2024-04-20T21:00:12.986 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-20T21:00:12.987 DEBUG:teuthology.run:Teuthology command: teuthology --description fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} --verbose --name rishabh-2024-04-20_20:50:37-fs-wip-rishabh-testing-20240416.193735-testing-default-smithi --archive /home/teuthworker/archive/rishabh-2024-04-20_20:50:37-fs-wip-rishabh-testing-20240416.193735-testing-default-smithi/7665388 --owner scheduled_rishabh@teuthology -- /home/teuthworker/archive/rishabh-2024-04-20_20:50:37-fs-wip-rishabh-testing-20240416.193735-testing-default-smithi/7665388/orig.config.yaml 2024-04-20T21:00:13.042 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-20T21:00:13.128 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/rishabh-2024-04-20_20:50:37-fs-wip-rishabh-testing-20240416.193735-testing-default-smithi/7665388 branch: wip-rishabh-testing-20240416.193735 description: fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} email: null first_in_suite: false job_id: '7665388' kernel: flavor: default kdb: true sha1: 806ec36b21129f113ce356c007dd59694d340b37 last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/v18.2.0 ' name: rishabh-2024-04-20_20:50:37-fs-wip-rishabh-testing-20240416.193735-testing-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-rishabh-testing-20240416.193735 ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - OSD_DOWN - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability sha1: a8c84f5997950fd7d008204839253291d52e6eda ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: a8c84f5997950fd7d008204839253291d52e6eda kclient: syntax: v1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-rishabh-testing-20240416.193735 sha1: a8c84f5997950fd7d008204839253291d52e6eda owner: scheduled_rishabh@teuthology priority: 75 repo: https://git.ceph.com/ceph-ci.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 7954 sha1: a8c84f5997950fd7d008204839253291d52e6eda sleep_before_teardown: 0 subset: 364/512 suite: fs suite_branch: wip-rishabh-testing-20240416.193735 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_a8c84f5997950fd7d008204839253291d52e6eda/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: a8c84f5997950fd7d008204839253291d52e6eda targets: smithi062.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFWpe9SJsDurBtnuJZcMu88UOJClsGajFv7glq/3P42ZmjzSSU8PSr3FioBnSz6RNYQ5UHRSycxlgSv0tNeaubI= smithi189.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBG1JxUynIoCSQeK70sAnxX+sG5vDLI88pwfpaS1csAIL2TcSKN7hwWd/eHA8/2yTiI7gnIpEtIoQkvwNeb5Rjks= tasks: - install: exclude_packages: - ceph-volume tag: v18.2.0 - print: '**** done install task...' - cephadm: compiled_cephadm_branch: reef conf: osd: osd_class_default_list: '*' osd_class_load_list: '*' image: quay.io/ceph/ceph:v18.2.0 roleless: true - print: '**** done end installing v18.2.0 cephadm ...' - cephadm.shell: host.a: - ceph config set mgr mgr/cephadm/use_repo_digest true --force - print: '**** done cephadm.shell ceph config set mgr...' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 2 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay true - cephadm.shell: host.a: - ceph fs set cephfs inline_data true --yes-i-really-really-mean-it - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - ceph-fuse: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: [] meta: {} postmerge: [] variables: fail_fs: true teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-20_20:50:37 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph versions | jq -e '.mgr | length == 1' - ceph versions | jq -e '.mgr | keys' | grep $sha1 - ceph versions | jq -e '.overall | length == 2' - ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '.up_to_date | length == 2' - ceph orch ps - cephadm.shell: env: - sha1 host.a: - ceph config set mgr mgr/orchestrator/fail_fs true - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph orch upgrade status - ceph health detail - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: rishabh verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2024-04-20T21:00:13.128 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_a8c84f5997950fd7d008204839253291d52e6eda/qa; will attempt to use it 2024-04-20T21:00:13.129 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_a8c84f5997950fd7d008204839253291d52e6eda/qa/tasks 2024-04-20T21:00:13.129 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-20T21:00:13.131 INFO:teuthology.task.internal:Checking packages... 2024-04-20T21:00:13.152 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'a8c84f5997950fd7d008204839253291d52e6eda' 2024-04-20T21:00:13.152 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-20T21:00:13.152 INFO:teuthology.packaging:ref: None 2024-04-20T21:00:13.152 INFO:teuthology.packaging:tag: None 2024-04-20T21:00:13.152 INFO:teuthology.packaging:branch: wip-rishabh-testing-20240416.193735 2024-04-20T21:00:13.152 INFO:teuthology.packaging:sha1: a8c84f5997950fd7d008204839253291d52e6eda 2024-04-20T21:00:13.153 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-rishabh-testing-20240416.193735 2024-04-20T21:00:13.417 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3133.ga8c84f59 2024-04-20T21:00:13.418 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-20T21:00:13.426 INFO:teuthology.task.internal:no buildpackages task found 2024-04-20T21:00:13.426 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-20T21:00:13.442 INFO:teuthology.task.internal:Saving configuration 2024-04-20T21:00:13.462 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-20T21:00:13.468 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-20T21:00:13.496 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi062.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-04-20_20:50:37-fs-wip-rishabh-testing-20240416.193735-testing-default-smithi/7665388', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-20 20:51:38.764805', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFWpe9SJsDurBtnuJZcMu88UOJClsGajFv7glq/3P42ZmjzSSU8PSr3FioBnSz6RNYQ5UHRSycxlgSv0tNeaubI='} 2024-04-20T21:00:13.519 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi189.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-04-20_20:50:37-fs-wip-rishabh-testing-20240416.193735-testing-default-smithi/7665388', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-20 20:51:38.769118', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBG1JxUynIoCSQeK70sAnxX+sG5vDLI88pwfpaS1csAIL2TcSKN7hwWd/eHA8/2yTiI7gnIpEtIoQkvwNeb5Rjks='} 2024-04-20T21:00:13.520 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-20T21:00:13.526 INFO:teuthology.task.internal:roles: ubuntu@smithi062.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2024-04-20T21:00:13.526 INFO:teuthology.task.internal:roles: ubuntu@smithi189.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2024-04-20T21:00:13.526 INFO:teuthology.run_tasks:Running task console_log... 2024-04-20T21:00:13.638 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fe8d0ee0160>, signals=[15]) 2024-04-20T21:00:13.639 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-20T21:00:13.645 INFO:teuthology.task.internal:Opening connections... 2024-04-20T21:00:13.645 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi062.front.sepia.ceph.com 2024-04-20T21:00:13.647 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi062.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-20T21:00:13.725 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi189.front.sepia.ceph.com 2024-04-20T21:00:13.726 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-20T21:00:13.798 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-20T21:00:13.805 DEBUG:teuthology.orchestra.run.smithi062:> uname -m 2024-04-20T21:00:13.824 INFO:teuthology.orchestra.run.smithi062.stdout:x86_64 2024-04-20T21:00:13.824 DEBUG:teuthology.orchestra.run.smithi062:> cat /etc/os-release 2024-04-20T21:00:13.881 INFO:teuthology.orchestra.run.smithi062.stdout:NAME="CentOS Stream" 2024-04-20T21:00:13.881 INFO:teuthology.orchestra.run.smithi062.stdout:VERSION="9" 2024-04-20T21:00:13.882 INFO:teuthology.orchestra.run.smithi062.stdout:ID="centos" 2024-04-20T21:00:13.882 INFO:teuthology.orchestra.run.smithi062.stdout:ID_LIKE="rhel fedora" 2024-04-20T21:00:13.882 INFO:teuthology.orchestra.run.smithi062.stdout:VERSION_ID="9" 2024-04-20T21:00:13.882 INFO:teuthology.orchestra.run.smithi062.stdout:PLATFORM_ID="platform:el9" 2024-04-20T21:00:13.882 INFO:teuthology.orchestra.run.smithi062.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-20T21:00:13.882 INFO:teuthology.orchestra.run.smithi062.stdout:ANSI_COLOR="0;31" 2024-04-20T21:00:13.882 INFO:teuthology.orchestra.run.smithi062.stdout:LOGO="fedora-logo-icon" 2024-04-20T21:00:13.882 INFO:teuthology.orchestra.run.smithi062.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-20T21:00:13.882 INFO:teuthology.orchestra.run.smithi062.stdout:HOME_URL="https://centos.org/" 2024-04-20T21:00:13.882 INFO:teuthology.orchestra.run.smithi062.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-20T21:00:13.882 INFO:teuthology.orchestra.run.smithi062.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-20T21:00:13.882 INFO:teuthology.orchestra.run.smithi062.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-20T21:00:13.883 INFO:teuthology.lock.ops:Updating smithi062.front.sepia.ceph.com on lock server 2024-04-20T21:00:13.906 DEBUG:teuthology.orchestra.run.smithi189:> uname -m 2024-04-20T21:00:13.924 INFO:teuthology.orchestra.run.smithi189.stdout:x86_64 2024-04-20T21:00:13.924 DEBUG:teuthology.orchestra.run.smithi189:> cat /etc/os-release 2024-04-20T21:00:13.980 INFO:teuthology.orchestra.run.smithi189.stdout:NAME="CentOS Stream" 2024-04-20T21:00:13.980 INFO:teuthology.orchestra.run.smithi189.stdout:VERSION="9" 2024-04-20T21:00:13.980 INFO:teuthology.orchestra.run.smithi189.stdout:ID="centos" 2024-04-20T21:00:13.980 INFO:teuthology.orchestra.run.smithi189.stdout:ID_LIKE="rhel fedora" 2024-04-20T21:00:13.981 INFO:teuthology.orchestra.run.smithi189.stdout:VERSION_ID="9" 2024-04-20T21:00:13.981 INFO:teuthology.orchestra.run.smithi189.stdout:PLATFORM_ID="platform:el9" 2024-04-20T21:00:13.981 INFO:teuthology.orchestra.run.smithi189.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-20T21:00:13.981 INFO:teuthology.orchestra.run.smithi189.stdout:ANSI_COLOR="0;31" 2024-04-20T21:00:13.981 INFO:teuthology.orchestra.run.smithi189.stdout:LOGO="fedora-logo-icon" 2024-04-20T21:00:13.981 INFO:teuthology.orchestra.run.smithi189.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-20T21:00:13.981 INFO:teuthology.orchestra.run.smithi189.stdout:HOME_URL="https://centos.org/" 2024-04-20T21:00:13.981 INFO:teuthology.orchestra.run.smithi189.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-20T21:00:13.981 INFO:teuthology.orchestra.run.smithi189.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-20T21:00:13.981 INFO:teuthology.orchestra.run.smithi189.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-20T21:00:13.981 INFO:teuthology.lock.ops:Updating smithi189.front.sepia.ceph.com on lock server 2024-04-20T21:00:14.014 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-20T21:00:14.024 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-20T21:00:14.029 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-20T21:00:14.030 DEBUG:teuthology.orchestra.run.smithi062:> test '!' -e /home/ubuntu/cephtest 2024-04-20T21:00:14.032 DEBUG:teuthology.orchestra.run.smithi189:> test '!' -e /home/ubuntu/cephtest 2024-04-20T21:00:14.048 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-20T21:00:14.054 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-20T21:00:14.054 DEBUG:teuthology.orchestra.run.smithi062:> test -z $(ls -A /var/lib/ceph) 2024-04-20T21:00:14.089 DEBUG:teuthology.orchestra.run.smithi189:> test -z $(ls -A /var/lib/ceph) 2024-04-20T21:00:14.113 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-20T21:00:14.169 INFO:teuthology.run_tasks:Running task kernel... 2024-04-20T21:00:14.183 INFO:teuthology.task.kernel:normalize config orig: {'flavor': 'default', 'kdb': True, 'sha1': '806ec36b21129f113ce356c007dd59694d340b37'} 2024-04-20T21:00:14.183 INFO:teuthology.task.kernel:config {'host.a': {'flavor': 'default', 'kdb': True, 'sha1': '806ec36b21129f113ce356c007dd59694d340b37'}, 'host.b': {'flavor': 'default', 'kdb': True, 'sha1': '806ec36b21129f113ce356c007dd59694d340b37'}}, timeout 300 2024-04-20T21:00:14.183 DEBUG:teuthology.orchestra.run.smithi062:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-20T21:00:14.184 DEBUG:teuthology.orchestra.run.smithi189:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-20T21:00:14.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-20T21:00:14.199 DEBUG:teuthology.task.kernel:sha1 for host.b is 806ec36b21129f113ce356c007dd59694d340b37 2024-04-20T21:00:14.199 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "806ec36b21129f113ce356c007dd59694d340b37"... 2024-04-20T21:00:14.199 DEBUG:teuthology.orchestra.run.smithi189:> uname -r 2024-04-20T21:00:14.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-20T21:00:14.201 DEBUG:teuthology.task.kernel:sha1 for host.a is 806ec36b21129f113ce356c007dd59694d340b37 2024-04-20T21:00:14.201 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "806ec36b21129f113ce356c007dd59694d340b37"... 2024-04-20T21:00:14.201 DEBUG:teuthology.orchestra.run.smithi062:> uname -r 2024-04-20T21:00:14.255 INFO:teuthology.orchestra.run.smithi189.stdout:5.14.0-437.el9.x86_64 2024-04-20T21:00:14.255 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-437.el9.x86_64 vs 806ec36b21129f113ce356c007dd59694d340b37 2024-04-20T21:00:14.255 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-04-20T21:00:14.256 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=806ec36b21129f113ce356c007dd59694d340b37 2024-04-20T21:00:14.260 INFO:teuthology.orchestra.run.smithi062.stdout:5.14.0-437.el9.x86_64 2024-04-20T21:00:14.260 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-437.el9.x86_64 vs 806ec36b21129f113ce356c007dd59694d340b37 2024-04-20T21:00:14.260 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-04-20T21:00:14.260 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=806ec36b21129f113ce356c007dd59694d340b37 2024-04-20T21:00:14.524 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum upgrade -y linux-firmware 2024-04-20T21:00:14.527 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum upgrade -y linux-firmware 2024-04-20T21:00:17.262 INFO:teuthology.orchestra.run.smithi062.stdout:CentOS Stream 9 - BaseOS 6.6 MB/s | 8.0 MB 00:01 2024-04-20T21:00:18.758 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS Stream 9 - BaseOS 3.1 MB/s | 8.0 MB 00:02 2024-04-20T21:00:20.240 INFO:teuthology.orchestra.run.smithi062.stdout:CentOS Stream 9 - AppStream 10 MB/s | 19 MB 00:01 2024-04-20T21:00:24.007 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS Stream 9 - AppStream 4.5 MB/s | 19 MB 00:04 2024-04-20T21:00:24.355 INFO:teuthology.orchestra.run.smithi062.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.2 MB 00:00 2024-04-20T21:00:26.248 INFO:teuthology.orchestra.run.smithi062.stdout:CentOS Stream 9 - Extras packages 23 kB/s | 16 kB 00:00 2024-04-20T21:00:28.182 INFO:teuthology.orchestra.run.smithi062.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-04-20T21:00:33.465 INFO:teuthology.orchestra.run.smithi062.stdout:lab-extras 40 kB/s | 1.7 kB 00:00 2024-04-20T21:00:35.508 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2024-04-20T21:00:35.557 INFO:teuthology.orchestra.run.smithi062.stdout:Nothing to do. 2024-04-20T21:00:35.557 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2024-04-20T21:00:35.621 INFO:teuthology.task.kernel:Downloading kernel 806ec36b21129f113ce356c007dd59694d340b37 on host.a... 2024-04-20T21:00:35.621 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=806ec36b21129f113ce356c007dd59694d340b37 2024-04-20T21:00:35.897 INFO:teuthology.task.kernel:fetching, builder baseurl is https://4.chacra.ceph.com/r/kernel/testing/806ec36b21129f113ce356c007dd59694d340b37/centos/9/flavors/default/x86_64/ 2024-04-20T21:00:35.898 DEBUG:teuthology.orchestra.run.smithi062:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.9.0_rc3_g806ec36b2112-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://4.chacra.ceph.com/r/kernel/testing/806ec36b21129f113ce356c007dd59694d340b37/centos/9/flavors/default/x86_64/ --input-file=- 2024-04-20T21:00:35.900 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi062.front.sepia.ceph.com to complete... 2024-04-20T21:00:36.157 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS Stream 9 - CRB 738 kB/s | 6.2 MB 00:08 2024-04-20T21:00:37.890 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS Stream 9 - Extras packages 41 kB/s | 16 kB 00:00 2024-04-20T21:00:39.300 INFO:teuthology.orchestra.run.smithi189.stdout:Extra Packages for Enterprise Linux 16 MB/s | 21 MB 00:01 2024-04-20T21:00:44.333 INFO:teuthology.orchestra.run.smithi189.stdout:lab-extras 15 kB/s | 1.7 kB 00:00 2024-04-20T21:00:46.312 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-04-20T21:00:46.313 INFO:teuthology.orchestra.run.smithi189.stdout:Nothing to do. 2024-04-20T21:00:46.314 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-04-20T21:00:46.431 INFO:teuthology.task.kernel:Downloading kernel 806ec36b21129f113ce356c007dd59694d340b37 on host.b... 2024-04-20T21:00:46.431 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=806ec36b21129f113ce356c007dd59694d340b37 2024-04-20T21:00:46.654 INFO:teuthology.task.kernel:fetching, builder baseurl is https://4.chacra.ceph.com/r/kernel/testing/806ec36b21129f113ce356c007dd59694d340b37/centos/9/flavors/default/x86_64/ 2024-04-20T21:00:46.654 DEBUG:teuthology.orchestra.run.smithi189:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.9.0_rc3_g806ec36b2112-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://4.chacra.ceph.com/r/kernel/testing/806ec36b21129f113ce356c007dd59694d340b37/centos/9/flavors/default/x86_64/ --input-file=- 2024-04-20T21:00:46.656 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi189.front.sepia.ceph.com to complete... 2024-04-20T21:07:55.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20 21:07:55 URL:https://4.chacra.ceph.com/r/kernel/testing/806ec36b21129f113ce356c007dd59694d340b37/centos/9/flavors/default/x86_64/kernel-6.9.0_rc3_g806ec36b2112-1.x86_64.rpm [357367792/357367792] -> "/tmp/kernel.x86_64.rpm" [1] 2024-04-20T21:07:55.329 INFO:teuthology.orchestra.run.smithi062.stderr:FINISHED --2024-04-20 21:07:55-- 2024-04-20T21:07:55.329 INFO:teuthology.orchestra.run.smithi062.stderr:Total wall clock time: 7m 19s 2024-04-20T21:07:55.329 INFO:teuthology.orchestra.run.smithi062.stderr:Downloaded: 1 files, 341M in 7m 19s (795 KB/s) 2024-04-20T21:07:55.332 INFO:teuthology.task.kernel:Installing kernel 806ec36b21129f113ce356c007dd59694d340b37 on host.a... 2024-04-20T21:07:55.332 DEBUG:teuthology.orchestra.run.smithi062:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-04-20T21:07:57.513 INFO:teuthology.orchestra.run.smithi062.stdout:Verifying... ######################################## 2024-04-20T21:07:57.644 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing... ######################################## 2024-04-20T21:07:57.759 INFO:teuthology.orchestra.run.smithi062.stdout:Updating / installing... 2024-04-20T21:08:06.097 INFO:teuthology.orchestra.run.smithi062.stdout:kernel-6.9.0_rc3_g806ec36b2112-1 ######################################## 2024-04-20T21:08:44.027 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi062.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-04-20T21:08:44.027 DEBUG:teuthology.orchestra.run.smithi062:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-04-20T21:08:44.094 INFO:teuthology.orchestra.run.smithi062.stdout:/boot/System.map-6.9.0-rc3-g806ec36b2112 2024-04-20T21:08:44.094 INFO:teuthology.orchestra.run.smithi062.stdout:/boot/config-6.9.0-rc3-g806ec36b2112 2024-04-20T21:08:44.094 INFO:teuthology.orchestra.run.smithi062.stdout:/boot/vmlinuz-6.9.0-rc3-g806ec36b2112 2024-04-20T21:08:44.094 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/System.map 2024-04-20T21:08:44.094 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/config 2024-04-20T21:08:44.095 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel 2024-04-20T21:08:44.095 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch 2024-04-20T21:08:44.095 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86 2024-04-20T21:08:44.095 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto 2024-04-20T21:08:44.095 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-04-20T21:08:44.095 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-04-20T21:08:44.095 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-04-20T21:08:44.095 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-04-20T21:08:44.095 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-04-20T21:08:44.095 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-04-20T21:08:44.095 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-04-20T21:08:44.095 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/crc32c-intel.ko 2024-04-20T21:08:44.095 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-04-20T21:08:44.096 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-04-20T21:08:44.096 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-04-20T21:08:44.096 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-04-20T21:08:44.096 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/serpent-avx2.ko 2024-04-20T21:08:44.096 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-04-20T21:08:44.096 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-04-20T21:08:44.096 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-04-20T21:08:44.096 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-04-20T21:08:44.096 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-04-20T21:08:44.096 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-04-20T21:08:44.096 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-04-20T21:08:44.096 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel 2024-04-20T21:08:44.096 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel/cpu 2024-04-20T21:08:44.096 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel/cpu/mce 2024-04-20T21:08:44.096 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-04-20T21:08:44.097 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm 2024-04-20T21:08:44.097 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm/kvm-amd.ko 2024-04-20T21:08:44.097 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm/kvm-intel.ko 2024-04-20T21:08:44.097 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm/kvm.ko 2024-04-20T21:08:44.097 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto 2024-04-20T21:08:44.097 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/ansi_cprng.ko 2024-04-20T21:08:44.097 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/anubis.ko 2024-04-20T21:08:44.097 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/arc4.ko 2024-04-20T21:08:44.097 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx 2024-04-20T21:08:44.097 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_memcpy.ko 2024-04-20T21:08:44.097 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_pq.ko 2024-04-20T21:08:44.097 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_raid6_recov.ko 2024-04-20T21:08:44.097 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_tx.ko 2024-04-20T21:08:44.097 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_xor.ko 2024-04-20T21:08:44.097 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/raid6test.ko 2024-04-20T21:08:44.098 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/authenc.ko 2024-04-20T21:08:44.098 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/authencesn.ko 2024-04-20T21:08:44.098 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/blake2b_generic.ko 2024-04-20T21:08:44.098 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/blowfish_common.ko 2024-04-20T21:08:44.098 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/blowfish_generic.ko 2024-04-20T21:08:44.098 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/camellia_generic.ko 2024-04-20T21:08:44.098 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cast5_generic.ko 2024-04-20T21:08:44.098 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cast6_generic.ko 2024-04-20T21:08:44.098 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cast_common.ko 2024-04-20T21:08:44.098 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/ccm.ko 2024-04-20T21:08:44.098 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cmac.ko 2024-04-20T21:08:44.098 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/crc32_generic.ko 2024-04-20T21:08:44.098 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/crypto_engine.ko 2024-04-20T21:08:44.099 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/crypto_user.ko 2024-04-20T21:08:44.099 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cts.ko 2024-04-20T21:08:44.099 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/des_generic.ko 2024-04-20T21:08:44.099 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/dh_generic.ko 2024-04-20T21:08:44.099 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/echainiv.ko 2024-04-20T21:08:44.099 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/essiv.ko 2024-04-20T21:08:44.099 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/fcrypt.ko 2024-04-20T21:08:44.099 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/khazad.ko 2024-04-20T21:08:44.099 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/lz4.ko 2024-04-20T21:08:44.099 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/lz4hc.ko 2024-04-20T21:08:44.099 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/md4.ko 2024-04-20T21:08:44.099 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/michael_mic.ko 2024-04-20T21:08:44.099 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/pcbc.ko 2024-04-20T21:08:44.099 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/pcrypt.ko 2024-04-20T21:08:44.099 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/rmd160.ko 2024-04-20T21:08:44.100 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/seed.ko 2024-04-20T21:08:44.100 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/serpent_generic.ko 2024-04-20T21:08:44.100 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/tcrypt.ko 2024-04-20T21:08:44.100 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/tea.ko 2024-04-20T21:08:44.100 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/twofish_common.ko 2024-04-20T21:08:44.100 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/twofish_generic.ko 2024-04-20T21:08:44.100 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/vmac.ko 2024-04-20T21:08:44.100 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/wp512.ko 2024-04-20T21:08:44.100 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/xcbc.ko 2024-04-20T21:08:44.100 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/xor.ko 2024-04-20T21:08:44.100 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/xxhash_generic.ko 2024-04-20T21:08:44.100 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers 2024-04-20T21:08:44.100 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi 2024-04-20T21:08:44.100 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/acpi_ipmi.ko 2024-04-20T21:08:44.101 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/acpi_pad.ko 2024-04-20T21:08:44.101 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/ec_sys.ko 2024-04-20T21:08:44.101 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/platform_profile.ko 2024-04-20T21:08:44.101 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/sbs.ko 2024-04-20T21:08:44.101 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/sbshc.ko 2024-04-20T21:08:44.101 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/video.ko 2024-04-20T21:08:44.101 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata 2024-04-20T21:08:44.101 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/acard-ahci.ko 2024-04-20T21:08:44.101 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/ahci_platform.ko 2024-04-20T21:08:44.101 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/ata_generic.ko 2024-04-20T21:08:44.101 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/libahci_platform.ko 2024-04-20T21:08:44.101 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_acpi.ko 2024-04-20T21:08:44.101 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ali.ko 2024-04-20T21:08:44.101 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_amd.ko 2024-04-20T21:08:44.101 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_artop.ko 2024-04-20T21:08:44.101 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_atiixp.ko 2024-04-20T21:08:44.102 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_atp867x.ko 2024-04-20T21:08:44.102 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_cmd640.ko 2024-04-20T21:08:44.102 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_cmd64x.ko 2024-04-20T21:08:44.102 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_cypress.ko 2024-04-20T21:08:44.102 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_efar.ko 2024-04-20T21:08:44.102 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt366.ko 2024-04-20T21:08:44.102 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt37x.ko 2024-04-20T21:08:44.102 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt3x2n.ko 2024-04-20T21:08:44.102 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt3x3.ko 2024-04-20T21:08:44.102 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_it8213.ko 2024-04-20T21:08:44.102 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_it821x.ko 2024-04-20T21:08:44.102 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_jmicron.ko 2024-04-20T21:08:44.102 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_marvell.ko 2024-04-20T21:08:44.102 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_mpiix.ko 2024-04-20T21:08:44.103 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_netcell.ko 2024-04-20T21:08:44.103 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ninja32.ko 2024-04-20T21:08:44.103 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ns87410.ko 2024-04-20T21:08:44.103 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ns87415.ko 2024-04-20T21:08:44.103 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_oldpiix.ko 2024-04-20T21:08:44.103 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_opti.ko 2024-04-20T21:08:44.103 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_optidma.ko 2024-04-20T21:08:44.103 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_pcmcia.ko 2024-04-20T21:08:44.103 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_pdc2027x.ko 2024-04-20T21:08:44.103 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-04-20T21:08:44.103 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_piccolo.ko 2024-04-20T21:08:44.103 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_rdc.ko 2024-04-20T21:08:44.103 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sch.ko 2024-04-20T21:08:44.103 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_serverworks.ko 2024-04-20T21:08:44.104 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sil680.ko 2024-04-20T21:08:44.104 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sis.ko 2024-04-20T21:08:44.104 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sl82c105.ko 2024-04-20T21:08:44.104 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_triflex.ko 2024-04-20T21:08:44.104 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_via.ko 2024-04-20T21:08:44.104 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pdc_adma.ko 2024-04-20T21:08:44.104 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_inic162x.ko 2024-04-20T21:08:44.104 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_mv.ko 2024-04-20T21:08:44.104 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_nv.ko 2024-04-20T21:08:44.104 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_promise.ko 2024-04-20T21:08:44.104 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_qstor.ko 2024-04-20T21:08:44.104 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sil.ko 2024-04-20T21:08:44.104 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sil24.ko 2024-04-20T21:08:44.104 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sis.ko 2024-04-20T21:08:44.104 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_svw.ko 2024-04-20T21:08:44.105 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sx4.ko 2024-04-20T21:08:44.105 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_uli.ko 2024-04-20T21:08:44.105 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_via.ko 2024-04-20T21:08:44.105 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_vsc.ko 2024-04-20T21:08:44.105 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/auxdisplay 2024-04-20T21:08:44.105 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/auxdisplay/ks0108.ko 2024-04-20T21:08:44.105 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base 2024-04-20T21:08:44.105 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base/regmap 2024-04-20T21:08:44.105 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base/regmap/regmap-i2c.ko 2024-04-20T21:08:44.105 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base/regmap/regmap-mmio.ko 2024-04-20T21:08:44.105 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/bcma 2024-04-20T21:08:44.105 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/bcma/bcma.ko 2024-04-20T21:08:44.105 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block 2024-04-20T21:08:44.105 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/aoe 2024-04-20T21:08:44.105 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/aoe/aoe.ko 2024-04-20T21:08:44.105 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/brd.ko 2024-04-20T21:08:44.106 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/drbd 2024-04-20T21:08:44.106 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/drbd/drbd.ko 2024-04-20T21:08:44.106 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/floppy.ko 2024-04-20T21:08:44.106 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/loop.ko 2024-04-20T21:08:44.106 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/mtip32xx 2024-04-20T21:08:44.106 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-04-20T21:08:44.106 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/nbd.ko 2024-04-20T21:08:44.106 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/null_blk 2024-04-20T21:08:44.106 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/null_blk/null_blk.ko 2024-04-20T21:08:44.106 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/pktcdvd.ko 2024-04-20T21:08:44.106 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/rbd.ko 2024-04-20T21:08:44.106 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/virtio_blk.ko 2024-04-20T21:08:44.106 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/xen-blkback 2024-04-20T21:08:44.106 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-04-20T21:08:44.107 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/xen-blkfront.ko 2024-04-20T21:08:44.107 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/zram 2024-04-20T21:08:44.107 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/zram/zram.ko 2024-04-20T21:08:44.107 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char 2024-04-20T21:08:44.107 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hangcheck-timer.ko 2024-04-20T21:08:44.107 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random 2024-04-20T21:08:44.107 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/amd-rng.ko 2024-04-20T21:08:44.107 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/intel-rng.ko 2024-04-20T21:08:44.107 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-04-20T21:08:44.107 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/via-rng.ko 2024-04-20T21:08:44.107 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/virtio-rng.ko 2024-04-20T21:08:44.107 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi 2024-04-20T21:08:44.107 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-04-20T21:08:44.107 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-04-20T21:08:44.107 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-04-20T21:08:44.108 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_si.ko 2024-04-20T21:08:44.108 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-04-20T21:08:44.108 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-04-20T21:08:44.108 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/lp.ko 2024-04-20T21:08:44.108 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/mwave 2024-04-20T21:08:44.108 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/mwave/mwave.ko 2024-04-20T21:08:44.108 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ppdev.ko 2024-04-20T21:08:44.108 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tlclk.ko 2024-04-20T21:08:44.108 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm 2024-04-20T21:08:44.108 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm.ko 2024-04-20T21:08:44.108 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_atmel.ko 2024-04-20T21:08:44.108 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_infineon.ko 2024-04-20T21:08:44.108 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_nsc.ko 2024-04-20T21:08:44.108 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_tis.ko 2024-04-20T21:08:44.108 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-04-20T21:08:44.108 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/uv_mmtimer.ko 2024-04-20T21:08:44.109 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/virtio_console.ko 2024-04-20T21:08:44.109 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq 2024-04-20T21:08:44.109 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-04-20T21:08:44.109 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-04-20T21:08:44.109 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/p4-clockmod.ko 2024-04-20T21:08:44.109 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-04-20T21:08:44.109 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/powernow-k8.ko 2024-04-20T21:08:44.109 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/speedstep-lib.ko 2024-04-20T21:08:44.109 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto 2024-04-20T21:08:44.109 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/ccp 2024-04-20T21:08:44.109 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-04-20T21:08:44.109 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/ccp/ccp.ko 2024-04-20T21:08:44.109 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel 2024-04-20T21:08:44.109 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat 2024-04-20T21:08:44.109 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_common 2024-04-20T21:08:44.110 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-04-20T21:08:44.110 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-04-20T21:08:44.110 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-04-20T21:08:44.110 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/padlock-aes.ko 2024-04-20T21:08:44.110 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/padlock-sha.ko 2024-04-20T21:08:44.110 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/virtio 2024-04-20T21:08:44.110 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-04-20T21:08:44.110 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dca 2024-04-20T21:08:44.110 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dca/dca.ko 2024-04-20T21:08:44.110 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/devfreq 2024-04-20T21:08:44.110 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-04-20T21:08:44.110 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma 2024-04-20T21:08:44.110 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/dw 2024-04-20T21:08:44.110 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/dw/dw_dmac.ko 2024-04-20T21:08:44.111 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/ioat 2024-04-20T21:08:44.111 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/ioat/ioatdma.ko 2024-04-20T21:08:44.111 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac 2024-04-20T21:08:44.111 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/amd64_edac.ko 2024-04-20T21:08:44.111 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/e752x_edac.ko 2024-04-20T21:08:44.111 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/edac_mce_amd.ko 2024-04-20T21:08:44.111 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i3000_edac.ko 2024-04-20T21:08:44.111 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i3200_edac.ko 2024-04-20T21:08:44.111 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i5100_edac.ko 2024-04-20T21:08:44.111 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i5400_edac.ko 2024-04-20T21:08:44.111 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i7300_edac.ko 2024-04-20T21:08:44.111 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i7core_edac.ko 2024-04-20T21:08:44.111 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i82975x_edac.ko 2024-04-20T21:08:44.111 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/ie31200_edac.ko 2024-04-20T21:08:44.111 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/sb_edac.ko 2024-04-20T21:08:44.112 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/x38_edac.ko 2024-04-20T21:08:44.112 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/firmware 2024-04-20T21:08:44.112 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/firmware/edd.ko 2024-04-20T21:08:44.112 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/firmware/iscsi_ibft.ko 2024-04-20T21:08:44.112 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/gpio 2024-04-20T21:08:44.112 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/gpio/gpio-ich.ko 2024-04-20T21:08:44.112 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/gpio/gpio-viperboard.ko 2024-04-20T21:08:44.112 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid 2024-04-20T21:08:44.112 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-appleir.ko 2024-04-20T21:08:44.112 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-aureal.ko 2024-04-20T21:08:44.112 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-axff.ko 2024-04-20T21:08:44.112 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-dr.ko 2024-04-20T21:08:44.112 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-elecom.ko 2024-04-20T21:08:44.112 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-elo.ko 2024-04-20T21:08:44.112 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-emsff.ko 2024-04-20T21:08:44.113 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-gaff.ko 2024-04-20T21:08:44.113 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-gt683r.ko 2024-04-20T21:08:44.113 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-gyration.ko 2024-04-20T21:08:44.113 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-holtek-kbd.ko 2024-04-20T21:08:44.113 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-holtek-mouse.ko 2024-04-20T21:08:44.113 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-holtekff.ko 2024-04-20T21:08:44.113 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-hyperv.ko 2024-04-20T21:08:44.113 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-icade.ko 2024-04-20T21:08:44.113 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-keytouch.ko 2024-04-20T21:08:44.113 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-kye.ko 2024-04-20T21:08:44.113 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-lcpower.ko 2024-04-20T21:08:44.113 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-led.ko 2024-04-20T21:08:44.113 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-lenovo.ko 2024-04-20T21:08:44.113 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-logitech-dj.ko 2024-04-20T21:08:44.113 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-04-20T21:08:44.114 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-multitouch.ko 2024-04-20T21:08:44.114 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-ortek.ko 2024-04-20T21:08:44.114 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-penmount.ko 2024-04-20T21:08:44.114 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-petalynx.ko 2024-04-20T21:08:44.114 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-picolcd.ko 2024-04-20T21:08:44.114 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-pl.ko 2024-04-20T21:08:44.114 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-plantronics.ko 2024-04-20T21:08:44.114 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-primax.ko 2024-04-20T21:08:44.114 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-rmi.ko 2024-04-20T21:08:44.114 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-arvo.ko 2024-04-20T21:08:44.114 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-common.ko 2024-04-20T21:08:44.114 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-isku.ko 2024-04-20T21:08:44.114 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-kone.ko 2024-04-20T21:08:44.114 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-04-20T21:08:44.115 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-konepure.ko 2024-04-20T21:08:44.115 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-04-20T21:08:44.115 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-lua.ko 2024-04-20T21:08:44.115 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-pyra.ko 2024-04-20T21:08:44.115 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-ryos.ko 2024-04-20T21:08:44.115 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-savu.ko 2024-04-20T21:08:44.115 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat.ko 2024-04-20T21:08:44.115 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-saitek.ko 2024-04-20T21:08:44.115 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-samsung.ko 2024-04-20T21:08:44.115 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sensor-hub.ko 2024-04-20T21:08:44.115 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sjoy.ko 2024-04-20T21:08:44.115 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sony.ko 2024-04-20T21:08:44.115 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-speedlink.ko 2024-04-20T21:08:44.115 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-steelseries.ko 2024-04-20T21:08:44.115 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sunplus.ko 2024-04-20T21:08:44.116 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-thrustmaster.ko 2024-04-20T21:08:44.116 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-tivo.ko 2024-04-20T21:08:44.116 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-tmff.ko 2024-04-20T21:08:44.116 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-topseed.ko 2024-04-20T21:08:44.116 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-twinhan.ko 2024-04-20T21:08:44.116 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-uclogic.ko 2024-04-20T21:08:44.116 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-waltop.ko 2024-04-20T21:08:44.116 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-wiimote.ko 2024-04-20T21:08:44.116 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-xinmo.ko 2024-04-20T21:08:44.116 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-zpff.ko 2024-04-20T21:08:44.116 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-zydacron.ko 2024-04-20T21:08:44.116 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/uhid.ko 2024-04-20T21:08:44.116 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/wacom.ko 2024-04-20T21:08:44.116 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv 2024-04-20T21:08:44.116 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv/hv_balloon.ko 2024-04-20T21:08:44.116 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv/hv_utils.ko 2024-04-20T21:08:44.117 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv/hv_vmbus.ko 2024-04-20T21:08:44.117 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon 2024-04-20T21:08:44.117 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/abituguru.ko 2024-04-20T21:08:44.117 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/abituguru3.ko 2024-04-20T21:08:44.117 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/acpi_power_meter.ko 2024-04-20T21:08:44.117 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ad7414.ko 2024-04-20T21:08:44.117 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ad7418.ko 2024-04-20T21:08:44.117 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adc128d818.ko 2024-04-20T21:08:44.117 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1025.ko 2024-04-20T21:08:44.117 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1026.ko 2024-04-20T21:08:44.117 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1029.ko 2024-04-20T21:08:44.117 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1031.ko 2024-04-20T21:08:44.117 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm9240.ko 2024-04-20T21:08:44.117 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ads7828.ko 2024-04-20T21:08:44.117 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7410.ko 2024-04-20T21:08:44.117 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7411.ko 2024-04-20T21:08:44.118 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7462.ko 2024-04-20T21:08:44.118 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7470.ko 2024-04-20T21:08:44.118 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7475.ko 2024-04-20T21:08:44.118 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7x10.ko 2024-04-20T21:08:44.118 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/amc6821.ko 2024-04-20T21:08:44.118 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/applesmc.ko 2024-04-20T21:08:44.118 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/asb100.ko 2024-04-20T21:08:44.118 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/asc7621.ko 2024-04-20T21:08:44.118 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/asus_atk0110.ko 2024-04-20T21:08:44.118 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/atxp1.ko 2024-04-20T21:08:44.118 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/coretemp.ko 2024-04-20T21:08:44.118 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-04-20T21:08:44.118 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/dme1737.ko 2024-04-20T21:08:44.118 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ds1621.ko 2024-04-20T21:08:44.119 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ds620.ko 2024-04-20T21:08:44.119 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/emc1403.ko 2024-04-20T21:08:44.119 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/emc6w201.ko 2024-04-20T21:08:44.119 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/f71805f.ko 2024-04-20T21:08:44.119 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/f71882fg.ko 2024-04-20T21:08:44.119 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/f75375s.ko 2024-04-20T21:08:44.119 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/fam15h_power.ko 2024-04-20T21:08:44.119 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/fschmd.ko 2024-04-20T21:08:44.119 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/g760a.ko 2024-04-20T21:08:44.119 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/g762.ko 2024-04-20T21:08:44.119 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/gl518sm.ko 2024-04-20T21:08:44.119 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/gl520sm.ko 2024-04-20T21:08:44.119 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/hwmon-vid.ko 2024-04-20T21:08:44.119 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/i5500_temp.ko 2024-04-20T21:08:44.120 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/i5k_amb.ko 2024-04-20T21:08:44.120 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ibmaem.ko 2024-04-20T21:08:44.120 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ibmpex.ko 2024-04-20T21:08:44.120 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ina209.ko 2024-04-20T21:08:44.120 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ina2xx.ko 2024-04-20T21:08:44.120 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/it87.ko 2024-04-20T21:08:44.120 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/k10temp.ko 2024-04-20T21:08:44.120 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/k8temp.ko 2024-04-20T21:08:44.120 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lineage-pem.ko 2024-04-20T21:08:44.120 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm63.ko 2024-04-20T21:08:44.120 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm73.ko 2024-04-20T21:08:44.120 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm75.ko 2024-04-20T21:08:44.120 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm77.ko 2024-04-20T21:08:44.120 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm78.ko 2024-04-20T21:08:44.120 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm80.ko 2024-04-20T21:08:44.121 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm83.ko 2024-04-20T21:08:44.121 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm85.ko 2024-04-20T21:08:44.121 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm87.ko 2024-04-20T21:08:44.121 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm90.ko 2024-04-20T21:08:44.121 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm92.ko 2024-04-20T21:08:44.121 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm93.ko 2024-04-20T21:08:44.121 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm95234.ko 2024-04-20T21:08:44.121 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm95241.ko 2024-04-20T21:08:44.121 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm95245.ko 2024-04-20T21:08:44.121 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc2945.ko 2024-04-20T21:08:44.121 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4151.ko 2024-04-20T21:08:44.121 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4215.ko 2024-04-20T21:08:44.121 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4222.ko 2024-04-20T21:08:44.121 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4245.ko 2024-04-20T21:08:44.121 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4260.ko 2024-04-20T21:08:44.122 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4261.ko 2024-04-20T21:08:44.122 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max16065.ko 2024-04-20T21:08:44.122 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max1619.ko 2024-04-20T21:08:44.122 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max1668.ko 2024-04-20T21:08:44.122 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max197.ko 2024-04-20T21:08:44.122 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max6639.ko 2024-04-20T21:08:44.122 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max6650.ko 2024-04-20T21:08:44.122 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max6697.ko 2024-04-20T21:08:44.122 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/mcp3021.ko 2024-04-20T21:08:44.122 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct6683.ko 2024-04-20T21:08:44.122 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct6775-core.ko 2024-04-20T21:08:44.122 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct6775.ko 2024-04-20T21:08:44.122 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct7802.ko 2024-04-20T21:08:44.122 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pc87360.ko 2024-04-20T21:08:44.122 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pc87427.ko 2024-04-20T21:08:44.123 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pcf8591.ko 2024-04-20T21:08:44.123 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus 2024-04-20T21:08:44.123 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-04-20T21:08:44.123 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-04-20T21:08:44.123 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-04-20T21:08:44.123 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/max16064.ko 2024-04-20T21:08:44.123 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/max34440.ko 2024-04-20T21:08:44.123 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/max8688.ko 2024-04-20T21:08:44.123 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-04-20T21:08:44.123 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-04-20T21:08:44.123 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-04-20T21:08:44.123 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-04-20T21:08:44.123 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-04-20T21:08:44.123 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-04-20T21:08:44.123 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/powr1220.ko 2024-04-20T21:08:44.124 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sch5627.ko 2024-04-20T21:08:44.124 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sch5636.ko 2024-04-20T21:08:44.124 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sch56xx-common.ko 2024-04-20T21:08:44.124 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sht15.ko 2024-04-20T21:08:44.124 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sht21.ko 2024-04-20T21:08:44.124 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/shtc1.ko 2024-04-20T21:08:44.124 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sis5595.ko 2024-04-20T21:08:44.124 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/smsc47b397.ko 2024-04-20T21:08:44.124 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/smsc47m1.ko 2024-04-20T21:08:44.124 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/smsc47m192.ko 2024-04-20T21:08:44.124 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/thmc50.ko 2024-04-20T21:08:44.124 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp102.ko 2024-04-20T21:08:44.124 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp103.ko 2024-04-20T21:08:44.124 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp401.ko 2024-04-20T21:08:44.124 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp421.ko 2024-04-20T21:08:44.125 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/via-cputemp.ko 2024-04-20T21:08:44.125 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/via686a.ko 2024-04-20T21:08:44.125 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/vt1211.ko 2024-04-20T21:08:44.125 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/vt8231.ko 2024-04-20T21:08:44.125 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83627ehf.ko 2024-04-20T21:08:44.125 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83627hf.ko 2024-04-20T21:08:44.125 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83781d.ko 2024-04-20T21:08:44.125 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83791d.ko 2024-04-20T21:08:44.125 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83792d.ko 2024-04-20T21:08:44.125 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83793.ko 2024-04-20T21:08:44.125 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83795.ko 2024-04-20T21:08:44.125 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83l785ts.ko 2024-04-20T21:08:44.125 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83l786ng.ko 2024-04-20T21:08:44.125 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c 2024-04-20T21:08:44.125 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/algos 2024-04-20T21:08:44.125 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-04-20T21:08:44.126 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-04-20T21:08:44.126 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses 2024-04-20T21:08:44.126 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-04-20T21:08:44.126 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-04-20T21:08:44.126 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-04-20T21:08:44.126 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-04-20T21:08:44.126 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-04-20T21:08:44.126 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-04-20T21:08:44.126 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-04-20T21:08:44.126 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-04-20T21:08:44.126 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-i801.ko 2024-04-20T21:08:44.126 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-isch.ko 2024-04-20T21:08:44.126 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-04-20T21:08:44.126 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-04-20T21:08:44.127 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-04-20T21:08:44.127 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-parport.ko 2024-04-20T21:08:44.127 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-04-20T21:08:44.127 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-04-20T21:08:44.127 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-04-20T21:08:44.127 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-04-20T21:08:44.127 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-04-20T21:08:44.127 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-04-20T21:08:44.127 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-via.ko 2024-04-20T21:08:44.127 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-04-20T21:08:44.127 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-04-20T21:08:44.127 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-dev.ko 2024-04-20T21:08:44.127 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-mux.ko 2024-04-20T21:08:44.127 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-04-20T21:08:44.127 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-smbus.ko 2024-04-20T21:08:44.128 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-stub.ko 2024-04-20T21:08:44.128 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband 2024-04-20T21:08:44.128 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core 2024-04-20T21:08:44.128 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_cm.ko 2024-04-20T21:08:44.128 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_core.ko 2024-04-20T21:08:44.128 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_umad.ko 2024-04-20T21:08:44.128 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-04-20T21:08:44.128 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/iw_cm.ko 2024-04-20T21:08:44.128 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/rdma_cm.ko 2024-04-20T21:08:44.128 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-04-20T21:08:44.128 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw 2024-04-20T21:08:44.128 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/cxgb4 2024-04-20T21:08:44.128 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-04-20T21:08:44.128 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx4 2024-04-20T21:08:44.128 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-04-20T21:08:44.128 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx5 2024-04-20T21:08:44.129 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-04-20T21:08:44.129 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mthca 2024-04-20T21:08:44.129 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-04-20T21:08:44.129 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp 2024-04-20T21:08:44.129 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/ipoib 2024-04-20T21:08:44.129 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-04-20T21:08:44.129 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/iser 2024-04-20T21:08:44.129 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-04-20T21:08:44.129 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/isert 2024-04-20T21:08:44.129 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-04-20T21:08:44.129 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srp 2024-04-20T21:08:44.129 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-04-20T21:08:44.129 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srpt 2024-04-20T21:08:44.129 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-04-20T21:08:44.129 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input 2024-04-20T21:08:44.130 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport 2024-04-20T21:08:44.130 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-04-20T21:08:44.130 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport/fm801-gp.ko 2024-04-20T21:08:44.130 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport/gameport.ko 2024-04-20T21:08:44.130 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/joydev.ko 2024-04-20T21:08:44.130 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/keyboard 2024-04-20T21:08:44.130 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/keyboard/gpio_keys.ko 2024-04-20T21:08:44.130 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse 2024-04-20T21:08:44.130 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/appletouch.ko 2024-04-20T21:08:44.130 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/bcm5974.ko 2024-04-20T21:08:44.130 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/cyapatp.ko 2024-04-20T21:08:44.130 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/elan_i2c.ko 2024-04-20T21:08:44.130 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/sermouse.ko 2024-04-20T21:08:44.130 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-04-20T21:08:44.130 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/synaptics_usb.ko 2024-04-20T21:08:44.130 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/vsxxxaa.ko 2024-04-20T21:08:44.131 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/rmi4 2024-04-20T21:08:44.131 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/rmi4/rmi_core.ko 2024-04-20T21:08:44.131 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio 2024-04-20T21:08:44.131 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/altera_ps2.ko 2024-04-20T21:08:44.131 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/arc_ps2.ko 2024-04-20T21:08:44.131 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-04-20T21:08:44.131 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/serio_raw.ko 2024-04-20T21:08:44.131 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/sparse-keymap.ko 2024-04-20T21:08:44.131 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds 2024-04-20T21:08:44.131 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-blinkm.ko 2024-04-20T21:08:44.131 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-lm3530.ko 2024-04-20T21:08:44.131 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-lp3944.ko 2024-04-20T21:08:44.131 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-ss4200.ko 2024-04-20T21:08:44.131 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger 2024-04-20T21:08:44.131 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-04-20T21:08:44.132 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-04-20T21:08:44.132 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-04-20T21:08:44.132 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-04-20T21:08:44.132 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-04-20T21:08:44.132 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-04-20T21:08:44.132 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-04-20T21:08:44.132 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-04-20T21:08:44.132 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-04-20T21:08:44.132 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md 2024-04-20T21:08:44.132 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/bcache 2024-04-20T21:08:44.132 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/bcache/bcache.ko 2024-04-20T21:08:44.132 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-bio-prison.ko 2024-04-20T21:08:44.132 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-cache-smq.ko 2024-04-20T21:08:44.132 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-cache.ko 2024-04-20T21:08:44.132 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-crypt.ko 2024-04-20T21:08:44.133 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-delay.ko 2024-04-20T21:08:44.133 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-flakey.ko 2024-04-20T21:08:44.133 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-log-userspace.ko 2024-04-20T21:08:44.133 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-multipath.ko 2024-04-20T21:08:44.133 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-queue-length.ko 2024-04-20T21:08:44.133 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-raid.ko 2024-04-20T21:08:44.133 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-round-robin.ko 2024-04-20T21:08:44.133 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-service-time.ko 2024-04-20T21:08:44.133 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-switch.ko 2024-04-20T21:08:44.133 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-thin-pool.ko 2024-04-20T21:08:44.133 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-verity.ko 2024-04-20T21:08:44.133 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/persistent-data 2024-04-20T21:08:44.133 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-04-20T21:08:44.133 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid0.ko 2024-04-20T21:08:44.133 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid1.ko 2024-04-20T21:08:44.133 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid10.ko 2024-04-20T21:08:44.134 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid456.ko 2024-04-20T21:08:44.134 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message 2024-04-20T21:08:44.134 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion 2024-04-20T21:08:44.134 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptbase.ko 2024-04-20T21:08:44.134 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptctl.ko 2024-04-20T21:08:44.134 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptfc.ko 2024-04-20T21:08:44.134 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptlan.ko 2024-04-20T21:08:44.134 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptsas.ko 2024-04-20T21:08:44.134 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptscsih.ko 2024-04-20T21:08:44.134 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptspi.ko 2024-04-20T21:08:44.134 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd 2024-04-20T21:08:44.134 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/lpc_ich.ko 2024-04-20T21:08:44.134 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/lpc_sch.ko 2024-04-20T21:08:44.134 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/mfd-core.ko 2024-04-20T21:08:44.135 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/sm501.ko 2024-04-20T21:08:44.135 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/viperboard.ko 2024-04-20T21:08:44.135 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/vx855.ko 2024-04-20T21:08:44.135 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/wl1273-core.ko 2024-04-20T21:08:44.135 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc 2024-04-20T21:08:44.135 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/altera-stapl 2024-04-20T21:08:44.135 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-04-20T21:08:44.135 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/apds9802als.ko 2024-04-20T21:08:44.135 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/apds990x.ko 2024-04-20T21:08:44.135 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/bh1770glc.ko 2024-04-20T21:08:44.135 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/cb710 2024-04-20T21:08:44.135 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/cb710/cb710.ko 2024-04-20T21:08:44.135 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom 2024-04-20T21:08:44.135 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom/at24.ko 2024-04-20T21:08:44.136 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-04-20T21:08:44.136 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom/max6875.ko 2024-04-20T21:08:44.136 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/enclosure.ko 2024-04-20T21:08:44.136 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/hpilo.ko 2024-04-20T21:08:44.136 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/isl29003.ko 2024-04-20T21:08:44.136 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/isl29020.ko 2024-04-20T21:08:44.136 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/lis3lv02d 2024-04-20T21:08:44.136 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-04-20T21:08:44.136 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-04-20T21:08:44.136 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei 2024-04-20T21:08:44.136 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei/mei-me.ko 2024-04-20T21:08:44.136 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei/mei-txe.ko 2024-04-20T21:08:44.136 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei/mei.ko 2024-04-20T21:08:44.136 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-gru 2024-04-20T21:08:44.136 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-gru/gru.ko 2024-04-20T21:08:44.137 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp 2024-04-20T21:08:44.137 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp/xp.ko 2024-04-20T21:08:44.137 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp/xpc.ko 2024-04-20T21:08:44.137 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-04-20T21:08:44.137 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/tifm_7xx1.ko 2024-04-20T21:08:44.137 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/tifm_core.ko 2024-04-20T21:08:44.137 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/tsl2550.ko 2024-04-20T21:08:44.137 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/vmw_balloon.ko 2024-04-20T21:08:44.137 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/vmw_vmci 2024-04-20T21:08:44.137 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-04-20T21:08:44.137 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net 2024-04-20T21:08:44.137 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/bonding 2024-04-20T21:08:44.137 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/bonding/bonding.ko 2024-04-20T21:08:44.137 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa 2024-04-20T21:08:44.137 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/b53 2024-04-20T21:08:44.138 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/b53/b53_common.ko 2024-04-20T21:08:44.138 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/bcm-sf2.ko 2024-04-20T21:08:44.138 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/mv88e6060.ko 2024-04-20T21:08:44.138 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/mv88e6xxx 2024-04-20T21:08:44.138 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-04-20T21:08:44.138 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dummy.ko 2024-04-20T21:08:44.138 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/eql.ko 2024-04-20T21:08:44.138 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet 2024-04-20T21:08:44.138 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com 2024-04-20T21:08:44.138 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-04-20T21:08:44.138 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-04-20T21:08:44.138 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-04-20T21:08:44.138 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-04-20T21:08:44.138 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390 2024-04-20T21:08:44.138 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/8390.ko 2024-04-20T21:08:44.139 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-04-20T21:08:44.139 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-04-20T21:08:44.139 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-04-20T21:08:44.139 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/adaptec 2024-04-20T21:08:44.139 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-04-20T21:08:44.139 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/agere 2024-04-20T21:08:44.139 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/agere/et131x.ko 2024-04-20T21:08:44.139 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/alteon 2024-04-20T21:08:44.139 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-04-20T21:08:44.139 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/altera 2024-04-20T21:08:44.139 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-04-20T21:08:44.139 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd 2024-04-20T21:08:44.139 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-04-20T21:08:44.139 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-04-20T21:08:44.139 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-04-20T21:08:44.140 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros 2024-04-20T21:08:44.140 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/alx 2024-04-20T21:08:44.140 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-04-20T21:08:44.140 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1c 2024-04-20T21:08:44.140 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-04-20T21:08:44.140 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1e 2024-04-20T21:08:44.140 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-04-20T21:08:44.140 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atlx 2024-04-20T21:08:44.140 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-04-20T21:08:44.140 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-04-20T21:08:44.140 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom 2024-04-20T21:08:44.140 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-04-20T21:08:44.140 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-04-20T21:08:44.140 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-04-20T21:08:44.140 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-04-20T21:08:44.141 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-04-20T21:08:44.141 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/genet 2024-04-20T21:08:44.141 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-04-20T21:08:44.141 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-04-20T21:08:44.141 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/brocade 2024-04-20T21:08:44.141 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/brocade/bna 2024-04-20T21:08:44.141 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-04-20T21:08:44.141 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio 2024-04-20T21:08:44.141 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb 2024-04-20T21:08:44.141 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-04-20T21:08:44.141 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-04-20T21:08:44.141 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-04-20T21:08:44.141 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-04-20T21:08:44.141 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-04-20T21:08:44.141 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-04-20T21:08:44.142 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-04-20T21:08:44.142 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-04-20T21:08:44.142 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-04-20T21:08:44.142 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/cisco 2024-04-20T21:08:44.142 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/cisco/enic 2024-04-20T21:08:44.142 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-04-20T21:08:44.142 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec 2024-04-20T21:08:44.142 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip 2024-04-20T21:08:44.142 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-04-20T21:08:44.142 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-04-20T21:08:44.142 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-04-20T21:08:44.142 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-04-20T21:08:44.142 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-04-20T21:08:44.142 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-04-20T21:08:44.143 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dlink 2024-04-20T21:08:44.143 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-04-20T21:08:44.143 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-04-20T21:08:44.143 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dnet.ko 2024-04-20T21:08:44.143 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/emulex 2024-04-20T21:08:44.143 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/emulex/benet 2024-04-20T21:08:44.143 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-04-20T21:08:44.143 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/ethoc.ko 2024-04-20T21:08:44.143 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/fealnx.ko 2024-04-20T21:08:44.143 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel 2024-04-20T21:08:44.143 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e100.ko 2024-04-20T21:08:44.143 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000 2024-04-20T21:08:44.143 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-04-20T21:08:44.143 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000e 2024-04-20T21:08:44.143 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-04-20T21:08:44.143 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/fm10k 2024-04-20T21:08:44.144 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-04-20T21:08:44.144 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/i40e 2024-04-20T21:08:44.144 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-04-20T21:08:44.144 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/iavf 2024-04-20T21:08:44.144 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-04-20T21:08:44.144 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igb 2024-04-20T21:08:44.144 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-04-20T21:08:44.144 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igbvf 2024-04-20T21:08:44.144 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-04-20T21:08:44.144 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbe 2024-04-20T21:08:44.144 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-04-20T21:08:44.144 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbevf 2024-04-20T21:08:44.144 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-04-20T21:08:44.144 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/jme.ko 2024-04-20T21:08:44.144 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell 2024-04-20T21:08:44.145 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-04-20T21:08:44.145 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell/skge.ko 2024-04-20T21:08:44.145 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-04-20T21:08:44.145 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox 2024-04-20T21:08:44.145 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx4 2024-04-20T21:08:44.145 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-04-20T21:08:44.145 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-04-20T21:08:44.145 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx5 2024-04-20T21:08:44.145 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-04-20T21:08:44.145 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-04-20T21:08:44.145 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/micrel 2024-04-20T21:08:44.145 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-04-20T21:08:44.145 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/myricom 2024-04-20T21:08:44.145 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/myricom/myri10ge 2024-04-20T21:08:44.145 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-04-20T21:08:44.146 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/natsemi 2024-04-20T21:08:44.146 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-04-20T21:08:44.146 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-04-20T21:08:44.146 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/neterion 2024-04-20T21:08:44.146 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-04-20T21:08:44.146 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/nvidia 2024-04-20T21:08:44.146 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-04-20T21:08:44.146 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/packetengines 2024-04-20T21:08:44.146 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-04-20T21:08:44.146 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-04-20T21:08:44.146 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic 2024-04-20T21:08:44.146 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/netxen 2024-04-20T21:08:44.146 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-04-20T21:08:44.146 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-04-20T21:08:44.146 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-04-20T21:08:44.147 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-04-20T21:08:44.147 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rdc 2024-04-20T21:08:44.147 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-04-20T21:08:44.147 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek 2024-04-20T21:08:44.147 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-04-20T21:08:44.147 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-04-20T21:08:44.147 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/atp.ko 2024-04-20T21:08:44.147 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-04-20T21:08:44.147 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rocker 2024-04-20T21:08:44.147 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-04-20T21:08:44.147 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sfc 2024-04-20T21:08:44.147 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-04-20T21:08:44.147 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/silan 2024-04-20T21:08:44.147 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-04-20T21:08:44.147 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sis 2024-04-20T21:08:44.148 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sis/sis190.ko 2024-04-20T21:08:44.148 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sis/sis900.ko 2024-04-20T21:08:44.148 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc 2024-04-20T21:08:44.148 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-04-20T21:08:44.148 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-04-20T21:08:44.148 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-04-20T21:08:44.148 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-04-20T21:08:44.148 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro 2024-04-20T21:08:44.148 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro/stmmac 2024-04-20T21:08:44.148 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-04-20T21:08:44.148 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-04-20T21:08:44.148 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun 2024-04-20T21:08:44.148 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/cassini.ko 2024-04-20T21:08:44.148 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/niu.ko 2024-04-20T21:08:44.148 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/sungem.ko 2024-04-20T21:08:44.149 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-04-20T21:08:44.149 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/tehuti 2024-04-20T21:08:44.149 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-04-20T21:08:44.149 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/ti 2024-04-20T21:08:44.149 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/ti/tlan.ko 2024-04-20T21:08:44.149 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/via 2024-04-20T21:08:44.149 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-04-20T21:08:44.149 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-04-20T21:08:44.149 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/wiznet 2024-04-20T21:08:44.149 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-04-20T21:08:44.149 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-04-20T21:08:44.149 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/xircom 2024-04-20T21:08:44.149 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-04-20T21:08:44.149 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/geneve.ko 2024-04-20T21:08:44.149 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/hyperv 2024-04-20T21:08:44.150 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-04-20T21:08:44.150 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ieee802154 2024-04-20T21:08:44.150 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ieee802154/fakelb.ko 2024-04-20T21:08:44.150 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ifb.ko 2024-04-20T21:08:44.150 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ipvlan 2024-04-20T21:08:44.150 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ipvlan/ipvlan.ko 2024-04-20T21:08:44.150 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/macvlan.ko 2024-04-20T21:08:44.150 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/macvtap.ko 2024-04-20T21:08:44.150 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio 2024-04-20T21:08:44.150 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio.ko 2024-04-20T21:08:44.150 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-04-20T21:08:44.150 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-04-20T21:08:44.150 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio/mdio-regmap.ko 2024-04-20T21:08:44.150 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mii.ko 2024-04-20T21:08:44.150 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/net_failover.ko 2024-04-20T21:08:44.151 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/netconsole.ko 2024-04-20T21:08:44.151 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/nlmon.ko 2024-04-20T21:08:44.151 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/pcs 2024-04-20T21:08:44.151 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/pcs/pcs-lynx.ko 2024-04-20T21:08:44.151 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-04-20T21:08:44.151 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy 2024-04-20T21:08:44.151 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/amd.ko 2024-04-20T21:08:44.151 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-04-20T21:08:44.151 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-04-20T21:08:44.151 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm7xxx.ko 2024-04-20T21:08:44.152 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm87xx.ko 2024-04-20T21:08:44.152 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/broadcom.ko 2024-04-20T21:08:44.152 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/cicada.ko 2024-04-20T21:08:44.152 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/davicom.ko 2024-04-20T21:08:44.152 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/dp83640.ko 2024-04-20T21:08:44.152 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/et1011c.ko 2024-04-20T21:08:44.152 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/icplus.ko 2024-04-20T21:08:44.152 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/lxt.ko 2024-04-20T21:08:44.152 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/marvell.ko 2024-04-20T21:08:44.152 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/micrel.ko 2024-04-20T21:08:44.152 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/national.ko 2024-04-20T21:08:44.152 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/phylink.ko 2024-04-20T21:08:44.152 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/qsemi.ko 2024-04-20T21:08:44.152 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/realtek.ko 2024-04-20T21:08:44.153 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/smsc.ko 2024-04-20T21:08:44.153 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/ste10Xp.ko 2024-04-20T21:08:44.153 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/vitesse.ko 2024-04-20T21:08:44.153 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp 2024-04-20T21:08:44.153 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/bsd_comp.ko 2024-04-20T21:08:44.153 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_async.ko 2024-04-20T21:08:44.153 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_deflate.ko 2024-04-20T21:08:44.153 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_generic.ko 2024-04-20T21:08:44.153 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_mppe.ko 2024-04-20T21:08:44.153 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_synctty.ko 2024-04-20T21:08:44.153 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/pppoe.ko 2024-04-20T21:08:44.153 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/pppox.ko 2024-04-20T21:08:44.153 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/pptp.ko 2024-04-20T21:08:44.153 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/slip 2024-04-20T21:08:44.153 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/slip/slhc.ko 2024-04-20T21:08:44.154 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/slip/slip.ko 2024-04-20T21:08:44.154 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/sungem_phy.ko 2024-04-20T21:08:44.154 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/tap.ko 2024-04-20T21:08:44.154 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team 2024-04-20T21:08:44.154 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team.ko 2024-04-20T21:08:44.154 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_activebackup.ko 2024-04-20T21:08:44.154 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_broadcast.ko 2024-04-20T21:08:44.154 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-04-20T21:08:44.154 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_random.ko 2024-04-20T21:08:44.154 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-04-20T21:08:44.154 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/tun.ko 2024-04-20T21:08:44.154 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/veth.ko 2024-04-20T21:08:44.154 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/virtio_net.ko 2024-04-20T21:08:44.154 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vmxnet3 2024-04-20T21:08:44.154 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-04-20T21:08:44.155 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vxlan 2024-04-20T21:08:44.155 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vxlan/vxlan.ko 2024-04-20T21:08:44.155 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/xen-netback 2024-04-20T21:08:44.155 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/xen-netback/xen-netback.ko 2024-04-20T21:08:44.155 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/xen-netfront.ko 2024-04-20T21:08:44.155 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ntb 2024-04-20T21:08:44.155 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ntb/ntb.ko 2024-04-20T21:08:44.155 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme 2024-04-20T21:08:44.155 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme/host 2024-04-20T21:08:44.155 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme/host/nvme-core.ko 2024-04-20T21:08:44.155 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme/host/nvme.ko 2024-04-20T21:08:44.155 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport 2024-04-20T21:08:44.155 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport.ko 2024-04-20T21:08:44.155 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport_cs.ko 2024-04-20T21:08:44.155 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport_pc.ko 2024-04-20T21:08:44.156 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport_serial.ko 2024-04-20T21:08:44.156 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci 2024-04-20T21:08:44.156 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/hotplug 2024-04-20T21:08:44.156 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-04-20T21:08:44.156 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/pcie 2024-04-20T21:08:44.156 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/pcie/aer_inject.ko 2024-04-20T21:08:44.156 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/xen-pcifront.ko 2024-04-20T21:08:44.156 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia 2024-04-20T21:08:44.156 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia/i82092.ko 2024-04-20T21:08:44.156 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia/pd6729.ko 2024-04-20T21:08:44.156 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia/yenta_socket.ko 2024-04-20T21:08:44.156 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pinctrl 2024-04-20T21:08:44.156 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pinctrl/intel 2024-04-20T21:08:44.156 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-04-20T21:08:44.156 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform 2024-04-20T21:08:44.156 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86 2024-04-20T21:08:44.157 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/acer-wmi.ko 2024-04-20T21:08:44.157 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/acerhdf.ko 2024-04-20T21:08:44.157 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/apple-gmux.ko 2024-04-20T21:08:44.157 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/asus-laptop.ko 2024-04-20T21:08:44.157 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-04-20T21:08:44.157 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/asus-wmi.ko 2024-04-20T21:08:44.157 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/classmate-laptop.ko 2024-04-20T21:08:44.157 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-04-20T21:08:44.157 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-04-20T21:08:44.157 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-04-20T21:08:44.157 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-04-20T21:08:44.157 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/hdaps.ko 2024-04-20T21:08:44.157 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/intel 2024-04-20T21:08:44.157 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-04-20T21:08:44.157 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/intel_ips.ko 2024-04-20T21:08:44.157 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/msi-wmi.ko 2024-04-20T21:08:44.158 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/mxm-wmi.ko 2024-04-20T21:08:44.158 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-04-20T21:08:44.158 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/samsung-laptop.ko 2024-04-20T21:08:44.158 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/samsung-q10.ko 2024-04-20T21:08:44.158 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/topstar-laptop.ko 2024-04-20T21:08:44.158 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-04-20T21:08:44.158 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/toshiba_haps.ko 2024-04-20T21:08:44.158 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/wmi-bmof.ko 2024-04-20T21:08:44.158 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/wmi.ko 2024-04-20T21:08:44.158 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/powercap 2024-04-20T21:08:44.158 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/powercap/intel_rapl_common.ko 2024-04-20T21:08:44.158 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/powercap/intel_rapl_msr.ko 2024-04-20T21:08:44.158 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps 2024-04-20T21:08:44.158 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients 2024-04-20T21:08:44.158 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients/pps-gpio.ko 2024-04-20T21:08:44.159 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients/pps-ldisc.ko 2024-04-20T21:08:44.159 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients/pps_parport.ko 2024-04-20T21:08:44.159 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/pps_core.ko 2024-04-20T21:08:44.159 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ptp 2024-04-20T21:08:44.159 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ptp/ptp.ko 2024-04-20T21:08:44.159 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ptp/ptp_kvm.ko 2024-04-20T21:08:44.159 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras 2024-04-20T21:08:44.159 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd 2024-04-20T21:08:44.159 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd/atl 2024-04-20T21:08:44.159 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd/atl/amd_atl.ko 2024-04-20T21:08:44.159 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd/fmpm.ko 2024-04-20T21:08:44.159 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc 2024-04-20T21:08:44.159 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-bq32k.ko 2024-04-20T21:08:44.159 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1286.ko 2024-04-20T21:08:44.159 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1307.ko 2024-04-20T21:08:44.160 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1374.ko 2024-04-20T21:08:44.160 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1511.ko 2024-04-20T21:08:44.160 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1553.ko 2024-04-20T21:08:44.160 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1672.ko 2024-04-20T21:08:44.160 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1742.ko 2024-04-20T21:08:44.160 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds2404.ko 2024-04-20T21:08:44.160 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds3232.ko 2024-04-20T21:08:44.160 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-em3027.ko 2024-04-20T21:08:44.160 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-fm3130.ko 2024-04-20T21:08:44.160 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-isl12022.ko 2024-04-20T21:08:44.160 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-isl1208.ko 2024-04-20T21:08:44.160 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-m41t80.ko 2024-04-20T21:08:44.160 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-m48t35.ko 2024-04-20T21:08:44.160 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-m48t59.ko 2024-04-20T21:08:44.160 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-max6900.ko 2024-04-20T21:08:44.160 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-msm6242.ko 2024-04-20T21:08:44.161 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf2127.ko 2024-04-20T21:08:44.161 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf85063.ko 2024-04-20T21:08:44.161 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf8523.ko 2024-04-20T21:08:44.161 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf8563.ko 2024-04-20T21:08:44.161 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf8583.ko 2024-04-20T21:08:44.161 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rp5c01.ko 2024-04-20T21:08:44.161 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rs5c372.ko 2024-04-20T21:08:44.161 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-04-20T21:08:44.161 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rx8025.ko 2024-04-20T21:08:44.161 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rx8581.ko 2024-04-20T21:08:44.161 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-04-20T21:08:44.161 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-x1205.ko 2024-04-20T21:08:44.161 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi 2024-04-20T21:08:44.161 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/3w-9xxx.ko 2024-04-20T21:08:44.161 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/3w-sas.ko 2024-04-20T21:08:44.161 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/3w-xxxx.ko 2024-04-20T21:08:44.162 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/BusLogic.ko 2024-04-20T21:08:44.162 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/a100u2w.ko 2024-04-20T21:08:44.162 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aacraid 2024-04-20T21:08:44.162 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aacraid/aacraid.ko 2024-04-20T21:08:44.162 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/advansys.ko 2024-04-20T21:08:44.162 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aic7xxx 2024-04-20T21:08:44.162 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-04-20T21:08:44.162 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-04-20T21:08:44.162 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/am53c974.ko 2024-04-20T21:08:44.162 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/arcmsr 2024-04-20T21:08:44.162 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-04-20T21:08:44.162 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/atp870u.ko 2024-04-20T21:08:44.162 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/be2iscsi 2024-04-20T21:08:44.162 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-04-20T21:08:44.163 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bfa 2024-04-20T21:08:44.163 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bfa/bfa.ko 2024-04-20T21:08:44.163 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2fc 2024-04-20T21:08:44.163 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-04-20T21:08:44.163 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2i 2024-04-20T21:08:44.163 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-04-20T21:08:44.163 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ch.ko 2024-04-20T21:08:44.163 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/csiostor 2024-04-20T21:08:44.163 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/csiostor/csiostor.ko 2024-04-20T21:08:44.163 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi 2024-04-20T21:08:44.163 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb3i 2024-04-20T21:08:44.163 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-04-20T21:08:44.163 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb4i 2024-04-20T21:08:44.163 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-04-20T21:08:44.163 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-04-20T21:08:44.164 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/dc395x.ko 2024-04-20T21:08:44.164 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler 2024-04-20T21:08:44.164 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-04-20T21:08:44.164 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-04-20T21:08:44.164 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-04-20T21:08:44.164 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-04-20T21:08:44.164 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/dmx3191d.ko 2024-04-20T21:08:44.164 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/esas2r 2024-04-20T21:08:44.164 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/esas2r/esas2r.ko 2024-04-20T21:08:44.164 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/esp_scsi.ko 2024-04-20T21:08:44.164 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fcoe 2024-04-20T21:08:44.164 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fcoe/fcoe.ko 2024-04-20T21:08:44.164 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-04-20T21:08:44.164 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fnic 2024-04-20T21:08:44.164 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fnic/fnic.ko 2024-04-20T21:08:44.164 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/hpsa.ko 2024-04-20T21:08:44.165 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/hptiop.ko 2024-04-20T21:08:44.165 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/hv_storvsc.ko 2024-04-20T21:08:44.165 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/initio.ko 2024-04-20T21:08:44.165 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ipr.ko 2024-04-20T21:08:44.165 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ips.ko 2024-04-20T21:08:44.165 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/isci 2024-04-20T21:08:44.165 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/isci/isci.ko 2024-04-20T21:08:44.165 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-04-20T21:08:44.165 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/iscsi_tcp.ko 2024-04-20T21:08:44.165 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libfc 2024-04-20T21:08:44.165 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libfc/libfc.ko 2024-04-20T21:08:44.165 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libiscsi.ko 2024-04-20T21:08:44.165 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libiscsi_tcp.ko 2024-04-20T21:08:44.165 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libsas 2024-04-20T21:08:44.165 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libsas/libsas.ko 2024-04-20T21:08:44.166 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/lpfc 2024-04-20T21:08:44.166 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/lpfc/lpfc.ko 2024-04-20T21:08:44.166 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid 2024-04-20T21:08:44.166 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid.ko 2024-04-20T21:08:44.166 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-04-20T21:08:44.166 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-04-20T21:08:44.166 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-04-20T21:08:44.166 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mpt3sas 2024-04-20T21:08:44.166 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-04-20T21:08:44.166 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mvsas 2024-04-20T21:08:44.166 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mvsas/mvsas.ko 2024-04-20T21:08:44.166 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mvumi.ko 2024-04-20T21:08:44.166 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/pm8001 2024-04-20T21:08:44.166 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-04-20T21:08:44.166 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/pmcraid.ko 2024-04-20T21:08:44.167 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla1280.ko 2024-04-20T21:08:44.167 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla2xxx 2024-04-20T21:08:44.167 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-04-20T21:08:44.167 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-04-20T21:08:44.167 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla4xxx 2024-04-20T21:08:44.167 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-04-20T21:08:44.167 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/raid_class.ko 2024-04-20T21:08:44.167 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_debug.ko 2024-04-20T21:08:44.167 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_fc.ko 2024-04-20T21:08:44.167 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-04-20T21:08:44.167 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_sas.ko 2024-04-20T21:08:44.167 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_spi.ko 2024-04-20T21:08:44.167 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_srp.ko 2024-04-20T21:08:44.167 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ses.ko 2024-04-20T21:08:44.167 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/st.ko 2024-04-20T21:08:44.167 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/stex.ko 2024-04-20T21:08:44.168 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/sym53c8xx_2 2024-04-20T21:08:44.168 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-04-20T21:08:44.168 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/virtio_scsi.ko 2024-04-20T21:08:44.168 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/vmw_pvscsi.ko 2024-04-20T21:08:44.168 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/wd719x.ko 2024-04-20T21:08:44.168 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ssb 2024-04-20T21:08:44.168 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ssb/ssb.ko 2024-04-20T21:08:44.168 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target 2024-04-20T21:08:44.168 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/iscsi 2024-04-20T21:08:44.168 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-04-20T21:08:44.168 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/loopback 2024-04-20T21:08:44.168 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/loopback/tcm_loop.ko 2024-04-20T21:08:44.168 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_file.ko 2024-04-20T21:08:44.168 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_iblock.ko 2024-04-20T21:08:44.168 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_mod.ko 2024-04-20T21:08:44.168 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_pscsi.ko 2024-04-20T21:08:44.168 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_user.ko 2024-04-20T21:08:44.169 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/tcm_fc 2024-04-20T21:08:44.169 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-04-20T21:08:44.169 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal 2024-04-20T21:08:44.169 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel 2024-04-20T21:08:44.169 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal 2024-04-20T21:08:44.169 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-04-20T21:08:44.169 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-04-20T21:08:44.169 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-04-20T21:08:44.169 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-04-20T21:08:44.169 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-04-20T21:08:44.169 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-04-20T21:08:44.169 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-04-20T21:08:44.169 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-04-20T21:08:44.169 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-04-20T21:08:44.169 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-04-20T21:08:44.170 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-04-20T21:08:44.170 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-04-20T21:08:44.170 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-04-20T21:08:44.170 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-04-20T21:08:44.170 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-04-20T21:08:44.170 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-04-20T21:08:44.170 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-04-20T21:08:44.170 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-04-20T21:08:44.170 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty 2024-04-20T21:08:44.170 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/ipwireless 2024-04-20T21:08:44.170 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-04-20T21:08:44.170 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/n_gsm.ko 2024-04-20T21:08:44.170 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/n_hdlc.ko 2024-04-20T21:08:44.170 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/nozomi.ko 2024-04-20T21:08:44.171 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial 2024-04-20T21:08:44.171 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/8250 2024-04-20T21:08:44.171 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-04-20T21:08:44.171 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/arc_uart.ko 2024-04-20T21:08:44.171 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/jsm 2024-04-20T21:08:44.171 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/jsm/jsm.ko 2024-04-20T21:08:44.171 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs 2024-04-20T21:08:44.171 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/core 2024-04-20T21:08:44.171 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/core/ufshcd-core.ko 2024-04-20T21:08:44.171 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/host 2024-04-20T21:08:44.171 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-04-20T21:08:44.171 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio 2024-04-20T21:08:44.171 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio.ko 2024-04-20T21:08:44.171 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_aec.ko 2024-04-20T21:08:44.171 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_cif.ko 2024-04-20T21:08:44.172 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_pci_generic.ko 2024-04-20T21:08:44.172 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_sercos3.ko 2024-04-20T21:08:44.172 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb 2024-04-20T21:08:44.172 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class 2024-04-20T21:08:44.172 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/cdc-acm.ko 2024-04-20T21:08:44.172 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/cdc-wdm.ko 2024-04-20T21:08:44.172 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/usblp.ko 2024-04-20T21:08:44.172 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/usbtmc.ko 2024-04-20T21:08:44.172 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/host 2024-04-20T21:08:44.172 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/host/sl811-hcd.ko 2024-04-20T21:08:44.172 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/image 2024-04-20T21:08:44.172 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/image/mdc800.ko 2024-04-20T21:08:44.172 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/image/microtek.ko 2024-04-20T21:08:44.172 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc 2024-04-20T21:08:44.172 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/adutux.ko 2024-04-20T21:08:44.172 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/appledisplay.ko 2024-04-20T21:08:44.173 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/emi26.ko 2024-04-20T21:08:44.173 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/emi62.ko 2024-04-20T21:08:44.173 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/ezusb.ko 2024-04-20T21:08:44.173 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/idmouse.ko 2024-04-20T21:08:44.173 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/iowarrior.ko 2024-04-20T21:08:44.173 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/isight_firmware.ko 2024-04-20T21:08:44.173 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/ldusb.ko 2024-04-20T21:08:44.173 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/legousbtower.ko 2024-04-20T21:08:44.173 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/sisusbvga 2024-04-20T21:08:44.173 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-04-20T21:08:44.173 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/trancevibrator.ko 2024-04-20T21:08:44.173 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/usb3503.ko 2024-04-20T21:08:44.173 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/usblcd.ko 2024-04-20T21:08:44.173 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/usbsevseg.ko 2024-04-20T21:08:44.173 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/uss720.ko 2024-04-20T21:08:44.173 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/yurex.ko 2024-04-20T21:08:44.174 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/phy 2024-04-20T21:08:44.174 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/phy/phy-generic.ko 2024-04-20T21:08:44.174 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial 2024-04-20T21:08:44.174 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/aircable.ko 2024-04-20T21:08:44.174 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ark3116.ko 2024-04-20T21:08:44.174 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/belkin_sa.ko 2024-04-20T21:08:44.174 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ch341.ko 2024-04-20T21:08:44.174 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/cp210x.ko 2024-04-20T21:08:44.174 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/cyberjack.ko 2024-04-20T21:08:44.174 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/cypress_m8.ko 2024-04-20T21:08:44.174 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/digi_acceleport.ko 2024-04-20T21:08:44.174 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/empeg.ko 2024-04-20T21:08:44.174 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ftdi_sio.ko 2024-04-20T21:08:44.174 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/garmin_gps.ko 2024-04-20T21:08:44.174 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/io_edgeport.ko 2024-04-20T21:08:44.175 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/io_ti.ko 2024-04-20T21:08:44.175 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ipaq.ko 2024-04-20T21:08:44.175 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ipw.ko 2024-04-20T21:08:44.175 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ir-usb.ko 2024-04-20T21:08:44.175 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-04-20T21:08:44.175 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/keyspan.ko 2024-04-20T21:08:44.175 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/keyspan_pda.ko 2024-04-20T21:08:44.175 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/kl5kusb105.ko 2024-04-20T21:08:44.175 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/kobil_sct.ko 2024-04-20T21:08:44.175 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/mct_u232.ko 2024-04-20T21:08:44.175 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/mos7720.ko 2024-04-20T21:08:44.175 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/mos7840.ko 2024-04-20T21:08:44.175 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/navman.ko 2024-04-20T21:08:44.175 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/omninet.ko 2024-04-20T21:08:44.175 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/opticon.ko 2024-04-20T21:08:44.175 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/option.ko 2024-04-20T21:08:44.176 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/oti6858.ko 2024-04-20T21:08:44.176 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/pl2303.ko 2024-04-20T21:08:44.176 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/qcaux.ko 2024-04-20T21:08:44.176 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/qcserial.ko 2024-04-20T21:08:44.176 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/quatech2.ko 2024-04-20T21:08:44.176 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/safe_serial.ko 2024-04-20T21:08:44.176 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/sierra.ko 2024-04-20T21:08:44.176 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/spcp8x5.ko 2024-04-20T21:08:44.176 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ssu100.ko 2024-04-20T21:08:44.176 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/symbolserial.ko 2024-04-20T21:08:44.176 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-04-20T21:08:44.176 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-04-20T21:08:44.176 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/usb_debug.ko 2024-04-20T21:08:44.176 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/usb_wwan.ko 2024-04-20T21:08:44.176 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/visor.ko 2024-04-20T21:08:44.176 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/whiteheat.ko 2024-04-20T21:08:44.177 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/xsens_mt.ko 2024-04-20T21:08:44.177 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage 2024-04-20T21:08:44.177 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/uas.ko 2024-04-20T21:08:44.177 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-alauda.ko 2024-04-20T21:08:44.177 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-cypress.ko 2024-04-20T21:08:44.177 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-datafab.ko 2024-04-20T21:08:44.177 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-04-20T21:08:44.177 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-freecom.ko 2024-04-20T21:08:44.177 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-isd200.ko 2024-04-20T21:08:44.177 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-04-20T21:08:44.177 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-karma.ko 2024-04-20T21:08:44.177 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-onetouch.ko 2024-04-20T21:08:44.177 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-realtek.ko 2024-04-20T21:08:44.177 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-sddr09.ko 2024-04-20T21:08:44.177 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-sddr55.ko 2024-04-20T21:08:44.178 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-usbat.ko 2024-04-20T21:08:44.178 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/usb-storage.ko 2024-04-20T21:08:44.178 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip 2024-04-20T21:08:44.178 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip/usbip-core.ko 2024-04-20T21:08:44.178 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip/usbip-host.ko 2024-04-20T21:08:44.178 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-04-20T21:08:44.178 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio 2024-04-20T21:08:44.178 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/pci 2024-04-20T21:08:44.178 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-04-20T21:08:44.178 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/pci/vfio-pci.ko 2024-04-20T21:08:44.178 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/vfio.ko 2024-04-20T21:08:44.178 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-04-20T21:08:44.178 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost 2024-04-20T21:08:44.178 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost.ko 2024-04-20T21:08:44.178 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost_iotlb.ko 2024-04-20T21:08:44.179 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost_net.ko 2024-04-20T21:08:44.179 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost_scsi.ko 2024-04-20T21:08:44.179 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video 2024-04-20T21:08:44.179 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight 2024-04-20T21:08:44.179 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight/apple_bl.ko 2024-04-20T21:08:44.179 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight/lcd.ko 2024-04-20T21:08:44.179 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight/platform_lcd.ko 2024-04-20T21:08:44.179 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio 2024-04-20T21:08:44.179 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio.ko 2024-04-20T21:08:44.179 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_balloon.ko 2024-04-20T21:08:44.179 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_mmio.ko 2024-04-20T21:08:44.179 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_pci.ko 2024-04-20T21:08:44.179 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-04-20T21:08:44.179 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-04-20T21:08:44.179 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_ring.ko 2024-04-20T21:08:44.180 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1 2024-04-20T21:08:44.180 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/masters 2024-04-20T21:08:44.180 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/masters/ds2482.ko 2024-04-20T21:08:44.180 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/masters/ds2490.ko 2024-04-20T21:08:44.180 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves 2024-04-20T21:08:44.180 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-04-20T21:08:44.180 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-04-20T21:08:44.180 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-04-20T21:08:44.180 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-04-20T21:08:44.180 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-04-20T21:08:44.180 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-04-20T21:08:44.180 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-04-20T21:08:44.180 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-04-20T21:08:44.180 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-04-20T21:08:44.180 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_smem.ko 2024-04-20T21:08:44.180 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_therm.ko 2024-04-20T21:08:44.181 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/wire.ko 2024-04-20T21:08:44.181 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog 2024-04-20T21:08:44.181 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/alim1535_wdt.ko 2024-04-20T21:08:44.181 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/alim7101_wdt.ko 2024-04-20T21:08:44.181 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/f71808e_wdt.ko 2024-04-20T21:08:44.181 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/hpwdt.ko 2024-04-20T21:08:44.181 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/i6300esb.ko 2024-04-20T21:08:44.181 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-04-20T21:08:44.181 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/iTCO_wdt.ko 2024-04-20T21:08:44.181 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/ib700wdt.ko 2024-04-20T21:08:44.181 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/ibmasr.ko 2024-04-20T21:08:44.181 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-04-20T21:08:44.181 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/it8712f_wdt.ko 2024-04-20T21:08:44.181 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/it87_wdt.ko 2024-04-20T21:08:44.181 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/machzwd.ko 2024-04-20T21:08:44.182 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/nv_tco.ko 2024-04-20T21:08:44.182 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/pcwd_pci.ko 2024-04-20T21:08:44.182 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/pcwd_usb.ko 2024-04-20T21:08:44.182 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-04-20T21:08:44.182 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/sch311x_wdt.ko 2024-04-20T21:08:44.182 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/softdog.ko 2024-04-20T21:08:44.182 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/sp5100_tco.ko 2024-04-20T21:08:44.182 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/via_wdt.ko 2024-04-20T21:08:44.182 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-04-20T21:08:44.182 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/w83877f_wdt.ko 2024-04-20T21:08:44.182 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/w83977f_wdt.ko 2024-04-20T21:08:44.182 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/wdt_pci.ko 2024-04-20T21:08:44.182 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/xen_wdt.ko 2024-04-20T21:08:44.182 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen 2024-04-20T21:08:44.182 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-acpi-processor.ko 2024-04-20T21:08:44.183 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-evtchn.ko 2024-04-20T21:08:44.183 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-gntalloc.ko 2024-04-20T21:08:44.183 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-gntdev.ko 2024-04-20T21:08:44.183 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-pciback 2024-04-20T21:08:44.183 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-04-20T21:08:44.183 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-privcmd.ko 2024-04-20T21:08:44.183 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xenfs 2024-04-20T21:08:44.183 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xenfs/xenfs.ko 2024-04-20T21:08:44.183 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs 2024-04-20T21:08:44.183 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/9p 2024-04-20T21:08:44.183 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/9p/9p.ko 2024-04-20T21:08:44.183 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/affs 2024-04-20T21:08:44.183 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/affs/affs.ko 2024-04-20T21:08:44.183 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/befs 2024-04-20T21:08:44.183 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/befs/befs.ko 2024-04-20T21:08:44.183 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/binfmt_misc.ko 2024-04-20T21:08:44.184 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/btrfs 2024-04-20T21:08:44.184 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/btrfs/btrfs.ko 2024-04-20T21:08:44.184 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cachefiles 2024-04-20T21:08:44.184 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cachefiles/cachefiles.ko 2024-04-20T21:08:44.184 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ceph 2024-04-20T21:08:44.184 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ceph/ceph.ko 2024-04-20T21:08:44.184 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/coda 2024-04-20T21:08:44.184 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/coda/coda.ko 2024-04-20T21:08:44.184 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cramfs 2024-04-20T21:08:44.184 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cramfs/cramfs.ko 2024-04-20T21:08:44.184 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/dlm 2024-04-20T21:08:44.184 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/dlm/dlm.ko 2024-04-20T21:08:44.184 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ecryptfs 2024-04-20T21:08:44.184 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ecryptfs/ecryptfs.ko 2024-04-20T21:08:44.184 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/f2fs 2024-04-20T21:08:44.185 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/f2fs/f2fs.ko 2024-04-20T21:08:44.185 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat 2024-04-20T21:08:44.185 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat/fat.ko 2024-04-20T21:08:44.185 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat/msdos.ko 2024-04-20T21:08:44.185 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat/vfat.ko 2024-04-20T21:08:44.185 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fuse 2024-04-20T21:08:44.185 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fuse/cuse.ko 2024-04-20T21:08:44.185 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fuse/fuse.ko 2024-04-20T21:08:44.185 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/gfs2 2024-04-20T21:08:44.185 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/gfs2/gfs2.ko 2024-04-20T21:08:44.185 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfs 2024-04-20T21:08:44.185 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfs/hfs.ko 2024-04-20T21:08:44.185 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfsplus 2024-04-20T21:08:44.185 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfsplus/hfsplus.ko 2024-04-20T21:08:44.185 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/isofs 2024-04-20T21:08:44.185 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/isofs/isofs.ko 2024-04-20T21:08:44.186 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/jfs 2024-04-20T21:08:44.186 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/jfs/jfs.ko 2024-04-20T21:08:44.186 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/lockd 2024-04-20T21:08:44.186 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/lockd/lockd.ko 2024-04-20T21:08:44.186 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/minix 2024-04-20T21:08:44.186 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/minix/minix.ko 2024-04-20T21:08:44.186 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/netfs 2024-04-20T21:08:44.186 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/netfs/netfs.ko 2024-04-20T21:08:44.186 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs 2024-04-20T21:08:44.186 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/blocklayout 2024-04-20T21:08:44.186 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-04-20T21:08:44.186 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/filelayout 2024-04-20T21:08:44.186 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-04-20T21:08:44.186 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/flexfilelayout 2024-04-20T21:08:44.186 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-04-20T21:08:44.187 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/nfs.ko 2024-04-20T21:08:44.187 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/nfsv3.ko 2024-04-20T21:08:44.187 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/nfsv4.ko 2024-04-20T21:08:44.187 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs_common 2024-04-20T21:08:44.187 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs_common/grace.ko 2024-04-20T21:08:44.187 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs_common/nfs_acl.ko 2024-04-20T21:08:44.187 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfsd 2024-04-20T21:08:44.187 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfsd/nfsd.ko 2024-04-20T21:08:44.187 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nilfs2 2024-04-20T21:08:44.187 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nilfs2/nilfs2.ko 2024-04-20T21:08:44.187 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls 2024-04-20T21:08:44.187 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-celtic.ko 2024-04-20T21:08:44.187 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-centeuro.ko 2024-04-20T21:08:44.187 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-croatian.ko 2024-04-20T21:08:44.187 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-cyrillic.ko 2024-04-20T21:08:44.187 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-gaelic.ko 2024-04-20T21:08:44.188 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-greek.ko 2024-04-20T21:08:44.188 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-iceland.ko 2024-04-20T21:08:44.188 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-inuit.ko 2024-04-20T21:08:44.188 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-roman.ko 2024-04-20T21:08:44.188 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-romanian.ko 2024-04-20T21:08:44.188 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-turkish.ko 2024-04-20T21:08:44.188 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp1250.ko 2024-04-20T21:08:44.188 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp1251.ko 2024-04-20T21:08:44.188 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp1255.ko 2024-04-20T21:08:44.188 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp737.ko 2024-04-20T21:08:44.188 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp775.ko 2024-04-20T21:08:44.188 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp850.ko 2024-04-20T21:08:44.188 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp852.ko 2024-04-20T21:08:44.188 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp855.ko 2024-04-20T21:08:44.188 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp857.ko 2024-04-20T21:08:44.188 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp860.ko 2024-04-20T21:08:44.189 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp861.ko 2024-04-20T21:08:44.189 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp862.ko 2024-04-20T21:08:44.189 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp863.ko 2024-04-20T21:08:44.189 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp864.ko 2024-04-20T21:08:44.189 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp865.ko 2024-04-20T21:08:44.189 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp866.ko 2024-04-20T21:08:44.189 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp869.ko 2024-04-20T21:08:44.189 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp874.ko 2024-04-20T21:08:44.189 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp932.ko 2024-04-20T21:08:44.189 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp936.ko 2024-04-20T21:08:44.189 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp949.ko 2024-04-20T21:08:44.189 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp950.ko 2024-04-20T21:08:44.189 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_euc-jp.ko 2024-04-20T21:08:44.189 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-1.ko 2024-04-20T21:08:44.189 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-13.ko 2024-04-20T21:08:44.189 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-14.ko 2024-04-20T21:08:44.190 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-15.ko 2024-04-20T21:08:44.190 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-2.ko 2024-04-20T21:08:44.190 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-3.ko 2024-04-20T21:08:44.190 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-4.ko 2024-04-20T21:08:44.190 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-5.ko 2024-04-20T21:08:44.190 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-6.ko 2024-04-20T21:08:44.190 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-7.ko 2024-04-20T21:08:44.190 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-9.ko 2024-04-20T21:08:44.190 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_koi8-r.ko 2024-04-20T21:08:44.190 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_koi8-ru.ko 2024-04-20T21:08:44.190 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_koi8-u.ko 2024-04-20T21:08:44.190 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_ucs2_utils.ko 2024-04-20T21:08:44.190 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_utf8.ko 2024-04-20T21:08:44.190 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2 2024-04-20T21:08:44.191 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/cluster 2024-04-20T21:08:44.191 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-04-20T21:08:44.191 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlm 2024-04-20T21:08:44.191 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-04-20T21:08:44.191 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlmfs 2024-04-20T21:08:44.191 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-04-20T21:08:44.191 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2.ko 2024-04-20T21:08:44.191 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-04-20T21:08:44.191 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-04-20T21:08:44.191 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-04-20T21:08:44.191 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/overlayfs 2024-04-20T21:08:44.191 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/overlayfs/overlay.ko 2024-04-20T21:08:44.191 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/pstore 2024-04-20T21:08:44.191 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/pstore/ramoops.ko 2024-04-20T21:08:44.191 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/reiserfs 2024-04-20T21:08:44.191 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/reiserfs/reiserfs.ko 2024-04-20T21:08:44.192 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/romfs 2024-04-20T21:08:44.192 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/romfs/romfs.ko 2024-04-20T21:08:44.192 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb 2024-04-20T21:08:44.192 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/client 2024-04-20T21:08:44.192 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/client/cifs.ko 2024-04-20T21:08:44.192 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/common 2024-04-20T21:08:44.192 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/common/cifs_arc4.ko 2024-04-20T21:08:44.192 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/common/cifs_md4.ko 2024-04-20T21:08:44.192 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/squashfs 2024-04-20T21:08:44.192 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/squashfs/squashfs.ko 2024-04-20T21:08:44.192 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/sysv 2024-04-20T21:08:44.192 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/sysv/sysv.ko 2024-04-20T21:08:44.192 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/udf 2024-04-20T21:08:44.192 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/udf/udf.ko 2024-04-20T21:08:44.192 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ufs 2024-04-20T21:08:44.192 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ufs/ufs.ko 2024-04-20T21:08:44.193 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/xfs 2024-04-20T21:08:44.193 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/xfs/xfs.ko 2024-04-20T21:08:44.193 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel 2024-04-20T21:08:44.193 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/rcu 2024-04-20T21:08:44.193 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/rcu/rcutorture.ko 2024-04-20T21:08:44.193 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/torture.ko 2024-04-20T21:08:44.193 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/trace 2024-04-20T21:08:44.193 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-04-20T21:08:44.193 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib 2024-04-20T21:08:44.193 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/asn1_encoder.ko 2024-04-20T21:08:44.193 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crc-itu-t.ko 2024-04-20T21:08:44.193 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crc8.ko 2024-04-20T21:08:44.193 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crypto 2024-04-20T21:08:44.193 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crypto/libarc4.ko 2024-04-20T21:08:44.193 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crypto/libdes.ko 2024-04-20T21:08:44.194 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/libcrc32c.ko 2024-04-20T21:08:44.194 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lru_cache.ko 2024-04-20T21:08:44.194 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lz4 2024-04-20T21:08:44.194 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lz4/lz4_compress.ko 2024-04-20T21:08:44.194 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lz4/lz4hc_compress.ko 2024-04-20T21:08:44.194 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/math 2024-04-20T21:08:44.194 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/math/cordic.ko 2024-04-20T21:08:44.194 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/raid6 2024-04-20T21:08:44.194 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/raid6/raid6_pq.ko 2024-04-20T21:08:44.194 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/reed_solomon 2024-04-20T21:08:44.194 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/reed_solomon/reed_solomon.ko 2024-04-20T21:08:44.194 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/ts_bm.ko 2024-04-20T21:08:44.194 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/ts_fsm.ko 2024-04-20T21:08:44.194 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/ts_kmp.ko 2024-04-20T21:08:44.194 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/zstd 2024-04-20T21:08:44.195 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/zstd/zstd_compress.ko 2024-04-20T21:08:44.195 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/mm 2024-04-20T21:08:44.195 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/mm/hwpoison-inject.ko 2024-04-20T21:08:44.195 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net 2024-04-20T21:08:44.195 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan 2024-04-20T21:08:44.195 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/6lowpan.ko 2024-04-20T21:08:44.195 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_dest.ko 2024-04-20T21:08:44.195 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_fragment.ko 2024-04-20T21:08:44.195 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_hop.ko 2024-04-20T21:08:44.195 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_ipv6.ko 2024-04-20T21:08:44.195 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_mobility.ko 2024-04-20T21:08:44.195 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_routing.ko 2024-04-20T21:08:44.195 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_udp.ko 2024-04-20T21:08:44.195 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802 2024-04-20T21:08:44.195 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/garp.ko 2024-04-20T21:08:44.195 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/mrp.ko 2024-04-20T21:08:44.196 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/p8022.ko 2024-04-20T21:08:44.196 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/psnap.ko 2024-04-20T21:08:44.196 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/stp.ko 2024-04-20T21:08:44.196 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/8021q 2024-04-20T21:08:44.196 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/8021q/8021q.ko 2024-04-20T21:08:44.196 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p 2024-04-20T21:08:44.197 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet.ko 2024-04-20T21:08:44.197 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet_fd.ko 2024-04-20T21:08:44.197 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet_rdma.ko 2024-04-20T21:08:44.197 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet_virtio.ko 2024-04-20T21:08:44.197 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/batman-adv 2024-04-20T21:08:44.197 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/batman-adv/batman-adv.ko 2024-04-20T21:08:44.197 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge 2024-04-20T21:08:44.197 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/br_netfilter.ko 2024-04-20T21:08:44.198 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/bridge.ko 2024-04-20T21:08:44.198 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter 2024-04-20T21:08:44.198 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-04-20T21:08:44.198 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_among.ko 2024-04-20T21:08:44.198 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_arp.ko 2024-04-20T21:08:44.198 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-04-20T21:08:44.198 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-04-20T21:08:44.198 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_ip.ko 2024-04-20T21:08:44.198 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-04-20T21:08:44.198 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_limit.ko 2024-04-20T21:08:44.198 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_log.ko 2024-04-20T21:08:44.198 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_mark.ko 2024-04-20T21:08:44.198 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-04-20T21:08:44.198 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-04-20T21:08:44.199 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-04-20T21:08:44.199 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-04-20T21:08:44.199 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_snat.ko 2024-04-20T21:08:44.199 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_stp.ko 2024-04-20T21:08:44.199 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-04-20T21:08:44.199 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-04-20T21:08:44.199 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-04-20T21:08:44.199 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-04-20T21:08:44.199 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtables.ko 2024-04-20T21:08:44.199 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-04-20T21:08:44.199 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-04-20T21:08:44.199 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-04-20T21:08:44.199 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ceph 2024-04-20T21:08:44.199 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ceph/libceph.ko 2024-04-20T21:08:44.199 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/core 2024-04-20T21:08:44.200 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/core/failover.ko 2024-04-20T21:08:44.200 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/core/pktgen.ko 2024-04-20T21:08:44.200 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp 2024-04-20T21:08:44.200 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp.ko 2024-04-20T21:08:44.200 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp_diag.ko 2024-04-20T21:08:44.200 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp_ipv4.ko 2024-04-20T21:08:44.200 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp_ipv6.ko 2024-04-20T21:08:44.200 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dns_resolver 2024-04-20T21:08:44.200 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dns_resolver/dns_resolver.ko 2024-04-20T21:08:44.200 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa 2024-04-20T21:08:44.200 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/dsa_core.ko 2024-04-20T21:08:44.200 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_brcm.ko 2024-04-20T21:08:44.200 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_dsa.ko 2024-04-20T21:08:44.200 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_none.ko 2024-04-20T21:08:44.200 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_trailer.ko 2024-04-20T21:08:44.201 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154 2024-04-20T21:08:44.201 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/6lowpan 2024-04-20T21:08:44.201 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-04-20T21:08:44.201 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/ieee802154.ko 2024-04-20T21:08:44.201 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/ieee802154_socket.ko 2024-04-20T21:08:44.201 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4 2024-04-20T21:08:44.201 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ah4.ko 2024-04-20T21:08:44.201 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/esp4.ko 2024-04-20T21:08:44.201 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/fou.ko 2024-04-20T21:08:44.201 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/gre.ko 2024-04-20T21:08:44.201 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/inet_diag.ko 2024-04-20T21:08:44.201 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ip_gre.ko 2024-04-20T21:08:44.201 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ip_tunnel.ko 2024-04-20T21:08:44.201 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ip_vti.ko 2024-04-20T21:08:44.201 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ipcomp.ko 2024-04-20T21:08:44.202 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ipip.ko 2024-04-20T21:08:44.202 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter 2024-04-20T21:08:44.202 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/arp_tables.ko 2024-04-20T21:08:44.202 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-04-20T21:08:44.202 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-04-20T21:08:44.202 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-04-20T21:08:44.202 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-04-20T21:08:44.202 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-04-20T21:08:44.202 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-04-20T21:08:44.202 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-04-20T21:08:44.202 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-04-20T21:08:44.202 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-04-20T21:08:44.202 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_security.ko 2024-04-20T21:08:44.202 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-04-20T21:08:44.203 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-04-20T21:08:44.203 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-04-20T21:08:44.203 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-04-20T21:08:44.203 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-04-20T21:08:44.203 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-04-20T21:08:44.203 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-04-20T21:08:44.203 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-04-20T21:08:44.203 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-04-20T21:08:44.203 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-04-20T21:08:44.203 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_bic.ko 2024-04-20T21:08:44.203 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_dctcp.ko 2024-04-20T21:08:44.203 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_diag.ko 2024-04-20T21:08:44.203 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_highspeed.ko 2024-04-20T21:08:44.203 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_htcp.ko 2024-04-20T21:08:44.203 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_hybla.ko 2024-04-20T21:08:44.204 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_illinois.ko 2024-04-20T21:08:44.204 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_lp.ko 2024-04-20T21:08:44.204 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_scalable.ko 2024-04-20T21:08:44.204 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_vegas.ko 2024-04-20T21:08:44.204 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_veno.ko 2024-04-20T21:08:44.204 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_westwood.ko 2024-04-20T21:08:44.204 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_yeah.ko 2024-04-20T21:08:44.204 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tunnel4.ko 2024-04-20T21:08:44.204 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/udp_diag.ko 2024-04-20T21:08:44.204 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/udp_tunnel.ko 2024-04-20T21:08:44.204 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/xfrm4_tunnel.ko 2024-04-20T21:08:44.204 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6 2024-04-20T21:08:44.204 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ah6.ko 2024-04-20T21:08:44.204 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/esp6.ko 2024-04-20T21:08:44.204 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/fou6.ko 2024-04-20T21:08:44.204 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ip6_tunnel.ko 2024-04-20T21:08:44.205 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-04-20T21:08:44.205 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ip6_vti.ko 2024-04-20T21:08:44.205 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ipcomp6.ko 2024-04-20T21:08:44.205 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter 2024-04-20T21:08:44.205 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-04-20T21:08:44.205 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-04-20T21:08:44.205 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-04-20T21:08:44.205 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-04-20T21:08:44.205 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-04-20T21:08:44.205 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-04-20T21:08:44.205 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-04-20T21:08:44.205 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-04-20T21:08:44.205 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-04-20T21:08:44.205 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-04-20T21:08:44.205 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-04-20T21:08:44.206 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-04-20T21:08:44.206 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-04-20T21:08:44.206 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-04-20T21:08:44.206 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-04-20T21:08:44.206 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-04-20T21:08:44.206 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-04-20T21:08:44.206 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-04-20T21:08:44.206 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-04-20T21:08:44.206 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-04-20T21:08:44.206 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-04-20T21:08:44.206 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-04-20T21:08:44.206 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-04-20T21:08:44.206 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-04-20T21:08:44.206 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/sit.ko 2024-04-20T21:08:44.206 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/tunnel6.ko 2024-04-20T21:08:44.207 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/xfrm6_tunnel.ko 2024-04-20T21:08:44.207 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/key 2024-04-20T21:08:44.207 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/key/af_key.ko 2024-04-20T21:08:44.207 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp 2024-04-20T21:08:44.207 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_core.ko 2024-04-20T21:08:44.207 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_debugfs.ko 2024-04-20T21:08:44.207 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_eth.ko 2024-04-20T21:08:44.207 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_ip.ko 2024-04-20T21:08:44.207 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_ip6.ko 2024-04-20T21:08:44.207 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_netlink.ko 2024-04-20T21:08:44.207 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_ppp.ko 2024-04-20T21:08:44.207 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/llc 2024-04-20T21:08:44.207 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/llc/llc.ko 2024-04-20T21:08:44.207 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mac802154 2024-04-20T21:08:44.207 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mac802154/mac802154.ko 2024-04-20T21:08:44.208 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mpls 2024-04-20T21:08:44.208 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mpls/mpls_gso.ko 2024-04-20T21:08:44.208 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter 2024-04-20T21:08:44.208 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset 2024-04-20T21:08:44.208 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set.ko 2024-04-20T21:08:44.208 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-04-20T21:08:44.208 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-04-20T21:08:44.208 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-04-20T21:08:44.208 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-04-20T21:08:44.208 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-04-20T21:08:44.208 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-04-20T21:08:44.208 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-04-20T21:08:44.208 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-04-20T21:08:44.208 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-04-20T21:08:44.209 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-04-20T21:08:44.209 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-04-20T21:08:44.209 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-04-20T21:08:44.209 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-04-20T21:08:44.209 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-04-20T21:08:44.209 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-04-20T21:08:44.209 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs 2024-04-20T21:08:44.209 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs.ko 2024-04-20T21:08:44.209 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-04-20T21:08:44.209 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-04-20T21:08:44.209 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-04-20T21:08:44.209 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-04-20T21:08:44.209 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-04-20T21:08:44.209 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-04-20T21:08:44.209 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-04-20T21:08:44.209 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-04-20T21:08:44.210 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-04-20T21:08:44.210 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-04-20T21:08:44.210 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-04-20T21:08:44.210 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-04-20T21:08:44.210 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-04-20T21:08:44.210 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conncount.ko 2024-04-20T21:08:44.210 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack.ko 2024-04-20T21:08:44.210 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-04-20T21:08:44.210 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-04-20T21:08:44.210 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-04-20T21:08:44.210 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_h323.ko 2024-04-20T21:08:44.210 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_irc.ko 2024-04-20T21:08:44.210 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-04-20T21:08:44.211 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-04-20T21:08:44.211 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-04-20T21:08:44.211 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_sane.ko 2024-04-20T21:08:44.211 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_sip.ko 2024-04-20T21:08:44.211 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-04-20T21:08:44.211 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-04-20T21:08:44.211 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_dup_netdev.ko 2024-04-20T21:08:44.211 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_flow_table.ko 2024-04-20T21:08:44.211 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_flow_table_inet.ko 2024-04-20T21:08:44.211 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_log_syslog.ko 2024-04-20T21:08:44.211 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat.ko 2024-04-20T21:08:44.211 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_amanda.ko 2024-04-20T21:08:44.211 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_ftp.ko 2024-04-20T21:08:44.211 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_irc.ko 2024-04-20T21:08:44.212 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_sip.ko 2024-04-20T21:08:44.212 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_tftp.ko 2024-04-20T21:08:44.212 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_synproxy_core.ko 2024-04-20T21:08:44.212 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_tables.ko 2024-04-20T21:08:44.212 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink.ko 2024-04-20T21:08:44.212 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_acct.ko 2024-04-20T21:08:44.212 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_log.ko 2024-04-20T21:08:44.212 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_osf.ko 2024-04-20T21:08:44.212 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_queue.ko 2024-04-20T21:08:44.212 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_chain_nat.ko 2024-04-20T21:08:44.212 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_compat.ko 2024-04-20T21:08:44.212 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_ct.ko 2024-04-20T21:08:44.213 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_dup_netdev.ko 2024-04-20T21:08:44.213 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_fib.ko 2024-04-20T21:08:44.213 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_fwd_netdev.ko 2024-04-20T21:08:44.213 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_hash.ko 2024-04-20T21:08:44.213 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_limit.ko 2024-04-20T21:08:44.213 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_log.ko 2024-04-20T21:08:44.213 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_masq.ko 2024-04-20T21:08:44.213 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_nat.ko 2024-04-20T21:08:44.213 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_numgen.ko 2024-04-20T21:08:44.213 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_queue.ko 2024-04-20T21:08:44.213 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_quota.ko 2024-04-20T21:08:44.213 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_redir.ko 2024-04-20T21:08:44.213 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_reject.ko 2024-04-20T21:08:44.213 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_reject_inet.ko 2024-04-20T21:08:44.213 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_socket.ko 2024-04-20T21:08:44.214 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_tunnel.ko 2024-04-20T21:08:44.214 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_AUDIT.ko 2024-04-20T21:08:44.214 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CHECKSUM.ko 2024-04-20T21:08:44.214 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CLASSIFY.ko 2024-04-20T21:08:44.214 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-04-20T21:08:44.214 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CT.ko 2024-04-20T21:08:44.214 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_DSCP.ko 2024-04-20T21:08:44.214 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_HL.ko 2024-04-20T21:08:44.214 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_HMARK.ko 2024-04-20T21:08:44.214 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_IDLETIMER.ko 2024-04-20T21:08:44.215 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_LED.ko 2024-04-20T21:08:44.215 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_LOG.ko 2024-04-20T21:08:44.215 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_MASQUERADE.ko 2024-04-20T21:08:44.215 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_NETMAP.ko 2024-04-20T21:08:44.215 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_NFLOG.ko 2024-04-20T21:08:44.215 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_NFQUEUE.ko 2024-04-20T21:08:44.215 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_RATEEST.ko 2024-04-20T21:08:44.215 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_REDIRECT.ko 2024-04-20T21:08:44.215 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_SECMARK.ko 2024-04-20T21:08:44.215 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TCPMSS.ko 2024-04-20T21:08:44.215 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-04-20T21:08:44.215 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TEE.ko 2024-04-20T21:08:44.215 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TPROXY.ko 2024-04-20T21:08:44.215 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TRACE.ko 2024-04-20T21:08:44.215 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_addrtype.ko 2024-04-20T21:08:44.216 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_bpf.ko 2024-04-20T21:08:44.216 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_cgroup.ko 2024-04-20T21:08:44.216 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_cluster.ko 2024-04-20T21:08:44.216 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_comment.ko 2024-04-20T21:08:44.216 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connbytes.ko 2024-04-20T21:08:44.216 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connlabel.ko 2024-04-20T21:08:44.216 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connlimit.ko 2024-04-20T21:08:44.216 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connmark.ko 2024-04-20T21:08:44.216 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_conntrack.ko 2024-04-20T21:08:44.216 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_cpu.ko 2024-04-20T21:08:44.216 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_dccp.ko 2024-04-20T21:08:44.216 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_devgroup.ko 2024-04-20T21:08:44.216 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_dscp.ko 2024-04-20T21:08:44.216 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_ecn.ko 2024-04-20T21:08:44.216 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_esp.ko 2024-04-20T21:08:44.217 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_hashlimit.ko 2024-04-20T21:08:44.217 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_helper.ko 2024-04-20T21:08:44.217 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_hl.ko 2024-04-20T21:08:44.217 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_ipcomp.ko 2024-04-20T21:08:44.217 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_iprange.ko 2024-04-20T21:08:44.217 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_ipvs.ko 2024-04-20T21:08:44.217 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_l2tp.ko 2024-04-20T21:08:44.217 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_length.ko 2024-04-20T21:08:44.217 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_limit.ko 2024-04-20T21:08:44.217 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_mac.ko 2024-04-20T21:08:44.217 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_mark.ko 2024-04-20T21:08:44.217 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_multiport.ko 2024-04-20T21:08:44.217 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_nat.ko 2024-04-20T21:08:44.217 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_nfacct.ko 2024-04-20T21:08:44.218 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_osf.ko 2024-04-20T21:08:44.218 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_owner.ko 2024-04-20T21:08:44.218 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_physdev.ko 2024-04-20T21:08:44.218 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_pkttype.ko 2024-04-20T21:08:44.218 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_policy.ko 2024-04-20T21:08:44.218 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_quota.ko 2024-04-20T21:08:44.218 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_rateest.ko 2024-04-20T21:08:44.218 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_realm.ko 2024-04-20T21:08:44.218 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_recent.ko 2024-04-20T21:08:44.218 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_sctp.ko 2024-04-20T21:08:44.218 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_set.ko 2024-04-20T21:08:44.218 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_socket.ko 2024-04-20T21:08:44.218 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_state.ko 2024-04-20T21:08:44.218 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_statistic.ko 2024-04-20T21:08:44.218 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_string.ko 2024-04-20T21:08:44.219 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_tcpmss.ko 2024-04-20T21:08:44.219 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_time.ko 2024-04-20T21:08:44.219 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_u32.ko 2024-04-20T21:08:44.219 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netlink 2024-04-20T21:08:44.219 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netlink/netlink_diag.ko 2024-04-20T21:08:44.219 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/nsh 2024-04-20T21:08:44.219 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/nsh/nsh.ko 2024-04-20T21:08:44.219 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch 2024-04-20T21:08:44.219 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/openvswitch.ko 2024-04-20T21:08:44.219 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/vport-geneve.ko 2024-04-20T21:08:44.219 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/vport-gre.ko 2024-04-20T21:08:44.219 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/vport-vxlan.ko 2024-04-20T21:08:44.219 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/packet 2024-04-20T21:08:44.219 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/packet/af_packet_diag.ko 2024-04-20T21:08:44.219 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds 2024-04-20T21:08:44.220 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds/rds.ko 2024-04-20T21:08:44.220 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds/rds_rdma.ko 2024-04-20T21:08:44.220 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds/rds_tcp.ko 2024-04-20T21:08:44.220 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched 2024-04-20T21:08:44.220 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_csum.ko 2024-04-20T21:08:44.220 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_gact.ko 2024-04-20T21:08:44.220 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_mirred.ko 2024-04-20T21:08:44.220 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_nat.ko 2024-04-20T21:08:44.220 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_pedit.ko 2024-04-20T21:08:44.220 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_police.ko 2024-04-20T21:08:44.220 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_simple.ko 2024-04-20T21:08:44.220 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_skbedit.ko 2024-04-20T21:08:44.220 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_vlan.ko 2024-04-20T21:08:44.220 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_basic.ko 2024-04-20T21:08:44.220 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_bpf.ko 2024-04-20T21:08:44.221 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_flow.ko 2024-04-20T21:08:44.221 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_fw.ko 2024-04-20T21:08:44.221 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_route.ko 2024-04-20T21:08:44.221 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_u32.ko 2024-04-20T21:08:44.221 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_cmp.ko 2024-04-20T21:08:44.221 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_ipset.ko 2024-04-20T21:08:44.221 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_meta.ko 2024-04-20T21:08:44.221 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_nbyte.ko 2024-04-20T21:08:44.221 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_text.ko 2024-04-20T21:08:44.221 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_u32.ko 2024-04-20T21:08:44.221 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_choke.ko 2024-04-20T21:08:44.221 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_codel.ko 2024-04-20T21:08:44.221 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_drr.ko 2024-04-20T21:08:44.221 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_fq.ko 2024-04-20T21:08:44.221 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_gred.ko 2024-04-20T21:08:44.222 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_hfsc.ko 2024-04-20T21:08:44.222 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_hhf.ko 2024-04-20T21:08:44.222 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_htb.ko 2024-04-20T21:08:44.222 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_ingress.ko 2024-04-20T21:08:44.222 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_mqprio.ko 2024-04-20T21:08:44.222 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_mqprio_lib.ko 2024-04-20T21:08:44.222 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_multiq.ko 2024-04-20T21:08:44.222 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_netem.ko 2024-04-20T21:08:44.222 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_pie.ko 2024-04-20T21:08:44.222 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_plug.ko 2024-04-20T21:08:44.222 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_prio.ko 2024-04-20T21:08:44.225 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_qfq.ko 2024-04-20T21:08:44.226 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_red.ko 2024-04-20T21:08:44.226 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_sfb.ko 2024-04-20T21:08:44.226 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_sfq.ko 2024-04-20T21:08:44.226 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_tbf.ko 2024-04-20T21:08:44.226 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_teql.ko 2024-04-20T21:08:44.226 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sctp 2024-04-20T21:08:44.226 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sctp/sctp.ko 2024-04-20T21:08:44.226 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sctp/sctp_diag.ko 2024-04-20T21:08:44.226 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc 2024-04-20T21:08:44.226 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/auth_gss 2024-04-20T21:08:44.226 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-04-20T21:08:44.226 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-04-20T21:08:44.227 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/sunrpc.ko 2024-04-20T21:08:44.227 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/xprtrdma 2024-04-20T21:08:44.227 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-04-20T21:08:44.227 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/tipc 2024-04-20T21:08:44.227 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/tipc/tipc.ko 2024-04-20T21:08:44.227 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/tipc/tipc_diag.ko 2024-04-20T21:08:44.227 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/unix 2024-04-20T21:08:44.227 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/unix/unix_diag.ko 2024-04-20T21:08:44.227 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock 2024-04-20T21:08:44.227 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-04-20T21:08:44.227 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-04-20T21:08:44.227 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vsock.ko 2024-04-20T21:08:44.227 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vsock_diag.ko 2024-04-20T21:08:44.227 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vsock_loopback.ko 2024-04-20T21:08:44.227 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/xfrm 2024-04-20T21:08:44.228 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/xfrm/xfrm_ipcomp.ko 2024-04-20T21:08:44.228 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security 2024-04-20T21:08:44.228 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys 2024-04-20T21:08:44.228 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/encrypted-keys 2024-04-20T21:08:44.228 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-04-20T21:08:44.228 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/trusted-keys 2024-04-20T21:08:44.228 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/trusted-keys/trusted.ko 2024-04-20T21:08:44.228 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.alias 2024-04-20T21:08:44.228 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.alias.bin 2024-04-20T21:08:44.228 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin 2024-04-20T21:08:44.228 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin.alias.bin 2024-04-20T21:08:44.228 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin.bin 2024-04-20T21:08:44.228 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin.modinfo 2024-04-20T21:08:44.228 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.dep 2024-04-20T21:08:44.228 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.dep.bin 2024-04-20T21:08:44.229 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.devname 2024-04-20T21:08:44.229 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.order 2024-04-20T21:08:44.229 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.softdep 2024-04-20T21:08:44.229 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.symbols 2024-04-20T21:08:44.229 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.symbols.bin 2024-04-20T21:08:44.229 INFO:teuthology.orchestra.run.smithi062.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/vmlinuz 2024-04-20T21:08:44.233 DEBUG:teuthology.task.kernel:get_image_version: 6.9.0-rc3-g806ec36b2112 2024-04-20T21:08:44.233 DEBUG:teuthology.orchestra.run.smithi062:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-04-20T21:08:44.257 INFO:teuthology.orchestra.run.smithi062.stdout:postinstall scriptlet (using /bin/sh): 2024-04-20T21:08:44.257 INFO:teuthology.orchestra.run.smithi062.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-04-20T21:08:44.257 INFO:teuthology.orchestra.run.smithi062.stdout: /usr/bin/kernel-install add 6.9.0-rc3-g806ec36b2112 /lib/modules/6.9.0-rc3-g806ec36b2112/vmlinuz 2024-04-20T21:08:44.257 INFO:teuthology.orchestra.run.smithi062.stdout:fi 2024-04-20T21:08:44.257 INFO:teuthology.orchestra.run.smithi062.stdout:for file in vmlinuz System.map config; do 2024-04-20T21:08:44.257 INFO:teuthology.orchestra.run.smithi062.stdout: if ! cmp --silent "/lib/modules/6.9.0-rc3-g806ec36b2112/${file}" "/boot/${file}-6.9.0-rc3-g806ec36b2112"; then 2024-04-20T21:08:44.257 INFO:teuthology.orchestra.run.smithi062.stdout: cp "/lib/modules/6.9.0-rc3-g806ec36b2112/${file}" "/boot/${file}-6.9.0-rc3-g806ec36b2112" 2024-04-20T21:08:44.257 INFO:teuthology.orchestra.run.smithi062.stdout: fi 2024-04-20T21:08:44.257 INFO:teuthology.orchestra.run.smithi062.stdout:done 2024-04-20T21:08:44.257 INFO:teuthology.orchestra.run.smithi062.stdout:if [ -d "/lib/modules/6.9.0-rc3-g806ec36b2112/dtb" ] && \ 2024-04-20T21:08:44.257 INFO:teuthology.orchestra.run.smithi062.stdout: ! diff -rq "/lib/modules/6.9.0-rc3-g806ec36b2112/dtb" "/boot/dtb-6.9.0-rc3-g806ec36b2112" >/dev/null 2>&1; then 2024-04-20T21:08:44.258 INFO:teuthology.orchestra.run.smithi062.stdout: rm -rf "/boot/dtb-6.9.0-rc3-g806ec36b2112" 2024-04-20T21:08:44.258 INFO:teuthology.orchestra.run.smithi062.stdout: cp -r "/lib/modules/6.9.0-rc3-g806ec36b2112/dtb" "/boot/dtb-6.9.0-rc3-g806ec36b2112" 2024-04-20T21:08:44.258 INFO:teuthology.orchestra.run.smithi062.stdout:fi 2024-04-20T21:08:44.258 INFO:teuthology.orchestra.run.smithi062.stdout:if [ ! -e "/lib/modules/6.9.0-rc3-g806ec36b2112/modules.dep" ]; then 2024-04-20T21:08:44.258 INFO:teuthology.orchestra.run.smithi062.stdout: /usr/sbin/depmod 6.9.0-rc3-g806ec36b2112 2024-04-20T21:08:44.258 INFO:teuthology.orchestra.run.smithi062.stdout:fi 2024-04-20T21:08:44.258 INFO:teuthology.orchestra.run.smithi062.stdout:preuninstall scriptlet (using /bin/sh): 2024-04-20T21:08:44.258 INFO:teuthology.orchestra.run.smithi062.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-04-20T21:08:44.258 INFO:teuthology.orchestra.run.smithi062.stdout:kernel-install remove 6.9.0-rc3-g806ec36b2112 2024-04-20T21:08:44.258 INFO:teuthology.orchestra.run.smithi062.stdout:fi 2024-04-20T21:08:44.259 DEBUG:teuthology.orchestra.run.smithi062:> sudo rpm -qi grub2-tools 2024-04-20T21:08:44.332 INFO:teuthology.orchestra.run.smithi062.stdout:Name : grub2-tools 2024-04-20T21:08:44.332 INFO:teuthology.orchestra.run.smithi062.stdout:Epoch : 1 2024-04-20T21:08:44.333 INFO:teuthology.orchestra.run.smithi062.stdout:Version : 2.06 2024-04-20T21:08:44.333 INFO:teuthology.orchestra.run.smithi062.stdout:Release : 68.el9 2024-04-20T21:08:44.333 INFO:teuthology.orchestra.run.smithi062.stdout:Architecture: x86_64 2024-04-20T21:08:44.333 INFO:teuthology.orchestra.run.smithi062.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-04-20T21:08:44.333 INFO:teuthology.orchestra.run.smithi062.stdout:Group : Unspecified 2024-04-20T21:08:44.333 INFO:teuthology.orchestra.run.smithi062.stdout:Size : 8264254 2024-04-20T21:08:44.333 INFO:teuthology.orchestra.run.smithi062.stdout:License : GPLv3+ 2024-04-20T21:08:44.333 INFO:teuthology.orchestra.run.smithi062.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-20T21:08:44.333 INFO:teuthology.orchestra.run.smithi062.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-20T21:08:44.333 INFO:teuthology.orchestra.run.smithi062.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-20T21:08:44.333 INFO:teuthology.orchestra.run.smithi062.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-20T21:08:44.333 INFO:teuthology.orchestra.run.smithi062.stdout:Packager : builder@centos.org 2024-04-20T21:08:44.333 INFO:teuthology.orchestra.run.smithi062.stdout:Vendor : CentOS 2024-04-20T21:08:44.333 INFO:teuthology.orchestra.run.smithi062.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-20T21:08:44.333 INFO:teuthology.orchestra.run.smithi062.stdout:Summary : Support tools for GRUB. 2024-04-20T21:08:44.334 INFO:teuthology.orchestra.run.smithi062.stdout:Description : 2024-04-20T21:08:44.334 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:08:44.334 INFO:teuthology.orchestra.run.smithi062.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-20T21:08:44.334 INFO:teuthology.orchestra.run.smithi062.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-20T21:08:44.334 INFO:teuthology.orchestra.run.smithi062.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-20T21:08:44.334 INFO:teuthology.orchestra.run.smithi062.stdout:hardware devices. 2024-04-20T21:08:44.334 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:08:44.334 INFO:teuthology.orchestra.run.smithi062.stdout:This subpackage provides tools for support of all platforms. 2024-04-20T21:08:44.335 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-20T21:08:44.335 INFO:teuthology.task.kernel:Updating grub on smithi062 to boot 6.9.0-rc3-g806ec36b2112 2024-04-20T21:08:44.335 DEBUG:teuthology.orchestra.run.smithi062:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-20T21:08:44.872 INFO:teuthology.orchestra.run.smithi062.stderr:Generating grub configuration file ... 2024-04-20T21:08:46.314 INFO:teuthology.orchestra.run.smithi062.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-20T21:08:46.345 INFO:teuthology.orchestra.run.smithi062.stderr:done 2024-04-20T21:08:46.347 DEBUG:teuthology.orchestra.run.smithi062:> mktemp 2024-04-20T21:08:46.362 INFO:teuthology.orchestra.run.smithi062.stdout:/tmp/tmp.HvpfuZ4Zhd 2024-04-20T21:08:46.362 DEBUG:teuthology.orchestra.run.smithi062:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.HvpfuZ4Zhd 2024-04-20T21:08:46.427 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod 0666 /tmp/tmp.HvpfuZ4Zhd 2024-04-20T21:08:46.570 DEBUG:teuthology.orchestra.remote:smithi062:/tmp/tmp.HvpfuZ4Zhd is 6KB 2024-04-20T21:08:46.618 DEBUG:teuthology.orchestra.run.smithi062:> rm -fr /tmp/tmp.HvpfuZ4Zhd 2024-04-20T21:08:46.646 DEBUG:teuthology.orchestra.run.smithi062:> sudo /bin/ls /boot/loader/entries || true 2024-04-20T21:08:46.694 INFO:teuthology.orchestra.run.smithi062.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-04-20T21:08:46.695 INFO:teuthology.orchestra.run.smithi062.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-04-20T21:08:46.695 INFO:teuthology.orchestra.run.smithi062.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-04-20T21:08:46.695 INFO:teuthology.orchestra.run.smithi062.stdout:7fc8828e06d0482694bcf4a447d13cb6-6.9.0-rc3-g806ec36b2112.conf 2024-04-20T21:08:46.696 DEBUG:teuthology.orchestra.run.smithi062:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-6.9.0-rc3-g806ec36b2112 2024-04-20T21:08:46.860 DEBUG:teuthology.orchestra.run.smithi062:> sudo shutdown -r now 2024-04-20T21:09:16.864 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-04-20T21:09:16.864 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi062.front.sepia.ceph.com' 2024-04-20T21:09:16.865 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi062.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-20T21:09:28.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20 21:09:28 URL:https://4.chacra.ceph.com/r/kernel/testing/806ec36b21129f113ce356c007dd59694d340b37/centos/9/flavors/default/x86_64/kernel-6.9.0_rc3_g806ec36b2112-1.x86_64.rpm [357367792/357367792] -> "/tmp/kernel.x86_64.rpm" [1] 2024-04-20T21:09:28.641 INFO:teuthology.orchestra.run.smithi189.stderr:FINISHED --2024-04-20 21:09:28-- 2024-04-20T21:09:28.641 INFO:teuthology.orchestra.run.smithi189.stderr:Total wall clock time: 8m 42s 2024-04-20T21:09:28.641 INFO:teuthology.orchestra.run.smithi189.stderr:Downloaded: 1 files, 341M in 8m 41s (669 KB/s) 2024-04-20T21:09:28.644 INFO:teuthology.task.kernel:Installing kernel 806ec36b21129f113ce356c007dd59694d340b37 on host.b... 2024-04-20T21:09:28.644 DEBUG:teuthology.orchestra.run.smithi189:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-04-20T21:09:30.755 INFO:teuthology.orchestra.run.smithi189.stdout:Verifying... ######################################## 2024-04-20T21:09:30.885 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing... ######################################## 2024-04-20T21:09:30.992 INFO:teuthology.orchestra.run.smithi189.stdout:Updating / installing... 2024-04-20T21:09:38.883 INFO:teuthology.orchestra.run.smithi189.stdout:kernel-6.9.0_rc3_g806ec36b2112-1 ######################################## 2024-04-20T21:10:15.959 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi189.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-04-20T21:10:15.959 DEBUG:teuthology.orchestra.run.smithi189:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-04-20T21:10:16.042 INFO:teuthology.orchestra.run.smithi189.stdout:/boot/System.map-6.9.0-rc3-g806ec36b2112 2024-04-20T21:10:16.043 INFO:teuthology.orchestra.run.smithi189.stdout:/boot/config-6.9.0-rc3-g806ec36b2112 2024-04-20T21:10:16.043 INFO:teuthology.orchestra.run.smithi189.stdout:/boot/vmlinuz-6.9.0-rc3-g806ec36b2112 2024-04-20T21:10:16.043 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/System.map 2024-04-20T21:10:16.043 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/config 2024-04-20T21:10:16.043 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel 2024-04-20T21:10:16.043 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch 2024-04-20T21:10:16.043 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86 2024-04-20T21:10:16.043 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto 2024-04-20T21:10:16.043 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-04-20T21:10:16.043 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-04-20T21:10:16.043 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-04-20T21:10:16.043 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-04-20T21:10:16.043 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-04-20T21:10:16.044 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-04-20T21:10:16.044 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-04-20T21:10:16.044 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/crc32c-intel.ko 2024-04-20T21:10:16.044 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-04-20T21:10:16.044 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-04-20T21:10:16.044 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-04-20T21:10:16.044 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-04-20T21:10:16.044 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/serpent-avx2.ko 2024-04-20T21:10:16.044 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-04-20T21:10:16.044 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-04-20T21:10:16.044 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-04-20T21:10:16.044 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-04-20T21:10:16.044 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-04-20T21:10:16.044 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-04-20T21:10:16.045 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-04-20T21:10:16.045 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel 2024-04-20T21:10:16.045 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel/cpu 2024-04-20T21:10:16.045 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel/cpu/mce 2024-04-20T21:10:16.045 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-04-20T21:10:16.045 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm 2024-04-20T21:10:16.045 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm/kvm-amd.ko 2024-04-20T21:10:16.045 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm/kvm-intel.ko 2024-04-20T21:10:16.045 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm/kvm.ko 2024-04-20T21:10:16.045 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto 2024-04-20T21:10:16.045 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/ansi_cprng.ko 2024-04-20T21:10:16.045 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/anubis.ko 2024-04-20T21:10:16.045 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/arc4.ko 2024-04-20T21:10:16.045 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx 2024-04-20T21:10:16.045 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_memcpy.ko 2024-04-20T21:10:16.046 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_pq.ko 2024-04-20T21:10:16.046 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_raid6_recov.ko 2024-04-20T21:10:16.046 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_tx.ko 2024-04-20T21:10:16.046 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_xor.ko 2024-04-20T21:10:16.046 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/raid6test.ko 2024-04-20T21:10:16.046 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/authenc.ko 2024-04-20T21:10:16.046 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/authencesn.ko 2024-04-20T21:10:16.046 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/blake2b_generic.ko 2024-04-20T21:10:16.046 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/blowfish_common.ko 2024-04-20T21:10:16.046 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/blowfish_generic.ko 2024-04-20T21:10:16.046 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/camellia_generic.ko 2024-04-20T21:10:16.046 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cast5_generic.ko 2024-04-20T21:10:16.046 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cast6_generic.ko 2024-04-20T21:10:16.047 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cast_common.ko 2024-04-20T21:10:16.047 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/ccm.ko 2024-04-20T21:10:16.047 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cmac.ko 2024-04-20T21:10:16.047 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/crc32_generic.ko 2024-04-20T21:10:16.047 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/crypto_engine.ko 2024-04-20T21:10:16.047 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/crypto_user.ko 2024-04-20T21:10:16.047 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cts.ko 2024-04-20T21:10:16.047 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/des_generic.ko 2024-04-20T21:10:16.047 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/dh_generic.ko 2024-04-20T21:10:16.047 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/echainiv.ko 2024-04-20T21:10:16.047 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/essiv.ko 2024-04-20T21:10:16.047 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/fcrypt.ko 2024-04-20T21:10:16.047 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/khazad.ko 2024-04-20T21:10:16.047 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/lz4.ko 2024-04-20T21:10:16.047 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/lz4hc.ko 2024-04-20T21:10:16.048 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/md4.ko 2024-04-20T21:10:16.048 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/michael_mic.ko 2024-04-20T21:10:16.048 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/pcbc.ko 2024-04-20T21:10:16.048 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/pcrypt.ko 2024-04-20T21:10:16.048 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/rmd160.ko 2024-04-20T21:10:16.048 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/seed.ko 2024-04-20T21:10:16.048 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/serpent_generic.ko 2024-04-20T21:10:16.048 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/tcrypt.ko 2024-04-20T21:10:16.048 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/tea.ko 2024-04-20T21:10:16.048 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/twofish_common.ko 2024-04-20T21:10:16.048 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/twofish_generic.ko 2024-04-20T21:10:16.049 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/vmac.ko 2024-04-20T21:10:16.049 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/wp512.ko 2024-04-20T21:10:16.049 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/xcbc.ko 2024-04-20T21:10:16.049 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/xor.ko 2024-04-20T21:10:16.049 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/xxhash_generic.ko 2024-04-20T21:10:16.049 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers 2024-04-20T21:10:16.049 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi 2024-04-20T21:10:16.049 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/acpi_ipmi.ko 2024-04-20T21:10:16.049 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/acpi_pad.ko 2024-04-20T21:10:16.049 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/ec_sys.ko 2024-04-20T21:10:16.049 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/platform_profile.ko 2024-04-20T21:10:16.049 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/sbs.ko 2024-04-20T21:10:16.049 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/sbshc.ko 2024-04-20T21:10:16.049 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/video.ko 2024-04-20T21:10:16.049 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata 2024-04-20T21:10:16.050 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/acard-ahci.ko 2024-04-20T21:10:16.050 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/ahci_platform.ko 2024-04-20T21:10:16.050 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/ata_generic.ko 2024-04-20T21:10:16.050 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/libahci_platform.ko 2024-04-20T21:10:16.050 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_acpi.ko 2024-04-20T21:10:16.050 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ali.ko 2024-04-20T21:10:16.050 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_amd.ko 2024-04-20T21:10:16.050 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_artop.ko 2024-04-20T21:10:16.050 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_atiixp.ko 2024-04-20T21:10:16.050 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_atp867x.ko 2024-04-20T21:10:16.050 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_cmd640.ko 2024-04-20T21:10:16.050 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_cmd64x.ko 2024-04-20T21:10:16.050 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_cypress.ko 2024-04-20T21:10:16.050 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_efar.ko 2024-04-20T21:10:16.051 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt366.ko 2024-04-20T21:10:16.051 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt37x.ko 2024-04-20T21:10:16.051 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt3x2n.ko 2024-04-20T21:10:16.051 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt3x3.ko 2024-04-20T21:10:16.051 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_it8213.ko 2024-04-20T21:10:16.051 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_it821x.ko 2024-04-20T21:10:16.051 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_jmicron.ko 2024-04-20T21:10:16.051 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_marvell.ko 2024-04-20T21:10:16.051 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_mpiix.ko 2024-04-20T21:10:16.051 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_netcell.ko 2024-04-20T21:10:16.051 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ninja32.ko 2024-04-20T21:10:16.051 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ns87410.ko 2024-04-20T21:10:16.051 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ns87415.ko 2024-04-20T21:10:16.051 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_oldpiix.ko 2024-04-20T21:10:16.052 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_opti.ko 2024-04-20T21:10:16.052 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_optidma.ko 2024-04-20T21:10:16.052 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_pcmcia.ko 2024-04-20T21:10:16.052 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_pdc2027x.ko 2024-04-20T21:10:16.052 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-04-20T21:10:16.052 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_piccolo.ko 2024-04-20T21:10:16.052 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_rdc.ko 2024-04-20T21:10:16.052 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sch.ko 2024-04-20T21:10:16.052 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_serverworks.ko 2024-04-20T21:10:16.052 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sil680.ko 2024-04-20T21:10:16.052 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sis.ko 2024-04-20T21:10:16.052 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sl82c105.ko 2024-04-20T21:10:16.052 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_triflex.ko 2024-04-20T21:10:16.052 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_via.ko 2024-04-20T21:10:16.052 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pdc_adma.ko 2024-04-20T21:10:16.053 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_inic162x.ko 2024-04-20T21:10:16.053 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_mv.ko 2024-04-20T21:10:16.053 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_nv.ko 2024-04-20T21:10:16.053 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_promise.ko 2024-04-20T21:10:16.053 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_qstor.ko 2024-04-20T21:10:16.053 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sil.ko 2024-04-20T21:10:16.053 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sil24.ko 2024-04-20T21:10:16.053 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sis.ko 2024-04-20T21:10:16.053 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_svw.ko 2024-04-20T21:10:16.053 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sx4.ko 2024-04-20T21:10:16.053 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_uli.ko 2024-04-20T21:10:16.053 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_via.ko 2024-04-20T21:10:16.053 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_vsc.ko 2024-04-20T21:10:16.053 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/auxdisplay 2024-04-20T21:10:16.053 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/auxdisplay/ks0108.ko 2024-04-20T21:10:16.054 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base 2024-04-20T21:10:16.054 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base/regmap 2024-04-20T21:10:16.054 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base/regmap/regmap-i2c.ko 2024-04-20T21:10:16.054 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base/regmap/regmap-mmio.ko 2024-04-20T21:10:16.054 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/bcma 2024-04-20T21:10:16.054 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/bcma/bcma.ko 2024-04-20T21:10:16.054 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block 2024-04-20T21:10:16.054 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/aoe 2024-04-20T21:10:16.054 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/aoe/aoe.ko 2024-04-20T21:10:16.054 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/brd.ko 2024-04-20T21:10:16.054 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/drbd 2024-04-20T21:10:16.054 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/drbd/drbd.ko 2024-04-20T21:10:16.054 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/floppy.ko 2024-04-20T21:10:16.054 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/loop.ko 2024-04-20T21:10:16.055 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/mtip32xx 2024-04-20T21:10:16.055 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-04-20T21:10:16.055 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/nbd.ko 2024-04-20T21:10:16.055 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/null_blk 2024-04-20T21:10:16.055 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/null_blk/null_blk.ko 2024-04-20T21:10:16.055 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/pktcdvd.ko 2024-04-20T21:10:16.055 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/rbd.ko 2024-04-20T21:10:16.055 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/virtio_blk.ko 2024-04-20T21:10:16.055 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/xen-blkback 2024-04-20T21:10:16.055 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-04-20T21:10:16.055 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/xen-blkfront.ko 2024-04-20T21:10:16.055 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/zram 2024-04-20T21:10:16.055 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/zram/zram.ko 2024-04-20T21:10:16.055 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char 2024-04-20T21:10:16.056 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hangcheck-timer.ko 2024-04-20T21:10:16.056 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random 2024-04-20T21:10:16.056 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/amd-rng.ko 2024-04-20T21:10:16.056 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/intel-rng.ko 2024-04-20T21:10:16.056 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-04-20T21:10:16.056 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/via-rng.ko 2024-04-20T21:10:16.056 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/virtio-rng.ko 2024-04-20T21:10:16.056 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi 2024-04-20T21:10:16.056 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-04-20T21:10:16.056 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-04-20T21:10:16.056 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-04-20T21:10:16.056 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_si.ko 2024-04-20T21:10:16.056 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-04-20T21:10:16.056 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-04-20T21:10:16.056 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/lp.ko 2024-04-20T21:10:16.057 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/mwave 2024-04-20T21:10:16.057 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/mwave/mwave.ko 2024-04-20T21:10:16.057 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ppdev.ko 2024-04-20T21:10:16.057 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tlclk.ko 2024-04-20T21:10:16.057 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm 2024-04-20T21:10:16.057 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm.ko 2024-04-20T21:10:16.057 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_atmel.ko 2024-04-20T21:10:16.057 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_infineon.ko 2024-04-20T21:10:16.057 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_nsc.ko 2024-04-20T21:10:16.057 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_tis.ko 2024-04-20T21:10:16.057 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-04-20T21:10:16.057 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/uv_mmtimer.ko 2024-04-20T21:10:16.057 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/virtio_console.ko 2024-04-20T21:10:16.057 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq 2024-04-20T21:10:16.058 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-04-20T21:10:16.058 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-04-20T21:10:16.058 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/p4-clockmod.ko 2024-04-20T21:10:16.058 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-04-20T21:10:16.058 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/powernow-k8.ko 2024-04-20T21:10:16.058 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/speedstep-lib.ko 2024-04-20T21:10:16.058 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto 2024-04-20T21:10:16.058 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/ccp 2024-04-20T21:10:16.058 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-04-20T21:10:16.058 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/ccp/ccp.ko 2024-04-20T21:10:16.058 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel 2024-04-20T21:10:16.058 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat 2024-04-20T21:10:16.058 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_common 2024-04-20T21:10:16.058 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-04-20T21:10:16.058 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-04-20T21:10:16.059 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-04-20T21:10:16.059 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/padlock-aes.ko 2024-04-20T21:10:16.059 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/padlock-sha.ko 2024-04-20T21:10:16.059 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/virtio 2024-04-20T21:10:16.059 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-04-20T21:10:16.059 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dca 2024-04-20T21:10:16.059 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dca/dca.ko 2024-04-20T21:10:16.059 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/devfreq 2024-04-20T21:10:16.059 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-04-20T21:10:16.059 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma 2024-04-20T21:10:16.059 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/dw 2024-04-20T21:10:16.059 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/dw/dw_dmac.ko 2024-04-20T21:10:16.059 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/ioat 2024-04-20T21:10:16.059 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/ioat/ioatdma.ko 2024-04-20T21:10:16.060 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac 2024-04-20T21:10:16.060 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/amd64_edac.ko 2024-04-20T21:10:16.060 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/e752x_edac.ko 2024-04-20T21:10:16.060 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/edac_mce_amd.ko 2024-04-20T21:10:16.060 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i3000_edac.ko 2024-04-20T21:10:16.060 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i3200_edac.ko 2024-04-20T21:10:16.060 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i5100_edac.ko 2024-04-20T21:10:16.060 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i5400_edac.ko 2024-04-20T21:10:16.060 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i7300_edac.ko 2024-04-20T21:10:16.060 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i7core_edac.ko 2024-04-20T21:10:16.060 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i82975x_edac.ko 2024-04-20T21:10:16.060 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/ie31200_edac.ko 2024-04-20T21:10:16.060 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/sb_edac.ko 2024-04-20T21:10:16.060 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/x38_edac.ko 2024-04-20T21:10:16.060 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/firmware 2024-04-20T21:10:16.061 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/firmware/edd.ko 2024-04-20T21:10:16.061 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/firmware/iscsi_ibft.ko 2024-04-20T21:10:16.061 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/gpio 2024-04-20T21:10:16.061 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/gpio/gpio-ich.ko 2024-04-20T21:10:16.061 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/gpio/gpio-viperboard.ko 2024-04-20T21:10:16.061 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid 2024-04-20T21:10:16.061 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-appleir.ko 2024-04-20T21:10:16.061 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-aureal.ko 2024-04-20T21:10:16.061 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-axff.ko 2024-04-20T21:10:16.061 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-dr.ko 2024-04-20T21:10:16.061 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-elecom.ko 2024-04-20T21:10:16.061 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-elo.ko 2024-04-20T21:10:16.061 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-emsff.ko 2024-04-20T21:10:16.061 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-gaff.ko 2024-04-20T21:10:16.061 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-gt683r.ko 2024-04-20T21:10:16.062 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-gyration.ko 2024-04-20T21:10:16.062 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-holtek-kbd.ko 2024-04-20T21:10:16.062 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-holtek-mouse.ko 2024-04-20T21:10:16.062 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-holtekff.ko 2024-04-20T21:10:16.062 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-hyperv.ko 2024-04-20T21:10:16.062 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-icade.ko 2024-04-20T21:10:16.062 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-keytouch.ko 2024-04-20T21:10:16.062 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-kye.ko 2024-04-20T21:10:16.062 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-lcpower.ko 2024-04-20T21:10:16.062 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-led.ko 2024-04-20T21:10:16.062 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-lenovo.ko 2024-04-20T21:10:16.062 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-logitech-dj.ko 2024-04-20T21:10:16.062 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-04-20T21:10:16.062 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-multitouch.ko 2024-04-20T21:10:16.063 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-ortek.ko 2024-04-20T21:10:16.063 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-penmount.ko 2024-04-20T21:10:16.063 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-petalynx.ko 2024-04-20T21:10:16.063 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-picolcd.ko 2024-04-20T21:10:16.063 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-pl.ko 2024-04-20T21:10:16.063 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-plantronics.ko 2024-04-20T21:10:16.063 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-primax.ko 2024-04-20T21:10:16.063 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-rmi.ko 2024-04-20T21:10:16.063 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-arvo.ko 2024-04-20T21:10:16.063 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-common.ko 2024-04-20T21:10:16.063 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-isku.ko 2024-04-20T21:10:16.063 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-kone.ko 2024-04-20T21:10:16.063 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-04-20T21:10:16.063 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-konepure.ko 2024-04-20T21:10:16.063 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-04-20T21:10:16.064 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-lua.ko 2024-04-20T21:10:16.064 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-pyra.ko 2024-04-20T21:10:16.064 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-ryos.ko 2024-04-20T21:10:16.064 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-savu.ko 2024-04-20T21:10:16.064 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat.ko 2024-04-20T21:10:16.064 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-saitek.ko 2024-04-20T21:10:16.064 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-samsung.ko 2024-04-20T21:10:16.064 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sensor-hub.ko 2024-04-20T21:10:16.064 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sjoy.ko 2024-04-20T21:10:16.064 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sony.ko 2024-04-20T21:10:16.064 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-speedlink.ko 2024-04-20T21:10:16.064 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-steelseries.ko 2024-04-20T21:10:16.064 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sunplus.ko 2024-04-20T21:10:16.064 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-thrustmaster.ko 2024-04-20T21:10:16.065 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-tivo.ko 2024-04-20T21:10:16.065 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-tmff.ko 2024-04-20T21:10:16.065 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-topseed.ko 2024-04-20T21:10:16.065 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-twinhan.ko 2024-04-20T21:10:16.065 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-uclogic.ko 2024-04-20T21:10:16.065 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-waltop.ko 2024-04-20T21:10:16.065 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-wiimote.ko 2024-04-20T21:10:16.065 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-xinmo.ko 2024-04-20T21:10:16.065 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-zpff.ko 2024-04-20T21:10:16.065 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-zydacron.ko 2024-04-20T21:10:16.065 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/uhid.ko 2024-04-20T21:10:16.065 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/wacom.ko 2024-04-20T21:10:16.065 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv 2024-04-20T21:10:16.065 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv/hv_balloon.ko 2024-04-20T21:10:16.065 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv/hv_utils.ko 2024-04-20T21:10:16.066 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv/hv_vmbus.ko 2024-04-20T21:10:16.066 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon 2024-04-20T21:10:16.066 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/abituguru.ko 2024-04-20T21:10:16.066 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/abituguru3.ko 2024-04-20T21:10:16.066 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/acpi_power_meter.ko 2024-04-20T21:10:16.066 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ad7414.ko 2024-04-20T21:10:16.066 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ad7418.ko 2024-04-20T21:10:16.066 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adc128d818.ko 2024-04-20T21:10:16.066 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1025.ko 2024-04-20T21:10:16.066 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1026.ko 2024-04-20T21:10:16.066 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1029.ko 2024-04-20T21:10:16.066 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1031.ko 2024-04-20T21:10:16.066 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm9240.ko 2024-04-20T21:10:16.066 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ads7828.ko 2024-04-20T21:10:16.067 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7410.ko 2024-04-20T21:10:16.067 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7411.ko 2024-04-20T21:10:16.067 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7462.ko 2024-04-20T21:10:16.067 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7470.ko 2024-04-20T21:10:16.067 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7475.ko 2024-04-20T21:10:16.067 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7x10.ko 2024-04-20T21:10:16.067 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/amc6821.ko 2024-04-20T21:10:16.067 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/applesmc.ko 2024-04-20T21:10:16.067 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/asb100.ko 2024-04-20T21:10:16.067 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/asc7621.ko 2024-04-20T21:10:16.067 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/asus_atk0110.ko 2024-04-20T21:10:16.067 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/atxp1.ko 2024-04-20T21:10:16.067 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/coretemp.ko 2024-04-20T21:10:16.067 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-04-20T21:10:16.068 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/dme1737.ko 2024-04-20T21:10:16.068 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ds1621.ko 2024-04-20T21:10:16.068 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ds620.ko 2024-04-20T21:10:16.068 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/emc1403.ko 2024-04-20T21:10:16.068 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/emc6w201.ko 2024-04-20T21:10:16.068 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/f71805f.ko 2024-04-20T21:10:16.068 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/f71882fg.ko 2024-04-20T21:10:16.068 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/f75375s.ko 2024-04-20T21:10:16.068 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/fam15h_power.ko 2024-04-20T21:10:16.068 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/fschmd.ko 2024-04-20T21:10:16.068 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/g760a.ko 2024-04-20T21:10:16.068 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/g762.ko 2024-04-20T21:10:16.068 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/gl518sm.ko 2024-04-20T21:10:16.068 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/gl520sm.ko 2024-04-20T21:10:16.068 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/hwmon-vid.ko 2024-04-20T21:10:16.069 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/i5500_temp.ko 2024-04-20T21:10:16.069 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/i5k_amb.ko 2024-04-20T21:10:16.069 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ibmaem.ko 2024-04-20T21:10:16.069 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ibmpex.ko 2024-04-20T21:10:16.069 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ina209.ko 2024-04-20T21:10:16.069 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ina2xx.ko 2024-04-20T21:10:16.069 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/it87.ko 2024-04-20T21:10:16.069 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/k10temp.ko 2024-04-20T21:10:16.069 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/k8temp.ko 2024-04-20T21:10:16.069 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lineage-pem.ko 2024-04-20T21:10:16.069 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm63.ko 2024-04-20T21:10:16.069 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm73.ko 2024-04-20T21:10:16.069 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm75.ko 2024-04-20T21:10:16.069 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm77.ko 2024-04-20T21:10:16.069 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm78.ko 2024-04-20T21:10:16.070 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm80.ko 2024-04-20T21:10:16.070 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm83.ko 2024-04-20T21:10:16.070 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm85.ko 2024-04-20T21:10:16.070 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm87.ko 2024-04-20T21:10:16.070 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm90.ko 2024-04-20T21:10:16.070 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm92.ko 2024-04-20T21:10:16.070 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm93.ko 2024-04-20T21:10:16.070 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm95234.ko 2024-04-20T21:10:16.070 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm95241.ko 2024-04-20T21:10:16.070 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm95245.ko 2024-04-20T21:10:16.070 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc2945.ko 2024-04-20T21:10:16.070 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4151.ko 2024-04-20T21:10:16.070 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4215.ko 2024-04-20T21:10:16.070 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4222.ko 2024-04-20T21:10:16.070 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4245.ko 2024-04-20T21:10:16.071 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4260.ko 2024-04-20T21:10:16.071 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4261.ko 2024-04-20T21:10:16.071 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max16065.ko 2024-04-20T21:10:16.071 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max1619.ko 2024-04-20T21:10:16.071 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max1668.ko 2024-04-20T21:10:16.071 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max197.ko 2024-04-20T21:10:16.071 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max6639.ko 2024-04-20T21:10:16.071 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max6650.ko 2024-04-20T21:10:16.071 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max6697.ko 2024-04-20T21:10:16.071 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/mcp3021.ko 2024-04-20T21:10:16.071 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct6683.ko 2024-04-20T21:10:16.071 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct6775-core.ko 2024-04-20T21:10:16.071 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct6775.ko 2024-04-20T21:10:16.071 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct7802.ko 2024-04-20T21:10:16.071 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pc87360.ko 2024-04-20T21:10:16.072 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pc87427.ko 2024-04-20T21:10:16.072 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pcf8591.ko 2024-04-20T21:10:16.072 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus 2024-04-20T21:10:16.072 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-04-20T21:10:16.072 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-04-20T21:10:16.072 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-04-20T21:10:16.072 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/max16064.ko 2024-04-20T21:10:16.072 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/max34440.ko 2024-04-20T21:10:16.072 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/max8688.ko 2024-04-20T21:10:16.072 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-04-20T21:10:16.072 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-04-20T21:10:16.072 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-04-20T21:10:16.072 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-04-20T21:10:16.072 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-04-20T21:10:16.072 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-04-20T21:10:16.073 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/powr1220.ko 2024-04-20T21:10:16.073 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sch5627.ko 2024-04-20T21:10:16.073 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sch5636.ko 2024-04-20T21:10:16.073 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sch56xx-common.ko 2024-04-20T21:10:16.073 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sht15.ko 2024-04-20T21:10:16.073 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sht21.ko 2024-04-20T21:10:16.073 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/shtc1.ko 2024-04-20T21:10:16.073 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sis5595.ko 2024-04-20T21:10:16.073 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/smsc47b397.ko 2024-04-20T21:10:16.073 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/smsc47m1.ko 2024-04-20T21:10:16.073 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/smsc47m192.ko 2024-04-20T21:10:16.073 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/thmc50.ko 2024-04-20T21:10:16.073 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp102.ko 2024-04-20T21:10:16.073 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp103.ko 2024-04-20T21:10:16.073 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp401.ko 2024-04-20T21:10:16.074 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp421.ko 2024-04-20T21:10:16.074 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/via-cputemp.ko 2024-04-20T21:10:16.074 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/via686a.ko 2024-04-20T21:10:16.074 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/vt1211.ko 2024-04-20T21:10:16.074 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/vt8231.ko 2024-04-20T21:10:16.074 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83627ehf.ko 2024-04-20T21:10:16.074 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83627hf.ko 2024-04-20T21:10:16.074 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83781d.ko 2024-04-20T21:10:16.074 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83791d.ko 2024-04-20T21:10:16.074 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83792d.ko 2024-04-20T21:10:16.074 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83793.ko 2024-04-20T21:10:16.075 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83795.ko 2024-04-20T21:10:16.075 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83l785ts.ko 2024-04-20T21:10:16.075 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83l786ng.ko 2024-04-20T21:10:16.075 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c 2024-04-20T21:10:16.075 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/algos 2024-04-20T21:10:16.075 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-04-20T21:10:16.075 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-04-20T21:10:16.075 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses 2024-04-20T21:10:16.075 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-04-20T21:10:16.075 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-04-20T21:10:16.075 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-04-20T21:10:16.075 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-04-20T21:10:16.076 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-04-20T21:10:16.076 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-04-20T21:10:16.076 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-04-20T21:10:16.076 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-04-20T21:10:16.076 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-i801.ko 2024-04-20T21:10:16.076 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-isch.ko 2024-04-20T21:10:16.076 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-04-20T21:10:16.076 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-04-20T21:10:16.076 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-04-20T21:10:16.076 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-parport.ko 2024-04-20T21:10:16.076 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-04-20T21:10:16.076 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-04-20T21:10:16.076 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-04-20T21:10:16.076 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-04-20T21:10:16.077 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-04-20T21:10:16.077 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-04-20T21:10:16.077 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-via.ko 2024-04-20T21:10:16.077 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-04-20T21:10:16.077 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-04-20T21:10:16.077 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-dev.ko 2024-04-20T21:10:16.077 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-mux.ko 2024-04-20T21:10:16.077 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-04-20T21:10:16.077 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-smbus.ko 2024-04-20T21:10:16.077 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-stub.ko 2024-04-20T21:10:16.077 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband 2024-04-20T21:10:16.077 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core 2024-04-20T21:10:16.077 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_cm.ko 2024-04-20T21:10:16.077 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_core.ko 2024-04-20T21:10:16.077 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_umad.ko 2024-04-20T21:10:16.078 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-04-20T21:10:16.078 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/iw_cm.ko 2024-04-20T21:10:16.078 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/rdma_cm.ko 2024-04-20T21:10:16.078 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-04-20T21:10:16.078 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw 2024-04-20T21:10:16.078 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/cxgb4 2024-04-20T21:10:16.078 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-04-20T21:10:16.078 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx4 2024-04-20T21:10:16.078 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-04-20T21:10:16.078 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx5 2024-04-20T21:10:16.078 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-04-20T21:10:16.078 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mthca 2024-04-20T21:10:16.078 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-04-20T21:10:16.078 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp 2024-04-20T21:10:16.079 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/ipoib 2024-04-20T21:10:16.079 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-04-20T21:10:16.079 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/iser 2024-04-20T21:10:16.079 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-04-20T21:10:16.079 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/isert 2024-04-20T21:10:16.079 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-04-20T21:10:16.079 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srp 2024-04-20T21:10:16.079 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-04-20T21:10:16.079 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srpt 2024-04-20T21:10:16.079 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-04-20T21:10:16.079 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input 2024-04-20T21:10:16.079 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport 2024-04-20T21:10:16.079 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-04-20T21:10:16.079 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport/fm801-gp.ko 2024-04-20T21:10:16.079 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport/gameport.ko 2024-04-20T21:10:16.080 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/joydev.ko 2024-04-20T21:10:16.080 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/keyboard 2024-04-20T21:10:16.080 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/keyboard/gpio_keys.ko 2024-04-20T21:10:16.080 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse 2024-04-20T21:10:16.080 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/appletouch.ko 2024-04-20T21:10:16.080 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/bcm5974.ko 2024-04-20T21:10:16.080 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/cyapatp.ko 2024-04-20T21:10:16.080 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/elan_i2c.ko 2024-04-20T21:10:16.080 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/sermouse.ko 2024-04-20T21:10:16.080 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-04-20T21:10:16.080 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/synaptics_usb.ko 2024-04-20T21:10:16.080 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/vsxxxaa.ko 2024-04-20T21:10:16.080 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/rmi4 2024-04-20T21:10:16.080 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/rmi4/rmi_core.ko 2024-04-20T21:10:16.080 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio 2024-04-20T21:10:16.080 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/altera_ps2.ko 2024-04-20T21:10:16.081 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/arc_ps2.ko 2024-04-20T21:10:16.081 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-04-20T21:10:16.081 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/serio_raw.ko 2024-04-20T21:10:16.081 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/sparse-keymap.ko 2024-04-20T21:10:16.081 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds 2024-04-20T21:10:16.081 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-blinkm.ko 2024-04-20T21:10:16.081 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-lm3530.ko 2024-04-20T21:10:16.081 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-lp3944.ko 2024-04-20T21:10:16.081 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-ss4200.ko 2024-04-20T21:10:16.081 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger 2024-04-20T21:10:16.081 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-04-20T21:10:16.081 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-04-20T21:10:16.081 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-04-20T21:10:16.081 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-04-20T21:10:16.082 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-04-20T21:10:16.082 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-04-20T21:10:16.082 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-04-20T21:10:16.082 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-04-20T21:10:16.082 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-04-20T21:10:16.082 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md 2024-04-20T21:10:16.082 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/bcache 2024-04-20T21:10:16.082 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/bcache/bcache.ko 2024-04-20T21:10:16.082 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-bio-prison.ko 2024-04-20T21:10:16.082 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-cache-smq.ko 2024-04-20T21:10:16.082 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-cache.ko 2024-04-20T21:10:16.082 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-crypt.ko 2024-04-20T21:10:16.082 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-delay.ko 2024-04-20T21:10:16.082 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-flakey.ko 2024-04-20T21:10:16.082 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-log-userspace.ko 2024-04-20T21:10:16.083 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-multipath.ko 2024-04-20T21:10:16.083 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-queue-length.ko 2024-04-20T21:10:16.083 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-raid.ko 2024-04-20T21:10:16.083 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-round-robin.ko 2024-04-20T21:10:16.083 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-service-time.ko 2024-04-20T21:10:16.083 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-switch.ko 2024-04-20T21:10:16.083 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-thin-pool.ko 2024-04-20T21:10:16.083 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-verity.ko 2024-04-20T21:10:16.083 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/persistent-data 2024-04-20T21:10:16.083 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-04-20T21:10:16.083 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid0.ko 2024-04-20T21:10:16.083 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid1.ko 2024-04-20T21:10:16.084 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid10.ko 2024-04-20T21:10:16.084 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid456.ko 2024-04-20T21:10:16.084 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message 2024-04-20T21:10:16.084 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion 2024-04-20T21:10:16.084 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptbase.ko 2024-04-20T21:10:16.084 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptctl.ko 2024-04-20T21:10:16.084 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptfc.ko 2024-04-20T21:10:16.084 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptlan.ko 2024-04-20T21:10:16.084 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptsas.ko 2024-04-20T21:10:16.084 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptscsih.ko 2024-04-20T21:10:16.084 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptspi.ko 2024-04-20T21:10:16.084 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd 2024-04-20T21:10:16.084 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/lpc_ich.ko 2024-04-20T21:10:16.084 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/lpc_sch.ko 2024-04-20T21:10:16.085 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/mfd-core.ko 2024-04-20T21:10:16.085 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/sm501.ko 2024-04-20T21:10:16.085 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/viperboard.ko 2024-04-20T21:10:16.085 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/vx855.ko 2024-04-20T21:10:16.085 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/wl1273-core.ko 2024-04-20T21:10:16.085 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc 2024-04-20T21:10:16.085 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/altera-stapl 2024-04-20T21:10:16.085 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-04-20T21:10:16.085 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/apds9802als.ko 2024-04-20T21:10:16.085 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/apds990x.ko 2024-04-20T21:10:16.085 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/bh1770glc.ko 2024-04-20T21:10:16.085 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/cb710 2024-04-20T21:10:16.085 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/cb710/cb710.ko 2024-04-20T21:10:16.085 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom 2024-04-20T21:10:16.085 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom/at24.ko 2024-04-20T21:10:16.086 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-04-20T21:10:16.086 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom/max6875.ko 2024-04-20T21:10:16.086 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/enclosure.ko 2024-04-20T21:10:16.086 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/hpilo.ko 2024-04-20T21:10:16.086 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/isl29003.ko 2024-04-20T21:10:16.086 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/isl29020.ko 2024-04-20T21:10:16.086 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/lis3lv02d 2024-04-20T21:10:16.086 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-04-20T21:10:16.086 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-04-20T21:10:16.086 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei 2024-04-20T21:10:16.086 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei/mei-me.ko 2024-04-20T21:10:16.086 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei/mei-txe.ko 2024-04-20T21:10:16.086 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei/mei.ko 2024-04-20T21:10:16.086 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-gru 2024-04-20T21:10:16.087 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-gru/gru.ko 2024-04-20T21:10:16.087 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp 2024-04-20T21:10:16.087 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp/xp.ko 2024-04-20T21:10:16.087 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp/xpc.ko 2024-04-20T21:10:16.087 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-04-20T21:10:16.087 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/tifm_7xx1.ko 2024-04-20T21:10:16.087 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/tifm_core.ko 2024-04-20T21:10:16.087 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/tsl2550.ko 2024-04-20T21:10:16.087 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/vmw_balloon.ko 2024-04-20T21:10:16.087 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/vmw_vmci 2024-04-20T21:10:16.087 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-04-20T21:10:16.087 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net 2024-04-20T21:10:16.087 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/bonding 2024-04-20T21:10:16.087 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/bonding/bonding.ko 2024-04-20T21:10:16.087 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa 2024-04-20T21:10:16.088 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/b53 2024-04-20T21:10:16.088 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/b53/b53_common.ko 2024-04-20T21:10:16.088 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/bcm-sf2.ko 2024-04-20T21:10:16.088 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/mv88e6060.ko 2024-04-20T21:10:16.088 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/mv88e6xxx 2024-04-20T21:10:16.088 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-04-20T21:10:16.088 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dummy.ko 2024-04-20T21:10:16.088 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/eql.ko 2024-04-20T21:10:16.088 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet 2024-04-20T21:10:16.088 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com 2024-04-20T21:10:16.088 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-04-20T21:10:16.088 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-04-20T21:10:16.088 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-04-20T21:10:16.088 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-04-20T21:10:16.088 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390 2024-04-20T21:10:16.089 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/8390.ko 2024-04-20T21:10:16.089 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-04-20T21:10:16.089 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-04-20T21:10:16.089 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-04-20T21:10:16.089 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/adaptec 2024-04-20T21:10:16.089 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-04-20T21:10:16.089 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/agere 2024-04-20T21:10:16.089 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/agere/et131x.ko 2024-04-20T21:10:16.089 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/alteon 2024-04-20T21:10:16.089 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-04-20T21:10:16.089 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/altera 2024-04-20T21:10:16.089 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-04-20T21:10:16.089 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd 2024-04-20T21:10:16.089 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-04-20T21:10:16.089 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-04-20T21:10:16.090 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-04-20T21:10:16.090 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros 2024-04-20T21:10:16.090 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/alx 2024-04-20T21:10:16.090 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-04-20T21:10:16.090 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1c 2024-04-20T21:10:16.090 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-04-20T21:10:16.090 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1e 2024-04-20T21:10:16.090 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-04-20T21:10:16.090 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atlx 2024-04-20T21:10:16.090 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-04-20T21:10:16.090 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-04-20T21:10:16.090 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom 2024-04-20T21:10:16.090 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-04-20T21:10:16.090 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-04-20T21:10:16.090 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-04-20T21:10:16.091 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-04-20T21:10:16.091 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-04-20T21:10:16.091 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/genet 2024-04-20T21:10:16.091 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-04-20T21:10:16.091 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-04-20T21:10:16.091 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/brocade 2024-04-20T21:10:16.091 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/brocade/bna 2024-04-20T21:10:16.091 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-04-20T21:10:16.091 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio 2024-04-20T21:10:16.091 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb 2024-04-20T21:10:16.091 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-04-20T21:10:16.091 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-04-20T21:10:16.091 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-04-20T21:10:16.091 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-04-20T21:10:16.092 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-04-20T21:10:16.092 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-04-20T21:10:16.092 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-04-20T21:10:16.092 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-04-20T21:10:16.092 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-04-20T21:10:16.092 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/cisco 2024-04-20T21:10:16.092 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/cisco/enic 2024-04-20T21:10:16.092 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-04-20T21:10:16.092 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec 2024-04-20T21:10:16.092 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip 2024-04-20T21:10:16.092 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-04-20T21:10:16.092 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-04-20T21:10:16.092 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-04-20T21:10:16.092 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-04-20T21:10:16.093 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-04-20T21:10:16.093 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-04-20T21:10:16.093 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dlink 2024-04-20T21:10:16.093 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-04-20T21:10:16.093 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-04-20T21:10:16.093 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dnet.ko 2024-04-20T21:10:16.093 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/emulex 2024-04-20T21:10:16.093 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/emulex/benet 2024-04-20T21:10:16.093 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-04-20T21:10:16.093 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/ethoc.ko 2024-04-20T21:10:16.093 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/fealnx.ko 2024-04-20T21:10:16.093 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel 2024-04-20T21:10:16.093 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e100.ko 2024-04-20T21:10:16.093 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000 2024-04-20T21:10:16.093 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-04-20T21:10:16.093 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000e 2024-04-20T21:10:16.094 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-04-20T21:10:16.094 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/fm10k 2024-04-20T21:10:16.094 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-04-20T21:10:16.094 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/i40e 2024-04-20T21:10:16.094 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-04-20T21:10:16.094 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/iavf 2024-04-20T21:10:16.094 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-04-20T21:10:16.094 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igb 2024-04-20T21:10:16.094 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-04-20T21:10:16.094 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igbvf 2024-04-20T21:10:16.094 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-04-20T21:10:16.094 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbe 2024-04-20T21:10:16.094 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-04-20T21:10:16.094 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbevf 2024-04-20T21:10:16.095 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-04-20T21:10:16.095 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/jme.ko 2024-04-20T21:10:16.095 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell 2024-04-20T21:10:16.095 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-04-20T21:10:16.095 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell/skge.ko 2024-04-20T21:10:16.095 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-04-20T21:10:16.095 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox 2024-04-20T21:10:16.095 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx4 2024-04-20T21:10:16.095 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-04-20T21:10:16.095 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-04-20T21:10:16.095 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx5 2024-04-20T21:10:16.095 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-04-20T21:10:16.095 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-04-20T21:10:16.095 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/micrel 2024-04-20T21:10:16.096 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-04-20T21:10:16.096 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/myricom 2024-04-20T21:10:16.096 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/myricom/myri10ge 2024-04-20T21:10:16.096 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-04-20T21:10:16.096 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/natsemi 2024-04-20T21:10:16.096 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-04-20T21:10:16.096 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-04-20T21:10:16.096 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/neterion 2024-04-20T21:10:16.096 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-04-20T21:10:16.096 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/nvidia 2024-04-20T21:10:16.096 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-04-20T21:10:16.096 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/packetengines 2024-04-20T21:10:16.096 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-04-20T21:10:16.096 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-04-20T21:10:16.096 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic 2024-04-20T21:10:16.096 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/netxen 2024-04-20T21:10:16.097 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-04-20T21:10:16.097 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-04-20T21:10:16.097 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-04-20T21:10:16.097 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-04-20T21:10:16.097 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rdc 2024-04-20T21:10:16.097 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-04-20T21:10:16.097 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek 2024-04-20T21:10:16.097 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-04-20T21:10:16.097 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-04-20T21:10:16.097 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/atp.ko 2024-04-20T21:10:16.097 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-04-20T21:10:16.097 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rocker 2024-04-20T21:10:16.097 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-04-20T21:10:16.097 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sfc 2024-04-20T21:10:16.098 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-04-20T21:10:16.098 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/silan 2024-04-20T21:10:16.098 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-04-20T21:10:16.098 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sis 2024-04-20T21:10:16.098 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sis/sis190.ko 2024-04-20T21:10:16.098 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sis/sis900.ko 2024-04-20T21:10:16.098 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc 2024-04-20T21:10:16.098 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-04-20T21:10:16.098 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-04-20T21:10:16.098 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-04-20T21:10:16.098 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-04-20T21:10:16.098 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro 2024-04-20T21:10:16.098 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro/stmmac 2024-04-20T21:10:16.098 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-04-20T21:10:16.098 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-04-20T21:10:16.099 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun 2024-04-20T21:10:16.099 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/cassini.ko 2024-04-20T21:10:16.099 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/niu.ko 2024-04-20T21:10:16.099 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/sungem.ko 2024-04-20T21:10:16.099 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-04-20T21:10:16.099 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/tehuti 2024-04-20T21:10:16.099 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-04-20T21:10:16.099 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/ti 2024-04-20T21:10:16.099 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/ti/tlan.ko 2024-04-20T21:10:16.099 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/via 2024-04-20T21:10:16.099 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-04-20T21:10:16.099 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-04-20T21:10:16.099 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/wiznet 2024-04-20T21:10:16.099 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-04-20T21:10:16.099 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-04-20T21:10:16.100 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/xircom 2024-04-20T21:10:16.100 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-04-20T21:10:16.100 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/geneve.ko 2024-04-20T21:10:16.100 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/hyperv 2024-04-20T21:10:16.100 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-04-20T21:10:16.100 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ieee802154 2024-04-20T21:10:16.100 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ieee802154/fakelb.ko 2024-04-20T21:10:16.100 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ifb.ko 2024-04-20T21:10:16.100 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ipvlan 2024-04-20T21:10:16.100 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ipvlan/ipvlan.ko 2024-04-20T21:10:16.100 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/macvlan.ko 2024-04-20T21:10:16.100 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/macvtap.ko 2024-04-20T21:10:16.100 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio 2024-04-20T21:10:16.100 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio.ko 2024-04-20T21:10:16.100 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-04-20T21:10:16.101 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-04-20T21:10:16.101 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio/mdio-regmap.ko 2024-04-20T21:10:16.101 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mii.ko 2024-04-20T21:10:16.101 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/net_failover.ko 2024-04-20T21:10:16.101 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/netconsole.ko 2024-04-20T21:10:16.101 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/nlmon.ko 2024-04-20T21:10:16.101 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/pcs 2024-04-20T21:10:16.101 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/pcs/pcs-lynx.ko 2024-04-20T21:10:16.101 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-04-20T21:10:16.101 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy 2024-04-20T21:10:16.101 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/amd.ko 2024-04-20T21:10:16.101 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-04-20T21:10:16.101 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-04-20T21:10:16.101 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm7xxx.ko 2024-04-20T21:10:16.101 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm87xx.ko 2024-04-20T21:10:16.101 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/broadcom.ko 2024-04-20T21:10:16.102 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/cicada.ko 2024-04-20T21:10:16.102 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/davicom.ko 2024-04-20T21:10:16.102 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/dp83640.ko 2024-04-20T21:10:16.102 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/et1011c.ko 2024-04-20T21:10:16.102 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/icplus.ko 2024-04-20T21:10:16.102 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/lxt.ko 2024-04-20T21:10:16.102 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/marvell.ko 2024-04-20T21:10:16.102 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/micrel.ko 2024-04-20T21:10:16.102 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/national.ko 2024-04-20T21:10:16.102 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/phylink.ko 2024-04-20T21:10:16.102 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/qsemi.ko 2024-04-20T21:10:16.102 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/realtek.ko 2024-04-20T21:10:16.102 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/smsc.ko 2024-04-20T21:10:16.102 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/ste10Xp.ko 2024-04-20T21:10:16.103 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/vitesse.ko 2024-04-20T21:10:16.103 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp 2024-04-20T21:10:16.103 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/bsd_comp.ko 2024-04-20T21:10:16.103 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_async.ko 2024-04-20T21:10:16.103 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_deflate.ko 2024-04-20T21:10:16.103 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_generic.ko 2024-04-20T21:10:16.103 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_mppe.ko 2024-04-20T21:10:16.103 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_synctty.ko 2024-04-20T21:10:16.103 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/pppoe.ko 2024-04-20T21:10:16.103 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/pppox.ko 2024-04-20T21:10:16.103 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/pptp.ko 2024-04-20T21:10:16.103 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/slip 2024-04-20T21:10:16.103 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/slip/slhc.ko 2024-04-20T21:10:16.103 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/slip/slip.ko 2024-04-20T21:10:16.103 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/sungem_phy.ko 2024-04-20T21:10:16.103 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/tap.ko 2024-04-20T21:10:16.104 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team 2024-04-20T21:10:16.104 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team.ko 2024-04-20T21:10:16.104 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_activebackup.ko 2024-04-20T21:10:16.104 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_broadcast.ko 2024-04-20T21:10:16.104 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-04-20T21:10:16.104 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_random.ko 2024-04-20T21:10:16.104 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-04-20T21:10:16.104 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/tun.ko 2024-04-20T21:10:16.104 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/veth.ko 2024-04-20T21:10:16.104 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/virtio_net.ko 2024-04-20T21:10:16.104 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vmxnet3 2024-04-20T21:10:16.104 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-04-20T21:10:16.104 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vxlan 2024-04-20T21:10:16.104 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vxlan/vxlan.ko 2024-04-20T21:10:16.105 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/xen-netback 2024-04-20T21:10:16.105 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/xen-netback/xen-netback.ko 2024-04-20T21:10:16.105 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/xen-netfront.ko 2024-04-20T21:10:16.105 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ntb 2024-04-20T21:10:16.105 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ntb/ntb.ko 2024-04-20T21:10:16.105 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme 2024-04-20T21:10:16.105 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme/host 2024-04-20T21:10:16.105 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme/host/nvme-core.ko 2024-04-20T21:10:16.105 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme/host/nvme.ko 2024-04-20T21:10:16.105 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport 2024-04-20T21:10:16.105 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport.ko 2024-04-20T21:10:16.105 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport_cs.ko 2024-04-20T21:10:16.106 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport_pc.ko 2024-04-20T21:10:16.106 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport_serial.ko 2024-04-20T21:10:16.106 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci 2024-04-20T21:10:16.106 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/hotplug 2024-04-20T21:10:16.106 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-04-20T21:10:16.106 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/pcie 2024-04-20T21:10:16.106 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/pcie/aer_inject.ko 2024-04-20T21:10:16.106 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/xen-pcifront.ko 2024-04-20T21:10:16.106 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia 2024-04-20T21:10:16.106 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia/i82092.ko 2024-04-20T21:10:16.106 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia/pd6729.ko 2024-04-20T21:10:16.106 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia/yenta_socket.ko 2024-04-20T21:10:16.106 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pinctrl 2024-04-20T21:10:16.107 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pinctrl/intel 2024-04-20T21:10:16.107 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-04-20T21:10:16.107 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform 2024-04-20T21:10:16.107 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86 2024-04-20T21:10:16.107 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/acer-wmi.ko 2024-04-20T21:10:16.107 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/acerhdf.ko 2024-04-20T21:10:16.107 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/apple-gmux.ko 2024-04-20T21:10:16.107 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/asus-laptop.ko 2024-04-20T21:10:16.107 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-04-20T21:10:16.107 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/asus-wmi.ko 2024-04-20T21:10:16.107 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/classmate-laptop.ko 2024-04-20T21:10:16.107 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-04-20T21:10:16.107 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-04-20T21:10:16.107 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-04-20T21:10:16.107 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-04-20T21:10:16.108 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/hdaps.ko 2024-04-20T21:10:16.108 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/intel 2024-04-20T21:10:16.108 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-04-20T21:10:16.108 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/intel_ips.ko 2024-04-20T21:10:16.108 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/msi-wmi.ko 2024-04-20T21:10:16.108 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/mxm-wmi.ko 2024-04-20T21:10:16.108 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-04-20T21:10:16.108 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/samsung-laptop.ko 2024-04-20T21:10:16.108 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/samsung-q10.ko 2024-04-20T21:10:16.108 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/topstar-laptop.ko 2024-04-20T21:10:16.108 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-04-20T21:10:16.108 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/toshiba_haps.ko 2024-04-20T21:10:16.108 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/wmi-bmof.ko 2024-04-20T21:10:16.108 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/wmi.ko 2024-04-20T21:10:16.108 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/powercap 2024-04-20T21:10:16.109 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/powercap/intel_rapl_common.ko 2024-04-20T21:10:16.109 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/powercap/intel_rapl_msr.ko 2024-04-20T21:10:16.109 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps 2024-04-20T21:10:16.109 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients 2024-04-20T21:10:16.109 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients/pps-gpio.ko 2024-04-20T21:10:16.109 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients/pps-ldisc.ko 2024-04-20T21:10:16.109 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients/pps_parport.ko 2024-04-20T21:10:16.109 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/pps_core.ko 2024-04-20T21:10:16.109 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ptp 2024-04-20T21:10:16.109 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ptp/ptp.ko 2024-04-20T21:10:16.109 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ptp/ptp_kvm.ko 2024-04-20T21:10:16.109 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras 2024-04-20T21:10:16.109 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd 2024-04-20T21:10:16.109 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd/atl 2024-04-20T21:10:16.109 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd/atl/amd_atl.ko 2024-04-20T21:10:16.109 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd/fmpm.ko 2024-04-20T21:10:16.110 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc 2024-04-20T21:10:16.110 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-bq32k.ko 2024-04-20T21:10:16.110 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1286.ko 2024-04-20T21:10:16.110 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1307.ko 2024-04-20T21:10:16.110 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1374.ko 2024-04-20T21:10:16.110 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1511.ko 2024-04-20T21:10:16.110 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1553.ko 2024-04-20T21:10:16.110 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1672.ko 2024-04-20T21:10:16.110 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1742.ko 2024-04-20T21:10:16.110 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds2404.ko 2024-04-20T21:10:16.110 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds3232.ko 2024-04-20T21:10:16.110 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-em3027.ko 2024-04-20T21:10:16.110 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-fm3130.ko 2024-04-20T21:10:16.110 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-isl12022.ko 2024-04-20T21:10:16.110 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-isl1208.ko 2024-04-20T21:10:16.111 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-m41t80.ko 2024-04-20T21:10:16.111 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-m48t35.ko 2024-04-20T21:10:16.111 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-m48t59.ko 2024-04-20T21:10:16.111 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-max6900.ko 2024-04-20T21:10:16.111 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-msm6242.ko 2024-04-20T21:10:16.111 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf2127.ko 2024-04-20T21:10:16.111 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf85063.ko 2024-04-20T21:10:16.111 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf8523.ko 2024-04-20T21:10:16.111 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf8563.ko 2024-04-20T21:10:16.111 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf8583.ko 2024-04-20T21:10:16.111 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rp5c01.ko 2024-04-20T21:10:16.111 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rs5c372.ko 2024-04-20T21:10:16.111 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-04-20T21:10:16.111 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rx8025.ko 2024-04-20T21:10:16.112 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rx8581.ko 2024-04-20T21:10:16.112 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-04-20T21:10:16.112 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-x1205.ko 2024-04-20T21:10:16.112 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi 2024-04-20T21:10:16.112 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/3w-9xxx.ko 2024-04-20T21:10:16.112 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/3w-sas.ko 2024-04-20T21:10:16.112 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/3w-xxxx.ko 2024-04-20T21:10:16.112 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/BusLogic.ko 2024-04-20T21:10:16.112 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/a100u2w.ko 2024-04-20T21:10:16.112 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aacraid 2024-04-20T21:10:16.112 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aacraid/aacraid.ko 2024-04-20T21:10:16.112 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/advansys.ko 2024-04-20T21:10:16.112 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aic7xxx 2024-04-20T21:10:16.112 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-04-20T21:10:16.112 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-04-20T21:10:16.113 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/am53c974.ko 2024-04-20T21:10:16.113 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/arcmsr 2024-04-20T21:10:16.113 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-04-20T21:10:16.113 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/atp870u.ko 2024-04-20T21:10:16.113 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/be2iscsi 2024-04-20T21:10:16.113 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-04-20T21:10:16.113 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bfa 2024-04-20T21:10:16.113 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bfa/bfa.ko 2024-04-20T21:10:16.113 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2fc 2024-04-20T21:10:16.113 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-04-20T21:10:16.113 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2i 2024-04-20T21:10:16.113 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-04-20T21:10:16.113 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ch.ko 2024-04-20T21:10:16.113 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/csiostor 2024-04-20T21:10:16.113 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/csiostor/csiostor.ko 2024-04-20T21:10:16.114 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi 2024-04-20T21:10:16.114 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb3i 2024-04-20T21:10:16.114 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-04-20T21:10:16.114 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb4i 2024-04-20T21:10:16.114 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-04-20T21:10:16.114 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-04-20T21:10:16.114 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/dc395x.ko 2024-04-20T21:10:16.114 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler 2024-04-20T21:10:16.114 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-04-20T21:10:16.114 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-04-20T21:10:16.114 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-04-20T21:10:16.114 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-04-20T21:10:16.114 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/dmx3191d.ko 2024-04-20T21:10:16.114 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/esas2r 2024-04-20T21:10:16.114 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/esas2r/esas2r.ko 2024-04-20T21:10:16.115 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/esp_scsi.ko 2024-04-20T21:10:16.115 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fcoe 2024-04-20T21:10:16.115 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fcoe/fcoe.ko 2024-04-20T21:10:16.115 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-04-20T21:10:16.115 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fnic 2024-04-20T21:10:16.115 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fnic/fnic.ko 2024-04-20T21:10:16.115 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/hpsa.ko 2024-04-20T21:10:16.115 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/hptiop.ko 2024-04-20T21:10:16.115 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/hv_storvsc.ko 2024-04-20T21:10:16.115 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/initio.ko 2024-04-20T21:10:16.115 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ipr.ko 2024-04-20T21:10:16.115 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ips.ko 2024-04-20T21:10:16.115 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/isci 2024-04-20T21:10:16.115 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/isci/isci.ko 2024-04-20T21:10:16.115 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-04-20T21:10:16.116 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/iscsi_tcp.ko 2024-04-20T21:10:16.116 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libfc 2024-04-20T21:10:16.116 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libfc/libfc.ko 2024-04-20T21:10:16.116 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libiscsi.ko 2024-04-20T21:10:16.116 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libiscsi_tcp.ko 2024-04-20T21:10:16.116 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libsas 2024-04-20T21:10:16.116 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libsas/libsas.ko 2024-04-20T21:10:16.116 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/lpfc 2024-04-20T21:10:16.116 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/lpfc/lpfc.ko 2024-04-20T21:10:16.116 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid 2024-04-20T21:10:16.116 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid.ko 2024-04-20T21:10:16.116 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-04-20T21:10:16.116 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-04-20T21:10:16.116 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-04-20T21:10:16.116 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mpt3sas 2024-04-20T21:10:16.117 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-04-20T21:10:16.117 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mvsas 2024-04-20T21:10:16.117 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mvsas/mvsas.ko 2024-04-20T21:10:16.117 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mvumi.ko 2024-04-20T21:10:16.117 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/pm8001 2024-04-20T21:10:16.117 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-04-20T21:10:16.117 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/pmcraid.ko 2024-04-20T21:10:16.117 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla1280.ko 2024-04-20T21:10:16.117 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla2xxx 2024-04-20T21:10:16.117 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-04-20T21:10:16.117 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-04-20T21:10:16.117 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla4xxx 2024-04-20T21:10:16.117 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-04-20T21:10:16.117 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/raid_class.ko 2024-04-20T21:10:16.117 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_debug.ko 2024-04-20T21:10:16.118 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_fc.ko 2024-04-20T21:10:16.118 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-04-20T21:10:16.118 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_sas.ko 2024-04-20T21:10:16.118 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_spi.ko 2024-04-20T21:10:16.118 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_srp.ko 2024-04-20T21:10:16.118 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ses.ko 2024-04-20T21:10:16.118 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/st.ko 2024-04-20T21:10:16.118 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/stex.ko 2024-04-20T21:10:16.118 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/sym53c8xx_2 2024-04-20T21:10:16.118 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-04-20T21:10:16.118 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/virtio_scsi.ko 2024-04-20T21:10:16.118 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/vmw_pvscsi.ko 2024-04-20T21:10:16.118 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/wd719x.ko 2024-04-20T21:10:16.118 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ssb 2024-04-20T21:10:16.118 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ssb/ssb.ko 2024-04-20T21:10:16.119 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target 2024-04-20T21:10:16.119 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/iscsi 2024-04-20T21:10:16.119 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-04-20T21:10:16.119 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/loopback 2024-04-20T21:10:16.119 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/loopback/tcm_loop.ko 2024-04-20T21:10:16.119 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_file.ko 2024-04-20T21:10:16.119 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_iblock.ko 2024-04-20T21:10:16.119 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_mod.ko 2024-04-20T21:10:16.119 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_pscsi.ko 2024-04-20T21:10:16.119 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_user.ko 2024-04-20T21:10:16.119 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/tcm_fc 2024-04-20T21:10:16.119 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-04-20T21:10:16.119 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal 2024-04-20T21:10:16.119 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel 2024-04-20T21:10:16.119 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal 2024-04-20T21:10:16.120 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-04-20T21:10:16.120 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-04-20T21:10:16.120 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-04-20T21:10:16.120 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-04-20T21:10:16.120 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-04-20T21:10:16.120 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-04-20T21:10:16.120 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-04-20T21:10:16.120 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-04-20T21:10:16.120 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-04-20T21:10:16.120 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-04-20T21:10:16.120 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-04-20T21:10:16.120 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-04-20T21:10:16.120 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-04-20T21:10:16.120 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-04-20T21:10:16.121 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-04-20T21:10:16.121 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-04-20T21:10:16.121 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-04-20T21:10:16.121 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-04-20T21:10:16.121 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty 2024-04-20T21:10:16.121 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/ipwireless 2024-04-20T21:10:16.121 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-04-20T21:10:16.121 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/n_gsm.ko 2024-04-20T21:10:16.121 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/n_hdlc.ko 2024-04-20T21:10:16.121 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/nozomi.ko 2024-04-20T21:10:16.121 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial 2024-04-20T21:10:16.121 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/8250 2024-04-20T21:10:16.121 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-04-20T21:10:16.121 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/arc_uart.ko 2024-04-20T21:10:16.121 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/jsm 2024-04-20T21:10:16.122 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/jsm/jsm.ko 2024-04-20T21:10:16.122 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs 2024-04-20T21:10:16.122 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/core 2024-04-20T21:10:16.122 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/core/ufshcd-core.ko 2024-04-20T21:10:16.122 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/host 2024-04-20T21:10:16.122 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-04-20T21:10:16.122 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio 2024-04-20T21:10:16.122 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio.ko 2024-04-20T21:10:16.122 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_aec.ko 2024-04-20T21:10:16.122 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_cif.ko 2024-04-20T21:10:16.122 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_pci_generic.ko 2024-04-20T21:10:16.122 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_sercos3.ko 2024-04-20T21:10:16.122 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb 2024-04-20T21:10:16.122 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class 2024-04-20T21:10:16.123 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/cdc-acm.ko 2024-04-20T21:10:16.123 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/cdc-wdm.ko 2024-04-20T21:10:16.123 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/usblp.ko 2024-04-20T21:10:16.123 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/usbtmc.ko 2024-04-20T21:10:16.123 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/host 2024-04-20T21:10:16.123 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/host/sl811-hcd.ko 2024-04-20T21:10:16.123 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/image 2024-04-20T21:10:16.123 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/image/mdc800.ko 2024-04-20T21:10:16.123 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/image/microtek.ko 2024-04-20T21:10:16.123 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc 2024-04-20T21:10:16.123 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/adutux.ko 2024-04-20T21:10:16.123 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/appledisplay.ko 2024-04-20T21:10:16.123 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/emi26.ko 2024-04-20T21:10:16.123 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/emi62.ko 2024-04-20T21:10:16.123 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/ezusb.ko 2024-04-20T21:10:16.124 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/idmouse.ko 2024-04-20T21:10:16.124 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/iowarrior.ko 2024-04-20T21:10:16.124 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/isight_firmware.ko 2024-04-20T21:10:16.124 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/ldusb.ko 2024-04-20T21:10:16.124 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/legousbtower.ko 2024-04-20T21:10:16.124 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/sisusbvga 2024-04-20T21:10:16.124 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-04-20T21:10:16.124 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/trancevibrator.ko 2024-04-20T21:10:16.124 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/usb3503.ko 2024-04-20T21:10:16.124 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/usblcd.ko 2024-04-20T21:10:16.124 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/usbsevseg.ko 2024-04-20T21:10:16.124 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/uss720.ko 2024-04-20T21:10:16.124 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/yurex.ko 2024-04-20T21:10:16.124 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/phy 2024-04-20T21:10:16.124 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/phy/phy-generic.ko 2024-04-20T21:10:16.125 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial 2024-04-20T21:10:16.125 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/aircable.ko 2024-04-20T21:10:16.125 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ark3116.ko 2024-04-20T21:10:16.125 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/belkin_sa.ko 2024-04-20T21:10:16.125 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ch341.ko 2024-04-20T21:10:16.125 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/cp210x.ko 2024-04-20T21:10:16.125 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/cyberjack.ko 2024-04-20T21:10:16.125 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/cypress_m8.ko 2024-04-20T21:10:16.125 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/digi_acceleport.ko 2024-04-20T21:10:16.125 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/empeg.ko 2024-04-20T21:10:16.125 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ftdi_sio.ko 2024-04-20T21:10:16.125 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/garmin_gps.ko 2024-04-20T21:10:16.125 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/io_edgeport.ko 2024-04-20T21:10:16.125 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/io_ti.ko 2024-04-20T21:10:16.125 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ipaq.ko 2024-04-20T21:10:16.126 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ipw.ko 2024-04-20T21:10:16.126 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ir-usb.ko 2024-04-20T21:10:16.126 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-04-20T21:10:16.126 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/keyspan.ko 2024-04-20T21:10:16.126 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/keyspan_pda.ko 2024-04-20T21:10:16.126 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/kl5kusb105.ko 2024-04-20T21:10:16.126 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/kobil_sct.ko 2024-04-20T21:10:16.126 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/mct_u232.ko 2024-04-20T21:10:16.126 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/mos7720.ko 2024-04-20T21:10:16.126 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/mos7840.ko 2024-04-20T21:10:16.126 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/navman.ko 2024-04-20T21:10:16.126 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/omninet.ko 2024-04-20T21:10:16.126 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/opticon.ko 2024-04-20T21:10:16.126 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/option.ko 2024-04-20T21:10:16.126 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/oti6858.ko 2024-04-20T21:10:16.127 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/pl2303.ko 2024-04-20T21:10:16.127 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/qcaux.ko 2024-04-20T21:10:16.127 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/qcserial.ko 2024-04-20T21:10:16.127 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/quatech2.ko 2024-04-20T21:10:16.127 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/safe_serial.ko 2024-04-20T21:10:16.127 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/sierra.ko 2024-04-20T21:10:16.127 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/spcp8x5.ko 2024-04-20T21:10:16.127 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ssu100.ko 2024-04-20T21:10:16.127 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/symbolserial.ko 2024-04-20T21:10:16.127 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-04-20T21:10:16.127 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-04-20T21:10:16.127 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/usb_debug.ko 2024-04-20T21:10:16.127 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/usb_wwan.ko 2024-04-20T21:10:16.127 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/visor.ko 2024-04-20T21:10:16.127 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/whiteheat.ko 2024-04-20T21:10:16.128 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/xsens_mt.ko 2024-04-20T21:10:16.128 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage 2024-04-20T21:10:16.128 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/uas.ko 2024-04-20T21:10:16.128 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-alauda.ko 2024-04-20T21:10:16.128 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-cypress.ko 2024-04-20T21:10:16.128 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-datafab.ko 2024-04-20T21:10:16.128 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-04-20T21:10:16.128 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-freecom.ko 2024-04-20T21:10:16.128 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-isd200.ko 2024-04-20T21:10:16.128 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-04-20T21:10:16.128 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-karma.ko 2024-04-20T21:10:16.128 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-onetouch.ko 2024-04-20T21:10:16.128 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-realtek.ko 2024-04-20T21:10:16.128 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-sddr09.ko 2024-04-20T21:10:16.128 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-sddr55.ko 2024-04-20T21:10:16.128 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-usbat.ko 2024-04-20T21:10:16.129 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/usb-storage.ko 2024-04-20T21:10:16.129 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip 2024-04-20T21:10:16.129 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip/usbip-core.ko 2024-04-20T21:10:16.129 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip/usbip-host.ko 2024-04-20T21:10:16.129 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-04-20T21:10:16.129 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio 2024-04-20T21:10:16.129 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/pci 2024-04-20T21:10:16.129 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-04-20T21:10:16.129 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/pci/vfio-pci.ko 2024-04-20T21:10:16.129 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/vfio.ko 2024-04-20T21:10:16.129 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-04-20T21:10:16.129 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost 2024-04-20T21:10:16.129 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost.ko 2024-04-20T21:10:16.129 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost_iotlb.ko 2024-04-20T21:10:16.130 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost_net.ko 2024-04-20T21:10:16.130 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost_scsi.ko 2024-04-20T21:10:16.130 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video 2024-04-20T21:10:16.130 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight 2024-04-20T21:10:16.130 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight/apple_bl.ko 2024-04-20T21:10:16.130 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight/lcd.ko 2024-04-20T21:10:16.130 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight/platform_lcd.ko 2024-04-20T21:10:16.130 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio 2024-04-20T21:10:16.130 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio.ko 2024-04-20T21:10:16.130 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_balloon.ko 2024-04-20T21:10:16.130 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_mmio.ko 2024-04-20T21:10:16.130 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_pci.ko 2024-04-20T21:10:16.130 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-04-20T21:10:16.130 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-04-20T21:10:16.130 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_ring.ko 2024-04-20T21:10:16.131 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1 2024-04-20T21:10:16.131 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/masters 2024-04-20T21:10:16.131 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/masters/ds2482.ko 2024-04-20T21:10:16.131 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/masters/ds2490.ko 2024-04-20T21:10:16.131 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves 2024-04-20T21:10:16.131 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-04-20T21:10:16.131 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-04-20T21:10:16.131 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-04-20T21:10:16.131 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-04-20T21:10:16.131 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-04-20T21:10:16.131 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-04-20T21:10:16.131 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-04-20T21:10:16.131 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-04-20T21:10:16.131 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-04-20T21:10:16.131 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_smem.ko 2024-04-20T21:10:16.132 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_therm.ko 2024-04-20T21:10:16.132 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/wire.ko 2024-04-20T21:10:16.132 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog 2024-04-20T21:10:16.132 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/alim1535_wdt.ko 2024-04-20T21:10:16.132 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/alim7101_wdt.ko 2024-04-20T21:10:16.132 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/f71808e_wdt.ko 2024-04-20T21:10:16.132 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/hpwdt.ko 2024-04-20T21:10:16.132 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/i6300esb.ko 2024-04-20T21:10:16.132 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-04-20T21:10:16.132 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/iTCO_wdt.ko 2024-04-20T21:10:16.132 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/ib700wdt.ko 2024-04-20T21:10:16.132 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/ibmasr.ko 2024-04-20T21:10:16.132 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-04-20T21:10:16.132 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/it8712f_wdt.ko 2024-04-20T21:10:16.132 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/it87_wdt.ko 2024-04-20T21:10:16.133 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/machzwd.ko 2024-04-20T21:10:16.133 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/nv_tco.ko 2024-04-20T21:10:16.133 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/pcwd_pci.ko 2024-04-20T21:10:16.133 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/pcwd_usb.ko 2024-04-20T21:10:16.133 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-04-20T21:10:16.133 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/sch311x_wdt.ko 2024-04-20T21:10:16.133 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/softdog.ko 2024-04-20T21:10:16.133 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/sp5100_tco.ko 2024-04-20T21:10:16.133 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/via_wdt.ko 2024-04-20T21:10:16.133 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-04-20T21:10:16.133 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/w83877f_wdt.ko 2024-04-20T21:10:16.133 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/w83977f_wdt.ko 2024-04-20T21:10:16.133 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/wdt_pci.ko 2024-04-20T21:10:16.133 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/xen_wdt.ko 2024-04-20T21:10:16.133 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen 2024-04-20T21:10:16.134 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-acpi-processor.ko 2024-04-20T21:10:16.134 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-evtchn.ko 2024-04-20T21:10:16.134 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-gntalloc.ko 2024-04-20T21:10:16.134 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-gntdev.ko 2024-04-20T21:10:16.134 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-pciback 2024-04-20T21:10:16.134 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-04-20T21:10:16.134 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-privcmd.ko 2024-04-20T21:10:16.134 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xenfs 2024-04-20T21:10:16.134 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xenfs/xenfs.ko 2024-04-20T21:10:16.134 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs 2024-04-20T21:10:16.134 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/9p 2024-04-20T21:10:16.134 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/9p/9p.ko 2024-04-20T21:10:16.134 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/affs 2024-04-20T21:10:16.134 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/affs/affs.ko 2024-04-20T21:10:16.134 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/befs 2024-04-20T21:10:16.135 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/befs/befs.ko 2024-04-20T21:10:16.135 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/binfmt_misc.ko 2024-04-20T21:10:16.135 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/btrfs 2024-04-20T21:10:16.135 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/btrfs/btrfs.ko 2024-04-20T21:10:16.135 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cachefiles 2024-04-20T21:10:16.135 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cachefiles/cachefiles.ko 2024-04-20T21:10:16.135 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ceph 2024-04-20T21:10:16.135 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ceph/ceph.ko 2024-04-20T21:10:16.135 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/coda 2024-04-20T21:10:16.135 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/coda/coda.ko 2024-04-20T21:10:16.135 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cramfs 2024-04-20T21:10:16.135 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cramfs/cramfs.ko 2024-04-20T21:10:16.135 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/dlm 2024-04-20T21:10:16.135 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/dlm/dlm.ko 2024-04-20T21:10:16.135 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ecryptfs 2024-04-20T21:10:16.136 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ecryptfs/ecryptfs.ko 2024-04-20T21:10:16.136 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/f2fs 2024-04-20T21:10:16.136 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/f2fs/f2fs.ko 2024-04-20T21:10:16.136 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat 2024-04-20T21:10:16.136 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat/fat.ko 2024-04-20T21:10:16.136 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat/msdos.ko 2024-04-20T21:10:16.136 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat/vfat.ko 2024-04-20T21:10:16.136 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fuse 2024-04-20T21:10:16.136 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fuse/cuse.ko 2024-04-20T21:10:16.136 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fuse/fuse.ko 2024-04-20T21:10:16.136 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/gfs2 2024-04-20T21:10:16.136 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/gfs2/gfs2.ko 2024-04-20T21:10:16.136 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfs 2024-04-20T21:10:16.136 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfs/hfs.ko 2024-04-20T21:10:16.136 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfsplus 2024-04-20T21:10:16.137 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfsplus/hfsplus.ko 2024-04-20T21:10:16.137 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/isofs 2024-04-20T21:10:16.137 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/isofs/isofs.ko 2024-04-20T21:10:16.137 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/jfs 2024-04-20T21:10:16.137 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/jfs/jfs.ko 2024-04-20T21:10:16.137 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/lockd 2024-04-20T21:10:16.137 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/lockd/lockd.ko 2024-04-20T21:10:16.137 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/minix 2024-04-20T21:10:16.137 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/minix/minix.ko 2024-04-20T21:10:16.137 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/netfs 2024-04-20T21:10:16.137 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/netfs/netfs.ko 2024-04-20T21:10:16.137 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs 2024-04-20T21:10:16.137 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/blocklayout 2024-04-20T21:10:16.137 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-04-20T21:10:16.137 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/filelayout 2024-04-20T21:10:16.137 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-04-20T21:10:16.138 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/flexfilelayout 2024-04-20T21:10:16.138 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-04-20T21:10:16.138 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/nfs.ko 2024-04-20T21:10:16.138 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/nfsv3.ko 2024-04-20T21:10:16.138 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/nfsv4.ko 2024-04-20T21:10:16.138 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs_common 2024-04-20T21:10:16.138 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs_common/grace.ko 2024-04-20T21:10:16.138 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs_common/nfs_acl.ko 2024-04-20T21:10:16.138 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfsd 2024-04-20T21:10:16.138 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfsd/nfsd.ko 2024-04-20T21:10:16.138 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nilfs2 2024-04-20T21:10:16.138 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nilfs2/nilfs2.ko 2024-04-20T21:10:16.138 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls 2024-04-20T21:10:16.138 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-celtic.ko 2024-04-20T21:10:16.139 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-centeuro.ko 2024-04-20T21:10:16.139 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-croatian.ko 2024-04-20T21:10:16.139 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-cyrillic.ko 2024-04-20T21:10:16.139 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-gaelic.ko 2024-04-20T21:10:16.139 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-greek.ko 2024-04-20T21:10:16.139 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-iceland.ko 2024-04-20T21:10:16.139 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-inuit.ko 2024-04-20T21:10:16.139 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-roman.ko 2024-04-20T21:10:16.139 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-romanian.ko 2024-04-20T21:10:16.139 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-turkish.ko 2024-04-20T21:10:16.139 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp1250.ko 2024-04-20T21:10:16.139 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp1251.ko 2024-04-20T21:10:16.139 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp1255.ko 2024-04-20T21:10:16.139 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp737.ko 2024-04-20T21:10:16.139 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp775.ko 2024-04-20T21:10:16.140 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp850.ko 2024-04-20T21:10:16.140 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp852.ko 2024-04-20T21:10:16.140 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp855.ko 2024-04-20T21:10:16.140 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp857.ko 2024-04-20T21:10:16.140 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp860.ko 2024-04-20T21:10:16.140 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp861.ko 2024-04-20T21:10:16.140 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp862.ko 2024-04-20T21:10:16.140 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp863.ko 2024-04-20T21:10:16.140 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp864.ko 2024-04-20T21:10:16.140 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp865.ko 2024-04-20T21:10:16.140 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp866.ko 2024-04-20T21:10:16.140 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp869.ko 2024-04-20T21:10:16.140 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp874.ko 2024-04-20T21:10:16.140 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp932.ko 2024-04-20T21:10:16.140 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp936.ko 2024-04-20T21:10:16.140 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp949.ko 2024-04-20T21:10:16.141 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp950.ko 2024-04-20T21:10:16.141 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_euc-jp.ko 2024-04-20T21:10:16.141 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-1.ko 2024-04-20T21:10:16.141 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-13.ko 2024-04-20T21:10:16.141 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-14.ko 2024-04-20T21:10:16.141 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-15.ko 2024-04-20T21:10:16.141 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-2.ko 2024-04-20T21:10:16.141 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-3.ko 2024-04-20T21:10:16.141 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-4.ko 2024-04-20T21:10:16.141 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-5.ko 2024-04-20T21:10:16.141 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-6.ko 2024-04-20T21:10:16.141 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-7.ko 2024-04-20T21:10:16.141 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-9.ko 2024-04-20T21:10:16.141 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_koi8-r.ko 2024-04-20T21:10:16.142 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_koi8-ru.ko 2024-04-20T21:10:16.142 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_koi8-u.ko 2024-04-20T21:10:16.142 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_ucs2_utils.ko 2024-04-20T21:10:16.142 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_utf8.ko 2024-04-20T21:10:16.142 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2 2024-04-20T21:10:16.142 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/cluster 2024-04-20T21:10:16.142 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-04-20T21:10:16.142 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlm 2024-04-20T21:10:16.142 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-04-20T21:10:16.142 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlmfs 2024-04-20T21:10:16.142 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-04-20T21:10:16.142 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2.ko 2024-04-20T21:10:16.142 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-04-20T21:10:16.142 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-04-20T21:10:16.142 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-04-20T21:10:16.143 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/overlayfs 2024-04-20T21:10:16.143 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/overlayfs/overlay.ko 2024-04-20T21:10:16.143 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/pstore 2024-04-20T21:10:16.143 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/pstore/ramoops.ko 2024-04-20T21:10:16.143 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/reiserfs 2024-04-20T21:10:16.143 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/reiserfs/reiserfs.ko 2024-04-20T21:10:16.143 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/romfs 2024-04-20T21:10:16.143 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/romfs/romfs.ko 2024-04-20T21:10:16.143 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb 2024-04-20T21:10:16.143 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/client 2024-04-20T21:10:16.143 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/client/cifs.ko 2024-04-20T21:10:16.143 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/common 2024-04-20T21:10:16.143 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/common/cifs_arc4.ko 2024-04-20T21:10:16.143 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/common/cifs_md4.ko 2024-04-20T21:10:16.143 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/squashfs 2024-04-20T21:10:16.144 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/squashfs/squashfs.ko 2024-04-20T21:10:16.144 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/sysv 2024-04-20T21:10:16.144 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/sysv/sysv.ko 2024-04-20T21:10:16.144 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/udf 2024-04-20T21:10:16.144 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/udf/udf.ko 2024-04-20T21:10:16.144 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ufs 2024-04-20T21:10:16.144 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ufs/ufs.ko 2024-04-20T21:10:16.144 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/xfs 2024-04-20T21:10:16.144 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/xfs/xfs.ko 2024-04-20T21:10:16.144 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel 2024-04-20T21:10:16.144 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/rcu 2024-04-20T21:10:16.144 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/rcu/rcutorture.ko 2024-04-20T21:10:16.144 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/torture.ko 2024-04-20T21:10:16.144 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/trace 2024-04-20T21:10:16.144 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-04-20T21:10:16.145 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib 2024-04-20T21:10:16.145 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/asn1_encoder.ko 2024-04-20T21:10:16.145 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crc-itu-t.ko 2024-04-20T21:10:16.145 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crc8.ko 2024-04-20T21:10:16.145 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crypto 2024-04-20T21:10:16.145 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crypto/libarc4.ko 2024-04-20T21:10:16.145 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crypto/libdes.ko 2024-04-20T21:10:16.145 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/libcrc32c.ko 2024-04-20T21:10:16.145 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lru_cache.ko 2024-04-20T21:10:16.145 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lz4 2024-04-20T21:10:16.145 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lz4/lz4_compress.ko 2024-04-20T21:10:16.145 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lz4/lz4hc_compress.ko 2024-04-20T21:10:16.145 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/math 2024-04-20T21:10:16.145 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/math/cordic.ko 2024-04-20T21:10:16.145 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/raid6 2024-04-20T21:10:16.146 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/raid6/raid6_pq.ko 2024-04-20T21:10:16.146 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/reed_solomon 2024-04-20T21:10:16.146 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/reed_solomon/reed_solomon.ko 2024-04-20T21:10:16.146 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/ts_bm.ko 2024-04-20T21:10:16.146 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/ts_fsm.ko 2024-04-20T21:10:16.146 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/ts_kmp.ko 2024-04-20T21:10:16.146 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/zstd 2024-04-20T21:10:16.146 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/zstd/zstd_compress.ko 2024-04-20T21:10:16.146 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/mm 2024-04-20T21:10:16.146 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/mm/hwpoison-inject.ko 2024-04-20T21:10:16.146 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net 2024-04-20T21:10:16.146 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan 2024-04-20T21:10:16.146 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/6lowpan.ko 2024-04-20T21:10:16.146 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_dest.ko 2024-04-20T21:10:16.147 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_fragment.ko 2024-04-20T21:10:16.147 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_hop.ko 2024-04-20T21:10:16.147 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_ipv6.ko 2024-04-20T21:10:16.147 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_mobility.ko 2024-04-20T21:10:16.147 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_routing.ko 2024-04-20T21:10:16.147 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_udp.ko 2024-04-20T21:10:16.147 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802 2024-04-20T21:10:16.147 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/garp.ko 2024-04-20T21:10:16.147 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/mrp.ko 2024-04-20T21:10:16.147 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/p8022.ko 2024-04-20T21:10:16.147 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/psnap.ko 2024-04-20T21:10:16.147 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/stp.ko 2024-04-20T21:10:16.147 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/8021q 2024-04-20T21:10:16.147 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/8021q/8021q.ko 2024-04-20T21:10:16.147 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p 2024-04-20T21:10:16.148 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet.ko 2024-04-20T21:10:16.148 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet_fd.ko 2024-04-20T21:10:16.148 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet_rdma.ko 2024-04-20T21:10:16.148 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet_virtio.ko 2024-04-20T21:10:16.148 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/batman-adv 2024-04-20T21:10:16.148 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/batman-adv/batman-adv.ko 2024-04-20T21:10:16.148 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge 2024-04-20T21:10:16.148 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/br_netfilter.ko 2024-04-20T21:10:16.148 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/bridge.ko 2024-04-20T21:10:16.148 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter 2024-04-20T21:10:16.148 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-04-20T21:10:16.148 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_among.ko 2024-04-20T21:10:16.148 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_arp.ko 2024-04-20T21:10:16.148 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-04-20T21:10:16.148 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-04-20T21:10:16.149 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_ip.ko 2024-04-20T21:10:16.149 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-04-20T21:10:16.149 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_limit.ko 2024-04-20T21:10:16.149 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_log.ko 2024-04-20T21:10:16.149 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_mark.ko 2024-04-20T21:10:16.149 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-04-20T21:10:16.149 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-04-20T21:10:16.149 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-04-20T21:10:16.149 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-04-20T21:10:16.149 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_snat.ko 2024-04-20T21:10:16.149 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_stp.ko 2024-04-20T21:10:16.149 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-04-20T21:10:16.149 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-04-20T21:10:16.149 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-04-20T21:10:16.149 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-04-20T21:10:16.150 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtables.ko 2024-04-20T21:10:16.150 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-04-20T21:10:16.150 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-04-20T21:10:16.150 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-04-20T21:10:16.150 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ceph 2024-04-20T21:10:16.150 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ceph/libceph.ko 2024-04-20T21:10:16.150 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/core 2024-04-20T21:10:16.150 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/core/failover.ko 2024-04-20T21:10:16.150 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/core/pktgen.ko 2024-04-20T21:10:16.150 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp 2024-04-20T21:10:16.150 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp.ko 2024-04-20T21:10:16.150 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp_diag.ko 2024-04-20T21:10:16.150 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp_ipv4.ko 2024-04-20T21:10:16.150 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp_ipv6.ko 2024-04-20T21:10:16.150 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dns_resolver 2024-04-20T21:10:16.151 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dns_resolver/dns_resolver.ko 2024-04-20T21:10:16.151 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa 2024-04-20T21:10:16.151 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/dsa_core.ko 2024-04-20T21:10:16.151 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_brcm.ko 2024-04-20T21:10:16.151 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_dsa.ko 2024-04-20T21:10:16.151 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_none.ko 2024-04-20T21:10:16.151 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_trailer.ko 2024-04-20T21:10:16.151 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154 2024-04-20T21:10:16.151 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/6lowpan 2024-04-20T21:10:16.151 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-04-20T21:10:16.151 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/ieee802154.ko 2024-04-20T21:10:16.151 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/ieee802154_socket.ko 2024-04-20T21:10:16.151 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4 2024-04-20T21:10:16.151 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ah4.ko 2024-04-20T21:10:16.151 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/esp4.ko 2024-04-20T21:10:16.152 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/fou.ko 2024-04-20T21:10:16.152 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/gre.ko 2024-04-20T21:10:16.152 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/inet_diag.ko 2024-04-20T21:10:16.152 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ip_gre.ko 2024-04-20T21:10:16.152 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ip_tunnel.ko 2024-04-20T21:10:16.152 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ip_vti.ko 2024-04-20T21:10:16.152 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ipcomp.ko 2024-04-20T21:10:16.152 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ipip.ko 2024-04-20T21:10:16.152 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter 2024-04-20T21:10:16.152 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/arp_tables.ko 2024-04-20T21:10:16.152 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-04-20T21:10:16.152 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-04-20T21:10:16.152 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-04-20T21:10:16.152 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-04-20T21:10:16.152 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-04-20T21:10:16.153 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-04-20T21:10:16.153 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-04-20T21:10:16.153 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-04-20T21:10:16.153 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-04-20T21:10:16.153 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_security.ko 2024-04-20T21:10:16.153 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-04-20T21:10:16.153 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-04-20T21:10:16.153 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-04-20T21:10:16.153 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-04-20T21:10:16.153 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-04-20T21:10:16.153 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-04-20T21:10:16.153 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-04-20T21:10:16.153 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-04-20T21:10:16.153 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-04-20T21:10:16.154 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-04-20T21:10:16.154 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_bic.ko 2024-04-20T21:10:16.154 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_dctcp.ko 2024-04-20T21:10:16.154 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_diag.ko 2024-04-20T21:10:16.154 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_highspeed.ko 2024-04-20T21:10:16.154 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_htcp.ko 2024-04-20T21:10:16.154 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_hybla.ko 2024-04-20T21:10:16.154 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_illinois.ko 2024-04-20T21:10:16.154 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_lp.ko 2024-04-20T21:10:16.154 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_scalable.ko 2024-04-20T21:10:16.154 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_vegas.ko 2024-04-20T21:10:16.154 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_veno.ko 2024-04-20T21:10:16.154 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_westwood.ko 2024-04-20T21:10:16.154 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_yeah.ko 2024-04-20T21:10:16.154 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tunnel4.ko 2024-04-20T21:10:16.155 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/udp_diag.ko 2024-04-20T21:10:16.155 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/udp_tunnel.ko 2024-04-20T21:10:16.155 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/xfrm4_tunnel.ko 2024-04-20T21:10:16.155 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6 2024-04-20T21:10:16.155 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ah6.ko 2024-04-20T21:10:16.155 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/esp6.ko 2024-04-20T21:10:16.155 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/fou6.ko 2024-04-20T21:10:16.155 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ip6_tunnel.ko 2024-04-20T21:10:16.155 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-04-20T21:10:16.155 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ip6_vti.ko 2024-04-20T21:10:16.155 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ipcomp6.ko 2024-04-20T21:10:16.155 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter 2024-04-20T21:10:16.155 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-04-20T21:10:16.155 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-04-20T21:10:16.155 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-04-20T21:10:16.156 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-04-20T21:10:16.156 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-04-20T21:10:16.156 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-04-20T21:10:16.156 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-04-20T21:10:16.156 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-04-20T21:10:16.156 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-04-20T21:10:16.156 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-04-20T21:10:16.156 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-04-20T21:10:16.156 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-04-20T21:10:16.156 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-04-20T21:10:16.156 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-04-20T21:10:16.156 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-04-20T21:10:16.156 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-04-20T21:10:16.156 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-04-20T21:10:16.156 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-04-20T21:10:16.157 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-04-20T21:10:16.157 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-04-20T21:10:16.157 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-04-20T21:10:16.157 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-04-20T21:10:16.157 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-04-20T21:10:16.157 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-04-20T21:10:16.157 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/sit.ko 2024-04-20T21:10:16.157 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/tunnel6.ko 2024-04-20T21:10:16.157 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/xfrm6_tunnel.ko 2024-04-20T21:10:16.157 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/key 2024-04-20T21:10:16.157 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/key/af_key.ko 2024-04-20T21:10:16.157 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp 2024-04-20T21:10:16.157 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_core.ko 2024-04-20T21:10:16.157 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_debugfs.ko 2024-04-20T21:10:16.157 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_eth.ko 2024-04-20T21:10:16.158 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_ip.ko 2024-04-20T21:10:16.158 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_ip6.ko 2024-04-20T21:10:16.158 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_netlink.ko 2024-04-20T21:10:16.158 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_ppp.ko 2024-04-20T21:10:16.158 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/llc 2024-04-20T21:10:16.158 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/llc/llc.ko 2024-04-20T21:10:16.158 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mac802154 2024-04-20T21:10:16.158 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mac802154/mac802154.ko 2024-04-20T21:10:16.158 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mpls 2024-04-20T21:10:16.158 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mpls/mpls_gso.ko 2024-04-20T21:10:16.158 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter 2024-04-20T21:10:16.158 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset 2024-04-20T21:10:16.158 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set.ko 2024-04-20T21:10:16.158 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-04-20T21:10:16.159 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-04-20T21:10:16.159 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-04-20T21:10:16.159 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-04-20T21:10:16.159 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-04-20T21:10:16.159 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-04-20T21:10:16.159 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-04-20T21:10:16.159 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-04-20T21:10:16.159 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-04-20T21:10:16.159 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-04-20T21:10:16.159 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-04-20T21:10:16.159 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-04-20T21:10:16.159 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-04-20T21:10:16.159 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-04-20T21:10:16.159 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-04-20T21:10:16.159 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs 2024-04-20T21:10:16.160 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs.ko 2024-04-20T21:10:16.160 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-04-20T21:10:16.160 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-04-20T21:10:16.160 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-04-20T21:10:16.160 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-04-20T21:10:16.160 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-04-20T21:10:16.160 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-04-20T21:10:16.160 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-04-20T21:10:16.160 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-04-20T21:10:16.160 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-04-20T21:10:16.160 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-04-20T21:10:16.160 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-04-20T21:10:16.160 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-04-20T21:10:16.160 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-04-20T21:10:16.160 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conncount.ko 2024-04-20T21:10:16.161 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack.ko 2024-04-20T21:10:16.161 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-04-20T21:10:16.161 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-04-20T21:10:16.161 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-04-20T21:10:16.161 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_h323.ko 2024-04-20T21:10:16.161 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_irc.ko 2024-04-20T21:10:16.161 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-04-20T21:10:16.161 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-04-20T21:10:16.161 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-04-20T21:10:16.161 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_sane.ko 2024-04-20T21:10:16.161 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_sip.ko 2024-04-20T21:10:16.161 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-04-20T21:10:16.161 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-04-20T21:10:16.161 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_dup_netdev.ko 2024-04-20T21:10:16.162 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_flow_table.ko 2024-04-20T21:10:16.162 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_flow_table_inet.ko 2024-04-20T21:10:16.162 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_log_syslog.ko 2024-04-20T21:10:16.162 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat.ko 2024-04-20T21:10:16.162 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_amanda.ko 2024-04-20T21:10:16.162 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_ftp.ko 2024-04-20T21:10:16.162 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_irc.ko 2024-04-20T21:10:16.162 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_sip.ko 2024-04-20T21:10:16.162 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_tftp.ko 2024-04-20T21:10:16.162 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_synproxy_core.ko 2024-04-20T21:10:16.162 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_tables.ko 2024-04-20T21:10:16.162 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink.ko 2024-04-20T21:10:16.162 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_acct.ko 2024-04-20T21:10:16.162 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_log.ko 2024-04-20T21:10:16.162 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_osf.ko 2024-04-20T21:10:16.163 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_queue.ko 2024-04-20T21:10:16.163 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_chain_nat.ko 2024-04-20T21:10:16.163 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_compat.ko 2024-04-20T21:10:16.163 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_ct.ko 2024-04-20T21:10:16.163 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_dup_netdev.ko 2024-04-20T21:10:16.163 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_fib.ko 2024-04-20T21:10:16.163 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_fwd_netdev.ko 2024-04-20T21:10:16.163 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_hash.ko 2024-04-20T21:10:16.163 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_limit.ko 2024-04-20T21:10:16.163 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_log.ko 2024-04-20T21:10:16.163 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_masq.ko 2024-04-20T21:10:16.163 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_nat.ko 2024-04-20T21:10:16.163 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_numgen.ko 2024-04-20T21:10:16.163 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_queue.ko 2024-04-20T21:10:16.163 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_quota.ko 2024-04-20T21:10:16.164 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_redir.ko 2024-04-20T21:10:16.164 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_reject.ko 2024-04-20T21:10:16.164 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_reject_inet.ko 2024-04-20T21:10:16.164 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_socket.ko 2024-04-20T21:10:16.164 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_tunnel.ko 2024-04-20T21:10:16.164 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_AUDIT.ko 2024-04-20T21:10:16.164 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CHECKSUM.ko 2024-04-20T21:10:16.164 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CLASSIFY.ko 2024-04-20T21:10:16.164 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-04-20T21:10:16.164 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CT.ko 2024-04-20T21:10:16.164 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_DSCP.ko 2024-04-20T21:10:16.164 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_HL.ko 2024-04-20T21:10:16.164 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_HMARK.ko 2024-04-20T21:10:16.164 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_IDLETIMER.ko 2024-04-20T21:10:16.164 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_LED.ko 2024-04-20T21:10:16.165 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_LOG.ko 2024-04-20T21:10:16.165 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_MASQUERADE.ko 2024-04-20T21:10:16.165 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_NETMAP.ko 2024-04-20T21:10:16.165 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_NFLOG.ko 2024-04-20T21:10:16.165 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_NFQUEUE.ko 2024-04-20T21:10:16.165 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_RATEEST.ko 2024-04-20T21:10:16.165 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_REDIRECT.ko 2024-04-20T21:10:16.165 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_SECMARK.ko 2024-04-20T21:10:16.165 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TCPMSS.ko 2024-04-20T21:10:16.165 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-04-20T21:10:16.165 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TEE.ko 2024-04-20T21:10:16.165 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TPROXY.ko 2024-04-20T21:10:16.165 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TRACE.ko 2024-04-20T21:10:16.165 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_addrtype.ko 2024-04-20T21:10:16.166 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_bpf.ko 2024-04-20T21:10:16.166 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_cgroup.ko 2024-04-20T21:10:16.166 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_cluster.ko 2024-04-20T21:10:16.166 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_comment.ko 2024-04-20T21:10:16.166 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connbytes.ko 2024-04-20T21:10:16.166 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connlabel.ko 2024-04-20T21:10:16.166 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connlimit.ko 2024-04-20T21:10:16.166 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connmark.ko 2024-04-20T21:10:16.166 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_conntrack.ko 2024-04-20T21:10:16.166 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_cpu.ko 2024-04-20T21:10:16.166 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_dccp.ko 2024-04-20T21:10:16.166 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_devgroup.ko 2024-04-20T21:10:16.166 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_dscp.ko 2024-04-20T21:10:16.166 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_ecn.ko 2024-04-20T21:10:16.166 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_esp.ko 2024-04-20T21:10:16.167 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_hashlimit.ko 2024-04-20T21:10:16.167 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_helper.ko 2024-04-20T21:10:16.167 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_hl.ko 2024-04-20T21:10:16.167 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_ipcomp.ko 2024-04-20T21:10:16.167 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_iprange.ko 2024-04-20T21:10:16.167 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_ipvs.ko 2024-04-20T21:10:16.167 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_l2tp.ko 2024-04-20T21:10:16.167 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_length.ko 2024-04-20T21:10:16.167 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_limit.ko 2024-04-20T21:10:16.167 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_mac.ko 2024-04-20T21:10:16.167 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_mark.ko 2024-04-20T21:10:16.167 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_multiport.ko 2024-04-20T21:10:16.167 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_nat.ko 2024-04-20T21:10:16.167 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_nfacct.ko 2024-04-20T21:10:16.167 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_osf.ko 2024-04-20T21:10:16.168 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_owner.ko 2024-04-20T21:10:16.168 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_physdev.ko 2024-04-20T21:10:16.168 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_pkttype.ko 2024-04-20T21:10:16.168 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_policy.ko 2024-04-20T21:10:16.168 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_quota.ko 2024-04-20T21:10:16.168 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_rateest.ko 2024-04-20T21:10:16.168 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_realm.ko 2024-04-20T21:10:16.168 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_recent.ko 2024-04-20T21:10:16.168 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_sctp.ko 2024-04-20T21:10:16.168 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_set.ko 2024-04-20T21:10:16.168 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_socket.ko 2024-04-20T21:10:16.168 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_state.ko 2024-04-20T21:10:16.168 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_statistic.ko 2024-04-20T21:10:16.168 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_string.ko 2024-04-20T21:10:16.169 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_tcpmss.ko 2024-04-20T21:10:16.169 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_time.ko 2024-04-20T21:10:16.169 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_u32.ko 2024-04-20T21:10:16.169 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netlink 2024-04-20T21:10:16.169 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netlink/netlink_diag.ko 2024-04-20T21:10:16.169 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/nsh 2024-04-20T21:10:16.169 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/nsh/nsh.ko 2024-04-20T21:10:16.169 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch 2024-04-20T21:10:16.169 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/openvswitch.ko 2024-04-20T21:10:16.169 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/vport-geneve.ko 2024-04-20T21:10:16.169 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/vport-gre.ko 2024-04-20T21:10:16.169 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/vport-vxlan.ko 2024-04-20T21:10:16.169 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/packet 2024-04-20T21:10:16.169 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/packet/af_packet_diag.ko 2024-04-20T21:10:16.170 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds 2024-04-20T21:10:16.170 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds/rds.ko 2024-04-20T21:10:16.170 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds/rds_rdma.ko 2024-04-20T21:10:16.170 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds/rds_tcp.ko 2024-04-20T21:10:16.170 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched 2024-04-20T21:10:16.170 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_csum.ko 2024-04-20T21:10:16.170 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_gact.ko 2024-04-20T21:10:16.170 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_mirred.ko 2024-04-20T21:10:16.170 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_nat.ko 2024-04-20T21:10:16.170 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_pedit.ko 2024-04-20T21:10:16.170 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_police.ko 2024-04-20T21:10:16.170 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_simple.ko 2024-04-20T21:10:16.170 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_skbedit.ko 2024-04-20T21:10:16.170 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_vlan.ko 2024-04-20T21:10:16.171 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_basic.ko 2024-04-20T21:10:16.171 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_bpf.ko 2024-04-20T21:10:16.171 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_flow.ko 2024-04-20T21:10:16.171 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_fw.ko 2024-04-20T21:10:16.171 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_route.ko 2024-04-20T21:10:16.171 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_u32.ko 2024-04-20T21:10:16.171 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_cmp.ko 2024-04-20T21:10:16.171 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_ipset.ko 2024-04-20T21:10:16.171 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_meta.ko 2024-04-20T21:10:16.171 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_nbyte.ko 2024-04-20T21:10:16.171 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_text.ko 2024-04-20T21:10:16.171 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_u32.ko 2024-04-20T21:10:16.171 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_choke.ko 2024-04-20T21:10:16.171 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_codel.ko 2024-04-20T21:10:16.172 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_drr.ko 2024-04-20T21:10:16.172 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_fq.ko 2024-04-20T21:10:16.172 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_gred.ko 2024-04-20T21:10:16.172 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_hfsc.ko 2024-04-20T21:10:16.172 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_hhf.ko 2024-04-20T21:10:16.172 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_htb.ko 2024-04-20T21:10:16.172 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_ingress.ko 2024-04-20T21:10:16.172 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_mqprio.ko 2024-04-20T21:10:16.172 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_mqprio_lib.ko 2024-04-20T21:10:16.172 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_multiq.ko 2024-04-20T21:10:16.172 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_netem.ko 2024-04-20T21:10:16.172 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_pie.ko 2024-04-20T21:10:16.172 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_plug.ko 2024-04-20T21:10:16.172 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_prio.ko 2024-04-20T21:10:16.172 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_qfq.ko 2024-04-20T21:10:16.173 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_red.ko 2024-04-20T21:10:16.173 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_sfb.ko 2024-04-20T21:10:16.173 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_sfq.ko 2024-04-20T21:10:16.173 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_tbf.ko 2024-04-20T21:10:16.173 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_teql.ko 2024-04-20T21:10:16.173 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sctp 2024-04-20T21:10:16.173 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sctp/sctp.ko 2024-04-20T21:10:16.173 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sctp/sctp_diag.ko 2024-04-20T21:10:16.173 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc 2024-04-20T21:10:16.173 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/auth_gss 2024-04-20T21:10:16.173 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-04-20T21:10:16.173 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-04-20T21:10:16.173 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/sunrpc.ko 2024-04-20T21:10:16.173 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/xprtrdma 2024-04-20T21:10:16.173 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-04-20T21:10:16.174 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/tipc 2024-04-20T21:10:16.174 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/tipc/tipc.ko 2024-04-20T21:10:16.174 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/tipc/tipc_diag.ko 2024-04-20T21:10:16.174 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/unix 2024-04-20T21:10:16.174 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/unix/unix_diag.ko 2024-04-20T21:10:16.174 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock 2024-04-20T21:10:16.174 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-04-20T21:10:16.174 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-04-20T21:10:16.174 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vsock.ko 2024-04-20T21:10:16.174 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vsock_diag.ko 2024-04-20T21:10:16.174 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vsock_loopback.ko 2024-04-20T21:10:16.174 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/xfrm 2024-04-20T21:10:16.174 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/xfrm/xfrm_ipcomp.ko 2024-04-20T21:10:16.174 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security 2024-04-20T21:10:16.175 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys 2024-04-20T21:10:16.175 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/encrypted-keys 2024-04-20T21:10:16.175 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-04-20T21:10:16.175 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/trusted-keys 2024-04-20T21:10:16.175 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/trusted-keys/trusted.ko 2024-04-20T21:10:16.175 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.alias 2024-04-20T21:10:16.175 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.alias.bin 2024-04-20T21:10:16.175 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin 2024-04-20T21:10:16.175 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin.alias.bin 2024-04-20T21:10:16.175 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin.bin 2024-04-20T21:10:16.175 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin.modinfo 2024-04-20T21:10:16.175 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.dep 2024-04-20T21:10:16.176 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.dep.bin 2024-04-20T21:10:16.176 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.devname 2024-04-20T21:10:16.176 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.order 2024-04-20T21:10:16.176 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.softdep 2024-04-20T21:10:16.176 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.symbols 2024-04-20T21:10:16.176 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.symbols.bin 2024-04-20T21:10:16.176 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/vmlinuz 2024-04-20T21:10:16.180 DEBUG:teuthology.task.kernel:get_image_version: 6.9.0-rc3-g806ec36b2112 2024-04-20T21:10:16.180 DEBUG:teuthology.orchestra.run.smithi189:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-04-20T21:10:16.203 INFO:teuthology.orchestra.run.smithi189.stdout:postinstall scriptlet (using /bin/sh): 2024-04-20T21:10:16.203 INFO:teuthology.orchestra.run.smithi189.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-04-20T21:10:16.203 INFO:teuthology.orchestra.run.smithi189.stdout: /usr/bin/kernel-install add 6.9.0-rc3-g806ec36b2112 /lib/modules/6.9.0-rc3-g806ec36b2112/vmlinuz 2024-04-20T21:10:16.203 INFO:teuthology.orchestra.run.smithi189.stdout:fi 2024-04-20T21:10:16.203 INFO:teuthology.orchestra.run.smithi189.stdout:for file in vmlinuz System.map config; do 2024-04-20T21:10:16.203 INFO:teuthology.orchestra.run.smithi189.stdout: if ! cmp --silent "/lib/modules/6.9.0-rc3-g806ec36b2112/${file}" "/boot/${file}-6.9.0-rc3-g806ec36b2112"; then 2024-04-20T21:10:16.204 INFO:teuthology.orchestra.run.smithi189.stdout: cp "/lib/modules/6.9.0-rc3-g806ec36b2112/${file}" "/boot/${file}-6.9.0-rc3-g806ec36b2112" 2024-04-20T21:10:16.204 INFO:teuthology.orchestra.run.smithi189.stdout: fi 2024-04-20T21:10:16.204 INFO:teuthology.orchestra.run.smithi189.stdout:done 2024-04-20T21:10:16.204 INFO:teuthology.orchestra.run.smithi189.stdout:if [ -d "/lib/modules/6.9.0-rc3-g806ec36b2112/dtb" ] && \ 2024-04-20T21:10:16.204 INFO:teuthology.orchestra.run.smithi189.stdout: ! diff -rq "/lib/modules/6.9.0-rc3-g806ec36b2112/dtb" "/boot/dtb-6.9.0-rc3-g806ec36b2112" >/dev/null 2>&1; then 2024-04-20T21:10:16.204 INFO:teuthology.orchestra.run.smithi189.stdout: rm -rf "/boot/dtb-6.9.0-rc3-g806ec36b2112" 2024-04-20T21:10:16.204 INFO:teuthology.orchestra.run.smithi189.stdout: cp -r "/lib/modules/6.9.0-rc3-g806ec36b2112/dtb" "/boot/dtb-6.9.0-rc3-g806ec36b2112" 2024-04-20T21:10:16.204 INFO:teuthology.orchestra.run.smithi189.stdout:fi 2024-04-20T21:10:16.204 INFO:teuthology.orchestra.run.smithi189.stdout:if [ ! -e "/lib/modules/6.9.0-rc3-g806ec36b2112/modules.dep" ]; then 2024-04-20T21:10:16.204 INFO:teuthology.orchestra.run.smithi189.stdout: /usr/sbin/depmod 6.9.0-rc3-g806ec36b2112 2024-04-20T21:10:16.204 INFO:teuthology.orchestra.run.smithi189.stdout:fi 2024-04-20T21:10:16.204 INFO:teuthology.orchestra.run.smithi189.stdout:preuninstall scriptlet (using /bin/sh): 2024-04-20T21:10:16.204 INFO:teuthology.orchestra.run.smithi189.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-04-20T21:10:16.204 INFO:teuthology.orchestra.run.smithi189.stdout:kernel-install remove 6.9.0-rc3-g806ec36b2112 2024-04-20T21:10:16.204 INFO:teuthology.orchestra.run.smithi189.stdout:fi 2024-04-20T21:10:16.205 DEBUG:teuthology.orchestra.run.smithi189:> sudo rpm -qi grub2-tools 2024-04-20T21:10:16.279 INFO:teuthology.orchestra.run.smithi189.stdout:Name : grub2-tools 2024-04-20T21:10:16.279 INFO:teuthology.orchestra.run.smithi189.stdout:Epoch : 1 2024-04-20T21:10:16.279 INFO:teuthology.orchestra.run.smithi189.stdout:Version : 2.06 2024-04-20T21:10:16.279 INFO:teuthology.orchestra.run.smithi189.stdout:Release : 68.el9 2024-04-20T21:10:16.279 INFO:teuthology.orchestra.run.smithi189.stdout:Architecture: x86_64 2024-04-20T21:10:16.279 INFO:teuthology.orchestra.run.smithi189.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-04-20T21:10:16.279 INFO:teuthology.orchestra.run.smithi189.stdout:Group : Unspecified 2024-04-20T21:10:16.279 INFO:teuthology.orchestra.run.smithi189.stdout:Size : 8264254 2024-04-20T21:10:16.279 INFO:teuthology.orchestra.run.smithi189.stdout:License : GPLv3+ 2024-04-20T21:10:16.279 INFO:teuthology.orchestra.run.smithi189.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-20T21:10:16.279 INFO:teuthology.orchestra.run.smithi189.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-20T21:10:16.280 INFO:teuthology.orchestra.run.smithi189.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-20T21:10:16.280 INFO:teuthology.orchestra.run.smithi189.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-20T21:10:16.280 INFO:teuthology.orchestra.run.smithi189.stdout:Packager : builder@centos.org 2024-04-20T21:10:16.280 INFO:teuthology.orchestra.run.smithi189.stdout:Vendor : CentOS 2024-04-20T21:10:16.280 INFO:teuthology.orchestra.run.smithi189.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-20T21:10:16.280 INFO:teuthology.orchestra.run.smithi189.stdout:Summary : Support tools for GRUB. 2024-04-20T21:10:16.280 INFO:teuthology.orchestra.run.smithi189.stdout:Description : 2024-04-20T21:10:16.280 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-20T21:10:16.280 INFO:teuthology.orchestra.run.smithi189.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-20T21:10:16.280 INFO:teuthology.orchestra.run.smithi189.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-20T21:10:16.280 INFO:teuthology.orchestra.run.smithi189.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-20T21:10:16.280 INFO:teuthology.orchestra.run.smithi189.stdout:hardware devices. 2024-04-20T21:10:16.280 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-20T21:10:16.280 INFO:teuthology.orchestra.run.smithi189.stdout:This subpackage provides tools for support of all platforms. 2024-04-20T21:10:16.281 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-20T21:10:16.281 INFO:teuthology.task.kernel:Updating grub on smithi189 to boot 6.9.0-rc3-g806ec36b2112 2024-04-20T21:10:16.281 DEBUG:teuthology.orchestra.run.smithi189:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-20T21:10:16.861 INFO:teuthology.orchestra.run.smithi189.stderr:Generating grub configuration file ... 2024-04-20T21:10:16.867 DEBUG:teuthology.orchestra.remote:timed out 2024-04-20T21:10:18.192 INFO:teuthology.orchestra.run.smithi189.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-20T21:10:18.216 INFO:teuthology.orchestra.run.smithi189.stderr:done 2024-04-20T21:10:18.218 DEBUG:teuthology.orchestra.run.smithi189:> mktemp 2024-04-20T21:10:18.233 INFO:teuthology.orchestra.run.smithi189.stdout:/tmp/tmp.Rp00xusg71 2024-04-20T21:10:18.234 DEBUG:teuthology.orchestra.run.smithi189:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.Rp00xusg71 2024-04-20T21:10:18.299 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod 0666 /tmp/tmp.Rp00xusg71 2024-04-20T21:10:18.435 DEBUG:teuthology.orchestra.remote:smithi189:/tmp/tmp.Rp00xusg71 is 6KB 2024-04-20T21:10:18.482 DEBUG:teuthology.orchestra.run.smithi189:> rm -fr /tmp/tmp.Rp00xusg71 2024-04-20T21:10:18.497 DEBUG:teuthology.orchestra.run.smithi189:> sudo /bin/ls /boot/loader/entries || true 2024-04-20T21:10:18.568 INFO:teuthology.orchestra.run.smithi189.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-04-20T21:10:18.568 INFO:teuthology.orchestra.run.smithi189.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-04-20T21:10:18.568 INFO:teuthology.orchestra.run.smithi189.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-04-20T21:10:18.568 INFO:teuthology.orchestra.run.smithi189.stdout:7fc8828e06d0482694bcf4a447d13cb6-6.9.0-rc3-g806ec36b2112.conf 2024-04-20T21:10:18.570 DEBUG:teuthology.orchestra.run.smithi189:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-6.9.0-rc3-g806ec36b2112 2024-04-20T21:10:18.727 DEBUG:teuthology.orchestra.run.smithi189:> sudo shutdown -r now 2024-04-20T21:10:25.875 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi062.front.sepia.ceph.com' 2024-04-20T21:10:25.876 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi062.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-20T21:10:33.444 DEBUG:teuthology.orchestra.run.smithi062:> true 2024-04-20T21:10:34.036 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi062.front.sepia.ceph.com' 2024-04-20T21:10:34.037 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "6.9.0-rc3-g806ec36b2112"... 2024-04-20T21:10:34.037 DEBUG:teuthology.orchestra.run.smithi062:> uname -r 2024-04-20T21:10:34.076 INFO:teuthology.orchestra.run.smithi062.stdout:6.9.0-rc3-g806ec36b2112 2024-04-20T21:10:34.076 DEBUG:teuthology.task.kernel:current kernel version is 6.9.0-rc3-g806ec36b2112 vs 6.9.0-rc3-g806ec36b2112 2024-04-20T21:10:34.076 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-20T21:10:34.076 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-20T21:10:35.077 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-04-20T21:10:35.077 DEBUG:teuthology.orchestra.run.smithi062:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-20T21:10:35.193 INFO:teuthology.orchestra.run.smithi062.stdout:ttyS1 2024-04-20T21:10:35.259 DEBUG:teuthology.parallel:result is None 2024-04-20T21:10:48.729 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2024-04-20T21:10:48.730 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi189.front.sepia.ceph.com' 2024-04-20T21:10:48.730 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-20T21:11:07.294 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.189 2024-04-20T21:11:16.294 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi189.front.sepia.ceph.com' 2024-04-20T21:11:16.295 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-20T21:11:19.358 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.189 2024-04-20T21:11:31.403 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi189.front.sepia.ceph.com' 2024-04-20T21:11:31.404 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-20T21:11:49.790 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.189 2024-04-20T21:12:04.792 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi189.front.sepia.ceph.com' 2024-04-20T21:12:04.792 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-20T21:12:05.210 DEBUG:teuthology.orchestra.run.smithi189:> true 2024-04-20T21:12:05.764 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi189.front.sepia.ceph.com' 2024-04-20T21:12:05.764 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "6.9.0-rc3-g806ec36b2112"... 2024-04-20T21:12:05.764 DEBUG:teuthology.orchestra.run.smithi189:> uname -r 2024-04-20T21:12:05.802 INFO:teuthology.orchestra.run.smithi189.stdout:6.9.0-rc3-g806ec36b2112 2024-04-20T21:12:05.802 DEBUG:teuthology.task.kernel:current kernel version is 6.9.0-rc3-g806ec36b2112 vs 6.9.0-rc3-g806ec36b2112 2024-04-20T21:12:05.802 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-20T21:12:05.802 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-20T21:12:06.804 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-04-20T21:12:06.804 DEBUG:teuthology.orchestra.run.smithi189:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-20T21:12:06.931 INFO:teuthology.orchestra.run.smithi189.stdout:ttyS1 2024-04-20T21:12:06.975 DEBUG:teuthology.parallel:result is None 2024-04-20T21:12:06.976 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-20T21:12:06.983 INFO:teuthology.task.internal:Creating test directory... 2024-04-20T21:12:06.984 DEBUG:teuthology.orchestra.run.smithi062:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-20T21:12:06.993 DEBUG:teuthology.orchestra.run.smithi189:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-20T21:12:07.047 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-20T21:12:07.054 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-20T21:12:07.062 INFO:teuthology.task.internal:Creating archive directory... 2024-04-20T21:12:07.062 DEBUG:teuthology.orchestra.run.smithi062:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-20T21:12:07.068 DEBUG:teuthology.orchestra.run.smithi189:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-20T21:12:07.129 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-20T21:12:07.136 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-20T21:12:07.136 DEBUG:teuthology.orchestra.run.smithi062:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-20T21:12:07.144 DEBUG:teuthology.orchestra.run.smithi189:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-20T21:12:07.239 INFO:teuthology.orchestra.run.smithi062.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-20T21:12:07.269 INFO:teuthology.orchestra.run.smithi189.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-20T21:12:07.272 INFO:teuthology.orchestra.run.smithi062.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-20T21:12:07.302 INFO:teuthology.orchestra.run.smithi189.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-20T21:12:07.307 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-20T21:12:07.317 INFO:teuthology.task.internal:Configuring sudo... 2024-04-20T21:12:07.318 DEBUG:teuthology.orchestra.run.smithi062:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-20T21:12:07.323 DEBUG:teuthology.orchestra.run.smithi189:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-20T21:12:07.413 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-20T21:12:07.422 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-20T21:12:07.422 DEBUG:teuthology.orchestra.run.smithi062:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-20T21:12:07.430 DEBUG:teuthology.orchestra.run.smithi189:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-20T21:12:07.480 DEBUG:teuthology.orchestra.run.smithi062:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-20T21:12:07.567 DEBUG:teuthology.orchestra.run.smithi062:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-20T21:12:07.677 DEBUG:teuthology.orchestra.run.smithi062:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-20T21:12:07.764 DEBUG:teuthology.orchestra.run.smithi062:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-20T21:12:07.860 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-04-20T21:12:07.860 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-20T21:12:07.950 DEBUG:teuthology.orchestra.run.smithi189:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-20T21:12:08.028 DEBUG:teuthology.orchestra.run.smithi189:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-20T21:12:08.109 DEBUG:teuthology.orchestra.run.smithi189:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-20T21:12:08.179 DEBUG:teuthology.orchestra.run.smithi189:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-20T21:12:08.259 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-20T21:12:08.259 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-20T21:12:08.317 DEBUG:teuthology.orchestra.run.smithi062:> sudo service rsyslog restart 2024-04-20T21:12:08.326 DEBUG:teuthology.orchestra.run.smithi189:> sudo service rsyslog restart 2024-04-20T21:12:08.445 INFO:teuthology.orchestra.run.smithi062.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-20T21:12:08.469 INFO:teuthology.orchestra.run.smithi189.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-20T21:12:08.980 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-20T21:12:08.988 INFO:teuthology.task.internal:Starting timer... 2024-04-20T21:12:08.988 INFO:teuthology.run_tasks:Running task pcp... 2024-04-20T21:12:09.019 INFO:teuthology.run_tasks:Running task selinux... 2024-04-20T21:12:09.028 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-04-20T21:12:09.029 DEBUG:teuthology.orchestra.run.smithi062:> sudo service auditd rotate 2024-04-20T21:12:09.182 INFO:teuthology.orchestra.run.smithi062.stdout:Rotating logs: 2024-04-20T21:12:09.187 DEBUG:teuthology.orchestra.run.smithi189:> sudo service auditd rotate 2024-04-20T21:12:09.294 INFO:teuthology.orchestra.run.smithi189.stdout:Rotating logs: 2024-04-20T21:12:09.298 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-20T21:12:09.298 DEBUG:teuthology.orchestra.run.smithi062:> /usr/sbin/getenforce 2024-04-20T21:12:09.358 INFO:teuthology.orchestra.run.smithi062.stdout:Permissive 2024-04-20T21:12:09.358 DEBUG:teuthology.orchestra.run.smithi189:> /usr/sbin/getenforce 2024-04-20T21:12:09.404 INFO:teuthology.orchestra.run.smithi189.stdout:Permissive 2024-04-20T21:12:09.405 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi062.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi189.front.sepia.ceph.com': 'permissive'} 2024-04-20T21:12:09.405 DEBUG:teuthology.orchestra.run.smithi062:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-20T21:12:09.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-20T21:12:09.488 DEBUG:teuthology.orchestra.run.smithi189:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-20T21:12:09.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-20T21:12:09.557 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-20T21:12:09.557 DEBUG:teuthology.orchestra.run.smithi062:> sudo /usr/sbin/setenforce permissive 2024-04-20T21:12:09.634 DEBUG:teuthology.orchestra.run.smithi189:> sudo /usr/sbin/setenforce permissive 2024-04-20T21:12:09.712 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-20T21:12:09.722 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-20T21:12:09.745 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-20T21:12:09.747 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi062.front.sepia.ceph.com,smithi189.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-20T21:14:32.445 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi062.front.sepia.ceph.com'), Remote(name='ubuntu@smithi189.front.sepia.ceph.com')] 2024-04-20T21:14:32.446 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi062.front.sepia.ceph.com' 2024-04-20T21:14:32.447 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi062.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-20T21:14:32.561 DEBUG:teuthology.orchestra.run.smithi062:> true 2024-04-20T21:14:32.722 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi062.front.sepia.ceph.com' 2024-04-20T21:14:32.722 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi189.front.sepia.ceph.com' 2024-04-20T21:14:32.722 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-20T21:14:32.831 DEBUG:teuthology.orchestra.run.smithi189:> true 2024-04-20T21:14:32.987 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi189.front.sepia.ceph.com' 2024-04-20T21:14:32.987 INFO:teuthology.run_tasks:Running task clock... 2024-04-20T21:14:32.997 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-20T21:14:32.997 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-20T21:14:32.998 DEBUG:teuthology.orchestra.run.smithi062:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-20T21:14:33.006 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-20T21:14:33.006 DEBUG:teuthology.orchestra.run.smithi189:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-20T21:14:33.096 INFO:teuthology.orchestra.run.smithi189.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-20T21:14:33.102 INFO:teuthology.orchestra.run.smithi062.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-20T21:14:33.137 INFO:teuthology.orchestra.run.smithi189.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-20T21:14:33.146 INFO:teuthology.orchestra.run.smithi062.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-20T21:14:33.214 INFO:teuthology.orchestra.run.smithi189.stderr:sudo: ntpd: command not found 2024-04-20T21:14:33.229 INFO:teuthology.orchestra.run.smithi062.stderr:sudo: ntpd: command not found 2024-04-20T21:14:33.246 INFO:teuthology.orchestra.run.smithi189.stdout:506 Cannot talk to daemon 2024-04-20T21:14:33.263 INFO:teuthology.orchestra.run.smithi062.stdout:506 Cannot talk to daemon 2024-04-20T21:14:33.287 INFO:teuthology.orchestra.run.smithi189.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-20T21:14:33.306 INFO:teuthology.orchestra.run.smithi062.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-20T21:14:33.328 INFO:teuthology.orchestra.run.smithi189.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-20T21:14:33.351 INFO:teuthology.orchestra.run.smithi062.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-20T21:14:33.466 INFO:teuthology.orchestra.run.smithi189.stderr:bash: line 1: ntpq: command not found 2024-04-20T21:14:33.476 INFO:teuthology.orchestra.run.smithi189.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-20T21:14:33.476 INFO:teuthology.orchestra.run.smithi189.stdout:=============================================================================== 2024-04-20T21:14:33.476 INFO:teuthology.orchestra.run.smithi189.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-20T21:14:33.476 INFO:teuthology.orchestra.run.smithi189.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-20T21:14:33.476 INFO:teuthology.orchestra.run.smithi189.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-20T21:14:33.476 INFO:teuthology.orchestra.run.smithi189.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-20T21:14:33.504 INFO:teuthology.orchestra.run.smithi062.stderr:bash: line 1: ntpq: command not found 2024-04-20T21:14:33.513 INFO:teuthology.orchestra.run.smithi062.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-20T21:14:33.513 INFO:teuthology.orchestra.run.smithi062.stdout:=============================================================================== 2024-04-20T21:14:33.513 INFO:teuthology.orchestra.run.smithi062.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-20T21:14:33.513 INFO:teuthology.orchestra.run.smithi062.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-20T21:14:33.514 INFO:teuthology.orchestra.run.smithi062.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-20T21:14:33.514 INFO:teuthology.orchestra.run.smithi062.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-20T21:14:33.515 INFO:teuthology.run_tasks:Running task install... 2024-04-20T21:14:33.523 DEBUG:teuthology.task.install:project ceph 2024-04-20T21:14:33.523 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'a8c84f5997950fd7d008204839253291d52e6eda'}} 2024-04-20T21:14:33.524 DEBUG:teuthology.task.install:config {'exclude_packages': ['ceph-volume'], 'tag': 'v18.2.0', 'flavor': 'default', 'sha1': 'a8c84f5997950fd7d008204839253291d52e6eda'} 2024-04-20T21:14:33.524 INFO:teuthology.task.install:Using flavor: default 2024-04-20T21:14:33.533 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-20T21:14:33.533 INFO:teuthology.task.install:extra packages: [] 2024-04-20T21:14:33.533 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'a8c84f5997950fd7d008204839253291d52e6eda', 'tag': 'v18.2.0', 'wait_for_package': False} 2024-04-20T21:14:33.535 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-04-20T21:14:33.535 INFO:teuthology.packaging:ref: None 2024-04-20T21:14:33.535 INFO:teuthology.packaging:tag: v18.2.0 2024-04-20T21:14:33.535 INFO:teuthology.packaging:branch: None 2024-04-20T21:14:33.535 INFO:teuthology.packaging:sha1: a8c84f5997950fd7d008204839253291d52e6eda 2024-04-20T21:14:33.964 DEBUG:teuthology.repo_utils:git ls-remote https://git.ceph.com/ceph-ci v18.2.0^{} -> None 2024-04-20T21:14:33.965 INFO:teuthology.packaging:Tag 'v18.2.0' not found in https://git.ceph.com/ceph-ci; will also look in https://git.ceph.com/ceph 2024-04-20T21:14:36.496 DEBUG:teuthology.repo_utils:git ls-remote https://git.ceph.com/ceph v18.2.0^{} -> 5dd24139a1eada541a3bc16b6941c5dde975e26d 2024-04-20T21:14:36.496 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dd24139a1eada541a3bc16b6941c5dde975e26d 2024-04-20T21:14:36.498 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'a8c84f5997950fd7d008204839253291d52e6eda', 'tag': 'v18.2.0', 'wait_for_package': False} 2024-04-20T21:14:36.498 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-04-20T21:14:36.499 INFO:teuthology.packaging:ref: None 2024-04-20T21:14:36.499 INFO:teuthology.packaging:tag: v18.2.0 2024-04-20T21:14:36.499 INFO:teuthology.packaging:branch: None 2024-04-20T21:14:36.499 INFO:teuthology.packaging:sha1: a8c84f5997950fd7d008204839253291d52e6eda 2024-04-20T21:14:36.499 INFO:teuthology.packaging:Tag 'v18.2.0' not found in https://git.ceph.com/ceph-ci; will also look in https://git.ceph.com/ceph 2024-04-20T21:14:36.499 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dd24139a1eada541a3bc16b6941c5dde975e26d 2024-04-20T21:14:36.646 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/reef/5dd24139a1eada541a3bc16b6941c5dde975e26d/centos/9/flavors/default/ 2024-04-20T21:14:36.646 INFO:teuthology.task.install.rpm:Package version is 18.2.0-0 2024-04-20T21:14:36.685 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/reef/5dd24139a1eada541a3bc16b6941c5dde975e26d/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/reef/5dd24139a1eada541a3bc16b6941c5dde975e26d/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/reef/5dd24139a1eada541a3bc16b6941c5dde975e26d/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-20T21:14:36.685 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-20T21:14:36.686 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-20T21:14:36.725 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/reef/5dd24139a1eada541a3bc16b6941c5dde975e26d/centos/9/flavors/default/ 2024-04-20T21:14:36.725 INFO:teuthology.task.install.rpm:Package version is 18.2.0-0 2024-04-20T21:14:36.754 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/reef/5dd24139a1eada541a3bc16b6941c5dde975e26d/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/reef/5dd24139a1eada541a3bc16b6941c5dde975e26d/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/reef/5dd24139a1eada541a3bc16b6941c5dde975e26d/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-20T21:14:36.754 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-04-20T21:14:36.754 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-20T21:14:36.776 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-20T21:14:36.776 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-04-20T21:14:36.776 INFO:teuthology.packaging:ref: None 2024-04-20T21:14:36.777 INFO:teuthology.packaging:tag: v18.2.0 2024-04-20T21:14:36.777 INFO:teuthology.packaging:branch: None 2024-04-20T21:14:36.777 INFO:teuthology.packaging:sha1: a8c84f5997950fd7d008204839253291d52e6eda 2024-04-20T21:14:36.777 DEBUG:teuthology.orchestra.run.smithi189:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v18.2.0/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-20T21:14:36.850 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-20T21:14:36.850 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-04-20T21:14:36.850 INFO:teuthology.packaging:ref: None 2024-04-20T21:14:36.850 INFO:teuthology.packaging:tag: v18.2.0 2024-04-20T21:14:36.850 INFO:teuthology.packaging:branch: None 2024-04-20T21:14:36.850 INFO:teuthology.packaging:sha1: a8c84f5997950fd7d008204839253291d52e6eda 2024-04-20T21:14:36.850 DEBUG:teuthology.orchestra.run.smithi062:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v18.2.0/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-20T21:14:36.902 DEBUG:teuthology.orchestra.run.smithi189:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-20T21:14:36.983 DEBUG:teuthology.orchestra.run.smithi062:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-20T21:14:37.008 DEBUG:teuthology.orchestra.run.smithi189:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-20T21:14:37.078 INFO:teuthology.orchestra.run.smithi189.stdout:check_obsoletes = 1 2024-04-20T21:14:37.081 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum clean all 2024-04-20T21:14:37.092 DEBUG:teuthology.orchestra.run.smithi062:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-20T21:14:37.170 INFO:teuthology.orchestra.run.smithi062.stdout:check_obsoletes = 1 2024-04-20T21:14:37.175 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum clean all 2024-04-20T21:14:37.398 INFO:teuthology.orchestra.run.smithi189.stdout:48 files removed 2024-04-20T21:14:37.439 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install ceph-radosgw 2024-04-20T21:14:37.508 INFO:teuthology.orchestra.run.smithi062.stdout:48 files removed 2024-04-20T21:14:37.547 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install ceph-radosgw 2024-04-20T21:14:38.132 INFO:teuthology.orchestra.run.smithi189.stdout:ceph packages for x86_64 292 kB/s | 76 kB 00:00 2024-04-20T21:14:38.197 INFO:teuthology.orchestra.run.smithi062.stdout:ceph packages for x86_64 361 kB/s | 76 kB 00:00 2024-04-20T21:14:38.441 INFO:teuthology.orchestra.run.smithi189.stdout:ceph noarch packages 39 kB/s | 9.3 kB 00:00 2024-04-20T21:14:38.447 INFO:teuthology.orchestra.run.smithi062.stdout:ceph noarch packages 49 kB/s | 9.3 kB 00:00 2024-04-20T21:14:38.708 INFO:teuthology.orchestra.run.smithi189.stdout:ceph source packages 10 kB/s | 2.2 kB 00:00 2024-04-20T21:14:38.732 INFO:teuthology.orchestra.run.smithi062.stdout:ceph source packages 9.6 kB/s | 2.2 kB 00:00 2024-04-20T21:14:39.258 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 8.0 MB 00:00 2024-04-20T21:14:40.715 INFO:teuthology.orchestra.run.smithi062.stdout:CentOS Stream 9 - BaseOS 4.2 MB/s | 8.0 MB 00:01 2024-04-20T21:14:41.517 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS Stream 9 - AppStream 18 MB/s | 19 MB 00:01 2024-04-20T21:14:44.248 INFO:teuthology.orchestra.run.smithi062.stdout:CentOS Stream 9 - AppStream 8.3 MB/s | 19 MB 00:02 2024-04-20T21:14:47.334 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS Stream 9 - CRB 3.3 MB/s | 6.2 MB 00:01 2024-04-20T21:14:50.002 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS Stream 9 - Extras packages 12 kB/s | 16 kB 00:01 2024-04-20T21:14:51.628 INFO:teuthology.orchestra.run.smithi189.stdout:Extra Packages for Enterprise Linux 14 MB/s | 21 MB 00:01 2024-04-20T21:14:56.258 INFO:teuthology.orchestra.run.smithi062.stdout:CentOS Stream 9 - CRB 790 kB/s | 6.2 MB 00:08 2024-04-20T21:14:57.589 INFO:teuthology.orchestra.run.smithi189.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-04-20T21:14:58.182 INFO:teuthology.orchestra.run.smithi062.stdout:CentOS Stream 9 - Extras packages 31 kB/s | 16 kB 00:00 2024-04-20T21:15:00.110 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-04-20T21:15:00.119 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-20T21:15:00.120 INFO:teuthology.orchestra.run.smithi189.stdout: Package Arch Version Repository Size 2024-04-20T21:15:00.120 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-20T21:15:00.120 INFO:teuthology.orchestra.run.smithi189.stdout:Installing: 2024-04-20T21:15:00.120 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-radosgw x86_64 2:18.2.0-0.el9 ceph 7.6 M 2024-04-20T21:15:00.120 INFO:teuthology.orchestra.run.smithi189.stdout:Upgrading: 2024-04-20T21:15:00.120 INFO:teuthology.orchestra.run.smithi189.stdout: librados2 x86_64 2:18.2.0-0.el9 ceph 3.3 M 2024-04-20T21:15:00.120 INFO:teuthology.orchestra.run.smithi189.stdout: librbd1 x86_64 2:18.2.0-0.el9 ceph 3.0 M 2024-04-20T21:15:00.120 INFO:teuthology.orchestra.run.smithi189.stdout:Installing dependencies: 2024-04-20T21:15:00.120 INFO:teuthology.orchestra.run.smithi189.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-20T21:15:00.120 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-base x86_64 2:18.2.0-0.el9 ceph 5.2 M 2024-04-20T21:15:00.120 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-common x86_64 2:18.2.0-0.el9 ceph 18 M 2024-04-20T21:15:00.120 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-selinux x86_64 2:18.2.0-0.el9 ceph 24 k 2024-04-20T21:15:00.120 INFO:teuthology.orchestra.run.smithi189.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2024-04-20T21:15:00.121 INFO:teuthology.orchestra.run.smithi189.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-20T21:15:00.121 INFO:teuthology.orchestra.run.smithi189.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-20T21:15:00.121 INFO:teuthology.orchestra.run.smithi189.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-20T21:15:00.121 INFO:teuthology.orchestra.run.smithi189.stdout: libcephfs2 x86_64 2:18.2.0-0.el9 ceph 653 k 2024-04-20T21:15:00.121 INFO:teuthology.orchestra.run.smithi189.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-20T21:15:00.121 INFO:teuthology.orchestra.run.smithi189.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-20T21:15:00.121 INFO:teuthology.orchestra.run.smithi189.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-20T21:15:00.121 INFO:teuthology.orchestra.run.smithi189.stdout: libradosstriper1 x86_64 2:18.2.0-0.el9 ceph 474 k 2024-04-20T21:15:00.121 INFO:teuthology.orchestra.run.smithi189.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-20T21:15:00.121 INFO:teuthology.orchestra.run.smithi189.stdout: librgw2 x86_64 2:18.2.0-0.el9 ceph 4.4 M 2024-04-20T21:15:00.121 INFO:teuthology.orchestra.run.smithi189.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-20T21:15:00.121 INFO:teuthology.orchestra.run.smithi189.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-20T21:15:00.121 INFO:teuthology.orchestra.run.smithi189.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-20T21:15:00.122 INFO:teuthology.orchestra.run.smithi189.stdout: python3-ceph-argparse x86_64 2:18.2.0-0.el9 ceph 45 k 2024-04-20T21:15:00.122 INFO:teuthology.orchestra.run.smithi189.stdout: python3-ceph-common x86_64 2:18.2.0-0.el9 ceph 119 k 2024-04-20T21:15:00.122 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cephfs x86_64 2:18.2.0-0.el9 ceph 155 k 2024-04-20T21:15:00.122 INFO:teuthology.orchestra.run.smithi189.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-20T21:15:00.122 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rados x86_64 2:18.2.0-0.el9 ceph 321 k 2024-04-20T21:15:00.122 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rbd x86_64 2:18.2.0-0.el9 ceph 297 k 2024-04-20T21:15:00.123 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rgw x86_64 2:18.2.0-0.el9 ceph 99 k 2024-04-20T21:15:00.123 INFO:teuthology.orchestra.run.smithi189.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-20T21:15:00.123 INFO:teuthology.orchestra.run.smithi189.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-20T21:15:00.123 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-20T21:15:00.123 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction Summary 2024-04-20T21:15:00.123 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-20T21:15:00.123 INFO:teuthology.orchestra.run.smithi189.stdout:Install 28 Packages 2024-04-20T21:15:00.123 INFO:teuthology.orchestra.run.smithi189.stdout:Upgrade 2 Packages 2024-04-20T21:15:00.123 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-20T21:15:00.124 INFO:teuthology.orchestra.run.smithi189.stdout:Total download size: 53 M 2024-04-20T21:15:00.124 INFO:teuthology.orchestra.run.smithi189.stdout:Downloading Packages: 2024-04-20T21:15:00.749 INFO:teuthology.orchestra.run.smithi189.stdout:(1/30): ceph-base-18.2.0-0.el9.x86_64.rpm 16 MB/s | 5.2 MB 00:00 2024-04-20T21:15:00.766 INFO:teuthology.orchestra.run.smithi189.stdout:(2/30): ceph-selinux-18.2.0-0.el9.x86_64.rpm 1.4 MB/s | 24 kB 00:00 2024-04-20T21:15:00.816 INFO:teuthology.orchestra.run.smithi189.stdout:(3/30): libcephfs2-18.2.0-0.el9.x86_64.rpm 13 MB/s | 653 kB 00:00 2024-04-20T21:15:00.858 INFO:teuthology.orchestra.run.smithi189.stdout:(4/30): libradosstriper1-18.2.0-0.el9.x86_64.rp 11 MB/s | 474 kB 00:00 2024-04-20T21:15:01.014 INFO:teuthology.orchestra.run.smithi189.stdout:(5/30): ceph-radosgw-18.2.0-0.el9.x86_64.rpm 13 MB/s | 7.6 MB 00:00 2024-04-20T21:15:01.042 INFO:teuthology.orchestra.run.smithi189.stdout:(6/30): python3-ceph-argparse-18.2.0-0.el9.x86_ 1.4 MB/s | 45 kB 00:00 2024-04-20T21:15:01.079 INFO:teuthology.orchestra.run.smithi189.stdout:(7/30): python3-ceph-common-18.2.0-0.el9.x86_64 3.6 MB/s | 119 kB 00:00 2024-04-20T21:15:01.109 INFO:teuthology.orchestra.run.smithi189.stdout:(8/30): python3-cephfs-18.2.0-0.el9.x86_64.rpm 4.7 MB/s | 155 kB 00:00 2024-04-20T21:15:01.152 INFO:teuthology.orchestra.run.smithi189.stdout:(9/30): python3-rados-18.2.0-0.el9.x86_64.rpm 7.7 MB/s | 321 kB 00:00 2024-04-20T21:15:01.193 INFO:teuthology.orchestra.run.smithi189.stdout:(10/30): python3-rbd-18.2.0-0.el9.x86_64.rpm 7.1 MB/s | 297 kB 00:00 2024-04-20T21:15:01.236 INFO:teuthology.orchestra.run.smithi189.stdout:(11/30): python3-rgw-18.2.0-0.el9.x86_64.rpm 3.0 MB/s | 99 kB 00:00 2024-04-20T21:15:01.381 INFO:teuthology.orchestra.run.smithi189.stdout:(12/30): librgw2-18.2.0-0.el9.x86_64.rpm 8.5 MB/s | 4.4 MB 00:00 2024-04-20T21:15:01.402 INFO:teuthology.orchestra.run.smithi189.stdout:(13/30): boost-program-options-1.75.0-8.el9.x86 614 kB/s | 107 kB 00:00 2024-04-20T21:15:01.452 INFO:teuthology.orchestra.run.smithi189.stdout:(14/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 920 kB/s | 45 kB 00:00 2024-04-20T21:15:01.841 INFO:teuthology.orchestra.run.smithi189.stdout:(15/30): ceph-common-18.2.0-0.el9.x86_64.rpm 13 MB/s | 18 MB 00:01 2024-04-20T21:15:01.870 INFO:teuthology.orchestra.run.smithi189.stdout:(16/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 326 kB/s | 160 kB 00:00 2024-04-20T21:15:01.953 INFO:teuthology.orchestra.run.smithi189.stdout:(17/30): librdkafka-1.6.1-102.el9.x86_64.rpm 1.3 MB/s | 662 kB 00:00 2024-04-20T21:15:01.986 INFO:teuthology.orchestra.run.smithi189.stdout:(18/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.0 MB/s | 292 kB 00:00 2024-04-20T21:15:02.012 INFO:teuthology.orchestra.run.smithi189.stdout:(19/30): python3-prettytable-0.7.2-27.el9.noarc 299 kB/s | 42 kB 00:00 2024-04-20T21:15:02.679 INFO:teuthology.orchestra.run.smithi189.stdout:(20/30): fmt-8.1.1-5.el9.x86_64.rpm 153 kB/s | 111 kB 00:00 2024-04-20T21:15:02.821 INFO:teuthology.orchestra.run.smithi189.stdout:(21/30): libarrow-doc-9.0.0-10.el9.noarch.rpm 184 kB/s | 26 kB 00:00 2024-04-20T21:15:02.962 INFO:teuthology.orchestra.run.smithi189.stdout:(22/30): liboath-2.6.7-2.el9.x86_64.rpm 344 kB/s | 49 kB 00:00 2024-04-20T21:15:03.029 INFO:teuthology.orchestra.run.smithi189.stdout:(23/30): gperftools-libs-2.9.1-2.el9.x86_64.rpm 304 kB/s | 309 kB 00:01 2024-04-20T21:15:03.213 INFO:teuthology.orchestra.run.smithi189.stdout:(24/30): libunwind-1.6.2-1.el9.x86_64.rpm 269 kB/s | 67 kB 00:00 2024-04-20T21:15:03.538 INFO:teuthology.orchestra.run.smithi189.stdout:(25/30): re2-20211101-3.el9.x86_64.rpm 591 kB/s | 192 kB 00:00 2024-04-20T21:15:04.214 INFO:teuthology.orchestra.run.smithi189.stdout:(26/30): parquet-libs-9.0.0-10.el9.x86_64.rpm 708 kB/s | 838 kB 00:01 2024-04-20T21:15:04.356 INFO:teuthology.orchestra.run.smithi189.stdout:(27/30): librados2-18.2.0-0.el9.x86_64.rpm 23 MB/s | 3.3 MB 00:00 2024-04-20T21:15:04.472 INFO:teuthology.orchestra.run.smithi189.stdout:(28/30): librbd1-18.2.0-0.el9.x86_64.rpm 26 MB/s | 3.0 MB 00:00 2024-04-20T21:15:05.339 INFO:teuthology.orchestra.run.smithi189.stdout:(29/30): thrift-0.15.0-2.el9.x86_64.rpm 900 kB/s | 1.6 MB 00:01 2024-04-20T21:15:07.506 INFO:teuthology.orchestra.run.smithi189.stdout:(30/30): libarrow-9.0.0-10.el9.x86_64.rpm 824 kB/s | 4.4 MB 00:05 2024-04-20T21:15:07.509 INFO:teuthology.orchestra.run.smithi189.stdout:-------------------------------------------------------------------------------- 2024-04-20T21:15:07.510 INFO:teuthology.orchestra.run.smithi189.stdout:Total 7.1 MB/s | 53 MB 00:07 2024-04-20T21:15:07.732 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction check 2024-04-20T21:15:07.831 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction check succeeded. 2024-04-20T21:15:07.831 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction test 2024-04-20T21:15:08.738 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction test succeeded. 2024-04-20T21:15:08.739 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction 2024-04-20T21:15:10.341 INFO:teuthology.orchestra.run.smithi189.stdout: Preparing : 1/1 2024-04-20T21:15:10.451 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2024-04-20T21:15:10.507 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/32 2024-04-20T21:15:10.613 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/32 2024-04-20T21:15:11.022 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2024-04-20T21:15:11.118 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : librados2-2:18.2.0-0.el9.x86_64 5/32 2024-04-20T21:15:11.254 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: librados2-2:18.2.0-0.el9.x86_64 5/32 2024-04-20T21:15:11.360 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-rados-2:18.2.0-0.el9.x86_64 6/32 2024-04-20T21:15:11.386 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libcephfs2-2:18.2.0-0.el9.x86_64 7/32 2024-04-20T21:15:11.470 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: libcephfs2-2:18.2.0-0.el9.x86_64 7/32 2024-04-20T21:15:11.538 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2024-04-20T21:15:11.611 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-ceph-argparse-2:18.2.0-0.el9.x86_64 9/32 2024-04-20T21:15:11.713 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-cephfs-2:18.2.0-0.el9.x86_64 10/32 2024-04-20T21:15:11.739 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libradosstriper1-2:18.2.0-0.el9.x86_64 11/32 2024-04-20T21:15:11.814 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: libradosstriper1-2:18.2.0-0.el9.x86_64 11/32 2024-04-20T21:15:12.009 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/32 2024-04-20T21:15:12.049 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 13/32 2024-04-20T21:15:12.096 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2024-04-20T21:15:12.130 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/32 2024-04-20T21:15:12.168 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 16/32 2024-04-20T21:15:12.218 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2024-04-20T21:15:12.267 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2024-04-20T21:15:12.678 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2024-04-20T21:15:12.719 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : librbd1-2:18.2.0-0.el9.x86_64 20/32 2024-04-20T21:15:12.834 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: librbd1-2:18.2.0-0.el9.x86_64 20/32 2024-04-20T21:15:12.869 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-rbd-2:18.2.0-0.el9.x86_64 21/32 2024-04-20T21:15:12.936 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2024-04-20T21:15:13.569 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 23/32 2024-04-20T21:15:13.605 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : librgw2-2:18.2.0-0.el9.x86_64 24/32 2024-04-20T21:15:13.723 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: librgw2-2:18.2.0-0.el9.x86_64 24/32 2024-04-20T21:15:13.823 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-rgw-2:18.2.0-0.el9.x86_64 25/32 2024-04-20T21:15:13.849 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-ceph-common-2:18.2.0-0.el9.x86_64 26/32 2024-04-20T21:15:14.085 INFO:teuthology.orchestra.run.smithi062.stdout:Extra Packages for Enterprise Linux 1.3 MB/s | 21 MB 00:15 2024-04-20T21:15:16.460 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-common-2:18.2.0-0.el9.x86_64 27/32 2024-04-20T21:15:16.504 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-common-2:18.2.0-0.el9.x86_64 27/32 2024-04-20T21:15:17.143 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-common-2:18.2.0-0.el9.x86_64 27/32 2024-04-20T21:15:17.191 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-base-2:18.2.0-0.el9.x86_64 28/32 2024-04-20T21:15:17.361 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-base-2:18.2.0-0.el9.x86_64 28/32 2024-04-20T21:15:17.361 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-20T21:15:17.361 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-20T21:15:17.361 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-20T21:15:17.400 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-selinux-2:18.2.0-0.el9.x86_64 29/32 2024-04-20T21:15:20.185 INFO:teuthology.orchestra.run.smithi062.stdout:lab-extras 23 kB/s | 1.7 kB 00:00 2024-04-20T21:15:22.766 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2024-04-20T21:15:22.779 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-04-20T21:15:22.779 INFO:teuthology.orchestra.run.smithi062.stdout: Package Arch Version Repository Size 2024-04-20T21:15:22.779 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-04-20T21:15:22.779 INFO:teuthology.orchestra.run.smithi062.stdout:Installing: 2024-04-20T21:15:22.779 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-radosgw x86_64 2:18.2.0-0.el9 ceph 7.6 M 2024-04-20T21:15:22.779 INFO:teuthology.orchestra.run.smithi062.stdout:Upgrading: 2024-04-20T21:15:22.779 INFO:teuthology.orchestra.run.smithi062.stdout: librados2 x86_64 2:18.2.0-0.el9 ceph 3.3 M 2024-04-20T21:15:22.779 INFO:teuthology.orchestra.run.smithi062.stdout: librbd1 x86_64 2:18.2.0-0.el9 ceph 3.0 M 2024-04-20T21:15:22.779 INFO:teuthology.orchestra.run.smithi062.stdout:Installing dependencies: 2024-04-20T21:15:22.779 INFO:teuthology.orchestra.run.smithi062.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-20T21:15:22.779 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-base x86_64 2:18.2.0-0.el9 ceph 5.2 M 2024-04-20T21:15:22.780 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-common x86_64 2:18.2.0-0.el9 ceph 18 M 2024-04-20T21:15:22.780 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-selinux x86_64 2:18.2.0-0.el9 ceph 24 k 2024-04-20T21:15:22.780 INFO:teuthology.orchestra.run.smithi062.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2024-04-20T21:15:22.780 INFO:teuthology.orchestra.run.smithi062.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-20T21:15:22.780 INFO:teuthology.orchestra.run.smithi062.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-20T21:15:22.780 INFO:teuthology.orchestra.run.smithi062.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-20T21:15:22.780 INFO:teuthology.orchestra.run.smithi062.stdout: libcephfs2 x86_64 2:18.2.0-0.el9 ceph 653 k 2024-04-20T21:15:22.780 INFO:teuthology.orchestra.run.smithi062.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-20T21:15:22.780 INFO:teuthology.orchestra.run.smithi062.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-20T21:15:22.780 INFO:teuthology.orchestra.run.smithi062.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-20T21:15:22.780 INFO:teuthology.orchestra.run.smithi062.stdout: libradosstriper1 x86_64 2:18.2.0-0.el9 ceph 474 k 2024-04-20T21:15:22.780 INFO:teuthology.orchestra.run.smithi062.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-20T21:15:22.780 INFO:teuthology.orchestra.run.smithi062.stdout: librgw2 x86_64 2:18.2.0-0.el9 ceph 4.4 M 2024-04-20T21:15:22.781 INFO:teuthology.orchestra.run.smithi062.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-20T21:15:22.781 INFO:teuthology.orchestra.run.smithi062.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-20T21:15:22.781 INFO:teuthology.orchestra.run.smithi062.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-20T21:15:22.781 INFO:teuthology.orchestra.run.smithi062.stdout: python3-ceph-argparse x86_64 2:18.2.0-0.el9 ceph 45 k 2024-04-20T21:15:22.781 INFO:teuthology.orchestra.run.smithi062.stdout: python3-ceph-common x86_64 2:18.2.0-0.el9 ceph 119 k 2024-04-20T21:15:22.781 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cephfs x86_64 2:18.2.0-0.el9 ceph 155 k 2024-04-20T21:15:22.781 INFO:teuthology.orchestra.run.smithi062.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-20T21:15:22.781 INFO:teuthology.orchestra.run.smithi062.stdout: python3-rados x86_64 2:18.2.0-0.el9 ceph 321 k 2024-04-20T21:15:22.781 INFO:teuthology.orchestra.run.smithi062.stdout: python3-rbd x86_64 2:18.2.0-0.el9 ceph 297 k 2024-04-20T21:15:22.781 INFO:teuthology.orchestra.run.smithi062.stdout: python3-rgw x86_64 2:18.2.0-0.el9 ceph 99 k 2024-04-20T21:15:22.781 INFO:teuthology.orchestra.run.smithi062.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-20T21:15:22.781 INFO:teuthology.orchestra.run.smithi062.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-20T21:15:22.781 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:15:22.781 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction Summary 2024-04-20T21:15:22.781 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-04-20T21:15:22.782 INFO:teuthology.orchestra.run.smithi062.stdout:Install 28 Packages 2024-04-20T21:15:22.782 INFO:teuthology.orchestra.run.smithi062.stdout:Upgrade 2 Packages 2024-04-20T21:15:22.782 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:15:22.782 INFO:teuthology.orchestra.run.smithi062.stdout:Total download size: 53 M 2024-04-20T21:15:22.782 INFO:teuthology.orchestra.run.smithi062.stdout:Downloading Packages: 2024-04-20T21:15:23.877 INFO:teuthology.orchestra.run.smithi062.stdout:(1/30): ceph-base-18.2.0-0.el9.x86_64.rpm 15 MB/s | 5.2 MB 00:00 2024-04-20T21:15:23.902 INFO:teuthology.orchestra.run.smithi062.stdout:(2/30): ceph-selinux-18.2.0-0.el9.x86_64.rpm 989 kB/s | 24 kB 00:00 2024-04-20T21:15:23.953 INFO:teuthology.orchestra.run.smithi062.stdout:(3/30): libcephfs2-18.2.0-0.el9.x86_64.rpm 13 MB/s | 653 kB 00:00 2024-04-20T21:15:24.022 INFO:teuthology.orchestra.run.smithi062.stdout:(4/30): libradosstriper1-18.2.0-0.el9.x86_64.rp 6.9 MB/s | 474 kB 00:00 2024-04-20T21:15:24.204 INFO:teuthology.orchestra.run.smithi062.stdout:(5/30): ceph-radosgw-18.2.0-0.el9.x86_64.rpm 11 MB/s | 7.6 MB 00:00 2024-04-20T21:15:24.229 INFO:teuthology.orchestra.run.smithi062.stdout:(6/30): python3-ceph-argparse-18.2.0-0.el9.x86_ 1.8 MB/s | 45 kB 00:00 2024-04-20T21:15:24.254 INFO:teuthology.orchestra.run.smithi062.stdout:(7/30): python3-ceph-common-18.2.0-0.el9.x86_64 4.9 MB/s | 119 kB 00:00 2024-04-20T21:15:24.288 INFO:teuthology.orchestra.run.smithi062.stdout:(8/30): python3-cephfs-18.2.0-0.el9.x86_64.rpm 4.7 MB/s | 155 kB 00:00 2024-04-20T21:15:24.330 INFO:teuthology.orchestra.run.smithi062.stdout:(9/30): python3-rados-18.2.0-0.el9.x86_64.rpm 7.7 MB/s | 321 kB 00:00 2024-04-20T21:15:24.371 INFO:teuthology.orchestra.run.smithi062.stdout:(10/30): python3-rbd-18.2.0-0.el9.x86_64.rpm 7.2 MB/s | 297 kB 00:00 2024-04-20T21:15:24.405 INFO:teuthology.orchestra.run.smithi062.stdout:(11/30): python3-rgw-18.2.0-0.el9.x86_64.rpm 3.0 MB/s | 99 kB 00:00 2024-04-20T21:15:24.531 INFO:teuthology.orchestra.run.smithi062.stdout:(12/30): librgw2-18.2.0-0.el9.x86_64.rpm 8.7 MB/s | 4.4 MB 00:00 2024-04-20T21:15:24.572 INFO:teuthology.orchestra.run.smithi062.stdout:(13/30): boost-program-options-1.75.0-8.el9.x86 645 kB/s | 107 kB 00:00 2024-04-20T21:15:24.614 INFO:teuthology.orchestra.run.smithi062.stdout:(14/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.1 MB/s | 45 kB 00:00 2024-04-20T21:15:24.681 INFO:teuthology.orchestra.run.smithi062.stdout:(15/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.0 MB/s | 160 kB 00:00 2024-04-20T21:15:24.731 INFO:teuthology.orchestra.run.smithi062.stdout:(16/30): librdkafka-1.6.1-102.el9.x86_64.rpm 5.6 MB/s | 662 kB 00:00 2024-04-20T21:15:24.765 INFO:teuthology.orchestra.run.smithi062.stdout:(17/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.4 MB/s | 292 kB 00:00 2024-04-20T21:15:24.790 INFO:teuthology.orchestra.run.smithi062.stdout:(18/30): python3-prettytable-0.7.2-27.el9.noarc 734 kB/s | 42 kB 00:00 2024-04-20T21:15:25.058 INFO:teuthology.orchestra.run.smithi062.stdout:(19/30): ceph-common-18.2.0-0.el9.x86_64.rpm 12 MB/s | 18 MB 00:01 2024-04-20T21:15:25.216 INFO:teuthology.orchestra.run.smithi062.stdout:(20/30): fmt-8.1.1-5.el9.x86_64.rpm 246 kB/s | 111 kB 00:00 2024-04-20T21:15:25.308 INFO:teuthology.orchestra.run.smithi062.stdout:(21/30): libarrow-doc-9.0.0-10.el9.noarch.rpm 286 kB/s | 26 kB 00:00 2024-04-20T21:15:25.400 INFO:teuthology.orchestra.run.smithi062.stdout:(22/30): liboath-2.6.7-2.el9.x86_64.rpm 533 kB/s | 49 kB 00:00 2024-04-20T21:15:25.442 INFO:teuthology.orchestra.run.smithi062.stdout:(23/30): gperftools-libs-2.9.1-2.el9.x86_64.rpm 475 kB/s | 309 kB 00:00 2024-04-20T21:15:25.492 INFO:teuthology.orchestra.run.smithi062.stdout:(24/30): libunwind-1.6.2-1.el9.x86_64.rpm 738 kB/s | 67 kB 00:00 2024-04-20T21:15:25.634 INFO:teuthology.orchestra.run.smithi062.stdout:(25/30): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.3 MB/s | 838 kB 00:00 2024-04-20T21:15:25.667 INFO:teuthology.orchestra.run.smithi062.stdout:(26/30): re2-20211101-3.el9.x86_64.rpm 1.1 MB/s | 192 kB 00:00 2024-04-20T21:15:25.818 INFO:teuthology.orchestra.run.smithi062.stdout:(27/30): librados2-18.2.0-0.el9.x86_64.rpm 22 MB/s | 3.3 MB 00:00 2024-04-20T21:15:25.885 INFO:teuthology.orchestra.run.smithi062.stdout:(28/30): thrift-0.15.0-2.el9.x86_64.rpm 6.3 MB/s | 1.6 MB 00:00 2024-04-20T21:15:26.018 INFO:teuthology.orchestra.run.smithi062.stdout:(29/30): libarrow-9.0.0-10.el9.x86_64.rpm 4.6 MB/s | 4.4 MB 00:00 2024-04-20T21:15:26.110 INFO:teuthology.orchestra.run.smithi062.stdout:(30/30): librbd1-18.2.0-0.el9.x86_64.rpm 10 MB/s | 3.0 MB 00:00 2024-04-20T21:15:26.115 INFO:teuthology.orchestra.run.smithi062.stdout:-------------------------------------------------------------------------------- 2024-04-20T21:15:26.115 INFO:teuthology.orchestra.run.smithi062.stdout:Total 16 MB/s | 53 MB 00:03 2024-04-20T21:15:26.343 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction check 2024-04-20T21:15:26.452 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction check succeeded. 2024-04-20T21:15:26.452 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction test 2024-04-20T21:15:27.392 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction test succeeded. 2024-04-20T21:15:27.394 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction 2024-04-20T21:15:28.279 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-selinux-2:18.2.0-0.el9.x86_64 29/32 2024-04-20T21:15:28.279 INFO:teuthology.orchestra.run.smithi189.stdout:skipping the directory /sys 2024-04-20T21:15:28.279 INFO:teuthology.orchestra.run.smithi189.stdout:skipping the directory /proc 2024-04-20T21:15:28.279 INFO:teuthology.orchestra.run.smithi189.stdout:skipping the directory /mnt 2024-04-20T21:15:28.279 INFO:teuthology.orchestra.run.smithi189.stdout:skipping the directory /var/tmp 2024-04-20T21:15:28.279 INFO:teuthology.orchestra.run.smithi189.stdout:skipping the directory /home 2024-04-20T21:15:28.279 INFO:teuthology.orchestra.run.smithi189.stdout:skipping the directory /root 2024-04-20T21:15:28.279 INFO:teuthology.orchestra.run.smithi189.stdout:skipping the directory /tmp 2024-04-20T21:15:28.279 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-20T21:15:28.975 INFO:teuthology.orchestra.run.smithi062.stdout: Preparing : 1/1 2024-04-20T21:15:29.088 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2024-04-20T21:15:29.137 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/32 2024-04-20T21:15:29.158 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-radosgw-2:18.2.0-0.el9.x86_64 30/32 2024-04-20T21:15:29.216 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-radosgw-2:18.2.0-0.el9.x86_64 30/32 2024-04-20T21:15:29.216 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-20T21:15:29.216 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-20T21:15:29.216 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-20T21:15:29.216 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-20T21:15:29.216 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-20T21:15:29.221 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-04-20T21:15:29.287 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/32 2024-04-20T21:15:29.335 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-04-20T21:15:29.335 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2024-04-20T21:15:29.704 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2024-04-20T21:15:29.814 INFO:teuthology.orchestra.run.smithi062.stdout: Upgrading : librados2-2:18.2.0-0.el9.x86_64 5/32 2024-04-20T21:15:29.953 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: librados2-2:18.2.0-0.el9.x86_64 5/32 2024-04-20T21:15:30.071 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-rados-2:18.2.0-0.el9.x86_64 6/32 2024-04-20T21:15:30.099 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libcephfs2-2:18.2.0-0.el9.x86_64 7/32 2024-04-20T21:15:30.234 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: libcephfs2-2:18.2.0-0.el9.x86_64 7/32 2024-04-20T21:15:30.285 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2024-04-20T21:15:30.358 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-ceph-argparse-2:18.2.0-0.el9.x86_64 9/32 2024-04-20T21:15:30.462 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-cephfs-2:18.2.0-0.el9.x86_64 10/32 2024-04-20T21:15:30.492 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libradosstriper1-2:18.2.0-0.el9.x86_64 11/32 2024-04-20T21:15:30.595 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: libradosstriper1-2:18.2.0-0.el9.x86_64 11/32 2024-04-20T21:15:30.799 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/32 2024-04-20T21:15:30.836 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 13/32 2024-04-20T21:15:30.900 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2024-04-20T21:15:30.901 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-base-2:18.2.0-0.el9.x86_64 1/32 2024-04-20T21:15:30.901 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-common-2:18.2.0-0.el9.x86_64 2/32 2024-04-20T21:15:30.901 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-radosgw-2:18.2.0-0.el9.x86_64 3/32 2024-04-20T21:15:30.901 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-selinux-2:18.2.0-0.el9.x86_64 4/32 2024-04-20T21:15:30.901 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libcephfs2-2:18.2.0-0.el9.x86_64 5/32 2024-04-20T21:15:30.901 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libradosstriper1-2:18.2.0-0.el9.x86_64 6/32 2024-04-20T21:15:30.901 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librgw2-2:18.2.0-0.el9.x86_64 7/32 2024-04-20T21:15:30.901 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-ceph-argparse-2:18.2.0-0.el9.x86_64 8/32 2024-04-20T21:15:30.901 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-ceph-common-2:18.2.0-0.el9.x86_64 9/32 2024-04-20T21:15:30.901 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-cephfs-2:18.2.0-0.el9.x86_64 10/32 2024-04-20T21:15:30.901 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-rados-2:18.2.0-0.el9.x86_64 11/32 2024-04-20T21:15:30.901 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-rbd-2:18.2.0-0.el9.x86_64 12/32 2024-04-20T21:15:30.901 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-rgw-2:18.2.0-0.el9.x86_64 13/32 2024-04-20T21:15:30.901 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2024-04-20T21:15:30.902 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2024-04-20T21:15:30.902 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/32 2024-04-20T21:15:30.902 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/32 2024-04-20T21:15:30.902 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/32 2024-04-20T21:15:30.902 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/32 2024-04-20T21:15:30.902 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2024-04-20T21:15:30.902 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 21/32 2024-04-20T21:15:30.902 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 22/32 2024-04-20T21:15:30.902 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 23/32 2024-04-20T21:15:30.902 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2024-04-20T21:15:30.902 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2024-04-20T21:15:30.902 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 26/32 2024-04-20T21:15:30.902 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : re2-1:20211101-3.el9.x86_64 27/32 2024-04-20T21:15:30.903 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2024-04-20T21:15:30.903 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2024-04-20T21:15:30.904 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librados2-2:18.2.0-0.el9.x86_64 29/32 2024-04-20T21:15:30.904 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2024-04-20T21:15:30.904 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librbd1-2:18.2.0-0.el9.x86_64 31/32 2024-04-20T21:15:30.936 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/32 2024-04-20T21:15:30.976 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 16/32 2024-04-20T21:15:31.035 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2024-04-20T21:15:31.083 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2024-04-20T21:15:31.487 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2024-04-20T21:15:31.523 INFO:teuthology.orchestra.run.smithi062.stdout: Upgrading : librbd1-2:18.2.0-0.el9.x86_64 20/32 2024-04-20T21:15:31.647 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: librbd1-2:18.2.0-0.el9.x86_64 20/32 2024-04-20T21:15:31.653 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2024-04-20T21:15:31.654 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-20T21:15:31.654 INFO:teuthology.orchestra.run.smithi189.stdout:Upgraded: 2024-04-20T21:15:31.654 INFO:teuthology.orchestra.run.smithi189.stdout: librados2-2:18.2.0-0.el9.x86_64 librbd1-2:18.2.0-0.el9.x86_64 2024-04-20T21:15:31.654 INFO:teuthology.orchestra.run.smithi189.stdout:Installed: 2024-04-20T21:15:31.654 INFO:teuthology.orchestra.run.smithi189.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-20T21:15:31.654 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-base-2:18.2.0-0.el9.x86_64 2024-04-20T21:15:31.654 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-common-2:18.2.0-0.el9.x86_64 2024-04-20T21:15:31.654 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-radosgw-2:18.2.0-0.el9.x86_64 2024-04-20T21:15:31.654 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-selinux-2:18.2.0-0.el9.x86_64 2024-04-20T21:15:31.654 INFO:teuthology.orchestra.run.smithi189.stdout: fmt-8.1.1-5.el9.x86_64 2024-04-20T21:15:31.654 INFO:teuthology.orchestra.run.smithi189.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-20T21:15:31.654 INFO:teuthology.orchestra.run.smithi189.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-20T21:15:31.654 INFO:teuthology.orchestra.run.smithi189.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-20T21:15:31.655 INFO:teuthology.orchestra.run.smithi189.stdout: libcephfs2-2:18.2.0-0.el9.x86_64 2024-04-20T21:15:31.655 INFO:teuthology.orchestra.run.smithi189.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-20T21:15:31.655 INFO:teuthology.orchestra.run.smithi189.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-20T21:15:31.655 INFO:teuthology.orchestra.run.smithi189.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-20T21:15:31.655 INFO:teuthology.orchestra.run.smithi189.stdout: libradosstriper1-2:18.2.0-0.el9.x86_64 2024-04-20T21:15:31.655 INFO:teuthology.orchestra.run.smithi189.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-20T21:15:31.655 INFO:teuthology.orchestra.run.smithi189.stdout: librgw2-2:18.2.0-0.el9.x86_64 2024-04-20T21:15:31.655 INFO:teuthology.orchestra.run.smithi189.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-20T21:15:31.655 INFO:teuthology.orchestra.run.smithi189.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-20T21:15:31.655 INFO:teuthology.orchestra.run.smithi189.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-20T21:15:31.655 INFO:teuthology.orchestra.run.smithi189.stdout: python3-ceph-argparse-2:18.2.0-0.el9.x86_64 2024-04-20T21:15:31.655 INFO:teuthology.orchestra.run.smithi189.stdout: python3-ceph-common-2:18.2.0-0.el9.x86_64 2024-04-20T21:15:31.655 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cephfs-2:18.2.0-0.el9.x86_64 2024-04-20T21:15:31.655 INFO:teuthology.orchestra.run.smithi189.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-20T21:15:31.656 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rados-2:18.2.0-0.el9.x86_64 2024-04-20T21:15:31.656 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rbd-2:18.2.0-0.el9.x86_64 2024-04-20T21:15:31.656 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rgw-2:18.2.0-0.el9.x86_64 2024-04-20T21:15:31.656 INFO:teuthology.orchestra.run.smithi189.stdout: re2-1:20211101-3.el9.x86_64 2024-04-20T21:15:31.656 INFO:teuthology.orchestra.run.smithi189.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-20T21:15:31.656 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-20T21:15:31.656 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-04-20T21:15:31.681 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-rbd-2:18.2.0-0.el9.x86_64 21/32 2024-04-20T21:15:31.741 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2024-04-20T21:15:32.063 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install ceph-test 2024-04-20T21:15:32.250 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 23/32 2024-04-20T21:15:32.284 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : librgw2-2:18.2.0-0.el9.x86_64 24/32 2024-04-20T21:15:32.385 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: librgw2-2:18.2.0-0.el9.x86_64 24/32 2024-04-20T21:15:32.490 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-rgw-2:18.2.0-0.el9.x86_64 25/32 2024-04-20T21:15:32.528 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-ceph-common-2:18.2.0-0.el9.x86_64 26/32 2024-04-20T21:15:32.830 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:00:35 ago on Sat 20 Apr 2024 09:14:57 PM UTC. 2024-04-20T21:15:32.967 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-04-20T21:15:32.968 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-20T21:15:32.968 INFO:teuthology.orchestra.run.smithi189.stdout: Package Architecture Version Repository Size 2024-04-20T21:15:32.968 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-20T21:15:32.968 INFO:teuthology.orchestra.run.smithi189.stdout:Installing: 2024-04-20T21:15:32.968 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-test x86_64 2:18.2.0-0.el9 ceph 40 M 2024-04-20T21:15:32.968 INFO:teuthology.orchestra.run.smithi189.stdout:Installing dependencies: 2024-04-20T21:15:32.968 INFO:teuthology.orchestra.run.smithi189.stdout: libcephsqlite x86_64 2:18.2.0-0.el9 ceph 161 k 2024-04-20T21:15:32.968 INFO:teuthology.orchestra.run.smithi189.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-20T21:15:32.969 INFO:teuthology.orchestra.run.smithi189.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-20T21:15:32.969 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-20T21:15:32.969 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction Summary 2024-04-20T21:15:32.969 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-20T21:15:32.969 INFO:teuthology.orchestra.run.smithi189.stdout:Install 4 Packages 2024-04-20T21:15:32.969 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-20T21:15:32.969 INFO:teuthology.orchestra.run.smithi189.stdout:Total download size: 40 M 2024-04-20T21:15:32.969 INFO:teuthology.orchestra.run.smithi189.stdout:Installed size: 165 M 2024-04-20T21:15:32.970 INFO:teuthology.orchestra.run.smithi189.stdout:Downloading Packages: 2024-04-20T21:15:33.290 INFO:teuthology.orchestra.run.smithi189.stdout:(1/4): libcephsqlite-18.2.0-0.el9.x86_64.rpm 1.1 MB/s | 161 kB 00:00 2024-04-20T21:15:33.340 INFO:teuthology.orchestra.run.smithi189.stdout:(2/4): socat-1.7.4.1-5.el9.x86_64.rpm 1.6 MB/s | 305 kB 00:00 2024-04-20T21:15:33.392 INFO:teuthology.orchestra.run.smithi189.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 631 kB/s | 64 kB 00:00 2024-04-20T21:15:34.049 INFO:teuthology.orchestra.run.smithi189.stdout:(4/4): ceph-test-18.2.0-0.el9.x86_64.rpm 44 MB/s | 40 MB 00:00 2024-04-20T21:15:34.052 INFO:teuthology.orchestra.run.smithi189.stdout:-------------------------------------------------------------------------------- 2024-04-20T21:15:34.052 INFO:teuthology.orchestra.run.smithi189.stdout:Total 37 MB/s | 40 MB 00:01 2024-04-20T21:15:34.107 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction check 2024-04-20T21:15:34.132 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction check succeeded. 2024-04-20T21:15:34.132 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction test 2024-04-20T21:15:34.568 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction test succeeded. 2024-04-20T21:15:34.569 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction 2024-04-20T21:15:35.192 INFO:teuthology.orchestra.run.smithi189.stdout: Preparing : 1/1 2024-04-20T21:15:35.261 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-common-2:18.2.0-0.el9.x86_64 27/32 2024-04-20T21:15:35.278 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-04-20T21:15:35.299 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-common-2:18.2.0-0.el9.x86_64 27/32 2024-04-20T21:15:35.346 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/4 2024-04-20T21:15:35.376 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libcephsqlite-2:18.2.0-0.el9.x86_64 3/4 2024-04-20T21:15:36.003 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-common-2:18.2.0-0.el9.x86_64 27/32 2024-04-20T21:15:36.052 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-base-2:18.2.0-0.el9.x86_64 28/32 2024-04-20T21:15:36.219 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-base-2:18.2.0-0.el9.x86_64 28/32 2024-04-20T21:15:36.219 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-20T21:15:36.219 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-20T21:15:36.220 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:15:36.262 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-selinux-2:18.2.0-0.el9.x86_64 29/32 2024-04-20T21:15:39.577 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: libcephsqlite-2:18.2.0-0.el9.x86_64 3/4 2024-04-20T21:15:39.772 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-test-2:18.2.0-0.el9.x86_64 4/4 2024-04-20T21:15:41.485 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-test-2:18.2.0-0.el9.x86_64 4/4 2024-04-20T21:15:41.485 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-test-2:18.2.0-0.el9.x86_64 1/4 2024-04-20T21:15:41.485 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libcephsqlite-2:18.2.0-0.el9.x86_64 2/4 2024-04-20T21:15:41.485 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/4 2024-04-20T21:15:41.780 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-04-20T21:15:41.780 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-20T21:15:41.780 INFO:teuthology.orchestra.run.smithi189.stdout:Installed: 2024-04-20T21:15:41.780 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-test-2:18.2.0-0.el9.x86_64 libcephsqlite-2:18.2.0-0.el9.x86_64 2024-04-20T21:15:41.780 INFO:teuthology.orchestra.run.smithi189.stdout: socat-1.7.4.1-5.el9.x86_64 xmlstarlet-1.6.1-20.el9.x86_64 2024-04-20T21:15:41.780 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-20T21:15:41.780 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-04-20T21:15:42.086 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install ceph 2024-04-20T21:15:42.899 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:00:45 ago on Sat 20 Apr 2024 09:14:57 PM UTC. 2024-04-20T21:15:43.033 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-04-20T21:15:43.036 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-20T21:15:43.036 INFO:teuthology.orchestra.run.smithi189.stdout: Package Arch Version Repository Size 2024-04-20T21:15:43.036 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-20T21:15:43.036 INFO:teuthology.orchestra.run.smithi189.stdout:Installing: 2024-04-20T21:15:43.036 INFO:teuthology.orchestra.run.smithi189.stdout: ceph x86_64 2:18.2.0-0.el9 ceph 6.4 k 2024-04-20T21:15:43.036 INFO:teuthology.orchestra.run.smithi189.stdout:Installing dependencies: 2024-04-20T21:15:43.037 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mds x86_64 2:18.2.0-0.el9 ceph 2.1 M 2024-04-20T21:15:43.037 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr x86_64 2:18.2.0-0.el9 ceph 1.4 M 2024-04-20T21:15:43.037 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-modules-core noarch 2:18.2.0-0.el9 ceph-noarch 240 k 2024-04-20T21:15:43.037 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mon x86_64 2:18.2.0-0.el9 ceph 4.4 M 2024-04-20T21:15:43.037 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-osd x86_64 2:18.2.0-0.el9 ceph 18 M 2024-04-20T21:15:43.037 INFO:teuthology.orchestra.run.smithi189.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-20T21:15:43.037 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-20T21:15:43.037 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-20T21:15:43.037 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-20T21:15:43.037 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-20T21:15:43.037 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-20T21:15:43.037 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-20T21:15:43.037 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-20T21:15:43.038 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-20T21:15:43.038 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-20T21:15:43.038 INFO:teuthology.orchestra.run.smithi189.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-20T21:15:43.038 INFO:teuthology.orchestra.run.smithi189.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-20T21:15:43.038 INFO:teuthology.orchestra.run.smithi189.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-20T21:15:43.038 INFO:teuthology.orchestra.run.smithi189.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-20T21:15:43.038 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-20T21:15:43.038 INFO:teuthology.orchestra.run.smithi189.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-20T21:15:43.038 INFO:teuthology.orchestra.run.smithi189.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-20T21:15:43.038 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-20T21:15:43.038 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-20T21:15:43.038 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-20T21:15:43.038 INFO:teuthology.orchestra.run.smithi189.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-20T21:15:43.039 INFO:teuthology.orchestra.run.smithi189.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-20T21:15:43.039 INFO:teuthology.orchestra.run.smithi189.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-20T21:15:43.039 INFO:teuthology.orchestra.run.smithi189.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-20T21:15:43.039 INFO:teuthology.orchestra.run.smithi189.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-20T21:15:43.039 INFO:teuthology.orchestra.run.smithi189.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-20T21:15:43.039 INFO:teuthology.orchestra.run.smithi189.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-20T21:15:43.039 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-20T21:15:43.039 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction Summary 2024-04-20T21:15:43.039 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-20T21:15:43.039 INFO:teuthology.orchestra.run.smithi189.stdout:Install 33 Packages 2024-04-20T21:15:43.039 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-20T21:15:43.040 INFO:teuthology.orchestra.run.smithi189.stdout:Total download size: 30 M 2024-04-20T21:15:43.040 INFO:teuthology.orchestra.run.smithi189.stdout:Installed size: 109 M 2024-04-20T21:15:43.040 INFO:teuthology.orchestra.run.smithi189.stdout:Downloading Packages: 2024-04-20T21:15:43.927 INFO:teuthology.orchestra.run.smithi189.stdout:(1/33): ceph-18.2.0-0.el9.x86_64.rpm 32 kB/s | 6.4 kB 00:00 2024-04-20T21:15:44.003 INFO:teuthology.orchestra.run.smithi189.stdout:(2/33): ceph-mgr-18.2.0-0.el9.x86_64.rpm 5.2 MB/s | 1.4 MB 00:00 2024-04-20T21:15:44.070 INFO:teuthology.orchestra.run.smithi189.stdout:(3/33): ceph-mds-18.2.0-0.el9.x86_64.rpm 6.1 MB/s | 2.1 MB 00:00 2024-04-20T21:15:44.104 INFO:teuthology.orchestra.run.smithi189.stdout:(4/33): ceph-mgr-modules-core-18.2.0-0.el9.noar 7.1 MB/s | 240 kB 00:00 2024-04-20T21:15:44.237 INFO:teuthology.orchestra.run.smithi189.stdout:(5/33): ceph-mon-18.2.0-0.el9.x86_64.rpm 14 MB/s | 4.4 MB 00:00 2024-04-20T21:15:44.296 INFO:teuthology.orchestra.run.smithi189.stdout:(6/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.3 MB/s | 253 kB 00:00 2024-04-20T21:15:44.387 INFO:teuthology.orchestra.run.smithi189.stdout:(7/33): python3-ply-3.11-14.el9.noarch.rpm 1.1 MB/s | 106 kB 00:00 2024-04-20T21:15:44.646 INFO:teuthology.orchestra.run.smithi189.stdout:(8/33): ceph-osd-18.2.0-0.el9.x86_64.rpm 27 MB/s | 18 MB 00:00 2024-04-20T21:15:44.671 INFO:teuthology.orchestra.run.smithi189.stdout:(9/33): python3-pycparser-2.20-6.el9.noarch.rpm 477 kB/s | 135 kB 00:00 2024-04-20T21:15:44.755 INFO:teuthology.orchestra.run.smithi189.stdout:(10/33): python3-cryptography-36.0.1-4.el9.x86_ 2.4 MB/s | 1.2 MB 00:00 2024-04-20T21:15:44.780 INFO:teuthology.orchestra.run.smithi189.stdout:(11/33): python3-requests-2.25.1-8.el9.noarch.r 940 kB/s | 125 kB 00:00 2024-04-20T21:15:44.814 INFO:teuthology.orchestra.run.smithi189.stdout:(12/33): python3-urllib3-1.26.5-5.el9.noarch.rp 1.5 MB/s | 215 kB 00:00 2024-04-20T21:15:45.047 INFO:teuthology.orchestra.run.smithi189.stdout:(13/33): python3-markupsafe-1.1.1-12.el9.x86_64 131 kB/s | 35 kB 00:00 2024-04-20T21:15:45.164 INFO:teuthology.orchestra.run.smithi189.stdout:(14/33): python3-pytz-2021.1-5.el9.noarch.rpm 146 kB/s | 51 kB 00:00 2024-04-20T21:15:45.231 INFO:teuthology.orchestra.run.smithi189.stdout:(15/33): python3-toml-0.10.2-6.el9.noarch.rpm 227 kB/s | 42 kB 00:00 2024-04-20T21:15:45.290 INFO:teuthology.orchestra.run.smithi189.stdout:(16/33): python3-mako-1.1.4-6.el9.noarch.rpm 323 kB/s | 172 kB 00:00 2024-04-20T21:15:45.515 INFO:teuthology.orchestra.run.smithi189.stdout:(17/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 124 kB/s | 43 kB 00:00 2024-04-20T21:15:45.687 INFO:teuthology.orchestra.run.smithi189.stdout:(18/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 378 kB/s | 172 kB 00:00 2024-04-20T21:15:45.716 INFO:teuthology.orchestra.run.smithi189.stdout:(19/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 53 kB/s | 11 kB 00:00 2024-04-20T21:15:45.891 INFO:teuthology.orchestra.run.smithi189.stdout:(20/33): python3-cherrypy-18.6.1-2.el9.noarch.r 597 kB/s | 358 kB 00:00 2024-04-20T21:15:45.950 INFO:teuthology.orchestra.run.smithi189.stdout:(21/33): python3-jaraco-classes-3.2.1-5.el9.noa 68 kB/s | 18 kB 00:00 2024-04-20T21:15:45.983 INFO:teuthology.orchestra.run.smithi189.stdout:(22/33): python3-jaraco-collections-3.0.0-8.el9 87 kB/s | 23 kB 00:00 2024-04-20T21:15:46.167 INFO:teuthology.orchestra.run.smithi189.stdout:(23/33): python3-jaraco-functools-3.5.0-2.el9.n 71 kB/s | 19 kB 00:00 2024-04-20T21:15:46.217 INFO:teuthology.orchestra.run.smithi189.stdout:(24/33): python3-jaraco-text-3.2.0-6.el9.noarch 74 kB/s | 20 kB 00:00 2024-04-20T21:15:46.342 INFO:teuthology.orchestra.run.smithi189.stdout:(25/33): python3-logutils-0.3.5-21.el9.noarch.r 129 kB/s | 46 kB 00:00 2024-04-20T21:15:46.509 INFO:teuthology.orchestra.run.smithi189.stdout:(26/33): python3-more-itertools-8.12.0-2.el9.no 231 kB/s | 79 kB 00:00 2024-04-20T21:15:46.609 INFO:teuthology.orchestra.run.smithi189.stdout:(27/33): python3-portend-3.1.0-2.el9.noarch.rpm 62 kB/s | 16 kB 00:00 2024-04-20T21:15:46.726 INFO:teuthology.orchestra.run.smithi189.stdout:(28/33): python3-pecan-1.4.2-3.el9.noarch.rpm 535 kB/s | 272 kB 00:00 2024-04-20T21:15:46.885 INFO:teuthology.orchestra.run.smithi189.stdout:(29/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 240 kB/s | 90 kB 00:00 2024-04-20T21:15:46.910 INFO:teuthology.orchestra.run.smithi189.stdout:(30/33): python3-tempora-5.0.0-2.el9.noarch.rpm 119 kB/s | 36 kB 00:00 2024-04-20T21:15:47.178 INFO:teuthology.orchestra.run.smithi189.stdout:(31/33): python3-zc-lockfile-2.0-10.el9.noarch. 75 kB/s | 20 kB 00:00 2024-04-20T21:15:47.244 INFO:teuthology.orchestra.run.smithi189.stdout:(32/33): python3-webob-1.8.7-6.el9.noarch.rpm 444 kB/s | 230 kB 00:00 2024-04-20T21:15:47.502 INFO:teuthology.orchestra.run.smithi189.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 723 kB/s | 427 kB 00:00 2024-04-20T21:15:47.509 INFO:teuthology.orchestra.run.smithi189.stdout:-------------------------------------------------------------------------------- 2024-04-20T21:15:47.522 INFO:teuthology.orchestra.run.smithi189.stdout:Total 6.7 MB/s | 30 MB 00:04 2024-04-20T21:15:47.734 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-selinux-2:18.2.0-0.el9.x86_64 29/32 2024-04-20T21:15:47.734 INFO:teuthology.orchestra.run.smithi062.stdout:skipping the directory /sys 2024-04-20T21:15:47.734 INFO:teuthology.orchestra.run.smithi062.stdout:skipping the directory /proc 2024-04-20T21:15:47.734 INFO:teuthology.orchestra.run.smithi062.stdout:skipping the directory /mnt 2024-04-20T21:15:47.734 INFO:teuthology.orchestra.run.smithi062.stdout:skipping the directory /var/tmp 2024-04-20T21:15:47.734 INFO:teuthology.orchestra.run.smithi062.stdout:skipping the directory /home 2024-04-20T21:15:47.734 INFO:teuthology.orchestra.run.smithi062.stdout:skipping the directory /root 2024-04-20T21:15:47.735 INFO:teuthology.orchestra.run.smithi062.stdout:skipping the directory /tmp 2024-04-20T21:15:47.735 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:15:47.875 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction check 2024-04-20T21:15:47.961 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction check succeeded. 2024-04-20T21:15:47.961 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction test 2024-04-20T21:15:48.508 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction test succeeded. 2024-04-20T21:15:48.509 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction 2024-04-20T21:15:48.629 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-radosgw-2:18.2.0-0.el9.x86_64 30/32 2024-04-20T21:15:48.713 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-radosgw-2:18.2.0-0.el9.x86_64 30/32 2024-04-20T21:15:48.713 INFO:teuthology.orchestra.run.smithi062.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-20T21:15:48.713 INFO:teuthology.orchestra.run.smithi062.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-20T21:15:48.713 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-20T21:15:48.713 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-20T21:15:48.714 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:15:48.719 INFO:teuthology.orchestra.run.smithi062.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-04-20T21:15:48.839 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-04-20T21:15:48.839 INFO:teuthology.orchestra.run.smithi062.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2024-04-20T21:15:49.796 INFO:teuthology.orchestra.run.smithi189.stdout: Preparing : 1/1 2024-04-20T21:15:49.856 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-04-20T21:15:49.894 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-04-20T21:15:49.936 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-04-20T21:15:50.073 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-04-20T21:15:50.172 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-04-20T21:15:50.229 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-04-20T21:15:50.279 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-04-20T21:15:50.317 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/33 2024-04-20T21:15:50.353 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/33 2024-04-20T21:15:50.396 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/33 2024-04-20T21:15:50.458 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/33 2024-04-20T21:15:50.468 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2024-04-20T21:15:50.468 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-base-2:18.2.0-0.el9.x86_64 1/32 2024-04-20T21:15:50.468 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-common-2:18.2.0-0.el9.x86_64 2/32 2024-04-20T21:15:50.468 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-radosgw-2:18.2.0-0.el9.x86_64 3/32 2024-04-20T21:15:50.469 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-selinux-2:18.2.0-0.el9.x86_64 4/32 2024-04-20T21:15:50.469 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libcephfs2-2:18.2.0-0.el9.x86_64 5/32 2024-04-20T21:15:50.469 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libradosstriper1-2:18.2.0-0.el9.x86_64 6/32 2024-04-20T21:15:50.469 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : librgw2-2:18.2.0-0.el9.x86_64 7/32 2024-04-20T21:15:50.469 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-ceph-argparse-2:18.2.0-0.el9.x86_64 8/32 2024-04-20T21:15:50.469 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-ceph-common-2:18.2.0-0.el9.x86_64 9/32 2024-04-20T21:15:50.469 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-cephfs-2:18.2.0-0.el9.x86_64 10/32 2024-04-20T21:15:50.469 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-rados-2:18.2.0-0.el9.x86_64 11/32 2024-04-20T21:15:50.469 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-rbd-2:18.2.0-0.el9.x86_64 12/32 2024-04-20T21:15:50.469 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-rgw-2:18.2.0-0.el9.x86_64 13/32 2024-04-20T21:15:50.469 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2024-04-20T21:15:50.469 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2024-04-20T21:15:50.469 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/32 2024-04-20T21:15:50.469 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/32 2024-04-20T21:15:50.470 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/32 2024-04-20T21:15:50.470 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/32 2024-04-20T21:15:50.470 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2024-04-20T21:15:50.470 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 21/32 2024-04-20T21:15:50.470 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 22/32 2024-04-20T21:15:50.470 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 23/32 2024-04-20T21:15:50.470 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2024-04-20T21:15:50.470 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2024-04-20T21:15:50.470 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 26/32 2024-04-20T21:15:50.470 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : re2-1:20211101-3.el9.x86_64 27/32 2024-04-20T21:15:50.470 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2024-04-20T21:15:50.470 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : librados2-2:18.2.0-0.el9.x86_64 29/32 2024-04-20T21:15:50.470 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2024-04-20T21:15:50.470 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : librbd1-2:18.2.0-0.el9.x86_64 31/32 2024-04-20T21:15:50.518 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/33 2024-04-20T21:15:50.556 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/33 2024-04-20T21:15:50.612 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/33 2024-04-20T21:15:50.740 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 15/33 2024-04-20T21:15:50.947 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 16/33 2024-04-20T21:15:51.055 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 17/33 2024-04-20T21:15:51.131 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 18/33 2024-04-20T21:15:51.174 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 19/33 2024-04-20T21:15:51.239 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-ply-3.11-14.el9.noarch 20/33 2024-04-20T21:15:51.268 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2024-04-20T21:15:51.268 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:15:51.269 INFO:teuthology.orchestra.run.smithi062.stdout:Upgraded: 2024-04-20T21:15:51.269 INFO:teuthology.orchestra.run.smithi062.stdout: librados2-2:18.2.0-0.el9.x86_64 librbd1-2:18.2.0-0.el9.x86_64 2024-04-20T21:15:51.269 INFO:teuthology.orchestra.run.smithi062.stdout:Installed: 2024-04-20T21:15:51.269 INFO:teuthology.orchestra.run.smithi062.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-20T21:15:51.269 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-base-2:18.2.0-0.el9.x86_64 2024-04-20T21:15:51.269 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-common-2:18.2.0-0.el9.x86_64 2024-04-20T21:15:51.269 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-radosgw-2:18.2.0-0.el9.x86_64 2024-04-20T21:15:51.269 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-selinux-2:18.2.0-0.el9.x86_64 2024-04-20T21:15:51.269 INFO:teuthology.orchestra.run.smithi062.stdout: fmt-8.1.1-5.el9.x86_64 2024-04-20T21:15:51.269 INFO:teuthology.orchestra.run.smithi062.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-20T21:15:51.269 INFO:teuthology.orchestra.run.smithi062.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-20T21:15:51.270 INFO:teuthology.orchestra.run.smithi062.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-20T21:15:51.270 INFO:teuthology.orchestra.run.smithi062.stdout: libcephfs2-2:18.2.0-0.el9.x86_64 2024-04-20T21:15:51.270 INFO:teuthology.orchestra.run.smithi062.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-20T21:15:51.270 INFO:teuthology.orchestra.run.smithi062.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-20T21:15:51.270 INFO:teuthology.orchestra.run.smithi062.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-20T21:15:51.270 INFO:teuthology.orchestra.run.smithi062.stdout: libradosstriper1-2:18.2.0-0.el9.x86_64 2024-04-20T21:15:51.270 INFO:teuthology.orchestra.run.smithi062.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-20T21:15:51.270 INFO:teuthology.orchestra.run.smithi062.stdout: librgw2-2:18.2.0-0.el9.x86_64 2024-04-20T21:15:51.270 INFO:teuthology.orchestra.run.smithi062.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-20T21:15:51.270 INFO:teuthology.orchestra.run.smithi062.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-20T21:15:51.270 INFO:teuthology.orchestra.run.smithi062.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-20T21:15:51.270 INFO:teuthology.orchestra.run.smithi062.stdout: python3-ceph-argparse-2:18.2.0-0.el9.x86_64 2024-04-20T21:15:51.270 INFO:teuthology.orchestra.run.smithi062.stdout: python3-ceph-common-2:18.2.0-0.el9.x86_64 2024-04-20T21:15:51.270 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cephfs-2:18.2.0-0.el9.x86_64 2024-04-20T21:15:51.271 INFO:teuthology.orchestra.run.smithi062.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-20T21:15:51.271 INFO:teuthology.orchestra.run.smithi062.stdout: python3-rados-2:18.2.0-0.el9.x86_64 2024-04-20T21:15:51.271 INFO:teuthology.orchestra.run.smithi062.stdout: python3-rbd-2:18.2.0-0.el9.x86_64 2024-04-20T21:15:51.271 INFO:teuthology.orchestra.run.smithi062.stdout: python3-rgw-2:18.2.0-0.el9.x86_64 2024-04-20T21:15:51.271 INFO:teuthology.orchestra.run.smithi062.stdout: re2-1:20211101-3.el9.x86_64 2024-04-20T21:15:51.271 INFO:teuthology.orchestra.run.smithi062.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-20T21:15:51.271 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:15:51.271 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2024-04-20T21:15:51.318 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 21/33 2024-04-20T21:15:51.573 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 22/33 2024-04-20T21:15:51.646 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 23/33 2024-04-20T21:15:51.724 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install ceph-test 2024-04-20T21:15:51.736 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 24/33 2024-04-20T21:15:51.855 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 25/33 2024-04-20T21:15:51.901 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 26/33 2024-04-20T21:15:52.046 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 27/33 2024-04-20T21:15:52.243 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mgr-modules-core-2:18.2.0-0.el9.noarch 28/33 2024-04-20T21:15:52.275 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mgr-2:18.2.0-0.el9.x86_64 29/33 2024-04-20T21:15:52.343 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-mgr-2:18.2.0-0.el9.x86_64 29/33 2024-04-20T21:15:52.343 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-20T21:15:52.343 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-20T21:15:52.343 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-20T21:15:52.343 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-20T21:15:52.343 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-20T21:15:52.513 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:00:32 ago on Sat 20 Apr 2024 09:15:20 PM UTC. 2024-04-20T21:15:52.648 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2024-04-20T21:15:52.649 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-04-20T21:15:52.649 INFO:teuthology.orchestra.run.smithi062.stdout: Package Architecture Version Repository Size 2024-04-20T21:15:52.649 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-04-20T21:15:52.649 INFO:teuthology.orchestra.run.smithi062.stdout:Installing: 2024-04-20T21:15:52.649 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-test x86_64 2:18.2.0-0.el9 ceph 40 M 2024-04-20T21:15:52.650 INFO:teuthology.orchestra.run.smithi062.stdout:Installing dependencies: 2024-04-20T21:15:52.650 INFO:teuthology.orchestra.run.smithi062.stdout: libcephsqlite x86_64 2:18.2.0-0.el9 ceph 161 k 2024-04-20T21:15:52.650 INFO:teuthology.orchestra.run.smithi062.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-20T21:15:52.650 INFO:teuthology.orchestra.run.smithi062.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-20T21:15:52.650 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:15:52.650 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction Summary 2024-04-20T21:15:52.650 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-04-20T21:15:52.650 INFO:teuthology.orchestra.run.smithi062.stdout:Install 4 Packages 2024-04-20T21:15:52.650 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:15:52.651 INFO:teuthology.orchestra.run.smithi062.stdout:Total download size: 40 M 2024-04-20T21:15:52.651 INFO:teuthology.orchestra.run.smithi062.stdout:Installed size: 165 M 2024-04-20T21:15:52.651 INFO:teuthology.orchestra.run.smithi062.stdout:Downloading Packages: 2024-04-20T21:15:53.065 INFO:teuthology.orchestra.run.smithi062.stdout:(1/4): libcephsqlite-18.2.0-0.el9.x86_64.rpm 1.3 MB/s | 161 kB 00:00 2024-04-20T21:15:53.132 INFO:teuthology.orchestra.run.smithi062.stdout:(2/4): socat-1.7.4.1-5.el9.x86_64.rpm 1.5 MB/s | 305 kB 00:00 2024-04-20T21:15:53.157 INFO:teuthology.orchestra.run.smithi062.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 695 kB/s | 64 kB 00:00 2024-04-20T21:15:53.800 INFO:teuthology.orchestra.run.smithi062.stdout:(4/4): ceph-test-18.2.0-0.el9.x86_64.rpm 46 MB/s | 40 MB 00:00 2024-04-20T21:15:53.803 INFO:teuthology.orchestra.run.smithi062.stdout:-------------------------------------------------------------------------------- 2024-04-20T21:15:53.804 INFO:teuthology.orchestra.run.smithi062.stdout:Total 35 MB/s | 40 MB 00:01 2024-04-20T21:15:53.870 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction check 2024-04-20T21:15:53.898 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction check succeeded. 2024-04-20T21:15:53.898 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction test 2024-04-20T21:15:54.052 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-osd-2:18.2.0-0.el9.x86_64 30/33 2024-04-20T21:15:54.134 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-osd-2:18.2.0-0.el9.x86_64 30/33 2024-04-20T21:15:54.134 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-20T21:15:54.135 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-20T21:15:54.135 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-20T21:15:54.135 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-20T21:15:54.135 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-20T21:15:54.326 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction test succeeded. 2024-04-20T21:15:54.327 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction 2024-04-20T21:15:54.636 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mon-2:18.2.0-0.el9.x86_64 31/33 2024-04-20T21:15:54.705 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-mon-2:18.2.0-0.el9.x86_64 31/33 2024-04-20T21:15:54.705 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-20T21:15:54.705 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-20T21:15:54.705 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-20T21:15:54.705 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-20T21:15:54.705 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-20T21:15:54.937 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mds-2:18.2.0-0.el9.x86_64 32/33 2024-04-20T21:15:54.950 INFO:teuthology.orchestra.run.smithi062.stdout: Preparing : 1/1 2024-04-20T21:15:55.003 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-mds-2:18.2.0-0.el9.x86_64 32/33 2024-04-20T21:15:55.003 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-20T21:15:55.003 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-20T21:15:55.003 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-20T21:15:55.004 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-20T21:15:55.004 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-20T21:15:55.085 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-04-20T21:15:55.095 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-2:18.2.0-0.el9.x86_64 33/33 2024-04-20T21:15:55.182 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/4 2024-04-20T21:15:55.217 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-2:18.2.0-0.el9.x86_64 33/33 2024-04-20T21:15:55.218 INFO:teuthology.orchestra.run.smithi189.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-04-20T21:15:55.218 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-20T21:15:55.226 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libcephsqlite-2:18.2.0-0.el9.x86_64 3/4 2024-04-20T21:15:56.959 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-2:18.2.0-0.el9.x86_64 1/33 2024-04-20T21:15:56.960 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mds-2:18.2.0-0.el9.x86_64 2/33 2024-04-20T21:15:56.960 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mgr-2:18.2.0-0.el9.x86_64 3/33 2024-04-20T21:15:56.960 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mon-2:18.2.0-0.el9.x86_64 4/33 2024-04-20T21:15:56.960 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-osd-2:18.2.0-0.el9.x86_64 5/33 2024-04-20T21:15:56.960 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mgr-modules-core-2:18.2.0-0.el9.noarch 6/33 2024-04-20T21:15:56.960 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/33 2024-04-20T21:15:56.960 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/33 2024-04-20T21:15:56.960 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/33 2024-04-20T21:15:56.960 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/33 2024-04-20T21:15:56.960 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/33 2024-04-20T21:15:56.960 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/33 2024-04-20T21:15:56.960 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/33 2024-04-20T21:15:56.960 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/33 2024-04-20T21:15:56.960 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 15/33 2024-04-20T21:15:56.961 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 16/33 2024-04-20T21:15:56.961 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-04-20T21:15:56.961 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-04-20T21:15:56.961 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-04-20T21:15:56.961 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-04-20T21:15:56.961 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-04-20T21:15:56.961 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-04-20T21:15:56.961 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-04-20T21:15:56.961 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-04-20T21:15:56.961 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-04-20T21:15:56.961 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-04-20T21:15:56.961 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-04-20T21:15:56.961 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-04-20T21:15:56.961 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-04-20T21:15:56.961 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-04-20T21:15:56.962 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-04-20T21:15:56.962 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-04-20T21:15:57.936 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-04-20T21:15:57.936 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-20T21:15:57.936 INFO:teuthology.orchestra.run.smithi189.stdout:Installed: 2024-04-20T21:15:57.936 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-2:18.2.0-0.el9.x86_64 2024-04-20T21:15:57.936 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mds-2:18.2.0-0.el9.x86_64 2024-04-20T21:15:57.936 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-2:18.2.0-0.el9.x86_64 2024-04-20T21:15:57.936 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-modules-core-2:18.2.0-0.el9.noarch 2024-04-20T21:15:57.936 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mon-2:18.2.0-0.el9.x86_64 2024-04-20T21:15:57.936 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-osd-2:18.2.0-0.el9.x86_64 2024-04-20T21:15:57.937 INFO:teuthology.orchestra.run.smithi189.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-20T21:15:57.937 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-20T21:15:57.937 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-20T21:15:57.937 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-20T21:15:57.937 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-20T21:15:57.937 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-20T21:15:57.937 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-20T21:15:57.937 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-20T21:15:57.937 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-20T21:15:57.937 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-20T21:15:57.937 INFO:teuthology.orchestra.run.smithi189.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-20T21:15:57.938 INFO:teuthology.orchestra.run.smithi189.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-20T21:15:57.938 INFO:teuthology.orchestra.run.smithi189.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-20T21:15:57.938 INFO:teuthology.orchestra.run.smithi189.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-20T21:15:57.938 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-20T21:15:57.938 INFO:teuthology.orchestra.run.smithi189.stdout: python3-ply-3.11-14.el9.noarch 2024-04-20T21:15:57.938 INFO:teuthology.orchestra.run.smithi189.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-20T21:15:57.938 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-20T21:15:57.938 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-20T21:15:57.938 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-20T21:15:57.938 INFO:teuthology.orchestra.run.smithi189.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-20T21:15:57.938 INFO:teuthology.orchestra.run.smithi189.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-20T21:15:57.938 INFO:teuthology.orchestra.run.smithi189.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-20T21:15:57.938 INFO:teuthology.orchestra.run.smithi189.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-20T21:15:57.938 INFO:teuthology.orchestra.run.smithi189.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-20T21:15:57.939 INFO:teuthology.orchestra.run.smithi189.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-20T21:15:57.939 INFO:teuthology.orchestra.run.smithi189.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-20T21:15:57.939 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-20T21:15:57.939 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-04-20T21:15:58.321 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install ceph-base 2024-04-20T21:15:59.089 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:01:02 ago on Sat 20 Apr 2024 09:14:57 PM UTC. 2024-04-20T21:15:59.169 INFO:teuthology.orchestra.run.smithi189.stdout:Package ceph-base-2:18.2.0-0.el9.x86_64 is already installed. 2024-04-20T21:15:59.211 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-04-20T21:15:59.212 INFO:teuthology.orchestra.run.smithi189.stdout:Nothing to do. 2024-04-20T21:15:59.212 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-04-20T21:15:59.297 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install cephadm 2024-04-20T21:15:59.857 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: libcephsqlite-2:18.2.0-0.el9.x86_64 3/4 2024-04-20T21:16:00.010 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:01:03 ago on Sat 20 Apr 2024 09:14:57 PM UTC. 2024-04-20T21:16:00.094 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-test-2:18.2.0-0.el9.x86_64 4/4 2024-04-20T21:16:00.132 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-04-20T21:16:00.133 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-20T21:16:00.133 INFO:teuthology.orchestra.run.smithi189.stdout: Package Architecture Version Repository Size 2024-04-20T21:16:00.133 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-20T21:16:00.133 INFO:teuthology.orchestra.run.smithi189.stdout:Installing: 2024-04-20T21:16:00.133 INFO:teuthology.orchestra.run.smithi189.stdout: cephadm noarch 2:18.2.0-0.el9 ceph-noarch 209 k 2024-04-20T21:16:00.133 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-20T21:16:00.133 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction Summary 2024-04-20T21:16:00.133 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-20T21:16:00.134 INFO:teuthology.orchestra.run.smithi189.stdout:Install 1 Package 2024-04-20T21:16:00.134 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-20T21:16:00.134 INFO:teuthology.orchestra.run.smithi189.stdout:Total download size: 209 k 2024-04-20T21:16:00.134 INFO:teuthology.orchestra.run.smithi189.stdout:Installed size: 200 k 2024-04-20T21:16:00.134 INFO:teuthology.orchestra.run.smithi189.stdout:Downloading Packages: 2024-04-20T21:16:00.211 INFO:teuthology.orchestra.run.smithi189.stdout:cephadm-18.2.0-0.el9.noarch.rpm 2.7 MB/s | 209 kB 00:00 2024-04-20T21:16:00.211 INFO:teuthology.orchestra.run.smithi189.stdout:-------------------------------------------------------------------------------- 2024-04-20T21:16:00.212 INFO:teuthology.orchestra.run.smithi189.stdout:Total 2.6 MB/s | 209 kB 00:00 2024-04-20T21:16:00.212 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction check 2024-04-20T21:16:00.218 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction check succeeded. 2024-04-20T21:16:00.219 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction test 2024-04-20T21:16:00.222 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction test succeeded. 2024-04-20T21:16:00.222 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction 2024-04-20T21:16:00.317 INFO:teuthology.orchestra.run.smithi189.stdout: Preparing : 1/1 2024-04-20T21:16:00.949 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: cephadm-2:18.2.0-0.el9.noarch 1/1 2024-04-20T21:16:01.088 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : cephadm-2:18.2.0-0.el9.noarch 1/1 2024-04-20T21:16:01.416 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: cephadm-2:18.2.0-0.el9.noarch 1/1 2024-04-20T21:16:01.718 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : cephadm-2:18.2.0-0.el9.noarch 1/1 2024-04-20T21:16:01.718 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-20T21:16:01.718 INFO:teuthology.orchestra.run.smithi189.stdout:Installed: 2024-04-20T21:16:01.718 INFO:teuthology.orchestra.run.smithi189.stdout: cephadm-2:18.2.0-0.el9.noarch 2024-04-20T21:16:01.718 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-20T21:16:01.718 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-04-20T21:16:02.029 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-test-2:18.2.0-0.el9.x86_64 4/4 2024-04-20T21:16:02.029 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-test-2:18.2.0-0.el9.x86_64 1/4 2024-04-20T21:16:02.029 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libcephsqlite-2:18.2.0-0.el9.x86_64 2/4 2024-04-20T21:16:02.029 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/4 2024-04-20T21:16:02.044 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install ceph-immutable-object-cache 2024-04-20T21:16:02.361 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-04-20T21:16:02.361 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:16:02.361 INFO:teuthology.orchestra.run.smithi062.stdout:Installed: 2024-04-20T21:16:02.361 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-test-2:18.2.0-0.el9.x86_64 libcephsqlite-2:18.2.0-0.el9.x86_64 2024-04-20T21:16:02.361 INFO:teuthology.orchestra.run.smithi062.stdout: socat-1.7.4.1-5.el9.x86_64 xmlstarlet-1.6.1-20.el9.x86_64 2024-04-20T21:16:02.361 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:16:02.361 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2024-04-20T21:16:02.829 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:01:05 ago on Sat 20 Apr 2024 09:14:57 PM UTC. 2024-04-20T21:16:02.873 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install ceph 2024-04-20T21:16:02.958 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-04-20T21:16:02.959 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-20T21:16:02.959 INFO:teuthology.orchestra.run.smithi189.stdout: Package Arch Version Repo Size 2024-04-20T21:16:02.959 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-20T21:16:02.959 INFO:teuthology.orchestra.run.smithi189.stdout:Installing: 2024-04-20T21:16:02.959 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-immutable-object-cache x86_64 2:18.2.0-0.el9 ceph 142 k 2024-04-20T21:16:02.959 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-20T21:16:02.960 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction Summary 2024-04-20T21:16:02.960 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-20T21:16:02.960 INFO:teuthology.orchestra.run.smithi189.stdout:Install 1 Package 2024-04-20T21:16:02.960 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-20T21:16:02.960 INFO:teuthology.orchestra.run.smithi189.stdout:Total download size: 142 k 2024-04-20T21:16:02.960 INFO:teuthology.orchestra.run.smithi189.stdout:Installed size: 399 k 2024-04-20T21:16:02.960 INFO:teuthology.orchestra.run.smithi189.stdout:Downloading Packages: 2024-04-20T21:16:03.042 INFO:teuthology.orchestra.run.smithi189.stdout:ceph-immutable-object-cache-18.2.0-0.el9.x86_64 1.7 MB/s | 142 kB 00:00 2024-04-20T21:16:03.042 INFO:teuthology.orchestra.run.smithi189.stdout:-------------------------------------------------------------------------------- 2024-04-20T21:16:03.042 INFO:teuthology.orchestra.run.smithi189.stdout:Total 1.7 MB/s | 142 kB 00:00 2024-04-20T21:16:03.042 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction check 2024-04-20T21:16:03.052 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction check succeeded. 2024-04-20T21:16:03.052 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction test 2024-04-20T21:16:03.117 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction test succeeded. 2024-04-20T21:16:03.117 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction 2024-04-20T21:16:03.270 INFO:teuthology.orchestra.run.smithi189.stdout: Preparing : 1/1 2024-04-20T21:16:03.342 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-immutable-object-cache-2:18.2.0-0.el9.x86_64 1/1 2024-04-20T21:16:03.412 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.0-0.el9.x86_64 1/1 2024-04-20T21:16:03.412 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-20T21:16:03.412 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-20T21:16:03.412 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-20T21:16:03.668 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:00:43 ago on Sat 20 Apr 2024 09:15:20 PM UTC. 2024-04-20T21:16:03.803 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2024-04-20T21:16:03.806 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-04-20T21:16:03.806 INFO:teuthology.orchestra.run.smithi062.stdout: Package Arch Version Repository Size 2024-04-20T21:16:03.806 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-04-20T21:16:03.806 INFO:teuthology.orchestra.run.smithi062.stdout:Installing: 2024-04-20T21:16:03.806 INFO:teuthology.orchestra.run.smithi062.stdout: ceph x86_64 2:18.2.0-0.el9 ceph 6.4 k 2024-04-20T21:16:03.806 INFO:teuthology.orchestra.run.smithi062.stdout:Installing dependencies: 2024-04-20T21:16:03.806 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mds x86_64 2:18.2.0-0.el9 ceph 2.1 M 2024-04-20T21:16:03.806 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr x86_64 2:18.2.0-0.el9 ceph 1.4 M 2024-04-20T21:16:03.806 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-modules-core noarch 2:18.2.0-0.el9 ceph-noarch 240 k 2024-04-20T21:16:03.806 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mon x86_64 2:18.2.0-0.el9 ceph 4.4 M 2024-04-20T21:16:03.806 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-osd x86_64 2:18.2.0-0.el9 ceph 18 M 2024-04-20T21:16:03.806 INFO:teuthology.orchestra.run.smithi062.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-20T21:16:03.807 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-20T21:16:03.807 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-20T21:16:03.807 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-20T21:16:03.807 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-20T21:16:03.807 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-20T21:16:03.807 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-20T21:16:03.807 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-20T21:16:03.807 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-20T21:16:03.807 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-20T21:16:03.807 INFO:teuthology.orchestra.run.smithi062.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-20T21:16:03.807 INFO:teuthology.orchestra.run.smithi062.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-20T21:16:03.807 INFO:teuthology.orchestra.run.smithi062.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-20T21:16:03.808 INFO:teuthology.orchestra.run.smithi062.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-20T21:16:03.808 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-20T21:16:03.808 INFO:teuthology.orchestra.run.smithi062.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-20T21:16:03.808 INFO:teuthology.orchestra.run.smithi062.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-20T21:16:03.808 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-20T21:16:03.808 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-20T21:16:03.808 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-20T21:16:03.808 INFO:teuthology.orchestra.run.smithi062.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-20T21:16:03.808 INFO:teuthology.orchestra.run.smithi062.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-20T21:16:03.808 INFO:teuthology.orchestra.run.smithi062.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-20T21:16:03.808 INFO:teuthology.orchestra.run.smithi062.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-20T21:16:03.809 INFO:teuthology.orchestra.run.smithi062.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-20T21:16:03.809 INFO:teuthology.orchestra.run.smithi062.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-20T21:16:03.809 INFO:teuthology.orchestra.run.smithi062.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-20T21:16:03.809 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:16:03.809 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction Summary 2024-04-20T21:16:03.809 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-04-20T21:16:03.809 INFO:teuthology.orchestra.run.smithi062.stdout:Install 33 Packages 2024-04-20T21:16:03.809 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:16:03.810 INFO:teuthology.orchestra.run.smithi062.stdout:Total download size: 30 M 2024-04-20T21:16:03.810 INFO:teuthology.orchestra.run.smithi062.stdout:Installed size: 109 M 2024-04-20T21:16:03.810 INFO:teuthology.orchestra.run.smithi062.stdout:Downloading Packages: 2024-04-20T21:16:04.650 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-immutable-object-cache-2:18.2.0-0.el9.x86_64 1/1 2024-04-20T21:16:04.651 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-20T21:16:04.651 INFO:teuthology.orchestra.run.smithi189.stdout:Installed: 2024-04-20T21:16:04.651 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-immutable-object-cache-2:18.2.0-0.el9.x86_64 2024-04-20T21:16:04.651 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-20T21:16:04.651 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-04-20T21:16:04.787 INFO:teuthology.orchestra.run.smithi062.stdout:(1/33): ceph-18.2.0-0.el9.x86_64.rpm 35 kB/s | 6.4 kB 00:00 2024-04-20T21:16:04.862 INFO:teuthology.orchestra.run.smithi062.stdout:(2/33): ceph-mgr-18.2.0-0.el9.x86_64.rpm 5.6 MB/s | 1.4 MB 00:00 2024-04-20T21:16:04.906 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install ceph-mgr 2024-04-20T21:16:04.937 INFO:teuthology.orchestra.run.smithi062.stdout:(3/33): ceph-mds-18.2.0-0.el9.x86_64.rpm 6.3 MB/s | 2.1 MB 00:00 2024-04-20T21:16:04.979 INFO:teuthology.orchestra.run.smithi062.stdout:(4/33): ceph-mgr-modules-core-18.2.0-0.el9.noar 5.7 MB/s | 240 kB 00:00 2024-04-20T21:16:05.229 INFO:teuthology.orchestra.run.smithi062.stdout:(5/33): ceph-mon-18.2.0-0.el9.x86_64.rpm 10 MB/s | 4.4 MB 00:00 2024-04-20T21:16:05.321 INFO:teuthology.orchestra.run.smithi062.stdout:(6/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 743 kB/s | 253 kB 00:00 2024-04-20T21:16:05.454 INFO:teuthology.orchestra.run.smithi062.stdout:(7/33): python3-ply-3.11-14.el9.noarch.rpm 814 kB/s | 106 kB 00:00 2024-04-20T21:16:05.538 INFO:teuthology.orchestra.run.smithi062.stdout:(8/33): python3-cryptography-36.0.1-4.el9.x86_6 4.0 MB/s | 1.2 MB 00:00 2024-04-20T21:16:05.630 INFO:teuthology.orchestra.run.smithi062.stdout:(9/33): python3-pycparser-2.20-6.el9.noarch.rpm 777 kB/s | 135 kB 00:00 2024-04-20T21:16:05.678 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:01:08 ago on Sat 20 Apr 2024 09:14:57 PM UTC. 2024-04-20T21:16:05.716 INFO:teuthology.orchestra.run.smithi062.stdout:(10/33): python3-requests-2.25.1-8.el9.noarch.r 710 kB/s | 125 kB 00:00 2024-04-20T21:16:05.759 INFO:teuthology.orchestra.run.smithi189.stdout:Package ceph-mgr-2:18.2.0-0.el9.x86_64 is already installed. 2024-04-20T21:16:05.801 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-04-20T21:16:05.802 INFO:teuthology.orchestra.run.smithi189.stdout:Nothing to do. 2024-04-20T21:16:05.802 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-04-20T21:16:05.885 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install ceph-mgr-dashboard 2024-04-20T21:16:05.964 INFO:teuthology.orchestra.run.smithi062.stdout:(11/33): ceph-osd-18.2.0-0.el9.x86_64.rpm 16 MB/s | 18 MB 00:01 2024-04-20T21:16:05.998 INFO:teuthology.orchestra.run.smithi062.stdout:(12/33): python3-mako-1.1.4-6.el9.noarch.rpm 616 kB/s | 172 kB 00:00 2024-04-20T21:16:06.031 INFO:teuthology.orchestra.run.smithi062.stdout:(13/33): python3-urllib3-1.26.5-5.el9.noarch.rp 537 kB/s | 215 kB 00:00 2024-04-20T21:16:06.048 INFO:teuthology.orchestra.run.smithi062.stdout:(14/33): python3-markupsafe-1.1.1-12.el9.x86_64 421 kB/s | 35 kB 00:00 2024-04-20T21:16:06.074 INFO:teuthology.orchestra.run.smithi062.stdout:(15/33): python3-pytz-2021.1-5.el9.noarch.rpm 684 kB/s | 51 kB 00:00 2024-04-20T21:16:06.099 INFO:teuthology.orchestra.run.smithi062.stdout:(16/33): python3-toml-0.10.2-6.el9.noarch.rpm 634 kB/s | 42 kB 00:00 2024-04-20T21:16:06.216 INFO:teuthology.orchestra.run.smithi062.stdout:(17/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 262 kB/s | 43 kB 00:00 2024-04-20T21:16:06.274 INFO:teuthology.orchestra.run.smithi062.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 186 kB/s | 11 kB 00:00 2024-04-20T21:16:06.316 INFO:teuthology.orchestra.run.smithi062.stdout:(19/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 711 kB/s | 172 kB 00:00 2024-04-20T21:16:06.333 INFO:teuthology.orchestra.run.smithi062.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 306 kB/s | 18 kB 00:00 2024-04-20T21:16:06.392 INFO:teuthology.orchestra.run.smithi062.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 311 kB/s | 23 kB 00:00 2024-04-20T21:16:06.434 INFO:teuthology.orchestra.run.smithi062.stdout:(22/33): python3-cherrypy-18.6.1-2.el9.noarch.r 1.0 MB/s | 358 kB 00:00 2024-04-20T21:16:06.459 INFO:teuthology.orchestra.run.smithi062.stdout:(23/33): python3-jaraco-functools-3.5.0-2.el9.n 155 kB/s | 19 kB 00:00 2024-04-20T21:16:06.484 INFO:teuthology.orchestra.run.smithi062.stdout:(24/33): python3-jaraco-text-3.2.0-6.el9.noarch 215 kB/s | 20 kB 00:00 2024-04-20T21:16:06.509 INFO:teuthology.orchestra.run.smithi062.stdout:(25/33): python3-logutils-0.3.5-21.el9.noarch.r 618 kB/s | 46 kB 00:00 2024-04-20T21:16:06.535 INFO:teuthology.orchestra.run.smithi062.stdout:(26/33): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-04-20T21:16:06.568 INFO:teuthology.orchestra.run.smithi062.stdout:(27/33): python3-pecan-1.4.2-3.el9.noarch.rpm 3.2 MB/s | 272 kB 00:00 2024-04-20T21:16:06.594 INFO:teuthology.orchestra.run.smithi062.stdout:(28/33): python3-portend-3.1.0-2.el9.noarch.rpm 197 kB/s | 16 kB 00:00 2024-04-20T21:16:06.616 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:01:09 ago on Sat 20 Apr 2024 09:14:57 PM UTC. 2024-04-20T21:16:06.619 INFO:teuthology.orchestra.run.smithi062.stdout:(29/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.1 MB/s | 90 kB 00:00 2024-04-20T21:16:06.644 INFO:teuthology.orchestra.run.smithi062.stdout:(30/33): python3-tempora-5.0.0-2.el9.noarch.rpm 481 kB/s | 36 kB 00:00 2024-04-20T21:16:06.678 INFO:teuthology.orchestra.run.smithi062.stdout:(31/33): python3-webob-1.8.7-6.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-04-20T21:16:06.712 INFO:teuthology.orchestra.run.smithi062.stdout:(32/33): python3-zc-lockfile-2.0-10.el9.noarch. 302 kB/s | 20 kB 00:00 2024-04-20T21:16:06.745 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-04-20T21:16:06.746 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-20T21:16:06.746 INFO:teuthology.orchestra.run.smithi189.stdout: Package Arch Version Repository Size 2024-04-20T21:16:06.746 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-20T21:16:06.746 INFO:teuthology.orchestra.run.smithi189.stdout:Installing: 2024-04-20T21:16:06.746 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-dashboard noarch 2:18.2.0-0.el9 ceph-noarch 1.7 M 2024-04-20T21:16:06.746 INFO:teuthology.orchestra.run.smithi189.stdout:Installing dependencies: 2024-04-20T21:16:06.746 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-grafana-dashboards noarch 2:18.2.0-0.el9 ceph-noarch 23 k 2024-04-20T21:16:06.747 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-prometheus-alerts noarch 2:18.2.0-0.el9 ceph-noarch 15 k 2024-04-20T21:16:06.747 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-04-20T21:16:06.747 INFO:teuthology.orchestra.run.smithi189.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-20T21:16:06.747 INFO:teuthology.orchestra.run.smithi189.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-20T21:16:06.747 INFO:teuthology.orchestra.run.smithi189.stdout:Installing weak dependencies: 2024-04-20T21:16:06.747 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-04-20T21:16:06.747 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-20T21:16:06.747 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction Summary 2024-04-20T21:16:06.747 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-20T21:16:06.747 INFO:teuthology.orchestra.run.smithi189.stdout:Install 7 Packages 2024-04-20T21:16:06.747 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-20T21:16:06.748 INFO:teuthology.orchestra.run.smithi189.stdout:Total download size: 2.0 M 2024-04-20T21:16:06.748 INFO:teuthology.orchestra.run.smithi189.stdout:Installed size: 8.1 M 2024-04-20T21:16:06.748 INFO:teuthology.orchestra.run.smithi189.stdout:Downloading Packages: 2024-04-20T21:16:06.761 INFO:teuthology.orchestra.run.smithi062.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.0 MB/s | 427 kB 00:00 2024-04-20T21:16:06.767 INFO:teuthology.orchestra.run.smithi062.stdout:-------------------------------------------------------------------------------- 2024-04-20T21:16:06.768 INFO:teuthology.orchestra.run.smithi062.stdout:Total 10 MB/s | 30 MB 00:02 2024-04-20T21:16:07.146 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction check 2024-04-20T21:16:07.242 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction check succeeded. 2024-04-20T21:16:07.242 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction test 2024-04-20T21:16:07.303 INFO:teuthology.orchestra.run.smithi189.stdout:(1/7): ceph-grafana-dashboards-18.2.0-0.el9.noa 124 kB/s | 23 kB 00:00 2024-04-20T21:16:07.328 INFO:teuthology.orchestra.run.smithi189.stdout:(2/7): ceph-prometheus-alerts-18.2.0-0.el9.noar 69 kB/s | 15 kB 00:00 2024-04-20T21:16:07.412 INFO:teuthology.orchestra.run.smithi189.stdout:(3/7): ceph-mgr-dashboard-18.2.0-0.el9.noarch.r 5.8 MB/s | 1.7 MB 00:00 2024-04-20T21:16:07.504 INFO:teuthology.orchestra.run.smithi189.stdout:(4/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 45 kB/s | 9.0 kB 00:00 2024-04-20T21:16:07.529 INFO:teuthology.orchestra.run.smithi189.stdout:(5/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 265 kB/s | 31 kB 00:00 2024-04-20T21:16:07.579 INFO:teuthology.orchestra.run.smithi189.stdout:(6/7): python3-jwt-2.4.0-1.el9.noarch.rpm 163 kB/s | 41 kB 00:00 2024-04-20T21:16:07.696 INFO:teuthology.orchestra.run.smithi189.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 987 kB/s | 188 kB 00:00 2024-04-20T21:16:07.701 INFO:teuthology.orchestra.run.smithi189.stdout:-------------------------------------------------------------------------------- 2024-04-20T21:16:07.702 INFO:teuthology.orchestra.run.smithi189.stdout:Total 2.1 MB/s | 2.0 MB 00:00 2024-04-20T21:16:07.702 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction check 2024-04-20T21:16:07.731 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction check succeeded. 2024-04-20T21:16:07.731 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction test 2024-04-20T21:16:07.754 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction test succeeded. 2024-04-20T21:16:07.755 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction 2024-04-20T21:16:07.883 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction test succeeded. 2024-04-20T21:16:07.883 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction 2024-04-20T21:16:08.273 INFO:teuthology.orchestra.run.smithi189.stdout: Preparing : 1/1 2024-04-20T21:16:08.331 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-04-20T21:16:08.381 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-04-20T21:16:08.493 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-04-20T21:16:08.530 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-04-20T21:16:08.577 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-prometheus-alerts-2:18.2.0-0.el9.noarch 5/7 2024-04-20T21:16:08.896 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-grafana-dashboards-2:18.2.0-0.el9.noarch 6/7 2024-04-20T21:16:08.933 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mgr-dashboard-2:18.2.0-0.el9.noarch 7/7 2024-04-20T21:16:09.051 INFO:teuthology.orchestra.run.smithi062.stdout: Preparing : 1/1 2024-04-20T21:16:09.115 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-04-20T21:16:09.153 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-04-20T21:16:09.196 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-04-20T21:16:09.337 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-04-20T21:16:09.437 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-04-20T21:16:09.497 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-04-20T21:16:09.520 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.0-0.el9.noarch 7/7 2024-04-20T21:16:09.520 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-grafana-dashboards-2:18.2.0-0.el9.noarch 1/7 2024-04-20T21:16:09.520 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mgr-dashboard-2:18.2.0-0.el9.noarch 2/7 2024-04-20T21:16:09.520 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-prometheus-alerts-2:18.2.0-0.el9.noarch 3/7 2024-04-20T21:16:09.520 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-04-20T21:16:09.520 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-04-20T21:16:09.520 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-04-20T21:16:09.555 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-04-20T21:16:09.593 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/33 2024-04-20T21:16:09.640 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/33 2024-04-20T21:16:09.683 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/33 2024-04-20T21:16:09.746 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/33 2024-04-20T21:16:09.814 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/33 2024-04-20T21:16:09.827 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-04-20T21:16:09.827 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-20T21:16:09.827 INFO:teuthology.orchestra.run.smithi189.stdout:Installed: 2024-04-20T21:16:09.827 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-grafana-dashboards-2:18.2.0-0.el9.noarch 2024-04-20T21:16:09.827 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-dashboard-2:18.2.0-0.el9.noarch 2024-04-20T21:16:09.827 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-prometheus-alerts-2:18.2.0-0.el9.noarch 2024-04-20T21:16:09.827 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-04-20T21:16:09.828 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-04-20T21:16:09.828 INFO:teuthology.orchestra.run.smithi189.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-20T21:16:09.828 INFO:teuthology.orchestra.run.smithi189.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-20T21:16:09.828 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-20T21:16:09.828 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-04-20T21:16:09.866 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/33 2024-04-20T21:16:09.926 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/33 2024-04-20T21:16:10.061 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 15/33 2024-04-20T21:16:10.117 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-20T21:16:10.277 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 16/33 2024-04-20T21:16:10.382 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 17/33 2024-04-20T21:16:10.458 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 18/33 2024-04-20T21:16:10.508 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 19/33 2024-04-20T21:16:10.577 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-ply-3.11-14.el9.noarch 20/33 2024-04-20T21:16:10.664 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 21/33 2024-04-20T21:16:10.866 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:01:13 ago on Sat 20 Apr 2024 09:14:57 PM UTC. 2024-04-20T21:16:10.943 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 22/33 2024-04-20T21:16:10.994 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-04-20T21:16:10.996 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-20T21:16:10.996 INFO:teuthology.orchestra.run.smithi189.stdout: Package Arch Version Repository Size 2024-04-20T21:16:10.996 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-20T21:16:10.996 INFO:teuthology.orchestra.run.smithi189.stdout:Installing: 2024-04-20T21:16:10.996 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.0-0.el9 ceph-noarch 7.4 M 2024-04-20T21:16:10.996 INFO:teuthology.orchestra.run.smithi189.stdout:Installing dependencies: 2024-04-20T21:16:10.996 INFO:teuthology.orchestra.run.smithi189.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-20T21:16:10.996 INFO:teuthology.orchestra.run.smithi189.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-20T21:16:10.996 INFO:teuthology.orchestra.run.smithi189.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-20T21:16:10.996 INFO:teuthology.orchestra.run.smithi189.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-20T21:16:10.996 INFO:teuthology.orchestra.run.smithi189.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-20T21:16:10.996 INFO:teuthology.orchestra.run.smithi189.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-20T21:16:10.996 INFO:teuthology.orchestra.run.smithi189.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-20T21:16:10.996 INFO:teuthology.orchestra.run.smithi189.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-20T21:16:10.997 INFO:teuthology.orchestra.run.smithi189.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-20T21:16:10.997 INFO:teuthology.orchestra.run.smithi189.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-20T21:16:10.997 INFO:teuthology.orchestra.run.smithi189.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-20T21:16:10.997 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-20T21:16:10.997 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction Summary 2024-04-20T21:16:10.997 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-20T21:16:10.997 INFO:teuthology.orchestra.run.smithi189.stdout:Install 12 Packages 2024-04-20T21:16:10.997 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-20T21:16:10.997 INFO:teuthology.orchestra.run.smithi189.stdout:Total download size: 38 M 2024-04-20T21:16:10.998 INFO:teuthology.orchestra.run.smithi189.stdout:Installed size: 209 M 2024-04-20T21:16:10.998 INFO:teuthology.orchestra.run.smithi189.stdout:Downloading Packages: 2024-04-20T21:16:11.004 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 23/33 2024-04-20T21:16:11.107 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 24/33 2024-04-20T21:16:11.224 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 25/33 2024-04-20T21:16:11.277 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 26/33 2024-04-20T21:16:11.434 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 27/33 2024-04-20T21:16:11.603 INFO:teuthology.orchestra.run.smithi189.stdout:(1/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.2 MB/s | 195 kB 00:00 2024-04-20T21:16:11.614 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-mgr-modules-core-2:18.2.0-0.el9.noarch 28/33 2024-04-20T21:16:11.642 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-mgr-2:18.2.0-0.el9.x86_64 29/33 2024-04-20T21:16:11.645 INFO:teuthology.orchestra.run.smithi189.stdout:(2/12): libgfortran-11.4.1-3.el9.x86_64.rpm 3.9 MB/s | 803 kB 00:00 2024-04-20T21:16:11.722 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-mgr-2:18.2.0-0.el9.x86_64 29/33 2024-04-20T21:16:11.722 INFO:teuthology.orchestra.run.smithi062.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-20T21:16:11.722 INFO:teuthology.orchestra.run.smithi062.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-20T21:16:11.722 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-20T21:16:11.722 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-20T21:16:11.722 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:16:11.820 INFO:teuthology.orchestra.run.smithi189.stdout:(3/12): ceph-mgr-diskprediction-local-18.2.0-0. 20 MB/s | 7.4 MB 00:00 2024-04-20T21:16:11.887 INFO:teuthology.orchestra.run.smithi189.stdout:(4/12): flexiblas-3.0.4-8.el9.x86_64.rpm 115 kB/s | 33 kB 00:00 2024-04-20T21:16:12.079 INFO:teuthology.orchestra.run.smithi189.stdout:(5/12): openblas-0.3.21-2.el9.x86_64.rpm 184 kB/s | 35 kB 00:00 2024-04-20T21:16:12.104 INFO:teuthology.orchestra.run.smithi189.stdout:(6/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 64 kB/s | 18 kB 00:00 2024-04-20T21:16:12.629 INFO:teuthology.orchestra.run.smithi189.stdout:(7/12): python3-devel-3.9.18-3.el9.x86_64.rpm 466 kB/s | 244 kB 00:00 2024-04-20T21:16:13.468 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-osd-2:18.2.0-0.el9.x86_64 30/33 2024-04-20T21:16:13.550 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-osd-2:18.2.0-0.el9.x86_64 30/33 2024-04-20T21:16:13.550 INFO:teuthology.orchestra.run.smithi062.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-20T21:16:13.550 INFO:teuthology.orchestra.run.smithi062.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-20T21:16:13.550 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-20T21:16:13.550 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-20T21:16:13.551 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:16:14.094 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-mon-2:18.2.0-0.el9.x86_64 31/33 2024-04-20T21:16:14.166 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-mon-2:18.2.0-0.el9.x86_64 31/33 2024-04-20T21:16:14.166 INFO:teuthology.orchestra.run.smithi062.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-20T21:16:14.166 INFO:teuthology.orchestra.run.smithi062.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-20T21:16:14.167 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-20T21:16:14.167 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-20T21:16:14.167 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:16:14.411 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-mds-2:18.2.0-0.el9.x86_64 32/33 2024-04-20T21:16:14.482 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-mds-2:18.2.0-0.el9.x86_64 32/33 2024-04-20T21:16:14.482 INFO:teuthology.orchestra.run.smithi062.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-20T21:16:14.482 INFO:teuthology.orchestra.run.smithi062.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-20T21:16:14.483 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-20T21:16:14.483 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-20T21:16:14.483 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:16:14.583 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-2:18.2.0-0.el9.x86_64 33/33 2024-04-20T21:16:14.712 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-2:18.2.0-0.el9.x86_64 33/33 2024-04-20T21:16:14.712 INFO:teuthology.orchestra.run.smithi062.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-04-20T21:16:14.712 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:16:15.146 INFO:teuthology.orchestra.run.smithi189.stdout:(8/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 2.0 MB/s | 5.1 MB 00:02 2024-04-20T21:16:15.441 INFO:teuthology.orchestra.run.smithi189.stdout:(9/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 807 kB/s | 3.0 MB 00:03 2024-04-20T21:16:15.621 INFO:teuthology.orchestra.run.smithi189.stdout:(10/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 528 kB/s | 250 kB 00:00 2024-04-20T21:16:15.880 INFO:teuthology.orchestra.run.smithi189.stdout:(11/12): openblas-openmp-0.3.21-2.el9.x86_64.rp 1.2 MB/s | 4.7 MB 00:03 2024-04-20T21:16:16.513 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-2:18.2.0-0.el9.x86_64 1/33 2024-04-20T21:16:16.513 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-mds-2:18.2.0-0.el9.x86_64 2/33 2024-04-20T21:16:16.514 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-mgr-2:18.2.0-0.el9.x86_64 3/33 2024-04-20T21:16:16.514 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-mon-2:18.2.0-0.el9.x86_64 4/33 2024-04-20T21:16:16.514 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-osd-2:18.2.0-0.el9.x86_64 5/33 2024-04-20T21:16:16.514 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-mgr-modules-core-2:18.2.0-0.el9.noarch 6/33 2024-04-20T21:16:16.514 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/33 2024-04-20T21:16:16.514 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/33 2024-04-20T21:16:16.514 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/33 2024-04-20T21:16:16.514 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/33 2024-04-20T21:16:16.514 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/33 2024-04-20T21:16:16.514 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/33 2024-04-20T21:16:16.514 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/33 2024-04-20T21:16:16.514 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/33 2024-04-20T21:16:16.514 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 15/33 2024-04-20T21:16:16.515 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 16/33 2024-04-20T21:16:16.515 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-04-20T21:16:16.515 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-04-20T21:16:16.515 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-04-20T21:16:16.515 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-04-20T21:16:16.515 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-04-20T21:16:16.515 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-04-20T21:16:16.515 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-04-20T21:16:16.515 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-04-20T21:16:16.515 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-04-20T21:16:16.515 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-04-20T21:16:16.515 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-04-20T21:16:16.515 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-04-20T21:16:16.515 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-04-20T21:16:16.516 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-04-20T21:16:16.516 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-04-20T21:16:16.516 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-04-20T21:16:17.474 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-04-20T21:16:17.475 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:16:17.475 INFO:teuthology.orchestra.run.smithi062.stdout:Installed: 2024-04-20T21:16:17.475 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-2:18.2.0-0.el9.x86_64 2024-04-20T21:16:17.475 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mds-2:18.2.0-0.el9.x86_64 2024-04-20T21:16:17.475 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-2:18.2.0-0.el9.x86_64 2024-04-20T21:16:17.475 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-modules-core-2:18.2.0-0.el9.noarch 2024-04-20T21:16:17.475 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mon-2:18.2.0-0.el9.x86_64 2024-04-20T21:16:17.475 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-osd-2:18.2.0-0.el9.x86_64 2024-04-20T21:16:17.475 INFO:teuthology.orchestra.run.smithi062.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-20T21:16:17.475 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-20T21:16:17.475 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-20T21:16:17.475 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-20T21:16:17.475 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-20T21:16:17.476 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-20T21:16:17.476 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-20T21:16:17.476 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-20T21:16:17.476 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-20T21:16:17.476 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-20T21:16:17.476 INFO:teuthology.orchestra.run.smithi062.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-20T21:16:17.476 INFO:teuthology.orchestra.run.smithi062.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-20T21:16:17.476 INFO:teuthology.orchestra.run.smithi062.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-20T21:16:17.476 INFO:teuthology.orchestra.run.smithi062.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-20T21:16:17.476 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-20T21:16:17.476 INFO:teuthology.orchestra.run.smithi062.stdout: python3-ply-3.11-14.el9.noarch 2024-04-20T21:16:17.476 INFO:teuthology.orchestra.run.smithi062.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-20T21:16:17.476 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-20T21:16:17.476 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-20T21:16:17.476 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-20T21:16:17.477 INFO:teuthology.orchestra.run.smithi062.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-20T21:16:17.477 INFO:teuthology.orchestra.run.smithi062.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-20T21:16:17.477 INFO:teuthology.orchestra.run.smithi062.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-20T21:16:17.477 INFO:teuthology.orchestra.run.smithi062.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-20T21:16:17.477 INFO:teuthology.orchestra.run.smithi062.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-20T21:16:17.477 INFO:teuthology.orchestra.run.smithi062.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-20T21:16:17.477 INFO:teuthology.orchestra.run.smithi062.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-20T21:16:17.477 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:16:17.477 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2024-04-20T21:16:17.890 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install ceph-base 2024-04-20T21:16:18.675 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:00:58 ago on Sat 20 Apr 2024 09:15:20 PM UTC. 2024-04-20T21:16:18.758 INFO:teuthology.orchestra.run.smithi062.stdout:Package ceph-base-2:18.2.0-0.el9.x86_64 is already installed. 2024-04-20T21:16:18.803 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2024-04-20T21:16:18.804 INFO:teuthology.orchestra.run.smithi062.stdout:Nothing to do. 2024-04-20T21:16:18.804 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2024-04-20T21:16:18.887 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install cephadm 2024-04-20T21:16:19.670 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:00:59 ago on Sat 20 Apr 2024 09:15:20 PM UTC. 2024-04-20T21:16:19.798 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2024-04-20T21:16:19.799 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-04-20T21:16:19.799 INFO:teuthology.orchestra.run.smithi062.stdout: Package Architecture Version Repository Size 2024-04-20T21:16:19.799 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-04-20T21:16:19.799 INFO:teuthology.orchestra.run.smithi062.stdout:Installing: 2024-04-20T21:16:19.799 INFO:teuthology.orchestra.run.smithi062.stdout: cephadm noarch 2:18.2.0-0.el9 ceph-noarch 209 k 2024-04-20T21:16:19.799 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:16:19.800 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction Summary 2024-04-20T21:16:19.800 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-04-20T21:16:19.800 INFO:teuthology.orchestra.run.smithi062.stdout:Install 1 Package 2024-04-20T21:16:19.800 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:16:19.800 INFO:teuthology.orchestra.run.smithi062.stdout:Total download size: 209 k 2024-04-20T21:16:19.800 INFO:teuthology.orchestra.run.smithi062.stdout:Installed size: 200 k 2024-04-20T21:16:19.800 INFO:teuthology.orchestra.run.smithi062.stdout:Downloading Packages: 2024-04-20T21:16:19.877 INFO:teuthology.orchestra.run.smithi062.stdout:cephadm-18.2.0-0.el9.noarch.rpm 2.7 MB/s | 209 kB 00:00 2024-04-20T21:16:19.877 INFO:teuthology.orchestra.run.smithi062.stdout:-------------------------------------------------------------------------------- 2024-04-20T21:16:19.877 INFO:teuthology.orchestra.run.smithi062.stdout:Total 2.6 MB/s | 209 kB 00:00 2024-04-20T21:16:19.877 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction check 2024-04-20T21:16:19.884 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction check succeeded. 2024-04-20T21:16:19.884 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction test 2024-04-20T21:16:19.888 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction test succeeded. 2024-04-20T21:16:19.888 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction 2024-04-20T21:16:19.921 INFO:teuthology.orchestra.run.smithi189.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 3.6 MB/s | 16 MB 00:04 2024-04-20T21:16:19.924 INFO:teuthology.orchestra.run.smithi189.stdout:-------------------------------------------------------------------------------- 2024-04-20T21:16:19.925 INFO:teuthology.orchestra.run.smithi189.stdout:Total 4.3 MB/s | 38 MB 00:08 2024-04-20T21:16:19.970 INFO:teuthology.orchestra.run.smithi062.stdout: Preparing : 1/1 2024-04-20T21:16:20.537 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction check 2024-04-20T21:16:20.615 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction check succeeded. 2024-04-20T21:16:20.615 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction test 2024-04-20T21:16:20.638 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: cephadm-2:18.2.0-0.el9.noarch 1/1 2024-04-20T21:16:20.764 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : cephadm-2:18.2.0-0.el9.noarch 1/1 2024-04-20T21:16:21.216 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: cephadm-2:18.2.0-0.el9.noarch 1/1 2024-04-20T21:16:21.244 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction test succeeded. 2024-04-20T21:16:21.245 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction 2024-04-20T21:16:21.502 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : cephadm-2:18.2.0-0.el9.noarch 1/1 2024-04-20T21:16:21.502 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:16:21.502 INFO:teuthology.orchestra.run.smithi062.stdout:Installed: 2024-04-20T21:16:21.503 INFO:teuthology.orchestra.run.smithi062.stdout: cephadm-2:18.2.0-0.el9.noarch 2024-04-20T21:16:21.503 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:16:21.503 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2024-04-20T21:16:21.785 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install ceph-immutable-object-cache 2024-04-20T21:16:22.161 INFO:teuthology.orchestra.run.smithi189.stdout: Preparing : 1/1 2024-04-20T21:16:22.231 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-04-20T21:16:22.292 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-04-20T21:16:22.438 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-04-20T21:16:22.481 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-04-20T21:16:22.595 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:01:02 ago on Sat 20 Apr 2024 09:15:20 PM UTC. 2024-04-20T21:16:22.724 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2024-04-20T21:16:22.725 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-04-20T21:16:22.725 INFO:teuthology.orchestra.run.smithi062.stdout: Package Arch Version Repo Size 2024-04-20T21:16:22.725 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-04-20T21:16:22.725 INFO:teuthology.orchestra.run.smithi062.stdout:Installing: 2024-04-20T21:16:22.725 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-immutable-object-cache x86_64 2:18.2.0-0.el9 ceph 142 k 2024-04-20T21:16:22.725 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:16:22.725 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction Summary 2024-04-20T21:16:22.725 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-04-20T21:16:22.726 INFO:teuthology.orchestra.run.smithi062.stdout:Install 1 Package 2024-04-20T21:16:22.726 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:16:22.726 INFO:teuthology.orchestra.run.smithi062.stdout:Total download size: 142 k 2024-04-20T21:16:22.726 INFO:teuthology.orchestra.run.smithi062.stdout:Installed size: 399 k 2024-04-20T21:16:22.726 INFO:teuthology.orchestra.run.smithi062.stdout:Downloading Packages: 2024-04-20T21:16:22.735 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-20T21:16:22.770 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-20T21:16:22.797 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-immutable-object-cache-18.2.0-0.el9.x86_64 2.0 MB/s | 142 kB 00:00 2024-04-20T21:16:22.798 INFO:teuthology.orchestra.run.smithi062.stdout:-------------------------------------------------------------------------------- 2024-04-20T21:16:22.799 INFO:teuthology.orchestra.run.smithi062.stdout:Total 1.9 MB/s | 142 kB 00:00 2024-04-20T21:16:22.799 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction check 2024-04-20T21:16:22.808 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction check succeeded. 2024-04-20T21:16:22.809 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction test 2024-04-20T21:16:22.877 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction test succeeded. 2024-04-20T21:16:22.877 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction 2024-04-20T21:16:22.896 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-04-20T21:16:23.046 INFO:teuthology.orchestra.run.smithi062.stdout: Preparing : 1/1 2024-04-20T21:16:23.124 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-immutable-object-cache-2:18.2.0-0.el9.x86_64 1/1 2024-04-20T21:16:23.207 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.0-0.el9.x86_64 1/1 2024-04-20T21:16:23.207 INFO:teuthology.orchestra.run.smithi062.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-20T21:16:23.207 INFO:teuthology.orchestra.run.smithi062.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-20T21:16:23.207 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:16:23.794 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-04-20T21:16:23.940 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-04-20T21:16:24.431 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-immutable-object-cache-2:18.2.0-0.el9.x86_64 1/1 2024-04-20T21:16:24.431 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:16:24.431 INFO:teuthology.orchestra.run.smithi062.stdout:Installed: 2024-04-20T21:16:24.431 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-immutable-object-cache-2:18.2.0-0.el9.x86_64 2024-04-20T21:16:24.431 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:16:24.431 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2024-04-20T21:16:24.732 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install ceph-mgr 2024-04-20T21:16:25.483 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-04-20T21:16:25.532 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:01:05 ago on Sat 20 Apr 2024 09:15:20 PM UTC. 2024-04-20T21:16:25.615 INFO:teuthology.orchestra.run.smithi062.stdout:Package ceph-mgr-2:18.2.0-0.el9.x86_64 is already installed. 2024-04-20T21:16:25.659 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2024-04-20T21:16:25.661 INFO:teuthology.orchestra.run.smithi062.stdout:Nothing to do. 2024-04-20T21:16:25.661 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2024-04-20T21:16:25.746 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install ceph-mgr-dashboard 2024-04-20T21:16:26.536 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:01:06 ago on Sat 20 Apr 2024 09:15:20 PM UTC. 2024-04-20T21:16:26.651 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-04-20T21:16:26.670 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2024-04-20T21:16:26.671 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-04-20T21:16:26.671 INFO:teuthology.orchestra.run.smithi062.stdout: Package Arch Version Repository Size 2024-04-20T21:16:26.671 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-04-20T21:16:26.671 INFO:teuthology.orchestra.run.smithi062.stdout:Installing: 2024-04-20T21:16:26.671 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-dashboard noarch 2:18.2.0-0.el9 ceph-noarch 1.7 M 2024-04-20T21:16:26.671 INFO:teuthology.orchestra.run.smithi062.stdout:Installing dependencies: 2024-04-20T21:16:26.671 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-grafana-dashboards noarch 2:18.2.0-0.el9 ceph-noarch 23 k 2024-04-20T21:16:26.671 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-prometheus-alerts noarch 2:18.2.0-0.el9 ceph-noarch 15 k 2024-04-20T21:16:26.671 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-04-20T21:16:26.671 INFO:teuthology.orchestra.run.smithi062.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-20T21:16:26.672 INFO:teuthology.orchestra.run.smithi062.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-20T21:16:26.672 INFO:teuthology.orchestra.run.smithi062.stdout:Installing weak dependencies: 2024-04-20T21:16:26.672 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-04-20T21:16:26.672 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:16:26.672 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction Summary 2024-04-20T21:16:26.672 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-04-20T21:16:26.672 INFO:teuthology.orchestra.run.smithi062.stdout:Install 7 Packages 2024-04-20T21:16:26.672 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:16:26.673 INFO:teuthology.orchestra.run.smithi062.stdout:Total download size: 2.0 M 2024-04-20T21:16:26.673 INFO:teuthology.orchestra.run.smithi062.stdout:Installed size: 8.1 M 2024-04-20T21:16:26.673 INFO:teuthology.orchestra.run.smithi062.stdout:Downloading Packages: 2024-04-20T21:16:26.686 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.0-0.el9.noarc 12/12 2024-04-20T21:16:27.275 INFO:teuthology.orchestra.run.smithi062.stdout:(1/7): ceph-grafana-dashboards-18.2.0-0.el9.noa 102 kB/s | 23 kB 00:00 2024-04-20T21:16:27.300 INFO:teuthology.orchestra.run.smithi062.stdout:(2/7): ceph-prometheus-alerts-18.2.0-0.el9.noar 58 kB/s | 15 kB 00:00 2024-04-20T21:16:27.375 INFO:teuthology.orchestra.run.smithi062.stdout:(3/7): ceph-mgr-dashboard-18.2.0-0.el9.noarch.r 5.2 MB/s | 1.7 MB 00:00 2024-04-20T21:16:27.619 INFO:teuthology.orchestra.run.smithi062.stdout:(4/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 26 kB/s | 9.0 kB 00:00 2024-04-20T21:16:27.735 INFO:teuthology.orchestra.run.smithi062.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 94 kB/s | 41 kB 00:00 2024-04-20T21:16:27.768 INFO:teuthology.orchestra.run.smithi062.stdout:(6/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 107 kB/s | 31 kB 00:00 2024-04-20T21:16:27.843 INFO:teuthology.orchestra.run.smithi062.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 838 kB/s | 188 kB 00:00 2024-04-20T21:16:27.854 INFO:teuthology.orchestra.run.smithi062.stdout:-------------------------------------------------------------------------------- 2024-04-20T21:16:27.855 INFO:teuthology.orchestra.run.smithi062.stdout:Total 1.7 MB/s | 2.0 MB 00:01 2024-04-20T21:16:27.855 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction check 2024-04-20T21:16:27.884 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction check succeeded. 2024-04-20T21:16:27.884 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction test 2024-04-20T21:16:28.006 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction test succeeded. 2024-04-20T21:16:28.006 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction 2024-04-20T21:16:28.407 INFO:teuthology.orchestra.run.smithi062.stdout: Preparing : 1/1 2024-04-20T21:16:28.470 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-04-20T21:16:28.522 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-04-20T21:16:28.617 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-04-20T21:16:28.652 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-04-20T21:16:28.697 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-prometheus-alerts-2:18.2.0-0.el9.noarch 5/7 2024-04-20T21:16:29.008 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-grafana-dashboards-2:18.2.0-0.el9.noarch 6/7 2024-04-20T21:16:29.047 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-mgr-dashboard-2:18.2.0-0.el9.noarch 7/7 2024-04-20T21:16:29.069 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.0-0.el9.noarc 12/12 2024-04-20T21:16:29.069 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.0-0.el9.noarc 1/12 2024-04-20T21:16:29.069 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-04-20T21:16:29.069 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-04-20T21:16:29.069 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-04-20T21:16:29.069 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-04-20T21:16:29.069 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-04-20T21:16:29.070 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-04-20T21:16:29.070 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-04-20T21:16:29.070 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-04-20T21:16:29.070 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-04-20T21:16:29.070 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-04-20T21:16:29.417 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-04-20T21:16:29.418 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-20T21:16:29.418 INFO:teuthology.orchestra.run.smithi189.stdout:Installed: 2024-04-20T21:16:29.418 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-diskprediction-local-2:18.2.0-0.el9.noarch 2024-04-20T21:16:29.418 INFO:teuthology.orchestra.run.smithi189.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-20T21:16:29.418 INFO:teuthology.orchestra.run.smithi189.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-20T21:16:29.418 INFO:teuthology.orchestra.run.smithi189.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-20T21:16:29.418 INFO:teuthology.orchestra.run.smithi189.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-20T21:16:29.418 INFO:teuthology.orchestra.run.smithi189.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-20T21:16:29.418 INFO:teuthology.orchestra.run.smithi189.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-20T21:16:29.418 INFO:teuthology.orchestra.run.smithi189.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-20T21:16:29.418 INFO:teuthology.orchestra.run.smithi189.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-20T21:16:29.418 INFO:teuthology.orchestra.run.smithi189.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-20T21:16:29.418 INFO:teuthology.orchestra.run.smithi189.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-20T21:16:29.418 INFO:teuthology.orchestra.run.smithi189.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-20T21:16:29.419 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-20T21:16:29.419 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-04-20T21:16:29.607 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.0-0.el9.noarch 7/7 2024-04-20T21:16:29.607 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-grafana-dashboards-2:18.2.0-0.el9.noarch 1/7 2024-04-20T21:16:29.607 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-mgr-dashboard-2:18.2.0-0.el9.noarch 2/7 2024-04-20T21:16:29.607 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-prometheus-alerts-2:18.2.0-0.el9.noarch 3/7 2024-04-20T21:16:29.607 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-04-20T21:16:29.607 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-04-20T21:16:29.607 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-04-20T21:16:29.823 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install ceph-mgr-rook 2024-04-20T21:16:29.925 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-04-20T21:16:29.925 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:16:29.925 INFO:teuthology.orchestra.run.smithi062.stdout:Installed: 2024-04-20T21:16:29.925 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-grafana-dashboards-2:18.2.0-0.el9.noarch 2024-04-20T21:16:29.925 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-dashboard-2:18.2.0-0.el9.noarch 2024-04-20T21:16:29.925 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-prometheus-alerts-2:18.2.0-0.el9.noarch 2024-04-20T21:16:29.925 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-04-20T21:16:29.925 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-04-20T21:16:29.925 INFO:teuthology.orchestra.run.smithi062.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-20T21:16:29.925 INFO:teuthology.orchestra.run.smithi062.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-20T21:16:29.926 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:16:29.926 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2024-04-20T21:16:30.253 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-20T21:16:30.594 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:01:33 ago on Sat 20 Apr 2024 09:14:57 PM UTC. 2024-04-20T21:16:30.727 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-04-20T21:16:30.729 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-20T21:16:30.729 INFO:teuthology.orchestra.run.smithi189.stdout: Package Arch Version Repository Size 2024-04-20T21:16:30.729 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-20T21:16:30.729 INFO:teuthology.orchestra.run.smithi189.stdout:Installing: 2024-04-20T21:16:30.729 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-rook noarch 2:18.2.0-0.el9 ceph-noarch 47 k 2024-04-20T21:16:30.729 INFO:teuthology.orchestra.run.smithi189.stdout:Installing dependencies: 2024-04-20T21:16:30.729 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-20T21:16:30.729 INFO:teuthology.orchestra.run.smithi189.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-20T21:16:30.729 INFO:teuthology.orchestra.run.smithi189.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-20T21:16:30.730 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-20T21:16:30.730 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-20T21:16:30.730 INFO:teuthology.orchestra.run.smithi189.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-20T21:16:30.730 INFO:teuthology.orchestra.run.smithi189.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-20T21:16:30.730 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-20T21:16:30.730 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-20T21:16:30.730 INFO:teuthology.orchestra.run.smithi189.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-20T21:16:30.730 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-20T21:16:30.730 INFO:teuthology.orchestra.run.smithi189.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-20T21:16:30.730 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-20T21:16:30.730 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction Summary 2024-04-20T21:16:30.730 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-20T21:16:30.730 INFO:teuthology.orchestra.run.smithi189.stdout:Install 13 Packages 2024-04-20T21:16:30.730 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-20T21:16:30.731 INFO:teuthology.orchestra.run.smithi189.stdout:Total download size: 2.2 M 2024-04-20T21:16:30.731 INFO:teuthology.orchestra.run.smithi189.stdout:Installed size: 26 M 2024-04-20T21:16:30.731 INFO:teuthology.orchestra.run.smithi189.stdout:Downloading Packages: 2024-04-20T21:16:31.049 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:01:11 ago on Sat 20 Apr 2024 09:15:20 PM UTC. 2024-04-20T21:16:31.185 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2024-04-20T21:16:31.187 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-04-20T21:16:31.187 INFO:teuthology.orchestra.run.smithi062.stdout: Package Arch Version Repository Size 2024-04-20T21:16:31.187 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-04-20T21:16:31.187 INFO:teuthology.orchestra.run.smithi062.stdout:Installing: 2024-04-20T21:16:31.187 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.0-0.el9 ceph-noarch 7.4 M 2024-04-20T21:16:31.187 INFO:teuthology.orchestra.run.smithi062.stdout:Installing dependencies: 2024-04-20T21:16:31.187 INFO:teuthology.orchestra.run.smithi062.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-20T21:16:31.187 INFO:teuthology.orchestra.run.smithi062.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-20T21:16:31.187 INFO:teuthology.orchestra.run.smithi062.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-20T21:16:31.187 INFO:teuthology.orchestra.run.smithi062.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-20T21:16:31.187 INFO:teuthology.orchestra.run.smithi062.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-20T21:16:31.187 INFO:teuthology.orchestra.run.smithi062.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-20T21:16:31.188 INFO:teuthology.orchestra.run.smithi062.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-20T21:16:31.188 INFO:teuthology.orchestra.run.smithi062.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-20T21:16:31.188 INFO:teuthology.orchestra.run.smithi062.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-20T21:16:31.188 INFO:teuthology.orchestra.run.smithi062.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-20T21:16:31.188 INFO:teuthology.orchestra.run.smithi062.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-20T21:16:31.188 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:16:31.188 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction Summary 2024-04-20T21:16:31.188 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-04-20T21:16:31.188 INFO:teuthology.orchestra.run.smithi062.stdout:Install 12 Packages 2024-04-20T21:16:31.188 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:16:31.189 INFO:teuthology.orchestra.run.smithi062.stdout:Total download size: 38 M 2024-04-20T21:16:31.189 INFO:teuthology.orchestra.run.smithi062.stdout:Installed size: 209 M 2024-04-20T21:16:31.189 INFO:teuthology.orchestra.run.smithi062.stdout:Downloading Packages: 2024-04-20T21:16:31.346 INFO:teuthology.orchestra.run.smithi189.stdout:(1/13): ceph-mgr-rook-18.2.0-0.el9.noarch.rpm 428 kB/s | 47 kB 00:00 2024-04-20T21:16:31.396 INFO:teuthology.orchestra.run.smithi189.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 119 kB/s | 19 kB 00:00 2024-04-20T21:16:31.421 INFO:teuthology.orchestra.run.smithi189.stdout:(3/13): python3-jsonpatch-1.21-16.el9.noarch.rp 138 kB/s | 26 kB 00:00 2024-04-20T21:16:31.530 INFO:teuthology.orchestra.run.smithi189.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.2 MB/s | 159 kB 00:00 2024-04-20T21:16:31.580 INFO:teuthology.orchestra.run.smithi189.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 953 kB/s | 222 kB 00:00 2024-04-20T21:16:31.614 INFO:teuthology.orchestra.run.smithi189.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.4 MB/s | 279 kB 00:00 2024-04-20T21:16:31.639 INFO:teuthology.orchestra.run.smithi189.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 496 kB/s | 54 kB 00:00 2024-04-20T21:16:31.942 INFO:teuthology.orchestra.run.smithi062.stdout:(1/12): ceph-mgr-diskprediction-local-18.2.0-0. 26 MB/s | 7.4 MB 00:00 2024-04-20T21:16:31.966 INFO:teuthology.orchestra.run.smithi062.stdout:(2/12): libquadmath-11.4.1-3.el9.x86_64.rpm 631 kB/s | 195 kB 00:00 2024-04-20T21:16:32.009 INFO:teuthology.orchestra.run.smithi062.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 487 kB/s | 33 kB 00:00 2024-04-20T21:16:32.059 INFO:teuthology.orchestra.run.smithi062.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 380 kB/s | 18 kB 00:00 2024-04-20T21:16:32.109 INFO:teuthology.orchestra.run.smithi062.stdout:(5/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.7 MB/s | 803 kB 00:00 2024-04-20T21:16:32.122 INFO:teuthology.orchestra.run.smithi189.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 59 kB/s | 32 kB 00:00 2024-04-20T21:16:32.159 INFO:teuthology.orchestra.run.smithi062.stdout:(6/12): openblas-0.3.21-2.el9.x86_64.rpm 359 kB/s | 35 kB 00:00 2024-04-20T21:16:32.164 INFO:teuthology.orchestra.run.smithi189.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 27 kB/s | 15 kB 00:00 2024-04-20T21:16:32.285 INFO:teuthology.orchestra.run.smithi062.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 9.4 MB/s | 3.0 MB 00:00 2024-04-20T21:16:32.310 INFO:teuthology.orchestra.run.smithi062.stdout:(8/12): python3-devel-3.9.18-3.el9.x86_64.rpm 1.6 MB/s | 244 kB 00:00 2024-04-20T21:16:32.431 INFO:teuthology.orchestra.run.smithi189.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 222 kB/s | 59 kB 00:00 2024-04-20T21:16:32.443 INFO:teuthology.orchestra.run.smithi062.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.8 MB/s | 250 kB 00:00 2024-04-20T21:16:32.540 INFO:teuthology.orchestra.run.smithi189.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 225 kB/s | 203 kB 00:00 2024-04-20T21:16:32.544 INFO:teuthology.orchestra.run.smithi062.stdout:(10/12): openblas-openmp-0.3.21-2.el9.x86_64.rp 11 MB/s | 4.7 MB 00:00 2024-04-20T21:16:32.632 INFO:teuthology.orchestra.run.smithi189.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 450 kB/s | 90 kB 00:00 2024-04-20T21:16:33.011 INFO:teuthology.orchestra.run.smithi062.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 7.0 MB/s | 5.1 MB 00:00 2024-04-20T21:16:33.440 INFO:teuthology.orchestra.run.smithi189.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 794 kB/s | 1.0 MB 00:01 2024-04-20T21:16:33.447 INFO:teuthology.orchestra.run.smithi189.stdout:-------------------------------------------------------------------------------- 2024-04-20T21:16:33.448 INFO:teuthology.orchestra.run.smithi189.stdout:Total 828 kB/s | 2.2 MB 00:02 2024-04-20T21:16:33.671 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction check 2024-04-20T21:16:33.718 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction check succeeded. 2024-04-20T21:16:33.718 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction test 2024-04-20T21:16:33.867 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction test succeeded. 2024-04-20T21:16:33.867 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction 2024-04-20T21:16:33.927 INFO:teuthology.orchestra.run.smithi062.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 11 MB/s | 16 MB 00:01 2024-04-20T21:16:33.929 INFO:teuthology.orchestra.run.smithi062.stdout:-------------------------------------------------------------------------------- 2024-04-20T21:16:33.930 INFO:teuthology.orchestra.run.smithi062.stdout:Total 14 MB/s | 38 MB 00:02 2024-04-20T21:16:34.580 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction check 2024-04-20T21:16:34.622 INFO:teuthology.orchestra.run.smithi189.stdout: Preparing : 1/1 2024-04-20T21:16:34.663 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction check succeeded. 2024-04-20T21:16:34.663 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction test 2024-04-20T21:16:34.821 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-04-20T21:16:34.893 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-04-20T21:16:34.959 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-04-20T21:16:34.997 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-04-20T21:16:35.051 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-04-20T21:16:35.213 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-04-20T21:16:35.353 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-04-20T21:16:35.354 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction test succeeded. 2024-04-20T21:16:35.354 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction 2024-04-20T21:16:35.420 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-04-20T21:16:36.301 INFO:teuthology.orchestra.run.smithi062.stdout: Preparing : 1/1 2024-04-20T21:16:36.341 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-04-20T21:16:36.380 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-04-20T21:16:36.403 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-04-20T21:16:36.436 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-04-20T21:16:36.442 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-04-20T21:16:36.488 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-04-20T21:16:36.512 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mgr-rook-2:18.2.0-0.el9.noarch 13/13 2024-04-20T21:16:36.592 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-04-20T21:16:36.637 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-04-20T21:16:36.901 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-20T21:16:36.934 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-20T21:16:37.059 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-04-20T21:16:37.395 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.0-0.el9.noarch 13/13 2024-04-20T21:16:37.395 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mgr-rook-2:18.2.0-0.el9.noarch 1/13 2024-04-20T21:16:37.395 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-04-20T21:16:37.395 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-04-20T21:16:37.395 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-04-20T21:16:37.395 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-04-20T21:16:37.395 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-04-20T21:16:37.396 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-04-20T21:16:37.396 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-04-20T21:16:37.396 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-04-20T21:16:37.396 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-04-20T21:16:37.396 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-04-20T21:16:37.396 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-04-20T21:16:37.817 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-04-20T21:16:37.817 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-20T21:16:37.817 INFO:teuthology.orchestra.run.smithi189.stdout:Installed: 2024-04-20T21:16:37.817 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-rook-2:18.2.0-0.el9.noarch 2024-04-20T21:16:37.817 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-20T21:16:37.817 INFO:teuthology.orchestra.run.smithi189.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-20T21:16:37.817 INFO:teuthology.orchestra.run.smithi189.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-20T21:16:37.817 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-20T21:16:37.818 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-20T21:16:37.818 INFO:teuthology.orchestra.run.smithi189.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-20T21:16:37.818 INFO:teuthology.orchestra.run.smithi189.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-20T21:16:37.818 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-20T21:16:37.818 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-20T21:16:37.818 INFO:teuthology.orchestra.run.smithi189.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-20T21:16:37.818 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-20T21:16:37.818 INFO:teuthology.orchestra.run.smithi189.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-20T21:16:37.818 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-20T21:16:37.818 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-04-20T21:16:37.983 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-04-20T21:16:38.119 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install ceph-mgr-cephadm 2024-04-20T21:16:38.126 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-04-20T21:16:38.908 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:01:41 ago on Sat 20 Apr 2024 09:14:57 PM UTC. 2024-04-20T21:16:39.034 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-04-20T21:16:39.035 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-20T21:16:39.035 INFO:teuthology.orchestra.run.smithi189.stdout: Package Arch Version Repository Size 2024-04-20T21:16:39.035 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-20T21:16:39.035 INFO:teuthology.orchestra.run.smithi189.stdout:Installing: 2024-04-20T21:16:39.036 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-cephadm noarch 2:18.2.0-0.el9 ceph-noarch 127 k 2024-04-20T21:16:39.036 INFO:teuthology.orchestra.run.smithi189.stdout:Installing dependencies: 2024-04-20T21:16:39.036 INFO:teuthology.orchestra.run.smithi189.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-20T21:16:39.036 INFO:teuthology.orchestra.run.smithi189.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-20T21:16:39.036 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-20T21:16:39.036 INFO:teuthology.orchestra.run.smithi189.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-20T21:16:39.036 INFO:teuthology.orchestra.run.smithi189.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-20T21:16:39.036 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-20T21:16:39.036 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction Summary 2024-04-20T21:16:39.036 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-20T21:16:39.036 INFO:teuthology.orchestra.run.smithi189.stdout:Install 6 Packages 2024-04-20T21:16:39.036 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-20T21:16:39.037 INFO:teuthology.orchestra.run.smithi189.stdout:Total download size: 7.0 M 2024-04-20T21:16:39.037 INFO:teuthology.orchestra.run.smithi189.stdout:Installed size: 33 M 2024-04-20T21:16:39.037 INFO:teuthology.orchestra.run.smithi189.stdout:Downloading Packages: 2024-04-20T21:16:39.719 INFO:teuthology.orchestra.run.smithi189.stdout:(1/6): ceph-mgr-cephadm-18.2.0-0.el9.noarch.rpm 1.0 MB/s | 127 kB 00:00 2024-04-20T21:16:39.791 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-04-20T21:16:40.086 INFO:teuthology.orchestra.run.smithi189.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.5 MB/s | 548 kB 00:00 2024-04-20T21:16:40.136 INFO:teuthology.orchestra.run.smithi189.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 1.1 MB/s | 58 kB 00:00 2024-04-20T21:16:40.186 INFO:teuthology.orchestra.run.smithi189.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 1.0 MB/s | 51 kB 00:00 2024-04-20T21:16:40.245 INFO:teuthology.orchestra.run.smithi189.stdout:(5/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 384 kB/s | 248 kB 00:00 2024-04-20T21:16:41.031 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-04-20T21:16:41.066 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.0-0.el9.noarc 12/12 2024-04-20T21:16:42.146 INFO:teuthology.orchestra.run.smithi189.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 2.3 MB/s | 6.0 MB 00:02 2024-04-20T21:16:42.149 INFO:teuthology.orchestra.run.smithi189.stdout:-------------------------------------------------------------------------------- 2024-04-20T21:16:42.150 INFO:teuthology.orchestra.run.smithi189.stdout:Total 2.2 MB/s | 7.0 MB 00:03 2024-04-20T21:16:42.276 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction check 2024-04-20T21:16:42.315 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction check succeeded. 2024-04-20T21:16:42.315 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction test 2024-04-20T21:16:42.469 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction test succeeded. 2024-04-20T21:16:42.469 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction 2024-04-20T21:16:42.890 INFO:teuthology.orchestra.run.smithi189.stdout: Preparing : 1/1 2024-04-20T21:16:43.112 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-04-20T21:16:43.170 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-04-20T21:16:43.609 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.0-0.el9.noarc 12/12 2024-04-20T21:16:43.609 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.0-0.el9.noarc 1/12 2024-04-20T21:16:43.610 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-04-20T21:16:43.610 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-04-20T21:16:43.610 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-04-20T21:16:43.610 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-04-20T21:16:43.610 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-04-20T21:16:43.610 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-04-20T21:16:43.610 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-04-20T21:16:43.610 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-04-20T21:16:43.610 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-04-20T21:16:43.610 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-04-20T21:16:43.837 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-04-20T21:16:43.943 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-04-20T21:16:43.986 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-04-20T21:16:43.987 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:16:43.987 INFO:teuthology.orchestra.run.smithi062.stdout:Installed: 2024-04-20T21:16:43.987 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-diskprediction-local-2:18.2.0-0.el9.noarch 2024-04-20T21:16:43.987 INFO:teuthology.orchestra.run.smithi062.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-20T21:16:43.987 INFO:teuthology.orchestra.run.smithi062.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-20T21:16:43.987 INFO:teuthology.orchestra.run.smithi062.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-20T21:16:43.987 INFO:teuthology.orchestra.run.smithi062.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-20T21:16:43.987 INFO:teuthology.orchestra.run.smithi062.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-20T21:16:43.987 INFO:teuthology.orchestra.run.smithi062.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-20T21:16:43.987 INFO:teuthology.orchestra.run.smithi062.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-20T21:16:43.987 INFO:teuthology.orchestra.run.smithi062.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-20T21:16:43.988 INFO:teuthology.orchestra.run.smithi062.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-20T21:16:43.988 INFO:teuthology.orchestra.run.smithi062.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-20T21:16:43.988 INFO:teuthology.orchestra.run.smithi062.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-20T21:16:43.988 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:16:43.988 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2024-04-20T21:16:44.015 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-04-20T21:16:44.042 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mgr-cephadm-2:18.2.0-0.el9.noarch 6/6 2024-04-20T21:16:44.425 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install ceph-mgr-rook 2024-04-20T21:16:44.831 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.0-0.el9.noarch 6/6 2024-04-20T21:16:44.831 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mgr-cephadm-2:18.2.0-0.el9.noarch 1/6 2024-04-20T21:16:44.831 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-04-20T21:16:44.831 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-04-20T21:16:44.831 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-04-20T21:16:44.831 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-04-20T21:16:45.129 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-04-20T21:16:45.129 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-20T21:16:45.129 INFO:teuthology.orchestra.run.smithi189.stdout:Installed: 2024-04-20T21:16:45.129 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-cephadm-2:18.2.0-0.el9.noarch 2024-04-20T21:16:45.129 INFO:teuthology.orchestra.run.smithi189.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-20T21:16:45.130 INFO:teuthology.orchestra.run.smithi189.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-20T21:16:45.130 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-20T21:16:45.130 INFO:teuthology.orchestra.run.smithi189.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-20T21:16:45.130 INFO:teuthology.orchestra.run.smithi189.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-20T21:16:45.130 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-20T21:16:45.130 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-04-20T21:16:45.247 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:01:25 ago on Sat 20 Apr 2024 09:15:20 PM UTC. 2024-04-20T21:16:45.377 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2024-04-20T21:16:45.379 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-04-20T21:16:45.379 INFO:teuthology.orchestra.run.smithi062.stdout: Package Arch Version Repository Size 2024-04-20T21:16:45.379 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-04-20T21:16:45.380 INFO:teuthology.orchestra.run.smithi062.stdout:Installing: 2024-04-20T21:16:45.380 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-rook noarch 2:18.2.0-0.el9 ceph-noarch 47 k 2024-04-20T21:16:45.380 INFO:teuthology.orchestra.run.smithi062.stdout:Installing dependencies: 2024-04-20T21:16:45.380 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-20T21:16:45.380 INFO:teuthology.orchestra.run.smithi062.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-20T21:16:45.380 INFO:teuthology.orchestra.run.smithi062.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-20T21:16:45.380 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-20T21:16:45.380 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-20T21:16:45.380 INFO:teuthology.orchestra.run.smithi062.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-20T21:16:45.380 INFO:teuthology.orchestra.run.smithi062.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-20T21:16:45.381 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-20T21:16:45.381 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-20T21:16:45.381 INFO:teuthology.orchestra.run.smithi062.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-20T21:16:45.381 INFO:teuthology.orchestra.run.smithi062.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-20T21:16:45.381 INFO:teuthology.orchestra.run.smithi062.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-20T21:16:45.381 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:16:45.381 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction Summary 2024-04-20T21:16:45.381 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-04-20T21:16:45.381 INFO:teuthology.orchestra.run.smithi062.stdout:Install 13 Packages 2024-04-20T21:16:45.381 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:16:45.382 INFO:teuthology.orchestra.run.smithi062.stdout:Total download size: 2.2 M 2024-04-20T21:16:45.382 INFO:teuthology.orchestra.run.smithi062.stdout:Installed size: 26 M 2024-04-20T21:16:45.382 INFO:teuthology.orchestra.run.smithi062.stdout:Downloading Packages: 2024-04-20T21:16:45.439 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install ceph-fuse 2024-04-20T21:16:46.025 INFO:teuthology.orchestra.run.smithi062.stdout:(1/13): ceph-mgr-rook-18.2.0-0.el9.noarch.rpm 424 kB/s | 47 kB 00:00 2024-04-20T21:16:46.075 INFO:teuthology.orchestra.run.smithi062.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 119 kB/s | 19 kB 00:00 2024-04-20T21:16:46.100 INFO:teuthology.orchestra.run.smithi062.stdout:(3/13): python3-jsonpatch-1.21-16.el9.noarch.rp 137 kB/s | 26 kB 00:00 2024-04-20T21:16:46.192 INFO:teuthology.orchestra.run.smithi062.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.3 MB/s | 159 kB 00:00 2024-04-20T21:16:46.235 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:01:49 ago on Sat 20 Apr 2024 09:14:57 PM UTC. 2024-04-20T21:16:46.250 INFO:teuthology.orchestra.run.smithi062.stdout:(5/13): python3-requests-oauthlib-1.3.0-12.el9. 926 kB/s | 54 kB 00:00 2024-04-20T21:16:46.276 INFO:teuthology.orchestra.run.smithi062.stdout:(6/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 889 kB/s | 222 kB 00:00 2024-04-20T21:16:46.309 INFO:teuthology.orchestra.run.smithi062.stdout:(7/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.3 MB/s | 279 kB 00:00 2024-04-20T21:16:46.363 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-04-20T21:16:46.364 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-20T21:16:46.364 INFO:teuthology.orchestra.run.smithi189.stdout: Package Architecture Version Repository Size 2024-04-20T21:16:46.364 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-20T21:16:46.364 INFO:teuthology.orchestra.run.smithi189.stdout:Installing: 2024-04-20T21:16:46.364 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-fuse x86_64 2:18.2.0-0.el9 ceph 835 k 2024-04-20T21:16:46.364 INFO:teuthology.orchestra.run.smithi189.stdout:Installing dependencies: 2024-04-20T21:16:46.364 INFO:teuthology.orchestra.run.smithi189.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-20T21:16:46.364 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-20T21:16:46.364 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction Summary 2024-04-20T21:16:46.364 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-20T21:16:46.365 INFO:teuthology.orchestra.run.smithi189.stdout:Install 2 Packages 2024-04-20T21:16:46.365 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-20T21:16:46.365 INFO:teuthology.orchestra.run.smithi189.stdout:Total download size: 915 k 2024-04-20T21:16:46.365 INFO:teuthology.orchestra.run.smithi189.stdout:Installed size: 2.7 M 2024-04-20T21:16:46.365 INFO:teuthology.orchestra.run.smithi189.stdout:Downloading Packages: 2024-04-20T21:16:46.601 INFO:teuthology.orchestra.run.smithi062.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 46 kB/s | 15 kB 00:00 2024-04-20T21:16:46.668 INFO:teuthology.orchestra.run.smithi062.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 77 kB/s | 32 kB 00:00 2024-04-20T21:16:46.760 INFO:teuthology.orchestra.run.smithi062.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 451 kB/s | 203 kB 00:00 2024-04-20T21:16:46.772 INFO:teuthology.orchestra.run.smithi189.stdout:(1/2): ceph-fuse-18.2.0-0.el9.x86_64.rpm 6.6 MB/s | 835 kB 00:00 2024-04-20T21:16:46.844 INFO:teuthology.orchestra.run.smithi062.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 339 kB/s | 59 kB 00:00 2024-04-20T21:16:46.888 INFO:teuthology.orchestra.run.smithi189.stdout:(2/2): fuse-2.9.9-15.el9.x86_64.rpm 335 kB/s | 80 kB 00:00 2024-04-20T21:16:46.890 INFO:teuthology.orchestra.run.smithi189.stdout:-------------------------------------------------------------------------------- 2024-04-20T21:16:46.891 INFO:teuthology.orchestra.run.smithi189.stdout:Total 1.7 MB/s | 915 kB 00:00 2024-04-20T21:16:46.933 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction check 2024-04-20T21:16:46.990 INFO:teuthology.orchestra.run.smithi062.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 2.6 MB/s | 1.0 MB 00:00 2024-04-20T21:16:47.003 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction check succeeded. 2024-04-20T21:16:47.003 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction test 2024-04-20T21:16:47.069 INFO:teuthology.orchestra.run.smithi062.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 292 kB/s | 90 kB 00:00 2024-04-20T21:16:47.076 INFO:teuthology.orchestra.run.smithi062.stdout:-------------------------------------------------------------------------------- 2024-04-20T21:16:47.077 INFO:teuthology.orchestra.run.smithi062.stdout:Total 1.3 MB/s | 2.2 MB 00:01 2024-04-20T21:16:47.115 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction test succeeded. 2024-04-20T21:16:47.116 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction 2024-04-20T21:16:47.294 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction check 2024-04-20T21:16:47.342 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction check succeeded. 2024-04-20T21:16:47.342 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction test 2024-04-20T21:16:47.342 INFO:teuthology.orchestra.run.smithi189.stdout: Preparing : 1/1 2024-04-20T21:16:47.499 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction test succeeded. 2024-04-20T21:16:47.501 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction 2024-04-20T21:16:47.502 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-04-20T21:16:47.574 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-fuse-2:18.2.0-0.el9.x86_64 2/2 2024-04-20T21:16:48.234 INFO:teuthology.orchestra.run.smithi062.stdout: Preparing : 1/1 2024-04-20T21:16:48.431 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-fuse-2:18.2.0-0.el9.x86_64 2/2 2024-04-20T21:16:48.431 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-fuse-2:18.2.0-0.el9.x86_64 1/2 2024-04-20T21:16:48.452 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-04-20T21:16:48.522 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-04-20T21:16:48.596 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-04-20T21:16:48.658 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-04-20T21:16:48.690 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-04-20T21:16:48.690 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-20T21:16:48.690 INFO:teuthology.orchestra.run.smithi189.stdout:Installed: 2024-04-20T21:16:48.690 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-fuse-2:18.2.0-0.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-04-20T21:16:48.690 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-20T21:16:48.691 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-04-20T21:16:48.715 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-04-20T21:16:48.861 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-04-20T21:16:48.990 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install librados-devel 2024-04-20T21:16:49.000 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-04-20T21:16:49.074 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-04-20T21:16:49.765 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:01:52 ago on Sat 20 Apr 2024 09:14:57 PM UTC. 2024-04-20T21:16:49.901 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-04-20T21:16:49.901 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-20T21:16:49.901 INFO:teuthology.orchestra.run.smithi189.stdout: Package Architecture Version Repository Size 2024-04-20T21:16:49.901 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-20T21:16:49.901 INFO:teuthology.orchestra.run.smithi189.stdout:Installing: 2024-04-20T21:16:49.901 INFO:teuthology.orchestra.run.smithi189.stdout: librados-devel x86_64 2:18.2.0-0.el9 ceph 126 k 2024-04-20T21:16:49.901 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-20T21:16:49.902 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction Summary 2024-04-20T21:16:49.902 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-20T21:16:49.902 INFO:teuthology.orchestra.run.smithi189.stdout:Install 1 Package 2024-04-20T21:16:49.902 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-20T21:16:49.902 INFO:teuthology.orchestra.run.smithi189.stdout:Total download size: 126 k 2024-04-20T21:16:49.902 INFO:teuthology.orchestra.run.smithi189.stdout:Installed size: 456 k 2024-04-20T21:16:49.903 INFO:teuthology.orchestra.run.smithi189.stdout:Downloading Packages: 2024-04-20T21:16:49.977 INFO:teuthology.orchestra.run.smithi189.stdout:librados-devel-18.2.0-0.el9.x86_64.rpm 1.7 MB/s | 126 kB 00:00 2024-04-20T21:16:49.978 INFO:teuthology.orchestra.run.smithi189.stdout:-------------------------------------------------------------------------------- 2024-04-20T21:16:49.978 INFO:teuthology.orchestra.run.smithi189.stdout:Total 1.6 MB/s | 126 kB 00:00 2024-04-20T21:16:49.978 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction check 2024-04-20T21:16:49.985 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction check succeeded. 2024-04-20T21:16:49.985 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction test 2024-04-20T21:16:50.040 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-04-20T21:16:50.058 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction test succeeded. 2024-04-20T21:16:50.059 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction 2024-04-20T21:16:50.089 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-04-20T21:16:50.123 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-04-20T21:16:50.174 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-04-20T21:16:50.198 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-mgr-rook-2:18.2.0-0.el9.noarch 13/13 2024-04-20T21:16:50.246 INFO:teuthology.orchestra.run.smithi189.stdout: Preparing : 1/1 2024-04-20T21:16:50.388 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : librados-devel-2:18.2.0-0.el9.x86_64 1/1 2024-04-20T21:16:50.893 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: librados-devel-2:18.2.0-0.el9.x86_64 1/1 2024-04-20T21:16:51.022 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.0-0.el9.noarch 13/13 2024-04-20T21:16:51.022 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-mgr-rook-2:18.2.0-0.el9.noarch 1/13 2024-04-20T21:16:51.022 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-04-20T21:16:51.023 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-04-20T21:16:51.023 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-04-20T21:16:51.023 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-04-20T21:16:51.023 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-04-20T21:16:51.023 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-04-20T21:16:51.023 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-04-20T21:16:51.023 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-04-20T21:16:51.023 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-04-20T21:16:51.023 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-04-20T21:16:51.023 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-04-20T21:16:51.193 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librados-devel-2:18.2.0-0.el9.x86_64 1/1 2024-04-20T21:16:51.193 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-20T21:16:51.193 INFO:teuthology.orchestra.run.smithi189.stdout:Installed: 2024-04-20T21:16:51.193 INFO:teuthology.orchestra.run.smithi189.stdout: librados-devel-2:18.2.0-0.el9.x86_64 2024-04-20T21:16:51.193 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-20T21:16:51.193 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-04-20T21:16:51.435 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-04-20T21:16:51.435 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:16:51.435 INFO:teuthology.orchestra.run.smithi062.stdout:Installed: 2024-04-20T21:16:51.436 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-rook-2:18.2.0-0.el9.noarch 2024-04-20T21:16:51.436 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-20T21:16:51.436 INFO:teuthology.orchestra.run.smithi062.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-20T21:16:51.436 INFO:teuthology.orchestra.run.smithi062.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-20T21:16:51.436 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-20T21:16:51.436 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-20T21:16:51.436 INFO:teuthology.orchestra.run.smithi062.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-20T21:16:51.436 INFO:teuthology.orchestra.run.smithi062.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-20T21:16:51.436 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-20T21:16:51.436 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-20T21:16:51.436 INFO:teuthology.orchestra.run.smithi062.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-20T21:16:51.436 INFO:teuthology.orchestra.run.smithi062.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-20T21:16:51.436 INFO:teuthology.orchestra.run.smithi062.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-20T21:16:51.436 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:16:51.436 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2024-04-20T21:16:51.473 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install libcephfs2 2024-04-20T21:16:51.810 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install ceph-mgr-cephadm 2024-04-20T21:16:52.251 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:01:55 ago on Sat 20 Apr 2024 09:14:57 PM UTC. 2024-04-20T21:16:52.348 INFO:teuthology.orchestra.run.smithi189.stdout:Package libcephfs2-2:18.2.0-0.el9.x86_64 is already installed. 2024-04-20T21:16:52.393 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-04-20T21:16:52.394 INFO:teuthology.orchestra.run.smithi189.stdout:Nothing to do. 2024-04-20T21:16:52.395 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-04-20T21:16:52.485 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install libcephfs-devel 2024-04-20T21:16:52.621 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:01:32 ago on Sat 20 Apr 2024 09:15:20 PM UTC. 2024-04-20T21:16:52.750 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2024-04-20T21:16:52.751 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-04-20T21:16:52.751 INFO:teuthology.orchestra.run.smithi062.stdout: Package Arch Version Repository Size 2024-04-20T21:16:52.751 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-04-20T21:16:52.751 INFO:teuthology.orchestra.run.smithi062.stdout:Installing: 2024-04-20T21:16:52.752 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-cephadm noarch 2:18.2.0-0.el9 ceph-noarch 127 k 2024-04-20T21:16:52.752 INFO:teuthology.orchestra.run.smithi062.stdout:Installing dependencies: 2024-04-20T21:16:52.752 INFO:teuthology.orchestra.run.smithi062.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-20T21:16:52.752 INFO:teuthology.orchestra.run.smithi062.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-20T21:16:52.752 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-20T21:16:52.752 INFO:teuthology.orchestra.run.smithi062.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-20T21:16:52.753 INFO:teuthology.orchestra.run.smithi062.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-20T21:16:52.753 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:16:52.753 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction Summary 2024-04-20T21:16:52.753 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-04-20T21:16:52.753 INFO:teuthology.orchestra.run.smithi062.stdout:Install 6 Packages 2024-04-20T21:16:52.753 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:16:52.754 INFO:teuthology.orchestra.run.smithi062.stdout:Total download size: 7.0 M 2024-04-20T21:16:52.755 INFO:teuthology.orchestra.run.smithi062.stdout:Installed size: 33 M 2024-04-20T21:16:52.755 INFO:teuthology.orchestra.run.smithi062.stdout:Downloading Packages: 2024-04-20T21:16:53.205 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:01:56 ago on Sat 20 Apr 2024 09:14:57 PM UTC. 2024-04-20T21:16:53.335 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-04-20T21:16:53.335 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-20T21:16:53.335 INFO:teuthology.orchestra.run.smithi189.stdout: Package Architecture Version Repository Size 2024-04-20T21:16:53.335 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-20T21:16:53.335 INFO:teuthology.orchestra.run.smithi189.stdout:Installing: 2024-04-20T21:16:53.335 INFO:teuthology.orchestra.run.smithi189.stdout: libcephfs-devel x86_64 2:18.2.0-0.el9 ceph 30 k 2024-04-20T21:16:53.335 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-20T21:16:53.336 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction Summary 2024-04-20T21:16:53.336 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-20T21:16:53.336 INFO:teuthology.orchestra.run.smithi189.stdout:Install 1 Package 2024-04-20T21:16:53.336 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-20T21:16:53.336 INFO:teuthology.orchestra.run.smithi189.stdout:Total download size: 30 k 2024-04-20T21:16:53.336 INFO:teuthology.orchestra.run.smithi189.stdout:Installed size: 137 k 2024-04-20T21:16:53.336 INFO:teuthology.orchestra.run.smithi189.stdout:Downloading Packages: 2024-04-20T21:16:53.362 INFO:teuthology.orchestra.run.smithi062.stdout:(1/6): ceph-mgr-cephadm-18.2.0-0.el9.noarch.rpm 1.1 MB/s | 127 kB 00:00 2024-04-20T21:16:53.414 INFO:teuthology.orchestra.run.smithi189.stdout:libcephfs-devel-18.2.0-0.el9.x86_64.rpm 394 kB/s | 30 kB 00:00 2024-04-20T21:16:53.415 INFO:teuthology.orchestra.run.smithi189.stdout:-------------------------------------------------------------------------------- 2024-04-20T21:16:53.415 INFO:teuthology.orchestra.run.smithi189.stdout:Total 385 kB/s | 30 kB 00:00 2024-04-20T21:16:53.415 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction check 2024-04-20T21:16:53.420 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction check succeeded. 2024-04-20T21:16:53.420 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction test 2024-04-20T21:16:53.444 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction test succeeded. 2024-04-20T21:16:53.444 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction 2024-04-20T21:16:53.546 INFO:teuthology.orchestra.run.smithi062.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 832 kB/s | 248 kB 00:00 2024-04-20T21:16:53.569 INFO:teuthology.orchestra.run.smithi189.stdout: Preparing : 1/1 2024-04-20T21:16:53.713 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libcephfs-devel-2:18.2.0-0.el9.x86_64 1/1 2024-04-20T21:16:53.938 INFO:teuthology.orchestra.run.smithi062.stdout:(3/6): python3-babel-2.9.1-2.el9.noarch.rpm 8.6 MB/s | 6.0 MB 00:00 2024-04-20T21:16:53.963 INFO:teuthology.orchestra.run.smithi062.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 140 kB/s | 58 kB 00:00 2024-04-20T21:16:54.005 INFO:teuthology.orchestra.run.smithi062.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 855 kB/s | 548 kB 00:00 2024-04-20T21:16:54.140 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: libcephfs-devel-2:18.2.0-0.el9.x86_64 1/1 2024-04-20T21:16:54.305 INFO:teuthology.orchestra.run.smithi062.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 139 kB/s | 51 kB 00:00 2024-04-20T21:16:54.311 INFO:teuthology.orchestra.run.smithi062.stdout:-------------------------------------------------------------------------------- 2024-04-20T21:16:54.312 INFO:teuthology.orchestra.run.smithi062.stdout:Total 4.5 MB/s | 7.0 MB 00:01 2024-04-20T21:16:54.376 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libcephfs-devel-2:18.2.0-0.el9.x86_64 1/1 2024-04-20T21:16:54.377 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-20T21:16:54.377 INFO:teuthology.orchestra.run.smithi189.stdout:Installed: 2024-04-20T21:16:54.377 INFO:teuthology.orchestra.run.smithi189.stdout: libcephfs-devel-2:18.2.0-0.el9.x86_64 2024-04-20T21:16:54.377 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-20T21:16:54.377 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-04-20T21:16:54.446 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction check 2024-04-20T21:16:54.483 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction check succeeded. 2024-04-20T21:16:54.483 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction test 2024-04-20T21:16:54.619 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install librados2 2024-04-20T21:16:54.641 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction test succeeded. 2024-04-20T21:16:54.642 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction 2024-04-20T21:16:55.030 INFO:teuthology.orchestra.run.smithi062.stdout: Preparing : 1/1 2024-04-20T21:16:55.250 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-04-20T21:16:55.313 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-04-20T21:16:55.343 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:01:58 ago on Sat 20 Apr 2024 09:14:57 PM UTC. 2024-04-20T21:16:55.441 INFO:teuthology.orchestra.run.smithi189.stdout:Package librados2-2:18.2.0-0.el9.x86_64 is already installed. 2024-04-20T21:16:55.483 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-04-20T21:16:55.484 INFO:teuthology.orchestra.run.smithi189.stdout:Nothing to do. 2024-04-20T21:16:55.484 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-04-20T21:16:55.567 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install librbd1 2024-04-20T21:16:56.020 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-04-20T21:16:56.131 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-04-20T21:16:56.211 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-04-20T21:16:56.244 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-mgr-cephadm-2:18.2.0-0.el9.noarch 6/6 2024-04-20T21:16:56.308 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:01:59 ago on Sat 20 Apr 2024 09:14:57 PM UTC. 2024-04-20T21:16:56.394 INFO:teuthology.orchestra.run.smithi189.stdout:Package librbd1-2:18.2.0-0.el9.x86_64 is already installed. 2024-04-20T21:16:56.436 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-04-20T21:16:56.437 INFO:teuthology.orchestra.run.smithi189.stdout:Nothing to do. 2024-04-20T21:16:56.437 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-04-20T21:16:56.520 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install python3-rados 2024-04-20T21:16:57.046 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.0-0.el9.noarch 6/6 2024-04-20T21:16:57.046 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-mgr-cephadm-2:18.2.0-0.el9.noarch 1/6 2024-04-20T21:16:57.046 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-04-20T21:16:57.046 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-04-20T21:16:57.046 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-04-20T21:16:57.046 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-04-20T21:16:57.257 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:02:00 ago on Sat 20 Apr 2024 09:14:57 PM UTC. 2024-04-20T21:16:57.343 INFO:teuthology.orchestra.run.smithi189.stdout:Package python3-rados-2:18.2.0-0.el9.x86_64 is already installed. 2024-04-20T21:16:57.361 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-04-20T21:16:57.361 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:16:57.361 INFO:teuthology.orchestra.run.smithi062.stdout:Installed: 2024-04-20T21:16:57.361 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-cephadm-2:18.2.0-0.el9.noarch 2024-04-20T21:16:57.361 INFO:teuthology.orchestra.run.smithi062.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-20T21:16:57.361 INFO:teuthology.orchestra.run.smithi062.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-20T21:16:57.361 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-20T21:16:57.361 INFO:teuthology.orchestra.run.smithi062.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-20T21:16:57.361 INFO:teuthology.orchestra.run.smithi062.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-20T21:16:57.362 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:16:57.362 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2024-04-20T21:16:57.385 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-04-20T21:16:57.386 INFO:teuthology.orchestra.run.smithi189.stdout:Nothing to do. 2024-04-20T21:16:57.386 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-04-20T21:16:57.472 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install python3-rgw 2024-04-20T21:16:57.646 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install ceph-fuse 2024-04-20T21:16:58.221 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:02:01 ago on Sat 20 Apr 2024 09:14:57 PM UTC. 2024-04-20T21:16:58.305 INFO:teuthology.orchestra.run.smithi189.stdout:Package python3-rgw-2:18.2.0-0.el9.x86_64 is already installed. 2024-04-20T21:16:58.348 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-04-20T21:16:58.349 INFO:teuthology.orchestra.run.smithi189.stdout:Nothing to do. 2024-04-20T21:16:58.349 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-04-20T21:16:58.428 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install python3-cephfs 2024-04-20T21:16:58.441 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:01:38 ago on Sat 20 Apr 2024 09:15:20 PM UTC. 2024-04-20T21:16:58.588 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2024-04-20T21:16:58.588 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-04-20T21:16:58.589 INFO:teuthology.orchestra.run.smithi062.stdout: Package Architecture Version Repository Size 2024-04-20T21:16:58.589 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-04-20T21:16:58.589 INFO:teuthology.orchestra.run.smithi062.stdout:Installing: 2024-04-20T21:16:58.589 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-fuse x86_64 2:18.2.0-0.el9 ceph 835 k 2024-04-20T21:16:58.589 INFO:teuthology.orchestra.run.smithi062.stdout:Installing dependencies: 2024-04-20T21:16:58.589 INFO:teuthology.orchestra.run.smithi062.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-20T21:16:58.589 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:16:58.589 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction Summary 2024-04-20T21:16:58.589 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-04-20T21:16:58.589 INFO:teuthology.orchestra.run.smithi062.stdout:Install 2 Packages 2024-04-20T21:16:58.589 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:16:58.590 INFO:teuthology.orchestra.run.smithi062.stdout:Total download size: 915 k 2024-04-20T21:16:58.590 INFO:teuthology.orchestra.run.smithi062.stdout:Installed size: 2.7 M 2024-04-20T21:16:58.590 INFO:teuthology.orchestra.run.smithi062.stdout:Downloading Packages: 2024-04-20T21:16:58.974 INFO:teuthology.orchestra.run.smithi062.stdout:(1/2): ceph-fuse-18.2.0-0.el9.x86_64.rpm 8.4 MB/s | 835 kB 00:00 2024-04-20T21:16:59.090 INFO:teuthology.orchestra.run.smithi062.stdout:(2/2): fuse-2.9.9-15.el9.x86_64.rpm 376 kB/s | 80 kB 00:00 2024-04-20T21:16:59.092 INFO:teuthology.orchestra.run.smithi062.stdout:-------------------------------------------------------------------------------- 2024-04-20T21:16:59.093 INFO:teuthology.orchestra.run.smithi062.stdout:Total 1.8 MB/s | 915 kB 00:00 2024-04-20T21:16:59.133 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction check 2024-04-20T21:16:59.158 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:02:02 ago on Sat 20 Apr 2024 09:14:57 PM UTC. 2024-04-20T21:16:59.213 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction check succeeded. 2024-04-20T21:16:59.213 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction test 2024-04-20T21:16:59.244 INFO:teuthology.orchestra.run.smithi189.stdout:Package python3-cephfs-2:18.2.0-0.el9.x86_64 is already installed. 2024-04-20T21:16:59.285 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-04-20T21:16:59.286 INFO:teuthology.orchestra.run.smithi189.stdout:Nothing to do. 2024-04-20T21:16:59.286 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-04-20T21:16:59.331 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction test succeeded. 2024-04-20T21:16:59.332 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction 2024-04-20T21:16:59.369 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install python3-rbd 2024-04-20T21:16:59.605 INFO:teuthology.orchestra.run.smithi062.stdout: Preparing : 1/1 2024-04-20T21:16:59.789 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-04-20T21:16:59.876 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-fuse-2:18.2.0-0.el9.x86_64 2/2 2024-04-20T21:17:00.108 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:02:03 ago on Sat 20 Apr 2024 09:14:57 PM UTC. 2024-04-20T21:17:00.194 INFO:teuthology.orchestra.run.smithi189.stdout:Package python3-rbd-2:18.2.0-0.el9.x86_64 is already installed. 2024-04-20T21:17:00.235 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-04-20T21:17:00.236 INFO:teuthology.orchestra.run.smithi189.stdout:Nothing to do. 2024-04-20T21:17:00.237 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-04-20T21:17:00.316 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install rbd-fuse 2024-04-20T21:17:00.720 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-fuse-2:18.2.0-0.el9.x86_64 2/2 2024-04-20T21:17:00.720 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-fuse-2:18.2.0-0.el9.x86_64 1/2 2024-04-20T21:17:01.034 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-04-20T21:17:01.034 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:17:01.034 INFO:teuthology.orchestra.run.smithi062.stdout:Installed: 2024-04-20T21:17:01.034 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-fuse-2:18.2.0-0.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-04-20T21:17:01.034 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:17:01.034 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2024-04-20T21:17:01.060 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:02:04 ago on Sat 20 Apr 2024 09:14:57 PM UTC. 2024-04-20T21:17:01.200 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-04-20T21:17:01.200 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-20T21:17:01.200 INFO:teuthology.orchestra.run.smithi189.stdout: Package Architecture Version Repository Size 2024-04-20T21:17:01.200 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-20T21:17:01.200 INFO:teuthology.orchestra.run.smithi189.stdout:Installing: 2024-04-20T21:17:01.200 INFO:teuthology.orchestra.run.smithi189.stdout: rbd-fuse x86_64 2:18.2.0-0.el9 ceph 86 k 2024-04-20T21:17:01.201 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-20T21:17:01.201 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction Summary 2024-04-20T21:17:01.201 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-20T21:17:01.201 INFO:teuthology.orchestra.run.smithi189.stdout:Install 1 Package 2024-04-20T21:17:01.201 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-20T21:17:01.201 INFO:teuthology.orchestra.run.smithi189.stdout:Total download size: 86 k 2024-04-20T21:17:01.201 INFO:teuthology.orchestra.run.smithi189.stdout:Installed size: 230 k 2024-04-20T21:17:01.202 INFO:teuthology.orchestra.run.smithi189.stdout:Downloading Packages: 2024-04-20T21:17:01.285 INFO:teuthology.orchestra.run.smithi189.stdout:rbd-fuse-18.2.0-0.el9.x86_64.rpm 1.0 MB/s | 86 kB 00:00 2024-04-20T21:17:01.286 INFO:teuthology.orchestra.run.smithi189.stdout:-------------------------------------------------------------------------------- 2024-04-20T21:17:01.286 INFO:teuthology.orchestra.run.smithi189.stdout:Total 1.0 MB/s | 86 kB 00:00 2024-04-20T21:17:01.286 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction check 2024-04-20T21:17:01.295 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction check succeeded. 2024-04-20T21:17:01.295 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction test 2024-04-20T21:17:01.370 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction test succeeded. 2024-04-20T21:17:01.370 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction 2024-04-20T21:17:01.382 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install librados-devel 2024-04-20T21:17:01.533 INFO:teuthology.orchestra.run.smithi189.stdout: Preparing : 1/1 2024-04-20T21:17:01.642 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : rbd-fuse-2:18.2.0-0.el9.x86_64 1/1 2024-04-20T21:17:02.190 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:01:42 ago on Sat 20 Apr 2024 09:15:20 PM UTC. 2024-04-20T21:17:02.196 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: rbd-fuse-2:18.2.0-0.el9.x86_64 1/1 2024-04-20T21:17:02.325 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2024-04-20T21:17:02.325 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-04-20T21:17:02.325 INFO:teuthology.orchestra.run.smithi062.stdout: Package Architecture Version Repository Size 2024-04-20T21:17:02.326 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-04-20T21:17:02.326 INFO:teuthology.orchestra.run.smithi062.stdout:Installing: 2024-04-20T21:17:02.326 INFO:teuthology.orchestra.run.smithi062.stdout: librados-devel x86_64 2:18.2.0-0.el9 ceph 126 k 2024-04-20T21:17:02.326 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:17:02.326 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction Summary 2024-04-20T21:17:02.326 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-04-20T21:17:02.326 INFO:teuthology.orchestra.run.smithi062.stdout:Install 1 Package 2024-04-20T21:17:02.326 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:17:02.326 INFO:teuthology.orchestra.run.smithi062.stdout:Total download size: 126 k 2024-04-20T21:17:02.326 INFO:teuthology.orchestra.run.smithi062.stdout:Installed size: 456 k 2024-04-20T21:17:02.326 INFO:teuthology.orchestra.run.smithi062.stdout:Downloading Packages: 2024-04-20T21:17:02.403 INFO:teuthology.orchestra.run.smithi062.stdout:librados-devel-18.2.0-0.el9.x86_64.rpm 1.6 MB/s | 126 kB 00:00 2024-04-20T21:17:02.403 INFO:teuthology.orchestra.run.smithi062.stdout:-------------------------------------------------------------------------------- 2024-04-20T21:17:02.404 INFO:teuthology.orchestra.run.smithi062.stdout:Total 1.6 MB/s | 126 kB 00:00 2024-04-20T21:17:02.404 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction check 2024-04-20T21:17:02.411 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction check succeeded. 2024-04-20T21:17:02.411 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction test 2024-04-20T21:17:02.465 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : rbd-fuse-2:18.2.0-0.el9.x86_64 1/1 2024-04-20T21:17:02.466 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-20T21:17:02.466 INFO:teuthology.orchestra.run.smithi189.stdout:Installed: 2024-04-20T21:17:02.466 INFO:teuthology.orchestra.run.smithi189.stdout: rbd-fuse-2:18.2.0-0.el9.x86_64 2024-04-20T21:17:02.466 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-20T21:17:02.466 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-04-20T21:17:02.488 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction test succeeded. 2024-04-20T21:17:02.488 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction 2024-04-20T21:17:02.663 INFO:teuthology.orchestra.run.smithi062.stdout: Preparing : 1/1 2024-04-20T21:17:02.745 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install rbd-mirror 2024-04-20T21:17:02.787 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : librados-devel-2:18.2.0-0.el9.x86_64 1/1 2024-04-20T21:17:03.343 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: librados-devel-2:18.2.0-0.el9.x86_64 1/1 2024-04-20T21:17:03.527 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:02:06 ago on Sat 20 Apr 2024 09:14:57 PM UTC. 2024-04-20T21:17:03.609 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : librados-devel-2:18.2.0-0.el9.x86_64 1/1 2024-04-20T21:17:03.609 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:17:03.609 INFO:teuthology.orchestra.run.smithi062.stdout:Installed: 2024-04-20T21:17:03.609 INFO:teuthology.orchestra.run.smithi062.stdout: librados-devel-2:18.2.0-0.el9.x86_64 2024-04-20T21:17:03.609 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:17:03.609 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2024-04-20T21:17:03.669 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-04-20T21:17:03.669 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-20T21:17:03.670 INFO:teuthology.orchestra.run.smithi189.stdout: Package Architecture Version Repository Size 2024-04-20T21:17:03.670 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-20T21:17:03.670 INFO:teuthology.orchestra.run.smithi189.stdout:Installing: 2024-04-20T21:17:03.670 INFO:teuthology.orchestra.run.smithi189.stdout: rbd-mirror x86_64 2:18.2.0-0.el9 ceph 3.0 M 2024-04-20T21:17:03.670 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-20T21:17:03.670 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction Summary 2024-04-20T21:17:03.670 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-20T21:17:03.670 INFO:teuthology.orchestra.run.smithi189.stdout:Install 1 Package 2024-04-20T21:17:03.670 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-20T21:17:03.671 INFO:teuthology.orchestra.run.smithi189.stdout:Total download size: 3.0 M 2024-04-20T21:17:03.671 INFO:teuthology.orchestra.run.smithi189.stdout:Installed size: 12 M 2024-04-20T21:17:03.671 INFO:teuthology.orchestra.run.smithi189.stdout:Downloading Packages: 2024-04-20T21:17:03.788 INFO:teuthology.orchestra.run.smithi189.stdout:rbd-mirror-18.2.0-0.el9.x86_64.rpm 26 MB/s | 3.0 MB 00:00 2024-04-20T21:17:03.789 INFO:teuthology.orchestra.run.smithi189.stdout:-------------------------------------------------------------------------------- 2024-04-20T21:17:03.789 INFO:teuthology.orchestra.run.smithi189.stdout:Total 25 MB/s | 3.0 MB 00:00 2024-04-20T21:17:03.790 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction check 2024-04-20T21:17:03.801 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction check succeeded. 2024-04-20T21:17:03.801 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction test 2024-04-20T21:17:03.893 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction test succeeded. 2024-04-20T21:17:03.893 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction 2024-04-20T21:17:03.944 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install libcephfs2 2024-04-20T21:17:04.366 INFO:teuthology.orchestra.run.smithi189.stdout: Preparing : 1/1 2024-04-20T21:17:04.440 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : rbd-mirror-2:18.2.0-0.el9.x86_64 1/1 2024-04-20T21:17:04.538 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: rbd-mirror-2:18.2.0-0.el9.x86_64 1/1 2024-04-20T21:17:04.538 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-20T21:17:04.538 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-20T21:17:04.538 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-20T21:17:04.538 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-20T21:17:04.538 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-20T21:17:04.721 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:01:44 ago on Sat 20 Apr 2024 09:15:20 PM UTC. 2024-04-20T21:17:04.805 INFO:teuthology.orchestra.run.smithi062.stdout:Package libcephfs2-2:18.2.0-0.el9.x86_64 is already installed. 2024-04-20T21:17:04.849 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2024-04-20T21:17:04.851 INFO:teuthology.orchestra.run.smithi062.stdout:Nothing to do. 2024-04-20T21:17:04.851 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2024-04-20T21:17:04.926 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install libcephfs-devel 2024-04-20T21:17:05.729 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:01:45 ago on Sat 20 Apr 2024 09:15:20 PM UTC. 2024-04-20T21:17:05.863 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2024-04-20T21:17:05.863 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-04-20T21:17:05.864 INFO:teuthology.orchestra.run.smithi062.stdout: Package Architecture Version Repository Size 2024-04-20T21:17:05.864 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-04-20T21:17:05.864 INFO:teuthology.orchestra.run.smithi062.stdout:Installing: 2024-04-20T21:17:05.864 INFO:teuthology.orchestra.run.smithi062.stdout: libcephfs-devel x86_64 2:18.2.0-0.el9 ceph 30 k 2024-04-20T21:17:05.864 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:17:05.864 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction Summary 2024-04-20T21:17:05.864 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-04-20T21:17:05.864 INFO:teuthology.orchestra.run.smithi062.stdout:Install 1 Package 2024-04-20T21:17:05.864 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:17:05.864 INFO:teuthology.orchestra.run.smithi062.stdout:Total download size: 30 k 2024-04-20T21:17:05.864 INFO:teuthology.orchestra.run.smithi062.stdout:Installed size: 137 k 2024-04-20T21:17:05.864 INFO:teuthology.orchestra.run.smithi062.stdout:Downloading Packages: 2024-04-20T21:17:05.898 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : rbd-mirror-2:18.2.0-0.el9.x86_64 1/1 2024-04-20T21:17:05.898 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-20T21:17:05.898 INFO:teuthology.orchestra.run.smithi189.stdout:Installed: 2024-04-20T21:17:05.898 INFO:teuthology.orchestra.run.smithi189.stdout: rbd-mirror-2:18.2.0-0.el9.x86_64 2024-04-20T21:17:05.898 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-20T21:17:05.898 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-04-20T21:17:05.940 INFO:teuthology.orchestra.run.smithi062.stdout:libcephfs-devel-18.2.0-0.el9.x86_64.rpm 407 kB/s | 30 kB 00:00 2024-04-20T21:17:05.940 INFO:teuthology.orchestra.run.smithi062.stdout:-------------------------------------------------------------------------------- 2024-04-20T21:17:05.941 INFO:teuthology.orchestra.run.smithi062.stdout:Total 397 kB/s | 30 kB 00:00 2024-04-20T21:17:05.941 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction check 2024-04-20T21:17:05.946 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction check succeeded. 2024-04-20T21:17:05.946 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction test 2024-04-20T21:17:05.969 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction test succeeded. 2024-04-20T21:17:05.970 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction 2024-04-20T21:17:06.139 INFO:teuthology.orchestra.run.smithi062.stdout: Preparing : 1/1 2024-04-20T21:17:06.199 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install rbd-nbd 2024-04-20T21:17:06.308 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libcephfs-devel-2:18.2.0-0.el9.x86_64 1/1 2024-04-20T21:17:06.739 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: libcephfs-devel-2:18.2.0-0.el9.x86_64 1/1 2024-04-20T21:17:06.974 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:02:09 ago on Sat 20 Apr 2024 09:14:57 PM UTC. 2024-04-20T21:17:06.990 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libcephfs-devel-2:18.2.0-0.el9.x86_64 1/1 2024-04-20T21:17:06.990 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:17:06.990 INFO:teuthology.orchestra.run.smithi062.stdout:Installed: 2024-04-20T21:17:06.990 INFO:teuthology.orchestra.run.smithi062.stdout: libcephfs-devel-2:18.2.0-0.el9.x86_64 2024-04-20T21:17:06.990 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:17:06.990 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2024-04-20T21:17:07.096 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-04-20T21:17:07.098 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-20T21:17:07.098 INFO:teuthology.orchestra.run.smithi189.stdout: Package Architecture Version Repository Size 2024-04-20T21:17:07.098 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-20T21:17:07.098 INFO:teuthology.orchestra.run.smithi189.stdout:Installing: 2024-04-20T21:17:07.098 INFO:teuthology.orchestra.run.smithi189.stdout: rbd-nbd x86_64 2:18.2.0-0.el9 ceph 169 k 2024-04-20T21:17:07.099 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-20T21:17:07.099 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction Summary 2024-04-20T21:17:07.099 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-20T21:17:07.099 INFO:teuthology.orchestra.run.smithi189.stdout:Install 1 Package 2024-04-20T21:17:07.099 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-20T21:17:07.099 INFO:teuthology.orchestra.run.smithi189.stdout:Total download size: 169 k 2024-04-20T21:17:07.100 INFO:teuthology.orchestra.run.smithi189.stdout:Installed size: 490 k 2024-04-20T21:17:07.100 INFO:teuthology.orchestra.run.smithi189.stdout:Downloading Packages: 2024-04-20T21:17:07.168 INFO:teuthology.orchestra.run.smithi189.stdout:rbd-nbd-18.2.0-0.el9.x86_64.rpm 2.4 MB/s | 169 kB 00:00 2024-04-20T21:17:07.168 INFO:teuthology.orchestra.run.smithi189.stdout:-------------------------------------------------------------------------------- 2024-04-20T21:17:07.168 INFO:teuthology.orchestra.run.smithi189.stdout:Total 2.3 MB/s | 169 kB 00:00 2024-04-20T21:17:07.169 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction check 2024-04-20T21:17:07.178 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction check succeeded. 2024-04-20T21:17:07.178 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction test 2024-04-20T21:17:07.249 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install librados2 2024-04-20T21:17:07.250 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction test succeeded. 2024-04-20T21:17:07.251 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction 2024-04-20T21:17:07.438 INFO:teuthology.orchestra.run.smithi189.stdout: Preparing : 1/1 2024-04-20T21:17:07.564 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : rbd-nbd-2:18.2.0-0.el9.x86_64 1/1 2024-04-20T21:17:08.029 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:01:48 ago on Sat 20 Apr 2024 09:15:20 PM UTC. 2024-04-20T21:17:08.053 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: rbd-nbd-2:18.2.0-0.el9.x86_64 1/1 2024-04-20T21:17:08.117 INFO:teuthology.orchestra.run.smithi062.stdout:Package librados2-2:18.2.0-0.el9.x86_64 is already installed. 2024-04-20T21:17:08.163 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2024-04-20T21:17:08.165 INFO:teuthology.orchestra.run.smithi062.stdout:Nothing to do. 2024-04-20T21:17:08.165 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2024-04-20T21:17:08.242 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install librbd1 2024-04-20T21:17:08.378 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : rbd-nbd-2:18.2.0-0.el9.x86_64 1/1 2024-04-20T21:17:08.378 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-20T21:17:08.378 INFO:teuthology.orchestra.run.smithi189.stdout:Installed: 2024-04-20T21:17:08.378 INFO:teuthology.orchestra.run.smithi189.stdout: rbd-nbd-2:18.2.0-0.el9.x86_64 2024-04-20T21:17:08.378 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-20T21:17:08.378 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-04-20T21:17:08.726 DEBUG:teuthology.parallel:result is None 2024-04-20T21:17:09.006 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:01:49 ago on Sat 20 Apr 2024 09:15:20 PM UTC. 2024-04-20T21:17:09.089 INFO:teuthology.orchestra.run.smithi062.stdout:Package librbd1-2:18.2.0-0.el9.x86_64 is already installed. 2024-04-20T21:17:09.133 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2024-04-20T21:17:09.134 INFO:teuthology.orchestra.run.smithi062.stdout:Nothing to do. 2024-04-20T21:17:09.134 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2024-04-20T21:17:09.333 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install python3-rados 2024-04-20T21:17:10.109 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:01:50 ago on Sat 20 Apr 2024 09:15:20 PM UTC. 2024-04-20T21:17:10.199 INFO:teuthology.orchestra.run.smithi062.stdout:Package python3-rados-2:18.2.0-0.el9.x86_64 is already installed. 2024-04-20T21:17:10.243 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2024-04-20T21:17:10.244 INFO:teuthology.orchestra.run.smithi062.stdout:Nothing to do. 2024-04-20T21:17:10.245 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2024-04-20T21:17:10.449 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install python3-rgw 2024-04-20T21:17:11.231 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:01:51 ago on Sat 20 Apr 2024 09:15:20 PM UTC. 2024-04-20T21:17:11.318 INFO:teuthology.orchestra.run.smithi062.stdout:Package python3-rgw-2:18.2.0-0.el9.x86_64 is already installed. 2024-04-20T21:17:11.361 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2024-04-20T21:17:11.362 INFO:teuthology.orchestra.run.smithi062.stdout:Nothing to do. 2024-04-20T21:17:11.362 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2024-04-20T21:17:11.449 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install python3-cephfs 2024-04-20T21:17:12.206 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:01:52 ago on Sat 20 Apr 2024 09:15:20 PM UTC. 2024-04-20T21:17:12.294 INFO:teuthology.orchestra.run.smithi062.stdout:Package python3-cephfs-2:18.2.0-0.el9.x86_64 is already installed. 2024-04-20T21:17:12.339 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2024-04-20T21:17:12.340 INFO:teuthology.orchestra.run.smithi062.stdout:Nothing to do. 2024-04-20T21:17:12.340 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2024-04-20T21:17:12.418 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install python3-rbd 2024-04-20T21:17:13.159 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:01:53 ago on Sat 20 Apr 2024 09:15:20 PM UTC. 2024-04-20T21:17:13.243 INFO:teuthology.orchestra.run.smithi062.stdout:Package python3-rbd-2:18.2.0-0.el9.x86_64 is already installed. 2024-04-20T21:17:13.287 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2024-04-20T21:17:13.288 INFO:teuthology.orchestra.run.smithi062.stdout:Nothing to do. 2024-04-20T21:17:13.288 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2024-04-20T21:17:13.361 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install rbd-fuse 2024-04-20T21:17:14.110 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:01:54 ago on Sat 20 Apr 2024 09:15:20 PM UTC. 2024-04-20T21:17:14.241 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2024-04-20T21:17:14.242 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-04-20T21:17:14.242 INFO:teuthology.orchestra.run.smithi062.stdout: Package Architecture Version Repository Size 2024-04-20T21:17:14.242 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-04-20T21:17:14.242 INFO:teuthology.orchestra.run.smithi062.stdout:Installing: 2024-04-20T21:17:14.242 INFO:teuthology.orchestra.run.smithi062.stdout: rbd-fuse x86_64 2:18.2.0-0.el9 ceph 86 k 2024-04-20T21:17:14.242 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:17:14.242 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction Summary 2024-04-20T21:17:14.242 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-04-20T21:17:14.242 INFO:teuthology.orchestra.run.smithi062.stdout:Install 1 Package 2024-04-20T21:17:14.242 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:17:14.243 INFO:teuthology.orchestra.run.smithi062.stdout:Total download size: 86 k 2024-04-20T21:17:14.243 INFO:teuthology.orchestra.run.smithi062.stdout:Installed size: 230 k 2024-04-20T21:17:14.243 INFO:teuthology.orchestra.run.smithi062.stdout:Downloading Packages: 2024-04-20T21:17:14.401 INFO:teuthology.orchestra.run.smithi062.stdout:rbd-fuse-18.2.0-0.el9.x86_64.rpm 544 kB/s | 86 kB 00:00 2024-04-20T21:17:14.402 INFO:teuthology.orchestra.run.smithi062.stdout:-------------------------------------------------------------------------------- 2024-04-20T21:17:14.402 INFO:teuthology.orchestra.run.smithi062.stdout:Total 538 kB/s | 86 kB 00:00 2024-04-20T21:17:14.402 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction check 2024-04-20T21:17:14.412 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction check succeeded. 2024-04-20T21:17:14.412 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction test 2024-04-20T21:17:14.488 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction test succeeded. 2024-04-20T21:17:14.488 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction 2024-04-20T21:17:14.659 INFO:teuthology.orchestra.run.smithi062.stdout: Preparing : 1/1 2024-04-20T21:17:14.781 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : rbd-fuse-2:18.2.0-0.el9.x86_64 1/1 2024-04-20T21:17:15.300 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: rbd-fuse-2:18.2.0-0.el9.x86_64 1/1 2024-04-20T21:17:15.564 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : rbd-fuse-2:18.2.0-0.el9.x86_64 1/1 2024-04-20T21:17:15.564 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:17:15.564 INFO:teuthology.orchestra.run.smithi062.stdout:Installed: 2024-04-20T21:17:15.564 INFO:teuthology.orchestra.run.smithi062.stdout: rbd-fuse-2:18.2.0-0.el9.x86_64 2024-04-20T21:17:15.564 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:17:15.564 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2024-04-20T21:17:16.015 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install rbd-mirror 2024-04-20T21:17:16.814 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:01:56 ago on Sat 20 Apr 2024 09:15:20 PM UTC. 2024-04-20T21:17:16.953 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2024-04-20T21:17:16.954 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-04-20T21:17:16.954 INFO:teuthology.orchestra.run.smithi062.stdout: Package Architecture Version Repository Size 2024-04-20T21:17:16.954 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-04-20T21:17:16.954 INFO:teuthology.orchestra.run.smithi062.stdout:Installing: 2024-04-20T21:17:16.954 INFO:teuthology.orchestra.run.smithi062.stdout: rbd-mirror x86_64 2:18.2.0-0.el9 ceph 3.0 M 2024-04-20T21:17:16.954 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:17:16.954 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction Summary 2024-04-20T21:17:16.954 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-04-20T21:17:16.954 INFO:teuthology.orchestra.run.smithi062.stdout:Install 1 Package 2024-04-20T21:17:16.955 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:17:16.955 INFO:teuthology.orchestra.run.smithi062.stdout:Total download size: 3.0 M 2024-04-20T21:17:16.955 INFO:teuthology.orchestra.run.smithi062.stdout:Installed size: 12 M 2024-04-20T21:17:16.955 INFO:teuthology.orchestra.run.smithi062.stdout:Downloading Packages: 2024-04-20T21:17:17.079 INFO:teuthology.orchestra.run.smithi062.stdout:rbd-mirror-18.2.0-0.el9.x86_64.rpm 24 MB/s | 3.0 MB 00:00 2024-04-20T21:17:17.079 INFO:teuthology.orchestra.run.smithi062.stdout:-------------------------------------------------------------------------------- 2024-04-20T21:17:17.079 INFO:teuthology.orchestra.run.smithi062.stdout:Total 24 MB/s | 3.0 MB 00:00 2024-04-20T21:17:17.080 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction check 2024-04-20T21:17:17.091 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction check succeeded. 2024-04-20T21:17:17.091 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction test 2024-04-20T21:17:17.185 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction test succeeded. 2024-04-20T21:17:17.186 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction 2024-04-20T21:17:17.692 INFO:teuthology.orchestra.run.smithi062.stdout: Preparing : 1/1 2024-04-20T21:17:17.763 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : rbd-mirror-2:18.2.0-0.el9.x86_64 1/1 2024-04-20T21:17:17.838 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: rbd-mirror-2:18.2.0-0.el9.x86_64 1/1 2024-04-20T21:17:17.839 INFO:teuthology.orchestra.run.smithi062.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-20T21:17:17.839 INFO:teuthology.orchestra.run.smithi062.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-20T21:17:17.839 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-20T21:17:17.839 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-20T21:17:17.839 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:17:19.299 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : rbd-mirror-2:18.2.0-0.el9.x86_64 1/1 2024-04-20T21:17:19.300 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:17:19.300 INFO:teuthology.orchestra.run.smithi062.stdout:Installed: 2024-04-20T21:17:19.300 INFO:teuthology.orchestra.run.smithi062.stdout: rbd-mirror-2:18.2.0-0.el9.x86_64 2024-04-20T21:17:19.300 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:17:19.300 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2024-04-20T21:17:19.621 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install rbd-nbd 2024-04-20T21:17:20.414 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:02:00 ago on Sat 20 Apr 2024 09:15:20 PM UTC. 2024-04-20T21:17:20.546 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2024-04-20T21:17:20.547 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-04-20T21:17:20.547 INFO:teuthology.orchestra.run.smithi062.stdout: Package Architecture Version Repository Size 2024-04-20T21:17:20.547 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-04-20T21:17:20.547 INFO:teuthology.orchestra.run.smithi062.stdout:Installing: 2024-04-20T21:17:20.547 INFO:teuthology.orchestra.run.smithi062.stdout: rbd-nbd x86_64 2:18.2.0-0.el9 ceph 169 k 2024-04-20T21:17:20.547 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:17:20.548 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction Summary 2024-04-20T21:17:20.548 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-04-20T21:17:20.548 INFO:teuthology.orchestra.run.smithi062.stdout:Install 1 Package 2024-04-20T21:17:20.548 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:17:20.548 INFO:teuthology.orchestra.run.smithi062.stdout:Total download size: 169 k 2024-04-20T21:17:20.548 INFO:teuthology.orchestra.run.smithi062.stdout:Installed size: 490 k 2024-04-20T21:17:20.548 INFO:teuthology.orchestra.run.smithi062.stdout:Downloading Packages: 2024-04-20T21:17:20.615 INFO:teuthology.orchestra.run.smithi062.stdout:rbd-nbd-18.2.0-0.el9.x86_64.rpm 2.5 MB/s | 169 kB 00:00 2024-04-20T21:17:20.616 INFO:teuthology.orchestra.run.smithi062.stdout:-------------------------------------------------------------------------------- 2024-04-20T21:17:20.616 INFO:teuthology.orchestra.run.smithi062.stdout:Total 2.4 MB/s | 169 kB 00:00 2024-04-20T21:17:20.616 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction check 2024-04-20T21:17:20.626 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction check succeeded. 2024-04-20T21:17:20.626 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction test 2024-04-20T21:17:20.701 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction test succeeded. 2024-04-20T21:17:20.702 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction 2024-04-20T21:17:20.891 INFO:teuthology.orchestra.run.smithi062.stdout: Preparing : 1/1 2024-04-20T21:17:20.994 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : rbd-nbd-2:18.2.0-0.el9.x86_64 1/1 2024-04-20T21:17:21.555 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: rbd-nbd-2:18.2.0-0.el9.x86_64 1/1 2024-04-20T21:17:21.838 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : rbd-nbd-2:18.2.0-0.el9.x86_64 1/1 2024-04-20T21:17:21.838 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:17:21.838 INFO:teuthology.orchestra.run.smithi062.stdout:Installed: 2024-04-20T21:17:21.838 INFO:teuthology.orchestra.run.smithi062.stdout: rbd-nbd-2:18.2.0-0.el9.x86_64 2024-04-20T21:17:21.838 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:17:21.838 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2024-04-20T21:17:22.184 DEBUG:teuthology.parallel:result is None 2024-04-20T21:17:22.184 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-04-20T21:17:22.184 INFO:teuthology.packaging:ref: None 2024-04-20T21:17:22.184 INFO:teuthology.packaging:tag: v18.2.0 2024-04-20T21:17:22.184 INFO:teuthology.packaging:branch: None 2024-04-20T21:17:22.184 INFO:teuthology.packaging:sha1: a8c84f5997950fd7d008204839253291d52e6eda 2024-04-20T21:17:22.184 INFO:teuthology.packaging:Tag 'v18.2.0' not found in https://git.ceph.com/ceph-ci; will also look in https://git.ceph.com/ceph 2024-04-20T21:17:22.185 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dd24139a1eada541a3bc16b6941c5dde975e26d 2024-04-20T21:17:22.327 DEBUG:teuthology.orchestra.run.smithi062:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-20T21:17:22.401 INFO:teuthology.orchestra.run.smithi062.stdout:18.2.0-0.el9 2024-04-20T21:17:22.402 INFO:teuthology.packaging:The installed version of ceph is 18.2.0-0.el9 2024-04-20T21:17:22.402 INFO:teuthology.task.install:The correct ceph version 18.2.0-0 is installed. 2024-04-20T21:17:22.403 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-04-20T21:17:22.404 INFO:teuthology.packaging:ref: None 2024-04-20T21:17:22.404 INFO:teuthology.packaging:tag: v18.2.0 2024-04-20T21:17:22.404 INFO:teuthology.packaging:branch: None 2024-04-20T21:17:22.404 INFO:teuthology.packaging:sha1: a8c84f5997950fd7d008204839253291d52e6eda 2024-04-20T21:17:22.404 INFO:teuthology.packaging:Tag 'v18.2.0' not found in https://git.ceph.com/ceph-ci; will also look in https://git.ceph.com/ceph 2024-04-20T21:17:22.404 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dd24139a1eada541a3bc16b6941c5dde975e26d 2024-04-20T21:17:22.542 DEBUG:teuthology.orchestra.run.smithi189:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-20T21:17:22.613 INFO:teuthology.orchestra.run.smithi189.stdout:18.2.0-0.el9 2024-04-20T21:17:22.614 INFO:teuthology.packaging:The installed version of ceph is 18.2.0-0.el9 2024-04-20T21:17:22.614 INFO:teuthology.task.install:The correct ceph version 18.2.0-0 is installed. 2024-04-20T21:17:22.616 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-20T21:17:22.616 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-04-20T21:17:22.616 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-20T21:17:22.707 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-20T21:17:22.707 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-20T21:17:22.798 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-20T21:17:22.798 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-04-20T21:17:22.798 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/usr/bin/daemon-helper 2024-04-20T21:17:22.891 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-20T21:17:22.989 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-20T21:17:22.989 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/usr/bin/daemon-helper 2024-04-20T21:17:23.078 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-20T21:17:23.204 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-20T21:17:23.204 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-04-20T21:17:23.205 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-20T21:17:23.299 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-20T21:17:23.425 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-20T21:17:23.425 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-20T21:17:23.517 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-20T21:17:23.644 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-20T21:17:23.644 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-04-20T21:17:23.644 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/usr/bin/stdin-killer 2024-04-20T21:17:23.733 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-20T21:17:23.820 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-20T21:17:23.820 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/usr/bin/stdin-killer 2024-04-20T21:17:23.910 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-20T21:17:23.991 INFO:teuthology.run_tasks:Running task print... 2024-04-20T21:17:23.998 INFO:teuthology.task.print:**** done install task... 2024-04-20T21:17:23.998 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-20T21:17:24.097 INFO:tasks.cephadm:Config: {'compiled_cephadm_branch': 'reef', 'conf': {'osd': {'osd_class_default_list': '*', 'osd_class_load_list': '*', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}}, 'image': 'quay.io/ceph/ceph:v18.2.0', 'roleless': True, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'OSD_DOWN', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability'], 'sha1': 'a8c84f5997950fd7d008204839253291d52e6eda'} 2024-04-20T21:17:24.097 INFO:tasks.cephadm:Cluster image is quay.io/ceph/ceph:v18.2.0 2024-04-20T21:17:24.097 INFO:tasks.cephadm:Cluster fsid is 61ae3628-ff5b-11ee-bc93-c7b262605968 2024-04-20T21:17:24.097 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-20T21:17:24.098 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-04-20T21:17:24.098 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi062': '172.21.15.62', 'mon.smithi189': '172.21.15.189'} 2024-04-20T21:17:24.098 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-20T21:17:24.098 DEBUG:teuthology.orchestra.run.smithi062:> sudo hostname $(hostname -s) 2024-04-20T21:17:24.194 DEBUG:teuthology.orchestra.run.smithi189:> sudo hostname $(hostname -s) 2024-04-20T21:17:24.284 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra for reef 2024-04-20T21:17:24.285 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=a8c84f5997950fd7d008204839253291d52e6eda 2024-04-20T21:17:24.502 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'a8c84f5997950fd7d008204839253291d52e6eda', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/78637/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-3133-ga8c84f59', 'node_name': '172.21.5.35+adami05', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-3133.ga8c84f59'}, 'url': 'https://2.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240416.193735/a8c84f5997950fd7d008204839253291d52e6eda/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-04-16 21:41:56.138585', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-rishabh-testing-20240416.193735', 'chacra_url': 'https://2.chacra.ceph.com/repos/ceph/wip-rishabh-testing-20240416.193735/a8c84f5997950fd7d008204839253291d52e6eda/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-04-20T21:17:24.813 INFO:tasks.util.chacra:got chacra host chacra.ceph.com, ref reef, sha1 9d98999acebbad2663cc4938e94c9ffd83485cbd from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&ref=reef 2024-04-20T21:17:24.814 INFO:tasks.cephadm:Discovered cachra url: https://chacra.ceph.com/binaries/ceph/reef/9d98999acebbad2663cc4938e94c9ffd83485cbd/centos/9/x86_64/flavors/default/cephadm 2024-04-20T21:17:24.814 DEBUG:teuthology.orchestra.run.smithi062:> curl --silent -L https://chacra.ceph.com/binaries/ceph/reef/9d98999acebbad2663cc4938e94c9ffd83485cbd/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-20T21:17:25.179 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-r--r--. 1 ubuntu ubuntu 212578 Apr 20 21:17 /home/ubuntu/cephtest/cephadm 2024-04-20T21:17:25.179 DEBUG:teuthology.orchestra.run.smithi189:> curl --silent -L https://chacra.ceph.com/binaries/ceph/reef/9d98999acebbad2663cc4938e94c9ffd83485cbd/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-20T21:17:25.539 INFO:teuthology.orchestra.run.smithi189.stdout:-rw-r--r--. 1 ubuntu ubuntu 212578 Apr 20 21:17 /home/ubuntu/cephtest/cephadm 2024-04-20T21:17:25.539 DEBUG:teuthology.orchestra.run.smithi062:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-20T21:17:25.602 DEBUG:teuthology.orchestra.run.smithi189:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-20T21:17:25.673 INFO:tasks.cephadm:Pulling image quay.io/ceph/ceph:v18.2.0 on all hosts... 2024-04-20T21:17:25.674 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 pull 2024-04-20T21:17:25.681 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 pull 2024-04-20T21:17:27.550 INFO:teuthology.orchestra.run.smithi189.stderr:Pulling container image quay.io/ceph/ceph:v18.2.0... 2024-04-20T21:17:27.670 INFO:teuthology.orchestra.run.smithi062.stderr:Pulling container image quay.io/ceph/ceph:v18.2.0... 2024-04-20T21:17:53.664 INFO:teuthology.orchestra.run.smithi189.stdout:{ 2024-04-20T21:17:53.664 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_version": "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)", 2024-04-20T21:17:53.665 INFO:teuthology.orchestra.run.smithi189.stdout: "image_id": "dc2bc1663786ad5862ef6efcff18243ab7dfeda0bcb911ddf757a0d20feca946", 2024-04-20T21:17:53.665 INFO:teuthology.orchestra.run.smithi189.stdout: "repo_digests": [ 2024-04-20T21:17:53.665 INFO:teuthology.orchestra.run.smithi189.stdout: "quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007", 2024-04-20T21:17:53.665 INFO:teuthology.orchestra.run.smithi189.stdout: "quay.io/ceph/ceph@sha256:d58cf65589d0abf9d5261cd46fb62be3bfb29098febc78c0fcdf116a15274d27" 2024-04-20T21:17:53.665 INFO:teuthology.orchestra.run.smithi189.stdout: ] 2024-04-20T21:17:53.665 INFO:teuthology.orchestra.run.smithi189.stdout:} 2024-04-20T21:17:56.574 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:17:56.575 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph_version": "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)", 2024-04-20T21:17:56.575 INFO:teuthology.orchestra.run.smithi062.stdout: "image_id": "dc2bc1663786ad5862ef6efcff18243ab7dfeda0bcb911ddf757a0d20feca946", 2024-04-20T21:17:56.575 INFO:teuthology.orchestra.run.smithi062.stdout: "repo_digests": [ 2024-04-20T21:17:56.575 INFO:teuthology.orchestra.run.smithi062.stdout: "quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007", 2024-04-20T21:17:56.575 INFO:teuthology.orchestra.run.smithi062.stdout: "quay.io/ceph/ceph@sha256:d58cf65589d0abf9d5261cd46fb62be3bfb29098febc78c0fcdf116a15274d27" 2024-04-20T21:17:56.575 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-20T21:17:56.575 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:17:56.618 DEBUG:teuthology.orchestra.run.smithi062:> sudo mkdir -p /etc/ceph 2024-04-20T21:17:56.712 DEBUG:teuthology.orchestra.run.smithi189:> sudo mkdir -p /etc/ceph 2024-04-20T21:17:56.807 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod 777 /etc/ceph 2024-04-20T21:17:56.900 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod 777 /etc/ceph 2024-04-20T21:17:56.988 INFO:tasks.cephadm:Writing seed config... 2024-04-20T21:17:56.989 INFO:tasks.cephadm: override: [osd] osd_class_default_list = * 2024-04-20T21:17:56.990 INFO:tasks.cephadm: override: [osd] osd_class_load_list = * 2024-04-20T21:17:56.990 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-04-20T21:17:56.990 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-04-20T21:17:56.990 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-04-20T21:17:56.990 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-04-20T21:17:56.990 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-04-20T21:17:56.990 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-04-20T21:17:56.990 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-04-20T21:17:56.990 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-20T21:17:56.990 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-20T21:17:56.990 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-04-20T21:17:56.991 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-04-20T21:17:56.991 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-04-20T21:17:56.991 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-04-20T21:17:56.991 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-04-20T21:17:56.991 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-04-20T21:17:56.991 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-04-20T21:17:56.991 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-04-20T21:17:56.991 INFO:tasks.cephadm: override: [client] debug client = 20 2024-04-20T21:17:56.991 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-04-20T21:17:56.991 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-04-20T21:17:56.991 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-04-20T21:17:56.991 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2024-04-20T21:17:56.991 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-04-20T21:17:56.992 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-04-20T21:17:56.992 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-04-20T21:17:56.992 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-04-20T21:17:56.992 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-04-20T21:17:56.992 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-04-20T21:17:56.992 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-04-20T21:17:56.992 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-04-20T21:17:56.992 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-04-20T21:17:56.992 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-04-20T21:17:56.992 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-04-20T21:17:56.992 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-20T21:17:56.992 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-20T21:17:56.992 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-20T21:17:56.992 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-20T21:17:56.992 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-20T21:17:56.992 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-04-20T21:17:56.993 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-04-20T21:17:56.993 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-04-20T21:17:56.993 DEBUG:teuthology.orchestra.run.smithi062:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-20T21:17:57.054 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 61ae3628-ff5b-11ee-bc93-c7b262605968 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true osd_class_default_list = * osd_class_load_list = * bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-04-20T21:17:57.054 DEBUG:teuthology.orchestra.run.smithi062:mon.smithi062> sudo journalctl -f -n 0 -u ceph-61ae3628-ff5b-11ee-bc93-c7b262605968@mon.smithi062.service 2024-04-20T21:17:57.101 INFO:tasks.cephadm:Bootstrapping... 2024-04-20T21:17:57.102 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 -v bootstrap --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.62 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-20T21:17:57.347 INFO:teuthology.orchestra.run.smithi062.stdout:-------------------------------------------------------------------------------- 2024-04-20T21:17:57.347 INFO:teuthology.orchestra.run.smithi062.stdout:cephadm ['--image', 'quay.io/ceph/ceph:v18.2.0', '-v', 'bootstrap', '--fsid', '61ae3628-ff5b-11ee-bc93-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-ip', '172.21.15.62', '--skip-admin-label'] 2024-04-20T21:17:57.406 INFO:teuthology.orchestra.run.smithi062.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-20T21:17:57.407 INFO:teuthology.orchestra.run.smithi062.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-04-20T21:17:57.407 INFO:teuthology.orchestra.run.smithi062.stdout:Verifying podman|docker is present... 2024-04-20T21:17:57.458 INFO:teuthology.orchestra.run.smithi062.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-20T21:17:57.459 INFO:teuthology.orchestra.run.smithi062.stdout:Verifying lvm2 is present... 2024-04-20T21:17:57.459 INFO:teuthology.orchestra.run.smithi062.stdout:Verifying time synchronization is in place... 2024-04-20T21:17:57.476 INFO:teuthology.orchestra.run.smithi062.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-20T21:17:57.476 INFO:teuthology.orchestra.run.smithi062.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-20T21:17:57.494 INFO:teuthology.orchestra.run.smithi062.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-20T21:17:57.494 INFO:teuthology.orchestra.run.smithi062.stdout:systemctl: stdout inactive 2024-04-20T21:17:57.513 INFO:teuthology.orchestra.run.smithi062.stdout:systemctl: stdout enabled 2024-04-20T21:17:57.532 INFO:teuthology.orchestra.run.smithi062.stdout:systemctl: stdout active 2024-04-20T21:17:57.532 INFO:teuthology.orchestra.run.smithi062.stdout:Unit chronyd.service is enabled and running 2024-04-20T21:17:57.532 INFO:teuthology.orchestra.run.smithi062.stdout:Repeating the final host check... 2024-04-20T21:17:57.586 INFO:teuthology.orchestra.run.smithi062.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-20T21:17:57.586 INFO:teuthology.orchestra.run.smithi062.stdout:podman (/bin/podman) version 4.9.4 is present 2024-04-20T21:17:57.587 INFO:teuthology.orchestra.run.smithi062.stdout:systemctl is present 2024-04-20T21:17:57.587 INFO:teuthology.orchestra.run.smithi062.stdout:lvcreate is present 2024-04-20T21:17:57.606 INFO:teuthology.orchestra.run.smithi062.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-20T21:17:57.606 INFO:teuthology.orchestra.run.smithi062.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-20T21:17:57.626 INFO:teuthology.orchestra.run.smithi062.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-20T21:17:57.627 INFO:teuthology.orchestra.run.smithi062.stdout:systemctl: stdout inactive 2024-04-20T21:17:57.650 INFO:teuthology.orchestra.run.smithi062.stdout:systemctl: stdout enabled 2024-04-20T21:17:57.669 INFO:teuthology.orchestra.run.smithi062.stdout:systemctl: stdout active 2024-04-20T21:17:57.669 INFO:teuthology.orchestra.run.smithi062.stdout:Unit chronyd.service is enabled and running 2024-04-20T21:17:57.670 INFO:teuthology.orchestra.run.smithi062.stdout:Host looks OK 2024-04-20T21:17:57.670 INFO:teuthology.orchestra.run.smithi062.stdout:Cluster fsid: 61ae3628-ff5b-11ee-bc93-c7b262605968 2024-04-20T21:17:57.671 INFO:teuthology.orchestra.run.smithi062.stdout:Acquiring lock 139830779686048 on /run/cephadm/61ae3628-ff5b-11ee-bc93-c7b262605968.lock 2024-04-20T21:17:57.671 INFO:teuthology.orchestra.run.smithi062.stdout:Lock 139830779686048 acquired on /run/cephadm/61ae3628-ff5b-11ee-bc93-c7b262605968.lock 2024-04-20T21:17:57.672 INFO:teuthology.orchestra.run.smithi062.stdout:Verifying IP 172.21.15.62 port 3300 ... 2024-04-20T21:17:57.674 INFO:teuthology.orchestra.run.smithi062.stdout:Verifying IP 172.21.15.62 port 6789 ... 2024-04-20T21:17:57.674 INFO:teuthology.orchestra.run.smithi062.stdout:Base mon IP(s) is [172.21.15.62:3300, 172.21.15.62:6789], mon addrv is [v2:172.21.15.62:3300,v1:172.21.15.62:6789] 2024-04-20T21:17:57.689 INFO:teuthology.orchestra.run.smithi062.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.62 metric 100 2024-04-20T21:17:57.689 INFO:teuthology.orchestra.run.smithi062.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.62 metric 100 2024-04-20T21:17:57.702 INFO:teuthology.orchestra.run.smithi062.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-04-20T21:17:57.702 INFO:teuthology.orchestra.run.smithi062.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-04-20T21:17:57.702 INFO:teuthology.orchestra.run.smithi062.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1717sec hoplimit 64 pref medium 2024-04-20T21:17:57.713 INFO:teuthology.orchestra.run.smithi062.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-20T21:17:57.714 INFO:teuthology.orchestra.run.smithi062.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-04-20T21:17:57.714 INFO:teuthology.orchestra.run.smithi062.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-20T21:17:57.714 INFO:teuthology.orchestra.run.smithi062.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-04-20T21:17:57.714 INFO:teuthology.orchestra.run.smithi062.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febb:d1c3/64 scope link 2024-04-20T21:17:57.715 INFO:teuthology.orchestra.run.smithi062.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-20T21:17:57.718 INFO:teuthology.orchestra.run.smithi062.stdout:Mon IP `172.21.15.62` is in CIDR network `172.21.0.0/20` 2024-04-20T21:17:57.718 INFO:teuthology.orchestra.run.smithi062.stdout:Mon IP `172.21.15.62` is in CIDR network `172.21.0.0/20` 2024-04-20T21:17:57.718 INFO:teuthology.orchestra.run.smithi062.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-04-20T21:17:57.721 INFO:teuthology.orchestra.run.smithi062.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-20T21:17:57.723 INFO:teuthology.orchestra.run.smithi062.stdout:Pulling container image quay.io/ceph/ceph:v18.2.0... 2024-04-20T21:17:58.776 INFO:teuthology.orchestra.run.smithi062.stdout:/bin/podman: stdout dc2bc1663786ad5862ef6efcff18243ab7dfeda0bcb911ddf757a0d20feca946 2024-04-20T21:17:58.776 INFO:teuthology.orchestra.run.smithi062.stdout:/bin/podman: stderr Trying to pull quay.io/ceph/ceph:v18.2.0... 2024-04-20T21:17:58.776 INFO:teuthology.orchestra.run.smithi062.stdout:/bin/podman: stderr Getting image source signatures 2024-04-20T21:17:58.776 INFO:teuthology.orchestra.run.smithi062.stdout:/bin/podman: stderr Copying blob sha256:3bd20aeff60302f668275dc2005d10679ae56492967a3a5a54fd3dde85333aec 2024-04-20T21:17:58.776 INFO:teuthology.orchestra.run.smithi062.stdout:/bin/podman: stderr Copying blob sha256:46af8f5390d4e94fc57efb422ccb97bb53dfe5b948546bfc191b46557eb2dbd9 2024-04-20T21:17:58.776 INFO:teuthology.orchestra.run.smithi062.stdout:/bin/podman: stderr Copying config sha256:dc2bc1663786ad5862ef6efcff18243ab7dfeda0bcb911ddf757a0d20feca946 2024-04-20T21:17:58.776 INFO:teuthology.orchestra.run.smithi062.stdout:/bin/podman: stderr Writing manifest to image destination 2024-04-20T21:18:00.037 INFO:teuthology.orchestra.run.smithi062.stdout:ceph: stdout ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable) 2024-04-20T21:18:00.037 INFO:teuthology.orchestra.run.smithi062.stdout:Ceph version: ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable) 2024-04-20T21:18:00.037 INFO:teuthology.orchestra.run.smithi062.stdout:Extracting ceph user uid/gid from container image... 2024-04-20T21:18:01.160 INFO:teuthology.orchestra.run.smithi062.stdout:stat: stdout 167 167 2024-04-20T21:18:01.160 INFO:teuthology.orchestra.run.smithi062.stdout:Creating initial keys... 2024-04-20T21:18:02.454 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph-authtool: stdout AQAJMSRm8hlSKBAAod9SSC46Pp821j56WFJDsw== 2024-04-20T21:18:03.700 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph-authtool: stdout AQAKMSRmOTxYNRAADAP8t+B7G0wKfNgvWNk+Kw== 2024-04-20T21:18:14.305 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph-authtool: stdout AQAMMSRmDch6ERAAgagrsYqU+ZZSjPzzcg5a6Q== 2024-04-20T21:18:14.307 INFO:teuthology.orchestra.run.smithi062.stdout:Creating initial monmap... 2024-04-20T21:18:15.485 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-20T21:18:15.485 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = pacific 2024-04-20T21:18:15.485 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 61ae3628-ff5b-11ee-bc93-c7b262605968 2024-04-20T21:18:15.485 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-20T21:18:15.486 INFO:teuthology.orchestra.run.smithi062.stdout:monmaptool for smithi062 [v2:172.21.15.62:3300,v1:172.21.15.62:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-20T21:18:15.486 INFO:teuthology.orchestra.run.smithi062.stdout:setting min_mon_release = pacific 2024-04-20T21:18:15.486 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/monmaptool: set fsid to 61ae3628-ff5b-11ee-bc93-c7b262605968 2024-04-20T21:18:15.486 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-20T21:18:15.486 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:18:15.486 INFO:teuthology.orchestra.run.smithi062.stdout:Creating mon... 2024-04-20T21:18:16.686 INFO:teuthology.orchestra.run.smithi062.stdout:create mon.smithi062 on 2024-04-20T21:18:17.021 INFO:teuthology.orchestra.run.smithi062.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-04-20T21:18:17.270 INFO:teuthology.orchestra.run.smithi062.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-20T21:18:17.557 INFO:teuthology.orchestra.run.smithi062.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-61ae3628-ff5b-11ee-bc93-c7b262605968.target → /etc/systemd/system/ceph-61ae3628-ff5b-11ee-bc93-c7b262605968.target. 2024-04-20T21:18:17.557 INFO:teuthology.orchestra.run.smithi062.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-61ae3628-ff5b-11ee-bc93-c7b262605968.target → /etc/systemd/system/ceph-61ae3628-ff5b-11ee-bc93-c7b262605968.target. 2024-04-20T21:18:17.919 INFO:teuthology.orchestra.run.smithi062.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-61ae3628-ff5b-11ee-bc93-c7b262605968@mon.smithi062 2024-04-20T21:18:17.919 INFO:teuthology.orchestra.run.smithi062.stdout:systemctl: stderr Failed to reset failed state of unit ceph-61ae3628-ff5b-11ee-bc93-c7b262605968@mon.smithi062.service: Unit ceph-61ae3628-ff5b-11ee-bc93-c7b262605968@mon.smithi062.service not loaded. 2024-04-20T21:18:18.221 INFO:teuthology.orchestra.run.smithi062.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-61ae3628-ff5b-11ee-bc93-c7b262605968.target.wants/ceph-61ae3628-ff5b-11ee-bc93-c7b262605968@mon.smithi062.service → /etc/systemd/system/ceph-61ae3628-ff5b-11ee-bc93-c7b262605968@.service. 2024-04-20T21:18:19.037 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:18 smithi062 podman[15832]: 2024-04-20 21:18:18.541279242 +0000 UTC m=+0.051486086 image pull dc2bc1663786ad5862ef6efcff18243ab7dfeda0bcb911ddf757a0d20feca946 quay.io/ceph/ceph:v18.2.0 2024-04-20T21:18:19.871 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:19 smithi062 podman[15832]: 2024-04-20 21:18:19.615876278 +0000 UTC m=+1.126083162 container create ec5495668187ed6345b1c3fcb8705e5190770590062ea5da279d4afb24f712e4 (image=quay.io/ceph/ceph:v18.2.0, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-mon-smithi062, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20231212, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, ceph=True, org.label-schema.license=GPLv2, RELEASE=HEAD, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-04-20T21:18:19.871 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:19 smithi062 podman[15832]: 2024-04-20 21:18:19.860182837 +0000 UTC m=+1.370389724 container init ec5495668187ed6345b1c3fcb8705e5190770590062ea5da279d4afb24f712e4 (image=quay.io/ceph/ceph:v18.2.0, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-mon-smithi062, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.0, ceph=True, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, RELEASE=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20231212, GIT_BRANCH=HEAD) 2024-04-20T21:18:19.872 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:19 smithi062 podman[15832]: 2024-04-20 21:18:19.872094606 +0000 UTC m=+1.382301494 container start ec5495668187ed6345b1c3fcb8705e5190770590062ea5da279d4afb24f712e4 (image=quay.io/ceph/ceph:v18.2.0, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-mon-smithi062, ceph=True, GIT_BRANCH=HEAD, RELEASE=HEAD, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, org.label-schema.vendor=CentOS, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20231212, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.0, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-04-20T21:18:19.982 INFO:teuthology.orchestra.run.smithi062.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-20T21:18:19.982 INFO:teuthology.orchestra.run.smithi062.stdout:systemctl: stdout disabled 2024-04-20T21:18:19.996 INFO:teuthology.orchestra.run.smithi062.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-20T21:18:19.996 INFO:teuthology.orchestra.run.smithi062.stdout:systemctl: stdout inactive 2024-04-20T21:18:19.996 INFO:teuthology.orchestra.run.smithi062.stdout:firewalld.service is not enabled 2024-04-20T21:18:19.996 INFO:teuthology.orchestra.run.smithi062.stdout:Not possible to enable service . firewalld.service is not available 2024-04-20T21:18:19.997 INFO:teuthology.orchestra.run.smithi062.stdout:Waiting for mon to start... 2024-04-20T21:18:19.997 INFO:teuthology.orchestra.run.smithi062.stdout:Waiting for mon... 2024-04-20T21:18:20.146 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:19 smithi062 bash[15832]: ec5495668187ed6345b1c3fcb8705e5190770590062ea5da279d4afb24f712e4 2024-04-20T21:18:20.146 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:19 smithi062 ceph-mon[15851]: mkfs 61ae3628-ff5b-11ee-bc93-c7b262605968 2024-04-20T21:18:20.146 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:19 smithi062 ceph-mon[15851]: mon.smithi062 is new leader, mons smithi062 in quorum (ranks 0) 2024-04-20T21:18:20.146 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:19 smithi062 systemd[1]: Started Ceph mon.smithi062 for 61ae3628-ff5b-11ee-bc93-c7b262605968. 2024-04-20T21:18:21.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:20 smithi062 ceph-mon[15851]: mon.smithi062 is new leader, mons smithi062 in quorum (ranks 0) 2024-04-20T21:18:21.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:20 smithi062 ceph-mon[15851]: monmap e1: 1 mons at {smithi062=[v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0]} removed_ranks: {} 2024-04-20T21:18:21.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:20 smithi062 ceph-mon[15851]: fsmap 2024-04-20T21:18:21.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:20 smithi062 ceph-mon[15851]: osdmap e1: 0 total, 0 up, 0 in 2024-04-20T21:18:21.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:20 smithi062 ceph-mon[15851]: mgrmap e1: no daemons active 2024-04-20T21:18:21.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:20 smithi062 ceph-mon[15851]: from='client.? 172.21.15.62:0/1993126935' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-20T21:18:21.412 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout cluster: 2024-04-20T21:18:21.412 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout id: 61ae3628-ff5b-11ee-bc93-c7b262605968 2024-04-20T21:18:21.412 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-04-20T21:18:21.412 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout 2024-04-20T21:18:21.412 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout services: 2024-04-20T21:18:21.412 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum smithi062 (age 0.772495s) 2024-04-20T21:18:21.413 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-04-20T21:18:21.413 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-04-20T21:18:21.413 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout 2024-04-20T21:18:21.413 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout data: 2024-04-20T21:18:21.413 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-04-20T21:18:21.413 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-04-20T21:18:21.413 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-04-20T21:18:21.413 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout pgs: 2024-04-20T21:18:21.413 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout 2024-04-20T21:18:21.413 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:20.628+0000 7f8299400700 1 Processor -- start 2024-04-20T21:18:21.413 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:20.628+0000 7f8299400700 1 -- start start 2024-04-20T21:18:21.413 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:20.628+0000 7f8299400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8294105b10 0x7f8294105f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:18:21.413 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:20.628+0000 7f8299400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8294106460 con 0x7f8294105b10 2024-04-20T21:18:21.413 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:20.630+0000 7f8292800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8294105b10 0x7f8294105f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:18:21.414 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:20.630+0000 7f8292800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8294105b10 0x7f8294105f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52020/0 (socket says 172.21.15.62:52020) 2024-04-20T21:18:21.414 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:20.630+0000 7f8292800700 1 -- 172.21.15.62:0/2880233225 learned_addr learned my addr 172.21.15.62:0/2880233225 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:18:21.414 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:20.630+0000 7f8292800700 1 -- 172.21.15.62:0/2880233225 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82941065a0 con 0x7f8294105b10 2024-04-20T21:18:21.414 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:20.631+0000 7f8292800700 1 --2- 172.21.15.62:0/2880233225 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8294105b10 0x7f8294105f20 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f82840098a0 tx=0x7f8284009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d7b2fe82f204303 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:18:21.414 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:20.631+0000 7f8291400700 1 -- 172.21.15.62:0/2880233225 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8284004180 con 0x7f8294105b10 2024-04-20T21:18:21.414 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:20.631+0000 7f8291400700 1 -- 172.21.15.62:0/2880233225 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82840042e0 con 0x7f8294105b10 2024-04-20T21:18:21.414 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:20.631+0000 7f8291400700 1 -- 172.21.15.62:0/2880233225 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f82840036a0 con 0x7f8294105b10 2024-04-20T21:18:21.414 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:20.632+0000 7f8299400700 1 -- 172.21.15.62:0/2880233225 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8294105b10 msgr2=0x7f8294105f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:18:21.414 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:20.632+0000 7f8299400700 1 --2- 172.21.15.62:0/2880233225 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8294105b10 0x7f8294105f20 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f82840098a0 tx=0x7f8284009bb0 comp rx=0 tx=0).stop 2024-04-20T21:18:21.414 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:20.632+0000 7f8299400700 1 -- 172.21.15.62:0/2880233225 shutdown_connections 2024-04-20T21:18:21.414 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:20.632+0000 7f8299400700 1 --2- 172.21.15.62:0/2880233225 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8294105b10 0x7f8294105f20 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:18:21.414 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:20.632+0000 7f8299400700 1 -- 172.21.15.62:0/2880233225 >> 172.21.15.62:0/2880233225 conn(0x7f8294101180 msgr2=0x7f8294103590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:18:21.414 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:20.632+0000 7f8299400700 1 -- 172.21.15.62:0/2880233225 shutdown_connections 2024-04-20T21:18:21.415 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:20.632+0000 7f8299400700 1 -- 172.21.15.62:0/2880233225 wait complete. 2024-04-20T21:18:21.415 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:20.633+0000 7f8299400700 1 Processor -- start 2024-04-20T21:18:21.415 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:20.633+0000 7f8299400700 1 -- start start 2024-04-20T21:18:21.415 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:20.634+0000 7f8299400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8294105b10 0x7f829419b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:18:21.415 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:20.634+0000 7f8299400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f829419b700 con 0x7f8294105b10 2024-04-20T21:18:21.415 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:20.634+0000 7f8292800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8294105b10 0x7f829419b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:18:21.415 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:20.634+0000 7f8292800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8294105b10 0x7f829419b1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52026/0 (socket says 172.21.15.62:52026) 2024-04-20T21:18:21.415 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:20.634+0000 7f8292800700 1 -- 172.21.15.62:0/1993126935 learned_addr learned my addr 172.21.15.62:0/1993126935 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:18:21.415 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:20.635+0000 7f8292800700 1 -- 172.21.15.62:0/1993126935 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8284009550 con 0x7f8294105b10 2024-04-20T21:18:21.415 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:20.635+0000 7f8292800700 1 --2- 172.21.15.62:0/1993126935 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8294105b10 0x7f829419b1c0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f8294106840 tx=0x7f8284004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:18:21.418 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:20.635+0000 7f828b400700 1 -- 172.21.15.62:0/1993126935 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8284003ce0 con 0x7f8294105b10 2024-04-20T21:18:21.418 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:20.635+0000 7f828b400700 1 -- 172.21.15.62:0/1993126935 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8284020070 con 0x7f8294105b10 2024-04-20T21:18:21.418 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:20.635+0000 7f8299400700 1 -- 172.21.15.62:0/1993126935 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f829419b900 con 0x7f8294105b10 2024-04-20T21:18:21.418 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:20.635+0000 7f828b400700 1 -- 172.21.15.62:0/1993126935 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8284011630 con 0x7f8294105b10 2024-04-20T21:18:21.418 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:20.635+0000 7f8299400700 1 -- 172.21.15.62:0/1993126935 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f829419bd20 con 0x7f8294105b10 2024-04-20T21:18:21.418 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:20.636+0000 7f828b400700 1 -- 172.21.15.62:0/1993126935 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f8284022030 con 0x7f8294105b10 2024-04-20T21:18:21.418 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:20.636+0000 7f828b400700 1 -- 172.21.15.62:0/1993126935 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f8284025d40 con 0x7f8294105b10 2024-04-20T21:18:21.418 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:20.637+0000 7f8299400700 1 -- 172.21.15.62:0/1993126935 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8294106ed0 con 0x7f8294105b10 2024-04-20T21:18:21.418 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:20.640+0000 7f828b400700 1 -- 172.21.15.62:0/1993126935 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7f8284025480 con 0x7f8294105b10 2024-04-20T21:18:21.418 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:20.720+0000 7f8299400700 1 -- 172.21.15.62:0/1993126935 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f829419edb0 con 0x7f8294105b10 2024-04-20T21:18:21.418 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:20.721+0000 7f828b400700 1 -- 172.21.15.62:0/1993126935 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+325 (secure 0 0 0) 0x7f8284011bd0 con 0x7f8294105b10 2024-04-20T21:18:21.418 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:20.723+0000 7f8299400700 1 -- 172.21.15.62:0/1993126935 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8294105b10 msgr2=0x7f829419b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:18:21.419 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:20.723+0000 7f8299400700 1 --2- 172.21.15.62:0/1993126935 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8294105b10 0x7f829419b1c0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f8294106840 tx=0x7f8284004750 comp rx=0 tx=0).stop 2024-04-20T21:18:21.419 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:20.723+0000 7f8299400700 1 -- 172.21.15.62:0/1993126935 shutdown_connections 2024-04-20T21:18:21.419 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:20.723+0000 7f8299400700 1 --2- 172.21.15.62:0/1993126935 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8294105b10 0x7f829419b1c0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:18:21.419 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:20.723+0000 7f8299400700 1 -- 172.21.15.62:0/1993126935 >> 172.21.15.62:0/1993126935 conn(0x7f8294101180 msgr2=0x7f829407a0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:18:21.419 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:20.723+0000 7f8299400700 1 -- 172.21.15.62:0/1993126935 shutdown_connections 2024-04-20T21:18:21.419 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:20.723+0000 7f8299400700 1 -- 172.21.15.62:0/1993126935 wait complete. 2024-04-20T21:18:21.419 INFO:teuthology.orchestra.run.smithi062.stdout:mon is available 2024-04-20T21:18:21.419 INFO:teuthology.orchestra.run.smithi062.stdout:Assimilating anything we can from ceph.conf... 2024-04-20T21:18:22.531 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:22 smithi062 ceph-mon[15851]: from='client.? 172.21.15.62:0/1525557019' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-20T21:18:22.531 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:22 smithi062 ceph-mon[15851]: from='client.? 172.21.15.62:0/1525557019' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-20T21:18:22.834 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout 2024-04-20T21:18:22.834 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout [global] 2024-04-20T21:18:22.834 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout fsid = 61ae3628-ff5b-11ee-bc93-c7b262605968 2024-04-20T21:18:22.834 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.62:3300,v1:172.21.15.62:6789] 2024-04-20T21:18:22.834 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-20T21:18:22.835 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-20T21:18:22.835 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-20T21:18:22.835 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-20T21:18:22.840 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout 2024-04-20T21:18:22.840 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-20T21:18:22.840 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-20T21:18:22.840 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout 2024-04-20T21:18:22.840 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout [osd] 2024-04-20T21:18:22.840 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-20T21:18:22.840 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-20T21:18:22.840 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-20T21:18:22.840 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-20T21:18:22.840 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-20T21:18:22.840 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:22.085+0000 7f1136000700 1 Processor -- start 2024-04-20T21:18:22.841 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:22.085+0000 7f1136000700 1 -- start start 2024-04-20T21:18:22.841 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:22.086+0000 7f1136000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1130105b70 0x7f1130105f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:18:22.841 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:22.086+0000 7f1136000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11301064c0 con 0x7f1130105b70 2024-04-20T21:18:22.841 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:22.086+0000 7f112f400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1130105b70 0x7f1130105f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:18:22.841 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:22.086+0000 7f112f400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1130105b70 0x7f1130105f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48722/0 (socket says 172.21.15.62:48722) 2024-04-20T21:18:22.841 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:22.086+0000 7f112f400700 1 -- 172.21.15.62:0/654726316 learned_addr learned my addr 172.21.15.62:0/654726316 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:18:22.841 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:22.087+0000 7f112f400700 1 -- 172.21.15.62:0/654726316 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1130106600 con 0x7f1130105b70 2024-04-20T21:18:22.841 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:22.087+0000 7f112f400700 1 --2- 172.21.15.62:0/654726316 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1130105b70 0x7f1130105f80 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f11200098a0 tx=0x7f1120009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67d3fa257c493216 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:18:22.841 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:22.087+0000 7f112e000700 1 -- 172.21.15.62:0/654726316 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1120004180 con 0x7f1130105b70 2024-04-20T21:18:22.841 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:22.087+0000 7f112e000700 1 -- 172.21.15.62:0/654726316 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11200042e0 con 0x7f1130105b70 2024-04-20T21:18:22.841 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:22.087+0000 7f112e000700 1 -- 172.21.15.62:0/654726316 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f11200036a0 con 0x7f1130105b70 2024-04-20T21:18:22.841 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:22.088+0000 7f1136000700 1 -- 172.21.15.62:0/654726316 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1130105b70 msgr2=0x7f1130105f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:18:22.841 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:22.088+0000 7f1136000700 1 --2- 172.21.15.62:0/654726316 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1130105b70 0x7f1130105f80 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f11200098a0 tx=0x7f1120009bb0 comp rx=0 tx=0).stop 2024-04-20T21:18:22.841 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:22.088+0000 7f1136000700 1 -- 172.21.15.62:0/654726316 shutdown_connections 2024-04-20T21:18:22.842 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:22.088+0000 7f1136000700 1 --2- 172.21.15.62:0/654726316 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1130105b70 0x7f1130105f80 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:18:22.842 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:22.088+0000 7f1136000700 1 -- 172.21.15.62:0/654726316 >> 172.21.15.62:0/654726316 conn(0x7f11301011a0 msgr2=0x7f11301035f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:18:22.842 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:22.088+0000 7f1136000700 1 -- 172.21.15.62:0/654726316 shutdown_connections 2024-04-20T21:18:22.842 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:22.088+0000 7f1136000700 1 -- 172.21.15.62:0/654726316 wait complete. 2024-04-20T21:18:22.842 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:22.089+0000 7f1136000700 1 Processor -- start 2024-04-20T21:18:22.842 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:22.089+0000 7f1136000700 1 -- start start 2024-04-20T21:18:22.842 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:22.090+0000 7f1136000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1130105b70 0x7f113019b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:18:22.842 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:22.090+0000 7f1136000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f113019b760 con 0x7f1130105b70 2024-04-20T21:18:22.842 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:22.090+0000 7f112f400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1130105b70 0x7f113019b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:18:22.842 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:22.090+0000 7f112f400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1130105b70 0x7f113019b220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48730/0 (socket says 172.21.15.62:48730) 2024-04-20T21:18:22.842 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:22.090+0000 7f112f400700 1 -- 172.21.15.62:0/1525557019 learned_addr learned my addr 172.21.15.62:0/1525557019 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:18:22.842 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:22.090+0000 7f112f400700 1 -- 172.21.15.62:0/1525557019 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1120009550 con 0x7f1130105b70 2024-04-20T21:18:22.842 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:22.091+0000 7f112f400700 1 --2- 172.21.15.62:0/1525557019 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1130105b70 0x7f113019b220 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f11301068a0 tx=0x7f1120004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:18:22.842 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:22.091+0000 7f1127e00700 1 -- 172.21.15.62:0/1525557019 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1120003ce0 con 0x7f1130105b70 2024-04-20T21:18:22.843 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:22.091+0000 7f1127e00700 1 -- 172.21.15.62:0/1525557019 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1120020070 con 0x7f1130105b70 2024-04-20T21:18:22.845 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:22.091+0000 7f1127e00700 1 -- 172.21.15.62:0/1525557019 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1120011630 con 0x7f1130105b70 2024-04-20T21:18:22.845 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:22.091+0000 7f1136000700 1 -- 172.21.15.62:0/1525557019 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f113019b960 con 0x7f1130105b70 2024-04-20T21:18:22.845 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:22.091+0000 7f1136000700 1 -- 172.21.15.62:0/1525557019 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f113019bd80 con 0x7f1130105b70 2024-04-20T21:18:22.845 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:22.091+0000 7f1127e00700 1 -- 172.21.15.62:0/1525557019 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f1120022030 con 0x7f1130105b70 2024-04-20T21:18:22.845 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:22.092+0000 7f1127e00700 1 -- 172.21.15.62:0/1525557019 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f1120025d40 con 0x7f1130105b70 2024-04-20T21:18:22.845 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:22.092+0000 7f1136000700 1 -- 172.21.15.62:0/1525557019 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1130062380 con 0x7f1130105b70 2024-04-20T21:18:22.845 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:22.095+0000 7f1127e00700 1 -- 172.21.15.62:0/1525557019 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7f1120011ae0 con 0x7f1130105b70 2024-04-20T21:18:22.845 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:22.173+0000 7f1136000700 1 -- 172.21.15.62:0/1525557019 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f113019e7c0 con 0x7f1130105b70 2024-04-20T21:18:22.845 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:22.183+0000 7f1127e00700 1 -- 172.21.15.62:0/1525557019 <== mon.0 v2:172.21.15.62:3300/0 7 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f1120011c80 con 0x7f1130105b70 2024-04-20T21:18:22.845 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:22.183+0000 7f1127e00700 1 -- 172.21.15.62:0/1525557019 <== mon.0 v2:172.21.15.62:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+429 (secure 0 0 0) 0x7f1120036020 con 0x7f1130105b70 2024-04-20T21:18:22.846 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:22.185+0000 7f1136000700 1 -- 172.21.15.62:0/1525557019 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1130105b70 msgr2=0x7f113019b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:18:22.846 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:22.185+0000 7f1136000700 1 --2- 172.21.15.62:0/1525557019 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1130105b70 0x7f113019b220 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f11301068a0 tx=0x7f1120004750 comp rx=0 tx=0).stop 2024-04-20T21:18:22.846 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:22.185+0000 7f1136000700 1 -- 172.21.15.62:0/1525557019 shutdown_connections 2024-04-20T21:18:22.846 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:22.185+0000 7f1136000700 1 --2- 172.21.15.62:0/1525557019 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1130105b70 0x7f113019b220 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:18:22.846 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:22.185+0000 7f1136000700 1 -- 172.21.15.62:0/1525557019 >> 172.21.15.62:0/1525557019 conn(0x7f11301011a0 msgr2=0x7f113007a0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:18:22.846 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:22.185+0000 7f1136000700 1 -- 172.21.15.62:0/1525557019 shutdown_connections 2024-04-20T21:18:22.846 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:22.185+0000 7f1136000700 1 -- 172.21.15.62:0/1525557019 wait complete. 2024-04-20T21:18:22.846 INFO:teuthology.orchestra.run.smithi062.stdout:Generating new minimal ceph.conf... 2024-04-20T21:18:23.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:23 smithi062 ceph-mon[15851]: from='client.? 172.21.15.62:0/815681041' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:18:24.121 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:23.478+0000 7fe8d3e00700 1 Processor -- start 2024-04-20T21:18:24.122 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:23.478+0000 7fe8d3e00700 1 -- start start 2024-04-20T21:18:24.122 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:23.479+0000 7fe8d3e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe8cc105b10 0x7fe8cc105f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:18:24.122 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:23.479+0000 7fe8d3e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8cc106460 con 0x7fe8cc105b10 2024-04-20T21:18:24.124 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:23.479+0000 7fe8d1400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe8cc105b10 0x7fe8cc105f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:18:24.124 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:23.479+0000 7fe8d1400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe8cc105b10 0x7fe8cc105f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48738/0 (socket says 172.21.15.62:48738) 2024-04-20T21:18:24.124 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:23.479+0000 7fe8d1400700 1 -- 172.21.15.62:0/4241417860 learned_addr learned my addr 172.21.15.62:0/4241417860 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:18:24.124 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:23.480+0000 7fe8d1400700 1 -- 172.21.15.62:0/4241417860 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8cc1065a0 con 0x7fe8cc105b10 2024-04-20T21:18:24.124 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:23.480+0000 7fe8d1400700 1 --2- 172.21.15.62:0/4241417860 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe8cc105b10 0x7fe8cc105f20 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fe8c00098a0 tx=0x7fe8c0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6745897e6f7ac36 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:18:24.124 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:23.481+0000 7fe8cbe00700 1 -- 172.21.15.62:0/4241417860 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe8c0004180 con 0x7fe8cc105b10 2024-04-20T21:18:24.124 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:23.481+0000 7fe8cbe00700 1 -- 172.21.15.62:0/4241417860 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fe8c00042e0 con 0x7fe8cc105b10 2024-04-20T21:18:24.124 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:23.481+0000 7fe8cbe00700 1 -- 172.21.15.62:0/4241417860 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe8c0011a50 con 0x7fe8cc105b10 2024-04-20T21:18:24.125 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:23.481+0000 7fe8d3e00700 1 -- 172.21.15.62:0/4241417860 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe8cc105b10 msgr2=0x7fe8cc105f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:18:24.125 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:23.481+0000 7fe8d3e00700 1 --2- 172.21.15.62:0/4241417860 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe8cc105b10 0x7fe8cc105f20 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fe8c00098a0 tx=0x7fe8c0009bb0 comp rx=0 tx=0).stop 2024-04-20T21:18:24.125 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:23.481+0000 7fe8d3e00700 1 -- 172.21.15.62:0/4241417860 shutdown_connections 2024-04-20T21:18:24.125 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:23.481+0000 7fe8d3e00700 1 --2- 172.21.15.62:0/4241417860 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe8cc105b10 0x7fe8cc105f20 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:18:24.125 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:23.481+0000 7fe8d3e00700 1 -- 172.21.15.62:0/4241417860 >> 172.21.15.62:0/4241417860 conn(0x7fe8cc101180 msgr2=0x7fe8cc103590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:18:24.125 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:23.482+0000 7fe8d3e00700 1 -- 172.21.15.62:0/4241417860 shutdown_connections 2024-04-20T21:18:24.125 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:23.482+0000 7fe8d3e00700 1 -- 172.21.15.62:0/4241417860 wait complete. 2024-04-20T21:18:24.125 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:23.483+0000 7fe8d3e00700 1 Processor -- start 2024-04-20T21:18:24.125 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:23.483+0000 7fe8d3e00700 1 -- start start 2024-04-20T21:18:24.125 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:23.483+0000 7fe8d3e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe8cc105b10 0x7fe8cc07bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:18:24.128 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:23.483+0000 7fe8d3e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8cc07c180 con 0x7fe8cc105b10 2024-04-20T21:18:24.128 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:23.483+0000 7fe8d1400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe8cc105b10 0x7fe8cc07bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:18:24.128 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:23.483+0000 7fe8d1400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe8cc105b10 0x7fe8cc07bc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48742/0 (socket says 172.21.15.62:48742) 2024-04-20T21:18:24.128 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:23.483+0000 7fe8d1400700 1 -- 172.21.15.62:0/815681041 learned_addr learned my addr 172.21.15.62:0/815681041 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:18:24.128 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:23.484+0000 7fe8d1400700 1 -- 172.21.15.62:0/815681041 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8c0009550 con 0x7fe8cc105b10 2024-04-20T21:18:24.128 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:23.484+0000 7fe8d1400700 1 --2- 172.21.15.62:0/815681041 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe8cc105b10 0x7fe8cc07bc40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fe8c0000c00 tx=0x7fe8c0011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:18:24.128 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:23.484+0000 7fe8ca000700 1 -- 172.21.15.62:0/815681041 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe8c0003b90 con 0x7fe8cc105b10 2024-04-20T21:18:24.128 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:23.484+0000 7fe8ca000700 1 -- 172.21.15.62:0/815681041 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fe8c00248b0 con 0x7fe8cc105b10 2024-04-20T21:18:24.128 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:23.484+0000 7fe8ca000700 1 -- 172.21.15.62:0/815681041 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe8c001a8e0 con 0x7fe8cc105b10 2024-04-20T21:18:24.128 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:23.484+0000 7fe8d3e00700 1 -- 172.21.15.62:0/815681041 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8cc07a290 con 0x7fe8cc105b10 2024-04-20T21:18:24.129 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:23.485+0000 7fe8d3e00700 1 -- 172.21.15.62:0/815681041 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8cc07a6b0 con 0x7fe8cc105b10 2024-04-20T21:18:24.129 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:23.485+0000 7fe8ca000700 1 -- 172.21.15.62:0/815681041 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fe8c002d3f0 con 0x7fe8cc105b10 2024-04-20T21:18:24.129 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:23.485+0000 7fe8ca000700 1 -- 172.21.15.62:0/815681041 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fe8c0019450 con 0x7fe8cc105b10 2024-04-20T21:18:24.129 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:23.486+0000 7fe8d3e00700 1 -- 172.21.15.62:0/815681041 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8cc062380 con 0x7fe8cc105b10 2024-04-20T21:18:24.129 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:23.489+0000 7fe8ca000700 1 -- 172.21.15.62:0/815681041 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7fe8c0024db0 con 0x7fe8cc105b10 2024-04-20T21:18:24.129 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:23.567+0000 7fe8d3e00700 1 -- 172.21.15.62:0/815681041 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fe8cc107980 con 0x7fe8cc105b10 2024-04-20T21:18:24.129 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:23.568+0000 7fe8ca000700 1 -- 172.21.15.62:0/815681041 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7fe8c0024ad0 con 0x7fe8cc105b10 2024-04-20T21:18:24.129 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:23.570+0000 7fe8d3e00700 1 -- 172.21.15.62:0/815681041 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe8cc105b10 msgr2=0x7fe8cc07bc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:18:24.129 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:23.570+0000 7fe8d3e00700 1 --2- 172.21.15.62:0/815681041 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe8cc105b10 0x7fe8cc07bc40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fe8c0000c00 tx=0x7fe8c0011ab0 comp rx=0 tx=0).stop 2024-04-20T21:18:24.129 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:23.570+0000 7fe8d3e00700 1 -- 172.21.15.62:0/815681041 shutdown_connections 2024-04-20T21:18:24.129 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:23.570+0000 7fe8d3e00700 1 --2- 172.21.15.62:0/815681041 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe8cc105b10 0x7fe8cc07bc40 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:18:24.129 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:23.570+0000 7fe8d3e00700 1 -- 172.21.15.62:0/815681041 >> 172.21.15.62:0/815681041 conn(0x7fe8cc101180 msgr2=0x7fe8cc101f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:18:24.129 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:23.570+0000 7fe8d3e00700 1 -- 172.21.15.62:0/815681041 shutdown_connections 2024-04-20T21:18:24.129 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:23.570+0000 7fe8d3e00700 1 -- 172.21.15.62:0/815681041 wait complete. 2024-04-20T21:18:24.130 INFO:teuthology.orchestra.run.smithi062.stdout:Restarting the monitor... 2024-04-20T21:18:24.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:24 smithi062 systemd[1]: Stopping Ceph mon.smithi062 for 61ae3628-ff5b-11ee-bc93-c7b262605968... 2024-04-20T21:18:25.037 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:24 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-mon-smithi062[15847]: 2024-04-20T21:18:24.654+0000 7f864a000700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi062 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-20T21:18:25.037 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:24 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-mon-smithi062[15847]: 2024-04-20T21:18:24.654+0000 7f864a000700 -1 mon.smithi062@0(leader) e1 *** Got Signal Terminated *** 2024-04-20T21:18:25.037 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:24 smithi062 podman[16095]: 2024-04-20 21:18:24.757679515 +0000 UTC m=+0.223548336 container died ec5495668187ed6345b1c3fcb8705e5190770590062ea5da279d4afb24f712e4 (image=quay.io/ceph/ceph:v18.2.0, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-mon-smithi062, ceph=True, org.label-schema.build-date=20231212, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-20T21:18:25.364 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:25 smithi062 podman[16095]: 2024-04-20 21:18:25.164857889 +0000 UTC m=+0.630726703 container cleanup ec5495668187ed6345b1c3fcb8705e5190770590062ea5da279d4afb24f712e4 (image=quay.io/ceph/ceph:v18.2.0, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-mon-smithi062, org.label-schema.license=GPLv2, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20231212, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True) 2024-04-20T21:18:25.364 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:25 smithi062 bash[16095]: ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-mon-smithi062 2024-04-20T21:18:25.649 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:25 smithi062 podman[16109]: 2024-04-20 21:18:25.36526089 +0000 UTC m=+0.589662009 container remove ec5495668187ed6345b1c3fcb8705e5190770590062ea5da279d4afb24f712e4 (image=quay.io/ceph/ceph:v18.2.0, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-mon-smithi062, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20231212, CEPH_POINT_RELEASE=-18.2.0, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-20T21:18:25.943 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:25 smithi062 systemd[1]: ceph-61ae3628-ff5b-11ee-bc93-c7b262605968@mon.smithi062.service: Deactivated successfully. 2024-04-20T21:18:25.943 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:25 smithi062 systemd[1]: Stopped Ceph mon.smithi062 for 61ae3628-ff5b-11ee-bc93-c7b262605968. 2024-04-20T21:18:25.944 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:25 smithi062 systemd[1]: ceph-61ae3628-ff5b-11ee-bc93-c7b262605968@mon.smithi062.service: Consumed 2.036s CPU time. 2024-04-20T21:18:25.944 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:25 smithi062 systemd[1]: Starting Ceph mon.smithi062 for 61ae3628-ff5b-11ee-bc93-c7b262605968... 2024-04-20T21:18:26.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 podman[16201]: 2024-04-20 21:18:26.157763205 +0000 UTC m=+0.039750152 image pull dc2bc1663786ad5862ef6efcff18243ab7dfeda0bcb911ddf757a0d20feca946 quay.io/ceph/ceph:v18.2.0 2024-04-20T21:18:26.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 podman[16201]: 2024-04-20 21:18:26.266220679 +0000 UTC m=+0.148207660 container create bd6efbd5c380cda9118a1a2c96ddddc0fe61d4a9ab8109dbe18acf00687d2e4f (image=quay.io/ceph/ceph:v18.2.0, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-mon-smithi062, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=HEAD, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, org.label-schema.vendor=CentOS, org.label-schema.build-date=20231212, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-04-20T21:18:26.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 podman[16201]: 2024-04-20 21:18:26.486193226 +0000 UTC m=+0.368180226 container init bd6efbd5c380cda9118a1a2c96ddddc0fe61d4a9ab8109dbe18acf00687d2e4f (image=quay.io/ceph/ceph:v18.2.0, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-mon-smithi062, org.label-schema.schema-version=1.0, org.label-schema.build-date=20231212, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.0, maintainer=Guillaume Abrioux , GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, RELEASE=HEAD, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-04-20T21:18:26.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 podman[16201]: 2024-04-20 21:18:26.495871925 +0000 UTC m=+0.377858909 container start bd6efbd5c380cda9118a1a2c96ddddc0fe61d4a9ab8109dbe18acf00687d2e4f (image=quay.io/ceph/ceph:v18.2.0, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-mon-smithi062, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.0, ceph=True, org.label-schema.license=GPLv2, RELEASE=HEAD, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20231212, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308) 2024-04-20T21:18:26.585 INFO:teuthology.orchestra.run.smithi062.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-04-20T21:18:26.874 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: set uid:gid to 167:167 (ceph:ceph) 2024-04-20T21:18:26.874 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable), process ceph-mon, pid 2 2024-04-20T21:18:26.874 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: pidfile_write: ignore empty --pid-file 2024-04-20T21:18:26.874 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: load: jerasure load: lrc 2024-04-20T21:18:26.874 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: RocksDB version: 7.9.2 2024-04-20T21:18:26.874 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Git sha 0 2024-04-20T21:18:26.874 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Compile date 2023-08-03 19:21:13 2024-04-20T21:18:26.874 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: DB SUMMARY 2024-04-20T21:18:26.874 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: DB Session ID: 98UV0L1FYLBYYURZQ562 2024-04-20T21:18:26.874 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: CURRENT file: CURRENT 2024-04-20T21:18:26.874 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: IDENTITY file: IDENTITY 2024-04-20T21:18:26.875 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-04-20T21:18:26.875 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi062/store.db dir, Total Num: 1, files: 000008.sst 2024-04-20T21:18:26.875 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi062/store.db: 000009.log size: 98264 ; 2024-04-20T21:18:26.875 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.error_if_exists: 0 2024-04-20T21:18:26.875 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.create_if_missing: 0 2024-04-20T21:18:26.875 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.paranoid_checks: 1 2024-04-20T21:18:26.875 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-20T21:18:26.875 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-20T21:18:26.875 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-20T21:18:26.875 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.env: 0x562bd3815720 2024-04-20T21:18:26.875 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.fs: PosixFileSystem 2024-04-20T21:18:26.875 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.info_log: 0x562bedbc72c0 2024-04-20T21:18:26.875 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.max_file_opening_threads: 16 2024-04-20T21:18:26.875 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.statistics: (nil) 2024-04-20T21:18:26.875 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.use_fsync: 0 2024-04-20T21:18:26.875 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.max_log_file_size: 0 2024-04-20T21:18:26.876 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-20T21:18:26.876 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-20T21:18:26.876 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.keep_log_file_num: 1000 2024-04-20T21:18:26.876 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.recycle_log_file_num: 0 2024-04-20T21:18:26.876 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.allow_fallocate: 1 2024-04-20T21:18:26.876 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.allow_mmap_reads: 0 2024-04-20T21:18:26.876 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.allow_mmap_writes: 0 2024-04-20T21:18:26.876 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.use_direct_reads: 0 2024-04-20T21:18:26.876 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-20T21:18:26.876 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.create_missing_column_families: 0 2024-04-20T21:18:26.876 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.db_log_dir: 2024-04-20T21:18:26.876 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.wal_dir: 2024-04-20T21:18:26.876 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-20T21:18:26.877 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-20T21:18:26.877 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-20T21:18:26.877 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-20T21:18:26.877 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-20T21:18:26.877 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-20T21:18:26.877 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.advise_random_on_open: 1 2024-04-20T21:18:26.877 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.db_write_buffer_size: 0 2024-04-20T21:18:26.877 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.write_buffer_manager: 0x562bece383c0 2024-04-20T21:18:26.877 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-20T21:18:26.877 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-20T21:18:26.877 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-20T21:18:26.877 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.rate_limiter: (nil) 2024-04-20T21:18:26.877 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-20T21:18:26.877 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.wal_recovery_mode: 2 2024-04-20T21:18:26.878 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.enable_thread_tracking: 0 2024-04-20T21:18:26.878 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.enable_pipelined_write: 0 2024-04-20T21:18:26.878 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.unordered_write: 0 2024-04-20T21:18:26.878 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-20T21:18:26.878 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-20T21:18:26.878 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-20T21:18:26.878 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-20T21:18:26.878 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.row_cache: None 2024-04-20T21:18:26.878 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.wal_filter: None 2024-04-20T21:18:26.878 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-20T21:18:26.878 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.allow_ingest_behind: 0 2024-04-20T21:18:26.878 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.two_write_queues: 0 2024-04-20T21:18:26.878 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.manual_wal_flush: 0 2024-04-20T21:18:26.878 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.wal_compression: 0 2024-04-20T21:18:26.878 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.atomic_flush: 0 2024-04-20T21:18:26.879 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-20T21:18:26.879 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-20T21:18:26.879 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-20T21:18:26.879 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.log_readahead_size: 0 2024-04-20T21:18:26.879 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-20T21:18:26.879 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.best_efforts_recovery: 0 2024-04-20T21:18:26.879 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-20T21:18:26.879 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-20T21:18:26.879 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.allow_data_in_errors: 0 2024-04-20T21:18:26.879 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.db_host_id: __hostname__ 2024-04-20T21:18:26.879 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-20T21:18:26.880 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.max_background_jobs: 2 2024-04-20T21:18:26.880 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.max_background_compactions: -1 2024-04-20T21:18:26.880 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.max_subcompactions: 1 2024-04-20T21:18:26.880 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-20T21:18:26.880 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-20T21:18:26.880 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-20T21:18:26.880 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.max_total_wal_size: 0 2024-04-20T21:18:26.880 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-20T21:18:26.880 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-20T21:18:26.880 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-20T21:18:26.880 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-20T21:18:26.880 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.max_open_files: -1 2024-04-20T21:18:26.880 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.bytes_per_sync: 0 2024-04-20T21:18:26.881 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-20T21:18:26.881 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-20T21:18:26.881 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.compaction_readahead_size: 0 2024-04-20T21:18:26.881 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.max_background_flushes: -1 2024-04-20T21:18:26.881 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Compression algorithms supported: 2024-04-20T21:18:26.881 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-20T21:18:26.881 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: kZSTD supported: 0 2024-04-20T21:18:26.881 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: kXpressCompression supported: 0 2024-04-20T21:18:26.881 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: kLZ4HCCompression supported: 1 2024-04-20T21:18:26.881 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: kZlibCompression supported: 1 2024-04-20T21:18:26.881 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: kSnappyCompression supported: 1 2024-04-20T21:18:26.881 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: kLZ4Compression supported: 1 2024-04-20T21:18:26.881 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: kBZip2Compression supported: 0 2024-04-20T21:18:26.881 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-20T21:18:26.881 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-20T21:18:26.881 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi062/store.db/MANIFEST-000010 2024-04-20T21:18:26.882 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-20T21:18:26.882 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-20T21:18:26.882 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.merge_operator: 2024-04-20T21:18:26.882 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.compaction_filter: None 2024-04-20T21:18:26.882 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.compaction_filter_factory: None 2024-04-20T21:18:26.882 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.sst_partitioner_factory: None 2024-04-20T21:18:26.882 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-20T21:18:26.882 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-20T21:18:26.882 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x562bedbc73c0) 2024-04-20T21:18:26.882 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: cache_index_and_filter_blocks: 1 2024-04-20T21:18:26.882 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-20T21:18:26.882 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-20T21:18:26.882 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: pin_top_level_index_and_filter: 1 2024-04-20T21:18:26.882 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: index_type: 0 2024-04-20T21:18:26.882 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: data_block_index_type: 0 2024-04-20T21:18:26.883 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: index_shortening: 1 2024-04-20T21:18:26.883 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-20T21:18:26.883 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: checksum: 4 2024-04-20T21:18:26.883 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: no_block_cache: 0 2024-04-20T21:18:26.883 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: block_cache: 0x562beced9350 2024-04-20T21:18:26.883 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: block_cache_name: BinnedLRUCache 2024-04-20T21:18:26.883 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: block_cache_options: 2024-04-20T21:18:26.883 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: capacity : 536870912 2024-04-20T21:18:26.883 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: num_shard_bits : 4 2024-04-20T21:18:26.883 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: strict_capacity_limit : 0 2024-04-20T21:18:26.883 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: high_pri_pool_ratio: 0.000 2024-04-20T21:18:26.883 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: block_cache_compressed: (nil) 2024-04-20T21:18:26.883 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: persistent_cache: (nil) 2024-04-20T21:18:26.883 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: block_size: 4096 2024-04-20T21:18:26.884 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: block_size_deviation: 10 2024-04-20T21:18:26.884 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: block_restart_interval: 16 2024-04-20T21:18:26.884 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: index_block_restart_interval: 1 2024-04-20T21:18:26.884 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: metadata_block_size: 4096 2024-04-20T21:18:26.884 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: partition_filters: 0 2024-04-20T21:18:26.884 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: use_delta_encoding: 1 2024-04-20T21:18:26.884 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: filter_policy: bloomfilter 2024-04-20T21:18:26.884 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: whole_key_filtering: 1 2024-04-20T21:18:26.884 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: verify_compression: 0 2024-04-20T21:18:26.884 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: read_amp_bytes_per_bit: 0 2024-04-20T21:18:26.884 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: format_version: 5 2024-04-20T21:18:26.884 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: enable_index_compression: 1 2024-04-20T21:18:26.884 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: block_align: 0 2024-04-20T21:18:26.884 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: max_auto_readahead_size: 262144 2024-04-20T21:18:26.884 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: prepopulate_block_cache: 0 2024-04-20T21:18:26.884 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: initial_auto_readahead_size: 8192 2024-04-20T21:18:26.885 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: num_file_reads_for_auto_readahead: 2 2024-04-20T21:18:26.885 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.write_buffer_size: 33554432 2024-04-20T21:18:26.885 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.max_write_buffer_number: 2 2024-04-20T21:18:26.885 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.compression: NoCompression 2024-04-20T21:18:26.885 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.bottommost_compression: Disabled 2024-04-20T21:18:26.885 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.prefix_extractor: nullptr 2024-04-20T21:18:26.885 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-20T21:18:26.885 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.num_levels: 7 2024-04-20T21:18:26.885 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-20T21:18:26.885 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-20T21:18:26.885 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-20T21:18:26.885 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-20T21:18:26.885 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-20T21:18:26.885 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-20T21:18:26.885 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-20T21:18:26.885 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-20T21:18:26.889 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-20T21:18:26.890 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-20T21:18:26.890 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-20T21:18:26.890 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-20T21:18:26.890 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-20T21:18:26.890 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.compression_opts.level: 32767 2024-04-20T21:18:26.890 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.compression_opts.strategy: 0 2024-04-20T21:18:26.890 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-20T21:18:26.890 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-20T21:18:26.890 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-20T21:18:26.890 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-20T21:18:26.890 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.compression_opts.enabled: false 2024-04-20T21:18:26.890 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-20T21:18:26.891 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-20T21:18:26.891 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-20T21:18:26.891 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-20T21:18:26.891 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.target_file_size_base: 67108864 2024-04-20T21:18:26.891 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-20T21:18:26.891 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-20T21:18:26.891 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-20T21:18:26.891 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-20T21:18:26.891 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-20T21:18:26.891 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-20T21:18:26.891 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-20T21:18:26.891 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-20T21:18:26.891 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-20T21:18:26.891 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-20T21:18:26.892 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-20T21:18:26.892 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-20T21:18:26.892 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-20T21:18:26.892 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-20T21:18:26.892 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.arena_block_size: 1048576 2024-04-20T21:18:26.892 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-20T21:18:26.892 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-20T21:18:26.892 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.disable_auto_compactions: 0 2024-04-20T21:18:26.892 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-20T21:18:26.892 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-20T21:18:26.892 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-20T21:18:26.892 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-20T21:18:26.892 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-20T21:18:26.892 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-20T21:18:26.893 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-20T21:18:26.893 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-20T21:18:26.893 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-20T21:18:26.893 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-20T21:18:26.893 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.table_properties_collectors: 2024-04-20T21:18:26.893 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.inplace_update_support: 0 2024-04-20T21:18:26.893 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-20T21:18:26.893 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-20T21:18:26.893 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-20T21:18:26.893 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-20T21:18:26.893 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.bloom_locality: 0 2024-04-20T21:18:26.893 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.max_successive_merges: 0 2024-04-20T21:18:26.893 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-20T21:18:26.893 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.paranoid_file_checks: 0 2024-04-20T21:18:26.894 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.force_consistency_checks: 1 2024-04-20T21:18:26.894 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.report_bg_io_stats: 0 2024-04-20T21:18:26.894 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.ttl: 2592000 2024-04-20T21:18:26.894 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-20T21:18:26.894 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-20T21:18:26.894 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-20T21:18:26.894 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.enable_blob_files: false 2024-04-20T21:18:26.894 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.min_blob_size: 0 2024-04-20T21:18:26.894 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.blob_file_size: 268435456 2024-04-20T21:18:26.894 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-20T21:18:26.894 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-20T21:18:26.894 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-20T21:18:26.894 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-20T21:18:26.894 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-20T21:18:26.895 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.blob_file_starting_level: 0 2024-04-20T21:18:26.895 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-20T21:18:26.895 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi062/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-04-20T21:18:26.895 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-04-20T21:18:26.895 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 6e1ad0b0-7dc2-473c-b9e3-099eb8c1edd2 2024-04-20T21:18:26.895 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713647906552584, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-04-20T21:18:26.895 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-04-20T21:18:26.895 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713647906554480, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 93515, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 314, "table_properties": {"data_size": 91537, "index_size": 260, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13576, "raw_average_key_size": 49, "raw_value_size": 84146, "raw_average_value_size": 308, "num_data_blocks": 11, "num_entries": 273, "num_filter_entries": 273, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1713647906, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "6e1ad0b0-7dc2-473c-b9e3-099eb8c1edd2", "db_session_id": "98UV0L1FYLBYYURZQ562", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-04-20T21:18:26.895 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713647906554633, "job": 1, "event": "recovery_finished"} 2024-04-20T21:18:26.895 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-04-20T21:18:26.895 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi062/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-20T21:18:26.895 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x562becf76000 2024-04-20T21:18:26.895 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: DB pointer 0x562becf62000 2024-04-20T21:18:26.895 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-20T21:18:26.895 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-20T21:18:26.896 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: ** DB Stats ** 2024-04-20T21:18:26.896 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-20T21:18:26.896 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-20T21:18:26.896 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-20T21:18:26.896 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-20T21:18:26.896 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-20T21:18:26.896 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-20T21:18:26.896 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-20T21:18:26.896 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: 2024-04-20T21:18:26.896 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: ** Compaction Stats [default] ** 2024-04-20T21:18:26.896 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-20T21:18:26.896 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-20T21:18:26.896 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: L0 2/0 93.20 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 77.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-20T21:18:26.896 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: Sum 2/0 93.20 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 77.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-20T21:18:26.896 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 77.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-20T21:18:26.896 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: 2024-04-20T21:18:26.897 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: ** Compaction Stats [default] ** 2024-04-20T21:18:26.897 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-20T21:18:26.897 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-20T21:18:26.897 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 77.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-20T21:18:26.897 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: 2024-04-20T21:18:26.897 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-20T21:18:26.897 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: 2024-04-20T21:18:26.897 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-20T21:18:26.897 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-20T21:18:26.897 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-20T21:18:26.897 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-20T21:18:26.897 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-20T21:18:26.897 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-20T21:18:26.897 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: Cumulative compaction: 0.00 GB write, 13.05 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-20T21:18:26.898 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: Interval compaction: 0.00 GB write, 13.05 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-20T21:18:26.898 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-20T21:18:26.898 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: Block cache BinnedLRUCache@0x562beced9350#2 capacity: 512.00 MB usage: 1.36 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-04-20T21:18:26.898 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2024-04-20T21:18:26.898 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: 2024-04-20T21:18:26.898 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-20T21:18:26.898 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: starting mon.smithi062 rank 0 at public addrs [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] at bind addrs [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi062 fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 2024-04-20T21:18:26.898 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: mon.smithi062@-1(???) e1 preinit fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 2024-04-20T21:18:26.898 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: mon.smithi062@-1(???).mds e1 new map 2024-04-20T21:18:26.898 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: mon.smithi062@-1(???).mds e1 print_map 2024-04-20T21:18:26.898 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: e1 2024-04-20T21:18:26.898 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T21:18:26.898 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T21:18:26.898 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: legacy client fscid: -1 2024-04-20T21:18:26.898 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: 2024-04-20T21:18:26.899 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: No filesystems configured 2024-04-20T21:18:26.899 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: mon.smithi062@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-20T21:18:26.899 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: mon.smithi062@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-20T21:18:26.899 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: mon.smithi062@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-20T21:18:26.899 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: mon.smithi062@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-20T21:18:26.899 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: mon.smithi062@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-20T21:18:26.899 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 bash[16201]: bd6efbd5c380cda9118a1a2c96ddddc0fe61d4a9ab8109dbe18acf00687d2e4f 2024-04-20T21:18:26.899 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 systemd[1]: Started Ceph mon.smithi062 for 61ae3628-ff5b-11ee-bc93-c7b262605968. 2024-04-20T21:18:26.899 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: mon.smithi062 is new leader, mons smithi062 in quorum (ranks 0) 2024-04-20T21:18:26.899 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: monmap e1: 1 mons at {smithi062=[v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0]} removed_ranks: {} 2024-04-20T21:18:26.899 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: fsmap 2024-04-20T21:18:26.899 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: osdmap e1: 0 total, 0 up, 0 in 2024-04-20T21:18:26.899 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:26 smithi062 ceph-mon[16221]: mgrmap e1: no daemons active 2024-04-20T21:18:27.908 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:27.164+0000 7f6df7c00700 1 Processor -- start 2024-04-20T21:18:27.908 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:27.164+0000 7f6df7c00700 1 -- start start 2024-04-20T21:18:27.908 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:27.164+0000 7f6df7c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6df007ace0 0x7f6df00791e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:18:27.908 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:27.164+0000 7f6df7c00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6df0079720 con 0x7f6df007ace0 2024-04-20T21:18:27.908 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:27.165+0000 7f6df5200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6df007ace0 0x7f6df00791e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:18:27.908 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:27.165+0000 7f6df5200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6df007ace0 0x7f6df00791e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48756/0 (socket says 172.21.15.62:48756) 2024-04-20T21:18:27.908 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:27.165+0000 7f6df5200700 1 -- 172.21.15.62:0/3210294507 learned_addr learned my addr 172.21.15.62:0/3210294507 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:18:27.908 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:27.165+0000 7f6df5200700 1 -- 172.21.15.62:0/3210294507 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6df0079860 con 0x7f6df007ace0 2024-04-20T21:18:27.908 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:27.166+0000 7f6df5200700 1 --2- 172.21.15.62:0/3210294507 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6df007ace0 0x7f6df00791e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f6de40098a0 tx=0x7f6de4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddd6ad9efaea0100 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:18:27.908 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:27.166+0000 7f6def400700 1 -- 172.21.15.62:0/3210294507 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6de4004180 con 0x7f6df007ace0 2024-04-20T21:18:27.908 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:27.166+0000 7f6def400700 1 -- 172.21.15.62:0/3210294507 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f6de40042e0 con 0x7f6df007ace0 2024-04-20T21:18:27.909 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:27.166+0000 7f6def400700 1 -- 172.21.15.62:0/3210294507 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6de4011a50 con 0x7f6df007ace0 2024-04-20T21:18:27.909 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:27.167+0000 7f6df7c00700 1 -- 172.21.15.62:0/3210294507 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6df007ace0 msgr2=0x7f6df00791e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:18:27.909 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:27.167+0000 7f6df7c00700 1 --2- 172.21.15.62:0/3210294507 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6df007ace0 0x7f6df00791e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f6de40098a0 tx=0x7f6de4009bb0 comp rx=0 tx=0).stop 2024-04-20T21:18:27.909 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:27.167+0000 7f6df7c00700 1 -- 172.21.15.62:0/3210294507 shutdown_connections 2024-04-20T21:18:27.909 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:27.167+0000 7f6df7c00700 1 --2- 172.21.15.62:0/3210294507 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6df007ace0 0x7f6df00791e0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:18:27.909 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:27.167+0000 7f6df7c00700 1 -- 172.21.15.62:0/3210294507 >> 172.21.15.62:0/3210294507 conn(0x7f6df0101160 msgr2=0x7f6df0103590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:18:27.909 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:27.167+0000 7f6df7c00700 1 -- 172.21.15.62:0/3210294507 shutdown_connections 2024-04-20T21:18:27.909 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:27.167+0000 7f6df7c00700 1 -- 172.21.15.62:0/3210294507 wait complete. 2024-04-20T21:18:27.909 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:27.169+0000 7f6df7c00700 1 Processor -- start 2024-04-20T21:18:27.909 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:27.169+0000 7f6df7c00700 1 -- start start 2024-04-20T21:18:27.909 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:27.169+0000 7f6df7c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6df007ace0 0x7f6df019ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:18:27.909 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:27.169+0000 7f6df7c00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6df019bfb0 con 0x7f6df007ace0 2024-04-20T21:18:27.909 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:27.169+0000 7f6df5200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6df007ace0 0x7f6df019ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:18:27.910 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:27.170+0000 7f6df5200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6df007ace0 0x7f6df019ba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48764/0 (socket says 172.21.15.62:48764) 2024-04-20T21:18:27.910 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:27.170+0000 7f6df5200700 1 -- 172.21.15.62:0/4227537429 learned_addr learned my addr 172.21.15.62:0/4227537429 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:18:27.910 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:27.170+0000 7f6df5200700 1 -- 172.21.15.62:0/4227537429 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6de4009550 con 0x7f6df007ace0 2024-04-20T21:18:27.910 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:27.170+0000 7f6df5200700 1 --2- 172.21.15.62:0/4227537429 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6df007ace0 0x7f6df019ba70 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f6de4006af0 tx=0x7f6de4011a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:18:27.910 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:27.171+0000 7f6ded600700 1 -- 172.21.15.62:0/4227537429 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6de4003b90 con 0x7f6df007ace0 2024-04-20T21:18:27.910 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:27.171+0000 7f6ded600700 1 -- 172.21.15.62:0/4227537429 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f6de401a8a0 con 0x7f6df007ace0 2024-04-20T21:18:27.910 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:27.171+0000 7f6df7c00700 1 -- 172.21.15.62:0/4227537429 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6df019c1b0 con 0x7f6df007ace0 2024-04-20T21:18:27.910 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:27.171+0000 7f6ded600700 1 -- 172.21.15.62:0/4227537429 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6de4019880 con 0x7f6df007ace0 2024-04-20T21:18:27.910 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:27.171+0000 7f6df7c00700 1 -- 172.21.15.62:0/4227537429 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6df019c5d0 con 0x7f6df007ace0 2024-04-20T21:18:27.910 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:27.172+0000 7f6ded600700 1 -- 172.21.15.62:0/4227537429 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f6de4019af0 con 0x7f6df007ace0 2024-04-20T21:18:27.910 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:27.172+0000 7f6ded600700 1 -- 172.21.15.62:0/4227537429 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f6de40243f0 con 0x7f6df007ace0 2024-04-20T21:18:27.910 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:27.172+0000 7f6df7c00700 1 -- 172.21.15.62:0/4227537429 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6dd0005320 con 0x7f6df007ace0 2024-04-20T21:18:27.910 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:27.176+0000 7f6ded600700 1 -- 172.21.15.62:0/4227537429 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7f6de401aa10 con 0x7f6df007ace0 2024-04-20T21:18:27.911 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:27.255+0000 7f6df7c00700 1 -- 172.21.15.62:0/4227537429 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f6dd0005cc0 con 0x7f6df007ace0 2024-04-20T21:18:27.911 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:27.260+0000 7f6ded600700 1 -- 172.21.15.62:0/4227537429 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f6de402d440 con 0x7f6df007ace0 2024-04-20T21:18:27.911 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:27.262+0000 7f6df7c00700 1 -- 172.21.15.62:0/4227537429 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6df007ace0 msgr2=0x7f6df019ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:18:27.911 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:27.262+0000 7f6df7c00700 1 --2- 172.21.15.62:0/4227537429 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6df007ace0 0x7f6df019ba70 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f6de4006af0 tx=0x7f6de4011a90 comp rx=0 tx=0).stop 2024-04-20T21:18:27.911 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:27.262+0000 7f6df7c00700 1 -- 172.21.15.62:0/4227537429 shutdown_connections 2024-04-20T21:18:27.911 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:27.262+0000 7f6df7c00700 1 --2- 172.21.15.62:0/4227537429 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6df007ace0 0x7f6df019ba70 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:18:27.911 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:27.262+0000 7f6df7c00700 1 -- 172.21.15.62:0/4227537429 >> 172.21.15.62:0/4227537429 conn(0x7f6df0101160 msgr2=0x7f6df0102810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:18:27.911 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:27.262+0000 7f6df7c00700 1 -- 172.21.15.62:0/4227537429 shutdown_connections 2024-04-20T21:18:27.911 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:27.262+0000 7f6df7c00700 1 -- 172.21.15.62:0/4227537429 wait complete. 2024-04-20T21:18:27.911 INFO:teuthology.orchestra.run.smithi062.stdout:Wrote config to /etc/ceph/ceph.conf 2024-04-20T21:18:27.930 INFO:teuthology.orchestra.run.smithi062.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-20T21:18:27.930 INFO:teuthology.orchestra.run.smithi062.stdout:Creating mgr... 2024-04-20T21:18:27.931 INFO:teuthology.orchestra.run.smithi062.stdout:Verifying port 0.0.0.0:9283 ... 2024-04-20T21:18:27.932 INFO:teuthology.orchestra.run.smithi062.stdout:Verifying port 0.0.0.0:8765 ... 2024-04-20T21:18:27.932 INFO:teuthology.orchestra.run.smithi062.stdout:Verifying port 0.0.0.0:8443 ... 2024-04-20T21:18:28.341 INFO:teuthology.orchestra.run.smithi062.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-61ae3628-ff5b-11ee-bc93-c7b262605968@mgr.smithi062.rqxuxi 2024-04-20T21:18:28.342 INFO:teuthology.orchestra.run.smithi062.stdout:systemctl: stderr Failed to reset failed state of unit ceph-61ae3628-ff5b-11ee-bc93-c7b262605968@mgr.smithi062.rqxuxi.service: Unit ceph-61ae3628-ff5b-11ee-bc93-c7b262605968@mgr.smithi062.rqxuxi.service not loaded. 2024-04-20T21:18:28.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:28 smithi062 ceph-mon[16221]: from='client.? 172.21.15.62:0/4227537429' entity='client.admin' 2024-04-20T21:18:28.601 INFO:teuthology.orchestra.run.smithi062.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-61ae3628-ff5b-11ee-bc93-c7b262605968.target.wants/ceph-61ae3628-ff5b-11ee-bc93-c7b262605968@mgr.smithi062.rqxuxi.service → /etc/systemd/system/ceph-61ae3628-ff5b-11ee-bc93-c7b262605968@.service. 2024-04-20T21:18:29.551 INFO:teuthology.orchestra.run.smithi062.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-20T21:18:29.552 INFO:teuthology.orchestra.run.smithi062.stdout:systemctl: stdout disabled 2024-04-20T21:18:29.575 INFO:teuthology.orchestra.run.smithi062.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-20T21:18:29.576 INFO:teuthology.orchestra.run.smithi062.stdout:systemctl: stdout inactive 2024-04-20T21:18:29.576 INFO:teuthology.orchestra.run.smithi062.stdout:firewalld.service is not enabled 2024-04-20T21:18:29.576 INFO:teuthology.orchestra.run.smithi062.stdout:Not possible to enable service . firewalld.service is not available 2024-04-20T21:18:29.609 INFO:teuthology.orchestra.run.smithi062.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-20T21:18:29.609 INFO:teuthology.orchestra.run.smithi062.stdout:systemctl: stdout disabled 2024-04-20T21:18:29.633 INFO:teuthology.orchestra.run.smithi062.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-20T21:18:29.633 INFO:teuthology.orchestra.run.smithi062.stdout:systemctl: stdout inactive 2024-04-20T21:18:29.633 INFO:teuthology.orchestra.run.smithi062.stdout:firewalld.service is not enabled 2024-04-20T21:18:29.633 INFO:teuthology.orchestra.run.smithi062.stdout:Not possible to open ports <[9283, 8765, 8443]>. firewalld.service is not available 2024-04-20T21:18:29.634 INFO:teuthology.orchestra.run.smithi062.stdout:Waiting for mgr to start... 2024-04-20T21:18:29.634 INFO:teuthology.orchestra.run.smithi062.stdout:Waiting for mgr... 2024-04-20T21:18:30.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:30 smithi062 ceph-mon[16221]: from='client.? 172.21.15.62:0/3359004569' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-20T21:18:31.224 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout 2024-04-20T21:18:31.224 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout { 2024-04-20T21:18:31.225 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "fsid": "61ae3628-ff5b-11ee-bc93-c7b262605968", 2024-04-20T21:18:31.225 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "health": { 2024-04-20T21:18:31.225 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-20T21:18:31.225 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-20T21:18:31.225 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-20T21:18:31.225 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout }, 2024-04-20T21:18:31.225 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-20T21:18:31.225 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-20T21:18:31.225 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout 0 2024-04-20T21:18:31.225 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout ], 2024-04-20T21:18:31.225 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-20T21:18:31.225 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "smithi062" 2024-04-20T21:18:31.226 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout ], 2024-04-20T21:18:31.226 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "quorum_age": 3, 2024-04-20T21:18:31.226 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-20T21:18:31.226 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-20T21:18:31.226 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-04-20T21:18:31.226 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-20T21:18:31.226 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout }, 2024-04-20T21:18:31.226 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-20T21:18:31.226 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-20T21:18:31.226 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-20T21:18:31.226 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-20T21:18:31.226 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-20T21:18:31.235 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-20T21:18:31.235 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-20T21:18:31.235 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-20T21:18:31.235 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout }, 2024-04-20T21:18:31.235 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-20T21:18:31.235 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-20T21:18:31.235 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-20T21:18:31.235 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-20T21:18:31.235 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-20T21:18:31.236 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-20T21:18:31.236 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-20T21:18:31.236 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-20T21:18:31.236 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-20T21:18:31.236 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout }, 2024-04-20T21:18:31.236 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-20T21:18:31.236 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-20T21:18:31.236 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-20T21:18:31.236 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-20T21:18:31.236 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout }, 2024-04-20T21:18:31.236 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-20T21:18:31.236 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-20T21:18:31.236 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-20T21:18:31.237 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-20T21:18:31.237 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-20T21:18:31.237 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-20T21:18:31.237 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "restful" 2024-04-20T21:18:31.237 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout ], 2024-04-20T21:18:31.237 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-20T21:18:31.237 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout }, 2024-04-20T21:18:31.237 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-20T21:18:31.237 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-20T21:18:31.237 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "modified": "2024-04-20T21:18:19.951841+0000", 2024-04-20T21:18:31.237 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-20T21:18:31.237 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout }, 2024-04-20T21:18:31.237 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-20T21:18:31.237 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout } 2024-04-20T21:18:31.237 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:30.261+0000 7fa75ac00700 1 Processor -- start 2024-04-20T21:18:31.238 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:30.262+0000 7fa75ac00700 1 -- start start 2024-04-20T21:18:31.238 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:30.262+0000 7fa75ac00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa754105b10 0x7fa754105f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:18:31.238 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:30.262+0000 7fa75ac00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa754106460 con 0x7fa754105b10 2024-04-20T21:18:31.238 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:30.263+0000 7fa753e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa754105b10 0x7fa754105f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:18:31.238 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:30.263+0000 7fa753e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa754105b10 0x7fa754105f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48798/0 (socket says 172.21.15.62:48798) 2024-04-20T21:18:31.238 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:30.263+0000 7fa753e00700 1 -- 172.21.15.62:0/603869680 learned_addr learned my addr 172.21.15.62:0/603869680 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:18:31.238 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:30.263+0000 7fa753e00700 1 -- 172.21.15.62:0/603869680 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7541065a0 con 0x7fa754105b10 2024-04-20T21:18:31.238 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:30.264+0000 7fa753e00700 1 --2- 172.21.15.62:0/603869680 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa754105b10 0x7fa754105f20 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fa7440098a0 tx=0x7fa744009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1caed827c35d6e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:18:31.238 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:30.264+0000 7fa752a00700 1 -- 172.21.15.62:0/603869680 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa744004180 con 0x7fa754105b10 2024-04-20T21:18:31.238 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:30.264+0000 7fa752a00700 1 -- 172.21.15.62:0/603869680 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fa7440042e0 con 0x7fa754105b10 2024-04-20T21:18:31.238 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:30.264+0000 7fa752a00700 1 -- 172.21.15.62:0/603869680 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa744011a50 con 0x7fa754105b10 2024-04-20T21:18:31.238 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:30.265+0000 7fa75ac00700 1 -- 172.21.15.62:0/603869680 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa754105b10 msgr2=0x7fa754105f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:18:31.238 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:30.265+0000 7fa75ac00700 1 --2- 172.21.15.62:0/603869680 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa754105b10 0x7fa754105f20 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fa7440098a0 tx=0x7fa744009bb0 comp rx=0 tx=0).stop 2024-04-20T21:18:31.239 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:30.265+0000 7fa75ac00700 1 -- 172.21.15.62:0/603869680 shutdown_connections 2024-04-20T21:18:31.239 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:30.265+0000 7fa75ac00700 1 --2- 172.21.15.62:0/603869680 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa754105b10 0x7fa754105f20 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:18:31.239 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:30.265+0000 7fa75ac00700 1 -- 172.21.15.62:0/603869680 >> 172.21.15.62:0/603869680 conn(0x7fa754101180 msgr2=0x7fa754103590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:18:31.239 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:30.265+0000 7fa75ac00700 1 -- 172.21.15.62:0/603869680 shutdown_connections 2024-04-20T21:18:31.239 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:30.265+0000 7fa75ac00700 1 -- 172.21.15.62:0/603869680 wait complete. 2024-04-20T21:18:31.239 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:30.266+0000 7fa75ac00700 1 Processor -- start 2024-04-20T21:18:31.239 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:30.266+0000 7fa75ac00700 1 -- start start 2024-04-20T21:18:31.239 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:30.267+0000 7fa75ac00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa754105b10 0x7fa75407bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:18:31.239 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:30.267+0000 7fa75ac00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa75407c180 con 0x7fa754105b10 2024-04-20T21:18:31.239 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:30.267+0000 7fa753e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa754105b10 0x7fa75407bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:18:31.239 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:30.267+0000 7fa753e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa754105b10 0x7fa75407bc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48800/0 (socket says 172.21.15.62:48800) 2024-04-20T21:18:31.239 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:30.267+0000 7fa753e00700 1 -- 172.21.15.62:0/3359004569 learned_addr learned my addr 172.21.15.62:0/3359004569 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:18:31.239 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:30.267+0000 7fa753e00700 1 -- 172.21.15.62:0/3359004569 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa744009550 con 0x7fa754105b10 2024-04-20T21:18:31.240 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:30.268+0000 7fa753e00700 1 --2- 172.21.15.62:0/3359004569 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa754105b10 0x7fa75407bc40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fa744000c00 tx=0x7fa744011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:18:31.240 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:30.268+0000 7fa750c00700 1 -- 172.21.15.62:0/3359004569 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa744003b90 con 0x7fa754105b10 2024-04-20T21:18:31.240 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:30.268+0000 7fa750c00700 1 -- 172.21.15.62:0/3359004569 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fa7440248b0 con 0x7fa754105b10 2024-04-20T21:18:31.240 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:30.268+0000 7fa75ac00700 1 -- 172.21.15.62:0/3359004569 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa75407a290 con 0x7fa754105b10 2024-04-20T21:18:31.240 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:30.268+0000 7fa750c00700 1 -- 172.21.15.62:0/3359004569 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa74401a8e0 con 0x7fa754105b10 2024-04-20T21:18:31.240 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:30.268+0000 7fa75ac00700 1 -- 172.21.15.62:0/3359004569 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa75407a6b0 con 0x7fa754105b10 2024-04-20T21:18:31.240 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:30.268+0000 7fa750c00700 1 -- 172.21.15.62:0/3359004569 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fa74402d3f0 con 0x7fa754105b10 2024-04-20T21:18:31.240 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:30.268+0000 7fa750c00700 1 -- 172.21.15.62:0/3359004569 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fa744019520 con 0x7fa754105b10 2024-04-20T21:18:31.240 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:30.269+0000 7fa75ac00700 1 -- 172.21.15.62:0/3359004569 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa754062380 con 0x7fa754105b10 2024-04-20T21:18:31.240 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:30.274+0000 7fa750c00700 1 -- 172.21.15.62:0/3359004569 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7fa744024db0 con 0x7fa754105b10 2024-04-20T21:18:31.240 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:30.355+0000 7fa75ac00700 1 -- 172.21.15.62:0/3359004569 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fa75407ad10 con 0x7fa754105b10 2024-04-20T21:18:31.240 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:30.356+0000 7fa750c00700 1 -- 172.21.15.62:0/3359004569 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1246 (secure 0 0 0) 0x7fa75407ad10 con 0x7fa754105b10 2024-04-20T21:18:31.240 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:30.357+0000 7fa75ac00700 1 -- 172.21.15.62:0/3359004569 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa754105b10 msgr2=0x7fa75407bc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:18:31.240 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:30.357+0000 7fa75ac00700 1 --2- 172.21.15.62:0/3359004569 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa754105b10 0x7fa75407bc40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fa744000c00 tx=0x7fa744011ab0 comp rx=0 tx=0).stop 2024-04-20T21:18:31.240 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:30.357+0000 7fa75ac00700 1 -- 172.21.15.62:0/3359004569 shutdown_connections 2024-04-20T21:18:31.241 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:30.357+0000 7fa75ac00700 1 --2- 172.21.15.62:0/3359004569 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa754105b10 0x7fa75407bc40 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:18:31.241 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:30.357+0000 7fa75ac00700 1 -- 172.21.15.62:0/3359004569 >> 172.21.15.62:0/3359004569 conn(0x7fa754101180 msgr2=0x7fa754101f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:18:31.241 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:30.357+0000 7fa75ac00700 1 -- 172.21.15.62:0/3359004569 shutdown_connections 2024-04-20T21:18:31.241 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:30.358+0000 7fa75ac00700 1 -- 172.21.15.62:0/3359004569 wait complete. 2024-04-20T21:18:31.241 INFO:teuthology.orchestra.run.smithi062.stdout:mgr not available, waiting (1/15)... 2024-04-20T21:18:34.283 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:33 smithi062 ceph-mon[16221]: from='client.? 172.21.15.62:0/255388222' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-20T21:18:34.578 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout 2024-04-20T21:18:34.581 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout { 2024-04-20T21:18:34.581 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "fsid": "61ae3628-ff5b-11ee-bc93-c7b262605968", 2024-04-20T21:18:34.581 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "health": { 2024-04-20T21:18:34.581 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-20T21:18:34.581 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-20T21:18:34.581 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-20T21:18:34.581 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout }, 2024-04-20T21:18:34.582 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-20T21:18:34.582 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-20T21:18:34.582 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout 0 2024-04-20T21:18:34.582 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout ], 2024-04-20T21:18:34.582 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-20T21:18:34.582 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "smithi062" 2024-04-20T21:18:34.582 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout ], 2024-04-20T21:18:34.582 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "quorum_age": 7, 2024-04-20T21:18:34.582 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-20T21:18:34.582 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-20T21:18:34.582 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-04-20T21:18:34.582 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-20T21:18:34.582 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout }, 2024-04-20T21:18:34.582 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-20T21:18:34.582 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-20T21:18:34.583 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-20T21:18:34.583 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-20T21:18:34.583 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-20T21:18:34.583 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-20T21:18:34.583 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-20T21:18:34.583 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-20T21:18:34.583 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout }, 2024-04-20T21:18:34.583 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-20T21:18:34.589 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-20T21:18:34.590 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-20T21:18:34.590 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-20T21:18:34.590 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-20T21:18:34.590 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-20T21:18:34.590 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-20T21:18:34.590 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-20T21:18:34.590 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-20T21:18:34.590 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout }, 2024-04-20T21:18:34.590 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-20T21:18:34.590 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-20T21:18:34.590 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-20T21:18:34.590 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-20T21:18:34.590 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout }, 2024-04-20T21:18:34.591 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-20T21:18:34.591 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-20T21:18:34.591 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-20T21:18:34.591 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-20T21:18:34.591 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-20T21:18:34.591 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-20T21:18:34.591 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "restful" 2024-04-20T21:18:34.591 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout ], 2024-04-20T21:18:34.591 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-20T21:18:34.591 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout }, 2024-04-20T21:18:34.591 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-20T21:18:34.591 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-20T21:18:34.591 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "modified": "2024-04-20T21:18:19.951841+0000", 2024-04-20T21:18:34.591 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-20T21:18:34.591 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout }, 2024-04-20T21:18:34.592 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-20T21:18:34.592 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout } 2024-04-20T21:18:34.592 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:33.822+0000 7f9678e00700 1 Processor -- start 2024-04-20T21:18:34.592 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:33.823+0000 7f9678e00700 1 -- start start 2024-04-20T21:18:34.592 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:33.823+0000 7f9678e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9674105b10 0x7f9674105f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:18:34.592 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:33.823+0000 7f9678e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9674106460 con 0x7f9674105b10 2024-04-20T21:18:34.592 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:33.824+0000 7f9671e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9674105b10 0x7f9674105f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:18:34.592 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:33.824+0000 7f9671e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9674105b10 0x7f9674105f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:36988/0 (socket says 172.21.15.62:36988) 2024-04-20T21:18:34.592 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:33.824+0000 7f9671e00700 1 -- 172.21.15.62:0/1795651065 learned_addr learned my addr 172.21.15.62:0/1795651065 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:18:34.592 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:33.824+0000 7f9671e00700 1 -- 172.21.15.62:0/1795651065 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96741065a0 con 0x7f9674105b10 2024-04-20T21:18:34.592 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:33.824+0000 7f9671e00700 1 --2- 172.21.15.62:0/1795651065 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9674105b10 0x7f9674105f20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f96640098a0 tx=0x7f9664009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77768c12b1f3f947 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:18:34.592 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:33.825+0000 7f9670a00700 1 -- 172.21.15.62:0/1795651065 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9664004180 con 0x7f9674105b10 2024-04-20T21:18:34.593 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:33.825+0000 7f9670a00700 1 -- 172.21.15.62:0/1795651065 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f96640042e0 con 0x7f9674105b10 2024-04-20T21:18:34.593 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:33.825+0000 7f9670a00700 1 -- 172.21.15.62:0/1795651065 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9664011a50 con 0x7f9674105b10 2024-04-20T21:18:34.593 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:33.825+0000 7f9678e00700 1 -- 172.21.15.62:0/1795651065 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9674105b10 msgr2=0x7f9674105f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:18:34.593 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:33.825+0000 7f9678e00700 1 --2- 172.21.15.62:0/1795651065 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9674105b10 0x7f9674105f20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f96640098a0 tx=0x7f9664009bb0 comp rx=0 tx=0).stop 2024-04-20T21:18:34.593 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:33.826+0000 7f9678e00700 1 -- 172.21.15.62:0/1795651065 shutdown_connections 2024-04-20T21:18:34.593 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:33.826+0000 7f9678e00700 1 --2- 172.21.15.62:0/1795651065 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9674105b10 0x7f9674105f20 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:18:34.593 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:33.826+0000 7f9678e00700 1 -- 172.21.15.62:0/1795651065 >> 172.21.15.62:0/1795651065 conn(0x7f9674101160 msgr2=0x7f9674103590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:18:34.593 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:33.826+0000 7f9678e00700 1 -- 172.21.15.62:0/1795651065 shutdown_connections 2024-04-20T21:18:34.593 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:33.826+0000 7f9678e00700 1 -- 172.21.15.62:0/1795651065 wait complete. 2024-04-20T21:18:34.593 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:33.827+0000 7f9678e00700 1 Processor -- start 2024-04-20T21:18:34.593 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:33.827+0000 7f9678e00700 1 -- start start 2024-04-20T21:18:34.594 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:33.827+0000 7f9678e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9674105b10 0x7f967419ba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:18:34.594 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:33.827+0000 7f9678e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f967419bf70 con 0x7f9674105b10 2024-04-20T21:18:34.594 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:33.828+0000 7f9671e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9674105b10 0x7f967419ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:18:34.594 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:33.828+0000 7f9671e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9674105b10 0x7f967419ba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:36994/0 (socket says 172.21.15.62:36994) 2024-04-20T21:18:34.594 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:33.828+0000 7f9671e00700 1 -- 172.21.15.62:0/255388222 learned_addr learned my addr 172.21.15.62:0/255388222 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:18:34.594 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:33.828+0000 7f9671e00700 1 -- 172.21.15.62:0/255388222 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9664009550 con 0x7f9674105b10 2024-04-20T21:18:34.594 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:33.828+0000 7f9671e00700 1 --2- 172.21.15.62:0/255388222 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9674105b10 0x7f967419ba30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f96640119e0 tx=0x7f9664011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:18:34.594 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:33.829+0000 7f966aa00700 1 -- 172.21.15.62:0/255388222 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9664003b90 con 0x7f9674105b10 2024-04-20T21:18:34.594 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:33.829+0000 7f966aa00700 1 -- 172.21.15.62:0/255388222 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f966401a8a0 con 0x7f9674105b10 2024-04-20T21:18:34.594 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:33.829+0000 7f966aa00700 1 -- 172.21.15.62:0/255388222 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9664019930 con 0x7f9674105b10 2024-04-20T21:18:34.594 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:33.829+0000 7f9678e00700 1 -- 172.21.15.62:0/255388222 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f967419c170 con 0x7f9674105b10 2024-04-20T21:18:34.594 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:33.829+0000 7f9678e00700 1 -- 172.21.15.62:0/255388222 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f967419c590 con 0x7f9674105b10 2024-04-20T21:18:34.595 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:33.829+0000 7f966aa00700 1 -- 172.21.15.62:0/255388222 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f966402d440 con 0x7f9674105b10 2024-04-20T21:18:34.595 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:33.829+0000 7f966aa00700 1 -- 172.21.15.62:0/255388222 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f9664024850 con 0x7f9674105b10 2024-04-20T21:18:34.595 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:33.830+0000 7f9678e00700 1 -- 172.21.15.62:0/255388222 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9674062380 con 0x7f9674105b10 2024-04-20T21:18:34.595 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:33.833+0000 7f966aa00700 1 -- 172.21.15.62:0/255388222 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7f966401aa10 con 0x7f9674105b10 2024-04-20T21:18:34.595 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:33.912+0000 7f9678e00700 1 -- 172.21.15.62:0/255388222 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f9674107980 con 0x7f9674105b10 2024-04-20T21:18:34.595 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:33.913+0000 7f966aa00700 1 -- 172.21.15.62:0/255388222 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1246 (secure 0 0 0) 0x7f966401abb0 con 0x7f9674105b10 2024-04-20T21:18:34.595 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:33.915+0000 7f9678e00700 1 -- 172.21.15.62:0/255388222 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9674105b10 msgr2=0x7f967419ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:18:34.595 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:33.915+0000 7f9678e00700 1 --2- 172.21.15.62:0/255388222 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9674105b10 0x7f967419ba30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f96640119e0 tx=0x7f9664011ac0 comp rx=0 tx=0).stop 2024-04-20T21:18:34.595 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:33.915+0000 7f9678e00700 1 -- 172.21.15.62:0/255388222 shutdown_connections 2024-04-20T21:18:34.595 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:33.915+0000 7f9678e00700 1 --2- 172.21.15.62:0/255388222 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9674105b10 0x7f967419ba30 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:18:34.595 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:33.915+0000 7f9678e00700 1 -- 172.21.15.62:0/255388222 >> 172.21.15.62:0/255388222 conn(0x7f9674101160 msgr2=0x7f9674101f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:18:34.595 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:33.915+0000 7f9678e00700 1 -- 172.21.15.62:0/255388222 shutdown_connections 2024-04-20T21:18:34.595 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:33.915+0000 7f9678e00700 1 -- 172.21.15.62:0/255388222 wait complete. 2024-04-20T21:18:34.595 INFO:teuthology.orchestra.run.smithi062.stdout:mgr not available, waiting (2/15)... 2024-04-20T21:18:37.951 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:37 smithi062 ceph-mon[16221]: from='client.? 172.21.15.62:0/28111666' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-20T21:18:37.957 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout 2024-04-20T21:18:37.958 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout { 2024-04-20T21:18:37.958 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "fsid": "61ae3628-ff5b-11ee-bc93-c7b262605968", 2024-04-20T21:18:37.958 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "health": { 2024-04-20T21:18:37.958 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-20T21:18:37.958 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-20T21:18:37.958 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-20T21:18:37.958 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout }, 2024-04-20T21:18:37.958 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-20T21:18:37.958 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-20T21:18:37.960 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout 0 2024-04-20T21:18:37.960 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout ], 2024-04-20T21:18:37.960 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-20T21:18:37.960 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "smithi062" 2024-04-20T21:18:37.960 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout ], 2024-04-20T21:18:37.960 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "quorum_age": 10, 2024-04-20T21:18:37.960 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-20T21:18:37.960 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-20T21:18:37.960 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-04-20T21:18:37.960 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-20T21:18:37.960 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout }, 2024-04-20T21:18:37.960 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-20T21:18:37.960 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-20T21:18:37.961 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-20T21:18:37.961 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-20T21:18:37.961 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-20T21:18:37.968 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-20T21:18:37.968 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-20T21:18:37.968 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-20T21:18:37.968 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout }, 2024-04-20T21:18:37.968 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-20T21:18:37.968 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-20T21:18:37.968 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-20T21:18:37.968 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-20T21:18:37.968 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-20T21:18:37.968 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-20T21:18:37.968 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-20T21:18:37.969 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-20T21:18:37.969 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-20T21:18:37.969 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout }, 2024-04-20T21:18:37.969 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-20T21:18:37.969 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-20T21:18:37.969 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-20T21:18:37.969 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-20T21:18:37.969 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout }, 2024-04-20T21:18:37.969 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-20T21:18:37.969 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-20T21:18:37.969 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-20T21:18:37.969 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-20T21:18:37.969 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-20T21:18:37.969 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-20T21:18:37.969 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "restful" 2024-04-20T21:18:37.970 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout ], 2024-04-20T21:18:37.970 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-20T21:18:37.970 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout }, 2024-04-20T21:18:37.970 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-20T21:18:37.970 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-20T21:18:37.970 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "modified": "2024-04-20T21:18:19.951841+0000", 2024-04-20T21:18:37.970 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-20T21:18:37.970 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout }, 2024-04-20T21:18:37.970 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-20T21:18:37.970 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout } 2024-04-20T21:18:37.970 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:37.181+0000 7f8742200700 1 Processor -- start 2024-04-20T21:18:37.970 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:37.181+0000 7f8742200700 1 -- start start 2024-04-20T21:18:37.970 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:37.181+0000 7f8742200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f873c105b10 0x7f873c105f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:18:37.970 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:37.181+0000 7f8742200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f873c106460 con 0x7f873c105b10 2024-04-20T21:18:37.970 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:37.182+0000 7f873b400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f873c105b10 0x7f873c105f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:18:37.971 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:37.182+0000 7f873b400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f873c105b10 0x7f873c105f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37000/0 (socket says 172.21.15.62:37000) 2024-04-20T21:18:37.971 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:37.182+0000 7f873b400700 1 -- 172.21.15.62:0/1748721921 learned_addr learned my addr 172.21.15.62:0/1748721921 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:18:37.971 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:37.182+0000 7f873b400700 1 -- 172.21.15.62:0/1748721921 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f873c1065a0 con 0x7f873c105b10 2024-04-20T21:18:37.971 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:37.183+0000 7f873b400700 1 --2- 172.21.15.62:0/1748721921 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f873c105b10 0x7f873c105f20 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f872c0098a0 tx=0x7f872c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=595e98f93da146e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:18:37.971 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:37.183+0000 7f873a000700 1 -- 172.21.15.62:0/1748721921 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f872c004180 con 0x7f873c105b10 2024-04-20T21:18:37.971 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:37.183+0000 7f873a000700 1 -- 172.21.15.62:0/1748721921 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f872c0042e0 con 0x7f873c105b10 2024-04-20T21:18:37.971 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:37.183+0000 7f873a000700 1 -- 172.21.15.62:0/1748721921 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f872c011a50 con 0x7f873c105b10 2024-04-20T21:18:37.971 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:37.183+0000 7f8742200700 1 -- 172.21.15.62:0/1748721921 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f873c105b10 msgr2=0x7f873c105f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:18:37.971 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:37.183+0000 7f8742200700 1 --2- 172.21.15.62:0/1748721921 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f873c105b10 0x7f873c105f20 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f872c0098a0 tx=0x7f872c009bb0 comp rx=0 tx=0).stop 2024-04-20T21:18:37.971 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:37.184+0000 7f8742200700 1 -- 172.21.15.62:0/1748721921 shutdown_connections 2024-04-20T21:18:37.971 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:37.184+0000 7f8742200700 1 --2- 172.21.15.62:0/1748721921 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f873c105b10 0x7f873c105f20 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:18:37.971 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:37.184+0000 7f8742200700 1 -- 172.21.15.62:0/1748721921 >> 172.21.15.62:0/1748721921 conn(0x7f873c101160 msgr2=0x7f873c103590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:18:37.971 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:37.184+0000 7f8742200700 1 -- 172.21.15.62:0/1748721921 shutdown_connections 2024-04-20T21:18:37.971 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:37.184+0000 7f8742200700 1 -- 172.21.15.62:0/1748721921 wait complete. 2024-04-20T21:18:37.971 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:37.185+0000 7f8742200700 1 Processor -- start 2024-04-20T21:18:37.972 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:37.185+0000 7f8742200700 1 -- start start 2024-04-20T21:18:37.972 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:37.185+0000 7f8742200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f873c105b10 0x7f873c07ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:18:37.972 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:37.185+0000 7f8742200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f873c07b070 con 0x7f873c105b10 2024-04-20T21:18:37.972 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:37.186+0000 7f873b400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f873c105b10 0x7f873c07ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:18:37.972 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:37.186+0000 7f873b400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f873c105b10 0x7f873c07ab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37008/0 (socket says 172.21.15.62:37008) 2024-04-20T21:18:37.972 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:37.186+0000 7f873b400700 1 -- 172.21.15.62:0/28111666 learned_addr learned my addr 172.21.15.62:0/28111666 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:18:37.972 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:37.186+0000 7f873b400700 1 -- 172.21.15.62:0/28111666 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f872c009550 con 0x7f873c105b10 2024-04-20T21:18:37.972 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:37.186+0000 7f873b400700 1 --2- 172.21.15.62:0/28111666 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f873c105b10 0x7f873c07ab30 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f872c000c00 tx=0x7f872c011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:18:37.972 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:37.187+0000 7f8733e00700 1 -- 172.21.15.62:0/28111666 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f872c003b90 con 0x7f873c105b10 2024-04-20T21:18:37.972 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:37.187+0000 7f8733e00700 1 -- 172.21.15.62:0/28111666 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f872c0248b0 con 0x7f873c105b10 2024-04-20T21:18:37.972 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:37.187+0000 7f8742200700 1 -- 172.21.15.62:0/28111666 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f873c079180 con 0x7f873c105b10 2024-04-20T21:18:37.972 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:37.187+0000 7f8733e00700 1 -- 172.21.15.62:0/28111666 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f872c01a8e0 con 0x7f873c105b10 2024-04-20T21:18:37.972 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:37.187+0000 7f8742200700 1 -- 172.21.15.62:0/28111666 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f873c0795a0 con 0x7f873c105b10 2024-04-20T21:18:37.972 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:37.187+0000 7f8733e00700 1 -- 172.21.15.62:0/28111666 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f872c02d3f0 con 0x7f873c105b10 2024-04-20T21:18:37.972 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:37.187+0000 7f8733e00700 1 -- 172.21.15.62:0/28111666 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f872c019450 con 0x7f873c105b10 2024-04-20T21:18:37.972 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:37.188+0000 7f8742200700 1 -- 172.21.15.62:0/28111666 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f873c062380 con 0x7f873c105b10 2024-04-20T21:18:37.973 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:37.191+0000 7f8733e00700 1 -- 172.21.15.62:0/28111666 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7f872c024db0 con 0x7f873c105b10 2024-04-20T21:18:37.973 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:37.271+0000 7f8742200700 1 -- 172.21.15.62:0/28111666 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f873c079c00 con 0x7f873c105b10 2024-04-20T21:18:37.973 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:37.271+0000 7f8733e00700 1 -- 172.21.15.62:0/28111666 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1247 (secure 0 0 0) 0x7f873c079c00 con 0x7f873c105b10 2024-04-20T21:18:37.973 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:37.273+0000 7f8742200700 1 -- 172.21.15.62:0/28111666 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f873c105b10 msgr2=0x7f873c07ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:18:37.973 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:37.273+0000 7f8742200700 1 --2- 172.21.15.62:0/28111666 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f873c105b10 0x7f873c07ab30 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f872c000c00 tx=0x7f872c011ab0 comp rx=0 tx=0).stop 2024-04-20T21:18:37.973 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:37.273+0000 7f8742200700 1 -- 172.21.15.62:0/28111666 shutdown_connections 2024-04-20T21:18:37.973 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:37.273+0000 7f8742200700 1 --2- 172.21.15.62:0/28111666 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f873c105b10 0x7f873c07ab30 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:18:37.973 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:37.273+0000 7f8742200700 1 -- 172.21.15.62:0/28111666 >> 172.21.15.62:0/28111666 conn(0x7f873c101160 msgr2=0x7f873c101f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:18:37.973 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:37.273+0000 7f8742200700 1 -- 172.21.15.62:0/28111666 shutdown_connections 2024-04-20T21:18:37.973 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:37.273+0000 7f8742200700 1 -- 172.21.15.62:0/28111666 wait complete. 2024-04-20T21:18:37.973 INFO:teuthology.orchestra.run.smithi062.stdout:mgr not available, waiting (3/15)... 2024-04-20T21:18:38.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:38 smithi062 ceph-mon[16221]: Activating manager daemon smithi062.rqxuxi 2024-04-20T21:18:38.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:38 smithi062 ceph-mon[16221]: mgrmap e2: smithi062.rqxuxi(active, starting, since 0.0124173s) 2024-04-20T21:18:38.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:38 smithi062 ceph-mon[16221]: from='mgr.14100 172.21.15.62:0/2359350732' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-20T21:18:38.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:38 smithi062 ceph-mon[16221]: from='mgr.14100 172.21.15.62:0/2359350732' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-20T21:18:38.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:38 smithi062 ceph-mon[16221]: from='mgr.14100 172.21.15.62:0/2359350732' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-20T21:18:38.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:38 smithi062 ceph-mon[16221]: from='mgr.14100 172.21.15.62:0/2359350732' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mon metadata", "id": "smithi062"}]: dispatch 2024-04-20T21:18:38.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:38 smithi062 ceph-mon[16221]: from='mgr.14100 172.21.15.62:0/2359350732' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mgr metadata", "who": "smithi062.rqxuxi", "id": "smithi062.rqxuxi"}]: dispatch 2024-04-20T21:18:38.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:38 smithi062 ceph-mon[16221]: Manager daemon smithi062.rqxuxi is now available 2024-04-20T21:18:38.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:38 smithi062 ceph-mon[16221]: from='mgr.14100 172.21.15.62:0/2359350732' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi062.rqxuxi/mirror_snapshot_schedule"}]: dispatch 2024-04-20T21:18:38.787 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:38 smithi062 ceph-mon[16221]: from='mgr.14100 172.21.15.62:0/2359350732' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:18:38.787 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:38 smithi062 ceph-mon[16221]: from='mgr.14100 172.21.15.62:0/2359350732' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi062.rqxuxi/trash_purge_schedule"}]: dispatch 2024-04-20T21:18:38.787 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:38 smithi062 ceph-mon[16221]: from='mgr.14100 172.21.15.62:0/2359350732' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:18:38.787 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:38 smithi062 ceph-mon[16221]: from='mgr.14100 172.21.15.62:0/2359350732' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:18:40.448 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:40 smithi062 ceph-mon[16221]: mgrmap e3: smithi062.rqxuxi(active, since 1.02227s) 2024-04-20T21:18:41.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:41 smithi062 ceph-mon[16221]: mgrmap e4: smithi062.rqxuxi(active, since 2s) 2024-04-20T21:18:41.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:41 smithi062 ceph-mon[16221]: from='client.? 172.21.15.62:0/2265287396' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-20T21:18:41.607 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout 2024-04-20T21:18:41.607 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout { 2024-04-20T21:18:41.607 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "fsid": "61ae3628-ff5b-11ee-bc93-c7b262605968", 2024-04-20T21:18:41.607 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "health": { 2024-04-20T21:18:41.607 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-20T21:18:41.607 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-20T21:18:41.607 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-20T21:18:41.607 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout }, 2024-04-20T21:18:41.607 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-20T21:18:41.607 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-20T21:18:41.607 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout 0 2024-04-20T21:18:41.607 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout ], 2024-04-20T21:18:41.607 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-20T21:18:41.608 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "smithi062" 2024-04-20T21:18:41.608 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout ], 2024-04-20T21:18:41.608 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "quorum_age": 14, 2024-04-20T21:18:41.608 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-20T21:18:41.608 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-20T21:18:41.608 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-04-20T21:18:41.608 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-20T21:18:41.608 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout }, 2024-04-20T21:18:41.608 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-20T21:18:41.608 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-20T21:18:41.608 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-20T21:18:41.608 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-20T21:18:41.608 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-20T21:18:41.616 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-20T21:18:41.616 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-20T21:18:41.616 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-20T21:18:41.616 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout }, 2024-04-20T21:18:41.616 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-20T21:18:41.616 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-20T21:18:41.616 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-20T21:18:41.616 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-20T21:18:41.616 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-20T21:18:41.616 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-20T21:18:41.616 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-20T21:18:41.616 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-20T21:18:41.617 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-20T21:18:41.617 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout }, 2024-04-20T21:18:41.617 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-20T21:18:41.617 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-20T21:18:41.617 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-20T21:18:41.617 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-20T21:18:41.617 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout }, 2024-04-20T21:18:41.617 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-20T21:18:41.617 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-20T21:18:41.617 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-20T21:18:41.617 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-20T21:18:41.617 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-20T21:18:41.617 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-20T21:18:41.617 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "restful" 2024-04-20T21:18:41.618 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout ], 2024-04-20T21:18:41.618 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-20T21:18:41.618 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout }, 2024-04-20T21:18:41.618 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-20T21:18:41.618 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-20T21:18:41.618 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "modified": "2024-04-20T21:18:19.951841+0000", 2024-04-20T21:18:41.618 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-20T21:18:41.618 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout }, 2024-04-20T21:18:41.618 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-20T21:18:41.618 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout } 2024-04-20T21:18:41.618 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:40.614+0000 7f9b45200700 1 Processor -- start 2024-04-20T21:18:41.618 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:40.614+0000 7f9b45200700 1 -- start start 2024-04-20T21:18:41.618 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:40.615+0000 7f9b45200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9b40107d00 0x7f9b40108110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:18:41.618 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:40.615+0000 7f9b45200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b40108650 con 0x7f9b40107d00 2024-04-20T21:18:41.618 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:40.615+0000 7f9b3de00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9b40107d00 0x7f9b40108110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:18:41.618 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:40.615+0000 7f9b3de00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9b40107d00 0x7f9b40108110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37064/0 (socket says 172.21.15.62:37064) 2024-04-20T21:18:41.619 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:40.615+0000 7f9b3de00700 1 -- 172.21.15.62:0/3554027609 learned_addr learned my addr 172.21.15.62:0/3554027609 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:18:41.619 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:40.616+0000 7f9b3de00700 1 -- 172.21.15.62:0/3554027609 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b40108790 con 0x7f9b40107d00 2024-04-20T21:18:41.619 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:40.616+0000 7f9b3de00700 1 --2- 172.21.15.62:0/3554027609 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9b40107d00 0x7f9b40108110 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f9b300098a0 tx=0x7f9b30009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=685c39725c0fa43e server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:18:41.619 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:40.616+0000 7f9b3ca00700 1 -- 172.21.15.62:0/3554027609 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b30004180 con 0x7f9b40107d00 2024-04-20T21:18:41.619 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:40.617+0000 7f9b3ca00700 1 -- 172.21.15.62:0/3554027609 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f9b300042e0 con 0x7f9b40107d00 2024-04-20T21:18:41.619 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:40.617+0000 7f9b3ca00700 1 -- 172.21.15.62:0/3554027609 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b30011a50 con 0x7f9b40107d00 2024-04-20T21:18:41.619 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:40.617+0000 7f9b45200700 1 -- 172.21.15.62:0/3554027609 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9b40107d00 msgr2=0x7f9b40108110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:18:41.619 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:40.617+0000 7f9b45200700 1 --2- 172.21.15.62:0/3554027609 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9b40107d00 0x7f9b40108110 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f9b300098a0 tx=0x7f9b30009bb0 comp rx=0 tx=0).stop 2024-04-20T21:18:41.619 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:40.617+0000 7f9b45200700 1 -- 172.21.15.62:0/3554027609 shutdown_connections 2024-04-20T21:18:41.619 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:40.617+0000 7f9b45200700 1 --2- 172.21.15.62:0/3554027609 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9b40107d00 0x7f9b40108110 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:18:41.619 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:40.617+0000 7f9b45200700 1 -- 172.21.15.62:0/3554027609 >> 172.21.15.62:0/3554027609 conn(0x7f9b4007b220 msgr2=0x7f9b4007b640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:18:41.619 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:40.618+0000 7f9b45200700 1 -- 172.21.15.62:0/3554027609 shutdown_connections 2024-04-20T21:18:41.619 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:40.618+0000 7f9b45200700 1 -- 172.21.15.62:0/3554027609 wait complete. 2024-04-20T21:18:41.619 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:40.619+0000 7f9b45200700 1 Processor -- start 2024-04-20T21:18:41.619 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:40.619+0000 7f9b45200700 1 -- start start 2024-04-20T21:18:41.619 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:40.619+0000 7f9b45200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9b40107d00 0x7f9b4019ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:18:41.620 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:40.619+0000 7f9b45200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b4019bfb0 con 0x7f9b40107d00 2024-04-20T21:18:41.620 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:40.619+0000 7f9b3de00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9b40107d00 0x7f9b4019ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:18:41.620 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:40.619+0000 7f9b3de00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9b40107d00 0x7f9b4019ba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37066/0 (socket says 172.21.15.62:37066) 2024-04-20T21:18:41.620 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:40.619+0000 7f9b3de00700 1 -- 172.21.15.62:0/2265287396 learned_addr learned my addr 172.21.15.62:0/2265287396 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:18:41.620 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:40.620+0000 7f9b3de00700 1 -- 172.21.15.62:0/2265287396 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b30009550 con 0x7f9b40107d00 2024-04-20T21:18:41.620 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:40.620+0000 7f9b3de00700 1 --2- 172.21.15.62:0/2265287396 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9b40107d00 0x7f9b4019ba70 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f9b300119e0 tx=0x7f9b30011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:18:41.620 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:40.620+0000 7f9b36a00700 1 -- 172.21.15.62:0/2265287396 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b30003b90 con 0x7f9b40107d00 2024-04-20T21:18:41.620 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:40.620+0000 7f9b36a00700 1 -- 172.21.15.62:0/2265287396 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f9b3001a8a0 con 0x7f9b40107d00 2024-04-20T21:18:41.620 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:40.620+0000 7f9b36a00700 1 -- 172.21.15.62:0/2265287396 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b30019930 con 0x7f9b40107d00 2024-04-20T21:18:41.620 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:40.621+0000 7f9b45200700 1 -- 172.21.15.62:0/2265287396 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b4019c1b0 con 0x7f9b40107d00 2024-04-20T21:18:41.620 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:40.621+0000 7f9b45200700 1 -- 172.21.15.62:0/2265287396 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b4019c5d0 con 0x7f9b40107d00 2024-04-20T21:18:41.620 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:40.622+0000 7f9b36a00700 1 -- 172.21.15.62:0/2265287396 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 4) v1 ==== 45073+0+0 (secure 0 0 0) 0x7f9b30019ba0 con 0x7f9b40107d00 2024-04-20T21:18:41.620 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:40.622+0000 7f9b45200700 1 -- 172.21.15.62:0/2265287396 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b40062380 con 0x7f9b40107d00 2024-04-20T21:18:41.620 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:40.622+0000 7f9b36a00700 1 --2- 172.21.15.62:0/2265287396 >> [v2:172.21.15.62:6800/871423274,v1:172.21.15.62:6801/871423274] conn(0x7f9b240383b0 0x7f9b2403a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:18:41.620 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:40.622+0000 7f9b36a00700 1 -- 172.21.15.62:0/2265287396 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f9b3004c760 con 0x7f9b40107d00 2024-04-20T21:18:41.620 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:40.622+0000 7f9b3d400700 1 --2- 172.21.15.62:0/2265287396 >> [v2:172.21.15.62:6800/871423274,v1:172.21.15.62:6801/871423274] conn(0x7f9b240383b0 0x7f9b2403a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:18:41.621 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:40.623+0000 7f9b3d400700 1 --2- 172.21.15.62:0/2265287396 >> [v2:172.21.15.62:6800/871423274,v1:172.21.15.62:6801/871423274] conn(0x7f9b240383b0 0x7f9b2403a860 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f9b2c006fd0 tx=0x7f9b2c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:18:41.621 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:40.629+0000 7f9b36a00700 1 -- 172.21.15.62:0/2265287396 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f9b300189a0 con 0x7f9b40107d00 2024-04-20T21:18:41.621 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:40.935+0000 7f9b45200700 1 -- 172.21.15.62:0/2265287396 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f9b4019f770 con 0x7f9b40107d00 2024-04-20T21:18:41.621 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:40.935+0000 7f9b36a00700 1 -- 172.21.15.62:0/2265287396 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1246 (secure 0 0 0) 0x7f9b30018350 con 0x7f9b40107d00 2024-04-20T21:18:41.621 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:40.939+0000 7f9b45200700 1 -- 172.21.15.62:0/2265287396 >> [v2:172.21.15.62:6800/871423274,v1:172.21.15.62:6801/871423274] conn(0x7f9b240383b0 msgr2=0x7f9b2403a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:18:41.621 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:40.939+0000 7f9b45200700 1 --2- 172.21.15.62:0/2265287396 >> [v2:172.21.15.62:6800/871423274,v1:172.21.15.62:6801/871423274] conn(0x7f9b240383b0 0x7f9b2403a860 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f9b2c006fd0 tx=0x7f9b2c006e40 comp rx=0 tx=0).stop 2024-04-20T21:18:41.621 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:40.939+0000 7f9b45200700 1 -- 172.21.15.62:0/2265287396 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9b40107d00 msgr2=0x7f9b4019ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:18:41.621 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:40.939+0000 7f9b45200700 1 --2- 172.21.15.62:0/2265287396 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9b40107d00 0x7f9b4019ba70 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f9b300119e0 tx=0x7f9b30011ac0 comp rx=0 tx=0).stop 2024-04-20T21:18:41.621 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:40.939+0000 7f9b45200700 1 -- 172.21.15.62:0/2265287396 shutdown_connections 2024-04-20T21:18:41.621 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:40.939+0000 7f9b45200700 1 --2- 172.21.15.62:0/2265287396 >> [v2:172.21.15.62:6800/871423274,v1:172.21.15.62:6801/871423274] conn(0x7f9b240383b0 0x7f9b2403a860 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:18:41.621 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:40.939+0000 7f9b45200700 1 --2- 172.21.15.62:0/2265287396 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9b40107d00 0x7f9b4019ba70 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:18:41.621 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:40.939+0000 7f9b45200700 1 -- 172.21.15.62:0/2265287396 >> 172.21.15.62:0/2265287396 conn(0x7f9b4007b220 msgr2=0x7f9b40105ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:18:41.621 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:40.940+0000 7f9b45200700 1 -- 172.21.15.62:0/2265287396 shutdown_connections 2024-04-20T21:18:41.621 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:40.940+0000 7f9b45200700 1 -- 172.21.15.62:0/2265287396 wait complete. 2024-04-20T21:18:41.621 INFO:teuthology.orchestra.run.smithi062.stdout:mgr is available 2024-04-20T21:18:42.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:42 smithi062 ceph-mon[16221]: from='client.? 172.21.15.62:0/1555113775' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-20T21:18:43.078 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout 2024-04-20T21:18:43.078 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout [global] 2024-04-20T21:18:43.078 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout fsid = 61ae3628-ff5b-11ee-bc93-c7b262605968 2024-04-20T21:18:43.078 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-20T21:18:43.078 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-20T21:18:43.078 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-20T21:18:43.078 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-20T21:18:43.078 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout 2024-04-20T21:18:43.078 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-20T21:18:43.079 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-20T21:18:43.079 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout 2024-04-20T21:18:43.079 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout [osd] 2024-04-20T21:18:43.079 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-20T21:18:43.079 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-20T21:18:43.079 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-20T21:18:43.084 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-20T21:18:43.084 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-20T21:18:43.085 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:42.215+0000 7f1174e00700 1 Processor -- start 2024-04-20T21:18:43.085 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:42.215+0000 7f1174e00700 1 -- start start 2024-04-20T21:18:43.085 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:42.216+0000 7f1174e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f117007be30 0x7f117007a330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:18:43.085 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:42.216+0000 7f1174e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f117007a870 con 0x7f117007be30 2024-04-20T21:18:43.085 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:42.216+0000 7f116de00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f117007be30 0x7f117007a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:18:43.085 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:42.216+0000 7f116de00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f117007be30 0x7f117007a330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48044/0 (socket says 172.21.15.62:48044) 2024-04-20T21:18:43.085 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:42.216+0000 7f116de00700 1 -- 172.21.15.62:0/1444340885 learned_addr learned my addr 172.21.15.62:0/1444340885 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:18:43.085 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:42.217+0000 7f116de00700 1 -- 172.21.15.62:0/1444340885 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f117007a9b0 con 0x7f117007be30 2024-04-20T21:18:43.085 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:42.217+0000 7f116de00700 1 --2- 172.21.15.62:0/1444340885 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f117007be30 0x7f117007a330 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f11600098a0 tx=0x7f1160009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c5a371f7c49be66 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:18:43.085 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:42.217+0000 7f116d400700 1 -- 172.21.15.62:0/1444340885 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1160004180 con 0x7f117007be30 2024-04-20T21:18:43.085 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:42.218+0000 7f116d400700 1 -- 172.21.15.62:0/1444340885 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f11600042e0 con 0x7f117007be30 2024-04-20T21:18:43.085 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:42.218+0000 7f116d400700 1 -- 172.21.15.62:0/1444340885 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1160011a50 con 0x7f117007be30 2024-04-20T21:18:43.085 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:42.218+0000 7f1174e00700 1 -- 172.21.15.62:0/1444340885 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f117007be30 msgr2=0x7f117007a330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:18:43.086 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:42.218+0000 7f1174e00700 1 --2- 172.21.15.62:0/1444340885 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f117007be30 0x7f117007a330 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f11600098a0 tx=0x7f1160009bb0 comp rx=0 tx=0).stop 2024-04-20T21:18:43.086 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:42.218+0000 7f1174e00700 1 -- 172.21.15.62:0/1444340885 shutdown_connections 2024-04-20T21:18:43.086 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:42.218+0000 7f1174e00700 1 --2- 172.21.15.62:0/1444340885 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f117007be30 0x7f117007a330 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:18:43.086 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:42.218+0000 7f1174e00700 1 -- 172.21.15.62:0/1444340885 >> 172.21.15.62:0/1444340885 conn(0x7f1170101180 msgr2=0x7f1170103590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:18:43.086 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:42.218+0000 7f1174e00700 1 -- 172.21.15.62:0/1444340885 shutdown_connections 2024-04-20T21:18:43.086 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:42.218+0000 7f1174e00700 1 -- 172.21.15.62:0/1444340885 wait complete. 2024-04-20T21:18:43.086 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:42.219+0000 7f1174e00700 1 Processor -- start 2024-04-20T21:18:43.086 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:42.220+0000 7f1174e00700 1 -- start start 2024-04-20T21:18:43.086 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:42.220+0000 7f1174e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f117007be30 0x7f117019fe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:18:43.086 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:42.220+0000 7f1174e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11701a03c0 con 0x7f117007be30 2024-04-20T21:18:43.086 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:42.220+0000 7f116de00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f117007be30 0x7f117019fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:18:43.086 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:42.220+0000 7f116de00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f117007be30 0x7f117019fe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48046/0 (socket says 172.21.15.62:48046) 2024-04-20T21:18:43.086 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:42.220+0000 7f116de00700 1 -- 172.21.15.62:0/1555113775 learned_addr learned my addr 172.21.15.62:0/1555113775 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:18:43.086 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:42.221+0000 7f116de00700 1 -- 172.21.15.62:0/1555113775 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1160009550 con 0x7f117007be30 2024-04-20T21:18:43.087 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:42.221+0000 7f116de00700 1 --2- 172.21.15.62:0/1555113775 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f117007be30 0x7f117019fe80 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f1160000c00 tx=0x7f1160011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:18:43.087 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:42.221+0000 7f1166a00700 1 -- 172.21.15.62:0/1555113775 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1160003b90 con 0x7f117007be30 2024-04-20T21:18:43.087 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:42.221+0000 7f1166a00700 1 -- 172.21.15.62:0/1555113775 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f11600248b0 con 0x7f117007be30 2024-04-20T21:18:43.087 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:42.221+0000 7f1166a00700 1 -- 172.21.15.62:0/1555113775 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f116001a8e0 con 0x7f117007be30 2024-04-20T21:18:43.087 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:42.221+0000 7f1174e00700 1 -- 172.21.15.62:0/1555113775 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11701a05c0 con 0x7f117007be30 2024-04-20T21:18:43.087 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:42.222+0000 7f1174e00700 1 -- 172.21.15.62:0/1555113775 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11701a09e0 con 0x7f117007be30 2024-04-20T21:18:43.087 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:42.223+0000 7f1166a00700 1 -- 172.21.15.62:0/1555113775 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 4) v1 ==== 45073+0+0 (secure 0 0 0) 0x7f1160024a20 con 0x7f117007be30 2024-04-20T21:18:43.087 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:42.223+0000 7f1174e00700 1 -- 172.21.15.62:0/1555113775 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1170062380 con 0x7f117007be30 2024-04-20T21:18:43.087 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:42.223+0000 7f1166a00700 1 --2- 172.21.15.62:0/1555113775 >> [v2:172.21.15.62:6800/871423274,v1:172.21.15.62:6801/871423274] conn(0x7f11540383f0 0x7f115403a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:18:43.087 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:42.223+0000 7f1166a00700 1 -- 172.21.15.62:0/1555113775 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f116004bfe0 con 0x7f117007be30 2024-04-20T21:18:43.087 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:42.223+0000 7f1167e00700 1 --2- 172.21.15.62:0/1555113775 >> [v2:172.21.15.62:6800/871423274,v1:172.21.15.62:6801/871423274] conn(0x7f11540383f0 0x7f115403a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:18:43.087 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:42.224+0000 7f1167e00700 1 --2- 172.21.15.62:0/1555113775 >> [v2:172.21.15.62:6800/871423274,v1:172.21.15.62:6801/871423274] conn(0x7f11540383f0 0x7f115403a8a0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1158006fd0 tx=0x7f1158006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:18:43.087 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:42.229+0000 7f1166a00700 1 -- 172.21.15.62:0/1555113775 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f1160018b80 con 0x7f117007be30 2024-04-20T21:18:43.087 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:42.449+0000 7f1174e00700 1 -- 172.21.15.62:0/1555113775 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f11701a38d0 con 0x7f117007be30 2024-04-20T21:18:43.088 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:42.451+0000 7f1166a00700 1 -- 172.21.15.62:0/1555113775 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7f1160018350 con 0x7f117007be30 2024-04-20T21:18:43.088 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:42.454+0000 7f1174e00700 1 -- 172.21.15.62:0/1555113775 >> [v2:172.21.15.62:6800/871423274,v1:172.21.15.62:6801/871423274] conn(0x7f11540383f0 msgr2=0x7f115403a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:18:43.088 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:42.454+0000 7f1174e00700 1 --2- 172.21.15.62:0/1555113775 >> [v2:172.21.15.62:6800/871423274,v1:172.21.15.62:6801/871423274] conn(0x7f11540383f0 0x7f115403a8a0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1158006fd0 tx=0x7f1158006e40 comp rx=0 tx=0).stop 2024-04-20T21:18:43.088 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:42.454+0000 7f1174e00700 1 -- 172.21.15.62:0/1555113775 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f117007be30 msgr2=0x7f117019fe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:18:43.088 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:42.454+0000 7f1174e00700 1 --2- 172.21.15.62:0/1555113775 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f117007be30 0x7f117019fe80 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f1160000c00 tx=0x7f1160011ab0 comp rx=0 tx=0).stop 2024-04-20T21:18:43.088 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:42.454+0000 7f1174e00700 1 -- 172.21.15.62:0/1555113775 shutdown_connections 2024-04-20T21:18:43.088 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:42.454+0000 7f1174e00700 1 --2- 172.21.15.62:0/1555113775 >> [v2:172.21.15.62:6800/871423274,v1:172.21.15.62:6801/871423274] conn(0x7f11540383f0 0x7f115403a8a0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:18:43.088 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:42.454+0000 7f1174e00700 1 --2- 172.21.15.62:0/1555113775 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f117007be30 0x7f117019fe80 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:18:43.088 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:42.454+0000 7f1174e00700 1 -- 172.21.15.62:0/1555113775 >> 172.21.15.62:0/1555113775 conn(0x7f1170101180 msgr2=0x7f1170106d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:18:43.088 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:42.454+0000 7f1174e00700 1 -- 172.21.15.62:0/1555113775 shutdown_connections 2024-04-20T21:18:43.088 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:42.455+0000 7f1174e00700 1 -- 172.21.15.62:0/1555113775 wait complete. 2024-04-20T21:18:43.088 INFO:teuthology.orchestra.run.smithi062.stdout:Enabling cephadm module... 2024-04-20T21:18:44.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:44 smithi062 ceph-mon[16221]: from='client.? 172.21.15.62:0/3641939557' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-20T21:18:44.672 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:43.714+0000 7f2839400700 1 Processor -- start 2024-04-20T21:18:44.672 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:43.714+0000 7f2839400700 1 -- start start 2024-04-20T21:18:44.672 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:43.714+0000 7f2839400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2834107d00 0x7f2834108110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:18:44.672 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:43.715+0000 7f2839400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2834108650 con 0x7f2834107d00 2024-04-20T21:18:44.672 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:43.715+0000 7f2832800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2834107d00 0x7f2834108110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:18:44.672 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:43.715+0000 7f2832800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2834107d00 0x7f2834108110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48062/0 (socket says 172.21.15.62:48062) 2024-04-20T21:18:44.672 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:43.715+0000 7f2832800700 1 -- 172.21.15.62:0/1890939344 learned_addr learned my addr 172.21.15.62:0/1890939344 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:18:44.672 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:43.715+0000 7f2832800700 1 -- 172.21.15.62:0/1890939344 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2834108790 con 0x7f2834107d00 2024-04-20T21:18:44.672 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:43.716+0000 7f2832800700 1 --2- 172.21.15.62:0/1890939344 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2834107d00 0x7f2834108110 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f28240098a0 tx=0x7f2824009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fc87cfc27ea6d47 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:18:44.673 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:43.716+0000 7f2831400700 1 -- 172.21.15.62:0/1890939344 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2824004180 con 0x7f2834107d00 2024-04-20T21:18:44.673 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:43.716+0000 7f2831400700 1 -- 172.21.15.62:0/1890939344 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f28240042e0 con 0x7f2834107d00 2024-04-20T21:18:44.673 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:43.716+0000 7f2831400700 1 -- 172.21.15.62:0/1890939344 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2824011a50 con 0x7f2834107d00 2024-04-20T21:18:44.673 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:43.717+0000 7f2839400700 1 -- 172.21.15.62:0/1890939344 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2834107d00 msgr2=0x7f2834108110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:18:44.673 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:43.717+0000 7f2839400700 1 --2- 172.21.15.62:0/1890939344 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2834107d00 0x7f2834108110 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f28240098a0 tx=0x7f2824009bb0 comp rx=0 tx=0).stop 2024-04-20T21:18:44.673 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:43.717+0000 7f2839400700 1 -- 172.21.15.62:0/1890939344 shutdown_connections 2024-04-20T21:18:44.673 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:43.717+0000 7f2839400700 1 --2- 172.21.15.62:0/1890939344 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2834107d00 0x7f2834108110 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:18:44.673 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:43.717+0000 7f2839400700 1 -- 172.21.15.62:0/1890939344 >> 172.21.15.62:0/1890939344 conn(0x7f283407b220 msgr2=0x7f283407b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:18:44.673 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:43.717+0000 7f2839400700 1 -- 172.21.15.62:0/1890939344 shutdown_connections 2024-04-20T21:18:44.673 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:43.717+0000 7f2839400700 1 -- 172.21.15.62:0/1890939344 wait complete. 2024-04-20T21:18:44.673 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:43.718+0000 7f2839400700 1 Processor -- start 2024-04-20T21:18:44.673 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:43.719+0000 7f2839400700 1 -- start start 2024-04-20T21:18:44.673 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:43.719+0000 7f2839400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2834107d00 0x7f283419ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:18:44.674 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:43.719+0000 7f2839400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f283419bfc0 con 0x7f2834107d00 2024-04-20T21:18:44.674 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:43.719+0000 7f2832800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2834107d00 0x7f283419ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:18:44.674 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:43.719+0000 7f2832800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2834107d00 0x7f283419ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48076/0 (socket says 172.21.15.62:48076) 2024-04-20T21:18:44.674 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:43.719+0000 7f2832800700 1 -- 172.21.15.62:0/3641939557 learned_addr learned my addr 172.21.15.62:0/3641939557 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:18:44.674 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:43.720+0000 7f2832800700 1 -- 172.21.15.62:0/3641939557 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2824009550 con 0x7f2834107d00 2024-04-20T21:18:44.674 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:43.720+0000 7f2832800700 1 --2- 172.21.15.62:0/3641939557 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2834107d00 0x7f283419ba80 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f2824003de0 tx=0x7f2824011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:18:44.674 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:43.720+0000 7f282b400700 1 -- 172.21.15.62:0/3641939557 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2824003930 con 0x7f2834107d00 2024-04-20T21:18:44.674 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:43.720+0000 7f282b400700 1 -- 172.21.15.62:0/3641939557 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f2824003a90 con 0x7f2834107d00 2024-04-20T21:18:44.674 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:43.720+0000 7f282b400700 1 -- 172.21.15.62:0/3641939557 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2824019730 con 0x7f2834107d00 2024-04-20T21:18:44.674 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:43.720+0000 7f2839400700 1 -- 172.21.15.62:0/3641939557 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f283419c1c0 con 0x7f2834107d00 2024-04-20T21:18:44.674 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:43.721+0000 7f2839400700 1 -- 172.21.15.62:0/3641939557 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f283419c5e0 con 0x7f2834107d00 2024-04-20T21:18:44.674 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:43.722+0000 7f282b400700 1 -- 172.21.15.62:0/3641939557 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 4) v1 ==== 45073+0+0 (secure 0 0 0) 0x7f2824019960 con 0x7f2834107d00 2024-04-20T21:18:44.675 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:43.722+0000 7f2839400700 1 -- 172.21.15.62:0/3641939557 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2834062380 con 0x7f2834107d00 2024-04-20T21:18:44.675 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:43.722+0000 7f282b400700 1 --2- 172.21.15.62:0/3641939557 >> [v2:172.21.15.62:6800/871423274,v1:172.21.15.62:6801/871423274] conn(0x7f28180383f0 0x7f281803a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:18:44.675 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:43.722+0000 7f282b400700 1 -- 172.21.15.62:0/3641939557 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f282404c0a0 con 0x7f2834107d00 2024-04-20T21:18:44.675 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:43.722+0000 7f2831e00700 1 --2- 172.21.15.62:0/3641939557 >> [v2:172.21.15.62:6800/871423274,v1:172.21.15.62:6801/871423274] conn(0x7f28180383f0 0x7f281803a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:18:44.675 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:43.723+0000 7f2831e00700 1 --2- 172.21.15.62:0/3641939557 >> [v2:172.21.15.62:6800/871423274,v1:172.21.15.62:6801/871423274] conn(0x7f28180383f0 0x7f281803a8a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2820006fd0 tx=0x7f2820006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:18:44.675 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:43.728+0000 7f282b400700 1 -- 172.21.15.62:0/3641939557 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f2824010770 con 0x7f2834107d00 2024-04-20T21:18:44.675 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:44.007+0000 7f2839400700 1 -- 172.21.15.62:0/3641939557 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f2834109b70 con 0x7f2834107d00 2024-04-20T21:18:44.675 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:44.064+0000 7f282b400700 1 -- 172.21.15.62:0/3641939557 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mgrmap(e 5) v1 ==== 45084+0+0 (secure 0 0 0) 0x7f282401a430 con 0x7f2834107d00 2024-04-20T21:18:44.675 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:44.066+0000 7f282b400700 1 -- 172.21.15.62:0/3641939557 <== mon.0 v2:172.21.15.62:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f282401a950 con 0x7f2834107d00 2024-04-20T21:18:44.675 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:44.069+0000 7f2839400700 1 -- 172.21.15.62:0/3641939557 >> [v2:172.21.15.62:6800/871423274,v1:172.21.15.62:6801/871423274] conn(0x7f28180383f0 msgr2=0x7f281803a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:18:44.675 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:44.069+0000 7f2839400700 1 --2- 172.21.15.62:0/3641939557 >> [v2:172.21.15.62:6800/871423274,v1:172.21.15.62:6801/871423274] conn(0x7f28180383f0 0x7f281803a8a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2820006fd0 tx=0x7f2820006e40 comp rx=0 tx=0).stop 2024-04-20T21:18:44.675 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:44.069+0000 7f2839400700 1 -- 172.21.15.62:0/3641939557 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2834107d00 msgr2=0x7f283419ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:18:44.675 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:44.069+0000 7f2839400700 1 --2- 172.21.15.62:0/3641939557 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2834107d00 0x7f283419ba80 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f2824003de0 tx=0x7f2824011ab0 comp rx=0 tx=0).stop 2024-04-20T21:18:44.675 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:44.069+0000 7f2839400700 1 -- 172.21.15.62:0/3641939557 shutdown_connections 2024-04-20T21:18:44.676 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:44.069+0000 7f2839400700 1 --2- 172.21.15.62:0/3641939557 >> [v2:172.21.15.62:6800/871423274,v1:172.21.15.62:6801/871423274] conn(0x7f28180383f0 0x7f281803a8a0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:18:44.676 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:44.069+0000 7f2839400700 1 --2- 172.21.15.62:0/3641939557 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2834107d00 0x7f283419ba80 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:18:44.676 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:44.069+0000 7f2839400700 1 -- 172.21.15.62:0/3641939557 >> 172.21.15.62:0/3641939557 conn(0x7f283407b220 msgr2=0x7f28341060e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:18:44.676 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:44.070+0000 7f2839400700 1 -- 172.21.15.62:0/3641939557 shutdown_connections 2024-04-20T21:18:44.676 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:44.070+0000 7f2839400700 1 -- 172.21.15.62:0/3641939557 wait complete. 2024-04-20T21:18:45.329 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:45 smithi062 ceph-mon[16221]: from='client.? 172.21.15.62:0/3641939557' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-20T21:18:45.329 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:45 smithi062 ceph-mon[16221]: mgrmap e5: smithi062.rqxuxi(active, since 5s) 2024-04-20T21:18:46.257 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:46 smithi062 ceph-mon[16221]: from='client.? 172.21.15.62:0/3029712865' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-20T21:18:46.269 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout { 2024-04-20T21:18:46.269 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-04-20T21:18:46.270 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-20T21:18:46.270 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "active_name": "smithi062.rqxuxi", 2024-04-20T21:18:46.270 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-20T21:18:46.270 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout } 2024-04-20T21:18:46.270 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:45.329+0000 7f75a4600700 1 Processor -- start 2024-04-20T21:18:46.270 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:45.330+0000 7f75a4600700 1 -- start start 2024-04-20T21:18:46.270 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:45.330+0000 7f75a4600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f759c105b10 0x7f759c105f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:18:46.270 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:45.330+0000 7f75a4600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f759c106460 con 0x7f759c105b10 2024-04-20T21:18:46.270 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:45.331+0000 7f75a1c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f759c105b10 0x7f759c105f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:18:46.270 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:45.331+0000 7f75a1c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f759c105b10 0x7f759c105f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48092/0 (socket says 172.21.15.62:48092) 2024-04-20T21:18:46.270 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:45.331+0000 7f75a1c00700 1 -- 172.21.15.62:0/4235075096 learned_addr learned my addr 172.21.15.62:0/4235075096 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:18:46.270 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:45.331+0000 7f75a1c00700 1 -- 172.21.15.62:0/4235075096 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f759c1065a0 con 0x7f759c105b10 2024-04-20T21:18:46.270 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:45.332+0000 7f75a1c00700 1 --2- 172.21.15.62:0/4235075096 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f759c105b10 0x7f759c105f20 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f75900098a0 tx=0x7f7590009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13f8ca6a957f586c server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:18:46.271 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:45.332+0000 7f759be00700 1 -- 172.21.15.62:0/4235075096 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7590004180 con 0x7f759c105b10 2024-04-20T21:18:46.271 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:45.332+0000 7f759be00700 1 -- 172.21.15.62:0/4235075096 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f75900042e0 con 0x7f759c105b10 2024-04-20T21:18:46.271 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:45.332+0000 7f759be00700 1 -- 172.21.15.62:0/4235075096 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7590011a50 con 0x7f759c105b10 2024-04-20T21:18:46.271 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:45.333+0000 7f75a4600700 1 -- 172.21.15.62:0/4235075096 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f759c105b10 msgr2=0x7f759c105f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:18:46.271 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:45.333+0000 7f75a4600700 1 --2- 172.21.15.62:0/4235075096 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f759c105b10 0x7f759c105f20 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f75900098a0 tx=0x7f7590009bb0 comp rx=0 tx=0).stop 2024-04-20T21:18:46.271 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:45.333+0000 7f75a4600700 1 -- 172.21.15.62:0/4235075096 shutdown_connections 2024-04-20T21:18:46.271 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:45.333+0000 7f75a4600700 1 --2- 172.21.15.62:0/4235075096 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f759c105b10 0x7f759c105f20 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:18:46.271 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:45.333+0000 7f75a4600700 1 -- 172.21.15.62:0/4235075096 >> 172.21.15.62:0/4235075096 conn(0x7f759c101160 msgr2=0x7f759c103590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:18:46.271 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:45.333+0000 7f75a4600700 1 -- 172.21.15.62:0/4235075096 shutdown_connections 2024-04-20T21:18:46.271 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:45.333+0000 7f75a4600700 1 -- 172.21.15.62:0/4235075096 wait complete. 2024-04-20T21:18:46.271 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:45.334+0000 7f75a4600700 1 Processor -- start 2024-04-20T21:18:46.271 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:45.334+0000 7f75a4600700 1 -- start start 2024-04-20T21:18:46.271 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:45.335+0000 7f75a4600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f759c105b10 0x7f759c07ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:18:46.271 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:45.335+0000 7f75a4600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f759c07b070 con 0x7f759c105b10 2024-04-20T21:18:46.272 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:45.335+0000 7f75a1c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f759c105b10 0x7f759c07ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:18:46.272 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:45.335+0000 7f75a1c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f759c105b10 0x7f759c07ab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48096/0 (socket says 172.21.15.62:48096) 2024-04-20T21:18:46.272 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:45.335+0000 7f75a1c00700 1 -- 172.21.15.62:0/3029712865 learned_addr learned my addr 172.21.15.62:0/3029712865 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:18:46.272 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:45.335+0000 7f75a1c00700 1 -- 172.21.15.62:0/3029712865 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7590009550 con 0x7f759c105b10 2024-04-20T21:18:46.272 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:45.336+0000 7f75a1c00700 1 --2- 172.21.15.62:0/3029712865 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f759c105b10 0x7f759c07ab30 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7590000c00 tx=0x7f7590011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:18:46.272 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:45.336+0000 7f759a000700 1 -- 172.21.15.62:0/3029712865 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7590003b90 con 0x7f759c105b10 2024-04-20T21:18:46.272 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:45.336+0000 7f759a000700 1 -- 172.21.15.62:0/3029712865 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f75900248b0 con 0x7f759c105b10 2024-04-20T21:18:46.272 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:45.336+0000 7f75a4600700 1 -- 172.21.15.62:0/3029712865 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f759c079180 con 0x7f759c105b10 2024-04-20T21:18:46.272 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:45.336+0000 7f759a000700 1 -- 172.21.15.62:0/3029712865 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f759001a8e0 con 0x7f759c105b10 2024-04-20T21:18:46.272 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:45.336+0000 7f75a4600700 1 -- 172.21.15.62:0/3029712865 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f759c0795a0 con 0x7f759c105b10 2024-04-20T21:18:46.272 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:45.337+0000 7f75a4600700 1 -- 172.21.15.62:0/3029712865 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f759c062380 con 0x7f759c105b10 2024-04-20T21:18:46.272 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:45.337+0000 7f759a000700 1 -- 172.21.15.62:0/3029712865 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 45084+0+0 (secure 0 0 0) 0x7f7590024440 con 0x7f759c105b10 2024-04-20T21:18:46.272 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:45.338+0000 7f759a000700 1 --2- 172.21.15.62:0/3029712865 >> [v2:172.21.15.62:6800/871423274,v1:172.21.15.62:6801/871423274] conn(0x7f7584038440 0x7f758403a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:18:46.273 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:45.338+0000 7f759a000700 1 -- 172.21.15.62:0/3029712865 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f759004c0b0 con 0x7f759c105b10 2024-04-20T21:18:46.273 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:45.338+0000 7f75a1200700 1 -- 172.21.15.62:0/3029712865 >> [v2:172.21.15.62:6800/871423274,v1:172.21.15.62:6801/871423274] conn(0x7f7584038440 msgr2=0x7f758403a8f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.62:6800/871423274 2024-04-20T21:18:46.273 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:45.338+0000 7f75a1200700 1 --2- 172.21.15.62:0/3029712865 >> [v2:172.21.15.62:6800/871423274,v1:172.21.15.62:6801/871423274] conn(0x7f7584038440 0x7f758403a8f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-20T21:18:46.273 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:45.343+0000 7f759a000700 1 -- 172.21.15.62:0/3029712865 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f759007e720 con 0x7f759c105b10 2024-04-20T21:18:46.273 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:45.539+0000 7f75a1200700 1 -- 172.21.15.62:0/3029712865 >> [v2:172.21.15.62:6800/871423274,v1:172.21.15.62:6801/871423274] conn(0x7f7584038440 msgr2=0x7f758403a8f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.62:6800/871423274 2024-04-20T21:18:46.273 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:45.539+0000 7f75a1200700 1 --2- 172.21.15.62:0/3029712865 >> [v2:172.21.15.62:6800/871423274,v1:172.21.15.62:6801/871423274] conn(0x7f7584038440 0x7f758403a8f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-20T21:18:46.273 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:45.617+0000 7f75a4600700 1 -- 172.21.15.62:0/3029712865 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f759c107980 con 0x7f759c105b10 2024-04-20T21:18:46.273 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:45.618+0000 7f759a000700 1 -- 172.21.15.62:0/3029712865 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7f75900189a0 con 0x7f759c105b10 2024-04-20T21:18:46.273 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:45.622+0000 7f75a4600700 1 -- 172.21.15.62:0/3029712865 >> [v2:172.21.15.62:6800/871423274,v1:172.21.15.62:6801/871423274] conn(0x7f7584038440 msgr2=0x7f758403a8f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-20T21:18:46.273 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:45.622+0000 7f75a4600700 1 --2- 172.21.15.62:0/3029712865 >> [v2:172.21.15.62:6800/871423274,v1:172.21.15.62:6801/871423274] conn(0x7f7584038440 0x7f758403a8f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:18:46.273 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:45.622+0000 7f75a4600700 1 -- 172.21.15.62:0/3029712865 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f759c105b10 msgr2=0x7f759c07ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:18:46.273 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:45.622+0000 7f75a4600700 1 --2- 172.21.15.62:0/3029712865 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f759c105b10 0x7f759c07ab30 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7590000c00 tx=0x7f7590011ab0 comp rx=0 tx=0).stop 2024-04-20T21:18:46.273 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:45.622+0000 7f75a4600700 1 -- 172.21.15.62:0/3029712865 shutdown_connections 2024-04-20T21:18:46.273 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:45.622+0000 7f75a4600700 1 --2- 172.21.15.62:0/3029712865 >> [v2:172.21.15.62:6800/871423274,v1:172.21.15.62:6801/871423274] conn(0x7f7584038440 0x7f758403a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:18:46.274 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:45.622+0000 7f75a4600700 1 --2- 172.21.15.62:0/3029712865 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f759c105b10 0x7f759c07ab30 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:18:46.274 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:45.622+0000 7f75a4600700 1 -- 172.21.15.62:0/3029712865 >> 172.21.15.62:0/3029712865 conn(0x7f759c101160 msgr2=0x7f759c101f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:18:46.274 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:45.623+0000 7f75a4600700 1 -- 172.21.15.62:0/3029712865 shutdown_connections 2024-04-20T21:18:46.274 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:45.623+0000 7f75a4600700 1 -- 172.21.15.62:0/3029712865 wait complete. 2024-04-20T21:18:46.274 INFO:teuthology.orchestra.run.smithi062.stdout:Waiting for the mgr to restart... 2024-04-20T21:18:46.274 INFO:teuthology.orchestra.run.smithi062.stdout:Waiting for mgr epoch 5... 2024-04-20T21:18:53.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:53 smithi062 ceph-mon[16221]: Active manager daemon smithi062.rqxuxi restarted 2024-04-20T21:18:53.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:53 smithi062 ceph-mon[16221]: Activating manager daemon smithi062.rqxuxi 2024-04-20T21:18:53.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:53 smithi062 ceph-mon[16221]: osdmap e2: 0 total, 0 up, 0 in 2024-04-20T21:18:53.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:53 smithi062 ceph-mon[16221]: mgrmap e6: smithi062.rqxuxi(active, starting, since 0.00827717s) 2024-04-20T21:18:53.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:53 smithi062 ceph-mon[16221]: from='mgr.14120 172.21.15.62:0/292923938' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mon metadata", "id": "smithi062"}]: dispatch 2024-04-20T21:18:53.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:53 smithi062 ceph-mon[16221]: from='mgr.14120 172.21.15.62:0/292923938' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mgr metadata", "who": "smithi062.rqxuxi", "id": "smithi062.rqxuxi"}]: dispatch 2024-04-20T21:18:53.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:53 smithi062 ceph-mon[16221]: from='mgr.14120 172.21.15.62:0/292923938' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-20T21:18:53.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:53 smithi062 ceph-mon[16221]: from='mgr.14120 172.21.15.62:0/292923938' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-20T21:18:53.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:53 smithi062 ceph-mon[16221]: from='mgr.14120 172.21.15.62:0/292923938' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-20T21:18:53.287 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:53 smithi062 ceph-mon[16221]: Manager daemon smithi062.rqxuxi is now available 2024-04-20T21:18:53.287 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:53 smithi062 ceph-mon[16221]: from='mgr.14120 172.21.15.62:0/292923938' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:18:53.287 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:53 smithi062 ceph-mon[16221]: from='mgr.14120 172.21.15.62:0/292923938' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:18:54.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:54 smithi062 ceph-mon[16221]: Found migration_current of "None". Setting to last migration. 2024-04-20T21:18:54.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:54 smithi062 ceph-mon[16221]: from='mgr.14120 172.21.15.62:0/292923938' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:18:54.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:54 smithi062 ceph-mon[16221]: from='mgr.14120 172.21.15.62:0/292923938' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:18:54.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:54 smithi062 ceph-mon[16221]: from='mgr.14120 172.21.15.62:0/292923938' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:18:54.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:54 smithi062 ceph-mon[16221]: from='mgr.14120 172.21.15.62:0/292923938' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi062.rqxuxi/mirror_snapshot_schedule"}]: dispatch 2024-04-20T21:18:54.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:54 smithi062 ceph-mon[16221]: from='mgr.14120 172.21.15.62:0/292923938' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi062.rqxuxi/trash_purge_schedule"}]: dispatch 2024-04-20T21:18:54.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:54 smithi062 ceph-mon[16221]: from='mgr.14120 172.21.15.62:0/292923938' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:18:54.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:54 smithi062 ceph-mon[16221]: from='mgr.14120 172.21.15.62:0/292923938' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:18:54.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:54 smithi062 ceph-mon[16221]: mgrmap e7: smithi062.rqxuxi(active, since 1.01514s) 2024-04-20T21:18:54.632 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout { 2024-04-20T21:18:54.632 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-04-20T21:18:54.633 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-20T21:18:54.633 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout } 2024-04-20T21:18:54.633 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:46.848+0000 7f9a18600700 1 Processor -- start 2024-04-20T21:18:54.633 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:46.848+0000 7f9a18600700 1 -- start start 2024-04-20T21:18:54.633 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:46.849+0000 7f9a18600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9a1007acc0 0x7f9a100791c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:18:54.633 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:46.849+0000 7f9a18600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a10079700 con 0x7f9a1007acc0 2024-04-20T21:18:54.633 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:46.849+0000 7f9a15c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9a1007acc0 0x7f9a100791c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:18:54.633 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:46.849+0000 7f9a15c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9a1007acc0 0x7f9a100791c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48100/0 (socket says 172.21.15.62:48100) 2024-04-20T21:18:54.633 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:46.849+0000 7f9a15c00700 1 -- 172.21.15.62:0/4053245871 learned_addr learned my addr 172.21.15.62:0/4053245871 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:18:54.633 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:46.850+0000 7f9a15c00700 1 -- 172.21.15.62:0/4053245871 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a10079840 con 0x7f9a1007acc0 2024-04-20T21:18:54.633 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:46.850+0000 7f9a15c00700 1 --2- 172.21.15.62:0/4053245871 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9a1007acc0 0x7f9a100791c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9a040098a0 tx=0x7f9a04009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=643433c82d97fed7 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:18:54.633 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:46.850+0000 7f9a0fe00700 1 -- 172.21.15.62:0/4053245871 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9a04004180 con 0x7f9a1007acc0 2024-04-20T21:18:54.633 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:46.850+0000 7f9a0fe00700 1 -- 172.21.15.62:0/4053245871 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f9a040042e0 con 0x7f9a1007acc0 2024-04-20T21:18:54.634 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:46.850+0000 7f9a0fe00700 1 -- 172.21.15.62:0/4053245871 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9a04011a50 con 0x7f9a1007acc0 2024-04-20T21:18:54.634 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:46.851+0000 7f9a18600700 1 -- 172.21.15.62:0/4053245871 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9a1007acc0 msgr2=0x7f9a100791c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:18:54.634 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:46.851+0000 7f9a18600700 1 --2- 172.21.15.62:0/4053245871 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9a1007acc0 0x7f9a100791c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9a040098a0 tx=0x7f9a04009bb0 comp rx=0 tx=0).stop 2024-04-20T21:18:54.634 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:46.851+0000 7f9a18600700 1 -- 172.21.15.62:0/4053245871 shutdown_connections 2024-04-20T21:18:54.634 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:46.851+0000 7f9a18600700 1 --2- 172.21.15.62:0/4053245871 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9a1007acc0 0x7f9a100791c0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:18:54.634 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:46.851+0000 7f9a18600700 1 -- 172.21.15.62:0/4053245871 >> 172.21.15.62:0/4053245871 conn(0x7f9a10101110 msgr2=0x7f9a10103520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:18:54.634 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:46.851+0000 7f9a18600700 1 -- 172.21.15.62:0/4053245871 shutdown_connections 2024-04-20T21:18:54.634 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:46.851+0000 7f9a18600700 1 -- 172.21.15.62:0/4053245871 wait complete. 2024-04-20T21:18:54.634 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:46.852+0000 7f9a18600700 1 Processor -- start 2024-04-20T21:18:54.634 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:46.852+0000 7f9a18600700 1 -- start start 2024-04-20T21:18:54.634 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:46.853+0000 7f9a18600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9a1007acc0 0x7f9a1019ba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:18:54.634 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:46.853+0000 7f9a18600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a1019bf50 con 0x7f9a1007acc0 2024-04-20T21:18:54.635 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:46.853+0000 7f9a15c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9a1007acc0 0x7f9a1019ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:18:54.635 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:46.853+0000 7f9a15c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9a1007acc0 0x7f9a1019ba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48110/0 (socket says 172.21.15.62:48110) 2024-04-20T21:18:54.635 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:46.853+0000 7f9a15c00700 1 -- 172.21.15.62:0/4205982559 learned_addr learned my addr 172.21.15.62:0/4205982559 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:18:54.635 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:46.853+0000 7f9a15c00700 1 -- 172.21.15.62:0/4205982559 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a04009550 con 0x7f9a1007acc0 2024-04-20T21:18:54.635 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:46.854+0000 7f9a15c00700 1 --2- 172.21.15.62:0/4205982559 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9a1007acc0 0x7f9a1019ba10 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f9a04000c00 tx=0x7f9a040047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:18:54.635 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:46.854+0000 7f9a0e000700 1 -- 172.21.15.62:0/4205982559 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9a04003a00 con 0x7f9a1007acc0 2024-04-20T21:18:54.635 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:46.854+0000 7f9a0e000700 1 -- 172.21.15.62:0/4205982559 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f9a04003b60 con 0x7f9a1007acc0 2024-04-20T21:18:54.635 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:46.854+0000 7f9a18600700 1 -- 172.21.15.62:0/4205982559 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a1019c150 con 0x7f9a1007acc0 2024-04-20T21:18:54.635 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:46.854+0000 7f9a0e000700 1 -- 172.21.15.62:0/4205982559 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9a0401a800 con 0x7f9a1007acc0 2024-04-20T21:18:54.635 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:46.854+0000 7f9a18600700 1 -- 172.21.15.62:0/4205982559 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a1019c570 con 0x7f9a1007acc0 2024-04-20T21:18:54.635 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:46.855+0000 7f9a0e000700 1 -- 172.21.15.62:0/4205982559 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 45084+0+0 (secure 0 0 0) 0x7f9a04024440 con 0x7f9a1007acc0 2024-04-20T21:18:54.635 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:46.856+0000 7f9a0e000700 1 --2- 172.21.15.62:0/4205982559 >> [v2:172.21.15.62:6800/871423274,v1:172.21.15.62:6801/871423274] conn(0x7f99f80383f0 0x7f99f803a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:18:54.635 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:46.856+0000 7f9a18600700 1 -- 172.21.15.62:0/4205982559 --> [v2:172.21.15.62:6800/871423274,v1:172.21.15.62:6801/871423274] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9a10062380 con 0x7f99f80383f0 2024-04-20T21:18:54.635 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:46.856+0000 7f9a0e000700 1 -- 172.21.15.62:0/4205982559 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f9a0404c060 con 0x7f9a1007acc0 2024-04-20T21:18:54.635 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:46.856+0000 7f9a15200700 1 -- 172.21.15.62:0/4205982559 >> [v2:172.21.15.62:6800/871423274,v1:172.21.15.62:6801/871423274] conn(0x7f99f80383f0 msgr2=0x7f99f803a8a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.62:6800/871423274 2024-04-20T21:18:54.636 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:46.856+0000 7f9a15200700 1 --2- 172.21.15.62:0/4205982559 >> [v2:172.21.15.62:6800/871423274,v1:172.21.15.62:6801/871423274] conn(0x7f99f80383f0 0x7f99f803a8a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-20T21:18:54.636 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:47.057+0000 7f9a15200700 1 -- 172.21.15.62:0/4205982559 >> [v2:172.21.15.62:6800/871423274,v1:172.21.15.62:6801/871423274] conn(0x7f99f80383f0 msgr2=0x7f99f803a8a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.62:6800/871423274 2024-04-20T21:18:54.636 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:47.057+0000 7f9a15200700 1 --2- 172.21.15.62:0/4205982559 >> [v2:172.21.15.62:6800/871423274,v1:172.21.15.62:6801/871423274] conn(0x7f99f80383f0 0x7f99f803a8a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-20T21:18:54.636 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:47.458+0000 7f9a15200700 1 -- 172.21.15.62:0/4205982559 >> [v2:172.21.15.62:6800/871423274,v1:172.21.15.62:6801/871423274] conn(0x7f99f80383f0 msgr2=0x7f99f803a8a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.62:6800/871423274 2024-04-20T21:18:54.636 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:47.458+0000 7f9a15200700 1 --2- 172.21.15.62:0/4205982559 >> [v2:172.21.15.62:6800/871423274,v1:172.21.15.62:6801/871423274] conn(0x7f99f80383f0 0x7f99f803a8a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-20T21:18:54.636 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:48.259+0000 7f9a15200700 1 -- 172.21.15.62:0/4205982559 >> [v2:172.21.15.62:6800/871423274,v1:172.21.15.62:6801/871423274] conn(0x7f99f80383f0 msgr2=0x7f99f803a8a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.62:6800/871423274 2024-04-20T21:18:54.636 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:48.259+0000 7f9a15200700 1 --2- 172.21.15.62:0/4205982559 >> [v2:172.21.15.62:6800/871423274,v1:172.21.15.62:6801/871423274] conn(0x7f99f80383f0 0x7f99f803a8a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-20T21:18:54.636 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:49.861+0000 7f9a15200700 1 -- 172.21.15.62:0/4205982559 >> [v2:172.21.15.62:6800/871423274,v1:172.21.15.62:6801/871423274] conn(0x7f99f80383f0 msgr2=0x7f99f803a8a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.62:6800/871423274 2024-04-20T21:18:54.636 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:49.861+0000 7f9a15200700 1 --2- 172.21.15.62:0/4205982559 >> [v2:172.21.15.62:6800/871423274,v1:172.21.15.62:6801/871423274] conn(0x7f99f80383f0 0x7f99f803a8a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-20T21:18:54.636 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:52.996+0000 7f9a0e000700 1 -- 172.21.15.62:0/4205982559 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mgrmap(e 6) v1 ==== 44851+0+0 (secure 0 0 0) 0x7f9a0402d910 con 0x7f9a1007acc0 2024-04-20T21:18:54.636 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:52.996+0000 7f9a0e000700 1 -- 172.21.15.62:0/4205982559 >> [v2:172.21.15.62:6800/871423274,v1:172.21.15.62:6801/871423274] conn(0x7f99f80383f0 msgr2=0x7f99f803a8a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-20T21:18:54.636 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:52.996+0000 7f9a0e000700 1 --2- 172.21.15.62:0/4205982559 >> [v2:172.21.15.62:6800/871423274,v1:172.21.15.62:6801/871423274] conn(0x7f99f80383f0 0x7f99f803a8a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:18:54.636 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:53.999+0000 7f9a0e000700 1 -- 172.21.15.62:0/4205982559 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mgrmap(e 7) v1 ==== 44978+0+0 (secure 0 0 0) 0x7f9a0404d530 con 0x7f9a1007acc0 2024-04-20T21:18:54.637 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:53.999+0000 7f9a0e000700 1 --2- 172.21.15.62:0/4205982559 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7f99f803bb10 0x7f99f803def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:18:54.637 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:53.999+0000 7f9a0e000700 1 -- 172.21.15.62:0/4205982559 --> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9a10062380 con 0x7f99f803bb10 2024-04-20T21:18:54.637 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:54.000+0000 7f9a15200700 1 --2- 172.21.15.62:0/4205982559 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7f99f803bb10 0x7f99f803def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:18:54.637 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:54.002+0000 7f9a15200700 1 --2- 172.21.15.62:0/4205982559 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7f99f803bb10 0x7f99f803def0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f99fc003a10 tx=0x7f99fc009290 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:18:54.637 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:54.002+0000 7f9a0e000700 1 -- 172.21.15.62:0/4205982559 <== mgr.14120 v2:172.21.15.62:6800/4147738586 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7f9a10062380 con 0x7f99f803bb10 2024-04-20T21:18:54.637 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:54.010+0000 7f9a18600700 1 -- 172.21.15.62:0/4205982559 --> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f9a1019cd50 con 0x7f99f803bb10 2024-04-20T21:18:54.637 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:54.011+0000 7f9a0e000700 1 -- 172.21.15.62:0/4205982559 <== mgr.14120 v2:172.21.15.62:6800/4147738586 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f9a1019cd50 con 0x7f99f803bb10 2024-04-20T21:18:54.637 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:54.011+0000 7f9a18600700 1 -- 172.21.15.62:0/4205982559 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7f99f803bb10 msgr2=0x7f99f803def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:18:54.637 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:54.011+0000 7f9a18600700 1 --2- 172.21.15.62:0/4205982559 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7f99f803bb10 0x7f99f803def0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f99fc003a10 tx=0x7f99fc009290 comp rx=0 tx=0).stop 2024-04-20T21:18:54.637 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:54.011+0000 7f9a18600700 1 -- 172.21.15.62:0/4205982559 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9a1007acc0 msgr2=0x7f9a1019ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:18:54.637 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:54.011+0000 7f9a18600700 1 --2- 172.21.15.62:0/4205982559 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9a1007acc0 0x7f9a1019ba10 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f9a04000c00 tx=0x7f9a040047c0 comp rx=0 tx=0).stop 2024-04-20T21:18:54.637 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:54.012+0000 7f9a18600700 1 -- 172.21.15.62:0/4205982559 shutdown_connections 2024-04-20T21:18:54.637 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:54.012+0000 7f9a18600700 1 --2- 172.21.15.62:0/4205982559 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7f99f803bb10 0x7f99f803def0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:18:54.637 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:54.012+0000 7f9a18600700 1 --2- 172.21.15.62:0/4205982559 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9a1007acc0 0x7f9a1019ba10 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:18:54.638 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:54.012+0000 7f9a18600700 1 --2- 172.21.15.62:0/4205982559 >> [v2:172.21.15.62:6800/871423274,v1:172.21.15.62:6801/871423274] conn(0x7f99f80383f0 0x7f99f803a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:18:54.638 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:54.012+0000 7f9a18600700 1 -- 172.21.15.62:0/4205982559 >> 172.21.15.62:0/4205982559 conn(0x7f9a10101110 msgr2=0x7f9a10101ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:18:54.638 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:54.012+0000 7f9a18600700 1 -- 172.21.15.62:0/4205982559 shutdown_connections 2024-04-20T21:18:54.638 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:54.012+0000 7f9a18600700 1 -- 172.21.15.62:0/4205982559 wait complete. 2024-04-20T21:18:54.638 INFO:teuthology.orchestra.run.smithi062.stdout:mgr epoch 5 is available 2024-04-20T21:18:54.638 INFO:teuthology.orchestra.run.smithi062.stdout:Setting orchestrator backend to cephadm... 2024-04-20T21:18:55.205 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:55 smithi062 ceph-mon[16221]: [20/Apr/2024:21:18:53] ENGINE Bus STARTING 2024-04-20T21:18:55.205 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:55 smithi062 ceph-mon[16221]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-20T21:18:55.205 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:55 smithi062 ceph-mon[16221]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-20T21:18:55.206 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:55 smithi062 ceph-mon[16221]: [20/Apr/2024:21:18:54] ENGINE Serving on http://172.21.15.62:8765 2024-04-20T21:18:55.206 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:55 smithi062 ceph-mon[16221]: from='mgr.14120 172.21.15.62:0/292923938' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:18:56.153 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:56 smithi062 ceph-mon[16221]: [20/Apr/2024:21:18:54] ENGINE Serving on https://172.21.15.62:7150 2024-04-20T21:18:56.154 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:56 smithi062 ceph-mon[16221]: [20/Apr/2024:21:18:54] ENGINE Bus STARTED 2024-04-20T21:18:56.154 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:56 smithi062 ceph-mon[16221]: mgrmap e8: smithi062.rqxuxi(active, since 2s) 2024-04-20T21:18:56.154 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:56 smithi062 ceph-mon[16221]: from='mgr.14120 172.21.15.62:0/292923938' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:18:56.154 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:56 smithi062 ceph-mon[16221]: from='mgr.14120 172.21.15.62:0/292923938' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:18:56.166 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:55.205+0000 7f8d34800700 1 Processor -- start 2024-04-20T21:18:56.167 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:55.205+0000 7f8d34800700 1 -- start start 2024-04-20T21:18:56.167 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:55.206+0000 7f8d34800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d2c106610 0x7f8d2c1089f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:18:56.167 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:55.206+0000 7f8d34800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d2c074510 con 0x7f8d2c106610 2024-04-20T21:18:56.167 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:55.207+0000 7f8d31e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d2c106610 0x7f8d2c1089f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:18:56.169 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:55.207+0000 7f8d31e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d2c106610 0x7f8d2c1089f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34998/0 (socket says 172.21.15.62:34998) 2024-04-20T21:18:56.169 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:55.207+0000 7f8d31e00700 1 -- 172.21.15.62:0/736997132 learned_addr learned my addr 172.21.15.62:0/736997132 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:18:56.169 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:55.207+0000 7f8d31e00700 1 -- 172.21.15.62:0/736997132 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d2c074650 con 0x7f8d2c106610 2024-04-20T21:18:56.169 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:55.208+0000 7f8d31e00700 1 --2- 172.21.15.62:0/736997132 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d2c106610 0x7f8d2c1089f0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f8d200098a0 tx=0x7f8d20009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=447b4f616544e1e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:18:56.169 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:55.208+0000 7f8d30a00700 1 -- 172.21.15.62:0/736997132 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8d20004180 con 0x7f8d2c106610 2024-04-20T21:18:56.169 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:55.208+0000 7f8d30a00700 1 -- 172.21.15.62:0/736997132 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f8d200042e0 con 0x7f8d2c106610 2024-04-20T21:18:56.169 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:55.208+0000 7f8d30a00700 1 -- 172.21.15.62:0/736997132 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8d20011a50 con 0x7f8d2c106610 2024-04-20T21:18:56.169 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:55.209+0000 7f8d34800700 1 -- 172.21.15.62:0/736997132 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d2c106610 msgr2=0x7f8d2c1089f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:18:56.169 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:55.209+0000 7f8d34800700 1 --2- 172.21.15.62:0/736997132 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d2c106610 0x7f8d2c1089f0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f8d200098a0 tx=0x7f8d20009bb0 comp rx=0 tx=0).stop 2024-04-20T21:18:56.169 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:55.209+0000 7f8d34800700 1 -- 172.21.15.62:0/736997132 shutdown_connections 2024-04-20T21:18:56.169 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:55.209+0000 7f8d34800700 1 --2- 172.21.15.62:0/736997132 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d2c106610 0x7f8d2c1089f0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:18:56.169 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:55.209+0000 7f8d34800700 1 -- 172.21.15.62:0/736997132 >> 172.21.15.62:0/736997132 conn(0x7f8d2c100070 msgr2=0x7f8d2c102480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:18:56.169 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:55.209+0000 7f8d34800700 1 -- 172.21.15.62:0/736997132 shutdown_connections 2024-04-20T21:18:56.171 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:55.209+0000 7f8d34800700 1 -- 172.21.15.62:0/736997132 wait complete. 2024-04-20T21:18:56.171 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:55.210+0000 7f8d34800700 1 Processor -- start 2024-04-20T21:18:56.171 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:55.210+0000 7f8d34800700 1 -- start start 2024-04-20T21:18:56.171 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:55.211+0000 7f8d34800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d2c106610 0x7f8d2c19ba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:18:56.171 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:55.211+0000 7f8d34800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d2c19bf70 con 0x7f8d2c106610 2024-04-20T21:18:56.171 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:55.211+0000 7f8d31e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d2c106610 0x7f8d2c19ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:18:56.172 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:55.211+0000 7f8d31e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d2c106610 0x7f8d2c19ba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35008/0 (socket says 172.21.15.62:35008) 2024-04-20T21:18:56.172 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:55.211+0000 7f8d31e00700 1 -- 172.21.15.62:0/2249694423 learned_addr learned my addr 172.21.15.62:0/2249694423 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:18:56.172 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:55.211+0000 7f8d31e00700 1 -- 172.21.15.62:0/2249694423 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d20009550 con 0x7f8d2c106610 2024-04-20T21:18:56.172 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:55.212+0000 7f8d31e00700 1 --2- 172.21.15.62:0/2249694423 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d2c106610 0x7f8d2c19ba30 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f8d20000c00 tx=0x7f8d20011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:18:56.172 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:55.212+0000 7f8d26a00700 1 -- 172.21.15.62:0/2249694423 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8d20003b90 con 0x7f8d2c106610 2024-04-20T21:18:56.172 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:55.212+0000 7f8d26a00700 1 -- 172.21.15.62:0/2249694423 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f8d200248b0 con 0x7f8d2c106610 2024-04-20T21:18:56.172 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:55.212+0000 7f8d34800700 1 -- 172.21.15.62:0/2249694423 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d2c19c170 con 0x7f8d2c106610 2024-04-20T21:18:56.172 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:55.212+0000 7f8d26a00700 1 -- 172.21.15.62:0/2249694423 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8d2001a8e0 con 0x7f8d2c106610 2024-04-20T21:18:56.172 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:55.212+0000 7f8d34800700 1 -- 172.21.15.62:0/2249694423 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d2c19c590 con 0x7f8d2c106610 2024-04-20T21:18:56.172 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:55.213+0000 7f8d26a00700 1 -- 172.21.15.62:0/2249694423 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 8) v1 ==== 45084+0+0 (secure 0 0 0) 0x7f8d20024a20 con 0x7f8d2c106610 2024-04-20T21:18:56.174 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:55.213+0000 7f8d34800700 1 -- 172.21.15.62:0/2249694423 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d2c062380 con 0x7f8d2c106610 2024-04-20T21:18:56.174 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:55.214+0000 7f8d26a00700 1 --2- 172.21.15.62:0/2249694423 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7f8d14038440 0x7f8d1403a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:18:56.174 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:55.214+0000 7f8d26a00700 1 -- 172.21.15.62:0/2249694423 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f8d2004c3a0 con 0x7f8d2c106610 2024-04-20T21:18:56.174 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:55.214+0000 7f8d31400700 1 --2- 172.21.15.62:0/2249694423 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7f8d14038440 0x7f8d1403a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:18:56.174 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:55.215+0000 7f8d31400700 1 --2- 172.21.15.62:0/2249694423 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7f8d14038440 0x7f8d1403a8f0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f8d1c006fd0 tx=0x7f8d1c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:18:56.174 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:55.219+0000 7f8d26a00700 1 -- 172.21.15.62:0/2249694423 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f8d200189a0 con 0x7f8d2c106610 2024-04-20T21:18:56.174 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:55.444+0000 7f8d34800700 1 -- 172.21.15.62:0/2249694423 --> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f8d2c102330 con 0x7f8d14038440 2024-04-20T21:18:56.175 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:55.461+0000 7f8d26a00700 1 -- 172.21.15.62:0/2249694423 <== mgr.14120 v2:172.21.15.62:6800/4147738586 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f8d2c102330 con 0x7f8d14038440 2024-04-20T21:18:56.175 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:55.465+0000 7f8d34800700 1 -- 172.21.15.62:0/2249694423 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7f8d14038440 msgr2=0x7f8d1403a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:18:56.175 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:55.465+0000 7f8d34800700 1 --2- 172.21.15.62:0/2249694423 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7f8d14038440 0x7f8d1403a8f0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f8d1c006fd0 tx=0x7f8d1c006e40 comp rx=0 tx=0).stop 2024-04-20T21:18:56.175 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:55.465+0000 7f8d34800700 1 -- 172.21.15.62:0/2249694423 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d2c106610 msgr2=0x7f8d2c19ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:18:56.175 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:55.465+0000 7f8d34800700 1 --2- 172.21.15.62:0/2249694423 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d2c106610 0x7f8d2c19ba30 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f8d20000c00 tx=0x7f8d20011ab0 comp rx=0 tx=0).stop 2024-04-20T21:18:56.175 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:55.465+0000 7f8d34800700 1 -- 172.21.15.62:0/2249694423 shutdown_connections 2024-04-20T21:18:56.175 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:55.465+0000 7f8d34800700 1 --2- 172.21.15.62:0/2249694423 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7f8d14038440 0x7f8d1403a8f0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:18:56.175 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:55.465+0000 7f8d34800700 1 --2- 172.21.15.62:0/2249694423 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d2c106610 0x7f8d2c19ba30 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:18:56.175 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:55.465+0000 7f8d34800700 1 -- 172.21.15.62:0/2249694423 >> 172.21.15.62:0/2249694423 conn(0x7f8d2c100070 msgr2=0x7f8d2c101710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:18:56.175 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:55.466+0000 7f8d34800700 1 -- 172.21.15.62:0/2249694423 shutdown_connections 2024-04-20T21:18:56.175 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:55.466+0000 7f8d34800700 1 -- 172.21.15.62:0/2249694423 wait complete. 2024-04-20T21:18:57.382 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:57 smithi062 ceph-mon[16221]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:18:57.642 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout value unchanged 2024-04-20T21:18:57.643 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:56.775+0000 7f5baec00700 1 Processor -- start 2024-04-20T21:18:57.644 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:56.775+0000 7f5baec00700 1 -- start start 2024-04-20T21:18:57.644 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:56.776+0000 7f5baec00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5ba8074490 0x7f5ba81086a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:18:57.644 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:56.776+0000 7f5baec00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ba8108be0 con 0x7f5ba8074490 2024-04-20T21:18:57.644 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:56.776+0000 7f5ba7e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5ba8074490 0x7f5ba81086a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:18:57.644 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:56.776+0000 7f5ba7e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5ba8074490 0x7f5ba81086a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35014/0 (socket says 172.21.15.62:35014) 2024-04-20T21:18:57.644 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:56.776+0000 7f5ba7e00700 1 -- 172.21.15.62:0/1888489036 learned_addr learned my addr 172.21.15.62:0/1888489036 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:18:57.644 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:56.777+0000 7f5ba7e00700 1 -- 172.21.15.62:0/1888489036 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ba8108d20 con 0x7f5ba8074490 2024-04-20T21:18:57.644 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:56.777+0000 7f5ba7e00700 1 --2- 172.21.15.62:0/1888489036 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5ba8074490 0x7f5ba81086a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5b980098a0 tx=0x7f5b98009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed014fe12e44ff48 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:18:57.644 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:56.777+0000 7f5ba6a00700 1 -- 172.21.15.62:0/1888489036 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5b98004180 con 0x7f5ba8074490 2024-04-20T21:18:57.644 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:56.777+0000 7f5ba6a00700 1 -- 172.21.15.62:0/1888489036 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f5b980042e0 con 0x7f5ba8074490 2024-04-20T21:18:57.648 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:56.777+0000 7f5ba6a00700 1 -- 172.21.15.62:0/1888489036 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5b98011a50 con 0x7f5ba8074490 2024-04-20T21:18:57.648 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:56.778+0000 7f5baec00700 1 -- 172.21.15.62:0/1888489036 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5ba8074490 msgr2=0x7f5ba81086a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:18:57.648 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:56.778+0000 7f5baec00700 1 --2- 172.21.15.62:0/1888489036 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5ba8074490 0x7f5ba81086a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5b980098a0 tx=0x7f5b98009bb0 comp rx=0 tx=0).stop 2024-04-20T21:18:57.648 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:56.778+0000 7f5baec00700 1 -- 172.21.15.62:0/1888489036 shutdown_connections 2024-04-20T21:18:57.648 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:56.778+0000 7f5baec00700 1 --2- 172.21.15.62:0/1888489036 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5ba8074490 0x7f5ba81086a0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:18:57.648 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:56.778+0000 7f5baec00700 1 -- 172.21.15.62:0/1888489036 >> 172.21.15.62:0/1888489036 conn(0x7f5ba8100070 msgr2=0x7f5ba8102480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:18:57.648 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:56.778+0000 7f5baec00700 1 -- 172.21.15.62:0/1888489036 shutdown_connections 2024-04-20T21:18:57.648 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:56.779+0000 7f5baec00700 1 -- 172.21.15.62:0/1888489036 wait complete. 2024-04-20T21:18:57.649 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:56.780+0000 7f5baec00700 1 Processor -- start 2024-04-20T21:18:57.649 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:56.780+0000 7f5baec00700 1 -- start start 2024-04-20T21:18:57.649 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:56.780+0000 7f5baec00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5ba819ba30 0x7f5ba819be40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:18:57.649 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:56.780+0000 7f5baec00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ba819c380 con 0x7f5ba819ba30 2024-04-20T21:18:57.649 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:56.780+0000 7f5ba7e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5ba819ba30 0x7f5ba819be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:18:57.649 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:56.780+0000 7f5ba7e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5ba819ba30 0x7f5ba819be40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35024/0 (socket says 172.21.15.62:35024) 2024-04-20T21:18:57.649 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:56.780+0000 7f5ba7e00700 1 -- 172.21.15.62:0/2141807989 learned_addr learned my addr 172.21.15.62:0/2141807989 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:18:57.649 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:56.781+0000 7f5ba7e00700 1 -- 172.21.15.62:0/2141807989 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b98009550 con 0x7f5ba819ba30 2024-04-20T21:18:57.649 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:56.781+0000 7f5ba7e00700 1 --2- 172.21.15.62:0/2141807989 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5ba819ba30 0x7f5ba819be40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5b98000c00 tx=0x7f5b98011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:18:57.649 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:56.781+0000 7f5ba4c00700 1 -- 172.21.15.62:0/2141807989 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5b98003b90 con 0x7f5ba819ba30 2024-04-20T21:18:57.649 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:56.781+0000 7f5ba4c00700 1 -- 172.21.15.62:0/2141807989 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f5b980248b0 con 0x7f5ba819ba30 2024-04-20T21:18:57.650 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:56.781+0000 7f5ba4c00700 1 -- 172.21.15.62:0/2141807989 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5b9801a8e0 con 0x7f5ba819ba30 2024-04-20T21:18:57.650 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:56.781+0000 7f5baec00700 1 -- 172.21.15.62:0/2141807989 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ba819c580 con 0x7f5ba819ba30 2024-04-20T21:18:57.650 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:56.782+0000 7f5baec00700 1 -- 172.21.15.62:0/2141807989 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ba819f1e0 con 0x7f5ba819ba30 2024-04-20T21:18:57.650 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:56.783+0000 7f5ba4c00700 1 -- 172.21.15.62:0/2141807989 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 8) v1 ==== 45084+0+0 (secure 0 0 0) 0x7f5b98024a20 con 0x7f5ba819ba30 2024-04-20T21:18:57.650 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:56.783+0000 7f5baec00700 1 -- 172.21.15.62:0/2141807989 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ba8074490 con 0x7f5ba819ba30 2024-04-20T21:18:57.650 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:56.783+0000 7f5ba4c00700 1 --2- 172.21.15.62:0/2141807989 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7f5b8c038440 0x7f5b8c03a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:18:57.650 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:56.783+0000 7f5ba4c00700 1 -- 172.21.15.62:0/2141807989 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f5b9804c460 con 0x7f5ba819ba30 2024-04-20T21:18:57.650 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:56.784+0000 7f5ba7400700 1 --2- 172.21.15.62:0/2141807989 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7f5b8c038440 0x7f5b8c03a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:18:57.650 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:56.784+0000 7f5ba7400700 1 --2- 172.21.15.62:0/2141807989 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7f5b8c038440 0x7f5b8c03a8f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5b94006fd0 tx=0x7f5b94006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:18:57.650 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:56.789+0000 7f5ba4c00700 1 -- 172.21.15.62:0/2141807989 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f5b980189a0 con 0x7f5ba819ba30 2024-04-20T21:18:57.650 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:57.014+0000 7f5baec00700 1 -- 172.21.15.62:0/2141807989 --> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f5ba8102330 con 0x7f5b8c038440 2024-04-20T21:18:57.650 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:57.015+0000 7f5ba4c00700 1 -- 172.21.15.62:0/2141807989 <== mgr.14120 v2:172.21.15.62:6800/4147738586 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f5ba8102330 con 0x7f5b8c038440 2024-04-20T21:18:57.650 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:57.018+0000 7f5baec00700 1 -- 172.21.15.62:0/2141807989 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7f5b8c038440 msgr2=0x7f5b8c03a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:18:57.651 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:57.018+0000 7f5baec00700 1 --2- 172.21.15.62:0/2141807989 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7f5b8c038440 0x7f5b8c03a8f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5b94006fd0 tx=0x7f5b94006e40 comp rx=0 tx=0).stop 2024-04-20T21:18:57.651 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:57.018+0000 7f5baec00700 1 -- 172.21.15.62:0/2141807989 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5ba819ba30 msgr2=0x7f5ba819be40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:18:57.652 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:57.018+0000 7f5baec00700 1 --2- 172.21.15.62:0/2141807989 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5ba819ba30 0x7f5ba819be40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5b98000c00 tx=0x7f5b98011ab0 comp rx=0 tx=0).stop 2024-04-20T21:18:57.652 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:57.018+0000 7f5baec00700 1 -- 172.21.15.62:0/2141807989 shutdown_connections 2024-04-20T21:18:57.652 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:57.018+0000 7f5baec00700 1 --2- 172.21.15.62:0/2141807989 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7f5b8c038440 0x7f5b8c03a8f0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:18:57.652 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:57.018+0000 7f5baec00700 1 --2- 172.21.15.62:0/2141807989 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5ba819ba30 0x7f5ba819be40 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:18:57.652 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:57.018+0000 7f5baec00700 1 -- 172.21.15.62:0/2141807989 >> 172.21.15.62:0/2141807989 conn(0x7f5ba8100070 msgr2=0x7f5ba81016d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:18:57.652 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:57.018+0000 7f5baec00700 1 -- 172.21.15.62:0/2141807989 shutdown_connections 2024-04-20T21:18:57.652 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:57.018+0000 7f5baec00700 1 -- 172.21.15.62:0/2141807989 wait complete. 2024-04-20T21:18:57.652 INFO:teuthology.orchestra.run.smithi062.stdout:Generating ssh key... 2024-04-20T21:18:58.249 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:58 smithi062 ceph-mon[16221]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:18:59.593 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:58.248+0000 7fdf13e00700 1 Processor -- start 2024-04-20T21:18:59.599 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:58.249+0000 7fdf13e00700 1 -- start start 2024-04-20T21:18:59.599 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:58.249+0000 7fdf13e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdf14105920 0x7fdf14105d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:18:59.599 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:58.249+0000 7fdf13e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf14106270 con 0x7fdf14105920 2024-04-20T21:18:59.599 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:58.249+0000 7fdf12a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdf14105920 0x7fdf14105d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:18:59.599 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:58.250+0000 7fdf12a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdf14105920 0x7fdf14105d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35030/0 (socket says 172.21.15.62:35030) 2024-04-20T21:18:59.599 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:58.250+0000 7fdf12a00700 1 -- 172.21.15.62:0/3609324851 learned_addr learned my addr 172.21.15.62:0/3609324851 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:18:59.599 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:58.250+0000 7fdf12a00700 1 -- 172.21.15.62:0/3609324851 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf141063b0 con 0x7fdf14105920 2024-04-20T21:18:59.599 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:58.250+0000 7fdf12a00700 1 --2- 172.21.15.62:0/3609324851 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdf14105920 0x7fdf14105d30 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fdf040098a0 tx=0x7fdf04009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2540c7d46afbe8ca server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:18:59.599 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:58.251+0000 7fdf11600700 1 -- 172.21.15.62:0/3609324851 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf04004180 con 0x7fdf14105920 2024-04-20T21:18:59.599 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:58.251+0000 7fdf11600700 1 -- 172.21.15.62:0/3609324851 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fdf040042e0 con 0x7fdf14105920 2024-04-20T21:18:59.600 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:58.251+0000 7fdf11600700 1 -- 172.21.15.62:0/3609324851 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf04011a50 con 0x7fdf14105920 2024-04-20T21:18:59.600 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:58.251+0000 7fdf13e00700 1 -- 172.21.15.62:0/3609324851 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdf14105920 msgr2=0x7fdf14105d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:18:59.600 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:58.251+0000 7fdf13e00700 1 --2- 172.21.15.62:0/3609324851 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdf14105920 0x7fdf14105d30 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fdf040098a0 tx=0x7fdf04009bb0 comp rx=0 tx=0).stop 2024-04-20T21:18:59.600 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:58.252+0000 7fdf13e00700 1 -- 172.21.15.62:0/3609324851 shutdown_connections 2024-04-20T21:18:59.600 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:58.252+0000 7fdf13e00700 1 --2- 172.21.15.62:0/3609324851 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdf14105920 0x7fdf14105d30 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:18:59.602 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:58.252+0000 7fdf13e00700 1 -- 172.21.15.62:0/3609324851 >> 172.21.15.62:0/3609324851 conn(0x7fdf14100fb0 msgr2=0x7fdf141033c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:18:59.602 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:58.252+0000 7fdf13e00700 1 -- 172.21.15.62:0/3609324851 shutdown_connections 2024-04-20T21:18:59.602 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:58.252+0000 7fdf13e00700 1 -- 172.21.15.62:0/3609324851 wait complete. 2024-04-20T21:18:59.602 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:58.253+0000 7fdf13e00700 1 Processor -- start 2024-04-20T21:18:59.602 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:58.253+0000 7fdf13e00700 1 -- start start 2024-04-20T21:18:59.602 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:58.253+0000 7fdf13e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdf14105920 0x7fdf1419b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:18:59.602 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:58.253+0000 7fdf13e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf1419bdb0 con 0x7fdf14105920 2024-04-20T21:18:59.603 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:58.254+0000 7fdf12a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdf14105920 0x7fdf1419b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:18:59.603 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:58.254+0000 7fdf12a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdf14105920 0x7fdf1419b870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35040/0 (socket says 172.21.15.62:35040) 2024-04-20T21:18:59.603 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:58.254+0000 7fdf12a00700 1 -- 172.21.15.62:0/3510570449 learned_addr learned my addr 172.21.15.62:0/3510570449 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:18:59.603 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:58.254+0000 7fdf12a00700 1 -- 172.21.15.62:0/3510570449 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf04009550 con 0x7fdf14105920 2024-04-20T21:18:59.603 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:58.254+0000 7fdf12a00700 1 --2- 172.21.15.62:0/3510570449 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdf14105920 0x7fdf1419b870 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fdf04000c00 tx=0x7fdf04011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:18:59.603 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:58.255+0000 7fdf18a00700 1 -- 172.21.15.62:0/3510570449 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf04003b90 con 0x7fdf14105920 2024-04-20T21:18:59.603 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:58.255+0000 7fdf18a00700 1 -- 172.21.15.62:0/3510570449 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fdf040248b0 con 0x7fdf14105920 2024-04-20T21:18:59.603 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:58.255+0000 7fdf13e00700 1 -- 172.21.15.62:0/3510570449 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf1419bfb0 con 0x7fdf14105920 2024-04-20T21:18:59.603 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:58.255+0000 7fdf18a00700 1 -- 172.21.15.62:0/3510570449 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf0401a8e0 con 0x7fdf14105920 2024-04-20T21:18:59.603 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:58.255+0000 7fdf13e00700 1 -- 172.21.15.62:0/3510570449 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf1419c3d0 con 0x7fdf14105920 2024-04-20T21:18:59.603 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:58.256+0000 7fdf13e00700 1 -- 172.21.15.62:0/3510570449 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdef4005320 con 0x7fdf14105920 2024-04-20T21:18:59.603 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:58.256+0000 7fdf18a00700 1 -- 172.21.15.62:0/3510570449 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 8) v1 ==== 45084+0+0 (secure 0 0 0) 0x7fdf0401ab50 con 0x7fdf14105920 2024-04-20T21:18:59.604 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:58.256+0000 7fdf18a00700 1 --2- 172.21.15.62:0/3510570449 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7fdef8038440 0x7fdef803a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:18:59.604 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:58.256+0000 7fdf18a00700 1 -- 172.21.15.62:0/3510570449 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fdf0404c440 con 0x7fdf14105920 2024-04-20T21:18:59.604 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:58.257+0000 7fdf12000700 1 --2- 172.21.15.62:0/3510570449 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7fdef8038440 0x7fdef803a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:18:59.604 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:58.257+0000 7fdf12000700 1 --2- 172.21.15.62:0/3510570449 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7fdef8038440 0x7fdef803a8f0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fdf00006fd0 tx=0x7fdf00006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:18:59.604 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:58.262+0000 7fdf18a00700 1 -- 172.21.15.62:0/3510570449 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fdf0401ae00 con 0x7fdf14105920 2024-04-20T21:18:59.604 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:58.484+0000 7fdf13e00700 1 -- 172.21.15.62:0/3510570449 --> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fdef4000bf0 con 0x7fdef8038440 2024-04-20T21:18:59.604 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:58.909+0000 7fdf18a00700 1 -- 172.21.15.62:0/3510570449 <== mgr.14120 v2:172.21.15.62:6800/4147738586 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fdef4000bf0 con 0x7fdef8038440 2024-04-20T21:18:59.604 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:58.912+0000 7fdf13e00700 1 -- 172.21.15.62:0/3510570449 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7fdef8038440 msgr2=0x7fdef803a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:18:59.604 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:58.912+0000 7fdf13e00700 1 --2- 172.21.15.62:0/3510570449 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7fdef8038440 0x7fdef803a8f0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fdf00006fd0 tx=0x7fdf00006e40 comp rx=0 tx=0).stop 2024-04-20T21:18:59.604 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:58.912+0000 7fdf13e00700 1 -- 172.21.15.62:0/3510570449 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdf14105920 msgr2=0x7fdf1419b870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:18:59.604 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:58.912+0000 7fdf13e00700 1 --2- 172.21.15.62:0/3510570449 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdf14105920 0x7fdf1419b870 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fdf04000c00 tx=0x7fdf04011ab0 comp rx=0 tx=0).stop 2024-04-20T21:18:59.605 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:58.912+0000 7fdf13e00700 1 -- 172.21.15.62:0/3510570449 shutdown_connections 2024-04-20T21:18:59.605 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:58.912+0000 7fdf13e00700 1 --2- 172.21.15.62:0/3510570449 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7fdef8038440 0x7fdef803a8f0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:18:59.605 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:58.912+0000 7fdf13e00700 1 --2- 172.21.15.62:0/3510570449 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdf14105920 0x7fdf1419b870 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:18:59.605 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:58.912+0000 7fdf13e00700 1 -- 172.21.15.62:0/3510570449 >> 172.21.15.62:0/3510570449 conn(0x7fdf14100fb0 msgr2=0x7fdf14102690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:18:59.605 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:58.912+0000 7fdf13e00700 1 -- 172.21.15.62:0/3510570449 shutdown_connections 2024-04-20T21:18:59.605 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:18:58.912+0000 7fdf13e00700 1 -- 172.21.15.62:0/3510570449 wait complete. 2024-04-20T21:19:00.189 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:59 smithi062 ceph-mon[16221]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:19:00.189 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:59 smithi062 ceph-mon[16221]: Generating ssh key... 2024-04-20T21:19:00.189 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:59 smithi062 ceph-mon[16221]: from='mgr.14120 172.21.15.62:0/292923938' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:19:00.189 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:18:59 smithi062 ceph-mon[16221]: from='mgr.14120 172.21.15.62:0/292923938' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:19:01.090 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDo8heeGJn1ioyL6p7G0QD0cZdLrSEmX4SZLE4ugWVEC27Z0Qz92rrI+1HFetWULR62uaVx+NYWQq5JT9vyo3fZ9YJaZg4phajNPqKoccbpD0WPkARZBDll4aknfmqPRWbIBPVJZI1/gYX7bXT7/D18I7BeX84zh0P6U9oEGTzpwv3AEWaFS9lki3pR9cw/NE4K3C8G7J6vydxs0TWudV4c9/VxFloHcN+/lZqzvgT4tiD6iVORvmTiUcVa7jfhdNMx7gewmP9ijoKtHomik2Hx6ymiYJLybw+yWW+Rut/aTIX1RcUkZ7F/hjd0hv5PgrpzAsJZezLp72hG3tS3WNaxqEjp7ao/FzaQ6zMgh4XvOUDiGT3B+bF7LYTVyDCRJTh/D0GUwsdPmVV71SAeY4F9aPF3M78y0Jn4QhS8AKSIAx5Fl+3TEQqarEZRfFVcUnPJ6DBUWdZysHoOTM+zYJWZL2TMPDZgmv0VPlQvJlkHT1vjP/Lqjzuao1kwQO3bdeU= ceph-61ae3628-ff5b-11ee-bc93-c7b262605968 2024-04-20T21:19:01.091 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:00.188+0000 7f9db9200700 1 Processor -- start 2024-04-20T21:19:01.091 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:00.189+0000 7f9db9200700 1 -- start start 2024-04-20T21:19:01.091 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:00.189+0000 7f9db9200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9db4104810 0x7f9db4104c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:19:01.091 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:00.189+0000 7f9db9200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9db4105160 con 0x7f9db4104810 2024-04-20T21:19:01.091 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:00.190+0000 7f9db3400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9db4104810 0x7f9db4104c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:19:01.091 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:00.190+0000 7f9db3400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9db4104810 0x7f9db4104c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35056/0 (socket says 172.21.15.62:35056) 2024-04-20T21:19:01.092 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:00.190+0000 7f9db3400700 1 -- 172.21.15.62:0/2707389274 learned_addr learned my addr 172.21.15.62:0/2707389274 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:19:01.093 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:00.190+0000 7f9db3400700 1 -- 172.21.15.62:0/2707389274 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9db41052a0 con 0x7f9db4104810 2024-04-20T21:19:01.093 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:00.191+0000 7f9db3400700 1 --2- 172.21.15.62:0/2707389274 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9db4104810 0x7f9db4104c20 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f9da400bb40 tx=0x7f9da400be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0180f6416d856e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:19:01.093 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:00.191+0000 7f9db2000700 1 -- 172.21.15.62:0/2707389274 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9da400dc00 con 0x7f9db4104810 2024-04-20T21:19:01.093 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:00.191+0000 7f9db2000700 1 -- 172.21.15.62:0/2707389274 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f9da40044a0 con 0x7f9db4104810 2024-04-20T21:19:01.093 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:00.191+0000 7f9db2000700 1 -- 172.21.15.62:0/2707389274 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9da4010470 con 0x7f9db4104810 2024-04-20T21:19:01.093 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:00.192+0000 7f9db9200700 1 -- 172.21.15.62:0/2707389274 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9db4104810 msgr2=0x7f9db4104c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:19:01.093 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:00.192+0000 7f9db9200700 1 --2- 172.21.15.62:0/2707389274 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9db4104810 0x7f9db4104c20 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f9da400bb40 tx=0x7f9da400be50 comp rx=0 tx=0).stop 2024-04-20T21:19:01.093 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:00.192+0000 7f9db9200700 1 -- 172.21.15.62:0/2707389274 shutdown_connections 2024-04-20T21:19:01.093 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:00.192+0000 7f9db9200700 1 --2- 172.21.15.62:0/2707389274 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9db4104810 0x7f9db4104c20 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:19:01.093 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:00.192+0000 7f9db9200700 1 -- 172.21.15.62:0/2707389274 >> 172.21.15.62:0/2707389274 conn(0x7f9db40ffe80 msgr2=0x7f9db4102290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:19:01.093 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:00.192+0000 7f9db9200700 1 -- 172.21.15.62:0/2707389274 shutdown_connections 2024-04-20T21:19:01.093 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:00.192+0000 7f9db9200700 1 -- 172.21.15.62:0/2707389274 wait complete. 2024-04-20T21:19:01.096 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:00.193+0000 7f9db9200700 1 Processor -- start 2024-04-20T21:19:01.096 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:00.193+0000 7f9db9200700 1 -- start start 2024-04-20T21:19:01.096 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:00.194+0000 7f9db9200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9db4104810 0x7f9db419b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:19:01.096 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:00.194+0000 7f9db9200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9db419bda0 con 0x7f9db4104810 2024-04-20T21:19:01.096 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:00.194+0000 7f9db3400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9db4104810 0x7f9db419b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:19:01.096 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:00.194+0000 7f9db3400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9db4104810 0x7f9db419b860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35064/0 (socket says 172.21.15.62:35064) 2024-04-20T21:19:01.096 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:00.194+0000 7f9db3400700 1 -- 172.21.15.62:0/856652372 learned_addr learned my addr 172.21.15.62:0/856652372 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:19:01.096 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:00.194+0000 7f9db3400700 1 -- 172.21.15.62:0/856652372 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9da400b7f0 con 0x7f9db4104810 2024-04-20T21:19:01.096 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:00.195+0000 7f9db3400700 1 --2- 172.21.15.62:0/856652372 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9db4104810 0x7f9db419b860 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f9da40046d0 tx=0x7f9da4003d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:19:01.096 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:00.195+0000 7f9dabe00700 1 -- 172.21.15.62:0/856652372 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9da40042a0 con 0x7f9db4104810 2024-04-20T21:19:01.096 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:00.195+0000 7f9dabe00700 1 -- 172.21.15.62:0/856652372 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f9da4010b50 con 0x7f9db4104810 2024-04-20T21:19:01.096 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:00.195+0000 7f9dabe00700 1 -- 172.21.15.62:0/856652372 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9da4021ae0 con 0x7f9db4104810 2024-04-20T21:19:01.097 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:00.195+0000 7f9db9200700 1 -- 172.21.15.62:0/856652372 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9db419bfa0 con 0x7f9db4104810 2024-04-20T21:19:01.097 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:00.195+0000 7f9db9200700 1 -- 172.21.15.62:0/856652372 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9db419c360 con 0x7f9db4104810 2024-04-20T21:19:01.097 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:00.196+0000 7f9dabe00700 1 -- 172.21.15.62:0/856652372 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 8) v1 ==== 45084+0+0 (secure 0 0 0) 0x7f9da4010cc0 con 0x7f9db4104810 2024-04-20T21:19:01.097 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:00.196+0000 7f9da9600700 1 -- 172.21.15.62:0/856652372 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9db4062380 con 0x7f9db4104810 2024-04-20T21:19:01.097 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:00.197+0000 7f9dabe00700 1 --2- 172.21.15.62:0/856652372 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7f9d980383f0 0x7f9d9803a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:19:01.097 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:00.197+0000 7f9dabe00700 1 -- 172.21.15.62:0/856652372 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f9da404c4a0 con 0x7f9db4104810 2024-04-20T21:19:01.097 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:00.197+0000 7f9db2a00700 1 --2- 172.21.15.62:0/856652372 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7f9d980383f0 0x7f9d9803a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:19:01.097 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:00.198+0000 7f9db2a00700 1 --2- 172.21.15.62:0/856652372 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7f9d980383f0 0x7f9d9803a8a0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f9da0006fd0 tx=0x7f9da0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:19:01.097 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:00.202+0000 7f9dabe00700 1 -- 172.21.15.62:0/856652372 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f9da400f5b0 con 0x7f9db4104810 2024-04-20T21:19:01.099 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:00.422+0000 7f9da9600700 1 -- 172.21.15.62:0/856652372 --> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f9db4106680 con 0x7f9d980383f0 2024-04-20T21:19:01.099 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:00.424+0000 7f9dabe00700 1 -- 172.21.15.62:0/856652372 <== mgr.14120 v2:172.21.15.62:6800/4147738586 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f9db4106680 con 0x7f9d980383f0 2024-04-20T21:19:01.099 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:00.427+0000 7f9da9600700 1 -- 172.21.15.62:0/856652372 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7f9d980383f0 msgr2=0x7f9d9803a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:19:01.099 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:00.427+0000 7f9da9600700 1 --2- 172.21.15.62:0/856652372 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7f9d980383f0 0x7f9d9803a8a0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f9da0006fd0 tx=0x7f9da0006e40 comp rx=0 tx=0).stop 2024-04-20T21:19:01.099 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:00.427+0000 7f9da9600700 1 -- 172.21.15.62:0/856652372 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9db4104810 msgr2=0x7f9db419b860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:19:01.099 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:00.427+0000 7f9da9600700 1 --2- 172.21.15.62:0/856652372 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9db4104810 0x7f9db419b860 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f9da40046d0 tx=0x7f9da4003d70 comp rx=0 tx=0).stop 2024-04-20T21:19:01.099 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:00.427+0000 7f9da9600700 1 -- 172.21.15.62:0/856652372 shutdown_connections 2024-04-20T21:19:01.099 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:00.427+0000 7f9da9600700 1 --2- 172.21.15.62:0/856652372 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7f9d980383f0 0x7f9d9803a8a0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:19:01.099 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:00.427+0000 7f9da9600700 1 --2- 172.21.15.62:0/856652372 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9db4104810 0x7f9db419b860 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:19:01.099 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:00.427+0000 7f9da9600700 1 -- 172.21.15.62:0/856652372 >> 172.21.15.62:0/856652372 conn(0x7f9db40ffe80 msgr2=0x7f9db4192690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:19:01.100 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:00.427+0000 7f9da9600700 1 -- 172.21.15.62:0/856652372 shutdown_connections 2024-04-20T21:19:01.100 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:00.427+0000 7f9da9600700 1 -- 172.21.15.62:0/856652372 wait complete. 2024-04-20T21:19:01.100 INFO:teuthology.orchestra.run.smithi062.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-20T21:19:01.100 INFO:teuthology.orchestra.run.smithi062.stdout:Adding key to root@localhost authorized_keys... 2024-04-20T21:19:01.104 INFO:teuthology.orchestra.run.smithi062.stdout:Adding host smithi062... 2024-04-20T21:19:01.398 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:01 smithi062 ceph-mon[16221]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:19:02.930 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:02 smithi062 ceph-mon[16221]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi062", "addr": "172.21.15.62", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:19:04.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:04 smithi062 ceph-mon[16221]: Deploying cephadm binary to smithi062 2024-04-20T21:19:06.955 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout Added host 'smithi062' with addr '172.21.15.62' 2024-04-20T21:19:06.956 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:01.699+0000 7ff936000700 1 Processor -- start 2024-04-20T21:19:06.956 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:01.699+0000 7ff936000700 1 -- start start 2024-04-20T21:19:06.956 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:01.700+0000 7ff936000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff93007be10 0x7ff93007a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:19:06.956 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:01.700+0000 7ff936000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff93007a850 con 0x7ff93007be10 2024-04-20T21:19:06.956 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:01.700+0000 7ff934c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff93007be10 0x7ff93007a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:19:06.957 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:01.700+0000 7ff934c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff93007be10 0x7ff93007a310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33370/0 (socket says 172.21.15.62:33370) 2024-04-20T21:19:06.957 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:01.700+0000 7ff934c00700 1 -- 172.21.15.62:0/585837503 learned_addr learned my addr 172.21.15.62:0/585837503 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:19:06.957 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:01.701+0000 7ff934c00700 1 -- 172.21.15.62:0/585837503 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff93007a990 con 0x7ff93007be10 2024-04-20T21:19:06.957 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:01.701+0000 7ff934c00700 1 --2- 172.21.15.62:0/585837503 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff93007be10 0x7ff93007a310 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff9240098a0 tx=0x7ff924009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=757b8e5b2f91b9b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:19:06.957 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:01.701+0000 7ff92f400700 1 -- 172.21.15.62:0/585837503 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff924004180 con 0x7ff93007be10 2024-04-20T21:19:06.958 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:01.701+0000 7ff92f400700 1 -- 172.21.15.62:0/585837503 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7ff9240042e0 con 0x7ff93007be10 2024-04-20T21:19:06.958 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:01.701+0000 7ff92f400700 1 -- 172.21.15.62:0/585837503 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff924011a50 con 0x7ff93007be10 2024-04-20T21:19:06.958 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:01.702+0000 7ff936000700 1 -- 172.21.15.62:0/585837503 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff93007be10 msgr2=0x7ff93007a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:19:06.959 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:01.702+0000 7ff936000700 1 --2- 172.21.15.62:0/585837503 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff93007be10 0x7ff93007a310 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff9240098a0 tx=0x7ff924009bb0 comp rx=0 tx=0).stop 2024-04-20T21:19:06.959 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:01.702+0000 7ff936000700 1 -- 172.21.15.62:0/585837503 shutdown_connections 2024-04-20T21:19:06.959 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:01.702+0000 7ff936000700 1 --2- 172.21.15.62:0/585837503 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff93007be10 0x7ff93007a310 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:19:06.960 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:01.702+0000 7ff936000700 1 -- 172.21.15.62:0/585837503 >> 172.21.15.62:0/585837503 conn(0x7ff930100fa0 msgr2=0x7ff9301033b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:19:06.960 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:01.702+0000 7ff936000700 1 -- 172.21.15.62:0/585837503 shutdown_connections 2024-04-20T21:19:06.960 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:01.702+0000 7ff936000700 1 -- 172.21.15.62:0/585837503 wait complete. 2024-04-20T21:19:06.963 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:01.703+0000 7ff936000700 1 Processor -- start 2024-04-20T21:19:06.963 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:01.703+0000 7ff936000700 1 -- start start 2024-04-20T21:19:06.963 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:01.704+0000 7ff936000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff93007be10 0x7ff93019b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:19:06.963 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:01.704+0000 7ff936000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff93019be00 con 0x7ff93007be10 2024-04-20T21:19:06.963 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:01.704+0000 7ff934c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff93007be10 0x7ff93019b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:19:06.963 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:01.704+0000 7ff934c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff93007be10 0x7ff93019b8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33382/0 (socket says 172.21.15.62:33382) 2024-04-20T21:19:06.963 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:01.704+0000 7ff934c00700 1 -- 172.21.15.62:0/3499913527 learned_addr learned my addr 172.21.15.62:0/3499913527 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:19:06.963 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:01.704+0000 7ff934c00700 1 -- 172.21.15.62:0/3499913527 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff924009550 con 0x7ff93007be10 2024-04-20T21:19:06.963 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:01.705+0000 7ff934c00700 1 --2- 172.21.15.62:0/3499913527 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff93007be10 0x7ff93019b8c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7ff924006af0 tx=0x7ff924011a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:19:06.963 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:01.705+0000 7ff92d600700 1 -- 172.21.15.62:0/3499913527 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff924003b90 con 0x7ff93007be10 2024-04-20T21:19:06.964 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:01.705+0000 7ff92d600700 1 -- 172.21.15.62:0/3499913527 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7ff92401a8a0 con 0x7ff93007be10 2024-04-20T21:19:06.964 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:01.705+0000 7ff936000700 1 -- 172.21.15.62:0/3499913527 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff93019c000 con 0x7ff93007be10 2024-04-20T21:19:06.964 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:01.705+0000 7ff92d600700 1 -- 172.21.15.62:0/3499913527 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff924019880 con 0x7ff93007be10 2024-04-20T21:19:06.964 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:01.705+0000 7ff936000700 1 -- 172.21.15.62:0/3499913527 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff93019c420 con 0x7ff93007be10 2024-04-20T21:19:06.964 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:01.706+0000 7ff936000700 1 -- 172.21.15.62:0/3499913527 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff930062380 con 0x7ff93007be10 2024-04-20T21:19:06.964 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:01.706+0000 7ff92d600700 1 -- 172.21.15.62:0/3499913527 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 8) v1 ==== 45084+0+0 (secure 0 0 0) 0x7ff92401aa10 con 0x7ff93007be10 2024-04-20T21:19:06.965 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:01.707+0000 7ff92d600700 1 --2- 172.21.15.62:0/3499913527 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7ff9180383b0 0x7ff91803a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:19:06.965 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:01.707+0000 7ff92d600700 1 -- 172.21.15.62:0/3499913527 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7ff92404c790 con 0x7ff93007be10 2024-04-20T21:19:06.965 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:01.707+0000 7ff92fe00700 1 --2- 172.21.15.62:0/3499913527 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7ff9180383b0 0x7ff91803a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:19:06.965 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:01.708+0000 7ff92fe00700 1 --2- 172.21.15.62:0/3499913527 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7ff9180383b0 0x7ff91803a860 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ff920006fd0 tx=0x7ff920006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:19:06.965 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:01.713+0000 7ff92d600700 1 -- 172.21.15.62:0/3499913527 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7ff924018530 con 0x7ff93007be10 2024-04-20T21:19:06.966 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:01.938+0000 7ff936000700 1 -- 172.21.15.62:0/3499913527 --> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi062", "addr": "172.21.15.62", "target": ["mon-mgr", ""]}) v1 -- 0x7ff930103280 con 0x7ff9180383b0 2024-04-20T21:19:06.966 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:06.247+0000 7ff92d600700 1 -- 172.21.15.62:0/3499913527 <== mgr.14120 v2:172.21.15.62:6800/4147738586 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7ff930103280 con 0x7ff9180383b0 2024-04-20T21:19:06.966 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:06.251+0000 7ff936000700 1 -- 172.21.15.62:0/3499913527 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7ff9180383b0 msgr2=0x7ff91803a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:19:06.966 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:06.251+0000 7ff936000700 1 --2- 172.21.15.62:0/3499913527 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7ff9180383b0 0x7ff91803a860 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ff920006fd0 tx=0x7ff920006e40 comp rx=0 tx=0).stop 2024-04-20T21:19:06.967 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:06.251+0000 7ff936000700 1 -- 172.21.15.62:0/3499913527 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff93007be10 msgr2=0x7ff93019b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:19:06.967 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:06.251+0000 7ff936000700 1 --2- 172.21.15.62:0/3499913527 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff93007be10 0x7ff93019b8c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7ff924006af0 tx=0x7ff924011a90 comp rx=0 tx=0).stop 2024-04-20T21:19:06.967 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:06.251+0000 7ff936000700 1 -- 172.21.15.62:0/3499913527 shutdown_connections 2024-04-20T21:19:06.967 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:06.251+0000 7ff936000700 1 --2- 172.21.15.62:0/3499913527 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7ff9180383b0 0x7ff91803a860 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:19:06.967 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:06.251+0000 7ff936000700 1 --2- 172.21.15.62:0/3499913527 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff93007be10 0x7ff93019b8c0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:19:06.967 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:06.251+0000 7ff936000700 1 -- 172.21.15.62:0/3499913527 >> 172.21.15.62:0/3499913527 conn(0x7ff930100fa0 msgr2=0x7ff930102660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:19:06.968 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:06.252+0000 7ff936000700 1 -- 172.21.15.62:0/3499913527 shutdown_connections 2024-04-20T21:19:06.968 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:06.252+0000 7ff936000700 1 -- 172.21.15.62:0/3499913527 wait complete. 2024-04-20T21:19:06.968 INFO:teuthology.orchestra.run.smithi062.stdout:Deploying mon service with default placement... 2024-04-20T21:19:07.514 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:07 smithi062 ceph-mon[16221]: from='mgr.14120 172.21.15.62:0/292923938' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:19:07.514 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:07 smithi062 ceph-mon[16221]: Added host smithi062 2024-04-20T21:19:07.514 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:07 smithi062 ceph-mon[16221]: from='mgr.14120 172.21.15.62:0/292923938' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:19:09.221 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-04-20T21:19:09.221 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:07.802+0000 7f4529000700 1 Processor -- start 2024-04-20T21:19:09.221 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:07.803+0000 7f4529000700 1 -- start start 2024-04-20T21:19:09.221 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:07.803+0000 7f4529000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4524107d00 0x7f4524108110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:19:09.221 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:07.803+0000 7f4529000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4524108650 con 0x7f4524107d00 2024-04-20T21:19:09.221 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:07.804+0000 7f4521e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4524107d00 0x7f4524108110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:19:09.221 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:07.804+0000 7f4521e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4524107d00 0x7f4524108110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33384/0 (socket says 172.21.15.62:33384) 2024-04-20T21:19:09.221 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:07.804+0000 7f4521e00700 1 -- 172.21.15.62:0/3397782824 learned_addr learned my addr 172.21.15.62:0/3397782824 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:19:09.221 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:07.805+0000 7f4521e00700 1 -- 172.21.15.62:0/3397782824 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4524108790 con 0x7f4524107d00 2024-04-20T21:19:09.221 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:07.805+0000 7f4521e00700 1 --2- 172.21.15.62:0/3397782824 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4524107d00 0x7f4524108110 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f45140098a0 tx=0x7f4514009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a32c2120bfcb252 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:19:09.221 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:07.805+0000 7f4520a00700 1 -- 172.21.15.62:0/3397782824 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4514004180 con 0x7f4524107d00 2024-04-20T21:19:09.221 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:07.805+0000 7f4520a00700 1 -- 172.21.15.62:0/3397782824 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f45140042e0 con 0x7f4524107d00 2024-04-20T21:19:09.222 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:07.805+0000 7f4520a00700 1 -- 172.21.15.62:0/3397782824 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4514011a50 con 0x7f4524107d00 2024-04-20T21:19:09.222 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:07.806+0000 7f4529000700 1 -- 172.21.15.62:0/3397782824 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4524107d00 msgr2=0x7f4524108110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:19:09.222 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:07.806+0000 7f4529000700 1 --2- 172.21.15.62:0/3397782824 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4524107d00 0x7f4524108110 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f45140098a0 tx=0x7f4514009bb0 comp rx=0 tx=0).stop 2024-04-20T21:19:09.222 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:07.806+0000 7f4529000700 1 -- 172.21.15.62:0/3397782824 shutdown_connections 2024-04-20T21:19:09.222 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:07.806+0000 7f4529000700 1 --2- 172.21.15.62:0/3397782824 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4524107d00 0x7f4524108110 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:19:09.222 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:07.806+0000 7f4529000700 1 -- 172.21.15.62:0/3397782824 >> 172.21.15.62:0/3397782824 conn(0x7f452407b220 msgr2=0x7f452407b640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:19:09.222 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:07.806+0000 7f4529000700 1 -- 172.21.15.62:0/3397782824 shutdown_connections 2024-04-20T21:19:09.222 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:07.807+0000 7f4529000700 1 -- 172.21.15.62:0/3397782824 wait complete. 2024-04-20T21:19:09.222 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:07.808+0000 7f4529000700 1 Processor -- start 2024-04-20T21:19:09.222 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:07.808+0000 7f4529000700 1 -- start start 2024-04-20T21:19:09.222 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:07.808+0000 7f4529000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4524107d00 0x7f452419ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:19:09.222 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:07.808+0000 7f4529000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f452419bfb0 con 0x7f4524107d00 2024-04-20T21:19:09.222 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:07.808+0000 7f4521e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4524107d00 0x7f452419ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:19:09.222 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:07.808+0000 7f4521e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4524107d00 0x7f452419ba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33400/0 (socket says 172.21.15.62:33400) 2024-04-20T21:19:09.223 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:07.808+0000 7f4521e00700 1 -- 172.21.15.62:0/1025402460 learned_addr learned my addr 172.21.15.62:0/1025402460 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:19:09.223 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:07.809+0000 7f4521e00700 1 -- 172.21.15.62:0/1025402460 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4514009550 con 0x7f4524107d00 2024-04-20T21:19:09.223 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:07.809+0000 7f4521e00700 1 --2- 172.21.15.62:0/1025402460 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4524107d00 0x7f452419ba70 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4514000c00 tx=0x7f4514011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:19:09.223 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:07.809+0000 7f451aa00700 1 -- 172.21.15.62:0/1025402460 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4514003b90 con 0x7f4524107d00 2024-04-20T21:19:09.223 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:07.809+0000 7f451aa00700 1 -- 172.21.15.62:0/1025402460 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f45140248b0 con 0x7f4524107d00 2024-04-20T21:19:09.223 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:07.809+0000 7f4529000700 1 -- 172.21.15.62:0/1025402460 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f452419c1b0 con 0x7f4524107d00 2024-04-20T21:19:09.223 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:07.810+0000 7f451aa00700 1 -- 172.21.15.62:0/1025402460 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f451401a8e0 con 0x7f4524107d00 2024-04-20T21:19:09.223 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:07.810+0000 7f4529000700 1 -- 172.21.15.62:0/1025402460 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f452419c5d0 con 0x7f4524107d00 2024-04-20T21:19:09.223 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:07.811+0000 7f451aa00700 1 -- 172.21.15.62:0/1025402460 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 8) v1 ==== 45084+0+0 (secure 0 0 0) 0x7f4514024440 con 0x7f4524107d00 2024-04-20T21:19:09.223 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:07.811+0000 7f4529000700 1 -- 172.21.15.62:0/1025402460 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4524062380 con 0x7f4524107d00 2024-04-20T21:19:09.223 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:07.811+0000 7f451aa00700 1 --2- 172.21.15.62:0/1025402460 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7f4508038440 0x7f450803a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:19:09.223 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:07.811+0000 7f451aa00700 1 -- 172.21.15.62:0/1025402460 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f451404c490 con 0x7f4524107d00 2024-04-20T21:19:09.223 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:07.812+0000 7f4521400700 1 --2- 172.21.15.62:0/1025402460 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7f4508038440 0x7f450803a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:19:09.223 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:07.812+0000 7f4521400700 1 --2- 172.21.15.62:0/1025402460 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7f4508038440 0x7f450803a8f0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4510006fd0 tx=0x7f4510006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:19:09.224 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:07.817+0000 7f451aa00700 1 -- 172.21.15.62:0/1025402460 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f45140189a0 con 0x7f4524107d00 2024-04-20T21:19:09.224 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:08.059+0000 7f4529000700 1 -- 172.21.15.62:0/1025402460 --> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7f4524106c50 con 0x7f4508038440 2024-04-20T21:19:09.224 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:08.072+0000 7f451aa00700 1 -- 172.21.15.62:0/1025402460 <== mgr.14120 v2:172.21.15.62:6800/4147738586 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f4524106c50 con 0x7f4508038440 2024-04-20T21:19:09.224 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:08.075+0000 7f4529000700 1 -- 172.21.15.62:0/1025402460 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7f4508038440 msgr2=0x7f450803a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:19:09.224 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:08.075+0000 7f4529000700 1 --2- 172.21.15.62:0/1025402460 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7f4508038440 0x7f450803a8f0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4510006fd0 tx=0x7f4510006e40 comp rx=0 tx=0).stop 2024-04-20T21:19:09.224 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:08.075+0000 7f4529000700 1 -- 172.21.15.62:0/1025402460 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4524107d00 msgr2=0x7f452419ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:19:09.224 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:08.075+0000 7f4529000700 1 --2- 172.21.15.62:0/1025402460 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4524107d00 0x7f452419ba70 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4514000c00 tx=0x7f4514011ab0 comp rx=0 tx=0).stop 2024-04-20T21:19:09.224 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:08.075+0000 7f4529000700 1 -- 172.21.15.62:0/1025402460 shutdown_connections 2024-04-20T21:19:09.224 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:08.075+0000 7f4529000700 1 --2- 172.21.15.62:0/1025402460 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7f4508038440 0x7f450803a8f0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:19:09.224 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:08.075+0000 7f4529000700 1 --2- 172.21.15.62:0/1025402460 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4524107d00 0x7f452419ba70 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:19:09.224 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:08.075+0000 7f4529000700 1 -- 172.21.15.62:0/1025402460 >> 172.21.15.62:0/1025402460 conn(0x7f452407b220 msgr2=0x7f4524105ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:19:09.224 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:08.075+0000 7f4529000700 1 -- 172.21.15.62:0/1025402460 shutdown_connections 2024-04-20T21:19:09.224 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:08.076+0000 7f4529000700 1 -- 172.21.15.62:0/1025402460 wait complete. 2024-04-20T21:19:09.224 INFO:teuthology.orchestra.run.smithi062.stdout:Deploying mgr service with default placement... 2024-04-20T21:19:09.368 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:09 smithi062 ceph-mon[16221]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:19:09.368 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:09 smithi062 ceph-mon[16221]: Saving service mon spec with placement count:5 2024-04-20T21:19:09.368 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:09 smithi062 ceph-mon[16221]: from='mgr.14120 172.21.15.62:0/292923938' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:19:10.443 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:10 smithi062 ceph-mon[16221]: from='mgr.14120 172.21.15.62:0/292923938' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:19:10.443 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:10 smithi062 ceph-mon[16221]: from='mgr.14120 172.21.15.62:0/292923938' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:19:10.443 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:10 smithi062 ceph-mon[16221]: from='mgr.14120 172.21.15.62:0/292923938' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:19:10.765 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-04-20T21:19:10.765 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:09.806+0000 7f1041e00700 1 Processor -- start 2024-04-20T21:19:10.765 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:09.806+0000 7f1041e00700 1 -- start start 2024-04-20T21:19:10.765 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:09.806+0000 7f1041e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f103c105920 0x7f103c105d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:19:10.765 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:09.806+0000 7f1041e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f103c106270 con 0x7f103c105920 2024-04-20T21:19:10.765 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:09.807+0000 7f1040a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f103c105920 0x7f103c105d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:19:10.766 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:09.807+0000 7f1040a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f103c105920 0x7f103c105d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33410/0 (socket says 172.21.15.62:33410) 2024-04-20T21:19:10.766 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:09.807+0000 7f1040a00700 1 -- 172.21.15.62:0/2471871708 learned_addr learned my addr 172.21.15.62:0/2471871708 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:19:10.766 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:09.807+0000 7f1040a00700 1 -- 172.21.15.62:0/2471871708 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f103c1063b0 con 0x7f103c105920 2024-04-20T21:19:10.766 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:09.807+0000 7f1040a00700 1 --2- 172.21.15.62:0/2471871708 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f103c105920 0x7f103c105d30 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f10300098a0 tx=0x7f1030009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f70317167bb8ff83 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:19:10.766 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:09.808+0000 7f103b400700 1 -- 172.21.15.62:0/2471871708 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1030004180 con 0x7f103c105920 2024-04-20T21:19:10.766 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:09.808+0000 7f103b400700 1 -- 172.21.15.62:0/2471871708 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f10300042e0 con 0x7f103c105920 2024-04-20T21:19:10.766 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:09.808+0000 7f103b400700 1 -- 172.21.15.62:0/2471871708 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1030011af0 con 0x7f103c105920 2024-04-20T21:19:10.766 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:09.808+0000 7f1041e00700 1 -- 172.21.15.62:0/2471871708 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f103c105920 msgr2=0x7f103c105d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:19:10.766 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:09.808+0000 7f1041e00700 1 --2- 172.21.15.62:0/2471871708 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f103c105920 0x7f103c105d30 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f10300098a0 tx=0x7f1030009bb0 comp rx=0 tx=0).stop 2024-04-20T21:19:10.766 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:09.809+0000 7f1041e00700 1 -- 172.21.15.62:0/2471871708 shutdown_connections 2024-04-20T21:19:10.766 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:09.809+0000 7f1041e00700 1 --2- 172.21.15.62:0/2471871708 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f103c105920 0x7f103c105d30 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:19:10.766 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:09.809+0000 7f1041e00700 1 -- 172.21.15.62:0/2471871708 >> 172.21.15.62:0/2471871708 conn(0x7f103c100fb0 msgr2=0x7f103c1033c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:19:10.766 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:09.809+0000 7f1041e00700 1 -- 172.21.15.62:0/2471871708 shutdown_connections 2024-04-20T21:19:10.767 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:09.809+0000 7f1041e00700 1 -- 172.21.15.62:0/2471871708 wait complete. 2024-04-20T21:19:10.767 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:09.810+0000 7f1041e00700 1 Processor -- start 2024-04-20T21:19:10.767 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:09.810+0000 7f1041e00700 1 -- start start 2024-04-20T21:19:10.767 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:09.810+0000 7f1041e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f103c105920 0x7f103c1996a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:19:10.767 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:09.811+0000 7f1041e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f103c199be0 con 0x7f103c105920 2024-04-20T21:19:10.767 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:09.811+0000 7f1040a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f103c105920 0x7f103c1996a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:19:10.767 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:09.811+0000 7f1040a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f103c105920 0x7f103c1996a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33420/0 (socket says 172.21.15.62:33420) 2024-04-20T21:19:10.767 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:09.811+0000 7f1040a00700 1 -- 172.21.15.62:0/532620238 learned_addr learned my addr 172.21.15.62:0/532620238 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:19:10.767 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:09.811+0000 7f1040a00700 1 -- 172.21.15.62:0/532620238 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1030009550 con 0x7f103c105920 2024-04-20T21:19:10.767 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:09.812+0000 7f1040a00700 1 --2- 172.21.15.62:0/532620238 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f103c105920 0x7f103c1996a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f1030011ad0 tx=0x7f1030011bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:19:10.767 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:09.812+0000 7f1039600700 1 -- 172.21.15.62:0/532620238 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1030003ab0 con 0x7f103c105920 2024-04-20T21:19:10.768 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:09.812+0000 7f1039600700 1 -- 172.21.15.62:0/532620238 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f1030003c10 con 0x7f103c105920 2024-04-20T21:19:10.768 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:09.812+0000 7f1039600700 1 -- 172.21.15.62:0/532620238 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f10300198e0 con 0x7f103c105920 2024-04-20T21:19:10.768 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:09.812+0000 7f1041e00700 1 -- 172.21.15.62:0/532620238 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f103c199de0 con 0x7f103c105920 2024-04-20T21:19:10.768 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:09.812+0000 7f1041e00700 1 -- 172.21.15.62:0/532620238 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f103c19a200 con 0x7f103c105920 2024-04-20T21:19:10.768 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:09.813+0000 7f1041e00700 1 -- 172.21.15.62:0/532620238 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f103c062380 con 0x7f103c105920 2024-04-20T21:19:10.768 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:09.814+0000 7f1039600700 1 -- 172.21.15.62:0/532620238 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 8) v1 ==== 45084+0+0 (secure 0 0 0) 0x7f1030019a40 con 0x7f103c105920 2024-04-20T21:19:10.768 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:09.814+0000 7f1039600700 1 --2- 172.21.15.62:0/532620238 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7f1024038440 0x7f102403a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:19:10.768 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:09.814+0000 7f1039600700 1 -- 172.21.15.62:0/532620238 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f103004c180 con 0x7f103c105920 2024-04-20T21:19:10.768 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:09.815+0000 7f103be00700 1 --2- 172.21.15.62:0/532620238 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7f1024038440 0x7f102403a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:19:10.768 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:09.815+0000 7f103be00700 1 --2- 172.21.15.62:0/532620238 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7f1024038440 0x7f102403a8f0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f1028006fd0 tx=0x7f1028006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:19:10.769 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:09.819+0000 7f1039600700 1 -- 172.21.15.62:0/532620238 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f103001a430 con 0x7f103c105920 2024-04-20T21:19:10.769 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:10.079+0000 7f1041e00700 1 -- 172.21.15.62:0/532620238 --> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7f103c19d060 con 0x7f1024038440 2024-04-20T21:19:10.769 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:10.085+0000 7f1039600700 1 -- 172.21.15.62:0/532620238 <== mgr.14120 v2:172.21.15.62:6800/4147738586 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f103c19d060 con 0x7f1024038440 2024-04-20T21:19:10.769 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:10.088+0000 7f1041e00700 1 -- 172.21.15.62:0/532620238 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7f1024038440 msgr2=0x7f102403a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:19:10.769 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:10.088+0000 7f1041e00700 1 --2- 172.21.15.62:0/532620238 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7f1024038440 0x7f102403a8f0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f1028006fd0 tx=0x7f1028006e40 comp rx=0 tx=0).stop 2024-04-20T21:19:10.769 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:10.088+0000 7f1041e00700 1 -- 172.21.15.62:0/532620238 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f103c105920 msgr2=0x7f103c1996a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:19:10.769 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:10.088+0000 7f1041e00700 1 --2- 172.21.15.62:0/532620238 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f103c105920 0x7f103c1996a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f1030011ad0 tx=0x7f1030011bb0 comp rx=0 tx=0).stop 2024-04-20T21:19:10.769 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:10.088+0000 7f1041e00700 1 -- 172.21.15.62:0/532620238 shutdown_connections 2024-04-20T21:19:10.769 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:10.088+0000 7f1041e00700 1 --2- 172.21.15.62:0/532620238 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7f1024038440 0x7f102403a8f0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:19:10.769 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:10.088+0000 7f1041e00700 1 --2- 172.21.15.62:0/532620238 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f103c105920 0x7f103c1996a0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:19:10.769 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:10.088+0000 7f1041e00700 1 -- 172.21.15.62:0/532620238 >> 172.21.15.62:0/532620238 conn(0x7f103c100fb0 msgr2=0x7f103c102690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:19:10.769 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:10.088+0000 7f1041e00700 1 -- 172.21.15.62:0/532620238 shutdown_connections 2024-04-20T21:19:10.769 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:10.088+0000 7f1041e00700 1 -- 172.21.15.62:0/532620238 wait complete. 2024-04-20T21:19:10.769 INFO:teuthology.orchestra.run.smithi062.stdout:Deploying crash service with default placement... 2024-04-20T21:19:11.391 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:11 smithi062 ceph-mon[16221]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:19:11.391 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:11 smithi062 ceph-mon[16221]: Saving service mgr spec with placement count:2 2024-04-20T21:19:12.646 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout Scheduled crash update... 2024-04-20T21:19:12.646 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:11.391+0000 7ff2ac000700 1 Processor -- start 2024-04-20T21:19:12.646 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:11.391+0000 7ff2ac000700 1 -- start start 2024-04-20T21:19:12.646 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:11.391+0000 7ff2ac000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff2a407acc0 0x7ff2a40791c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:19:12.646 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:11.391+0000 7ff2ac000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2a4079700 con 0x7ff2a407acc0 2024-04-20T21:19:12.646 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:11.392+0000 7ff2a9600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff2a407acc0 0x7ff2a40791c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:19:12.646 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:11.392+0000 7ff2a9600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff2a407acc0 0x7ff2a40791c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:39028/0 (socket says 172.21.15.62:39028) 2024-04-20T21:19:12.646 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:11.392+0000 7ff2a9600700 1 -- 172.21.15.62:0/28193987 learned_addr learned my addr 172.21.15.62:0/28193987 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:19:12.646 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:11.392+0000 7ff2a9600700 1 -- 172.21.15.62:0/28193987 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2a4079840 con 0x7ff2a407acc0 2024-04-20T21:19:12.647 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:11.393+0000 7ff2a9600700 1 --2- 172.21.15.62:0/28193987 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff2a407acc0 0x7ff2a40791c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff2980098a0 tx=0x7ff298009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9311ca429da6e11f server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:19:12.647 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:11.393+0000 7ff2a3e00700 1 -- 172.21.15.62:0/28193987 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff298004180 con 0x7ff2a407acc0 2024-04-20T21:19:12.647 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:11.393+0000 7ff2a3e00700 1 -- 172.21.15.62:0/28193987 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff2980042e0 con 0x7ff2a407acc0 2024-04-20T21:19:12.647 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:11.393+0000 7ff2a3e00700 1 -- 172.21.15.62:0/28193987 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff298011af0 con 0x7ff2a407acc0 2024-04-20T21:19:12.647 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:11.394+0000 7ff2ac000700 1 -- 172.21.15.62:0/28193987 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff2a407acc0 msgr2=0x7ff2a40791c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:19:12.647 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:11.394+0000 7ff2ac000700 1 --2- 172.21.15.62:0/28193987 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff2a407acc0 0x7ff2a40791c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff2980098a0 tx=0x7ff298009bb0 comp rx=0 tx=0).stop 2024-04-20T21:19:12.647 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:11.394+0000 7ff2ac000700 1 -- 172.21.15.62:0/28193987 shutdown_connections 2024-04-20T21:19:12.647 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:11.394+0000 7ff2ac000700 1 --2- 172.21.15.62:0/28193987 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff2a407acc0 0x7ff2a40791c0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:19:12.651 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:11.394+0000 7ff2ac000700 1 -- 172.21.15.62:0/28193987 >> 172.21.15.62:0/28193987 conn(0x7ff2a4101160 msgr2=0x7ff2a4103590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:19:12.651 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:11.394+0000 7ff2ac000700 1 -- 172.21.15.62:0/28193987 shutdown_connections 2024-04-20T21:19:12.651 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:11.394+0000 7ff2ac000700 1 -- 172.21.15.62:0/28193987 wait complete. 2024-04-20T21:19:12.651 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:11.396+0000 7ff2ac000700 1 Processor -- start 2024-04-20T21:19:12.651 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:11.396+0000 7ff2ac000700 1 -- start start 2024-04-20T21:19:12.651 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:11.396+0000 7ff2ac000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff2a407acc0 0x7ff2a419bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:19:12.651 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:11.396+0000 7ff2ac000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2a419c020 con 0x7ff2a407acc0 2024-04-20T21:19:12.651 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:11.396+0000 7ff2a9600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff2a407acc0 0x7ff2a419bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:19:12.651 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:11.397+0000 7ff2a9600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff2a407acc0 0x7ff2a419bae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:39044/0 (socket says 172.21.15.62:39044) 2024-04-20T21:19:12.651 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:11.397+0000 7ff2a9600700 1 -- 172.21.15.62:0/1073935407 learned_addr learned my addr 172.21.15.62:0/1073935407 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:19:12.652 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:11.397+0000 7ff2a9600700 1 -- 172.21.15.62:0/1073935407 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff298009550 con 0x7ff2a407acc0 2024-04-20T21:19:12.652 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:11.397+0000 7ff2a9600700 1 --2- 172.21.15.62:0/1073935407 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff2a407acc0 0x7ff2a419bae0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff298000c00 tx=0x7ff298011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:19:12.652 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:11.398+0000 7ff2a2000700 1 -- 172.21.15.62:0/1073935407 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff298003a70 con 0x7ff2a407acc0 2024-04-20T21:19:12.652 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:11.398+0000 7ff2a2000700 1 -- 172.21.15.62:0/1073935407 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff298003bd0 con 0x7ff2a407acc0 2024-04-20T21:19:12.652 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:11.398+0000 7ff2ac000700 1 -- 172.21.15.62:0/1073935407 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2a419c220 con 0x7ff2a407acc0 2024-04-20T21:19:12.652 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:11.398+0000 7ff2ac000700 1 -- 172.21.15.62:0/1073935407 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2a419c640 con 0x7ff2a407acc0 2024-04-20T21:19:12.652 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:11.398+0000 7ff2a2000700 1 -- 172.21.15.62:0/1073935407 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff2980238c0 con 0x7ff2a407acc0 2024-04-20T21:19:12.652 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:11.399+0000 7ff2a2000700 1 -- 172.21.15.62:0/1073935407 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 8) v1 ==== 45084+0+0 (secure 0 0 0) 0x7ff298023b80 con 0x7ff2a407acc0 2024-04-20T21:19:12.652 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:11.399+0000 7ff2ac000700 1 -- 172.21.15.62:0/1073935407 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff284005320 con 0x7ff2a407acc0 2024-04-20T21:19:12.652 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:11.400+0000 7ff2a2000700 1 --2- 172.21.15.62:0/1073935407 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7ff28c0383f0 0x7ff28c03a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:19:12.652 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:11.400+0000 7ff2a2000700 1 -- 172.21.15.62:0/1073935407 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7ff29801a430 con 0x7ff2a407acc0 2024-04-20T21:19:12.652 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:11.400+0000 7ff2a8c00700 1 --2- 172.21.15.62:0/1073935407 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7ff28c0383f0 0x7ff28c03a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:19:12.652 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:11.401+0000 7ff2a8c00700 1 --2- 172.21.15.62:0/1073935407 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7ff28c0383f0 0x7ff28c03a8a0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7ff290006fd0 tx=0x7ff290006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:19:12.652 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:11.405+0000 7ff2a2000700 1 -- 172.21.15.62:0/1073935407 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7ff29802b300 con 0x7ff2a407acc0 2024-04-20T21:19:12.653 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:11.653+0000 7ff2ac000700 1 -- 172.21.15.62:0/1073935407 --> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7ff284000bf0 con 0x7ff28c0383f0 2024-04-20T21:19:12.653 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:11.662+0000 7ff2a2000700 1 -- 172.21.15.62:0/1073935407 <== mgr.14120 v2:172.21.15.62:6800/4147738586 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7ff284000bf0 con 0x7ff28c0383f0 2024-04-20T21:19:12.653 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:11.666+0000 7ff2ac000700 1 -- 172.21.15.62:0/1073935407 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7ff28c0383f0 msgr2=0x7ff28c03a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:19:12.653 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:11.666+0000 7ff2ac000700 1 --2- 172.21.15.62:0/1073935407 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7ff28c0383f0 0x7ff28c03a8a0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7ff290006fd0 tx=0x7ff290006e40 comp rx=0 tx=0).stop 2024-04-20T21:19:12.653 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:11.666+0000 7ff2ac000700 1 -- 172.21.15.62:0/1073935407 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff2a407acc0 msgr2=0x7ff2a419bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:19:12.653 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:11.666+0000 7ff2ac000700 1 --2- 172.21.15.62:0/1073935407 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff2a407acc0 0x7ff2a419bae0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff298000c00 tx=0x7ff298011b50 comp rx=0 tx=0).stop 2024-04-20T21:19:12.653 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:11.666+0000 7ff2ac000700 1 -- 172.21.15.62:0/1073935407 shutdown_connections 2024-04-20T21:19:12.653 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:11.666+0000 7ff2ac000700 1 --2- 172.21.15.62:0/1073935407 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7ff28c0383f0 0x7ff28c03a8a0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:19:12.653 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:11.666+0000 7ff2ac000700 1 --2- 172.21.15.62:0/1073935407 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff2a407acc0 0x7ff2a419bae0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:19:12.653 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:11.666+0000 7ff2ac000700 1 -- 172.21.15.62:0/1073935407 >> 172.21.15.62:0/1073935407 conn(0x7ff2a4101160 msgr2=0x7ff2a4102840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:19:12.653 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:11.667+0000 7ff2ac000700 1 -- 172.21.15.62:0/1073935407 shutdown_connections 2024-04-20T21:19:12.653 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:11.667+0000 7ff2ac000700 1 -- 172.21.15.62:0/1073935407 wait complete. 2024-04-20T21:19:12.653 INFO:teuthology.orchestra.run.smithi062.stdout:Deploying ceph-exporter service with default placement... 2024-04-20T21:19:12.921 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:12 smithi062 ceph-mon[16221]: from='client.14146 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:19:12.921 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:12 smithi062 ceph-mon[16221]: Saving service crash spec with placement * 2024-04-20T21:19:12.921 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:12 smithi062 ceph-mon[16221]: from='mgr.14120 172.21.15.62:0/292923938' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:19:12.921 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:12 smithi062 ceph-mon[16221]: from='mgr.14120 172.21.15.62:0/292923938' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:19:14.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:14 smithi062 ceph-mon[16221]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:19:14.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:14 smithi062 ceph-mon[16221]: from='mgr.14120 172.21.15.62:0/292923938' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:19:14.446 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout Scheduled ceph-exporter update... 2024-04-20T21:19:14.447 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:13.247+0000 7fd7d7200700 1 Processor -- start 2024-04-20T21:19:14.447 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:13.247+0000 7fd7d7200700 1 -- start start 2024-04-20T21:19:14.447 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:13.247+0000 7fd7d7200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd7d807acc0 0x7fd7d80791c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:19:14.447 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:13.247+0000 7fd7d7200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7d8079700 con 0x7fd7d807acc0 2024-04-20T21:19:14.447 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:13.248+0000 7fd7d5e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd7d807acc0 0x7fd7d80791c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:19:14.447 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:13.248+0000 7fd7d5e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd7d807acc0 0x7fd7d80791c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:39046/0 (socket says 172.21.15.62:39046) 2024-04-20T21:19:14.447 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:13.248+0000 7fd7d5e00700 1 -- 172.21.15.62:0/709066875 learned_addr learned my addr 172.21.15.62:0/709066875 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:19:14.447 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:13.248+0000 7fd7d5e00700 1 -- 172.21.15.62:0/709066875 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7d8079840 con 0x7fd7d807acc0 2024-04-20T21:19:14.447 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:13.249+0000 7fd7d5e00700 1 --2- 172.21.15.62:0/709066875 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd7d807acc0 0x7fd7d80791c0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd7c80098a0 tx=0x7fd7c8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f864bfecdb59926f server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:19:14.447 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:13.249+0000 7fd7d4a00700 1 -- 172.21.15.62:0/709066875 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd7c8004180 con 0x7fd7d807acc0 2024-04-20T21:19:14.447 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:13.249+0000 7fd7d4a00700 1 -- 172.21.15.62:0/709066875 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fd7c80042e0 con 0x7fd7d807acc0 2024-04-20T21:19:14.447 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:13.249+0000 7fd7d4a00700 1 -- 172.21.15.62:0/709066875 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd7c8011af0 con 0x7fd7d807acc0 2024-04-20T21:19:14.448 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:13.250+0000 7fd7d7200700 1 -- 172.21.15.62:0/709066875 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd7d807acc0 msgr2=0x7fd7d80791c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:19:14.448 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:13.250+0000 7fd7d7200700 1 --2- 172.21.15.62:0/709066875 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd7d807acc0 0x7fd7d80791c0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd7c80098a0 tx=0x7fd7c8009bb0 comp rx=0 tx=0).stop 2024-04-20T21:19:14.448 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:13.250+0000 7fd7d7200700 1 -- 172.21.15.62:0/709066875 shutdown_connections 2024-04-20T21:19:14.448 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:13.250+0000 7fd7d7200700 1 --2- 172.21.15.62:0/709066875 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd7d807acc0 0x7fd7d80791c0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:19:14.448 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:13.250+0000 7fd7d7200700 1 -- 172.21.15.62:0/709066875 >> 172.21.15.62:0/709066875 conn(0x7fd7d8100f30 msgr2=0x7fd7d8103340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:19:14.448 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:13.250+0000 7fd7d7200700 1 -- 172.21.15.62:0/709066875 shutdown_connections 2024-04-20T21:19:14.448 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:13.250+0000 7fd7d7200700 1 -- 172.21.15.62:0/709066875 wait complete. 2024-04-20T21:19:14.448 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:13.251+0000 7fd7d7200700 1 Processor -- start 2024-04-20T21:19:14.448 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:13.251+0000 7fd7d7200700 1 -- start start 2024-04-20T21:19:14.448 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:13.252+0000 7fd7d7200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd7d807acc0 0x7fd7d8108cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:19:14.448 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:13.252+0000 7fd7d7200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7d8109230 con 0x7fd7d807acc0 2024-04-20T21:19:14.448 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:13.252+0000 7fd7d5e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd7d807acc0 0x7fd7d8108cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:19:14.448 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:13.252+0000 7fd7d5e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd7d807acc0 0x7fd7d8108cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:39056/0 (socket says 172.21.15.62:39056) 2024-04-20T21:19:14.449 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:13.252+0000 7fd7d5e00700 1 -- 172.21.15.62:0/3298380716 learned_addr learned my addr 172.21.15.62:0/3298380716 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:19:14.449 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:13.252+0000 7fd7d5e00700 1 -- 172.21.15.62:0/3298380716 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7c8009550 con 0x7fd7d807acc0 2024-04-20T21:19:14.449 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:13.253+0000 7fd7d5e00700 1 --2- 172.21.15.62:0/3298380716 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd7d807acc0 0x7fd7d8108cf0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd7c8000c00 tx=0x7fd7c8011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:19:14.449 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:13.253+0000 7fd7cea00700 1 -- 172.21.15.62:0/3298380716 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd7c8003a70 con 0x7fd7d807acc0 2024-04-20T21:19:14.449 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:13.253+0000 7fd7cea00700 1 -- 172.21.15.62:0/3298380716 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fd7c8003bd0 con 0x7fd7d807acc0 2024-04-20T21:19:14.449 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:13.253+0000 7fd7cea00700 1 -- 172.21.15.62:0/3298380716 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd7c80238c0 con 0x7fd7d807acc0 2024-04-20T21:19:14.449 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:13.253+0000 7fd7d7200700 1 -- 172.21.15.62:0/3298380716 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7d8109430 con 0x7fd7d807acc0 2024-04-20T21:19:14.449 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:13.253+0000 7fd7d7200700 1 -- 172.21.15.62:0/3298380716 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7d8105920 con 0x7fd7d807acc0 2024-04-20T21:19:14.449 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:13.254+0000 7fd7d7200700 1 -- 172.21.15.62:0/3298380716 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7d8062380 con 0x7fd7d807acc0 2024-04-20T21:19:14.449 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:13.255+0000 7fd7cea00700 1 -- 172.21.15.62:0/3298380716 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 8) v1 ==== 45084+0+0 (secure 0 0 0) 0x7fd7c8023b30 con 0x7fd7d807acc0 2024-04-20T21:19:14.449 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:13.255+0000 7fd7cea00700 1 --2- 172.21.15.62:0/3298380716 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7fd7bc038440 0x7fd7bc03a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:19:14.449 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:13.255+0000 7fd7cea00700 1 -- 172.21.15.62:0/3298380716 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fd7c804c210 con 0x7fd7d807acc0 2024-04-20T21:19:14.450 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:13.255+0000 7fd7d5400700 1 --2- 172.21.15.62:0/3298380716 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7fd7bc038440 0x7fd7bc03a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:19:14.450 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:13.256+0000 7fd7d5400700 1 --2- 172.21.15.62:0/3298380716 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7fd7bc038440 0x7fd7bc03a8f0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fd7c4006fd0 tx=0x7fd7c4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:19:14.450 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:13.266+0000 7fd7cea00700 1 -- 172.21.15.62:0/3298380716 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fd7c8018d20 con 0x7fd7d807acc0 2024-04-20T21:19:14.450 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:13.526+0000 7fd7d7200700 1 -- 172.21.15.62:0/3298380716 --> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "ceph-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7fd7d81063c0 con 0x7fd7bc038440 2024-04-20T21:19:14.450 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:13.537+0000 7fd7cea00700 1 -- 172.21.15.62:0/3298380716 <== mgr.14120 v2:172.21.15.62:6800/4147738586 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7fd7d81063c0 con 0x7fd7bc038440 2024-04-20T21:19:14.450 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:13.540+0000 7fd7d7200700 1 -- 172.21.15.62:0/3298380716 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7fd7bc038440 msgr2=0x7fd7bc03a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:19:14.450 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:13.540+0000 7fd7d7200700 1 --2- 172.21.15.62:0/3298380716 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7fd7bc038440 0x7fd7bc03a8f0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fd7c4006fd0 tx=0x7fd7c4006e40 comp rx=0 tx=0).stop 2024-04-20T21:19:14.450 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:13.540+0000 7fd7d7200700 1 -- 172.21.15.62:0/3298380716 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd7d807acc0 msgr2=0x7fd7d8108cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:19:14.450 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:13.540+0000 7fd7d7200700 1 --2- 172.21.15.62:0/3298380716 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd7d807acc0 0x7fd7d8108cf0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd7c8000c00 tx=0x7fd7c8011b50 comp rx=0 tx=0).stop 2024-04-20T21:19:14.450 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:13.540+0000 7fd7d7200700 1 -- 172.21.15.62:0/3298380716 shutdown_connections 2024-04-20T21:19:14.450 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:13.540+0000 7fd7d7200700 1 --2- 172.21.15.62:0/3298380716 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7fd7bc038440 0x7fd7bc03a8f0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:19:14.450 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:13.540+0000 7fd7d7200700 1 --2- 172.21.15.62:0/3298380716 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd7d807acc0 0x7fd7d8108cf0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:19:14.450 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:13.540+0000 7fd7d7200700 1 -- 172.21.15.62:0/3298380716 >> 172.21.15.62:0/3298380716 conn(0x7fd7d8100f30 msgr2=0x7fd7d8101d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:19:14.450 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:13.540+0000 7fd7d7200700 1 -- 172.21.15.62:0/3298380716 shutdown_connections 2024-04-20T21:19:14.450 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:13.540+0000 7fd7d7200700 1 -- 172.21.15.62:0/3298380716 wait complete. 2024-04-20T21:19:14.451 INFO:teuthology.orchestra.run.smithi062.stdout:Deploying prometheus service with default placement... 2024-04-20T21:19:15.267 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:15 smithi062 ceph-mon[16221]: from='client.14148 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "ceph-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:19:15.268 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:15 smithi062 ceph-mon[16221]: Saving service ceph-exporter spec with placement * 2024-04-20T21:19:15.268 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:15 smithi062 ceph-mon[16221]: from='mgr.14120 172.21.15.62:0/292923938' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:19:15.889 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout Scheduled prometheus update... 2024-04-20T21:19:15.889 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:15.004+0000 7f9535600700 1 Processor -- start 2024-04-20T21:19:15.889 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:15.004+0000 7f9535600700 1 -- start start 2024-04-20T21:19:15.893 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:15.004+0000 7f9535600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9530074490 0x7f95301086a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:19:15.894 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:15.004+0000 7f9535600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9530108be0 con 0x7f9530074490 2024-04-20T21:19:15.894 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:15.005+0000 7f952ea00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9530074490 0x7f95301086a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:19:15.894 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:15.005+0000 7f952ea00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9530074490 0x7f95301086a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:39070/0 (socket says 172.21.15.62:39070) 2024-04-20T21:19:15.894 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:15.005+0000 7f952ea00700 1 -- 172.21.15.62:0/2413116315 learned_addr learned my addr 172.21.15.62:0/2413116315 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:19:15.894 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:15.005+0000 7f952ea00700 1 -- 172.21.15.62:0/2413116315 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9530108d20 con 0x7f9530074490 2024-04-20T21:19:15.894 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:15.006+0000 7f952ea00700 1 --2- 172.21.15.62:0/2413116315 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9530074490 0x7f95301086a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f95200098a0 tx=0x7f9520009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af1bed1de86ba57b server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:19:15.894 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:15.006+0000 7f952d600700 1 -- 172.21.15.62:0/2413116315 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9520004180 con 0x7f9530074490 2024-04-20T21:19:15.894 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:15.006+0000 7f952d600700 1 -- 172.21.15.62:0/2413116315 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f95200042e0 con 0x7f9530074490 2024-04-20T21:19:15.894 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:15.006+0000 7f952d600700 1 -- 172.21.15.62:0/2413116315 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9520011af0 con 0x7f9530074490 2024-04-20T21:19:15.894 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:15.007+0000 7f9535600700 1 -- 172.21.15.62:0/2413116315 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9530074490 msgr2=0x7f95301086a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:19:15.894 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:15.007+0000 7f9535600700 1 --2- 172.21.15.62:0/2413116315 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9530074490 0x7f95301086a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f95200098a0 tx=0x7f9520009bb0 comp rx=0 tx=0).stop 2024-04-20T21:19:15.895 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:15.007+0000 7f9535600700 1 -- 172.21.15.62:0/2413116315 shutdown_connections 2024-04-20T21:19:15.895 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:15.007+0000 7f9535600700 1 --2- 172.21.15.62:0/2413116315 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9530074490 0x7f95301086a0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:19:15.895 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:15.007+0000 7f9535600700 1 -- 172.21.15.62:0/2413116315 >> 172.21.15.62:0/2413116315 conn(0x7f9530100070 msgr2=0x7f9530102480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:19:15.895 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:15.007+0000 7f9535600700 1 -- 172.21.15.62:0/2413116315 shutdown_connections 2024-04-20T21:19:15.895 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:15.007+0000 7f9535600700 1 -- 172.21.15.62:0/2413116315 wait complete. 2024-04-20T21:19:15.895 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:15.009+0000 7f9535600700 1 Processor -- start 2024-04-20T21:19:15.895 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:15.009+0000 7f9535600700 1 -- start start 2024-04-20T21:19:15.895 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:15.010+0000 7f9535600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9530105fb0 0x7f9530104640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:19:15.895 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:15.010+0000 7f9535600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95301063c0 con 0x7f9530105fb0 2024-04-20T21:19:15.895 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:15.010+0000 7f952ea00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9530105fb0 0x7f9530104640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:19:15.895 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:15.010+0000 7f952ea00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9530105fb0 0x7f9530104640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:39078/0 (socket says 172.21.15.62:39078) 2024-04-20T21:19:15.895 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:15.010+0000 7f952ea00700 1 -- 172.21.15.62:0/859914812 learned_addr learned my addr 172.21.15.62:0/859914812 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:19:15.895 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:15.010+0000 7f952ea00700 1 -- 172.21.15.62:0/859914812 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9520009550 con 0x7f9530105fb0 2024-04-20T21:19:15.895 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:15.011+0000 7f952ea00700 1 --2- 172.21.15.62:0/859914812 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9530105fb0 0x7f9530104640 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9520000c00 tx=0x7f9520011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:19:15.896 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:15.011+0000 7f9527400700 1 -- 172.21.15.62:0/859914812 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9520003a70 con 0x7f9530105fb0 2024-04-20T21:19:15.896 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:15.011+0000 7f9527400700 1 -- 172.21.15.62:0/859914812 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f9520003bd0 con 0x7f9530105fb0 2024-04-20T21:19:15.896 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:15.011+0000 7f9527400700 1 -- 172.21.15.62:0/859914812 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f95200238c0 con 0x7f9530105fb0 2024-04-20T21:19:15.896 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:15.011+0000 7f9535600700 1 -- 172.21.15.62:0/859914812 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9530104b80 con 0x7f9530105fb0 2024-04-20T21:19:15.896 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:15.011+0000 7f9535600700 1 -- 172.21.15.62:0/859914812 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9530104fa0 con 0x7f9530105fb0 2024-04-20T21:19:15.896 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:15.012+0000 7f9535600700 1 -- 172.21.15.62:0/859914812 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9530062380 con 0x7f9530105fb0 2024-04-20T21:19:15.896 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:15.013+0000 7f9527400700 1 -- 172.21.15.62:0/859914812 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 8) v1 ==== 45084+0+0 (secure 0 0 0) 0x7f9520023a20 con 0x7f9530105fb0 2024-04-20T21:19:15.896 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:15.013+0000 7f9527400700 1 --2- 172.21.15.62:0/859914812 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7f9514038440 0x7f951403a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:19:15.898 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:15.014+0000 7f9527400700 1 -- 172.21.15.62:0/859914812 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f952004c290 con 0x7f9530105fb0 2024-04-20T21:19:15.898 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:15.014+0000 7f952e000700 1 --2- 172.21.15.62:0/859914812 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7f9514038440 0x7f951403a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:19:15.898 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:15.015+0000 7f952e000700 1 --2- 172.21.15.62:0/859914812 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7f9514038440 0x7f951403a8f0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f951c006fd0 tx=0x7f951c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:19:15.898 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:15.019+0000 7f9527400700 1 -- 172.21.15.62:0/859914812 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f9520010380 con 0x7f9530105fb0 2024-04-20T21:19:15.898 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:15.267+0000 7f9535600700 1 -- 172.21.15.62:0/859914812 --> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7f9530102330 con 0x7f9514038440 2024-04-20T21:19:15.898 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:15.278+0000 7f9527400700 1 -- 172.21.15.62:0/859914812 <== mgr.14120 v2:172.21.15.62:6800/4147738586 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7f9530102330 con 0x7f9514038440 2024-04-20T21:19:15.898 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:15.281+0000 7f9535600700 1 -- 172.21.15.62:0/859914812 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7f9514038440 msgr2=0x7f951403a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:19:15.898 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:15.281+0000 7f9535600700 1 --2- 172.21.15.62:0/859914812 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7f9514038440 0x7f951403a8f0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f951c006fd0 tx=0x7f951c006e40 comp rx=0 tx=0).stop 2024-04-20T21:19:15.898 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:15.281+0000 7f9535600700 1 -- 172.21.15.62:0/859914812 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9530105fb0 msgr2=0x7f9530104640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:19:15.899 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:15.281+0000 7f9535600700 1 --2- 172.21.15.62:0/859914812 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9530105fb0 0x7f9530104640 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9520000c00 tx=0x7f9520011b50 comp rx=0 tx=0).stop 2024-04-20T21:19:15.899 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:15.281+0000 7f9535600700 1 -- 172.21.15.62:0/859914812 shutdown_connections 2024-04-20T21:19:15.899 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:15.281+0000 7f9535600700 1 --2- 172.21.15.62:0/859914812 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7f9514038440 0x7f951403a8f0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:19:15.899 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:15.281+0000 7f9535600700 1 --2- 172.21.15.62:0/859914812 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9530105fb0 0x7f9530104640 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:19:15.899 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:15.281+0000 7f9535600700 1 -- 172.21.15.62:0/859914812 >> 172.21.15.62:0/859914812 conn(0x7f9530100070 msgr2=0x7f95301016d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:19:15.899 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:15.281+0000 7f9535600700 1 -- 172.21.15.62:0/859914812 shutdown_connections 2024-04-20T21:19:15.899 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:15.282+0000 7f9535600700 1 -- 172.21.15.62:0/859914812 wait complete. 2024-04-20T21:19:15.899 INFO:teuthology.orchestra.run.smithi062.stdout:Deploying grafana service with default placement... 2024-04-20T21:19:16.513 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:16 smithi062 ceph-mon[16221]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:19:16.513 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:16 smithi062 ceph-mon[16221]: from='mgr.14120 172.21.15.62:0/292923938' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:19:17.427 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:17 smithi062 ceph-mon[16221]: from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:19:17.427 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:17 smithi062 ceph-mon[16221]: Saving service prometheus spec with placement count:1 2024-04-20T21:19:18.431 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:18 smithi062 ceph-mon[16221]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:19:18.431 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:18 smithi062 ceph-mon[16221]: from='mgr.14120 172.21.15.62:0/292923938' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:19:18.445 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout Scheduled grafana update... 2024-04-20T21:19:18.446 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:17.500+0000 7f3794e00700 1 Processor -- start 2024-04-20T21:19:18.446 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:17.500+0000 7f3794e00700 1 -- start start 2024-04-20T21:19:18.447 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:17.500+0000 7f3794e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3790107d00 0x7f3790108110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:19:18.447 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:17.501+0000 7f3794e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3790108650 con 0x7f3790107d00 2024-04-20T21:19:18.447 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:17.501+0000 7f378de00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3790107d00 0x7f3790108110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:19:18.447 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:17.501+0000 7f378de00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3790107d00 0x7f3790108110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:39084/0 (socket says 172.21.15.62:39084) 2024-04-20T21:19:18.447 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:17.501+0000 7f378de00700 1 -- 172.21.15.62:0/4201068550 learned_addr learned my addr 172.21.15.62:0/4201068550 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:19:18.448 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:17.502+0000 7f378de00700 1 -- 172.21.15.62:0/4201068550 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3790108790 con 0x7f3790107d00 2024-04-20T21:19:18.449 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:17.502+0000 7f378de00700 1 --2- 172.21.15.62:0/4201068550 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3790107d00 0x7f3790108110 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f3780009e80 tx=0x7f378000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=365c7ac5898f48a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:19:18.449 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:17.502+0000 7f378d400700 1 -- 172.21.15.62:0/4201068550 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f378000e070 con 0x7f3790107d00 2024-04-20T21:19:18.449 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:17.502+0000 7f378d400700 1 -- 172.21.15.62:0/4201068550 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f378000b8c0 con 0x7f3790107d00 2024-04-20T21:19:18.449 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:17.502+0000 7f378d400700 1 -- 172.21.15.62:0/4201068550 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f37800044d0 con 0x7f3790107d00 2024-04-20T21:19:18.449 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:17.503+0000 7f3794e00700 1 -- 172.21.15.62:0/4201068550 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3790107d00 msgr2=0x7f3790108110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:19:18.449 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:17.503+0000 7f3794e00700 1 --2- 172.21.15.62:0/4201068550 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3790107d00 0x7f3790108110 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f3780009e80 tx=0x7f378000b330 comp rx=0 tx=0).stop 2024-04-20T21:19:18.449 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:17.503+0000 7f3794e00700 1 -- 172.21.15.62:0/4201068550 shutdown_connections 2024-04-20T21:19:18.449 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:17.503+0000 7f3794e00700 1 --2- 172.21.15.62:0/4201068550 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3790107d00 0x7f3790108110 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:19:18.449 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:17.503+0000 7f3794e00700 1 -- 172.21.15.62:0/4201068550 >> 172.21.15.62:0/4201068550 conn(0x7f379007b220 msgr2=0x7f379007b640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:19:18.449 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:17.503+0000 7f3794e00700 1 -- 172.21.15.62:0/4201068550 shutdown_connections 2024-04-20T21:19:18.449 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:17.503+0000 7f3794e00700 1 -- 172.21.15.62:0/4201068550 wait complete. 2024-04-20T21:19:18.449 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:17.505+0000 7f3794e00700 1 Processor -- start 2024-04-20T21:19:18.449 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:17.505+0000 7f3794e00700 1 -- start start 2024-04-20T21:19:18.453 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:17.505+0000 7f3794e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f379019bc50 0x7f379019c060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:19:18.453 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:17.505+0000 7f3794e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f379019c5a0 con 0x7f379019bc50 2024-04-20T21:19:18.453 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:17.506+0000 7f378de00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f379019bc50 0x7f379019c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:19:18.453 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:17.506+0000 7f378de00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f379019bc50 0x7f379019c060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:39086/0 (socket says 172.21.15.62:39086) 2024-04-20T21:19:18.453 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:17.506+0000 7f378de00700 1 -- 172.21.15.62:0/2140899050 learned_addr learned my addr 172.21.15.62:0/2140899050 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:19:18.453 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:17.506+0000 7f378de00700 1 -- 172.21.15.62:0/2140899050 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3780009b30 con 0x7f379019bc50 2024-04-20T21:19:18.453 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:17.506+0000 7f378de00700 1 --2- 172.21.15.62:0/2140899050 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f379019bc50 0x7f379019c060 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f3780009fd0 tx=0x7f37800049f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:19:18.453 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:17.507+0000 7f3787400700 1 -- 172.21.15.62:0/2140899050 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f378000e040 con 0x7f379019bc50 2024-04-20T21:19:18.453 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:17.507+0000 7f3787400700 1 -- 172.21.15.62:0/2140899050 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f378000bc40 con 0x7f379019bc50 2024-04-20T21:19:18.453 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:17.507+0000 7f3794e00700 1 -- 172.21.15.62:0/2140899050 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f379019c7a0 con 0x7f379019bc50 2024-04-20T21:19:18.453 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:17.507+0000 7f3787400700 1 -- 172.21.15.62:0/2140899050 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f378001b5a0 con 0x7f379019bc50 2024-04-20T21:19:18.454 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:17.507+0000 7f3794e00700 1 -- 172.21.15.62:0/2140899050 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f379019f3f0 con 0x7f379019bc50 2024-04-20T21:19:18.454 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:17.508+0000 7f3787400700 1 -- 172.21.15.62:0/2140899050 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 8) v1 ==== 45084+0+0 (secure 0 0 0) 0x7f3780019040 con 0x7f379019bc50 2024-04-20T21:19:18.454 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:17.508+0000 7f3794e00700 1 -- 172.21.15.62:0/2140899050 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3790195b30 con 0x7f379019bc50 2024-04-20T21:19:18.454 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:17.508+0000 7f3787400700 1 --2- 172.21.15.62:0/2140899050 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7f37740383f0 0x7f377403a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:19:18.454 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:17.509+0000 7f3787400700 1 -- 172.21.15.62:0/2140899050 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f378004b8a0 con 0x7f379019bc50 2024-04-20T21:19:18.454 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:17.509+0000 7f3785400700 1 --2- 172.21.15.62:0/2140899050 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7f37740383f0 0x7f377403a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:19:18.454 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:17.509+0000 7f3785400700 1 --2- 172.21.15.62:0/2140899050 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7f37740383f0 0x7f377403a8a0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f3778006fd0 tx=0x7f3778006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:19:18.454 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:17.515+0000 7f3787400700 1 -- 172.21.15.62:0/2140899050 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f3780023be0 con 0x7f379019bc50 2024-04-20T21:19:18.454 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:17.747+0000 7f3794e00700 1 -- 172.21.15.62:0/2140899050 --> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7f3790061190 con 0x7f37740383f0 2024-04-20T21:19:18.454 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:17.759+0000 7f3787400700 1 -- 172.21.15.62:0/2140899050 <== mgr.14120 v2:172.21.15.62:6800/4147738586 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f3790061190 con 0x7f37740383f0 2024-04-20T21:19:18.454 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:17.762+0000 7f3794e00700 1 -- 172.21.15.62:0/2140899050 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7f37740383f0 msgr2=0x7f377403a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:19:18.454 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:17.762+0000 7f3794e00700 1 --2- 172.21.15.62:0/2140899050 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7f37740383f0 0x7f377403a8a0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f3778006fd0 tx=0x7f3778006e40 comp rx=0 tx=0).stop 2024-04-20T21:19:18.454 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:17.762+0000 7f3794e00700 1 -- 172.21.15.62:0/2140899050 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f379019bc50 msgr2=0x7f379019c060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:19:18.454 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:17.762+0000 7f3794e00700 1 --2- 172.21.15.62:0/2140899050 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f379019bc50 0x7f379019c060 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f3780009fd0 tx=0x7f37800049f0 comp rx=0 tx=0).stop 2024-04-20T21:19:18.455 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:17.762+0000 7f3794e00700 1 -- 172.21.15.62:0/2140899050 shutdown_connections 2024-04-20T21:19:18.455 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:17.762+0000 7f3794e00700 1 --2- 172.21.15.62:0/2140899050 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7f37740383f0 0x7f377403a8a0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:19:18.455 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:17.762+0000 7f3794e00700 1 --2- 172.21.15.62:0/2140899050 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f379019bc50 0x7f379019c060 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:19:18.455 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:17.762+0000 7f3794e00700 1 -- 172.21.15.62:0/2140899050 >> 172.21.15.62:0/2140899050 conn(0x7f379007b220 msgr2=0x7f3790106d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:19:18.455 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:17.763+0000 7f3794e00700 1 -- 172.21.15.62:0/2140899050 shutdown_connections 2024-04-20T21:19:18.455 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:17.763+0000 7f3794e00700 1 -- 172.21.15.62:0/2140899050 wait complete. 2024-04-20T21:19:18.455 INFO:teuthology.orchestra.run.smithi062.stdout:Deploying node-exporter service with default placement... 2024-04-20T21:19:19.303 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:19 smithi062 ceph-mon[16221]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:19:19.303 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:19 smithi062 ceph-mon[16221]: Saving service grafana spec with placement count:1 2024-04-20T21:19:19.952 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout Scheduled node-exporter update... 2024-04-20T21:19:19.961 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:19.045+0000 7f21c6000700 1 Processor -- start 2024-04-20T21:19:19.961 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:19.045+0000 7f21c6000700 1 -- start start 2024-04-20T21:19:19.961 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:19.045+0000 7f21c6000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f21c007bdd0 0x7f21c007a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:19:19.961 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:19.045+0000 7f21c6000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21c007a810 con 0x7f21c007bdd0 2024-04-20T21:19:19.961 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:19.046+0000 7f21bf400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f21c007bdd0 0x7f21c007a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:19:19.961 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:19.046+0000 7f21bf400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f21c007bdd0 0x7f21c007a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:39092/0 (socket says 172.21.15.62:39092) 2024-04-20T21:19:19.961 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:19.046+0000 7f21bf400700 1 -- 172.21.15.62:0/3933668085 learned_addr learned my addr 172.21.15.62:0/3933668085 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:19:19.961 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:19.046+0000 7f21bf400700 1 -- 172.21.15.62:0/3933668085 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21c007a950 con 0x7f21c007bdd0 2024-04-20T21:19:19.961 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:19.047+0000 7f21bf400700 1 --2- 172.21.15.62:0/3933668085 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f21c007bdd0 0x7f21c007a2d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f21b00098a0 tx=0x7f21b0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a412b7e80187968d server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:19:19.962 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:19.047+0000 7f21be000700 1 -- 172.21.15.62:0/3933668085 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f21b0004180 con 0x7f21c007bdd0 2024-04-20T21:19:19.962 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:19.047+0000 7f21be000700 1 -- 172.21.15.62:0/3933668085 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f21b00042e0 con 0x7f21c007bdd0 2024-04-20T21:19:19.962 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:19.047+0000 7f21be000700 1 -- 172.21.15.62:0/3933668085 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f21b0011af0 con 0x7f21c007bdd0 2024-04-20T21:19:19.962 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:19.048+0000 7f21c6000700 1 -- 172.21.15.62:0/3933668085 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f21c007bdd0 msgr2=0x7f21c007a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:19:19.962 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:19.048+0000 7f21c6000700 1 --2- 172.21.15.62:0/3933668085 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f21c007bdd0 0x7f21c007a2d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f21b00098a0 tx=0x7f21b0009bb0 comp rx=0 tx=0).stop 2024-04-20T21:19:19.962 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:19.048+0000 7f21c6000700 1 -- 172.21.15.62:0/3933668085 shutdown_connections 2024-04-20T21:19:19.962 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:19.048+0000 7f21c6000700 1 --2- 172.21.15.62:0/3933668085 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f21c007bdd0 0x7f21c007a2d0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:19:19.962 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:19.048+0000 7f21c6000700 1 -- 172.21.15.62:0/3933668085 >> 172.21.15.62:0/3933668085 conn(0x7f21c0101180 msgr2=0x7f21c0103590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:19:19.962 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:19.048+0000 7f21c6000700 1 -- 172.21.15.62:0/3933668085 shutdown_connections 2024-04-20T21:19:19.962 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:19.048+0000 7f21c6000700 1 -- 172.21.15.62:0/3933668085 wait complete. 2024-04-20T21:19:19.962 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:19.050+0000 7f21c6000700 1 Processor -- start 2024-04-20T21:19:19.962 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:19.050+0000 7f21c6000700 1 -- start start 2024-04-20T21:19:19.963 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:19.050+0000 7f21c6000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f21c007bdd0 0x7f21c019ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:19:19.963 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:19.050+0000 7f21c6000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21c019bfd0 con 0x7f21c007bdd0 2024-04-20T21:19:19.963 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:19.051+0000 7f21bf400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f21c007bdd0 0x7f21c019ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:19:19.963 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:19.051+0000 7f21bf400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f21c007bdd0 0x7f21c019ba90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:39106/0 (socket says 172.21.15.62:39106) 2024-04-20T21:19:19.963 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:19.051+0000 7f21bf400700 1 -- 172.21.15.62:0/882184367 learned_addr learned my addr 172.21.15.62:0/882184367 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:19:19.963 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:19.051+0000 7f21bf400700 1 -- 172.21.15.62:0/882184367 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21b0009550 con 0x7f21c007bdd0 2024-04-20T21:19:19.963 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:19.051+0000 7f21bf400700 1 --2- 172.21.15.62:0/882184367 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f21c007bdd0 0x7f21c019ba90 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f21b0000c00 tx=0x7f21b0011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:19:19.963 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:19.052+0000 7f21b7e00700 1 -- 172.21.15.62:0/882184367 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f21b0003a70 con 0x7f21c007bdd0 2024-04-20T21:19:19.963 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:19.052+0000 7f21b7e00700 1 -- 172.21.15.62:0/882184367 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f21b0003bd0 con 0x7f21c007bdd0 2024-04-20T21:19:19.963 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:19.052+0000 7f21b7e00700 1 -- 172.21.15.62:0/882184367 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f21b00238c0 con 0x7f21c007bdd0 2024-04-20T21:19:19.963 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:19.052+0000 7f21c6000700 1 -- 172.21.15.62:0/882184367 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21c019c1d0 con 0x7f21c007bdd0 2024-04-20T21:19:19.963 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:19.052+0000 7f21c6000700 1 -- 172.21.15.62:0/882184367 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21c019c5f0 con 0x7f21c007bdd0 2024-04-20T21:19:19.963 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:19.053+0000 7f21b7e00700 1 -- 172.21.15.62:0/882184367 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 8) v1 ==== 45084+0+0 (secure 0 0 0) 0x7f21b0023b30 con 0x7f21c007bdd0 2024-04-20T21:19:19.964 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:19.053+0000 7f21c6000700 1 -- 172.21.15.62:0/882184367 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21c010be60 con 0x7f21c007bdd0 2024-04-20T21:19:19.964 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:19.054+0000 7f21b7e00700 1 --2- 172.21.15.62:0/882184367 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7f21a4038440 0x7f21a403a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:19:19.964 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:19.054+0000 7f21b7e00700 1 -- 172.21.15.62:0/882184367 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f21b004c160 con 0x7f21c007bdd0 2024-04-20T21:19:19.964 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:19.054+0000 7f21bea00700 1 --2- 172.21.15.62:0/882184367 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7f21a4038440 0x7f21a403a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:19:19.964 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:19.055+0000 7f21bea00700 1 --2- 172.21.15.62:0/882184367 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7f21a4038440 0x7f21a403a8f0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f21ac006fd0 tx=0x7f21ac006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:19:19.964 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:19.059+0000 7f21b7e00700 1 -- 172.21.15.62:0/882184367 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f21b0010380 con 0x7f21c007bdd0 2024-04-20T21:19:19.964 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:19.292+0000 7f21c6000700 1 -- 172.21.15.62:0/882184367 --> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7f21c019f450 con 0x7f21a4038440 2024-04-20T21:19:19.964 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:19.303+0000 7f21b7e00700 1 -- 172.21.15.62:0/882184367 <== mgr.14120 v2:172.21.15.62:6800/4147738586 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f21c019f450 con 0x7f21a4038440 2024-04-20T21:19:19.964 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:19.307+0000 7f21c6000700 1 -- 172.21.15.62:0/882184367 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7f21a4038440 msgr2=0x7f21a403a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:19:19.964 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:19.307+0000 7f21c6000700 1 --2- 172.21.15.62:0/882184367 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7f21a4038440 0x7f21a403a8f0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f21ac006fd0 tx=0x7f21ac006e40 comp rx=0 tx=0).stop 2024-04-20T21:19:19.964 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:19.307+0000 7f21c6000700 1 -- 172.21.15.62:0/882184367 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f21c007bdd0 msgr2=0x7f21c019ba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:19:19.964 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:19.307+0000 7f21c6000700 1 --2- 172.21.15.62:0/882184367 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f21c007bdd0 0x7f21c019ba90 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f21b0000c00 tx=0x7f21b0011b50 comp rx=0 tx=0).stop 2024-04-20T21:19:19.964 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:19.307+0000 7f21c6000700 1 -- 172.21.15.62:0/882184367 shutdown_connections 2024-04-20T21:19:19.965 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:19.307+0000 7f21c6000700 1 --2- 172.21.15.62:0/882184367 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7f21a4038440 0x7f21a403a8f0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:19:19.965 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:19.308+0000 7f21c6000700 1 --2- 172.21.15.62:0/882184367 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f21c007bdd0 0x7f21c019ba90 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:19:19.965 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:19.308+0000 7f21c6000700 1 -- 172.21.15.62:0/882184367 >> 172.21.15.62:0/882184367 conn(0x7f21c0101180 msgr2=0x7f21c0101f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:19:19.965 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:19.308+0000 7f21c6000700 1 -- 172.21.15.62:0/882184367 shutdown_connections 2024-04-20T21:19:19.965 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:19.308+0000 7f21c6000700 1 -- 172.21.15.62:0/882184367 wait complete. 2024-04-20T21:19:19.965 INFO:teuthology.orchestra.run.smithi062.stdout:Deploying alertmanager service with default placement... 2024-04-20T21:19:20.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:20 smithi062 ceph-mon[16221]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:19:20.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:20 smithi062 ceph-mon[16221]: from='mgr.14120 172.21.15.62:0/292923938' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:19:21.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:21 smithi062 ceph-mon[16221]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:19:21.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:21 smithi062 ceph-mon[16221]: Saving service node-exporter spec with placement * 2024-04-20T21:19:21.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:21 smithi062 ceph-mon[16221]: from='mgr.14120 172.21.15.62:0/292923938' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:19:21.573 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout Scheduled alertmanager update... 2024-04-20T21:19:21.574 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:20.619+0000 7ff10f200700 1 Processor -- start 2024-04-20T21:19:21.574 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:20.620+0000 7ff10f200700 1 -- start start 2024-04-20T21:19:21.574 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:20.620+0000 7ff10f200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff11007f8c0 0x7ff11007fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:19:21.574 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:20.620+0000 7ff10f200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff110080210 con 0x7ff11007f8c0 2024-04-20T21:19:21.574 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:20.620+0000 7ff10de00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff11007f8c0 0x7ff11007fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:19:21.574 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:20.620+0000 7ff10de00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff11007f8c0 0x7ff11007fcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:39114/0 (socket says 172.21.15.62:39114) 2024-04-20T21:19:21.574 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:20.620+0000 7ff10de00700 1 -- 172.21.15.62:0/1518921392 learned_addr learned my addr 172.21.15.62:0/1518921392 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:19:21.574 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:20.621+0000 7ff10de00700 1 -- 172.21.15.62:0/1518921392 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff110080350 con 0x7ff11007f8c0 2024-04-20T21:19:21.574 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:20.621+0000 7ff10de00700 1 --2- 172.21.15.62:0/1518921392 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff11007f8c0 0x7ff11007fcd0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff1000098a0 tx=0x7ff100009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f0b5e3aca82d625 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:19:21.574 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:20.622+0000 7ff10ca00700 1 -- 172.21.15.62:0/1518921392 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff100004180 con 0x7ff11007f8c0 2024-04-20T21:19:21.574 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:20.622+0000 7ff10ca00700 1 -- 172.21.15.62:0/1518921392 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff1000042e0 con 0x7ff11007f8c0 2024-04-20T21:19:21.574 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:20.622+0000 7ff10ca00700 1 -- 172.21.15.62:0/1518921392 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff100011af0 con 0x7ff11007f8c0 2024-04-20T21:19:21.575 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:20.622+0000 7ff10f200700 1 -- 172.21.15.62:0/1518921392 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff11007f8c0 msgr2=0x7ff11007fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:19:21.575 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:20.622+0000 7ff10f200700 1 --2- 172.21.15.62:0/1518921392 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff11007f8c0 0x7ff11007fcd0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff1000098a0 tx=0x7ff100009bb0 comp rx=0 tx=0).stop 2024-04-20T21:19:21.575 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:20.622+0000 7ff10f200700 1 -- 172.21.15.62:0/1518921392 shutdown_connections 2024-04-20T21:19:21.575 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:20.622+0000 7ff10f200700 1 --2- 172.21.15.62:0/1518921392 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff11007f8c0 0x7ff11007fcd0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:19:21.575 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:20.622+0000 7ff10f200700 1 -- 172.21.15.62:0/1518921392 >> 172.21.15.62:0/1518921392 conn(0x7ff11007afe0 msgr2=0x7ff11007d3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:19:21.575 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:20.623+0000 7ff10f200700 1 -- 172.21.15.62:0/1518921392 shutdown_connections 2024-04-20T21:19:21.575 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:20.623+0000 7ff10f200700 1 -- 172.21.15.62:0/1518921392 wait complete. 2024-04-20T21:19:21.575 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:20.624+0000 7ff10f200700 1 Processor -- start 2024-04-20T21:19:21.575 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:20.624+0000 7ff10f200700 1 -- start start 2024-04-20T21:19:21.575 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:20.624+0000 7ff10f200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff11007f8c0 0x7ff11019fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:19:21.575 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:20.624+0000 7ff10f200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1101a01d0 con 0x7ff11007f8c0 2024-04-20T21:19:21.575 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:20.624+0000 7ff10de00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff11007f8c0 0x7ff11019fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:19:21.575 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:20.624+0000 7ff10de00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff11007f8c0 0x7ff11019fc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:39124/0 (socket says 172.21.15.62:39124) 2024-04-20T21:19:21.575 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:20.624+0000 7ff10de00700 1 -- 172.21.15.62:0/2175533561 learned_addr learned my addr 172.21.15.62:0/2175533561 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:19:21.575 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:20.625+0000 7ff10de00700 1 -- 172.21.15.62:0/2175533561 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff100009550 con 0x7ff11007f8c0 2024-04-20T21:19:21.576 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:20.625+0000 7ff10de00700 1 --2- 172.21.15.62:0/2175533561 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff11007f8c0 0x7ff11019fc90 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff100006ba0 tx=0x7ff100011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:19:21.576 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:20.625+0000 7ff106a00700 1 -- 172.21.15.62:0/2175533561 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff100003b00 con 0x7ff11007f8c0 2024-04-20T21:19:21.576 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:20.626+0000 7ff10f200700 1 -- 172.21.15.62:0/2175533561 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1101a03d0 con 0x7ff11007f8c0 2024-04-20T21:19:21.576 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:20.626+0000 7ff106a00700 1 -- 172.21.15.62:0/2175533561 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff10001a8e0 con 0x7ff11007f8c0 2024-04-20T21:19:21.576 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:20.626+0000 7ff106a00700 1 -- 172.21.15.62:0/2175533561 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff1000199c0 con 0x7ff11007f8c0 2024-04-20T21:19:21.576 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:20.626+0000 7ff10f200700 1 -- 172.21.15.62:0/2175533561 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1101a07f0 con 0x7ff11007f8c0 2024-04-20T21:19:21.576 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:20.627+0000 7ff106a00700 1 -- 172.21.15.62:0/2175533561 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 8) v1 ==== 45084+0+0 (secure 0 0 0) 0x7ff100019c30 con 0x7ff11007f8c0 2024-04-20T21:19:21.576 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:20.627+0000 7ff10f200700 1 -- 172.21.15.62:0/2175533561 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff110062380 con 0x7ff11007f8c0 2024-04-20T21:19:21.576 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:20.628+0000 7ff106a00700 1 --2- 172.21.15.62:0/2175533561 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7ff0f40383f0 0x7ff0f403a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:19:21.576 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:20.628+0000 7ff106a00700 1 -- 172.21.15.62:0/2175533561 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7ff10004c5a0 con 0x7ff11007f8c0 2024-04-20T21:19:21.576 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:20.628+0000 7ff10d400700 1 --2- 172.21.15.62:0/2175533561 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7ff0f40383f0 0x7ff0f403a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:19:21.576 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:20.629+0000 7ff10d400700 1 --2- 172.21.15.62:0/2175533561 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7ff0f40383f0 0x7ff0f403a8a0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ff0fc006fd0 tx=0x7ff0fc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:19:21.576 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:20.633+0000 7ff106a00700 1 -- 172.21.15.62:0/2175533561 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7ff10001aa50 con 0x7ff11007f8c0 2024-04-20T21:19:21.576 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:20.867+0000 7ff10f200700 1 -- 172.21.15.62:0/2175533561 --> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7ff1101a3640 con 0x7ff0f40383f0 2024-04-20T21:19:21.577 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:20.877+0000 7ff106a00700 1 -- 172.21.15.62:0/2175533561 <== mgr.14120 v2:172.21.15.62:6800/4147738586 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7ff1101a3640 con 0x7ff0f40383f0 2024-04-20T21:19:21.577 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:20.880+0000 7ff10f200700 1 -- 172.21.15.62:0/2175533561 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7ff0f40383f0 msgr2=0x7ff0f403a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:19:21.577 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:20.880+0000 7ff10f200700 1 --2- 172.21.15.62:0/2175533561 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7ff0f40383f0 0x7ff0f403a8a0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ff0fc006fd0 tx=0x7ff0fc006e40 comp rx=0 tx=0).stop 2024-04-20T21:19:21.577 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:20.880+0000 7ff10f200700 1 -- 172.21.15.62:0/2175533561 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff11007f8c0 msgr2=0x7ff11019fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:19:21.577 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:20.880+0000 7ff10f200700 1 --2- 172.21.15.62:0/2175533561 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff11007f8c0 0x7ff11019fc90 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff100006ba0 tx=0x7ff100011b50 comp rx=0 tx=0).stop 2024-04-20T21:19:21.577 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:20.880+0000 7ff10f200700 1 -- 172.21.15.62:0/2175533561 shutdown_connections 2024-04-20T21:19:21.577 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:20.880+0000 7ff10f200700 1 --2- 172.21.15.62:0/2175533561 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7ff0f40383f0 0x7ff0f403a8a0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:19:21.577 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:20.880+0000 7ff10f200700 1 --2- 172.21.15.62:0/2175533561 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff11007f8c0 0x7ff11019fc90 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:19:21.577 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:20.880+0000 7ff10f200700 1 -- 172.21.15.62:0/2175533561 >> 172.21.15.62:0/2175533561 conn(0x7ff11007afe0 msgr2=0x7ff11007c6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:19:21.577 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:20.881+0000 7ff10f200700 1 -- 172.21.15.62:0/2175533561 shutdown_connections 2024-04-20T21:19:21.577 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:20.881+0000 7ff10f200700 1 -- 172.21.15.62:0/2175533561 wait complete. 2024-04-20T21:19:22.432 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:22 smithi062 ceph-mon[16221]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:19:22.432 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:22 smithi062 ceph-mon[16221]: Saving service alertmanager spec with placement count:1 2024-04-20T21:19:22.432 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:22 smithi062 ceph-mon[16221]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:19:23.067 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:22.197+0000 7fdd0a400700 1 Processor -- start 2024-04-20T21:19:23.067 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:22.198+0000 7fdd0a400700 1 -- start start 2024-04-20T21:19:23.067 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:22.198+0000 7fdd0a400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd04105920 0x7fdd04105d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:19:23.067 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:22.198+0000 7fdd0a400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd04106270 con 0x7fdd04105920 2024-04-20T21:19:23.067 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:22.198+0000 7fdd09000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd04105920 0x7fdd04105d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:19:23.067 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:22.198+0000 7fdd09000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd04105920 0x7fdd04105d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41708/0 (socket says 172.21.15.62:41708) 2024-04-20T21:19:23.069 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:22.199+0000 7fdd09000700 1 -- 172.21.15.62:0/1858090426 learned_addr learned my addr 172.21.15.62:0/1858090426 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:19:23.069 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:22.199+0000 7fdd09000700 1 -- 172.21.15.62:0/1858090426 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd041063b0 con 0x7fdd04105920 2024-04-20T21:19:23.069 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:22.199+0000 7fdd09000700 1 --2- 172.21.15.62:0/1858090426 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd04105920 0x7fdd04105d30 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fdcec0098a0 tx=0x7fdcec009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ec7df946d30621f server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:19:23.069 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:22.200+0000 7fdd03e00700 1 -- 172.21.15.62:0/1858090426 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdcec004180 con 0x7fdd04105920 2024-04-20T21:19:23.069 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:22.200+0000 7fdd03e00700 1 -- 172.21.15.62:0/1858090426 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fdcec0042e0 con 0x7fdd04105920 2024-04-20T21:19:23.069 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:22.200+0000 7fdd03e00700 1 -- 172.21.15.62:0/1858090426 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdcec011af0 con 0x7fdd04105920 2024-04-20T21:19:23.070 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:22.200+0000 7fdd0a400700 1 -- 172.21.15.62:0/1858090426 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd04105920 msgr2=0x7fdd04105d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:19:23.070 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:22.200+0000 7fdd0a400700 1 --2- 172.21.15.62:0/1858090426 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd04105920 0x7fdd04105d30 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fdcec0098a0 tx=0x7fdcec009bb0 comp rx=0 tx=0).stop 2024-04-20T21:19:23.070 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:22.200+0000 7fdd0a400700 1 -- 172.21.15.62:0/1858090426 shutdown_connections 2024-04-20T21:19:23.070 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:22.200+0000 7fdd0a400700 1 --2- 172.21.15.62:0/1858090426 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd04105920 0x7fdd04105d30 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:19:23.070 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:22.200+0000 7fdd0a400700 1 -- 172.21.15.62:0/1858090426 >> 172.21.15.62:0/1858090426 conn(0x7fdd04100fb0 msgr2=0x7fdd041033c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:19:23.070 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:22.201+0000 7fdd0a400700 1 -- 172.21.15.62:0/1858090426 shutdown_connections 2024-04-20T21:19:23.070 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:22.201+0000 7fdd0a400700 1 -- 172.21.15.62:0/1858090426 wait complete. 2024-04-20T21:19:23.070 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:22.202+0000 7fdd0a400700 1 Processor -- start 2024-04-20T21:19:23.070 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:22.202+0000 7fdd0a400700 1 -- start start 2024-04-20T21:19:23.070 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:22.202+0000 7fdd0a400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd04105920 0x7fdd0407ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:19:23.070 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:22.202+0000 7fdd0a400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd0407b070 con 0x7fdd04105920 2024-04-20T21:19:23.070 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:22.203+0000 7fdd09000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd04105920 0x7fdd0407ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:19:23.073 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:22.203+0000 7fdd09000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd04105920 0x7fdd0407ab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41712/0 (socket says 172.21.15.62:41712) 2024-04-20T21:19:23.073 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:22.203+0000 7fdd09000700 1 -- 172.21.15.62:0/3685080418 learned_addr learned my addr 172.21.15.62:0/3685080418 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:19:23.073 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:22.203+0000 7fdd09000700 1 -- 172.21.15.62:0/3685080418 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcec009550 con 0x7fdd04105920 2024-04-20T21:19:23.073 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:22.203+0000 7fdd09000700 1 --2- 172.21.15.62:0/3685080418 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd04105920 0x7fdd0407ab30 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fdcec011ad0 tx=0x7fdcec011bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:19:23.073 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:22.204+0000 7fdd02000700 1 -- 172.21.15.62:0/3685080418 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdcec003ab0 con 0x7fdd04105920 2024-04-20T21:19:23.073 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:22.204+0000 7fdd02000700 1 -- 172.21.15.62:0/3685080418 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fdcec003c10 con 0x7fdd04105920 2024-04-20T21:19:23.073 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:22.204+0000 7fdd02000700 1 -- 172.21.15.62:0/3685080418 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdcec0198e0 con 0x7fdd04105920 2024-04-20T21:19:23.073 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:22.204+0000 7fdd0a400700 1 -- 172.21.15.62:0/3685080418 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd04079180 con 0x7fdd04105920 2024-04-20T21:19:23.073 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:22.204+0000 7fdd0a400700 1 -- 172.21.15.62:0/3685080418 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd040795a0 con 0x7fdd04105920 2024-04-20T21:19:23.073 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:22.205+0000 7fdd02000700 1 -- 172.21.15.62:0/3685080418 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 8) v1 ==== 45084+0+0 (secure 0 0 0) 0x7fdcec019b50 con 0x7fdd04105920 2024-04-20T21:19:23.073 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:22.205+0000 7fdd0a400700 1 -- 172.21.15.62:0/3685080418 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd04062380 con 0x7fdd04105920 2024-04-20T21:19:23.073 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:22.206+0000 7fdd02000700 1 --2- 172.21.15.62:0/3685080418 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7fdce4038440 0x7fdce403a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:19:23.074 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:22.206+0000 7fdd02000700 1 -- 172.21.15.62:0/3685080418 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fdcec04c150 con 0x7fdd04105920 2024-04-20T21:19:23.074 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:22.206+0000 7fdcfbe00700 1 --2- 172.21.15.62:0/3685080418 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7fdce4038440 0x7fdce403a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:19:23.074 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:22.207+0000 7fdcfbe00700 1 --2- 172.21.15.62:0/3685080418 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7fdce4038440 0x7fdce403a8f0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fdcf4006fd0 tx=0x7fdcf4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:19:23.074 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:22.211+0000 7fdd02000700 1 -- 172.21.15.62:0/3685080418 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fdcec01a430 con 0x7fdd04105920 2024-04-20T21:19:23.074 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:22.432+0000 7fdd0a400700 1 -- 172.21.15.62:0/3685080418 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7fdd04107820 con 0x7fdd04105920 2024-04-20T21:19:23.074 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:22.444+0000 7fdd02000700 1 -- 172.21.15.62:0/3685080418 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7fdcec018b00 con 0x7fdd04105920 2024-04-20T21:19:23.074 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:22.447+0000 7fdd0a400700 1 -- 172.21.15.62:0/3685080418 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7fdce4038440 msgr2=0x7fdce403a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:19:23.074 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:22.447+0000 7fdd0a400700 1 --2- 172.21.15.62:0/3685080418 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7fdce4038440 0x7fdce403a8f0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fdcf4006fd0 tx=0x7fdcf4006e40 comp rx=0 tx=0).stop 2024-04-20T21:19:23.074 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:22.447+0000 7fdd0a400700 1 -- 172.21.15.62:0/3685080418 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd04105920 msgr2=0x7fdd0407ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:19:23.074 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:22.447+0000 7fdd0a400700 1 --2- 172.21.15.62:0/3685080418 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd04105920 0x7fdd0407ab30 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fdcec011ad0 tx=0x7fdcec011bb0 comp rx=0 tx=0).stop 2024-04-20T21:19:23.075 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:22.447+0000 7fdd0a400700 1 -- 172.21.15.62:0/3685080418 shutdown_connections 2024-04-20T21:19:23.075 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:22.447+0000 7fdd0a400700 1 --2- 172.21.15.62:0/3685080418 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7fdce4038440 0x7fdce403a8f0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:19:23.075 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:22.447+0000 7fdd0a400700 1 --2- 172.21.15.62:0/3685080418 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd04105920 0x7fdd0407ab30 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:19:23.075 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:22.447+0000 7fdd0a400700 1 -- 172.21.15.62:0/3685080418 >> 172.21.15.62:0/3685080418 conn(0x7fdd04100fb0 msgr2=0x7fdd041026b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:19:23.075 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:22.448+0000 7fdd0a400700 1 -- 172.21.15.62:0/3685080418 shutdown_connections 2024-04-20T21:19:23.075 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:22.448+0000 7fdd0a400700 1 -- 172.21.15.62:0/3685080418 wait complete. 2024-04-20T21:19:23.718 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:23 smithi062 ceph-mon[16221]: from='client.? 172.21.15.62:0/3685080418' entity='client.admin' 2024-04-20T21:19:23.718 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:23 smithi062 ceph-mon[16221]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:19:24.607 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:23.718+0000 7f4cf8c00700 1 Processor -- start 2024-04-20T21:19:24.607 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:23.719+0000 7f4cf8c00700 1 -- start start 2024-04-20T21:19:24.607 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:23.719+0000 7f4cf8c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4cf4105b70 0x7f4cf4105f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:19:24.608 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:23.719+0000 7f4cf8c00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cf41064c0 con 0x7f4cf4105b70 2024-04-20T21:19:24.608 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:23.719+0000 7f4cf1e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4cf4105b70 0x7f4cf4105f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:19:24.608 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:23.720+0000 7f4cf1e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4cf4105b70 0x7f4cf4105f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41716/0 (socket says 172.21.15.62:41716) 2024-04-20T21:19:24.608 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:23.720+0000 7f4cf1e00700 1 -- 172.21.15.62:0/1008169639 learned_addr learned my addr 172.21.15.62:0/1008169639 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:19:24.608 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:23.720+0000 7f4cf1e00700 1 -- 172.21.15.62:0/1008169639 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cf4106600 con 0x7f4cf4105b70 2024-04-20T21:19:24.609 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:23.720+0000 7f4cf1e00700 1 --2- 172.21.15.62:0/1008169639 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4cf4105b70 0x7f4cf4105f80 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4ce40098a0 tx=0x7f4ce4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=770a3afd25268821 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:19:24.609 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:23.721+0000 7f4cf0a00700 1 -- 172.21.15.62:0/1008169639 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ce4004180 con 0x7f4cf4105b70 2024-04-20T21:19:24.609 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:23.721+0000 7f4cf0a00700 1 -- 172.21.15.62:0/1008169639 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f4ce40042e0 con 0x7f4cf4105b70 2024-04-20T21:19:24.609 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:23.721+0000 7f4cf0a00700 1 -- 172.21.15.62:0/1008169639 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ce4011af0 con 0x7f4cf4105b70 2024-04-20T21:19:24.609 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:23.721+0000 7f4cf8c00700 1 -- 172.21.15.62:0/1008169639 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4cf4105b70 msgr2=0x7f4cf4105f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:19:24.610 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:23.721+0000 7f4cf8c00700 1 --2- 172.21.15.62:0/1008169639 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4cf4105b70 0x7f4cf4105f80 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4ce40098a0 tx=0x7f4ce4009bb0 comp rx=0 tx=0).stop 2024-04-20T21:19:24.610 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:23.722+0000 7f4cf8c00700 1 -- 172.21.15.62:0/1008169639 shutdown_connections 2024-04-20T21:19:24.610 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:23.722+0000 7f4cf8c00700 1 --2- 172.21.15.62:0/1008169639 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4cf4105b70 0x7f4cf4105f80 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:19:24.610 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:23.722+0000 7f4cf8c00700 1 -- 172.21.15.62:0/1008169639 >> 172.21.15.62:0/1008169639 conn(0x7f4cf41011a0 msgr2=0x7f4cf41035f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:19:24.610 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:23.722+0000 7f4cf8c00700 1 -- 172.21.15.62:0/1008169639 shutdown_connections 2024-04-20T21:19:24.611 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:23.722+0000 7f4cf8c00700 1 -- 172.21.15.62:0/1008169639 wait complete. 2024-04-20T21:19:24.611 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:23.723+0000 7f4cf8c00700 1 Processor -- start 2024-04-20T21:19:24.611 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:23.723+0000 7f4cf8c00700 1 -- start start 2024-04-20T21:19:24.611 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:23.723+0000 7f4cf8c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4cf4105b70 0x7f4cf419ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:19:24.612 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:23.723+0000 7f4cf8c00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cf419bfb0 con 0x7f4cf4105b70 2024-04-20T21:19:24.612 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:23.723+0000 7f4cf1e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4cf4105b70 0x7f4cf419ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:19:24.612 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:23.724+0000 7f4cf1e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4cf4105b70 0x7f4cf419ba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41718/0 (socket says 172.21.15.62:41718) 2024-04-20T21:19:24.613 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:23.724+0000 7f4cf1e00700 1 -- 172.21.15.62:0/698728537 learned_addr learned my addr 172.21.15.62:0/698728537 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:19:24.613 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:23.724+0000 7f4cf1e00700 1 -- 172.21.15.62:0/698728537 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ce4009550 con 0x7f4cf4105b70 2024-04-20T21:19:24.613 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:23.724+0000 7f4cf1e00700 1 --2- 172.21.15.62:0/698728537 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4cf4105b70 0x7f4cf419ba70 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4ce4011ad0 tx=0x7f4ce4011bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:19:24.613 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:23.724+0000 7f4ceaa00700 1 -- 172.21.15.62:0/698728537 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ce4003ab0 con 0x7f4cf4105b70 2024-04-20T21:19:24.613 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:23.725+0000 7f4ceaa00700 1 -- 172.21.15.62:0/698728537 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f4ce4003c10 con 0x7f4cf4105b70 2024-04-20T21:19:24.613 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:23.725+0000 7f4ceaa00700 1 -- 172.21.15.62:0/698728537 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ce40198e0 con 0x7f4cf4105b70 2024-04-20T21:19:24.614 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:23.725+0000 7f4cf8c00700 1 -- 172.21.15.62:0/698728537 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cf419c1b0 con 0x7f4cf4105b70 2024-04-20T21:19:24.614 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:23.725+0000 7f4cf8c00700 1 -- 172.21.15.62:0/698728537 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cf419c5d0 con 0x7f4cf4105b70 2024-04-20T21:19:24.614 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:23.726+0000 7f4cf8c00700 1 -- 172.21.15.62:0/698728537 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cf4062380 con 0x7f4cf4105b70 2024-04-20T21:19:24.615 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:23.726+0000 7f4ceaa00700 1 -- 172.21.15.62:0/698728537 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 8) v1 ==== 45084+0+0 (secure 0 0 0) 0x7f4ce4019b50 con 0x7f4cf4105b70 2024-04-20T21:19:24.615 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:23.726+0000 7f4ceaa00700 1 --2- 172.21.15.62:0/698728537 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7f4cd8038440 0x7f4cd803a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:19:24.615 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:23.726+0000 7f4ceaa00700 1 -- 172.21.15.62:0/698728537 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f4ce404c1e0 con 0x7f4cf4105b70 2024-04-20T21:19:24.616 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:23.727+0000 7f4cf1400700 1 --2- 172.21.15.62:0/698728537 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7f4cd8038440 0x7f4cd803a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:19:24.616 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:23.727+0000 7f4cf1400700 1 --2- 172.21.15.62:0/698728537 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7f4cd8038440 0x7f4cd803a8f0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f4ce0006fd0 tx=0x7f4ce0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:19:24.616 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:23.731+0000 7f4ceaa00700 1 -- 172.21.15.62:0/698728537 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f4ce401a430 con 0x7f4cf4105b70 2024-04-20T21:19:24.617 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:23.950+0000 7f4cf8c00700 1 -- 172.21.15.62:0/698728537 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f4cf41079b0 con 0x7f4cf4105b70 2024-04-20T21:19:24.617 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:23.954+0000 7f4ceaa00700 1 -- 172.21.15.62:0/698728537 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7f4ce4018b00 con 0x7f4cf4105b70 2024-04-20T21:19:24.617 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:23.958+0000 7f4cf8c00700 1 -- 172.21.15.62:0/698728537 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7f4cd8038440 msgr2=0x7f4cd803a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:19:24.617 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:23.958+0000 7f4cf8c00700 1 --2- 172.21.15.62:0/698728537 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7f4cd8038440 0x7f4cd803a8f0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f4ce0006fd0 tx=0x7f4ce0006e40 comp rx=0 tx=0).stop 2024-04-20T21:19:24.618 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:23.958+0000 7f4cf8c00700 1 -- 172.21.15.62:0/698728537 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4cf4105b70 msgr2=0x7f4cf419ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:19:24.618 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:23.958+0000 7f4cf8c00700 1 --2- 172.21.15.62:0/698728537 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4cf4105b70 0x7f4cf419ba70 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4ce4011ad0 tx=0x7f4ce4011bb0 comp rx=0 tx=0).stop 2024-04-20T21:19:24.618 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:23.958+0000 7f4cf8c00700 1 -- 172.21.15.62:0/698728537 shutdown_connections 2024-04-20T21:19:24.618 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:23.958+0000 7f4cf8c00700 1 --2- 172.21.15.62:0/698728537 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7f4cd8038440 0x7f4cd803a8f0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:19:24.618 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:23.958+0000 7f4cf8c00700 1 --2- 172.21.15.62:0/698728537 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4cf4105b70 0x7f4cf419ba70 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:19:24.618 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:23.958+0000 7f4cf8c00700 1 -- 172.21.15.62:0/698728537 >> 172.21.15.62:0/698728537 conn(0x7f4cf41011a0 msgr2=0x7f4cf4101f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:19:24.618 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:23.959+0000 7f4cf8c00700 1 -- 172.21.15.62:0/698728537 shutdown_connections 2024-04-20T21:19:24.618 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:23.959+0000 7f4cf8c00700 1 -- 172.21.15.62:0/698728537 wait complete. 2024-04-20T21:19:24.618 INFO:teuthology.orchestra.run.smithi062.stdout:Enabling the dashboard module... 2024-04-20T21:19:25.164 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:24 smithi062 ceph-mon[16221]: from='client.? 172.21.15.62:0/698728537' entity='client.admin' 2024-04-20T21:19:26.283 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:25 smithi062 ceph-mon[16221]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:19:26.283 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:25 smithi062 ceph-mon[16221]: from='client.? 172.21.15.62:0/3744735270' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-20T21:19:26.647 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:25.247+0000 7fda8a200700 1 Processor -- start 2024-04-20T21:19:26.647 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:25.248+0000 7fda8a200700 1 -- start start 2024-04-20T21:19:26.647 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:25.248+0000 7fda8a200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fda84107d20 0x7fda84108130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:19:26.647 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:25.248+0000 7fda8a200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda84108670 con 0x7fda84107d20 2024-04-20T21:19:26.648 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:25.248+0000 7fda83400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fda84107d20 0x7fda84108130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:19:26.648 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:25.249+0000 7fda83400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fda84107d20 0x7fda84108130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41724/0 (socket says 172.21.15.62:41724) 2024-04-20T21:19:26.648 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:25.249+0000 7fda83400700 1 -- 172.21.15.62:0/867505493 learned_addr learned my addr 172.21.15.62:0/867505493 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:19:26.648 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:25.249+0000 7fda83400700 1 -- 172.21.15.62:0/867505493 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda841087b0 con 0x7fda84107d20 2024-04-20T21:19:26.648 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:25.249+0000 7fda83400700 1 --2- 172.21.15.62:0/867505493 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fda84107d20 0x7fda84108130 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fda740098a0 tx=0x7fda74009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e987f5b5940ca8a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:19:26.648 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:25.250+0000 7fda82000700 1 -- 172.21.15.62:0/867505493 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda74004180 con 0x7fda84107d20 2024-04-20T21:19:26.648 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:25.250+0000 7fda82000700 1 -- 172.21.15.62:0/867505493 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fda740042e0 con 0x7fda84107d20 2024-04-20T21:19:26.648 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:25.250+0000 7fda82000700 1 -- 172.21.15.62:0/867505493 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda74011af0 con 0x7fda84107d20 2024-04-20T21:19:26.648 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:25.250+0000 7fda8a200700 1 -- 172.21.15.62:0/867505493 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fda84107d20 msgr2=0x7fda84108130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:19:26.648 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:25.250+0000 7fda8a200700 1 --2- 172.21.15.62:0/867505493 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fda84107d20 0x7fda84108130 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fda740098a0 tx=0x7fda74009bb0 comp rx=0 tx=0).stop 2024-04-20T21:19:26.648 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:25.251+0000 7fda8a200700 1 -- 172.21.15.62:0/867505493 shutdown_connections 2024-04-20T21:19:26.648 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:25.251+0000 7fda8a200700 1 --2- 172.21.15.62:0/867505493 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fda84107d20 0x7fda84108130 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:19:26.648 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:25.251+0000 7fda8a200700 1 -- 172.21.15.62:0/867505493 >> 172.21.15.62:0/867505493 conn(0x7fda8407b220 msgr2=0x7fda8407b660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:19:26.648 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:25.251+0000 7fda8a200700 1 -- 172.21.15.62:0/867505493 shutdown_connections 2024-04-20T21:19:26.649 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:25.251+0000 7fda8a200700 1 -- 172.21.15.62:0/867505493 wait complete. 2024-04-20T21:19:26.649 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:25.252+0000 7fda8a200700 1 Processor -- start 2024-04-20T21:19:26.649 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:25.253+0000 7fda8a200700 1 -- start start 2024-04-20T21:19:26.649 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:25.253+0000 7fda8a200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fda84107d20 0x7fda8419baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:19:26.649 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:25.253+0000 7fda8a200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda8419c030 con 0x7fda84107d20 2024-04-20T21:19:26.649 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:25.253+0000 7fda83400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fda84107d20 0x7fda8419baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:19:26.649 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:25.253+0000 7fda83400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fda84107d20 0x7fda8419baf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41730/0 (socket says 172.21.15.62:41730) 2024-04-20T21:19:26.649 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:25.253+0000 7fda83400700 1 -- 172.21.15.62:0/3744735270 learned_addr learned my addr 172.21.15.62:0/3744735270 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:19:26.649 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:25.254+0000 7fda83400700 1 -- 172.21.15.62:0/3744735270 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda74009550 con 0x7fda84107d20 2024-04-20T21:19:26.649 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:25.254+0000 7fda83400700 1 --2- 172.21.15.62:0/3744735270 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fda84107d20 0x7fda8419baf0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fda74011670 tx=0x7fda74011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:19:26.649 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:25.254+0000 7fda7be00700 1 -- 172.21.15.62:0/3744735270 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda74003a40 con 0x7fda84107d20 2024-04-20T21:19:26.649 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:25.255+0000 7fda7be00700 1 -- 172.21.15.62:0/3744735270 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fda74003ba0 con 0x7fda84107d20 2024-04-20T21:19:26.649 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:25.255+0000 7fda7be00700 1 -- 172.21.15.62:0/3744735270 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda74024790 con 0x7fda84107d20 2024-04-20T21:19:26.649 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:25.255+0000 7fda8a200700 1 -- 172.21.15.62:0/3744735270 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda8419c230 con 0x7fda84107d20 2024-04-20T21:19:26.649 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:25.255+0000 7fda8a200700 1 -- 172.21.15.62:0/3744735270 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda8419c650 con 0x7fda84107d20 2024-04-20T21:19:26.650 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:25.256+0000 7fda7be00700 1 -- 172.21.15.62:0/3744735270 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 8) v1 ==== 45084+0+0 (secure 0 0 0) 0x7fda7401a430 con 0x7fda84107d20 2024-04-20T21:19:26.650 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:25.256+0000 7fda8a200700 1 -- 172.21.15.62:0/3744735270 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda8410c1b0 con 0x7fda84107d20 2024-04-20T21:19:26.650 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:25.257+0000 7fda7be00700 1 --2- 172.21.15.62:0/3744735270 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7fda68038440 0x7fda6803a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:19:26.650 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:25.257+0000 7fda7be00700 1 -- 172.21.15.62:0/3744735270 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fda7404c450 con 0x7fda84107d20 2024-04-20T21:19:26.650 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:25.257+0000 7fda82a00700 1 --2- 172.21.15.62:0/3744735270 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7fda68038440 0x7fda6803a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:19:26.650 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:25.258+0000 7fda82a00700 1 --2- 172.21.15.62:0/3744735270 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7fda68038440 0x7fda6803a8f0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fda70006fd0 tx=0x7fda70006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:19:26.650 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:25.262+0000 7fda7be00700 1 -- 172.21.15.62:0/3744735270 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fda74018b40 con 0x7fda84107d20 2024-04-20T21:19:26.650 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:25.548+0000 7fda8a200700 1 -- 172.21.15.62:0/3744735270 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7fda840623d0 con 0x7fda84107d20 2024-04-20T21:19:26.650 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:25.963+0000 7fda7be00700 1 -- 172.21.15.62:0/3744735270 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mgrmap(e 9) v1 ==== 45097+0+0 (secure 0 0 0) 0x7fda7401a7e0 con 0x7fda84107d20 2024-04-20T21:19:26.650 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:25.964+0000 7fda7be00700 1 -- 172.21.15.62:0/3744735270 <== mon.0 v2:172.21.15.62:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7fda7401ac50 con 0x7fda84107d20 2024-04-20T21:19:26.650 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:25.969+0000 7fda8a200700 1 -- 172.21.15.62:0/3744735270 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7fda68038440 msgr2=0x7fda6803a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:19:26.650 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:25.969+0000 7fda8a200700 1 --2- 172.21.15.62:0/3744735270 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7fda68038440 0x7fda6803a8f0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fda70006fd0 tx=0x7fda70006e40 comp rx=0 tx=0).stop 2024-04-20T21:19:26.651 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:25.969+0000 7fda8a200700 1 -- 172.21.15.62:0/3744735270 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fda84107d20 msgr2=0x7fda8419baf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:19:26.651 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:25.969+0000 7fda8a200700 1 --2- 172.21.15.62:0/3744735270 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fda84107d20 0x7fda8419baf0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fda74011670 tx=0x7fda74011b50 comp rx=0 tx=0).stop 2024-04-20T21:19:26.651 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:25.969+0000 7fda8a200700 1 -- 172.21.15.62:0/3744735270 shutdown_connections 2024-04-20T21:19:26.651 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:25.969+0000 7fda8a200700 1 --2- 172.21.15.62:0/3744735270 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7fda68038440 0x7fda6803a8f0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:19:26.651 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:25.969+0000 7fda8a200700 1 --2- 172.21.15.62:0/3744735270 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fda84107d20 0x7fda8419baf0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:19:26.651 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:25.969+0000 7fda8a200700 1 -- 172.21.15.62:0/3744735270 >> 172.21.15.62:0/3744735270 conn(0x7fda8407b220 msgr2=0x7fda84106060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:19:26.651 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:25.969+0000 7fda8a200700 1 -- 172.21.15.62:0/3744735270 shutdown_connections 2024-04-20T21:19:26.651 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:25.969+0000 7fda8a200700 1 -- 172.21.15.62:0/3744735270 wait complete. 2024-04-20T21:19:27.163 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:26 smithi062 ceph-mon[16221]: from='client.? 172.21.15.62:0/3744735270' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-20T21:19:27.163 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:26 smithi062 ceph-mon[16221]: mgrmap e9: smithi062.rqxuxi(active, since 32s) 2024-04-20T21:19:28.241 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:27 smithi062 ceph-mon[16221]: from='client.? 172.21.15.62:0/2851760486' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-20T21:19:28.248 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout { 2024-04-20T21:19:28.255 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-04-20T21:19:28.255 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-20T21:19:28.255 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "active_name": "smithi062.rqxuxi", 2024-04-20T21:19:28.255 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-20T21:19:28.255 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout } 2024-04-20T21:19:28.255 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:27.241+0000 7f2830600700 1 Processor -- start 2024-04-20T21:19:28.255 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:27.241+0000 7f2830600700 1 -- start start 2024-04-20T21:19:28.255 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:27.242+0000 7f2830600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2828106670 0x7f2828108a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:19:28.255 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:27.242+0000 7f2830600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28280744b0 con 0x7f2828106670 2024-04-20T21:19:28.255 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:27.242+0000 7f282dc00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2828106670 0x7f2828108a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:19:28.256 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:27.242+0000 7f282dc00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2828106670 0x7f2828108a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41760/0 (socket says 172.21.15.62:41760) 2024-04-20T21:19:28.256 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:27.242+0000 7f282dc00700 1 -- 172.21.15.62:0/2324995137 learned_addr learned my addr 172.21.15.62:0/2324995137 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:19:28.256 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:27.243+0000 7f282dc00700 1 -- 172.21.15.62:0/2324995137 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28280745f0 con 0x7f2828106670 2024-04-20T21:19:28.256 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:27.243+0000 7f282dc00700 1 --2- 172.21.15.62:0/2324995137 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2828106670 0x7f2828108a50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f281c0098a0 tx=0x7f281c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6f2cb14601d9e66 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:19:28.256 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:27.243+0000 7f2827e00700 1 -- 172.21.15.62:0/2324995137 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f281c004180 con 0x7f2828106670 2024-04-20T21:19:28.256 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:27.244+0000 7f2827e00700 1 -- 172.21.15.62:0/2324995137 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f281c0042e0 con 0x7f2828106670 2024-04-20T21:19:28.256 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:27.244+0000 7f2827e00700 1 -- 172.21.15.62:0/2324995137 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f281c011af0 con 0x7f2828106670 2024-04-20T21:19:28.256 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:27.244+0000 7f2830600700 1 -- 172.21.15.62:0/2324995137 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2828106670 msgr2=0x7f2828108a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:19:28.256 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:27.244+0000 7f2830600700 1 --2- 172.21.15.62:0/2324995137 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2828106670 0x7f2828108a50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f281c0098a0 tx=0x7f281c009bb0 comp rx=0 tx=0).stop 2024-04-20T21:19:28.256 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:27.245+0000 7f2830600700 1 -- 172.21.15.62:0/2324995137 shutdown_connections 2024-04-20T21:19:28.256 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:27.245+0000 7f2830600700 1 --2- 172.21.15.62:0/2324995137 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2828106670 0x7f2828108a50 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:19:28.256 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:27.245+0000 7f2830600700 1 -- 172.21.15.62:0/2324995137 >> 172.21.15.62:0/2324995137 conn(0x7f2828100070 msgr2=0x7f2828102480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:19:28.256 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:27.245+0000 7f2830600700 1 -- 172.21.15.62:0/2324995137 shutdown_connections 2024-04-20T21:19:28.256 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:27.245+0000 7f2830600700 1 -- 172.21.15.62:0/2324995137 wait complete. 2024-04-20T21:19:28.256 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:27.246+0000 7f2830600700 1 Processor -- start 2024-04-20T21:19:28.257 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:27.246+0000 7f2830600700 1 -- start start 2024-04-20T21:19:28.257 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:27.247+0000 7f2830600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2828106670 0x7f282819ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:19:28.257 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:27.247+0000 7f2830600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f282819bfc0 con 0x7f2828106670 2024-04-20T21:19:28.257 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:27.247+0000 7f282dc00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2828106670 0x7f282819ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:19:28.257 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:27.247+0000 7f282dc00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2828106670 0x7f282819ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41770/0 (socket says 172.21.15.62:41770) 2024-04-20T21:19:28.257 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:27.247+0000 7f282dc00700 1 -- 172.21.15.62:0/2851760486 learned_addr learned my addr 172.21.15.62:0/2851760486 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:19:28.257 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:27.247+0000 7f282dc00700 1 -- 172.21.15.62:0/2851760486 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f281c009550 con 0x7f2828106670 2024-04-20T21:19:28.257 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:27.248+0000 7f282dc00700 1 --2- 172.21.15.62:0/2851760486 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2828106670 0x7f282819ba80 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f281c000c00 tx=0x7f281c011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:19:28.257 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:27.248+0000 7f2826000700 1 -- 172.21.15.62:0/2851760486 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f281c003a70 con 0x7f2828106670 2024-04-20T21:19:28.257 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:27.248+0000 7f2826000700 1 -- 172.21.15.62:0/2851760486 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f281c003bd0 con 0x7f2828106670 2024-04-20T21:19:28.257 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:27.248+0000 7f2830600700 1 -- 172.21.15.62:0/2851760486 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f282819c1c0 con 0x7f2828106670 2024-04-20T21:19:28.257 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:27.248+0000 7f2826000700 1 -- 172.21.15.62:0/2851760486 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f281c0238c0 con 0x7f2828106670 2024-04-20T21:19:28.257 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:27.248+0000 7f2830600700 1 -- 172.21.15.62:0/2851760486 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f282819c5e0 con 0x7f2828106670 2024-04-20T21:19:28.257 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:27.249+0000 7f2826000700 1 -- 172.21.15.62:0/2851760486 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 9) v1 ==== 45097+0+0 (secure 0 0 0) 0x7f281c023b80 con 0x7f2828106670 2024-04-20T21:19:28.257 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:27.250+0000 7f281b400700 1 -- 172.21.15.62:0/2851760486 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28040052f0 con 0x7f2828106670 2024-04-20T21:19:28.258 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:27.250+0000 7f2826000700 1 --2- 172.21.15.62:0/2851760486 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7f2810038440 0x7f281003a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:19:28.258 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:27.250+0000 7f2826000700 1 -- 172.21.15.62:0/2851760486 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f281c04c400 con 0x7f2828106670 2024-04-20T21:19:28.258 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:27.250+0000 7f282d200700 1 -- 172.21.15.62:0/2851760486 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7f2810038440 msgr2=0x7f281003a8f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.62:6800/4147738586 2024-04-20T21:19:28.258 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:27.250+0000 7f282d200700 1 --2- 172.21.15.62:0/2851760486 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7f2810038440 0x7f281003a8f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-20T21:19:28.258 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:27.256+0000 7f2826000700 1 -- 172.21.15.62:0/2851760486 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f281c010970 con 0x7f2828106670 2024-04-20T21:19:28.258 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:27.451+0000 7f282d200700 1 -- 172.21.15.62:0/2851760486 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7f2810038440 msgr2=0x7f281003a8f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.62:6800/4147738586 2024-04-20T21:19:28.258 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:27.451+0000 7f282d200700 1 --2- 172.21.15.62:0/2851760486 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7f2810038440 0x7f281003a8f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-20T21:19:28.258 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:27.535+0000 7f281b400700 1 -- 172.21.15.62:0/2851760486 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f28040061d0 con 0x7f2828106670 2024-04-20T21:19:28.258 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:27.535+0000 7f2826000700 1 -- 172.21.15.62:0/2851760486 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+103 (secure 0 0 0) 0x7f281c02c350 con 0x7f2828106670 2024-04-20T21:19:28.258 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:27.538+0000 7f281b400700 1 -- 172.21.15.62:0/2851760486 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7f2810038440 msgr2=0x7f281003a8f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-20T21:19:28.258 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:27.538+0000 7f281b400700 1 --2- 172.21.15.62:0/2851760486 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7f2810038440 0x7f281003a8f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:19:28.258 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:27.538+0000 7f281b400700 1 -- 172.21.15.62:0/2851760486 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2828106670 msgr2=0x7f282819ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:19:28.258 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:27.538+0000 7f281b400700 1 --2- 172.21.15.62:0/2851760486 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2828106670 0x7f282819ba80 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f281c000c00 tx=0x7f281c011b50 comp rx=0 tx=0).stop 2024-04-20T21:19:28.258 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:27.538+0000 7f281b400700 1 -- 172.21.15.62:0/2851760486 shutdown_connections 2024-04-20T21:19:28.259 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:27.538+0000 7f281b400700 1 --2- 172.21.15.62:0/2851760486 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7f2810038440 0x7f281003a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:19:28.259 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:27.538+0000 7f281b400700 1 --2- 172.21.15.62:0/2851760486 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2828106670 0x7f282819ba80 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:19:28.259 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:27.538+0000 7f281b400700 1 -- 172.21.15.62:0/2851760486 >> 172.21.15.62:0/2851760486 conn(0x7f2828100070 msgr2=0x7f2828101760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:19:28.259 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:27.538+0000 7f281b400700 1 -- 172.21.15.62:0/2851760486 shutdown_connections 2024-04-20T21:19:28.259 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:27.538+0000 7f281b400700 1 -- 172.21.15.62:0/2851760486 wait complete. 2024-04-20T21:19:28.259 INFO:teuthology.orchestra.run.smithi062.stdout:Waiting for the mgr to restart... 2024-04-20T21:19:28.259 INFO:teuthology.orchestra.run.smithi062.stdout:Waiting for mgr epoch 9... 2024-04-20T21:19:35.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:35 smithi062 ceph-mon[16221]: Active manager daemon smithi062.rqxuxi restarted 2024-04-20T21:19:35.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:35 smithi062 ceph-mon[16221]: Activating manager daemon smithi062.rqxuxi 2024-04-20T21:19:35.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:35 smithi062 ceph-mon[16221]: osdmap e3: 0 total, 0 up, 0 in 2024-04-20T21:19:35.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:35 smithi062 ceph-mon[16221]: mgrmap e10: smithi062.rqxuxi(active, starting, since 0.00864641s) 2024-04-20T21:19:35.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:35 smithi062 ceph-mon[16221]: from='mgr.14164 172.21.15.62:0/570876096' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mon metadata", "id": "smithi062"}]: dispatch 2024-04-20T21:19:35.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:35 smithi062 ceph-mon[16221]: from='mgr.14164 172.21.15.62:0/570876096' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mgr metadata", "who": "smithi062.rqxuxi", "id": "smithi062.rqxuxi"}]: dispatch 2024-04-20T21:19:35.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:35 smithi062 ceph-mon[16221]: from='mgr.14164 172.21.15.62:0/570876096' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-20T21:19:35.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:35 smithi062 ceph-mon[16221]: from='mgr.14164 172.21.15.62:0/570876096' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-20T21:19:35.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:35 smithi062 ceph-mon[16221]: from='mgr.14164 172.21.15.62:0/570876096' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-20T21:19:35.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:35 smithi062 ceph-mon[16221]: Manager daemon smithi062.rqxuxi is now available 2024-04-20T21:19:36.270 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:36 smithi062 ceph-mon[16221]: from='mgr.14164 172.21.15.62:0/570876096' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:19:36.270 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:36 smithi062 ceph-mon[16221]: from='mgr.14164 172.21.15.62:0/570876096' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:19:36.271 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:36 smithi062 ceph-mon[16221]: from='mgr.14164 172.21.15.62:0/570876096' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi062.rqxuxi/mirror_snapshot_schedule"}]: dispatch 2024-04-20T21:19:36.271 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:36 smithi062 ceph-mon[16221]: from='mgr.14164 172.21.15.62:0/570876096' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi062.rqxuxi/trash_purge_schedule"}]: dispatch 2024-04-20T21:19:36.271 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:36 smithi062 ceph-mon[16221]: mgrmap e11: smithi062.rqxuxi(active, since 1.00996s) 2024-04-20T21:19:37.250 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:37 smithi062 ceph-mon[16221]: from='client.14168 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-20T21:19:37.250 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:37 smithi062 ceph-mon[16221]: from='client.14168 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-20T21:19:37.268 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout { 2024-04-20T21:19:37.268 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-04-20T21:19:37.268 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-20T21:19:37.268 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout } 2024-04-20T21:19:37.269 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:28.830+0000 7f67f9200700 1 Processor -- start 2024-04-20T21:19:37.269 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:28.831+0000 7f67f9200700 1 -- start start 2024-04-20T21:19:37.269 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:28.831+0000 7f67f9200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f67f407ace0 0x7f67f40791e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:19:37.269 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:28.831+0000 7f67f9200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67f4079720 con 0x7f67f407ace0 2024-04-20T21:19:37.269 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:28.831+0000 7f67f3400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f67f407ace0 0x7f67f40791e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:19:37.269 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:28.831+0000 7f67f3400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f67f407ace0 0x7f67f40791e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41778/0 (socket says 172.21.15.62:41778) 2024-04-20T21:19:37.269 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:28.831+0000 7f67f3400700 1 -- 172.21.15.62:0/4193529318 learned_addr learned my addr 172.21.15.62:0/4193529318 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:19:37.269 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:28.832+0000 7f67f3400700 1 -- 172.21.15.62:0/4193529318 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67f4079860 con 0x7f67f407ace0 2024-04-20T21:19:37.269 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:28.832+0000 7f67f3400700 1 --2- 172.21.15.62:0/4193529318 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f67f407ace0 0x7f67f40791e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f67e40098a0 tx=0x7f67e4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e415bff10f864fc server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:19:37.269 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:28.833+0000 7f67f2000700 1 -- 172.21.15.62:0/4193529318 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f67e4004180 con 0x7f67f407ace0 2024-04-20T21:19:37.269 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:28.833+0000 7f67f2000700 1 -- 172.21.15.62:0/4193529318 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f67e40042e0 con 0x7f67f407ace0 2024-04-20T21:19:37.269 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:28.833+0000 7f67f2000700 1 -- 172.21.15.62:0/4193529318 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f67e4011af0 con 0x7f67f407ace0 2024-04-20T21:19:37.269 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:28.833+0000 7f67f9200700 1 -- 172.21.15.62:0/4193529318 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f67f407ace0 msgr2=0x7f67f40791e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:19:37.269 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:28.833+0000 7f67f9200700 1 --2- 172.21.15.62:0/4193529318 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f67f407ace0 0x7f67f40791e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f67e40098a0 tx=0x7f67e4009bb0 comp rx=0 tx=0).stop 2024-04-20T21:19:37.269 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:28.834+0000 7f67f9200700 1 -- 172.21.15.62:0/4193529318 shutdown_connections 2024-04-20T21:19:37.270 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:28.834+0000 7f67f9200700 1 --2- 172.21.15.62:0/4193529318 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f67f407ace0 0x7f67f40791e0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:19:37.270 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:28.834+0000 7f67f9200700 1 -- 172.21.15.62:0/4193529318 >> 172.21.15.62:0/4193529318 conn(0x7f67f4100fb0 msgr2=0x7f67f41033c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:19:37.270 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:28.834+0000 7f67f9200700 1 -- 172.21.15.62:0/4193529318 shutdown_connections 2024-04-20T21:19:37.270 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:28.834+0000 7f67f9200700 1 -- 172.21.15.62:0/4193529318 wait complete. 2024-04-20T21:19:37.270 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:28.835+0000 7f67f9200700 1 Processor -- start 2024-04-20T21:19:37.270 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:28.835+0000 7f67f9200700 1 -- start start 2024-04-20T21:19:37.270 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:28.835+0000 7f67f9200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f67f407ace0 0x7f67f419b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:19:37.270 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:28.835+0000 7f67f9200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67f419be30 con 0x7f67f407ace0 2024-04-20T21:19:37.270 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:28.836+0000 7f67f3400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f67f407ace0 0x7f67f419b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:19:37.270 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:28.836+0000 7f67f3400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f67f407ace0 0x7f67f419b8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41788/0 (socket says 172.21.15.62:41788) 2024-04-20T21:19:37.270 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:28.836+0000 7f67f3400700 1 -- 172.21.15.62:0/3042695234 learned_addr learned my addr 172.21.15.62:0/3042695234 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:19:37.270 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:28.836+0000 7f67f3400700 1 -- 172.21.15.62:0/3042695234 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67e4009550 con 0x7f67f407ace0 2024-04-20T21:19:37.270 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:28.836+0000 7f67f3400700 1 --2- 172.21.15.62:0/3042695234 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f67f407ace0 0x7f67f419b8f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f67e4006ba0 tx=0x7f67e4011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:19:37.270 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:28.836+0000 7f67ebe00700 1 -- 172.21.15.62:0/3042695234 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f67e4003b00 con 0x7f67f407ace0 2024-04-20T21:19:37.271 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:28.837+0000 7f67ebe00700 1 -- 172.21.15.62:0/3042695234 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f67e4003c60 con 0x7f67f407ace0 2024-04-20T21:19:37.271 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:28.837+0000 7f67ebe00700 1 -- 172.21.15.62:0/3042695234 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f67e4019820 con 0x7f67f407ace0 2024-04-20T21:19:37.271 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:28.837+0000 7f67f9200700 1 -- 172.21.15.62:0/3042695234 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67f419c030 con 0x7f67f407ace0 2024-04-20T21:19:37.271 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:28.837+0000 7f67f9200700 1 -- 172.21.15.62:0/3042695234 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67f419c450 con 0x7f67f407ace0 2024-04-20T21:19:37.271 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:28.838+0000 7f67ebe00700 1 -- 172.21.15.62:0/3042695234 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 9) v1 ==== 45097+0+0 (secure 0 0 0) 0x7f67e401ac90 con 0x7f67f407ace0 2024-04-20T21:19:37.271 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:28.838+0000 7f67ebe00700 1 --2- 172.21.15.62:0/3042695234 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7f67d80383f0 0x7f67d803a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:19:37.271 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:28.838+0000 7f67ebe00700 1 -- 172.21.15.62:0/3042695234 --> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f67d803afb0 con 0x7f67d80383f0 2024-04-20T21:19:37.271 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:28.838+0000 7f67ebe00700 1 -- 172.21.15.62:0/3042695234 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f67e404c590 con 0x7f67f407ace0 2024-04-20T21:19:37.271 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:28.838+0000 7f67f2a00700 1 -- 172.21.15.62:0/3042695234 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7f67d80383f0 msgr2=0x7f67d803a8a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.62:6800/4147738586 2024-04-20T21:19:37.271 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:28.838+0000 7f67f2a00700 1 --2- 172.21.15.62:0/3042695234 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7f67d80383f0 0x7f67d803a8a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-20T21:19:37.271 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:29.039+0000 7f67f2a00700 1 -- 172.21.15.62:0/3042695234 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7f67d80383f0 msgr2=0x7f67d803a8a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.62:6800/4147738586 2024-04-20T21:19:37.271 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:29.039+0000 7f67f2a00700 1 --2- 172.21.15.62:0/3042695234 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7f67d80383f0 0x7f67d803a8a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-20T21:19:37.271 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:29.440+0000 7f67f2a00700 1 -- 172.21.15.62:0/3042695234 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7f67d80383f0 msgr2=0x7f67d803a8a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.62:6800/4147738586 2024-04-20T21:19:37.271 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:29.440+0000 7f67f2a00700 1 --2- 172.21.15.62:0/3042695234 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7f67d80383f0 0x7f67d803a8a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-20T21:19:37.272 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:30.241+0000 7f67f2a00700 1 -- 172.21.15.62:0/3042695234 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7f67d80383f0 msgr2=0x7f67d803a8a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.62:6800/4147738586 2024-04-20T21:19:37.272 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:30.241+0000 7f67f2a00700 1 --2- 172.21.15.62:0/3042695234 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7f67d80383f0 0x7f67d803a8a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-20T21:19:37.272 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:31.843+0000 7f67f2a00700 1 -- 172.21.15.62:0/3042695234 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7f67d80383f0 msgr2=0x7f67d803a8a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.62:6800/4147738586 2024-04-20T21:19:37.272 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:31.843+0000 7f67f2a00700 1 --2- 172.21.15.62:0/3042695234 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7f67d80383f0 0x7f67d803a8a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-20T21:19:37.272 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:34.971+0000 7f67ebe00700 1 -- 172.21.15.62:0/3042695234 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mgrmap(e 10) v1 ==== 44864+0+0 (secure 0 0 0) 0x7f67e400fda0 con 0x7f67f407ace0 2024-04-20T21:19:37.272 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:34.971+0000 7f67ebe00700 1 -- 172.21.15.62:0/3042695234 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7f67d80383f0 msgr2=0x7f67d803a8a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-20T21:19:37.272 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:34.971+0000 7f67ebe00700 1 --2- 172.21.15.62:0/3042695234 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7f67d80383f0 0x7f67d803a8a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:19:37.272 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:35.972+0000 7f67ebe00700 1 -- 172.21.15.62:0/3042695234 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mgrmap(e 11) v1 ==== 44991+0+0 (secure 0 0 0) 0x7f67e404da60 con 0x7f67f407ace0 2024-04-20T21:19:37.272 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:35.973+0000 7f67ebe00700 1 --2- 172.21.15.62:0/3042695234 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7f67d803be50 0x7f67d803e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:19:37.272 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:35.973+0000 7f67ebe00700 1 -- 172.21.15.62:0/3042695234 --> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f67d803afb0 con 0x7f67d803be50 2024-04-20T21:19:37.272 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:35.974+0000 7f67f2a00700 1 --2- 172.21.15.62:0/3042695234 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7f67d803be50 0x7f67d803e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:19:37.272 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:35.976+0000 7f67f2a00700 1 --2- 172.21.15.62:0/3042695234 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7f67d803be50 0x7f67d803e230 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f67e0003a10 tx=0x7f67e00092b0 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:19:37.272 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:35.977+0000 7f67ebe00700 1 -- 172.21.15.62:0/3042695234 <== mgr.14164 v2:172.21.15.62:6800/831506538 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7f67d803afb0 con 0x7f67d803be50 2024-04-20T21:19:37.272 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:35.988+0000 7f67f9200700 1 -- 172.21.15.62:0/3042695234 --> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f67f419cc30 con 0x7f67d803be50 2024-04-20T21:19:37.273 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:35.989+0000 7f67ebe00700 1 -- 172.21.15.62:0/3042695234 <== mgr.14164 v2:172.21.15.62:6800/831506538 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f67f419cc30 con 0x7f67d803be50 2024-04-20T21:19:37.273 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:35.990+0000 7f67f9200700 1 -- 172.21.15.62:0/3042695234 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7f67d803be50 msgr2=0x7f67d803e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:19:37.273 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:35.990+0000 7f67f9200700 1 --2- 172.21.15.62:0/3042695234 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7f67d803be50 0x7f67d803e230 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f67e0003a10 tx=0x7f67e00092b0 comp rx=0 tx=0).stop 2024-04-20T21:19:37.273 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:35.990+0000 7f67f9200700 1 -- 172.21.15.62:0/3042695234 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f67f407ace0 msgr2=0x7f67f419b8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:19:37.273 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:35.990+0000 7f67f9200700 1 --2- 172.21.15.62:0/3042695234 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f67f407ace0 0x7f67f419b8f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f67e4006ba0 tx=0x7f67e4011b50 comp rx=0 tx=0).stop 2024-04-20T21:19:37.273 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:35.990+0000 7f67f9200700 1 -- 172.21.15.62:0/3042695234 shutdown_connections 2024-04-20T21:19:37.273 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:35.990+0000 7f67f9200700 1 --2- 172.21.15.62:0/3042695234 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7f67d803be50 0x7f67d803e230 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:19:37.273 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:35.990+0000 7f67f9200700 1 --2- 172.21.15.62:0/3042695234 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f67f407ace0 0x7f67f419b8f0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:19:37.273 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:35.990+0000 7f67f9200700 1 --2- 172.21.15.62:0/3042695234 >> [v2:172.21.15.62:6800/4147738586,v1:172.21.15.62:6801/4147738586] conn(0x7f67d80383f0 0x7f67d803a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:19:37.273 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:35.990+0000 7f67f9200700 1 -- 172.21.15.62:0/3042695234 >> 172.21.15.62:0/3042695234 conn(0x7f67f4100fb0 msgr2=0x7f67f41026b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:19:37.273 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:35.990+0000 7f67f9200700 1 -- 172.21.15.62:0/3042695234 shutdown_connections 2024-04-20T21:19:37.273 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:35.990+0000 7f67f9200700 1 -- 172.21.15.62:0/3042695234 wait complete. 2024-04-20T21:19:37.273 INFO:teuthology.orchestra.run.smithi062.stdout:mgr epoch 9 is available 2024-04-20T21:19:37.273 INFO:teuthology.orchestra.run.smithi062.stdout:Generating a dashboard self-signed certificate... 2024-04-20T21:19:38.149 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:38 smithi062 ceph-mon[16221]: [20/Apr/2024:21:19:36] ENGINE Bus STARTING 2024-04-20T21:19:38.150 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:38 smithi062 ceph-mon[16221]: [20/Apr/2024:21:19:36] ENGINE Serving on http://172.21.15.62:8765 2024-04-20T21:19:38.150 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:38 smithi062 ceph-mon[16221]: [20/Apr/2024:21:19:36] ENGINE Serving on https://172.21.15.62:7150 2024-04-20T21:19:38.150 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:38 smithi062 ceph-mon[16221]: [20/Apr/2024:21:19:36] ENGINE Bus STARTED 2024-04-20T21:19:38.150 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:38 smithi062 ceph-mon[16221]: from='mgr.14164 172.21.15.62:0/570876096' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:19:38.150 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:38 smithi062 ceph-mon[16221]: mgrmap e12: smithi062.rqxuxi(active, since 2s) 2024-04-20T21:19:38.879 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-04-20T21:19:38.879 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:37.861+0000 7f9639800700 1 Processor -- start 2024-04-20T21:19:38.880 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:37.862+0000 7f9639800700 1 -- start start 2024-04-20T21:19:38.880 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:37.862+0000 7f9639800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9634074490 0x7f963410aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:19:38.880 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:37.862+0000 7f9639800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9634106490 con 0x7f9634074490 2024-04-20T21:19:38.880 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:37.863+0000 7f9632a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9634074490 0x7f963410aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:19:38.880 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:37.863+0000 7f9632a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9634074490 0x7f963410aaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47416/0 (socket says 172.21.15.62:47416) 2024-04-20T21:19:38.880 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:37.863+0000 7f9632a00700 1 -- 172.21.15.62:0/1629220840 learned_addr learned my addr 172.21.15.62:0/1629220840 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:19:38.880 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:37.863+0000 7f9632a00700 1 -- 172.21.15.62:0/1629220840 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f963410afe0 con 0x7f9634074490 2024-04-20T21:19:38.880 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:37.864+0000 7f9632a00700 1 --2- 172.21.15.62:0/1629220840 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9634074490 0x7f963410aaa0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f96240098a0 tx=0x7f9624009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=730f753c2b77559c server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:19:38.880 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:37.864+0000 7f9632000700 1 -- 172.21.15.62:0/1629220840 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9624004180 con 0x7f9634074490 2024-04-20T21:19:38.880 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:37.864+0000 7f9632000700 1 -- 172.21.15.62:0/1629220840 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f96240042e0 con 0x7f9634074490 2024-04-20T21:19:38.880 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:37.864+0000 7f9632000700 1 -- 172.21.15.62:0/1629220840 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9624011af0 con 0x7f9634074490 2024-04-20T21:19:38.880 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:37.865+0000 7f9639800700 1 -- 172.21.15.62:0/1629220840 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9634074490 msgr2=0x7f963410aaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:19:38.880 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:37.865+0000 7f9639800700 1 --2- 172.21.15.62:0/1629220840 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9634074490 0x7f963410aaa0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f96240098a0 tx=0x7f9624009bb0 comp rx=0 tx=0).stop 2024-04-20T21:19:38.880 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:37.865+0000 7f9639800700 1 -- 172.21.15.62:0/1629220840 shutdown_connections 2024-04-20T21:19:38.880 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:37.865+0000 7f9639800700 1 --2- 172.21.15.62:0/1629220840 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9634074490 0x7f963410aaa0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:19:38.881 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:37.865+0000 7f9639800700 1 -- 172.21.15.62:0/1629220840 >> 172.21.15.62:0/1629220840 conn(0x7f9634100070 msgr2=0x7f9634102480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:19:38.881 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:37.865+0000 7f9639800700 1 -- 172.21.15.62:0/1629220840 shutdown_connections 2024-04-20T21:19:38.881 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:37.865+0000 7f9639800700 1 -- 172.21.15.62:0/1629220840 wait complete. 2024-04-20T21:19:38.881 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:37.866+0000 7f9639800700 1 Processor -- start 2024-04-20T21:19:38.881 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:37.867+0000 7f9639800700 1 -- start start 2024-04-20T21:19:38.881 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:37.867+0000 7f9639800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f963419ba50 0x7f963419be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:19:38.881 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:37.867+0000 7f9639800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f963419c3a0 con 0x7f963419ba50 2024-04-20T21:19:38.881 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:37.867+0000 7f9632a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f963419ba50 0x7f963419be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:19:38.881 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:37.867+0000 7f9632a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f963419ba50 0x7f963419be60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47422/0 (socket says 172.21.15.62:47422) 2024-04-20T21:19:38.881 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:37.867+0000 7f9632a00700 1 -- 172.21.15.62:0/1005552158 learned_addr learned my addr 172.21.15.62:0/1005552158 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:19:38.881 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:37.868+0000 7f9632a00700 1 -- 172.21.15.62:0/1005552158 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9624009550 con 0x7f963419ba50 2024-04-20T21:19:38.881 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:37.868+0000 7f9632a00700 1 --2- 172.21.15.62:0/1005552158 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f963419ba50 0x7f963419be60 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f9624000c00 tx=0x7f9624011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:19:38.881 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:37.868+0000 7f962b400700 1 -- 172.21.15.62:0/1005552158 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9624003a70 con 0x7f963419ba50 2024-04-20T21:19:38.881 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:37.868+0000 7f962b400700 1 -- 172.21.15.62:0/1005552158 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f9624003bd0 con 0x7f963419ba50 2024-04-20T21:19:38.881 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:37.868+0000 7f9639800700 1 -- 172.21.15.62:0/1005552158 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f963419c5a0 con 0x7f963419ba50 2024-04-20T21:19:38.881 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:37.868+0000 7f962b400700 1 -- 172.21.15.62:0/1005552158 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f96240238c0 con 0x7f963419ba50 2024-04-20T21:19:38.882 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:37.869+0000 7f9639800700 1 -- 172.21.15.62:0/1005552158 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f963419f200 con 0x7f963419ba50 2024-04-20T21:19:38.882 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:37.870+0000 7f9628c00700 1 -- 172.21.15.62:0/1005552158 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9634074490 con 0x7f963419ba50 2024-04-20T21:19:38.882 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:37.870+0000 7f962b400700 1 -- 172.21.15.62:0/1005552158 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 12) v1 ==== 45097+0+0 (secure 0 0 0) 0x7f9624023b30 con 0x7f963419ba50 2024-04-20T21:19:38.882 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:37.871+0000 7f962b400700 1 --2- 172.21.15.62:0/1005552158 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7f9618038490 0x7f961803a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:19:38.882 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:37.871+0000 7f962b400700 1 -- 172.21.15.62:0/1005552158 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f962404c2e0 con 0x7f963419ba50 2024-04-20T21:19:38.882 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:37.871+0000 7f962be00700 1 --2- 172.21.15.62:0/1005552158 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7f9618038490 0x7f961803a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:19:38.882 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:37.872+0000 7f962be00700 1 --2- 172.21.15.62:0/1005552158 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7f9618038490 0x7f961803a940 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f961c006fd0 tx=0x7f961c006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:19:38.882 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:37.877+0000 7f962b400700 1 -- 172.21.15.62:0/1005552158 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f962402b4e0 con 0x7f963419ba50 2024-04-20T21:19:38.882 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:38.149+0000 7f9628c00700 1 -- 172.21.15.62:0/1005552158 --> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f963419c730 con 0x7f9618038490 2024-04-20T21:19:38.882 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:38.196+0000 7f962b400700 1 -- 172.21.15.62:0/1005552158 <== mgr.14164 v2:172.21.15.62:6800/831506538 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f963419c730 con 0x7f9618038490 2024-04-20T21:19:38.882 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:38.198+0000 7f9628c00700 1 -- 172.21.15.62:0/1005552158 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7f9618038490 msgr2=0x7f961803a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:19:38.882 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:38.198+0000 7f9628c00700 1 --2- 172.21.15.62:0/1005552158 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7f9618038490 0x7f961803a940 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f961c006fd0 tx=0x7f961c006e40 comp rx=0 tx=0).stop 2024-04-20T21:19:38.882 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:38.198+0000 7f9628c00700 1 -- 172.21.15.62:0/1005552158 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f963419ba50 msgr2=0x7f963419be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:19:38.882 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:38.198+0000 7f9628c00700 1 --2- 172.21.15.62:0/1005552158 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f963419ba50 0x7f963419be60 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f9624000c00 tx=0x7f9624011b50 comp rx=0 tx=0).stop 2024-04-20T21:19:38.883 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:38.199+0000 7f9628c00700 1 -- 172.21.15.62:0/1005552158 shutdown_connections 2024-04-20T21:19:38.883 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:38.199+0000 7f9628c00700 1 --2- 172.21.15.62:0/1005552158 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7f9618038490 0x7f961803a940 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:19:38.883 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:38.199+0000 7f9628c00700 1 --2- 172.21.15.62:0/1005552158 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f963419ba50 0x7f963419be60 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:19:38.883 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:38.199+0000 7f9628c00700 1 -- 172.21.15.62:0/1005552158 >> 172.21.15.62:0/1005552158 conn(0x7f9634100070 msgr2=0x7f9634101730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:19:38.883 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:38.199+0000 7f9628c00700 1 -- 172.21.15.62:0/1005552158 shutdown_connections 2024-04-20T21:19:38.883 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:38.199+0000 7f9628c00700 1 -- 172.21.15.62:0/1005552158 wait complete. 2024-04-20T21:19:38.883 INFO:teuthology.orchestra.run.smithi062.stdout:Creating initial admin user... 2024-04-20T21:19:39.278 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:39 smithi062 ceph-mon[16221]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:19:39.278 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:39 smithi062 ceph-mon[16221]: from='mgr.14164 172.21.15.62:0/570876096' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:19:39.278 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:39 smithi062 ceph-mon[16221]: from='mgr.14164 172.21.15.62:0/570876096' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:19:39.278 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:39 smithi062 ceph-mon[16221]: from='mgr.14164 172.21.15.62:0/570876096' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:19:40.769 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$sC27JDyHW8umprHMsLew9O.1DVaEL9nUdgCaEPCKNlUUN/0nJo/fS", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1713647979, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-20T21:19:40.772 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:39.427+0000 7ff3e0200700 1 Processor -- start 2024-04-20T21:19:40.773 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:39.427+0000 7ff3e0200700 1 -- start start 2024-04-20T21:19:40.773 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:39.427+0000 7ff3e0200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff3d8106650 0x7ff3d8108a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:19:40.773 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:39.427+0000 7ff3e0200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d80744d0 con 0x7ff3d8106650 2024-04-20T21:19:40.773 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:39.428+0000 7ff3dd800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff3d8106650 0x7ff3d8108a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:19:40.773 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:39.428+0000 7ff3dd800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff3d8106650 0x7ff3d8108a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47426/0 (socket says 172.21.15.62:47426) 2024-04-20T21:19:40.773 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:39.428+0000 7ff3dd800700 1 -- 172.21.15.62:0/950812136 learned_addr learned my addr 172.21.15.62:0/950812136 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:19:40.773 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:39.428+0000 7ff3dd800700 1 -- 172.21.15.62:0/950812136 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3d8074610 con 0x7ff3d8106650 2024-04-20T21:19:40.773 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:39.429+0000 7ff3dd800700 1 --2- 172.21.15.62:0/950812136 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff3d8106650 0x7ff3d8108a30 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ff3cc0098a0 tx=0x7ff3cc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0ffd2c612d101fb server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:19:40.773 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:39.429+0000 7ff3d7e00700 1 -- 172.21.15.62:0/950812136 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff3cc004180 con 0x7ff3d8106650 2024-04-20T21:19:40.773 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:39.429+0000 7ff3d7e00700 1 -- 172.21.15.62:0/950812136 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff3cc0042e0 con 0x7ff3d8106650 2024-04-20T21:19:40.773 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:39.429+0000 7ff3d7e00700 1 -- 172.21.15.62:0/950812136 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff3cc011af0 con 0x7ff3d8106650 2024-04-20T21:19:40.773 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:39.430+0000 7ff3e0200700 1 -- 172.21.15.62:0/950812136 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff3d8106650 msgr2=0x7ff3d8108a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:19:40.773 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:39.430+0000 7ff3e0200700 1 --2- 172.21.15.62:0/950812136 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff3d8106650 0x7ff3d8108a30 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ff3cc0098a0 tx=0x7ff3cc009bb0 comp rx=0 tx=0).stop 2024-04-20T21:19:40.773 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:39.430+0000 7ff3e0200700 1 -- 172.21.15.62:0/950812136 shutdown_connections 2024-04-20T21:19:40.774 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:39.430+0000 7ff3e0200700 1 --2- 172.21.15.62:0/950812136 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff3d8106650 0x7ff3d8108a30 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:19:40.774 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:39.430+0000 7ff3e0200700 1 -- 172.21.15.62:0/950812136 >> 172.21.15.62:0/950812136 conn(0x7ff3d8100070 msgr2=0x7ff3d8102480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:19:40.774 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:39.430+0000 7ff3e0200700 1 -- 172.21.15.62:0/950812136 shutdown_connections 2024-04-20T21:19:40.774 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:39.431+0000 7ff3e0200700 1 -- 172.21.15.62:0/950812136 wait complete. 2024-04-20T21:19:40.774 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:39.432+0000 7ff3e0200700 1 Processor -- start 2024-04-20T21:19:40.774 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:39.432+0000 7ff3e0200700 1 -- start start 2024-04-20T21:19:40.774 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:39.432+0000 7ff3e0200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff3d8106650 0x7ff3d8199840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:19:40.774 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:39.432+0000 7ff3e0200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d8199d80 con 0x7ff3d8106650 2024-04-20T21:19:40.774 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:39.433+0000 7ff3dd800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff3d8106650 0x7ff3d8199840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:19:40.774 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:39.433+0000 7ff3dd800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff3d8106650 0x7ff3d8199840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47438/0 (socket says 172.21.15.62:47438) 2024-04-20T21:19:40.774 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:39.433+0000 7ff3dd800700 1 -- 172.21.15.62:0/3893762662 learned_addr learned my addr 172.21.15.62:0/3893762662 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:19:40.774 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:39.433+0000 7ff3dd800700 1 -- 172.21.15.62:0/3893762662 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3cc009550 con 0x7ff3d8106650 2024-04-20T21:19:40.774 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:39.433+0000 7ff3dd800700 1 --2- 172.21.15.62:0/3893762662 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff3d8106650 0x7ff3d8199840 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff3cc000c00 tx=0x7ff3cc011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:19:40.774 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:39.434+0000 7ff3d6000700 1 -- 172.21.15.62:0/3893762662 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff3cc003a70 con 0x7ff3d8106650 2024-04-20T21:19:40.775 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:39.434+0000 7ff3d6000700 1 -- 172.21.15.62:0/3893762662 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff3cc003bd0 con 0x7ff3d8106650 2024-04-20T21:19:40.777 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:39.434+0000 7ff3e0200700 1 -- 172.21.15.62:0/3893762662 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3d8199fe0 con 0x7ff3d8106650 2024-04-20T21:19:40.777 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:39.434+0000 7ff3d6000700 1 -- 172.21.15.62:0/3893762662 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff3cc0238c0 con 0x7ff3d8106650 2024-04-20T21:19:40.777 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:39.434+0000 7ff3e0200700 1 -- 172.21.15.62:0/3893762662 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3d819a460 con 0x7ff3d8106650 2024-04-20T21:19:40.777 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:39.435+0000 7ff3e0200700 1 -- 172.21.15.62:0/3893762662 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3d8109b70 con 0x7ff3d8106650 2024-04-20T21:19:40.777 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:39.436+0000 7ff3d6000700 1 -- 172.21.15.62:0/3893762662 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 12) v1 ==== 45097+0+0 (secure 0 0 0) 0x7ff3cc023b80 con 0x7ff3d8106650 2024-04-20T21:19:40.777 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:39.436+0000 7ff3d6000700 1 --2- 172.21.15.62:0/3893762662 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7ff3c0038440 0x7ff3c003a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:19:40.777 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:39.436+0000 7ff3d6000700 1 -- 172.21.15.62:0/3893762662 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7ff3cc04c480 con 0x7ff3d8106650 2024-04-20T21:19:40.777 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:39.436+0000 7ff3dce00700 1 --2- 172.21.15.62:0/3893762662 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7ff3c0038440 0x7ff3c003a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:19:40.777 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:39.437+0000 7ff3dce00700 1 --2- 172.21.15.62:0/3893762662 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7ff3c0038440 0x7ff3c003a8f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff3c4006fd0 tx=0x7ff3c4006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:19:40.778 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:39.445+0000 7ff3d6000700 1 -- 172.21.15.62:0/3893762662 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7ff3cc0193c0 con 0x7ff3d8106650 2024-04-20T21:19:40.778 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:39.692+0000 7ff3e0200700 1 -- 172.21.15.62:0/3893762662 --> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7ff3d8109d60 con 0x7ff3c0038440 2024-04-20T21:19:40.778 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:39.987+0000 7ff3d6000700 1 -- 172.21.15.62:0/3893762662 <== mgr.14164 v2:172.21.15.62:6800/831506538 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7ff3d8109d60 con 0x7ff3c0038440 2024-04-20T21:19:40.778 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:39.990+0000 7ff3e0200700 1 -- 172.21.15.62:0/3893762662 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7ff3c0038440 msgr2=0x7ff3c003a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:19:40.778 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:39.990+0000 7ff3e0200700 1 --2- 172.21.15.62:0/3893762662 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7ff3c0038440 0x7ff3c003a8f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff3c4006fd0 tx=0x7ff3c4006e40 comp rx=0 tx=0).stop 2024-04-20T21:19:40.778 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:39.990+0000 7ff3e0200700 1 -- 172.21.15.62:0/3893762662 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff3d8106650 msgr2=0x7ff3d8199840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:19:40.778 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:39.990+0000 7ff3e0200700 1 --2- 172.21.15.62:0/3893762662 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff3d8106650 0x7ff3d8199840 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff3cc000c00 tx=0x7ff3cc011b50 comp rx=0 tx=0).stop 2024-04-20T21:19:40.779 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:39.990+0000 7ff3e0200700 1 -- 172.21.15.62:0/3893762662 shutdown_connections 2024-04-20T21:19:40.779 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:39.990+0000 7ff3e0200700 1 --2- 172.21.15.62:0/3893762662 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7ff3c0038440 0x7ff3c003a8f0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:19:40.779 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:39.990+0000 7ff3e0200700 1 --2- 172.21.15.62:0/3893762662 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff3d8106650 0x7ff3d8199840 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:19:40.779 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:39.990+0000 7ff3e0200700 1 -- 172.21.15.62:0/3893762662 >> 172.21.15.62:0/3893762662 conn(0x7ff3d8100070 msgr2=0x7ff3d8101780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:19:40.779 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:39.990+0000 7ff3e0200700 1 -- 172.21.15.62:0/3893762662 shutdown_connections 2024-04-20T21:19:40.779 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:39.991+0000 7ff3e0200700 1 -- 172.21.15.62:0/3893762662 wait complete. 2024-04-20T21:19:40.779 INFO:teuthology.orchestra.run.smithi062.stdout:Fetching dashboard port number... 2024-04-20T21:19:41.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:40 smithi062 ceph-mon[16221]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:19:41.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:40 smithi062 ceph-mon[16221]: from='mgr.14164 172.21.15.62:0/570876096' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:19:42.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:41 smithi062 ceph-mon[16221]: mgrmap e13: smithi062.rqxuxi(active, since 6s) 2024-04-20T21:19:42.757 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout 8443 2024-04-20T21:19:42.758 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:41.853+0000 7fc04b200700 1 Processor -- start 2024-04-20T21:19:42.758 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:41.854+0000 7fc04b200700 1 -- start start 2024-04-20T21:19:42.758 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:41.854+0000 7fc04b200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc04c104840 0x7fc04c104c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:19:42.758 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:41.854+0000 7fc04b200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc04c105190 con 0x7fc04c104840 2024-04-20T21:19:42.758 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:41.854+0000 7fc049e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc04c104840 0x7fc04c104c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:19:42.758 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:41.855+0000 7fc049e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc04c104840 0x7fc04c104c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40088/0 (socket says 172.21.15.62:40088) 2024-04-20T21:19:42.758 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:41.855+0000 7fc049e00700 1 -- 172.21.15.62:0/1873981820 learned_addr learned my addr 172.21.15.62:0/1873981820 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:19:42.758 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:41.855+0000 7fc049e00700 1 -- 172.21.15.62:0/1873981820 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc04c1052d0 con 0x7fc04c104840 2024-04-20T21:19:42.758 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:41.855+0000 7fc049e00700 1 --2- 172.21.15.62:0/1873981820 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc04c104840 0x7fc04c104c50 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc03c0098a0 tx=0x7fc03c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea1d411a24c8e1cb server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:19:42.758 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:41.856+0000 7fc049400700 1 -- 172.21.15.62:0/1873981820 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc03c004180 con 0x7fc04c104840 2024-04-20T21:19:42.758 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:41.856+0000 7fc049400700 1 -- 172.21.15.62:0/1873981820 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc03c0042e0 con 0x7fc04c104840 2024-04-20T21:19:42.758 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:41.856+0000 7fc049400700 1 -- 172.21.15.62:0/1873981820 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc03c011af0 con 0x7fc04c104840 2024-04-20T21:19:42.758 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:41.856+0000 7fc04b200700 1 -- 172.21.15.62:0/1873981820 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc04c104840 msgr2=0x7fc04c104c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:19:42.759 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:41.856+0000 7fc04b200700 1 --2- 172.21.15.62:0/1873981820 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc04c104840 0x7fc04c104c50 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc03c0098a0 tx=0x7fc03c009bb0 comp rx=0 tx=0).stop 2024-04-20T21:19:42.759 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:41.857+0000 7fc04b200700 1 -- 172.21.15.62:0/1873981820 shutdown_connections 2024-04-20T21:19:42.759 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:41.857+0000 7fc04b200700 1 --2- 172.21.15.62:0/1873981820 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc04c104840 0x7fc04c104c50 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:19:42.759 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:41.857+0000 7fc04b200700 1 -- 172.21.15.62:0/1873981820 >> 172.21.15.62:0/1873981820 conn(0x7fc04c0ffeb0 msgr2=0x7fc04c1022c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:19:42.759 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:41.857+0000 7fc04b200700 1 -- 172.21.15.62:0/1873981820 shutdown_connections 2024-04-20T21:19:42.759 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:41.857+0000 7fc04b200700 1 -- 172.21.15.62:0/1873981820 wait complete. 2024-04-20T21:19:42.759 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:41.858+0000 7fc04b200700 1 Processor -- start 2024-04-20T21:19:42.759 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:41.858+0000 7fc04b200700 1 -- start start 2024-04-20T21:19:42.759 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:41.858+0000 7fc04b200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc04c104840 0x7fc04c18fa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:19:42.759 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:41.859+0000 7fc04b200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc04c18ff60 con 0x7fc04c104840 2024-04-20T21:19:42.759 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:41.859+0000 7fc049e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc04c104840 0x7fc04c18fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:19:42.759 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:41.859+0000 7fc049e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc04c104840 0x7fc04c18fa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40100/0 (socket says 172.21.15.62:40100) 2024-04-20T21:19:42.759 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:41.859+0000 7fc049e00700 1 -- 172.21.15.62:0/762577529 learned_addr learned my addr 172.21.15.62:0/762577529 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:19:42.759 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:41.859+0000 7fc049e00700 1 -- 172.21.15.62:0/762577529 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc03c009550 con 0x7fc04c104840 2024-04-20T21:19:42.760 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:41.859+0000 7fc049e00700 1 --2- 172.21.15.62:0/762577529 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc04c104840 0x7fc04c18fa20 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fc03c011ad0 tx=0x7fc03c011bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:19:42.760 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:41.860+0000 7fc042a00700 1 -- 172.21.15.62:0/762577529 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc03c003ab0 con 0x7fc04c104840 2024-04-20T21:19:42.760 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:41.860+0000 7fc042a00700 1 -- 172.21.15.62:0/762577529 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc03c003c10 con 0x7fc04c104840 2024-04-20T21:19:42.760 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:41.860+0000 7fc042a00700 1 -- 172.21.15.62:0/762577529 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc03c0198e0 con 0x7fc04c104840 2024-04-20T21:19:42.760 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:41.860+0000 7fc04b200700 1 -- 172.21.15.62:0/762577529 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc04c18e070 con 0x7fc04c104840 2024-04-20T21:19:42.760 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:41.860+0000 7fc04b200700 1 -- 172.21.15.62:0/762577529 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc04c18e490 con 0x7fc04c104840 2024-04-20T21:19:42.760 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:41.861+0000 7fc042a00700 1 -- 172.21.15.62:0/762577529 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 13) v1 ==== 45140+0+0 (secure 0 0 0) 0x7fc03c019b50 con 0x7fc04c104840 2024-04-20T21:19:42.760 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:41.861+0000 7fc04b200700 1 -- 172.21.15.62:0/762577529 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc04c062380 con 0x7fc04c104840 2024-04-20T21:19:42.760 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:41.862+0000 7fc042a00700 1 --2- 172.21.15.62:0/762577529 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7fc030038530 0x7fc03003a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:19:42.760 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:41.862+0000 7fc042a00700 1 -- 172.21.15.62:0/762577529 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fc03c04c420 con 0x7fc04c104840 2024-04-20T21:19:42.760 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:41.862+0000 7fc043e00700 1 --2- 172.21.15.62:0/762577529 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7fc030038530 0x7fc03003a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:19:42.760 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:41.863+0000 7fc043e00700 1 --2- 172.21.15.62:0/762577529 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7fc030038530 0x7fc03003a9e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fc034006fd0 tx=0x7fc034006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:19:42.760 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:41.867+0000 7fc042a00700 1 -- 172.21.15.62:0/762577529 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fc03c01a430 con 0x7fc04c104840 2024-04-20T21:19:42.760 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:42.094+0000 7fc04b200700 1 -- 172.21.15.62:0/762577529 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7fc04c1066f0 con 0x7fc04c104840 2024-04-20T21:19:42.761 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:42.095+0000 7fc042a00700 1 -- 172.21.15.62:0/762577529 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7fc03c010380 con 0x7fc04c104840 2024-04-20T21:19:42.761 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:42.098+0000 7fc04b200700 1 -- 172.21.15.62:0/762577529 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7fc030038530 msgr2=0x7fc03003a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:19:42.761 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:42.098+0000 7fc04b200700 1 --2- 172.21.15.62:0/762577529 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7fc030038530 0x7fc03003a9e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fc034006fd0 tx=0x7fc034006e40 comp rx=0 tx=0).stop 2024-04-20T21:19:42.761 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:42.098+0000 7fc04b200700 1 -- 172.21.15.62:0/762577529 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc04c104840 msgr2=0x7fc04c18fa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:19:42.761 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:42.098+0000 7fc04b200700 1 --2- 172.21.15.62:0/762577529 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc04c104840 0x7fc04c18fa20 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fc03c011ad0 tx=0x7fc03c011bb0 comp rx=0 tx=0).stop 2024-04-20T21:19:42.761 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:42.098+0000 7fc04b200700 1 -- 172.21.15.62:0/762577529 shutdown_connections 2024-04-20T21:19:42.761 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:42.098+0000 7fc04b200700 1 --2- 172.21.15.62:0/762577529 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7fc030038530 0x7fc03003a9e0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:19:42.761 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:42.098+0000 7fc04b200700 1 --2- 172.21.15.62:0/762577529 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc04c104840 0x7fc04c18fa20 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:19:42.761 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:42.098+0000 7fc04b200700 1 -- 172.21.15.62:0/762577529 >> 172.21.15.62:0/762577529 conn(0x7fc04c0ffeb0 msgr2=0x7fc04c074030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:19:42.761 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:42.098+0000 7fc04b200700 1 -- 172.21.15.62:0/762577529 shutdown_connections 2024-04-20T21:19:42.761 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:42.098+0000 7fc04b200700 1 -- 172.21.15.62:0/762577529 wait complete. 2024-04-20T21:19:42.795 INFO:teuthology.orchestra.run.smithi062.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-20T21:19:42.795 INFO:teuthology.orchestra.run.smithi062.stdout:systemctl: stdout disabled 2024-04-20T21:19:42.822 INFO:teuthology.orchestra.run.smithi062.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-20T21:19:42.823 INFO:teuthology.orchestra.run.smithi062.stdout:systemctl: stdout inactive 2024-04-20T21:19:42.823 INFO:teuthology.orchestra.run.smithi062.stdout:firewalld.service is not enabled 2024-04-20T21:19:42.823 INFO:teuthology.orchestra.run.smithi062.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-20T21:19:42.831 INFO:teuthology.orchestra.run.smithi062.stdout:Ceph Dashboard is now available at: 2024-04-20T21:19:42.831 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:19:42.831 INFO:teuthology.orchestra.run.smithi062.stdout: URL: https://smithi062.front.sepia.ceph.com:8443/ 2024-04-20T21:19:42.831 INFO:teuthology.orchestra.run.smithi062.stdout: User: admin 2024-04-20T21:19:42.831 INFO:teuthology.orchestra.run.smithi062.stdout: Password: 3og7iexgcn 2024-04-20T21:19:42.831 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:19:42.832 INFO:teuthology.orchestra.run.smithi062.stdout:Saving cluster configuration to /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/config directory 2024-04-20T21:19:42.834 INFO:teuthology.orchestra.run.smithi062.stdout:Enabling autotune for osd_memory_target 2024-04-20T21:19:43.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:42 smithi062 ceph-mon[16221]: from='client.? 172.21.15.62:0/762577529' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-20T21:19:44.417 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:43.463+0000 7fcf1ae00700 1 Processor -- start 2024-04-20T21:19:44.417 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:43.463+0000 7fcf1ae00700 1 -- start start 2024-04-20T21:19:44.418 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:43.464+0000 7fcf1ae00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf14107d10 0x7fcf14108120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:19:44.418 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:43.464+0000 7fcf1ae00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf14108660 con 0x7fcf14107d10 2024-04-20T21:19:44.418 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:43.464+0000 7fcf13e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf14107d10 0x7fcf14108120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:19:44.418 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:43.464+0000 7fcf13e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf14107d10 0x7fcf14108120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40108/0 (socket says 172.21.15.62:40108) 2024-04-20T21:19:44.418 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:43.464+0000 7fcf13e00700 1 -- 172.21.15.62:0/2901106052 learned_addr learned my addr 172.21.15.62:0/2901106052 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:19:44.418 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:43.465+0000 7fcf13e00700 1 -- 172.21.15.62:0/2901106052 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf141087a0 con 0x7fcf14107d10 2024-04-20T21:19:44.418 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:43.465+0000 7fcf13e00700 1 --2- 172.21.15.62:0/2901106052 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf14107d10 0x7fcf14108120 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fcf040098a0 tx=0x7fcf04009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5ca1ddf99b95c7a server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:19:44.418 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:43.465+0000 7fcf12a00700 1 -- 172.21.15.62:0/2901106052 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcf04004180 con 0x7fcf14107d10 2024-04-20T21:19:44.418 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:43.465+0000 7fcf12a00700 1 -- 172.21.15.62:0/2901106052 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fcf040042e0 con 0x7fcf14107d10 2024-04-20T21:19:44.418 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:43.465+0000 7fcf12a00700 1 -- 172.21.15.62:0/2901106052 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcf04011af0 con 0x7fcf14107d10 2024-04-20T21:19:44.418 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:43.466+0000 7fcf1ae00700 1 -- 172.21.15.62:0/2901106052 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf14107d10 msgr2=0x7fcf14108120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:19:44.418 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:43.466+0000 7fcf1ae00700 1 --2- 172.21.15.62:0/2901106052 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf14107d10 0x7fcf14108120 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fcf040098a0 tx=0x7fcf04009bb0 comp rx=0 tx=0).stop 2024-04-20T21:19:44.419 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:43.466+0000 7fcf1ae00700 1 -- 172.21.15.62:0/2901106052 shutdown_connections 2024-04-20T21:19:44.419 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:43.466+0000 7fcf1ae00700 1 --2- 172.21.15.62:0/2901106052 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf14107d10 0x7fcf14108120 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:19:44.419 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:43.466+0000 7fcf1ae00700 1 -- 172.21.15.62:0/2901106052 >> 172.21.15.62:0/2901106052 conn(0x7fcf14103570 msgr2=0x7fcf14105950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:19:44.419 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:43.466+0000 7fcf1ae00700 1 -- 172.21.15.62:0/2901106052 shutdown_connections 2024-04-20T21:19:44.419 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:43.466+0000 7fcf1ae00700 1 -- 172.21.15.62:0/2901106052 wait complete. 2024-04-20T21:19:44.419 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:43.467+0000 7fcf1ae00700 1 Processor -- start 2024-04-20T21:19:44.419 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:43.467+0000 7fcf1ae00700 1 -- start start 2024-04-20T21:19:44.419 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:43.468+0000 7fcf1ae00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf14107d10 0x7fcf1419ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:19:44.419 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:43.468+0000 7fcf1ae00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf1419bf80 con 0x7fcf14107d10 2024-04-20T21:19:44.419 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:43.468+0000 7fcf13e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf14107d10 0x7fcf1419ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:19:44.419 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:43.468+0000 7fcf13e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf14107d10 0x7fcf1419ba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40112/0 (socket says 172.21.15.62:40112) 2024-04-20T21:19:44.419 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:43.468+0000 7fcf13e00700 1 -- 172.21.15.62:0/1786195899 learned_addr learned my addr 172.21.15.62:0/1786195899 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:19:44.419 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:43.468+0000 7fcf13e00700 1 -- 172.21.15.62:0/1786195899 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf04009550 con 0x7fcf14107d10 2024-04-20T21:19:44.419 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:43.469+0000 7fcf13e00700 1 --2- 172.21.15.62:0/1786195899 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf14107d10 0x7fcf1419ba40 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fcf04011ad0 tx=0x7fcf04011bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:19:44.420 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:43.469+0000 7fcf10c00700 1 -- 172.21.15.62:0/1786195899 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcf04003ab0 con 0x7fcf14107d10 2024-04-20T21:19:44.420 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:43.469+0000 7fcf10c00700 1 -- 172.21.15.62:0/1786195899 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fcf04003c10 con 0x7fcf14107d10 2024-04-20T21:19:44.420 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:43.469+0000 7fcf1ae00700 1 -- 172.21.15.62:0/1786195899 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf1419c180 con 0x7fcf14107d10 2024-04-20T21:19:44.420 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:43.469+0000 7fcf10c00700 1 -- 172.21.15.62:0/1786195899 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcf040198e0 con 0x7fcf14107d10 2024-04-20T21:19:44.420 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:43.469+0000 7fcf1ae00700 1 -- 172.21.15.62:0/1786195899 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf1419c5a0 con 0x7fcf14107d10 2024-04-20T21:19:44.420 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:43.470+0000 7fcf10c00700 1 -- 172.21.15.62:0/1786195899 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 13) v1 ==== 45140+0+0 (secure 0 0 0) 0x7fcf04019b50 con 0x7fcf14107d10 2024-04-20T21:19:44.420 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:43.470+0000 7fcf1ae00700 1 -- 172.21.15.62:0/1786195899 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf14062380 con 0x7fcf14107d10 2024-04-20T21:19:44.420 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:43.471+0000 7fcf10c00700 1 --2- 172.21.15.62:0/1786195899 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7fcef8038530 0x7fcef803a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:19:44.420 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:43.471+0000 7fcf10c00700 1 -- 172.21.15.62:0/1786195899 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fcf0404c350 con 0x7fcf14107d10 2024-04-20T21:19:44.420 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:43.471+0000 7fcf13400700 1 --2- 172.21.15.62:0/1786195899 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7fcef8038530 0x7fcef803a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:19:44.420 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:43.472+0000 7fcf13400700 1 --2- 172.21.15.62:0/1786195899 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7fcef8038530 0x7fcef803a9e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fcf00006fd0 tx=0x7fcf00006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:19:44.420 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:43.476+0000 7fcf10c00700 1 -- 172.21.15.62:0/1786195899 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fcf0401a430 con 0x7fcf14107d10 2024-04-20T21:19:44.420 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:43.697+0000 7fcf1ae00700 1 -- 172.21.15.62:0/1786195899 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7fcf1419f470 con 0x7fcf14107d10 2024-04-20T21:19:44.420 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:43.698+0000 7fcf10c00700 1 -- 172.21.15.62:0/1786195899 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7fcf04018350 con 0x7fcf14107d10 2024-04-20T21:19:44.421 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:43.702+0000 7fcf1ae00700 1 -- 172.21.15.62:0/1786195899 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7fcef8038530 msgr2=0x7fcef803a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:19:44.421 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:43.702+0000 7fcf1ae00700 1 --2- 172.21.15.62:0/1786195899 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7fcef8038530 0x7fcef803a9e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fcf00006fd0 tx=0x7fcf00006e40 comp rx=0 tx=0).stop 2024-04-20T21:19:44.421 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:43.702+0000 7fcf1ae00700 1 -- 172.21.15.62:0/1786195899 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf14107d10 msgr2=0x7fcf1419ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:19:44.421 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:43.702+0000 7fcf1ae00700 1 --2- 172.21.15.62:0/1786195899 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf14107d10 0x7fcf1419ba40 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fcf04011ad0 tx=0x7fcf04011bb0 comp rx=0 tx=0).stop 2024-04-20T21:19:44.421 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:43.702+0000 7fcf1ae00700 1 -- 172.21.15.62:0/1786195899 shutdown_connections 2024-04-20T21:19:44.421 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:43.702+0000 7fcf1ae00700 1 --2- 172.21.15.62:0/1786195899 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7fcef8038530 0x7fcef803a9e0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:19:44.421 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:43.702+0000 7fcf1ae00700 1 --2- 172.21.15.62:0/1786195899 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf14107d10 0x7fcf1419ba40 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:19:44.421 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:43.702+0000 7fcf1ae00700 1 -- 172.21.15.62:0/1786195899 >> 172.21.15.62:0/1786195899 conn(0x7fcf14103570 msgr2=0x7fcf14105320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:19:44.421 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:43.702+0000 7fcf1ae00700 1 -- 172.21.15.62:0/1786195899 shutdown_connections 2024-04-20T21:19:44.421 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:43.702+0000 7fcf1ae00700 1 -- 172.21.15.62:0/1786195899 wait complete. 2024-04-20T21:19:46.130 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:45.161+0000 7f4560800700 1 Processor -- start 2024-04-20T21:19:46.130 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:45.162+0000 7f4560800700 1 -- start start 2024-04-20T21:19:46.131 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:45.162+0000 7f4560800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4558107d00 0x7f4558108110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:19:46.131 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:45.162+0000 7f4560800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4558108650 con 0x7f4558107d00 2024-04-20T21:19:46.131 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:45.162+0000 7f455de00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4558107d00 0x7f4558108110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:19:46.131 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:45.162+0000 7f455de00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4558107d00 0x7f4558108110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40128/0 (socket says 172.21.15.62:40128) 2024-04-20T21:19:46.131 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:45.163+0000 7f455de00700 1 -- 172.21.15.62:0/1652940413 learned_addr learned my addr 172.21.15.62:0/1652940413 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:19:46.132 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:45.163+0000 7f455de00700 1 -- 172.21.15.62:0/1652940413 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4558108790 con 0x7f4558107d00 2024-04-20T21:19:46.132 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:45.163+0000 7f455de00700 1 --2- 172.21.15.62:0/1652940413 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4558107d00 0x7f4558108110 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f454c0098a0 tx=0x7f454c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=400b2b98ba8aebdf server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:19:46.132 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:45.163+0000 7f455ca00700 1 -- 172.21.15.62:0/1652940413 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f454c004180 con 0x7f4558107d00 2024-04-20T21:19:46.132 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:45.164+0000 7f455ca00700 1 -- 172.21.15.62:0/1652940413 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f454c0042e0 con 0x7f4558107d00 2024-04-20T21:19:46.132 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:45.164+0000 7f455ca00700 1 -- 172.21.15.62:0/1652940413 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f454c011af0 con 0x7f4558107d00 2024-04-20T21:19:46.132 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:45.164+0000 7f4560800700 1 -- 172.21.15.62:0/1652940413 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4558107d00 msgr2=0x7f4558108110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:19:46.133 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:45.164+0000 7f4560800700 1 --2- 172.21.15.62:0/1652940413 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4558107d00 0x7f4558108110 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f454c0098a0 tx=0x7f454c009bb0 comp rx=0 tx=0).stop 2024-04-20T21:19:46.133 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:45.164+0000 7f4560800700 1 -- 172.21.15.62:0/1652940413 shutdown_connections 2024-04-20T21:19:46.134 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:45.164+0000 7f4560800700 1 --2- 172.21.15.62:0/1652940413 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4558107d00 0x7f4558108110 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:19:46.134 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:45.164+0000 7f4560800700 1 -- 172.21.15.62:0/1652940413 >> 172.21.15.62:0/1652940413 conn(0x7f455807b220 msgr2=0x7f455807b640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:19:46.134 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:45.164+0000 7f4560800700 1 -- 172.21.15.62:0/1652940413 shutdown_connections 2024-04-20T21:19:46.134 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:45.165+0000 7f4560800700 1 -- 172.21.15.62:0/1652940413 wait complete. 2024-04-20T21:19:46.134 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:45.166+0000 7f4560800700 1 Processor -- start 2024-04-20T21:19:46.134 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:45.166+0000 7f4560800700 1 -- start start 2024-04-20T21:19:46.134 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:45.166+0000 7f4560800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4558107d00 0x7f455819bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:19:46.134 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:45.166+0000 7f4560800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f455819bff0 con 0x7f4558107d00 2024-04-20T21:19:46.136 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:45.167+0000 7f455de00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4558107d00 0x7f455819bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:19:46.136 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:45.167+0000 7f455de00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4558107d00 0x7f455819bab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40132/0 (socket says 172.21.15.62:40132) 2024-04-20T21:19:46.136 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:45.167+0000 7f455de00700 1 -- 172.21.15.62:0/3394098836 learned_addr learned my addr 172.21.15.62:0/3394098836 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:19:46.136 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:45.167+0000 7f455de00700 1 -- 172.21.15.62:0/3394098836 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f454c009550 con 0x7f4558107d00 2024-04-20T21:19:46.136 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:45.168+0000 7f455de00700 1 --2- 172.21.15.62:0/3394098836 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4558107d00 0x7f455819bab0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f454c011ad0 tx=0x7f454c011bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:19:46.136 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:45.168+0000 7f4552a00700 1 -- 172.21.15.62:0/3394098836 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f454c003ab0 con 0x7f4558107d00 2024-04-20T21:19:46.136 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:45.168+0000 7f4552a00700 1 -- 172.21.15.62:0/3394098836 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f454c003c10 con 0x7f4558107d00 2024-04-20T21:19:46.136 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:45.168+0000 7f4560800700 1 -- 172.21.15.62:0/3394098836 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f455819c1f0 con 0x7f4558107d00 2024-04-20T21:19:46.136 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:45.168+0000 7f4552a00700 1 -- 172.21.15.62:0/3394098836 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f454c0198e0 con 0x7f4558107d00 2024-04-20T21:19:46.136 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:45.168+0000 7f4560800700 1 -- 172.21.15.62:0/3394098836 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f455819c610 con 0x7f4558107d00 2024-04-20T21:19:46.136 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:45.169+0000 7f4552a00700 1 -- 172.21.15.62:0/3394098836 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 13) v1 ==== 45140+0+0 (secure 0 0 0) 0x7f454c019b50 con 0x7f4558107d00 2024-04-20T21:19:46.138 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:45.169+0000 7f4560800700 1 -- 172.21.15.62:0/3394098836 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4558062380 con 0x7f4558107d00 2024-04-20T21:19:46.138 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:45.170+0000 7f4552a00700 1 --2- 172.21.15.62:0/3394098836 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7f4540038530 0x7f454003a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:19:46.138 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:45.170+0000 7f4552a00700 1 -- 172.21.15.62:0/3394098836 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f454c04c610 con 0x7f4558107d00 2024-04-20T21:19:46.138 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:45.170+0000 7f455d400700 1 --2- 172.21.15.62:0/3394098836 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7f4540038530 0x7f454003a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:19:46.138 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:45.171+0000 7f455d400700 1 --2- 172.21.15.62:0/3394098836 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7f4540038530 0x7f454003a9e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f4548006fd0 tx=0x7f4548006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:19:46.138 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:45.175+0000 7f4552a00700 1 -- 172.21.15.62:0/3394098836 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f454c02b300 con 0x7f4558107d00 2024-04-20T21:19:46.138 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:45.517+0000 7f4560800700 1 -- 172.21.15.62:0/3394098836 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7f455819f470 con 0x7f4558107d00 2024-04-20T21:19:46.138 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:45.523+0000 7f4552a00700 1 -- 172.21.15.62:0/3394098836 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v34)=0 set mgr/dashboard/cluster/status v34) v1 ==== 153+0+0 (secure 0 0 0) 0x7f454c010b70 con 0x7f4558107d00 2024-04-20T21:19:46.139 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:45.525+0000 7f4560800700 1 -- 172.21.15.62:0/3394098836 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7f4540038530 msgr2=0x7f454003a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:19:46.139 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:45.525+0000 7f4560800700 1 --2- 172.21.15.62:0/3394098836 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7f4540038530 0x7f454003a9e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f4548006fd0 tx=0x7f4548006e40 comp rx=0 tx=0).stop 2024-04-20T21:19:46.139 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:45.525+0000 7f4560800700 1 -- 172.21.15.62:0/3394098836 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4558107d00 msgr2=0x7f455819bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:19:46.139 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:45.525+0000 7f4560800700 1 --2- 172.21.15.62:0/3394098836 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4558107d00 0x7f455819bab0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f454c011ad0 tx=0x7f454c011bb0 comp rx=0 tx=0).stop 2024-04-20T21:19:46.139 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:45.525+0000 7f4560800700 1 -- 172.21.15.62:0/3394098836 shutdown_connections 2024-04-20T21:19:46.140 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:45.525+0000 7f4560800700 1 --2- 172.21.15.62:0/3394098836 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7f4540038530 0x7f454003a9e0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:19:46.140 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:45.525+0000 7f4560800700 1 --2- 172.21.15.62:0/3394098836 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4558107d00 0x7f455819bab0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:19:46.140 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:45.526+0000 7f4560800700 1 -- 172.21.15.62:0/3394098836 >> 172.21.15.62:0/3394098836 conn(0x7f455807b220 msgr2=0x7f4558105ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:19:46.140 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:45.526+0000 7f4560800700 1 -- 172.21.15.62:0/3394098836 shutdown_connections 2024-04-20T21:19:46.140 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-04-20T21:19:45.526+0000 7f4560800700 1 -- 172.21.15.62:0/3394098836 wait complete. 2024-04-20T21:19:46.141 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-04-20T21:19:46.141 INFO:teuthology.orchestra.run.smithi062.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-04-20T21:19:46.141 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:19:46.141 INFO:teuthology.orchestra.run.smithi062.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-20T21:19:46.141 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:19:46.141 INFO:teuthology.orchestra.run.smithi062.stdout:Or, if you are only running a single cluster on this host: 2024-04-20T21:19:46.141 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:19:46.141 INFO:teuthology.orchestra.run.smithi062.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-04-20T21:19:46.141 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:19:46.141 INFO:teuthology.orchestra.run.smithi062.stdout:Please consider enabling telemetry to help improve Ceph: 2024-04-20T21:19:46.141 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:19:46.141 INFO:teuthology.orchestra.run.smithi062.stdout: ceph telemetry on 2024-04-20T21:19:46.141 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:19:46.141 INFO:teuthology.orchestra.run.smithi062.stdout:For more information see: 2024-04-20T21:19:46.142 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:19:46.142 INFO:teuthology.orchestra.run.smithi062.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-04-20T21:19:46.142 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:19:46.142 INFO:teuthology.orchestra.run.smithi062.stdout:Bootstrap complete. 2024-04-20T21:19:46.249 INFO:tasks.cephadm:Fetching config... 2024-04-20T21:19:46.250 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-04-20T21:19:46.250 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-20T21:19:46.305 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-20T21:19:46.305 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-04-20T21:19:46.305 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-20T21:19:46.349 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-20T21:19:46.349 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-04-20T21:19:46.349 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/keyring of=/dev/stdout 2024-04-20T21:19:46.465 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-20T21:19:46.465 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-04-20T21:19:46.465 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-20T21:19:46.562 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-20T21:19:46.562 DEBUG:teuthology.orchestra.run.smithi062:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDo8heeGJn1ioyL6p7G0QD0cZdLrSEmX4SZLE4ugWVEC27Z0Qz92rrI+1HFetWULR62uaVx+NYWQq5JT9vyo3fZ9YJaZg4phajNPqKoccbpD0WPkARZBDll4aknfmqPRWbIBPVJZI1/gYX7bXT7/D18I7BeX84zh0P6U9oEGTzpwv3AEWaFS9lki3pR9cw/NE4K3C8G7J6vydxs0TWudV4c9/VxFloHcN+/lZqzvgT4tiD6iVORvmTiUcVa7jfhdNMx7gewmP9ijoKtHomik2Hx6ymiYJLybw+yWW+Rut/aTIX1RcUkZ7F/hjd0hv5PgrpzAsJZezLp72hG3tS3WNaxqEjp7ao/FzaQ6zMgh4XvOUDiGT3B+bF7LYTVyDCRJTh/D0GUwsdPmVV71SAeY4F9aPF3M78y0Jn4QhS8AKSIAx5Fl+3TEQqarEZRfFVcUnPJ6DBUWdZysHoOTM+zYJWZL2TMPDZgmv0VPlQvJlkHT1vjP/Lqjzuao1kwQO3bdeU= ceph-61ae3628-ff5b-11ee-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-20T21:19:46.666 INFO:teuthology.orchestra.run.smithi062.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDo8heeGJn1ioyL6p7G0QD0cZdLrSEmX4SZLE4ugWVEC27Z0Qz92rrI+1HFetWULR62uaVx+NYWQq5JT9vyo3fZ9YJaZg4phajNPqKoccbpD0WPkARZBDll4aknfmqPRWbIBPVJZI1/gYX7bXT7/D18I7BeX84zh0P6U9oEGTzpwv3AEWaFS9lki3pR9cw/NE4K3C8G7J6vydxs0TWudV4c9/VxFloHcN+/lZqzvgT4tiD6iVORvmTiUcVa7jfhdNMx7gewmP9ijoKtHomik2Hx6ymiYJLybw+yWW+Rut/aTIX1RcUkZ7F/hjd0hv5PgrpzAsJZezLp72hG3tS3WNaxqEjp7ao/FzaQ6zMgh4XvOUDiGT3B+bF7LYTVyDCRJTh/D0GUwsdPmVV71SAeY4F9aPF3M78y0Jn4QhS8AKSIAx5Fl+3TEQqarEZRfFVcUnPJ6DBUWdZysHoOTM+zYJWZL2TMPDZgmv0VPlQvJlkHT1vjP/Lqjzuao1kwQO3bdeU= ceph-61ae3628-ff5b-11ee-bc93-c7b262605968 2024-04-20T21:19:46.695 DEBUG:teuthology.orchestra.run.smithi189:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDo8heeGJn1ioyL6p7G0QD0cZdLrSEmX4SZLE4ugWVEC27Z0Qz92rrI+1HFetWULR62uaVx+NYWQq5JT9vyo3fZ9YJaZg4phajNPqKoccbpD0WPkARZBDll4aknfmqPRWbIBPVJZI1/gYX7bXT7/D18I7BeX84zh0P6U9oEGTzpwv3AEWaFS9lki3pR9cw/NE4K3C8G7J6vydxs0TWudV4c9/VxFloHcN+/lZqzvgT4tiD6iVORvmTiUcVa7jfhdNMx7gewmP9ijoKtHomik2Hx6ymiYJLybw+yWW+Rut/aTIX1RcUkZ7F/hjd0hv5PgrpzAsJZezLp72hG3tS3WNaxqEjp7ao/FzaQ6zMgh4XvOUDiGT3B+bF7LYTVyDCRJTh/D0GUwsdPmVV71SAeY4F9aPF3M78y0Jn4QhS8AKSIAx5Fl+3TEQqarEZRfFVcUnPJ6DBUWdZysHoOTM+zYJWZL2TMPDZgmv0VPlQvJlkHT1vjP/Lqjzuao1kwQO3bdeU= ceph-61ae3628-ff5b-11ee-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-20T21:19:46.716 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:46 smithi062 ceph-mon[16221]: from='client.? 172.21.15.62:0/3394098836' entity='client.admin' 2024-04-20T21:19:47.106 INFO:teuthology.orchestra.run.smithi189.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDo8heeGJn1ioyL6p7G0QD0cZdLrSEmX4SZLE4ugWVEC27Z0Qz92rrI+1HFetWULR62uaVx+NYWQq5JT9vyo3fZ9YJaZg4phajNPqKoccbpD0WPkARZBDll4aknfmqPRWbIBPVJZI1/gYX7bXT7/D18I7BeX84zh0P6U9oEGTzpwv3AEWaFS9lki3pR9cw/NE4K3C8G7J6vydxs0TWudV4c9/VxFloHcN+/lZqzvgT4tiD6iVORvmTiUcVa7jfhdNMx7gewmP9ijoKtHomik2Hx6ymiYJLybw+yWW+Rut/aTIX1RcUkZ7F/hjd0hv5PgrpzAsJZezLp72hG3tS3WNaxqEjp7ao/FzaQ6zMgh4XvOUDiGT3B+bF7LYTVyDCRJTh/D0GUwsdPmVV71SAeY4F9aPF3M78y0Jn4QhS8AKSIAx5Fl+3TEQqarEZRfFVcUnPJ6DBUWdZysHoOTM+zYJWZL2TMPDZgmv0VPlQvJlkHT1vjP/Lqjzuao1kwQO3bdeU= ceph-61ae3628-ff5b-11ee-bc93-c7b262605968 2024-04-20T21:19:47.168 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-20T21:19:48.072 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T21:19:50.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:50.178+0000 7f45a7e00700 1 -- 172.21.15.62:0/3957175170 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f45a8101e50 msgr2=0x7f45a8102260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:19:50.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:50.178+0000 7f45a7e00700 1 --2- 172.21.15.62:0/3957175170 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f45a8101e50 0x7f45a8102260 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f4598009910 tx=0x7f4598009c20 comp rx=0 tx=0).stop 2024-04-20T21:19:50.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:50.178+0000 7f45a7e00700 1 -- 172.21.15.62:0/3957175170 shutdown_connections 2024-04-20T21:19:50.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:50.178+0000 7f45a7e00700 1 --2- 172.21.15.62:0/3957175170 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f45a8101e50 0x7f45a8102260 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:19:50.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:50.178+0000 7f45a7e00700 1 -- 172.21.15.62:0/3957175170 >> 172.21.15.62:0/3957175170 conn(0x7f45a80fd530 msgr2=0x7f45a80ff910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:19:50.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:50.178+0000 7f45a7e00700 1 -- 172.21.15.62:0/3957175170 shutdown_connections 2024-04-20T21:19:50.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:50.178+0000 7f45a7e00700 1 -- 172.21.15.62:0/3957175170 wait complete. 2024-04-20T21:19:50.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:50.179+0000 7f45a7e00700 1 Processor -- start 2024-04-20T21:19:50.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:50.180+0000 7f45a7e00700 1 -- start start 2024-04-20T21:19:50.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:50.180+0000 7f45a7e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f45a8101e50 0x7f45a8197000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:19:50.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:50.180+0000 7f45a7e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45a8197540 con 0x7f45a8101e50 2024-04-20T21:19:50.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:50.180+0000 7f45a6a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f45a8101e50 0x7f45a8197000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:19:50.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:50.180+0000 7f45a6a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f45a8101e50 0x7f45a8197000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40156/0 (socket says 172.21.15.62:40156) 2024-04-20T21:19:50.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:50.180+0000 7f45a6a00700 1 -- 172.21.15.62:0/521347874 learned_addr learned my addr 172.21.15.62:0/521347874 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:19:50.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:50.181+0000 7f45a6a00700 1 -- 172.21.15.62:0/521347874 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45980095f0 con 0x7f45a8101e50 2024-04-20T21:19:50.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:50.181+0000 7f45a6a00700 1 --2- 172.21.15.62:0/521347874 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f45a8101e50 0x7f45a8197000 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f45980092e0 tx=0x7f4598009e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:19:50.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:50.181+0000 7f459f400700 1 -- 172.21.15.62:0/521347874 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f459801d070 con 0x7f45a8101e50 2024-04-20T21:19:50.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:50.181+0000 7f459f400700 1 -- 172.21.15.62:0/521347874 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f459800be10 con 0x7f45a8101e50 2024-04-20T21:19:50.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:50.181+0000 7f45a7e00700 1 -- 172.21.15.62:0/521347874 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45a8197740 con 0x7f45a8101e50 2024-04-20T21:19:50.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:50.181+0000 7f459f400700 1 -- 172.21.15.62:0/521347874 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f459800f460 con 0x7f45a8101e50 2024-04-20T21:19:50.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:50.182+0000 7f45a7e00700 1 -- 172.21.15.62:0/521347874 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45a8197be0 con 0x7f45a8101e50 2024-04-20T21:19:50.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:50.183+0000 7f45a7e00700 1 -- 172.21.15.62:0/521347874 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45a8190ab0 con 0x7f45a8101e50 2024-04-20T21:19:50.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:50.183+0000 7f459f400700 1 -- 172.21.15.62:0/521347874 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 13) v1 ==== 45140+0+0 (secure 0 0 0) 0x7f4598022470 con 0x7f45a8101e50 2024-04-20T21:19:50.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:50.184+0000 7f459f400700 1 --2- 172.21.15.62:0/521347874 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7f458c038490 0x7f458c03a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:19:50.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:50.184+0000 7f459f400700 1 -- 172.21.15.62:0/521347874 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f459804c3b0 con 0x7f45a8101e50 2024-04-20T21:19:50.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:50.184+0000 7f45a6000700 1 --2- 172.21.15.62:0/521347874 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7f458c038490 0x7f458c03a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:19:50.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:50.185+0000 7f45a6000700 1 --2- 172.21.15.62:0/521347874 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7f458c038490 0x7f458c03a940 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4594006fd0 tx=0x7f4594006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:19:50.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:50.189+0000 7f459f400700 1 -- 172.21.15.62:0/521347874 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f459800b960 con 0x7f45a8101e50 2024-04-20T21:19:50.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:50.417+0000 7f45a7e00700 1 -- 172.21.15.62:0/521347874 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f45a81028e0 con 0x7f45a8101e50 2024-04-20T21:19:50.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:50.425+0000 7f459f400700 1 -- 172.21.15.62:0/521347874 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7f45a81028e0 con 0x7f45a8101e50 2024-04-20T21:19:50.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:50.428+0000 7f45a7e00700 1 -- 172.21.15.62:0/521347874 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7f458c038490 msgr2=0x7f458c03a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:19:50.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:50.428+0000 7f45a7e00700 1 --2- 172.21.15.62:0/521347874 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7f458c038490 0x7f458c03a940 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4594006fd0 tx=0x7f4594006e40 comp rx=0 tx=0).stop 2024-04-20T21:19:50.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:50.428+0000 7f45a7e00700 1 -- 172.21.15.62:0/521347874 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f45a8101e50 msgr2=0x7f45a8197000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:19:50.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:50.428+0000 7f45a7e00700 1 --2- 172.21.15.62:0/521347874 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f45a8101e50 0x7f45a8197000 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f45980092e0 tx=0x7f4598009e60 comp rx=0 tx=0).stop 2024-04-20T21:19:50.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:50.428+0000 7f45a7e00700 1 -- 172.21.15.62:0/521347874 shutdown_connections 2024-04-20T21:19:50.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:50.428+0000 7f45a7e00700 1 --2- 172.21.15.62:0/521347874 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7f458c038490 0x7f458c03a940 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:19:50.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:50.428+0000 7f45a7e00700 1 --2- 172.21.15.62:0/521347874 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f45a8101e50 0x7f45a8197000 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:19:50.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:50.428+0000 7f45a7e00700 1 -- 172.21.15.62:0/521347874 >> 172.21.15.62:0/521347874 conn(0x7f45a80fd530 msgr2=0x7f45a80ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:19:50.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:50.428+0000 7f45a7e00700 1 -- 172.21.15.62:0/521347874 shutdown_connections 2024-04-20T21:19:50.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:50.429+0000 7f45a7e00700 1 -- 172.21.15.62:0/521347874 wait complete. 2024-04-20T21:19:51.231 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-20T21:19:51.231 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-20T21:19:51.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:51 smithi062 ceph-mon[16221]: from='client.? 172.21.15.62:0/521347874' entity='client.admin' 2024-04-20T21:19:51.769 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T21:19:52.943 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:52 smithi062 ceph-mon[16221]: from='mgr.14164 172.21.15.62:0/570876096' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:19:52.943 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:52 smithi062 ceph-mon[16221]: from='mgr.14164 172.21.15.62:0/570876096' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:19:52.943 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:52 smithi062 ceph-mon[16221]: from='mgr.14164 172.21.15.62:0/570876096' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "who": "osd/host:smithi062", "name": "osd_memory_target"}]: dispatch 2024-04-20T21:19:52.943 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:52 smithi062 ceph-mon[16221]: from='mgr.14164 172.21.15.62:0/570876096' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:19:52.943 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:52 smithi062 ceph-mon[16221]: from='mgr.14164 172.21.15.62:0/570876096' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi062", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-20T21:19:52.943 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:52 smithi062 ceph-mon[16221]: from='mgr.14164 172.21.15.62:0/570876096' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi062", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]': finished 2024-04-20T21:19:52.943 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:52 smithi062 ceph-mon[16221]: from='mgr.14164 172.21.15.62:0/570876096' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:19:52.944 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:52 smithi062 ceph-mon[16221]: Deploying daemon ceph-exporter.smithi062 on smithi062 2024-04-20T21:19:54.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:54.269+0000 7f13df200700 1 -- 172.21.15.62:0/284945762 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f13d8106440 msgr2=0x7f13d8106850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:19:54.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:54.269+0000 7f13df200700 1 --2- 172.21.15.62:0/284945762 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f13d8106440 0x7f13d8106850 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f13c8009910 tx=0x7f13c8009c20 comp rx=0 tx=0).stop 2024-04-20T21:19:54.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:54.269+0000 7f13df200700 1 -- 172.21.15.62:0/284945762 shutdown_connections 2024-04-20T21:19:54.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:54.269+0000 7f13df200700 1 --2- 172.21.15.62:0/284945762 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f13d8106440 0x7f13d8106850 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:19:54.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:54.269+0000 7f13df200700 1 -- 172.21.15.62:0/284945762 >> 172.21.15.62:0/284945762 conn(0x7f13d80756e0 msgr2=0x7f13d8077b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:19:54.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:54.269+0000 7f13df200700 1 -- 172.21.15.62:0/284945762 shutdown_connections 2024-04-20T21:19:54.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:54.269+0000 7f13df200700 1 -- 172.21.15.62:0/284945762 wait complete. 2024-04-20T21:19:54.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:54.271+0000 7f13df200700 1 Processor -- start 2024-04-20T21:19:54.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:54.271+0000 7f13df200700 1 -- start start 2024-04-20T21:19:54.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:54.271+0000 7f13df200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f13d8106440 0x7f13d819b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:19:54.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:54.271+0000 7f13df200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13d819bae0 con 0x7f13d8106440 2024-04-20T21:19:54.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:54.271+0000 7f13d7e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f13d8106440 0x7f13d819b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:19:54.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:54.271+0000 7f13d7e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f13d8106440 0x7f13d819b5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:57210/0 (socket says 172.21.15.62:57210) 2024-04-20T21:19:54.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:54.271+0000 7f13d7e00700 1 -- 172.21.15.62:0/1129775361 learned_addr learned my addr 172.21.15.62:0/1129775361 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:19:54.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:54.272+0000 7f13d7e00700 1 -- 172.21.15.62:0/1129775361 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13c80095f0 con 0x7f13d8106440 2024-04-20T21:19:54.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:54.272+0000 7f13d7e00700 1 --2- 172.21.15.62:0/1129775361 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f13d8106440 0x7f13d819b5a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f13c8005950 tx=0x7f13c8009e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:19:54.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:54.272+0000 7f13d4c00700 1 -- 172.21.15.62:0/1129775361 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f13c801d070 con 0x7f13d8106440 2024-04-20T21:19:54.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:54.273+0000 7f13d4c00700 1 -- 172.21.15.62:0/1129775361 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f13c800be10 con 0x7f13d8106440 2024-04-20T21:19:54.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:54.273+0000 7f13d4c00700 1 -- 172.21.15.62:0/1129775361 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f13c800f460 con 0x7f13d8106440 2024-04-20T21:19:54.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:54.273+0000 7f13df200700 1 -- 172.21.15.62:0/1129775361 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13d819bce0 con 0x7f13d8106440 2024-04-20T21:19:54.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:54.273+0000 7f13df200700 1 -- 172.21.15.62:0/1129775361 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13d819c180 con 0x7f13d8106440 2024-04-20T21:19:54.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:54.274+0000 7f13d4c00700 1 -- 172.21.15.62:0/1129775361 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 13) v1 ==== 45140+0+0 (secure 0 0 0) 0x7f13c80227c0 con 0x7f13d8106440 2024-04-20T21:19:54.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:54.274+0000 7f13df200700 1 -- 172.21.15.62:0/1129775361 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13d8195110 con 0x7f13d8106440 2024-04-20T21:19:54.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:54.274+0000 7f13d4c00700 1 --2- 172.21.15.62:0/1129775361 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7f13bc0384e0 0x7f13bc03a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:19:54.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:54.275+0000 7f13d4c00700 1 -- 172.21.15.62:0/1129775361 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f13c804c430 con 0x7f13d8106440 2024-04-20T21:19:54.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:54.275+0000 7f13d7400700 1 --2- 172.21.15.62:0/1129775361 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7f13bc0384e0 0x7f13bc03a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:19:54.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:54.276+0000 7f13d7400700 1 --2- 172.21.15.62:0/1129775361 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7f13bc0384e0 0x7f13bc03a990 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f13c4006fd0 tx=0x7f13c4006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:19:54.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:54.280+0000 7f13d4c00700 1 -- 172.21.15.62:0/1129775361 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f13c802ba00 con 0x7f13d8106440 2024-04-20T21:19:54.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:54.511+0000 7f13df200700 1 -- 172.21.15.62:0/1129775361 --> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f13d8061220 con 0x7f13bc0384e0 2024-04-20T21:19:54.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:54.517+0000 7f13d4c00700 1 -- 172.21.15.62:0/1129775361 <== mgr.14164 v2:172.21.15.62:6800/831506538 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f13d8061220 con 0x7f13bc0384e0 2024-04-20T21:19:54.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:54.520+0000 7f13df200700 1 -- 172.21.15.62:0/1129775361 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7f13bc0384e0 msgr2=0x7f13bc03a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:19:54.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:54.520+0000 7f13df200700 1 --2- 172.21.15.62:0/1129775361 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7f13bc0384e0 0x7f13bc03a990 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f13c4006fd0 tx=0x7f13c4006e40 comp rx=0 tx=0).stop 2024-04-20T21:19:54.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:54.520+0000 7f13df200700 1 -- 172.21.15.62:0/1129775361 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f13d8106440 msgr2=0x7f13d819b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:19:54.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:54.520+0000 7f13df200700 1 --2- 172.21.15.62:0/1129775361 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f13d8106440 0x7f13d819b5a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f13c8005950 tx=0x7f13c8009e60 comp rx=0 tx=0).stop 2024-04-20T21:19:54.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:54.520+0000 7f13df200700 1 -- 172.21.15.62:0/1129775361 shutdown_connections 2024-04-20T21:19:54.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:54.520+0000 7f13df200700 1 --2- 172.21.15.62:0/1129775361 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7f13bc0384e0 0x7f13bc03a990 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:19:54.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:54.520+0000 7f13df200700 1 --2- 172.21.15.62:0/1129775361 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f13d8106440 0x7f13d819b5a0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:19:54.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:54.520+0000 7f13df200700 1 -- 172.21.15.62:0/1129775361 >> 172.21.15.62:0/1129775361 conn(0x7f13d80756e0 msgr2=0x7f13d80773a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:19:54.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:54.520+0000 7f13df200700 1 -- 172.21.15.62:0/1129775361 shutdown_connections 2024-04-20T21:19:54.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:54.520+0000 7f13df200700 1 -- 172.21.15.62:0/1129775361 wait complete. 2024-04-20T21:19:55.506 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi189 2024-04-20T21:19:55.507 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-20T21:19:55.507 DEBUG:teuthology.orchestra.run.smithi189:> dd of=/etc/ceph/ceph.conf 2024-04-20T21:19:55.564 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-20T21:19:55.564 DEBUG:teuthology.orchestra.run.smithi189:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-20T21:19:55.652 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:55 smithi062 ceph-mon[16221]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:19:55.652 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:55 smithi062 ceph-mon[16221]: from='mgr.14164 172.21.15.62:0/570876096' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:19:55.652 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:55 smithi062 ceph-mon[16221]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:19:55.659 INFO:tasks.cephadm:Adding host smithi189 to orchestrator... 2024-04-20T21:19:55.659 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -- ceph orch host add smithi189 2024-04-20T21:19:56.470 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T21:19:57.648 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:57 smithi062 ceph-mon[16221]: from='mgr.14164 172.21.15.62:0/570876096' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:19:57.649 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:57 smithi062 ceph-mon[16221]: from='mgr.14164 172.21.15.62:0/570876096' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:19:57.649 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:57 smithi062 ceph-mon[16221]: from='mgr.14164 172.21.15.62:0/570876096' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:19:57.649 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:57 smithi062 ceph-mon[16221]: from='mgr.14164 172.21.15.62:0/570876096' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:19:57.649 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:57 smithi062 ceph-mon[16221]: from='mgr.14164 172.21.15.62:0/570876096' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi062", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-20T21:19:57.649 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:57 smithi062 ceph-mon[16221]: from='mgr.14164 172.21.15.62:0/570876096' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi062", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-04-20T21:19:57.649 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:57 smithi062 ceph-mon[16221]: from='mgr.14164 172.21.15.62:0/570876096' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:19:57.649 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:57 smithi062 ceph-mon[16221]: Deploying daemon crash.smithi062 on smithi062 2024-04-20T21:19:57.649 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:19:57 smithi062 ceph-mon[16221]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:19:58.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:58.853+0000 7eff44600700 1 -- 172.21.15.62:0/965872154 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff3c0ffe20 msgr2=0x7eff3c100230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:19:58.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:58.853+0000 7eff44600700 1 --2- 172.21.15.62:0/965872154 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff3c0ffe20 0x7eff3c100230 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7eff30009910 tx=0x7eff30009c20 comp rx=0 tx=0).stop 2024-04-20T21:19:58.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:58.853+0000 7eff44600700 1 -- 172.21.15.62:0/965872154 shutdown_connections 2024-04-20T21:19:58.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:58.853+0000 7eff44600700 1 --2- 172.21.15.62:0/965872154 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff3c0ffe20 0x7eff3c100230 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:19:58.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:58.853+0000 7eff44600700 1 -- 172.21.15.62:0/965872154 >> 172.21.15.62:0/965872154 conn(0x7eff3c0fb3d0 msgr2=0x7eff3c0fd800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:19:58.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:58.853+0000 7eff44600700 1 -- 172.21.15.62:0/965872154 shutdown_connections 2024-04-20T21:19:58.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:58.853+0000 7eff44600700 1 -- 172.21.15.62:0/965872154 wait complete. 2024-04-20T21:19:58.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:58.854+0000 7eff44600700 1 Processor -- start 2024-04-20T21:19:58.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:58.855+0000 7eff44600700 1 -- start start 2024-04-20T21:19:58.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:58.855+0000 7eff44600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff3c0ffe20 0x7eff3c194f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:19:58.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:58.855+0000 7eff44600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff3c1954c0 con 0x7eff3c0ffe20 2024-04-20T21:19:58.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:58.855+0000 7eff41c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff3c0ffe20 0x7eff3c194f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:19:58.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:58.855+0000 7eff41c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff3c0ffe20 0x7eff3c194f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:57224/0 (socket says 172.21.15.62:57224) 2024-04-20T21:19:58.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:58.855+0000 7eff41c00700 1 -- 172.21.15.62:0/3411812108 learned_addr learned my addr 172.21.15.62:0/3411812108 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:19:58.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:58.856+0000 7eff41c00700 1 -- 172.21.15.62:0/3411812108 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff300095f0 con 0x7eff3c0ffe20 2024-04-20T21:19:58.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:58.856+0000 7eff41c00700 1 --2- 172.21.15.62:0/3411812108 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff3c0ffe20 0x7eff3c194f80 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7eff300092e0 tx=0x7eff30009e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:19:58.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:58.856+0000 7eff3a000700 1 -- 172.21.15.62:0/3411812108 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff3001d070 con 0x7eff3c0ffe20 2024-04-20T21:19:58.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:58.856+0000 7eff3a000700 1 -- 172.21.15.62:0/3411812108 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7eff3000be10 con 0x7eff3c0ffe20 2024-04-20T21:19:58.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:58.856+0000 7eff44600700 1 -- 172.21.15.62:0/3411812108 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff3c1956c0 con 0x7eff3c0ffe20 2024-04-20T21:19:58.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:58.856+0000 7eff3a000700 1 -- 172.21.15.62:0/3411812108 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff3000f460 con 0x7eff3c0ffe20 2024-04-20T21:19:58.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:58.856+0000 7eff44600700 1 -- 172.21.15.62:0/3411812108 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff3c195b60 con 0x7eff3c0ffe20 2024-04-20T21:19:58.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:58.858+0000 7eff3a000700 1 -- 172.21.15.62:0/3411812108 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 13) v1 ==== 45140+0+0 (secure 0 0 0) 0x7eff3000b960 con 0x7eff3c0ffe20 2024-04-20T21:19:58.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:58.858+0000 7eff44600700 1 -- 172.21.15.62:0/3411812108 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff3c18ea80 con 0x7eff3c0ffe20 2024-04-20T21:19:58.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:58.858+0000 7eff3a000700 1 --2- 172.21.15.62:0/3411812108 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7eff240384e0 0x7eff2403a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:19:58.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:58.858+0000 7eff3a000700 1 -- 172.21.15.62:0/3411812108 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7eff3004c280 con 0x7eff3c0ffe20 2024-04-20T21:19:58.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:58.858+0000 7eff41200700 1 --2- 172.21.15.62:0/3411812108 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7eff240384e0 0x7eff2403a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:19:58.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:58.859+0000 7eff41200700 1 --2- 172.21.15.62:0/3411812108 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7eff240384e0 0x7eff2403a990 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7eff28006fd0 tx=0x7eff28006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:19:58.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:58.864+0000 7eff3a000700 1 -- 172.21.15.62:0/3411812108 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7eff30022790 con 0x7eff3c0ffe20 2024-04-20T21:19:59.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:19:59.091+0000 7eff44600700 1 -- 172.21.15.62:0/3411812108 --> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi189", "target": ["mon-mgr", ""]}) v1 -- 0x7eff3c1008b0 con 0x7eff240384e0 2024-04-20T21:20:00.272 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:20:00 smithi062 ceph-mon[16221]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:20:00.273 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:20:00 smithi062 ceph-mon[16221]: from='client.14191 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi189", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:20:01.446 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:20:01 smithi062 ceph-mon[16221]: from='mgr.14164 172.21.15.62:0/570876096' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:20:01.446 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:20:01 smithi062 ceph-mon[16221]: from='mgr.14164 172.21.15.62:0/570876096' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:20:01.446 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:20:01 smithi062 ceph-mon[16221]: from='mgr.14164 172.21.15.62:0/570876096' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:20:01.446 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:20:01 smithi062 ceph-mon[16221]: from='mgr.14164 172.21.15.62:0/570876096' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:20:01.446 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:20:01 smithi062 ceph-mon[16221]: from='mgr.14164 172.21.15.62:0/570876096' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:20:02.176 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:20:02 smithi062 ceph-mon[16221]: Deploying cephadm binary to smithi189 2024-04-20T21:20:02.176 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:20:02 smithi062 ceph-mon[16221]: Deploying daemon node-exporter.smithi062 on smithi062 2024-04-20T21:20:02.176 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:20:02 smithi062 ceph-mon[16221]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:20:03.383 INFO:teuthology.orchestra.run.smithi062.stdout:Added host 'smithi189' with addr '172.21.15.189' 2024-04-20T21:20:03.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:20:03.378+0000 7eff3a000700 1 -- 172.21.15.62:0/3411812108 <== mgr.14164 v2:172.21.15.62:6800/831506538 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7eff3c1008b0 con 0x7eff240384e0 2024-04-20T21:20:03.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:20:03.382+0000 7eff44600700 1 -- 172.21.15.62:0/3411812108 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7eff240384e0 msgr2=0x7eff2403a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:20:03.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:20:03.382+0000 7eff44600700 1 --2- 172.21.15.62:0/3411812108 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7eff240384e0 0x7eff2403a990 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7eff28006fd0 tx=0x7eff28006e40 comp rx=0 tx=0).stop 2024-04-20T21:20:03.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:20:03.383+0000 7eff44600700 1 -- 172.21.15.62:0/3411812108 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff3c0ffe20 msgr2=0x7eff3c194f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:20:03.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:20:03.383+0000 7eff44600700 1 --2- 172.21.15.62:0/3411812108 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff3c0ffe20 0x7eff3c194f80 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7eff300092e0 tx=0x7eff30009e60 comp rx=0 tx=0).stop 2024-04-20T21:20:03.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:20:03.383+0000 7eff44600700 1 -- 172.21.15.62:0/3411812108 shutdown_connections 2024-04-20T21:20:03.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:20:03.383+0000 7eff44600700 1 --2- 172.21.15.62:0/3411812108 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7eff240384e0 0x7eff2403a990 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:20:03.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:20:03.383+0000 7eff44600700 1 --2- 172.21.15.62:0/3411812108 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff3c0ffe20 0x7eff3c194f80 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:20:03.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:20:03.383+0000 7eff44600700 1 -- 172.21.15.62:0/3411812108 >> 172.21.15.62:0/3411812108 conn(0x7eff3c0fb3d0 msgr2=0x7eff3c0fc080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:20:03.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:20:03.383+0000 7eff44600700 1 -- 172.21.15.62:0/3411812108 shutdown_connections 2024-04-20T21:20:03.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:20:03.384+0000 7eff44600700 1 -- 172.21.15.62:0/3411812108 wait complete. 2024-04-20T21:20:04.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:20:04 smithi062 ceph-mon[16221]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:20:04.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:20:04 smithi062 ceph-mon[16221]: from='mgr.14164 172.21.15.62:0/570876096' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:20:05.063 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-20T21:20:05.105 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:20:05 smithi062 ceph-mon[16221]: Added host smithi189 2024-04-20T21:20:05.105 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:20:05 smithi062 ceph-mon[16221]: from='mgr.14164 172.21.15.62:0/570876096' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:20:06.156 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T21:20:06.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:20:06 smithi062 ceph-mon[16221]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:20:07.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:20:07 smithi062 ceph-mon[16221]: from='mgr.14164 172.21.15.62:0/570876096' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:20:07.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:20:07 smithi062 ceph-mon[16221]: from='mgr.14164 172.21.15.62:0/570876096' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:20:07.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:20:07 smithi062 ceph-mon[16221]: from='mgr.14164 172.21.15.62:0/570876096' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:20:07.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:20:07 smithi062 ceph-mon[16221]: from='mgr.14164 172.21.15.62:0/570876096' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:20:07.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:20:07 smithi062 ceph-mon[16221]: Deploying daemon alertmanager.smithi062 on smithi062 2024-04-20T21:20:07.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:20:07 smithi062 ceph-mon[16221]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:20:08.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:20:08.109+0000 7f92d2200700 1 -- 172.21.15.62:0/1605744661 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f92cc0ffe20 msgr2=0x7f92cc100230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:20:08.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:20:08.109+0000 7f92d2200700 1 --2- 172.21.15.62:0/1605744661 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f92cc0ffe20 0x7f92cc100230 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f92bc009910 tx=0x7f92bc009c20 comp rx=0 tx=0).stop 2024-04-20T21:20:08.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:20:08.109+0000 7f92d2200700 1 -- 172.21.15.62:0/1605744661 shutdown_connections 2024-04-20T21:20:08.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:20:08.109+0000 7f92d2200700 1 --2- 172.21.15.62:0/1605744661 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f92cc0ffe20 0x7f92cc100230 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:20:08.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:20:08.109+0000 7f92d2200700 1 -- 172.21.15.62:0/1605744661 >> 172.21.15.62:0/1605744661 conn(0x7f92cc0fb3d0 msgr2=0x7f92cc0fd800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:20:08.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:20:08.109+0000 7f92d2200700 1 -- 172.21.15.62:0/1605744661 shutdown_connections 2024-04-20T21:20:08.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:20:08.109+0000 7f92d2200700 1 -- 172.21.15.62:0/1605744661 wait complete. 2024-04-20T21:20:08.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:20:08.110+0000 7f92d2200700 1 Processor -- start 2024-04-20T21:20:08.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:20:08.111+0000 7f92d2200700 1 -- start start 2024-04-20T21:20:08.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:20:08.111+0000 7f92d2200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f92cc0ffe20 0x7f92cc197170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:20:08.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:20:08.111+0000 7f92d2200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92cc1976b0 con 0x7f92cc0ffe20 2024-04-20T21:20:08.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:20:08.111+0000 7f92cb400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f92cc0ffe20 0x7f92cc197170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:20:08.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:20:08.111+0000 7f92cb400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f92cc0ffe20 0x7f92cc197170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51674/0 (socket says 172.21.15.62:51674) 2024-04-20T21:20:08.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:20:08.111+0000 7f92cb400700 1 -- 172.21.15.62:0/1344756115 learned_addr learned my addr 172.21.15.62:0/1344756115 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:20:08.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:20:08.112+0000 7f92cb400700 1 -- 172.21.15.62:0/1344756115 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92bc0095f0 con 0x7f92cc0ffe20 2024-04-20T21:20:08.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:20:08.112+0000 7f92cb400700 1 --2- 172.21.15.62:0/1344756115 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f92cc0ffe20 0x7f92cc197170 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f92bc0092e0 tx=0x7f92bc009e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:20:08.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:20:08.113+0000 7f92c3e00700 1 -- 172.21.15.62:0/1344756115 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f92bc01d070 con 0x7f92cc0ffe20 2024-04-20T21:20:08.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:20:08.113+0000 7f92c3e00700 1 -- 172.21.15.62:0/1344756115 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f92bc00be10 con 0x7f92cc0ffe20 2024-04-20T21:20:08.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:20:08.113+0000 7f92d2200700 1 -- 172.21.15.62:0/1344756115 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92cc1978b0 con 0x7f92cc0ffe20 2024-04-20T21:20:08.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:20:08.113+0000 7f92c3e00700 1 -- 172.21.15.62:0/1344756115 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f92bc00f460 con 0x7f92cc0ffe20 2024-04-20T21:20:08.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:20:08.113+0000 7f92d2200700 1 -- 172.21.15.62:0/1344756115 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92cc197d50 con 0x7f92cc0ffe20 2024-04-20T21:20:08.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:20:08.114+0000 7f92c3e00700 1 -- 172.21.15.62:0/1344756115 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 13) v1 ==== 45140+0+0 (secure 0 0 0) 0x7f92bc00f5c0 con 0x7f92cc0ffe20 2024-04-20T21:20:08.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:20:08.114+0000 7f92d2200700 1 -- 172.21.15.62:0/1344756115 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92cc190e80 con 0x7f92cc0ffe20 2024-04-20T21:20:08.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:20:08.115+0000 7f92c3e00700 1 --2- 172.21.15.62:0/1344756115 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7f92b00384e0 0x7f92b003a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:20:08.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:20:08.115+0000 7f92c3e00700 1 -- 172.21.15.62:0/1344756115 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f92bc04c2f0 con 0x7f92cc0ffe20 2024-04-20T21:20:08.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:20:08.115+0000 7f92caa00700 1 --2- 172.21.15.62:0/1344756115 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7f92b00384e0 0x7f92b003a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:20:08.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:20:08.116+0000 7f92caa00700 1 --2- 172.21.15.62:0/1344756115 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7f92b00384e0 0x7f92b003a990 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f92b8006fd0 tx=0x7f92b8006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:20:08.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:20:08.120+0000 7f92c3e00700 1 -- 172.21.15.62:0/1344756115 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f92bc020960 con 0x7f92cc0ffe20 2024-04-20T21:20:08.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:20:08.363+0000 7f92d2200700 1 -- 172.21.15.62:0/1344756115 --> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f92cc1008b0 con 0x7f92b00384e0 2024-04-20T21:20:08.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:20:08.365+0000 7f92c3e00700 1 -- 172.21.15.62:0/1344756115 <== mgr.14164 v2:172.21.15.62:6800/831506538 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+160 (secure 0 0 0) 0x7f92cc1008b0 con 0x7f92b00384e0 2024-04-20T21:20:08.365 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:20:08.365 INFO:teuthology.orchestra.run.smithi062.stdout:[{"addr": "172.21.15.62", "hostname": "smithi062", "labels": [], "status": ""}, {"addr": "172.21.15.189", "hostname": "smithi189", "labels": [], "status": ""}] 2024-04-20T21:20:08.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:20:08.369+0000 7f92d2200700 1 -- 172.21.15.62:0/1344756115 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7f92b00384e0 msgr2=0x7f92b003a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:20:08.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:20:08.369+0000 7f92d2200700 1 --2- 172.21.15.62:0/1344756115 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7f92b00384e0 0x7f92b003a990 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f92b8006fd0 tx=0x7f92b8006e40 comp rx=0 tx=0).stop 2024-04-20T21:20:08.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:20:08.369+0000 7f92d2200700 1 -- 172.21.15.62:0/1344756115 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f92cc0ffe20 msgr2=0x7f92cc197170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:20:08.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:20:08.369+0000 7f92d2200700 1 --2- 172.21.15.62:0/1344756115 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f92cc0ffe20 0x7f92cc197170 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f92bc0092e0 tx=0x7f92bc009e60 comp rx=0 tx=0).stop 2024-04-20T21:20:08.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:20:08.369+0000 7f92d2200700 1 -- 172.21.15.62:0/1344756115 shutdown_connections 2024-04-20T21:20:08.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:20:08.369+0000 7f92d2200700 1 --2- 172.21.15.62:0/1344756115 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7f92b00384e0 0x7f92b003a990 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:20:08.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:20:08.369+0000 7f92d2200700 1 --2- 172.21.15.62:0/1344756115 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f92cc0ffe20 0x7f92cc197170 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:20:08.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:20:08.369+0000 7f92d2200700 1 -- 172.21.15.62:0/1344756115 >> 172.21.15.62:0/1344756115 conn(0x7f92cc0fb3d0 msgr2=0x7f92cc0fd800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:20:08.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:20:08.370+0000 7f92d2200700 1 -- 172.21.15.62:0/1344756115 shutdown_connections 2024-04-20T21:20:08.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:20:08.370+0000 7f92d2200700 1 -- 172.21.15.62:0/1344756115 wait complete. 2024-04-20T21:20:09.451 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-20T21:20:09.452 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -- ceph osd crush tunables default 2024-04-20T21:20:09.913 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T21:20:10.260 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:20:10 smithi062 ceph-mon[16221]: from='client.14193 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T21:20:10.260 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:20:10 smithi062 ceph-mon[16221]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:20:11.468 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:20:11 smithi062 ceph-mon[16221]: from='mgr.14164 172.21.15.62:0/570876096' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:20:12.304 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:20:12 smithi062 ceph-mon[16221]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:20:13.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:20:13.486+0000 7f35e2600700 1 -- 172.21.15.62:0/4054719969 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f35dc106440 msgr2=0x7f35dc106850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:20:13.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:20:13.486+0000 7f35e2600700 1 --2- 172.21.15.62:0/4054719969 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f35dc106440 0x7f35dc106850 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f35cc009910 tx=0x7f35cc009c20 comp rx=0 tx=0).stop 2024-04-20T21:20:13.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:20:13.486+0000 7f35e2600700 1 -- 172.21.15.62:0/4054719969 shutdown_connections 2024-04-20T21:20:13.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:20:13.486+0000 7f35e2600700 1 --2- 172.21.15.62:0/4054719969 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f35dc106440 0x7f35dc106850 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:20:13.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:20:13.486+0000 7f35e2600700 1 -- 172.21.15.62:0/4054719969 >> 172.21.15.62:0/4054719969 conn(0x7f35dc0756e0 msgr2=0x7f35dc077b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:20:13.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:20:13.486+0000 7f35e2600700 1 -- 172.21.15.62:0/4054719969 shutdown_connections 2024-04-20T21:20:13.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:20:13.486+0000 7f35e2600700 1 -- 172.21.15.62:0/4054719969 wait complete. 2024-04-20T21:20:13.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:20:13.488+0000 7f35e2600700 1 Processor -- start 2024-04-20T21:20:13.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:20:13.488+0000 7f35e2600700 1 -- start start 2024-04-20T21:20:13.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:20:13.488+0000 7f35e2600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f35dc106440 0x7f35dc19b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:20:13.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:20:13.488+0000 7f35e2600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35dc19bae0 con 0x7f35dc106440 2024-04-20T21:20:13.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:20:13.489+0000 7f35db400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f35dc106440 0x7f35dc19b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:20:13.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:20:13.489+0000 7f35db400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f35dc106440 0x7f35dc19b5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33432/0 (socket says 172.21.15.62:33432) 2024-04-20T21:20:13.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:20:13.489+0000 7f35db400700 1 -- 172.21.15.62:0/3215201747 learned_addr learned my addr 172.21.15.62:0/3215201747 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:20:13.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:20:13.489+0000 7f35db400700 1 -- 172.21.15.62:0/3215201747 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35cc0095f0 con 0x7f35dc106440 2024-04-20T21:20:13.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:20:13.489+0000 7f35db400700 1 --2- 172.21.15.62:0/3215201747 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f35dc106440 0x7f35dc19b5a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f35cc005950 tx=0x7f35cc009e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:20:13.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:20:13.490+0000 7f35d8c00700 1 -- 172.21.15.62:0/3215201747 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f35cc01d070 con 0x7f35dc106440 2024-04-20T21:20:13.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:20:13.490+0000 7f35d8c00700 1 -- 172.21.15.62:0/3215201747 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f35cc00be10 con 0x7f35dc106440 2024-04-20T21:20:13.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:20:13.490+0000 7f35d8c00700 1 -- 172.21.15.62:0/3215201747 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f35cc00f460 con 0x7f35dc106440 2024-04-20T21:20:13.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:20:13.490+0000 7f35e2600700 1 -- 172.21.15.62:0/3215201747 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35dc19bce0 con 0x7f35dc106440 2024-04-20T21:20:13.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:20:13.490+0000 7f35e2600700 1 -- 172.21.15.62:0/3215201747 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35dc19c180 con 0x7f35dc106440 2024-04-20T21:20:13.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:20:13.491+0000 7f35d8c00700 1 -- 172.21.15.62:0/3215201747 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 13) v1 ==== 45140+0+0 (secure 0 0 0) 0x7f35cc0227c0 con 0x7f35dc106440 2024-04-20T21:20:13.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:20:13.491+0000 7f35e2600700 1 -- 172.21.15.62:0/3215201747 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35dc195110 con 0x7f35dc106440 2024-04-20T21:20:13.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:20:13.492+0000 7f35d8c00700 1 --2- 172.21.15.62:0/3215201747 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7f35c00384e0 0x7f35c003a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:20:13.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:20:13.492+0000 7f35d8c00700 1 -- 172.21.15.62:0/3215201747 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f35cc04c410 con 0x7f35dc106440 2024-04-20T21:20:13.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:20:13.492+0000 7f35d2a00700 1 --2- 172.21.15.62:0/3215201747 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7f35c00384e0 0x7f35c003a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:20:13.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:20:13.493+0000 7f35d2a00700 1 --2- 172.21.15.62:0/3215201747 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7f35c00384e0 0x7f35c003a990 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f35c4006fd0 tx=0x7f35c4006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:20:13.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:20:13.497+0000 7f35d8c00700 1 -- 172.21.15.62:0/3215201747 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f35cc02ba00 con 0x7f35dc106440 2024-04-20T21:20:13.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:20:13.729+0000 7f35e2600700 1 -- 172.21.15.62:0/3215201747 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f35dc02cc30 con 0x7f35dc106440 2024-04-20T21:20:14.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:20:14.067+0000 7f35d8c00700 1 -- 172.21.15.62:0/3215201747 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7f35cc027020 con 0x7f35dc106440 2024-04-20T21:20:14.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:20:14.071+0000 7f35e2600700 1 -- 172.21.15.62:0/3215201747 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7f35c00384e0 msgr2=0x7f35c003a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:20:14.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:20:14.071+0000 7f35e2600700 1 --2- 172.21.15.62:0/3215201747 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7f35c00384e0 0x7f35c003a990 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f35c4006fd0 tx=0x7f35c4006e40 comp rx=0 tx=0).stop 2024-04-20T21:20:14.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:20:14.071+0000 7f35e2600700 1 -- 172.21.15.62:0/3215201747 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f35dc106440 msgr2=0x7f35dc19b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:20:14.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:20:14.071+0000 7f35e2600700 1 --2- 172.21.15.62:0/3215201747 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f35dc106440 0x7f35dc19b5a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f35cc005950 tx=0x7f35cc009e60 comp rx=0 tx=0).stop 2024-04-20T21:20:14.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:20:14.072+0000 7f35e2600700 1 -- 172.21.15.62:0/3215201747 shutdown_connections 2024-04-20T21:20:14.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:20:14.072+0000 7f35e2600700 1 --2- 172.21.15.62:0/3215201747 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7f35c00384e0 0x7f35c003a990 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:20:14.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:20:14.072+0000 7f35e2600700 1 --2- 172.21.15.62:0/3215201747 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f35dc106440 0x7f35dc19b5a0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:20:14.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:20:14.072+0000 7f35e2600700 1 -- 172.21.15.62:0/3215201747 >> 172.21.15.62:0/3215201747 conn(0x7f35dc0756e0 msgr2=0x7f35dc0773a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:20:14.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:20:14.073+0000 7f35e2600700 1 -- 172.21.15.62:0/3215201747 shutdown_connections 2024-04-20T21:20:14.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:20:14.073+0000 7f35e2600700 1 -- 172.21.15.62:0/3215201747 wait complete. 2024-04-20T21:20:14.074 INFO:teuthology.orchestra.run.smithi062.stderr:adjusted tunables profile to default 2024-04-20T21:20:14.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:20:14 smithi062 ceph-mon[16221]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:20:14.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:20:14 smithi062 ceph-mon[16221]: from='client.? 172.21.15.62:0/3215201747' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-20T21:20:15.130 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:20:15 smithi062 ceph-mon[16221]: from='client.? 172.21.15.62:0/3215201747' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-20T21:20:15.130 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:20:15 smithi062 ceph-mon[16221]: osdmap e4: 0 total, 0 up, 0 in 2024-04-20T21:20:15.176 INFO:tasks.cephadm:Adding mon.smithi062 on smithi062 2024-04-20T21:20:15.176 INFO:tasks.cephadm:Adding mon.smithi189 on smithi189 2024-04-20T21:20:15.176 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -- ceph orch apply mon '2;smithi062:172.21.15.62=smithi062;smithi189:172.21.15.189=smithi189' 2024-04-20T21:20:15.564 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-20T21:20:15.671 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-20T21:20:16.411 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:20:16 smithi062 ceph-mon[16221]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:20:17.845 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:17.845+0000 7f4f90600700 1 -- 172.21.15.189:0/1926911856 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f880fed30 msgr2=0x7f4f880ff140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:20:17.845 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:17.845+0000 7f4f90600700 1 --2- 172.21.15.189:0/1926911856 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f880fed30 0x7f4f880ff140 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f4f7c009910 tx=0x7f4f7c009c20 comp rx=0 tx=0).stop 2024-04-20T21:20:17.845 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:17.845+0000 7f4f90600700 1 -- 172.21.15.189:0/1926911856 shutdown_connections 2024-04-20T21:20:17.845 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:17.845+0000 7f4f90600700 1 --2- 172.21.15.189:0/1926911856 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f880fed30 0x7f4f880ff140 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:20:17.845 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:17.845+0000 7f4f90600700 1 -- 172.21.15.189:0/1926911856 >> 172.21.15.189:0/1926911856 conn(0x7f4f880fa2c0 msgr2=0x7f4f880fc710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:20:17.845 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:17.845+0000 7f4f90600700 1 -- 172.21.15.189:0/1926911856 shutdown_connections 2024-04-20T21:20:17.845 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:17.845+0000 7f4f90600700 1 -- 172.21.15.189:0/1926911856 wait complete. 2024-04-20T21:20:17.846 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:17.846+0000 7f4f90600700 1 Processor -- start 2024-04-20T21:20:17.846 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:17.847+0000 7f4f90600700 1 -- start start 2024-04-20T21:20:17.846 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:17.847+0000 7f4f90600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f880fed30 0x7f4f88194fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:20:17.847 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:17.847+0000 7f4f90600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f88195510 con 0x7f4f880fed30 2024-04-20T21:20:17.847 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:17.848+0000 7f4f8dc00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f880fed30 0x7f4f88194fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:20:17.847 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:17.848+0000 7f4f8dc00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f880fed30 0x7f4f88194fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.189:53796/0 (socket says 172.21.15.189:53796) 2024-04-20T21:20:17.847 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:17.848+0000 7f4f8dc00700 1 -- 172.21.15.189:0/4281785132 learned_addr learned my addr 172.21.15.189:0/4281785132 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-20T21:20:17.848 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:17.848+0000 7f4f8dc00700 1 -- 172.21.15.189:0/4281785132 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f7c0095f0 con 0x7f4f880fed30 2024-04-20T21:20:17.848 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:17.849+0000 7f4f8dc00700 1 --2- 172.21.15.189:0/4281785132 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f880fed30 0x7f4f88194fd0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f4f7c005950 tx=0x7f4f7c009e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:20:17.849 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:17.849+0000 7f4f86000700 1 -- 172.21.15.189:0/4281785132 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4f7c01d070 con 0x7f4f880fed30 2024-04-20T21:20:17.849 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:17.849+0000 7f4f90600700 1 -- 172.21.15.189:0/4281785132 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f88195710 con 0x7f4f880fed30 2024-04-20T21:20:17.849 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:17.849+0000 7f4f86000700 1 -- 172.21.15.189:0/4281785132 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f4f7c00be10 con 0x7f4f880fed30 2024-04-20T21:20:17.849 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:17.849+0000 7f4f86000700 1 -- 172.21.15.189:0/4281785132 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4f7c00f460 con 0x7f4f880fed30 2024-04-20T21:20:17.849 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:17.849+0000 7f4f90600700 1 -- 172.21.15.189:0/4281785132 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f88195bb0 con 0x7f4f880fed30 2024-04-20T21:20:17.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:17.850+0000 7f4f90600700 1 -- 172.21.15.189:0/4281785132 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f8818ea20 con 0x7f4f880fed30 2024-04-20T21:20:17.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:17.851+0000 7f4f86000700 1 -- 172.21.15.189:0/4281785132 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 13) v1 ==== 45140+0+0 (secure 0 0 0) 0x7f4f7c0227c0 con 0x7f4f880fed30 2024-04-20T21:20:17.850 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:17.851+0000 7f4f86000700 1 --2- 172.21.15.189:0/4281785132 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7f4f70038490 0x7f4f7003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:20:17.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:17.851+0000 7f4f86000700 1 -- 172.21.15.189:0/4281785132 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f4f7c04c4b0 con 0x7f4f880fed30 2024-04-20T21:20:17.852 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:17.852+0000 7f4f8d200700 1 --2- 172.21.15.189:0/4281785132 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7f4f70038490 0x7f4f7003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:20:17.852 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:17.853+0000 7f4f8d200700 1 --2- 172.21.15.189:0/4281785132 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7f4f70038490 0x7f4f7003a940 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f4f74006fd0 tx=0x7f4f74006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:20:17.867 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:17.867+0000 7f4f86000700 1 -- 172.21.15.189:0/4281785132 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f4f7c02bba0 con 0x7f4f880fed30 2024-04-20T21:20:17.868 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:20:17 smithi062 ceph-mon[16221]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:20:18.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:18.093+0000 7f4f90600700 1 -- 172.21.15.189:0/4281785132 --> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi062:172.21.15.62=smithi062;smithi189:172.21.15.189=smithi189", "target": ["mon-mgr", ""]}) v1 -- 0x7f4f88061300 con 0x7f4f70038490 2024-04-20T21:20:18.104 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:18.105+0000 7f4f86000700 1 -- 172.21.15.189:0/4281785132 <== mgr.14164 v2:172.21.15.62:6800/831506538 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f4f88061300 con 0x7f4f70038490 2024-04-20T21:20:18.105 INFO:teuthology.orchestra.run.smithi189.stdout:Scheduled mon update... 2024-04-20T21:20:18.108 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:18.108+0000 7f4f90600700 1 -- 172.21.15.189:0/4281785132 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7f4f70038490 msgr2=0x7f4f7003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:20:18.108 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:18.108+0000 7f4f90600700 1 --2- 172.21.15.189:0/4281785132 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7f4f70038490 0x7f4f7003a940 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f4f74006fd0 tx=0x7f4f74006e40 comp rx=0 tx=0).stop 2024-04-20T21:20:18.108 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:18.108+0000 7f4f90600700 1 -- 172.21.15.189:0/4281785132 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f880fed30 msgr2=0x7f4f88194fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:20:18.108 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:18.108+0000 7f4f90600700 1 --2- 172.21.15.189:0/4281785132 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f880fed30 0x7f4f88194fd0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f4f7c005950 tx=0x7f4f7c009e60 comp rx=0 tx=0).stop 2024-04-20T21:20:18.108 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:18.108+0000 7f4f90600700 1 -- 172.21.15.189:0/4281785132 shutdown_connections 2024-04-20T21:20:18.108 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:18.108+0000 7f4f90600700 1 --2- 172.21.15.189:0/4281785132 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7f4f70038490 0x7f4f7003a940 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:20:18.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:18.108+0000 7f4f90600700 1 --2- 172.21.15.189:0/4281785132 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f880fed30 0x7f4f88194fd0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:20:18.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:18.108+0000 7f4f90600700 1 -- 172.21.15.189:0/4281785132 >> 172.21.15.189:0/4281785132 conn(0x7f4f880fa2c0 msgr2=0x7f4f88069070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:20:18.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:18.109+0000 7f4f90600700 1 -- 172.21.15.189:0/4281785132 shutdown_connections 2024-04-20T21:20:18.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:18.109+0000 7f4f90600700 1 -- 172.21.15.189:0/4281785132 wait complete. 2024-04-20T21:20:18.853 DEBUG:teuthology.orchestra.run.smithi189:mon.smithi189> sudo journalctl -f -n 0 -u ceph-61ae3628-ff5b-11ee-bc93-c7b262605968@mon.smithi189.service 2024-04-20T21:20:18.861 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-20T21:20:18.861 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-20T21:20:19.236 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-20T21:20:19.340 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-20T21:20:19.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:20:19 smithi062 ceph-mon[16221]: from='client.14197 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi062:172.21.15.62=smithi062;smithi189:172.21.15.189=smithi189", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:20:19.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:20:19 smithi062 ceph-mon[16221]: Saving service mon spec with placement smithi062:172.21.15.62=smithi062;smithi189:172.21.15.189=smithi189;count:2 2024-04-20T21:20:19.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:20:19 smithi062 ceph-mon[16221]: from='mgr.14164 172.21.15.62:0/570876096' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:20:19.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:20:19 smithi062 ceph-mon[16221]: from='mgr.14164 172.21.15.62:0/570876096' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:20:19.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:20:19 smithi062 ceph-mon[16221]: from='mgr.14164 172.21.15.62:0/570876096' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:20:19.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:20:19 smithi062 ceph-mon[16221]: from='mgr.14164 172.21.15.62:0/570876096' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:20:19.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:20:19 smithi062 ceph-mon[16221]: from='mgr.14164 172.21.15.62:0/570876096' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:20:19.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:20:19 smithi062 ceph-mon[16221]: from='mgr.14164 172.21.15.62:0/570876096' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:20:19.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:20:19 smithi062 ceph-mon[16221]: from='mgr.14164 172.21.15.62:0/570876096' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:20:19.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:20:19 smithi062 ceph-mon[16221]: from='mgr.14164 172.21.15.62:0/570876096' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-20T21:20:19.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:20:19 smithi062 ceph-mon[16221]: from='mgr.14164 172.21.15.62:0/570876096' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:20:20.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:20:20 smithi062 ceph-mon[16221]: Regenerating cephadm self-signed grafana TLS certificates 2024-04-20T21:20:20.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:20:20 smithi062 ceph-mon[16221]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-20T21:20:20.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:20:20 smithi062 ceph-mon[16221]: Deploying daemon grafana.smithi062 on smithi062 2024-04-20T21:20:20.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:20:20 smithi062 ceph-mon[16221]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:20:20.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:20:20 smithi062 ceph-mon[16221]: from='mgr.14164 172.21.15.62:0/570876096' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:20:21.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:21.274+0000 7f74dca00700 1 -- 172.21.15.189:0/2480365849 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f74d8106440 msgr2=0x7f74d8106850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:20:21.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:21.274+0000 7f74dca00700 1 --2- 172.21.15.189:0/2480365849 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f74d8106440 0x7f74d8106850 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f74c8009910 tx=0x7f74c8009c20 comp rx=0 tx=0).stop 2024-04-20T21:20:21.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:21.274+0000 7f74dca00700 1 -- 172.21.15.189:0/2480365849 shutdown_connections 2024-04-20T21:20:21.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:21.274+0000 7f74dca00700 1 --2- 172.21.15.189:0/2480365849 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f74d8106440 0x7f74d8106850 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:20:21.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:21.274+0000 7f74dca00700 1 -- 172.21.15.189:0/2480365849 >> 172.21.15.189:0/2480365849 conn(0x7f74d80756e0 msgr2=0x7f74d8077b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:20:21.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:21.274+0000 7f74dca00700 1 -- 172.21.15.189:0/2480365849 shutdown_connections 2024-04-20T21:20:21.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:21.274+0000 7f74dca00700 1 -- 172.21.15.189:0/2480365849 wait complete. 2024-04-20T21:20:21.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:21.276+0000 7f74dca00700 1 Processor -- start 2024-04-20T21:20:21.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:21.276+0000 7f74dca00700 1 -- start start 2024-04-20T21:20:21.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:21.276+0000 7f74dca00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f74d8106440 0x7f74d819b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:20:21.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:21.276+0000 7f74dca00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74d819bae0 con 0x7f74d8106440 2024-04-20T21:20:21.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:21.277+0000 7f74d5e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f74d8106440 0x7f74d819b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:20:21.277 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:21.277+0000 7f74d5e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f74d8106440 0x7f74d819b5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.189:53812/0 (socket says 172.21.15.189:53812) 2024-04-20T21:20:21.277 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:21.277+0000 7f74d5e00700 1 -- 172.21.15.189:0/2113105648 learned_addr learned my addr 172.21.15.189:0/2113105648 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-20T21:20:21.277 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:21.278+0000 7f74d5e00700 1 -- 172.21.15.189:0/2113105648 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74c80095f0 con 0x7f74d8106440 2024-04-20T21:20:21.278 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:21.279+0000 7f74d5e00700 1 --2- 172.21.15.189:0/2113105648 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f74d8106440 0x7f74d819b5a0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f74c8005b40 tx=0x7f74c8004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:20:21.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:21.279+0000 7f74cea00700 1 -- 172.21.15.189:0/2113105648 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f74c801d070 con 0x7f74d8106440 2024-04-20T21:20:21.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:21.279+0000 7f74dca00700 1 -- 172.21.15.189:0/2113105648 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74d819bce0 con 0x7f74d8106440 2024-04-20T21:20:21.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:21.279+0000 7f74cea00700 1 -- 172.21.15.189:0/2113105648 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f74c800be10 con 0x7f74d8106440 2024-04-20T21:20:21.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:21.279+0000 7f74cea00700 1 -- 172.21.15.189:0/2113105648 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f74c800f460 con 0x7f74d8106440 2024-04-20T21:20:21.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:21.279+0000 7f74dca00700 1 -- 172.21.15.189:0/2113105648 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74d819c180 con 0x7f74d8106440 2024-04-20T21:20:21.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:21.280+0000 7f74dca00700 1 -- 172.21.15.189:0/2113105648 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74d8195110 con 0x7f74d8106440 2024-04-20T21:20:21.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:21.281+0000 7f74cea00700 1 -- 172.21.15.189:0/2113105648 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 13) v1 ==== 45140+0+0 (secure 0 0 0) 0x7f74c80227e0 con 0x7f74d8106440 2024-04-20T21:20:21.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:21.282+0000 7f74cea00700 1 --2- 172.21.15.189:0/2113105648 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7f74bc0384e0 0x7f74bc03a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:20:21.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:21.282+0000 7f74cea00700 1 -- 172.21.15.189:0/2113105648 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f74c804cae0 con 0x7f74d8106440 2024-04-20T21:20:21.282 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:21.282+0000 7f74d5400700 1 --2- 172.21.15.189:0/2113105648 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7f74bc0384e0 0x7f74bc03a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:20:21.283 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:21.284+0000 7f74d5400700 1 --2- 172.21.15.189:0/2113105648 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7f74bc0384e0 0x7f74bc03a990 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f74c4006fd0 tx=0x7f74c4006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:20:21.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:21.304+0000 7f74cea00700 1 -- 172.21.15.189:0/2113105648 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f74c802bdf0 con 0x7f74d8106440 2024-04-20T21:20:21.601 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:21.601+0000 7f74dca00700 1 -- 172.21.15.189:0/2113105648 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f74d802cc30 con 0x7f74d8106440 2024-04-20T21:20:21.602 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:21.603+0000 7f74cea00700 1 -- 172.21.15.189:0/2113105648 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f74c8027030 con 0x7f74d8106440 2024-04-20T21:20:21.603 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-20T21:20:21.603 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":1,"fsid":"61ae3628-ff5b-11ee-bc93-c7b262605968","modified":"2024-04-20T21:18:14.797594Z","created":"2024-04-20T21:18:14.797594Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-20T21:20:21.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:21.606+0000 7f74dca00700 1 -- 172.21.15.189:0/2113105648 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7f74bc0384e0 msgr2=0x7f74bc03a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:20:21.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:21.606+0000 7f74dca00700 1 --2- 172.21.15.189:0/2113105648 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7f74bc0384e0 0x7f74bc03a990 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f74c4006fd0 tx=0x7f74c4006e40 comp rx=0 tx=0).stop 2024-04-20T21:20:21.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:21.606+0000 7f74dca00700 1 -- 172.21.15.189:0/2113105648 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f74d8106440 msgr2=0x7f74d819b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:20:21.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:21.606+0000 7f74dca00700 1 --2- 172.21.15.189:0/2113105648 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f74d8106440 0x7f74d819b5a0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f74c8005b40 tx=0x7f74c8004740 comp rx=0 tx=0).stop 2024-04-20T21:20:21.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:21.606+0000 7f74dca00700 1 -- 172.21.15.189:0/2113105648 shutdown_connections 2024-04-20T21:20:21.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:21.606+0000 7f74dca00700 1 --2- 172.21.15.189:0/2113105648 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7f74bc0384e0 0x7f74bc03a990 secure :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f74c4006fd0 tx=0x7f74c4006e40 comp rx=0 tx=0).stop 2024-04-20T21:20:21.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:21.606+0000 7f74dca00700 1 --2- 172.21.15.189:0/2113105648 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f74d8106440 0x7f74d819b5a0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:20:21.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:21.606+0000 7f74dca00700 1 -- 172.21.15.189:0/2113105648 >> 172.21.15.189:0/2113105648 conn(0x7f74d80756e0 msgr2=0x7f74d80773a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:20:21.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:21.606+0000 7f74dca00700 1 -- 172.21.15.189:0/2113105648 shutdown_connections 2024-04-20T21:20:21.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:21.606+0000 7f74dca00700 1 -- 172.21.15.189:0/2113105648 wait complete. 2024-04-20T21:20:21.607 INFO:teuthology.orchestra.run.smithi189.stderr:dumped monmap epoch 1 2024-04-20T21:20:22.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:20:22 smithi062 ceph-mon[16221]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:20:22.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:20:22 smithi062 ceph-mon[16221]: from='client.? 172.21.15.189:0/2113105648' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-20T21:20:23.321 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-20T21:20:23.321 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-20T21:20:23.688 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-20T21:20:23.789 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-20T21:20:24.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:20:24 smithi062 ceph-mon[16221]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:20:25.639 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:25.638+0000 7f3a76400700 1 -- 172.21.15.189:0/1193033830 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3a70102020 msgr2=0x7f3a70102430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:20:25.639 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:25.638+0000 7f3a76400700 1 --2- 172.21.15.189:0/1193033830 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3a70102020 0x7f3a70102430 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f3a60009910 tx=0x7f3a60009c20 comp rx=0 tx=0).stop 2024-04-20T21:20:25.639 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:25.638+0000 7f3a76400700 1 -- 172.21.15.189:0/1193033830 shutdown_connections 2024-04-20T21:20:25.639 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:25.638+0000 7f3a76400700 1 --2- 172.21.15.189:0/1193033830 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3a70102020 0x7f3a70102430 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:20:25.639 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:25.639+0000 7f3a76400700 1 -- 172.21.15.189:0/1193033830 >> 172.21.15.189:0/1193033830 conn(0x7f3a700fd700 msgr2=0x7f3a700ffae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:20:25.639 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:25.639+0000 7f3a76400700 1 -- 172.21.15.189:0/1193033830 shutdown_connections 2024-04-20T21:20:25.639 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:25.639+0000 7f3a76400700 1 -- 172.21.15.189:0/1193033830 wait complete. 2024-04-20T21:20:25.640 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:25.640+0000 7f3a76400700 1 Processor -- start 2024-04-20T21:20:25.640 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:25.641+0000 7f3a76400700 1 -- start start 2024-04-20T21:20:25.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:25.641+0000 7f3a76400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3a70102020 0x7f3a701971a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:20:25.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:25.641+0000 7f3a76400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a701976e0 con 0x7f3a70102020 2024-04-20T21:20:25.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:25.642+0000 7f3a6f400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3a70102020 0x7f3a701971a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:20:25.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:25.642+0000 7f3a6f400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3a70102020 0x7f3a701971a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.189:53822/0 (socket says 172.21.15.189:53822) 2024-04-20T21:20:25.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:25.642+0000 7f3a6f400700 1 -- 172.21.15.189:0/181512519 learned_addr learned my addr 172.21.15.189:0/181512519 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-20T21:20:25.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:25.642+0000 7f3a6f400700 1 -- 172.21.15.189:0/181512519 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a600095f0 con 0x7f3a70102020 2024-04-20T21:20:25.642 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:25.643+0000 7f3a6f400700 1 --2- 172.21.15.189:0/181512519 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3a70102020 0x7f3a701971a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f3a60005b40 tx=0x7f3a60004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:20:25.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:25.643+0000 7f3a67e00700 1 -- 172.21.15.189:0/181512519 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3a6001d070 con 0x7f3a70102020 2024-04-20T21:20:25.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:25.644+0000 7f3a76400700 1 -- 172.21.15.189:0/181512519 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a701978e0 con 0x7f3a70102020 2024-04-20T21:20:25.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:25.644+0000 7f3a67e00700 1 -- 172.21.15.189:0/181512519 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f3a6000be10 con 0x7f3a70102020 2024-04-20T21:20:25.643 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:25.644+0000 7f3a67e00700 1 -- 172.21.15.189:0/181512519 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3a6000f460 con 0x7f3a70102020 2024-04-20T21:20:25.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:25.644+0000 7f3a76400700 1 -- 172.21.15.189:0/181512519 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a70197d80 con 0x7f3a70102020 2024-04-20T21:20:25.644 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:25.645+0000 7f3a76400700 1 -- 172.21.15.189:0/181512519 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a70190d00 con 0x7f3a70102020 2024-04-20T21:20:25.645 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:25.646+0000 7f3a67e00700 1 -- 172.21.15.189:0/181512519 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 13) v1 ==== 45140+0+0 (secure 0 0 0) 0x7f3a600227c0 con 0x7f3a70102020 2024-04-20T21:20:25.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:25.646+0000 7f3a67e00700 1 --2- 172.21.15.189:0/181512519 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7f3a54038530 0x7f3a5403a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:20:25.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:25.646+0000 7f3a67e00700 1 -- 172.21.15.189:0/181512519 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f3a6004c640 con 0x7f3a70102020 2024-04-20T21:20:25.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:25.647+0000 7f3a6ea00700 1 --2- 172.21.15.189:0/181512519 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7f3a54038530 0x7f3a5403a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:20:25.648 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:25.648+0000 7f3a6ea00700 1 --2- 172.21.15.189:0/181512519 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7f3a54038530 0x7f3a5403a9e0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f3a5c006fd0 tx=0x7f3a5c006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:20:25.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:25.665+0000 7f3a67e00700 1 -- 172.21.15.189:0/181512519 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f3a6000b960 con 0x7f3a70102020 2024-04-20T21:20:25.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:25.966+0000 7f3a76400700 1 -- 172.21.15.189:0/181512519 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3a7002cc30 con 0x7f3a70102020 2024-04-20T21:20:25.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:25.967+0000 7f3a67e00700 1 -- 172.21.15.189:0/181512519 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f3a60027020 con 0x7f3a70102020 2024-04-20T21:20:25.967 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-20T21:20:25.967 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":1,"fsid":"61ae3628-ff5b-11ee-bc93-c7b262605968","modified":"2024-04-20T21:18:14.797594Z","created":"2024-04-20T21:18:14.797594Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-20T21:20:25.970 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:25.970+0000 7f3a76400700 1 -- 172.21.15.189:0/181512519 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7f3a54038530 msgr2=0x7f3a5403a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:20:25.970 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:25.970+0000 7f3a76400700 1 --2- 172.21.15.189:0/181512519 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7f3a54038530 0x7f3a5403a9e0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f3a5c006fd0 tx=0x7f3a5c006e40 comp rx=0 tx=0).stop 2024-04-20T21:20:25.970 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:25.970+0000 7f3a76400700 1 -- 172.21.15.189:0/181512519 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3a70102020 msgr2=0x7f3a701971a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:20:25.970 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:25.970+0000 7f3a76400700 1 --2- 172.21.15.189:0/181512519 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3a70102020 0x7f3a701971a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f3a60005b40 tx=0x7f3a60004740 comp rx=0 tx=0).stop 2024-04-20T21:20:25.970 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:25.970+0000 7f3a76400700 1 -- 172.21.15.189:0/181512519 shutdown_connections 2024-04-20T21:20:25.970 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:25.970+0000 7f3a76400700 1 --2- 172.21.15.189:0/181512519 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7f3a54038530 0x7f3a5403a9e0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:20:25.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:25.970+0000 7f3a76400700 1 --2- 172.21.15.189:0/181512519 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3a70102020 0x7f3a701971a0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:20:25.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:25.970+0000 7f3a76400700 1 -- 172.21.15.189:0/181512519 >> 172.21.15.189:0/181512519 conn(0x7f3a700fd700 msgr2=0x7f3a700ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:20:25.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:25.970+0000 7f3a76400700 1 -- 172.21.15.189:0/181512519 shutdown_connections 2024-04-20T21:20:25.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:25.971+0000 7f3a76400700 1 -- 172.21.15.189:0/181512519 wait complete. 2024-04-20T21:20:25.971 INFO:teuthology.orchestra.run.smithi189.stderr:dumped monmap epoch 1 2024-04-20T21:20:26.519 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:20:26 smithi062 ceph-mon[16221]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:20:26.519 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:20:26 smithi062 ceph-mon[16221]: from='client.? 172.21.15.189:0/181512519' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-20T21:20:27.662 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-20T21:20:27.662 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-20T21:20:27.721 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:20:27 smithi062 ceph-mon[16221]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:20:28.065 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-20T21:20:28.173 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-20T21:20:30.041 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:30.040+0000 7f3ee9e00700 1 -- 172.21.15.189:0/3168230712 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3ee4106440 msgr2=0x7f3ee4106850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:20:30.041 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:30.040+0000 7f3ee9e00700 1 --2- 172.21.15.189:0/3168230712 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3ee4106440 0x7f3ee4106850 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f3ed4009910 tx=0x7f3ed4009c20 comp rx=0 tx=0).stop 2024-04-20T21:20:30.041 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:30.040+0000 7f3ee9e00700 1 -- 172.21.15.189:0/3168230712 shutdown_connections 2024-04-20T21:20:30.041 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:30.040+0000 7f3ee9e00700 1 --2- 172.21.15.189:0/3168230712 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3ee4106440 0x7f3ee4106850 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:20:30.041 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:30.040+0000 7f3ee9e00700 1 -- 172.21.15.189:0/3168230712 >> 172.21.15.189:0/3168230712 conn(0x7f3ee40756e0 msgr2=0x7f3ee4077b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:20:30.041 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:30.040+0000 7f3ee9e00700 1 -- 172.21.15.189:0/3168230712 shutdown_connections 2024-04-20T21:20:30.041 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:30.040+0000 7f3ee9e00700 1 -- 172.21.15.189:0/3168230712 wait complete. 2024-04-20T21:20:30.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:30.042+0000 7f3ee9e00700 1 Processor -- start 2024-04-20T21:20:30.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:30.042+0000 7f3ee9e00700 1 -- start start 2024-04-20T21:20:30.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:30.042+0000 7f3ee9e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3ee419b7b0 0x7f3ee419bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:20:30.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:30.042+0000 7f3ee9e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee419c100 con 0x7f3ee419b7b0 2024-04-20T21:20:30.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:30.043+0000 7f3ee2a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3ee419b7b0 0x7f3ee419bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:20:30.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:30.043+0000 7f3ee2a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3ee419b7b0 0x7f3ee419bbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.189:59456/0 (socket says 172.21.15.189:59456) 2024-04-20T21:20:30.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:30.043+0000 7f3ee2a00700 1 -- 172.21.15.189:0/897108901 learned_addr learned my addr 172.21.15.189:0/897108901 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-20T21:20:30.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:30.044+0000 7f3ee2a00700 1 -- 172.21.15.189:0/897108901 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ed40095f0 con 0x7f3ee419b7b0 2024-04-20T21:20:30.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:30.044+0000 7f3ee2a00700 1 --2- 172.21.15.189:0/897108901 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3ee419b7b0 0x7f3ee419bbc0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f3ee4107450 tx=0x7f3ed4009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:20:30.048 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:30.048+0000 7f3ee8a00700 1 -- 172.21.15.189:0/897108901 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ed401d070 con 0x7f3ee419b7b0 2024-04-20T21:20:30.048 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:30.048+0000 7f3ee9e00700 1 -- 172.21.15.189:0/897108901 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ee419c300 con 0x7f3ee419b7b0 2024-04-20T21:20:30.048 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:30.048+0000 7f3ee8a00700 1 -- 172.21.15.189:0/897108901 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f3ed4005dc0 con 0x7f3ee419b7b0 2024-04-20T21:20:30.048 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:30.048+0000 7f3ee8a00700 1 -- 172.21.15.189:0/897108901 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ed4022da0 con 0x7f3ee419b7b0 2024-04-20T21:20:30.048 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:30.048+0000 7f3ee9e00700 1 -- 172.21.15.189:0/897108901 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ee419ef60 con 0x7f3ee419b7b0 2024-04-20T21:20:30.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:30.050+0000 7f3ee9e00700 1 -- 172.21.15.189:0/897108901 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ee4195110 con 0x7f3ee419b7b0 2024-04-20T21:20:30.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:30.050+0000 7f3ee8a00700 1 -- 172.21.15.189:0/897108901 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 13) v1 ==== 45140+0+0 (secure 0 0 0) 0x7f3ed4022690 con 0x7f3ee419b7b0 2024-04-20T21:20:30.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:30.051+0000 7f3ee8a00700 1 --2- 172.21.15.189:0/897108901 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7f3ec80384e0 0x7f3ec803a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:20:30.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:30.051+0000 7f3ee8a00700 1 -- 172.21.15.189:0/897108901 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f3ed404c2c0 con 0x7f3ee419b7b0 2024-04-20T21:20:30.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:30.051+0000 7f3ee2000700 1 --2- 172.21.15.189:0/897108901 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7f3ec80384e0 0x7f3ec803a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:20:30.052 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:30.052+0000 7f3ee2000700 1 --2- 172.21.15.189:0/897108901 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7f3ec80384e0 0x7f3ec803a990 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f3ed0006fd0 tx=0x7f3ed0006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:20:30.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:30.058+0000 7f3ee8a00700 1 -- 172.21.15.189:0/897108901 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f3ed4022470 con 0x7f3ee419b7b0 2024-04-20T21:20:30.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:30.355+0000 7f3ee9e00700 1 -- 172.21.15.189:0/897108901 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3ee402cc30 con 0x7f3ee419b7b0 2024-04-20T21:20:30.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:30.356+0000 7f3ee8a00700 1 -- 172.21.15.189:0/897108901 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f3ed4027030 con 0x7f3ee419b7b0 2024-04-20T21:20:30.356 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-20T21:20:30.356 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":1,"fsid":"61ae3628-ff5b-11ee-bc93-c7b262605968","modified":"2024-04-20T21:18:14.797594Z","created":"2024-04-20T21:18:14.797594Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-20T21:20:30.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:30.358+0000 7f3ee9e00700 1 -- 172.21.15.189:0/897108901 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7f3ec80384e0 msgr2=0x7f3ec803a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:20:30.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:30.358+0000 7f3ee9e00700 1 --2- 172.21.15.189:0/897108901 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7f3ec80384e0 0x7f3ec803a990 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f3ed0006fd0 tx=0x7f3ed0006e40 comp rx=0 tx=0).stop 2024-04-20T21:20:30.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:30.358+0000 7f3ee9e00700 1 -- 172.21.15.189:0/897108901 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3ee419b7b0 msgr2=0x7f3ee419bbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:20:30.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:30.359+0000 7f3ee9e00700 1 --2- 172.21.15.189:0/897108901 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3ee419b7b0 0x7f3ee419bbc0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f3ee4107450 tx=0x7f3ed4009f80 comp rx=0 tx=0).stop 2024-04-20T21:20:30.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:30.359+0000 7f3ee9e00700 1 -- 172.21.15.189:0/897108901 shutdown_connections 2024-04-20T21:20:30.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:30.359+0000 7f3ee9e00700 1 --2- 172.21.15.189:0/897108901 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7f3ec80384e0 0x7f3ec803a990 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:20:30.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:30.359+0000 7f3ee9e00700 1 --2- 172.21.15.189:0/897108901 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3ee419b7b0 0x7f3ee419bbc0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:20:30.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:30.359+0000 7f3ee9e00700 1 -- 172.21.15.189:0/897108901 >> 172.21.15.189:0/897108901 conn(0x7f3ee40756e0 msgr2=0x7f3ee4077490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:20:30.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:30.359+0000 7f3ee9e00700 1 -- 172.21.15.189:0/897108901 shutdown_connections 2024-04-20T21:20:30.359 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:30.359+0000 7f3ee9e00700 1 -- 172.21.15.189:0/897108901 wait complete. 2024-04-20T21:20:30.360 INFO:teuthology.orchestra.run.smithi189.stderr:dumped monmap epoch 1 2024-04-20T21:20:30.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:20:30 smithi062 ceph-mon[16221]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:20:31.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:20:31 smithi062 ceph-mon[16221]: from='client.? 172.21.15.189:0/897108901' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-20T21:20:32.037 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-20T21:20:32.037 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-20T21:20:32.413 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-20T21:20:32.529 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-20T21:20:32.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:20:32 smithi062 ceph-mon[16221]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:20:34.505 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:34.505+0000 7fdc65400700 1 -- 172.21.15.189:0/750046159 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc60102020 msgr2=0x7fdc60102430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:20:34.506 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:34.505+0000 7fdc65400700 1 --2- 172.21.15.189:0/750046159 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc60102020 0x7fdc60102430 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fdc5000b0a0 tx=0x7fdc5000b3b0 comp rx=0 tx=0).stop 2024-04-20T21:20:34.506 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:34.505+0000 7fdc65400700 1 -- 172.21.15.189:0/750046159 shutdown_connections 2024-04-20T21:20:34.506 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:34.505+0000 7fdc65400700 1 --2- 172.21.15.189:0/750046159 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc60102020 0x7fdc60102430 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:20:34.506 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:34.505+0000 7fdc65400700 1 -- 172.21.15.189:0/750046159 >> 172.21.15.189:0/750046159 conn(0x7fdc600fd700 msgr2=0x7fdc600ffae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:20:34.506 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:34.505+0000 7fdc65400700 1 -- 172.21.15.189:0/750046159 shutdown_connections 2024-04-20T21:20:34.506 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:34.505+0000 7fdc65400700 1 -- 172.21.15.189:0/750046159 wait complete. 2024-04-20T21:20:34.507 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:34.507+0000 7fdc65400700 1 Processor -- start 2024-04-20T21:20:34.507 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:34.507+0000 7fdc65400700 1 -- start start 2024-04-20T21:20:34.507 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:34.507+0000 7fdc65400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc60102020 0x7fdc601971a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:20:34.507 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:34.507+0000 7fdc65400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc601976e0 con 0x7fdc60102020 2024-04-20T21:20:34.507 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:34.508+0000 7fdc5e800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc60102020 0x7fdc601971a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:20:34.508 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:34.508+0000 7fdc5e800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc60102020 0x7fdc601971a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.189:59486/0 (socket says 172.21.15.189:59486) 2024-04-20T21:20:34.508 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:34.508+0000 7fdc5e800700 1 -- 172.21.15.189:0/2263163462 learned_addr learned my addr 172.21.15.189:0/2263163462 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-20T21:20:34.508 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:34.508+0000 7fdc5e800700 1 -- 172.21.15.189:0/2263163462 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc50009d00 con 0x7fdc60102020 2024-04-20T21:20:34.508 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:34.509+0000 7fdc5e800700 1 --2- 172.21.15.189:0/2263163462 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc60102020 0x7fdc601971a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fdc5000b830 tx=0x7fdc5000bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:20:34.509 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:34.509+0000 7fdc57e00700 1 -- 172.21.15.189:0/2263163462 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdc5000e040 con 0x7fdc60102020 2024-04-20T21:20:34.509 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:34.510+0000 7fdc57e00700 1 -- 172.21.15.189:0/2263163462 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fdc50012af0 con 0x7fdc60102020 2024-04-20T21:20:34.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:34.510+0000 7fdc65400700 1 -- 172.21.15.189:0/2263163462 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc601978e0 con 0x7fdc60102020 2024-04-20T21:20:34.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:34.510+0000 7fdc57e00700 1 -- 172.21.15.189:0/2263163462 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdc5001d9f0 con 0x7fdc60102020 2024-04-20T21:20:34.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:34.510+0000 7fdc65400700 1 -- 172.21.15.189:0/2263163462 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc60197d00 con 0x7fdc60102020 2024-04-20T21:20:34.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:34.511+0000 7fdc65400700 1 -- 172.21.15.189:0/2263163462 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc60190d00 con 0x7fdc60102020 2024-04-20T21:20:34.511 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:34.512+0000 7fdc57e00700 1 -- 172.21.15.189:0/2263163462 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 13) v1 ==== 45140+0+0 (secure 0 0 0) 0x7fdc50012c60 con 0x7fdc60102020 2024-04-20T21:20:34.511 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:34.512+0000 7fdc57e00700 1 --2- 172.21.15.189:0/2263163462 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7fdc440384e0 0x7fdc4403a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:20:34.511 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:34.512+0000 7fdc57e00700 1 -- 172.21.15.189:0/2263163462 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fdc5004d800 con 0x7fdc60102020 2024-04-20T21:20:34.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:34.513+0000 7fdc55e00700 1 --2- 172.21.15.189:0/2263163462 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7fdc440384e0 0x7fdc4403a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:20:34.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:34.514+0000 7fdc55e00700 1 --2- 172.21.15.189:0/2263163462 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7fdc440384e0 0x7fdc4403a990 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fdc48006fd0 tx=0x7fdc48006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:20:34.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:34.528+0000 7fdc57e00700 1 -- 172.21.15.189:0/2263163462 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fdc5001c9d0 con 0x7fdc60102020 2024-04-20T21:20:34.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:20:34 smithi062 ceph-mon[16221]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:20:34.828 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:34.828+0000 7fdc65400700 1 -- 172.21.15.189:0/2263163462 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdc60062380 con 0x7fdc60102020 2024-04-20T21:20:34.829 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:34.831+0000 7fdc57e00700 1 -- 172.21.15.189:0/2263163462 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7fdc5001c3e0 con 0x7fdc60102020 2024-04-20T21:20:34.830 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-20T21:20:34.830 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":1,"fsid":"61ae3628-ff5b-11ee-bc93-c7b262605968","modified":"2024-04-20T21:18:14.797594Z","created":"2024-04-20T21:18:14.797594Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-20T21:20:34.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:34.832+0000 7fdc65400700 1 -- 172.21.15.189:0/2263163462 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7fdc440384e0 msgr2=0x7fdc4403a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:20:34.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:34.832+0000 7fdc65400700 1 --2- 172.21.15.189:0/2263163462 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7fdc440384e0 0x7fdc4403a990 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fdc48006fd0 tx=0x7fdc48006e40 comp rx=0 tx=0).stop 2024-04-20T21:20:34.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:34.832+0000 7fdc65400700 1 -- 172.21.15.189:0/2263163462 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc60102020 msgr2=0x7fdc601971a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:20:34.832 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:34.832+0000 7fdc65400700 1 --2- 172.21.15.189:0/2263163462 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc60102020 0x7fdc601971a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fdc5000b830 tx=0x7fdc5000bd40 comp rx=0 tx=0).stop 2024-04-20T21:20:34.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:34.833+0000 7fdc65400700 1 -- 172.21.15.189:0/2263163462 shutdown_connections 2024-04-20T21:20:34.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:34.833+0000 7fdc65400700 1 --2- 172.21.15.189:0/2263163462 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7fdc440384e0 0x7fdc4403a990 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:20:34.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:34.833+0000 7fdc65400700 1 --2- 172.21.15.189:0/2263163462 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc60102020 0x7fdc601971a0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:20:34.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:34.833+0000 7fdc65400700 1 -- 172.21.15.189:0/2263163462 >> 172.21.15.189:0/2263163462 conn(0x7fdc600fd700 msgr2=0x7fdc600ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:20:34.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:34.833+0000 7fdc65400700 1 -- 172.21.15.189:0/2263163462 shutdown_connections 2024-04-20T21:20:34.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:34.833+0000 7fdc65400700 1 -- 172.21.15.189:0/2263163462 wait complete. 2024-04-20T21:20:34.834 INFO:teuthology.orchestra.run.smithi189.stderr:dumped monmap epoch 1 2024-04-20T21:20:35.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:20:35 smithi062 ceph-mon[16221]: from='client.? 172.21.15.189:0/2263163462' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-20T21:20:36.494 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-20T21:20:36.495 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-20T21:20:36.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:20:36 smithi062 ceph-mon[16221]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:20:36.871 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-20T21:20:36.969 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-20T21:20:37.930 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:20:37 smithi062 ceph-mon[16221]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:20:38.927 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:38.927+0000 7f3c1dc00700 1 -- 172.21.15.189:0/66192272 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3c18101fe0 msgr2=0x7f3c181023f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:20:38.927 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:38.927+0000 7f3c1dc00700 1 --2- 172.21.15.189:0/66192272 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3c18101fe0 0x7f3c181023f0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f3c08009910 tx=0x7f3c08009c20 comp rx=0 tx=0).stop 2024-04-20T21:20:38.927 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:38.928+0000 7f3c1dc00700 1 -- 172.21.15.189:0/66192272 shutdown_connections 2024-04-20T21:20:38.927 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:38.928+0000 7f3c1dc00700 1 --2- 172.21.15.189:0/66192272 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3c18101fe0 0x7f3c181023f0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:20:38.927 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:38.928+0000 7f3c1dc00700 1 -- 172.21.15.189:0/66192272 >> 172.21.15.189:0/66192272 conn(0x7f3c180fd590 msgr2=0x7f3c180ff9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:20:38.927 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:38.928+0000 7f3c1dc00700 1 -- 172.21.15.189:0/66192272 shutdown_connections 2024-04-20T21:20:38.927 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:38.928+0000 7f3c1dc00700 1 -- 172.21.15.189:0/66192272 wait complete. 2024-04-20T21:20:38.928 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:38.929+0000 7f3c1dc00700 1 Processor -- start 2024-04-20T21:20:38.928 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:38.929+0000 7f3c1dc00700 1 -- start start 2024-04-20T21:20:38.928 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:38.930+0000 7f3c1dc00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3c18101fe0 0x7f3c18197150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:20:38.928 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:38.930+0000 7f3c1dc00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c18197690 con 0x7f3c18101fe0 2024-04-20T21:20:38.929 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:38.930+0000 7f3c16a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3c18101fe0 0x7f3c18197150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:20:38.929 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:38.930+0000 7f3c16a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3c18101fe0 0x7f3c18197150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.189:34418/0 (socket says 172.21.15.189:34418) 2024-04-20T21:20:38.929 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:38.930+0000 7f3c16a00700 1 -- 172.21.15.189:0/549274685 learned_addr learned my addr 172.21.15.189:0/549274685 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-20T21:20:38.929 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:38.931+0000 7f3c16a00700 1 -- 172.21.15.189:0/549274685 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c080095f0 con 0x7f3c18101fe0 2024-04-20T21:20:38.929 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:38.931+0000 7f3c16a00700 1 --2- 172.21.15.189:0/549274685 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3c18101fe0 0x7f3c18197150 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f3c08005950 tx=0x7f3c08009e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:20:38.930 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:38.932+0000 7f3c0f400700 1 -- 172.21.15.189:0/549274685 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3c0801d070 con 0x7f3c18101fe0 2024-04-20T21:20:38.930 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:38.932+0000 7f3c1dc00700 1 -- 172.21.15.189:0/549274685 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c18197890 con 0x7f3c18101fe0 2024-04-20T21:20:38.930 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:38.932+0000 7f3c0f400700 1 -- 172.21.15.189:0/549274685 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f3c0800be10 con 0x7f3c18101fe0 2024-04-20T21:20:38.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:38.932+0000 7f3c0f400700 1 -- 172.21.15.189:0/549274685 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3c0800f460 con 0x7f3c18101fe0 2024-04-20T21:20:38.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:38.932+0000 7f3c1dc00700 1 -- 172.21.15.189:0/549274685 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c18197d30 con 0x7f3c18101fe0 2024-04-20T21:20:38.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:38.933+0000 7f3c1dc00700 1 -- 172.21.15.189:0/549274685 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c18190cc0 con 0x7f3c18101fe0 2024-04-20T21:20:38.932 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:38.933+0000 7f3c0f400700 1 -- 172.21.15.189:0/549274685 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 13) v1 ==== 45140+0+0 (secure 0 0 0) 0x7f3c080227c0 con 0x7f3c18101fe0 2024-04-20T21:20:38.932 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:38.934+0000 7f3c0f400700 1 --2- 172.21.15.189:0/549274685 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7f3bfc038490 0x7f3bfc03a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:20:38.932 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:38.934+0000 7f3c0f400700 1 -- 172.21.15.189:0/549274685 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f3c0804c450 con 0x7f3c18101fe0 2024-04-20T21:20:38.933 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:38.934+0000 7f3c16000700 1 --2- 172.21.15.189:0/549274685 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7f3bfc038490 0x7f3bfc03a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:20:38.934 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:38.935+0000 7f3c16000700 1 --2- 172.21.15.189:0/549274685 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7f3bfc038490 0x7f3bfc03a940 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f3c04006fd0 tx=0x7f3c04006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:20:38.947 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:38.949+0000 7f3c0f400700 1 -- 172.21.15.189:0/549274685 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f3c0802bba0 con 0x7f3c18101fe0 2024-04-20T21:20:39.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:39.247+0000 7f3c1dc00700 1 -- 172.21.15.189:0/549274685 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3c1802cc30 con 0x7f3c18101fe0 2024-04-20T21:20:39.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:39.248+0000 7f3c0f400700 1 -- 172.21.15.189:0/549274685 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f3c08027030 con 0x7f3c18101fe0 2024-04-20T21:20:39.247 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-20T21:20:39.247 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":1,"fsid":"61ae3628-ff5b-11ee-bc93-c7b262605968","modified":"2024-04-20T21:18:14.797594Z","created":"2024-04-20T21:18:14.797594Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-20T21:20:39.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:39.252+0000 7f3c1dc00700 1 -- 172.21.15.189:0/549274685 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7f3bfc038490 msgr2=0x7f3bfc03a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:20:39.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:39.252+0000 7f3c1dc00700 1 --2- 172.21.15.189:0/549274685 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7f3bfc038490 0x7f3bfc03a940 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f3c04006fd0 tx=0x7f3c04006e40 comp rx=0 tx=0).stop 2024-04-20T21:20:39.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:39.252+0000 7f3c1dc00700 1 -- 172.21.15.189:0/549274685 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3c18101fe0 msgr2=0x7f3c18197150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:20:39.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:39.252+0000 7f3c1dc00700 1 --2- 172.21.15.189:0/549274685 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3c18101fe0 0x7f3c18197150 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f3c08005950 tx=0x7f3c08009e60 comp rx=0 tx=0).stop 2024-04-20T21:20:39.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:39.252+0000 7f3c1dc00700 1 -- 172.21.15.189:0/549274685 shutdown_connections 2024-04-20T21:20:39.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:39.252+0000 7f3c1dc00700 1 --2- 172.21.15.189:0/549274685 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7f3bfc038490 0x7f3bfc03a940 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:20:39.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:39.252+0000 7f3c1dc00700 1 --2- 172.21.15.189:0/549274685 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3c18101fe0 0x7f3c18197150 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:20:39.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:39.252+0000 7f3c1dc00700 1 -- 172.21.15.189:0/549274685 >> 172.21.15.189:0/549274685 conn(0x7f3c180fd590 msgr2=0x7f3c180fe240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:20:39.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:39.252+0000 7f3c1dc00700 1 -- 172.21.15.189:0/549274685 shutdown_connections 2024-04-20T21:20:39.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:39.252+0000 7f3c1dc00700 1 -- 172.21.15.189:0/549274685 wait complete. 2024-04-20T21:20:39.252 INFO:teuthology.orchestra.run.smithi189.stderr:dumped monmap epoch 1 2024-04-20T21:20:40.119 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:20:40 smithi062 ceph-mon[16221]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:20:40.120 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:20:40 smithi062 ceph-mon[16221]: from='client.? 172.21.15.189:0/549274685' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-20T21:20:40.971 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-20T21:20:40.971 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-20T21:20:41.339 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-20T21:20:41.449 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-20T21:20:41.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:20:41 smithi062 ceph-mon[16221]: from='mgr.14164 172.21.15.62:0/570876096' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:20:41.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:20:41 smithi062 ceph-mon[16221]: from='mgr.14164 172.21.15.62:0/570876096' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:20:41.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:20:41 smithi062 ceph-mon[16221]: from='mgr.14164 172.21.15.62:0/570876096' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:20:41.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:20:41 smithi062 ceph-mon[16221]: from='mgr.14164 172.21.15.62:0/570876096' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:20:41.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:20:41 smithi062 ceph-mon[16221]: from='mgr.14164 172.21.15.62:0/570876096' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:20:41.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:20:41 smithi062 ceph-mon[16221]: from='mgr.14164 172.21.15.62:0/570876096' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:20:41.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:20:41 smithi062 ceph-mon[16221]: Deploying daemon prometheus.smithi062 on smithi062 2024-04-20T21:20:41.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:20:41 smithi062 ceph-mon[16221]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:20:43.502 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:43.503+0000 7f9091000700 1 -- 172.21.15.189:0/4124055208 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f908c0ffe20 msgr2=0x7f908c100230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:20:43.502 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:43.503+0000 7f9091000700 1 --2- 172.21.15.189:0/4124055208 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f908c0ffe20 0x7f908c100230 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f907c009910 tx=0x7f907c009c20 comp rx=0 tx=0).stop 2024-04-20T21:20:43.502 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:43.503+0000 7f9091000700 1 -- 172.21.15.189:0/4124055208 shutdown_connections 2024-04-20T21:20:43.502 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:43.503+0000 7f9091000700 1 --2- 172.21.15.189:0/4124055208 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f908c0ffe20 0x7f908c100230 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:20:43.502 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:43.503+0000 7f9091000700 1 -- 172.21.15.189:0/4124055208 >> 172.21.15.189:0/4124055208 conn(0x7f908c0fb3d0 msgr2=0x7f908c0fd800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:20:43.502 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:43.503+0000 7f9091000700 1 -- 172.21.15.189:0/4124055208 shutdown_connections 2024-04-20T21:20:43.503 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:43.503+0000 7f9091000700 1 -- 172.21.15.189:0/4124055208 wait complete. 2024-04-20T21:20:43.503 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:43.504+0000 7f9091000700 1 Processor -- start 2024-04-20T21:20:43.503 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:43.504+0000 7f9091000700 1 -- start start 2024-04-20T21:20:43.504 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:43.505+0000 7f9091000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f908c0ffe20 0x7f908c197150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:20:43.504 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:43.505+0000 7f9091000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f908c197690 con 0x7f908c0ffe20 2024-04-20T21:20:43.504 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:43.505+0000 7f9089e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f908c0ffe20 0x7f908c197150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:20:43.504 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:43.505+0000 7f9089e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f908c0ffe20 0x7f908c197150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.189:34436/0 (socket says 172.21.15.189:34436) 2024-04-20T21:20:43.504 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:43.505+0000 7f9089e00700 1 -- 172.21.15.189:0/1984551512 learned_addr learned my addr 172.21.15.189:0/1984551512 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-20T21:20:43.505 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:43.506+0000 7f9089e00700 1 -- 172.21.15.189:0/1984551512 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f907c0095f0 con 0x7f908c0ffe20 2024-04-20T21:20:43.505 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:43.507+0000 7f9089e00700 1 --2- 172.21.15.189:0/1984551512 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f908c0ffe20 0x7f908c197150 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f907c0092e0 tx=0x7f907c009e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:20:43.506 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:43.507+0000 7f9082a00700 1 -- 172.21.15.189:0/1984551512 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f907c01d070 con 0x7f908c0ffe20 2024-04-20T21:20:43.507 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:43.508+0000 7f9082a00700 1 -- 172.21.15.189:0/1984551512 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f907c00be10 con 0x7f908c0ffe20 2024-04-20T21:20:43.507 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:43.508+0000 7f9091000700 1 -- 172.21.15.189:0/1984551512 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f908c197890 con 0x7f908c0ffe20 2024-04-20T21:20:43.507 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:43.508+0000 7f9082a00700 1 -- 172.21.15.189:0/1984551512 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f907c00f460 con 0x7f908c0ffe20 2024-04-20T21:20:43.507 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:43.508+0000 7f9091000700 1 -- 172.21.15.189:0/1984551512 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f908c197d30 con 0x7f908c0ffe20 2024-04-20T21:20:43.507 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:43.509+0000 7f9091000700 1 -- 172.21.15.189:0/1984551512 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f908c190e80 con 0x7f908c0ffe20 2024-04-20T21:20:43.508 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:43.510+0000 7f9082a00700 1 -- 172.21.15.189:0/1984551512 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 13) v1 ==== 45140+0+0 (secure 0 0 0) 0x7f907c022470 con 0x7f908c0ffe20 2024-04-20T21:20:43.509 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:43.510+0000 7f9082a00700 1 --2- 172.21.15.189:0/1984551512 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7f9070038490 0x7f907003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:20:43.509 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:43.510+0000 7f9082a00700 1 -- 172.21.15.189:0/1984551512 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f907c04ca90 con 0x7f908c0ffe20 2024-04-20T21:20:43.509 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:43.511+0000 7f9083e00700 1 --2- 172.21.15.189:0/1984551512 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7f9070038490 0x7f907003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:20:43.510 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:43.512+0000 7f9083e00700 1 --2- 172.21.15.189:0/1984551512 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7f9070038490 0x7f907003a940 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f9074006fd0 tx=0x7f9074006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:20:43.530 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:43.532+0000 7f9082a00700 1 -- 172.21.15.189:0/1984551512 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f907c022790 con 0x7f908c0ffe20 2024-04-20T21:20:43.807 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:43.808+0000 7f9091000700 1 -- 172.21.15.189:0/1984551512 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f908c02cc30 con 0x7f908c0ffe20 2024-04-20T21:20:43.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:43.809+0000 7f9082a00700 1 -- 172.21.15.189:0/1984551512 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f907c027020 con 0x7f908c0ffe20 2024-04-20T21:20:43.808 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-20T21:20:43.808 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":1,"fsid":"61ae3628-ff5b-11ee-bc93-c7b262605968","modified":"2024-04-20T21:18:14.797594Z","created":"2024-04-20T21:18:14.797594Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-20T21:20:43.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:43.812+0000 7f9091000700 1 -- 172.21.15.189:0/1984551512 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7f9070038490 msgr2=0x7f907003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:20:43.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:43.813+0000 7f9091000700 1 --2- 172.21.15.189:0/1984551512 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7f9070038490 0x7f907003a940 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f9074006fd0 tx=0x7f9074006e40 comp rx=0 tx=0).stop 2024-04-20T21:20:43.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:43.813+0000 7f9091000700 1 -- 172.21.15.189:0/1984551512 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f908c0ffe20 msgr2=0x7f908c197150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:20:43.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:43.813+0000 7f9091000700 1 --2- 172.21.15.189:0/1984551512 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f908c0ffe20 0x7f908c197150 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f907c0092e0 tx=0x7f907c009e60 comp rx=0 tx=0).stop 2024-04-20T21:20:43.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:43.813+0000 7f9091000700 1 -- 172.21.15.189:0/1984551512 shutdown_connections 2024-04-20T21:20:43.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:43.813+0000 7f9091000700 1 --2- 172.21.15.189:0/1984551512 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7f9070038490 0x7f907003a940 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:20:43.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:43.813+0000 7f9091000700 1 --2- 172.21.15.189:0/1984551512 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f908c0ffe20 0x7f908c197150 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:20:43.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:43.813+0000 7f9091000700 1 -- 172.21.15.189:0/1984551512 >> 172.21.15.189:0/1984551512 conn(0x7f908c0fb3d0 msgr2=0x7f908c0fd800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:20:43.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:43.813+0000 7f9091000700 1 -- 172.21.15.189:0/1984551512 shutdown_connections 2024-04-20T21:20:43.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:43.813+0000 7f9091000700 1 -- 172.21.15.189:0/1984551512 wait complete. 2024-04-20T21:20:43.813 INFO:teuthology.orchestra.run.smithi189.stderr:dumped monmap epoch 1 2024-04-20T21:20:44.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:20:44 smithi062 ceph-mon[16221]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:20:44.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:20:44 smithi062 ceph-mon[16221]: from='client.? 172.21.15.189:0/1984551512' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-20T21:20:45.716 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-20T21:20:45.716 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-20T21:20:46.070 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-20T21:20:46.173 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-20T21:20:46.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:20:46 smithi062 ceph-mon[16221]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:20:46.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:20:46 smithi062 ceph-mon[16221]: from='mgr.14164 172.21.15.62:0/570876096' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:20:48.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:20:47 smithi062 ceph-mon[16221]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:20:48.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:48.310+0000 7f0e87a00700 1 -- 172.21.15.189:0/2073753243 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e800ffe20 msgr2=0x7f0e80100230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:20:48.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:48.310+0000 7f0e87a00700 1 --2- 172.21.15.189:0/2073753243 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e800ffe20 0x7f0e80100230 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f0e74009910 tx=0x7f0e74009c20 comp rx=0 tx=0).stop 2024-04-20T21:20:48.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:48.310+0000 7f0e87a00700 1 -- 172.21.15.189:0/2073753243 shutdown_connections 2024-04-20T21:20:48.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:48.310+0000 7f0e87a00700 1 --2- 172.21.15.189:0/2073753243 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e800ffe20 0x7f0e80100230 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:20:48.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:48.310+0000 7f0e87a00700 1 -- 172.21.15.189:0/2073753243 >> 172.21.15.189:0/2073753243 conn(0x7f0e800fb3d0 msgr2=0x7f0e800fd800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:20:48.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:48.310+0000 7f0e87a00700 1 -- 172.21.15.189:0/2073753243 shutdown_connections 2024-04-20T21:20:48.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:48.310+0000 7f0e87a00700 1 -- 172.21.15.189:0/2073753243 wait complete. 2024-04-20T21:20:48.312 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:48.312+0000 7f0e87a00700 1 Processor -- start 2024-04-20T21:20:48.312 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:48.312+0000 7f0e87a00700 1 -- start start 2024-04-20T21:20:48.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:48.312+0000 7f0e87a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e800ffe20 0x7f0e80197150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:20:48.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:48.312+0000 7f0e87a00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e80197690 con 0x7f0e800ffe20 2024-04-20T21:20:48.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:48.313+0000 7f0e85000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e800ffe20 0x7f0e80197150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:20:48.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:48.313+0000 7f0e85000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e800ffe20 0x7f0e80197150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.189:39984/0 (socket says 172.21.15.189:39984) 2024-04-20T21:20:48.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:48.313+0000 7f0e85000700 1 -- 172.21.15.189:0/2235042523 learned_addr learned my addr 172.21.15.189:0/2235042523 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-20T21:20:48.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:48.314+0000 7f0e85000700 1 -- 172.21.15.189:0/2235042523 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e740095f0 con 0x7f0e800ffe20 2024-04-20T21:20:48.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:48.314+0000 7f0e85000700 1 --2- 172.21.15.189:0/2235042523 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e800ffe20 0x7f0e80197150 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f0e74005b40 tx=0x7f0e74004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:20:48.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:48.315+0000 7f0e7d600700 1 -- 172.21.15.189:0/2235042523 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0e7401d070 con 0x7f0e800ffe20 2024-04-20T21:20:48.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:48.315+0000 7f0e87a00700 1 -- 172.21.15.189:0/2235042523 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e80197890 con 0x7f0e800ffe20 2024-04-20T21:20:48.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:48.315+0000 7f0e87a00700 1 -- 172.21.15.189:0/2235042523 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e80197d30 con 0x7f0e800ffe20 2024-04-20T21:20:48.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:48.315+0000 7f0e7d600700 1 -- 172.21.15.189:0/2235042523 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f0e7400be10 con 0x7f0e800ffe20 2024-04-20T21:20:48.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:48.315+0000 7f0e7d600700 1 -- 172.21.15.189:0/2235042523 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0e7400f460 con 0x7f0e800ffe20 2024-04-20T21:20:48.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:48.316+0000 7f0e87a00700 1 -- 172.21.15.189:0/2235042523 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e80190e80 con 0x7f0e800ffe20 2024-04-20T21:20:48.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:48.317+0000 7f0e7d600700 1 -- 172.21.15.189:0/2235042523 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 13) v1 ==== 45140+0+0 (secure 0 0 0) 0x7f0e740227c0 con 0x7f0e800ffe20 2024-04-20T21:20:48.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:48.317+0000 7f0e7d600700 1 --2- 172.21.15.189:0/2235042523 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7f0e68038490 0x7f0e6803a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:20:48.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:48.317+0000 7f0e7d600700 1 -- 172.21.15.189:0/2235042523 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f0e7404c580 con 0x7f0e800ffe20 2024-04-20T21:20:48.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:48.318+0000 7f0e7fe00700 1 --2- 172.21.15.189:0/2235042523 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7f0e68038490 0x7f0e6803a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:20:48.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:48.319+0000 7f0e7fe00700 1 --2- 172.21.15.189:0/2235042523 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7f0e68038490 0x7f0e6803a940 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f0e6c006fd0 tx=0x7f0e6c006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:20:48.332 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:48.332+0000 7f0e7d600700 1 -- 172.21.15.189:0/2235042523 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f0e74022470 con 0x7f0e800ffe20 2024-04-20T21:20:48.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:48.634+0000 7f0e87a00700 1 -- 172.21.15.189:0/2235042523 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0e8002cc30 con 0x7f0e800ffe20 2024-04-20T21:20:48.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:48.635+0000 7f0e7d600700 1 -- 172.21.15.189:0/2235042523 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f0e74027020 con 0x7f0e800ffe20 2024-04-20T21:20:48.635 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-20T21:20:48.635 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":1,"fsid":"61ae3628-ff5b-11ee-bc93-c7b262605968","modified":"2024-04-20T21:18:14.797594Z","created":"2024-04-20T21:18:14.797594Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-20T21:20:48.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:48.638+0000 7f0e87a00700 1 -- 172.21.15.189:0/2235042523 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7f0e68038490 msgr2=0x7f0e6803a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:20:48.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:48.638+0000 7f0e87a00700 1 --2- 172.21.15.189:0/2235042523 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7f0e68038490 0x7f0e6803a940 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f0e6c006fd0 tx=0x7f0e6c006e40 comp rx=0 tx=0).stop 2024-04-20T21:20:48.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:48.638+0000 7f0e87a00700 1 -- 172.21.15.189:0/2235042523 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e800ffe20 msgr2=0x7f0e80197150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:20:48.637 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:48.638+0000 7f0e87a00700 1 --2- 172.21.15.189:0/2235042523 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e800ffe20 0x7f0e80197150 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f0e74005b40 tx=0x7f0e74004740 comp rx=0 tx=0).stop 2024-04-20T21:20:48.638 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:48.638+0000 7f0e87a00700 1 -- 172.21.15.189:0/2235042523 shutdown_connections 2024-04-20T21:20:48.639 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:48.638+0000 7f0e87a00700 1 --2- 172.21.15.189:0/2235042523 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7f0e68038490 0x7f0e6803a940 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:20:48.639 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:48.638+0000 7f0e87a00700 1 --2- 172.21.15.189:0/2235042523 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e800ffe20 0x7f0e80197150 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:20:48.639 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:48.638+0000 7f0e87a00700 1 -- 172.21.15.189:0/2235042523 >> 172.21.15.189:0/2235042523 conn(0x7f0e800fb3d0 msgr2=0x7f0e800fd800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:20:48.639 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:48.638+0000 7f0e87a00700 1 -- 172.21.15.189:0/2235042523 shutdown_connections 2024-04-20T21:20:48.639 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:48.638+0000 7f0e87a00700 1 -- 172.21.15.189:0/2235042523 wait complete. 2024-04-20T21:20:48.639 INFO:teuthology.orchestra.run.smithi189.stderr:dumped monmap epoch 1 2024-04-20T21:20:49.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:20:48 smithi062 ceph-mon[16221]: from='client.? 172.21.15.189:0/2235042523' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-20T21:20:50.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:20:49 smithi062 ceph-mon[16221]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:20:50.329 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-20T21:20:50.329 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-20T21:20:50.701 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-20T21:20:50.804 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-20T21:20:52.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:20:52 smithi062 ceph-mon[16221]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:20:52.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:52.733+0000 7fe226a00700 1 -- 172.21.15.189:0/1956217475 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe220102020 msgr2=0x7fe220102430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:20:52.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:52.733+0000 7fe226a00700 1 --2- 172.21.15.189:0/1956217475 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe220102020 0x7fe220102430 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fe210009910 tx=0x7fe210009c20 comp rx=0 tx=0).stop 2024-04-20T21:20:52.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:52.733+0000 7fe226a00700 1 -- 172.21.15.189:0/1956217475 shutdown_connections 2024-04-20T21:20:52.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:52.733+0000 7fe226a00700 1 --2- 172.21.15.189:0/1956217475 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe220102020 0x7fe220102430 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:20:52.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:52.733+0000 7fe226a00700 1 -- 172.21.15.189:0/1956217475 >> 172.21.15.189:0/1956217475 conn(0x7fe2200fd700 msgr2=0x7fe2200ffae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:20:52.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:52.734+0000 7fe226a00700 1 -- 172.21.15.189:0/1956217475 shutdown_connections 2024-04-20T21:20:52.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:52.734+0000 7fe226a00700 1 -- 172.21.15.189:0/1956217475 wait complete. 2024-04-20T21:20:52.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:52.735+0000 7fe226a00700 1 Processor -- start 2024-04-20T21:20:52.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:52.735+0000 7fe226a00700 1 -- start start 2024-04-20T21:20:52.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:52.735+0000 7fe226a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe220102020 0x7fe2201971a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:20:52.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:52.735+0000 7fe226a00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2201976e0 con 0x7fe220102020 2024-04-20T21:20:52.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:52.736+0000 7fe21fe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe220102020 0x7fe2201971a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:20:52.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:52.736+0000 7fe21fe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe220102020 0x7fe2201971a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.189:40004/0 (socket says 172.21.15.189:40004) 2024-04-20T21:20:52.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:52.736+0000 7fe21fe00700 1 -- 172.21.15.189:0/2607288662 learned_addr learned my addr 172.21.15.189:0/2607288662 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-20T21:20:52.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:52.737+0000 7fe21fe00700 1 -- 172.21.15.189:0/2607288662 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2100095f0 con 0x7fe220102020 2024-04-20T21:20:52.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:52.737+0000 7fe21fe00700 1 --2- 172.21.15.189:0/2607288662 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe220102020 0x7fe2201971a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fe210005b40 tx=0x7fe210004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:20:52.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:52.737+0000 7fe21d600700 1 -- 172.21.15.189:0/2607288662 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe21001d070 con 0x7fe220102020 2024-04-20T21:20:52.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:52.737+0000 7fe226a00700 1 -- 172.21.15.189:0/2607288662 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2201978e0 con 0x7fe220102020 2024-04-20T21:20:52.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:52.738+0000 7fe226a00700 1 -- 172.21.15.189:0/2607288662 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe220197d80 con 0x7fe220102020 2024-04-20T21:20:52.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:52.738+0000 7fe21d600700 1 -- 172.21.15.189:0/2607288662 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fe21000be10 con 0x7fe220102020 2024-04-20T21:20:52.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:52.738+0000 7fe21d600700 1 -- 172.21.15.189:0/2607288662 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe21000f460 con 0x7fe220102020 2024-04-20T21:20:52.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:52.739+0000 7fe226a00700 1 -- 172.21.15.189:0/2607288662 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1fc005320 con 0x7fe220102020 2024-04-20T21:20:52.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:52.739+0000 7fe21d600700 1 -- 172.21.15.189:0/2607288662 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 13) v1 ==== 45140+0+0 (secure 0 0 0) 0x7fe21000b960 con 0x7fe220102020 2024-04-20T21:20:52.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:52.739+0000 7fe21d600700 1 --2- 172.21.15.189:0/2607288662 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7fe2040384e0 0x7fe20403a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:20:52.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:52.739+0000 7fe21d600700 1 -- 172.21.15.189:0/2607288662 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fe21004c240 con 0x7fe220102020 2024-04-20T21:20:52.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:52.740+0000 7fe217400700 1 --2- 172.21.15.189:0/2607288662 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7fe2040384e0 0x7fe20403a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:20:52.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:52.741+0000 7fe217400700 1 --2- 172.21.15.189:0/2607288662 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7fe2040384e0 0x7fe20403a990 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe20800ad30 tx=0x7fe2080093f0 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:20:52.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:52.756+0000 7fe21d600700 1 -- 172.21.15.189:0/2607288662 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fe21000f5c0 con 0x7fe220102020 2024-04-20T21:20:53.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:53.055+0000 7fe226a00700 1 -- 172.21.15.189:0/2607288662 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe1fc005190 con 0x7fe220102020 2024-04-20T21:20:53.056 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:53.057+0000 7fe21d600700 1 -- 172.21.15.189:0/2607288662 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7fe210027020 con 0x7fe220102020 2024-04-20T21:20:53.056 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-20T21:20:53.057 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":1,"fsid":"61ae3628-ff5b-11ee-bc93-c7b262605968","modified":"2024-04-20T21:18:14.797594Z","created":"2024-04-20T21:18:14.797594Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-20T21:20:53.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:53.059+0000 7fe226a00700 1 -- 172.21.15.189:0/2607288662 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7fe2040384e0 msgr2=0x7fe20403a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:20:53.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:53.059+0000 7fe226a00700 1 --2- 172.21.15.189:0/2607288662 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7fe2040384e0 0x7fe20403a990 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe20800ad30 tx=0x7fe2080093f0 comp rx=0 tx=0).stop 2024-04-20T21:20:53.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:53.060+0000 7fe226a00700 1 -- 172.21.15.189:0/2607288662 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe220102020 msgr2=0x7fe2201971a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:20:53.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:53.060+0000 7fe226a00700 1 --2- 172.21.15.189:0/2607288662 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe220102020 0x7fe2201971a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fe210005b40 tx=0x7fe210004740 comp rx=0 tx=0).stop 2024-04-20T21:20:53.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:53.060+0000 7fe226a00700 1 -- 172.21.15.189:0/2607288662 shutdown_connections 2024-04-20T21:20:53.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:53.060+0000 7fe226a00700 1 --2- 172.21.15.189:0/2607288662 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7fe2040384e0 0x7fe20403a990 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:20:53.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:53.060+0000 7fe226a00700 1 --2- 172.21.15.189:0/2607288662 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe220102020 0x7fe2201971a0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:20:53.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:53.060+0000 7fe226a00700 1 -- 172.21.15.189:0/2607288662 >> 172.21.15.189:0/2607288662 conn(0x7fe2200fd700 msgr2=0x7fe2200ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:20:53.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:53.060+0000 7fe226a00700 1 -- 172.21.15.189:0/2607288662 shutdown_connections 2024-04-20T21:20:53.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:53.060+0000 7fe226a00700 1 -- 172.21.15.189:0/2607288662 wait complete. 2024-04-20T21:20:53.062 INFO:teuthology.orchestra.run.smithi189.stderr:dumped monmap epoch 1 2024-04-20T21:20:54.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:20:54 smithi062 ceph-mon[16221]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:20:54.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:20:54 smithi062 ceph-mon[16221]: from='client.? 172.21.15.189:0/2607288662' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-20T21:20:54.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:20:54 smithi062 ceph-mon[16221]: from='mgr.14164 172.21.15.62:0/570876096' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:20:54.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:20:54 smithi062 ceph-mon[16221]: from='mgr.14164 172.21.15.62:0/570876096' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:20:54.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:20:54 smithi062 ceph-mon[16221]: from='mgr.14164 172.21.15.62:0/570876096' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:20:54.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:20:54 smithi062 ceph-mon[16221]: from='mgr.14164 172.21.15.62:0/570876096' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-20T21:20:54.744 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-20T21:20:54.745 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-20T21:20:55.154 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-20T21:20:55.251 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-20T21:20:56.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:20:55 smithi062 ceph-mon[16221]: from='mgr.14164 172.21.15.62:0/570876096' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-04-20T21:20:56.037 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:20:55 smithi062 ceph-mon[16221]: mgrmap e14: smithi062.rqxuxi(active, since 79s) 2024-04-20T21:20:57.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:57.146+0000 7f178ec00700 1 -- 172.21.15.189:0/3706672032 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1788106440 msgr2=0x7f1788106850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:20:57.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:57.146+0000 7f178ec00700 1 --2- 172.21.15.189:0/3706672032 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1788106440 0x7f1788106850 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f1778009910 tx=0x7f1778009c20 comp rx=0 tx=0).stop 2024-04-20T21:20:57.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:57.146+0000 7f178ec00700 1 -- 172.21.15.189:0/3706672032 shutdown_connections 2024-04-20T21:20:57.146 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:57.146+0000 7f178ec00700 1 --2- 172.21.15.189:0/3706672032 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1788106440 0x7f1788106850 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:20:57.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:57.146+0000 7f178ec00700 1 -- 172.21.15.189:0/3706672032 >> 172.21.15.189:0/3706672032 conn(0x7f17880756e0 msgr2=0x7f1788077b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:20:57.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:57.146+0000 7f178ec00700 1 -- 172.21.15.189:0/3706672032 shutdown_connections 2024-04-20T21:20:57.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:57.146+0000 7f178ec00700 1 -- 172.21.15.189:0/3706672032 wait complete. 2024-04-20T21:20:57.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:57.147+0000 7f178ec00700 1 Processor -- start 2024-04-20T21:20:57.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:57.148+0000 7f178ec00700 1 -- start start 2024-04-20T21:20:57.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:57.148+0000 7f178ec00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1788106440 0x7f178819b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:20:57.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:57.148+0000 7f178ec00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f178819bae0 con 0x7f1788106440 2024-04-20T21:20:57.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:57.148+0000 7f1787e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1788106440 0x7f178819b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:20:57.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:57.149+0000 7f1787e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1788106440 0x7f178819b5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.189:40036/0 (socket says 172.21.15.189:40036) 2024-04-20T21:20:57.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:57.149+0000 7f1787e00700 1 -- 172.21.15.189:0/1615382402 learned_addr learned my addr 172.21.15.189:0/1615382402 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-20T21:20:57.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:57.149+0000 7f1787e00700 1 -- 172.21.15.189:0/1615382402 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17780095f0 con 0x7f1788106440 2024-04-20T21:20:57.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:57.150+0000 7f1787e00700 1 --2- 172.21.15.189:0/1615382402 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1788106440 0x7f178819b5a0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f1778005950 tx=0x7f1778009e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:20:57.150 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:57.150+0000 7f1784c00700 1 -- 172.21.15.189:0/1615382402 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f177801d070 con 0x7f1788106440 2024-04-20T21:20:57.150 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:57.150+0000 7f178ec00700 1 -- 172.21.15.189:0/1615382402 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f178819bce0 con 0x7f1788106440 2024-04-20T21:20:57.150 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:57.150+0000 7f1784c00700 1 -- 172.21.15.189:0/1615382402 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f177800be10 con 0x7f1788106440 2024-04-20T21:20:57.150 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:57.150+0000 7f1784c00700 1 -- 172.21.15.189:0/1615382402 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f177800f460 con 0x7f1788106440 2024-04-20T21:20:57.150 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:57.150+0000 7f178ec00700 1 -- 172.21.15.189:0/1615382402 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f178819c180 con 0x7f1788106440 2024-04-20T21:20:57.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:57.151+0000 7f178ec00700 1 -- 172.21.15.189:0/1615382402 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1788195110 con 0x7f1788106440 2024-04-20T21:20:57.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:57.152+0000 7f1784c00700 1 -- 172.21.15.189:0/1615382402 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 14) v1 ==== 45154+0+0 (secure 0 0 0) 0x7f17780227c0 con 0x7f1788106440 2024-04-20T21:20:57.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:57.152+0000 7f1784c00700 1 --2- 172.21.15.189:0/1615382402 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7f176c0384e0 0x7f176c03a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:20:57.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:57.152+0000 7f1784c00700 1 -- 172.21.15.189:0/1615382402 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f177804c4c0 con 0x7f1788106440 2024-04-20T21:20:57.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:57.152+0000 7f1787400700 1 -- 172.21.15.189:0/1615382402 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7f176c0384e0 msgr2=0x7f176c03a990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.62:6800/831506538 2024-04-20T21:20:57.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:57.152+0000 7f1787400700 1 --2- 172.21.15.189:0/1615382402 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7f176c0384e0 0x7f176c03a990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-20T21:20:57.162 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:57.162+0000 7f1784c00700 1 -- 172.21.15.189:0/1615382402 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f177802bba0 con 0x7f1788106440 2024-04-20T21:20:57.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:57.353+0000 7f1787400700 1 -- 172.21.15.189:0/1615382402 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7f176c0384e0 msgr2=0x7f176c03a990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.62:6800/831506538 2024-04-20T21:20:57.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:57.353+0000 7f1787400700 1 --2- 172.21.15.189:0/1615382402 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7f176c0384e0 0x7f176c03a990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-20T21:20:57.462 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:57.462+0000 7f178ec00700 1 -- 172.21.15.189:0/1615382402 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f178802cc30 con 0x7f1788106440 2024-04-20T21:20:57.463 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:57.464+0000 7f1784c00700 1 -- 172.21.15.189:0/1615382402 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f1778027030 con 0x7f1788106440 2024-04-20T21:20:57.464 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-20T21:20:57.464 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":1,"fsid":"61ae3628-ff5b-11ee-bc93-c7b262605968","modified":"2024-04-20T21:18:14.797594Z","created":"2024-04-20T21:18:14.797594Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-20T21:20:57.467 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:57.466+0000 7f178ec00700 1 -- 172.21.15.189:0/1615382402 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7f176c0384e0 msgr2=0x7f176c03a990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-20T21:20:57.467 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:57.467+0000 7f178ec00700 1 --2- 172.21.15.189:0/1615382402 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7f176c0384e0 0x7f176c03a990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:20:57.467 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:57.467+0000 7f178ec00700 1 -- 172.21.15.189:0/1615382402 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1788106440 msgr2=0x7f178819b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:20:57.467 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:57.467+0000 7f178ec00700 1 --2- 172.21.15.189:0/1615382402 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1788106440 0x7f178819b5a0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f1778005950 tx=0x7f1778009e60 comp rx=0 tx=0).stop 2024-04-20T21:20:57.467 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:57.467+0000 7f178ec00700 1 -- 172.21.15.189:0/1615382402 shutdown_connections 2024-04-20T21:20:57.467 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:57.467+0000 7f178ec00700 1 --2- 172.21.15.189:0/1615382402 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7f176c0384e0 0x7f176c03a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:20:57.467 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:57.467+0000 7f178ec00700 1 --2- 172.21.15.189:0/1615382402 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1788106440 0x7f178819b5a0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:20:57.467 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:57.467+0000 7f178ec00700 1 -- 172.21.15.189:0/1615382402 >> 172.21.15.189:0/1615382402 conn(0x7f17880756e0 msgr2=0x7f17880773a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:20:57.467 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:57.467+0000 7f178ec00700 1 -- 172.21.15.189:0/1615382402 shutdown_connections 2024-04-20T21:20:57.467 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:20:57.467+0000 7f178ec00700 1 -- 172.21.15.189:0/1615382402 wait complete. 2024-04-20T21:20:57.468 INFO:teuthology.orchestra.run.smithi189.stderr:dumped monmap epoch 1 2024-04-20T21:20:57.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:20:57 smithi062 ceph-mon[16221]: from='client.? 172.21.15.189:0/1615382402' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-20T21:20:59.194 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-20T21:20:59.194 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-20T21:20:59.572 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-20T21:20:59.677 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-20T21:21:01.608 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:01.608+0000 7fc6ec800700 1 -- 172.21.15.189:0/192904519 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc6e4106440 msgr2=0x7fc6e4106850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:21:01.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:01.608+0000 7fc6ec800700 1 --2- 172.21.15.189:0/192904519 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc6e4106440 0x7fc6e4106850 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fc6d8009910 tx=0x7fc6d8009c20 comp rx=0 tx=0).stop 2024-04-20T21:21:01.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:01.608+0000 7fc6ec800700 1 -- 172.21.15.189:0/192904519 shutdown_connections 2024-04-20T21:21:01.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:01.608+0000 7fc6ec800700 1 --2- 172.21.15.189:0/192904519 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc6e4106440 0x7fc6e4106850 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:21:01.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:01.608+0000 7fc6ec800700 1 -- 172.21.15.189:0/192904519 >> 172.21.15.189:0/192904519 conn(0x7fc6e40756e0 msgr2=0x7fc6e4077b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:21:01.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:01.608+0000 7fc6ec800700 1 -- 172.21.15.189:0/192904519 shutdown_connections 2024-04-20T21:21:01.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:01.608+0000 7fc6ec800700 1 -- 172.21.15.189:0/192904519 wait complete. 2024-04-20T21:21:01.609 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:01.610+0000 7fc6ec800700 1 Processor -- start 2024-04-20T21:21:01.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:01.610+0000 7fc6ec800700 1 -- start start 2024-04-20T21:21:01.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:01.610+0000 7fc6ec800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc6e419b7b0 0x7fc6e419bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:21:01.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:01.610+0000 7fc6ec800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6e419c100 con 0x7fc6e419b7b0 2024-04-20T21:21:01.610 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:01.611+0000 7fc6e9e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc6e419b7b0 0x7fc6e419bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:21:01.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:01.611+0000 7fc6e9e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc6e419b7b0 0x7fc6e419bbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.189:56496/0 (socket says 172.21.15.189:56496) 2024-04-20T21:21:01.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:01.611+0000 7fc6e9e00700 1 -- 172.21.15.189:0/739337681 learned_addr learned my addr 172.21.15.189:0/739337681 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-20T21:21:01.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:01.612+0000 7fc6e9e00700 1 -- 172.21.15.189:0/739337681 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6d80095f0 con 0x7fc6e419b7b0 2024-04-20T21:21:01.611 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:01.612+0000 7fc6e9e00700 1 --2- 172.21.15.189:0/739337681 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc6e419b7b0 0x7fc6e419bbc0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fc6d8005950 tx=0x7fc6d8009f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:21:01.612 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:01.612+0000 7fc6dea00700 1 -- 172.21.15.189:0/739337681 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc6d801d070 con 0x7fc6e419b7b0 2024-04-20T21:21:01.612 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:01.612+0000 7fc6ec800700 1 -- 172.21.15.189:0/739337681 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6e419c300 con 0x7fc6e419b7b0 2024-04-20T21:21:01.612 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:01.613+0000 7fc6dea00700 1 -- 172.21.15.189:0/739337681 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc6d800be10 con 0x7fc6e419b7b0 2024-04-20T21:21:01.612 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:01.613+0000 7fc6ec800700 1 -- 172.21.15.189:0/739337681 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6e419ef60 con 0x7fc6e419b7b0 2024-04-20T21:21:01.612 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:01.613+0000 7fc6dea00700 1 -- 172.21.15.189:0/739337681 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc6d8022db0 con 0x7fc6e419b7b0 2024-04-20T21:21:01.614 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:01.614+0000 7fc6ec800700 1 -- 172.21.15.189:0/739337681 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6e4195110 con 0x7fc6e419b7b0 2024-04-20T21:21:01.614 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:01.614+0000 7fc6dea00700 1 -- 172.21.15.189:0/739337681 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 14) v1 ==== 45154+0+0 (secure 0 0 0) 0x7fc6d8022690 con 0x7fc6e419b7b0 2024-04-20T21:21:01.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:01.615+0000 7fc6dea00700 1 --2- 172.21.15.189:0/739337681 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7fc6cc0384e0 0x7fc6cc03a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:21:01.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:01.615+0000 7fc6dea00700 1 -- 172.21.15.189:0/739337681 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fc6d804c350 con 0x7fc6e419b7b0 2024-04-20T21:21:01.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:01.615+0000 7fc6e9400700 1 -- 172.21.15.189:0/739337681 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7fc6cc0384e0 msgr2=0x7fc6cc03a990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.62:6800/831506538 2024-04-20T21:21:01.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:01.615+0000 7fc6e9400700 1 --2- 172.21.15.189:0/739337681 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7fc6cc0384e0 0x7fc6cc03a990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-20T21:21:01.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:01.629+0000 7fc6dea00700 1 -- 172.21.15.189:0/739337681 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fc6d800b960 con 0x7fc6e419b7b0 2024-04-20T21:21:01.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:01.816+0000 7fc6e9400700 1 -- 172.21.15.189:0/739337681 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7fc6cc0384e0 msgr2=0x7fc6cc03a990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.62:6800/831506538 2024-04-20T21:21:01.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:01.816+0000 7fc6e9400700 1 --2- 172.21.15.189:0/739337681 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7fc6cc0384e0 0x7fc6cc03a990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-20T21:21:01.929 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:01.930+0000 7fc6ec800700 1 -- 172.21.15.189:0/739337681 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc6e402cc30 con 0x7fc6e419b7b0 2024-04-20T21:21:01.931 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:01.931+0000 7fc6dea00700 1 -- 172.21.15.189:0/739337681 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7fc6d8027030 con 0x7fc6e419b7b0 2024-04-20T21:21:01.931 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-20T21:21:01.932 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":1,"fsid":"61ae3628-ff5b-11ee-bc93-c7b262605968","modified":"2024-04-20T21:18:14.797594Z","created":"2024-04-20T21:18:14.797594Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-20T21:21:01.933 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:01.934+0000 7fc6ec800700 1 -- 172.21.15.189:0/739337681 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7fc6cc0384e0 msgr2=0x7fc6cc03a990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-20T21:21:01.934 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:01.934+0000 7fc6ec800700 1 --2- 172.21.15.189:0/739337681 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7fc6cc0384e0 0x7fc6cc03a990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:21:01.934 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:01.934+0000 7fc6ec800700 1 -- 172.21.15.189:0/739337681 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc6e419b7b0 msgr2=0x7fc6e419bbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:21:01.934 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:01.934+0000 7fc6ec800700 1 --2- 172.21.15.189:0/739337681 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc6e419b7b0 0x7fc6e419bbc0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fc6d8005950 tx=0x7fc6d8009f10 comp rx=0 tx=0).stop 2024-04-20T21:21:01.934 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:01.934+0000 7fc6ec800700 1 -- 172.21.15.189:0/739337681 shutdown_connections 2024-04-20T21:21:01.934 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:01.934+0000 7fc6ec800700 1 --2- 172.21.15.189:0/739337681 >> [v2:172.21.15.62:6800/831506538,v1:172.21.15.62:6801/831506538] conn(0x7fc6cc0384e0 0x7fc6cc03a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:21:01.934 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:01.934+0000 7fc6ec800700 1 --2- 172.21.15.189:0/739337681 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc6e419b7b0 0x7fc6e419bbc0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:21:01.934 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:01.934+0000 7fc6ec800700 1 -- 172.21.15.189:0/739337681 >> 172.21.15.189:0/739337681 conn(0x7fc6e40756e0 msgr2=0x7fc6e4077490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:21:01.934 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:01.934+0000 7fc6ec800700 1 -- 172.21.15.189:0/739337681 shutdown_connections 2024-04-20T21:21:01.934 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:01.934+0000 7fc6ec800700 1 -- 172.21.15.189:0/739337681 wait complete. 2024-04-20T21:21:01.935 INFO:teuthology.orchestra.run.smithi189.stderr:dumped monmap epoch 1 2024-04-20T21:21:03.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:02 smithi062 ceph-mon[16221]: from='client.? 172.21.15.189:0/739337681' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-20T21:21:03.591 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-20T21:21:03.592 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-20T21:21:03.973 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-20T21:21:04.028 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:03 smithi062 ceph-mon[16221]: Active manager daemon smithi062.rqxuxi restarted 2024-04-20T21:21:04.028 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:03 smithi062 ceph-mon[16221]: Activating manager daemon smithi062.rqxuxi 2024-04-20T21:21:04.028 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:03 smithi062 ceph-mon[16221]: osdmap e5: 0 total, 0 up, 0 in 2024-04-20T21:21:04.028 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:03 smithi062 ceph-mon[16221]: mgrmap e15: smithi062.rqxuxi(active, starting, since 0.00811463s) 2024-04-20T21:21:04.028 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:03 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mon metadata", "id": "smithi062"}]: dispatch 2024-04-20T21:21:04.028 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:03 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mgr metadata", "who": "smithi062.rqxuxi", "id": "smithi062.rqxuxi"}]: dispatch 2024-04-20T21:21:04.028 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:03 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-20T21:21:04.029 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:03 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-20T21:21:04.029 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:03 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-20T21:21:04.029 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:03 smithi062 ceph-mon[16221]: Manager daemon smithi062.rqxuxi is now available 2024-04-20T21:21:04.074 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-20T21:21:05.208 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:04 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:21:05.208 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:04 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:21:05.208 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:04 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:21:05.209 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:04 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:21:05.209 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:04 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi062.rqxuxi/mirror_snapshot_schedule"}]: dispatch 2024-04-20T21:21:05.209 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:04 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi062.rqxuxi/trash_purge_schedule"}]: dispatch 2024-04-20T21:21:05.209 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:04 smithi062 ceph-mon[16221]: mgrmap e16: smithi062.rqxuxi(active, since 1.0084s) 2024-04-20T21:21:06.126 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:06.125+0000 7fcacb200700 1 -- 172.21.15.189:0/1153051132 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcacc1006f0 msgr2=0x7fcacc102ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:21:06.126 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:06.125+0000 7fcacb200700 1 --2- 172.21.15.189:0/1153051132 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcacc1006f0 0x7fcacc102ad0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fcabc009910 tx=0x7fcabc009c20 comp rx=0 tx=0).stop 2024-04-20T21:21:06.126 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:06.125+0000 7fcacb200700 1 -- 172.21.15.189:0/1153051132 shutdown_connections 2024-04-20T21:21:06.126 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:06.125+0000 7fcacb200700 1 --2- 172.21.15.189:0/1153051132 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcacc1006f0 0x7fcacc102ad0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:21:06.126 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:06.125+0000 7fcacb200700 1 -- 172.21.15.189:0/1153051132 >> 172.21.15.189:0/1153051132 conn(0x7fcacc0fa110 msgr2=0x7fcacc0fc540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:21:06.126 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:06.126+0000 7fcacb200700 1 -- 172.21.15.189:0/1153051132 shutdown_connections 2024-04-20T21:21:06.126 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:06.126+0000 7fcacb200700 1 -- 172.21.15.189:0/1153051132 wait complete. 2024-04-20T21:21:06.127 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:06.127+0000 7fcacb200700 1 Processor -- start 2024-04-20T21:21:06.127 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:06.127+0000 7fcacb200700 1 -- start start 2024-04-20T21:21:06.127 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:06.128+0000 7fcacb200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcacc1006f0 0x7fcacc194db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:21:06.127 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:06.128+0000 7fcacb200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcacc1952f0 con 0x7fcacc1006f0 2024-04-20T21:21:06.128 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:06.128+0000 7fcac9e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcacc1006f0 0x7fcacc194db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:21:06.128 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:06.128+0000 7fcac9e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcacc1006f0 0x7fcacc194db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.189:56512/0 (socket says 172.21.15.189:56512) 2024-04-20T21:21:06.128 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:06.128+0000 7fcac9e00700 1 -- 172.21.15.189:0/1221007350 learned_addr learned my addr 172.21.15.189:0/1221007350 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-20T21:21:06.128 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:06.129+0000 7fcac9e00700 1 -- 172.21.15.189:0/1221007350 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcabc0095f0 con 0x7fcacc1006f0 2024-04-20T21:21:06.129 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:06.129+0000 7fcac9e00700 1 --2- 172.21.15.189:0/1221007350 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcacc1006f0 0x7fcacc194db0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fcabc0092e0 tx=0x7fcabc009e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:21:06.129 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:06.129+0000 7fcac2a00700 1 -- 172.21.15.189:0/1221007350 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcabc01d070 con 0x7fcacc1006f0 2024-04-20T21:21:06.129 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:06.129+0000 7fcac2a00700 1 -- 172.21.15.189:0/1221007350 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fcabc00be10 con 0x7fcacc1006f0 2024-04-20T21:21:06.129 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:06.129+0000 7fcacb200700 1 -- 172.21.15.189:0/1221007350 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcacc1954f0 con 0x7fcacc1006f0 2024-04-20T21:21:06.129 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:06.129+0000 7fcac2a00700 1 -- 172.21.15.189:0/1221007350 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcabc00f460 con 0x7fcacc1006f0 2024-04-20T21:21:06.129 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:06.129+0000 7fcacb200700 1 -- 172.21.15.189:0/1221007350 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcacc195990 con 0x7fcacc1006f0 2024-04-20T21:21:06.130 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:06.131+0000 7fcac2a00700 1 -- 172.21.15.189:0/1221007350 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 45197+0+0 (secure 0 0 0) 0x7fcabc022470 con 0x7fcacc1006f0 2024-04-20T21:21:06.130 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:06.131+0000 7fcacb200700 1 -- 172.21.15.189:0/1221007350 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcacc18e940 con 0x7fcacc1006f0 2024-04-20T21:21:06.131 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:06.131+0000 7fcac2a00700 1 --2- 172.21.15.189:0/1221007350 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fcab00385d0 0x7fcab003aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:21:06.131 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:06.131+0000 7fcac2a00700 1 -- 172.21.15.189:0/1221007350 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7fcabc04c750 con 0x7fcacc1006f0 2024-04-20T21:21:06.131 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:06.132+0000 7fcac9400700 1 --2- 172.21.15.189:0/1221007350 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fcab00385d0 0x7fcab003aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:21:06.132 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:06.133+0000 7fcac9400700 1 --2- 172.21.15.189:0/1221007350 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fcab00385d0 0x7fcab003aa80 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fcab8006fd0 tx=0x7fcab8006e40 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:21:06.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:06.137+0000 7fcac2a00700 1 -- 172.21.15.189:0/1221007350 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fcabc00b960 con 0x7fcacc1006f0 2024-04-20T21:21:06.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:05 smithi062 ceph-mon[16221]: [20/Apr/2024:21:21:04] ENGINE Bus STARTING 2024-04-20T21:21:06.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:05 smithi062 ceph-mon[16221]: [20/Apr/2024:21:21:04] ENGINE Serving on https://172.21.15.62:7150 2024-04-20T21:21:06.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:05 smithi062 ceph-mon[16221]: [20/Apr/2024:21:21:04] ENGINE Serving on http://172.21.15.62:8765 2024-04-20T21:21:06.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:05 smithi062 ceph-mon[16221]: [20/Apr/2024:21:21:04] ENGINE Bus STARTED 2024-04-20T21:21:06.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:05 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:21:06.442 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:06.442+0000 7fcacb200700 1 -- 172.21.15.189:0/1221007350 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcacc02cc30 con 0x7fcacc1006f0 2024-04-20T21:21:06.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:06.443+0000 7fcac2a00700 1 -- 172.21.15.189:0/1221007350 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7fcabc027070 con 0x7fcacc1006f0 2024-04-20T21:21:06.446 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-20T21:21:06.446 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":1,"fsid":"61ae3628-ff5b-11ee-bc93-c7b262605968","modified":"2024-04-20T21:18:14.797594Z","created":"2024-04-20T21:18:14.797594Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-20T21:21:06.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:06.446+0000 7fcacb200700 1 -- 172.21.15.189:0/1221007350 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fcab00385d0 msgr2=0x7fcab003aa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:21:06.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:06.446+0000 7fcacb200700 1 --2- 172.21.15.189:0/1221007350 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fcab00385d0 0x7fcab003aa80 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fcab8006fd0 tx=0x7fcab8006e40 comp rx=0 tx=0).stop 2024-04-20T21:21:06.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:06.446+0000 7fcacb200700 1 -- 172.21.15.189:0/1221007350 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcacc1006f0 msgr2=0x7fcacc194db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:21:06.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:06.446+0000 7fcacb200700 1 --2- 172.21.15.189:0/1221007350 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcacc1006f0 0x7fcacc194db0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fcabc0092e0 tx=0x7fcabc009e60 comp rx=0 tx=0).stop 2024-04-20T21:21:06.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:06.446+0000 7fcacb200700 1 -- 172.21.15.189:0/1221007350 shutdown_connections 2024-04-20T21:21:06.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:06.446+0000 7fcacb200700 1 --2- 172.21.15.189:0/1221007350 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fcab00385d0 0x7fcab003aa80 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:21:06.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:06.446+0000 7fcacb200700 1 --2- 172.21.15.189:0/1221007350 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcacc1006f0 0x7fcacc194db0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:21:06.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:06.446+0000 7fcacb200700 1 -- 172.21.15.189:0/1221007350 >> 172.21.15.189:0/1221007350 conn(0x7fcacc0fa110 msgr2=0x7fcacc0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:21:06.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:06.446+0000 7fcacb200700 1 -- 172.21.15.189:0/1221007350 shutdown_connections 2024-04-20T21:21:06.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:06.446+0000 7fcacb200700 1 -- 172.21.15.189:0/1221007350 wait complete. 2024-04-20T21:21:06.448 INFO:teuthology.orchestra.run.smithi189.stderr:dumped monmap epoch 1 2024-04-20T21:21:07.369 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:07 smithi062 ceph-mon[16221]: mgrmap e17: smithi062.rqxuxi(active, since 2s) 2024-04-20T21:21:07.369 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:07 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:21:07.369 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:07 smithi062 ceph-mon[16221]: from='client.? 172.21.15.189:0/1221007350' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-20T21:21:08.207 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-20T21:21:08.208 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-20T21:21:08.575 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-20T21:21:09.252 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-20T21:21:09.284 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:08 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:21:11.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:11 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:21:11.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:11 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:21:12.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:12.106+0000 7fc975e00700 1 -- 172.21.15.189:0/2380215690 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9700ffe20 msgr2=0x7fc970100230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:21:12.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:12.106+0000 7fc975e00700 1 --2- 172.21.15.189:0/2380215690 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9700ffe20 0x7fc970100230 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fc96000b0a0 tx=0x7fc96000b3b0 comp rx=0 tx=0).stop 2024-04-20T21:21:12.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:12.106+0000 7fc975e00700 1 -- 172.21.15.189:0/2380215690 shutdown_connections 2024-04-20T21:21:12.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:12.106+0000 7fc975e00700 1 --2- 172.21.15.189:0/2380215690 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9700ffe20 0x7fc970100230 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:21:12.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:12.106+0000 7fc975e00700 1 -- 172.21.15.189:0/2380215690 >> 172.21.15.189:0/2380215690 conn(0x7fc9700fb3d0 msgr2=0x7fc9700fd800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:21:12.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:12.107+0000 7fc975e00700 1 -- 172.21.15.189:0/2380215690 shutdown_connections 2024-04-20T21:21:12.274 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:12.107+0000 7fc975e00700 1 -- 172.21.15.189:0/2380215690 wait complete. 2024-04-20T21:21:12.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:12.108+0000 7fc975e00700 1 Processor -- start 2024-04-20T21:21:12.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:12.108+0000 7fc975e00700 1 -- start start 2024-04-20T21:21:12.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:12.108+0000 7fc975e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9700ffe20 0x7fc970194f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:21:12.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:12.108+0000 7fc975e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9701954c0 con 0x7fc9700ffe20 2024-04-20T21:21:12.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:12.109+0000 7fc96ea00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9700ffe20 0x7fc970194f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:21:12.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:12.109+0000 7fc96ea00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9700ffe20 0x7fc970194f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.189:50462/0 (socket says 172.21.15.189:50462) 2024-04-20T21:21:12.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:12.109+0000 7fc96ea00700 1 -- 172.21.15.189:0/3918970147 learned_addr learned my addr 172.21.15.189:0/3918970147 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-20T21:21:12.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:12.110+0000 7fc96ea00700 1 -- 172.21.15.189:0/3918970147 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc960009d00 con 0x7fc9700ffe20 2024-04-20T21:21:12.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:12.110+0000 7fc96ea00700 1 --2- 172.21.15.189:0/3918970147 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9700ffe20 0x7fc970194f80 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fc960015040 tx=0x7fc96000bb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:21:12.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:12.110+0000 7fc974a00700 1 -- 172.21.15.189:0/3918970147 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc96000e040 con 0x7fc9700ffe20 2024-04-20T21:21:12.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:12.111+0000 7fc975e00700 1 -- 172.21.15.189:0/3918970147 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9701956c0 con 0x7fc9700ffe20 2024-04-20T21:21:12.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:12.111+0000 7fc975e00700 1 -- 172.21.15.189:0/3918970147 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc970195ae0 con 0x7fc9700ffe20 2024-04-20T21:21:12.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:12.111+0000 7fc974a00700 1 -- 172.21.15.189:0/3918970147 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc96001d930 con 0x7fc9700ffe20 2024-04-20T21:21:12.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:12.111+0000 7fc974a00700 1 -- 172.21.15.189:0/3918970147 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc960027980 con 0x7fc9700ffe20 2024-04-20T21:21:12.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:12.112+0000 7fc975e00700 1 -- 172.21.15.189:0/3918970147 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc97018ea80 con 0x7fc9700ffe20 2024-04-20T21:21:12.275 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:12.112+0000 7fc974a00700 1 -- 172.21.15.189:0/3918970147 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 45197+0+0 (secure 0 0 0) 0x7fc96001daa0 con 0x7fc9700ffe20 2024-04-20T21:21:12.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:12.113+0000 7fc974a00700 1 --2- 172.21.15.189:0/3918970147 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fc9540385d0 0x7fc95403aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:21:12.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:12.113+0000 7fc974a00700 1 -- 172.21.15.189:0/3918970147 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7fc96004c830 con 0x7fc9700ffe20 2024-04-20T21:21:12.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:12.113+0000 7fc966000700 1 --2- 172.21.15.189:0/3918970147 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fc9540385d0 0x7fc95403aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:21:12.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:12.114+0000 7fc966000700 1 --2- 172.21.15.189:0/3918970147 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fc9540385d0 0x7fc95403aa80 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fc958006fd0 tx=0x7fc958006e40 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:21:12.276 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:12.128+0000 7fc974a00700 1 -- 172.21.15.189:0/3918970147 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fc960026a10 con 0x7fc9700ffe20 2024-04-20T21:21:12.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:12.443+0000 7fc975e00700 1 -- 172.21.15.189:0/3918970147 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc970062380 con 0x7fc9700ffe20 2024-04-20T21:21:12.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:12.444+0000 7fc974a00700 1 -- 172.21.15.189:0/3918970147 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7fc960026830 con 0x7fc9700ffe20 2024-04-20T21:21:12.444 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-20T21:21:12.444 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":1,"fsid":"61ae3628-ff5b-11ee-bc93-c7b262605968","modified":"2024-04-20T21:18:14.797594Z","created":"2024-04-20T21:18:14.797594Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-20T21:21:12.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:12.447+0000 7fc975e00700 1 -- 172.21.15.189:0/3918970147 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fc9540385d0 msgr2=0x7fc95403aa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:21:12.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:12.447+0000 7fc975e00700 1 --2- 172.21.15.189:0/3918970147 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fc9540385d0 0x7fc95403aa80 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fc958006fd0 tx=0x7fc958006e40 comp rx=0 tx=0).stop 2024-04-20T21:21:12.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:12.447+0000 7fc975e00700 1 -- 172.21.15.189:0/3918970147 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9700ffe20 msgr2=0x7fc970194f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:21:12.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:12.447+0000 7fc975e00700 1 --2- 172.21.15.189:0/3918970147 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9700ffe20 0x7fc970194f80 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fc960015040 tx=0x7fc96000bb10 comp rx=0 tx=0).stop 2024-04-20T21:21:12.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:12.447+0000 7fc975e00700 1 -- 172.21.15.189:0/3918970147 shutdown_connections 2024-04-20T21:21:12.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:12.447+0000 7fc975e00700 1 --2- 172.21.15.189:0/3918970147 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fc9540385d0 0x7fc95403aa80 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:21:12.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:12.447+0000 7fc975e00700 1 --2- 172.21.15.189:0/3918970147 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9700ffe20 0x7fc970194f80 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:21:12.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:12.447+0000 7fc975e00700 1 -- 172.21.15.189:0/3918970147 >> 172.21.15.189:0/3918970147 conn(0x7fc9700fb3d0 msgr2=0x7fc9700fc080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:21:12.448 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:12.447+0000 7fc975e00700 1 -- 172.21.15.189:0/3918970147 shutdown_connections 2024-04-20T21:21:12.448 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:12.447+0000 7fc975e00700 1 -- 172.21.15.189:0/3918970147 wait complete. 2024-04-20T21:21:12.448 INFO:teuthology.orchestra.run.smithi189.stderr:dumped monmap epoch 1 2024-04-20T21:21:13.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:12 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:21:13.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:12 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:21:13.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:12 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "who": "osd/host:smithi062", "name": "osd_memory_target"}]: dispatch 2024-04-20T21:21:13.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:12 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:21:13.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:12 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:21:13.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:12 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:21:13.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:12 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:21:13.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:12 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "who": "osd/host:smithi189", "name": "osd_memory_target"}]: dispatch 2024-04-20T21:21:13.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:12 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:21:13.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:12 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T21:21:13.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:12 smithi062 ceph-mon[16221]: Updating smithi062:/etc/ceph/ceph.conf 2024-04-20T21:21:13.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:12 smithi062 ceph-mon[16221]: Updating smithi189:/etc/ceph/ceph.conf 2024-04-20T21:21:13.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:12 smithi062 ceph-mon[16221]: from='client.? 172.21.15.189:0/3918970147' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-20T21:21:14.161 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-20T21:21:14.162 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-20T21:21:14.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:13 smithi062 ceph-mon[16221]: Updating smithi189:/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-20T21:21:14.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:13 smithi062 ceph-mon[16221]: Updating smithi062:/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-20T21:21:14.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:13 smithi062 ceph-mon[16221]: Updating smithi189:/etc/ceph/ceph.client.admin.keyring 2024-04-20T21:21:14.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:13 smithi062 ceph-mon[16221]: Updating smithi062:/etc/ceph/ceph.client.admin.keyring 2024-04-20T21:21:14.506 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-20T21:21:15.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:14 smithi062 ceph-mon[16221]: Updating smithi189:/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-20T21:21:15.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:14 smithi062 ceph-mon[16221]: Updating smithi062:/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-20T21:21:16.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:16.443+0000 7fc47ea00700 1 -- 172.21.15.189:0/428943363 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc478100090 msgr2=0x7fc478100460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:21:16.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:16.443+0000 7fc47ea00700 1 --2- 172.21.15.189:0/428943363 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc478100090 0x7fc478100460 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fc468009910 tx=0x7fc468009c20 comp rx=0 tx=0).stop 2024-04-20T21:21:16.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:16.443+0000 7fc47ea00700 1 -- 172.21.15.189:0/428943363 shutdown_connections 2024-04-20T21:21:16.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:16.443+0000 7fc47ea00700 1 --2- 172.21.15.189:0/428943363 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc478100090 0x7fc478100460 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:21:16.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:16.443+0000 7fc47ea00700 1 -- 172.21.15.189:0/428943363 >> 172.21.15.189:0/428943363 conn(0x7fc4780fb970 msgr2=0x7fc4780fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:21:16.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:16.444+0000 7fc47ea00700 1 -- 172.21.15.189:0/428943363 shutdown_connections 2024-04-20T21:21:16.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:16.444+0000 7fc47ea00700 1 -- 172.21.15.189:0/428943363 wait complete. 2024-04-20T21:21:16.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:16.445+0000 7fc47ea00700 1 Processor -- start 2024-04-20T21:21:16.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:16.445+0000 7fc47ea00700 1 -- start start 2024-04-20T21:21:16.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:16.446+0000 7fc47ea00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc478100090 0x7fc478197520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:21:16.445 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:16.446+0000 7fc47ea00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc478197a60 con 0x7fc478100090 2024-04-20T21:21:16.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:16.446+0000 7fc477e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc478100090 0x7fc478197520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:21:16.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:16.447+0000 7fc477e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc478100090 0x7fc478197520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.189:50482/0 (socket says 172.21.15.189:50482) 2024-04-20T21:21:16.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:16.447+0000 7fc477e00700 1 -- 172.21.15.189:0/1820978294 learned_addr learned my addr 172.21.15.189:0/1820978294 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-20T21:21:16.446 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:16.447+0000 7fc477e00700 1 -- 172.21.15.189:0/1820978294 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4680095f0 con 0x7fc478100090 2024-04-20T21:21:16.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:16.448+0000 7fc477e00700 1 --2- 172.21.15.189:0/1820978294 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc478100090 0x7fc478197520 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fc4680092e0 tx=0x7fc468004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:21:16.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:16.448+0000 7fc474c00700 1 -- 172.21.15.189:0/1820978294 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc46801d070 con 0x7fc478100090 2024-04-20T21:21:16.448 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:16.448+0000 7fc47ea00700 1 -- 172.21.15.189:0/1820978294 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc478197c60 con 0x7fc478100090 2024-04-20T21:21:16.448 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:16.448+0000 7fc474c00700 1 -- 172.21.15.189:0/1820978294 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc46800be10 con 0x7fc478100090 2024-04-20T21:21:16.448 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:16.448+0000 7fc474c00700 1 -- 172.21.15.189:0/1820978294 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc468022e10 con 0x7fc478100090 2024-04-20T21:21:16.448 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:16.448+0000 7fc47ea00700 1 -- 172.21.15.189:0/1820978294 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc478198080 con 0x7fc478100090 2024-04-20T21:21:16.449 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:16.450+0000 7fc474c00700 1 -- 172.21.15.189:0/1820978294 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 45197+0+0 (secure 0 0 0) 0x7fc46800f460 con 0x7fc478100090 2024-04-20T21:21:16.450 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:16.450+0000 7fc47ea00700 1 -- 172.21.15.189:0/1820978294 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc478062380 con 0x7fc478100090 2024-04-20T21:21:16.450 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:16.450+0000 7fc474c00700 1 --2- 172.21.15.189:0/1820978294 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fc45c038530 0x7fc45c03a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:21:16.450 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:16.450+0000 7fc474c00700 1 -- 172.21.15.189:0/1820978294 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7fc46804c4e0 con 0x7fc478100090 2024-04-20T21:21:16.450 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:16.451+0000 7fc477400700 1 --2- 172.21.15.189:0/1820978294 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fc45c038530 0x7fc45c03a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:21:16.451 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:16.452+0000 7fc477400700 1 --2- 172.21.15.189:0/1820978294 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fc45c038530 0x7fc45c03a9e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc464006fd0 tx=0x7fc464006e40 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:21:16.464 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:16.465+0000 7fc474c00700 1 -- 172.21.15.189:0/1820978294 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fc46800fe60 con 0x7fc478100090 2024-04-20T21:21:16.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:16 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:21:16.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:16 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:21:16.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:16 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:21:16.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:16 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:21:16.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:16 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:21:16.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:16 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi189", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-20T21:21:16.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:16 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi189", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]': finished 2024-04-20T21:21:16.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:16 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:21:16.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:16 smithi062 ceph-mon[16221]: Deploying daemon ceph-exporter.smithi189 on smithi189 2024-04-20T21:21:16.770 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:16.770+0000 7fc47ea00700 1 -- 172.21.15.189:0/1820978294 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc47819c010 con 0x7fc478100090 2024-04-20T21:21:16.771 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:16.771+0000 7fc474c00700 1 -- 172.21.15.189:0/1820978294 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7fc468027030 con 0x7fc478100090 2024-04-20T21:21:16.771 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-20T21:21:16.771 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":1,"fsid":"61ae3628-ff5b-11ee-bc93-c7b262605968","modified":"2024-04-20T21:18:14.797594Z","created":"2024-04-20T21:18:14.797594Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-20T21:21:16.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:16.774+0000 7fc47ea00700 1 -- 172.21.15.189:0/1820978294 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fc45c038530 msgr2=0x7fc45c03a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:21:16.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:16.774+0000 7fc47ea00700 1 --2- 172.21.15.189:0/1820978294 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fc45c038530 0x7fc45c03a9e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc464006fd0 tx=0x7fc464006e40 comp rx=0 tx=0).stop 2024-04-20T21:21:16.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:16.774+0000 7fc47ea00700 1 -- 172.21.15.189:0/1820978294 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc478100090 msgr2=0x7fc478197520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:21:16.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:16.774+0000 7fc47ea00700 1 --2- 172.21.15.189:0/1820978294 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc478100090 0x7fc478197520 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fc4680092e0 tx=0x7fc468004c80 comp rx=0 tx=0).stop 2024-04-20T21:21:16.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:16.774+0000 7fc47ea00700 1 -- 172.21.15.189:0/1820978294 shutdown_connections 2024-04-20T21:21:16.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:16.774+0000 7fc47ea00700 1 --2- 172.21.15.189:0/1820978294 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fc45c038530 0x7fc45c03a9e0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:21:16.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:16.774+0000 7fc47ea00700 1 --2- 172.21.15.189:0/1820978294 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc478100090 0x7fc478197520 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:21:16.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:16.774+0000 7fc47ea00700 1 -- 172.21.15.189:0/1820978294 >> 172.21.15.189:0/1820978294 conn(0x7fc4780fb970 msgr2=0x7fc478102260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:21:16.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:16.775+0000 7fc47ea00700 1 -- 172.21.15.189:0/1820978294 shutdown_connections 2024-04-20T21:21:16.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:16.775+0000 7fc47ea00700 1 -- 172.21.15.189:0/1820978294 wait complete. 2024-04-20T21:21:16.776 INFO:teuthology.orchestra.run.smithi189.stderr:dumped monmap epoch 1 2024-04-20T21:21:18.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:17 smithi062 ceph-mon[16221]: from='client.? 172.21.15.189:0/1820978294' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-20T21:21:19.269 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-20T21:21:19.269 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-20T21:21:19.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:18 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:21:19.658 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-20T21:21:21.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:21 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:21:21.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:21 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:21:21.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:21 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:21:21.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:21 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:21:21.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:21 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi189", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-20T21:21:21.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:21 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi189", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-04-20T21:21:21.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:21 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:21:21.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:21 smithi062 ceph-mon[16221]: Deploying daemon crash.smithi189 on smithi189 2024-04-20T21:21:21.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:21.965+0000 7f3d81000700 1 -- 172.21.15.189:0/3146902481 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3d7c100f30 msgr2=0x7f3d7c101300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:21:21.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:21.966+0000 7f3d81000700 1 --2- 172.21.15.189:0/3146902481 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3d7c100f30 0x7f3d7c101300 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f3d6c009910 tx=0x7f3d6c009c20 comp rx=0 tx=0).stop 2024-04-20T21:21:21.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:21.966+0000 7f3d81000700 1 -- 172.21.15.189:0/3146902481 shutdown_connections 2024-04-20T21:21:21.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:21.966+0000 7f3d81000700 1 --2- 172.21.15.189:0/3146902481 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3d7c100f30 0x7f3d7c101300 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:21:21.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:21.966+0000 7f3d81000700 1 -- 172.21.15.189:0/3146902481 >> 172.21.15.189:0/3146902481 conn(0x7f3d7c074af0 msgr2=0x7f3d7c074ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:21:21.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:21.966+0000 7f3d81000700 1 -- 172.21.15.189:0/3146902481 shutdown_connections 2024-04-20T21:21:21.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:21.966+0000 7f3d81000700 1 -- 172.21.15.189:0/3146902481 wait complete. 2024-04-20T21:21:21.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:21.967+0000 7f3d81000700 1 Processor -- start 2024-04-20T21:21:21.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:21.967+0000 7f3d81000700 1 -- start start 2024-04-20T21:21:21.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:21.968+0000 7f3d81000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3d7c100f30 0x7f3d7c195390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:21:21.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:21.968+0000 7f3d81000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d7c1958d0 con 0x7f3d7c100f30 2024-04-20T21:21:21.968 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:21.968+0000 7f3d79e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3d7c100f30 0x7f3d7c195390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:21:21.968 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:21.968+0000 7f3d79e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3d7c100f30 0x7f3d7c195390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.189:44256/0 (socket says 172.21.15.189:44256) 2024-04-20T21:21:21.968 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:21.968+0000 7f3d79e00700 1 -- 172.21.15.189:0/1681731334 learned_addr learned my addr 172.21.15.189:0/1681731334 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-20T21:21:21.968 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:21.969+0000 7f3d79e00700 1 -- 172.21.15.189:0/1681731334 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d6c0095f0 con 0x7f3d7c100f30 2024-04-20T21:21:21.968 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:21.969+0000 7f3d79e00700 1 --2- 172.21.15.189:0/1681731334 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3d7c100f30 0x7f3d7c195390 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f3d6c0092e0 tx=0x7f3d6c004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:21:21.969 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:21.970+0000 7f3d72a00700 1 -- 172.21.15.189:0/1681731334 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3d6c01d070 con 0x7f3d7c100f30 2024-04-20T21:21:21.969 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:21.970+0000 7f3d81000700 1 -- 172.21.15.189:0/1681731334 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d7c195ad0 con 0x7f3d7c100f30 2024-04-20T21:21:21.969 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:21.970+0000 7f3d72a00700 1 -- 172.21.15.189:0/1681731334 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f3d6c00be10 con 0x7f3d7c100f30 2024-04-20T21:21:21.970 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:21.970+0000 7f3d72a00700 1 -- 172.21.15.189:0/1681731334 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3d6c022e10 con 0x7f3d7c100f30 2024-04-20T21:21:21.970 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:21.970+0000 7f3d81000700 1 -- 172.21.15.189:0/1681731334 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d7c195ef0 con 0x7f3d7c100f30 2024-04-20T21:21:21.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:21.971+0000 7f3d81000700 1 -- 172.21.15.189:0/1681731334 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d7c062380 con 0x7f3d7c100f30 2024-04-20T21:21:21.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:21.972+0000 7f3d72a00700 1 -- 172.21.15.189:0/1681731334 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 45197+0+0 (secure 0 0 0) 0x7f3d6c022470 con 0x7f3d7c100f30 2024-04-20T21:21:21.973 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:21.972+0000 7f3d72a00700 1 --2- 172.21.15.189:0/1681731334 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f3d60038530 0x7f3d6003a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:21:21.974 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:21.972+0000 7f3d72a00700 1 -- 172.21.15.189:0/1681731334 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f3d6c04c520 con 0x7f3d7c100f30 2024-04-20T21:21:21.974 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:21.973+0000 7f3d79400700 1 --2- 172.21.15.189:0/1681731334 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f3d60038530 0x7f3d6003a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:21:21.974 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:21.974+0000 7f3d79400700 1 --2- 172.21.15.189:0/1681731334 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f3d60038530 0x7f3d6003a9e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f3d68006fd0 tx=0x7f3d68006e40 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:21:21.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:21.987+0000 7f3d72a00700 1 -- 172.21.15.189:0/1681731334 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f3d6c02a540 con 0x7f3d7c100f30 2024-04-20T21:21:22.304 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:22.305+0000 7f3d81000700 1 -- 172.21.15.189:0/1681731334 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3d7c107130 con 0x7f3d7c100f30 2024-04-20T21:21:22.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:22.306+0000 7f3d72a00700 1 -- 172.21.15.189:0/1681731334 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f3d6c027030 con 0x7f3d7c100f30 2024-04-20T21:21:22.306 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-20T21:21:22.306 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":1,"fsid":"61ae3628-ff5b-11ee-bc93-c7b262605968","modified":"2024-04-20T21:18:14.797594Z","created":"2024-04-20T21:18:14.797594Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-20T21:21:22.309 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:22.309+0000 7f3d81000700 1 -- 172.21.15.189:0/1681731334 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f3d60038530 msgr2=0x7f3d6003a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:21:22.309 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:22.309+0000 7f3d81000700 1 --2- 172.21.15.189:0/1681731334 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f3d60038530 0x7f3d6003a9e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f3d68006fd0 tx=0x7f3d68006e40 comp rx=0 tx=0).stop 2024-04-20T21:21:22.309 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:22.309+0000 7f3d81000700 1 -- 172.21.15.189:0/1681731334 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3d7c100f30 msgr2=0x7f3d7c195390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:21:22.309 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:22.309+0000 7f3d81000700 1 --2- 172.21.15.189:0/1681731334 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3d7c100f30 0x7f3d7c195390 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f3d6c0092e0 tx=0x7f3d6c004c80 comp rx=0 tx=0).stop 2024-04-20T21:21:22.309 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:22.309+0000 7f3d81000700 1 -- 172.21.15.189:0/1681731334 shutdown_connections 2024-04-20T21:21:22.309 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:22.309+0000 7f3d81000700 1 --2- 172.21.15.189:0/1681731334 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f3d60038530 0x7f3d6003a9e0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:21:22.309 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:22.309+0000 7f3d81000700 1 --2- 172.21.15.189:0/1681731334 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3d7c100f30 0x7f3d7c195390 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:21:22.309 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:22.309+0000 7f3d81000700 1 -- 172.21.15.189:0/1681731334 >> 172.21.15.189:0/1681731334 conn(0x7f3d7c074af0 msgr2=0x7f3d7c104750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:21:22.309 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:22.309+0000 7f3d81000700 1 -- 172.21.15.189:0/1681731334 shutdown_connections 2024-04-20T21:21:22.309 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:22.309+0000 7f3d81000700 1 -- 172.21.15.189:0/1681731334 wait complete. 2024-04-20T21:21:22.310 INFO:teuthology.orchestra.run.smithi189.stderr:dumped monmap epoch 1 2024-04-20T21:21:23.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:22 smithi062 ceph-mon[16221]: from='client.? 172.21.15.189:0/1681731334' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-20T21:21:24.348 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-20T21:21:24.348 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-20T21:21:24.931 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-20T21:21:25.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:24 smithi062 ceph-mon[16221]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:21:25.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:24 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:21:26.187 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:25 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:21:26.187 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:25 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:21:26.187 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:25 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:21:26.187 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:25 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:21:26.188 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:25 smithi062 ceph-mon[16221]: Deploying daemon node-exporter.smithi189 on smithi189 2024-04-20T21:21:26.188 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:25 smithi062 ceph-mon[16221]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:21:27.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:27.108+0000 7f8988200700 1 -- 172.21.15.189:0/843243105 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8980100f30 msgr2=0x7f8980101300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:21:27.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:27.108+0000 7f8988200700 1 --2- 172.21.15.189:0/843243105 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8980100f30 0x7f8980101300 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f8974009910 tx=0x7f8974009c20 comp rx=0 tx=0).stop 2024-04-20T21:21:27.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:27.108+0000 7f8988200700 1 -- 172.21.15.189:0/843243105 shutdown_connections 2024-04-20T21:21:27.109 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:27.108+0000 7f8988200700 1 --2- 172.21.15.189:0/843243105 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8980100f30 0x7f8980101300 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:21:27.110 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:27.108+0000 7f8988200700 1 -- 172.21.15.189:0/843243105 >> 172.21.15.189:0/843243105 conn(0x7f8980074af0 msgr2=0x7f8980074ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:21:27.110 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:27.108+0000 7f8988200700 1 -- 172.21.15.189:0/843243105 shutdown_connections 2024-04-20T21:21:27.110 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:27.108+0000 7f8988200700 1 -- 172.21.15.189:0/843243105 wait complete. 2024-04-20T21:21:27.110 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:27.110+0000 7f8988200700 1 Processor -- start 2024-04-20T21:21:27.110 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:27.110+0000 7f8988200700 1 -- start start 2024-04-20T21:21:27.110 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:27.111+0000 7f8988200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8980100f30 0x7f89801a1790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:21:27.111 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:27.111+0000 7f8988200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89801a1cd0 con 0x7f8980100f30 2024-04-20T21:21:27.111 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:27.111+0000 7f8985800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8980100f30 0x7f89801a1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:21:27.111 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:27.112+0000 7f8985800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8980100f30 0x7f89801a1790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.189:44288/0 (socket says 172.21.15.189:44288) 2024-04-20T21:21:27.111 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:27.112+0000 7f8985800700 1 -- 172.21.15.189:0/3239765353 learned_addr learned my addr 172.21.15.189:0/3239765353 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-20T21:21:27.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:27.112+0000 7f8985800700 1 -- 172.21.15.189:0/3239765353 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89740095f0 con 0x7f8980100f30 2024-04-20T21:21:27.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:27.113+0000 7f8985800700 1 --2- 172.21.15.189:0/3239765353 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8980100f30 0x7f89801a1790 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f8974005b40 tx=0x7f8974004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:21:27.112 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:27.113+0000 7f897e000700 1 -- 172.21.15.189:0/3239765353 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f897401d070 con 0x7f8980100f30 2024-04-20T21:21:27.113 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:27.113+0000 7f8988200700 1 -- 172.21.15.189:0/3239765353 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89801a1ed0 con 0x7f8980100f30 2024-04-20T21:21:27.113 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:27.113+0000 7f897e000700 1 -- 172.21.15.189:0/3239765353 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f89740044d0 con 0x7f8980100f30 2024-04-20T21:21:27.113 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:27.113+0000 7f897e000700 1 -- 172.21.15.189:0/3239765353 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8974003930 con 0x7f8980100f30 2024-04-20T21:21:27.113 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:27.113+0000 7f8988200700 1 -- 172.21.15.189:0/3239765353 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89801a2290 con 0x7f8980100f30 2024-04-20T21:21:27.114 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:27.115+0000 7f8988200700 1 -- 172.21.15.189:0/3239765353 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8980062380 con 0x7f8980100f30 2024-04-20T21:21:27.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:27.115+0000 7f897e000700 1 -- 172.21.15.189:0/3239765353 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 45197+0+0 (secure 0 0 0) 0x7f897400b960 con 0x7f8980100f30 2024-04-20T21:21:27.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:27.115+0000 7f897e000700 1 --2- 172.21.15.189:0/3239765353 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f89680385d0 0x7f896803aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:21:27.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:27.116+0000 7f897e000700 1 -- 172.21.15.189:0/3239765353 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f897404c370 con 0x7f8980100f30 2024-04-20T21:21:27.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:27.116+0000 7f8984e00700 1 --2- 172.21.15.189:0/3239765353 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f89680385d0 0x7f896803aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:21:27.117 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:27.117+0000 7f8984e00700 1 --2- 172.21.15.189:0/3239765353 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f89680385d0 0x7f896803aa80 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f896c006fd0 tx=0x7f896c006e40 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:21:27.128 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:27.129+0000 7f897e000700 1 -- 172.21.15.189:0/3239765353 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f897402a540 con 0x7f8980100f30 2024-04-20T21:21:27.456 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:27.457+0000 7f8988200700 1 -- 172.21.15.189:0/3239765353 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f898019cd20 con 0x7f8980100f30 2024-04-20T21:21:27.457 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:27.458+0000 7f897e000700 1 -- 172.21.15.189:0/3239765353 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f897402a360 con 0x7f8980100f30 2024-04-20T21:21:27.458 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-20T21:21:27.458 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":1,"fsid":"61ae3628-ff5b-11ee-bc93-c7b262605968","modified":"2024-04-20T21:18:14.797594Z","created":"2024-04-20T21:18:14.797594Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-20T21:21:27.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:27.461+0000 7f8988200700 1 -- 172.21.15.189:0/3239765353 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f89680385d0 msgr2=0x7f896803aa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:21:27.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:27.461+0000 7f8988200700 1 --2- 172.21.15.189:0/3239765353 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f89680385d0 0x7f896803aa80 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f896c006fd0 tx=0x7f896c006e40 comp rx=0 tx=0).stop 2024-04-20T21:21:27.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:27.461+0000 7f8988200700 1 -- 172.21.15.189:0/3239765353 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8980100f30 msgr2=0x7f89801a1790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:21:27.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:27.461+0000 7f8988200700 1 --2- 172.21.15.189:0/3239765353 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8980100f30 0x7f89801a1790 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f8974005b40 tx=0x7f8974004c80 comp rx=0 tx=0).stop 2024-04-20T21:21:27.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:27.461+0000 7f8988200700 1 -- 172.21.15.189:0/3239765353 shutdown_connections 2024-04-20T21:21:27.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:27.461+0000 7f8988200700 1 --2- 172.21.15.189:0/3239765353 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f89680385d0 0x7f896803aa80 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:21:27.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:27.461+0000 7f8988200700 1 --2- 172.21.15.189:0/3239765353 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8980100f30 0x7f89801a1790 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:21:27.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:27.461+0000 7f8988200700 1 -- 172.21.15.189:0/3239765353 >> 172.21.15.189:0/3239765353 conn(0x7f8980074af0 msgr2=0x7f8980102740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:21:27.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:27.461+0000 7f8988200700 1 -- 172.21.15.189:0/3239765353 shutdown_connections 2024-04-20T21:21:27.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:27.461+0000 7f8988200700 1 -- 172.21.15.189:0/3239765353 wait complete. 2024-04-20T21:21:27.462 INFO:teuthology.orchestra.run.smithi189.stderr:dumped monmap epoch 1 2024-04-20T21:21:27.910 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:27 smithi062 ceph-mon[16221]: from='client.? 172.21.15.189:0/3239765353' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-20T21:21:29.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:28 smithi062 ceph-mon[16221]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:21:29.315 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-20T21:21:29.316 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-20T21:21:29.795 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-20T21:21:30.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:29 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:21:30.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:29 smithi062 ceph-mon[16221]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:21:32.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:32.405+0000 7f7428800700 1 -- 172.21.15.189:0/400253675 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7420100090 msgr2=0x7f7420100460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:21:32.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:32.405+0000 7f7428800700 1 --2- 172.21.15.189:0/400253675 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7420100090 0x7f7420100460 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f7414009910 tx=0x7f7414009c20 comp rx=0 tx=0).stop 2024-04-20T21:21:32.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:32.405+0000 7f7428800700 1 -- 172.21.15.189:0/400253675 shutdown_connections 2024-04-20T21:21:32.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:32.405+0000 7f7428800700 1 --2- 172.21.15.189:0/400253675 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7420100090 0x7f7420100460 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:21:32.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:32.405+0000 7f7428800700 1 -- 172.21.15.189:0/400253675 >> 172.21.15.189:0/400253675 conn(0x7f74200fb970 msgr2=0x7f74200fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:21:32.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:32.405+0000 7f7428800700 1 -- 172.21.15.189:0/400253675 shutdown_connections 2024-04-20T21:21:32.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:32.406+0000 7f7428800700 1 -- 172.21.15.189:0/400253675 wait complete. 2024-04-20T21:21:32.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:32.406+0000 7f7428800700 1 Processor -- start 2024-04-20T21:21:32.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:32.407+0000 7f7428800700 1 -- start start 2024-04-20T21:21:32.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:32.407+0000 7f7428800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7420100090 0x7f7420197590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:21:32.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:32.407+0000 7f7428800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7420197ad0 con 0x7f7420100090 2024-04-20T21:21:32.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:32.408+0000 7f7425e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7420100090 0x7f7420197590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:21:32.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:32.408+0000 7f7425e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7420100090 0x7f7420197590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.189:38482/0 (socket says 172.21.15.189:38482) 2024-04-20T21:21:32.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:32.408+0000 7f7425e00700 1 -- 172.21.15.189:0/4077884798 learned_addr learned my addr 172.21.15.189:0/4077884798 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-20T21:21:32.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:32.408+0000 7f7425e00700 1 -- 172.21.15.189:0/4077884798 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74140095f0 con 0x7f7420100090 2024-04-20T21:21:32.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:32.409+0000 7f7425e00700 1 --2- 172.21.15.189:0/4077884798 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7420100090 0x7f7420197590 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f7414005b40 tx=0x7f7414004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:21:32.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:32.409+0000 7f741aa00700 1 -- 172.21.15.189:0/4077884798 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f741401d070 con 0x7f7420100090 2024-04-20T21:21:32.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:32.409+0000 7f7428800700 1 -- 172.21.15.189:0/4077884798 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7420197cd0 con 0x7f7420100090 2024-04-20T21:21:32.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:32.409+0000 7f741aa00700 1 -- 172.21.15.189:0/4077884798 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f74140044d0 con 0x7f7420100090 2024-04-20T21:21:32.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:32.409+0000 7f741aa00700 1 -- 172.21.15.189:0/4077884798 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7414003930 con 0x7f7420100090 2024-04-20T21:21:32.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:32.409+0000 7f7428800700 1 -- 172.21.15.189:0/4077884798 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7420198090 con 0x7f7420100090 2024-04-20T21:21:32.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:32.410+0000 7f7428800700 1 -- 172.21.15.189:0/4077884798 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7420062380 con 0x7f7420100090 2024-04-20T21:21:32.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:32.411+0000 7f741aa00700 1 -- 172.21.15.189:0/4077884798 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 45197+0+0 (secure 0 0 0) 0x7f741400b960 con 0x7f7420100090 2024-04-20T21:21:32.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:32.411+0000 7f741aa00700 1 --2- 172.21.15.189:0/4077884798 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f7408038580 0x7f740803aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:21:32.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:32.411+0000 7f741aa00700 1 -- 172.21.15.189:0/4077884798 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f741404c4b0 con 0x7f7420100090 2024-04-20T21:21:32.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:32.412+0000 7f7425400700 1 --2- 172.21.15.189:0/4077884798 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f7408038580 0x7f740803aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:21:32.412 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:32.414+0000 7f7425400700 1 --2- 172.21.15.189:0/4077884798 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f7408038580 0x7f740803aa30 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f7410006fd0 tx=0x7f7410006e40 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:21:32.426 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:32.428+0000 7f741aa00700 1 -- 172.21.15.189:0/4077884798 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f741402a540 con 0x7f7420100090 2024-04-20T21:21:32.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:32.737+0000 7f7428800700 1 -- 172.21.15.189:0/4077884798 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f742019c010 con 0x7f7420100090 2024-04-20T21:21:32.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:32.738+0000 7f741aa00700 1 -- 172.21.15.189:0/4077884798 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f741402a360 con 0x7f7420100090 2024-04-20T21:21:32.736 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-20T21:21:32.737 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":1,"fsid":"61ae3628-ff5b-11ee-bc93-c7b262605968","modified":"2024-04-20T21:18:14.797594Z","created":"2024-04-20T21:18:14.797594Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-20T21:21:32.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:32.741+0000 7f7428800700 1 -- 172.21.15.189:0/4077884798 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f7408038580 msgr2=0x7f740803aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:21:32.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:32.741+0000 7f7428800700 1 --2- 172.21.15.189:0/4077884798 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f7408038580 0x7f740803aa30 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f7410006fd0 tx=0x7f7410006e40 comp rx=0 tx=0).stop 2024-04-20T21:21:32.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:32.741+0000 7f7428800700 1 -- 172.21.15.189:0/4077884798 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7420100090 msgr2=0x7f7420197590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:21:32.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:32.741+0000 7f7428800700 1 --2- 172.21.15.189:0/4077884798 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7420100090 0x7f7420197590 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f7414005b40 tx=0x7f7414004c80 comp rx=0 tx=0).stop 2024-04-20T21:21:32.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:32.741+0000 7f7428800700 1 -- 172.21.15.189:0/4077884798 shutdown_connections 2024-04-20T21:21:32.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:32.741+0000 7f7428800700 1 --2- 172.21.15.189:0/4077884798 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f7408038580 0x7f740803aa30 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:21:32.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:32.741+0000 7f7428800700 1 --2- 172.21.15.189:0/4077884798 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7420100090 0x7f7420197590 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:21:32.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:32.741+0000 7f7428800700 1 -- 172.21.15.189:0/4077884798 >> 172.21.15.189:0/4077884798 conn(0x7f74200fb970 msgr2=0x7f74201039e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:21:32.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:32.741+0000 7f7428800700 1 -- 172.21.15.189:0/4077884798 shutdown_connections 2024-04-20T21:21:32.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:32.741+0000 7f7428800700 1 -- 172.21.15.189:0/4077884798 wait complete. 2024-04-20T21:21:32.740 INFO:teuthology.orchestra.run.smithi189.stderr:dumped monmap epoch 1 2024-04-20T21:21:32.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:32 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:21:32.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:32 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:21:32.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:32 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:21:32.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:32 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:21:32.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:32 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi189.tawavz", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-20T21:21:32.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:32 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi189.tawavz", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-20T21:21:32.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:32 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-20T21:21:32.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:32 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:21:32.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:32 smithi062 ceph-mon[16221]: Deploying daemon mgr.smithi189.tawavz on smithi189 2024-04-20T21:21:32.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:32 smithi062 ceph-mon[16221]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:21:33.969 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:33 smithi062 ceph-mon[16221]: from='client.? 172.21.15.189:0/4077884798' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-20T21:21:35.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:34 smithi062 ceph-mon[16221]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:21:35.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:34 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:21:35.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:34 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:21:35.294 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-20T21:21:35.294 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-20T21:21:35.715 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-20T21:21:36.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:36 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:21:36.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:36 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:21:36.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:36 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:21:36.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:36 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:21:36.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:36 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-20T21:21:36.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:36 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:21:36.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:36 smithi062 ceph-mon[16221]: Deploying daemon mon.smithi189 on smithi189 2024-04-20T21:21:36.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:36 smithi062 ceph-mon[16221]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:21:38.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:38.368+0000 7fcad3200700 1 -- 172.21.15.189:0/2467833189 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcad40fe3d0 msgr2=0x7fcad40fe7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:21:38.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:38.368+0000 7fcad3200700 1 --2- 172.21.15.189:0/2467833189 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcad40fe3d0 0x7fcad40fe7a0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fcac4009910 tx=0x7fcac4009c20 comp rx=0 tx=0).stop 2024-04-20T21:21:38.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:38.368+0000 7fcad3200700 1 -- 172.21.15.189:0/2467833189 shutdown_connections 2024-04-20T21:21:38.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:38.368+0000 7fcad3200700 1 --2- 172.21.15.189:0/2467833189 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcad40fe3d0 0x7fcad40fe7a0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:21:38.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:38.368+0000 7fcad3200700 1 -- 172.21.15.189:0/2467833189 >> 172.21.15.189:0/2467833189 conn(0x7fcad40f9e20 msgr2=0x7fcad40fc230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:21:38.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:38.368+0000 7fcad3200700 1 -- 172.21.15.189:0/2467833189 shutdown_connections 2024-04-20T21:21:38.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:38.368+0000 7fcad3200700 1 -- 172.21.15.189:0/2467833189 wait complete. 2024-04-20T21:21:38.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:38.369+0000 7fcad3200700 1 Processor -- start 2024-04-20T21:21:38.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:38.369+0000 7fcad3200700 1 -- start start 2024-04-20T21:21:38.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:38.370+0000 7fcad3200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcad40fe3d0 0x7fcad4197390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:21:38.369 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:38.370+0000 7fcad3200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcad41978d0 con 0x7fcad40fe3d0 2024-04-20T21:21:38.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:38.370+0000 7fcad1e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcad40fe3d0 0x7fcad4197390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:21:38.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:38.370+0000 7fcad1e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcad40fe3d0 0x7fcad4197390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.189:47916/0 (socket says 172.21.15.189:47916) 2024-04-20T21:21:38.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:38.370+0000 7fcad1e00700 1 -- 172.21.15.189:0/3358366996 learned_addr learned my addr 172.21.15.189:0/3358366996 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-20T21:21:38.370 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:38.371+0000 7fcad1e00700 1 -- 172.21.15.189:0/3358366996 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcac40095f0 con 0x7fcad40fe3d0 2024-04-20T21:21:38.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:38.371+0000 7fcad1e00700 1 --2- 172.21.15.189:0/3358366996 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcad40fe3d0 0x7fcad4197390 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fcac40092e0 tx=0x7fcac4004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:21:38.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:38.372+0000 7fcacaa00700 1 -- 172.21.15.189:0/3358366996 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcac401d070 con 0x7fcad40fe3d0 2024-04-20T21:21:38.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:38.372+0000 7fcad3200700 1 -- 172.21.15.189:0/3358366996 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcad4197ad0 con 0x7fcad40fe3d0 2024-04-20T21:21:38.371 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:38.372+0000 7fcacaa00700 1 -- 172.21.15.189:0/3358366996 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fcac400be10 con 0x7fcad40fe3d0 2024-04-20T21:21:38.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:38.372+0000 7fcacaa00700 1 -- 172.21.15.189:0/3358366996 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcac4022e10 con 0x7fcad40fe3d0 2024-04-20T21:21:38.372 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:38.372+0000 7fcad3200700 1 -- 172.21.15.189:0/3358366996 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcad4197ef0 con 0x7fcad40fe3d0 2024-04-20T21:21:38.373 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:38.373+0000 7fcad3200700 1 -- 172.21.15.189:0/3358366996 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcad4062380 con 0x7fcad40fe3d0 2024-04-20T21:21:38.373 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:38.374+0000 7fcacaa00700 1 -- 172.21.15.189:0/3358366996 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 45197+0+0 (secure 0 0 0) 0x7fcac4022470 con 0x7fcad40fe3d0 2024-04-20T21:21:38.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:38.374+0000 7fcacaa00700 1 --2- 172.21.15.189:0/3358366996 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fcab8038530 0x7fcab803a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:21:38.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:38.374+0000 7fcacaa00700 1 -- 172.21.15.189:0/3358366996 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7fcac404c540 con 0x7fcad40fe3d0 2024-04-20T21:21:38.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:38.375+0000 7fcad1400700 1 --2- 172.21.15.189:0/3358366996 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fcab8038530 0x7fcab803a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:21:38.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:38.376+0000 7fcad1400700 1 --2- 172.21.15.189:0/3358366996 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fcab8038530 0x7fcab803a9e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fcac0006fd0 tx=0x7fcac0006e40 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:21:38.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:38.390+0000 7fcacaa00700 1 -- 172.21.15.189:0/3358366996 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fcac402a540 con 0x7fcad40fe3d0 2024-04-20T21:21:38.692 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:38.692+0000 7fcad3200700 1 -- 172.21.15.189:0/3358366996 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcad419be80 con 0x7fcad40fe3d0 2024-04-20T21:21:38.693 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:38.693+0000 7fcacaa00700 1 -- 172.21.15.189:0/3358366996 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7fcac4027030 con 0x7fcad40fe3d0 2024-04-20T21:21:38.693 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-20T21:21:38.693 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":1,"fsid":"61ae3628-ff5b-11ee-bc93-c7b262605968","modified":"2024-04-20T21:18:14.797594Z","created":"2024-04-20T21:18:14.797594Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-20T21:21:38.697 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:38.697+0000 7fcad3200700 1 -- 172.21.15.189:0/3358366996 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fcab8038530 msgr2=0x7fcab803a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:21:38.697 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:38.697+0000 7fcad3200700 1 --2- 172.21.15.189:0/3358366996 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fcab8038530 0x7fcab803a9e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fcac0006fd0 tx=0x7fcac0006e40 comp rx=0 tx=0).stop 2024-04-20T21:21:38.697 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:38.697+0000 7fcad3200700 1 -- 172.21.15.189:0/3358366996 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcad40fe3d0 msgr2=0x7fcad4197390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:21:38.697 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:38.697+0000 7fcad3200700 1 --2- 172.21.15.189:0/3358366996 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcad40fe3d0 0x7fcad4197390 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fcac40092e0 tx=0x7fcac4004c80 comp rx=0 tx=0).stop 2024-04-20T21:21:38.697 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:38.697+0000 7fcad3200700 1 -- 172.21.15.189:0/3358366996 shutdown_connections 2024-04-20T21:21:38.697 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:38.697+0000 7fcad3200700 1 --2- 172.21.15.189:0/3358366996 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fcab8038530 0x7fcab803a9e0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:21:38.697 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:38.697+0000 7fcad3200700 1 --2- 172.21.15.189:0/3358366996 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcad40fe3d0 0x7fcad4197390 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:21:38.697 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:38.697+0000 7fcad3200700 1 -- 172.21.15.189:0/3358366996 >> 172.21.15.189:0/3358366996 conn(0x7fcad40f9e20 msgr2=0x7fcad4103830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:21:38.697 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:38.697+0000 7fcad3200700 1 -- 172.21.15.189:0/3358366996 shutdown_connections 2024-04-20T21:21:38.697 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:38.697+0000 7fcad3200700 1 -- 172.21.15.189:0/3358366996 wait complete. 2024-04-20T21:21:38.698 INFO:teuthology.orchestra.run.smithi189.stderr:dumped monmap epoch 1 2024-04-20T21:21:39.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:38 smithi062 ceph-mon[16221]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:21:39.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:38 smithi062 ceph-mon[16221]: from='client.? 172.21.15.189:0/3358366996' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-20T21:21:40.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:39 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:21:40.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:39 smithi062 ceph-mon[16221]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:21:40.491 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-20T21:21:40.491 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-20T21:21:40.892 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi189/config 2024-04-20T21:21:43.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:42 smithi062 ceph-mon[16221]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:21:43.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:21:43 smithi189 systemd[1]: Started Ceph mon.smithi189 for 61ae3628-ff5b-11ee-bc93-c7b262605968. 2024-04-20T21:21:48.537 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:48 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mon metadata", "id": "smithi062"}]: dispatch 2024-04-20T21:21:48.537 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:48 smithi062 ceph-mon[16221]: mon.smithi062 calling monitor election 2024-04-20T21:21:48.537 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:48 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mon metadata", "id": "smithi189"}]: dispatch 2024-04-20T21:21:48.537 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:48 smithi062 ceph-mon[16221]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:21:48.537 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:48 smithi062 ceph-mon[16221]: from='mgr.? 172.21.15.189:0/3076593902' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi189.tawavz/crt"}]: dispatch 2024-04-20T21:21:48.537 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:48 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mon metadata", "id": "smithi189"}]: dispatch 2024-04-20T21:21:48.537 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:48 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mon metadata", "id": "smithi189"}]: dispatch 2024-04-20T21:21:48.537 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:48 smithi062 ceph-mon[16221]: mon.smithi189 calling monitor election 2024-04-20T21:21:48.537 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:48 smithi062 ceph-mon[16221]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:21:48.537 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:48 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mon metadata", "id": "smithi189"}]: dispatch 2024-04-20T21:21:48.537 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:48 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mon metadata", "id": "smithi189"}]: dispatch 2024-04-20T21:21:48.537 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:48 smithi062 ceph-mon[16221]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:21:48.538 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:48 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mon metadata", "id": "smithi189"}]: dispatch 2024-04-20T21:21:48.538 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:48 smithi062 ceph-mon[16221]: mon.smithi062 is new leader, mons smithi062,smithi189 in quorum (ranks 0,1) 2024-04-20T21:21:48.538 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:48 smithi062 ceph-mon[16221]: monmap e2: 2 mons at {smithi062=[v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0],smithi189=[v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0]} removed_ranks: {} 2024-04-20T21:21:48.538 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:48 smithi062 ceph-mon[16221]: fsmap 2024-04-20T21:21:48.538 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:48 smithi062 ceph-mon[16221]: osdmap e5: 0 total, 0 up, 0 in 2024-04-20T21:21:48.538 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:48 smithi062 ceph-mon[16221]: mgrmap e17: smithi062.rqxuxi(active, since 44s) 2024-04-20T21:21:48.538 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:48 smithi062 ceph-mon[16221]: Standby manager daemon smithi189.tawavz started 2024-04-20T21:21:48.538 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:48 smithi062 ceph-mon[16221]: from='mgr.? 172.21.15.189:0/3076593902' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-20T21:21:48.538 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:48 smithi062 ceph-mon[16221]: overall HEALTH_OK 2024-04-20T21:21:48.538 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:48 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:21:48.538 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:48 smithi062 ceph-mon[16221]: from='mgr.? 172.21.15.189:0/3076593902' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi189.tawavz/key"}]: dispatch 2024-04-20T21:21:48.538 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:48 smithi062 ceph-mon[16221]: from='mgr.? 172.21.15.189:0/3076593902' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-20T21:21:48.538 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:48 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:21:48.539 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:48 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:21:48.539 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:48 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:21:48.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:21:48 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mon metadata", "id": "smithi062"}]: dispatch 2024-04-20T21:21:48.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:21:48 smithi189 ceph-mon[21562]: mon.smithi062 calling monitor election 2024-04-20T21:21:48.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:21:48 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mon metadata", "id": "smithi189"}]: dispatch 2024-04-20T21:21:48.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:21:48 smithi189 ceph-mon[21562]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:21:48.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:21:48 smithi189 ceph-mon[21562]: from='mgr.? 172.21.15.189:0/3076593902' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi189.tawavz/crt"}]: dispatch 2024-04-20T21:21:48.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:21:48 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mon metadata", "id": "smithi189"}]: dispatch 2024-04-20T21:21:48.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:21:48 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mon metadata", "id": "smithi189"}]: dispatch 2024-04-20T21:21:48.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:21:48 smithi189 ceph-mon[21562]: mon.smithi189 calling monitor election 2024-04-20T21:21:48.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:21:48 smithi189 ceph-mon[21562]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:21:48.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:21:48 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mon metadata", "id": "smithi189"}]: dispatch 2024-04-20T21:21:48.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:21:48 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mon metadata", "id": "smithi189"}]: dispatch 2024-04-20T21:21:48.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:21:48 smithi189 ceph-mon[21562]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:21:48.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:21:48 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mon metadata", "id": "smithi189"}]: dispatch 2024-04-20T21:21:48.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:21:48 smithi189 ceph-mon[21562]: mon.smithi062 is new leader, mons smithi062,smithi189 in quorum (ranks 0,1) 2024-04-20T21:21:48.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:21:48 smithi189 ceph-mon[21562]: monmap e2: 2 mons at {smithi062=[v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0],smithi189=[v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0]} removed_ranks: {} 2024-04-20T21:21:48.739 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:21:48 smithi189 ceph-mon[21562]: fsmap 2024-04-20T21:21:48.739 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:21:48 smithi189 ceph-mon[21562]: osdmap e5: 0 total, 0 up, 0 in 2024-04-20T21:21:48.739 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:21:48 smithi189 ceph-mon[21562]: mgrmap e17: smithi062.rqxuxi(active, since 44s) 2024-04-20T21:21:48.739 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:21:48 smithi189 ceph-mon[21562]: Standby manager daemon smithi189.tawavz started 2024-04-20T21:21:48.739 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:21:48 smithi189 ceph-mon[21562]: from='mgr.? 172.21.15.189:0/3076593902' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-20T21:21:48.739 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:21:48 smithi189 ceph-mon[21562]: overall HEALTH_OK 2024-04-20T21:21:48.739 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:21:48 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:21:48.739 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:21:48 smithi189 ceph-mon[21562]: from='mgr.? 172.21.15.189:0/3076593902' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi189.tawavz/key"}]: dispatch 2024-04-20T21:21:48.739 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:21:48 smithi189 ceph-mon[21562]: from='mgr.? 172.21.15.189:0/3076593902' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-20T21:21:48.739 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:21:48 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:21:48.739 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:21:48 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:21:48.739 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:21:48 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:21:49.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:49.314+0000 7f9f38a00700 1 -- 172.21.15.189:0/3541718505 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9f18006230 msgr2=0x7f9f180039b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:21:49.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:49.314+0000 7f9f38a00700 1 --2- 172.21.15.189:0/3541718505 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9f18006230 0x7f9f180039b0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f9f34101030 tx=0x7f9f240095d0 comp rx=0 tx=0).stop 2024-04-20T21:21:49.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:49.314+0000 7f9f38a00700 1 -- 172.21.15.189:0/3541718505 shutdown_connections 2024-04-20T21:21:49.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:49.314+0000 7f9f38a00700 1 --2- 172.21.15.189:0/3541718505 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9f18006230 0x7f9f180039b0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:21:49.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:49.314+0000 7f9f38a00700 1 -- 172.21.15.189:0/3541718505 >> 172.21.15.189:0/3541718505 conn(0x7f9f340fba50 msgr2=0x7f9f340fde60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:21:49.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:49.314+0000 7f9f38a00700 1 -- 172.21.15.189:0/3541718505 shutdown_connections 2024-04-20T21:21:49.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:49.314+0000 7f9f38a00700 1 -- 172.21.15.189:0/3541718505 wait complete. 2024-04-20T21:21:49.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:49.317+0000 7f9f38a00700 1 Processor -- start 2024-04-20T21:21:49.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:49.317+0000 7f9f38a00700 1 -- start start 2024-04-20T21:21:49.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:49.317+0000 7f9f38a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9f18006230 0x7f9f341a2bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:21:49.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:49.317+0000 7f9f38a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f341a3110 0x7f9f3419e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:21:49.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:49.317+0000 7f9f38a00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f341a36d0 con 0x7f9f18006230 2024-04-20T21:21:49.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:49.317+0000 7f9f38a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f3419e9b0 con 0x7f9f341a3110 2024-04-20T21:21:49.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:49.318+0000 7f9f2be00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f341a3110 0x7f9f3419e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:21:49.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:49.318+0000 7f9f2be00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f341a3110 0x7f9f3419e470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:45176/0 (socket says 172.21.15.189:45176) 2024-04-20T21:21:49.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:49.318+0000 7f9f2be00700 1 -- 172.21.15.189:0/2254415068 learned_addr learned my addr 172.21.15.189:0/2254415068 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-20T21:21:49.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:49.318+0000 7f9f31e00700 1 --2- 172.21.15.189:0/2254415068 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9f18006230 0x7f9f341a2bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:21:49.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:49.319+0000 7f9f2be00700 1 -- 172.21.15.189:0/2254415068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f341a3110 msgr2=0x7f9f3419e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-20T21:21:49.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:49.319+0000 7f9f2be00700 1 -- 172.21.15.189:0/2254415068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f341a3110 msgr2=0x7f9f3419e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-20T21:21:49.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:49.319+0000 7f9f2be00700 1 --2- 172.21.15.189:0/2254415068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f341a3110 0x7f9f3419e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-20T21:21:49.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:49.319+0000 7f9f2be00700 1 --2- 172.21.15.189:0/2254415068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f341a3110 0x7f9f3419e470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-20T21:21:49.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:49.319+0000 7f9f31e00700 1 -- 172.21.15.189:0/2254415068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f341a3110 msgr2=0x7f9f3419e470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-20T21:21:49.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:49.319+0000 7f9f31e00700 1 --2- 172.21.15.189:0/2254415068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f341a3110 0x7f9f3419e470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:21:49.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:49.319+0000 7f9f31e00700 1 -- 172.21.15.189:0/2254415068 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f240098d0 con 0x7f9f18006230 2024-04-20T21:21:49.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:49.320+0000 7f9f31e00700 1 --2- 172.21.15.189:0/2254415068 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9f18006230 0x7f9f341a2bd0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f9f34101030 tx=0x7f9f24005b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:21:49.320 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:49.321+0000 7f9f2aa00700 1 -- 172.21.15.189:0/2254415068 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9f24004480 con 0x7f9f18006230 2024-04-20T21:21:49.320 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:49.321+0000 7f9f38a00700 1 -- 172.21.15.189:0/2254415068 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f3419ec30 con 0x7f9f18006230 2024-04-20T21:21:49.320 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:49.321+0000 7f9f2aa00700 1 -- 172.21.15.189:0/2254415068 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f9f2402d8f0 con 0x7f9f18006230 2024-04-20T21:21:49.320 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:49.321+0000 7f9f2aa00700 1 -- 172.21.15.189:0/2254415068 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9f2401aa70 con 0x7f9f18006230 2024-04-20T21:21:49.320 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:49.321+0000 7f9f38a00700 1 -- 172.21.15.189:0/2254415068 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f3419f120 con 0x7f9f18006230 2024-04-20T21:21:49.321 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:49.322+0000 7f9f38a00700 1 -- 172.21.15.189:0/2254415068 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f34062380 con 0x7f9f18006230 2024-04-20T21:21:49.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:49.324+0000 7f9f2aa00700 1 -- 172.21.15.189:0/2254415068 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) v1 ==== 89858+0+0 (secure 0 0 0) 0x7f9f2401abd0 con 0x7f9f18006230 2024-04-20T21:21:49.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:49.325+0000 7f9f2aa00700 1 --2- 172.21.15.189:0/2254415068 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f9f1006c750 0x7f9f1006ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:21:49.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:49.325+0000 7f9f2aa00700 1 -- 172.21.15.189:0/2254415068 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f9f2408d420 con 0x7f9f18006230 2024-04-20T21:21:49.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:49.326+0000 7f9f2be00700 1 --2- 172.21.15.189:0/2254415068 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f9f1006c750 0x7f9f1006ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:21:49.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:49.327+0000 7f9f2be00700 1 --2- 172.21.15.189:0/2254415068 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f9f1006c750 0x7f9f1006ec00 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f9f1c009180 tx=0x7f9f1c008040 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:21:49.340 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:49.340+0000 7f9f2aa00700 1 -- 172.21.15.189:0/2254415068 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f9f24058bc0 con 0x7f9f18006230 2024-04-20T21:21:49.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:21:49 smithi189 ceph-mon[21562]: mgrmap e18: smithi062.rqxuxi(active, since 44s), standbys: smithi189.tawavz 2024-04-20T21:21:49.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:21:49 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mgr metadata", "who": "smithi189.tawavz", "id": "smithi189.tawavz"}]: dispatch 2024-04-20T21:21:49.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:21:49 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:21:49.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:21:49 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:21:49.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:21:49 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:21:49.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:21:49 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:21:49.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:21:49 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mon metadata", "id": "smithi189"}]: dispatch 2024-04-20T21:21:49.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:49 smithi062 ceph-mon[16221]: mgrmap e18: smithi062.rqxuxi(active, since 44s), standbys: smithi189.tawavz 2024-04-20T21:21:49.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:49 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mgr metadata", "who": "smithi189.tawavz", "id": "smithi189.tawavz"}]: dispatch 2024-04-20T21:21:49.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:49 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:21:49.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:49 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:21:49.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:49 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:21:49.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:49 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:21:49.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:49 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mon metadata", "id": "smithi189"}]: dispatch 2024-04-20T21:21:49.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:49.672+0000 7f9f38a00700 1 -- 172.21.15.189:0/2254415068 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9f34072d20 con 0x7f9f18006230 2024-04-20T21:21:49.673 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:49.674+0000 7f9f2aa00700 1 -- 172.21.15.189:0/2254415068 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1022 (secure 0 0 0) 0x7f9f2405c1e0 con 0x7f9f18006230 2024-04-20T21:21:49.673 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-20T21:21:49.673 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":2,"fsid":"61ae3628-ff5b-11ee-bc93-c7b262605968","modified":"2024-04-20T21:21:43.183410Z","created":"2024-04-20T21:18:14.797594Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"smithi189","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:3300","nonce":0},{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-20T21:21:49.677 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:49.677+0000 7f9f38a00700 1 -- 172.21.15.189:0/2254415068 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f9f1006c750 msgr2=0x7f9f1006ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:21:49.677 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:49.677+0000 7f9f38a00700 1 --2- 172.21.15.189:0/2254415068 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f9f1006c750 0x7f9f1006ec00 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f9f1c009180 tx=0x7f9f1c008040 comp rx=0 tx=0).stop 2024-04-20T21:21:49.677 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:49.677+0000 7f9f38a00700 1 -- 172.21.15.189:0/2254415068 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9f18006230 msgr2=0x7f9f341a2bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:21:49.677 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:49.677+0000 7f9f38a00700 1 --2- 172.21.15.189:0/2254415068 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9f18006230 0x7f9f341a2bd0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f9f34101030 tx=0x7f9f24005b50 comp rx=0 tx=0).stop 2024-04-20T21:21:49.677 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:49.677+0000 7f9f38a00700 1 -- 172.21.15.189:0/2254415068 shutdown_connections 2024-04-20T21:21:49.677 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:49.677+0000 7f9f38a00700 1 --2- 172.21.15.189:0/2254415068 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f9f1006c750 0x7f9f1006ec00 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:21:49.678 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:49.677+0000 7f9f38a00700 1 --2- 172.21.15.189:0/2254415068 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9f18006230 0x7f9f341a2bd0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:21:49.678 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:49.678+0000 7f9f38a00700 1 --2- 172.21.15.189:0/2254415068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f341a3110 0x7f9f3419e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:21:49.678 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:49.678+0000 7f9f38a00700 1 -- 172.21.15.189:0/2254415068 >> 172.21.15.189:0/2254415068 conn(0x7f9f340fba50 msgr2=0x7f9f340fcfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:21:49.678 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:49.678+0000 7f9f38a00700 1 -- 172.21.15.189:0/2254415068 shutdown_connections 2024-04-20T21:21:49.678 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:21:49.678+0000 7f9f38a00700 1 -- 172.21.15.189:0/2254415068 wait complete. 2024-04-20T21:21:49.678 INFO:teuthology.orchestra.run.smithi189.stderr:dumped monmap epoch 2 2024-04-20T21:21:50.355 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:21:50 smithi189 ceph-mon[21562]: from='client.? 172.21.15.189:0/2254415068' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-20T21:21:50.356 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:21:50 smithi189 ceph-mon[21562]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:21:50.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:50 smithi062 ceph-mon[16221]: from='client.? 172.21.15.189:0/2254415068' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-20T21:21:50.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:50 smithi062 ceph-mon[16221]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:21:50.807 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-20T21:21:50.808 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -- ceph config generate-minimal-conf 2024-04-20T21:21:51.239 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T21:21:52.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:52 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:21:52.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:52 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:21:52.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:52 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:21:52.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:52 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T21:21:52.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:52 smithi062 ceph-mon[16221]: Updating smithi062:/etc/ceph/ceph.conf 2024-04-20T21:21:52.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:52 smithi062 ceph-mon[16221]: Updating smithi189:/etc/ceph/ceph.conf 2024-04-20T21:21:52.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:52 smithi062 ceph-mon[16221]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:21:52.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:21:52 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:21:52.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:21:52 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:21:52.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:21:52 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:21:52.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:21:52 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T21:21:52.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:21:52 smithi189 ceph-mon[21562]: Updating smithi062:/etc/ceph/ceph.conf 2024-04-20T21:21:52.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:21:52 smithi189 ceph-mon[21562]: Updating smithi189:/etc/ceph/ceph.conf 2024-04-20T21:21:52.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:21:52 smithi189 ceph-mon[21562]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:21:53.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:21:53.465+0000 7f6754800700 1 -- 172.21.15.62:0/2023232896 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f674c100090 msgr2=0x7f674c100460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:21:53.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:21:53.465+0000 7f6754800700 1 --2- 172.21.15.62:0/2023232896 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f674c100090 0x7f674c100460 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f6740009910 tx=0x7f6740009c20 comp rx=0 tx=0).stop 2024-04-20T21:21:53.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:21:53.465+0000 7f6754800700 1 -- 172.21.15.62:0/2023232896 shutdown_connections 2024-04-20T21:21:53.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:21:53.465+0000 7f6754800700 1 --2- 172.21.15.62:0/2023232896 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f674c100090 0x7f674c100460 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:21:53.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:21:53.466+0000 7f6754800700 1 -- 172.21.15.62:0/2023232896 >> 172.21.15.62:0/2023232896 conn(0x7f674c0fb970 msgr2=0x7f674c0fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:21:53.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:21:53.466+0000 7f6754800700 1 -- 172.21.15.62:0/2023232896 shutdown_connections 2024-04-20T21:21:53.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:21:53.466+0000 7f6754800700 1 -- 172.21.15.62:0/2023232896 wait complete. 2024-04-20T21:21:53.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:21:53.467+0000 7f6754800700 1 Processor -- start 2024-04-20T21:21:53.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:21:53.467+0000 7f6754800700 1 -- start start 2024-04-20T21:21:53.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:21:53.467+0000 7f6754800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f674c100090 0x7f674c1976f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:21:53.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:21:53.467+0000 7f6754800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f674c197c30 0x7f674c19aff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:21:53.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:21:53.467+0000 7f6754800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f674c198240 con 0x7f674c197c30 2024-04-20T21:21:53.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:21:53.467+0000 7f6754800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f674c198380 con 0x7f674c100090 2024-04-20T21:21:53.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:21:53.468+0000 7f6751400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f674c197c30 0x7f674c19aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:21:53.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:21:53.468+0000 7f6751400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f674c197c30 0x7f674c19aff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42772/0 (socket says 172.21.15.62:42772) 2024-04-20T21:21:53.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:21:53.468+0000 7f6751400700 1 -- 172.21.15.62:0/1554860247 learned_addr learned my addr 172.21.15.62:0/1554860247 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:21:53.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:21:53.468+0000 7f6751e00700 1 --2- 172.21.15.62:0/1554860247 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f674c100090 0x7f674c1976f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:21:53.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:21:53.468+0000 7f6751400700 1 -- 172.21.15.62:0/1554860247 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f674c100090 msgr2=0x7f674c1976f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:21:53.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:21:53.468+0000 7f6751400700 1 --2- 172.21.15.62:0/1554860247 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f674c100090 0x7f674c1976f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:21:53.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:21:53.468+0000 7f6751400700 1 -- 172.21.15.62:0/1554860247 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67400095f0 con 0x7f674c197c30 2024-04-20T21:21:53.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:21:53.469+0000 7f6751400700 1 --2- 172.21.15.62:0/1554860247 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f674c197c30 0x7f674c19aff0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f673c00b700 tx=0x7f673c00bac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:21:53.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:21:53.469+0000 7f6746a00700 1 -- 172.21.15.62:0/1554860247 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f673c011800 con 0x7f674c197c30 2024-04-20T21:21:53.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:21:53.469+0000 7f6746a00700 1 -- 172.21.15.62:0/1554860247 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f673c011e10 con 0x7f674c197c30 2024-04-20T21:21:53.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:21:53.469+0000 7f6746a00700 1 -- 172.21.15.62:0/1554860247 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f673c00f2e0 con 0x7f674c197c30 2024-04-20T21:21:53.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:21:53.469+0000 7f6754800700 1 -- 172.21.15.62:0/1554860247 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f674c19b5f0 con 0x7f674c197c30 2024-04-20T21:21:53.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:21:53.469+0000 7f6754800700 1 -- 172.21.15.62:0/1554860247 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f674c19bb40 con 0x7f674c197c30 2024-04-20T21:21:53.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:21:53.470+0000 7f6754800700 1 -- 172.21.15.62:0/1554860247 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f674c062380 con 0x7f674c197c30 2024-04-20T21:21:53.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:21:53.471+0000 7f6746a00700 1 -- 172.21.15.62:0/1554860247 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) v1 ==== 89858+0+0 (secure 0 0 0) 0x7f673c010380 con 0x7f674c197c30 2024-04-20T21:21:53.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:21:53.473+0000 7f6746a00700 1 --2- 172.21.15.62:0/1554860247 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f673406c760 0x7f673406ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:21:53.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:21:53.473+0000 7f6746a00700 1 -- 172.21.15.62:0/1554860247 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f673c08a4c0 con 0x7f674c197c30 2024-04-20T21:21:53.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:21:53.473+0000 7f6751e00700 1 --2- 172.21.15.62:0/1554860247 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f673406c760 0x7f673406ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:21:53.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:21:53.474+0000 7f6751e00700 1 --2- 172.21.15.62:0/1554860247 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f673406c760 0x7f673406ec10 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f6740006010 tx=0x7f674000b540 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:21:53.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:21:53.478+0000 7f6746a00700 1 -- 172.21.15.62:0/1554860247 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f673c055be0 con 0x7f674c197c30 2024-04-20T21:21:53.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:21:53.713+0000 7f6754800700 1 -- 172.21.15.62:0/1554860247 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f674c1a2370 con 0x7f674c197c30 2024-04-20T21:21:53.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:21:53.713+0000 7f6746a00700 1 -- 172.21.15.62:0/1554860247 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v10) v1 ==== 76+0+225 (secure 0 0 0) 0x7f673c015100 con 0x7f674c197c30 2024-04-20T21:21:53.713 INFO:teuthology.orchestra.run.smithi062.stdout:# minimal ceph.conf for 61ae3628-ff5b-11ee-bc93-c7b262605968 2024-04-20T21:21:53.714 INFO:teuthology.orchestra.run.smithi062.stdout:[global] 2024-04-20T21:21:53.714 INFO:teuthology.orchestra.run.smithi062.stdout: fsid = 61ae3628-ff5b-11ee-bc93-c7b262605968 2024-04-20T21:21:53.714 INFO:teuthology.orchestra.run.smithi062.stdout: mon_host = [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-20T21:21:53.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:21:53.716+0000 7f6754800700 1 -- 172.21.15.62:0/1554860247 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f673406c760 msgr2=0x7f673406ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:21:53.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:21:53.716+0000 7f6754800700 1 --2- 172.21.15.62:0/1554860247 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f673406c760 0x7f673406ec10 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f6740006010 tx=0x7f674000b540 comp rx=0 tx=0).stop 2024-04-20T21:21:53.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:21:53.716+0000 7f6754800700 1 -- 172.21.15.62:0/1554860247 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f674c197c30 msgr2=0x7f674c19aff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:21:53.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:21:53.716+0000 7f6754800700 1 --2- 172.21.15.62:0/1554860247 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f674c197c30 0x7f674c19aff0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f673c00b700 tx=0x7f673c00bac0 comp rx=0 tx=0).stop 2024-04-20T21:21:53.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:21:53.716+0000 7f6754800700 1 -- 172.21.15.62:0/1554860247 shutdown_connections 2024-04-20T21:21:53.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:21:53.716+0000 7f6754800700 1 --2- 172.21.15.62:0/1554860247 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f673406c760 0x7f673406ec10 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:21:53.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:21:53.716+0000 7f6754800700 1 --2- 172.21.15.62:0/1554860247 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f674c100090 0x7f674c1976f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:21:53.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:21:53.716+0000 7f6754800700 1 --2- 172.21.15.62:0/1554860247 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f674c197c30 0x7f674c19aff0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:21:53.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:21:53.716+0000 7f6754800700 1 -- 172.21.15.62:0/1554860247 >> 172.21.15.62:0/1554860247 conn(0x7f674c0fb970 msgr2=0x7f674c1039e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:21:53.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:21:53.716+0000 7f6754800700 1 -- 172.21.15.62:0/1554860247 shutdown_connections 2024-04-20T21:21:53.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:21:53.717+0000 7f6754800700 1 -- 172.21.15.62:0/1554860247 wait complete. 2024-04-20T21:21:53.969 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:53 smithi062 ceph-mon[16221]: Updating smithi062:/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-20T21:21:53.969 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:53 smithi062 ceph-mon[16221]: Updating smithi189:/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-20T21:21:53.969 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:53 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:21:53.969 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:53 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:21:53.969 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:53 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:21:53.969 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:53 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:21:53.969 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:53 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:21:53.969 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:53 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-20T21:21:53.970 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:53 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-20T21:21:53.970 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:53 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:21:53.970 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:53 smithi062 ceph-mon[16221]: from='client.? 172.21.15.62:0/1554860247' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:21:53.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:21:53 smithi189 ceph-mon[21562]: Updating smithi062:/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-20T21:21:53.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:21:53 smithi189 ceph-mon[21562]: Updating smithi189:/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-20T21:21:53.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:21:53 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:21:53.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:21:53 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:21:53.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:21:53 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:21:53.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:21:53 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:21:53.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:21:53 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:21:53.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:21:53 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-20T21:21:53.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:21:53 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-20T21:21:53.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:21:53 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:21:53.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:21:53 smithi189 ceph-mon[21562]: from='client.? 172.21.15.62:0/1554860247' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:21:54.540 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-20T21:21:54.540 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-04-20T21:21:54.540 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/etc/ceph/ceph.conf 2024-04-20T21:21:54.612 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-04-20T21:21:54.612 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-20T21:21:54.729 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-20T21:21:54.729 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/etc/ceph/ceph.conf 2024-04-20T21:21:54.745 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:54 smithi062 ceph-mon[16221]: Reconfiguring mon.smithi062 (unknown last config time)... 2024-04-20T21:21:54.823 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-20T21:21:54.824 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-20T21:21:54.952 INFO:tasks.cephadm:Deploying OSDs... 2024-04-20T21:21:54.952 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-04-20T21:21:54.952 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/scratch_devs of=/dev/stdout 2024-04-20T21:21:55.002 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-20T21:21:55.003 DEBUG:teuthology.orchestra.run.smithi062:> stat /dev/vg_nvme/lv_1 2024-04-20T21:21:55.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:54 smithi062 ceph-mon[16221]: Reconfiguring daemon mon.smithi062 on smithi062 2024-04-20T21:21:55.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:54 smithi062 ceph-mon[16221]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:21:55.080 INFO:teuthology.orchestra.run.smithi062.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-20T21:21:55.080 INFO:teuthology.orchestra.run.smithi062.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-20T21:21:55.081 INFO:teuthology.orchestra.run.smithi062.stdout:Device: 6h/6d Inode: 632 Links: 1 2024-04-20T21:21:55.081 INFO:teuthology.orchestra.run.smithi062.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-20T21:21:55.081 INFO:teuthology.orchestra.run.smithi062.stdout:Context: system_u:object_r:device_t:s0 2024-04-20T21:21:55.081 INFO:teuthology.orchestra.run.smithi062.stdout:Access: 2024-04-20 21:20:52.711944575 +0000 2024-04-20T21:21:55.081 INFO:teuthology.orchestra.run.smithi062.stdout:Modify: 2024-04-20 21:14:03.478250081 +0000 2024-04-20T21:21:55.081 INFO:teuthology.orchestra.run.smithi062.stdout:Change: 2024-04-20 21:14:03.478250081 +0000 2024-04-20T21:21:55.081 INFO:teuthology.orchestra.run.smithi062.stdout: Birth: 2024-04-20 21:14:03.478250081 +0000 2024-04-20T21:21:55.081 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-20T21:21:55.208 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records in 2024-04-20T21:21:55.208 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records out 2024-04-20T21:21:55.209 INFO:teuthology.orchestra.run.smithi062.stderr:512 bytes copied, 0.000471093 s, 1.1 MB/s 2024-04-20T21:21:55.213 DEBUG:teuthology.orchestra.run.smithi062:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-20T21:21:55.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:21:54 smithi189 ceph-mon[21562]: Reconfiguring mon.smithi062 (unknown last config time)... 2024-04-20T21:21:55.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:21:54 smithi189 ceph-mon[21562]: Reconfiguring daemon mon.smithi062 on smithi062 2024-04-20T21:21:55.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:21:54 smithi189 ceph-mon[21562]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:21:55.271 DEBUG:teuthology.orchestra.run.smithi062:> stat /dev/vg_nvme/lv_2 2024-04-20T21:21:55.323 INFO:teuthology.orchestra.run.smithi062.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-20T21:21:55.324 INFO:teuthology.orchestra.run.smithi062.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-20T21:21:55.324 INFO:teuthology.orchestra.run.smithi062.stdout:Device: 6h/6d Inode: 651 Links: 1 2024-04-20T21:21:55.324 INFO:teuthology.orchestra.run.smithi062.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-20T21:21:55.324 INFO:teuthology.orchestra.run.smithi062.stdout:Context: system_u:object_r:device_t:s0 2024-04-20T21:21:55.324 INFO:teuthology.orchestra.run.smithi062.stdout:Access: 2024-04-20 21:20:52.711944575 +0000 2024-04-20T21:21:55.324 INFO:teuthology.orchestra.run.smithi062.stdout:Modify: 2024-04-20 21:14:04.103240026 +0000 2024-04-20T21:21:55.324 INFO:teuthology.orchestra.run.smithi062.stdout:Change: 2024-04-20 21:14:04.103240026 +0000 2024-04-20T21:21:55.324 INFO:teuthology.orchestra.run.smithi062.stdout: Birth: 2024-04-20 21:14:04.103240026 +0000 2024-04-20T21:21:55.324 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-20T21:21:55.402 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records in 2024-04-20T21:21:55.402 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records out 2024-04-20T21:21:55.402 INFO:teuthology.orchestra.run.smithi062.stderr:512 bytes copied, 0.000385628 s, 1.3 MB/s 2024-04-20T21:21:55.406 DEBUG:teuthology.orchestra.run.smithi062:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-20T21:21:55.462 DEBUG:teuthology.orchestra.run.smithi062:> stat /dev/vg_nvme/lv_3 2024-04-20T21:21:55.512 INFO:teuthology.orchestra.run.smithi062.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-20T21:21:55.512 INFO:teuthology.orchestra.run.smithi062.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-20T21:21:55.512 INFO:teuthology.orchestra.run.smithi062.stdout:Device: 6h/6d Inode: 671 Links: 1 2024-04-20T21:21:55.512 INFO:teuthology.orchestra.run.smithi062.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-20T21:21:55.512 INFO:teuthology.orchestra.run.smithi062.stdout:Context: system_u:object_r:device_t:s0 2024-04-20T21:21:55.512 INFO:teuthology.orchestra.run.smithi062.stdout:Access: 2024-04-20 21:20:52.712944559 +0000 2024-04-20T21:21:55.513 INFO:teuthology.orchestra.run.smithi062.stdout:Modify: 2024-04-20 21:14:04.758229489 +0000 2024-04-20T21:21:55.513 INFO:teuthology.orchestra.run.smithi062.stdout:Change: 2024-04-20 21:14:04.758229489 +0000 2024-04-20T21:21:55.513 INFO:teuthology.orchestra.run.smithi062.stdout: Birth: 2024-04-20 21:14:04.758229489 +0000 2024-04-20T21:21:55.513 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-20T21:21:55.592 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records in 2024-04-20T21:21:55.592 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records out 2024-04-20T21:21:55.592 INFO:teuthology.orchestra.run.smithi062.stderr:512 bytes copied, 0.00038271 s, 1.3 MB/s 2024-04-20T21:21:55.596 DEBUG:teuthology.orchestra.run.smithi062:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-20T21:21:55.654 DEBUG:teuthology.orchestra.run.smithi062:> stat /dev/vg_nvme/lv_4 2024-04-20T21:21:55.705 INFO:teuthology.orchestra.run.smithi062.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-20T21:21:55.706 INFO:teuthology.orchestra.run.smithi062.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-20T21:21:55.706 INFO:teuthology.orchestra.run.smithi062.stdout:Device: 6h/6d Inode: 689 Links: 1 2024-04-20T21:21:55.706 INFO:teuthology.orchestra.run.smithi062.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-20T21:21:55.706 INFO:teuthology.orchestra.run.smithi062.stdout:Context: system_u:object_r:device_t:s0 2024-04-20T21:21:55.706 INFO:teuthology.orchestra.run.smithi062.stdout:Access: 2024-04-20 21:20:52.713944544 +0000 2024-04-20T21:21:55.706 INFO:teuthology.orchestra.run.smithi062.stdout:Modify: 2024-04-20 21:14:05.358219836 +0000 2024-04-20T21:21:55.706 INFO:teuthology.orchestra.run.smithi062.stdout:Change: 2024-04-20 21:14:05.358219836 +0000 2024-04-20T21:21:55.706 INFO:teuthology.orchestra.run.smithi062.stdout: Birth: 2024-04-20 21:14:05.358219836 +0000 2024-04-20T21:21:55.706 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-20T21:21:55.784 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records in 2024-04-20T21:21:55.785 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records out 2024-04-20T21:21:55.785 INFO:teuthology.orchestra.run.smithi062.stderr:512 bytes copied, 0.00036612 s, 1.4 MB/s 2024-04-20T21:21:55.788 DEBUG:teuthology.orchestra.run.smithi062:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-20T21:21:55.846 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-20T21:21:55.846 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/scratch_devs of=/dev/stdout 2024-04-20T21:21:55.904 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-20T21:21:55.904 DEBUG:teuthology.orchestra.run.smithi189:> stat /dev/vg_nvme/lv_1 2024-04-20T21:21:55.954 INFO:teuthology.orchestra.run.smithi189.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-20T21:21:55.954 INFO:teuthology.orchestra.run.smithi189.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-20T21:21:55.954 INFO:teuthology.orchestra.run.smithi189.stdout:Device: 6h/6d Inode: 633 Links: 1 2024-04-20T21:21:55.955 INFO:teuthology.orchestra.run.smithi189.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-20T21:21:55.955 INFO:teuthology.orchestra.run.smithi189.stdout:Context: system_u:object_r:device_t:s0 2024-04-20T21:21:55.955 INFO:teuthology.orchestra.run.smithi189.stdout:Access: 2024-04-20 21:21:41.303906595 +0000 2024-04-20T21:21:55.955 INFO:teuthology.orchestra.run.smithi189.stdout:Modify: 2024-04-20 21:14:03.317488360 +0000 2024-04-20T21:21:55.955 INFO:teuthology.orchestra.run.smithi189.stdout:Change: 2024-04-20 21:14:03.317488360 +0000 2024-04-20T21:21:55.955 INFO:teuthology.orchestra.run.smithi189.stdout: Birth: 2024-04-20 21:14:03.317488360 +0000 2024-04-20T21:21:55.955 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-20T21:21:56.033 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records in 2024-04-20T21:21:56.033 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records out 2024-04-20T21:21:56.033 INFO:teuthology.orchestra.run.smithi189.stderr:512 bytes copied, 0.00046307 s, 1.1 MB/s 2024-04-20T21:21:56.037 DEBUG:teuthology.orchestra.run.smithi189:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-20T21:21:56.091 DEBUG:teuthology.orchestra.run.smithi189:> stat /dev/vg_nvme/lv_2 2024-04-20T21:21:56.138 INFO:teuthology.orchestra.run.smithi189.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-20T21:21:56.138 INFO:teuthology.orchestra.run.smithi189.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-20T21:21:56.138 INFO:teuthology.orchestra.run.smithi189.stdout:Device: 6h/6d Inode: 653 Links: 1 2024-04-20T21:21:56.138 INFO:teuthology.orchestra.run.smithi189.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-20T21:21:56.138 INFO:teuthology.orchestra.run.smithi189.stdout:Context: system_u:object_r:device_t:s0 2024-04-20T21:21:56.138 INFO:teuthology.orchestra.run.smithi189.stdout:Access: 2024-04-20 21:21:41.304906575 +0000 2024-04-20T21:21:56.138 INFO:teuthology.orchestra.run.smithi189.stdout:Modify: 2024-04-20 21:14:03.898476689 +0000 2024-04-20T21:21:56.138 INFO:teuthology.orchestra.run.smithi189.stdout:Change: 2024-04-20 21:14:03.898476689 +0000 2024-04-20T21:21:56.138 INFO:teuthology.orchestra.run.smithi189.stdout: Birth: 2024-04-20 21:14:03.898476689 +0000 2024-04-20T21:21:56.138 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-20T21:21:56.215 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records in 2024-04-20T21:21:56.215 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records out 2024-04-20T21:21:56.215 INFO:teuthology.orchestra.run.smithi189.stderr:512 bytes copied, 0.000358968 s, 1.4 MB/s 2024-04-20T21:21:56.218 DEBUG:teuthology.orchestra.run.smithi189:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-20T21:21:56.272 DEBUG:teuthology.orchestra.run.smithi189:> stat /dev/vg_nvme/lv_3 2024-04-20T21:21:56.321 INFO:teuthology.orchestra.run.smithi189.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-20T21:21:56.321 INFO:teuthology.orchestra.run.smithi189.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-20T21:21:56.321 INFO:teuthology.orchestra.run.smithi189.stdout:Device: 6h/6d Inode: 672 Links: 1 2024-04-20T21:21:56.321 INFO:teuthology.orchestra.run.smithi189.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-20T21:21:56.321 INFO:teuthology.orchestra.run.smithi189.stdout:Context: system_u:object_r:device_t:s0 2024-04-20T21:21:56.321 INFO:teuthology.orchestra.run.smithi189.stdout:Access: 2024-04-20 21:21:41.305906556 +0000 2024-04-20T21:21:56.321 INFO:teuthology.orchestra.run.smithi189.stdout:Modify: 2024-04-20 21:14:04.484464917 +0000 2024-04-20T21:21:56.321 INFO:teuthology.orchestra.run.smithi189.stdout:Change: 2024-04-20 21:14:04.484464917 +0000 2024-04-20T21:21:56.321 INFO:teuthology.orchestra.run.smithi189.stdout: Birth: 2024-04-20 21:14:04.484464917 +0000 2024-04-20T21:21:56.322 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-20T21:21:56.397 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records in 2024-04-20T21:21:56.397 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records out 2024-04-20T21:21:56.397 INFO:teuthology.orchestra.run.smithi189.stderr:512 bytes copied, 0.0003817 s, 1.3 MB/s 2024-04-20T21:21:56.401 DEBUG:teuthology.orchestra.run.smithi189:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-20T21:21:56.452 DEBUG:teuthology.orchestra.run.smithi189:> stat /dev/vg_nvme/lv_4 2024-04-20T21:21:56.500 INFO:teuthology.orchestra.run.smithi189.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-20T21:21:56.500 INFO:teuthology.orchestra.run.smithi189.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-20T21:21:56.500 INFO:teuthology.orchestra.run.smithi189.stdout:Device: 6h/6d Inode: 690 Links: 1 2024-04-20T21:21:56.500 INFO:teuthology.orchestra.run.smithi189.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-20T21:21:56.500 INFO:teuthology.orchestra.run.smithi189.stdout:Context: system_u:object_r:device_t:s0 2024-04-20T21:21:56.500 INFO:teuthology.orchestra.run.smithi189.stdout:Access: 2024-04-20 21:21:41.305906556 +0000 2024-04-20T21:21:56.500 INFO:teuthology.orchestra.run.smithi189.stdout:Modify: 2024-04-20 21:14:05.119452161 +0000 2024-04-20T21:21:56.500 INFO:teuthology.orchestra.run.smithi189.stdout:Change: 2024-04-20 21:14:05.119452161 +0000 2024-04-20T21:21:56.500 INFO:teuthology.orchestra.run.smithi189.stdout: Birth: 2024-04-20 21:14:05.119452161 +0000 2024-04-20T21:21:56.501 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-20T21:21:56.577 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records in 2024-04-20T21:21:56.585 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records out 2024-04-20T21:21:56.585 INFO:teuthology.orchestra.run.smithi189.stderr:512 bytes copied, 0.000331164 s, 1.5 MB/s 2024-04-20T21:21:56.586 DEBUG:teuthology.orchestra.run.smithi189:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-20T21:21:56.636 INFO:tasks.cephadm:Deploying osd.0 on smithi062 with /dev/vg_nvme/lv_4... 2024-04-20T21:21:56.636 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-20T21:21:56.993 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:56 smithi062 ceph-mon[16221]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:21:56.993 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:56 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:21:56.993 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:56 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:21:56.993 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:56 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi062.rqxuxi", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-20T21:21:56.993 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:56 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-20T21:21:56.994 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:56 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:21:57.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:21:56 smithi189 ceph-mon[21562]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:21:57.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:21:56 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:21:57.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:21:56 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:21:57.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:21:56 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi062.rqxuxi", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-20T21:21:57.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:21:56 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-20T21:21:57.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:21:56 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:21:58.047 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T21:21:58.197 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:57 smithi062 ceph-mon[16221]: Reconfiguring mgr.smithi062.rqxuxi (unknown last config time)... 2024-04-20T21:21:58.197 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:57 smithi062 ceph-mon[16221]: Reconfiguring daemon mgr.smithi062.rqxuxi on smithi062 2024-04-20T21:21:58.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:21:57 smithi189 ceph-mon[21562]: Reconfiguring mgr.smithi062.rqxuxi (unknown last config time)... 2024-04-20T21:21:58.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:21:57 smithi189 ceph-mon[21562]: Reconfiguring daemon mgr.smithi062.rqxuxi on smithi062 2024-04-20T21:21:59.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:58 smithi062 ceph-mon[16221]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:21:59.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:58 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:21:59.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:58 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:21:59.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:58 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi062", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-20T21:21:59.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:58 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:21:59.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:21:58 smithi189 ceph-mon[21562]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:21:59.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:21:58 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:21:59.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:21:58 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:21:59.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:21:58 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi062", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-20T21:21:59.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:21:58 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:21:59.907 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:59 smithi062 ceph-mon[16221]: Reconfiguring ceph-exporter.smithi062 (monmap changed)... 2024-04-20T21:21:59.907 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:59 smithi062 ceph-mon[16221]: Reconfiguring daemon ceph-exporter.smithi062 on smithi062 2024-04-20T21:21:59.907 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:21:59 smithi062 ceph-mon[16221]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:22:00.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:21:59 smithi189 ceph-mon[21562]: Reconfiguring ceph-exporter.smithi062 (monmap changed)... 2024-04-20T21:22:00.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:21:59 smithi189 ceph-mon[21562]: Reconfiguring daemon ceph-exporter.smithi062 on smithi062 2024-04-20T21:22:00.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:21:59 smithi189 ceph-mon[21562]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:22:01.279 INFO:teuthology.orchestra.run.smithi062.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-20T21:22:01.279 INFO:teuthology.orchestra.run.smithi062.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-20T21:22:01.279 INFO:teuthology.orchestra.run.smithi062.stderr: stderr: 10+0 records in 2024-04-20T21:22:01.279 INFO:teuthology.orchestra.run.smithi062.stderr:10+0 records out 2024-04-20T21:22:01.279 INFO:teuthology.orchestra.run.smithi062.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0365121 s, 287 MB/s 2024-04-20T21:22:01.279 INFO:teuthology.orchestra.run.smithi062.stderr:--> Zapping successful for: 2024-04-20T21:22:02.211 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -- ceph orch daemon add osd smithi062:vg_nvme/lv_4 2024-04-20T21:22:02.404 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:02 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:22:02.404 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:02 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:22:02.404 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:02 smithi062 ceph-mon[16221]: Reconfiguring crash.smithi062 (monmap changed)... 2024-04-20T21:22:02.404 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:02 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi062", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-20T21:22:02.404 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:02 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:22:02.404 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:02 smithi062 ceph-mon[16221]: Reconfiguring daemon crash.smithi062 on smithi062 2024-04-20T21:22:02.404 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:02 smithi062 ceph-mon[16221]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:22:02.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:02 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:22:02.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:02 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:22:02.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:02 smithi189 ceph-mon[21562]: Reconfiguring crash.smithi062 (monmap changed)... 2024-04-20T21:22:02.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:02 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi062", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-20T21:22:02.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:02 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:22:02.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:02 smithi189 ceph-mon[21562]: Reconfiguring daemon crash.smithi062 on smithi062 2024-04-20T21:22:02.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:02 smithi189 ceph-mon[21562]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:22:03.407 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T21:22:04.772 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:04 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:22:04.772 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:04 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:22:04.772 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:04 smithi062 ceph-mon[16221]: Reconfiguring alertmanager.smithi062 (dependencies changed)... 2024-04-20T21:22:04.772 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:04 smithi062 ceph-mon[16221]: Reconfiguring daemon alertmanager.smithi062 on smithi062 2024-04-20T21:22:04.772 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:04 smithi062 ceph-mon[16221]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:22:04.772 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:04 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:22:04.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:04 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:22:04.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:04 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:22:04.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:04 smithi189 ceph-mon[21562]: Reconfiguring alertmanager.smithi062 (dependencies changed)... 2024-04-20T21:22:04.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:04 smithi189 ceph-mon[21562]: Reconfiguring daemon alertmanager.smithi062 on smithi062 2024-04-20T21:22:04.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:04 smithi189 ceph-mon[21562]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:22:04.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:04 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:22:07.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:06 smithi062 ceph-mon[16221]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:22:07.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:06 smithi189 ceph-mon[21562]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:22:07.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:22:07.416+0000 7fa419200700 1 -- 172.21.15.62:0/1693639324 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa414074ce0 msgr2=0x7fa4140731e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:22:07.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:22:07.416+0000 7fa419200700 1 --2- 172.21.15.62:0/1693639324 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa414074ce0 0x7fa4140731e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fa404009910 tx=0x7fa404009c20 comp rx=0 tx=0).stop 2024-04-20T21:22:07.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:22:07.416+0000 7fa419200700 1 -- 172.21.15.62:0/1693639324 shutdown_connections 2024-04-20T21:22:07.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:22:07.416+0000 7fa419200700 1 --2- 172.21.15.62:0/1693639324 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa414073720 0x7fa414073b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:22:07.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:22:07.416+0000 7fa419200700 1 --2- 172.21.15.62:0/1693639324 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa414074ce0 0x7fa4140731e0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:22:07.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:22:07.416+0000 7fa419200700 1 -- 172.21.15.62:0/1693639324 >> 172.21.15.62:0/1693639324 conn(0x7fa4140fb890 msgr2=0x7fa4140fdcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:22:07.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:22:07.417+0000 7fa419200700 1 -- 172.21.15.62:0/1693639324 shutdown_connections 2024-04-20T21:22:07.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:22:07.417+0000 7fa419200700 1 -- 172.21.15.62:0/1693639324 wait complete. 2024-04-20T21:22:07.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:22:07.418+0000 7fa419200700 1 Processor -- start 2024-04-20T21:22:07.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:22:07.418+0000 7fa419200700 1 -- start start 2024-04-20T21:22:07.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:22:07.418+0000 7fa419200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa414073720 0x7fa41419c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:22:07.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:22:07.418+0000 7fa419200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa414074ce0 0x7fa41419c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:22:07.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:22:07.418+0000 7fa419200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa41419cc40 con 0x7fa414074ce0 2024-04-20T21:22:07.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:22:07.418+0000 7fa419200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa41419cd80 con 0x7fa414073720 2024-04-20T21:22:07.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:22:07.419+0000 7fa411400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa414074ce0 0x7fa41419c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:22:07.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:22:07.419+0000 7fa411400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa414074ce0 0x7fa41419c620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48994/0 (socket says 172.21.15.62:48994) 2024-04-20T21:22:07.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:22:07.419+0000 7fa411400700 1 -- 172.21.15.62:0/3300993859 learned_addr learned my addr 172.21.15.62:0/3300993859 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:22:07.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:22:07.419+0000 7fa411e00700 1 --2- 172.21.15.62:0/3300993859 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa414073720 0x7fa41419c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:22:07.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:22:07.419+0000 7fa411400700 1 -- 172.21.15.62:0/3300993859 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa414073720 msgr2=0x7fa41419c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:22:07.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:22:07.419+0000 7fa411400700 1 --2- 172.21.15.62:0/3300993859 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa414073720 0x7fa41419c0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:22:07.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:22:07.419+0000 7fa411400700 1 -- 172.21.15.62:0/3300993859 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4040095f0 con 0x7fa414074ce0 2024-04-20T21:22:07.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:22:07.419+0000 7fa411400700 1 --2- 172.21.15.62:0/3300993859 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa414074ce0 0x7fa41419c620 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fa40000d8d0 tx=0x7fa40000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:22:07.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:22:07.420+0000 7fa40aa00700 1 -- 172.21.15.62:0/3300993859 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa400009940 con 0x7fa414074ce0 2024-04-20T21:22:07.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:22:07.420+0000 7fa40aa00700 1 -- 172.21.15.62:0/3300993859 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa400010460 con 0x7fa414074ce0 2024-04-20T21:22:07.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:22:07.420+0000 7fa419200700 1 -- 172.21.15.62:0/3300993859 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4141a1830 con 0x7fa414074ce0 2024-04-20T21:22:07.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:22:07.420+0000 7fa40aa00700 1 -- 172.21.15.62:0/3300993859 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa40000f5f0 con 0x7fa414074ce0 2024-04-20T21:22:07.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:22:07.420+0000 7fa419200700 1 -- 172.21.15.62:0/3300993859 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4141a1d80 con 0x7fa414074ce0 2024-04-20T21:22:07.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:22:07.421+0000 7fa419200700 1 -- 172.21.15.62:0/3300993859 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa41410f300 con 0x7fa414074ce0 2024-04-20T21:22:07.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:22:07.422+0000 7fa40aa00700 1 -- 172.21.15.62:0/3300993859 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) v1 ==== 89858+0+0 (secure 0 0 0) 0x7fa4000105d0 con 0x7fa414074ce0 2024-04-20T21:22:07.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:22:07.423+0000 7fa40aa00700 1 --2- 172.21.15.62:0/3300993859 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fa3f806c610 0x7fa3f806eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:22:07.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:22:07.423+0000 7fa40aa00700 1 -- 172.21.15.62:0/3300993859 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7fa40008a810 con 0x7fa414074ce0 2024-04-20T21:22:07.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:22:07.423+0000 7fa411e00700 1 --2- 172.21.15.62:0/3300993859 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fa3f806c610 0x7fa3f806eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:22:07.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:22:07.424+0000 7fa411e00700 1 --2- 172.21.15.62:0/3300993859 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fa3f806c610 0x7fa3f806eac0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa404006010 tx=0x7fa40400b540 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:22:07.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:22:07.428+0000 7fa40aa00700 1 -- 172.21.15.62:0/3300993859 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fa4000599c0 con 0x7fa414074ce0 2024-04-20T21:22:07.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:22:07.663+0000 7fa419200700 1 -- 172.21.15.62:0/3300993859 --> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi062:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7fa414103390 con 0x7fa3f806c610 2024-04-20T21:22:08.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:07 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-20T21:22:08.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:07 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-20T21:22:08.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:07 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:22:08.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:07 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-20T21:22:08.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:07 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-20T21:22:08.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:07 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:22:09.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:08 smithi062 ceph-mon[16221]: from='client.14254 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi062:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:22:09.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:08 smithi062 ceph-mon[16221]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:22:09.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:08 smithi189 ceph-mon[21562]: from='client.14254 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi062:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:22:09.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:08 smithi189 ceph-mon[21562]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:22:09.927 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:09 smithi062 ceph-mon[16221]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:22:10.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:09 smithi189 ceph-mon[21562]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:22:12.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:12 smithi189 ceph-mon[21562]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:22:12.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:12 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:22:12.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:12 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:22:13.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:12 smithi062 ceph-mon[16221]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:22:13.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:12 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:22:13.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:12 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:22:13.969 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:13 smithi062 ceph-mon[16221]: Reconfiguring grafana.smithi062 (dependencies changed)... 2024-04-20T21:22:13.969 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:13 smithi062 ceph-mon[16221]: Reconfiguring daemon grafana.smithi062 on smithi062 2024-04-20T21:22:13.969 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:13 smithi062 ceph-mon[16221]: from='client.? 172.21.15.62:0/3180044768' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4a3f8216-c641-4338-813d-d836efc3f4e8"}]: dispatch 2024-04-20T21:22:13.969 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:13 smithi062 ceph-mon[16221]: from='client.? 172.21.15.62:0/3180044768' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4a3f8216-c641-4338-813d-d836efc3f4e8"}]': finished 2024-04-20T21:22:13.969 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:13 smithi062 ceph-mon[16221]: osdmap e6: 1 total, 0 up, 1 in 2024-04-20T21:22:13.970 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:13 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-20T21:22:13.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:13 smithi189 ceph-mon[21562]: Reconfiguring grafana.smithi062 (dependencies changed)... 2024-04-20T21:22:13.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:13 smithi189 ceph-mon[21562]: Reconfiguring daemon grafana.smithi062 on smithi062 2024-04-20T21:22:13.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:13 smithi189 ceph-mon[21562]: from='client.? 172.21.15.62:0/3180044768' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4a3f8216-c641-4338-813d-d836efc3f4e8"}]: dispatch 2024-04-20T21:22:13.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:13 smithi189 ceph-mon[21562]: from='client.? 172.21.15.62:0/3180044768' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4a3f8216-c641-4338-813d-d836efc3f4e8"}]': finished 2024-04-20T21:22:13.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:13 smithi189 ceph-mon[21562]: osdmap e6: 1 total, 0 up, 1 in 2024-04-20T21:22:13.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:13 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-20T21:22:15.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:14 smithi062 ceph-mon[16221]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:22:15.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:14 smithi062 ceph-mon[16221]: from='client.? 172.21.15.62:0/2955439224' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-20T21:22:15.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:14 smithi189 ceph-mon[21562]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:22:15.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:14 smithi189 ceph-mon[21562]: from='client.? 172.21.15.62:0/2955439224' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-20T21:22:17.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:16 smithi189 ceph-mon[21562]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:22:17.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:16 smithi062 ceph-mon[16221]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:22:19.153 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:18 smithi062 ceph-mon[16221]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:22:19.153 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:18 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:22:19.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:18 smithi189 ceph-mon[21562]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:22:19.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:18 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:22:19.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:19 smithi189 ceph-mon[21562]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:22:20.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:19 smithi062 ceph-mon[16221]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:22:22.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:22 smithi189 ceph-mon[21562]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:22:23.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:22 smithi062 ceph-mon[16221]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:22:25.122 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:24 smithi062 ceph-mon[16221]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:22:25.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:24 smithi189 ceph-mon[21562]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:22:26.392 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:26 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:22:26.392 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:26 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:22:26.392 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:26 smithi062 ceph-mon[16221]: Reconfiguring prometheus.smithi062 (dependencies changed)... 2024-04-20T21:22:26.392 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:26 smithi062 ceph-mon[16221]: Reconfiguring daemon prometheus.smithi062 on smithi062 2024-04-20T21:22:26.392 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:26 smithi062 ceph-mon[16221]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:22:26.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:26 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:22:26.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:26 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:22:26.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:26 smithi189 ceph-mon[21562]: Reconfiguring prometheus.smithi062 (dependencies changed)... 2024-04-20T21:22:26.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:26 smithi189 ceph-mon[21562]: Reconfiguring daemon prometheus.smithi062 on smithi062 2024-04-20T21:22:26.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:26 smithi189 ceph-mon[21562]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:22:27.976 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:27 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-20T21:22:27.976 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:27 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:22:27.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:27 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-20T21:22:27.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:27 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:22:28.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:28 smithi189 ceph-mon[21562]: Deploying daemon osd.0 on smithi062 2024-04-20T21:22:28.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:28 smithi189 ceph-mon[21562]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:22:29.028 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:28 smithi062 ceph-mon[16221]: Deploying daemon osd.0 on smithi062 2024-04-20T21:22:29.028 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:28 smithi062 ceph-mon[16221]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:22:31.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:30 smithi189 ceph-mon[21562]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:22:31.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:30 smithi062 ceph-mon[16221]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:22:32.978 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:32 smithi062 ceph-mon[16221]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:22:32.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:32 smithi189 ceph-mon[21562]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:22:34.154 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:34 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:22:34.154 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:34 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:22:34.154 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:34 smithi189 ceph-mon[21562]: Reconfiguring ceph-exporter.smithi189 (monmap changed)... 2024-04-20T21:22:34.154 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:34 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi189", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-20T21:22:34.154 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:34 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:22:34.154 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:34 smithi189 ceph-mon[21562]: Reconfiguring daemon ceph-exporter.smithi189 on smithi189 2024-04-20T21:22:34.154 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:34 smithi189 ceph-mon[21562]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:22:34.154 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:34 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:22:34.485 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:34 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:22:34.485 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:34 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:22:34.486 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:34 smithi062 ceph-mon[16221]: Reconfiguring ceph-exporter.smithi189 (monmap changed)... 2024-04-20T21:22:34.486 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:34 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi189", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-20T21:22:34.486 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:34 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:22:34.486 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:34 smithi062 ceph-mon[16221]: Reconfiguring daemon ceph-exporter.smithi189 on smithi189 2024-04-20T21:22:34.486 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:34 smithi062 ceph-mon[16221]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:22:34.486 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:34 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:22:36.142 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:36 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:22:36.142 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:36 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:22:36.142 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:36 smithi189 ceph-mon[21562]: Reconfiguring crash.smithi189 (monmap changed)... 2024-04-20T21:22:36.142 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:36 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi189", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-20T21:22:36.142 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:36 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:22:36.143 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:36 smithi189 ceph-mon[21562]: Reconfiguring daemon crash.smithi189 on smithi189 2024-04-20T21:22:36.143 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:36 smithi189 ceph-mon[21562]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:22:36.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:36 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:22:36.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:36 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:22:36.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:36 smithi062 ceph-mon[16221]: Reconfiguring crash.smithi189 (monmap changed)... 2024-04-20T21:22:36.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:36 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi189", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-20T21:22:36.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:36 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:22:36.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:36 smithi062 ceph-mon[16221]: Reconfiguring daemon crash.smithi189 on smithi189 2024-04-20T21:22:36.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:36 smithi062 ceph-mon[16221]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:22:38.050 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:38 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:22:38.050 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:38 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:22:38.050 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:38 smithi189 ceph-mon[21562]: Reconfiguring mgr.smithi189.tawavz (monmap changed)... 2024-04-20T21:22:38.050 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:38 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi189.tawavz", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-20T21:22:38.050 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:38 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-20T21:22:38.050 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:38 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:22:38.050 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:38 smithi189 ceph-mon[21562]: Reconfiguring daemon mgr.smithi189.tawavz on smithi189 2024-04-20T21:22:38.050 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:38 smithi189 ceph-mon[21562]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:22:38.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:38 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:22:38.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:38 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:22:38.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:38 smithi062 ceph-mon[16221]: Reconfiguring mgr.smithi189.tawavz (monmap changed)... 2024-04-20T21:22:38.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:38 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi189.tawavz", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-20T21:22:38.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:38 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-20T21:22:38.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:38 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:22:38.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:38 smithi062 ceph-mon[16221]: Reconfiguring daemon mgr.smithi189.tawavz on smithi189 2024-04-20T21:22:38.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:38 smithi062 ceph-mon[16221]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:22:39.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:39 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:22:39.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:39 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:22:39.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:39 smithi189 ceph-mon[21562]: Reconfiguring mon.smithi189 (monmap changed)... 2024-04-20T21:22:39.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:39 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-20T21:22:39.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:39 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-20T21:22:39.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:39 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:22:39.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:39 smithi189 ceph-mon[21562]: Reconfiguring daemon mon.smithi189 on smithi189 2024-04-20T21:22:39.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:39 smithi189 ceph-mon[21562]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:22:40.195 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:39 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:22:40.195 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:39 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:22:40.195 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:39 smithi062 ceph-mon[16221]: Reconfiguring mon.smithi189 (monmap changed)... 2024-04-20T21:22:40.195 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:39 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-20T21:22:40.195 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:39 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-20T21:22:40.195 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:39 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:22:40.195 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:39 smithi062 ceph-mon[16221]: Reconfiguring daemon mon.smithi189 on smithi189 2024-04-20T21:22:40.195 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:39 smithi062 ceph-mon[16221]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:22:41.726 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:41 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:22:41.726 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:41 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:22:41.726 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:41 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:22:41.726 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:41 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:22:41.726 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:41 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-20T21:22:41.726 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:41 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi062.front.sepia.ceph.com:9093"}]: dispatch 2024-04-20T21:22:41.726 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:41 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:22:41.726 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:41 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-20T21:22:41.726 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:41 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi062.front.sepia.ceph.com:3000"}]: dispatch 2024-04-20T21:22:41.727 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:41 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:22:41.727 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:41 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-20T21:22:41.727 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:41 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi062.front.sepia.ceph.com:9095"}]: dispatch 2024-04-20T21:22:41.727 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:41 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:22:41.727 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:41 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:22:41.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:41 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:22:41.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:41 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:22:41.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:41 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:22:41.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:41 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:22:41.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:41 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-20T21:22:41.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:41 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi062.front.sepia.ceph.com:9093"}]: dispatch 2024-04-20T21:22:41.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:41 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:22:41.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:41 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-20T21:22:41.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:41 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi062.front.sepia.ceph.com:3000"}]: dispatch 2024-04-20T21:22:41.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:41 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:22:41.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:41 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-20T21:22:41.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:41 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi062.front.sepia.ceph.com:9095"}]: dispatch 2024-04-20T21:22:41.989 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:41 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:22:41.989 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:41 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:22:42.934 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:42 smithi189 ceph-mon[21562]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-20T21:22:42.934 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:42 smithi189 ceph-mon[21562]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi062.front.sepia.ceph.com:9093"}]: dispatch 2024-04-20T21:22:42.935 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:42 smithi189 ceph-mon[21562]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-20T21:22:42.935 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:42 smithi189 ceph-mon[21562]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi062.front.sepia.ceph.com:3000"}]: dispatch 2024-04-20T21:22:42.935 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:42 smithi189 ceph-mon[21562]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-20T21:22:42.935 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:42 smithi189 ceph-mon[21562]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi062.front.sepia.ceph.com:9095"}]: dispatch 2024-04-20T21:22:42.935 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:42 smithi189 ceph-mon[21562]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:22:42.935 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:42 smithi062 ceph-mon[16221]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-20T21:22:42.935 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:42 smithi062 ceph-mon[16221]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi062.front.sepia.ceph.com:9093"}]: dispatch 2024-04-20T21:22:42.935 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:42 smithi062 ceph-mon[16221]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-20T21:22:42.935 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:42 smithi062 ceph-mon[16221]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi062.front.sepia.ceph.com:3000"}]: dispatch 2024-04-20T21:22:42.935 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:42 smithi062 ceph-mon[16221]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-20T21:22:42.936 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:42 smithi062 ceph-mon[16221]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi062.front.sepia.ceph.com:9095"}]: dispatch 2024-04-20T21:22:42.936 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:42 smithi062 ceph-mon[16221]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:22:44.533 INFO:teuthology.orchestra.run.smithi062.stdout:Created osd(s) 0 on host 'smithi062' 2024-04-20T21:22:44.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:22:44.532+0000 7fa40aa00700 1 -- 172.21.15.62:0/3300993859 <== mgr.14215 v2:172.21.15.62:6800/4116451241 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fa414103390 con 0x7fa3f806c610 2024-04-20T21:22:44.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:22:44.535+0000 7fa419200700 1 -- 172.21.15.62:0/3300993859 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fa3f806c610 msgr2=0x7fa3f806eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:22:44.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:22:44.535+0000 7fa419200700 1 --2- 172.21.15.62:0/3300993859 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fa3f806c610 0x7fa3f806eac0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa404006010 tx=0x7fa40400b540 comp rx=0 tx=0).stop 2024-04-20T21:22:44.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:22:44.535+0000 7fa419200700 1 -- 172.21.15.62:0/3300993859 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa414074ce0 msgr2=0x7fa41419c620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:22:44.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:22:44.535+0000 7fa419200700 1 --2- 172.21.15.62:0/3300993859 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa414074ce0 0x7fa41419c620 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fa40000d8d0 tx=0x7fa40000dc90 comp rx=0 tx=0).stop 2024-04-20T21:22:44.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:22:44.536+0000 7fa419200700 1 -- 172.21.15.62:0/3300993859 shutdown_connections 2024-04-20T21:22:44.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:22:44.536+0000 7fa419200700 1 --2- 172.21.15.62:0/3300993859 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fa3f806c610 0x7fa3f806eac0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:22:44.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:22:44.536+0000 7fa419200700 1 --2- 172.21.15.62:0/3300993859 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa414073720 0x7fa41419c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:22:44.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:22:44.536+0000 7fa419200700 1 --2- 172.21.15.62:0/3300993859 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa414074ce0 0x7fa41419c620 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:22:44.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:22:44.536+0000 7fa419200700 1 -- 172.21.15.62:0/3300993859 >> 172.21.15.62:0/3300993859 conn(0x7fa4140fb890 msgr2=0x7fa414101c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:22:44.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:22:44.536+0000 7fa419200700 1 -- 172.21.15.62:0/3300993859 shutdown_connections 2024-04-20T21:22:44.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:22:44.536+0000 7fa419200700 1 -- 172.21.15.62:0/3300993859 wait complete. 2024-04-20T21:22:44.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:44 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:22:44.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:44 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:22:44.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:44 smithi062 ceph-mon[16221]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:22:44.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:44 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:22:44.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:44 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:22:44.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:44 smithi062 ceph-mon[16221]: from='osd.0 [v2:172.21.15.62:6802/2687023571,v1:172.21.15.62:6803/2687023571]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-20T21:22:44.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:44 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:22:44.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:44 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:22:44.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:44 smithi189 ceph-mon[21562]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:22:44.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:44 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:22:44.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:44 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:22:44.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:44 smithi189 ceph-mon[21562]: from='osd.0 [v2:172.21.15.62:6802/2687023571,v1:172.21.15.62:6803/2687023571]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-20T21:22:45.225 DEBUG:teuthology.orchestra.run.smithi062:osd.0> sudo journalctl -f -n 0 -u ceph-61ae3628-ff5b-11ee-bc93-c7b262605968@osd.0.service 2024-04-20T21:22:45.230 INFO:tasks.cephadm:Deploying osd.1 on smithi062 with /dev/vg_nvme/lv_3... 2024-04-20T21:22:45.231 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-20T21:22:46.101 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T21:22:46.504 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:46 smithi062 ceph-mon[16221]: from='osd.0 [v2:172.21.15.62:6802/2687023571,v1:172.21.15.62:6803/2687023571]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-20T21:22:46.504 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:46 smithi062 ceph-mon[16221]: osdmap e7: 1 total, 0 up, 1 in 2024-04-20T21:22:46.504 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:46 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-20T21:22:46.504 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:46 smithi062 ceph-mon[16221]: from='osd.0 [v2:172.21.15.62:6802/2687023571,v1:172.21.15.62:6803/2687023571]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-04-20T21:22:46.504 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:46 smithi062 ceph-mon[16221]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:22:46.786 INFO:journalctl@ceph.osd.0.smithi062.stdout:Apr 20 21:22:46 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-0[31088]: 2024-04-20T21:22:46.503+0000 7faf0a600700 -1 osd.0 0 waiting for initial osdmap 2024-04-20T21:22:46.787 INFO:journalctl@ceph.osd.0.smithi062.stdout:Apr 20 21:22:46 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-0[31088]: 2024-04-20T21:22:46.515+0000 7faf04a00700 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-20T21:22:46.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:46 smithi189 ceph-mon[21562]: from='osd.0 [v2:172.21.15.62:6802/2687023571,v1:172.21.15.62:6803/2687023571]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-20T21:22:46.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:46 smithi189 ceph-mon[21562]: osdmap e7: 1 total, 0 up, 1 in 2024-04-20T21:22:46.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:46 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-20T21:22:46.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:46 smithi189 ceph-mon[21562]: from='osd.0 [v2:172.21.15.62:6802/2687023571,v1:172.21.15.62:6803/2687023571]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-04-20T21:22:46.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:46 smithi189 ceph-mon[21562]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:22:47.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:47 smithi062 ceph-mon[16221]: from='osd.0 [v2:172.21.15.62:6802/2687023571,v1:172.21.15.62:6803/2687023571]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]': finished 2024-04-20T21:22:47.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:47 smithi062 ceph-mon[16221]: osdmap e8: 1 total, 0 up, 1 in 2024-04-20T21:22:47.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:47 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-20T21:22:47.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:47 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-20T21:22:47.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:47 smithi189 ceph-mon[21562]: from='osd.0 [v2:172.21.15.62:6802/2687023571,v1:172.21.15.62:6803/2687023571]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]': finished 2024-04-20T21:22:47.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:47 smithi189 ceph-mon[21562]: osdmap e8: 1 total, 0 up, 1 in 2024-04-20T21:22:47.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:47 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-20T21:22:47.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:47 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-20T21:22:48.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:48 smithi062 ceph-mon[16221]: purged_snaps scrub starts 2024-04-20T21:22:48.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:48 smithi062 ceph-mon[16221]: purged_snaps scrub ok 2024-04-20T21:22:48.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:48 smithi062 ceph-mon[16221]: osd.0 [v2:172.21.15.62:6802/2687023571,v1:172.21.15.62:6803/2687023571] boot 2024-04-20T21:22:48.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:48 smithi062 ceph-mon[16221]: osdmap e9: 1 total, 1 up, 1 in 2024-04-20T21:22:48.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:48 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-20T21:22:48.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:48 smithi062 ceph-mon[16221]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:22:48.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:48 smithi189 ceph-mon[21562]: purged_snaps scrub starts 2024-04-20T21:22:48.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:48 smithi189 ceph-mon[21562]: purged_snaps scrub ok 2024-04-20T21:22:48.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:48 smithi189 ceph-mon[21562]: osd.0 [v2:172.21.15.62:6802/2687023571,v1:172.21.15.62:6803/2687023571] boot 2024-04-20T21:22:48.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:48 smithi189 ceph-mon[21562]: osdmap e9: 1 total, 1 up, 1 in 2024-04-20T21:22:48.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:48 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-20T21:22:48.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:48 smithi189 ceph-mon[21562]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T21:22:49.015 INFO:teuthology.orchestra.run.smithi062.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-20T21:22:49.015 INFO:teuthology.orchestra.run.smithi062.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-20T21:22:49.015 INFO:teuthology.orchestra.run.smithi062.stderr: stderr: 10+0 records in 2024-04-20T21:22:49.015 INFO:teuthology.orchestra.run.smithi062.stderr:10+0 records out 2024-04-20T21:22:49.015 INFO:teuthology.orchestra.run.smithi062.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0337327 s, 311 MB/s 2024-04-20T21:22:49.015 INFO:teuthology.orchestra.run.smithi062.stderr:--> Zapping successful for: 2024-04-20T21:22:49.779 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:49 smithi062 ceph-mon[16221]: osdmap e10: 1 total, 1 up, 1 in 2024-04-20T21:22:49.779 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:49 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:22:49.837 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -- ceph orch daemon add osd smithi062:vg_nvme/lv_3 2024-04-20T21:22:49.889 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:49 smithi189 ceph-mon[21562]: osdmap e10: 1 total, 1 up, 1 in 2024-04-20T21:22:49.889 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:49 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:22:50.525 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T21:22:50.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:50 smithi062 ceph-mon[16221]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-20T21:22:50.916 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:50 smithi189 ceph-mon[21562]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-20T21:22:51.854 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:51 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:22:51.854 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:51 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:22:51.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:51 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:22:51.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:51 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:22:52.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:22:52.599+0000 7f17a4200700 1 -- 172.21.15.62:0/1949764423 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f179c1036a0 msgr2=0x7f179c105a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:22:52.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:22:52.599+0000 7f17a4200700 1 --2- 172.21.15.62:0/1949764423 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f179c1036a0 0x7f179c105a80 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f1788009910 tx=0x7f1788009c20 comp rx=0 tx=0).stop 2024-04-20T21:22:52.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:22:52.600+0000 7f17a4200700 1 -- 172.21.15.62:0/1949764423 shutdown_connections 2024-04-20T21:22:52.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:22:52.600+0000 7f17a4200700 1 --2- 172.21.15.62:0/1949764423 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f179c1036a0 0x7f179c105a80 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:22:52.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:22:52.600+0000 7f17a4200700 1 --2- 172.21.15.62:0/1949764423 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f179c100d80 0x7f179c103160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:22:52.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:22:52.600+0000 7f17a4200700 1 -- 172.21.15.62:0/1949764423 >> 172.21.15.62:0/1949764423 conn(0x7f179c0fa7a0 msgr2=0x7f179c0fcbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:22:52.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:22:52.600+0000 7f17a4200700 1 -- 172.21.15.62:0/1949764423 shutdown_connections 2024-04-20T21:22:52.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:22:52.600+0000 7f17a4200700 1 -- 172.21.15.62:0/1949764423 wait complete. 2024-04-20T21:22:52.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:22:52.601+0000 7f17a4200700 1 Processor -- start 2024-04-20T21:22:52.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:22:52.602+0000 7f17a4200700 1 -- start start 2024-04-20T21:22:52.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:22:52.602+0000 7f17a4200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f179c100d80 0x7f179c1006e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:22:52.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:22:52.602+0000 7f17a4200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f179c1036a0 0x7f179c0fed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:22:52.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:22:52.602+0000 7f17a4200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f179c0ff270 con 0x7f179c1036a0 2024-04-20T21:22:52.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:22:52.602+0000 7f17a4200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f179c0ff3b0 con 0x7f179c100d80 2024-04-20T21:22:52.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:22:52.602+0000 7f17a0e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f179c1036a0 0x7f179c0fed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:22:52.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:22:52.602+0000 7f17a0e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f179c1036a0 0x7f179c0fed30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:38112/0 (socket says 172.21.15.62:38112) 2024-04-20T21:22:52.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:22:52.602+0000 7f17a0e00700 1 -- 172.21.15.62:0/1147008417 learned_addr learned my addr 172.21.15.62:0/1147008417 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:22:52.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:22:52.603+0000 7f17a1800700 1 --2- 172.21.15.62:0/1147008417 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f179c100d80 0x7f179c1006e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:22:52.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:22:52.603+0000 7f17a0e00700 1 -- 172.21.15.62:0/1147008417 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f179c100d80 msgr2=0x7f179c1006e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:22:52.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:22:52.603+0000 7f17a0e00700 1 --2- 172.21.15.62:0/1147008417 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f179c100d80 0x7f179c1006e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:22:52.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:22:52.603+0000 7f17a0e00700 1 -- 172.21.15.62:0/1147008417 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17880095f0 con 0x7f179c1036a0 2024-04-20T21:22:52.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:22:52.603+0000 7f17a0e00700 1 --2- 172.21.15.62:0/1147008417 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f179c1036a0 0x7f179c0fed30 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f1788000c00 tx=0x7f178800f740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:22:52.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:22:52.603+0000 7f179a000700 1 -- 172.21.15.62:0/1147008417 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f178801d070 con 0x7f179c1036a0 2024-04-20T21:22:52.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:22:52.604+0000 7f179a000700 1 -- 172.21.15.62:0/1147008417 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f1788005dc0 con 0x7f179c1036a0 2024-04-20T21:22:52.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:22:52.604+0000 7f17a4200700 1 -- 172.21.15.62:0/1147008417 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f179c0ff630 con 0x7f179c1036a0 2024-04-20T21:22:52.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:22:52.604+0000 7f179a000700 1 -- 172.21.15.62:0/1147008417 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f17880226a0 con 0x7f179c1036a0 2024-04-20T21:22:52.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:22:52.604+0000 7f17a4200700 1 -- 172.21.15.62:0/1147008417 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f179c0ffb20 con 0x7f179c1036a0 2024-04-20T21:22:52.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:22:52.605+0000 7f17a4200700 1 -- 172.21.15.62:0/1147008417 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f179c18fc00 con 0x7f179c1036a0 2024-04-20T21:22:52.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:22:52.607+0000 7f179a000700 1 -- 172.21.15.62:0/1147008417 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) v1 ==== 89858+0+0 (secure 0 0 0) 0x7f1788003680 con 0x7f179c1036a0 2024-04-20T21:22:52.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:22:52.608+0000 7f179a000700 1 --2- 172.21.15.62:0/1147008417 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f178406c610 0x7f178406eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:22:52.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:22:52.608+0000 7f179a000700 1 -- 172.21.15.62:0/1147008417 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1982+0+0 (secure 0 0 0) 0x7f178808bfc0 con 0x7f179c1036a0 2024-04-20T21:22:52.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:22:52.609+0000 7f17a1800700 1 --2- 172.21.15.62:0/1147008417 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f178406c610 0x7f178406eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:22:52.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:22:52.610+0000 7f17a1800700 1 --2- 172.21.15.62:0/1147008417 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f178406c610 0x7f178406eac0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f179c069890 tx=0x7f1790009260 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:22:52.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:22:52.613+0000 7f179a000700 1 -- 172.21.15.62:0/1147008417 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f178805b010 con 0x7f179c1036a0 2024-04-20T21:22:52.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:52 smithi189 ceph-mon[21562]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-20T21:22:52.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:22:52.857+0000 7f17a4200700 1 -- 172.21.15.62:0/1147008417 --> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi062:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f179c0008d0 con 0x7f178406c610 2024-04-20T21:22:52.938 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:52 smithi062 ceph-mon[16221]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-20T21:22:53.751 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:53 smithi062 ceph-mon[16221]: from='client.14272 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi062:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:22:53.751 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:53 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-20T21:22:53.751 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:53 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-20T21:22:53.751 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:53 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:22:53.990 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:53 smithi189 ceph-mon[21562]: from='client.14272 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi062:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:22:53.990 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:53 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-20T21:22:53.990 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:53 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-20T21:22:53.990 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:53 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:22:54.984 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:54 smithi062 ceph-mon[16221]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-20T21:22:54.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:54 smithi189 ceph-mon[21562]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-20T21:22:57.008 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:56 smithi062 ceph-mon[16221]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-20T21:22:57.008 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:56 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:22:57.008 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:56 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:22:57.008 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:56 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-20T21:22:57.008 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:56 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:22:57.008 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:56 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:22:57.008 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:56 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T21:22:57.008 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:56 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:22:57.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:56 smithi189 ceph-mon[21562]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-20T21:22:57.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:56 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:22:57.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:56 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:22:57.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:56 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-20T21:22:57.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:56 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:22:57.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:56 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:22:57.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:56 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T21:22:57.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:56 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:22:58.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:57 smithi062 ceph-mon[16221]: Detected new or changed devices on smithi062 2024-04-20T21:22:58.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:57 smithi062 ceph-mon[16221]: Adjusting osd_memory_target on smithi062 to 11841M 2024-04-20T21:22:58.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:57 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:22:58.216 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:57 smithi189 ceph-mon[21562]: Detected new or changed devices on smithi062 2024-04-20T21:22:58.216 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:57 smithi189 ceph-mon[21562]: Adjusting osd_memory_target on smithi062 to 11841M 2024-04-20T21:22:58.216 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:57 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:22:59.158 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:58 smithi189 ceph-mon[21562]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-20T21:22:59.158 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:58 smithi189 ceph-mon[21562]: from='client.? 172.21.15.62:0/2214693853' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f0c451a4-9002-4ddc-be0e-d8ddebd4418d"}]: dispatch 2024-04-20T21:22:59.158 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:58 smithi189 ceph-mon[21562]: from='client.? 172.21.15.62:0/2214693853' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f0c451a4-9002-4ddc-be0e-d8ddebd4418d"}]': finished 2024-04-20T21:22:59.158 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:58 smithi189 ceph-mon[21562]: osdmap e11: 2 total, 1 up, 2 in 2024-04-20T21:22:59.158 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:58 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-20T21:22:59.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:58 smithi062 ceph-mon[16221]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-20T21:22:59.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:58 smithi062 ceph-mon[16221]: from='client.? 172.21.15.62:0/2214693853' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f0c451a4-9002-4ddc-be0e-d8ddebd4418d"}]: dispatch 2024-04-20T21:22:59.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:58 smithi062 ceph-mon[16221]: from='client.? 172.21.15.62:0/2214693853' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f0c451a4-9002-4ddc-be0e-d8ddebd4418d"}]': finished 2024-04-20T21:22:59.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:58 smithi062 ceph-mon[16221]: osdmap e11: 2 total, 1 up, 2 in 2024-04-20T21:22:59.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:58 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-20T21:23:00.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:59 smithi189 ceph-mon[21562]: from='client.? 172.21.15.62:0/2150804198' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-20T21:23:00.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:22:59 smithi189 ceph-mon[21562]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-20T21:23:00.260 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:59 smithi062 ceph-mon[16221]: from='client.? 172.21.15.62:0/2150804198' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-20T21:23:00.261 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:22:59 smithi062 ceph-mon[16221]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-20T21:23:01.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:00 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:23:01.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:00 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:23:01.282 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:00 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:23:01.282 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:00 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:23:02.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:02 smithi189 ceph-mon[21562]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-20T21:23:03.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:02 smithi062 ceph-mon[16221]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-20T21:23:04.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:03 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:23:04.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:03 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:23:04.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:03 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:23:04.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:03 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T21:23:04.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:03 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:23:04.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:03 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:23:04.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:03 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:23:04.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:03 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:23:04.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:03 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T21:23:04.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:03 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:23:04.954 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:04 smithi062 ceph-mon[16221]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-20T21:23:04.955 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:04 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:23:05.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:04 smithi189 ceph-mon[21562]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-20T21:23:05.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:04 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:23:07.188 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:06 smithi062 ceph-mon[16221]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-20T21:23:07.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:06 smithi189 ceph-mon[21562]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-20T21:23:08.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:07 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-20T21:23:08.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:07 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:23:08.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:07 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-20T21:23:08.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:07 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:23:08.979 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:08 smithi062 ceph-mon[16221]: Deploying daemon osd.1 on smithi062 2024-04-20T21:23:08.980 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:08 smithi062 ceph-mon[16221]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-20T21:23:09.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:08 smithi189 ceph-mon[21562]: Deploying daemon osd.1 on smithi062 2024-04-20T21:23:09.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:08 smithi189 ceph-mon[21562]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-20T21:23:10.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:09 smithi189 ceph-mon[21562]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-20T21:23:10.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:09 smithi062 ceph-mon[16221]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-20T21:23:12.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:12 smithi189 ceph-mon[21562]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-20T21:23:13.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:12 smithi062 ceph-mon[16221]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-20T21:23:15.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:14 smithi062 ceph-mon[16221]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-20T21:23:15.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:14 smithi189 ceph-mon[21562]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-20T21:23:16.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:15 smithi189 ceph-mon[21562]: pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-20T21:23:16.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:15 smithi062 ceph-mon[16221]: pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-20T21:23:17.203 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:17 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:23:17.204 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:17 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:23:17.204 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:17 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:23:17.362 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:17 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:23:17.362 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:17 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:23:17.362 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:17 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:23:18.380 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:18 smithi189 ceph-mon[21562]: pgmap v66: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-20T21:23:18.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:18 smithi062 ceph-mon[16221]: pgmap v66: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-20T21:23:19.257 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:19 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:23:19.257 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:19 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:23:19.258 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:19 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:23:19.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:19 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:23:19.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:19 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:23:19.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:19 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:23:20.177 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:20 smithi189 ceph-mon[21562]: pgmap v67: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-20T21:23:20.177 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:20 smithi189 ceph-mon[21562]: from='osd.1 [v2:172.21.15.62:6810/1027109374,v1:172.21.15.62:6811/1027109374]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-20T21:23:20.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:20 smithi062 ceph-mon[16221]: pgmap v67: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-20T21:23:20.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:20 smithi062 ceph-mon[16221]: from='osd.1 [v2:172.21.15.62:6810/1027109374,v1:172.21.15.62:6811/1027109374]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-20T21:23:21.303 INFO:teuthology.orchestra.run.smithi062.stdout:Created osd(s) 1 on host 'smithi062' 2024-04-20T21:23:21.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:23:21.302+0000 7f179a000700 1 -- 172.21.15.62:0/1147008417 <== mgr.14215 v2:172.21.15.62:6800/4116451241 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f179c0008d0 con 0x7f178406c610 2024-04-20T21:23:21.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:23:21.307+0000 7f17a4200700 1 -- 172.21.15.62:0/1147008417 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f178406c610 msgr2=0x7f178406eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:23:21.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:23:21.307+0000 7f17a4200700 1 --2- 172.21.15.62:0/1147008417 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f178406c610 0x7f178406eac0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f179c069890 tx=0x7f1790009260 comp rx=0 tx=0).stop 2024-04-20T21:23:21.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:23:21.307+0000 7f17a4200700 1 -- 172.21.15.62:0/1147008417 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f179c1036a0 msgr2=0x7f179c0fed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:23:21.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:23:21.307+0000 7f17a4200700 1 --2- 172.21.15.62:0/1147008417 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f179c1036a0 0x7f179c0fed30 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f1788000c00 tx=0x7f178800f740 comp rx=0 tx=0).stop 2024-04-20T21:23:21.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:23:21.307+0000 7f17a4200700 1 -- 172.21.15.62:0/1147008417 shutdown_connections 2024-04-20T21:23:21.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:23:21.307+0000 7f17a4200700 1 --2- 172.21.15.62:0/1147008417 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f178406c610 0x7f178406eac0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:23:21.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:23:21.307+0000 7f17a4200700 1 --2- 172.21.15.62:0/1147008417 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f179c100d80 0x7f179c1006e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:23:21.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:23:21.307+0000 7f17a4200700 1 --2- 172.21.15.62:0/1147008417 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f179c1036a0 0x7f179c0fed30 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:23:21.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:23:21.307+0000 7f17a4200700 1 -- 172.21.15.62:0/1147008417 >> 172.21.15.62:0/1147008417 conn(0x7f179c0fa7a0 msgr2=0x7f179c0fcbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:23:21.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:23:21.307+0000 7f17a4200700 1 -- 172.21.15.62:0/1147008417 shutdown_connections 2024-04-20T21:23:21.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:23:21.307+0000 7f17a4200700 1 -- 172.21.15.62:0/1147008417 wait complete. 2024-04-20T21:23:21.433 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:21 smithi062 ceph-mon[16221]: from='osd.1 [v2:172.21.15.62:6810/1027109374,v1:172.21.15.62:6811/1027109374]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-20T21:23:21.433 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:21 smithi062 ceph-mon[16221]: osdmap e12: 2 total, 1 up, 2 in 2024-04-20T21:23:21.434 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:21 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-20T21:23:21.434 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:21 smithi062 ceph-mon[16221]: from='osd.1 [v2:172.21.15.62:6810/1027109374,v1:172.21.15.62:6811/1027109374]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-04-20T21:23:21.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:21 smithi189 ceph-mon[21562]: from='osd.1 [v2:172.21.15.62:6810/1027109374,v1:172.21.15.62:6811/1027109374]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-20T21:23:21.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:21 smithi189 ceph-mon[21562]: osdmap e12: 2 total, 1 up, 2 in 2024-04-20T21:23:21.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:21 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-20T21:23:21.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:21 smithi189 ceph-mon[21562]: from='osd.1 [v2:172.21.15.62:6810/1027109374,v1:172.21.15.62:6811/1027109374]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-04-20T21:23:22.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:22 smithi062 ceph-mon[16221]: from='osd.1 [v2:172.21.15.62:6810/1027109374,v1:172.21.15.62:6811/1027109374]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]': finished 2024-04-20T21:23:22.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:22 smithi062 ceph-mon[16221]: osdmap e13: 2 total, 1 up, 2 in 2024-04-20T21:23:22.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:22 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-20T21:23:22.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:22 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-20T21:23:22.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:22 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:23:22.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:22 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:23:22.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:22 smithi062 ceph-mon[16221]: pgmap v70: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-20T21:23:22.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:22 smithi189 ceph-mon[21562]: from='osd.1 [v2:172.21.15.62:6810/1027109374,v1:172.21.15.62:6811/1027109374]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]': finished 2024-04-20T21:23:22.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:22 smithi189 ceph-mon[21562]: osdmap e13: 2 total, 1 up, 2 in 2024-04-20T21:23:22.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:22 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-20T21:23:22.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:22 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-20T21:23:22.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:22 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:23:22.489 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:22 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:23:22.489 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:22 smithi189 ceph-mon[21562]: pgmap v70: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-20T21:23:22.584 DEBUG:teuthology.orchestra.run.smithi062:osd.1> sudo journalctl -f -n 0 -u ceph-61ae3628-ff5b-11ee-bc93-c7b262605968@osd.1.service 2024-04-20T21:23:22.591 INFO:tasks.cephadm:Deploying osd.2 on smithi062 with /dev/vg_nvme/lv_2... 2024-04-20T21:23:22.592 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-20T21:23:23.075 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T21:23:23.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:23 smithi189 ceph-mon[21562]: purged_snaps scrub starts 2024-04-20T21:23:23.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:23 smithi189 ceph-mon[21562]: purged_snaps scrub ok 2024-04-20T21:23:23.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:23 smithi189 ceph-mon[21562]: osd.1 [v2:172.21.15.62:6810/1027109374,v1:172.21.15.62:6811/1027109374] boot 2024-04-20T21:23:23.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:23 smithi189 ceph-mon[21562]: osdmap e14: 2 total, 2 up, 2 in 2024-04-20T21:23:23.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:23 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-20T21:23:23.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:23 smithi062 ceph-mon[16221]: purged_snaps scrub starts 2024-04-20T21:23:23.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:23 smithi062 ceph-mon[16221]: purged_snaps scrub ok 2024-04-20T21:23:23.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:23 smithi062 ceph-mon[16221]: osd.1 [v2:172.21.15.62:6810/1027109374,v1:172.21.15.62:6811/1027109374] boot 2024-04-20T21:23:23.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:23 smithi062 ceph-mon[16221]: osdmap e14: 2 total, 2 up, 2 in 2024-04-20T21:23:23.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:23 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-20T21:23:24.726 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:24 smithi062 ceph-mon[16221]: osdmap e15: 2 total, 2 up, 2 in 2024-04-20T21:23:24.727 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:24 smithi062 ceph-mon[16221]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-20T21:23:24.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:24 smithi189 ceph-mon[21562]: osdmap e15: 2 total, 2 up, 2 in 2024-04-20T21:23:24.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:24 smithi189 ceph-mon[21562]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-20T21:23:26.885 INFO:teuthology.orchestra.run.smithi062.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-20T21:23:26.886 INFO:teuthology.orchestra.run.smithi062.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-20T21:23:26.886 INFO:teuthology.orchestra.run.smithi062.stderr: stderr: 10+0 records in 2024-04-20T21:23:26.886 INFO:teuthology.orchestra.run.smithi062.stderr:10+0 records out 2024-04-20T21:23:26.886 INFO:teuthology.orchestra.run.smithi062.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0588043 s, 178 MB/s 2024-04-20T21:23:26.886 INFO:teuthology.orchestra.run.smithi062.stderr:--> Zapping successful for: 2024-04-20T21:23:26.924 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:26 smithi062 ceph-mon[16221]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-20T21:23:26.924 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:26 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:23:26.924 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:26 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:23:27.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:26 smithi189 ceph-mon[21562]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-20T21:23:27.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:26 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:23:27.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:26 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:23:27.792 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -- ceph orch daemon add osd smithi062:vg_nvme/lv_2 2024-04-20T21:23:28.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:27 smithi062 ceph-mon[16221]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-20T21:23:28.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:27 smithi189 ceph-mon[21562]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-20T21:23:29.250 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T21:23:30.906 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:30 smithi062 ceph-mon[16221]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-20T21:23:31.198 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:30 smithi189 ceph-mon[21562]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-20T21:23:32.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:23:32.516+0000 7ff4b5000700 1 -- 172.21.15.62:0/2814420674 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff4b0074d20 msgr2=0x7ff4b0073180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:23:32.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:23:32.516+0000 7ff4b5000700 1 --2- 172.21.15.62:0/2814420674 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff4b0074d20 0x7ff4b0073180 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7ff4a0009910 tx=0x7ff4a0009c20 comp rx=0 tx=0).stop 2024-04-20T21:23:32.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:23:32.516+0000 7ff4b5000700 1 -- 172.21.15.62:0/2814420674 shutdown_connections 2024-04-20T21:23:32.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:23:32.516+0000 7ff4b5000700 1 --2- 172.21.15.62:0/2814420674 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4b0073750 0x7ff4b0073bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:23:32.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:23:32.516+0000 7ff4b5000700 1 --2- 172.21.15.62:0/2814420674 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff4b0074d20 0x7ff4b0073180 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:23:32.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:23:32.516+0000 7ff4b5000700 1 -- 172.21.15.62:0/2814420674 >> 172.21.15.62:0/2814420674 conn(0x7ff4b00fb7f0 msgr2=0x7ff4b00fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:23:32.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:23:32.516+0000 7ff4b5000700 1 -- 172.21.15.62:0/2814420674 shutdown_connections 2024-04-20T21:23:32.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:23:32.517+0000 7ff4b5000700 1 -- 172.21.15.62:0/2814420674 wait complete. 2024-04-20T21:23:32.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:23:32.518+0000 7ff4b5000700 1 Processor -- start 2024-04-20T21:23:32.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:23:32.519+0000 7ff4b5000700 1 -- start start 2024-04-20T21:23:32.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:23:32.519+0000 7ff4b5000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff4b0073750 0x7ff4b0197ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:23:32.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:23:32.519+0000 7ff4b5000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4b0074d20 0x7ff4b0198220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:23:32.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:23:32.519+0000 7ff4b5000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4b0198840 con 0x7ff4b0073750 2024-04-20T21:23:32.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:23:32.519+0000 7ff4b5000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4b019d250 con 0x7ff4b0074d20 2024-04-20T21:23:32.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:23:32.519+0000 7ff4ade00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff4b0073750 0x7ff4b0197ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:23:32.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:23:32.519+0000 7ff4ade00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff4b0073750 0x7ff4b0197ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56234/0 (socket says 172.21.15.62:56234) 2024-04-20T21:23:32.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:23:32.519+0000 7ff4ade00700 1 -- 172.21.15.62:0/1366874725 learned_addr learned my addr 172.21.15.62:0/1366874725 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:23:32.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:23:32.520+0000 7ff4a5400700 1 --2- 172.21.15.62:0/1366874725 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4b0074d20 0x7ff4b0198220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:23:32.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:23:32.520+0000 7ff4ade00700 1 -- 172.21.15.62:0/1366874725 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4b0074d20 msgr2=0x7ff4b0198220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:23:32.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:23:32.520+0000 7ff4ade00700 1 --2- 172.21.15.62:0/1366874725 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4b0074d20 0x7ff4b0198220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:23:32.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:23:32.520+0000 7ff4ade00700 1 -- 172.21.15.62:0/1366874725 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4a00095f0 con 0x7ff4b0073750 2024-04-20T21:23:32.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:23:32.520+0000 7ff4ade00700 1 --2- 172.21.15.62:0/1366874725 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff4b0073750 0x7ff4b0197ce0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7ff4a0000c00 tx=0x7ff4a000f740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:23:32.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:23:32.522+0000 7ff4a7400700 1 -- 172.21.15.62:0/1366874725 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff4a001d070 con 0x7ff4b0073750 2024-04-20T21:23:32.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:23:32.522+0000 7ff4b5000700 1 -- 172.21.15.62:0/1366874725 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4b019d3f0 con 0x7ff4b0073750 2024-04-20T21:23:32.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:23:32.522+0000 7ff4a7400700 1 -- 172.21.15.62:0/1366874725 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff4a0005dc0 con 0x7ff4b0073750 2024-04-20T21:23:32.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:23:32.522+0000 7ff4a7400700 1 -- 172.21.15.62:0/1366874725 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff4a00226a0 con 0x7ff4b0073750 2024-04-20T21:23:32.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:23:32.522+0000 7ff4b5000700 1 -- 172.21.15.62:0/1366874725 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4b019d8b0 con 0x7ff4b0073750 2024-04-20T21:23:32.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:23:32.524+0000 7ff4b5000700 1 -- 172.21.15.62:0/1366874725 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4b010af80 con 0x7ff4b0073750 2024-04-20T21:23:32.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:23:32.524+0000 7ff4a7400700 1 -- 172.21.15.62:0/1366874725 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) v1 ==== 89858+0+0 (secure 0 0 0) 0x7ff4a0003680 con 0x7ff4b0073750 2024-04-20T21:23:32.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:23:32.525+0000 7ff4a7400700 1 --2- 172.21.15.62:0/1366874725 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7ff49406c600 0x7ff49406eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:23:32.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:23:32.526+0000 7ff4a7400700 1 -- 172.21.15.62:0/1366874725 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 2414+0+0 (secure 0 0 0) 0x7ff4a008c020 con 0x7ff4b0073750 2024-04-20T21:23:32.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:23:32.526+0000 7ff4a5400700 1 --2- 172.21.15.62:0/1366874725 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7ff49406c600 0x7ff49406eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:23:32.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:23:32.527+0000 7ff4a5400700 1 --2- 172.21.15.62:0/1366874725 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7ff49406c600 0x7ff49406eab0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff498009bf0 tx=0x7ff498009260 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:23:32.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:23:32.532+0000 7ff4a7400700 1 -- 172.21.15.62:0/1366874725 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7ff4a0057450 con 0x7ff4b0073750 2024-04-20T21:23:32.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:23:32.773+0000 7ff4b5000700 1 -- 172.21.15.62:0/1366874725 --> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi062:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7ff4b019df10 con 0x7ff49406c600 2024-04-20T21:23:32.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:32 smithi189 ceph-mon[21562]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-20T21:23:32.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:32 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:23:32.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:32 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:23:32.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:32 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-20T21:23:32.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:32 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-20T21:23:32.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:32 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:23:32.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:32 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:23:32.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:32 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T21:23:32.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:32 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:23:32.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:32 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:23:33.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:32 smithi062 ceph-mon[16221]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-20T21:23:33.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:32 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:23:33.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:32 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:23:33.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:32 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-20T21:23:33.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:32 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-20T21:23:33.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:32 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:23:33.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:32 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:23:33.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:32 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T21:23:33.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:32 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:23:33.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:32 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:23:33.968 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:33 smithi062 ceph-mon[16221]: Detected new or changed devices on smithi062 2024-04-20T21:23:33.969 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:33 smithi062 ceph-mon[16221]: Adjusting osd_memory_target on smithi062 to 5920M 2024-04-20T21:23:33.969 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:33 smithi062 ceph-mon[16221]: from='client.14290 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi062:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:23:33.969 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:33 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-20T21:23:33.969 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:33 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-20T21:23:33.969 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:33 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:23:33.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:33 smithi189 ceph-mon[21562]: Detected new or changed devices on smithi062 2024-04-20T21:23:33.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:33 smithi189 ceph-mon[21562]: Adjusting osd_memory_target on smithi062 to 5920M 2024-04-20T21:23:33.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:33 smithi189 ceph-mon[21562]: from='client.14290 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi062:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:23:33.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:33 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-20T21:23:33.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:33 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-20T21:23:33.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:33 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:23:34.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:34 smithi189 ceph-mon[21562]: pgmap v78: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-20T21:23:34.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:34 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:23:35.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:34 smithi062 ceph-mon[16221]: pgmap v78: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-20T21:23:35.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:34 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:23:36.464 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:36 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:23:36.464 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:36 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:23:36.464 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:36 smithi062 ceph-mon[16221]: pgmap v79: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-20T21:23:36.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:36 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:23:36.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:36 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:23:36.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:36 smithi189 ceph-mon[21562]: pgmap v79: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-20T21:23:37.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:37 smithi189 ceph-mon[21562]: from='client.? 172.21.15.62:0/1773036288' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e1a8769d-5da2-49c8-942a-83a94c7ac761"}]: dispatch 2024-04-20T21:23:37.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:37 smithi189 ceph-mon[21562]: from='client.? 172.21.15.62:0/1773036288' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e1a8769d-5da2-49c8-942a-83a94c7ac761"}]': finished 2024-04-20T21:23:37.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:37 smithi189 ceph-mon[21562]: osdmap e16: 3 total, 2 up, 3 in 2024-04-20T21:23:37.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:37 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-20T21:23:38.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:37 smithi062 ceph-mon[16221]: from='client.? 172.21.15.62:0/1773036288' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e1a8769d-5da2-49c8-942a-83a94c7ac761"}]: dispatch 2024-04-20T21:23:38.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:37 smithi062 ceph-mon[16221]: from='client.? 172.21.15.62:0/1773036288' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e1a8769d-5da2-49c8-942a-83a94c7ac761"}]': finished 2024-04-20T21:23:38.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:37 smithi062 ceph-mon[16221]: osdmap e16: 3 total, 2 up, 3 in 2024-04-20T21:23:38.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:37 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-20T21:23:38.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:38 smithi062 ceph-mon[16221]: pgmap v81: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-20T21:23:38.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:38 smithi062 ceph-mon[16221]: from='client.? 172.21.15.62:0/3343625181' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-20T21:23:38.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:38 smithi189 ceph-mon[21562]: pgmap v81: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-20T21:23:38.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:38 smithi189 ceph-mon[21562]: from='client.? 172.21.15.62:0/3343625181' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-20T21:23:41.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:40 smithi189 ceph-mon[21562]: pgmap v82: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-20T21:23:41.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:40 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:23:41.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:40 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:23:41.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:40 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:23:41.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:40 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T21:23:41.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:40 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:23:41.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:40 smithi062 ceph-mon[16221]: pgmap v82: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-20T21:23:41.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:40 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:23:41.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:40 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:23:41.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:40 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:23:41.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:40 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T21:23:41.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:40 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:23:42.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:42 smithi189 ceph-mon[21562]: pgmap v83: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-20T21:23:43.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:42 smithi062 ceph-mon[16221]: pgmap v83: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-20T21:23:45.150 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:44 smithi062 ceph-mon[16221]: pgmap v84: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-20T21:23:45.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:44 smithi189 ceph-mon[21562]: pgmap v84: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-20T21:23:46.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:45 smithi189 ceph-mon[21562]: pgmap v85: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-20T21:23:46.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:45 smithi062 ceph-mon[16221]: pgmap v85: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-20T21:23:47.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:47 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-20T21:23:47.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:47 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:23:48.014 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:47 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-20T21:23:48.014 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:47 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:23:48.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:48 smithi189 ceph-mon[21562]: Deploying daemon osd.2 on smithi062 2024-04-20T21:23:48.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:48 smithi189 ceph-mon[21562]: pgmap v86: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-20T21:23:49.016 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:48 smithi062 ceph-mon[16221]: Deploying daemon osd.2 on smithi062 2024-04-20T21:23:49.016 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:48 smithi062 ceph-mon[16221]: pgmap v86: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-20T21:23:49.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:49 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:23:49.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:49 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:23:50.956 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:50 smithi062 ceph-mon[16221]: pgmap v87: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-20T21:23:50.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:50 smithi189 ceph-mon[21562]: pgmap v87: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-20T21:23:52.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:52 smithi189 ceph-mon[21562]: pgmap v88: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-20T21:23:53.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:52 smithi062 ceph-mon[16221]: pgmap v88: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-20T21:23:55.213 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:54 smithi062 ceph-mon[16221]: pgmap v89: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-20T21:23:55.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:54 smithi189 ceph-mon[21562]: pgmap v89: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-20T21:23:56.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:55 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:23:56.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:55 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:23:56.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:55 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:23:56.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:55 smithi189 ceph-mon[21562]: pgmap v90: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-20T21:23:56.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:55 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:23:56.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:55 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:23:56.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:55 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:23:56.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:55 smithi062 ceph-mon[16221]: pgmap v90: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-20T21:23:59.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:58 smithi062 ceph-mon[16221]: pgmap v91: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-20T21:23:59.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:58 smithi189 ceph-mon[21562]: pgmap v91: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-20T21:24:00.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:23:59 smithi062 ceph-mon[16221]: from='osd.2 [v2:172.21.15.62:6818/47205739,v1:172.21.15.62:6819/47205739]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-20T21:24:00.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:23:59 smithi189 ceph-mon[21562]: from='osd.2 [v2:172.21.15.62:6818/47205739,v1:172.21.15.62:6819/47205739]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-20T21:24:00.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:24:00.469+0000 7ff4a7400700 1 -- 172.21.15.62:0/1366874725 <== mgr.14215 v2:172.21.15.62:6800/4116451241 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7ff4b019df10 con 0x7ff49406c600 2024-04-20T21:24:00.468 INFO:teuthology.orchestra.run.smithi062.stdout:Created osd(s) 2 on host 'smithi062' 2024-04-20T21:24:00.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:24:00.472+0000 7ff4b5000700 1 -- 172.21.15.62:0/1366874725 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7ff49406c600 msgr2=0x7ff49406eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:24:00.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:24:00.472+0000 7ff4b5000700 1 --2- 172.21.15.62:0/1366874725 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7ff49406c600 0x7ff49406eab0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff498009bf0 tx=0x7ff498009260 comp rx=0 tx=0).stop 2024-04-20T21:24:00.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:24:00.472+0000 7ff4b5000700 1 -- 172.21.15.62:0/1366874725 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff4b0073750 msgr2=0x7ff4b0197ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:24:00.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:24:00.472+0000 7ff4b5000700 1 --2- 172.21.15.62:0/1366874725 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff4b0073750 0x7ff4b0197ce0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7ff4a0000c00 tx=0x7ff4a000f740 comp rx=0 tx=0).stop 2024-04-20T21:24:00.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:24:00.473+0000 7ff4b5000700 1 -- 172.21.15.62:0/1366874725 shutdown_connections 2024-04-20T21:24:00.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:24:00.473+0000 7ff4b5000700 1 --2- 172.21.15.62:0/1366874725 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7ff49406c600 0x7ff49406eab0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:24:00.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:24:00.473+0000 7ff4b5000700 1 --2- 172.21.15.62:0/1366874725 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff4b0073750 0x7ff4b0197ce0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:24:00.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:24:00.473+0000 7ff4b5000700 1 --2- 172.21.15.62:0/1366874725 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4b0074d20 0x7ff4b0198220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:24:00.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:24:00.473+0000 7ff4b5000700 1 -- 172.21.15.62:0/1366874725 >> 172.21.15.62:0/1366874725 conn(0x7ff4b00fb7f0 msgr2=0x7ff4b0106080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:24:00.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:24:00.473+0000 7ff4b5000700 1 -- 172.21.15.62:0/1366874725 shutdown_connections 2024-04-20T21:24:00.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:24:00.473+0000 7ff4b5000700 1 -- 172.21.15.62:0/1366874725 wait complete. 2024-04-20T21:24:01.204 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:00 smithi062 ceph-mon[16221]: pgmap v92: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-20T21:24:01.204 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:00 smithi062 ceph-mon[16221]: from='osd.2 [v2:172.21.15.62:6818/47205739,v1:172.21.15.62:6819/47205739]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-20T21:24:01.204 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:00 smithi062 ceph-mon[16221]: osdmap e17: 3 total, 2 up, 3 in 2024-04-20T21:24:01.204 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:00 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-20T21:24:01.204 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:00 smithi062 ceph-mon[16221]: from='osd.2 [v2:172.21.15.62:6818/47205739,v1:172.21.15.62:6819/47205739]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-04-20T21:24:01.204 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:00 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:24:01.204 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:00 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:24:01.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:00 smithi189 ceph-mon[21562]: pgmap v92: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-20T21:24:01.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:00 smithi189 ceph-mon[21562]: from='osd.2 [v2:172.21.15.62:6818/47205739,v1:172.21.15.62:6819/47205739]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-20T21:24:01.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:00 smithi189 ceph-mon[21562]: osdmap e17: 3 total, 2 up, 3 in 2024-04-20T21:24:01.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:00 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-20T21:24:01.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:00 smithi189 ceph-mon[21562]: from='osd.2 [v2:172.21.15.62:6818/47205739,v1:172.21.15.62:6819/47205739]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-04-20T21:24:01.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:00 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:24:01.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:00 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:24:01.258 DEBUG:teuthology.orchestra.run.smithi062:osd.2> sudo journalctl -f -n 0 -u ceph-61ae3628-ff5b-11ee-bc93-c7b262605968@osd.2.service 2024-04-20T21:24:01.263 INFO:tasks.cephadm:Deploying osd.3 on smithi189 with /dev/vg_nvme/lv_4... 2024-04-20T21:24:01.263 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-20T21:24:01.652 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi189/config 2024-04-20T21:24:02.088 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:01 smithi189 ceph-mon[21562]: from='osd.2 [v2:172.21.15.62:6818/47205739,v1:172.21.15.62:6819/47205739]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]': finished 2024-04-20T21:24:02.088 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:01 smithi189 ceph-mon[21562]: osdmap e18: 3 total, 2 up, 3 in 2024-04-20T21:24:02.088 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:01 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-20T21:24:02.088 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:01 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-20T21:24:02.088 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:01 smithi189 ceph-mon[21562]: pgmap v95: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-20T21:24:02.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:01 smithi062 ceph-mon[16221]: from='osd.2 [v2:172.21.15.62:6818/47205739,v1:172.21.15.62:6819/47205739]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]': finished 2024-04-20T21:24:02.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:01 smithi062 ceph-mon[16221]: osdmap e18: 3 total, 2 up, 3 in 2024-04-20T21:24:02.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:01 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-20T21:24:02.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:01 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-20T21:24:02.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:01 smithi062 ceph-mon[16221]: pgmap v95: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-20T21:24:03.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:02 smithi189 ceph-mon[21562]: purged_snaps scrub starts 2024-04-20T21:24:03.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:02 smithi189 ceph-mon[21562]: purged_snaps scrub ok 2024-04-20T21:24:03.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:02 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-20T21:24:03.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:02 smithi189 ceph-mon[21562]: osd.2 [v2:172.21.15.62:6818/47205739,v1:172.21.15.62:6819/47205739] boot 2024-04-20T21:24:03.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:02 smithi189 ceph-mon[21562]: osdmap e19: 3 total, 3 up, 3 in 2024-04-20T21:24:03.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:02 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-20T21:24:03.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:02 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-20T21:24:03.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:02 smithi062 ceph-mon[16221]: purged_snaps scrub starts 2024-04-20T21:24:03.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:02 smithi062 ceph-mon[16221]: purged_snaps scrub ok 2024-04-20T21:24:03.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:02 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-20T21:24:03.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:02 smithi062 ceph-mon[16221]: osd.2 [v2:172.21.15.62:6818/47205739,v1:172.21.15.62:6819/47205739] boot 2024-04-20T21:24:03.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:02 smithi062 ceph-mon[16221]: osdmap e19: 3 total, 3 up, 3 in 2024-04-20T21:24:03.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:02 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-20T21:24:03.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:02 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-20T21:24:04.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:03 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-20T21:24:04.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:03 smithi189 ceph-mon[21562]: osdmap e20: 3 total, 3 up, 3 in 2024-04-20T21:24:04.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:03 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-20T21:24:04.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:03 smithi189 ceph-mon[21562]: pgmap v98: 1 pgs: 1 creating+peering; 0 B data, 79 MiB used, 268 GiB / 268 GiB avail 2024-04-20T21:24:04.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:03 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:24:04.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:03 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-20T21:24:04.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:03 smithi062 ceph-mon[16221]: osdmap e20: 3 total, 3 up, 3 in 2024-04-20T21:24:04.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:03 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-20T21:24:04.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:03 smithi062 ceph-mon[16221]: pgmap v98: 1 pgs: 1 creating+peering; 0 B data, 79 MiB used, 268 GiB / 268 GiB avail 2024-04-20T21:24:04.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:03 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:24:04.303 INFO:teuthology.orchestra.run.smithi189.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-20T21:24:04.303 INFO:teuthology.orchestra.run.smithi189.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-20T21:24:04.303 INFO:teuthology.orchestra.run.smithi189.stderr: stderr: 10+0 records in 2024-04-20T21:24:04.303 INFO:teuthology.orchestra.run.smithi189.stderr:10+0 records out 2024-04-20T21:24:04.303 INFO:teuthology.orchestra.run.smithi189.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0326253 s, 321 MB/s 2024-04-20T21:24:04.303 INFO:teuthology.orchestra.run.smithi189.stderr:--> Zapping successful for: 2024-04-20T21:24:05.066 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -- ceph orch daemon add osd smithi189:vg_nvme/lv_4 2024-04-20T21:24:05.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:04 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-20T21:24:05.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:04 smithi189 ceph-mon[21562]: osdmap e21: 3 total, 3 up, 3 in 2024-04-20T21:24:05.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:04 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:24:05.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:04 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:24:05.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:04 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-20T21:24:05.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:04 smithi062 ceph-mon[16221]: osdmap e21: 3 total, 3 up, 3 in 2024-04-20T21:24:05.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:04 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:24:05.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:04 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:24:05.456 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi189/config 2024-04-20T21:24:05.785 INFO:journalctl@ceph.osd.0.smithi062.stdout:Apr 20 21:24:05 smithi062 sudo[44011]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-20T21:24:05.785 INFO:journalctl@ceph.osd.0.smithi062.stdout:Apr 20 21:24:05 smithi062 sudo[44011]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-20T21:24:05.785 INFO:journalctl@ceph.osd.0.smithi062.stdout:Apr 20 21:24:05 smithi062 sudo[44011]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-20T21:24:05.785 INFO:journalctl@ceph.osd.0.smithi062.stdout:Apr 20 21:24:05 smithi062 sudo[44011]: pam_unix(sudo:session): session closed for user root 2024-04-20T21:24:06.393 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:06 smithi062 ceph-mon[16221]: osdmap e22: 3 total, 3 up, 3 in 2024-04-20T21:24:06.393 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:06 smithi062 ceph-mon[16221]: pgmap v101: 1 pgs: 1 creating+peering; 0 B data, 79 MiB used, 268 GiB / 268 GiB avail 2024-04-20T21:24:06.393 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:06 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:24:06.393 INFO:journalctl@ceph.osd.0.smithi062.stdout:Apr 20 21:24:06 smithi062 sudo[44076]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-20T21:24:06.393 INFO:journalctl@ceph.osd.0.smithi062.stdout:Apr 20 21:24:06 smithi062 sudo[44076]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-20T21:24:06.393 INFO:journalctl@ceph.osd.0.smithi062.stdout:Apr 20 21:24:06 smithi062 sudo[44076]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-20T21:24:06.393 INFO:journalctl@ceph.osd.0.smithi062.stdout:Apr 20 21:24:06 smithi062 sudo[44076]: pam_unix(sudo:session): session closed for user root 2024-04-20T21:24:06.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:06 smithi189 ceph-mon[21562]: osdmap e22: 3 total, 3 up, 3 in 2024-04-20T21:24:06.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:06 smithi189 ceph-mon[21562]: pgmap v101: 1 pgs: 1 creating+peering; 0 B data, 79 MiB used, 268 GiB / 268 GiB avail 2024-04-20T21:24:06.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:06 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:24:06.996 INFO:journalctl@ceph.osd.1.smithi062.stdout:Apr 20 21:24:06 smithi062 sudo[44096]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-20T21:24:06.996 INFO:journalctl@ceph.osd.1.smithi062.stdout:Apr 20 21:24:06 smithi062 sudo[44096]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-20T21:24:06.996 INFO:journalctl@ceph.osd.1.smithi062.stdout:Apr 20 21:24:06 smithi062 sudo[44096]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-20T21:24:06.996 INFO:journalctl@ceph.osd.1.smithi062.stdout:Apr 20 21:24:06 smithi062 sudo[44096]: pam_unix(sudo:session): session closed for user root 2024-04-20T21:24:07.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:07 smithi062 ceph-mon[16221]: mgrmap e19: smithi062.rqxuxi(active, since 3m), standbys: smithi189.tawavz 2024-04-20T21:24:07.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:07 smithi189 ceph-mon[21562]: mgrmap e19: smithi062.rqxuxi(active, since 3m), standbys: smithi189.tawavz 2024-04-20T21:24:07.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:24:07.488+0000 7f5140400700 1 -- 172.21.15.189:0/1623686141 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51380737b0 msgr2=0x7f5138073c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:24:07.487 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:24:07.488+0000 7f5140400700 1 --2- 172.21.15.189:0/1623686141 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51380737b0 0x7f5138073c20 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f5124009940 tx=0x7f5124009c50 comp rx=0 tx=0).stop 2024-04-20T21:24:07.488 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:24:07.488+0000 7f5140400700 1 -- 172.21.15.189:0/1623686141 shutdown_connections 2024-04-20T21:24:07.488 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:24:07.488+0000 7f5140400700 1 --2- 172.21.15.189:0/1623686141 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51380737b0 0x7f5138073c20 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:24:07.488 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:24:07.488+0000 7f5140400700 1 --2- 172.21.15.189:0/1623686141 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5138074d80 0x7f51380731e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:24:07.488 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:24:07.488+0000 7f5140400700 1 -- 172.21.15.189:0/1623686141 >> 172.21.15.189:0/1623686141 conn(0x7f51380fb7f0 msgr2=0x7f51380fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:24:07.488 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:24:07.489+0000 7f5140400700 1 -- 172.21.15.189:0/1623686141 shutdown_connections 2024-04-20T21:24:07.488 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:24:07.489+0000 7f5140400700 1 -- 172.21.15.189:0/1623686141 wait complete. 2024-04-20T21:24:07.488 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:24:07.490+0000 7f5140400700 1 Processor -- start 2024-04-20T21:24:07.488 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:24:07.490+0000 7f5140400700 1 -- start start 2024-04-20T21:24:07.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:24:07.491+0000 7f5140400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f51380737b0 0x7f5138197ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:24:07.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:24:07.491+0000 7f5140400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5138074d80 0x7f51381981e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:24:07.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:24:07.491+0000 7f5140400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51381987e0 con 0x7f5138074d80 2024-04-20T21:24:07.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:24:07.491+0000 7f5140400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5138198950 con 0x7f51380737b0 2024-04-20T21:24:07.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:24:07.491+0000 7f513da00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f51380737b0 0x7f5138197ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:24:07.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:24:07.491+0000 7f513da00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f51380737b0 0x7f5138197ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:33908/0 (socket says 172.21.15.189:33908) 2024-04-20T21:24:07.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:24:07.491+0000 7f513da00700 1 -- 172.21.15.189:0/2438340216 learned_addr learned my addr 172.21.15.189:0/2438340216 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-20T21:24:07.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:24:07.491+0000 7f513d000700 1 --2- 172.21.15.189:0/2438340216 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5138074d80 0x7f51381981e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:24:07.491 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:24:07.491+0000 7f513da00700 1 -- 172.21.15.189:0/2438340216 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5138074d80 msgr2=0x7f51381981e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:24:07.491 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:24:07.491+0000 7f513da00700 1 --2- 172.21.15.189:0/2438340216 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5138074d80 0x7f51381981e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:24:07.491 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:24:07.491+0000 7f513da00700 1 -- 172.21.15.189:0/2438340216 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51240095f0 con 0x7f51380737b0 2024-04-20T21:24:07.491 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:24:07.492+0000 7f513da00700 1 --2- 172.21.15.189:0/2438340216 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f51380737b0 0x7f5138197ca0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f512c00e920 tx=0x7f512c00ec30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:24:07.491 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:24:07.492+0000 7f5136000700 1 -- 172.21.15.189:0/2438340216 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f512c00c9d0 con 0x7f51380737b0 2024-04-20T21:24:07.491 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:24:07.492+0000 7f5136000700 1 -- 172.21.15.189:0/2438340216 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f512c010430 con 0x7f51380737b0 2024-04-20T21:24:07.491 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:24:07.492+0000 7f5136000700 1 -- 172.21.15.189:0/2438340216 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f512c018650 con 0x7f51380737b0 2024-04-20T21:24:07.491 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:24:07.492+0000 7f5140400700 1 -- 172.21.15.189:0/2438340216 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f513819d3e0 con 0x7f51380737b0 2024-04-20T21:24:07.491 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:24:07.492+0000 7f5140400700 1 -- 172.21.15.189:0/2438340216 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f513819d900 con 0x7f51380737b0 2024-04-20T21:24:07.492 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:24:07.493+0000 7f5140400700 1 -- 172.21.15.189:0/2438340216 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f513810af80 con 0x7f51380737b0 2024-04-20T21:24:07.492 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:24:07.494+0000 7f5136000700 1 -- 172.21.15.189:0/2438340216 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f512c0105a0 con 0x7f51380737b0 2024-04-20T21:24:07.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:24:07.495+0000 7f5136000700 1 --2- 172.21.15.189:0/2438340216 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f512006c680 0x7f512006eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:24:07.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:24:07.496+0000 7f5136000700 1 -- 172.21.15.189:0/2438340216 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 3232+0+0 (secure 0 0 0) 0x7f512c014070 con 0x7f51380737b0 2024-04-20T21:24:07.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:24:07.496+0000 7f513d000700 1 --2- 172.21.15.189:0/2438340216 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f512006c680 0x7f512006eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:24:07.495 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:24:07.497+0000 7f513d000700 1 --2- 172.21.15.189:0/2438340216 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f512006c680 0x7f512006eb30 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5124000c00 tx=0x7f5124005e80 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:24:07.499 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:24:07.501+0000 7f5136000700 1 -- 172.21.15.189:0/2438340216 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f512c051720 con 0x7f51380737b0 2024-04-20T21:24:07.586 INFO:journalctl@ceph.osd.1.smithi062.stdout:Apr 20 21:24:07 smithi062 sudo[44102]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-20T21:24:07.586 INFO:journalctl@ceph.osd.1.smithi062.stdout:Apr 20 21:24:07 smithi062 sudo[44102]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-20T21:24:07.586 INFO:journalctl@ceph.osd.1.smithi062.stdout:Apr 20 21:24:07 smithi062 sudo[44102]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-20T21:24:07.586 INFO:journalctl@ceph.osd.1.smithi062.stdout:Apr 20 21:24:07 smithi062 sudo[44102]: pam_unix(sudo:session): session closed for user root 2024-04-20T21:24:07.720 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:24:07.722+0000 7f5140400700 1 -- 172.21.15.189:0/2438340216 --> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f5138107740 con 0x7f512006c680 2024-04-20T21:24:08.229 INFO:journalctl@ceph.osd.2.smithi062.stdout:Apr 20 21:24:07 smithi062 sudo[44131]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-20T21:24:08.229 INFO:journalctl@ceph.osd.2.smithi062.stdout:Apr 20 21:24:07 smithi062 sudo[44131]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-20T21:24:08.229 INFO:journalctl@ceph.osd.2.smithi062.stdout:Apr 20 21:24:07 smithi062 sudo[44131]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-20T21:24:08.229 INFO:journalctl@ceph.osd.2.smithi062.stdout:Apr 20 21:24:07 smithi062 sudo[44131]: pam_unix(sudo:session): session closed for user root 2024-04-20T21:24:08.483 INFO:journalctl@ceph.osd.2.smithi062.stdout:Apr 20 21:24:08 smithi062 sudo[44145]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-20T21:24:08.483 INFO:journalctl@ceph.osd.2.smithi062.stdout:Apr 20 21:24:08 smithi062 sudo[44145]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-20T21:24:08.483 INFO:journalctl@ceph.osd.2.smithi062.stdout:Apr 20 21:24:08 smithi062 sudo[44145]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-20T21:24:08.483 INFO:journalctl@ceph.osd.2.smithi062.stdout:Apr 20 21:24:08 smithi062 sudo[44145]: pam_unix(sudo:session): session closed for user root 2024-04-20T21:24:08.483 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:08 smithi062 ceph-mon[16221]: from='client.24137 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:24:08.483 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:08 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-20T21:24:08.484 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:08 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-20T21:24:08.484 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:08 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:24:08.484 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:08 smithi062 ceph-mon[16221]: pgmap v102: 1 pgs: 1 creating+peering; 0 B data, 79 MiB used, 268 GiB / 268 GiB avail 2024-04-20T21:24:08.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:08 smithi189 ceph-mon[21562]: from='client.24137 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:24:08.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:08 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-20T21:24:08.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:08 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-20T21:24:08.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:08 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:24:08.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:08 smithi189 ceph-mon[21562]: pgmap v102: 1 pgs: 1 creating+peering; 0 B data, 79 MiB used, 268 GiB / 268 GiB avail 2024-04-20T21:24:09.504 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:09 smithi062 sudo[44482]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-20T21:24:09.504 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:09 smithi062 sudo[44482]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-20T21:24:09.504 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:09 smithi062 sudo[44482]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-20T21:24:09.504 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:09 smithi062 sudo[44482]: pam_unix(sudo:session): session closed for user root 2024-04-20T21:24:09.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:09 smithi062 sudo[44485]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-20T21:24:09.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:09 smithi062 sudo[44485]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-20T21:24:09.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:09 smithi062 sudo[44485]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-20T21:24:09.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:09 smithi062 sudo[44485]: pam_unix(sudo:session): session closed for user root 2024-04-20T21:24:10.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:10 smithi189 sudo[25304]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-20T21:24:10.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:10 smithi189 sudo[25304]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-20T21:24:10.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:10 smithi189 sudo[25304]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-20T21:24:10.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:10 smithi189 sudo[25304]: pam_unix(sudo:session): session closed for user root 2024-04-20T21:24:11.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:10 smithi189 sudo[25323]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-20T21:24:11.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:10 smithi189 sudo[25323]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-20T21:24:11.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:10 smithi189 sudo[25323]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-20T21:24:11.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:10 smithi189 sudo[25323]: pam_unix(sudo:session): session closed for user root 2024-04-20T21:24:11.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:10 smithi189 ceph-mon[21562]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-20T21:24:11.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:10 smithi189 ceph-mon[21562]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-20T21:24:11.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:10 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mon metadata", "id": "smithi062"}]: dispatch 2024-04-20T21:24:11.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:10 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mon metadata", "id": "smithi189"}]: dispatch 2024-04-20T21:24:11.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:10 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:24:11.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:10 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mon metadata", "id": "smithi062"}]: dispatch 2024-04-20T21:24:11.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:10 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mon metadata", "id": "smithi189"}]: dispatch 2024-04-20T21:24:11.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:10 smithi062 ceph-mon[16221]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-20T21:24:11.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:10 smithi062 ceph-mon[16221]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-20T21:24:11.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:10 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mon metadata", "id": "smithi062"}]: dispatch 2024-04-20T21:24:11.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:10 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mon metadata", "id": "smithi189"}]: dispatch 2024-04-20T21:24:11.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:10 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:24:11.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:10 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mon metadata", "id": "smithi062"}]: dispatch 2024-04-20T21:24:11.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:10 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mon metadata", "id": "smithi189"}]: dispatch 2024-04-20T21:24:11.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:11 smithi189 ceph-mon[21562]: Detected new or changed devices on smithi062 2024-04-20T21:24:11.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:11 smithi189 ceph-mon[21562]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-20T21:24:11.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:11 smithi189 ceph-mon[21562]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-04-20T21:24:11.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:11 smithi189 ceph-mon[21562]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-20T21:24:11.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:11 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:24:11.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:11 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-20T21:24:11.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:11 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-20T21:24:11.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:11 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-20T21:24:11.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:11 smithi189 ceph-mon[21562]: Adjusting osd_memory_target on smithi062 to 3947M 2024-04-20T21:24:11.989 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:11 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:24:11.989 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:11 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:24:11.989 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:11 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T21:24:11.989 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:11 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:24:11.989 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:11 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:24:11.989 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:11 smithi189 ceph-mon[21562]: from='client.? 172.21.15.189:0/166731464' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eaa2fe2b-87f8-4d01-8ddf-e72f244f31b6"}]: dispatch 2024-04-20T21:24:11.989 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:11 smithi189 ceph-mon[21562]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eaa2fe2b-87f8-4d01-8ddf-e72f244f31b6"}]: dispatch 2024-04-20T21:24:11.989 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:11 smithi189 ceph-mon[21562]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "eaa2fe2b-87f8-4d01-8ddf-e72f244f31b6"}]': finished 2024-04-20T21:24:11.989 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:11 smithi189 ceph-mon[21562]: osdmap e23: 4 total, 3 up, 4 in 2024-04-20T21:24:11.989 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:11 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-20T21:24:11.989 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:11 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:24:12.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:11 smithi062 ceph-mon[16221]: Detected new or changed devices on smithi062 2024-04-20T21:24:12.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:11 smithi062 ceph-mon[16221]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-20T21:24:12.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:11 smithi062 ceph-mon[16221]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-04-20T21:24:12.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:11 smithi062 ceph-mon[16221]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-20T21:24:12.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:11 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:24:12.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:11 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-20T21:24:12.287 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:11 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-20T21:24:12.287 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:11 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-20T21:24:12.287 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:11 smithi062 ceph-mon[16221]: Adjusting osd_memory_target on smithi062 to 3947M 2024-04-20T21:24:12.287 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:11 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:24:12.287 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:11 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:24:12.287 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:11 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T21:24:12.287 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:11 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:24:12.287 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:11 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:24:12.287 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:11 smithi062 ceph-mon[16221]: from='client.? 172.21.15.189:0/166731464' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eaa2fe2b-87f8-4d01-8ddf-e72f244f31b6"}]: dispatch 2024-04-20T21:24:12.287 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:11 smithi062 ceph-mon[16221]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eaa2fe2b-87f8-4d01-8ddf-e72f244f31b6"}]: dispatch 2024-04-20T21:24:12.287 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:11 smithi062 ceph-mon[16221]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "eaa2fe2b-87f8-4d01-8ddf-e72f244f31b6"}]': finished 2024-04-20T21:24:12.287 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:11 smithi062 ceph-mon[16221]: osdmap e23: 4 total, 3 up, 4 in 2024-04-20T21:24:12.287 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:11 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-20T21:24:12.288 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:11 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:24:13.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:12 smithi189 ceph-mon[21562]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-04-20T21:24:13.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:12 smithi189 ceph-mon[21562]: from='client.? 172.21.15.189:0/4049583995' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-20T21:24:13.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:12 smithi062 ceph-mon[16221]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-04-20T21:24:13.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:12 smithi062 ceph-mon[16221]: from='client.? 172.21.15.189:0/4049583995' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-20T21:24:15.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:14 smithi189 ceph-mon[21562]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 196 KiB/s wr, 13 op/s 2024-04-20T21:24:15.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:14 smithi062 ceph-mon[16221]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 196 KiB/s wr, 13 op/s 2024-04-20T21:24:17.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:16 smithi189 ceph-mon[21562]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-04-20T21:24:17.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:16 smithi062 ceph-mon[16221]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-04-20T21:24:18.334 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:18 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:24:18.334 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:18 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:24:18.334 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:18 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:24:18.334 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:18 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T21:24:18.335 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:18 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:24:18.335 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:18 smithi189 ceph-mon[21562]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-04-20T21:24:18.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:18 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:24:18.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:18 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:24:18.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:18 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:24:18.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:18 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T21:24:18.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:18 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:24:18.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:18 smithi062 ceph-mon[16221]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-04-20T21:24:19.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:19 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:24:19.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:19 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:24:20.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:20 smithi189 ceph-mon[21562]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-04-20T21:24:20.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:20 smithi062 ceph-mon[16221]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-04-20T21:24:21.401 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:21 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-20T21:24:21.401 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:21 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:24:21.402 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:21 smithi189 ceph-mon[21562]: Deploying daemon osd.3 on smithi189 2024-04-20T21:24:21.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:21 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-20T21:24:21.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:21 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:24:21.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:21 smithi062 ceph-mon[16221]: Deploying daemon osd.3 on smithi189 2024-04-20T21:24:22.859 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:22 smithi189 ceph-mon[21562]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 73 KiB/s wr, 5 op/s 2024-04-20T21:24:23.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:22 smithi062 ceph-mon[16221]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 73 KiB/s wr, 5 op/s 2024-04-20T21:24:23.969 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:23 smithi062 ceph-mon[16221]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-04-20T21:24:24.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:23 smithi189 ceph-mon[21562]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-04-20T21:24:26.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:25 smithi189 ceph-mon[21562]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-04-20T21:24:26.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:25 smithi062 ceph-mon[16221]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-04-20T21:24:28.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:27 smithi189 ceph-mon[21562]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-04-20T21:24:28.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:27 smithi062 ceph-mon[16221]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-04-20T21:24:29.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:28 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:24:29.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:28 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:24:29.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:28 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:24:29.059 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:28 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:24:29.059 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:28 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:24:29.060 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:28 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:24:30.125 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:29 smithi189 ceph-mon[21562]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-04-20T21:24:30.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:29 smithi062 ceph-mon[16221]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-04-20T21:24:32.663 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:24:32.663+0000 7f5136000700 1 -- 172.21.15.189:0/2438340216 <== mgr.14215 v2:172.21.15.62:6800/4116451241 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f5138107740 con 0x7f512006c680 2024-04-20T21:24:32.663 INFO:teuthology.orchestra.run.smithi189.stdout:Created osd(s) 3 on host 'smithi189' 2024-04-20T21:24:32.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:24:32.667+0000 7f5140400700 1 -- 172.21.15.189:0/2438340216 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f512006c680 msgr2=0x7f512006eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:24:32.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:24:32.667+0000 7f5140400700 1 --2- 172.21.15.189:0/2438340216 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f512006c680 0x7f512006eb30 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5124000c00 tx=0x7f5124005e80 comp rx=0 tx=0).stop 2024-04-20T21:24:32.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:24:32.667+0000 7f5140400700 1 -- 172.21.15.189:0/2438340216 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f51380737b0 msgr2=0x7f5138197ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:24:32.666 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:24:32.667+0000 7f5140400700 1 --2- 172.21.15.189:0/2438340216 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f51380737b0 0x7f5138197ca0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f512c00e920 tx=0x7f512c00ec30 comp rx=0 tx=0).stop 2024-04-20T21:24:32.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:24:32.667+0000 7f5140400700 1 -- 172.21.15.189:0/2438340216 shutdown_connections 2024-04-20T21:24:32.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:24:32.667+0000 7f5140400700 1 --2- 172.21.15.189:0/2438340216 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f512006c680 0x7f512006eb30 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:24:32.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:24:32.667+0000 7f5140400700 1 --2- 172.21.15.189:0/2438340216 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f51380737b0 0x7f5138197ca0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:24:32.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:24:32.667+0000 7f5140400700 1 --2- 172.21.15.189:0/2438340216 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5138074d80 0x7f51381981e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:24:32.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:24:32.667+0000 7f5140400700 1 -- 172.21.15.189:0/2438340216 >> 172.21.15.189:0/2438340216 conn(0x7f51380fb7f0 msgr2=0x7f5138106020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:24:32.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:24:32.668+0000 7f5140400700 1 -- 172.21.15.189:0/2438340216 shutdown_connections 2024-04-20T21:24:32.667 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:24:32.668+0000 7f5140400700 1 -- 172.21.15.189:0/2438340216 wait complete. 2024-04-20T21:24:32.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:32 smithi189 ceph-mon[21562]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-04-20T21:24:32.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:32 smithi189 ceph-mon[21562]: from='osd.3 [v2:172.21.15.189:6800/3169871320,v1:172.21.15.189:6801/3169871320]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-20T21:24:32.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:32 smithi189 ceph-mon[21562]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-20T21:24:32.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:32 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:24:32.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:32 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:24:32.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:32 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:24:32.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:32 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:24:33.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:32 smithi062 ceph-mon[16221]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-04-20T21:24:33.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:32 smithi062 ceph-mon[16221]: from='osd.3 [v2:172.21.15.189:6800/3169871320,v1:172.21.15.189:6801/3169871320]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-20T21:24:33.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:32 smithi062 ceph-mon[16221]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-20T21:24:33.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:32 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:24:33.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:32 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:24:33.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:32 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:24:33.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:32 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:24:33.434 DEBUG:teuthology.orchestra.run.smithi189:osd.3> sudo journalctl -f -n 0 -u ceph-61ae3628-ff5b-11ee-bc93-c7b262605968@osd.3.service 2024-04-20T21:24:33.444 INFO:tasks.cephadm:Deploying osd.4 on smithi189 with /dev/vg_nvme/lv_3... 2024-04-20T21:24:33.444 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-20T21:24:33.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:33 smithi189 ceph-mon[21562]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-20T21:24:33.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:33 smithi189 ceph-mon[21562]: osdmap e24: 4 total, 3 up, 4 in 2024-04-20T21:24:33.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:33 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-20T21:24:33.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:33 smithi189 ceph-mon[21562]: from='osd.3 [v2:172.21.15.189:6800/3169871320,v1:172.21.15.189:6801/3169871320]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-04-20T21:24:33.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:33 smithi189 ceph-mon[21562]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-04-20T21:24:33.988 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 20 21:24:33 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-3[27490]: 2024-04-20T21:24:33.743+0000 7f6f92e00700 -1 osd.3 0 waiting for initial osdmap 2024-04-20T21:24:33.988 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 20 21:24:33 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-3[27490]: 2024-04-20T21:24:33.757+0000 7f6f8c800700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-20T21:24:34.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:33 smithi062 ceph-mon[16221]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-20T21:24:34.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:33 smithi062 ceph-mon[16221]: osdmap e24: 4 total, 3 up, 4 in 2024-04-20T21:24:34.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:33 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-20T21:24:34.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:33 smithi062 ceph-mon[16221]: from='osd.3 [v2:172.21.15.189:6800/3169871320,v1:172.21.15.189:6801/3169871320]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-04-20T21:24:34.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:33 smithi062 ceph-mon[16221]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-04-20T21:24:34.899 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi189/config 2024-04-20T21:24:35.028 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:34 smithi189 ceph-mon[21562]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-04-20T21:24:35.028 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:34 smithi189 ceph-mon[21562]: osdmap e25: 4 total, 3 up, 4 in 2024-04-20T21:24:35.028 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:34 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-20T21:24:35.028 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:34 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-20T21:24:35.028 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:34 smithi189 ceph-mon[21562]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-04-20T21:24:35.028 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:34 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:24:35.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:34 smithi062 ceph-mon[16221]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-04-20T21:24:35.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:34 smithi062 ceph-mon[16221]: osdmap e25: 4 total, 3 up, 4 in 2024-04-20T21:24:35.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:34 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-20T21:24:35.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:34 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-20T21:24:35.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:34 smithi062 ceph-mon[16221]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-04-20T21:24:35.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:34 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:24:35.859 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:35 smithi189 ceph-mon[21562]: purged_snaps scrub starts 2024-04-20T21:24:35.859 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:35 smithi189 ceph-mon[21562]: purged_snaps scrub ok 2024-04-20T21:24:35.859 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:35 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-20T21:24:35.859 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:35 smithi189 ceph-mon[21562]: osd.3 [v2:172.21.15.189:6800/3169871320,v1:172.21.15.189:6801/3169871320] boot 2024-04-20T21:24:35.859 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:35 smithi189 ceph-mon[21562]: osdmap e26: 4 total, 4 up, 4 in 2024-04-20T21:24:35.860 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:35 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-20T21:24:36.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:35 smithi062 ceph-mon[16221]: purged_snaps scrub starts 2024-04-20T21:24:36.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:35 smithi062 ceph-mon[16221]: purged_snaps scrub ok 2024-04-20T21:24:36.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:35 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-20T21:24:36.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:35 smithi062 ceph-mon[16221]: osd.3 [v2:172.21.15.189:6800/3169871320,v1:172.21.15.189:6801/3169871320] boot 2024-04-20T21:24:36.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:35 smithi062 ceph-mon[16221]: osdmap e26: 4 total, 4 up, 4 in 2024-04-20T21:24:36.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:35 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-20T21:24:37.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:36 smithi062 ceph-mon[16221]: osdmap e27: 4 total, 4 up, 4 in 2024-04-20T21:24:37.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:36 smithi062 ceph-mon[16221]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-20T21:24:37.037 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:36 smithi189 ceph-mon[21562]: osdmap e27: 4 total, 4 up, 4 in 2024-04-20T21:24:37.037 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:36 smithi189 ceph-mon[21562]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-20T21:24:37.864 INFO:teuthology.orchestra.run.smithi189.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-20T21:24:37.864 INFO:teuthology.orchestra.run.smithi189.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-20T21:24:37.865 INFO:teuthology.orchestra.run.smithi189.stderr: stderr: 10+0 records in 2024-04-20T21:24:37.865 INFO:teuthology.orchestra.run.smithi189.stderr:10+0 records out 2024-04-20T21:24:37.865 INFO:teuthology.orchestra.run.smithi189.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0511279 s, 205 MB/s 2024-04-20T21:24:37.865 INFO:teuthology.orchestra.run.smithi189.stderr:--> Zapping successful for: 2024-04-20T21:24:38.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:37 smithi062 ceph-mon[16221]: osdmap e28: 4 total, 4 up, 4 in 2024-04-20T21:24:38.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:37 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:24:38.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:37 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:24:38.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:37 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-20T21:24:38.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:37 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:24:38.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:37 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:24:38.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:37 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T21:24:38.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:37 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:24:38.199 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:37 smithi189 ceph-mon[21562]: osdmap e28: 4 total, 4 up, 4 in 2024-04-20T21:24:38.199 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:37 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:24:38.200 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:37 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:24:38.200 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:37 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-20T21:24:38.200 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:37 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:24:38.200 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:37 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:24:38.200 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:37 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T21:24:38.200 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:37 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:24:38.538 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -- ceph orch daemon add osd smithi189:vg_nvme/lv_3 2024-04-20T21:24:38.918 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi189/config 2024-04-20T21:24:39.054 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:38 smithi062 ceph-mon[16221]: Detected new or changed devices on smithi189 2024-04-20T21:24:39.054 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:38 smithi062 ceph-mon[16221]: Adjusting osd_memory_target on smithi189 to 14913M 2024-04-20T21:24:39.054 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:38 smithi062 ceph-mon[16221]: osdmap e29: 4 total, 4 up, 4 in 2024-04-20T21:24:39.054 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:38 smithi062 ceph-mon[16221]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-20T21:24:39.054 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:38 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:24:39.054 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:38 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:24:39.054 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:38 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T21:24:39.054 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:38 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:24:39.062 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:38 smithi189 ceph-mon[21562]: Detected new or changed devices on smithi189 2024-04-20T21:24:39.063 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:38 smithi189 ceph-mon[21562]: Adjusting osd_memory_target on smithi189 to 14913M 2024-04-20T21:24:39.063 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:38 smithi189 ceph-mon[21562]: osdmap e29: 4 total, 4 up, 4 in 2024-04-20T21:24:39.063 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:38 smithi189 ceph-mon[21562]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-20T21:24:39.063 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:38 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:24:39.063 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:38 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:24:39.063 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:38 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T21:24:39.063 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:38 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:24:40.113 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:39 smithi189 ceph-mon[21562]: osdmap e30: 4 total, 4 up, 4 in 2024-04-20T21:24:40.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:39 smithi062 ceph-mon[16221]: osdmap e30: 4 total, 4 up, 4 in 2024-04-20T21:24:41.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:24:41.023+0000 7f0ce7000700 1 -- 172.21.15.189:0/1060570523 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ce0100d60 msgr2=0x7f0ce0103140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:24:41.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:24:41.023+0000 7f0ce7000700 1 --2- 172.21.15.189:0/1060570523 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ce0100d60 0x7f0ce0103140 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f0cd0009910 tx=0x7f0cd0009c20 comp rx=0 tx=0).stop 2024-04-20T21:24:41.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:24:41.023+0000 7f0ce7000700 1 -- 172.21.15.189:0/1060570523 shutdown_connections 2024-04-20T21:24:41.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:24:41.023+0000 7f0ce7000700 1 --2- 172.21.15.189:0/1060570523 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0ce0103680 0x7f0ce0105a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:24:41.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:24:41.023+0000 7f0ce7000700 1 --2- 172.21.15.189:0/1060570523 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ce0100d60 0x7f0ce0103140 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:24:41.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:24:41.023+0000 7f0ce7000700 1 -- 172.21.15.189:0/1060570523 >> 172.21.15.189:0/1060570523 conn(0x7f0ce00fa780 msgr2=0x7f0ce00fcbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:24:41.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:24:41.023+0000 7f0ce7000700 1 -- 172.21.15.189:0/1060570523 shutdown_connections 2024-04-20T21:24:41.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:24:41.023+0000 7f0ce7000700 1 -- 172.21.15.189:0/1060570523 wait complete. 2024-04-20T21:24:41.024 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:24:41.024+0000 7f0ce7000700 1 Processor -- start 2024-04-20T21:24:41.024 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:24:41.025+0000 7f0ce7000700 1 -- start start 2024-04-20T21:24:41.024 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:24:41.025+0000 7f0ce7000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ce0100d60 0x7f0ce0197c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:24:41.024 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:24:41.025+0000 7f0ce7000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0ce0103680 0x7f0ce01981a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:24:41.024 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:24:41.025+0000 7f0ce7000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ce01987a0 con 0x7f0ce0103680 2024-04-20T21:24:41.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:24:41.025+0000 7f0ce7000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ce0198910 con 0x7f0ce0100d60 2024-04-20T21:24:41.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:24:41.025+0000 7f0cdfe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ce0100d60 0x7f0ce0197c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:24:41.025 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:24:41.025+0000 7f0cdfe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ce0100d60 0x7f0ce0197c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:41944/0 (socket says 172.21.15.189:41944) 2024-04-20T21:24:41.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:24:41.025+0000 7f0cdfe00700 1 -- 172.21.15.189:0/2118201807 learned_addr learned my addr 172.21.15.189:0/2118201807 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-20T21:24:41.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:24:41.026+0000 7f0cdf400700 1 --2- 172.21.15.189:0/2118201807 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0ce0103680 0x7f0ce01981a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:24:41.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:24:41.026+0000 7f0cdfe00700 1 -- 172.21.15.189:0/2118201807 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0ce0103680 msgr2=0x7f0ce01981a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:24:41.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:24:41.026+0000 7f0cdfe00700 1 --2- 172.21.15.189:0/2118201807 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0ce0103680 0x7f0ce01981a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:24:41.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:24:41.026+0000 7f0cdfe00700 1 -- 172.21.15.189:0/2118201807 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cd00095f0 con 0x7f0ce0100d60 2024-04-20T21:24:41.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:24:41.026+0000 7f0cdf400700 1 --2- 172.21.15.189:0/2118201807 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0ce0103680 0x7f0ce01981a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T21:24:41.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:24:41.026+0000 7f0cdfe00700 1 --2- 172.21.15.189:0/2118201807 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ce0100d60 0x7f0ce0197c60 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f0cd000bda0 tx=0x7f0cd000be80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:24:41.026 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:24:41.027+0000 7f0cdcc00700 1 -- 172.21.15.189:0/2118201807 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0cd001d070 con 0x7f0ce0100d60 2024-04-20T21:24:41.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:24:41.027+0000 7f0ce7000700 1 -- 172.21.15.189:0/2118201807 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ce019d340 con 0x7f0ce0100d60 2024-04-20T21:24:41.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:24:41.027+0000 7f0cdcc00700 1 -- 172.21.15.189:0/2118201807 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f0cd000f460 con 0x7f0ce0100d60 2024-04-20T21:24:41.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:24:41.027+0000 7f0cdcc00700 1 -- 172.21.15.189:0/2118201807 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0cd0021620 con 0x7f0ce0100d60 2024-04-20T21:24:41.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:24:41.027+0000 7f0ce7000700 1 -- 172.21.15.189:0/2118201807 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ce019d830 con 0x7f0ce0100d60 2024-04-20T21:24:41.027 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:24:41.028+0000 7f0ce7000700 1 -- 172.21.15.189:0/2118201807 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ce00fc340 con 0x7f0ce0100d60 2024-04-20T21:24:41.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:24:41.029+0000 7f0cdcc00700 1 -- 172.21.15.189:0/2118201807 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f0cd000f5d0 con 0x7f0ce0100d60 2024-04-20T21:24:41.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:24:41.031+0000 7f0cdcc00700 1 --2- 172.21.15.189:0/2118201807 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f0cc406c680 0x7f0cc406eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:24:41.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:24:41.031+0000 7f0cdcc00700 1 -- 172.21.15.189:0/2118201807 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 3774+0+0 (secure 0 0 0) 0x7f0cd008c520 con 0x7f0ce0100d60 2024-04-20T21:24:41.030 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:24:41.031+0000 7f0cdf400700 1 --2- 172.21.15.189:0/2118201807 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f0cc406c680 0x7f0cc406eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:24:41.031 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:24:41.032+0000 7f0cdf400700 1 --2- 172.21.15.189:0/2118201807 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f0cc406c680 0x7f0cc406eb30 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0ccc005fd0 tx=0x7f0ccc005ee0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:24:41.037 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:24:41.036+0000 7f0cdcc00700 1 -- 172.21.15.189:0/2118201807 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f0cd0057200 con 0x7f0ce0100d60 2024-04-20T21:24:41.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:40 smithi189 ceph-mon[21562]: pgmap v126: 1 pgs: 1 peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 141 KiB/s, 0 objects/s recovering 2024-04-20T21:24:41.263 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:24:41.263+0000 7f0ce7000700 1 -- 172.21.15.189:0/2118201807 --> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f0ce0061190 con 0x7f0cc406c680 2024-04-20T21:24:41.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:40 smithi062 ceph-mon[16221]: pgmap v126: 1 pgs: 1 peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 141 KiB/s, 0 objects/s recovering 2024-04-20T21:24:42.076 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:41 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-20T21:24:42.077 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:41 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-20T21:24:42.077 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:41 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:24:42.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:41 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-20T21:24:42.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:41 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-20T21:24:42.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:41 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:24:43.080 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:42 smithi189 ceph-mon[21562]: from='client.24157 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:24:43.080 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:42 smithi189 ceph-mon[21562]: pgmap v127: 1 pgs: 1 peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-20T21:24:43.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:42 smithi062 ceph-mon[16221]: from='client.24157 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:24:43.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:42 smithi062 ceph-mon[16221]: pgmap v127: 1 pgs: 1 peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-20T21:24:44.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:43 smithi189 ceph-mon[21562]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-04-20T21:24:44.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:43 smithi062 ceph-mon[16221]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-04-20T21:24:45.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:44 smithi189 ceph-mon[21562]: from='client.? 172.21.15.189:0/2918078275' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "72be8597-52e5-45fd-b4f2-06ac50df77e6"}]: dispatch 2024-04-20T21:24:45.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:44 smithi189 ceph-mon[21562]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "72be8597-52e5-45fd-b4f2-06ac50df77e6"}]: dispatch 2024-04-20T21:24:45.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:44 smithi189 ceph-mon[21562]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "72be8597-52e5-45fd-b4f2-06ac50df77e6"}]': finished 2024-04-20T21:24:45.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:44 smithi189 ceph-mon[21562]: osdmap e31: 5 total, 4 up, 5 in 2024-04-20T21:24:45.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:44 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-20T21:24:45.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:44 smithi062 ceph-mon[16221]: from='client.? 172.21.15.189:0/2918078275' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "72be8597-52e5-45fd-b4f2-06ac50df77e6"}]: dispatch 2024-04-20T21:24:45.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:44 smithi062 ceph-mon[16221]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "72be8597-52e5-45fd-b4f2-06ac50df77e6"}]: dispatch 2024-04-20T21:24:45.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:44 smithi062 ceph-mon[16221]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "72be8597-52e5-45fd-b4f2-06ac50df77e6"}]': finished 2024-04-20T21:24:45.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:44 smithi062 ceph-mon[16221]: osdmap e31: 5 total, 4 up, 5 in 2024-04-20T21:24:45.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:44 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-20T21:24:46.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:45 smithi062 ceph-mon[16221]: from='client.? 172.21.15.189:0/351695087' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-20T21:24:46.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:45 smithi062 ceph-mon[16221]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-20T21:24:46.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:46 smithi189 ceph-mon[21562]: from='client.? 172.21.15.189:0/351695087' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-20T21:24:46.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:46 smithi189 ceph-mon[21562]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-20T21:24:48.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:47 smithi189 ceph-mon[21562]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-20T21:24:48.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:47 smithi062 ceph-mon[16221]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-20T21:24:49.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:48 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:24:49.281 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:48 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:24:50.089 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:49 smithi189 ceph-mon[21562]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-20T21:24:50.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:49 smithi062 ceph-mon[16221]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-20T21:24:52.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:52 smithi189 ceph-mon[21562]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-20T21:24:53.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:52 smithi062 ceph-mon[16221]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-20T21:24:53.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:53 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-20T21:24:53.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:53 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:24:54.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:53 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-20T21:24:54.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:53 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:24:54.796 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:54 smithi189 ceph-mon[21562]: Deploying daemon osd.4 on smithi189 2024-04-20T21:24:54.796 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:54 smithi189 ceph-mon[21562]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-20T21:24:55.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:54 smithi062 ceph-mon[16221]: Deploying daemon osd.4 on smithi189 2024-04-20T21:24:55.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:54 smithi062 ceph-mon[16221]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-20T21:24:56.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:55 smithi189 ceph-mon[21562]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-20T21:24:56.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:55 smithi062 ceph-mon[16221]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-20T21:24:58.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:57 smithi189 ceph-mon[21562]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-20T21:24:58.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:57 smithi062 ceph-mon[16221]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-20T21:25:00.104 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:24:59 smithi189 ceph-mon[21562]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-20T21:25:00.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:24:59 smithi062 ceph-mon[16221]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-20T21:25:02.568 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:02 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:25:02.568 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:02 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:25:02.568 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:02 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:25:02.568 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:02 smithi189 ceph-mon[21562]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-20T21:25:02.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:02 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:25:02.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:02 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:25:02.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:02 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:25:02.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:02 smithi062 ceph-mon[16221]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-20T21:25:03.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:03 smithi189 ceph-mon[21562]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-20T21:25:03.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:03 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:25:04.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:03 smithi062 ceph-mon[16221]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-20T21:25:04.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:03 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:25:05.687 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:05 smithi189 ceph-mon[21562]: from='osd.4 [v2:172.21.15.189:6808/191133671,v1:172.21.15.189:6809/191133671]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-20T21:25:05.688 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:05 smithi189 ceph-mon[21562]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-20T21:25:05.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:05 smithi062 ceph-mon[16221]: from='osd.4 [v2:172.21.15.189:6808/191133671,v1:172.21.15.189:6809/191133671]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-20T21:25:05.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:05 smithi062 ceph-mon[16221]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-20T21:25:06.630 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:06 smithi189 ceph-mon[21562]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-20T21:25:06.630 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:06 smithi189 ceph-mon[21562]: osdmap e32: 5 total, 4 up, 5 in 2024-04-20T21:25:06.631 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:06 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-20T21:25:06.631 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:06 smithi189 ceph-mon[21562]: from='osd.4 [v2:172.21.15.189:6808/191133671,v1:172.21.15.189:6809/191133671]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-04-20T21:25:06.631 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:06 smithi189 ceph-mon[21562]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-04-20T21:25:06.631 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:06 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:25:06.631 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:06 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:25:06.631 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:06 smithi189 ceph-mon[21562]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-20T21:25:06.736 INFO:teuthology.orchestra.run.smithi189.stdout:Created osd(s) 4 on host 'smithi189' 2024-04-20T21:25:06.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:25:06.736+0000 7f0cdcc00700 1 -- 172.21.15.189:0/2118201807 <== mgr.14215 v2:172.21.15.62:6800/4116451241 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f0ce0061190 con 0x7f0cc406c680 2024-04-20T21:25:06.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:25:06.739+0000 7f0ce7000700 1 -- 172.21.15.189:0/2118201807 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f0cc406c680 msgr2=0x7f0cc406eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:25:06.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:25:06.739+0000 7f0ce7000700 1 --2- 172.21.15.189:0/2118201807 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f0cc406c680 0x7f0cc406eb30 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0ccc005fd0 tx=0x7f0ccc005ee0 comp rx=0 tx=0).stop 2024-04-20T21:25:06.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:25:06.739+0000 7f0ce7000700 1 -- 172.21.15.189:0/2118201807 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ce0100d60 msgr2=0x7f0ce0197c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:25:06.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:25:06.739+0000 7f0ce7000700 1 --2- 172.21.15.189:0/2118201807 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ce0100d60 0x7f0ce0197c60 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f0cd000bda0 tx=0x7f0cd000be80 comp rx=0 tx=0).stop 2024-04-20T21:25:06.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:25:06.739+0000 7f0ce7000700 1 -- 172.21.15.189:0/2118201807 shutdown_connections 2024-04-20T21:25:06.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:25:06.739+0000 7f0ce7000700 1 --2- 172.21.15.189:0/2118201807 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f0cc406c680 0x7f0cc406eb30 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:25:06.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:25:06.739+0000 7f0ce7000700 1 --2- 172.21.15.189:0/2118201807 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ce0100d60 0x7f0ce0197c60 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:25:06.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:25:06.739+0000 7f0ce7000700 1 --2- 172.21.15.189:0/2118201807 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0ce0103680 0x7f0ce01981a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:25:06.739 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:25:06.739+0000 7f0ce7000700 1 -- 172.21.15.189:0/2118201807 >> 172.21.15.189:0/2118201807 conn(0x7f0ce00fa780 msgr2=0x7f0ce00ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:25:06.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:25:06.739+0000 7f0ce7000700 1 -- 172.21.15.189:0/2118201807 shutdown_connections 2024-04-20T21:25:06.740 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:25:06.739+0000 7f0ce7000700 1 -- 172.21.15.189:0/2118201807 wait complete. 2024-04-20T21:25:06.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:06 smithi062 ceph-mon[16221]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-20T21:25:06.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:06 smithi062 ceph-mon[16221]: osdmap e32: 5 total, 4 up, 5 in 2024-04-20T21:25:06.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:06 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-20T21:25:06.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:06 smithi062 ceph-mon[16221]: from='osd.4 [v2:172.21.15.189:6808/191133671,v1:172.21.15.189:6809/191133671]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-04-20T21:25:06.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:06 smithi062 ceph-mon[16221]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-04-20T21:25:06.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:06 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:25:06.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:06 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:25:06.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:06 smithi062 ceph-mon[16221]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-20T21:25:07.456 DEBUG:teuthology.orchestra.run.smithi189:osd.4> sudo journalctl -f -n 0 -u ceph-61ae3628-ff5b-11ee-bc93-c7b262605968@osd.4.service 2024-04-20T21:25:07.460 INFO:tasks.cephadm:Deploying osd.5 on smithi189 with /dev/vg_nvme/lv_2... 2024-04-20T21:25:07.460 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-20T21:25:07.689 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:07 smithi189 ceph-mon[21562]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-04-20T21:25:07.829 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:07 smithi189 ceph-mon[21562]: osdmap e33: 5 total, 4 up, 5 in 2024-04-20T21:25:07.830 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:07 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-20T21:25:07.830 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:07 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-20T21:25:07.830 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:07 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:25:07.830 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:07 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:25:07.830 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:07 smithi062 ceph-mon[16221]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-04-20T21:25:07.831 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:07 smithi062 ceph-mon[16221]: osdmap e33: 5 total, 4 up, 5 in 2024-04-20T21:25:07.831 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:07 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-20T21:25:07.831 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:07 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-20T21:25:07.831 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:07 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:25:07.831 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:07 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:25:08.694 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:08 smithi189 ceph-mon[21562]: purged_snaps scrub starts 2024-04-20T21:25:08.695 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:08 smithi189 ceph-mon[21562]: purged_snaps scrub ok 2024-04-20T21:25:08.695 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:08 smithi189 ceph-mon[21562]: osd.4 [v2:172.21.15.189:6808/191133671,v1:172.21.15.189:6809/191133671] boot 2024-04-20T21:25:08.695 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:08 smithi189 ceph-mon[21562]: osdmap e34: 5 total, 5 up, 5 in 2024-04-20T21:25:08.695 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:08 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-20T21:25:08.695 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:08 smithi189 ceph-mon[21562]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-20T21:25:08.707 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi189/config 2024-04-20T21:25:08.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:08 smithi062 ceph-mon[16221]: purged_snaps scrub starts 2024-04-20T21:25:08.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:08 smithi062 ceph-mon[16221]: purged_snaps scrub ok 2024-04-20T21:25:08.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:08 smithi062 ceph-mon[16221]: osd.4 [v2:172.21.15.189:6808/191133671,v1:172.21.15.189:6809/191133671] boot 2024-04-20T21:25:08.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:08 smithi062 ceph-mon[16221]: osdmap e34: 5 total, 5 up, 5 in 2024-04-20T21:25:08.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:08 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-20T21:25:08.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:08 smithi062 ceph-mon[16221]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-20T21:25:09.583 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:09 smithi189 ceph-mon[21562]: osdmap e35: 5 total, 5 up, 5 in 2024-04-20T21:25:09.730 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:09 smithi062 ceph-mon[16221]: osdmap e35: 5 total, 5 up, 5 in 2024-04-20T21:25:10.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:10 smithi189 ceph-mon[21562]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-20T21:25:10.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:10 smithi062 ceph-mon[16221]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-20T21:25:12.241 INFO:teuthology.orchestra.run.smithi189.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-20T21:25:12.241 INFO:teuthology.orchestra.run.smithi189.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-20T21:25:12.241 INFO:teuthology.orchestra.run.smithi189.stderr: stderr: 10+0 records in 2024-04-20T21:25:12.241 INFO:teuthology.orchestra.run.smithi189.stderr:10+0 records out 2024-04-20T21:25:12.241 INFO:teuthology.orchestra.run.smithi189.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0364931 s, 287 MB/s 2024-04-20T21:25:12.241 INFO:teuthology.orchestra.run.smithi189.stderr:--> Zapping successful for: 2024-04-20T21:25:12.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:12 smithi189 ceph-mon[21562]: Detected new or changed devices on smithi189 2024-04-20T21:25:12.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:12 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:25:12.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:12 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:25:12.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:12 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-20T21:25:12.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:12 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-20T21:25:12.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:12 smithi189 ceph-mon[21562]: Adjusting osd_memory_target on smithi189 to 7456M 2024-04-20T21:25:12.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:12 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:25:12.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:12 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:25:12.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:12 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T21:25:12.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:12 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:25:12.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:12 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:25:12.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:12 smithi189 ceph-mon[21562]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-20T21:25:12.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:12 smithi062 ceph-mon[16221]: Detected new or changed devices on smithi189 2024-04-20T21:25:12.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:12 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:25:12.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:12 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:25:12.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:12 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-20T21:25:12.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:12 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-20T21:25:12.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:12 smithi062 ceph-mon[16221]: Adjusting osd_memory_target on smithi189 to 7456M 2024-04-20T21:25:12.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:12 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:25:12.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:12 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:25:12.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:12 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T21:25:12.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:12 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:25:12.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:12 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:25:12.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:12 smithi062 ceph-mon[16221]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-20T21:25:12.914 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -- ceph orch daemon add osd smithi189:vg_nvme/lv_2 2024-04-20T21:25:13.291 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi189/config 2024-04-20T21:25:13.443 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:13 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:25:13.443 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:13 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T21:25:13.443 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:13 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:25:13.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:13 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:25:13.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:13 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T21:25:13.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:13 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:25:14.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:14 smithi062 ceph-mon[16221]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-20T21:25:14.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:14 smithi189 ceph-mon[21562]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-20T21:25:15.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:25:15.382+0000 7fdbe5800700 1 -- 172.21.15.189:0/2352459841 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdbe0074d80 msgr2=0x7fdbe00731e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:25:15.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:25:15.382+0000 7fdbe5800700 1 --2- 172.21.15.189:0/2352459841 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdbe0074d80 0x7fdbe00731e0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fdbd000b0a0 tx=0x7fdbd000b3b0 comp rx=0 tx=0).stop 2024-04-20T21:25:15.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:25:15.382+0000 7fdbe5800700 1 -- 172.21.15.189:0/2352459841 shutdown_connections 2024-04-20T21:25:15.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:25:15.382+0000 7fdbe5800700 1 --2- 172.21.15.189:0/2352459841 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdbe00737b0 0x7fdbe0073c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:25:15.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:25:15.382+0000 7fdbe5800700 1 --2- 172.21.15.189:0/2352459841 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdbe0074d80 0x7fdbe00731e0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:25:15.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:25:15.382+0000 7fdbe5800700 1 -- 172.21.15.189:0/2352459841 >> 172.21.15.189:0/2352459841 conn(0x7fdbe00fb7f0 msgr2=0x7fdbe00fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:25:15.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:25:15.382+0000 7fdbe5800700 1 -- 172.21.15.189:0/2352459841 shutdown_connections 2024-04-20T21:25:15.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:25:15.383+0000 7fdbe5800700 1 -- 172.21.15.189:0/2352459841 wait complete. 2024-04-20T21:25:15.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:25:15.384+0000 7fdbe5800700 1 Processor -- start 2024-04-20T21:25:15.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:25:15.384+0000 7fdbe5800700 1 -- start start 2024-04-20T21:25:15.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:25:15.384+0000 7fdbe5800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdbe00737b0 0x7fdbe019c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:25:15.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:25:15.384+0000 7fdbe5800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdbe0074d80 0x7fdbe019c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:25:15.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:25:15.384+0000 7fdbe5800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbe019cc10 con 0x7fdbe00737b0 2024-04-20T21:25:15.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:25:15.384+0000 7fdbe5800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbe019cd50 con 0x7fdbe0074d80 2024-04-20T21:25:15.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:25:15.384+0000 7fdbd6000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdbe0074d80 0x7fdbe019c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:25:15.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:25:15.385+0000 7fdbd6000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdbe0074d80 0x7fdbe019c5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50420/0 (socket says 172.21.15.189:50420) 2024-04-20T21:25:15.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:25:15.385+0000 7fdbd6000700 1 -- 172.21.15.189:0/1237057333 learned_addr learned my addr 172.21.15.189:0/1237057333 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-20T21:25:15.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:25:15.385+0000 7fdbdea00700 1 --2- 172.21.15.189:0/1237057333 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdbe00737b0 0x7fdbe019c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:25:15.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:25:15.385+0000 7fdbd6000700 1 -- 172.21.15.189:0/1237057333 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdbe00737b0 msgr2=0x7fdbe019c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:25:15.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:25:15.385+0000 7fdbd6000700 1 --2- 172.21.15.189:0/1237057333 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdbe00737b0 0x7fdbe019c0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:25:15.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:25:15.385+0000 7fdbd6000700 1 -- 172.21.15.189:0/1237057333 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbd0009d00 con 0x7fdbe0074d80 2024-04-20T21:25:15.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:25:15.385+0000 7fdbd6000700 1 --2- 172.21.15.189:0/1237057333 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdbe0074d80 0x7fdbe019c5f0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fdbc800c8a0 tx=0x7fdbc800cc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:25:15.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:25:15.386+0000 7fdbd7e00700 1 -- 172.21.15.189:0/1237057333 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdbc800cea0 con 0x7fdbe0074d80 2024-04-20T21:25:15.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:25:15.386+0000 7fdbd7e00700 1 -- 172.21.15.189:0/1237057333 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fdbc8004830 con 0x7fdbe0074d80 2024-04-20T21:25:15.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:25:15.386+0000 7fdbe5800700 1 -- 172.21.15.189:0/1237057333 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbe01a1800 con 0x7fdbe0074d80 2024-04-20T21:25:15.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:25:15.386+0000 7fdbd7e00700 1 -- 172.21.15.189:0/1237057333 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdbc80055b0 con 0x7fdbe0074d80 2024-04-20T21:25:15.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:25:15.386+0000 7fdbe5800700 1 -- 172.21.15.189:0/1237057333 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbe01a1d50 con 0x7fdbe0074d80 2024-04-20T21:25:15.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:25:15.387+0000 7fdbe5800700 1 -- 172.21.15.189:0/1237057333 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbe010f370 con 0x7fdbe0074d80 2024-04-20T21:25:15.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:25:15.388+0000 7fdbd7e00700 1 -- 172.21.15.189:0/1237057333 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7fdbc8007ad0 con 0x7fdbe0074d80 2024-04-20T21:25:15.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:25:15.389+0000 7fdbd7e00700 1 --2- 172.21.15.189:0/1237057333 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fdbc406c680 0x7fdbc406eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:25:15.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:25:15.389+0000 7fdbd7e00700 1 -- 172.21.15.189:0/1237057333 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4206+0+0 (secure 0 0 0) 0x7fdbc808b0e0 con 0x7fdbe0074d80 2024-04-20T21:25:15.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:25:15.390+0000 7fdbdea00700 1 --2- 172.21.15.189:0/1237057333 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fdbc406c680 0x7fdbc406eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:25:15.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:25:15.391+0000 7fdbdea00700 1 --2- 172.21.15.189:0/1237057333 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fdbc406c680 0x7fdbc406eb30 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fdbd0000f80 tx=0x7fdbd000b7b0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:25:15.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:25:15.394+0000 7fdbd7e00700 1 -- 172.21.15.189:0/1237057333 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fdbc8059720 con 0x7fdbe0074d80 2024-04-20T21:25:15.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:25:15.619+0000 7fdbe5800700 1 -- 172.21.15.189:0/1237057333 --> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fdbe01032f0 con 0x7fdbc406c680 2024-04-20T21:25:15.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:15 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-20T21:25:15.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:15 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-20T21:25:15.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:15 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:25:16.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:15 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-20T21:25:16.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:15 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-20T21:25:16.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:15 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:25:16.912 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:16 smithi189 ceph-mon[21562]: from='client.24175 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:25:16.912 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:16 smithi189 ceph-mon[21562]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-20T21:25:17.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:16 smithi062 ceph-mon[16221]: from='client.24175 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:25:17.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:16 smithi062 ceph-mon[16221]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-20T21:25:18.222 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:17 smithi189 ceph-mon[21562]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-20T21:25:18.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:17 smithi062 ceph-mon[16221]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-20T21:25:19.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:18 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:25:19.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:18 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:25:20.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:19 smithi189 ceph-mon[21562]: from='client.? 172.21.15.189:0/833994016' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c650c63b-41eb-4d4f-8e14-907c7b3614c3"}]: dispatch 2024-04-20T21:25:20.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:19 smithi189 ceph-mon[21562]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c650c63b-41eb-4d4f-8e14-907c7b3614c3"}]: dispatch 2024-04-20T21:25:20.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:19 smithi189 ceph-mon[21562]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c650c63b-41eb-4d4f-8e14-907c7b3614c3"}]': finished 2024-04-20T21:25:20.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:19 smithi189 ceph-mon[21562]: osdmap e36: 6 total, 5 up, 6 in 2024-04-20T21:25:20.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:19 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-20T21:25:20.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:19 smithi189 ceph-mon[21562]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-20T21:25:20.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:19 smithi062 ceph-mon[16221]: from='client.? 172.21.15.189:0/833994016' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c650c63b-41eb-4d4f-8e14-907c7b3614c3"}]: dispatch 2024-04-20T21:25:20.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:19 smithi062 ceph-mon[16221]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c650c63b-41eb-4d4f-8e14-907c7b3614c3"}]: dispatch 2024-04-20T21:25:20.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:19 smithi062 ceph-mon[16221]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c650c63b-41eb-4d4f-8e14-907c7b3614c3"}]': finished 2024-04-20T21:25:20.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:19 smithi062 ceph-mon[16221]: osdmap e36: 6 total, 5 up, 6 in 2024-04-20T21:25:20.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:19 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-20T21:25:20.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:19 smithi062 ceph-mon[16221]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-20T21:25:21.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:20 smithi189 ceph-mon[21562]: from='client.? 172.21.15.189:0/3888050574' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-20T21:25:21.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:20 smithi062 ceph-mon[16221]: from='client.? 172.21.15.189:0/3888050574' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-20T21:25:23.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:22 smithi062 ceph-mon[16221]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-20T21:25:23.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:22 smithi189 ceph-mon[21562]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-20T21:25:24.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:23 smithi189 ceph-mon[21562]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-20T21:25:24.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:23 smithi062 ceph-mon[16221]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-20T21:25:26.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:25 smithi189 ceph-mon[21562]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-20T21:25:26.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:25 smithi062 ceph-mon[16221]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-20T21:25:27.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:27 smithi189 ceph-mon[21562]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-20T21:25:28.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:27 smithi062 ceph-mon[16221]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-20T21:25:29.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:28 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-20T21:25:29.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:28 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:25:29.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:28 smithi189 ceph-mon[21562]: Deploying daemon osd.5 on smithi189 2024-04-20T21:25:29.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:28 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-20T21:25:29.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:28 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:25:29.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:28 smithi062 ceph-mon[16221]: Deploying daemon osd.5 on smithi189 2024-04-20T21:25:30.188 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:29 smithi189 ceph-mon[21562]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-20T21:25:30.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:29 smithi062 ceph-mon[16221]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-20T21:25:32.864 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:32 smithi189 ceph-mon[21562]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-20T21:25:33.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:32 smithi062 ceph-mon[16221]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-20T21:25:34.225 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:33 smithi189 ceph-mon[21562]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-20T21:25:34.225 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:33 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:25:34.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:33 smithi062 ceph-mon[16221]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-20T21:25:34.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:33 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:25:35.990 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:35 smithi189 ceph-mon[21562]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-20T21:25:36.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:35 smithi062 ceph-mon[16221]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-20T21:25:37.916 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:37 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:25:37.916 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:37 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:25:37.916 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:37 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:25:38.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:37 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:25:38.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:37 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:25:38.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:37 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:25:39.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:38 smithi189 ceph-mon[21562]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-20T21:25:39.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:38 smithi062 ceph-mon[16221]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-20T21:25:41.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:40 smithi189 ceph-mon[21562]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-20T21:25:41.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:40 smithi062 ceph-mon[16221]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-20T21:25:41.580 INFO:teuthology.orchestra.run.smithi189.stdout:Created osd(s) 5 on host 'smithi189' 2024-04-20T21:25:41.580 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:25:41.580+0000 7fdbd7e00700 1 -- 172.21.15.189:0/1237057333 <== mgr.14215 v2:172.21.15.62:6800/4116451241 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fdbe01032f0 con 0x7fdbc406c680 2024-04-20T21:25:41.583 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:25:41.583+0000 7fdbe5800700 1 -- 172.21.15.189:0/1237057333 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fdbc406c680 msgr2=0x7fdbc406eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:25:41.583 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:25:41.583+0000 7fdbe5800700 1 --2- 172.21.15.189:0/1237057333 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fdbc406c680 0x7fdbc406eb30 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fdbd0000f80 tx=0x7fdbd000b7b0 comp rx=0 tx=0).stop 2024-04-20T21:25:41.583 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:25:41.583+0000 7fdbe5800700 1 -- 172.21.15.189:0/1237057333 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdbe0074d80 msgr2=0x7fdbe019c5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:25:41.583 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:25:41.583+0000 7fdbe5800700 1 --2- 172.21.15.189:0/1237057333 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdbe0074d80 0x7fdbe019c5f0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fdbc800c8a0 tx=0x7fdbc800cc60 comp rx=0 tx=0).stop 2024-04-20T21:25:41.583 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:25:41.584+0000 7fdbe5800700 1 -- 172.21.15.189:0/1237057333 shutdown_connections 2024-04-20T21:25:41.583 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:25:41.584+0000 7fdbe5800700 1 --2- 172.21.15.189:0/1237057333 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fdbc406c680 0x7fdbc406eb30 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:25:41.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:25:41.584+0000 7fdbe5800700 1 --2- 172.21.15.189:0/1237057333 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdbe00737b0 0x7fdbe019c0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:25:41.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:25:41.584+0000 7fdbe5800700 1 --2- 172.21.15.189:0/1237057333 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdbe0074d80 0x7fdbe019c5f0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:25:41.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:25:41.584+0000 7fdbe5800700 1 -- 172.21.15.189:0/1237057333 >> 172.21.15.189:0/1237057333 conn(0x7fdbe00fb7f0 msgr2=0x7fdbe0101bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:25:41.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:25:41.584+0000 7fdbe5800700 1 -- 172.21.15.189:0/1237057333 shutdown_connections 2024-04-20T21:25:41.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:25:41.584+0000 7fdbe5800700 1 -- 172.21.15.189:0/1237057333 wait complete. 2024-04-20T21:25:42.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:41 smithi189 ceph-mon[21562]: from='osd.5 [v2:172.21.15.189:6816/893231043,v1:172.21.15.189:6817/893231043]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-20T21:25:42.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:41 smithi189 ceph-mon[21562]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-20T21:25:42.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:41 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:25:42.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:41 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:25:42.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:41 smithi062 ceph-mon[16221]: from='osd.5 [v2:172.21.15.189:6816/893231043,v1:172.21.15.189:6817/893231043]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-20T21:25:42.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:41 smithi062 ceph-mon[16221]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-20T21:25:42.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:41 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:25:42.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:41 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:25:42.377 DEBUG:teuthology.orchestra.run.smithi189:osd.5> sudo journalctl -f -n 0 -u ceph-61ae3628-ff5b-11ee-bc93-c7b262605968@osd.5.service 2024-04-20T21:25:42.382 INFO:tasks.cephadm:Waiting for 6 OSDs to come up... 2024-04-20T21:25:42.382 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-20T21:25:42.804 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T21:25:43.141 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:42 smithi189 ceph-mon[21562]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-20T21:25:43.142 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:42 smithi189 ceph-mon[21562]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-20T21:25:43.142 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:42 smithi189 ceph-mon[21562]: osdmap e37: 6 total, 5 up, 6 in 2024-04-20T21:25:43.142 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:42 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-20T21:25:43.142 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:42 smithi189 ceph-mon[21562]: from='osd.5 [v2:172.21.15.189:6816/893231043,v1:172.21.15.189:6817/893231043]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-04-20T21:25:43.142 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:42 smithi189 ceph-mon[21562]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-04-20T21:25:43.142 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:42 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:25:43.142 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:42 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:25:43.142 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 21:25:42 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-5[36899]: 2024-04-20T21:25:42.895+0000 7f4618c00700 -1 osd.5 0 waiting for initial osdmap 2024-04-20T21:25:43.143 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 21:25:42 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-5[36899]: 2024-04-20T21:25:42.914+0000 7f460fe00700 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-20T21:25:43.180 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:42 smithi062 ceph-mon[16221]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-20T21:25:43.180 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:42 smithi062 ceph-mon[16221]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-20T21:25:43.180 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:42 smithi062 ceph-mon[16221]: osdmap e37: 6 total, 5 up, 6 in 2024-04-20T21:25:43.180 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:42 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-20T21:25:43.180 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:42 smithi062 ceph-mon[16221]: from='osd.5 [v2:172.21.15.189:6816/893231043,v1:172.21.15.189:6817/893231043]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-04-20T21:25:43.180 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:42 smithi062 ceph-mon[16221]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-04-20T21:25:43.180 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:42 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:25:43.180 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:42 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:25:44.047 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:43 smithi062 ceph-mon[16221]: purged_snaps scrub starts 2024-04-20T21:25:44.047 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:43 smithi062 ceph-mon[16221]: purged_snaps scrub ok 2024-04-20T21:25:44.047 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:43 smithi062 ceph-mon[16221]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-04-20T21:25:44.047 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:43 smithi062 ceph-mon[16221]: osdmap e38: 6 total, 5 up, 6 in 2024-04-20T21:25:44.047 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:43 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-20T21:25:44.047 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:43 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-20T21:25:44.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:43 smithi189 ceph-mon[21562]: purged_snaps scrub starts 2024-04-20T21:25:44.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:43 smithi189 ceph-mon[21562]: purged_snaps scrub ok 2024-04-20T21:25:44.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:43 smithi189 ceph-mon[21562]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-04-20T21:25:44.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:43 smithi189 ceph-mon[21562]: osdmap e38: 6 total, 5 up, 6 in 2024-04-20T21:25:44.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:43 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-20T21:25:44.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:43 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-20T21:25:45.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:44 smithi062 ceph-mon[16221]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-20T21:25:45.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:44 smithi062 ceph-mon[16221]: osd.5 [v2:172.21.15.189:6816/893231043,v1:172.21.15.189:6817/893231043] boot 2024-04-20T21:25:45.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:44 smithi062 ceph-mon[16221]: osdmap e39: 6 total, 6 up, 6 in 2024-04-20T21:25:45.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:44 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-20T21:25:45.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:44 smithi189 ceph-mon[21562]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-20T21:25:45.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:44 smithi189 ceph-mon[21562]: osd.5 [v2:172.21.15.189:6816/893231043,v1:172.21.15.189:6817/893231043] boot 2024-04-20T21:25:45.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:44 smithi189 ceph-mon[21562]: osdmap e39: 6 total, 6 up, 6 in 2024-04-20T21:25:45.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:44 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-20T21:25:45.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:45.269+0000 7f811f600700 1 -- 172.21.15.62:0/3889721904 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8118103680 msgr2=0x7f8118105a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:25:45.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:45.269+0000 7f811f600700 1 --2- 172.21.15.62:0/3889721904 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8118103680 0x7f8118105a60 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f8104009910 tx=0x7f8104009c20 comp rx=0 tx=0).stop 2024-04-20T21:25:45.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:45.270+0000 7f811f600700 1 -- 172.21.15.62:0/3889721904 shutdown_connections 2024-04-20T21:25:45.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:45.270+0000 7f811f600700 1 --2- 172.21.15.62:0/3889721904 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8118103680 0x7f8118105a60 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:25:45.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:45.270+0000 7f811f600700 1 --2- 172.21.15.62:0/3889721904 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8118100d60 0x7f8118103140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:25:45.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:45.270+0000 7f811f600700 1 -- 172.21.15.62:0/3889721904 >> 172.21.15.62:0/3889721904 conn(0x7f81180fa780 msgr2=0x7f81180fcbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:25:45.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:45.270+0000 7f811f600700 1 -- 172.21.15.62:0/3889721904 shutdown_connections 2024-04-20T21:25:45.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:45.271+0000 7f811f600700 1 -- 172.21.15.62:0/3889721904 wait complete. 2024-04-20T21:25:45.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:45.272+0000 7f811f600700 1 Processor -- start 2024-04-20T21:25:45.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:45.272+0000 7f811f600700 1 -- start start 2024-04-20T21:25:45.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:45.272+0000 7f811f600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8118100d60 0x7f8118195af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:25:45.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:45.272+0000 7f811f600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8118103680 0x7f8118196030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:25:45.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:45.272+0000 7f811f600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8118196650 con 0x7f8118100d60 2024-04-20T21:25:45.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:45.272+0000 7f811f600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8118196790 con 0x7f8118103680 2024-04-20T21:25:45.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:45.273+0000 7f811cc00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8118100d60 0x7f8118195af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:25:45.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:45.273+0000 7f811cc00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8118100d60 0x7f8118195af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50004/0 (socket says 172.21.15.62:50004) 2024-04-20T21:25:45.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:45.273+0000 7f811cc00700 1 -- 172.21.15.62:0/974386861 learned_addr learned my addr 172.21.15.62:0/974386861 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:25:45.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:45.273+0000 7f8117e00700 1 --2- 172.21.15.62:0/974386861 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8118103680 0x7f8118196030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:25:45.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:45.273+0000 7f811cc00700 1 -- 172.21.15.62:0/974386861 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8118103680 msgr2=0x7f8118196030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:25:45.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:45.273+0000 7f811cc00700 1 --2- 172.21.15.62:0/974386861 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8118103680 0x7f8118196030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:25:45.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:45.273+0000 7f811cc00700 1 -- 172.21.15.62:0/974386861 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81040095f0 con 0x7f8118100d60 2024-04-20T21:25:45.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:45.274+0000 7f811cc00700 1 --2- 172.21.15.62:0/974386861 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8118100d60 0x7f8118195af0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f810c00d8d0 tx=0x7f810c00dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:25:45.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:45.274+0000 7f8115600700 1 -- 172.21.15.62:0/974386861 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f810c004a00 con 0x7f8118100d60 2024-04-20T21:25:45.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:45.274+0000 7f8115600700 1 -- 172.21.15.62:0/974386861 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f810c0054d0 con 0x7f8118100d60 2024-04-20T21:25:45.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:45.274+0000 7f8115600700 1 -- 172.21.15.62:0/974386861 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f810c00fc20 con 0x7f8118100d60 2024-04-20T21:25:45.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:45.274+0000 7f811f600700 1 -- 172.21.15.62:0/974386861 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f811819b240 con 0x7f8118100d60 2024-04-20T21:25:45.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:45.274+0000 7f811f600700 1 -- 172.21.15.62:0/974386861 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f811819b790 con 0x7f8118100d60 2024-04-20T21:25:45.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:45.276+0000 7f811f600700 1 -- 172.21.15.62:0/974386861 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f811818fc00 con 0x7f8118100d60 2024-04-20T21:25:45.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:45.277+0000 7f8115600700 1 -- 172.21.15.62:0/974386861 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f810c005020 con 0x7f8118100d60 2024-04-20T21:25:45.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:45.278+0000 7f8115600700 1 --2- 172.21.15.62:0/974386861 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f810006c680 0x7f810006eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:25:45.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:45.278+0000 7f8115600700 1 -- 172.21.15.62:0/974386861 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f810c08aef0 con 0x7f8118100d60 2024-04-20T21:25:45.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:45.278+0000 7f8117e00700 1 --2- 172.21.15.62:0/974386861 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f810006c680 0x7f810006eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:25:45.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:45.279+0000 7f8117e00700 1 --2- 172.21.15.62:0/974386861 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f810006c680 0x7f810006eb30 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f8104006010 tx=0x7f810400b560 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:25:45.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:45.283+0000 7f8115600700 1 -- 172.21.15.62:0/974386861 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f810c059300 con 0x7f8118100d60 2024-04-20T21:25:45.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:45.510+0000 7f811f600700 1 -- 172.21.15.62:0/974386861 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f811802cc30 con 0x7f8118100d60 2024-04-20T21:25:45.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:45.511+0000 7f8115600700 1 -- 172.21.15.62:0/974386861 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v40) v1 ==== 74+0+130 (secure 0 0 0) 0x7f810c0170a0 con 0x7f8118100d60 2024-04-20T21:25:45.511 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:25:45.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:45.514+0000 7f811f600700 1 -- 172.21.15.62:0/974386861 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f810006c680 msgr2=0x7f810006eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:25:45.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:45.514+0000 7f811f600700 1 --2- 172.21.15.62:0/974386861 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f810006c680 0x7f810006eb30 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f8104006010 tx=0x7f810400b560 comp rx=0 tx=0).stop 2024-04-20T21:25:45.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:45.514+0000 7f811f600700 1 -- 172.21.15.62:0/974386861 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8118100d60 msgr2=0x7f8118195af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:25:45.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:45.514+0000 7f811f600700 1 --2- 172.21.15.62:0/974386861 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8118100d60 0x7f8118195af0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f810c00d8d0 tx=0x7f810c00dbe0 comp rx=0 tx=0).stop 2024-04-20T21:25:45.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:45.515+0000 7f811f600700 1 -- 172.21.15.62:0/974386861 shutdown_connections 2024-04-20T21:25:45.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:45.515+0000 7f811f600700 1 --2- 172.21.15.62:0/974386861 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f810006c680 0x7f810006eb30 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:25:45.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:45.515+0000 7f811f600700 1 --2- 172.21.15.62:0/974386861 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8118100d60 0x7f8118195af0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:25:45.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:45.515+0000 7f811f600700 1 --2- 172.21.15.62:0/974386861 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8118103680 0x7f8118196030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:25:45.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:45.515+0000 7f811f600700 1 -- 172.21.15.62:0/974386861 >> 172.21.15.62:0/974386861 conn(0x7f81180fa780 msgr2=0x7f81180fcbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:25:45.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:45.515+0000 7f811f600700 1 -- 172.21.15.62:0/974386861 shutdown_connections 2024-04-20T21:25:45.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:45.515+0000 7f811f600700 1 -- 172.21.15.62:0/974386861 wait complete. 2024-04-20T21:25:46.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:45 smithi189 ceph-mon[21562]: osdmap e40: 6 total, 6 up, 6 in 2024-04-20T21:25:46.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:45 smithi189 ceph-mon[21562]: from='client.? 172.21.15.62:0/974386861' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-20T21:25:46.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:45 smithi189 ceph-mon[21562]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:25:46.250 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:45 smithi062 ceph-mon[16221]: osdmap e40: 6 total, 6 up, 6 in 2024-04-20T21:25:46.250 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:45 smithi062 ceph-mon[16221]: from='client.? 172.21.15.62:0/974386861' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-20T21:25:46.250 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:45 smithi062 ceph-mon[16221]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:25:46.327 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":40,"num_osds":6,"num_up_osds":6,"osd_up_since":1713648343,"num_in_osds":6,"osd_in_since":1713648319,"num_remapped_pgs":0} 2024-04-20T21:25:46.327 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-20T21:25:46.753 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T21:25:47.981 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:47 smithi062 ceph-mon[16221]: Detected new or changed devices on smithi189 2024-04-20T21:25:47.981 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:47 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:25:47.981 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:47 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:25:47.981 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:47 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-20T21:25:47.981 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:47 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-20T21:25:47.981 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:47 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-20T21:25:47.981 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:47 smithi062 ceph-mon[16221]: Adjusting osd_memory_target on smithi189 to 4971M 2024-04-20T21:25:47.981 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:47 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:25:47.981 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:47 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:25:47.981 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:47 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T21:25:47.981 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:47 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:25:47.981 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:47 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:25:47.982 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:47 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:25:47.982 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:47 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T21:25:47.982 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:47 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:25:47.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:47 smithi189 ceph-mon[21562]: Detected new or changed devices on smithi189 2024-04-20T21:25:47.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:47 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:25:47.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:47 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:25:47.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:47 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-20T21:25:47.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:47 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-20T21:25:47.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:47 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-20T21:25:47.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:47 smithi189 ceph-mon[21562]: Adjusting osd_memory_target on smithi189 to 4971M 2024-04-20T21:25:47.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:47 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:25:47.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:47 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:25:47.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:47 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T21:25:47.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:47 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:25:47.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:47 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:25:47.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:47 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:25:47.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:47 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T21:25:47.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:47 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:25:48.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:48.928+0000 7ff20e200700 1 -- 172.21.15.62:0/530171434 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff208102460 msgr2=0x7ff2081028d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:25:48.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:48.928+0000 7ff20e200700 1 --2- 172.21.15.62:0/530171434 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff208102460 0x7ff2081028d0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff1f4009910 tx=0x7ff1f4009c20 comp rx=0 tx=0).stop 2024-04-20T21:25:48.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:48.928+0000 7ff20e200700 1 -- 172.21.15.62:0/530171434 shutdown_connections 2024-04-20T21:25:48.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:48.928+0000 7ff20e200700 1 --2- 172.21.15.62:0/530171434 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff208102460 0x7ff2081028d0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:25:48.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:48.928+0000 7ff20e200700 1 --2- 172.21.15.62:0/530171434 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff208108460 0x7ff208108830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:25:48.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:48.928+0000 7ff20e200700 1 -- 172.21.15.62:0/530171434 >> 172.21.15.62:0/530171434 conn(0x7ff2080fdf40 msgr2=0x7ff208100350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:25:48.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:48.928+0000 7ff20e200700 1 -- 172.21.15.62:0/530171434 shutdown_connections 2024-04-20T21:25:48.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:48.928+0000 7ff20e200700 1 -- 172.21.15.62:0/530171434 wait complete. 2024-04-20T21:25:48.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:48.929+0000 7ff20e200700 1 Processor -- start 2024-04-20T21:25:48.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:48.930+0000 7ff20e200700 1 -- start start 2024-04-20T21:25:48.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:48.930+0000 7ff20e200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff208102460 0x7ff208198140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:25:48.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:48.930+0000 7ff20e200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff208108460 0x7ff208198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:25:48.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:48.930+0000 7ff20e200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff208198d60 con 0x7ff208108460 2024-04-20T21:25:48.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:48.930+0000 7ff20e200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff20819caf0 con 0x7ff208102460 2024-04-20T21:25:48.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:48.930+0000 7ff206a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff208108460 0x7ff208198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:25:48.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:48.930+0000 7ff206a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff208108460 0x7ff208198680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50018/0 (socket says 172.21.15.62:50018) 2024-04-20T21:25:48.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:48.930+0000 7ff206a00700 1 -- 172.21.15.62:0/3639903501 learned_addr learned my addr 172.21.15.62:0/3639903501 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:25:48.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:48.931+0000 7ff206a00700 1 -- 172.21.15.62:0/3639903501 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff208102460 msgr2=0x7ff208198140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:25:48.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:48.931+0000 7ff207400700 1 --2- 172.21.15.62:0/3639903501 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff208102460 0x7ff208198140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:25:48.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:48.931+0000 7ff206a00700 1 --2- 172.21.15.62:0/3639903501 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff208102460 0x7ff208198140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:25:48.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:48.931+0000 7ff206a00700 1 -- 172.21.15.62:0/3639903501 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1f40095f0 con 0x7ff208108460 2024-04-20T21:25:48.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:48.931+0000 7ff207400700 1 --2- 172.21.15.62:0/3639903501 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff208102460 0x7ff208198140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T21:25:48.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:48.931+0000 7ff206a00700 1 --2- 172.21.15.62:0/3639903501 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff208108460 0x7ff208198680 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7ff1f400b5c0 tx=0x7ff1f400be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:25:48.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:48.931+0000 7ff1ffe00700 1 -- 172.21.15.62:0/3639903501 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff1f401d070 con 0x7ff208108460 2024-04-20T21:25:48.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:48.932+0000 7ff1ffe00700 1 -- 172.21.15.62:0/3639903501 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff1f4022470 con 0x7ff208108460 2024-04-20T21:25:48.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:48.932+0000 7ff1ffe00700 1 -- 172.21.15.62:0/3639903501 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff1f400f670 con 0x7ff208108460 2024-04-20T21:25:48.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:48.932+0000 7ff20e200700 1 -- 172.21.15.62:0/3639903501 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff20819cd70 con 0x7ff208108460 2024-04-20T21:25:48.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:48.932+0000 7ff20e200700 1 -- 172.21.15.62:0/3639903501 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff20819d210 con 0x7ff208108460 2024-04-20T21:25:48.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:48.933+0000 7ff20e200700 1 -- 172.21.15.62:0/3639903501 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff20810a9f0 con 0x7ff208108460 2024-04-20T21:25:48.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:48.934+0000 7ff1ffe00700 1 -- 172.21.15.62:0/3639903501 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7ff1f4004c30 con 0x7ff208108460 2024-04-20T21:25:48.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:48.935+0000 7ff1ffe00700 1 --2- 172.21.15.62:0/3639903501 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7ff1ec06c680 0x7ff1ec06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:25:48.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:48.935+0000 7ff1ffe00700 1 -- 172.21.15.62:0/3639903501 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7ff1f408cb90 con 0x7ff208108460 2024-04-20T21:25:48.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:48.935+0000 7ff207400700 1 --2- 172.21.15.62:0/3639903501 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7ff1ec06c680 0x7ff1ec06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:25:48.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:48.936+0000 7ff207400700 1 --2- 172.21.15.62:0/3639903501 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7ff1ec06c680 0x7ff1ec06eb30 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ff1f8007900 tx=0x7ff1f8008040 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:25:48.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:48.940+0000 7ff1ffe00700 1 -- 172.21.15.62:0/3639903501 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7ff1f4057640 con 0x7ff208108460 2024-04-20T21:25:48.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:48 smithi189 ceph-mon[21562]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:25:49.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:48 smithi062 ceph-mon[16221]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:25:49.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:49.164+0000 7ff20e200700 1 -- 172.21.15.62:0/3639903501 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff20804f2a0 con 0x7ff208108460 2024-04-20T21:25:49.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:49.166+0000 7ff1ffe00700 1 -- 172.21.15.62:0/3639903501 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+11296 (secure 0 0 0) 0x7ff1f405ac60 con 0x7ff208108460 2024-04-20T21:25:49.166 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:25:49.167 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":40,"fsid":"61ae3628-ff5b-11ee-bc93-c7b262605968","created":"2024-04-20T21:18:19.951305+0000","modified":"2024-04-20T21:25:44.906419+0000","last_up_change":"2024-04-20T21:25:43.888770+0000","last_in_change":"2024-04-20T21:25:19.201764+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-20T21:24:02.500375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a3f8216-c641-4338-813d-d836efc3f4e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":2687023571},{"type":"v1","addr":"172.21.15.62:6803","nonce":2687023571}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":2687023571},{"type":"v1","addr":"172.21.15.62:6805","nonce":2687023571}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":2687023571},{"type":"v1","addr":"172.21.15.62:6809","nonce":2687023571}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":2687023571},{"type":"v1","addr":"172.21.15.62:6807","nonce":2687023571}]},"public_addr":"172.21.15.62:6803/2687023571","cluster_addr":"172.21.15.62:6805/2687023571","heartbeat_back_addr":"172.21.15.62:6809/2687023571","heartbeat_front_addr":"172.21.15.62:6807/2687023571","state":["exists","up"]},{"osd":1,"uuid":"f0c451a4-9002-4ddc-be0e-d8ddebd4418d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":1027109374},{"type":"v1","addr":"172.21.15.62:6811","nonce":1027109374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":1027109374},{"type":"v1","addr":"172.21.15.62:6813","nonce":1027109374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":1027109374},{"type":"v1","addr":"172.21.15.62:6817","nonce":1027109374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":1027109374},{"type":"v1","addr":"172.21.15.62:6815","nonce":1027109374}]},"public_addr":"172.21.15.62:6811/1027109374","cluster_addr":"172.21.15.62:6813/1027109374","heartbeat_back_addr":"172.21.15.62:6817/1027109374","heartbeat_front_addr":"172.21.15.62:6815/1027109374","state":["exists","up"]},{"osd":2,"uuid":"e1a8769d-5da2-49c8-942a-83a94c7ac761","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":47205739},{"type":"v1","addr":"172.21.15.62:6819","nonce":47205739}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":47205739},{"type":"v1","addr":"172.21.15.62:6821","nonce":47205739}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":47205739},{"type":"v1","addr":"172.21.15.62:6825","nonce":47205739}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6822","nonce":47205739},{"type":"v1","addr":"172.21.15.62:6823","nonce":47205739}]},"public_addr":"172.21.15.62:6819/47205739","cluster_addr":"172.21.15.62:6821/47205739","heartbeat_back_addr":"172.21.15.62:6825/47205739","heartbeat_front_addr":"172.21.15.62:6823/47205739","state":["exists","up"]},{"osd":3,"uuid":"eaa2fe2b-87f8-4d01-8ddf-e72f244f31b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":3169871320},{"type":"v1","addr":"172.21.15.189:6801","nonce":3169871320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":3169871320},{"type":"v1","addr":"172.21.15.189:6803","nonce":3169871320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":3169871320},{"type":"v1","addr":"172.21.15.189:6807","nonce":3169871320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":3169871320},{"type":"v1","addr":"172.21.15.189:6805","nonce":3169871320}]},"public_addr":"172.21.15.189:6801/3169871320","cluster_addr":"172.21.15.189:6803/3169871320","heartbeat_back_addr":"172.21.15.189:6807/3169871320","heartbeat_front_addr":"172.21.15.189:6805/3169871320","state":["exists","up"]},{"osd":4,"uuid":"72be8597-52e5-45fd-b4f2-06ac50df77e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":191133671},{"type":"v1","addr":"172.21.15.189:6809","nonce":191133671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":191133671},{"type":"v1","addr":"172.21.15.189:6811","nonce":191133671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":191133671},{"type":"v1","addr":"172.21.15.189:6815","nonce":191133671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":191133671},{"type":"v1","addr":"172.21.15.189:6813","nonce":191133671}]},"public_addr":"172.21.15.189:6809/191133671","cluster_addr":"172.21.15.189:6811/191133671","heartbeat_back_addr":"172.21.15.189:6815/191133671","heartbeat_front_addr":"172.21.15.189:6813/191133671","state":["exists","up"]},{"osd":5,"uuid":"c650c63b-41eb-4d4f-8e14-907c7b3614c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":893231043},{"type":"v1","addr":"172.21.15.189:6817","nonce":893231043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":893231043},{"type":"v1","addr":"172.21.15.189:6819","nonce":893231043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":893231043},{"type":"v1","addr":"172.21.15.189:6823","nonce":893231043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":893231043},{"type":"v1","addr":"172.21.15.189:6821","nonce":893231043}]},"public_addr":"172.21.15.189:6817/893231043","cluster_addr":"172.21.15.189:6819/893231043","heartbeat_back_addr":"172.21.15.189:6823/893231043","heartbeat_front_addr":"172.21.15.189:6821/893231043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T21:22:45.605185+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T21:23:21.062873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T21:24:00.502044+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T21:24:32.944093+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T21:25:06.278988+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T21:25:41.872982+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.62:0/1116048562":"2024-04-21T21:21:03.778739+0000","172.21.15.62:6800/831506538":"2024-04-21T21:21:03.778739+0000","172.21.15.62:0/4158176989":"2024-04-21T21:19:34.965981+0000","172.21.15.62:6800/4147738586":"2024-04-21T21:19:34.965981+0000","172.21.15.62:0/2525292829":"2024-04-21T21:18:52.990685+0000","172.21.15.62:0/301716985":"2024-04-21T21:21:03.778739+0000","172.21.15.62:0/634566145":"2024-04-21T21:18:52.990685+0000","172.21.15.62:0/3192460755":"2024-04-21T21:21:03.778739+0000","172.21.15.62:6801/831506538":"2024-04-21T21:21:03.778739+0000","172.21.15.62:6800/871423274":"2024-04-21T21:18:52.990685+0000","172.21.15.62:6801/4147738586":"2024-04-21T21:19:34.965981+0000","172.21.15.62:0/771955286":"2024-04-21T21:19:34.965981+0000","172.21.15.62:6801/871423274":"2024-04-21T21:18:52.990685+0000","172.21.15.62:0/3416563026":"2024-04-21T21:18:52.990685+0000","172.21.15.62:0/4274508569":"2024-04-21T21:19:34.965981+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-20T21:25:49.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:49.169+0000 7ff20e200700 1 -- 172.21.15.62:0/3639903501 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7ff1ec06c680 msgr2=0x7ff1ec06eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:25:49.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:49.169+0000 7ff20e200700 1 --2- 172.21.15.62:0/3639903501 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7ff1ec06c680 0x7ff1ec06eb30 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ff1f8007900 tx=0x7ff1f8008040 comp rx=0 tx=0).stop 2024-04-20T21:25:49.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:49.169+0000 7ff20e200700 1 -- 172.21.15.62:0/3639903501 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff208108460 msgr2=0x7ff208198680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:25:49.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:49.170+0000 7ff20e200700 1 --2- 172.21.15.62:0/3639903501 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff208108460 0x7ff208198680 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7ff1f400b5c0 tx=0x7ff1f400be00 comp rx=0 tx=0).stop 2024-04-20T21:25:49.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:49.170+0000 7ff20e200700 1 -- 172.21.15.62:0/3639903501 shutdown_connections 2024-04-20T21:25:49.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:49.170+0000 7ff20e200700 1 --2- 172.21.15.62:0/3639903501 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7ff1ec06c680 0x7ff1ec06eb30 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:25:49.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:49.170+0000 7ff20e200700 1 --2- 172.21.15.62:0/3639903501 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff208102460 0x7ff208198140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:25:49.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:49.170+0000 7ff20e200700 1 --2- 172.21.15.62:0/3639903501 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff208108460 0x7ff208198680 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:25:49.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:49.170+0000 7ff20e200700 1 -- 172.21.15.62:0/3639903501 >> 172.21.15.62:0/3639903501 conn(0x7ff2080fdf40 msgr2=0x7ff2080fe980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:25:49.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:49.170+0000 7ff20e200700 1 -- 172.21.15.62:0/3639903501 shutdown_connections 2024-04-20T21:25:49.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:49.170+0000 7ff20e200700 1 -- 172.21.15.62:0/3639903501 wait complete. 2024-04-20T21:25:49.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:49 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:25:49.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:49 smithi062 ceph-mon[16221]: from='client.? 172.21.15.62:0/3639903501' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-20T21:25:49.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:49 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:25:49.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:49 smithi189 ceph-mon[21562]: from='client.? 172.21.15.62:0/3639903501' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-20T21:25:50.036 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-20T21:24:02.500375+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-20T21:25:50.036 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-04-20T21:25:50.457 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T21:25:50.841 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:50 smithi062 ceph-mon[16221]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:25:50.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:50 smithi189 ceph-mon[21562]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:25:52.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:52.604+0000 7f7aec200700 1 -- 172.21.15.62:0/2614416538 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ae40687a0 msgr2=0x7f7ae4068b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:25:52.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:52.604+0000 7f7aec200700 1 --2- 172.21.15.62:0/2614416538 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ae40687a0 0x7f7ae4068b70 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f7ad8009910 tx=0x7f7ad8009c20 comp rx=0 tx=0).stop 2024-04-20T21:25:52.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:52.605+0000 7f7aec200700 1 -- 172.21.15.62:0/2614416538 shutdown_connections 2024-04-20T21:25:52.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:52.605+0000 7f7aec200700 1 --2- 172.21.15.62:0/2614416538 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ae4069140 0x7f7ae4104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:25:52.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:52.605+0000 7f7aec200700 1 --2- 172.21.15.62:0/2614416538 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ae40687a0 0x7f7ae4068b70 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:25:52.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:52.605+0000 7f7aec200700 1 -- 172.21.15.62:0/2614416538 >> 172.21.15.62:0/2614416538 conn(0x7f7ae4075130 msgr2=0x7f7ae4075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:25:52.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:52.605+0000 7f7aec200700 1 -- 172.21.15.62:0/2614416538 shutdown_connections 2024-04-20T21:25:52.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:52.605+0000 7f7aec200700 1 -- 172.21.15.62:0/2614416538 wait complete. 2024-04-20T21:25:52.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:52.606+0000 7f7aec200700 1 Processor -- start 2024-04-20T21:25:52.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:52.606+0000 7f7aec200700 1 -- start start 2024-04-20T21:25:52.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:52.606+0000 7f7aec200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ae40687a0 0x7f7ae40ffca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:25:52.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:52.606+0000 7f7aec200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ae4069140 0x7f7ae41001e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:25:52.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:52.606+0000 7f7aec200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ae4100720 con 0x7f7ae40687a0 2024-04-20T21:25:52.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:52.606+0000 7f7aec200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ae4100860 con 0x7f7ae4069140 2024-04-20T21:25:52.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:52.607+0000 7f7ae9800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ae40687a0 0x7f7ae40ffca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:25:52.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:52.607+0000 7f7ae9800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ae40687a0 0x7f7ae40ffca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:38102/0 (socket says 172.21.15.62:38102) 2024-04-20T21:25:52.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:52.607+0000 7f7ae9800700 1 -- 172.21.15.62:0/2177293727 learned_addr learned my addr 172.21.15.62:0/2177293727 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:25:52.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:52.607+0000 7f7ae8e00700 1 --2- 172.21.15.62:0/2177293727 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ae4069140 0x7f7ae41001e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:25:52.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:52.607+0000 7f7ae9800700 1 -- 172.21.15.62:0/2177293727 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ae4069140 msgr2=0x7f7ae41001e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:25:52.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:52.607+0000 7f7ae9800700 1 --2- 172.21.15.62:0/2177293727 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ae4069140 0x7f7ae41001e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:25:52.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:52.607+0000 7f7ae9800700 1 -- 172.21.15.62:0/2177293727 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ad80095f0 con 0x7f7ae40687a0 2024-04-20T21:25:52.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:52.607+0000 7f7ae9800700 1 --2- 172.21.15.62:0/2177293727 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ae40687a0 0x7f7ae40ffca0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f7ad8006010 tx=0x7f7ad800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:25:52.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:52.608+0000 7f7ae2000700 1 -- 172.21.15.62:0/2177293727 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7ad801d070 con 0x7f7ae40687a0 2024-04-20T21:25:52.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:52.608+0000 7f7ae2000700 1 -- 172.21.15.62:0/2177293727 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f7ad800f460 con 0x7f7ae40687a0 2024-04-20T21:25:52.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:52.608+0000 7f7ae2000700 1 -- 172.21.15.62:0/2177293727 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7ad8017630 con 0x7f7ae40687a0 2024-04-20T21:25:52.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:52.608+0000 7f7aec200700 1 -- 172.21.15.62:0/2177293727 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ae4100ae0 con 0x7f7ae40687a0 2024-04-20T21:25:52.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:52.608+0000 7f7aec200700 1 -- 172.21.15.62:0/2177293727 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ae41a2380 con 0x7f7ae40687a0 2024-04-20T21:25:52.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:52.609+0000 7f7aec200700 1 -- 172.21.15.62:0/2177293727 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ae4108790 con 0x7f7ae40687a0 2024-04-20T21:25:52.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:52.610+0000 7f7ae2000700 1 -- 172.21.15.62:0/2177293727 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f7ad800f5d0 con 0x7f7ae40687a0 2024-04-20T21:25:52.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:52.611+0000 7f7ae2000700 1 --2- 172.21.15.62:0/2177293727 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f7acc06c680 0x7f7acc06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:25:52.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:52.611+0000 7f7ae2000700 1 -- 172.21.15.62:0/2177293727 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f7ad808cad0 con 0x7f7ae40687a0 2024-04-20T21:25:52.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:52.611+0000 7f7ae8e00700 1 --2- 172.21.15.62:0/2177293727 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f7acc06c680 0x7f7acc06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:25:52.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:52.612+0000 7f7ae8e00700 1 --2- 172.21.15.62:0/2177293727 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f7acc06c680 0x7f7acc06eb30 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f7ad0005ea0 tx=0x7f7ad0005e30 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:25:52.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:52.616+0000 7f7ae2000700 1 -- 172.21.15.62:0/2177293727 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f7ad8057570 con 0x7f7ae40687a0 2024-04-20T21:25:52.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:52.840+0000 7f7aec200700 1 -- 172.21.15.62:0/2177293727 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7ae41012a0 con 0x7f7ae40687a0 2024-04-20T21:25:52.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:52.841+0000 7f7ae2000700 1 -- 172.21.15.62:0/2177293727 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v40) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7ad805ab90 con 0x7f7ae40687a0 2024-04-20T21:25:52.841 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 1 2024-04-20T21:25:52.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:52.845+0000 7f7aec200700 1 -- 172.21.15.62:0/2177293727 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f7acc06c680 msgr2=0x7f7acc06eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:25:52.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:52.845+0000 7f7aec200700 1 --2- 172.21.15.62:0/2177293727 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f7acc06c680 0x7f7acc06eb30 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f7ad0005ea0 tx=0x7f7ad0005e30 comp rx=0 tx=0).stop 2024-04-20T21:25:52.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:52.845+0000 7f7aec200700 1 -- 172.21.15.62:0/2177293727 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ae40687a0 msgr2=0x7f7ae40ffca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:25:52.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:52.845+0000 7f7aec200700 1 --2- 172.21.15.62:0/2177293727 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ae40687a0 0x7f7ae40ffca0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f7ad8006010 tx=0x7f7ad800be20 comp rx=0 tx=0).stop 2024-04-20T21:25:52.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:52.845+0000 7f7aec200700 1 -- 172.21.15.62:0/2177293727 shutdown_connections 2024-04-20T21:25:52.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:52.845+0000 7f7aec200700 1 --2- 172.21.15.62:0/2177293727 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f7acc06c680 0x7f7acc06eb30 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:25:52.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:52.845+0000 7f7aec200700 1 --2- 172.21.15.62:0/2177293727 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ae40687a0 0x7f7ae40ffca0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:25:52.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:52.845+0000 7f7aec200700 1 --2- 172.21.15.62:0/2177293727 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ae4069140 0x7f7ae41001e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:25:52.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:52.845+0000 7f7aec200700 1 -- 172.21.15.62:0/2177293727 >> 172.21.15.62:0/2177293727 conn(0x7f7ae4075130 msgr2=0x7f7ae40fe7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:25:52.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:52.845+0000 7f7aec200700 1 -- 172.21.15.62:0/2177293727 shutdown_connections 2024-04-20T21:25:52.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:52.845+0000 7f7aec200700 1 -- 172.21.15.62:0/2177293727 wait complete. 2024-04-20T21:25:53.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:52 smithi062 ceph-mon[16221]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:25:53.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:52 smithi189 ceph-mon[21562]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:25:53.550 INFO:tasks.cephadm:Setting up client nodes... 2024-04-20T21:25:53.551 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-20T21:25:53.980 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T21:25:54.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:53 smithi062 ceph-mon[16221]: from='client.? 172.21.15.62:0/2177293727' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-20T21:25:54.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:53 smithi189 ceph-mon[21562]: from='client.? 172.21.15.62:0/2177293727' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-20T21:25:55.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:54 smithi062 ceph-mon[16221]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:25:55.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:54 smithi189 ceph-mon[21562]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:25:56.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:56.186+0000 7f2c4f200700 1 -- 172.21.15.62:0/276121231 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c50100050 msgr2=0x7f2c50100460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:25:56.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:56.186+0000 7f2c4f200700 1 --2- 172.21.15.62:0/276121231 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c50100050 0x7f2c50100460 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f2c40009910 tx=0x7f2c40009c20 comp rx=0 tx=0).stop 2024-04-20T21:25:56.187 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:55 smithi062 ceph-mon[16221]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:25:56.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:56.186+0000 7f2c4f200700 1 -- 172.21.15.62:0/276121231 shutdown_connections 2024-04-20T21:25:56.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:56.186+0000 7f2c4f200700 1 --2- 172.21.15.62:0/276121231 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c50101250 0x7f2c501016a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:25:56.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:56.186+0000 7f2c4f200700 1 --2- 172.21.15.62:0/276121231 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c50100050 0x7f2c50100460 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:25:56.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:56.186+0000 7f2c4f200700 1 -- 172.21.15.62:0/276121231 >> 172.21.15.62:0/276121231 conn(0x7f2c500fb5e0 msgr2=0x7f2c500fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:25:56.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:56.187+0000 7f2c4f200700 1 -- 172.21.15.62:0/276121231 shutdown_connections 2024-04-20T21:25:56.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:56.187+0000 7f2c4f200700 1 -- 172.21.15.62:0/276121231 wait complete. 2024-04-20T21:25:56.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:56.188+0000 7f2c4f200700 1 Processor -- start 2024-04-20T21:25:56.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:56.188+0000 7f2c4f200700 1 -- start start 2024-04-20T21:25:56.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:56.188+0000 7f2c4f200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c50100050 0x7f2c50197a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:25:56.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:56.188+0000 7f2c4f200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c50101250 0x7f2c50197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:25:56.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:56.188+0000 7f2c4f200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c501985d0 con 0x7f2c50100050 2024-04-20T21:25:56.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:56.188+0000 7f2c4f200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c50198710 con 0x7f2c50101250 2024-04-20T21:25:56.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:56.189+0000 7f2c4de00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c50100050 0x7f2c50197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:25:56.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:56.189+0000 7f2c4de00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c50100050 0x7f2c50197a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:38122/0 (socket says 172.21.15.62:38122) 2024-04-20T21:25:56.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:56.189+0000 7f2c4de00700 1 -- 172.21.15.62:0/388720885 learned_addr learned my addr 172.21.15.62:0/388720885 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:25:56.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:56.189+0000 7f2c4d400700 1 --2- 172.21.15.62:0/388720885 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c50101250 0x7f2c50197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:25:56.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:56.189+0000 7f2c4de00700 1 -- 172.21.15.62:0/388720885 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c50101250 msgr2=0x7f2c50197fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:25:56.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:56.189+0000 7f2c4de00700 1 --2- 172.21.15.62:0/388720885 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c50101250 0x7f2c50197fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:25:56.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:56.189+0000 7f2c4de00700 1 -- 172.21.15.62:0/388720885 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c400095f0 con 0x7f2c50100050 2024-04-20T21:25:56.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:56.189+0000 7f2c4d400700 1 --2- 172.21.15.62:0/388720885 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c50101250 0x7f2c50197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-20T21:25:56.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:56.190+0000 7f2c4de00700 1 --2- 172.21.15.62:0/388720885 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c50100050 0x7f2c50197a70 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f2c4000bda0 tx=0x7f2c4000be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:25:56.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:56.190+0000 7f2c46a00700 1 -- 172.21.15.62:0/388720885 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2c4001d070 con 0x7f2c50100050 2024-04-20T21:25:56.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:56.190+0000 7f2c46a00700 1 -- 172.21.15.62:0/388720885 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f2c40004c50 con 0x7f2c50100050 2024-04-20T21:25:56.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:56.190+0000 7f2c4f200700 1 -- 172.21.15.62:0/388720885 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c5019d160 con 0x7f2c50100050 2024-04-20T21:25:56.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:56.190+0000 7f2c46a00700 1 -- 172.21.15.62:0/388720885 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2c4000f690 con 0x7f2c50100050 2024-04-20T21:25:56.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:56.191+0000 7f2c4f200700 1 -- 172.21.15.62:0/388720885 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c5019d650 con 0x7f2c50100050 2024-04-20T21:25:56.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:56.192+0000 7f2c4f200700 1 -- 172.21.15.62:0/388720885 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c50105330 con 0x7f2c50100050 2024-04-20T21:25:56.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:56.193+0000 7f2c46a00700 1 -- 172.21.15.62:0/388720885 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f2c40022a20 con 0x7f2c50100050 2024-04-20T21:25:56.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:56.194+0000 7f2c46a00700 1 --2- 172.21.15.62:0/388720885 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f2c3406c680 0x7f2c3406eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:25:56.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:56.194+0000 7f2c46a00700 1 -- 172.21.15.62:0/388720885 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f2c4008cc90 con 0x7f2c50100050 2024-04-20T21:25:56.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:56.195+0000 7f2c4d400700 1 --2- 172.21.15.62:0/388720885 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f2c3406c680 0x7f2c3406eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:25:56.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:56.195+0000 7f2c4d400700 1 --2- 172.21.15.62:0/388720885 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f2c3406c680 0x7f2c3406eb30 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f2c3c005950 tx=0x7f2c3c005f50 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:25:56.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:56.199+0000 7f2c46a00700 1 -- 172.21.15.62:0/388720885 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f2c4005b190 con 0x7f2c50100050 2024-04-20T21:25:56.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:55 smithi189 ceph-mon[21562]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:25:56.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:56.506+0000 7f2c4f200700 1 -- 172.21.15.62:0/388720885 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f2c5002cc30 con 0x7f2c50100050 2024-04-20T21:25:56.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:56.512+0000 7f2c46a00700 1 -- 172.21.15.62:0/388720885 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v16) v1 ==== 170+0+59 (secure 0 0 0) 0x7f2c4005ad20 con 0x7f2c50100050 2024-04-20T21:25:56.511 INFO:teuthology.orchestra.run.smithi062.stdout:[client.0] 2024-04-20T21:25:56.512 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQDkMiRm+f9PHhAA/A9Qbp8GI27nX3oFB2Gvjw== 2024-04-20T21:25:56.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:56.515+0000 7f2c4f200700 1 -- 172.21.15.62:0/388720885 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f2c3406c680 msgr2=0x7f2c3406eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:25:56.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:56.515+0000 7f2c4f200700 1 --2- 172.21.15.62:0/388720885 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f2c3406c680 0x7f2c3406eb30 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f2c3c005950 tx=0x7f2c3c005f50 comp rx=0 tx=0).stop 2024-04-20T21:25:56.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:56.515+0000 7f2c4f200700 1 -- 172.21.15.62:0/388720885 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c50100050 msgr2=0x7f2c50197a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:25:56.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:56.515+0000 7f2c4f200700 1 --2- 172.21.15.62:0/388720885 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c50100050 0x7f2c50197a70 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f2c4000bda0 tx=0x7f2c4000be80 comp rx=0 tx=0).stop 2024-04-20T21:25:56.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:56.515+0000 7f2c4f200700 1 -- 172.21.15.62:0/388720885 shutdown_connections 2024-04-20T21:25:56.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:56.515+0000 7f2c4f200700 1 --2- 172.21.15.62:0/388720885 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f2c3406c680 0x7f2c3406eb30 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:25:56.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:56.515+0000 7f2c4f200700 1 --2- 172.21.15.62:0/388720885 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c50100050 0x7f2c50197a70 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:25:56.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:56.515+0000 7f2c4f200700 1 --2- 172.21.15.62:0/388720885 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c50101250 0x7f2c50197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:25:56.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:56.515+0000 7f2c4f200700 1 -- 172.21.15.62:0/388720885 >> 172.21.15.62:0/388720885 conn(0x7f2c500fb5e0 msgr2=0x7f2c50102470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:25:56.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:56.516+0000 7f2c4f200700 1 -- 172.21.15.62:0/388720885 shutdown_connections 2024-04-20T21:25:56.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:25:56.516+0000 7f2c4f200700 1 -- 172.21.15.62:0/388720885 wait complete. 2024-04-20T21:25:57.189 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:56 smithi062 ceph-mon[16221]: from='client.? 172.21.15.62:0/388720885' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-20T21:25:57.189 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:56 smithi062 ceph-mon[16221]: from='client.? 172.21.15.62:0/388720885' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-20T21:25:57.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:56 smithi189 ceph-mon[21562]: from='client.? 172.21.15.62:0/388720885' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-20T21:25:57.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:56 smithi189 ceph-mon[21562]: from='client.? 172.21.15.62:0/388720885' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-20T21:25:57.263 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-04-20T21:25:57.263 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-20T21:25:57.263 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-20T21:25:57.386 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-20T21:25:57.803 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi189/config 2024-04-20T21:25:58.218 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:57 smithi062 ceph-mon[16221]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:25:58.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:57 smithi189 ceph-mon[21562]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:25:59.898 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:25:59.898+0000 7fc7c4600700 1 -- 172.21.15.189:0/424474327 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc7bc074d80 msgr2=0x7fc7bc0731e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:25:59.898 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:25:59.898+0000 7fc7c4600700 1 --2- 172.21.15.189:0/424474327 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc7bc074d80 0x7fc7bc0731e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fc7b0009910 tx=0x7fc7b0009c20 comp rx=0 tx=0).stop 2024-04-20T21:25:59.898 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:25:59.898+0000 7fc7c4600700 1 -- 172.21.15.189:0/424474327 shutdown_connections 2024-04-20T21:25:59.898 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:25:59.898+0000 7fc7c4600700 1 --2- 172.21.15.189:0/424474327 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc7bc0737b0 0x7fc7bc073c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:25:59.899 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:25:59.898+0000 7fc7c4600700 1 --2- 172.21.15.189:0/424474327 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc7bc074d80 0x7fc7bc0731e0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:25:59.899 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:25:59.898+0000 7fc7c4600700 1 -- 172.21.15.189:0/424474327 >> 172.21.15.189:0/424474327 conn(0x7fc7bc0fb7f0 msgr2=0x7fc7bc0fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:25:59.899 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:25:59.898+0000 7fc7c4600700 1 -- 172.21.15.189:0/424474327 shutdown_connections 2024-04-20T21:25:59.899 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:25:59.898+0000 7fc7c4600700 1 -- 172.21.15.189:0/424474327 wait complete. 2024-04-20T21:25:59.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:25:59.899+0000 7fc7c4600700 1 Processor -- start 2024-04-20T21:25:59.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:25:59.900+0000 7fc7c4600700 1 -- start start 2024-04-20T21:25:59.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:25:59.900+0000 7fc7c4600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc7bc0737b0 0x7fc7bc197d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:25:59.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:25:59.900+0000 7fc7c4600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc7bc074d80 0x7fc7bc198260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:25:59.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:25:59.900+0000 7fc7c4600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7bc198880 con 0x7fc7bc0737b0 2024-04-20T21:25:59.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:25:59.900+0000 7fc7c4600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7bc19d290 con 0x7fc7bc074d80 2024-04-20T21:25:59.901 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:25:59.900+0000 7fc7c1200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc7bc074d80 0x7fc7bc198260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:25:59.902 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:25:59.900+0000 7fc7c1200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc7bc074d80 0x7fc7bc198260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:43752/0 (socket says 172.21.15.189:43752) 2024-04-20T21:25:59.902 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:25:59.900+0000 7fc7c1200700 1 -- 172.21.15.189:0/2849963429 learned_addr learned my addr 172.21.15.189:0/2849963429 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-20T21:25:59.902 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:25:59.901+0000 7fc7c1c00700 1 --2- 172.21.15.189:0/2849963429 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc7bc0737b0 0x7fc7bc197d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:25:59.902 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:25:59.901+0000 7fc7c1200700 1 -- 172.21.15.189:0/2849963429 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc7bc0737b0 msgr2=0x7fc7bc197d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:25:59.902 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:25:59.901+0000 7fc7c1200700 1 --2- 172.21.15.189:0/2849963429 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc7bc0737b0 0x7fc7bc197d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:25:59.902 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:25:59.901+0000 7fc7c1200700 1 -- 172.21.15.189:0/2849963429 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7b00095f0 con 0x7fc7bc074d80 2024-04-20T21:25:59.902 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:25:59.901+0000 7fc7c1200700 1 --2- 172.21.15.189:0/2849963429 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc7bc074d80 0x7fc7bc198260 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc7a800b4b0 tx=0x7fc7a800b870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:25:59.902 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:25:59.901+0000 7fc7ba000700 1 -- 172.21.15.189:0/2849963429 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc7a8011800 con 0x7fc7bc074d80 2024-04-20T21:25:59.902 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:25:59.901+0000 7fc7ba000700 1 -- 172.21.15.189:0/2849963429 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc7a8011e10 con 0x7fc7bc074d80 2024-04-20T21:25:59.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:25:59.901+0000 7fc7ba000700 1 -- 172.21.15.189:0/2849963429 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc7a800f2e0 con 0x7fc7bc074d80 2024-04-20T21:25:59.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:25:59.901+0000 7fc7c4600700 1 -- 172.21.15.189:0/2849963429 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7bc19d490 con 0x7fc7bc074d80 2024-04-20T21:25:59.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:25:59.902+0000 7fc7c4600700 1 -- 172.21.15.189:0/2849963429 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7bc19d9b0 con 0x7fc7bc074d80 2024-04-20T21:25:59.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:25:59.903+0000 7fc7c4600700 1 -- 172.21.15.189:0/2849963429 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7bc10b000 con 0x7fc7bc074d80 2024-04-20T21:25:59.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:25:59.904+0000 7fc7ba000700 1 -- 172.21.15.189:0/2849963429 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7fc7a8010380 con 0x7fc7bc074d80 2024-04-20T21:25:59.904 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:25:59.905+0000 7fc7ba000700 1 --2- 172.21.15.189:0/2849963429 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fc7a406c680 0x7fc7a406eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:25:59.904 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:25:59.905+0000 7fc7ba000700 1 -- 172.21.15.189:0/2849963429 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fc7a808b090 con 0x7fc7bc074d80 2024-04-20T21:25:59.904 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:25:59.905+0000 7fc7c1c00700 1 --2- 172.21.15.189:0/2849963429 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fc7a406c680 0x7fc7a406eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:25:59.906 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:25:59.907+0000 7fc7c1c00700 1 --2- 172.21.15.189:0/2849963429 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fc7a406c680 0x7fc7a406eb30 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc7b0006010 tx=0x7fc7b000b560 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:25:59.908 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:25:59.909+0000 7fc7ba000700 1 -- 172.21.15.189:0/2849963429 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fc7a8055b30 con 0x7fc7bc074d80 2024-04-20T21:25:59.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:25:59 smithi189 ceph-mon[21562]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:26:00.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:26:00.205+0000 7fc7c4600700 1 -- 172.21.15.189:0/2849963429 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7fc7bc19e070 con 0x7fc7bc074d80 2024-04-20T21:26:00.218 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:26:00.218+0000 7fc7ba000700 1 -- 172.21.15.189:0/2849963429 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v17) v1 ==== 170+0+59 (secure 0 0 0) 0x7fc7a8059150 con 0x7fc7bc074d80 2024-04-20T21:26:00.218 INFO:teuthology.orchestra.run.smithi189.stdout:[client.1] 2024-04-20T21:26:00.218 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQDoMiRm9otzDBAAHMqp6bvG4f80mKhUb4N6Cw== 2024-04-20T21:26:00.221 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:26:00.221+0000 7fc7c4600700 1 -- 172.21.15.189:0/2849963429 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fc7a406c680 msgr2=0x7fc7a406eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:26:00.221 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:26:00.221+0000 7fc7c4600700 1 --2- 172.21.15.189:0/2849963429 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fc7a406c680 0x7fc7a406eb30 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc7b0006010 tx=0x7fc7b000b560 comp rx=0 tx=0).stop 2024-04-20T21:26:00.221 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:26:00.221+0000 7fc7c4600700 1 -- 172.21.15.189:0/2849963429 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc7bc074d80 msgr2=0x7fc7bc198260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:26:00.221 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:26:00.221+0000 7fc7c4600700 1 --2- 172.21.15.189:0/2849963429 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc7bc074d80 0x7fc7bc198260 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc7a800b4b0 tx=0x7fc7a800b870 comp rx=0 tx=0).stop 2024-04-20T21:26:00.221 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:26:00.221+0000 7fc7c4600700 1 -- 172.21.15.189:0/2849963429 shutdown_connections 2024-04-20T21:26:00.221 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:26:00.221+0000 7fc7c4600700 1 --2- 172.21.15.189:0/2849963429 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fc7a406c680 0x7fc7a406eb30 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:00.222 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:26:00.221+0000 7fc7c4600700 1 --2- 172.21.15.189:0/2849963429 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc7bc0737b0 0x7fc7bc197d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:00.222 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:26:00.221+0000 7fc7c4600700 1 --2- 172.21.15.189:0/2849963429 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc7bc074d80 0x7fc7bc198260 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:00.222 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:26:00.221+0000 7fc7c4600700 1 -- 172.21.15.189:0/2849963429 >> 172.21.15.189:0/2849963429 conn(0x7fc7bc0fb7f0 msgr2=0x7fc7bc106020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:26:00.222 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:26:00.222+0000 7fc7c4600700 1 -- 172.21.15.189:0/2849963429 shutdown_connections 2024-04-20T21:26:00.222 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:26:00.222+0000 7fc7c4600700 1 -- 172.21.15.189:0/2849963429 wait complete. 2024-04-20T21:26:00.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:25:59 smithi062 ceph-mon[16221]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:26:01.094 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-20T21:26:01.094 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-04-20T21:26:01.094 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-04-20T21:26:01.180 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:26:00 smithi189 ceph-mon[21562]: from='client.? 172.21.15.189:0/2849963429' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-20T21:26:01.180 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:26:00 smithi189 ceph-mon[21562]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-20T21:26:01.180 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:26:00 smithi189 ceph-mon[21562]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-20T21:26:01.215 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-20T21:26:01.215 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-20T21:26:01.215 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -- ceph mgr dump --format=json 2024-04-20T21:26:01.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:26:00 smithi062 ceph-mon[16221]: from='client.? 172.21.15.189:0/2849963429' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-20T21:26:01.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:26:00 smithi062 ceph-mon[16221]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-20T21:26:01.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:26:00 smithi062 ceph-mon[16221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-20T21:26:01.654 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T21:26:03.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:26:02 smithi062 ceph-mon[16221]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:26:03.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:26:02 smithi189 ceph-mon[21562]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:26:03.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:03.885+0000 7f688d000700 1 -- 172.21.15.62:0/541180537 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6888068540 msgr2=0x7f68880689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:26:03.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:03.885+0000 7f688d000700 1 --2- 172.21.15.62:0/541180537 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6888068540 0x7f68880689b0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f6874009910 tx=0x7f6874009c20 comp rx=0 tx=0).stop 2024-04-20T21:26:03.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:03.885+0000 7f688d000700 1 -- 172.21.15.62:0/541180537 shutdown_connections 2024-04-20T21:26:03.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:03.885+0000 7f688d000700 1 --2- 172.21.15.62:0/541180537 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6888068540 0x7f68880689b0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:03.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:03.885+0000 7f688d000700 1 --2- 172.21.15.62:0/541180537 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68881061b0 0x7f6888106580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:03.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:03.885+0000 7f688d000700 1 -- 172.21.15.62:0/541180537 >> 172.21.15.62:0/541180537 conn(0x7f6888074fa0 msgr2=0x7f68880753a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:26:03.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:03.885+0000 7f688d000700 1 -- 172.21.15.62:0/541180537 shutdown_connections 2024-04-20T21:26:03.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:03.885+0000 7f688d000700 1 -- 172.21.15.62:0/541180537 wait complete. 2024-04-20T21:26:03.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:03.886+0000 7f688d000700 1 Processor -- start 2024-04-20T21:26:03.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:03.886+0000 7f688d000700 1 -- start start 2024-04-20T21:26:03.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:03.887+0000 7f688d000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6888068540 0x7f6888197d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:26:03.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:03.887+0000 7f688d000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f68881061b0 0x7f6888198280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:26:03.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:03.887+0000 7f688d000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6888198960 con 0x7f68881061b0 2024-04-20T21:26:03.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:03.887+0000 7f688d000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f688819c6f0 con 0x7f6888068540 2024-04-20T21:26:03.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:03.887+0000 7f6886a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f68881061b0 0x7f6888198280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:26:03.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:03.887+0000 7f6886a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f68881061b0 0x7f6888198280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50590/0 (socket says 172.21.15.62:50590) 2024-04-20T21:26:03.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:03.887+0000 7f6886a00700 1 -- 172.21.15.62:0/3680588466 learned_addr learned my addr 172.21.15.62:0/3680588466 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:26:03.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:03.887+0000 7f6887400700 1 --2- 172.21.15.62:0/3680588466 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6888068540 0x7f6888197d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:26:03.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:03.888+0000 7f6886a00700 1 -- 172.21.15.62:0/3680588466 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6888068540 msgr2=0x7f6888197d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:26:03.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:03.888+0000 7f6886a00700 1 --2- 172.21.15.62:0/3680588466 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6888068540 0x7f6888197d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:03.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:03.888+0000 7f6886a00700 1 -- 172.21.15.62:0/3680588466 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68740095f0 con 0x7f68881061b0 2024-04-20T21:26:03.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:03.888+0000 7f6886a00700 1 --2- 172.21.15.62:0/3680588466 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f68881061b0 0x7f6888198280 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f6874006010 tx=0x7f687400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:26:03.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:03.888+0000 7f687fe00700 1 -- 172.21.15.62:0/3680588466 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f687401d070 con 0x7f68881061b0 2024-04-20T21:26:03.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:03.888+0000 7f687fe00700 1 -- 172.21.15.62:0/3680588466 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f687400f460 con 0x7f68881061b0 2024-04-20T21:26:03.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:03.888+0000 7f687fe00700 1 -- 172.21.15.62:0/3680588466 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6874021640 con 0x7f68881061b0 2024-04-20T21:26:03.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:03.888+0000 7f688d000700 1 -- 172.21.15.62:0/3680588466 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f688819c970 con 0x7f68881061b0 2024-04-20T21:26:03.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:03.889+0000 7f688d000700 1 -- 172.21.15.62:0/3680588466 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f688819ce60 con 0x7f68881061b0 2024-04-20T21:26:03.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:03.890+0000 7f688d000700 1 -- 172.21.15.62:0/3680588466 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6888100a20 con 0x7f68881061b0 2024-04-20T21:26:03.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:03.891+0000 7f687fe00700 1 -- 172.21.15.62:0/3680588466 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f687400fab0 con 0x7f68881061b0 2024-04-20T21:26:03.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:03.892+0000 7f687fe00700 1 --2- 172.21.15.62:0/3680588466 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f686c06c680 0x7f686c06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:26:03.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:03.893+0000 7f687fe00700 1 -- 172.21.15.62:0/3680588466 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f687408c8a0 con 0x7f68881061b0 2024-04-20T21:26:03.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:03.893+0000 7f6887400700 1 --2- 172.21.15.62:0/3680588466 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f686c06c680 0x7f686c06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:26:03.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:03.894+0000 7f6887400700 1 --2- 172.21.15.62:0/3680588466 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f686c06c680 0x7f686c06eb30 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f6878005fd0 tx=0x7f687800a560 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:26:03.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:03.897+0000 7f687fe00700 1 -- 172.21.15.62:0/3680588466 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f68740572a0 con 0x7f68881061b0 2024-04-20T21:26:03.970 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:26:03 smithi062 ceph-mon[16221]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:26:03.971 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:26:03 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:26:04.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:04.183+0000 7f688d000700 1 -- 172.21.15.62:0/3680588466 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f688819d190 con 0x7f68881061b0 2024-04-20T21:26:04.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:04.188+0000 7f687fe00700 1 -- 172.21.15.62:0/3680588466 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v19) v1 ==== 74+0+172853 (secure 0 0 0) 0x7f6874026090 con 0x7f68881061b0 2024-04-20T21:26:04.188 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:26:04.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:04.192+0000 7f688d000700 1 -- 172.21.15.62:0/3680588466 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f686c06c680 msgr2=0x7f686c06eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:26:04.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:04.192+0000 7f688d000700 1 --2- 172.21.15.62:0/3680588466 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f686c06c680 0x7f686c06eb30 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f6878005fd0 tx=0x7f687800a560 comp rx=0 tx=0).stop 2024-04-20T21:26:04.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:04.192+0000 7f688d000700 1 -- 172.21.15.62:0/3680588466 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f68881061b0 msgr2=0x7f6888198280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:26:04.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:04.192+0000 7f688d000700 1 --2- 172.21.15.62:0/3680588466 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f68881061b0 0x7f6888198280 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f6874006010 tx=0x7f687400be20 comp rx=0 tx=0).stop 2024-04-20T21:26:04.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:04.193+0000 7f688d000700 1 -- 172.21.15.62:0/3680588466 shutdown_connections 2024-04-20T21:26:04.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:04.193+0000 7f688d000700 1 --2- 172.21.15.62:0/3680588466 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f686c06c680 0x7f686c06eb30 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:04.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:04.193+0000 7f688d000700 1 --2- 172.21.15.62:0/3680588466 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6888068540 0x7f6888197d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:04.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:04.193+0000 7f688d000700 1 --2- 172.21.15.62:0/3680588466 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f68881061b0 0x7f6888198280 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:04.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:04.193+0000 7f688d000700 1 -- 172.21.15.62:0/3680588466 >> 172.21.15.62:0/3680588466 conn(0x7f6888074fa0 msgr2=0x7f688810a5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:26:04.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:04.193+0000 7f688d000700 1 -- 172.21.15.62:0/3680588466 shutdown_connections 2024-04-20T21:26:04.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:04.193+0000 7f688d000700 1 -- 172.21.15.62:0/3680588466 wait complete. 2024-04-20T21:26:04.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:26:03 smithi189 ceph-mon[21562]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:26:04.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:26:03 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:26:04.970 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":19,"active_gid":14215,"active_name":"smithi062.rqxuxi","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":4116451241},{"type":"v1","addr":"172.21.15.62:6801","nonce":4116451241}]},"active_addr":"172.21.15.62:6801/4116451241","active_change":"2024-04-20T21:21:03.778920+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":14240,"name":"smithi189.tawavz","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_password":{"name":"alertmanager_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web password","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_user":{"name":"alertmanager_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web user","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph:v18","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"prometheus_web_password":{"name":"prometheus_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web password","long_desc":"","tags":[],"see_also":[]},"prometheus_web_user":{"name":"prometheus_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web user","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_password":{"name":"alertmanager_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web password","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_user":{"name":"alertmanager_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web user","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph:v18","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"prometheus_web_password":{"name":"prometheus_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web password","long_desc":"","tags":[],"see_also":[]},"prometheus_web_user":{"name":"prometheus_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web user","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.62:8443/","prometheus":"http://172.21.15.62:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":5,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.62:0","nonce":868116748}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.62:0","nonce":3673769314}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.62:0","nonce":2348636520}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.62:0","nonce":717708557}]}]} 2024-04-20T21:26:04.974 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-20T21:26:04.974 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-20T21:26:04.974 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-20T21:26:05.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:26:04 smithi189 ceph-mon[21562]: from='client.? 172.21.15.62:0/3680588466' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-20T21:26:05.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:26:04 smithi062 ceph-mon[16221]: from='client.? 172.21.15.62:0/3680588466' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-20T21:26:05.385 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T21:26:06.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:26:05 smithi189 ceph-mon[21562]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:26:06.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:26:05 smithi062 ceph-mon[16221]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:26:07.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:07.610+0000 7fe56d400700 1 -- 172.21.15.62:0/1214905847 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe568101c00 msgr2=0x7fe56810a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:26:07.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:07.610+0000 7fe56d400700 1 --2- 172.21.15.62:0/1214905847 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe568101c00 0x7fe56810a2d0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fe550009910 tx=0x7fe550009c20 comp rx=0 tx=0).stop 2024-04-20T21:26:07.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:07.610+0000 7fe56d400700 1 -- 172.21.15.62:0/1214905847 shutdown_connections 2024-04-20T21:26:07.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:07.610+0000 7fe56d400700 1 --2- 172.21.15.62:0/1214905847 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe568101c00 0x7fe56810a2d0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:07.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:07.610+0000 7fe56d400700 1 --2- 172.21.15.62:0/1214905847 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe5681012f0 0x7fe5681016c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:07.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:07.610+0000 7fe56d400700 1 -- 172.21.15.62:0/1214905847 >> 172.21.15.62:0/1214905847 conn(0x7fe5680fac80 msgr2=0x7fe5680fd090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:26:07.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:07.610+0000 7fe56d400700 1 -- 172.21.15.62:0/1214905847 shutdown_connections 2024-04-20T21:26:07.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:07.610+0000 7fe56d400700 1 -- 172.21.15.62:0/1214905847 wait complete. 2024-04-20T21:26:07.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:07.612+0000 7fe56d400700 1 Processor -- start 2024-04-20T21:26:07.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:07.612+0000 7fe56d400700 1 -- start start 2024-04-20T21:26:07.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:07.612+0000 7fe56d400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe5681012f0 0x7fe568195e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:26:07.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:07.612+0000 7fe56d400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe568101c00 0x7fe5681963b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:26:07.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:07.612+0000 7fe56d400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe568196a90 con 0x7fe568101c00 2024-04-20T21:26:07.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:07.612+0000 7fe56d400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe56819a820 con 0x7fe5681012f0 2024-04-20T21:26:07.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:07.613+0000 7fe55fe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe568101c00 0x7fe5681963b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:26:07.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:07.613+0000 7fe55fe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe568101c00 0x7fe5681963b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50606/0 (socket says 172.21.15.62:50606) 2024-04-20T21:26:07.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:07.613+0000 7fe55fe00700 1 -- 172.21.15.62:0/865460659 learned_addr learned my addr 172.21.15.62:0/865460659 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:26:07.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:07.613+0000 7fe566800700 1 --2- 172.21.15.62:0/865460659 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe5681012f0 0x7fe568195e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:26:07.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:07.613+0000 7fe55fe00700 1 -- 172.21.15.62:0/865460659 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe5681012f0 msgr2=0x7fe568195e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:26:07.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:07.613+0000 7fe55fe00700 1 --2- 172.21.15.62:0/865460659 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe5681012f0 0x7fe568195e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:07.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:07.613+0000 7fe55fe00700 1 -- 172.21.15.62:0/865460659 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5500095f0 con 0x7fe568101c00 2024-04-20T21:26:07.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:07.613+0000 7fe566800700 1 --2- 172.21.15.62:0/865460659 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe5681012f0 0x7fe568195e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-20T21:26:07.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:07.614+0000 7fe55fe00700 1 --2- 172.21.15.62:0/865460659 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe568101c00 0x7fe5681963b0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fe55000b5c0 tx=0x7fe55000be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:26:07.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:07.614+0000 7fe55f400700 1 -- 172.21.15.62:0/865460659 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe55001d070 con 0x7fe568101c00 2024-04-20T21:26:07.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:07.614+0000 7fe55f400700 1 -- 172.21.15.62:0/865460659 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fe550022470 con 0x7fe568101c00 2024-04-20T21:26:07.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:07.614+0000 7fe56d400700 1 -- 172.21.15.62:0/865460659 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe56819aaa0 con 0x7fe568101c00 2024-04-20T21:26:07.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:07.614+0000 7fe55f400700 1 -- 172.21.15.62:0/865460659 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe55000f670 con 0x7fe568101c00 2024-04-20T21:26:07.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:07.614+0000 7fe56d400700 1 -- 172.21.15.62:0/865460659 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe56819af90 con 0x7fe568101c00 2024-04-20T21:26:07.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:07.615+0000 7fe56d400700 1 -- 172.21.15.62:0/865460659 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe568107970 con 0x7fe568101c00 2024-04-20T21:26:07.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:07.617+0000 7fe55f400700 1 -- 172.21.15.62:0/865460659 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7fe550004c30 con 0x7fe568101c00 2024-04-20T21:26:07.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:07.617+0000 7fe55f400700 1 --2- 172.21.15.62:0/865460659 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fe54c06c680 0x7fe54c06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:26:07.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:07.618+0000 7fe55f400700 1 -- 172.21.15.62:0/865460659 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fe55008cb10 con 0x7fe568101c00 2024-04-20T21:26:07.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:07.618+0000 7fe566800700 1 --2- 172.21.15.62:0/865460659 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fe54c06c680 0x7fe54c06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:26:07.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:07.619+0000 7fe566800700 1 --2- 172.21.15.62:0/865460659 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fe54c06c680 0x7fe54c06eb30 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe558005950 tx=0x7fe558005f50 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:26:07.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:07.623+0000 7fe55f400700 1 -- 172.21.15.62:0/865460659 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fe5500575c0 con 0x7fe568101c00 2024-04-20T21:26:07.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:07.845+0000 7fe56d400700 1 -- 172.21.15.62:0/865460659 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe56804f2a0 con 0x7fe568101c00 2024-04-20T21:26:07.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:07.848+0000 7fe55f400700 1 -- 172.21.15.62:0/865460659 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+11296 (secure 0 0 0) 0x7fe55005abe0 con 0x7fe568101c00 2024-04-20T21:26:07.847 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:26:07.847 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":40,"fsid":"61ae3628-ff5b-11ee-bc93-c7b262605968","created":"2024-04-20T21:18:19.951305+0000","modified":"2024-04-20T21:25:44.906419+0000","last_up_change":"2024-04-20T21:25:43.888770+0000","last_in_change":"2024-04-20T21:25:19.201764+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-20T21:24:02.500375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a3f8216-c641-4338-813d-d836efc3f4e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":2687023571},{"type":"v1","addr":"172.21.15.62:6803","nonce":2687023571}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":2687023571},{"type":"v1","addr":"172.21.15.62:6805","nonce":2687023571}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":2687023571},{"type":"v1","addr":"172.21.15.62:6809","nonce":2687023571}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":2687023571},{"type":"v1","addr":"172.21.15.62:6807","nonce":2687023571}]},"public_addr":"172.21.15.62:6803/2687023571","cluster_addr":"172.21.15.62:6805/2687023571","heartbeat_back_addr":"172.21.15.62:6809/2687023571","heartbeat_front_addr":"172.21.15.62:6807/2687023571","state":["exists","up"]},{"osd":1,"uuid":"f0c451a4-9002-4ddc-be0e-d8ddebd4418d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":1027109374},{"type":"v1","addr":"172.21.15.62:6811","nonce":1027109374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":1027109374},{"type":"v1","addr":"172.21.15.62:6813","nonce":1027109374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":1027109374},{"type":"v1","addr":"172.21.15.62:6817","nonce":1027109374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":1027109374},{"type":"v1","addr":"172.21.15.62:6815","nonce":1027109374}]},"public_addr":"172.21.15.62:6811/1027109374","cluster_addr":"172.21.15.62:6813/1027109374","heartbeat_back_addr":"172.21.15.62:6817/1027109374","heartbeat_front_addr":"172.21.15.62:6815/1027109374","state":["exists","up"]},{"osd":2,"uuid":"e1a8769d-5da2-49c8-942a-83a94c7ac761","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":47205739},{"type":"v1","addr":"172.21.15.62:6819","nonce":47205739}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":47205739},{"type":"v1","addr":"172.21.15.62:6821","nonce":47205739}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":47205739},{"type":"v1","addr":"172.21.15.62:6825","nonce":47205739}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6822","nonce":47205739},{"type":"v1","addr":"172.21.15.62:6823","nonce":47205739}]},"public_addr":"172.21.15.62:6819/47205739","cluster_addr":"172.21.15.62:6821/47205739","heartbeat_back_addr":"172.21.15.62:6825/47205739","heartbeat_front_addr":"172.21.15.62:6823/47205739","state":["exists","up"]},{"osd":3,"uuid":"eaa2fe2b-87f8-4d01-8ddf-e72f244f31b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":3169871320},{"type":"v1","addr":"172.21.15.189:6801","nonce":3169871320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":3169871320},{"type":"v1","addr":"172.21.15.189:6803","nonce":3169871320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":3169871320},{"type":"v1","addr":"172.21.15.189:6807","nonce":3169871320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":3169871320},{"type":"v1","addr":"172.21.15.189:6805","nonce":3169871320}]},"public_addr":"172.21.15.189:6801/3169871320","cluster_addr":"172.21.15.189:6803/3169871320","heartbeat_back_addr":"172.21.15.189:6807/3169871320","heartbeat_front_addr":"172.21.15.189:6805/3169871320","state":["exists","up"]},{"osd":4,"uuid":"72be8597-52e5-45fd-b4f2-06ac50df77e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":191133671},{"type":"v1","addr":"172.21.15.189:6809","nonce":191133671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":191133671},{"type":"v1","addr":"172.21.15.189:6811","nonce":191133671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":191133671},{"type":"v1","addr":"172.21.15.189:6815","nonce":191133671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":191133671},{"type":"v1","addr":"172.21.15.189:6813","nonce":191133671}]},"public_addr":"172.21.15.189:6809/191133671","cluster_addr":"172.21.15.189:6811/191133671","heartbeat_back_addr":"172.21.15.189:6815/191133671","heartbeat_front_addr":"172.21.15.189:6813/191133671","state":["exists","up"]},{"osd":5,"uuid":"c650c63b-41eb-4d4f-8e14-907c7b3614c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":893231043},{"type":"v1","addr":"172.21.15.189:6817","nonce":893231043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":893231043},{"type":"v1","addr":"172.21.15.189:6819","nonce":893231043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":893231043},{"type":"v1","addr":"172.21.15.189:6823","nonce":893231043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":893231043},{"type":"v1","addr":"172.21.15.189:6821","nonce":893231043}]},"public_addr":"172.21.15.189:6817/893231043","cluster_addr":"172.21.15.189:6819/893231043","heartbeat_back_addr":"172.21.15.189:6823/893231043","heartbeat_front_addr":"172.21.15.189:6821/893231043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T21:22:45.605185+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T21:23:21.062873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T21:24:00.502044+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T21:24:32.944093+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T21:25:06.278988+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T21:25:41.872982+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.62:0/1116048562":"2024-04-21T21:21:03.778739+0000","172.21.15.62:6800/831506538":"2024-04-21T21:21:03.778739+0000","172.21.15.62:0/4158176989":"2024-04-21T21:19:34.965981+0000","172.21.15.62:6800/4147738586":"2024-04-21T21:19:34.965981+0000","172.21.15.62:0/2525292829":"2024-04-21T21:18:52.990685+0000","172.21.15.62:0/301716985":"2024-04-21T21:21:03.778739+0000","172.21.15.62:0/634566145":"2024-04-21T21:18:52.990685+0000","172.21.15.62:0/3192460755":"2024-04-21T21:21:03.778739+0000","172.21.15.62:6801/831506538":"2024-04-21T21:21:03.778739+0000","172.21.15.62:6800/871423274":"2024-04-21T21:18:52.990685+0000","172.21.15.62:6801/4147738586":"2024-04-21T21:19:34.965981+0000","172.21.15.62:0/771955286":"2024-04-21T21:19:34.965981+0000","172.21.15.62:6801/871423274":"2024-04-21T21:18:52.990685+0000","172.21.15.62:0/3416563026":"2024-04-21T21:18:52.990685+0000","172.21.15.62:0/4274508569":"2024-04-21T21:19:34.965981+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-20T21:26:07.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:07.851+0000 7fe56d400700 1 -- 172.21.15.62:0/865460659 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fe54c06c680 msgr2=0x7fe54c06eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:26:07.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:07.851+0000 7fe56d400700 1 --2- 172.21.15.62:0/865460659 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fe54c06c680 0x7fe54c06eb30 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe558005950 tx=0x7fe558005f50 comp rx=0 tx=0).stop 2024-04-20T21:26:07.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:07.851+0000 7fe56d400700 1 -- 172.21.15.62:0/865460659 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe568101c00 msgr2=0x7fe5681963b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:26:07.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:07.851+0000 7fe56d400700 1 --2- 172.21.15.62:0/865460659 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe568101c00 0x7fe5681963b0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fe55000b5c0 tx=0x7fe55000be00 comp rx=0 tx=0).stop 2024-04-20T21:26:07.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:07.851+0000 7fe56d400700 1 -- 172.21.15.62:0/865460659 shutdown_connections 2024-04-20T21:26:07.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:07.851+0000 7fe56d400700 1 --2- 172.21.15.62:0/865460659 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fe54c06c680 0x7fe54c06eb30 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:07.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:07.851+0000 7fe56d400700 1 --2- 172.21.15.62:0/865460659 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe5681012f0 0x7fe568195e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:07.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:07.851+0000 7fe56d400700 1 --2- 172.21.15.62:0/865460659 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe568101c00 0x7fe5681963b0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:07.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:07.851+0000 7fe56d400700 1 -- 172.21.15.62:0/865460659 >> 172.21.15.62:0/865460659 conn(0x7fe5680fac80 msgr2=0x7fe568104ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:26:07.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:07.851+0000 7fe56d400700 1 -- 172.21.15.62:0/865460659 shutdown_connections 2024-04-20T21:26:07.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:07.851+0000 7fe56d400700 1 -- 172.21.15.62:0/865460659 wait complete. 2024-04-20T21:26:08.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:26:07 smithi189 ceph-mon[21562]: from='client.? 172.21.15.62:0/865460659' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-20T21:26:08.251 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:26:07 smithi062 ceph-mon[16221]: from='client.? 172.21.15.62:0/865460659' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-20T21:26:08.639 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-20T21:26:08.639 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-20T21:26:09.057 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T21:26:09.211 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:26:08 smithi062 ceph-mon[16221]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:26:09.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:26:08 smithi189 ceph-mon[21562]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:26:10.949 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:26:10 smithi062 ceph-mon[16221]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:26:11.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:11.235+0000 7f40f5800700 1 -- 172.21.15.62:0/3332614177 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40f01004e0 msgr2=0x7f40f01008b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:26:11.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:11.235+0000 7f40f5800700 1 --2- 172.21.15.62:0/3332614177 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40f01004e0 0x7f40f01008b0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f40e0009910 tx=0x7f40e0009c20 comp rx=0 tx=0).stop 2024-04-20T21:26:11.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:11.235+0000 7f40f5800700 1 -- 172.21.15.62:0/3332614177 shutdown_connections 2024-04-20T21:26:11.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:11.235+0000 7f40f5800700 1 --2- 172.21.15.62:0/3332614177 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f40f0100e80 0x7f40f0104e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:11.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:11.235+0000 7f40f5800700 1 --2- 172.21.15.62:0/3332614177 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40f01004e0 0x7f40f01008b0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:11.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:11.235+0000 7f40f5800700 1 -- 172.21.15.62:0/3332614177 >> 172.21.15.62:0/3332614177 conn(0x7f40f00fbda0 msgr2=0x7f40f00fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:26:11.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:11.236+0000 7f40f5800700 1 -- 172.21.15.62:0/3332614177 shutdown_connections 2024-04-20T21:26:11.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:11.236+0000 7f40f5800700 1 -- 172.21.15.62:0/3332614177 wait complete. 2024-04-20T21:26:11.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:11.237+0000 7f40f5800700 1 Processor -- start 2024-04-20T21:26:11.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:11.237+0000 7f40f5800700 1 -- start start 2024-04-20T21:26:11.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:26:10 smithi189 ceph-mon[21562]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:26:11.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:11.238+0000 7f40f5800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f40f01004e0 0x7f40f0195ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:26:11.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:11.238+0000 7f40f5800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40f0100e80 0x7f40f0196430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:26:11.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:11.238+0000 7f40f5800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40f0196b10 con 0x7f40f0100e80 2024-04-20T21:26:11.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:11.238+0000 7f40f5800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40f019a8a0 con 0x7f40f01004e0 2024-04-20T21:26:11.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:11.238+0000 7f40ee000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40f0100e80 0x7f40f0196430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:26:11.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:11.238+0000 7f40ee000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40f0100e80 0x7f40f0196430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50630/0 (socket says 172.21.15.62:50630) 2024-04-20T21:26:11.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:11.238+0000 7f40ee000700 1 -- 172.21.15.62:0/1973490049 learned_addr learned my addr 172.21.15.62:0/1973490049 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:26:11.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:11.238+0000 7f40eea00700 1 --2- 172.21.15.62:0/1973490049 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f40f01004e0 0x7f40f0195ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:26:11.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:11.239+0000 7f40ee000700 1 -- 172.21.15.62:0/1973490049 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f40f01004e0 msgr2=0x7f40f0195ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:26:11.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:11.239+0000 7f40ee000700 1 --2- 172.21.15.62:0/1973490049 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f40f01004e0 0x7f40f0195ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:11.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:11.239+0000 7f40ee000700 1 -- 172.21.15.62:0/1973490049 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40e00095f0 con 0x7f40f0100e80 2024-04-20T21:26:11.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:11.239+0000 7f40ee000700 1 --2- 172.21.15.62:0/1973490049 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40f0100e80 0x7f40f0196430 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f40dc00d900 tx=0x7f40dc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:26:11.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:11.239+0000 7f40e7400700 1 -- 172.21.15.62:0/1973490049 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f40dc009900 con 0x7f40f0100e80 2024-04-20T21:26:11.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:11.239+0000 7f40e7400700 1 -- 172.21.15.62:0/1973490049 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f40dc010460 con 0x7f40f0100e80 2024-04-20T21:26:11.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:11.239+0000 7f40e7400700 1 -- 172.21.15.62:0/1973490049 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f40dc00f5f0 con 0x7f40f0100e80 2024-04-20T21:26:11.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:11.239+0000 7f40f5800700 1 -- 172.21.15.62:0/1973490049 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40f019ab80 con 0x7f40f0100e80 2024-04-20T21:26:11.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:11.240+0000 7f40f5800700 1 -- 172.21.15.62:0/1973490049 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40f019b0d0 con 0x7f40f0100e80 2024-04-20T21:26:11.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:11.241+0000 7f40f5800700 1 -- 172.21.15.62:0/1973490049 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40f01087f0 con 0x7f40f0100e80 2024-04-20T21:26:11.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:11.242+0000 7f40e7400700 1 -- 172.21.15.62:0/1973490049 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f40dc0105d0 con 0x7f40f0100e80 2024-04-20T21:26:11.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:11.243+0000 7f40e7400700 1 --2- 172.21.15.62:0/1973490049 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f40d406c680 0x7f40d406eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:26:11.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:11.243+0000 7f40e7400700 1 -- 172.21.15.62:0/1973490049 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f40dc08b370 con 0x7f40f0100e80 2024-04-20T21:26:11.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:11.243+0000 7f40eea00700 1 --2- 172.21.15.62:0/1973490049 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f40d406c680 0x7f40d406eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:26:11.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:11.244+0000 7f40eea00700 1 --2- 172.21.15.62:0/1973490049 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f40d406c680 0x7f40d406eb30 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f40e0006010 tx=0x7f40e001a040 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:26:11.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:11.247+0000 7f40e7400700 1 -- 172.21.15.62:0/1973490049 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f40dc055d70 con 0x7f40f0100e80 2024-04-20T21:26:11.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:11.468+0000 7f40f5800700 1 -- 172.21.15.62:0/1973490049 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f40f019b340 con 0x7f40f0100e80 2024-04-20T21:26:11.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:11.471+0000 7f40e7400700 1 -- 172.21.15.62:0/1973490049 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+11296 (secure 0 0 0) 0x7f40dc059390 con 0x7f40f0100e80 2024-04-20T21:26:11.470 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:26:11.470 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":40,"fsid":"61ae3628-ff5b-11ee-bc93-c7b262605968","created":"2024-04-20T21:18:19.951305+0000","modified":"2024-04-20T21:25:44.906419+0000","last_up_change":"2024-04-20T21:25:43.888770+0000","last_in_change":"2024-04-20T21:25:19.201764+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-20T21:24:02.500375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a3f8216-c641-4338-813d-d836efc3f4e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":2687023571},{"type":"v1","addr":"172.21.15.62:6803","nonce":2687023571}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":2687023571},{"type":"v1","addr":"172.21.15.62:6805","nonce":2687023571}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":2687023571},{"type":"v1","addr":"172.21.15.62:6809","nonce":2687023571}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":2687023571},{"type":"v1","addr":"172.21.15.62:6807","nonce":2687023571}]},"public_addr":"172.21.15.62:6803/2687023571","cluster_addr":"172.21.15.62:6805/2687023571","heartbeat_back_addr":"172.21.15.62:6809/2687023571","heartbeat_front_addr":"172.21.15.62:6807/2687023571","state":["exists","up"]},{"osd":1,"uuid":"f0c451a4-9002-4ddc-be0e-d8ddebd4418d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":1027109374},{"type":"v1","addr":"172.21.15.62:6811","nonce":1027109374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":1027109374},{"type":"v1","addr":"172.21.15.62:6813","nonce":1027109374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":1027109374},{"type":"v1","addr":"172.21.15.62:6817","nonce":1027109374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":1027109374},{"type":"v1","addr":"172.21.15.62:6815","nonce":1027109374}]},"public_addr":"172.21.15.62:6811/1027109374","cluster_addr":"172.21.15.62:6813/1027109374","heartbeat_back_addr":"172.21.15.62:6817/1027109374","heartbeat_front_addr":"172.21.15.62:6815/1027109374","state":["exists","up"]},{"osd":2,"uuid":"e1a8769d-5da2-49c8-942a-83a94c7ac761","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":47205739},{"type":"v1","addr":"172.21.15.62:6819","nonce":47205739}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":47205739},{"type":"v1","addr":"172.21.15.62:6821","nonce":47205739}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":47205739},{"type":"v1","addr":"172.21.15.62:6825","nonce":47205739}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6822","nonce":47205739},{"type":"v1","addr":"172.21.15.62:6823","nonce":47205739}]},"public_addr":"172.21.15.62:6819/47205739","cluster_addr":"172.21.15.62:6821/47205739","heartbeat_back_addr":"172.21.15.62:6825/47205739","heartbeat_front_addr":"172.21.15.62:6823/47205739","state":["exists","up"]},{"osd":3,"uuid":"eaa2fe2b-87f8-4d01-8ddf-e72f244f31b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":3169871320},{"type":"v1","addr":"172.21.15.189:6801","nonce":3169871320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":3169871320},{"type":"v1","addr":"172.21.15.189:6803","nonce":3169871320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":3169871320},{"type":"v1","addr":"172.21.15.189:6807","nonce":3169871320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":3169871320},{"type":"v1","addr":"172.21.15.189:6805","nonce":3169871320}]},"public_addr":"172.21.15.189:6801/3169871320","cluster_addr":"172.21.15.189:6803/3169871320","heartbeat_back_addr":"172.21.15.189:6807/3169871320","heartbeat_front_addr":"172.21.15.189:6805/3169871320","state":["exists","up"]},{"osd":4,"uuid":"72be8597-52e5-45fd-b4f2-06ac50df77e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":191133671},{"type":"v1","addr":"172.21.15.189:6809","nonce":191133671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":191133671},{"type":"v1","addr":"172.21.15.189:6811","nonce":191133671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":191133671},{"type":"v1","addr":"172.21.15.189:6815","nonce":191133671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":191133671},{"type":"v1","addr":"172.21.15.189:6813","nonce":191133671}]},"public_addr":"172.21.15.189:6809/191133671","cluster_addr":"172.21.15.189:6811/191133671","heartbeat_back_addr":"172.21.15.189:6815/191133671","heartbeat_front_addr":"172.21.15.189:6813/191133671","state":["exists","up"]},{"osd":5,"uuid":"c650c63b-41eb-4d4f-8e14-907c7b3614c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":893231043},{"type":"v1","addr":"172.21.15.189:6817","nonce":893231043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":893231043},{"type":"v1","addr":"172.21.15.189:6819","nonce":893231043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":893231043},{"type":"v1","addr":"172.21.15.189:6823","nonce":893231043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":893231043},{"type":"v1","addr":"172.21.15.189:6821","nonce":893231043}]},"public_addr":"172.21.15.189:6817/893231043","cluster_addr":"172.21.15.189:6819/893231043","heartbeat_back_addr":"172.21.15.189:6823/893231043","heartbeat_front_addr":"172.21.15.189:6821/893231043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T21:22:45.605185+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T21:23:21.062873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T21:24:00.502044+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T21:24:32.944093+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T21:25:06.278988+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T21:25:41.872982+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.62:0/1116048562":"2024-04-21T21:21:03.778739+0000","172.21.15.62:6800/831506538":"2024-04-21T21:21:03.778739+0000","172.21.15.62:0/4158176989":"2024-04-21T21:19:34.965981+0000","172.21.15.62:6800/4147738586":"2024-04-21T21:19:34.965981+0000","172.21.15.62:0/2525292829":"2024-04-21T21:18:52.990685+0000","172.21.15.62:0/301716985":"2024-04-21T21:21:03.778739+0000","172.21.15.62:0/634566145":"2024-04-21T21:18:52.990685+0000","172.21.15.62:0/3192460755":"2024-04-21T21:21:03.778739+0000","172.21.15.62:6801/831506538":"2024-04-21T21:21:03.778739+0000","172.21.15.62:6800/871423274":"2024-04-21T21:18:52.990685+0000","172.21.15.62:6801/4147738586":"2024-04-21T21:19:34.965981+0000","172.21.15.62:0/771955286":"2024-04-21T21:19:34.965981+0000","172.21.15.62:6801/871423274":"2024-04-21T21:18:52.990685+0000","172.21.15.62:0/3416563026":"2024-04-21T21:18:52.990685+0000","172.21.15.62:0/4274508569":"2024-04-21T21:19:34.965981+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-20T21:26:11.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:11.473+0000 7f40f5800700 1 -- 172.21.15.62:0/1973490049 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f40d406c680 msgr2=0x7f40d406eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:26:11.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:11.473+0000 7f40f5800700 1 --2- 172.21.15.62:0/1973490049 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f40d406c680 0x7f40d406eb30 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f40e0006010 tx=0x7f40e001a040 comp rx=0 tx=0).stop 2024-04-20T21:26:11.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:11.474+0000 7f40f5800700 1 -- 172.21.15.62:0/1973490049 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40f0100e80 msgr2=0x7f40f0196430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:26:11.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:11.474+0000 7f40f5800700 1 --2- 172.21.15.62:0/1973490049 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40f0100e80 0x7f40f0196430 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f40dc00d900 tx=0x7f40dc00dcc0 comp rx=0 tx=0).stop 2024-04-20T21:26:11.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:11.474+0000 7f40f5800700 1 -- 172.21.15.62:0/1973490049 shutdown_connections 2024-04-20T21:26:11.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:11.474+0000 7f40f5800700 1 --2- 172.21.15.62:0/1973490049 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f40d406c680 0x7f40d406eb30 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:11.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:11.474+0000 7f40f5800700 1 --2- 172.21.15.62:0/1973490049 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f40f01004e0 0x7f40f0195ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:11.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:11.474+0000 7f40f5800700 1 --2- 172.21.15.62:0/1973490049 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40f0100e80 0x7f40f0196430 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:11.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:11.474+0000 7f40f5800700 1 -- 172.21.15.62:0/1973490049 >> 172.21.15.62:0/1973490049 conn(0x7f40f00fbda0 msgr2=0x7f40f00fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:26:11.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:11.474+0000 7f40f5800700 1 -- 172.21.15.62:0/1973490049 shutdown_connections 2024-04-20T21:26:11.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:11.474+0000 7f40f5800700 1 -- 172.21.15.62:0/1973490049 wait complete. 2024-04-20T21:26:12.179 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:26:11 smithi062 ceph-mon[16221]: from='client.? 172.21.15.62:0/1973490049' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-20T21:26:12.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:26:11 smithi189 ceph-mon[21562]: from='client.? 172.21.15.62:0/1973490049' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-20T21:26:12.254 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-04-20T21:26:12.254 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-04-20T21:26:12.254 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-04-20T21:26:12.255 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-04-20T21:26:12.255 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-04-20T21:26:12.256 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-04-20T21:26:12.767 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T21:26:12.771 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T21:26:12.816 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T21:26:13.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:26:12 smithi062 ceph-mon[16221]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:26:13.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:26:12 smithi189 ceph-mon[21562]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:26:13.970 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:26:13 smithi062 ceph-mon[16221]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:26:14.055 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T21:26:14.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:26:13 smithi189 ceph-mon[21562]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:26:14.337 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T21:26:14.478 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T21:26:16.214 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:26:15 smithi062 ceph-mon[16221]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:26:16.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:26:15 smithi189 ceph-mon[21562]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:26:18.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:26:17 smithi189 ceph-mon[21562]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:26:18.280 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:26:17 smithi062 ceph-mon[16221]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:26:19.232 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:26:18 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:26:19.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:26:18 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:26:19.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:19.406+0000 7f8021800700 1 -- 172.21.15.62:0/1706749514 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f801c100e80 msgr2=0x7f801c104e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:26:19.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:19.406+0000 7f8021800700 1 --2- 172.21.15.62:0/1706749514 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f801c100e80 0x7f801c104e50 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f8008009910 tx=0x7f8008009c20 comp rx=0 tx=0).stop 2024-04-20T21:26:19.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:19.406+0000 7f8021800700 1 -- 172.21.15.62:0/1706749514 shutdown_connections 2024-04-20T21:26:19.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:19.406+0000 7f8021800700 1 --2- 172.21.15.62:0/1706749514 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f801c100e80 0x7f801c104e50 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:19.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:19.406+0000 7f8021800700 1 --2- 172.21.15.62:0/1706749514 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f801c1004e0 0x7f801c1008b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:19.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:19.406+0000 7f8021800700 1 -- 172.21.15.62:0/1706749514 >> 172.21.15.62:0/1706749514 conn(0x7f801c0fbda0 msgr2=0x7f801c0fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:26:19.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:19.406+0000 7f8021800700 1 -- 172.21.15.62:0/1706749514 shutdown_connections 2024-04-20T21:26:19.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:19.406+0000 7f8021800700 1 -- 172.21.15.62:0/1706749514 wait complete. 2024-04-20T21:26:19.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:19.407+0000 7f8021800700 1 Processor -- start 2024-04-20T21:26:19.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:19.408+0000 7f8021800700 1 -- start start 2024-04-20T21:26:19.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:19.408+0000 7f8021800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f801c1004e0 0x7f801c1980e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:26:19.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:19.408+0000 7f8021800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f801c100e80 0x7f801c198620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:26:19.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:19.408+0000 7f8021800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f801c198d00 con 0x7f801c1004e0 2024-04-20T21:26:19.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:19.408+0000 7f8021800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f801c19ca90 con 0x7f801c100e80 2024-04-20T21:26:19.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:19.408+0000 7f801aa00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f801c1004e0 0x7f801c1980e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:26:19.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:19.408+0000 7f801aa00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f801c1004e0 0x7f801c1980e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43218/0 (socket says 172.21.15.62:43218) 2024-04-20T21:26:19.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:19.408+0000 7f801aa00700 1 -- 172.21.15.62:0/1334181808 learned_addr learned my addr 172.21.15.62:0/1334181808 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:26:19.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:19.409+0000 7f801a000700 1 --2- 172.21.15.62:0/1334181808 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f801c100e80 0x7f801c198620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:26:19.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:19.409+0000 7f801aa00700 1 -- 172.21.15.62:0/1334181808 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f801c100e80 msgr2=0x7f801c198620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:26:19.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:19.409+0000 7f801aa00700 1 --2- 172.21.15.62:0/1334181808 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f801c100e80 0x7f801c198620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:19.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:19.409+0000 7f801aa00700 1 -- 172.21.15.62:0/1334181808 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80080095f0 con 0x7f801c1004e0 2024-04-20T21:26:19.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:19.409+0000 7f801aa00700 1 --2- 172.21.15.62:0/1334181808 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f801c1004e0 0x7f801c1980e0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f800c00d8d0 tx=0x7f800c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:26:19.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:19.409+0000 7f8013400700 1 -- 172.21.15.62:0/1334181808 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f800c009940 con 0x7f801c1004e0 2024-04-20T21:26:19.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:19.409+0000 7f8013400700 1 -- 172.21.15.62:0/1334181808 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f800c010460 con 0x7f801c1004e0 2024-04-20T21:26:19.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:19.409+0000 7f8013400700 1 -- 172.21.15.62:0/1334181808 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f800c00f5f0 con 0x7f801c1004e0 2024-04-20T21:26:19.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:19.409+0000 7f8021800700 1 -- 172.21.15.62:0/1334181808 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f801c19cd70 con 0x7f801c1004e0 2024-04-20T21:26:19.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:19.410+0000 7f8021800700 1 -- 172.21.15.62:0/1334181808 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f801c19d2c0 con 0x7f801c1004e0 2024-04-20T21:26:19.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:19.410+0000 7f8021800700 1 -- 172.21.15.62:0/1334181808 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f801c10d650 con 0x7f801c1004e0 2024-04-20T21:26:19.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:19.412+0000 7f8013400700 1 -- 172.21.15.62:0/1334181808 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f800c0105d0 con 0x7f801c1004e0 2024-04-20T21:26:19.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:19.413+0000 7f8013400700 1 --2- 172.21.15.62:0/1334181808 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f800006c680 0x7f800006eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:26:19.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:19.413+0000 7f8013400700 1 -- 172.21.15.62:0/1334181808 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f800c08b510 con 0x7f801c1004e0 2024-04-20T21:26:19.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:19.413+0000 7f801a000700 1 --2- 172.21.15.62:0/1334181808 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f800006c680 0x7f800006eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:26:19.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:19.413+0000 7f8013400700 1 --2- 172.21.15.62:0/1334181808 >> [v2:172.21.15.189:6800/3169871320,v1:172.21.15.189:6801/3169871320] conn(0x7f8000072350 0x7f8000074760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:26:19.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:19.413+0000 7f8013400700 1 -- 172.21.15.62:0/1334181808 --> [v2:172.21.15.189:6800/3169871320,v1:172.21.15.189:6801/3169871320] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8000074e10 con 0x7f8000072350 2024-04-20T21:26:19.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:19.413+0000 7f8013400700 1 -- 172.21.15.62:0/1334181808 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f800c08db40 con 0x7f801c1004e0 2024-04-20T21:26:19.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:19.414+0000 7f801b400700 1 --2- 172.21.15.62:0/1334181808 >> [v2:172.21.15.189:6800/3169871320,v1:172.21.15.189:6801/3169871320] conn(0x7f8000072350 0x7f8000074760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:26:19.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:19.414+0000 7f801a000700 1 --2- 172.21.15.62:0/1334181808 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f800006c680 0x7f800006eb30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f801c199700 tx=0x7f800801a040 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:26:19.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:19.415+0000 7f801b400700 1 --2- 172.21.15.62:0/1334181808 >> [v2:172.21.15.189:6800/3169871320,v1:172.21.15.189:6801/3169871320] conn(0x7f8000072350 0x7f8000074760 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:26:19.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:19.417+0000 7f8013400700 1 -- 172.21.15.62:0/1334181808 <== osd.3 v2:172.21.15.189:6800/3169871320 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f8000074e10 con 0x7f8000072350 2024-04-20T21:26:19.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:19.439+0000 7f8021800700 1 -- 172.21.15.62:0/1334181808 --> [v2:172.21.15.189:6800/3169871320,v1:172.21.15.189:6801/3169871320] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f801c04f2a0 con 0x7f8000072350 2024-04-20T21:26:19.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:19.440+0000 7f8013400700 1 -- 172.21.15.62:0/1334181808 <== osd.3 v2:172.21.15.189:6800/3169871320 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f801c04f2a0 con 0x7f8000072350 2024-04-20T21:26:19.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:19.440+0000 7f8021800700 1 -- 172.21.15.62:0/1334181808 >> [v2:172.21.15.189:6800/3169871320,v1:172.21.15.189:6801/3169871320] conn(0x7f8000072350 msgr2=0x7f8000074760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:26:19.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:19.440+0000 7f8021800700 1 --2- 172.21.15.62:0/1334181808 >> [v2:172.21.15.189:6800/3169871320,v1:172.21.15.189:6801/3169871320] conn(0x7f8000072350 0x7f8000074760 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:19.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:19.440+0000 7f8021800700 1 -- 172.21.15.62:0/1334181808 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f800006c680 msgr2=0x7f800006eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:26:19.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:19.441+0000 7f8021800700 1 --2- 172.21.15.62:0/1334181808 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f800006c680 0x7f800006eb30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f801c199700 tx=0x7f800801a040 comp rx=0 tx=0).stop 2024-04-20T21:26:19.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:19.441+0000 7f8021800700 1 -- 172.21.15.62:0/1334181808 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f801c1004e0 msgr2=0x7f801c1980e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:26:19.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:19.441+0000 7f8021800700 1 --2- 172.21.15.62:0/1334181808 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f801c1004e0 0x7f801c1980e0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f800c00d8d0 tx=0x7f800c00dc90 comp rx=0 tx=0).stop 2024-04-20T21:26:19.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:19.441+0000 7f8021800700 1 -- 172.21.15.62:0/1334181808 shutdown_connections 2024-04-20T21:26:19.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:19.441+0000 7f8021800700 1 --2- 172.21.15.62:0/1334181808 >> [v2:172.21.15.189:6800/3169871320,v1:172.21.15.189:6801/3169871320] conn(0x7f8000072350 0x7f8000074760 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:19.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:19.441+0000 7f8021800700 1 --2- 172.21.15.62:0/1334181808 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f800006c680 0x7f800006eb30 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:19.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:19.441+0000 7f8021800700 1 --2- 172.21.15.62:0/1334181808 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f801c1004e0 0x7f801c1980e0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:19.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:19.441+0000 7f8021800700 1 --2- 172.21.15.62:0/1334181808 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f801c100e80 0x7f801c198620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:19.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:19.441+0000 7f8021800700 1 -- 172.21.15.62:0/1334181808 >> 172.21.15.62:0/1334181808 conn(0x7f801c0fbda0 msgr2=0x7f801c0fd3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:26:19.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:19.441+0000 7f8021800700 1 -- 172.21.15.62:0/1334181808 shutdown_connections 2024-04-20T21:26:19.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:19.441+0000 7f8021800700 1 -- 172.21.15.62:0/1334181808 wait complete. 2024-04-20T21:26:20.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:20.227+0000 7f68a7000700 1 -- 172.21.15.62:0/1667069405 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f68a0100e60 msgr2=0x7f68a0104e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:26:20.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:20.227+0000 7f68a7000700 1 --2- 172.21.15.62:0/1667069405 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f68a0100e60 0x7f68a0104e30 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f688c009910 tx=0x7f688c009c20 comp rx=0 tx=0).stop 2024-04-20T21:26:20.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:20.227+0000 7f68a7000700 1 -- 172.21.15.62:0/1667069405 shutdown_connections 2024-04-20T21:26:20.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:20.227+0000 7f68a7000700 1 --2- 172.21.15.62:0/1667069405 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f68a0100e60 0x7f68a0104e30 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:20.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:20.227+0000 7f68a7000700 1 --2- 172.21.15.62:0/1667069405 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68a01004c0 0x7f68a0100890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:20.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:20.227+0000 7f68a7000700 1 -- 172.21.15.62:0/1667069405 >> 172.21.15.62:0/1667069405 conn(0x7f68a00fbd80 msgr2=0x7f68a00fe190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:26:20.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:20.227+0000 7f68a7000700 1 -- 172.21.15.62:0/1667069405 shutdown_connections 2024-04-20T21:26:20.228 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:26:19 smithi062 ceph-mon[16221]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:26:20.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:20.227+0000 7f68a7000700 1 -- 172.21.15.62:0/1667069405 wait complete. 2024-04-20T21:26:20.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:20.228+0000 7f68a7000700 1 Processor -- start 2024-04-20T21:26:20.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:20.228+0000 7f68a7000700 1 -- start start 2024-04-20T21:26:20.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:20.229+0000 7f68a7000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68a01004c0 0x7f68a01980f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:26:20.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:20.229+0000 7f68a7000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f68a0100e60 0x7f68a0198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:26:20.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:20.229+0000 7f68a7000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68a0198d10 con 0x7f68a0100e60 2024-04-20T21:26:20.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:20.229+0000 7f68a7000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68a019caa0 con 0x7f68a01004c0 2024-04-20T21:26:20.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:20.229+0000 7f689f400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f68a0100e60 0x7f68a0198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:26:20.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:20.229+0000 7f689f400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f68a0100e60 0x7f68a0198630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43240/0 (socket says 172.21.15.62:43240) 2024-04-20T21:26:20.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:20.229+0000 7f689f400700 1 -- 172.21.15.62:0/2857128455 learned_addr learned my addr 172.21.15.62:0/2857128455 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:26:20.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:20.229+0000 7f689fe00700 1 --2- 172.21.15.62:0/2857128455 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68a01004c0 0x7f68a01980f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:26:20.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:20.230+0000 7f689f400700 1 -- 172.21.15.62:0/2857128455 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68a01004c0 msgr2=0x7f68a01980f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:26:20.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:20.230+0000 7f689f400700 1 --2- 172.21.15.62:0/2857128455 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68a01004c0 0x7f68a01980f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:20.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:20.230+0000 7f689f400700 1 -- 172.21.15.62:0/2857128455 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f688c0095f0 con 0x7f68a0100e60 2024-04-20T21:26:20.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:20.230+0000 7f689fe00700 1 --2- 172.21.15.62:0/2857128455 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68a01004c0 0x7f68a01980f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-20T21:26:20.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:20.230+0000 7f689f400700 1 --2- 172.21.15.62:0/2857128455 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f68a0100e60 0x7f68a0198630 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f688c00bda0 tx=0x7f688c00be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:26:20.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:20.230+0000 7f689cc00700 1 -- 172.21.15.62:0/2857128455 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f688c01d070 con 0x7f68a0100e60 2024-04-20T21:26:20.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:20.230+0000 7f689cc00700 1 -- 172.21.15.62:0/2857128455 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f688c022470 con 0x7f68a0100e60 2024-04-20T21:26:20.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:20.230+0000 7f689cc00700 1 -- 172.21.15.62:0/2857128455 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f688c00f690 con 0x7f68a0100e60 2024-04-20T21:26:20.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:20.230+0000 7f68a7000700 1 -- 172.21.15.62:0/2857128455 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68a019cd20 con 0x7f68a0100e60 2024-04-20T21:26:20.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:20.230+0000 7f68a7000700 1 -- 172.21.15.62:0/2857128455 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68a019d1c0 con 0x7f68a0100e60 2024-04-20T21:26:20.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:20.231+0000 7f68a7000700 1 -- 172.21.15.62:0/2857128455 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f68a010d630 con 0x7f68a0100e60 2024-04-20T21:26:20.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:20.233+0000 7f689cc00700 1 -- 172.21.15.62:0/2857128455 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f688c022ae0 con 0x7f68a0100e60 2024-04-20T21:26:20.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:20.233+0000 7f689cc00700 1 --2- 172.21.15.62:0/2857128455 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f688406c680 0x7f688406eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:26:20.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:20.234+0000 7f689cc00700 1 -- 172.21.15.62:0/2857128455 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f688c08ca00 con 0x7f68a0100e60 2024-04-20T21:26:20.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:20.234+0000 7f689fe00700 1 --2- 172.21.15.62:0/2857128455 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f688406c680 0x7f688406eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:26:20.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:20.234+0000 7f689cc00700 1 --2- 172.21.15.62:0/2857128455 >> [v2:172.21.15.62:6818/47205739,v1:172.21.15.62:6819/47205739] conn(0x7f6884072350 0x7f6884074760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:26:20.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:20.234+0000 7f689cc00700 1 -- 172.21.15.62:0/2857128455 --> [v2:172.21.15.62:6818/47205739,v1:172.21.15.62:6819/47205739] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6884074e10 con 0x7f6884072350 2024-04-20T21:26:20.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:20.234+0000 7f689cc00700 1 -- 172.21.15.62:0/2857128455 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f688c0912a0 con 0x7f68a0100e60 2024-04-20T21:26:20.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:20.234+0000 7f68a5000700 1 --2- 172.21.15.62:0/2857128455 >> [v2:172.21.15.62:6818/47205739,v1:172.21.15.62:6819/47205739] conn(0x7f6884072350 0x7f6884074760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:26:20.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:20.235+0000 7f689fe00700 1 --2- 172.21.15.62:0/2857128455 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f688406c680 0x7f688406eb30 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f6890005950 tx=0x7f6890005f50 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:26:20.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:20.235+0000 7f68a5000700 1 --2- 172.21.15.62:0/2857128455 >> [v2:172.21.15.62:6818/47205739,v1:172.21.15.62:6819/47205739] conn(0x7f6884072350 0x7f6884074760 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:26:20.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:20.236+0000 7f689cc00700 1 -- 172.21.15.62:0/2857128455 <== osd.2 v2:172.21.15.62:6818/47205739 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f6884074e10 con 0x7f6884072350 2024-04-20T21:26:20.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:26:19 smithi189 ceph-mon[21562]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:26:20.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:20.255+0000 7f68a7000700 1 -- 172.21.15.62:0/2857128455 --> [v2:172.21.15.62:6818/47205739,v1:172.21.15.62:6819/47205739] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f68a004f2a0 con 0x7f6884072350 2024-04-20T21:26:20.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:20.256+0000 7f689cc00700 1 -- 172.21.15.62:0/2857128455 <== osd.2 v2:172.21.15.62:6818/47205739 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f68a004f2a0 con 0x7f6884072350 2024-04-20T21:26:20.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:20.256+0000 7f68a7000700 1 -- 172.21.15.62:0/2857128455 >> [v2:172.21.15.62:6818/47205739,v1:172.21.15.62:6819/47205739] conn(0x7f6884072350 msgr2=0x7f6884074760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:26:20.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:20.256+0000 7f68a7000700 1 --2- 172.21.15.62:0/2857128455 >> [v2:172.21.15.62:6818/47205739,v1:172.21.15.62:6819/47205739] conn(0x7f6884072350 0x7f6884074760 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:20.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:20.256+0000 7f68a7000700 1 -- 172.21.15.62:0/2857128455 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f688406c680 msgr2=0x7f688406eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:26:20.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:20.256+0000 7f68a7000700 1 --2- 172.21.15.62:0/2857128455 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f688406c680 0x7f688406eb30 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f6890005950 tx=0x7f6890005f50 comp rx=0 tx=0).stop 2024-04-20T21:26:20.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:20.256+0000 7f68a7000700 1 -- 172.21.15.62:0/2857128455 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f68a0100e60 msgr2=0x7f68a0198630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:26:20.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:20.256+0000 7f68a7000700 1 --2- 172.21.15.62:0/2857128455 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f68a0100e60 0x7f68a0198630 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f688c00bda0 tx=0x7f688c00be80 comp rx=0 tx=0).stop 2024-04-20T21:26:20.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:20.257+0000 7f68a7000700 1 -- 172.21.15.62:0/2857128455 shutdown_connections 2024-04-20T21:26:20.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:20.257+0000 7f68a7000700 1 --2- 172.21.15.62:0/2857128455 >> [v2:172.21.15.62:6818/47205739,v1:172.21.15.62:6819/47205739] conn(0x7f6884072350 0x7f6884074760 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:20.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:20.257+0000 7f68a7000700 1 --2- 172.21.15.62:0/2857128455 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f688406c680 0x7f688406eb30 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:20.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:20.257+0000 7f68a7000700 1 --2- 172.21.15.62:0/2857128455 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68a01004c0 0x7f68a01980f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:20.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:20.257+0000 7f68a7000700 1 --2- 172.21.15.62:0/2857128455 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f68a0100e60 0x7f68a0198630 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:20.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:20.257+0000 7f68a7000700 1 -- 172.21.15.62:0/2857128455 >> 172.21.15.62:0/2857128455 conn(0x7f68a00fbd80 msgr2=0x7f68a00fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:26:20.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:20.257+0000 7f68a7000700 1 -- 172.21.15.62:0/2857128455 shutdown_connections 2024-04-20T21:26:20.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:20.257+0000 7f68a7000700 1 -- 172.21.15.62:0/2857128455 wait complete. 2024-04-20T21:26:21.562 INFO:teuthology.orchestra.run.smithi062.stdout:81604378655 2024-04-20T21:26:21.563 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-04-20T21:26:22.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.675+0000 7f4882a00700 1 -- 172.21.15.62:0/42512892 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f487c0ffbd0 msgr2=0x7f487c100040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:26:22.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.675+0000 7f4882a00700 1 --2- 172.21.15.62:0/42512892 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f487c0ffbd0 0x7f487c100040 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f4868009910 tx=0x7f4868009c20 comp rx=0 tx=0).stop 2024-04-20T21:26:22.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.675+0000 7f4882a00700 1 -- 172.21.15.62:0/42512892 shutdown_connections 2024-04-20T21:26:22.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.675+0000 7f4882a00700 1 --2- 172.21.15.62:0/42512892 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f487c0ffbd0 0x7f487c100040 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:22.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.675+0000 7f4882a00700 1 --2- 172.21.15.62:0/42512892 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f487c104130 0x7f487c104500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:22.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.675+0000 7f4882a00700 1 -- 172.21.15.62:0/42512892 >> 172.21.15.62:0/42512892 conn(0x7f487c075130 msgr2=0x7f487c075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:26:22.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.675+0000 7f4882a00700 1 -- 172.21.15.62:0/42512892 shutdown_connections 2024-04-20T21:26:22.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.675+0000 7f4882a00700 1 -- 172.21.15.62:0/42512892 wait complete. 2024-04-20T21:26:22.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.676+0000 7f4882a00700 1 Processor -- start 2024-04-20T21:26:22.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.676+0000 7f4882a00700 1 -- start start 2024-04-20T21:26:22.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.676+0000 7f4882a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f487c0ffbd0 0x7f487c196d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:26:22.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.677+0000 7f4882a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f487c104130 0x7f487c197260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:26:22.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.677+0000 7f4882a00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f487c197880 con 0x7f487c0ffbd0 2024-04-20T21:26:22.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.677+0000 7f4882a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f487c1979c0 con 0x7f487c104130 2024-04-20T21:26:22.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.677+0000 7f487be00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f487c0ffbd0 0x7f487c196d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:26:22.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.677+0000 7f487be00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f487c0ffbd0 0x7f487c196d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51882/0 (socket says 172.21.15.62:51882) 2024-04-20T21:26:22.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.677+0000 7f487be00700 1 -- 172.21.15.62:0/3653937692 learned_addr learned my addr 172.21.15.62:0/3653937692 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:26:22.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.677+0000 7f487b400700 1 --2- 172.21.15.62:0/3653937692 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f487c104130 0x7f487c197260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:26:22.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.677+0000 7f487be00700 1 -- 172.21.15.62:0/3653937692 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f487c104130 msgr2=0x7f487c197260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:26:22.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.677+0000 7f487be00700 1 --2- 172.21.15.62:0/3653937692 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f487c104130 0x7f487c197260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:22.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.677+0000 7f487be00700 1 -- 172.21.15.62:0/3653937692 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48680095f0 con 0x7f487c0ffbd0 2024-04-20T21:26:22.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.678+0000 7f487be00700 1 --2- 172.21.15.62:0/3653937692 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f487c0ffbd0 0x7f487c196d20 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f486c00b4b0 tx=0x7f486c00b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:26:22.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.678+0000 7f4878c00700 1 -- 172.21.15.62:0/3653937692 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f486c011800 con 0x7f487c0ffbd0 2024-04-20T21:26:22.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.678+0000 7f4878c00700 1 -- 172.21.15.62:0/3653937692 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f486c011e10 con 0x7f487c0ffbd0 2024-04-20T21:26:22.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.678+0000 7f4882a00700 1 -- 172.21.15.62:0/3653937692 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f487c19fd90 con 0x7f487c0ffbd0 2024-04-20T21:26:22.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.678+0000 7f4878c00700 1 -- 172.21.15.62:0/3653937692 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f486c00f2e0 con 0x7f487c0ffbd0 2024-04-20T21:26:22.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.678+0000 7f4882a00700 1 -- 172.21.15.62:0/3653937692 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f487c1a0250 con 0x7f487c0ffbd0 2024-04-20T21:26:22.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.679+0000 7f4882a00700 1 -- 172.21.15.62:0/3653937692 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f487c10c600 con 0x7f487c0ffbd0 2024-04-20T21:26:22.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.680+0000 7f4878c00700 1 -- 172.21.15.62:0/3653937692 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f486c010380 con 0x7f487c0ffbd0 2024-04-20T21:26:22.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.681+0000 7f4878c00700 1 --2- 172.21.15.62:0/3653937692 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f486006c680 0x7f486006eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:26:22.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.681+0000 7f4878c00700 1 -- 172.21.15.62:0/3653937692 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f486c08b220 con 0x7f487c0ffbd0 2024-04-20T21:26:22.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.682+0000 7f487b400700 1 --2- 172.21.15.62:0/3653937692 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f486006c680 0x7f486006eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:26:22.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.682+0000 7f4878c00700 1 --2- 172.21.15.62:0/3653937692 >> [v2:172.21.15.189:6816/893231043,v1:172.21.15.189:6817/893231043] conn(0x7f4860072350 0x7f4860074760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:26:22.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.682+0000 7f4878c00700 1 -- 172.21.15.62:0/3653937692 --> [v2:172.21.15.189:6816/893231043,v1:172.21.15.189:6817/893231043] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4860074e10 con 0x7f4860072350 2024-04-20T21:26:22.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.682+0000 7f4878c00700 1 -- 172.21.15.62:0/3653937692 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f486c08d850 con 0x7f487c0ffbd0 2024-04-20T21:26:22.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.682+0000 7f487b400700 1 --2- 172.21.15.62:0/3653937692 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f486006c680 0x7f486006eb30 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4868006010 tx=0x7f486800b540 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:26:22.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.682+0000 7f4880a00700 1 --2- 172.21.15.62:0/3653937692 >> [v2:172.21.15.189:6816/893231043,v1:172.21.15.189:6817/893231043] conn(0x7f4860072350 0x7f4860074760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:26:22.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.683+0000 7f4880a00700 1 --2- 172.21.15.62:0/3653937692 >> [v2:172.21.15.189:6816/893231043,v1:172.21.15.189:6817/893231043] conn(0x7f4860072350 0x7f4860074760 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:26:22.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.685+0000 7f4878c00700 1 -- 172.21.15.62:0/3653937692 <== osd.5 v2:172.21.15.189:6816/893231043 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f4860074e10 con 0x7f4860072350 2024-04-20T21:26:22.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.703+0000 7f4882a00700 1 -- 172.21.15.62:0/3653937692 --> [v2:172.21.15.189:6816/893231043,v1:172.21.15.189:6817/893231043] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f487c04f2a0 con 0x7f4860072350 2024-04-20T21:26:22.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.704+0000 7f4878c00700 1 -- 172.21.15.62:0/3653937692 <== osd.5 v2:172.21.15.189:6816/893231043 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f487c04f2a0 con 0x7f4860072350 2024-04-20T21:26:22.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.704+0000 7f4882a00700 1 -- 172.21.15.62:0/3653937692 >> [v2:172.21.15.189:6816/893231043,v1:172.21.15.189:6817/893231043] conn(0x7f4860072350 msgr2=0x7f4860074760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:26:22.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.704+0000 7f4882a00700 1 --2- 172.21.15.62:0/3653937692 >> [v2:172.21.15.189:6816/893231043,v1:172.21.15.189:6817/893231043] conn(0x7f4860072350 0x7f4860074760 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:22.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.705+0000 7f4882a00700 1 -- 172.21.15.62:0/3653937692 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f486006c680 msgr2=0x7f486006eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:26:22.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.705+0000 7f4882a00700 1 --2- 172.21.15.62:0/3653937692 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f486006c680 0x7f486006eb30 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4868006010 tx=0x7f486800b540 comp rx=0 tx=0).stop 2024-04-20T21:26:22.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.705+0000 7f4882a00700 1 -- 172.21.15.62:0/3653937692 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f487c0ffbd0 msgr2=0x7f487c196d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:26:22.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.705+0000 7f4882a00700 1 --2- 172.21.15.62:0/3653937692 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f487c0ffbd0 0x7f487c196d20 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f486c00b4b0 tx=0x7f486c00b870 comp rx=0 tx=0).stop 2024-04-20T21:26:22.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.705+0000 7f4882a00700 1 -- 172.21.15.62:0/3653937692 shutdown_connections 2024-04-20T21:26:22.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.705+0000 7f4882a00700 1 --2- 172.21.15.62:0/3653937692 >> [v2:172.21.15.189:6816/893231043,v1:172.21.15.189:6817/893231043] conn(0x7f4860072350 0x7f4860074760 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:22.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.705+0000 7f4882a00700 1 --2- 172.21.15.62:0/3653937692 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f486006c680 0x7f486006eb30 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:22.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.705+0000 7f4882a00700 1 --2- 172.21.15.62:0/3653937692 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f487c0ffbd0 0x7f487c196d20 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:22.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.705+0000 7f4882a00700 1 --2- 172.21.15.62:0/3653937692 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f487c104130 0x7f487c197260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:22.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.705+0000 7f4882a00700 1 -- 172.21.15.62:0/3653937692 >> 172.21.15.62:0/3653937692 conn(0x7f487c075130 msgr2=0x7f487c0fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:26:22.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.705+0000 7f4882a00700 1 -- 172.21.15.62:0/3653937692 shutdown_connections 2024-04-20T21:26:22.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.705+0000 7f4882a00700 1 -- 172.21.15.62:0/3653937692 wait complete. 2024-04-20T21:26:23.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:26:22 smithi062 ceph-mon[16221]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:26:23.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:26:22 smithi189 ceph-mon[21562]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:26:23.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:23.339+0000 7f0f25a00700 1 -- 172.21.15.62:0/1266068673 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0f20108470 msgr2=0x7f0f20108840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:26:23.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:23.339+0000 7f0f25a00700 1 --2- 172.21.15.62:0/1266068673 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0f20108470 0x7f0f20108840 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f0f10009910 tx=0x7f0f10009c20 comp rx=0 tx=0).stop 2024-04-20T21:26:23.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:23.340+0000 7f0f25a00700 1 -- 172.21.15.62:0/1266068673 shutdown_connections 2024-04-20T21:26:23.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:23.340+0000 7f0f25a00700 1 --2- 172.21.15.62:0/1266068673 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f20102470 0x7f0f201028e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:23.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:23.340+0000 7f0f25a00700 1 --2- 172.21.15.62:0/1266068673 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0f20108470 0x7f0f20108840 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:23.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:23.340+0000 7f0f25a00700 1 -- 172.21.15.62:0/1266068673 >> 172.21.15.62:0/1266068673 conn(0x7f0f200fdf90 msgr2=0x7f0f201003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:26:23.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:23.340+0000 7f0f25a00700 1 -- 172.21.15.62:0/1266068673 shutdown_connections 2024-04-20T21:26:23.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:23.340+0000 7f0f25a00700 1 -- 172.21.15.62:0/1266068673 wait complete. 2024-04-20T21:26:23.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:23.341+0000 7f0f25a00700 1 Processor -- start 2024-04-20T21:26:23.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:23.341+0000 7f0f25a00700 1 -- start start 2024-04-20T21:26:23.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:23.341+0000 7f0f25a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f20102470 0x7f0f201981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:26:23.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:23.341+0000 7f0f25a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0f20108470 0x7f0f20198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:26:23.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:23.341+0000 7f0f25a00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f20198e10 con 0x7f0f20108470 2024-04-20T21:26:23.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:23.341+0000 7f0f25a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f2019cba0 con 0x7f0f20102470 2024-04-20T21:26:23.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:23.342+0000 7f0f1e000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0f20108470 0x7f0f20198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:26:23.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:23.342+0000 7f0f1e000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0f20108470 0x7f0f20198730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51912/0 (socket says 172.21.15.62:51912) 2024-04-20T21:26:23.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:23.342+0000 7f0f1e000700 1 -- 172.21.15.62:0/154602774 learned_addr learned my addr 172.21.15.62:0/154602774 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:26:23.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:23.342+0000 7f0f1ea00700 1 --2- 172.21.15.62:0/154602774 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f20102470 0x7f0f201981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:26:23.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:23.342+0000 7f0f1e000700 1 -- 172.21.15.62:0/154602774 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f20102470 msgr2=0x7f0f201981f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:26:23.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:23.342+0000 7f0f1e000700 1 --2- 172.21.15.62:0/154602774 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f20102470 0x7f0f201981f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:23.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:23.342+0000 7f0f1e000700 1 -- 172.21.15.62:0/154602774 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f100095f0 con 0x7f0f20108470 2024-04-20T21:26:23.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:23.343+0000 7f0f1e000700 1 --2- 172.21.15.62:0/154602774 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0f20108470 0x7f0f20198730 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f0f0c00c8a0 tx=0x7f0f0c00cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:26:23.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:23.343+0000 7f0f17400700 1 -- 172.21.15.62:0/154602774 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0f0c0078e0 con 0x7f0f20108470 2024-04-20T21:26:23.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:23.343+0000 7f0f17400700 1 -- 172.21.15.62:0/154602774 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f0f0c00f450 con 0x7f0f20108470 2024-04-20T21:26:23.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:23.343+0000 7f0f17400700 1 -- 172.21.15.62:0/154602774 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0f0c00e5e0 con 0x7f0f20108470 2024-04-20T21:26:23.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:23.343+0000 7f0f25a00700 1 -- 172.21.15.62:0/154602774 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f2019ce80 con 0x7f0f20108470 2024-04-20T21:26:23.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:23.343+0000 7f0f25a00700 1 -- 172.21.15.62:0/154602774 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f2019d3d0 con 0x7f0f20108470 2024-04-20T21:26:23.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:23.344+0000 7f0f25a00700 1 -- 172.21.15.62:0/154602774 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0f2010f860 con 0x7f0f20108470 2024-04-20T21:26:23.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:23.346+0000 7f0f17400700 1 -- 172.21.15.62:0/154602774 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f0f0c00f5c0 con 0x7f0f20108470 2024-04-20T21:26:23.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:23.346+0000 7f0f17400700 1 --2- 172.21.15.62:0/154602774 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f0f0406c680 0x7f0f0406eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:26:23.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:23.347+0000 7f0f17400700 1 -- 172.21.15.62:0/154602774 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f0f0c08b7d0 con 0x7f0f20108470 2024-04-20T21:26:23.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:23.347+0000 7f0f1ea00700 1 --2- 172.21.15.62:0/154602774 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f0f0406c680 0x7f0f0406eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:26:23.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:23.347+0000 7f0f17400700 1 --2- 172.21.15.62:0/154602774 >> [v2:172.21.15.62:6810/1027109374,v1:172.21.15.62:6811/1027109374] conn(0x7f0f04072350 0x7f0f04074760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:26:23.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:23.347+0000 7f0f17400700 1 -- 172.21.15.62:0/154602774 --> [v2:172.21.15.62:6810/1027109374,v1:172.21.15.62:6811/1027109374] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0f04074e10 con 0x7f0f04072350 2024-04-20T21:26:23.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:23.347+0000 7f0f17400700 1 -- 172.21.15.62:0/154602774 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0f0c08bb90 con 0x7f0f20108470 2024-04-20T21:26:23.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:23.347+0000 7f0f1f400700 1 --2- 172.21.15.62:0/154602774 >> [v2:172.21.15.62:6810/1027109374,v1:172.21.15.62:6811/1027109374] conn(0x7f0f04072350 0x7f0f04074760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:26:23.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:23.348+0000 7f0f1ea00700 1 --2- 172.21.15.62:0/154602774 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f0f0406c680 0x7f0f0406eb30 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0f10009fd0 tx=0x7f0f1001a040 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:26:23.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:23.348+0000 7f0f1f400700 1 --2- 172.21.15.62:0/154602774 >> [v2:172.21.15.62:6810/1027109374,v1:172.21.15.62:6811/1027109374] conn(0x7f0f04072350 0x7f0f04074760 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:26:23.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:23.349+0000 7f0f17400700 1 -- 172.21.15.62:0/154602774 <== osd.1 v2:172.21.15.62:6810/1027109374 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f0f04074e10 con 0x7f0f04072350 2024-04-20T21:26:23.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:23.367+0000 7f0f25a00700 1 -- 172.21.15.62:0/154602774 --> [v2:172.21.15.62:6810/1027109374,v1:172.21.15.62:6811/1027109374] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f0f2004f2a0 con 0x7f0f04072350 2024-04-20T21:26:23.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:23.368+0000 7f0f17400700 1 -- 172.21.15.62:0/154602774 <== osd.1 v2:172.21.15.62:6810/1027109374 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f0f2004f2a0 con 0x7f0f04072350 2024-04-20T21:26:23.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:23.368+0000 7f0f25a00700 1 -- 172.21.15.62:0/154602774 >> [v2:172.21.15.62:6810/1027109374,v1:172.21.15.62:6811/1027109374] conn(0x7f0f04072350 msgr2=0x7f0f04074760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:26:23.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:23.368+0000 7f0f25a00700 1 --2- 172.21.15.62:0/154602774 >> [v2:172.21.15.62:6810/1027109374,v1:172.21.15.62:6811/1027109374] conn(0x7f0f04072350 0x7f0f04074760 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:23.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:23.368+0000 7f0f25a00700 1 -- 172.21.15.62:0/154602774 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f0f0406c680 msgr2=0x7f0f0406eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:26:23.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:23.368+0000 7f0f25a00700 1 --2- 172.21.15.62:0/154602774 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f0f0406c680 0x7f0f0406eb30 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0f10009fd0 tx=0x7f0f1001a040 comp rx=0 tx=0).stop 2024-04-20T21:26:23.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:23.368+0000 7f0f25a00700 1 -- 172.21.15.62:0/154602774 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0f20108470 msgr2=0x7f0f20198730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:26:23.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:23.368+0000 7f0f25a00700 1 --2- 172.21.15.62:0/154602774 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0f20108470 0x7f0f20198730 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f0f0c00c8a0 tx=0x7f0f0c00cbb0 comp rx=0 tx=0).stop 2024-04-20T21:26:23.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:23.369+0000 7f0f25a00700 1 -- 172.21.15.62:0/154602774 shutdown_connections 2024-04-20T21:26:23.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:23.369+0000 7f0f25a00700 1 --2- 172.21.15.62:0/154602774 >> [v2:172.21.15.62:6810/1027109374,v1:172.21.15.62:6811/1027109374] conn(0x7f0f04072350 0x7f0f04074760 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:23.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:23.369+0000 7f0f25a00700 1 --2- 172.21.15.62:0/154602774 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f0f0406c680 0x7f0f0406eb30 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:23.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:23.369+0000 7f0f25a00700 1 --2- 172.21.15.62:0/154602774 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f20102470 0x7f0f201981f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:23.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:23.369+0000 7f0f25a00700 1 --2- 172.21.15.62:0/154602774 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0f20108470 0x7f0f20198730 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:23.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:23.369+0000 7f0f25a00700 1 -- 172.21.15.62:0/154602774 >> 172.21.15.62:0/154602774 conn(0x7f0f200fdf90 msgr2=0x7f0f201002c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:26:23.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:23.369+0000 7f0f25a00700 1 -- 172.21.15.62:0/154602774 shutdown_connections 2024-04-20T21:26:23.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:23.369+0000 7f0f25a00700 1 -- 172.21.15.62:0/154602774 wait complete. 2024-04-20T21:26:23.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.545+0000 7f123d600700 1 -- 172.21.15.62:0/979027829 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12380732f0 msgr2=0x7f12380736c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:26:23.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.545+0000 7f123d600700 1 --2- 172.21.15.62:0/979027829 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12380732f0 0x7f12380736c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f1228009910 tx=0x7f1228009c20 comp rx=0 tx=0).stop 2024-04-20T21:26:23.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.545+0000 7f123d600700 1 -- 172.21.15.62:0/979027829 shutdown_connections 2024-04-20T21:26:23.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.545+0000 7f123d600700 1 --2- 172.21.15.62:0/979027829 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1238073c00 0x7f123810c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:23.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.545+0000 7f123d600700 1 --2- 172.21.15.62:0/979027829 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12380732f0 0x7f12380736c0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:23.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.545+0000 7f123d600700 1 -- 172.21.15.62:0/979027829 >> 172.21.15.62:0/979027829 conn(0x7f12380fbbd0 msgr2=0x7f12380fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:26:23.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.545+0000 7f123d600700 1 -- 172.21.15.62:0/979027829 shutdown_connections 2024-04-20T21:26:23.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.545+0000 7f123d600700 1 -- 172.21.15.62:0/979027829 wait complete. 2024-04-20T21:26:23.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.547+0000 7f123d600700 1 Processor -- start 2024-04-20T21:26:23.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.547+0000 7f123d600700 1 -- start start 2024-04-20T21:26:23.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.548+0000 7f123d600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12380732f0 0x7f1238197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:26:23.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.548+0000 7f123d600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1238073c00 0x7f1238198440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:26:23.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.548+0000 7f123d600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1238198b20 con 0x7f1238073c00 2024-04-20T21:26:23.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.548+0000 7f123d600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f123819c8b0 con 0x7f12380732f0 2024-04-20T21:26:23.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.548+0000 7f122f400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1238073c00 0x7f1238198440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:26:23.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.548+0000 7f122f400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1238073c00 0x7f1238198440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51858/0 (socket says 172.21.15.62:51858) 2024-04-20T21:26:23.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.548+0000 7f122f400700 1 -- 172.21.15.62:0/2988056612 learned_addr learned my addr 172.21.15.62:0/2988056612 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:26:23.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.549+0000 7f1237e00700 1 --2- 172.21.15.62:0/2988056612 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12380732f0 0x7f1238197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:26:23.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.549+0000 7f122f400700 1 -- 172.21.15.62:0/2988056612 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12380732f0 msgr2=0x7f1238197f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:26:23.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.549+0000 7f122f400700 1 --2- 172.21.15.62:0/2988056612 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12380732f0 0x7f1238197f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:23.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.549+0000 7f122f400700 1 -- 172.21.15.62:0/2988056612 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12280095f0 con 0x7f1238073c00 2024-04-20T21:26:23.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.549+0000 7f122f400700 1 --2- 172.21.15.62:0/2988056612 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1238073c00 0x7f1238198440 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f12200060b0 tx=0x7f122000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:26:23.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.550+0000 7f1235600700 1 -- 172.21.15.62:0/2988056612 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f122000dda0 con 0x7f1238073c00 2024-04-20T21:26:23.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.550+0000 7f123d600700 1 -- 172.21.15.62:0/2988056612 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f123819cb90 con 0x7f1238073c00 2024-04-20T21:26:23.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.550+0000 7f1235600700 1 -- 172.21.15.62:0/2988056612 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f122000f040 con 0x7f1238073c00 2024-04-20T21:26:23.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.550+0000 7f1235600700 1 -- 172.21.15.62:0/2988056612 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1220014790 con 0x7f1238073c00 2024-04-20T21:26:23.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.550+0000 7f123d600700 1 -- 172.21.15.62:0/2988056612 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f123819d0e0 con 0x7f1238073c00 2024-04-20T21:26:23.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.551+0000 7f123d600700 1 -- 172.21.15.62:0/2988056612 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f123810f7a0 con 0x7f1238073c00 2024-04-20T21:26:23.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.552+0000 7f1235600700 1 -- 172.21.15.62:0/2988056612 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f12200148f0 con 0x7f1238073c00 2024-04-20T21:26:23.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.553+0000 7f1235600700 1 --2- 172.21.15.62:0/2988056612 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f121c06c680 0x7f121c06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:26:23.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.553+0000 7f1235600700 1 -- 172.21.15.62:0/2988056612 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f122008b8b0 con 0x7f1238073c00 2024-04-20T21:26:23.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.554+0000 7f1237e00700 1 --2- 172.21.15.62:0/2988056612 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f121c06c680 0x7f121c06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:26:23.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.554+0000 7f1235600700 1 --2- 172.21.15.62:0/2988056612 >> [v2:172.21.15.189:6808/191133671,v1:172.21.15.189:6809/191133671] conn(0x7f121c072350 0x7f121c074760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:26:23.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.554+0000 7f1235600700 1 -- 172.21.15.62:0/2988056612 --> [v2:172.21.15.189:6808/191133671,v1:172.21.15.189:6809/191133671] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f121c074e10 con 0x7f121c072350 2024-04-20T21:26:23.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.554+0000 7f1235600700 1 -- 172.21.15.62:0/2988056612 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f122008bc70 con 0x7f1238073c00 2024-04-20T21:26:23.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.554+0000 7f1237e00700 1 --2- 172.21.15.62:0/2988056612 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f121c06c680 0x7f121c06eb30 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f122800b850 tx=0x7f1228005ad0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:26:23.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.554+0000 7f123cc00700 1 --2- 172.21.15.62:0/2988056612 >> [v2:172.21.15.189:6808/191133671,v1:172.21.15.189:6809/191133671] conn(0x7f121c072350 0x7f121c074760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:26:23.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.555+0000 7f123cc00700 1 --2- 172.21.15.62:0/2988056612 >> [v2:172.21.15.189:6808/191133671,v1:172.21.15.189:6809/191133671] conn(0x7f121c072350 0x7f121c074760 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:26:23.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.557+0000 7f1235600700 1 -- 172.21.15.62:0/2988056612 <== osd.4 v2:172.21.15.189:6808/191133671 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f121c074e10 con 0x7f121c072350 2024-04-20T21:26:23.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.575+0000 7f123d600700 1 -- 172.21.15.62:0/2988056612 --> [v2:172.21.15.189:6808/191133671,v1:172.21.15.189:6809/191133671] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f1238062380 con 0x7f121c072350 2024-04-20T21:26:23.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.576+0000 7f1235600700 1 -- 172.21.15.62:0/2988056612 <== osd.4 v2:172.21.15.189:6808/191133671 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f1238062380 con 0x7f121c072350 2024-04-20T21:26:23.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.576+0000 7f123d600700 1 -- 172.21.15.62:0/2988056612 >> [v2:172.21.15.189:6808/191133671,v1:172.21.15.189:6809/191133671] conn(0x7f121c072350 msgr2=0x7f121c074760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:26:23.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.576+0000 7f123d600700 1 --2- 172.21.15.62:0/2988056612 >> [v2:172.21.15.189:6808/191133671,v1:172.21.15.189:6809/191133671] conn(0x7f121c072350 0x7f121c074760 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:23.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.577+0000 7f123d600700 1 -- 172.21.15.62:0/2988056612 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f121c06c680 msgr2=0x7f121c06eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:26:23.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.577+0000 7f123d600700 1 --2- 172.21.15.62:0/2988056612 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f121c06c680 0x7f121c06eb30 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f122800b850 tx=0x7f1228005ad0 comp rx=0 tx=0).stop 2024-04-20T21:26:23.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.577+0000 7f123d600700 1 -- 172.21.15.62:0/2988056612 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1238073c00 msgr2=0x7f1238198440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:26:23.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.577+0000 7f123d600700 1 --2- 172.21.15.62:0/2988056612 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1238073c00 0x7f1238198440 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f12200060b0 tx=0x7f122000d3b0 comp rx=0 tx=0).stop 2024-04-20T21:26:23.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.577+0000 7f123d600700 1 -- 172.21.15.62:0/2988056612 shutdown_connections 2024-04-20T21:26:23.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.577+0000 7f123d600700 1 --2- 172.21.15.62:0/2988056612 >> [v2:172.21.15.189:6808/191133671,v1:172.21.15.189:6809/191133671] conn(0x7f121c072350 0x7f121c074760 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:23.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.577+0000 7f123d600700 1 --2- 172.21.15.62:0/2988056612 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f121c06c680 0x7f121c06eb30 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:23.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.577+0000 7f123d600700 1 --2- 172.21.15.62:0/2988056612 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12380732f0 0x7f1238197f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:23.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.577+0000 7f123d600700 1 --2- 172.21.15.62:0/2988056612 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1238073c00 0x7f1238198440 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:23.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.577+0000 7f123d600700 1 -- 172.21.15.62:0/2988056612 >> 172.21.15.62:0/2988056612 conn(0x7f12380fbbd0 msgr2=0x7f1238106c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:26:23.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.577+0000 7f123d600700 1 -- 172.21.15.62:0/2988056612 shutdown_connections 2024-04-20T21:26:23.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:22.577+0000 7f123d600700 1 -- 172.21.15.62:0/2988056612 wait complete. 2024-04-20T21:26:24.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:26:23 smithi189 ceph-mon[21562]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:26:24.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:26:23 smithi062 ceph-mon[16221]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:26:24.512 INFO:teuthology.orchestra.run.smithi062.stdout:146028888082 2024-04-20T21:26:24.512 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-04-20T21:26:25.195 INFO:teuthology.orchestra.run.smithi062.stdout:60129542182 2024-04-20T21:26:25.195 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-04-20T21:26:25.855 INFO:teuthology.orchestra.run.smithi062.stdout:111669149720 2024-04-20T21:26:25.855 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-04-20T21:26:25.951 INFO:teuthology.orchestra.run.smithi062.stdout:167503724554 2024-04-20T21:26:25.951 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-04-20T21:26:26.057 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:26:25 smithi062 ceph-mon[16221]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:26:26.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:26:25 smithi189 ceph-mon[21562]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:26:26.237 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T21:26:26.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:26.242+0000 7fb15f600700 1 -- 172.21.15.62:0/2690191083 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb158107de0 msgr2=0x7fb1581081b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:26:26.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:26.242+0000 7fb15f600700 1 --2- 172.21.15.62:0/2690191083 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb158107de0 0x7fb1581081b0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fb14c009910 tx=0x7fb14c009c20 comp rx=0 tx=0).stop 2024-04-20T21:26:26.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:26.243+0000 7fb15f600700 1 -- 172.21.15.62:0/2690191083 shutdown_connections 2024-04-20T21:26:26.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:26.243+0000 7fb15f600700 1 --2- 172.21.15.62:0/2690191083 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb158108780 0x7fb158110dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:26.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:26.243+0000 7fb15f600700 1 --2- 172.21.15.62:0/2690191083 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb158107de0 0x7fb1581081b0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:26.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:26.243+0000 7fb15f600700 1 -- 172.21.15.62:0/2690191083 >> 172.21.15.62:0/2690191083 conn(0x7fb15806ca90 msgr2=0x7fb15806ce90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:26:26.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:26.243+0000 7fb15f600700 1 -- 172.21.15.62:0/2690191083 shutdown_connections 2024-04-20T21:26:26.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:26.243+0000 7fb15f600700 1 -- 172.21.15.62:0/2690191083 wait complete. 2024-04-20T21:26:26.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:26.245+0000 7fb15f600700 1 Processor -- start 2024-04-20T21:26:26.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:26.245+0000 7fb15f600700 1 -- start start 2024-04-20T21:26:26.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:26.245+0000 7fb15f600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb158107de0 0x7fb15819c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:26:26.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:26.245+0000 7fb15f600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb158108780 0x7fb15819c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:26:26.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:26.245+0000 7fb15f600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb15819cf10 con 0x7fb158107de0 2024-04-20T21:26:26.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:26.245+0000 7fb15f600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1581a0ca0 con 0x7fb158108780 2024-04-20T21:26:26.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:26.246+0000 7fb15cc00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb158107de0 0x7fb15819c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:26:26.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:26.246+0000 7fb15cc00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb158107de0 0x7fb15819c2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51922/0 (socket says 172.21.15.62:51922) 2024-04-20T21:26:26.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:26.246+0000 7fb15cc00700 1 -- 172.21.15.62:0/3275362960 learned_addr learned my addr 172.21.15.62:0/3275362960 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:26:26.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:26.246+0000 7fb157e00700 1 --2- 172.21.15.62:0/3275362960 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb158108780 0x7fb15819c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:26:26.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:26.246+0000 7fb15cc00700 1 -- 172.21.15.62:0/3275362960 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb158108780 msgr2=0x7fb15819c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:26:26.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:26.248+0000 7fb15cc00700 1 --2- 172.21.15.62:0/3275362960 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb158108780 0x7fb15819c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:26.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:26.248+0000 7fb15cc00700 1 -- 172.21.15.62:0/3275362960 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb14c0095f0 con 0x7fb158107de0 2024-04-20T21:26:26.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:26.248+0000 7fb157e00700 1 --2- 172.21.15.62:0/3275362960 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb158108780 0x7fb15819c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-20T21:26:26.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:26.249+0000 7fb15cc00700 1 --2- 172.21.15.62:0/3275362960 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb158107de0 0x7fb15819c2f0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fb14c00b5c0 tx=0x7fb14c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:26:26.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:26.250+0000 7fb155600700 1 -- 172.21.15.62:0/3275362960 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb14c023aa0 con 0x7fb158107de0 2024-04-20T21:26:26.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:26.250+0000 7fb155600700 1 -- 172.21.15.62:0/3275362960 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fb14c027070 con 0x7fb158107de0 2024-04-20T21:26:26.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:26.250+0000 7fb155600700 1 -- 172.21.15.62:0/3275362960 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb14c00f690 con 0x7fb158107de0 2024-04-20T21:26:26.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:26.250+0000 7fb15f600700 1 -- 172.21.15.62:0/3275362960 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1581a0f80 con 0x7fb158107de0 2024-04-20T21:26:26.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:26.250+0000 7fb15f600700 1 -- 172.21.15.62:0/3275362960 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1581a14d0 con 0x7fb158107de0 2024-04-20T21:26:26.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:26.252+0000 7fb15f600700 1 -- 172.21.15.62:0/3275362960 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb158113be0 con 0x7fb158107de0 2024-04-20T21:26:26.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:26.255+0000 7fb155600700 1 -- 172.21.15.62:0/3275362960 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7fb14c029030 con 0x7fb158107de0 2024-04-20T21:26:26.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:26.256+0000 7fb155600700 1 --2- 172.21.15.62:0/3275362960 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fb14006c680 0x7fb14006eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:26:26.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:26.256+0000 7fb155600700 1 -- 172.21.15.62:0/3275362960 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fb14c095220 con 0x7fb158107de0 2024-04-20T21:26:26.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:26.257+0000 7fb155600700 1 --2- 172.21.15.62:0/3275362960 >> [v2:172.21.15.62:6802/2687023571,v1:172.21.15.62:6803/2687023571] conn(0x7fb140072350 0x7fb140074760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:26:26.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:26.257+0000 7fb155600700 1 -- 172.21.15.62:0/3275362960 --> [v2:172.21.15.62:6802/2687023571,v1:172.21.15.62:6803/2687023571] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb140074e10 con 0x7fb140072350 2024-04-20T21:26:26.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:26.257+0000 7fb155600700 1 -- 172.21.15.62:0/3275362960 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb14c0955e0 con 0x7fb158107de0 2024-04-20T21:26:26.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:26.257+0000 7fb157e00700 1 --2- 172.21.15.62:0/3275362960 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fb14006c680 0x7fb14006eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:26:26.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:26.258+0000 7fb15d600700 1 --2- 172.21.15.62:0/3275362960 >> [v2:172.21.15.62:6802/2687023571,v1:172.21.15.62:6803/2687023571] conn(0x7fb140072350 0x7fb140074760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:26:26.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:26.259+0000 7fb15d600700 1 --2- 172.21.15.62:0/3275362960 >> [v2:172.21.15.62:6802/2687023571,v1:172.21.15.62:6803/2687023571] conn(0x7fb140072350 0x7fb140074760 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:26:26.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:26.259+0000 7fb157e00700 1 --2- 172.21.15.62:0/3275362960 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fb14006c680 0x7fb14006eb30 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb15819d910 tx=0x7fb14400b5f0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:26:26.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:26.264+0000 7fb155600700 1 -- 172.21.15.62:0/3275362960 <== osd.0 v2:172.21.15.62:6802/2687023571 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fb140074e10 con 0x7fb140072350 2024-04-20T21:26:26.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:26.287+0000 7fb15f600700 1 -- 172.21.15.62:0/3275362960 --> [v2:172.21.15.62:6802/2687023571,v1:172.21.15.62:6803/2687023571] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fb1580685c0 con 0x7fb140072350 2024-04-20T21:26:26.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:26.287+0000 7fb155600700 1 -- 172.21.15.62:0/3275362960 <== osd.0 v2:172.21.15.62:6802/2687023571 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fb1580685c0 con 0x7fb140072350 2024-04-20T21:26:26.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:26.288+0000 7fb15f600700 1 -- 172.21.15.62:0/3275362960 >> [v2:172.21.15.62:6802/2687023571,v1:172.21.15.62:6803/2687023571] conn(0x7fb140072350 msgr2=0x7fb140074760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:26:26.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:26.288+0000 7fb15f600700 1 --2- 172.21.15.62:0/3275362960 >> [v2:172.21.15.62:6802/2687023571,v1:172.21.15.62:6803/2687023571] conn(0x7fb140072350 0x7fb140074760 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:26.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:26.288+0000 7fb15f600700 1 -- 172.21.15.62:0/3275362960 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fb14006c680 msgr2=0x7fb14006eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:26:26.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:26.288+0000 7fb15f600700 1 --2- 172.21.15.62:0/3275362960 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fb14006c680 0x7fb14006eb30 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb15819d910 tx=0x7fb14400b5f0 comp rx=0 tx=0).stop 2024-04-20T21:26:26.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:26.288+0000 7fb15f600700 1 -- 172.21.15.62:0/3275362960 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb158107de0 msgr2=0x7fb15819c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:26:26.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:26.288+0000 7fb15f600700 1 --2- 172.21.15.62:0/3275362960 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb158107de0 0x7fb15819c2f0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fb14c00b5c0 tx=0x7fb14c00be20 comp rx=0 tx=0).stop 2024-04-20T21:26:26.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:26.288+0000 7fb15f600700 1 -- 172.21.15.62:0/3275362960 shutdown_connections 2024-04-20T21:26:26.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:26.288+0000 7fb15f600700 1 --2- 172.21.15.62:0/3275362960 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fb14006c680 0x7fb14006eb30 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:26.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:26.289+0000 7fb15f600700 1 --2- 172.21.15.62:0/3275362960 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb158107de0 0x7fb15819c2f0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:26.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:26.289+0000 7fb15f600700 1 --2- 172.21.15.62:0/3275362960 >> [v2:172.21.15.62:6802/2687023571,v1:172.21.15.62:6803/2687023571] conn(0x7fb140072350 0x7fb140074760 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:26.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:26.289+0000 7fb15f600700 1 --2- 172.21.15.62:0/3275362960 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb158108780 0x7fb15819c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:26.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:26.289+0000 7fb15f600700 1 -- 172.21.15.62:0/3275362960 >> 172.21.15.62:0/3275362960 conn(0x7fb15806ca90 msgr2=0x7fb15810b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:26:26.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:26.289+0000 7fb15f600700 1 -- 172.21.15.62:0/3275362960 shutdown_connections 2024-04-20T21:26:26.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:26.289+0000 7fb15f600700 1 -- 172.21.15.62:0/3275362960 wait complete. 2024-04-20T21:26:27.788 INFO:teuthology.orchestra.run.smithi062.stdout:38654705710 2024-04-20T21:26:27.789 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-04-20T21:26:27.825 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T21:26:27.829 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T21:26:27.840 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T21:26:27.842 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T21:26:27.988 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:26:27 smithi062 ceph-mon[16221]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:26:28.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:26:27 smithi189 ceph-mon[21562]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:26:30.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:26:29 smithi062 ceph-mon[16221]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:26:30.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:26:29 smithi189 ceph-mon[21562]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:26:31.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:31.789+0000 7ff527800700 1 -- 172.21.15.62:0/3017835507 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff5200730b0 msgr2=0x7ff520073480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:26:31.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:31.789+0000 7ff527800700 1 --2- 172.21.15.62:0/3017835507 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff5200730b0 0x7ff520073480 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7ff510009940 tx=0x7ff510009c50 comp rx=0 tx=0).stop 2024-04-20T21:26:31.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:31.790+0000 7ff527800700 1 -- 172.21.15.62:0/3017835507 shutdown_connections 2024-04-20T21:26:31.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:31.790+0000 7ff527800700 1 --2- 172.21.15.62:0/3017835507 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff5200739c0 0x7ff52010c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:31.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:31.790+0000 7ff527800700 1 --2- 172.21.15.62:0/3017835507 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff5200730b0 0x7ff520073480 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:31.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:31.790+0000 7ff527800700 1 -- 172.21.15.62:0/3017835507 >> 172.21.15.62:0/3017835507 conn(0x7ff5200fbda0 msgr2=0x7ff5200fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:26:31.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:31.790+0000 7ff527800700 1 -- 172.21.15.62:0/3017835507 shutdown_connections 2024-04-20T21:26:31.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:31.790+0000 7ff527800700 1 -- 172.21.15.62:0/3017835507 wait complete. 2024-04-20T21:26:31.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:31.791+0000 7ff527800700 1 Processor -- start 2024-04-20T21:26:31.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:31.791+0000 7ff527800700 1 -- start start 2024-04-20T21:26:31.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:31.792+0000 7ff527800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff5200730b0 0x7ff520198210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:26:31.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:31.792+0000 7ff527800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff5200739c0 0x7ff520198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:26:31.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:31.792+0000 7ff527800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff520198e30 con 0x7ff5200739c0 2024-04-20T21:26:31.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:31.792+0000 7ff527800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff52019cbc0 con 0x7ff5200730b0 2024-04-20T21:26:31.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:31.792+0000 7ff517e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff5200739c0 0x7ff520198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:26:31.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:31.792+0000 7ff517e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff5200739c0 0x7ff520198750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40284/0 (socket says 172.21.15.62:40284) 2024-04-20T21:26:31.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:31.792+0000 7ff517e00700 1 -- 172.21.15.62:0/90824021 learned_addr learned my addr 172.21.15.62:0/90824021 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:26:31.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:31.792+0000 7ff524e00700 1 --2- 172.21.15.62:0/90824021 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff5200730b0 0x7ff520198210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:26:31.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:31.792+0000 7ff517e00700 1 -- 172.21.15.62:0/90824021 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff5200730b0 msgr2=0x7ff520198210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:26:31.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:31.792+0000 7ff517e00700 1 --2- 172.21.15.62:0/90824021 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff5200730b0 0x7ff520198210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:31.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:31.792+0000 7ff517e00700 1 -- 172.21.15.62:0/90824021 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5100095f0 con 0x7ff5200739c0 2024-04-20T21:26:31.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:31.793+0000 7ff517e00700 1 --2- 172.21.15.62:0/90824021 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff5200739c0 0x7ff520198750 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7ff50800e920 tx=0x7ff50800ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:26:31.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:31.793+0000 7ff51e000700 1 -- 172.21.15.62:0/90824021 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff50800c9d0 con 0x7ff5200739c0 2024-04-20T21:26:31.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:31.793+0000 7ff51e000700 1 -- 172.21.15.62:0/90824021 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff508010430 con 0x7ff5200739c0 2024-04-20T21:26:31.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:31.793+0000 7ff527800700 1 -- 172.21.15.62:0/90824021 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff52019cea0 con 0x7ff5200739c0 2024-04-20T21:26:31.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:31.793+0000 7ff51e000700 1 -- 172.21.15.62:0/90824021 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff508018650 con 0x7ff5200739c0 2024-04-20T21:26:31.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:31.793+0000 7ff527800700 1 -- 172.21.15.62:0/90824021 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff52019d3f0 con 0x7ff5200739c0 2024-04-20T21:26:31.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:31.794+0000 7ff527800700 1 -- 172.21.15.62:0/90824021 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff520109cc0 con 0x7ff5200739c0 2024-04-20T21:26:31.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:31.795+0000 7ff51e000700 1 -- 172.21.15.62:0/90824021 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7ff5080105a0 con 0x7ff5200739c0 2024-04-20T21:26:31.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:31.797+0000 7ff51e000700 1 --2- 172.21.15.62:0/90824021 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7ff50406c680 0x7ff50406eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:26:31.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:31.797+0000 7ff51e000700 1 -- 172.21.15.62:0/90824021 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7ff508014070 con 0x7ff5200739c0 2024-04-20T21:26:31.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:31.797+0000 7ff524e00700 1 --2- 172.21.15.62:0/90824021 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7ff50406c680 0x7ff50406eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:26:31.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:31.798+0000 7ff524e00700 1 --2- 172.21.15.62:0/90824021 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7ff50406c680 0x7ff50406eb30 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff510006010 tx=0x7ff510005ad0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:26:31.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:31.802+0000 7ff51e000700 1 -- 172.21.15.62:0/90824021 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7ff508051720 con 0x7ff5200739c0 2024-04-20T21:26:32.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:32.029+0000 7ff527800700 1 -- 172.21.15.62:0/90824021 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7ff52019d660 con 0x7ff5200739c0 2024-04-20T21:26:32.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:32.030+0000 7ff51e000700 1 -- 172.21.15.62:0/90824021 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ff508056350 con 0x7ff5200739c0 2024-04-20T21:26:32.030 INFO:teuthology.orchestra.run.smithi062.stdout:81604378657 2024-04-20T21:26:32.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:32.033+0000 7ff527800700 1 -- 172.21.15.62:0/90824021 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7ff50406c680 msgr2=0x7ff50406eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:26:32.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:32.033+0000 7ff527800700 1 --2- 172.21.15.62:0/90824021 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7ff50406c680 0x7ff50406eb30 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff510006010 tx=0x7ff510005ad0 comp rx=0 tx=0).stop 2024-04-20T21:26:32.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:32.033+0000 7ff527800700 1 -- 172.21.15.62:0/90824021 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff5200739c0 msgr2=0x7ff520198750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:26:32.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:32.033+0000 7ff527800700 1 --2- 172.21.15.62:0/90824021 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff5200739c0 0x7ff520198750 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7ff50800e920 tx=0x7ff50800ec30 comp rx=0 tx=0).stop 2024-04-20T21:26:32.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:32.033+0000 7ff527800700 1 -- 172.21.15.62:0/90824021 shutdown_connections 2024-04-20T21:26:32.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:32.033+0000 7ff527800700 1 --2- 172.21.15.62:0/90824021 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7ff50406c680 0x7ff50406eb30 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:32.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:32.033+0000 7ff527800700 1 --2- 172.21.15.62:0/90824021 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff5200730b0 0x7ff520198210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:32.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:32.033+0000 7ff527800700 1 --2- 172.21.15.62:0/90824021 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff5200739c0 0x7ff520198750 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:32.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:32.033+0000 7ff527800700 1 -- 172.21.15.62:0/90824021 >> 172.21.15.62:0/90824021 conn(0x7ff5200fbda0 msgr2=0x7ff520106e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:26:32.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:32.034+0000 7ff527800700 1 -- 172.21.15.62:0/90824021 shutdown_connections 2024-04-20T21:26:32.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:32.034+0000 7ff527800700 1 -- 172.21.15.62:0/90824021 wait complete. 2024-04-20T21:26:32.872 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:26:32 smithi062 ceph-mon[16221]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:26:32.873 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:26:32 smithi062 ceph-mon[16221]: from='client.? 172.21.15.62:0/90824021' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-20T21:26:33.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:26:32 smithi189 ceph-mon[21562]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:26:33.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:26:32 smithi189 ceph-mon[21562]: from='client.? 172.21.15.62:0/90824021' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-20T21:26:33.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:33.834+0000 7fdb1ca00700 1 -- 172.21.15.62:0/2178303639 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdb18100290 msgr2=0x7fdb18100660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:26:33.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:33.834+0000 7fdb1ca00700 1 --2- 172.21.15.62:0/2178303639 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdb18100290 0x7fdb18100660 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fdb0800b0a0 tx=0x7fdb0800b3b0 comp rx=0 tx=0).stop 2024-04-20T21:26:33.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:33.834+0000 7fdb1ca00700 1 -- 172.21.15.62:0/2178303639 shutdown_connections 2024-04-20T21:26:33.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:33.834+0000 7fdb1ca00700 1 --2- 172.21.15.62:0/2178303639 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb18100c30 0x7fdb18104c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:33.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:33.834+0000 7fdb1ca00700 1 --2- 172.21.15.62:0/2178303639 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdb18100290 0x7fdb18100660 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:33.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:33.834+0000 7fdb1ca00700 1 -- 172.21.15.62:0/2178303639 >> 172.21.15.62:0/2178303639 conn(0x7fdb180fbb10 msgr2=0x7fdb180fdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:26:33.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:33.834+0000 7fdb1ca00700 1 -- 172.21.15.62:0/2178303639 shutdown_connections 2024-04-20T21:26:33.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:33.834+0000 7fdb1ca00700 1 -- 172.21.15.62:0/2178303639 wait complete. 2024-04-20T21:26:33.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:33.835+0000 7fdb1ca00700 1 Processor -- start 2024-04-20T21:26:33.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:33.835+0000 7fdb1ca00700 1 -- start start 2024-04-20T21:26:33.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:33.836+0000 7fdb1ca00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdb18100290 0x7fdb18197e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:26:33.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:33.836+0000 7fdb1ca00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb18100c30 0x7fdb18198340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:26:33.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:33.836+0000 7fdb1ca00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb18198a20 con 0x7fdb18100290 2024-04-20T21:26:33.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:33.836+0000 7fdb1ca00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb1819c7b0 con 0x7fdb18100c30 2024-04-20T21:26:33.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:33.836+0000 7fdb16800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdb18100290 0x7fdb18197e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:26:33.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:33.836+0000 7fdb16800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdb18100290 0x7fdb18197e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40296/0 (socket says 172.21.15.62:40296) 2024-04-20T21:26:33.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:33.836+0000 7fdb16800700 1 -- 172.21.15.62:0/1446437193 learned_addr learned my addr 172.21.15.62:0/1446437193 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:26:33.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:33.836+0000 7fdb15e00700 1 --2- 172.21.15.62:0/1446437193 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb18100c30 0x7fdb18198340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:26:33.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:33.836+0000 7fdb16800700 1 -- 172.21.15.62:0/1446437193 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb18100c30 msgr2=0x7fdb18198340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:26:33.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:33.836+0000 7fdb16800700 1 --2- 172.21.15.62:0/1446437193 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb18100c30 0x7fdb18198340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:33.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:33.836+0000 7fdb16800700 1 -- 172.21.15.62:0/1446437193 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb08009d00 con 0x7fdb18100290 2024-04-20T21:26:33.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:33.836+0000 7fdb15e00700 1 --2- 172.21.15.62:0/1446437193 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb18100c30 0x7fdb18198340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T21:26:33.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:33.837+0000 7fdb16800700 1 --2- 172.21.15.62:0/1446437193 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdb18100290 0x7fdb18197e00 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fdb08000f80 tx=0x7fdb0800cdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:26:33.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:33.837+0000 7fdb0f400700 1 -- 172.21.15.62:0/1446437193 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdb0800e040 con 0x7fdb18100290 2024-04-20T21:26:33.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:33.837+0000 7fdb1ca00700 1 -- 172.21.15.62:0/1446437193 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb1819ca30 con 0x7fdb18100290 2024-04-20T21:26:33.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:33.837+0000 7fdb0f400700 1 -- 172.21.15.62:0/1446437193 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fdb08003de0 con 0x7fdb18100290 2024-04-20T21:26:33.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:33.837+0000 7fdb0f400700 1 -- 172.21.15.62:0/1446437193 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdb0801d9a0 con 0x7fdb18100290 2024-04-20T21:26:33.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:33.837+0000 7fdb1ca00700 1 -- 172.21.15.62:0/1446437193 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb1819cf20 con 0x7fdb18100290 2024-04-20T21:26:33.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:33.839+0000 7fdb1ca00700 1 -- 172.21.15.62:0/1446437193 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb18104380 con 0x7fdb18100290 2024-04-20T21:26:33.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:33.840+0000 7fdb0f400700 1 -- 172.21.15.62:0/1446437193 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7fdb08004040 con 0x7fdb18100290 2024-04-20T21:26:33.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:33.841+0000 7fdb0f400700 1 --2- 172.21.15.62:0/1446437193 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fdafc06c680 0x7fdafc06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:26:33.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:33.841+0000 7fdb0f400700 1 -- 172.21.15.62:0/1446437193 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fdb0808cf20 con 0x7fdb18100290 2024-04-20T21:26:33.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:33.841+0000 7fdb15e00700 1 --2- 172.21.15.62:0/1446437193 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fdafc06c680 0x7fdafc06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:26:33.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:33.842+0000 7fdb15e00700 1 --2- 172.21.15.62:0/1446437193 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fdafc06c680 0x7fdafc06eb30 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fdb04005fd0 tx=0x7fdb04005ee0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:26:33.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:33.846+0000 7fdb0f400700 1 -- 172.21.15.62:0/1446437193 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fdb080578a0 con 0x7fdb18100290 2024-04-20T21:26:34.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:34.102+0000 7fdb1ca00700 1 -- 172.21.15.62:0/1446437193 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fdb18062380 con 0x7fdb18100290 2024-04-20T21:26:34.102 INFO:teuthology.orchestra.run.smithi062.stdout:167503724556 2024-04-20T21:26:34.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:34.103+0000 7fdb0f400700 1 -- 172.21.15.62:0/1446437193 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fdb08057070 con 0x7fdb18100290 2024-04-20T21:26:34.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:34.105+0000 7fdb1ca00700 1 -- 172.21.15.62:0/1446437193 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fdafc06c680 msgr2=0x7fdafc06eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:26:34.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:34.105+0000 7fdb1ca00700 1 --2- 172.21.15.62:0/1446437193 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fdafc06c680 0x7fdafc06eb30 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fdb04005fd0 tx=0x7fdb04005ee0 comp rx=0 tx=0).stop 2024-04-20T21:26:34.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:34.106+0000 7fdb1ca00700 1 -- 172.21.15.62:0/1446437193 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdb18100290 msgr2=0x7fdb18197e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:26:34.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:34.106+0000 7fdb1ca00700 1 --2- 172.21.15.62:0/1446437193 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdb18100290 0x7fdb18197e00 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fdb08000f80 tx=0x7fdb0800cdd0 comp rx=0 tx=0).stop 2024-04-20T21:26:34.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:34.106+0000 7fdb1ca00700 1 -- 172.21.15.62:0/1446437193 shutdown_connections 2024-04-20T21:26:34.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:34.106+0000 7fdb1ca00700 1 --2- 172.21.15.62:0/1446437193 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fdafc06c680 0x7fdafc06eb30 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:34.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:34.106+0000 7fdb1ca00700 1 --2- 172.21.15.62:0/1446437193 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdb18100290 0x7fdb18197e00 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:34.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:34.106+0000 7fdb1ca00700 1 --2- 172.21.15.62:0/1446437193 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb18100c30 0x7fdb18198340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:34.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:34.106+0000 7fdb1ca00700 1 -- 172.21.15.62:0/1446437193 >> 172.21.15.62:0/1446437193 conn(0x7fdb180fbb10 msgr2=0x7fdb180fd1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:26:34.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:34.106+0000 7fdb1ca00700 1 -- 172.21.15.62:0/1446437193 shutdown_connections 2024-04-20T21:26:34.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:34.106+0000 7fdb1ca00700 1 -- 172.21.15.62:0/1446437193 wait complete. 2024-04-20T21:26:34.205 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:26:33 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:26:34.205 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:26:33 smithi062 ceph-mon[16221]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:26:34.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:26:33 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:26:34.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:26:33 smithi189 ceph-mon[21562]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:26:35.055 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:26:34 smithi062 ceph-mon[16221]: from='client.? 172.21.15.62:0/1446437193' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-20T21:26:35.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:35.103+0000 7f3b89400700 1 -- 172.21.15.62:0/3451874038 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3b8410c8f0 msgr2=0x7f3b8410ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:26:35.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:35.103+0000 7f3b89400700 1 --2- 172.21.15.62:0/3451874038 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3b8410c8f0 0x7f3b8410ccc0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f3b74009910 tx=0x7f3b74009c20 comp rx=0 tx=0).stop 2024-04-20T21:26:35.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:35.103+0000 7f3b89400700 1 -- 172.21.15.62:0/3451874038 shutdown_connections 2024-04-20T21:26:35.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:35.103+0000 7f3b89400700 1 --2- 172.21.15.62:0/3451874038 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b84106b80 0x7f3b84106ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:35.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:35.103+0000 7f3b89400700 1 --2- 172.21.15.62:0/3451874038 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3b8410c8f0 0x7f3b8410ccc0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:35.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:35.104+0000 7f3b89400700 1 -- 172.21.15.62:0/3451874038 >> 172.21.15.62:0/3451874038 conn(0x7f3b84076020 msgr2=0x7f3b84078430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:26:35.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:35.104+0000 7f3b89400700 1 -- 172.21.15.62:0/3451874038 shutdown_connections 2024-04-20T21:26:35.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:35.104+0000 7f3b89400700 1 -- 172.21.15.62:0/3451874038 wait complete. 2024-04-20T21:26:35.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:35.105+0000 7f3b89400700 1 Processor -- start 2024-04-20T21:26:35.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:35.105+0000 7f3b89400700 1 -- start start 2024-04-20T21:26:35.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:35.105+0000 7f3b89400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3b84106b80 0x7f3b8419c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:26:35.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:35.105+0000 7f3b89400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b8410c8f0 0x7f3b8419c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:26:35.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:35.105+0000 7f3b89400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b8419cff0 con 0x7f3b84106b80 2024-04-20T21:26:35.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:35.105+0000 7f3b89400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b841a0d80 con 0x7f3b8410c8f0 2024-04-20T21:26:35.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:35.106+0000 7f3b82800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3b84106b80 0x7f3b8419c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:26:35.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:35.106+0000 7f3b82800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3b84106b80 0x7f3b8419c3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40316/0 (socket says 172.21.15.62:40316) 2024-04-20T21:26:35.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:35.106+0000 7f3b82800700 1 -- 172.21.15.62:0/222494660 learned_addr learned my addr 172.21.15.62:0/222494660 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:26:35.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:35.106+0000 7f3b81e00700 1 --2- 172.21.15.62:0/222494660 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b8410c8f0 0x7f3b8419c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:26:35.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:35.106+0000 7f3b82800700 1 -- 172.21.15.62:0/222494660 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b8410c8f0 msgr2=0x7f3b8419c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:26:35.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:35.106+0000 7f3b82800700 1 --2- 172.21.15.62:0/222494660 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b8410c8f0 0x7f3b8419c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:35.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:35.106+0000 7f3b82800700 1 -- 172.21.15.62:0/222494660 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b740095f0 con 0x7f3b84106b80 2024-04-20T21:26:35.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:35.107+0000 7f3b82800700 1 --2- 172.21.15.62:0/222494660 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3b84106b80 0x7f3b8419c3d0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f3b7400b5c0 tx=0x7f3b7400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:26:35.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:35.107+0000 7f3b7b400700 1 -- 172.21.15.62:0/222494660 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3b7401d070 con 0x7f3b84106b80 2024-04-20T21:26:35.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:35.107+0000 7f3b7b400700 1 -- 172.21.15.62:0/222494660 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f3b74004fa0 con 0x7f3b84106b80 2024-04-20T21:26:35.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:35.107+0000 7f3b7b400700 1 -- 172.21.15.62:0/222494660 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3b7400f460 con 0x7f3b84106b80 2024-04-20T21:26:35.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:35.107+0000 7f3b89400700 1 -- 172.21.15.62:0/222494660 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b841a1000 con 0x7f3b84106b80 2024-04-20T21:26:35.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:35.107+0000 7f3b89400700 1 -- 172.21.15.62:0/222494660 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b841a14f0 con 0x7f3b84106b80 2024-04-20T21:26:35.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:35.108+0000 7f3b89400700 1 -- 172.21.15.62:0/222494660 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b8404f2a0 con 0x7f3b84106b80 2024-04-20T21:26:35.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:35.109+0000 7f3b7b400700 1 -- 172.21.15.62:0/222494660 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f3b74022810 con 0x7f3b84106b80 2024-04-20T21:26:35.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:35.110+0000 7f3b7b400700 1 --2- 172.21.15.62:0/222494660 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f3b6806c680 0x7f3b6806eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:26:35.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:35.110+0000 7f3b7b400700 1 -- 172.21.15.62:0/222494660 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f3b7408cae0 con 0x7f3b84106b80 2024-04-20T21:26:35.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:35.111+0000 7f3b81e00700 1 --2- 172.21.15.62:0/222494660 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f3b6806c680 0x7f3b6806eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:26:35.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:35.111+0000 7f3b81e00700 1 --2- 172.21.15.62:0/222494660 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f3b6806c680 0x7f3b6806eb30 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f3b70005950 tx=0x7f3b700058e0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:26:35.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:35.115+0000 7f3b7b400700 1 -- 172.21.15.62:0/222494660 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f3b74057510 con 0x7f3b84106b80 2024-04-20T21:26:35.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:26:34 smithi189 ceph-mon[21562]: from='client.? 172.21.15.62:0/1446437193' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-20T21:26:35.297 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378655 got 81604378657 for osd.2 2024-04-20T21:26:35.297 DEBUG:teuthology.parallel:result is None 2024-04-20T21:26:35.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:35.361+0000 7f3b89400700 1 -- 172.21.15.62:0/222494660 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f3b8419d730 con 0x7f3b84106b80 2024-04-20T21:26:35.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:35.362+0000 7f3b7b400700 1 -- 172.21.15.62:0/222494660 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3b74027090 con 0x7f3b84106b80 2024-04-20T21:26:35.361 INFO:teuthology.orchestra.run.smithi062.stdout:60129542184 2024-04-20T21:26:35.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:35.365+0000 7f3b89400700 1 -- 172.21.15.62:0/222494660 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f3b6806c680 msgr2=0x7f3b6806eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:26:35.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:35.365+0000 7f3b89400700 1 --2- 172.21.15.62:0/222494660 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f3b6806c680 0x7f3b6806eb30 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f3b70005950 tx=0x7f3b700058e0 comp rx=0 tx=0).stop 2024-04-20T21:26:35.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:35.365+0000 7f3b89400700 1 -- 172.21.15.62:0/222494660 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3b84106b80 msgr2=0x7f3b8419c3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:26:35.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:35.365+0000 7f3b89400700 1 --2- 172.21.15.62:0/222494660 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3b84106b80 0x7f3b8419c3d0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f3b7400b5c0 tx=0x7f3b7400be20 comp rx=0 tx=0).stop 2024-04-20T21:26:35.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:35.365+0000 7f3b89400700 1 -- 172.21.15.62:0/222494660 shutdown_connections 2024-04-20T21:26:35.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:35.365+0000 7f3b89400700 1 --2- 172.21.15.62:0/222494660 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f3b6806c680 0x7f3b6806eb30 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:35.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:35.365+0000 7f3b89400700 1 --2- 172.21.15.62:0/222494660 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3b84106b80 0x7f3b8419c3d0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:35.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:35.365+0000 7f3b89400700 1 --2- 172.21.15.62:0/222494660 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b8410c8f0 0x7f3b8419c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:35.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:35.365+0000 7f3b89400700 1 -- 172.21.15.62:0/222494660 >> 172.21.15.62:0/222494660 conn(0x7f3b84076020 msgr2=0x7f3b84077940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:26:35.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:35.365+0000 7f3b89400700 1 -- 172.21.15.62:0/222494660 shutdown_connections 2024-04-20T21:26:35.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:35.366+0000 7f3b89400700 1 -- 172.21.15.62:0/222494660 wait complete. 2024-04-20T21:26:36.034 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724554 got 167503724556 for osd.5 2024-04-20T21:26:36.034 DEBUG:teuthology.parallel:result is None 2024-04-20T21:26:36.144 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:26:35 smithi062 ceph-mon[16221]: from='client.? 172.21.15.62:0/222494660' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-20T21:26:36.145 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:26:35 smithi062 ceph-mon[16221]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:26:36.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:26:35 smithi189 ceph-mon[21562]: from='client.? 172.21.15.62:0/222494660' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-20T21:26:36.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:26:35 smithi189 ceph-mon[21562]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:26:36.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:35.880+0000 7f2fff200700 1 -- 172.21.15.62:0/2401538716 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2ff81001b0 msgr2=0x7f2ff8100580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:26:36.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:35.880+0000 7f2fff200700 1 --2- 172.21.15.62:0/2401538716 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2ff81001b0 0x7f2ff8100580 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f2fec009910 tx=0x7f2fec009c20 comp rx=0 tx=0).stop 2024-04-20T21:26:36.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:35.881+0000 7f2fff200700 1 -- 172.21.15.62:0/2401538716 shutdown_connections 2024-04-20T21:26:36.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:35.881+0000 7f2fff200700 1 --2- 172.21.15.62:0/2401538716 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ff8100b50 0x7f2ff8104b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:36.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:35.881+0000 7f2fff200700 1 --2- 172.21.15.62:0/2401538716 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2ff81001b0 0x7f2ff8100580 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:36.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:35.881+0000 7f2fff200700 1 -- 172.21.15.62:0/2401538716 >> 172.21.15.62:0/2401538716 conn(0x7f2ff80fba70 msgr2=0x7f2ff80fde80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:26:36.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:35.881+0000 7f2fff200700 1 -- 172.21.15.62:0/2401538716 shutdown_connections 2024-04-20T21:26:36.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:35.881+0000 7f2fff200700 1 -- 172.21.15.62:0/2401538716 wait complete. 2024-04-20T21:26:36.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:35.882+0000 7f2fff200700 1 Processor -- start 2024-04-20T21:26:36.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:35.882+0000 7f2fff200700 1 -- start start 2024-04-20T21:26:36.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:35.882+0000 7f2fff200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2ff81001b0 0x7f2ff8073050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:26:36.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:35.882+0000 7f2fff200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ff8100b50 0x7f2ff8073590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:26:36.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:35.882+0000 7f2fff200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff8073ad0 con 0x7f2ff81001b0 2024-04-20T21:26:36.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:35.882+0000 7f2fff200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff8073c10 con 0x7f2ff8100b50 2024-04-20T21:26:36.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:35.883+0000 7f2ffde00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2ff81001b0 0x7f2ff8073050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:26:36.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:35.883+0000 7f2ffde00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2ff81001b0 0x7f2ff8073050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40336/0 (socket says 172.21.15.62:40336) 2024-04-20T21:26:36.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:35.883+0000 7f2ffde00700 1 -- 172.21.15.62:0/451908 learned_addr learned my addr 172.21.15.62:0/451908 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:26:36.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:35.883+0000 7f2ffd400700 1 --2- 172.21.15.62:0/451908 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ff8100b50 0x7f2ff8073590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:26:36.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:35.883+0000 7f2ffde00700 1 -- 172.21.15.62:0/451908 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ff8100b50 msgr2=0x7f2ff8073590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:26:36.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:35.883+0000 7f2ffde00700 1 --2- 172.21.15.62:0/451908 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ff8100b50 0x7f2ff8073590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:36.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:35.883+0000 7f2ffde00700 1 -- 172.21.15.62:0/451908 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fec0095f0 con 0x7f2ff81001b0 2024-04-20T21:26:36.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:35.883+0000 7f2ffde00700 1 --2- 172.21.15.62:0/451908 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2ff81001b0 0x7f2ff8073050 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f2fec00b5c0 tx=0x7f2fec00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:26:36.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:35.884+0000 7f2ff2a00700 1 -- 172.21.15.62:0/451908 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2fec01d070 con 0x7f2ff81001b0 2024-04-20T21:26:36.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:35.884+0000 7f2ff2a00700 1 -- 172.21.15.62:0/451908 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f2fec004fa0 con 0x7f2ff81001b0 2024-04-20T21:26:36.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:35.884+0000 7f2fff200700 1 -- 172.21.15.62:0/451908 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ff8073e90 con 0x7f2ff81001b0 2024-04-20T21:26:36.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:35.884+0000 7f2ff2a00700 1 -- 172.21.15.62:0/451908 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2fec00f460 con 0x7f2ff81001b0 2024-04-20T21:26:36.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:35.884+0000 7f2fff200700 1 -- 172.21.15.62:0/451908 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ff81a2010 con 0x7f2ff81001b0 2024-04-20T21:26:36.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:35.885+0000 7f2fff200700 1 -- 172.21.15.62:0/451908 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ff81084c0 con 0x7f2ff81001b0 2024-04-20T21:26:36.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:35.886+0000 7f2ff2a00700 1 -- 172.21.15.62:0/451908 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f2fec022810 con 0x7f2ff81001b0 2024-04-20T21:26:36.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:35.887+0000 7f2ff2a00700 1 --2- 172.21.15.62:0/451908 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f2fe006c680 0x7f2fe006eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:26:36.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:35.887+0000 7f2ff2a00700 1 -- 172.21.15.62:0/451908 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f2fec08c960 con 0x7f2ff81001b0 2024-04-20T21:26:36.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:35.888+0000 7f2ffd400700 1 --2- 172.21.15.62:0/451908 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f2fe006c680 0x7f2fe006eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:26:36.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:35.888+0000 7f2ffd400700 1 --2- 172.21.15.62:0/451908 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f2fe006c680 0x7f2fe006eb30 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f2fe8005950 tx=0x7f2fe80058e0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:26:36.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:35.891+0000 7f2ff2a00700 1 -- 172.21.15.62:0/451908 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f2fec0574b0 con 0x7f2ff81001b0 2024-04-20T21:26:36.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:36.124+0000 7f2fff200700 1 -- 172.21.15.62:0/451908 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f2ff8062380 con 0x7f2ff81001b0 2024-04-20T21:26:36.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:36.125+0000 7f2ff2a00700 1 -- 172.21.15.62:0/451908 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f2fec027020 con 0x7f2ff81001b0 2024-04-20T21:26:36.652 INFO:teuthology.orchestra.run.smithi062.stdout:111669149724 2024-04-20T21:26:36.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:36.128+0000 7f2fff200700 1 -- 172.21.15.62:0/451908 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f2fe006c680 msgr2=0x7f2fe006eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:26:36.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:36.128+0000 7f2fff200700 1 --2- 172.21.15.62:0/451908 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f2fe006c680 0x7f2fe006eb30 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f2fe8005950 tx=0x7f2fe80058e0 comp rx=0 tx=0).stop 2024-04-20T21:26:36.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:36.128+0000 7f2fff200700 1 -- 172.21.15.62:0/451908 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2ff81001b0 msgr2=0x7f2ff8073050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:26:36.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:36.128+0000 7f2fff200700 1 --2- 172.21.15.62:0/451908 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2ff81001b0 0x7f2ff8073050 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f2fec00b5c0 tx=0x7f2fec00be20 comp rx=0 tx=0).stop 2024-04-20T21:26:36.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:36.128+0000 7f2fff200700 1 -- 172.21.15.62:0/451908 shutdown_connections 2024-04-20T21:26:36.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:36.128+0000 7f2fff200700 1 --2- 172.21.15.62:0/451908 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f2fe006c680 0x7f2fe006eb30 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:36.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:36.128+0000 7f2fff200700 1 --2- 172.21.15.62:0/451908 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2ff81001b0 0x7f2ff8073050 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:36.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:36.128+0000 7f2fff200700 1 --2- 172.21.15.62:0/451908 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ff8100b50 0x7f2ff8073590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:36.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:36.128+0000 7f2fff200700 1 -- 172.21.15.62:0/451908 >> 172.21.15.62:0/451908 conn(0x7f2ff80fba70 msgr2=0x7f2ff80fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:26:36.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:36.128+0000 7f2fff200700 1 -- 172.21.15.62:0/451908 shutdown_connections 2024-04-20T21:26:36.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:36.128+0000 7f2fff200700 1 -- 172.21.15.62:0/451908 wait complete. 2024-04-20T21:26:36.875 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T21:26:36.993 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:26:36 smithi062 ceph-mon[16221]: from='client.? 172.21.15.62:0/451908' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-20T21:26:37.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:26:36 smithi189 ceph-mon[21562]: from='client.? 172.21.15.62:0/451908' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-20T21:26:37.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:37.263+0000 7f4e91a00700 1 -- 172.21.15.62:0/155942147 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4e8c1002d0 msgr2=0x7f4e8c1006a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:26:37.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:37.264+0000 7f4e91a00700 1 --2- 172.21.15.62:0/155942147 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4e8c1002d0 0x7f4e8c1006a0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f4e7c009910 tx=0x7f4e7c009c20 comp rx=0 tx=0).stop 2024-04-20T21:26:37.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:37.264+0000 7f4e91a00700 1 -- 172.21.15.62:0/155942147 shutdown_connections 2024-04-20T21:26:37.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:37.264+0000 7f4e91a00700 1 --2- 172.21.15.62:0/155942147 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e8c100c70 0x7f4e8c104c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:37.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:37.264+0000 7f4e91a00700 1 --2- 172.21.15.62:0/155942147 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4e8c1002d0 0x7f4e8c1006a0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:37.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:37.264+0000 7f4e91a00700 1 -- 172.21.15.62:0/155942147 >> 172.21.15.62:0/155942147 conn(0x7f4e8c0fbbb0 msgr2=0x7f4e8c0fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:26:37.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:37.264+0000 7f4e91a00700 1 -- 172.21.15.62:0/155942147 shutdown_connections 2024-04-20T21:26:37.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:37.264+0000 7f4e91a00700 1 -- 172.21.15.62:0/155942147 wait complete. 2024-04-20T21:26:37.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:37.265+0000 7f4e91a00700 1 Processor -- start 2024-04-20T21:26:37.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:37.265+0000 7f4e91a00700 1 -- start start 2024-04-20T21:26:37.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:37.265+0000 7f4e91a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4e8c1002d0 0x7f4e8c073050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:26:37.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:37.265+0000 7f4e91a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e8c100c70 0x7f4e8c073590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:26:37.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:37.266+0000 7f4e91a00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e8c073ad0 con 0x7f4e8c1002d0 2024-04-20T21:26:37.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:37.266+0000 7f4e91a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e8c073c10 con 0x7f4e8c100c70 2024-04-20T21:26:37.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:37.266+0000 7f4e8be00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4e8c1002d0 0x7f4e8c073050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:26:37.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:37.266+0000 7f4e8be00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4e8c1002d0 0x7f4e8c073050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40346/0 (socket says 172.21.15.62:40346) 2024-04-20T21:26:37.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:37.266+0000 7f4e8be00700 1 -- 172.21.15.62:0/1991443862 learned_addr learned my addr 172.21.15.62:0/1991443862 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:26:37.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:37.266+0000 7f4e8b400700 1 --2- 172.21.15.62:0/1991443862 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e8c100c70 0x7f4e8c073590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:26:37.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:37.266+0000 7f4e8be00700 1 -- 172.21.15.62:0/1991443862 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e8c100c70 msgr2=0x7f4e8c073590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:26:37.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:37.266+0000 7f4e8be00700 1 --2- 172.21.15.62:0/1991443862 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e8c100c70 0x7f4e8c073590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:37.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:37.266+0000 7f4e8be00700 1 -- 172.21.15.62:0/1991443862 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e7c0095f0 con 0x7f4e8c1002d0 2024-04-20T21:26:37.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:37.267+0000 7f4e8be00700 1 --2- 172.21.15.62:0/1991443862 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4e8c1002d0 0x7f4e8c073050 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f4e7c00b5c0 tx=0x7f4e7c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:26:37.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:37.267+0000 7f4e88c00700 1 -- 172.21.15.62:0/1991443862 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4e7c01d070 con 0x7f4e8c1002d0 2024-04-20T21:26:37.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:37.267+0000 7f4e88c00700 1 -- 172.21.15.62:0/1991443862 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f4e7c004fa0 con 0x7f4e8c1002d0 2024-04-20T21:26:37.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:37.267+0000 7f4e91a00700 1 -- 172.21.15.62:0/1991443862 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e8c073e90 con 0x7f4e8c1002d0 2024-04-20T21:26:37.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:37.267+0000 7f4e88c00700 1 -- 172.21.15.62:0/1991443862 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4e7c00f460 con 0x7f4e8c1002d0 2024-04-20T21:26:37.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:37.267+0000 7f4e91a00700 1 -- 172.21.15.62:0/1991443862 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e8c1a21d0 con 0x7f4e8c1002d0 2024-04-20T21:26:37.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:37.268+0000 7f4e91a00700 1 -- 172.21.15.62:0/1991443862 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e8c1085e0 con 0x7f4e8c1002d0 2024-04-20T21:26:37.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:37.269+0000 7f4e88c00700 1 -- 172.21.15.62:0/1991443862 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f4e7c022810 con 0x7f4e8c1002d0 2024-04-20T21:26:37.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:37.270+0000 7f4e88c00700 1 --2- 172.21.15.62:0/1991443862 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f4e7006c680 0x7f4e7006eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:26:37.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:37.270+0000 7f4e88c00700 1 -- 172.21.15.62:0/1991443862 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f4e7c08ca00 con 0x7f4e8c1002d0 2024-04-20T21:26:37.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:37.271+0000 7f4e8b400700 1 --2- 172.21.15.62:0/1991443862 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f4e7006c680 0x7f4e7006eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:26:37.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:37.271+0000 7f4e8b400700 1 --2- 172.21.15.62:0/1991443862 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f4e7006c680 0x7f4e7006eb30 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4e78005950 tx=0x7f4e780058e0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:26:37.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:37.275+0000 7f4e88c00700 1 -- 172.21.15.62:0/1991443862 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f4e7c0574b0 con 0x7f4e8c1002d0 2024-04-20T21:26:37.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:37.501+0000 7f4e91a00700 1 -- 172.21.15.62:0/1991443862 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f4e8c062380 con 0x7f4e8c1002d0 2024-04-20T21:26:37.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:37.502+0000 7f4e88c00700 1 -- 172.21.15.62:0/1991443862 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f4e7c027020 con 0x7f4e8c1002d0 2024-04-20T21:26:37.501 INFO:teuthology.orchestra.run.smithi062.stdout:146028888084 2024-04-20T21:26:37.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:37.505+0000 7f4e91a00700 1 -- 172.21.15.62:0/1991443862 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f4e7006c680 msgr2=0x7f4e7006eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:26:37.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:37.505+0000 7f4e91a00700 1 --2- 172.21.15.62:0/1991443862 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f4e7006c680 0x7f4e7006eb30 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4e78005950 tx=0x7f4e780058e0 comp rx=0 tx=0).stop 2024-04-20T21:26:37.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:37.505+0000 7f4e91a00700 1 -- 172.21.15.62:0/1991443862 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4e8c1002d0 msgr2=0x7f4e8c073050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:26:37.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:37.505+0000 7f4e91a00700 1 --2- 172.21.15.62:0/1991443862 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4e8c1002d0 0x7f4e8c073050 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f4e7c00b5c0 tx=0x7f4e7c00be20 comp rx=0 tx=0).stop 2024-04-20T21:26:37.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:37.505+0000 7f4e91a00700 1 -- 172.21.15.62:0/1991443862 shutdown_connections 2024-04-20T21:26:37.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:37.505+0000 7f4e91a00700 1 --2- 172.21.15.62:0/1991443862 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f4e7006c680 0x7f4e7006eb30 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:37.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:37.505+0000 7f4e91a00700 1 --2- 172.21.15.62:0/1991443862 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4e8c1002d0 0x7f4e8c073050 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:37.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:37.505+0000 7f4e91a00700 1 --2- 172.21.15.62:0/1991443862 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e8c100c70 0x7f4e8c073590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:37.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:37.505+0000 7f4e91a00700 1 -- 172.21.15.62:0/1991443862 >> 172.21.15.62:0/1991443862 conn(0x7f4e8c0fbbb0 msgr2=0x7f4e8c0fdfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:26:37.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:37.506+0000 7f4e91a00700 1 -- 172.21.15.62:0/1991443862 shutdown_connections 2024-04-20T21:26:37.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:37.506+0000 7f4e91a00700 1 -- 172.21.15.62:0/1991443862 wait complete. 2024-04-20T21:26:37.670 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149720 got 111669149724 for osd.3 2024-04-20T21:26:37.670 DEBUG:teuthology.parallel:result is None 2024-04-20T21:26:38.235 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:26:37 smithi062 ceph-mon[16221]: from='client.? 172.21.15.62:0/1991443862' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-20T21:26:38.235 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:26:37 smithi062 ceph-mon[16221]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:26:38.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:26:37 smithi189 ceph-mon[21562]: from='client.? 172.21.15.62:0/1991443862' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-20T21:26:38.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:26:37 smithi189 ceph-mon[21562]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:26:38.559 INFO:tasks.cephadm.ceph_manager.ceph:need seq 60129542182 got 60129542184 for osd.1 2024-04-20T21:26:38.559 DEBUG:teuthology.parallel:result is None 2024-04-20T21:26:39.179 INFO:tasks.cephadm.ceph_manager.ceph:need seq 146028888082 got 146028888084 for osd.4 2024-04-20T21:26:39.179 DEBUG:teuthology.parallel:result is None 2024-04-20T21:26:40.225 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:26:39 smithi062 ceph-mon[16221]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:26:40.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:26:39 smithi189 ceph-mon[21562]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:26:40.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:40.765+0000 7f01ab800700 1 -- 172.21.15.62:0/2418400996 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f01a410c8f0 msgr2=0x7f01a410ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:26:40.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:40.765+0000 7f01ab800700 1 --2- 172.21.15.62:0/2418400996 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f01a410c8f0 0x7f01a410ccc0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f0198009910 tx=0x7f0198009c20 comp rx=0 tx=0).stop 2024-04-20T21:26:40.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:40.766+0000 7f01ab800700 1 -- 172.21.15.62:0/2418400996 shutdown_connections 2024-04-20T21:26:40.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:40.766+0000 7f01ab800700 1 --2- 172.21.15.62:0/2418400996 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f01a4106b80 0x7f01a4106ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:40.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:40.766+0000 7f01ab800700 1 --2- 172.21.15.62:0/2418400996 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f01a410c8f0 0x7f01a410ccc0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:40.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:40.766+0000 7f01ab800700 1 -- 172.21.15.62:0/2418400996 >> 172.21.15.62:0/2418400996 conn(0x7f01a4076060 msgr2=0x7f01a4078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:26:40.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:40.766+0000 7f01ab800700 1 -- 172.21.15.62:0/2418400996 shutdown_connections 2024-04-20T21:26:40.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:40.766+0000 7f01ab800700 1 -- 172.21.15.62:0/2418400996 wait complete. 2024-04-20T21:26:40.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:40.767+0000 7f01ab800700 1 Processor -- start 2024-04-20T21:26:40.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:40.767+0000 7f01ab800700 1 -- start start 2024-04-20T21:26:40.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:40.767+0000 7f01ab800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f01a4106b80 0x7f01a419c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:26:40.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:40.768+0000 7f01ab800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f01a410c8f0 0x7f01a419ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:26:40.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:40.768+0000 7f01ab800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01a419d090 con 0x7f01a4106b80 2024-04-20T21:26:40.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:40.768+0000 7f01ab800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01a419d1d0 con 0x7f01a410c8f0 2024-04-20T21:26:40.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:40.768+0000 7f01a8e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f01a4106b80 0x7f01a419c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:26:40.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:40.768+0000 7f01a8e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f01a4106b80 0x7f01a419c500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40374/0 (socket says 172.21.15.62:40374) 2024-04-20T21:26:40.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:40.768+0000 7f01a8e00700 1 -- 172.21.15.62:0/3724867699 learned_addr learned my addr 172.21.15.62:0/3724867699 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:26:40.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:40.768+0000 7f01a3e00700 1 --2- 172.21.15.62:0/3724867699 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f01a410c8f0 0x7f01a419ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:26:40.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:40.768+0000 7f01a8e00700 1 -- 172.21.15.62:0/3724867699 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f01a410c8f0 msgr2=0x7f01a419ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:26:40.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:40.768+0000 7f01a8e00700 1 --2- 172.21.15.62:0/3724867699 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f01a410c8f0 0x7f01a419ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:40.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:40.768+0000 7f01a8e00700 1 -- 172.21.15.62:0/3724867699 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01980095f0 con 0x7f01a4106b80 2024-04-20T21:26:40.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:40.769+0000 7f01a8e00700 1 --2- 172.21.15.62:0/3724867699 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f01a4106b80 0x7f01a419c500 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f0198006010 tx=0x7f019800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:26:40.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:40.769+0000 7f01a1600700 1 -- 172.21.15.62:0/3724867699 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f019801d070 con 0x7f01a4106b80 2024-04-20T21:26:40.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:40.769+0000 7f01a1600700 1 -- 172.21.15.62:0/3724867699 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f019800f460 con 0x7f01a4106b80 2024-04-20T21:26:40.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:40.769+0000 7f01a1600700 1 -- 172.21.15.62:0/3724867699 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0198017630 con 0x7f01a4106b80 2024-04-20T21:26:40.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:40.769+0000 7f01ab800700 1 -- 172.21.15.62:0/3724867699 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01a41a0fc0 con 0x7f01a4106b80 2024-04-20T21:26:40.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:40.769+0000 7f01ab800700 1 -- 172.21.15.62:0/3724867699 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01a41a14b0 con 0x7f01a4106b80 2024-04-20T21:26:40.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:40.770+0000 7f0196a00700 1 -- 172.21.15.62:0/3724867699 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01800052f0 con 0x7f01a4106b80 2024-04-20T21:26:40.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:40.771+0000 7f01a1600700 1 -- 172.21.15.62:0/3724867699 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f019800f5d0 con 0x7f01a4106b80 2024-04-20T21:26:40.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:40.773+0000 7f01a1600700 1 --2- 172.21.15.62:0/3724867699 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f018c06c680 0x7f018c06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:26:40.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:40.773+0000 7f01a1600700 1 -- 172.21.15.62:0/3724867699 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f019808cb30 con 0x7f01a4106b80 2024-04-20T21:26:40.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:40.773+0000 7f01a3e00700 1 --2- 172.21.15.62:0/3724867699 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f018c06c680 0x7f018c06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:26:40.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:40.774+0000 7f01a3e00700 1 --2- 172.21.15.62:0/3724867699 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f018c06c680 0x7f018c06eb30 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f0190005ea0 tx=0x7f0190005e30 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:26:40.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:40.777+0000 7f01a1600700 1 -- 172.21.15.62:0/3724867699 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f0198057530 con 0x7f01a4106b80 2024-04-20T21:26:40.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:40.999+0000 7f0196a00700 1 -- 172.21.15.62:0/3724867699 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f0180005160 con 0x7f01a4106b80 2024-04-20T21:26:41.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:41.000+0000 7f01a1600700 1 -- 172.21.15.62:0/3724867699 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f019805ab50 con 0x7f01a4106b80 2024-04-20T21:26:41.000 INFO:teuthology.orchestra.run.smithi062.stdout:38654705713 2024-04-20T21:26:41.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:41.003+0000 7f0196a00700 1 -- 172.21.15.62:0/3724867699 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f018c06c680 msgr2=0x7f018c06eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:26:41.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:41.003+0000 7f0196a00700 1 --2- 172.21.15.62:0/3724867699 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f018c06c680 0x7f018c06eb30 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f0190005ea0 tx=0x7f0190005e30 comp rx=0 tx=0).stop 2024-04-20T21:26:41.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:41.003+0000 7f0196a00700 1 -- 172.21.15.62:0/3724867699 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f01a4106b80 msgr2=0x7f01a419c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:26:41.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:41.003+0000 7f0196a00700 1 --2- 172.21.15.62:0/3724867699 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f01a4106b80 0x7f01a419c500 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f0198006010 tx=0x7f019800be20 comp rx=0 tx=0).stop 2024-04-20T21:26:41.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:41.003+0000 7f0196a00700 1 -- 172.21.15.62:0/3724867699 shutdown_connections 2024-04-20T21:26:41.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:41.003+0000 7f0196a00700 1 --2- 172.21.15.62:0/3724867699 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f018c06c680 0x7f018c06eb30 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:41.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:41.003+0000 7f0196a00700 1 --2- 172.21.15.62:0/3724867699 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f01a4106b80 0x7f01a419c500 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:41.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:41.003+0000 7f0196a00700 1 --2- 172.21.15.62:0/3724867699 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f01a410c8f0 0x7f01a419ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:41.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:41.003+0000 7f0196a00700 1 -- 172.21.15.62:0/3724867699 >> 172.21.15.62:0/3724867699 conn(0x7f01a4076060 msgr2=0x7f01a4077aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:26:41.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:41.004+0000 7f0196a00700 1 -- 172.21.15.62:0/3724867699 shutdown_connections 2024-04-20T21:26:41.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:41.004+0000 7f0196a00700 1 -- 172.21.15.62:0/3724867699 wait complete. 2024-04-20T21:26:41.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:26:41 smithi062 ceph-mon[16221]: from='client.? 172.21.15.62:0/3724867699' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-20T21:26:41.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:26:41 smithi189 ceph-mon[21562]: from='client.? 172.21.15.62:0/3724867699' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-20T21:26:41.843 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705710 got 38654705713 for osd.0 2024-04-20T21:26:41.844 DEBUG:teuthology.parallel:result is None 2024-04-20T21:26:41.844 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-20T21:26:41.844 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-20T21:26:42.322 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T21:26:43.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:26:42 smithi062 ceph-mon[16221]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:26:43.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:26:42 smithi189 ceph-mon[21562]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:26:44.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:26:43 smithi062 ceph-mon[16221]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:26:44.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:26:43 smithi189 ceph-mon[21562]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:26:44.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:44.599+0000 7ff245800700 1 -- 172.21.15.62:0/3317753870 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff240101c90 msgr2=0x7ff24010a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:26:44.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:44.599+0000 7ff245800700 1 --2- 172.21.15.62:0/3317753870 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff240101c90 0x7ff24010a2d0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7ff22c009910 tx=0x7ff22c009c20 comp rx=0 tx=0).stop 2024-04-20T21:26:44.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:44.600+0000 7ff245800700 1 -- 172.21.15.62:0/3317753870 shutdown_connections 2024-04-20T21:26:44.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:44.600+0000 7ff245800700 1 --2- 172.21.15.62:0/3317753870 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff240101c90 0x7ff24010a2d0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:44.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:44.600+0000 7ff245800700 1 --2- 172.21.15.62:0/3317753870 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2401012f0 0x7ff2401016c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:44.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:44.600+0000 7ff245800700 1 -- 172.21.15.62:0/3317753870 >> 172.21.15.62:0/3317753870 conn(0x7ff2400fac80 msgr2=0x7ff2400fd090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:26:44.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:44.600+0000 7ff245800700 1 -- 172.21.15.62:0/3317753870 shutdown_connections 2024-04-20T21:26:44.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:44.600+0000 7ff245800700 1 -- 172.21.15.62:0/3317753870 wait complete. 2024-04-20T21:26:44.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:44.601+0000 7ff245800700 1 Processor -- start 2024-04-20T21:26:44.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:44.601+0000 7ff245800700 1 -- start start 2024-04-20T21:26:44.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:44.602+0000 7ff245800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2401012f0 0x7ff2401980e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:26:44.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:44.602+0000 7ff245800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff240101c90 0x7ff240198620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:26:44.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:44.602+0000 7ff245800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff240198d00 con 0x7ff240101c90 2024-04-20T21:26:44.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:44.602+0000 7ff245800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff24019ca90 con 0x7ff2401012f0 2024-04-20T21:26:44.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:44.602+0000 7ff23e000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff240101c90 0x7ff240198620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:26:44.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:44.602+0000 7ff23e000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff240101c90 0x7ff240198620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37236/0 (socket says 172.21.15.62:37236) 2024-04-20T21:26:44.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:44.602+0000 7ff23e000700 1 -- 172.21.15.62:0/3373886979 learned_addr learned my addr 172.21.15.62:0/3373886979 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:26:44.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:44.602+0000 7ff23ea00700 1 --2- 172.21.15.62:0/3373886979 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2401012f0 0x7ff2401980e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:26:44.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:44.603+0000 7ff23e000700 1 -- 172.21.15.62:0/3373886979 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2401012f0 msgr2=0x7ff2401980e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:26:44.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:44.603+0000 7ff23e000700 1 --2- 172.21.15.62:0/3373886979 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2401012f0 0x7ff2401980e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:44.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:44.603+0000 7ff23e000700 1 -- 172.21.15.62:0/3373886979 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff22c0095f0 con 0x7ff240101c90 2024-04-20T21:26:44.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:44.603+0000 7ff23e000700 1 --2- 172.21.15.62:0/3373886979 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff240101c90 0x7ff240198620 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7ff22c005b40 tx=0x7ff22c0056a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:26:44.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:44.603+0000 7ff237400700 1 -- 172.21.15.62:0/3373886979 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff22c01d070 con 0x7ff240101c90 2024-04-20T21:26:44.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:44.603+0000 7ff237400700 1 -- 172.21.15.62:0/3373886979 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff22c00f460 con 0x7ff240101c90 2024-04-20T21:26:44.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:44.603+0000 7ff237400700 1 -- 172.21.15.62:0/3373886979 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff22c021640 con 0x7ff240101c90 2024-04-20T21:26:44.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:44.603+0000 7ff245800700 1 -- 172.21.15.62:0/3373886979 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff24019cd10 con 0x7ff240101c90 2024-04-20T21:26:44.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:44.604+0000 7ff245800700 1 -- 172.21.15.62:0/3373886979 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff24019d260 con 0x7ff240101c90 2024-04-20T21:26:44.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:44.605+0000 7ff245800700 1 -- 172.21.15.62:0/3373886979 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff240109e30 con 0x7ff240101c90 2024-04-20T21:26:44.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:44.606+0000 7ff237400700 1 -- 172.21.15.62:0/3373886979 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7ff22c00f5d0 con 0x7ff240101c90 2024-04-20T21:26:44.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:44.606+0000 7ff237400700 1 --2- 172.21.15.62:0/3373886979 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7ff22406c680 0x7ff22406eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:26:44.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:44.607+0000 7ff237400700 1 -- 172.21.15.62:0/3373886979 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7ff22c08c8e0 con 0x7ff240101c90 2024-04-20T21:26:44.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:44.607+0000 7ff23ea00700 1 --2- 172.21.15.62:0/3373886979 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7ff22406c680 0x7ff22406eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:26:44.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:44.607+0000 7ff23ea00700 1 --2- 172.21.15.62:0/3373886979 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7ff22406c680 0x7ff22406eb30 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff230005950 tx=0x7ff230005cf0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:26:44.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:44.612+0000 7ff237400700 1 -- 172.21.15.62:0/3373886979 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7ff22c0572e0 con 0x7ff240101c90 2024-04-20T21:26:44.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:44.827+0000 7ff245800700 1 -- 172.21.15.62:0/3373886979 --> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff24019d4d0 con 0x7ff22406c680 2024-04-20T21:26:44.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:44.831+0000 7ff237400700 1 -- 172.21.15.62:0/3373886979 <== mgr.14215 v2:172.21.15.62:6800/4116451241 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+27106 (secure 0 0 0) 0x7ff24019d4d0 con 0x7ff22406c680 2024-04-20T21:26:44.831 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:26:44.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:44.834+0000 7ff245800700 1 -- 172.21.15.62:0/3373886979 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7ff22406c680 msgr2=0x7ff22406eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:26:44.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:44.834+0000 7ff245800700 1 --2- 172.21.15.62:0/3373886979 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7ff22406c680 0x7ff22406eb30 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff230005950 tx=0x7ff230005cf0 comp rx=0 tx=0).stop 2024-04-20T21:26:44.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:44.834+0000 7ff245800700 1 -- 172.21.15.62:0/3373886979 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff240101c90 msgr2=0x7ff240198620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:26:44.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:44.834+0000 7ff245800700 1 --2- 172.21.15.62:0/3373886979 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff240101c90 0x7ff240198620 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7ff22c005b40 tx=0x7ff22c0056a0 comp rx=0 tx=0).stop 2024-04-20T21:26:44.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:44.835+0000 7ff245800700 1 -- 172.21.15.62:0/3373886979 shutdown_connections 2024-04-20T21:26:44.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:44.835+0000 7ff245800700 1 --2- 172.21.15.62:0/3373886979 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7ff22406c680 0x7ff22406eb30 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:44.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:44.835+0000 7ff245800700 1 --2- 172.21.15.62:0/3373886979 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2401012f0 0x7ff2401980e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:44.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:44.835+0000 7ff245800700 1 --2- 172.21.15.62:0/3373886979 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff240101c90 0x7ff240198620 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:44.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:44.835+0000 7ff245800700 1 -- 172.21.15.62:0/3373886979 >> 172.21.15.62:0/3373886979 conn(0x7ff2400fac80 msgr2=0x7ff2400ff8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:26:44.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:44.835+0000 7ff245800700 1 -- 172.21.15.62:0/3373886979 shutdown_connections 2024-04-20T21:26:44.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:44.835+0000 7ff245800700 1 -- 172.21.15.62:0/3373886979 wait complete. 2024-04-20T21:26:44.836 INFO:teuthology.orchestra.run.smithi062.stderr:dumped all 2024-04-20T21:26:45.228 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:26:44 smithi062 ceph-mon[16221]: from='client.14418 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T21:26:45.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:26:44 smithi189 ceph-mon[21562]: from='client.14418 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T21:26:45.615 INFO:teuthology.orchestra.run.smithi062.stdout:{"pg_ready":true,"pg_map":{"version":198,"stamp":"2024-04-20T21:26:43.876441+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":165588,"kb_used_data":3876,"kb_used_omap":0,"kb_used_meta":161664,"kb_avail":562305324,"statfs":{"total":575970213888,"available":575800651776,"internally_reserved":0,"allocated":3969024,"data_stored":2609166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":165543936},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003586"},"pg_stats":[{"pgid":"1.0","version":"22'117","reported_seq":187,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-20T21:25:43.897902+0000","last_change":"2024-04-20T21:25:07.409019+0000","last_active":"2024-04-20T21:25:43.897902+0000","last_peered":"2024-04-20T21:25:43.897902+0000","last_clean":"2024-04-20T21:25:43.897902+0000","last_became_active":"2024-04-20T21:25:07.408597+0000","last_became_peered":"2024-04-20T21:25:07.408597+0000","last_unstale":"2024-04-20T21:25:43.897902+0000","last_undegraded":"2024-04-20T21:25:43.897902+0000","last_fullsized":"2024-04-20T21:25:43.897902+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T21:24:02.914396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T21:24:02.914396+0000","last_clean_scrub_stamp":"2024-04-20T21:24:02.914396+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:09:39.663012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":39,"seq":167503724558,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27308,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717844,"statfs":{"total":95995035648,"available":95967072256,"internally_reserved":0,"allocated":364544,"data_stored":139677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79100000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2150000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4430000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.29}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.026}]}]},{"osd":4,"up_from":34,"seq":146028888086,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27308,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717844,"statfs":{"total":95995035648,"available":95967072256,"internally_reserved":0,"allocated":364544,"data_stored":139677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 21:26:08 2024","interfaces":[{"interface":"back","average":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"min":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":1.5840000000000001,"5min":1.5840000000000001,"15min":1.5840000000000001},"last":1.339},{"interface":"front","average":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"min":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"max":{"1min":1.514,"5min":1.514,"15min":1.514},"last":1.272}]},{"osd":1,"last update":"Sat Apr 20 21:26:08 2024","interfaces":[{"interface":"back","average":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"min":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":1.369,"5min":1.369,"15min":1.369},"last":1.4099999999999999},{"interface":"front","average":{"1min":1.1160000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"min":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"max":{"1min":1.821,"5min":1.821,"15min":1.821},"last":1.1519999999999999}]},{"osd":2,"last update":"Sat Apr 20 21:26:08 2024","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":1.127,"15min":1.127},"min":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":1.7490000000000001,"5min":1.7490000000000001,"15min":1.7490000000000001},"last":1.2430000000000001},{"interface":"front","average":{"1min":1.111,"5min":1.111,"15min":1.111},"min":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"max":{"1min":1.8560000000000001,"5min":1.8560000000000001,"15min":1.8560000000000001},"last":1.1859999999999999}]},{"osd":3,"last update":"Sat Apr 20 21:26:08 2024","interfaces":[{"interface":"back","average":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.653,"5min":1.653,"15min":1.653},"last":0.91300000000000003},{"interface":"front","average":{"1min":1.089,"5min":1.089,"15min":1.089},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.798,"5min":1.798,"15min":1.798},"last":1.087}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.298}]}]},{"osd":3,"up_from":26,"seq":111669149725,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27888,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717264,"statfs":{"total":95995035648,"available":95966478336,"internally_reserved":0,"allocated":958464,"data_stored":730045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 21:26:40 2024","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":0.95499999999999996,"15min":0.95699999999999996},"min":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":1.272,"5min":1.395,"15min":1.395},"last":1.272},{"interface":"front","average":{"1min":1.016,"5min":1.0309999999999999,"15min":1.0329999999999999},"min":{"1min":0.81299999999999994,"5min":0.80500000000000005,"15min":0.80500000000000005},"max":{"1min":1.232,"5min":1.288,"15min":1.288},"last":1.034}]},{"osd":1,"last update":"Sat Apr 20 21:26:40 2024","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":1.022,"15min":1.0209999999999999},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.2849999999999999,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":1.2250000000000001},{"interface":"front","average":{"1min":0.98899999999999999,"5min":1.0489999999999999,"15min":1.0589999999999999},"min":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.3080000000000001,"5min":1.335,"15min":1.335},"last":1}]},{"osd":2,"last update":"Sat Apr 20 21:26:40 2024","interfaces":[{"interface":"back","average":{"1min":1.048,"5min":1.097,"15min":1.105},"min":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":1.419,"5min":1.419,"15min":1.419},"last":1.1539999999999999},{"interface":"front","average":{"1min":1.1639999999999999,"5min":1.1140000000000001,"15min":1.1060000000000001},"min":{"1min":0.85499999999999998,"5min":0.84599999999999997,"15min":0.84599999999999997},"max":{"1min":1.4159999999999999,"5min":1.4159999999999999,"15min":1.4159999999999999},"last":1.4159999999999999}]},{"osd":4,"last update":"Sat Apr 20 21:26:09 2024","interfaces":[{"interface":"back","average":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":1.4339999999999999,"5min":1.4339999999999999,"15min":1.4339999999999999},"last":1.0920000000000001},{"interface":"front","average":{"1min":1.1890000000000001,"5min":1.1890000000000001,"15min":1.1890000000000001},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.528,"5min":1.528,"15min":1.528},"last":1.48}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3340000000000001}]}]},{"osd":2,"up_from":19,"seq":81604378660,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27308,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717844,"statfs":{"total":95995035648,"available":95967072256,"internally_reserved":0,"allocated":364544,"data_stored":139677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 21:26:05 2024","interfaces":[{"interface":"back","average":{"1min":0.97299999999999998,"5min":0.83399999999999996,"15min":0.81100000000000005},"min":{"1min":0.67000000000000004,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.6299999999999999,"5min":1.6299999999999999,"15min":1.6299999999999999},"last":0.83699999999999997},{"interface":"front","average":{"1min":0.96399999999999997,"5min":0.86599999999999999,"15min":0.84999999999999998},"min":{"1min":0.76600000000000001,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.3,"5min":1.3,"15min":1.3},"last":0.96399999999999997}]},{"osd":1,"last update":"Sat Apr 20 21:26:05 2024","interfaces":[{"interface":"back","average":{"1min":0.996,"5min":0.91000000000000003,"15min":0.89600000000000002},"min":{"1min":0.76500000000000001,"5min":0.432,"15min":0.432},"max":{"1min":1.2549999999999999,"5min":1.3600000000000001,"15min":1.3600000000000001},"last":1.121},{"interface":"front","average":{"1min":1.119,"5min":0.96199999999999997,"15min":0.93600000000000005},"min":{"1min":0.77600000000000002,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.379,"5min":1.379,"15min":1.379},"last":1.3100000000000001}]},{"osd":3,"last update":"Sat Apr 20 21:26:39 2024","interfaces":[{"interface":"back","average":{"1min":1.157,"5min":1.133,"15min":1.129},"min":{"1min":0.78700000000000003,"5min":0.745,"15min":0.745},"max":{"1min":1.556,"5min":1.556,"15min":1.556},"last":1.2170000000000001},{"interface":"front","average":{"1min":1.1819999999999999,"5min":1.1679999999999999,"15min":1.1659999999999999},"min":{"1min":0.78000000000000003,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.6879999999999999,"5min":1.6879999999999999,"15min":1.6879999999999999},"last":1.1830000000000001}]},{"osd":4,"last update":"Sat Apr 20 21:26:09 2024","interfaces":[{"interface":"back","average":{"1min":1.2569999999999999,"5min":1.2569999999999999,"15min":1.2569999999999999},"min":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"max":{"1min":1.5089999999999999,"5min":1.5089999999999999,"15min":1.5089999999999999},"last":1.345},{"interface":"front","average":{"1min":1.3180000000000001,"5min":1.3180000000000001,"15min":1.3180000000000001},"min":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"max":{"1min":1.8340000000000001,"5min":1.8340000000000001,"15min":1.8340000000000001},"last":1.2549999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5209999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705714,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27888,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717264,"statfs":{"total":95995035648,"available":95966478336,"internally_reserved":0,"allocated":958464,"data_stored":730045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 20 21:26:31 2024","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.80800000000000005,"15min":0.78200000000000003},"min":{"1min":0.432,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.7230000000000001,"5min":2.2789999999999999,"15min":2.2789999999999999},"last":0.84499999999999997},{"interface":"front","average":{"1min":0.95999999999999996,"5min":0.88500000000000001,"15min":0.86299999999999999},"min":{"1min":0.55600000000000005,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.357,"5min":2.4369999999999998,"15min":2.4369999999999998},"last":1.323}]},{"osd":2,"last update":"Sat Apr 20 21:26:07 2024","interfaces":[{"interface":"back","average":{"1min":1.0489999999999999,"5min":1.0720000000000001,"15min":1.0760000000000001},"min":{"1min":0.71299999999999997,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.462,"5min":1.8959999999999999,"15min":1.8959999999999999},"last":1.0620000000000001},{"interface":"front","average":{"1min":0.997,"5min":1.0529999999999999,"15min":1.0620000000000001},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.29,"5min":2.0790000000000002,"15min":2.0790000000000002},"last":1.5589999999999999}]},{"osd":3,"last update":"Sat Apr 20 21:26:41 2024","interfaces":[{"interface":"back","average":{"1min":1.1439999999999999,"5min":1.141,"15min":1.141},"min":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"max":{"1min":1.5620000000000001,"5min":1.5620000000000001,"15min":1.5620000000000001},"last":1.3580000000000001},{"interface":"front","average":{"1min":1.2709999999999999,"5min":1.2,"15min":1.1879999999999999},"min":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"max":{"1min":2.0059999999999998,"5min":2.0059999999999998,"15min":2.0059999999999998},"last":1.2669999999999999}]},{"osd":4,"last update":"Sat Apr 20 21:26:12 2024","interfaces":[{"interface":"back","average":{"1min":1.286,"5min":1.286,"15min":1.286},"min":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"max":{"1min":1.8540000000000001,"5min":1.8540000000000001,"15min":1.8540000000000001},"last":1.411},{"interface":"front","average":{"1min":1.365,"5min":1.365,"15min":1.365},"min":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"max":{"1min":1.8009999999999999,"5min":1.8009999999999999,"15min":1.8009999999999999},"last":1.617}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4890000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542186,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27888,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717264,"statfs":{"total":95995035648,"available":95966478336,"internally_reserved":0,"allocated":958464,"data_stored":730045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 21:26:26 2024","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.77500000000000002,"15min":0.749},"min":{"1min":0.46300000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.357,"5min":1.357,"15min":1.357},"last":0.91000000000000003},{"interface":"front","average":{"1min":0.94199999999999995,"5min":0.83199999999999996,"15min":0.77400000000000002},"min":{"1min":0.47299999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":1.429,"5min":1.429,"15min":1.429},"last":1.2569999999999999}]},{"osd":2,"last update":"Sat Apr 20 21:26:07 2024","interfaces":[{"interface":"back","average":{"1min":1.0720000000000001,"5min":1.0409999999999999,"15min":1.036},"min":{"1min":0.52300000000000002,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.706,"5min":1.706,"15min":1.706},"last":1.5700000000000001},{"interface":"front","average":{"1min":0.97299999999999998,"5min":1.0589999999999999,"15min":1.073},"min":{"1min":0.44700000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.4510000000000001,"5min":1.9730000000000001,"15min":1.9730000000000001},"last":1.321}]},{"osd":3,"last update":"Sat Apr 20 21:26:41 2024","interfaces":[{"interface":"back","average":{"1min":1.171,"5min":1.1819999999999999,"15min":1.1839999999999999},"min":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.8919999999999999,"5min":1.8919999999999999,"15min":1.8919999999999999},"last":1.3620000000000001},{"interface":"front","average":{"1min":1.1120000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":1.4750000000000001,"5min":1.5269999999999999,"15min":1.5269999999999999},"last":1.1879999999999999}]},{"osd":4,"last update":"Sat Apr 20 21:26:07 2024","interfaces":[{"interface":"back","average":{"1min":1.1919999999999999,"5min":1.1919999999999999,"15min":1.1919999999999999},"min":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.786,"5min":1.786,"15min":1.786},"last":1.401},{"interface":"front","average":{"1min":1.327,"5min":1.327,"15min":1.327},"min":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"max":{"1min":1.8939999999999999,"5min":1.8939999999999999,"15min":1.8939999999999999},"last":1.5249999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4670000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T21:26:45.616 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-20T21:26:46.055 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T21:26:46.233 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:26:45 smithi062 ceph-mon[16221]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:26:46.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:26:45 smithi189 ceph-mon[21562]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:26:48.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:26:47 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:26:48.085 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:26:47 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:26:48.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:48.282+0000 7fe376000700 1 -- 172.21.15.62:0/2619893332 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe3700739a0 msgr2=0x7fe37010c5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:26:48.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:48.282+0000 7fe376000700 1 --2- 172.21.15.62:0/2619893332 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe3700739a0 0x7fe37010c5e0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fe35c009940 tx=0x7fe35c009c50 comp rx=0 tx=0).stop 2024-04-20T21:26:48.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:48.282+0000 7fe376000700 1 -- 172.21.15.62:0/2619893332 shutdown_connections 2024-04-20T21:26:48.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:48.282+0000 7fe376000700 1 --2- 172.21.15.62:0/2619893332 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe3700739a0 0x7fe37010c5e0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:48.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:48.282+0000 7fe376000700 1 --2- 172.21.15.62:0/2619893332 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe370073090 0x7fe370073460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:48.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:48.282+0000 7fe376000700 1 -- 172.21.15.62:0/2619893332 >> 172.21.15.62:0/2619893332 conn(0x7fe3700fbda0 msgr2=0x7fe3700fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:26:48.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:48.282+0000 7fe376000700 1 -- 172.21.15.62:0/2619893332 shutdown_connections 2024-04-20T21:26:48.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:48.282+0000 7fe376000700 1 -- 172.21.15.62:0/2619893332 wait complete. 2024-04-20T21:26:48.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:48.283+0000 7fe376000700 1 Processor -- start 2024-04-20T21:26:48.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:48.283+0000 7fe376000700 1 -- start start 2024-04-20T21:26:48.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:48.284+0000 7fe376000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe370073090 0x7fe3701981b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:26:48.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:48.284+0000 7fe376000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3700739a0 0x7fe3701986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:26:48.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:48.284+0000 7fe376000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe370198dd0 con 0x7fe370073090 2024-04-20T21:26:48.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:48.284+0000 7fe376000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe37019cb60 con 0x7fe3700739a0 2024-04-20T21:26:48.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:48.284+0000 7fe36f400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe370073090 0x7fe3701981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:26:48.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:48.284+0000 7fe36f400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe370073090 0x7fe3701981b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37252/0 (socket says 172.21.15.62:37252) 2024-04-20T21:26:48.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:48.284+0000 7fe36f400700 1 -- 172.21.15.62:0/1425939118 learned_addr learned my addr 172.21.15.62:0/1425939118 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:26:48.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:48.284+0000 7fe36ea00700 1 --2- 172.21.15.62:0/1425939118 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3700739a0 0x7fe3701986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:26:48.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:48.285+0000 7fe36f400700 1 -- 172.21.15.62:0/1425939118 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3700739a0 msgr2=0x7fe3701986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:26:48.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:48.285+0000 7fe36f400700 1 --2- 172.21.15.62:0/1425939118 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3700739a0 0x7fe3701986f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:48.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:48.285+0000 7fe36f400700 1 -- 172.21.15.62:0/1425939118 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe360009520 con 0x7fe370073090 2024-04-20T21:26:48.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:48.285+0000 7fe36f400700 1 --2- 172.21.15.62:0/1425939118 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe370073090 0x7fe3701981b0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fe36000eab0 tx=0x7fe36000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:26:48.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:48.286+0000 7fe367e00700 1 -- 172.21.15.62:0/1425939118 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe36000cc60 con 0x7fe370073090 2024-04-20T21:26:48.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:48.286+0000 7fe367e00700 1 -- 172.21.15.62:0/1425939118 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fe3600044d0 con 0x7fe370073090 2024-04-20T21:26:48.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:48.286+0000 7fe376000700 1 -- 172.21.15.62:0/1425939118 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe35c0095f0 con 0x7fe370073090 2024-04-20T21:26:48.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:48.286+0000 7fe367e00700 1 -- 172.21.15.62:0/1425939118 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe360005250 con 0x7fe370073090 2024-04-20T21:26:48.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:48.286+0000 7fe376000700 1 -- 172.21.15.62:0/1425939118 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe37019d200 con 0x7fe370073090 2024-04-20T21:26:48.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:48.287+0000 7fe376000700 1 -- 172.21.15.62:0/1425939118 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe350005320 con 0x7fe370073090 2024-04-20T21:26:48.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:48.290+0000 7fe367e00700 1 -- 172.21.15.62:0/1425939118 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7fe360004930 con 0x7fe370073090 2024-04-20T21:26:48.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:48.291+0000 7fe367e00700 1 --2- 172.21.15.62:0/1425939118 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fe35406c680 0x7fe35406eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:26:48.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:48.291+0000 7fe367e00700 1 -- 172.21.15.62:0/1425939118 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fe360014070 con 0x7fe370073090 2024-04-20T21:26:48.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:48.291+0000 7fe36ea00700 1 --2- 172.21.15.62:0/1425939118 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fe35406c680 0x7fe35406eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:26:48.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:48.292+0000 7fe36ea00700 1 --2- 172.21.15.62:0/1425939118 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fe35406c680 0x7fe35406eb30 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fe3701997d0 tx=0x7fe35c005a90 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:26:48.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:48.296+0000 7fe367e00700 1 -- 172.21.15.62:0/1425939118 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fe3600560e0 con 0x7fe370073090 2024-04-20T21:26:48.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:48.516+0000 7fe376000700 1 -- 172.21.15.62:0/1425939118 --> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe350000bf0 con 0x7fe35406c680 2024-04-20T21:26:48.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:48.518+0000 7fe367e00700 1 -- 172.21.15.62:0/1425939118 <== mgr.14215 v2:172.21.15.62:6800/4116451241 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+29427 (secure 0 0 0) 0x7fe350000bf0 con 0x7fe35406c680 2024-04-20T21:26:48.518 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:26:48.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:48.521+0000 7fe376000700 1 -- 172.21.15.62:0/1425939118 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fe35406c680 msgr2=0x7fe35406eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:26:48.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:48.521+0000 7fe376000700 1 --2- 172.21.15.62:0/1425939118 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fe35406c680 0x7fe35406eb30 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fe3701997d0 tx=0x7fe35c005a90 comp rx=0 tx=0).stop 2024-04-20T21:26:48.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:48.521+0000 7fe376000700 1 -- 172.21.15.62:0/1425939118 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe370073090 msgr2=0x7fe3701981b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:26:48.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:48.521+0000 7fe376000700 1 --2- 172.21.15.62:0/1425939118 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe370073090 0x7fe3701981b0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fe36000eab0 tx=0x7fe36000ee70 comp rx=0 tx=0).stop 2024-04-20T21:26:48.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:48.521+0000 7fe376000700 1 -- 172.21.15.62:0/1425939118 shutdown_connections 2024-04-20T21:26:48.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:48.521+0000 7fe376000700 1 --2- 172.21.15.62:0/1425939118 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fe35406c680 0x7fe35406eb30 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:48.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:48.521+0000 7fe376000700 1 --2- 172.21.15.62:0/1425939118 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe370073090 0x7fe3701981b0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:48.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:48.521+0000 7fe376000700 1 --2- 172.21.15.62:0/1425939118 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3700739a0 0x7fe3701986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:48.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:48.521+0000 7fe376000700 1 -- 172.21.15.62:0/1425939118 >> 172.21.15.62:0/1425939118 conn(0x7fe3700fbda0 msgr2=0x7fe370106e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:26:48.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:48.521+0000 7fe376000700 1 -- 172.21.15.62:0/1425939118 shutdown_connections 2024-04-20T21:26:48.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:48.521+0000 7fe376000700 1 -- 172.21.15.62:0/1425939118 wait complete. 2024-04-20T21:26:48.522 INFO:teuthology.orchestra.run.smithi062.stderr:dumped all 2024-04-20T21:26:49.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:26:48 smithi189 ceph-mon[21562]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:26:49.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:26:48 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:26:49.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:26:48 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T21:26:49.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:26:48 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:26:49.271 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:26:48 smithi062 ceph-mon[16221]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:26:49.271 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:26:48 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:26:49.271 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:26:48 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T21:26:49.271 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:26:48 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:26:49.323 INFO:teuthology.orchestra.run.smithi062.stdout:{"pg_ready":true,"pg_map":{"version":200,"stamp":"2024-04-20T21:26:47.877364+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":165588,"kb_used_data":3876,"kb_used_omap":0,"kb_used_meta":161664,"kb_avail":562305324,"statfs":{"total":575970213888,"available":575800651776,"internally_reserved":0,"allocated":3969024,"data_stored":2609166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":165543936},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003442"},"pg_stats":[{"pgid":"1.0","version":"22'117","reported_seq":187,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-20T21:25:43.897902+0000","last_change":"2024-04-20T21:25:07.409019+0000","last_active":"2024-04-20T21:25:43.897902+0000","last_peered":"2024-04-20T21:25:43.897902+0000","last_clean":"2024-04-20T21:25:43.897902+0000","last_became_active":"2024-04-20T21:25:07.408597+0000","last_became_peered":"2024-04-20T21:25:07.408597+0000","last_unstale":"2024-04-20T21:25:43.897902+0000","last_undegraded":"2024-04-20T21:25:43.897902+0000","last_fullsized":"2024-04-20T21:25:43.897902+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T21:24:02.914396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T21:24:02.914396+0000","last_clean_scrub_stamp":"2024-04-20T21:24:02.914396+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:09:39.663012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":39,"seq":167503724559,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27308,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717844,"statfs":{"total":95995035648,"available":95967072256,"internally_reserved":0,"allocated":364544,"data_stored":139677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 21:26:43 2024","interfaces":[{"interface":"back","average":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"min":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"max":{"1min":1.5189999999999999,"5min":1.5189999999999999,"15min":1.5189999999999999},"last":0.98499999999999999},{"interface":"front","average":{"1min":1.0720000000000001,"5min":1.0720000000000001,"15min":1.0720000000000001},"min":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"max":{"1min":1.5109999999999999,"5min":1.5109999999999999,"15min":1.5109999999999999},"last":1.1619999999999999}]},{"osd":1,"last update":"Sat Apr 20 21:26:43 2024","interfaces":[{"interface":"back","average":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"min":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":1.2949999999999999,"5min":1.2949999999999999,"15min":1.2949999999999999},"last":0.81000000000000005},{"interface":"front","average":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"min":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"max":{"1min":1.4410000000000001,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.74299999999999999}]},{"osd":2,"last update":"Sat Apr 20 21:26:43 2024","interfaces":[{"interface":"back","average":{"1min":1.133,"5min":1.133,"15min":1.133},"min":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"max":{"1min":1.4850000000000001,"5min":1.4850000000000001,"15min":1.4850000000000001},"last":0.86499999999999999},{"interface":"front","average":{"1min":1.1970000000000001,"5min":1.1970000000000001,"15min":1.1970000000000001},"min":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":1.5309999999999999,"5min":1.5309999999999999,"15min":1.5309999999999999},"last":1.091}]},{"osd":3,"last update":"Sat Apr 20 21:26:43 2024","interfaces":[{"interface":"back","average":{"1min":1.1739999999999999,"5min":1.1739999999999999,"15min":1.1739999999999999},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.6399999999999999,"5min":1.6399999999999999,"15min":1.6399999999999999},"last":1.0609999999999999},{"interface":"front","average":{"1min":1.2170000000000001,"5min":1.2170000000000001,"15min":1.2170000000000001},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.6970000000000001,"5min":1.6970000000000001,"15min":1.6970000000000001},"last":1.2070000000000001}]},{"osd":4,"last update":"Sat Apr 20 21:26:43 2024","interfaces":[{"interface":"back","average":{"1min":1.2549999999999999,"5min":1.2549999999999999,"15min":1.2549999999999999},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.5880000000000001,"5min":1.5880000000000001,"15min":1.5880000000000001},"last":1.296},{"interface":"front","average":{"1min":1.3100000000000001,"5min":1.3100000000000001,"15min":1.3100000000000001},"min":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":1.7669999999999999,"5min":1.7669999999999999,"15min":1.7669999999999999},"last":1.1439999999999999}]}]},{"osd":4,"up_from":34,"seq":146028888087,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27308,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717844,"statfs":{"total":95995035648,"available":95967072256,"internally_reserved":0,"allocated":364544,"data_stored":139677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 21:26:08 2024","interfaces":[{"interface":"back","average":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"min":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":1.5840000000000001,"5min":1.5840000000000001,"15min":1.5840000000000001},"last":0.90300000000000002},{"interface":"front","average":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"min":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"max":{"1min":1.514,"5min":1.514,"15min":1.514},"last":0.84099999999999997}]},{"osd":1,"last update":"Sat Apr 20 21:26:08 2024","interfaces":[{"interface":"back","average":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"min":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":1.369,"5min":1.369,"15min":1.369},"last":1.135},{"interface":"front","average":{"1min":1.1160000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"min":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"max":{"1min":1.821,"5min":1.821,"15min":1.821},"last":1.208}]},{"osd":2,"last update":"Sat Apr 20 21:26:08 2024","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":1.127,"15min":1.127},"min":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":1.7490000000000001,"5min":1.7490000000000001,"15min":1.7490000000000001},"last":1.0960000000000001},{"interface":"front","average":{"1min":1.111,"5min":1.111,"15min":1.111},"min":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"max":{"1min":1.8560000000000001,"5min":1.8560000000000001,"15min":1.8560000000000001},"last":1.159}]},{"osd":3,"last update":"Sat Apr 20 21:26:08 2024","interfaces":[{"interface":"back","average":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.653,"5min":1.653,"15min":1.653},"last":1.046},{"interface":"front","average":{"1min":1.089,"5min":1.089,"15min":1.089},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.798,"5min":1.798,"15min":1.798},"last":0.78200000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.022}]}]},{"osd":3,"up_from":26,"seq":111669149726,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27888,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717264,"statfs":{"total":95995035648,"available":95966478336,"internally_reserved":0,"allocated":958464,"data_stored":730045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 21:26:40 2024","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":0.95499999999999996,"15min":0.95699999999999996},"min":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":1.272,"5min":1.395,"15min":1.395},"last":0.94899999999999995},{"interface":"front","average":{"1min":1.016,"5min":1.0309999999999999,"15min":1.0329999999999999},"min":{"1min":0.81299999999999994,"5min":0.80500000000000005,"15min":0.80500000000000005},"max":{"1min":1.232,"5min":1.288,"15min":1.288},"last":1.1799999999999999}]},{"osd":1,"last update":"Sat Apr 20 21:26:40 2024","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":1.022,"15min":1.0209999999999999},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.2849999999999999,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.97399999999999998},{"interface":"front","average":{"1min":0.98899999999999999,"5min":1.0489999999999999,"15min":1.0589999999999999},"min":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.3080000000000001,"5min":1.335,"15min":1.335},"last":1.266}]},{"osd":2,"last update":"Sat Apr 20 21:26:40 2024","interfaces":[{"interface":"back","average":{"1min":1.048,"5min":1.097,"15min":1.105},"min":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":1.419,"5min":1.419,"15min":1.419},"last":1.0029999999999999},{"interface":"front","average":{"1min":1.1639999999999999,"5min":1.1140000000000001,"15min":1.1060000000000001},"min":{"1min":0.85499999999999998,"5min":0.84599999999999997,"15min":0.84599999999999997},"max":{"1min":1.4159999999999999,"5min":1.4159999999999999,"15min":1.4159999999999999},"last":1.113}]},{"osd":4,"last update":"Sat Apr 20 21:26:09 2024","interfaces":[{"interface":"back","average":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":1.4339999999999999,"5min":1.4339999999999999,"15min":1.4339999999999999},"last":0.876},{"interface":"front","average":{"1min":1.1890000000000001,"5min":1.1890000000000001,"15min":1.1890000000000001},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.528,"5min":1.528,"15min":1.528},"last":1.226}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3340000000000001}]}]},{"osd":2,"up_from":19,"seq":81604378660,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27308,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717844,"statfs":{"total":95995035648,"available":95967072256,"internally_reserved":0,"allocated":364544,"data_stored":139677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 21:26:05 2024","interfaces":[{"interface":"back","average":{"1min":0.97299999999999998,"5min":0.83399999999999996,"15min":0.81100000000000005},"min":{"1min":0.67000000000000004,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.6299999999999999,"5min":1.6299999999999999,"15min":1.6299999999999999},"last":0.83699999999999997},{"interface":"front","average":{"1min":0.96399999999999997,"5min":0.86599999999999999,"15min":0.84999999999999998},"min":{"1min":0.76600000000000001,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.3,"5min":1.3,"15min":1.3},"last":0.96399999999999997}]},{"osd":1,"last update":"Sat Apr 20 21:26:05 2024","interfaces":[{"interface":"back","average":{"1min":0.996,"5min":0.91000000000000003,"15min":0.89600000000000002},"min":{"1min":0.76500000000000001,"5min":0.432,"15min":0.432},"max":{"1min":1.2549999999999999,"5min":1.3600000000000001,"15min":1.3600000000000001},"last":1.121},{"interface":"front","average":{"1min":1.119,"5min":0.96199999999999997,"15min":0.93600000000000005},"min":{"1min":0.77600000000000002,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.379,"5min":1.379,"15min":1.379},"last":1.3100000000000001}]},{"osd":3,"last update":"Sat Apr 20 21:26:39 2024","interfaces":[{"interface":"back","average":{"1min":1.157,"5min":1.133,"15min":1.129},"min":{"1min":0.78700000000000003,"5min":0.745,"15min":0.745},"max":{"1min":1.556,"5min":1.556,"15min":1.556},"last":1.2170000000000001},{"interface":"front","average":{"1min":1.1819999999999999,"5min":1.1679999999999999,"15min":1.1659999999999999},"min":{"1min":0.78000000000000003,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.6879999999999999,"5min":1.6879999999999999,"15min":1.6879999999999999},"last":1.1830000000000001}]},{"osd":4,"last update":"Sat Apr 20 21:26:09 2024","interfaces":[{"interface":"back","average":{"1min":1.2569999999999999,"5min":1.2569999999999999,"15min":1.2569999999999999},"min":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"max":{"1min":1.5089999999999999,"5min":1.5089999999999999,"15min":1.5089999999999999},"last":1.345},{"interface":"front","average":{"1min":1.3180000000000001,"5min":1.3180000000000001,"15min":1.3180000000000001},"min":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"max":{"1min":1.8340000000000001,"5min":1.8340000000000001,"15min":1.8340000000000001},"last":1.2549999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5209999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705714,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27888,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717264,"statfs":{"total":95995035648,"available":95966478336,"internally_reserved":0,"allocated":958464,"data_stored":730045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 20 21:26:31 2024","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.80800000000000005,"15min":0.78200000000000003},"min":{"1min":0.432,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.7230000000000001,"5min":2.2789999999999999,"15min":2.2789999999999999},"last":0.84499999999999997},{"interface":"front","average":{"1min":0.95999999999999996,"5min":0.88500000000000001,"15min":0.86299999999999999},"min":{"1min":0.55600000000000005,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.357,"5min":2.4369999999999998,"15min":2.4369999999999998},"last":1.323}]},{"osd":2,"last update":"Sat Apr 20 21:26:07 2024","interfaces":[{"interface":"back","average":{"1min":1.0489999999999999,"5min":1.0720000000000001,"15min":1.0760000000000001},"min":{"1min":0.71299999999999997,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.462,"5min":1.8959999999999999,"15min":1.8959999999999999},"last":1.0620000000000001},{"interface":"front","average":{"1min":0.997,"5min":1.0529999999999999,"15min":1.0620000000000001},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.29,"5min":2.0790000000000002,"15min":2.0790000000000002},"last":1.5589999999999999}]},{"osd":3,"last update":"Sat Apr 20 21:26:41 2024","interfaces":[{"interface":"back","average":{"1min":1.1439999999999999,"5min":1.141,"15min":1.141},"min":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"max":{"1min":1.5620000000000001,"5min":1.5620000000000001,"15min":1.5620000000000001},"last":1.3580000000000001},{"interface":"front","average":{"1min":1.2709999999999999,"5min":1.2,"15min":1.1879999999999999},"min":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"max":{"1min":2.0059999999999998,"5min":2.0059999999999998,"15min":2.0059999999999998},"last":1.2669999999999999}]},{"osd":4,"last update":"Sat Apr 20 21:26:12 2024","interfaces":[{"interface":"back","average":{"1min":1.286,"5min":1.286,"15min":1.286},"min":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"max":{"1min":1.8540000000000001,"5min":1.8540000000000001,"15min":1.8540000000000001},"last":1.411},{"interface":"front","average":{"1min":1.365,"5min":1.365,"15min":1.365},"min":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"max":{"1min":1.8009999999999999,"5min":1.8009999999999999,"15min":1.8009999999999999},"last":1.617}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4890000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542186,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27888,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717264,"statfs":{"total":95995035648,"available":95966478336,"internally_reserved":0,"allocated":958464,"data_stored":730045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 21:26:26 2024","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.77500000000000002,"15min":0.749},"min":{"1min":0.46300000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.357,"5min":1.357,"15min":1.357},"last":0.91000000000000003},{"interface":"front","average":{"1min":0.94199999999999995,"5min":0.83199999999999996,"15min":0.77400000000000002},"min":{"1min":0.47299999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":1.429,"5min":1.429,"15min":1.429},"last":1.2569999999999999}]},{"osd":2,"last update":"Sat Apr 20 21:26:07 2024","interfaces":[{"interface":"back","average":{"1min":1.0720000000000001,"5min":1.0409999999999999,"15min":1.036},"min":{"1min":0.52300000000000002,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.706,"5min":1.706,"15min":1.706},"last":1.5700000000000001},{"interface":"front","average":{"1min":0.97299999999999998,"5min":1.0589999999999999,"15min":1.073},"min":{"1min":0.44700000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.4510000000000001,"5min":1.9730000000000001,"15min":1.9730000000000001},"last":1.321}]},{"osd":3,"last update":"Sat Apr 20 21:26:41 2024","interfaces":[{"interface":"back","average":{"1min":1.171,"5min":1.1819999999999999,"15min":1.1839999999999999},"min":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.8919999999999999,"5min":1.8919999999999999,"15min":1.8919999999999999},"last":1.3620000000000001},{"interface":"front","average":{"1min":1.1120000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":1.4750000000000001,"5min":1.5269999999999999,"15min":1.5269999999999999},"last":1.1879999999999999}]},{"osd":4,"last update":"Sat Apr 20 21:26:07 2024","interfaces":[{"interface":"back","average":{"1min":1.1919999999999999,"5min":1.1919999999999999,"15min":1.1919999999999999},"min":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.786,"5min":1.786,"15min":1.786},"last":1.401},{"interface":"front","average":{"1min":1.327,"5min":1.327,"15min":1.327},"min":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"max":{"1min":1.8939999999999999,"5min":1.8939999999999999,"15min":1.8939999999999999},"last":1.5249999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4670000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T21:26:49.324 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-20T21:26:49.324 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-20T21:26:49.324 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-20T21:26:49.325 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -- ceph health --format=json 2024-04-20T21:26:49.751 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T21:26:50.117 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:26:49 smithi062 ceph-mon[16221]: from='client.14422 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T21:26:50.117 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:26:49 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:26:50.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:26:49 smithi189 ceph-mon[21562]: from='client.14422 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T21:26:50.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:26:49 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:26:51.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:26:50 smithi062 ceph-mon[16221]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:26:51.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:26:50 smithi189 ceph-mon[21562]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:26:51.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:51.993+0000 7fb947e00700 1 -- 172.21.15.62:0/3443255096 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb9481061b0 msgr2=0x7fb948106580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:26:51.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:51.993+0000 7fb947e00700 1 --2- 172.21.15.62:0/3443255096 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb9481061b0 0x7fb948106580 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fb938009910 tx=0x7fb938009c20 comp rx=0 tx=0).stop 2024-04-20T21:26:51.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:51.993+0000 7fb947e00700 1 -- 172.21.15.62:0/3443255096 shutdown_connections 2024-04-20T21:26:51.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:51.993+0000 7fb947e00700 1 --2- 172.21.15.62:0/3443255096 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb948068540 0x7fb9480689b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:51.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:51.993+0000 7fb947e00700 1 --2- 172.21.15.62:0/3443255096 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb9481061b0 0x7fb948106580 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:51.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:51.993+0000 7fb947e00700 1 -- 172.21.15.62:0/3443255096 >> 172.21.15.62:0/3443255096 conn(0x7fb948074fa0 msgr2=0x7fb9480753a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:26:51.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:51.993+0000 7fb947e00700 1 -- 172.21.15.62:0/3443255096 shutdown_connections 2024-04-20T21:26:51.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:51.993+0000 7fb947e00700 1 -- 172.21.15.62:0/3443255096 wait complete. 2024-04-20T21:26:51.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:51.994+0000 7fb947e00700 1 Processor -- start 2024-04-20T21:26:51.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:51.994+0000 7fb947e00700 1 -- start start 2024-04-20T21:26:51.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:51.995+0000 7fb947e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb948068540 0x7fb94819b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:26:51.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:51.995+0000 7fb947e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb9481061b0 0x7fb94819bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:26:51.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:51.995+0000 7fb947e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb94819c4a0 con 0x7fb9481061b0 2024-04-20T21:26:51.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:51.995+0000 7fb947e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9481959e0 con 0x7fb948068540 2024-04-20T21:26:51.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:51.995+0000 7fb93fe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb9481061b0 0x7fb94819bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:26:51.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:51.995+0000 7fb93fe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb9481061b0 0x7fb94819bea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33900/0 (socket says 172.21.15.62:33900) 2024-04-20T21:26:51.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:51.995+0000 7fb93fe00700 1 -- 172.21.15.62:0/3160930284 learned_addr learned my addr 172.21.15.62:0/3160930284 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:26:51.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:51.996+0000 7fb946a00700 1 --2- 172.21.15.62:0/3160930284 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb948068540 0x7fb94819b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:26:51.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:51.996+0000 7fb93fe00700 1 -- 172.21.15.62:0/3160930284 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb948068540 msgr2=0x7fb94819b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:26:51.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:51.996+0000 7fb93fe00700 1 --2- 172.21.15.62:0/3160930284 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb948068540 0x7fb94819b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:51.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:51.996+0000 7fb93fe00700 1 -- 172.21.15.62:0/3160930284 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9380095f0 con 0x7fb9481061b0 2024-04-20T21:26:51.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:51.996+0000 7fb93fe00700 1 --2- 172.21.15.62:0/3160930284 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb9481061b0 0x7fb94819bea0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fb93000c930 tx=0x7fb93000ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:26:51.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:51.996+0000 7fb93f400700 1 -- 172.21.15.62:0/3160930284 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb930007ad0 con 0x7fb9481061b0 2024-04-20T21:26:51.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:51.996+0000 7fb93f400700 1 -- 172.21.15.62:0/3160930284 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fb930007c30 con 0x7fb9481061b0 2024-04-20T21:26:51.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:51.996+0000 7fb947e00700 1 -- 172.21.15.62:0/3160930284 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb948195cc0 con 0x7fb9481061b0 2024-04-20T21:26:51.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:51.996+0000 7fb93f400700 1 -- 172.21.15.62:0/3160930284 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb930018630 con 0x7fb9481061b0 2024-04-20T21:26:51.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:51.997+0000 7fb947e00700 1 -- 172.21.15.62:0/3160930284 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb948196210 con 0x7fb9481061b0 2024-04-20T21:26:51.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:51.998+0000 7fb947e00700 1 -- 172.21.15.62:0/3160930284 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb948062380 con 0x7fb9481061b0 2024-04-20T21:26:51.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:51.999+0000 7fb93f400700 1 -- 172.21.15.62:0/3160930284 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7fb93000f450 con 0x7fb9481061b0 2024-04-20T21:26:51.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:52.000+0000 7fb93f400700 1 --2- 172.21.15.62:0/3160930284 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fb92c06c680 0x7fb92c06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:26:51.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:52.000+0000 7fb93f400700 1 -- 172.21.15.62:0/3160930284 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fb93008b710 con 0x7fb9481061b0 2024-04-20T21:26:51.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:52.000+0000 7fb946a00700 1 --2- 172.21.15.62:0/3160930284 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fb92c06c680 0x7fb92c06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:26:52.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:52.001+0000 7fb946a00700 1 --2- 172.21.15.62:0/3160930284 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fb92c06c680 0x7fb92c06eb30 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb938006010 tx=0x7fb93801a040 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:26:52.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:52.005+0000 7fb93f400700 1 -- 172.21.15.62:0/3160930284 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fb930059ba0 con 0x7fb9481061b0 2024-04-20T21:26:52.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:52.282+0000 7fb947e00700 1 -- 172.21.15.62:0/3160930284 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fb948196490 con 0x7fb9481061b0 2024-04-20T21:26:52.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:52.283+0000 7fb93f400700 1 -- 172.21.15.62:0/3160930284 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7fb930059730 con 0x7fb9481061b0 2024-04-20T21:26:52.283 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:26:52.283 INFO:teuthology.orchestra.run.smithi062.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-20T21:26:52.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:52.286+0000 7fb947e00700 1 -- 172.21.15.62:0/3160930284 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fb92c06c680 msgr2=0x7fb92c06eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:26:52.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:52.286+0000 7fb947e00700 1 --2- 172.21.15.62:0/3160930284 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fb92c06c680 0x7fb92c06eb30 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb938006010 tx=0x7fb93801a040 comp rx=0 tx=0).stop 2024-04-20T21:26:52.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:52.286+0000 7fb947e00700 1 -- 172.21.15.62:0/3160930284 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb9481061b0 msgr2=0x7fb94819bea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:26:52.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:52.286+0000 7fb947e00700 1 --2- 172.21.15.62:0/3160930284 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb9481061b0 0x7fb94819bea0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fb93000c930 tx=0x7fb93000ccf0 comp rx=0 tx=0).stop 2024-04-20T21:26:52.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:52.287+0000 7fb947e00700 1 -- 172.21.15.62:0/3160930284 shutdown_connections 2024-04-20T21:26:52.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:52.287+0000 7fb947e00700 1 --2- 172.21.15.62:0/3160930284 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fb92c06c680 0x7fb92c06eb30 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:52.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:52.287+0000 7fb947e00700 1 --2- 172.21.15.62:0/3160930284 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb948068540 0x7fb94819b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:52.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:52.287+0000 7fb947e00700 1 --2- 172.21.15.62:0/3160930284 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb9481061b0 0x7fb94819bea0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:52.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:52.287+0000 7fb947e00700 1 -- 172.21.15.62:0/3160930284 >> 172.21.15.62:0/3160930284 conn(0x7fb948074fa0 msgr2=0x7fb9480fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:26:52.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:52.287+0000 7fb947e00700 1 -- 172.21.15.62:0/3160930284 shutdown_connections 2024-04-20T21:26:52.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:52.287+0000 7fb947e00700 1 -- 172.21.15.62:0/3160930284 wait complete. 2024-04-20T21:26:53.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:26:52 smithi062 ceph-mon[16221]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:26:53.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:26:52 smithi062 ceph-mon[16221]: from='client.? 172.21.15.62:0/3160930284' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-20T21:26:53.117 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-20T21:26:53.117 INFO:tasks.cephadm:Setup complete, yielding 2024-04-20T21:26:53.117 INFO:teuthology.run_tasks:Running task print... 2024-04-20T21:26:53.126 INFO:teuthology.task.print:**** done end installing v18.2.0 cephadm ... 2024-04-20T21:26:53.126 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-20T21:26:53.135 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi062.front.sepia.ceph.com 2024-04-20T21:26:53.135 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -- bash -c 'ceph config set mgr mgr/cephadm/use_repo_digest true --force' 2024-04-20T21:26:53.177 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:26:52 smithi189 ceph-mon[21562]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:26:53.177 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:26:52 smithi189 ceph-mon[21562]: from='client.? 172.21.15.62:0/3160930284' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-20T21:26:53.554 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T21:26:54.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:26:53 smithi189 ceph-mon[21562]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:26:54.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:26:53 smithi062 ceph-mon[16221]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:26:55.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:55.880+0000 7f4785e00700 1 -- 172.21.15.62:0/2224720204 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4780103640 msgr2=0x7f4780103a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:26:55.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:55.880+0000 7f4785e00700 1 --2- 172.21.15.62:0/2224720204 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4780103640 0x7f4780103a90 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f4768009910 tx=0x7f4768009c20 comp rx=0 tx=0).stop 2024-04-20T21:26:55.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:55.880+0000 7f4785e00700 1 -- 172.21.15.62:0/2224720204 shutdown_connections 2024-04-20T21:26:55.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:55.880+0000 7f4785e00700 1 --2- 172.21.15.62:0/2224720204 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4780103640 0x7f4780103a90 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:55.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:55.880+0000 7f4785e00700 1 --2- 172.21.15.62:0/2224720204 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4780102440 0x7f4780102850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:55.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:55.880+0000 7f4785e00700 1 -- 172.21.15.62:0/2224720204 >> 172.21.15.62:0/2224720204 conn(0x7f47800fd9f0 msgr2=0x7f47800ffe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:26:55.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:55.880+0000 7f4785e00700 1 -- 172.21.15.62:0/2224720204 shutdown_connections 2024-04-20T21:26:55.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:55.880+0000 7f4785e00700 1 -- 172.21.15.62:0/2224720204 wait complete. 2024-04-20T21:26:55.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:55.881+0000 7f4785e00700 1 Processor -- start 2024-04-20T21:26:55.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:55.881+0000 7f4785e00700 1 -- start start 2024-04-20T21:26:55.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:55.881+0000 7f4785e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4780102440 0x7f4780197cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:26:55.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:55.881+0000 7f4785e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4780103640 0x7f47801981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:26:55.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:55.881+0000 7f4785e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47801987f0 con 0x7f4780103640 2024-04-20T21:26:55.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:55.881+0000 7f4785e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4780198960 con 0x7f4780102440 2024-04-20T21:26:55.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:55.882+0000 7f4777e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4780103640 0x7f47801981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:26:55.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:55.882+0000 7f4777e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4780103640 0x7f47801981f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33922/0 (socket says 172.21.15.62:33922) 2024-04-20T21:26:55.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:55.882+0000 7f4777e00700 1 -- 172.21.15.62:0/4254523375 learned_addr learned my addr 172.21.15.62:0/4254523375 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:26:55.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:55.882+0000 7f477ea00700 1 --2- 172.21.15.62:0/4254523375 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4780102440 0x7f4780197cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:26:55.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:55.882+0000 7f4777e00700 1 -- 172.21.15.62:0/4254523375 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4780102440 msgr2=0x7f4780197cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:26:55.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:55.882+0000 7f4777e00700 1 --2- 172.21.15.62:0/4254523375 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4780102440 0x7f4780197cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:55.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:55.882+0000 7f4777e00700 1 -- 172.21.15.62:0/4254523375 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47680095f0 con 0x7f4780103640 2024-04-20T21:26:55.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:55.883+0000 7f4777e00700 1 --2- 172.21.15.62:0/4254523375 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4780103640 0x7f47801981f0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f4768005b40 tx=0x7f476800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:26:55.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:55.883+0000 7f4784a00700 1 -- 172.21.15.62:0/4254523375 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f476801d070 con 0x7f4780103640 2024-04-20T21:26:55.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:55.883+0000 7f4784a00700 1 -- 172.21.15.62:0/4254523375 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f4768022470 con 0x7f4780103640 2024-04-20T21:26:55.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:55.883+0000 7f4784a00700 1 -- 172.21.15.62:0/4254523375 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f476800f690 con 0x7f4780103640 2024-04-20T21:26:55.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:55.883+0000 7f4785e00700 1 -- 172.21.15.62:0/4254523375 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f478019d390 con 0x7f4780103640 2024-04-20T21:26:55.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:55.883+0000 7f4785e00700 1 -- 172.21.15.62:0/4254523375 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f478019d880 con 0x7f4780103640 2024-04-20T21:26:55.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:55.884+0000 7f4785e00700 1 -- 172.21.15.62:0/4254523375 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f478010af90 con 0x7f4780103640 2024-04-20T21:26:55.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:55.885+0000 7f4784a00700 1 -- 172.21.15.62:0/4254523375 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f47680225e0 con 0x7f4780103640 2024-04-20T21:26:55.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:55.886+0000 7f4784a00700 1 --2- 172.21.15.62:0/4254523375 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f476406c680 0x7f476406eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:26:55.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:55.887+0000 7f4784a00700 1 -- 172.21.15.62:0/4254523375 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f476808cc90 con 0x7f4780103640 2024-04-20T21:26:55.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:55.887+0000 7f477ea00700 1 --2- 172.21.15.62:0/4254523375 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f476406c680 0x7f476406eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:26:55.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:55.887+0000 7f477ea00700 1 --2- 172.21.15.62:0/4254523375 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f476406c680 0x7f476406eb30 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f4770005fd0 tx=0x7f4770005dd0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:26:55.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:55.891+0000 7f4784a00700 1 -- 172.21.15.62:0/4254523375 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f4768057710 con 0x7f4780103640 2024-04-20T21:26:56.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:56.111+0000 7f4785e00700 1 -- 172.21.15.62:0/4254523375 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/use_repo_digest}] v 0) v1 -- 0x7f478010b180 con 0x7f4780103640 2024-04-20T21:26:56.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:56.123+0000 7f4784a00700 1 -- 172.21.15.62:0/4254523375 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/use_repo_digest}]=0 v20)=0 v20) v1 ==== 143+0+0 (secure 0 0 0) 0x7f476805ad30 con 0x7f4780103640 2024-04-20T21:26:56.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:56.125+0000 7f4785e00700 1 -- 172.21.15.62:0/4254523375 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f476406c680 msgr2=0x7f476406eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:26:56.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:56.125+0000 7f4785e00700 1 --2- 172.21.15.62:0/4254523375 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f476406c680 0x7f476406eb30 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f4770005fd0 tx=0x7f4770005dd0 comp rx=0 tx=0).stop 2024-04-20T21:26:56.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:56.125+0000 7f4785e00700 1 -- 172.21.15.62:0/4254523375 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4780103640 msgr2=0x7f47801981f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:26:56.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:56.125+0000 7f4785e00700 1 --2- 172.21.15.62:0/4254523375 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4780103640 0x7f47801981f0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f4768005b40 tx=0x7f476800be20 comp rx=0 tx=0).stop 2024-04-20T21:26:56.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:56.126+0000 7f4785e00700 1 -- 172.21.15.62:0/4254523375 shutdown_connections 2024-04-20T21:26:56.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:56.126+0000 7f4785e00700 1 --2- 172.21.15.62:0/4254523375 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f476406c680 0x7f476406eb30 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:56.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:56.126+0000 7f4785e00700 1 --2- 172.21.15.62:0/4254523375 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4780102440 0x7f4780197cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:56.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:56.126+0000 7f4785e00700 1 --2- 172.21.15.62:0/4254523375 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4780103640 0x7f47801981f0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:56.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:56.126+0000 7f4785e00700 1 -- 172.21.15.62:0/4254523375 >> 172.21.15.62:0/4254523375 conn(0x7f47800fd9f0 msgr2=0x7f4780106870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:26:56.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:56.126+0000 7f4785e00700 1 -- 172.21.15.62:0/4254523375 shutdown_connections 2024-04-20T21:26:56.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:56.127+0000 7f4785e00700 1 -- 172.21.15.62:0/4254523375 wait complete. 2024-04-20T21:26:56.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:26:55 smithi189 ceph-mon[21562]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:26:56.249 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:26:55 smithi062 ceph-mon[16221]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:26:56.984 INFO:teuthology.run_tasks:Running task print... 2024-04-20T21:26:56.993 INFO:teuthology.task.print:**** done cephadm.shell ceph config set mgr... 2024-04-20T21:26:56.993 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-20T21:26:57.023 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi062.front.sepia.ceph.com 2024-04-20T21:26:57.023 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -- bash -c 'ceph orch status' 2024-04-20T21:26:57.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:26:57 smithi062 ceph-mon[16221]: from='client.? 172.21.15.62:0/4254523375' entity='client.admin' 2024-04-20T21:26:57.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:26:57 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:26:57.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:26:57 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:26:57.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:26:57 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T21:26:57.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:26:57 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:26:57.455 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T21:26:57.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:26:57 smithi189 ceph-mon[21562]: from='client.? 172.21.15.62:0/4254523375' entity='client.admin' 2024-04-20T21:26:57.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:26:57 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:26:57.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:26:57 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:26:57.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:26:57 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T21:26:57.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:26:57 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:26:58.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:26:58 smithi189 ceph-mon[21562]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:26:58.523 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:26:58 smithi062 ceph-mon[16221]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:26:59.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:59.848+0000 7f56bb000700 1 -- 172.21.15.62:0/761375098 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f56b4101440 msgr2=0x7f56b4101890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:26:59.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:59.848+0000 7f56bb000700 1 --2- 172.21.15.62:0/761375098 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f56b4101440 0x7f56b4101890 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f569c009910 tx=0x7f569c009c20 comp rx=0 tx=0).stop 2024-04-20T21:26:59.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:59.848+0000 7f56bb000700 1 -- 172.21.15.62:0/761375098 shutdown_connections 2024-04-20T21:26:59.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:59.848+0000 7f56bb000700 1 --2- 172.21.15.62:0/761375098 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f56b4101440 0x7f56b4101890 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:59.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:59.848+0000 7f56bb000700 1 --2- 172.21.15.62:0/761375098 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56b4100240 0x7f56b4100650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:59.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:59.848+0000 7f56bb000700 1 -- 172.21.15.62:0/761375098 >> 172.21.15.62:0/761375098 conn(0x7f56b40fb7f0 msgr2=0x7f56b40fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:26:59.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:59.849+0000 7f56bb000700 1 -- 172.21.15.62:0/761375098 shutdown_connections 2024-04-20T21:26:59.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:59.849+0000 7f56bb000700 1 -- 172.21.15.62:0/761375098 wait complete. 2024-04-20T21:26:59.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:59.850+0000 7f56bb000700 1 Processor -- start 2024-04-20T21:26:59.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:59.850+0000 7f56bb000700 1 -- start start 2024-04-20T21:26:59.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:59.850+0000 7f56bb000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f56b4100240 0x7f56b4195ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:26:59.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:59.850+0000 7f56bb000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56b4101440 0x7f56b4195ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:26:59.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:59.850+0000 7f56bb000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56b4196580 con 0x7f56b4100240 2024-04-20T21:26:59.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:59.850+0000 7f56bb000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56b41966c0 con 0x7f56b4101440 2024-04-20T21:26:59.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:59.850+0000 7f56b3e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f56b4100240 0x7f56b4195ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:26:59.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:59.850+0000 7f56b3e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f56b4100240 0x7f56b4195ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33944/0 (socket says 172.21.15.62:33944) 2024-04-20T21:26:59.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:59.850+0000 7f56b3e00700 1 -- 172.21.15.62:0/4227856354 learned_addr learned my addr 172.21.15.62:0/4227856354 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:26:59.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:59.851+0000 7f56ab400700 1 --2- 172.21.15.62:0/4227856354 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56b4101440 0x7f56b4195ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:26:59.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:59.851+0000 7f56b3e00700 1 -- 172.21.15.62:0/4227856354 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56b4101440 msgr2=0x7f56b4195ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:26:59.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:59.851+0000 7f56b3e00700 1 --2- 172.21.15.62:0/4227856354 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56b4101440 0x7f56b4195ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:26:59.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:59.851+0000 7f56b3e00700 1 -- 172.21.15.62:0/4227856354 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f569c0095f0 con 0x7f56b4100240 2024-04-20T21:26:59.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:59.851+0000 7f56b3e00700 1 --2- 172.21.15.62:0/4227856354 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f56b4100240 0x7f56b4195ab0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f56a400cfd0 tx=0x7f56a400a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:26:59.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:59.852+0000 7f56b1600700 1 -- 172.21.15.62:0/4227856354 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f56a400ad70 con 0x7f56b4100240 2024-04-20T21:26:59.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:59.852+0000 7f56b1600700 1 -- 172.21.15.62:0/4227856354 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f56a4004020 con 0x7f56b4100240 2024-04-20T21:26:59.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:59.852+0000 7f56b1600700 1 -- 172.21.15.62:0/4227856354 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f56a4012650 con 0x7f56b4100240 2024-04-20T21:26:59.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:59.852+0000 7f56bb000700 1 -- 172.21.15.62:0/4227856354 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56b419b180 con 0x7f56b4100240 2024-04-20T21:26:59.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:59.852+0000 7f56bb000700 1 -- 172.21.15.62:0/4227856354 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56b419b6a0 con 0x7f56b4100240 2024-04-20T21:26:59.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:59.853+0000 7f56bb000700 1 -- 172.21.15.62:0/4227856354 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56b4108d90 con 0x7f56b4100240 2024-04-20T21:26:59.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:59.854+0000 7f56b1600700 1 -- 172.21.15.62:0/4227856354 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f56a4004190 con 0x7f56b4100240 2024-04-20T21:26:59.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:59.855+0000 7f56b1600700 1 --2- 172.21.15.62:0/4227856354 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f569806c680 0x7f569806eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:26:59.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:59.856+0000 7f56b1600700 1 -- 172.21.15.62:0/4227856354 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f56a408b6d0 con 0x7f56b4100240 2024-04-20T21:26:59.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:59.856+0000 7f56ab400700 1 --2- 172.21.15.62:0/4227856354 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f569806c680 0x7f569806eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:26:59.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:59.856+0000 7f56ab400700 1 --2- 172.21.15.62:0/4227856354 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f569806c680 0x7f569806eb30 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f569c006010 tx=0x7f569c00b540 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:26:59.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:26:59.860+0000 7f56b1600700 1 -- 172.21.15.62:0/4227856354 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f56a4059c00 con 0x7f56b4100240 2024-04-20T21:27:00.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:00.102+0000 7f56bb000700 1 -- 172.21.15.62:0/4227856354 --> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7f56b4105d90 con 0x7f569806c680 2024-04-20T21:27:00.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:00.104+0000 7f56b1600700 1 -- 172.21.15.62:0/4227856354 <== mgr.14215 v2:172.21.15.62:6800/4116451241 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f56b4105d90 con 0x7f569806c680 2024-04-20T21:27:00.104 INFO:teuthology.orchestra.run.smithi062.stdout:Backend: cephadm 2024-04-20T21:27:00.104 INFO:teuthology.orchestra.run.smithi062.stdout:Available: Yes 2024-04-20T21:27:00.104 INFO:teuthology.orchestra.run.smithi062.stdout:Paused: No 2024-04-20T21:27:00.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:00.107+0000 7f56bb000700 1 -- 172.21.15.62:0/4227856354 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f569806c680 msgr2=0x7f569806eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:27:00.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:00.107+0000 7f56bb000700 1 --2- 172.21.15.62:0/4227856354 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f569806c680 0x7f569806eb30 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f569c006010 tx=0x7f569c00b540 comp rx=0 tx=0).stop 2024-04-20T21:27:00.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:00.107+0000 7f56bb000700 1 -- 172.21.15.62:0/4227856354 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f56b4100240 msgr2=0x7f56b4195ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:27:00.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:00.107+0000 7f56bb000700 1 --2- 172.21.15.62:0/4227856354 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f56b4100240 0x7f56b4195ab0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f56a400cfd0 tx=0x7f56a400a340 comp rx=0 tx=0).stop 2024-04-20T21:27:00.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:00.107+0000 7f56bb000700 1 -- 172.21.15.62:0/4227856354 shutdown_connections 2024-04-20T21:27:00.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:00.107+0000 7f56bb000700 1 --2- 172.21.15.62:0/4227856354 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f569806c680 0x7f569806eb30 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:00.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:00.107+0000 7f56bb000700 1 --2- 172.21.15.62:0/4227856354 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f56b4100240 0x7f56b4195ab0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:00.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:00.107+0000 7f56bb000700 1 --2- 172.21.15.62:0/4227856354 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56b4101440 0x7f56b4195ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:00.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:00.107+0000 7f56bb000700 1 -- 172.21.15.62:0/4227856354 >> 172.21.15.62:0/4227856354 conn(0x7f56b40fb7f0 msgr2=0x7f56b4104670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:27:00.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:00.108+0000 7f56bb000700 1 -- 172.21.15.62:0/4227856354 shutdown_connections 2024-04-20T21:27:00.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:00.108+0000 7f56bb000700 1 -- 172.21.15.62:0/4227856354 wait complete. 2024-04-20T21:27:00.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:26:59 smithi189 ceph-mon[21562]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:27:00.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:26:59 smithi062 ceph-mon[16221]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:27:00.965 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -- bash -c 'ceph orch ps' 2024-04-20T21:27:01.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:00 smithi189 ceph-mon[21562]: from='client.14434 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:27:01.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:00 smithi062 ceph-mon[16221]: from='client.14434 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:27:01.386 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T21:27:02.897 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:02 smithi062 ceph-mon[16221]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:27:03.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:02 smithi189 ceph-mon[21562]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:27:03.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:03.702+0000 7f4ed6800700 1 -- 172.21.15.62:0/2658988506 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4ed0100d60 msgr2=0x7f4ed0103140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:27:03.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:03.702+0000 7f4ed6800700 1 --2- 172.21.15.62:0/2658988506 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4ed0100d60 0x7f4ed0103140 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f4ec000b0a0 tx=0x7f4ec000b3b0 comp rx=0 tx=0).stop 2024-04-20T21:27:03.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:03.703+0000 7f4ed6800700 1 -- 172.21.15.62:0/2658988506 shutdown_connections 2024-04-20T21:27:03.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:03.703+0000 7f4ed6800700 1 --2- 172.21.15.62:0/2658988506 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ed0103680 0x7f4ed0105a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:03.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:03.703+0000 7f4ed6800700 1 --2- 172.21.15.62:0/2658988506 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4ed0100d60 0x7f4ed0103140 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:03.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:03.703+0000 7f4ed6800700 1 -- 172.21.15.62:0/2658988506 >> 172.21.15.62:0/2658988506 conn(0x7f4ed00fa780 msgr2=0x7f4ed00fcbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:27:03.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:03.703+0000 7f4ed6800700 1 -- 172.21.15.62:0/2658988506 shutdown_connections 2024-04-20T21:27:03.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:03.703+0000 7f4ed6800700 1 -- 172.21.15.62:0/2658988506 wait complete. 2024-04-20T21:27:03.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:03.704+0000 7f4ed6800700 1 Processor -- start 2024-04-20T21:27:03.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:03.704+0000 7f4ed6800700 1 -- start start 2024-04-20T21:27:03.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:03.705+0000 7f4ed6800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ed0100d60 0x7f4ed0195ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:27:03.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:03.705+0000 7f4ed6800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4ed0103680 0x7f4ed0196020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:27:03.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:03.705+0000 7f4ed6800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed01965b0 con 0x7f4ed0103680 2024-04-20T21:27:03.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:03.705+0000 7f4ed6800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed01966f0 con 0x7f4ed0100d60 2024-04-20T21:27:03.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:03.705+0000 7f4ec6a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4ed0103680 0x7f4ed0196020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:27:03.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:03.705+0000 7f4ec6a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4ed0103680 0x7f4ed0196020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:58280/0 (socket says 172.21.15.62:58280) 2024-04-20T21:27:03.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:03.705+0000 7f4ec6a00700 1 -- 172.21.15.62:0/1121424721 learned_addr learned my addr 172.21.15.62:0/1121424721 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:27:03.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:03.705+0000 7f4ecf400700 1 --2- 172.21.15.62:0/1121424721 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ed0100d60 0x7f4ed0195ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:27:03.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:03.706+0000 7f4ec6a00700 1 -- 172.21.15.62:0/1121424721 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ed0100d60 msgr2=0x7f4ed0195ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:27:03.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:03.706+0000 7f4ec6a00700 1 --2- 172.21.15.62:0/1121424721 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ed0100d60 0x7f4ed0195ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:03.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:03.706+0000 7f4ec6a00700 1 -- 172.21.15.62:0/1121424721 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ec0009d00 con 0x7f4ed0103680 2024-04-20T21:27:03.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:03.706+0000 7f4ec6a00700 1 --2- 172.21.15.62:0/1121424721 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4ed0103680 0x7f4ed0196020 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f4eb800c930 tx=0x7f4eb800ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:27:03.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:03.706+0000 7f4eccc00700 1 -- 172.21.15.62:0/1121424721 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4eb8007ad0 con 0x7f4ed0103680 2024-04-20T21:27:03.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:03.706+0000 7f4eccc00700 1 -- 172.21.15.62:0/1121424721 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f4eb8007c30 con 0x7f4ed0103680 2024-04-20T21:27:03.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:03.706+0000 7f4ed6800700 1 -- 172.21.15.62:0/1121424721 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ed019b1b0 con 0x7f4ed0103680 2024-04-20T21:27:03.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:03.706+0000 7f4eccc00700 1 -- 172.21.15.62:0/1121424721 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4eb8018630 con 0x7f4ed0103680 2024-04-20T21:27:03.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:03.707+0000 7f4ed6800700 1 -- 172.21.15.62:0/1121424721 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ed019b6d0 con 0x7f4ed0103680 2024-04-20T21:27:03.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:03.708+0000 7f4ed6800700 1 -- 172.21.15.62:0/1121424721 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ed018fbc0 con 0x7f4ed0103680 2024-04-20T21:27:03.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:03.709+0000 7f4eccc00700 1 -- 172.21.15.62:0/1121424721 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f4eb800f450 con 0x7f4ed0103680 2024-04-20T21:27:03.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:03.710+0000 7f4eccc00700 1 --2- 172.21.15.62:0/1121424721 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f4eb406c680 0x7f4eb406eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:27:03.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:03.710+0000 7f4eccc00700 1 -- 172.21.15.62:0/1121424721 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f4eb808b710 con 0x7f4ed0103680 2024-04-20T21:27:03.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:03.710+0000 7f4ecf400700 1 --2- 172.21.15.62:0/1121424721 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f4eb406c680 0x7f4eb406eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:27:03.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:03.711+0000 7f4ecf400700 1 --2- 172.21.15.62:0/1121424721 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f4eb406c680 0x7f4eb406eb30 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4ec0009fd0 tx=0x7f4ec0007a60 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:27:03.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:03.716+0000 7f4eccc00700 1 -- 172.21.15.62:0/1121424721 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f4eb8059ba0 con 0x7f4ed0103680 2024-04-20T21:27:03.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:03.958+0000 7f4ed6800700 1 -- 172.21.15.62:0/1121424721 --> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f4ed0061190 con 0x7f4eb406c680 2024-04-20T21:27:03.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:03.988+0000 7f4eccc00700 1 -- 172.21.15.62:0/1121424721 <== mgr.14215 v2:172.21.15.62:6800/4116451241 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+2820 (secure 0 0 0) 0x7f4ed0061190 con 0x7f4eb406c680 2024-04-20T21:27:03.988 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T21:27:03.988 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (4m) 2m ago 6m 13.6M - 0.25.0 c8568f914cd2 957f663a9cd7 2024-04-20T21:27:03.988 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (7m) 2m ago 7m 7088k - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T21:27:03.988 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (5m) 81s ago 5m 6924k - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T21:27:03.988 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (7m) 2m ago 7m 6656k - 18.2.0 dc2bc1663786 36e10ef5be39 2024-04-20T21:27:03.988 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (5m) 81s ago 5m 6665k - 18.2.0 dc2bc1663786 4101f9953e3d 2024-04-20T21:27:03.988 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (4m) 2m ago 6m 74.4M - 9.4.7 954c08fa6188 99363bb095d8 2024-04-20T21:27:03.988 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:9283,8765,8443 running (8m) 2m ago 8m 480M - 18.2.0 dc2bc1663786 e411b3dc7233 2024-04-20T21:27:03.988 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (5m) 81s ago 5m 436M - 18.2.0 dc2bc1663786 a31cf3b36efc 2024-04-20T21:27:03.988 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (8m) 2m ago 8m 43.4M 2048M 18.2.0 dc2bc1663786 bd6efbd5c380 2024-04-20T21:27:03.989 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (5m) 81s ago 5m 35.2M 2048M 18.2.0 dc2bc1663786 3735c8e5db9a 2024-04-20T21:27:03.989 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (6m) 2m ago 6m 7638k - 1.5.0 0da6a335fe13 fc746d8ae72e 2024-04-20T21:27:03.989 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (5m) 81s ago 5m 7478k - 1.5.0 0da6a335fe13 db03af85b286 2024-04-20T21:27:03.989 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (4m) 2m ago 4m 40.2M 3947M 18.2.0 dc2bc1663786 3f315c3afc1f 2024-04-20T21:27:03.989 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (3m) 2m ago 3m 41.3M 3947M 18.2.0 dc2bc1663786 44e4d07416ad 2024-04-20T21:27:03.989 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (3m) 2m ago 3m 39.4M 3947M 18.2.0 dc2bc1663786 31c5d9586e98 2024-04-20T21:27:03.989 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (2m) 81s ago 2m 40.7M 4971M 18.2.0 dc2bc1663786 611c1e242693 2024-04-20T21:27:03.989 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (2m) 81s ago 2m 38.2M 4971M 18.2.0 dc2bc1663786 6c8fce91d45d 2024-04-20T21:27:03.989 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (87s) 81s ago 87s 11.0M 4971M 18.2.0 dc2bc1663786 8e4574bae8db 2024-04-20T21:27:03.989 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (4m) 2m ago 6m 39.1M - 2.43.0 a07b618ecd1d cd390b7e44d0 2024-04-20T21:27:03.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:03.990+0000 7f4ed6800700 1 -- 172.21.15.62:0/1121424721 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f4eb406c680 msgr2=0x7f4eb406eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:27:03.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:03.990+0000 7f4ed6800700 1 --2- 172.21.15.62:0/1121424721 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f4eb406c680 0x7f4eb406eb30 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4ec0009fd0 tx=0x7f4ec0007a60 comp rx=0 tx=0).stop 2024-04-20T21:27:03.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:03.991+0000 7f4ed6800700 1 -- 172.21.15.62:0/1121424721 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4ed0103680 msgr2=0x7f4ed0196020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:27:03.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:03.991+0000 7f4ed6800700 1 --2- 172.21.15.62:0/1121424721 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4ed0103680 0x7f4ed0196020 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f4eb800c930 tx=0x7f4eb800ccf0 comp rx=0 tx=0).stop 2024-04-20T21:27:03.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:03.991+0000 7f4ed6800700 1 -- 172.21.15.62:0/1121424721 shutdown_connections 2024-04-20T21:27:03.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:03.991+0000 7f4ed6800700 1 --2- 172.21.15.62:0/1121424721 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f4eb406c680 0x7f4eb406eb30 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:03.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:03.991+0000 7f4ed6800700 1 --2- 172.21.15.62:0/1121424721 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ed0100d60 0x7f4ed0195ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:03.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:03.991+0000 7f4ed6800700 1 --2- 172.21.15.62:0/1121424721 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4ed0103680 0x7f4ed0196020 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:03.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:03.991+0000 7f4ed6800700 1 -- 172.21.15.62:0/1121424721 >> 172.21.15.62:0/1121424721 conn(0x7f4ed00fa780 msgr2=0x7f4ed00fcbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:27:03.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:03.991+0000 7f4ed6800700 1 -- 172.21.15.62:0/1121424721 shutdown_connections 2024-04-20T21:27:03.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:03.992+0000 7f4ed6800700 1 -- 172.21.15.62:0/1121424721 wait complete. 2024-04-20T21:27:04.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:03 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:27:04.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:03 smithi189 ceph-mon[21562]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:27:04.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:03 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:27:04.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:03 smithi062 ceph-mon[16221]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:27:04.871 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -- bash -c 'ceph orch ls' 2024-04-20T21:27:05.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:04 smithi189 ceph-mon[21562]: from='client.14438 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:27:05.284 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:04 smithi062 ceph-mon[16221]: from='client.14438 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:27:05.308 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T21:27:06.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:05 smithi062 ceph-mon[16221]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:27:06.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:05 smithi189 ceph-mon[21562]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:27:07.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:07.635+0000 7fdc1ca00700 1 -- 172.21.15.62:0/522567482 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc18103500 msgr2=0x7fdc18103950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:27:07.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:07.635+0000 7fdc1ca00700 1 --2- 172.21.15.62:0/522567482 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc18103500 0x7fdc18103950 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fdc04009910 tx=0x7fdc04009c20 comp rx=0 tx=0).stop 2024-04-20T21:27:07.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:07.635+0000 7fdc1ca00700 1 -- 172.21.15.62:0/522567482 shutdown_connections 2024-04-20T21:27:07.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:07.635+0000 7fdc1ca00700 1 --2- 172.21.15.62:0/522567482 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc18103500 0x7fdc18103950 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:07.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:07.635+0000 7fdc1ca00700 1 --2- 172.21.15.62:0/522567482 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc18102300 0x7fdc18102710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:07.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:07.635+0000 7fdc1ca00700 1 -- 172.21.15.62:0/522567482 >> 172.21.15.62:0/522567482 conn(0x7fdc180fd8d0 msgr2=0x7fdc180ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:27:07.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:07.635+0000 7fdc1ca00700 1 -- 172.21.15.62:0/522567482 shutdown_connections 2024-04-20T21:27:07.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:07.635+0000 7fdc1ca00700 1 -- 172.21.15.62:0/522567482 wait complete. 2024-04-20T21:27:07.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:07.637+0000 7fdc1ca00700 1 Processor -- start 2024-04-20T21:27:07.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:07.637+0000 7fdc1ca00700 1 -- start start 2024-04-20T21:27:07.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:07.637+0000 7fdc1ca00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc18102300 0x7fdc18197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:27:07.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:07.637+0000 7fdc1ca00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc18103500 0x7fdc18197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:27:07.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:07.637+0000 7fdc1ca00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc18198500 con 0x7fdc18103500 2024-04-20T21:27:07.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:07.637+0000 7fdc1ca00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc18198640 con 0x7fdc18102300 2024-04-20T21:27:07.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:07.638+0000 7fdc15e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc18103500 0x7fdc18197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:27:07.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:07.638+0000 7fdc15e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc18103500 0x7fdc18197f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:58302/0 (socket says 172.21.15.62:58302) 2024-04-20T21:27:07.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:07.638+0000 7fdc15e00700 1 -- 172.21.15.62:0/1680165526 learned_addr learned my addr 172.21.15.62:0/1680165526 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:27:07.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:07.638+0000 7fdc16800700 1 --2- 172.21.15.62:0/1680165526 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc18102300 0x7fdc18197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:27:07.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:07.638+0000 7fdc15e00700 1 -- 172.21.15.62:0/1680165526 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc18102300 msgr2=0x7fdc18197a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:27:07.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:07.638+0000 7fdc15e00700 1 --2- 172.21.15.62:0/1680165526 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc18102300 0x7fdc18197a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:07.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:07.638+0000 7fdc15e00700 1 -- 172.21.15.62:0/1680165526 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc040095f0 con 0x7fdc18103500 2024-04-20T21:27:07.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:07.639+0000 7fdc15e00700 1 --2- 172.21.15.62:0/1680165526 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc18103500 0x7fdc18197f70 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fdc04005b40 tx=0x7fdc0400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:27:07.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:07.639+0000 7fdc0f400700 1 -- 172.21.15.62:0/1680165526 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdc0401d070 con 0x7fdc18103500 2024-04-20T21:27:07.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:07.639+0000 7fdc0f400700 1 -- 172.21.15.62:0/1680165526 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fdc04022470 con 0x7fdc18103500 2024-04-20T21:27:07.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:07.639+0000 7fdc0f400700 1 -- 172.21.15.62:0/1680165526 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdc0400f690 con 0x7fdc18103500 2024-04-20T21:27:07.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:07.639+0000 7fdc1ca00700 1 -- 172.21.15.62:0/1680165526 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc1819d0a0 con 0x7fdc18103500 2024-04-20T21:27:07.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:07.639+0000 7fdc1ca00700 1 -- 172.21.15.62:0/1680165526 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc1819d510 con 0x7fdc18103500 2024-04-20T21:27:07.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:07.640+0000 7fdc1ca00700 1 -- 172.21.15.62:0/1680165526 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc1810ae50 con 0x7fdc18103500 2024-04-20T21:27:07.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:07.641+0000 7fdc0f400700 1 -- 172.21.15.62:0/1680165526 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7fdc040225e0 con 0x7fdc18103500 2024-04-20T21:27:07.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:07.642+0000 7fdc0f400700 1 --2- 172.21.15.62:0/1680165526 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fdbfc06c680 0x7fdbfc06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:27:07.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:07.642+0000 7fdc0f400700 1 -- 172.21.15.62:0/1680165526 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fdc0408ce20 con 0x7fdc18103500 2024-04-20T21:27:07.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:07.643+0000 7fdc16800700 1 --2- 172.21.15.62:0/1680165526 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fdbfc06c680 0x7fdbfc06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:27:07.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:07.643+0000 7fdc16800700 1 --2- 172.21.15.62:0/1680165526 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fdbfc06c680 0x7fdbfc06eb30 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fdc08005fd0 tx=0x7fdc08005e50 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:27:07.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:07.647+0000 7fdc0f400700 1 -- 172.21.15.62:0/1680165526 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fdc0405b2b0 con 0x7fdc18103500 2024-04-20T21:27:07.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:07.879+0000 7fdc1ca00700 1 -- 172.21.15.62:0/1680165526 --> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fdc18107e50 con 0x7fdbfc06c680 2024-04-20T21:27:07.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:07.884+0000 7fdc0f400700 1 -- 172.21.15.62:0/1680165526 <== mgr.14215 v2:172.21.15.62:6800/4116451241 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+1300 (secure 0 0 0) 0x7fdc18107e50 con 0x7fdbfc06c680 2024-04-20T21:27:07.883 INFO:teuthology.orchestra.run.smithi062.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-04-20T21:27:07.883 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager ?:9093,9094 1/1 2m ago 7m count:1 2024-04-20T21:27:07.884 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter 2/2 2m ago 7m * 2024-04-20T21:27:07.884 INFO:teuthology.orchestra.run.smithi062.stdout:crash 2/2 2m ago 7m * 2024-04-20T21:27:07.884 INFO:teuthology.orchestra.run.smithi062.stdout:grafana ?:3000 1/1 2m ago 7m count:1 2024-04-20T21:27:07.884 INFO:teuthology.orchestra.run.smithi062.stdout:mgr 2/2 2m ago 7m count:2 2024-04-20T21:27:07.884 INFO:teuthology.orchestra.run.smithi062.stdout:mon 2/2 2m ago 6m smithi062:172.21.15.62=smithi062;smithi189:172.21.15.189=smithi189;count:2 2024-04-20T21:27:07.884 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter ?:9100 2/2 2m ago 7m * 2024-04-20T21:27:07.884 INFO:teuthology.orchestra.run.smithi062.stdout:osd 6 2m ago - 2024-04-20T21:27:07.884 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus ?:9095 1/1 2m ago 7m count:1 2024-04-20T21:27:07.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:07.887+0000 7fdc1ca00700 1 -- 172.21.15.62:0/1680165526 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fdbfc06c680 msgr2=0x7fdbfc06eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:27:07.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:07.887+0000 7fdc1ca00700 1 --2- 172.21.15.62:0/1680165526 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fdbfc06c680 0x7fdbfc06eb30 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fdc08005fd0 tx=0x7fdc08005e50 comp rx=0 tx=0).stop 2024-04-20T21:27:07.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:07.887+0000 7fdc1ca00700 1 -- 172.21.15.62:0/1680165526 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc18103500 msgr2=0x7fdc18197f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:27:07.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:07.887+0000 7fdc1ca00700 1 --2- 172.21.15.62:0/1680165526 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc18103500 0x7fdc18197f70 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fdc04005b40 tx=0x7fdc0400be20 comp rx=0 tx=0).stop 2024-04-20T21:27:07.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:07.888+0000 7fdc1ca00700 1 -- 172.21.15.62:0/1680165526 shutdown_connections 2024-04-20T21:27:07.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:07.888+0000 7fdc1ca00700 1 --2- 172.21.15.62:0/1680165526 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fdbfc06c680 0x7fdbfc06eb30 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:07.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:07.888+0000 7fdc1ca00700 1 --2- 172.21.15.62:0/1680165526 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc18102300 0x7fdc18197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:07.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:07.888+0000 7fdc1ca00700 1 --2- 172.21.15.62:0/1680165526 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc18103500 0x7fdc18197f70 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:07.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:07.888+0000 7fdc1ca00700 1 -- 172.21.15.62:0/1680165526 >> 172.21.15.62:0/1680165526 conn(0x7fdc180fd8d0 msgr2=0x7fdc18106730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:27:07.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:07.888+0000 7fdc1ca00700 1 -- 172.21.15.62:0/1680165526 shutdown_connections 2024-04-20T21:27:07.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:07.888+0000 7fdc1ca00700 1 -- 172.21.15.62:0/1680165526 wait complete. 2024-04-20T21:27:08.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:07 smithi189 ceph-mon[21562]: from='client.14442 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:27:08.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:07 smithi189 ceph-mon[21562]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:27:08.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:07 smithi062 ceph-mon[16221]: from='client.14442 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:27:08.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:07 smithi062 ceph-mon[16221]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:27:08.772 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -- bash -c 'ceph orch host ls' 2024-04-20T21:27:09.191 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T21:27:10.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:09 smithi189 ceph-mon[21562]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:27:10.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:09 smithi062 ceph-mon[16221]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:27:11.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:11.418+0000 7f5faa400700 1 -- 172.21.15.62:0/730306630 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5fa4103480 msgr2=0x7f5fa4105860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:27:11.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:11.418+0000 7f5faa400700 1 --2- 172.21.15.62:0/730306630 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5fa4103480 0x7f5fa4105860 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f5f90009910 tx=0x7f5f90009c20 comp rx=0 tx=0).stop 2024-04-20T21:27:11.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:11.418+0000 7f5faa400700 1 -- 172.21.15.62:0/730306630 shutdown_connections 2024-04-20T21:27:11.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:11.418+0000 7f5faa400700 1 --2- 172.21.15.62:0/730306630 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5fa4103480 0x7f5fa4105860 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:11.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:11.418+0000 7f5faa400700 1 --2- 172.21.15.62:0/730306630 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5fa4100b60 0x7f5fa4102f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:11.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:11.418+0000 7f5faa400700 1 -- 172.21.15.62:0/730306630 >> 172.21.15.62:0/730306630 conn(0x7f5fa40fa5a0 msgr2=0x7f5fa40fc9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:27:11.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:11.418+0000 7f5faa400700 1 -- 172.21.15.62:0/730306630 shutdown_connections 2024-04-20T21:27:11.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:11.418+0000 7f5faa400700 1 -- 172.21.15.62:0/730306630 wait complete. 2024-04-20T21:27:11.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:11.419+0000 7f5faa400700 1 Processor -- start 2024-04-20T21:27:11.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:11.419+0000 7f5faa400700 1 -- start start 2024-04-20T21:27:11.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:11.420+0000 7f5faa400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5fa4100b60 0x7f5fa4197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:27:11.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:11.420+0000 7f5faa400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5fa4103480 0x7f5fa4197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:27:11.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:11.420+0000 7f5faa400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa41985c0 con 0x7f5fa4103480 2024-04-20T21:27:11.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:11.420+0000 7f5faa400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa4198700 con 0x7f5fa4100b60 2024-04-20T21:27:11.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:11.420+0000 7f5fa3e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5fa4103480 0x7f5fa4197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:27:11.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:11.420+0000 7f5fa3e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5fa4103480 0x7f5fa4197fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40266/0 (socket says 172.21.15.62:40266) 2024-04-20T21:27:11.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:11.420+0000 7f5fa3e00700 1 -- 172.21.15.62:0/419839407 learned_addr learned my addr 172.21.15.62:0/419839407 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:27:11.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:11.420+0000 7f5fa9000700 1 --2- 172.21.15.62:0/419839407 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5fa4100b60 0x7f5fa4197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:27:11.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:11.420+0000 7f5fa3e00700 1 -- 172.21.15.62:0/419839407 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5fa4100b60 msgr2=0x7f5fa4197a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:27:11.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:11.420+0000 7f5fa3e00700 1 --2- 172.21.15.62:0/419839407 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5fa4100b60 0x7f5fa4197a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:11.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:11.420+0000 7f5fa3e00700 1 -- 172.21.15.62:0/419839407 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f900095f0 con 0x7f5fa4103480 2024-04-20T21:27:11.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:11.421+0000 7f5fa3e00700 1 --2- 172.21.15.62:0/419839407 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5fa4103480 0x7f5fa4197fa0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f5f90005b40 tx=0x7f5f9000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:27:11.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:11.421+0000 7f5fa1600700 1 -- 172.21.15.62:0/419839407 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5f9001d070 con 0x7f5fa4103480 2024-04-20T21:27:11.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:11.421+0000 7f5fa1600700 1 -- 172.21.15.62:0/419839407 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f5f9000f460 con 0x7f5fa4103480 2024-04-20T21:27:11.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:11.421+0000 7f5fa1600700 1 -- 172.21.15.62:0/419839407 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5f90021640 con 0x7f5fa4103480 2024-04-20T21:27:11.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:11.421+0000 7f5faa400700 1 -- 172.21.15.62:0/419839407 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fa419d150 con 0x7f5fa4103480 2024-04-20T21:27:11.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:11.421+0000 7f5faa400700 1 -- 172.21.15.62:0/419839407 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fa419d640 con 0x7f5fa4103480 2024-04-20T21:27:11.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:11.423+0000 7f5faa400700 1 -- 172.21.15.62:0/419839407 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fa40fc110 con 0x7f5fa4103480 2024-04-20T21:27:11.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:11.424+0000 7f5fa1600700 1 -- 172.21.15.62:0/419839407 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f5f9000fa90 con 0x7f5fa4103480 2024-04-20T21:27:11.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:11.425+0000 7f5fa1600700 1 --2- 172.21.15.62:0/419839407 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f5f8c06c680 0x7f5f8c06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:27:11.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:11.425+0000 7f5fa1600700 1 -- 172.21.15.62:0/419839407 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f5f9008c8b0 con 0x7f5fa4103480 2024-04-20T21:27:11.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:11.425+0000 7f5fa9000700 1 --2- 172.21.15.62:0/419839407 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f5f8c06c680 0x7f5f8c06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:27:11.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:11.426+0000 7f5fa9000700 1 --2- 172.21.15.62:0/419839407 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f5f8c06c680 0x7f5f8c06eb30 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f5f98005950 tx=0x7f5f980058e0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:27:11.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:11.430+0000 7f5fa1600700 1 -- 172.21.15.62:0/419839407 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f5f9005ad40 con 0x7f5fa4103480 2024-04-20T21:27:11.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:11.657+0000 7f5faa400700 1 -- 172.21.15.62:0/419839407 --> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f5fa4061190 con 0x7f5f8c06c680 2024-04-20T21:27:11.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:11.660+0000 7f5fa1600700 1 -- 172.21.15.62:0/419839407 <== mgr.14215 v2:172.21.15.62:6800/4116451241 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+148 (secure 0 0 0) 0x7f5fa4061190 con 0x7f5f8c06c680 2024-04-20T21:27:11.661 INFO:teuthology.orchestra.run.smithi062.stdout:HOST ADDR LABELS STATUS 2024-04-20T21:27:11.661 INFO:teuthology.orchestra.run.smithi062.stdout:smithi062 172.21.15.62 2024-04-20T21:27:11.662 INFO:teuthology.orchestra.run.smithi062.stdout:smithi189 172.21.15.189 2024-04-20T21:27:11.662 INFO:teuthology.orchestra.run.smithi062.stdout:2 hosts in cluster 2024-04-20T21:27:11.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:11.663+0000 7f5faa400700 1 -- 172.21.15.62:0/419839407 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f5f8c06c680 msgr2=0x7f5f8c06eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:27:11.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:11.663+0000 7f5faa400700 1 --2- 172.21.15.62:0/419839407 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f5f8c06c680 0x7f5f8c06eb30 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f5f98005950 tx=0x7f5f980058e0 comp rx=0 tx=0).stop 2024-04-20T21:27:11.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:11.664+0000 7f5faa400700 1 -- 172.21.15.62:0/419839407 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5fa4103480 msgr2=0x7f5fa4197fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:27:11.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:11.664+0000 7f5faa400700 1 --2- 172.21.15.62:0/419839407 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5fa4103480 0x7f5fa4197fa0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f5f90005b40 tx=0x7f5f9000be20 comp rx=0 tx=0).stop 2024-04-20T21:27:11.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:11.664+0000 7f5faa400700 1 -- 172.21.15.62:0/419839407 shutdown_connections 2024-04-20T21:27:11.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:11.664+0000 7f5faa400700 1 --2- 172.21.15.62:0/419839407 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f5f8c06c680 0x7f5f8c06eb30 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:11.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:11.664+0000 7f5faa400700 1 --2- 172.21.15.62:0/419839407 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5fa4100b60 0x7f5fa4197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:11.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:11.664+0000 7f5faa400700 1 --2- 172.21.15.62:0/419839407 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5fa4103480 0x7f5fa4197fa0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:11.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:11.664+0000 7f5faa400700 1 -- 172.21.15.62:0/419839407 >> 172.21.15.62:0/419839407 conn(0x7f5fa40fa5a0 msgr2=0x7f5fa40ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:27:11.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:11.664+0000 7f5faa400700 1 -- 172.21.15.62:0/419839407 shutdown_connections 2024-04-20T21:27:11.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:11.664+0000 7f5faa400700 1 -- 172.21.15.62:0/419839407 wait complete. 2024-04-20T21:27:12.460 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -- bash -c 'ceph orch device ls' 2024-04-20T21:27:12.901 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T21:27:13.056 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:12 smithi062 ceph-mon[16221]: from='client.14446 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:27:13.056 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:12 smithi062 ceph-mon[16221]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:27:13.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:12 smithi189 ceph-mon[21562]: from='client.14446 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:27:13.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:12 smithi189 ceph-mon[21562]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:27:14.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:13 smithi189 ceph-mon[21562]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:27:14.251 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:13 smithi062 ceph-mon[16221]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:27:15.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:15.087+0000 7f5469000700 1 -- 172.21.15.62:0/4138950443 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5464107a40 msgr2=0x7f5464107e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:27:15.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:15.087+0000 7f5469000700 1 --2- 172.21.15.62:0/4138950443 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5464107a40 0x7f5464107e90 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f544c009910 tx=0x7f544c009c20 comp rx=0 tx=0).stop 2024-04-20T21:27:15.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:15.087+0000 7f5469000700 1 -- 172.21.15.62:0/4138950443 shutdown_connections 2024-04-20T21:27:15.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:15.087+0000 7f5469000700 1 --2- 172.21.15.62:0/4138950443 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5464107a40 0x7f5464107e90 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:15.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:15.087+0000 7f5469000700 1 --2- 172.21.15.62:0/4138950443 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5464106840 0x7f5464106c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:15.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:15.088+0000 7f5469000700 1 -- 172.21.15.62:0/4138950443 >> 172.21.15.62:0/4138950443 conn(0x7f5464075970 msgr2=0x7f5464077dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:27:15.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:15.088+0000 7f5469000700 1 -- 172.21.15.62:0/4138950443 shutdown_connections 2024-04-20T21:27:15.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:15.088+0000 7f5469000700 1 -- 172.21.15.62:0/4138950443 wait complete. 2024-04-20T21:27:15.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:15.089+0000 7f5469000700 1 Processor -- start 2024-04-20T21:27:15.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:15.089+0000 7f5469000700 1 -- start start 2024-04-20T21:27:15.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:15.089+0000 7f5469000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5464106840 0x7f546419c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:27:15.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:15.089+0000 7f5469000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5464107a40 0x7f546419c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:27:15.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:15.089+0000 7f5469000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f546419cc40 con 0x7f5464106840 2024-04-20T21:27:15.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:15.089+0000 7f5469000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f546419cd80 con 0x7f5464107a40 2024-04-20T21:27:15.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:15.090+0000 7f5461e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5464106840 0x7f546419c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:27:15.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:15.090+0000 7f5461e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5464106840 0x7f546419c0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40292/0 (socket says 172.21.15.62:40292) 2024-04-20T21:27:15.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:15.090+0000 7f5461e00700 1 -- 172.21.15.62:0/1547545372 learned_addr learned my addr 172.21.15.62:0/1547545372 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:27:15.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:15.090+0000 7f545be00700 1 --2- 172.21.15.62:0/1547545372 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5464107a40 0x7f546419c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:27:15.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:15.090+0000 7f5461e00700 1 -- 172.21.15.62:0/1547545372 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5464107a40 msgr2=0x7f546419c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:27:15.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:15.090+0000 7f5461e00700 1 --2- 172.21.15.62:0/1547545372 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5464107a40 0x7f546419c620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:15.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:15.090+0000 7f5461e00700 1 -- 172.21.15.62:0/1547545372 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f544c0095f0 con 0x7f5464106840 2024-04-20T21:27:15.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:15.090+0000 7f5461e00700 1 --2- 172.21.15.62:0/1547545372 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5464106840 0x7f546419c0e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f545400d900 tx=0x7f545400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:27:15.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:15.091+0000 7f545aa00700 1 -- 172.21.15.62:0/1547545372 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f54540041f0 con 0x7f5464106840 2024-04-20T21:27:15.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:15.091+0000 7f545aa00700 1 -- 172.21.15.62:0/1547545372 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f5454004350 con 0x7f5464106840 2024-04-20T21:27:15.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:15.091+0000 7f5469000700 1 -- 172.21.15.62:0/1547545372 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54641a1830 con 0x7f5464106840 2024-04-20T21:27:15.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:15.091+0000 7f545aa00700 1 -- 172.21.15.62:0/1547545372 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5454003da0 con 0x7f5464106840 2024-04-20T21:27:15.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:15.091+0000 7f5469000700 1 -- 172.21.15.62:0/1547545372 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54641a1d80 con 0x7f5464106840 2024-04-20T21:27:15.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:15.092+0000 7f5469000700 1 -- 172.21.15.62:0/1547545372 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f546410f390 con 0x7f5464106840 2024-04-20T21:27:15.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:15.093+0000 7f545aa00700 1 -- 172.21.15.62:0/1547545372 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f5454009730 con 0x7f5464106840 2024-04-20T21:27:15.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:15.094+0000 7f545aa00700 1 --2- 172.21.15.62:0/1547545372 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f544806c680 0x7f544806eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:27:15.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:15.094+0000 7f545aa00700 1 -- 172.21.15.62:0/1547545372 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f545408b750 con 0x7f5464106840 2024-04-20T21:27:15.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:15.094+0000 7f545be00700 1 --2- 172.21.15.62:0/1547545372 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f544806c680 0x7f544806eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:27:15.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:15.095+0000 7f545be00700 1 --2- 172.21.15.62:0/1547545372 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f544806c680 0x7f544806eb30 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f544c006010 tx=0x7f544c00b580 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:27:15.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:15.099+0000 7f545aa00700 1 -- 172.21.15.62:0/1547545372 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f5454059b60 con 0x7f5464106840 2024-04-20T21:27:15.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:15.328+0000 7f5469000700 1 -- 172.21.15.62:0/1547545372 --> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f546410c390 con 0x7f544806c680 2024-04-20T21:27:15.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:15.331+0000 7f545aa00700 1 -- 172.21.15.62:0/1547545372 <== mgr.14215 v2:172.21.15.62:6800/4116451241 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+369 (secure 0 0 0) 0x7f546410c390 con 0x7f544806c680 2024-04-20T21:27:15.331 INFO:teuthology.orchestra.run.smithi062.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-04-20T21:27:15.331 INFO:teuthology.orchestra.run.smithi062.stdout:smithi062 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT53440094400BGN 372G No 3m ago LVM detected, locked 2024-04-20T21:27:15.331 INFO:teuthology.orchestra.run.smithi062.stdout:smithi189 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT6232004L400BGN 372G No 88s ago LVM detected, locked 2024-04-20T21:27:15.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:15.334+0000 7f5469000700 1 -- 172.21.15.62:0/1547545372 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f544806c680 msgr2=0x7f544806eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:27:15.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:15.334+0000 7f5469000700 1 --2- 172.21.15.62:0/1547545372 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f544806c680 0x7f544806eb30 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f544c006010 tx=0x7f544c00b580 comp rx=0 tx=0).stop 2024-04-20T21:27:15.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:15.334+0000 7f5469000700 1 -- 172.21.15.62:0/1547545372 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5464106840 msgr2=0x7f546419c0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:27:15.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:15.334+0000 7f5469000700 1 --2- 172.21.15.62:0/1547545372 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5464106840 0x7f546419c0e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f545400d900 tx=0x7f545400dcc0 comp rx=0 tx=0).stop 2024-04-20T21:27:15.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:15.334+0000 7f5469000700 1 -- 172.21.15.62:0/1547545372 shutdown_connections 2024-04-20T21:27:15.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:15.334+0000 7f5469000700 1 --2- 172.21.15.62:0/1547545372 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f544806c680 0x7f544806eb30 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:15.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:15.334+0000 7f5469000700 1 --2- 172.21.15.62:0/1547545372 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5464106840 0x7f546419c0e0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:15.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:15.334+0000 7f5469000700 1 --2- 172.21.15.62:0/1547545372 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5464107a40 0x7f546419c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:15.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:15.334+0000 7f5469000700 1 -- 172.21.15.62:0/1547545372 >> 172.21.15.62:0/1547545372 conn(0x7f5464075970 msgr2=0x7f546410ac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:27:15.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:15.334+0000 7f5469000700 1 -- 172.21.15.62:0/1547545372 shutdown_connections 2024-04-20T21:27:15.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:15.335+0000 7f5469000700 1 -- 172.21.15.62:0/1547545372 wait complete. 2024-04-20T21:27:16.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:15 smithi062 ceph-mon[16221]: from='client.14450 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:27:16.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:15 smithi062 ceph-mon[16221]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:27:16.147 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-20T21:27:16.157 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi062.front.sepia.ceph.com 2024-04-20T21:27:16.157 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -- bash -c 'ceph fs volume create cephfs --placement=4' 2024-04-20T21:27:16.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:15 smithi189 ceph-mon[21562]: from='client.14450 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:27:16.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:15 smithi189 ceph-mon[21562]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:27:16.596 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T21:27:18.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:17 smithi189 ceph-mon[21562]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:27:18.244 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:17 smithi062 ceph-mon[16221]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:27:18.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:18.743+0000 7ff555c00700 1 -- 172.21.15.62:0/3167852402 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff550074d80 msgr2=0x7ff5500731e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:27:18.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:18.743+0000 7ff555c00700 1 --2- 172.21.15.62:0/3167852402 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff550074d80 0x7ff5500731e0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7ff540009910 tx=0x7ff540009c20 comp rx=0 tx=0).stop 2024-04-20T21:27:18.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:18.743+0000 7ff555c00700 1 -- 172.21.15.62:0/3167852402 shutdown_connections 2024-04-20T21:27:18.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:18.743+0000 7ff555c00700 1 --2- 172.21.15.62:0/3167852402 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff5500737b0 0x7ff550073c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:18.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:18.743+0000 7ff555c00700 1 --2- 172.21.15.62:0/3167852402 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff550074d80 0x7ff5500731e0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:18.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:18.743+0000 7ff555c00700 1 -- 172.21.15.62:0/3167852402 >> 172.21.15.62:0/3167852402 conn(0x7ff5500fb7f0 msgr2=0x7ff5500fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:27:18.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:18.743+0000 7ff555c00700 1 -- 172.21.15.62:0/3167852402 shutdown_connections 2024-04-20T21:27:18.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:18.743+0000 7ff555c00700 1 -- 172.21.15.62:0/3167852402 wait complete. 2024-04-20T21:27:18.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:18.744+0000 7ff555c00700 1 Processor -- start 2024-04-20T21:27:18.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:18.744+0000 7ff555c00700 1 -- start start 2024-04-20T21:27:18.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:18.745+0000 7ff555c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff5500737b0 0x7ff55019c060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:27:18.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:18.745+0000 7ff555c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff550074d80 0x7ff55019c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:27:18.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:18.745+0000 7ff555c00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff55019cbc0 con 0x7ff5500737b0 2024-04-20T21:27:18.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:18.745+0000 7ff555c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff55019cd00 con 0x7ff550074d80 2024-04-20T21:27:18.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:18.745+0000 7ff54ea00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff5500737b0 0x7ff55019c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:27:18.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:18.745+0000 7ff54ea00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff5500737b0 0x7ff55019c060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40306/0 (socket says 172.21.15.62:40306) 2024-04-20T21:27:18.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:18.745+0000 7ff54ea00700 1 -- 172.21.15.62:0/3463255377 learned_addr learned my addr 172.21.15.62:0/3463255377 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:27:18.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:18.745+0000 7ff54e000700 1 --2- 172.21.15.62:0/3463255377 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff550074d80 0x7ff55019c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:27:18.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:18.746+0000 7ff54ea00700 1 -- 172.21.15.62:0/3463255377 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff550074d80 msgr2=0x7ff55019c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:27:18.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:18.746+0000 7ff54ea00700 1 --2- 172.21.15.62:0/3463255377 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff550074d80 0x7ff55019c5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:18.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:18.746+0000 7ff54ea00700 1 -- 172.21.15.62:0/3463255377 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5400095f0 con 0x7ff5500737b0 2024-04-20T21:27:18.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:18.746+0000 7ff54ea00700 1 --2- 172.21.15.62:0/3463255377 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff5500737b0 0x7ff55019c060 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7ff540005b40 tx=0x7ff54000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:27:18.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:18.746+0000 7ff547400700 1 -- 172.21.15.62:0/3463255377 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff54001d070 con 0x7ff5500737b0 2024-04-20T21:27:18.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:18.746+0000 7ff547400700 1 -- 172.21.15.62:0/3463255377 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff540022470 con 0x7ff5500737b0 2024-04-20T21:27:18.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:18.746+0000 7ff547400700 1 -- 172.21.15.62:0/3463255377 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff54000f690 con 0x7ff5500737b0 2024-04-20T21:27:18.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:18.746+0000 7ff555c00700 1 -- 172.21.15.62:0/3463255377 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5501a1750 con 0x7ff5500737b0 2024-04-20T21:27:18.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:18.746+0000 7ff555c00700 1 -- 172.21.15.62:0/3463255377 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5501a1c40 con 0x7ff5500737b0 2024-04-20T21:27:18.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:18.748+0000 7ff555c00700 1 -- 172.21.15.62:0/3463255377 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff55010f370 con 0x7ff5500737b0 2024-04-20T21:27:18.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:18.749+0000 7ff547400700 1 -- 172.21.15.62:0/3463255377 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7ff5400225e0 con 0x7ff5500737b0 2024-04-20T21:27:18.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:18.750+0000 7ff547400700 1 --2- 172.21.15.62:0/3463255377 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7ff53406c680 0x7ff53406eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:27:18.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:18.750+0000 7ff547400700 1 -- 172.21.15.62:0/3463255377 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7ff54008cc90 con 0x7ff5500737b0 2024-04-20T21:27:18.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:18.750+0000 7ff54e000700 1 --2- 172.21.15.62:0/3463255377 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7ff53406c680 0x7ff53406eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:27:18.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:18.751+0000 7ff54e000700 1 --2- 172.21.15.62:0/3463255377 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7ff53406c680 0x7ff53406eb30 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ff53c005950 tx=0x7ff53c005cf0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:27:18.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:18.755+0000 7ff547400700 1 -- 172.21.15.62:0/3463255377 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7ff54005b1a0 con 0x7ff5500737b0 2024-04-20T21:27:19.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:19.021+0000 7ff555c00700 1 -- 172.21.15.62:0/3463255377 --> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] -- mgr_command(tid 0: {"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7ff5501a2390 con 0x7ff53406c680 2024-04-20T21:27:19.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:18 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:27:19.284 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:18 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:27:20.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:19 smithi189 ceph-mon[21562]: from='client.14454 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:27:20.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:19 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2024-04-20T21:27:20.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:19 smithi189 ceph-mon[21562]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:27:20.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:19 smithi062 ceph-mon[16221]: from='client.14454 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:27:20.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:19 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2024-04-20T21:27:20.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:19 smithi062 ceph-mon[16221]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:27:21.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:21.060+0000 7ff547400700 1 -- 172.21.15.62:0/3463255377 <== mgr.14215 v2:172.21.15.62:6800/4116451241 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7ff5501a2390 con 0x7ff53406c680 2024-04-20T21:27:21.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:21.064+0000 7ff555c00700 1 -- 172.21.15.62:0/3463255377 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7ff53406c680 msgr2=0x7ff53406eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:27:21.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:21.064+0000 7ff555c00700 1 --2- 172.21.15.62:0/3463255377 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7ff53406c680 0x7ff53406eb30 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ff53c005950 tx=0x7ff53c005cf0 comp rx=0 tx=0).stop 2024-04-20T21:27:21.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:21.064+0000 7ff555c00700 1 -- 172.21.15.62:0/3463255377 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff5500737b0 msgr2=0x7ff55019c060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:27:21.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:21.064+0000 7ff555c00700 1 --2- 172.21.15.62:0/3463255377 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff5500737b0 0x7ff55019c060 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7ff540005b40 tx=0x7ff54000be20 comp rx=0 tx=0).stop 2024-04-20T21:27:21.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:21.064+0000 7ff555c00700 1 -- 172.21.15.62:0/3463255377 shutdown_connections 2024-04-20T21:27:21.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:21.064+0000 7ff555c00700 1 --2- 172.21.15.62:0/3463255377 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7ff53406c680 0x7ff53406eb30 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:21.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:21.064+0000 7ff555c00700 1 --2- 172.21.15.62:0/3463255377 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff5500737b0 0x7ff55019c060 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:21.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:21.064+0000 7ff555c00700 1 --2- 172.21.15.62:0/3463255377 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff550074d80 0x7ff55019c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:21.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:21.064+0000 7ff555c00700 1 -- 172.21.15.62:0/3463255377 >> 172.21.15.62:0/3463255377 conn(0x7ff5500fb7f0 msgr2=0x7ff550101bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:27:21.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:21.064+0000 7ff555c00700 1 -- 172.21.15.62:0/3463255377 shutdown_connections 2024-04-20T21:27:21.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:21.064+0000 7ff555c00700 1 -- 172.21.15.62:0/3463255377 wait complete. 2024-04-20T21:27:21.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:21 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2024-04-20T21:27:21.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:21 smithi062 ceph-mon[16221]: osdmap e41: 6 total, 6 up, 6 in 2024-04-20T21:27:21.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:21 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2024-04-20T21:27:21.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:21 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-mon-smithi062[16216]: 2024-04-20T21:27:21.031+0000 7fa3c7200700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-20T21:27:21.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:21 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2024-04-20T21:27:21.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:21 smithi189 ceph-mon[21562]: osdmap e41: 6 total, 6 up, 6 in 2024-04-20T21:27:21.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:21 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2024-04-20T21:27:21.938 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -- bash -c 'ceph fs dump' 2024-04-20T21:27:22.180 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:22 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2024-04-20T21:27:22.180 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:22 smithi062 ceph-mon[16221]: osdmap e42: 6 total, 6 up, 6 in 2024-04-20T21:27:22.181 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:22 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2024-04-20T21:27:22.181 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:22 smithi062 ceph-mon[16221]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-20T21:27:22.181 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:22 smithi062 ceph-mon[16221]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-20T21:27:22.181 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:22 smithi062 ceph-mon[16221]: osdmap e43: 6 total, 6 up, 6 in 2024-04-20T21:27:22.181 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:22 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2024-04-20T21:27:22.182 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:22 smithi062 ceph-mon[16221]: fsmap cephfs:0 2024-04-20T21:27:22.182 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:22 smithi062 ceph-mon[16221]: Saving service mds.cephfs spec with placement count:4 2024-04-20T21:27:22.182 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:22 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:27:22.182 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:22 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:27:22.182 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:22 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:27:22.183 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:22 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T21:27:22.183 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:22 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:27:22.183 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:22 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi062.ryqblb", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-20T21:27:22.183 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:22 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi062.ryqblb", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-20T21:27:22.184 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:22 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:27:22.184 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:22 smithi062 ceph-mon[16221]: Deploying daemon mds.cephfs.smithi062.ryqblb on smithi062 2024-04-20T21:27:22.184 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:22 smithi062 ceph-mon[16221]: pgmap v220: 65 pgs: 9 creating+peering, 55 unknown, 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:27:22.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:22 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2024-04-20T21:27:22.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:22 smithi189 ceph-mon[21562]: osdmap e42: 6 total, 6 up, 6 in 2024-04-20T21:27:22.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:22 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2024-04-20T21:27:22.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:22 smithi189 ceph-mon[21562]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-20T21:27:22.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:22 smithi189 ceph-mon[21562]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-20T21:27:22.489 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:22 smithi189 ceph-mon[21562]: osdmap e43: 6 total, 6 up, 6 in 2024-04-20T21:27:22.489 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:22 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2024-04-20T21:27:22.489 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:22 smithi189 ceph-mon[21562]: fsmap cephfs:0 2024-04-20T21:27:22.489 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:22 smithi189 ceph-mon[21562]: Saving service mds.cephfs spec with placement count:4 2024-04-20T21:27:22.489 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:22 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:27:22.489 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:22 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:27:22.490 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:22 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:27:22.490 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:22 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T21:27:22.490 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:22 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:27:22.490 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:22 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi062.ryqblb", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-20T21:27:22.490 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:22 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi062.ryqblb", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-20T21:27:22.490 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:22 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:27:22.491 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:22 smithi189 ceph-mon[21562]: Deploying daemon mds.cephfs.smithi062.ryqblb on smithi062 2024-04-20T21:27:22.491 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:22 smithi189 ceph-mon[21562]: pgmap v220: 65 pgs: 9 creating+peering, 55 unknown, 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:27:23.451 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:23 smithi062 ceph-mon[16221]: osdmap e44: 6 total, 6 up, 6 in 2024-04-20T21:27:23.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:23 smithi189 ceph-mon[21562]: osdmap e44: 6 total, 6 up, 6 in 2024-04-20T21:27:23.587 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T21:27:24.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:24 smithi189 ceph-mon[21562]: osdmap e45: 6 total, 6 up, 6 in 2024-04-20T21:27:24.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:24 smithi189 ceph-mon[21562]: pgmap v223: 65 pgs: 9 creating+peering, 19 unknown, 37 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:27:24.430 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:24 smithi062 ceph-mon[16221]: osdmap e45: 6 total, 6 up, 6 in 2024-04-20T21:27:24.430 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:24 smithi062 ceph-mon[16221]: pgmap v223: 65 pgs: 9 creating+peering, 19 unknown, 37 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:27:26.223 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:25 smithi062 ceph-mon[16221]: pgmap v224: 65 pgs: 65 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:27:26.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:25 smithi189 ceph-mon[21562]: pgmap v224: 65 pgs: 65 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:27:26.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:26.404+0000 7fa087c00700 1 -- 172.21.15.62:0/3869491872 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0801036d0 msgr2=0x7fa080103b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:27:26.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:26.404+0000 7fa087c00700 1 --2- 172.21.15.62:0/3869491872 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0801036d0 0x7fa080103b40 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fa06c009910 tx=0x7fa06c009c20 comp rx=0 tx=0).stop 2024-04-20T21:27:26.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:26.404+0000 7fa087c00700 1 -- 172.21.15.62:0/3869491872 shutdown_connections 2024-04-20T21:27:26.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:26.404+0000 7fa087c00700 1 --2- 172.21.15.62:0/3869491872 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0801036d0 0x7fa080103b40 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:26.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:26.404+0000 7fa087c00700 1 --2- 172.21.15.62:0/3869491872 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa080102430 0x7fa080102840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:26.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:26.404+0000 7fa087c00700 1 -- 172.21.15.62:0/3869491872 >> 172.21.15.62:0/3869491872 conn(0x7fa0800fda80 msgr2=0x7fa0800ffeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:27:26.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:26.404+0000 7fa087c00700 1 -- 172.21.15.62:0/3869491872 shutdown_connections 2024-04-20T21:27:26.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:26.404+0000 7fa087c00700 1 -- 172.21.15.62:0/3869491872 wait complete. 2024-04-20T21:27:26.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:26.405+0000 7fa087c00700 1 Processor -- start 2024-04-20T21:27:26.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:26.406+0000 7fa087c00700 1 -- start start 2024-04-20T21:27:26.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:26.406+0000 7fa087c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa080102430 0x7fa080197c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:27:26.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:26.406+0000 7fa087c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0801036d0 0x7fa0801981d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:27:26.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:26.406+0000 7fa087c00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0801987d0 con 0x7fa080102430 2024-04-20T21:27:26.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:26.406+0000 7fa087c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa080198940 con 0x7fa0801036d0 2024-04-20T21:27:26.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:26.406+0000 7fa085200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa080102430 0x7fa080197c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:27:26.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:26.406+0000 7fa085200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa080102430 0x7fa080197c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33960/0 (socket says 172.21.15.62:33960) 2024-04-20T21:27:26.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:26.406+0000 7fa085200700 1 -- 172.21.15.62:0/2876645491 learned_addr learned my addr 172.21.15.62:0/2876645491 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:27:26.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:26.406+0000 7fa07fe00700 1 --2- 172.21.15.62:0/2876645491 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0801036d0 0x7fa0801981d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:27:26.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:26.407+0000 7fa085200700 1 -- 172.21.15.62:0/2876645491 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0801036d0 msgr2=0x7fa0801981d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:27:26.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:26.407+0000 7fa085200700 1 --2- 172.21.15.62:0/2876645491 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0801036d0 0x7fa0801981d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:26.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:26.407+0000 7fa085200700 1 -- 172.21.15.62:0/2876645491 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa06c0095f0 con 0x7fa080102430 2024-04-20T21:27:26.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:26.407+0000 7fa07fe00700 1 --2- 172.21.15.62:0/2876645491 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0801036d0 0x7fa0801981d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-20T21:27:26.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:26.407+0000 7fa085200700 1 --2- 172.21.15.62:0/2876645491 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa080102430 0x7fa080197c90 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fa07400d8d0 tx=0x7fa07400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:27:26.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:26.407+0000 7fa07d600700 1 -- 172.21.15.62:0/2876645491 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa074009940 con 0x7fa080102430 2024-04-20T21:27:26.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:26.407+0000 7fa07d600700 1 -- 172.21.15.62:0/2876645491 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa074010460 con 0x7fa080102430 2024-04-20T21:27:26.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:26.407+0000 7fa087c00700 1 -- 172.21.15.62:0/2876645491 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa08019d3d0 con 0x7fa080102430 2024-04-20T21:27:26.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:26.407+0000 7fa07d600700 1 -- 172.21.15.62:0/2876645491 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa07400f5f0 con 0x7fa080102430 2024-04-20T21:27:26.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:26.407+0000 7fa087c00700 1 -- 172.21.15.62:0/2876645491 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa08019d920 con 0x7fa080102430 2024-04-20T21:27:26.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:26.409+0000 7fa087c00700 1 -- 172.21.15.62:0/2876645491 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa08010af10 con 0x7fa080102430 2024-04-20T21:27:26.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:26.410+0000 7fa07d600700 1 -- 172.21.15.62:0/2876645491 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7fa0740105d0 con 0x7fa080102430 2024-04-20T21:27:26.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:26.411+0000 7fa07d600700 1 --2- 172.21.15.62:0/2876645491 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fa06806c680 0x7fa06806eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:27:26.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:26.411+0000 7fa07d600700 1 -- 172.21.15.62:0/2876645491 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5468+0+0 (secure 0 0 0) 0x7fa07408b7b0 con 0x7fa080102430 2024-04-20T21:27:26.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:26.411+0000 7fa07fe00700 1 --2- 172.21.15.62:0/2876645491 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fa06806c680 0x7fa06806eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:27:26.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:26.412+0000 7fa07fe00700 1 --2- 172.21.15.62:0/2876645491 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fa06806c680 0x7fa06806eb30 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa0801991a0 tx=0x7fa06c00b560 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:27:26.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:26.416+0000 7fa07d600700 1 -- 172.21.15.62:0/2876645491 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fa074055f10 con 0x7fa080102430 2024-04-20T21:27:26.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:26.689+0000 7fa087c00700 1 -- 172.21.15.62:0/2876645491 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fa08019e360 con 0x7fa080102430 2024-04-20T21:27:26.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:26.690+0000 7fa07d600700 1 -- 172.21.15.62:0/2876645491 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 2 v2) v1 ==== 75+0+1093 (secure 0 0 0) 0x7fa074016460 con 0x7fa080102430 2024-04-20T21:27:26.691 INFO:teuthology.orchestra.run.smithi062.stdout:e2 2024-04-20T21:27:26.691 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T21:27:26.691 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T21:27:26.691 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T21:27:26.692 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:27:26.692 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T21:27:26.692 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T21:27:26.692 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 2 2024-04-20T21:27:26.692 INFO:teuthology.orchestra.run.smithi062.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-20T21:27:26.692 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T21:27:26.692 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T21:27:21.032406+0000 2024-04-20T21:27:26.692 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T21:27:26.692 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T21:27:26.692 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T21:27:26.692 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T21:27:26.692 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T21:27:26.692 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T21:27:26.692 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T21:27:26.692 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 0 2024-04-20T21:27:26.693 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T21:27:26.693 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 1 2024-04-20T21:27:26.693 INFO:teuthology.orchestra.run.smithi062.stdout:in 2024-04-20T21:27:26.693 INFO:teuthology.orchestra.run.smithi062.stdout:up {} 2024-04-20T21:27:26.693 INFO:teuthology.orchestra.run.smithi062.stdout:failed 2024-04-20T21:27:26.693 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T21:27:26.693 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T21:27:26.693 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T21:27:26.693 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T21:27:26.693 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data disabled 2024-04-20T21:27:26.693 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T21:27:26.693 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T21:27:26.693 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 0 2024-04-20T21:27:26.694 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:27:26.694 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:27:26.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:26.694+0000 7fa087c00700 1 -- 172.21.15.62:0/2876645491 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fa06806c680 msgr2=0x7fa06806eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:27:26.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:26.694+0000 7fa087c00700 1 --2- 172.21.15.62:0/2876645491 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fa06806c680 0x7fa06806eb30 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa0801991a0 tx=0x7fa06c00b560 comp rx=0 tx=0).stop 2024-04-20T21:27:26.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:26.694+0000 7fa087c00700 1 -- 172.21.15.62:0/2876645491 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa080102430 msgr2=0x7fa080197c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:27:26.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:26.694+0000 7fa087c00700 1 --2- 172.21.15.62:0/2876645491 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa080102430 0x7fa080197c90 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fa07400d8d0 tx=0x7fa07400dc90 comp rx=0 tx=0).stop 2024-04-20T21:27:26.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:26.694+0000 7fa087c00700 1 -- 172.21.15.62:0/2876645491 shutdown_connections 2024-04-20T21:27:26.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:26.694+0000 7fa087c00700 1 --2- 172.21.15.62:0/2876645491 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fa06806c680 0x7fa06806eb30 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:26.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:26.694+0000 7fa087c00700 1 --2- 172.21.15.62:0/2876645491 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa080102430 0x7fa080197c90 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:26.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:26.694+0000 7fa087c00700 1 --2- 172.21.15.62:0/2876645491 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0801036d0 0x7fa0801981d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:26.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:26.694+0000 7fa087c00700 1 -- 172.21.15.62:0/2876645491 >> 172.21.15.62:0/2876645491 conn(0x7fa0800fda80 msgr2=0x7fa080106900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:27:26.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:26.694+0000 7fa087c00700 1 -- 172.21.15.62:0/2876645491 shutdown_connections 2024-04-20T21:27:26.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:26.694+0000 7fa087c00700 1 -- 172.21.15.62:0/2876645491 wait complete. 2024-04-20T21:27:26.695 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 2 2024-04-20T21:27:27.418 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:27 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:27:27.419 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:27 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:27:27.419 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:27 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:27:27.419 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:27 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi189.dvimer", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-20T21:27:27.419 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:27 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi189.dvimer", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-20T21:27:27.419 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:27 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:27:27.419 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:27 smithi189 ceph-mon[21562]: Deploying daemon mds.cephfs.smithi189.dvimer on smithi189 2024-04-20T21:27:27.419 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:27 smithi189 ceph-mon[21562]: from='client.? 172.21.15.62:0/2876645491' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T21:27:27.419 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:27 smithi189 ceph-mon[21562]: daemon mds.cephfs.smithi062.ryqblb assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2024-04-20T21:27:27.419 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:27 smithi189 ceph-mon[21562]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-20T21:27:27.419 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:27 smithi189 ceph-mon[21562]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-20T21:27:27.419 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:27 smithi189 ceph-mon[21562]: Cluster is now healthy 2024-04-20T21:27:27.420 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:27 smithi189 ceph-mon[21562]: mds.? [v2:172.21.15.62:6826/1147127123,v1:172.21.15.62:6827/1147127123] up:boot 2024-04-20T21:27:27.420 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:27 smithi189 ceph-mon[21562]: fsmap cephfs:1 {0=cephfs.smithi062.ryqblb=up:creating} 2024-04-20T21:27:27.420 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:27 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi062.ryqblb"}]: dispatch 2024-04-20T21:27:27.420 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:27 smithi189 ceph-mon[21562]: daemon mds.cephfs.smithi062.ryqblb is now active in filesystem cephfs as rank 0 2024-04-20T21:27:27.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:27 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:27:27.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:27 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:27:27.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:27 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:27:27.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:27 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi189.dvimer", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-20T21:27:27.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:27 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi189.dvimer", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-20T21:27:27.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:27 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:27:27.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:27 smithi062 ceph-mon[16221]: Deploying daemon mds.cephfs.smithi189.dvimer on smithi189 2024-04-20T21:27:27.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:27 smithi062 ceph-mon[16221]: from='client.? 172.21.15.62:0/2876645491' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T21:27:27.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:27 smithi062 ceph-mon[16221]: daemon mds.cephfs.smithi062.ryqblb assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2024-04-20T21:27:27.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:27 smithi062 ceph-mon[16221]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-20T21:27:27.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:27 smithi062 ceph-mon[16221]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-20T21:27:27.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:27 smithi062 ceph-mon[16221]: Cluster is now healthy 2024-04-20T21:27:27.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:27 smithi062 ceph-mon[16221]: mds.? [v2:172.21.15.62:6826/1147127123,v1:172.21.15.62:6827/1147127123] up:boot 2024-04-20T21:27:27.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:27 smithi062 ceph-mon[16221]: fsmap cephfs:1 {0=cephfs.smithi062.ryqblb=up:creating} 2024-04-20T21:27:27.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:27 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi062.ryqblb"}]: dispatch 2024-04-20T21:27:27.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:27 smithi062 ceph-mon[16221]: daemon mds.cephfs.smithi062.ryqblb is now active in filesystem cephfs as rank 0 2024-04-20T21:27:27.588 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-20T21:27:27.599 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi062.front.sepia.ceph.com 2024-04-20T21:27:27.599 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -- bash -c 'ceph fs set cephfs max_mds 2' 2024-04-20T21:27:28.025 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T21:27:28.933 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:28 smithi189 ceph-mon[21562]: mds.? [v2:172.21.15.62:6826/1147127123,v1:172.21.15.62:6827/1147127123] up:active 2024-04-20T21:27:28.933 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:28 smithi189 ceph-mon[21562]: fsmap cephfs:1 {0=cephfs.smithi062.ryqblb=up:active} 2024-04-20T21:27:28.933 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:28 smithi189 ceph-mon[21562]: pgmap v225: 65 pgs: 65 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 149 B/s wr, 0 op/s 2024-04-20T21:27:29.072 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:28 smithi062 ceph-mon[16221]: mds.? [v2:172.21.15.62:6826/1147127123,v1:172.21.15.62:6827/1147127123] up:active 2024-04-20T21:27:29.072 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:28 smithi062 ceph-mon[16221]: fsmap cephfs:1 {0=cephfs.smithi062.ryqblb=up:active} 2024-04-20T21:27:29.072 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:28 smithi062 ceph-mon[16221]: pgmap v225: 65 pgs: 65 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 149 B/s wr, 0 op/s 2024-04-20T21:27:30.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:30.297+0000 7f39be800700 1 -- 172.21.15.62:0/1360776837 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39b8101440 msgr2=0x7f39b8101890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:27:30.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:30.297+0000 7f39be800700 1 --2- 172.21.15.62:0/1360776837 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39b8101440 0x7f39b8101890 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f39a0009910 tx=0x7f39a0009c20 comp rx=0 tx=0).stop 2024-04-20T21:27:30.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:30.298+0000 7f39be800700 1 -- 172.21.15.62:0/1360776837 shutdown_connections 2024-04-20T21:27:30.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:30.298+0000 7f39be800700 1 --2- 172.21.15.62:0/1360776837 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39b8101440 0x7f39b8101890 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:30.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:30.298+0000 7f39be800700 1 --2- 172.21.15.62:0/1360776837 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39b8100240 0x7f39b8100650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:30.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:30.298+0000 7f39be800700 1 -- 172.21.15.62:0/1360776837 >> 172.21.15.62:0/1360776837 conn(0x7f39b80fb7f0 msgr2=0x7f39b80fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:27:30.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:30.298+0000 7f39be800700 1 -- 172.21.15.62:0/1360776837 shutdown_connections 2024-04-20T21:27:30.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:30.298+0000 7f39be800700 1 -- 172.21.15.62:0/1360776837 wait complete. 2024-04-20T21:27:30.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:30.299+0000 7f39be800700 1 Processor -- start 2024-04-20T21:27:30.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:30.299+0000 7f39be800700 1 -- start start 2024-04-20T21:27:30.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:30.299+0000 7f39be800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39b8100240 0x7f39b8197c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:27:30.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:30.299+0000 7f39be800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39b8101440 0x7f39b81981d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:27:30.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:30.299+0000 7f39be800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39b81987d0 con 0x7f39b8101440 2024-04-20T21:27:30.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:30.299+0000 7f39be800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39b8198940 con 0x7f39b8100240 2024-04-20T21:27:30.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:30.300+0000 7f39aea00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39b8101440 0x7f39b81981d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:27:30.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:30.300+0000 7f39aea00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39b8101440 0x7f39b81981d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33980/0 (socket says 172.21.15.62:33980) 2024-04-20T21:27:30.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:30.300+0000 7f39aea00700 1 -- 172.21.15.62:0/3915488221 learned_addr learned my addr 172.21.15.62:0/3915488221 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:27:30.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:30.300+0000 7f39b7400700 1 --2- 172.21.15.62:0/3915488221 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39b8100240 0x7f39b8197c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:27:30.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:30.300+0000 7f39aea00700 1 -- 172.21.15.62:0/3915488221 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39b8100240 msgr2=0x7f39b8197c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:27:30.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:30.300+0000 7f39aea00700 1 --2- 172.21.15.62:0/3915488221 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39b8100240 0x7f39b8197c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:30.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:30.300+0000 7f39aea00700 1 -- 172.21.15.62:0/3915488221 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39a00095f0 con 0x7f39b8101440 2024-04-20T21:27:30.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:30.301+0000 7f39aea00700 1 --2- 172.21.15.62:0/3915488221 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39b8101440 0x7f39b81981d0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f39a0006010 tx=0x7f39a000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:27:30.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:30.301+0000 7f39b4c00700 1 -- 172.21.15.62:0/3915488221 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f39a001d070 con 0x7f39b8101440 2024-04-20T21:27:30.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:30.301+0000 7f39b4c00700 1 -- 172.21.15.62:0/3915488221 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f39a000f460 con 0x7f39b8101440 2024-04-20T21:27:30.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:30.301+0000 7f39b4c00700 1 -- 172.21.15.62:0/3915488221 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f39a0021640 con 0x7f39b8101440 2024-04-20T21:27:30.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:30.301+0000 7f39be800700 1 -- 172.21.15.62:0/3915488221 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39b819d370 con 0x7f39b8101440 2024-04-20T21:27:30.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:30.301+0000 7f39be800700 1 -- 172.21.15.62:0/3915488221 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39b819d860 con 0x7f39b8101440 2024-04-20T21:27:30.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:30.303+0000 7f39be800700 1 -- 172.21.15.62:0/3915488221 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39b8105520 con 0x7f39b8101440 2024-04-20T21:27:30.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:30.304+0000 7f39b4c00700 1 -- 172.21.15.62:0/3915488221 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f39a000fab0 con 0x7f39b8101440 2024-04-20T21:27:30.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:30.305+0000 7f39b4c00700 1 --2- 172.21.15.62:0/3915488221 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f399c06c680 0x7f399c06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:27:30.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:30.305+0000 7f39b4c00700 1 -- 172.21.15.62:0/3915488221 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5468+0+0 (secure 0 0 0) 0x7f39a008cad0 con 0x7f39b8101440 2024-04-20T21:27:30.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:30.305+0000 7f39b7400700 1 --2- 172.21.15.62:0/3915488221 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f399c06c680 0x7f399c06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:27:30.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:30.306+0000 7f39b7400700 1 --2- 172.21.15.62:0/3915488221 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f399c06c680 0x7f399c06eb30 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f39b8198b70 tx=0x7f39a800b410 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:27:30.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:30.310+0000 7f39b4c00700 1 -- 172.21.15.62:0/3915488221 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f39a0057190 con 0x7f39b8101440 2024-04-20T21:27:30.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:30.594+0000 7f39be800700 1 -- 172.21.15.62:0/3915488221 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"} v 0) v1 -- 0x7f39b8061a00 con 0x7f39b8101440 2024-04-20T21:27:30.859 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:30 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:27:30.859 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:30 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:27:30.859 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:30 smithi062 ceph-mon[16221]: mds.? [v2:172.21.15.189:6824/237262514,v1:172.21.15.189:6825/237262514] up:boot 2024-04-20T21:27:30.860 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:30 smithi062 ceph-mon[16221]: fsmap cephfs:1 {0=cephfs.smithi062.ryqblb=up:active} 1 up:standby 2024-04-20T21:27:30.860 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:30 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi189.dvimer"}]: dispatch 2024-04-20T21:27:30.860 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:30 smithi062 ceph-mon[16221]: fsmap cephfs:1 {0=cephfs.smithi062.ryqblb=up:active} 1 up:standby 2024-04-20T21:27:30.860 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:30 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:27:30.860 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:30 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi062.ozizwi", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-20T21:27:30.860 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:30 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi062.ozizwi", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-20T21:27:30.860 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:30 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:27:30.860 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:30 smithi062 ceph-mon[16221]: Deploying daemon mds.cephfs.smithi062.ozizwi on smithi062 2024-04-20T21:27:30.860 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:30 smithi062 ceph-mon[16221]: pgmap v226: 65 pgs: 65 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.4 KiB/s wr, 3 op/s 2024-04-20T21:27:30.860 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:30 smithi062 ceph-mon[16221]: from='client.? 172.21.15.62:0/3915488221' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]: dispatch 2024-04-20T21:27:30.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:30.868+0000 7f39b4c00700 1 -- 172.21.15.62:0/3915488221 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]=0 v7) v1 ==== 105+0+0 (secure 0 0 0) 0x7f39a005a7b0 con 0x7f39b8101440 2024-04-20T21:27:30.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:30.873+0000 7f39be800700 1 -- 172.21.15.62:0/3915488221 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f399c06c680 msgr2=0x7f399c06eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:27:30.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:30.873+0000 7f39be800700 1 --2- 172.21.15.62:0/3915488221 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f399c06c680 0x7f399c06eb30 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f39b8198b70 tx=0x7f39a800b410 comp rx=0 tx=0).stop 2024-04-20T21:27:30.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:30.873+0000 7f39be800700 1 -- 172.21.15.62:0/3915488221 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39b8101440 msgr2=0x7f39b81981d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:27:30.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:30.873+0000 7f39be800700 1 --2- 172.21.15.62:0/3915488221 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39b8101440 0x7f39b81981d0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f39a0006010 tx=0x7f39a000bd70 comp rx=0 tx=0).stop 2024-04-20T21:27:30.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:30.874+0000 7f39be800700 1 -- 172.21.15.62:0/3915488221 shutdown_connections 2024-04-20T21:27:30.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:30.874+0000 7f39be800700 1 --2- 172.21.15.62:0/3915488221 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f399c06c680 0x7f399c06eb30 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:30.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:30.874+0000 7f39be800700 1 --2- 172.21.15.62:0/3915488221 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39b8100240 0x7f39b8197c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:30.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:30.874+0000 7f39be800700 1 --2- 172.21.15.62:0/3915488221 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39b8101440 0x7f39b81981d0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:30.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:30.874+0000 7f39be800700 1 -- 172.21.15.62:0/3915488221 >> 172.21.15.62:0/3915488221 conn(0x7f39b80fb7f0 msgr2=0x7f39b8102660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:27:30.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:30.874+0000 7f39be800700 1 -- 172.21.15.62:0/3915488221 shutdown_connections 2024-04-20T21:27:30.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:30.875+0000 7f39be800700 1 -- 172.21.15.62:0/3915488221 wait complete. 2024-04-20T21:27:31.047 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:30 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:27:31.047 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:30 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:27:31.048 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:30 smithi189 ceph-mon[21562]: mds.? [v2:172.21.15.189:6824/237262514,v1:172.21.15.189:6825/237262514] up:boot 2024-04-20T21:27:31.048 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:30 smithi189 ceph-mon[21562]: fsmap cephfs:1 {0=cephfs.smithi062.ryqblb=up:active} 1 up:standby 2024-04-20T21:27:31.048 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:30 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi189.dvimer"}]: dispatch 2024-04-20T21:27:31.048 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:30 smithi189 ceph-mon[21562]: fsmap cephfs:1 {0=cephfs.smithi062.ryqblb=up:active} 1 up:standby 2024-04-20T21:27:31.048 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:30 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:27:31.048 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:30 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi062.ozizwi", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-20T21:27:31.048 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:30 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi062.ozizwi", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-20T21:27:31.048 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:30 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:27:31.048 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:30 smithi189 ceph-mon[21562]: Deploying daemon mds.cephfs.smithi062.ozizwi on smithi062 2024-04-20T21:27:31.048 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:30 smithi189 ceph-mon[21562]: pgmap v226: 65 pgs: 65 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.4 KiB/s wr, 3 op/s 2024-04-20T21:27:31.048 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:30 smithi189 ceph-mon[21562]: from='client.? 172.21.15.62:0/3915488221' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]: dispatch 2024-04-20T21:27:32.013 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-20T21:27:32.024 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi062.front.sepia.ceph.com 2024-04-20T21:27:32.024 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -- bash -c 'ceph fs set cephfs allow_standby_replay true' 2024-04-20T21:27:32.095 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:31 smithi062 ceph-mon[16221]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-20T21:27:32.096 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:31 smithi062 ceph-mon[16221]: from='client.? 172.21.15.62:0/3915488221' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]': finished 2024-04-20T21:27:32.096 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:31 smithi062 ceph-mon[16221]: daemon mds.cephfs.smithi189.dvimer assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-20T21:27:32.096 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:31 smithi062 ceph-mon[16221]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-04-20T21:27:32.096 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:31 smithi062 ceph-mon[16221]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-20T21:27:32.096 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:31 smithi062 ceph-mon[16221]: fsmap cephfs:1 {0=cephfs.smithi062.ryqblb=up:active} 1 up:standby 2024-04-20T21:27:32.096 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:31 smithi062 ceph-mon[16221]: fsmap cephfs:2 {0=cephfs.smithi062.ryqblb=up:active,1=cephfs.smithi189.dvimer=up:creating} 2024-04-20T21:27:32.096 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:31 smithi062 ceph-mon[16221]: daemon mds.cephfs.smithi189.dvimer is now active in filesystem cephfs as rank 1 2024-04-20T21:27:32.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:31 smithi189 ceph-mon[21562]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-20T21:27:32.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:31 smithi189 ceph-mon[21562]: from='client.? 172.21.15.62:0/3915488221' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]': finished 2024-04-20T21:27:32.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:31 smithi189 ceph-mon[21562]: daemon mds.cephfs.smithi189.dvimer assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-20T21:27:32.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:31 smithi189 ceph-mon[21562]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-04-20T21:27:32.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:31 smithi189 ceph-mon[21562]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-20T21:27:32.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:31 smithi189 ceph-mon[21562]: fsmap cephfs:1 {0=cephfs.smithi062.ryqblb=up:active} 1 up:standby 2024-04-20T21:27:32.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:31 smithi189 ceph-mon[21562]: fsmap cephfs:2 {0=cephfs.smithi062.ryqblb=up:active,1=cephfs.smithi189.dvimer=up:creating} 2024-04-20T21:27:32.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:31 smithi189 ceph-mon[21562]: daemon mds.cephfs.smithi189.dvimer is now active in filesystem cephfs as rank 1 2024-04-20T21:27:32.679 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T21:27:33.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:32 smithi189 ceph-mon[21562]: mds.? [v2:172.21.15.189:6824/237262514,v1:172.21.15.189:6825/237262514] up:active 2024-04-20T21:27:33.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:32 smithi189 ceph-mon[21562]: fsmap cephfs:2 {0=cephfs.smithi062.ryqblb=up:active,1=cephfs.smithi189.dvimer=up:active} 2024-04-20T21:27:33.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:32 smithi189 ceph-mon[21562]: pgmap v227: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.4 KiB/s wr, 4 op/s 2024-04-20T21:27:33.279 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:32 smithi062 ceph-mon[16221]: mds.? [v2:172.21.15.189:6824/237262514,v1:172.21.15.189:6825/237262514] up:active 2024-04-20T21:27:33.280 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:32 smithi062 ceph-mon[16221]: fsmap cephfs:2 {0=cephfs.smithi062.ryqblb=up:active,1=cephfs.smithi189.dvimer=up:active} 2024-04-20T21:27:33.280 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:32 smithi062 ceph-mon[16221]: pgmap v227: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.4 KiB/s wr, 4 op/s 2024-04-20T21:27:35.172 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:34 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:27:35.173 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:34 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:27:35.173 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:34 smithi062 ceph-mon[16221]: pgmap v228: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s wr, 5 op/s 2024-04-20T21:27:35.173 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:34 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:27:35.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:34 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:27:35.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:34 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:27:35.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:34 smithi189 ceph-mon[21562]: pgmap v228: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s wr, 5 op/s 2024-04-20T21:27:35.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:34 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:27:35.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:35.555+0000 7fca51200700 1 -- 172.21.15.62:0/3593548438 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fca4c100cd0 msgr2=0x7fca4c1030b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:27:35.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:35.555+0000 7fca51200700 1 --2- 172.21.15.62:0/3593548438 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fca4c100cd0 0x7fca4c1030b0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fca3c009910 tx=0x7fca3c009c20 comp rx=0 tx=0).stop 2024-04-20T21:27:35.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:35.555+0000 7fca51200700 1 -- 172.21.15.62:0/3593548438 shutdown_connections 2024-04-20T21:27:35.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:35.555+0000 7fca51200700 1 --2- 172.21.15.62:0/3593548438 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca4c1035f0 0x7fca4c1059d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:35.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:35.555+0000 7fca51200700 1 --2- 172.21.15.62:0/3593548438 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fca4c100cd0 0x7fca4c1030b0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:35.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:35.555+0000 7fca51200700 1 -- 172.21.15.62:0/3593548438 >> 172.21.15.62:0/3593548438 conn(0x7fca4c0fa6d0 msgr2=0x7fca4c0fcb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:27:35.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:35.555+0000 7fca51200700 1 -- 172.21.15.62:0/3593548438 shutdown_connections 2024-04-20T21:27:35.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:35.555+0000 7fca51200700 1 -- 172.21.15.62:0/3593548438 wait complete. 2024-04-20T21:27:35.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:35.556+0000 7fca51200700 1 Processor -- start 2024-04-20T21:27:35.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:35.556+0000 7fca51200700 1 -- start start 2024-04-20T21:27:35.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:35.557+0000 7fca51200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fca4c100cd0 0x7fca4c197c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:27:35.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:35.557+0000 7fca51200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca4c1035f0 0x7fca4c1981b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:27:35.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:35.557+0000 7fca51200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca4c1987d0 con 0x7fca4c100cd0 2024-04-20T21:27:35.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:35.557+0000 7fca51200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca4c19d1e0 con 0x7fca4c1035f0 2024-04-20T21:27:35.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:35.557+0000 7fca49e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fca4c100cd0 0x7fca4c197c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:27:35.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:35.557+0000 7fca49e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fca4c100cd0 0x7fca4c197c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42292/0 (socket says 172.21.15.62:42292) 2024-04-20T21:27:35.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:35.557+0000 7fca49e00700 1 -- 172.21.15.62:0/2638748956 learned_addr learned my addr 172.21.15.62:0/2638748956 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:27:35.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:35.557+0000 7fca49400700 1 --2- 172.21.15.62:0/2638748956 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca4c1035f0 0x7fca4c1981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:27:35.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:35.557+0000 7fca49e00700 1 -- 172.21.15.62:0/2638748956 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca4c1035f0 msgr2=0x7fca4c1981b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:27:35.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:35.557+0000 7fca49e00700 1 --2- 172.21.15.62:0/2638748956 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca4c1035f0 0x7fca4c1981b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:35.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:35.557+0000 7fca49e00700 1 -- 172.21.15.62:0/2638748956 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca3c0095f0 con 0x7fca4c100cd0 2024-04-20T21:27:35.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:35.558+0000 7fca49e00700 1 --2- 172.21.15.62:0/2638748956 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fca4c100cd0 0x7fca4c197c70 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fca3c000c00 tx=0x7fca3c00f740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:27:35.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:35.558+0000 7fca42a00700 1 -- 172.21.15.62:0/2638748956 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fca3c01d070 con 0x7fca4c100cd0 2024-04-20T21:27:35.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:35.558+0000 7fca42a00700 1 -- 172.21.15.62:0/2638748956 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fca3c005dc0 con 0x7fca4c100cd0 2024-04-20T21:27:35.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:35.558+0000 7fca42a00700 1 -- 172.21.15.62:0/2638748956 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fca3c0226a0 con 0x7fca4c100cd0 2024-04-20T21:27:35.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:35.558+0000 7fca51200700 1 -- 172.21.15.62:0/2638748956 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca4c19d380 con 0x7fca4c100cd0 2024-04-20T21:27:35.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:35.558+0000 7fca51200700 1 -- 172.21.15.62:0/2638748956 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca4c19d840 con 0x7fca4c100cd0 2024-04-20T21:27:35.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:35.560+0000 7fca51200700 1 -- 172.21.15.62:0/2638748956 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca4c0fc2b0 con 0x7fca4c100cd0 2024-04-20T21:27:35.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:35.561+0000 7fca42a00700 1 -- 172.21.15.62:0/2638748956 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7fca3c003680 con 0x7fca4c100cd0 2024-04-20T21:27:35.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:35.562+0000 7fca42a00700 1 --2- 172.21.15.62:0/2638748956 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fca3006c680 0x7fca3006eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:27:35.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:35.562+0000 7fca42a00700 1 -- 172.21.15.62:0/2638748956 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5468+0+0 (secure 0 0 0) 0x7fca3c08cd20 con 0x7fca4c100cd0 2024-04-20T21:27:35.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:35.562+0000 7fca49400700 1 --2- 172.21.15.62:0/2638748956 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fca3006c680 0x7fca3006eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:27:35.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:35.563+0000 7fca49400700 1 --2- 172.21.15.62:0/2638748956 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fca3006c680 0x7fca3006eb30 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fca38009bf0 tx=0x7fca38009260 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:27:35.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:35.567+0000 7fca42a00700 1 -- 172.21.15.62:0/2638748956 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fca3c057480 con 0x7fca4c100cd0 2024-04-20T21:27:35.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:35.838+0000 7fca51200700 1 -- 172.21.15.62:0/2638748956 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7fca4c02ce90 con 0x7fca4c100cd0 2024-04-20T21:27:35.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:35.887+0000 7fca42a00700 1 -- 172.21.15.62:0/2638748956 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v10) v1 ==== 121+0+0 (secure 0 0 0) 0x7fca3c003930 con 0x7fca4c100cd0 2024-04-20T21:27:35.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:35.891+0000 7fca51200700 1 -- 172.21.15.62:0/2638748956 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fca3006c680 msgr2=0x7fca3006eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:27:35.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:35.891+0000 7fca51200700 1 --2- 172.21.15.62:0/2638748956 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fca3006c680 0x7fca3006eb30 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fca38009bf0 tx=0x7fca38009260 comp rx=0 tx=0).stop 2024-04-20T21:27:35.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:35.891+0000 7fca51200700 1 -- 172.21.15.62:0/2638748956 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fca4c100cd0 msgr2=0x7fca4c197c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:27:35.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:35.891+0000 7fca51200700 1 --2- 172.21.15.62:0/2638748956 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fca4c100cd0 0x7fca4c197c70 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fca3c000c00 tx=0x7fca3c00f740 comp rx=0 tx=0).stop 2024-04-20T21:27:35.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:35.892+0000 7fca51200700 1 -- 172.21.15.62:0/2638748956 shutdown_connections 2024-04-20T21:27:35.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:35.892+0000 7fca51200700 1 --2- 172.21.15.62:0/2638748956 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fca3006c680 0x7fca3006eb30 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:35.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:35.892+0000 7fca51200700 1 --2- 172.21.15.62:0/2638748956 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fca4c100cd0 0x7fca4c197c70 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:35.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:35.892+0000 7fca51200700 1 --2- 172.21.15.62:0/2638748956 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca4c1035f0 0x7fca4c1981b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:35.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:35.892+0000 7fca51200700 1 -- 172.21.15.62:0/2638748956 >> 172.21.15.62:0/2638748956 conn(0x7fca4c0fa6d0 msgr2=0x7fca4c0ff350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:27:35.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:35.892+0000 7fca51200700 1 -- 172.21.15.62:0/2638748956 shutdown_connections 2024-04-20T21:27:35.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:35.892+0000 7fca51200700 1 -- 172.21.15.62:0/2638748956 wait complete. 2024-04-20T21:27:36.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:36 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:27:36.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:36 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:27:36.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:36 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:27:36.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:36 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi189.xbgspw", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-20T21:27:36.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:36 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi189.xbgspw", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-20T21:27:36.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:36 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:27:36.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:36 smithi062 ceph-mon[16221]: Deploying daemon mds.cephfs.smithi189.xbgspw on smithi189 2024-04-20T21:27:36.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:36 smithi062 ceph-mon[16221]: from='client.? 172.21.15.62:0/2638748956' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-20T21:27:36.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:36 smithi062 ceph-mon[16221]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-04-20T21:27:36.537 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:36 smithi062 ceph-mon[16221]: Cluster is now healthy 2024-04-20T21:27:36.537 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:36 smithi062 ceph-mon[16221]: mds.? [v2:172.21.15.62:6828/3494174293,v1:172.21.15.62:6829/3494174293] up:boot 2024-04-20T21:27:36.537 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:36 smithi062 ceph-mon[16221]: from='client.? 172.21.15.62:0/2638748956' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-20T21:27:36.537 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:36 smithi062 ceph-mon[16221]: fsmap cephfs:2 {0=cephfs.smithi062.ryqblb=up:active,1=cephfs.smithi189.dvimer=up:active} 1 up:standby 2024-04-20T21:27:36.537 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:36 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi062.ozizwi"}]: dispatch 2024-04-20T21:27:36.537 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:36 smithi062 ceph-mon[16221]: pgmap v229: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.2 KiB/s wr, 6 op/s 2024-04-20T21:27:36.537 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:36 smithi062 ceph-mon[16221]: fsmap cephfs:2 {0=cephfs.smithi062.ryqblb=up:active,1=cephfs.smithi189.dvimer=up:active} 1 up:standby-replay 2024-04-20T21:27:36.600 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:36 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:27:36.600 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:36 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:27:36.600 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:36 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:27:36.600 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:36 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi189.xbgspw", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-20T21:27:36.600 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:36 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi189.xbgspw", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-20T21:27:36.600 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:36 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:27:36.600 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:36 smithi189 ceph-mon[21562]: Deploying daemon mds.cephfs.smithi189.xbgspw on smithi189 2024-04-20T21:27:36.600 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:36 smithi189 ceph-mon[21562]: from='client.? 172.21.15.62:0/2638748956' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-20T21:27:36.601 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:36 smithi189 ceph-mon[21562]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-04-20T21:27:36.601 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:36 smithi189 ceph-mon[21562]: Cluster is now healthy 2024-04-20T21:27:36.601 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:36 smithi189 ceph-mon[21562]: mds.? [v2:172.21.15.62:6828/3494174293,v1:172.21.15.62:6829/3494174293] up:boot 2024-04-20T21:27:36.601 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:36 smithi189 ceph-mon[21562]: from='client.? 172.21.15.62:0/2638748956' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-20T21:27:36.601 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:36 smithi189 ceph-mon[21562]: fsmap cephfs:2 {0=cephfs.smithi062.ryqblb=up:active,1=cephfs.smithi189.dvimer=up:active} 1 up:standby 2024-04-20T21:27:36.601 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:36 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi062.ozizwi"}]: dispatch 2024-04-20T21:27:36.601 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:36 smithi189 ceph-mon[21562]: pgmap v229: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.2 KiB/s wr, 6 op/s 2024-04-20T21:27:36.601 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:36 smithi189 ceph-mon[21562]: fsmap cephfs:2 {0=cephfs.smithi062.ryqblb=up:active,1=cephfs.smithi189.dvimer=up:active} 1 up:standby-replay 2024-04-20T21:27:36.748 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-20T21:27:36.762 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi062.front.sepia.ceph.com 2024-04-20T21:27:36.762 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -- bash -c 'ceph fs set cephfs inline_data true --yes-i-really-really-mean-it' 2024-04-20T21:27:37.188 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T21:27:39.170 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:38 smithi062 ceph-mon[16221]: pgmap v230: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 85 B/s rd, 2.2 KiB/s wr, 7 op/s 2024-04-20T21:27:39.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:38 smithi189 ceph-mon[21562]: pgmap v230: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 85 B/s rd, 2.2 KiB/s wr, 7 op/s 2024-04-20T21:27:39.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:39.538+0000 7f6748800700 1 -- 172.21.15.62:0/63984748 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f67400737b0 msgr2=0x7f6740073c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:27:39.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:39.538+0000 7f6748800700 1 --2- 172.21.15.62:0/63984748 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f67400737b0 0x7f6740073c20 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f6730009940 tx=0x7f6730009c50 comp rx=0 tx=0).stop 2024-04-20T21:27:39.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:39.538+0000 7f6748800700 1 -- 172.21.15.62:0/63984748 shutdown_connections 2024-04-20T21:27:39.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:39.538+0000 7f6748800700 1 --2- 172.21.15.62:0/63984748 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f67400737b0 0x7f6740073c20 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:39.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:39.538+0000 7f6748800700 1 --2- 172.21.15.62:0/63984748 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6740074d80 0x7f67400731e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:39.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:39.538+0000 7f6748800700 1 -- 172.21.15.62:0/63984748 >> 172.21.15.62:0/63984748 conn(0x7f67400fb7f0 msgr2=0x7f67400fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:27:39.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:39.538+0000 7f6748800700 1 -- 172.21.15.62:0/63984748 shutdown_connections 2024-04-20T21:27:39.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:39.538+0000 7f6748800700 1 -- 172.21.15.62:0/63984748 wait complete. 2024-04-20T21:27:39.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:39.539+0000 7f6748800700 1 Processor -- start 2024-04-20T21:27:39.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:39.539+0000 7f6748800700 1 -- start start 2024-04-20T21:27:39.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:39.540+0000 7f6748800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f67400737b0 0x7f6740197ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:27:39.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:39.540+0000 7f6748800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6740074d80 0x7f67401981e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:27:39.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:39.540+0000 7f6748800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67401987e0 con 0x7f67400737b0 2024-04-20T21:27:39.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:39.540+0000 7f6748800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6740198950 con 0x7f6740074d80 2024-04-20T21:27:39.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:39.540+0000 7f6745e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f67400737b0 0x7f6740197ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:27:39.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:39.540+0000 7f6745e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f67400737b0 0x7f6740197ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42306/0 (socket says 172.21.15.62:42306) 2024-04-20T21:27:39.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:39.540+0000 7f6745e00700 1 -- 172.21.15.62:0/3424111312 learned_addr learned my addr 172.21.15.62:0/3424111312 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:27:39.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:39.540+0000 7f6745400700 1 --2- 172.21.15.62:0/3424111312 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6740074d80 0x7f67401981e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:27:39.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:39.540+0000 7f6745e00700 1 -- 172.21.15.62:0/3424111312 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6740074d80 msgr2=0x7f67401981e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:27:39.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:39.541+0000 7f6745e00700 1 --2- 172.21.15.62:0/3424111312 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6740074d80 0x7f67401981e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:39.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:39.541+0000 7f6745e00700 1 -- 172.21.15.62:0/3424111312 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67300095f0 con 0x7f67400737b0 2024-04-20T21:27:39.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:39.541+0000 7f6745e00700 1 --2- 172.21.15.62:0/3424111312 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f67400737b0 0x7f6740197ca0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f673400e920 tx=0x7f673400ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:27:39.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:39.541+0000 7f673aa00700 1 -- 172.21.15.62:0/3424111312 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f673400c9d0 con 0x7f67400737b0 2024-04-20T21:27:39.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:39.541+0000 7f673aa00700 1 -- 172.21.15.62:0/3424111312 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f6734010430 con 0x7f67400737b0 2024-04-20T21:27:39.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:39.541+0000 7f673aa00700 1 -- 172.21.15.62:0/3424111312 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6734018650 con 0x7f67400737b0 2024-04-20T21:27:39.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:39.541+0000 7f6748800700 1 -- 172.21.15.62:0/3424111312 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f674019d3e0 con 0x7f67400737b0 2024-04-20T21:27:39.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:39.541+0000 7f6748800700 1 -- 172.21.15.62:0/3424111312 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f674019d900 con 0x7f67400737b0 2024-04-20T21:27:39.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:39.542+0000 7f6748800700 1 -- 172.21.15.62:0/3424111312 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f674010af80 con 0x7f67400737b0 2024-04-20T21:27:39.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:39.544+0000 7f673aa00700 1 -- 172.21.15.62:0/3424111312 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f67340105a0 con 0x7f67400737b0 2024-04-20T21:27:39.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:39.546+0000 7f673aa00700 1 --2- 172.21.15.62:0/3424111312 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f672806c680 0x7f672806eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:27:39.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:39.546+0000 7f673aa00700 1 -- 172.21.15.62:0/3424111312 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5468+0+0 (secure 0 0 0) 0x7f6734014070 con 0x7f67400737b0 2024-04-20T21:27:39.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:39.546+0000 7f6745400700 1 --2- 172.21.15.62:0/3424111312 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f672806c680 0x7f672806eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:27:39.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:39.547+0000 7f6745400700 1 --2- 172.21.15.62:0/3424111312 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f672806c680 0x7f672806eb30 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f6730000c00 tx=0x7f6730005e80 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:27:39.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:39.551+0000 7f673aa00700 1 -- 172.21.15.62:0/3424111312 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f6734056840 con 0x7f67400737b0 2024-04-20T21:27:39.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:39.825+0000 7f6748800700 1 -- 172.21.15.62:0/3424111312 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f674010b140 con 0x7f67400737b0 2024-04-20T21:27:39.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:39.907+0000 7f673aa00700 1 -- 172.21.15.62:0/3424111312 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]=0 inline data enabled v12) v1 ==== 168+0+0 (secure 0 0 0) 0x7f6734059e60 con 0x7f67400737b0 2024-04-20T21:27:39.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:39.911+0000 7f6748800700 1 -- 172.21.15.62:0/3424111312 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f672806c680 msgr2=0x7f672806eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:27:39.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:39.911+0000 7f6748800700 1 --2- 172.21.15.62:0/3424111312 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f672806c680 0x7f672806eb30 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f6730000c00 tx=0x7f6730005e80 comp rx=0 tx=0).stop 2024-04-20T21:27:39.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:39.911+0000 7f6748800700 1 -- 172.21.15.62:0/3424111312 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f67400737b0 msgr2=0x7f6740197ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:27:39.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:39.911+0000 7f6748800700 1 --2- 172.21.15.62:0/3424111312 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f67400737b0 0x7f6740197ca0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f673400e920 tx=0x7f673400ec30 comp rx=0 tx=0).stop 2024-04-20T21:27:39.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:39.911+0000 7f6748800700 1 -- 172.21.15.62:0/3424111312 shutdown_connections 2024-04-20T21:27:39.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:39.911+0000 7f6748800700 1 --2- 172.21.15.62:0/3424111312 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f672806c680 0x7f672806eb30 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:39.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:39.911+0000 7f6748800700 1 --2- 172.21.15.62:0/3424111312 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f67400737b0 0x7f6740197ca0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:39.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:39.911+0000 7f6748800700 1 --2- 172.21.15.62:0/3424111312 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6740074d80 0x7f67401981e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:39.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:39.911+0000 7f6748800700 1 -- 172.21.15.62:0/3424111312 >> 172.21.15.62:0/3424111312 conn(0x7f67400fb7f0 msgr2=0x7f6740106020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:27:39.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:39.912+0000 7f6748800700 1 -- 172.21.15.62:0/3424111312 shutdown_connections 2024-04-20T21:27:39.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:39.912+0000 7f6748800700 1 -- 172.21.15.62:0/3424111312 wait complete. 2024-04-20T21:27:39.913 INFO:teuthology.orchestra.run.smithi062.stderr:inline data enabled 2024-04-20T21:27:40.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:40 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:27:40.489 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:40 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:27:40.489 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:40 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:27:40.489 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:40 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:27:40.489 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:40 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:27:40.489 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:40 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:27:40.489 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:40 smithi189 ceph-mon[21562]: from='client.? 172.21.15.62:0/3424111312' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-20T21:27:40.489 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:40 smithi189 ceph-mon[21562]: pgmap v231: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 341 B/s rd, 2.1 KiB/s wr, 7 op/s 2024-04-20T21:27:40.489 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:40 smithi189 ceph-mon[21562]: Health check failed: 1 filesystem with deprecated feature inline_data (FS_INLINE_DATA_DEPRECATED) 2024-04-20T21:27:40.489 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:40 smithi189 ceph-mon[21562]: mds.? [v2:172.21.15.189:6826/1495356258,v1:172.21.15.189:6827/1495356258] up:boot 2024-04-20T21:27:40.489 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:40 smithi189 ceph-mon[21562]: from='client.? 172.21.15.62:0/3424111312' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]': finished 2024-04-20T21:27:40.489 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:40 smithi189 ceph-mon[21562]: fsmap cephfs:2 {0=cephfs.smithi062.ryqblb=up:active,1=cephfs.smithi189.dvimer=up:active} 1 up:standby-replay 1 up:standby 2024-04-20T21:27:40.489 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:40 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi189.xbgspw"}]: dispatch 2024-04-20T21:27:40.489 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:40 smithi189 ceph-mon[21562]: fsmap cephfs:2 {0=cephfs.smithi062.ryqblb=up:active,1=cephfs.smithi189.dvimer=up:active} 2 up:standby-replay 2024-04-20T21:27:40.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:40 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:27:40.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:40 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:27:40.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:40 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:27:40.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:40 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:27:40.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:40 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:27:40.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:40 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:27:40.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:40 smithi062 ceph-mon[16221]: from='client.? 172.21.15.62:0/3424111312' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-20T21:27:40.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:40 smithi062 ceph-mon[16221]: pgmap v231: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 341 B/s rd, 2.1 KiB/s wr, 7 op/s 2024-04-20T21:27:40.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:40 smithi062 ceph-mon[16221]: Health check failed: 1 filesystem with deprecated feature inline_data (FS_INLINE_DATA_DEPRECATED) 2024-04-20T21:27:40.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:40 smithi062 ceph-mon[16221]: mds.? [v2:172.21.15.189:6826/1495356258,v1:172.21.15.189:6827/1495356258] up:boot 2024-04-20T21:27:40.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:40 smithi062 ceph-mon[16221]: from='client.? 172.21.15.62:0/3424111312' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]': finished 2024-04-20T21:27:40.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:40 smithi062 ceph-mon[16221]: fsmap cephfs:2 {0=cephfs.smithi062.ryqblb=up:active,1=cephfs.smithi189.dvimer=up:active} 1 up:standby-replay 1 up:standby 2024-04-20T21:27:40.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:40 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi189.xbgspw"}]: dispatch 2024-04-20T21:27:40.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:40 smithi062 ceph-mon[16221]: fsmap cephfs:2 {0=cephfs.smithi062.ryqblb=up:active,1=cephfs.smithi189.dvimer=up:active} 2 up:standby-replay 2024-04-20T21:27:40.706 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-20T21:27:40.723 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi062.front.sepia.ceph.com 2024-04-20T21:27:40.723 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -- bash -c 'ceph fs dump' 2024-04-20T21:27:41.728 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T21:27:43.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:42 smithi062 ceph-mon[16221]: pgmap v232: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-04-20T21:27:43.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:42 smithi189 ceph-mon[21562]: pgmap v232: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-04-20T21:27:44.202 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:43 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:27:44.202 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:43 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:27:44.202 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:43 smithi062 ceph-mon[16221]: mds.? [v2:172.21.15.62:6826/1147127123,v1:172.21.15.62:6827/1147127123] up:active 2024-04-20T21:27:44.202 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:43 smithi062 ceph-mon[16221]: mds.? [v2:172.21.15.189:6824/237262514,v1:172.21.15.189:6825/237262514] up:active 2024-04-20T21:27:44.203 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:43 smithi062 ceph-mon[16221]: mds.? [v2:172.21.15.189:6826/1495356258,v1:172.21.15.189:6827/1495356258] up:standby-replay 2024-04-20T21:27:44.203 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:43 smithi062 ceph-mon[16221]: mds.? [v2:172.21.15.62:6828/3494174293,v1:172.21.15.62:6829/3494174293] up:standby-replay 2024-04-20T21:27:44.203 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:43 smithi062 ceph-mon[16221]: fsmap cephfs:2 {0=cephfs.smithi062.ryqblb=up:active,1=cephfs.smithi189.dvimer=up:active} 2 up:standby-replay 2024-04-20T21:27:44.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:43 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:27:44.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:43 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:27:44.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:43 smithi189 ceph-mon[21562]: mds.? [v2:172.21.15.62:6826/1147127123,v1:172.21.15.62:6827/1147127123] up:active 2024-04-20T21:27:44.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:43 smithi189 ceph-mon[21562]: mds.? [v2:172.21.15.189:6824/237262514,v1:172.21.15.189:6825/237262514] up:active 2024-04-20T21:27:44.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:43 smithi189 ceph-mon[21562]: mds.? [v2:172.21.15.189:6826/1495356258,v1:172.21.15.189:6827/1495356258] up:standby-replay 2024-04-20T21:27:44.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:43 smithi189 ceph-mon[21562]: mds.? [v2:172.21.15.62:6828/3494174293,v1:172.21.15.62:6829/3494174293] up:standby-replay 2024-04-20T21:27:44.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:43 smithi189 ceph-mon[21562]: fsmap cephfs:2 {0=cephfs.smithi062.ryqblb=up:active,1=cephfs.smithi189.dvimer=up:active} 2 up:standby-replay 2024-04-20T21:27:44.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:44.736+0000 7fe195e00700 1 -- 172.21.15.62:0/1753812344 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe190074d20 msgr2=0x7fe190073180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:27:44.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:44.736+0000 7fe195e00700 1 --2- 172.21.15.62:0/1753812344 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe190074d20 0x7fe190073180 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fe180009910 tx=0x7fe180009c20 comp rx=0 tx=0).stop 2024-04-20T21:27:44.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:44.736+0000 7fe195e00700 1 -- 172.21.15.62:0/1753812344 shutdown_connections 2024-04-20T21:27:44.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:44.736+0000 7fe195e00700 1 --2- 172.21.15.62:0/1753812344 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe190073750 0x7fe190073bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:44.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:44.736+0000 7fe195e00700 1 --2- 172.21.15.62:0/1753812344 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe190074d20 0x7fe190073180 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:44.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:44.736+0000 7fe195e00700 1 -- 172.21.15.62:0/1753812344 >> 172.21.15.62:0/1753812344 conn(0x7fe1900fb7f0 msgr2=0x7fe1900fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:27:44.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:44.737+0000 7fe195e00700 1 -- 172.21.15.62:0/1753812344 shutdown_connections 2024-04-20T21:27:44.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:44.737+0000 7fe195e00700 1 -- 172.21.15.62:0/1753812344 wait complete. 2024-04-20T21:27:44.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:44.738+0000 7fe195e00700 1 Processor -- start 2024-04-20T21:27:44.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:44.738+0000 7fe195e00700 1 -- start start 2024-04-20T21:27:44.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:44.738+0000 7fe195e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe190073750 0x7fe190197ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:27:44.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:44.738+0000 7fe195e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe190074d20 0x7fe1901981e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:27:44.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:44.738+0000 7fe195e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe190198750 con 0x7fe190074d20 2024-04-20T21:27:44.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:44.738+0000 7fe195e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1901988c0 con 0x7fe190073750 2024-04-20T21:27:44.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:44.738+0000 7fe18e000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe190074d20 0x7fe1901981e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:27:44.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:44.739+0000 7fe18e000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe190074d20 0x7fe1901981e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60800/0 (socket says 172.21.15.62:60800) 2024-04-20T21:27:44.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:44.739+0000 7fe18e000700 1 -- 172.21.15.62:0/1603173448 learned_addr learned my addr 172.21.15.62:0/1603173448 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:27:44.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:44.739+0000 7fe18ea00700 1 --2- 172.21.15.62:0/1603173448 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe190073750 0x7fe190197ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:27:44.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:44.739+0000 7fe18e000700 1 -- 172.21.15.62:0/1603173448 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe190073750 msgr2=0x7fe190197ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:27:44.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:44.739+0000 7fe18e000700 1 --2- 172.21.15.62:0/1603173448 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe190073750 0x7fe190197ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:44.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:44.739+0000 7fe18e000700 1 -- 172.21.15.62:0/1603173448 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1800095f0 con 0x7fe190074d20 2024-04-20T21:27:44.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:44.739+0000 7fe18ea00700 1 --2- 172.21.15.62:0/1603173448 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe190073750 0x7fe190197ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T21:27:44.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:44.739+0000 7fe18e000700 1 --2- 172.21.15.62:0/1603173448 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe190074d20 0x7fe1901981e0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fe17c00c930 tx=0x7fe17c00ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:27:44.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:44.740+0000 7fe194a00700 1 -- 172.21.15.62:0/1603173448 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe17c007ad0 con 0x7fe190074d20 2024-04-20T21:27:44.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:44.740+0000 7fe194a00700 1 -- 172.21.15.62:0/1603173448 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fe17c007c30 con 0x7fe190074d20 2024-04-20T21:27:44.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:44.740+0000 7fe195e00700 1 -- 172.21.15.62:0/1603173448 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe19019d360 con 0x7fe190074d20 2024-04-20T21:27:44.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:44.740+0000 7fe194a00700 1 -- 172.21.15.62:0/1603173448 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe17c018630 con 0x7fe190074d20 2024-04-20T21:27:44.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:44.740+0000 7fe195e00700 1 -- 172.21.15.62:0/1603173448 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe19019d8b0 con 0x7fe190074d20 2024-04-20T21:27:44.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:44.741+0000 7fe195e00700 1 -- 172.21.15.62:0/1603173448 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe19010af80 con 0x7fe190074d20 2024-04-20T21:27:44.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:44.742+0000 7fe194a00700 1 -- 172.21.15.62:0/1603173448 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7fe17c00f450 con 0x7fe190074d20 2024-04-20T21:27:44.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:44.743+0000 7fe194a00700 1 --2- 172.21.15.62:0/1603173448 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fe17406c680 0x7fe17406eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:27:44.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:44.743+0000 7fe194a00700 1 -- 172.21.15.62:0/1603173448 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5468+0+0 (secure 0 0 0) 0x7fe17c08ba70 con 0x7fe190074d20 2024-04-20T21:27:44.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:44.744+0000 7fe18ea00700 1 --2- 172.21.15.62:0/1603173448 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fe17406c680 0x7fe17406eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:27:44.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:44.744+0000 7fe18ea00700 1 --2- 172.21.15.62:0/1603173448 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fe17406c680 0x7fe17406eb30 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fe180006010 tx=0x7fe18000b540 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:27:44.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:44.748+0000 7fe194a00700 1 -- 172.21.15.62:0/1603173448 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fe17c056130 con 0x7fe190074d20 2024-04-20T21:27:45.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:45.054+0000 7fe195e00700 1 -- 172.21.15.62:0/1603173448 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fe19010b170 con 0x7fe190074d20 2024-04-20T21:27:45.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:45.055+0000 7fe194a00700 1 -- 172.21.15.62:0/1603173448 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 14 v14) v1 ==== 76+0+1847 (secure 0 0 0) 0x7fe17c059750 con 0x7fe190074d20 2024-04-20T21:27:45.058 INFO:teuthology.orchestra.run.smithi062.stdout:e14 2024-04-20T21:27:45.058 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T21:27:45.058 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T21:27:45.058 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T21:27:45.058 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:27:45.058 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T21:27:45.058 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T21:27:45.058 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 14 2024-04-20T21:27:45.058 INFO:teuthology.orchestra.run.smithi062.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-20T21:27:45.059 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T21:27:45.059 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T21:27:43.814916+0000 2024-04-20T21:27:45.059 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T21:27:45.059 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T21:27:45.059 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T21:27:45.059 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T21:27:45.059 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T21:27:45.059 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T21:27:45.059 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T21:27:45.059 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 0 2024-04-20T21:27:45.059 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T21:27:45.059 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 2 2024-04-20T21:27:45.059 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1 2024-04-20T21:27:45.059 INFO:teuthology.orchestra.run.smithi062.stdout:up {0=14458,1=24279} 2024-04-20T21:27:45.059 INFO:teuthology.orchestra.run.smithi062.stdout:failed 2024-04-20T21:27:45.060 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T21:27:45.060 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T21:27:45.060 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T21:27:45.060 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T21:27:45.060 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data enabled 2024-04-20T21:27:45.060 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T21:27:45.060 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T21:27:45.060 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 1 2024-04-20T21:27:45.060 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ryqblb{0:14458} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.62:6826/1147127123,v1:172.21.15.62:6827/1147127123] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:27:45.060 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ozizwi{0:14474} state up:standby-replay seq 3 join_fscid=1 addr [v2:172.21.15.62:6828/3494174293,v1:172.21.15.62:6829/3494174293] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:27:45.060 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.dvimer{1:24279} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.189:6824/237262514,v1:172.21.15.189:6825/237262514] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:27:45.060 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.xbgspw{1:24291} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.189:6826/1495356258,v1:172.21.15.189:6827/1495356258] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:27:45.060 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:27:45.060 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:27:45.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:45.059+0000 7fe195e00700 1 -- 172.21.15.62:0/1603173448 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fe17406c680 msgr2=0x7fe17406eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:27:45.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:45.059+0000 7fe195e00700 1 --2- 172.21.15.62:0/1603173448 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fe17406c680 0x7fe17406eb30 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fe180006010 tx=0x7fe18000b540 comp rx=0 tx=0).stop 2024-04-20T21:27:45.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:45.059+0000 7fe195e00700 1 -- 172.21.15.62:0/1603173448 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe190074d20 msgr2=0x7fe1901981e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:27:45.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:45.059+0000 7fe195e00700 1 --2- 172.21.15.62:0/1603173448 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe190074d20 0x7fe1901981e0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fe17c00c930 tx=0x7fe17c00ccf0 comp rx=0 tx=0).stop 2024-04-20T21:27:45.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:45.060+0000 7fe195e00700 1 -- 172.21.15.62:0/1603173448 shutdown_connections 2024-04-20T21:27:45.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:45.060+0000 7fe195e00700 1 --2- 172.21.15.62:0/1603173448 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fe17406c680 0x7fe17406eb30 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:45.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:45.060+0000 7fe195e00700 1 --2- 172.21.15.62:0/1603173448 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe190073750 0x7fe190197ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:45.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:45.060+0000 7fe195e00700 1 --2- 172.21.15.62:0/1603173448 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe190074d20 0x7fe1901981e0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:45.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:45.060+0000 7fe195e00700 1 -- 172.21.15.62:0/1603173448 >> 172.21.15.62:0/1603173448 conn(0x7fe1900fb7f0 msgr2=0x7fe190106080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:27:45.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:45.060+0000 7fe195e00700 1 -- 172.21.15.62:0/1603173448 shutdown_connections 2024-04-20T21:27:45.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:45.060+0000 7fe195e00700 1 -- 172.21.15.62:0/1603173448 wait complete. 2024-04-20T21:27:45.062 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 14 2024-04-20T21:27:45.206 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:44 smithi062 ceph-mon[16221]: pgmap v233: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.4 KiB/s rd, 1023 B/s wr, 5 op/s 2024-04-20T21:27:45.206 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:44 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:27:45.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:44 smithi189 ceph-mon[21562]: pgmap v233: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.4 KiB/s rd, 1023 B/s wr, 5 op/s 2024-04-20T21:27:45.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:44 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:27:46.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:45 smithi189 ceph-mon[21562]: from='client.? 172.21.15.62:0/1603173448' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T21:27:46.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:45 smithi189 ceph-mon[21562]: pgmap v234: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.4 KiB/s rd, 682 B/s wr, 5 op/s 2024-04-20T21:27:46.274 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:45 smithi062 ceph-mon[16221]: from='client.? 172.21.15.62:0/1603173448' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T21:27:46.275 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:45 smithi062 ceph-mon[16221]: pgmap v234: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.4 KiB/s rd, 682 B/s wr, 5 op/s 2024-04-20T21:27:46.325 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -- bash -c 'ceph --format=json fs dump | jq -e ".filesystems | length == 1"' 2024-04-20T21:27:46.811 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T21:27:48.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:47 smithi189 ceph-mon[21562]: pgmap v235: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.4 KiB/s rd, 0 B/s wr, 3 op/s 2024-04-20T21:27:48.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:47 smithi062 ceph-mon[16221]: pgmap v235: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.4 KiB/s rd, 0 B/s wr, 3 op/s 2024-04-20T21:27:49.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:49.909+0000 7f6dbd000700 1 -- 172.21.15.62:0/3429988611 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6db8103680 msgr2=0x7f6db8105a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:27:49.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:49.909+0000 7f6dbd000700 1 --2- 172.21.15.62:0/3429988611 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6db8103680 0x7f6db8105a60 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f6da4009910 tx=0x7f6da4009c20 comp rx=0 tx=0).stop 2024-04-20T21:27:49.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:49.909+0000 7f6dbd000700 1 -- 172.21.15.62:0/3429988611 shutdown_connections 2024-04-20T21:27:49.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:49.909+0000 7f6dbd000700 1 --2- 172.21.15.62:0/3429988611 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6db8103680 0x7f6db8105a60 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:49.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:49.909+0000 7f6dbd000700 1 --2- 172.21.15.62:0/3429988611 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6db8100d60 0x7f6db8103140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:49.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:49.909+0000 7f6dbd000700 1 -- 172.21.15.62:0/3429988611 >> 172.21.15.62:0/3429988611 conn(0x7f6db80fa780 msgr2=0x7f6db80fcbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:27:49.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:49.909+0000 7f6dbd000700 1 -- 172.21.15.62:0/3429988611 shutdown_connections 2024-04-20T21:27:49.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:49.909+0000 7f6dbd000700 1 -- 172.21.15.62:0/3429988611 wait complete. 2024-04-20T21:27:49.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:49.911+0000 7f6dbd000700 1 Processor -- start 2024-04-20T21:27:49.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:49.911+0000 7f6dbd000700 1 -- start start 2024-04-20T21:27:49.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:49.912+0000 7f6dbd000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6db8100d60 0x7f6db8195ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:27:49.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:49.912+0000 7f6dbd000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6db8103680 0x7f6db8196020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:27:49.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:49.912+0000 7f6dbd000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6db8196640 con 0x7f6db8103680 2024-04-20T21:27:49.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:49.912+0000 7f6dbd000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6db8196780 con 0x7f6db8100d60 2024-04-20T21:27:49.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:49.912+0000 7f6db5400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6db8103680 0x7f6db8196020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:27:49.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:49.912+0000 7f6db5400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6db8103680 0x7f6db8196020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60806/0 (socket says 172.21.15.62:60806) 2024-04-20T21:27:49.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:49.912+0000 7f6db5400700 1 -- 172.21.15.62:0/3514088300 learned_addr learned my addr 172.21.15.62:0/3514088300 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:27:49.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:49.912+0000 7f6db5e00700 1 --2- 172.21.15.62:0/3514088300 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6db8100d60 0x7f6db8195ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:27:49.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:49.913+0000 7f6db5400700 1 -- 172.21.15.62:0/3514088300 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6db8100d60 msgr2=0x7f6db8195ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:27:49.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:49.913+0000 7f6db5400700 1 --2- 172.21.15.62:0/3514088300 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6db8100d60 0x7f6db8195ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:49.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:49.913+0000 7f6db5400700 1 -- 172.21.15.62:0/3514088300 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6da40095f0 con 0x7f6db8103680 2024-04-20T21:27:49.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:49.913+0000 7f6db5400700 1 --2- 172.21.15.62:0/3514088300 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6db8103680 0x7f6db8196020 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f6da4005b40 tx=0x7f6da400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:27:49.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:49.914+0000 7f6daea00700 1 -- 172.21.15.62:0/3514088300 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6da401d070 con 0x7f6db8103680 2024-04-20T21:27:49.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:49.914+0000 7f6daea00700 1 -- 172.21.15.62:0/3514088300 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f6da400f460 con 0x7f6db8103680 2024-04-20T21:27:49.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:49.914+0000 7f6daea00700 1 -- 172.21.15.62:0/3514088300 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6da4021620 con 0x7f6db8103680 2024-04-20T21:27:49.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:49.914+0000 7f6dbd000700 1 -- 172.21.15.62:0/3514088300 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6db819b1d0 con 0x7f6db8103680 2024-04-20T21:27:49.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:49.914+0000 7f6dbd000700 1 -- 172.21.15.62:0/3514088300 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6db819b670 con 0x7f6db8103680 2024-04-20T21:27:49.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:49.916+0000 7f6dbd000700 1 -- 172.21.15.62:0/3514088300 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6db818fbe0 con 0x7f6db8103680 2024-04-20T21:27:49.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:49.917+0000 7f6daea00700 1 -- 172.21.15.62:0/3514088300 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f6da400fa40 con 0x7f6db8103680 2024-04-20T21:27:49.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:49.918+0000 7f6daea00700 1 --2- 172.21.15.62:0/3514088300 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f6d9c06c6d0 0x7f6d9c06eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:27:49.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:49.918+0000 7f6daea00700 1 -- 172.21.15.62:0/3514088300 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5468+0+0 (secure 0 0 0) 0x7f6da408cbd0 con 0x7f6db8103680 2024-04-20T21:27:49.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:49.919+0000 7f6db5e00700 1 --2- 172.21.15.62:0/3514088300 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f6d9c06c6d0 0x7f6d9c06eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:27:49.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:49.920+0000 7f6db5e00700 1 --2- 172.21.15.62:0/3514088300 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f6d9c06c6d0 0x7f6d9c06eb80 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f6da8005950 tx=0x7f6da800b410 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:27:49.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:49.928+0000 7f6daea00700 1 -- 172.21.15.62:0/3514088300 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f6da405aca0 con 0x7f6db8103680 2024-04-20T21:27:49.997 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:49 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:27:49.998 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:49 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:27:50.105 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:49 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:27:50.105 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:49 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:27:50.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:50.281+0000 7f6dbd000700 1 -- 172.21.15.62:0/3514088300 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6db802cc30 con 0x7f6db8103680 2024-04-20T21:27:50.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:50.282+0000 7f6daea00700 1 -- 172.21.15.62:0/3514088300 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+4771 (secure 0 0 0) 0x7f6da405a830 con 0x7f6db8103680 2024-04-20T21:27:50.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:50.285+0000 7f6dbd000700 1 -- 172.21.15.62:0/3514088300 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f6d9c06c6d0 msgr2=0x7f6d9c06eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:27:50.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:50.285+0000 7f6dbd000700 1 --2- 172.21.15.62:0/3514088300 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f6d9c06c6d0 0x7f6d9c06eb80 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f6da8005950 tx=0x7f6da800b410 comp rx=0 tx=0).stop 2024-04-20T21:27:50.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:50.285+0000 7f6dbd000700 1 -- 172.21.15.62:0/3514088300 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6db8103680 msgr2=0x7f6db8196020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:27:50.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:50.285+0000 7f6dbd000700 1 --2- 172.21.15.62:0/3514088300 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6db8103680 0x7f6db8196020 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f6da4005b40 tx=0x7f6da400bd70 comp rx=0 tx=0).stop 2024-04-20T21:27:50.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:50.285+0000 7f6dbd000700 1 -- 172.21.15.62:0/3514088300 shutdown_connections 2024-04-20T21:27:50.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:50.285+0000 7f6dbd000700 1 --2- 172.21.15.62:0/3514088300 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f6d9c06c6d0 0x7f6d9c06eb80 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:50.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:50.285+0000 7f6dbd000700 1 --2- 172.21.15.62:0/3514088300 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6db8100d60 0x7f6db8195ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:50.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:50.285+0000 7f6dbd000700 1 --2- 172.21.15.62:0/3514088300 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6db8103680 0x7f6db8196020 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:50.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:50.285+0000 7f6dbd000700 1 -- 172.21.15.62:0/3514088300 >> 172.21.15.62:0/3514088300 conn(0x7f6db80fa780 msgr2=0x7f6db80fcbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:27:50.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:50.285+0000 7f6dbd000700 1 -- 172.21.15.62:0/3514088300 shutdown_connections 2024-04-20T21:27:50.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:50.285+0000 7f6dbd000700 1 -- 172.21.15.62:0/3514088300 wait complete. 2024-04-20T21:27:50.285 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 14 2024-04-20T21:27:50.299 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T21:27:50.959 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:50 smithi062 ceph-mon[16221]: pgmap v236: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.3 KiB/s rd, 0 B/s wr, 3 op/s 2024-04-20T21:27:50.959 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:50 smithi062 ceph-mon[16221]: from='client.? 172.21.15.62:0/3514088300' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-20T21:27:50.959 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:50 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:27:50.959 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:50 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:27:51.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:50 smithi189 ceph-mon[21562]: pgmap v236: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.3 KiB/s rd, 0 B/s wr, 3 op/s 2024-04-20T21:27:51.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:50 smithi189 ceph-mon[21562]: from='client.? 172.21.15.62:0/3514088300' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-20T21:27:51.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:50 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:27:51.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:50 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:27:51.260 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -- bash -c 'while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done' 2024-04-20T21:27:51.647 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T21:27:52.055 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:51 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:27:52.055 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:51 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T21:27:52.055 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:51 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:27:52.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:51 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:27:52.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:51 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T21:27:52.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:51 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:27:53.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:52 smithi189 ceph-mon[21562]: pgmap v237: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.9 KiB/s rd, 4 op/s 2024-04-20T21:27:53.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:52 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:27:53.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:52 smithi062 ceph-mon[16221]: pgmap v237: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.9 KiB/s rd, 4 op/s 2024-04-20T21:27:53.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:52 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:27:53.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:53.844+0000 7fa5c7200700 1 -- 172.21.15.62:0/2500371944 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5c0103680 msgr2=0x7fa5c0105a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:27:53.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:53.844+0000 7fa5c7200700 1 --2- 172.21.15.62:0/2500371944 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5c0103680 0x7fa5c0105a60 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fa5ac009910 tx=0x7fa5ac009c20 comp rx=0 tx=0).stop 2024-04-20T21:27:53.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:53.844+0000 7fa5c7200700 1 -- 172.21.15.62:0/2500371944 shutdown_connections 2024-04-20T21:27:53.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:53.844+0000 7fa5c7200700 1 --2- 172.21.15.62:0/2500371944 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5c0103680 0x7fa5c0105a60 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:53.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:53.844+0000 7fa5c7200700 1 --2- 172.21.15.62:0/2500371944 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5c0100d60 0x7fa5c0103140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:53.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:53.844+0000 7fa5c7200700 1 -- 172.21.15.62:0/2500371944 >> 172.21.15.62:0/2500371944 conn(0x7fa5c00fa780 msgr2=0x7fa5c00fcbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:27:53.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:53.844+0000 7fa5c7200700 1 -- 172.21.15.62:0/2500371944 shutdown_connections 2024-04-20T21:27:53.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:53.844+0000 7fa5c7200700 1 -- 172.21.15.62:0/2500371944 wait complete. 2024-04-20T21:27:53.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:53.845+0000 7fa5c7200700 1 Processor -- start 2024-04-20T21:27:53.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:53.845+0000 7fa5c7200700 1 -- start start 2024-04-20T21:27:53.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:53.846+0000 7fa5c7200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5c0100d60 0x7fa5c0197cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:27:53.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:53.846+0000 7fa5c7200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5c0103680 0x7fa5c0198230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:27:53.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:53.846+0000 7fa5c7200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5c0198850 con 0x7fa5c0103680 2024-04-20T21:27:53.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:53.846+0000 7fa5c7200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5c019d260 con 0x7fa5c0100d60 2024-04-20T21:27:53.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:53.846+0000 7fa5bf400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5c0103680 0x7fa5c0198230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:27:53.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:53.846+0000 7fa5bf400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5c0103680 0x7fa5c0198230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35964/0 (socket says 172.21.15.62:35964) 2024-04-20T21:27:53.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:53.846+0000 7fa5bfe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5c0100d60 0x7fa5c0197cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:27:53.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:53.846+0000 7fa5bf400700 1 -- 172.21.15.62:0/1664443865 learned_addr learned my addr 172.21.15.62:0/1664443865 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:27:53.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:53.846+0000 7fa5bf400700 1 -- 172.21.15.62:0/1664443865 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5c0100d60 msgr2=0x7fa5c0197cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:27:53.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:53.846+0000 7fa5bf400700 1 --2- 172.21.15.62:0/1664443865 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5c0100d60 0x7fa5c0197cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:53.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:53.846+0000 7fa5bf400700 1 -- 172.21.15.62:0/1664443865 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5ac0095f0 con 0x7fa5c0103680 2024-04-20T21:27:53.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:53.847+0000 7fa5bf400700 1 --2- 172.21.15.62:0/1664443865 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5c0103680 0x7fa5c0198230 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fa5ac000c00 tx=0x7fa5ac00f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:27:53.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:53.847+0000 7fa5bcc00700 1 -- 172.21.15.62:0/1664443865 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa5ac01d070 con 0x7fa5c0103680 2024-04-20T21:27:53.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:53.847+0000 7fa5bcc00700 1 -- 172.21.15.62:0/1664443865 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa5ac005dc0 con 0x7fa5c0103680 2024-04-20T21:27:53.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:53.847+0000 7fa5bcc00700 1 -- 172.21.15.62:0/1664443865 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa5ac0226a0 con 0x7fa5c0103680 2024-04-20T21:27:53.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:53.847+0000 7fa5c7200700 1 -- 172.21.15.62:0/1664443865 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5c019d400 con 0x7fa5c0103680 2024-04-20T21:27:53.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:53.847+0000 7fa5c7200700 1 -- 172.21.15.62:0/1664443865 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5c019d8c0 con 0x7fa5c0103680 2024-04-20T21:27:53.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:53.849+0000 7fa5c7200700 1 -- 172.21.15.62:0/1664443865 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5c00fc340 con 0x7fa5c0103680 2024-04-20T21:27:53.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:53.850+0000 7fa5bcc00700 1 -- 172.21.15.62:0/1664443865 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7fa5ac0038f0 con 0x7fa5c0103680 2024-04-20T21:27:53.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:53.851+0000 7fa5bcc00700 1 --2- 172.21.15.62:0/1664443865 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fa5a406c680 0x7fa5a406eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:27:53.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:53.851+0000 7fa5bcc00700 1 -- 172.21.15.62:0/1664443865 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5468+0+0 (secure 0 0 0) 0x7fa5ac08ccf0 con 0x7fa5c0103680 2024-04-20T21:27:53.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:53.852+0000 7fa5bfe00700 1 --2- 172.21.15.62:0/1664443865 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fa5a406c680 0x7fa5a406eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:27:53.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:53.852+0000 7fa5bfe00700 1 --2- 172.21.15.62:0/1664443865 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fa5a406c680 0x7fa5a406eb30 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fa5c01991d0 tx=0x7fa5b0009260 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:27:53.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:53.857+0000 7fa5bcc00700 1 -- 172.21.15.62:0/1664443865 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fa5ac0573b0 con 0x7fa5c0103680 2024-04-20T21:27:54.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:54.180+0000 7fa5c7200700 1 -- 172.21.15.62:0/1664443865 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mds versions", "format": "json"} v 0) v1 -- 0x7fa5c002cc30 con 0x7fa5c0103680 2024-04-20T21:27:54.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:54.182+0000 7fa5bcc00700 1 -- 172.21.15.62:0/1664443865 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mds versions", "format": "json"}]=0 v14) v1 ==== 78+0+83 (secure 0 0 0) 0x7fa5ac003680 con 0x7fa5c0103680 2024-04-20T21:27:54.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:54.185+0000 7fa5c7200700 1 -- 172.21.15.62:0/1664443865 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fa5a406c680 msgr2=0x7fa5a406eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:27:54.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:54.185+0000 7fa5c7200700 1 --2- 172.21.15.62:0/1664443865 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fa5a406c680 0x7fa5a406eb30 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fa5c01991d0 tx=0x7fa5b0009260 comp rx=0 tx=0).stop 2024-04-20T21:27:54.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:54.185+0000 7fa5c7200700 1 -- 172.21.15.62:0/1664443865 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5c0103680 msgr2=0x7fa5c0198230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:27:54.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:54.185+0000 7fa5c7200700 1 --2- 172.21.15.62:0/1664443865 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5c0103680 0x7fa5c0198230 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fa5ac000c00 tx=0x7fa5ac00f690 comp rx=0 tx=0).stop 2024-04-20T21:27:54.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:54.185+0000 7fa5c7200700 1 -- 172.21.15.62:0/1664443865 shutdown_connections 2024-04-20T21:27:54.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:54.185+0000 7fa5c7200700 1 --2- 172.21.15.62:0/1664443865 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fa5a406c680 0x7fa5a406eb30 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:54.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:54.185+0000 7fa5c7200700 1 --2- 172.21.15.62:0/1664443865 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5c0100d60 0x7fa5c0197cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:54.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:54.185+0000 7fa5c7200700 1 --2- 172.21.15.62:0/1664443865 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5c0103680 0x7fa5c0198230 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:54.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:54.185+0000 7fa5c7200700 1 -- 172.21.15.62:0/1664443865 >> 172.21.15.62:0/1664443865 conn(0x7fa5c00fa780 msgr2=0x7fa5c00ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:27:54.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:54.185+0000 7fa5c7200700 1 -- 172.21.15.62:0/1664443865 shutdown_connections 2024-04-20T21:27:54.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:54.185+0000 7fa5c7200700 1 -- 172.21.15.62:0/1664443865 wait complete. 2024-04-20T21:27:54.205 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T21:27:54.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:53 smithi189 ceph-mon[21562]: pgmap v238: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-20T21:27:54.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:53 smithi062 ceph-mon[16221]: pgmap v238: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-20T21:27:55.236 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:54 smithi189 ceph-mon[21562]: from='client.? 172.21.15.62:0/1664443865' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2024-04-20T21:27:55.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:54 smithi062 ceph-mon[16221]: from='client.? 172.21.15.62:0/1664443865' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2024-04-20T21:27:55.520 INFO:teuthology.run_tasks:Running task fs.pre_upgrade_save... 2024-04-20T21:27:55.532 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-20T21:27:55.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:55.764+0000 7ff4fa400640 1 -- 172.21.15.62:0/499337855 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff4f40fdd20 msgr2=0x7ff4f40fe130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:27:55.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:55.764+0000 7ff4fa400640 1 --2- 172.21.15.62:0/499337855 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff4f40fdd20 0x7ff4f40fe130 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7ff4e8009880 tx=0x7ff4e802bce0 comp rx=0 tx=0).stop 2024-04-20T21:27:55.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:55.764+0000 7ff4fa400640 1 -- 172.21.15.62:0/499337855 shutdown_connections 2024-04-20T21:27:55.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:55.764+0000 7ff4fa400640 1 --2- 172.21.15.62:0/499337855 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4f40fe670 0x7ff4f40feac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:55.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:55.764+0000 7ff4fa400640 1 --2- 172.21.15.62:0/499337855 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff4f40fdd20 0x7ff4f40fe130 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:55.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:55.764+0000 7ff4fa400640 1 -- 172.21.15.62:0/499337855 >> 172.21.15.62:0/499337855 conn(0x7ff4f40f9890 msgr2=0x7ff4f40fbd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:27:55.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:55.764+0000 7ff4fa400640 1 -- 172.21.15.62:0/499337855 shutdown_connections 2024-04-20T21:27:55.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:55.764+0000 7ff4fa400640 1 -- 172.21.15.62:0/499337855 wait complete. 2024-04-20T21:27:55.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:55.765+0000 7ff4fa400640 1 Processor -- start 2024-04-20T21:27:55.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:55.765+0000 7ff4fa400640 1 -- start start 2024-04-20T21:27:55.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:55.765+0000 7ff4fa400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4f40fdd20 0x7ff4f4196e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:27:55.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:55.765+0000 7ff4fa400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff4f40fe670 0x7ff4f41973c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:27:55.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:55.765+0000 7ff4fa400640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4f4197990 con 0x7ff4f40fe670 2024-04-20T21:27:55.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:55.765+0000 7ff4fa400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4f4197b00 con 0x7ff4f40fdd20 2024-04-20T21:27:55.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:55.766+0000 7ff4f3e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff4f40fe670 0x7ff4f41973c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:27:55.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:55.766+0000 7ff4f3e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff4f40fe670 0x7ff4f41973c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35996/0 (socket says 172.21.15.62:35996) 2024-04-20T21:27:55.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:55.766+0000 7ff4f3e00640 1 -- 172.21.15.62:0/23409919 learned_addr learned my addr 172.21.15.62:0/23409919 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:27:55.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:55.766+0000 7ff4f9000640 1 --2- 172.21.15.62:0/23409919 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4f40fdd20 0x7ff4f4196e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:27:55.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:55.766+0000 7ff4f3e00640 1 -- 172.21.15.62:0/23409919 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4f40fdd20 msgr2=0x7ff4f4196e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:27:55.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:55.766+0000 7ff4f3e00640 1 --2- 172.21.15.62:0/23409919 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4f40fdd20 0x7ff4f4196e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:55.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:55.766+0000 7ff4f3e00640 1 -- 172.21.15.62:0/23409919 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4e8009560 con 0x7ff4f40fe670 2024-04-20T21:27:55.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:55.766+0000 7ff4f9000640 1 --2- 172.21.15.62:0/23409919 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4f40fdd20 0x7ff4f4196e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-20T21:27:55.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:55.767+0000 7ff4f3e00640 1 --2- 172.21.15.62:0/23409919 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff4f40fe670 0x7ff4f41973c0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7ff4e000c810 tx=0x7ff4e000ccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:27:55.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:55.767+0000 7ff4f1600640 1 -- 172.21.15.62:0/23409919 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff4e00041f0 con 0x7ff4f40fe670 2024-04-20T21:27:55.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:55.767+0000 7ff4f1600640 1 -- 172.21.15.62:0/23409919 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff4e0007e60 con 0x7ff4f40fe670 2024-04-20T21:27:55.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:55.767+0000 7ff4f1600640 1 -- 172.21.15.62:0/23409919 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff4e00049d0 con 0x7ff4f40fe670 2024-04-20T21:27:55.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:55.767+0000 7ff4fa400640 1 -- 172.21.15.62:0/23409919 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4f419c590 con 0x7ff4f40fe670 2024-04-20T21:27:55.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:55.767+0000 7ff4fa400640 1 -- 172.21.15.62:0/23409919 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4f419caa0 con 0x7ff4f40fe670 2024-04-20T21:27:55.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:55.768+0000 7ff4fa400640 1 -- 172.21.15.62:0/23409919 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4f410a350 con 0x7ff4f40fe670 2024-04-20T21:27:55.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:55.769+0000 7ff4f1600640 1 -- 172.21.15.62:0/23409919 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7ff4e00079b0 con 0x7ff4f40fe670 2024-04-20T21:27:55.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:55.770+0000 7ff4f1600640 1 --2- 172.21.15.62:0/23409919 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7ff4dc06c6b0 0x7ff4dc06eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:27:55.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:55.771+0000 7ff4f1600640 1 -- 172.21.15.62:0/23409919 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5468+0+0 (secure 0 0 0) 0x7ff4e008b6a0 con 0x7ff4f40fe670 2024-04-20T21:27:55.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:55.771+0000 7ff4f9000640 1 --2- 172.21.15.62:0/23409919 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7ff4dc06c6b0 0x7ff4dc06eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:27:55.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:55.772+0000 7ff4f9000640 1 --2- 172.21.15.62:0/23409919 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7ff4dc06c6b0 0x7ff4dc06eb60 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ff4e8000c00 tx=0x7ff4e802cf90 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:27:55.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:55.775+0000 7ff4f1600640 1 -- 172.21.15.62:0/23409919 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7ff4e0055d80 con 0x7ff4f40fe670 2024-04-20T21:27:55.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:55.987+0000 7ff4fa400640 1 -- 172.21.15.62:0/23409919 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff4f419ce60 con 0x7ff4f40fe670 2024-04-20T21:27:55.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:55.988+0000 7ff4f1600640 1 -- 172.21.15.62:0/23409919 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v45) v1 ==== 74+0+14959 (secure 0 0 0) 0x7ff4e00593a0 con 0x7ff4f40fe670 2024-04-20T21:27:55.988 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:27:55.988 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":45,"fsid":"61ae3628-ff5b-11ee-bc93-c7b262605968","created":"2024-04-20T21:18:19.951305+0000","modified":"2024-04-20T21:27:23.044645+0000","last_up_change":"2024-04-20T21:25:43.888770+0000","last_in_change":"2024-04-20T21:25:19.201764+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-20T21:24:02.500375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-04-20T21:27:19.577559+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-04-20T21:27:20.560814+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a3f8216-c641-4338-813d-d836efc3f4e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":2687023571},{"type":"v1","addr":"172.21.15.62:6803","nonce":2687023571}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":2687023571},{"type":"v1","addr":"172.21.15.62:6805","nonce":2687023571}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":2687023571},{"type":"v1","addr":"172.21.15.62:6809","nonce":2687023571}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":2687023571},{"type":"v1","addr":"172.21.15.62:6807","nonce":2687023571}]},"public_addr":"172.21.15.62:6803/2687023571","cluster_addr":"172.21.15.62:6805/2687023571","heartbeat_back_addr":"172.21.15.62:6809/2687023571","heartbeat_front_addr":"172.21.15.62:6807/2687023571","state":["exists","up"]},{"osd":1,"uuid":"f0c451a4-9002-4ddc-be0e-d8ddebd4418d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":1027109374},{"type":"v1","addr":"172.21.15.62:6811","nonce":1027109374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":1027109374},{"type":"v1","addr":"172.21.15.62:6813","nonce":1027109374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":1027109374},{"type":"v1","addr":"172.21.15.62:6817","nonce":1027109374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":1027109374},{"type":"v1","addr":"172.21.15.62:6815","nonce":1027109374}]},"public_addr":"172.21.15.62:6811/1027109374","cluster_addr":"172.21.15.62:6813/1027109374","heartbeat_back_addr":"172.21.15.62:6817/1027109374","heartbeat_front_addr":"172.21.15.62:6815/1027109374","state":["exists","up"]},{"osd":2,"uuid":"e1a8769d-5da2-49c8-942a-83a94c7ac761","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":47205739},{"type":"v1","addr":"172.21.15.62:6819","nonce":47205739}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":47205739},{"type":"v1","addr":"172.21.15.62:6821","nonce":47205739}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":47205739},{"type":"v1","addr":"172.21.15.62:6825","nonce":47205739}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6822","nonce":47205739},{"type":"v1","addr":"172.21.15.62:6823","nonce":47205739}]},"public_addr":"172.21.15.62:6819/47205739","cluster_addr":"172.21.15.62:6821/47205739","heartbeat_back_addr":"172.21.15.62:6825/47205739","heartbeat_front_addr":"172.21.15.62:6823/47205739","state":["exists","up"]},{"osd":3,"uuid":"eaa2fe2b-87f8-4d01-8ddf-e72f244f31b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":3169871320},{"type":"v1","addr":"172.21.15.189:6801","nonce":3169871320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":3169871320},{"type":"v1","addr":"172.21.15.189:6803","nonce":3169871320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":3169871320},{"type":"v1","addr":"172.21.15.189:6807","nonce":3169871320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":3169871320},{"type":"v1","addr":"172.21.15.189:6805","nonce":3169871320}]},"public_addr":"172.21.15.189:6801/3169871320","cluster_addr":"172.21.15.189:6803/3169871320","heartbeat_back_addr":"172.21.15.189:6807/3169871320","heartbeat_front_addr":"172.21.15.189:6805/3169871320","state":["exists","up"]},{"osd":4,"uuid":"72be8597-52e5-45fd-b4f2-06ac50df77e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":191133671},{"type":"v1","addr":"172.21.15.189:6809","nonce":191133671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":191133671},{"type":"v1","addr":"172.21.15.189:6811","nonce":191133671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":191133671},{"type":"v1","addr":"172.21.15.189:6815","nonce":191133671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":191133671},{"type":"v1","addr":"172.21.15.189:6813","nonce":191133671}]},"public_addr":"172.21.15.189:6809/191133671","cluster_addr":"172.21.15.189:6811/191133671","heartbeat_back_addr":"172.21.15.189:6815/191133671","heartbeat_front_addr":"172.21.15.189:6813/191133671","state":["exists","up"]},{"osd":5,"uuid":"c650c63b-41eb-4d4f-8e14-907c7b3614c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":893231043},{"type":"v1","addr":"172.21.15.189:6817","nonce":893231043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":893231043},{"type":"v1","addr":"172.21.15.189:6819","nonce":893231043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":893231043},{"type":"v1","addr":"172.21.15.189:6823","nonce":893231043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":893231043},{"type":"v1","addr":"172.21.15.189:6821","nonce":893231043}]},"public_addr":"172.21.15.189:6817/893231043","cluster_addr":"172.21.15.189:6819/893231043","heartbeat_back_addr":"172.21.15.189:6823/893231043","heartbeat_front_addr":"172.21.15.189:6821/893231043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T21:22:45.605185+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T21:23:21.062873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T21:24:00.502044+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T21:24:32.944093+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T21:25:06.278988+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T21:25:41.872982+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.62:0/1116048562":"2024-04-21T21:21:03.778739+0000","172.21.15.62:6800/831506538":"2024-04-21T21:21:03.778739+0000","172.21.15.62:0/4158176989":"2024-04-21T21:19:34.965981+0000","172.21.15.62:6800/4147738586":"2024-04-21T21:19:34.965981+0000","172.21.15.62:0/2525292829":"2024-04-21T21:18:52.990685+0000","172.21.15.62:0/301716985":"2024-04-21T21:21:03.778739+0000","172.21.15.62:0/634566145":"2024-04-21T21:18:52.990685+0000","172.21.15.62:0/3192460755":"2024-04-21T21:21:03.778739+0000","172.21.15.62:6801/831506538":"2024-04-21T21:21:03.778739+0000","172.21.15.62:6800/871423274":"2024-04-21T21:18:52.990685+0000","172.21.15.62:6801/4147738586":"2024-04-21T21:19:34.965981+0000","172.21.15.62:0/771955286":"2024-04-21T21:19:34.965981+0000","172.21.15.62:6801/871423274":"2024-04-21T21:18:52.990685+0000","172.21.15.62:0/3416563026":"2024-04-21T21:18:52.990685+0000","172.21.15.62:0/4274508569":"2024-04-21T21:19:34.965981+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-20T21:27:55.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:55.991+0000 7ff4fa400640 1 -- 172.21.15.62:0/23409919 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7ff4dc06c6b0 msgr2=0x7ff4dc06eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:27:55.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:55.991+0000 7ff4fa400640 1 --2- 172.21.15.62:0/23409919 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7ff4dc06c6b0 0x7ff4dc06eb60 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ff4e8000c00 tx=0x7ff4e802cf90 comp rx=0 tx=0).stop 2024-04-20T21:27:55.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:55.991+0000 7ff4fa400640 1 -- 172.21.15.62:0/23409919 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff4f40fe670 msgr2=0x7ff4f41973c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:27:55.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:55.991+0000 7ff4fa400640 1 --2- 172.21.15.62:0/23409919 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff4f40fe670 0x7ff4f41973c0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7ff4e000c810 tx=0x7ff4e000ccd0 comp rx=0 tx=0).stop 2024-04-20T21:27:55.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:55.991+0000 7ff4fa400640 1 -- 172.21.15.62:0/23409919 shutdown_connections 2024-04-20T21:27:55.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:55.991+0000 7ff4fa400640 1 --2- 172.21.15.62:0/23409919 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7ff4dc06c6b0 0x7ff4dc06eb60 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:55.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:55.991+0000 7ff4fa400640 1 --2- 172.21.15.62:0/23409919 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff4f40fe670 0x7ff4f41973c0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:55.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:55.991+0000 7ff4fa400640 1 --2- 172.21.15.62:0/23409919 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4f40fdd20 0x7ff4f4196e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:55.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:55.991+0000 7ff4fa400640 1 -- 172.21.15.62:0/23409919 >> 172.21.15.62:0/23409919 conn(0x7ff4f40f9890 msgr2=0x7ff4f4106350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:27:55.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:55.991+0000 7ff4fa400640 1 -- 172.21.15.62:0/23409919 shutdown_connections 2024-04-20T21:27:55.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:55.991+0000 7ff4fa400640 1 -- 172.21.15.62:0/23409919 wait complete. 2024-04-20T21:27:56.010 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-20T21:24:02.500375+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-04-20T21:27:19.577559+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 1, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-04-20T21:27:20.560814+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-20T21:27:56.011 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-20T21:27:56.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:56.202+0000 7f7d8d800640 1 -- 172.21.15.62:0/2150548236 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d88074c20 msgr2=0x7f7d88075030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:27:56.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:56.202+0000 7f7d8d800640 1 --2- 172.21.15.62:0/2150548236 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d88074c20 0x7f7d88075030 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f7d78009c20 tx=0x7f7d7802c1b0 comp rx=0 tx=0).stop 2024-04-20T21:27:56.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:56.203+0000 7f7d8d800640 1 -- 172.21.15.62:0/2150548236 shutdown_connections 2024-04-20T21:27:56.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:56.203+0000 7f7d8d800640 1 --2- 172.21.15.62:0/2150548236 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d88075e60 0x7f7d880762b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:56.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:56.203+0000 7f7d8d800640 1 --2- 172.21.15.62:0/2150548236 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d88074c20 0x7f7d88075030 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:56.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:56.203+0000 7f7d8d800640 1 -- 172.21.15.62:0/2150548236 >> 172.21.15.62:0/2150548236 conn(0x7f7d880fce10 msgr2=0x7f7d880ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:27:56.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:56.203+0000 7f7d8d800640 1 -- 172.21.15.62:0/2150548236 shutdown_connections 2024-04-20T21:27:56.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:56.203+0000 7f7d8d800640 1 -- 172.21.15.62:0/2150548236 wait complete. 2024-04-20T21:27:56.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:56.204+0000 7f7d8d800640 1 Processor -- start 2024-04-20T21:27:56.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:56.204+0000 7f7d8d800640 1 -- start start 2024-04-20T21:27:56.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:56.204+0000 7f7d8d800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d88074c20 0x7f7d8819b590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:27:56.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:56.204+0000 7f7d8d800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d88075e60 0x7f7d8819bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:27:56.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:56.204+0000 7f7d8d800640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d8819c0a0 con 0x7f7d88075e60 2024-04-20T21:27:56.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:56.204+0000 7f7d8d800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d8819c210 con 0x7f7d88074c20 2024-04-20T21:27:56.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:56.204+0000 7f7d7e000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d88075e60 0x7f7d8819bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:27:56.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:56.204+0000 7f7d7e000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d88075e60 0x7f7d8819bad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:36012/0 (socket says 172.21.15.62:36012) 2024-04-20T21:27:56.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:56.204+0000 7f7d7e000640 1 -- 172.21.15.62:0/960100890 learned_addr learned my addr 172.21.15.62:0/960100890 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:27:56.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:56.205+0000 7f7d86a00640 1 --2- 172.21.15.62:0/960100890 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d88074c20 0x7f7d8819b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:27:56.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:56.205+0000 7f7d7e000640 1 -- 172.21.15.62:0/960100890 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d88074c20 msgr2=0x7f7d8819b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:27:56.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:56.205+0000 7f7d7e000640 1 --2- 172.21.15.62:0/960100890 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d88074c20 0x7f7d8819b590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:56.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:56.205+0000 7f7d7e000640 1 -- 172.21.15.62:0/960100890 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d78009900 con 0x7f7d88075e60 2024-04-20T21:27:56.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:56.205+0000 7f7d7e000640 1 --2- 172.21.15.62:0/960100890 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d88075e60 0x7f7d8819bad0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f7d7000d6f0 tx=0x7f7d7000dbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:27:56.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:56.205+0000 7f7d7fe00640 1 -- 172.21.15.62:0/960100890 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7d70009a30 con 0x7f7d88075e60 2024-04-20T21:27:56.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:56.206+0000 7f7d8d800640 1 -- 172.21.15.62:0/960100890 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d881a0ca0 con 0x7f7d88075e60 2024-04-20T21:27:56.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:56.206+0000 7f7d7fe00640 1 -- 172.21.15.62:0/960100890 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f7d700054d0 con 0x7f7d88075e60 2024-04-20T21:27:56.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:56.206+0000 7f7d7fe00640 1 -- 172.21.15.62:0/960100890 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7d70010460 con 0x7f7d88075e60 2024-04-20T21:27:56.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:56.206+0000 7f7d8d800640 1 -- 172.21.15.62:0/960100890 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d881a1280 con 0x7f7d88075e60 2024-04-20T21:27:56.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:56.207+0000 7f7d8d800640 1 -- 172.21.15.62:0/960100890 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d8810e9a0 con 0x7f7d88075e60 2024-04-20T21:27:56.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:56.208+0000 7f7d7fe00640 1 -- 172.21.15.62:0/960100890 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f7d70009b90 con 0x7f7d88075e60 2024-04-20T21:27:56.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:56.209+0000 7f7d7fe00640 1 --2- 172.21.15.62:0/960100890 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f7d6c06c5f0 0x7f7d6c06eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:27:56.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:56.209+0000 7f7d7fe00640 1 -- 172.21.15.62:0/960100890 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5468+0+0 (secure 0 0 0) 0x7f7d7008b4c0 con 0x7f7d88075e60 2024-04-20T21:27:56.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:56.209+0000 7f7d86a00640 1 --2- 172.21.15.62:0/960100890 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f7d6c06c5f0 0x7f7d6c06eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:27:56.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:56.210+0000 7f7d86a00640 1 --2- 172.21.15.62:0/960100890 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f7d6c06c5f0 0x7f7d6c06eaa0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f7d78009bf0 tx=0x7f7d78032340 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:27:56.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:56.213+0000 7f7d7fe00640 1 -- 172.21.15.62:0/960100890 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f7d70055b00 con 0x7f7d88075e60 2024-04-20T21:27:56.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:56.411+0000 7f7d8d800640 1 -- 172.21.15.62:0/960100890 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7d881a16a0 con 0x7f7d88075e60 2024-04-20T21:27:56.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:56.412+0000 7f7d7fe00640 1 -- 172.21.15.62:0/960100890 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v45) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7d70089e60 con 0x7f7d88075e60 2024-04-20T21:27:56.412 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 1 2024-04-20T21:27:56.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:56.415+0000 7f7d8d800640 1 -- 172.21.15.62:0/960100890 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f7d6c06c5f0 msgr2=0x7f7d6c06eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:27:56.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:56.415+0000 7f7d8d800640 1 --2- 172.21.15.62:0/960100890 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f7d6c06c5f0 0x7f7d6c06eaa0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f7d78009bf0 tx=0x7f7d78032340 comp rx=0 tx=0).stop 2024-04-20T21:27:56.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:56.415+0000 7f7d8d800640 1 -- 172.21.15.62:0/960100890 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d88075e60 msgr2=0x7f7d8819bad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:27:56.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:56.415+0000 7f7d8d800640 1 --2- 172.21.15.62:0/960100890 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d88075e60 0x7f7d8819bad0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f7d7000d6f0 tx=0x7f7d7000dbb0 comp rx=0 tx=0).stop 2024-04-20T21:27:56.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:56.415+0000 7f7d8d800640 1 -- 172.21.15.62:0/960100890 shutdown_connections 2024-04-20T21:27:56.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:56.415+0000 7f7d8d800640 1 --2- 172.21.15.62:0/960100890 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f7d6c06c5f0 0x7f7d6c06eaa0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:56.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:56.415+0000 7f7d8d800640 1 --2- 172.21.15.62:0/960100890 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d88075e60 0x7f7d8819bad0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:56.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:56.415+0000 7f7d8d800640 1 --2- 172.21.15.62:0/960100890 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d88074c20 0x7f7d8819b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:56.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:56.415+0000 7f7d8d800640 1 -- 172.21.15.62:0/960100890 >> 172.21.15.62:0/960100890 conn(0x7f7d880fce10 msgr2=0x7f7d8810baa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:27:56.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:56.415+0000 7f7d8d800640 1 -- 172.21.15.62:0/960100890 shutdown_connections 2024-04-20T21:27:56.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:56.415+0000 7f7d8d800640 1 -- 172.21.15.62:0/960100890 wait complete. 2024-04-20T21:27:56.434 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-04-20T21:27:56.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:56.636+0000 7fc309000640 1 -- 172.21.15.62:0/1538139667 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc3040ff580 msgr2=0x7fc3040ff9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:27:56.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:56.636+0000 7fc309000640 1 --2- 172.21.15.62:0/1538139667 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc3040ff580 0x7fc3040ff9f0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fc2ec009880 tx=0x7fc2ec02bce0 comp rx=0 tx=0).stop 2024-04-20T21:27:56.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:56.636+0000 7fc309000640 1 -- 172.21.15.62:0/1538139667 shutdown_connections 2024-04-20T21:27:56.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:56.636+0000 7fc309000640 1 --2- 172.21.15.62:0/1538139667 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc3040ff580 0x7fc3040ff9f0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:56.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:56.636+0000 7fc309000640 1 --2- 172.21.15.62:0/1538139667 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc3040fe390 0x7fc3040fe7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:56.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:56.636+0000 7fc309000640 1 -- 172.21.15.62:0/1538139667 >> 172.21.15.62:0/1538139667 conn(0x7fc3040f98f0 msgr2=0x7fc3040fbd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:27:56.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:56.636+0000 7fc309000640 1 -- 172.21.15.62:0/1538139667 shutdown_connections 2024-04-20T21:27:56.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:56.636+0000 7fc309000640 1 -- 172.21.15.62:0/1538139667 wait complete. 2024-04-20T21:27:56.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:56.637+0000 7fc309000640 1 Processor -- start 2024-04-20T21:27:56.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:56.637+0000 7fc309000640 1 -- start start 2024-04-20T21:27:56.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:56.638+0000 7fc309000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc3040fe390 0x7fc304102320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:27:56.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:56.638+0000 7fc309000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc3040ff580 0x7fc304100960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:27:56.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:56.638+0000 7fc309000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc304100ea0 con 0x7fc3040fe390 2024-04-20T21:27:56.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:56.638+0000 7fc309000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc304101010 con 0x7fc3040ff580 2024-04-20T21:27:56.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:56.638+0000 7fc303400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc3040fe390 0x7fc304102320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:27:56.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:56.638+0000 7fc303400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc3040fe390 0x7fc304102320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:36034/0 (socket says 172.21.15.62:36034) 2024-04-20T21:27:56.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:56.638+0000 7fc303400640 1 -- 172.21.15.62:0/2332213647 learned_addr learned my addr 172.21.15.62:0/2332213647 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:27:56.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:56.638+0000 7fc302a00640 1 --2- 172.21.15.62:0/2332213647 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc3040ff580 0x7fc304100960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:27:56.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:56.638+0000 7fc303400640 1 -- 172.21.15.62:0/2332213647 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc3040ff580 msgr2=0x7fc304100960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:27:56.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:56.638+0000 7fc303400640 1 --2- 172.21.15.62:0/2332213647 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc3040ff580 0x7fc304100960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:56.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:56.638+0000 7fc303400640 1 -- 172.21.15.62:0/2332213647 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2ec009560 con 0x7fc3040fe390 2024-04-20T21:27:56.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:56.639+0000 7fc303400640 1 --2- 172.21.15.62:0/2332213647 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc3040fe390 0x7fc304102320 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fc2f400ea20 tx=0x7fc2f400eee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:27:56.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:56.639+0000 7fc2fbe00640 1 -- 172.21.15.62:0/2332213647 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc2f400cdb0 con 0x7fc3040fe390 2024-04-20T21:27:56.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:56.639+0000 7fc2fbe00640 1 -- 172.21.15.62:0/2332213647 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc2f4003e20 con 0x7fc3040fe390 2024-04-20T21:27:56.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:56.639+0000 7fc309000640 1 -- 172.21.15.62:0/2332213647 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3041012f0 con 0x7fc3040fe390 2024-04-20T21:27:56.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:56.639+0000 7fc2fbe00640 1 -- 172.21.15.62:0/2332213647 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc2f40116a0 con 0x7fc3040fe390 2024-04-20T21:27:56.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:56.639+0000 7fc309000640 1 -- 172.21.15.62:0/2332213647 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc30419cde0 con 0x7fc3040fe390 2024-04-20T21:27:56.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:56.640+0000 7fc309000640 1 -- 172.21.15.62:0/2332213647 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc304106f70 con 0x7fc3040fe390 2024-04-20T21:27:56.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:56.641+0000 7fc2fbe00640 1 -- 172.21.15.62:0/2332213647 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7fc2f4004030 con 0x7fc3040fe390 2024-04-20T21:27:56.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:56.642+0000 7fc2fbe00640 1 --2- 172.21.15.62:0/2332213647 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fc2e806c5f0 0x7fc2e806eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:27:56.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:56.642+0000 7fc2fbe00640 1 -- 172.21.15.62:0/2332213647 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5468+0+0 (secure 0 0 0) 0x7fc2f408be70 con 0x7fc3040fe390 2024-04-20T21:27:56.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:56.643+0000 7fc302a00640 1 --2- 172.21.15.62:0/2332213647 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fc2e806c5f0 0x7fc2e806eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:27:56.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:56.643+0000 7fc302a00640 1 --2- 172.21.15.62:0/2332213647 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fc2e806c5f0 0x7fc2e806eaa0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fc2ec02c7c0 tx=0x7fc2ec005fd0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:27:56.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:56.647+0000 7fc2fbe00640 1 -- 172.21.15.62:0/2332213647 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fc2f40564b0 con 0x7fc3040fe390 2024-04-20T21:27:56.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:56 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:27:56.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:56 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:27:56.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:56 smithi189 ceph-mon[21562]: pgmap v239: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.7 KiB/s rd, 4 op/s 2024-04-20T21:27:56.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:56 smithi189 ceph-mon[21562]: from='client.? 172.21.15.62:0/23409919' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-20T21:27:56.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:56 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:27:56.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:56 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:27:56.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:56 smithi062 ceph-mon[16221]: pgmap v239: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.7 KiB/s rd, 4 op/s 2024-04-20T21:27:56.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:56 smithi062 ceph-mon[16221]: from='client.? 172.21.15.62:0/23409919' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-20T21:27:56.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:56.867+0000 7fc309000640 1 -- 172.21.15.62:0/2332213647 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7fc304107170 con 0x7fc3040fe390 2024-04-20T21:27:56.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:56.868+0000 7fc2fbe00640 1 -- 172.21.15.62:0/2332213647 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v45) v1 ==== 107+0+11 (secure 0 0 0) 0x7fc2f4059ad0 con 0x7fc3040fe390 2024-04-20T21:27:56.867 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 32 2024-04-20T21:27:56.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:56.870+0000 7fc309000640 1 -- 172.21.15.62:0/2332213647 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fc2e806c5f0 msgr2=0x7fc2e806eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:27:56.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:56.870+0000 7fc309000640 1 --2- 172.21.15.62:0/2332213647 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fc2e806c5f0 0x7fc2e806eaa0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fc2ec02c7c0 tx=0x7fc2ec005fd0 comp rx=0 tx=0).stop 2024-04-20T21:27:56.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:56.870+0000 7fc309000640 1 -- 172.21.15.62:0/2332213647 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc3040fe390 msgr2=0x7fc304102320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:27:56.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:56.870+0000 7fc309000640 1 --2- 172.21.15.62:0/2332213647 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc3040fe390 0x7fc304102320 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fc2f400ea20 tx=0x7fc2f400eee0 comp rx=0 tx=0).stop 2024-04-20T21:27:56.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:56.871+0000 7fc309000640 1 -- 172.21.15.62:0/2332213647 shutdown_connections 2024-04-20T21:27:56.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:56.871+0000 7fc309000640 1 --2- 172.21.15.62:0/2332213647 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fc2e806c5f0 0x7fc2e806eaa0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:56.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:56.871+0000 7fc309000640 1 --2- 172.21.15.62:0/2332213647 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc3040ff580 0x7fc304100960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:56.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:56.871+0000 7fc309000640 1 --2- 172.21.15.62:0/2332213647 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc3040fe390 0x7fc304102320 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:56.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:56.871+0000 7fc309000640 1 -- 172.21.15.62:0/2332213647 >> 172.21.15.62:0/2332213647 conn(0x7fc3040f98f0 msgr2=0x7fc304067310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:27:56.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:56.871+0000 7fc309000640 1 -- 172.21.15.62:0/2332213647 shutdown_connections 2024-04-20T21:27:56.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:56.871+0000 7fc309000640 1 -- 172.21.15.62:0/2332213647 wait complete. 2024-04-20T21:27:56.888 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-04-20T21:27:57.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:57.078+0000 7fc056200640 1 -- 172.21.15.62:0/3887608609 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc050074200 msgr2=0x7fc0500726f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:27:57.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:57.078+0000 7fc056200640 1 --2- 172.21.15.62:0/3887608609 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc050074200 0x7fc0500726f0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fc040009880 tx=0x7fc04002bce0 comp rx=0 tx=0).stop 2024-04-20T21:27:57.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:57.078+0000 7fc056200640 1 -- 172.21.15.62:0/3887608609 shutdown_connections 2024-04-20T21:27:57.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:57.078+0000 7fc056200640 1 --2- 172.21.15.62:0/3887608609 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc050072c30 0x7fc0500730a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:57.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:57.078+0000 7fc056200640 1 --2- 172.21.15.62:0/3887608609 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc050074200 0x7fc0500726f0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:57.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:57.078+0000 7fc056200640 1 -- 172.21.15.62:0/3887608609 >> 172.21.15.62:0/3887608609 conn(0x7fc0500faab0 msgr2=0x7fc0500fcec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:27:57.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:57.078+0000 7fc056200640 1 -- 172.21.15.62:0/3887608609 shutdown_connections 2024-04-20T21:27:57.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:57.078+0000 7fc056200640 1 -- 172.21.15.62:0/3887608609 wait complete. 2024-04-20T21:27:57.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:57.079+0000 7fc056200640 1 Processor -- start 2024-04-20T21:27:57.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:57.079+0000 7fc056200640 1 -- start start 2024-04-20T21:27:57.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:57.080+0000 7fc056200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc050072c30 0x7fc05019b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:27:57.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:57.080+0000 7fc056200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc050074200 0x7fc05019b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:27:57.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:57.080+0000 7fc056200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc05019bed0 con 0x7fc050072c30 2024-04-20T21:27:57.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:57.080+0000 7fc056200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc05019c040 con 0x7fc050074200 2024-04-20T21:27:57.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:57.080+0000 7fc054e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc050072c30 0x7fc05019b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:27:57.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:57.080+0000 7fc054e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc050072c30 0x7fc05019b3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:36048/0 (socket says 172.21.15.62:36048) 2024-04-20T21:27:57.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:57.080+0000 7fc054e00640 1 -- 172.21.15.62:0/1343523748 learned_addr learned my addr 172.21.15.62:0/1343523748 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:27:57.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:57.080+0000 7fc047e00640 1 --2- 172.21.15.62:0/1343523748 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc050074200 0x7fc05019b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:27:57.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:57.080+0000 7fc054e00640 1 -- 172.21.15.62:0/1343523748 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc050074200 msgr2=0x7fc05019b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:27:57.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:57.080+0000 7fc054e00640 1 --2- 172.21.15.62:0/1343523748 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc050074200 0x7fc05019b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:57.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:57.080+0000 7fc054e00640 1 -- 172.21.15.62:0/1343523748 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc040009560 con 0x7fc050072c30 2024-04-20T21:27:57.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:57.081+0000 7fc054e00640 1 --2- 172.21.15.62:0/1343523748 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc050072c30 0x7fc05019b3c0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fc04002c220 tx=0x7fc040004dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:27:57.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:57.081+0000 7fc04e000640 1 -- 172.21.15.62:0/1343523748 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc0400044a0 con 0x7fc050072c30 2024-04-20T21:27:57.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:57.081+0000 7fc04e000640 1 -- 172.21.15.62:0/1343523748 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc0400054b0 con 0x7fc050072c30 2024-04-20T21:27:57.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:57.081+0000 7fc056200640 1 -- 172.21.15.62:0/1343523748 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0501a0a70 con 0x7fc050072c30 2024-04-20T21:27:57.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:57.081+0000 7fc04e000640 1 -- 172.21.15.62:0/1343523748 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc040034440 con 0x7fc050072c30 2024-04-20T21:27:57.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:57.081+0000 7fc056200640 1 -- 172.21.15.62:0/1343523748 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0501a0f70 con 0x7fc050072c30 2024-04-20T21:27:57.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:57.082+0000 7fc056200640 1 -- 172.21.15.62:0/1343523748 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc05010e740 con 0x7fc050072c30 2024-04-20T21:27:57.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:57.083+0000 7fc04e000640 1 -- 172.21.15.62:0/1343523748 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7fc040005000 con 0x7fc050072c30 2024-04-20T21:27:57.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:57.084+0000 7fc04e000640 1 --2- 172.21.15.62:0/1343523748 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fc03406c5f0 0x7fc03406eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:27:57.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:57.084+0000 7fc04e000640 1 -- 172.21.15.62:0/1343523748 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5468+0+0 (secure 0 0 0) 0x7fc0400add90 con 0x7fc050072c30 2024-04-20T21:27:57.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:57.084+0000 7fc047e00640 1 --2- 172.21.15.62:0/1343523748 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fc03406c5f0 0x7fc03406eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:27:57.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:57.085+0000 7fc047e00640 1 --2- 172.21.15.62:0/1343523748 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fc03406c5f0 0x7fc03406eaa0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fc038005fd0 tx=0x7fc038005850 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:27:57.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:57.089+0000 7fc04e000640 1 -- 172.21.15.62:0/1343523748 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fc0400784f0 con 0x7fc050072c30 2024-04-20T21:27:57.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:57.293+0000 7fc056200640 1 -- 172.21.15.62:0/1343523748 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7fc0501a12a0 con 0x7fc050072c30 2024-04-20T21:27:57.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:57.294+0000 7fc04e000640 1 -- 172.21.15.62:0/1343523748 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v45) v1 ==== 107+0+11 (secure 0 0 0) 0x7fc040041020 con 0x7fc050072c30 2024-04-20T21:27:57.293 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 32 2024-04-20T21:27:57.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:57.296+0000 7fc056200640 1 -- 172.21.15.62:0/1343523748 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fc03406c5f0 msgr2=0x7fc03406eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:27:57.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:57.296+0000 7fc056200640 1 --2- 172.21.15.62:0/1343523748 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fc03406c5f0 0x7fc03406eaa0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fc038005fd0 tx=0x7fc038005850 comp rx=0 tx=0).stop 2024-04-20T21:27:57.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:57.297+0000 7fc056200640 1 -- 172.21.15.62:0/1343523748 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc050072c30 msgr2=0x7fc05019b3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:27:57.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:57.297+0000 7fc056200640 1 --2- 172.21.15.62:0/1343523748 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc050072c30 0x7fc05019b3c0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fc04002c220 tx=0x7fc040004dc0 comp rx=0 tx=0).stop 2024-04-20T21:27:57.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:57.297+0000 7fc056200640 1 -- 172.21.15.62:0/1343523748 shutdown_connections 2024-04-20T21:27:57.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:57.297+0000 7fc056200640 1 --2- 172.21.15.62:0/1343523748 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fc03406c5f0 0x7fc03406eaa0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:57.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:57.297+0000 7fc056200640 1 --2- 172.21.15.62:0/1343523748 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc050074200 0x7fc05019b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:57.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:57.297+0000 7fc056200640 1 --2- 172.21.15.62:0/1343523748 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc050072c30 0x7fc05019b3c0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:57.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:57.297+0000 7fc056200640 1 -- 172.21.15.62:0/1343523748 >> 172.21.15.62:0/1343523748 conn(0x7fc0500faab0 msgr2=0x7fc050100e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:27:57.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:57.297+0000 7fc056200640 1 -- 172.21.15.62:0/1343523748 shutdown_connections 2024-04-20T21:27:57.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:57.297+0000 7fc056200640 1 -- 172.21.15.62:0/1343523748 wait complete. 2024-04-20T21:27:57.316 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-20T21:27:57.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:57.529+0000 7f3ce0e00640 1 -- 172.21.15.62:0/2478472826 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3cdc102840 msgr2=0x7f3cdc104c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:27:57.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:57.529+0000 7f3ce0e00640 1 --2- 172.21.15.62:0/2478472826 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3cdc102840 0x7f3cdc104c20 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f3cc40098b0 tx=0x7f3cc402bcd0 comp rx=0 tx=0).stop 2024-04-20T21:27:57.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:57.529+0000 7f3ce0e00640 1 -- 172.21.15.62:0/2478472826 shutdown_connections 2024-04-20T21:27:57.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:57.529+0000 7f3ce0e00640 1 --2- 172.21.15.62:0/2478472826 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3cdc102840 0x7f3cdc104c20 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:57.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:57.529+0000 7f3ce0e00640 1 --2- 172.21.15.62:0/2478472826 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3cdc0fff20 0x7f3cdc102300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:57.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:57.529+0000 7f3ce0e00640 1 -- 172.21.15.62:0/2478472826 >> 172.21.15.62:0/2478472826 conn(0x7f3cdc0f9900 msgr2=0x7f3cdc0fbd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:27:57.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:57.529+0000 7f3ce0e00640 1 -- 172.21.15.62:0/2478472826 shutdown_connections 2024-04-20T21:27:57.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:57.529+0000 7f3ce0e00640 1 -- 172.21.15.62:0/2478472826 wait complete. 2024-04-20T21:27:57.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:57.530+0000 7f3ce0e00640 1 Processor -- start 2024-04-20T21:27:57.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:57.531+0000 7f3ce0e00640 1 -- start start 2024-04-20T21:27:57.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:57.531+0000 7f3ce0e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3cdc0fff20 0x7f3cdc196f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:27:57.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:57.531+0000 7f3ce0e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3cdc102840 0x7f3cdc197490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:27:57.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:57.531+0000 7f3ce0e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cdc197a60 con 0x7f3cdc0fff20 2024-04-20T21:27:57.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:57.531+0000 7f3ce0e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cdc197bd0 con 0x7f3cdc102840 2024-04-20T21:27:57.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:57.531+0000 7f3cdb400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3cdc0fff20 0x7f3cdc196f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:27:57.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:57.531+0000 7f3cdb400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3cdc0fff20 0x7f3cdc196f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:36064/0 (socket says 172.21.15.62:36064) 2024-04-20T21:27:57.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:57.531+0000 7f3cdb400640 1 -- 172.21.15.62:0/4192161406 learned_addr learned my addr 172.21.15.62:0/4192161406 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:27:57.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:57.531+0000 7f3cd3e00640 1 --2- 172.21.15.62:0/4192161406 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3cdc102840 0x7f3cdc197490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:27:57.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:57.532+0000 7f3cdb400640 1 -- 172.21.15.62:0/4192161406 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3cdc102840 msgr2=0x7f3cdc197490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:27:57.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:57.532+0000 7f3cdb400640 1 --2- 172.21.15.62:0/4192161406 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3cdc102840 0x7f3cdc197490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:57.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:57.532+0000 7f3cdb400640 1 -- 172.21.15.62:0/4192161406 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cc4009560 con 0x7f3cdc0fff20 2024-04-20T21:27:57.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:57.532+0000 7f3cd3e00640 1 --2- 172.21.15.62:0/4192161406 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3cdc102840 0x7f3cdc197490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T21:27:57.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:57.532+0000 7f3cdb400640 1 --2- 172.21.15.62:0/4192161406 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3cdc0fff20 0x7f3cdc196f50 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f3ccc00d6f0 tx=0x7f3ccc00dbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:27:57.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:57.532+0000 7f3cd8c00640 1 -- 172.21.15.62:0/4192161406 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3ccc009a30 con 0x7f3cdc0fff20 2024-04-20T21:27:57.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:57.532+0000 7f3cd8c00640 1 -- 172.21.15.62:0/4192161406 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f3ccc0054d0 con 0x7f3cdc0fff20 2024-04-20T21:27:57.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:57.532+0000 7f3ce0e00640 1 -- 172.21.15.62:0/4192161406 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cdc19c660 con 0x7f3cdc0fff20 2024-04-20T21:27:57.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:57.532+0000 7f3cd8c00640 1 -- 172.21.15.62:0/4192161406 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3ccc010460 con 0x7f3cdc0fff20 2024-04-20T21:27:57.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:57.532+0000 7f3ce0e00640 1 -- 172.21.15.62:0/4192161406 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cdc19cbc0 con 0x7f3cdc0fff20 2024-04-20T21:27:57.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:57.534+0000 7f3ce0e00640 1 -- 172.21.15.62:0/4192161406 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cdc0fb450 con 0x7f3cdc0fff20 2024-04-20T21:27:57.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:57.534+0000 7f3cd8c00640 1 -- 172.21.15.62:0/4192161406 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f3ccc009b90 con 0x7f3cdc0fff20 2024-04-20T21:27:57.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:57.535+0000 7f3cd8c00640 1 --2- 172.21.15.62:0/4192161406 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f3cc006c6b0 0x7f3cc006eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:27:57.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:57.535+0000 7f3cd8c00640 1 -- 172.21.15.62:0/4192161406 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5468+0+0 (secure 0 0 0) 0x7f3ccc08b420 con 0x7f3cdc0fff20 2024-04-20T21:27:57.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:57.536+0000 7f3cd3e00640 1 --2- 172.21.15.62:0/4192161406 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f3cc006c6b0 0x7f3cc006eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:27:57.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:57.537+0000 7f3cd3e00640 1 --2- 172.21.15.62:0/4192161406 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f3cc006c6b0 0x7f3cc006eb60 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f3cc4036040 tx=0x7f3cc4005fd0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:27:57.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:57.540+0000 7f3cd8c00640 1 -- 172.21.15.62:0/4192161406 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f3ccc055b00 con 0x7f3cdc0fff20 2024-04-20T21:27:57.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:57 smithi189 ceph-mon[21562]: from='client.? 172.21.15.62:0/960100890' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-20T21:27:57.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:57 smithi189 ceph-mon[21562]: from='client.? 172.21.15.62:0/2332213647' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-20T21:27:57.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:57 smithi189 ceph-mon[21562]: from='client.? 172.21.15.62:0/1343523748' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-20T21:27:57.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:57.778+0000 7f3ce0e00640 1 -- 172.21.15.62:0/4192161406 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3cdc108160 con 0x7f3cdc0fff20 2024-04-20T21:27:57.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:57.779+0000 7f3cd8c00640 1 -- 172.21.15.62:0/4192161406 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+4771 (secure 0 0 0) 0x7f3ccc089e60 con 0x7f3cdc0fff20 2024-04-20T21:27:57.779 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:27:57.779 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":14,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-20T21:27:21.032337+0000","modified":"2024-04-20T21:27:43.814916+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14458,"mds_1":24279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14458":{"gid":14458,"name":"cephfs.smithi062.ryqblb","rank":0,"incarnation":3,"state":"up:active","state_seq":6,"addr":"172.21.15.62:6827/1147127123","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":1147127123},{"type":"v1","addr":"172.21.15.62:6827","nonce":1147127123}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14474":{"gid":14474,"name":"cephfs.smithi062.ozizwi","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":3,"addr":"172.21.15.62:6829/3494174293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":3494174293},{"type":"v1","addr":"172.21.15.62:6829","nonce":3494174293}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24279":{"gid":24279,"name":"cephfs.smithi189.dvimer","rank":1,"incarnation":8,"state":"up:active","state_seq":5,"addr":"172.21.15.189:6825/237262514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":237262514},{"type":"v1","addr":"172.21.15.189:6825","nonce":237262514}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24291":{"gid":24291,"name":"cephfs.smithi189.xbgspw","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":2,"addr":"172.21.15.189:6827/1495356258","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":1495356258},{"type":"v1","addr":"172.21.15.189:6827","nonce":1495356258}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-20T21:27:57.779 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 14 2024-04-20T21:27:57.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:57.782+0000 7f3ce0e00640 1 -- 172.21.15.62:0/4192161406 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f3cc006c6b0 msgr2=0x7f3cc006eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:27:57.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:57.782+0000 7f3ce0e00640 1 --2- 172.21.15.62:0/4192161406 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f3cc006c6b0 0x7f3cc006eb60 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f3cc4036040 tx=0x7f3cc4005fd0 comp rx=0 tx=0).stop 2024-04-20T21:27:57.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:57.782+0000 7f3ce0e00640 1 -- 172.21.15.62:0/4192161406 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3cdc0fff20 msgr2=0x7f3cdc196f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:27:57.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:57.782+0000 7f3ce0e00640 1 --2- 172.21.15.62:0/4192161406 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3cdc0fff20 0x7f3cdc196f50 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f3ccc00d6f0 tx=0x7f3ccc00dbb0 comp rx=0 tx=0).stop 2024-04-20T21:27:57.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:57.782+0000 7f3ce0e00640 1 -- 172.21.15.62:0/4192161406 shutdown_connections 2024-04-20T21:27:57.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:57.782+0000 7f3ce0e00640 1 --2- 172.21.15.62:0/4192161406 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f3cc006c6b0 0x7f3cc006eb60 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:57.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:57.782+0000 7f3ce0e00640 1 --2- 172.21.15.62:0/4192161406 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3cdc102840 0x7f3cdc197490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:57.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:57.782+0000 7f3ce0e00640 1 --2- 172.21.15.62:0/4192161406 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3cdc0fff20 0x7f3cdc196f50 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:27:57.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:57.782+0000 7f3ce0e00640 1 -- 172.21.15.62:0/4192161406 >> 172.21.15.62:0/4192161406 conn(0x7f3cdc0f9900 msgr2=0x7f3cdc0fe5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:27:57.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:57.782+0000 7f3ce0e00640 1 -- 172.21.15.62:0/4192161406 shutdown_connections 2024-04-20T21:27:57.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:27:57.782+0000 7f3ce0e00640 1 -- 172.21.15.62:0/4192161406 wait complete. 2024-04-20T21:27:57.784 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:57 smithi062 ceph-mon[16221]: from='client.? 172.21.15.62:0/960100890' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-20T21:27:57.784 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:57 smithi062 ceph-mon[16221]: from='client.? 172.21.15.62:0/2332213647' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-20T21:27:57.784 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:57 smithi062 ceph-mon[16221]: from='client.? 172.21.15.62:0/1343523748' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-20T21:27:57.799 DEBUG:tasks.fs:fs fscid=1,name=cephfs state = {'epoch': 14, 'max_mds': 2, 'flags': 50} 2024-04-20T21:27:57.799 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-04-20T21:27:57.823 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-04-20T21:27:57.823 INFO:tasks.ceph_fuse:config is {'client.0': {}, 'client.1': {}} 2024-04-20T21:27:57.823 INFO:tasks.ceph_fuse:client.0 config is {} 2024-04-20T21:27:57.823 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-20T21:27:57.823 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-20T21:27:57.824 INFO:tasks.ceph_fuse:client.1 config is {} 2024-04-20T21:27:57.824 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-20T21:27:57.824 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-20T21:27:57.824 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-20T21:27:57.824 DEBUG:teuthology.orchestra.run.smithi189:> ip netns list 2024-04-20T21:27:57.883 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-20T21:27:57.883 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip link delete ceph-brx 2024-04-20T21:27:57.970 INFO:teuthology.orchestra.run.smithi189.stderr:Cannot find device "ceph-brx" 2024-04-20T21:27:57.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-20T21:27:57.974 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-20T21:27:57.974 DEBUG:teuthology.orchestra.run.smithi062:> ip netns list 2024-04-20T21:27:58.032 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-20T21:27:58.033 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip link delete ceph-brx 2024-04-20T21:27:58.117 INFO:teuthology.orchestra.run.smithi062.stderr:Cannot find device "ceph-brx" 2024-04-20T21:27:58.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-20T21:27:58.120 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-04-20T21:27:58.121 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -- ceph fs ls 2024-04-20T21:27:58.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:58 smithi189 ceph-mon[21562]: from='client.? 172.21.15.62:0/4192161406' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-20T21:27:58.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:58 smithi189 ceph-mon[21562]: pgmap v240: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:27:58.784 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:58 smithi062 ceph-mon[16221]: from='client.? 172.21.15.62:0/4192161406' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-20T21:27:58.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:58 smithi062 ceph-mon[16221]: pgmap v240: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:27:58.856 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T21:28:00.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:27:59 smithi189 ceph-mon[21562]: pgmap v241: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:28:00.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:27:59 smithi062 ceph-mon[16221]: pgmap v241: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:28:01.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:01.852+0000 7f207a400700 1 -- 172.21.15.62:0/2978436412 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f207410b640 msgr2=0x7f207410ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:28:01.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:01.852+0000 7f207a400700 1 --2- 172.21.15.62:0/2978436412 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f207410b640 0x7f207410ba10 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f2068009910 tx=0x7f2068009c20 comp rx=0 tx=0).stop 2024-04-20T21:28:01.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:01.852+0000 7f207a400700 1 -- 172.21.15.62:0/2978436412 shutdown_connections 2024-04-20T21:28:01.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:01.852+0000 7f207a400700 1 --2- 172.21.15.62:0/2978436412 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2074106b80 0x7f2074106ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:28:01.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:01.852+0000 7f207a400700 1 --2- 172.21.15.62:0/2978436412 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f207410b640 0x7f207410ba10 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:28:01.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:01.852+0000 7f207a400700 1 -- 172.21.15.62:0/2978436412 >> 172.21.15.62:0/2978436412 conn(0x7f2074075e90 msgr2=0x7f20740782a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:28:01.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:01.852+0000 7f207a400700 1 -- 172.21.15.62:0/2978436412 shutdown_connections 2024-04-20T21:28:01.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:01.852+0000 7f207a400700 1 -- 172.21.15.62:0/2978436412 wait complete. 2024-04-20T21:28:01.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:01.854+0000 7f207a400700 1 Processor -- start 2024-04-20T21:28:01.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:01.854+0000 7f207a400700 1 -- start start 2024-04-20T21:28:01.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:01.855+0000 7f207a400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2074106b80 0x7f207419c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:28:01.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:01.855+0000 7f207a400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f207410b640 0x7f207419c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:28:01.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:01.855+0000 7f207a400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f207419cdb0 con 0x7f207410b640 2024-04-20T21:28:01.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:01.855+0000 7f207a400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20741a0b40 con 0x7f2074106b80 2024-04-20T21:28:01.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:01.855+0000 7f2073e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f207410b640 0x7f207419c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:28:01.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:01.855+0000 7f2073e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f207410b640 0x7f207419c6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55346/0 (socket says 172.21.15.62:55346) 2024-04-20T21:28:01.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:01.855+0000 7f2073e00700 1 -- 172.21.15.62:0/1367933297 learned_addr learned my addr 172.21.15.62:0/1367933297 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:28:01.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:01.855+0000 7f2079000700 1 --2- 172.21.15.62:0/1367933297 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2074106b80 0x7f207419c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:28:01.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:01.856+0000 7f2073e00700 1 -- 172.21.15.62:0/1367933297 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2074106b80 msgr2=0x7f207419c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:28:01.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:01.856+0000 7f2073e00700 1 --2- 172.21.15.62:0/1367933297 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2074106b80 0x7f207419c190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:28:01.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:01.856+0000 7f2073e00700 1 -- 172.21.15.62:0/1367933297 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20680095f0 con 0x7f207410b640 2024-04-20T21:28:01.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:01.856+0000 7f2073e00700 1 --2- 172.21.15.62:0/1367933297 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f207410b640 0x7f207419c6d0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f206000d8d0 tx=0x7f206000dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:28:01.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:01.856+0000 7f2071600700 1 -- 172.21.15.62:0/1367933297 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f20600098a0 con 0x7f207410b640 2024-04-20T21:28:01.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:01.856+0000 7f2071600700 1 -- 172.21.15.62:0/1367933297 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f2060010460 con 0x7f207410b640 2024-04-20T21:28:01.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:01.856+0000 7f207a400700 1 -- 172.21.15.62:0/1367933297 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20741a0e20 con 0x7f207410b640 2024-04-20T21:28:01.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:01.856+0000 7f2071600700 1 -- 172.21.15.62:0/1367933297 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f206000f5f0 con 0x7f207410b640 2024-04-20T21:28:01.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:01.857+0000 7f207a400700 1 -- 172.21.15.62:0/1367933297 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20741a1370 con 0x7f207410b640 2024-04-20T21:28:01.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:01.858+0000 7f207a400700 1 -- 172.21.15.62:0/1367933297 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2074062380 con 0x7f207410b640 2024-04-20T21:28:01.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:01.859+0000 7f2071600700 1 -- 172.21.15.62:0/1367933297 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f2060009a00 con 0x7f207410b640 2024-04-20T21:28:01.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:01.860+0000 7f2071600700 1 --2- 172.21.15.62:0/1367933297 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f205c06c680 0x7f205c06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:28:01.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:01.860+0000 7f2071600700 1 -- 172.21.15.62:0/1367933297 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5468+0+0 (secure 0 0 0) 0x7f206008b750 con 0x7f207410b640 2024-04-20T21:28:01.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:01.860+0000 7f2079000700 1 --2- 172.21.15.62:0/1367933297 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f205c06c680 0x7f205c06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:28:01.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:01.861+0000 7f2079000700 1 --2- 172.21.15.62:0/1367933297 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f205c06c680 0x7f205c06eb30 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f206800b5c0 tx=0x7f206801a040 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:28:01.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:01.865+0000 7f2071600700 1 -- 172.21.15.62:0/1367933297 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f2060055e10 con 0x7f207410b640 2024-04-20T21:28:02.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:02.134+0000 7f207a400700 1 -- 172.21.15.62:0/1367933297 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f207419d4f0 con 0x7f207410b640 2024-04-20T21:28:02.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:02.135+0000 7f2071600700 1 -- 172.21.15.62:0/1367933297 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v14) v1 ==== 53+0+83 (secure 0 0 0) 0x7f2060059430 con 0x7f207410b640 2024-04-20T21:28:02.135 INFO:teuthology.orchestra.run.smithi062.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2024-04-20T21:28:02.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:02.138+0000 7f207a400700 1 -- 172.21.15.62:0/1367933297 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f205c06c680 msgr2=0x7f205c06eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:28:02.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:02.138+0000 7f207a400700 1 --2- 172.21.15.62:0/1367933297 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f205c06c680 0x7f205c06eb30 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f206800b5c0 tx=0x7f206801a040 comp rx=0 tx=0).stop 2024-04-20T21:28:02.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:02.138+0000 7f207a400700 1 -- 172.21.15.62:0/1367933297 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f207410b640 msgr2=0x7f207419c6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:28:02.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:02.138+0000 7f207a400700 1 --2- 172.21.15.62:0/1367933297 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f207410b640 0x7f207419c6d0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f206000d8d0 tx=0x7f206000dbe0 comp rx=0 tx=0).stop 2024-04-20T21:28:02.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:02.138+0000 7f207a400700 1 -- 172.21.15.62:0/1367933297 shutdown_connections 2024-04-20T21:28:02.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:02.138+0000 7f207a400700 1 --2- 172.21.15.62:0/1367933297 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f205c06c680 0x7f205c06eb30 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:28:02.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:02.138+0000 7f207a400700 1 --2- 172.21.15.62:0/1367933297 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2074106b80 0x7f207419c190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:28:02.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:02.138+0000 7f207a400700 1 --2- 172.21.15.62:0/1367933297 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f207410b640 0x7f207419c6d0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:28:02.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:02.138+0000 7f207a400700 1 -- 172.21.15.62:0/1367933297 >> 172.21.15.62:0/1367933297 conn(0x7f2074075e90 msgr2=0x7f2074076660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:28:02.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:02.139+0000 7f207a400700 1 -- 172.21.15.62:0/1367933297 shutdown_connections 2024-04-20T21:28:02.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:02.139+0000 7f207a400700 1 -- 172.21.15.62:0/1367933297 wait complete. 2024-04-20T21:28:02.528 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:28:02 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:28:02.528 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:28:02 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:28:02.528 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:28:02 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:28:02.528 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:28:02 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T21:28:02.528 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:28:02 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:28:02.528 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:28:02 smithi062 ceph-mon[16221]: pgmap v242: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:28:02.528 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:28:02 smithi062 ceph-mon[16221]: from='client.? 172.21.15.62:0/1367933297' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-20T21:28:02.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:28:02 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:28:02.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:28:02 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:28:02.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:28:02 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:28:02.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:28:02 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T21:28:02.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:28:02 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:28:02.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:28:02 smithi189 ceph-mon[21562]: pgmap v242: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:28:02.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:28:02 smithi189 ceph-mon[21562]: from='client.? 172.21.15.62:0/1367933297' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-20T21:28:03.005 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-20T21:28:03.005 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-20T21:28:03.005 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi062.front.sepia.ceph.com 2024-04-20T21:28:03.005 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-20T21:28:03.005 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-20T21:28:03.005 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-20T21:28:03.005 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-20T21:28:03.005 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-20T21:28:03.005 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-20T21:28:03.005 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-20T21:28:03.006 DEBUG:teuthology.orchestra.run.smithi062:> ip addr 2024-04-20T21:28:03.065 INFO:teuthology.orchestra.run.smithi062.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-20T21:28:03.065 INFO:teuthology.orchestra.run.smithi062.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-20T21:28:03.065 INFO:teuthology.orchestra.run.smithi062.stdout: inet 127.0.0.1/8 scope host lo 2024-04-20T21:28:03.065 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-04-20T21:28:03.066 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 ::1/128 scope host 2024-04-20T21:28:03.066 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-04-20T21:28:03.066 INFO:teuthology.orchestra.run.smithi062.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-20T21:28:03.066 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 0c:c4:7a:a9:aa:14 brd ff:ff:ff:ff:ff:ff 2024-04-20T21:28:03.066 INFO:teuthology.orchestra.run.smithi062.stdout: altname enp7s0f0 2024-04-20T21:28:03.066 INFO:teuthology.orchestra.run.smithi062.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-20T21:28:03.066 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 0c:c4:7a:a9:aa:15 brd ff:ff:ff:ff:ff:ff 2024-04-20T21:28:03.066 INFO:teuthology.orchestra.run.smithi062.stdout: altname enp7s0f1 2024-04-20T21:28:03.066 INFO:teuthology.orchestra.run.smithi062.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-20T21:28:03.066 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 0c:c4:7a:bb:d1:c2 brd ff:ff:ff:ff:ff:ff 2024-04-20T21:28:03.066 INFO:teuthology.orchestra.run.smithi062.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-20T21:28:03.066 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 0c:c4:7a:bb:d1:c3 brd ff:ff:ff:ff:ff:ff 2024-04-20T21:28:03.066 INFO:teuthology.orchestra.run.smithi062.stdout: inet 172.21.15.62/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-20T21:28:03.067 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft 42148sec preferred_lft 42148sec 2024-04-20T21:28:03.067 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 fe80::ec4:7aff:febb:d1c3/64 scope link 2024-04-20T21:28:03.067 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-04-20T21:28:03.069 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-20T21:28:03.069 DEBUG:teuthology.orchestra.run.smithi062:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-20T21:28:03.069 DEBUG:teuthology.orchestra.run.smithi062:> set -e 2024-04-20T21:28:03.069 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip link add name ceph-brx type bridge 2024-04-20T21:28:03.070 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip addr flush dev ceph-brx 2024-04-20T21:28:03.070 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip link set ceph-brx up 2024-04-20T21:28:03.070 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-20T21:28:03.070 DEBUG:teuthology.orchestra.run.smithi062:> ') 2024-04-20T21:28:03.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:03 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-20T21:28:03.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:03 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-20T21:28:03.568 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-20T21:28:03.569 DEBUG:teuthology.orchestra.run.smithi062:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-20T21:28:03.638 INFO:teuthology.orchestra.run.smithi062.stdout:1 2024-04-20T21:28:03.642 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-20T21:28:03.642 DEBUG:teuthology.orchestra.run.smithi062:> route 2024-04-20T21:28:03.694 INFO:teuthology.orchestra.run.smithi062.stdout:Kernel IP routing table 2024-04-20T21:28:03.694 INFO:teuthology.orchestra.run.smithi062.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-20T21:28:03.695 INFO:teuthology.orchestra.run.smithi062.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-20T21:28:03.695 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-20T21:28:03.695 INFO:teuthology.orchestra.run.smithi062.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-20T21:28:03.696 DEBUG:teuthology.orchestra.run.smithi062:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-20T21:28:03.696 DEBUG:teuthology.orchestra.run.smithi062:> set -e 2024-04-20T21:28:03.696 DEBUG:teuthology.orchestra.run.smithi062:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-20T21:28:03.696 DEBUG:teuthology.orchestra.run.smithi062:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-20T21:28:03.696 DEBUG:teuthology.orchestra.run.smithi062:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-20T21:28:03.696 DEBUG:teuthology.orchestra.run.smithi062:> ') 2024-04-20T21:28:03.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:03 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-20T21:28:04.090 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:28:03 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:28:04.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:28:03 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:28:04.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:04 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-20T21:28:04.428 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-20T21:28:04.428 DEBUG:teuthology.orchestra.run.smithi062:> ip netns list 2024-04-20T21:28:04.489 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-20T21:28:04.489 DEBUG:teuthology.orchestra.run.smithi062:> ip netns list-id 2024-04-20T21:28:04.545 DEBUG:teuthology.orchestra.run.smithi062:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-20T21:28:04.545 DEBUG:teuthology.orchestra.run.smithi062:> set -e 2024-04-20T21:28:04.545 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-20T21:28:04.545 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-04-20T21:28:04.545 DEBUG:teuthology.orchestra.run.smithi062:> ') 2024-04-20T21:28:04.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:04 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-20T21:28:04.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:04 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-20T21:28:04.741 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-04-20T21:28:04.741 DEBUG:teuthology.orchestra.run.smithi062:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-20T21:28:04.741 DEBUG:teuthology.orchestra.run.smithi062:> set -e 2024-04-20T21:28:04.741 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-04-20T21:28:04.741 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-20T21:28:04.741 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-04-20T21:28:04.742 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-04-20T21:28:04.742 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-04-20T21:28:04.742 DEBUG:teuthology.orchestra.run.smithi062:> ') 2024-04-20T21:28:04.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:04 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-20T21:28:04.934 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:28:04 smithi062 ceph-mon[16221]: pgmap v243: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:28:05.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:05 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-20T21:28:05.042 DEBUG:teuthology.orchestra.run.smithi062:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-20T21:28:05.043 DEBUG:teuthology.orchestra.run.smithi062:> set -e 2024-04-20T21:28:05.043 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip link set brx.0 up 2024-04-20T21:28:05.043 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip link set dev brx.0 master ceph-brx 2024-04-20T21:28:05.043 DEBUG:teuthology.orchestra.run.smithi062:> ') 2024-04-20T21:28:05.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:05 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-20T21:28:05.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:28:04 smithi189 ceph-mon[21562]: pgmap v243: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:28:05.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:05 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-20T21:28:05.247 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-04-20T21:28:05.247 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-20T21:28:05.247 DEBUG:teuthology.orchestra.run.smithi062:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-20T21:28:05.296 INFO:teuthology.orchestra.run.smithi062.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-20T21:28:05.297 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-20T21:28:05.297 DEBUG:teuthology.orchestra.run.smithi062:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-20T21:28:05.347 DEBUG:teuthology.orchestra.run.smithi062:> sudo modprobe fuse 2024-04-20T21:28:05.432 DEBUG:teuthology.orchestra.run.smithi062:> cat /proc/self/mounts | awk '{print $2}' 2024-04-20T21:28:05.482 INFO:teuthology.orchestra.run.smithi062.stdout:/proc 2024-04-20T21:28:05.482 INFO:teuthology.orchestra.run.smithi062.stdout:/sys 2024-04-20T21:28:05.482 INFO:teuthology.orchestra.run.smithi062.stdout:/dev 2024-04-20T21:28:05.482 INFO:teuthology.orchestra.run.smithi062.stdout:/sys/kernel/security 2024-04-20T21:28:05.482 INFO:teuthology.orchestra.run.smithi062.stdout:/dev/shm 2024-04-20T21:28:05.482 INFO:teuthology.orchestra.run.smithi062.stdout:/dev/pts 2024-04-20T21:28:05.482 INFO:teuthology.orchestra.run.smithi062.stdout:/run 2024-04-20T21:28:05.482 INFO:teuthology.orchestra.run.smithi062.stdout:/sys/fs/cgroup 2024-04-20T21:28:05.483 INFO:teuthology.orchestra.run.smithi062.stdout:/sys/fs/pstore 2024-04-20T21:28:05.483 INFO:teuthology.orchestra.run.smithi062.stdout:/sys/fs/bpf 2024-04-20T21:28:05.483 INFO:teuthology.orchestra.run.smithi062.stdout:/ 2024-04-20T21:28:05.483 INFO:teuthology.orchestra.run.smithi062.stdout:/sys/fs/selinux 2024-04-20T21:28:05.483 INFO:teuthology.orchestra.run.smithi062.stdout:/proc/sys/fs/binfmt_misc 2024-04-20T21:28:05.483 INFO:teuthology.orchestra.run.smithi062.stdout:/dev/mqueue 2024-04-20T21:28:05.483 INFO:teuthology.orchestra.run.smithi062.stdout:/dev/hugepages 2024-04-20T21:28:05.483 INFO:teuthology.orchestra.run.smithi062.stdout:/sys/kernel/debug 2024-04-20T21:28:05.483 INFO:teuthology.orchestra.run.smithi062.stdout:/sys/kernel/tracing 2024-04-20T21:28:05.483 INFO:teuthology.orchestra.run.smithi062.stdout:/sys/fs/fuse/connections 2024-04-20T21:28:05.483 INFO:teuthology.orchestra.run.smithi062.stdout:/sys/kernel/config 2024-04-20T21:28:05.483 INFO:teuthology.orchestra.run.smithi062.stdout:/run/credentials/systemd-sysctl.service 2024-04-20T21:28:05.483 INFO:teuthology.orchestra.run.smithi062.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-20T21:28:05.483 INFO:teuthology.orchestra.run.smithi062.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-20T21:28:05.483 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/nfs/rpc_pipefs 2024-04-20T21:28:05.484 INFO:teuthology.orchestra.run.smithi062.stdout:/run/user/1000 2024-04-20T21:28:05.484 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/ceph 2024-04-20T21:28:05.484 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/containers/storage/overlay 2024-04-20T21:28:05.484 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/containers/storage/overlay/16a1e24fe87108f9a09755623533eaa35aeb70fc8038e6830adfacbe474c0608/merged 2024-04-20T21:28:05.484 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/containers/storage/overlay/56363e63013dd494e11c2ea4ecb4c7961043ee43358e96db8b1b0b1d03592066/merged 2024-04-20T21:28:05.484 INFO:teuthology.orchestra.run.smithi062.stdout:/run/user/0 2024-04-20T21:28:05.484 INFO:teuthology.orchestra.run.smithi062.stdout:/proc/sys/fs/binfmt_misc 2024-04-20T21:28:05.484 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/containers/storage/overlay/959da03849f3a72c163dd6fe428cbdc6d94d2f0cd62fc135a7c8b5d7e3e04ace/merged 2024-04-20T21:28:05.484 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/containers/storage/overlay/6de5486caf91e71a2db7092bfb5a467d0266db1b0404fd888a5187f262dda45f/merged 2024-04-20T21:28:05.484 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/containers/storage/overlay/57a97588d72b587a47f3580f45494df70f19098938c97d46fa9ee23a8ed0b9d5/merged 2024-04-20T21:28:05.484 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/containers/storage/overlay/cf13e3e7c49c1c67631dfd73b5f40fef21fe8217e29ec2c0b81ea23ba8475512/merged 2024-04-20T21:28:05.484 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/containers/storage/overlay/6a4ccb7e994d03a40cc4a3e51debd19ecc5b469ddcc300dbf326bcb9f140559e/merged 2024-04-20T21:28:05.484 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/containers/storage/overlay/2a4ef8507fcca0903a625d54b08c9f2dcd14e9b15ad25caf0b2533c977029b88/merged 2024-04-20T21:28:05.484 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/containers/storage/overlay/63ec892759cd58426d3a69eda980e976ebc892b718242541e795568862864ef8/merged 2024-04-20T21:28:05.484 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/containers/storage/overlay/44aab6a826b6d6553294ce8529249712ab1183561807ac672e89e1978a7b78cd/merged 2024-04-20T21:28:05.484 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/containers/storage/overlay/4de94971d1aabfae39954003ae2f2b8a76ddc7975d702bfa65333b9ec17ee32e/merged 2024-04-20T21:28:05.484 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/containers/storage/overlay/495da4a982dca3c8a01fcd2336bb05322f814ca74dd8d35a7a121d1453ae4daf/merged 2024-04-20T21:28:05.485 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/containers/storage/overlay/e951e9eeb06eec6a50365c05f251e8350e307b977dcfd4b58f6c2a49aaa8527f/merged 2024-04-20T21:28:05.485 INFO:teuthology.orchestra.run.smithi062.stdout:/run/netns 2024-04-20T21:28:05.485 INFO:teuthology.orchestra.run.smithi062.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-20T21:28:05.485 INFO:teuthology.orchestra.run.smithi062.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-20T21:28:05.485 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-20T21:28:05.485 DEBUG:teuthology.orchestra.run.smithi062:> ls /sys/fs/fuse/connections 2024-04-20T21:28:05.528 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-20T21:28:05.529 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-04-20T21:28:05.576 DEBUG:teuthology.orchestra.run.smithi062:> sudo modprobe fuse 2024-04-20T21:28:05.646 DEBUG:teuthology.orchestra.run.smithi062:> cat /proc/self/mounts | awk '{print $2}' 2024-04-20T21:28:05.693 INFO:teuthology.orchestra.run.smithi062.stdout:/proc 2024-04-20T21:28:05.693 INFO:teuthology.orchestra.run.smithi062.stdout:/sys 2024-04-20T21:28:05.693 INFO:teuthology.orchestra.run.smithi062.stdout:/dev 2024-04-20T21:28:05.693 INFO:teuthology.orchestra.run.smithi062.stdout:/sys/kernel/security 2024-04-20T21:28:05.693 INFO:teuthology.orchestra.run.smithi062.stdout:/dev/shm 2024-04-20T21:28:05.693 INFO:teuthology.orchestra.run.smithi062.stdout:/dev/pts 2024-04-20T21:28:05.693 INFO:teuthology.orchestra.run.smithi062.stdout:/run 2024-04-20T21:28:05.693 INFO:teuthology.orchestra.run.smithi062.stdout:/sys/fs/cgroup 2024-04-20T21:28:05.693 INFO:teuthology.orchestra.run.smithi062.stdout:/sys/fs/pstore 2024-04-20T21:28:05.693 INFO:teuthology.orchestra.run.smithi062.stdout:/sys/fs/bpf 2024-04-20T21:28:05.693 INFO:teuthology.orchestra.run.smithi062.stdout:/ 2024-04-20T21:28:05.693 INFO:teuthology.orchestra.run.smithi062.stdout:/sys/fs/selinux 2024-04-20T21:28:05.693 INFO:teuthology.orchestra.run.smithi062.stdout:/proc/sys/fs/binfmt_misc 2024-04-20T21:28:05.694 INFO:teuthology.orchestra.run.smithi062.stdout:/dev/mqueue 2024-04-20T21:28:05.694 INFO:teuthology.orchestra.run.smithi062.stdout:/dev/hugepages 2024-04-20T21:28:05.694 INFO:teuthology.orchestra.run.smithi062.stdout:/sys/kernel/debug 2024-04-20T21:28:05.694 INFO:teuthology.orchestra.run.smithi062.stdout:/sys/kernel/tracing 2024-04-20T21:28:05.694 INFO:teuthology.orchestra.run.smithi062.stdout:/sys/fs/fuse/connections 2024-04-20T21:28:05.694 INFO:teuthology.orchestra.run.smithi062.stdout:/sys/kernel/config 2024-04-20T21:28:05.694 INFO:teuthology.orchestra.run.smithi062.stdout:/run/credentials/systemd-sysctl.service 2024-04-20T21:28:05.694 INFO:teuthology.orchestra.run.smithi062.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-20T21:28:05.694 INFO:teuthology.orchestra.run.smithi062.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-20T21:28:05.694 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/nfs/rpc_pipefs 2024-04-20T21:28:05.694 INFO:teuthology.orchestra.run.smithi062.stdout:/run/user/1000 2024-04-20T21:28:05.694 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/ceph 2024-04-20T21:28:05.694 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/containers/storage/overlay 2024-04-20T21:28:05.695 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/containers/storage/overlay/16a1e24fe87108f9a09755623533eaa35aeb70fc8038e6830adfacbe474c0608/merged 2024-04-20T21:28:05.695 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/containers/storage/overlay/56363e63013dd494e11c2ea4ecb4c7961043ee43358e96db8b1b0b1d03592066/merged 2024-04-20T21:28:05.695 INFO:teuthology.orchestra.run.smithi062.stdout:/run/user/0 2024-04-20T21:28:05.695 INFO:teuthology.orchestra.run.smithi062.stdout:/proc/sys/fs/binfmt_misc 2024-04-20T21:28:05.695 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/containers/storage/overlay/959da03849f3a72c163dd6fe428cbdc6d94d2f0cd62fc135a7c8b5d7e3e04ace/merged 2024-04-20T21:28:05.695 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/containers/storage/overlay/6de5486caf91e71a2db7092bfb5a467d0266db1b0404fd888a5187f262dda45f/merged 2024-04-20T21:28:05.695 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/containers/storage/overlay/57a97588d72b587a47f3580f45494df70f19098938c97d46fa9ee23a8ed0b9d5/merged 2024-04-20T21:28:05.695 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/containers/storage/overlay/cf13e3e7c49c1c67631dfd73b5f40fef21fe8217e29ec2c0b81ea23ba8475512/merged 2024-04-20T21:28:05.695 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/containers/storage/overlay/6a4ccb7e994d03a40cc4a3e51debd19ecc5b469ddcc300dbf326bcb9f140559e/merged 2024-04-20T21:28:05.695 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/containers/storage/overlay/2a4ef8507fcca0903a625d54b08c9f2dcd14e9b15ad25caf0b2533c977029b88/merged 2024-04-20T21:28:05.695 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/containers/storage/overlay/63ec892759cd58426d3a69eda980e976ebc892b718242541e795568862864ef8/merged 2024-04-20T21:28:05.695 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/containers/storage/overlay/44aab6a826b6d6553294ce8529249712ab1183561807ac672e89e1978a7b78cd/merged 2024-04-20T21:28:05.695 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/containers/storage/overlay/4de94971d1aabfae39954003ae2f2b8a76ddc7975d702bfa65333b9ec17ee32e/merged 2024-04-20T21:28:05.695 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/containers/storage/overlay/495da4a982dca3c8a01fcd2336bb05322f814ca74dd8d35a7a121d1453ae4daf/merged 2024-04-20T21:28:05.695 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/containers/storage/overlay/e951e9eeb06eec6a50365c05f251e8350e307b977dcfd4b58f6c2a49aaa8527f/merged 2024-04-20T21:28:05.696 INFO:teuthology.orchestra.run.smithi062.stdout:/run/netns 2024-04-20T21:28:05.696 INFO:teuthology.orchestra.run.smithi062.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-20T21:28:05.696 INFO:teuthology.orchestra.run.smithi062.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-20T21:28:05.696 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-20T21:28:05.697 DEBUG:teuthology.orchestra.run.smithi062:> ls /sys/fs/fuse/connections 2024-04-20T21:28:05.779 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi062.stderr:2024-04-20T21:28:05.780+0000 7f95d4a42480 -1 init, newargv = 0x55ca9bbf3730 newargc=15 2024-04-20T21:28:05.779 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi062.stderr:ceph-fuse[54098]: starting ceph client 2024-04-20T21:28:05.801 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi062.stderr:ceph-fuse[54098]: starting fuse 2024-04-20T21:28:06.840 DEBUG:teuthology.orchestra.run.smithi062:> sudo modprobe fuse 2024-04-20T21:28:06.945 DEBUG:teuthology.orchestra.run.smithi062:> cat /proc/self/mounts | awk '{print $2}' 2024-04-20T21:28:06.999 INFO:teuthology.orchestra.run.smithi062.stdout:/proc 2024-04-20T21:28:06.999 INFO:teuthology.orchestra.run.smithi062.stdout:/sys 2024-04-20T21:28:06.999 INFO:teuthology.orchestra.run.smithi062.stdout:/dev 2024-04-20T21:28:06.999 INFO:teuthology.orchestra.run.smithi062.stdout:/sys/kernel/security 2024-04-20T21:28:06.999 INFO:teuthology.orchestra.run.smithi062.stdout:/dev/shm 2024-04-20T21:28:06.999 INFO:teuthology.orchestra.run.smithi062.stdout:/dev/pts 2024-04-20T21:28:06.999 INFO:teuthology.orchestra.run.smithi062.stdout:/run 2024-04-20T21:28:06.999 INFO:teuthology.orchestra.run.smithi062.stdout:/sys/fs/cgroup 2024-04-20T21:28:06.999 INFO:teuthology.orchestra.run.smithi062.stdout:/sys/fs/pstore 2024-04-20T21:28:06.999 INFO:teuthology.orchestra.run.smithi062.stdout:/sys/fs/bpf 2024-04-20T21:28:06.999 INFO:teuthology.orchestra.run.smithi062.stdout:/ 2024-04-20T21:28:06.999 INFO:teuthology.orchestra.run.smithi062.stdout:/sys/fs/selinux 2024-04-20T21:28:06.999 INFO:teuthology.orchestra.run.smithi062.stdout:/proc/sys/fs/binfmt_misc 2024-04-20T21:28:07.000 INFO:teuthology.orchestra.run.smithi062.stdout:/dev/mqueue 2024-04-20T21:28:07.000 INFO:teuthology.orchestra.run.smithi062.stdout:/dev/hugepages 2024-04-20T21:28:07.000 INFO:teuthology.orchestra.run.smithi062.stdout:/sys/kernel/debug 2024-04-20T21:28:07.000 INFO:teuthology.orchestra.run.smithi062.stdout:/sys/kernel/tracing 2024-04-20T21:28:07.000 INFO:teuthology.orchestra.run.smithi062.stdout:/sys/fs/fuse/connections 2024-04-20T21:28:07.000 INFO:teuthology.orchestra.run.smithi062.stdout:/sys/kernel/config 2024-04-20T21:28:07.000 INFO:teuthology.orchestra.run.smithi062.stdout:/run/credentials/systemd-sysctl.service 2024-04-20T21:28:07.000 INFO:teuthology.orchestra.run.smithi062.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-20T21:28:07.000 INFO:teuthology.orchestra.run.smithi062.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-20T21:28:07.000 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/nfs/rpc_pipefs 2024-04-20T21:28:07.000 INFO:teuthology.orchestra.run.smithi062.stdout:/run/user/1000 2024-04-20T21:28:07.000 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/ceph 2024-04-20T21:28:07.000 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/containers/storage/overlay 2024-04-20T21:28:07.000 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/containers/storage/overlay/16a1e24fe87108f9a09755623533eaa35aeb70fc8038e6830adfacbe474c0608/merged 2024-04-20T21:28:07.000 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/containers/storage/overlay/56363e63013dd494e11c2ea4ecb4c7961043ee43358e96db8b1b0b1d03592066/merged 2024-04-20T21:28:07.001 INFO:teuthology.orchestra.run.smithi062.stdout:/run/user/0 2024-04-20T21:28:07.001 INFO:teuthology.orchestra.run.smithi062.stdout:/proc/sys/fs/binfmt_misc 2024-04-20T21:28:07.001 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/containers/storage/overlay/959da03849f3a72c163dd6fe428cbdc6d94d2f0cd62fc135a7c8b5d7e3e04ace/merged 2024-04-20T21:28:07.001 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/containers/storage/overlay/6de5486caf91e71a2db7092bfb5a467d0266db1b0404fd888a5187f262dda45f/merged 2024-04-20T21:28:07.001 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/containers/storage/overlay/57a97588d72b587a47f3580f45494df70f19098938c97d46fa9ee23a8ed0b9d5/merged 2024-04-20T21:28:07.001 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/containers/storage/overlay/cf13e3e7c49c1c67631dfd73b5f40fef21fe8217e29ec2c0b81ea23ba8475512/merged 2024-04-20T21:28:07.001 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/containers/storage/overlay/6a4ccb7e994d03a40cc4a3e51debd19ecc5b469ddcc300dbf326bcb9f140559e/merged 2024-04-20T21:28:07.001 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/containers/storage/overlay/2a4ef8507fcca0903a625d54b08c9f2dcd14e9b15ad25caf0b2533c977029b88/merged 2024-04-20T21:28:07.001 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/containers/storage/overlay/63ec892759cd58426d3a69eda980e976ebc892b718242541e795568862864ef8/merged 2024-04-20T21:28:07.001 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/containers/storage/overlay/44aab6a826b6d6553294ce8529249712ab1183561807ac672e89e1978a7b78cd/merged 2024-04-20T21:28:07.001 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/containers/storage/overlay/4de94971d1aabfae39954003ae2f2b8a76ddc7975d702bfa65333b9ec17ee32e/merged 2024-04-20T21:28:07.001 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/containers/storage/overlay/495da4a982dca3c8a01fcd2336bb05322f814ca74dd8d35a7a121d1453ae4daf/merged 2024-04-20T21:28:07.002 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/containers/storage/overlay/e951e9eeb06eec6a50365c05f251e8350e307b977dcfd4b58f6c2a49aaa8527f/merged 2024-04-20T21:28:07.002 INFO:teuthology.orchestra.run.smithi062.stdout:/run/netns 2024-04-20T21:28:07.002 INFO:teuthology.orchestra.run.smithi062.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-20T21:28:07.002 INFO:teuthology.orchestra.run.smithi062.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-20T21:28:07.002 INFO:teuthology.orchestra.run.smithi062.stdout:/home/ubuntu/cephtest/mnt.0 2024-04-20T21:28:07.003 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-20T21:28:07.003 DEBUG:teuthology.orchestra.run.smithi062:> ls /sys/fs/fuse/connections 2024-04-20T21:28:07.054 INFO:teuthology.orchestra.run.smithi062.stdout:155 2024-04-20T21:28:07.054 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [155] 2024-04-20T21:28:07.055 DEBUG:teuthology.orchestra.run.smithi062:> sudo stdin-killer -- python3 -c ' 2024-04-20T21:28:07.055 DEBUG:teuthology.orchestra.run.smithi062:> import glob 2024-04-20T21:28:07.055 DEBUG:teuthology.orchestra.run.smithi062:> import re 2024-04-20T21:28:07.055 DEBUG:teuthology.orchestra.run.smithi062:> import os 2024-04-20T21:28:07.055 DEBUG:teuthology.orchestra.run.smithi062:> import subprocess 2024-04-20T21:28:07.055 DEBUG:teuthology.orchestra.run.smithi062:> 2024-04-20T21:28:07.055 DEBUG:teuthology.orchestra.run.smithi062:> def _find_admin_socket(client_name): 2024-04-20T21:28:07.055 DEBUG:teuthology.orchestra.run.smithi062:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-04-20T21:28:07.055 DEBUG:teuthology.orchestra.run.smithi062:> files = glob.glob(asok_path) 2024-04-20T21:28:07.055 DEBUG:teuthology.orchestra.run.smithi062:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-04-20T21:28:07.055 DEBUG:teuthology.orchestra.run.smithi062:> 2024-04-20T21:28:07.055 DEBUG:teuthology.orchestra.run.smithi062:> # Given a non-glob path, it better be there 2024-04-20T21:28:07.055 DEBUG:teuthology.orchestra.run.smithi062:> if "*" not in asok_path: 2024-04-20T21:28:07.055 DEBUG:teuthology.orchestra.run.smithi062:> assert(len(files) == 1) 2024-04-20T21:28:07.055 DEBUG:teuthology.orchestra.run.smithi062:> return files[0] 2024-04-20T21:28:07.056 DEBUG:teuthology.orchestra.run.smithi062:> 2024-04-20T21:28:07.056 DEBUG:teuthology.orchestra.run.smithi062:> for f in files: 2024-04-20T21:28:07.056 DEBUG:teuthology.orchestra.run.smithi062:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-20T21:28:07.056 DEBUG:teuthology.orchestra.run.smithi062:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-20T21:28:07.056 DEBUG:teuthology.orchestra.run.smithi062:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-20T21:28:07.056 DEBUG:teuthology.orchestra.run.smithi062:> contents = proc_f.read() 2024-04-20T21:28:07.056 DEBUG:teuthology.orchestra.run.smithi062:> if mountpoint in contents: 2024-04-20T21:28:07.056 DEBUG:teuthology.orchestra.run.smithi062:> return f 2024-04-20T21:28:07.056 DEBUG:teuthology.orchestra.run.smithi062:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-20T21:28:07.056 DEBUG:teuthology.orchestra.run.smithi062:> 2024-04-20T21:28:07.056 DEBUG:teuthology.orchestra.run.smithi062:> print(_find_admin_socket("client.0")) 2024-04-20T21:28:07.056 DEBUG:teuthology.orchestra.run.smithi062:> ' 2024-04-20T21:28:07.226 INFO:teuthology.orchestra.run.smithi062.stdout:/var/run/ceph/ceph-client.0.54098.asok 2024-04-20T21:28:07.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:07 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-20T21:28:07.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:28:06 smithi189 ceph-mon[21562]: pgmap v244: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:28:07.240 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:28:06 smithi062 ceph-mon[16221]: pgmap v244: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:28:07.242 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.54098.asok 2024-04-20T21:28:07.242 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-20T21:28:07.242 DEBUG:teuthology.orchestra.run.smithi062:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.54098.asok status 2024-04-20T21:28:07.478 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:28:07.478 INFO:teuthology.orchestra.run.smithi062.stdout: "metadata": { 2024-04-20T21:28:07.478 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph_sha1": "5dd24139a1eada541a3bc16b6941c5dde975e26d", 2024-04-20T21:28:07.478 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph_version": "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)", 2024-04-20T21:28:07.478 INFO:teuthology.orchestra.run.smithi062.stdout: "entity_id": "0", 2024-04-20T21:28:07.478 INFO:teuthology.orchestra.run.smithi062.stdout: "hostname": "smithi062", 2024-04-20T21:28:07.478 INFO:teuthology.orchestra.run.smithi062.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-04-20T21:28:07.479 INFO:teuthology.orchestra.run.smithi062.stdout: "pid": "54098", 2024-04-20T21:28:07.479 INFO:teuthology.orchestra.run.smithi062.stdout: "root": "/" 2024-04-20T21:28:07.479 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:28:07.479 INFO:teuthology.orchestra.run.smithi062.stdout: "dentry_count": 0, 2024-04-20T21:28:07.479 INFO:teuthology.orchestra.run.smithi062.stdout: "dentry_pinned_count": 0, 2024-04-20T21:28:07.479 INFO:teuthology.orchestra.run.smithi062.stdout: "id": 14524, 2024-04-20T21:28:07.479 INFO:teuthology.orchestra.run.smithi062.stdout: "inst": { 2024-04-20T21:28:07.479 INFO:teuthology.orchestra.run.smithi062.stdout: "name": { 2024-04-20T21:28:07.479 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "client", 2024-04-20T21:28:07.479 INFO:teuthology.orchestra.run.smithi062.stdout: "num": 14524 2024-04-20T21:28:07.479 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:28:07.479 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": { 2024-04-20T21:28:07.479 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-04-20T21:28:07.479 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "192.168.0.1:0", 2024-04-20T21:28:07.479 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 3910576540 2024-04-20T21:28:07.479 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T21:28:07.480 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:28:07.480 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": { 2024-04-20T21:28:07.480 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-04-20T21:28:07.480 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "192.168.0.1:0", 2024-04-20T21:28:07.480 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 3910576540 2024-04-20T21:28:07.480 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:28:07.480 INFO:teuthology.orchestra.run.smithi062.stdout: "inst_str": "client.14524 192.168.0.1:0/3910576540", 2024-04-20T21:28:07.480 INFO:teuthology.orchestra.run.smithi062.stdout: "addr_str": "192.168.0.1:0/3910576540", 2024-04-20T21:28:07.480 INFO:teuthology.orchestra.run.smithi062.stdout: "inode_count": 1, 2024-04-20T21:28:07.480 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_epoch": 14, 2024-04-20T21:28:07.480 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_epoch": 45, 2024-04-20T21:28:07.480 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_epoch_barrier": 0, 2024-04-20T21:28:07.480 INFO:teuthology.orchestra.run.smithi062.stdout: "blocklisted": false, 2024-04-20T21:28:07.480 INFO:teuthology.orchestra.run.smithi062.stdout: "fs_name": "cephfs" 2024-04-20T21:28:07.480 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:28:07.494 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -- ceph fs ls 2024-04-20T21:28:07.908 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T21:28:08.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:28:07 smithi189 ceph-mon[21562]: pgmap v245: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:28:08.252 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:28:07 smithi062 ceph-mon[16221]: pgmap v245: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:28:10.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:10.198+0000 7f6bbc400700 1 -- 172.21.15.62:0/2392257280 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6bb4101420 msgr2=0x7f6bb41017f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:28:10.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:10.198+0000 7f6bbc400700 1 --2- 172.21.15.62:0/2392257280 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6bb4101420 0x7f6bb41017f0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f6ba8009910 tx=0x7f6ba8009c20 comp rx=0 tx=0).stop 2024-04-20T21:28:10.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:10.198+0000 7f6bbc400700 1 -- 172.21.15.62:0/2392257280 shutdown_connections 2024-04-20T21:28:10.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:10.198+0000 7f6bbc400700 1 --2- 172.21.15.62:0/2392257280 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6bb4101d30 0x7f6bb410a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:28:10.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:10.198+0000 7f6bbc400700 1 --2- 172.21.15.62:0/2392257280 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6bb4101420 0x7f6bb41017f0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:28:10.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:10.198+0000 7f6bbc400700 1 -- 172.21.15.62:0/2392257280 >> 172.21.15.62:0/2392257280 conn(0x7f6bb40fac80 msgr2=0x7f6bb40fd090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:28:10.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:10.198+0000 7f6bbc400700 1 -- 172.21.15.62:0/2392257280 shutdown_connections 2024-04-20T21:28:10.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:10.198+0000 7f6bbc400700 1 -- 172.21.15.62:0/2392257280 wait complete. 2024-04-20T21:28:10.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:10.199+0000 7f6bbc400700 1 Processor -- start 2024-04-20T21:28:10.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:10.199+0000 7f6bbc400700 1 -- start start 2024-04-20T21:28:10.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:10.200+0000 7f6bbc400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6bb4101420 0x7f6bb4195ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:28:10.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:10.200+0000 7f6bbc400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6bb4101d30 0x7f6bb4196410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:28:10.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:10.200+0000 7f6bbc400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb4196af0 con 0x7f6bb4101420 2024-04-20T21:28:10.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:10.200+0000 7f6bbc400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb419a880 con 0x7f6bb4101d30 2024-04-20T21:28:10.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:10.200+0000 7f6bb9a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6bb4101420 0x7f6bb4195ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:28:10.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:10.200+0000 7f6bb9a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6bb4101420 0x7f6bb4195ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55368/0 (socket says 172.21.15.62:55368) 2024-04-20T21:28:10.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:10.200+0000 7f6bb9a00700 1 -- 172.21.15.62:0/1547206962 learned_addr learned my addr 172.21.15.62:0/1547206962 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:28:10.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:10.201+0000 7f6bb9000700 1 --2- 172.21.15.62:0/1547206962 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6bb4101d30 0x7f6bb4196410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:28:10.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:10.201+0000 7f6bb9a00700 1 -- 172.21.15.62:0/1547206962 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6bb4101d30 msgr2=0x7f6bb4196410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:28:10.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:10.201+0000 7f6bb9a00700 1 --2- 172.21.15.62:0/1547206962 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6bb4101d30 0x7f6bb4196410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:28:10.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:10.201+0000 7f6bb9a00700 1 -- 172.21.15.62:0/1547206962 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ba80095f0 con 0x7f6bb4101420 2024-04-20T21:28:10.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:10.201+0000 7f6bb9000700 1 --2- 172.21.15.62:0/1547206962 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6bb4101d30 0x7f6bb4196410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T21:28:10.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:10.201+0000 7f6bb9a00700 1 --2- 172.21.15.62:0/1547206962 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6bb4101420 0x7f6bb4195ed0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f6ba800b5c0 tx=0x7f6ba800be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:28:10.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:10.201+0000 7f6bb2000700 1 -- 172.21.15.62:0/1547206962 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6ba801d070 con 0x7f6bb4101420 2024-04-20T21:28:10.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:10.201+0000 7f6bb2000700 1 -- 172.21.15.62:0/1547206962 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f6ba8022470 con 0x7f6bb4101420 2024-04-20T21:28:10.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:10.201+0000 7f6bbc400700 1 -- 172.21.15.62:0/1547206962 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bb419ab00 con 0x7f6bb4101420 2024-04-20T21:28:10.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:10.201+0000 7f6bb2000700 1 -- 172.21.15.62:0/1547206962 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6ba800f670 con 0x7f6bb4101420 2024-04-20T21:28:10.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:10.202+0000 7f6bbc400700 1 -- 172.21.15.62:0/1547206962 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bb419aff0 con 0x7f6bb4101420 2024-04-20T21:28:10.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:10.203+0000 7f6bbc400700 1 -- 172.21.15.62:0/1547206962 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bb41078e0 con 0x7f6bb4101420 2024-04-20T21:28:10.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:10.204+0000 7f6bb2000700 1 -- 172.21.15.62:0/1547206962 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f6ba8004c30 con 0x7f6bb4101420 2024-04-20T21:28:10.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:10.205+0000 7f6bb2000700 1 --2- 172.21.15.62:0/1547206962 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f6b9c06c680 0x7f6b9c06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:28:10.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:10.205+0000 7f6bb2000700 1 -- 172.21.15.62:0/1547206962 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5468+0+0 (secure 0 0 0) 0x7f6ba808cfd0 con 0x7f6bb4101420 2024-04-20T21:28:10.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:10.205+0000 7f6bb9000700 1 --2- 172.21.15.62:0/1547206962 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f6b9c06c680 0x7f6b9c06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:28:10.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:10.206+0000 7f6bb9000700 1 --2- 172.21.15.62:0/1547206962 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f6b9c06c680 0x7f6b9c06eb30 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f6bb41974f0 tx=0x7f6ba0008040 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:28:10.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:10.210+0000 7f6bb2000700 1 -- 172.21.15.62:0/1547206962 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f6ba80bf720 con 0x7f6bb4101420 2024-04-20T21:28:10.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:28:09 smithi189 ceph-mon[21562]: pgmap v246: 65 pgs: 65 active+clean; 581 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 85 B/s wr, 3 op/s 2024-04-20T21:28:10.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:28:09 smithi062 ceph-mon[16221]: pgmap v246: 65 pgs: 65 active+clean; 581 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 85 B/s wr, 3 op/s 2024-04-20T21:28:10.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:10.485+0000 7f6bbc400700 1 -- 172.21.15.62:0/1547206962 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f6bb404f2a0 con 0x7f6bb4101420 2024-04-20T21:28:10.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:10.486+0000 7f6bb2000700 1 -- 172.21.15.62:0/1547206962 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v14) v1 ==== 53+0+83 (secure 0 0 0) 0x7f6ba80576c0 con 0x7f6bb4101420 2024-04-20T21:28:10.485 INFO:teuthology.orchestra.run.smithi062.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2024-04-20T21:28:10.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:10.488+0000 7f6bbc400700 1 -- 172.21.15.62:0/1547206962 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f6b9c06c680 msgr2=0x7f6b9c06eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:28:10.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:10.488+0000 7f6bbc400700 1 --2- 172.21.15.62:0/1547206962 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f6b9c06c680 0x7f6b9c06eb30 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f6bb41974f0 tx=0x7f6ba0008040 comp rx=0 tx=0).stop 2024-04-20T21:28:10.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:10.488+0000 7f6bbc400700 1 -- 172.21.15.62:0/1547206962 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6bb4101420 msgr2=0x7f6bb4195ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:28:10.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:10.488+0000 7f6bbc400700 1 --2- 172.21.15.62:0/1547206962 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6bb4101420 0x7f6bb4195ed0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f6ba800b5c0 tx=0x7f6ba800be00 comp rx=0 tx=0).stop 2024-04-20T21:28:10.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:10.488+0000 7f6bbc400700 1 -- 172.21.15.62:0/1547206962 shutdown_connections 2024-04-20T21:28:10.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:10.488+0000 7f6bbc400700 1 --2- 172.21.15.62:0/1547206962 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f6b9c06c680 0x7f6b9c06eb30 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:28:10.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:10.488+0000 7f6bbc400700 1 --2- 172.21.15.62:0/1547206962 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6bb4101420 0x7f6bb4195ed0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:28:10.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:10.488+0000 7f6bbc400700 1 --2- 172.21.15.62:0/1547206962 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6bb4101d30 0x7f6bb4196410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:28:10.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:10.488+0000 7f6bbc400700 1 -- 172.21.15.62:0/1547206962 >> 172.21.15.62:0/1547206962 conn(0x7f6bb40fac80 msgr2=0x7f6bb4104a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:28:10.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:10.489+0000 7f6bbc400700 1 -- 172.21.15.62:0/1547206962 shutdown_connections 2024-04-20T21:28:10.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:10.489+0000 7f6bbc400700 1 -- 172.21.15.62:0/1547206962 wait complete. 2024-04-20T21:28:11.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:28:10 smithi189 ceph-mon[21562]: from='client.? 172.21.15.62:0/1547206962' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-20T21:28:11.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:28:10 smithi062 ceph-mon[16221]: from='client.? 172.21.15.62:0/1547206962' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-20T21:28:11.410 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-20T21:28:11.410 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-20T21:28:11.410 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi189.front.sepia.ceph.com 2024-04-20T21:28:11.410 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-04-20T21:28:11.410 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-20T21:28:11.410 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-20T21:28:11.411 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-20T21:28:11.411 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-20T21:28:11.411 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-20T21:28:11.411 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-20T21:28:11.411 DEBUG:teuthology.orchestra.run.smithi189:> ip addr 2024-04-20T21:28:11.468 INFO:teuthology.orchestra.run.smithi189.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-20T21:28:11.469 INFO:teuthology.orchestra.run.smithi189.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-20T21:28:11.469 INFO:teuthology.orchestra.run.smithi189.stdout: inet 127.0.0.1/8 scope host lo 2024-04-20T21:28:11.469 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-20T21:28:11.469 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 ::1/128 scope host 2024-04-20T21:28:11.469 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-20T21:28:11.469 INFO:teuthology.orchestra.run.smithi189.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-20T21:28:11.469 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 0c:c4:7a:d9:3b:18 brd ff:ff:ff:ff:ff:ff 2024-04-20T21:28:11.469 INFO:teuthology.orchestra.run.smithi189.stdout: altname enp7s0f0 2024-04-20T21:28:11.469 INFO:teuthology.orchestra.run.smithi189.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-20T21:28:11.469 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 0c:c4:7a:d9:3b:19 brd ff:ff:ff:ff:ff:ff 2024-04-20T21:28:11.469 INFO:teuthology.orchestra.run.smithi189.stdout: altname enp7s0f1 2024-04-20T21:28:11.469 INFO:teuthology.orchestra.run.smithi189.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-20T21:28:11.469 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 0c:c4:7a:88:6d:f0 brd ff:ff:ff:ff:ff:ff 2024-04-20T21:28:11.469 INFO:teuthology.orchestra.run.smithi189.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-20T21:28:11.469 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 0c:c4:7a:88:6d:f1 brd ff:ff:ff:ff:ff:ff 2024-04-20T21:28:11.470 INFO:teuthology.orchestra.run.smithi189.stdout: inet 172.21.15.189/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-20T21:28:11.470 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft 42227sec preferred_lft 42227sec 2024-04-20T21:28:11.470 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 fe80::ec4:7aff:fe88:6df1/64 scope link 2024-04-20T21:28:11.470 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-04-20T21:28:11.471 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-20T21:28:11.471 DEBUG:teuthology.orchestra.run.smithi189:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-20T21:28:11.471 DEBUG:teuthology.orchestra.run.smithi189:> set -e 2024-04-20T21:28:11.471 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip link add name ceph-brx type bridge 2024-04-20T21:28:11.471 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip addr flush dev ceph-brx 2024-04-20T21:28:11.471 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip link set ceph-brx up 2024-04-20T21:28:11.471 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-20T21:28:11.471 DEBUG:teuthology.orchestra.run.smithi189:> ') 2024-04-20T21:28:11.578 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:28:11 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-20T21:28:11.956 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:28:11 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-20T21:28:11.965 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-20T21:28:11.965 DEBUG:teuthology.orchestra.run.smithi189:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-20T21:28:12.038 INFO:teuthology.orchestra.run.smithi189.stdout:1 2024-04-20T21:28:12.042 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-20T21:28:12.042 DEBUG:teuthology.orchestra.run.smithi189:> route 2024-04-20T21:28:12.094 INFO:teuthology.orchestra.run.smithi189.stdout:Kernel IP routing table 2024-04-20T21:28:12.094 INFO:teuthology.orchestra.run.smithi189.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-20T21:28:12.095 INFO:teuthology.orchestra.run.smithi189.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-20T21:28:12.095 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-20T21:28:12.095 INFO:teuthology.orchestra.run.smithi189.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-20T21:28:12.095 DEBUG:teuthology.orchestra.run.smithi189:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-20T21:28:12.096 DEBUG:teuthology.orchestra.run.smithi189:> set -e 2024-04-20T21:28:12.096 DEBUG:teuthology.orchestra.run.smithi189:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-20T21:28:12.096 DEBUG:teuthology.orchestra.run.smithi189:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-20T21:28:12.096 DEBUG:teuthology.orchestra.run.smithi189:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-20T21:28:12.096 DEBUG:teuthology.orchestra.run.smithi189:> ') 2024-04-20T21:28:12.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:28:12 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-20T21:28:12.914 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:28:12 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-20T21:28:12.924 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-20T21:28:12.924 DEBUG:teuthology.orchestra.run.smithi189:> ip netns list 2024-04-20T21:28:12.973 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-20T21:28:12.973 DEBUG:teuthology.orchestra.run.smithi189:> ip netns list-id 2024-04-20T21:28:13.021 DEBUG:teuthology.orchestra.run.smithi189:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-20T21:28:13.021 DEBUG:teuthology.orchestra.run.smithi189:> set -e 2024-04-20T21:28:13.021 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-20T21:28:13.021 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-04-20T21:28:13.021 DEBUG:teuthology.orchestra.run.smithi189:> ') 2024-04-20T21:28:13.120 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:28:13 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-20T21:28:13.143 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:28:12 smithi189 ceph-mon[21562]: pgmap v247: 65 pgs: 65 active+clean; 581 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.7 KiB/s rd, 341 B/s wr, 5 op/s 2024-04-20T21:28:13.191 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:28:13 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-20T21:28:13.202 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-04-20T21:28:13.202 DEBUG:teuthology.orchestra.run.smithi189:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-20T21:28:13.202 DEBUG:teuthology.orchestra.run.smithi189:> set -e 2024-04-20T21:28:13.202 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-04-20T21:28:13.202 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-20T21:28:13.202 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-04-20T21:28:13.202 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-04-20T21:28:13.202 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-04-20T21:28:13.202 DEBUG:teuthology.orchestra.run.smithi189:> ') 2024-04-20T21:28:13.273 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:28:12 smithi062 ceph-mon[16221]: pgmap v247: 65 pgs: 65 active+clean; 581 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.7 KiB/s rd, 341 B/s wr, 5 op/s 2024-04-20T21:28:13.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:28:13 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-20T21:28:13.502 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:28:13 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-20T21:28:13.512 DEBUG:teuthology.orchestra.run.smithi189:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-20T21:28:13.512 DEBUG:teuthology.orchestra.run.smithi189:> set -e 2024-04-20T21:28:13.512 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip link set brx.0 up 2024-04-20T21:28:13.512 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip link set dev brx.0 master ceph-brx 2024-04-20T21:28:13.512 DEBUG:teuthology.orchestra.run.smithi189:> ') 2024-04-20T21:28:13.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:28:13 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-20T21:28:13.705 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:28:13 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-20T21:28:13.714 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-04-20T21:28:13.714 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-20T21:28:13.715 DEBUG:teuthology.orchestra.run.smithi189:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-04-20T21:28:13.761 INFO:teuthology.orchestra.run.smithi189.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-04-20T21:28:13.761 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-20T21:28:13.761 DEBUG:teuthology.orchestra.run.smithi189:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-04-20T21:28:13.852 DEBUG:teuthology.orchestra.run.smithi189:> sudo modprobe fuse 2024-04-20T21:28:13.937 DEBUG:teuthology.orchestra.run.smithi189:> cat /proc/self/mounts | awk '{print $2}' 2024-04-20T21:28:13.995 INFO:teuthology.orchestra.run.smithi189.stdout:/proc 2024-04-20T21:28:13.995 INFO:teuthology.orchestra.run.smithi189.stdout:/sys 2024-04-20T21:28:13.995 INFO:teuthology.orchestra.run.smithi189.stdout:/dev 2024-04-20T21:28:13.995 INFO:teuthology.orchestra.run.smithi189.stdout:/sys/kernel/security 2024-04-20T21:28:13.995 INFO:teuthology.orchestra.run.smithi189.stdout:/dev/shm 2024-04-20T21:28:13.995 INFO:teuthology.orchestra.run.smithi189.stdout:/dev/pts 2024-04-20T21:28:13.995 INFO:teuthology.orchestra.run.smithi189.stdout:/run 2024-04-20T21:28:13.995 INFO:teuthology.orchestra.run.smithi189.stdout:/sys/fs/cgroup 2024-04-20T21:28:13.995 INFO:teuthology.orchestra.run.smithi189.stdout:/sys/fs/pstore 2024-04-20T21:28:13.995 INFO:teuthology.orchestra.run.smithi189.stdout:/sys/fs/bpf 2024-04-20T21:28:13.995 INFO:teuthology.orchestra.run.smithi189.stdout:/ 2024-04-20T21:28:13.995 INFO:teuthology.orchestra.run.smithi189.stdout:/sys/fs/selinux 2024-04-20T21:28:13.995 INFO:teuthology.orchestra.run.smithi189.stdout:/proc/sys/fs/binfmt_misc 2024-04-20T21:28:13.995 INFO:teuthology.orchestra.run.smithi189.stdout:/dev/hugepages 2024-04-20T21:28:13.996 INFO:teuthology.orchestra.run.smithi189.stdout:/dev/mqueue 2024-04-20T21:28:13.996 INFO:teuthology.orchestra.run.smithi189.stdout:/sys/kernel/debug 2024-04-20T21:28:13.996 INFO:teuthology.orchestra.run.smithi189.stdout:/sys/kernel/tracing 2024-04-20T21:28:13.996 INFO:teuthology.orchestra.run.smithi189.stdout:/sys/fs/fuse/connections 2024-04-20T21:28:13.996 INFO:teuthology.orchestra.run.smithi189.stdout:/sys/kernel/config 2024-04-20T21:28:13.996 INFO:teuthology.orchestra.run.smithi189.stdout:/run/credentials/systemd-sysctl.service 2024-04-20T21:28:13.996 INFO:teuthology.orchestra.run.smithi189.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-20T21:28:13.996 INFO:teuthology.orchestra.run.smithi189.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-20T21:28:13.996 INFO:teuthology.orchestra.run.smithi189.stdout:/var/lib/nfs/rpc_pipefs 2024-04-20T21:28:13.996 INFO:teuthology.orchestra.run.smithi189.stdout:/run/user/1000 2024-04-20T21:28:13.996 INFO:teuthology.orchestra.run.smithi189.stdout:/var/lib/ceph 2024-04-20T21:28:13.996 INFO:teuthology.orchestra.run.smithi189.stdout:/run/user/0 2024-04-20T21:28:13.996 INFO:teuthology.orchestra.run.smithi189.stdout:/proc/sys/fs/binfmt_misc 2024-04-20T21:28:13.996 INFO:teuthology.orchestra.run.smithi189.stdout:/var/lib/containers/storage/overlay 2024-04-20T21:28:13.996 INFO:teuthology.orchestra.run.smithi189.stdout:/var/lib/containers/storage/overlay/52f64aa4bd870738c127951a4c4e91c7deb3a39761619d52fc8dee1ba586245e/merged 2024-04-20T21:28:13.997 INFO:teuthology.orchestra.run.smithi189.stdout:/var/lib/containers/storage/overlay/e2a76c7bb159a748fb86265cfd97606095cd3c8039fec313636bb80b1ce2502f/merged 2024-04-20T21:28:13.997 INFO:teuthology.orchestra.run.smithi189.stdout:/var/lib/containers/storage/overlay/e1bcabcb0ed940d835cf885e8512d8a55ce0676a8974aed2278dad31edd19f17/merged 2024-04-20T21:28:13.997 INFO:teuthology.orchestra.run.smithi189.stdout:/var/lib/containers/storage/overlay/2fa85e78693bec70c4635608a1bc61c40078970e88419d9cb1737272e7e83440/merged 2024-04-20T21:28:13.997 INFO:teuthology.orchestra.run.smithi189.stdout:/var/lib/containers/storage/overlay/77be9a94fa7ce24c9797bacf895dd3a51ca1aea9a8747bb0f7c130507604139f/merged 2024-04-20T21:28:13.997 INFO:teuthology.orchestra.run.smithi189.stdout:/var/lib/containers/storage/overlay/0a5aebd38fd3629d8ec8523e6b5f3c0372399ec69f12297771bad290abc28507/merged 2024-04-20T21:28:13.997 INFO:teuthology.orchestra.run.smithi189.stdout:/var/lib/containers/storage/overlay/8b4b03d4fe586c73490fbef26e0cff65fb0de12f111ac30a0028dfa354304357/merged 2024-04-20T21:28:13.997 INFO:teuthology.orchestra.run.smithi189.stdout:/var/lib/containers/storage/overlay/2121faec5dcd4b90d3407fb1355632d265abb213c8734fb9c861d5c7c4c45722/merged 2024-04-20T21:28:13.997 INFO:teuthology.orchestra.run.smithi189.stdout:/var/lib/containers/storage/overlay/b19e377c859b22076f3b6492d9f8f9f537d9b25a469852323585343aeef28e8b/merged 2024-04-20T21:28:13.997 INFO:teuthology.orchestra.run.smithi189.stdout:/var/lib/containers/storage/overlay/4407a615b1379018f4f969778f869fd8b874769a9e8a4756f8c82a1b24f8c1e4/merged 2024-04-20T21:28:13.997 INFO:teuthology.orchestra.run.smithi189.stdout:/run/netns 2024-04-20T21:28:13.997 INFO:teuthology.orchestra.run.smithi189.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-20T21:28:13.997 INFO:teuthology.orchestra.run.smithi189.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-20T21:28:13.998 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-20T21:28:13.998 DEBUG:teuthology.orchestra.run.smithi189:> ls /sys/fs/fuse/connections 2024-04-20T21:28:14.049 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-20T21:28:14.050 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-04-20T21:28:14.056 DEBUG:teuthology.orchestra.run.smithi189:> sudo modprobe fuse 2024-04-20T21:28:14.127 DEBUG:teuthology.orchestra.run.smithi189:> cat /proc/self/mounts | awk '{print $2}' 2024-04-20T21:28:14.168 INFO:teuthology.orchestra.run.smithi189.stdout:/proc 2024-04-20T21:28:14.168 INFO:teuthology.orchestra.run.smithi189.stdout:/sys 2024-04-20T21:28:14.168 INFO:teuthology.orchestra.run.smithi189.stdout:/dev 2024-04-20T21:28:14.168 INFO:teuthology.orchestra.run.smithi189.stdout:/sys/kernel/security 2024-04-20T21:28:14.168 INFO:teuthology.orchestra.run.smithi189.stdout:/dev/shm 2024-04-20T21:28:14.168 INFO:teuthology.orchestra.run.smithi189.stdout:/dev/pts 2024-04-20T21:28:14.168 INFO:teuthology.orchestra.run.smithi189.stdout:/run 2024-04-20T21:28:14.168 INFO:teuthology.orchestra.run.smithi189.stdout:/sys/fs/cgroup 2024-04-20T21:28:14.168 INFO:teuthology.orchestra.run.smithi189.stdout:/sys/fs/pstore 2024-04-20T21:28:14.168 INFO:teuthology.orchestra.run.smithi189.stdout:/sys/fs/bpf 2024-04-20T21:28:14.169 INFO:teuthology.orchestra.run.smithi189.stdout:/ 2024-04-20T21:28:14.169 INFO:teuthology.orchestra.run.smithi189.stdout:/sys/fs/selinux 2024-04-20T21:28:14.169 INFO:teuthology.orchestra.run.smithi189.stdout:/proc/sys/fs/binfmt_misc 2024-04-20T21:28:14.169 INFO:teuthology.orchestra.run.smithi189.stdout:/dev/hugepages 2024-04-20T21:28:14.169 INFO:teuthology.orchestra.run.smithi189.stdout:/dev/mqueue 2024-04-20T21:28:14.169 INFO:teuthology.orchestra.run.smithi189.stdout:/sys/kernel/debug 2024-04-20T21:28:14.169 INFO:teuthology.orchestra.run.smithi189.stdout:/sys/kernel/tracing 2024-04-20T21:28:14.169 INFO:teuthology.orchestra.run.smithi189.stdout:/sys/fs/fuse/connections 2024-04-20T21:28:14.169 INFO:teuthology.orchestra.run.smithi189.stdout:/sys/kernel/config 2024-04-20T21:28:14.169 INFO:teuthology.orchestra.run.smithi189.stdout:/run/credentials/systemd-sysctl.service 2024-04-20T21:28:14.169 INFO:teuthology.orchestra.run.smithi189.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-20T21:28:14.169 INFO:teuthology.orchestra.run.smithi189.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-20T21:28:14.169 INFO:teuthology.orchestra.run.smithi189.stdout:/var/lib/nfs/rpc_pipefs 2024-04-20T21:28:14.169 INFO:teuthology.orchestra.run.smithi189.stdout:/run/user/1000 2024-04-20T21:28:14.169 INFO:teuthology.orchestra.run.smithi189.stdout:/var/lib/ceph 2024-04-20T21:28:14.170 INFO:teuthology.orchestra.run.smithi189.stdout:/run/user/0 2024-04-20T21:28:14.170 INFO:teuthology.orchestra.run.smithi189.stdout:/proc/sys/fs/binfmt_misc 2024-04-20T21:28:14.170 INFO:teuthology.orchestra.run.smithi189.stdout:/var/lib/containers/storage/overlay 2024-04-20T21:28:14.170 INFO:teuthology.orchestra.run.smithi189.stdout:/var/lib/containers/storage/overlay/52f64aa4bd870738c127951a4c4e91c7deb3a39761619d52fc8dee1ba586245e/merged 2024-04-20T21:28:14.170 INFO:teuthology.orchestra.run.smithi189.stdout:/var/lib/containers/storage/overlay/e2a76c7bb159a748fb86265cfd97606095cd3c8039fec313636bb80b1ce2502f/merged 2024-04-20T21:28:14.170 INFO:teuthology.orchestra.run.smithi189.stdout:/var/lib/containers/storage/overlay/e1bcabcb0ed940d835cf885e8512d8a55ce0676a8974aed2278dad31edd19f17/merged 2024-04-20T21:28:14.170 INFO:teuthology.orchestra.run.smithi189.stdout:/var/lib/containers/storage/overlay/2fa85e78693bec70c4635608a1bc61c40078970e88419d9cb1737272e7e83440/merged 2024-04-20T21:28:14.170 INFO:teuthology.orchestra.run.smithi189.stdout:/var/lib/containers/storage/overlay/77be9a94fa7ce24c9797bacf895dd3a51ca1aea9a8747bb0f7c130507604139f/merged 2024-04-20T21:28:14.170 INFO:teuthology.orchestra.run.smithi189.stdout:/var/lib/containers/storage/overlay/0a5aebd38fd3629d8ec8523e6b5f3c0372399ec69f12297771bad290abc28507/merged 2024-04-20T21:28:14.170 INFO:teuthology.orchestra.run.smithi189.stdout:/var/lib/containers/storage/overlay/8b4b03d4fe586c73490fbef26e0cff65fb0de12f111ac30a0028dfa354304357/merged 2024-04-20T21:28:14.170 INFO:teuthology.orchestra.run.smithi189.stdout:/var/lib/containers/storage/overlay/2121faec5dcd4b90d3407fb1355632d265abb213c8734fb9c861d5c7c4c45722/merged 2024-04-20T21:28:14.170 INFO:teuthology.orchestra.run.smithi189.stdout:/var/lib/containers/storage/overlay/b19e377c859b22076f3b6492d9f8f9f537d9b25a469852323585343aeef28e8b/merged 2024-04-20T21:28:14.170 INFO:teuthology.orchestra.run.smithi189.stdout:/var/lib/containers/storage/overlay/4407a615b1379018f4f969778f869fd8b874769a9e8a4756f8c82a1b24f8c1e4/merged 2024-04-20T21:28:14.170 INFO:teuthology.orchestra.run.smithi189.stdout:/run/netns 2024-04-20T21:28:14.171 INFO:teuthology.orchestra.run.smithi189.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-20T21:28:14.171 INFO:teuthology.orchestra.run.smithi189.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-20T21:28:14.171 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-20T21:28:14.171 DEBUG:teuthology.orchestra.run.smithi189:> ls /sys/fs/fuse/connections 2024-04-20T21:28:14.236 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:28:13 smithi189 ceph-mon[21562]: pgmap v248: 65 pgs: 65 active+clean; 581 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.8 KiB/s rd, 341 B/s wr, 4 op/s 2024-04-20T21:28:14.275 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi189.stderr:2024-04-20T21:28:14.275+0000 7f5932d96480 -1 init, newargv = 0x5591433bdd80 newargc=15 2024-04-20T21:28:14.275 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi189.stderr:ceph-fuse[41413]: starting ceph client 2024-04-20T21:28:14.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:28:13 smithi062 ceph-mon[16221]: pgmap v248: 65 pgs: 65 active+clean; 581 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.8 KiB/s rd, 341 B/s wr, 4 op/s 2024-04-20T21:28:14.292 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi189.stderr:ceph-fuse[41413]: starting fuse 2024-04-20T21:28:15.358 DEBUG:teuthology.orchestra.run.smithi189:> sudo modprobe fuse 2024-04-20T21:28:15.449 DEBUG:teuthology.orchestra.run.smithi189:> cat /proc/self/mounts | awk '{print $2}' 2024-04-20T21:28:15.502 INFO:teuthology.orchestra.run.smithi189.stdout:/proc 2024-04-20T21:28:15.502 INFO:teuthology.orchestra.run.smithi189.stdout:/sys 2024-04-20T21:28:15.502 INFO:teuthology.orchestra.run.smithi189.stdout:/dev 2024-04-20T21:28:15.502 INFO:teuthology.orchestra.run.smithi189.stdout:/sys/kernel/security 2024-04-20T21:28:15.503 INFO:teuthology.orchestra.run.smithi189.stdout:/dev/shm 2024-04-20T21:28:15.503 INFO:teuthology.orchestra.run.smithi189.stdout:/dev/pts 2024-04-20T21:28:15.503 INFO:teuthology.orchestra.run.smithi189.stdout:/run 2024-04-20T21:28:15.503 INFO:teuthology.orchestra.run.smithi189.stdout:/sys/fs/cgroup 2024-04-20T21:28:15.503 INFO:teuthology.orchestra.run.smithi189.stdout:/sys/fs/pstore 2024-04-20T21:28:15.503 INFO:teuthology.orchestra.run.smithi189.stdout:/sys/fs/bpf 2024-04-20T21:28:15.503 INFO:teuthology.orchestra.run.smithi189.stdout:/ 2024-04-20T21:28:15.503 INFO:teuthology.orchestra.run.smithi189.stdout:/sys/fs/selinux 2024-04-20T21:28:15.503 INFO:teuthology.orchestra.run.smithi189.stdout:/proc/sys/fs/binfmt_misc 2024-04-20T21:28:15.503 INFO:teuthology.orchestra.run.smithi189.stdout:/dev/hugepages 2024-04-20T21:28:15.503 INFO:teuthology.orchestra.run.smithi189.stdout:/dev/mqueue 2024-04-20T21:28:15.503 INFO:teuthology.orchestra.run.smithi189.stdout:/sys/kernel/debug 2024-04-20T21:28:15.503 INFO:teuthology.orchestra.run.smithi189.stdout:/sys/kernel/tracing 2024-04-20T21:28:15.503 INFO:teuthology.orchestra.run.smithi189.stdout:/sys/fs/fuse/connections 2024-04-20T21:28:15.504 INFO:teuthology.orchestra.run.smithi189.stdout:/sys/kernel/config 2024-04-20T21:28:15.504 INFO:teuthology.orchestra.run.smithi189.stdout:/run/credentials/systemd-sysctl.service 2024-04-20T21:28:15.504 INFO:teuthology.orchestra.run.smithi189.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-20T21:28:15.504 INFO:teuthology.orchestra.run.smithi189.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-20T21:28:15.504 INFO:teuthology.orchestra.run.smithi189.stdout:/var/lib/nfs/rpc_pipefs 2024-04-20T21:28:15.504 INFO:teuthology.orchestra.run.smithi189.stdout:/run/user/1000 2024-04-20T21:28:15.504 INFO:teuthology.orchestra.run.smithi189.stdout:/var/lib/ceph 2024-04-20T21:28:15.504 INFO:teuthology.orchestra.run.smithi189.stdout:/run/user/0 2024-04-20T21:28:15.504 INFO:teuthology.orchestra.run.smithi189.stdout:/proc/sys/fs/binfmt_misc 2024-04-20T21:28:15.504 INFO:teuthology.orchestra.run.smithi189.stdout:/var/lib/containers/storage/overlay 2024-04-20T21:28:15.504 INFO:teuthology.orchestra.run.smithi189.stdout:/var/lib/containers/storage/overlay/52f64aa4bd870738c127951a4c4e91c7deb3a39761619d52fc8dee1ba586245e/merged 2024-04-20T21:28:15.504 INFO:teuthology.orchestra.run.smithi189.stdout:/var/lib/containers/storage/overlay/e2a76c7bb159a748fb86265cfd97606095cd3c8039fec313636bb80b1ce2502f/merged 2024-04-20T21:28:15.504 INFO:teuthology.orchestra.run.smithi189.stdout:/var/lib/containers/storage/overlay/e1bcabcb0ed940d835cf885e8512d8a55ce0676a8974aed2278dad31edd19f17/merged 2024-04-20T21:28:15.504 INFO:teuthology.orchestra.run.smithi189.stdout:/var/lib/containers/storage/overlay/2fa85e78693bec70c4635608a1bc61c40078970e88419d9cb1737272e7e83440/merged 2024-04-20T21:28:15.504 INFO:teuthology.orchestra.run.smithi189.stdout:/var/lib/containers/storage/overlay/77be9a94fa7ce24c9797bacf895dd3a51ca1aea9a8747bb0f7c130507604139f/merged 2024-04-20T21:28:15.504 INFO:teuthology.orchestra.run.smithi189.stdout:/var/lib/containers/storage/overlay/0a5aebd38fd3629d8ec8523e6b5f3c0372399ec69f12297771bad290abc28507/merged 2024-04-20T21:28:15.505 INFO:teuthology.orchestra.run.smithi189.stdout:/var/lib/containers/storage/overlay/8b4b03d4fe586c73490fbef26e0cff65fb0de12f111ac30a0028dfa354304357/merged 2024-04-20T21:28:15.505 INFO:teuthology.orchestra.run.smithi189.stdout:/var/lib/containers/storage/overlay/2121faec5dcd4b90d3407fb1355632d265abb213c8734fb9c861d5c7c4c45722/merged 2024-04-20T21:28:15.505 INFO:teuthology.orchestra.run.smithi189.stdout:/var/lib/containers/storage/overlay/b19e377c859b22076f3b6492d9f8f9f537d9b25a469852323585343aeef28e8b/merged 2024-04-20T21:28:15.505 INFO:teuthology.orchestra.run.smithi189.stdout:/var/lib/containers/storage/overlay/4407a615b1379018f4f969778f869fd8b874769a9e8a4756f8c82a1b24f8c1e4/merged 2024-04-20T21:28:15.505 INFO:teuthology.orchestra.run.smithi189.stdout:/run/netns 2024-04-20T21:28:15.505 INFO:teuthology.orchestra.run.smithi189.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-20T21:28:15.505 INFO:teuthology.orchestra.run.smithi189.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-20T21:28:15.505 INFO:teuthology.orchestra.run.smithi189.stdout:/home/ubuntu/cephtest/mnt.1 2024-04-20T21:28:15.506 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-20T21:28:15.506 DEBUG:teuthology.orchestra.run.smithi189:> ls /sys/fs/fuse/connections 2024-04-20T21:28:15.559 INFO:teuthology.orchestra.run.smithi189.stdout:122 2024-04-20T21:28:15.559 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [122] 2024-04-20T21:28:15.559 DEBUG:teuthology.orchestra.run.smithi189:> sudo stdin-killer -- python3 -c ' 2024-04-20T21:28:15.560 DEBUG:teuthology.orchestra.run.smithi189:> import glob 2024-04-20T21:28:15.560 DEBUG:teuthology.orchestra.run.smithi189:> import re 2024-04-20T21:28:15.560 DEBUG:teuthology.orchestra.run.smithi189:> import os 2024-04-20T21:28:15.560 DEBUG:teuthology.orchestra.run.smithi189:> import subprocess 2024-04-20T21:28:15.560 DEBUG:teuthology.orchestra.run.smithi189:> 2024-04-20T21:28:15.560 DEBUG:teuthology.orchestra.run.smithi189:> def _find_admin_socket(client_name): 2024-04-20T21:28:15.560 DEBUG:teuthology.orchestra.run.smithi189:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-04-20T21:28:15.560 DEBUG:teuthology.orchestra.run.smithi189:> files = glob.glob(asok_path) 2024-04-20T21:28:15.560 DEBUG:teuthology.orchestra.run.smithi189:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-04-20T21:28:15.560 DEBUG:teuthology.orchestra.run.smithi189:> 2024-04-20T21:28:15.560 DEBUG:teuthology.orchestra.run.smithi189:> # Given a non-glob path, it better be there 2024-04-20T21:28:15.560 DEBUG:teuthology.orchestra.run.smithi189:> if "*" not in asok_path: 2024-04-20T21:28:15.560 DEBUG:teuthology.orchestra.run.smithi189:> assert(len(files) == 1) 2024-04-20T21:28:15.560 DEBUG:teuthology.orchestra.run.smithi189:> return files[0] 2024-04-20T21:28:15.560 DEBUG:teuthology.orchestra.run.smithi189:> 2024-04-20T21:28:15.560 DEBUG:teuthology.orchestra.run.smithi189:> for f in files: 2024-04-20T21:28:15.560 DEBUG:teuthology.orchestra.run.smithi189:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-20T21:28:15.561 DEBUG:teuthology.orchestra.run.smithi189:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-20T21:28:15.561 DEBUG:teuthology.orchestra.run.smithi189:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-20T21:28:15.561 DEBUG:teuthology.orchestra.run.smithi189:> contents = proc_f.read() 2024-04-20T21:28:15.561 DEBUG:teuthology.orchestra.run.smithi189:> if mountpoint in contents: 2024-04-20T21:28:15.561 DEBUG:teuthology.orchestra.run.smithi189:> return f 2024-04-20T21:28:15.561 DEBUG:teuthology.orchestra.run.smithi189:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-20T21:28:15.561 DEBUG:teuthology.orchestra.run.smithi189:> 2024-04-20T21:28:15.561 DEBUG:teuthology.orchestra.run.smithi189:> print(_find_admin_socket("client.1")) 2024-04-20T21:28:15.561 DEBUG:teuthology.orchestra.run.smithi189:> ' 2024-04-20T21:28:15.723 INFO:teuthology.orchestra.run.smithi189.stdout:/var/run/ceph/ceph-client.1.41413.asok 2024-04-20T21:28:15.728 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-20T21:28:15 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-20T21:28:15.742 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.41413.asok 2024-04-20T21:28:15.742 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-20T21:28:15.743 DEBUG:teuthology.orchestra.run.smithi189:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.41413.asok status 2024-04-20T21:28:15.971 INFO:teuthology.orchestra.run.smithi189.stdout:{ 2024-04-20T21:28:15.971 INFO:teuthology.orchestra.run.smithi189.stdout: "metadata": { 2024-04-20T21:28:15.972 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_sha1": "5dd24139a1eada541a3bc16b6941c5dde975e26d", 2024-04-20T21:28:15.972 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_version": "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)", 2024-04-20T21:28:15.972 INFO:teuthology.orchestra.run.smithi189.stdout: "entity_id": "1", 2024-04-20T21:28:15.972 INFO:teuthology.orchestra.run.smithi189.stdout: "hostname": "smithi189", 2024-04-20T21:28:15.972 INFO:teuthology.orchestra.run.smithi189.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-04-20T21:28:15.973 INFO:teuthology.orchestra.run.smithi189.stdout: "pid": "41413", 2024-04-20T21:28:15.973 INFO:teuthology.orchestra.run.smithi189.stdout: "root": "/" 2024-04-20T21:28:15.973 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-20T21:28:15.973 INFO:teuthology.orchestra.run.smithi189.stdout: "dentry_count": 0, 2024-04-20T21:28:15.973 INFO:teuthology.orchestra.run.smithi189.stdout: "dentry_pinned_count": 0, 2024-04-20T21:28:15.974 INFO:teuthology.orchestra.run.smithi189.stdout: "id": 24329, 2024-04-20T21:28:15.974 INFO:teuthology.orchestra.run.smithi189.stdout: "inst": { 2024-04-20T21:28:15.974 INFO:teuthology.orchestra.run.smithi189.stdout: "name": { 2024-04-20T21:28:15.974 INFO:teuthology.orchestra.run.smithi189.stdout: "type": "client", 2024-04-20T21:28:15.975 INFO:teuthology.orchestra.run.smithi189.stdout: "num": 24329 2024-04-20T21:28:15.975 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-20T21:28:15.975 INFO:teuthology.orchestra.run.smithi189.stdout: "addr": { 2024-04-20T21:28:15.975 INFO:teuthology.orchestra.run.smithi189.stdout: "type": "v1", 2024-04-20T21:28:15.975 INFO:teuthology.orchestra.run.smithi189.stdout: "addr": "192.168.0.1:0", 2024-04-20T21:28:15.975 INFO:teuthology.orchestra.run.smithi189.stdout: "nonce": 2755061771 2024-04-20T21:28:15.976 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-20T21:28:15.976 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-20T21:28:15.976 INFO:teuthology.orchestra.run.smithi189.stdout: "addr": { 2024-04-20T21:28:15.976 INFO:teuthology.orchestra.run.smithi189.stdout: "type": "v1", 2024-04-20T21:28:15.976 INFO:teuthology.orchestra.run.smithi189.stdout: "addr": "192.168.0.1:0", 2024-04-20T21:28:15.976 INFO:teuthology.orchestra.run.smithi189.stdout: "nonce": 2755061771 2024-04-20T21:28:15.976 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-20T21:28:15.977 INFO:teuthology.orchestra.run.smithi189.stdout: "inst_str": "client.24329 192.168.0.1:0/2755061771", 2024-04-20T21:28:15.977 INFO:teuthology.orchestra.run.smithi189.stdout: "addr_str": "192.168.0.1:0/2755061771", 2024-04-20T21:28:15.977 INFO:teuthology.orchestra.run.smithi189.stdout: "inode_count": 1, 2024-04-20T21:28:15.977 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_epoch": 14, 2024-04-20T21:28:15.977 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_epoch": 45, 2024-04-20T21:28:15.977 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_epoch_barrier": 0, 2024-04-20T21:28:15.978 INFO:teuthology.orchestra.run.smithi189.stdout: "blocklisted": false, 2024-04-20T21:28:15.978 INFO:teuthology.orchestra.run.smithi189.stdout: "fs_name": "cephfs" 2024-04-20T21:28:15.978 INFO:teuthology.orchestra.run.smithi189.stdout:} 2024-04-20T21:28:15.987 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-20T21:28:15.987 DEBUG:teuthology.orchestra.run.smithi062:> stat --file-system '--printf=%T 2024-04-20T21:28:15.987 DEBUG:teuthology.orchestra.run.smithi062:> ' -- /home/ubuntu/cephtest/mnt.0 2024-04-20T21:28:16.050 INFO:teuthology.orchestra.run.smithi062.stdout:fuseblk 2024-04-20T21:28:16.050 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-04-20T21:28:16.050 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-20T21:28:16.050 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-20T21:28:16.139 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-20T21:28:16.139 DEBUG:teuthology.orchestra.run.smithi189:> stat --file-system '--printf=%T 2024-04-20T21:28:16.140 DEBUG:teuthology.orchestra.run.smithi189:> ' -- /home/ubuntu/cephtest/mnt.1 2024-04-20T21:28:16.197 INFO:teuthology.orchestra.run.smithi189.stdout:fuseblk 2024-04-20T21:28:16.197 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-04-20T21:28:16.197 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-20T21:28:16.197 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-04-20T21:28:16.236 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:28:15 smithi189 ceph-mon[21562]: pgmap v249: 65 pgs: 65 active+clean; 581 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-20T21:28:16.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:28:15 smithi062 ceph-mon[16221]: pgmap v249: 65 pgs: 65 active+clean; 581 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-20T21:28:18.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:28:17 smithi189 ceph-mon[21562]: pgmap v250: 65 pgs: 65 active+clean; 581 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.9 KiB/s rd, 426 B/s wr, 3 op/s 2024-04-20T21:28:18.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:28:17 smithi062 ceph-mon[16221]: pgmap v250: 65 pgs: 65 active+clean; 581 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.9 KiB/s rd, 426 B/s wr, 3 op/s 2024-04-20T21:28:19.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:28:18 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:28:19.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:28:18 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:28:20.236 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:28:20 smithi189 ceph-mon[21562]: pgmap v251: 65 pgs: 65 active+clean; 582 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 4 op/s 2024-04-20T21:28:20.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:28:20 smithi062 ceph-mon[16221]: pgmap v251: 65 pgs: 65 active+clean; 582 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 4 op/s 2024-04-20T21:28:20.989 INFO:teuthology.run_tasks:Running task print... 2024-04-20T21:28:20.998 INFO:teuthology.task.print:**** done client 2024-04-20T21:28:20.998 INFO:teuthology.run_tasks:Running task parallel... 2024-04-20T21:28:21.012 INFO:teuthology.task.parallel:starting parallel... 2024-04-20T21:28:21.012 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-04-20T21:28:21.013 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-04-20T21:28:21.013 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi062.front.sepia.ceph.com 2024-04-20T21:28:21.013 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -e sha1=a8c84f5997950fd7d008204839253291d52e6eda -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-04-20T21:28:21.013 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-04-20T21:28:21.013 INFO:teuthology.task.sequential:In sequential, running task workunit... 2024-04-20T21:28:21.016 INFO:tasks.workunit:Pulling workunits from ref a8c84f5997950fd7d008204839253291d52e6eda 2024-04-20T21:28:21.017 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-04-20T21:28:21.017 INFO:tasks.workunit:timeout=3h 2024-04-20T21:28:21.017 INFO:tasks.workunit:cleanup=True 2024-04-20T21:28:21.017 DEBUG:teuthology.orchestra.run.smithi062:> stat -- /home/ubuntu/cephtest/mnt.0 2024-04-20T21:28:21.078 INFO:teuthology.orchestra.run.smithi062.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-04-20T21:28:21.079 INFO:teuthology.orchestra.run.smithi062.stdout: Size: 0 Blocks: 1 IO Block: 4194304 directory 2024-04-20T21:28:21.079 INFO:teuthology.orchestra.run.smithi062.stdout:Device: 9bh/155d Inode: 1 Links: 2 2024-04-20T21:28:21.079 INFO:teuthology.orchestra.run.smithi062.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-20T21:28:21.079 INFO:teuthology.orchestra.run.smithi062.stdout:Context: system_u:object_r:fusefs_t:s0 2024-04-20T21:28:21.079 INFO:teuthology.orchestra.run.smithi062.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-04-20T21:28:21.079 INFO:teuthology.orchestra.run.smithi062.stdout:Modify: 2024-04-20 21:27:26.803895148 +0000 2024-04-20T21:28:21.079 INFO:teuthology.orchestra.run.smithi062.stdout:Change: 2024-04-20 21:28:16.132685015 +0000 2024-04-20T21:28:21.079 INFO:teuthology.orchestra.run.smithi062.stdout: Birth: - 2024-04-20T21:28:21.079 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-04-20T21:28:21.079 DEBUG:teuthology.orchestra.run.smithi062:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-04-20T21:28:21.222 DEBUG:teuthology.orchestra.run.smithi189:> stat -- /home/ubuntu/cephtest/mnt.1 2024-04-20T21:28:21.286 INFO:teuthology.orchestra.run.smithi189.stdout: File: /home/ubuntu/cephtest/mnt.1 2024-04-20T21:28:21.286 INFO:teuthology.orchestra.run.smithi189.stdout: Size: 0 Blocks: 1 IO Block: 4194304 directory 2024-04-20T21:28:21.287 INFO:teuthology.orchestra.run.smithi189.stdout:Device: 7ah/122d Inode: 1 Links: 3 2024-04-20T21:28:21.287 INFO:teuthology.orchestra.run.smithi189.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-20T21:28:21.287 INFO:teuthology.orchestra.run.smithi189.stdout:Context: system_u:object_r:fusefs_t:s0 2024-04-20T21:28:21.287 INFO:teuthology.orchestra.run.smithi189.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-04-20T21:28:21.287 INFO:teuthology.orchestra.run.smithi189.stdout:Modify: 2024-04-20 21:28:21.205419480 +0000 2024-04-20T21:28:21.287 INFO:teuthology.orchestra.run.smithi189.stdout:Change: 2024-04-20 21:28:21.205419480 +0000 2024-04-20T21:28:21.287 INFO:teuthology.orchestra.run.smithi189.stdout: Birth: - 2024-04-20T21:28:21.287 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.1 2024-04-20T21:28:21.287 DEBUG:teuthology.orchestra.run.smithi189:> cd -- /home/ubuntu/cephtest/mnt.1 && sudo install -d -m 0755 --owner=ubuntu -- client.1 2024-04-20T21:28:21.392 DEBUG:teuthology.orchestra.run.smithi062:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout a8c84f5997950fd7d008204839253291d52e6eda 2024-04-20T21:28:21.392 DEBUG:teuthology.orchestra.run.smithi189:> rm -rf /home/ubuntu/cephtest/clone.client.1 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.1 && cd /home/ubuntu/cephtest/clone.client.1 && git checkout a8c84f5997950fd7d008204839253291d52e6eda 2024-04-20T21:28:21.428 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T21:28:21.442 INFO:tasks.workunit.client.0.smithi062.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-04-20T21:28:21.452 INFO:tasks.workunit.client.1.smithi189.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.1'... 2024-04-20T21:28:23.221 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:28:22 smithi062 ceph-mon[16221]: pgmap v252: 65 pgs: 65 active+clean; 582 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-20T21:28:23.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:28:22 smithi189 ceph-mon[21562]: pgmap v252: 65 pgs: 65 active+clean; 582 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-20T21:28:23.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:23.810+0000 7f637d400700 1 -- 172.21.15.62:0/1712805596 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6378101440 msgr2=0x7f6378101890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:28:23.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:23.810+0000 7f637d400700 1 --2- 172.21.15.62:0/1712805596 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6378101440 0x7f6378101890 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f6364009910 tx=0x7f6364009c20 comp rx=0 tx=0).stop 2024-04-20T21:28:23.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:23.810+0000 7f637d400700 1 -- 172.21.15.62:0/1712805596 shutdown_connections 2024-04-20T21:28:23.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:23.810+0000 7f637d400700 1 --2- 172.21.15.62:0/1712805596 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6378101440 0x7f6378101890 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:28:23.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:23.810+0000 7f637d400700 1 --2- 172.21.15.62:0/1712805596 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6378100240 0x7f6378100650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:28:23.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:23.810+0000 7f637d400700 1 -- 172.21.15.62:0/1712805596 >> 172.21.15.62:0/1712805596 conn(0x7f63780fb7f0 msgr2=0x7f63780fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:28:23.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:23.810+0000 7f637d400700 1 -- 172.21.15.62:0/1712805596 shutdown_connections 2024-04-20T21:28:23.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:23.810+0000 7f637d400700 1 -- 172.21.15.62:0/1712805596 wait complete. 2024-04-20T21:28:23.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:23.811+0000 7f637d400700 1 Processor -- start 2024-04-20T21:28:23.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:23.812+0000 7f637d400700 1 -- start start 2024-04-20T21:28:23.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:23.812+0000 7f637d400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6378100240 0x7f6378195b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:28:23.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:23.812+0000 7f637d400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6378101440 0x7f63781960c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:28:23.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:23.812+0000 7f637d400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63781966e0 con 0x7f6378100240 2024-04-20T21:28:23.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:23.812+0000 7f637d400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6378196820 con 0x7f6378101440 2024-04-20T21:28:23.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:23.812+0000 7f6376800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6378100240 0x7f6378195b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:28:23.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:23.812+0000 7f6376800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6378100240 0x7f6378195b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56698/0 (socket says 172.21.15.62:56698) 2024-04-20T21:28:23.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:23.812+0000 7f6376800700 1 -- 172.21.15.62:0/859351515 learned_addr learned my addr 172.21.15.62:0/859351515 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:28:23.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:23.813+0000 7f6375e00700 1 --2- 172.21.15.62:0/859351515 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6378101440 0x7f63781960c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:28:23.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:23.813+0000 7f6376800700 1 -- 172.21.15.62:0/859351515 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6378101440 msgr2=0x7f63781960c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:28:23.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:23.813+0000 7f6376800700 1 --2- 172.21.15.62:0/859351515 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6378101440 0x7f63781960c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:28:23.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:23.813+0000 7f6376800700 1 -- 172.21.15.62:0/859351515 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63640095f0 con 0x7f6378100240 2024-04-20T21:28:23.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:23.813+0000 7f6375e00700 1 --2- 172.21.15.62:0/859351515 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6378101440 0x7f63781960c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T21:28:23.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:23.813+0000 7f6376800700 1 --2- 172.21.15.62:0/859351515 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6378100240 0x7f6378195b80 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f636800b480 tx=0x7f636800b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:28:23.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:23.813+0000 7f636f400700 1 -- 172.21.15.62:0/859351515 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f63680118c0 con 0x7f6378100240 2024-04-20T21:28:23.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:23.813+0000 7f636f400700 1 -- 172.21.15.62:0/859351515 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f6368010380 con 0x7f6378100240 2024-04-20T21:28:23.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:23.813+0000 7f636f400700 1 -- 172.21.15.62:0/859351515 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f636800f4f0 con 0x7f6378100240 2024-04-20T21:28:23.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:23.813+0000 7f637d400700 1 -- 172.21.15.62:0/859351515 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f637819b2d0 con 0x7f6378100240 2024-04-20T21:28:23.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:23.814+0000 7f637d400700 1 -- 172.21.15.62:0/859351515 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f637819b820 con 0x7f6378100240 2024-04-20T21:28:23.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:23.815+0000 7f637d400700 1 -- 172.21.15.62:0/859351515 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6378108e30 con 0x7f6378100240 2024-04-20T21:28:23.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:23.816+0000 7f636f400700 1 -- 172.21.15.62:0/859351515 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f63680104f0 con 0x7f6378100240 2024-04-20T21:28:23.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:23.817+0000 7f636f400700 1 --2- 172.21.15.62:0/859351515 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f635c06c680 0x7f635c06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:28:23.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:23.817+0000 7f636f400700 1 -- 172.21.15.62:0/859351515 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5468+0+0 (secure 0 0 0) 0x7f636808b4b0 con 0x7f6378100240 2024-04-20T21:28:23.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:23.817+0000 7f6375e00700 1 --2- 172.21.15.62:0/859351515 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f635c06c680 0x7f635c06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:28:23.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:23.818+0000 7f6375e00700 1 --2- 172.21.15.62:0/859351515 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f635c06c680 0x7f635c06eb30 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f6364006010 tx=0x7f636400b540 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:28:23.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:23.822+0000 7f636f400700 1 -- 172.21.15.62:0/859351515 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f6368059600 con 0x7f6378100240 2024-04-20T21:28:24.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:28:23 smithi062 ceph-mon[16221]: pgmap v253: 65 pgs: 65 active+clean; 582 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.8 KiB/s rd, 341 B/s wr, 4 op/s 2024-04-20T21:28:24.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:24.049+0000 7f637d400700 1 -- 172.21.15.62:0/859351515 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}] v 0) v1 -- 0x7f6378108ff0 con 0x7f6378100240 2024-04-20T21:28:24.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:24.050+0000 7f636f400700 1 -- 172.21.15.62:0/859351515 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}]=0 v21)=0 v21) v1 ==== 155+0+0 (secure 0 0 0) 0x7f6368059190 con 0x7f6378100240 2024-04-20T21:28:24.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:24.053+0000 7f637d400700 1 -- 172.21.15.62:0/859351515 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f635c06c680 msgr2=0x7f635c06eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:28:24.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:24.053+0000 7f637d400700 1 --2- 172.21.15.62:0/859351515 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f635c06c680 0x7f635c06eb30 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f6364006010 tx=0x7f636400b540 comp rx=0 tx=0).stop 2024-04-20T21:28:24.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:24.053+0000 7f637d400700 1 -- 172.21.15.62:0/859351515 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6378100240 msgr2=0x7f6378195b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:28:24.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:24.053+0000 7f637d400700 1 --2- 172.21.15.62:0/859351515 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6378100240 0x7f6378195b80 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f636800b480 tx=0x7f636800b840 comp rx=0 tx=0).stop 2024-04-20T21:28:24.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:24.053+0000 7f637d400700 1 -- 172.21.15.62:0/859351515 shutdown_connections 2024-04-20T21:28:24.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:24.053+0000 7f637d400700 1 --2- 172.21.15.62:0/859351515 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f635c06c680 0x7f635c06eb30 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:28:24.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:24.053+0000 7f637d400700 1 --2- 172.21.15.62:0/859351515 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6378100240 0x7f6378195b80 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:28:24.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:24.053+0000 7f637d400700 1 --2- 172.21.15.62:0/859351515 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6378101440 0x7f63781960c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:28:24.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:24.053+0000 7f637d400700 1 -- 172.21.15.62:0/859351515 >> 172.21.15.62:0/859351515 conn(0x7f63780fb7f0 msgr2=0x7f6378104670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:28:24.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:24.053+0000 7f637d400700 1 -- 172.21.15.62:0/859351515 shutdown_connections 2024-04-20T21:28:24.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:24.053+0000 7f637d400700 1 -- 172.21.15.62:0/859351515 wait complete. 2024-04-20T21:28:24.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:28:23 smithi189 ceph-mon[21562]: pgmap v253: 65 pgs: 65 active+clean; 582 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.8 KiB/s rd, 341 B/s wr, 4 op/s 2024-04-20T21:28:24.983 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -e sha1=a8c84f5997950fd7d008204839253291d52e6eda -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-04-20T21:28:25.456 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T21:28:26.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:28:25 smithi189 ceph-mon[21562]: pgmap v254: 65 pgs: 65 active+clean; 587 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 3.1 KiB/s rd, 852 B/s wr, 5 op/s 2024-04-20T21:28:26.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:28:25 smithi062 ceph-mon[16221]: pgmap v254: 65 pgs: 65 active+clean; 587 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 3.1 KiB/s rd, 852 B/s wr, 5 op/s 2024-04-20T21:28:27.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:27.938+0000 7fd054c00700 1 -- 172.21.15.62:0/2245730650 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd050100d60 msgr2=0x7fd050103140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:28:27.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:27.938+0000 7fd054c00700 1 --2- 172.21.15.62:0/2245730650 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd050100d60 0x7fd050103140 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fd040009910 tx=0x7fd040009c20 comp rx=0 tx=0).stop 2024-04-20T21:28:27.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:27.938+0000 7fd054c00700 1 -- 172.21.15.62:0/2245730650 shutdown_connections 2024-04-20T21:28:27.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:27.938+0000 7fd054c00700 1 --2- 172.21.15.62:0/2245730650 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd050103680 0x7fd050105a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:28:27.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:27.938+0000 7fd054c00700 1 --2- 172.21.15.62:0/2245730650 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd050100d60 0x7fd050103140 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:28:27.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:27.938+0000 7fd054c00700 1 -- 172.21.15.62:0/2245730650 >> 172.21.15.62:0/2245730650 conn(0x7fd0500fa780 msgr2=0x7fd0500fcbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:28:27.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:27.938+0000 7fd054c00700 1 -- 172.21.15.62:0/2245730650 shutdown_connections 2024-04-20T21:28:27.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:27.938+0000 7fd054c00700 1 -- 172.21.15.62:0/2245730650 wait complete. 2024-04-20T21:28:27.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:27.939+0000 7fd054c00700 1 Processor -- start 2024-04-20T21:28:27.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:27.940+0000 7fd054c00700 1 -- start start 2024-04-20T21:28:27.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:27.940+0000 7fd054c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd050100d60 0x7fd0501006c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:28:27.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:27.940+0000 7fd054c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd050103680 0x7fd0500fed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:28:27.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:27.940+0000 7fd054c00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0500ff250 con 0x7fd050103680 2024-04-20T21:28:27.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:27.940+0000 7fd054c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0500ff390 con 0x7fd050100d60 2024-04-20T21:28:27.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:27.940+0000 7fd047e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd050103680 0x7fd0500fed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:28:27.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:27.940+0000 7fd047e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd050103680 0x7fd0500fed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56712/0 (socket says 172.21.15.62:56712) 2024-04-20T21:28:27.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:27.940+0000 7fd047e00700 1 -- 172.21.15.62:0/2243804415 learned_addr learned my addr 172.21.15.62:0/2243804415 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:28:27.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:27.941+0000 7fd04de00700 1 --2- 172.21.15.62:0/2243804415 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd050100d60 0x7fd0501006c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:28:27.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:27.941+0000 7fd047e00700 1 -- 172.21.15.62:0/2243804415 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd050100d60 msgr2=0x7fd0501006c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:28:27.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:27.941+0000 7fd047e00700 1 --2- 172.21.15.62:0/2243804415 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd050100d60 0x7fd0501006c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:28:27.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:27.941+0000 7fd047e00700 1 -- 172.21.15.62:0/2243804415 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0400095f0 con 0x7fd050103680 2024-04-20T21:28:27.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:27.941+0000 7fd047e00700 1 --2- 172.21.15.62:0/2243804415 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd050103680 0x7fd0500fed10 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fd03800b4b0 tx=0x7fd03800b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:28:27.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:27.942+0000 7fd046a00700 1 -- 172.21.15.62:0/2243804415 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd038011800 con 0x7fd050103680 2024-04-20T21:28:27.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:27.942+0000 7fd054c00700 1 -- 172.21.15.62:0/2243804415 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0500ff670 con 0x7fd050103680 2024-04-20T21:28:27.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:27.942+0000 7fd046a00700 1 -- 172.21.15.62:0/2243804415 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fd038011e10 con 0x7fd050103680 2024-04-20T21:28:27.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:27.942+0000 7fd046a00700 1 -- 172.21.15.62:0/2243804415 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd03800f2e0 con 0x7fd050103680 2024-04-20T21:28:27.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:27.942+0000 7fd054c00700 1 -- 172.21.15.62:0/2243804415 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd05019db20 con 0x7fd050103680 2024-04-20T21:28:27.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:27.943+0000 7fd054c00700 1 -- 172.21.15.62:0/2243804415 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd05018fc40 con 0x7fd050103680 2024-04-20T21:28:27.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:27.944+0000 7fd046a00700 1 -- 172.21.15.62:0/2243804415 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7fd038011960 con 0x7fd050103680 2024-04-20T21:28:27.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:27.945+0000 7fd046a00700 1 --2- 172.21.15.62:0/2243804415 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fd03406c680 0x7fd03406eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:28:27.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:27.945+0000 7fd046a00700 1 -- 172.21.15.62:0/2243804415 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5468+0+0 (secure 0 0 0) 0x7fd03808b3b0 con 0x7fd050103680 2024-04-20T21:28:27.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:27.945+0000 7fd04de00700 1 --2- 172.21.15.62:0/2243804415 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fd03406c680 0x7fd03406eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:28:27.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:27.946+0000 7fd04de00700 1 --2- 172.21.15.62:0/2243804415 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fd03406c680 0x7fd03406eb30 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fd040006010 tx=0x7fd04000b540 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:28:27.949 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:27.950+0000 7fd046a00700 1 -- 172.21.15.62:0/2243804415 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fd0380595a0 con 0x7fd050103680 2024-04-20T21:28:28.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:28:27 smithi062 ceph-mon[16221]: pgmap v255: 65 pgs: 65 active+clean; 587 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.2 KiB/s rd, 767 B/s wr, 4 op/s 2024-04-20T21:28:28.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:28.172+0000 7fd054c00700 1 -- 172.21.15.62:0/2243804415 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}] v 0) v1 -- 0x7fd05002cc30 con 0x7fd050103680 2024-04-20T21:28:28.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:28.173+0000 7fd046a00700 1 -- 172.21.15.62:0/2243804415 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}]=0 v21)=0 v21) v1 ==== 163+0+0 (secure 0 0 0) 0x7fd038059130 con 0x7fd050103680 2024-04-20T21:28:28.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:28.176+0000 7fd054c00700 1 -- 172.21.15.62:0/2243804415 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fd03406c680 msgr2=0x7fd03406eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:28:28.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:28.176+0000 7fd054c00700 1 --2- 172.21.15.62:0/2243804415 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fd03406c680 0x7fd03406eb30 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fd040006010 tx=0x7fd04000b540 comp rx=0 tx=0).stop 2024-04-20T21:28:28.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:28.176+0000 7fd054c00700 1 -- 172.21.15.62:0/2243804415 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd050103680 msgr2=0x7fd0500fed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:28:28.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:28.176+0000 7fd054c00700 1 --2- 172.21.15.62:0/2243804415 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd050103680 0x7fd0500fed10 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fd03800b4b0 tx=0x7fd03800b870 comp rx=0 tx=0).stop 2024-04-20T21:28:28.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:28.176+0000 7fd054c00700 1 -- 172.21.15.62:0/2243804415 shutdown_connections 2024-04-20T21:28:28.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:28.176+0000 7fd054c00700 1 --2- 172.21.15.62:0/2243804415 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fd03406c680 0x7fd03406eb30 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:28:28.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:28.176+0000 7fd054c00700 1 --2- 172.21.15.62:0/2243804415 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd050100d60 0x7fd0501006c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:28:28.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:28.176+0000 7fd054c00700 1 --2- 172.21.15.62:0/2243804415 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd050103680 0x7fd0500fed10 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:28:28.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:28.176+0000 7fd054c00700 1 -- 172.21.15.62:0/2243804415 >> 172.21.15.62:0/2243804415 conn(0x7fd0500fa780 msgr2=0x7fd0500fcbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:28:28.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:28.177+0000 7fd054c00700 1 -- 172.21.15.62:0/2243804415 shutdown_connections 2024-04-20T21:28:28.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:28.177+0000 7fd054c00700 1 -- 172.21.15.62:0/2243804415 wait complete. 2024-04-20T21:28:28.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:28:27 smithi189 ceph-mon[21562]: pgmap v255: 65 pgs: 65 active+clean; 587 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.2 KiB/s rd, 767 B/s wr, 4 op/s 2024-04-20T21:28:29.090 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -e sha1=a8c84f5997950fd7d008204839253291d52e6eda -- bash -c 'ceph config set global log_to_journald false --force' 2024-04-20T21:28:29.517 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T21:28:30.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:28:29 smithi062 ceph-mon[16221]: pgmap v256: 65 pgs: 65 active+clean; 587 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.2 KiB/s rd, 853 B/s wr, 5 op/s 2024-04-20T21:28:30.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:28:29 smithi189 ceph-mon[21562]: pgmap v256: 65 pgs: 65 active+clean; 587 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.2 KiB/s rd, 853 B/s wr, 5 op/s 2024-04-20T21:28:31.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:31.956+0000 7feeb4800700 1 -- 172.21.15.62:0/2292013527 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feeac076950 msgr2=0x7feeac076dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:28:31.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:31.956+0000 7feeb4800700 1 --2- 172.21.15.62:0/2292013527 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feeac076950 0x7feeac076dc0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fee9c009910 tx=0x7fee9c009c20 comp rx=0 tx=0).stop 2024-04-20T21:28:31.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:31.956+0000 7feeb4800700 1 -- 172.21.15.62:0/2292013527 shutdown_connections 2024-04-20T21:28:31.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:31.956+0000 7feeb4800700 1 --2- 172.21.15.62:0/2292013527 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feeac076950 0x7feeac076dc0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:28:31.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:31.956+0000 7feeb4800700 1 --2- 172.21.15.62:0/2292013527 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feeac075700 0x7feeac075b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:28:31.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:31.956+0000 7feeb4800700 1 -- 172.21.15.62:0/2292013527 >> 172.21.15.62:0/2292013527 conn(0x7feeac0fd9f0 msgr2=0x7feeac0ffe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:28:31.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:31.956+0000 7feeb4800700 1 -- 172.21.15.62:0/2292013527 shutdown_connections 2024-04-20T21:28:31.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:31.956+0000 7feeb4800700 1 -- 172.21.15.62:0/2292013527 wait complete. 2024-04-20T21:28:31.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:31.957+0000 7feeb4800700 1 Processor -- start 2024-04-20T21:28:31.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:31.957+0000 7feeb4800700 1 -- start start 2024-04-20T21:28:31.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:31.958+0000 7feeb4800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feeac075700 0x7feeac19c070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:28:31.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:31.958+0000 7feeb4800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feeac076950 0x7feeac19c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:28:31.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:31.958+0000 7feeb4800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feeac19cbd0 con 0x7feeac075700 2024-04-20T21:28:31.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:31.958+0000 7feeb4800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feeac19cd10 con 0x7feeac076950 2024-04-20T21:28:31.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:31.958+0000 7feeb1e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feeac075700 0x7feeac19c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:28:31.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:31.958+0000 7feeb1e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feeac075700 0x7feeac19c070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43264/0 (socket says 172.21.15.62:43264) 2024-04-20T21:28:31.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:31.958+0000 7feeb1e00700 1 -- 172.21.15.62:0/2218348137 learned_addr learned my addr 172.21.15.62:0/2218348137 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:28:31.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:31.959+0000 7feeb1400700 1 --2- 172.21.15.62:0/2218348137 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feeac076950 0x7feeac19c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:28:31.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:31.959+0000 7feeb1e00700 1 -- 172.21.15.62:0/2218348137 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feeac076950 msgr2=0x7feeac19c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:28:31.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:31.959+0000 7feeb1e00700 1 --2- 172.21.15.62:0/2218348137 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feeac076950 0x7feeac19c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:28:31.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:31.959+0000 7feeb1e00700 1 -- 172.21.15.62:0/2218348137 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee9c0095f0 con 0x7feeac075700 2024-04-20T21:28:31.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:31.959+0000 7feeb1e00700 1 --2- 172.21.15.62:0/2218348137 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feeac075700 0x7feeac19c070 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7feea000c8a0 tx=0x7feea000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:28:31.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:31.960+0000 7feea6a00700 1 -- 172.21.15.62:0/2218348137 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feea000cea0 con 0x7feeac075700 2024-04-20T21:28:31.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:31.960+0000 7feea6a00700 1 -- 172.21.15.62:0/2218348137 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7feea0004830 con 0x7feeac075700 2024-04-20T21:28:31.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:31.960+0000 7feea6a00700 1 -- 172.21.15.62:0/2218348137 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feea00055b0 con 0x7feeac075700 2024-04-20T21:28:31.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:31.960+0000 7feeb4800700 1 -- 172.21.15.62:0/2218348137 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feeac1a17c0 con 0x7feeac075700 2024-04-20T21:28:31.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:31.960+0000 7feeb4800700 1 -- 172.21.15.62:0/2218348137 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feeac1a1d10 con 0x7feeac075700 2024-04-20T21:28:31.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:31.961+0000 7feeb4800700 1 -- 172.21.15.62:0/2218348137 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feeac10f390 con 0x7feeac075700 2024-04-20T21:28:31.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:31.962+0000 7feea6a00700 1 -- 172.21.15.62:0/2218348137 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7feea0007ad0 con 0x7feeac075700 2024-04-20T21:28:31.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:31.963+0000 7feea6a00700 1 --2- 172.21.15.62:0/2218348137 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fee9406c680 0x7fee9406eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:28:31.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:31.963+0000 7feea6a00700 1 -- 172.21.15.62:0/2218348137 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5468+0+0 (secure 0 0 0) 0x7feea008b5d0 con 0x7feeac075700 2024-04-20T21:28:31.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:31.963+0000 7feeb1400700 1 --2- 172.21.15.62:0/2218348137 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fee9406c680 0x7fee9406eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:28:31.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:31.964+0000 7feeb1400700 1 --2- 172.21.15.62:0/2218348137 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fee9406c680 0x7fee9406eb30 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fee9c006010 tx=0x7fee9c00b540 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:28:31.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:31.968+0000 7feea6a00700 1 -- 172.21.15.62:0/2218348137 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7feea0059720 con 0x7feeac075700 2024-04-20T21:28:32.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:32.187+0000 7feeb4800700 1 -- 172.21.15.62:0/2218348137 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command([{prefix=config set, name=log_to_journald}] v 0) v1 -- 0x7feeac10f580 con 0x7feeac075700 2024-04-20T21:28:32.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:32.188+0000 7feea6a00700 1 -- 172.21.15.62:0/2218348137 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{prefix=config set, name=log_to_journald}]=0 v21)=0 v21) v1 ==== 135+0+0 (secure 0 0 0) 0x7feea00592b0 con 0x7feeac075700 2024-04-20T21:28:32.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:32.191+0000 7feeb4800700 1 -- 172.21.15.62:0/2218348137 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fee9406c680 msgr2=0x7fee9406eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:28:32.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:32.191+0000 7feeb4800700 1 --2- 172.21.15.62:0/2218348137 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fee9406c680 0x7fee9406eb30 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fee9c006010 tx=0x7fee9c00b540 comp rx=0 tx=0).stop 2024-04-20T21:28:32.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:32.191+0000 7feeb4800700 1 -- 172.21.15.62:0/2218348137 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feeac075700 msgr2=0x7feeac19c070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:28:32.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:32.191+0000 7feeb4800700 1 --2- 172.21.15.62:0/2218348137 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feeac075700 0x7feeac19c070 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7feea000c8a0 tx=0x7feea000cc60 comp rx=0 tx=0).stop 2024-04-20T21:28:32.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:32.191+0000 7feeb4800700 1 -- 172.21.15.62:0/2218348137 shutdown_connections 2024-04-20T21:28:32.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:32.191+0000 7feeb4800700 1 --2- 172.21.15.62:0/2218348137 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fee9406c680 0x7fee9406eb30 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:28:32.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:32.191+0000 7feeb4800700 1 --2- 172.21.15.62:0/2218348137 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feeac075700 0x7feeac19c070 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:28:32.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:32.191+0000 7feeb4800700 1 --2- 172.21.15.62:0/2218348137 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feeac076950 0x7feeac19c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:28:32.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:32.191+0000 7feeb4800700 1 -- 172.21.15.62:0/2218348137 >> 172.21.15.62:0/2218348137 conn(0x7feeac0fd9f0 msgr2=0x7feeac10c6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:28:32.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:32.191+0000 7feeb4800700 1 -- 172.21.15.62:0/2218348137 shutdown_connections 2024-04-20T21:28:32.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:32.192+0000 7feeb4800700 1 -- 172.21.15.62:0/2218348137 wait complete. 2024-04-20T21:28:32.962 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -e sha1=a8c84f5997950fd7d008204839253291d52e6eda -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr' 2024-04-20T21:28:33.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:28:32 smithi189 ceph-mon[21562]: pgmap v257: 65 pgs: 65 active+clean; 587 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.9 KiB/s rd, 767 B/s wr, 5 op/s 2024-04-20T21:28:33.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:28:32 smithi062 ceph-mon[16221]: pgmap v257: 65 pgs: 65 active+clean; 587 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.9 KiB/s rd, 767 B/s wr, 5 op/s 2024-04-20T21:28:33.392 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T21:28:34.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:28:33 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:28:34.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:28:33 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:28:35.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:28:34 smithi062 ceph-mon[16221]: pgmap v258: 65 pgs: 65 active+clean; 587 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 852 B/s wr, 5 op/s 2024-04-20T21:28:35.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:28:34 smithi189 ceph-mon[21562]: pgmap v258: 65 pgs: 65 active+clean; 587 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 852 B/s wr, 5 op/s 2024-04-20T21:28:35.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:35.626+0000 7f9dba400700 1 -- 172.21.15.62:0/3602098153 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9db4107a40 msgr2=0x7f9db4107e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:28:35.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:35.626+0000 7f9dba400700 1 --2- 172.21.15.62:0/3602098153 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9db4107a40 0x7f9db4107e90 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f9da0009910 tx=0x7f9da0009c20 comp rx=0 tx=0).stop 2024-04-20T21:28:35.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:35.627+0000 7f9dba400700 1 -- 172.21.15.62:0/3602098153 shutdown_connections 2024-04-20T21:28:35.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:35.627+0000 7f9dba400700 1 --2- 172.21.15.62:0/3602098153 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9db4107a40 0x7f9db4107e90 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:28:35.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:35.627+0000 7f9dba400700 1 --2- 172.21.15.62:0/3602098153 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9db4106840 0x7f9db4106c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:28:35.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:35.627+0000 7f9dba400700 1 -- 172.21.15.62:0/3602098153 >> 172.21.15.62:0/3602098153 conn(0x7f9db4075970 msgr2=0x7f9db4077dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:28:35.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:35.627+0000 7f9dba400700 1 -- 172.21.15.62:0/3602098153 shutdown_connections 2024-04-20T21:28:35.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:35.627+0000 7f9dba400700 1 -- 172.21.15.62:0/3602098153 wait complete. 2024-04-20T21:28:35.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:35.628+0000 7f9dba400700 1 Processor -- start 2024-04-20T21:28:35.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:35.628+0000 7f9dba400700 1 -- start start 2024-04-20T21:28:35.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:35.628+0000 7f9dba400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9db4106840 0x7f9db419c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:28:35.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:35.628+0000 7f9dba400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9db4107a40 0x7f9db419c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:28:35.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:35.628+0000 7f9dba400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9db419cc40 con 0x7f9db4107a40 2024-04-20T21:28:35.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:35.628+0000 7f9dba400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9db419cd80 con 0x7f9db4106840 2024-04-20T21:28:35.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:35.629+0000 7f9db2a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9db4107a40 0x7f9db419c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:28:35.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:35.629+0000 7f9db2a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9db4107a40 0x7f9db419c620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43290/0 (socket says 172.21.15.62:43290) 2024-04-20T21:28:35.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:35.629+0000 7f9db2a00700 1 -- 172.21.15.62:0/1926775126 learned_addr learned my addr 172.21.15.62:0/1926775126 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:28:35.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:35.629+0000 7f9db3400700 1 --2- 172.21.15.62:0/1926775126 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9db4106840 0x7f9db419c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:28:35.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:35.629+0000 7f9db2a00700 1 -- 172.21.15.62:0/1926775126 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9db4106840 msgr2=0x7f9db419c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:28:35.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:35.629+0000 7f9db2a00700 1 --2- 172.21.15.62:0/1926775126 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9db4106840 0x7f9db419c0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:28:35.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:35.629+0000 7f9db2a00700 1 -- 172.21.15.62:0/1926775126 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9da00095f0 con 0x7f9db4107a40 2024-04-20T21:28:35.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:35.630+0000 7f9db2a00700 1 --2- 172.21.15.62:0/1926775126 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9db4107a40 0x7f9db419c620 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f9da0005b40 tx=0x7f9da000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:28:35.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:35.630+0000 7f9dabe00700 1 -- 172.21.15.62:0/1926775126 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9da001d070 con 0x7f9db4107a40 2024-04-20T21:28:35.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:35.630+0000 7f9dabe00700 1 -- 172.21.15.62:0/1926775126 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f9da000f460 con 0x7f9db4107a40 2024-04-20T21:28:35.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:35.630+0000 7f9dabe00700 1 -- 172.21.15.62:0/1926775126 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9da0021640 con 0x7f9db4107a40 2024-04-20T21:28:35.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:35.630+0000 7f9dba400700 1 -- 172.21.15.62:0/1926775126 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9db41a17d0 con 0x7f9db4107a40 2024-04-20T21:28:35.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:35.630+0000 7f9dba400700 1 -- 172.21.15.62:0/1926775126 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9db41a1c70 con 0x7f9db4107a40 2024-04-20T21:28:35.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:35.631+0000 7f9dba400700 1 -- 172.21.15.62:0/1926775126 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9db410f390 con 0x7f9db4107a40 2024-04-20T21:28:35.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:35.633+0000 7f9dabe00700 1 -- 172.21.15.62:0/1926775126 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f9da000fa90 con 0x7f9db4107a40 2024-04-20T21:28:35.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:35.634+0000 7f9dabe00700 1 --2- 172.21.15.62:0/1926775126 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f9d9806c680 0x7f9d9806eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:28:35.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:35.634+0000 7f9dabe00700 1 -- 172.21.15.62:0/1926775126 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5468+0+0 (secure 0 0 0) 0x7f9da008cc30 con 0x7f9db4107a40 2024-04-20T21:28:35.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:35.634+0000 7f9db3400700 1 --2- 172.21.15.62:0/1926775126 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f9d9806c680 0x7f9d9806eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:28:35.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:35.635+0000 7f9db3400700 1 --2- 172.21.15.62:0/1926775126 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f9d9806c680 0x7f9d9806eb30 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f9da4005950 tx=0x7f9da40058e0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:28:35.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:35.638+0000 7f9dabe00700 1 -- 172.21.15.62:0/1926775126 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f9da005ad00 con 0x7f9db4107a40 2024-04-20T21:28:35.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:28:35.878+0000 7f9dba400700 1 -- 172.21.15.62:0/1926775126 --> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] -- mgr_command(tid 0: {"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:a8c84f5997950fd7d008204839253291d52e6eda", "daemon_types": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7f9db410c390 con 0x7f9d9806c680 2024-04-20T21:28:37.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:28:36 smithi189 ceph-mon[21562]: from='client.14546 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:a8c84f5997950fd7d008204839253291d52e6eda", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:28:37.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:28:36 smithi189 ceph-mon[21562]: pgmap v259: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 3.2 KiB/s rd, 1023 B/s wr, 5 op/s 2024-04-20T21:28:37.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:28:36 smithi062 ceph-mon[16221]: from='client.14546 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:a8c84f5997950fd7d008204839253291d52e6eda", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:28:37.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:28:36 smithi062 ceph-mon[16221]: pgmap v259: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 3.2 KiB/s rd, 1023 B/s wr, 5 op/s 2024-04-20T21:28:39.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:28:38 smithi189 ceph-mon[21562]: pgmap v260: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 3 op/s 2024-04-20T21:28:39.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:28:38 smithi062 ceph-mon[16221]: pgmap v260: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 3 op/s 2024-04-20T21:28:40.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:28:39 smithi189 ceph-mon[21562]: pgmap v261: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 4 op/s 2024-04-20T21:28:40.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:28:39 smithi062 ceph-mon[16221]: pgmap v261: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 4 op/s 2024-04-20T21:28:43.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:28:42 smithi189 ceph-mon[21562]: pgmap v262: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.8 KiB/s rd, 426 B/s wr, 4 op/s 2024-04-20T21:28:43.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:28:42 smithi062 ceph-mon[16221]: pgmap v262: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.8 KiB/s rd, 426 B/s wr, 4 op/s 2024-04-20T21:28:44.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:28:43 smithi062 ceph-mon[16221]: pgmap v263: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.0 KiB/s rd, 426 B/s wr, 4 op/s 2024-04-20T21:28:44.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:28:43 smithi189 ceph-mon[21562]: pgmap v263: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.0 KiB/s rd, 426 B/s wr, 4 op/s 2024-04-20T21:28:46.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:28:45 smithi062 ceph-mon[16221]: pgmap v264: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.8 KiB/s rd, 341 B/s wr, 4 op/s 2024-04-20T21:28:46.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:28:45 smithi189 ceph-mon[21562]: pgmap v264: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.8 KiB/s rd, 341 B/s wr, 4 op/s 2024-04-20T21:28:48.284 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:28:47 smithi062 ceph-mon[16221]: pgmap v265: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:28:48.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:28:47 smithi189 ceph-mon[21562]: pgmap v265: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:28:49.268 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:28:48 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:28:49.389 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:28:48 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:28:50.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:28:49 smithi062 ceph-mon[16221]: pgmap v266: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:28:50.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:28:49 smithi189 ceph-mon[21562]: pgmap v266: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:28:53.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:28:52 smithi189 ceph-mon[21562]: pgmap v267: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:28:53.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:28:52 smithi062 ceph-mon[16221]: pgmap v267: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:28:54.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:28:53 smithi062 ceph-mon[16221]: pgmap v268: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:28:54.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:28:53 smithi189 ceph-mon[21562]: pgmap v268: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:28:56.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:28:55 smithi062 ceph-mon[16221]: pgmap v269: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:28:56.380 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:28:55 smithi189 ceph-mon[21562]: pgmap v269: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:28:58.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:28:57 smithi062 ceph-mon[16221]: pgmap v270: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:28:58.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:28:57 smithi189 ceph-mon[21562]: pgmap v270: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:29:00.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:28:59 smithi062 ceph-mon[16221]: pgmap v271: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:29:00.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:28:59 smithi189 ceph-mon[21562]: pgmap v271: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:29:01.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:29:01 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:29:02.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:29:01 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:29:03.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:29:02 smithi189 ceph-mon[21562]: pgmap v272: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:29:03.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:29:02 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:29:03.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:29:02 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T21:29:03.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:29:02 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:29:03.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:29:02 smithi062 ceph-mon[16221]: pgmap v272: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:29:03.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:29:02 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:29:03.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:29:02 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T21:29:03.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:29:02 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:29:04.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:29:03 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:29:04.284 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:29:03 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:29:05.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:29:04 smithi189 ceph-mon[21562]: pgmap v273: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:29:05.284 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:29:04 smithi062 ceph-mon[16221]: pgmap v273: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:29:06.933 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:29:06 smithi189 ceph-mon[21562]: pgmap v274: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:29:07.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:29:06 smithi062 ceph-mon[16221]: pgmap v274: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:29:09.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:29:08 smithi189 ceph-mon[21562]: pgmap v275: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:29:09.284 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:29:08 smithi062 ceph-mon[16221]: pgmap v275: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:29:11.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:29:10 smithi189 ceph-mon[21562]: pgmap v276: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:29:11.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:29:10 smithi062 ceph-mon[16221]: pgmap v276: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:29:13.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:29:12 smithi189 ceph-mon[21562]: pgmap v277: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:29:13.284 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:29:12 smithi062 ceph-mon[16221]: pgmap v277: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:29:14.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:29:13 smithi062 ceph-mon[16221]: pgmap v278: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:29:14.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:29:13 smithi189 ceph-mon[21562]: pgmap v278: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:29:16.255 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:29:15 smithi062 ceph-mon[16221]: pgmap v279: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:29:16.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:29:15 smithi189 ceph-mon[21562]: pgmap v279: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:29:18.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:29:17 smithi062 ceph-mon[16221]: pgmap v280: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:29:18.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:29:17 smithi189 ceph-mon[21562]: pgmap v280: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:29:19.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:29:18 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:29:19.387 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:29:18 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:29:20.284 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:29:20 smithi062 ceph-mon[16221]: pgmap v281: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:29:20.486 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:29:20 smithi189 ceph-mon[21562]: pgmap v281: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:29:23.215 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:29:22 smithi062 ceph-mon[16221]: pgmap v282: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:29:23.236 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:29:22 smithi189 ceph-mon[21562]: pgmap v282: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:29:23.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:23.270+0000 7f9dabe00700 1 -- 172.21.15.62:0/1926775126 <== mgr.14215 v2:172.21.15.62:6800/4116451241 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+89 (secure 0 0 0) 0x7f9db410c390 con 0x7f9d9806c680 2024-04-20T21:29:23.270 INFO:teuthology.orchestra.run.smithi062.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:a8c84f5997950fd7d008204839253291d52e6eda 2024-04-20T21:29:23.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:23.273+0000 7f9dba400700 1 -- 172.21.15.62:0/1926775126 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f9d9806c680 msgr2=0x7f9d9806eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:29:23.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:23.273+0000 7f9dba400700 1 --2- 172.21.15.62:0/1926775126 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f9d9806c680 0x7f9d9806eb30 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f9da4005950 tx=0x7f9da40058e0 comp rx=0 tx=0).stop 2024-04-20T21:29:23.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:23.273+0000 7f9dba400700 1 -- 172.21.15.62:0/1926775126 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9db4107a40 msgr2=0x7f9db419c620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:29:23.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:23.273+0000 7f9dba400700 1 --2- 172.21.15.62:0/1926775126 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9db4107a40 0x7f9db419c620 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f9da0005b40 tx=0x7f9da000be20 comp rx=0 tx=0).stop 2024-04-20T21:29:23.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:23.273+0000 7f9dba400700 1 -- 172.21.15.62:0/1926775126 shutdown_connections 2024-04-20T21:29:23.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:23.273+0000 7f9dba400700 1 --2- 172.21.15.62:0/1926775126 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f9d9806c680 0x7f9d9806eb30 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:29:23.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:23.273+0000 7f9dba400700 1 --2- 172.21.15.62:0/1926775126 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9db4106840 0x7f9db419c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:29:23.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:23.273+0000 7f9dba400700 1 --2- 172.21.15.62:0/1926775126 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9db4107a40 0x7f9db419c620 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:29:23.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:23.273+0000 7f9dba400700 1 -- 172.21.15.62:0/1926775126 >> 172.21.15.62:0/1926775126 conn(0x7f9db4075970 msgr2=0x7f9db410ac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:29:23.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:23.274+0000 7f9dba400700 1 -- 172.21.15.62:0/1926775126 shutdown_connections 2024-04-20T21:29:23.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:23.274+0000 7f9dba400700 1 -- 172.21.15.62:0/1926775126 wait complete. 2024-04-20T21:29:24.095 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -e sha1=a8c84f5997950fd7d008204839253291d52e6eda -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-04-20T21:29:24.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:29:24 smithi062 ceph-mon[16221]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:a8c84f5997950fd7d008204839253291d52e6eda 2024-04-20T21:29:24.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:29:24 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:29:24.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:29:24 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:29:24.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:29:24 smithi062 ceph-mon[16221]: pgmap v283: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:29:24.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:29:24 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:29:24.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:29:24 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:29:24.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:29:24 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:29:24.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:29:24 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T21:29:24.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:29:24 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:29:24.566 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T21:29:24.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:29:24 smithi189 ceph-mon[21562]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:a8c84f5997950fd7d008204839253291d52e6eda 2024-04-20T21:29:24.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:29:24 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:29:24.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:29:24 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:29:24.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:29:24 smithi189 ceph-mon[21562]: pgmap v283: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:29:24.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:29:24 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:29:24.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:29:24 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:29:24.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:29:24 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:29:24.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:29:24 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T21:29:24.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:29:24 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:29:25.534 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:29:25 smithi062 ceph-mon[16221]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:a8c84f5997950fd7d008204839253291d52e6eda 2024-04-20T21:29:25.736 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:29:25 smithi189 ceph-mon[21562]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:a8c84f5997950fd7d008204839253291d52e6eda 2024-04-20T21:29:26.523 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:29:26 smithi062 ceph-mon[16221]: pgmap v284: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:29:26.736 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:29:26 smithi189 ceph-mon[21562]: pgmap v284: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:29:27.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:27.019+0000 7f8e24e00700 1 -- 172.21.15.62:0/2177574419 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e201033d0 msgr2=0x7f8e201057b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:29:27.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:27.019+0000 7f8e24e00700 1 --2- 172.21.15.62:0/2177574419 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e201033d0 0x7f8e201057b0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f8e0c009910 tx=0x7f8e0c009c20 comp rx=0 tx=0).stop 2024-04-20T21:29:27.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:27.021+0000 7f8e24e00700 1 -- 172.21.15.62:0/2177574419 shutdown_connections 2024-04-20T21:29:27.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:27.021+0000 7f8e24e00700 1 --2- 172.21.15.62:0/2177574419 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e201033d0 0x7f8e201057b0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:29:27.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:27.021+0000 7f8e24e00700 1 --2- 172.21.15.62:0/2177574419 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e20100ab0 0x7f8e20102e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:29:27.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:27.021+0000 7f8e24e00700 1 -- 172.21.15.62:0/2177574419 >> 172.21.15.62:0/2177574419 conn(0x7f8e200fa4d0 msgr2=0x7f8e200fc900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:29:27.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:27.021+0000 7f8e24e00700 1 -- 172.21.15.62:0/2177574419 shutdown_connections 2024-04-20T21:29:27.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:27.021+0000 7f8e24e00700 1 -- 172.21.15.62:0/2177574419 wait complete. 2024-04-20T21:29:27.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:27.022+0000 7f8e24e00700 1 Processor -- start 2024-04-20T21:29:27.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:27.022+0000 7f8e24e00700 1 -- start start 2024-04-20T21:29:27.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:27.023+0000 7f8e24e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e20100ab0 0x7f8e20197a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:29:27.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:27.023+0000 7f8e24e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e201033d0 0x7f8e20197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:29:27.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:27.023+0000 7f8e24e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e201985b0 con 0x7f8e20100ab0 2024-04-20T21:29:27.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:27.023+0000 7f8e24e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e201986f0 con 0x7f8e201033d0 2024-04-20T21:29:27.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:27.023+0000 7f8e1f400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e20100ab0 0x7f8e20197a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:29:27.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:27.023+0000 7f8e1f400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e20100ab0 0x7f8e20197a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49892/0 (socket says 172.21.15.62:49892) 2024-04-20T21:29:27.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:27.023+0000 7f8e1f400700 1 -- 172.21.15.62:0/344668421 learned_addr learned my addr 172.21.15.62:0/344668421 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:29:27.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:27.023+0000 7f8e1ea00700 1 --2- 172.21.15.62:0/344668421 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e201033d0 0x7f8e20197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:29:27.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:27.024+0000 7f8e1f400700 1 -- 172.21.15.62:0/344668421 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e201033d0 msgr2=0x7f8e20197f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:29:27.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:27.024+0000 7f8e1f400700 1 --2- 172.21.15.62:0/344668421 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e201033d0 0x7f8e20197f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:29:27.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:27.024+0000 7f8e1f400700 1 -- 172.21.15.62:0/344668421 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e0c0095f0 con 0x7f8e20100ab0 2024-04-20T21:29:27.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:27.024+0000 7f8e1f400700 1 --2- 172.21.15.62:0/344668421 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e20100ab0 0x7f8e20197a50 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f8e1000e920 tx=0x7f8e1000ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:29:27.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:27.025+0000 7f8e17e00700 1 -- 172.21.15.62:0/344668421 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8e1000c9d0 con 0x7f8e20100ab0 2024-04-20T21:29:27.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:27.025+0000 7f8e17e00700 1 -- 172.21.15.62:0/344668421 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f8e10010430 con 0x7f8e20100ab0 2024-04-20T21:29:27.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:27.025+0000 7f8e17e00700 1 -- 172.21.15.62:0/344668421 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8e10018650 con 0x7f8e20100ab0 2024-04-20T21:29:27.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:27.025+0000 7f8e24e00700 1 -- 172.21.15.62:0/344668421 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e2019d1a0 con 0x7f8e20100ab0 2024-04-20T21:29:27.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:27.025+0000 7f8e24e00700 1 -- 172.21.15.62:0/344668421 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e2019d6f0 con 0x7f8e20100ab0 2024-04-20T21:29:27.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:27.028+0000 7f8e24e00700 1 -- 172.21.15.62:0/344668421 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e200fc090 con 0x7f8e20100ab0 2024-04-20T21:29:27.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:27.029+0000 7f8e17e00700 1 -- 172.21.15.62:0/344668421 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f8e100105a0 con 0x7f8e20100ab0 2024-04-20T21:29:27.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:27.030+0000 7f8e17e00700 1 --2- 172.21.15.62:0/344668421 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f8e0406c7f0 0x7f8e0406eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:29:27.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:27.031+0000 7f8e17e00700 1 -- 172.21.15.62:0/344668421 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5468+0+0 (secure 0 0 0) 0x7f8e10014070 con 0x7f8e20100ab0 2024-04-20T21:29:27.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:27.031+0000 7f8e1ea00700 1 --2- 172.21.15.62:0/344668421 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f8e0406c7f0 0x7f8e0406eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:29:27.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:27.032+0000 7f8e1ea00700 1 --2- 172.21.15.62:0/344668421 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f8e0406c7f0 0x7f8e0406eca0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f8e0c00b6f0 tx=0x7f8e0c009f70 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:29:27.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:27.039+0000 7f8e17e00700 1 -- 172.21.15.62:0/344668421 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f8e10059ef0 con 0x7f8e20100ab0 2024-04-20T21:29:27.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:27.460+0000 7f8e24e00700 1 -- 172.21.15.62:0/344668421 --> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8e20061190 con 0x7f8e0406c7f0 2024-04-20T21:29:27.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:27.461+0000 7f8e17e00700 1 -- 172.21.15.62:0/344668421 <== mgr.14215 v2:172.21.15.62:6800/4116451241 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+351 (secure 0 0 0) 0x7f8e20061190 con 0x7f8e0406c7f0 2024-04-20T21:29:27.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:27.464+0000 7f8e24e00700 1 -- 172.21.15.62:0/344668421 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f8e0406c7f0 msgr2=0x7f8e0406eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:29:27.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:27.464+0000 7f8e24e00700 1 --2- 172.21.15.62:0/344668421 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f8e0406c7f0 0x7f8e0406eca0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f8e0c00b6f0 tx=0x7f8e0c009f70 comp rx=0 tx=0).stop 2024-04-20T21:29:27.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:27.464+0000 7f8e24e00700 1 -- 172.21.15.62:0/344668421 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e20100ab0 msgr2=0x7f8e20197a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:29:27.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:27.464+0000 7f8e24e00700 1 --2- 172.21.15.62:0/344668421 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e20100ab0 0x7f8e20197a50 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f8e1000e920 tx=0x7f8e1000ec30 comp rx=0 tx=0).stop 2024-04-20T21:29:27.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:27.465+0000 7f8e24e00700 1 -- 172.21.15.62:0/344668421 shutdown_connections 2024-04-20T21:29:27.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:27.465+0000 7f8e24e00700 1 --2- 172.21.15.62:0/344668421 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f8e0406c7f0 0x7f8e0406eca0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:29:27.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:27.465+0000 7f8e24e00700 1 --2- 172.21.15.62:0/344668421 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e20100ab0 0x7f8e20197a50 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:29:27.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:27.465+0000 7f8e24e00700 1 --2- 172.21.15.62:0/344668421 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e201033d0 0x7f8e20197f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:29:27.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:27.465+0000 7f8e24e00700 1 -- 172.21.15.62:0/344668421 >> 172.21.15.62:0/344668421 conn(0x7f8e200fa4d0 msgr2=0x7f8e200ff130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:29:27.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:27.465+0000 7f8e24e00700 1 -- 172.21.15.62:0/344668421 shutdown_connections 2024-04-20T21:29:27.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:27.465+0000 7f8e24e00700 1 -- 172.21.15.62:0/344668421 wait complete. 2024-04-20T21:29:27.480 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T21:29:27.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:27.617+0000 7f9885c00700 1 -- 172.21.15.62:0/2480081664 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f98801033d0 msgr2=0x7f98801057b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:29:27.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:27.617+0000 7f9885c00700 1 --2- 172.21.15.62:0/2480081664 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f98801033d0 0x7f98801057b0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f986c009910 tx=0x7f986c009c20 comp rx=0 tx=0).stop 2024-04-20T21:29:27.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:27.618+0000 7f9885c00700 1 -- 172.21.15.62:0/2480081664 shutdown_connections 2024-04-20T21:29:27.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:27.618+0000 7f9885c00700 1 --2- 172.21.15.62:0/2480081664 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f98801033d0 0x7f98801057b0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:29:27.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:27.618+0000 7f9885c00700 1 --2- 172.21.15.62:0/2480081664 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9880100ab0 0x7f9880102e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:29:27.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:27.618+0000 7f9885c00700 1 -- 172.21.15.62:0/2480081664 >> 172.21.15.62:0/2480081664 conn(0x7f98800fa4d0 msgr2=0x7f98800fc900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:29:27.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:27.618+0000 7f9885c00700 1 -- 172.21.15.62:0/2480081664 shutdown_connections 2024-04-20T21:29:27.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:27.618+0000 7f9885c00700 1 -- 172.21.15.62:0/2480081664 wait complete. 2024-04-20T21:29:27.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:27.619+0000 7f9885c00700 1 Processor -- start 2024-04-20T21:29:27.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:27.620+0000 7f9885c00700 1 -- start start 2024-04-20T21:29:27.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:27.620+0000 7f9885c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9880100ab0 0x7f9880195830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:29:27.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:27.620+0000 7f9885c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f98801033d0 0x7f9880195d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:29:27.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:27.620+0000 7f9885c00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9880196390 con 0x7f98801033d0 2024-04-20T21:29:27.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:27.620+0000 7f9885c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98801964d0 con 0x7f9880100ab0 2024-04-20T21:29:27.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:27.620+0000 7f987fe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9880100ab0 0x7f9880195830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:29:27.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:27.620+0000 7f987fe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9880100ab0 0x7f9880195830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.62:56730/0 (socket says 172.21.15.62:56730) 2024-04-20T21:29:27.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:27.620+0000 7f987fe00700 1 -- 172.21.15.62:0/516815390 learned_addr learned my addr 172.21.15.62:0/516815390 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:29:27.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:27.620+0000 7f987f400700 1 --2- 172.21.15.62:0/516815390 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f98801033d0 0x7f9880195d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:29:27.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:27.621+0000 7f987fe00700 1 -- 172.21.15.62:0/516815390 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f98801033d0 msgr2=0x7f9880195d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:29:27.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:27.621+0000 7f987fe00700 1 --2- 172.21.15.62:0/516815390 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f98801033d0 0x7f9880195d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:29:27.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:27.621+0000 7f987fe00700 1 -- 172.21.15.62:0/516815390 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f986c0095f0 con 0x7f9880100ab0 2024-04-20T21:29:27.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:27.622+0000 7f987fe00700 1 --2- 172.21.15.62:0/516815390 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9880100ab0 0x7f9880195830 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f987000e920 tx=0x7f987000ec30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:29:27.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:27.622+0000 7f987cc00700 1 -- 172.21.15.62:0/516815390 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f987000c9d0 con 0x7f9880100ab0 2024-04-20T21:29:27.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:27.622+0000 7f9885c00700 1 -- 172.21.15.62:0/516815390 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f988019af80 con 0x7f9880100ab0 2024-04-20T21:29:27.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:27.622+0000 7f987cc00700 1 -- 172.21.15.62:0/516815390 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f9870010430 con 0x7f9880100ab0 2024-04-20T21:29:27.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:27.622+0000 7f987cc00700 1 -- 172.21.15.62:0/516815390 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9870018650 con 0x7f9880100ab0 2024-04-20T21:29:27.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:27.622+0000 7f9885c00700 1 -- 172.21.15.62:0/516815390 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98800fea70 con 0x7f9880100ab0 2024-04-20T21:29:27.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:27.624+0000 7f9885c00700 1 -- 172.21.15.62:0/516815390 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f988018fa40 con 0x7f9880100ab0 2024-04-20T21:29:27.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:27.625+0000 7f987cc00700 1 -- 172.21.15.62:0/516815390 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f98700105a0 con 0x7f9880100ab0 2024-04-20T21:29:27.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:27.626+0000 7f987cc00700 1 --2- 172.21.15.62:0/516815390 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f9864070a90 0x7f9864072f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:29:27.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:27.626+0000 7f987cc00700 1 -- 172.21.15.62:0/516815390 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5468+0+0 (secure 0 0 0) 0x7f9870014070 con 0x7f9880100ab0 2024-04-20T21:29:27.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:27.626+0000 7f987f400700 1 --2- 172.21.15.62:0/516815390 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f9864070a90 0x7f9864072f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:29:27.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:27.627+0000 7f987f400700 1 --2- 172.21.15.62:0/516815390 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f9864070a90 0x7f9864072f40 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f986c009fd0 tx=0x7f986c005fd0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:29:27.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:27.635+0000 7f987cc00700 1 -- 172.21.15.62:0/516815390 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f9870051720 con 0x7f9880100ab0 2024-04-20T21:29:28.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:28.092+0000 7f9885c00700 1 -- 172.21.15.62:0/516815390 --> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9880061190 con 0x7f9864070a90 2024-04-20T21:29:28.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:28.094+0000 7f987cc00700 1 -- 172.21.15.62:0/516815390 <== mgr.14215 v2:172.21.15.62:6800/4116451241 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+341 (secure 0 0 0) 0x7f9880061190 con 0x7f9864070a90 2024-04-20T21:29:28.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:28.097+0000 7f9885c00700 1 -- 172.21.15.62:0/516815390 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f9864070a90 msgr2=0x7f9864072f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:29:28.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:28.097+0000 7f9885c00700 1 --2- 172.21.15.62:0/516815390 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f9864070a90 0x7f9864072f40 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f986c009fd0 tx=0x7f986c005fd0 comp rx=0 tx=0).stop 2024-04-20T21:29:28.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:28.097+0000 7f9885c00700 1 -- 172.21.15.62:0/516815390 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9880100ab0 msgr2=0x7f9880195830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:29:28.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:28.097+0000 7f9885c00700 1 --2- 172.21.15.62:0/516815390 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9880100ab0 0x7f9880195830 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f987000e920 tx=0x7f987000ec30 comp rx=0 tx=0).stop 2024-04-20T21:29:28.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:28.097+0000 7f9885c00700 1 -- 172.21.15.62:0/516815390 shutdown_connections 2024-04-20T21:29:28.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:28.097+0000 7f9885c00700 1 --2- 172.21.15.62:0/516815390 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f9864070a90 0x7f9864072f40 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:29:28.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:28.097+0000 7f9885c00700 1 --2- 172.21.15.62:0/516815390 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9880100ab0 0x7f9880195830 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:29:28.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:28.097+0000 7f9885c00700 1 --2- 172.21.15.62:0/516815390 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f98801033d0 0x7f9880195d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:29:28.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:28.097+0000 7f9885c00700 1 -- 172.21.15.62:0/516815390 >> 172.21.15.62:0/516815390 conn(0x7f98800fa4d0 msgr2=0x7f98800fc900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:29:28.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:28.098+0000 7f9885c00700 1 -- 172.21.15.62:0/516815390 shutdown_connections 2024-04-20T21:29:28.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:28.098+0000 7f9885c00700 1 -- 172.21.15.62:0/516815390 wait complete. 2024-04-20T21:29:28.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:28.337+0000 7f2d41600700 1 -- 172.21.15.62:0/320376538 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2d3c1012c0 msgr2=0x7f2d3c101730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:29:28.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:28.337+0000 7f2d41600700 1 --2- 172.21.15.62:0/320376538 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2d3c1012c0 0x7f2d3c101730 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f2d28009910 tx=0x7f2d28009c20 comp rx=0 tx=0).stop 2024-04-20T21:29:28.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:28.337+0000 7f2d41600700 1 -- 172.21.15.62:0/320376538 shutdown_connections 2024-04-20T21:29:28.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:28.337+0000 7f2d41600700 1 --2- 172.21.15.62:0/320376538 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2d3c1012c0 0x7f2d3c101730 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:29:28.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:28.337+0000 7f2d41600700 1 --2- 172.21.15.62:0/320376538 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2d3c100020 0x7f2d3c100430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:29:28.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:28.337+0000 7f2d41600700 1 -- 172.21.15.62:0/320376538 >> 172.21.15.62:0/320376538 conn(0x7f2d3c0fb650 msgr2=0x7f2d3c0fdaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:29:28.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:28.337+0000 7f2d41600700 1 -- 172.21.15.62:0/320376538 shutdown_connections 2024-04-20T21:29:28.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:28.338+0000 7f2d41600700 1 -- 172.21.15.62:0/320376538 wait complete. 2024-04-20T21:29:28.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:28.339+0000 7f2d41600700 1 Processor -- start 2024-04-20T21:29:28.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:28.339+0000 7f2d41600700 1 -- start start 2024-04-20T21:29:28.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:28.340+0000 7f2d41600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2d3c100020 0x7f2d3c19bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:29:28.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:28.340+0000 7f2d41600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2d3c1012c0 0x7f2d3c19c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:29:28.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:28.340+0000 7f2d41600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d3c19c950 con 0x7f2d3c100020 2024-04-20T21:29:28.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:28.340+0000 7f2d41600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d3c19ca90 con 0x7f2d3c1012c0 2024-04-20T21:29:28.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:28.340+0000 7f2d3b400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2d3c1012c0 0x7f2d3c19c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:29:28.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:28.340+0000 7f2d3b400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2d3c1012c0 0x7f2d3c19c330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.62:56750/0 (socket says 172.21.15.62:56750) 2024-04-20T21:29:28.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:28.340+0000 7f2d3b400700 1 -- 172.21.15.62:0/1997121256 learned_addr learned my addr 172.21.15.62:0/1997121256 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:29:28.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:28.341+0000 7f2d3be00700 1 --2- 172.21.15.62:0/1997121256 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2d3c100020 0x7f2d3c19bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:29:28.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:28.341+0000 7f2d3b400700 1 -- 172.21.15.62:0/1997121256 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2d3c100020 msgr2=0x7f2d3c19bdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:29:28.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:28.341+0000 7f2d3b400700 1 --2- 172.21.15.62:0/1997121256 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2d3c100020 0x7f2d3c19bdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:29:28.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:28.341+0000 7f2d3b400700 1 -- 172.21.15.62:0/1997121256 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d280095f0 con 0x7f2d3c1012c0 2024-04-20T21:29:28.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:28.342+0000 7f2d3b400700 1 --2- 172.21.15.62:0/1997121256 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2d3c1012c0 0x7f2d3c19c330 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f2d280092e0 tx=0x7f2d2800f740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:29:28.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:28.342+0000 7f2d38c00700 1 -- 172.21.15.62:0/1997121256 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2d2801d070 con 0x7f2d3c1012c0 2024-04-20T21:29:28.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:28.342+0000 7f2d41600700 1 -- 172.21.15.62:0/1997121256 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d3c1a14e0 con 0x7f2d3c1012c0 2024-04-20T21:29:28.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:28.342+0000 7f2d38c00700 1 -- 172.21.15.62:0/1997121256 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f2d280052a0 con 0x7f2d3c1012c0 2024-04-20T21:29:28.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:28.342+0000 7f2d38c00700 1 -- 172.21.15.62:0/1997121256 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2d28017610 con 0x7f2d3c1012c0 2024-04-20T21:29:28.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:28.342+0000 7f2d41600700 1 -- 172.21.15.62:0/1997121256 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d3c1a19d0 con 0x7f2d3c1012c0 2024-04-20T21:29:28.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:28.344+0000 7f2d41600700 1 -- 172.21.15.62:0/1997121256 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d3c1053a0 con 0x7f2d3c1012c0 2024-04-20T21:29:28.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:28.347+0000 7f2d38c00700 1 -- 172.21.15.62:0/1997121256 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f2d2800fba0 con 0x7f2d3c1012c0 2024-04-20T21:29:28.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:28.349+0000 7f2d38c00700 1 --2- 172.21.15.62:0/1997121256 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f2d2006c6d0 0x7f2d2006eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:29:28.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:28.349+0000 7f2d38c00700 1 -- 172.21.15.62:0/1997121256 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5468+0+0 (secure 0 0 0) 0x7f2d2808d0a0 con 0x7f2d3c1012c0 2024-04-20T21:29:28.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:28.349+0000 7f2d3be00700 1 --2- 172.21.15.62:0/1997121256 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f2d2006c6d0 0x7f2d2006eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:29:28.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:28.350+0000 7f2d3be00700 1 --2- 172.21.15.62:0/1997121256 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f2d2006c6d0 0x7f2d2006eb80 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f2d2c009bf0 tx=0x7f2d2c009260 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:29:28.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:28.355+0000 7f2d38c00700 1 -- 172.21.15.62:0/1997121256 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f2d2805b270 con 0x7f2d3c1012c0 2024-04-20T21:29:28.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:28.602+0000 7f2d41600700 1 -- 172.21.15.62:0/1997121256 --> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f2d3c061190 con 0x7f2d2006c6d0 2024-04-20T21:29:28.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:28.622+0000 7f2d38c00700 1 -- 172.21.15.62:0/1997121256 <== mgr.14215 v2:172.21.15.62:6800/4116451241 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3528 (secure 0 0 0) 0x7f2d3c061190 con 0x7f2d2006c6d0 2024-04-20T21:29:28.621 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T21:29:28.621 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (7m) 87s ago 9m 13.1M - 0.25.0 c8568f914cd2 957f663a9cd7 2024-04-20T21:29:28.622 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (9m) 87s ago 9m 7583k - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T21:29:28.622 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (8m) 93s ago 8m 7377k - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T21:29:28.622 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (9m) 87s ago 9m 6656k - 18.2.0 dc2bc1663786 36e10ef5be39 2024-04-20T21:29:28.622 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (8m) 93s ago 8m 6665k - 18.2.0 dc2bc1663786 4101f9953e3d 2024-04-20T21:29:28.622 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (7m) 87s ago 8m 74.1M - 9.4.7 954c08fa6188 99363bb095d8 2024-04-20T21:29:28.622 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (114s) 87s ago 113s 13.6M - 18.2.0 dc2bc1663786 5e6460aa6159 2024-04-20T21:29:28.622 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (2m) 87s ago 2m 14.9M - 18.2.0 dc2bc1663786 e65f2267544e 2024-04-20T21:29:28.622 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (119s) 93s ago 118s 14.4M - 18.2.0 dc2bc1663786 fb2a8a223db8 2024-04-20T21:29:28.622 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (110s) 93s ago 109s 13.0M - 18.2.0 dc2bc1663786 f2eec2ee8951 2024-04-20T21:29:28.622 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:9283,8765,8443 running (10m) 87s ago 10m 488M - 18.2.0 dc2bc1663786 e411b3dc7233 2024-04-20T21:29:28.622 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (7m) 93s ago 7m 437M - 18.2.0 dc2bc1663786 a31cf3b36efc 2024-04-20T21:29:28.622 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (11m) 87s ago 11m 43.9M 2048M 18.2.0 dc2bc1663786 bd6efbd5c380 2024-04-20T21:29:28.622 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (7m) 93s ago 7m 36.1M 2048M 18.2.0 dc2bc1663786 3735c8e5db9a 2024-04-20T21:29:28.623 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (9m) 87s ago 9m 7738k - 1.5.0 0da6a335fe13 fc746d8ae72e 2024-04-20T21:29:28.623 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (7m) 93s ago 7m 7734k - 1.5.0 0da6a335fe13 db03af85b286 2024-04-20T21:29:28.623 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (6m) 87s ago 6m 46.1M 3947M 18.2.0 dc2bc1663786 3f315c3afc1f 2024-04-20T21:29:28.623 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (6m) 87s ago 6m 49.0M 3947M 18.2.0 dc2bc1663786 44e4d07416ad 2024-04-20T21:29:28.623 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (5m) 87s ago 5m 46.5M 3947M 18.2.0 dc2bc1663786 31c5d9586e98 2024-04-20T21:29:28.623 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (5m) 93s ago 5m 45.2M 4971M 18.2.0 dc2bc1663786 611c1e242693 2024-04-20T21:29:28.623 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (4m) 93s ago 4m 42.8M 4971M 18.2.0 dc2bc1663786 6c8fce91d45d 2024-04-20T21:29:28.623 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (3m) 93s ago 3m 42.4M 4971M 18.2.0 dc2bc1663786 8e4574bae8db 2024-04-20T21:29:28.623 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (6m) 87s ago 8m 51.5M - 2.43.0 a07b618ecd1d cd390b7e44d0 2024-04-20T21:29:28.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:28.625+0000 7f2d41600700 1 -- 172.21.15.62:0/1997121256 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f2d2006c6d0 msgr2=0x7f2d2006eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:29:28.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:28.625+0000 7f2d41600700 1 --2- 172.21.15.62:0/1997121256 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f2d2006c6d0 0x7f2d2006eb80 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f2d2c009bf0 tx=0x7f2d2c009260 comp rx=0 tx=0).stop 2024-04-20T21:29:28.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:28.625+0000 7f2d41600700 1 -- 172.21.15.62:0/1997121256 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2d3c1012c0 msgr2=0x7f2d3c19c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:29:28.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:28.625+0000 7f2d41600700 1 --2- 172.21.15.62:0/1997121256 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2d3c1012c0 0x7f2d3c19c330 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f2d280092e0 tx=0x7f2d2800f740 comp rx=0 tx=0).stop 2024-04-20T21:29:28.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:28.625+0000 7f2d41600700 1 -- 172.21.15.62:0/1997121256 shutdown_connections 2024-04-20T21:29:28.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:28.625+0000 7f2d41600700 1 --2- 172.21.15.62:0/1997121256 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f2d2006c6d0 0x7f2d2006eb80 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:29:28.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:28.625+0000 7f2d41600700 1 --2- 172.21.15.62:0/1997121256 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2d3c100020 0x7f2d3c19bdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:29:28.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:28.625+0000 7f2d41600700 1 --2- 172.21.15.62:0/1997121256 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2d3c1012c0 0x7f2d3c19c330 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:29:28.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:28.625+0000 7f2d41600700 1 -- 172.21.15.62:0/1997121256 >> 172.21.15.62:0/1997121256 conn(0x7f2d3c0fb650 msgr2=0x7f2d3c1024e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:29:28.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:28.626+0000 7f2d41600700 1 -- 172.21.15.62:0/1997121256 shutdown_connections 2024-04-20T21:29:28.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:28.626+0000 7f2d41600700 1 -- 172.21.15.62:0/1997121256 wait complete. 2024-04-20T21:29:28.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:28.838+0000 7f0c31c00700 1 -- 172.21.15.62:0/2087048421 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c2c1033d0 msgr2=0x7f0c2c1057b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:29:28.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:28.838+0000 7f0c31c00700 1 --2- 172.21.15.62:0/2087048421 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c2c1033d0 0x7f0c2c1057b0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f0c18009910 tx=0x7f0c18009c20 comp rx=0 tx=0).stop 2024-04-20T21:29:28.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:28.838+0000 7f0c31c00700 1 -- 172.21.15.62:0/2087048421 shutdown_connections 2024-04-20T21:29:28.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:28.838+0000 7f0c31c00700 1 --2- 172.21.15.62:0/2087048421 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c2c1033d0 0x7f0c2c1057b0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:29:28.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:28.838+0000 7f0c31c00700 1 --2- 172.21.15.62:0/2087048421 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c2c100ab0 0x7f0c2c102e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:29:28.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:28.838+0000 7f0c31c00700 1 -- 172.21.15.62:0/2087048421 >> 172.21.15.62:0/2087048421 conn(0x7f0c2c0fa4d0 msgr2=0x7f0c2c0fc900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:29:28.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:28.838+0000 7f0c31c00700 1 -- 172.21.15.62:0/2087048421 shutdown_connections 2024-04-20T21:29:28.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:28.838+0000 7f0c31c00700 1 -- 172.21.15.62:0/2087048421 wait complete. 2024-04-20T21:29:28.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:28.840+0000 7f0c31c00700 1 Processor -- start 2024-04-20T21:29:28.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:28.840+0000 7f0c31c00700 1 -- start start 2024-04-20T21:29:28.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:28.840+0000 7f0c31c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c2c100ab0 0x7f0c2c071f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:29:28.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:28.840+0000 7f0c31c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c2c072470 0x7f0c2c1a3830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:29:28.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:28.840+0000 7f0c31c00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c2c0728e0 con 0x7f0c2c072470 2024-04-20T21:29:28.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:28.840+0000 7f0c31c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c2c072a50 con 0x7f0c2c100ab0 2024-04-20T21:29:28.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:28.841+0000 7f0c2be00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c2c100ab0 0x7f0c2c071f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:29:28.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:28.841+0000 7f0c2be00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c2c100ab0 0x7f0c2c071f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.62:56770/0 (socket says 172.21.15.62:56770) 2024-04-20T21:29:28.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:28.841+0000 7f0c2be00700 1 -- 172.21.15.62:0/65041119 learned_addr learned my addr 172.21.15.62:0/65041119 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:29:28.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:28.841+0000 7f0c2b400700 1 --2- 172.21.15.62:0/65041119 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c2c072470 0x7f0c2c1a3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:29:28.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:28.841+0000 7f0c2be00700 1 -- 172.21.15.62:0/65041119 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c2c072470 msgr2=0x7f0c2c1a3830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:29:28.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:28.841+0000 7f0c2be00700 1 --2- 172.21.15.62:0/65041119 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c2c072470 0x7f0c2c1a3830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:29:28.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:28.842+0000 7f0c2be00700 1 -- 172.21.15.62:0/65041119 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c180095f0 con 0x7f0c2c100ab0 2024-04-20T21:29:28.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:28.842+0000 7f0c2b400700 1 --2- 172.21.15.62:0/65041119 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c2c072470 0x7f0c2c1a3830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-20T21:29:28.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:28.842+0000 7f0c2be00700 1 --2- 172.21.15.62:0/65041119 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c2c100ab0 0x7f0c2c071f30 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f0c1c00e920 tx=0x7f0c1c00ec30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:29:28.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:28.842+0000 7f0c28c00700 1 -- 172.21.15.62:0/65041119 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0c1c00c9d0 con 0x7f0c2c100ab0 2024-04-20T21:29:28.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:28.842+0000 7f0c31c00700 1 -- 172.21.15.62:0/65041119 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c2c1a3d70 con 0x7f0c2c100ab0 2024-04-20T21:29:28.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:28.843+0000 7f0c28c00700 1 -- 172.21.15.62:0/65041119 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f0c1c010430 con 0x7f0c2c100ab0 2024-04-20T21:29:28.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:28.843+0000 7f0c28c00700 1 -- 172.21.15.62:0/65041119 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0c1c018650 con 0x7f0c2c100ab0 2024-04-20T21:29:28.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:28.843+0000 7f0c31c00700 1 -- 172.21.15.62:0/65041119 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c2c1a4230 con 0x7f0c2c100ab0 2024-04-20T21:29:28.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:28.844+0000 7f0c31c00700 1 -- 172.21.15.62:0/65041119 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c0c005320 con 0x7f0c2c100ab0 2024-04-20T21:29:28.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:28.845+0000 7f0c28c00700 1 -- 172.21.15.62:0/65041119 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f0c1c0188c0 con 0x7f0c2c100ab0 2024-04-20T21:29:28.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:28.846+0000 7f0c28c00700 1 --2- 172.21.15.62:0/65041119 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f0c1006c6d0 0x7f0c1006eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:29:28.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:28.846+0000 7f0c28c00700 1 -- 172.21.15.62:0/65041119 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5468+0+0 (secure 0 0 0) 0x7f0c1c014070 con 0x7f0c2c100ab0 2024-04-20T21:29:28.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:28.847+0000 7f0c2b400700 1 --2- 172.21.15.62:0/65041119 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f0c1006c6d0 0x7f0c1006eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:29:28.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:28.847+0000 7f0c2b400700 1 --2- 172.21.15.62:0/65041119 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f0c1006c6d0 0x7f0c1006eb80 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f0c18000c00 tx=0x7f0c18009f70 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:29:28.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:28.853+0000 7f0c28c00700 1 -- 172.21.15.62:0/65041119 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f0c1c051720 con 0x7f0c2c100ab0 2024-04-20T21:29:28.853 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:29:28 smithi062 ceph-mon[16221]: from='client.14550 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:29:28.853 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:29:28 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:29:28.853 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:29:28 smithi062 ceph-mon[16221]: Upgrade: Target is version 19.0.0-3133-ga8c84f59 (unknown) 2024-04-20T21:29:28.853 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:29:28 smithi062 ceph-mon[16221]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, digests ['quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1'] 2024-04-20T21:29:28.853 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:29:28 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:29:28.853 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:29:28 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:29:28.853 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:29:28 smithi062 ceph-mon[16221]: Upgrade: Need to upgrade myself (mgr.smithi062.rqxuxi) 2024-04-20T21:29:28.853 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:29:28 smithi062 ceph-mon[16221]: pgmap v285: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:29:28.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:29:28 smithi189 ceph-mon[21562]: from='client.14550 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:29:28.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:29:28 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:29:28.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:29:28 smithi189 ceph-mon[21562]: Upgrade: Target is version 19.0.0-3133-ga8c84f59 (unknown) 2024-04-20T21:29:28.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:29:28 smithi189 ceph-mon[21562]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, digests ['quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1'] 2024-04-20T21:29:28.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:29:28 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:29:28.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:29:28 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:29:28.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:29:28 smithi189 ceph-mon[21562]: Upgrade: Need to upgrade myself (mgr.smithi062.rqxuxi) 2024-04-20T21:29:28.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:29:28 smithi189 ceph-mon[21562]: pgmap v285: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:29:29.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:29.356+0000 7f0c31c00700 1 -- 172.21.15.62:0/65041119 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f0c0c006200 con 0x7f0c2c100ab0 2024-04-20T21:29:29.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:29.357+0000 7f0c28c00700 1 -- 172.21.15.62:0/65041119 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+558 (secure 0 0 0) 0x7f0c1c056300 con 0x7f0c2c100ab0 2024-04-20T21:29:29.357 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:29:29.357 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T21:29:29.357 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 2 2024-04-20T21:29:29.357 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:29:29.357 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T21:29:29.357 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 2 2024-04-20T21:29:29.357 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:29:29.357 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T21:29:29.357 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 6 2024-04-20T21:29:29.357 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:29:29.357 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T21:29:29.357 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 4 2024-04-20T21:29:29.357 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:29:29.358 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T21:29:29.358 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 14 2024-04-20T21:29:29.358 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T21:29:29.358 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:29:29.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:29.361+0000 7f0c31c00700 1 -- 172.21.15.62:0/65041119 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f0c1006c6d0 msgr2=0x7f0c1006eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:29:29.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:29.361+0000 7f0c31c00700 1 --2- 172.21.15.62:0/65041119 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f0c1006c6d0 0x7f0c1006eb80 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f0c18000c00 tx=0x7f0c18009f70 comp rx=0 tx=0).stop 2024-04-20T21:29:29.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:29.361+0000 7f0c31c00700 1 -- 172.21.15.62:0/65041119 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c2c100ab0 msgr2=0x7f0c2c071f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:29:29.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:29.361+0000 7f0c31c00700 1 --2- 172.21.15.62:0/65041119 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c2c100ab0 0x7f0c2c071f30 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f0c1c00e920 tx=0x7f0c1c00ec30 comp rx=0 tx=0).stop 2024-04-20T21:29:29.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:29.361+0000 7f0c31c00700 1 -- 172.21.15.62:0/65041119 shutdown_connections 2024-04-20T21:29:29.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:29.361+0000 7f0c31c00700 1 --2- 172.21.15.62:0/65041119 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f0c1006c6d0 0x7f0c1006eb80 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:29:29.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:29.361+0000 7f0c31c00700 1 --2- 172.21.15.62:0/65041119 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c2c100ab0 0x7f0c2c071f30 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:29:29.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:29.361+0000 7f0c31c00700 1 --2- 172.21.15.62:0/65041119 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c2c072470 0x7f0c2c1a3830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:29:29.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:29.361+0000 7f0c31c00700 1 -- 172.21.15.62:0/65041119 >> 172.21.15.62:0/65041119 conn(0x7f0c2c0fa4d0 msgr2=0x7f0c2c0fc900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:29:29.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:29.362+0000 7f0c31c00700 1 -- 172.21.15.62:0/65041119 shutdown_connections 2024-04-20T21:29:29.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:29.362+0000 7f0c31c00700 1 -- 172.21.15.62:0/65041119 wait complete. 2024-04-20T21:29:29.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:29.535+0000 7f51e4400700 1 -- 172.21.15.62:0/3505104591 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51dc103350 msgr2=0x7f51dc105790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:29:29.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:29.535+0000 7f51e4400700 1 --2- 172.21.15.62:0/3505104591 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51dc103350 0x7f51dc105790 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f51c8009910 tx=0x7f51c8009c20 comp rx=0 tx=0).stop 2024-04-20T21:29:29.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:29.535+0000 7f51e4400700 1 -- 172.21.15.62:0/3505104591 shutdown_connections 2024-04-20T21:29:29.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:29.535+0000 7f51e4400700 1 --2- 172.21.15.62:0/3505104591 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51dc103350 0x7f51dc105790 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:29:29.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:29.535+0000 7f51e4400700 1 --2- 172.21.15.62:0/3505104591 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f51dc069230 0x7f51dc102e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:29:29.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:29.535+0000 7f51e4400700 1 -- 172.21.15.62:0/3505104591 >> 172.21.15.62:0/3505104591 conn(0x7f51dc0fa780 msgr2=0x7f51dc0fcbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:29:29.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:29.536+0000 7f51e4400700 1 -- 172.21.15.62:0/3505104591 shutdown_connections 2024-04-20T21:29:29.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:29.536+0000 7f51e4400700 1 -- 172.21.15.62:0/3505104591 wait complete. 2024-04-20T21:29:29.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:29.537+0000 7f51e4400700 1 Processor -- start 2024-04-20T21:29:29.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:29.537+0000 7f51e4400700 1 -- start start 2024-04-20T21:29:29.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:29.538+0000 7f51e4400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51dc069230 0x7f51dc100760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:29:29.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:29.538+0000 7f51e4400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f51dc103350 0x7f51dc0fedb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:29:29.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:29.538+0000 7f51e4400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51dc0ff2f0 con 0x7f51dc069230 2024-04-20T21:29:29.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:29.538+0000 7f51e4400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51dc0ff430 con 0x7f51dc103350 2024-04-20T21:29:29.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:29.538+0000 7f51e1a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51dc069230 0x7f51dc100760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:29:29.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:29.538+0000 7f51e1a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51dc069230 0x7f51dc100760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49970/0 (socket says 172.21.15.62:49970) 2024-04-20T21:29:29.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:29.538+0000 7f51e1a00700 1 -- 172.21.15.62:0/3217331268 learned_addr learned my addr 172.21.15.62:0/3217331268 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:29:29.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:29.538+0000 7f51e1000700 1 --2- 172.21.15.62:0/3217331268 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f51dc103350 0x7f51dc0fedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:29:29.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:29.539+0000 7f51e1a00700 1 -- 172.21.15.62:0/3217331268 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f51dc103350 msgr2=0x7f51dc0fedb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:29:29.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:29.539+0000 7f51e1a00700 1 --2- 172.21.15.62:0/3217331268 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f51dc103350 0x7f51dc0fedb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:29:29.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:29.539+0000 7f51e1a00700 1 -- 172.21.15.62:0/3217331268 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51c80095f0 con 0x7f51dc069230 2024-04-20T21:29:29.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:29.539+0000 7f51e1000700 1 --2- 172.21.15.62:0/3217331268 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f51dc103350 0x7f51dc0fedb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-20T21:29:29.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:29.539+0000 7f51e1a00700 1 --2- 172.21.15.62:0/3217331268 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51dc069230 0x7f51dc100760 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f51d000d8d0 tx=0x7f51d000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:29:29.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:29.540+0000 7f51da000700 1 -- 172.21.15.62:0/3217331268 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f51d00041f0 con 0x7f51dc069230 2024-04-20T21:29:29.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:29.540+0000 7f51da000700 1 -- 172.21.15.62:0/3217331268 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f51d0004350 con 0x7f51dc069230 2024-04-20T21:29:29.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:29.540+0000 7f51da000700 1 -- 172.21.15.62:0/3217331268 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f51d0003cf0 con 0x7f51dc069230 2024-04-20T21:29:29.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:29.540+0000 7f51e4400700 1 -- 172.21.15.62:0/3217331268 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51dc0ff710 con 0x7f51dc069230 2024-04-20T21:29:29.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:29.540+0000 7f51e4400700 1 -- 172.21.15.62:0/3217331268 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51dc19db30 con 0x7f51dc069230 2024-04-20T21:29:29.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:29.541+0000 7f51e4400700 1 -- 172.21.15.62:0/3217331268 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51dc18fbf0 con 0x7f51dc069230 2024-04-20T21:29:29.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:29.544+0000 7f51da000700 1 -- 172.21.15.62:0/3217331268 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f51d0009730 con 0x7f51dc069230 2024-04-20T21:29:29.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:29.545+0000 7f51da000700 1 --2- 172.21.15.62:0/3217331268 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f51c406c680 0x7f51c406eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:29:29.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:29.547+0000 7f51e1000700 1 --2- 172.21.15.62:0/3217331268 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f51c406c680 0x7f51c406eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:29:29.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:29.548+0000 7f51da000700 1 -- 172.21.15.62:0/3217331268 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5468+0+0 (secure 0 0 0) 0x7f51d00044c0 con 0x7f51dc069230 2024-04-20T21:29:29.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:29.548+0000 7f51e1000700 1 --2- 172.21.15.62:0/3217331268 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f51c406c680 0x7f51c406eb30 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f51c8006010 tx=0x7f51c800b580 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:29:29.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:29.551+0000 7f51da000700 1 -- 172.21.15.62:0/3217331268 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f51d00595d0 con 0x7f51dc069230 2024-04-20T21:29:29.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:29.811+0000 7f51e4400700 1 -- 172.21.15.62:0/3217331268 --> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f51dc061190 con 0x7f51c406c680 2024-04-20T21:29:29.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:29.813+0000 7f51da000700 1 -- 172.21.15.62:0/3217331268 <== mgr.14215 v2:172.21.15.62:6800/4116451241 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7f51dc061190 con 0x7f51c406c680 2024-04-20T21:29:29.813 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:29:29.813 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T21:29:29.813 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T21:29:29.813 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-20T21:29:29.813 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [], 2024-04-20T21:29:29.813 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "0/2 daemons upgraded", 2024-04-20T21:29:29.813 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 image on host smithi189", 2024-04-20T21:29:29.813 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T21:29:29.813 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:29:29.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:29.816+0000 7f51e4400700 1 -- 172.21.15.62:0/3217331268 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f51c406c680 msgr2=0x7f51c406eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:29:29.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:29.816+0000 7f51e4400700 1 --2- 172.21.15.62:0/3217331268 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f51c406c680 0x7f51c406eb30 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f51c8006010 tx=0x7f51c800b580 comp rx=0 tx=0).stop 2024-04-20T21:29:29.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:29.816+0000 7f51e4400700 1 -- 172.21.15.62:0/3217331268 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51dc069230 msgr2=0x7f51dc100760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:29:29.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:29.816+0000 7f51e4400700 1 --2- 172.21.15.62:0/3217331268 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51dc069230 0x7f51dc100760 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f51d000d8d0 tx=0x7f51d000dc90 comp rx=0 tx=0).stop 2024-04-20T21:29:29.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:29.816+0000 7f51e4400700 1 -- 172.21.15.62:0/3217331268 shutdown_connections 2024-04-20T21:29:29.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:29.816+0000 7f51e4400700 1 --2- 172.21.15.62:0/3217331268 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f51c406c680 0x7f51c406eb30 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:29:29.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:29.816+0000 7f51e4400700 1 --2- 172.21.15.62:0/3217331268 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51dc069230 0x7f51dc100760 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:29:29.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:29.816+0000 7f51e4400700 1 --2- 172.21.15.62:0/3217331268 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f51dc103350 0x7f51dc0fedb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:29:29.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:29.816+0000 7f51e4400700 1 -- 172.21.15.62:0/3217331268 >> 172.21.15.62:0/3217331268 conn(0x7f51dc0fa780 msgr2=0x7f51dc0fcbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:29:29.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:29.817+0000 7f51e4400700 1 -- 172.21.15.62:0/3217331268 shutdown_connections 2024-04-20T21:29:29.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:29:29.817+0000 7f51e4400700 1 -- 172.21.15.62:0/3217331268 wait complete. 2024-04-20T21:29:29.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:29:29 smithi189 ceph-mon[21562]: from='client.24345 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:29:29.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:29:29 smithi189 ceph-mon[21562]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 on smithi189 2024-04-20T21:29:29.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:29:29 smithi189 ceph-mon[21562]: from='client.24349 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:29:29.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:29:29 smithi189 ceph-mon[21562]: from='client.? 172.21.15.62:0/65041119' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:29:30.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:29:29 smithi062 ceph-mon[16221]: from='client.24345 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:29:30.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:29:29 smithi062 ceph-mon[16221]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 on smithi189 2024-04-20T21:29:30.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:29:29 smithi062 ceph-mon[16221]: from='client.24349 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:29:30.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:29:29 smithi062 ceph-mon[16221]: from='client.? 172.21.15.62:0/65041119' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:29:30.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:29:30 smithi189 ceph-mon[21562]: from='client.14566 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:29:30.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:29:30 smithi189 ceph-mon[21562]: pgmap v286: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:29:31.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:29:30 smithi062 ceph-mon[16221]: from='client.14566 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:29:31.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:29:30 smithi062 ceph-mon[16221]: pgmap v286: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:29:33.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:29:32 smithi189 ceph-mon[21562]: pgmap v287: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:29:33.284 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:29:32 smithi062 ceph-mon[16221]: pgmap v287: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:29:34.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:29:33 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:29:34.284 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:29:33 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:29:35.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:29:34 smithi189 ceph-mon[21562]: pgmap v288: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:29:35.284 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:29:34 smithi062 ceph-mon[16221]: pgmap v288: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:29:36.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:29:35 smithi062 ceph-mon[16221]: pgmap v289: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:29:36.492 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:29:35 smithi189 ceph-mon[21562]: pgmap v289: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:29:38.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:29:37 smithi062 ceph-mon[16221]: pgmap v290: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:29:38.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:29:38 smithi189 ceph-mon[21562]: pgmap v290: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:29:40.284 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:29:39 smithi062 ceph-mon[16221]: pgmap v291: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:29:40.486 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:29:39 smithi189 ceph-mon[21562]: pgmap v291: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:29:43.236 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:29:42 smithi189 ceph-mon[21562]: pgmap v292: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:29:43.284 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:29:42 smithi062 ceph-mon[16221]: pgmap v292: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:29:44.284 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:29:43 smithi062 ceph-mon[16221]: pgmap v293: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:29:44.486 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:29:43 smithi189 ceph-mon[21562]: pgmap v293: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:29:46.284 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:29:45 smithi062 ceph-mon[16221]: pgmap v294: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:29:46.486 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:29:45 smithi189 ceph-mon[21562]: pgmap v294: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:29:48.284 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:29:47 smithi062 ceph-mon[16221]: pgmap v295: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:29:48.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:29:47 smithi189 ceph-mon[21562]: pgmap v295: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:29:49.284 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:29:48 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:29:49.387 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:29:48 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:29:50.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:29:50 smithi062 ceph-mon[16221]: pgmap v296: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:29:50.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:29:50 smithi189 ceph-mon[21562]: pgmap v296: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:29:52.986 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:29:52 smithi189 ceph-mon[21562]: pgmap v297: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:29:53.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:29:52 smithi062 ceph-mon[16221]: pgmap v297: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:29:54.284 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:29:53 smithi062 ceph-mon[16221]: pgmap v298: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:29:54.486 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:29:54 smithi189 ceph-mon[21562]: pgmap v298: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:29:54.526 INFO:tasks.workunit.client.0.smithi062.stderr:Updating files: 58% (7170/12271) Updating files: 59% (7240/12271) Updating files: 60% (7363/12271) Updating files: 61% (7486/12271) Updating files: 62% (7609/12271) Updating files: 63% (7731/12271) Updating files: 64% (7854/12271) Updating files: 65% (7977/12271) Updating files: 66% (8099/12271) Updating files: 67% (8222/12271) Updating files: 68% (8345/12271) Updating files: 69% (8467/12271) Updating files: 70% (8590/12271) Updating files: 71% (8713/12271) Updating files: 72% (8836/12271) Updating files: 73% (8958/12271) Updating files: 74% (9081/12271) Updating files: 75% (9204/12271) Updating files: 76% (9326/12271) Updating files: 77% (9449/12271) Updating files: 78% (9572/12271) Updating files: 79% (9695/12271) Updating files: 80% (9817/12271) Updating files: 81% (9940/12271) Updating files: 82% (10063/12271) Updating files: 83% (10185/12271) Updating files: 84% (10308/12271) Updating files: 85% (10431/12271) Updating files: 86% (10554/12271) Updating files: 87% (10676/12271) Updating files: 88% (10799/12271) Updating files: 89% (10922/12271) Updating files: 90% (11044/12271) Updating files: 91% (11167/12271) Updating files: 92% (11290/12271) Updating files: 93% (11413/12271) Updating files: 94% (11535/12271) Updating files: 95% (11658/12271) Updating files: 95% (11760/12271) Updating files: 96% (11781/12271) Updating files: 97% (11903/12271) Updating files: 98% (12026/12271) Updating files: 99% (12149/12271) Updating files: 100% (12271/12271) Updating files: 100% (12271/12271), done. 2024-04-20T21:29:55.374 INFO:tasks.workunit.client.0.smithi062.stderr:Note: switching to 'a8c84f5997950fd7d008204839253291d52e6eda'. 2024-04-20T21:29:55.374 INFO:tasks.workunit.client.0.smithi062.stderr: 2024-04-20T21:29:55.374 INFO:tasks.workunit.client.0.smithi062.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-20T21:29:55.374 INFO:tasks.workunit.client.0.smithi062.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-20T21:29:55.374 INFO:tasks.workunit.client.0.smithi062.stderr:state without impacting any branches by switching back to a branch. 2024-04-20T21:29:55.374 INFO:tasks.workunit.client.0.smithi062.stderr: 2024-04-20T21:29:55.374 INFO:tasks.workunit.client.0.smithi062.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-20T21:29:55.374 INFO:tasks.workunit.client.0.smithi062.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-20T21:29:55.374 INFO:tasks.workunit.client.0.smithi062.stderr: 2024-04-20T21:29:55.374 INFO:tasks.workunit.client.0.smithi062.stderr: git switch -c 2024-04-20T21:29:55.375 INFO:tasks.workunit.client.0.smithi062.stderr: 2024-04-20T21:29:55.375 INFO:tasks.workunit.client.0.smithi062.stderr:Or undo this operation with: 2024-04-20T21:29:55.375 INFO:tasks.workunit.client.0.smithi062.stderr: 2024-04-20T21:29:55.375 INFO:tasks.workunit.client.0.smithi062.stderr: git switch - 2024-04-20T21:29:55.375 INFO:tasks.workunit.client.0.smithi062.stderr: 2024-04-20T21:29:55.375 INFO:tasks.workunit.client.0.smithi062.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-20T21:29:55.375 INFO:tasks.workunit.client.0.smithi062.stderr: 2024-04-20T21:29:55.375 INFO:tasks.workunit.client.0.smithi062.stderr:HEAD is now at a8c84f59979 Merge PR #51659 into wip-rishabh-testing-20240416.193735 2024-04-20T21:29:55.391 DEBUG:teuthology.orchestra.run.smithi062:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-04-20T21:29:55.456 INFO:tasks.workunit.client.0.smithi062.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-20T21:29:55.464 INFO:tasks.workunit.client.0.smithi062.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-20T21:29:55.464 INFO:tasks.workunit.client.0.smithi062.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-20T21:29:56.122 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:29:56 smithi062 ceph-mon[16221]: pgmap v299: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:29:56.368 INFO:tasks.workunit.client.0.smithi062.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-20T21:29:56.485 INFO:tasks.workunit.client.0.smithi062.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-20T21:29:56.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:29:56 smithi189 ceph-mon[21562]: pgmap v299: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:29:56.573 INFO:tasks.workunit.client.0.smithi062.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-20T21:29:56.579 INFO:tasks.workunit.client.0.smithi062.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-20T21:29:56.579 INFO:tasks.workunit.client.0.smithi062.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-20T21:29:56.660 INFO:tasks.workunit.client.0.smithi062.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-20T21:29:56.671 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-04-20T21:29:56.671 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-04-20T21:29:56.719 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.0... 2024-04-20T21:29:56.721 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2024-04-20T21:29:56.721 DEBUG:teuthology.orchestra.run.smithi062:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a8c84f5997950fd7d008204839253291d52e6eda TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh 2024-04-20T21:29:56.836 INFO:tasks.workunit.client.0.smithi062.stderr:+ mkdir -p fsstress 2024-04-20T21:29:56.843 INFO:tasks.workunit.client.0.smithi062.stderr:+ pushd fsstress 2024-04-20T21:29:56.848 INFO:tasks.workunit.client.0.smithi062.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp 2024-04-20T21:29:56.849 INFO:tasks.workunit.client.0.smithi062.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2024-04-20T21:29:57.962 INFO:tasks.workunit.client.0.smithi062.stderr:+ tar xzf ltp-full.tgz 2024-04-20T21:29:58.283 INFO:tasks.workunit.client.1.smithi189.stderr:Updating files: 43% (5284/12271) Updating files: 44% (5400/12271) Updating files: 45% (5522/12271) Updating files: 46% (5645/12271) Updating files: 47% (5768/12271) Updating files: 48% (5891/12271) Updating files: 49% (6013/12271) Updating files: 50% (6136/12271) Updating files: 51% (6259/12271) Updating files: 52% (6381/12271) Updating files: 53% (6504/12271) Updating files: 54% (6627/12271) Updating files: 55% (6750/12271) Updating files: 56% (6872/12271) Updating files: 57% (6995/12271) Updating files: 58% (7118/12271) Updating files: 59% (7240/12271) Updating files: 60% (7363/12271) Updating files: 61% (7486/12271) Updating files: 62% (7609/12271) Updating files: 63% (7731/12271) Updating files: 64% (7854/12271) Updating files: 65% (7977/12271) Updating files: 66% (8099/12271) Updating files: 67% (8222/12271) Updating files: 68% (8345/12271) Updating files: 69% (8467/12271) Updating files: 70% (8590/12271) Updating files: 71% (8713/12271) Updating files: 72% (8836/12271) Updating files: 73% (8958/12271) Updating files: 74% (9081/12271) Updating files: 74% (9185/12271) Updating files: 75% (9204/12271) Updating files: 76% (9326/12271) Updating files: 77% (9449/12271) Updating files: 78% (9572/12271) Updating files: 79% (9695/12271) Updating files: 80% (9817/12271) Updating files: 81% (9940/12271) Updating files: 82% (10063/12271) Updating files: 83% (10185/12271) Updating files: 84% (10308/12271) Updating files: 85% (10431/12271) Updating files: 86% (10554/12271) Updating files: 87% (10676/12271) Updating files: 88% (10799/12271) Updating files: 89% (10922/12271) Updating files: 90% (11044/12271) Updating files: 91% (11167/12271) Updating files: 92% (11290/12271) Updating files: 93% (11413/12271) Updating files: 94% (11535/12271) Updating files: 95% (11658/12271) Updating files: 96% (11781/12271) Updating files: 97% (11903/12271) Updating files: 98% (12026/12271) Updating files: 99% (12149/12271) Updating files: 100% (12271/12271) Updating files: 100% (12271/12271), done. 2024-04-20T21:29:58.284 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:29:58 smithi062 ceph-mon[16221]: pgmap v300: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:29:58.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:29:58 smithi189 ceph-mon[21562]: pgmap v300: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:29:59.034 INFO:tasks.workunit.client.1.smithi189.stderr:Note: switching to 'a8c84f5997950fd7d008204839253291d52e6eda'. 2024-04-20T21:29:59.034 INFO:tasks.workunit.client.1.smithi189.stderr: 2024-04-20T21:29:59.034 INFO:tasks.workunit.client.1.smithi189.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-20T21:29:59.034 INFO:tasks.workunit.client.1.smithi189.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-20T21:29:59.034 INFO:tasks.workunit.client.1.smithi189.stderr:state without impacting any branches by switching back to a branch. 2024-04-20T21:29:59.034 INFO:tasks.workunit.client.1.smithi189.stderr: 2024-04-20T21:29:59.034 INFO:tasks.workunit.client.1.smithi189.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-20T21:29:59.034 INFO:tasks.workunit.client.1.smithi189.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-20T21:29:59.034 INFO:tasks.workunit.client.1.smithi189.stderr: 2024-04-20T21:29:59.034 INFO:tasks.workunit.client.1.smithi189.stderr: git switch -c 2024-04-20T21:29:59.034 INFO:tasks.workunit.client.1.smithi189.stderr: 2024-04-20T21:29:59.034 INFO:tasks.workunit.client.1.smithi189.stderr:Or undo this operation with: 2024-04-20T21:29:59.035 INFO:tasks.workunit.client.1.smithi189.stderr: 2024-04-20T21:29:59.035 INFO:tasks.workunit.client.1.smithi189.stderr: git switch - 2024-04-20T21:29:59.035 INFO:tasks.workunit.client.1.smithi189.stderr: 2024-04-20T21:29:59.035 INFO:tasks.workunit.client.1.smithi189.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-20T21:29:59.035 INFO:tasks.workunit.client.1.smithi189.stderr: 2024-04-20T21:29:59.035 INFO:tasks.workunit.client.1.smithi189.stderr:HEAD is now at a8c84f59979 Merge PR #51659 into wip-rishabh-testing-20240416.193735 2024-04-20T21:29:59.050 DEBUG:teuthology.orchestra.run.smithi189:> cd -- /home/ubuntu/cephtest/clone.client.1/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.1 2024-04-20T21:29:59.106 INFO:tasks.workunit.client.1.smithi189.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-20T21:29:59.111 INFO:tasks.workunit.client.1.smithi189.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-04-20T21:29:59.111 INFO:tasks.workunit.client.1.smithi189.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-20T21:30:00.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.014+0000 7fe1f0800700 1 -- 172.21.15.62:0/11227323 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe1e8102490 msgr2=0x7fe1e81028a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:30:00.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.014+0000 7fe1f0800700 1 --2- 172.21.15.62:0/11227323 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe1e8102490 0x7fe1e81028a0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fe1dc009910 tx=0x7fe1dc009c20 comp rx=0 tx=0).stop 2024-04-20T21:30:00.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.015+0000 7fe1f0800700 1 -- 172.21.15.62:0/11227323 shutdown_connections 2024-04-20T21:30:00.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.015+0000 7fe1f0800700 1 --2- 172.21.15.62:0/11227323 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1e8103640 0x7fe1e8103a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:30:00.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.015+0000 7fe1f0800700 1 --2- 172.21.15.62:0/11227323 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe1e8102490 0x7fe1e81028a0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:30:00.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.015+0000 7fe1f0800700 1 -- 172.21.15.62:0/11227323 >> 172.21.15.62:0/11227323 conn(0x7fe1e80fda60 msgr2=0x7fe1e80ffe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:30:00.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.016+0000 7fe1f0800700 1 -- 172.21.15.62:0/11227323 shutdown_connections 2024-04-20T21:30:00.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.016+0000 7fe1f0800700 1 -- 172.21.15.62:0/11227323 wait complete. 2024-04-20T21:30:00.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.017+0000 7fe1f0800700 1 Processor -- start 2024-04-20T21:30:00.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.017+0000 7fe1f0800700 1 -- start start 2024-04-20T21:30:00.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.018+0000 7fe1f0800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1e8102490 0x7fe1e8197d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:30:00.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.018+0000 7fe1f0800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe1e8103640 0x7fe1e8198240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:30:00.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.018+0000 7fe1f0800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1e8198860 con 0x7fe1e8103640 2024-04-20T21:30:00.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.018+0000 7fe1f0800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1e819d270 con 0x7fe1e8102490 2024-04-20T21:30:00.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.018+0000 7fe1ed400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe1e8103640 0x7fe1e8198240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:30:00.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.018+0000 7fe1ed400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe1e8103640 0x7fe1e8198240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55682/0 (socket says 172.21.15.62:55682) 2024-04-20T21:30:00.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.018+0000 7fe1ed400700 1 -- 172.21.15.62:0/504177842 learned_addr learned my addr 172.21.15.62:0/504177842 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:30:00.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.018+0000 7fe1ede00700 1 --2- 172.21.15.62:0/504177842 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1e8102490 0x7fe1e8197d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:30:00.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.019+0000 7fe1ed400700 1 -- 172.21.15.62:0/504177842 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1e8102490 msgr2=0x7fe1e8197d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:30:00.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.019+0000 7fe1ed400700 1 --2- 172.21.15.62:0/504177842 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1e8102490 0x7fe1e8197d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:30:00.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.019+0000 7fe1ed400700 1 -- 172.21.15.62:0/504177842 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1dc0095f0 con 0x7fe1e8103640 2024-04-20T21:30:00.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.019+0000 7fe1ede00700 1 --2- 172.21.15.62:0/504177842 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1e8102490 0x7fe1e8197d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-20T21:30:00.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.019+0000 7fe1ed400700 1 --2- 172.21.15.62:0/504177842 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe1e8103640 0x7fe1e8198240 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fe1d800d8d0 tx=0x7fe1d800dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:30:00.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.020+0000 7fe1e2a00700 1 -- 172.21.15.62:0/504177842 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe1d80098a0 con 0x7fe1e8103640 2024-04-20T21:30:00.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.020+0000 7fe1e2a00700 1 -- 172.21.15.62:0/504177842 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fe1d8010460 con 0x7fe1e8103640 2024-04-20T21:30:00.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.020+0000 7fe1e2a00700 1 -- 172.21.15.62:0/504177842 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe1d800f5f0 con 0x7fe1e8103640 2024-04-20T21:30:00.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.020+0000 7fe1f0800700 1 -- 172.21.15.62:0/504177842 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1e819d470 con 0x7fe1e8103640 2024-04-20T21:30:00.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.020+0000 7fe1f0800700 1 -- 172.21.15.62:0/504177842 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1e819d990 con 0x7fe1e8103640 2024-04-20T21:30:00.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.021+0000 7fe1f0800700 1 -- 172.21.15.62:0/504177842 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1e810af90 con 0x7fe1e8103640 2024-04-20T21:30:00.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.023+0000 7fe1e2a00700 1 -- 172.21.15.62:0/504177842 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7fe1d8009a00 con 0x7fe1e8103640 2024-04-20T21:30:00.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.024+0000 7fe1e2a00700 1 --2- 172.21.15.62:0/504177842 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fe1d006c680 0x7fe1d006eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:30:00.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.024+0000 7fe1e2a00700 1 -- 172.21.15.62:0/504177842 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5468+0+0 (secure 0 0 0) 0x7fe1d808b4c0 con 0x7fe1e8103640 2024-04-20T21:30:00.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.024+0000 7fe1ede00700 1 --2- 172.21.15.62:0/504177842 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fe1d006c680 0x7fe1d006eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:30:00.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.025+0000 7fe1ede00700 1 --2- 172.21.15.62:0/504177842 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fe1d006c680 0x7fe1d006eb30 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fe1dc009fd0 tx=0x7fe1dc00b560 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:30:00.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.030+0000 7fe1e2a00700 1 -- 172.21.15.62:0/504177842 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fe1d8055b80 con 0x7fe1e8103640 2024-04-20T21:30:00.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:00 smithi062 ceph-mon[16221]: pgmap v301: 65 pgs: 65 active+clean; 8.8 MiB data, 177 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 699 KiB/s wr, 5 op/s 2024-04-20T21:30:00.291 INFO:tasks.workunit.client.1.smithi189.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-20T21:30:00.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.309+0000 7fe1f0800700 1 -- 172.21.15.62:0/504177842 --> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe1e8107f90 con 0x7fe1d006c680 2024-04-20T21:30:00.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.310+0000 7fe1e2a00700 1 -- 172.21.15.62:0/504177842 <== mgr.14215 v2:172.21.15.62:6800/4116451241 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7fe1e8107f90 con 0x7fe1d006c680 2024-04-20T21:30:00.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.313+0000 7fe1f0800700 1 -- 172.21.15.62:0/504177842 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fe1d006c680 msgr2=0x7fe1d006eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:30:00.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.313+0000 7fe1f0800700 1 --2- 172.21.15.62:0/504177842 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fe1d006c680 0x7fe1d006eb30 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fe1dc009fd0 tx=0x7fe1dc00b560 comp rx=0 tx=0).stop 2024-04-20T21:30:00.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.313+0000 7fe1f0800700 1 -- 172.21.15.62:0/504177842 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe1e8103640 msgr2=0x7fe1e8198240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:30:00.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.313+0000 7fe1f0800700 1 --2- 172.21.15.62:0/504177842 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe1e8103640 0x7fe1e8198240 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fe1d800d8d0 tx=0x7fe1d800dbe0 comp rx=0 tx=0).stop 2024-04-20T21:30:00.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.313+0000 7fe1f0800700 1 -- 172.21.15.62:0/504177842 shutdown_connections 2024-04-20T21:30:00.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.313+0000 7fe1f0800700 1 --2- 172.21.15.62:0/504177842 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fe1d006c680 0x7fe1d006eb30 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:30:00.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.313+0000 7fe1f0800700 1 --2- 172.21.15.62:0/504177842 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1e8102490 0x7fe1e8197d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:30:00.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.313+0000 7fe1f0800700 1 --2- 172.21.15.62:0/504177842 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe1e8103640 0x7fe1e8198240 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:30:00.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.313+0000 7fe1f0800700 1 -- 172.21.15.62:0/504177842 >> 172.21.15.62:0/504177842 conn(0x7fe1e80fda60 msgr2=0x7fe1e8106870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:30:00.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.313+0000 7fe1f0800700 1 -- 172.21.15.62:0/504177842 shutdown_connections 2024-04-20T21:30:00.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.314+0000 7fe1f0800700 1 -- 172.21.15.62:0/504177842 wait complete. 2024-04-20T21:30:00.330 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T21:30:00.398 INFO:tasks.workunit.client.1.smithi189.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-20T21:30:00.486 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:00 smithi189 ceph-mon[21562]: pgmap v301: 65 pgs: 65 active+clean; 8.8 MiB data, 177 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 699 KiB/s wr, 5 op/s 2024-04-20T21:30:00.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.494+0000 7f37c3e00700 1 -- 172.21.15.62:0/270829908 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f37c41033d0 msgr2=0x7f37c41057b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:30:00.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.494+0000 7f37c3e00700 1 --2- 172.21.15.62:0/270829908 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f37c41033d0 0x7f37c41057b0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f37b0009910 tx=0x7f37b0009c20 comp rx=0 tx=0).stop 2024-04-20T21:30:00.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.495+0000 7f37c3e00700 1 -- 172.21.15.62:0/270829908 shutdown_connections 2024-04-20T21:30:00.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.495+0000 7f37c3e00700 1 --2- 172.21.15.62:0/270829908 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f37c41033d0 0x7f37c41057b0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:30:00.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.495+0000 7f37c3e00700 1 --2- 172.21.15.62:0/270829908 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f37c4100ab0 0x7f37c4102e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:30:00.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.495+0000 7f37c3e00700 1 -- 172.21.15.62:0/270829908 >> 172.21.15.62:0/270829908 conn(0x7f37c40fa4d0 msgr2=0x7f37c40fc900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:30:00.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.495+0000 7f37c3e00700 1 -- 172.21.15.62:0/270829908 shutdown_connections 2024-04-20T21:30:00.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.495+0000 7f37c3e00700 1 -- 172.21.15.62:0/270829908 wait complete. 2024-04-20T21:30:00.495 INFO:tasks.workunit.client.1.smithi189.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-04-20T21:30:00.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.496+0000 7f37c3e00700 1 Processor -- start 2024-04-20T21:30:00.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.496+0000 7f37c3e00700 1 -- start start 2024-04-20T21:30:00.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.496+0000 7f37c3e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f37c4100ab0 0x7f37c4197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:30:00.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.496+0000 7f37c3e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f37c41033d0 0x7f37c4197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:30:00.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.496+0000 7f37c3e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37c4198590 con 0x7f37c4100ab0 2024-04-20T21:30:00.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.496+0000 7f37c3e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37c41986d0 con 0x7f37c41033d0 2024-04-20T21:30:00.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.497+0000 7f37c2a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f37c4100ab0 0x7f37c4197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:30:00.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.497+0000 7f37c2000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f37c41033d0 0x7f37c4197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:30:00.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.497+0000 7f37c2a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f37c4100ab0 0x7f37c4197a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55692/0 (socket says 172.21.15.62:55692) 2024-04-20T21:30:00.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.497+0000 7f37c2a00700 1 -- 172.21.15.62:0/774724643 learned_addr learned my addr 172.21.15.62:0/774724643 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:30:00.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.497+0000 7f37c2a00700 1 -- 172.21.15.62:0/774724643 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f37c41033d0 msgr2=0x7f37c4197f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:30:00.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.497+0000 7f37c2a00700 1 --2- 172.21.15.62:0/774724643 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f37c41033d0 0x7f37c4197f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:30:00.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.497+0000 7f37c2a00700 1 -- 172.21.15.62:0/774724643 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37b00095f0 con 0x7f37c4100ab0 2024-04-20T21:30:00.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.498+0000 7f37c2a00700 1 --2- 172.21.15.62:0/774724643 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f37c4100ab0 0x7f37c4197a30 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f37b400e920 tx=0x7f37b400ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:30:00.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.498+0000 7f37bb400700 1 -- 172.21.15.62:0/774724643 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f37b400c9d0 con 0x7f37c4100ab0 2024-04-20T21:30:00.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.498+0000 7f37bb400700 1 -- 172.21.15.62:0/774724643 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f37b4010430 con 0x7f37c4100ab0 2024-04-20T21:30:00.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.498+0000 7f37bb400700 1 -- 172.21.15.62:0/774724643 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f37b4018650 con 0x7f37c4100ab0 2024-04-20T21:30:00.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.498+0000 7f37c3e00700 1 -- 172.21.15.62:0/774724643 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37c419d180 con 0x7f37c4100ab0 2024-04-20T21:30:00.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.499+0000 7f37c3e00700 1 -- 172.21.15.62:0/774724643 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37c419d680 con 0x7f37c4100ab0 2024-04-20T21:30:00.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.500+0000 7f37c3e00700 1 -- 172.21.15.62:0/774724643 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37c40fc090 con 0x7f37c4100ab0 2024-04-20T21:30:00.499 INFO:tasks.workunit.client.1.smithi189.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-04-20T21:30:00.499 INFO:tasks.workunit.client.1.smithi189.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-20T21:30:00.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.502+0000 7f37bb400700 1 -- 172.21.15.62:0/774724643 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7f37b40105a0 con 0x7f37c4100ab0 2024-04-20T21:30:00.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.503+0000 7f37bb400700 1 --2- 172.21.15.62:0/774724643 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f37a806c680 0x7f37a806eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:30:00.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.504+0000 7f37bb400700 1 -- 172.21.15.62:0/774724643 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5468+0+0 (secure 0 0 0) 0x7f37b4014070 con 0x7f37c4100ab0 2024-04-20T21:30:00.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.504+0000 7f37c2000700 1 --2- 172.21.15.62:0/774724643 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f37a806c680 0x7f37a806eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:30:00.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.505+0000 7f37c2000700 1 --2- 172.21.15.62:0/774724643 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f37a806c680 0x7f37a806eb30 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f37b000b6f0 tx=0x7f37b0005a90 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:30:00.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.507+0000 7f37bb400700 1 -- 172.21.15.62:0/774724643 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f37b405a330 con 0x7f37c4100ab0 2024-04-20T21:30:00.574 INFO:tasks.workunit.client.1.smithi189.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-04-20T21:30:00.583 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-20T21:30:00.583 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/home/ubuntu/cephtest/workunits.list.client.1 of=/dev/stdout 2024-04-20T21:30:00.629 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.1... 2024-04-20T21:30:00.630 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2024-04-20T21:30:00.630 DEBUG:teuthology.orchestra.run.smithi189:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a8c84f5997950fd7d008204839253291d52e6eda TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh 2024-04-20T21:30:00.746 INFO:tasks.workunit.client.1.smithi189.stderr:+ mkdir -p fsstress 2024-04-20T21:30:00.753 INFO:tasks.workunit.client.1.smithi189.stderr:+ pushd fsstress 2024-04-20T21:30:00.757 INFO:tasks.workunit.client.1.smithi189.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-04-20T21:30:00.757 INFO:tasks.workunit.client.1.smithi189.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2024-04-20T21:30:00.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.766+0000 7f37c3e00700 1 -- 172.21.15.62:0/774724643 --> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f37c4061190 con 0x7f37a806c680 2024-04-20T21:30:00.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.768+0000 7f37bb400700 1 -- 172.21.15.62:0/774724643 <== mgr.14215 v2:172.21.15.62:6800/4116451241 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7f37c4061190 con 0x7f37a806c680 2024-04-20T21:30:00.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.771+0000 7f37c3e00700 1 -- 172.21.15.62:0/774724643 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f37a806c680 msgr2=0x7f37a806eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:30:00.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.771+0000 7f37c3e00700 1 --2- 172.21.15.62:0/774724643 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f37a806c680 0x7f37a806eb30 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f37b000b6f0 tx=0x7f37b0005a90 comp rx=0 tx=0).stop 2024-04-20T21:30:00.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.771+0000 7f37c3e00700 1 -- 172.21.15.62:0/774724643 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f37c4100ab0 msgr2=0x7f37c4197a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:30:00.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.771+0000 7f37c3e00700 1 --2- 172.21.15.62:0/774724643 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f37c4100ab0 0x7f37c4197a30 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f37b400e920 tx=0x7f37b400ec30 comp rx=0 tx=0).stop 2024-04-20T21:30:00.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.771+0000 7f37c3e00700 1 -- 172.21.15.62:0/774724643 shutdown_connections 2024-04-20T21:30:00.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.771+0000 7f37c3e00700 1 --2- 172.21.15.62:0/774724643 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f37a806c680 0x7f37a806eb30 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:30:00.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.771+0000 7f37c3e00700 1 --2- 172.21.15.62:0/774724643 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f37c4100ab0 0x7f37c4197a30 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:30:00.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.771+0000 7f37c3e00700 1 --2- 172.21.15.62:0/774724643 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f37c41033d0 0x7f37c4197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:30:00.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.771+0000 7f37c3e00700 1 -- 172.21.15.62:0/774724643 >> 172.21.15.62:0/774724643 conn(0x7f37c40fa4d0 msgr2=0x7f37c4104030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:30:00.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.772+0000 7f37c3e00700 1 -- 172.21.15.62:0/774724643 shutdown_connections 2024-04-20T21:30:00.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.772+0000 7f37c3e00700 1 -- 172.21.15.62:0/774724643 wait complete. 2024-04-20T21:30:00.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.938+0000 7fe49ea00700 1 -- 172.21.15.62:0/1284059468 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe498076900 msgr2=0x7fe498076d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:30:00.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.938+0000 7fe49ea00700 1 --2- 172.21.15.62:0/1284059468 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe498076900 0x7fe498076d70 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fe484009910 tx=0x7fe484009c20 comp rx=0 tx=0).stop 2024-04-20T21:30:00.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.938+0000 7fe49ea00700 1 -- 172.21.15.62:0/1284059468 shutdown_connections 2024-04-20T21:30:00.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.938+0000 7fe49ea00700 1 --2- 172.21.15.62:0/1284059468 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe498076900 0x7fe498076d70 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:30:00.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.938+0000 7fe49ea00700 1 --2- 172.21.15.62:0/1284059468 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe498075700 0x7fe498075b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:30:00.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.938+0000 7fe49ea00700 1 -- 172.21.15.62:0/1284059468 >> 172.21.15.62:0/1284059468 conn(0x7fe4980fda60 msgr2=0x7fe4980ffe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:30:00.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.939+0000 7fe49ea00700 1 -- 172.21.15.62:0/1284059468 shutdown_connections 2024-04-20T21:30:00.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.939+0000 7fe49ea00700 1 -- 172.21.15.62:0/1284059468 wait complete. 2024-04-20T21:30:00.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.940+0000 7fe49ea00700 1 Processor -- start 2024-04-20T21:30:00.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.940+0000 7fe49ea00700 1 -- start start 2024-04-20T21:30:00.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.941+0000 7fe49ea00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe498075700 0x7fe49819c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:30:00.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.941+0000 7fe49ea00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe498076900 0x7fe49819c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:30:00.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.941+0000 7fe49ea00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe49819cc00 con 0x7fe498075700 2024-04-20T21:30:00.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.941+0000 7fe49ea00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe49819cd40 con 0x7fe498076900 2024-04-20T21:30:00.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.941+0000 7fe497e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe498075700 0x7fe49819c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:30:00.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.941+0000 7fe497e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe498075700 0x7fe49819c0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55708/0 (socket says 172.21.15.62:55708) 2024-04-20T21:30:00.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.941+0000 7fe497e00700 1 -- 172.21.15.62:0/1078232278 learned_addr learned my addr 172.21.15.62:0/1078232278 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:30:00.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.941+0000 7fe497400700 1 --2- 172.21.15.62:0/1078232278 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe498076900 0x7fe49819c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:30:00.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.942+0000 7fe497e00700 1 -- 172.21.15.62:0/1078232278 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe498076900 msgr2=0x7fe49819c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:30:00.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.942+0000 7fe497e00700 1 --2- 172.21.15.62:0/1078232278 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe498076900 0x7fe49819c5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:30:00.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.942+0000 7fe497e00700 1 -- 172.21.15.62:0/1078232278 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4840095f0 con 0x7fe498075700 2024-04-20T21:30:00.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.942+0000 7fe497e00700 1 --2- 172.21.15.62:0/1078232278 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe498075700 0x7fe49819c0a0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fe48800d8d0 tx=0x7fe48800dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:30:00.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.943+0000 7fe494c00700 1 -- 172.21.15.62:0/1078232278 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe4880098a0 con 0x7fe498075700 2024-04-20T21:30:00.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.943+0000 7fe494c00700 1 -- 172.21.15.62:0/1078232278 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fe488010460 con 0x7fe498075700 2024-04-20T21:30:00.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.943+0000 7fe49ea00700 1 -- 172.21.15.62:0/1078232278 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4981a17f0 con 0x7fe498075700 2024-04-20T21:30:00.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.943+0000 7fe494c00700 1 -- 172.21.15.62:0/1078232278 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe48800f5f0 con 0x7fe498075700 2024-04-20T21:30:00.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.943+0000 7fe49ea00700 1 -- 172.21.15.62:0/1078232278 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4981a1d40 con 0x7fe498075700 2024-04-20T21:30:00.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.944+0000 7fe49ea00700 1 -- 172.21.15.62:0/1078232278 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe49810f380 con 0x7fe498075700 2024-04-20T21:30:00.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.946+0000 7fe494c00700 1 -- 172.21.15.62:0/1078232278 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7fe488009a00 con 0x7fe498075700 2024-04-20T21:30:00.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.947+0000 7fe494c00700 1 --2- 172.21.15.62:0/1078232278 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fe47c06c680 0x7fe47c06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:30:00.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.947+0000 7fe494c00700 1 -- 172.21.15.62:0/1078232278 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5468+0+0 (secure 0 0 0) 0x7fe48808b730 con 0x7fe498075700 2024-04-20T21:30:00.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.947+0000 7fe497400700 1 --2- 172.21.15.62:0/1078232278 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fe47c06c680 0x7fe47c06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:30:00.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.948+0000 7fe497400700 1 --2- 172.21.15.62:0/1078232278 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fe47c06c680 0x7fe47c06eb30 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fe484006010 tx=0x7fe48400b560 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:30:00.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:00.952+0000 7fe494c00700 1 -- 172.21.15.62:0/1078232278 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fe488059880 con 0x7fe498075700 2024-04-20T21:30:01.019 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:01 smithi062 ceph-mon[16221]: Health detail: HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-04-20T21:30:01.019 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:01 smithi062 ceph-mon[16221]: [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T21:30:01.019 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:01 smithi062 ceph-mon[16221]: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T21:30:01.019 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:01 smithi062 ceph-mon[16221]: from='client.14570 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:30:01.019 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:01 smithi062 ceph-mon[16221]: from='client.14574 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:30:01.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:01.296+0000 7fe49ea00700 1 -- 172.21.15.62:0/1078232278 --> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fe4981a2440 con 0x7fe47c06c680 2024-04-20T21:30:01.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:01.316+0000 7fe494c00700 1 -- 172.21.15.62:0/1078232278 <== mgr.14215 v2:172.21.15.62:6800/4116451241 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3480 (secure 0 0 0) 0x7fe4981a2440 con 0x7fe47c06c680 2024-04-20T21:30:01.316 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T21:30:01.316 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (7m) 2m ago 9m 13.1M - 0.25.0 c8568f914cd2 957f663a9cd7 2024-04-20T21:30:01.316 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (10m) 2m ago 10m 7583k - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T21:30:01.316 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (8m) 2m ago 8m 7377k - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T21:30:01.317 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (10m) 2m ago 10m 6656k - 18.2.0 dc2bc1663786 36e10ef5be39 2024-04-20T21:30:01.317 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (8m) 2m ago 8m 6665k - 18.2.0 dc2bc1663786 4101f9953e3d 2024-04-20T21:30:01.317 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (7m) 2m ago 9m 74.1M - 9.4.7 954c08fa6188 99363bb095d8 2024-04-20T21:30:01.317 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (2m) 2m ago 2m 13.6M - 18.2.0 dc2bc1663786 5e6460aa6159 2024-04-20T21:30:01.317 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (2m) 2m ago 2m 14.9M - 18.2.0 dc2bc1663786 e65f2267544e 2024-04-20T21:30:01.317 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (2m) 2m ago 2m 14.4M - 18.2.0 dc2bc1663786 fb2a8a223db8 2024-04-20T21:30:01.317 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (2m) 2m ago 2m 13.0M - 18.2.0 dc2bc1663786 f2eec2ee8951 2024-04-20T21:30:01.317 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:9283,8765,8443 running (11m) 2m ago 11m 488M - 18.2.0 dc2bc1663786 e411b3dc7233 2024-04-20T21:30:01.317 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (8m) 2m ago 8m 437M - 18.2.0 dc2bc1663786 a31cf3b36efc 2024-04-20T21:30:01.317 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (11m) 2m ago 11m 43.9M 2048M 18.2.0 dc2bc1663786 bd6efbd5c380 2024-04-20T21:30:01.317 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (8m) 2m ago 8m 36.1M 2048M 18.2.0 dc2bc1663786 3735c8e5db9a 2024-04-20T21:30:01.317 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (9m) 2m ago 9m 7738k - 1.5.0 0da6a335fe13 fc746d8ae72e 2024-04-20T21:30:01.317 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (8m) 2m ago 8m 7734k - 1.5.0 0da6a335fe13 db03af85b286 2024-04-20T21:30:01.317 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (7m) 2m ago 7m 46.1M 3947M 18.2.0 dc2bc1663786 3f315c3afc1f 2024-04-20T21:30:01.318 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (6m) 2m ago 6m 49.0M 3947M 18.2.0 dc2bc1663786 44e4d07416ad 2024-04-20T21:30:01.318 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (6m) 2m ago 6m 46.5M 3947M 18.2.0 dc2bc1663786 31c5d9586e98 2024-04-20T21:30:01.318 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (5m) 2m ago 5m 45.2M 4971M 18.2.0 dc2bc1663786 611c1e242693 2024-04-20T21:30:01.318 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (5m) 2m ago 5m 42.8M 4971M 18.2.0 dc2bc1663786 6c8fce91d45d 2024-04-20T21:30:01.318 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (4m) 2m ago 4m 42.4M 4971M 18.2.0 dc2bc1663786 8e4574bae8db 2024-04-20T21:30:01.318 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (7m) 2m ago 9m 51.5M - 2.43.0 a07b618ecd1d cd390b7e44d0 2024-04-20T21:30:01.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:01.319+0000 7fe49ea00700 1 -- 172.21.15.62:0/1078232278 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fe47c06c680 msgr2=0x7fe47c06eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:30:01.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:01.319+0000 7fe49ea00700 1 --2- 172.21.15.62:0/1078232278 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fe47c06c680 0x7fe47c06eb30 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fe484006010 tx=0x7fe48400b560 comp rx=0 tx=0).stop 2024-04-20T21:30:01.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:01.320+0000 7fe49ea00700 1 -- 172.21.15.62:0/1078232278 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe498075700 msgr2=0x7fe49819c0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:30:01.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:01.320+0000 7fe49ea00700 1 --2- 172.21.15.62:0/1078232278 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe498075700 0x7fe49819c0a0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fe48800d8d0 tx=0x7fe48800dbe0 comp rx=0 tx=0).stop 2024-04-20T21:30:01.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:01.320+0000 7fe49ea00700 1 -- 172.21.15.62:0/1078232278 shutdown_connections 2024-04-20T21:30:01.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:01.320+0000 7fe49ea00700 1 --2- 172.21.15.62:0/1078232278 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fe47c06c680 0x7fe47c06eb30 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:30:01.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:01.320+0000 7fe49ea00700 1 --2- 172.21.15.62:0/1078232278 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe498075700 0x7fe49819c0a0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:30:01.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:01.320+0000 7fe49ea00700 1 --2- 172.21.15.62:0/1078232278 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe498076900 0x7fe49819c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:30:01.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:01.320+0000 7fe49ea00700 1 -- 172.21.15.62:0/1078232278 >> 172.21.15.62:0/1078232278 conn(0x7fe4980fda60 msgr2=0x7fe49810c6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:30:01.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:01.320+0000 7fe49ea00700 1 -- 172.21.15.62:0/1078232278 shutdown_connections 2024-04-20T21:30:01.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:01.320+0000 7fe49ea00700 1 -- 172.21.15.62:0/1078232278 wait complete. 2024-04-20T21:30:01.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:01 smithi189 ceph-mon[21562]: Health detail: HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-04-20T21:30:01.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:01 smithi189 ceph-mon[21562]: [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T21:30:01.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:01 smithi189 ceph-mon[21562]: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T21:30:01.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:01 smithi189 ceph-mon[21562]: from='client.14570 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:30:01.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:01 smithi189 ceph-mon[21562]: from='client.14574 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:30:01.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:01.495+0000 7fc06b800700 1 -- 172.21.15.62:0/4134626026 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc0641003d0 msgr2=0x7fc064100840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:30:01.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:01.495+0000 7fc06b800700 1 --2- 172.21.15.62:0/4134626026 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc0641003d0 0x7fc064100840 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fc050009910 tx=0x7fc050009c20 comp rx=0 tx=0).stop 2024-04-20T21:30:01.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:01.496+0000 7fc06b800700 1 -- 172.21.15.62:0/4134626026 shutdown_connections 2024-04-20T21:30:01.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:01.496+0000 7fc06b800700 1 --2- 172.21.15.62:0/4134626026 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc0641003d0 0x7fc064100840 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:30:01.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:01.496+0000 7fc06b800700 1 --2- 172.21.15.62:0/4134626026 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc0640ff130 0x7fc0640ff540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:30:01.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:01.496+0000 7fc06b800700 1 -- 172.21.15.62:0/4134626026 >> 172.21.15.62:0/4134626026 conn(0x7fc0640fa780 msgr2=0x7fc0640fcbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:30:01.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:01.496+0000 7fc06b800700 1 -- 172.21.15.62:0/4134626026 shutdown_connections 2024-04-20T21:30:01.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:01.496+0000 7fc06b800700 1 -- 172.21.15.62:0/4134626026 wait complete. 2024-04-20T21:30:01.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:01.497+0000 7fc06b800700 1 Processor -- start 2024-04-20T21:30:01.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:01.497+0000 7fc06b800700 1 -- start start 2024-04-20T21:30:01.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:01.498+0000 7fc06b800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc0640ff130 0x7fc064103160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:30:01.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:01.498+0000 7fc06b800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc0641003d0 0x7fc0641017b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:30:01.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:01.498+0000 7fc06b800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc064101cf0 con 0x7fc0640ff130 2024-04-20T21:30:01.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:01.498+0000 7fc06b800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc064101e30 con 0x7fc0641003d0 2024-04-20T21:30:01.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:01.498+0000 7fc068e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc0640ff130 0x7fc064103160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:30:01.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:01.498+0000 7fc068e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc0640ff130 0x7fc064103160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50956/0 (socket says 172.21.15.62:50956) 2024-04-20T21:30:01.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:01.498+0000 7fc068e00700 1 -- 172.21.15.62:0/3198691287 learned_addr learned my addr 172.21.15.62:0/3198691287 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:30:01.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:01.498+0000 7fc063e00700 1 --2- 172.21.15.62:0/3198691287 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc0641003d0 0x7fc0641017b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:30:01.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:01.499+0000 7fc068e00700 1 -- 172.21.15.62:0/3198691287 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc0641003d0 msgr2=0x7fc0641017b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:30:01.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:01.499+0000 7fc068e00700 1 --2- 172.21.15.62:0/3198691287 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc0641003d0 0x7fc0641017b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:30:01.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:01.499+0000 7fc068e00700 1 -- 172.21.15.62:0/3198691287 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0500095f0 con 0x7fc0640ff130 2024-04-20T21:30:01.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:01.499+0000 7fc068e00700 1 --2- 172.21.15.62:0/3198691287 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc0640ff130 0x7fc064103160 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fc05800e920 tx=0x7fc05800ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:30:01.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:01.500+0000 7fc061600700 1 -- 172.21.15.62:0/3198691287 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc05800c9d0 con 0x7fc0640ff130 2024-04-20T21:30:01.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:01.500+0000 7fc061600700 1 -- 172.21.15.62:0/3198691287 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc058010430 con 0x7fc0640ff130 2024-04-20T21:30:01.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:01.500+0000 7fc061600700 1 -- 172.21.15.62:0/3198691287 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc058018650 con 0x7fc0640ff130 2024-04-20T21:30:01.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:01.500+0000 7fc06b800700 1 -- 172.21.15.62:0/3198691287 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc064102110 con 0x7fc0640ff130 2024-04-20T21:30:01.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:01.500+0000 7fc06b800700 1 -- 172.21.15.62:0/3198691287 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc06419dac0 con 0x7fc0640ff130 2024-04-20T21:30:01.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:01.501+0000 7fc06b800700 1 -- 172.21.15.62:0/3198691287 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc064107c10 con 0x7fc0640ff130 2024-04-20T21:30:01.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:01.503+0000 7fc061600700 1 -- 172.21.15.62:0/3198691287 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7fc0580105a0 con 0x7fc0640ff130 2024-04-20T21:30:01.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:01.504+0000 7fc061600700 1 --2- 172.21.15.62:0/3198691287 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fc04c06c680 0x7fc04c06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:30:01.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:01.504+0000 7fc061600700 1 -- 172.21.15.62:0/3198691287 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5468+0+0 (secure 0 0 0) 0x7fc058014070 con 0x7fc0640ff130 2024-04-20T21:30:01.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:01.504+0000 7fc063e00700 1 --2- 172.21.15.62:0/3198691287 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fc04c06c680 0x7fc04c06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:30:01.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:01.505+0000 7fc063e00700 1 --2- 172.21.15.62:0/3198691287 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fc04c06c680 0x7fc04c06eb30 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fc050009fd0 tx=0x7fc050005fd0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:30:01.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:01.510+0000 7fc061600700 1 -- 172.21.15.62:0/3198691287 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fc058051720 con 0x7fc0640ff130 2024-04-20T21:30:01.667 INFO:tasks.workunit.client.1.smithi189.stderr:+ tar xzf ltp-full.tgz 2024-04-20T21:30:01.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:01.885+0000 7fc06b800700 1 -- 172.21.15.62:0/3198691287 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fc06419e070 con 0x7fc0640ff130 2024-04-20T21:30:01.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:01.887+0000 7fc061600700 1 -- 172.21.15.62:0/3198691287 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+558 (secure 0 0 0) 0x7fc058059de0 con 0x7fc0640ff130 2024-04-20T21:30:01.886 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:30:01.886 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T21:30:01.886 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 2 2024-04-20T21:30:01.886 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:30:01.886 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T21:30:01.886 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 2 2024-04-20T21:30:01.887 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:30:01.887 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T21:30:01.887 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 6 2024-04-20T21:30:01.887 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:30:01.887 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T21:30:01.887 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 4 2024-04-20T21:30:01.887 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:30:01.887 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T21:30:01.887 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 14 2024-04-20T21:30:01.887 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T21:30:01.887 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:30:01.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:01.890+0000 7fc06b800700 1 -- 172.21.15.62:0/3198691287 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fc04c06c680 msgr2=0x7fc04c06eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:30:01.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:01.890+0000 7fc06b800700 1 --2- 172.21.15.62:0/3198691287 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fc04c06c680 0x7fc04c06eb30 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fc050009fd0 tx=0x7fc050005fd0 comp rx=0 tx=0).stop 2024-04-20T21:30:01.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:01.890+0000 7fc06b800700 1 -- 172.21.15.62:0/3198691287 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc0640ff130 msgr2=0x7fc064103160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:30:01.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:01.890+0000 7fc06b800700 1 --2- 172.21.15.62:0/3198691287 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc0640ff130 0x7fc064103160 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fc05800e920 tx=0x7fc05800ec30 comp rx=0 tx=0).stop 2024-04-20T21:30:01.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:01.890+0000 7fc06b800700 1 -- 172.21.15.62:0/3198691287 shutdown_connections 2024-04-20T21:30:01.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:01.890+0000 7fc06b800700 1 --2- 172.21.15.62:0/3198691287 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fc04c06c680 0x7fc04c06eb30 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:30:01.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:01.890+0000 7fc06b800700 1 --2- 172.21.15.62:0/3198691287 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc0640ff130 0x7fc064103160 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:30:01.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:01.890+0000 7fc06b800700 1 --2- 172.21.15.62:0/3198691287 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc0641003d0 0x7fc0641017b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:30:01.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:01.890+0000 7fc06b800700 1 -- 172.21.15.62:0/3198691287 >> 172.21.15.62:0/3198691287 conn(0x7fc0640fa780 msgr2=0x7fc064068540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:30:01.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:01.891+0000 7fc06b800700 1 -- 172.21.15.62:0/3198691287 shutdown_connections 2024-04-20T21:30:01.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:01.891+0000 7fc06b800700 1 -- 172.21.15.62:0/3198691287 wait complete. 2024-04-20T21:30:02.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:02.075+0000 7fbc7ee00700 1 -- 172.21.15.62:0/636752492 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbc78100d60 msgr2=0x7fbc78103140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:30:02.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:02.075+0000 7fbc7ee00700 1 --2- 172.21.15.62:0/636752492 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbc78100d60 0x7fbc78103140 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fbc68009910 tx=0x7fbc68009c20 comp rx=0 tx=0).stop 2024-04-20T21:30:02.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:02.075+0000 7fbc7ee00700 1 -- 172.21.15.62:0/636752492 shutdown_connections 2024-04-20T21:30:02.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:02.075+0000 7fbc7ee00700 1 --2- 172.21.15.62:0/636752492 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc78103680 0x7fbc78105a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:30:02.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:02.075+0000 7fbc7ee00700 1 --2- 172.21.15.62:0/636752492 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbc78100d60 0x7fbc78103140 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:30:02.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:02.075+0000 7fbc7ee00700 1 -- 172.21.15.62:0/636752492 >> 172.21.15.62:0/636752492 conn(0x7fbc780fa780 msgr2=0x7fbc780fcbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:30:02.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:02.075+0000 7fbc7ee00700 1 -- 172.21.15.62:0/636752492 shutdown_connections 2024-04-20T21:30:02.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:02.075+0000 7fbc7ee00700 1 -- 172.21.15.62:0/636752492 wait complete. 2024-04-20T21:30:02.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:02.076+0000 7fbc7ee00700 1 Processor -- start 2024-04-20T21:30:02.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:02.076+0000 7fbc7ee00700 1 -- start start 2024-04-20T21:30:02.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:02.077+0000 7fbc7ee00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc78100d60 0x7fbc781006c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:30:02.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:02.077+0000 7fbc7ee00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbc78103680 0x7fbc780fed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:30:02.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:02.077+0000 7fbc7ee00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc780ff250 con 0x7fbc78103680 2024-04-20T21:30:02.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:02.077+0000 7fbc7ee00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc780ff390 con 0x7fbc78100d60 2024-04-20T21:30:02.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:02.077+0000 7fbc6fe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbc78103680 0x7fbc780fed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:30:02.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:02.077+0000 7fbc6fe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbc78103680 0x7fbc780fed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50970/0 (socket says 172.21.15.62:50970) 2024-04-20T21:30:02.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:02.077+0000 7fbc6fe00700 1 -- 172.21.15.62:0/162569713 learned_addr learned my addr 172.21.15.62:0/162569713 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:30:02.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:02.077+0000 7fbc77e00700 1 --2- 172.21.15.62:0/162569713 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc78100d60 0x7fbc781006c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:30:02.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:02.078+0000 7fbc6fe00700 1 -- 172.21.15.62:0/162569713 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc78100d60 msgr2=0x7fbc781006c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:30:02.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:02.078+0000 7fbc6fe00700 1 --2- 172.21.15.62:0/162569713 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc78100d60 0x7fbc781006c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:30:02.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:02.078+0000 7fbc6fe00700 1 -- 172.21.15.62:0/162569713 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc680095f0 con 0x7fbc78103680 2024-04-20T21:30:02.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:02.078+0000 7fbc6fe00700 1 --2- 172.21.15.62:0/162569713 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbc78103680 0x7fbc780fed10 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fbc6000c930 tx=0x7fbc6000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:30:02.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:02.079+0000 7fbc75600700 1 -- 172.21.15.62:0/162569713 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbc60007ad0 con 0x7fbc78103680 2024-04-20T21:30:02.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:02.079+0000 7fbc75600700 1 -- 172.21.15.62:0/162569713 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fbc6000ce80 con 0x7fbc78103680 2024-04-20T21:30:02.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:02.079+0000 7fbc7ee00700 1 -- 172.21.15.62:0/162569713 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc780ff670 con 0x7fbc78103680 2024-04-20T21:30:02.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:02.079+0000 7fbc75600700 1 -- 172.21.15.62:0/162569713 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbc60018630 con 0x7fbc78103680 2024-04-20T21:30:02.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:02.079+0000 7fbc7ee00700 1 -- 172.21.15.62:0/162569713 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc7819db10 con 0x7fbc78103680 2024-04-20T21:30:02.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:02.081+0000 7fbc7ee00700 1 -- 172.21.15.62:0/162569713 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc7818fbc0 con 0x7fbc78103680 2024-04-20T21:30:02.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:02.082+0000 7fbc75600700 1 -- 172.21.15.62:0/162569713 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 19) v1 ==== 89914+0+0 (secure 0 0 0) 0x7fbc60007c30 con 0x7fbc78103680 2024-04-20T21:30:02.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:02.083+0000 7fbc75600700 1 --2- 172.21.15.62:0/162569713 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fbc5806c680 0x7fbc5806eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:30:02.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:02.083+0000 7fbc75600700 1 -- 172.21.15.62:0/162569713 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5468+0+0 (secure 0 0 0) 0x7fbc6008b7a0 con 0x7fbc78103680 2024-04-20T21:30:02.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:02.084+0000 7fbc77e00700 1 --2- 172.21.15.62:0/162569713 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fbc5806c680 0x7fbc5806eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:30:02.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:02.084+0000 7fbc77e00700 1 --2- 172.21.15.62:0/162569713 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fbc5806c680 0x7fbc5806eb30 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fbc68006010 tx=0x7fbc6800b540 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:30:02.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:02.089+0000 7fbc75600700 1 -- 172.21.15.62:0/162569713 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fbc60059990 con 0x7fbc78103680 2024-04-20T21:30:02.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:02.379+0000 7fbc7ee00700 1 -- 172.21.15.62:0/162569713 --> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fbc78061190 con 0x7fbc5806c680 2024-04-20T21:30:02.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:02.382+0000 7fbc75600700 1 -- 172.21.15.62:0/162569713 <== mgr.14215 v2:172.21.15.62:6800/4116451241 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7fbc78061190 con 0x7fbc5806c680 2024-04-20T21:30:02.381 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:30:02.381 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T21:30:02.381 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T21:30:02.381 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-20T21:30:02.381 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [], 2024-04-20T21:30:02.381 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "0/2 daemons upgraded", 2024-04-20T21:30:02.382 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 image on host smithi189", 2024-04-20T21:30:02.382 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T21:30:02.382 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:30:02.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:02.385+0000 7fbc7ee00700 1 -- 172.21.15.62:0/162569713 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fbc5806c680 msgr2=0x7fbc5806eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:30:02.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:02.385+0000 7fbc7ee00700 1 --2- 172.21.15.62:0/162569713 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fbc5806c680 0x7fbc5806eb30 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fbc68006010 tx=0x7fbc6800b540 comp rx=0 tx=0).stop 2024-04-20T21:30:02.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:02.385+0000 7fbc7ee00700 1 -- 172.21.15.62:0/162569713 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbc78103680 msgr2=0x7fbc780fed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:30:02.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:02.385+0000 7fbc7ee00700 1 --2- 172.21.15.62:0/162569713 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbc78103680 0x7fbc780fed10 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fbc6000c930 tx=0x7fbc6000cc40 comp rx=0 tx=0).stop 2024-04-20T21:30:02.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:02.386+0000 7fbc7ee00700 1 -- 172.21.15.62:0/162569713 shutdown_connections 2024-04-20T21:30:02.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:02.386+0000 7fbc7ee00700 1 --2- 172.21.15.62:0/162569713 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fbc5806c680 0x7fbc5806eb30 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:30:02.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:02.386+0000 7fbc7ee00700 1 --2- 172.21.15.62:0/162569713 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc78100d60 0x7fbc781006c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:30:02.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:02.386+0000 7fbc7ee00700 1 --2- 172.21.15.62:0/162569713 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbc78103680 0x7fbc780fed10 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:30:02.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:02.386+0000 7fbc7ee00700 1 -- 172.21.15.62:0/162569713 >> 172.21.15.62:0/162569713 conn(0x7fbc780fa780 msgr2=0x7fbc780fcbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:30:02.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:02.386+0000 7fbc7ee00700 1 -- 172.21.15.62:0/162569713 shutdown_connections 2024-04-20T21:30:02.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:02.386+0000 7fbc7ee00700 1 -- 172.21.15.62:0/162569713 wait complete. 2024-04-20T21:30:03.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:02 smithi189 ceph-mon[21562]: from='client.14578 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:30:03.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:02 smithi189 ceph-mon[21562]: from='client.? 172.21.15.62:0/3198691287' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:30:03.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:02 smithi189 ceph-mon[21562]: pgmap v302: 65 pgs: 65 active+clean; 11 MiB data, 183 MiB used, 536 GiB / 536 GiB avail; 4.0 KiB/s rd, 917 KiB/s wr, 9 op/s 2024-04-20T21:30:03.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:02 smithi062 ceph-mon[16221]: from='client.14578 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:30:03.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:02 smithi062 ceph-mon[16221]: from='client.? 172.21.15.62:0/3198691287' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:30:03.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:02 smithi062 ceph-mon[16221]: pgmap v302: 65 pgs: 65 active+clean; 11 MiB data, 183 MiB used, 536 GiB / 536 GiB avail; 4.0 KiB/s rd, 917 KiB/s wr, 9 op/s 2024-04-20T21:30:04.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:03 smithi189 ceph-mon[21562]: from='client.14586 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:30:04.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:03 smithi062 ceph-mon[16221]: from='client.14586 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:30:05.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:04 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:30:05.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:04 smithi189 ceph-mon[21562]: pgmap v303: 65 pgs: 65 active+clean; 19 MiB data, 221 MiB used, 536 GiB / 536 GiB avail; 3.2 KiB/s rd, 1.5 MiB/s wr, 21 op/s 2024-04-20T21:30:05.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:04 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:30:05.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:04 smithi062 ceph-mon[16221]: pgmap v303: 65 pgs: 65 active+clean; 19 MiB data, 221 MiB used, 536 GiB / 536 GiB avail; 3.2 KiB/s rd, 1.5 MiB/s wr, 21 op/s 2024-04-20T21:30:06.284 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:06 smithi062 ceph-mon[16221]: pgmap v304: 65 pgs: 65 active+clean; 25 MiB data, 234 MiB used, 536 GiB / 536 GiB avail; 98 KiB/s rd, 2.0 MiB/s wr, 36 op/s 2024-04-20T21:30:06.486 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:06 smithi189 ceph-mon[21562]: pgmap v304: 65 pgs: 65 active+clean; 25 MiB data, 234 MiB used, 536 GiB / 536 GiB avail; 98 KiB/s rd, 2.0 MiB/s wr, 36 op/s 2024-04-20T21:30:08.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:08 smithi062 ceph-mon[16221]: pgmap v305: 65 pgs: 65 active+clean; 25 MiB data, 239 MiB used, 536 GiB / 536 GiB avail; 97 KiB/s rd, 2.0 MiB/s wr, 37 op/s 2024-04-20T21:30:08.479 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:08 smithi189 ceph-mon[21562]: pgmap v305: 65 pgs: 65 active+clean; 25 MiB data, 239 MiB used, 536 GiB / 536 GiB avail; 97 KiB/s rd, 2.0 MiB/s wr, 37 op/s 2024-04-20T21:30:10.174 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:10 smithi189 ceph-mon[21562]: Upgrade: Updating mgr.smithi189.tawavz 2024-04-20T21:30:10.174 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:10 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:30:10.174 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:10 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi189.tawavz", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-20T21:30:10.174 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:10 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-20T21:30:10.174 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:10 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:30:10.174 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:10 smithi189 ceph-mon[21562]: Deploying daemon mgr.smithi189.tawavz on smithi189 2024-04-20T21:30:10.174 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:10 smithi189 ceph-mon[21562]: pgmap v306: 65 pgs: 65 active+clean; 26 MiB data, 246 MiB used, 536 GiB / 536 GiB avail; 97 KiB/s rd, 2.1 MiB/s wr, 61 op/s 2024-04-20T21:30:10.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:10 smithi062 ceph-mon[16221]: Upgrade: Updating mgr.smithi189.tawavz 2024-04-20T21:30:10.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:10 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:30:10.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:10 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi189.tawavz", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-20T21:30:10.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:10 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-20T21:30:10.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:10 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:30:10.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:10 smithi062 ceph-mon[16221]: Deploying daemon mgr.smithi189.tawavz on smithi189 2024-04-20T21:30:10.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:10 smithi062 ceph-mon[16221]: pgmap v306: 65 pgs: 65 active+clean; 26 MiB data, 246 MiB used, 536 GiB / 536 GiB avail; 97 KiB/s rd, 2.1 MiB/s wr, 61 op/s 2024-04-20T21:30:12.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:12 smithi189 ceph-mon[21562]: fsmap cephfs:2 {0=cephfs.smithi062.ryqblb=up:active,1=cephfs.smithi189.dvimer=up:active} 2 up:standby-replay 2024-04-20T21:30:12.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:12 smithi189 ceph-mon[21562]: pgmap v307: 65 pgs: 65 active+clean; 29 MiB data, 250 MiB used, 536 GiB / 536 GiB avail; 343 KiB/s rd, 1.7 MiB/s wr, 66 op/s 2024-04-20T21:30:12.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:12 smithi062 ceph-mon[16221]: fsmap cephfs:2 {0=cephfs.smithi062.ryqblb=up:active,1=cephfs.smithi189.dvimer=up:active} 2 up:standby-replay 2024-04-20T21:30:12.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:12 smithi062 ceph-mon[16221]: pgmap v307: 65 pgs: 65 active+clean; 29 MiB data, 250 MiB used, 536 GiB / 536 GiB avail; 343 KiB/s rd, 1.7 MiB/s wr, 66 op/s 2024-04-20T21:30:14.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:14 smithi062 ceph-mon[16221]: pgmap v308: 65 pgs: 65 active+clean; 34 MiB data, 261 MiB used, 536 GiB / 536 GiB avail; 683 KiB/s rd, 1.9 MiB/s wr, 78 op/s 2024-04-20T21:30:14.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:14 smithi189 ceph-mon[21562]: pgmap v308: 65 pgs: 65 active+clean; 34 MiB data, 261 MiB used, 536 GiB / 536 GiB avail; 683 KiB/s rd, 1.9 MiB/s wr, 78 op/s 2024-04-20T21:30:16.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:16 smithi062 ceph-mon[16221]: pgmap v309: 65 pgs: 65 active+clean; 36 MiB data, 269 MiB used, 536 GiB / 536 GiB avail; 783 KiB/s rd, 1.4 MiB/s wr, 77 op/s 2024-04-20T21:30:16.470 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:16 smithi189 ceph-mon[21562]: pgmap v309: 65 pgs: 65 active+clean; 36 MiB data, 269 MiB used, 536 GiB / 536 GiB avail; 783 KiB/s rd, 1.4 MiB/s wr, 77 op/s 2024-04-20T21:30:18.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:18 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:30:18.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:18 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:30:18.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:18 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:30:18.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:18 smithi062 ceph-mon[16221]: pgmap v310: 65 pgs: 65 active+clean; 36 MiB data, 278 MiB used, 536 GiB / 536 GiB avail; 688 KiB/s rd, 981 KiB/s wr, 64 op/s 2024-04-20T21:30:18.410 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:18 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:30:18.411 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:18 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:30:18.411 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:18 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:30:18.411 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:18 smithi189 ceph-mon[21562]: pgmap v310: 65 pgs: 65 active+clean; 36 MiB data, 278 MiB used, 536 GiB / 536 GiB avail; 688 KiB/s rd, 981 KiB/s wr, 64 op/s 2024-04-20T21:30:19.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:19 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:30:19.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:19 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:30:20.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:20 smithi189 ceph-mon[21562]: pgmap v311: 65 pgs: 65 active+clean; 37 MiB data, 288 MiB used, 536 GiB / 536 GiB avail; 811 KiB/s rd, 1.1 MiB/s wr, 78 op/s 2024-04-20T21:30:20.534 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:20 smithi062 ceph-mon[16221]: pgmap v311: 65 pgs: 65 active+clean; 37 MiB data, 288 MiB used, 536 GiB / 536 GiB avail; 811 KiB/s rd, 1.1 MiB/s wr, 78 op/s 2024-04-20T21:30:23.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:22 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:30:23.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:22 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:30:23.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:22 smithi189 ceph-mon[21562]: pgmap v312: 65 pgs: 65 active+clean; 40 MiB data, 300 MiB used, 536 GiB / 536 GiB avail; 1.0 MiB/s rd, 1.2 MiB/s wr, 60 op/s 2024-04-20T21:30:23.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:22 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:30:23.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:22 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:30:23.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:22 smithi062 ceph-mon[16221]: pgmap v312: 65 pgs: 65 active+clean; 40 MiB data, 300 MiB used, 536 GiB / 536 GiB avail; 1.0 MiB/s rd, 1.2 MiB/s wr, 60 op/s 2024-04-20T21:30:24.925 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:24 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:30:24.926 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:24 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:30:24.926 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:24 smithi189 ceph-mon[21562]: pgmap v313: 65 pgs: 65 active+clean; 48 MiB data, 322 MiB used, 536 GiB / 536 GiB avail; 1.4 MiB/s rd, 1.6 MiB/s wr, 71 op/s 2024-04-20T21:30:25.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:24 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:30:25.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:24 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:30:25.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:24 smithi062 ceph-mon[16221]: pgmap v313: 65 pgs: 65 active+clean; 48 MiB data, 322 MiB used, 536 GiB / 536 GiB avail; 1.4 MiB/s rd, 1.6 MiB/s wr, 71 op/s 2024-04-20T21:30:26.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:26 smithi189 ceph-mon[21562]: pgmap v314: 65 pgs: 65 active+clean; 53 MiB data, 336 MiB used, 536 GiB / 536 GiB avail; 1.4 MiB/s rd, 1.6 MiB/s wr, 76 op/s 2024-04-20T21:30:26.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:26 smithi062 ceph-mon[16221]: pgmap v314: 65 pgs: 65 active+clean; 53 MiB data, 336 MiB used, 536 GiB / 536 GiB avail; 1.4 MiB/s rd, 1.6 MiB/s wr, 76 op/s 2024-04-20T21:30:28.142 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:27 smithi062 ceph-mon[16221]: Standby manager daemon smithi189.tawavz restarted 2024-04-20T21:30:28.143 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:27 smithi062 ceph-mon[16221]: Standby manager daemon smithi189.tawavz started 2024-04-20T21:30:28.143 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:27 smithi062 ceph-mon[16221]: from='mgr.? 172.21.15.189:0/1768059451' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi189.tawavz/crt"}]: dispatch 2024-04-20T21:30:28.143 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:27 smithi062 ceph-mon[16221]: from='mgr.? 172.21.15.189:0/1768059451' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-20T21:30:28.143 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:27 smithi062 ceph-mon[16221]: from='mgr.? 172.21.15.189:0/1768059451' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi189.tawavz/key"}]: dispatch 2024-04-20T21:30:28.143 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:27 smithi062 ceph-mon[16221]: from='mgr.? 172.21.15.189:0/1768059451' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-20T21:30:28.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:27 smithi189 ceph-mon[21562]: Standby manager daemon smithi189.tawavz restarted 2024-04-20T21:30:28.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:27 smithi189 ceph-mon[21562]: Standby manager daemon smithi189.tawavz started 2024-04-20T21:30:28.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:27 smithi189 ceph-mon[21562]: from='mgr.? 172.21.15.189:0/1768059451' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi189.tawavz/crt"}]: dispatch 2024-04-20T21:30:28.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:27 smithi189 ceph-mon[21562]: from='mgr.? 172.21.15.189:0/1768059451' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-20T21:30:28.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:27 smithi189 ceph-mon[21562]: from='mgr.? 172.21.15.189:0/1768059451' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi189.tawavz/key"}]: dispatch 2024-04-20T21:30:28.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:27 smithi189 ceph-mon[21562]: from='mgr.? 172.21.15.189:0/1768059451' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-20T21:30:29.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:28 smithi189 ceph-mon[21562]: pgmap v315: 65 pgs: 65 active+clean; 53 MiB data, 340 MiB used, 536 GiB / 536 GiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 63 op/s 2024-04-20T21:30:29.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:28 smithi189 ceph-mon[21562]: mgrmap e20: smithi062.rqxuxi(active, since 9m), standbys: smithi189.tawavz 2024-04-20T21:30:29.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:28 smithi062 ceph-mon[16221]: pgmap v315: 65 pgs: 65 active+clean; 53 MiB data, 340 MiB used, 536 GiB / 536 GiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 63 op/s 2024-04-20T21:30:29.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:28 smithi062 ceph-mon[16221]: mgrmap e20: smithi062.rqxuxi(active, since 9m), standbys: smithi189.tawavz 2024-04-20T21:30:30.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:30 smithi189 ceph-mon[21562]: pgmap v316: 65 pgs: 65 active+clean; 58 MiB data, 359 MiB used, 536 GiB / 536 GiB avail; 1.4 MiB/s rd, 1.9 MiB/s wr, 85 op/s 2024-04-20T21:30:30.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:30 smithi062 ceph-mon[16221]: pgmap v316: 65 pgs: 65 active+clean; 58 MiB data, 359 MiB used, 536 GiB / 536 GiB avail; 1.4 MiB/s rd, 1.9 MiB/s wr, 85 op/s 2024-04-20T21:30:32.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:32.585+0000 7f50c9400700 1 -- 172.21.15.62:0/3066264095 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50c4100250 msgr2=0x7f50c4100660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:30:32.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:32.585+0000 7f50c9400700 1 --2- 172.21.15.62:0/3066264095 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50c4100250 0x7f50c4100660 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f50b4009910 tx=0x7f50b4009c20 comp rx=0 tx=0).stop 2024-04-20T21:30:32.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:32.585+0000 7f50c9400700 1 -- 172.21.15.62:0/3066264095 shutdown_connections 2024-04-20T21:30:32.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:32.585+0000 7f50c9400700 1 --2- 172.21.15.62:0/3066264095 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f50c4101450 0x7f50c41018a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:30:32.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:32.585+0000 7f50c9400700 1 --2- 172.21.15.62:0/3066264095 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50c4100250 0x7f50c4100660 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:30:32.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:32.585+0000 7f50c9400700 1 -- 172.21.15.62:0/3066264095 >> 172.21.15.62:0/3066264095 conn(0x7f50c40fb7e0 msgr2=0x7f50c40fdc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:30:32.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:32.585+0000 7f50c9400700 1 -- 172.21.15.62:0/3066264095 shutdown_connections 2024-04-20T21:30:32.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:32.585+0000 7f50c9400700 1 -- 172.21.15.62:0/3066264095 wait complete. 2024-04-20T21:30:32.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:32.587+0000 7f50c9400700 1 Processor -- start 2024-04-20T21:30:32.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:32.587+0000 7f50c9400700 1 -- start start 2024-04-20T21:30:32.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:32.587+0000 7f50c9400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50c4100250 0x7f50c4197c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:30:32.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:32.587+0000 7f50c9400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f50c4101450 0x7f50c41981b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:30:32.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:32.587+0000 7f50c9400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50c41987b0 con 0x7f50c4101450 2024-04-20T21:30:32.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:32.587+0000 7f50c9400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50c4198920 con 0x7f50c4100250 2024-04-20T21:30:32.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:32.588+0000 7f50c1e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f50c4101450 0x7f50c41981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:30:32.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:32.588+0000 7f50c1e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f50c4101450 0x7f50c41981b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34056/0 (socket says 172.21.15.62:34056) 2024-04-20T21:30:32.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:32.588+0000 7f50c1e00700 1 -- 172.21.15.62:0/2369656172 learned_addr learned my addr 172.21.15.62:0/2369656172 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:30:32.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:32.588+0000 7f50c2800700 1 --2- 172.21.15.62:0/2369656172 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50c4100250 0x7f50c4197c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:30:32.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:32.588+0000 7f50c1e00700 1 -- 172.21.15.62:0/2369656172 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50c4100250 msgr2=0x7f50c4197c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:30:32.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:32.588+0000 7f50c1e00700 1 --2- 172.21.15.62:0/2369656172 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50c4100250 0x7f50c4197c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:30:32.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:32.588+0000 7f50c1e00700 1 -- 172.21.15.62:0/2369656172 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50b40095f0 con 0x7f50c4101450 2024-04-20T21:30:32.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:32.588+0000 7f50c2800700 1 --2- 172.21.15.62:0/2369656172 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50c4100250 0x7f50c4197c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-20T21:30:32.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:32.589+0000 7f50c1e00700 1 --2- 172.21.15.62:0/2369656172 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f50c4101450 0x7f50c41981b0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f50b000e920 tx=0x7f50b000ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:30:32.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:32.589+0000 7f50bb400700 1 -- 172.21.15.62:0/2369656172 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f50b000c9d0 con 0x7f50c4101450 2024-04-20T21:30:32.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:32.589+0000 7f50bb400700 1 -- 172.21.15.62:0/2369656172 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f50b0010430 con 0x7f50c4101450 2024-04-20T21:30:32.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:32.589+0000 7f50c9400700 1 -- 172.21.15.62:0/2369656172 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50c419d3b0 con 0x7f50c4101450 2024-04-20T21:30:32.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:32.590+0000 7f50bb400700 1 -- 172.21.15.62:0/2369656172 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f50b0018650 con 0x7f50c4101450 2024-04-20T21:30:32.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:32.590+0000 7f50c9400700 1 -- 172.21.15.62:0/2369656172 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50c419d900 con 0x7f50c4101450 2024-04-20T21:30:32.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:32.591+0000 7f50c9400700 1 -- 172.21.15.62:0/2369656172 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50c4105530 con 0x7f50c4101450 2024-04-20T21:30:32.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:32.593+0000 7f50bb400700 1 -- 172.21.15.62:0/2369656172 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 20) v1 ==== 91256+0+0 (secure 0 0 0) 0x7f50b00105a0 con 0x7f50c4101450 2024-04-20T21:30:32.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:32.595+0000 7f50bb400700 1 --2- 172.21.15.62:0/2369656172 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f50a806d7e0 0x7f50a806fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:30:32.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:32.595+0000 7f50bb400700 1 -- 172.21.15.62:0/2369656172 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5468+0+0 (secure 0 0 0) 0x7f50b0014070 con 0x7f50c4101450 2024-04-20T21:30:32.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:32.595+0000 7f50c2800700 1 --2- 172.21.15.62:0/2369656172 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f50a806d7e0 0x7f50a806fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:30:32.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:32.596+0000 7f50c2800700 1 --2- 172.21.15.62:0/2369656172 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f50a806d7e0 0x7f50a806fc90 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f50b4009fd0 tx=0x7f50b4005fd0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:30:32.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:32.602+0000 7f50bb400700 1 -- 172.21.15.62:0/2369656172 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f50b0051720 con 0x7f50c4101450 2024-04-20T21:30:32.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:32.899+0000 7f50c9400700 1 -- 172.21.15.62:0/2369656172 --> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f50c4061190 con 0x7f50a806d7e0 2024-04-20T21:30:32.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:32.901+0000 7f50bb400700 1 -- 172.21.15.62:0/2369656172 <== mgr.14215 v2:172.21.15.62:6800/4116451241 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+341 (secure 0 0 0) 0x7f50c4061190 con 0x7f50a806d7e0 2024-04-20T21:30:32.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:32.904+0000 7f50c9400700 1 -- 172.21.15.62:0/2369656172 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f50a806d7e0 msgr2=0x7f50a806fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:30:32.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:32.904+0000 7f50c9400700 1 --2- 172.21.15.62:0/2369656172 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f50a806d7e0 0x7f50a806fc90 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f50b4009fd0 tx=0x7f50b4005fd0 comp rx=0 tx=0).stop 2024-04-20T21:30:32.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:32.904+0000 7f50c9400700 1 -- 172.21.15.62:0/2369656172 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f50c4101450 msgr2=0x7f50c41981b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:30:32.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:32.904+0000 7f50c9400700 1 --2- 172.21.15.62:0/2369656172 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f50c4101450 0x7f50c41981b0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f50b000e920 tx=0x7f50b000ec30 comp rx=0 tx=0).stop 2024-04-20T21:30:32.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:32.905+0000 7f50c9400700 1 -- 172.21.15.62:0/2369656172 shutdown_connections 2024-04-20T21:30:32.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:32.905+0000 7f50c9400700 1 --2- 172.21.15.62:0/2369656172 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f50a806d7e0 0x7f50a806fc90 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:30:32.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:32.905+0000 7f50c9400700 1 --2- 172.21.15.62:0/2369656172 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50c4100250 0x7f50c4197c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:30:32.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:32.905+0000 7f50c9400700 1 --2- 172.21.15.62:0/2369656172 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f50c4101450 0x7f50c41981b0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:30:32.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:32.905+0000 7f50c9400700 1 -- 172.21.15.62:0/2369656172 >> 172.21.15.62:0/2369656172 conn(0x7f50c40fb7e0 msgr2=0x7f50c4102670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:30:32.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:32.905+0000 7f50c9400700 1 -- 172.21.15.62:0/2369656172 shutdown_connections 2024-04-20T21:30:32.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:32.905+0000 7f50c9400700 1 -- 172.21.15.62:0/2369656172 wait complete. 2024-04-20T21:30:32.921 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T21:30:32.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:32 smithi189 ceph-mon[21562]: pgmap v317: 65 pgs: 65 active+clean; 59 MiB data, 363 MiB used, 536 GiB / 536 GiB avail; 1.3 MiB/s rd, 1.8 MiB/s wr, 74 op/s 2024-04-20T21:30:33.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:33.085+0000 7efe2ac00700 1 -- 172.21.15.62:0/2587303785 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efe241033d0 msgr2=0x7efe241057b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:30:33.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:33.085+0000 7efe2ac00700 1 --2- 172.21.15.62:0/2587303785 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efe241033d0 0x7efe241057b0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7efe10009910 tx=0x7efe10009c20 comp rx=0 tx=0).stop 2024-04-20T21:30:33.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:33.085+0000 7efe2ac00700 1 -- 172.21.15.62:0/2587303785 shutdown_connections 2024-04-20T21:30:33.085 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:33.085+0000 7efe2ac00700 1 --2- 172.21.15.62:0/2587303785 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efe241033d0 0x7efe241057b0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:30:33.085 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:33.085+0000 7efe2ac00700 1 --2- 172.21.15.62:0/2587303785 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efe24100ab0 0x7efe24102e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:30:33.085 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:33.085+0000 7efe2ac00700 1 -- 172.21.15.62:0/2587303785 >> 172.21.15.62:0/2587303785 conn(0x7efe240fa4d0 msgr2=0x7efe240fc900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:30:33.085 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:33.085+0000 7efe2ac00700 1 -- 172.21.15.62:0/2587303785 shutdown_connections 2024-04-20T21:30:33.085 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:33.085+0000 7efe2ac00700 1 -- 172.21.15.62:0/2587303785 wait complete. 2024-04-20T21:30:33.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:33.086+0000 7efe2ac00700 1 Processor -- start 2024-04-20T21:30:33.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:33.086+0000 7efe2ac00700 1 -- start start 2024-04-20T21:30:33.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:33.086+0000 7efe2ac00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efe24100ab0 0x7efe24100410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:30:33.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:33.086+0000 7efe2ac00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efe241033d0 0x7efe240fea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:30:33.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:33.087+0000 7efe2ac00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe240fefa0 con 0x7efe241033d0 2024-04-20T21:30:33.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:33.087+0000 7efe2ac00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe240ff0e0 con 0x7efe24100ab0 2024-04-20T21:30:33.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:33.087+0000 7efe28e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efe241033d0 0x7efe240fea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:30:33.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:33.087+0000 7efe28e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efe241033d0 0x7efe240fea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34072/0 (socket says 172.21.15.62:34072) 2024-04-20T21:30:33.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:33.087+0000 7efe28e00700 1 -- 172.21.15.62:0/3896690711 learned_addr learned my addr 172.21.15.62:0/3896690711 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:30:33.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:33.087+0000 7efe29800700 1 --2- 172.21.15.62:0/3896690711 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efe24100ab0 0x7efe24100410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:30:33.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:33.087+0000 7efe28e00700 1 -- 172.21.15.62:0/3896690711 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efe24100ab0 msgr2=0x7efe24100410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:30:33.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:33.087+0000 7efe28e00700 1 --2- 172.21.15.62:0/3896690711 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efe24100ab0 0x7efe24100410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:30:33.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:33.087+0000 7efe28e00700 1 -- 172.21.15.62:0/3896690711 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe100095f0 con 0x7efe241033d0 2024-04-20T21:30:33.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:33.087+0000 7efe29800700 1 --2- 172.21.15.62:0/3896690711 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efe24100ab0 0x7efe24100410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-20T21:30:33.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:33.088+0000 7efe28e00700 1 --2- 172.21.15.62:0/3896690711 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efe241033d0 0x7efe240fea60 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7efe10005b40 tx=0x7efe1000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:30:33.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:33.088+0000 7efe22000700 1 -- 172.21.15.62:0/3896690711 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efe1001d070 con 0x7efe241033d0 2024-04-20T21:30:33.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:33.088+0000 7efe22000700 1 -- 172.21.15.62:0/3896690711 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7efe10022470 con 0x7efe241033d0 2024-04-20T21:30:33.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:33.088+0000 7efe22000700 1 -- 172.21.15.62:0/3896690711 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efe1000f690 con 0x7efe241033d0 2024-04-20T21:30:33.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:33.088+0000 7efe2ac00700 1 -- 172.21.15.62:0/3896690711 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe240ff360 con 0x7efe241033d0 2024-04-20T21:30:33.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:33.089+0000 7efe2ac00700 1 -- 172.21.15.62:0/3896690711 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe240ff850 con 0x7efe241033d0 2024-04-20T21:30:33.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:33.090+0000 7efe2ac00700 1 -- 172.21.15.62:0/3896690711 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe2418fa40 con 0x7efe241033d0 2024-04-20T21:30:33.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:33.091+0000 7efe22000700 1 -- 172.21.15.62:0/3896690711 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 20) v1 ==== 91256+0+0 (secure 0 0 0) 0x7efe100225e0 con 0x7efe241033d0 2024-04-20T21:30:33.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:33.092+0000 7efe22000700 1 --2- 172.21.15.62:0/3896690711 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7efe0c06d7e0 0x7efe0c06fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:30:33.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:33.092+0000 7efe22000700 1 -- 172.21.15.62:0/3896690711 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5468+0+0 (secure 0 0 0) 0x7efe1008e790 con 0x7efe241033d0 2024-04-20T21:30:33.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:33.092+0000 7efe29800700 1 --2- 172.21.15.62:0/3896690711 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7efe0c06d7e0 0x7efe0c06fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:30:33.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:33.094+0000 7efe29800700 1 --2- 172.21.15.62:0/3896690711 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7efe0c06d7e0 0x7efe0c06fc90 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7efe18005950 tx=0x7efe18005f50 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:30:33.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:33.098+0000 7efe22000700 1 -- 172.21.15.62:0/3896690711 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7efe1005c960 con 0x7efe241033d0 2024-04-20T21:30:33.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:32 smithi062 ceph-mon[16221]: pgmap v317: 65 pgs: 65 active+clean; 59 MiB data, 363 MiB used, 536 GiB / 536 GiB avail; 1.3 MiB/s rd, 1.8 MiB/s wr, 74 op/s 2024-04-20T21:30:33.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:33.367+0000 7efe2ac00700 1 -- 172.21.15.62:0/3896690711 --> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7efe24061190 con 0x7efe0c06d7e0 2024-04-20T21:30:33.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:33.369+0000 7efe22000700 1 -- 172.21.15.62:0/3896690711 <== mgr.14215 v2:172.21.15.62:6800/4116451241 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+341 (secure 0 0 0) 0x7efe24061190 con 0x7efe0c06d7e0 2024-04-20T21:30:33.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:33.372+0000 7efe2ac00700 1 -- 172.21.15.62:0/3896690711 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7efe0c06d7e0 msgr2=0x7efe0c06fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:30:33.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:33.372+0000 7efe2ac00700 1 --2- 172.21.15.62:0/3896690711 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7efe0c06d7e0 0x7efe0c06fc90 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7efe18005950 tx=0x7efe18005f50 comp rx=0 tx=0).stop 2024-04-20T21:30:33.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:33.372+0000 7efe2ac00700 1 -- 172.21.15.62:0/3896690711 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efe241033d0 msgr2=0x7efe240fea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:30:33.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:33.372+0000 7efe2ac00700 1 --2- 172.21.15.62:0/3896690711 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efe241033d0 0x7efe240fea60 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7efe10005b40 tx=0x7efe1000be20 comp rx=0 tx=0).stop 2024-04-20T21:30:33.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:33.373+0000 7efe2ac00700 1 -- 172.21.15.62:0/3896690711 shutdown_connections 2024-04-20T21:30:33.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:33.373+0000 7efe2ac00700 1 --2- 172.21.15.62:0/3896690711 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7efe0c06d7e0 0x7efe0c06fc90 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:30:33.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:33.373+0000 7efe2ac00700 1 --2- 172.21.15.62:0/3896690711 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efe24100ab0 0x7efe24100410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:30:33.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:33.373+0000 7efe2ac00700 1 --2- 172.21.15.62:0/3896690711 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efe241033d0 0x7efe240fea60 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:30:33.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:33.373+0000 7efe2ac00700 1 -- 172.21.15.62:0/3896690711 >> 172.21.15.62:0/3896690711 conn(0x7efe240fa4d0 msgr2=0x7efe240fc900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:30:33.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:33.373+0000 7efe2ac00700 1 -- 172.21.15.62:0/3896690711 shutdown_connections 2024-04-20T21:30:33.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:33.373+0000 7efe2ac00700 1 -- 172.21.15.62:0/3896690711 wait complete. 2024-04-20T21:30:33.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:33.557+0000 7f404ea00700 1 -- 172.21.15.62:0/3332126556 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4048100d60 msgr2=0x7f4048103140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:30:33.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:33.557+0000 7f404ea00700 1 --2- 172.21.15.62:0/3332126556 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4048100d60 0x7f4048103140 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f4038009910 tx=0x7f4038009c20 comp rx=0 tx=0).stop 2024-04-20T21:30:33.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:33.558+0000 7f404ea00700 1 -- 172.21.15.62:0/3332126556 shutdown_connections 2024-04-20T21:30:33.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:33.558+0000 7f404ea00700 1 --2- 172.21.15.62:0/3332126556 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4048103680 0x7f4048105a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:30:33.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:33.558+0000 7f404ea00700 1 --2- 172.21.15.62:0/3332126556 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4048100d60 0x7f4048103140 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:30:33.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:33.558+0000 7f404ea00700 1 -- 172.21.15.62:0/3332126556 >> 172.21.15.62:0/3332126556 conn(0x7f40480fa780 msgr2=0x7f40480fcbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:30:33.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:33.558+0000 7f404ea00700 1 -- 172.21.15.62:0/3332126556 shutdown_connections 2024-04-20T21:30:33.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:33.558+0000 7f404ea00700 1 -- 172.21.15.62:0/3332126556 wait complete. 2024-04-20T21:30:33.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:33.559+0000 7f404ea00700 1 Processor -- start 2024-04-20T21:30:33.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:33.559+0000 7f404ea00700 1 -- start start 2024-04-20T21:30:33.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:33.560+0000 7f404ea00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4048100d60 0x7f40481006c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:30:33.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:33.560+0000 7f404ea00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4048103680 0x7f40480fed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:30:33.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:33.560+0000 7f404ea00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40480ff250 con 0x7f4048100d60 2024-04-20T21:30:33.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:33.560+0000 7f404ea00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40480ff390 con 0x7f4048103680 2024-04-20T21:30:33.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:33.560+0000 7f4047e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4048100d60 0x7f40481006c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:30:33.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:33.560+0000 7f4047e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4048100d60 0x7f40481006c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34096/0 (socket says 172.21.15.62:34096) 2024-04-20T21:30:33.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:33.560+0000 7f4047e00700 1 -- 172.21.15.62:0/1924497215 learned_addr learned my addr 172.21.15.62:0/1924497215 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:30:33.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:33.561+0000 7f4047400700 1 --2- 172.21.15.62:0/1924497215 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4048103680 0x7f40480fed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:30:33.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:33.561+0000 7f4047e00700 1 -- 172.21.15.62:0/1924497215 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4048103680 msgr2=0x7f40480fed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:30:33.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:33.561+0000 7f4047e00700 1 --2- 172.21.15.62:0/1924497215 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4048103680 0x7f40480fed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:30:33.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:33.561+0000 7f4047e00700 1 -- 172.21.15.62:0/1924497215 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40380095f0 con 0x7f4048100d60 2024-04-20T21:30:33.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:33.561+0000 7f4047400700 1 --2- 172.21.15.62:0/1924497215 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4048103680 0x7f40480fed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-20T21:30:33.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:33.561+0000 7f4047e00700 1 --2- 172.21.15.62:0/1924497215 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4048100d60 0x7f40481006c0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f4038005b40 tx=0x7f403800fab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:30:33.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:33.561+0000 7f4044c00700 1 -- 172.21.15.62:0/1924497215 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f403801d070 con 0x7f4048100d60 2024-04-20T21:30:33.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:33.562+0000 7f4044c00700 1 -- 172.21.15.62:0/1924497215 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f4038005dc0 con 0x7f4048100d60 2024-04-20T21:30:33.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:33.562+0000 7f404ea00700 1 -- 172.21.15.62:0/1924497215 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40480ff610 con 0x7f4048100d60 2024-04-20T21:30:33.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:33.562+0000 7f4044c00700 1 -- 172.21.15.62:0/1924497215 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f40380226a0 con 0x7f4048100d60 2024-04-20T21:30:33.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:33.562+0000 7f404ea00700 1 -- 172.21.15.62:0/1924497215 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40480ffb00 con 0x7f4048100d60 2024-04-20T21:30:33.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:33.563+0000 7f404ea00700 1 -- 172.21.15.62:0/1924497215 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f404818fbc0 con 0x7f4048100d60 2024-04-20T21:30:33.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:33.565+0000 7f4044c00700 1 -- 172.21.15.62:0/1924497215 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 20) v1 ==== 91256+0+0 (secure 0 0 0) 0x7f4038003680 con 0x7f4048100d60 2024-04-20T21:30:33.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:33.566+0000 7f4044c00700 1 --2- 172.21.15.62:0/1924497215 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f402c06d7e0 0x7f402c06fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:30:33.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:33.566+0000 7f4044c00700 1 -- 172.21.15.62:0/1924497215 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5468+0+0 (secure 0 0 0) 0x7f403808e500 con 0x7f4048100d60 2024-04-20T21:30:33.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:33.566+0000 7f4047400700 1 --2- 172.21.15.62:0/1924497215 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f402c06d7e0 0x7f402c06fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:30:33.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:33.567+0000 7f4047400700 1 --2- 172.21.15.62:0/1924497215 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f402c06d7e0 0x7f402c06fc90 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f4034009cb0 tx=0x7f4034009260 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:30:33.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:33.571+0000 7f4044c00700 1 -- 172.21.15.62:0/1924497215 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f403805c650 con 0x7f4048100d60 2024-04-20T21:30:33.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:33.846+0000 7f404ea00700 1 -- 172.21.15.62:0/1924497215 --> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f4048061190 con 0x7f402c06d7e0 2024-04-20T21:30:33.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:33.859+0000 7f4044c00700 1 -- 172.21.15.62:0/1924497215 <== mgr.14215 v2:172.21.15.62:6800/4116451241 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f4048061190 con 0x7f402c06d7e0 2024-04-20T21:30:33.860 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T21:30:33.860 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (8m) 2m ago 10m 13.1M - 0.25.0 c8568f914cd2 957f663a9cd7 2024-04-20T21:30:33.860 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (10m) 2m ago 10m 7583k - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T21:30:33.860 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (9m) 12s ago 9m 7650k - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T21:30:33.860 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (10m) 2m ago 10m 6656k - 18.2.0 dc2bc1663786 36e10ef5be39 2024-04-20T21:30:33.861 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (9m) 12s ago 9m 6665k - 18.2.0 dc2bc1663786 4101f9953e3d 2024-04-20T21:30:33.861 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (8m) 2m ago 9m 74.1M - 9.4.7 954c08fa6188 99363bb095d8 2024-04-20T21:30:33.861 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (2m) 2m ago 2m 13.6M - 18.2.0 dc2bc1663786 5e6460aa6159 2024-04-20T21:30:33.861 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (3m) 2m ago 3m 14.9M - 18.2.0 dc2bc1663786 e65f2267544e 2024-04-20T21:30:33.861 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (3m) 12s ago 3m 27.6M - 18.2.0 dc2bc1663786 fb2a8a223db8 2024-04-20T21:30:33.861 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (2m) 12s ago 2m 27.3M - 18.2.0 dc2bc1663786 f2eec2ee8951 2024-04-20T21:30:33.861 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:9283,8765,8443 running (12m) 2m ago 12m 488M - 18.2.0 dc2bc1663786 e411b3dc7233 2024-04-20T21:30:33.861 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (17s) 12s ago 8m 48.6M - 19.0.0-3133-ga8c84f59 76bc6b91b726 4cde2bc4de24 2024-04-20T21:30:33.861 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (12m) 2m ago 12m 43.9M 2048M 18.2.0 dc2bc1663786 bd6efbd5c380 2024-04-20T21:30:33.861 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (8m) 12s ago 8m 36.7M 2048M 18.2.0 dc2bc1663786 3735c8e5db9a 2024-04-20T21:30:33.861 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (10m) 2m ago 10m 7738k - 1.5.0 0da6a335fe13 fc746d8ae72e 2024-04-20T21:30:33.861 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (9m) 12s ago 9m 7998k - 1.5.0 0da6a335fe13 db03af85b286 2024-04-20T21:30:33.861 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (7m) 2m ago 7m 46.1M 3947M 18.2.0 dc2bc1663786 3f315c3afc1f 2024-04-20T21:30:33.861 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (7m) 2m ago 7m 49.0M 3947M 18.2.0 dc2bc1663786 44e4d07416ad 2024-04-20T21:30:33.861 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (6m) 2m ago 6m 46.5M 3947M 18.2.0 dc2bc1663786 31c5d9586e98 2024-04-20T21:30:33.861 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (6m) 12s ago 6m 58.3M 4971M 18.2.0 dc2bc1663786 611c1e242693 2024-04-20T21:30:33.861 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (5m) 12s ago 5m 56.4M 4971M 18.2.0 dc2bc1663786 6c8fce91d45d 2024-04-20T21:30:33.862 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (4m) 12s ago 4m 53.4M 4971M 18.2.0 dc2bc1663786 8e4574bae8db 2024-04-20T21:30:33.862 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (8m) 2m ago 9m 51.5M - 2.43.0 a07b618ecd1d cd390b7e44d0 2024-04-20T21:30:33.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:33.863+0000 7f404ea00700 1 -- 172.21.15.62:0/1924497215 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f402c06d7e0 msgr2=0x7f402c06fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:30:33.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:33.863+0000 7f404ea00700 1 --2- 172.21.15.62:0/1924497215 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f402c06d7e0 0x7f402c06fc90 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f4034009cb0 tx=0x7f4034009260 comp rx=0 tx=0).stop 2024-04-20T21:30:33.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:33.863+0000 7f404ea00700 1 -- 172.21.15.62:0/1924497215 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4048100d60 msgr2=0x7f40481006c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:30:33.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:33.863+0000 7f404ea00700 1 --2- 172.21.15.62:0/1924497215 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4048100d60 0x7f40481006c0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f4038005b40 tx=0x7f403800fab0 comp rx=0 tx=0).stop 2024-04-20T21:30:33.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:33.863+0000 7f404ea00700 1 -- 172.21.15.62:0/1924497215 shutdown_connections 2024-04-20T21:30:33.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:33.863+0000 7f404ea00700 1 --2- 172.21.15.62:0/1924497215 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f402c06d7e0 0x7f402c06fc90 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:30:33.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:33.863+0000 7f404ea00700 1 --2- 172.21.15.62:0/1924497215 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4048100d60 0x7f40481006c0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:30:33.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:33.863+0000 7f404ea00700 1 --2- 172.21.15.62:0/1924497215 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4048103680 0x7f40480fed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:30:33.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:33.863+0000 7f404ea00700 1 -- 172.21.15.62:0/1924497215 >> 172.21.15.62:0/1924497215 conn(0x7f40480fa780 msgr2=0x7f40480fcbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:30:33.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:33.864+0000 7f404ea00700 1 -- 172.21.15.62:0/1924497215 shutdown_connections 2024-04-20T21:30:33.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:33.864+0000 7f404ea00700 1 -- 172.21.15.62:0/1924497215 wait complete. 2024-04-20T21:30:34.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:34.039+0000 7fe20e400700 1 -- 172.21.15.62:0/2975141437 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2080fef30 msgr2=0x7fe2080ff340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:30:34.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:34.039+0000 7fe20e400700 1 --2- 172.21.15.62:0/2975141437 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2080fef30 0x7fe2080ff340 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fe1fc009910 tx=0x7fe1fc009c20 comp rx=0 tx=0).stop 2024-04-20T21:30:34.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:34.040+0000 7fe20e400700 1 -- 172.21.15.62:0/2975141437 shutdown_connections 2024-04-20T21:30:34.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:34.040+0000 7fe20e400700 1 --2- 172.21.15.62:0/2975141437 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe2081001d0 0x7fe208100640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:30:34.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:34.040+0000 7fe20e400700 1 --2- 172.21.15.62:0/2975141437 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2080fef30 0x7fe2080ff340 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:30:34.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:34.040+0000 7fe20e400700 1 -- 172.21.15.62:0/2975141437 >> 172.21.15.62:0/2975141437 conn(0x7fe2080fa580 msgr2=0x7fe2080fc9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:30:34.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:34.040+0000 7fe20e400700 1 -- 172.21.15.62:0/2975141437 shutdown_connections 2024-04-20T21:30:34.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:34.040+0000 7fe20e400700 1 -- 172.21.15.62:0/2975141437 wait complete. 2024-04-20T21:30:34.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:34.041+0000 7fe20e400700 1 Processor -- start 2024-04-20T21:30:34.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:34.041+0000 7fe20e400700 1 -- start start 2024-04-20T21:30:34.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:34.042+0000 7fe20e400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2080fef30 0x7fe208197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:30:34.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:34.042+0000 7fe20e400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe2081001d0 0x7fe208197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:30:34.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:34.042+0000 7fe20e400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe208198560 con 0x7fe2080fef30 2024-04-20T21:30:34.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:34.042+0000 7fe20e400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2081986a0 con 0x7fe2081001d0 2024-04-20T21:30:34.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:34.042+0000 7fe20d000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2080fef30 0x7fe208197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:30:34.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:34.042+0000 7fe20d000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2080fef30 0x7fe208197a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34114/0 (socket says 172.21.15.62:34114) 2024-04-20T21:30:34.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:34.042+0000 7fe20d000700 1 -- 172.21.15.62:0/3876663376 learned_addr learned my addr 172.21.15.62:0/3876663376 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:30:34.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:34.042+0000 7fe207e00700 1 --2- 172.21.15.62:0/3876663376 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe2081001d0 0x7fe208197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:30:34.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:34.043+0000 7fe20d000700 1 -- 172.21.15.62:0/3876663376 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe2081001d0 msgr2=0x7fe208197f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:30:34.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:34.043+0000 7fe20d000700 1 --2- 172.21.15.62:0/3876663376 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe2081001d0 0x7fe208197f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:30:34.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:34.043+0000 7fe20d000700 1 -- 172.21.15.62:0/3876663376 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1fc0095f0 con 0x7fe2080fef30 2024-04-20T21:30:34.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:34.043+0000 7fe20d000700 1 --2- 172.21.15.62:0/3876663376 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2080fef30 0x7fe208197a00 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fe1fc005b40 tx=0x7fe1fc00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:30:34.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:34.043+0000 7fe205600700 1 -- 172.21.15.62:0/3876663376 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe1fc01d070 con 0x7fe2080fef30 2024-04-20T21:30:34.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:34.043+0000 7fe205600700 1 -- 172.21.15.62:0/3876663376 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fe1fc022470 con 0x7fe2080fef30 2024-04-20T21:30:34.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:34.043+0000 7fe205600700 1 -- 172.21.15.62:0/3876663376 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe1fc00f690 con 0x7fe2080fef30 2024-04-20T21:30:34.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:34.043+0000 7fe20e400700 1 -- 172.21.15.62:0/3876663376 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe20819d0f0 con 0x7fe2080fef30 2024-04-20T21:30:34.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:34.044+0000 7fe20e400700 1 -- 172.21.15.62:0/3876663376 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe20819d5e0 con 0x7fe2080fef30 2024-04-20T21:30:34.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:34.045+0000 7fe20e400700 1 -- 172.21.15.62:0/3876663376 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe208107a10 con 0x7fe2080fef30 2024-04-20T21:30:34.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:34.046+0000 7fe205600700 1 -- 172.21.15.62:0/3876663376 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 20) v1 ==== 91256+0+0 (secure 0 0 0) 0x7fe1fc0225e0 con 0x7fe2080fef30 2024-04-20T21:30:34.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:34.048+0000 7fe205600700 1 --2- 172.21.15.62:0/3876663376 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fe1f006d7e0 0x7fe1f006fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:30:34.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:34.048+0000 7fe205600700 1 -- 172.21.15.62:0/3876663376 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5468+0+0 (secure 0 0 0) 0x7fe1fc08e7a0 con 0x7fe2080fef30 2024-04-20T21:30:34.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:34.048+0000 7fe207e00700 1 --2- 172.21.15.62:0/3876663376 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fe1f006d7e0 0x7fe1f006fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:30:34.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:34.049+0000 7fe207e00700 1 --2- 172.21.15.62:0/3876663376 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fe1f006d7e0 0x7fe1f006fc90 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fe1f4005fd0 tx=0x7fe1f400a560 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:30:34.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:34.053+0000 7fe205600700 1 -- 172.21.15.62:0/3876663376 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fe1fc05c970 con 0x7fe2080fef30 2024-04-20T21:30:34.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:33 smithi189 ceph-mon[21562]: from='client.14594 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:30:34.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:33 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:30:34.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:33 smithi062 ceph-mon[16221]: from='client.14594 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:30:34.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:33 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:30:34.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:34.431+0000 7fe20e400700 1 -- 172.21.15.62:0/3876663376 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fe208107c00 con 0x7fe2080fef30 2024-04-20T21:30:34.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:34.432+0000 7fe205600700 1 -- 172.21.15.62:0/3876663376 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+766 (secure 0 0 0) 0x7fe1fc05c500 con 0x7fe2080fef30 2024-04-20T21:30:34.432 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:30:34.432 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T21:30:34.432 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 2 2024-04-20T21:30:34.432 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:30:34.432 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T21:30:34.432 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 1, 2024-04-20T21:30:34.432 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 1 2024-04-20T21:30:34.432 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:30:34.433 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T21:30:34.433 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 6 2024-04-20T21:30:34.433 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:30:34.433 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T21:30:34.433 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 4 2024-04-20T21:30:34.433 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:30:34.433 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T21:30:34.433 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 13, 2024-04-20T21:30:34.433 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 1 2024-04-20T21:30:34.433 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T21:30:34.433 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:30:34.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:34.435+0000 7fe20e400700 1 -- 172.21.15.62:0/3876663376 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fe1f006d7e0 msgr2=0x7fe1f006fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:30:34.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:34.435+0000 7fe20e400700 1 --2- 172.21.15.62:0/3876663376 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fe1f006d7e0 0x7fe1f006fc90 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fe1f4005fd0 tx=0x7fe1f400a560 comp rx=0 tx=0).stop 2024-04-20T21:30:34.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:34.435+0000 7fe20e400700 1 -- 172.21.15.62:0/3876663376 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2080fef30 msgr2=0x7fe208197a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:30:34.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:34.435+0000 7fe20e400700 1 --2- 172.21.15.62:0/3876663376 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2080fef30 0x7fe208197a00 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fe1fc005b40 tx=0x7fe1fc00be20 comp rx=0 tx=0).stop 2024-04-20T21:30:34.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:34.435+0000 7fe20e400700 1 -- 172.21.15.62:0/3876663376 shutdown_connections 2024-04-20T21:30:34.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:34.435+0000 7fe20e400700 1 --2- 172.21.15.62:0/3876663376 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7fe1f006d7e0 0x7fe1f006fc90 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:30:34.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:34.435+0000 7fe20e400700 1 --2- 172.21.15.62:0/3876663376 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2080fef30 0x7fe208197a00 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:30:34.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:34.435+0000 7fe20e400700 1 --2- 172.21.15.62:0/3876663376 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe2081001d0 0x7fe208197f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:30:34.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:34.435+0000 7fe20e400700 1 -- 172.21.15.62:0/3876663376 >> 172.21.15.62:0/3876663376 conn(0x7fe2080fa580 msgr2=0x7fe2080fc0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:30:34.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:34.436+0000 7fe20e400700 1 -- 172.21.15.62:0/3876663376 shutdown_connections 2024-04-20T21:30:34.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:34.436+0000 7fe20e400700 1 -- 172.21.15.62:0/3876663376 wait complete. 2024-04-20T21:30:34.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:34.610+0000 7f6d67200700 1 -- 172.21.15.62:0/3404196971 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d68100ab0 msgr2=0x7f6d68102e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:30:34.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:34.610+0000 7f6d67200700 1 --2- 172.21.15.62:0/3404196971 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d68100ab0 0x7f6d68102e90 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f6d58009910 tx=0x7f6d58009c20 comp rx=0 tx=0).stop 2024-04-20T21:30:34.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:34.610+0000 7f6d67200700 1 -- 172.21.15.62:0/3404196971 shutdown_connections 2024-04-20T21:30:34.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:34.610+0000 7f6d67200700 1 --2- 172.21.15.62:0/3404196971 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6d681033d0 0x7f6d681057b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:30:34.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:34.610+0000 7f6d67200700 1 --2- 172.21.15.62:0/3404196971 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d68100ab0 0x7f6d68102e90 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:30:34.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:34.610+0000 7f6d67200700 1 -- 172.21.15.62:0/3404196971 >> 172.21.15.62:0/3404196971 conn(0x7f6d680fa4d0 msgr2=0x7f6d680fc900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:30:34.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:34.611+0000 7f6d67200700 1 -- 172.21.15.62:0/3404196971 shutdown_connections 2024-04-20T21:30:34.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:34.611+0000 7f6d67200700 1 -- 172.21.15.62:0/3404196971 wait complete. 2024-04-20T21:30:34.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:34.612+0000 7f6d67200700 1 Processor -- start 2024-04-20T21:30:34.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:34.612+0000 7f6d67200700 1 -- start start 2024-04-20T21:30:34.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:34.613+0000 7f6d67200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6d68100ab0 0x7f6d68100410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:30:34.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:34.613+0000 7f6d67200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d681033d0 0x7f6d680fea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:30:34.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:34.613+0000 7f6d67200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d680fefa0 con 0x7f6d681033d0 2024-04-20T21:30:34.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:34.613+0000 7f6d67200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d680ff0e0 con 0x7f6d68100ab0 2024-04-20T21:30:34.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:34.613+0000 7f6d65400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d681033d0 0x7f6d680fea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:30:34.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:34.613+0000 7f6d65400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d681033d0 0x7f6d680fea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34130/0 (socket says 172.21.15.62:34130) 2024-04-20T21:30:34.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:34.613+0000 7f6d65400700 1 -- 172.21.15.62:0/2191608677 learned_addr learned my addr 172.21.15.62:0/2191608677 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:30:34.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:34.614+0000 7f6d65e00700 1 --2- 172.21.15.62:0/2191608677 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6d68100ab0 0x7f6d68100410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:30:34.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:34.614+0000 7f6d65400700 1 -- 172.21.15.62:0/2191608677 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6d68100ab0 msgr2=0x7f6d68100410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:30:34.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:34.614+0000 7f6d65400700 1 --2- 172.21.15.62:0/2191608677 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6d68100ab0 0x7f6d68100410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:30:34.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:34.614+0000 7f6d65400700 1 -- 172.21.15.62:0/2191608677 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d580095f0 con 0x7f6d681033d0 2024-04-20T21:30:34.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:34.614+0000 7f6d65400700 1 --2- 172.21.15.62:0/2191608677 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d681033d0 0x7f6d680fea60 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f6d5400d8d0 tx=0x7f6d5400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:30:34.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:34.615+0000 7f6d5ea00700 1 -- 172.21.15.62:0/2191608677 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6d54009940 con 0x7f6d681033d0 2024-04-20T21:30:34.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:34.615+0000 7f6d5ea00700 1 -- 172.21.15.62:0/2191608677 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f6d54010460 con 0x7f6d681033d0 2024-04-20T21:30:34.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:34.615+0000 7f6d5ea00700 1 -- 172.21.15.62:0/2191608677 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6d5400f5f0 con 0x7f6d681033d0 2024-04-20T21:30:34.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:34.615+0000 7f6d67200700 1 -- 172.21.15.62:0/2191608677 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d680ff3c0 con 0x7f6d681033d0 2024-04-20T21:30:34.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:34.615+0000 7f6d67200700 1 -- 172.21.15.62:0/2191608677 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d6819d7e0 con 0x7f6d681033d0 2024-04-20T21:30:34.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:34.617+0000 7f6d67200700 1 -- 172.21.15.62:0/2191608677 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d6818fa40 con 0x7f6d681033d0 2024-04-20T21:30:34.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:34.619+0000 7f6d5ea00700 1 -- 172.21.15.62:0/2191608677 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 20) v1 ==== 91256+0+0 (secure 0 0 0) 0x7f6d540105d0 con 0x7f6d681033d0 2024-04-20T21:30:34.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:34.620+0000 7f6d5ea00700 1 --2- 172.21.15.62:0/2191608677 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f6d4c06d830 0x7f6d4c06fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:30:34.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:34.620+0000 7f6d5ea00700 1 -- 172.21.15.62:0/2191608677 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5468+0+0 (secure 0 0 0) 0x7f6d5408ce60 con 0x7f6d681033d0 2024-04-20T21:30:34.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:34.621+0000 7f6d65e00700 1 --2- 172.21.15.62:0/2191608677 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f6d4c06d830 0x7f6d4c06fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:30:34.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:34.622+0000 7f6d65e00700 1 --2- 172.21.15.62:0/2191608677 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f6d4c06d830 0x7f6d4c06fce0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f6d58006010 tx=0x7f6d5800b540 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:30:34.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:34.625+0000 7f6d5ea00700 1 -- 172.21.15.62:0/2191608677 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f6d5405b030 con 0x7f6d681033d0 2024-04-20T21:30:34.904 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:34 smithi189 ceph-mon[21562]: from='client.14598 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:30:34.905 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:34 smithi189 ceph-mon[21562]: from='client.14602 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:30:34.905 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:34 smithi189 ceph-mon[21562]: pgmap v318: 65 pgs: 65 active+clean; 65 MiB data, 379 MiB used, 536 GiB / 536 GiB avail; 1.5 MiB/s rd, 2.1 MiB/s wr, 80 op/s 2024-04-20T21:30:34.905 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:34 smithi189 ceph-mon[21562]: from='client.? 172.21.15.62:0/3876663376' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:30:34.905 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:34 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:30:34.905 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:34 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:30:34.905 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:34 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:30:34.905 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:34 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T21:30:34.905 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:34 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:30:35.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:35.038+0000 7f6d67200700 1 -- 172.21.15.62:0/2191608677 --> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6d68061190 con 0x7f6d4c06d830 2024-04-20T21:30:35.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:35.040+0000 7f6d5ea00700 1 -- 172.21.15.62:0/2191608677 <== mgr.14215 v2:172.21.15.62:6800/4116451241 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+341 (secure 0 0 0) 0x7f6d68061190 con 0x7f6d4c06d830 2024-04-20T21:30:35.040 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:30:35.040 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T21:30:35.040 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T21:30:35.040 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-20T21:30:35.040 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [], 2024-04-20T21:30:35.040 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "1/2 daemons upgraded", 2024-04-20T21:30:35.040 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading mgr daemons", 2024-04-20T21:30:35.040 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T21:30:35.040 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:30:35.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:35.045+0000 7f6d67200700 1 -- 172.21.15.62:0/2191608677 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f6d4c06d830 msgr2=0x7f6d4c06fce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:30:35.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:35.045+0000 7f6d67200700 1 --2- 172.21.15.62:0/2191608677 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f6d4c06d830 0x7f6d4c06fce0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f6d58006010 tx=0x7f6d5800b540 comp rx=0 tx=0).stop 2024-04-20T21:30:35.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:35.045+0000 7f6d67200700 1 -- 172.21.15.62:0/2191608677 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d681033d0 msgr2=0x7f6d680fea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:30:35.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:35.045+0000 7f6d67200700 1 --2- 172.21.15.62:0/2191608677 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d681033d0 0x7f6d680fea60 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f6d5400d8d0 tx=0x7f6d5400dc90 comp rx=0 tx=0).stop 2024-04-20T21:30:35.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:35.045+0000 7f6d67200700 1 -- 172.21.15.62:0/2191608677 shutdown_connections 2024-04-20T21:30:35.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:35.045+0000 7f6d67200700 1 --2- 172.21.15.62:0/2191608677 >> [v2:172.21.15.62:6800/4116451241,v1:172.21.15.62:6801/4116451241] conn(0x7f6d4c06d830 0x7f6d4c06fce0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:30:35.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:35.045+0000 7f6d67200700 1 --2- 172.21.15.62:0/2191608677 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6d68100ab0 0x7f6d68100410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:30:35.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:35.045+0000 7f6d67200700 1 --2- 172.21.15.62:0/2191608677 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d681033d0 0x7f6d680fea60 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:30:35.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:35.045+0000 7f6d67200700 1 -- 172.21.15.62:0/2191608677 >> 172.21.15.62:0/2191608677 conn(0x7f6d680fa4d0 msgr2=0x7f6d680fc900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:30:35.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:35.046+0000 7f6d67200700 1 -- 172.21.15.62:0/2191608677 shutdown_connections 2024-04-20T21:30:35.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:30:35.046+0000 7f6d67200700 1 -- 172.21.15.62:0/2191608677 wait complete. 2024-04-20T21:30:35.191 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:34 smithi062 ceph-mon[16221]: from='client.14598 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:30:35.191 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:34 smithi062 ceph-mon[16221]: from='client.14602 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:30:35.191 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:34 smithi062 ceph-mon[16221]: pgmap v318: 65 pgs: 65 active+clean; 65 MiB data, 379 MiB used, 536 GiB / 536 GiB avail; 1.5 MiB/s rd, 2.1 MiB/s wr, 80 op/s 2024-04-20T21:30:35.191 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:34 smithi062 ceph-mon[16221]: from='client.? 172.21.15.62:0/3876663376' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:30:35.191 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:34 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:30:35.191 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:34 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:30:35.191 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:34 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:30:35.191 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:34 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T21:30:35.191 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:34 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:30:36.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:35 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:30:36.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:35 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mgr fail", "who": "smithi062.rqxuxi"}]: dispatch 2024-04-20T21:30:36.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:35 smithi062 ceph-mon[16221]: osdmap e46: 6 total, 6 up, 6 in 2024-04-20T21:30:36.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:35 smithi062 ceph-mon[16221]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "mgr fail", "who": "smithi062.rqxuxi"}]': finished 2024-04-20T21:30:36.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:35 smithi062 ceph-mon[16221]: mgrmap e21: smithi189.tawavz(active, starting, since 0.0131365s) 2024-04-20T21:30:36.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:35 smithi062 ceph-mon[16221]: from='mgr.24379 172.21.15.189:0/1768059451' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "mon metadata", "id": "smithi062"}]: dispatch 2024-04-20T21:30:36.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:35 smithi062 ceph-mon[16221]: from='mgr.24379 172.21.15.189:0/1768059451' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "mon metadata", "id": "smithi189"}]: dispatch 2024-04-20T21:30:36.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:35 smithi062 ceph-mon[16221]: from='mgr.24379 172.21.15.189:0/1768059451' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi062.ryqblb"}]: dispatch 2024-04-20T21:30:36.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:35 smithi062 ceph-mon[16221]: from='mgr.24379 172.21.15.189:0/1768059451' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi062.ozizwi"}]: dispatch 2024-04-20T21:30:36.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:35 smithi062 ceph-mon[16221]: from='mgr.24379 172.21.15.189:0/1768059451' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi189.dvimer"}]: dispatch 2024-04-20T21:30:36.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:35 smithi062 ceph-mon[16221]: from='mgr.24379 172.21.15.189:0/1768059451' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi189.xbgspw"}]: dispatch 2024-04-20T21:30:36.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:35 smithi062 ceph-mon[16221]: from='mgr.24379 172.21.15.189:0/1768059451' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "mgr metadata", "who": "smithi189.tawavz", "id": "smithi189.tawavz"}]: dispatch 2024-04-20T21:30:36.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:35 smithi062 ceph-mon[16221]: from='mgr.24379 172.21.15.189:0/1768059451' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-20T21:30:36.037 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:35 smithi062 ceph-mon[16221]: from='mgr.24379 172.21.15.189:0/1768059451' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-20T21:30:36.037 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:35 smithi062 ceph-mon[16221]: from='mgr.24379 172.21.15.189:0/1768059451' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-20T21:30:36.037 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:35 smithi062 ceph-mon[16221]: from='mgr.24379 172.21.15.189:0/1768059451' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-20T21:30:36.037 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:35 smithi062 ceph-mon[16221]: from='mgr.24379 172.21.15.189:0/1768059451' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-20T21:30:36.037 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:35 smithi062 ceph-mon[16221]: from='mgr.24379 172.21.15.189:0/1768059451' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-20T21:30:36.037 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:35 smithi062 ceph-mon[16221]: from='mgr.24379 172.21.15.189:0/1768059451' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-20T21:30:36.037 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:35 smithi062 ceph-mon[16221]: from='mgr.24379 172.21.15.189:0/1768059451' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-20T21:30:36.037 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:35 smithi062 ceph-mon[16221]: from='mgr.24379 172.21.15.189:0/1768059451' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-20T21:30:36.037 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:35 smithi062 ceph-mon[16221]: Manager daemon smithi189.tawavz is now available 2024-04-20T21:30:36.037 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:35 smithi062 ceph-mon[16221]: from='mgr.24379 172.21.15.189:0/1768059451' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:30:36.037 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:35 smithi062 ceph-mon[16221]: from='mgr.24379 172.21.15.189:0/1768059451' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:30:36.037 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:35 smithi062 ceph-mon[16221]: from='mgr.24379 172.21.15.189:0/1768059451' entity='mgr.smithi189.tawavz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi189.tawavz/mirror_snapshot_schedule"}]: dispatch 2024-04-20T21:30:36.037 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:35 smithi062 ceph-mon[16221]: from='mgr.24379 ' entity='mgr.smithi189.tawavz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi189.tawavz/mirror_snapshot_schedule"}]: dispatch 2024-04-20T21:30:36.038 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:35 smithi062 ceph-mon[16221]: from='mgr.24379 172.21.15.189:0/1768059451' entity='mgr.smithi189.tawavz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi189.tawavz/trash_purge_schedule"}]: dispatch 2024-04-20T21:30:36.038 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:35 smithi062 ceph-mon[16221]: from='mgr.24379 ' entity='mgr.smithi189.tawavz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi189.tawavz/trash_purge_schedule"}]: dispatch 2024-04-20T21:30:36.038 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:35 smithi062 ceph-mon[16221]: [20/Apr/2024:21:30:35] ENGINE Bus STARTING 2024-04-20T21:30:36.038 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:35 smithi062 ceph-mon[16221]: [20/Apr/2024:21:30:35] ENGINE Serving on https://172.21.15.189:7150 2024-04-20T21:30:36.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:35 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:30:36.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:35 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mgr fail", "who": "smithi062.rqxuxi"}]: dispatch 2024-04-20T21:30:36.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:35 smithi189 ceph-mon[21562]: osdmap e46: 6 total, 6 up, 6 in 2024-04-20T21:30:36.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:35 smithi189 ceph-mon[21562]: from='mgr.14215 172.21.15.62:0/22341909' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "mgr fail", "who": "smithi062.rqxuxi"}]': finished 2024-04-20T21:30:36.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:35 smithi189 ceph-mon[21562]: mgrmap e21: smithi189.tawavz(active, starting, since 0.0131365s) 2024-04-20T21:30:36.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:35 smithi189 ceph-mon[21562]: from='mgr.24379 172.21.15.189:0/1768059451' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "mon metadata", "id": "smithi062"}]: dispatch 2024-04-20T21:30:36.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:35 smithi189 ceph-mon[21562]: from='mgr.24379 172.21.15.189:0/1768059451' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "mon metadata", "id": "smithi189"}]: dispatch 2024-04-20T21:30:36.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:35 smithi189 ceph-mon[21562]: from='mgr.24379 172.21.15.189:0/1768059451' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi062.ryqblb"}]: dispatch 2024-04-20T21:30:36.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:35 smithi189 ceph-mon[21562]: from='mgr.24379 172.21.15.189:0/1768059451' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi062.ozizwi"}]: dispatch 2024-04-20T21:30:36.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:35 smithi189 ceph-mon[21562]: from='mgr.24379 172.21.15.189:0/1768059451' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi189.dvimer"}]: dispatch 2024-04-20T21:30:36.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:35 smithi189 ceph-mon[21562]: from='mgr.24379 172.21.15.189:0/1768059451' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi189.xbgspw"}]: dispatch 2024-04-20T21:30:36.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:35 smithi189 ceph-mon[21562]: from='mgr.24379 172.21.15.189:0/1768059451' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "mgr metadata", "who": "smithi189.tawavz", "id": "smithi189.tawavz"}]: dispatch 2024-04-20T21:30:36.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:35 smithi189 ceph-mon[21562]: from='mgr.24379 172.21.15.189:0/1768059451' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-20T21:30:36.239 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:35 smithi189 ceph-mon[21562]: from='mgr.24379 172.21.15.189:0/1768059451' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-20T21:30:36.239 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:35 smithi189 ceph-mon[21562]: from='mgr.24379 172.21.15.189:0/1768059451' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-20T21:30:36.239 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:35 smithi189 ceph-mon[21562]: from='mgr.24379 172.21.15.189:0/1768059451' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-20T21:30:36.239 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:35 smithi189 ceph-mon[21562]: from='mgr.24379 172.21.15.189:0/1768059451' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-20T21:30:36.239 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:35 smithi189 ceph-mon[21562]: from='mgr.24379 172.21.15.189:0/1768059451' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-20T21:30:36.239 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:35 smithi189 ceph-mon[21562]: from='mgr.24379 172.21.15.189:0/1768059451' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-20T21:30:36.239 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:35 smithi189 ceph-mon[21562]: from='mgr.24379 172.21.15.189:0/1768059451' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-20T21:30:36.239 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:35 smithi189 ceph-mon[21562]: from='mgr.24379 172.21.15.189:0/1768059451' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-20T21:30:36.239 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:35 smithi189 ceph-mon[21562]: Manager daemon smithi189.tawavz is now available 2024-04-20T21:30:36.239 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:35 smithi189 ceph-mon[21562]: from='mgr.24379 172.21.15.189:0/1768059451' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:30:36.239 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:35 smithi189 ceph-mon[21562]: from='mgr.24379 172.21.15.189:0/1768059451' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:30:36.239 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:35 smithi189 ceph-mon[21562]: from='mgr.24379 172.21.15.189:0/1768059451' entity='mgr.smithi189.tawavz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi189.tawavz/mirror_snapshot_schedule"}]: dispatch 2024-04-20T21:30:36.239 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:35 smithi189 ceph-mon[21562]: from='mgr.24379 ' entity='mgr.smithi189.tawavz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi189.tawavz/mirror_snapshot_schedule"}]: dispatch 2024-04-20T21:30:36.239 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:35 smithi189 ceph-mon[21562]: from='mgr.24379 172.21.15.189:0/1768059451' entity='mgr.smithi189.tawavz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi189.tawavz/trash_purge_schedule"}]: dispatch 2024-04-20T21:30:36.239 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:35 smithi189 ceph-mon[21562]: from='mgr.24379 ' entity='mgr.smithi189.tawavz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi189.tawavz/trash_purge_schedule"}]: dispatch 2024-04-20T21:30:36.240 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:35 smithi189 ceph-mon[21562]: [20/Apr/2024:21:30:35] ENGINE Bus STARTING 2024-04-20T21:30:36.240 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:35 smithi189 ceph-mon[21562]: [20/Apr/2024:21:30:35] ENGINE Serving on https://172.21.15.189:7150 2024-04-20T21:30:37.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:37 smithi189 ceph-mon[21562]: [20/Apr/2024:21:30:35] ENGINE Serving on http://172.21.15.189:8765 2024-04-20T21:30:37.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:37 smithi189 ceph-mon[21562]: [20/Apr/2024:21:30:35] ENGINE Bus STARTED 2024-04-20T21:30:37.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:37 smithi189 ceph-mon[21562]: Deploying cephadm binary to smithi189 2024-04-20T21:30:37.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:37 smithi189 ceph-mon[21562]: Deploying cephadm binary to smithi062 2024-04-20T21:30:37.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:37 smithi189 ceph-mon[21562]: mgrmap e22: smithi189.tawavz(active, since 1.01488s) 2024-04-20T21:30:37.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:37 smithi189 ceph-mon[21562]: pgmap v3: 65 pgs: 65 active+clean; 76 MiB data, 411 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:30:37.534 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:37 smithi062 ceph-mon[16221]: [20/Apr/2024:21:30:35] ENGINE Serving on http://172.21.15.189:8765 2024-04-20T21:30:37.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:37 smithi062 ceph-mon[16221]: [20/Apr/2024:21:30:35] ENGINE Bus STARTED 2024-04-20T21:30:37.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:37 smithi062 ceph-mon[16221]: Deploying cephadm binary to smithi189 2024-04-20T21:30:37.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:37 smithi062 ceph-mon[16221]: Deploying cephadm binary to smithi062 2024-04-20T21:30:37.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:37 smithi062 ceph-mon[16221]: mgrmap e22: smithi189.tawavz(active, since 1.01488s) 2024-04-20T21:30:37.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:37 smithi062 ceph-mon[16221]: pgmap v3: 65 pgs: 65 active+clean; 76 MiB data, 411 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:30:38.486 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:38 smithi189 ceph-mon[21562]: pgmap v4: 65 pgs: 65 active+clean; 76 MiB data, 411 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:30:38.515 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:38 smithi062 ceph-mon[16221]: pgmap v4: 65 pgs: 65 active+clean; 76 MiB data, 411 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:30:39.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:39 smithi189 ceph-mon[21562]: mgrmap e23: smithi189.tawavz(active, since 3s) 2024-04-20T21:30:39.534 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:39 smithi062 ceph-mon[16221]: mgrmap e23: smithi189.tawavz(active, since 3s) 2024-04-20T21:30:40.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:40 smithi062 ceph-mon[16221]: pgmap v5: 65 pgs: 65 active+clean; 76 MiB data, 411 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:30:40.486 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:40 smithi189 ceph-mon[21562]: pgmap v5: 65 pgs: 65 active+clean; 76 MiB data, 411 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:30:43.236 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:42 smithi189 ceph-mon[21562]: pgmap v6: 65 pgs: 65 active+clean; 76 MiB data, 411 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:30:43.246 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:42 smithi062 ceph-mon[16221]: pgmap v6: 65 pgs: 65 active+clean; 76 MiB data, 411 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:30:45.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:44 smithi189 ceph-mon[21562]: pgmap v7: 65 pgs: 65 active+clean; 94 MiB data, 474 MiB used, 536 GiB / 536 GiB avail; 777 KiB/s rd, 2.6 MiB/s wr, 147 op/s 2024-04-20T21:30:45.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:44 smithi062 ceph-mon[16221]: pgmap v7: 65 pgs: 65 active+clean; 94 MiB data, 474 MiB used, 536 GiB / 536 GiB avail; 777 KiB/s rd, 2.6 MiB/s wr, 147 op/s 2024-04-20T21:30:47.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:46 smithi189 ceph-mon[21562]: pgmap v8: 65 pgs: 65 active+clean; 94 MiB data, 474 MiB used, 536 GiB / 536 GiB avail; 604 KiB/s rd, 2.0 MiB/s wr, 114 op/s 2024-04-20T21:30:47.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:46 smithi189 ceph-mon[21562]: Standby manager daemon smithi062.rqxuxi started 2024-04-20T21:30:47.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:46 smithi189 ceph-mon[21562]: from='mgr.? 172.21.15.62:0/875003493' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi062.rqxuxi/crt"}]: dispatch 2024-04-20T21:30:47.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:46 smithi189 ceph-mon[21562]: from='mgr.? 172.21.15.62:0/875003493' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-20T21:30:47.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:46 smithi189 ceph-mon[21562]: from='mgr.? 172.21.15.62:0/875003493' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi062.rqxuxi/key"}]: dispatch 2024-04-20T21:30:47.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:46 smithi189 ceph-mon[21562]: from='mgr.? 172.21.15.62:0/875003493' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-20T21:30:47.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:46 smithi062 ceph-mon[16221]: pgmap v8: 65 pgs: 65 active+clean; 94 MiB data, 474 MiB used, 536 GiB / 536 GiB avail; 604 KiB/s rd, 2.0 MiB/s wr, 114 op/s 2024-04-20T21:30:47.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:46 smithi062 ceph-mon[16221]: Standby manager daemon smithi062.rqxuxi started 2024-04-20T21:30:47.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:46 smithi062 ceph-mon[16221]: from='mgr.? 172.21.15.62:0/875003493' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi062.rqxuxi/crt"}]: dispatch 2024-04-20T21:30:47.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:46 smithi062 ceph-mon[16221]: from='mgr.? 172.21.15.62:0/875003493' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-20T21:30:47.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:46 smithi062 ceph-mon[16221]: from='mgr.? 172.21.15.62:0/875003493' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi062.rqxuxi/key"}]: dispatch 2024-04-20T21:30:47.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:46 smithi062 ceph-mon[16221]: from='mgr.? 172.21.15.62:0/875003493' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-20T21:30:48.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:47 smithi189 ceph-mon[21562]: from='mgr.24379 172.21.15.189:0/1768059451' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "mgr metadata", "who": "smithi062.rqxuxi", "id": "smithi062.rqxuxi"}]: dispatch 2024-04-20T21:30:48.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:47 smithi189 ceph-mon[21562]: mgrmap e24: smithi189.tawavz(active, since 11s), standbys: smithi062.rqxuxi 2024-04-20T21:30:48.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:47 smithi189 ceph-mon[21562]: from='mgr.24379 ' entity='mgr.smithi189.tawavz' 2024-04-20T21:30:48.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:47 smithi189 ceph-mon[21562]: from='mgr.24379 ' entity='mgr.smithi189.tawavz' 2024-04-20T21:30:48.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:47 smithi189 ceph-mon[21562]: pgmap v9: 65 pgs: 65 active+clean; 103 MiB data, 515 MiB used, 536 GiB / 536 GiB avail; 1.2 MiB/s rd, 2.5 MiB/s wr, 134 op/s 2024-04-20T21:30:48.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:47 smithi062 ceph-mon[16221]: from='mgr.24379 172.21.15.189:0/1768059451' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "mgr metadata", "who": "smithi062.rqxuxi", "id": "smithi062.rqxuxi"}]: dispatch 2024-04-20T21:30:48.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:47 smithi062 ceph-mon[16221]: mgrmap e24: smithi189.tawavz(active, since 11s), standbys: smithi062.rqxuxi 2024-04-20T21:30:48.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:47 smithi062 ceph-mon[16221]: from='mgr.24379 ' entity='mgr.smithi189.tawavz' 2024-04-20T21:30:48.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:47 smithi062 ceph-mon[16221]: from='mgr.24379 ' entity='mgr.smithi189.tawavz' 2024-04-20T21:30:48.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:47 smithi062 ceph-mon[16221]: pgmap v9: 65 pgs: 65 active+clean; 103 MiB data, 515 MiB used, 536 GiB / 536 GiB avail; 1.2 MiB/s rd, 2.5 MiB/s wr, 134 op/s 2024-04-20T21:30:50.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:50 smithi189 ceph-mon[21562]: pgmap v10: 65 pgs: 65 active+clean; 103 MiB data, 515 MiB used, 536 GiB / 536 GiB avail; 1.1 MiB/s rd, 2.3 MiB/s wr, 123 op/s 2024-04-20T21:30:50.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:50 smithi189 ceph-mon[21562]: from='mgr.24379 ' entity='mgr.smithi189.tawavz' 2024-04-20T21:30:50.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:50 smithi189 ceph-mon[21562]: from='mgr.24379 ' entity='mgr.smithi189.tawavz' 2024-04-20T21:30:50.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:50 smithi189 ceph-mon[21562]: from='mgr.24379 172.21.15.189:0/1768059451' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:30:50.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:50 smithi062 ceph-mon[16221]: pgmap v10: 65 pgs: 65 active+clean; 103 MiB data, 515 MiB used, 536 GiB / 536 GiB avail; 1.1 MiB/s rd, 2.3 MiB/s wr, 123 op/s 2024-04-20T21:30:50.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:50 smithi062 ceph-mon[16221]: from='mgr.24379 ' entity='mgr.smithi189.tawavz' 2024-04-20T21:30:50.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:50 smithi062 ceph-mon[16221]: from='mgr.24379 ' entity='mgr.smithi189.tawavz' 2024-04-20T21:30:50.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:50 smithi062 ceph-mon[16221]: from='mgr.24379 172.21.15.189:0/1768059451' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:30:53.172 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:52 smithi062 ceph-mon[16221]: pgmap v11: 65 pgs: 65 active+clean; 103 MiB data, 520 MiB used, 536 GiB / 536 GiB avail; 1.1 MiB/s rd, 2.3 MiB/s wr, 125 op/s 2024-04-20T21:30:53.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:52 smithi189 ceph-mon[21562]: pgmap v11: 65 pgs: 65 active+clean; 103 MiB data, 520 MiB used, 536 GiB / 536 GiB avail; 1.1 MiB/s rd, 2.3 MiB/s wr, 125 op/s 2024-04-20T21:30:55.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:54 smithi189 ceph-mon[21562]: pgmap v12: 65 pgs: 65 active+clean; 109 MiB data, 531 MiB used, 536 GiB / 536 GiB avail; 1.4 MiB/s rd, 2.8 MiB/s wr, 140 op/s 2024-04-20T21:30:55.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:54 smithi189 ceph-mon[21562]: from='mgr.24379 ' entity='mgr.smithi189.tawavz' 2024-04-20T21:30:55.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:54 smithi189 ceph-mon[21562]: from='mgr.24379 ' entity='mgr.smithi189.tawavz' 2024-04-20T21:30:55.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:54 smithi189 ceph-mon[21562]: from='mgr.24379 ' entity='mgr.smithi189.tawavz' 2024-04-20T21:30:55.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:54 smithi189 ceph-mon[21562]: from='mgr.24379 ' entity='mgr.smithi189.tawavz' 2024-04-20T21:30:55.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:54 smithi189 ceph-mon[21562]: from='mgr.24379 172.21.15.189:0/1768059451' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-20T21:30:55.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:54 smithi189 ceph-mon[21562]: from='mgr.24379 ' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-20T21:30:55.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:54 smithi189 ceph-mon[21562]: from='mgr.24379 172.21.15.189:0/1768059451' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-20T21:30:55.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:54 smithi189 ceph-mon[21562]: from='mgr.24379 ' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-20T21:30:55.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:54 smithi189 ceph-mon[21562]: from='mgr.24379 172.21.15.189:0/1768059451' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-20T21:30:55.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:54 smithi189 ceph-mon[21562]: from='mgr.24379 ' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-20T21:30:55.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:54 smithi189 ceph-mon[21562]: from='mgr.24379 ' entity='mgr.smithi189.tawavz' 2024-04-20T21:30:55.284 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:54 smithi062 ceph-mon[16221]: pgmap v12: 65 pgs: 65 active+clean; 109 MiB data, 531 MiB used, 536 GiB / 536 GiB avail; 1.4 MiB/s rd, 2.8 MiB/s wr, 140 op/s 2024-04-20T21:30:55.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:54 smithi062 ceph-mon[16221]: from='mgr.24379 ' entity='mgr.smithi189.tawavz' 2024-04-20T21:30:55.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:54 smithi062 ceph-mon[16221]: from='mgr.24379 ' entity='mgr.smithi189.tawavz' 2024-04-20T21:30:55.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:54 smithi062 ceph-mon[16221]: from='mgr.24379 ' entity='mgr.smithi189.tawavz' 2024-04-20T21:30:55.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:54 smithi062 ceph-mon[16221]: from='mgr.24379 ' entity='mgr.smithi189.tawavz' 2024-04-20T21:30:55.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:54 smithi062 ceph-mon[16221]: from='mgr.24379 172.21.15.189:0/1768059451' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-20T21:30:55.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:54 smithi062 ceph-mon[16221]: from='mgr.24379 ' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-20T21:30:55.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:54 smithi062 ceph-mon[16221]: from='mgr.24379 172.21.15.189:0/1768059451' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-20T21:30:55.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:54 smithi062 ceph-mon[16221]: from='mgr.24379 ' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-20T21:30:55.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:54 smithi062 ceph-mon[16221]: from='mgr.24379 172.21.15.189:0/1768059451' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-20T21:30:55.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:54 smithi062 ceph-mon[16221]: from='mgr.24379 ' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-20T21:30:55.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:54 smithi062 ceph-mon[16221]: from='mgr.24379 ' entity='mgr.smithi189.tawavz' 2024-04-20T21:30:56.185 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:55 smithi062 ceph-mon[16221]: Adjusting osd_memory_target on smithi189 to 2240M 2024-04-20T21:30:56.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:55 smithi189 ceph-mon[21562]: Adjusting osd_memory_target on smithi189 to 2240M 2024-04-20T21:30:57.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:56 smithi062 ceph-mon[16221]: pgmap v13: 65 pgs: 65 active+clean; 109 MiB data, 531 MiB used, 536 GiB / 536 GiB avail; 954 KiB/s rd, 1.3 MiB/s wr, 55 op/s 2024-04-20T21:30:57.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:56 smithi062 ceph-mon[16221]: from='mgr.24379 ' entity='mgr.smithi189.tawavz' 2024-04-20T21:30:57.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:56 smithi062 ceph-mon[16221]: from='mgr.24379 ' entity='mgr.smithi189.tawavz' 2024-04-20T21:30:57.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:56 smithi062 ceph-mon[16221]: from='mgr.24379 172.21.15.189:0/1768059451' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-20T21:30:57.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:56 smithi062 ceph-mon[16221]: from='mgr.24379 ' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-20T21:30:57.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:56 smithi062 ceph-mon[16221]: from='mgr.24379 172.21.15.189:0/1768059451' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-20T21:30:57.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:56 smithi062 ceph-mon[16221]: from='mgr.24379 ' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-20T21:30:57.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:56 smithi062 ceph-mon[16221]: from='mgr.24379 172.21.15.189:0/1768059451' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-20T21:30:57.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:56 smithi062 ceph-mon[16221]: from='mgr.24379 ' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-20T21:30:57.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:56 smithi062 ceph-mon[16221]: Adjusting osd_memory_target on smithi062 to 1216M 2024-04-20T21:30:57.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:56 smithi062 ceph-mon[16221]: from='mgr.24379 ' entity='mgr.smithi189.tawavz' 2024-04-20T21:30:57.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:56 smithi062 ceph-mon[16221]: from='mgr.24379 172.21.15.189:0/1768059451' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:30:57.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:56 smithi062 ceph-mon[16221]: from='mgr.24379 172.21.15.189:0/1768059451' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T21:30:57.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:56 smithi062 ceph-mon[16221]: Updating smithi062:/etc/ceph/ceph.conf 2024-04-20T21:30:57.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:56 smithi062 ceph-mon[16221]: Updating smithi189:/etc/ceph/ceph.conf 2024-04-20T21:30:57.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:56 smithi189 ceph-mon[21562]: pgmap v13: 65 pgs: 65 active+clean; 109 MiB data, 531 MiB used, 536 GiB / 536 GiB avail; 954 KiB/s rd, 1.3 MiB/s wr, 55 op/s 2024-04-20T21:30:57.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:56 smithi189 ceph-mon[21562]: from='mgr.24379 ' entity='mgr.smithi189.tawavz' 2024-04-20T21:30:57.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:56 smithi189 ceph-mon[21562]: from='mgr.24379 ' entity='mgr.smithi189.tawavz' 2024-04-20T21:30:57.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:56 smithi189 ceph-mon[21562]: from='mgr.24379 172.21.15.189:0/1768059451' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-20T21:30:57.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:56 smithi189 ceph-mon[21562]: from='mgr.24379 ' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-20T21:30:57.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:56 smithi189 ceph-mon[21562]: from='mgr.24379 172.21.15.189:0/1768059451' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-20T21:30:57.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:56 smithi189 ceph-mon[21562]: from='mgr.24379 ' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-20T21:30:57.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:56 smithi189 ceph-mon[21562]: from='mgr.24379 172.21.15.189:0/1768059451' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-20T21:30:57.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:56 smithi189 ceph-mon[21562]: from='mgr.24379 ' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-20T21:30:57.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:56 smithi189 ceph-mon[21562]: Adjusting osd_memory_target on smithi062 to 1216M 2024-04-20T21:30:57.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:56 smithi189 ceph-mon[21562]: from='mgr.24379 ' entity='mgr.smithi189.tawavz' 2024-04-20T21:30:57.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:56 smithi189 ceph-mon[21562]: from='mgr.24379 172.21.15.189:0/1768059451' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:30:57.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:56 smithi189 ceph-mon[21562]: from='mgr.24379 172.21.15.189:0/1768059451' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T21:30:57.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:56 smithi189 ceph-mon[21562]: Updating smithi062:/etc/ceph/ceph.conf 2024-04-20T21:30:57.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:56 smithi189 ceph-mon[21562]: Updating smithi189:/etc/ceph/ceph.conf 2024-04-20T21:30:58.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:57 smithi189 ceph-mon[21562]: pgmap v14: 65 pgs: 65 active+clean; 116 MiB data, 563 MiB used, 536 GiB / 536 GiB avail; 1.6 MiB/s rd, 1.9 MiB/s wr, 100 op/s 2024-04-20T21:30:58.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:57 smithi189 ceph-mon[21562]: Updating smithi189:/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-20T21:30:58.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:57 smithi189 ceph-mon[21562]: Updating smithi062:/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-20T21:30:58.284 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:57 smithi062 ceph-mon[16221]: pgmap v14: 65 pgs: 65 active+clean; 116 MiB data, 563 MiB used, 536 GiB / 536 GiB avail; 1.6 MiB/s rd, 1.9 MiB/s wr, 100 op/s 2024-04-20T21:30:58.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:57 smithi062 ceph-mon[16221]: Updating smithi189:/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-20T21:30:58.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:57 smithi062 ceph-mon[16221]: Updating smithi062:/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-20T21:30:59.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:58 smithi189 ceph-mon[21562]: Updating smithi189:/etc/ceph/ceph.client.admin.keyring 2024-04-20T21:30:59.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:58 smithi189 ceph-mon[21562]: Updating smithi062:/etc/ceph/ceph.client.admin.keyring 2024-04-20T21:30:59.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:58 smithi189 ceph-mon[21562]: Updating smithi189:/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-20T21:30:59.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:30:58 smithi189 ceph-mon[21562]: Updating smithi062:/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-20T21:30:59.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:58 smithi062 ceph-mon[16221]: Updating smithi189:/etc/ceph/ceph.client.admin.keyring 2024-04-20T21:30:59.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:58 smithi062 ceph-mon[16221]: Updating smithi062:/etc/ceph/ceph.client.admin.keyring 2024-04-20T21:30:59.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:58 smithi062 ceph-mon[16221]: Updating smithi189:/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-20T21:30:59.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:30:58 smithi062 ceph-mon[16221]: Updating smithi062:/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-20T21:31:00.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:00 smithi062 ceph-mon[16221]: pgmap v15: 65 pgs: 65 active+clean; 116 MiB data, 563 MiB used, 536 GiB / 536 GiB avail; 968 KiB/s rd, 1.1 MiB/s wr, 63 op/s 2024-04-20T21:31:00.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:00 smithi062 ceph-mon[16221]: from='mgr.24379 ' entity='mgr.smithi189.tawavz' 2024-04-20T21:31:00.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:00 smithi062 ceph-mon[16221]: from='mgr.24379 ' entity='mgr.smithi189.tawavz' 2024-04-20T21:31:00.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:00 smithi062 ceph-mon[16221]: from='mgr.24379 ' entity='mgr.smithi189.tawavz' 2024-04-20T21:31:00.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:00 smithi062 ceph-mon[16221]: from='mgr.24379 ' entity='mgr.smithi189.tawavz' 2024-04-20T21:31:00.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:00 smithi062 ceph-mon[16221]: from='mgr.24379 ' entity='mgr.smithi189.tawavz' 2024-04-20T21:31:00.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:00 smithi062 ceph-mon[16221]: from='mgr.24379 ' entity='mgr.smithi189.tawavz' 2024-04-20T21:31:00.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:00 smithi062 ceph-mon[16221]: from='mgr.24379 ' entity='mgr.smithi189.tawavz' 2024-04-20T21:31:00.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:00 smithi062 ceph-mon[16221]: from='mgr.24379 ' entity='mgr.smithi189.tawavz' 2024-04-20T21:31:00.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:00 smithi062 ceph-mon[16221]: from='mgr.24379 ' entity='mgr.smithi189.tawavz' 2024-04-20T21:31:00.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:00 smithi062 ceph-mon[16221]: Reconfiguring prometheus.smithi062 (dependencies changed)... 2024-04-20T21:31:00.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:00 smithi189 ceph-mon[21562]: pgmap v15: 65 pgs: 65 active+clean; 116 MiB data, 563 MiB used, 536 GiB / 536 GiB avail; 968 KiB/s rd, 1.1 MiB/s wr, 63 op/s 2024-04-20T21:31:00.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:00 smithi189 ceph-mon[21562]: from='mgr.24379 ' entity='mgr.smithi189.tawavz' 2024-04-20T21:31:00.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:00 smithi189 ceph-mon[21562]: from='mgr.24379 ' entity='mgr.smithi189.tawavz' 2024-04-20T21:31:00.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:00 smithi189 ceph-mon[21562]: from='mgr.24379 ' entity='mgr.smithi189.tawavz' 2024-04-20T21:31:00.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:00 smithi189 ceph-mon[21562]: from='mgr.24379 ' entity='mgr.smithi189.tawavz' 2024-04-20T21:31:00.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:00 smithi189 ceph-mon[21562]: from='mgr.24379 ' entity='mgr.smithi189.tawavz' 2024-04-20T21:31:00.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:00 smithi189 ceph-mon[21562]: from='mgr.24379 ' entity='mgr.smithi189.tawavz' 2024-04-20T21:31:00.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:00 smithi189 ceph-mon[21562]: from='mgr.24379 ' entity='mgr.smithi189.tawavz' 2024-04-20T21:31:00.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:00 smithi189 ceph-mon[21562]: from='mgr.24379 ' entity='mgr.smithi189.tawavz' 2024-04-20T21:31:00.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:00 smithi189 ceph-mon[21562]: from='mgr.24379 ' entity='mgr.smithi189.tawavz' 2024-04-20T21:31:00.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:00 smithi189 ceph-mon[21562]: Reconfiguring prometheus.smithi062 (dependencies changed)... 2024-04-20T21:31:01.751 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:01 smithi062 ceph-mon[16221]: Reconfiguring daemon prometheus.smithi062 on smithi062 2024-04-20T21:31:01.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:01 smithi189 ceph-mon[21562]: Reconfiguring daemon prometheus.smithi062 on smithi062 2024-04-20T21:31:03.193 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:02 smithi062 ceph-mon[16221]: pgmap v16: 65 pgs: 65 active+clean; 117 MiB data, 568 MiB used, 536 GiB / 536 GiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 70 op/s 2024-04-20T21:31:03.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:02 smithi189 ceph-mon[21562]: pgmap v16: 65 pgs: 65 active+clean; 117 MiB data, 568 MiB used, 536 GiB / 536 GiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 70 op/s 2024-04-20T21:31:05.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:04 smithi062 ceph-mon[16221]: pgmap v17: 65 pgs: 65 active+clean; 124 MiB data, 587 MiB used, 536 GiB / 536 GiB avail; 1.5 MiB/s rd, 1.7 MiB/s wr, 100 op/s 2024-04-20T21:31:05.236 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:04 smithi189 ceph-mon[21562]: pgmap v17: 65 pgs: 65 active+clean; 124 MiB data, 587 MiB used, 536 GiB / 536 GiB avail; 1.5 MiB/s rd, 1.7 MiB/s wr, 100 op/s 2024-04-20T21:31:05.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:05.299+0000 7feb0d000700 1 -- 172.21.15.62:0/2329945980 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb08106860 msgr2=0x7feb08106c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:31:05.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:05.299+0000 7feb0d000700 1 --2- 172.21.15.62:0/2329945980 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb08106860 0x7feb08106c70 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7feaf0009910 tx=0x7feaf0009c20 comp rx=0 tx=0).stop 2024-04-20T21:31:05.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:05.299+0000 7feb0d000700 1 -- 172.21.15.62:0/2329945980 shutdown_connections 2024-04-20T21:31:05.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:05.299+0000 7feb0d000700 1 --2- 172.21.15.62:0/2329945980 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb08107a60 0x7feb08107eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:31:05.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:05.299+0000 7feb0d000700 1 --2- 172.21.15.62:0/2329945980 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb08106860 0x7feb08106c70 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:31:05.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:05.299+0000 7feb0d000700 1 -- 172.21.15.62:0/2329945980 >> 172.21.15.62:0/2329945980 conn(0x7feb08069b40 msgr2=0x7feb08069f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:31:05.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:05.300+0000 7feb0d000700 1 -- 172.21.15.62:0/2329945980 shutdown_connections 2024-04-20T21:31:05.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:05.300+0000 7feb0d000700 1 -- 172.21.15.62:0/2329945980 wait complete. 2024-04-20T21:31:05.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:05.301+0000 7feb0d000700 1 Processor -- start 2024-04-20T21:31:05.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:05.302+0000 7feb0d000700 1 -- start start 2024-04-20T21:31:05.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:05.303+0000 7feb0d000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb08106860 0x7feb08071d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:31:05.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:05.303+0000 7feb0d000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb08107a60 0x7feb080722a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:31:05.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:05.304+0000 7feb0d000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb080728c0 con 0x7feb08107a60 2024-04-20T21:31:05.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:05.304+0000 7feb05e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb08106860 0x7feb08071d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:31:05.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:05.304+0000 7feb0d000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb081a9e90 con 0x7feb08106860 2024-04-20T21:31:05.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:05.304+0000 7feb05e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb08106860 0x7feb08071d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.62:33516/0 (socket says 172.21.15.62:33516) 2024-04-20T21:31:05.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:05.304+0000 7feb05e00700 1 -- 172.21.15.62:0/3919667233 learned_addr learned my addr 172.21.15.62:0/3919667233 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:31:05.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:05.304+0000 7feafd400700 1 --2- 172.21.15.62:0/3919667233 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb08107a60 0x7feb080722a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:31:05.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:05.305+0000 7feb05e00700 1 -- 172.21.15.62:0/3919667233 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb08107a60 msgr2=0x7feb080722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:31:05.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:05.305+0000 7feb05e00700 1 --2- 172.21.15.62:0/3919667233 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb08107a60 0x7feb080722a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:31:05.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:05.305+0000 7feb05e00700 1 -- 172.21.15.62:0/3919667233 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaf00095f0 con 0x7feb08106860 2024-04-20T21:31:05.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:05.307+0000 7feb05e00700 1 --2- 172.21.15.62:0/3919667233 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb08106860 0x7feb08071d60 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7feaf000bd70 tx=0x7feaf000bda0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:31:05.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:05.307+0000 7feaff400700 1 -- 172.21.15.62:0/3919667233 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feaf001d070 con 0x7feb08106860 2024-04-20T21:31:05.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:05.307+0000 7feaff400700 1 -- 172.21.15.62:0/3919667233 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7feaf0005dc0 con 0x7feb08106860 2024-04-20T21:31:05.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:05.307+0000 7feaff400700 1 -- 172.21.15.62:0/3919667233 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feaf0021410 con 0x7feb08106860 2024-04-20T21:31:05.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:05.307+0000 7feb0d000700 1 -- 172.21.15.62:0/3919667233 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb081aa030 con 0x7feb08106860 2024-04-20T21:31:05.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:05.307+0000 7feb0d000700 1 -- 172.21.15.62:0/3919667233 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb081aa4d0 con 0x7feb08106860 2024-04-20T21:31:05.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:05.309+0000 7feb0d000700 1 -- 172.21.15.62:0/3919667233 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feae4005320 con 0x7feb08106860 2024-04-20T21:31:05.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:05.311+0000 7feaff400700 1 -- 172.21.15.62:0/3919667233 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 24) v1 ==== 91360+0+0 (secure 0 0 0) 0x7feaf0003a60 con 0x7feb08106860 2024-04-20T21:31:05.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:05.313+0000 7feaff400700 1 --2- 172.21.15.62:0/3919667233 >> [v2:172.21.15.189:6828/2010473407,v1:172.21.15.189:6829/2010473407] conn(0x7feaec06d960 0x7feaec06fe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:31:05.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:05.313+0000 7feaff400700 1 -- 172.21.15.62:0/3919667233 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5726+0+0 (secure 0 0 0) 0x7feaf008e470 con 0x7feb08106860 2024-04-20T21:31:05.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:05.313+0000 7feafd400700 1 --2- 172.21.15.62:0/3919667233 >> [v2:172.21.15.189:6828/2010473407,v1:172.21.15.189:6829/2010473407] conn(0x7feaec06d960 0x7feaec06fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:31:05.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:05.315+0000 7feafd400700 1 --2- 172.21.15.62:0/3919667233 >> [v2:172.21.15.189:6828/2010473407,v1:172.21.15.189:6829/2010473407] conn(0x7feaec06d960 0x7feaec06fe10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7feb08072cc0 tx=0x7feaf800b410 comp rx=0 tx=0).ready entity=mgr.24379 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:31:05.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:05.329+0000 7feaff400700 1 -- 172.21.15.62:0/3919667233 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187527 (secure 0 0 0) 0x7feaf005b220 con 0x7feb08106860 2024-04-20T21:31:05.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:05.664+0000 7feb0d000700 1 -- 172.21.15.62:0/3919667233 --> [v2:172.21.15.189:6828/2010473407,v1:172.21.15.189:6829/2010473407] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7feae4000bf0 con 0x7feaec06d960 2024-04-20T21:31:05.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:05.668+0000 7feaff400700 1 -- 172.21.15.62:0/3919667233 <== mgr.24379 v2:172.21.15.189:6828/2010473407 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+310 (secure 0 0 0) 0x7feae4000bf0 con 0x7feaec06d960 2024-04-20T21:31:05.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:05.671+0000 7feb0d000700 1 -- 172.21.15.62:0/3919667233 >> [v2:172.21.15.189:6828/2010473407,v1:172.21.15.189:6829/2010473407] conn(0x7feaec06d960 msgr2=0x7feaec06fe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:31:05.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:05.671+0000 7feb0d000700 1 --2- 172.21.15.62:0/3919667233 >> [v2:172.21.15.189:6828/2010473407,v1:172.21.15.189:6829/2010473407] conn(0x7feaec06d960 0x7feaec06fe10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7feb08072cc0 tx=0x7feaf800b410 comp rx=0 tx=0).stop 2024-04-20T21:31:05.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:05.671+0000 7feb0d000700 1 -- 172.21.15.62:0/3919667233 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb08106860 msgr2=0x7feb08071d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:31:05.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:05.671+0000 7feb0d000700 1 --2- 172.21.15.62:0/3919667233 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb08106860 0x7feb08071d60 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7feaf000bd70 tx=0x7feaf000bda0 comp rx=0 tx=0).stop 2024-04-20T21:31:05.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:05.671+0000 7feb0d000700 1 -- 172.21.15.62:0/3919667233 shutdown_connections 2024-04-20T21:31:05.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:05.671+0000 7feb0d000700 1 --2- 172.21.15.62:0/3919667233 >> [v2:172.21.15.189:6828/2010473407,v1:172.21.15.189:6829/2010473407] conn(0x7feaec06d960 0x7feaec06fe10 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:31:05.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:05.671+0000 7feb0d000700 1 --2- 172.21.15.62:0/3919667233 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb08106860 0x7feb08071d60 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:31:05.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:05.671+0000 7feb0d000700 1 --2- 172.21.15.62:0/3919667233 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb08107a60 0x7feb080722a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:31:05.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:05.671+0000 7feb0d000700 1 -- 172.21.15.62:0/3919667233 >> 172.21.15.62:0/3919667233 conn(0x7feb08069b40 msgr2=0x7feb0810ac90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:31:05.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:05.671+0000 7feb0d000700 1 -- 172.21.15.62:0/3919667233 shutdown_connections 2024-04-20T21:31:05.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:05.672+0000 7feb0d000700 1 -- 172.21.15.62:0/3919667233 wait complete. 2024-04-20T21:31:05.689 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T21:31:05.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:05.845+0000 7f7a27e00700 1 -- 172.21.15.62:0/3622607079 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a28102280 msgr2=0x7f7a28102690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:31:05.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:05.845+0000 7f7a27e00700 1 --2- 172.21.15.62:0/3622607079 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a28102280 0x7f7a28102690 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f7a1800b0a0 tx=0x7f7a1800b3b0 comp rx=0 tx=0).stop 2024-04-20T21:31:05.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:05.845+0000 7f7a27e00700 1 -- 172.21.15.62:0/3622607079 shutdown_connections 2024-04-20T21:31:05.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:05.845+0000 7f7a27e00700 1 --2- 172.21.15.62:0/3622607079 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7a28103480 0x7f7a281038d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:31:05.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:05.845+0000 7f7a27e00700 1 --2- 172.21.15.62:0/3622607079 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a28102280 0x7f7a28102690 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:31:05.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:05.845+0000 7f7a27e00700 1 -- 172.21.15.62:0/3622607079 >> 172.21.15.62:0/3622607079 conn(0x7f7a280fd8d0 msgr2=0x7f7a280ffcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:31:05.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:05.846+0000 7f7a27e00700 1 -- 172.21.15.62:0/3622607079 shutdown_connections 2024-04-20T21:31:05.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:05.846+0000 7f7a27e00700 1 -- 172.21.15.62:0/3622607079 wait complete. 2024-04-20T21:31:05.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:05.847+0000 7f7a27e00700 1 Processor -- start 2024-04-20T21:31:05.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:05.847+0000 7f7a27e00700 1 -- start start 2024-04-20T21:31:05.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:05.847+0000 7f7a27e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a28102280 0x7f7a28197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:31:05.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:05.847+0000 7f7a27e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7a28103480 0x7f7a28198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:31:05.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:05.847+0000 7f7a27e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a28198620 con 0x7f7a28102280 2024-04-20T21:31:05.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:05.847+0000 7f7a27e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a28198760 con 0x7f7a28103480 2024-04-20T21:31:05.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:05.848+0000 7f7a26a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a28102280 0x7f7a28197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:31:05.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:05.848+0000 7f7a26a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a28102280 0x7f7a28197ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42382/0 (socket says 172.21.15.62:42382) 2024-04-20T21:31:05.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:05.848+0000 7f7a26a00700 1 -- 172.21.15.62:0/1294057639 learned_addr learned my addr 172.21.15.62:0/1294057639 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:31:05.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:05.848+0000 7f7a26000700 1 --2- 172.21.15.62:0/1294057639 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7a28103480 0x7f7a28198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:31:05.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:05.848+0000 7f7a26a00700 1 -- 172.21.15.62:0/1294057639 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7a28103480 msgr2=0x7f7a28198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:31:05.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:05.848+0000 7f7a26a00700 1 --2- 172.21.15.62:0/1294057639 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7a28103480 0x7f7a28198000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:31:05.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:05.848+0000 7f7a26a00700 1 -- 172.21.15.62:0/1294057639 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a18009d00 con 0x7f7a28102280 2024-04-20T21:31:05.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:05.849+0000 7f7a26a00700 1 --2- 172.21.15.62:0/1294057639 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a28102280 0x7f7a28197ac0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f7a18009fd0 tx=0x7f7a18009790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:31:05.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:05.849+0000 7f7a2ca00700 1 -- 172.21.15.62:0/1294057639 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7a1800e040 con 0x7f7a28102280 2024-04-20T21:31:05.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:05.849+0000 7f7a2ca00700 1 -- 172.21.15.62:0/1294057639 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f7a18003de0 con 0x7f7a28102280 2024-04-20T21:31:05.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:05.849+0000 7f7a27e00700 1 -- 172.21.15.62:0/1294057639 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a2819d1b0 con 0x7f7a28102280 2024-04-20T21:31:05.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:05.849+0000 7f7a2ca00700 1 -- 172.21.15.62:0/1294057639 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7a1801d980 con 0x7f7a28102280 2024-04-20T21:31:05.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:05.849+0000 7f7a27e00700 1 -- 172.21.15.62:0/1294057639 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a2819d720 con 0x7f7a28102280 2024-04-20T21:31:05.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:05.852+0000 7f7a27e00700 1 -- 172.21.15.62:0/1294057639 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a2810add0 con 0x7f7a28102280 2024-04-20T21:31:05.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:05.852+0000 7f7a2ca00700 1 -- 172.21.15.62:0/1294057639 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 24) v1 ==== 91360+0+0 (secure 0 0 0) 0x7f7a18004040 con 0x7f7a28102280 2024-04-20T21:31:05.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:05.855+0000 7f7a2ca00700 1 --2- 172.21.15.62:0/1294057639 >> [v2:172.21.15.189:6828/2010473407,v1:172.21.15.189:6829/2010473407] conn(0x7f7a0c06d910 0x7f7a0c06fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:31:05.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:05.855+0000 7f7a2ca00700 1 -- 172.21.15.62:0/1294057639 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5726+0+0 (secure 0 0 0) 0x7f7a1808ed50 con 0x7f7a28102280 2024-04-20T21:31:05.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:05.856+0000 7f7a26000700 1 --2- 172.21.15.62:0/1294057639 >> [v2:172.21.15.189:6828/2010473407,v1:172.21.15.189:6829/2010473407] conn(0x7f7a0c06d910 0x7f7a0c06fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:31:05.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:05.857+0000 7f7a26000700 1 --2- 172.21.15.62:0/1294057639 >> [v2:172.21.15.189:6828/2010473407,v1:172.21.15.189:6829/2010473407] conn(0x7f7a0c06d910 0x7f7a0c06fdc0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f7a14005950 tx=0x7f7a140058e0 comp rx=0 tx=0).ready entity=mgr.24379 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:31:05.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:05.860+0000 7f7a2ca00700 1 -- 172.21.15.62:0/1294057639 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187527 (secure 0 0 0) 0x7f7a1805ba00 con 0x7f7a28102280 2024-04-20T21:31:06.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:06.189+0000 7f7a27e00700 1 -- 172.21.15.62:0/1294057639 --> [v2:172.21.15.189:6828/2010473407,v1:172.21.15.189:6829/2010473407] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7a2819dd60 con 0x7f7a0c06d910 2024-04-20T21:31:06.189 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:05 smithi062 ceph-mon[16221]: from='mgr.24379 172.21.15.189:0/1768059451' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:31:06.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:06.192+0000 7f7a2ca00700 1 -- 172.21.15.62:0/1294057639 <== mgr.24379 v2:172.21.15.189:6828/2010473407 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+310 (secure 0 0 0) 0x7f7a2819dd60 con 0x7f7a0c06d910 2024-04-20T21:31:06.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:06.197+0000 7f7a27e00700 1 -- 172.21.15.62:0/1294057639 >> [v2:172.21.15.189:6828/2010473407,v1:172.21.15.189:6829/2010473407] conn(0x7f7a0c06d910 msgr2=0x7f7a0c06fdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:31:06.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:06.197+0000 7f7a27e00700 1 --2- 172.21.15.62:0/1294057639 >> [v2:172.21.15.189:6828/2010473407,v1:172.21.15.189:6829/2010473407] conn(0x7f7a0c06d910 0x7f7a0c06fdc0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f7a14005950 tx=0x7f7a140058e0 comp rx=0 tx=0).stop 2024-04-20T21:31:06.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:06.197+0000 7f7a27e00700 1 -- 172.21.15.62:0/1294057639 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a28102280 msgr2=0x7f7a28197ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:31:06.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:06.197+0000 7f7a27e00700 1 --2- 172.21.15.62:0/1294057639 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a28102280 0x7f7a28197ac0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f7a18009fd0 tx=0x7f7a18009790 comp rx=0 tx=0).stop 2024-04-20T21:31:06.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:06.197+0000 7f7a27e00700 1 -- 172.21.15.62:0/1294057639 shutdown_connections 2024-04-20T21:31:06.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:06.197+0000 7f7a27e00700 1 --2- 172.21.15.62:0/1294057639 >> [v2:172.21.15.189:6828/2010473407,v1:172.21.15.189:6829/2010473407] conn(0x7f7a0c06d910 0x7f7a0c06fdc0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:31:06.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:06.197+0000 7f7a27e00700 1 --2- 172.21.15.62:0/1294057639 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a28102280 0x7f7a28197ac0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:31:06.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:06.197+0000 7f7a27e00700 1 --2- 172.21.15.62:0/1294057639 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7a28103480 0x7f7a28198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:31:06.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:06.197+0000 7f7a27e00700 1 -- 172.21.15.62:0/1294057639 >> 172.21.15.62:0/1294057639 conn(0x7f7a280fd8d0 msgr2=0x7f7a281066b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:31:06.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:06.197+0000 7f7a27e00700 1 -- 172.21.15.62:0/1294057639 shutdown_connections 2024-04-20T21:31:06.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:06.197+0000 7f7a27e00700 1 -- 172.21.15.62:0/1294057639 wait complete. 2024-04-20T21:31:06.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:05 smithi189 ceph-mon[21562]: from='mgr.24379 172.21.15.189:0/1768059451' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:31:06.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:06.398+0000 7fae10e00700 1 -- 172.21.15.62:0/3933661778 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fae0c101440 msgr2=0x7fae0c101890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:31:06.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:06.398+0000 7fae10e00700 1 --2- 172.21.15.62:0/3933661778 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fae0c101440 0x7fae0c101890 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fadf8009910 tx=0x7fadf8009c20 comp rx=0 tx=0).stop 2024-04-20T21:31:06.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:06.399+0000 7fae10e00700 1 -- 172.21.15.62:0/3933661778 shutdown_connections 2024-04-20T21:31:06.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:06.399+0000 7fae10e00700 1 --2- 172.21.15.62:0/3933661778 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fae0c101440 0x7fae0c101890 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:31:06.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:06.399+0000 7fae10e00700 1 --2- 172.21.15.62:0/3933661778 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae0c100240 0x7fae0c100650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:31:06.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:06.399+0000 7fae10e00700 1 -- 172.21.15.62:0/3933661778 >> 172.21.15.62:0/3933661778 conn(0x7fae0c0fb7f0 msgr2=0x7fae0c0fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:31:06.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:06.400+0000 7fae10e00700 1 -- 172.21.15.62:0/3933661778 shutdown_connections 2024-04-20T21:31:06.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:06.400+0000 7fae10e00700 1 -- 172.21.15.62:0/3933661778 wait complete. 2024-04-20T21:31:06.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:06.401+0000 7fae10e00700 1 Processor -- start 2024-04-20T21:31:06.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:06.402+0000 7fae10e00700 1 -- start start 2024-04-20T21:31:06.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:06.402+0000 7fae10e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae0c100240 0x7fae0c074af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:31:06.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:06.402+0000 7fae10e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fae0c101440 0x7fae0c073140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:31:06.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:06.402+0000 7fae10e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae0c073680 con 0x7fae0c101440 2024-04-20T21:31:06.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:06.402+0000 7fae10e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae0c0737c0 con 0x7fae0c100240 2024-04-20T21:31:06.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:06.403+0000 7fae09e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae0c100240 0x7fae0c074af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:31:06.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:06.403+0000 7fae09e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae0c100240 0x7fae0c074af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.62:33554/0 (socket says 172.21.15.62:33554) 2024-04-20T21:31:06.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:06.403+0000 7fae09e00700 1 -- 172.21.15.62:0/321169032 learned_addr learned my addr 172.21.15.62:0/321169032 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:31:06.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:06.403+0000 7fae09400700 1 --2- 172.21.15.62:0/321169032 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fae0c101440 0x7fae0c073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:31:06.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:06.404+0000 7fae09400700 1 -- 172.21.15.62:0/321169032 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae0c100240 msgr2=0x7fae0c074af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:31:06.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:06.404+0000 7fae09400700 1 --2- 172.21.15.62:0/321169032 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae0c100240 0x7fae0c074af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:31:06.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:06.404+0000 7fae09400700 1 -- 172.21.15.62:0/321169032 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadf80095f0 con 0x7fae0c101440 2024-04-20T21:31:06.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:06.404+0000 7fae09400700 1 --2- 172.21.15.62:0/321169032 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fae0c101440 0x7fae0c073140 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fadf8000c00 tx=0x7fadf800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:31:06.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:06.405+0000 7fae02a00700 1 -- 172.21.15.62:0/321169032 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fadf801d070 con 0x7fae0c101440 2024-04-20T21:31:06.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:06.405+0000 7fae02a00700 1 -- 172.21.15.62:0/321169032 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fadf8005dc0 con 0x7fae0c101440 2024-04-20T21:31:06.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:06.405+0000 7fae10e00700 1 -- 172.21.15.62:0/321169032 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae0c073a40 con 0x7fae0c101440 2024-04-20T21:31:06.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:06.405+0000 7fae02a00700 1 -- 172.21.15.62:0/321169032 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fadf8021410 con 0x7fae0c101440 2024-04-20T21:31:06.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:06.405+0000 7fae10e00700 1 -- 172.21.15.62:0/321169032 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae0c073f30 con 0x7fae0c101440 2024-04-20T21:31:06.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:06.406+0000 7fae10e00700 1 -- 172.21.15.62:0/321169032 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae0c108d90 con 0x7fae0c101440 2024-04-20T21:31:06.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:06.408+0000 7fae02a00700 1 -- 172.21.15.62:0/321169032 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 24) v1 ==== 91360+0+0 (secure 0 0 0) 0x7fadf800f7d0 con 0x7fae0c101440 2024-04-20T21:31:06.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:06.408+0000 7fae02a00700 1 --2- 172.21.15.62:0/321169032 >> [v2:172.21.15.189:6828/2010473407,v1:172.21.15.189:6829/2010473407] conn(0x7fadf006d910 0x7fadf006fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:31:06.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:06.409+0000 7fae02a00700 1 -- 172.21.15.62:0/321169032 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5726+0+0 (secure 0 0 0) 0x7fadf808e410 con 0x7fae0c101440 2024-04-20T21:31:06.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:06.409+0000 7fae09e00700 1 --2- 172.21.15.62:0/321169032 >> [v2:172.21.15.189:6828/2010473407,v1:172.21.15.189:6829/2010473407] conn(0x7fadf006d910 0x7fadf006fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:31:06.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:06.411+0000 7fae09e00700 1 --2- 172.21.15.62:0/321169032 >> [v2:172.21.15.189:6828/2010473407,v1:172.21.15.189:6829/2010473407] conn(0x7fadf006d910 0x7fadf006fdc0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fadfc00bd70 tx=0x7fadfc00b4a0 comp rx=0 tx=0).ready entity=mgr.24379 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:31:06.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:06.414+0000 7fae02a00700 1 -- 172.21.15.62:0/321169032 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187527 (secure 0 0 0) 0x7fadf805b1e0 con 0x7fae0c101440 2024-04-20T21:31:06.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:06.718+0000 7fae10e00700 1 -- 172.21.15.62:0/321169032 --> [v2:172.21.15.189:6828/2010473407,v1:172.21.15.189:6829/2010473407] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fae0c19dff0 con 0x7fadf006d910 2024-04-20T21:31:06.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:06.761+0000 7fae02a00700 1 -- 172.21.15.62:0/321169032 <== mgr.24379 v2:172.21.15.189:6828/2010473407 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fae0c19dff0 con 0x7fadf006d910 2024-04-20T21:31:06.761 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T21:31:06.761 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (8m) 12s ago 10m 13.9M - 0.25.0 c8568f914cd2 957f663a9cd7 2024-04-20T21:31:06.761 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (11m) 12s ago 11m 7856k - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T21:31:06.761 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (9m) 19s ago 9m 7684k - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T21:31:06.761 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (11m) 12s ago 11m 6656k - 18.2.0 dc2bc1663786 36e10ef5be39 2024-04-20T21:31:06.761 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (9m) 19s ago 9m 6665k - 18.2.0 dc2bc1663786 4101f9953e3d 2024-04-20T21:31:06.761 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (8m) 12s ago 10m 75.5M - 9.4.7 954c08fa6188 99363bb095d8 2024-04-20T21:31:06.762 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (3m) 12s ago 3m 70.7M - 18.2.0 dc2bc1663786 5e6460aa6159 2024-04-20T21:31:06.762 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (3m) 12s ago 3m 74.9M - 18.2.0 dc2bc1663786 e65f2267544e 2024-04-20T21:31:06.762 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (3m) 19s ago 3m 73.3M - 18.2.0 dc2bc1663786 fb2a8a223db8 2024-04-20T21:31:06.762 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (3m) 19s ago 3m 53.0M - 18.2.0 dc2bc1663786 f2eec2ee8951 2024-04-20T21:31:06.762 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:9283,8765,8443 running (12m) 12s ago 12m 323M - 18.2.0 dc2bc1663786 e411b3dc7233 2024-04-20T21:31:06.762 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (50s) 19s ago 9m 488M - 19.0.0-3133-ga8c84f59 76bc6b91b726 4cde2bc4de24 2024-04-20T21:31:06.762 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (12m) 12s ago 12m 49.3M 2048M 18.2.0 dc2bc1663786 bd6efbd5c380 2024-04-20T21:31:06.762 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (9m) 19s ago 9m 40.4M 2048M 18.2.0 dc2bc1663786 3735c8e5db9a 2024-04-20T21:31:06.762 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (11m) 12s ago 11m 7902k - 1.5.0 0da6a335fe13 fc746d8ae72e 2024-04-20T21:31:06.762 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (9m) 19s ago 9m 8363k - 1.5.0 0da6a335fe13 db03af85b286 2024-04-20T21:31:06.762 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (8m) 12s ago 8m 68.4M 1216M 18.2.0 dc2bc1663786 3f315c3afc1f 2024-04-20T21:31:06.762 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (7m) 12s ago 7m 76.2M 1216M 18.2.0 dc2bc1663786 44e4d07416ad 2024-04-20T21:31:06.762 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (7m) 12s ago 7m 62.6M 1216M 18.2.0 dc2bc1663786 31c5d9586e98 2024-04-20T21:31:06.762 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (6m) 19s ago 6m 73.6M 2240M 18.2.0 dc2bc1663786 611c1e242693 2024-04-20T21:31:06.763 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (6m) 19s ago 6m 68.5M 2240M 18.2.0 dc2bc1663786 6c8fce91d45d 2024-04-20T21:31:06.763 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (5m) 19s ago 5m 67.1M 2240M 18.2.0 dc2bc1663786 8e4574bae8db 2024-04-20T21:31:06.763 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (8m) 12s ago 10m 62.3M - 2.43.0 a07b618ecd1d cd390b7e44d0 2024-04-20T21:31:06.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:06.765+0000 7fae10e00700 1 -- 172.21.15.62:0/321169032 >> [v2:172.21.15.189:6828/2010473407,v1:172.21.15.189:6829/2010473407] conn(0x7fadf006d910 msgr2=0x7fadf006fdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:31:06.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:06.765+0000 7fae10e00700 1 --2- 172.21.15.62:0/321169032 >> [v2:172.21.15.189:6828/2010473407,v1:172.21.15.189:6829/2010473407] conn(0x7fadf006d910 0x7fadf006fdc0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fadfc00bd70 tx=0x7fadfc00b4a0 comp rx=0 tx=0).stop 2024-04-20T21:31:06.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:06.765+0000 7fae10e00700 1 -- 172.21.15.62:0/321169032 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fae0c101440 msgr2=0x7fae0c073140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:31:06.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:06.765+0000 7fae10e00700 1 --2- 172.21.15.62:0/321169032 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fae0c101440 0x7fae0c073140 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fadf8000c00 tx=0x7fadf800bd70 comp rx=0 tx=0).stop 2024-04-20T21:31:06.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:06.765+0000 7fae10e00700 1 -- 172.21.15.62:0/321169032 shutdown_connections 2024-04-20T21:31:06.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:06.765+0000 7fae10e00700 1 --2- 172.21.15.62:0/321169032 >> [v2:172.21.15.189:6828/2010473407,v1:172.21.15.189:6829/2010473407] conn(0x7fadf006d910 0x7fadf006fdc0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:31:06.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:06.765+0000 7fae10e00700 1 --2- 172.21.15.62:0/321169032 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae0c100240 0x7fae0c074af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:31:06.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:06.765+0000 7fae10e00700 1 --2- 172.21.15.62:0/321169032 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fae0c101440 0x7fae0c073140 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:31:06.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:06.765+0000 7fae10e00700 1 -- 172.21.15.62:0/321169032 >> 172.21.15.62:0/321169032 conn(0x7fae0c0fb7f0 msgr2=0x7fae0c104670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:31:06.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:06.765+0000 7fae10e00700 1 -- 172.21.15.62:0/321169032 shutdown_connections 2024-04-20T21:31:06.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:06.765+0000 7fae10e00700 1 -- 172.21.15.62:0/321169032 wait complete. 2024-04-20T21:31:06.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:06.936+0000 7fd4bf800700 1 -- 172.21.15.62:0/2070987165 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd4b8073770 msgr2=0x7fd4b8073be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:31:06.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:06.936+0000 7fd4bf800700 1 --2- 172.21.15.62:0/2070987165 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd4b8073770 0x7fd4b8073be0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fd4a4009910 tx=0x7fd4a4009c20 comp rx=0 tx=0).stop 2024-04-20T21:31:06.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:06.936+0000 7fd4bf800700 1 -- 172.21.15.62:0/2070987165 shutdown_connections 2024-04-20T21:31:06.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:06.936+0000 7fd4bf800700 1 --2- 172.21.15.62:0/2070987165 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd4b8073770 0x7fd4b8073be0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:31:06.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:06.936+0000 7fd4bf800700 1 --2- 172.21.15.62:0/2070987165 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd4b8074d40 0x7fd4b80731a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:31:06.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:06.936+0000 7fd4bf800700 1 -- 172.21.15.62:0/2070987165 >> 172.21.15.62:0/2070987165 conn(0x7fd4b80fb7f0 msgr2=0x7fd4b80fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:31:06.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:06.937+0000 7fd4bf800700 1 -- 172.21.15.62:0/2070987165 shutdown_connections 2024-04-20T21:31:06.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:06.937+0000 7fd4bf800700 1 -- 172.21.15.62:0/2070987165 wait complete. 2024-04-20T21:31:06.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:06.938+0000 7fd4bf800700 1 Processor -- start 2024-04-20T21:31:06.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:06.938+0000 7fd4bf800700 1 -- start start 2024-04-20T21:31:06.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:06.938+0000 7fd4bf800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd4b8073770 0x7fd4b819c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:31:06.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:06.939+0000 7fd4bf800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd4b8074d40 0x7fd4b819c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:31:06.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:06.939+0000 7fd4bf800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4b819cc10 con 0x7fd4b8074d40 2024-04-20T21:31:06.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:06.939+0000 7fd4bf800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4b819cd50 con 0x7fd4b8073770 2024-04-20T21:31:06.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:06.939+0000 7fd4b7e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd4b8074d40 0x7fd4b819c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:31:06.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:06.939+0000 7fd4b7e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd4b8074d40 0x7fd4b819c5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42408/0 (socket says 172.21.15.62:42408) 2024-04-20T21:31:06.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:06.939+0000 7fd4b7e00700 1 -- 172.21.15.62:0/3875751600 learned_addr learned my addr 172.21.15.62:0/3875751600 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:31:06.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:06.939+0000 7fd4bce00700 1 --2- 172.21.15.62:0/3875751600 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd4b8073770 0x7fd4b819c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:31:06.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:06.939+0000 7fd4b7e00700 1 -- 172.21.15.62:0/3875751600 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd4b8073770 msgr2=0x7fd4b819c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:31:06.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:06.939+0000 7fd4b7e00700 1 --2- 172.21.15.62:0/3875751600 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd4b8073770 0x7fd4b819c0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:31:06.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:06.939+0000 7fd4b7e00700 1 -- 172.21.15.62:0/3875751600 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4a40095f0 con 0x7fd4b8074d40 2024-04-20T21:31:06.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:06.940+0000 7fd4b7e00700 1 --2- 172.21.15.62:0/3875751600 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd4b8074d40 0x7fd4b819c5f0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fd4a400b5c0 tx=0x7fd4a400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:31:06.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:06.940+0000 7fd4b5600700 1 -- 172.21.15.62:0/3875751600 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd4a401d070 con 0x7fd4b8074d40 2024-04-20T21:31:06.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:06.940+0000 7fd4b5600700 1 -- 172.21.15.62:0/3875751600 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fd4a4022470 con 0x7fd4b8074d40 2024-04-20T21:31:06.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:06.940+0000 7fd4bf800700 1 -- 172.21.15.62:0/3875751600 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4b81a17a0 con 0x7fd4b8074d40 2024-04-20T21:31:06.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:06.940+0000 7fd4b5600700 1 -- 172.21.15.62:0/3875751600 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd4a400f690 con 0x7fd4b8074d40 2024-04-20T21:31:06.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:06.940+0000 7fd4bf800700 1 -- 172.21.15.62:0/3875751600 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4b81a1c90 con 0x7fd4b8074d40 2024-04-20T21:31:06.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:06.942+0000 7fd4bf800700 1 -- 172.21.15.62:0/3875751600 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4b8104b10 con 0x7fd4b8074d40 2024-04-20T21:31:06.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:06.943+0000 7fd4b5600700 1 -- 172.21.15.62:0/3875751600 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 24) v1 ==== 91360+0+0 (secure 0 0 0) 0x7fd4a4004b70 con 0x7fd4b8074d40 2024-04-20T21:31:06.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:06.945+0000 7fd4b5600700 1 --2- 172.21.15.62:0/3875751600 >> [v2:172.21.15.189:6828/2010473407,v1:172.21.15.189:6829/2010473407] conn(0x7fd4a006d910 0x7fd4a006fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:31:06.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:06.945+0000 7fd4b5600700 1 -- 172.21.15.62:0/3875751600 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5726+0+0 (secure 0 0 0) 0x7fd4a408e8a0 con 0x7fd4b8074d40 2024-04-20T21:31:06.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:06.946+0000 7fd4bce00700 1 --2- 172.21.15.62:0/3875751600 >> [v2:172.21.15.189:6828/2010473407,v1:172.21.15.189:6829/2010473407] conn(0x7fd4a006d910 0x7fd4a006fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:31:06.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:06.947+0000 7fd4bce00700 1 --2- 172.21.15.62:0/3875751600 >> [v2:172.21.15.189:6828/2010473407,v1:172.21.15.189:6829/2010473407] conn(0x7fd4a006d910 0x7fd4a006fdc0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fd4ac005fd0 tx=0x7fd4ac00a560 comp rx=0 tx=0).ready entity=mgr.24379 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:31:06.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:06.951+0000 7fd4b5600700 1 -- 172.21.15.62:0/3875751600 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187527 (secure 0 0 0) 0x7fd4a405b720 con 0x7fd4b8074d40 2024-04-20T21:31:07.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:06 smithi189 ceph-mon[21562]: pgmap v18: 65 pgs: 65 active+clean; 124 MiB data, 587 MiB used, 536 GiB / 536 GiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 84 op/s 2024-04-20T21:31:07.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:06 smithi189 ceph-mon[21562]: from='client.24427 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:31:07.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:06 smithi189 ceph-mon[21562]: from='client.14642 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:31:07.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:06 smithi189 ceph-mon[21562]: from='client.14646 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:31:07.284 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:06 smithi062 ceph-mon[16221]: pgmap v18: 65 pgs: 65 active+clean; 124 MiB data, 587 MiB used, 536 GiB / 536 GiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 84 op/s 2024-04-20T21:31:07.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:06 smithi062 ceph-mon[16221]: from='client.24427 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:31:07.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:06 smithi062 ceph-mon[16221]: from='client.14642 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:31:07.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:06 smithi062 ceph-mon[16221]: from='client.14646 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:31:07.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:07.373+0000 7fd4bf800700 1 -- 172.21.15.62:0/3875751600 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fd4b802ce40 con 0x7fd4b8074d40 2024-04-20T21:31:07.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:07.375+0000 7fd4b5600700 1 -- 172.21.15.62:0/3875751600 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+766 (secure 0 0 0) 0x7fd4a405b2b0 con 0x7fd4b8074d40 2024-04-20T21:31:07.374 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:31:07.374 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T21:31:07.375 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 2 2024-04-20T21:31:07.375 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:31:07.375 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T21:31:07.375 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 1, 2024-04-20T21:31:07.375 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 1 2024-04-20T21:31:07.375 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:31:07.375 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T21:31:07.375 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 6 2024-04-20T21:31:07.375 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:31:07.375 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T21:31:07.375 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 4 2024-04-20T21:31:07.375 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:31:07.375 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T21:31:07.375 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 13, 2024-04-20T21:31:07.376 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 1 2024-04-20T21:31:07.376 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T21:31:07.376 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:31:07.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:07.378+0000 7fd4bf800700 1 -- 172.21.15.62:0/3875751600 >> [v2:172.21.15.189:6828/2010473407,v1:172.21.15.189:6829/2010473407] conn(0x7fd4a006d910 msgr2=0x7fd4a006fdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:31:07.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:07.378+0000 7fd4bf800700 1 --2- 172.21.15.62:0/3875751600 >> [v2:172.21.15.189:6828/2010473407,v1:172.21.15.189:6829/2010473407] conn(0x7fd4a006d910 0x7fd4a006fdc0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fd4ac005fd0 tx=0x7fd4ac00a560 comp rx=0 tx=0).stop 2024-04-20T21:31:07.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:07.378+0000 7fd4bf800700 1 -- 172.21.15.62:0/3875751600 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd4b8074d40 msgr2=0x7fd4b819c5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:31:07.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:07.378+0000 7fd4bf800700 1 --2- 172.21.15.62:0/3875751600 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd4b8074d40 0x7fd4b819c5f0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fd4a400b5c0 tx=0x7fd4a400be20 comp rx=0 tx=0).stop 2024-04-20T21:31:07.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:07.378+0000 7fd4bf800700 1 -- 172.21.15.62:0/3875751600 shutdown_connections 2024-04-20T21:31:07.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:07.378+0000 7fd4bf800700 1 --2- 172.21.15.62:0/3875751600 >> [v2:172.21.15.189:6828/2010473407,v1:172.21.15.189:6829/2010473407] conn(0x7fd4a006d910 0x7fd4a006fdc0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:31:07.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:07.378+0000 7fd4bf800700 1 --2- 172.21.15.62:0/3875751600 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd4b8073770 0x7fd4b819c0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:31:07.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:07.378+0000 7fd4bf800700 1 --2- 172.21.15.62:0/3875751600 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd4b8074d40 0x7fd4b819c5f0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:31:07.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:07.378+0000 7fd4bf800700 1 -- 172.21.15.62:0/3875751600 >> 172.21.15.62:0/3875751600 conn(0x7fd4b80fb7f0 msgr2=0x7fd4b8101c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:31:07.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:07.379+0000 7fd4bf800700 1 -- 172.21.15.62:0/3875751600 shutdown_connections 2024-04-20T21:31:07.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:07.379+0000 7fd4bf800700 1 -- 172.21.15.62:0/3875751600 wait complete. 2024-04-20T21:31:07.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:07.564+0000 7f44b4c00700 1 -- 172.21.15.62:0/3691412412 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f44b01014b0 msgr2=0x7f44b0103930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:31:07.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:07.564+0000 7f44b4c00700 1 --2- 172.21.15.62:0/3691412412 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f44b01014b0 0x7f44b0103930 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f449c009910 tx=0x7f449c009c20 comp rx=0 tx=0).stop 2024-04-20T21:31:07.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:07.565+0000 7f44b4c00700 1 -- 172.21.15.62:0/3691412412 shutdown_connections 2024-04-20T21:31:07.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:07.565+0000 7f44b4c00700 1 --2- 172.21.15.62:0/3691412412 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f44b01014b0 0x7f44b0103930 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:31:07.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:07.565+0000 7f44b4c00700 1 --2- 172.21.15.62:0/3691412412 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f44b00feb50 0x7f44b0100f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:31:07.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:07.565+0000 7f44b4c00700 1 -- 172.21.15.62:0/3691412412 >> 172.21.15.62:0/3691412412 conn(0x7f44b00fa780 msgr2=0x7f44b00fcbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:31:07.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:07.565+0000 7f44b4c00700 1 -- 172.21.15.62:0/3691412412 shutdown_connections 2024-04-20T21:31:07.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:07.565+0000 7f44b4c00700 1 -- 172.21.15.62:0/3691412412 wait complete. 2024-04-20T21:31:07.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:07.566+0000 7f44b4c00700 1 Processor -- start 2024-04-20T21:31:07.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:07.566+0000 7f44b4c00700 1 -- start start 2024-04-20T21:31:07.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:07.566+0000 7f44b4c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f44b00feb50 0x7f44b019c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:31:07.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:07.567+0000 7f44b4c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f44b01014b0 0x7f44b019c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:31:07.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:07.567+0000 7f44b4c00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44b019cb70 con 0x7f44b00feb50 2024-04-20T21:31:07.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:07.567+0000 7f44b4c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44b019ccb0 con 0x7f44b01014b0 2024-04-20T21:31:07.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:07.567+0000 7f44ade00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f44b00feb50 0x7f44b019c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:31:07.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:07.567+0000 7f44ade00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f44b00feb50 0x7f44b019c0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42432/0 (socket says 172.21.15.62:42432) 2024-04-20T21:31:07.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:07.567+0000 7f44ade00700 1 -- 172.21.15.62:0/3074103295 learned_addr learned my addr 172.21.15.62:0/3074103295 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:31:07.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:07.567+0000 7f44ad400700 1 --2- 172.21.15.62:0/3074103295 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f44b01014b0 0x7f44b019c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:31:07.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:07.567+0000 7f44ade00700 1 -- 172.21.15.62:0/3074103295 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f44b01014b0 msgr2=0x7f44b019c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:31:07.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:07.567+0000 7f44ade00700 1 --2- 172.21.15.62:0/3074103295 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f44b01014b0 0x7f44b019c5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:31:07.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:07.567+0000 7f44ade00700 1 -- 172.21.15.62:0/3074103295 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f449c0095f0 con 0x7f44b00feb50 2024-04-20T21:31:07.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:07.568+0000 7f44ade00700 1 --2- 172.21.15.62:0/3074103295 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f44b00feb50 0x7f44b019c0a0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f44a000c910 tx=0x7f44a000ccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:31:07.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:07.568+0000 7f44a6a00700 1 -- 172.21.15.62:0/3074103295 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f44a00041f0 con 0x7f44b00feb50 2024-04-20T21:31:07.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:07.568+0000 7f44a6a00700 1 -- 172.21.15.62:0/3074103295 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f44a0007db0 con 0x7f44b00feb50 2024-04-20T21:31:07.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:07.568+0000 7f44a6a00700 1 -- 172.21.15.62:0/3074103295 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f44a0003d60 con 0x7f44b00feb50 2024-04-20T21:31:07.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:07.568+0000 7f44b4c00700 1 -- 172.21.15.62:0/3074103295 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44b01a1770 con 0x7f44b00feb50 2024-04-20T21:31:07.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:07.568+0000 7f44b4c00700 1 -- 172.21.15.62:0/3074103295 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44b01a1c90 con 0x7f44b00feb50 2024-04-20T21:31:07.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:07.570+0000 7f44b4c00700 1 -- 172.21.15.62:0/3074103295 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44b01961c0 con 0x7f44b00feb50 2024-04-20T21:31:07.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:07.571+0000 7f44a6a00700 1 -- 172.21.15.62:0/3074103295 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 24) v1 ==== 91360+0+0 (secure 0 0 0) 0x7f44a0007900 con 0x7f44b00feb50 2024-04-20T21:31:07.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:07.572+0000 7f44a6a00700 1 --2- 172.21.15.62:0/3074103295 >> [v2:172.21.15.189:6828/2010473407,v1:172.21.15.189:6829/2010473407] conn(0x7f449406d910 0x7f449406fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:31:07.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:07.573+0000 7f44a6a00700 1 -- 172.21.15.62:0/3074103295 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5726+0+0 (secure 0 0 0) 0x7f44a008cfc0 con 0x7f44b00feb50 2024-04-20T21:31:07.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:07.573+0000 7f44ad400700 1 --2- 172.21.15.62:0/3074103295 >> [v2:172.21.15.189:6828/2010473407,v1:172.21.15.189:6829/2010473407] conn(0x7f449406d910 0x7f449406fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:31:07.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:07.575+0000 7f44ad400700 1 --2- 172.21.15.62:0/3074103295 >> [v2:172.21.15.189:6828/2010473407,v1:172.21.15.189:6829/2010473407] conn(0x7f449406d910 0x7f449406fdc0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f449c006010 tx=0x7f449c00b540 comp rx=0 tx=0).ready entity=mgr.24379 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:31:07.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:07.578+0000 7f44a6a00700 1 -- 172.21.15.62:0/3074103295 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187527 (secure 0 0 0) 0x7f44a0059cf0 con 0x7f44b00feb50 2024-04-20T21:31:07.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:07.918+0000 7f44b4c00700 1 -- 172.21.15.62:0/3074103295 --> [v2:172.21.15.189:6828/2010473407,v1:172.21.15.189:6829/2010473407] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f44b0061190 con 0x7f449406d910 2024-04-20T21:31:07.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:07.920+0000 7f44a6a00700 1 -- 172.21.15.62:0/3074103295 <== mgr.24379 v2:172.21.15.189:6828/2010473407 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+310 (secure 0 0 0) 0x7f44b0061190 con 0x7f449406d910 2024-04-20T21:31:07.920 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:31:07.920 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T21:31:07.920 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T21:31:07.920 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-20T21:31:07.920 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [], 2024-04-20T21:31:07.920 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "1/2 daemons upgraded", 2024-04-20T21:31:07.920 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "", 2024-04-20T21:31:07.920 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T21:31:07.920 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:31:07.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:07.923+0000 7f44b4c00700 1 -- 172.21.15.62:0/3074103295 >> [v2:172.21.15.189:6828/2010473407,v1:172.21.15.189:6829/2010473407] conn(0x7f449406d910 msgr2=0x7f449406fdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:31:07.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:07.923+0000 7f44b4c00700 1 --2- 172.21.15.62:0/3074103295 >> [v2:172.21.15.189:6828/2010473407,v1:172.21.15.189:6829/2010473407] conn(0x7f449406d910 0x7f449406fdc0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f449c006010 tx=0x7f449c00b540 comp rx=0 tx=0).stop 2024-04-20T21:31:07.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:07.923+0000 7f44b4c00700 1 -- 172.21.15.62:0/3074103295 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f44b00feb50 msgr2=0x7f44b019c0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:31:07.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:07.923+0000 7f44b4c00700 1 --2- 172.21.15.62:0/3074103295 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f44b00feb50 0x7f44b019c0a0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f44a000c910 tx=0x7f44a000ccd0 comp rx=0 tx=0).stop 2024-04-20T21:31:07.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:07.924+0000 7f44b4c00700 1 -- 172.21.15.62:0/3074103295 shutdown_connections 2024-04-20T21:31:07.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:07.924+0000 7f44b4c00700 1 --2- 172.21.15.62:0/3074103295 >> [v2:172.21.15.189:6828/2010473407,v1:172.21.15.189:6829/2010473407] conn(0x7f449406d910 0x7f449406fdc0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:31:07.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:07.924+0000 7f44b4c00700 1 --2- 172.21.15.62:0/3074103295 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f44b00feb50 0x7f44b019c0a0 secure :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f44a000c910 tx=0x7f44a000ccd0 comp rx=0 tx=0).stop 2024-04-20T21:31:07.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:07.924+0000 7f44b4c00700 1 --2- 172.21.15.62:0/3074103295 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f44b01014b0 0x7f44b019c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:31:07.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:07.924+0000 7f44b4c00700 1 -- 172.21.15.62:0/3074103295 >> 172.21.15.62:0/3074103295 conn(0x7f44b00fa780 msgr2=0x7f44b00fcbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:31:07.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:07.924+0000 7f44b4c00700 1 -- 172.21.15.62:0/3074103295 shutdown_connections 2024-04-20T21:31:07.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:07.924+0000 7f44b4c00700 1 -- 172.21.15.62:0/3074103295 wait complete. 2024-04-20T21:31:08.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:07 smithi189 ceph-mon[21562]: pgmap v19: 65 pgs: 65 active+clean; 131 MiB data, 616 MiB used, 536 GiB / 536 GiB avail; 1.8 MiB/s rd, 1.9 MiB/s wr, 109 op/s 2024-04-20T21:31:08.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:07 smithi189 ceph-mon[21562]: from='client.? 172.21.15.62:0/3875751600' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:31:08.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:07 smithi189 ceph-mon[21562]: from='mgr.24379 ' entity='mgr.smithi189.tawavz' 2024-04-20T21:31:08.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:07 smithi189 ceph-mon[21562]: from='mgr.24379 ' entity='mgr.smithi189.tawavz' 2024-04-20T21:31:08.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:07 smithi189 ceph-mon[21562]: from='mgr.24379 172.21.15.189:0/1768059451' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-20T21:31:08.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:07 smithi062 ceph-mon[16221]: pgmap v19: 65 pgs: 65 active+clean; 131 MiB data, 616 MiB used, 536 GiB / 536 GiB avail; 1.8 MiB/s rd, 1.9 MiB/s wr, 109 op/s 2024-04-20T21:31:08.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:07 smithi062 ceph-mon[16221]: from='client.? 172.21.15.62:0/3875751600' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:31:08.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:07 smithi062 ceph-mon[16221]: from='mgr.24379 ' entity='mgr.smithi189.tawavz' 2024-04-20T21:31:08.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:07 smithi062 ceph-mon[16221]: from='mgr.24379 ' entity='mgr.smithi189.tawavz' 2024-04-20T21:31:08.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:07 smithi062 ceph-mon[16221]: from='mgr.24379 172.21.15.189:0/1768059451' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-20T21:31:09.195 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:08 smithi062 ceph-mon[16221]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-20T21:31:09.195 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:08 smithi062 ceph-mon[16221]: from='client.14654 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:31:09.195 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:08 smithi062 ceph-mon[16221]: from='mgr.24379 172.21.15.189:0/1768059451' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:31:09.195 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:08 smithi062 ceph-mon[16221]: Upgrade: Need to upgrade myself (mgr.smithi189.tawavz) 2024-04-20T21:31:09.195 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:08 smithi062 ceph-mon[16221]: Upgrade: Need to upgrade myself (mgr.smithi189.tawavz) 2024-04-20T21:31:09.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:08 smithi189 ceph-mon[21562]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-20T21:31:09.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:08 smithi189 ceph-mon[21562]: from='client.14654 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:31:09.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:08 smithi189 ceph-mon[21562]: from='mgr.24379 172.21.15.189:0/1768059451' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:31:09.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:08 smithi189 ceph-mon[21562]: Upgrade: Need to upgrade myself (mgr.smithi189.tawavz) 2024-04-20T21:31:09.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:08 smithi189 ceph-mon[21562]: Upgrade: Need to upgrade myself (mgr.smithi189.tawavz) 2024-04-20T21:31:09.939 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:09 smithi189 ceph-mon[21562]: pgmap v20: 65 pgs: 65 active+clean; 131 MiB data, 616 MiB used, 536 GiB / 536 GiB avail; 1.1 MiB/s rd, 1.3 MiB/s wr, 63 op/s 2024-04-20T21:31:10.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:09 smithi062 ceph-mon[16221]: pgmap v20: 65 pgs: 65 active+clean; 131 MiB data, 616 MiB used, 536 GiB / 536 GiB avail; 1.1 MiB/s rd, 1.3 MiB/s wr, 63 op/s 2024-04-20T21:31:11.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:11 smithi189 ceph-mon[21562]: Upgrade: Updating mgr.smithi062.rqxuxi 2024-04-20T21:31:11.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:11 smithi189 ceph-mon[21562]: from='mgr.24379 ' entity='mgr.smithi189.tawavz' 2024-04-20T21:31:11.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:11 smithi189 ceph-mon[21562]: from='mgr.24379 172.21.15.189:0/1768059451' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi062.rqxuxi", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-20T21:31:11.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:11 smithi189 ceph-mon[21562]: from='mgr.24379 ' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi062.rqxuxi", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-20T21:31:11.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:11 smithi189 ceph-mon[21562]: from='mgr.24379 172.21.15.189:0/1768059451' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-20T21:31:11.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:11 smithi189 ceph-mon[21562]: from='mgr.24379 172.21.15.189:0/1768059451' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:31:11.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:11 smithi189 ceph-mon[21562]: Deploying daemon mgr.smithi062.rqxuxi on smithi062 2024-04-20T21:31:11.765 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:11 smithi062 ceph-mon[16221]: Upgrade: Updating mgr.smithi062.rqxuxi 2024-04-20T21:31:11.765 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:11 smithi062 ceph-mon[16221]: from='mgr.24379 ' entity='mgr.smithi189.tawavz' 2024-04-20T21:31:11.765 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:11 smithi062 ceph-mon[16221]: from='mgr.24379 172.21.15.189:0/1768059451' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi062.rqxuxi", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-20T21:31:11.765 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:11 smithi062 ceph-mon[16221]: from='mgr.24379 ' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi062.rqxuxi", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-20T21:31:11.766 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:11 smithi062 ceph-mon[16221]: from='mgr.24379 172.21.15.189:0/1768059451' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-20T21:31:11.766 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:11 smithi062 ceph-mon[16221]: from='mgr.24379 172.21.15.189:0/1768059451' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:31:11.766 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:11 smithi062 ceph-mon[16221]: Deploying daemon mgr.smithi062.rqxuxi on smithi062 2024-04-20T21:31:13.201 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:12 smithi062 ceph-mon[16221]: pgmap v21: 65 pgs: 65 active+clean; 133 MiB data, 624 MiB used, 536 GiB / 536 GiB avail; 1.1 MiB/s rd, 1.4 MiB/s wr, 83 op/s 2024-04-20T21:31:13.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:12 smithi189 ceph-mon[21562]: pgmap v21: 65 pgs: 65 active+clean; 133 MiB data, 624 MiB used, 536 GiB / 536 GiB avail; 1.1 MiB/s rd, 1.4 MiB/s wr, 83 op/s 2024-04-20T21:31:14.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:14 smithi189 ceph-mon[21562]: pgmap v22: 65 pgs: 65 active+clean; 140 MiB data, 643 MiB used, 536 GiB / 536 GiB avail; 1.2 MiB/s rd, 1.9 MiB/s wr, 110 op/s 2024-04-20T21:31:15.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:14 smithi062 ceph-mon[16221]: pgmap v22: 65 pgs: 65 active+clean; 140 MiB data, 643 MiB used, 536 GiB / 536 GiB avail; 1.2 MiB/s rd, 1.9 MiB/s wr, 110 op/s 2024-04-20T21:31:16.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:15 smithi189 ceph-mon[21562]: fsmap cephfs:2 {0=cephfs.smithi062.ryqblb=up:active,1=cephfs.smithi189.dvimer=up:active} 2 up:standby-replay 2024-04-20T21:31:16.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:15 smithi189 ceph-mon[21562]: pgmap v23: 65 pgs: 65 active+clean; 140 MiB data, 643 MiB used, 536 GiB / 536 GiB avail; 810 KiB/s rd, 1.4 MiB/s wr, 78 op/s 2024-04-20T21:31:16.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:15 smithi062 ceph-mon[16221]: fsmap cephfs:2 {0=cephfs.smithi062.ryqblb=up:active,1=cephfs.smithi189.dvimer=up:active} 2 up:standby-replay 2024-04-20T21:31:16.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:15 smithi062 ceph-mon[16221]: pgmap v23: 65 pgs: 65 active+clean; 140 MiB data, 643 MiB used, 536 GiB / 536 GiB avail; 810 KiB/s rd, 1.4 MiB/s wr, 78 op/s 2024-04-20T21:31:18.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:17 smithi062 ceph-mon[16221]: from='mgr.24379 ' entity='mgr.smithi189.tawavz' 2024-04-20T21:31:18.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:17 smithi062 ceph-mon[16221]: from='mgr.24379 ' entity='mgr.smithi189.tawavz' 2024-04-20T21:31:18.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:17 smithi062 ceph-mon[16221]: from='mgr.24379 172.21.15.189:0/1768059451' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:31:18.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:17 smithi062 ceph-mon[16221]: pgmap v24: 65 pgs: 65 active+clean; 152 MiB data, 689 MiB used, 536 GiB / 536 GiB avail; 1.4 MiB/s rd, 2.4 MiB/s wr, 134 op/s 2024-04-20T21:31:18.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:17 smithi189 ceph-mon[21562]: from='mgr.24379 ' entity='mgr.smithi189.tawavz' 2024-04-20T21:31:18.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:17 smithi189 ceph-mon[21562]: from='mgr.24379 ' entity='mgr.smithi189.tawavz' 2024-04-20T21:31:18.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:17 smithi189 ceph-mon[21562]: from='mgr.24379 172.21.15.189:0/1768059451' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:31:18.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:17 smithi189 ceph-mon[21562]: pgmap v24: 65 pgs: 65 active+clean; 152 MiB data, 689 MiB used, 536 GiB / 536 GiB avail; 1.4 MiB/s rd, 2.4 MiB/s wr, 134 op/s 2024-04-20T21:31:19.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:19 smithi062 ceph-mon[16221]: from='mgr.24379 ' entity='mgr.smithi189.tawavz' 2024-04-20T21:31:19.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:19 smithi062 ceph-mon[16221]: from='mgr.24379 ' entity='mgr.smithi189.tawavz' 2024-04-20T21:31:19.389 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:19 smithi189 ceph-mon[21562]: from='mgr.24379 ' entity='mgr.smithi189.tawavz' 2024-04-20T21:31:19.390 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:19 smithi189 ceph-mon[21562]: from='mgr.24379 ' entity='mgr.smithi189.tawavz' 2024-04-20T21:31:20.412 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:20 smithi189 ceph-mon[21562]: pgmap v25: 65 pgs: 65 active+clean; 152 MiB data, 689 MiB used, 536 GiB / 536 GiB avail; 899 KiB/s rd, 1.8 MiB/s wr, 110 op/s 2024-04-20T21:31:20.534 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:20 smithi062 ceph-mon[16221]: pgmap v25: 65 pgs: 65 active+clean; 152 MiB data, 689 MiB used, 536 GiB / 536 GiB avail; 899 KiB/s rd, 1.8 MiB/s wr, 110 op/s 2024-04-20T21:31:21.132 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:21 smithi062 ceph-mon[16221]: from='mgr.24379 172.21.15.189:0/1768059451' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:31:21.154 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:21 smithi189 ceph-mon[21562]: from='mgr.24379 172.21.15.189:0/1768059451' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:31:23.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:22 smithi189 ceph-mon[21562]: pgmap v26: 65 pgs: 65 active+clean; 155 MiB data, 712 MiB used, 536 GiB / 536 GiB avail; 1.0 MiB/s rd, 2.0 MiB/s wr, 140 op/s 2024-04-20T21:31:23.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:22 smithi062 ceph-mon[16221]: pgmap v26: 65 pgs: 65 active+clean; 155 MiB data, 712 MiB used, 536 GiB / 536 GiB avail; 1.0 MiB/s rd, 2.0 MiB/s wr, 140 op/s 2024-04-20T21:31:25.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:24 smithi189 ceph-mon[21562]: pgmap v27: 65 pgs: 65 active+clean; 162 MiB data, 729 MiB used, 536 GiB / 536 GiB avail; 1.3 MiB/s rd, 2.5 MiB/s wr, 158 op/s 2024-04-20T21:31:25.284 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:24 smithi062 ceph-mon[16221]: pgmap v27: 65 pgs: 65 active+clean; 162 MiB data, 729 MiB used, 536 GiB / 536 GiB avail; 1.3 MiB/s rd, 2.5 MiB/s wr, 158 op/s 2024-04-20T21:31:26.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:25 smithi189 ceph-mon[21562]: pgmap v28: 65 pgs: 65 active+clean; 162 MiB data, 729 MiB used, 536 GiB / 536 GiB avail; 1024 KiB/s rd, 1.9 MiB/s wr, 124 op/s 2024-04-20T21:31:26.284 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:25 smithi062 ceph-mon[16221]: pgmap v28: 65 pgs: 65 active+clean; 162 MiB data, 729 MiB used, 536 GiB / 536 GiB avail; 1024 KiB/s rd, 1.9 MiB/s wr, 124 op/s 2024-04-20T21:31:29.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:28 smithi062 ceph-mon[16221]: pgmap v29: 65 pgs: 65 active+clean; 170 MiB data, 765 MiB used, 536 GiB / 536 GiB avail; 1.4 MiB/s rd, 2.6 MiB/s wr, 190 op/s 2024-04-20T21:31:29.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:28 smithi062 ceph-mon[16221]: from='mgr.24379 ' entity='mgr.smithi189.tawavz' 2024-04-20T21:31:29.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:28 smithi062 ceph-mon[16221]: from='mgr.24379 ' entity='mgr.smithi189.tawavz' 2024-04-20T21:31:29.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:28 smithi189 ceph-mon[21562]: pgmap v29: 65 pgs: 65 active+clean; 170 MiB data, 765 MiB used, 536 GiB / 536 GiB avail; 1.4 MiB/s rd, 2.6 MiB/s wr, 190 op/s 2024-04-20T21:31:29.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:28 smithi189 ceph-mon[21562]: from='mgr.24379 ' entity='mgr.smithi189.tawavz' 2024-04-20T21:31:29.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:28 smithi189 ceph-mon[21562]: from='mgr.24379 ' entity='mgr.smithi189.tawavz' 2024-04-20T21:31:30.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:29 smithi189 ceph-mon[21562]: Standby manager daemon smithi062.rqxuxi restarted 2024-04-20T21:31:30.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:29 smithi189 ceph-mon[21562]: Standby manager daemon smithi062.rqxuxi started 2024-04-20T21:31:30.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:29 smithi189 ceph-mon[21562]: from='mgr.? 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi062.rqxuxi/crt"}]: dispatch 2024-04-20T21:31:30.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:29 smithi189 ceph-mon[21562]: from='mgr.? 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-20T21:31:30.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:29 smithi189 ceph-mon[21562]: from='mgr.? 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi062.rqxuxi/key"}]: dispatch 2024-04-20T21:31:30.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:29 smithi189 ceph-mon[21562]: from='mgr.? 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-20T21:31:30.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:29 smithi189 ceph-mon[21562]: pgmap v30: 65 pgs: 65 active+clean; 170 MiB data, 765 MiB used, 536 GiB / 536 GiB avail; 794 KiB/s rd, 1.5 MiB/s wr, 133 op/s 2024-04-20T21:31:30.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:29 smithi062 ceph-mon[16221]: Standby manager daemon smithi062.rqxuxi restarted 2024-04-20T21:31:30.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:29 smithi062 ceph-mon[16221]: Standby manager daemon smithi062.rqxuxi started 2024-04-20T21:31:30.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:29 smithi062 ceph-mon[16221]: from='mgr.? 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi062.rqxuxi/crt"}]: dispatch 2024-04-20T21:31:30.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:29 smithi062 ceph-mon[16221]: from='mgr.? 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-20T21:31:30.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:29 smithi062 ceph-mon[16221]: from='mgr.? 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi062.rqxuxi/key"}]: dispatch 2024-04-20T21:31:30.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:29 smithi062 ceph-mon[16221]: from='mgr.? 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-20T21:31:30.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:29 smithi062 ceph-mon[16221]: pgmap v30: 65 pgs: 65 active+clean; 170 MiB data, 765 MiB used, 536 GiB / 536 GiB avail; 794 KiB/s rd, 1.5 MiB/s wr, 133 op/s 2024-04-20T21:31:31.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:30 smithi189 ceph-mon[21562]: mgrmap e25: smithi189.tawavz(active, since 54s), standbys: smithi062.rqxuxi 2024-04-20T21:31:31.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:30 smithi189 ceph-mon[21562]: from='mgr.24379 ' entity='mgr.smithi189.tawavz' 2024-04-20T21:31:31.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:30 smithi189 ceph-mon[21562]: from='mgr.24379 ' entity='mgr.smithi189.tawavz' 2024-04-20T21:31:31.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:30 smithi062 ceph-mon[16221]: mgrmap e25: smithi189.tawavz(active, since 54s), standbys: smithi062.rqxuxi 2024-04-20T21:31:31.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:30 smithi062 ceph-mon[16221]: from='mgr.24379 ' entity='mgr.smithi189.tawavz' 2024-04-20T21:31:31.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:30 smithi062 ceph-mon[16221]: from='mgr.24379 ' entity='mgr.smithi189.tawavz' 2024-04-20T21:31:33.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:32 smithi189 ceph-mon[21562]: pgmap v31: 65 pgs: 65 active+clean; 177 MiB data, 756 MiB used, 536 GiB / 536 GiB avail; 1.2 MiB/s rd, 2.1 MiB/s wr, 194 op/s 2024-04-20T21:31:33.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:32 smithi062 ceph-mon[16221]: pgmap v31: 65 pgs: 65 active+clean; 177 MiB data, 756 MiB used, 536 GiB / 536 GiB avail; 1.2 MiB/s rd, 2.1 MiB/s wr, 194 op/s 2024-04-20T21:31:34.042 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:33 smithi062 ceph-mon[16221]: pgmap v32: 65 pgs: 65 active+clean; 178 MiB data, 750 MiB used, 536 GiB / 536 GiB avail; 1.0 MiB/s rd, 1.9 MiB/s wr, 178 op/s 2024-04-20T21:31:34.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:33 smithi189 ceph-mon[21562]: pgmap v32: 65 pgs: 65 active+clean; 178 MiB data, 750 MiB used, 536 GiB / 536 GiB avail; 1.0 MiB/s rd, 1.9 MiB/s wr, 178 op/s 2024-04-20T21:31:36.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:36 smithi189 ceph-mon[21562]: pgmap v33: 65 pgs: 65 active+clean; 178 MiB data, 750 MiB used, 536 GiB / 536 GiB avail; 831 KiB/s rd, 1.3 MiB/s wr, 141 op/s 2024-04-20T21:31:36.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:36 smithi189 ceph-mon[21562]: from='mgr.24379 172.21.15.189:0/1768059451' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:31:36.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:36 smithi062 ceph-mon[16221]: pgmap v33: 65 pgs: 65 active+clean; 178 MiB data, 750 MiB used, 536 GiB / 536 GiB avail; 831 KiB/s rd, 1.3 MiB/s wr, 141 op/s 2024-04-20T21:31:36.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:36 smithi062 ceph-mon[16221]: from='mgr.24379 172.21.15.189:0/1768059451' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:31:37.188 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:37 smithi189 ceph-mon[21562]: from='mgr.24379 ' entity='mgr.smithi189.tawavz' 2024-04-20T21:31:37.188 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:37 smithi189 ceph-mon[21562]: from='mgr.24379 ' entity='mgr.smithi189.tawavz' 2024-04-20T21:31:37.188 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:37 smithi189 ceph-mon[21562]: from='mgr.24379 172.21.15.189:0/1768059451' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:31:37.188 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:37 smithi189 ceph-mon[21562]: from='mgr.24379 172.21.15.189:0/1768059451' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T21:31:37.188 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:37 smithi189 ceph-mon[21562]: from='mgr.24379 ' entity='mgr.smithi189.tawavz' 2024-04-20T21:31:37.189 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:37 smithi189 ceph-mon[21562]: from='mgr.24379 172.21.15.189:0/1768059451' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:31:37.189 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:37 smithi189 ceph-mon[21562]: from='mgr.24379 172.21.15.189:0/1768059451' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "mgr fail", "who": "smithi189.tawavz"}]: dispatch 2024-04-20T21:31:37.189 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:37 smithi189 ceph-mon[21562]: from='mgr.24379 ' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "mgr fail", "who": "smithi189.tawavz"}]: dispatch 2024-04-20T21:31:37.189 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:37 smithi189 ceph-mon[21562]: osdmap e47: 6 total, 6 up, 6 in 2024-04-20T21:31:37.189 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:37 smithi189 ceph-mon[21562]: from='mgr.24379 ' entity='mgr.smithi189.tawavz' cmd='[{"prefix": "mgr fail", "who": "smithi189.tawavz"}]': finished 2024-04-20T21:31:37.189 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:37 smithi189 ceph-mon[21562]: mgrmap e26: smithi062.rqxuxi(active, starting, since 0.0114718s) 2024-04-20T21:31:37.189 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:37 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mon metadata", "id": "smithi062"}]: dispatch 2024-04-20T21:31:37.189 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:37 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mon metadata", "id": "smithi189"}]: dispatch 2024-04-20T21:31:37.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:37 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi062.ryqblb"}]: dispatch 2024-04-20T21:31:37.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:37 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi062.ozizwi"}]: dispatch 2024-04-20T21:31:37.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:37 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi189.dvimer"}]: dispatch 2024-04-20T21:31:37.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:37 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi189.xbgspw"}]: dispatch 2024-04-20T21:31:37.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:37 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mgr metadata", "who": "smithi062.rqxuxi", "id": "smithi062.rqxuxi"}]: dispatch 2024-04-20T21:31:37.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:37 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-20T21:31:37.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:37 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-20T21:31:37.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:37 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-20T21:31:37.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:37 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-20T21:31:37.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:37 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-20T21:31:37.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:37 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-20T21:31:37.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:37 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-20T21:31:37.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:37 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-20T21:31:37.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:37 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-20T21:31:37.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:37 smithi189 ceph-mon[21562]: Manager daemon smithi062.rqxuxi is now available 2024-04-20T21:31:37.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:37 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:31:37.489 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:37 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:31:37.489 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:37 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi062.rqxuxi/mirror_snapshot_schedule"}]: dispatch 2024-04-20T21:31:37.489 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:37 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi062.rqxuxi/trash_purge_schedule"}]: dispatch 2024-04-20T21:31:37.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:37 smithi062 ceph-mon[16221]: from='mgr.24379 ' entity='mgr.smithi189.tawavz' 2024-04-20T21:31:37.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:37 smithi062 ceph-mon[16221]: from='mgr.24379 ' entity='mgr.smithi189.tawavz' 2024-04-20T21:31:37.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:37 smithi062 ceph-mon[16221]: from='mgr.24379 172.21.15.189:0/1768059451' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:31:37.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:37 smithi062 ceph-mon[16221]: from='mgr.24379 172.21.15.189:0/1768059451' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T21:31:37.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:37 smithi062 ceph-mon[16221]: from='mgr.24379 ' entity='mgr.smithi189.tawavz' 2024-04-20T21:31:37.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:37 smithi062 ceph-mon[16221]: from='mgr.24379 172.21.15.189:0/1768059451' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:31:37.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:37 smithi062 ceph-mon[16221]: from='mgr.24379 172.21.15.189:0/1768059451' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "mgr fail", "who": "smithi189.tawavz"}]: dispatch 2024-04-20T21:31:37.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:37 smithi062 ceph-mon[16221]: from='mgr.24379 ' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "mgr fail", "who": "smithi189.tawavz"}]: dispatch 2024-04-20T21:31:37.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:37 smithi062 ceph-mon[16221]: osdmap e47: 6 total, 6 up, 6 in 2024-04-20T21:31:37.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:37 smithi062 ceph-mon[16221]: from='mgr.24379 ' entity='mgr.smithi189.tawavz' cmd='[{"prefix": "mgr fail", "who": "smithi189.tawavz"}]': finished 2024-04-20T21:31:37.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:37 smithi062 ceph-mon[16221]: mgrmap e26: smithi062.rqxuxi(active, starting, since 0.0114718s) 2024-04-20T21:31:37.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:37 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mon metadata", "id": "smithi062"}]: dispatch 2024-04-20T21:31:37.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:37 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mon metadata", "id": "smithi189"}]: dispatch 2024-04-20T21:31:37.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:37 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi062.ryqblb"}]: dispatch 2024-04-20T21:31:37.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:37 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi062.ozizwi"}]: dispatch 2024-04-20T21:31:37.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:37 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi189.dvimer"}]: dispatch 2024-04-20T21:31:37.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:37 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi189.xbgspw"}]: dispatch 2024-04-20T21:31:37.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:37 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mgr metadata", "who": "smithi062.rqxuxi", "id": "smithi062.rqxuxi"}]: dispatch 2024-04-20T21:31:37.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:37 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-20T21:31:37.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:37 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-20T21:31:37.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:37 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-20T21:31:37.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:37 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-20T21:31:37.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:37 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-20T21:31:37.537 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:37 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-20T21:31:37.537 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:37 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-20T21:31:37.537 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:37 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-20T21:31:37.537 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:37 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-20T21:31:37.537 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:37 smithi062 ceph-mon[16221]: Manager daemon smithi062.rqxuxi is now available 2024-04-20T21:31:37.537 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:37 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:31:37.537 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:37 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:31:37.537 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:37 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi062.rqxuxi/mirror_snapshot_schedule"}]: dispatch 2024-04-20T21:31:37.537 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:37 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi062.rqxuxi/trash_purge_schedule"}]: dispatch 2024-04-20T21:31:38.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:38.139+0000 7ffb50a00700 1 -- 172.21.15.62:0/1263000216 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb4c103640 msgr2=0x7ffb4c103a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:31:38.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:38.139+0000 7ffb50a00700 1 --2- 172.21.15.62:0/1263000216 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb4c103640 0x7ffb4c103a90 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ffb38009910 tx=0x7ffb38009c20 comp rx=0 tx=0).stop 2024-04-20T21:31:38.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:38.139+0000 7ffb50a00700 1 -- 172.21.15.62:0/1263000216 shutdown_connections 2024-04-20T21:31:38.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:38.139+0000 7ffb50a00700 1 --2- 172.21.15.62:0/1263000216 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb4c103640 0x7ffb4c103a90 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:31:38.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:38.139+0000 7ffb50a00700 1 --2- 172.21.15.62:0/1263000216 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffb4c102490 0x7ffb4c1028a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:31:38.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:38.139+0000 7ffb50a00700 1 -- 172.21.15.62:0/1263000216 >> 172.21.15.62:0/1263000216 conn(0x7ffb4c0fda60 msgr2=0x7ffb4c0ffe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:31:38.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:38.141+0000 7ffb50a00700 1 -- 172.21.15.62:0/1263000216 shutdown_connections 2024-04-20T21:31:38.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:38.141+0000 7ffb50a00700 1 -- 172.21.15.62:0/1263000216 wait complete. 2024-04-20T21:31:38.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:38.143+0000 7ffb50a00700 1 Processor -- start 2024-04-20T21:31:38.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:38.143+0000 7ffb50a00700 1 -- start start 2024-04-20T21:31:38.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:38.143+0000 7ffb50a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffb4c102490 0x7ffb4c1938a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:31:38.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:38.143+0000 7ffb50a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb4c103640 0x7ffb4c193de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:31:38.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:38.143+0000 7ffb50a00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb4c194400 con 0x7ffb4c102490 2024-04-20T21:31:38.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:38.143+0000 7ffb50a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb4c194540 con 0x7ffb4c103640 2024-04-20T21:31:38.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:38.144+0000 7ffb49e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffb4c102490 0x7ffb4c1938a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:31:38.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:38.144+0000 7ffb49e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffb4c102490 0x7ffb4c1938a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49560/0 (socket says 172.21.15.62:49560) 2024-04-20T21:31:38.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:38.144+0000 7ffb49e00700 1 -- 172.21.15.62:0/1308772372 learned_addr learned my addr 172.21.15.62:0/1308772372 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:31:38.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:38.145+0000 7ffb49e00700 1 -- 172.21.15.62:0/1308772372 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb4c103640 msgr2=0x7ffb4c193de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:31:38.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:38.145+0000 7ffb49e00700 1 --2- 172.21.15.62:0/1308772372 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb4c103640 0x7ffb4c193de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:31:38.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:38.145+0000 7ffb49e00700 1 -- 172.21.15.62:0/1308772372 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb380095f0 con 0x7ffb4c102490 2024-04-20T21:31:38.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:38.145+0000 7ffb49e00700 1 --2- 172.21.15.62:0/1308772372 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffb4c102490 0x7ffb4c1938a0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7ffb3c014340 tx=0x7ffb3c014700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:31:38.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:38.145+0000 7ffb42a00700 1 -- 172.21.15.62:0/1308772372 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffb3c01f070 con 0x7ffb4c102490 2024-04-20T21:31:38.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:38.146+0000 7ffb42a00700 1 -- 172.21.15.62:0/1308772372 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ffb3c017040 con 0x7ffb4c102490 2024-04-20T21:31:38.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:38.146+0000 7ffb50a00700 1 -- 172.21.15.62:0/1308772372 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb4c198f90 con 0x7ffb4c102490 2024-04-20T21:31:38.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:38.146+0000 7ffb42a00700 1 -- 172.21.15.62:0/1308772372 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffb3c016050 con 0x7ffb4c102490 2024-04-20T21:31:38.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:38.147+0000 7ffb50a00700 1 -- 172.21.15.62:0/1308772372 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb4c199430 con 0x7ffb4c102490 2024-04-20T21:31:38.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:38.149+0000 7ffb37e00700 1 -- 172.21.15.62:0/1308772372 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb4c0778a0 con 0x7ffb4c102490 2024-04-20T21:31:38.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:38.150+0000 7ffb42a00700 1 -- 172.21.15.62:0/1308772372 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 27) v1 ==== 46641+0+0 (secure 0 0 0) 0x7ffb3c0115d0 con 0x7ffb4c102490 2024-04-20T21:31:38.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:38.153+0000 7ffb42a00700 1 --2- 172.21.15.62:0/1308772372 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7ffb30039840 0x7ffb3003bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:31:38.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:38.153+0000 7ffb42a00700 1 -- 172.21.15.62:0/1308772372 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6070+0+0 (secure 0 0 0) 0x7ffb3c0554b0 con 0x7ffb4c102490 2024-04-20T21:31:38.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:38.154+0000 7ffb49400700 1 --2- 172.21.15.62:0/1308772372 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7ffb30039840 0x7ffb3003bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:31:38.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:38.156+0000 7ffb49400700 1 --2- 172.21.15.62:0/1308772372 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7ffb30039840 0x7ffb3003bcf0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7ffb38006010 tx=0x7ffb38009f90 comp rx=0 tx=0).ready entity=mgr.14658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:31:38.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:38.163+0000 7ffb42a00700 1 -- 172.21.15.62:0/1308772372 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187527 (secure 0 0 0) 0x7ffb3c01a770 con 0x7ffb4c102490 2024-04-20T21:31:38.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:38.543+0000 7ffb37e00700 1 -- 172.21.15.62:0/1308772372 --> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ffb4c077af0 con 0x7ffb30039840 2024-04-20T21:31:38.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:38.546+0000 7ffb42a00700 1 -- 172.21.15.62:0/1308772372 <== mgr.14658 v2:172.21.15.62:6800/1585760929 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+329 (secure 0 0 0) 0x7ffb4c077af0 con 0x7ffb30039840 2024-04-20T21:31:38.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:38.549+0000 7ffb37e00700 1 -- 172.21.15.62:0/1308772372 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7ffb30039840 msgr2=0x7ffb3003bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:31:38.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:38.549+0000 7ffb37e00700 1 --2- 172.21.15.62:0/1308772372 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7ffb30039840 0x7ffb3003bcf0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7ffb38006010 tx=0x7ffb38009f90 comp rx=0 tx=0).stop 2024-04-20T21:31:38.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:38.549+0000 7ffb37e00700 1 -- 172.21.15.62:0/1308772372 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffb4c102490 msgr2=0x7ffb4c1938a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:31:38.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:38.549+0000 7ffb37e00700 1 --2- 172.21.15.62:0/1308772372 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffb4c102490 0x7ffb4c1938a0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7ffb3c014340 tx=0x7ffb3c014700 comp rx=0 tx=0).stop 2024-04-20T21:31:38.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:38.549+0000 7ffb37e00700 1 -- 172.21.15.62:0/1308772372 shutdown_connections 2024-04-20T21:31:38.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:38.549+0000 7ffb37e00700 1 --2- 172.21.15.62:0/1308772372 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7ffb30039840 0x7ffb3003bcf0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:31:38.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:38.549+0000 7ffb37e00700 1 --2- 172.21.15.62:0/1308772372 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffb4c102490 0x7ffb4c1938a0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:31:38.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:38.549+0000 7ffb37e00700 1 --2- 172.21.15.62:0/1308772372 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb4c103640 0x7ffb4c193de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:31:38.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:38.549+0000 7ffb37e00700 1 -- 172.21.15.62:0/1308772372 >> 172.21.15.62:0/1308772372 conn(0x7ffb4c0fda60 msgr2=0x7ffb4c0fe3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:31:38.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:38.550+0000 7ffb37e00700 1 -- 172.21.15.62:0/1308772372 shutdown_connections 2024-04-20T21:31:38.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:38.550+0000 7ffb37e00700 1 -- 172.21.15.62:0/1308772372 wait complete. 2024-04-20T21:31:38.566 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T21:31:38.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:38.764+0000 7f1e2b200700 1 -- 172.21.15.62:0/1895400321 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1e24075700 msgr2=0x7f1e24075b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:31:38.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:38.764+0000 7f1e2b200700 1 --2- 172.21.15.62:0/1895400321 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1e24075700 0x7f1e24075b10 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f1e14009910 tx=0x7f1e14009c20 comp rx=0 tx=0).stop 2024-04-20T21:31:38.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:38.764+0000 7f1e2b200700 1 -- 172.21.15.62:0/1895400321 shutdown_connections 2024-04-20T21:31:38.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:38.764+0000 7f1e2b200700 1 --2- 172.21.15.62:0/1895400321 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1e24076900 0x7f1e24076d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:31:38.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:38.764+0000 7f1e2b200700 1 --2- 172.21.15.62:0/1895400321 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1e24075700 0x7f1e24075b10 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:31:38.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:38.764+0000 7f1e2b200700 1 -- 172.21.15.62:0/1895400321 >> 172.21.15.62:0/1895400321 conn(0x7f1e240fda60 msgr2=0x7f1e240ffe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:31:38.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:38.764+0000 7f1e2b200700 1 -- 172.21.15.62:0/1895400321 shutdown_connections 2024-04-20T21:31:38.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:38.764+0000 7f1e2b200700 1 -- 172.21.15.62:0/1895400321 wait complete. 2024-04-20T21:31:38.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:38.766+0000 7f1e2b200700 1 Processor -- start 2024-04-20T21:31:38.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:38.766+0000 7f1e2b200700 1 -- start start 2024-04-20T21:31:38.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:38.766+0000 7f1e2b200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1e24075700 0x7f1e2410eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:31:38.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:38.766+0000 7f1e2b200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1e24076900 0x7f1e2410f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:31:38.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:38.766+0000 7f1e2b200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e2410f6f0 con 0x7f1e24075700 2024-04-20T21:31:38.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:38.766+0000 7f1e2b200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e2410f830 con 0x7f1e24076900 2024-04-20T21:31:38.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:38.767+0000 7f1e23e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1e24075700 0x7f1e2410eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:31:38.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:38.767+0000 7f1e23e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1e24075700 0x7f1e2410eb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49604/0 (socket says 172.21.15.62:49604) 2024-04-20T21:31:38.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:38.767+0000 7f1e23e00700 1 -- 172.21.15.62:0/4232601027 learned_addr learned my addr 172.21.15.62:0/4232601027 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:31:38.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:38.767+0000 7f1e23400700 1 --2- 172.21.15.62:0/4232601027 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1e24076900 0x7f1e2410f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:31:38.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:38.767+0000 7f1e23e00700 1 -- 172.21.15.62:0/4232601027 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1e24076900 msgr2=0x7f1e2410f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:31:38.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:38.767+0000 7f1e23e00700 1 --2- 172.21.15.62:0/4232601027 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1e24076900 0x7f1e2410f0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:31:38.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:38.767+0000 7f1e23e00700 1 -- 172.21.15.62:0/4232601027 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e140095f0 con 0x7f1e24075700 2024-04-20T21:31:38.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:38.767+0000 7f1e23e00700 1 --2- 172.21.15.62:0/4232601027 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1e24075700 0x7f1e2410eb90 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f1e140092e0 tx=0x7f1e14005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:31:38.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:38.768+0000 7f1e20c00700 1 -- 172.21.15.62:0/4232601027 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1e140052a0 con 0x7f1e24075700 2024-04-20T21:31:38.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:38.768+0000 7f1e20c00700 1 -- 172.21.15.62:0/4232601027 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f1e1402e070 con 0x7f1e24075700 2024-04-20T21:31:38.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:38.768+0000 7f1e2b200700 1 -- 172.21.15.62:0/4232601027 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e2406a770 con 0x7f1e24075700 2024-04-20T21:31:38.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:38.768+0000 7f1e20c00700 1 -- 172.21.15.62:0/4232601027 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1e14017400 con 0x7f1e24075700 2024-04-20T21:31:38.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:38.768+0000 7f1e2b200700 1 -- 172.21.15.62:0/4232601027 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e2406ac60 con 0x7f1e24075700 2024-04-20T21:31:38.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:38.769+0000 7f1e2b200700 1 -- 172.21.15.62:0/4232601027 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e24108450 con 0x7f1e24075700 2024-04-20T21:31:38.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:38.770+0000 7f1e20c00700 1 -- 172.21.15.62:0/4232601027 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 27) v1 ==== 46641+0+0 (secure 0 0 0) 0x7f1e14003710 con 0x7f1e24075700 2024-04-20T21:31:38.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:38.770+0000 7f1e20c00700 1 --2- 172.21.15.62:0/4232601027 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f1e08039840 0x7f1e0803bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:31:38.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:38.770+0000 7f1e20c00700 1 -- 172.21.15.62:0/4232601027 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f1e1405fea0 con 0x7f1e24075700 2024-04-20T21:31:38.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:38.771+0000 7f1e23400700 1 --2- 172.21.15.62:0/4232601027 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f1e08039840 0x7f1e0803bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:31:38.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:38.772+0000 7f1e23400700 1 --2- 172.21.15.62:0/4232601027 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f1e08039840 0x7f1e0803bcf0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f1e10009c60 tx=0x7f1e10009260 comp rx=0 tx=0).ready entity=mgr.14658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:31:38.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:38.776+0000 7f1e20c00700 1 -- 172.21.15.62:0/4232601027 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187527 (secure 0 0 0) 0x7f1e140167c0 con 0x7f1e24075700 2024-04-20T21:31:38.940 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:38 smithi062 ceph-mon[16221]: mgrmap e27: smithi062.rqxuxi(active, since 1.01948s) 2024-04-20T21:31:38.941 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:38 smithi062 ceph-mon[16221]: pgmap v3: 65 pgs: 65 active+clean; 190 MiB data, 791 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:31:38.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:38 smithi189 ceph-mon[21562]: mgrmap e27: smithi062.rqxuxi(active, since 1.01948s) 2024-04-20T21:31:38.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:38 smithi189 ceph-mon[21562]: pgmap v3: 65 pgs: 65 active+clean; 190 MiB data, 791 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:31:39.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:39.130+0000 7f1e2b200700 1 -- 172.21.15.62:0/4232601027 --> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1e24061190 con 0x7f1e08039840 2024-04-20T21:31:39.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:39.132+0000 7f1e20c00700 1 -- 172.21.15.62:0/4232601027 <== mgr.14658 v2:172.21.15.62:6800/1585760929 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+329 (secure 0 0 0) 0x7f1e24061190 con 0x7f1e08039840 2024-04-20T21:31:39.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:39.135+0000 7f1e2b200700 1 -- 172.21.15.62:0/4232601027 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f1e08039840 msgr2=0x7f1e0803bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:31:39.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:39.135+0000 7f1e2b200700 1 --2- 172.21.15.62:0/4232601027 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f1e08039840 0x7f1e0803bcf0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f1e10009c60 tx=0x7f1e10009260 comp rx=0 tx=0).stop 2024-04-20T21:31:39.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:39.135+0000 7f1e2b200700 1 -- 172.21.15.62:0/4232601027 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1e24075700 msgr2=0x7f1e2410eb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:31:39.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:39.135+0000 7f1e2b200700 1 --2- 172.21.15.62:0/4232601027 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1e24075700 0x7f1e2410eb90 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f1e140092e0 tx=0x7f1e14005dc0 comp rx=0 tx=0).stop 2024-04-20T21:31:39.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:39.136+0000 7f1e2b200700 1 -- 172.21.15.62:0/4232601027 shutdown_connections 2024-04-20T21:31:39.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:39.136+0000 7f1e2b200700 1 --2- 172.21.15.62:0/4232601027 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f1e08039840 0x7f1e0803bcf0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:31:39.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:39.136+0000 7f1e2b200700 1 --2- 172.21.15.62:0/4232601027 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1e24075700 0x7f1e2410eb90 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:31:39.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:39.136+0000 7f1e2b200700 1 --2- 172.21.15.62:0/4232601027 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1e24076900 0x7f1e2410f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:31:39.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:39.136+0000 7f1e2b200700 1 -- 172.21.15.62:0/4232601027 >> 172.21.15.62:0/4232601027 conn(0x7f1e240fda60 msgr2=0x7f1e24105590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:31:39.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:39.136+0000 7f1e2b200700 1 -- 172.21.15.62:0/4232601027 shutdown_connections 2024-04-20T21:31:39.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:39.136+0000 7f1e2b200700 1 -- 172.21.15.62:0/4232601027 wait complete. 2024-04-20T21:31:39.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:39.333+0000 7fcc7a600700 1 -- 172.21.15.62:0/427098135 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcc74100ab0 msgr2=0x7fcc74102e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:31:39.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:39.333+0000 7fcc7a600700 1 --2- 172.21.15.62:0/427098135 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcc74100ab0 0x7fcc74102e90 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fcc68009910 tx=0x7fcc68009c20 comp rx=0 tx=0).stop 2024-04-20T21:31:39.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:39.333+0000 7fcc7a600700 1 -- 172.21.15.62:0/427098135 shutdown_connections 2024-04-20T21:31:39.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:39.333+0000 7fcc7a600700 1 --2- 172.21.15.62:0/427098135 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc741033d0 0x7fcc741057b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:31:39.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:39.333+0000 7fcc7a600700 1 --2- 172.21.15.62:0/427098135 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcc74100ab0 0x7fcc74102e90 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:31:39.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:39.333+0000 7fcc7a600700 1 -- 172.21.15.62:0/427098135 >> 172.21.15.62:0/427098135 conn(0x7fcc740fa4d0 msgr2=0x7fcc740fc900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:31:39.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:39.333+0000 7fcc7a600700 1 -- 172.21.15.62:0/427098135 shutdown_connections 2024-04-20T21:31:39.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:39.334+0000 7fcc7a600700 1 -- 172.21.15.62:0/427098135 wait complete. 2024-04-20T21:31:39.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:39.335+0000 7fcc7a600700 1 Processor -- start 2024-04-20T21:31:39.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:39.335+0000 7fcc7a600700 1 -- start start 2024-04-20T21:31:39.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:39.336+0000 7fcc7a600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc74100ab0 0x7fcc74197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:31:39.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:39.336+0000 7fcc7a600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcc741033d0 0x7fcc74197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:31:39.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:39.336+0000 7fcc7a600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc74198560 con 0x7fcc741033d0 2024-04-20T21:31:39.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:39.336+0000 7fcc7a600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc741986a0 con 0x7fcc74100ab0 2024-04-20T21:31:39.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:39.336+0000 7fcc79200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc74100ab0 0x7fcc74197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:31:39.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:39.336+0000 7fcc73e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcc741033d0 0x7fcc74197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:31:39.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:39.337+0000 7fcc79200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc74100ab0 0x7fcc74197a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.62:38240/0 (socket says 172.21.15.62:38240) 2024-04-20T21:31:39.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:39.337+0000 7fcc79200700 1 -- 172.21.15.62:0/2469536360 learned_addr learned my addr 172.21.15.62:0/2469536360 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:31:39.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:39.338+0000 7fcc79200700 1 -- 172.21.15.62:0/2469536360 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcc741033d0 msgr2=0x7fcc74197f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:31:39.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:39.338+0000 7fcc79200700 1 --2- 172.21.15.62:0/2469536360 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcc741033d0 0x7fcc74197f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:31:39.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:39.338+0000 7fcc79200700 1 -- 172.21.15.62:0/2469536360 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc680095f0 con 0x7fcc74100ab0 2024-04-20T21:31:39.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:39.338+0000 7fcc73e00700 1 --2- 172.21.15.62:0/2469536360 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcc741033d0 0x7fcc74197f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-20T21:31:39.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:39.341+0000 7fcc79200700 1 --2- 172.21.15.62:0/2469536360 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc74100ab0 0x7fcc74197a00 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fcc680092e0 tx=0x7fcc6800f690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:31:39.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:39.341+0000 7fcc71600700 1 -- 172.21.15.62:0/2469536360 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcc6801d070 con 0x7fcc74100ab0 2024-04-20T21:31:39.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:39.341+0000 7fcc71600700 1 -- 172.21.15.62:0/2469536360 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fcc68005dc0 con 0x7fcc74100ab0 2024-04-20T21:31:39.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:39.341+0000 7fcc7a600700 1 -- 172.21.15.62:0/2469536360 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc7419d0f0 con 0x7fcc74100ab0 2024-04-20T21:31:39.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:39.341+0000 7fcc71600700 1 -- 172.21.15.62:0/2469536360 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcc68017630 con 0x7fcc74100ab0 2024-04-20T21:31:39.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:39.341+0000 7fcc7a600700 1 -- 172.21.15.62:0/2469536360 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc7419d5e0 con 0x7fcc74100ab0 2024-04-20T21:31:39.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:39.343+0000 7fcc7a600700 1 -- 172.21.15.62:0/2469536360 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc740fc090 con 0x7fcc74100ab0 2024-04-20T21:31:39.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:39.343+0000 7fcc71600700 1 -- 172.21.15.62:0/2469536360 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) v1 ==== 46641+0+0 (secure 0 0 0) 0x7fcc6800faf0 con 0x7fcc74100ab0 2024-04-20T21:31:39.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:39.344+0000 7fcc71600700 1 --2- 172.21.15.62:0/2469536360 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7fcc5c039840 0x7fcc5c03bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:31:39.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:39.344+0000 7fcc71600700 1 -- 172.21.15.62:0/2469536360 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6070+0+0 (secure 0 0 0) 0x7fcc6804f070 con 0x7fcc74100ab0 2024-04-20T21:31:39.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:39.344+0000 7fcc73e00700 1 --2- 172.21.15.62:0/2469536360 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7fcc5c039840 0x7fcc5c03bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:31:39.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:39.345+0000 7fcc73e00700 1 --2- 172.21.15.62:0/2469536360 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7fcc5c039840 0x7fcc5c03bcf0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fcc60011490 tx=0x7fcc60009260 comp rx=0 tx=0).ready entity=mgr.14658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:31:39.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:39.350+0000 7fcc71600700 1 -- 172.21.15.62:0/2469536360 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187527 (secure 0 0 0) 0x7fcc68083ca0 con 0x7fcc74100ab0 2024-04-20T21:31:39.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:39.579+0000 7fcc71600700 1 -- 172.21.15.62:0/2469536360 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 28) v1 ==== 46697+0+0 (secure 0 0 0) 0x7fcc68003680 con 0x7fcc74100ab0 2024-04-20T21:31:39.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:39.706+0000 7fcc7a600700 1 -- 172.21.15.62:0/2469536360 --> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fcc74061190 con 0x7fcc5c039840 2024-04-20T21:31:39.706 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:39 smithi062 ceph-mon[16221]: [20/Apr/2024:21:31:37] ENGINE Bus STARTING 2024-04-20T21:31:39.706 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:39 smithi062 ceph-mon[16221]: [20/Apr/2024:21:31:38] ENGINE Serving on http://172.21.15.62:8765 2024-04-20T21:31:39.706 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:39 smithi062 ceph-mon[16221]: [20/Apr/2024:21:31:38] ENGINE Serving on https://172.21.15.62:7150 2024-04-20T21:31:39.706 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:39 smithi062 ceph-mon[16221]: [20/Apr/2024:21:31:38] ENGINE Bus STARTED 2024-04-20T21:31:39.706 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:39 smithi062 ceph-mon[16221]: from='client.14684 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:31:39.707 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:39 smithi062 ceph-mon[16221]: pgmap v4: 65 pgs: 65 active+clean; 190 MiB data, 791 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:31:39.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:39.723+0000 7fcc71600700 1 -- 172.21.15.62:0/2469536360 <== mgr.14658 v2:172.21.15.62:6800/1585760929 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fcc74061190 con 0x7fcc5c039840 2024-04-20T21:31:39.723 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T21:31:39.723 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (9m) 11s ago 11m 14.0M - 0.25.0 c8568f914cd2 957f663a9cd7 2024-04-20T21:31:39.723 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (11m) 11s ago 11m 7902k - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T21:31:39.723 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (10m) 52s ago 10m 7684k - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T21:31:39.723 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (11m) 11s ago 11m 6656k - 18.2.0 dc2bc1663786 36e10ef5be39 2024-04-20T21:31:39.724 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (10m) 52s ago 10m 6665k - 18.2.0 dc2bc1663786 4101f9953e3d 2024-04-20T21:31:39.724 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (9m) 11s ago 10m 75.5M - 9.4.7 954c08fa6188 99363bb095d8 2024-04-20T21:31:39.724 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (4m) 11s ago 4m 108M - 18.2.0 dc2bc1663786 5e6460aa6159 2024-04-20T21:31:39.724 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (4m) 11s ago 4m 102M - 18.2.0 dc2bc1663786 e65f2267544e 2024-04-20T21:31:39.724 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (4m) 52s ago 4m 73.3M - 18.2.0 dc2bc1663786 fb2a8a223db8 2024-04-20T21:31:39.724 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (4m) 52s ago 4m 53.0M - 18.2.0 dc2bc1663786 f2eec2ee8951 2024-04-20T21:31:39.724 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (23s) 11s ago 13m 64.4M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T21:31:39.724 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (83s) 52s ago 10m 488M - 19.0.0-3133-ga8c84f59 76bc6b91b726 4cde2bc4de24 2024-04-20T21:31:39.724 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (13m) 11s ago 13m 51.1M 2048M 18.2.0 dc2bc1663786 bd6efbd5c380 2024-04-20T21:31:39.724 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (9m) 52s ago 9m 40.4M 2048M 18.2.0 dc2bc1663786 3735c8e5db9a 2024-04-20T21:31:39.724 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (11m) 11s ago 11m 7776k - 1.5.0 0da6a335fe13 fc746d8ae72e 2024-04-20T21:31:39.725 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (10m) 52s ago 10m 8363k - 1.5.0 0da6a335fe13 db03af85b286 2024-04-20T21:31:39.725 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (8m) 11s ago 8m 80.7M 1216M 18.2.0 dc2bc1663786 3f315c3afc1f 2024-04-20T21:31:39.725 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (8m) 11s ago 8m 85.8M 1216M 18.2.0 dc2bc1663786 44e4d07416ad 2024-04-20T21:31:39.725 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (7m) 11s ago 7m 72.6M 1216M 18.2.0 dc2bc1663786 31c5d9586e98 2024-04-20T21:31:39.725 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (7m) 52s ago 7m 73.6M 2240M 18.2.0 dc2bc1663786 611c1e242693 2024-04-20T21:31:39.725 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (6m) 52s ago 6m 68.5M 2240M 18.2.0 dc2bc1663786 6c8fce91d45d 2024-04-20T21:31:39.725 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (6m) 52s ago 6m 67.1M 2240M 18.2.0 dc2bc1663786 8e4574bae8db 2024-04-20T21:31:39.725 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (32s) 11s ago 10m 45.5M - 2.43.0 a07b618ecd1d 6777f15d9248 2024-04-20T21:31:39.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:39.727+0000 7fcc7a600700 1 -- 172.21.15.62:0/2469536360 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7fcc5c039840 msgr2=0x7fcc5c03bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:31:39.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:39.727+0000 7fcc7a600700 1 --2- 172.21.15.62:0/2469536360 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7fcc5c039840 0x7fcc5c03bcf0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fcc60011490 tx=0x7fcc60009260 comp rx=0 tx=0).stop 2024-04-20T21:31:39.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:39.727+0000 7fcc7a600700 1 -- 172.21.15.62:0/2469536360 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc74100ab0 msgr2=0x7fcc74197a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:31:39.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:39.727+0000 7fcc7a600700 1 --2- 172.21.15.62:0/2469536360 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc74100ab0 0x7fcc74197a00 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fcc680092e0 tx=0x7fcc6800f690 comp rx=0 tx=0).stop 2024-04-20T21:31:39.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:39.727+0000 7fcc7a600700 1 -- 172.21.15.62:0/2469536360 shutdown_connections 2024-04-20T21:31:39.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:39.727+0000 7fcc7a600700 1 --2- 172.21.15.62:0/2469536360 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7fcc5c039840 0x7fcc5c03bcf0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:31:39.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:39.727+0000 7fcc7a600700 1 --2- 172.21.15.62:0/2469536360 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc74100ab0 0x7fcc74197a00 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:31:39.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:39.727+0000 7fcc7a600700 1 --2- 172.21.15.62:0/2469536360 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcc741033d0 0x7fcc74197f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:31:39.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:39.727+0000 7fcc7a600700 1 -- 172.21.15.62:0/2469536360 >> 172.21.15.62:0/2469536360 conn(0x7fcc740fa4d0 msgr2=0x7fcc740ff130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:31:39.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:39.727+0000 7fcc7a600700 1 -- 172.21.15.62:0/2469536360 shutdown_connections 2024-04-20T21:31:39.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:39.727+0000 7fcc7a600700 1 -- 172.21.15.62:0/2469536360 wait complete. 2024-04-20T21:31:39.854 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:39 smithi189 ceph-mon[21562]: [20/Apr/2024:21:31:37] ENGINE Bus STARTING 2024-04-20T21:31:39.854 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:39 smithi189 ceph-mon[21562]: [20/Apr/2024:21:31:38] ENGINE Serving on http://172.21.15.62:8765 2024-04-20T21:31:39.854 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:39 smithi189 ceph-mon[21562]: [20/Apr/2024:21:31:38] ENGINE Serving on https://172.21.15.62:7150 2024-04-20T21:31:39.854 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:39 smithi189 ceph-mon[21562]: [20/Apr/2024:21:31:38] ENGINE Bus STARTED 2024-04-20T21:31:39.855 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:39 smithi189 ceph-mon[21562]: from='client.14684 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:31:39.855 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:39 smithi189 ceph-mon[21562]: pgmap v4: 65 pgs: 65 active+clean; 190 MiB data, 791 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:31:39.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:39.908+0000 7f685e000700 1 -- 172.21.15.62:0/194754470 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6858100080 msgr2=0x7f6858100490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:31:39.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:39.908+0000 7f685e000700 1 --2- 172.21.15.62:0/194754470 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6858100080 0x7f6858100490 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f684c009870 tx=0x7f684c009b80 comp rx=0 tx=0).stop 2024-04-20T21:31:39.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:39.908+0000 7f685e000700 1 -- 172.21.15.62:0/194754470 shutdown_connections 2024-04-20T21:31:39.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:39.908+0000 7f685e000700 1 --2- 172.21.15.62:0/194754470 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6858101280 0x7f68581016d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:31:39.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:39.908+0000 7f685e000700 1 --2- 172.21.15.62:0/194754470 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6858100080 0x7f6858100490 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:31:39.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:39.908+0000 7f685e000700 1 -- 172.21.15.62:0/194754470 >> 172.21.15.62:0/194754470 conn(0x7f68580fb610 msgr2=0x7f68580fda60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:31:39.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:39.909+0000 7f685e000700 1 -- 172.21.15.62:0/194754470 shutdown_connections 2024-04-20T21:31:39.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:39.909+0000 7f685e000700 1 -- 172.21.15.62:0/194754470 wait complete. 2024-04-20T21:31:39.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:39.910+0000 7f685e000700 1 Processor -- start 2024-04-20T21:31:39.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:39.911+0000 7f685e000700 1 -- start start 2024-04-20T21:31:39.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:39.911+0000 7f685e000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6858101280 0x7f6858071f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:31:39.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:39.911+0000 7f685e000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68580724b0 0x7f6858073500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:31:39.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:39.911+0000 7f685e000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68581a33d0 con 0x7f6858101280 2024-04-20T21:31:39.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:39.911+0000 7f685e000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68581a3540 con 0x7f68580724b0 2024-04-20T21:31:39.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:39.911+0000 7f685cc00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6858101280 0x7f6858071f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:31:39.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:39.912+0000 7f685cc00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6858101280 0x7f6858071f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49636/0 (socket says 172.21.15.62:49636) 2024-04-20T21:31:39.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:39.912+0000 7f685cc00700 1 -- 172.21.15.62:0/3263404199 learned_addr learned my addr 172.21.15.62:0/3263404199 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:31:39.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:39.912+0000 7f6857e00700 1 --2- 172.21.15.62:0/3263404199 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68580724b0 0x7f6858073500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:31:39.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:39.912+0000 7f685cc00700 1 -- 172.21.15.62:0/3263404199 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68580724b0 msgr2=0x7f6858073500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:31:39.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:39.912+0000 7f685cc00700 1 --2- 172.21.15.62:0/3263404199 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68580724b0 0x7f6858073500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:31:39.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:39.912+0000 7f685cc00700 1 -- 172.21.15.62:0/3263404199 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f684c009520 con 0x7f6858101280 2024-04-20T21:31:39.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:39.912+0000 7f6857e00700 1 --2- 172.21.15.62:0/3263404199 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68580724b0 0x7f6858073500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-20T21:31:39.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:39.913+0000 7f685cc00700 1 --2- 172.21.15.62:0/3263404199 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6858101280 0x7f6858071f70 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f684c009230 tx=0x7f684c00f740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:31:39.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:39.913+0000 7f6855600700 1 -- 172.21.15.62:0/3263404199 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f684c01c070 con 0x7f6858101280 2024-04-20T21:31:39.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:39.913+0000 7f6855600700 1 -- 172.21.15.62:0/3263404199 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f684c00b4e0 con 0x7f6858101280 2024-04-20T21:31:39.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:39.913+0000 7f685e000700 1 -- 172.21.15.62:0/3263404199 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68581a37c0 con 0x7f6858101280 2024-04-20T21:31:39.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:39.914+0000 7f6855600700 1 -- 172.21.15.62:0/3263404199 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f684c022d90 con 0x7f6858101280 2024-04-20T21:31:39.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:39.914+0000 7f685e000700 1 -- 172.21.15.62:0/3263404199 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68581a3cb0 con 0x7f6858101280 2024-04-20T21:31:39.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:39.915+0000 7f685e000700 1 -- 172.21.15.62:0/3263404199 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6858108bd0 con 0x7f6858101280 2024-04-20T21:31:39.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:39.915+0000 7f6855600700 1 -- 172.21.15.62:0/3263404199 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 28) v1 ==== 46697+0+0 (secure 0 0 0) 0x7f684c020070 con 0x7f6858101280 2024-04-20T21:31:39.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:39.916+0000 7f6855600700 1 --2- 172.21.15.62:0/3263404199 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f68400398c0 0x7f684003bd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:31:39.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:39.916+0000 7f6855600700 1 -- 172.21.15.62:0/3263404199 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f684c04ecb0 con 0x7f6858101280 2024-04-20T21:31:39.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:39.917+0000 7f6857e00700 1 --2- 172.21.15.62:0/3263404199 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f68400398c0 0x7f684003bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:31:39.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:39.918+0000 7f6857e00700 1 --2- 172.21.15.62:0/3263404199 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f68400398c0 0x7f684003bd70 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f6844005fd0 tx=0x7f6844009310 comp rx=0 tx=0).ready entity=mgr.14658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:31:39.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:39.922+0000 7f6855600700 1 -- 172.21.15.62:0/3263404199 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187527 (secure 0 0 0) 0x7f684c0173f0 con 0x7f6858101280 2024-04-20T21:31:40.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:40.346+0000 7f685e000700 1 -- 172.21.15.62:0/3263404199 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f6858073db0 con 0x7f6858101280 2024-04-20T21:31:40.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:40.347+0000 7f6855600700 1 -- 172.21.15.62:0/3263404199 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+675 (secure 0 0 0) 0x7f684c027020 con 0x7f6858101280 2024-04-20T21:31:40.347 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:31:40.347 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T21:31:40.347 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 2 2024-04-20T21:31:40.347 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:31:40.347 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T21:31:40.347 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 1 2024-04-20T21:31:40.347 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:31:40.347 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T21:31:40.347 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 6 2024-04-20T21:31:40.347 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:31:40.347 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T21:31:40.348 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 4 2024-04-20T21:31:40.348 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:31:40.348 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T21:31:40.348 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 12, 2024-04-20T21:31:40.348 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 1 2024-04-20T21:31:40.348 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T21:31:40.348 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:31:40.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:40.350+0000 7f685e000700 1 -- 172.21.15.62:0/3263404199 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f68400398c0 msgr2=0x7f684003bd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:31:40.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:40.350+0000 7f685e000700 1 --2- 172.21.15.62:0/3263404199 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f68400398c0 0x7f684003bd70 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f6844005fd0 tx=0x7f6844009310 comp rx=0 tx=0).stop 2024-04-20T21:31:40.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:40.350+0000 7f685e000700 1 -- 172.21.15.62:0/3263404199 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6858101280 msgr2=0x7f6858071f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:31:40.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:40.350+0000 7f685e000700 1 --2- 172.21.15.62:0/3263404199 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6858101280 0x7f6858071f70 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f684c009230 tx=0x7f684c00f740 comp rx=0 tx=0).stop 2024-04-20T21:31:40.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:40.351+0000 7f685e000700 1 -- 172.21.15.62:0/3263404199 shutdown_connections 2024-04-20T21:31:40.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:40.351+0000 7f685e000700 1 --2- 172.21.15.62:0/3263404199 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f68400398c0 0x7f684003bd70 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:31:40.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:40.351+0000 7f685e000700 1 --2- 172.21.15.62:0/3263404199 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6858101280 0x7f6858071f70 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:31:40.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:40.351+0000 7f685e000700 1 --2- 172.21.15.62:0/3263404199 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68580724b0 0x7f6858073500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:31:40.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:40.351+0000 7f685e000700 1 -- 172.21.15.62:0/3263404199 >> 172.21.15.62:0/3263404199 conn(0x7f68580fb610 msgr2=0x7f68581044b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:31:40.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:40.351+0000 7f685e000700 1 -- 172.21.15.62:0/3263404199 shutdown_connections 2024-04-20T21:31:40.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:40.351+0000 7f685e000700 1 -- 172.21.15.62:0/3263404199 wait complete. 2024-04-20T21:31:40.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:40.545+0000 7f5656e00700 1 -- 172.21.15.62:0/788979662 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5650074d20 msgr2=0x7f5650073180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:31:40.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:40.545+0000 7f5656e00700 1 --2- 172.21.15.62:0/788979662 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5650074d20 0x7f5650073180 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f5640009910 tx=0x7f5640009c20 comp rx=0 tx=0).stop 2024-04-20T21:31:40.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:40.545+0000 7f5656e00700 1 -- 172.21.15.62:0/788979662 shutdown_connections 2024-04-20T21:31:40.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:40.545+0000 7f5656e00700 1 --2- 172.21.15.62:0/788979662 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5650073750 0x7f5650073bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:31:40.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:40.545+0000 7f5656e00700 1 --2- 172.21.15.62:0/788979662 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5650074d20 0x7f5650073180 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:31:40.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:40.546+0000 7f5656e00700 1 -- 172.21.15.62:0/788979662 >> 172.21.15.62:0/788979662 conn(0x7f56500fb7f0 msgr2=0x7f56500fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:31:40.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:40.546+0000 7f5656e00700 1 -- 172.21.15.62:0/788979662 shutdown_connections 2024-04-20T21:31:40.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:40.546+0000 7f5656e00700 1 -- 172.21.15.62:0/788979662 wait complete. 2024-04-20T21:31:40.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:40.547+0000 7f5656e00700 1 Processor -- start 2024-04-20T21:31:40.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:40.548+0000 7f5656e00700 1 -- start start 2024-04-20T21:31:40.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:40.548+0000 7f5656e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5650073750 0x7f5650197ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:31:40.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:40.548+0000 7f5656e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5650074d20 0x7f5650198220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:31:40.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:40.548+0000 7f5656e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5650198840 con 0x7f5650074d20 2024-04-20T21:31:40.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:40.548+0000 7f5656e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f565019d250 con 0x7f5650073750 2024-04-20T21:31:40.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:40.549+0000 7f564fe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5650073750 0x7f5650197ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:31:40.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:40.549+0000 7f564f400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5650074d20 0x7f5650198220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:31:40.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:40.549+0000 7f564f400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5650074d20 0x7f5650198220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49652/0 (socket says 172.21.15.62:49652) 2024-04-20T21:31:40.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:40.549+0000 7f564f400700 1 -- 172.21.15.62:0/1225042040 learned_addr learned my addr 172.21.15.62:0/1225042040 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:31:40.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:40.549+0000 7f564fe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5650073750 0x7f5650197ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.62:38278/0 (socket says 172.21.15.62:38278) 2024-04-20T21:31:40.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:40.549+0000 7f564f400700 1 -- 172.21.15.62:0/1225042040 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5650073750 msgr2=0x7f5650197ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:31:40.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:40.549+0000 7f564f400700 1 --2- 172.21.15.62:0/1225042040 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5650073750 0x7f5650197ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:31:40.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:40.549+0000 7f564f400700 1 -- 172.21.15.62:0/1225042040 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56400095f0 con 0x7f5650074d20 2024-04-20T21:31:40.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:40.550+0000 7f564f400700 1 --2- 172.21.15.62:0/1225042040 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5650074d20 0x7f5650198220 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f563c00d8d0 tx=0x7f563c00dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:31:40.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:40.550+0000 7f564cc00700 1 -- 172.21.15.62:0/1225042040 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f563c0098a0 con 0x7f5650074d20 2024-04-20T21:31:40.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:40.550+0000 7f5656e00700 1 -- 172.21.15.62:0/1225042040 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f565019d450 con 0x7f5650074d20 2024-04-20T21:31:40.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:40.550+0000 7f564cc00700 1 -- 172.21.15.62:0/1225042040 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f563c010460 con 0x7f5650074d20 2024-04-20T21:31:40.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:40.550+0000 7f564cc00700 1 -- 172.21.15.62:0/1225042040 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f563c00f5f0 con 0x7f5650074d20 2024-04-20T21:31:40.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:40.550+0000 7f5656e00700 1 -- 172.21.15.62:0/1225042040 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f565019d970 con 0x7f5650074d20 2024-04-20T21:31:40.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:40.552+0000 7f564cc00700 1 -- 172.21.15.62:0/1225042040 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 28) v1 ==== 46697+0+0 (secure 0 0 0) 0x7f563c00f750 con 0x7f5650074d20 2024-04-20T21:31:40.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:40.552+0000 7f5656e00700 1 -- 172.21.15.62:0/1225042040 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f565010af80 con 0x7f5650074d20 2024-04-20T21:31:40.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:40.552+0000 7f564cc00700 1 --2- 172.21.15.62:0/1225042040 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f56340398c0 0x7f563403bd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:31:40.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:40.552+0000 7f564cc00700 1 -- 172.21.15.62:0/1225042040 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f563c0226c0 con 0x7f5650074d20 2024-04-20T21:31:40.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:40.554+0000 7f564fe00700 1 --2- 172.21.15.62:0/1225042040 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f56340398c0 0x7f563403bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:31:40.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:40.554+0000 7f564fe00700 1 --2- 172.21.15.62:0/1225042040 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f56340398c0 0x7f563403bd70 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f5640006010 tx=0x7f564000b560 comp rx=0 tx=0).ready entity=mgr.14658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:31:40.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:40.560+0000 7f564cc00700 1 -- 172.21.15.62:0/1225042040 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187527 (secure 0 0 0) 0x7f563c016020 con 0x7f5650074d20 2024-04-20T21:31:40.708 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:40 smithi189 ceph-mon[21562]: from='client.14692 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:31:40.708 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:40 smithi189 ceph-mon[21562]: mgrmap e28: smithi062.rqxuxi(active, since 3s) 2024-04-20T21:31:40.708 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:40 smithi189 ceph-mon[21562]: from='client.24473 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:31:40.708 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:40 smithi189 ceph-mon[21562]: from='client.? 172.21.15.62:0/3263404199' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:31:40.784 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:40 smithi062 ceph-mon[16221]: from='client.14692 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:31:40.784 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:40 smithi062 ceph-mon[16221]: mgrmap e28: smithi062.rqxuxi(active, since 3s) 2024-04-20T21:31:40.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:40 smithi062 ceph-mon[16221]: from='client.24473 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:31:40.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:40 smithi062 ceph-mon[16221]: from='client.? 172.21.15.62:0/3263404199' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:31:40.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:40.897+0000 7f5656e00700 1 -- 172.21.15.62:0/1225042040 --> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f56501077a0 con 0x7f56340398c0 2024-04-20T21:31:40.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:40.899+0000 7f564cc00700 1 -- 172.21.15.62:0/1225042040 <== mgr.14658 v2:172.21.15.62:6800/1585760929 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+329 (secure 0 0 0) 0x7f56501077a0 con 0x7f56340398c0 2024-04-20T21:31:40.900 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:31:40.900 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T21:31:40.900 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T21:31:40.900 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-20T21:31:40.900 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T21:31:40.900 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr" 2024-04-20T21:31:40.900 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T21:31:40.900 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "2/2 daemons upgraded", 2024-04-20T21:31:40.900 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "", 2024-04-20T21:31:40.900 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T21:31:40.900 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:31:40.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:40.901+0000 7f5656e00700 1 -- 172.21.15.62:0/1225042040 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f56340398c0 msgr2=0x7f563403bd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:31:40.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:40.901+0000 7f5656e00700 1 --2- 172.21.15.62:0/1225042040 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f56340398c0 0x7f563403bd70 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f5640006010 tx=0x7f564000b560 comp rx=0 tx=0).stop 2024-04-20T21:31:40.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:40.902+0000 7f5656e00700 1 -- 172.21.15.62:0/1225042040 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5650074d20 msgr2=0x7f5650198220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:31:40.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:40.902+0000 7f5656e00700 1 --2- 172.21.15.62:0/1225042040 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5650074d20 0x7f5650198220 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f563c00d8d0 tx=0x7f563c00dbe0 comp rx=0 tx=0).stop 2024-04-20T21:31:40.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:40.902+0000 7f5656e00700 1 -- 172.21.15.62:0/1225042040 shutdown_connections 2024-04-20T21:31:40.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:40.902+0000 7f5656e00700 1 --2- 172.21.15.62:0/1225042040 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f56340398c0 0x7f563403bd70 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:31:40.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:40.902+0000 7f5656e00700 1 --2- 172.21.15.62:0/1225042040 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5650073750 0x7f5650197ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:31:40.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:40.902+0000 7f5656e00700 1 --2- 172.21.15.62:0/1225042040 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5650074d20 0x7f5650198220 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:31:40.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:40.902+0000 7f5656e00700 1 -- 172.21.15.62:0/1225042040 >> 172.21.15.62:0/1225042040 conn(0x7f56500fb7f0 msgr2=0x7f5650106080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:31:40.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:40.902+0000 7f5656e00700 1 -- 172.21.15.62:0/1225042040 shutdown_connections 2024-04-20T21:31:40.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:31:40.902+0000 7f5656e00700 1 -- 172.21.15.62:0/1225042040 wait complete. 2024-04-20T21:31:41.986 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:41 smithi189 ceph-mon[21562]: pgmap v5: 65 pgs: 65 active+clean; 190 MiB data, 791 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:31:42.013 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:41 smithi062 ceph-mon[16221]: pgmap v5: 65 pgs: 65 active+clean; 190 MiB data, 791 MiB used, 536 GiB / 536 GiB avail 2024-04-20T21:31:42.959 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:42 smithi062 ceph-mon[16221]: from='client.14704 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:31:42.959 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:42 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:31:42.959 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:42 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:31:42.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:42 smithi189 ceph-mon[21562]: from='client.14704 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:31:42.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:42 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:31:42.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:42 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:31:43.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:43 smithi062 ceph-mon[16221]: pgmap v6: 65 pgs: 65 active+clean; 193 MiB data, 804 MiB used, 536 GiB / 536 GiB avail; 274 KiB/s rd, 800 KiB/s wr, 91 op/s 2024-04-20T21:31:43.901 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:43 smithi189 ceph-mon[21562]: pgmap v6: 65 pgs: 65 active+clean; 193 MiB data, 804 MiB used, 536 GiB / 536 GiB avail; 274 KiB/s rd, 800 KiB/s wr, 91 op/s 2024-04-20T21:31:45.178 INFO:tasks.workunit.client.0.smithi062.stderr:+ pushd ltp-full-20091231/testcases/kernel/fs/fsstress 2024-04-20T21:31:45.237 INFO:tasks.workunit.client.0.smithi062.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress ~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp 2024-04-20T21:31:45.237 INFO:tasks.workunit.client.0.smithi062.stderr:+ make 2024-04-20T21:31:45.424 INFO:tasks.workunit.client.0.smithi062.stdout:cc -DNO_XFS -I/home/ubuntu/cephtest/mnt.0/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress -D_LARGEFILE64_SOURCE -D_GNU_SOURCE -I../../../../include -I../../../../include -L../../../../lib fsstress.c -o fsstress 2024-04-20T21:31:45.430 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:45 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:31:45.430 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:45 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:31:45.430 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:45 smithi189 ceph-mon[21562]: pgmap v7: 65 pgs: 65 active+clean; 203 MiB data, 811 MiB used, 536 GiB / 536 GiB avail; 759 KiB/s rd, 1.9 MiB/s wr, 211 op/s 2024-04-20T21:31:45.534 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:45 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:31:45.534 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:45 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:31:45.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:45 smithi062 ceph-mon[16221]: pgmap v7: 65 pgs: 65 active+clean; 203 MiB data, 811 MiB used, 536 GiB / 536 GiB avail; 759 KiB/s rd, 1.9 MiB/s wr, 211 op/s 2024-04-20T21:31:47.149 INFO:tasks.workunit.client.0.smithi062.stderr:++ readlink -f fsstress 2024-04-20T21:31:47.155 INFO:tasks.workunit.client.0.smithi062.stderr:+ BIN=/home/ubuntu/cephtest/mnt.0/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress/fsstress 2024-04-20T21:31:47.155 INFO:tasks.workunit.client.0.smithi062.stderr:+ popd 2024-04-20T21:31:47.158 INFO:tasks.workunit.client.0.smithi062.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp 2024-04-20T21:31:47.159 INFO:tasks.workunit.client.0.smithi062.stderr:+ popd 2024-04-20T21:31:47.160 INFO:tasks.workunit.client.0.smithi062.stdout:~/cephtest/mnt.0/client.0/tmp 2024-04-20T21:31:47.161 INFO:tasks.workunit.client.0.smithi062.stderr:++ mktemp -d -p . 2024-04-20T21:31:47.167 INFO:tasks.workunit.client.0.smithi062.stderr:+ T=./tmp.uS6ubwsDVL 2024-04-20T21:31:47.167 INFO:tasks.workunit.client.0.smithi062.stderr:+ /home/ubuntu/cephtest/mnt.0/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress/fsstress -d ./tmp.uS6ubwsDVL -l 1 -n 1000 -p 10 -v 2024-04-20T21:31:47.175 INFO:tasks.workunit.client.0.smithi062.stdout:seed = 1713799332 2024-04-20T21:31:47.222 INFO:tasks.workunit.client.0.smithi062.stdout:0/0: stat - no entries 2024-04-20T21:31:47.223 INFO:tasks.workunit.client.0.smithi062.stdout:0/1: dread - no filename 2024-04-20T21:31:47.225 INFO:tasks.workunit.client.0.smithi062.stdout:1/0: write - no filename 2024-04-20T21:31:47.226 INFO:tasks.workunit.client.0.smithi062.stdout:0/2: mknod c0 0 2024-04-20T21:31:47.226 INFO:tasks.workunit.client.0.smithi062.stdout:0/3: fsync - no filename 2024-04-20T21:31:47.226 INFO:tasks.workunit.client.0.smithi062.stdout:0/4: fsync - no filename 2024-04-20T21:31:47.226 INFO:tasks.workunit.client.0.smithi062.stdout:0/5: read - no filename 2024-04-20T21:31:47.229 INFO:tasks.workunit.client.0.smithi062.stdout:0/6: creat f1 x:0 0 0 2024-04-20T21:31:47.231 INFO:tasks.workunit.client.0.smithi062.stdout:1/1: symlink l0 0 2024-04-20T21:31:47.231 INFO:tasks.workunit.client.0.smithi062.stdout:1/2: dread - no filename 2024-04-20T21:31:47.232 INFO:tasks.workunit.client.0.smithi062.stdout:0/7: write f1 [645069,114547] 0 2024-04-20T21:31:47.232 INFO:tasks.workunit.client.0.smithi062.stdout:0/8: rmdir - no directory 2024-04-20T21:31:47.233 INFO:tasks.workunit.client.0.smithi062.stdout:1/3: chown l0 9 1 2024-04-20T21:31:47.233 INFO:tasks.workunit.client.0.smithi062.stdout:1/4: fdatasync - no filename 2024-04-20T21:31:47.233 INFO:tasks.workunit.client.0.smithi062.stdout:1/5: dwrite - no filename 2024-04-20T21:31:47.234 INFO:tasks.workunit.client.0.smithi062.stdout:2/0: symlink l0 0 2024-04-20T21:31:47.234 INFO:tasks.workunit.client.0.smithi062.stdout:2/1: dwrite - no filename 2024-04-20T21:31:47.234 INFO:tasks.workunit.client.0.smithi062.stdout:2/2: rmdir - no directory 2024-04-20T21:31:47.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:46 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:31:47.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:46 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:31:47.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:46 smithi189 ceph-mon[21562]: pgmap v8: 65 pgs: 65 active+clean; 203 MiB data, 811 MiB used, 536 GiB / 536 GiB avail; 590 KiB/s rd, 1.5 MiB/s wr, 164 op/s 2024-04-20T21:31:47.238 INFO:tasks.workunit.client.0.smithi062.stdout:4/0: chown . 0 1 2024-04-20T21:31:47.238 INFO:tasks.workunit.client.0.smithi062.stdout:4/1: fdatasync - no filename 2024-04-20T21:31:47.240 INFO:tasks.workunit.client.0.smithi062.stdout:0/9: creat f2 x:0 0 0 2024-04-20T21:31:47.242 INFO:tasks.workunit.client.0.smithi062.stdout:2/3: mknod c1 0 2024-04-20T21:31:47.246 INFO:tasks.workunit.client.0.smithi062.stdout:1/6: unlink l0 0 2024-04-20T21:31:47.249 INFO:tasks.workunit.client.0.smithi062.stdout:3/0: mkdir d0 0 2024-04-20T21:31:47.249 INFO:tasks.workunit.client.0.smithi062.stdout:3/1: chown d0 2065730 1 2024-04-20T21:31:47.259 INFO:tasks.workunit.client.0.smithi062.stdout:4/2: mknod c0 0 2024-04-20T21:31:47.264 INFO:tasks.workunit.client.0.smithi062.stdout:0/10: dwrite f2 [0,4194304] 0 2024-04-20T21:31:47.265 INFO:tasks.workunit.client.0.smithi062.stdout:3/2: mkdir d0/d1 0 2024-04-20T21:31:47.267 INFO:tasks.workunit.client.0.smithi062.stdout:3/3: rename d0 to d0/d2 22 2024-04-20T21:31:47.267 INFO:tasks.workunit.client.0.smithi062.stdout:3/4: write - no filename 2024-04-20T21:31:47.270 INFO:tasks.workunit.client.0.smithi062.stdout:4/3: creat f1 x:0 0 0 2024-04-20T21:31:47.272 INFO:tasks.workunit.client.0.smithi062.stdout:2/4: link c1 c2 0 2024-04-20T21:31:47.272 INFO:tasks.workunit.client.0.smithi062.stdout:2/5: read - no filename 2024-04-20T21:31:47.272 INFO:tasks.workunit.client.0.smithi062.stdout:4/4: write f1 [944336,113008] 0 2024-04-20T21:31:47.273 INFO:tasks.workunit.client.0.smithi062.stdout:7/0: dread - no filename 2024-04-20T21:31:47.274 INFO:tasks.workunit.client.0.smithi062.stdout:5/0: symlink l0 0 2024-04-20T21:31:47.275 INFO:tasks.workunit.client.0.smithi062.stdout:3/5: symlink d0/l3 0 2024-04-20T21:31:47.275 INFO:tasks.workunit.client.0.smithi062.stdout:3/6: write - no filename 2024-04-20T21:31:47.276 INFO:tasks.workunit.client.0.smithi062.stdout:3/7: fdatasync - no filename 2024-04-20T21:31:47.278 INFO:tasks.workunit.client.0.smithi062.stdout:3/8: rename d0 to d0/d1/d4 22 2024-04-20T21:31:47.281 INFO:tasks.workunit.client.0.smithi062.stdout:2/6: symlink l3 0 2024-04-20T21:31:47.282 INFO:tasks.workunit.client.0.smithi062.stdout:6/0: mkdir d0 0 2024-04-20T21:31:47.282 INFO:tasks.workunit.client.0.smithi062.stdout:6/1: unlink - no file 2024-04-20T21:31:47.282 INFO:tasks.workunit.client.0.smithi062.stdout:6/2: read - no filename 2024-04-20T21:31:47.282 INFO:tasks.workunit.client.0.smithi062.stdout:6/3: dread - no filename 2024-04-20T21:31:47.284 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:46 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:31:47.284 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:46 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:31:47.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:46 smithi062 ceph-mon[16221]: pgmap v8: 65 pgs: 65 active+clean; 203 MiB data, 811 MiB used, 536 GiB / 536 GiB avail; 590 KiB/s rd, 1.5 MiB/s wr, 164 op/s 2024-04-20T21:31:47.286 INFO:tasks.workunit.client.0.smithi062.stdout:5/1: creat f1 x:0 0 0 2024-04-20T21:31:47.286 INFO:tasks.workunit.client.0.smithi062.stdout:4/5: rename f1 to f2 0 2024-04-20T21:31:47.286 INFO:tasks.workunit.client.0.smithi062.stdout:7/1: symlink l0 0 2024-04-20T21:31:47.286 INFO:tasks.workunit.client.0.smithi062.stdout:7/2: fsync - no filename 2024-04-20T21:31:47.287 INFO:tasks.workunit.client.0.smithi062.stdout:7/3: fsync - no filename 2024-04-20T21:31:47.288 INFO:tasks.workunit.client.0.smithi062.stdout:5/2: truncate f1 683738 0 2024-04-20T21:31:47.289 INFO:tasks.workunit.client.0.smithi062.stdout:9/0: rmdir - no directory 2024-04-20T21:31:47.290 INFO:tasks.workunit.client.0.smithi062.stdout:2/7: symlink l4 0 2024-04-20T21:31:47.294 INFO:tasks.workunit.client.0.smithi062.stdout:3/9: symlink d0/d1/l5 0 2024-04-20T21:31:47.294 INFO:tasks.workunit.client.0.smithi062.stdout:3/10: dwrite - no filename 2024-04-20T21:31:47.295 INFO:tasks.workunit.client.0.smithi062.stdout:3/11: write - no filename 2024-04-20T21:31:47.295 INFO:tasks.workunit.client.0.smithi062.stdout:6/4: mknod d0/c1 0 2024-04-20T21:31:47.295 INFO:tasks.workunit.client.0.smithi062.stdout:6/5: truncate - no filename 2024-04-20T21:31:47.295 INFO:tasks.workunit.client.0.smithi062.stdout:6/6: dread - no filename 2024-04-20T21:31:47.295 INFO:tasks.workunit.client.0.smithi062.stdout:6/7: truncate - no filename 2024-04-20T21:31:47.298 INFO:tasks.workunit.client.0.smithi062.stdout:4/6: creat f3 x:0 0 0 2024-04-20T21:31:47.301 INFO:tasks.workunit.client.0.smithi062.stdout:7/4: mknod c1 0 2024-04-20T21:31:47.301 INFO:tasks.workunit.client.0.smithi062.stdout:8/0: creat f0 x:0 0 0 2024-04-20T21:31:47.302 INFO:tasks.workunit.client.0.smithi062.stdout:3/12: symlink d0/l6 0 2024-04-20T21:31:47.304 INFO:tasks.workunit.client.0.smithi062.stdout:8/1: truncate f0 610543 0 2024-04-20T21:31:47.305 INFO:tasks.workunit.client.0.smithi062.stdout:9/1: creat f0 x:0 0 0 2024-04-20T21:31:47.306 INFO:tasks.workunit.client.0.smithi062.stdout:8/2: fdatasync f0 0 2024-04-20T21:31:47.307 INFO:tasks.workunit.client.0.smithi062.stdout:4/7: creat f4 x:0 0 0 2024-04-20T21:31:47.307 INFO:tasks.workunit.client.0.smithi062.stdout:7/5: rename c1 to c2 0 2024-04-20T21:31:47.310 INFO:tasks.workunit.client.0.smithi062.stdout:3/13: mknod d0/d1/c7 0 2024-04-20T21:31:47.311 INFO:tasks.workunit.client.0.smithi062.stdout:3/14: dread - no filename 2024-04-20T21:31:47.311 INFO:tasks.workunit.client.0.smithi062.stdout:3/15: chown d0 282270077 1 2024-04-20T21:31:47.314 INFO:tasks.workunit.client.0.smithi062.stdout:4/8: write f2 [777127,106] 0 2024-04-20T21:31:47.314 INFO:tasks.workunit.client.0.smithi062.stdout:6/8: link d0/c1 d0/c2 0 2024-04-20T21:31:47.317 INFO:tasks.workunit.client.0.smithi062.stdout:3/16: creat d0/f8 x:0 0 0 2024-04-20T21:31:47.318 INFO:tasks.workunit.client.0.smithi062.stdout:3/17: chown d0/d1/c7 55964496 1 2024-04-20T21:31:47.318 INFO:tasks.workunit.client.0.smithi062.stdout:9/2: link f0 f1 0 2024-04-20T21:31:47.319 INFO:tasks.workunit.client.0.smithi062.stdout:3/18: dread - d0/f8 zero size 2024-04-20T21:31:47.321 INFO:tasks.workunit.client.0.smithi062.stdout:4/9: creat f5 x:0 0 0 2024-04-20T21:31:47.321 INFO:tasks.workunit.client.0.smithi062.stdout:4/10: readlink - no filename 2024-04-20T21:31:47.322 INFO:tasks.workunit.client.0.smithi062.stdout:6/9: creat d0/f3 x:0 0 0 2024-04-20T21:31:47.324 INFO:tasks.workunit.client.0.smithi062.stdout:3/19: rmdir d0/d1 39 2024-04-20T21:31:47.328 INFO:tasks.workunit.client.0.smithi062.stdout:3/20: symlink d0/l9 0 2024-04-20T21:31:47.331 INFO:tasks.workunit.client.0.smithi062.stdout:3/21: mknod d0/ca 0 2024-04-20T21:31:47.333 INFO:tasks.workunit.client.0.smithi062.stdout:3/22: write d0/f8 [522951,40133] 0 2024-04-20T21:31:47.575 INFO:tasks.workunit.client.0.smithi062.stdout:1/7: getdents . 0 2024-04-20T21:31:47.575 INFO:tasks.workunit.client.0.smithi062.stdout:1/8: read - no filename 2024-04-20T21:31:47.575 INFO:tasks.workunit.client.0.smithi062.stdout:1/9: dwrite - no filename 2024-04-20T21:31:47.575 INFO:tasks.workunit.client.0.smithi062.stdout:1/10: dwrite - no filename 2024-04-20T21:31:47.576 INFO:tasks.workunit.client.0.smithi062.stdout:1/11: rmdir - no directory 2024-04-20T21:31:47.576 INFO:tasks.workunit.client.0.smithi062.stdout:1/12: write - no filename 2024-04-20T21:31:47.579 INFO:tasks.workunit.client.0.smithi062.stdout:1/13: creat f1 x:0 0 0 2024-04-20T21:31:47.581 INFO:tasks.workunit.client.0.smithi062.stdout:3/23: creat d0/d1/fb x:0 0 0 2024-04-20T21:31:47.582 INFO:tasks.workunit.client.0.smithi062.stdout:1/14: creat f2 x:0 0 0 2024-04-20T21:31:47.586 INFO:tasks.workunit.client.0.smithi062.stdout:0/11: truncate f2 3976353 0 2024-04-20T21:31:47.588 INFO:tasks.workunit.client.0.smithi062.stdout:2/8: getdents . 0 2024-04-20T21:31:47.590 INFO:tasks.workunit.client.0.smithi062.stdout:4/11: fsync f3 0 2024-04-20T21:31:47.591 INFO:tasks.workunit.client.0.smithi062.stdout:2/9: readlink l3 0 2024-04-20T21:31:47.591 INFO:tasks.workunit.client.0.smithi062.stdout:2/10: write - no filename 2024-04-20T21:31:47.592 INFO:tasks.workunit.client.0.smithi062.stdout:2/11: stat l4 0 2024-04-20T21:31:47.592 INFO:tasks.workunit.client.0.smithi062.stdout:0/12: fdatasync f2 0 2024-04-20T21:31:47.592 INFO:tasks.workunit.client.0.smithi062.stdout:2/12: write - no filename 2024-04-20T21:31:47.592 INFO:tasks.workunit.client.0.smithi062.stdout:2/13: read - no filename 2024-04-20T21:31:47.592 INFO:tasks.workunit.client.0.smithi062.stdout:2/14: rmdir - no directory 2024-04-20T21:31:47.592 INFO:tasks.workunit.client.0.smithi062.stdout:2/15: stat l0 0 2024-04-20T21:31:47.593 INFO:tasks.workunit.client.0.smithi062.stdout:2/16: dwrite - no filename 2024-04-20T21:31:47.593 INFO:tasks.workunit.client.0.smithi062.stdout:8/3: truncate f0 476781 0 2024-04-20T21:31:47.594 INFO:tasks.workunit.client.0.smithi062.stdout:4/12: symlink l6 0 2024-04-20T21:31:47.596 INFO:tasks.workunit.client.0.smithi062.stdout:0/13: symlink l3 0 2024-04-20T21:31:47.597 INFO:tasks.workunit.client.0.smithi062.stdout:4/13: truncate f2 2019822 0 2024-04-20T21:31:47.598 INFO:tasks.workunit.client.0.smithi062.stdout:3/24: rename d0/d1/c7 to d0/cc 0 2024-04-20T21:31:47.599 INFO:tasks.workunit.client.0.smithi062.stdout:9/3: getdents . 0 2024-04-20T21:31:47.599 INFO:tasks.workunit.client.0.smithi062.stdout:9/4: rmdir - no directory 2024-04-20T21:31:47.600 INFO:tasks.workunit.client.0.smithi062.stdout:2/17: mknod c5 0 2024-04-20T21:31:47.603 INFO:tasks.workunit.client.0.smithi062.stdout:9/5: creat f2 x:0 0 0 2024-04-20T21:31:47.605 INFO:tasks.workunit.client.0.smithi062.stdout:3/25: unlink d0/l3 0 2024-04-20T21:31:47.606 INFO:tasks.workunit.client.0.smithi062.stdout:4/14: rename c0 to c7 0 2024-04-20T21:31:47.606 INFO:tasks.workunit.client.0.smithi062.stdout:9/6: chown f1 155011 1 2024-04-20T21:31:47.608 INFO:tasks.workunit.client.0.smithi062.stdout:3/26: readlink d0/d1/l5 0 2024-04-20T21:31:47.610 INFO:tasks.workunit.client.0.smithi062.stdout:9/7: truncate f0 727683 0 2024-04-20T21:31:47.610 INFO:tasks.workunit.client.0.smithi062.stdout:9/8: readlink - no filename 2024-04-20T21:31:47.614 INFO:tasks.workunit.client.0.smithi062.stdout:9/9: mknod c3 0 2024-04-20T21:31:47.620 INFO:tasks.workunit.client.0.smithi062.stdout:9/10: truncate f2 400087 0 2024-04-20T21:31:47.623 INFO:tasks.workunit.client.0.smithi062.stdout:1/15: dwrite f2 [0,4194304] 0 2024-04-20T21:31:47.629 INFO:tasks.workunit.client.0.smithi062.stdout:1/16: dread - f1 zero size 2024-04-20T21:31:47.630 INFO:tasks.workunit.client.0.smithi062.stdout:9/11: rename f0 to f4 0 2024-04-20T21:31:47.640 INFO:tasks.workunit.client.0.smithi062.stdout:1/17: write f2 [360236,43719] 0 2024-04-20T21:31:47.645 INFO:tasks.workunit.client.0.smithi062.stdout:1/18: mknod c3 0 2024-04-20T21:31:47.647 INFO:tasks.workunit.client.0.smithi062.stdout:4/15: dwrite f5 [0,4194304] 0 2024-04-20T21:31:47.652 INFO:tasks.workunit.client.0.smithi062.stdout:3/27: dwrite d0/f8 [0,4194304] 0 2024-04-20T21:31:47.653 INFO:tasks.workunit.client.0.smithi062.stdout:4/16: mknod c8 0 2024-04-20T21:31:47.655 INFO:tasks.workunit.client.0.smithi062.stdout:4/17: write f5 [405293,44285] 0 2024-04-20T21:31:47.656 INFO:tasks.workunit.client.0.smithi062.stdout:3/28: write d0/d1/fb [853356,126625] 0 2024-04-20T21:31:47.657 INFO:tasks.workunit.client.0.smithi062.stdout:4/18: write f2 [1560237,39234] 0 2024-04-20T21:31:47.658 INFO:tasks.workunit.client.0.smithi062.stdout:3/29: chown d0/d1/l5 603887837 1 2024-04-20T21:31:47.659 INFO:tasks.workunit.client.0.smithi062.stdout:4/19: write f5 [2099602,124113] 0 2024-04-20T21:31:47.662 INFO:tasks.workunit.client.0.smithi062.stdout:4/20: rename l6 to l9 0 2024-04-20T21:31:47.665 INFO:tasks.workunit.client.0.smithi062.stdout:3/30: creat d0/d1/fd x:0 0 0 2024-04-20T21:31:47.668 INFO:tasks.workunit.client.0.smithi062.stdout:3/31: symlink d0/d1/le 0 2024-04-20T21:31:47.683 INFO:tasks.workunit.client.0.smithi062.stdout:3/32: creat d0/ff x:0 0 0 2024-04-20T21:31:47.683 INFO:tasks.workunit.client.0.smithi062.stdout:1/19: fsync f1 0 2024-04-20T21:31:47.684 INFO:tasks.workunit.client.0.smithi062.stdout:3/33: read - d0/ff zero size 2024-04-20T21:31:47.686 INFO:tasks.workunit.client.0.smithi062.stdout:1/20: truncate f1 732477 0 2024-04-20T21:31:47.687 INFO:tasks.workunit.client.0.smithi062.stdout:8/4: fsync f0 0 2024-04-20T21:31:47.689 INFO:tasks.workunit.client.0.smithi062.stdout:3/34: mkdir d0/d1/d10 0 2024-04-20T21:31:47.690 INFO:tasks.workunit.client.0.smithi062.stdout:2/18: getdents . 0 2024-04-20T21:31:47.690 INFO:tasks.workunit.client.0.smithi062.stdout:2/19: write - no filename 2024-04-20T21:31:47.691 INFO:tasks.workunit.client.0.smithi062.stdout:1/21: dread f1 [0,4194304] 0 2024-04-20T21:31:47.692 INFO:tasks.workunit.client.0.smithi062.stdout:3/35: readlink d0/d1/l5 0 2024-04-20T21:31:47.693 INFO:tasks.workunit.client.0.smithi062.stdout:8/5: creat f1 x:0 0 0 2024-04-20T21:31:47.695 INFO:tasks.workunit.client.0.smithi062.stdout:8/6: chown f0 502 1 2024-04-20T21:31:47.696 INFO:tasks.workunit.client.0.smithi062.stdout:3/36: truncate d0/d1/fb 1506032 0 2024-04-20T21:31:47.696 INFO:tasks.workunit.client.0.smithi062.stdout:9/12: fsync f4 0 2024-04-20T21:31:47.700 INFO:tasks.workunit.client.0.smithi062.stdout:8/7: link f1 f2 0 2024-04-20T21:31:47.701 INFO:tasks.workunit.client.0.smithi062.stdout:9/13: mknod c5 0 2024-04-20T21:31:47.702 INFO:tasks.workunit.client.0.smithi062.stdout:4/21: truncate f5 4090333 0 2024-04-20T21:31:47.704 INFO:tasks.workunit.client.0.smithi062.stdout:0/14: dread f2 [0,4194304] 0 2024-04-20T21:31:47.711 INFO:tasks.workunit.client.0.smithi062.stdout:9/14: dread f2 [0,4194304] 0 2024-04-20T21:31:47.713 INFO:tasks.workunit.client.0.smithi062.stdout:0/15: read f2 [1431957,112911] 0 2024-04-20T21:31:47.717 INFO:tasks.workunit.client.0.smithi062.stdout:9/15: fdatasync f4 0 2024-04-20T21:31:47.719 INFO:tasks.workunit.client.0.smithi062.stdout:8/8: write f2 [163807,27397] 0 2024-04-20T21:31:47.721 INFO:tasks.workunit.client.0.smithi062.stdout:0/16: mknod c4 0 2024-04-20T21:31:47.722 INFO:tasks.workunit.client.0.smithi062.stdout:0/17: rmdir - no directory 2024-04-20T21:31:47.724 INFO:tasks.workunit.client.0.smithi062.stdout:9/16: chown f2 5 1 2024-04-20T21:31:47.725 INFO:tasks.workunit.client.0.smithi062.stdout:4/22: read f5 [1193731,101852] 0 2024-04-20T21:31:47.726 INFO:tasks.workunit.client.0.smithi062.stdout:8/9: unlink f0 0 2024-04-20T21:31:47.729 INFO:tasks.workunit.client.0.smithi062.stdout:0/18: mknod c5 0 2024-04-20T21:31:47.731 INFO:tasks.workunit.client.0.smithi062.stdout:9/17: rename f1 to f6 0 2024-04-20T21:31:47.732 INFO:tasks.workunit.client.0.smithi062.stdout:4/23: chown c8 175592149 1 2024-04-20T21:31:47.740 INFO:tasks.workunit.client.0.smithi062.stdout:9/18: creat f7 x:0 0 0 2024-04-20T21:31:47.740 INFO:tasks.workunit.client.0.smithi062.stdout:3/37: dwrite d0/ff [0,4194304] 0 2024-04-20T21:31:47.742 INFO:tasks.workunit.client.0.smithi062.stdout:4/24: write f3 [917784,25326] 0 2024-04-20T21:31:47.744 INFO:tasks.workunit.client.0.smithi062.stdout:1/22: dwrite f2 [4194304,4194304] 0 2024-04-20T21:31:47.753 INFO:tasks.workunit.client.0.smithi062.stdout:4/25: chown f3 28643410 1 2024-04-20T21:31:47.753 INFO:tasks.workunit.client.0.smithi062.stdout:0/19: dread f1 [0,4194304] 0 2024-04-20T21:31:47.754 INFO:tasks.workunit.client.0.smithi062.stdout:4/26: stat f2 0 2024-04-20T21:31:47.760 INFO:tasks.workunit.client.0.smithi062.stdout:9/19: dread f6 [0,4194304] 0 2024-04-20T21:31:47.760 INFO:tasks.workunit.client.0.smithi062.stdout:1/23: truncate f1 291006 0 2024-04-20T21:31:47.762 INFO:tasks.workunit.client.0.smithi062.stdout:4/27: mkdir da 0 2024-04-20T21:31:47.765 INFO:tasks.workunit.client.0.smithi062.stdout:4/28: chown da 13285 1 2024-04-20T21:31:47.768 INFO:tasks.workunit.client.0.smithi062.stdout:8/10: dwrite f1 [0,4194304] 0 2024-04-20T21:31:47.768 INFO:tasks.workunit.client.0.smithi062.stdout:3/38: truncate d0/ff 2104440 0 2024-04-20T21:31:47.774 INFO:tasks.workunit.client.0.smithi062.stdout:0/20: dread f1 [0,4194304] 0 2024-04-20T21:31:47.774 INFO:tasks.workunit.client.0.smithi062.stdout:8/11: mknod c3 0 2024-04-20T21:31:47.777 INFO:tasks.workunit.client.0.smithi062.stdout:0/21: rename c5 to c6 0 2024-04-20T21:31:47.778 INFO:tasks.workunit.client.0.smithi062.stdout:8/12: mknod c4 0 2024-04-20T21:31:47.779 INFO:tasks.workunit.client.0.smithi062.stdout:4/29: rename f2 to da/fb 0 2024-04-20T21:31:47.783 INFO:tasks.workunit.client.0.smithi062.stdout:8/13: chown c4 3 1 2024-04-20T21:31:47.784 INFO:tasks.workunit.client.0.smithi062.stdout:3/39: read d0/ff [1935780,99510] 0 2024-04-20T21:31:47.791 INFO:tasks.workunit.client.0.smithi062.stdout:8/14: write f1 [4181463,87521] 0 2024-04-20T21:31:47.795 INFO:tasks.workunit.client.0.smithi062.stdout:4/30: rename c7 to da/cc 0 2024-04-20T21:31:47.803 INFO:tasks.workunit.client.0.smithi062.stdout:4/31: mkdir da/dd 0 2024-04-20T21:31:47.805 INFO:tasks.workunit.client.0.smithi062.stdout:8/15: link f1 f5 0 2024-04-20T21:31:47.813 INFO:tasks.workunit.client.0.smithi062.stdout:9/20: dwrite f2 [0,4194304] 0 2024-04-20T21:31:47.816 INFO:tasks.workunit.client.0.smithi062.stdout:1/24: dwrite f2 [4194304,4194304] 0 2024-04-20T21:31:47.816 INFO:tasks.workunit.client.0.smithi062.stdout:1/25: readlink - no filename 2024-04-20T21:31:47.819 INFO:tasks.workunit.client.0.smithi062.stdout:4/32: truncate f4 54134 0 2024-04-20T21:31:47.821 INFO:tasks.workunit.client.0.smithi062.stdout:8/16: rename f5 to f6 0 2024-04-20T21:31:47.821 INFO:tasks.workunit.client.0.smithi062.stdout:8/17: readlink - no filename 2024-04-20T21:31:47.824 INFO:tasks.workunit.client.0.smithi062.stdout:1/26: chown f2 37 1 2024-04-20T21:31:47.825 INFO:tasks.workunit.client.0.smithi062.stdout:9/21: rename f2 to f8 0 2024-04-20T21:31:47.826 INFO:tasks.workunit.client.0.smithi062.stdout:4/33: mknod da/ce 0 2024-04-20T21:31:47.836 INFO:tasks.workunit.client.0.smithi062.stdout:3/40: dwrite d0/d1/fb [0,4194304] 0 2024-04-20T21:31:47.836 INFO:tasks.workunit.client.0.smithi062.stdout:8/18: write f6 [2287910,13446] 0 2024-04-20T21:31:47.842 INFO:tasks.workunit.client.0.smithi062.stdout:0/22: dwrite f2 [0,4194304] 0 2024-04-20T21:31:47.847 INFO:tasks.workunit.client.0.smithi062.stdout:4/34: write f5 [1049093,104082] 0 2024-04-20T21:31:47.852 INFO:tasks.workunit.client.0.smithi062.stdout:9/22: dread f4 [0,4194304] 0 2024-04-20T21:31:47.853 INFO:tasks.workunit.client.0.smithi062.stdout:3/41: mkdir d0/d11 0 2024-04-20T21:31:47.853 INFO:tasks.workunit.client.0.smithi062.stdout:0/23: read f1 [545720,55569] 0 2024-04-20T21:31:47.856 INFO:tasks.workunit.client.0.smithi062.stdout:9/23: mknod c9 0 2024-04-20T21:31:47.858 INFO:tasks.workunit.client.0.smithi062.stdout:4/35: mkdir da/dd/df 0 2024-04-20T21:31:47.860 INFO:tasks.workunit.client.0.smithi062.stdout:3/42: mknod d0/d1/c12 0 2024-04-20T21:31:47.860 INFO:tasks.workunit.client.0.smithi062.stdout:0/24: mkdir d7 0 2024-04-20T21:31:47.867 INFO:tasks.workunit.client.0.smithi062.stdout:1/27: dwrite f2 [4194304,4194304] 0 2024-04-20T21:31:47.868 INFO:tasks.workunit.client.0.smithi062.stdout:9/24: link c3 ca 0 2024-04-20T21:31:47.868 INFO:tasks.workunit.client.0.smithi062.stdout:8/19: dread f2 [0,4194304] 0 2024-04-20T21:31:47.870 INFO:tasks.workunit.client.0.smithi062.stdout:4/36: creat da/dd/df/f10 x:0 0 0 2024-04-20T21:31:47.870 INFO:tasks.workunit.client.0.smithi062.stdout:3/43: symlink d0/l13 0 2024-04-20T21:31:47.871 INFO:tasks.workunit.client.0.smithi062.stdout:0/25: creat d7/f8 x:0 0 0 2024-04-20T21:31:47.872 INFO:tasks.workunit.client.0.smithi062.stdout:9/25: unlink c9 0 2024-04-20T21:31:47.876 INFO:tasks.workunit.client.0.smithi062.stdout:0/26: creat d7/f9 x:0 0 0 2024-04-20T21:31:47.877 INFO:tasks.workunit.client.0.smithi062.stdout:3/44: read d0/d1/fb [1108482,12142] 0 2024-04-20T21:31:47.878 INFO:tasks.workunit.client.0.smithi062.stdout:8/20: mkdir d7 0 2024-04-20T21:31:47.880 INFO:tasks.workunit.client.0.smithi062.stdout:4/37: rename f5 to da/dd/f11 0 2024-04-20T21:31:47.882 INFO:tasks.workunit.client.0.smithi062.stdout:9/26: creat fb x:0 0 0 2024-04-20T21:31:47.885 INFO:tasks.workunit.client.0.smithi062.stdout:0/27: mknod d7/ca 0 2024-04-20T21:31:47.888 INFO:tasks.workunit.client.0.smithi062.stdout:4/38: link f3 da/dd/f12 0 2024-04-20T21:31:47.890 INFO:tasks.workunit.client.0.smithi062.stdout:0/28: creat d7/fb x:0 0 0 2024-04-20T21:31:47.892 INFO:tasks.workunit.client.0.smithi062.stdout:1/28: truncate f2 566264 0 2024-04-20T21:31:47.892 INFO:tasks.workunit.client.0.smithi062.stdout:0/29: truncate d7/f9 23120 0 2024-04-20T21:31:47.893 INFO:tasks.workunit.client.0.smithi062.stdout:4/39: write da/fb [3037227,84042] 0 2024-04-20T21:31:47.893 INFO:tasks.workunit.client.0.smithi062.stdout:8/21: dread f1 [0,4194304] 0 2024-04-20T21:31:47.894 INFO:tasks.workunit.client.0.smithi062.stdout:0/30: rename d7 to d7/dc 22 2024-04-20T21:31:47.896 INFO:tasks.workunit.client.0.smithi062.stdout:4/40: dread f4 [0,4194304] 0 2024-04-20T21:31:47.896 INFO:tasks.workunit.client.0.smithi062.stdout:1/29: chown f2 1334779627 1 2024-04-20T21:31:47.897 INFO:tasks.workunit.client.0.smithi062.stdout:9/27: dread f8 [0,4194304] 0 2024-04-20T21:31:47.899 INFO:tasks.workunit.client.0.smithi062.stdout:0/31: mknod d7/cd 0 2024-04-20T21:31:47.902 INFO:tasks.workunit.client.0.smithi062.stdout:9/28: creat fc x:0 0 0 2024-04-20T21:31:47.905 INFO:tasks.workunit.client.0.smithi062.stdout:4/41: creat da/f13 x:0 0 0 2024-04-20T21:31:47.906 INFO:tasks.workunit.client.0.smithi062.stdout:0/32: mkdir d7/de 0 2024-04-20T21:31:47.907 INFO:tasks.workunit.client.0.smithi062.stdout:9/29: dread f4 [0,4194304] 0 2024-04-20T21:31:47.907 INFO:tasks.workunit.client.0.smithi062.stdout:1/30: getdents . 0 2024-04-20T21:31:47.908 INFO:tasks.workunit.client.0.smithi062.stdout:4/42: fdatasync da/dd/f12 0 2024-04-20T21:31:47.910 INFO:tasks.workunit.client.0.smithi062.stdout:9/30: creat fd x:0 0 0 2024-04-20T21:31:47.914 INFO:tasks.workunit.client.0.smithi062.stdout:1/31: fdatasync f1 0 2024-04-20T21:31:47.916 INFO:tasks.workunit.client.0.smithi062.stdout:4/43: write da/dd/f11 [4356942,27928] 0 2024-04-20T21:31:47.917 INFO:tasks.workunit.client.0.smithi062.stdout:9/31: creat fe x:0 0 0 2024-04-20T21:31:47.918 INFO:tasks.workunit.client.0.smithi062.stdout:8/22: truncate f2 4255367 0 2024-04-20T21:31:47.918 INFO:tasks.workunit.client.0.smithi062.stdout:8/23: readlink - no filename 2024-04-20T21:31:47.919 INFO:tasks.workunit.client.0.smithi062.stdout:0/33: creat d7/de/ff x:0 0 0 2024-04-20T21:31:47.921 INFO:tasks.workunit.client.0.smithi062.stdout:1/32: mknod c4 0 2024-04-20T21:31:47.923 INFO:tasks.workunit.client.0.smithi062.stdout:4/44: creat da/dd/f14 x:0 0 0 2024-04-20T21:31:47.924 INFO:tasks.workunit.client.0.smithi062.stdout:9/32: rename c5 to cf 0 2024-04-20T21:31:47.925 INFO:tasks.workunit.client.0.smithi062.stdout:9/33: write f7 [956357,24779] 0 2024-04-20T21:31:47.927 INFO:tasks.workunit.client.0.smithi062.stdout:8/24: chown f6 16396045 1 2024-04-20T21:31:47.928 INFO:tasks.workunit.client.0.smithi062.stdout:9/34: mknod c10 0 2024-04-20T21:31:47.929 INFO:tasks.workunit.client.0.smithi062.stdout:9/35: fdatasync fb 0 2024-04-20T21:31:47.932 INFO:tasks.workunit.client.0.smithi062.stdout:8/25: creat d7/f8 x:0 0 0 2024-04-20T21:31:47.933 INFO:tasks.workunit.client.0.smithi062.stdout:8/26: chown c4 14388 1 2024-04-20T21:31:47.933 INFO:tasks.workunit.client.0.smithi062.stdout:0/34: rmdir d7/de 39 2024-04-20T21:31:47.935 INFO:tasks.workunit.client.0.smithi062.stdout:8/27: mkdir d7/d9 0 2024-04-20T21:31:47.939 INFO:tasks.workunit.client.0.smithi062.stdout:4/45: getdents da/dd 0 2024-04-20T21:31:47.942 INFO:tasks.workunit.client.0.smithi062.stdout:0/35: mknod d7/c10 0 2024-04-20T21:31:47.944 INFO:tasks.workunit.client.0.smithi062.stdout:4/46: stat l9 0 2024-04-20T21:31:47.950 INFO:tasks.workunit.client.0.smithi062.stdout:4/47: mkdir da/dd/d15 0 2024-04-20T21:31:47.950 INFO:tasks.workunit.client.0.smithi062.stdout:1/33: dwrite f1 [0,4194304] 0 2024-04-20T21:31:47.951 INFO:tasks.workunit.client.0.smithi062.stdout:9/36: link cf c11 0 2024-04-20T21:31:47.956 INFO:tasks.workunit.client.0.smithi062.stdout:1/34: creat f5 x:0 0 0 2024-04-20T21:31:47.960 INFO:tasks.workunit.client.0.smithi062.stdout:0/36: link d7/fb d7/de/f11 0 2024-04-20T21:31:47.960 INFO:tasks.workunit.client.0.smithi062.stdout:0/37: chown c0 688 1 2024-04-20T21:31:47.962 INFO:tasks.workunit.client.0.smithi062.stdout:0/38: dread - d7/de/f11 zero size 2024-04-20T21:31:47.964 INFO:tasks.workunit.client.0.smithi062.stdout:1/35: write f2 [355149,6843] 0 2024-04-20T21:31:47.965 INFO:tasks.workunit.client.0.smithi062.stdout:1/36: chown c3 0 1 2024-04-20T21:31:47.969 INFO:tasks.workunit.client.0.smithi062.stdout:1/37: write f5 [646142,118949] 0 2024-04-20T21:31:47.969 INFO:tasks.workunit.client.0.smithi062.stdout:0/39: symlink d7/de/l12 0 2024-04-20T21:31:47.970 INFO:tasks.workunit.client.0.smithi062.stdout:9/37: getdents . 0 2024-04-20T21:31:47.971 INFO:tasks.workunit.client.0.smithi062.stdout:8/28: write f6 [4312191,128635] 0 2024-04-20T21:31:47.972 INFO:tasks.workunit.client.0.smithi062.stdout:9/38: read f4 [231174,38758] 0 2024-04-20T21:31:47.973 INFO:tasks.workunit.client.0.smithi062.stdout:1/38: write f1 [3321715,106842] 0 2024-04-20T21:31:47.975 INFO:tasks.workunit.client.0.smithi062.stdout:0/40: unlink c6 0 2024-04-20T21:31:47.977 INFO:tasks.workunit.client.0.smithi062.stdout:1/39: creat f6 x:0 0 0 2024-04-20T21:31:47.979 INFO:tasks.workunit.client.0.smithi062.stdout:1/40: mknod c7 0 2024-04-20T21:31:47.984 INFO:tasks.workunit.client.0.smithi062.stdout:1/41: chown c4 2 1 2024-04-20T21:31:47.985 INFO:tasks.workunit.client.0.smithi062.stdout:1/42: stat c7 0 2024-04-20T21:31:47.985 INFO:tasks.workunit.client.0.smithi062.stdout:1/43: chown f1 1747772540 1 2024-04-20T21:31:47.986 INFO:tasks.workunit.client.0.smithi062.stdout:1/44: truncate f2 1200646 0 2024-04-20T21:31:47.989 INFO:tasks.workunit.client.0.smithi062.stdout:1/45: creat f8 x:0 0 0 2024-04-20T21:31:47.992 INFO:tasks.workunit.client.0.smithi062.stdout:8/29: dread f1 [0,4194304] 0 2024-04-20T21:31:47.992 INFO:tasks.workunit.client.0.smithi062.stdout:1/46: fdatasync f5 0 2024-04-20T21:31:47.996 INFO:tasks.workunit.client.0.smithi062.stdout:8/30: creat d7/fa x:0 0 0 2024-04-20T21:31:47.997 INFO:tasks.workunit.client.0.smithi062.stdout:1/47: write f5 [627076,77670] 0 2024-04-20T21:31:48.002 INFO:tasks.workunit.client.0.smithi062.stdout:1/48: dread - f8 zero size 2024-04-20T21:31:48.003 INFO:tasks.workunit.client.0.smithi062.stdout:9/39: dwrite fc [0,4194304] 0 2024-04-20T21:31:48.007 INFO:tasks.workunit.client.0.smithi062.stdout:8/31: rename d7/f8 to d7/d9/fb 0 2024-04-20T21:31:48.007 INFO:tasks.workunit.client.0.smithi062.stdout:0/41: dwrite d7/de/f11 [0,4194304] 0 2024-04-20T21:31:48.008 INFO:tasks.workunit.client.0.smithi062.stdout:9/40: mknod c12 0 2024-04-20T21:31:48.013 INFO:tasks.workunit.client.0.smithi062.stdout:8/32: creat d7/fc x:0 0 0 2024-04-20T21:31:48.013 INFO:tasks.workunit.client.0.smithi062.stdout:9/41: read fc [3168280,116529] 0 2024-04-20T21:31:48.016 INFO:tasks.workunit.client.0.smithi062.stdout:8/33: fdatasync f6 0 2024-04-20T21:31:48.020 INFO:tasks.workunit.client.0.smithi062.stdout:8/34: creat d7/fd x:0 0 0 2024-04-20T21:31:48.021 INFO:tasks.workunit.client.0.smithi062.stdout:0/42: mkdir d7/de/d13 0 2024-04-20T21:31:48.023 INFO:tasks.workunit.client.0.smithi062.stdout:0/43: dread - d7/de/ff zero size 2024-04-20T21:31:48.024 INFO:tasks.workunit.client.0.smithi062.stdout:8/35: mknod d7/d9/ce 0 2024-04-20T21:31:48.027 INFO:tasks.workunit.client.0.smithi062.stdout:8/36: write f1 [3418480,3046] 0 2024-04-20T21:31:48.030 INFO:tasks.workunit.client.0.smithi062.stdout:0/44: write d7/de/f11 [3704356,69754] 0 2024-04-20T21:31:48.031 INFO:tasks.workunit.client.0.smithi062.stdout:8/37: dread - d7/fa zero size 2024-04-20T21:31:48.032 INFO:tasks.workunit.client.0.smithi062.stdout:1/49: dwrite f2 [0,4194304] 0 2024-04-20T21:31:48.034 INFO:tasks.workunit.client.0.smithi062.stdout:8/38: stat d7/fc 0 2024-04-20T21:31:48.036 INFO:tasks.workunit.client.0.smithi062.stdout:9/42: dread f7 [0,4194304] 0 2024-04-20T21:31:48.036 INFO:tasks.workunit.client.0.smithi062.stdout:8/39: write f1 [1315100,43947] 0 2024-04-20T21:31:48.038 INFO:tasks.workunit.client.0.smithi062.stdout:1/50: creat f9 x:0 0 0 2024-04-20T21:31:48.039 INFO:tasks.workunit.client.0.smithi062.stdout:9/43: creat f13 x:0 0 0 2024-04-20T21:31:48.041 INFO:tasks.workunit.client.0.smithi062.stdout:8/40: stat f1 0 2024-04-20T21:31:48.041 INFO:tasks.workunit.client.0.smithi062.stdout:9/44: fsync fb 0 2024-04-20T21:31:48.042 INFO:tasks.workunit.client.0.smithi062.stdout:0/45: rmdir d7/de/d13 0 2024-04-20T21:31:48.043 INFO:tasks.workunit.client.0.smithi062.stdout:1/51: creat fa x:0 0 0 2024-04-20T21:31:48.043 INFO:tasks.workunit.client.0.smithi062.stdout:9/45: symlink l14 0 2024-04-20T21:31:48.044 INFO:tasks.workunit.client.0.smithi062.stdout:9/46: dread - fd zero size 2024-04-20T21:31:48.044 INFO:tasks.workunit.client.0.smithi062.stdout:1/52: creat fb x:0 0 0 2024-04-20T21:31:48.045 INFO:tasks.workunit.client.0.smithi062.stdout:1/53: fdatasync f6 0 2024-04-20T21:31:48.045 INFO:tasks.workunit.client.0.smithi062.stdout:0/46: creat d7/de/f14 x:0 0 0 2024-04-20T21:31:48.046 INFO:tasks.workunit.client.0.smithi062.stdout:1/54: stat c3 0 2024-04-20T21:31:48.048 INFO:tasks.workunit.client.0.smithi062.stdout:1/55: mkdir dc 0 2024-04-20T21:31:48.049 INFO:tasks.workunit.client.0.smithi062.stdout:1/56: read - f6 zero size 2024-04-20T21:31:48.050 INFO:tasks.workunit.client.0.smithi062.stdout:1/57: truncate f8 767414 0 2024-04-20T21:31:48.052 INFO:tasks.workunit.client.0.smithi062.stdout:1/58: mknod dc/cd 0 2024-04-20T21:31:48.053 INFO:tasks.workunit.client.0.smithi062.stdout:1/59: dread - fb zero size 2024-04-20T21:31:48.055 INFO:tasks.workunit.client.0.smithi062.stdout:1/60: read - f9 zero size 2024-04-20T21:31:48.055 INFO:tasks.workunit.client.0.smithi062.stdout:1/61: readlink - no filename 2024-04-20T21:31:48.058 INFO:tasks.workunit.client.0.smithi062.stdout:1/62: creat dc/fe x:0 0 0 2024-04-20T21:31:48.062 INFO:tasks.workunit.client.0.smithi062.stdout:1/63: rename fa to dc/ff 0 2024-04-20T21:31:48.067 INFO:tasks.workunit.client.0.smithi062.stdout:0/47: dwrite d7/f9 [0,4194304] 0 2024-04-20T21:31:48.069 INFO:tasks.workunit.client.0.smithi062.stdout:0/48: read - d7/f8 zero size 2024-04-20T21:31:48.070 INFO:tasks.workunit.client.0.smithi062.stdout:1/64: link c4 dc/c10 0 2024-04-20T21:31:48.072 INFO:tasks.workunit.client.0.smithi062.stdout:0/49: write d7/f8 [283124,5821] 0 2024-04-20T21:31:48.076 INFO:tasks.workunit.client.0.smithi062.stdout:0/50: truncate d7/f8 1020153 0 2024-04-20T21:31:48.076 INFO:tasks.workunit.client.0.smithi062.stdout:0/51: stat f1 0 2024-04-20T21:31:48.079 INFO:tasks.workunit.client.0.smithi062.stdout:0/52: write d7/de/f11 [4851143,28558] 0 2024-04-20T21:31:48.094 INFO:tasks.workunit.client.0.smithi062.stdout:1/65: dwrite f1 [0,4194304] 0 2024-04-20T21:31:48.102 INFO:tasks.workunit.client.0.smithi062.stdout:1/66: mknod dc/c11 0 2024-04-20T21:31:48.102 INFO:tasks.workunit.client.0.smithi062.stdout:0/53: dwrite d7/fb [4194304,4194304] 0 2024-04-20T21:31:48.103 INFO:tasks.workunit.client.0.smithi062.stdout:0/54: stat f2 0 2024-04-20T21:31:48.110 INFO:tasks.workunit.client.0.smithi062.stdout:0/55: mkdir d7/de/d15 0 2024-04-20T21:31:48.117 INFO:tasks.workunit.client.0.smithi062.stdout:0/56: link c0 d7/c16 0 2024-04-20T21:31:48.117 INFO:tasks.workunit.client.0.smithi062.stdout:1/67: dread f1 [0,4194304] 0 2024-04-20T21:31:48.118 INFO:tasks.workunit.client.0.smithi062.stdout:0/57: chown d7/cd 11239865 1 2024-04-20T21:31:48.120 INFO:tasks.workunit.client.0.smithi062.stdout:1/68: mknod dc/c12 0 2024-04-20T21:31:48.124 INFO:tasks.workunit.client.0.smithi062.stdout:1/69: mknod dc/c13 0 2024-04-20T21:31:48.125 INFO:tasks.workunit.client.0.smithi062.stdout:1/70: chown dc/c12 37481 1 2024-04-20T21:31:48.133 INFO:tasks.workunit.client.0.smithi062.stdout:0/58: truncate d7/de/f11 1252444 0 2024-04-20T21:31:48.140 INFO:tasks.workunit.client.0.smithi062.stdout:1/71: dwrite f8 [0,4194304] 0 2024-04-20T21:31:48.142 INFO:tasks.workunit.client.0.smithi062.stdout:1/72: truncate f5 1117163 0 2024-04-20T21:31:48.145 INFO:tasks.workunit.client.0.smithi062.stdout:1/73: creat dc/f14 x:0 0 0 2024-04-20T21:31:48.146 INFO:tasks.workunit.client.0.smithi062.stdout:0/59: dread d7/f9 [0,4194304] 0 2024-04-20T21:31:48.149 INFO:tasks.workunit.client.0.smithi062.stdout:0/60: mkdir d7/de/d15/d17 0 2024-04-20T21:31:48.151 INFO:tasks.workunit.client.0.smithi062.stdout:0/61: creat d7/de/f18 x:0 0 0 2024-04-20T21:31:48.155 INFO:tasks.workunit.client.0.smithi062.stdout:0/62: mkdir d7/de/d15/d17/d19 0 2024-04-20T21:31:48.156 INFO:tasks.workunit.client.0.smithi062.stdout:0/63: stat d7/c10 0 2024-04-20T21:31:48.157 INFO:tasks.workunit.client.0.smithi062.stdout:0/64: stat f1 0 2024-04-20T21:31:48.161 INFO:tasks.workunit.client.0.smithi062.stdout:1/74: dwrite f1 [0,4194304] 0 2024-04-20T21:31:48.162 INFO:tasks.workunit.client.0.smithi062.stdout:0/65: creat d7/de/d15/d17/d19/f1a x:0 0 0 2024-04-20T21:31:48.174 INFO:tasks.workunit.client.0.smithi062.stdout:1/75: rmdir dc 39 2024-04-20T21:31:48.177 INFO:tasks.workunit.client.0.smithi062.stdout:1/76: mkdir dc/d15 0 2024-04-20T21:31:48.177 INFO:tasks.workunit.client.0.smithi062.stdout:0/66: truncate d7/f8 172821 0 2024-04-20T21:31:48.178 INFO:tasks.workunit.client.0.smithi062.stdout:1/77: chown f2 6 1 2024-04-20T21:31:48.182 INFO:tasks.workunit.client.0.smithi062.stdout:1/78: creat dc/d15/f16 x:0 0 0 2024-04-20T21:31:48.182 INFO:tasks.workunit.client.0.smithi062.stdout:0/67: truncate d7/de/d15/d17/d19/f1a 176123 0 2024-04-20T21:31:48.182 INFO:tasks.workunit.client.0.smithi062.stdout:1/79: dread - dc/ff zero size 2024-04-20T21:31:48.184 INFO:tasks.workunit.client.0.smithi062.stdout:1/80: write dc/ff [947541,95792] 0 2024-04-20T21:31:48.187 INFO:tasks.workunit.client.0.smithi062.stdout:0/68: rename d7/de/d15/d17/d19 to d7/de/d15/d17/d1b 0 2024-04-20T21:31:48.188 INFO:tasks.workunit.client.0.smithi062.stdout:1/81: mkdir dc/d15/d17 0 2024-04-20T21:31:48.189 INFO:tasks.workunit.client.0.smithi062.stdout:0/69: chown c4 7524854 1 2024-04-20T21:31:48.190 INFO:tasks.workunit.client.0.smithi062.stdout:0/70: dread - d7/de/ff zero size 2024-04-20T21:31:48.191 INFO:tasks.workunit.client.0.smithi062.stdout:0/71: readlink d7/de/l12 0 2024-04-20T21:31:48.192 INFO:tasks.workunit.client.0.smithi062.stdout:1/82: creat dc/d15/f18 x:0 0 0 2024-04-20T21:31:48.192 INFO:tasks.workunit.client.0.smithi062.stdout:1/83: fdatasync dc/ff 0 2024-04-20T21:31:48.193 INFO:tasks.workunit.client.0.smithi062.stdout:0/72: write d7/f9 [2410741,113691] 0 2024-04-20T21:31:48.195 INFO:tasks.workunit.client.0.smithi062.stdout:0/73: read d7/de/f11 [429932,2952] 0 2024-04-20T21:31:48.214 INFO:tasks.workunit.client.0.smithi062.stdout:1/84: dwrite f6 [0,4194304] 0 2024-04-20T21:31:48.218 INFO:tasks.workunit.client.0.smithi062.stdout:0/74: dwrite d7/de/f14 [0,4194304] 0 2024-04-20T21:31:48.221 INFO:tasks.workunit.client.0.smithi062.stdout:1/85: truncate dc/fe 552359 0 2024-04-20T21:31:48.221 INFO:tasks.workunit.client.0.smithi062.stdout:1/86: stat dc/d15 0 2024-04-20T21:31:48.222 INFO:tasks.workunit.client.0.smithi062.stdout:0/75: write d7/de/ff [912226,17191] 0 2024-04-20T21:31:48.231 INFO:tasks.workunit.client.0.smithi062.stdout:1/87: link dc/d15/f18 dc/d15/d17/f19 0 2024-04-20T21:31:48.238 INFO:tasks.workunit.client.0.smithi062.stdout:0/76: dread f2 [0,4194304] 0 2024-04-20T21:31:48.250 INFO:tasks.workunit.client.0.smithi062.stdout:0/77: dread d7/f8 [0,4194304] 0 2024-04-20T21:31:48.255 INFO:tasks.workunit.client.0.smithi062.stdout:1/88: dwrite f8 [0,4194304] 0 2024-04-20T21:31:48.257 INFO:tasks.workunit.client.0.smithi062.stdout:1/89: fsync f6 0 2024-04-20T21:31:48.260 INFO:tasks.workunit.client.0.smithi062.stdout:0/78: rename d7/de to d7/d1c 0 2024-04-20T21:31:48.263 INFO:tasks.workunit.client.0.smithi062.stdout:0/79: readlink l3 0 2024-04-20T21:31:48.263 INFO:tasks.workunit.client.0.smithi062.stdout:1/90: link dc/f14 dc/f1a 0 2024-04-20T21:31:48.267 INFO:tasks.workunit.client.0.smithi062.stdout:0/80: write d7/d1c/f18 [427947,27654] 0 2024-04-20T21:31:48.275 INFO:tasks.workunit.client.0.smithi062.stdout:0/81: rename d7/f8 to d7/d1c/d15/d17/f1d 0 2024-04-20T21:31:48.276 INFO:tasks.workunit.client.0.smithi062.stdout:0/82: read f2 [1464274,109155] 0 2024-04-20T21:31:48.281 INFO:tasks.workunit.client.0.smithi062.stdout:0/83: mkdir d7/d1e 0 2024-04-20T21:31:48.290 INFO:tasks.workunit.client.0.smithi062.stdout:0/84: write d7/d1c/d15/d17/f1d [505077,16313] 0 2024-04-20T21:31:48.293 INFO:tasks.workunit.client.0.smithi062.stdout:0/85: write d7/d1c/f18 [1262447,19385] 0 2024-04-20T21:31:48.295 INFO:tasks.workunit.client.0.smithi062.stdout:0/86: write d7/d1c/f14 [3279062,57076] 0 2024-04-20T21:31:48.299 INFO:tasks.workunit.client.0.smithi062.stdout:0/87: mknod d7/d1c/d15/d17/d1b/c1f 0 2024-04-20T21:31:48.304 INFO:tasks.workunit.client.0.smithi062.stdout:0/88: rmdir d7/d1c/d15 39 2024-04-20T21:31:48.307 INFO:tasks.workunit.client.0.smithi062.stdout:0/89: truncate d7/d1c/d15/d17/f1d 1216675 0 2024-04-20T21:31:48.308 INFO:tasks.workunit.client.0.smithi062.stdout:0/90: fsync d7/d1c/f14 0 2024-04-20T21:31:48.311 INFO:tasks.workunit.client.0.smithi062.stdout:0/91: read d7/d1c/f18 [1100387,83942] 0 2024-04-20T21:31:48.333 INFO:tasks.workunit.client.0.smithi062.stdout:0/92: dwrite d7/d1c/f18 [0,4194304] 0 2024-04-20T21:31:48.534 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:48 smithi062 ceph-mon[16221]: Standby manager daemon smithi189.tawavz started 2024-04-20T21:31:48.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:48 smithi062 ceph-mon[16221]: from='mgr.? 172.21.15.189:0/3757179262' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi189.tawavz/crt"}]: dispatch 2024-04-20T21:31:48.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:48 smithi062 ceph-mon[16221]: from='mgr.? 172.21.15.189:0/3757179262' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-20T21:31:48.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:48 smithi062 ceph-mon[16221]: from='mgr.? 172.21.15.189:0/3757179262' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi189.tawavz/key"}]: dispatch 2024-04-20T21:31:48.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:48 smithi062 ceph-mon[16221]: from='mgr.? 172.21.15.189:0/3757179262' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-20T21:31:48.569 INFO:tasks.workunit.client.0.smithi062.stdout:5/3: sync 2024-04-20T21:31:48.569 INFO:tasks.workunit.client.0.smithi062.stdout:3/45: sync 2024-04-20T21:31:48.569 INFO:tasks.workunit.client.0.smithi062.stdout:4/48: sync 2024-04-20T21:31:48.569 INFO:tasks.workunit.client.0.smithi062.stdout:6/10: sync 2024-04-20T21:31:48.569 INFO:tasks.workunit.client.0.smithi062.stdout:8/41: sync 2024-04-20T21:31:48.569 INFO:tasks.workunit.client.0.smithi062.stdout:2/20: sync 2024-04-20T21:31:48.569 INFO:tasks.workunit.client.0.smithi062.stdout:2/21: write - no filename 2024-04-20T21:31:48.569 INFO:tasks.workunit.client.0.smithi062.stdout:2/22: dread - no filename 2024-04-20T21:31:48.569 INFO:tasks.workunit.client.0.smithi062.stdout:2/23: read - no filename 2024-04-20T21:31:48.569 INFO:tasks.workunit.client.0.smithi062.stdout:2/24: rmdir - no directory 2024-04-20T21:31:48.569 INFO:tasks.workunit.client.0.smithi062.stdout:7/6: sync 2024-04-20T21:31:48.569 INFO:tasks.workunit.client.0.smithi062.stdout:7/7: fsync - no filename 2024-04-20T21:31:48.572 INFO:tasks.workunit.client.0.smithi062.stdout:3/46: stat d0/d1/fb 0 2024-04-20T21:31:48.573 INFO:tasks.workunit.client.0.smithi062.stdout:5/4: write f1 [71015,75663] 0 2024-04-20T21:31:48.573 INFO:tasks.workunit.client.0.smithi062.stdout:5/5: readlink l0 0 2024-04-20T21:31:48.574 INFO:tasks.workunit.client.0.smithi062.stdout:7/8: chown c2 4237 1 2024-04-20T21:31:48.574 INFO:tasks.workunit.client.0.smithi062.stdout:7/9: dread - no filename 2024-04-20T21:31:48.574 INFO:tasks.workunit.client.0.smithi062.stdout:8/42: dread f1 [4194304,4194304] 0 2024-04-20T21:31:48.574 INFO:tasks.workunit.client.0.smithi062.stdout:8/43: readlink - no filename 2024-04-20T21:31:48.576 INFO:tasks.workunit.client.0.smithi062.stdout:2/25: unlink c1 0 2024-04-20T21:31:48.576 INFO:tasks.workunit.client.0.smithi062.stdout:2/26: truncate - no filename 2024-04-20T21:31:48.576 INFO:tasks.workunit.client.0.smithi062.stdout:2/27: write - no filename 2024-04-20T21:31:48.576 INFO:tasks.workunit.client.0.smithi062.stdout:2/28: dread - no filename 2024-04-20T21:31:48.576 INFO:tasks.workunit.client.0.smithi062.stdout:2/29: fsync - no filename 2024-04-20T21:31:48.576 INFO:tasks.workunit.client.0.smithi062.stdout:2/30: write - no filename 2024-04-20T21:31:48.576 INFO:tasks.workunit.client.0.smithi062.stdout:2/31: write - no filename 2024-04-20T21:31:48.577 INFO:tasks.workunit.client.0.smithi062.stdout:4/49: creat da/f16 x:0 0 0 2024-04-20T21:31:48.578 INFO:tasks.workunit.client.0.smithi062.stdout:6/11: unlink d0/c1 0 2024-04-20T21:31:48.579 INFO:tasks.workunit.client.0.smithi062.stdout:5/6: mknod c2 0 2024-04-20T21:31:48.580 INFO:tasks.workunit.client.0.smithi062.stdout:4/50: write da/f13 [298638,58246] 0 2024-04-20T21:31:48.581 INFO:tasks.workunit.client.0.smithi062.stdout:3/47: fdatasync d0/ff 0 2024-04-20T21:31:48.581 INFO:tasks.workunit.client.0.smithi062.stdout:7/10: creat f3 x:0 0 0 2024-04-20T21:31:48.582 INFO:tasks.workunit.client.0.smithi062.stdout:6/12: fsync d0/f3 0 2024-04-20T21:31:48.582 INFO:tasks.workunit.client.0.smithi062.stdout:2/32: creat f6 x:0 0 0 2024-04-20T21:31:48.584 INFO:tasks.workunit.client.0.smithi062.stdout:8/44: rename d7/fd to d7/d9/ff 0 2024-04-20T21:31:48.589 INFO:tasks.workunit.client.0.smithi062.stdout:6/13: symlink d0/l4 0 2024-04-20T21:31:48.590 INFO:tasks.workunit.client.0.smithi062.stdout:7/11: creat f4 x:0 0 0 2024-04-20T21:31:48.590 INFO:tasks.workunit.client.0.smithi062.stdout:5/7: dread f1 [0,4194304] 0 2024-04-20T21:31:48.590 INFO:tasks.workunit.client.0.smithi062.stdout:6/14: readlink d0/l4 0 2024-04-20T21:31:48.591 INFO:tasks.workunit.client.0.smithi062.stdout:7/12: write f3 [409074,112497] 0 2024-04-20T21:31:48.592 INFO:tasks.workunit.client.0.smithi062.stdout:5/8: chown l0 13 1 2024-04-20T21:31:48.592 INFO:tasks.workunit.client.0.smithi062.stdout:3/48: mknod d0/d1/d10/c14 0 2024-04-20T21:31:48.593 INFO:tasks.workunit.client.0.smithi062.stdout:6/15: write d0/f3 [252057,2763] 0 2024-04-20T21:31:48.598 INFO:tasks.workunit.client.0.smithi062.stdout:7/13: write f4 [957250,84890] 0 2024-04-20T21:31:48.599 INFO:tasks.workunit.client.0.smithi062.stdout:6/16: chown d0/f3 977306115 1 2024-04-20T21:31:48.599 INFO:tasks.workunit.client.0.smithi062.stdout:5/9: truncate f1 827400 0 2024-04-20T21:31:48.604 INFO:tasks.workunit.client.0.smithi062.stdout:7/14: creat f5 x:0 0 0 2024-04-20T21:31:48.605 INFO:tasks.workunit.client.0.smithi062.stdout:5/10: symlink l3 0 2024-04-20T21:31:48.612 INFO:tasks.workunit.client.0.smithi062.stdout:7/15: creat f6 x:0 0 0 2024-04-20T21:31:48.612 INFO:tasks.workunit.client.0.smithi062.stdout:8/45: dread f1 [0,4194304] 0 2024-04-20T21:31:48.618 INFO:tasks.workunit.client.0.smithi062.stdout:7/16: mkdir d7 0 2024-04-20T21:31:48.623 INFO:tasks.workunit.client.0.smithi062.stdout:4/51: dwrite da/dd/f11 [0,4194304] 0 2024-04-20T21:31:48.632 INFO:tasks.workunit.client.0.smithi062.stdout:2/33: dwrite f6 [0,4194304] 0 2024-04-20T21:31:48.637 INFO:tasks.workunit.client.0.smithi062.stdout:4/52: chown l9 12 1 2024-04-20T21:31:48.641 INFO:tasks.workunit.client.0.smithi062.stdout:2/34: read f6 [2765563,22323] 0 2024-04-20T21:31:48.652 INFO:tasks.workunit.client.0.smithi062.stdout:3/49: dwrite d0/ff [0,4194304] 0 2024-04-20T21:31:48.653 INFO:tasks.workunit.client.0.smithi062.stdout:6/17: dwrite d0/f3 [0,4194304] 0 2024-04-20T21:31:48.654 INFO:tasks.workunit.client.0.smithi062.stdout:7/17: mknod d7/c8 0 2024-04-20T21:31:48.656 INFO:tasks.workunit.client.0.smithi062.stdout:2/35: write f6 [1452998,19885] 0 2024-04-20T21:31:48.662 INFO:tasks.workunit.client.0.smithi062.stdout:5/11: dwrite f1 [0,4194304] 0 2024-04-20T21:31:48.668 INFO:tasks.workunit.client.0.smithi062.stdout:4/53: creat da/dd/df/f17 x:0 0 0 2024-04-20T21:31:48.673 INFO:tasks.workunit.client.0.smithi062.stdout:7/18: mkdir d7/d9 0 2024-04-20T21:31:48.673 INFO:tasks.workunit.client.0.smithi062.stdout:2/36: mkdir d7 0 2024-04-20T21:31:48.673 INFO:tasks.workunit.client.0.smithi062.stdout:5/12: rename f1 to f4 0 2024-04-20T21:31:48.674 INFO:tasks.workunit.client.0.smithi062.stdout:2/37: readlink l3 0 2024-04-20T21:31:48.674 INFO:tasks.workunit.client.0.smithi062.stdout:8/46: dwrite d7/fa [0,4194304] 0 2024-04-20T21:31:48.676 INFO:tasks.workunit.client.0.smithi062.stdout:3/50: symlink d0/d11/l15 0 2024-04-20T21:31:48.677 INFO:tasks.workunit.client.0.smithi062.stdout:4/54: symlink da/dd/l18 0 2024-04-20T21:31:48.677 INFO:tasks.workunit.client.0.smithi062.stdout:2/38: write f6 [172280,121820] 0 2024-04-20T21:31:48.680 INFO:tasks.workunit.client.0.smithi062.stdout:3/51: dread - d0/d1/fd zero size 2024-04-20T21:31:48.684 INFO:tasks.workunit.client.0.smithi062.stdout:4/55: write da/dd/f14 [373679,50969] 0 2024-04-20T21:31:48.684 INFO:tasks.workunit.client.0.smithi062.stdout:7/19: dread f3 [0,4194304] 0 2024-04-20T21:31:48.684 INFO:tasks.workunit.client.0.smithi062.stdout:7/20: stat f5 0 2024-04-20T21:31:48.686 INFO:tasks.workunit.client.0.smithi062.stdout:2/39: rename l0 to d7/l8 0 2024-04-20T21:31:48.689 INFO:tasks.workunit.client.0.smithi062.stdout:3/52: mknod d0/d1/c16 0 2024-04-20T21:31:48.691 INFO:tasks.workunit.client.0.smithi062.stdout:4/56: fdatasync f3 0 2024-04-20T21:31:48.694 INFO:tasks.workunit.client.0.smithi062.stdout:7/21: creat d7/fa x:0 0 0 2024-04-20T21:31:48.695 INFO:tasks.workunit.client.0.smithi062.stdout:3/53: write d0/f8 [1319165,37577] 0 2024-04-20T21:31:48.697 INFO:tasks.workunit.client.0.smithi062.stdout:3/54: stat d0/d11 0 2024-04-20T21:31:48.700 INFO:tasks.workunit.client.0.smithi062.stdout:4/57: creat da/dd/f19 x:0 0 0 2024-04-20T21:31:48.703 INFO:tasks.workunit.client.0.smithi062.stdout:4/58: write da/f16 [878786,93170] 0 2024-04-20T21:31:48.708 INFO:tasks.workunit.client.0.smithi062.stdout:4/59: chown da/f16 142102390 1 2024-04-20T21:31:48.711 INFO:tasks.workunit.client.0.smithi062.stdout:2/40: dwrite f6 [0,4194304] 0 2024-04-20T21:31:48.713 INFO:tasks.workunit.client.0.smithi062.stdout:2/41: mknod d7/c9 0 2024-04-20T21:31:48.714 INFO:tasks.workunit.client.0.smithi062.stdout:2/42: chown d7 1061804 1 2024-04-20T21:31:48.716 INFO:tasks.workunit.client.0.smithi062.stdout:2/43: chown d7/c9 3547272 1 2024-04-20T21:31:48.731 INFO:tasks.workunit.client.0.smithi062.stdout:6/18: rmdir d0 39 2024-04-20T21:31:48.736 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:48 smithi189 ceph-mon[21562]: Standby manager daemon smithi189.tawavz started 2024-04-20T21:31:48.736 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:48 smithi189 ceph-mon[21562]: from='mgr.? 172.21.15.189:0/3757179262' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi189.tawavz/crt"}]: dispatch 2024-04-20T21:31:48.736 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:48 smithi189 ceph-mon[21562]: from='mgr.? 172.21.15.189:0/3757179262' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-20T21:31:48.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:48 smithi189 ceph-mon[21562]: from='mgr.? 172.21.15.189:0/3757179262' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi189.tawavz/key"}]: dispatch 2024-04-20T21:31:48.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:48 smithi189 ceph-mon[21562]: from='mgr.? 172.21.15.189:0/3757179262' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-20T21:31:48.738 INFO:tasks.workunit.client.0.smithi062.stdout:4/60: getdents da/dd/df 0 2024-04-20T21:31:48.742 INFO:tasks.workunit.client.0.smithi062.stdout:3/55: rmdir d0/d1 39 2024-04-20T21:31:48.744 INFO:tasks.workunit.client.0.smithi062.stdout:2/44: dwrite f6 [0,4194304] 0 2024-04-20T21:31:48.753 INFO:tasks.workunit.client.0.smithi062.stdout:8/47: dwrite d7/fa [4194304,4194304] 0 2024-04-20T21:31:48.753 INFO:tasks.workunit.client.0.smithi062.stdout:4/61: write f4 [1008266,100543] 0 2024-04-20T21:31:48.754 INFO:tasks.workunit.client.0.smithi062.stdout:3/56: symlink d0/d1/d10/l17 0 2024-04-20T21:31:48.757 INFO:tasks.workunit.client.0.smithi062.stdout:4/62: stat da/dd/d15 0 2024-04-20T21:31:48.763 INFO:tasks.workunit.client.0.smithi062.stdout:6/19: mknod d0/c5 0 2024-04-20T21:31:48.764 INFO:tasks.workunit.client.0.smithi062.stdout:8/48: write f1 [1666972,14176] 0 2024-04-20T21:31:48.769 INFO:tasks.workunit.client.0.smithi062.stdout:8/49: write f1 [590580,101248] 0 2024-04-20T21:31:48.773 INFO:tasks.workunit.client.0.smithi062.stdout:4/63: getdents da/dd 0 2024-04-20T21:31:48.774 INFO:tasks.workunit.client.0.smithi062.stdout:8/50: creat d7/f10 x:0 0 0 2024-04-20T21:31:48.777 INFO:tasks.workunit.client.0.smithi062.stdout:8/51: chown d7/d9/fb 62077 1 2024-04-20T21:31:48.778 INFO:tasks.workunit.client.0.smithi062.stdout:3/57: write d0/d1/fb [3641660,81573] 0 2024-04-20T21:31:48.780 INFO:tasks.workunit.client.0.smithi062.stdout:4/64: creat da/dd/df/f1a x:0 0 0 2024-04-20T21:31:48.783 INFO:tasks.workunit.client.0.smithi062.stdout:2/45: dwrite f6 [0,4194304] 0 2024-04-20T21:31:48.785 INFO:tasks.workunit.client.0.smithi062.stdout:3/58: chown d0/d1/fd 276 1 2024-04-20T21:31:48.786 INFO:tasks.workunit.client.0.smithi062.stdout:4/65: creat da/f1b x:0 0 0 2024-04-20T21:31:48.788 INFO:tasks.workunit.client.0.smithi062.stdout:8/52: link c3 d7/d9/c11 0 2024-04-20T21:31:48.789 INFO:tasks.workunit.client.0.smithi062.stdout:8/53: fsync f1 0 2024-04-20T21:31:48.790 INFO:tasks.workunit.client.0.smithi062.stdout:3/59: creat d0/d1/f18 x:0 0 0 2024-04-20T21:31:48.791 INFO:tasks.workunit.client.0.smithi062.stdout:4/66: chown da/cc 0 1 2024-04-20T21:31:48.791 INFO:tasks.workunit.client.0.smithi062.stdout:2/46: mkdir d7/da 0 2024-04-20T21:31:48.793 INFO:tasks.workunit.client.0.smithi062.stdout:8/54: fdatasync d7/d9/fb 0 2024-04-20T21:31:48.795 INFO:tasks.workunit.client.0.smithi062.stdout:2/47: chown d7/l8 14026 1 2024-04-20T21:31:48.799 INFO:tasks.workunit.client.0.smithi062.stdout:4/67: creat da/dd/f1c x:0 0 0 2024-04-20T21:31:48.799 INFO:tasks.workunit.client.0.smithi062.stdout:6/20: truncate d0/f3 2236826 0 2024-04-20T21:31:48.800 INFO:tasks.workunit.client.0.smithi062.stdout:3/60: rename d0/d1/f18 to d0/d1/f19 0 2024-04-20T21:31:48.801 INFO:tasks.workunit.client.0.smithi062.stdout:8/55: rename d7/fa to d7/d9/f12 0 2024-04-20T21:31:48.803 INFO:tasks.workunit.client.0.smithi062.stdout:2/48: creat d7/fb x:0 0 0 2024-04-20T21:31:48.804 INFO:tasks.workunit.client.0.smithi062.stdout:8/56: write d7/f10 [245890,71501] 0 2024-04-20T21:31:48.809 INFO:tasks.workunit.client.0.smithi062.stdout:8/57: fsync d7/d9/ff 0 2024-04-20T21:31:48.818 INFO:tasks.workunit.client.0.smithi062.stdout:3/61: dread d0/f8 [0,4194304] 0 2024-04-20T21:31:48.823 INFO:tasks.workunit.client.0.smithi062.stdout:9/47: sync 2024-04-20T21:31:48.823 INFO:tasks.workunit.client.0.smithi062.stdout:3/62: creat d0/d11/f1a x:0 0 0 2024-04-20T21:31:48.826 INFO:tasks.workunit.client.0.smithi062.stdout:9/48: dread - fe zero size 2024-04-20T21:31:48.829 INFO:tasks.workunit.client.0.smithi062.stdout:6/21: truncate d0/f3 1585587 0 2024-04-20T21:31:48.831 INFO:tasks.workunit.client.0.smithi062.stdout:3/63: unlink d0/l9 0 2024-04-20T21:31:48.840 INFO:tasks.workunit.client.0.smithi062.stdout:9/49: truncate f13 860952 0 2024-04-20T21:31:48.841 INFO:tasks.workunit.client.0.smithi062.stdout:2/49: dwrite d7/fb [0,4194304] 0 2024-04-20T21:31:48.847 INFO:tasks.workunit.client.0.smithi062.stdout:3/64: write d0/d11/f1a [131637,29557] 0 2024-04-20T21:31:48.852 INFO:tasks.workunit.client.0.smithi062.stdout:9/50: creat f15 x:0 0 0 2024-04-20T21:31:48.852 INFO:tasks.workunit.client.0.smithi062.stdout:6/22: read d0/f3 [163291,85309] 0 2024-04-20T21:31:48.854 INFO:tasks.workunit.client.0.smithi062.stdout:2/50: getdents d7/da 0 2024-04-20T21:31:48.860 INFO:tasks.workunit.client.0.smithi062.stdout:4/68: dwrite f3 [0,4194304] 0 2024-04-20T21:31:48.860 INFO:tasks.workunit.client.0.smithi062.stdout:3/65: mkdir d0/d1b 0 2024-04-20T21:31:48.862 INFO:tasks.workunit.client.0.smithi062.stdout:8/58: dwrite d7/d9/fb [0,4194304] 0 2024-04-20T21:31:48.862 INFO:tasks.workunit.client.0.smithi062.stdout:6/23: creat d0/f6 x:0 0 0 2024-04-20T21:31:48.863 INFO:tasks.workunit.client.0.smithi062.stdout:8/59: fsync d7/fc 0 2024-04-20T21:31:48.864 INFO:tasks.workunit.client.0.smithi062.stdout:2/51: mknod d7/da/cc 0 2024-04-20T21:31:48.873 INFO:tasks.workunit.client.0.smithi062.stdout:9/51: truncate f8 3065288 0 2024-04-20T21:31:48.880 INFO:tasks.workunit.client.0.smithi062.stdout:6/24: rename d0/l4 to d0/l7 0 2024-04-20T21:31:48.881 INFO:tasks.workunit.client.0.smithi062.stdout:9/52: stat f8 0 2024-04-20T21:31:48.884 INFO:tasks.workunit.client.0.smithi062.stdout:2/52: rmdir d7 39 2024-04-20T21:31:48.884 INFO:tasks.workunit.client.0.smithi062.stdout:9/53: read f7 [687086,75413] 0 2024-04-20T21:31:48.884 INFO:tasks.workunit.client.0.smithi062.stdout:8/60: getdents d7/d9 0 2024-04-20T21:31:48.890 INFO:tasks.workunit.client.0.smithi062.stdout:2/53: chown d7/fb 21826356 1 2024-04-20T21:31:48.892 INFO:tasks.workunit.client.0.smithi062.stdout:2/54: chown d7 4 1 2024-04-20T21:31:48.893 INFO:tasks.workunit.client.0.smithi062.stdout:9/54: mkdir d16 0 2024-04-20T21:31:48.895 INFO:tasks.workunit.client.0.smithi062.stdout:6/25: rmdir d0 39 2024-04-20T21:31:48.895 INFO:tasks.workunit.client.0.smithi062.stdout:8/61: creat d7/d9/f13 x:0 0 0 2024-04-20T21:31:48.897 INFO:tasks.workunit.client.0.smithi062.stdout:3/66: dwrite d0/d11/f1a [0,4194304] 0 2024-04-20T21:31:48.897 INFO:tasks.workunit.client.0.smithi062.stdout:9/55: fsync f6 0 2024-04-20T21:31:48.899 INFO:tasks.workunit.client.0.smithi062.stdout:2/55: write f6 [4902803,251] 0 2024-04-20T21:31:48.900 INFO:tasks.workunit.client.0.smithi062.stdout:3/67: chown d0/d11 484870526 1 2024-04-20T21:31:48.901 INFO:tasks.workunit.client.0.smithi062.stdout:9/56: fsync fc 0 2024-04-20T21:31:48.902 INFO:tasks.workunit.client.0.smithi062.stdout:6/26: read d0/f3 [738591,84948] 0 2024-04-20T21:31:48.904 INFO:tasks.workunit.client.0.smithi062.stdout:8/62: link d7/d9/ff d7/f14 0 2024-04-20T21:31:48.907 INFO:tasks.workunit.client.0.smithi062.stdout:8/63: write f6 [3955097,102624] 0 2024-04-20T21:31:48.909 INFO:tasks.workunit.client.0.smithi062.stdout:9/57: rename c12 to d16/c17 0 2024-04-20T21:31:48.910 INFO:tasks.workunit.client.0.smithi062.stdout:9/58: stat f13 0 2024-04-20T21:31:48.912 INFO:tasks.workunit.client.0.smithi062.stdout:3/68: mkdir d0/d1c 0 2024-04-20T21:31:48.914 INFO:tasks.workunit.client.0.smithi062.stdout:9/59: write fe [339418,72436] 0 2024-04-20T21:31:48.915 INFO:tasks.workunit.client.0.smithi062.stdout:6/27: getdents d0 0 2024-04-20T21:31:48.918 INFO:tasks.workunit.client.0.smithi062.stdout:6/28: dread - d0/f6 zero size 2024-04-20T21:31:48.919 INFO:tasks.workunit.client.0.smithi062.stdout:3/69: read d0/f8 [1719660,47596] 0 2024-04-20T21:31:48.922 INFO:tasks.workunit.client.0.smithi062.stdout:2/56: dread d7/fb [0,4194304] 0 2024-04-20T21:31:48.923 INFO:tasks.workunit.client.0.smithi062.stdout:3/70: mkdir d0/d1b/d1d 0 2024-04-20T21:31:48.924 INFO:tasks.workunit.client.0.smithi062.stdout:6/29: unlink d0/l7 0 2024-04-20T21:31:48.926 INFO:tasks.workunit.client.0.smithi062.stdout:3/71: write d0/d11/f1a [830626,34826] 0 2024-04-20T21:31:48.928 INFO:tasks.workunit.client.0.smithi062.stdout:9/60: getdents d16 0 2024-04-20T21:31:48.930 INFO:tasks.workunit.client.0.smithi062.stdout:8/64: dread f1 [0,4194304] 0 2024-04-20T21:31:48.930 INFO:tasks.workunit.client.0.smithi062.stdout:2/57: rename c5 to d7/cd 0 2024-04-20T21:31:48.932 INFO:tasks.workunit.client.0.smithi062.stdout:3/72: mknod d0/d11/c1e 0 2024-04-20T21:31:48.932 INFO:tasks.workunit.client.0.smithi062.stdout:8/65: write d7/f10 [202956,121554] 0 2024-04-20T21:31:48.933 INFO:tasks.workunit.client.0.smithi062.stdout:3/73: readlink d0/d1/d10/l17 0 2024-04-20T21:31:48.935 INFO:tasks.workunit.client.0.smithi062.stdout:9/61: creat d16/f18 x:0 0 0 2024-04-20T21:31:48.938 INFO:tasks.workunit.client.0.smithi062.stdout:3/74: fsync d0/f8 0 2024-04-20T21:31:48.939 INFO:tasks.workunit.client.0.smithi062.stdout:2/58: write d7/fb [2677654,54184] 0 2024-04-20T21:31:48.941 INFO:tasks.workunit.client.0.smithi062.stdout:9/62: rename d16/f18 to d16/f19 0 2024-04-20T21:31:48.948 INFO:tasks.workunit.client.0.smithi062.stdout:9/63: symlink d16/l1a 0 2024-04-20T21:31:48.948 INFO:tasks.workunit.client.0.smithi062.stdout:3/75: mknod d0/d1/d10/c1f 0 2024-04-20T21:31:48.949 INFO:tasks.workunit.client.0.smithi062.stdout:2/59: link d7/c9 d7/ce 0 2024-04-20T21:31:48.956 INFO:tasks.workunit.client.0.smithi062.stdout:2/60: creat d7/ff x:0 0 0 2024-04-20T21:31:48.960 INFO:tasks.workunit.client.0.smithi062.stdout:2/61: symlink d7/da/l10 0 2024-04-20T21:31:48.963 INFO:tasks.workunit.client.0.smithi062.stdout:8/66: dwrite d7/d9/f12 [0,4194304] 0 2024-04-20T21:31:48.971 INFO:tasks.workunit.client.0.smithi062.stdout:8/67: write d7/d9/f13 [313230,120752] 0 2024-04-20T21:31:48.971 INFO:tasks.workunit.client.0.smithi062.stdout:6/30: dwrite d0/f3 [0,4194304] 0 2024-04-20T21:31:48.972 INFO:tasks.workunit.client.0.smithi062.stdout:8/68: stat d7 0 2024-04-20T21:31:48.975 INFO:tasks.workunit.client.0.smithi062.stdout:3/76: dread d0/d1/fb [0,4194304] 0 2024-04-20T21:31:48.982 INFO:tasks.workunit.client.0.smithi062.stdout:3/77: dread - d0/d1/fd zero size 2024-04-20T21:31:48.983 INFO:tasks.workunit.client.0.smithi062.stdout:9/64: dwrite fc [4194304,4194304] 0 2024-04-20T21:31:48.993 INFO:tasks.workunit.client.0.smithi062.stdout:9/65: dread f4 [0,4194304] 0 2024-04-20T21:31:48.993 INFO:tasks.workunit.client.0.smithi062.stdout:3/78: rmdir d0/d1b/d1d 0 2024-04-20T21:31:48.994 INFO:tasks.workunit.client.0.smithi062.stdout:2/62: dread f6 [0,4194304] 0 2024-04-20T21:31:48.994 INFO:tasks.workunit.client.0.smithi062.stdout:9/66: fdatasync fe 0 2024-04-20T21:31:48.998 INFO:tasks.workunit.client.0.smithi062.stdout:8/69: write d7/d9/ff [76868,91238] 0 2024-04-20T21:31:48.998 INFO:tasks.workunit.client.0.smithi062.stdout:9/67: fsync f15 0 2024-04-20T21:31:48.998 INFO:tasks.workunit.client.0.smithi062.stdout:3/79: chown d0/d1/l5 0 1 2024-04-20T21:31:49.000 INFO:tasks.workunit.client.0.smithi062.stdout:2/63: unlink d7/c9 0 2024-04-20T21:31:49.001 INFO:tasks.workunit.client.0.smithi062.stdout:8/70: write d7/d9/f12 [5756668,68834] 0 2024-04-20T21:31:49.002 INFO:tasks.workunit.client.0.smithi062.stdout:3/80: creat d0/d1/f20 x:0 0 0 2024-04-20T21:31:49.004 INFO:tasks.workunit.client.0.smithi062.stdout:2/64: write d7/ff [844533,44665] 0 2024-04-20T21:31:49.005 INFO:tasks.workunit.client.0.smithi062.stdout:3/81: chown d0/d1 20097001 1 2024-04-20T21:31:49.005 INFO:tasks.workunit.client.0.smithi062.stdout:6/31: fsync d0/f3 0 2024-04-20T21:31:49.007 INFO:tasks.workunit.client.0.smithi062.stdout:9/68: rename d16/c17 to d16/c1b 0 2024-04-20T21:31:49.008 INFO:tasks.workunit.client.0.smithi062.stdout:8/71: creat d7/d9/f15 x:0 0 0 2024-04-20T21:31:49.011 INFO:tasks.workunit.client.0.smithi062.stdout:6/32: write d0/f6 [161595,1197] 0 2024-04-20T21:31:49.012 INFO:tasks.workunit.client.0.smithi062.stdout:1/91: sync 2024-04-20T21:31:49.012 INFO:tasks.workunit.client.0.smithi062.stdout:0/93: sync 2024-04-20T21:31:49.012 INFO:tasks.workunit.client.0.smithi062.stdout:5/13: sync 2024-04-20T21:31:49.012 INFO:tasks.workunit.client.0.smithi062.stdout:5/14: rmdir - no directory 2024-04-20T21:31:49.016 INFO:tasks.workunit.client.0.smithi062.stdout:8/72: write d7/d9/fb [1814769,34719] 0 2024-04-20T21:31:49.017 INFO:tasks.workunit.client.0.smithi062.stdout:2/65: chown l3 46055 1 2024-04-20T21:31:49.020 INFO:tasks.workunit.client.0.smithi062.stdout:1/92: mknod dc/d15/d17/c1b 0 2024-04-20T21:31:49.025 INFO:tasks.workunit.client.0.smithi062.stdout:8/73: unlink d7/d9/fb 0 2024-04-20T21:31:49.030 INFO:tasks.workunit.client.0.smithi062.stdout:8/74: truncate d7/f14 1152391 0 2024-04-20T21:31:49.033 INFO:tasks.workunit.client.0.smithi062.stdout:3/82: truncate d0/ff 386502 0 2024-04-20T21:31:49.037 INFO:tasks.workunit.client.0.smithi062.stdout:1/93: rmdir dc/d15 39 2024-04-20T21:31:49.039 INFO:tasks.workunit.client.0.smithi062.stdout:8/75: chown d7/d9/ce 17450435 1 2024-04-20T21:31:49.046 INFO:tasks.workunit.client.0.smithi062.stdout:8/76: mknod d7/d9/c16 0 2024-04-20T21:31:49.052 INFO:tasks.workunit.client.0.smithi062.stdout:6/33: dwrite d0/f6 [0,4194304] 0 2024-04-20T21:31:49.052 INFO:tasks.workunit.client.0.smithi062.stdout:3/83: mknod d0/d1c/c21 0 2024-04-20T21:31:49.057 INFO:tasks.workunit.client.0.smithi062.stdout:5/15: dread f4 [0,4194304] 0 2024-04-20T21:31:49.062 INFO:tasks.workunit.client.0.smithi062.stdout:1/94: dread dc/ff [0,4194304] 0 2024-04-20T21:31:49.063 INFO:tasks.workunit.client.0.smithi062.stdout:9/69: dwrite f8 [0,4194304] 0 2024-04-20T21:31:49.063 INFO:tasks.workunit.client.0.smithi062.stdout:0/94: dwrite d7/d1c/f14 [0,4194304] 0 2024-04-20T21:31:49.070 INFO:tasks.workunit.client.0.smithi062.stdout:5/16: creat f5 x:0 0 0 2024-04-20T21:31:49.070 INFO:tasks.workunit.client.0.smithi062.stdout:0/95: chown l3 62713441 1 2024-04-20T21:31:49.077 INFO:tasks.workunit.client.0.smithi062.stdout:2/66: dwrite f6 [0,4194304] 0 2024-04-20T21:31:49.077 INFO:tasks.workunit.client.0.smithi062.stdout:0/96: write d7/f9 [4205630,47445] 0 2024-04-20T21:31:49.077 INFO:tasks.workunit.client.0.smithi062.stdout:3/84: mkdir d0/d22 0 2024-04-20T21:31:49.078 INFO:tasks.workunit.client.0.smithi062.stdout:0/97: read d7/fb [206516,18943] 0 2024-04-20T21:31:49.078 INFO:tasks.workunit.client.0.smithi062.stdout:9/70: creat d16/f1c x:0 0 0 2024-04-20T21:31:49.080 INFO:tasks.workunit.client.0.smithi062.stdout:5/17: symlink l6 0 2024-04-20T21:31:49.080 INFO:tasks.workunit.client.0.smithi062.stdout:8/77: link d7/d9/ff d7/d9/f17 0 2024-04-20T21:31:49.083 INFO:tasks.workunit.client.0.smithi062.stdout:5/18: readlink l0 0 2024-04-20T21:31:49.085 INFO:tasks.workunit.client.0.smithi062.stdout:1/95: rename c3 to dc/c1c 0 2024-04-20T21:31:49.088 INFO:tasks.workunit.client.0.smithi062.stdout:3/85: creat d0/d1b/f23 x:0 0 0 2024-04-20T21:31:49.088 INFO:tasks.workunit.client.0.smithi062.stdout:9/71: creat d16/f1d x:0 0 0 2024-04-20T21:31:49.089 INFO:tasks.workunit.client.0.smithi062.stdout:5/19: mknod c7 0 2024-04-20T21:31:49.089 INFO:tasks.workunit.client.0.smithi062.stdout:2/67: fsync d7/ff 0 2024-04-20T21:31:49.091 INFO:tasks.workunit.client.0.smithi062.stdout:6/34: dread d0/f6 [0,4194304] 0 2024-04-20T21:31:49.091 INFO:tasks.workunit.client.0.smithi062.stdout:0/98: mkdir d7/d20 0 2024-04-20T21:31:49.092 INFO:tasks.workunit.client.0.smithi062.stdout:5/20: mkdir d8 0 2024-04-20T21:31:49.099 INFO:tasks.workunit.client.0.smithi062.stdout:5/21: rename l3 to d8/l9 0 2024-04-20T21:31:49.101 INFO:tasks.workunit.client.0.smithi062.stdout:5/22: stat f4 0 2024-04-20T21:31:49.101 INFO:tasks.workunit.client.0.smithi062.stdout:4/69: sync 2024-04-20T21:31:49.102 INFO:tasks.workunit.client.0.smithi062.stdout:7/22: sync 2024-04-20T21:31:49.105 INFO:tasks.workunit.client.0.smithi062.stdout:6/35: mknod d0/c8 0 2024-04-20T21:31:49.105 INFO:tasks.workunit.client.0.smithi062.stdout:3/86: write d0/d1/f19 [765641,95802] 0 2024-04-20T21:31:49.106 INFO:tasks.workunit.client.0.smithi062.stdout:4/70: write da/dd/f19 [37261,10249] 0 2024-04-20T21:31:49.109 INFO:tasks.workunit.client.0.smithi062.stdout:7/23: mknod d7/cb 0 2024-04-20T21:31:49.109 INFO:tasks.workunit.client.0.smithi062.stdout:0/99: mknod d7/d20/c21 0 2024-04-20T21:31:49.109 INFO:tasks.workunit.client.0.smithi062.stdout:5/23: mknod d8/ca 0 2024-04-20T21:31:49.111 INFO:tasks.workunit.client.0.smithi062.stdout:5/24: creat d8/fb x:0 0 0 2024-04-20T21:31:49.111 INFO:tasks.workunit.client.0.smithi062.stdout:0/100: readlink d7/d1c/l12 0 2024-04-20T21:31:49.113 INFO:tasks.workunit.client.0.smithi062.stdout:2/68: link d7/da/cc d7/da/c11 0 2024-04-20T21:31:49.114 INFO:tasks.workunit.client.0.smithi062.stdout:1/96: dread f1 [0,4194304] 0 2024-04-20T21:31:49.115 INFO:tasks.workunit.client.0.smithi062.stdout:4/71: creat da/dd/d15/f1d x:0 0 0 2024-04-20T21:31:49.117 INFO:tasks.workunit.client.0.smithi062.stdout:6/36: mkdir d0/d9 0 2024-04-20T21:31:49.120 INFO:tasks.workunit.client.0.smithi062.stdout:5/25: creat d8/fc x:0 0 0 2024-04-20T21:31:49.123 INFO:tasks.workunit.client.0.smithi062.stdout:2/69: readlink l4 0 2024-04-20T21:31:49.123 INFO:tasks.workunit.client.0.smithi062.stdout:0/101: fdatasync d7/d1c/d15/d17/d1b/f1a 0 2024-04-20T21:31:49.129 INFO:tasks.workunit.client.0.smithi062.stdout:9/72: dwrite fe [0,4194304] 0 2024-04-20T21:31:49.204 INFO:tasks.workunit.client.0.smithi062.stdout:5/26: symlink d8/ld 0 2024-04-20T21:31:49.206 INFO:tasks.workunit.client.0.smithi062.stdout:3/87: dread d0/d1/fb [0,4194304] 0 2024-04-20T21:31:49.209 INFO:tasks.workunit.client.0.smithi062.stdout:6/37: mknod d0/ca 0 2024-04-20T21:31:49.217 INFO:tasks.workunit.client.0.smithi062.stdout:8/78: dwrite d7/d9/ff [0,4194304] 0 2024-04-20T21:31:49.220 INFO:tasks.workunit.client.0.smithi062.stdout:2/70: write d7/fb [1815965,1421] 0 2024-04-20T21:31:49.225 INFO:tasks.workunit.client.0.smithi062.stdout:5/27: write f4 [1278543,77621] 0 2024-04-20T21:31:49.227 INFO:tasks.workunit.client.0.smithi062.stdout:3/88: mknod d0/d1b/c24 0 2024-04-20T21:31:49.230 INFO:tasks.workunit.client.0.smithi062.stdout:7/24: dwrite f5 [0,4194304] 0 2024-04-20T21:31:49.239 INFO:tasks.workunit.client.0.smithi062.stdout:6/38: write d0/f6 [2383969,66234] 0 2024-04-20T21:31:49.240 INFO:tasks.workunit.client.0.smithi062.stdout:7/25: chown d7/cb 13750886 1 2024-04-20T21:31:49.243 INFO:tasks.workunit.client.0.smithi062.stdout:5/28: creat d8/fe x:0 0 0 2024-04-20T21:31:49.249 INFO:tasks.workunit.client.0.smithi062.stdout:4/72: dwrite da/f1b [0,4194304] 0 2024-04-20T21:31:49.250 INFO:tasks.workunit.client.0.smithi062.stdout:4/73: chown da 223 1 2024-04-20T21:31:49.253 INFO:tasks.workunit.client.0.smithi062.stdout:5/29: unlink f5 0 2024-04-20T21:31:49.255 INFO:tasks.workunit.client.0.smithi062.stdout:7/26: unlink f3 0 2024-04-20T21:31:49.257 INFO:tasks.workunit.client.0.smithi062.stdout:6/39: chown d0/ca 226 1 2024-04-20T21:31:49.258 INFO:tasks.workunit.client.0.smithi062.stdout:8/79: dread d7/d9/f13 [0,4194304] 0 2024-04-20T21:31:49.259 INFO:tasks.workunit.client.0.smithi062.stdout:2/71: truncate f6 2252480 0 2024-04-20T21:31:49.260 INFO:tasks.workunit.client.0.smithi062.stdout:5/30: chown c7 7 1 2024-04-20T21:31:49.261 INFO:tasks.workunit.client.0.smithi062.stdout:0/102: dwrite d7/d1c/f14 [0,4194304] 0 2024-04-20T21:31:49.263 INFO:tasks.workunit.client.0.smithi062.stdout:9/73: dwrite fd [0,4194304] 0 2024-04-20T21:31:49.267 INFO:tasks.workunit.client.0.smithi062.stdout:7/27: write f4 [923422,66297] 0 2024-04-20T21:31:49.267 INFO:tasks.workunit.client.0.smithi062.stdout:8/80: unlink f2 0 2024-04-20T21:31:49.271 INFO:tasks.workunit.client.0.smithi062.stdout:0/103: rmdir d7 39 2024-04-20T21:31:49.272 INFO:tasks.workunit.client.0.smithi062.stdout:6/40: chown d0/f6 96284 1 2024-04-20T21:31:49.274 INFO:tasks.workunit.client.0.smithi062.stdout:7/28: write f5 [4774646,7295] 0 2024-04-20T21:31:49.279 INFO:tasks.workunit.client.0.smithi062.stdout:5/31: creat d8/ff x:0 0 0 2024-04-20T21:31:49.279 INFO:tasks.workunit.client.0.smithi062.stdout:4/74: getdents da/dd/d15 0 2024-04-20T21:31:49.281 INFO:tasks.workunit.client.0.smithi062.stdout:3/89: dwrite d0/d1/f20 [0,4194304] 0 2024-04-20T21:31:49.281 INFO:tasks.workunit.client.0.smithi062.stdout:9/74: rename f8 to d16/f1e 0 2024-04-20T21:31:49.283 INFO:tasks.workunit.client.0.smithi062.stdout:1/97: sync 2024-04-20T21:31:49.284 INFO:tasks.workunit.client.0.smithi062.stdout:7/29: mknod d7/cc 0 2024-04-20T21:31:49.286 INFO:tasks.workunit.client.0.smithi062.stdout:3/90: dread - d0/d1b/f23 zero size 2024-04-20T21:31:49.286 INFO:tasks.workunit.client.0.smithi062.stdout:2/72: creat d7/f12 x:0 0 0 2024-04-20T21:31:49.287 INFO:tasks.workunit.client.0.smithi062.stdout:7/30: truncate f4 1541230 0 2024-04-20T21:31:49.287 INFO:tasks.workunit.client.0.smithi062.stdout:5/32: truncate d8/fe 1016666 0 2024-04-20T21:31:49.287 INFO:tasks.workunit.client.0.smithi062.stdout:4/75: rmdir da/dd/df 39 2024-04-20T21:31:49.288 INFO:tasks.workunit.client.0.smithi062.stdout:7/31: chown f4 2 1 2024-04-20T21:31:49.290 INFO:tasks.workunit.client.0.smithi062.stdout:0/104: creat d7/d1c/d15/f22 x:0 0 0 2024-04-20T21:31:49.291 INFO:tasks.workunit.client.0.smithi062.stdout:4/76: chown da/dd/f12 31667293 1 2024-04-20T21:31:49.294 INFO:tasks.workunit.client.0.smithi062.stdout:8/81: rename d7/f14 to d7/d9/f18 0 2024-04-20T21:31:49.297 INFO:tasks.workunit.client.0.smithi062.stdout:9/75: mknod d16/c1f 0 2024-04-20T21:31:49.299 INFO:tasks.workunit.client.0.smithi062.stdout:9/76: write fc [8176955,799] 0 2024-04-20T21:31:49.300 INFO:tasks.workunit.client.0.smithi062.stdout:2/73: chown d7/l8 11 1 2024-04-20T21:31:49.304 INFO:tasks.workunit.client.0.smithi062.stdout:1/98: mknod dc/d15/d17/c1d 0 2024-04-20T21:31:49.313 INFO:tasks.workunit.client.0.smithi062.stdout:8/82: mkdir d7/d9/d19 0 2024-04-20T21:31:49.314 INFO:tasks.workunit.client.0.smithi062.stdout:3/91: mknod d0/c25 0 2024-04-20T21:31:49.314 INFO:tasks.workunit.client.0.smithi062.stdout:9/77: symlink d16/l20 0 2024-04-20T21:31:49.316 INFO:tasks.workunit.client.0.smithi062.stdout:2/74: creat d7/da/f13 x:0 0 0 2024-04-20T21:31:49.316 INFO:tasks.workunit.client.0.smithi062.stdout:6/41: truncate d0/f3 2478527 0 2024-04-20T21:31:49.317 INFO:tasks.workunit.client.0.smithi062.stdout:9/78: dread - d16/f19 zero size 2024-04-20T21:31:49.317 INFO:tasks.workunit.client.0.smithi062.stdout:3/92: fsync d0/d1/fd 0 2024-04-20T21:31:49.321 INFO:tasks.workunit.client.0.smithi062.stdout:8/83: rename c4 to d7/c1a 0 2024-04-20T21:31:49.323 INFO:tasks.workunit.client.0.smithi062.stdout:0/105: link d7/f9 d7/d1c/f23 0 2024-04-20T21:31:49.323 INFO:tasks.workunit.client.0.smithi062.stdout:9/79: symlink d16/l21 0 2024-04-20T21:31:49.324 INFO:tasks.workunit.client.0.smithi062.stdout:4/77: dread f3 [0,4194304] 0 2024-04-20T21:31:49.327 INFO:tasks.workunit.client.0.smithi062.stdout:9/80: dread - f15 zero size 2024-04-20T21:31:49.327 INFO:tasks.workunit.client.0.smithi062.stdout:3/93: symlink d0/d1/d10/l26 0 2024-04-20T21:31:49.328 INFO:tasks.workunit.client.0.smithi062.stdout:6/42: mknod d0/cb 0 2024-04-20T21:31:49.330 INFO:tasks.workunit.client.0.smithi062.stdout:7/32: dwrite f4 [0,4194304] 0 2024-04-20T21:31:49.331 INFO:tasks.workunit.client.0.smithi062.stdout:8/84: mkdir d7/d9/d1b 0 2024-04-20T21:31:49.334 INFO:tasks.workunit.client.0.smithi062.stdout:9/81: read f13 [86560,101208] 0 2024-04-20T21:31:49.336 INFO:tasks.workunit.client.0.smithi062.stdout:6/43: write d0/f6 [978904,42215] 0 2024-04-20T21:31:49.338 INFO:tasks.workunit.client.0.smithi062.stdout:0/106: write d7/d1c/d15/d17/d1b/f1a [751412,95022] 0 2024-04-20T21:31:49.343 INFO:tasks.workunit.client.0.smithi062.stdout:6/44: write d0/f6 [3048487,85548] 0 2024-04-20T21:31:49.345 INFO:tasks.workunit.client.0.smithi062.stdout:4/78: creat da/dd/d15/f1e x:0 0 0 2024-04-20T21:31:49.345 INFO:tasks.workunit.client.0.smithi062.stdout:9/82: mknod d16/c22 0 2024-04-20T21:31:49.347 INFO:tasks.workunit.client.0.smithi062.stdout:0/107: fdatasync d7/d1c/f14 0 2024-04-20T21:31:49.355 INFO:tasks.workunit.client.0.smithi062.stdout:3/94: link d0/d1/f19 d0/d1/d10/f27 0 2024-04-20T21:31:49.355 INFO:tasks.workunit.client.0.smithi062.stdout:8/85: creat d7/d9/d1b/f1c x:0 0 0 2024-04-20T21:31:49.356 INFO:tasks.workunit.client.0.smithi062.stdout:1/99: dwrite dc/fe [0,4194304] 0 2024-04-20T21:31:49.357 INFO:tasks.workunit.client.0.smithi062.stdout:3/95: read d0/d1/fb [532883,108478] 0 2024-04-20T21:31:49.358 INFO:tasks.workunit.client.0.smithi062.stdout:6/45: mknod d0/cc 0 2024-04-20T21:31:49.363 INFO:tasks.workunit.client.0.smithi062.stdout:2/75: dwrite f6 [0,4194304] 0 2024-04-20T21:31:49.366 INFO:tasks.workunit.client.0.smithi062.stdout:0/108: creat d7/d1e/f24 x:0 0 0 2024-04-20T21:31:49.372 INFO:tasks.workunit.client.0.smithi062.stdout:1/100: creat dc/d15/d17/f1e x:0 0 0 2024-04-20T21:31:49.372 INFO:tasks.workunit.client.0.smithi062.stdout:3/96: write d0/d1/fb [2462328,24742] 0 2024-04-20T21:31:49.376 INFO:tasks.workunit.client.0.smithi062.stdout:2/76: chown c2 21 1 2024-04-20T21:31:49.379 INFO:tasks.workunit.client.0.smithi062.stdout:4/79: getdents da/dd/df 0 2024-04-20T21:31:49.381 INFO:tasks.workunit.client.0.smithi062.stdout:1/101: mknod dc/c1f 0 2024-04-20T21:31:49.386 INFO:tasks.workunit.client.0.smithi062.stdout:2/77: chown d7/da/cc 28280214 1 2024-04-20T21:31:49.391 INFO:tasks.workunit.client.0.smithi062.stdout:0/109: write d7/d1c/f23 [4906188,43258] 0 2024-04-20T21:31:49.392 INFO:tasks.workunit.client.0.smithi062.stdout:4/80: fsync da/dd/f12 0 2024-04-20T21:31:49.394 INFO:tasks.workunit.client.0.smithi062.stdout:3/97: mknod d0/d22/c28 0 2024-04-20T21:31:49.394 INFO:tasks.workunit.client.0.smithi062.stdout:9/83: dwrite d16/f1e [0,4194304] 0 2024-04-20T21:31:49.396 INFO:tasks.workunit.client.0.smithi062.stdout:0/110: chown d7/c10 82098 1 2024-04-20T21:31:49.396 INFO:tasks.workunit.client.0.smithi062.stdout:4/81: fsync da/f1b 0 2024-04-20T21:31:49.398 INFO:tasks.workunit.client.0.smithi062.stdout:2/78: rename d7 to d7/d14 22 2024-04-20T21:31:49.398 INFO:tasks.workunit.client.0.smithi062.stdout:8/86: dread f6 [0,4194304] 0 2024-04-20T21:31:49.399 INFO:tasks.workunit.client.0.smithi062.stdout:6/46: dread d0/f3 [0,4194304] 0 2024-04-20T21:31:49.415 INFO:tasks.workunit.client.0.smithi062.stdout:3/98: creat d0/d1/f29 x:0 0 0 2024-04-20T21:31:49.415 INFO:tasks.workunit.client.0.smithi062.stdout:9/84: write d16/f19 [606038,93727] 0 2024-04-20T21:31:49.415 INFO:tasks.workunit.client.0.smithi062.stdout:2/79: mkdir d7/da/d15 0 2024-04-20T21:31:49.416 INFO:tasks.workunit.client.0.smithi062.stdout:3/99: fdatasync d0/f8 0 2024-04-20T21:31:49.416 INFO:tasks.workunit.client.0.smithi062.stdout:4/82: symlink da/dd/d15/l1f 0 2024-04-20T21:31:49.417 INFO:tasks.workunit.client.0.smithi062.stdout:0/111: chown c0 954241396 1 2024-04-20T21:31:49.417 INFO:tasks.workunit.client.0.smithi062.stdout:9/85: stat d16/l20 0 2024-04-20T21:31:49.418 INFO:tasks.workunit.client.0.smithi062.stdout:8/87: rename d7/f10 to d7/d9/d19/f1d 0 2024-04-20T21:31:49.418 INFO:tasks.workunit.client.0.smithi062.stdout:2/80: stat d7/f12 0 2024-04-20T21:31:49.418 INFO:tasks.workunit.client.0.smithi062.stdout:6/47: symlink d0/ld 0 2024-04-20T21:31:49.418 INFO:tasks.workunit.client.0.smithi062.stdout:3/100: dread - d0/d1/f29 zero size 2024-04-20T21:31:49.418 INFO:tasks.workunit.client.0.smithi062.stdout:0/112: symlink d7/d1e/l25 0 2024-04-20T21:31:49.427 INFO:tasks.workunit.client.0.smithi062.stdout:8/88: creat d7/d9/d1b/f1e x:0 0 0 2024-04-20T21:31:49.428 INFO:tasks.workunit.client.0.smithi062.stdout:3/101: dread - d0/d1/f29 zero size 2024-04-20T21:31:49.428 INFO:tasks.workunit.client.0.smithi062.stdout:1/102: dwrite dc/d15/f18 [0,4194304] 0 2024-04-20T21:31:49.428 INFO:tasks.workunit.client.0.smithi062.stdout:9/86: getdents d16 0 2024-04-20T21:31:49.428 INFO:tasks.workunit.client.0.smithi062.stdout:0/113: dread - d7/d1e/f24 zero size 2024-04-20T21:31:49.432 INFO:tasks.workunit.client.0.smithi062.stdout:3/102: dread - d0/d1/f29 zero size 2024-04-20T21:31:49.433 INFO:tasks.workunit.client.0.smithi062.stdout:8/89: chown d7/d9/ce 1403 1 2024-04-20T21:31:49.438 INFO:tasks.workunit.client.0.smithi062.stdout:3/103: chown d0/f8 28573793 1 2024-04-20T21:31:49.441 INFO:tasks.workunit.client.0.smithi062.stdout:3/104: read - d0/d1/f29 zero size 2024-04-20T21:31:49.443 INFO:tasks.workunit.client.0.smithi062.stdout:6/48: dread d0/f3 [0,4194304] 0 2024-04-20T21:31:49.444 INFO:tasks.workunit.client.0.smithi062.stdout:4/83: dwrite da/dd/df/f17 [0,4194304] 0 2024-04-20T21:31:49.466 INFO:tasks.workunit.client.0.smithi062.stdout:5/33: sync 2024-04-20T21:31:49.490 INFO:tasks.workunit.client.0.smithi062.stdout:5/34: dwrite f4 [0,4194304] 0 2024-04-20T21:31:49.512 INFO:tasks.workunit.client.0.smithi062.stdout:2/81: rename l3 to d7/l16 0 2024-04-20T21:31:49.514 INFO:tasks.workunit.client.0.smithi062.stdout:9/87: mknod d16/c23 0 2024-04-20T21:31:49.515 INFO:tasks.workunit.client.0.smithi062.stdout:0/114: mkdir d7/d1c/d15/d17/d26 0 2024-04-20T21:31:49.516 INFO:tasks.workunit.client.0.smithi062.stdout:1/103: mkdir dc/d15/d20 0 2024-04-20T21:31:49.517 INFO:tasks.workunit.client.0.smithi062.stdout:1/104: chown dc/d15/d17 1 1 2024-04-20T21:31:49.517 INFO:tasks.workunit.client.0.smithi062.stdout:1/105: readlink - no filename 2024-04-20T21:31:49.517 INFO:tasks.workunit.client.0.smithi062.stdout:1/106: read - f9 zero size 2024-04-20T21:31:49.519 INFO:tasks.workunit.client.0.smithi062.stdout:8/90: creat d7/d9/d1b/f1f x:0 0 0 2024-04-20T21:31:49.521 INFO:tasks.workunit.client.0.smithi062.stdout:8/91: write d7/d9/f15 [484470,106984] 0 2024-04-20T21:31:49.523 INFO:tasks.workunit.client.0.smithi062.stdout:4/84: creat da/dd/d15/f20 x:0 0 0 2024-04-20T21:31:49.525 INFO:tasks.workunit.client.0.smithi062.stdout:5/35: mkdir d8/d10 0 2024-04-20T21:31:49.552 INFO:tasks.workunit.client.0.smithi062.stdout:9/88: chown c3 6551750 1 2024-04-20T21:31:49.553 INFO:tasks.workunit.client.0.smithi062.stdout:8/92: creat d7/d9/d19/f20 x:0 0 0 2024-04-20T21:31:49.553 INFO:tasks.workunit.client.0.smithi062.stdout:9/89: readlink d16/l1a 0 2024-04-20T21:31:49.553 INFO:tasks.workunit.client.0.smithi062.stdout:4/85: symlink da/dd/d15/l21 0 2024-04-20T21:31:49.554 INFO:tasks.workunit.client.0.smithi062.stdout:8/93: chown d7/d9/ff 1287184 1 2024-04-20T21:31:49.557 INFO:tasks.workunit.client.0.smithi062.stdout:4/86: truncate da/dd/f19 366392 0 2024-04-20T21:31:49.557 INFO:tasks.workunit.client.0.smithi062.stdout:6/49: creat d0/fe x:0 0 0 2024-04-20T21:31:49.557 INFO:tasks.workunit.client.0.smithi062.stdout:8/94: chown d7/d9/ce 390510 1 2024-04-20T21:31:49.559 INFO:tasks.workunit.client.0.smithi062.stdout:5/36: symlink d8/d10/l11 0 2024-04-20T21:31:49.561 INFO:tasks.workunit.client.0.smithi062.stdout:9/90: rename l14 to d16/l24 0 2024-04-20T21:31:49.561 INFO:tasks.workunit.client.0.smithi062.stdout:5/37: fdatasync d8/fb 0 2024-04-20T21:31:49.563 INFO:tasks.workunit.client.0.smithi062.stdout:1/107: rename dc/f1a to dc/f21 0 2024-04-20T21:31:49.564 INFO:tasks.workunit.client.0.smithi062.stdout:3/105: write d0/d1/d10/f27 [1436908,2084] 0 2024-04-20T21:31:49.564 INFO:tasks.workunit.client.0.smithi062.stdout:0/115: getdents d7/d1e 0 2024-04-20T21:31:49.566 INFO:tasks.workunit.client.0.smithi062.stdout:0/116: readlink l3 0 2024-04-20T21:31:49.568 INFO:tasks.workunit.client.0.smithi062.stdout:9/91: unlink f7 0 2024-04-20T21:31:49.568 INFO:tasks.workunit.client.0.smithi062.stdout:8/95: rename d7/d9/d19/f1d to d7/d9/d1b/f21 0 2024-04-20T21:31:49.569 INFO:tasks.workunit.client.0.smithi062.stdout:0/117: readlink d7/d1e/l25 0 2024-04-20T21:31:49.571 INFO:tasks.workunit.client.0.smithi062.stdout:5/38: symlink d8/d10/l12 0 2024-04-20T21:31:49.572 INFO:tasks.workunit.client.0.smithi062.stdout:3/106: write d0/d1/d10/f27 [344382,110050] 0 2024-04-20T21:31:49.575 INFO:tasks.workunit.client.0.smithi062.stdout:9/92: symlink d16/l25 0 2024-04-20T21:31:49.576 INFO:tasks.workunit.client.0.smithi062.stdout:5/39: read d8/fe [530931,112464] 0 2024-04-20T21:31:49.578 INFO:tasks.workunit.client.0.smithi062.stdout:0/118: fdatasync f1 0 2024-04-20T21:31:49.580 INFO:tasks.workunit.client.0.smithi062.stdout:9/93: fdatasync f13 0 2024-04-20T21:31:49.581 INFO:tasks.workunit.client.0.smithi062.stdout:4/87: getdents da/dd 0 2024-04-20T21:31:49.581 INFO:tasks.workunit.client.0.smithi062.stdout:0/119: stat d7/cd 0 2024-04-20T21:31:49.583 INFO:tasks.workunit.client.0.smithi062.stdout:5/40: mknod d8/d10/c13 0 2024-04-20T21:31:49.585 INFO:tasks.workunit.client.0.smithi062.stdout:1/108: getdents dc 0 2024-04-20T21:31:49.585 INFO:tasks.workunit.client.0.smithi062.stdout:9/94: read - d16/f1d zero size 2024-04-20T21:31:49.586 INFO:tasks.workunit.client.0.smithi062.stdout:6/50: dwrite d0/fe [0,4194304] 0 2024-04-20T21:31:49.586 INFO:tasks.workunit.client.0.smithi062.stdout:3/107: rename d0/d1/d10/c14 to d0/d11/c2a 0 2024-04-20T21:31:49.588 INFO:tasks.workunit.client.0.smithi062.stdout:4/88: write da/fb [2897948,123640] 0 2024-04-20T21:31:49.590 INFO:tasks.workunit.client.0.smithi062.stdout:0/120: mknod d7/d1c/d15/d17/c27 0 2024-04-20T21:31:49.591 INFO:tasks.workunit.client.0.smithi062.stdout:5/41: mknod d8/d10/c14 0 2024-04-20T21:31:49.592 INFO:tasks.workunit.client.0.smithi062.stdout:3/108: dread - d0/d1/fd zero size 2024-04-20T21:31:49.594 INFO:tasks.workunit.client.0.smithi062.stdout:1/109: mknod dc/d15/c22 0 2024-04-20T21:31:49.595 INFO:tasks.workunit.client.0.smithi062.stdout:9/95: symlink d16/l26 0 2024-04-20T21:31:49.595 INFO:tasks.workunit.client.0.smithi062.stdout:5/42: chown d8 6351237 1 2024-04-20T21:31:49.598 INFO:tasks.workunit.client.0.smithi062.stdout:0/121: mknod d7/d20/c28 0 2024-04-20T21:31:49.598 INFO:tasks.workunit.client.0.smithi062.stdout:4/89: chown da/dd/f12 34005 1 2024-04-20T21:31:49.603 INFO:tasks.workunit.client.0.smithi062.stdout:8/96: dwrite d7/fc [0,4194304] 0 2024-04-20T21:31:49.603 INFO:tasks.workunit.client.0.smithi062.stdout:0/122: chown d7/d1c/f23 41361 1 2024-04-20T21:31:49.604 INFO:tasks.workunit.client.0.smithi062.stdout:1/110: write dc/d15/f18 [417309,15376] 0 2024-04-20T21:31:49.606 INFO:tasks.workunit.client.0.smithi062.stdout:6/51: write d0/f3 [2957250,19126] 0 2024-04-20T21:31:49.609 INFO:tasks.workunit.client.0.smithi062.stdout:5/43: truncate d8/fb 838615 0 2024-04-20T21:31:49.611 INFO:tasks.workunit.client.0.smithi062.stdout:4/90: mknod da/dd/d15/c22 0 2024-04-20T21:31:49.619 INFO:tasks.workunit.client.0.smithi062.stdout:4/91: fdatasync da/dd/f19 0 2024-04-20T21:31:49.622 INFO:tasks.workunit.client.0.smithi062.stdout:1/111: truncate dc/d15/d17/f1e 109814 0 2024-04-20T21:31:49.626 INFO:tasks.workunit.client.0.smithi062.stdout:6/52: getdents d0 0 2024-04-20T21:31:49.629 INFO:tasks.workunit.client.0.smithi062.stdout:5/44: truncate f4 1528842 0 2024-04-20T21:31:49.636 INFO:tasks.workunit.client.0.smithi062.stdout:1/112: write dc/d15/f18 [2198203,98902] 0 2024-04-20T21:31:49.640 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:49 smithi189 ceph-mon[21562]: mgrmap e29: smithi062.rqxuxi(active, since 11s), standbys: smithi189.tawavz 2024-04-20T21:31:49.640 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:49 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mgr metadata", "who": "smithi189.tawavz", "id": "smithi189.tawavz"}]: dispatch 2024-04-20T21:31:49.640 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:49 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:31:49.640 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:49 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:31:49.641 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:49 smithi189 ceph-mon[21562]: pgmap v9: 65 pgs: 65 active+clean; 226 MiB data, 881 MiB used, 536 GiB / 536 GiB avail; 1.1 MiB/s rd, 3.2 MiB/s wr, 222 op/s 2024-04-20T21:31:49.645 INFO:tasks.workunit.client.0.smithi062.stdout:3/109: dwrite d0/f8 [0,4194304] 0 2024-04-20T21:31:49.646 INFO:tasks.workunit.client.0.smithi062.stdout:9/96: dwrite d16/f1e [0,4194304] 0 2024-04-20T21:31:49.646 INFO:tasks.workunit.client.0.smithi062.stdout:4/92: mknod da/dd/c23 0 2024-04-20T21:31:49.654 INFO:tasks.workunit.client.0.smithi062.stdout:7/33: sync 2024-04-20T21:31:49.657 INFO:tasks.workunit.client.0.smithi062.stdout:0/123: dwrite d7/d1c/f23 [0,4194304] 0 2024-04-20T21:31:49.659 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:49 smithi062 ceph-mon[16221]: mgrmap e29: smithi062.rqxuxi(active, since 11s), standbys: smithi189.tawavz 2024-04-20T21:31:49.659 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:49 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mgr metadata", "who": "smithi189.tawavz", "id": "smithi189.tawavz"}]: dispatch 2024-04-20T21:31:49.659 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:49 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:31:49.659 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:49 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:31:49.659 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:49 smithi062 ceph-mon[16221]: pgmap v9: 65 pgs: 65 active+clean; 226 MiB data, 881 MiB used, 536 GiB / 536 GiB avail; 1.1 MiB/s rd, 3.2 MiB/s wr, 222 op/s 2024-04-20T21:31:49.660 INFO:tasks.workunit.client.0.smithi062.stdout:8/97: dwrite d7/d9/ff [0,4194304] 0 2024-04-20T21:31:49.668 INFO:tasks.workunit.client.0.smithi062.stdout:5/45: rename d8/d10/l11 to d8/d10/l15 0 2024-04-20T21:31:49.674 INFO:tasks.workunit.client.0.smithi062.stdout:3/110: creat d0/d1b/f2b x:0 0 0 2024-04-20T21:31:49.680 INFO:tasks.workunit.client.0.smithi062.stdout:7/34: symlink d7/ld 0 2024-04-20T21:31:49.683 INFO:tasks.workunit.client.0.smithi062.stdout:6/53: dwrite d0/fe [4194304,4194304] 0 2024-04-20T21:31:49.692 INFO:tasks.workunit.client.0.smithi062.stdout:9/97: dread fc [4194304,4194304] 0 2024-04-20T21:31:49.696 INFO:tasks.workunit.client.0.smithi062.stdout:8/98: link d7/d9/d19/f20 d7/f22 0 2024-04-20T21:31:49.700 INFO:tasks.workunit.client.0.smithi062.stdout:6/54: rename d0/c8 to d0/d9/cf 0 2024-04-20T21:31:49.702 INFO:tasks.workunit.client.0.smithi062.stdout:9/98: mknod d16/c27 0 2024-04-20T21:31:49.702 INFO:tasks.workunit.client.0.smithi062.stdout:4/93: fsync da/dd/df/f17 0 2024-04-20T21:31:49.702 INFO:tasks.workunit.client.0.smithi062.stdout:3/111: read d0/d1/d10/f27 [1393759,15828] 0 2024-04-20T21:31:49.706 INFO:tasks.workunit.client.0.smithi062.stdout:9/99: chown fe 41293 1 2024-04-20T21:31:49.706 INFO:tasks.workunit.client.0.smithi062.stdout:4/94: stat da/dd/df/f1a 0 2024-04-20T21:31:49.708 INFO:tasks.workunit.client.0.smithi062.stdout:8/99: creat d7/f23 x:0 0 0 2024-04-20T21:31:49.710 INFO:tasks.workunit.client.0.smithi062.stdout:7/35: dread f4 [0,4194304] 0 2024-04-20T21:31:49.713 INFO:tasks.workunit.client.0.smithi062.stdout:3/112: mkdir d0/d1/d10/d2c 0 2024-04-20T21:31:49.714 INFO:tasks.workunit.client.0.smithi062.stdout:0/124: dwrite f2 [0,4194304] 0 2024-04-20T21:31:49.718 INFO:tasks.workunit.client.0.smithi062.stdout:1/113: write dc/f14 [1016881,63854] 0 2024-04-20T21:31:49.718 INFO:tasks.workunit.client.0.smithi062.stdout:3/113: write d0/d1/fb [2396226,806] 0 2024-04-20T21:31:49.719 INFO:tasks.workunit.client.0.smithi062.stdout:4/95: creat da/dd/df/f24 x:0 0 0 2024-04-20T21:31:49.719 INFO:tasks.workunit.client.0.smithi062.stdout:5/46: rmdir d8 39 2024-04-20T21:31:49.726 INFO:tasks.workunit.client.0.smithi062.stdout:2/82: sync 2024-04-20T21:31:49.726 INFO:tasks.workunit.client.0.smithi062.stdout:4/96: read - da/dd/df/f10 zero size 2024-04-20T21:31:49.726 INFO:tasks.workunit.client.0.smithi062.stdout:8/100: rename d7/d9/d19 to d7/d9/d1b/d24 0 2024-04-20T21:31:49.727 INFO:tasks.workunit.client.0.smithi062.stdout:0/125: mkdir d7/d20/d29 0 2024-04-20T21:31:49.729 INFO:tasks.workunit.client.0.smithi062.stdout:0/126: read f1 [359445,45298] 0 2024-04-20T21:31:49.729 INFO:tasks.workunit.client.0.smithi062.stdout:1/114: creat dc/d15/d17/f23 x:0 0 0 2024-04-20T21:31:49.730 INFO:tasks.workunit.client.0.smithi062.stdout:3/114: creat d0/d22/f2d x:0 0 0 2024-04-20T21:31:49.732 INFO:tasks.workunit.client.0.smithi062.stdout:7/36: rmdir d7/d9 0 2024-04-20T21:31:49.734 INFO:tasks.workunit.client.0.smithi062.stdout:8/101: mkdir d7/d25 0 2024-04-20T21:31:49.734 INFO:tasks.workunit.client.0.smithi062.stdout:0/127: write d7/d1c/ff [1938668,89268] 0 2024-04-20T21:31:49.740 INFO:tasks.workunit.client.0.smithi062.stdout:5/47: symlink d8/d10/l16 0 2024-04-20T21:31:49.740 INFO:tasks.workunit.client.0.smithi062.stdout:9/100: getdents d16 0 2024-04-20T21:31:49.740 INFO:tasks.workunit.client.0.smithi062.stdout:6/55: read d0/f6 [2065402,103781] 0 2024-04-20T21:31:49.741 INFO:tasks.workunit.client.0.smithi062.stdout:1/115: mknod dc/d15/d17/c24 0 2024-04-20T21:31:49.741 INFO:tasks.workunit.client.0.smithi062.stdout:9/101: fsync fb 0 2024-04-20T21:31:49.742 INFO:tasks.workunit.client.0.smithi062.stdout:7/37: rename d7/fa to d7/fe 0 2024-04-20T21:31:49.742 INFO:tasks.workunit.client.0.smithi062.stdout:5/48: stat d8/fc 0 2024-04-20T21:31:49.743 INFO:tasks.workunit.client.0.smithi062.stdout:8/102: symlink d7/d9/d1b/d24/l26 0 2024-04-20T21:31:49.744 INFO:tasks.workunit.client.0.smithi062.stdout:4/97: getdents da/dd 0 2024-04-20T21:31:49.744 INFO:tasks.workunit.client.0.smithi062.stdout:6/56: readlink d0/ld 0 2024-04-20T21:31:49.745 INFO:tasks.workunit.client.0.smithi062.stdout:9/102: unlink c3 0 2024-04-20T21:31:49.746 INFO:tasks.workunit.client.0.smithi062.stdout:0/128: mknod d7/d1c/d15/d17/c2a 0 2024-04-20T21:31:49.746 INFO:tasks.workunit.client.0.smithi062.stdout:3/115: link d0/d1/le d0/d1/d10/d2c/l2e 0 2024-04-20T21:31:49.746 INFO:tasks.workunit.client.0.smithi062.stdout:5/49: write d8/ff [451904,63594] 0 2024-04-20T21:31:49.747 INFO:tasks.workunit.client.0.smithi062.stdout:6/57: rmdir d0 39 2024-04-20T21:31:49.750 INFO:tasks.workunit.client.0.smithi062.stdout:8/103: mknod d7/d9/d1b/c27 0 2024-04-20T21:31:49.752 INFO:tasks.workunit.client.0.smithi062.stdout:3/116: chown d0/d11/c1e 605 1 2024-04-20T21:31:49.752 INFO:tasks.workunit.client.0.smithi062.stdout:9/103: getdents d16 0 2024-04-20T21:31:49.752 INFO:tasks.workunit.client.0.smithi062.stdout:3/117: chown d0/d1 10727360 1 2024-04-20T21:31:49.753 INFO:tasks.workunit.client.0.smithi062.stdout:4/98: link da/dd/c23 da/dd/df/c25 0 2024-04-20T21:31:49.755 INFO:tasks.workunit.client.0.smithi062.stdout:5/50: creat d8/d10/f17 x:0 0 0 2024-04-20T21:31:49.756 INFO:tasks.workunit.client.0.smithi062.stdout:3/118: chown d0/d1/c16 43210 1 2024-04-20T21:31:49.757 INFO:tasks.workunit.client.0.smithi062.stdout:6/58: creat d0/f10 x:0 0 0 2024-04-20T21:31:49.757 INFO:tasks.workunit.client.0.smithi062.stdout:4/99: dread - da/dd/df/f1a zero size 2024-04-20T21:31:49.760 INFO:tasks.workunit.client.0.smithi062.stdout:5/51: mknod d8/d10/c18 0 2024-04-20T21:31:49.763 INFO:tasks.workunit.client.0.smithi062.stdout:5/52: fdatasync d8/d10/f17 0 2024-04-20T21:31:49.765 INFO:tasks.workunit.client.0.smithi062.stdout:3/119: write d0/d1/f20 [3056301,58963] 0 2024-04-20T21:31:49.767 INFO:tasks.workunit.client.0.smithi062.stdout:4/100: write da/dd/f14 [1243661,29386] 0 2024-04-20T21:31:49.770 INFO:tasks.workunit.client.0.smithi062.stdout:5/53: dread - d8/fc zero size 2024-04-20T21:31:49.778 INFO:tasks.workunit.client.0.smithi062.stdout:8/104: dwrite d7/d9/f18 [0,4194304] 0 2024-04-20T21:31:49.781 INFO:tasks.workunit.client.0.smithi062.stdout:3/120: creat d0/f2f x:0 0 0 2024-04-20T21:31:49.784 INFO:tasks.workunit.client.0.smithi062.stdout:0/129: dwrite d7/f9 [0,4194304] 0 2024-04-20T21:31:49.785 INFO:tasks.workunit.client.0.smithi062.stdout:9/104: dwrite d16/f19 [0,4194304] 0 2024-04-20T21:31:49.787 INFO:tasks.workunit.client.0.smithi062.stdout:3/121: stat d0/d11/c1e 0 2024-04-20T21:31:49.790 INFO:tasks.workunit.client.0.smithi062.stdout:5/54: dread d8/fe [0,4194304] 0 2024-04-20T21:31:49.795 INFO:tasks.workunit.client.0.smithi062.stdout:9/105: symlink d16/l28 0 2024-04-20T21:31:49.797 INFO:tasks.workunit.client.0.smithi062.stdout:5/55: write d8/d10/f17 [1039922,81577] 0 2024-04-20T21:31:49.799 INFO:tasks.workunit.client.0.smithi062.stdout:8/105: link d7/d9/d1b/c27 d7/d9/d1b/d24/c28 0 2024-04-20T21:31:49.803 INFO:tasks.workunit.client.0.smithi062.stdout:5/56: creat d8/f19 x:0 0 0 2024-04-20T21:31:49.803 INFO:tasks.workunit.client.0.smithi062.stdout:8/106: read - d7/f23 zero size 2024-04-20T21:31:49.807 INFO:tasks.workunit.client.0.smithi062.stdout:0/130: link d7/d20/c21 d7/d1e/c2b 0 2024-04-20T21:31:49.808 INFO:tasks.workunit.client.0.smithi062.stdout:6/59: dread d0/fe [0,4194304] 0 2024-04-20T21:31:49.809 INFO:tasks.workunit.client.0.smithi062.stdout:8/107: mkdir d7/d9/d29 0 2024-04-20T21:31:49.810 INFO:tasks.workunit.client.0.smithi062.stdout:0/131: symlink d7/d1e/l2c 0 2024-04-20T21:31:49.815 INFO:tasks.workunit.client.0.smithi062.stdout:6/60: mkdir d0/d9/d11 0 2024-04-20T21:31:49.819 INFO:tasks.workunit.client.0.smithi062.stdout:0/132: unlink d7/d1c/l12 0 2024-04-20T21:31:49.821 INFO:tasks.workunit.client.0.smithi062.stdout:6/61: stat d0/cc 0 2024-04-20T21:31:49.834 INFO:tasks.workunit.client.0.smithi062.stdout:6/62: symlink d0/d9/l12 0 2024-04-20T21:31:49.835 INFO:tasks.workunit.client.0.smithi062.stdout:3/122: dwrite d0/ff [0,4194304] 0 2024-04-20T21:31:49.835 INFO:tasks.workunit.client.0.smithi062.stdout:0/133: write d7/d1c/ff [2435301,16705] 0 2024-04-20T21:31:49.848 INFO:tasks.workunit.client.0.smithi062.stdout:6/63: creat d0/d9/f13 x:0 0 0 2024-04-20T21:31:49.851 INFO:tasks.workunit.client.0.smithi062.stdout:9/106: dwrite fb [0,4194304] 0 2024-04-20T21:31:49.852 INFO:tasks.workunit.client.0.smithi062.stdout:5/57: dwrite d8/d10/f17 [0,4194304] 0 2024-04-20T21:31:49.859 INFO:tasks.workunit.client.0.smithi062.stdout:0/134: creat d7/d1c/d15/f2d x:0 0 0 2024-04-20T21:31:49.862 INFO:tasks.workunit.client.0.smithi062.stdout:8/108: dwrite d7/d9/f12 [0,4194304] 0 2024-04-20T21:31:49.867 INFO:tasks.workunit.client.0.smithi062.stdout:0/135: chown d7/d1c/d15/d17/d1b/c1f 3514 1 2024-04-20T21:31:49.867 INFO:tasks.workunit.client.0.smithi062.stdout:9/107: dread f4 [0,4194304] 0 2024-04-20T21:31:49.871 INFO:tasks.workunit.client.0.smithi062.stdout:8/109: creat d7/d9/d1b/d24/f2a x:0 0 0 2024-04-20T21:31:49.871 INFO:tasks.workunit.client.0.smithi062.stdout:9/108: symlink d16/l29 0 2024-04-20T21:31:49.872 INFO:tasks.workunit.client.0.smithi062.stdout:5/58: write d8/fe [1683018,33663] 0 2024-04-20T21:31:49.874 INFO:tasks.workunit.client.0.smithi062.stdout:0/136: truncate d7/f9 5893487 0 2024-04-20T21:31:49.880 INFO:tasks.workunit.client.0.smithi062.stdout:5/59: creat d8/f1a x:0 0 0 2024-04-20T21:31:49.883 INFO:tasks.workunit.client.0.smithi062.stdout:3/123: dwrite d0/d1b/f2b [0,4194304] 0 2024-04-20T21:31:49.883 INFO:tasks.workunit.client.0.smithi062.stdout:8/110: creat d7/d9/d29/f2b x:0 0 0 2024-04-20T21:31:49.887 INFO:tasks.workunit.client.0.smithi062.stdout:8/111: chown d7/d9/d1b 1 1 2024-04-20T21:31:49.887 INFO:tasks.workunit.client.0.smithi062.stdout:0/137: link d7/c16 d7/d1c/c2e 0 2024-04-20T21:31:49.887 INFO:tasks.workunit.client.0.smithi062.stdout:6/64: dwrite d0/f6 [0,4194304] 0 2024-04-20T21:31:49.888 INFO:tasks.workunit.client.0.smithi062.stdout:3/124: dread - d0/d1b/f23 zero size 2024-04-20T21:31:49.891 INFO:tasks.workunit.client.0.smithi062.stdout:5/60: getdents d8/d10 0 2024-04-20T21:31:49.894 INFO:tasks.workunit.client.0.smithi062.stdout:0/138: symlink d7/d20/l2f 0 2024-04-20T21:31:49.898 INFO:tasks.workunit.client.0.smithi062.stdout:8/112: mknod d7/d9/d1b/c2c 0 2024-04-20T21:31:49.899 INFO:tasks.workunit.client.0.smithi062.stdout:9/109: dwrite fb [0,4194304] 0 2024-04-20T21:31:49.901 INFO:tasks.workunit.client.0.smithi062.stdout:3/125: link d0/d1b/f2b d0/d1/d10/d2c/f30 0 2024-04-20T21:31:49.905 INFO:tasks.workunit.client.0.smithi062.stdout:3/126: mkdir d0/d1/d10/d31 0 2024-04-20T21:31:49.907 INFO:tasks.workunit.client.0.smithi062.stdout:6/65: dread d0/fe [4194304,4194304] 0 2024-04-20T21:31:49.907 INFO:tasks.workunit.client.0.smithi062.stdout:5/61: link d8/f19 d8/d10/f1b 0 2024-04-20T21:31:49.910 INFO:tasks.workunit.client.0.smithi062.stdout:3/127: write d0/d1/f19 [2410882,94178] 0 2024-04-20T21:31:49.912 INFO:tasks.workunit.client.0.smithi062.stdout:6/66: mknod d0/d9/d11/c14 0 2024-04-20T21:31:49.914 INFO:tasks.workunit.client.0.smithi062.stdout:6/67: dread - d0/d9/f13 zero size 2024-04-20T21:31:49.916 INFO:tasks.workunit.client.0.smithi062.stdout:8/113: dread d7/d9/f17 [0,4194304] 0 2024-04-20T21:31:49.916 INFO:tasks.workunit.client.0.smithi062.stdout:3/128: truncate d0/d1/fd 426384 0 2024-04-20T21:31:49.919 INFO:tasks.workunit.client.0.smithi062.stdout:6/68: creat d0/d9/f15 x:0 0 0 2024-04-20T21:31:49.922 INFO:tasks.workunit.client.0.smithi062.stdout:9/110: dwrite d16/f19 [0,4194304] 0 2024-04-20T21:31:49.927 INFO:tasks.workunit.client.0.smithi062.stdout:3/129: write d0/d1/d10/d2c/f30 [3675979,17520] 0 2024-04-20T21:31:49.932 INFO:tasks.workunit.client.0.smithi062.stdout:6/69: link d0/f6 d0/d9/f16 0 2024-04-20T21:31:49.933 INFO:tasks.workunit.client.0.smithi062.stdout:5/62: dwrite d8/f1a [0,4194304] 0 2024-04-20T21:31:49.941 INFO:tasks.workunit.client.0.smithi062.stdout:6/70: creat d0/f17 x:0 0 0 2024-04-20T21:31:49.946 INFO:tasks.workunit.client.0.smithi062.stdout:8/114: dwrite d7/d9/d1b/f21 [0,4194304] 0 2024-04-20T21:31:49.946 INFO:tasks.workunit.client.0.smithi062.stdout:6/71: chown d0/d9/f13 219740596 1 2024-04-20T21:31:49.947 INFO:tasks.workunit.client.0.smithi062.stdout:8/115: dread - d7/d9/d1b/f1c zero size 2024-04-20T21:31:49.949 INFO:tasks.workunit.client.0.smithi062.stdout:3/130: dread d0/d11/f1a [0,4194304] 0 2024-04-20T21:31:49.963 INFO:tasks.workunit.client.0.smithi062.stdout:8/116: dread d7/d9/f12 [0,4194304] 0 2024-04-20T21:31:49.965 INFO:tasks.workunit.client.0.smithi062.stdout:8/117: fsync d7/fc 0 2024-04-20T21:31:49.970 INFO:tasks.workunit.client.0.smithi062.stdout:8/118: write d7/d9/f17 [2019520,60255] 0 2024-04-20T21:31:49.974 INFO:tasks.workunit.client.0.smithi062.stdout:3/131: dwrite d0/d11/f1a [4194304,4194304] 0 2024-04-20T21:31:49.976 INFO:tasks.workunit.client.0.smithi062.stdout:8/119: rename d7/c1a to d7/d25/c2d 0 2024-04-20T21:31:49.977 INFO:tasks.workunit.client.0.smithi062.stdout:3/132: creat d0/d1/f32 x:0 0 0 2024-04-20T21:31:49.979 INFO:tasks.workunit.client.0.smithi062.stdout:8/120: symlink d7/d9/l2e 0 2024-04-20T21:31:49.980 INFO:tasks.workunit.client.0.smithi062.stdout:8/121: fdatasync d7/f23 0 2024-04-20T21:31:49.980 INFO:tasks.workunit.client.0.smithi062.stdout:3/133: symlink d0/d22/l33 0 2024-04-20T21:31:49.984 INFO:tasks.workunit.client.0.smithi062.stdout:8/122: rename d7/d9/c16 to d7/d25/c2f 0 2024-04-20T21:31:49.987 INFO:tasks.workunit.client.0.smithi062.stdout:8/123: mkdir d7/d30 0 2024-04-20T21:31:49.996 INFO:tasks.workunit.client.0.smithi062.stdout:8/124: link d7/d9/c11 d7/d9/d1b/d24/c31 0 2024-04-20T21:31:50.000 INFO:tasks.workunit.client.0.smithi062.stdout:8/125: creat d7/d9/d29/f32 x:0 0 0 2024-04-20T21:31:50.001 INFO:tasks.workunit.client.0.smithi062.stdout:3/134: dwrite d0/f8 [4194304,4194304] 0 2024-04-20T21:31:50.003 INFO:tasks.workunit.client.0.smithi062.stdout:8/126: symlink d7/d25/l33 0 2024-04-20T21:31:50.004 INFO:tasks.workunit.client.0.smithi062.stdout:8/127: fsync d7/d9/d29/f2b 0 2024-04-20T21:31:50.005 INFO:tasks.workunit.client.0.smithi062.stdout:3/135: rename d0/d1/c12 to d0/d1/d10/d31/c34 0 2024-04-20T21:31:50.006 INFO:tasks.workunit.client.0.smithi062.stdout:8/128: dread - d7/d9/d1b/f1e zero size 2024-04-20T21:31:50.007 INFO:tasks.workunit.client.0.smithi062.stdout:3/136: creat d0/d1b/f35 x:0 0 0 2024-04-20T21:31:50.008 INFO:tasks.workunit.client.0.smithi062.stdout:3/137: dread - d0/d1/f32 zero size 2024-04-20T21:31:50.011 INFO:tasks.workunit.client.0.smithi062.stdout:3/138: chown d0/d22/l33 1529 1 2024-04-20T21:31:50.013 INFO:tasks.workunit.client.0.smithi062.stdout:3/139: chown d0/d1/fd 3 1 2024-04-20T21:31:50.017 INFO:tasks.workunit.client.0.smithi062.stdout:3/140: mknod d0/d1b/c36 0 2024-04-20T21:31:50.032 INFO:tasks.workunit.client.0.smithi062.stdout:8/129: dwrite d7/f22 [0,4194304] 0 2024-04-20T21:31:50.035 INFO:tasks.workunit.client.0.smithi062.stdout:8/130: mknod d7/d30/c34 0 2024-04-20T21:31:50.036 INFO:tasks.workunit.client.0.smithi062.stdout:3/141: dwrite d0/d1/d10/d2c/f30 [0,4194304] 0 2024-04-20T21:31:50.046 INFO:tasks.workunit.client.0.smithi062.stdout:8/131: link d7/d25/l33 d7/d9/d1b/l35 0 2024-04-20T21:31:50.048 INFO:tasks.workunit.client.0.smithi062.stdout:8/132: read - d7/d9/d1b/f1f zero size 2024-04-20T21:31:50.051 INFO:tasks.workunit.client.0.smithi062.stdout:8/133: write d7/d9/ff [1446988,90135] 0 2024-04-20T21:31:50.052 INFO:tasks.workunit.client.0.smithi062.stdout:8/134: chown d7/f22 9 1 2024-04-20T21:31:50.055 INFO:tasks.workunit.client.0.smithi062.stdout:3/142: truncate d0/d1/d10/f27 1403903 0 2024-04-20T21:31:50.058 INFO:tasks.workunit.client.0.smithi062.stdout:3/143: symlink d0/d1/d10/d31/l37 0 2024-04-20T21:31:50.063 INFO:tasks.workunit.client.0.smithi062.stdout:8/135: truncate d7/d9/f12 889307 0 2024-04-20T21:31:50.065 INFO:tasks.workunit.client.0.smithi062.stdout:8/136: read - d7/d9/d29/f2b zero size 2024-04-20T21:31:50.066 INFO:tasks.workunit.client.0.smithi062.stdout:8/137: chown d7/d9/f15 4211837 1 2024-04-20T21:31:50.067 INFO:tasks.workunit.client.0.smithi062.stdout:8/138: stat d7 0 2024-04-20T21:31:50.068 INFO:tasks.workunit.client.0.smithi062.stdout:3/144: dread d0/ff [0,4194304] 0 2024-04-20T21:31:50.071 INFO:tasks.workunit.client.0.smithi062.stdout:8/139: rename d7/d9/d1b/d24/l26 to d7/d25/l36 0 2024-04-20T21:31:50.071 INFO:tasks.workunit.client.0.smithi062.stdout:3/145: symlink d0/d1b/l38 0 2024-04-20T21:31:50.072 INFO:tasks.workunit.client.0.smithi062.stdout:8/140: dread - d7/d9/d1b/f1c zero size 2024-04-20T21:31:50.074 INFO:tasks.workunit.client.0.smithi062.stdout:3/146: mknod d0/c39 0 2024-04-20T21:31:50.074 INFO:tasks.workunit.client.0.smithi062.stdout:8/141: write d7/d9/f15 [662406,53254] 0 2024-04-20T21:31:50.077 INFO:tasks.workunit.client.0.smithi062.stdout:8/142: symlink d7/d25/l37 0 2024-04-20T21:31:50.084 INFO:tasks.workunit.client.0.smithi062.stdout:8/143: rmdir d7/d9/d1b 39 2024-04-20T21:31:50.089 INFO:tasks.workunit.client.0.smithi062.stdout:3/147: dwrite d0/f2f [0,4194304] 0 2024-04-20T21:31:50.090 INFO:tasks.workunit.client.0.smithi062.stdout:3/148: readlink d0/d22/l33 0 2024-04-20T21:31:50.094 INFO:tasks.workunit.client.0.smithi062.stdout:3/149: mkdir d0/d1/d10/d3a 0 2024-04-20T21:31:50.095 INFO:tasks.workunit.client.0.smithi062.stdout:8/144: stat d7/d9/f12 0 2024-04-20T21:31:50.100 INFO:tasks.workunit.client.0.smithi062.stdout:3/150: unlink d0/d22/f2d 0 2024-04-20T21:31:50.100 INFO:tasks.workunit.client.0.smithi062.stdout:8/145: creat d7/f38 x:0 0 0 2024-04-20T21:31:50.106 INFO:tasks.workunit.client.0.smithi062.stdout:8/146: unlink d7/d9/d1b/d24/f2a 0 2024-04-20T21:31:50.108 INFO:tasks.workunit.client.0.smithi062.stdout:3/151: link d0/d1/d10/d2c/l2e d0/d1/d10/d2c/l3b 0 2024-04-20T21:31:50.110 INFO:tasks.workunit.client.0.smithi062.stdout:3/152: chown d0/d1/d10/c1f 508979 1 2024-04-20T21:31:50.113 INFO:tasks.workunit.client.0.smithi062.stdout:3/153: mkdir d0/d1/d3c 0 2024-04-20T21:31:50.115 INFO:tasks.workunit.client.0.smithi062.stdout:3/154: read - d0/d1b/f35 zero size 2024-04-20T21:31:50.126 INFO:tasks.workunit.client.0.smithi062.stdout:8/147: dwrite d7/f38 [0,4194304] 0 2024-04-20T21:31:50.134 INFO:tasks.workunit.client.0.smithi062.stdout:3/155: dread d0/ff [0,4194304] 0 2024-04-20T21:31:50.134 INFO:tasks.workunit.client.0.smithi062.stdout:8/148: link d7/d9/c11 d7/d9/d1b/c39 0 2024-04-20T21:31:50.138 INFO:tasks.workunit.client.0.smithi062.stdout:3/156: symlink d0/d1/d10/d31/l3d 0 2024-04-20T21:31:50.141 INFO:tasks.workunit.client.0.smithi062.stdout:3/157: readlink d0/d1/d10/d2c/l2e 0 2024-04-20T21:31:50.145 INFO:tasks.workunit.client.0.smithi062.stdout:8/149: dread d7/d9/ff [0,4194304] 0 2024-04-20T21:31:50.145 INFO:tasks.workunit.client.0.smithi062.stdout:3/158: link d0/d1/f20 d0/d1/f3e 0 2024-04-20T21:31:50.146 INFO:tasks.workunit.client.0.smithi062.stdout:3/159: fsync d0/f8 0 2024-04-20T21:31:50.150 INFO:tasks.workunit.client.0.smithi062.stdout:3/160: creat d0/d1/d10/f3f x:0 0 0 2024-04-20T21:31:50.151 INFO:tasks.workunit.client.0.smithi062.stdout:8/150: creat d7/d9/f3a x:0 0 0 2024-04-20T21:31:50.153 INFO:tasks.workunit.client.0.smithi062.stdout:3/161: mknod d0/d1c/c40 0 2024-04-20T21:31:50.154 INFO:tasks.workunit.client.0.smithi062.stdout:3/162: fsync d0/d1/f32 0 2024-04-20T21:31:50.163 INFO:tasks.workunit.client.0.smithi062.stdout:8/151: dread d7/d9/d1b/d24/f20 [0,4194304] 0 2024-04-20T21:31:50.222 INFO:tasks.workunit.client.0.smithi062.stdout:8/152: mkdir d7/d25/d3b 0 2024-04-20T21:31:50.228 INFO:tasks.workunit.client.0.smithi062.stdout:8/153: readlink d7/d25/l36 0 2024-04-20T21:31:50.239 INFO:tasks.workunit.client.0.smithi062.stdout:8/154: creat d7/d30/f3c x:0 0 0 2024-04-20T21:31:50.246 INFO:tasks.workunit.client.0.smithi062.stdout:8/155: write d7/d9/d29/f32 [148116,80677] 0 2024-04-20T21:31:50.247 INFO:tasks.workunit.client.0.smithi062.stdout:3/163: dwrite d0/f2f [4194304,4194304] 0 2024-04-20T21:31:50.263 INFO:tasks.workunit.client.0.smithi062.stdout:3/164: truncate d0/d1/f20 607960 0 2024-04-20T21:31:50.277 INFO:tasks.workunit.client.0.smithi062.stdout:8/156: dwrite d7/f22 [0,4194304] 0 2024-04-20T21:31:50.281 INFO:tasks.workunit.client.0.smithi062.stdout:8/157: chown d7/d25/c2f 11647614 1 2024-04-20T21:31:50.288 INFO:tasks.workunit.client.0.smithi062.stdout:8/158: rename d7/d30/c34 to d7/c3d 0 2024-04-20T21:31:50.296 INFO:tasks.workunit.client.0.smithi062.stdout:8/159: write d7/d9/f12 [611568,90297] 0 2024-04-20T21:31:50.307 INFO:tasks.workunit.client.0.smithi062.stdout:8/160: getdents d7/d9/d1b/d24 0 2024-04-20T21:31:50.311 INFO:tasks.workunit.client.0.smithi062.stdout:3/165: dwrite d0/d1/f3e [0,4194304] 0 2024-04-20T21:31:50.316 INFO:tasks.workunit.client.0.smithi062.stdout:3/166: unlink d0/d1/f3e 0 2024-04-20T21:31:50.319 INFO:tasks.workunit.client.0.smithi062.stdout:8/161: dread d7/d9/d29/f32 [0,4194304] 0 2024-04-20T21:31:50.321 INFO:tasks.workunit.client.0.smithi062.stdout:3/167: write d0/d1b/f2b [4002666,4428] 0 2024-04-20T21:31:50.322 INFO:tasks.workunit.client.0.smithi062.stdout:8/162: truncate d7/d9/f13 582244 0 2024-04-20T21:31:50.323 INFO:tasks.workunit.client.0.smithi062.stdout:8/163: stat d7/d9/d1b/d24 0 2024-04-20T21:31:50.325 INFO:tasks.workunit.client.0.smithi062.stdout:8/164: chown d7/f22 216203 1 2024-04-20T21:31:50.326 INFO:tasks.workunit.client.0.smithi062.stdout:8/165: truncate d7/f23 692242 0 2024-04-20T21:31:50.328 INFO:tasks.workunit.client.0.smithi062.stdout:8/166: chown d7/c3d 506358 1 2024-04-20T21:31:50.330 INFO:tasks.workunit.client.0.smithi062.stdout:8/167: rmdir d7/d9/d1b 39 2024-04-20T21:31:50.334 INFO:tasks.workunit.client.0.smithi062.stdout:8/168: truncate d7/d9/d1b/f1e 643689 0 2024-04-20T21:31:50.363 INFO:tasks.workunit.client.0.smithi062.stdout:1/116: sync 2024-04-20T21:31:50.363 INFO:tasks.workunit.client.0.smithi062.stdout:2/83: sync 2024-04-20T21:31:50.365 INFO:tasks.workunit.client.0.smithi062.stdout:2/84: chown c2 785879211 1 2024-04-20T21:31:50.406 INFO:tasks.workunit.client.0.smithi062.stdout:1/117: dwrite fb [0,4194304] 0 2024-04-20T21:31:50.408 INFO:tasks.workunit.client.0.smithi062.stdout:2/85: dwrite d7/fb [0,4194304] 0 2024-04-20T21:31:50.413 INFO:tasks.workunit.client.0.smithi062.stdout:2/86: dread - d7/da/f13 zero size 2024-04-20T21:31:50.417 INFO:tasks.workunit.client.0.smithi062.stdout:1/118: link c7 dc/d15/d17/c25 0 2024-04-20T21:31:50.419 INFO:tasks.workunit.client.0.smithi062.stdout:1/119: write f5 [1167488,54188] 0 2024-04-20T21:31:50.421 INFO:tasks.workunit.client.0.smithi062.stdout:2/87: creat d7/f17 x:0 0 0 2024-04-20T21:31:50.426 INFO:tasks.workunit.client.0.smithi062.stdout:1/120: creat dc/f26 x:0 0 0 2024-04-20T21:31:50.435 INFO:tasks.workunit.client.0.smithi062.stdout:1/121: dread - f9 zero size 2024-04-20T21:31:50.451 INFO:tasks.workunit.client.0.smithi062.stdout:2/88: truncate f6 4015361 0 2024-04-20T21:31:50.508 INFO:tasks.workunit.client.0.smithi062.stdout:2/89: symlink d7/l18 0 2024-04-20T21:31:50.511 INFO:tasks.workunit.client.0.smithi062.stdout:1/122: mkdir dc/d27 0 2024-04-20T21:31:50.513 INFO:tasks.workunit.client.0.smithi062.stdout:4/101: sync 2024-04-20T21:31:50.513 INFO:tasks.workunit.client.0.smithi062.stdout:7/38: sync 2024-04-20T21:31:50.515 INFO:tasks.workunit.client.0.smithi062.stdout:2/90: chown d7/f17 342 1 2024-04-20T21:31:50.515 INFO:tasks.workunit.client.0.smithi062.stdout:7/39: dread - d7/fe zero size 2024-04-20T21:31:50.520 INFO:tasks.workunit.client.0.smithi062.stdout:4/102: creat da/dd/d15/f26 x:0 0 0 2024-04-20T21:31:50.521 INFO:tasks.workunit.client.0.smithi062.stdout:2/91: truncate d7/f17 398369 0 2024-04-20T21:31:50.522 INFO:tasks.workunit.client.0.smithi062.stdout:4/103: readlink da/dd/l18 0 2024-04-20T21:31:50.525 INFO:tasks.workunit.client.0.smithi062.stdout:2/92: read d7/ff [231512,123787] 0 2024-04-20T21:31:50.528 INFO:tasks.workunit.client.0.smithi062.stdout:4/104: mkdir da/dd/df/d27 0 2024-04-20T21:31:50.528 INFO:tasks.workunit.client.0.smithi062.stdout:2/93: dread - d7/da/f13 zero size 2024-04-20T21:31:50.530 INFO:tasks.workunit.client.0.smithi062.stdout:4/105: chown f4 123 1 2024-04-20T21:31:50.534 INFO:tasks.workunit.client.0.smithi062.stdout:4/106: write da/dd/f11 [3165801,115156] 0 2024-04-20T21:31:50.553 INFO:tasks.workunit.client.0.smithi062.stdout:7/40: truncate f4 2041160 0 2024-04-20T21:31:50.555 INFO:tasks.workunit.client.0.smithi062.stdout:7/41: fsync f6 0 2024-04-20T21:31:50.570 INFO:tasks.workunit.client.0.smithi062.stdout:4/107: dwrite da/dd/d15/f1e [0,4194304] 0 2024-04-20T21:31:50.572 INFO:tasks.workunit.client.0.smithi062.stdout:2/94: dwrite d7/da/f13 [0,4194304] 0 2024-04-20T21:31:50.582 INFO:tasks.workunit.client.0.smithi062.stdout:4/108: getdents da/dd/df/d27 0 2024-04-20T21:31:50.588 INFO:tasks.workunit.client.0.smithi062.stdout:4/109: creat da/dd/df/f28 x:0 0 0 2024-04-20T21:31:50.594 INFO:tasks.workunit.client.0.smithi062.stdout:7/42: dwrite f4 [0,4194304] 0 2024-04-20T21:31:50.594 INFO:tasks.workunit.client.0.smithi062.stdout:4/110: truncate da/dd/d15/f26 591434 0 2024-04-20T21:31:50.596 INFO:tasks.workunit.client.0.smithi062.stdout:7/43: fdatasync f6 0 2024-04-20T21:31:50.597 INFO:tasks.workunit.client.0.smithi062.stdout:7/44: dread - f6 zero size 2024-04-20T21:31:50.599 INFO:tasks.workunit.client.0.smithi062.stdout:7/45: chown d7/fe 16512070 1 2024-04-20T21:31:50.599 INFO:tasks.workunit.client.0.smithi062.stdout:4/111: symlink da/dd/df/d27/l29 0 2024-04-20T21:31:50.600 INFO:tasks.workunit.client.0.smithi062.stdout:0/139: sync 2024-04-20T21:31:50.601 INFO:tasks.workunit.client.0.smithi062.stdout:9/111: sync 2024-04-20T21:31:50.601 INFO:tasks.workunit.client.0.smithi062.stdout:0/140: chown d7/d1c 87 1 2024-04-20T21:31:50.602 INFO:tasks.workunit.client.0.smithi062.stdout:7/46: stat c2 0 2024-04-20T21:31:50.602 INFO:tasks.workunit.client.0.smithi062.stdout:4/112: write da/dd/f19 [11623,48114] 0 2024-04-20T21:31:50.603 INFO:tasks.workunit.client.0.smithi062.stdout:9/112: fsync d16/f19 0 2024-04-20T21:31:50.607 INFO:tasks.workunit.client.0.smithi062.stdout:9/113: mknod d16/c2a 0 2024-04-20T21:31:50.607 INFO:tasks.workunit.client.0.smithi062.stdout:7/47: creat d7/ff x:0 0 0 2024-04-20T21:31:50.608 INFO:tasks.workunit.client.0.smithi062.stdout:4/113: symlink da/dd/l2a 0 2024-04-20T21:31:50.608 INFO:tasks.workunit.client.0.smithi062.stdout:9/114: chown d16/f19 1 1 2024-04-20T21:31:50.610 INFO:tasks.workunit.client.0.smithi062.stdout:9/115: readlink d16/l29 0 2024-04-20T21:31:50.610 INFO:tasks.workunit.client.0.smithi062.stdout:4/114: stat da/f16 0 2024-04-20T21:31:50.613 INFO:tasks.workunit.client.0.smithi062.stdout:9/116: chown c11 11381552 1 2024-04-20T21:31:50.619 INFO:tasks.workunit.client.0.smithi062.stdout:0/141: dread d7/f9 [0,4194304] 0 2024-04-20T21:31:50.619 INFO:tasks.workunit.client.0.smithi062.stdout:9/117: rename c11 to d16/c2b 0 2024-04-20T21:31:50.623 INFO:tasks.workunit.client.0.smithi062.stdout:9/118: mkdir d16/d2c 0 2024-04-20T21:31:50.626 INFO:tasks.workunit.client.0.smithi062.stdout:0/142: dread d7/fb [0,4194304] 0 2024-04-20T21:31:50.629 INFO:tasks.workunit.client.0.smithi062.stdout:0/143: rename d7/d20/c28 to d7/d1e/c30 0 2024-04-20T21:31:50.632 INFO:tasks.workunit.client.0.smithi062.stdout:0/144: chown d7/d20/d29 350321 1 2024-04-20T21:31:50.636 INFO:tasks.workunit.client.0.smithi062.stdout:7/48: dwrite d7/fe [0,4194304] 0 2024-04-20T21:31:50.638 INFO:tasks.workunit.client.0.smithi062.stdout:0/145: mkdir d7/d31 0 2024-04-20T21:31:50.645 INFO:tasks.workunit.client.0.smithi062.stdout:0/146: chown d7/cd 12486 1 2024-04-20T21:31:50.650 INFO:tasks.workunit.client.0.smithi062.stdout:7/49: getdents d7 0 2024-04-20T21:31:50.653 INFO:tasks.workunit.client.0.smithi062.stdout:0/147: chown d7/d1e/f24 3829138 1 2024-04-20T21:31:50.655 INFO:tasks.workunit.client.0.smithi062.stdout:7/50: write f4 [2711191,62237] 0 2024-04-20T21:31:50.655 INFO:tasks.workunit.client.0.smithi062.stdout:9/119: dwrite d16/f1d [0,4194304] 0 2024-04-20T21:31:50.656 INFO:tasks.workunit.client.0.smithi062.stdout:4/115: dwrite da/dd/d15/f26 [0,4194304] 0 2024-04-20T21:31:50.658 INFO:tasks.workunit.client.0.smithi062.stdout:9/120: write d16/f1e [3748039,119454] 0 2024-04-20T21:31:50.665 INFO:tasks.workunit.client.0.smithi062.stdout:0/148: creat d7/d1c/f32 x:0 0 0 2024-04-20T21:31:50.667 INFO:tasks.workunit.client.0.smithi062.stdout:9/121: rename ca to d16/d2c/c2d 0 2024-04-20T21:31:50.670 INFO:tasks.workunit.client.0.smithi062.stdout:7/51: dread f5 [0,4194304] 0 2024-04-20T21:31:50.672 INFO:tasks.workunit.client.0.smithi062.stdout:9/122: creat d16/f2e x:0 0 0 2024-04-20T21:31:50.674 INFO:tasks.workunit.client.0.smithi062.stdout:5/63: sync 2024-04-20T21:31:50.674 INFO:tasks.workunit.client.0.smithi062.stdout:9/123: chown d16/l28 142 1 2024-04-20T21:31:50.676 INFO:tasks.workunit.client.0.smithi062.stdout:0/149: mknod d7/d1c/c33 0 2024-04-20T21:31:50.676 INFO:tasks.workunit.client.0.smithi062.stdout:9/124: readlink d16/l26 0 2024-04-20T21:31:50.680 INFO:tasks.workunit.client.0.smithi062.stdout:4/116: dwrite da/dd/df/f24 [0,4194304] 0 2024-04-20T21:31:50.681 INFO:tasks.workunit.client.0.smithi062.stdout:9/125: truncate d16/f1c 283903 0 2024-04-20T21:31:50.682 INFO:tasks.workunit.client.0.smithi062.stdout:9/126: truncate f4 985952 0 2024-04-20T21:31:50.689 INFO:tasks.workunit.client.0.smithi062.stdout:0/150: mknod d7/d1c/d15/d17/c34 0 2024-04-20T21:31:50.689 INFO:tasks.workunit.client.0.smithi062.stdout:7/52: link d7/fe d7/f10 0 2024-04-20T21:31:50.691 INFO:tasks.workunit.client.0.smithi062.stdout:5/64: fsync f4 0 2024-04-20T21:31:50.692 INFO:tasks.workunit.client.0.smithi062.stdout:4/117: write da/dd/f11 [4019349,56830] 0 2024-04-20T21:31:50.695 INFO:tasks.workunit.client.0.smithi062.stdout:4/118: dread - da/dd/df/f10 zero size 2024-04-20T21:31:50.698 INFO:tasks.workunit.client.0.smithi062.stdout:9/127: truncate f6 1594297 0 2024-04-20T21:31:50.700 INFO:tasks.workunit.client.0.smithi062.stdout:7/53: symlink d7/l11 0 2024-04-20T21:31:50.701 INFO:tasks.workunit.client.0.smithi062.stdout:9/128: write d16/f1e [5165817,93582] 0 2024-04-20T21:31:50.706 INFO:tasks.workunit.client.0.smithi062.stdout:5/65: rename d8/f19 to d8/f1c 0 2024-04-20T21:31:50.711 INFO:tasks.workunit.client.0.smithi062.stdout:9/129: mkdir d16/d2f 0 2024-04-20T21:31:50.714 INFO:tasks.workunit.client.0.smithi062.stdout:7/54: link f5 d7/f12 0 2024-04-20T21:31:50.714 INFO:tasks.workunit.client.0.smithi062.stdout:4/119: fsync da/dd/d15/f26 0 2024-04-20T21:31:50.717 INFO:tasks.workunit.client.0.smithi062.stdout:9/130: symlink d16/d2c/l30 0 2024-04-20T21:31:50.718 INFO:tasks.workunit.client.0.smithi062.stdout:0/151: link d7/d1e/l2c d7/d1c/l35 0 2024-04-20T21:31:50.720 INFO:tasks.workunit.client.0.smithi062.stdout:9/131: write d16/f1e [2515846,129606] 0 2024-04-20T21:31:50.721 INFO:tasks.workunit.client.0.smithi062.stdout:4/120: write da/dd/f14 [108224,121077] 0 2024-04-20T21:31:50.722 INFO:tasks.workunit.client.0.smithi062.stdout:2/95: sync 2024-04-20T21:31:50.723 INFO:tasks.workunit.client.0.smithi062.stdout:8/169: sync 2024-04-20T21:31:50.724 INFO:tasks.workunit.client.0.smithi062.stdout:1/123: sync 2024-04-20T21:31:50.724 INFO:tasks.workunit.client.0.smithi062.stdout:1/124: readlink - no filename 2024-04-20T21:31:50.724 INFO:tasks.workunit.client.0.smithi062.stdout:6/72: sync 2024-04-20T21:31:50.724 INFO:tasks.workunit.client.0.smithi062.stdout:3/168: sync 2024-04-20T21:31:50.724 INFO:tasks.workunit.client.0.smithi062.stdout:0/152: rmdir d7 39 2024-04-20T21:31:50.725 INFO:tasks.workunit.client.0.smithi062.stdout:5/66: link c2 d8/c1d 0 2024-04-20T21:31:50.726 INFO:tasks.workunit.client.0.smithi062.stdout:4/121: write da/dd/f14 [408122,49163] 0 2024-04-20T21:31:50.733 INFO:tasks.workunit.client.0.smithi062.stdout:6/73: readlink d0/ld 0 2024-04-20T21:31:50.735 INFO:tasks.workunit.client.0.smithi062.stdout:9/132: creat d16/d2f/f31 x:0 0 0 2024-04-20T21:31:50.738 INFO:tasks.workunit.client.0.smithi062.stdout:8/170: creat d7/d9/d29/f3e x:0 0 0 2024-04-20T21:31:50.741 INFO:tasks.workunit.client.0.smithi062.stdout:6/74: truncate d0/f17 555357 0 2024-04-20T21:31:50.745 INFO:tasks.workunit.client.0.smithi062.stdout:7/55: truncate f4 432730 0 2024-04-20T21:31:50.747 INFO:tasks.workunit.client.0.smithi062.stdout:3/169: dread d0/d1/d10/f27 [0,4194304] 0 2024-04-20T21:31:50.750 INFO:tasks.workunit.client.0.smithi062.stdout:9/133: link f4 d16/f32 0 2024-04-20T21:31:50.751 INFO:tasks.workunit.client.0.smithi062.stdout:8/171: truncate d7/d9/f15 1074032 0 2024-04-20T21:31:50.752 INFO:tasks.workunit.client.0.smithi062.stdout:6/75: rmdir d0/d9 39 2024-04-20T21:31:50.762 INFO:tasks.workunit.client.0.smithi062.stdout:5/67: dwrite d8/fe [0,4194304] 0 2024-04-20T21:31:50.762 INFO:tasks.workunit.client.0.smithi062.stdout:0/153: dread d7/d1c/f14 [0,4194304] 0 2024-04-20T21:31:50.763 INFO:tasks.workunit.client.0.smithi062.stdout:8/172: creat d7/d9/d1b/f3f x:0 0 0 2024-04-20T21:31:50.764 INFO:tasks.workunit.client.0.smithi062.stdout:9/134: truncate f13 855979 0 2024-04-20T21:31:50.765 INFO:tasks.workunit.client.0.smithi062.stdout:2/96: dwrite d7/fb [0,4194304] 0 2024-04-20T21:31:50.768 INFO:tasks.workunit.client.0.smithi062.stdout:6/76: fdatasync d0/d9/f13 0 2024-04-20T21:31:50.768 INFO:tasks.workunit.client.0.smithi062.stdout:1/125: dwrite dc/ff [0,4194304] 0 2024-04-20T21:31:50.771 INFO:tasks.workunit.client.0.smithi062.stdout:9/135: fsync f6 0 2024-04-20T21:31:50.772 INFO:tasks.workunit.client.0.smithi062.stdout:8/173: chown d7/d9/f18 18 1 2024-04-20T21:31:50.784 INFO:tasks.workunit.client.0.smithi062.stdout:2/97: mknod d7/da/c19 0 2024-04-20T21:31:50.784 INFO:tasks.workunit.client.0.smithi062.stdout:6/77: dread - d0/d9/f15 zero size 2024-04-20T21:31:50.786 INFO:tasks.workunit.client.0.smithi062.stdout:1/126: write dc/ff [4792831,109115] 0 2024-04-20T21:31:50.787 INFO:tasks.workunit.client.0.smithi062.stdout:9/136: truncate d16/f1c 403757 0 2024-04-20T21:31:50.787 INFO:tasks.workunit.client.0.smithi062.stdout:8/174: fsync d7/d9/d29/f32 0 2024-04-20T21:31:50.788 INFO:tasks.workunit.client.0.smithi062.stdout:1/127: fsync fb 0 2024-04-20T21:31:50.791 INFO:tasks.workunit.client.0.smithi062.stdout:2/98: chown d7/da 0 1 2024-04-20T21:31:50.802 INFO:tasks.workunit.client.0.smithi062.stdout:7/56: dread d7/f12 [0,4194304] 0 2024-04-20T21:31:50.802 INFO:tasks.workunit.client.0.smithi062.stdout:1/128: write f5 [572290,92185] 0 2024-04-20T21:31:50.803 INFO:tasks.workunit.client.0.smithi062.stdout:8/175: truncate d7/d9/f3a 785008 0 2024-04-20T21:31:50.804 INFO:tasks.workunit.client.0.smithi062.stdout:0/154: dread d7/d1c/d15/d17/f1d [0,4194304] 0 2024-04-20T21:31:50.812 INFO:tasks.workunit.client.0.smithi062.stdout:7/57: mknod d7/c13 0 2024-04-20T21:31:50.813 INFO:tasks.workunit.client.0.smithi062.stdout:3/170: dwrite d0/d1b/f35 [0,4194304] 0 2024-04-20T21:31:50.813 INFO:tasks.workunit.client.0.smithi062.stdout:2/99: creat d7/f1a x:0 0 0 2024-04-20T21:31:50.815 INFO:tasks.workunit.client.0.smithi062.stdout:7/58: chown c2 377113780 1 2024-04-20T21:31:50.815 INFO:tasks.workunit.client.0.smithi062.stdout:2/100: chown d7/da 1095 1 2024-04-20T21:31:50.816 INFO:tasks.workunit.client.0.smithi062.stdout:8/176: rename d7/fc to d7/d9/d29/f40 0 2024-04-20T21:31:50.819 INFO:tasks.workunit.client.0.smithi062.stdout:6/78: truncate d0/f17 260967 0 2024-04-20T21:31:50.820 INFO:tasks.workunit.client.0.smithi062.stdout:3/171: fdatasync d0/d1/f29 0 2024-04-20T21:31:50.820 INFO:tasks.workunit.client.0.smithi062.stdout:7/59: creat d7/f14 x:0 0 0 2024-04-20T21:31:50.820 INFO:tasks.workunit.client.0.smithi062.stdout:2/101: read d7/ff [117134,2368] 0 2024-04-20T21:31:50.821 INFO:tasks.workunit.client.0.smithi062.stdout:5/68: dwrite f4 [0,4194304] 0 2024-04-20T21:31:50.822 INFO:tasks.workunit.client.0.smithi062.stdout:1/129: mkdir dc/d15/d20/d28 0 2024-04-20T21:31:50.822 INFO:tasks.workunit.client.0.smithi062.stdout:7/60: stat d7/l11 0 2024-04-20T21:31:50.824 INFO:tasks.workunit.client.0.smithi062.stdout:7/61: read d7/f12 [1518778,101010] 0 2024-04-20T21:31:50.824 INFO:tasks.workunit.client.0.smithi062.stdout:6/79: creat d0/d9/f18 x:0 0 0 2024-04-20T21:31:50.825 INFO:tasks.workunit.client.0.smithi062.stdout:1/130: chown f1 22141501 1 2024-04-20T21:31:50.828 INFO:tasks.workunit.client.0.smithi062.stdout:8/177: creat d7/d9/f41 x:0 0 0 2024-04-20T21:31:50.831 INFO:tasks.workunit.client.0.smithi062.stdout:5/69: dread d8/fb [0,4194304] 0 2024-04-20T21:31:50.833 INFO:tasks.workunit.client.0.smithi062.stdout:3/172: dread d0/d1/d10/f27 [0,4194304] 0 2024-04-20T21:31:50.840 INFO:tasks.workunit.client.0.smithi062.stdout:5/70: mkdir d8/d1e 0 2024-04-20T21:31:50.842 INFO:tasks.workunit.client.0.smithi062.stdout:3/173: mkdir d0/d1/d10/d31/d41 0 2024-04-20T21:31:50.844 INFO:tasks.workunit.client.0.smithi062.stdout:7/62: write d7/fe [1165053,113301] 0 2024-04-20T21:31:50.847 INFO:tasks.workunit.client.0.smithi062.stdout:5/71: chown d8/d10/c14 66859 1 2024-04-20T21:31:50.847 INFO:tasks.workunit.client.0.smithi062.stdout:6/80: truncate d0/f6 805776 0 2024-04-20T21:31:50.848 INFO:tasks.workunit.client.0.smithi062.stdout:3/174: chown d0/d11/l15 23938221 1 2024-04-20T21:31:50.851 INFO:tasks.workunit.client.0.smithi062.stdout:0/155: dwrite d7/d1c/d15/d17/d1b/f1a [0,4194304] 0 2024-04-20T21:31:50.851 INFO:tasks.workunit.client.0.smithi062.stdout:1/131: link dc/d15/d17/f23 dc/d27/f29 0 2024-04-20T21:31:50.851 INFO:tasks.workunit.client.0.smithi062.stdout:2/102: dread f6 [0,4194304] 0 2024-04-20T21:31:50.859 INFO:tasks.workunit.client.0.smithi062.stdout:1/132: fdatasync dc/fe 0 2024-04-20T21:31:50.867 INFO:tasks.workunit.client.0.smithi062.stdout:0/156: rmdir d7/d1c/d15 39 2024-04-20T21:31:50.868 INFO:tasks.workunit.client.0.smithi062.stdout:5/72: symlink d8/d1e/l1f 0 2024-04-20T21:31:50.869 INFO:tasks.workunit.client.0.smithi062.stdout:7/63: rename d7/ld to d7/l15 0 2024-04-20T21:31:50.869 INFO:tasks.workunit.client.0.smithi062.stdout:1/133: rename dc/d15/d17/c1d to dc/d15/c2a 0 2024-04-20T21:31:50.870 INFO:tasks.workunit.client.0.smithi062.stdout:8/178: dwrite d7/f23 [0,4194304] 0 2024-04-20T21:31:50.872 INFO:tasks.workunit.client.0.smithi062.stdout:9/137: dwrite f13 [0,4194304] 0 2024-04-20T21:31:50.877 INFO:tasks.workunit.client.0.smithi062.stdout:8/179: stat d7/d9/f41 0 2024-04-20T21:31:50.878 INFO:tasks.workunit.client.0.smithi062.stdout:1/134: write dc/f26 [682197,128301] 0 2024-04-20T21:31:50.880 INFO:tasks.workunit.client.0.smithi062.stdout:0/157: mknod d7/d1e/c36 0 2024-04-20T21:31:50.885 INFO:tasks.workunit.client.0.smithi062.stdout:2/103: getdents d7 0 2024-04-20T21:31:50.886 INFO:tasks.workunit.client.0.smithi062.stdout:7/64: symlink d7/l16 0 2024-04-20T21:31:50.887 INFO:tasks.workunit.client.0.smithi062.stdout:8/180: chown d7/d9/d29/f3e 54788 1 2024-04-20T21:31:50.887 INFO:tasks.workunit.client.0.smithi062.stdout:5/73: mknod d8/c20 0 2024-04-20T21:31:50.891 INFO:tasks.workunit.client.0.smithi062.stdout:2/104: write d7/ff [92670,106582] 0 2024-04-20T21:31:50.893 INFO:tasks.workunit.client.0.smithi062.stdout:7/65: symlink d7/l17 0 2024-04-20T21:31:50.898 INFO:tasks.workunit.client.0.smithi062.stdout:3/175: dwrite d0/d1/fb [0,4194304] 0 2024-04-20T21:31:50.898 INFO:tasks.workunit.client.0.smithi062.stdout:8/181: creat d7/d25/f42 x:0 0 0 2024-04-20T21:31:50.899 INFO:tasks.workunit.client.0.smithi062.stdout:0/158: mkdir d7/d1c/d15/d17/d37 0 2024-04-20T21:31:50.903 INFO:tasks.workunit.client.0.smithi062.stdout:0/159: stat d7/ca 0 2024-04-20T21:31:50.906 INFO:tasks.workunit.client.0.smithi062.stdout:6/81: dwrite d0/fe [4194304,4194304] 0 2024-04-20T21:31:50.909 INFO:tasks.workunit.client.0.smithi062.stdout:3/176: truncate d0/d1/d10/f3f 412174 0 2024-04-20T21:31:50.911 INFO:tasks.workunit.client.0.smithi062.stdout:1/135: getdents dc/d15/d17 0 2024-04-20T21:31:50.911 INFO:tasks.workunit.client.0.smithi062.stdout:8/182: chown d7/d25/l33 153529206 1 2024-04-20T21:31:50.911 INFO:tasks.workunit.client.0.smithi062.stdout:7/66: truncate f4 662349 0 2024-04-20T21:31:50.917 INFO:tasks.workunit.client.0.smithi062.stdout:6/82: mknod d0/d9/d11/c19 0 2024-04-20T21:31:50.920 INFO:tasks.workunit.client.0.smithi062.stdout:9/138: dwrite d16/f1c [0,4194304] 0 2024-04-20T21:31:50.920 INFO:tasks.workunit.client.0.smithi062.stdout:2/105: write f6 [20335,127978] 0 2024-04-20T21:31:50.924 INFO:tasks.workunit.client.0.smithi062.stdout:7/67: rmdir d7 39 2024-04-20T21:31:50.926 INFO:tasks.workunit.client.0.smithi062.stdout:8/183: mknod d7/c43 0 2024-04-20T21:31:50.926 INFO:tasks.workunit.client.0.smithi062.stdout:7/68: fdatasync f6 0 2024-04-20T21:31:50.926 INFO:tasks.workunit.client.0.smithi062.stdout:9/139: dread - d16/d2f/f31 zero size 2024-04-20T21:31:50.927 INFO:tasks.workunit.client.0.smithi062.stdout:4/122: sync 2024-04-20T21:31:50.931 INFO:tasks.workunit.client.0.smithi062.stdout:3/177: mkdir d0/d1/d10/d2c/d42 0 2024-04-20T21:31:50.931 INFO:tasks.workunit.client.0.smithi062.stdout:6/83: mkdir d0/d1a 0 2024-04-20T21:31:50.934 INFO:tasks.workunit.client.0.smithi062.stdout:3/178: readlink d0/d1/l5 0 2024-04-20T21:31:50.934 INFO:tasks.workunit.client.0.smithi062.stdout:1/136: creat dc/f2b x:0 0 0 2024-04-20T21:31:50.937 INFO:tasks.workunit.client.0.smithi062.stdout:2/106: mkdir d7/da/d15/d1b 0 2024-04-20T21:31:50.937 INFO:tasks.workunit.client.0.smithi062.stdout:4/123: truncate da/dd/d15/f1d 390686 0 2024-04-20T21:31:50.943 INFO:tasks.workunit.client.0.smithi062.stdout:7/69: creat d7/f18 x:0 0 0 2024-04-20T21:31:50.945 INFO:tasks.workunit.client.0.smithi062.stdout:5/74: dwrite d8/d10/f1b [0,4194304] 0 2024-04-20T21:31:50.950 INFO:tasks.workunit.client.0.smithi062.stdout:3/179: mkdir d0/d1/d3c/d43 0 2024-04-20T21:31:50.953 INFO:tasks.workunit.client.0.smithi062.stdout:5/75: mknod d8/d10/c21 0 2024-04-20T21:31:50.953 INFO:tasks.workunit.client.0.smithi062.stdout:7/70: write f5 [3009384,86604] 0 2024-04-20T21:31:50.955 INFO:tasks.workunit.client.0.smithi062.stdout:4/124: write da/dd/df/f1a [638233,101010] 0 2024-04-20T21:31:50.965 INFO:tasks.workunit.client.0.smithi062.stdout:7/71: fsync f4 0 2024-04-20T21:31:50.967 INFO:tasks.workunit.client.0.smithi062.stdout:2/107: dread d7/fb [0,4194304] 0 2024-04-20T21:31:50.977 INFO:tasks.workunit.client.0.smithi062.stdout:5/76: rename c2 to d8/c22 0 2024-04-20T21:31:50.982 INFO:tasks.workunit.client.0.smithi062.stdout:8/184: dwrite d7/d9/d1b/f3f [0,4194304] 0 2024-04-20T21:31:50.983 INFO:tasks.workunit.client.0.smithi062.stdout:3/180: write d0/d1/d10/f27 [808788,103250] 0 2024-04-20T21:31:50.984 INFO:tasks.workunit.client.0.smithi062.stdout:7/72: chown d7/cb 259 1 2024-04-20T21:31:50.986 INFO:tasks.workunit.client.0.smithi062.stdout:5/77: write f4 [5039835,87618] 0 2024-04-20T21:31:50.988 INFO:tasks.workunit.client.0.smithi062.stdout:6/84: dwrite d0/f10 [0,4194304] 0 2024-04-20T21:31:50.988 INFO:tasks.workunit.client.0.smithi062.stdout:7/73: fsync f5 0 2024-04-20T21:31:50.993 INFO:tasks.workunit.client.0.smithi062.stdout:5/78: fdatasync d8/ff 0 2024-04-20T21:31:50.994 INFO:tasks.workunit.client.0.smithi062.stdout:1/137: dread f8 [0,4194304] 0 2024-04-20T21:31:50.994 INFO:tasks.workunit.client.0.smithi062.stdout:4/125: symlink da/dd/l2b 0 2024-04-20T21:31:50.998 INFO:tasks.workunit.client.0.smithi062.stdout:6/85: mknod d0/d1a/c1b 0 2024-04-20T21:31:51.002 INFO:tasks.workunit.client.0.smithi062.stdout:7/74: mkdir d7/d19 0 2024-04-20T21:31:51.003 INFO:tasks.workunit.client.0.smithi062.stdout:3/181: mkdir d0/d1/d10/d2c/d42/d44 0 2024-04-20T21:31:51.004 INFO:tasks.workunit.client.0.smithi062.stdout:9/140: dwrite d16/f32 [0,4194304] 0 2024-04-20T21:31:51.006 INFO:tasks.workunit.client.0.smithi062.stdout:4/126: fdatasync da/f13 0 2024-04-20T21:31:51.008 INFO:tasks.workunit.client.0.smithi062.stdout:1/138: unlink dc/f2b 0 2024-04-20T21:31:51.016 INFO:tasks.workunit.client.0.smithi062.stdout:9/141: rmdir d16/d2c 39 2024-04-20T21:31:51.018 INFO:tasks.workunit.client.0.smithi062.stdout:3/182: creat d0/d1c/f45 x:0 0 0 2024-04-20T21:31:51.019 INFO:tasks.workunit.client.0.smithi062.stdout:9/142: write f6 [5058117,75985] 0 2024-04-20T21:31:51.025 INFO:tasks.workunit.client.0.smithi062.stdout:7/75: link d7/f10 d7/d19/f1a 0 2024-04-20T21:31:51.030 INFO:tasks.workunit.client.0.smithi062.stdout:4/127: creat da/dd/df/d27/f2c x:0 0 0 2024-04-20T21:31:51.031 INFO:tasks.workunit.client.0.smithi062.stdout:2/108: dwrite d7/f12 [0,4194304] 0 2024-04-20T21:31:51.033 INFO:tasks.workunit.client.0.smithi062.stdout:6/86: link d0/ca d0/c1c 0 2024-04-20T21:31:51.035 INFO:tasks.workunit.client.0.smithi062.stdout:3/183: write d0/d1/f19 [389428,76402] 0 2024-04-20T21:31:51.041 INFO:tasks.workunit.client.0.smithi062.stdout:7/76: dread f4 [0,4194304] 0 2024-04-20T21:31:51.043 INFO:tasks.workunit.client.0.smithi062.stdout:5/79: dread d8/f1a [0,4194304] 0 2024-04-20T21:31:51.043 INFO:tasks.workunit.client.0.smithi062.stdout:3/184: unlink d0/ff 0 2024-04-20T21:31:51.047 INFO:tasks.workunit.client.0.smithi062.stdout:7/77: read f4 [353274,26281] 0 2024-04-20T21:31:51.049 INFO:tasks.workunit.client.0.smithi062.stdout:2/109: truncate d7/f17 480532 0 2024-04-20T21:31:51.050 INFO:tasks.workunit.client.0.smithi062.stdout:8/185: dwrite d7/d9/d1b/f1e [0,4194304] 0 2024-04-20T21:31:51.055 INFO:tasks.workunit.client.0.smithi062.stdout:6/87: link d0/d9/d11/c14 d0/d1a/c1d 0 2024-04-20T21:31:51.057 INFO:tasks.workunit.client.0.smithi062.stdout:7/78: chown d7/c8 3992 1 2024-04-20T21:31:51.058 INFO:tasks.workunit.client.0.smithi062.stdout:2/110: read d7/f12 [3838481,51322] 0 2024-04-20T21:31:51.061 INFO:tasks.workunit.client.0.smithi062.stdout:5/80: chown d8/l9 46368 1 2024-04-20T21:31:51.066 INFO:tasks.workunit.client.0.smithi062.stdout:7/79: symlink d7/l1b 0 2024-04-20T21:31:51.070 INFO:tasks.workunit.client.0.smithi062.stdout:8/186: fsync d7/d9/f17 0 2024-04-20T21:31:51.071 INFO:tasks.workunit.client.0.smithi062.stdout:1/139: dwrite dc/d15/f16 [0,4194304] 0 2024-04-20T21:31:51.071 INFO:tasks.workunit.client.0.smithi062.stdout:2/111: write d7/f1a [201212,113325] 0 2024-04-20T21:31:51.076 INFO:tasks.workunit.client.0.smithi062.stdout:7/80: write d7/ff [50192,125847] 0 2024-04-20T21:31:51.078 INFO:tasks.workunit.client.0.smithi062.stdout:5/81: creat d8/d1e/f23 x:0 0 0 2024-04-20T21:31:51.078 INFO:tasks.workunit.client.0.smithi062.stdout:2/112: chown l4 47121 1 2024-04-20T21:31:51.082 INFO:tasks.workunit.client.0.smithi062.stdout:9/143: dwrite d16/f32 [0,4194304] 0 2024-04-20T21:31:51.083 INFO:tasks.workunit.client.0.smithi062.stdout:8/187: write d7/d9/f18 [3630349,108050] 0 2024-04-20T21:31:51.087 INFO:tasks.workunit.client.0.smithi062.stdout:2/113: mknod d7/c1c 0 2024-04-20T21:31:51.090 INFO:tasks.workunit.client.0.smithi062.stdout:1/140: truncate dc/d15/d17/f23 1020797 0 2024-04-20T21:31:51.090 INFO:tasks.workunit.client.0.smithi062.stdout:9/144: dread - d16/f2e zero size 2024-04-20T21:31:51.092 INFO:tasks.workunit.client.0.smithi062.stdout:9/145: fdatasync d16/f1d 0 2024-04-20T21:31:51.093 INFO:tasks.workunit.client.0.smithi062.stdout:5/82: truncate d8/d10/f1b 4478156 0 2024-04-20T21:31:51.094 INFO:tasks.workunit.client.0.smithi062.stdout:3/185: dread d0/f8 [4194304,4194304] 0 2024-04-20T21:31:51.102 INFO:tasks.workunit.client.0.smithi062.stdout:0/160: sync 2024-04-20T21:31:51.104 INFO:tasks.workunit.client.0.smithi062.stdout:8/188: symlink d7/d25/l44 0 2024-04-20T21:31:51.104 INFO:tasks.workunit.client.0.smithi062.stdout:1/141: mknod dc/c2c 0 2024-04-20T21:31:51.105 INFO:tasks.workunit.client.0.smithi062.stdout:5/83: readlink l6 0 2024-04-20T21:31:51.109 INFO:tasks.workunit.client.0.smithi062.stdout:6/88: dwrite d0/d9/f18 [0,4194304] 0 2024-04-20T21:31:51.111 INFO:tasks.workunit.client.0.smithi062.stdout:8/189: chown d7/d25/d3b 37273 1 2024-04-20T21:31:51.111 INFO:tasks.workunit.client.0.smithi062.stdout:5/84: chown d8/d10/f17 4 1 2024-04-20T21:31:51.111 INFO:tasks.workunit.client.0.smithi062.stdout:9/146: symlink d16/d2c/l33 0 2024-04-20T21:31:51.114 INFO:tasks.workunit.client.0.smithi062.stdout:1/142: mkdir dc/d2d 0 2024-04-20T21:31:51.115 INFO:tasks.workunit.client.0.smithi062.stdout:5/85: write f4 [2648967,3718] 0 2024-04-20T21:31:51.117 INFO:tasks.workunit.client.0.smithi062.stdout:8/190: read d7/d9/d1b/f21 [3507587,5699] 0 2024-04-20T21:31:51.119 INFO:tasks.workunit.client.0.smithi062.stdout:6/89: truncate d0/d9/f13 484213 0 2024-04-20T21:31:51.120 INFO:tasks.workunit.client.0.smithi062.stdout:8/191: fdatasync d7/f23 0 2024-04-20T21:31:51.122 INFO:tasks.workunit.client.0.smithi062.stdout:0/161: creat d7/d1c/d15/d17/f38 x:0 0 0 2024-04-20T21:31:51.125 INFO:tasks.workunit.client.0.smithi062.stdout:5/86: creat d8/d1e/f24 x:0 0 0 2024-04-20T21:31:51.127 INFO:tasks.workunit.client.0.smithi062.stdout:1/143: read dc/f26 [667243,49567] 0 2024-04-20T21:31:51.130 INFO:tasks.workunit.client.0.smithi062.stdout:6/90: unlink d0/d1a/c1b 0 2024-04-20T21:31:51.136 INFO:tasks.workunit.client.0.smithi062.stdout:1/144: mknod dc/d15/d20/c2e 0 2024-04-20T21:31:51.137 INFO:tasks.workunit.client.0.smithi062.stdout:6/91: symlink d0/d9/d11/l1e 0 2024-04-20T21:31:51.138 INFO:tasks.workunit.client.0.smithi062.stdout:4/128: dread da/f1b [0,4194304] 0 2024-04-20T21:31:51.138 INFO:tasks.workunit.client.0.smithi062.stdout:2/114: unlink d7/fb 0 2024-04-20T21:31:51.138 INFO:tasks.workunit.client.0.smithi062.stdout:3/186: dread d0/d1/fb [0,4194304] 0 2024-04-20T21:31:51.140 INFO:tasks.workunit.client.0.smithi062.stdout:0/162: creat d7/d1c/f39 x:0 0 0 2024-04-20T21:31:51.144 INFO:tasks.workunit.client.0.smithi062.stdout:3/187: fdatasync d0/d1b/f35 0 2024-04-20T21:31:51.145 INFO:tasks.workunit.client.0.smithi062.stdout:9/147: truncate d16/f1d 3736892 0 2024-04-20T21:31:51.147 INFO:tasks.workunit.client.0.smithi062.stdout:0/163: write f1 [975760,19913] 0 2024-04-20T21:31:51.151 INFO:tasks.workunit.client.0.smithi062.stdout:1/145: creat dc/d15/d20/d28/f2f x:0 0 0 2024-04-20T21:31:51.154 INFO:tasks.workunit.client.0.smithi062.stdout:4/129: stat da/dd/f14 0 2024-04-20T21:31:51.158 INFO:tasks.workunit.client.0.smithi062.stdout:0/164: mkdir d7/d20/d3a 0 2024-04-20T21:31:51.160 INFO:tasks.workunit.client.0.smithi062.stdout:0/165: stat f1 0 2024-04-20T21:31:51.164 INFO:tasks.workunit.client.0.smithi062.stdout:3/188: rename d0/c25 to d0/d1/d3c/d43/c46 0 2024-04-20T21:31:51.167 INFO:tasks.workunit.client.0.smithi062.stdout:1/146: chown c7 150471 1 2024-04-20T21:31:51.171 INFO:tasks.workunit.client.0.smithi062.stdout:5/87: dwrite d8/d1e/f24 [0,4194304] 0 2024-04-20T21:31:51.177 INFO:tasks.workunit.client.0.smithi062.stdout:8/192: dwrite d7/d9/d1b/f1f [0,4194304] 0 2024-04-20T21:31:51.177 INFO:tasks.workunit.client.0.smithi062.stdout:4/130: rename da/f1b to da/dd/df/d27/f2d 0 2024-04-20T21:31:51.177 INFO:tasks.workunit.client.0.smithi062.stdout:0/166: write d7/d1c/d15/f22 [877771,96393] 0 2024-04-20T21:31:51.187 INFO:tasks.workunit.client.0.smithi062.stdout:8/193: chown d7/d9/d1b/f1e 12046315 1 2024-04-20T21:31:51.189 INFO:tasks.workunit.client.0.smithi062.stdout:9/148: truncate fb 3871916 0 2024-04-20T21:31:51.190 INFO:tasks.workunit.client.0.smithi062.stdout:6/92: dwrite d0/d9/f13 [0,4194304] 0 2024-04-20T21:31:51.192 INFO:tasks.workunit.client.0.smithi062.stdout:9/149: chown d16/l29 28 1 2024-04-20T21:31:51.193 INFO:tasks.workunit.client.0.smithi062.stdout:2/115: dwrite d7/ff [0,4194304] 0 2024-04-20T21:31:51.193 INFO:tasks.workunit.client.0.smithi062.stdout:5/88: creat d8/d1e/f25 x:0 0 0 2024-04-20T21:31:51.198 INFO:tasks.workunit.client.0.smithi062.stdout:0/167: fsync d7/d1c/d15/d17/f1d 0 2024-04-20T21:31:51.201 INFO:tasks.workunit.client.0.smithi062.stdout:4/131: stat da/dd/d15/f1d 0 2024-04-20T21:31:51.204 INFO:tasks.workunit.client.0.smithi062.stdout:9/150: rename d16/l28 to d16/d2c/l34 0 2024-04-20T21:31:51.206 INFO:tasks.workunit.client.0.smithi062.stdout:4/132: mknod da/c2e 0 2024-04-20T21:31:51.210 INFO:tasks.workunit.client.0.smithi062.stdout:0/168: symlink d7/d1c/l3b 0 2024-04-20T21:31:51.212 INFO:tasks.workunit.client.0.smithi062.stdout:6/93: link d0/d9/f15 d0/d9/d11/f1f 0 2024-04-20T21:31:51.216 INFO:tasks.workunit.client.0.smithi062.stdout:5/89: dread d8/ff [0,4194304] 0 2024-04-20T21:31:51.216 INFO:tasks.workunit.client.0.smithi062.stdout:4/133: creat da/f2f x:0 0 0 2024-04-20T21:31:51.220 INFO:tasks.workunit.client.0.smithi062.stdout:7/81: sync 2024-04-20T21:31:51.228 INFO:tasks.workunit.client.0.smithi062.stdout:3/189: dwrite d0/d1b/f35 [0,4194304] 0 2024-04-20T21:31:51.228 INFO:tasks.workunit.client.0.smithi062.stdout:6/94: write d0/d9/f18 [2810303,79785] 0 2024-04-20T21:31:51.229 INFO:tasks.workunit.client.0.smithi062.stdout:4/134: dread - da/dd/df/d27/f2c zero size 2024-04-20T21:31:51.229 INFO:tasks.workunit.client.0.smithi062.stdout:5/90: write d8/d10/f17 [4368247,53528] 0 2024-04-20T21:31:51.243 INFO:tasks.workunit.client.0.smithi062.stdout:5/91: creat d8/d10/f26 x:0 0 0 2024-04-20T21:31:51.243 INFO:tasks.workunit.client.0.smithi062.stdout:3/190: rmdir d0/d1/d10/d2c/d42 39 2024-04-20T21:31:51.247 INFO:tasks.workunit.client.0.smithi062.stdout:3/191: fsync d0/d11/f1a 0 2024-04-20T21:31:51.251 INFO:tasks.workunit.client.0.smithi062.stdout:3/192: dread - d0/d1/f32 zero size 2024-04-20T21:31:51.253 INFO:tasks.workunit.client.0.smithi062.stdout:5/92: fsync d8/f1a 0 2024-04-20T21:31:51.257 INFO:tasks.workunit.client.0.smithi062.stdout:3/193: fsync d0/d1/f29 0 2024-04-20T21:31:51.259 INFO:tasks.workunit.client.0.smithi062.stdout:2/116: dwrite f6 [0,4194304] 0 2024-04-20T21:31:51.262 INFO:tasks.workunit.client.0.smithi062.stdout:0/169: dwrite d7/f9 [4194304,4194304] 0 2024-04-20T21:31:51.264 INFO:tasks.workunit.client.0.smithi062.stdout:5/93: write d8/d1e/f23 [426546,99125] 0 2024-04-20T21:31:51.266 INFO:tasks.workunit.client.0.smithi062.stdout:8/194: dread d7/d9/d29/f40 [0,4194304] 0 2024-04-20T21:31:51.276 INFO:tasks.workunit.client.0.smithi062.stdout:0/170: rmdir d7/d1c/d15 39 2024-04-20T21:31:51.279 INFO:tasks.workunit.client.0.smithi062.stdout:6/95: dread d0/d9/f16 [0,4194304] 0 2024-04-20T21:31:51.279 INFO:tasks.workunit.client.0.smithi062.stdout:5/94: write d8/f1c [93578,26449] 0 2024-04-20T21:31:51.281 INFO:tasks.workunit.client.0.smithi062.stdout:8/195: mknod d7/d9/d1b/d24/c45 0 2024-04-20T21:31:51.283 INFO:tasks.workunit.client.0.smithi062.stdout:2/117: dread d7/f17 [0,4194304] 0 2024-04-20T21:31:51.284 INFO:tasks.workunit.client.0.smithi062.stdout:4/135: dwrite da/dd/df/d27/f2d [0,4194304] 0 2024-04-20T21:31:51.288 INFO:tasks.workunit.client.0.smithi062.stdout:7/82: dread d7/d19/f1a [0,4194304] 0 2024-04-20T21:31:51.288 INFO:tasks.workunit.client.0.smithi062.stdout:5/95: write d8/d10/f26 [190812,37507] 0 2024-04-20T21:31:51.289 INFO:tasks.workunit.client.0.smithi062.stdout:1/147: dread f2 [0,4194304] 0 2024-04-20T21:31:51.289 INFO:tasks.workunit.client.0.smithi062.stdout:8/196: stat d7/d9/d29/f32 0 2024-04-20T21:31:51.291 INFO:tasks.workunit.client.0.smithi062.stdout:0/171: dread d7/fb [0,4194304] 0 2024-04-20T21:31:51.293 INFO:tasks.workunit.client.0.smithi062.stdout:8/197: chown d7/d9/f15 26193 1 2024-04-20T21:31:51.297 INFO:tasks.workunit.client.0.smithi062.stdout:6/96: symlink d0/d9/l20 0 2024-04-20T21:31:51.297 INFO:tasks.workunit.client.0.smithi062.stdout:5/96: write d8/d10/f26 [1239726,9425] 0 2024-04-20T21:31:51.301 INFO:tasks.workunit.client.0.smithi062.stdout:2/118: creat d7/da/d15/f1d x:0 0 0 2024-04-20T21:31:51.302 INFO:tasks.workunit.client.0.smithi062.stdout:5/97: stat d8/d10/c21 0 2024-04-20T21:31:51.303 INFO:tasks.workunit.client.0.smithi062.stdout:4/136: truncate da/dd/df/d27/f2c 279426 0 2024-04-20T21:31:51.307 INFO:tasks.workunit.client.0.smithi062.stdout:2/119: chown d7/da/d15/d1b 61 1 2024-04-20T21:31:51.307 INFO:tasks.workunit.client.0.smithi062.stdout:6/97: rename d0/ld to d0/d9/d11/l21 0 2024-04-20T21:31:51.309 INFO:tasks.workunit.client.0.smithi062.stdout:0/172: mkdir d7/d20/d29/d3c 0 2024-04-20T21:31:51.311 INFO:tasks.workunit.client.0.smithi062.stdout:9/151: dread d16/f1e [0,4194304] 0 2024-04-20T21:31:51.312 INFO:tasks.workunit.client.0.smithi062.stdout:5/98: write d8/d1e/f23 [382433,77322] 0 2024-04-20T21:31:51.313 INFO:tasks.workunit.client.0.smithi062.stdout:4/137: mkdir da/dd/df/d27/d30 0 2024-04-20T21:31:51.315 INFO:tasks.workunit.client.0.smithi062.stdout:9/152: fsync d16/f2e 0 2024-04-20T21:31:51.317 INFO:tasks.workunit.client.0.smithi062.stdout:8/198: getdents d7/d9/d1b 0 2024-04-20T21:31:51.319 INFO:tasks.workunit.client.0.smithi062.stdout:3/194: dwrite d0/d1/f20 [0,4194304] 0 2024-04-20T21:31:51.320 INFO:tasks.workunit.client.0.smithi062.stdout:2/120: write d7/f12 [629536,24316] 0 2024-04-20T21:31:51.321 INFO:tasks.workunit.client.0.smithi062.stdout:0/173: dread d7/fb [0,4194304] 0 2024-04-20T21:31:51.321 INFO:tasks.workunit.client.0.smithi062.stdout:9/153: write fe [3115556,57906] 0 2024-04-20T21:31:51.326 INFO:tasks.workunit.client.0.smithi062.stdout:7/83: truncate d7/f10 2930778 0 2024-04-20T21:31:51.328 INFO:tasks.workunit.client.0.smithi062.stdout:9/154: symlink d16/d2c/l35 0 2024-04-20T21:31:51.328 INFO:tasks.workunit.client.0.smithi062.stdout:8/199: mkdir d7/d9/d1b/d24/d46 0 2024-04-20T21:31:51.330 INFO:tasks.workunit.client.0.smithi062.stdout:2/121: dread d7/f17 [0,4194304] 0 2024-04-20T21:31:51.331 INFO:tasks.workunit.client.0.smithi062.stdout:4/138: rename da/dd/df/d27/f2d to da/dd/df/d27/f31 0 2024-04-20T21:31:51.335 INFO:tasks.workunit.client.0.smithi062.stdout:6/98: truncate d0/f10 665989 0 2024-04-20T21:31:51.336 INFO:tasks.workunit.client.0.smithi062.stdout:0/174: mkdir d7/d1c/d15/d17/d3d 0 2024-04-20T21:31:51.340 INFO:tasks.workunit.client.0.smithi062.stdout:2/122: read f6 [388440,51576] 0 2024-04-20T21:31:51.340 INFO:tasks.workunit.client.0.smithi062.stdout:8/200: symlink d7/d9/d1b/d24/l47 0 2024-04-20T21:31:51.340 INFO:tasks.workunit.client.0.smithi062.stdout:9/155: write d16/f1e [5365818,35742] 0 2024-04-20T21:31:51.340 INFO:tasks.workunit.client.0.smithi062.stdout:7/84: mkdir d7/d19/d1c 0 2024-04-20T21:31:51.344 INFO:tasks.workunit.client.0.smithi062.stdout:4/139: read da/f16 [289487,125104] 0 2024-04-20T21:31:51.346 INFO:tasks.workunit.client.0.smithi062.stdout:5/99: dwrite d8/d1e/f25 [0,4194304] 0 2024-04-20T21:31:51.347 INFO:tasks.workunit.client.0.smithi062.stdout:4/140: fsync da/dd/df/f10 0 2024-04-20T21:31:51.349 INFO:tasks.workunit.client.0.smithi062.stdout:2/123: fdatasync f6 0 2024-04-20T21:31:51.349 INFO:tasks.workunit.client.0.smithi062.stdout:1/148: dread dc/d15/f18 [0,4194304] 0 2024-04-20T21:31:51.350 INFO:tasks.workunit.client.0.smithi062.stdout:9/156: rmdir d16/d2c 39 2024-04-20T21:31:51.351 INFO:tasks.workunit.client.0.smithi062.stdout:9/157: stat d16/l26 0 2024-04-20T21:31:51.352 INFO:tasks.workunit.client.0.smithi062.stdout:7/85: fsync f4 0 2024-04-20T21:31:51.353 INFO:tasks.workunit.client.0.smithi062.stdout:7/86: read - d7/f14 zero size 2024-04-20T21:31:51.358 INFO:tasks.workunit.client.0.smithi062.stdout:8/201: creat d7/d9/d1b/d24/f48 x:0 0 0 2024-04-20T21:31:51.362 INFO:tasks.workunit.client.0.smithi062.stdout:4/141: mkdir da/dd/df/d27/d32 0 2024-04-20T21:31:51.369 INFO:tasks.workunit.client.0.smithi062.stdout:0/175: creat d7/d1c/f3e x:0 0 0 2024-04-20T21:31:51.372 INFO:tasks.workunit.client.0.smithi062.stdout:0/176: write d7/d1c/f32 [596578,72493] 0 2024-04-20T21:31:51.374 INFO:tasks.workunit.client.0.smithi062.stdout:4/142: mkdir da/dd/df/d27/d33 0 2024-04-20T21:31:51.375 INFO:tasks.workunit.client.0.smithi062.stdout:1/149: dwrite dc/f14 [0,4194304] 0 2024-04-20T21:31:51.377 INFO:tasks.workunit.client.0.smithi062.stdout:0/177: truncate d7/d1c/f39 541650 0 2024-04-20T21:31:51.379 INFO:tasks.workunit.client.0.smithi062.stdout:8/202: creat d7/d9/d1b/d24/d46/f49 x:0 0 0 2024-04-20T21:31:51.379 INFO:tasks.workunit.client.0.smithi062.stdout:5/100: link l0 d8/l27 0 2024-04-20T21:31:51.380 INFO:tasks.workunit.client.0.smithi062.stdout:9/158: chown d16/d2c/l34 86 1 2024-04-20T21:31:51.381 INFO:tasks.workunit.client.0.smithi062.stdout:0/178: stat d7/d1c/d15/d17/c34 0 2024-04-20T21:31:51.382 INFO:tasks.workunit.client.0.smithi062.stdout:2/124: link d7/ff d7/f1e 0 2024-04-20T21:31:51.383 INFO:tasks.workunit.client.0.smithi062.stdout:1/150: write f2 [5029203,116436] 0 2024-04-20T21:31:51.384 INFO:tasks.workunit.client.0.smithi062.stdout:7/87: getdents d7/d19 0 2024-04-20T21:31:51.385 INFO:tasks.workunit.client.0.smithi062.stdout:4/143: symlink da/dd/df/d27/d33/l34 0 2024-04-20T21:31:51.387 INFO:tasks.workunit.client.0.smithi062.stdout:7/88: dread - d7/f14 zero size 2024-04-20T21:31:51.388 INFO:tasks.workunit.client.0.smithi062.stdout:6/99: dwrite d0/f10 [0,4194304] 0 2024-04-20T21:31:51.390 INFO:tasks.workunit.client.0.smithi062.stdout:1/151: unlink f5 0 2024-04-20T21:31:51.396 INFO:tasks.workunit.client.0.smithi062.stdout:5/101: getdents d8/d10 0 2024-04-20T21:31:51.400 INFO:tasks.workunit.client.0.smithi062.stdout:1/152: write dc/d15/d17/f19 [3008903,96075] 0 2024-04-20T21:31:51.400 INFO:tasks.workunit.client.0.smithi062.stdout:2/125: rename d7/da/cc to d7/c1f 0 2024-04-20T21:31:51.402 INFO:tasks.workunit.client.0.smithi062.stdout:4/144: creat da/dd/f35 x:0 0 0 2024-04-20T21:31:51.404 INFO:tasks.workunit.client.0.smithi062.stdout:0/179: rmdir d7/d1c/d15/d17/d3d 0 2024-04-20T21:31:51.408 INFO:tasks.workunit.client.0.smithi062.stdout:9/159: dwrite f4 [0,4194304] 0 2024-04-20T21:31:51.408 INFO:tasks.workunit.client.0.smithi062.stdout:2/126: unlink l4 0 2024-04-20T21:31:51.412 INFO:tasks.workunit.client.0.smithi062.stdout:7/89: dread f5 [0,4194304] 0 2024-04-20T21:31:51.412 INFO:tasks.workunit.client.0.smithi062.stdout:8/203: dwrite d7/d9/f18 [0,4194304] 0 2024-04-20T21:31:51.413 INFO:tasks.workunit.client.0.smithi062.stdout:1/153: mknod dc/d27/c30 0 2024-04-20T21:31:51.414 INFO:tasks.workunit.client.0.smithi062.stdout:1/154: chown dc/fe 2715632 1 2024-04-20T21:31:51.420 INFO:tasks.workunit.client.0.smithi062.stdout:2/127: mknod d7/da/d15/c20 0 2024-04-20T21:31:51.420 INFO:tasks.workunit.client.0.smithi062.stdout:6/100: write d0/d9/f16 [381037,128028] 0 2024-04-20T21:31:51.422 INFO:tasks.workunit.client.0.smithi062.stdout:8/204: mknod d7/d25/c4a 0 2024-04-20T21:31:51.426 INFO:tasks.workunit.client.0.smithi062.stdout:9/160: dread d16/f1e [0,4194304] 0 2024-04-20T21:31:51.426 INFO:tasks.workunit.client.0.smithi062.stdout:4/145: dread da/dd/f14 [0,4194304] 0 2024-04-20T21:31:51.427 INFO:tasks.workunit.client.0.smithi062.stdout:8/205: write d7/d9/d1b/f1f [795967,54664] 0 2024-04-20T21:31:51.434 INFO:tasks.workunit.client.0.smithi062.stdout:5/102: truncate f4 1873870 0 2024-04-20T21:31:51.435 INFO:tasks.workunit.client.0.smithi062.stdout:2/128: dread d7/ff [0,4194304] 0 2024-04-20T21:31:51.439 INFO:tasks.workunit.client.0.smithi062.stdout:8/206: unlink f6 0 2024-04-20T21:31:51.440 INFO:tasks.workunit.client.0.smithi062.stdout:8/207: chown d7/d25/l33 2250482 1 2024-04-20T21:31:51.443 INFO:tasks.workunit.client.0.smithi062.stdout:5/103: rename d8/fc to d8/d10/f28 0 2024-04-20T21:31:51.445 INFO:tasks.workunit.client.0.smithi062.stdout:5/104: read d8/d1e/f24 [901208,54160] 0 2024-04-20T21:31:51.446 INFO:tasks.workunit.client.0.smithi062.stdout:4/146: link da/dd/d15/f1d da/dd/df/d27/f36 0 2024-04-20T21:31:51.447 INFO:tasks.workunit.client.0.smithi062.stdout:2/129: symlink d7/da/l21 0 2024-04-20T21:31:51.450 INFO:tasks.workunit.client.0.smithi062.stdout:8/208: rmdir d7/d9 39 2024-04-20T21:31:51.452 INFO:tasks.workunit.client.0.smithi062.stdout:0/180: truncate d7/d1c/ff 891037 0 2024-04-20T21:31:51.457 INFO:tasks.workunit.client.0.smithi062.stdout:0/181: truncate d7/d1c/f3e 487448 0 2024-04-20T21:31:51.457 INFO:tasks.workunit.client.0.smithi062.stdout:7/90: dwrite f4 [0,4194304] 0 2024-04-20T21:31:51.459 INFO:tasks.workunit.client.0.smithi062.stdout:9/161: creat d16/f36 x:0 0 0 2024-04-20T21:31:51.460 INFO:tasks.workunit.client.0.smithi062.stdout:5/105: symlink d8/d10/l29 0 2024-04-20T21:31:51.461 INFO:tasks.workunit.client.0.smithi062.stdout:4/147: unlink da/dd/d15/f26 0 2024-04-20T21:31:51.462 INFO:tasks.workunit.client.0.smithi062.stdout:0/182: write d7/d1c/d15/d17/f38 [96408,106132] 0 2024-04-20T21:31:51.462 INFO:tasks.workunit.client.0.smithi062.stdout:2/130: creat d7/da/f22 x:0 0 0 2024-04-20T21:31:51.465 INFO:tasks.workunit.client.0.smithi062.stdout:4/148: read da/fb [2817981,10748] 0 2024-04-20T21:31:51.468 INFO:tasks.workunit.client.0.smithi062.stdout:1/155: link dc/c13 dc/c31 0 2024-04-20T21:31:51.469 INFO:tasks.workunit.client.0.smithi062.stdout:6/101: dwrite d0/d9/d11/f1f [0,4194304] 0 2024-04-20T21:31:51.470 INFO:tasks.workunit.client.0.smithi062.stdout:1/156: stat dc/d27/c30 0 2024-04-20T21:31:51.474 INFO:tasks.workunit.client.0.smithi062.stdout:7/91: symlink d7/l1d 0 2024-04-20T21:31:51.477 INFO:tasks.workunit.client.0.smithi062.stdout:5/106: rmdir d8/d10 39 2024-04-20T21:31:51.481 INFO:tasks.workunit.client.0.smithi062.stdout:2/131: chown d7/l16 574978 1 2024-04-20T21:31:51.485 INFO:tasks.workunit.client.0.smithi062.stdout:3/195: sync 2024-04-20T21:31:51.485 INFO:tasks.workunit.client.0.smithi062.stdout:0/183: symlink d7/d1c/d15/d17/l3f 0 2024-04-20T21:31:51.489 INFO:tasks.workunit.client.0.smithi062.stdout:6/102: mkdir d0/d9/d11/d22 0 2024-04-20T21:31:51.490 INFO:tasks.workunit.client.0.smithi062.stdout:7/92: mknod d7/d19/c1e 0 2024-04-20T21:31:51.490 INFO:tasks.workunit.client.0.smithi062.stdout:6/103: fdatasync d0/d9/f13 0 2024-04-20T21:31:51.491 INFO:tasks.workunit.client.0.smithi062.stdout:9/162: symlink d16/l37 0 2024-04-20T21:31:51.492 INFO:tasks.workunit.client.0.smithi062.stdout:5/107: readlink d8/d10/l12 0 2024-04-20T21:31:51.492 INFO:tasks.workunit.client.0.smithi062.stdout:6/104: fsync d0/d9/f16 0 2024-04-20T21:31:51.493 INFO:tasks.workunit.client.0.smithi062.stdout:5/108: readlink d8/d10/l12 0 2024-04-20T21:31:51.493 INFO:tasks.workunit.client.0.smithi062.stdout:2/132: stat d7/cd 0 2024-04-20T21:31:51.498 INFO:tasks.workunit.client.0.smithi062.stdout:4/149: creat da/dd/df/d27/d32/f37 x:0 0 0 2024-04-20T21:31:51.498 INFO:tasks.workunit.client.0.smithi062.stdout:0/184: write d7/d1c/f11 [2268807,51619] 0 2024-04-20T21:31:51.498 INFO:tasks.workunit.client.0.smithi062.stdout:3/196: creat d0/d1/d10/f47 x:0 0 0 2024-04-20T21:31:51.507 INFO:tasks.workunit.client.0.smithi062.stdout:1/157: dwrite dc/f21 [0,4194304] 0 2024-04-20T21:31:51.507 INFO:tasks.workunit.client.0.smithi062.stdout:4/150: truncate da/dd/f19 1304752 0 2024-04-20T21:31:51.508 INFO:tasks.workunit.client.0.smithi062.stdout:9/163: dread d16/f1c [0,4194304] 0 2024-04-20T21:31:51.509 INFO:tasks.workunit.client.0.smithi062.stdout:1/158: readlink - no filename 2024-04-20T21:31:51.510 INFO:tasks.workunit.client.0.smithi062.stdout:0/185: rename d7/d20/d29 to d7/d1c/d15/d17/d40 0 2024-04-20T21:31:51.512 INFO:tasks.workunit.client.0.smithi062.stdout:4/151: chown da/dd/l2b 41955168 1 2024-04-20T21:31:51.513 INFO:tasks.workunit.client.0.smithi062.stdout:8/209: truncate d7/d9/f17 2232647 0 2024-04-20T21:31:51.513 INFO:tasks.workunit.client.0.smithi062.stdout:9/164: chown d16/f32 41 1 2024-04-20T21:31:51.517 INFO:tasks.workunit.client.0.smithi062.stdout:5/109: dread d8/d1e/f25 [0,4194304] 0 2024-04-20T21:31:51.519 INFO:tasks.workunit.client.0.smithi062.stdout:2/133: dwrite d7/da/f22 [0,4194304] 0 2024-04-20T21:31:51.522 INFO:tasks.workunit.client.0.smithi062.stdout:0/186: unlink d7/cd 0 2024-04-20T21:31:51.523 INFO:tasks.workunit.client.0.smithi062.stdout:6/105: getdents d0/d1a 0 2024-04-20T21:31:51.525 INFO:tasks.workunit.client.0.smithi062.stdout:2/134: chown d7/cd 0 1 2024-04-20T21:31:51.527 INFO:tasks.workunit.client.0.smithi062.stdout:0/187: write d7/d1c/f11 [2163297,99043] 0 2024-04-20T21:31:51.527 INFO:tasks.workunit.client.0.smithi062.stdout:8/210: creat d7/d9/d29/f4b x:0 0 0 2024-04-20T21:31:51.528 INFO:tasks.workunit.client.0.smithi062.stdout:7/93: link d7/fe d7/d19/d1c/f1f 0 2024-04-20T21:31:51.528 INFO:tasks.workunit.client.0.smithi062.stdout:5/110: mkdir d8/d10/d2a 0 2024-04-20T21:31:51.528 INFO:tasks.workunit.client.0.smithi062.stdout:6/106: fsync d0/d9/f18 0 2024-04-20T21:31:51.529 INFO:tasks.workunit.client.0.smithi062.stdout:1/159: dread dc/ff [0,4194304] 0 2024-04-20T21:31:51.535 INFO:tasks.workunit.client.0.smithi062.stdout:9/165: dread f6 [0,4194304] 0 2024-04-20T21:31:51.535 INFO:tasks.workunit.client.0.smithi062.stdout:5/111: write d8/fe [5116724,61936] 0 2024-04-20T21:31:51.537 INFO:tasks.workunit.client.0.smithi062.stdout:2/135: creat d7/da/d15/f23 x:0 0 0 2024-04-20T21:31:51.540 INFO:tasks.workunit.client.0.smithi062.stdout:4/152: dwrite da/f13 [0,4194304] 0 2024-04-20T21:31:51.542 INFO:tasks.workunit.client.0.smithi062.stdout:7/94: creat d7/d19/f20 x:0 0 0 2024-04-20T21:31:51.543 INFO:tasks.workunit.client.0.smithi062.stdout:2/136: dread d7/f17 [0,4194304] 0 2024-04-20T21:31:51.544 INFO:tasks.workunit.client.0.smithi062.stdout:7/95: write f5 [3168252,100430] 0 2024-04-20T21:31:51.545 INFO:tasks.workunit.client.0.smithi062.stdout:2/137: dread - d7/da/d15/f23 zero size 2024-04-20T21:31:51.547 INFO:tasks.workunit.client.0.smithi062.stdout:8/211: creat d7/d9/d29/f4c x:0 0 0 2024-04-20T21:31:51.549 INFO:tasks.workunit.client.0.smithi062.stdout:1/160: unlink c4 0 2024-04-20T21:31:51.551 INFO:tasks.workunit.client.0.smithi062.stdout:4/153: rename da/dd/l2a to da/dd/df/d27/l38 0 2024-04-20T21:31:51.553 INFO:tasks.workunit.client.0.smithi062.stdout:6/107: getdents d0/d9/d11/d22 0 2024-04-20T21:31:51.553 INFO:tasks.workunit.client.0.smithi062.stdout:8/212: rmdir d7/d9/d1b/d24 39 2024-04-20T21:31:51.554 INFO:tasks.workunit.client.0.smithi062.stdout:1/161: fsync f1 0 2024-04-20T21:31:51.556 INFO:tasks.workunit.client.0.smithi062.stdout:7/96: fsync d7/f10 0 2024-04-20T21:31:51.556 INFO:tasks.workunit.client.0.smithi062.stdout:1/162: write fb [3442332,91326] 0 2024-04-20T21:31:51.559 INFO:tasks.workunit.client.0.smithi062.stdout:1/163: fdatasync f6 0 2024-04-20T21:31:51.560 INFO:tasks.workunit.client.0.smithi062.stdout:8/213: rename d7/d9 to d7/d9/d4d 22 2024-04-20T21:31:51.561 INFO:tasks.workunit.client.0.smithi062.stdout:7/97: write f4 [4456041,81034] 0 2024-04-20T21:31:51.562 INFO:tasks.workunit.client.0.smithi062.stdout:6/108: rename d0 to d0/d9/d23 22 2024-04-20T21:31:51.565 INFO:tasks.workunit.client.0.smithi062.stdout:9/166: link d16/d2c/c2d d16/d2c/c38 0 2024-04-20T21:31:51.568 INFO:tasks.workunit.client.0.smithi062.stdout:1/164: chown dc/d15/d17/f23 0 1 2024-04-20T21:31:51.571 INFO:tasks.workunit.client.0.smithi062.stdout:1/165: chown f1 184108123 1 2024-04-20T21:31:51.576 INFO:tasks.workunit.client.0.smithi062.stdout:8/214: creat d7/d9/d1b/f4e x:0 0 0 2024-04-20T21:31:51.578 INFO:tasks.workunit.client.0.smithi062.stdout:3/197: dread d0/f2f [4194304,4194304] 0 2024-04-20T21:31:51.578 INFO:tasks.workunit.client.0.smithi062.stdout:1/166: write dc/d15/f16 [2712074,6477] 0 2024-04-20T21:31:51.578 INFO:tasks.workunit.client.0.smithi062.stdout:6/109: link d0/f10 d0/d9/d11/d22/f24 0 2024-04-20T21:31:51.580 INFO:tasks.workunit.client.0.smithi062.stdout:9/167: fsync d16/f1d 0 2024-04-20T21:31:51.589 INFO:tasks.workunit.client.0.smithi062.stdout:3/198: dread - d0/d1/f32 zero size 2024-04-20T21:31:51.591 INFO:tasks.workunit.client.0.smithi062.stdout:7/98: dread f4 [0,4194304] 0 2024-04-20T21:31:51.591 INFO:tasks.workunit.client.0.smithi062.stdout:1/167: mkdir dc/d27/d32 0 2024-04-20T21:31:51.592 INFO:tasks.workunit.client.0.smithi062.stdout:9/168: stat d16/d2c/l30 0 2024-04-20T21:31:51.597 INFO:tasks.workunit.client.0.smithi062.stdout:3/199: mknod d0/d1/d3c/d43/c48 0 2024-04-20T21:31:51.597 INFO:tasks.workunit.client.0.smithi062.stdout:7/99: symlink d7/d19/l21 0 2024-04-20T21:31:51.598 INFO:tasks.workunit.client.0.smithi062.stdout:5/112: dwrite d8/d1e/f24 [0,4194304] 0 2024-04-20T21:31:51.598 INFO:tasks.workunit.client.0.smithi062.stdout:6/110: dread d0/f6 [0,4194304] 0 2024-04-20T21:31:51.599 INFO:tasks.workunit.client.0.smithi062.stdout:0/188: dread d7/d1c/f18 [0,4194304] 0 2024-04-20T21:31:51.600 INFO:tasks.workunit.client.0.smithi062.stdout:5/113: fdatasync d8/f1a 0 2024-04-20T21:31:51.602 INFO:tasks.workunit.client.0.smithi062.stdout:3/200: chown d0/d1/c16 2683 1 2024-04-20T21:31:51.602 INFO:tasks.workunit.client.0.smithi062.stdout:1/168: unlink f6 0 2024-04-20T21:31:51.604 INFO:tasks.workunit.client.0.smithi062.stdout:9/169: creat d16/d2f/f39 x:0 0 0 2024-04-20T21:31:51.604 INFO:tasks.workunit.client.0.smithi062.stdout:7/100: fsync d7/d19/f20 0 2024-04-20T21:31:51.608 INFO:tasks.workunit.client.0.smithi062.stdout:4/154: dread da/fb [0,4194304] 0 2024-04-20T21:31:51.609 INFO:tasks.workunit.client.0.smithi062.stdout:7/101: chown f4 16 1 2024-04-20T21:31:51.611 INFO:tasks.workunit.client.0.smithi062.stdout:5/114: mkdir d8/d10/d2a/d2b 0 2024-04-20T21:31:51.613 INFO:tasks.workunit.client.0.smithi062.stdout:7/102: dread - d7/f18 zero size 2024-04-20T21:31:51.614 INFO:tasks.workunit.client.0.smithi062.stdout:3/201: rename d0/d1/l5 to d0/d11/l49 0 2024-04-20T21:31:51.616 INFO:tasks.workunit.client.0.smithi062.stdout:8/215: dwrite d7/d9/f12 [0,4194304] 0 2024-04-20T21:31:51.616 INFO:tasks.workunit.client.0.smithi062.stdout:4/155: chown f3 743022 1 2024-04-20T21:31:51.620 INFO:tasks.workunit.client.0.smithi062.stdout:9/170: creat d16/f3a x:0 0 0 2024-04-20T21:31:51.621 INFO:tasks.workunit.client.0.smithi062.stdout:5/115: mkdir d8/d1e/d2c 0 2024-04-20T21:31:51.621 INFO:tasks.workunit.client.0.smithi062.stdout:7/103: chown d7/f12 205 1 2024-04-20T21:31:51.622 INFO:tasks.workunit.client.0.smithi062.stdout:9/171: readlink d16/l24 0 2024-04-20T21:31:51.623 INFO:tasks.workunit.client.0.smithi062.stdout:0/189: dread f1 [0,4194304] 0 2024-04-20T21:31:51.624 INFO:tasks.workunit.client.0.smithi062.stdout:4/156: symlink da/dd/df/l39 0 2024-04-20T21:31:51.624 INFO:tasks.workunit.client.0.smithi062.stdout:5/116: write d8/f1c [214492,55318] 0 2024-04-20T21:31:51.625 INFO:tasks.workunit.client.0.smithi062.stdout:7/104: chown d7/f10 3710326 1 2024-04-20T21:31:51.636 INFO:tasks.workunit.client.0.smithi062.stdout:6/111: dread d0/d9/f13 [0,4194304] 0 2024-04-20T21:31:51.638 INFO:tasks.workunit.client.0.smithi062.stdout:9/172: creat d16/f3b x:0 0 0 2024-04-20T21:31:51.641 INFO:tasks.workunit.client.0.smithi062.stdout:8/216: dread d7/d9/d1b/f21 [0,4194304] 0 2024-04-20T21:31:51.642 INFO:tasks.workunit.client.0.smithi062.stdout:6/112: creat d0/d9/d11/f25 x:0 0 0 2024-04-20T21:31:51.643 INFO:tasks.workunit.client.0.smithi062.stdout:9/173: write fd [5107876,79084] 0 2024-04-20T21:31:51.647 INFO:tasks.workunit.client.0.smithi062.stdout:0/190: link d7/d1c/d15/f2d d7/d1c/f41 0 2024-04-20T21:31:51.652 INFO:tasks.workunit.client.0.smithi062.stdout:9/174: mknod d16/d2c/c3c 0 2024-04-20T21:31:51.653 INFO:tasks.workunit.client.0.smithi062.stdout:6/113: mkdir d0/d9/d11/d22/d26 0 2024-04-20T21:31:51.657 INFO:tasks.workunit.client.0.smithi062.stdout:0/191: dread d7/d1c/d15/d17/f38 [0,4194304] 0 2024-04-20T21:31:51.663 INFO:tasks.workunit.client.0.smithi062.stdout:1/169: dwrite dc/d15/d20/d28/f2f [0,4194304] 0 2024-04-20T21:31:51.664 INFO:tasks.workunit.client.0.smithi062.stdout:9/175: write d16/f36 [50026,27808] 0 2024-04-20T21:31:51.668 INFO:tasks.workunit.client.0.smithi062.stdout:0/192: chown d7/d1c/d15/f2d 10560924 1 2024-04-20T21:31:51.672 INFO:tasks.workunit.client.0.smithi062.stdout:7/105: dwrite d7/d19/f20 [0,4194304] 0 2024-04-20T21:31:51.674 INFO:tasks.workunit.client.0.smithi062.stdout:5/117: dwrite f4 [0,4194304] 0 2024-04-20T21:31:51.676 INFO:tasks.workunit.client.0.smithi062.stdout:0/193: chown d7/d1c/f32 12815 1 2024-04-20T21:31:51.676 INFO:tasks.workunit.client.0.smithi062.stdout:8/217: dread d7/f23 [0,4194304] 0 2024-04-20T21:31:51.676 INFO:tasks.workunit.client.0.smithi062.stdout:9/176: read d16/f19 [1000459,120001] 0 2024-04-20T21:31:51.677 INFO:tasks.workunit.client.0.smithi062.stdout:1/170: link dc/fe dc/d15/d20/f33 0 2024-04-20T21:31:51.684 INFO:tasks.workunit.client.0.smithi062.stdout:0/194: chown c0 13567 1 2024-04-20T21:31:51.687 INFO:tasks.workunit.client.0.smithi062.stdout:9/177: mkdir d16/d3d 0 2024-04-20T21:31:51.687 INFO:tasks.workunit.client.0.smithi062.stdout:4/157: dread da/dd/f12 [0,4194304] 0 2024-04-20T21:31:51.688 INFO:tasks.workunit.client.0.smithi062.stdout:7/106: write d7/f10 [2468619,77726] 0 2024-04-20T21:31:51.691 INFO:tasks.workunit.client.0.smithi062.stdout:7/107: fsync d7/f14 0 2024-04-20T21:31:51.691 INFO:tasks.workunit.client.0.smithi062.stdout:4/158: rename da to da/dd/df/d27/d30/d3a 22 2024-04-20T21:31:51.692 INFO:tasks.workunit.client.0.smithi062.stdout:1/171: fdatasync dc/d15/d17/f23 0 2024-04-20T21:31:51.692 INFO:tasks.workunit.client.0.smithi062.stdout:6/114: dwrite d0/d9/d11/f25 [0,4194304] 0 2024-04-20T21:31:51.696 INFO:tasks.workunit.client.0.smithi062.stdout:1/172: write dc/d15/f16 [4074197,37429] 0 2024-04-20T21:31:51.698 INFO:tasks.workunit.client.0.smithi062.stdout:8/218: mkdir d7/d25/d4f 0 2024-04-20T21:31:51.700 INFO:tasks.workunit.client.0.smithi062.stdout:7/108: dread d7/ff [0,4194304] 0 2024-04-20T21:31:51.705 INFO:tasks.workunit.client.0.smithi062.stdout:7/109: write d7/d19/f1a [3331539,94336] 0 2024-04-20T21:31:51.710 INFO:tasks.workunit.client.0.smithi062.stdout:7/110: write f6 [563212,58105] 0 2024-04-20T21:31:51.712 INFO:tasks.workunit.client.0.smithi062.stdout:1/173: chown dc/c12 1686 1 2024-04-20T21:31:51.717 INFO:tasks.workunit.client.0.smithi062.stdout:7/111: mknod d7/c22 0 2024-04-20T21:31:51.718 INFO:tasks.workunit.client.0.smithi062.stdout:1/174: truncate f9 354679 0 2024-04-20T21:31:51.719 INFO:tasks.workunit.client.0.smithi062.stdout:9/178: dread f4 [0,4194304] 0 2024-04-20T21:31:51.723 INFO:tasks.workunit.client.0.smithi062.stdout:0/195: write d7/d1c/ff [523477,122484] 0 2024-04-20T21:31:51.725 INFO:tasks.workunit.client.0.smithi062.stdout:7/112: readlink d7/l11 0 2024-04-20T21:31:51.725 INFO:tasks.workunit.client.0.smithi062.stdout:8/219: getdents d7/d9/d1b/d24/d46 0 2024-04-20T21:31:51.730 INFO:tasks.workunit.client.0.smithi062.stdout:9/179: write d16/f3b [554814,103265] 0 2024-04-20T21:31:51.731 INFO:tasks.workunit.client.0.smithi062.stdout:6/115: dread d0/d9/f15 [0,4194304] 0 2024-04-20T21:31:51.734 INFO:tasks.workunit.client.0.smithi062.stdout:7/113: mknod d7/c23 0 2024-04-20T21:31:51.734 INFO:tasks.workunit.client.0.smithi062.stdout:8/220: rmdir d7/d9 39 2024-04-20T21:31:51.739 INFO:tasks.workunit.client.0.smithi062.stdout:9/180: stat d16/d2c/l30 0 2024-04-20T21:31:51.740 INFO:tasks.workunit.client.0.smithi062.stdout:4/159: dwrite da/dd/d15/f20 [0,4194304] 0 2024-04-20T21:31:51.742 INFO:tasks.workunit.client.0.smithi062.stdout:7/114: dread - d7/f18 zero size 2024-04-20T21:31:51.744 INFO:tasks.workunit.client.0.smithi062.stdout:5/118: dwrite d8/d10/f28 [0,4194304] 0 2024-04-20T21:31:51.746 INFO:tasks.workunit.client.0.smithi062.stdout:3/202: sync 2024-04-20T21:31:51.746 INFO:tasks.workunit.client.0.smithi062.stdout:2/138: sync 2024-04-20T21:31:51.747 INFO:tasks.workunit.client.0.smithi062.stdout:8/221: truncate d7/d9/f41 850219 0 2024-04-20T21:31:51.747 INFO:tasks.workunit.client.0.smithi062.stdout:0/196: rename d7/d1c/d15/d17/c34 to d7/d1c/d15/d17/d26/c42 0 2024-04-20T21:31:51.749 INFO:tasks.workunit.client.0.smithi062.stdout:6/116: creat d0/d9/d11/d22/f27 x:0 0 0 2024-04-20T21:31:51.750 INFO:tasks.workunit.client.0.smithi062.stdout:3/203: write d0/d1/d10/f27 [2438252,122686] 0 2024-04-20T21:31:51.751 INFO:tasks.workunit.client.0.smithi062.stdout:4/160: chown da/dd/d15/l1f 0 1 2024-04-20T21:31:51.755 INFO:tasks.workunit.client.0.smithi062.stdout:1/175: link dc/d15/d17/f23 dc/d15/d17/f34 0 2024-04-20T21:31:51.757 INFO:tasks.workunit.client.0.smithi062.stdout:6/117: write d0/d9/f16 [351313,125517] 0 2024-04-20T21:31:51.758 INFO:tasks.workunit.client.0.smithi062.stdout:7/115: creat d7/f24 x:0 0 0 2024-04-20T21:31:51.761 INFO:tasks.workunit.client.0.smithi062.stdout:4/161: write da/dd/df/f10 [633472,117887] 0 2024-04-20T21:31:51.761 INFO:tasks.workunit.client.0.smithi062.stdout:0/197: readlink d7/d1c/l35 0 2024-04-20T21:31:51.761 INFO:tasks.workunit.client.0.smithi062.stdout:3/204: dread - d0/d1/d10/f47 zero size 2024-04-20T21:31:51.764 INFO:tasks.workunit.client.0.smithi062.stdout:2/139: mkdir d7/da/d15/d1b/d24 0 2024-04-20T21:31:51.765 INFO:tasks.workunit.client.0.smithi062.stdout:7/116: unlink d7/c22 0 2024-04-20T21:31:51.765 INFO:tasks.workunit.client.0.smithi062.stdout:4/162: write da/dd/df/d27/f2c [704307,7083] 0 2024-04-20T21:31:51.766 INFO:tasks.workunit.client.0.smithi062.stdout:3/205: fdatasync d0/d1/f20 0 2024-04-20T21:31:51.767 INFO:tasks.workunit.client.0.smithi062.stdout:6/118: symlink d0/d9/d11/l28 0 2024-04-20T21:31:51.771 INFO:tasks.workunit.client.0.smithi062.stdout:9/181: getdents d16 0 2024-04-20T21:31:51.771 INFO:tasks.workunit.client.0.smithi062.stdout:7/117: chown d7/d19/l21 49390 1 2024-04-20T21:31:51.772 INFO:tasks.workunit.client.0.smithi062.stdout:0/198: symlink d7/d1e/l43 0 2024-04-20T21:31:51.772 INFO:tasks.workunit.client.0.smithi062.stdout:6/119: readlink d0/d9/l12 0 2024-04-20T21:31:51.773 INFO:tasks.workunit.client.0.smithi062.stdout:2/140: mknod d7/da/c25 0 2024-04-20T21:31:51.776 INFO:tasks.workunit.client.0.smithi062.stdout:8/222: getdents d7/d25 0 2024-04-20T21:31:51.777 INFO:tasks.workunit.client.0.smithi062.stdout:0/199: read d7/d1c/ff [34074,14837] 0 2024-04-20T21:31:51.778 INFO:tasks.workunit.client.0.smithi062.stdout:5/119: dread d8/f1a [0,4194304] 0 2024-04-20T21:31:51.780 INFO:tasks.workunit.client.0.smithi062.stdout:1/176: creat dc/f35 x:0 0 0 2024-04-20T21:31:51.787 INFO:tasks.workunit.client.0.smithi062.stdout:5/120: chown d8/ca 4880 1 2024-04-20T21:31:51.788 INFO:tasks.workunit.client.0.smithi062.stdout:4/163: mkdir da/dd/df/d27/d30/d3b 0 2024-04-20T21:31:51.791 INFO:tasks.workunit.client.0.smithi062.stdout:2/141: creat d7/da/d15/f26 x:0 0 0 2024-04-20T21:31:51.793 INFO:tasks.workunit.client.0.smithi062.stdout:3/206: creat d0/d1/d10/d2c/d42/f4a x:0 0 0 2024-04-20T21:31:51.795 INFO:tasks.workunit.client.0.smithi062.stdout:1/177: mkdir dc/d15/d17/d36 0 2024-04-20T21:31:51.797 INFO:tasks.workunit.client.0.smithi062.stdout:5/121: write d8/d10/f1b [3662636,106224] 0 2024-04-20T21:31:51.798 INFO:tasks.workunit.client.0.smithi062.stdout:2/142: symlink d7/da/l27 0 2024-04-20T21:31:51.800 INFO:tasks.workunit.client.0.smithi062.stdout:0/200: link d7/d1c/d15/d17/f38 d7/d20/d3a/f44 0 2024-04-20T21:31:51.802 INFO:tasks.workunit.client.0.smithi062.stdout:7/118: dread d7/f12 [0,4194304] 0 2024-04-20T21:31:51.802 INFO:tasks.workunit.client.0.smithi062.stdout:9/182: dwrite d16/f36 [0,4194304] 0 2024-04-20T21:31:51.803 INFO:tasks.workunit.client.0.smithi062.stdout:5/122: readlink d8/d10/l29 0 2024-04-20T21:31:51.804 INFO:tasks.workunit.client.0.smithi062.stdout:2/143: mkdir d7/da/d28 0 2024-04-20T21:31:51.808 INFO:tasks.workunit.client.0.smithi062.stdout:6/120: dread d0/d9/d11/f25 [0,4194304] 0 2024-04-20T21:31:51.812 INFO:tasks.workunit.client.0.smithi062.stdout:0/201: fdatasync d7/d1c/f3e 0 2024-04-20T21:31:51.813 INFO:tasks.workunit.client.0.smithi062.stdout:7/119: dread - d7/f18 zero size 2024-04-20T21:31:51.813 INFO:tasks.workunit.client.0.smithi062.stdout:1/178: symlink dc/d27/d32/l37 0 2024-04-20T21:31:51.814 INFO:tasks.workunit.client.0.smithi062.stdout:3/207: mknod d0/d1/d10/d31/d41/c4b 0 2024-04-20T21:31:51.815 INFO:tasks.workunit.client.0.smithi062.stdout:0/202: chown c0 8900 1 2024-04-20T21:31:51.817 INFO:tasks.workunit.client.0.smithi062.stdout:2/144: rename d7/da/c25 to d7/da/d15/c29 0 2024-04-20T21:31:51.895 INFO:tasks.workunit.client.0.smithi062.stdout:1/179: mknod dc/d15/d20/d28/c38 0 2024-04-20T21:31:51.896 INFO:tasks.workunit.client.0.smithi062.stdout:0/203: creat d7/d1e/f45 x:0 0 0 2024-04-20T21:31:51.897 INFO:tasks.workunit.client.0.smithi062.stdout:8/223: dwrite d7/d9/f3a [0,4194304] 0 2024-04-20T21:31:51.901 INFO:tasks.workunit.client.0.smithi062.stdout:5/123: truncate d8/d10/f1b 1855231 0 2024-04-20T21:31:51.907 INFO:tasks.workunit.client.0.smithi062.stdout:7/120: getdents d7/d19 0 2024-04-20T21:31:51.907 INFO:tasks.workunit.client.0.smithi062.stdout:3/208: creat d0/f4c x:0 0 0 2024-04-20T21:31:51.907 INFO:tasks.workunit.client.0.smithi062.stdout:2/145: creat d7/da/d28/f2a x:0 0 0 2024-04-20T21:31:51.907 INFO:tasks.workunit.client.0.smithi062.stdout:4/164: dwrite da/dd/f35 [0,4194304] 0 2024-04-20T21:31:51.908 INFO:tasks.workunit.client.0.smithi062.stdout:0/204: dread f1 [0,4194304] 0 2024-04-20T21:31:51.908 INFO:tasks.workunit.client.0.smithi062.stdout:8/224: creat d7/d9/d1b/d24/f50 x:0 0 0 2024-04-20T21:31:51.912 INFO:tasks.workunit.client.0.smithi062.stdout:1/180: unlink dc/c1c 0 2024-04-20T21:31:51.912 INFO:tasks.workunit.client.0.smithi062.stdout:5/124: rmdir d8/d10/d2a 39 2024-04-20T21:31:51.914 INFO:tasks.workunit.client.0.smithi062.stdout:7/121: chown d7/l1d 0 1 2024-04-20T21:31:51.915 INFO:tasks.workunit.client.0.smithi062.stdout:9/183: dwrite d16/f1d [4194304,4194304] 0 2024-04-20T21:31:51.916 INFO:tasks.workunit.client.0.smithi062.stdout:2/146: symlink d7/da/l2b 0 2024-04-20T21:31:51.918 INFO:tasks.workunit.client.0.smithi062.stdout:0/205: chown d7/d1c/d15/d17/d40/d3c 2585189 1 2024-04-20T21:31:51.918 INFO:tasks.workunit.client.0.smithi062.stdout:1/181: stat dc/d27/d32/l37 0 2024-04-20T21:31:51.921 INFO:tasks.workunit.client.0.smithi062.stdout:5/125: mknod d8/d1e/c2d 0 2024-04-20T21:31:51.921 INFO:tasks.workunit.client.0.smithi062.stdout:7/122: symlink d7/l25 0 2024-04-20T21:31:51.922 INFO:tasks.workunit.client.0.smithi062.stdout:2/147: unlink f6 0 2024-04-20T21:31:51.923 INFO:tasks.workunit.client.0.smithi062.stdout:9/184: mkdir d16/d3e 0 2024-04-20T21:31:51.926 INFO:tasks.workunit.client.0.smithi062.stdout:0/206: creat d7/d1c/d15/f46 x:0 0 0 2024-04-20T21:31:51.929 INFO:tasks.workunit.client.0.smithi062.stdout:8/225: link d7/d9/d1b/f1f d7/d9/f51 0 2024-04-20T21:31:51.931 INFO:tasks.workunit.client.0.smithi062.stdout:2/148: creat d7/da/d15/d1b/f2c x:0 0 0 2024-04-20T21:31:51.932 INFO:tasks.workunit.client.0.smithi062.stdout:5/126: symlink d8/d1e/d2c/l2e 0 2024-04-20T21:31:51.933 INFO:tasks.workunit.client.0.smithi062.stdout:4/165: rename da/dd/df/d27/d33 to da/dd/d3c 0 2024-04-20T21:31:51.933 INFO:tasks.workunit.client.0.smithi062.stdout:1/182: truncate dc/d27/f29 1339085 0 2024-04-20T21:31:51.936 INFO:tasks.workunit.client.0.smithi062.stdout:3/209: dread d0/f2f [4194304,4194304] 0 2024-04-20T21:31:51.938 INFO:tasks.workunit.client.0.smithi062.stdout:4/166: fsync da/dd/f35 0 2024-04-20T21:31:51.940 INFO:tasks.workunit.client.0.smithi062.stdout:2/149: truncate d7/ff 4145776 0 2024-04-20T21:31:51.942 INFO:tasks.workunit.client.0.smithi062.stdout:3/210: creat d0/d1b/f4d x:0 0 0 2024-04-20T21:31:51.946 INFO:tasks.workunit.client.0.smithi062.stdout:3/211: chown d0/d1b/c36 3 1 2024-04-20T21:31:51.946 INFO:tasks.workunit.client.0.smithi062.stdout:7/123: dwrite d7/d19/d1c/f1f [0,4194304] 0 2024-04-20T21:31:51.948 INFO:tasks.workunit.client.0.smithi062.stdout:7/124: dread - d7/f18 zero size 2024-04-20T21:31:51.950 INFO:tasks.workunit.client.0.smithi062.stdout:5/127: getdents d8 0 2024-04-20T21:31:51.951 INFO:tasks.workunit.client.0.smithi062.stdout:8/226: dwrite d7/f22 [0,4194304] 0 2024-04-20T21:31:51.954 INFO:tasks.workunit.client.0.smithi062.stdout:7/125: mkdir d7/d26 0 2024-04-20T21:31:51.955 INFO:tasks.workunit.client.0.smithi062.stdout:8/227: chown d7/f23 31781668 1 2024-04-20T21:31:51.957 INFO:tasks.workunit.client.0.smithi062.stdout:4/167: rmdir da/dd/df/d27/d30/d3b 0 2024-04-20T21:31:51.957 INFO:tasks.workunit.client.0.smithi062.stdout:0/207: write d7/d1c/f41 [136225,11124] 0 2024-04-20T21:31:51.964 INFO:tasks.workunit.client.0.smithi062.stdout:4/168: dread da/f16 [0,4194304] 0 2024-04-20T21:31:51.966 INFO:tasks.workunit.client.0.smithi062.stdout:8/228: unlink d7/d25/l44 0 2024-04-20T21:31:51.967 INFO:tasks.workunit.client.0.smithi062.stdout:0/208: write f1 [1544773,108466] 0 2024-04-20T21:31:51.968 INFO:tasks.workunit.client.0.smithi062.stdout:9/185: rename d16/c23 to d16/d3d/c3f 0 2024-04-20T21:31:51.971 INFO:tasks.workunit.client.0.smithi062.stdout:5/128: mkdir d8/d10/d2a/d2f 0 2024-04-20T21:31:51.971 INFO:tasks.workunit.client.0.smithi062.stdout:3/212: truncate d0/f8 6015805 0 2024-04-20T21:31:51.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:51 smithi189 ceph-mon[21562]: pgmap v10: 65 pgs: 65 active+clean; 226 MiB data, 881 MiB used, 536 GiB / 536 GiB avail; 1011 KiB/s rd, 2.9 MiB/s wr, 203 op/s 2024-04-20T21:31:51.995 INFO:tasks.workunit.client.0.smithi062.stdout:1/183: write dc/d27/f29 [1071473,120741] 0 2024-04-20T21:31:51.998 INFO:tasks.workunit.client.0.smithi062.stdout:3/213: creat d0/d1/d10/f4e x:0 0 0 2024-04-20T21:31:52.000 INFO:tasks.workunit.client.0.smithi062.stdout:9/186: creat d16/d3e/f40 x:0 0 0 2024-04-20T21:31:52.001 INFO:tasks.workunit.client.0.smithi062.stdout:5/129: truncate d8/ff 445889 0 2024-04-20T21:31:52.001 INFO:tasks.workunit.client.0.smithi062.stdout:1/184: mknod dc/d15/d17/c39 0 2024-04-20T21:31:52.002 INFO:tasks.workunit.client.0.smithi062.stdout:3/214: chown d0/d1/d10/d2c/d42/f4a 319 1 2024-04-20T21:31:52.004 INFO:tasks.workunit.client.0.smithi062.stdout:1/185: chown dc/d27/f29 824008 1 2024-04-20T21:31:52.007 INFO:tasks.workunit.client.0.smithi062.stdout:2/150: dwrite d7/f1e [0,4194304] 0 2024-04-20T21:31:52.007 INFO:tasks.workunit.client.0.smithi062.stdout:5/130: getdents d8/d10/d2a/d2b 0 2024-04-20T21:31:52.010 INFO:tasks.workunit.client.0.smithi062.stdout:3/215: creat d0/f4f x:0 0 0 2024-04-20T21:31:52.010 INFO:tasks.workunit.client.0.smithi062.stdout:9/187: getdents d16/d3d 0 2024-04-20T21:31:52.011 INFO:tasks.workunit.client.0.smithi062.stdout:1/186: mkdir dc/d15/d17/d36/d3a 0 2024-04-20T21:31:52.011 INFO:tasks.workunit.client.0.smithi062.stdout:2/151: write d7/da/f22 [1240406,97296] 0 2024-04-20T21:31:52.015 INFO:tasks.workunit.client.0.smithi062.stdout:8/229: dwrite d7/d9/d29/f40 [0,4194304] 0 2024-04-20T21:31:52.017 INFO:tasks.workunit.client.0.smithi062.stdout:9/188: stat d16/f3a 0 2024-04-20T21:31:52.018 INFO:tasks.workunit.client.0.smithi062.stdout:5/131: rename l0 to d8/d10/d2a/d2b/l30 0 2024-04-20T21:31:52.023 INFO:tasks.workunit.client.0.smithi062.stdout:9/189: chown d16/f3a 56018158 1 2024-04-20T21:31:52.023 INFO:tasks.workunit.client.0.smithi062.stdout:1/187: symlink dc/d15/d20/l3b 0 2024-04-20T21:31:52.023 INFO:tasks.workunit.client.0.smithi062.stdout:2/152: read d7/f1a [2301,12410] 0 2024-04-20T21:31:52.023 INFO:tasks.workunit.client.0.smithi062.stdout:8/230: dread d7/d9/f41 [0,4194304] 0 2024-04-20T21:31:52.027 INFO:tasks.workunit.client.0.smithi062.stdout:8/231: chown d7/d9/d1b/d24/l47 32158882 1 2024-04-20T21:31:52.027 INFO:tasks.workunit.client.0.smithi062.stdout:1/188: write dc/d15/d17/f34 [139274,81632] 0 2024-04-20T21:31:52.029 INFO:tasks.workunit.client.0.smithi062.stdout:8/232: write d7/f38 [411753,41722] 0 2024-04-20T21:31:52.034 INFO:tasks.workunit.client.0.smithi062.stdout:2/153: creat d7/da/d15/d1b/f2d x:0 0 0 2024-04-20T21:31:52.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:51 smithi062 ceph-mon[16221]: pgmap v10: 65 pgs: 65 active+clean; 226 MiB data, 881 MiB used, 536 GiB / 536 GiB avail; 1011 KiB/s rd, 2.9 MiB/s wr, 203 op/s 2024-04-20T21:31:52.036 INFO:tasks.workunit.client.0.smithi062.stdout:2/154: chown d7/da/d15/d1b/d24 277 1 2024-04-20T21:31:52.037 INFO:tasks.workunit.client.0.smithi062.stdout:9/190: symlink d16/d2f/l41 0 2024-04-20T21:31:52.037 INFO:tasks.workunit.client.0.smithi062.stdout:3/216: link d0/d1/d10/d31/l3d d0/d1/d3c/l50 0 2024-04-20T21:31:52.039 INFO:tasks.workunit.client.0.smithi062.stdout:8/233: symlink d7/d25/d4f/l52 0 2024-04-20T21:31:52.043 INFO:tasks.workunit.client.0.smithi062.stdout:3/217: chown d0/d1b/c36 0 1 2024-04-20T21:31:52.044 INFO:tasks.workunit.client.0.smithi062.stdout:9/191: mkdir d16/d2c/d42 0 2024-04-20T21:31:52.045 INFO:tasks.workunit.client.0.smithi062.stdout:5/132: dwrite f4 [0,4194304] 0 2024-04-20T21:31:52.046 INFO:tasks.workunit.client.0.smithi062.stdout:3/218: chown d0/f4c 10376 1 2024-04-20T21:31:52.049 INFO:tasks.workunit.client.0.smithi062.stdout:9/192: write f4 [1276698,102054] 0 2024-04-20T21:31:52.051 INFO:tasks.workunit.client.0.smithi062.stdout:3/219: fsync d0/f2f 0 2024-04-20T21:31:52.051 INFO:tasks.workunit.client.0.smithi062.stdout:1/189: dwrite f2 [4194304,4194304] 0 2024-04-20T21:31:52.057 INFO:tasks.workunit.client.0.smithi062.stdout:3/220: readlink d0/d11/l49 0 2024-04-20T21:31:52.058 INFO:tasks.workunit.client.0.smithi062.stdout:8/234: truncate d7/d9/d1b/f1e 2917064 0 2024-04-20T21:31:52.059 INFO:tasks.workunit.client.0.smithi062.stdout:9/193: truncate d16/d2f/f31 438615 0 2024-04-20T21:31:52.060 INFO:tasks.workunit.client.0.smithi062.stdout:1/190: creat dc/d15/d20/f3c x:0 0 0 2024-04-20T21:31:52.063 INFO:tasks.workunit.client.0.smithi062.stdout:2/155: dwrite d7/ff [4194304,4194304] 0 2024-04-20T21:31:52.070 INFO:tasks.workunit.client.0.smithi062.stdout:9/194: rename f13 to d16/d3d/f43 0 2024-04-20T21:31:52.071 INFO:tasks.workunit.client.0.smithi062.stdout:5/133: dwrite d8/d10/f26 [0,4194304] 0 2024-04-20T21:31:52.080 INFO:tasks.workunit.client.0.smithi062.stdout:1/191: link dc/d27/d32/l37 dc/d15/l3d 0 2024-04-20T21:31:52.084 INFO:tasks.workunit.client.0.smithi062.stdout:6/121: sync 2024-04-20T21:31:52.087 INFO:tasks.workunit.client.0.smithi062.stdout:1/192: creat dc/d15/d20/f3e x:0 0 0 2024-04-20T21:31:52.087 INFO:tasks.workunit.client.0.smithi062.stdout:2/156: dwrite d7/ff [0,4194304] 0 2024-04-20T21:31:52.090 INFO:tasks.workunit.client.0.smithi062.stdout:2/157: readlink d7/da/l21 0 2024-04-20T21:31:52.094 INFO:tasks.workunit.client.0.smithi062.stdout:6/122: dread d0/d9/f16 [0,4194304] 0 2024-04-20T21:31:52.095 INFO:tasks.workunit.client.0.smithi062.stdout:9/195: dwrite fd [0,4194304] 0 2024-04-20T21:31:52.097 INFO:tasks.workunit.client.0.smithi062.stdout:1/193: chown dc/d15/c2a 185558 1 2024-04-20T21:31:52.099 INFO:tasks.workunit.client.0.smithi062.stdout:2/158: chown d7/da/d15/d1b/f2c 867502884 1 2024-04-20T21:31:52.100 INFO:tasks.workunit.client.0.smithi062.stdout:5/134: dwrite d8/d1e/f23 [0,4194304] 0 2024-04-20T21:31:52.106 INFO:tasks.workunit.client.0.smithi062.stdout:2/159: creat d7/da/d15/d1b/f2e x:0 0 0 2024-04-20T21:31:52.107 INFO:tasks.workunit.client.0.smithi062.stdout:9/196: fdatasync d16/f32 0 2024-04-20T21:31:52.108 INFO:tasks.workunit.client.0.smithi062.stdout:2/160: dread - d7/da/d28/f2a zero size 2024-04-20T21:31:52.110 INFO:tasks.workunit.client.0.smithi062.stdout:9/197: chown d16/d2f/l41 197764 1 2024-04-20T21:31:52.113 INFO:tasks.workunit.client.0.smithi062.stdout:5/135: symlink d8/d10/d2a/d2b/l31 0 2024-04-20T21:31:52.115 INFO:tasks.workunit.client.0.smithi062.stdout:2/161: mkdir d7/da/d15/d2f 0 2024-04-20T21:31:52.117 INFO:tasks.workunit.client.0.smithi062.stdout:2/162: chown d7/da/d15/d2f 48455656 1 2024-04-20T21:31:52.118 INFO:tasks.workunit.client.0.smithi062.stdout:2/163: chown d7/da 1436 1 2024-04-20T21:31:52.119 INFO:tasks.workunit.client.0.smithi062.stdout:2/164: readlink d7/da/l10 0 2024-04-20T21:31:52.122 INFO:tasks.workunit.client.0.smithi062.stdout:5/136: dread d8/fe [4194304,4194304] 0 2024-04-20T21:31:52.123 INFO:tasks.workunit.client.0.smithi062.stdout:2/165: read - d7/da/d15/d1b/f2d zero size 2024-04-20T21:31:52.128 INFO:tasks.workunit.client.0.smithi062.stdout:2/166: creat d7/da/d15/d2f/f30 x:0 0 0 2024-04-20T21:31:52.131 INFO:tasks.workunit.client.0.smithi062.stdout:9/198: dread d16/f1c [0,4194304] 0 2024-04-20T21:31:52.140 INFO:tasks.workunit.client.0.smithi062.stdout:1/194: dread fb [0,4194304] 0 2024-04-20T21:31:52.146 INFO:tasks.workunit.client.0.smithi062.stdout:5/137: dread d8/f1c [0,4194304] 0 2024-04-20T21:31:52.147 INFO:tasks.workunit.client.0.smithi062.stdout:2/167: dwrite d7/f17 [0,4194304] 0 2024-04-20T21:31:52.149 INFO:tasks.workunit.client.0.smithi062.stdout:5/138: write d8/d1e/f23 [150998,33461] 0 2024-04-20T21:31:52.151 INFO:tasks.workunit.client.0.smithi062.stdout:6/123: dread d0/fe [0,4194304] 0 2024-04-20T21:31:52.152 INFO:tasks.workunit.client.0.smithi062.stdout:5/139: chown d8/fe 186 1 2024-04-20T21:31:52.153 INFO:tasks.workunit.client.0.smithi062.stdout:2/168: symlink d7/da/d15/l31 0 2024-04-20T21:31:52.155 INFO:tasks.workunit.client.0.smithi062.stdout:6/124: creat d0/d1a/f29 x:0 0 0 2024-04-20T21:31:52.157 INFO:tasks.workunit.client.0.smithi062.stdout:1/195: truncate dc/d15/d20/d28/f2f 361607 0 2024-04-20T21:31:52.158 INFO:tasks.workunit.client.0.smithi062.stdout:2/169: unlink d7/da/d15/d1b/f2d 0 2024-04-20T21:31:52.163 INFO:tasks.workunit.client.0.smithi062.stdout:5/140: mkdir d8/d10/d32 0 2024-04-20T21:31:52.163 INFO:tasks.workunit.client.0.smithi062.stdout:2/170: mknod d7/da/d15/d2f/c32 0 2024-04-20T21:31:52.163 INFO:tasks.workunit.client.0.smithi062.stdout:1/196: readlink dc/d15/l3d 0 2024-04-20T21:31:52.164 INFO:tasks.workunit.client.0.smithi062.stdout:5/141: mknod d8/d10/d2a/d2f/c33 0 2024-04-20T21:31:52.167 INFO:tasks.workunit.client.0.smithi062.stdout:5/142: creat d8/d1e/d2c/f34 x:0 0 0 2024-04-20T21:31:52.170 INFO:tasks.workunit.client.0.smithi062.stdout:5/143: truncate d8/fb 1603833 0 2024-04-20T21:31:52.175 INFO:tasks.workunit.client.0.smithi062.stdout:6/125: dwrite d0/d9/d11/f25 [0,4194304] 0 2024-04-20T21:31:52.177 INFO:tasks.workunit.client.0.smithi062.stdout:6/126: readlink d0/d9/l20 0 2024-04-20T21:31:52.178 INFO:tasks.workunit.client.0.smithi062.stdout:2/171: getdents d7/da/d15/d2f 0 2024-04-20T21:31:52.178 INFO:tasks.workunit.client.0.smithi062.stdout:1/197: dwrite f1 [0,4194304] 0 2024-04-20T21:31:52.180 INFO:tasks.workunit.client.0.smithi062.stdout:1/198: rename dc/d15 to dc/d15/d3f 22 2024-04-20T21:31:52.180 INFO:tasks.workunit.client.0.smithi062.stdout:6/127: symlink d0/d9/d11/l2a 0 2024-04-20T21:31:52.181 INFO:tasks.workunit.client.0.smithi062.stdout:5/144: link d8/d10/c18 d8/d10/d2a/c35 0 2024-04-20T21:31:52.181 INFO:tasks.workunit.client.0.smithi062.stdout:2/172: creat d7/da/d15/d2f/f33 x:0 0 0 2024-04-20T21:31:52.182 INFO:tasks.workunit.client.0.smithi062.stdout:6/128: fdatasync d0/d1a/f29 0 2024-04-20T21:31:52.184 INFO:tasks.workunit.client.0.smithi062.stdout:5/145: unlink d8/ca 0 2024-04-20T21:31:52.184 INFO:tasks.workunit.client.0.smithi062.stdout:2/173: dread - d7/da/d15/d1b/f2c zero size 2024-04-20T21:31:52.188 INFO:tasks.workunit.client.0.smithi062.stdout:6/129: symlink d0/l2b 0 2024-04-20T21:31:52.192 INFO:tasks.workunit.client.0.smithi062.stdout:2/174: write d7/da/d15/f23 [246840,39147] 0 2024-04-20T21:31:52.196 INFO:tasks.workunit.client.0.smithi062.stdout:2/175: read - d7/da/d15/d1b/f2c zero size 2024-04-20T21:31:52.197 INFO:tasks.workunit.client.0.smithi062.stdout:1/199: truncate dc/d15/d20/d28/f2f 1244106 0 2024-04-20T21:31:52.200 INFO:tasks.workunit.client.0.smithi062.stdout:2/176: creat d7/da/d15/d1b/f34 x:0 0 0 2024-04-20T21:31:52.202 INFO:tasks.workunit.client.0.smithi062.stdout:2/177: write d7/f1e [4103940,76508] 0 2024-04-20T21:31:52.202 INFO:tasks.workunit.client.0.smithi062.stdout:5/146: dread d8/f1a [0,4194304] 0 2024-04-20T21:31:52.204 INFO:tasks.workunit.client.0.smithi062.stdout:2/178: fsync d7/da/f22 0 2024-04-20T21:31:52.206 INFO:tasks.workunit.client.0.smithi062.stdout:5/147: creat d8/d10/d2a/d2b/f36 x:0 0 0 2024-04-20T21:31:52.208 INFO:tasks.workunit.client.0.smithi062.stdout:6/130: dread d0/fe [4194304,4194304] 0 2024-04-20T21:31:52.208 INFO:tasks.workunit.client.0.smithi062.stdout:5/148: readlink d8/d10/l29 0 2024-04-20T21:31:52.209 INFO:tasks.workunit.client.0.smithi062.stdout:2/179: rename d7/da/l21 to d7/da/d15/d1b/d24/l35 0 2024-04-20T21:31:52.213 INFO:tasks.workunit.client.0.smithi062.stdout:6/131: truncate d0/f10 1795163 0 2024-04-20T21:31:52.214 INFO:tasks.workunit.client.0.smithi062.stdout:2/180: unlink d7/ce 0 2024-04-20T21:31:52.215 INFO:tasks.workunit.client.0.smithi062.stdout:1/200: dwrite fb [4194304,4194304] 0 2024-04-20T21:31:52.216 INFO:tasks.workunit.client.0.smithi062.stdout:5/149: link d8/d1e/c2d d8/d10/d2a/d2f/c37 0 2024-04-20T21:31:52.216 INFO:tasks.workunit.client.0.smithi062.stdout:2/181: chown d7/da/l10 6567927 1 2024-04-20T21:31:52.217 INFO:tasks.workunit.client.0.smithi062.stdout:6/132: mkdir d0/d2c 0 2024-04-20T21:31:52.220 INFO:tasks.workunit.client.0.smithi062.stdout:5/150: mkdir d8/d38 0 2024-04-20T21:31:52.226 INFO:tasks.workunit.client.0.smithi062.stdout:6/133: truncate d0/f17 751005 0 2024-04-20T21:31:52.229 INFO:tasks.workunit.client.0.smithi062.stdout:6/134: mknod d0/d1a/c2d 0 2024-04-20T21:31:52.233 INFO:tasks.workunit.client.0.smithi062.stdout:2/182: dwrite d7/f17 [0,4194304] 0 2024-04-20T21:31:52.233 INFO:tasks.workunit.client.0.smithi062.stdout:5/151: dread d8/d1e/f24 [0,4194304] 0 2024-04-20T21:31:52.235 INFO:tasks.workunit.client.0.smithi062.stdout:5/152: truncate d8/d10/f17 5412090 0 2024-04-20T21:31:52.240 INFO:tasks.workunit.client.0.smithi062.stdout:2/183: rmdir d7/da 39 2024-04-20T21:31:52.242 INFO:tasks.workunit.client.0.smithi062.stdout:6/135: creat d0/d2c/f2e x:0 0 0 2024-04-20T21:31:52.243 INFO:tasks.workunit.client.0.smithi062.stdout:5/153: readlink d8/l27 0 2024-04-20T21:31:52.245 INFO:tasks.workunit.client.0.smithi062.stdout:1/201: write dc/fe [2531113,38547] 0 2024-04-20T21:31:52.247 INFO:tasks.workunit.client.0.smithi062.stdout:5/154: rmdir d8/d10/d2a/d2f 39 2024-04-20T21:31:52.249 INFO:tasks.workunit.client.0.smithi062.stdout:6/136: mknod d0/c2f 0 2024-04-20T21:31:52.252 INFO:tasks.workunit.client.0.smithi062.stdout:1/202: mknod dc/d2d/c40 0 2024-04-20T21:31:52.253 INFO:tasks.workunit.client.0.smithi062.stdout:1/203: fsync f1 0 2024-04-20T21:31:52.254 INFO:tasks.workunit.client.0.smithi062.stdout:7/126: sync 2024-04-20T21:31:52.256 INFO:tasks.workunit.client.0.smithi062.stdout:5/155: rename d8/fe to d8/d10/d2a/f39 0 2024-04-20T21:31:52.256 INFO:tasks.workunit.client.0.smithi062.stdout:1/204: read dc/fe [3158611,114478] 0 2024-04-20T21:31:52.260 INFO:tasks.workunit.client.0.smithi062.stdout:7/127: creat d7/f27 x:0 0 0 2024-04-20T21:31:52.260 INFO:tasks.workunit.client.0.smithi062.stdout:2/184: dwrite d7/da/d15/d1b/f2e [0,4194304] 0 2024-04-20T21:31:52.265 INFO:tasks.workunit.client.0.smithi062.stdout:2/185: read - d7/da/d15/d1b/f2c zero size 2024-04-20T21:31:52.270 INFO:tasks.workunit.client.0.smithi062.stdout:6/137: truncate d0/d9/d11/f25 175681 0 2024-04-20T21:31:52.272 INFO:tasks.workunit.client.0.smithi062.stdout:6/138: dread - d0/d9/d11/d22/f27 zero size 2024-04-20T21:31:52.275 INFO:tasks.workunit.client.0.smithi062.stdout:5/156: truncate d8/d10/d2a/f39 2494256 0 2024-04-20T21:31:52.277 INFO:tasks.workunit.client.0.smithi062.stdout:6/139: readlink d0/d9/d11/l2a 0 2024-04-20T21:31:52.281 INFO:tasks.workunit.client.0.smithi062.stdout:6/140: fsync d0/f6 0 2024-04-20T21:31:52.285 INFO:tasks.workunit.client.0.smithi062.stdout:2/186: dread d7/da/f22 [0,4194304] 0 2024-04-20T21:31:52.285 INFO:tasks.workunit.client.0.smithi062.stdout:7/128: dwrite d7/f14 [0,4194304] 0 2024-04-20T21:31:52.287 INFO:tasks.workunit.client.0.smithi062.stdout:1/205: dwrite dc/f14 [4194304,4194304] 0 2024-04-20T21:31:52.288 INFO:tasks.workunit.client.0.smithi062.stdout:7/129: chown d7/d19/d1c 105 1 2024-04-20T21:31:52.288 INFO:tasks.workunit.client.0.smithi062.stdout:6/141: rmdir d0 39 2024-04-20T21:31:52.290 INFO:tasks.workunit.client.0.smithi062.stdout:5/157: getdents d8/d10 0 2024-04-20T21:31:52.294 INFO:tasks.workunit.client.0.smithi062.stdout:6/142: creat d0/d2c/f30 x:0 0 0 2024-04-20T21:31:52.295 INFO:tasks.workunit.client.0.smithi062.stdout:2/187: fsync d7/da/d15/d1b/f2e 0 2024-04-20T21:31:52.296 INFO:tasks.workunit.client.0.smithi062.stdout:1/206: mknod dc/c41 0 2024-04-20T21:31:52.298 INFO:tasks.workunit.client.0.smithi062.stdout:7/130: creat d7/d19/d1c/f28 x:0 0 0 2024-04-20T21:31:52.299 INFO:tasks.workunit.client.0.smithi062.stdout:7/131: readlink d7/l1b 0 2024-04-20T21:31:52.300 INFO:tasks.workunit.client.0.smithi062.stdout:2/188: mknod d7/da/d15/d2f/c36 0 2024-04-20T21:31:52.303 INFO:tasks.workunit.client.0.smithi062.stdout:2/189: fdatasync d7/da/d15/d1b/f2c 0 2024-04-20T21:31:52.304 INFO:tasks.workunit.client.0.smithi062.stdout:6/143: link d0/l2b d0/d1a/l31 0 2024-04-20T21:31:52.306 INFO:tasks.workunit.client.0.smithi062.stdout:7/132: creat d7/d26/f29 x:0 0 0 2024-04-20T21:31:52.309 INFO:tasks.workunit.client.0.smithi062.stdout:2/190: rename d7/da/l10 to d7/da/d15/d2f/l37 0 2024-04-20T21:31:52.310 INFO:tasks.workunit.client.0.smithi062.stdout:5/158: dread d8/d1e/f25 [0,4194304] 0 2024-04-20T21:31:52.311 INFO:tasks.workunit.client.0.smithi062.stdout:7/133: write d7/d26/f29 [414785,25921] 0 2024-04-20T21:31:52.312 INFO:tasks.workunit.client.0.smithi062.stdout:6/144: mknod d0/d9/d11/d22/d26/c32 0 2024-04-20T21:31:52.314 INFO:tasks.workunit.client.0.smithi062.stdout:2/191: mkdir d7/da/d15/d2f/d38 0 2024-04-20T21:31:52.315 INFO:tasks.workunit.client.0.smithi062.stdout:7/134: creat d7/f2a x:0 0 0 2024-04-20T21:31:52.318 INFO:tasks.workunit.client.0.smithi062.stdout:2/192: mkdir d7/d39 0 2024-04-20T21:31:52.320 INFO:tasks.workunit.client.0.smithi062.stdout:7/135: chown d7/d19/f1a 98320 1 2024-04-20T21:31:52.320 INFO:tasks.workunit.client.0.smithi062.stdout:1/207: dwrite dc/d15/d17/f23 [0,4194304] 0 2024-04-20T21:31:52.320 INFO:tasks.workunit.client.0.smithi062.stdout:5/159: getdents d8/d1e 0 2024-04-20T21:31:52.323 INFO:tasks.workunit.client.0.smithi062.stdout:2/193: symlink d7/da/d15/d1b/l3a 0 2024-04-20T21:31:52.325 INFO:tasks.workunit.client.0.smithi062.stdout:2/194: write d7/ff [6999861,59137] 0 2024-04-20T21:31:52.326 INFO:tasks.workunit.client.0.smithi062.stdout:7/136: creat d7/d19/d1c/f2b x:0 0 0 2024-04-20T21:31:52.328 INFO:tasks.workunit.client.0.smithi062.stdout:1/208: rename f8 to dc/d15/f42 0 2024-04-20T21:31:52.329 INFO:tasks.workunit.client.0.smithi062.stdout:7/137: read - d7/d19/d1c/f28 zero size 2024-04-20T21:31:52.331 INFO:tasks.workunit.client.0.smithi062.stdout:7/138: write d7/fe [3959584,97717] 0 2024-04-20T21:31:52.334 INFO:tasks.workunit.client.0.smithi062.stdout:1/209: write dc/f21 [2491745,33944] 0 2024-04-20T21:31:52.334 INFO:tasks.workunit.client.0.smithi062.stdout:6/145: dread d0/d9/f15 [0,4194304] 0 2024-04-20T21:31:52.334 INFO:tasks.workunit.client.0.smithi062.stdout:7/139: write d7/f27 [76057,55543] 0 2024-04-20T21:31:52.339 INFO:tasks.workunit.client.0.smithi062.stdout:1/210: chown fb 806383080 1 2024-04-20T21:31:52.339 INFO:tasks.workunit.client.0.smithi062.stdout:7/140: write f6 [1065347,34601] 0 2024-04-20T21:31:52.342 INFO:tasks.workunit.client.0.smithi062.stdout:5/160: dread d8/d1e/f23 [0,4194304] 0 2024-04-20T21:31:52.342 INFO:tasks.workunit.client.0.smithi062.stdout:7/141: readlink d7/l1b 0 2024-04-20T21:31:52.343 INFO:tasks.workunit.client.0.smithi062.stdout:5/161: creat d8/d1e/d2c/f3a x:0 0 0 2024-04-20T21:31:52.343 INFO:tasks.workunit.client.0.smithi062.stdout:1/211: creat dc/d15/d17/d36/d3a/f43 x:0 0 0 2024-04-20T21:31:52.344 INFO:tasks.workunit.client.0.smithi062.stdout:7/142: chown d7/d19/d1c/f2b 1454476 1 2024-04-20T21:31:52.344 INFO:tasks.workunit.client.0.smithi062.stdout:6/146: rename d0/f10 to d0/d9/d11/d22/f33 0 2024-04-20T21:31:52.350 INFO:tasks.workunit.client.0.smithi062.stdout:6/147: link d0/d9/d11/l1e d0/d2c/l34 0 2024-04-20T21:31:52.354 INFO:tasks.workunit.client.0.smithi062.stdout:6/148: mknod d0/d9/d11/c35 0 2024-04-20T21:31:52.356 INFO:tasks.workunit.client.0.smithi062.stdout:1/212: dread dc/f14 [4194304,4194304] 0 2024-04-20T21:31:52.358 INFO:tasks.workunit.client.0.smithi062.stdout:1/213: chown dc/d15/d17/c24 2 1 2024-04-20T21:31:52.360 INFO:tasks.workunit.client.0.smithi062.stdout:6/149: unlink d0/d9/d11/l1e 0 2024-04-20T21:31:52.360 INFO:tasks.workunit.client.0.smithi062.stdout:1/214: readlink dc/d15/l3d 0 2024-04-20T21:31:52.361 INFO:tasks.workunit.client.0.smithi062.stdout:5/162: dread d8/d10/d2a/f39 [0,4194304] 0 2024-04-20T21:31:52.361 INFO:tasks.workunit.client.0.smithi062.stdout:7/143: dwrite d7/d19/d1c/f28 [0,4194304] 0 2024-04-20T21:31:52.363 INFO:tasks.workunit.client.0.smithi062.stdout:7/144: chown d7/f2a 79 1 2024-04-20T21:31:52.364 INFO:tasks.workunit.client.0.smithi062.stdout:1/215: mknod dc/c44 0 2024-04-20T21:31:52.364 INFO:tasks.workunit.client.0.smithi062.stdout:6/150: write d0/d9/d11/d22/f27 [192103,42251] 0 2024-04-20T21:31:52.367 INFO:tasks.workunit.client.0.smithi062.stdout:7/145: write d7/d19/f1a [2430786,124282] 0 2024-04-20T21:31:52.367 INFO:tasks.workunit.client.0.smithi062.stdout:6/151: readlink d0/l2b 0 2024-04-20T21:31:52.369 INFO:tasks.workunit.client.0.smithi062.stdout:1/216: creat dc/d15/d20/f45 x:0 0 0 2024-04-20T21:31:52.373 INFO:tasks.workunit.client.0.smithi062.stdout:7/146: creat d7/d26/f2c x:0 0 0 2024-04-20T21:31:52.373 INFO:tasks.workunit.client.0.smithi062.stdout:5/163: dread d8/d10/d2a/f39 [0,4194304] 0 2024-04-20T21:31:52.377 INFO:tasks.workunit.client.0.smithi062.stdout:5/164: readlink d8/d10/l15 0 2024-04-20T21:31:52.379 INFO:tasks.workunit.client.0.smithi062.stdout:1/217: rename dc/d27/f29 to dc/d15/d17/d36/d3a/f46 0 2024-04-20T21:31:52.383 INFO:tasks.workunit.client.0.smithi062.stdout:1/218: write dc/d15/f42 [2341550,100219] 0 2024-04-20T21:31:52.387 INFO:tasks.workunit.client.0.smithi062.stdout:7/147: rmdir d7/d26 39 2024-04-20T21:31:52.387 INFO:tasks.workunit.client.0.smithi062.stdout:7/148: chown d7 149 1 2024-04-20T21:31:52.389 INFO:tasks.workunit.client.0.smithi062.stdout:6/152: dwrite d0/d9/f16 [0,4194304] 0 2024-04-20T21:31:52.390 INFO:tasks.workunit.client.0.smithi062.stdout:6/153: readlink d0/d9/l12 0 2024-04-20T21:31:52.393 INFO:tasks.workunit.client.0.smithi062.stdout:1/219: mknod dc/d15/d20/c47 0 2024-04-20T21:31:52.398 INFO:tasks.workunit.client.0.smithi062.stdout:5/165: write d8/fb [1961446,125113] 0 2024-04-20T21:31:52.401 INFO:tasks.workunit.client.0.smithi062.stdout:1/220: creat dc/d15/f48 x:0 0 0 2024-04-20T21:31:52.405 INFO:tasks.workunit.client.0.smithi062.stdout:7/149: dwrite d7/f14 [0,4194304] 0 2024-04-20T21:31:52.406 INFO:tasks.workunit.client.0.smithi062.stdout:7/150: stat d7/cc 0 2024-04-20T21:31:52.409 INFO:tasks.workunit.client.0.smithi062.stdout:1/221: fdatasync dc/d15/d17/d36/d3a/f46 0 2024-04-20T21:31:52.410 INFO:tasks.workunit.client.0.smithi062.stdout:5/166: link d8/d10/l29 d8/d10/d2a/d2b/l3b 0 2024-04-20T21:31:52.414 INFO:tasks.workunit.client.0.smithi062.stdout:5/167: write d8/d10/d2a/d2b/f36 [503054,41184] 0 2024-04-20T21:31:52.415 INFO:tasks.workunit.client.0.smithi062.stdout:1/222: creat dc/d15/d20/d28/f49 x:0 0 0 2024-04-20T21:31:52.416 INFO:tasks.workunit.client.0.smithi062.stdout:7/151: creat d7/f2d x:0 0 0 2024-04-20T21:31:52.416 INFO:tasks.workunit.client.0.smithi062.stdout:5/168: rmdir d8/d1e 39 2024-04-20T21:31:52.417 INFO:tasks.workunit.client.0.smithi062.stdout:1/223: readlink dc/d15/l3d 0 2024-04-20T21:31:52.419 INFO:tasks.workunit.client.0.smithi062.stdout:6/154: dwrite d0/f3 [0,4194304] 0 2024-04-20T21:31:52.419 INFO:tasks.workunit.client.0.smithi062.stdout:7/152: dread f4 [4194304,4194304] 0 2024-04-20T21:31:52.420 INFO:tasks.workunit.client.0.smithi062.stdout:5/169: chown d8/d1e/d2c/f34 227 1 2024-04-20T21:31:52.420 INFO:tasks.workunit.client.0.smithi062.stdout:1/224: fsync dc/d15/d20/f33 0 2024-04-20T21:31:52.422 INFO:tasks.workunit.client.0.smithi062.stdout:5/170: fdatasync d8/d10/d2a/d2b/f36 0 2024-04-20T21:31:52.423 INFO:tasks.workunit.client.0.smithi062.stdout:1/225: write dc/d15/d17/f34 [3661899,22635] 0 2024-04-20T21:31:52.425 INFO:tasks.workunit.client.0.smithi062.stdout:5/171: write f4 [2507309,89103] 0 2024-04-20T21:31:52.432 INFO:tasks.workunit.client.0.smithi062.stdout:1/226: write dc/fe [4378080,68050] 0 2024-04-20T21:31:52.433 INFO:tasks.workunit.client.0.smithi062.stdout:5/172: link d8/d10/f28 d8/d10/d32/f3c 0 2024-04-20T21:31:52.440 INFO:tasks.workunit.client.0.smithi062.stdout:4/169: sync 2024-04-20T21:31:52.440 INFO:tasks.workunit.client.0.smithi062.stdout:5/173: creat d8/d38/f3d x:0 0 0 2024-04-20T21:31:52.441 INFO:tasks.workunit.client.0.smithi062.stdout:1/227: rename dc/d15/c2a to dc/c4a 0 2024-04-20T21:31:52.442 INFO:tasks.workunit.client.0.smithi062.stdout:6/155: dwrite d0/d9/d11/f1f [0,4194304] 0 2024-04-20T21:31:52.444 INFO:tasks.workunit.client.0.smithi062.stdout:6/156: read - d0/d1a/f29 zero size 2024-04-20T21:31:52.454 INFO:tasks.workunit.client.0.smithi062.stdout:5/174: write d8/ff [1361835,54287] 0 2024-04-20T21:31:52.457 INFO:tasks.workunit.client.0.smithi062.stdout:1/228: truncate f9 11630 0 2024-04-20T21:31:52.461 INFO:tasks.workunit.client.0.smithi062.stdout:1/229: mknod dc/c4b 0 2024-04-20T21:31:52.463 INFO:tasks.workunit.client.0.smithi062.stdout:4/170: dwrite da/dd/df/d27/f31 [4194304,4194304] 0 2024-04-20T21:31:52.463 INFO:tasks.workunit.client.0.smithi062.stdout:1/230: read dc/d15/d17/f23 [4136113,4466] 0 2024-04-20T21:31:52.473 INFO:tasks.workunit.client.0.smithi062.stdout:1/231: creat dc/d15/d17/f4c x:0 0 0 2024-04-20T21:31:52.476 INFO:tasks.workunit.client.0.smithi062.stdout:6/157: dread d0/d9/d11/d22/f33 [0,4194304] 0 2024-04-20T21:31:52.476 INFO:tasks.workunit.client.0.smithi062.stdout:4/171: getdents da/dd/df/d27 0 2024-04-20T21:31:52.479 INFO:tasks.workunit.client.0.smithi062.stdout:5/175: dwrite d8/d1e/f25 [0,4194304] 0 2024-04-20T21:31:52.485 INFO:tasks.workunit.client.0.smithi062.stdout:4/172: getdents da/dd/df 0 2024-04-20T21:31:52.485 INFO:tasks.workunit.client.0.smithi062.stdout:5/176: readlink d8/d10/l29 0 2024-04-20T21:31:52.486 INFO:tasks.workunit.client.0.smithi062.stdout:6/158: rename d0/d9/d11/l21 to d0/d9/d11/d22/d26/l36 0 2024-04-20T21:31:52.488 INFO:tasks.workunit.client.0.smithi062.stdout:1/232: truncate dc/d15/f16 699775 0 2024-04-20T21:31:52.491 INFO:tasks.workunit.client.0.smithi062.stdout:4/173: unlink da/dd/d15/f1d 0 2024-04-20T21:31:52.494 INFO:tasks.workunit.client.0.smithi062.stdout:1/233: unlink dc/d15/d17/f23 0 2024-04-20T21:31:52.494 INFO:tasks.workunit.client.0.smithi062.stdout:4/174: rmdir da/dd/df/d27 39 2024-04-20T21:31:52.496 INFO:tasks.workunit.client.0.smithi062.stdout:4/175: write da/f2f [973138,93210] 0 2024-04-20T21:31:52.497 INFO:tasks.workunit.client.0.smithi062.stdout:5/177: write d8/d10/f28 [4990778,19839] 0 2024-04-20T21:31:52.499 INFO:tasks.workunit.client.0.smithi062.stdout:1/234: creat dc/d15/d17/d36/d3a/f4d x:0 0 0 2024-04-20T21:31:52.500 INFO:tasks.workunit.client.0.smithi062.stdout:6/159: write d0/d9/d11/d22/f24 [475955,95366] 0 2024-04-20T21:31:52.501 INFO:tasks.workunit.client.0.smithi062.stdout:5/178: chown d8/d10/f1b 54 1 2024-04-20T21:31:52.506 INFO:tasks.workunit.client.0.smithi062.stdout:5/179: chown d8/d10/d2a/d2f/c33 27323 1 2024-04-20T21:31:52.508 INFO:tasks.workunit.client.0.smithi062.stdout:1/235: link dc/fe dc/d15/d20/d28/f4e 0 2024-04-20T21:31:52.512 INFO:tasks.workunit.client.0.smithi062.stdout:5/180: mkdir d8/d10/d3e 0 2024-04-20T21:31:52.514 INFO:tasks.workunit.client.0.smithi062.stdout:5/181: chown d8/d10/c21 3 1 2024-04-20T21:31:52.515 INFO:tasks.workunit.client.0.smithi062.stdout:5/182: fsync d8/fb 0 2024-04-20T21:31:52.516 INFO:tasks.workunit.client.0.smithi062.stdout:4/176: dwrite da/dd/df/f1a [0,4194304] 0 2024-04-20T21:31:52.517 INFO:tasks.workunit.client.0.smithi062.stdout:5/183: rename d8/d10 to d8/d10/d2a/d2f/d3f 22 2024-04-20T21:31:52.518 INFO:tasks.workunit.client.0.smithi062.stdout:4/177: dread - da/dd/df/f28 zero size 2024-04-20T21:31:52.520 INFO:tasks.workunit.client.0.smithi062.stdout:4/178: read - da/dd/df/f28 zero size 2024-04-20T21:31:52.522 INFO:tasks.workunit.client.0.smithi062.stdout:4/179: chown da/dd/f12 3 1 2024-04-20T21:31:52.524 INFO:tasks.workunit.client.0.smithi062.stdout:5/184: rename d8/d1e/f23 to d8/d10/d2a/d2b/f40 0 2024-04-20T21:31:52.529 INFO:tasks.workunit.client.0.smithi062.stdout:6/160: getdents d0/d2c 0 2024-04-20T21:31:52.530 INFO:tasks.workunit.client.0.smithi062.stdout:5/185: mknod d8/d1e/d2c/c41 0 2024-04-20T21:31:52.532 INFO:tasks.workunit.client.0.smithi062.stdout:4/180: dread da/dd/f35 [0,4194304] 0 2024-04-20T21:31:52.532 INFO:tasks.workunit.client.0.smithi062.stdout:1/236: rename dc/f21 to dc/d15/d17/d36/d3a/f4f 0 2024-04-20T21:31:52.534 INFO:tasks.workunit.client.0.smithi062.stdout:5/186: symlink d8/d10/l42 0 2024-04-20T21:31:52.538 INFO:tasks.workunit.client.0.smithi062.stdout:9/199: sync 2024-04-20T21:31:52.538 INFO:tasks.workunit.client.0.smithi062.stdout:0/209: sync 2024-04-20T21:31:52.538 INFO:tasks.workunit.client.0.smithi062.stdout:1/237: rmdir dc/d15/d20 39 2024-04-20T21:31:52.538 INFO:tasks.workunit.client.0.smithi062.stdout:3/221: sync 2024-04-20T21:31:52.538 INFO:tasks.workunit.client.0.smithi062.stdout:8/235: sync 2024-04-20T21:31:52.540 INFO:tasks.workunit.client.0.smithi062.stdout:3/222: dread - d0/f4f zero size 2024-04-20T21:31:52.542 INFO:tasks.workunit.client.0.smithi062.stdout:6/161: getdents d0/d9/d11 0 2024-04-20T21:31:52.544 INFO:tasks.workunit.client.0.smithi062.stdout:1/238: unlink dc/c4b 0 2024-04-20T21:31:52.545 INFO:tasks.workunit.client.0.smithi062.stdout:5/187: symlink d8/d10/d3e/l43 0 2024-04-20T21:31:52.548 INFO:tasks.workunit.client.0.smithi062.stdout:8/236: symlink d7/d30/l53 0 2024-04-20T21:31:52.550 INFO:tasks.workunit.client.0.smithi062.stdout:6/162: rename d0/d9/l20 to d0/d9/d11/d22/l37 0 2024-04-20T21:31:52.555 INFO:tasks.workunit.client.0.smithi062.stdout:8/237: symlink d7/l54 0 2024-04-20T21:31:52.557 INFO:tasks.workunit.client.0.smithi062.stdout:6/163: fsync d0/d9/d11/f25 0 2024-04-20T21:31:52.559 INFO:tasks.workunit.client.0.smithi062.stdout:8/238: chown d7/d9/d29/f40 426424 1 2024-04-20T21:31:52.560 INFO:tasks.workunit.client.0.smithi062.stdout:5/188: dread d8/d1e/f24 [0,4194304] 0 2024-04-20T21:31:52.561 INFO:tasks.workunit.client.0.smithi062.stdout:0/210: truncate d7/d1c/f32 625652 0 2024-04-20T21:31:52.562 INFO:tasks.workunit.client.0.smithi062.stdout:4/181: dwrite da/dd/df/d27/f31 [4194304,4194304] 0 2024-04-20T21:31:52.563 INFO:tasks.workunit.client.0.smithi062.stdout:9/200: truncate fc 8188685 0 2024-04-20T21:31:52.567 INFO:tasks.workunit.client.0.smithi062.stdout:3/223: dwrite d0/d1/f29 [0,4194304] 0 2024-04-20T21:31:52.569 INFO:tasks.workunit.client.0.smithi062.stdout:6/164: write d0/fe [6444793,60349] 0 2024-04-20T21:31:52.570 INFO:tasks.workunit.client.0.smithi062.stdout:5/189: chown d8/d1e/f25 3 1 2024-04-20T21:31:52.572 INFO:tasks.workunit.client.0.smithi062.stdout:0/211: unlink d7/d1e/l2c 0 2024-04-20T21:31:52.574 INFO:tasks.workunit.client.0.smithi062.stdout:4/182: read - da/dd/df/f28 zero size 2024-04-20T21:31:52.574 INFO:tasks.workunit.client.0.smithi062.stdout:5/190: chown d8/d10/c21 203857 1 2024-04-20T21:31:52.580 INFO:tasks.workunit.client.0.smithi062.stdout:7/153: sync 2024-04-20T21:31:52.580 INFO:tasks.workunit.client.0.smithi062.stdout:2/195: sync 2024-04-20T21:31:52.581 INFO:tasks.workunit.client.0.smithi062.stdout:3/224: readlink d0/d1/d10/d31/l37 0 2024-04-20T21:31:52.588 INFO:tasks.workunit.client.0.smithi062.stdout:2/196: chown d7/da/d15/l31 1 1 2024-04-20T21:31:52.588 INFO:tasks.workunit.client.0.smithi062.stdout:7/154: stat d7/d26/f2c 0 2024-04-20T21:31:52.589 INFO:tasks.workunit.client.0.smithi062.stdout:1/239: dwrite dc/d15/d17/d36/d3a/f4f [8388608,4194304] 0 2024-04-20T21:31:52.589 INFO:tasks.workunit.client.0.smithi062.stdout:0/212: creat d7/f47 x:0 0 0 2024-04-20T21:31:52.590 INFO:tasks.workunit.client.0.smithi062.stdout:4/183: mkdir da/dd/df/d27/d30/d3d 0 2024-04-20T21:31:52.592 INFO:tasks.workunit.client.0.smithi062.stdout:9/201: getdents d16/d3e 0 2024-04-20T21:31:52.593 INFO:tasks.workunit.client.0.smithi062.stdout:5/191: creat d8/f44 x:0 0 0 2024-04-20T21:31:52.596 INFO:tasks.workunit.client.0.smithi062.stdout:0/213: chown f1 2 1 2024-04-20T21:31:52.597 INFO:tasks.workunit.client.0.smithi062.stdout:8/239: dwrite d7/f38 [0,4194304] 0 2024-04-20T21:31:52.598 INFO:tasks.workunit.client.0.smithi062.stdout:3/225: truncate d0/d1/d10/f47 178427 0 2024-04-20T21:31:52.602 INFO:tasks.workunit.client.0.smithi062.stdout:1/240: write dc/d15/d17/f19 [1020149,46447] 0 2024-04-20T21:31:52.602 INFO:tasks.workunit.client.0.smithi062.stdout:7/155: creat d7/d19/d1c/f2e x:0 0 0 2024-04-20T21:31:52.611 INFO:tasks.workunit.client.0.smithi062.stdout:9/202: fdatasync fb 0 2024-04-20T21:31:52.614 INFO:tasks.workunit.client.0.smithi062.stdout:3/226: rename d0/d1/d10/f3f to d0/d1c/f51 0 2024-04-20T21:31:52.619 INFO:tasks.workunit.client.0.smithi062.stdout:2/197: dwrite d7/f12 [0,4194304] 0 2024-04-20T21:31:52.619 INFO:tasks.workunit.client.0.smithi062.stdout:3/227: write d0/d1/f20 [2145850,66682] 0 2024-04-20T21:31:52.620 INFO:tasks.workunit.client.0.smithi062.stdout:7/156: creat d7/d26/f2f x:0 0 0 2024-04-20T21:31:52.620 INFO:tasks.workunit.client.0.smithi062.stdout:4/184: dwrite da/dd/f19 [0,4194304] 0 2024-04-20T21:31:52.624 INFO:tasks.workunit.client.0.smithi062.stdout:3/228: dread - d0/d1b/f4d zero size 2024-04-20T21:31:52.624 INFO:tasks.workunit.client.0.smithi062.stdout:2/198: chown d7/da/d28/f2a 0 1 2024-04-20T21:31:52.626 INFO:tasks.workunit.client.0.smithi062.stdout:6/165: link d0/d9/d11/d22/l37 d0/d9/d11/d22/l38 0 2024-04-20T21:31:52.626 INFO:tasks.workunit.client.0.smithi062.stdout:0/214: symlink d7/d1c/d15/d17/d26/l48 0 2024-04-20T21:31:52.627 INFO:tasks.workunit.client.0.smithi062.stdout:5/192: write d8/f1c [2784138,36090] 0 2024-04-20T21:31:52.631 INFO:tasks.workunit.client.0.smithi062.stdout:5/193: chown d8/d10/l29 58716409 1 2024-04-20T21:31:52.633 INFO:tasks.workunit.client.0.smithi062.stdout:6/166: write d0/d9/d11/d22/f24 [1936662,105598] 0 2024-04-20T21:31:52.639 INFO:tasks.workunit.client.0.smithi062.stdout:3/229: rmdir d0/d1/d10 39 2024-04-20T21:31:52.641 INFO:tasks.workunit.client.0.smithi062.stdout:7/157: dread d7/fe [0,4194304] 0 2024-04-20T21:31:52.644 INFO:tasks.workunit.client.0.smithi062.stdout:8/240: write d7/d9/d1b/f1e [2584367,519] 0 2024-04-20T21:31:52.646 INFO:tasks.workunit.client.0.smithi062.stdout:6/167: rename d0/c5 to d0/d9/d11/d22/c39 0 2024-04-20T21:31:52.648 INFO:tasks.workunit.client.0.smithi062.stdout:1/241: truncate f1 1802808 0 2024-04-20T21:31:52.648 INFO:tasks.workunit.client.0.smithi062.stdout:2/199: mknod d7/d39/c3b 0 2024-04-20T21:31:52.649 INFO:tasks.workunit.client.0.smithi062.stdout:3/230: mkdir d0/d1c/d52 0 2024-04-20T21:31:52.651 INFO:tasks.workunit.client.0.smithi062.stdout:2/200: fsync d7/ff 0 2024-04-20T21:31:52.652 INFO:tasks.workunit.client.0.smithi062.stdout:5/194: symlink d8/d10/d32/l45 0 2024-04-20T21:31:52.653 INFO:tasks.workunit.client.0.smithi062.stdout:6/168: write d0/d9/d11/f1f [3029432,51797] 0 2024-04-20T21:31:52.654 INFO:tasks.workunit.client.0.smithi062.stdout:8/241: mknod d7/d25/c55 0 2024-04-20T21:31:52.655 INFO:tasks.workunit.client.0.smithi062.stdout:4/185: creat da/dd/df/f3e x:0 0 0 2024-04-20T21:31:52.655 INFO:tasks.workunit.client.0.smithi062.stdout:9/203: dwrite d16/f3b [0,4194304] 0 2024-04-20T21:31:52.656 INFO:tasks.workunit.client.0.smithi062.stdout:2/201: stat d7/d39/c3b 0 2024-04-20T21:31:52.656 INFO:tasks.workunit.client.0.smithi062.stdout:3/231: chown d0/d11/c1e 951 1 2024-04-20T21:31:52.659 INFO:tasks.workunit.client.0.smithi062.stdout:5/195: readlink d8/d10/d2a/d2b/l30 0 2024-04-20T21:31:52.664 INFO:tasks.workunit.client.0.smithi062.stdout:5/196: fsync d8/f1c 0 2024-04-20T21:31:52.666 INFO:tasks.workunit.client.0.smithi062.stdout:7/158: dread d7/d19/d1c/f28 [0,4194304] 0 2024-04-20T21:31:52.667 INFO:tasks.workunit.client.0.smithi062.stdout:9/204: mknod d16/d2c/c44 0 2024-04-20T21:31:52.671 INFO:tasks.workunit.client.0.smithi062.stdout:6/169: rename d0/d9/d11/d22/f33 to d0/d1a/f3a 0 2024-04-20T21:31:52.674 INFO:tasks.workunit.client.0.smithi062.stdout:8/242: truncate d7/d9/d1b/f21 3873872 0 2024-04-20T21:31:52.675 INFO:tasks.workunit.client.0.smithi062.stdout:5/197: dread d8/d10/d2a/d2b/f36 [0,4194304] 0 2024-04-20T21:31:52.678 INFO:tasks.workunit.client.0.smithi062.stdout:2/202: link d7/da/f13 d7/da/f3c 0 2024-04-20T21:31:52.678 INFO:tasks.workunit.client.0.smithi062.stdout:8/243: fdatasync d7/d9/d29/f4b 0 2024-04-20T21:31:52.678 INFO:tasks.workunit.client.0.smithi062.stdout:3/232: creat d0/f53 x:0 0 0 2024-04-20T21:31:52.679 INFO:tasks.workunit.client.0.smithi062.stdout:7/159: fdatasync d7/f12 0 2024-04-20T21:31:52.681 INFO:tasks.workunit.client.0.smithi062.stdout:5/198: write d8/d1e/d2c/f3a [1034403,97860] 0 2024-04-20T21:31:52.681 INFO:tasks.workunit.client.0.smithi062.stdout:6/170: creat d0/d2c/f3b x:0 0 0 2024-04-20T21:31:52.683 INFO:tasks.workunit.client.0.smithi062.stdout:1/242: dread dc/d15/f42 [0,4194304] 0 2024-04-20T21:31:52.688 INFO:tasks.workunit.client.0.smithi062.stdout:4/186: dwrite da/f2f [0,4194304] 0 2024-04-20T21:31:52.692 INFO:tasks.workunit.client.0.smithi062.stdout:3/233: write d0/d1/d10/d2c/d42/f4a [581770,78340] 0 2024-04-20T21:31:52.694 INFO:tasks.workunit.client.0.smithi062.stdout:5/199: write d8/f1a [3836139,129548] 0 2024-04-20T21:31:52.697 INFO:tasks.workunit.client.0.smithi062.stdout:7/160: getdents d7/d26 0 2024-04-20T21:31:52.698 INFO:tasks.workunit.client.0.smithi062.stdout:3/234: dread - d0/d1b/f23 zero size 2024-04-20T21:31:52.701 INFO:tasks.workunit.client.0.smithi062.stdout:4/187: unlink da/dd/df/f24 0 2024-04-20T21:31:52.704 INFO:tasks.workunit.client.0.smithi062.stdout:5/200: mknod d8/d10/c46 0 2024-04-20T21:31:52.706 INFO:tasks.workunit.client.0.smithi062.stdout:6/171: stat d0/d2c/l34 0 2024-04-20T21:31:52.706 INFO:tasks.workunit.client.0.smithi062.stdout:3/235: write d0/d1b/f35 [4349389,118504] 0 2024-04-20T21:31:52.708 INFO:tasks.workunit.client.0.smithi062.stdout:1/243: mknod dc/d15/d20/c50 0 2024-04-20T21:31:52.710 INFO:tasks.workunit.client.0.smithi062.stdout:2/203: dread d7/f1e [4194304,4194304] 0 2024-04-20T21:31:52.711 INFO:tasks.workunit.client.0.smithi062.stdout:9/205: dread d16/f1e [0,4194304] 0 2024-04-20T21:31:52.715 INFO:tasks.workunit.client.0.smithi062.stdout:8/244: dwrite d7/d9/d1b/f4e [0,4194304] 0 2024-04-20T21:31:52.715 INFO:tasks.workunit.client.0.smithi062.stdout:6/172: fdatasync d0/d9/f16 0 2024-04-20T21:31:52.717 INFO:tasks.workunit.client.0.smithi062.stdout:9/206: write fe [942265,128634] 0 2024-04-20T21:31:52.723 INFO:tasks.workunit.client.0.smithi062.stdout:6/173: read d0/d9/f16 [2361550,120290] 0 2024-04-20T21:31:52.726 INFO:tasks.workunit.client.0.smithi062.stdout:4/188: truncate da/f13 1216489 0 2024-04-20T21:31:52.732 INFO:tasks.workunit.client.0.smithi062.stdout:6/174: mknod d0/d9/c3c 0 2024-04-20T21:31:52.734 INFO:tasks.workunit.client.0.smithi062.stdout:7/161: dwrite d7/d19/d1c/f1f [0,4194304] 0 2024-04-20T21:31:52.734 INFO:tasks.workunit.client.0.smithi062.stdout:2/204: getdents d7 0 2024-04-20T21:31:52.735 INFO:tasks.workunit.client.0.smithi062.stdout:4/189: mkdir da/dd/df/d27/d3f 0 2024-04-20T21:31:52.736 INFO:tasks.workunit.client.0.smithi062.stdout:7/162: readlink d7/l16 0 2024-04-20T21:31:52.737 INFO:tasks.workunit.client.0.smithi062.stdout:2/205: chown d7/da/d28 340693 1 2024-04-20T21:31:52.738 INFO:tasks.workunit.client.0.smithi062.stdout:5/201: dwrite d8/d10/f26 [0,4194304] 0 2024-04-20T21:31:52.739 INFO:tasks.workunit.client.0.smithi062.stdout:9/207: rename d16/l29 to d16/d2c/d42/l45 0 2024-04-20T21:31:52.739 INFO:tasks.workunit.client.0.smithi062.stdout:7/163: fsync d7/f18 0 2024-04-20T21:31:52.740 INFO:tasks.workunit.client.0.smithi062.stdout:6/175: rmdir d0/d9/d11/d22/d26 39 2024-04-20T21:31:52.740 INFO:tasks.workunit.client.0.smithi062.stdout:1/244: write dc/d15/d20/d28/f2f [1084358,123587] 0 2024-04-20T21:31:52.743 INFO:tasks.workunit.client.0.smithi062.stdout:8/245: truncate d7/f38 4022770 0 2024-04-20T21:31:52.750 INFO:tasks.workunit.client.0.smithi062.stdout:1/245: truncate dc/d15/d20/f3c 910961 0 2024-04-20T21:31:52.752 INFO:tasks.workunit.client.0.smithi062.stdout:7/164: creat d7/d19/f30 x:0 0 0 2024-04-20T21:31:52.753 INFO:tasks.workunit.client.0.smithi062.stdout:8/246: mknod d7/d30/c56 0 2024-04-20T21:31:52.756 INFO:tasks.workunit.client.0.smithi062.stdout:4/190: creat da/dd/df/d27/d30/d3d/f40 x:0 0 0 2024-04-20T21:31:52.761 INFO:tasks.workunit.client.0.smithi062.stdout:3/236: dread d0/d1b/f35 [0,4194304] 0 2024-04-20T21:31:52.765 INFO:tasks.workunit.client.0.smithi062.stdout:0/215: sync 2024-04-20T21:31:52.776 INFO:tasks.workunit.client.0.smithi062.stdout:9/208: rename d16/d2c/c44 to d16/d3d/c46 0 2024-04-20T21:31:52.781 INFO:tasks.workunit.client.0.smithi062.stdout:2/206: dwrite d7/da/f3c [0,4194304] 0 2024-04-20T21:31:52.785 INFO:tasks.workunit.client.0.smithi062.stdout:6/176: dwrite d0/d9/d11/f1f [0,4194304] 0 2024-04-20T21:31:52.790 INFO:tasks.workunit.client.0.smithi062.stdout:8/247: dread d7/d9/f13 [0,4194304] 0 2024-04-20T21:31:52.791 INFO:tasks.workunit.client.0.smithi062.stdout:3/237: rename d0/d1/d10/d2c/d42/f4a to d0/d1/f54 0 2024-04-20T21:31:52.791 INFO:tasks.workunit.client.0.smithi062.stdout:7/165: dread d7/fe [0,4194304] 0 2024-04-20T21:31:52.792 INFO:tasks.workunit.client.0.smithi062.stdout:5/202: dwrite d8/d1e/f25 [0,4194304] 0 2024-04-20T21:31:52.796 INFO:tasks.workunit.client.0.smithi062.stdout:6/177: read d0/d9/d11/d22/f24 [210461,80356] 0 2024-04-20T21:31:52.801 INFO:tasks.workunit.client.0.smithi062.stdout:2/207: getdents d7/da/d15/d2f/d38 0 2024-04-20T21:31:52.801 INFO:tasks.workunit.client.0.smithi062.stdout:1/246: dwrite dc/d15/d20/d28/f4e [0,4194304] 0 2024-04-20T21:31:52.804 INFO:tasks.workunit.client.0.smithi062.stdout:7/166: readlink d7/l15 0 2024-04-20T21:31:52.804 INFO:tasks.workunit.client.0.smithi062.stdout:5/203: write d8/fb [2951119,38348] 0 2024-04-20T21:31:52.809 INFO:tasks.workunit.client.0.smithi062.stdout:3/238: symlink d0/d1/d10/d31/d41/l55 0 2024-04-20T21:31:52.809 INFO:tasks.workunit.client.0.smithi062.stdout:6/178: mknod d0/d1a/c3d 0 2024-04-20T21:31:52.813 INFO:tasks.workunit.client.0.smithi062.stdout:6/179: chown d0/c2f 5428 1 2024-04-20T21:31:52.816 INFO:tasks.workunit.client.0.smithi062.stdout:5/204: symlink d8/d10/d3e/l47 0 2024-04-20T21:31:52.816 INFO:tasks.workunit.client.0.smithi062.stdout:6/180: chown d0/d9/f13 3951 1 2024-04-20T21:31:52.829 INFO:tasks.workunit.client.0.smithi062.stdout:9/209: link d16/l26 d16/l47 0 2024-04-20T21:31:52.832 INFO:tasks.workunit.client.0.smithi062.stdout:7/167: dread d7/f12 [4194304,4194304] 0 2024-04-20T21:31:52.833 INFO:tasks.workunit.client.0.smithi062.stdout:2/208: getdents d7/d39 0 2024-04-20T21:31:52.835 INFO:tasks.workunit.client.0.smithi062.stdout:9/210: stat d16/d3e/f40 0 2024-04-20T21:31:52.836 INFO:tasks.workunit.client.0.smithi062.stdout:0/216: dread d7/fb [0,4194304] 0 2024-04-20T21:31:52.839 INFO:tasks.workunit.client.0.smithi062.stdout:9/211: stat fb 0 2024-04-20T21:31:52.842 INFO:tasks.workunit.client.0.smithi062.stdout:4/191: dread da/dd/f11 [0,4194304] 0 2024-04-20T21:31:52.843 INFO:tasks.workunit.client.0.smithi062.stdout:1/247: dread dc/d15/d17/f19 [0,4194304] 0 2024-04-20T21:31:52.845 INFO:tasks.workunit.client.0.smithi062.stdout:7/168: creat d7/d26/f31 x:0 0 0 2024-04-20T21:31:52.848 INFO:tasks.workunit.client.0.smithi062.stdout:5/205: dwrite d8/d1e/f25 [0,4194304] 0 2024-04-20T21:31:52.849 INFO:tasks.workunit.client.0.smithi062.stdout:4/192: write da/f2f [3007062,45262] 0 2024-04-20T21:31:52.852 INFO:tasks.workunit.client.0.smithi062.stdout:3/239: dwrite d0/d1b/f23 [0,4194304] 0 2024-04-20T21:31:52.852 INFO:tasks.workunit.client.0.smithi062.stdout:9/212: chown d16/d2f/f39 0 1 2024-04-20T21:31:52.853 INFO:tasks.workunit.client.0.smithi062.stdout:8/248: dwrite d7/d9/f41 [0,4194304] 0 2024-04-20T21:31:52.853 INFO:tasks.workunit.client.0.smithi062.stdout:1/248: fsync dc/d15/d20/d28/f49 0 2024-04-20T21:31:52.855 INFO:tasks.workunit.client.0.smithi062.stdout:4/193: chown c8 0 1 2024-04-20T21:31:52.861 INFO:tasks.workunit.client.0.smithi062.stdout:4/194: write da/dd/f12 [3491215,106566] 0 2024-04-20T21:31:52.861 INFO:tasks.workunit.client.0.smithi062.stdout:9/213: write fe [51649,20293] 0 2024-04-20T21:31:52.864 INFO:tasks.workunit.client.0.smithi062.stdout:5/206: unlink d8/d1e/d2c/c41 0 2024-04-20T21:31:52.867 INFO:tasks.workunit.client.0.smithi062.stdout:3/240: creat d0/d11/f56 x:0 0 0 2024-04-20T21:31:52.869 INFO:tasks.workunit.client.0.smithi062.stdout:4/195: chown da/dd/d15/l21 6613312 1 2024-04-20T21:31:52.869 INFO:tasks.workunit.client.0.smithi062.stdout:8/249: unlink d7/d9/f13 0 2024-04-20T21:31:52.869 INFO:tasks.workunit.client.0.smithi062.stdout:0/217: dread d7/d1c/d15/d17/f38 [0,4194304] 0 2024-04-20T21:31:52.872 INFO:tasks.workunit.client.0.smithi062.stdout:6/181: dread d0/d9/f18 [0,4194304] 0 2024-04-20T21:31:52.873 INFO:tasks.workunit.client.0.smithi062.stdout:2/209: dread d7/ff [4194304,4194304] 0 2024-04-20T21:31:52.880 INFO:tasks.workunit.client.0.smithi062.stdout:8/250: write d7/d9/d29/f32 [363576,120780] 0 2024-04-20T21:31:52.880 INFO:tasks.workunit.client.0.smithi062.stdout:3/241: unlink d0/d1/d10/f27 0 2024-04-20T21:31:52.880 INFO:tasks.workunit.client.0.smithi062.stdout:6/182: read d0/d9/f16 [313990,9471] 0 2024-04-20T21:31:52.880 INFO:tasks.workunit.client.0.smithi062.stdout:0/218: stat d7/d1c/d15/f46 0 2024-04-20T21:31:52.881 INFO:tasks.workunit.client.0.smithi062.stdout:7/169: dread d7/d19/f1a [0,4194304] 0 2024-04-20T21:31:52.884 INFO:tasks.workunit.client.0.smithi062.stdout:3/242: read d0/d1b/f35 [1164414,27746] 0 2024-04-20T21:31:52.885 INFO:tasks.workunit.client.0.smithi062.stdout:6/183: fdatasync d0/d2c/f30 0 2024-04-20T21:31:52.887 INFO:tasks.workunit.client.0.smithi062.stdout:0/219: symlink d7/d1c/d15/d17/d1b/l49 0 2024-04-20T21:31:52.888 INFO:tasks.workunit.client.0.smithi062.stdout:4/196: link da/dd/df/f28 da/dd/df/f41 0 2024-04-20T21:31:52.896 INFO:tasks.workunit.client.0.smithi062.stdout:7/170: symlink d7/d19/l32 0 2024-04-20T21:31:52.897 INFO:tasks.workunit.client.0.smithi062.stdout:2/210: dread d7/da/f13 [0,4194304] 0 2024-04-20T21:31:52.900 INFO:tasks.workunit.client.0.smithi062.stdout:9/214: dwrite d16/d2f/f39 [0,4194304] 0 2024-04-20T21:31:52.902 INFO:tasks.workunit.client.0.smithi062.stdout:6/184: rmdir d0/d9 39 2024-04-20T21:31:52.905 INFO:tasks.workunit.client.0.smithi062.stdout:7/171: mkdir d7/d19/d33 0 2024-04-20T21:31:52.905 INFO:tasks.workunit.client.0.smithi062.stdout:1/249: dwrite dc/d15/d17/f4c [0,4194304] 0 2024-04-20T21:31:52.908 INFO:tasks.workunit.client.0.smithi062.stdout:9/215: read - d16/d3e/f40 zero size 2024-04-20T21:31:52.911 INFO:tasks.workunit.client.0.smithi062.stdout:6/185: unlink d0/c2f 0 2024-04-20T21:31:52.913 INFO:tasks.workunit.client.0.smithi062.stdout:8/251: link d7/d9/d1b/d24/d46/f49 d7/f57 0 2024-04-20T21:31:52.914 INFO:tasks.workunit.client.0.smithi062.stdout:4/197: dread da/dd/f35 [0,4194304] 0 2024-04-20T21:31:52.915 INFO:tasks.workunit.client.0.smithi062.stdout:7/172: creat d7/d26/f34 x:0 0 0 2024-04-20T21:31:52.917 INFO:tasks.workunit.client.0.smithi062.stdout:5/207: dwrite d8/d10/d2a/f39 [0,4194304] 0 2024-04-20T21:31:52.918 INFO:tasks.workunit.client.0.smithi062.stdout:1/250: creat dc/d27/d32/f51 x:0 0 0 2024-04-20T21:31:52.926 INFO:tasks.workunit.client.0.smithi062.stdout:8/252: mkdir d7/d25/d3b/d58 0 2024-04-20T21:31:52.928 INFO:tasks.workunit.client.0.smithi062.stdout:3/243: dwrite d0/f53 [0,4194304] 0 2024-04-20T21:31:52.930 INFO:tasks.workunit.client.0.smithi062.stdout:4/198: symlink da/dd/df/d27/l42 0 2024-04-20T21:31:52.934 INFO:tasks.workunit.client.0.smithi062.stdout:7/173: creat d7/d19/d33/f35 x:0 0 0 2024-04-20T21:31:52.943 INFO:tasks.workunit.client.0.smithi062.stdout:3/244: write d0/d11/f1a [7966859,108181] 0 2024-04-20T21:31:52.951 INFO:tasks.workunit.client.0.smithi062.stdout:2/211: dwrite d7/da/f13 [0,4194304] 0 2024-04-20T21:31:52.951 INFO:tasks.workunit.client.0.smithi062.stdout:6/186: write d0/f17 [1702979,17636] 0 2024-04-20T21:31:52.951 INFO:tasks.workunit.client.0.smithi062.stdout:5/208: dwrite d8/f1a [0,4194304] 0 2024-04-20T21:31:52.952 INFO:tasks.workunit.client.0.smithi062.stdout:8/253: chown d7/d9/d1b/c27 80017081 1 2024-04-20T21:31:52.956 INFO:tasks.workunit.client.0.smithi062.stdout:9/216: dwrite d16/f2e [0,4194304] 0 2024-04-20T21:31:52.956 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:52 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:31:52.957 INFO:tasks.workunit.client.0.smithi062.stdout:3/245: truncate d0/d1/fd 1379937 0 2024-04-20T21:31:52.961 INFO:tasks.workunit.client.0.smithi062.stdout:1/251: dread dc/d15/f18 [0,4194304] 0 2024-04-20T21:31:52.969 INFO:tasks.workunit.client.0.smithi062.stdout:5/209: write d8/d1e/d2c/f3a [619348,47999] 0 2024-04-20T21:31:52.970 INFO:tasks.workunit.client.0.smithi062.stdout:9/217: chown d16/l37 1884 1 2024-04-20T21:31:52.973 INFO:tasks.workunit.client.0.smithi062.stdout:3/246: rename d0/d1/f54 to d0/d1b/f57 0 2024-04-20T21:31:52.974 INFO:tasks.workunit.client.0.smithi062.stdout:4/199: dwrite f3 [0,4194304] 0 2024-04-20T21:31:52.974 INFO:tasks.workunit.client.0.smithi062.stdout:5/210: fdatasync d8/d10/f26 0 2024-04-20T21:31:52.976 INFO:tasks.workunit.client.0.smithi062.stdout:8/254: symlink d7/d25/d3b/d58/l59 0 2024-04-20T21:31:52.977 INFO:tasks.workunit.client.0.smithi062.stdout:9/218: read d16/f2e [4067095,100821] 0 2024-04-20T21:31:52.982 INFO:tasks.workunit.client.0.smithi062.stdout:5/211: readlink d8/d10/l16 0 2024-04-20T21:31:52.982 INFO:tasks.workunit.client.0.smithi062.stdout:4/200: fdatasync da/dd/f35 0 2024-04-20T21:31:52.983 INFO:tasks.workunit.client.0.smithi062.stdout:6/187: getdents d0/d9/d11/d22 0 2024-04-20T21:31:52.987 INFO:tasks.workunit.client.0.smithi062.stdout:5/212: readlink d8/d1e/l1f 0 2024-04-20T21:31:52.991 INFO:tasks.workunit.client.0.smithi062.stdout:2/212: truncate d7/f12 3204153 0 2024-04-20T21:31:52.995 INFO:tasks.workunit.client.0.smithi062.stdout:3/247: rename d0/d1/d10/d31/d41/c4b to d0/d1/c58 0 2024-04-20T21:31:52.996 INFO:tasks.workunit.client.0.smithi062.stdout:7/174: dwrite d7/d19/f1a [4194304,4194304] 0 2024-04-20T21:31:53.005 INFO:tasks.workunit.client.0.smithi062.stdout:3/248: mknod d0/d1/d10/d31/c59 0 2024-04-20T21:31:53.007 INFO:tasks.workunit.client.0.smithi062.stdout:2/213: chown d7/da/d15/d1b/f2c 29204 1 2024-04-20T21:31:53.013 INFO:tasks.workunit.client.0.smithi062.stdout:3/249: readlink d0/d1/d10/d2c/l2e 0 2024-04-20T21:31:53.013 INFO:tasks.workunit.client.0.smithi062.stdout:9/219: dread d16/f2e [0,4194304] 0 2024-04-20T21:31:53.014 INFO:tasks.workunit.client.0.smithi062.stdout:1/252: dread f2 [0,4194304] 0 2024-04-20T21:31:53.014 INFO:tasks.workunit.client.0.smithi062.stdout:7/175: creat d7/f36 x:0 0 0 2024-04-20T21:31:53.014 INFO:tasks.workunit.client.0.smithi062.stdout:5/213: truncate d8/d1e/f24 1956050 0 2024-04-20T21:31:53.015 INFO:tasks.workunit.client.0.smithi062.stdout:3/250: chown d0/d1b/f35 425 1 2024-04-20T21:31:53.022 INFO:tasks.workunit.client.0.smithi062.stdout:7/176: creat d7/d19/d33/f37 x:0 0 0 2024-04-20T21:31:53.023 INFO:tasks.workunit.client.0.smithi062.stdout:5/214: stat d8/d1e/d2c/l2e 0 2024-04-20T21:31:53.025 INFO:tasks.workunit.client.0.smithi062.stdout:5/215: chown c7 1875 1 2024-04-20T21:31:53.026 INFO:tasks.workunit.client.0.smithi062.stdout:6/188: dwrite d0/f17 [0,4194304] 0 2024-04-20T21:31:53.028 INFO:tasks.workunit.client.0.smithi062.stdout:3/251: truncate d0/d1/f19 763888 0 2024-04-20T21:31:53.029 INFO:tasks.workunit.client.0.smithi062.stdout:9/220: creat d16/d2c/d42/f48 x:0 0 0 2024-04-20T21:31:53.030 INFO:tasks.workunit.client.0.smithi062.stdout:7/177: mknod d7/d19/c38 0 2024-04-20T21:31:53.031 INFO:tasks.workunit.client.0.smithi062.stdout:3/252: chown d0/l13 363 1 2024-04-20T21:31:53.033 INFO:tasks.workunit.client.0.smithi062.stdout:5/216: mknod d8/d1e/c48 0 2024-04-20T21:31:53.033 INFO:tasks.workunit.client.0.smithi062.stdout:3/253: write d0/f4f [128603,126420] 0 2024-04-20T21:31:53.034 INFO:tasks.workunit.client.0.smithi062.stdout:2/214: truncate d7/da/f13 2721130 0 2024-04-20T21:31:53.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:52 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:31:53.036 INFO:tasks.workunit.client.0.smithi062.stdout:4/201: dwrite da/dd/df/d27/f36 [0,4194304] 0 2024-04-20T21:31:53.039 INFO:tasks.workunit.client.0.smithi062.stdout:9/221: mknod d16/d3e/c49 0 2024-04-20T21:31:53.041 INFO:tasks.workunit.client.0.smithi062.stdout:7/178: mkdir d7/d26/d39 0 2024-04-20T21:31:53.041 INFO:tasks.workunit.client.0.smithi062.stdout:5/217: mknod d8/d10/c49 0 2024-04-20T21:31:53.044 INFO:tasks.workunit.client.0.smithi062.stdout:8/255: dwrite d7/f38 [0,4194304] 0 2024-04-20T21:31:53.045 INFO:tasks.workunit.client.0.smithi062.stdout:5/218: dread - d8/d1e/d2c/f34 zero size 2024-04-20T21:31:53.048 INFO:tasks.workunit.client.0.smithi062.stdout:9/222: symlink d16/d2c/d42/l4a 0 2024-04-20T21:31:53.048 INFO:tasks.workunit.client.0.smithi062.stdout:6/189: creat d0/f3e x:0 0 0 2024-04-20T21:31:53.048 INFO:tasks.workunit.client.0.smithi062.stdout:3/254: creat d0/d22/f5a x:0 0 0 2024-04-20T21:31:53.051 INFO:tasks.workunit.client.0.smithi062.stdout:8/256: creat d7/d25/f5a x:0 0 0 2024-04-20T21:31:53.052 INFO:tasks.workunit.client.0.smithi062.stdout:4/202: mknod da/dd/df/d27/c43 0 2024-04-20T21:31:53.052 INFO:tasks.workunit.client.0.smithi062.stdout:9/223: write f6 [6058245,77922] 0 2024-04-20T21:31:53.057 INFO:tasks.workunit.client.0.smithi062.stdout:8/257: chown d7/d25/l33 36074987 1 2024-04-20T21:31:53.058 INFO:tasks.workunit.client.0.smithi062.stdout:5/219: mknod d8/d10/d32/c4a 0 2024-04-20T21:31:53.058 INFO:tasks.workunit.client.0.smithi062.stdout:4/203: dread - da/dd/df/f28 zero size 2024-04-20T21:31:53.061 INFO:tasks.workunit.client.0.smithi062.stdout:3/255: mkdir d0/d1/d10/d2c/d42/d5b 0 2024-04-20T21:31:53.061 INFO:tasks.workunit.client.0.smithi062.stdout:1/253: dwrite dc/d15/d20/f3c [0,4194304] 0 2024-04-20T21:31:53.069 INFO:tasks.workunit.client.0.smithi062.stdout:8/258: write d7/d9/d29/f32 [67209,108627] 0 2024-04-20T21:31:53.069 INFO:tasks.workunit.client.0.smithi062.stdout:5/220: dread - d8/f44 zero size 2024-04-20T21:31:53.071 INFO:tasks.workunit.client.0.smithi062.stdout:9/224: rename d16/d3e/c49 to d16/d2f/c4b 0 2024-04-20T21:31:53.077 INFO:tasks.workunit.client.0.smithi062.stdout:4/204: symlink da/dd/df/d27/d30/d3d/l44 0 2024-04-20T21:31:53.079 INFO:tasks.workunit.client.0.smithi062.stdout:8/259: write d7/d9/d1b/d24/f50 [16409,98116] 0 2024-04-20T21:31:53.080 INFO:tasks.workunit.client.0.smithi062.stdout:5/221: write f4 [476014,19202] 0 2024-04-20T21:31:53.082 INFO:tasks.workunit.client.0.smithi062.stdout:1/254: symlink dc/d15/d17/d36/d3a/l52 0 2024-04-20T21:31:53.083 INFO:tasks.workunit.client.0.smithi062.stdout:7/179: dwrite d7/f36 [0,4194304] 0 2024-04-20T21:31:53.083 INFO:tasks.workunit.client.0.smithi062.stdout:9/225: mkdir d16/d4c 0 2024-04-20T21:31:53.085 INFO:tasks.workunit.client.0.smithi062.stdout:6/190: dread d0/d9/f16 [0,4194304] 0 2024-04-20T21:31:53.086 INFO:tasks.workunit.client.0.smithi062.stdout:5/222: readlink d8/d10/d2a/d2b/l31 0 2024-04-20T21:31:53.086 INFO:tasks.workunit.client.0.smithi062.stdout:3/256: rename d0/d22 to d0/d5c 0 2024-04-20T21:31:53.091 INFO:tasks.workunit.client.0.smithi062.stdout:9/226: mknod d16/d2f/c4d 0 2024-04-20T21:31:53.093 INFO:tasks.workunit.client.0.smithi062.stdout:8/260: symlink d7/d9/d1b/d24/d46/l5b 0 2024-04-20T21:31:53.095 INFO:tasks.workunit.client.0.smithi062.stdout:7/180: symlink d7/d19/d1c/l3a 0 2024-04-20T21:31:53.098 INFO:tasks.workunit.client.0.smithi062.stdout:5/223: unlink d8/d10/d2a/d2b/l30 0 2024-04-20T21:31:53.100 INFO:tasks.workunit.client.0.smithi062.stdout:4/205: creat da/dd/f45 x:0 0 0 2024-04-20T21:31:53.101 INFO:tasks.workunit.client.0.smithi062.stdout:5/224: write d8/f1c [3729668,127096] 0 2024-04-20T21:31:53.102 INFO:tasks.workunit.client.0.smithi062.stdout:8/261: mkdir d7/d25/d5c 0 2024-04-20T21:31:53.107 INFO:tasks.workunit.client.0.smithi062.stdout:3/257: dwrite d0/d1/f20 [0,4194304] 0 2024-04-20T21:31:53.108 INFO:tasks.workunit.client.0.smithi062.stdout:9/227: dread d16/d2f/f39 [0,4194304] 0 2024-04-20T21:31:53.109 INFO:tasks.workunit.client.0.smithi062.stdout:3/258: chown d0/d1/d10 339 1 2024-04-20T21:31:53.109 INFO:tasks.workunit.client.0.smithi062.stdout:7/181: link l0 d7/d19/d33/l3b 0 2024-04-20T21:31:53.113 INFO:tasks.workunit.client.0.smithi062.stdout:3/259: write d0/f2f [3605337,111944] 0 2024-04-20T21:31:53.114 INFO:tasks.workunit.client.0.smithi062.stdout:4/206: rename da/dd/l18 to da/dd/d15/l46 0 2024-04-20T21:31:53.116 INFO:tasks.workunit.client.0.smithi062.stdout:4/207: truncate f4 1699086 0 2024-04-20T21:31:53.117 INFO:tasks.workunit.client.0.smithi062.stdout:7/182: mknod d7/d26/c3c 0 2024-04-20T21:31:53.119 INFO:tasks.workunit.client.0.smithi062.stdout:7/183: fsync d7/d26/f34 0 2024-04-20T21:31:53.119 INFO:tasks.workunit.client.0.smithi062.stdout:9/228: mkdir d16/d4e 0 2024-04-20T21:31:53.120 INFO:tasks.workunit.client.0.smithi062.stdout:6/191: dread d0/fe [4194304,4194304] 0 2024-04-20T21:31:53.124 INFO:tasks.workunit.client.0.smithi062.stdout:3/260: rename d0/d11/l49 to d0/d1c/d52/l5d 0 2024-04-20T21:31:53.128 INFO:tasks.workunit.client.0.smithi062.stdout:7/184: mkdir d7/d19/d33/d3d 0 2024-04-20T21:31:53.129 INFO:tasks.workunit.client.0.smithi062.stdout:8/262: write d7/d9/d1b/f1f [5030217,64699] 0 2024-04-20T21:31:53.130 INFO:tasks.workunit.client.0.smithi062.stdout:6/192: unlink d0/d2c/l34 0 2024-04-20T21:31:53.132 INFO:tasks.workunit.client.0.smithi062.stdout:8/263: write d7/d9/d29/f4c [435403,6605] 0 2024-04-20T21:31:53.139 INFO:tasks.workunit.client.0.smithi062.stdout:5/225: truncate d8/f1a 3665332 0 2024-04-20T21:31:53.140 INFO:tasks.workunit.client.0.smithi062.stdout:8/264: rmdir d7/d25/d3b/d58 39 2024-04-20T21:31:53.141 INFO:tasks.workunit.client.0.smithi062.stdout:3/261: mkdir d0/d1/d10/d3a/d5e 0 2024-04-20T21:31:53.141 INFO:tasks.workunit.client.0.smithi062.stdout:4/208: link c8 da/dd/df/c47 0 2024-04-20T21:31:53.142 INFO:tasks.workunit.client.0.smithi062.stdout:9/229: dwrite d16/f32 [0,4194304] 0 2024-04-20T21:31:53.146 INFO:tasks.workunit.client.0.smithi062.stdout:0/220: sync 2024-04-20T21:31:53.146 INFO:tasks.workunit.client.0.smithi062.stdout:1/255: dread dc/ff [0,4194304] 0 2024-04-20T21:31:53.148 INFO:tasks.workunit.client.0.smithi062.stdout:0/221: chown d7 18 1 2024-04-20T21:31:53.148 INFO:tasks.workunit.client.0.smithi062.stdout:6/193: symlink d0/l3f 0 2024-04-20T21:31:53.157 INFO:tasks.workunit.client.0.smithi062.stdout:7/185: rename d7/d19 to d7/d3e 0 2024-04-20T21:31:53.158 INFO:tasks.workunit.client.0.smithi062.stdout:3/262: creat d0/d1/d10/d31/f5f x:0 0 0 2024-04-20T21:31:53.159 INFO:tasks.workunit.client.0.smithi062.stdout:9/230: fsync d16/f3a 0 2024-04-20T21:31:53.160 INFO:tasks.workunit.client.0.smithi062.stdout:4/209: mknod da/dd/df/d27/d30/c48 0 2024-04-20T21:31:53.162 INFO:tasks.workunit.client.0.smithi062.stdout:4/210: write da/dd/df/f10 [1044036,23761] 0 2024-04-20T21:31:53.164 INFO:tasks.workunit.client.0.smithi062.stdout:0/222: symlink d7/d1c/d15/d17/d1b/l4a 0 2024-04-20T21:31:53.165 INFO:tasks.workunit.client.0.smithi062.stdout:8/265: write d7/f57 [390066,37914] 0 2024-04-20T21:31:53.166 INFO:tasks.workunit.client.0.smithi062.stdout:5/226: chown d8/d10/f26 23325218 1 2024-04-20T21:31:53.167 INFO:tasks.workunit.client.0.smithi062.stdout:8/266: readlink d7/d25/l36 0 2024-04-20T21:31:53.168 INFO:tasks.workunit.client.0.smithi062.stdout:7/186: creat d7/d3e/d1c/f3f x:0 0 0 2024-04-20T21:31:53.171 INFO:tasks.workunit.client.0.smithi062.stdout:5/227: creat d8/d10/d2a/d2f/f4b x:0 0 0 2024-04-20T21:31:53.172 INFO:tasks.workunit.client.0.smithi062.stdout:8/267: creat d7/d30/f5d x:0 0 0 2024-04-20T21:31:53.172 INFO:tasks.workunit.client.0.smithi062.stdout:9/231: fsync d16/d3d/f43 0 2024-04-20T21:31:53.174 INFO:tasks.workunit.client.0.smithi062.stdout:7/187: symlink d7/d26/l40 0 2024-04-20T21:31:53.179 INFO:tasks.workunit.client.0.smithi062.stdout:0/223: write d7/d1c/f18 [576886,80858] 0 2024-04-20T21:31:53.179 INFO:tasks.workunit.client.0.smithi062.stdout:8/268: rmdir d7/d25/d3b 39 2024-04-20T21:31:53.181 INFO:tasks.workunit.client.0.smithi062.stdout:5/228: truncate d8/fb 4003300 0 2024-04-20T21:31:53.184 INFO:tasks.workunit.client.0.smithi062.stdout:1/256: dwrite dc/f26 [0,4194304] 0 2024-04-20T21:31:53.184 INFO:tasks.workunit.client.0.smithi062.stdout:9/232: symlink d16/d2c/d42/l4f 0 2024-04-20T21:31:53.188 INFO:tasks.workunit.client.0.smithi062.stdout:9/233: stat d16/d2c 0 2024-04-20T21:31:53.189 INFO:tasks.workunit.client.0.smithi062.stdout:8/269: chown d7/d9/l2e 14 1 2024-04-20T21:31:53.190 INFO:tasks.workunit.client.0.smithi062.stdout:5/229: chown d8/d10/d3e 1365495 1 2024-04-20T21:31:53.195 INFO:tasks.workunit.client.0.smithi062.stdout:3/263: dwrite d0/d5c/f5a [0,4194304] 0 2024-04-20T21:31:53.197 INFO:tasks.workunit.client.0.smithi062.stdout:9/234: fsync d16/f19 0 2024-04-20T21:31:53.199 INFO:tasks.workunit.client.0.smithi062.stdout:0/224: link d7/fb d7/d1c/f4b 0 2024-04-20T21:31:53.199 INFO:tasks.workunit.client.0.smithi062.stdout:7/188: truncate d7/fe 5134243 0 2024-04-20T21:31:53.199 INFO:tasks.workunit.client.0.smithi062.stdout:6/194: dwrite d0/d2c/f30 [0,4194304] 0 2024-04-20T21:31:53.206 INFO:tasks.workunit.client.0.smithi062.stdout:9/235: creat d16/d2c/d42/f50 x:0 0 0 2024-04-20T21:31:53.206 INFO:tasks.workunit.client.0.smithi062.stdout:8/270: mknod d7/d9/c5e 0 2024-04-20T21:31:53.207 INFO:tasks.workunit.client.0.smithi062.stdout:4/211: dwrite da/dd/df/f1a [0,4194304] 0 2024-04-20T21:31:53.209 INFO:tasks.workunit.client.0.smithi062.stdout:6/195: write d0/f3e [917791,116091] 0 2024-04-20T21:31:53.211 INFO:tasks.workunit.client.0.smithi062.stdout:8/271: chown d7/d9/d1b 1 1 2024-04-20T21:31:53.215 INFO:tasks.workunit.client.0.smithi062.stdout:9/236: write d16/f2e [232081,22558] 0 2024-04-20T21:31:53.217 INFO:tasks.workunit.client.0.smithi062.stdout:6/196: write d0/d9/d11/f1f [321141,55999] 0 2024-04-20T21:31:53.219 INFO:tasks.workunit.client.0.smithi062.stdout:2/215: sync 2024-04-20T21:31:53.222 INFO:tasks.workunit.client.0.smithi062.stdout:8/272: creat d7/d30/f5f x:0 0 0 2024-04-20T21:31:53.222 INFO:tasks.workunit.client.0.smithi062.stdout:9/237: chown d16/d2c/d42/f48 876084 1 2024-04-20T21:31:53.222 INFO:tasks.workunit.client.0.smithi062.stdout:0/225: rename d7/d1c/f4b to d7/f4c 0 2024-04-20T21:31:53.223 INFO:tasks.workunit.client.0.smithi062.stdout:1/257: dwrite fb [0,4194304] 0 2024-04-20T21:31:53.227 INFO:tasks.workunit.client.0.smithi062.stdout:6/197: read - d0/d1a/f29 zero size 2024-04-20T21:31:53.233 INFO:tasks.workunit.client.0.smithi062.stdout:8/273: write d7/d30/f3c [1013583,65134] 0 2024-04-20T21:31:53.235 INFO:tasks.workunit.client.0.smithi062.stdout:5/230: dwrite d8/f44 [0,4194304] 0 2024-04-20T21:31:53.237 INFO:tasks.workunit.client.0.smithi062.stdout:0/226: stat d7/d1c/l35 0 2024-04-20T21:31:53.238 INFO:tasks.workunit.client.0.smithi062.stdout:7/189: rename d7/f14 to d7/d3e/f41 0 2024-04-20T21:31:53.238 INFO:tasks.workunit.client.0.smithi062.stdout:2/216: chown d7/da/d15/d1b/f34 2811 1 2024-04-20T21:31:53.238 INFO:tasks.workunit.client.0.smithi062.stdout:1/258: write dc/d27/d32/f51 [11667,103591] 0 2024-04-20T21:31:53.244 INFO:tasks.workunit.client.0.smithi062.stdout:8/274: creat d7/d9/d29/f60 x:0 0 0 2024-04-20T21:31:53.244 INFO:tasks.workunit.client.0.smithi062.stdout:2/217: chown d7/f12 534521214 1 2024-04-20T21:31:53.244 INFO:tasks.workunit.client.0.smithi062.stdout:5/231: mkdir d8/d10/d2a/d2b/d4c 0 2024-04-20T21:31:53.245 INFO:tasks.workunit.client.0.smithi062.stdout:3/264: dwrite d0/d11/f56 [0,4194304] 0 2024-04-20T21:31:53.247 INFO:tasks.workunit.client.0.smithi062.stdout:6/198: mknod d0/c40 0 2024-04-20T21:31:53.247 INFO:tasks.workunit.client.0.smithi062.stdout:1/259: dread - dc/d15/d17/d36/d3a/f4d zero size 2024-04-20T21:31:53.249 INFO:tasks.workunit.client.0.smithi062.stdout:0/227: mknod d7/d1e/c4d 0 2024-04-20T21:31:53.252 INFO:tasks.workunit.client.0.smithi062.stdout:7/190: truncate d7/d3e/d1c/f2e 690370 0 2024-04-20T21:31:53.254 INFO:tasks.workunit.client.0.smithi062.stdout:4/212: dread da/dd/df/f1a [0,4194304] 0 2024-04-20T21:31:53.258 INFO:tasks.workunit.client.0.smithi062.stdout:2/218: mkdir d7/da/d15/d1b/d3d 0 2024-04-20T21:31:53.258 INFO:tasks.workunit.client.0.smithi062.stdout:0/228: unlink d7/d1e/l25 0 2024-04-20T21:31:53.260 INFO:tasks.workunit.client.0.smithi062.stdout:2/219: fdatasync d7/da/f22 0 2024-04-20T21:31:53.267 INFO:tasks.workunit.client.0.smithi062.stdout:9/238: dread d16/f32 [4194304,4194304] 0 2024-04-20T21:31:53.267 INFO:tasks.workunit.client.0.smithi062.stdout:2/220: write d7/da/d15/d1b/f2c [668021,34429] 0 2024-04-20T21:31:53.268 INFO:tasks.workunit.client.0.smithi062.stdout:0/229: mkdir d7/d1c/d15/d17/d1b/d4e 0 2024-04-20T21:31:53.272 INFO:tasks.workunit.client.0.smithi062.stdout:1/260: rename f2 to dc/f53 0 2024-04-20T21:31:53.273 INFO:tasks.workunit.client.0.smithi062.stdout:4/213: creat da/dd/df/d27/d32/f49 x:0 0 0 2024-04-20T21:31:53.275 INFO:tasks.workunit.client.0.smithi062.stdout:5/232: link l6 d8/d38/l4d 0 2024-04-20T21:31:53.275 INFO:tasks.workunit.client.0.smithi062.stdout:1/261: readlink dc/d15/l3d 0 2024-04-20T21:31:53.276 INFO:tasks.workunit.client.0.smithi062.stdout:3/265: dwrite d0/d5c/f5a [4194304,4194304] 0 2024-04-20T21:31:53.279 INFO:tasks.workunit.client.0.smithi062.stdout:6/199: link d0/cb d0/d1a/c41 0 2024-04-20T21:31:53.279 INFO:tasks.workunit.client.0.smithi062.stdout:8/275: dwrite d7/d30/f5d [0,4194304] 0 2024-04-20T21:31:53.286 INFO:tasks.workunit.client.0.smithi062.stdout:7/191: getdents d7/d3e/d1c 0 2024-04-20T21:31:53.289 INFO:tasks.workunit.client.0.smithi062.stdout:5/233: creat d8/d10/f4e x:0 0 0 2024-04-20T21:31:53.289 INFO:tasks.workunit.client.0.smithi062.stdout:8/276: write d7/d9/d1b/f4e [230814,109407] 0 2024-04-20T21:31:53.289 INFO:tasks.workunit.client.0.smithi062.stdout:2/221: symlink d7/da/d15/d2f/d38/l3e 0 2024-04-20T21:31:53.292 INFO:tasks.workunit.client.0.smithi062.stdout:3/266: creat d0/d1/d10/d2c/d42/f60 x:0 0 0 2024-04-20T21:31:53.293 INFO:tasks.workunit.client.0.smithi062.stdout:1/262: fsync dc/d15/f18 0 2024-04-20T21:31:53.295 INFO:tasks.workunit.client.0.smithi062.stdout:9/239: rmdir d16/d4c 0 2024-04-20T21:31:53.297 INFO:tasks.workunit.client.0.smithi062.stdout:1/263: write dc/d15/f18 [4988213,9035] 0 2024-04-20T21:31:53.304 INFO:tasks.workunit.client.0.smithi062.stdout:4/214: dwrite da/dd/df/d27/d32/f37 [0,4194304] 0 2024-04-20T21:31:53.307 INFO:tasks.workunit.client.0.smithi062.stdout:8/277: mknod d7/d9/d1b/c61 0 2024-04-20T21:31:53.311 INFO:tasks.workunit.client.0.smithi062.stdout:9/240: rename d16/l47 to d16/d2f/l51 0 2024-04-20T21:31:53.315 INFO:tasks.workunit.client.0.smithi062.stdout:9/241: write fe [1588267,80097] 0 2024-04-20T21:31:53.318 INFO:tasks.workunit.client.0.smithi062.stdout:5/234: creat d8/f4f x:0 0 0 2024-04-20T21:31:53.322 INFO:tasks.workunit.client.0.smithi062.stdout:1/264: dread dc/d15/d20/d28/f2f [0,4194304] 0 2024-04-20T21:31:53.331 INFO:tasks.workunit.client.0.smithi062.stdout:4/215: truncate da/dd/f12 2282370 0 2024-04-20T21:31:53.333 INFO:tasks.workunit.client.0.smithi062.stdout:6/200: dwrite d0/d9/d11/d22/f24 [0,4194304] 0 2024-04-20T21:31:53.333 INFO:tasks.workunit.client.0.smithi062.stdout:8/278: mknod d7/d25/c62 0 2024-04-20T21:31:53.334 INFO:tasks.workunit.client.0.smithi062.stdout:9/242: symlink d16/l52 0 2024-04-20T21:31:53.339 INFO:tasks.workunit.client.0.smithi062.stdout:5/235: fsync d8/d1e/f24 0 2024-04-20T21:31:53.343 INFO:tasks.workunit.client.0.smithi062.stdout:4/216: fsync da/dd/df/d27/d32/f37 0 2024-04-20T21:31:53.344 INFO:tasks.workunit.client.0.smithi062.stdout:9/243: dread - d16/d3e/f40 zero size 2024-04-20T21:31:53.354 INFO:tasks.workunit.client.0.smithi062.stdout:7/192: dwrite d7/d3e/d1c/f1f [0,4194304] 0 2024-04-20T21:31:53.355 INFO:tasks.workunit.client.0.smithi062.stdout:2/222: dwrite d7/da/f3c [0,4194304] 0 2024-04-20T21:31:53.357 INFO:tasks.workunit.client.0.smithi062.stdout:0/230: dread d7/d1c/f23 [4194304,4194304] 0 2024-04-20T21:31:53.361 INFO:tasks.workunit.client.0.smithi062.stdout:0/231: chown d7/d1c 31 1 2024-04-20T21:31:53.361 INFO:tasks.workunit.client.0.smithi062.stdout:9/244: readlink d16/l26 0 2024-04-20T21:31:53.365 INFO:tasks.workunit.client.0.smithi062.stdout:1/265: dwrite dc/d15/d20/f45 [0,4194304] 0 2024-04-20T21:31:53.365 INFO:tasks.workunit.client.0.smithi062.stdout:3/267: dread d0/d1b/f2b [0,4194304] 0 2024-04-20T21:31:53.367 INFO:tasks.workunit.client.0.smithi062.stdout:2/223: rmdir d7/da/d15/d1b 39 2024-04-20T21:31:53.368 INFO:tasks.workunit.client.0.smithi062.stdout:8/279: creat d7/d25/d3b/d58/f63 x:0 0 0 2024-04-20T21:31:53.370 INFO:tasks.workunit.client.0.smithi062.stdout:3/268: chown d0/d1/d10 338 1 2024-04-20T21:31:53.372 INFO:tasks.workunit.client.0.smithi062.stdout:6/201: rename d0/d1a/c2d to d0/c42 0 2024-04-20T21:31:53.378 INFO:tasks.workunit.client.0.smithi062.stdout:0/232: write d7/d1c/d15/d17/f1d [1473134,88410] 0 2024-04-20T21:31:53.385 INFO:tasks.workunit.client.0.smithi062.stdout:6/202: symlink d0/d9/d11/l43 0 2024-04-20T21:31:53.396 INFO:tasks.workunit.client.0.smithi062.stdout:1/266: rename dc/d15/d17/f4c to dc/d15/d17/d36/f54 0 2024-04-20T21:31:53.397 INFO:tasks.workunit.client.0.smithi062.stdout:6/203: mkdir d0/d1a/d44 0 2024-04-20T21:31:53.400 INFO:tasks.workunit.client.0.smithi062.stdout:3/269: link d0/d11/l15 d0/d1c/d52/l61 0 2024-04-20T21:31:53.402 INFO:tasks.workunit.client.0.smithi062.stdout:4/217: dwrite da/dd/df/d27/d30/d3d/f40 [0,4194304] 0 2024-04-20T21:31:53.403 INFO:tasks.workunit.client.0.smithi062.stdout:2/224: truncate d7/f1e 4306206 0 2024-04-20T21:31:53.404 INFO:tasks.workunit.client.0.smithi062.stdout:7/193: dwrite d7/f10 [0,4194304] 0 2024-04-20T21:31:53.407 INFO:tasks.workunit.client.0.smithi062.stdout:2/225: stat d7/f1e 0 2024-04-20T21:31:53.412 INFO:tasks.workunit.client.0.smithi062.stdout:8/280: rename d7/d9/d1b/f21 to d7/d9/d1b/d24/d46/f64 0 2024-04-20T21:31:53.414 INFO:tasks.workunit.client.0.smithi062.stdout:0/233: link d7/d20/l2f d7/l4f 0 2024-04-20T21:31:53.417 INFO:tasks.workunit.client.0.smithi062.stdout:9/245: dwrite f4 [0,4194304] 0 2024-04-20T21:31:53.421 INFO:tasks.workunit.client.0.smithi062.stdout:1/267: creat dc/f55 x:0 0 0 2024-04-20T21:31:53.423 INFO:tasks.workunit.client.0.smithi062.stdout:8/281: truncate d7/d9/d1b/d24/d46/f49 1073137 0 2024-04-20T21:31:53.424 INFO:tasks.workunit.client.0.smithi062.stdout:7/194: rename d7/d26/l40 to d7/d3e/l42 0 2024-04-20T21:31:53.427 INFO:tasks.workunit.client.0.smithi062.stdout:6/204: dwrite d0/d9/f18 [0,4194304] 0 2024-04-20T21:31:53.427 INFO:tasks.workunit.client.0.smithi062.stdout:2/226: creat d7/da/d15/d1b/d24/f3f x:0 0 0 2024-04-20T21:31:53.429 INFO:tasks.workunit.client.0.smithi062.stdout:9/246: mkdir d16/d2c/d53 0 2024-04-20T21:31:53.434 INFO:tasks.workunit.client.0.smithi062.stdout:6/205: stat d0/d9/d11/l43 0 2024-04-20T21:31:53.434 INFO:tasks.workunit.client.0.smithi062.stdout:1/268: creat dc/d15/d17/d36/d3a/f56 x:0 0 0 2024-04-20T21:31:53.434 INFO:tasks.workunit.client.0.smithi062.stdout:3/270: dwrite d0/d1/d10/d31/f5f [0,4194304] 0 2024-04-20T21:31:53.438 INFO:tasks.workunit.client.0.smithi062.stdout:7/195: symlink d7/d3e/d33/d3d/l43 0 2024-04-20T21:31:53.438 INFO:tasks.workunit.client.0.smithi062.stdout:4/218: getdents da/dd/df/d27/d30/d3d 0 2024-04-20T21:31:53.439 INFO:tasks.workunit.client.0.smithi062.stdout:8/282: dread d7/d9/d29/f32 [0,4194304] 0 2024-04-20T21:31:53.440 INFO:tasks.workunit.client.0.smithi062.stdout:9/247: rename d16/d3e/f40 to d16/d2c/d42/f54 0 2024-04-20T21:31:53.444 INFO:tasks.workunit.client.0.smithi062.stdout:6/206: mknod d0/d9/d11/d22/d26/c45 0 2024-04-20T21:31:53.444 INFO:tasks.workunit.client.0.smithi062.stdout:1/269: mknod dc/d15/d20/c57 0 2024-04-20T21:31:53.444 INFO:tasks.workunit.client.0.smithi062.stdout:4/219: write f4 [2033979,92261] 0 2024-04-20T21:31:53.445 INFO:tasks.workunit.client.0.smithi062.stdout:3/271: readlink d0/d1/d3c/l50 0 2024-04-20T21:31:53.448 INFO:tasks.workunit.client.0.smithi062.stdout:6/207: read - d0/d1a/f29 zero size 2024-04-20T21:31:53.449 INFO:tasks.workunit.client.0.smithi062.stdout:4/220: chown da/dd/df 1621 1 2024-04-20T21:31:53.452 INFO:tasks.workunit.client.0.smithi062.stdout:7/196: link d7/fe d7/d3e/f44 0 2024-04-20T21:31:53.454 INFO:tasks.workunit.client.0.smithi062.stdout:6/208: write d0/d9/d11/d22/f27 [1161159,48618] 0 2024-04-20T21:31:53.454 INFO:tasks.workunit.client.0.smithi062.stdout:7/197: readlink d7/l16 0 2024-04-20T21:31:53.456 INFO:tasks.workunit.client.0.smithi062.stdout:8/283: write f1 [1097922,7306] 0 2024-04-20T21:31:53.456 INFO:tasks.workunit.client.0.smithi062.stdout:0/234: write d7/d1c/f11 [558351,122176] 0 2024-04-20T21:31:53.457 INFO:tasks.workunit.client.0.smithi062.stdout:7/198: fdatasync d7/f2a 0 2024-04-20T21:31:53.459 INFO:tasks.workunit.client.0.smithi062.stdout:3/272: creat d0/d1/d10/d3a/d5e/f62 x:0 0 0 2024-04-20T21:31:53.459 INFO:tasks.workunit.client.0.smithi062.stdout:4/221: symlink da/dd/df/l4a 0 2024-04-20T21:31:53.460 INFO:tasks.workunit.client.0.smithi062.stdout:8/284: dread - d7/d25/d3b/d58/f63 zero size 2024-04-20T21:31:53.462 INFO:tasks.workunit.client.0.smithi062.stdout:7/199: creat d7/d3e/f45 x:0 0 0 2024-04-20T21:31:53.462 INFO:tasks.workunit.client.0.smithi062.stdout:6/209: truncate d0/d1a/f29 210700 0 2024-04-20T21:31:53.465 INFO:tasks.workunit.client.0.smithi062.stdout:0/235: readlink d7/d1c/d15/d17/d26/l48 0 2024-04-20T21:31:53.469 INFO:tasks.workunit.client.0.smithi062.stdout:4/222: fdatasync da/dd/df/d27/d32/f37 0 2024-04-20T21:31:53.470 INFO:tasks.workunit.client.0.smithi062.stdout:7/200: truncate d7/fe 5639268 0 2024-04-20T21:31:53.472 INFO:tasks.workunit.client.0.smithi062.stdout:6/210: getdents d0/d1a/d44 0 2024-04-20T21:31:53.472 INFO:tasks.workunit.client.0.smithi062.stdout:1/270: dread dc/d15/d20/f33 [0,4194304] 0 2024-04-20T21:31:53.476 INFO:tasks.workunit.client.0.smithi062.stdout:4/223: mkdir da/dd/d15/d4b 0 2024-04-20T21:31:53.477 INFO:tasks.workunit.client.0.smithi062.stdout:5/236: sync 2024-04-20T21:31:53.479 INFO:tasks.workunit.client.0.smithi062.stdout:7/201: write d7/d26/f29 [1296342,113776] 0 2024-04-20T21:31:53.482 INFO:tasks.workunit.client.0.smithi062.stdout:0/236: dread d7/d1c/f39 [0,4194304] 0 2024-04-20T21:31:53.482 INFO:tasks.workunit.client.0.smithi062.stdout:6/211: write d0/d9/d11/d22/f27 [1221673,99289] 0 2024-04-20T21:31:53.489 INFO:tasks.workunit.client.0.smithi062.stdout:1/271: write dc/d15/f42 [3819242,85766] 0 2024-04-20T21:31:53.489 INFO:tasks.workunit.client.0.smithi062.stdout:7/202: truncate d7/d3e/f1a 6263502 0 2024-04-20T21:31:53.490 INFO:tasks.workunit.client.0.smithi062.stdout:0/237: symlink d7/d1c/d15/d17/l50 0 2024-04-20T21:31:53.491 INFO:tasks.workunit.client.0.smithi062.stdout:2/227: dread d7/da/f22 [0,4194304] 0 2024-04-20T21:31:53.492 INFO:tasks.workunit.client.0.smithi062.stdout:5/237: mkdir d8/d10/d2a/d50 0 2024-04-20T21:31:53.492 INFO:tasks.workunit.client.0.smithi062.stdout:4/224: creat da/dd/df/d27/d3f/f4c x:0 0 0 2024-04-20T21:31:53.493 INFO:tasks.workunit.client.0.smithi062.stdout:9/248: dread fd [0,4194304] 0 2024-04-20T21:31:53.496 INFO:tasks.workunit.client.0.smithi062.stdout:5/238: symlink d8/d10/d2a/d2b/l51 0 2024-04-20T21:31:53.499 INFO:tasks.workunit.client.0.smithi062.stdout:1/272: mknod dc/d15/d17/d36/c58 0 2024-04-20T21:31:53.500 INFO:tasks.workunit.client.0.smithi062.stdout:7/203: link d7/d3e/c38 d7/d3e/d1c/c46 0 2024-04-20T21:31:53.502 INFO:tasks.workunit.client.0.smithi062.stdout:4/225: readlink da/dd/df/d27/l29 0 2024-04-20T21:31:53.503 INFO:tasks.workunit.client.0.smithi062.stdout:3/273: dread d0/d1/f29 [0,4194304] 0 2024-04-20T21:31:53.503 INFO:tasks.workunit.client.0.smithi062.stdout:6/212: link d0/c2 d0/c46 0 2024-04-20T21:31:53.504 INFO:tasks.workunit.client.0.smithi062.stdout:1/273: readlink dc/d15/d17/d36/d3a/l52 0 2024-04-20T21:31:53.505 INFO:tasks.workunit.client.0.smithi062.stdout:5/239: read d8/ff [81996,5202] 0 2024-04-20T21:31:53.507 INFO:tasks.workunit.client.0.smithi062.stdout:7/204: creat d7/d26/f47 x:0 0 0 2024-04-20T21:31:53.507 INFO:tasks.workunit.client.0.smithi062.stdout:8/285: dread d7/d9/d1b/f3f [0,4194304] 0 2024-04-20T21:31:53.507 INFO:tasks.workunit.client.0.smithi062.stdout:6/213: creat d0/d9/f47 x:0 0 0 2024-04-20T21:31:53.508 INFO:tasks.workunit.client.0.smithi062.stdout:9/249: rmdir d16/d2c/d53 0 2024-04-20T21:31:53.511 INFO:tasks.workunit.client.0.smithi062.stdout:4/226: truncate da/dd/f19 5202668 0 2024-04-20T21:31:53.514 INFO:tasks.workunit.client.0.smithi062.stdout:1/274: unlink dc/d15/d20/d28/f2f 0 2024-04-20T21:31:53.516 INFO:tasks.workunit.client.0.smithi062.stdout:3/274: rename d0/d1/d10/d2c to d0/d1c/d63 0 2024-04-20T21:31:53.516 INFO:tasks.workunit.client.0.smithi062.stdout:5/240: creat d8/d10/d3e/f52 x:0 0 0 2024-04-20T21:31:53.517 INFO:tasks.workunit.client.0.smithi062.stdout:6/214: chown d0/d9/d11/f1f 55718 1 2024-04-20T21:31:53.517 INFO:tasks.workunit.client.0.smithi062.stdout:9/250: mkdir d16/d3d/d55 0 2024-04-20T21:31:53.520 INFO:tasks.workunit.client.0.smithi062.stdout:8/286: write d7/d9/d1b/f1f [2197128,29986] 0 2024-04-20T21:31:53.523 INFO:tasks.workunit.client.0.smithi062.stdout:0/238: dwrite d7/d1c/f18 [0,4194304] 0 2024-04-20T21:31:53.524 INFO:tasks.workunit.client.0.smithi062.stdout:2/228: dwrite d7/f1a [0,4194304] 0 2024-04-20T21:31:53.525 INFO:tasks.workunit.client.0.smithi062.stdout:9/251: mkdir d16/d3d/d56 0 2024-04-20T21:31:53.530 INFO:tasks.workunit.client.0.smithi062.stdout:6/215: chown d0/d9/f18 0 1 2024-04-20T21:31:53.534 INFO:tasks.workunit.client.0.smithi062.stdout:1/275: rename dc/d15/d17/d36/d3a/f56 to dc/d15/d17/d36/d3a/f59 0 2024-04-20T21:31:53.537 INFO:tasks.workunit.client.0.smithi062.stdout:0/239: rmdir d7/d1c 39 2024-04-20T21:31:53.541 INFO:tasks.workunit.client.0.smithi062.stdout:8/287: symlink d7/d25/d3b/l65 0 2024-04-20T21:31:53.543 INFO:tasks.workunit.client.0.smithi062.stdout:7/205: dwrite d7/f18 [0,4194304] 0 2024-04-20T21:31:53.546 INFO:tasks.workunit.client.0.smithi062.stdout:4/227: dread da/dd/f14 [0,4194304] 0 2024-04-20T21:31:53.547 INFO:tasks.workunit.client.0.smithi062.stdout:1/276: creat dc/d15/d17/d36/d3a/f5a x:0 0 0 2024-04-20T21:31:53.548 INFO:tasks.workunit.client.0.smithi062.stdout:9/252: getdents d16/d3d/d55 0 2024-04-20T21:31:53.549 INFO:tasks.workunit.client.0.smithi062.stdout:8/288: fsync d7/d9/d29/f4c 0 2024-04-20T21:31:53.550 INFO:tasks.workunit.client.0.smithi062.stdout:3/275: getdents d0/d1/d10/d3a/d5e 0 2024-04-20T21:31:53.557 INFO:tasks.workunit.client.0.smithi062.stdout:3/276: read d0/d1/f29 [2742256,124242] 0 2024-04-20T21:31:53.561 INFO:tasks.workunit.client.0.smithi062.stdout:7/206: write d7/d3e/d33/f35 [659424,66362] 0 2024-04-20T21:31:53.561 INFO:tasks.workunit.client.0.smithi062.stdout:5/241: dwrite d8/fb [0,4194304] 0 2024-04-20T21:31:53.561 INFO:tasks.workunit.client.0.smithi062.stdout:4/228: rmdir da/dd/d15 39 2024-04-20T21:31:53.562 INFO:tasks.workunit.client.0.smithi062.stdout:9/253: dread fd [4194304,4194304] 0 2024-04-20T21:31:53.563 INFO:tasks.workunit.client.0.smithi062.stdout:0/240: mknod d7/d1c/d15/d17/d1b/d4e/c51 0 2024-04-20T21:31:53.569 INFO:tasks.workunit.client.0.smithi062.stdout:3/277: creat d0/d1/d10/f64 x:0 0 0 2024-04-20T21:31:53.573 INFO:tasks.workunit.client.0.smithi062.stdout:6/216: dwrite d0/fe [0,4194304] 0 2024-04-20T21:31:53.574 INFO:tasks.workunit.client.0.smithi062.stdout:9/254: fsync d16/d2f/f39 0 2024-04-20T21:31:53.575 INFO:tasks.workunit.client.0.smithi062.stdout:7/207: fdatasync d7/d3e/d1c/f1f 0 2024-04-20T21:31:53.576 INFO:tasks.workunit.client.0.smithi062.stdout:2/229: dwrite d7/da/d15/d1b/d24/f3f [0,4194304] 0 2024-04-20T21:31:53.581 INFO:tasks.workunit.client.0.smithi062.stdout:2/230: chown d7/l16 9308 1 2024-04-20T21:31:53.583 INFO:tasks.workunit.client.0.smithi062.stdout:1/277: truncate dc/d15/f18 3587711 0 2024-04-20T21:31:53.585 INFO:tasks.workunit.client.0.smithi062.stdout:2/231: readlink d7/da/l2b 0 2024-04-20T21:31:53.586 INFO:tasks.workunit.client.0.smithi062.stdout:2/232: chown d7/l8 856 1 2024-04-20T21:31:53.587 INFO:tasks.workunit.client.0.smithi062.stdout:9/255: truncate d16/d2c/d42/f54 423744 0 2024-04-20T21:31:53.587 INFO:tasks.workunit.client.0.smithi062.stdout:0/241: dread d7/d1c/f3e [0,4194304] 0 2024-04-20T21:31:53.588 INFO:tasks.workunit.client.0.smithi062.stdout:4/229: creat da/dd/f4d x:0 0 0 2024-04-20T21:31:53.589 INFO:tasks.workunit.client.0.smithi062.stdout:5/242: link f4 d8/d1e/f53 0 2024-04-20T21:31:53.589 INFO:tasks.workunit.client.0.smithi062.stdout:8/289: dwrite d7/d9/d29/f40 [0,4194304] 0 2024-04-20T21:31:53.592 INFO:tasks.workunit.client.0.smithi062.stdout:1/278: read f9 [9731,38207] 0 2024-04-20T21:31:53.596 INFO:tasks.workunit.client.0.smithi062.stdout:6/217: link d0/d9/f47 d0/d9/d11/f48 0 2024-04-20T21:31:53.596 INFO:tasks.workunit.client.0.smithi062.stdout:5/243: write d8/f4f [216349,53663] 0 2024-04-20T21:31:53.601 INFO:tasks.workunit.client.0.smithi062.stdout:0/242: mknod d7/d1c/d15/d17/d40/c52 0 2024-04-20T21:31:53.603 INFO:tasks.workunit.client.0.smithi062.stdout:5/244: write d8/f44 [5003517,75836] 0 2024-04-20T21:31:53.603 INFO:tasks.workunit.client.0.smithi062.stdout:8/290: unlink d7/d9/d1b/c27 0 2024-04-20T21:31:53.607 INFO:tasks.workunit.client.0.smithi062.stdout:2/233: dread d7/ff [4194304,4194304] 0 2024-04-20T21:31:53.608 INFO:tasks.workunit.client.0.smithi062.stdout:9/256: unlink d16/d2f/c4b 0 2024-04-20T21:31:53.611 INFO:tasks.workunit.client.0.smithi062.stdout:8/291: fdatasync d7/d9/d29/f2b 0 2024-04-20T21:31:53.613 INFO:tasks.workunit.client.0.smithi062.stdout:0/243: mkdir d7/d1c/d15/d53 0 2024-04-20T21:31:53.614 INFO:tasks.workunit.client.0.smithi062.stdout:6/218: symlink d0/l49 0 2024-04-20T21:31:53.615 INFO:tasks.workunit.client.0.smithi062.stdout:7/208: dwrite d7/f12 [0,4194304] 0 2024-04-20T21:31:53.616 INFO:tasks.workunit.client.0.smithi062.stdout:9/257: symlink d16/d2c/d42/l57 0 2024-04-20T21:31:53.619 INFO:tasks.workunit.client.0.smithi062.stdout:1/279: link dc/d15/d20/f3c dc/d15/f5b 0 2024-04-20T21:31:53.619 INFO:tasks.workunit.client.0.smithi062.stdout:3/278: dwrite d0/d1/f19 [0,4194304] 0 2024-04-20T21:31:53.621 INFO:tasks.workunit.client.0.smithi062.stdout:0/244: write d7/d1c/d15/d17/d1b/f1a [1247816,6836] 0 2024-04-20T21:31:53.627 INFO:tasks.workunit.client.0.smithi062.stdout:9/258: fdatasync d16/f3b 0 2024-04-20T21:31:53.632 INFO:tasks.workunit.client.0.smithi062.stdout:2/234: link d7/l18 d7/da/d15/l40 0 2024-04-20T21:31:53.635 INFO:tasks.workunit.client.0.smithi062.stdout:4/230: dwrite da/f16 [0,4194304] 0 2024-04-20T21:31:53.639 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:53 smithi062 ceph-mon[16221]: pgmap v11: 65 pgs: 65 active+clean; 290 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.4 MiB/s rd, 11 MiB/s wr, 215 op/s 2024-04-20T21:31:53.641 INFO:tasks.workunit.client.0.smithi062.stdout:7/209: creat d7/f48 x:0 0 0 2024-04-20T21:31:53.641 INFO:tasks.workunit.client.0.smithi062.stdout:3/279: mkdir d0/d1/d10/d3a/d5e/d65 0 2024-04-20T21:31:53.642 INFO:tasks.workunit.client.0.smithi062.stdout:7/210: fdatasync d7/f24 0 2024-04-20T21:31:53.643 INFO:tasks.workunit.client.0.smithi062.stdout:0/245: symlink d7/d20/d3a/l54 0 2024-04-20T21:31:53.644 INFO:tasks.workunit.client.0.smithi062.stdout:4/231: symlink da/dd/d3c/l4e 0 2024-04-20T21:31:53.647 INFO:tasks.workunit.client.0.smithi062.stdout:9/259: mknod d16/c58 0 2024-04-20T21:31:53.649 INFO:tasks.workunit.client.0.smithi062.stdout:3/280: unlink d0/d1b/c36 0 2024-04-20T21:31:53.655 INFO:tasks.workunit.client.0.smithi062.stdout:7/211: rename d7/f27 to d7/d3e/d33/f49 0 2024-04-20T21:31:53.657 INFO:tasks.workunit.client.0.smithi062.stdout:0/246: mknod d7/d31/c55 0 2024-04-20T21:31:53.661 INFO:tasks.workunit.client.0.smithi062.stdout:0/247: chown d7/d1c/d15/f46 183537123 1 2024-04-20T21:31:53.666 INFO:tasks.workunit.client.0.smithi062.stdout:1/280: write dc/d15/f18 [778522,74048] 0 2024-04-20T21:31:53.674 INFO:tasks.workunit.client.0.smithi062.stdout:4/232: dwrite da/dd/df/d27/d30/d3d/f40 [4194304,4194304] 0 2024-04-20T21:31:53.675 INFO:tasks.workunit.client.0.smithi062.stdout:1/281: rmdir dc/d15/d17/d36 39 2024-04-20T21:31:53.675 INFO:tasks.workunit.client.0.smithi062.stdout:0/248: symlink d7/d1c/d15/d17/d40/d3c/l56 0 2024-04-20T21:31:53.677 INFO:tasks.workunit.client.0.smithi062.stdout:6/219: dread d0/d9/f13 [0,4194304] 0 2024-04-20T21:31:53.678 INFO:tasks.workunit.client.0.smithi062.stdout:4/233: stat da/dd/f45 0 2024-04-20T21:31:53.682 INFO:tasks.workunit.client.0.smithi062.stdout:1/282: fdatasync dc/d15/d20/f45 0 2024-04-20T21:31:53.683 INFO:tasks.workunit.client.0.smithi062.stdout:3/281: dwrite d0/f53 [0,4194304] 0 2024-04-20T21:31:53.685 INFO:tasks.workunit.client.0.smithi062.stdout:4/234: dread - da/dd/f4d zero size 2024-04-20T21:31:53.689 INFO:tasks.workunit.client.0.smithi062.stdout:6/220: truncate d0/d1a/f29 790374 0 2024-04-20T21:31:53.689 INFO:tasks.workunit.client.0.smithi062.stdout:9/260: dwrite fd [4194304,4194304] 0 2024-04-20T21:31:53.694 INFO:tasks.workunit.client.0.smithi062.stdout:0/249: link d7/d1c/f11 d7/d31/f57 0 2024-04-20T21:31:53.694 INFO:tasks.workunit.client.0.smithi062.stdout:1/283: dread dc/d15/d17/f1e [0,4194304] 0 2024-04-20T21:31:53.697 INFO:tasks.workunit.client.0.smithi062.stdout:2/235: dread d7/f17 [0,4194304] 0 2024-04-20T21:31:53.704 INFO:tasks.workunit.client.0.smithi062.stdout:7/212: dwrite d7/d3e/d1c/f28 [4194304,4194304] 0 2024-04-20T21:31:53.704 INFO:tasks.workunit.client.0.smithi062.stdout:2/236: chown d7/c1c 45517999 1 2024-04-20T21:31:53.704 INFO:tasks.workunit.client.0.smithi062.stdout:1/284: dread - dc/f35 zero size 2024-04-20T21:31:53.704 INFO:tasks.workunit.client.0.smithi062.stdout:0/250: chown d7/d1c/d15/d17/l3f 336 1 2024-04-20T21:31:53.709 INFO:tasks.workunit.client.0.smithi062.stdout:9/261: link d16/d2c/l33 d16/d2c/l59 0 2024-04-20T21:31:53.714 INFO:tasks.workunit.client.0.smithi062.stdout:7/213: unlink f5 0 2024-04-20T21:31:53.722 INFO:tasks.workunit.client.0.smithi062.stdout:7/214: write d7/d3e/d1c/f2e [694883,55408] 0 2024-04-20T21:31:53.723 INFO:tasks.workunit.client.0.smithi062.stdout:3/282: dwrite d0/d1/d10/f4e [0,4194304] 0 2024-04-20T21:31:53.733 INFO:tasks.workunit.client.0.smithi062.stdout:9/262: rename f6 to d16/d3d/d55/f5a 0 2024-04-20T21:31:53.734 INFO:tasks.workunit.client.0.smithi062.stdout:2/237: link d7/da/d15/d1b/l3a d7/da/d15/d1b/l41 0 2024-04-20T21:31:53.739 INFO:tasks.workunit.client.0.smithi062.stdout:6/221: dwrite d0/d2c/f30 [0,4194304] 0 2024-04-20T21:31:53.739 INFO:tasks.workunit.client.0.smithi062.stdout:1/285: link dc/c1f dc/d15/d17/c5c 0 2024-04-20T21:31:53.745 INFO:tasks.workunit.client.0.smithi062.stdout:2/238: write d7/da/d15/d1b/d24/f3f [3497713,4744] 0 2024-04-20T21:31:53.745 INFO:tasks.workunit.client.0.smithi062.stdout:6/222: read d0/f17 [2725886,23642] 0 2024-04-20T21:31:53.746 INFO:tasks.workunit.client.0.smithi062.stdout:7/215: stat d7/d3e/d1c 0 2024-04-20T21:31:53.746 INFO:tasks.workunit.client.0.smithi062.stdout:1/286: chown dc/f53 4083643 1 2024-04-20T21:31:53.748 INFO:tasks.workunit.client.0.smithi062.stdout:0/251: dwrite d7/d1c/f39 [0,4194304] 0 2024-04-20T21:31:53.752 INFO:tasks.workunit.client.0.smithi062.stdout:6/223: write d0/d1a/f29 [295685,5320] 0 2024-04-20T21:31:53.752 INFO:tasks.workunit.client.0.smithi062.stdout:9/263: unlink cf 0 2024-04-20T21:31:53.757 INFO:tasks.workunit.client.0.smithi062.stdout:7/216: creat d7/d3e/d33/d3d/f4a x:0 0 0 2024-04-20T21:31:53.758 INFO:tasks.workunit.client.0.smithi062.stdout:3/283: link d0/d1c/d63/l2e d0/d1/d10/l66 0 2024-04-20T21:31:53.759 INFO:tasks.workunit.client.0.smithi062.stdout:0/252: dread d7/d1c/d15/f2d [0,4194304] 0 2024-04-20T21:31:53.759 INFO:tasks.workunit.client.0.smithi062.stdout:3/284: write d0/d1/d10/f4e [3245240,13431] 0 2024-04-20T21:31:53.760 INFO:tasks.workunit.client.0.smithi062.stdout:1/287: creat dc/d15/d17/d36/f5d x:0 0 0 2024-04-20T21:31:53.762 INFO:tasks.workunit.client.0.smithi062.stdout:6/224: symlink d0/d1a/d44/l4a 0 2024-04-20T21:31:53.837 INFO:tasks.workunit.client.0.smithi062.stdout:0/253: creat d7/d1c/d15/d17/d40/f58 x:0 0 0 2024-04-20T21:31:53.837 INFO:tasks.workunit.client.0.smithi062.stdout:3/285: mkdir d0/d1/d3c/d43/d67 0 2024-04-20T21:31:53.840 INFO:tasks.workunit.client.0.smithi062.stdout:2/239: write d7/f12 [1365696,47098] 0 2024-04-20T21:31:53.844 INFO:tasks.workunit.client.0.smithi062.stdout:7/217: rename d7/l15 to d7/d26/l4b 0 2024-04-20T21:31:53.844 INFO:tasks.workunit.client.0.smithi062.stdout:3/286: read d0/f4f [73116,126941] 0 2024-04-20T21:31:53.848 INFO:tasks.workunit.client.0.smithi062.stdout:2/240: creat d7/da/d15/d2f/f42 x:0 0 0 2024-04-20T21:31:53.852 INFO:tasks.workunit.client.0.smithi062.stdout:1/288: dwrite dc/fe [0,4194304] 0 2024-04-20T21:31:53.852 INFO:tasks.workunit.client.0.smithi062.stdout:7/218: symlink d7/d26/d39/l4c 0 2024-04-20T21:31:53.857 INFO:tasks.workunit.client.0.smithi062.stdout:1/289: write dc/d15/f48 [925530,113146] 0 2024-04-20T21:31:53.858 INFO:tasks.workunit.client.0.smithi062.stdout:2/241: mknod d7/da/d15/d2f/d38/c43 0 2024-04-20T21:31:53.863 INFO:tasks.workunit.client.0.smithi062.stdout:7/219: mkdir d7/d3e/d33/d3d/d4d 0 2024-04-20T21:31:53.865 INFO:tasks.workunit.client.0.smithi062.stdout:6/225: dwrite d0/d2c/f2e [0,4194304] 0 2024-04-20T21:31:53.866 INFO:tasks.workunit.client.0.smithi062.stdout:1/290: symlink dc/d15/d20/l5e 0 2024-04-20T21:31:53.871 INFO:tasks.workunit.client.0.smithi062.stdout:7/220: write d7/d3e/d33/d3d/f4a [1016310,118098] 0 2024-04-20T21:31:53.872 INFO:tasks.workunit.client.0.smithi062.stdout:1/291: creat dc/d15/d17/d36/f5f x:0 0 0 2024-04-20T21:31:53.873 INFO:tasks.workunit.client.0.smithi062.stdout:0/254: dread d7/d1c/f23 [0,4194304] 0 2024-04-20T21:31:53.874 INFO:tasks.workunit.client.0.smithi062.stdout:2/242: rename d7/da/d15/d2f/l37 to d7/da/d15/d2f/l44 0 2024-04-20T21:31:53.874 INFO:tasks.workunit.client.0.smithi062.stdout:3/287: dwrite d0/d1b/f2b [4194304,4194304] 0 2024-04-20T21:31:53.875 INFO:tasks.workunit.client.0.smithi062.stdout:1/292: readlink dc/d15/d20/l3b 0 2024-04-20T21:31:53.876 INFO:tasks.workunit.client.0.smithi062.stdout:6/226: creat d0/d1a/d44/f4b x:0 0 0 2024-04-20T21:31:53.877 INFO:tasks.workunit.client.0.smithi062.stdout:1/293: chown dc/d15/d20/d28 1 1 2024-04-20T21:31:53.882 INFO:tasks.workunit.client.0.smithi062.stdout:2/243: creat d7/d39/f45 x:0 0 0 2024-04-20T21:31:53.884 INFO:tasks.workunit.client.0.smithi062.stdout:2/244: rename d7/da to d7/da/d15/d2f/d46 22 2024-04-20T21:31:53.887 INFO:tasks.workunit.client.0.smithi062.stdout:1/294: creat dc/d15/d20/d28/f60 x:0 0 0 2024-04-20T21:31:53.891 INFO:tasks.workunit.client.0.smithi062.stdout:1/295: chown dc/d15/d20/f3e 1922703 1 2024-04-20T21:31:53.897 INFO:tasks.workunit.client.0.smithi062.stdout:8/292: sync 2024-04-20T21:31:53.897 INFO:tasks.workunit.client.0.smithi062.stdout:9/264: sync 2024-04-20T21:31:53.898 INFO:tasks.workunit.client.0.smithi062.stdout:5/245: sync 2024-04-20T21:31:53.898 INFO:tasks.workunit.client.0.smithi062.stdout:4/235: sync 2024-04-20T21:31:53.898 INFO:tasks.workunit.client.0.smithi062.stdout:7/221: dwrite d7/f18 [4194304,4194304] 0 2024-04-20T21:31:53.898 INFO:tasks.workunit.client.0.smithi062.stdout:3/288: dwrite d0/d1/f32 [0,4194304] 0 2024-04-20T21:31:53.899 INFO:tasks.workunit.client.0.smithi062.stdout:1/296: mknod dc/d15/d20/c61 0 2024-04-20T21:31:53.901 INFO:tasks.workunit.client.0.smithi062.stdout:6/227: write d0/f17 [1967325,22515] 0 2024-04-20T21:31:53.903 INFO:tasks.workunit.client.0.smithi062.stdout:3/289: write d0/d1/d10/f64 [869064,114520] 0 2024-04-20T21:31:53.904 INFO:tasks.workunit.client.0.smithi062.stdout:8/293: truncate d7/d25/d3b/d58/f63 296637 0 2024-04-20T21:31:53.905 INFO:tasks.workunit.client.0.smithi062.stdout:3/290: chown d0/d1/d10/d31 90 1 2024-04-20T21:31:53.905 INFO:tasks.workunit.client.0.smithi062.stdout:5/246: rmdir d8/d10/d3e 39 2024-04-20T21:31:53.908 INFO:tasks.workunit.client.0.smithi062.stdout:7/222: fsync d7/f12 0 2024-04-20T21:31:53.911 INFO:tasks.workunit.client.0.smithi062.stdout:1/297: rename dc/d15/d20/f33 to dc/d27/f62 0 2024-04-20T21:31:53.913 INFO:tasks.workunit.client.0.smithi062.stdout:2/245: write d7/ff [1803020,4058] 0 2024-04-20T21:31:53.913 INFO:tasks.workunit.client.0.smithi062.stdout:7/223: chown d7/d26 41 1 2024-04-20T21:31:53.917 INFO:tasks.workunit.client.0.smithi062.stdout:7/224: fdatasync d7/d26/f34 0 2024-04-20T21:31:53.917 INFO:tasks.workunit.client.0.smithi062.stdout:3/291: truncate d0/d1/fd 1452833 0 2024-04-20T21:31:53.921 INFO:tasks.workunit.client.0.smithi062.stdout:8/294: read - d7/d9/d1b/d24/f48 zero size 2024-04-20T21:31:53.922 INFO:tasks.workunit.client.0.smithi062.stdout:1/298: rmdir dc/d15/d20/d28 39 2024-04-20T21:31:53.924 INFO:tasks.workunit.client.0.smithi062.stdout:8/295: write d7/f23 [4523294,113283] 0 2024-04-20T21:31:53.925 INFO:tasks.workunit.client.0.smithi062.stdout:2/246: symlink d7/da/d15/d1b/l47 0 2024-04-20T21:31:53.928 INFO:tasks.workunit.client.0.smithi062.stdout:0/255: dwrite d7/d1c/f32 [0,4194304] 0 2024-04-20T21:31:53.929 INFO:tasks.workunit.client.0.smithi062.stdout:1/299: chown dc/d15/d17/c39 727 1 2024-04-20T21:31:53.929 INFO:tasks.workunit.client.0.smithi062.stdout:6/228: truncate d0/f3 3053740 0 2024-04-20T21:31:53.930 INFO:tasks.workunit.client.0.smithi062.stdout:7/225: chown d7/l1d 7082273 1 2024-04-20T21:31:53.930 INFO:tasks.workunit.client.0.smithi062.stdout:9/265: getdents d16/d3d 0 2024-04-20T21:31:53.940 INFO:tasks.workunit.client.0.smithi062.stdout:2/247: mknod d7/da/d15/c48 0 2024-04-20T21:31:53.942 INFO:tasks.workunit.client.0.smithi062.stdout:1/300: write dc/d15/f48 [518300,49804] 0 2024-04-20T21:31:53.943 INFO:tasks.workunit.client.0.smithi062.stdout:0/256: creat d7/d1c/d15/d17/d37/f59 x:0 0 0 2024-04-20T21:31:53.947 INFO:tasks.workunit.client.0.smithi062.stdout:7/226: truncate d7/f24 900889 0 2024-04-20T21:31:53.947 INFO:tasks.workunit.client.0.smithi062.stdout:9/266: write d16/f3b [2720374,111463] 0 2024-04-20T21:31:53.950 INFO:tasks.workunit.client.0.smithi062.stdout:7/227: stat d7/d3e/d33/d3d/f4a 0 2024-04-20T21:31:53.951 INFO:tasks.workunit.client.0.smithi062.stdout:2/248: unlink d7/l18 0 2024-04-20T21:31:53.951 INFO:tasks.workunit.client.0.smithi062.stdout:3/292: dwrite d0/f53 [4194304,4194304] 0 2024-04-20T21:31:53.953 INFO:tasks.workunit.client.0.smithi062.stdout:7/228: chown d7/d26 433679401 1 2024-04-20T21:31:53.956 INFO:tasks.workunit.client.0.smithi062.stdout:0/257: read f2 [2965704,71646] 0 2024-04-20T21:31:53.958 INFO:tasks.workunit.client.0.smithi062.stdout:4/236: dread da/dd/f35 [0,4194304] 0 2024-04-20T21:31:53.958 INFO:tasks.workunit.client.0.smithi062.stdout:7/229: symlink d7/d3e/l4e 0 2024-04-20T21:31:53.960 INFO:tasks.workunit.client.0.smithi062.stdout:3/293: creat d0/d1c/d63/d42/d44/f68 x:0 0 0 2024-04-20T21:31:53.962 INFO:tasks.workunit.client.0.smithi062.stdout:6/229: dwrite d0/fe [0,4194304] 0 2024-04-20T21:31:53.966 INFO:tasks.workunit.client.0.smithi062.stdout:6/230: read - d0/d1a/d44/f4b zero size 2024-04-20T21:31:53.973 INFO:tasks.workunit.client.0.smithi062.stdout:5/247: dread d8/d1e/f25 [0,4194304] 0 2024-04-20T21:31:53.975 INFO:tasks.workunit.client.0.smithi062.stdout:7/230: rename d7/d26/d39 to d7/d3e/d1c/d4f 0 2024-04-20T21:31:53.978 INFO:tasks.workunit.client.0.smithi062.stdout:8/296: dread d7/d9/f17 [0,4194304] 0 2024-04-20T21:31:53.981 INFO:tasks.workunit.client.0.smithi062.stdout:4/237: write da/dd/df/f1a [4715048,86799] 0 2024-04-20T21:31:53.983 INFO:tasks.workunit.client.0.smithi062.stdout:7/231: mknod d7/d26/c50 0 2024-04-20T21:31:53.984 INFO:tasks.workunit.client.0.smithi062.stdout:5/248: read f4 [3226466,44970] 0 2024-04-20T21:31:53.986 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:53 smithi189 ceph-mon[21562]: pgmap v11: 65 pgs: 65 active+clean; 290 MiB data, 1024 MiB used, 535 GiB / 536 GiB avail; 1.4 MiB/s rd, 11 MiB/s wr, 215 op/s 2024-04-20T21:31:53.987 INFO:tasks.workunit.client.0.smithi062.stdout:3/294: link d0/d1b/l38 d0/d1c/d63/d42/l69 0 2024-04-20T21:31:53.991 INFO:tasks.workunit.client.0.smithi062.stdout:3/295: chown d0/d1b 356354222 1 2024-04-20T21:31:53.992 INFO:tasks.workunit.client.0.smithi062.stdout:8/297: write d7/d9/d1b/f4e [5134896,29028] 0 2024-04-20T21:31:53.992 INFO:tasks.workunit.client.0.smithi062.stdout:2/249: dwrite d7/da/f22 [0,4194304] 0 2024-04-20T21:31:53.995 INFO:tasks.workunit.client.0.smithi062.stdout:4/238: write da/dd/f35 [3047779,122268] 0 2024-04-20T21:31:54.006 INFO:tasks.workunit.client.0.smithi062.stdout:1/301: dwrite dc/d15/f16 [0,4194304] 0 2024-04-20T21:31:54.007 INFO:tasks.workunit.client.0.smithi062.stdout:6/231: dread d0/fe [0,4194304] 0 2024-04-20T21:31:54.007 INFO:tasks.workunit.client.0.smithi062.stdout:3/296: symlink d0/d1/d10/d3a/l6a 0 2024-04-20T21:31:54.008 INFO:tasks.workunit.client.0.smithi062.stdout:5/249: creat d8/d10/d2a/d2b/d4c/f54 x:0 0 0 2024-04-20T21:31:54.008 INFO:tasks.workunit.client.0.smithi062.stdout:0/258: dwrite d7/d1c/f23 [4194304,4194304] 0 2024-04-20T21:31:54.009 INFO:tasks.workunit.client.0.smithi062.stdout:9/267: dread fb [0,4194304] 0 2024-04-20T21:31:54.009 INFO:tasks.workunit.client.0.smithi062.stdout:7/232: rename d7/c8 to d7/d3e/d33/d3d/c51 0 2024-04-20T21:31:54.014 INFO:tasks.workunit.client.0.smithi062.stdout:4/239: chown da/dd/f1c 16688604 1 2024-04-20T21:31:54.015 INFO:tasks.workunit.client.0.smithi062.stdout:2/250: chown d7/da/d15/f26 14865 1 2024-04-20T21:31:54.017 INFO:tasks.workunit.client.0.smithi062.stdout:3/297: rmdir d0/d1/d3c 39 2024-04-20T21:31:54.017 INFO:tasks.workunit.client.0.smithi062.stdout:7/233: rmdir d7/d3e/d1c/d4f 39 2024-04-20T21:31:54.020 INFO:tasks.workunit.client.0.smithi062.stdout:9/268: mkdir d16/d2f/d5b 0 2024-04-20T21:31:54.022 INFO:tasks.workunit.client.0.smithi062.stdout:4/240: rmdir da 39 2024-04-20T21:31:54.022 INFO:tasks.workunit.client.0.smithi062.stdout:7/234: write d7/d26/f31 [375291,123777] 0 2024-04-20T21:31:54.022 INFO:tasks.workunit.client.0.smithi062.stdout:3/298: readlink d0/d1/d10/d31/l3d 0 2024-04-20T21:31:54.022 INFO:tasks.workunit.client.0.smithi062.stdout:5/250: creat d8/d10/d2a/d2b/d4c/f55 x:0 0 0 2024-04-20T21:31:54.023 INFO:tasks.workunit.client.0.smithi062.stdout:8/298: dread d7/d9/f18 [0,4194304] 0 2024-04-20T21:31:54.024 INFO:tasks.workunit.client.0.smithi062.stdout:3/299: fsync d0/d1b/f4d 0 2024-04-20T21:31:54.030 INFO:tasks.workunit.client.0.smithi062.stdout:1/302: creat dc/d15/d20/d28/f63 x:0 0 0 2024-04-20T21:31:54.031 INFO:tasks.workunit.client.0.smithi062.stdout:2/251: write d7/da/d15/d1b/f34 [645477,77534] 0 2024-04-20T21:31:54.031 INFO:tasks.workunit.client.0.smithi062.stdout:6/232: link d0/d9/d11/f1f d0/d1a/d44/f4c 0 2024-04-20T21:31:54.039 INFO:tasks.workunit.client.0.smithi062.stdout:1/303: mknod dc/d27/d32/c64 0 2024-04-20T21:31:54.039 INFO:tasks.workunit.client.0.smithi062.stdout:0/259: dread d7/d1c/d15/d17/f1d [0,4194304] 0 2024-04-20T21:31:54.041 INFO:tasks.workunit.client.0.smithi062.stdout:3/300: rename d0/d1/le to d0/d5c/l6b 0 2024-04-20T21:31:54.041 INFO:tasks.workunit.client.0.smithi062.stdout:2/252: chown d7/da/d15/d2f/d38/c43 1321842 1 2024-04-20T21:31:54.043 INFO:tasks.workunit.client.0.smithi062.stdout:8/299: getdents d7/d25/d3b 0 2024-04-20T21:31:54.043 INFO:tasks.workunit.client.0.smithi062.stdout:1/304: stat dc/d15/d17/d36/f5f 0 2024-04-20T21:31:54.045 INFO:tasks.workunit.client.0.smithi062.stdout:0/260: stat d7/d1c/d15/d17/d1b/l49 0 2024-04-20T21:31:54.047 INFO:tasks.workunit.client.0.smithi062.stdout:8/300: chown d7/d9/d1b/c61 29392027 1 2024-04-20T21:31:54.048 INFO:tasks.workunit.client.0.smithi062.stdout:1/305: creat dc/d2d/f65 x:0 0 0 2024-04-20T21:31:54.049 INFO:tasks.workunit.client.0.smithi062.stdout:5/251: getdents d8/d10 0 2024-04-20T21:31:54.053 INFO:tasks.workunit.client.0.smithi062.stdout:0/261: write d7/f47 [99188,8357] 0 2024-04-20T21:31:54.058 INFO:tasks.workunit.client.0.smithi062.stdout:3/301: mknod d0/d1c/d63/d42/d5b/c6c 0 2024-04-20T21:31:54.058 INFO:tasks.workunit.client.0.smithi062.stdout:8/301: rmdir d7/d25 39 2024-04-20T21:31:54.060 INFO:tasks.workunit.client.0.smithi062.stdout:3/302: stat d0/d1/d10/d31/f5f 0 2024-04-20T21:31:54.062 INFO:tasks.workunit.client.0.smithi062.stdout:0/262: rename d7/d1c/d15/d17 to d7/d1c/d15/d17/d5a 22 2024-04-20T21:31:54.062 INFO:tasks.workunit.client.0.smithi062.stdout:9/269: dwrite d16/f1e [0,4194304] 0 2024-04-20T21:31:54.062 INFO:tasks.workunit.client.0.smithi062.stdout:3/303: fsync d0/d11/f56 0 2024-04-20T21:31:54.063 INFO:tasks.workunit.client.0.smithi062.stdout:8/302: rename d7 to d7/d25/d4f/d66 22 2024-04-20T21:31:54.065 INFO:tasks.workunit.client.0.smithi062.stdout:0/263: chown d7/d1e/c36 15245515 1 2024-04-20T21:31:54.067 INFO:tasks.workunit.client.0.smithi062.stdout:1/306: link dc/d15/f18 dc/d15/d20/d28/f66 0 2024-04-20T21:31:54.068 INFO:tasks.workunit.client.0.smithi062.stdout:3/304: unlink d0/d1/d10/c1f 0 2024-04-20T21:31:54.072 INFO:tasks.workunit.client.0.smithi062.stdout:0/264: fsync d7/f4c 0 2024-04-20T21:31:54.073 INFO:tasks.workunit.client.0.smithi062.stdout:8/303: readlink d7/d9/l2e 0 2024-04-20T21:31:54.086 INFO:tasks.workunit.client.0.smithi062.stdout:5/252: dwrite d8/ff [0,4194304] 0 2024-04-20T21:31:54.086 INFO:tasks.workunit.client.0.smithi062.stdout:1/307: rename dc/f26 to dc/d15/d17/f67 0 2024-04-20T21:31:54.091 INFO:tasks.workunit.client.0.smithi062.stdout:7/235: dread d7/d3e/d1c/f28 [0,4194304] 0 2024-04-20T21:31:54.092 INFO:tasks.workunit.client.0.smithi062.stdout:5/253: fsync d8/d10/d2a/f39 0 2024-04-20T21:31:54.095 INFO:tasks.workunit.client.0.smithi062.stdout:0/265: link d7/d1c/f18 d7/d1c/d15/d17/d37/f5b 0 2024-04-20T21:31:54.099 INFO:tasks.workunit.client.0.smithi062.stdout:7/236: stat d7/l11 0 2024-04-20T21:31:54.099 INFO:tasks.workunit.client.0.smithi062.stdout:8/304: link d7/d25/d3b/d58/f63 d7/d9/d1b/d24/f67 0 2024-04-20T21:31:54.100 INFO:tasks.workunit.client.0.smithi062.stdout:7/237: fsync d7/d26/f34 0 2024-04-20T21:31:54.100 INFO:tasks.workunit.client.0.smithi062.stdout:4/241: dread da/dd/df/f17 [0,4194304] 0 2024-04-20T21:31:54.101 INFO:tasks.workunit.client.0.smithi062.stdout:0/266: creat d7/d20/f5c x:0 0 0 2024-04-20T21:31:54.105 INFO:tasks.workunit.client.0.smithi062.stdout:3/305: dwrite d0/f53 [4194304,4194304] 0 2024-04-20T21:31:54.105 INFO:tasks.workunit.client.0.smithi062.stdout:5/254: rename d8/d10/d2a to d8/d10/d32/d56 0 2024-04-20T21:31:54.108 INFO:tasks.workunit.client.0.smithi062.stdout:3/306: readlink d0/d1/d10/d31/l3d 0 2024-04-20T21:31:54.108 INFO:tasks.workunit.client.0.smithi062.stdout:8/305: fsync d7/d9/ff 0 2024-04-20T21:31:54.109 INFO:tasks.workunit.client.0.smithi062.stdout:4/242: mkdir da/d4f 0 2024-04-20T21:31:54.113 INFO:tasks.workunit.client.0.smithi062.stdout:4/243: fdatasync da/dd/df/f17 0 2024-04-20T21:31:54.115 INFO:tasks.workunit.client.0.smithi062.stdout:7/238: rmdir d7/d3e/d1c/d4f 39 2024-04-20T21:31:54.117 INFO:tasks.workunit.client.0.smithi062.stdout:8/306: mknod d7/d25/d3b/d58/c68 0 2024-04-20T21:31:54.119 INFO:tasks.workunit.client.0.smithi062.stdout:7/239: chown d7/d3e/d33/d3d/d4d 15605784 1 2024-04-20T21:31:54.121 INFO:tasks.workunit.client.0.smithi062.stdout:0/267: dread d7/d1c/d15/d17/d37/f5b [0,4194304] 0 2024-04-20T21:31:54.122 INFO:tasks.workunit.client.0.smithi062.stdout:4/244: mkdir da/dd/d15/d50 0 2024-04-20T21:31:54.125 INFO:tasks.workunit.client.0.smithi062.stdout:7/240: chown d7/d26/l4b 561 1 2024-04-20T21:31:54.126 INFO:tasks.workunit.client.0.smithi062.stdout:8/307: creat d7/d25/d5c/f69 x:0 0 0 2024-04-20T21:31:54.126 INFO:tasks.workunit.client.0.smithi062.stdout:9/270: dread d16/f1d [4194304,4194304] 0 2024-04-20T21:31:54.128 INFO:tasks.workunit.client.0.smithi062.stdout:3/307: link d0/d1c/d63/d42/l69 d0/d1c/l6d 0 2024-04-20T21:31:54.128 INFO:tasks.workunit.client.0.smithi062.stdout:4/245: stat da/dd/df/c25 0 2024-04-20T21:31:54.129 INFO:tasks.workunit.client.0.smithi062.stdout:0/268: chown d7/d1e/c2b 688 1 2024-04-20T21:31:54.130 INFO:tasks.workunit.client.0.smithi062.stdout:7/241: creat d7/d3e/d33/f52 x:0 0 0 2024-04-20T21:31:54.133 INFO:tasks.workunit.client.0.smithi062.stdout:9/271: rmdir d16/d2f 39 2024-04-20T21:31:54.134 INFO:tasks.workunit.client.0.smithi062.stdout:1/308: dread dc/d15/d17/f34 [0,4194304] 0 2024-04-20T21:31:54.135 INFO:tasks.workunit.client.0.smithi062.stdout:7/242: unlink d7/d3e/d33/d3d/l43 0 2024-04-20T21:31:54.135 INFO:tasks.workunit.client.0.smithi062.stdout:1/309: readlink dc/d15/l3d 0 2024-04-20T21:31:54.137 INFO:tasks.workunit.client.0.smithi062.stdout:4/246: symlink da/dd/d15/d4b/l51 0 2024-04-20T21:31:54.138 INFO:tasks.workunit.client.0.smithi062.stdout:3/308: mknod d0/d1/c6e 0 2024-04-20T21:31:54.140 INFO:tasks.workunit.client.0.smithi062.stdout:0/269: rename d7/d1c/f14 to d7/d1c/d15/d17/d37/f5d 0 2024-04-20T21:31:54.144 INFO:tasks.workunit.client.0.smithi062.stdout:1/310: mknod dc/d15/c68 0 2024-04-20T21:31:54.146 INFO:tasks.workunit.client.0.smithi062.stdout:7/243: dread d7/d26/f31 [0,4194304] 0 2024-04-20T21:31:54.147 INFO:tasks.workunit.client.0.smithi062.stdout:9/272: rename d16/l26 to d16/d3e/l5c 0 2024-04-20T21:31:54.149 INFO:tasks.workunit.client.0.smithi062.stdout:4/247: link da/dd/d3c/l34 da/dd/df/d27/d32/l52 0 2024-04-20T21:31:54.150 INFO:tasks.workunit.client.0.smithi062.stdout:7/244: write d7/d26/f29 [2205329,116617] 0 2024-04-20T21:31:54.152 INFO:tasks.workunit.client.0.smithi062.stdout:0/270: rename d7/d1c/l35 to d7/d1c/l5e 0 2024-04-20T21:31:54.152 INFO:tasks.workunit.client.0.smithi062.stdout:5/255: dread d8/d10/f28 [0,4194304] 0 2024-04-20T21:31:54.155 INFO:tasks.workunit.client.0.smithi062.stdout:0/271: write d7/d1c/f32 [3095404,89255] 0 2024-04-20T21:31:54.159 INFO:tasks.workunit.client.0.smithi062.stdout:3/309: dwrite d0/d1/fb [0,4194304] 0 2024-04-20T21:31:54.212 INFO:tasks.workunit.client.0.smithi062.stdout:1/311: write f9 [636790,35895] 0 2024-04-20T21:31:54.215 INFO:tasks.workunit.client.0.smithi062.stdout:5/256: symlink d8/d10/d32/d56/d2b/d4c/l57 0 2024-04-20T21:31:54.215 INFO:tasks.workunit.client.0.smithi062.stdout:9/273: link d16/d2c/c3c d16/c5d 0 2024-04-20T21:31:54.216 INFO:tasks.workunit.client.0.smithi062.stdout:8/308: dread d7/d9/f51 [0,4194304] 0 2024-04-20T21:31:54.216 INFO:tasks.workunit.client.0.smithi062.stdout:0/272: write d7/fb [854379,42008] 0 2024-04-20T21:31:54.218 INFO:tasks.workunit.client.0.smithi062.stdout:8/309: fdatasync d7/d30/f5d 0 2024-04-20T21:31:54.220 INFO:tasks.workunit.client.0.smithi062.stdout:3/310: rename d0/d1/d10/d31/l37 to d0/d1/d10/l6f 0 2024-04-20T21:31:54.220 INFO:tasks.workunit.client.0.smithi062.stdout:0/273: write d7/d1c/d15/d17/d37/f59 [998246,75960] 0 2024-04-20T21:31:54.221 INFO:tasks.workunit.client.0.smithi062.stdout:9/274: fsync d16/f36 0 2024-04-20T21:31:54.224 INFO:tasks.workunit.client.0.smithi062.stdout:3/311: chown d0/d1/d10/d3a/d5e/d65 5235 1 2024-04-20T21:31:54.225 INFO:tasks.workunit.client.0.smithi062.stdout:9/275: mkdir d16/d2c/d5e 0 2024-04-20T21:31:54.226 INFO:tasks.workunit.client.0.smithi062.stdout:1/312: creat dc/d15/d17/f69 x:0 0 0 2024-04-20T21:31:54.227 INFO:tasks.workunit.client.0.smithi062.stdout:8/310: mknod d7/d9/d1b/c6a 0 2024-04-20T21:31:54.227 INFO:tasks.workunit.client.0.smithi062.stdout:3/312: fsync d0/d1b/f2b 0 2024-04-20T21:31:54.228 INFO:tasks.workunit.client.0.smithi062.stdout:3/313: chown d0 673 1 2024-04-20T21:31:54.231 INFO:tasks.workunit.client.0.smithi062.stdout:0/274: mknod d7/d1c/d15/d17/d26/c5f 0 2024-04-20T21:31:54.232 INFO:tasks.workunit.client.0.smithi062.stdout:7/245: dwrite d7/d3e/d1c/f28 [4194304,4194304] 0 2024-04-20T21:31:54.235 INFO:tasks.workunit.client.0.smithi062.stdout:3/314: write d0/d1/d10/d3a/d5e/f62 [122643,38877] 0 2024-04-20T21:31:54.238 INFO:tasks.workunit.client.0.smithi062.stdout:7/246: fsync d7/f2a 0 2024-04-20T21:31:54.240 INFO:tasks.workunit.client.0.smithi062.stdout:8/311: dread d7/d9/f15 [0,4194304] 0 2024-04-20T21:31:54.241 INFO:tasks.workunit.client.0.smithi062.stdout:1/313: link dc/d15/d17/f19 dc/d15/d17/d36/d3a/f6a 0 2024-04-20T21:31:54.243 INFO:tasks.workunit.client.0.smithi062.stdout:3/315: rmdir d0/d1/d10/d3a 39 2024-04-20T21:31:54.245 INFO:tasks.workunit.client.0.smithi062.stdout:7/247: creat d7/d3e/d1c/d4f/f53 x:0 0 0 2024-04-20T21:31:54.246 INFO:tasks.workunit.client.0.smithi062.stdout:5/257: dwrite d8/d10/d32/f3c [0,4194304] 0 2024-04-20T21:31:54.247 INFO:tasks.workunit.client.0.smithi062.stdout:1/314: chown dc/d15/d17/d36 21118 1 2024-04-20T21:31:54.247 INFO:tasks.workunit.client.0.smithi062.stdout:8/312: creat d7/d25/d3b/d58/f6b x:0 0 0 2024-04-20T21:31:54.251 INFO:tasks.workunit.client.0.smithi062.stdout:7/248: dread - d7/d26/f34 zero size 2024-04-20T21:31:54.256 INFO:tasks.workunit.client.0.smithi062.stdout:8/313: symlink d7/l6c 0 2024-04-20T21:31:54.260 INFO:tasks.workunit.client.0.smithi062.stdout:9/276: dwrite d16/f1d [0,4194304] 0 2024-04-20T21:31:54.261 INFO:tasks.workunit.client.0.smithi062.stdout:8/314: readlink d7/d25/l37 0 2024-04-20T21:31:54.264 INFO:tasks.workunit.client.0.smithi062.stdout:3/316: link d0/d1c/d63/f30 d0/d1/d10/d31/f70 0 2024-04-20T21:31:54.267 INFO:tasks.workunit.client.0.smithi062.stdout:0/275: dwrite d7/d1c/d15/d17/d37/f59 [0,4194304] 0 2024-04-20T21:31:54.276 INFO:tasks.workunit.client.0.smithi062.stdout:8/315: symlink d7/d9/l6d 0 2024-04-20T21:31:54.276 INFO:tasks.workunit.client.0.smithi062.stdout:5/258: dwrite d8/d10/d3e/f52 [0,4194304] 0 2024-04-20T21:31:54.276 INFO:tasks.workunit.client.0.smithi062.stdout:7/249: rename d7/d3e/d1c to d7/d3e/d33/d3d/d4d/d54 0 2024-04-20T21:31:54.278 INFO:tasks.workunit.client.0.smithi062.stdout:1/315: truncate dc/ff 2973245 0 2024-04-20T21:31:54.281 INFO:tasks.workunit.client.0.smithi062.stdout:3/317: symlink d0/d1/l71 0 2024-04-20T21:31:54.282 INFO:tasks.workunit.client.0.smithi062.stdout:8/316: fsync d7/d9/f51 0 2024-04-20T21:31:54.284 INFO:tasks.workunit.client.0.smithi062.stdout:0/276: rename l3 to d7/d1e/l60 0 2024-04-20T21:31:54.285 INFO:tasks.workunit.client.0.smithi062.stdout:9/277: truncate d16/f19 2816232 0 2024-04-20T21:31:54.287 INFO:tasks.workunit.client.0.smithi062.stdout:5/259: symlink d8/l58 0 2024-04-20T21:31:54.289 INFO:tasks.workunit.client.0.smithi062.stdout:2/253: sync 2024-04-20T21:31:54.289 INFO:tasks.workunit.client.0.smithi062.stdout:6/233: sync 2024-04-20T21:31:54.290 INFO:tasks.workunit.client.0.smithi062.stdout:3/318: chown d0/d1/d10/d31/c34 7736 1 2024-04-20T21:31:54.291 INFO:tasks.workunit.client.0.smithi062.stdout:8/317: creat d7/d9/d29/f6e x:0 0 0 2024-04-20T21:31:54.291 INFO:tasks.workunit.client.0.smithi062.stdout:7/250: dread d7/f18 [4194304,4194304] 0 2024-04-20T21:31:54.293 INFO:tasks.workunit.client.0.smithi062.stdout:3/319: chown d0/d1c/f45 0 1 2024-04-20T21:31:54.296 INFO:tasks.workunit.client.0.smithi062.stdout:2/254: fsync d7/da/d15/d2f/f42 0 2024-04-20T21:31:54.300 INFO:tasks.workunit.client.0.smithi062.stdout:2/255: write d7/f1a [3310215,7518] 0 2024-04-20T21:31:54.307 INFO:tasks.workunit.client.0.smithi062.stdout:8/318: rename d7/d30/c56 to d7/d9/d1b/d24/d46/c6f 0 2024-04-20T21:31:54.308 INFO:tasks.workunit.client.0.smithi062.stdout:1/316: dread f1 [0,4194304] 0 2024-04-20T21:31:54.313 INFO:tasks.workunit.client.0.smithi062.stdout:3/320: mkdir d0/d1/d3c/d43/d72 0 2024-04-20T21:31:54.313 INFO:tasks.workunit.client.0.smithi062.stdout:2/256: readlink d7/da/d15/d1b/l3a 0 2024-04-20T21:31:54.316 INFO:tasks.workunit.client.0.smithi062.stdout:8/319: read - d7/d9/d1b/d24/f48 zero size 2024-04-20T21:31:54.318 INFO:tasks.workunit.client.0.smithi062.stdout:7/251: creat d7/f55 x:0 0 0 2024-04-20T21:31:54.329 INFO:tasks.workunit.client.0.smithi062.stdout:0/277: dwrite d7/d1c/d15/d17/f1d [0,4194304] 0 2024-04-20T21:31:54.329 INFO:tasks.workunit.client.0.smithi062.stdout:6/234: dwrite d0/d2c/f3b [0,4194304] 0 2024-04-20T21:31:54.331 INFO:tasks.workunit.client.0.smithi062.stdout:7/252: chown d7/c13 15410000 1 2024-04-20T21:31:54.334 INFO:tasks.workunit.client.0.smithi062.stdout:8/320: mkdir d7/d25/d70 0 2024-04-20T21:31:54.335 INFO:tasks.workunit.client.0.smithi062.stdout:9/278: dwrite d16/d2f/f31 [0,4194304] 0 2024-04-20T21:31:54.337 INFO:tasks.workunit.client.0.smithi062.stdout:0/278: creat d7/d1c/d15/d17/d37/f61 x:0 0 0 2024-04-20T21:31:54.339 INFO:tasks.workunit.client.0.smithi062.stdout:1/317: link dc/c11 dc/d27/d32/c6b 0 2024-04-20T21:31:54.343 INFO:tasks.workunit.client.0.smithi062.stdout:7/253: symlink d7/d3e/d33/l56 0 2024-04-20T21:31:54.345 INFO:tasks.workunit.client.0.smithi062.stdout:6/235: unlink d0/d9/cf 0 2024-04-20T21:31:54.347 INFO:tasks.workunit.client.0.smithi062.stdout:0/279: mkdir d7/d20/d62 0 2024-04-20T21:31:54.348 INFO:tasks.workunit.client.0.smithi062.stdout:9/279: creat d16/d3e/f5f x:0 0 0 2024-04-20T21:31:54.348 INFO:tasks.workunit.client.0.smithi062.stdout:8/321: dread f1 [4194304,4194304] 0 2024-04-20T21:31:54.349 INFO:tasks.workunit.client.0.smithi062.stdout:3/321: dwrite d0/d1/d10/f4e [0,4194304] 0 2024-04-20T21:31:54.349 INFO:tasks.workunit.client.0.smithi062.stdout:9/280: read fd [5882588,66044] 0 2024-04-20T21:31:54.351 INFO:tasks.workunit.client.0.smithi062.stdout:8/322: write d7/d9/d1b/d24/f48 [960046,122261] 0 2024-04-20T21:31:54.351 INFO:tasks.workunit.client.0.smithi062.stdout:9/281: read d16/d2c/d42/f54 [224160,29809] 0 2024-04-20T21:31:54.354 INFO:tasks.workunit.client.0.smithi062.stdout:2/257: truncate d7/ff 1455837 0 2024-04-20T21:31:54.354 INFO:tasks.workunit.client.0.smithi062.stdout:1/318: creat dc/d2d/f6c x:0 0 0 2024-04-20T21:31:54.360 INFO:tasks.workunit.client.0.smithi062.stdout:9/282: mknod d16/c60 0 2024-04-20T21:31:54.364 INFO:tasks.workunit.client.0.smithi062.stdout:1/319: creat dc/f6d x:0 0 0 2024-04-20T21:31:54.367 INFO:tasks.workunit.client.0.smithi062.stdout:9/283: chown d16/d2c/l34 23 1 2024-04-20T21:31:54.372 INFO:tasks.workunit.client.0.smithi062.stdout:1/320: chown dc/d15/d17/f67 1327 1 2024-04-20T21:31:54.374 INFO:tasks.workunit.client.0.smithi062.stdout:0/280: dwrite d7/d1c/f32 [0,4194304] 0 2024-04-20T21:31:54.377 INFO:tasks.workunit.client.0.smithi062.stdout:1/321: chown dc/c10 96118 1 2024-04-20T21:31:54.377 INFO:tasks.workunit.client.0.smithi062.stdout:9/284: rename d16/d3d/d56 to d16/d2f/d5b/d61 0 2024-04-20T21:31:54.381 INFO:tasks.workunit.client.0.smithi062.stdout:2/258: dwrite d7/f1a [0,4194304] 0 2024-04-20T21:31:54.382 INFO:tasks.workunit.client.1.smithi189.stderr:+ pushd ltp-full-20091231/testcases/kernel/fs/fsstress 2024-04-20T21:31:54.382 INFO:tasks.workunit.client.0.smithi062.stdout:9/285: chown d16/d2f/l41 11 1 2024-04-20T21:31:54.383 INFO:tasks.workunit.client.0.smithi062.stdout:8/323: dwrite d7/d9/d1b/f1f [0,4194304] 0 2024-04-20T21:31:54.385 INFO:tasks.workunit.client.0.smithi062.stdout:2/259: read d7/f12 [1017891,11140] 0 2024-04-20T21:31:54.385 INFO:tasks.workunit.client.0.smithi062.stdout:1/322: truncate dc/d15/f48 1104294 0 2024-04-20T21:31:54.387 INFO:tasks.workunit.client.0.smithi062.stdout:0/281: dread d7/d1c/d15/d17/f38 [0,4194304] 0 2024-04-20T21:31:54.389 INFO:tasks.workunit.client.0.smithi062.stdout:9/286: mkdir d16/d2c/d62 0 2024-04-20T21:31:54.393 INFO:tasks.workunit.client.0.smithi062.stdout:2/260: creat d7/da/d15/d1b/d3d/f49 x:0 0 0 2024-04-20T21:31:54.394 INFO:tasks.workunit.client.0.smithi062.stdout:0/282: dread d7/d1c/f3e [0,4194304] 0 2024-04-20T21:31:54.395 INFO:tasks.workunit.client.1.smithi189.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress ~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-04-20T21:31:54.395 INFO:tasks.workunit.client.0.smithi062.stdout:1/323: creat dc/d15/f6e x:0 0 0 2024-04-20T21:31:54.395 INFO:tasks.workunit.client.1.smithi189.stderr:+ make 2024-04-20T21:31:54.449 INFO:tasks.workunit.client.0.smithi062.stdout:4/248: sync 2024-04-20T21:31:54.454 INFO:tasks.workunit.client.0.smithi062.stdout:9/287: rename d16/l21 to d16/d3d/l63 0 2024-04-20T21:31:54.455 INFO:tasks.workunit.client.0.smithi062.stdout:2/261: mknod d7/da/d15/d1b/c4a 0 2024-04-20T21:31:54.457 INFO:tasks.workunit.client.0.smithi062.stdout:9/288: write d16/d3d/f43 [1834903,62727] 0 2024-04-20T21:31:54.458 INFO:tasks.workunit.client.0.smithi062.stdout:4/249: creat da/dd/d3c/f53 x:0 0 0 2024-04-20T21:31:54.458 INFO:tasks.workunit.client.0.smithi062.stdout:9/289: chown d16/d3d 2 1 2024-04-20T21:31:54.459 INFO:tasks.workunit.client.0.smithi062.stdout:1/324: mkdir dc/d15/d17/d6f 0 2024-04-20T21:31:54.464 INFO:tasks.workunit.client.0.smithi062.stdout:4/250: truncate da/dd/df/d27/f2c 1673205 0 2024-04-20T21:31:54.465 INFO:tasks.workunit.client.0.smithi062.stdout:4/251: chown da/d4f 1452312 1 2024-04-20T21:31:54.466 INFO:tasks.workunit.client.0.smithi062.stdout:4/252: read da/f16 [1407392,82432] 0 2024-04-20T21:31:54.470 INFO:tasks.workunit.client.0.smithi062.stdout:4/253: symlink da/dd/d3c/l54 0 2024-04-20T21:31:54.475 INFO:tasks.workunit.client.0.smithi062.stdout:4/254: symlink da/dd/d15/l55 0 2024-04-20T21:31:54.477 INFO:tasks.workunit.client.0.smithi062.stdout:2/262: truncate d7/da/d15/d1b/f2c 33093 0 2024-04-20T21:31:54.478 INFO:tasks.workunit.client.0.smithi062.stdout:4/255: mkdir da/d56 0 2024-04-20T21:31:54.479 INFO:tasks.workunit.client.0.smithi062.stdout:1/325: dwrite dc/d2d/f6c [0,4194304] 0 2024-04-20T21:31:54.480 INFO:tasks.workunit.client.0.smithi062.stdout:4/256: chown da/dd/c23 4085746 1 2024-04-20T21:31:54.481 INFO:tasks.workunit.client.0.smithi062.stdout:2/263: chown d7/da/l27 31872 1 2024-04-20T21:31:54.486 INFO:tasks.workunit.client.0.smithi062.stdout:2/264: mkdir d7/da/d28/d4b 0 2024-04-20T21:31:54.491 INFO:tasks.workunit.client.0.smithi062.stdout:4/257: mkdir da/dd/df/d27/d30/d57 0 2024-04-20T21:31:54.492 INFO:tasks.workunit.client.0.smithi062.stdout:2/265: mknod d7/da/d15/d1b/c4c 0 2024-04-20T21:31:54.498 INFO:tasks.workunit.client.0.smithi062.stdout:2/266: chown d7/da/d15/f26 0 1 2024-04-20T21:31:54.498 INFO:tasks.workunit.client.0.smithi062.stdout:2/267: read d7/f1a [1434377,69812] 0 2024-04-20T21:31:54.500 INFO:tasks.workunit.client.0.smithi062.stdout:2/268: readlink d7/l8 0 2024-04-20T21:31:54.501 INFO:tasks.workunit.client.0.smithi062.stdout:1/326: dwrite dc/f14 [4194304,4194304] 0 2024-04-20T21:31:54.502 INFO:tasks.workunit.client.0.smithi062.stdout:2/269: dread - d7/da/d15/f26 zero size 2024-04-20T21:31:54.504 INFO:tasks.workunit.client.0.smithi062.stdout:1/327: mkdir dc/d70 0 2024-04-20T21:31:54.506 INFO:tasks.workunit.client.0.smithi062.stdout:1/328: read - dc/d2d/f65 zero size 2024-04-20T21:31:54.509 INFO:tasks.workunit.client.0.smithi062.stdout:1/329: symlink dc/d27/l71 0 2024-04-20T21:31:54.518 INFO:tasks.workunit.client.0.smithi062.stdout:2/270: dwrite d7/da/f22 [0,4194304] 0 2024-04-20T21:31:54.522 INFO:tasks.workunit.client.0.smithi062.stdout:2/271: fsync d7/da/d15/f23 0 2024-04-20T21:31:54.529 INFO:tasks.workunit.client.0.smithi062.stdout:2/272: rename d7/f1e to d7/da/f4d 0 2024-04-20T21:31:54.533 INFO:tasks.workunit.client.0.smithi062.stdout:2/273: rmdir d7/da/d15/d1b 39 2024-04-20T21:31:54.533 INFO:tasks.workunit.client.0.smithi062.stdout:4/258: dread da/dd/df/d27/f36 [0,4194304] 0 2024-04-20T21:31:54.567 INFO:tasks.workunit.client.0.smithi062.stdout:2/274: write d7/ff [762411,2496] 0 2024-04-20T21:31:54.572 INFO:tasks.workunit.client.0.smithi062.stdout:2/275: creat d7/da/d15/d2f/d38/f4e x:0 0 0 2024-04-20T21:31:54.580 INFO:tasks.workunit.client.0.smithi062.stdout:2/276: rename d7/da/d15/d1b/l41 to d7/da/d15/d1b/d24/l4f 0 2024-04-20T21:31:54.591 INFO:tasks.workunit.client.0.smithi062.stdout:2/277: dread d7/da/f22 [0,4194304] 0 2024-04-20T21:31:54.597 INFO:tasks.workunit.client.0.smithi062.stdout:2/278: mknod d7/da/d15/c50 0 2024-04-20T21:31:54.616 INFO:tasks.workunit.client.0.smithi062.stdout:2/279: dwrite d7/da/f13 [0,4194304] 0 2024-04-20T21:31:54.624 INFO:tasks.workunit.client.0.smithi062.stdout:2/280: mkdir d7/da/d15/d2f/d38/d51 0 2024-04-20T21:31:54.631 INFO:tasks.workunit.client.0.smithi062.stdout:2/281: creat d7/da/d15/d1b/d3d/f52 x:0 0 0 2024-04-20T21:31:54.635 INFO:tasks.workunit.client.0.smithi062.stdout:2/282: write d7/f17 [3879159,51956] 0 2024-04-20T21:31:54.642 INFO:tasks.workunit.client.0.smithi062.stdout:2/283: truncate d7/f12 2927909 0 2024-04-20T21:31:54.678 INFO:tasks.workunit.client.1.smithi189.stdout:cc -DNO_XFS -I/home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress -D_LARGEFILE64_SOURCE -D_GNU_SOURCE -I../../../../include -I../../../../include -L../../../../lib fsstress.c -o fsstress 2024-04-20T21:31:54.678 INFO:tasks.workunit.client.0.smithi062.stdout:2/284: creat d7/da/d28/f53 x:0 0 0 2024-04-20T21:31:54.687 INFO:tasks.workunit.client.0.smithi062.stdout:2/285: write d7/da/f22 [4328302,122301] 0 2024-04-20T21:31:54.691 INFO:tasks.workunit.client.0.smithi062.stdout:2/286: write d7/d39/f45 [402925,31339] 0 2024-04-20T21:31:54.697 INFO:tasks.workunit.client.0.smithi062.stdout:2/287: creat d7/da/d15/d2f/d38/f54 x:0 0 0 2024-04-20T21:31:54.702 INFO:tasks.workunit.client.0.smithi062.stdout:2/288: read - d7/da/d15/d2f/d38/f4e zero size 2024-04-20T21:31:54.704 INFO:tasks.workunit.client.0.smithi062.stdout:5/260: sync 2024-04-20T21:31:54.713 INFO:tasks.workunit.client.0.smithi062.stdout:2/289: rmdir d7/d39 39 2024-04-20T21:31:54.714 INFO:tasks.workunit.client.0.smithi062.stdout:5/261: write d8/d1e/f24 [2687565,63767] 0 2024-04-20T21:31:54.715 INFO:tasks.workunit.client.0.smithi062.stdout:2/290: fdatasync d7/da/f4d 0 2024-04-20T21:31:54.719 INFO:tasks.workunit.client.0.smithi062.stdout:2/291: write d7/da/d15/d2f/f30 [500701,56369] 0 2024-04-20T21:31:54.720 INFO:tasks.workunit.client.0.smithi062.stdout:5/262: creat d8/d10/d32/d56/f59 x:0 0 0 2024-04-20T21:31:54.725 INFO:tasks.workunit.client.0.smithi062.stdout:2/292: write d7/da/d15/d2f/d38/f4e [589091,6026] 0 2024-04-20T21:31:54.727 INFO:tasks.workunit.client.0.smithi062.stdout:2/293: chown d7/da/c19 8650 1 2024-04-20T21:31:54.730 INFO:tasks.workunit.client.0.smithi062.stdout:5/263: link l6 d8/d10/d32/d56/l5a 0 2024-04-20T21:31:54.734 INFO:tasks.workunit.client.0.smithi062.stdout:5/264: write d8/d10/d32/d56/d2f/f4b [196610,48977] 0 2024-04-20T21:31:54.738 INFO:tasks.workunit.client.0.smithi062.stdout:5/265: fsync d8/d10/f1b 0 2024-04-20T21:31:54.743 INFO:tasks.workunit.client.0.smithi062.stdout:5/266: creat d8/d10/d3e/f5b x:0 0 0 2024-04-20T21:31:54.746 INFO:tasks.workunit.client.0.smithi062.stdout:2/294: dread d7/ff [0,4194304] 0 2024-04-20T21:31:54.749 INFO:tasks.workunit.client.0.smithi062.stdout:2/295: dread - d7/da/d15/d2f/d38/f54 zero size 2024-04-20T21:31:54.752 INFO:tasks.workunit.client.0.smithi062.stdout:2/296: chown d7/d39/f45 31506 1 2024-04-20T21:31:54.753 INFO:tasks.workunit.client.0.smithi062.stdout:5/267: truncate d8/f44 1882344 0 2024-04-20T21:31:54.754 INFO:tasks.workunit.client.0.smithi062.stdout:2/297: creat d7/d39/f55 x:0 0 0 2024-04-20T21:31:54.760 INFO:tasks.workunit.client.0.smithi062.stdout:2/298: readlink d7/da/d15/l40 0 2024-04-20T21:31:54.776 INFO:tasks.workunit.client.0.smithi062.stdout:2/299: dwrite d7/d39/f55 [0,4194304] 0 2024-04-20T21:31:54.779 INFO:tasks.workunit.client.0.smithi062.stdout:2/300: dread - d7/da/d28/f53 zero size 2024-04-20T21:31:54.786 INFO:tasks.workunit.client.0.smithi062.stdout:5/268: dwrite d8/d1e/f53 [4194304,4194304] 0 2024-04-20T21:31:54.786 INFO:tasks.workunit.client.0.smithi062.stdout:2/301: unlink d7/da/l2b 0 2024-04-20T21:31:54.789 INFO:tasks.workunit.client.0.smithi062.stdout:5/269: write d8/d10/d3e/f52 [1042923,111763] 0 2024-04-20T21:31:54.791 INFO:tasks.workunit.client.0.smithi062.stdout:5/270: chown d8/d10/d32/d56/d2b/d4c 8 1 2024-04-20T21:31:54.792 INFO:tasks.workunit.client.0.smithi062.stdout:2/302: unlink d7/da/c11 0 2024-04-20T21:31:54.820 INFO:tasks.workunit.client.0.smithi062.stdout:5/271: dwrite d8/d1e/f24 [0,4194304] 0 2024-04-20T21:31:54.824 INFO:tasks.workunit.client.0.smithi062.stdout:2/303: dwrite d7/da/d15/d1b/d3d/f52 [0,4194304] 0 2024-04-20T21:31:54.831 INFO:tasks.workunit.client.0.smithi062.stdout:5/272: write d8/d10/d32/d56/d2b/f40 [3295475,111702] 0 2024-04-20T21:31:54.833 INFO:tasks.workunit.client.0.smithi062.stdout:2/304: creat d7/f56 x:0 0 0 2024-04-20T21:31:54.838 INFO:tasks.workunit.client.0.smithi062.stdout:5/273: rmdir d8/d10/d32 39 2024-04-20T21:31:54.845 INFO:tasks.workunit.client.0.smithi062.stdout:2/305: link d7/da/d15/d1b/d24/l4f d7/da/d28/l57 0 2024-04-20T21:31:54.847 INFO:tasks.workunit.client.0.smithi062.stdout:2/306: fdatasync d7/da/d15/d2f/f33 0 2024-04-20T21:31:54.851 INFO:tasks.workunit.client.0.smithi062.stdout:2/307: symlink d7/da/d15/l58 0 2024-04-20T21:31:54.854 INFO:tasks.workunit.client.0.smithi062.stdout:7/254: sync 2024-04-20T21:31:54.855 INFO:tasks.workunit.client.0.smithi062.stdout:2/308: rename c2 to d7/da/d28/c59 0 2024-04-20T21:31:54.856 INFO:tasks.workunit.client.0.smithi062.stdout:2/309: chown d7/da/d28/d4b 14890 1 2024-04-20T21:31:54.858 INFO:tasks.workunit.client.0.smithi062.stdout:7/255: chown d7/f2d 581203 1 2024-04-20T21:31:54.858 INFO:tasks.workunit.client.0.smithi062.stdout:5/274: dread d8/d10/d3e/f52 [0,4194304] 0 2024-04-20T21:31:54.858 INFO:tasks.workunit.client.0.smithi062.stdout:2/310: chown d7/f17 183451261 1 2024-04-20T21:31:54.860 INFO:tasks.workunit.client.0.smithi062.stdout:7/256: read d7/d3e/f44 [5836656,35345] 0 2024-04-20T21:31:54.861 INFO:tasks.workunit.client.0.smithi062.stdout:2/311: symlink d7/da/d15/d1b/l5a 0 2024-04-20T21:31:54.863 INFO:tasks.workunit.client.0.smithi062.stdout:5/275: truncate d8/d10/d32/d56/d2f/f4b 460856 0 2024-04-20T21:31:54.863 INFO:tasks.workunit.client.0.smithi062.stdout:7/257: write d7/d3e/d33/f35 [600550,111177] 0 2024-04-20T21:31:54.865 INFO:tasks.workunit.client.0.smithi062.stdout:7/258: write d7/f2d [693417,78228] 0 2024-04-20T21:31:54.867 INFO:tasks.workunit.client.0.smithi062.stdout:2/312: rename d7/da/d15/d1b/l5a to d7/da/l5b 0 2024-04-20T21:31:54.869 INFO:tasks.workunit.client.0.smithi062.stdout:5/276: write d8/d10/d3e/f52 [926735,130882] 0 2024-04-20T21:31:54.870 INFO:tasks.workunit.client.0.smithi062.stdout:7/259: rename d7/f12 to d7/d3e/d33/f57 0 2024-04-20T21:31:54.872 INFO:tasks.workunit.client.0.smithi062.stdout:7/260: fsync d7/d3e/d33/f35 0 2024-04-20T21:31:54.873 INFO:tasks.workunit.client.0.smithi062.stdout:5/277: dread d8/f4f [0,4194304] 0 2024-04-20T21:31:54.875 INFO:tasks.workunit.client.0.smithi062.stdout:5/278: write d8/fb [1314113,26014] 0 2024-04-20T21:31:54.888 INFO:tasks.workunit.client.0.smithi062.stdout:2/313: dwrite d7/da/d28/f2a [0,4194304] 0 2024-04-20T21:31:54.895 INFO:tasks.workunit.client.0.smithi062.stdout:7/261: dwrite d7/d3e/d33/d3d/d4d/d54/f1f [0,4194304] 0 2024-04-20T21:31:54.897 INFO:tasks.workunit.client.0.smithi062.stdout:8/324: sync 2024-04-20T21:31:54.897 INFO:tasks.workunit.client.0.smithi062.stdout:6/236: sync 2024-04-20T21:31:54.898 INFO:tasks.workunit.client.0.smithi062.stdout:2/314: write d7/f1a [3925991,19370] 0 2024-04-20T21:31:54.899 INFO:tasks.workunit.client.0.smithi062.stdout:7/262: chown d7/l16 45109 1 2024-04-20T21:31:54.900 INFO:tasks.workunit.client.0.smithi062.stdout:3/322: sync 2024-04-20T21:31:54.904 INFO:tasks.workunit.client.0.smithi062.stdout:2/315: symlink d7/da/d15/d1b/l5c 0 2024-04-20T21:31:54.912 INFO:tasks.workunit.client.0.smithi062.stdout:3/323: creat d0/d1c/d63/d42/f73 x:0 0 0 2024-04-20T21:31:54.916 INFO:tasks.workunit.client.0.smithi062.stdout:6/237: rename d0/cb to d0/d9/d11/d22/c4d 0 2024-04-20T21:31:54.917 INFO:tasks.workunit.client.0.smithi062.stdout:3/324: mkdir d0/d1b/d74 0 2024-04-20T21:31:54.917 INFO:tasks.workunit.client.0.smithi062.stdout:8/325: dread d7/d9/d1b/d24/f48 [0,4194304] 0 2024-04-20T21:31:54.919 INFO:tasks.workunit.client.0.smithi062.stdout:2/316: creat d7/f5d x:0 0 0 2024-04-20T21:31:54.919 INFO:tasks.workunit.client.0.smithi062.stdout:8/326: stat d7/d9/ce 0 2024-04-20T21:31:54.924 INFO:tasks.workunit.client.0.smithi062.stdout:5/279: dread d8/fb [0,4194304] 0 2024-04-20T21:31:54.925 INFO:tasks.workunit.client.0.smithi062.stdout:8/327: dread - d7/d9/d29/f4b zero size 2024-04-20T21:31:54.925 INFO:tasks.workunit.client.0.smithi062.stdout:5/280: rmdir d8/d10 39 2024-04-20T21:31:54.926 INFO:tasks.workunit.client.0.smithi062.stdout:7/263: dwrite d7/f48 [0,4194304] 0 2024-04-20T21:31:54.927 INFO:tasks.workunit.client.0.smithi062.stdout:8/328: chown d7/d9/d1b/d24/d46/f49 3311747 1 2024-04-20T21:31:54.932 INFO:tasks.workunit.client.0.smithi062.stdout:5/281: chown d8/d10/d32/d56/c35 18603 1 2024-04-20T21:31:54.934 INFO:tasks.workunit.client.0.smithi062.stdout:6/238: fdatasync d0/d9/d11/d22/f27 0 2024-04-20T21:31:54.938 INFO:tasks.workunit.client.0.smithi062.stdout:2/317: dread d7/da/f22 [0,4194304] 0 2024-04-20T21:31:54.939 INFO:tasks.workunit.client.0.smithi062.stdout:6/239: symlink d0/d1a/d44/l4e 0 2024-04-20T21:31:54.940 INFO:tasks.workunit.client.0.smithi062.stdout:3/325: dwrite d0/d1b/f35 [0,4194304] 0 2024-04-20T21:31:54.941 INFO:tasks.workunit.client.0.smithi062.stdout:7/264: rename d7/d3e/d33/d3d/d4d/d54/d4f/l4c to d7/l58 0 2024-04-20T21:31:54.944 INFO:tasks.workunit.client.0.smithi062.stdout:7/265: chown d7/d3e/d33/d3d/d4d/d54/l3a 2 1 2024-04-20T21:31:54.946 INFO:tasks.workunit.client.0.smithi062.stdout:6/240: mkdir d0/d4f 0 2024-04-20T21:31:54.946 INFO:tasks.workunit.client.0.smithi062.stdout:8/329: truncate d7/d9/d1b/f4e 4839601 0 2024-04-20T21:31:54.949 INFO:tasks.workunit.client.0.smithi062.stdout:4/259: sync 2024-04-20T21:31:54.949 INFO:tasks.workunit.client.0.smithi062.stdout:0/283: sync 2024-04-20T21:31:54.949 INFO:tasks.workunit.client.0.smithi062.stdout:9/290: sync 2024-04-20T21:31:54.949 INFO:tasks.workunit.client.0.smithi062.stdout:1/330: sync 2024-04-20T21:31:54.952 INFO:tasks.workunit.client.0.smithi062.stdout:6/241: truncate d0/d1a/f29 1532272 0 2024-04-20T21:31:54.954 INFO:tasks.workunit.client.0.smithi062.stdout:8/330: write d7/d9/d1b/f1e [1578599,106599] 0 2024-04-20T21:31:54.954 INFO:tasks.workunit.client.0.smithi062.stdout:7/266: link d7/c13 d7/c59 0 2024-04-20T21:31:54.957 INFO:tasks.workunit.client.0.smithi062.stdout:9/291: creat d16/d3e/f64 x:0 0 0 2024-04-20T21:31:54.959 INFO:tasks.workunit.client.0.smithi062.stdout:4/260: mknod da/d4f/c58 0 2024-04-20T21:31:54.964 INFO:tasks.workunit.client.0.smithi062.stdout:5/282: dwrite d8/d10/f1b [0,4194304] 0 2024-04-20T21:31:54.967 INFO:tasks.workunit.client.0.smithi062.stdout:2/318: dwrite d7/f17 [0,4194304] 0 2024-04-20T21:31:54.981 INFO:tasks.workunit.client.0.smithi062.stdout:5/283: chown d8/d10/f17 61311969 1 2024-04-20T21:31:54.981 INFO:tasks.workunit.client.0.smithi062.stdout:4/261: mknod da/dd/df/d27/d30/c59 0 2024-04-20T21:31:54.982 INFO:tasks.workunit.client.0.smithi062.stdout:2/319: symlink d7/da/d15/d1b/l5e 0 2024-04-20T21:31:54.986 INFO:tasks.workunit.client.0.smithi062.stdout:7/267: dwrite d7/d26/f2f [0,4194304] 0 2024-04-20T21:31:54.986 INFO:tasks.workunit.client.0.smithi062.stdout:1/331: link dc/d15/d20/c2e dc/d15/d17/d36/c72 0 2024-04-20T21:31:54.986 INFO:tasks.workunit.client.0.smithi062.stdout:0/284: dread d7/f4c [0,4194304] 0 2024-04-20T21:31:54.986 INFO:tasks.workunit.client.0.smithi062.stdout:9/292: mknod d16/d4e/c65 0 2024-04-20T21:31:54.988 INFO:tasks.workunit.client.0.smithi062.stdout:2/320: fsync d7/da/d28/f2a 0 2024-04-20T21:31:54.988 INFO:tasks.workunit.client.0.smithi062.stdout:7/268: fdatasync d7/f55 0 2024-04-20T21:31:54.989 INFO:tasks.workunit.client.0.smithi062.stdout:0/285: readlink d7/d1e/l43 0 2024-04-20T21:31:54.995 INFO:tasks.workunit.client.0.smithi062.stdout:7/269: dread d7/f24 [0,4194304] 0 2024-04-20T21:31:54.995 INFO:tasks.workunit.client.0.smithi062.stdout:3/326: dread d0/d11/f1a [4194304,4194304] 0 2024-04-20T21:31:54.996 INFO:tasks.workunit.client.0.smithi062.stdout:7/270: readlink d7/d3e/l32 0 2024-04-20T21:31:54.997 INFO:tasks.workunit.client.0.smithi062.stdout:6/242: link d0/d9/d11/c35 d0/d9/d11/d22/d26/c50 0 2024-04-20T21:31:54.998 INFO:tasks.workunit.client.0.smithi062.stdout:4/262: rename da/c2e to da/dd/df/d27/d30/c5a 0 2024-04-20T21:31:54.998 INFO:tasks.workunit.client.0.smithi062.stdout:7/271: chown d7/d3e/f44 249700 1 2024-04-20T21:31:55.000 INFO:tasks.workunit.client.0.smithi062.stdout:8/331: link d7/d9/d1b/d24/d46/l5b d7/d9/l71 0 2024-04-20T21:31:55.003 INFO:tasks.workunit.client.0.smithi062.stdout:8/332: read d7/d9/f3a [1515659,16460] 0 2024-04-20T21:31:55.003 INFO:tasks.workunit.client.0.smithi062.stdout:2/321: dread d7/f17 [0,4194304] 0 2024-04-20T21:31:55.005 INFO:tasks.workunit.client.0.smithi062.stdout:1/332: mknod dc/d15/d17/c73 0 2024-04-20T21:31:55.006 INFO:tasks.workunit.client.0.smithi062.stdout:3/327: creat d0/d1b/f75 x:0 0 0 2024-04-20T21:31:55.009 INFO:tasks.workunit.client.0.smithi062.stdout:1/333: write dc/d15/d20/d28/f63 [685717,106937] 0 2024-04-20T21:31:55.013 INFO:tasks.workunit.client.0.smithi062.stdout:9/293: creat d16/f66 x:0 0 0 2024-04-20T21:31:55.015 INFO:tasks.workunit.client.0.smithi062.stdout:8/333: creat d7/d25/d5c/f72 x:0 0 0 2024-04-20T21:31:55.017 INFO:tasks.workunit.client.0.smithi062.stdout:4/263: dread da/dd/df/d27/f36 [0,4194304] 0 2024-04-20T21:31:55.018 INFO:tasks.workunit.client.0.smithi062.stdout:7/272: dwrite d7/d3e/d33/d3d/d4d/d54/f2e [0,4194304] 0 2024-04-20T21:31:55.020 INFO:tasks.workunit.client.0.smithi062.stdout:1/334: creat dc/d27/d32/f74 x:0 0 0 2024-04-20T21:31:55.021 INFO:tasks.workunit.client.0.smithi062.stdout:7/273: fdatasync d7/d3e/d33/f35 0 2024-04-20T21:31:55.021 INFO:tasks.workunit.client.0.smithi062.stdout:2/322: link d7/f5d d7/d39/f5f 0 2024-04-20T21:31:55.022 INFO:tasks.workunit.client.0.smithi062.stdout:9/294: mknod d16/d2c/d42/c67 0 2024-04-20T21:31:55.025 INFO:tasks.workunit.client.0.smithi062.stdout:2/323: truncate d7/d39/f45 1222710 0 2024-04-20T21:31:55.025 INFO:tasks.workunit.client.0.smithi062.stdout:5/284: dread d8/d10/d32/d56/d2b/f40 [0,4194304] 0 2024-04-20T21:31:55.027 INFO:tasks.workunit.client.0.smithi062.stdout:0/286: rename f2 to d7/d1c/f63 0 2024-04-20T21:31:55.028 INFO:tasks.workunit.client.0.smithi062.stdout:5/285: stat d8/d10/d32/l45 0 2024-04-20T21:31:55.029 INFO:tasks.workunit.client.0.smithi062.stdout:6/243: rename d0/c1c to d0/c51 0 2024-04-20T21:31:55.031 INFO:tasks.workunit.client.0.smithi062.stdout:0/287: truncate d7/d1c/f18 4928172 0 2024-04-20T21:31:55.033 INFO:tasks.workunit.client.0.smithi062.stdout:7/274: creat d7/d3e/d33/d3d/d4d/f5a x:0 0 0 2024-04-20T21:31:55.033 INFO:tasks.workunit.client.0.smithi062.stdout:9/295: symlink d16/d2f/d5b/l68 0 2024-04-20T21:31:55.035 INFO:tasks.workunit.client.0.smithi062.stdout:8/334: dread d7/d9/f51 [0,4194304] 0 2024-04-20T21:31:55.037 INFO:tasks.workunit.client.0.smithi062.stdout:2/324: dread d7/da/f4d [0,4194304] 0 2024-04-20T21:31:55.037 INFO:tasks.workunit.client.0.smithi062.stdout:0/288: write d7/d1c/f32 [1619583,35705] 0 2024-04-20T21:31:55.038 INFO:tasks.workunit.client.0.smithi062.stdout:4/264: mkdir da/dd/df/d27/d30/d57/d5b 0 2024-04-20T21:31:55.039 INFO:tasks.workunit.client.0.smithi062.stdout:5/286: symlink d8/d10/d32/l5c 0 2024-04-20T21:31:55.045 INFO:tasks.workunit.client.0.smithi062.stdout:5/287: write d8/d10/f1b [3681523,72285] 0 2024-04-20T21:31:55.046 INFO:tasks.workunit.client.0.smithi062.stdout:7/275: symlink d7/d3e/d33/d3d/d4d/d54/l5b 0 2024-04-20T21:31:55.047 INFO:tasks.workunit.client.0.smithi062.stdout:3/328: dwrite d0/f4f [0,4194304] 0 2024-04-20T21:31:55.048 INFO:tasks.workunit.client.0.smithi062.stdout:4/265: rename da/dd/df/c25 to da/d56/c5c 0 2024-04-20T21:31:55.049 INFO:tasks.workunit.client.0.smithi062.stdout:8/335: write d7/d25/d5c/f69 [258174,13274] 0 2024-04-20T21:31:55.049 INFO:tasks.workunit.client.0.smithi062.stdout:2/325: readlink d7/da/d15/d1b/d24/l4f 0 2024-04-20T21:31:55.052 INFO:tasks.workunit.client.0.smithi062.stdout:0/289: dread d7/d1c/d15/f2d [0,4194304] 0 2024-04-20T21:31:55.054 INFO:tasks.workunit.client.0.smithi062.stdout:2/326: write d7/da/f4d [1463521,69192] 0 2024-04-20T21:31:55.054 INFO:tasks.workunit.client.0.smithi062.stdout:4/266: chown da/dd/df/f28 209285 1 2024-04-20T21:31:55.054 INFO:tasks.workunit.client.0.smithi062.stdout:1/335: dread dc/d15/f18 [0,4194304] 0 2024-04-20T21:31:55.056 INFO:tasks.workunit.client.0.smithi062.stdout:5/288: read - d8/d10/d32/d56/d2b/d4c/f55 zero size 2024-04-20T21:31:55.062 INFO:tasks.workunit.client.0.smithi062.stdout:2/327: creat d7/da/d15/d2f/d38/d51/f60 x:0 0 0 2024-04-20T21:31:55.064 INFO:tasks.workunit.client.0.smithi062.stdout:5/289: write d8/d10/d3e/f52 [1298301,16510] 0 2024-04-20T21:31:55.064 INFO:tasks.workunit.client.0.smithi062.stdout:0/290: unlink d7/d1c/c33 0 2024-04-20T21:31:55.064 INFO:tasks.workunit.client.0.smithi062.stdout:6/244: dread d0/d2c/f3b [0,4194304] 0 2024-04-20T21:31:55.065 INFO:tasks.workunit.client.0.smithi062.stdout:1/336: mkdir dc/d15/d75 0 2024-04-20T21:31:55.069 INFO:tasks.workunit.client.0.smithi062.stdout:2/328: truncate d7/d39/f5f 145914 0 2024-04-20T21:31:55.072 INFO:tasks.workunit.client.0.smithi062.stdout:6/245: fdatasync d0/d9/d11/f1f 0 2024-04-20T21:31:55.075 INFO:tasks.workunit.client.0.smithi062.stdout:2/329: chown d7/d39/f55 37886303 1 2024-04-20T21:31:55.079 INFO:tasks.workunit.client.0.smithi062.stdout:7/276: dwrite d7/f55 [0,4194304] 0 2024-04-20T21:31:55.101 INFO:tasks.workunit.client.0.smithi062.stdout:2/330: mknod d7/da/d15/d1b/d3d/c61 0 2024-04-20T21:31:55.102 INFO:tasks.workunit.client.0.smithi062.stdout:8/336: dwrite d7/d9/d29/f32 [0,4194304] 0 2024-04-20T21:31:55.109 INFO:tasks.workunit.client.0.smithi062.stdout:8/337: rmdir d7 39 2024-04-20T21:31:55.112 INFO:tasks.workunit.client.0.smithi062.stdout:7/277: write d7/f10 [5711696,121089] 0 2024-04-20T21:31:55.115 INFO:tasks.workunit.client.0.smithi062.stdout:3/329: dwrite d0/d1/d10/d31/f70 [0,4194304] 0 2024-04-20T21:31:55.115 INFO:tasks.workunit.client.0.smithi062.stdout:2/331: symlink d7/da/d15/l62 0 2024-04-20T21:31:55.121 INFO:tasks.workunit.client.0.smithi062.stdout:0/291: dwrite d7/d1e/f24 [0,4194304] 0 2024-04-20T21:31:55.121 INFO:tasks.workunit.client.0.smithi062.stdout:5/290: dwrite d8/d1e/f25 [4194304,4194304] 0 2024-04-20T21:31:55.122 INFO:tasks.workunit.client.0.smithi062.stdout:0/292: fsync f1 0 2024-04-20T21:31:55.124 INFO:tasks.workunit.client.0.smithi062.stdout:4/267: dread da/f2f [0,4194304] 0 2024-04-20T21:31:55.125 INFO:tasks.workunit.client.0.smithi062.stdout:7/278: symlink d7/d3e/d33/d3d/d4d/l5c 0 2024-04-20T21:31:55.125 INFO:tasks.workunit.client.0.smithi062.stdout:3/330: fsync d0/d11/f1a 0 2024-04-20T21:31:55.126 INFO:tasks.workunit.client.0.smithi062.stdout:2/332: readlink d7/da/d15/l31 0 2024-04-20T21:31:55.129 INFO:tasks.workunit.client.0.smithi062.stdout:8/338: symlink d7/d25/d70/l73 0 2024-04-20T21:31:55.131 INFO:tasks.workunit.client.0.smithi062.stdout:5/291: creat d8/d1e/f5d x:0 0 0 2024-04-20T21:31:55.131 INFO:tasks.workunit.client.0.smithi062.stdout:1/337: dread dc/fe [0,4194304] 0 2024-04-20T21:31:55.132 INFO:tasks.workunit.client.0.smithi062.stdout:2/333: write d7/da/f3c [567452,121290] 0 2024-04-20T21:31:55.132 INFO:tasks.workunit.client.0.smithi062.stdout:4/268: creat da/dd/d15/f5d x:0 0 0 2024-04-20T21:31:55.134 INFO:tasks.workunit.client.0.smithi062.stdout:7/279: creat d7/d3e/f5d x:0 0 0 2024-04-20T21:31:55.135 INFO:tasks.workunit.client.0.smithi062.stdout:5/292: write d8/d38/f3d [140122,2336] 0 2024-04-20T21:31:55.135 INFO:tasks.workunit.client.0.smithi062.stdout:3/331: mkdir d0/d5c/d76 0 2024-04-20T21:31:55.138 INFO:tasks.workunit.client.0.smithi062.stdout:2/334: dread - d7/da/d15/d1b/d3d/f49 zero size 2024-04-20T21:31:55.138 INFO:tasks.workunit.client.0.smithi062.stdout:0/293: rename d7/d1c/d15/d17/d1b/d4e to d7/d1c/d15/d17/d37/d64 0 2024-04-20T21:31:55.139 INFO:tasks.workunit.client.0.smithi062.stdout:1/338: mkdir dc/d15/d20/d28/d76 0 2024-04-20T21:31:55.140 INFO:tasks.workunit.client.0.smithi062.stdout:7/280: write d7/d3e/f44 [235985,77279] 0 2024-04-20T21:31:55.142 INFO:tasks.workunit.client.0.smithi062.stdout:4/269: creat da/dd/df/d27/d30/d3d/f5e x:0 0 0 2024-04-20T21:31:55.144 INFO:tasks.workunit.client.0.smithi062.stdout:2/335: symlink d7/da/d15/d2f/l63 0 2024-04-20T21:31:55.148 INFO:tasks.workunit.client.0.smithi062.stdout:7/281: mkdir d7/d3e/d33/d3d/d5e 0 2024-04-20T21:31:55.150 INFO:tasks.workunit.client.0.smithi062.stdout:4/270: mknod da/dd/df/d27/d30/c5f 0 2024-04-20T21:31:55.152 INFO:tasks.workunit.client.0.smithi062.stdout:3/332: creat d0/d1/f77 x:0 0 0 2024-04-20T21:31:55.157 INFO:tasks.workunit.client.0.smithi062.stdout:2/336: truncate d7/f12 2891677 0 2024-04-20T21:31:55.157 INFO:tasks.workunit.client.0.smithi062.stdout:3/333: fdatasync d0/d1/d10/f4e 0 2024-04-20T21:31:55.158 INFO:tasks.workunit.client.0.smithi062.stdout:1/339: dread dc/d15/d17/f34 [0,4194304] 0 2024-04-20T21:31:55.160 INFO:tasks.workunit.client.0.smithi062.stdout:7/282: creat d7/f5f x:0 0 0 2024-04-20T21:31:55.161 INFO:tasks.workunit.client.0.smithi062.stdout:4/271: dread da/dd/df/d27/f2c [0,4194304] 0 2024-04-20T21:31:55.163 INFO:tasks.workunit.client.0.smithi062.stdout:0/294: dwrite d7/d1c/d15/d17/d37/f59 [0,4194304] 0 2024-04-20T21:31:55.164 INFO:tasks.workunit.client.0.smithi062.stdout:5/293: truncate d8/d1e/f25 1032875 0 2024-04-20T21:31:55.165 INFO:tasks.workunit.client.0.smithi062.stdout:7/283: unlink d7/f10 0 2024-04-20T21:31:55.165 INFO:tasks.workunit.client.0.smithi062.stdout:4/272: write da/dd/f4d [651390,112650] 0 2024-04-20T21:31:55.167 INFO:tasks.workunit.client.0.smithi062.stdout:7/284: readlink d7/l1d 0 2024-04-20T21:31:55.168 INFO:tasks.workunit.client.0.smithi062.stdout:2/337: unlink d7/da/d15/d1b/d24/f3f 0 2024-04-20T21:31:55.169 INFO:tasks.workunit.client.0.smithi062.stdout:1/340: symlink dc/d15/d75/l77 0 2024-04-20T21:31:55.172 INFO:tasks.workunit.client.0.smithi062.stdout:0/295: symlink d7/d1c/d15/d17/d37/l65 0 2024-04-20T21:31:55.173 INFO:tasks.workunit.client.0.smithi062.stdout:5/294: symlink d8/d10/d32/d56/d2f/l5e 0 2024-04-20T21:31:55.175 INFO:tasks.workunit.client.0.smithi062.stdout:4/273: mknod da/dd/df/d27/d32/c60 0 2024-04-20T21:31:55.176 INFO:tasks.workunit.client.0.smithi062.stdout:7/285: read d7/f36 [1076111,79679] 0 2024-04-20T21:31:55.177 INFO:tasks.workunit.client.0.smithi062.stdout:2/338: symlink d7/da/l64 0 2024-04-20T21:31:55.177 INFO:tasks.workunit.client.0.smithi062.stdout:5/295: fdatasync d8/d10/d32/d56/d2b/d4c/f55 0 2024-04-20T21:31:55.178 INFO:tasks.workunit.client.0.smithi062.stdout:7/286: readlink d7/l1d 0 2024-04-20T21:31:55.178 INFO:tasks.workunit.client.0.smithi062.stdout:0/296: fdatasync d7/d1c/d15/d17/d37/f5b 0 2024-04-20T21:31:55.182 INFO:tasks.workunit.client.0.smithi062.stdout:4/274: mknod da/c61 0 2024-04-20T21:31:55.183 INFO:tasks.workunit.client.0.smithi062.stdout:4/275: truncate da/f2f 4691095 0 2024-04-20T21:31:55.184 INFO:tasks.workunit.client.0.smithi062.stdout:2/339: rmdir d7/da/d28 39 2024-04-20T21:31:55.184 INFO:tasks.workunit.client.0.smithi062.stdout:3/334: dread d0/f53 [4194304,4194304] 0 2024-04-20T21:31:55.186 INFO:tasks.workunit.client.0.smithi062.stdout:7/287: symlink d7/d3e/d33/l60 0 2024-04-20T21:31:55.189 INFO:tasks.workunit.client.0.smithi062.stdout:4/276: symlink da/dd/df/d27/d30/l62 0 2024-04-20T21:31:55.192 INFO:tasks.workunit.client.0.smithi062.stdout:2/340: mknod d7/da/d15/d1b/d3d/c65 0 2024-04-20T21:31:55.195 INFO:tasks.workunit.client.0.smithi062.stdout:7/288: rename f6 to d7/d3e/d33/d3d/d5e/f61 0 2024-04-20T21:31:55.195 INFO:tasks.workunit.client.0.smithi062.stdout:1/341: dwrite dc/d15/d20/d28/f4e [0,4194304] 0 2024-04-20T21:31:55.198 INFO:tasks.workunit.client.0.smithi062.stdout:7/289: fdatasync d7/d3e/f30 0 2024-04-20T21:31:55.198 INFO:tasks.workunit.client.0.smithi062.stdout:5/296: getdents d8/d10/d32/d56/d2b/d4c 0 2024-04-20T21:31:55.199 INFO:tasks.workunit.client.0.smithi062.stdout:2/341: chown d7/da/d15/d2f/d38/f4e 39370 1 2024-04-20T21:31:55.202 INFO:tasks.workunit.client.0.smithi062.stdout:4/277: creat da/dd/df/d27/d30/d57/f63 x:0 0 0 2024-04-20T21:31:55.207 INFO:tasks.workunit.client.0.smithi062.stdout:1/342: mkdir dc/d15/d17/d36/d78 0 2024-04-20T21:31:55.210 INFO:tasks.workunit.client.0.smithi062.stdout:0/297: dwrite d7/f4c [0,4194304] 0 2024-04-20T21:31:55.212 INFO:tasks.workunit.client.0.smithi062.stdout:4/278: stat da/dd/d15/l46 0 2024-04-20T21:31:55.212 INFO:tasks.workunit.client.0.smithi062.stdout:5/297: creat d8/d10/d32/d56/d50/f5f x:0 0 0 2024-04-20T21:31:55.216 INFO:tasks.workunit.client.0.smithi062.stdout:3/335: dwrite d0/d1/f32 [0,4194304] 0 2024-04-20T21:31:55.219 INFO:tasks.workunit.client.0.smithi062.stdout:1/343: write dc/d15/d17/d36/d3a/f59 [73330,50102] 0 2024-04-20T21:31:55.219 INFO:tasks.workunit.client.0.smithi062.stdout:0/298: symlink d7/d1c/d15/d17/d1b/l66 0 2024-04-20T21:31:55.223 INFO:tasks.workunit.client.0.smithi062.stdout:7/290: dread d7/d26/f2f [0,4194304] 0 2024-04-20T21:31:55.229 INFO:tasks.workunit.client.0.smithi062.stdout:4/279: rename da/dd/df/d27/d30 to da/dd/d15/d50/d64 0 2024-04-20T21:31:55.230 INFO:tasks.workunit.client.0.smithi062.stdout:3/336: creat d0/d1/d10/d3a/d5e/f78 x:0 0 0 2024-04-20T21:31:55.230 INFO:tasks.workunit.client.0.smithi062.stdout:2/342: dwrite d7/da/f4d [0,4194304] 0 2024-04-20T21:31:55.233 INFO:tasks.workunit.client.0.smithi062.stdout:1/344: unlink dc/d15/d17/c5c 0 2024-04-20T21:31:55.235 INFO:tasks.workunit.client.0.smithi062.stdout:3/337: write d0/d11/f1a [5245111,244] 0 2024-04-20T21:31:55.238 INFO:tasks.workunit.client.0.smithi062.stdout:4/280: chown da/dd/d15/l46 0 1 2024-04-20T21:31:55.239 INFO:tasks.workunit.client.0.smithi062.stdout:2/343: creat d7/da/d15/d1b/d24/f66 x:0 0 0 2024-04-20T21:31:55.249 INFO:tasks.workunit.client.0.smithi062.stdout:5/298: dwrite d8/d10/d32/d56/d2b/f40 [0,4194304] 0 2024-04-20T21:31:55.249 INFO:tasks.workunit.client.0.smithi062.stdout:1/345: getdents dc/d27 0 2024-04-20T21:31:55.250 INFO:tasks.workunit.client.0.smithi062.stdout:3/338: creat d0/d1/d10/f79 x:0 0 0 2024-04-20T21:31:55.252 INFO:tasks.workunit.client.0.smithi062.stdout:1/346: write dc/d27/f62 [3674907,36198] 0 2024-04-20T21:31:55.252 INFO:tasks.workunit.client.0.smithi062.stdout:5/299: rmdir d8/d38 39 2024-04-20T21:31:55.255 INFO:tasks.workunit.client.0.smithi062.stdout:0/299: dwrite d7/d1c/f63 [0,4194304] 0 2024-04-20T21:31:55.255 INFO:tasks.workunit.client.0.smithi062.stdout:5/300: chown d8/d10/d3e 3644084 1 2024-04-20T21:31:55.256 INFO:tasks.workunit.client.0.smithi062.stdout:0/300: chown d7/f4c 1 1 2024-04-20T21:31:55.257 INFO:tasks.workunit.client.0.smithi062.stdout:1/347: write dc/d27/d32/f51 [162768,29915] 0 2024-04-20T21:31:55.258 INFO:tasks.workunit.client.0.smithi062.stdout:0/301: write f1 [623435,19536] 0 2024-04-20T21:31:55.263 INFO:tasks.workunit.client.0.smithi062.stdout:5/301: rmdir d8/d1e 39 2024-04-20T21:31:55.263 INFO:tasks.workunit.client.0.smithi062.stdout:1/348: read - dc/d15/d20/f3e zero size 2024-04-20T21:31:55.265 INFO:tasks.workunit.client.0.smithi062.stdout:3/339: creat d0/d1/d10/f7a x:0 0 0 2024-04-20T21:31:55.276 INFO:tasks.workunit.client.0.smithi062.stdout:4/281: dread da/f13 [0,4194304] 0 2024-04-20T21:31:55.277 INFO:tasks.workunit.client.0.smithi062.stdout:3/340: rename d0/d1c/d63/d42/d44 to d0/d1c/d52/d7b 0 2024-04-20T21:31:55.278 INFO:tasks.workunit.client.0.smithi062.stdout:2/344: dwrite d7/f5d [0,4194304] 0 2024-04-20T21:31:55.280 INFO:tasks.workunit.client.0.smithi062.stdout:4/282: readlink da/dd/d3c/l4e 0 2024-04-20T21:31:55.281 INFO:tasks.workunit.client.0.smithi062.stdout:2/345: stat d7/da/d15/d1b/f2e 0 2024-04-20T21:31:55.281 INFO:tasks.workunit.client.0.smithi062.stdout:7/291: dread d7/f36 [0,4194304] 0 2024-04-20T21:31:55.281 INFO:tasks.workunit.client.0.smithi062.stdout:5/302: link d8/d1e/f53 d8/d10/d32/f60 0 2024-04-20T21:31:55.282 INFO:tasks.workunit.client.0.smithi062.stdout:4/283: chown da/dd/d15/l21 5829 1 2024-04-20T21:31:55.285 INFO:tasks.workunit.client.0.smithi062.stdout:3/341: creat d0/d1/d10/d31/d41/f7c x:0 0 0 2024-04-20T21:31:55.288 INFO:tasks.workunit.client.0.smithi062.stdout:4/284: chown da/d4f 222654343 1 2024-04-20T21:31:55.288 INFO:tasks.workunit.client.0.smithi062.stdout:5/303: symlink d8/d10/d32/l61 0 2024-04-20T21:31:55.290 INFO:tasks.workunit.client.0.smithi062.stdout:0/302: dread d7/d1c/ff [0,4194304] 0 2024-04-20T21:31:55.292 INFO:tasks.workunit.client.0.smithi062.stdout:3/342: write d0/d1/d10/f64 [1365763,8760] 0 2024-04-20T21:31:55.297 INFO:tasks.workunit.client.0.smithi062.stdout:4/285: creat da/dd/d15/d50/d64/d57/f65 x:0 0 0 2024-04-20T21:31:55.297 INFO:tasks.workunit.client.0.smithi062.stdout:1/349: dwrite dc/d15/d17/d36/d3a/f43 [0,4194304] 0 2024-04-20T21:31:55.299 INFO:tasks.workunit.client.0.smithi062.stdout:3/343: creat d0/d1/d10/d3a/d5e/f7d x:0 0 0 2024-04-20T21:31:55.300 INFO:tasks.workunit.client.0.smithi062.stdout:1/350: stat dc/d15 0 2024-04-20T21:31:55.302 INFO:tasks.workunit.client.0.smithi062.stdout:0/303: read d7/d1c/f39 [4148441,20848] 0 2024-04-20T21:31:55.302 INFO:tasks.workunit.client.0.smithi062.stdout:5/304: link d8/d1e/d2c/f34 d8/d10/d3e/f62 0 2024-04-20T21:31:55.302 INFO:tasks.workunit.client.0.smithi062.stdout:3/344: write d0/d1b/f2b [444311,68759] 0 2024-04-20T21:31:55.304 INFO:tasks.workunit.client.0.smithi062.stdout:4/286: symlink da/d4f/l66 0 2024-04-20T21:31:55.306 INFO:tasks.workunit.client.0.smithi062.stdout:5/305: dread - d8/d1e/f5d zero size 2024-04-20T21:31:55.307 INFO:tasks.workunit.client.0.smithi062.stdout:4/287: readlink da/dd/d3c/l54 0 2024-04-20T21:31:55.309 INFO:tasks.workunit.client.0.smithi062.stdout:7/292: dwrite d7/d3e/f41 [0,4194304] 0 2024-04-20T21:31:55.311 INFO:tasks.workunit.client.0.smithi062.stdout:3/345: rename d0/d1/c6e to d0/d11/c7e 0 2024-04-20T21:31:55.314 INFO:tasks.workunit.client.0.smithi062.stdout:7/293: write d7/f55 [5127816,3684] 0 2024-04-20T21:31:55.317 INFO:tasks.workunit.client.0.smithi062.stdout:4/288: mknod da/dd/d15/d50/d64/d57/c67 0 2024-04-20T21:31:55.321 INFO:tasks.workunit.client.0.smithi062.stdout:3/346: link d0/f4f d0/d1c/d52/d7b/f7f 0 2024-04-20T21:31:55.322 INFO:tasks.workunit.client.0.smithi062.stdout:4/289: write da/dd/d15/f1e [3415472,40530] 0 2024-04-20T21:31:55.322 INFO:tasks.workunit.client.0.smithi062.stdout:0/304: dread d7/d31/f57 [0,4194304] 0 2024-04-20T21:31:55.323 INFO:tasks.workunit.client.0.smithi062.stdout:7/294: symlink d7/d3e/d33/d3d/d4d/l62 0 2024-04-20T21:31:55.324 INFO:tasks.workunit.client.0.smithi062.stdout:5/306: dread d8/d10/d32/d56/d2b/f36 [0,4194304] 0 2024-04-20T21:31:55.329 INFO:tasks.workunit.client.0.smithi062.stdout:3/347: readlink d0/d1c/d63/d42/l69 0 2024-04-20T21:31:55.329 INFO:tasks.workunit.client.0.smithi062.stdout:1/351: dwrite dc/d15/f48 [0,4194304] 0 2024-04-20T21:31:55.329 INFO:tasks.workunit.client.0.smithi062.stdout:4/290: write da/dd/d15/f20 [4678468,96496] 0 2024-04-20T21:31:55.329 INFO:tasks.workunit.client.0.smithi062.stdout:7/295: mkdir d7/d63 0 2024-04-20T21:31:55.332 INFO:tasks.workunit.client.0.smithi062.stdout:4/291: truncate da/dd/f45 570606 0 2024-04-20T21:31:55.332 INFO:tasks.workunit.client.0.smithi062.stdout:0/305: read d7/d1c/d15/d17/d37/f5b [1086151,77959] 0 2024-04-20T21:31:55.335 INFO:tasks.workunit.client.0.smithi062.stdout:3/348: mknod d0/d1/d10/c80 0 2024-04-20T21:31:55.337 INFO:tasks.workunit.client.0.smithi062.stdout:3/349: write d0/d5c/f5a [4252774,58405] 0 2024-04-20T21:31:55.339 INFO:tasks.workunit.client.0.smithi062.stdout:1/352: truncate dc/d15/d17/d36/d3a/f46 1186500 0 2024-04-20T21:31:55.339 INFO:tasks.workunit.client.0.smithi062.stdout:4/292: rmdir da/dd/d15/d50/d64/d57 39 2024-04-20T21:31:55.344 INFO:tasks.workunit.client.0.smithi062.stdout:5/307: getdents d8/d10/d32/d56/d2b 0 2024-04-20T21:31:55.346 INFO:tasks.workunit.client.0.smithi062.stdout:4/293: rmdir da/dd/d3c 39 2024-04-20T21:31:55.348 INFO:tasks.workunit.client.0.smithi062.stdout:3/350: symlink d0/d1c/d63/d42/d5b/l81 0 2024-04-20T21:31:55.352 INFO:tasks.workunit.client.0.smithi062.stdout:4/294: write da/dd/d15/f5d [489496,60767] 0 2024-04-20T21:31:55.352 INFO:tasks.workunit.client.0.smithi062.stdout:7/296: dwrite d7/d3e/d33/f35 [0,4194304] 0 2024-04-20T21:31:55.363 INFO:tasks.workunit.client.0.smithi062.stdout:5/308: link d8/d10/d32/d56/l5a d8/d10/l63 0 2024-04-20T21:31:55.363 INFO:tasks.workunit.client.0.smithi062.stdout:7/297: link d7/d26/c3c d7/d26/c64 0 2024-04-20T21:31:55.365 INFO:tasks.workunit.client.0.smithi062.stdout:0/306: dwrite d7/d1c/f11 [0,4194304] 0 2024-04-20T21:31:55.368 INFO:tasks.workunit.client.0.smithi062.stdout:5/309: truncate d8/ff 4505378 0 2024-04-20T21:31:55.368 INFO:tasks.workunit.client.0.smithi062.stdout:0/307: chown d7/d1c/d15/f2d 142075608 1 2024-04-20T21:31:55.372 INFO:tasks.workunit.client.0.smithi062.stdout:0/308: read d7/d1c/d15/d17/f1d [3317936,21133] 0 2024-04-20T21:31:55.375 INFO:tasks.workunit.client.0.smithi062.stdout:5/310: truncate d8/d10/d3e/f5b 966506 0 2024-04-20T21:31:55.378 INFO:tasks.workunit.client.0.smithi062.stdout:5/311: fdatasync d8/d10/d32/f3c 0 2024-04-20T21:31:55.378 INFO:tasks.workunit.client.0.smithi062.stdout:3/351: dwrite d0/d1/d10/d3a/d5e/f7d [0,4194304] 0 2024-04-20T21:31:55.380 INFO:tasks.workunit.client.0.smithi062.stdout:1/353: dread dc/d15/d17/d36/d3a/f4f [8388608,4194304] 0 2024-04-20T21:31:55.382 INFO:tasks.workunit.client.0.smithi062.stdout:3/352: write d0/d1/f20 [3117518,75551] 0 2024-04-20T21:31:55.387 INFO:tasks.workunit.client.0.smithi062.stdout:4/295: dwrite da/dd/df/d27/d3f/f4c [0,4194304] 0 2024-04-20T21:31:55.390 INFO:tasks.workunit.client.0.smithi062.stdout:7/298: truncate d7/d26/f2f 212979 0 2024-04-20T21:31:55.396 INFO:tasks.workunit.client.0.smithi062.stdout:4/296: write da/dd/f14 [261980,109600] 0 2024-04-20T21:31:55.396 INFO:tasks.workunit.client.0.smithi062.stdout:7/299: rmdir d7/d3e/d33 39 2024-04-20T21:31:55.398 INFO:tasks.workunit.client.0.smithi062.stdout:0/309: dwrite d7/d1c/f11 [0,4194304] 0 2024-04-20T21:31:55.401 INFO:tasks.workunit.client.0.smithi062.stdout:3/353: fdatasync d0/d1/d10/d3a/d5e/f7d 0 2024-04-20T21:31:55.402 INFO:tasks.workunit.client.0.smithi062.stdout:7/300: chown d7/d3e/d33/d3d/d5e 160 1 2024-04-20T21:31:55.404 INFO:tasks.workunit.client.0.smithi062.stdout:1/354: dread dc/f53 [0,4194304] 0 2024-04-20T21:31:55.405 INFO:tasks.workunit.client.0.smithi062.stdout:4/297: stat da/dd/d15/d50/d64/c5a 0 2024-04-20T21:31:55.410 INFO:tasks.workunit.client.0.smithi062.stdout:0/310: write d7/d1c/d15/f2d [916554,65537] 0 2024-04-20T21:31:55.413 INFO:tasks.workunit.client.0.smithi062.stdout:0/311: write d7/f9 [7832942,99436] 0 2024-04-20T21:31:55.416 INFO:tasks.workunit.client.0.smithi062.stdout:3/354: creat d0/d1/f82 x:0 0 0 2024-04-20T21:31:55.419 INFO:tasks.workunit.client.0.smithi062.stdout:0/312: creat d7/d1e/f67 x:0 0 0 2024-04-20T21:31:55.421 INFO:tasks.workunit.client.0.smithi062.stdout:4/298: creat da/dd/d15/d50/d64/d57/d5b/f68 x:0 0 0 2024-04-20T21:31:55.422 INFO:tasks.workunit.client.0.smithi062.stdout:5/312: dwrite d8/d1e/d2c/f34 [0,4194304] 0 2024-04-20T21:31:55.424 INFO:tasks.workunit.client.0.smithi062.stdout:1/355: dread dc/d15/d17/f19 [0,4194304] 0 2024-04-20T21:31:55.426 INFO:tasks.workunit.client.0.smithi062.stdout:0/313: creat d7/d1c/d15/d17/d26/f68 x:0 0 0 2024-04-20T21:31:55.427 INFO:tasks.workunit.client.0.smithi062.stdout:1/356: readlink dc/d15/d20/l3b 0 2024-04-20T21:31:55.429 INFO:tasks.workunit.client.0.smithi062.stdout:5/313: creat d8/d10/d32/d56/d50/f64 x:0 0 0 2024-04-20T21:31:55.432 INFO:tasks.workunit.client.0.smithi062.stdout:0/314: mknod d7/d1c/d15/d17/c69 0 2024-04-20T21:31:55.432 INFO:tasks.workunit.client.0.smithi062.stdout:3/355: getdents d0/d1c/d52/d7b 0 2024-04-20T21:31:55.433 INFO:tasks.workunit.client.0.smithi062.stdout:0/315: chown d7/d1c/f18 368 1 2024-04-20T21:31:55.433 INFO:tasks.workunit.client.0.smithi062.stdout:1/357: mknod dc/d70/c79 0 2024-04-20T21:31:55.435 INFO:tasks.workunit.client.0.smithi062.stdout:5/314: creat d8/d10/d32/f65 x:0 0 0 2024-04-20T21:31:55.438 INFO:tasks.workunit.client.0.smithi062.stdout:0/316: mknod d7/d1e/c6a 0 2024-04-20T21:31:55.441 INFO:tasks.workunit.client.0.smithi062.stdout:1/358: unlink dc/d15/d17/c24 0 2024-04-20T21:31:55.442 INFO:tasks.workunit.client.0.smithi062.stdout:5/315: mkdir d8/d10/d32/d56/d66 0 2024-04-20T21:31:55.442 INFO:tasks.workunit.client.0.smithi062.stdout:7/301: dread d7/d3e/d33/f57 [0,4194304] 0 2024-04-20T21:31:55.443 INFO:tasks.workunit.client.0.smithi062.stdout:0/317: mkdir d7/d1c/d15/d6b 0 2024-04-20T21:31:55.448 INFO:tasks.workunit.client.0.smithi062.stdout:5/316: mknod d8/d10/d3e/c67 0 2024-04-20T21:31:55.449 INFO:tasks.workunit.client.0.smithi062.stdout:4/299: dwrite da/dd/df/f28 [0,4194304] 0 2024-04-20T21:31:55.450 INFO:tasks.workunit.client.0.smithi062.stdout:1/359: mknod dc/d15/d17/d36/c7a 0 2024-04-20T21:31:55.450 INFO:tasks.workunit.client.0.smithi062.stdout:3/356: dread d0/f4f [0,4194304] 0 2024-04-20T21:31:55.453 INFO:tasks.workunit.client.0.smithi062.stdout:0/318: link d7/c16 d7/d20/c6c 0 2024-04-20T21:31:55.454 INFO:tasks.workunit.client.0.smithi062.stdout:1/360: write dc/d15/d17/d36/f54 [2835961,4316] 0 2024-04-20T21:31:55.458 INFO:tasks.workunit.client.0.smithi062.stdout:9/296: sync 2024-04-20T21:31:55.460 INFO:tasks.workunit.client.0.smithi062.stdout:3/357: dread d0/d1/d10/f47 [0,4194304] 0 2024-04-20T21:31:55.463 INFO:tasks.workunit.client.0.smithi062.stdout:5/317: getdents d8/d10/d32/d56 0 2024-04-20T21:31:55.464 INFO:tasks.workunit.client.0.smithi062.stdout:9/297: rmdir d16/d3e 39 2024-04-20T21:31:55.465 INFO:tasks.workunit.client.0.smithi062.stdout:4/300: getdents da/dd/df/d27/d32 0 2024-04-20T21:31:55.466 INFO:tasks.workunit.client.0.smithi062.stdout:1/361: symlink dc/d15/d17/d36/l7b 0 2024-04-20T21:31:55.467 INFO:tasks.workunit.client.0.smithi062.stdout:0/319: mknod d7/d1c/d15/d17/d1b/c6d 0 2024-04-20T21:31:55.468 INFO:tasks.workunit.client.0.smithi062.stdout:1/362: chown dc/f6d 40039631 1 2024-04-20T21:31:55.468 INFO:tasks.workunit.client.0.smithi062.stdout:5/318: rename d8/d10/d32/d56 to d8/d10/d32/d56/d2f/d68 22 2024-04-20T21:31:55.472 INFO:tasks.workunit.client.0.smithi062.stdout:0/320: write d7/d1c/f32 [92036,126932] 0 2024-04-20T21:31:55.472 INFO:tasks.workunit.client.0.smithi062.stdout:4/301: creat da/dd/f69 x:0 0 0 2024-04-20T21:31:55.473 INFO:tasks.workunit.client.0.smithi062.stdout:1/363: mkdir dc/d2d/d7c 0 2024-04-20T21:31:55.475 INFO:tasks.workunit.client.0.smithi062.stdout:5/319: creat d8/d1e/d2c/f69 x:0 0 0 2024-04-20T21:31:55.476 INFO:tasks.workunit.client.0.smithi062.stdout:7/302: dwrite d7/d26/f47 [0,4194304] 0 2024-04-20T21:31:55.478 INFO:tasks.workunit.client.0.smithi062.stdout:4/302: fdatasync da/dd/f45 0 2024-04-20T21:31:55.483 INFO:tasks.workunit.client.0.smithi062.stdout:4/303: fsync da/dd/d15/d50/d64/d57/f65 0 2024-04-20T21:31:55.486 INFO:tasks.workunit.client.0.smithi062.stdout:7/303: rename d7/d3e/c1e to d7/d63/c65 0 2024-04-20T21:31:55.487 INFO:tasks.workunit.client.0.smithi062.stdout:3/358: dwrite d0/f4c [0,4194304] 0 2024-04-20T21:31:55.487 INFO:tasks.workunit.client.1.smithi189.stderr:++ readlink -f fsstress 2024-04-20T21:31:55.488 INFO:tasks.workunit.client.0.smithi062.stdout:1/364: creat dc/d15/d17/d6f/f7d x:0 0 0 2024-04-20T21:31:55.492 INFO:tasks.workunit.client.0.smithi062.stdout:5/320: mknod d8/d10/d32/d56/d66/c6a 0 2024-04-20T21:31:55.492 INFO:tasks.workunit.client.0.smithi062.stdout:4/304: creat da/dd/df/d27/d32/f6a x:0 0 0 2024-04-20T21:31:55.492 INFO:tasks.workunit.client.0.smithi062.stdout:0/321: dread d7/fb [0,4194304] 0 2024-04-20T21:31:55.493 INFO:tasks.workunit.client.0.smithi062.stdout:3/359: symlink d0/d1/d3c/l83 0 2024-04-20T21:31:55.494 INFO:tasks.workunit.client.0.smithi062.stdout:1/365: creat dc/d15/d17/d36/f7e x:0 0 0 2024-04-20T21:31:55.494 INFO:tasks.workunit.client.1.smithi189.stderr:+ BIN=/home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress/fsstress 2024-04-20T21:31:55.494 INFO:tasks.workunit.client.1.smithi189.stderr:+ popd 2024-04-20T21:31:55.494 INFO:tasks.workunit.client.0.smithi062.stdout:0/322: chown d7 7738 1 2024-04-20T21:31:55.496 INFO:tasks.workunit.client.1.smithi189.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-04-20T21:31:55.496 INFO:tasks.workunit.client.1.smithi189.stderr:+ popd 2024-04-20T21:31:55.497 INFO:tasks.workunit.client.1.smithi189.stdout:~/cephtest/mnt.1/client.1/tmp 2024-04-20T21:31:55.498 INFO:tasks.workunit.client.0.smithi062.stdout:1/366: write dc/d15/d20/f45 [1425299,28243] 0 2024-04-20T21:31:55.498 INFO:tasks.workunit.client.1.smithi189.stderr:++ mktemp -d -p . 2024-04-20T21:31:55.500 INFO:tasks.workunit.client.0.smithi062.stdout:1/367: chown dc/d15/d20/c57 7283 1 2024-04-20T21:31:55.502 INFO:tasks.workunit.client.0.smithi062.stdout:5/321: rename d8/d10/d3e/f5b to d8/d10/d32/d56/f6b 0 2024-04-20T21:31:55.504 INFO:tasks.workunit.client.0.smithi062.stdout:9/298: dread d16/f1e [0,4194304] 0 2024-04-20T21:31:55.505 INFO:tasks.workunit.client.0.smithi062.stdout:3/360: symlink d0/d1/d10/d3a/d5e/l84 0 2024-04-20T21:31:55.505 INFO:tasks.workunit.client.0.smithi062.stdout:4/305: symlink da/dd/d15/d50/l6b 0 2024-04-20T21:31:55.506 INFO:tasks.workunit.client.0.smithi062.stdout:9/299: chown d16/d2c 2 1 2024-04-20T21:31:55.506 INFO:tasks.workunit.client.1.smithi189.stderr:+ T=./tmp.3tuvMuLnfy 2024-04-20T21:31:55.506 INFO:tasks.workunit.client.1.smithi189.stderr:+ /home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress/fsstress -d ./tmp.3tuvMuLnfy -l 1 -n 1000 -p 10 -v 2024-04-20T21:31:55.507 INFO:tasks.workunit.client.0.smithi062.stdout:7/304: write d7/f36 [1840495,75993] 0 2024-04-20T21:31:55.508 INFO:tasks.workunit.client.0.smithi062.stdout:1/368: mkdir dc/d70/d7f 0 2024-04-20T21:31:55.509 INFO:tasks.workunit.client.0.smithi062.stdout:9/300: mkdir d16/d2f/d69 0 2024-04-20T21:31:55.510 INFO:tasks.workunit.client.0.smithi062.stdout:1/369: read - dc/f55 zero size 2024-04-20T21:31:55.511 INFO:tasks.workunit.client.0.smithi062.stdout:7/305: readlink d7/d3e/l21 0 2024-04-20T21:31:55.512 INFO:tasks.workunit.client.0.smithi062.stdout:3/361: write d0/d1/d10/f47 [421233,31647] 0 2024-04-20T21:31:55.513 INFO:tasks.workunit.client.0.smithi062.stdout:6/246: sync 2024-04-20T21:31:55.513 INFO:tasks.workunit.client.0.smithi062.stdout:2/346: sync 2024-04-20T21:31:55.513 INFO:tasks.workunit.client.0.smithi062.stdout:8/339: sync 2024-04-20T21:31:55.514 INFO:tasks.workunit.client.1.smithi189.stdout:seed = 1713625046 2024-04-20T21:31:55.516 INFO:tasks.workunit.client.0.smithi062.stdout:1/370: write dc/d2d/f65 [413675,24940] 0 2024-04-20T21:31:55.517 INFO:tasks.workunit.client.0.smithi062.stdout:4/306: truncate da/dd/df/d27/d32/f6a 11288 0 2024-04-20T21:31:55.520 INFO:tasks.workunit.client.0.smithi062.stdout:3/362: write d0/d1c/d63/d42/f73 [67796,78195] 0 2024-04-20T21:31:55.521 INFO:tasks.workunit.client.0.smithi062.stdout:0/323: dwrite d7/d1c/d15/d17/d37/f61 [0,4194304] 0 2024-04-20T21:31:55.522 INFO:tasks.workunit.client.0.smithi062.stdout:4/307: write da/dd/f4d [310526,93313] 0 2024-04-20T21:31:55.524 INFO:tasks.workunit.client.0.smithi062.stdout:6/247: unlink d0/c42 0 2024-04-20T21:31:55.526 INFO:tasks.workunit.client.0.smithi062.stdout:2/347: creat d7/da/d15/d2f/d38/d51/f67 x:0 0 0 2024-04-20T21:31:55.528 INFO:tasks.workunit.client.1.smithi189.stdout:1/0: dwrite - no filename 2024-04-20T21:31:55.528 INFO:tasks.workunit.client.1.smithi189.stdout:1/1: stat - no entries 2024-04-20T21:31:55.528 INFO:tasks.workunit.client.1.smithi189.stdout:1/2: readlink - no filename 2024-04-20T21:31:55.528 INFO:tasks.workunit.client.1.smithi189.stdout:0/0: mknod c0 0 2024-04-20T21:31:55.531 INFO:tasks.workunit.client.1.smithi189.stdout:1/3: creat f0 x:0 0 0 2024-04-20T21:31:55.531 INFO:tasks.workunit.client.1.smithi189.stdout:2/0: truncate - no filename 2024-04-20T21:31:55.531 INFO:tasks.workunit.client.1.smithi189.stdout:2/1: fdatasync - no filename 2024-04-20T21:31:55.531 INFO:tasks.workunit.client.1.smithi189.stdout:2/2: write - no filename 2024-04-20T21:31:55.531 INFO:tasks.workunit.client.1.smithi189.stdout:2/3: stat - no entries 2024-04-20T21:31:55.538 INFO:tasks.workunit.client.0.smithi062.stdout:2/348: fsync d7/da/d15/d1b/f2e 0 2024-04-20T21:31:55.538 INFO:tasks.workunit.client.0.smithi062.stdout:3/363: creat d0/d1c/d52/d7b/f85 x:0 0 0 2024-04-20T21:31:55.540 INFO:tasks.workunit.client.0.smithi062.stdout:5/322: dread d8/d10/d32/d56/f6b [0,4194304] 0 2024-04-20T21:31:55.543 INFO:tasks.workunit.client.1.smithi189.stdout:0/1: rename c0 to c1 0 2024-04-20T21:31:55.543 INFO:tasks.workunit.client.1.smithi189.stdout:0/2: fsync - no filename 2024-04-20T21:31:55.543 INFO:tasks.workunit.client.1.smithi189.stdout:3/0: dwrite - no filename 2024-04-20T21:31:55.544 INFO:tasks.workunit.client.1.smithi189.stdout:2/4: mknod c0 0 2024-04-20T21:31:55.545 INFO:tasks.workunit.client.0.smithi062.stdout:7/306: dwrite d7/d3e/f41 [0,4194304] 0 2024-04-20T21:31:55.545 INFO:tasks.workunit.client.1.smithi189.stdout:0/3: creat f2 x:0 0 0 2024-04-20T21:31:55.546 INFO:tasks.workunit.client.1.smithi189.stdout:0/4: dread - f2 zero size 2024-04-20T21:31:55.546 INFO:tasks.workunit.client.0.smithi062.stdout:6/248: write d0/d9/f47 [806412,111601] 0 2024-04-20T21:31:55.547 INFO:tasks.workunit.client.1.smithi189.stdout:4/0: dwrite - no filename 2024-04-20T21:31:55.547 INFO:tasks.workunit.client.1.smithi189.stdout:4/1: fdatasync - no filename 2024-04-20T21:31:55.548 INFO:tasks.workunit.client.1.smithi189.stdout:4/2: readlink - no filename 2024-04-20T21:31:55.548 INFO:tasks.workunit.client.1.smithi189.stdout:4/3: fdatasync - no filename 2024-04-20T21:31:55.548 INFO:tasks.workunit.client.1.smithi189.stdout:2/5: unlink c0 0 2024-04-20T21:31:55.548 INFO:tasks.workunit.client.1.smithi189.stdout:2/6: truncate - no filename 2024-04-20T21:31:55.550 INFO:tasks.workunit.client.0.smithi062.stdout:7/307: read d7/f36 [1536235,93730] 0 2024-04-20T21:31:55.552 INFO:tasks.workunit.client.1.smithi189.stdout:0/5: mkdir d3 0 2024-04-20T21:31:55.552 INFO:tasks.workunit.client.1.smithi189.stdout:3/1: symlink l0 0 2024-04-20T21:31:55.552 INFO:tasks.workunit.client.1.smithi189.stdout:3/2: dwrite - no filename 2024-04-20T21:31:55.555 INFO:tasks.workunit.client.1.smithi189.stdout:5/0: write - no filename 2024-04-20T21:31:55.555 INFO:tasks.workunit.client.1.smithi189.stdout:5/1: chown . 1577180 1 2024-04-20T21:31:55.555 INFO:tasks.workunit.client.0.smithi062.stdout:0/324: dread d7/d1c/f11 [0,4194304] 0 2024-04-20T21:31:55.555 INFO:tasks.workunit.client.0.smithi062.stdout:4/308: dwrite da/dd/f69 [0,4194304] 0 2024-04-20T21:31:55.556 INFO:tasks.workunit.client.0.smithi062.stdout:8/340: link d7/d9/d29/f4b d7/d9/d1b/d24/f74 0 2024-04-20T21:31:55.556 INFO:tasks.workunit.client.1.smithi189.stdout:3/3: chown l0 1473 1 2024-04-20T21:31:55.556 INFO:tasks.workunit.client.1.smithi189.stdout:4/4: creat f0 x:0 0 0 2024-04-20T21:31:55.557 INFO:tasks.workunit.client.0.smithi062.stdout:2/349: dread - d7/da/d15/f1d zero size 2024-04-20T21:31:55.561 INFO:tasks.workunit.client.0.smithi062.stdout:1/371: dwrite dc/f53 [8388608,4194304] 0 2024-04-20T21:31:55.561 INFO:tasks.workunit.client.0.smithi062.stdout:8/341: readlink d7/d9/d1b/l35 0 2024-04-20T21:31:55.563 INFO:tasks.workunit.client.1.smithi189.stdout:6/0: truncate - no filename 2024-04-20T21:31:55.563 INFO:tasks.workunit.client.1.smithi189.stdout:6/1: write - no filename 2024-04-20T21:31:55.563 INFO:tasks.workunit.client.1.smithi189.stdout:6/2: dwrite - no filename 2024-04-20T21:31:55.563 INFO:tasks.workunit.client.1.smithi189.stdout:6/3: write - no filename 2024-04-20T21:31:55.563 INFO:tasks.workunit.client.1.smithi189.stdout:6/4: chown . 0 1 2024-04-20T21:31:55.564 INFO:tasks.workunit.client.1.smithi189.stdout:0/6: creat d3/f4 x:0 0 0 2024-04-20T21:31:55.565 INFO:tasks.workunit.client.1.smithi189.stdout:3/4: rename l0 to l1 0 2024-04-20T21:31:55.566 INFO:tasks.workunit.client.0.smithi062.stdout:3/364: creat d0/d1/d3c/d43/d67/f86 x:0 0 0 2024-04-20T21:31:55.567 INFO:tasks.workunit.client.1.smithi189.stdout:4/5: symlink l1 0 2024-04-20T21:31:55.568 INFO:tasks.workunit.client.1.smithi189.stdout:5/2: symlink l0 0 2024-04-20T21:31:55.569 INFO:tasks.workunit.client.0.smithi062.stdout:2/350: chown d7/da/d15/l31 359881 1 2024-04-20T21:31:55.570 INFO:tasks.workunit.client.1.smithi189.stdout:4/6: chown l1 455 1 2024-04-20T21:31:55.570 INFO:tasks.workunit.client.0.smithi062.stdout:6/249: mkdir d0/d52 0 2024-04-20T21:31:55.572 INFO:tasks.workunit.client.0.smithi062.stdout:4/309: rename da/dd/d15/f5d to da/d56/f6c 0 2024-04-20T21:31:55.578 INFO:tasks.workunit.client.0.smithi062.stdout:7/308: link d7/d3e/d33/l60 d7/d3e/d33/d3d/d4d/d54/d4f/l66 0 2024-04-20T21:31:55.580 INFO:tasks.workunit.client.1.smithi189.stdout:7/0: dread - no filename 2024-04-20T21:31:55.580 INFO:tasks.workunit.client.1.smithi189.stdout:7/1: write - no filename 2024-04-20T21:31:55.581 INFO:tasks.workunit.client.0.smithi062.stdout:2/351: rename d7/f17 to d7/da/d15/d2f/d38/d51/f68 0 2024-04-20T21:31:55.583 INFO:tasks.workunit.client.0.smithi062.stdout:3/365: link d0/d1/f82 d0/d1b/f87 0 2024-04-20T21:31:55.589 INFO:tasks.workunit.client.0.smithi062.stdout:5/323: dwrite d8/d10/d32/f60 [4194304,4194304] 0 2024-04-20T21:31:55.591 INFO:tasks.workunit.client.0.smithi062.stdout:6/250: getdents d0/d2c 0 2024-04-20T21:31:55.592 INFO:tasks.workunit.client.1.smithi189.stdout:0/7: dwrite d3/f4 [0,4194304] 0 2024-04-20T21:31:55.594 INFO:tasks.workunit.client.1.smithi189.stdout:0/8: rename d3 to d3/d5 22 2024-04-20T21:31:55.595 INFO:tasks.workunit.client.0.smithi062.stdout:1/372: dwrite dc/d15/d20/d28/f60 [0,4194304] 0 2024-04-20T21:31:55.599 INFO:tasks.workunit.client.0.smithi062.stdout:6/251: read d0/d9/d11/f25 [3303,77625] 0 2024-04-20T21:31:55.600 INFO:tasks.workunit.client.0.smithi062.stdout:7/309: symlink d7/d3e/d33/d3d/d5e/l67 0 2024-04-20T21:31:55.600 INFO:tasks.workunit.client.0.smithi062.stdout:4/310: dwrite da/dd/f69 [4194304,4194304] 0 2024-04-20T21:31:55.619 INFO:tasks.workunit.client.0.smithi062.stdout:0/325: dread d7/f9 [4194304,4194304] 0 2024-04-20T21:31:55.637 INFO:tasks.workunit.client.1.smithi189.stdout:3/5: mkdir d2 0 2024-04-20T21:31:55.638 INFO:tasks.workunit.client.1.smithi189.stdout:5/3: creat f1 x:0 0 0 2024-04-20T21:31:55.639 INFO:tasks.workunit.client.1.smithi189.stdout:4/7: mkdir d2 0 2024-04-20T21:31:55.639 INFO:tasks.workunit.client.1.smithi189.stdout:5/4: read - f1 zero size 2024-04-20T21:31:55.639 INFO:tasks.workunit.client.1.smithi189.stdout:6/5: creat f0 x:0 0 0 2024-04-20T21:31:55.641 INFO:tasks.workunit.client.1.smithi189.stdout:4/8: truncate f0 540324 0 2024-04-20T21:31:55.641 INFO:tasks.workunit.client.1.smithi189.stdout:5/5: write f1 [975510,58108] 0 2024-04-20T21:31:55.642 INFO:tasks.workunit.client.1.smithi189.stdout:8/0: readlink - no filename 2024-04-20T21:31:55.642 INFO:tasks.workunit.client.1.smithi189.stdout:8/1: dread - no filename 2024-04-20T21:31:55.642 INFO:tasks.workunit.client.1.smithi189.stdout:8/2: dwrite - no filename 2024-04-20T21:31:55.642 INFO:tasks.workunit.client.1.smithi189.stdout:8/3: write - no filename 2024-04-20T21:31:55.642 INFO:tasks.workunit.client.1.smithi189.stdout:8/4: fsync - no filename 2024-04-20T21:31:55.643 INFO:tasks.workunit.client.1.smithi189.stdout:8/5: link - no file 2024-04-20T21:31:55.644 INFO:tasks.workunit.client.1.smithi189.stdout:6/6: mknod c1 0 2024-04-20T21:31:55.644 INFO:tasks.workunit.client.1.smithi189.stdout:7/2: mknod c0 0 2024-04-20T21:31:55.645 INFO:tasks.workunit.client.1.smithi189.stdout:5/6: write f1 [1291628,106144] 0 2024-04-20T21:31:55.646 INFO:tasks.workunit.client.1.smithi189.stdout:6/7: write f0 [243635,10027] 0 2024-04-20T21:31:55.647 INFO:tasks.workunit.client.1.smithi189.stdout:9/0: write - no filename 2024-04-20T21:31:55.648 INFO:tasks.workunit.client.1.smithi189.stdout:9/1: write - no filename 2024-04-20T21:31:55.648 INFO:tasks.workunit.client.1.smithi189.stdout:9/2: fsync - no filename 2024-04-20T21:31:55.650 INFO:tasks.workunit.client.1.smithi189.stdout:3/6: mkdir d2/d3 0 2024-04-20T21:31:55.651 INFO:tasks.workunit.client.1.smithi189.stdout:3/7: stat d2 0 2024-04-20T21:31:55.651 INFO:tasks.workunit.client.1.smithi189.stdout:3/8: write - no filename 2024-04-20T21:31:55.651 INFO:tasks.workunit.client.1.smithi189.stdout:3/9: dwrite - no filename 2024-04-20T21:31:55.651 INFO:tasks.workunit.client.1.smithi189.stdout:3/10: truncate - no filename 2024-04-20T21:31:55.652 INFO:tasks.workunit.client.1.smithi189.stdout:7/3: creat f1 x:0 0 0 2024-04-20T21:31:55.653 INFO:tasks.workunit.client.1.smithi189.stdout:4/9: symlink d2/l3 0 2024-04-20T21:31:55.653 INFO:tasks.workunit.client.1.smithi189.stdout:7/4: dread - f1 zero size 2024-04-20T21:31:55.654 INFO:tasks.workunit.client.1.smithi189.stdout:5/7: creat f2 x:0 0 0 2024-04-20T21:31:55.654 INFO:tasks.workunit.client.1.smithi189.stdout:5/8: rmdir - no directory 2024-04-20T21:31:55.655 INFO:tasks.workunit.client.1.smithi189.stdout:7/5: write f1 [220502,35934] 0 2024-04-20T21:31:55.655 INFO:tasks.workunit.client.1.smithi189.stdout:8/6: symlink l0 0 2024-04-20T21:31:55.658 INFO:tasks.workunit.client.1.smithi189.stdout:6/8: link f0 f2 0 2024-04-20T21:31:55.658 INFO:tasks.workunit.client.1.smithi189.stdout:9/3: symlink l0 0 2024-04-20T21:31:55.658 INFO:tasks.workunit.client.1.smithi189.stdout:7/6: write f1 [634406,110911] 0 2024-04-20T21:31:55.660 INFO:tasks.workunit.client.1.smithi189.stdout:8/7: symlink l1 0 2024-04-20T21:31:55.662 INFO:tasks.workunit.client.1.smithi189.stdout:9/4: creat f1 x:0 0 0 2024-04-20T21:31:55.662 INFO:tasks.workunit.client.1.smithi189.stdout:9/5: rmdir - no directory 2024-04-20T21:31:55.663 INFO:tasks.workunit.client.1.smithi189.stdout:5/9: dread f1 [0,4194304] 0 2024-04-20T21:31:55.663 INFO:tasks.workunit.client.1.smithi189.stdout:7/7: rename f1 to f2 0 2024-04-20T21:31:55.664 INFO:tasks.workunit.client.1.smithi189.stdout:9/6: truncate f1 871868 0 2024-04-20T21:31:55.664 INFO:tasks.workunit.client.1.smithi189.stdout:6/9: mknod c3 0 2024-04-20T21:31:55.665 INFO:tasks.workunit.client.1.smithi189.stdout:7/8: stat c0 0 2024-04-20T21:31:55.665 INFO:tasks.workunit.client.1.smithi189.stdout:5/10: read - f2 zero size 2024-04-20T21:31:55.666 INFO:tasks.workunit.client.1.smithi189.stdout:8/8: rename l0 to l2 0 2024-04-20T21:31:55.666 INFO:tasks.workunit.client.1.smithi189.stdout:9/7: write f1 [668861,86658] 0 2024-04-20T21:31:55.666 INFO:tasks.workunit.client.1.smithi189.stdout:5/11: chown f2 0 1 2024-04-20T21:31:55.669 INFO:tasks.workunit.client.1.smithi189.stdout:7/9: write f2 [1651056,58420] 0 2024-04-20T21:31:55.669 INFO:tasks.workunit.client.1.smithi189.stdout:6/10: unlink f0 0 2024-04-20T21:31:55.671 INFO:tasks.workunit.client.1.smithi189.stdout:9/8: mknod c2 0 2024-04-20T21:31:55.672 INFO:tasks.workunit.client.1.smithi189.stdout:9/9: fdatasync f1 0 2024-04-20T21:31:55.674 INFO:tasks.workunit.client.1.smithi189.stdout:6/11: unlink c1 0 2024-04-20T21:31:55.678 INFO:tasks.workunit.client.1.smithi189.stdout:6/12: truncate f2 811826 0 2024-04-20T21:31:55.680 INFO:tasks.workunit.client.1.smithi189.stdout:9/10: dread f1 [0,4194304] 0 2024-04-20T21:31:55.683 INFO:tasks.workunit.client.1.smithi189.stdout:9/11: creat f3 x:0 0 0 2024-04-20T21:31:55.686 INFO:tasks.workunit.client.1.smithi189.stdout:9/12: creat f4 x:0 0 0 2024-04-20T21:31:55.697 INFO:tasks.workunit.client.1.smithi189.stdout:5/12: dwrite f1 [0,4194304] 0 2024-04-20T21:31:55.700 INFO:tasks.workunit.client.1.smithi189.stdout:5/13: mkdir d3 0 2024-04-20T21:31:55.704 INFO:tasks.workunit.client.1.smithi189.stdout:5/14: stat f1 0 2024-04-20T21:31:55.704 INFO:tasks.workunit.client.1.smithi189.stdout:6/13: dwrite f2 [0,4194304] 0 2024-04-20T21:31:55.706 INFO:tasks.workunit.client.1.smithi189.stdout:1/4: fsync f0 0 2024-04-20T21:31:55.708 INFO:tasks.workunit.client.1.smithi189.stdout:1/5: dread - f0 zero size 2024-04-20T21:31:55.709 INFO:tasks.workunit.client.0.smithi062.stdout:9/301: sync 2024-04-20T21:31:55.713 INFO:tasks.workunit.client.1.smithi189.stdout:1/6: link f0 f1 0 2024-04-20T21:31:55.715 INFO:tasks.workunit.client.1.smithi189.stdout:1/7: creat f2 x:0 0 0 2024-04-20T21:31:55.718 INFO:tasks.workunit.client.1.smithi189.stdout:1/8: symlink l3 0 2024-04-20T21:31:55.719 INFO:tasks.workunit.client.1.smithi189.stdout:1/9: write f2 [287604,9686] 0 2024-04-20T21:31:55.723 INFO:tasks.workunit.client.1.smithi189.stdout:1/10: unlink f0 0 2024-04-20T21:31:55.725 INFO:tasks.workunit.client.1.smithi189.stdout:3/11: readlink l1 0 2024-04-20T21:31:55.728 INFO:tasks.workunit.client.1.smithi189.stdout:1/11: rename l3 to l4 0 2024-04-20T21:31:55.730 INFO:tasks.workunit.client.1.smithi189.stdout:3/12: symlink d2/d3/l4 0 2024-04-20T21:31:55.731 INFO:tasks.workunit.client.0.smithi062.stdout:9/302: dwrite d16/d2c/d42/f50 [0,4194304] 0 2024-04-20T21:31:55.731 INFO:tasks.workunit.client.1.smithi189.stdout:1/12: mknod c5 0 2024-04-20T21:31:55.734 INFO:tasks.workunit.client.1.smithi189.stdout:3/13: creat d2/f5 x:0 0 0 2024-04-20T21:31:55.737 INFO:tasks.workunit.client.1.smithi189.stdout:3/14: creat d2/f6 x:0 0 0 2024-04-20T21:31:55.739 INFO:tasks.workunit.client.1.smithi189.stdout:0/9: truncate d3/f4 3664268 0 2024-04-20T21:31:55.739 INFO:tasks.workunit.client.1.smithi189.stdout:3/15: chown d2/d3/l4 30 1 2024-04-20T21:31:55.739 INFO:tasks.workunit.client.1.smithi189.stdout:1/13: write f1 [381059,118240] 0 2024-04-20T21:31:55.741 INFO:tasks.workunit.client.1.smithi189.stdout:3/16: dread - d2/f6 zero size 2024-04-20T21:31:55.741 INFO:tasks.workunit.client.1.smithi189.stdout:1/14: creat f6 x:0 0 0 2024-04-20T21:31:55.743 INFO:tasks.workunit.client.1.smithi189.stdout:1/15: chown f2 126283 1 2024-04-20T21:31:55.743 INFO:tasks.workunit.client.1.smithi189.stdout:3/17: write d2/f6 [900365,933] 0 2024-04-20T21:31:55.748 INFO:tasks.workunit.client.1.smithi189.stdout:0/10: rename f2 to d3/f6 0 2024-04-20T21:31:55.750 INFO:tasks.workunit.client.0.smithi062.stdout:8/342: sync 2024-04-20T21:31:55.757 INFO:tasks.workunit.client.1.smithi189.stdout:3/18: link d2/f5 d2/f7 0 2024-04-20T21:31:55.758 INFO:tasks.workunit.client.1.smithi189.stdout:8/9: stat l2 0 2024-04-20T21:31:55.758 INFO:tasks.workunit.client.1.smithi189.stdout:8/10: dread - no filename 2024-04-20T21:31:55.758 INFO:tasks.workunit.client.1.smithi189.stdout:0/11: chown d3/f6 113509084 1 2024-04-20T21:31:55.759 INFO:tasks.workunit.client.1.smithi189.stdout:7/10: truncate f2 596318 0 2024-04-20T21:31:55.761 INFO:tasks.workunit.client.1.smithi189.stdout:3/19: write d2/f6 [1578542,40006] 0 2024-04-20T21:31:55.761 INFO:tasks.workunit.client.1.smithi189.stdout:9/13: getdents . 0 2024-04-20T21:31:55.763 INFO:tasks.workunit.client.1.smithi189.stdout:6/14: truncate f2 1292505 0 2024-04-20T21:31:55.763 INFO:tasks.workunit.client.1.smithi189.stdout:9/14: truncate f3 529240 0 2024-04-20T21:31:55.764 INFO:tasks.workunit.client.1.smithi189.stdout:9/15: truncate f1 1747426 0 2024-04-20T21:31:55.851 INFO:tasks.workunit.client.1.smithi189.stdout:3/20: chown l1 2 1 2024-04-20T21:31:55.853 INFO:tasks.workunit.client.1.smithi189.stdout:1/16: chown l4 84162 1 2024-04-20T21:31:55.858 INFO:tasks.workunit.client.1.smithi189.stdout:0/12: unlink d3/f4 0 2024-04-20T21:31:55.860 INFO:tasks.workunit.client.1.smithi189.stdout:9/16: link f4 f5 0 2024-04-20T21:31:55.862 INFO:tasks.workunit.client.1.smithi189.stdout:3/21: rmdir d2 39 2024-04-20T21:31:55.864 INFO:tasks.workunit.client.1.smithi189.stdout:3/22: chown l1 139847 1 2024-04-20T21:31:55.866 INFO:tasks.workunit.client.1.smithi189.stdout:0/13: symlink d3/l7 0 2024-04-20T21:31:55.869 INFO:tasks.workunit.client.1.smithi189.stdout:8/11: getdents . 0 2024-04-20T21:31:55.869 INFO:tasks.workunit.client.1.smithi189.stdout:8/12: write - no filename 2024-04-20T21:31:55.873 INFO:tasks.workunit.client.1.smithi189.stdout:8/13: creat f3 x:0 0 0 2024-04-20T21:31:55.874 INFO:tasks.workunit.client.1.smithi189.stdout:0/14: write d3/f6 [80205,78580] 0 2024-04-20T21:31:55.879 INFO:tasks.workunit.client.1.smithi189.stdout:8/14: creat f4 x:0 0 0 2024-04-20T21:31:55.883 INFO:tasks.workunit.client.1.smithi189.stdout:8/15: write f4 [959815,79996] 0 2024-04-20T21:31:55.884 INFO:tasks.workunit.client.1.smithi189.stdout:0/15: dread d3/f6 [0,4194304] 0 2024-04-20T21:31:55.890 INFO:tasks.workunit.client.1.smithi189.stdout:3/23: dread d2/f6 [0,4194304] 0 2024-04-20T21:31:55.890 INFO:tasks.workunit.client.1.smithi189.stdout:0/16: truncate d3/f6 316426 0 2024-04-20T21:31:55.892 INFO:tasks.workunit.client.1.smithi189.stdout:0/17: chown d3 680001 1 2024-04-20T21:31:55.894 INFO:tasks.workunit.client.1.smithi189.stdout:3/24: read d2/f6 [1127589,17048] 0 2024-04-20T21:31:55.895 INFO:tasks.workunit.client.1.smithi189.stdout:0/18: rename d3 to d3/d8 22 2024-04-20T21:31:55.904 INFO:tasks.workunit.client.1.smithi189.stdout:3/25: mkdir d2/d8 0 2024-04-20T21:31:55.904 INFO:tasks.workunit.client.1.smithi189.stdout:7/11: dread f2 [0,4194304] 0 2024-04-20T21:31:55.907 INFO:tasks.workunit.client.1.smithi189.stdout:1/17: mknod c7 0 2024-04-20T21:31:55.910 INFO:tasks.workunit.client.1.smithi189.stdout:0/19: write d3/f6 [282627,101917] 0 2024-04-20T21:31:55.911 INFO:tasks.workunit.client.1.smithi189.stdout:9/17: dwrite f5 [0,4194304] 0 2024-04-20T21:31:55.912 INFO:tasks.workunit.client.1.smithi189.stdout:7/12: dread f2 [0,4194304] 0 2024-04-20T21:31:55.917 INFO:tasks.workunit.client.1.smithi189.stdout:3/26: symlink d2/d3/l9 0 2024-04-20T21:31:55.921 INFO:tasks.workunit.client.1.smithi189.stdout:7/13: mknod c3 0 2024-04-20T21:31:55.921 INFO:tasks.workunit.client.1.smithi189.stdout:0/20: creat d3/f9 x:0 0 0 2024-04-20T21:31:55.922 INFO:tasks.workunit.client.1.smithi189.stdout:9/18: write f4 [5070444,35920] 0 2024-04-20T21:31:55.924 INFO:tasks.workunit.client.1.smithi189.stdout:9/19: rmdir - no directory 2024-04-20T21:31:55.924 INFO:tasks.workunit.client.1.smithi189.stdout:6/15: getdents . 0 2024-04-20T21:31:55.927 INFO:tasks.workunit.client.0.smithi062.stdout:5/324: unlink d8/ff 0 2024-04-20T21:31:55.927 INFO:tasks.workunit.client.1.smithi189.stdout:3/27: rmdir d2/d3 39 2024-04-20T21:31:55.929 INFO:tasks.workunit.client.1.smithi189.stdout:3/28: chown d2 10235 1 2024-04-20T21:31:55.929 INFO:tasks.workunit.client.1.smithi189.stdout:7/14: unlink c3 0 2024-04-20T21:31:55.933 INFO:tasks.workunit.client.0.smithi062.stdout:2/352: creat d7/da/d28/d4b/f69 x:0 0 0 2024-04-20T21:31:55.934 INFO:tasks.workunit.client.0.smithi062.stdout:1/373: fdatasync dc/d15/d17/f1e 0 2024-04-20T21:31:55.935 INFO:tasks.workunit.client.1.smithi189.stdout:0/21: truncate d3/f6 845082 0 2024-04-20T21:31:55.935 INFO:tasks.workunit.client.1.smithi189.stdout:8/16: dwrite f3 [0,4194304] 0 2024-04-20T21:31:55.935 INFO:tasks.workunit.client.1.smithi189.stdout:1/18: dread f1 [0,4194304] 0 2024-04-20T21:31:55.937 INFO:tasks.workunit.client.1.smithi189.stdout:7/15: truncate f2 550910 0 2024-04-20T21:31:55.942 INFO:tasks.workunit.client.0.smithi062.stdout:4/311: mkdir da/dd/d15/d6d 0 2024-04-20T21:31:55.942 INFO:tasks.workunit.client.0.smithi062.stdout:7/310: chown d7/cb 196 1 2024-04-20T21:31:55.942 INFO:tasks.workunit.client.1.smithi189.stdout:0/22: unlink d3/f9 0 2024-04-20T21:31:55.942 INFO:tasks.workunit.client.1.smithi189.stdout:6/16: dread f2 [0,4194304] 0 2024-04-20T21:31:55.942 INFO:tasks.workunit.client.1.smithi189.stdout:0/23: write d3/f6 [476618,130048] 0 2024-04-20T21:31:55.943 INFO:tasks.workunit.client.1.smithi189.stdout:7/16: mknod c4 0 2024-04-20T21:31:55.945 INFO:tasks.workunit.client.1.smithi189.stdout:3/29: dread d2/f6 [0,4194304] 0 2024-04-20T21:31:55.945 INFO:tasks.workunit.client.0.smithi062.stdout:0/326: mkdir d7/d1e/d6e 0 2024-04-20T21:31:55.948 INFO:tasks.workunit.client.0.smithi062.stdout:9/303: read d16/f19 [2400348,8999] 0 2024-04-20T21:31:55.950 INFO:tasks.workunit.client.0.smithi062.stdout:3/366: mkdir d0/d1/d10/d3a/d5e/d65/d88 0 2024-04-20T21:31:55.953 INFO:tasks.workunit.client.1.smithi189.stdout:0/24: rename d3/f6 to d3/fa 0 2024-04-20T21:31:55.953 INFO:tasks.workunit.client.0.smithi062.stdout:5/325: rename d8/d10/d32 to d8/d1e/d2c/d6c 0 2024-04-20T21:31:55.954 INFO:tasks.workunit.client.1.smithi189.stdout:0/25: chown d3 217011 1 2024-04-20T21:31:55.955 INFO:tasks.workunit.client.0.smithi062.stdout:5/326: truncate d8/d1e/f5d 830464 0 2024-04-20T21:31:55.955 INFO:tasks.workunit.client.1.smithi189.stdout:8/17: dread f4 [0,4194304] 0 2024-04-20T21:31:55.956 INFO:tasks.workunit.client.1.smithi189.stdout:0/26: chown d3 1638 1 2024-04-20T21:31:55.957 INFO:tasks.workunit.client.0.smithi062.stdout:2/353: mkdir d7/da/d15/d2f/d38/d6a 0 2024-04-20T21:31:55.959 INFO:tasks.workunit.client.1.smithi189.stdout:8/18: write f3 [151674,129559] 0 2024-04-20T21:31:55.962 INFO:tasks.workunit.client.0.smithi062.stdout:4/312: creat da/dd/df/d27/d3f/f6e x:0 0 0 2024-04-20T21:31:55.962 INFO:tasks.workunit.client.1.smithi189.stdout:0/27: write d3/fa [88317,16120] 0 2024-04-20T21:31:55.963 INFO:tasks.workunit.client.0.smithi062.stdout:7/311: creat d7/d3e/d33/d3d/d4d/d54/d4f/f68 x:0 0 0 2024-04-20T21:31:55.964 INFO:tasks.workunit.client.1.smithi189.stdout:0/28: readlink d3/l7 0 2024-04-20T21:31:55.967 INFO:tasks.workunit.client.1.smithi189.stdout:0/29: truncate d3/fa 1650789 0 2024-04-20T21:31:55.967 INFO:tasks.workunit.client.1.smithi189.stdout:8/19: dread f4 [0,4194304] 0 2024-04-20T21:31:55.969 INFO:tasks.workunit.client.0.smithi062.stdout:9/304: chown d16/d3e/f5f 3235771 1 2024-04-20T21:31:55.977 INFO:tasks.workunit.client.0.smithi062.stdout:5/327: chown d8/d1e/d2c/d6c/d56/d50/f64 135827882 1 2024-04-20T21:31:55.978 INFO:tasks.workunit.client.0.smithi062.stdout:6/252: write d0/d9/d11/f25 [1092932,102453] 0 2024-04-20T21:31:55.979 INFO:tasks.workunit.client.1.smithi189.stdout:8/20: link l2 l5 0 2024-04-20T21:31:55.980 INFO:tasks.workunit.client.0.smithi062.stdout:2/354: mkdir d7/da/d15/d2f/d38/d6b 0 2024-04-20T21:31:55.982 INFO:tasks.workunit.client.1.smithi189.stdout:0/30: getdents d3 0 2024-04-20T21:31:55.983 INFO:tasks.workunit.client.0.smithi062.stdout:8/343: creat d7/d9/d1b/d24/f75 x:0 0 0 2024-04-20T21:31:55.984 INFO:tasks.workunit.client.0.smithi062.stdout:2/355: write d7/da/d15/d2f/f33 [123167,36574] 0 2024-04-20T21:31:55.985 INFO:tasks.workunit.client.1.smithi189.stdout:8/21: creat f6 x:0 0 0 2024-04-20T21:31:55.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:55 smithi189 ceph-mon[21562]: pgmap v12: 65 pgs: 65 active+clean; 654 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 7.4 MiB/s rd, 64 MiB/s wr, 288 op/s 2024-04-20T21:31:55.989 INFO:tasks.workunit.client.0.smithi062.stdout:0/327: dwrite d7/f4c [0,4194304] 0 2024-04-20T21:31:55.989 INFO:tasks.workunit.client.0.smithi062.stdout:4/313: rename da/dd/df/d27/d32/f6a to da/dd/d15/d50/d64/d57/d5b/f6f 0 2024-04-20T21:31:55.991 INFO:tasks.workunit.client.0.smithi062.stdout:3/367: rmdir d0/d1/d10/d3a/d5e 39 2024-04-20T21:31:55.994 INFO:tasks.workunit.client.0.smithi062.stdout:0/328: chown d7/d1c/d15/d17/d40/c52 10831733 1 2024-04-20T21:31:55.994 INFO:tasks.workunit.client.0.smithi062.stdout:9/305: dread - f15 zero size 2024-04-20T21:31:55.995 INFO:tasks.workunit.client.1.smithi189.stdout:0/31: write d3/fa [1187376,66000] 0 2024-04-20T21:31:55.995 INFO:tasks.workunit.client.0.smithi062.stdout:9/306: stat d16/d2c/l30 0 2024-04-20T21:31:55.997 INFO:tasks.workunit.client.0.smithi062.stdout:1/374: link dc/d15/c22 dc/d70/d7f/c80 0 2024-04-20T21:31:56.002 INFO:tasks.workunit.client.1.smithi189.stdout:7/17: write f2 [74108,7790] 0 2024-04-20T21:31:56.003 INFO:tasks.workunit.client.1.smithi189.stdout:1/19: dwrite f1 [0,4194304] 0 2024-04-20T21:31:56.004 INFO:tasks.workunit.client.0.smithi062.stdout:7/312: symlink d7/d3e/l69 0 2024-04-20T21:31:56.004 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:55 smithi062 ceph-mon[16221]: pgmap v12: 65 pgs: 65 active+clean; 654 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 7.4 MiB/s rd, 64 MiB/s wr, 288 op/s 2024-04-20T21:31:56.004 INFO:tasks.workunit.client.1.smithi189.stdout:0/32: chown d3/fa 7324 1 2024-04-20T21:31:56.006 INFO:tasks.workunit.client.0.smithi062.stdout:7/313: write d7/d3e/f45 [119327,21413] 0 2024-04-20T21:31:56.007 INFO:tasks.workunit.client.0.smithi062.stdout:7/314: chown d7/f18 2662 1 2024-04-20T21:31:56.007 INFO:tasks.workunit.client.0.smithi062.stdout:1/375: dread dc/d15/d17/f19 [0,4194304] 0 2024-04-20T21:31:56.008 INFO:tasks.workunit.client.0.smithi062.stdout:1/376: stat fb 0 2024-04-20T21:31:56.015 INFO:tasks.workunit.client.1.smithi189.stdout:6/17: dwrite f2 [0,4194304] 0 2024-04-20T21:31:56.025 INFO:tasks.workunit.client.0.smithi062.stdout:1/377: truncate dc/f14 12597031 0 2024-04-20T21:31:56.028 INFO:tasks.workunit.client.0.smithi062.stdout:7/315: dread d7/d3e/f45 [0,4194304] 0 2024-04-20T21:31:56.028 INFO:tasks.workunit.client.0.smithi062.stdout:5/328: mknod d8/d1e/d2c/d6c/d56/d2b/d4c/c6d 0 2024-04-20T21:31:56.028 INFO:tasks.workunit.client.0.smithi062.stdout:1/378: fsync dc/f6d 0 2024-04-20T21:31:56.028 INFO:tasks.workunit.client.0.smithi062.stdout:6/253: unlink d0/d9/d11/d22/l38 0 2024-04-20T21:31:56.030 INFO:tasks.workunit.client.0.smithi062.stdout:9/307: mknod d16/d2c/d62/c6a 0 2024-04-20T21:31:56.036 INFO:tasks.workunit.client.0.smithi062.stdout:9/308: chown d16/d3d/f43 0 1 2024-04-20T21:31:56.037 INFO:tasks.workunit.client.0.smithi062.stdout:3/368: unlink d0/d1/d3c/d43/c46 0 2024-04-20T21:31:56.038 INFO:tasks.workunit.client.1.smithi189.stdout:7/18: mkdir d5 0 2024-04-20T21:31:56.039 INFO:tasks.workunit.client.1.smithi189.stdout:7/19: chown f2 512277328 1 2024-04-20T21:31:56.039 INFO:tasks.workunit.client.1.smithi189.stdout:1/20: write f6 [648466,119986] 0 2024-04-20T21:31:56.040 INFO:tasks.workunit.client.1.smithi189.stdout:0/33: dread d3/fa [0,4194304] 0 2024-04-20T21:31:56.042 INFO:tasks.workunit.client.1.smithi189.stdout:6/18: creat f4 x:0 0 0 2024-04-20T21:31:56.045 INFO:tasks.workunit.client.1.smithi189.stdout:6/19: write f2 [669498,126484] 0 2024-04-20T21:31:56.045 INFO:tasks.workunit.client.0.smithi062.stdout:1/379: mknod dc/d15/d20/d28/c81 0 2024-04-20T21:31:56.046 INFO:tasks.workunit.client.1.smithi189.stdout:7/20: dread f2 [0,4194304] 0 2024-04-20T21:31:56.046 INFO:tasks.workunit.client.1.smithi189.stdout:6/20: dread - f4 zero size 2024-04-20T21:31:56.046 INFO:tasks.workunit.client.1.smithi189.stdout:8/22: rename l5 to l7 0 2024-04-20T21:31:56.050 INFO:tasks.workunit.client.0.smithi062.stdout:9/309: mknod d16/d3e/c6b 0 2024-04-20T21:31:56.056 INFO:tasks.workunit.client.1.smithi189.stdout:0/34: unlink d3/l7 0 2024-04-20T21:31:56.056 INFO:tasks.workunit.client.1.smithi189.stdout:8/23: symlink l8 0 2024-04-20T21:31:56.057 INFO:tasks.workunit.client.0.smithi062.stdout:8/344: dread f1 [0,4194304] 0 2024-04-20T21:31:56.058 INFO:tasks.workunit.client.1.smithi189.stdout:1/21: mkdir d8 0 2024-04-20T21:31:56.061 INFO:tasks.workunit.client.0.smithi062.stdout:8/345: write d7/d30/f3c [1733570,20477] 0 2024-04-20T21:31:56.061 INFO:tasks.workunit.client.0.smithi062.stdout:3/369: creat d0/d1/d3c/d43/d72/f89 x:0 0 0 2024-04-20T21:31:56.062 INFO:tasks.workunit.client.0.smithi062.stdout:7/316: creat d7/d3e/f6a x:0 0 0 2024-04-20T21:31:56.063 INFO:tasks.workunit.client.1.smithi189.stdout:0/35: creat d3/fb x:0 0 0 2024-04-20T21:31:56.064 INFO:tasks.workunit.client.1.smithi189.stdout:7/21: creat d5/f6 x:0 0 0 2024-04-20T21:31:56.065 INFO:tasks.workunit.client.0.smithi062.stdout:5/329: link d8/d1e/c48 d8/d1e/d2c/d6c/d56/c6e 0 2024-04-20T21:31:56.066 INFO:tasks.workunit.client.0.smithi062.stdout:6/254: getdents d0/d9 0 2024-04-20T21:31:56.066 INFO:tasks.workunit.client.0.smithi062.stdout:1/380: link dc/d15/d20/d28/c81 dc/d15/d17/d6f/c82 0 2024-04-20T21:31:56.067 INFO:tasks.workunit.client.1.smithi189.stdout:8/24: creat f9 x:0 0 0 2024-04-20T21:31:56.070 INFO:tasks.workunit.client.1.smithi189.stdout:6/21: dwrite f4 [0,4194304] 0 2024-04-20T21:31:56.071 INFO:tasks.workunit.client.1.smithi189.stdout:7/22: creat d5/f7 x:0 0 0 2024-04-20T21:31:56.072 INFO:tasks.workunit.client.1.smithi189.stdout:1/22: creat d8/f9 x:0 0 0 2024-04-20T21:31:56.073 INFO:tasks.workunit.client.1.smithi189.stdout:0/36: symlink d3/lc 0 2024-04-20T21:31:56.074 INFO:tasks.workunit.client.1.smithi189.stdout:8/25: mkdir da 0 2024-04-20T21:31:56.075 INFO:tasks.workunit.client.1.smithi189.stdout:6/22: creat f5 x:0 0 0 2024-04-20T21:31:56.077 INFO:tasks.workunit.client.0.smithi062.stdout:1/381: stat dc/d15/d17/f34 0 2024-04-20T21:31:56.079 INFO:tasks.workunit.client.0.smithi062.stdout:7/317: symlink d7/d3e/l6b 0 2024-04-20T21:31:56.080 INFO:tasks.workunit.client.0.smithi062.stdout:2/356: dwrite d7/f12 [0,4194304] 0 2024-04-20T21:31:56.080 INFO:tasks.workunit.client.0.smithi062.stdout:8/346: mkdir d7/d25/d76 0 2024-04-20T21:31:56.081 INFO:tasks.workunit.client.0.smithi062.stdout:4/314: dread da/dd/d15/d50/d64/d3d/f40 [4194304,4194304] 0 2024-04-20T21:31:56.081 INFO:tasks.workunit.client.1.smithi189.stdout:0/37: fdatasync d3/fa 0 2024-04-20T21:31:56.082 INFO:tasks.workunit.client.0.smithi062.stdout:6/255: mknod d0/d4f/c53 0 2024-04-20T21:31:56.085 INFO:tasks.workunit.client.1.smithi189.stdout:6/23: creat f6 x:0 0 0 2024-04-20T21:31:56.085 INFO:tasks.workunit.client.0.smithi062.stdout:9/310: dwrite d16/f3a [0,4194304] 0 2024-04-20T21:31:56.085 INFO:tasks.workunit.client.0.smithi062.stdout:5/330: creat d8/d38/f6f x:0 0 0 2024-04-20T21:31:56.086 INFO:tasks.workunit.client.1.smithi189.stdout:8/26: creat da/fb x:0 0 0 2024-04-20T21:31:56.086 INFO:tasks.workunit.client.1.smithi189.stdout:6/24: write f4 [3337431,21138] 0 2024-04-20T21:31:56.089 INFO:tasks.workunit.client.0.smithi062.stdout:6/256: dread - d0/d1a/d44/f4b zero size 2024-04-20T21:31:56.091 INFO:tasks.workunit.client.0.smithi062.stdout:1/382: mkdir dc/d15/d20/d28/d83 0 2024-04-20T21:31:56.092 INFO:tasks.workunit.client.0.smithi062.stdout:2/357: chown d7/da/d15/d1b/l3a 135676 1 2024-04-20T21:31:56.092 INFO:tasks.workunit.client.0.smithi062.stdout:8/347: stat d7/d9/f18 0 2024-04-20T21:31:56.092 INFO:tasks.workunit.client.1.smithi189.stdout:0/38: rename d3/fa to d3/fd 0 2024-04-20T21:31:56.093 INFO:tasks.workunit.client.0.smithi062.stdout:5/331: readlink d8/d38/l4d 0 2024-04-20T21:31:56.095 INFO:tasks.workunit.client.0.smithi062.stdout:8/348: stat d7/d30/f5d 0 2024-04-20T21:31:56.096 INFO:tasks.workunit.client.1.smithi189.stdout:0/39: truncate d3/fb 434496 0 2024-04-20T21:31:56.096 INFO:tasks.workunit.client.0.smithi062.stdout:5/332: write d8/d10/d3e/f52 [1159989,51113] 0 2024-04-20T21:31:56.098 INFO:tasks.workunit.client.0.smithi062.stdout:2/358: stat d7/da/d15/d2f/d38/d51/f60 0 2024-04-20T21:31:56.099 INFO:tasks.workunit.client.1.smithi189.stdout:6/25: unlink f6 0 2024-04-20T21:31:56.100 INFO:tasks.workunit.client.1.smithi189.stdout:7/23: truncate f2 665739 0 2024-04-20T21:31:56.103 INFO:tasks.workunit.client.1.smithi189.stdout:7/24: read - d5/f7 zero size 2024-04-20T21:31:56.106 INFO:tasks.workunit.client.1.smithi189.stdout:0/40: creat d3/fe x:0 0 0 2024-04-20T21:31:56.107 INFO:tasks.workunit.client.0.smithi062.stdout:5/333: creat d8/d38/f70 x:0 0 0 2024-04-20T21:31:56.108 INFO:tasks.workunit.client.1.smithi189.stdout:6/26: creat f7 x:0 0 0 2024-04-20T21:31:56.108 INFO:tasks.workunit.client.0.smithi062.stdout:9/311: link d16/d3d/c46 d16/d3d/d55/c6c 0 2024-04-20T21:31:56.109 INFO:tasks.workunit.client.1.smithi189.stdout:8/27: link l8 da/lc 0 2024-04-20T21:31:56.109 INFO:tasks.workunit.client.0.smithi062.stdout:3/370: truncate d0/d1b/f2b 7797179 0 2024-04-20T21:31:56.111 INFO:tasks.workunit.client.0.smithi062.stdout:6/257: dread d0/d9/d11/f25 [0,4194304] 0 2024-04-20T21:31:56.111 INFO:tasks.workunit.client.0.smithi062.stdout:5/334: fdatasync d8/d1e/f24 0 2024-04-20T21:31:56.111 INFO:tasks.workunit.client.0.smithi062.stdout:8/349: rename d7/d9/d1b/c6a to d7/d9/d1b/d24/d46/c77 0 2024-04-20T21:31:56.113 INFO:tasks.workunit.client.1.smithi189.stdout:0/41: mkdir d3/df 0 2024-04-20T21:31:56.114 INFO:tasks.workunit.client.1.smithi189.stdout:0/42: rename d3 to d3/d10 22 2024-04-20T21:31:56.115 INFO:tasks.workunit.client.1.smithi189.stdout:8/28: creat da/fd x:0 0 0 2024-04-20T21:31:56.115 INFO:tasks.workunit.client.1.smithi189.stdout:1/23: getdents d8 0 2024-04-20T21:31:56.116 INFO:tasks.workunit.client.0.smithi062.stdout:4/315: write da/f13 [624616,94659] 0 2024-04-20T21:31:56.117 INFO:tasks.workunit.client.0.smithi062.stdout:7/318: dread d7/d3e/d33/f35 [0,4194304] 0 2024-04-20T21:31:56.119 INFO:tasks.workunit.client.1.smithi189.stdout:0/43: write d3/fd [471621,46072] 0 2024-04-20T21:31:56.122 INFO:tasks.workunit.client.0.smithi062.stdout:9/312: rename d16/c22 to d16/d4e/c6d 0 2024-04-20T21:31:56.123 INFO:tasks.workunit.client.1.smithi189.stdout:8/29: mkdir da/de 0 2024-04-20T21:31:56.123 INFO:tasks.workunit.client.0.smithi062.stdout:4/316: chown da/dd/d15/f1e 330565 1 2024-04-20T21:31:56.125 INFO:tasks.workunit.client.1.smithi189.stdout:1/24: creat d8/fa x:0 0 0 2024-04-20T21:31:56.126 INFO:tasks.workunit.client.0.smithi062.stdout:8/350: readlink d7/d9/d1b/d24/l47 0 2024-04-20T21:31:56.130 INFO:tasks.workunit.client.1.smithi189.stdout:0/44: mkdir d3/d11 0 2024-04-20T21:31:56.130 INFO:tasks.workunit.client.1.smithi189.stdout:7/25: getdents d5 0 2024-04-20T21:31:56.132 INFO:tasks.workunit.client.1.smithi189.stdout:1/25: symlink d8/lb 0 2024-04-20T21:31:56.133 INFO:tasks.workunit.client.0.smithi062.stdout:9/313: readlink d16/d2c/l59 0 2024-04-20T21:31:56.133 INFO:tasks.workunit.client.0.smithi062.stdout:3/371: rename d0/d1c/d52/d7b/f7f to d0/d1/d10/d3a/d5e/d65/d88/f8a 0 2024-04-20T21:31:56.134 INFO:tasks.workunit.client.0.smithi062.stdout:1/383: dwrite dc/d15/f42 [0,4194304] 0 2024-04-20T21:31:56.134 INFO:tasks.workunit.client.0.smithi062.stdout:6/258: dread d0/d9/f47 [0,4194304] 0 2024-04-20T21:31:56.139 INFO:tasks.workunit.client.1.smithi189.stdout:1/26: symlink d8/lc 0 2024-04-20T21:31:56.139 INFO:tasks.workunit.client.0.smithi062.stdout:3/372: fdatasync d0/d1/d10/f64 0 2024-04-20T21:31:56.140 INFO:tasks.workunit.client.0.smithi062.stdout:7/319: dread d7/f55 [4194304,4194304] 0 2024-04-20T21:31:56.140 INFO:tasks.workunit.client.0.smithi062.stdout:9/314: mkdir d16/d3d/d55/d6e 0 2024-04-20T21:31:56.140 INFO:tasks.workunit.client.0.smithi062.stdout:1/384: fdatasync dc/d15/f6e 0 2024-04-20T21:31:56.142 INFO:tasks.workunit.client.0.smithi062.stdout:2/359: dwrite d7/da/d28/d4b/f69 [0,4194304] 0 2024-04-20T21:31:56.142 INFO:tasks.workunit.client.1.smithi189.stdout:6/27: getdents . 0 2024-04-20T21:31:56.143 INFO:tasks.workunit.client.0.smithi062.stdout:8/351: dread d7/f57 [0,4194304] 0 2024-04-20T21:31:56.145 INFO:tasks.workunit.client.0.smithi062.stdout:3/373: write d0/d1/d10/f64 [2038069,87420] 0 2024-04-20T21:31:56.147 INFO:tasks.workunit.client.0.smithi062.stdout:4/317: link da/dd/df/f1a da/dd/df/f70 0 2024-04-20T21:31:56.148 INFO:tasks.workunit.client.1.smithi189.stdout:6/28: mkdir d8 0 2024-04-20T21:31:56.149 INFO:tasks.workunit.client.1.smithi189.stdout:0/45: rmdir d3/d11 0 2024-04-20T21:31:56.150 INFO:tasks.workunit.client.0.smithi062.stdout:9/315: rename d16/d2f/f31 to d16/d4e/f6f 0 2024-04-20T21:31:56.151 INFO:tasks.workunit.client.1.smithi189.stdout:7/26: write f2 [745567,104024] 0 2024-04-20T21:31:56.152 INFO:tasks.workunit.client.0.smithi062.stdout:2/360: rmdir d7/da/d15/d2f/d38/d51 39 2024-04-20T21:31:56.152 INFO:tasks.workunit.client.0.smithi062.stdout:4/318: write da/dd/df/f3e [107821,74679] 0 2024-04-20T21:31:56.154 INFO:tasks.workunit.client.1.smithi189.stdout:0/46: stat d3/fb 0 2024-04-20T21:31:56.157 INFO:tasks.workunit.client.1.smithi189.stdout:8/30: dwrite f6 [0,4194304] 0 2024-04-20T21:31:56.159 INFO:tasks.workunit.client.0.smithi062.stdout:7/320: creat d7/f6c x:0 0 0 2024-04-20T21:31:56.159 INFO:tasks.workunit.client.0.smithi062.stdout:7/321: dread - d7/f5f zero size 2024-04-20T21:31:56.161 INFO:tasks.workunit.client.1.smithi189.stdout:7/27: rename c0 to d5/c8 0 2024-04-20T21:31:56.161 INFO:tasks.workunit.client.0.smithi062.stdout:2/361: creat d7/da/d15/d1b/f6c x:0 0 0 2024-04-20T21:31:56.162 INFO:tasks.workunit.client.1.smithi189.stdout:6/29: symlink d8/l9 0 2024-04-20T21:31:56.163 INFO:tasks.workunit.client.0.smithi062.stdout:9/316: rename d16/d2f/f39 to d16/d2c/d42/f70 0 2024-04-20T21:31:56.164 INFO:tasks.workunit.client.0.smithi062.stdout:4/319: mkdir da/d56/d71 0 2024-04-20T21:31:56.165 INFO:tasks.workunit.client.0.smithi062.stdout:3/374: creat d0/d1/f8b x:0 0 0 2024-04-20T21:31:56.167 INFO:tasks.workunit.client.1.smithi189.stdout:8/31: mkdir da/df 0 2024-04-20T21:31:56.167 INFO:tasks.workunit.client.1.smithi189.stdout:7/28: dread - d5/f6 zero size 2024-04-20T21:31:56.170 INFO:tasks.workunit.client.0.smithi062.stdout:3/375: chown d0/d1/d10/d3a/d5e/l84 5 1 2024-04-20T21:31:56.172 INFO:tasks.workunit.client.1.smithi189.stdout:6/30: chown d8/l9 9830 1 2024-04-20T21:31:56.172 INFO:tasks.workunit.client.0.smithi062.stdout:1/385: dwrite dc/f35 [0,4194304] 0 2024-04-20T21:31:56.172 INFO:tasks.workunit.client.0.smithi062.stdout:7/322: rename d7/d26/f34 to d7/d63/f6d 0 2024-04-20T21:31:56.173 INFO:tasks.workunit.client.1.smithi189.stdout:8/32: write f3 [1541346,71587] 0 2024-04-20T21:31:56.173 INFO:tasks.workunit.client.0.smithi062.stdout:3/376: fdatasync d0/d1b/f35 0 2024-04-20T21:31:56.176 INFO:tasks.workunit.client.0.smithi062.stdout:4/320: creat da/d56/f72 x:0 0 0 2024-04-20T21:31:56.177 INFO:tasks.workunit.client.0.smithi062.stdout:3/377: dread - d0/d1b/f75 zero size 2024-04-20T21:31:56.177 INFO:tasks.workunit.client.0.smithi062.stdout:8/352: dread d7/d30/f3c [0,4194304] 0 2024-04-20T21:31:56.177 INFO:tasks.workunit.client.0.smithi062.stdout:7/323: fdatasync d7/d63/f6d 0 2024-04-20T21:31:56.180 INFO:tasks.workunit.client.0.smithi062.stdout:1/386: chown dc/c4a 970961 1 2024-04-20T21:31:56.181 INFO:tasks.workunit.client.0.smithi062.stdout:4/321: chown da/cc 940331590 1 2024-04-20T21:31:56.181 INFO:tasks.workunit.client.1.smithi189.stdout:7/29: creat d5/f9 x:0 0 0 2024-04-20T21:31:56.187 INFO:tasks.workunit.client.0.smithi062.stdout:1/387: mkdir dc/d2d/d84 0 2024-04-20T21:31:56.189 INFO:tasks.workunit.client.1.smithi189.stdout:7/30: write f2 [443009,114053] 0 2024-04-20T21:31:56.189 INFO:tasks.workunit.client.1.smithi189.stdout:8/33: creat da/de/f10 x:0 0 0 2024-04-20T21:31:56.189 INFO:tasks.workunit.client.1.smithi189.stdout:7/31: readlink - no filename 2024-04-20T21:31:56.189 INFO:tasks.workunit.client.1.smithi189.stdout:7/32: readlink - no filename 2024-04-20T21:31:56.191 INFO:tasks.workunit.client.0.smithi062.stdout:8/353: rename d7/d9/d29/f6e to d7/d30/f78 0 2024-04-20T21:31:56.193 INFO:tasks.workunit.client.0.smithi062.stdout:3/378: truncate d0/d1b/f2b 2349449 0 2024-04-20T21:31:56.193 INFO:tasks.workunit.client.0.smithi062.stdout:7/324: mkdir d7/d3e/d6e 0 2024-04-20T21:31:56.195 INFO:tasks.workunit.client.0.smithi062.stdout:1/388: rmdir dc/d70/d7f 39 2024-04-20T21:31:56.195 INFO:tasks.workunit.client.1.smithi189.stdout:8/34: chown f6 352 1 2024-04-20T21:31:56.199 INFO:tasks.workunit.client.0.smithi062.stdout:8/354: creat d7/d25/d3b/f79 x:0 0 0 2024-04-20T21:31:56.200 INFO:tasks.workunit.client.1.smithi189.stdout:7/33: write d5/f6 [178858,103401] 0 2024-04-20T21:31:56.201 INFO:tasks.workunit.client.1.smithi189.stdout:1/27: dwrite d8/f9 [0,4194304] 0 2024-04-20T21:31:56.203 INFO:tasks.workunit.client.0.smithi062.stdout:3/379: symlink d0/d1b/l8c 0 2024-04-20T21:31:56.205 INFO:tasks.workunit.client.0.smithi062.stdout:4/322: link f4 da/dd/f73 0 2024-04-20T21:31:56.206 INFO:tasks.workunit.client.0.smithi062.stdout:1/389: symlink dc/d15/d17/d36/l85 0 2024-04-20T21:31:56.207 INFO:tasks.workunit.client.1.smithi189.stdout:8/35: rmdir da/de 39 2024-04-20T21:31:56.207 INFO:tasks.workunit.client.1.smithi189.stdout:1/28: symlink d8/ld 0 2024-04-20T21:31:56.208 INFO:tasks.workunit.client.1.smithi189.stdout:0/47: dwrite d3/fb [0,4194304] 0 2024-04-20T21:31:56.208 INFO:tasks.workunit.client.0.smithi062.stdout:7/325: symlink d7/d3e/d6e/l6f 0 2024-04-20T21:31:56.210 INFO:tasks.workunit.client.1.smithi189.stdout:7/34: symlink d5/la 0 2024-04-20T21:31:56.214 INFO:tasks.workunit.client.1.smithi189.stdout:7/35: write d5/f6 [549737,109749] 0 2024-04-20T21:31:56.215 INFO:tasks.workunit.client.0.smithi062.stdout:1/390: symlink dc/d2d/l86 0 2024-04-20T21:31:56.217 INFO:tasks.workunit.client.0.smithi062.stdout:9/317: dwrite f15 [0,4194304] 0 2024-04-20T21:31:56.217 INFO:tasks.workunit.client.0.smithi062.stdout:4/323: creat da/d56/f74 x:0 0 0 2024-04-20T21:31:56.221 INFO:tasks.workunit.client.1.smithi189.stdout:6/31: dwrite f7 [0,4194304] 0 2024-04-20T21:31:56.222 INFO:tasks.workunit.client.0.smithi062.stdout:9/318: mknod d16/d3d/c71 0 2024-04-20T21:31:56.227 INFO:tasks.workunit.client.0.smithi062.stdout:9/319: stat d16/d3d/c3f 0 2024-04-20T21:31:56.227 INFO:tasks.workunit.client.0.smithi062.stdout:1/391: rename dc/d15/d17/d36/d3a/l52 to dc/d15/d17/d36/l87 0 2024-04-20T21:31:56.229 INFO:tasks.workunit.client.0.smithi062.stdout:4/324: fdatasync da/dd/f12 0 2024-04-20T21:31:56.233 INFO:tasks.workunit.client.0.smithi062.stdout:3/380: dwrite d0/d1c/d63/d42/f73 [0,4194304] 0 2024-04-20T21:31:56.234 INFO:tasks.workunit.client.0.smithi062.stdout:9/320: readlink d16/l25 0 2024-04-20T21:31:56.234 INFO:tasks.workunit.client.0.smithi062.stdout:1/392: write dc/d15/d17/d36/f5d [683403,41699] 0 2024-04-20T21:31:56.239 INFO:tasks.workunit.client.0.smithi062.stdout:9/321: write d16/f36 [4671005,32916] 0 2024-04-20T21:31:56.242 INFO:tasks.workunit.client.0.smithi062.stdout:7/326: dwrite d7/d63/f6d [0,4194304] 0 2024-04-20T21:31:56.243 INFO:tasks.workunit.client.1.smithi189.stdout:7/36: dread d5/f6 [0,4194304] 0 2024-04-20T21:31:56.244 INFO:tasks.workunit.client.0.smithi062.stdout:3/381: unlink d0/d1/d10/d31/l3d 0 2024-04-20T21:31:56.257 INFO:tasks.workunit.client.0.smithi062.stdout:4/325: rename da/dd/d15/d50/d64/d3d/l44 to da/dd/d15/d4b/l75 0 2024-04-20T21:31:56.260 INFO:tasks.workunit.client.1.smithi189.stdout:7/37: write d5/f6 [1091112,69140] 0 2024-04-20T21:31:56.261 INFO:tasks.workunit.client.1.smithi189.stdout:0/48: dwrite d3/fd [0,4194304] 0 2024-04-20T21:31:56.261 INFO:tasks.workunit.client.1.smithi189.stdout:8/36: dwrite da/fd [0,4194304] 0 2024-04-20T21:31:56.263 INFO:tasks.workunit.client.1.smithi189.stdout:0/49: fsync d3/fe 0 2024-04-20T21:31:56.266 INFO:tasks.workunit.client.0.smithi062.stdout:9/322: creat d16/d3d/d55/d6e/f72 x:0 0 0 2024-04-20T21:31:56.267 INFO:tasks.workunit.client.1.smithi189.stdout:8/37: write da/fb [506736,48912] 0 2024-04-20T21:31:56.268 INFO:tasks.workunit.client.0.smithi062.stdout:4/326: fsync da/dd/df/d27/f36 0 2024-04-20T21:31:56.269 INFO:tasks.workunit.client.1.smithi189.stdout:0/50: mknod d3/c12 0 2024-04-20T21:31:56.272 INFO:tasks.workunit.client.1.smithi189.stdout:6/32: dwrite f2 [0,4194304] 0 2024-04-20T21:31:56.273 INFO:tasks.workunit.client.0.smithi062.stdout:9/323: unlink d16/l24 0 2024-04-20T21:31:56.273 INFO:tasks.workunit.client.1.smithi189.stdout:8/38: write f9 [776809,35689] 0 2024-04-20T21:31:56.274 INFO:tasks.workunit.client.1.smithi189.stdout:7/38: link d5/f6 d5/fb 0 2024-04-20T21:31:56.277 INFO:tasks.workunit.client.1.smithi189.stdout:8/39: write f3 [4255103,38747] 0 2024-04-20T21:31:56.278 INFO:tasks.workunit.client.1.smithi189.stdout:0/51: symlink d3/df/l13 0 2024-04-20T21:31:56.278 INFO:tasks.workunit.client.1.smithi189.stdout:6/33: mkdir d8/da 0 2024-04-20T21:31:56.280 INFO:tasks.workunit.client.0.smithi062.stdout:4/327: fdatasync da/f16 0 2024-04-20T21:31:56.281 INFO:tasks.workunit.client.1.smithi189.stdout:0/52: fsync d3/fb 0 2024-04-20T21:31:56.281 INFO:tasks.workunit.client.0.smithi062.stdout:9/324: creat d16/d3d/d55/f73 x:0 0 0 2024-04-20T21:31:56.284 INFO:tasks.workunit.client.0.smithi062.stdout:9/325: read d16/d2c/d42/f50 [683491,111194] 0 2024-04-20T21:31:56.285 INFO:tasks.workunit.client.1.smithi189.stdout:8/40: dread - da/de/f10 zero size 2024-04-20T21:31:56.286 INFO:tasks.workunit.client.0.smithi062.stdout:8/355: write d7/d9/d1b/d24/d46/f64 [1067841,74144] 0 2024-04-20T21:31:56.287 INFO:tasks.workunit.client.1.smithi189.stdout:7/39: rename c4 to d5/cc 0 2024-04-20T21:31:56.287 INFO:tasks.workunit.client.1.smithi189.stdout:0/53: write d3/fd [4163409,8193] 0 2024-04-20T21:31:56.291 INFO:tasks.workunit.client.1.smithi189.stdout:6/34: creat d8/da/fb x:0 0 0 2024-04-20T21:31:56.291 INFO:tasks.workunit.client.1.smithi189.stdout:0/54: read d3/fb [1677576,100423] 0 2024-04-20T21:31:56.291 INFO:tasks.workunit.client.1.smithi189.stdout:8/41: write da/de/f10 [377781,29103] 0 2024-04-20T21:31:56.293 INFO:tasks.workunit.client.0.smithi062.stdout:9/326: write d16/d3d/d55/d6e/f72 [524830,117363] 0 2024-04-20T21:31:56.294 INFO:tasks.workunit.client.1.smithi189.stdout:7/40: chown d5/f7 174409897 1 2024-04-20T21:31:56.295 INFO:tasks.workunit.client.0.smithi062.stdout:8/356: chown d7/d30/f3c 296 1 2024-04-20T21:31:56.296 INFO:tasks.workunit.client.1.smithi189.stdout:6/35: symlink d8/da/lc 0 2024-04-20T21:31:56.296 INFO:tasks.workunit.client.1.smithi189.stdout:0/55: mkdir d3/df/d14 0 2024-04-20T21:31:56.300 INFO:tasks.workunit.client.1.smithi189.stdout:0/56: dread - d3/fe zero size 2024-04-20T21:31:56.300 INFO:tasks.workunit.client.1.smithi189.stdout:7/41: rename d5/fb to d5/fd 0 2024-04-20T21:31:56.301 INFO:tasks.workunit.client.1.smithi189.stdout:8/42: creat da/df/f11 x:0 0 0 2024-04-20T21:31:56.304 INFO:tasks.workunit.client.1.smithi189.stdout:0/57: truncate d3/fe 283195 0 2024-04-20T21:31:56.305 INFO:tasks.workunit.client.0.smithi062.stdout:8/357: mknod d7/d9/d29/c7a 0 2024-04-20T21:31:56.305 INFO:tasks.workunit.client.1.smithi189.stdout:0/58: chown d3/fd 3119 1 2024-04-20T21:31:56.306 INFO:tasks.workunit.client.0.smithi062.stdout:1/393: dwrite dc/d15/d20/f3c [0,4194304] 0 2024-04-20T21:31:56.306 INFO:tasks.workunit.client.1.smithi189.stdout:7/42: creat d5/fe x:0 0 0 2024-04-20T21:31:56.307 INFO:tasks.workunit.client.1.smithi189.stdout:7/43: dread - d5/fe zero size 2024-04-20T21:31:56.310 INFO:tasks.workunit.client.1.smithi189.stdout:7/44: chown d5/f7 11681394 1 2024-04-20T21:31:56.311 INFO:tasks.workunit.client.1.smithi189.stdout:0/59: creat d3/df/f15 x:0 0 0 2024-04-20T21:31:56.312 INFO:tasks.workunit.client.1.smithi189.stdout:7/45: creat d5/ff x:0 0 0 2024-04-20T21:31:56.316 INFO:tasks.workunit.client.1.smithi189.stdout:7/46: write d5/ff [395063,80734] 0 2024-04-20T21:31:56.323 INFO:tasks.workunit.client.0.smithi062.stdout:1/394: creat dc/d2d/d7c/f88 x:0 0 0 2024-04-20T21:31:56.324 INFO:tasks.workunit.client.0.smithi062.stdout:3/382: dread d0/f8 [0,4194304] 0 2024-04-20T21:31:56.325 INFO:tasks.workunit.client.0.smithi062.stdout:4/328: link da/dd/d15/d4b/l75 da/dd/d15/l76 0 2024-04-20T21:31:56.326 INFO:tasks.workunit.client.1.smithi189.stdout:6/36: rmdir d8/da 39 2024-04-20T21:31:56.326 INFO:tasks.workunit.client.0.smithi062.stdout:3/383: fdatasync d0/d1/d10/f79 0 2024-04-20T21:31:56.327 INFO:tasks.workunit.client.0.smithi062.stdout:4/329: stat da/dd/df/d27/d32 0 2024-04-20T21:31:56.330 INFO:tasks.workunit.client.0.smithi062.stdout:1/395: write dc/d15/d20/d28/f4e [4664556,40037] 0 2024-04-20T21:31:56.333 INFO:tasks.workunit.client.0.smithi062.stdout:9/327: getdents d16/d2c/d62 0 2024-04-20T21:31:56.333 INFO:tasks.workunit.client.1.smithi189.stdout:8/43: dwrite da/df/f11 [0,4194304] 0 2024-04-20T21:31:56.333 INFO:tasks.workunit.client.1.smithi189.stdout:8/44: stat da/fb 0 2024-04-20T21:31:56.335 INFO:tasks.workunit.client.1.smithi189.stdout:8/45: chown da/df/f11 464 1 2024-04-20T21:31:56.336 INFO:tasks.workunit.client.0.smithi062.stdout:4/330: creat da/d56/f77 x:0 0 0 2024-04-20T21:31:56.337 INFO:tasks.workunit.client.1.smithi189.stdout:0/60: truncate d3/fd 3185345 0 2024-04-20T21:31:56.337 INFO:tasks.workunit.client.1.smithi189.stdout:8/46: mknod da/df/c12 0 2024-04-20T21:31:56.342 INFO:tasks.workunit.client.0.smithi062.stdout:9/328: fsync d16/f2e 0 2024-04-20T21:31:56.344 INFO:tasks.workunit.client.0.smithi062.stdout:8/358: getdents d7/d25/d3b/d58 0 2024-04-20T21:31:56.348 INFO:tasks.workunit.client.0.smithi062.stdout:9/329: creat d16/d2c/f74 x:0 0 0 2024-04-20T21:31:56.350 INFO:tasks.workunit.client.0.smithi062.stdout:4/331: rename da/d56/f77 to da/dd/f78 0 2024-04-20T21:31:56.351 INFO:tasks.workunit.client.0.smithi062.stdout:8/359: mknod d7/c7b 0 2024-04-20T21:31:56.352 INFO:tasks.workunit.client.0.smithi062.stdout:9/330: truncate d16/d2c/d42/f48 662826 0 2024-04-20T21:31:56.352 INFO:tasks.workunit.client.0.smithi062.stdout:1/396: dwrite dc/f35 [0,4194304] 0 2024-04-20T21:31:56.353 INFO:tasks.workunit.client.0.smithi062.stdout:8/360: stat d7/d9/f3a 0 2024-04-20T21:31:56.354 INFO:tasks.workunit.client.0.smithi062.stdout:1/397: dread - dc/d15/d20/f3e zero size 2024-04-20T21:31:56.354 INFO:tasks.workunit.client.0.smithi062.stdout:9/331: truncate d16/d3e/f64 657670 0 2024-04-20T21:31:56.354 INFO:tasks.workunit.client.0.smithi062.stdout:4/332: unlink da/d56/f6c 0 2024-04-20T21:31:56.356 INFO:tasks.workunit.client.1.smithi189.stdout:6/37: dwrite f7 [4194304,4194304] 0 2024-04-20T21:31:56.357 INFO:tasks.workunit.client.0.smithi062.stdout:8/361: truncate d7/d9/d1b/f3f 4834835 0 2024-04-20T21:31:56.358 INFO:tasks.workunit.client.1.smithi189.stdout:8/47: dread f6 [0,4194304] 0 2024-04-20T21:31:56.358 INFO:tasks.workunit.client.0.smithi062.stdout:9/332: read d16/f19 [1994941,67005] 0 2024-04-20T21:31:56.361 INFO:tasks.workunit.client.0.smithi062.stdout:1/398: creat dc/d27/d32/f89 x:0 0 0 2024-04-20T21:31:56.362 INFO:tasks.workunit.client.0.smithi062.stdout:8/362: read d7/d9/d1b/f1e [1308702,15757] 0 2024-04-20T21:31:56.362 INFO:tasks.workunit.client.0.smithi062.stdout:1/399: dread - dc/d27/d32/f89 zero size 2024-04-20T21:31:56.363 INFO:tasks.workunit.client.0.smithi062.stdout:9/333: mkdir d16/d2c/d5e/d75 0 2024-04-20T21:31:56.364 INFO:tasks.workunit.client.1.smithi189.stdout:8/48: chown da/lc 1516 1 2024-04-20T21:31:56.365 INFO:tasks.workunit.client.0.smithi062.stdout:9/334: fdatasync d16/d3d/d55/f73 0 2024-04-20T21:31:56.365 INFO:tasks.workunit.client.1.smithi189.stdout:8/49: stat da/df/f11 0 2024-04-20T21:31:56.366 INFO:tasks.workunit.client.0.smithi062.stdout:8/363: symlink d7/d25/d4f/l7c 0 2024-04-20T21:31:56.367 INFO:tasks.workunit.client.1.smithi189.stdout:0/61: dwrite d3/fb [0,4194304] 0 2024-04-20T21:31:56.368 INFO:tasks.workunit.client.0.smithi062.stdout:1/400: fdatasync dc/ff 0 2024-04-20T21:31:56.372 INFO:tasks.workunit.client.1.smithi189.stdout:0/62: stat d3/df/f15 0 2024-04-20T21:31:56.374 INFO:tasks.workunit.client.0.smithi062.stdout:4/333: dwrite da/dd/df/d27/f36 [0,4194304] 0 2024-04-20T21:31:56.375 INFO:tasks.workunit.client.1.smithi189.stdout:0/63: chown d3/df/l13 1839 1 2024-04-20T21:31:56.378 INFO:tasks.workunit.client.0.smithi062.stdout:9/335: mkdir d16/d2c/d5e/d75/d76 0 2024-04-20T21:31:56.378 INFO:tasks.workunit.client.0.smithi062.stdout:8/364: write d7/d9/f18 [2678867,33114] 0 2024-04-20T21:31:56.380 INFO:tasks.workunit.client.0.smithi062.stdout:4/334: creat da/d56/d71/f79 x:0 0 0 2024-04-20T21:31:56.381 INFO:tasks.workunit.client.0.smithi062.stdout:9/336: mkdir d16/d3e/d77 0 2024-04-20T21:31:56.385 INFO:tasks.workunit.client.0.smithi062.stdout:8/365: unlink d7/d25/d4f/l7c 0 2024-04-20T21:31:56.393 INFO:tasks.workunit.client.0.smithi062.stdout:4/335: mkdir da/dd/df/d27/d7a 0 2024-04-20T21:31:56.397 INFO:tasks.workunit.client.0.smithi062.stdout:8/366: creat d7/d25/f7d x:0 0 0 2024-04-20T21:31:56.397 INFO:tasks.workunit.client.0.smithi062.stdout:1/401: dwrite f9 [0,4194304] 0 2024-04-20T21:31:56.401 INFO:tasks.workunit.client.0.smithi062.stdout:8/367: read d7/d25/d3b/d58/f63 [64578,97427] 0 2024-04-20T21:31:56.402 INFO:tasks.workunit.client.0.smithi062.stdout:1/402: dread - dc/d15/d17/d6f/f7d zero size 2024-04-20T21:31:56.402 INFO:tasks.workunit.client.0.smithi062.stdout:4/336: rename da/dd/df/f1a to da/f7b 0 2024-04-20T21:31:56.402 INFO:tasks.workunit.client.0.smithi062.stdout:9/337: rmdir d16 39 2024-04-20T21:31:56.403 INFO:tasks.workunit.client.0.smithi062.stdout:8/368: write d7/d9/d1b/d24/f50 [909711,39407] 0 2024-04-20T21:31:56.409 INFO:tasks.workunit.client.0.smithi062.stdout:1/403: rename c7 to dc/c8a 0 2024-04-20T21:31:56.410 INFO:tasks.workunit.client.0.smithi062.stdout:9/338: creat d16/d2f/d5b/f78 x:0 0 0 2024-04-20T21:31:56.411 INFO:tasks.workunit.client.0.smithi062.stdout:8/369: write d7/d9/d1b/d24/f48 [1493580,81156] 0 2024-04-20T21:31:56.412 INFO:tasks.workunit.client.0.smithi062.stdout:1/404: readlink dc/d15/d20/l3b 0 2024-04-20T21:31:56.415 INFO:tasks.workunit.client.0.smithi062.stdout:9/339: mknod d16/d2c/d5e/d75/c79 0 2024-04-20T21:31:56.422 INFO:tasks.workunit.client.0.smithi062.stdout:4/337: dwrite da/dd/d15/d50/d64/d57/f63 [0,4194304] 0 2024-04-20T21:31:56.427 INFO:tasks.workunit.client.1.smithi189.stdout:6/38: mkdir d8/da/dd 0 2024-04-20T21:31:56.428 INFO:tasks.workunit.client.0.smithi062.stdout:8/370: dwrite d7/d9/d29/f32 [0,4194304] 0 2024-04-20T21:31:56.428 INFO:tasks.workunit.client.1.smithi189.stdout:8/50: mkdir da/df/d13 0 2024-04-20T21:31:56.430 INFO:tasks.workunit.client.1.smithi189.stdout:0/64: chown d3/fd 84 1 2024-04-20T21:31:56.431 INFO:tasks.workunit.client.1.smithi189.stdout:6/39: unlink d8/l9 0 2024-04-20T21:31:56.433 INFO:tasks.workunit.client.1.smithi189.stdout:8/51: creat da/df/d13/f14 x:0 0 0 2024-04-20T21:31:56.435 INFO:tasks.workunit.client.0.smithi062.stdout:4/338: rename da/dd/df/d27/d32/l52 to da/dd/df/d27/d7a/l7c 0 2024-04-20T21:31:56.435 INFO:tasks.workunit.client.1.smithi189.stdout:8/52: fdatasync da/df/f11 0 2024-04-20T21:31:56.437 INFO:tasks.workunit.client.1.smithi189.stdout:8/53: chown f4 54010 1 2024-04-20T21:31:56.438 INFO:tasks.workunit.client.1.smithi189.stdout:6/40: creat d8/da/dd/fe x:0 0 0 2024-04-20T21:31:56.439 INFO:tasks.workunit.client.0.smithi062.stdout:8/371: rename d7/f38 to d7/d25/d4f/f7e 0 2024-04-20T21:31:56.441 INFO:tasks.workunit.client.1.smithi189.stdout:8/54: mknod da/c15 0 2024-04-20T21:31:56.442 INFO:tasks.workunit.client.0.smithi062.stdout:8/372: mknod d7/d30/c7f 0 2024-04-20T21:31:56.444 INFO:tasks.workunit.client.0.smithi062.stdout:8/373: stat d7/d9/d1b/d24/d46/f64 0 2024-04-20T21:31:56.444 INFO:tasks.workunit.client.0.smithi062.stdout:9/340: dread d16/f3b [0,4194304] 0 2024-04-20T21:31:56.446 INFO:tasks.workunit.client.1.smithi189.stdout:8/55: rename f6 to da/de/f16 0 2024-04-20T21:31:56.448 INFO:tasks.workunit.client.1.smithi189.stdout:0/65: write d3/fd [2061416,3406] 0 2024-04-20T21:31:56.449 INFO:tasks.workunit.client.1.smithi189.stdout:8/56: creat da/df/f17 x:0 0 0 2024-04-20T21:31:56.449 INFO:tasks.workunit.client.1.smithi189.stdout:0/66: dread - d3/df/f15 zero size 2024-04-20T21:31:56.451 INFO:tasks.workunit.client.1.smithi189.stdout:0/67: write d3/fe [788032,109163] 0 2024-04-20T21:31:56.451 INFO:tasks.workunit.client.1.smithi189.stdout:8/57: creat da/df/d13/f18 x:0 0 0 2024-04-20T21:31:56.452 INFO:tasks.workunit.client.0.smithi062.stdout:4/339: dread da/dd/f69 [4194304,4194304] 0 2024-04-20T21:31:56.453 INFO:tasks.workunit.client.1.smithi189.stdout:0/68: write d3/fe [323161,3560] 0 2024-04-20T21:31:56.454 INFO:tasks.workunit.client.1.smithi189.stdout:8/58: creat da/df/f19 x:0 0 0 2024-04-20T21:31:56.455 INFO:tasks.workunit.client.0.smithi062.stdout:4/340: dread - da/d56/d71/f79 zero size 2024-04-20T21:31:56.455 INFO:tasks.workunit.client.1.smithi189.stdout:0/69: mkdir d3/d16 0 2024-04-20T21:31:56.456 INFO:tasks.workunit.client.0.smithi062.stdout:8/374: dread d7/d9/ff [0,4194304] 0 2024-04-20T21:31:56.457 INFO:tasks.workunit.client.0.smithi062.stdout:4/341: write da/dd/d15/f20 [3233012,85639] 0 2024-04-20T21:31:56.457 INFO:tasks.workunit.client.1.smithi189.stdout:8/59: symlink da/df/d13/l1a 0 2024-04-20T21:31:56.458 INFO:tasks.workunit.client.0.smithi062.stdout:8/375: stat d7/d25/d3b/d58 0 2024-04-20T21:31:56.459 INFO:tasks.workunit.client.1.smithi189.stdout:0/70: mknod d3/d16/c17 0 2024-04-20T21:31:56.461 INFO:tasks.workunit.client.1.smithi189.stdout:0/71: write d3/df/f15 [560393,25500] 0 2024-04-20T21:31:56.461 INFO:tasks.workunit.client.1.smithi189.stdout:8/60: creat da/df/f1b x:0 0 0 2024-04-20T21:31:56.462 INFO:tasks.workunit.client.1.smithi189.stdout:8/61: fdatasync da/de/f10 0 2024-04-20T21:31:56.464 INFO:tasks.workunit.client.1.smithi189.stdout:8/62: symlink da/df/d13/l1c 0 2024-04-20T21:31:56.465 INFO:tasks.workunit.client.1.smithi189.stdout:0/72: mknod d3/df/d14/c18 0 2024-04-20T21:31:56.466 INFO:tasks.workunit.client.1.smithi189.stdout:0/73: fsync d3/df/f15 0 2024-04-20T21:31:56.467 INFO:tasks.workunit.client.0.smithi062.stdout:4/342: rename da/dd/d15/d50/d64/d57/f63 to da/dd/df/d27/d7a/f7d 0 2024-04-20T21:31:56.468 INFO:tasks.workunit.client.1.smithi189.stdout:0/74: write d3/fb [4060533,102725] 0 2024-04-20T21:31:56.469 INFO:tasks.workunit.client.0.smithi062.stdout:4/343: read da/dd/f11 [1852411,42851] 0 2024-04-20T21:31:56.473 INFO:tasks.workunit.client.1.smithi189.stdout:0/75: rename c1 to d3/d16/c19 0 2024-04-20T21:31:56.475 INFO:tasks.workunit.client.0.smithi062.stdout:4/344: creat da/dd/d15/d50/d64/d3d/f7e x:0 0 0 2024-04-20T21:31:56.476 INFO:tasks.workunit.client.0.smithi062.stdout:4/345: chown da/dd/d15/d50/d64/d3d 45996172 1 2024-04-20T21:31:56.477 INFO:tasks.workunit.client.0.smithi062.stdout:8/376: dwrite d7/d30/f3c [0,4194304] 0 2024-04-20T21:31:56.482 INFO:tasks.workunit.client.0.smithi062.stdout:4/346: mknod da/dd/d15/d50/d64/d3d/c7f 0 2024-04-20T21:31:56.482 INFO:tasks.workunit.client.0.smithi062.stdout:8/377: symlink d7/d25/d5c/l80 0 2024-04-20T21:31:56.485 INFO:tasks.workunit.client.0.smithi062.stdout:4/347: write da/dd/d3c/f53 [742024,51056] 0 2024-04-20T21:31:56.485 INFO:tasks.workunit.client.0.smithi062.stdout:8/378: mknod d7/d9/d1b/d24/d46/c81 0 2024-04-20T21:31:56.490 INFO:tasks.workunit.client.1.smithi189.stdout:8/63: dwrite da/df/f1b [0,4194304] 0 2024-04-20T21:31:56.492 INFO:tasks.workunit.client.0.smithi062.stdout:4/348: getdents da/dd/d15/d50/d64 0 2024-04-20T21:31:56.492 INFO:tasks.workunit.client.1.smithi189.stdout:8/64: chown da/df/d13/l1c 301842446 1 2024-04-20T21:31:56.495 INFO:tasks.workunit.client.0.smithi062.stdout:4/349: dread da/dd/d15/d50/d64/d57/d5b/f6f [0,4194304] 0 2024-04-20T21:31:56.500 INFO:tasks.workunit.client.0.smithi062.stdout:4/350: creat da/dd/d15/d50/d64/d57/f80 x:0 0 0 2024-04-20T21:31:56.502 INFO:tasks.workunit.client.1.smithi189.stdout:8/65: dread da/de/f10 [0,4194304] 0 2024-04-20T21:31:56.502 INFO:tasks.workunit.client.0.smithi062.stdout:4/351: truncate da/dd/f35 5227409 0 2024-04-20T21:31:56.503 INFO:tasks.workunit.client.1.smithi189.stdout:0/76: dwrite d3/fe [0,4194304] 0 2024-04-20T21:31:56.504 INFO:tasks.workunit.client.1.smithi189.stdout:0/77: write d3/df/f15 [825136,40999] 0 2024-04-20T21:31:56.505 INFO:tasks.workunit.client.1.smithi189.stdout:8/66: symlink da/de/l1d 0 2024-04-20T21:31:56.506 INFO:tasks.workunit.client.0.smithi062.stdout:4/352: creat da/dd/df/d27/d32/f81 x:0 0 0 2024-04-20T21:31:56.517 INFO:tasks.workunit.client.0.smithi062.stdout:4/353: write da/dd/f12 [758300,22098] 0 2024-04-20T21:31:56.518 INFO:tasks.workunit.client.1.smithi189.stdout:8/67: truncate f9 770174 0 2024-04-20T21:31:56.520 INFO:tasks.workunit.client.0.smithi062.stdout:4/354: mkdir da/dd/df/d27/d3f/d82 0 2024-04-20T21:31:56.522 INFO:tasks.workunit.client.1.smithi189.stdout:8/68: mkdir da/de/d1e 0 2024-04-20T21:31:56.522 INFO:tasks.workunit.client.1.smithi189.stdout:8/69: stat f4 0 2024-04-20T21:31:56.523 INFO:tasks.workunit.client.0.smithi062.stdout:4/355: dread da/dd/d15/d50/d64/d57/d5b/f6f [0,4194304] 0 2024-04-20T21:31:56.524 INFO:tasks.workunit.client.0.smithi062.stdout:4/356: chown da/f13 1 1 2024-04-20T21:31:56.524 INFO:tasks.workunit.client.1.smithi189.stdout:8/70: mkdir da/d1f 0 2024-04-20T21:31:56.526 INFO:tasks.workunit.client.0.smithi062.stdout:4/357: truncate da/d56/f72 1031015 0 2024-04-20T21:31:56.526 INFO:tasks.workunit.client.1.smithi189.stdout:8/71: truncate da/df/f1b 4238707 0 2024-04-20T21:31:56.526 INFO:tasks.workunit.client.1.smithi189.stdout:0/78: fsync d3/fe 0 2024-04-20T21:31:56.528 INFO:tasks.workunit.client.0.smithi062.stdout:4/358: creat da/f83 x:0 0 0 2024-04-20T21:31:56.529 INFO:tasks.workunit.client.1.smithi189.stdout:0/79: rename d3/d16/c17 to d3/d16/c1a 0 2024-04-20T21:31:56.530 INFO:tasks.workunit.client.0.smithi062.stdout:4/359: mknod da/c84 0 2024-04-20T21:31:56.537 INFO:tasks.workunit.client.1.smithi189.stdout:8/72: dread f3 [0,4194304] 0 2024-04-20T21:31:56.538 INFO:tasks.workunit.client.1.smithi189.stdout:0/80: dread d3/fd [0,4194304] 0 2024-04-20T21:31:56.538 INFO:tasks.workunit.client.1.smithi189.stdout:8/73: write da/fb [1494896,90692] 0 2024-04-20T21:31:56.541 INFO:tasks.workunit.client.1.smithi189.stdout:8/74: mkdir da/df/d20 0 2024-04-20T21:31:56.544 INFO:tasks.workunit.client.1.smithi189.stdout:0/81: getdents d3/df/d14 0 2024-04-20T21:31:56.545 INFO:tasks.workunit.client.0.smithi062.stdout:4/360: dwrite da/dd/f1c [0,4194304] 0 2024-04-20T21:31:56.546 INFO:tasks.workunit.client.1.smithi189.stdout:0/82: creat d3/df/f1b x:0 0 0 2024-04-20T21:31:56.548 INFO:tasks.workunit.client.0.smithi062.stdout:4/361: readlink da/dd/df/d27/d7a/l7c 0 2024-04-20T21:31:56.549 INFO:tasks.workunit.client.1.smithi189.stdout:0/83: truncate d3/df/f1b 521102 0 2024-04-20T21:31:56.551 INFO:tasks.workunit.client.0.smithi062.stdout:4/362: symlink da/d56/l85 0 2024-04-20T21:31:56.557 INFO:tasks.workunit.client.0.smithi062.stdout:4/363: link f3 da/dd/d15/d50/f86 0 2024-04-20T21:31:56.558 INFO:tasks.workunit.client.0.smithi062.stdout:4/364: chown da/dd/d15/d50/d64/d3d/f5e 18186 1 2024-04-20T21:31:56.563 INFO:tasks.workunit.client.1.smithi189.stdout:8/75: dwrite da/df/f19 [0,4194304] 0 2024-04-20T21:31:56.566 INFO:tasks.workunit.client.1.smithi189.stdout:8/76: dread da/de/f10 [0,4194304] 0 2024-04-20T21:31:56.568 INFO:tasks.workunit.client.1.smithi189.stdout:0/84: dread d3/fb [0,4194304] 0 2024-04-20T21:31:56.574 INFO:tasks.workunit.client.0.smithi062.stdout:4/365: dread da/fb [0,4194304] 0 2024-04-20T21:31:56.578 INFO:tasks.workunit.client.1.smithi189.stdout:8/77: truncate da/de/f16 3259989 0 2024-04-20T21:31:56.580 INFO:tasks.workunit.client.1.smithi189.stdout:8/78: chown l7 37 1 2024-04-20T21:31:56.585 INFO:tasks.workunit.client.1.smithi189.stdout:8/79: mkdir da/de/d1e/d21 0 2024-04-20T21:31:56.586 INFO:tasks.workunit.client.0.smithi062.stdout:0/329: sync 2024-04-20T21:31:56.590 INFO:tasks.workunit.client.1.smithi189.stdout:0/85: dwrite d3/df/f1b [0,4194304] 0 2024-04-20T21:31:56.595 INFO:tasks.workunit.client.1.smithi189.stdout:8/80: getdents da/de/d1e/d21 0 2024-04-20T21:31:56.595 INFO:tasks.workunit.client.1.smithi189.stdout:0/86: write d3/fb [3645630,37870] 0 2024-04-20T21:31:56.598 INFO:tasks.workunit.client.0.smithi062.stdout:4/366: dread f3 [0,4194304] 0 2024-04-20T21:31:56.599 INFO:tasks.workunit.client.1.smithi189.stdout:0/87: mkdir d3/d1c 0 2024-04-20T21:31:56.600 INFO:tasks.workunit.client.1.smithi189.stdout:8/81: mkdir da/df/d22 0 2024-04-20T21:31:56.600 INFO:tasks.workunit.client.1.smithi189.stdout:0/88: stat d3/df/f15 0 2024-04-20T21:31:56.603 INFO:tasks.workunit.client.1.smithi189.stdout:0/89: creat d3/df/d14/f1d x:0 0 0 2024-04-20T21:31:56.604 INFO:tasks.workunit.client.0.smithi062.stdout:0/330: dwrite d7/d1c/f11 [0,4194304] 0 2024-04-20T21:31:56.607 INFO:tasks.workunit.client.1.smithi189.stdout:0/90: creat d3/df/f1e x:0 0 0 2024-04-20T21:31:56.607 INFO:tasks.workunit.client.0.smithi062.stdout:0/331: write d7/d1c/d15/d17/d26/f68 [992298,49158] 0 2024-04-20T21:31:56.611 INFO:tasks.workunit.client.1.smithi189.stdout:0/91: creat d3/f1f x:0 0 0 2024-04-20T21:31:56.618 INFO:tasks.workunit.client.0.smithi062.stdout:4/367: dwrite da/dd/d15/f20 [4194304,4194304] 0 2024-04-20T21:31:56.620 INFO:tasks.workunit.client.1.smithi189.stdout:8/82: dwrite da/df/f1b [0,4194304] 0 2024-04-20T21:31:56.621 INFO:tasks.workunit.client.0.smithi062.stdout:4/368: mknod da/dd/d3c/c87 0 2024-04-20T21:31:56.623 INFO:tasks.workunit.client.1.smithi189.stdout:8/83: truncate f4 1258087 0 2024-04-20T21:31:56.623 INFO:tasks.workunit.client.0.smithi062.stdout:4/369: write da/f83 [169369,75889] 0 2024-04-20T21:31:56.626 INFO:tasks.workunit.client.1.smithi189.stdout:0/92: dread d3/fd [0,4194304] 0 2024-04-20T21:31:56.648 INFO:tasks.workunit.client.0.smithi062.stdout:4/370: truncate da/dd/f4d 727603 0 2024-04-20T21:31:56.652 INFO:tasks.workunit.client.1.smithi189.stdout:8/84: dwrite da/fb [0,4194304] 0 2024-04-20T21:31:56.652 INFO:tasks.workunit.client.0.smithi062.stdout:0/332: dread d7/d1e/f24 [0,4194304] 0 2024-04-20T21:31:56.656 INFO:tasks.workunit.client.1.smithi189.stdout:0/93: dwrite d3/f1f [0,4194304] 0 2024-04-20T21:31:56.656 INFO:tasks.workunit.client.0.smithi062.stdout:0/333: mkdir d7/d20/d62/d6f 0 2024-04-20T21:31:56.656 INFO:tasks.workunit.client.1.smithi189.stdout:8/85: read da/fb [874085,23219] 0 2024-04-20T21:31:56.660 INFO:tasks.workunit.client.1.smithi189.stdout:0/94: read - d3/df/f1e zero size 2024-04-20T21:31:56.662 INFO:tasks.workunit.client.1.smithi189.stdout:0/95: readlink d3/df/l13 0 2024-04-20T21:31:56.663 INFO:tasks.workunit.client.0.smithi062.stdout:4/371: dread da/dd/df/d27/d3f/f4c [0,4194304] 0 2024-04-20T21:31:56.664 INFO:tasks.workunit.client.0.smithi062.stdout:0/334: rename d7/d1c/d15/d17/d40/c52 to d7/d31/c70 0 2024-04-20T21:31:56.665 INFO:tasks.workunit.client.1.smithi189.stdout:8/86: link da/df/d13/f18 da/df/f23 0 2024-04-20T21:31:56.666 INFO:tasks.workunit.client.0.smithi062.stdout:0/335: chown c0 58 1 2024-04-20T21:31:56.667 INFO:tasks.workunit.client.1.smithi189.stdout:8/87: truncate da/df/f17 812165 0 2024-04-20T21:31:56.671 INFO:tasks.workunit.client.0.smithi062.stdout:0/336: unlink d7/d1e/l60 0 2024-04-20T21:31:56.673 INFO:tasks.workunit.client.1.smithi189.stdout:8/88: creat da/d1f/f24 x:0 0 0 2024-04-20T21:31:56.673 INFO:tasks.workunit.client.0.smithi062.stdout:0/337: chown d7/ca 12 1 2024-04-20T21:31:56.675 INFO:tasks.workunit.client.0.smithi062.stdout:0/338: mkdir d7/d31/d71 0 2024-04-20T21:31:56.679 INFO:tasks.workunit.client.0.smithi062.stdout:0/339: creat d7/d1c/d15/d17/f72 x:0 0 0 2024-04-20T21:31:56.681 INFO:tasks.workunit.client.0.smithi062.stdout:4/372: truncate da/dd/f19 425971 0 2024-04-20T21:31:56.681 INFO:tasks.workunit.client.1.smithi189.stdout:8/89: getdents da/de/d1e 0 2024-04-20T21:31:56.682 INFO:tasks.workunit.client.0.smithi062.stdout:0/340: creat d7/d20/d62/d6f/f73 x:0 0 0 2024-04-20T21:31:56.684 INFO:tasks.workunit.client.1.smithi189.stdout:8/90: rmdir da/de/d1e 39 2024-04-20T21:31:56.684 INFO:tasks.workunit.client.1.smithi189.stdout:8/91: readlink l8 0 2024-04-20T21:31:56.685 INFO:tasks.workunit.client.1.smithi189.stdout:0/96: dwrite d3/df/f15 [0,4194304] 0 2024-04-20T21:31:56.685 INFO:tasks.workunit.client.0.smithi062.stdout:0/341: mkdir d7/d1c/d15/d17/d1b/d74 0 2024-04-20T21:31:56.687 INFO:tasks.workunit.client.1.smithi189.stdout:0/97: write d3/f1f [2109400,2640] 0 2024-04-20T21:31:56.688 INFO:tasks.workunit.client.1.smithi189.stdout:8/92: write f3 [3910594,66653] 0 2024-04-20T21:31:56.690 INFO:tasks.workunit.client.0.smithi062.stdout:0/342: dread d7/d1c/f3e [0,4194304] 0 2024-04-20T21:31:56.690 INFO:tasks.workunit.client.1.smithi189.stdout:0/98: write d3/df/f1e [768495,12922] 0 2024-04-20T21:31:56.692 INFO:tasks.workunit.client.1.smithi189.stdout:8/93: creat da/d1f/f25 x:0 0 0 2024-04-20T21:31:56.695 INFO:tasks.workunit.client.1.smithi189.stdout:0/99: symlink d3/d16/l20 0 2024-04-20T21:31:56.695 INFO:tasks.workunit.client.1.smithi189.stdout:0/100: chown d3/d1c 5 1 2024-04-20T21:31:56.696 INFO:tasks.workunit.client.1.smithi189.stdout:0/101: rename d3 to d3/d21 22 2024-04-20T21:31:56.700 INFO:tasks.workunit.client.0.smithi062.stdout:0/343: read d7/d1c/d15/d17/d1b/f1a [2997350,128815] 0 2024-04-20T21:31:56.702 INFO:tasks.workunit.client.1.smithi189.stdout:8/94: dread da/df/f11 [0,4194304] 0 2024-04-20T21:31:56.704 INFO:tasks.workunit.client.0.smithi062.stdout:0/344: creat d7/d1e/d6e/f75 x:0 0 0 2024-04-20T21:31:56.706 INFO:tasks.workunit.client.1.smithi189.stdout:8/95: symlink da/df/d22/l26 0 2024-04-20T21:31:56.706 INFO:tasks.workunit.client.1.smithi189.stdout:0/102: dread d3/fe [0,4194304] 0 2024-04-20T21:31:56.707 INFO:tasks.workunit.client.0.smithi062.stdout:0/345: mkdir d7/d1c/d15/d17/d26/d76 0 2024-04-20T21:31:56.709 INFO:tasks.workunit.client.1.smithi189.stdout:8/96: mkdir da/df/d20/d27 0 2024-04-20T21:31:56.710 INFO:tasks.workunit.client.1.smithi189.stdout:8/97: truncate da/d1f/f25 768967 0 2024-04-20T21:31:56.710 INFO:tasks.workunit.client.1.smithi189.stdout:0/103: link d3/fe d3/d16/f22 0 2024-04-20T21:31:56.714 INFO:tasks.workunit.client.0.smithi062.stdout:0/346: link d7/d1c/l5e d7/d20/d62/l77 0 2024-04-20T21:31:56.716 INFO:tasks.workunit.client.1.smithi189.stdout:8/98: getdents da/df 0 2024-04-20T21:31:56.716 INFO:tasks.workunit.client.0.smithi062.stdout:0/347: chown d7/d1e/c6a 159 1 2024-04-20T21:31:56.716 INFO:tasks.workunit.client.0.smithi062.stdout:0/348: stat d7/d20 0 2024-04-20T21:31:56.717 INFO:tasks.workunit.client.1.smithi189.stdout:8/99: read da/df/f1b [3043786,2942] 0 2024-04-20T21:31:56.720 INFO:tasks.workunit.client.0.smithi062.stdout:0/349: unlink d7/d1c/c2e 0 2024-04-20T21:31:56.725 INFO:tasks.workunit.client.0.smithi062.stdout:0/350: rename d7/d1c/d15/d17/d37/l65 to d7/d31/l78 0 2024-04-20T21:31:56.726 INFO:tasks.workunit.client.1.smithi189.stdout:0/104: dwrite d3/fb [0,4194304] 0 2024-04-20T21:31:56.733 INFO:tasks.workunit.client.1.smithi189.stdout:8/100: dread da/df/f1b [0,4194304] 0 2024-04-20T21:31:56.735 INFO:tasks.workunit.client.0.smithi062.stdout:5/335: sync 2024-04-20T21:31:56.738 INFO:tasks.workunit.client.1.smithi189.stdout:8/101: dread da/df/f17 [0,4194304] 0 2024-04-20T21:31:56.741 INFO:tasks.workunit.client.0.smithi062.stdout:5/336: truncate d8/d1e/d2c/d6c/d56/d2b/f40 4554299 0 2024-04-20T21:31:56.741 INFO:tasks.workunit.client.0.smithi062.stdout:0/351: rename d7/d31 to d7/d1c/d15/d17/d40/d79 0 2024-04-20T21:31:56.743 INFO:tasks.workunit.client.1.smithi189.stdout:0/105: dread d3/df/f1b [0,4194304] 0 2024-04-20T21:31:56.743 INFO:tasks.workunit.client.0.smithi062.stdout:5/337: fsync d8/d1e/f53 0 2024-04-20T21:31:56.745 INFO:tasks.workunit.client.1.smithi189.stdout:0/106: creat d3/d16/f23 x:0 0 0 2024-04-20T21:31:56.746 INFO:tasks.workunit.client.0.smithi062.stdout:5/338: creat d8/d1e/d2c/f71 x:0 0 0 2024-04-20T21:31:56.747 INFO:tasks.workunit.client.1.smithi189.stdout:0/107: write d3/df/f1e [786036,60541] 0 2024-04-20T21:31:56.748 INFO:tasks.workunit.client.0.smithi062.stdout:0/352: rename d7/d1c/d15/d17/d26/l48 to d7/d1c/d15/d17/d37/d64/l7a 0 2024-04-20T21:31:56.750 INFO:tasks.workunit.client.1.smithi189.stdout:8/102: dread da/fd [0,4194304] 0 2024-04-20T21:31:56.751 INFO:tasks.workunit.client.0.smithi062.stdout:5/339: symlink d8/d1e/d2c/d6c/d56/l72 0 2024-04-20T21:31:56.752 INFO:tasks.workunit.client.1.smithi189.stdout:0/108: rmdir d3/d1c 0 2024-04-20T21:31:56.753 INFO:tasks.workunit.client.0.smithi062.stdout:0/353: unlink d7/fb 0 2024-04-20T21:31:56.757 INFO:tasks.workunit.client.1.smithi189.stdout:8/103: link da/df/f23 da/de/f28 0 2024-04-20T21:31:56.758 INFO:tasks.workunit.client.0.smithi062.stdout:5/340: chown d8/f1a 55 1 2024-04-20T21:31:56.758 INFO:tasks.workunit.client.0.smithi062.stdout:0/354: write d7/d1c/f39 [3948046,122371] 0 2024-04-20T21:31:56.760 INFO:tasks.workunit.client.0.smithi062.stdout:0/355: mknod d7/d20/c7b 0 2024-04-20T21:31:56.768 INFO:tasks.workunit.client.1.smithi189.stdout:0/109: dread d3/d16/f22 [0,4194304] 0 2024-04-20T21:31:56.782 INFO:tasks.workunit.client.0.smithi062.stdout:5/341: dread d8/d1e/d2c/d6c/d56/f39 [0,4194304] 0 2024-04-20T21:31:56.790 INFO:tasks.workunit.client.0.smithi062.stdout:5/342: creat d8/d1e/d2c/d6c/d56/f73 x:0 0 0 2024-04-20T21:31:56.798 INFO:tasks.workunit.client.0.smithi062.stdout:5/343: rename d8/f1c to d8/d10/f74 0 2024-04-20T21:31:56.802 INFO:tasks.workunit.client.0.smithi062.stdout:5/344: symlink d8/d1e/d2c/d6c/d56/d2b/l75 0 2024-04-20T21:31:56.807 INFO:tasks.workunit.client.0.smithi062.stdout:5/345: write d8/d1e/d2c/d6c/d56/f59 [147499,49976] 0 2024-04-20T21:31:56.824 INFO:tasks.workunit.client.1.smithi189.stdout:0/110: stat d3/d16/c1a 0 2024-04-20T21:31:56.831 INFO:tasks.workunit.client.1.smithi189.stdout:0/111: rename d3/d16 to d3/df/d14/d24 0 2024-04-20T21:31:56.835 INFO:tasks.workunit.client.1.smithi189.stdout:0/112: rename d3/lc to d3/df/d14/d24/l25 0 2024-04-20T21:31:56.843 INFO:tasks.workunit.client.0.smithi062.stdout:5/346: dwrite d8/f44 [0,4194304] 0 2024-04-20T21:31:56.854 INFO:tasks.workunit.client.0.smithi062.stdout:5/347: link d8/c1d d8/d1e/c76 0 2024-04-20T21:31:56.860 INFO:tasks.workunit.client.1.smithi189.stdout:0/113: dwrite d3/df/d14/f1d [0,4194304] 0 2024-04-20T21:31:56.871 INFO:tasks.workunit.client.0.smithi062.stdout:5/348: dwrite d8/d1e/d2c/d6c/d56/f39 [0,4194304] 0 2024-04-20T21:31:56.871 INFO:tasks.workunit.client.1.smithi189.stdout:0/114: dread d3/df/f1e [0,4194304] 0 2024-04-20T21:31:56.872 INFO:tasks.workunit.client.1.smithi189.stdout:8/104: write da/de/f16 [1949644,44356] 0 2024-04-20T21:31:56.877 INFO:tasks.workunit.client.0.smithi062.stdout:5/349: read d8/d10/f1b [3330859,38476] 0 2024-04-20T21:31:56.877 INFO:tasks.workunit.client.1.smithi189.stdout:0/115: rename d3/df/d14 to d3/d26 0 2024-04-20T21:31:56.878 INFO:tasks.workunit.client.1.smithi189.stdout:8/105: symlink da/l29 0 2024-04-20T21:31:56.878 INFO:tasks.workunit.client.1.smithi189.stdout:0/116: stat d3/df 0 2024-04-20T21:31:56.882 INFO:tasks.workunit.client.1.smithi189.stdout:8/106: fdatasync da/df/d13/f14 0 2024-04-20T21:31:56.883 INFO:tasks.workunit.client.0.smithi062.stdout:5/350: mknod d8/d1e/d2c/d6c/d56/d66/c77 0 2024-04-20T21:31:56.884 INFO:tasks.workunit.client.0.smithi062.stdout:5/351: readlink d8/d1e/d2c/l2e 0 2024-04-20T21:31:56.886 INFO:tasks.workunit.client.1.smithi189.stdout:8/107: write da/df/d13/f14 [303800,8888] 0 2024-04-20T21:31:56.890 INFO:tasks.workunit.client.1.smithi189.stdout:8/108: unlink da/df/f17 0 2024-04-20T21:31:56.890 INFO:tasks.workunit.client.0.smithi062.stdout:5/352: unlink d8/d1e/d2c/d6c/d56/d2f/l5e 0 2024-04-20T21:31:56.891 INFO:tasks.workunit.client.1.smithi189.stdout:8/109: write da/df/f19 [834265,8682] 0 2024-04-20T21:31:56.894 INFO:tasks.workunit.client.1.smithi189.stdout:0/117: dread d3/df/f15 [0,4194304] 0 2024-04-20T21:31:56.895 INFO:tasks.workunit.client.1.smithi189.stdout:8/110: dread da/d1f/f25 [0,4194304] 0 2024-04-20T21:31:56.897 INFO:tasks.workunit.client.1.smithi189.stdout:8/111: mkdir da/d1f/d2a 0 2024-04-20T21:31:56.903 INFO:tasks.workunit.client.1.smithi189.stdout:0/118: truncate d3/d26/d24/f22 67697 0 2024-04-20T21:31:56.905 INFO:tasks.workunit.client.0.smithi062.stdout:6/259: sync 2024-04-20T21:31:56.906 INFO:tasks.workunit.client.0.smithi062.stdout:5/353: dwrite d8/d1e/d2c/d6c/d56/d2b/d4c/f55 [0,4194304] 0 2024-04-20T21:31:56.909 INFO:tasks.workunit.client.0.smithi062.stdout:5/354: truncate d8/d38/f70 228273 0 2024-04-20T21:31:56.910 INFO:tasks.workunit.client.0.smithi062.stdout:6/260: mknod d0/d1a/c54 0 2024-04-20T21:31:56.911 INFO:tasks.workunit.client.1.smithi189.stdout:8/112: dread da/df/f11 [0,4194304] 0 2024-04-20T21:31:56.918 INFO:tasks.workunit.client.1.smithi189.stdout:8/113: rename da/df/d13/l1c to da/l2b 0 2024-04-20T21:31:56.919 INFO:tasks.workunit.client.0.smithi062.stdout:6/261: mknod d0/d9/d11/d22/d26/c55 0 2024-04-20T21:31:56.921 INFO:tasks.workunit.client.1.smithi189.stdout:8/114: mkdir da/df/d20/d2c 0 2024-04-20T21:31:56.923 INFO:tasks.workunit.client.0.smithi062.stdout:6/262: creat d0/d9/f56 x:0 0 0 2024-04-20T21:31:56.924 INFO:tasks.workunit.client.1.smithi189.stdout:0/119: dwrite d3/d26/f1d [4194304,4194304] 0 2024-04-20T21:31:56.926 INFO:tasks.workunit.client.1.smithi189.stdout:0/120: fsync d3/d26/d24/f23 0 2024-04-20T21:31:56.932 INFO:tasks.workunit.client.1.smithi189.stdout:8/115: dread da/df/d13/f14 [0,4194304] 0 2024-04-20T21:31:56.934 INFO:tasks.workunit.client.1.smithi189.stdout:8/116: write da/df/f11 [3744545,10786] 0 2024-04-20T21:31:56.949 INFO:tasks.workunit.client.1.smithi189.stdout:8/117: dwrite da/de/f16 [0,4194304] 0 2024-04-20T21:31:57.011 INFO:tasks.workunit.client.1.smithi189.stdout:0/121: readlink d3/d26/d24/l25 0 2024-04-20T21:31:57.015 INFO:tasks.workunit.client.1.smithi189.stdout:0/122: rmdir d3/df 39 2024-04-20T21:31:57.017 INFO:tasks.workunit.client.1.smithi189.stdout:8/118: symlink da/d1f/d2a/l2d 0 2024-04-20T21:31:57.023 INFO:tasks.workunit.client.1.smithi189.stdout:8/119: symlink da/df/d20/l2e 0 2024-04-20T21:31:57.030 INFO:tasks.workunit.client.1.smithi189.stdout:8/120: write da/df/d13/f14 [1132631,78205] 0 2024-04-20T21:31:57.032 INFO:tasks.workunit.client.1.smithi189.stdout:8/121: fsync f3 0 2024-04-20T21:31:57.040 INFO:tasks.workunit.client.1.smithi189.stdout:8/122: fsync da/df/f1b 0 2024-04-20T21:31:57.050 INFO:tasks.workunit.client.1.smithi189.stdout:8/123: unlink f4 0 2024-04-20T21:31:57.053 INFO:tasks.workunit.client.1.smithi189.stdout:8/124: stat da/l29 0 2024-04-20T21:31:57.055 INFO:tasks.workunit.client.1.smithi189.stdout:0/123: dwrite d3/d26/d24/f23 [0,4194304] 0 2024-04-20T21:31:57.058 INFO:tasks.workunit.client.1.smithi189.stdout:8/125: write da/df/f19 [5068303,28327] 0 2024-04-20T21:31:57.064 INFO:tasks.workunit.client.1.smithi189.stdout:8/126: creat da/d1f/f2f x:0 0 0 2024-04-20T21:31:57.067 INFO:tasks.workunit.client.1.smithi189.stdout:0/124: creat d3/f27 x:0 0 0 2024-04-20T21:31:57.070 INFO:tasks.workunit.client.1.smithi189.stdout:8/127: mkdir da/d30 0 2024-04-20T21:31:57.077 INFO:tasks.workunit.client.1.smithi189.stdout:8/128: mkdir da/df/d13/d31 0 2024-04-20T21:31:57.079 INFO:tasks.workunit.client.1.smithi189.stdout:0/125: getdents d3/d26/d24 0 2024-04-20T21:31:57.079 INFO:tasks.workunit.client.1.smithi189.stdout:8/129: chown da/df/c12 169 1 2024-04-20T21:31:57.087 INFO:tasks.workunit.client.1.smithi189.stdout:0/126: mkdir d3/df/d28 0 2024-04-20T21:31:57.092 INFO:tasks.workunit.client.1.smithi189.stdout:8/130: symlink da/de/d1e/d21/l32 0 2024-04-20T21:31:57.094 INFO:tasks.workunit.client.1.smithi189.stdout:0/127: mkdir d3/d26/d24/d29 0 2024-04-20T21:31:57.095 INFO:tasks.workunit.client.0.smithi062.stdout:2/362: sync 2024-04-20T21:31:57.096 INFO:tasks.workunit.client.1.smithi189.stdout:8/131: stat da/l2b 0 2024-04-20T21:31:57.096 INFO:tasks.workunit.client.1.smithi189.stdout:0/128: stat d3/df/d28 0 2024-04-20T21:31:57.098 INFO:tasks.workunit.client.0.smithi062.stdout:2/363: fdatasync d7/da/f13 0 2024-04-20T21:31:57.102 INFO:tasks.workunit.client.1.smithi189.stdout:8/132: mkdir da/df/d22/d33 0 2024-04-20T21:31:57.105 INFO:tasks.workunit.client.1.smithi189.stdout:0/129: creat d3/df/d28/f2a x:0 0 0 2024-04-20T21:31:57.106 INFO:tasks.workunit.client.0.smithi062.stdout:2/364: write d7/da/f22 [3243619,102524] 0 2024-04-20T21:31:57.108 INFO:tasks.workunit.client.1.smithi189.stdout:8/133: mknod da/d1f/d2a/c34 0 2024-04-20T21:31:57.111 INFO:tasks.workunit.client.1.smithi189.stdout:0/130: mkdir d3/df/d28/d2b 0 2024-04-20T21:31:57.114 INFO:tasks.workunit.client.0.smithi062.stdout:2/365: creat d7/da/d28/f6d x:0 0 0 2024-04-20T21:31:57.117 INFO:tasks.workunit.client.1.smithi189.stdout:8/134: fsync f9 0 2024-04-20T21:31:57.120 INFO:tasks.workunit.client.1.smithi189.stdout:0/131: mkdir d3/df/d28/d2b/d2c 0 2024-04-20T21:31:57.127 INFO:tasks.workunit.client.1.smithi189.stdout:8/135: mknod da/d30/c35 0 2024-04-20T21:31:57.129 INFO:tasks.workunit.client.1.smithi189.stdout:8/136: chown da/fb 66129 1 2024-04-20T21:31:57.137 INFO:tasks.workunit.client.1.smithi189.stdout:8/137: creat da/df/d22/d33/f36 x:0 0 0 2024-04-20T21:31:57.154 INFO:tasks.workunit.client.0.smithi062.stdout:2/366: dwrite d7/da/f13 [0,4194304] 0 2024-04-20T21:31:57.170 INFO:tasks.workunit.client.0.smithi062.stdout:2/367: link d7/f1a d7/da/d28/d4b/f6e 0 2024-04-20T21:31:57.179 INFO:tasks.workunit.client.0.smithi062.stdout:9/341: sync 2024-04-20T21:31:57.179 INFO:tasks.workunit.client.0.smithi062.stdout:0/356: sync 2024-04-20T21:31:57.180 INFO:tasks.workunit.client.0.smithi062.stdout:8/379: sync 2024-04-20T21:31:57.180 INFO:tasks.workunit.client.0.smithi062.stdout:1/405: sync 2024-04-20T21:31:57.180 INFO:tasks.workunit.client.0.smithi062.stdout:4/373: sync 2024-04-20T21:31:57.180 INFO:tasks.workunit.client.0.smithi062.stdout:3/384: sync 2024-04-20T21:31:57.180 INFO:tasks.workunit.client.0.smithi062.stdout:7/327: sync 2024-04-20T21:31:57.183 INFO:tasks.workunit.client.0.smithi062.stdout:8/380: dread - d7/d9/d29/f4b zero size 2024-04-20T21:31:57.186 INFO:tasks.workunit.client.0.smithi062.stdout:8/381: stat d7/d9/d1b/f1c 0 2024-04-20T21:31:57.188 INFO:tasks.workunit.client.0.smithi062.stdout:7/328: write d7/d3e/d33/d3d/d4d/d54/d4f/f68 [848021,68568] 0 2024-04-20T21:31:57.191 INFO:tasks.workunit.client.0.smithi062.stdout:7/329: chown d7/d3e/d33/d3d/d5e/l67 70902 1 2024-04-20T21:31:57.202 INFO:tasks.workunit.client.0.smithi062.stdout:3/385: dwrite d0/d1b/f75 [0,4194304] 0 2024-04-20T21:31:57.229 INFO:tasks.workunit.client.0.smithi062.stdout:4/374: dread da/dd/d15/f20 [0,4194304] 0 2024-04-20T21:31:57.236 INFO:tasks.workunit.client.0.smithi062.stdout:4/375: truncate da/dd/d15/d50/d64/d57/d5b/f6f 609873 0 2024-04-20T21:31:57.313 INFO:tasks.workunit.client.0.smithi062.stdout:5/355: sync 2024-04-20T21:31:57.313 INFO:tasks.workunit.client.0.smithi062.stdout:6/263: sync 2024-04-20T21:31:57.323 INFO:tasks.workunit.client.0.smithi062.stdout:5/356: dread d8/d1e/d2c/d6c/d56/f59 [0,4194304] 0 2024-04-20T21:31:57.329 INFO:tasks.workunit.client.0.smithi062.stdout:5/357: write d8/d1e/d2c/d6c/d56/f39 [599212,950] 0 2024-04-20T21:31:57.344 INFO:tasks.workunit.client.1.smithi189.stdout:0/132: dread d3/f1f [0,4194304] 0 2024-04-20T21:31:57.347 INFO:tasks.workunit.client.0.smithi062.stdout:2/368: rename d7/l16 to d7/da/d15/d2f/d38/d6b/l6f 0 2024-04-20T21:31:57.352 INFO:tasks.workunit.client.1.smithi189.stdout:0/133: rmdir d3/d26/d24 39 2024-04-20T21:31:57.355 INFO:tasks.workunit.client.0.smithi062.stdout:9/342: creat d16/d3d/f7a x:0 0 0 2024-04-20T21:31:57.358 INFO:tasks.workunit.client.0.smithi062.stdout:7/330: sync 2024-04-20T21:31:57.358 INFO:tasks.workunit.client.0.smithi062.stdout:2/369: chown d7/da/d15/l62 552 1 2024-04-20T21:31:57.361 INFO:tasks.workunit.client.0.smithi062.stdout:1/406: mknod dc/d15/d20/d28/c8b 0 2024-04-20T21:31:57.363 INFO:tasks.workunit.client.1.smithi189.stdout:8/138: link da/d1f/d2a/c34 da/df/d13/d31/c37 0 2024-04-20T21:31:57.366 INFO:tasks.workunit.client.0.smithi062.stdout:3/386: rmdir d0/d1b 39 2024-04-20T21:31:57.369 INFO:tasks.workunit.client.1.smithi189.stdout:0/134: chown d3/d26/d24/f22 15 1 2024-04-20T21:31:57.372 INFO:tasks.workunit.client.1.smithi189.stdout:0/135: symlink d3/d26/d24/l2d 0 2024-04-20T21:31:57.378 INFO:tasks.workunit.client.1.smithi189.stdout:8/139: creat da/df/d20/d27/f38 x:0 0 0 2024-04-20T21:31:57.379 INFO:tasks.workunit.client.0.smithi062.stdout:5/358: dwrite d8/d38/f70 [0,4194304] 0 2024-04-20T21:31:57.380 INFO:tasks.workunit.client.1.smithi189.stdout:0/136: rename d3/c12 to d3/df/d28/c2e 0 2024-04-20T21:31:57.380 INFO:tasks.workunit.client.0.smithi062.stdout:9/343: unlink d16/d2c/f74 0 2024-04-20T21:31:57.381 INFO:tasks.workunit.client.0.smithi062.stdout:8/382: symlink d7/d9/d1b/l82 0 2024-04-20T21:31:57.386 INFO:tasks.workunit.client.1.smithi189.stdout:0/137: creat d3/d26/f2f x:0 0 0 2024-04-20T21:31:57.387 INFO:tasks.workunit.client.0.smithi062.stdout:7/331: rename d7/cb to d7/d63/c70 0 2024-04-20T21:31:57.396 INFO:tasks.workunit.client.1.smithi189.stdout:0/138: mkdir d3/df/d30 0 2024-04-20T21:31:57.396 INFO:tasks.workunit.client.0.smithi062.stdout:1/407: write f1 [1473763,95130] 0 2024-04-20T21:31:57.399 INFO:tasks.workunit.client.1.smithi189.stdout:0/139: truncate d3/f1f 4569178 0 2024-04-20T21:31:57.401 INFO:tasks.workunit.client.0.smithi062.stdout:5/359: write d8/fb [503951,82536] 0 2024-04-20T21:31:57.402 INFO:tasks.workunit.client.0.smithi062.stdout:4/376: dwrite da/dd/df/d27/f2c [0,4194304] 0 2024-04-20T21:31:57.404 INFO:tasks.workunit.client.0.smithi062.stdout:7/332: chown d7/d3e/d33/l3b 6 1 2024-04-20T21:31:57.404 INFO:tasks.workunit.client.0.smithi062.stdout:3/387: rmdir d0/d1/d3c 39 2024-04-20T21:31:57.405 INFO:tasks.workunit.client.0.smithi062.stdout:0/357: symlink d7/d1c/d15/d17/d1b/d74/l7c 0 2024-04-20T21:31:57.409 INFO:tasks.workunit.client.0.smithi062.stdout:7/333: readlink d7/d3e/d33/d3d/d5e/l67 0 2024-04-20T21:31:57.409 INFO:tasks.workunit.client.0.smithi062.stdout:8/383: write d7/d25/f42 [728632,119458] 0 2024-04-20T21:31:57.412 INFO:tasks.workunit.client.1.smithi189.stdout:0/140: rename d3/df/d28/f2a to d3/d26/d24/f31 0 2024-04-20T21:31:57.412 INFO:tasks.workunit.client.0.smithi062.stdout:6/264: truncate d0/d9/d11/d22/f24 2981558 0 2024-04-20T21:31:57.413 INFO:tasks.workunit.client.1.smithi189.stdout:8/140: dwrite da/df/f23 [0,4194304] 0 2024-04-20T21:31:57.414 INFO:tasks.workunit.client.0.smithi062.stdout:2/370: link d7/da/d15/d2f/c32 d7/da/c70 0 2024-04-20T21:31:57.419 INFO:tasks.workunit.client.0.smithi062.stdout:7/334: creat d7/d3e/f71 x:0 0 0 2024-04-20T21:31:57.419 INFO:tasks.workunit.client.0.smithi062.stdout:6/265: write d0/d9/f13 [2016896,124486] 0 2024-04-20T21:31:57.421 INFO:tasks.workunit.client.0.smithi062.stdout:5/360: truncate d8/d10/f4e 807811 0 2024-04-20T21:31:57.423 INFO:tasks.workunit.client.0.smithi062.stdout:2/371: mknod d7/da/d15/d1b/d24/c71 0 2024-04-20T21:31:57.424 INFO:tasks.workunit.client.0.smithi062.stdout:8/384: link d7/d9/d1b/l82 d7/d25/d3b/l83 0 2024-04-20T21:31:57.424 INFO:tasks.workunit.client.0.smithi062.stdout:6/266: creat d0/d4f/f57 x:0 0 0 2024-04-20T21:31:57.425 INFO:tasks.workunit.client.0.smithi062.stdout:5/361: chown d8/d38 513668 1 2024-04-20T21:31:57.426 INFO:tasks.workunit.client.0.smithi062.stdout:7/335: unlink d7/d3e/f1a 0 2024-04-20T21:31:57.431 INFO:tasks.workunit.client.0.smithi062.stdout:4/377: dwrite da/dd/d15/d50/d64/d57/f80 [0,4194304] 0 2024-04-20T21:31:57.439 INFO:tasks.workunit.client.0.smithi062.stdout:0/358: truncate d7/d1c/d15/d17/d26/f68 514499 0 2024-04-20T21:31:57.448 INFO:tasks.workunit.client.0.smithi062.stdout:2/372: write d7/da/d15/d2f/d38/f54 [254645,99585] 0 2024-04-20T21:31:57.451 INFO:tasks.workunit.client.0.smithi062.stdout:5/362: truncate d8/fb 3103959 0 2024-04-20T21:31:57.454 INFO:tasks.workunit.client.0.smithi062.stdout:4/378: mknod da/dd/df/d27/c88 0 2024-04-20T21:31:57.457 INFO:tasks.workunit.client.0.smithi062.stdout:1/408: dread f1 [0,4194304] 0 2024-04-20T21:31:57.460 INFO:tasks.workunit.client.0.smithi062.stdout:5/363: symlink d8/d1e/l78 0 2024-04-20T21:31:57.462 INFO:tasks.workunit.client.0.smithi062.stdout:3/388: dwrite d0/d1c/d63/f30 [0,4194304] 0 2024-04-20T21:31:57.464 INFO:tasks.workunit.client.0.smithi062.stdout:2/373: symlink d7/da/l72 0 2024-04-20T21:31:57.472 INFO:tasks.workunit.client.0.smithi062.stdout:9/344: dwrite fc [4194304,4194304] 0 2024-04-20T21:31:57.478 INFO:tasks.workunit.client.0.smithi062.stdout:5/364: chown d8/l27 1290 1 2024-04-20T21:31:57.485 INFO:tasks.workunit.client.0.smithi062.stdout:2/374: stat d7/da/d15/d1b/d3d/c61 0 2024-04-20T21:31:57.489 INFO:tasks.workunit.client.0.smithi062.stdout:6/267: dwrite d0/d9/f18 [0,4194304] 0 2024-04-20T21:31:57.492 INFO:tasks.workunit.client.0.smithi062.stdout:9/345: truncate d16/d3e/f64 708242 0 2024-04-20T21:31:57.493 INFO:tasks.workunit.client.0.smithi062.stdout:3/389: unlink d0/d1/d10/d3a/d5e/f62 0 2024-04-20T21:31:57.495 INFO:tasks.workunit.client.0.smithi062.stdout:8/385: dwrite d7/d25/f42 [0,4194304] 0 2024-04-20T21:31:57.497 INFO:tasks.workunit.client.0.smithi062.stdout:7/336: dwrite d7/d3e/d33/d3d/d4d/d54/d4f/f68 [0,4194304] 0 2024-04-20T21:31:57.503 INFO:tasks.workunit.client.0.smithi062.stdout:6/268: chown d0/d4f 106691121 1 2024-04-20T21:31:57.505 INFO:tasks.workunit.client.0.smithi062.stdout:7/337: dread - d7/f6c zero size 2024-04-20T21:31:57.505 INFO:tasks.workunit.client.0.smithi062.stdout:0/359: dwrite d7/d1c/d15/f46 [0,4194304] 0 2024-04-20T21:31:57.506 INFO:tasks.workunit.client.0.smithi062.stdout:9/346: chown d16/d2c/d42/l57 54204582 1 2024-04-20T21:31:57.508 INFO:tasks.workunit.client.0.smithi062.stdout:3/390: write d0/d1/d10/f64 [88403,81890] 0 2024-04-20T21:31:57.510 INFO:tasks.workunit.client.0.smithi062.stdout:4/379: dwrite da/dd/df/f28 [0,4194304] 0 2024-04-20T21:31:57.517 INFO:tasks.workunit.client.0.smithi062.stdout:8/386: getdents d7/d25/d5c 0 2024-04-20T21:31:57.520 INFO:tasks.workunit.client.0.smithi062.stdout:4/380: read - da/dd/d15/d50/d64/d3d/f7e zero size 2024-04-20T21:31:57.523 INFO:tasks.workunit.client.0.smithi062.stdout:7/338: rename d7/d3e/d6e to d7/d3e/d33/d3d/d5e/d72 0 2024-04-20T21:31:57.531 INFO:tasks.workunit.client.0.smithi062.stdout:1/409: truncate dc/d15/d17/d36/d3a/f43 1819199 0 2024-04-20T21:31:57.539 INFO:tasks.workunit.client.0.smithi062.stdout:5/365: dwrite d8/d1e/d2c/d6c/d56/d50/f64 [0,4194304] 0 2024-04-20T21:31:57.539 INFO:tasks.workunit.client.0.smithi062.stdout:1/410: write dc/d15/d17/d36/f54 [5056280,84192] 0 2024-04-20T21:31:57.542 INFO:tasks.workunit.client.0.smithi062.stdout:8/387: truncate d7/d25/d5c/f69 165195 0 2024-04-20T21:31:57.543 INFO:tasks.workunit.client.0.smithi062.stdout:5/366: write d8/d1e/f53 [1747412,128720] 0 2024-04-20T21:31:57.547 INFO:tasks.workunit.client.0.smithi062.stdout:1/411: chown dc/d15/d20/f3c 18937916 1 2024-04-20T21:31:57.556 INFO:tasks.workunit.client.0.smithi062.stdout:1/412: creat dc/d70/f8c x:0 0 0 2024-04-20T21:31:57.556 INFO:tasks.workunit.client.0.smithi062.stdout:8/388: link d7/d9/d1b/f1e d7/d25/d76/f84 0 2024-04-20T21:31:57.557 INFO:tasks.workunit.client.0.smithi062.stdout:7/339: dwrite d7/d3e/d33/d3d/d4d/d54/f2e [0,4194304] 0 2024-04-20T21:31:57.571 INFO:tasks.workunit.client.0.smithi062.stdout:5/367: write d8/d10/f17 [6418903,43971] 0 2024-04-20T21:31:57.573 INFO:tasks.workunit.client.0.smithi062.stdout:1/413: chown dc/d15/c22 817291 1 2024-04-20T21:31:57.574 INFO:tasks.workunit.client.0.smithi062.stdout:7/340: mknod d7/d3e/c73 0 2024-04-20T21:31:57.580 INFO:tasks.workunit.client.0.smithi062.stdout:9/347: dread d16/d3d/f43 [0,4194304] 0 2024-04-20T21:31:57.580 INFO:tasks.workunit.client.0.smithi062.stdout:3/391: dread d0/f4c [0,4194304] 0 2024-04-20T21:31:57.583 INFO:tasks.workunit.client.0.smithi062.stdout:5/368: read d8/d10/f26 [373905,59635] 0 2024-04-20T21:31:57.584 INFO:tasks.workunit.client.0.smithi062.stdout:7/341: mkdir d7/d26/d74 0 2024-04-20T21:31:57.585 INFO:tasks.workunit.client.0.smithi062.stdout:4/381: dread da/f7b [0,4194304] 0 2024-04-20T21:31:57.589 INFO:tasks.workunit.client.0.smithi062.stdout:5/369: creat d8/d1e/d2c/f79 x:0 0 0 2024-04-20T21:31:57.592 INFO:tasks.workunit.client.0.smithi062.stdout:4/382: unlink da/dd/d15/c22 0 2024-04-20T21:31:57.595 INFO:tasks.workunit.client.0.smithi062.stdout:7/342: creat d7/d3e/d33/d3d/f75 x:0 0 0 2024-04-20T21:31:57.596 INFO:tasks.workunit.client.0.smithi062.stdout:0/360: dread d7/d1c/f18 [0,4194304] 0 2024-04-20T21:31:57.600 INFO:tasks.workunit.client.0.smithi062.stdout:7/343: stat d7/f2a 0 2024-04-20T21:31:57.605 INFO:tasks.workunit.client.0.smithi062.stdout:1/414: dwrite dc/d15/d17/d36/d3a/f4d [0,4194304] 0 2024-04-20T21:31:57.608 INFO:tasks.workunit.client.0.smithi062.stdout:4/383: rename da/dd/df/d27/d7a/f7d to da/dd/df/d27/d32/f89 0 2024-04-20T21:31:57.610 INFO:tasks.workunit.client.0.smithi062.stdout:7/344: write d7/d3e/d33/d3d/d4d/d54/d4f/f53 [417950,82843] 0 2024-04-20T21:31:57.612 INFO:tasks.workunit.client.0.smithi062.stdout:0/361: getdents d7/d1c/d15 0 2024-04-20T21:31:57.615 INFO:tasks.workunit.client.0.smithi062.stdout:9/348: dread d16/d2c/d42/f50 [0,4194304] 0 2024-04-20T21:31:57.616 INFO:tasks.workunit.client.0.smithi062.stdout:4/384: readlink da/dd/df/d27/l42 0 2024-04-20T21:31:57.616 INFO:tasks.workunit.client.0.smithi062.stdout:8/389: dread d7/d9/d1b/f4e [0,4194304] 0 2024-04-20T21:31:57.622 INFO:tasks.workunit.client.0.smithi062.stdout:9/349: chown d16/d2c/l35 50 1 2024-04-20T21:31:57.622 INFO:tasks.workunit.client.0.smithi062.stdout:8/390: truncate d7/d30/f78 598130 0 2024-04-20T21:31:57.623 INFO:tasks.workunit.client.0.smithi062.stdout:3/392: dwrite d0/d1c/d52/d7b/f68 [0,4194304] 0 2024-04-20T21:31:57.624 INFO:tasks.workunit.client.0.smithi062.stdout:4/385: mknod da/dd/c8a 0 2024-04-20T21:31:57.626 INFO:tasks.workunit.client.0.smithi062.stdout:5/370: dread d8/d38/f70 [0,4194304] 0 2024-04-20T21:31:57.629 INFO:tasks.workunit.client.0.smithi062.stdout:9/350: creat d16/d2c/d5e/f7b x:0 0 0 2024-04-20T21:31:57.631 INFO:tasks.workunit.client.0.smithi062.stdout:0/362: truncate d7/d1c/f32 1934871 0 2024-04-20T21:31:57.636 INFO:tasks.workunit.client.0.smithi062.stdout:0/363: chown d7/d1c/d15/d17/f72 115749 1 2024-04-20T21:31:57.638 INFO:tasks.workunit.client.0.smithi062.stdout:0/364: readlink d7/d1c/d15/d17/l3f 0 2024-04-20T21:31:57.639 INFO:tasks.workunit.client.0.smithi062.stdout:5/371: rename d8/d1e/d2c/d6c/d56/d2b to d8/d38/d7a 0 2024-04-20T21:31:57.641 INFO:tasks.workunit.client.0.smithi062.stdout:3/393: link d0/d1/d10/d3a/d5e/f7d d0/d1/d10/d3a/d5e/f8d 0 2024-04-20T21:31:57.642 INFO:tasks.workunit.client.0.smithi062.stdout:7/345: dwrite d7/d3e/d33/d3d/d4d/f5a [0,4194304] 0 2024-04-20T21:31:57.646 INFO:tasks.workunit.client.0.smithi062.stdout:9/351: getdents d16/d2c/d5e/d75/d76 0 2024-04-20T21:31:57.650 INFO:tasks.workunit.client.0.smithi062.stdout:0/365: creat d7/d1e/d6e/f7d x:0 0 0 2024-04-20T21:31:57.653 INFO:tasks.workunit.client.0.smithi062.stdout:9/352: unlink d16/d2c/d5e/d75/c79 0 2024-04-20T21:31:57.654 INFO:tasks.workunit.client.0.smithi062.stdout:7/346: read d7/d3e/f20 [2545010,107350] 0 2024-04-20T21:31:57.657 INFO:tasks.workunit.client.0.smithi062.stdout:0/366: chown d7/d1e/d6e/f75 93434 1 2024-04-20T21:31:57.659 INFO:tasks.workunit.client.0.smithi062.stdout:8/391: dwrite d7/d9/f12 [0,4194304] 0 2024-04-20T21:31:57.661 INFO:tasks.workunit.client.0.smithi062.stdout:9/353: chown d16/c2b 16766838 1 2024-04-20T21:31:57.664 INFO:tasks.workunit.client.0.smithi062.stdout:4/386: dwrite da/dd/df/f41 [4194304,4194304] 0 2024-04-20T21:31:57.668 INFO:tasks.workunit.client.0.smithi062.stdout:0/367: symlink d7/d20/d3a/l7e 0 2024-04-20T21:31:57.671 INFO:tasks.workunit.client.0.smithi062.stdout:9/354: creat d16/d2f/d69/f7c x:0 0 0 2024-04-20T21:31:57.671 INFO:tasks.workunit.client.0.smithi062.stdout:0/368: chown d7/d1c/d15/d17/d37/d64 9747257 1 2024-04-20T21:31:57.671 INFO:tasks.workunit.client.0.smithi062.stdout:3/394: dread d0/f8 [0,4194304] 0 2024-04-20T21:31:57.672 INFO:tasks.workunit.client.0.smithi062.stdout:5/372: dwrite d8/d10/f28 [0,4194304] 0 2024-04-20T21:31:57.681 INFO:tasks.workunit.client.0.smithi062.stdout:0/369: write d7/d1c/f39 [4384656,65384] 0 2024-04-20T21:31:57.683 INFO:tasks.workunit.client.0.smithi062.stdout:5/373: creat d8/d38/d7a/d4c/f7b x:0 0 0 2024-04-20T21:31:57.685 INFO:tasks.workunit.client.0.smithi062.stdout:4/387: getdents da/dd/d15/d50/d64/d57/d5b 0 2024-04-20T21:31:57.688 INFO:tasks.workunit.client.0.smithi062.stdout:5/374: chown d8/d38/d7a/f40 5592880 1 2024-04-20T21:31:57.689 INFO:tasks.workunit.client.0.smithi062.stdout:0/370: write d7/d1e/d6e/f75 [481986,8757] 0 2024-04-20T21:31:57.694 INFO:tasks.workunit.client.0.smithi062.stdout:5/375: chown d8/d10/d3e/f62 342593 1 2024-04-20T21:31:57.695 INFO:tasks.workunit.client.0.smithi062.stdout:9/355: truncate d16/d2c/d42/f54 293636 0 2024-04-20T21:31:57.697 INFO:tasks.workunit.client.0.smithi062.stdout:7/347: dwrite d7/f24 [0,4194304] 0 2024-04-20T21:31:57.701 INFO:tasks.workunit.client.0.smithi062.stdout:4/388: creat da/dd/df/d27/d3f/d82/f8b x:0 0 0 2024-04-20T21:31:57.701 INFO:tasks.workunit.client.0.smithi062.stdout:5/376: mknod d8/d38/c7c 0 2024-04-20T21:31:57.702 INFO:tasks.workunit.client.0.smithi062.stdout:7/348: stat d7/d26/f29 0 2024-04-20T21:31:57.705 INFO:tasks.workunit.client.0.smithi062.stdout:9/356: mkdir d16/d3d/d55/d6e/d7d 0 2024-04-20T21:31:57.706 INFO:tasks.workunit.client.0.smithi062.stdout:4/389: stat da/dd/df/d27/c43 0 2024-04-20T21:31:57.707 INFO:tasks.workunit.client.0.smithi062.stdout:3/395: dread d0/d1b/f75 [0,4194304] 0 2024-04-20T21:31:57.710 INFO:tasks.workunit.client.0.smithi062.stdout:5/377: mkdir d8/d38/d7a/d4c/d7d 0 2024-04-20T21:31:57.711 INFO:tasks.workunit.client.0.smithi062.stdout:3/396: fsync d0/d1/f19 0 2024-04-20T21:31:57.712 INFO:tasks.workunit.client.0.smithi062.stdout:4/390: readlink da/dd/d15/d50/l6b 0 2024-04-20T21:31:57.717 INFO:tasks.workunit.client.0.smithi062.stdout:3/397: symlink d0/d1/d10/d3a/l8e 0 2024-04-20T21:31:57.718 INFO:tasks.workunit.client.0.smithi062.stdout:0/371: dread d7/d1c/f32 [0,4194304] 0 2024-04-20T21:31:57.722 INFO:tasks.workunit.client.0.smithi062.stdout:7/349: write d7/d3e/f44 [6376636,52954] 0 2024-04-20T21:31:57.725 INFO:tasks.workunit.client.0.smithi062.stdout:0/372: mknod d7/d1c/d15/d17/c7f 0 2024-04-20T21:31:57.727 INFO:tasks.workunit.client.0.smithi062.stdout:7/350: fsync d7/d3e/d33/f52 0 2024-04-20T21:31:57.729 INFO:tasks.workunit.client.0.smithi062.stdout:0/373: mkdir d7/d1c/d15/d80 0 2024-04-20T21:31:57.730 INFO:tasks.workunit.client.0.smithi062.stdout:9/357: dwrite d16/d3d/f43 [0,4194304] 0 2024-04-20T21:31:57.734 INFO:tasks.workunit.client.0.smithi062.stdout:0/374: fsync d7/d1e/f24 0 2024-04-20T21:31:57.737 INFO:tasks.workunit.client.0.smithi062.stdout:9/358: creat d16/d2f/d69/f7e x:0 0 0 2024-04-20T21:31:57.740 INFO:tasks.workunit.client.0.smithi062.stdout:4/391: dwrite da/dd/f73 [0,4194304] 0 2024-04-20T21:31:57.743 INFO:tasks.workunit.client.0.smithi062.stdout:0/375: mknod d7/d1c/d15/d17/d40/d3c/c81 0 2024-04-20T21:31:57.744 INFO:tasks.workunit.client.0.smithi062.stdout:9/359: write d16/d3d/f7a [782445,82063] 0 2024-04-20T21:31:57.746 INFO:tasks.workunit.client.0.smithi062.stdout:4/392: stat da/dd/d15/d50/d64/l62 0 2024-04-20T21:31:57.747 INFO:tasks.workunit.client.0.smithi062.stdout:3/398: dwrite d0/d1/fd [0,4194304] 0 2024-04-20T21:31:57.750 INFO:tasks.workunit.client.0.smithi062.stdout:9/360: creat d16/d2f/f7f x:0 0 0 2024-04-20T21:31:57.751 INFO:tasks.workunit.client.1.smithi189.stdout:0/141: chown d3/d26/d24/d29 0 1 2024-04-20T21:31:57.751 INFO:tasks.workunit.client.0.smithi062.stdout:0/376: symlink d7/d1c/d15/d17/d1b/l82 0 2024-04-20T21:31:57.753 INFO:tasks.workunit.client.0.smithi062.stdout:4/393: rmdir da/dd/d15 39 2024-04-20T21:31:57.760 INFO:tasks.workunit.client.1.smithi189.stdout:0/142: write d3/fd [3428495,32986] 0 2024-04-20T21:31:57.783 INFO:tasks.workunit.client.1.smithi189.stdout:8/141: dwrite da/de/f16 [0,4194304] 0 2024-04-20T21:31:57.808 INFO:tasks.workunit.client.0.smithi062.stdout:0/377: mkdir d7/d1c/d15/d17/d26/d83 0 2024-04-20T21:31:57.808 INFO:tasks.workunit.client.0.smithi062.stdout:7/351: dwrite d7/d3e/d33/d3d/f4a [0,4194304] 0 2024-04-20T21:31:57.809 INFO:tasks.workunit.client.0.smithi062.stdout:4/394: truncate da/dd/f69 9179770 0 2024-04-20T21:31:57.813 INFO:tasks.workunit.client.0.smithi062.stdout:0/378: fsync d7/d1c/d15/d17/d37/f59 0 2024-04-20T21:31:57.813 INFO:tasks.workunit.client.0.smithi062.stdout:7/352: write d7/f5f [565388,84289] 0 2024-04-20T21:31:57.814 INFO:tasks.workunit.client.0.smithi062.stdout:9/361: truncate d16/d4e/f6f 1751030 0 2024-04-20T21:31:57.817 INFO:tasks.workunit.client.0.smithi062.stdout:0/379: write d7/d20/d62/d6f/f73 [529528,102237] 0 2024-04-20T21:31:57.818 INFO:tasks.workunit.client.0.smithi062.stdout:7/353: creat d7/d3e/d33/f76 x:0 0 0 2024-04-20T21:31:57.819 INFO:tasks.workunit.client.0.smithi062.stdout:9/362: readlink d16/d2f/l51 0 2024-04-20T21:31:57.820 INFO:tasks.workunit.client.0.smithi062.stdout:9/363: stat d16/d2c/d5e 0 2024-04-20T21:31:57.822 INFO:tasks.workunit.client.0.smithi062.stdout:4/395: truncate da/dd/f69 5990863 0 2024-04-20T21:31:57.822 INFO:tasks.workunit.client.0.smithi062.stdout:7/354: chown d7/d26/c3c 36 1 2024-04-20T21:31:57.825 INFO:tasks.workunit.client.0.smithi062.stdout:0/380: read d7/d1c/d15/d17/d40/d79/f57 [719039,116110] 0 2024-04-20T21:31:57.825 INFO:tasks.workunit.client.0.smithi062.stdout:9/364: chown d16/d3d/l63 79 1 2024-04-20T21:31:57.826 INFO:tasks.workunit.client.0.smithi062.stdout:4/396: creat da/d56/f8c x:0 0 0 2024-04-20T21:31:57.829 INFO:tasks.workunit.client.0.smithi062.stdout:7/355: read f4 [2182904,65292] 0 2024-04-20T21:31:57.831 INFO:tasks.workunit.client.0.smithi062.stdout:0/381: creat d7/d1c/d15/d17/d37/f84 x:0 0 0 2024-04-20T21:31:57.832 INFO:tasks.workunit.client.0.smithi062.stdout:9/365: truncate d16/f2e 4075894 0 2024-04-20T21:31:57.833 INFO:tasks.workunit.client.0.smithi062.stdout:7/356: symlink d7/d3e/d33/d3d/d5e/d72/l77 0 2024-04-20T21:31:57.835 INFO:tasks.workunit.client.0.smithi062.stdout:9/366: fdatasync d16/f2e 0 2024-04-20T21:31:57.840 INFO:tasks.workunit.client.0.smithi062.stdout:4/397: write da/f16 [3904673,102258] 0 2024-04-20T21:31:57.842 INFO:tasks.workunit.client.0.smithi062.stdout:7/357: fsync d7/d3e/d33/d3d/d4d/d54/f2b 0 2024-04-20T21:31:57.849 INFO:tasks.workunit.client.0.smithi062.stdout:4/398: creat da/dd/d15/d50/d64/d57/d5b/f8d x:0 0 0 2024-04-20T21:31:57.851 INFO:tasks.workunit.client.0.smithi062.stdout:9/367: dread d16/d3d/f43 [0,4194304] 0 2024-04-20T21:31:57.852 INFO:tasks.workunit.client.0.smithi062.stdout:0/382: truncate d7/d1c/f41 725365 0 2024-04-20T21:31:57.853 INFO:tasks.workunit.client.0.smithi062.stdout:4/399: stat da/dd/df/d27/c43 0 2024-04-20T21:31:57.861 INFO:tasks.workunit.client.0.smithi062.stdout:4/400: truncate da/dd/d15/d50/d64/d57/d5b/f68 661952 0 2024-04-20T21:31:57.862 INFO:tasks.workunit.client.0.smithi062.stdout:0/383: rename d7/d1e/f24 to d7/d1c/d15/d17/d40/f85 0 2024-04-20T21:31:57.862 INFO:tasks.workunit.client.1.smithi189.stdout:0/143: symlink d3/df/d28/d2b/d2c/l32 0 2024-04-20T21:31:57.867 INFO:tasks.workunit.client.0.smithi062.stdout:7/358: dwrite d7/fe [4194304,4194304] 0 2024-04-20T21:31:57.874 INFO:tasks.workunit.client.0.smithi062.stdout:7/359: mkdir d7/d63/d78 0 2024-04-20T21:31:57.875 INFO:tasks.workunit.client.0.smithi062.stdout:4/401: getdents da/d56 0 2024-04-20T21:31:57.877 INFO:tasks.workunit.client.0.smithi062.stdout:9/368: dwrite d16/d2c/d42/f70 [0,4194304] 0 2024-04-20T21:31:57.886 INFO:tasks.workunit.client.0.smithi062.stdout:7/360: truncate d7/d3e/f20 4417278 0 2024-04-20T21:31:57.888 INFO:tasks.workunit.client.0.smithi062.stdout:9/369: read d16/d2c/d42/f48 [578125,51554] 0 2024-04-20T21:31:57.902 INFO:tasks.workunit.client.0.smithi062.stdout:7/361: symlink d7/d3e/d33/d3d/d4d/d54/d4f/l79 0 2024-04-20T21:31:57.907 INFO:tasks.workunit.client.0.smithi062.stdout:0/384: dwrite d7/d1c/f32 [0,4194304] 0 2024-04-20T21:31:57.907 INFO:tasks.workunit.client.0.smithi062.stdout:9/370: dread d16/d3d/d55/d6e/f72 [0,4194304] 0 2024-04-20T21:31:57.911 INFO:tasks.workunit.client.0.smithi062.stdout:7/362: unlink d7/d3e/l42 0 2024-04-20T21:31:57.912 INFO:tasks.workunit.client.0.smithi062.stdout:3/399: sync 2024-04-20T21:31:57.912 INFO:tasks.workunit.client.0.smithi062.stdout:8/392: sync 2024-04-20T21:31:57.912 INFO:tasks.workunit.client.0.smithi062.stdout:5/378: sync 2024-04-20T21:31:57.913 INFO:tasks.workunit.client.0.smithi062.stdout:6/269: sync 2024-04-20T21:31:57.913 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:57 smithi062 ceph-mon[16221]: pgmap v13: 65 pgs: 65 active+clean; 654 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 7.1 MiB/s rd, 63 MiB/s wr, 203 op/s 2024-04-20T21:31:57.913 INFO:tasks.workunit.client.0.smithi062.stdout:2/375: sync 2024-04-20T21:31:57.913 INFO:tasks.workunit.client.0.smithi062.stdout:1/415: sync 2024-04-20T21:31:57.917 INFO:tasks.workunit.client.0.smithi062.stdout:4/402: dwrite f4 [0,4194304] 0 2024-04-20T21:31:57.919 INFO:tasks.workunit.client.0.smithi062.stdout:3/400: unlink d0/d1/f32 0 2024-04-20T21:31:57.933 INFO:tasks.workunit.client.0.smithi062.stdout:5/379: symlink d8/d1e/d2c/l7e 0 2024-04-20T21:31:57.933 INFO:tasks.workunit.client.0.smithi062.stdout:6/270: symlink d0/d9/d11/d22/l58 0 2024-04-20T21:31:57.934 INFO:tasks.workunit.client.0.smithi062.stdout:0/385: chown d7/d1c/d15/d17/d37/f59 188709638 1 2024-04-20T21:31:57.948 INFO:tasks.workunit.client.0.smithi062.stdout:0/386: mknod d7/d20/c86 0 2024-04-20T21:31:57.953 INFO:tasks.workunit.client.0.smithi062.stdout:9/371: dwrite d16/d2c/d42/f70 [0,4194304] 0 2024-04-20T21:31:57.956 INFO:tasks.workunit.client.0.smithi062.stdout:8/393: creat d7/d9/d1b/d24/f85 x:0 0 0 2024-04-20T21:31:57.956 INFO:tasks.workunit.client.0.smithi062.stdout:7/363: dread d7/d3e/d33/d3d/d4d/d54/d4f/f68 [0,4194304] 0 2024-04-20T21:31:57.959 INFO:tasks.workunit.client.0.smithi062.stdout:9/372: stat d16/d3d 0 2024-04-20T21:31:57.963 INFO:tasks.workunit.client.0.smithi062.stdout:0/387: write d7/d1c/f3e [829434,26054] 0 2024-04-20T21:31:57.963 INFO:tasks.workunit.client.1.smithi189.stdout:8/142: truncate da/de/f28 642398 0 2024-04-20T21:31:57.966 INFO:tasks.workunit.client.1.smithi189.stdout:8/143: dread - da/df/d20/d27/f38 zero size 2024-04-20T21:31:57.967 INFO:tasks.workunit.client.0.smithi062.stdout:3/401: getdents d0/d1/d3c/d43/d72 0 2024-04-20T21:31:57.967 INFO:tasks.workunit.client.0.smithi062.stdout:9/373: fdatasync d16/f36 0 2024-04-20T21:31:57.968 INFO:tasks.workunit.client.1.smithi189.stdout:8/144: readlink da/de/d1e/d21/l32 0 2024-04-20T21:31:57.976 INFO:tasks.workunit.client.0.smithi062.stdout:1/416: dwrite dc/d15/d17/f34 [0,4194304] 0 2024-04-20T21:31:57.976 INFO:tasks.workunit.client.0.smithi062.stdout:2/376: dwrite d7/da/d28/f53 [0,4194304] 0 2024-04-20T21:31:57.977 INFO:tasks.workunit.client.0.smithi062.stdout:9/374: dread - d16/d3e/f5f zero size 2024-04-20T21:31:57.977 INFO:tasks.workunit.client.0.smithi062.stdout:7/364: chown d7/d3e/d33/f57 3 1 2024-04-20T21:31:57.983 INFO:tasks.workunit.client.0.smithi062.stdout:2/377: stat d7/d39/c3b 0 2024-04-20T21:31:57.983 INFO:tasks.workunit.client.0.smithi062.stdout:3/402: stat d0/d1/d10/l26 0 2024-04-20T21:31:57.984 INFO:tasks.workunit.client.1.smithi189.stdout:8/145: dwrite f3 [0,4194304] 0 2024-04-20T21:31:57.987 INFO:tasks.workunit.client.0.smithi062.stdout:0/388: write d7/d1c/d15/d17/d40/f58 [5788,86929] 0 2024-04-20T21:31:57.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:57 smithi189 ceph-mon[21562]: pgmap v13: 65 pgs: 65 active+clean; 654 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 7.1 MiB/s rd, 63 MiB/s wr, 203 op/s 2024-04-20T21:31:57.993 INFO:tasks.workunit.client.0.smithi062.stdout:7/365: write d7/d26/f29 [91952,21734] 0 2024-04-20T21:31:57.996 INFO:tasks.workunit.client.1.smithi189.stdout:8/146: rmdir da/de/d1e 39 2024-04-20T21:31:58.001 INFO:tasks.workunit.client.0.smithi062.stdout:9/375: chown f15 320282605 1 2024-04-20T21:31:58.001 INFO:tasks.workunit.client.0.smithi062.stdout:6/271: dwrite d0/d1a/f29 [0,4194304] 0 2024-04-20T21:31:58.007 INFO:tasks.workunit.client.1.smithi189.stdout:8/147: creat da/f39 x:0 0 0 2024-04-20T21:31:58.008 INFO:tasks.workunit.client.0.smithi062.stdout:2/378: creat d7/da/d28/d4b/f73 x:0 0 0 2024-04-20T21:31:58.015 INFO:tasks.workunit.client.0.smithi062.stdout:5/380: dwrite d8/d38/f6f [0,4194304] 0 2024-04-20T21:31:58.020 INFO:tasks.workunit.client.1.smithi189.stdout:8/148: rmdir da/df 39 2024-04-20T21:31:58.021 INFO:tasks.workunit.client.0.smithi062.stdout:9/376: mkdir d16/d4e/d80 0 2024-04-20T21:31:58.021 INFO:tasks.workunit.client.0.smithi062.stdout:4/403: dread da/dd/df/f17 [0,4194304] 0 2024-04-20T21:31:58.023 INFO:tasks.workunit.client.0.smithi062.stdout:5/381: dread - d8/d38/d7a/d4c/f7b zero size 2024-04-20T21:31:58.025 INFO:tasks.workunit.client.0.smithi062.stdout:7/366: rename d7/d3e/f30 to d7/d63/f7a 0 2024-04-20T21:31:58.026 INFO:tasks.workunit.client.0.smithi062.stdout:0/389: link d7/f4c d7/d20/d3a/f87 0 2024-04-20T21:31:58.029 INFO:tasks.workunit.client.0.smithi062.stdout:3/403: dread d0/d1b/f2b [0,4194304] 0 2024-04-20T21:31:58.043 INFO:tasks.workunit.client.1.smithi189.stdout:8/149: dwrite da/df/d20/d27/f38 [0,4194304] 0 2024-04-20T21:31:58.046 INFO:tasks.workunit.client.1.smithi189.stdout:8/150: mkdir da/df/d3a 0 2024-04-20T21:31:58.051 INFO:tasks.workunit.client.0.smithi062.stdout:2/379: rename d7/da/d15/d1b/l3a to d7/d39/l74 0 2024-04-20T21:31:58.052 INFO:tasks.workunit.client.1.smithi189.stdout:8/151: symlink da/df/d13/d31/l3b 0 2024-04-20T21:31:58.053 INFO:tasks.workunit.client.0.smithi062.stdout:4/404: creat da/dd/d15/d6d/f8e x:0 0 0 2024-04-20T21:31:58.054 INFO:tasks.workunit.client.0.smithi062.stdout:9/377: chown d16/c58 21853922 1 2024-04-20T21:31:58.056 INFO:tasks.workunit.client.1.smithi189.stdout:8/152: mkdir da/df/d3c 0 2024-04-20T21:31:58.057 INFO:tasks.workunit.client.0.smithi062.stdout:5/382: write d8/f1a [352694,32791] 0 2024-04-20T21:31:58.058 INFO:tasks.workunit.client.1.smithi189.stdout:8/153: chown da/df/d22/d33 24 1 2024-04-20T21:31:58.062 INFO:tasks.workunit.client.0.smithi062.stdout:0/390: chown d7/d1c/d15/d17/d40/d79/c70 498 1 2024-04-20T21:31:58.065 INFO:tasks.workunit.client.1.smithi189.stdout:8/154: dread da/d1f/f25 [0,4194304] 0 2024-04-20T21:31:58.071 INFO:tasks.workunit.client.0.smithi062.stdout:3/404: mknod d0/d1/d10/c8f 0 2024-04-20T21:31:58.071 INFO:tasks.workunit.client.0.smithi062.stdout:4/405: fsync da/dd/f69 0 2024-04-20T21:31:58.072 INFO:tasks.workunit.client.0.smithi062.stdout:7/367: truncate d7/d3e/d33/d3d/d4d/d54/f2b 695600 0 2024-04-20T21:31:58.073 INFO:tasks.workunit.client.0.smithi062.stdout:2/380: symlink d7/da/d15/d1b/d24/l75 0 2024-04-20T21:31:58.075 INFO:tasks.workunit.client.1.smithi189.stdout:8/155: getdents da/df/d13 0 2024-04-20T21:31:58.075 INFO:tasks.workunit.client.0.smithi062.stdout:8/394: dread d7/d9/d1b/f3f [0,4194304] 0 2024-04-20T21:31:58.076 INFO:tasks.workunit.client.0.smithi062.stdout:1/417: dwrite dc/d15/d17/f67 [0,4194304] 0 2024-04-20T21:31:58.078 INFO:tasks.workunit.client.0.smithi062.stdout:4/406: write da/dd/f14 [1286603,8639] 0 2024-04-20T21:31:58.078 INFO:tasks.workunit.client.1.smithi189.stdout:8/156: mknod da/d1f/c3d 0 2024-04-20T21:31:58.079 INFO:tasks.workunit.client.0.smithi062.stdout:5/383: link d8/d1e/d2c/f71 d8/d38/d7a/d4c/f7f 0 2024-04-20T21:31:58.079 INFO:tasks.workunit.client.1.smithi189.stdout:8/157: readlink da/lc 0 2024-04-20T21:31:58.081 INFO:tasks.workunit.client.0.smithi062.stdout:5/384: readlink d8/d38/d7a/l31 0 2024-04-20T21:31:58.081 INFO:tasks.workunit.client.0.smithi062.stdout:3/405: write d0/d1c/f51 [483786,125704] 0 2024-04-20T21:31:58.082 INFO:tasks.workunit.client.0.smithi062.stdout:4/407: stat da/dd/d15/d6d/f8e 0 2024-04-20T21:31:58.085 INFO:tasks.workunit.client.0.smithi062.stdout:0/391: creat d7/d1c/d15/d17/d1b/d74/f88 x:0 0 0 2024-04-20T21:31:58.085 INFO:tasks.workunit.client.0.smithi062.stdout:9/378: truncate d16/f1d 5913373 0 2024-04-20T21:31:58.087 INFO:tasks.workunit.client.0.smithi062.stdout:5/385: mkdir d8/d1e/d80 0 2024-04-20T21:31:58.088 INFO:tasks.workunit.client.0.smithi062.stdout:1/418: write dc/d15/f16 [3934980,81828] 0 2024-04-20T21:31:58.090 INFO:tasks.workunit.client.1.smithi189.stdout:8/158: dread da/de/f28 [0,4194304] 0 2024-04-20T21:31:58.090 INFO:tasks.workunit.client.0.smithi062.stdout:8/395: mknod d7/d25/c86 0 2024-04-20T21:31:58.092 INFO:tasks.workunit.client.0.smithi062.stdout:4/408: symlink da/dd/df/d27/d7a/l8f 0 2024-04-20T21:31:58.095 INFO:tasks.workunit.client.1.smithi189.stdout:8/159: unlink da/d1f/c3d 0 2024-04-20T21:31:58.097 INFO:tasks.workunit.client.0.smithi062.stdout:7/368: rename d7/d26/f31 to d7/d3e/f7b 0 2024-04-20T21:31:58.097 INFO:tasks.workunit.client.0.smithi062.stdout:5/386: mknod d8/d38/c81 0 2024-04-20T21:31:58.098 INFO:tasks.workunit.client.0.smithi062.stdout:8/396: write d7/d9/f12 [4425529,17451] 0 2024-04-20T21:31:58.101 INFO:tasks.workunit.client.1.smithi189.stdout:8/160: mkdir da/df/d20/d3e 0 2024-04-20T21:31:58.101 INFO:tasks.workunit.client.0.smithi062.stdout:9/379: creat d16/d2c/d5e/d75/d76/f81 x:0 0 0 2024-04-20T21:31:58.104 INFO:tasks.workunit.client.0.smithi062.stdout:4/409: creat da/dd/d15/d4b/f90 x:0 0 0 2024-04-20T21:31:58.105 INFO:tasks.workunit.client.0.smithi062.stdout:3/406: link d0/d1c/l6d d0/d1/d10/d31/d41/l90 0 2024-04-20T21:31:58.106 INFO:tasks.workunit.client.0.smithi062.stdout:7/369: mknod d7/d3e/d33/d3d/c7c 0 2024-04-20T21:31:58.108 INFO:tasks.workunit.client.0.smithi062.stdout:8/397: mknod d7/d25/d76/c87 0 2024-04-20T21:31:58.109 INFO:tasks.workunit.client.0.smithi062.stdout:5/387: write d8/d1e/d2c/f34 [1116081,115901] 0 2024-04-20T21:31:58.110 INFO:tasks.workunit.client.1.smithi189.stdout:8/161: link da/df/d20/l2e da/df/d13/l3f 0 2024-04-20T21:31:58.114 INFO:tasks.workunit.client.0.smithi062.stdout:2/381: dwrite d7/da/d28/d4b/f73 [0,4194304] 0 2024-04-20T21:31:58.114 INFO:tasks.workunit.client.0.smithi062.stdout:1/419: getdents dc/d27/d32 0 2024-04-20T21:31:58.119 INFO:tasks.workunit.client.0.smithi062.stdout:4/410: mkdir da/dd/df/d27/d7a/d91 0 2024-04-20T21:31:58.119 INFO:tasks.workunit.client.0.smithi062.stdout:2/382: stat d7/da/d15/d2f/l63 0 2024-04-20T21:31:58.119 INFO:tasks.workunit.client.0.smithi062.stdout:9/380: truncate d16/f2e 3289048 0 2024-04-20T21:31:58.125 INFO:tasks.workunit.client.0.smithi062.stdout:5/388: unlink d8/d10/f28 0 2024-04-20T21:31:58.125 INFO:tasks.workunit.client.0.smithi062.stdout:1/420: creat dc/d2d/d7c/f8d x:0 0 0 2024-04-20T21:31:58.126 INFO:tasks.workunit.client.0.smithi062.stdout:9/381: readlink d16/l37 0 2024-04-20T21:31:58.127 INFO:tasks.workunit.client.0.smithi062.stdout:1/421: chown dc/d15/f18 840011 1 2024-04-20T21:31:58.128 INFO:tasks.workunit.client.0.smithi062.stdout:2/383: chown d7/da/d15/d1b/d3d/f52 700 1 2024-04-20T21:31:58.130 INFO:tasks.workunit.client.0.smithi062.stdout:9/382: chown d16/d3e/l5c 16 1 2024-04-20T21:31:58.133 INFO:tasks.workunit.client.0.smithi062.stdout:9/383: fsync d16/f66 0 2024-04-20T21:31:58.136 INFO:tasks.workunit.client.0.smithi062.stdout:1/422: write dc/d27/d32/f51 [116531,3516] 0 2024-04-20T21:31:58.146 INFO:tasks.workunit.client.0.smithi062.stdout:1/423: symlink dc/d15/d75/l8e 0 2024-04-20T21:31:58.153 INFO:tasks.workunit.client.1.smithi189.stdout:8/162: dwrite da/df/f1b [4194304,4194304] 0 2024-04-20T21:31:58.154 INFO:tasks.workunit.client.0.smithi062.stdout:7/370: dwrite d7/d3e/d33/d3d/f75 [0,4194304] 0 2024-04-20T21:31:58.161 INFO:tasks.workunit.client.0.smithi062.stdout:4/411: dwrite f4 [0,4194304] 0 2024-04-20T21:31:58.161 INFO:tasks.workunit.client.0.smithi062.stdout:1/424: truncate dc/d27/d32/f74 108291 0 2024-04-20T21:31:58.163 INFO:tasks.workunit.client.0.smithi062.stdout:8/398: dwrite d7/d9/d1b/d24/f75 [0,4194304] 0 2024-04-20T21:31:58.163 INFO:tasks.workunit.client.0.smithi062.stdout:5/389: truncate d8/d10/d3e/f62 3454049 0 2024-04-20T21:31:58.163 INFO:tasks.workunit.client.0.smithi062.stdout:2/384: rename d7/da/d28/d4b/f69 to d7/da/f76 0 2024-04-20T21:31:58.168 INFO:tasks.workunit.client.0.smithi062.stdout:3/407: dread d0/d11/f56 [0,4194304] 0 2024-04-20T21:31:58.174 INFO:tasks.workunit.client.0.smithi062.stdout:1/425: dread dc/d15/d17/d36/d3a/f59 [0,4194304] 0 2024-04-20T21:31:58.177 INFO:tasks.workunit.client.0.smithi062.stdout:8/399: fdatasync d7/d9/d1b/f1c 0 2024-04-20T21:31:58.178 INFO:tasks.workunit.client.0.smithi062.stdout:4/412: mknod da/dd/d15/d50/d64/d3d/c92 0 2024-04-20T21:31:58.183 INFO:tasks.workunit.client.0.smithi062.stdout:9/384: dwrite d16/d2c/d5e/d75/d76/f81 [0,4194304] 0 2024-04-20T21:31:58.186 INFO:tasks.workunit.client.0.smithi062.stdout:3/408: write d0/d1/d3c/d43/d72/f89 [647649,71329] 0 2024-04-20T21:31:58.194 INFO:tasks.workunit.client.0.smithi062.stdout:8/400: truncate d7/d9/d1b/f1e 3811595 0 2024-04-20T21:31:58.194 INFO:tasks.workunit.client.0.smithi062.stdout:4/413: write da/dd/d15/d50/d64/d57/d5b/f6f [54920,84127] 0 2024-04-20T21:31:58.195 INFO:tasks.workunit.client.1.smithi189.stdout:8/163: dwrite da/f39 [0,4194304] 0 2024-04-20T21:31:58.196 INFO:tasks.workunit.client.0.smithi062.stdout:1/426: rename dc/d15/d17/d36/l87 to dc/d2d/d84/l8f 0 2024-04-20T21:31:58.200 INFO:tasks.workunit.client.0.smithi062.stdout:8/401: write d7/d30/f3c [5061803,6091] 0 2024-04-20T21:31:58.201 INFO:tasks.workunit.client.1.smithi189.stdout:8/164: symlink da/d1f/d2a/l40 0 2024-04-20T21:31:58.204 INFO:tasks.workunit.client.0.smithi062.stdout:4/414: truncate da/dd/f78 393938 0 2024-04-20T21:31:58.207 INFO:tasks.workunit.client.0.smithi062.stdout:7/371: dread d7/d3e/d33/d3d/d5e/f61 [0,4194304] 0 2024-04-20T21:31:58.207 INFO:tasks.workunit.client.1.smithi189.stdout:8/165: dread da/df/f23 [0,4194304] 0 2024-04-20T21:31:58.208 INFO:tasks.workunit.client.0.smithi062.stdout:2/385: dread d7/da/f13 [0,4194304] 0 2024-04-20T21:31:58.212 INFO:tasks.workunit.client.1.smithi189.stdout:8/166: dread da/df/d13/f18 [0,4194304] 0 2024-04-20T21:31:58.212 INFO:tasks.workunit.client.0.smithi062.stdout:8/402: mknod d7/d25/d3b/d58/c88 0 2024-04-20T21:31:58.217 INFO:tasks.workunit.client.0.smithi062.stdout:7/372: creat d7/d3e/d33/d3d/d4d/d54/d4f/f7d x:0 0 0 2024-04-20T21:31:58.220 INFO:tasks.workunit.client.0.smithi062.stdout:8/403: stat d7/d9/d29/f2b 0 2024-04-20T21:31:58.220 INFO:tasks.workunit.client.0.smithi062.stdout:2/386: read d7/da/f4d [2632959,4310] 0 2024-04-20T21:31:58.227 INFO:tasks.workunit.client.0.smithi062.stdout:7/373: unlink d7/d3e/l69 0 2024-04-20T21:31:58.231 INFO:tasks.workunit.client.0.smithi062.stdout:3/409: write d0/f8 [413102,123258] 0 2024-04-20T21:31:58.232 INFO:tasks.workunit.client.0.smithi062.stdout:9/385: dwrite f4 [0,4194304] 0 2024-04-20T21:31:58.238 INFO:tasks.workunit.client.0.smithi062.stdout:2/387: rmdir d7/da/d15/d2f/d38 39 2024-04-20T21:31:58.238 INFO:tasks.workunit.client.0.smithi062.stdout:8/404: read d7/d9/d1b/f1f [3089899,111726] 0 2024-04-20T21:31:58.243 INFO:tasks.workunit.client.0.smithi062.stdout:3/410: stat d0/d1c/d63/d42/d5b 0 2024-04-20T21:31:58.244 INFO:tasks.workunit.client.0.smithi062.stdout:4/415: dread da/dd/d15/d50/f86 [0,4194304] 0 2024-04-20T21:31:58.246 INFO:tasks.workunit.client.1.smithi189.stdout:8/167: dwrite f9 [0,4194304] 0 2024-04-20T21:31:58.252 INFO:tasks.workunit.client.0.smithi062.stdout:9/386: rename d16/d2c/l33 to d16/d3d/l82 0 2024-04-20T21:31:58.253 INFO:tasks.workunit.client.0.smithi062.stdout:1/427: dwrite dc/d15/d20/d28/f63 [0,4194304] 0 2024-04-20T21:31:58.254 INFO:tasks.workunit.client.0.smithi062.stdout:2/388: symlink d7/da/d28/l77 0 2024-04-20T21:31:58.255 INFO:tasks.workunit.client.0.smithi062.stdout:4/416: chown da/dd/d3c/c87 8697 1 2024-04-20T21:31:58.255 INFO:tasks.workunit.client.1.smithi189.stdout:8/168: symlink da/l41 0 2024-04-20T21:31:58.259 INFO:tasks.workunit.client.0.smithi062.stdout:1/428: chown dc/d27/f62 3263809 1 2024-04-20T21:31:58.262 INFO:tasks.workunit.client.0.smithi062.stdout:9/387: mkdir d16/d2f/d5b/d83 0 2024-04-20T21:31:58.265 INFO:tasks.workunit.client.0.smithi062.stdout:8/405: creat d7/d9/f89 x:0 0 0 2024-04-20T21:31:58.265 INFO:tasks.workunit.client.0.smithi062.stdout:4/417: dread da/f83 [0,4194304] 0 2024-04-20T21:31:58.270 INFO:tasks.workunit.client.0.smithi062.stdout:1/429: creat dc/d70/f90 x:0 0 0 2024-04-20T21:31:58.275 INFO:tasks.workunit.client.0.smithi062.stdout:8/406: dread - d7/d9/d29/f2b zero size 2024-04-20T21:31:58.276 INFO:tasks.workunit.client.0.smithi062.stdout:2/389: chown d7/da/d15/d1b/d24/l4f 1048959 1 2024-04-20T21:31:58.276 INFO:tasks.workunit.client.0.smithi062.stdout:7/374: dwrite d7/f5f [0,4194304] 0 2024-04-20T21:31:58.279 INFO:tasks.workunit.client.0.smithi062.stdout:1/430: readlink dc/d15/d17/d36/l85 0 2024-04-20T21:31:58.280 INFO:tasks.workunit.client.0.smithi062.stdout:9/388: rename d16/d2f/d5b/d83 to d16/d3e/d77/d84 0 2024-04-20T21:31:58.282 INFO:tasks.workunit.client.0.smithi062.stdout:1/431: chown dc/c41 5021194 1 2024-04-20T21:31:58.284 INFO:tasks.workunit.client.0.smithi062.stdout:4/418: creat da/d56/d71/f93 x:0 0 0 2024-04-20T21:31:58.289 INFO:tasks.workunit.client.0.smithi062.stdout:7/375: mkdir d7/d3e/d33/d3d/d5e/d7e 0 2024-04-20T21:31:58.297 INFO:tasks.workunit.client.0.smithi062.stdout:2/390: truncate d7/da/d15/d2f/f33 880341 0 2024-04-20T21:31:58.298 INFO:tasks.workunit.client.0.smithi062.stdout:7/376: rmdir d7/d63/d78 0 2024-04-20T21:31:58.299 INFO:tasks.workunit.client.0.smithi062.stdout:3/411: dwrite d0/d1/d3c/d43/d67/f86 [0,4194304] 0 2024-04-20T21:31:58.300 INFO:tasks.workunit.client.0.smithi062.stdout:8/407: rename d7/d9/d1b/c2c to d7/d25/d3b/c8a 0 2024-04-20T21:31:58.301 INFO:tasks.workunit.client.0.smithi062.stdout:4/419: creat da/dd/d15/d50/d64/d57/d5b/f94 x:0 0 0 2024-04-20T21:31:58.302 INFO:tasks.workunit.client.0.smithi062.stdout:2/391: creat d7/da/d15/d1b/d3d/f78 x:0 0 0 2024-04-20T21:31:58.304 INFO:tasks.workunit.client.0.smithi062.stdout:7/377: mkdir d7/d3e/d33/d3d/d7f 0 2024-04-20T21:31:58.306 INFO:tasks.workunit.client.0.smithi062.stdout:9/389: rename d16/d2f to d16/d2f/d5b/d61/d85 22 2024-04-20T21:31:58.307 INFO:tasks.workunit.client.0.smithi062.stdout:4/420: write da/dd/df/d27/d3f/f6e [457686,81628] 0 2024-04-20T21:31:58.307 INFO:tasks.workunit.client.0.smithi062.stdout:7/378: fsync d7/d26/f29 0 2024-04-20T21:31:58.308 INFO:tasks.workunit.client.1.smithi189.stdout:8/169: truncate da/f39 2162396 0 2024-04-20T21:31:58.315 INFO:tasks.workunit.client.1.smithi189.stdout:8/170: dread da/d1f/f25 [0,4194304] 0 2024-04-20T21:31:58.316 INFO:tasks.workunit.client.0.smithi062.stdout:2/392: readlink d7/da/d15/d1b/d24/l35 0 2024-04-20T21:31:58.319 INFO:tasks.workunit.client.0.smithi062.stdout:2/393: creat d7/da/d28/f79 x:0 0 0 2024-04-20T21:31:58.319 INFO:tasks.workunit.client.0.smithi062.stdout:0/392: sync 2024-04-20T21:31:58.320 INFO:tasks.workunit.client.0.smithi062.stdout:6/272: sync 2024-04-20T21:31:58.321 INFO:tasks.workunit.client.0.smithi062.stdout:4/421: creat da/dd/d3c/f95 x:0 0 0 2024-04-20T21:31:58.322 INFO:tasks.workunit.client.0.smithi062.stdout:7/379: mkdir d7/d3e/d33/d3d/d5e/d7e/d80 0 2024-04-20T21:31:58.326 INFO:tasks.workunit.client.0.smithi062.stdout:8/408: rename d7/d9/d1b to d7/d25/d3b/d58/d8b 0 2024-04-20T21:31:58.328 INFO:tasks.workunit.client.0.smithi062.stdout:1/432: dwrite dc/d27/f62 [0,4194304] 0 2024-04-20T21:31:58.330 INFO:tasks.workunit.client.0.smithi062.stdout:4/422: read da/dd/f14 [916823,105439] 0 2024-04-20T21:31:58.333 INFO:tasks.workunit.client.0.smithi062.stdout:2/394: creat d7/da/d15/d1b/d24/f7a x:0 0 0 2024-04-20T21:31:58.333 INFO:tasks.workunit.client.0.smithi062.stdout:0/393: truncate d7/d1e/f67 138474 0 2024-04-20T21:31:58.334 INFO:tasks.workunit.client.1.smithi189.stdout:8/171: dwrite da/df/d13/f14 [0,4194304] 0 2024-04-20T21:31:58.337 INFO:tasks.workunit.client.0.smithi062.stdout:7/380: creat d7/d26/d74/f81 x:0 0 0 2024-04-20T21:31:58.337 INFO:tasks.workunit.client.0.smithi062.stdout:6/273: truncate d0/d4f/f57 737713 0 2024-04-20T21:31:58.341 INFO:tasks.workunit.client.0.smithi062.stdout:6/274: write d0/d9/f16 [233698,46076] 0 2024-04-20T21:31:58.344 INFO:tasks.workunit.client.0.smithi062.stdout:0/394: symlink d7/d1c/d15/d17/d40/d3c/l89 0 2024-04-20T21:31:58.344 INFO:tasks.workunit.client.0.smithi062.stdout:2/395: write d7/da/d15/f26 [809137,46159] 0 2024-04-20T21:31:58.344 INFO:tasks.workunit.client.0.smithi062.stdout:8/409: link d7/d25/d4f/l52 d7/d30/l8c 0 2024-04-20T21:31:58.347 INFO:tasks.workunit.client.1.smithi189.stdout:8/172: write da/df/d13/f14 [4558798,16330] 0 2024-04-20T21:31:58.349 INFO:tasks.workunit.client.0.smithi062.stdout:4/423: creat da/dd/d15/d4b/f96 x:0 0 0 2024-04-20T21:31:58.353 INFO:tasks.workunit.client.0.smithi062.stdout:0/395: write d7/d20/f5c [978481,21921] 0 2024-04-20T21:31:58.357 INFO:tasks.workunit.client.0.smithi062.stdout:8/410: chown d7/d25/c2f 3496433 1 2024-04-20T21:31:58.359 INFO:tasks.workunit.client.0.smithi062.stdout:3/412: dwrite d0/d1/d10/d3a/d5e/f8d [0,4194304] 0 2024-04-20T21:31:58.359 INFO:tasks.workunit.client.0.smithi062.stdout:7/381: write d7/d63/f7a [199639,98198] 0 2024-04-20T21:31:58.359 INFO:tasks.workunit.client.0.smithi062.stdout:6/275: rename d0/d9/f18 to d0/d1a/f59 0 2024-04-20T21:31:58.361 INFO:tasks.workunit.client.0.smithi062.stdout:9/390: dread d16/d2c/d42/f70 [0,4194304] 0 2024-04-20T21:31:58.365 INFO:tasks.workunit.client.1.smithi189.stdout:8/173: dwrite da/df/f1b [4194304,4194304] 0 2024-04-20T21:31:58.366 INFO:tasks.workunit.client.1.smithi189.stdout:8/174: fdatasync f9 0 2024-04-20T21:31:58.367 INFO:tasks.workunit.client.0.smithi062.stdout:4/424: mkdir da/dd/df/d27/d32/d97 0 2024-04-20T21:31:58.371 INFO:tasks.workunit.client.0.smithi062.stdout:2/396: fsync d7/da/d28/d4b/f6e 0 2024-04-20T21:31:58.372 INFO:tasks.workunit.client.0.smithi062.stdout:5/390: sync 2024-04-20T21:31:58.373 INFO:tasks.workunit.client.0.smithi062.stdout:8/411: write d7/d9/d29/f40 [1198256,120898] 0 2024-04-20T21:31:58.374 INFO:tasks.workunit.client.0.smithi062.stdout:3/413: write d0/d1/d10/f4e [1620311,84383] 0 2024-04-20T21:31:58.375 INFO:tasks.workunit.client.0.smithi062.stdout:9/391: symlink d16/d4e/l86 0 2024-04-20T21:31:58.382 INFO:tasks.workunit.client.0.smithi062.stdout:0/396: truncate d7/d1c/ff 1044967 0 2024-04-20T21:31:58.382 INFO:tasks.workunit.client.0.smithi062.stdout:8/412: write d7/d25/f7d [235529,61348] 0 2024-04-20T21:31:58.382 INFO:tasks.workunit.client.0.smithi062.stdout:5/391: dread - d8/d38/d7a/d4c/f54 zero size 2024-04-20T21:31:58.383 INFO:tasks.workunit.client.0.smithi062.stdout:1/433: dread dc/f14 [8388608,4194304] 0 2024-04-20T21:31:58.386 INFO:tasks.workunit.client.0.smithi062.stdout:8/413: chown d7/d25/l36 8986 1 2024-04-20T21:31:58.389 INFO:tasks.workunit.client.0.smithi062.stdout:3/414: truncate d0/f4c 4679641 0 2024-04-20T21:31:58.389 INFO:tasks.workunit.client.0.smithi062.stdout:5/392: chown d8/d10/d3e/l43 8629959 1 2024-04-20T21:31:58.393 INFO:tasks.workunit.client.0.smithi062.stdout:9/392: write d16/d3d/f43 [1776319,24133] 0 2024-04-20T21:31:58.394 INFO:tasks.workunit.client.0.smithi062.stdout:3/415: rmdir d0/d1b 39 2024-04-20T21:31:58.397 INFO:tasks.workunit.client.0.smithi062.stdout:3/416: chown d0/d1c/d63/d42/f73 812 1 2024-04-20T21:31:58.398 INFO:tasks.workunit.client.1.smithi189.stdout:8/175: write da/df/f23 [1179358,15986] 0 2024-04-20T21:31:58.400 INFO:tasks.workunit.client.0.smithi062.stdout:2/397: symlink d7/da/d15/d2f/d38/l7b 0 2024-04-20T21:31:58.402 INFO:tasks.workunit.client.0.smithi062.stdout:1/434: creat dc/d15/d75/f91 x:0 0 0 2024-04-20T21:31:58.412 INFO:tasks.workunit.client.0.smithi062.stdout:5/393: rmdir d8/d1e/d2c/d6c/d56/d50 39 2024-04-20T21:31:58.415 INFO:tasks.workunit.client.0.smithi062.stdout:2/398: mkdir d7/da/d15/d2f/d38/d6a/d7c 0 2024-04-20T21:31:58.416 INFO:tasks.workunit.client.0.smithi062.stdout:4/425: dwrite da/dd/d3c/f53 [0,4194304] 0 2024-04-20T21:31:58.417 INFO:tasks.workunit.client.0.smithi062.stdout:3/417: creat d0/d1c/d63/f91 x:0 0 0 2024-04-20T21:31:58.418 INFO:tasks.workunit.client.0.smithi062.stdout:8/414: dread d7/d25/d3b/d58/d8b/d24/f48 [0,4194304] 0 2024-04-20T21:31:58.421 INFO:tasks.workunit.client.0.smithi062.stdout:1/435: truncate dc/d15/d17/d36/d3a/f6a 662178 0 2024-04-20T21:31:58.423 INFO:tasks.workunit.client.0.smithi062.stdout:2/399: chown d7/da/d28/f6d 0 1 2024-04-20T21:31:58.425 INFO:tasks.workunit.client.0.smithi062.stdout:5/394: unlink d8/d10/l12 0 2024-04-20T21:31:58.428 INFO:tasks.workunit.client.0.smithi062.stdout:1/436: write dc/d15/f5b [19866,92105] 0 2024-04-20T21:31:58.436 INFO:tasks.workunit.client.0.smithi062.stdout:3/418: creat d0/d1/d10/d31/f92 x:0 0 0 2024-04-20T21:31:58.436 INFO:tasks.workunit.client.0.smithi062.stdout:4/426: mknod da/dd/d15/d50/d64/c98 0 2024-04-20T21:31:58.441 INFO:tasks.workunit.client.0.smithi062.stdout:4/427: chown da/dd/df/d27/d7a/d91 10746 1 2024-04-20T21:31:58.442 INFO:tasks.workunit.client.0.smithi062.stdout:7/382: dwrite d7/d3e/d33/d3d/d4d/d54/d4f/f53 [0,4194304] 0 2024-04-20T21:31:58.443 INFO:tasks.workunit.client.0.smithi062.stdout:2/400: chown d7/da/d15/c29 87 1 2024-04-20T21:31:58.443 INFO:tasks.workunit.client.0.smithi062.stdout:5/395: creat d8/d38/d7a/d4c/d7d/f82 x:0 0 0 2024-04-20T21:31:58.443 INFO:tasks.workunit.client.0.smithi062.stdout:1/437: write dc/d15/d17/d36/f54 [226219,107462] 0 2024-04-20T21:31:58.446 INFO:tasks.workunit.client.0.smithi062.stdout:3/419: mknod d0/d1c/d52/c93 0 2024-04-20T21:31:58.448 INFO:tasks.workunit.client.0.smithi062.stdout:9/393: dwrite d16/f66 [0,4194304] 0 2024-04-20T21:31:58.450 INFO:tasks.workunit.client.0.smithi062.stdout:2/401: creat d7/da/d15/d1b/f7d x:0 0 0 2024-04-20T21:31:58.451 INFO:tasks.workunit.client.0.smithi062.stdout:9/394: write f15 [1367244,54185] 0 2024-04-20T21:31:58.454 INFO:tasks.workunit.client.0.smithi062.stdout:1/438: mknod dc/d15/d20/d28/c92 0 2024-04-20T21:31:58.461 INFO:tasks.workunit.client.0.smithi062.stdout:5/396: symlink d8/d10/l83 0 2024-04-20T21:31:58.461 INFO:tasks.workunit.client.0.smithi062.stdout:8/415: dwrite d7/d25/d3b/d58/d8b/d24/d46/f64 [4194304,4194304] 0 2024-04-20T21:31:58.474 INFO:tasks.workunit.client.0.smithi062.stdout:5/397: dread - d8/d1e/d2c/f71 zero size 2024-04-20T21:31:58.475 INFO:tasks.workunit.client.0.smithi062.stdout:7/383: rename d7/l25 to d7/d3e/d33/d3d/d5e/d72/l82 0 2024-04-20T21:31:58.476 INFO:tasks.workunit.client.0.smithi062.stdout:2/402: creat d7/da/d15/d2f/f7e x:0 0 0 2024-04-20T21:31:58.476 INFO:tasks.workunit.client.0.smithi062.stdout:8/416: truncate d7/d25/f7d 402275 0 2024-04-20T21:31:58.482 INFO:tasks.workunit.client.0.smithi062.stdout:8/417: readlink d7/d25/d3b/d58/d8b/l35 0 2024-04-20T21:31:58.483 INFO:tasks.workunit.client.0.smithi062.stdout:1/439: dwrite dc/d15/d75/f91 [0,4194304] 0 2024-04-20T21:31:58.488 INFO:tasks.workunit.client.1.smithi189.stdout:2/7: sync 2024-04-20T21:31:58.488 INFO:tasks.workunit.client.1.smithi189.stdout:2/8: dwrite - no filename 2024-04-20T21:31:58.488 INFO:tasks.workunit.client.1.smithi189.stdout:2/9: read - no filename 2024-04-20T21:31:58.491 INFO:tasks.workunit.client.0.smithi062.stdout:9/395: dwrite d16/d3d/d55/f73 [0,4194304] 0 2024-04-20T21:31:58.493 INFO:tasks.workunit.client.1.smithi189.stdout:2/10: mkdir d1 0 2024-04-20T21:31:58.493 INFO:tasks.workunit.client.1.smithi189.stdout:2/11: read - no filename 2024-04-20T21:31:58.493 INFO:tasks.workunit.client.1.smithi189.stdout:2/12: dwrite - no filename 2024-04-20T21:31:58.493 INFO:tasks.workunit.client.1.smithi189.stdout:2/13: dread - no filename 2024-04-20T21:31:58.494 INFO:tasks.workunit.client.1.smithi189.stdout:8/176: truncate da/df/f11 2186787 0 2024-04-20T21:31:58.501 INFO:tasks.workunit.client.0.smithi062.stdout:5/398: mknod d8/d38/d7a/d4c/c84 0 2024-04-20T21:31:58.503 INFO:tasks.workunit.client.1.smithi189.stdout:2/14: creat d1/f2 x:0 0 0 2024-04-20T21:31:58.506 INFO:tasks.workunit.client.0.smithi062.stdout:7/384: read d7/f36 [147874,99798] 0 2024-04-20T21:31:58.506 INFO:tasks.workunit.client.0.smithi062.stdout:2/403: mknod d7/da/d15/d2f/d38/d6b/c7f 0 2024-04-20T21:31:58.507 INFO:tasks.workunit.client.0.smithi062.stdout:1/440: symlink dc/d15/d17/d36/d3a/l93 0 2024-04-20T21:31:58.510 INFO:tasks.workunit.client.1.smithi189.stdout:8/177: dread da/df/d13/f18 [0,4194304] 0 2024-04-20T21:31:58.510 INFO:tasks.workunit.client.0.smithi062.stdout:1/441: chown dc/fe 34732 1 2024-04-20T21:31:58.513 INFO:tasks.workunit.client.1.smithi189.stdout:8/178: write da/de/f10 [370341,56345] 0 2024-04-20T21:31:58.514 INFO:tasks.workunit.client.0.smithi062.stdout:8/418: mknod d7/d25/d3b/d58/d8b/c8d 0 2024-04-20T21:31:58.519 INFO:tasks.workunit.client.0.smithi062.stdout:6/276: dread d0/d1a/f59 [0,4194304] 0 2024-04-20T21:31:58.520 INFO:tasks.workunit.client.0.smithi062.stdout:7/385: write d7/d3e/d33/d3d/d4d/d54/f28 [2417102,5382] 0 2024-04-20T21:31:58.522 INFO:tasks.workunit.client.0.smithi062.stdout:2/404: rename d7/da/f76 to d7/da/d15/d1b/f80 0 2024-04-20T21:31:58.527 INFO:tasks.workunit.client.0.smithi062.stdout:9/396: getdents d16/d2f/d69 0 2024-04-20T21:31:58.530 INFO:tasks.workunit.client.0.smithi062.stdout:7/386: chown d7/l1b 1 1 2024-04-20T21:31:58.533 INFO:tasks.workunit.client.0.smithi062.stdout:6/277: symlink d0/d9/d11/d22/d26/l5a 0 2024-04-20T21:31:58.534 INFO:tasks.workunit.client.1.smithi189.stdout:2/15: getdents d1 0 2024-04-20T21:31:58.535 INFO:tasks.workunit.client.1.smithi189.stdout:2/16: chown d1 2 1 2024-04-20T21:31:58.536 INFO:tasks.workunit.client.0.smithi062.stdout:5/399: link d8/d10/c18 d8/d1e/d2c/c85 0 2024-04-20T21:31:58.538 INFO:tasks.workunit.client.1.smithi189.stdout:8/179: dwrite da/de/f10 [0,4194304] 0 2024-04-20T21:31:58.542 INFO:tasks.workunit.client.1.smithi189.stdout:2/17: mknod d1/c3 0 2024-04-20T21:31:58.542 INFO:tasks.workunit.client.0.smithi062.stdout:7/387: rename d7/d26/d74 to d7/d3e/d33/d3d/d4d/d83 0 2024-04-20T21:31:58.543 INFO:tasks.workunit.client.0.smithi062.stdout:5/400: read d8/d1e/f24 [3010367,53308] 0 2024-04-20T21:31:58.543 INFO:tasks.workunit.client.1.smithi189.stdout:8/180: truncate da/d1f/f2f 713271 0 2024-04-20T21:31:58.543 INFO:tasks.workunit.client.1.smithi189.stdout:2/18: chown d1 600 1 2024-04-20T21:31:58.543 INFO:tasks.workunit.client.0.smithi062.stdout:6/278: symlink d0/d1a/d44/l5b 0 2024-04-20T21:31:58.544 INFO:tasks.workunit.client.1.smithi189.stdout:2/19: chown d1/c3 3 1 2024-04-20T21:31:58.546 INFO:tasks.workunit.client.1.smithi189.stdout:2/20: write d1/f2 [886175,51428] 0 2024-04-20T21:31:58.546 INFO:tasks.workunit.client.0.smithi062.stdout:4/428: chown da/dd/d15/d50/d64 3 1 2024-04-20T21:31:58.550 INFO:tasks.workunit.client.1.smithi189.stdout:2/21: creat d1/f4 x:0 0 0 2024-04-20T21:31:58.556 INFO:tasks.workunit.client.0.smithi062.stdout:1/442: dread dc/d15/f18 [0,4194304] 0 2024-04-20T21:31:58.557 INFO:tasks.workunit.client.0.smithi062.stdout:7/388: mknod d7/d3e/d33/d3d/d5e/c84 0 2024-04-20T21:31:58.560 INFO:tasks.workunit.client.0.smithi062.stdout:4/429: rename da/c84 to da/dd/d3c/c99 0 2024-04-20T21:31:58.560 INFO:tasks.workunit.client.0.smithi062.stdout:5/401: truncate d8/d38/d7a/d4c/f54 514018 0 2024-04-20T21:31:58.560 INFO:tasks.workunit.client.0.smithi062.stdout:6/279: unlink d0/d9/d11/d22/d26/c32 0 2024-04-20T21:31:58.562 INFO:tasks.workunit.client.1.smithi189.stdout:8/181: dwrite da/df/d13/f14 [0,4194304] 0 2024-04-20T21:31:58.563 INFO:tasks.workunit.client.0.smithi062.stdout:5/402: chown d8/d1e/d2c/d6c/l45 90599 1 2024-04-20T21:31:58.566 INFO:tasks.workunit.client.1.smithi189.stdout:2/22: rmdir d1 39 2024-04-20T21:31:58.567 INFO:tasks.workunit.client.0.smithi062.stdout:6/280: write d0/fe [9127151,5093] 0 2024-04-20T21:31:58.569 INFO:tasks.workunit.client.0.smithi062.stdout:5/403: write d8/d1e/f24 [5217414,7349] 0 2024-04-20T21:31:58.569 INFO:tasks.workunit.client.0.smithi062.stdout:4/430: truncate da/dd/f69 5451050 0 2024-04-20T21:31:58.571 INFO:tasks.workunit.client.0.smithi062.stdout:1/443: write dc/d15/f18 [1374092,15455] 0 2024-04-20T21:31:58.574 INFO:tasks.workunit.client.0.smithi062.stdout:6/281: mkdir d0/d9/d11/d5c 0 2024-04-20T21:31:58.574 INFO:tasks.workunit.client.0.smithi062.stdout:5/404: readlink d8/d38/d7a/d4c/l57 0 2024-04-20T21:31:58.579 INFO:tasks.workunit.client.0.smithi062.stdout:4/431: read - da/dd/df/d27/d32/f49 zero size 2024-04-20T21:31:58.582 INFO:tasks.workunit.client.0.smithi062.stdout:1/444: rename dc/d15/d17 to dc/d15/d75/d94 0 2024-04-20T21:31:58.585 INFO:tasks.workunit.client.0.smithi062.stdout:4/432: symlink da/dd/df/d27/d32/l9a 0 2024-04-20T21:31:58.585 INFO:tasks.workunit.client.0.smithi062.stdout:5/405: creat d8/d1e/d2c/f86 x:0 0 0 2024-04-20T21:31:58.587 INFO:tasks.workunit.client.0.smithi062.stdout:6/282: getdents d0/d2c 0 2024-04-20T21:31:58.588 INFO:tasks.workunit.client.0.smithi062.stdout:6/283: fdatasync d0/f17 0 2024-04-20T21:31:58.589 INFO:tasks.workunit.client.1.smithi189.stdout:2/23: rmdir d1 39 2024-04-20T21:31:58.589 INFO:tasks.workunit.client.0.smithi062.stdout:5/406: chown d8/d38/d7a/d4c 63 1 2024-04-20T21:31:58.590 INFO:tasks.workunit.client.0.smithi062.stdout:1/445: symlink dc/d15/d75/d94/d36/d78/l95 0 2024-04-20T21:31:58.591 INFO:tasks.workunit.client.0.smithi062.stdout:7/389: dwrite d7/d3e/d33/d3d/d4d/d54/f28 [4194304,4194304] 0 2024-04-20T21:31:58.592 INFO:tasks.workunit.client.1.smithi189.stdout:2/24: write d1/f4 [957395,79342] 0 2024-04-20T21:31:58.596 INFO:tasks.workunit.client.0.smithi062.stdout:4/433: write da/dd/f11 [877844,14544] 0 2024-04-20T21:31:58.596 INFO:tasks.workunit.client.0.smithi062.stdout:6/284: rmdir d0/d2c 39 2024-04-20T21:31:58.600 INFO:tasks.workunit.client.0.smithi062.stdout:1/446: mkdir dc/d15/d75/d96 0 2024-04-20T21:31:58.600 INFO:tasks.workunit.client.0.smithi062.stdout:5/407: truncate d8/d10/f26 5091780 0 2024-04-20T21:31:58.601 INFO:tasks.workunit.client.1.smithi189.stdout:2/25: creat d1/f5 x:0 0 0 2024-04-20T21:31:58.601 INFO:tasks.workunit.client.0.smithi062.stdout:6/285: fsync d0/d1a/d44/f4b 0 2024-04-20T21:31:58.602 INFO:tasks.workunit.client.0.smithi062.stdout:9/397: dread fd [4194304,4194304] 0 2024-04-20T21:31:58.603 INFO:tasks.workunit.client.0.smithi062.stdout:7/390: mknod d7/c85 0 2024-04-20T21:31:58.604 INFO:tasks.workunit.client.0.smithi062.stdout:4/434: write da/f83 [188909,19209] 0 2024-04-20T21:31:58.605 INFO:tasks.workunit.client.0.smithi062.stdout:8/419: dread d7/d25/d76/f84 [0,4194304] 0 2024-04-20T21:31:58.607 INFO:tasks.workunit.client.0.smithi062.stdout:5/408: symlink d8/d38/l87 0 2024-04-20T21:31:58.608 INFO:tasks.workunit.client.0.smithi062.stdout:9/398: truncate d16/d2f/d69/f7e 123717 0 2024-04-20T21:31:58.611 INFO:tasks.workunit.client.0.smithi062.stdout:1/447: symlink dc/d15/d75/d96/l97 0 2024-04-20T21:31:58.611 INFO:tasks.workunit.client.0.smithi062.stdout:6/286: mknod d0/c5d 0 2024-04-20T21:31:58.613 INFO:tasks.workunit.client.0.smithi062.stdout:8/420: read d7/d25/d4f/f7e [3876316,93358] 0 2024-04-20T21:31:58.615 INFO:tasks.workunit.client.0.smithi062.stdout:7/391: rename d7/d63/f6d to d7/d3e/d33/d3d/d5e/d72/f86 0 2024-04-20T21:31:58.615 INFO:tasks.workunit.client.1.smithi189.stdout:2/26: rmdir d1 39 2024-04-20T21:31:58.617 INFO:tasks.workunit.client.0.smithi062.stdout:9/399: truncate d16/d2c/d5e/f7b 742792 0 2024-04-20T21:31:58.618 INFO:tasks.workunit.client.0.smithi062.stdout:1/448: dread - dc/d15/d20/f3e zero size 2024-04-20T21:31:58.620 INFO:tasks.workunit.client.0.smithi062.stdout:7/392: readlink d7/d3e/d33/d3d/d4d/d54/l5b 0 2024-04-20T21:31:58.621 INFO:tasks.workunit.client.0.smithi062.stdout:9/400: creat d16/d2c/d5e/f87 x:0 0 0 2024-04-20T21:31:58.622 INFO:tasks.workunit.client.0.smithi062.stdout:1/449: write dc/d27/d32/f51 [576414,76005] 0 2024-04-20T21:31:58.624 INFO:tasks.workunit.client.1.smithi189.stdout:8/182: dwrite da/df/f11 [0,4194304] 0 2024-04-20T21:31:58.626 INFO:tasks.workunit.client.0.smithi062.stdout:1/450: write dc/d70/f8c [1003902,130745] 0 2024-04-20T21:31:58.628 INFO:tasks.workunit.client.1.smithi189.stdout:2/27: dread d1/f4 [0,4194304] 0 2024-04-20T21:31:58.628 INFO:tasks.workunit.client.0.smithi062.stdout:6/287: dread d0/d9/d11/f25 [0,4194304] 0 2024-04-20T21:31:58.630 INFO:tasks.workunit.client.1.smithi189.stdout:2/28: truncate d1/f5 524394 0 2024-04-20T21:31:58.632 INFO:tasks.workunit.client.0.smithi062.stdout:0/397: sync 2024-04-20T21:31:58.635 INFO:tasks.workunit.client.0.smithi062.stdout:1/451: symlink dc/d15/d75/d94/d36/d78/l98 0 2024-04-20T21:31:58.637 INFO:tasks.workunit.client.1.smithi189.stdout:2/29: unlink d1/c3 0 2024-04-20T21:31:58.638 INFO:tasks.workunit.client.0.smithi062.stdout:1/452: stat dc/f14 0 2024-04-20T21:31:58.638 INFO:tasks.workunit.client.0.smithi062.stdout:9/401: link d16/d2c/d5e/f87 d16/d3d/f88 0 2024-04-20T21:31:58.639 INFO:tasks.workunit.client.0.smithi062.stdout:4/435: dwrite da/d56/d71/f93 [0,4194304] 0 2024-04-20T21:31:58.640 INFO:tasks.workunit.client.1.smithi189.stdout:8/183: symlink da/df/d20/d2c/l42 0 2024-04-20T21:31:58.641 INFO:tasks.workunit.client.0.smithi062.stdout:0/398: symlink d7/d20/d62/l8a 0 2024-04-20T21:31:58.642 INFO:tasks.workunit.client.0.smithi062.stdout:4/436: stat da/d4f/c58 0 2024-04-20T21:31:58.647 INFO:tasks.workunit.client.0.smithi062.stdout:6/288: link d0/d1a/f29 d0/f5e 0 2024-04-20T21:31:58.647 INFO:tasks.workunit.client.0.smithi062.stdout:7/393: dread d7/d3e/d33/d3d/f75 [0,4194304] 0 2024-04-20T21:31:58.648 INFO:tasks.workunit.client.0.smithi062.stdout:1/453: creat dc/d15/d75/d94/d6f/f99 x:0 0 0 2024-04-20T21:31:58.649 INFO:tasks.workunit.client.0.smithi062.stdout:4/437: symlink da/dd/df/d27/d3f/l9b 0 2024-04-20T21:31:58.653 INFO:tasks.workunit.client.0.smithi062.stdout:1/454: mkdir dc/d27/d9a 0 2024-04-20T21:31:58.657 INFO:tasks.workunit.client.0.smithi062.stdout:9/402: read f15 [456482,119850] 0 2024-04-20T21:31:58.658 INFO:tasks.workunit.client.0.smithi062.stdout:0/399: read d7/d1c/d15/d17/d37/f5d [2425877,19963] 0 2024-04-20T21:31:58.661 INFO:tasks.workunit.client.0.smithi062.stdout:9/403: fdatasync d16/d3d/f7a 0 2024-04-20T21:31:58.662 INFO:tasks.workunit.client.0.smithi062.stdout:8/421: dread d7/d9/d29/f40 [0,4194304] 0 2024-04-20T21:31:58.665 INFO:tasks.workunit.client.1.smithi189.stdout:8/184: dwrite da/df/d13/f18 [0,4194304] 0 2024-04-20T21:31:58.666 INFO:tasks.workunit.client.1.smithi189.stdout:8/185: chown da/l41 42376189 1 2024-04-20T21:31:58.668 INFO:tasks.workunit.client.1.smithi189.stdout:8/186: read da/df/f1b [4886317,99651] 0 2024-04-20T21:31:58.674 INFO:tasks.workunit.client.0.smithi062.stdout:7/394: dread d7/d3e/d33/f57 [0,4194304] 0 2024-04-20T21:31:58.688 INFO:tasks.workunit.client.1.smithi189.stdout:4/10: sync 2024-04-20T21:31:58.692 INFO:tasks.workunit.client.1.smithi189.stdout:4/11: creat d2/f4 x:0 0 0 2024-04-20T21:31:58.693 INFO:tasks.workunit.client.1.smithi189.stdout:8/187: dwrite da/df/d22/d33/f36 [0,4194304] 0 2024-04-20T21:31:58.695 INFO:tasks.workunit.client.1.smithi189.stdout:8/188: rmdir da/df/d22/d33 39 2024-04-20T21:31:58.720 INFO:tasks.workunit.client.0.smithi062.stdout:3/420: sync 2024-04-20T21:31:58.720 INFO:tasks.workunit.client.0.smithi062.stdout:2/405: sync 2024-04-20T21:31:58.736 INFO:tasks.workunit.client.0.smithi062.stdout:2/406: dwrite d7/da/d15/d1b/d3d/f49 [0,4194304] 0 2024-04-20T21:31:58.745 INFO:tasks.workunit.client.0.smithi062.stdout:7/395: symlink d7/d3e/d33/d3d/d5e/d7e/l87 0 2024-04-20T21:31:58.745 INFO:tasks.workunit.client.0.smithi062.stdout:2/407: unlink d7/da/l64 0 2024-04-20T21:31:58.747 INFO:tasks.workunit.client.0.smithi062.stdout:3/421: write d0/d1/d10/d3a/d5e/d65/d88/f8a [2427812,124110] 0 2024-04-20T21:31:58.748 INFO:tasks.workunit.client.0.smithi062.stdout:2/408: stat d7/da/l72 0 2024-04-20T21:31:58.751 INFO:tasks.workunit.client.0.smithi062.stdout:0/400: mknod d7/d1c/c8b 0 2024-04-20T21:31:58.755 INFO:tasks.workunit.client.0.smithi062.stdout:1/455: mknod dc/d15/d75/d94/c9b 0 2024-04-20T21:31:58.757 INFO:tasks.workunit.client.0.smithi062.stdout:9/404: fsync d16/f1d 0 2024-04-20T21:31:58.761 INFO:tasks.workunit.client.0.smithi062.stdout:8/422: rename d7/d25/d3b/d58/d8b/c39 to d7/d9/d29/c8e 0 2024-04-20T21:31:58.765 INFO:tasks.workunit.client.0.smithi062.stdout:6/289: getdents d0/d4f 0 2024-04-20T21:31:58.765 INFO:tasks.workunit.client.0.smithi062.stdout:3/422: dwrite d0/d1/d10/d3a/d5e/d65/d88/f8a [4194304,4194304] 0 2024-04-20T21:31:58.767 INFO:tasks.workunit.client.0.smithi062.stdout:0/401: fsync d7/d1e/f45 0 2024-04-20T21:31:58.776 INFO:tasks.workunit.client.0.smithi062.stdout:4/438: dwrite da/fb [0,4194304] 0 2024-04-20T21:31:58.777 INFO:tasks.workunit.client.0.smithi062.stdout:7/396: rename c2 to d7/d3e/d33/d3d/d5e/d7e/c88 0 2024-04-20T21:31:58.781 INFO:tasks.workunit.client.0.smithi062.stdout:6/290: symlink d0/d4f/l5f 0 2024-04-20T21:31:58.785 INFO:tasks.workunit.client.0.smithi062.stdout:3/423: symlink d0/d1/d10/d31/d41/l94 0 2024-04-20T21:31:58.791 INFO:tasks.workunit.client.0.smithi062.stdout:9/405: symlink d16/d4e/d80/l89 0 2024-04-20T21:31:58.792 INFO:tasks.workunit.client.0.smithi062.stdout:4/439: symlink da/l9c 0 2024-04-20T21:31:58.793 INFO:tasks.workunit.client.0.smithi062.stdout:8/423: symlink d7/d25/d3b/l8f 0 2024-04-20T21:31:58.795 INFO:tasks.workunit.client.0.smithi062.stdout:6/291: symlink d0/d9/d11/l60 0 2024-04-20T21:31:58.797 INFO:tasks.workunit.client.0.smithi062.stdout:3/424: readlink d0/d1c/l6d 0 2024-04-20T21:31:58.797 INFO:tasks.workunit.client.0.smithi062.stdout:8/424: dread - d7/d25/d3b/f79 zero size 2024-04-20T21:31:58.798 INFO:tasks.workunit.client.0.smithi062.stdout:0/402: symlink d7/d1c/l8c 0 2024-04-20T21:31:58.799 INFO:tasks.workunit.client.0.smithi062.stdout:9/406: unlink d16/d3d/d55/f5a 0 2024-04-20T21:31:58.802 INFO:tasks.workunit.client.0.smithi062.stdout:7/397: creat d7/d3e/d33/d3d/d7f/f89 x:0 0 0 2024-04-20T21:31:58.803 INFO:tasks.workunit.client.0.smithi062.stdout:6/292: mknod d0/d9/d11/c61 0 2024-04-20T21:31:58.808 INFO:tasks.workunit.client.0.smithi062.stdout:4/440: mkdir da/dd/d15/d50/d64/d57/d5b/d9d 0 2024-04-20T21:31:58.809 INFO:tasks.workunit.client.0.smithi062.stdout:1/456: dwrite fb [0,4194304] 0 2024-04-20T21:31:58.809 INFO:tasks.workunit.client.0.smithi062.stdout:3/425: rename d0/d1/d10/d3a to d0/d1/d10/d3a/d5e/d65/d95 22 2024-04-20T21:31:58.815 INFO:tasks.workunit.client.0.smithi062.stdout:7/398: truncate d7/d26/f2c 893253 0 2024-04-20T21:31:58.828 INFO:tasks.workunit.client.0.smithi062.stdout:5/409: sync 2024-04-20T21:31:58.836 INFO:tasks.workunit.client.0.smithi062.stdout:7/399: dread d7/fe [4194304,4194304] 0 2024-04-20T21:31:58.838 INFO:tasks.workunit.client.0.smithi062.stdout:8/425: dwrite d7/d25/d3b/d58/d8b/d24/f48 [0,4194304] 0 2024-04-20T21:31:58.841 INFO:tasks.workunit.client.0.smithi062.stdout:8/426: read - d7/d9/d29/f2b zero size 2024-04-20T21:31:58.849 INFO:tasks.workunit.client.1.smithi189.stdout:5/15: sync 2024-04-20T21:31:58.849 INFO:tasks.workunit.client.1.smithi189.stdout:9/20: sync 2024-04-20T21:31:58.852 INFO:tasks.workunit.client.0.smithi062.stdout:6/293: dread d0/d9/f16 [0,4194304] 0 2024-04-20T21:31:58.869 INFO:tasks.workunit.client.0.smithi062.stdout:1/457: rename dc/f14 to dc/d15/d75/d94/d36/f9c 0 2024-04-20T21:31:58.871 INFO:tasks.workunit.client.0.smithi062.stdout:9/407: link d16/d2c/l35 d16/l8a 0 2024-04-20T21:31:58.872 INFO:tasks.workunit.client.0.smithi062.stdout:1/458: write dc/fe [942601,92433] 0 2024-04-20T21:31:58.873 INFO:tasks.workunit.client.0.smithi062.stdout:4/441: mkdir da/d56/d71/d9e 0 2024-04-20T21:31:58.879 INFO:tasks.workunit.client.0.smithi062.stdout:8/427: mkdir d7/d30/d90 0 2024-04-20T21:31:58.880 INFO:tasks.workunit.client.0.smithi062.stdout:6/294: rmdir d0/d9/d11 39 2024-04-20T21:31:58.881 INFO:tasks.workunit.client.0.smithi062.stdout:0/403: getdents d7/d1c/d15/d17/d26 0 2024-04-20T21:31:58.883 INFO:tasks.workunit.client.0.smithi062.stdout:8/428: truncate d7/d9/f89 706839 0 2024-04-20T21:31:58.893 INFO:tasks.workunit.client.0.smithi062.stdout:9/408: dwrite d16/f36 [4194304,4194304] 0 2024-04-20T21:31:58.895 INFO:tasks.workunit.client.1.smithi189.stdout:9/21: getdents . 0 2024-04-20T21:31:58.897 INFO:tasks.workunit.client.0.smithi062.stdout:1/459: symlink dc/l9d 0 2024-04-20T21:31:58.897 INFO:tasks.workunit.client.1.smithi189.stdout:9/22: readlink l0 0 2024-04-20T21:31:58.904 INFO:tasks.workunit.client.0.smithi062.stdout:0/404: dwrite d7/d1e/f67 [0,4194304] 0 2024-04-20T21:31:58.904 INFO:tasks.workunit.client.0.smithi062.stdout:9/409: unlink d16/d2f/d5b/f78 0 2024-04-20T21:31:58.909 INFO:tasks.workunit.client.0.smithi062.stdout:8/429: dread d7/d9/f15 [0,4194304] 0 2024-04-20T21:31:58.914 INFO:tasks.workunit.client.0.smithi062.stdout:9/410: mknod d16/d3e/c8b 0 2024-04-20T21:31:58.914 INFO:tasks.workunit.client.0.smithi062.stdout:8/430: dread - d7/d25/d3b/f79 zero size 2024-04-20T21:31:58.920 INFO:tasks.workunit.client.0.smithi062.stdout:1/460: dwrite fb [4194304,4194304] 0 2024-04-20T21:31:58.921 INFO:tasks.workunit.client.0.smithi062.stdout:4/442: dwrite da/dd/d15/d50/d64/d3d/f7e [0,4194304] 0 2024-04-20T21:31:58.921 INFO:tasks.workunit.client.1.smithi189.stdout:9/23: dread f5 [0,4194304] 0 2024-04-20T21:31:58.921 INFO:tasks.workunit.client.0.smithi062.stdout:3/426: truncate d0/d1/d10/d3a/d5e/d65/d88/f8a 5009141 0 2024-04-20T21:31:58.923 INFO:tasks.workunit.client.0.smithi062.stdout:3/427: stat d0/l13 0 2024-04-20T21:31:58.925 INFO:tasks.workunit.client.0.smithi062.stdout:2/409: sync 2024-04-20T21:31:58.928 INFO:tasks.workunit.client.0.smithi062.stdout:9/411: mknod d16/d2f/d5b/c8c 0 2024-04-20T21:31:58.933 INFO:tasks.workunit.client.0.smithi062.stdout:8/431: write d7/d25/d3b/d58/d8b/d24/f67 [579012,103332] 0 2024-04-20T21:31:58.934 INFO:tasks.workunit.client.1.smithi189.stdout:9/24: dread f5 [0,4194304] 0 2024-04-20T21:31:58.936 INFO:tasks.workunit.client.0.smithi062.stdout:3/428: read d0/d1/d3c/d43/d72/f89 [26230,26489] 0 2024-04-20T21:31:58.939 INFO:tasks.workunit.client.0.smithi062.stdout:4/443: dread - da/dd/d15/d50/d64/d57/d5b/f8d zero size 2024-04-20T21:31:58.940 INFO:tasks.workunit.client.0.smithi062.stdout:7/400: dwrite d7/f36 [0,4194304] 0 2024-04-20T21:31:58.941 INFO:tasks.workunit.client.0.smithi062.stdout:3/429: chown d0/d1c/d63/d42/l69 56769 1 2024-04-20T21:31:58.948 INFO:tasks.workunit.client.0.smithi062.stdout:8/432: write d7/d25/d76/f84 [4592697,67399] 0 2024-04-20T21:31:58.950 INFO:tasks.workunit.client.0.smithi062.stdout:6/295: dread d0/f3 [0,4194304] 0 2024-04-20T21:31:58.951 INFO:tasks.workunit.client.1.smithi189.stdout:3/30: sync 2024-04-20T21:31:58.955 INFO:tasks.workunit.client.0.smithi062.stdout:7/401: rename d7/d3e/f44 to d7/d63/f8a 0 2024-04-20T21:31:58.955 INFO:tasks.workunit.client.0.smithi062.stdout:4/444: unlink da/dd/d15/d50/d64/d57/c67 0 2024-04-20T21:31:58.959 INFO:tasks.workunit.client.0.smithi062.stdout:8/433: write d7/d25/d3b/d58/d8b/d24/f48 [5173755,42497] 0 2024-04-20T21:31:58.963 INFO:tasks.workunit.client.0.smithi062.stdout:9/412: truncate d16/f1e 3814006 0 2024-04-20T21:31:58.964 INFO:tasks.workunit.client.0.smithi062.stdout:2/410: truncate d7/da/d15/d1b/f80 632463 0 2024-04-20T21:31:58.969 INFO:tasks.workunit.client.0.smithi062.stdout:7/402: read d7/d3e/d33/d3d/d4d/f5a [2687896,34055] 0 2024-04-20T21:31:58.970 INFO:tasks.workunit.client.0.smithi062.stdout:8/434: fdatasync d7/d25/d3b/d58/d8b/d24/f67 0 2024-04-20T21:31:58.971 INFO:tasks.workunit.client.0.smithi062.stdout:9/413: rename d16/c2b to d16/d3d/d55/d6e/d7d/c8d 0 2024-04-20T21:31:58.973 INFO:tasks.workunit.client.0.smithi062.stdout:3/430: dwrite d0/d1/f29 [0,4194304] 0 2024-04-20T21:31:58.973 INFO:tasks.workunit.client.0.smithi062.stdout:2/411: mknod d7/da/d28/c81 0 2024-04-20T21:31:58.978 INFO:tasks.workunit.client.0.smithi062.stdout:7/403: rename d7/d3e/d33 to d7/d3e/d33/d3d/d7f/d8b 22 2024-04-20T21:31:58.978 INFO:tasks.workunit.client.0.smithi062.stdout:8/435: fdatasync d7/d25/d3b/d58/d8b/d24/d46/f64 0 2024-04-20T21:31:58.978 INFO:tasks.workunit.client.0.smithi062.stdout:2/412: stat d7/da/d28/c81 0 2024-04-20T21:31:58.979 INFO:tasks.workunit.client.0.smithi062.stdout:8/436: chown d7/d9/l2e 135352739 1 2024-04-20T21:31:58.984 INFO:tasks.workunit.client.0.smithi062.stdout:4/445: getdents da/dd/d15/d50/d64 0 2024-04-20T21:31:58.985 INFO:tasks.workunit.client.0.smithi062.stdout:8/437: chown d7/l6c 0 1 2024-04-20T21:31:58.989 INFO:tasks.workunit.client.0.smithi062.stdout:2/413: rename d7/da to d7/da/d15/d2f/d38/d51/d82 22 2024-04-20T21:31:58.995 INFO:tasks.workunit.client.0.smithi062.stdout:7/404: creat d7/d3e/d33/d3d/d5e/d7e/d80/f8c x:0 0 0 2024-04-20T21:31:58.995 INFO:tasks.workunit.client.0.smithi062.stdout:6/296: getdents d0/d9/d11/d22/d26 0 2024-04-20T21:31:59.000 INFO:tasks.workunit.client.0.smithi062.stdout:6/297: chown d0/d1a/d44 5973 1 2024-04-20T21:31:59.001 INFO:tasks.workunit.client.0.smithi062.stdout:8/438: mknod d7/d25/d3b/d58/d8b/d24/d46/c91 0 2024-04-20T21:31:59.001 INFO:tasks.workunit.client.0.smithi062.stdout:7/405: stat d7/d3e/d33/d3d/d5e/d7e/l87 0 2024-04-20T21:31:59.006 INFO:tasks.workunit.client.0.smithi062.stdout:2/414: link d7/da/d28/f6d d7/f83 0 2024-04-20T21:31:59.008 INFO:tasks.workunit.client.0.smithi062.stdout:3/431: fsync d0/d1/f29 0 2024-04-20T21:31:59.010 INFO:tasks.workunit.client.0.smithi062.stdout:2/415: dread - d7/da/d15/d2f/d38/d51/f60 zero size 2024-04-20T21:31:59.010 INFO:tasks.workunit.client.0.smithi062.stdout:8/439: unlink d7/d25/d3b/d58/d8b/d24/c45 0 2024-04-20T21:31:59.013 INFO:tasks.workunit.client.0.smithi062.stdout:7/406: rename d7/d3e/d33/d3d/d4d/d54/d4f/f53 to d7/d63/f8d 0 2024-04-20T21:31:59.022 INFO:tasks.workunit.client.0.smithi062.stdout:9/414: dwrite d16/d2c/d5e/d75/d76/f81 [0,4194304] 0 2024-04-20T21:31:59.023 INFO:tasks.workunit.client.0.smithi062.stdout:7/407: creat d7/d3e/d33/d3d/f8e x:0 0 0 2024-04-20T21:31:59.030 INFO:tasks.workunit.client.0.smithi062.stdout:2/416: getdents d7/da/d15/d2f 0 2024-04-20T21:31:59.038 INFO:tasks.workunit.client.0.smithi062.stdout:9/415: chown d16/d2c/d5e 42980 1 2024-04-20T21:31:59.039 INFO:tasks.workunit.client.0.smithi062.stdout:7/408: mknod d7/d3e/d33/d3d/d5e/c8f 0 2024-04-20T21:31:59.040 INFO:tasks.workunit.client.0.smithi062.stdout:4/446: dwrite da/dd/df/d27/d32/f81 [0,4194304] 0 2024-04-20T21:31:59.044 INFO:tasks.workunit.client.0.smithi062.stdout:6/298: dwrite d0/d1a/f29 [0,4194304] 0 2024-04-20T21:31:59.048 INFO:tasks.workunit.client.1.smithi189.stdout:9/25: truncate f1 1075447 0 2024-04-20T21:31:59.053 INFO:tasks.workunit.client.0.smithi062.stdout:7/409: creat d7/d3e/d33/d3d/d5e/f90 x:0 0 0 2024-04-20T21:31:59.057 INFO:tasks.workunit.client.0.smithi062.stdout:6/299: fdatasync d0/d1a/f59 0 2024-04-20T21:31:59.059 INFO:tasks.workunit.client.0.smithi062.stdout:4/447: chown da/dd/d15/l76 7101095 1 2024-04-20T21:31:59.063 INFO:tasks.workunit.client.1.smithi189.stdout:7/47: sync 2024-04-20T21:31:59.063 INFO:tasks.workunit.client.1.smithi189.stdout:1/29: sync 2024-04-20T21:31:59.067 INFO:tasks.workunit.client.0.smithi062.stdout:9/416: dwrite d16/f32 [0,4194304] 0 2024-04-20T21:31:59.068 INFO:tasks.workunit.client.0.smithi062.stdout:6/300: symlink d0/d1a/l62 0 2024-04-20T21:31:59.069 INFO:tasks.workunit.client.1.smithi189.stdout:1/30: truncate f2 694869 0 2024-04-20T21:31:59.071 INFO:tasks.workunit.client.1.smithi189.stdout:9/26: dread f4 [0,4194304] 0 2024-04-20T21:31:59.071 INFO:tasks.workunit.client.1.smithi189.stdout:3/31: dwrite d2/f5 [0,4194304] 0 2024-04-20T21:31:59.073 INFO:tasks.workunit.client.0.smithi062.stdout:2/417: truncate d7/da/f13 2400396 0 2024-04-20T21:31:59.074 INFO:tasks.workunit.client.1.smithi189.stdout:7/48: rename d5/la to d5/l10 0 2024-04-20T21:31:59.075 INFO:tasks.workunit.client.1.smithi189.stdout:7/49: fdatasync d5/fe 0 2024-04-20T21:31:59.076 INFO:tasks.workunit.client.0.smithi062.stdout:4/448: link da/dd/df/f3e da/dd/df/d27/d7a/d91/f9f 0 2024-04-20T21:31:59.076 INFO:tasks.workunit.client.0.smithi062.stdout:8/440: dread d7/d25/d4f/f7e [0,4194304] 0 2024-04-20T21:31:59.079 INFO:tasks.workunit.client.1.smithi189.stdout:1/31: creat d8/fe x:0 0 0 2024-04-20T21:31:59.080 INFO:tasks.workunit.client.1.smithi189.stdout:1/32: dread - d8/fa zero size 2024-04-20T21:31:59.080 INFO:tasks.workunit.client.0.smithi062.stdout:6/301: rename d0/f6 to d0/d9/d11/f63 0 2024-04-20T21:31:59.082 INFO:tasks.workunit.client.1.smithi189.stdout:1/33: write f6 [850698,28495] 0 2024-04-20T21:31:59.083 INFO:tasks.workunit.client.1.smithi189.stdout:9/27: creat f6 x:0 0 0 2024-04-20T21:31:59.083 INFO:tasks.workunit.client.1.smithi189.stdout:1/34: fdatasync d8/fe 0 2024-04-20T21:31:59.084 INFO:tasks.workunit.client.1.smithi189.stdout:3/32: write d2/f6 [825952,90578] 0 2024-04-20T21:31:59.085 INFO:tasks.workunit.client.0.smithi062.stdout:4/449: getdents da/d56/d71/d9e 0 2024-04-20T21:31:59.086 INFO:tasks.workunit.client.0.smithi062.stdout:6/302: unlink d0/d1a/c54 0 2024-04-20T21:31:59.089 INFO:tasks.workunit.client.0.smithi062.stdout:8/441: creat d7/d30/d90/f92 x:0 0 0 2024-04-20T21:31:59.090 INFO:tasks.workunit.client.1.smithi189.stdout:9/28: creat f7 x:0 0 0 2024-04-20T21:31:59.090 INFO:tasks.workunit.client.0.smithi062.stdout:7/410: dwrite d7/d3e/f20 [0,4194304] 0 2024-04-20T21:31:59.091 INFO:tasks.workunit.client.1.smithi189.stdout:7/50: link d5/ff d5/f11 0 2024-04-20T21:31:59.091 INFO:tasks.workunit.client.0.smithi062.stdout:6/303: truncate d0/f5e 4925886 0 2024-04-20T21:31:59.094 INFO:tasks.workunit.client.0.smithi062.stdout:4/450: write da/dd/df/d27/d3f/d82/f8b [1025908,114449] 0 2024-04-20T21:31:59.096 INFO:tasks.workunit.client.1.smithi189.stdout:9/29: creat f8 x:0 0 0 2024-04-20T21:31:59.096 INFO:tasks.workunit.client.1.smithi189.stdout:7/51: symlink d5/l12 0 2024-04-20T21:31:59.098 INFO:tasks.workunit.client.1.smithi189.stdout:1/35: dread d8/f9 [0,4194304] 0 2024-04-20T21:31:59.099 INFO:tasks.workunit.client.1.smithi189.stdout:0/144: sync 2024-04-20T21:31:59.099 INFO:tasks.workunit.client.1.smithi189.stdout:2/30: sync 2024-04-20T21:31:59.099 INFO:tasks.workunit.client.1.smithi189.stdout:6/41: sync 2024-04-20T21:31:59.099 INFO:tasks.workunit.client.1.smithi189.stdout:8/189: sync 2024-04-20T21:31:59.099 INFO:tasks.workunit.client.1.smithi189.stdout:4/12: sync 2024-04-20T21:31:59.099 INFO:tasks.workunit.client.1.smithi189.stdout:5/16: sync 2024-04-20T21:31:59.101 INFO:tasks.workunit.client.1.smithi189.stdout:5/17: chown l0 2973677 1 2024-04-20T21:31:59.101 INFO:tasks.workunit.client.1.smithi189.stdout:6/42: dread - d8/da/fb zero size 2024-04-20T21:31:59.103 INFO:tasks.workunit.client.1.smithi189.stdout:6/43: write f7 [6536440,36483] 0 2024-04-20T21:31:59.103 INFO:tasks.workunit.client.0.smithi062.stdout:6/304: write d0/f17 [1780929,95301] 0 2024-04-20T21:31:59.104 INFO:tasks.workunit.client.0.smithi062.stdout:4/451: write da/dd/d15/d50/d64/d3d/f5e [460741,123530] 0 2024-04-20T21:31:59.106 INFO:tasks.workunit.client.0.smithi062.stdout:7/411: write d7/d3e/d33/d3d/f4a [1319973,52196] 0 2024-04-20T21:31:59.106 INFO:tasks.workunit.client.1.smithi189.stdout:3/33: link l1 d2/d8/la 0 2024-04-20T21:31:59.108 INFO:tasks.workunit.client.0.smithi062.stdout:8/442: fdatasync d7/d25/d3b/d58/d8b/f3f 0 2024-04-20T21:31:59.108 INFO:tasks.workunit.client.1.smithi189.stdout:3/34: chown d2 60067 1 2024-04-20T21:31:59.113 INFO:tasks.workunit.client.0.smithi062.stdout:2/418: dwrite d7/d39/f55 [0,4194304] 0 2024-04-20T21:31:59.113 INFO:tasks.workunit.client.1.smithi189.stdout:7/52: creat d5/f13 x:0 0 0 2024-04-20T21:31:59.115 INFO:tasks.workunit.client.0.smithi062.stdout:5/410: sync 2024-04-20T21:31:59.116 INFO:tasks.workunit.client.1.smithi189.stdout:4/13: rmdir d2 39 2024-04-20T21:31:59.118 INFO:tasks.workunit.client.1.smithi189.stdout:2/31: rmdir d1 39 2024-04-20T21:31:59.119 INFO:tasks.workunit.client.1.smithi189.stdout:7/53: chown d5/f6 35903 1 2024-04-20T21:31:59.121 INFO:tasks.workunit.client.0.smithi062.stdout:4/452: unlink da/dd/d15/d50/d64/d3d/c92 0 2024-04-20T21:31:59.121 INFO:tasks.workunit.client.0.smithi062.stdout:7/412: rename d7/d3e/l6b to d7/d26/l91 0 2024-04-20T21:31:59.121 INFO:tasks.workunit.client.0.smithi062.stdout:2/419: chown d7/da/d15/l40 5042 1 2024-04-20T21:31:59.123 INFO:tasks.workunit.client.0.smithi062.stdout:8/443: creat d7/d25/f93 x:0 0 0 2024-04-20T21:31:59.126 INFO:tasks.workunit.client.1.smithi189.stdout:9/30: dwrite f6 [0,4194304] 0 2024-04-20T21:31:59.128 INFO:tasks.workunit.client.0.smithi062.stdout:2/420: chown d7/da/d15/d1b/l5e 862813 1 2024-04-20T21:31:59.128 INFO:tasks.workunit.client.0.smithi062.stdout:4/453: creat da/dd/d15/d50/fa0 x:0 0 0 2024-04-20T21:31:59.128 INFO:tasks.workunit.client.1.smithi189.stdout:3/35: readlink d2/d3/l4 0 2024-04-20T21:31:59.131 INFO:tasks.workunit.client.1.smithi189.stdout:4/14: dread - d2/f4 zero size 2024-04-20T21:31:59.131 INFO:tasks.workunit.client.0.smithi062.stdout:7/413: rmdir d7/d3e/d33/d3d 39 2024-04-20T21:31:59.131 INFO:tasks.workunit.client.1.smithi189.stdout:0/145: dwrite d3/d26/f1d [4194304,4194304] 0 2024-04-20T21:31:59.137 INFO:tasks.workunit.client.0.smithi062.stdout:5/411: mkdir d8/d1e/d2c/d6c/d56/d50/d88 0 2024-04-20T21:31:59.138 INFO:tasks.workunit.client.0.smithi062.stdout:4/454: mkdir da/dd/d15/d50/d64/d3d/da1 0 2024-04-20T21:31:59.143 INFO:tasks.workunit.client.0.smithi062.stdout:2/421: mkdir d7/da/d15/d1b/d3d/d84 0 2024-04-20T21:31:59.145 INFO:tasks.workunit.client.0.smithi062.stdout:5/412: rmdir d8/d10/d3e 39 2024-04-20T21:31:59.147 INFO:tasks.workunit.client.0.smithi062.stdout:5/413: dread - d8/d1e/d2c/f86 zero size 2024-04-20T21:31:59.149 INFO:tasks.workunit.client.1.smithi189.stdout:4/15: symlink d2/l5 0 2024-04-20T21:31:59.150 INFO:tasks.workunit.client.1.smithi189.stdout:8/190: creat da/df/d20/d3e/f43 x:0 0 0 2024-04-20T21:31:59.150 INFO:tasks.workunit.client.0.smithi062.stdout:6/305: dwrite d0/d9/d11/f1f [0,4194304] 0 2024-04-20T21:31:59.153 INFO:tasks.workunit.client.1.smithi189.stdout:4/16: truncate d2/f4 364305 0 2024-04-20T21:31:59.154 INFO:tasks.workunit.client.1.smithi189.stdout:4/17: stat l1 0 2024-04-20T21:31:59.156 INFO:tasks.workunit.client.1.smithi189.stdout:2/32: dwrite d1/f2 [0,4194304] 0 2024-04-20T21:31:59.156 INFO:tasks.workunit.client.0.smithi062.stdout:6/306: symlink d0/d9/d11/l64 0 2024-04-20T21:31:59.163 INFO:tasks.workunit.client.0.smithi062.stdout:5/414: dread d8/d1e/d2c/d6c/d56/f6b [0,4194304] 0 2024-04-20T21:31:59.164 INFO:tasks.workunit.client.0.smithi062.stdout:6/307: chown d0/d2c/f30 129 1 2024-04-20T21:31:59.164 INFO:tasks.workunit.client.1.smithi189.stdout:3/36: creat d2/d8/fb x:0 0 0 2024-04-20T21:31:59.166 INFO:tasks.workunit.client.1.smithi189.stdout:8/191: fsync da/df/f1b 0 2024-04-20T21:31:59.170 INFO:tasks.workunit.client.0.smithi062.stdout:5/415: creat d8/d38/f89 x:0 0 0 2024-04-20T21:31:59.171 INFO:tasks.workunit.client.0.smithi062.stdout:2/422: dread d7/da/d15/d1b/f34 [0,4194304] 0 2024-04-20T21:31:59.184 INFO:tasks.workunit.client.0.smithi062.stdout:6/308: getdents d0/d4f 0 2024-04-20T21:31:59.185 INFO:tasks.workunit.client.1.smithi189.stdout:0/146: link d3/d26/d24/c19 d3/df/d28/c33 0 2024-04-20T21:31:59.187 INFO:tasks.workunit.client.0.smithi062.stdout:2/423: creat d7/da/d15/f85 x:0 0 0 2024-04-20T21:31:59.189 INFO:tasks.workunit.client.0.smithi062.stdout:8/444: truncate d7/d9/f17 237613 0 2024-04-20T21:31:59.190 INFO:tasks.workunit.client.1.smithi189.stdout:7/54: dwrite d5/f6 [0,4194304] 0 2024-04-20T21:31:59.193 INFO:tasks.workunit.client.0.smithi062.stdout:7/414: dwrite f4 [0,4194304] 0 2024-04-20T21:31:59.196 INFO:tasks.workunit.client.0.smithi062.stdout:7/415: readlink d7/l11 0 2024-04-20T21:31:59.196 INFO:tasks.workunit.client.0.smithi062.stdout:5/416: dwrite d8/d1e/d2c/d6c/d56/f39 [0,4194304] 0 2024-04-20T21:31:59.196 INFO:tasks.workunit.client.1.smithi189.stdout:5/18: truncate f1 1902884 0 2024-04-20T21:31:59.197 INFO:tasks.workunit.client.0.smithi062.stdout:6/309: truncate d0/d1a/f3a 3416689 0 2024-04-20T21:31:59.197 INFO:tasks.workunit.client.1.smithi189.stdout:9/31: write f6 [4924551,67265] 0 2024-04-20T21:31:59.198 INFO:tasks.workunit.client.0.smithi062.stdout:8/445: creat d7/d25/d5c/f94 x:0 0 0 2024-04-20T21:31:59.199 INFO:tasks.workunit.client.1.smithi189.stdout:7/55: creat d5/f14 x:0 0 0 2024-04-20T21:31:59.200 INFO:tasks.workunit.client.0.smithi062.stdout:7/416: stat d7/d3e/d33/f76 0 2024-04-20T21:31:59.200 INFO:tasks.workunit.client.0.smithi062.stdout:4/455: dread da/dd/df/d27/f2c [0,4194304] 0 2024-04-20T21:31:59.201 INFO:tasks.workunit.client.1.smithi189.stdout:9/32: chown f6 57577375 1 2024-04-20T21:31:59.201 INFO:tasks.workunit.client.1.smithi189.stdout:7/56: readlink d5/l12 0 2024-04-20T21:31:59.203 INFO:tasks.workunit.client.1.smithi189.stdout:4/18: dwrite f0 [0,4194304] 0 2024-04-20T21:31:59.204 INFO:tasks.workunit.client.0.smithi062.stdout:5/417: write d8/d1e/d2c/f86 [206182,29917] 0 2024-04-20T21:31:59.206 INFO:tasks.workunit.client.1.smithi189.stdout:8/192: rename da/df/d20/d3e to da/de/d1e/d21/d44 0 2024-04-20T21:31:59.206 INFO:tasks.workunit.client.1.smithi189.stdout:4/19: chown d2 0 1 2024-04-20T21:31:59.206 INFO:tasks.workunit.client.1.smithi189.stdout:0/147: rename d3/d26 to d3/d26/d24/d34 22 2024-04-20T21:31:59.207 INFO:tasks.workunit.client.0.smithi062.stdout:8/446: creat d7/d30/f95 x:0 0 0 2024-04-20T21:31:59.208 INFO:tasks.workunit.client.1.smithi189.stdout:4/20: write f0 [2067038,82830] 0 2024-04-20T21:31:59.209 INFO:tasks.workunit.client.1.smithi189.stdout:8/193: write da/df/f19 [2884294,125970] 0 2024-04-20T21:31:59.210 INFO:tasks.workunit.client.1.smithi189.stdout:9/33: mkdir d9 0 2024-04-20T21:31:59.213 INFO:tasks.workunit.client.1.smithi189.stdout:9/34: write f6 [2036639,66487] 0 2024-04-20T21:31:59.213 INFO:tasks.workunit.client.1.smithi189.stdout:8/194: write da/df/d13/f18 [1333602,8489] 0 2024-04-20T21:31:59.214 INFO:tasks.workunit.client.0.smithi062.stdout:7/417: mknod d7/d3e/d33/d3d/d4d/d83/c92 0 2024-04-20T21:31:59.216 INFO:tasks.workunit.client.0.smithi062.stdout:0/405: sync 2024-04-20T21:31:59.223 INFO:tasks.workunit.client.0.smithi062.stdout:6/310: getdents d0/d1a/d44 0 2024-04-20T21:31:59.224 INFO:tasks.workunit.client.0.smithi062.stdout:7/418: readlink d7/l17 0 2024-04-20T21:31:59.226 INFO:tasks.workunit.client.0.smithi062.stdout:8/447: mkdir d7/d25/d3b/d58/d8b/d96 0 2024-04-20T21:31:59.229 INFO:tasks.workunit.client.0.smithi062.stdout:7/419: read - d7/d3e/d33/d3d/d5e/f90 zero size 2024-04-20T21:31:59.229 INFO:tasks.workunit.client.0.smithi062.stdout:8/448: write d7/d30/f3c [4685035,63545] 0 2024-04-20T21:31:59.231 INFO:tasks.workunit.client.1.smithi189.stdout:5/19: rmdir d3 0 2024-04-20T21:31:59.231 INFO:tasks.workunit.client.1.smithi189.stdout:4/21: rename d2/l3 to d2/l6 0 2024-04-20T21:31:59.232 INFO:tasks.workunit.client.0.smithi062.stdout:7/420: stat d7/d3e/d33/d3d/d5e/d72 0 2024-04-20T21:31:59.232 INFO:tasks.workunit.client.1.smithi189.stdout:3/37: truncate d2/f5 1208538 0 2024-04-20T21:31:59.235 INFO:tasks.workunit.client.0.smithi062.stdout:6/311: creat d0/d9/d11/f65 x:0 0 0 2024-04-20T21:31:59.237 INFO:tasks.workunit.client.0.smithi062.stdout:0/406: truncate d7/d20/d3a/f87 530954 0 2024-04-20T21:31:59.237 INFO:tasks.workunit.client.1.smithi189.stdout:8/195: unlink da/df/f1b 0 2024-04-20T21:31:59.238 INFO:tasks.workunit.client.1.smithi189.stdout:0/148: mkdir d3/d26/d24/d29/d35 0 2024-04-20T21:31:59.241 INFO:tasks.workunit.client.1.smithi189.stdout:0/149: dread d3/d26/d24/f22 [0,4194304] 0 2024-04-20T21:31:59.242 INFO:tasks.workunit.client.1.smithi189.stdout:5/20: mkdir d4 0 2024-04-20T21:31:59.242 INFO:tasks.workunit.client.1.smithi189.stdout:0/150: fdatasync d3/fb 0 2024-04-20T21:31:59.243 INFO:tasks.workunit.client.0.smithi062.stdout:8/449: link d7/d25/d3b/f79 d7/d25/d4f/f97 0 2024-04-20T21:31:59.244 INFO:tasks.workunit.client.1.smithi189.stdout:3/38: mknod d2/d8/cc 0 2024-04-20T21:31:59.245 INFO:tasks.workunit.client.0.smithi062.stdout:8/450: chown d7/c3d 106456 1 2024-04-20T21:31:59.248 INFO:tasks.workunit.client.0.smithi062.stdout:8/451: dread - d7/d30/d90/f92 zero size 2024-04-20T21:31:59.248 INFO:tasks.workunit.client.1.smithi189.stdout:9/35: mkdir d9/da 0 2024-04-20T21:31:59.252 INFO:tasks.workunit.client.0.smithi062.stdout:2/424: dread d7/f5d [0,4194304] 0 2024-04-20T21:31:59.253 INFO:tasks.workunit.client.1.smithi189.stdout:8/196: unlink da/d1f/f24 0 2024-04-20T21:31:59.258 INFO:tasks.workunit.client.0.smithi062.stdout:5/418: write d8/fb [2405793,82042] 0 2024-04-20T21:31:59.258 INFO:tasks.workunit.client.0.smithi062.stdout:8/452: rename d7/d9/f15 to d7/d25/d5c/f98 0 2024-04-20T21:31:59.260 INFO:tasks.workunit.client.0.smithi062.stdout:2/425: creat d7/d39/f86 x:0 0 0 2024-04-20T21:31:59.269 INFO:tasks.workunit.client.0.smithi062.stdout:2/426: write d7/f56 [722562,62970] 0 2024-04-20T21:31:59.270 INFO:tasks.workunit.client.0.smithi062.stdout:8/453: symlink d7/d25/d3b/l99 0 2024-04-20T21:31:59.271 INFO:tasks.workunit.client.0.smithi062.stdout:5/419: mkdir d8/d10/d3e/d8a 0 2024-04-20T21:31:59.272 INFO:tasks.workunit.client.0.smithi062.stdout:6/312: dwrite d0/d2c/f2e [0,4194304] 0 2024-04-20T21:31:59.273 INFO:tasks.workunit.client.1.smithi189.stdout:0/151: rename d3/df/f1b to d3/df/d28/d2b/f36 0 2024-04-20T21:31:59.276 INFO:tasks.workunit.client.0.smithi062.stdout:4/456: dwrite da/dd/f19 [0,4194304] 0 2024-04-20T21:31:59.277 INFO:tasks.workunit.client.0.smithi062.stdout:2/427: fsync d7/da/d28/d4b/f6e 0 2024-04-20T21:31:59.278 INFO:tasks.workunit.client.0.smithi062.stdout:5/420: write d8/d1e/d2c/f69 [511472,36918] 0 2024-04-20T21:31:59.279 INFO:tasks.workunit.client.1.smithi189.stdout:9/36: creat d9/fb x:0 0 0 2024-04-20T21:31:59.284 INFO:tasks.workunit.client.1.smithi189.stdout:8/197: rename da/df/d13/d31/l3b to da/df/l45 0 2024-04-20T21:31:59.287 INFO:tasks.workunit.client.1.smithi189.stdout:8/198: stat da/d1f/d2a/l40 0 2024-04-20T21:31:59.289 INFO:tasks.workunit.client.1.smithi189.stdout:5/21: rmdir d4 0 2024-04-20T21:31:59.290 INFO:tasks.workunit.client.1.smithi189.stdout:5/22: chown f2 39992 1 2024-04-20T21:31:59.290 INFO:tasks.workunit.client.1.smithi189.stdout:3/39: creat d2/fd x:0 0 0 2024-04-20T21:31:59.293 INFO:tasks.workunit.client.0.smithi062.stdout:5/421: mknod d8/d1e/d2c/d6c/d56/d50/c8b 0 2024-04-20T21:31:59.295 INFO:tasks.workunit.client.0.smithi062.stdout:6/313: mknod d0/d9/d11/d22/c66 0 2024-04-20T21:31:59.297 INFO:tasks.workunit.client.1.smithi189.stdout:5/23: mknod c5 0 2024-04-20T21:31:59.298 INFO:tasks.workunit.client.0.smithi062.stdout:6/314: readlink d0/d1a/d44/l4e 0 2024-04-20T21:31:59.302 INFO:tasks.workunit.client.0.smithi062.stdout:2/428: link d7/da/d28/c59 d7/da/d15/d1b/d24/c87 0 2024-04-20T21:31:59.303 INFO:tasks.workunit.client.0.smithi062.stdout:0/407: dread d7/d1c/d15/d17/d37/f61 [0,4194304] 0 2024-04-20T21:31:59.304 INFO:tasks.workunit.client.0.smithi062.stdout:5/422: creat d8/d1e/d2c/d6c/d56/d50/d88/f8c x:0 0 0 2024-04-20T21:31:59.306 INFO:tasks.workunit.client.0.smithi062.stdout:6/315: truncate d0/d9/d11/d22/f27 1509219 0 2024-04-20T21:31:59.308 INFO:tasks.workunit.client.0.smithi062.stdout:5/423: chown d8/l9 1 1 2024-04-20T21:31:59.309 INFO:tasks.workunit.client.0.smithi062.stdout:2/429: creat d7/da/d15/d2f/d38/d6b/f88 x:0 0 0 2024-04-20T21:31:59.310 INFO:tasks.workunit.client.1.smithi189.stdout:8/199: dread da/df/d22/d33/f36 [0,4194304] 0 2024-04-20T21:31:59.311 INFO:tasks.workunit.client.0.smithi062.stdout:5/424: stat d8/d10/l63 0 2024-04-20T21:31:59.315 INFO:tasks.workunit.client.0.smithi062.stdout:4/457: dwrite da/dd/df/d27/f2c [0,4194304] 0 2024-04-20T21:31:59.315 INFO:tasks.workunit.client.1.smithi189.stdout:8/200: mkdir da/df/d3a/d46 0 2024-04-20T21:31:59.317 INFO:tasks.workunit.client.0.smithi062.stdout:6/316: dread d0/d4f/f57 [0,4194304] 0 2024-04-20T21:31:59.320 INFO:tasks.workunit.client.0.smithi062.stdout:2/430: symlink d7/da/d15/l89 0 2024-04-20T21:31:59.322 INFO:tasks.workunit.client.0.smithi062.stdout:5/425: creat d8/d1e/d2c/d6c/d56/d50/d88/f8d x:0 0 0 2024-04-20T21:31:59.322 INFO:tasks.workunit.client.1.smithi189.stdout:8/201: creat da/de/d1e/d21/d44/f47 x:0 0 0 2024-04-20T21:31:59.336 INFO:tasks.workunit.client.0.smithi062.stdout:4/458: link da/dd/f69 da/dd/df/fa2 0 2024-04-20T21:31:59.336 INFO:tasks.workunit.client.1.smithi189.stdout:0/152: truncate d3/d26/f1d 6596757 0 2024-04-20T21:31:59.340 INFO:tasks.workunit.client.1.smithi189.stdout:8/202: rename da/df/l45 to da/de/l48 0 2024-04-20T21:31:59.343 INFO:tasks.workunit.client.1.smithi189.stdout:9/37: truncate f4 826369 0 2024-04-20T21:31:59.344 INFO:tasks.workunit.client.1.smithi189.stdout:0/153: rename d3/d26/d24 to d3/df/d30/d37 0 2024-04-20T21:31:59.345 INFO:tasks.workunit.client.1.smithi189.stdout:5/24: write f1 [1046065,115135] 0 2024-04-20T21:31:59.346 INFO:tasks.workunit.client.1.smithi189.stdout:0/154: readlink d3/df/l13 0 2024-04-20T21:31:59.351 INFO:tasks.workunit.client.0.smithi062.stdout:0/408: dread d7/d1c/f23 [4194304,4194304] 0 2024-04-20T21:31:59.354 INFO:tasks.workunit.client.1.smithi189.stdout:0/155: rename d3/df/f1e to d3/df/d30/d37/d29/f38 0 2024-04-20T21:31:59.354 INFO:tasks.workunit.client.0.smithi062.stdout:5/426: dwrite d8/d10/f26 [0,4194304] 0 2024-04-20T21:31:59.357 INFO:tasks.workunit.client.0.smithi062.stdout:6/317: dwrite d0/d1a/f29 [0,4194304] 0 2024-04-20T21:31:59.357 INFO:tasks.workunit.client.1.smithi189.stdout:8/203: dwrite da/df/d20/d27/f38 [4194304,4194304] 0 2024-04-20T21:31:59.359 INFO:tasks.workunit.client.1.smithi189.stdout:0/156: chown d3/df/d30/d37/l2d 656149945 1 2024-04-20T21:31:59.361 INFO:tasks.workunit.client.0.smithi062.stdout:6/318: mkdir d0/d2c/d67 0 2024-04-20T21:31:59.363 INFO:tasks.workunit.client.0.smithi062.stdout:4/459: dwrite da/dd/df/f28 [4194304,4194304] 0 2024-04-20T21:31:59.364 INFO:tasks.workunit.client.0.smithi062.stdout:5/427: mkdir d8/d1e/d2c/d6c/d56/d2f/d8e 0 2024-04-20T21:31:59.365 INFO:tasks.workunit.client.1.smithi189.stdout:8/204: mknod da/df/d13/d31/c49 0 2024-04-20T21:31:59.366 INFO:tasks.workunit.client.0.smithi062.stdout:5/428: stat d8/d1e/d80 0 2024-04-20T21:31:59.366 INFO:tasks.workunit.client.0.smithi062.stdout:0/409: creat d7/d1c/d15/d17/f8d x:0 0 0 2024-04-20T21:31:59.369 INFO:tasks.workunit.client.0.smithi062.stdout:9/417: sync 2024-04-20T21:31:59.369 INFO:tasks.workunit.client.0.smithi062.stdout:1/461: sync 2024-04-20T21:31:59.369 INFO:tasks.workunit.client.0.smithi062.stdout:3/432: sync 2024-04-20T21:31:59.369 INFO:tasks.workunit.client.0.smithi062.stdout:5/429: truncate d8/d1e/f25 1552798 0 2024-04-20T21:31:59.370 INFO:tasks.workunit.client.0.smithi062.stdout:4/460: mkdir da/dd/d15/d50/d64/d57/d5b/da3 0 2024-04-20T21:31:59.372 INFO:tasks.workunit.client.1.smithi189.stdout:0/157: write d3/df/d30/d37/d29/f38 [146084,34983] 0 2024-04-20T21:31:59.373 INFO:tasks.workunit.client.0.smithi062.stdout:6/319: creat d0/d9/d11/d22/f68 x:0 0 0 2024-04-20T21:31:59.376 INFO:tasks.workunit.client.0.smithi062.stdout:3/433: fdatasync d0/d1/f19 0 2024-04-20T21:31:59.377 INFO:tasks.workunit.client.0.smithi062.stdout:0/410: symlink d7/d1c/d15/d17/d1b/d74/l8e 0 2024-04-20T21:31:59.377 INFO:tasks.workunit.client.0.smithi062.stdout:2/431: dread d7/da/d15/d2f/d38/d51/f68 [0,4194304] 0 2024-04-20T21:31:59.379 INFO:tasks.workunit.client.0.smithi062.stdout:5/430: write d8/d10/f1b [3531520,54439] 0 2024-04-20T21:31:59.379 INFO:tasks.workunit.client.1.smithi189.stdout:0/158: mknod d3/c39 0 2024-04-20T21:31:59.382 INFO:tasks.workunit.client.1.smithi189.stdout:0/159: readlink d3/df/d30/d37/l2d 0 2024-04-20T21:31:59.382 INFO:tasks.workunit.client.0.smithi062.stdout:3/434: mknod d0/d1c/d52/c96 0 2024-04-20T21:31:59.383 INFO:tasks.workunit.client.1.smithi189.stdout:0/160: fdatasync d3/fb 0 2024-04-20T21:31:59.384 INFO:tasks.workunit.client.0.smithi062.stdout:9/418: write d16/d3d/d55/d6e/f72 [225321,83521] 0 2024-04-20T21:31:59.384 INFO:tasks.workunit.client.0.smithi062.stdout:5/431: readlink d8/d38/d7a/d4c/l57 0 2024-04-20T21:31:59.385 INFO:tasks.workunit.client.0.smithi062.stdout:1/462: rename dc/d15/d75/d94/d36/d78/l98 to dc/d15/d20/d28/l9e 0 2024-04-20T21:31:59.387 INFO:tasks.workunit.client.1.smithi189.stdout:0/161: readlink d3/df/d30/d37/l20 0 2024-04-20T21:31:59.388 INFO:tasks.workunit.client.0.smithi062.stdout:2/432: readlink d7/da/d15/d1b/d24/l4f 0 2024-04-20T21:31:59.389 INFO:tasks.workunit.client.1.smithi189.stdout:8/205: dwrite da/fb [0,4194304] 0 2024-04-20T21:31:59.390 INFO:tasks.workunit.client.0.smithi062.stdout:9/419: unlink d16/d3d/f88 0 2024-04-20T21:31:59.394 INFO:tasks.workunit.client.0.smithi062.stdout:9/420: write fc [5430681,104603] 0 2024-04-20T21:31:59.395 INFO:tasks.workunit.client.0.smithi062.stdout:3/435: dread - d0/d1c/d63/f91 zero size 2024-04-20T21:31:59.400 INFO:tasks.workunit.client.1.smithi189.stdout:9/38: read f5 [236065,42534] 0 2024-04-20T21:31:59.401 INFO:tasks.workunit.client.0.smithi062.stdout:5/432: symlink d8/d10/d3e/d8a/l8f 0 2024-04-20T21:31:59.403 INFO:tasks.workunit.client.0.smithi062.stdout:1/463: mknod dc/d27/d9a/c9f 0 2024-04-20T21:31:59.404 INFO:tasks.workunit.client.0.smithi062.stdout:0/411: dread d7/d20/f5c [0,4194304] 0 2024-04-20T21:31:59.405 INFO:tasks.workunit.client.1.smithi189.stdout:8/206: rename da/fd to da/de/d1e/d21/d44/f4a 0 2024-04-20T21:31:59.407 INFO:tasks.workunit.client.0.smithi062.stdout:3/436: mkdir d0/d1c/d52/d97 0 2024-04-20T21:31:59.409 INFO:tasks.workunit.client.0.smithi062.stdout:2/433: dread d7/da/d15/d2f/f33 [0,4194304] 0 2024-04-20T21:31:59.411 INFO:tasks.workunit.client.1.smithi189.stdout:9/39: mkdir d9/dc 0 2024-04-20T21:31:59.413 INFO:tasks.workunit.client.0.smithi062.stdout:4/461: dwrite da/f83 [0,4194304] 0 2024-04-20T21:31:59.413 INFO:tasks.workunit.client.0.smithi062.stdout:9/421: link d16/l25 d16/d3d/l8e 0 2024-04-20T21:31:59.415 INFO:tasks.workunit.client.0.smithi062.stdout:6/320: dread d0/d2c/f2e [0,4194304] 0 2024-04-20T21:31:59.419 INFO:tasks.workunit.client.1.smithi189.stdout:9/40: creat d9/dc/fd x:0 0 0 2024-04-20T21:31:59.419 INFO:tasks.workunit.client.0.smithi062.stdout:5/433: symlink d8/d1e/d2c/d6c/d56/d2f/d8e/l90 0 2024-04-20T21:31:59.419 INFO:tasks.workunit.client.0.smithi062.stdout:0/412: mknod d7/d1c/d15/d17/d37/c8f 0 2024-04-20T21:31:59.420 INFO:tasks.workunit.client.1.smithi189.stdout:9/41: truncate f8 443560 0 2024-04-20T21:31:59.423 INFO:tasks.workunit.client.0.smithi062.stdout:2/434: symlink d7/da/d28/d4b/l8a 0 2024-04-20T21:31:59.424 INFO:tasks.workunit.client.0.smithi062.stdout:4/462: symlink da/dd/d15/d50/d64/d57/la4 0 2024-04-20T21:31:59.426 INFO:tasks.workunit.client.1.smithi189.stdout:9/42: chown f1 1465120099 1 2024-04-20T21:31:59.428 INFO:tasks.workunit.client.0.smithi062.stdout:5/434: readlink d8/d1e/d2c/d6c/l61 0 2024-04-20T21:31:59.428 INFO:tasks.workunit.client.0.smithi062.stdout:6/321: write d0/d2c/f2e [3176744,21324] 0 2024-04-20T21:31:59.429 INFO:tasks.workunit.client.0.smithi062.stdout:9/422: truncate d16/f1e 3723332 0 2024-04-20T21:31:59.431 INFO:tasks.workunit.client.0.smithi062.stdout:3/437: truncate d0/d1/d10/d3a/d5e/d65/d88/f8a 4619500 0 2024-04-20T21:31:59.433 INFO:tasks.workunit.client.0.smithi062.stdout:1/464: dread dc/d15/d75/d94/d36/f9c [8388608,4194304] 0 2024-04-20T21:31:59.433 INFO:tasks.workunit.client.0.smithi062.stdout:2/435: write d7/da/d15/d1b/d24/f66 [64883,95529] 0 2024-04-20T21:31:59.434 INFO:tasks.workunit.client.0.smithi062.stdout:0/413: rename d7/d1c/d15/d17/d40/f58 to d7/d1c/d15/d17/d26/f90 0 2024-04-20T21:31:59.434 INFO:tasks.workunit.client.1.smithi189.stdout:9/43: rename d9/dc/fd to d9/dc/fe 0 2024-04-20T21:31:59.435 INFO:tasks.workunit.client.0.smithi062.stdout:5/435: chown d8/d1e/d2c/f86 26 1 2024-04-20T21:31:59.435 INFO:tasks.workunit.client.0.smithi062.stdout:6/322: write d0/d1a/d44/f4c [322442,89455] 0 2024-04-20T21:31:59.439 INFO:tasks.workunit.client.0.smithi062.stdout:3/438: chown d0/d1c/d63/f30 2332559 1 2024-04-20T21:31:59.442 INFO:tasks.workunit.client.0.smithi062.stdout:1/465: readlink dc/d27/d32/l37 0 2024-04-20T21:31:59.442 INFO:tasks.workunit.client.1.smithi189.stdout:6/44: sync 2024-04-20T21:31:59.443 INFO:tasks.workunit.client.1.smithi189.stdout:1/36: sync 2024-04-20T21:31:59.445 INFO:tasks.workunit.client.0.smithi062.stdout:4/463: getdents da/dd/d15/d50 0 2024-04-20T21:31:59.445 INFO:tasks.workunit.client.0.smithi062.stdout:7/421: sync 2024-04-20T21:31:59.448 INFO:tasks.workunit.client.1.smithi189.stdout:9/44: dread f6 [0,4194304] 0 2024-04-20T21:31:59.449 INFO:tasks.workunit.client.1.smithi189.stdout:1/37: mkdir d8/df 0 2024-04-20T21:31:59.449 INFO:tasks.workunit.client.0.smithi062.stdout:1/466: mkdir dc/d27/d9a/da0 0 2024-04-20T21:31:59.452 INFO:tasks.workunit.client.0.smithi062.stdout:6/323: rename d0/d9/d11/c35 to d0/d1a/c69 0 2024-04-20T21:31:59.452 INFO:tasks.workunit.client.0.smithi062.stdout:3/439: rmdir d0/d1/d10/d3a/d5e/d65 39 2024-04-20T21:31:59.455 INFO:tasks.workunit.client.1.smithi189.stdout:6/45: creat d8/ff x:0 0 0 2024-04-20T21:31:59.456 INFO:tasks.workunit.client.1.smithi189.stdout:9/45: symlink d9/dc/lf 0 2024-04-20T21:31:59.456 INFO:tasks.workunit.client.0.smithi062.stdout:4/464: mkdir da/dd/df/d27/da5 0 2024-04-20T21:31:59.457 INFO:tasks.workunit.client.1.smithi189.stdout:5/25: truncate f1 1331716 0 2024-04-20T21:31:59.457 INFO:tasks.workunit.client.0.smithi062.stdout:6/324: write d0/d2c/f30 [1301083,89700] 0 2024-04-20T21:31:59.457 INFO:tasks.workunit.client.0.smithi062.stdout:7/422: read d7/d3e/f41 [4165986,48801] 0 2024-04-20T21:31:59.459 INFO:tasks.workunit.client.0.smithi062.stdout:2/436: link d7/da/d15/d2f/l63 d7/da/d28/d4b/l8b 0 2024-04-20T21:31:59.459 INFO:tasks.workunit.client.1.smithi189.stdout:9/46: dread f3 [0,4194304] 0 2024-04-20T21:31:59.459 INFO:tasks.workunit.client.0.smithi062.stdout:5/436: getdents d8 0 2024-04-20T21:31:59.461 INFO:tasks.workunit.client.0.smithi062.stdout:4/465: write f3 [1825316,11651] 0 2024-04-20T21:31:59.468 INFO:tasks.workunit.client.0.smithi062.stdout:7/423: rmdir d7/d3e/d33/d3d 39 2024-04-20T21:31:59.469 INFO:tasks.workunit.client.0.smithi062.stdout:1/467: getdents dc/d2d/d7c 0 2024-04-20T21:31:59.469 INFO:tasks.workunit.client.1.smithi189.stdout:6/46: mkdir d8/d10 0 2024-04-20T21:31:59.470 INFO:tasks.workunit.client.0.smithi062.stdout:6/325: creat d0/d2c/d67/f6a x:0 0 0 2024-04-20T21:31:59.478 INFO:tasks.workunit.client.1.smithi189.stdout:1/38: dwrite d8/f9 [0,4194304] 0 2024-04-20T21:31:59.478 INFO:tasks.workunit.client.1.smithi189.stdout:9/47: symlink d9/da/l10 0 2024-04-20T21:31:59.479 INFO:tasks.workunit.client.0.smithi062.stdout:5/437: read d8/d10/f1b [445290,61792] 0 2024-04-20T21:31:59.480 INFO:tasks.workunit.client.0.smithi062.stdout:0/414: getdents d7/d1c/d15/d17/d40 0 2024-04-20T21:31:59.481 INFO:tasks.workunit.client.1.smithi189.stdout:8/207: truncate da/df/d20/d27/f38 4431193 0 2024-04-20T21:31:59.481 INFO:tasks.workunit.client.1.smithi189.stdout:1/39: chown d8/lc 2 1 2024-04-20T21:31:59.484 INFO:tasks.workunit.client.0.smithi062.stdout:4/466: link da/dd/d15/d50/fa0 da/dd/d15/fa6 0 2024-04-20T21:31:59.484 INFO:tasks.workunit.client.1.smithi189.stdout:0/162: dwrite d3/fe [0,4194304] 0 2024-04-20T21:31:59.486 INFO:tasks.workunit.client.1.smithi189.stdout:8/208: rmdir da/de/d1e 39 2024-04-20T21:31:59.492 INFO:tasks.workunit.client.0.smithi062.stdout:3/440: dwrite d0/d1/f19 [0,4194304] 0 2024-04-20T21:31:59.493 INFO:tasks.workunit.client.1.smithi189.stdout:6/47: dread f2 [0,4194304] 0 2024-04-20T21:31:59.494 INFO:tasks.workunit.client.1.smithi189.stdout:0/163: write d3/df/d30/d37/f31 [904090,78474] 0 2024-04-20T21:31:59.495 INFO:tasks.workunit.client.0.smithi062.stdout:0/415: dread d7/d1c/d15/d17/d26/f68 [0,4194304] 0 2024-04-20T21:31:59.495 INFO:tasks.workunit.client.0.smithi062.stdout:0/416: fsync d7/d1e/d6e/f7d 0 2024-04-20T21:31:59.496 INFO:tasks.workunit.client.1.smithi189.stdout:6/48: truncate f4 4750909 0 2024-04-20T21:31:59.499 INFO:tasks.workunit.client.1.smithi189.stdout:2/33: sync 2024-04-20T21:31:59.500 INFO:tasks.workunit.client.1.smithi189.stdout:2/34: chown d1/f4 94943937 1 2024-04-20T21:31:59.502 INFO:tasks.workunit.client.1.smithi189.stdout:5/26: unlink f1 0 2024-04-20T21:31:59.502 INFO:tasks.workunit.client.0.smithi062.stdout:8/454: sync 2024-04-20T21:31:59.502 INFO:tasks.workunit.client.1.smithi189.stdout:0/164: fsync d3/df/d30/d37/f31 0 2024-04-20T21:31:59.506 INFO:tasks.workunit.client.0.smithi062.stdout:8/455: dread - d7/d9/d29/f2b zero size 2024-04-20T21:31:59.507 INFO:tasks.workunit.client.1.smithi189.stdout:1/40: dread d8/f9 [0,4194304] 0 2024-04-20T21:31:59.507 INFO:tasks.workunit.client.0.smithi062.stdout:2/437: rename d7/da/d15/d2f/d38/d51/f67 to d7/da/d15/d2f/d38/d6a/d7c/f8c 0 2024-04-20T21:31:59.507 INFO:tasks.workunit.client.1.smithi189.stdout:2/35: rename d1/f2 to d1/f6 0 2024-04-20T21:31:59.509 INFO:tasks.workunit.client.1.smithi189.stdout:0/165: truncate d3/df/d30/d37/f31 1398373 0 2024-04-20T21:31:59.510 INFO:tasks.workunit.client.1.smithi189.stdout:2/36: fdatasync d1/f5 0 2024-04-20T21:31:59.510 INFO:tasks.workunit.client.1.smithi189.stdout:5/27: rename f2 to f6 0 2024-04-20T21:31:59.512 INFO:tasks.workunit.client.1.smithi189.stdout:2/37: chown d1/f5 675 1 2024-04-20T21:31:59.514 INFO:tasks.workunit.client.1.smithi189.stdout:1/41: mkdir d8/d10 0 2024-04-20T21:31:59.515 INFO:tasks.workunit.client.0.smithi062.stdout:4/467: rmdir da/dd/df/d27/d7a/d91 39 2024-04-20T21:31:59.517 INFO:tasks.workunit.client.1.smithi189.stdout:8/209: write da/df/d20/d27/f38 [5159839,47279] 0 2024-04-20T21:31:59.517 INFO:tasks.workunit.client.1.smithi189.stdout:9/48: write f4 [1535642,4453] 0 2024-04-20T21:31:59.517 INFO:tasks.workunit.client.1.smithi189.stdout:2/38: dread d1/f5 [0,4194304] 0 2024-04-20T21:31:59.519 INFO:tasks.workunit.client.1.smithi189.stdout:5/28: mkdir d7 0 2024-04-20T21:31:59.519 INFO:tasks.workunit.client.1.smithi189.stdout:9/49: dread - f7 zero size 2024-04-20T21:31:59.520 INFO:tasks.workunit.client.1.smithi189.stdout:0/166: rmdir d3/d26 39 2024-04-20T21:31:59.520 INFO:tasks.workunit.client.0.smithi062.stdout:9/423: dread d16/d2c/d42/f50 [0,4194304] 0 2024-04-20T21:31:59.521 INFO:tasks.workunit.client.1.smithi189.stdout:8/210: write f9 [2129554,110757] 0 2024-04-20T21:31:59.522 INFO:tasks.workunit.client.1.smithi189.stdout:6/49: dwrite d8/ff [0,4194304] 0 2024-04-20T21:31:59.522 INFO:tasks.workunit.client.1.smithi189.stdout:5/29: chown c5 111361 1 2024-04-20T21:31:59.522 INFO:tasks.workunit.client.0.smithi062.stdout:9/424: chown d16/d2c/d5e 0 1 2024-04-20T21:31:59.525 INFO:tasks.workunit.client.1.smithi189.stdout:9/50: chown d9/fb 18029 1 2024-04-20T21:31:59.528 INFO:tasks.workunit.client.0.smithi062.stdout:8/456: rmdir d7/d25 39 2024-04-20T21:31:59.529 INFO:tasks.workunit.client.1.smithi189.stdout:2/39: creat d1/f7 x:0 0 0 2024-04-20T21:31:59.531 INFO:tasks.workunit.client.1.smithi189.stdout:0/167: creat d3/df/f3a x:0 0 0 2024-04-20T21:31:59.532 INFO:tasks.workunit.client.1.smithi189.stdout:9/51: symlink d9/da/l11 0 2024-04-20T21:31:59.533 INFO:tasks.workunit.client.0.smithi062.stdout:6/326: fsync d0/d9/d11/d22/f24 0 2024-04-20T21:31:59.534 INFO:tasks.workunit.client.1.smithi189.stdout:1/42: rmdir d8/d10 0 2024-04-20T21:31:59.535 INFO:tasks.workunit.client.0.smithi062.stdout:2/438: mknod d7/da/d15/d2f/c8d 0 2024-04-20T21:31:59.536 INFO:tasks.workunit.client.1.smithi189.stdout:2/40: dread d1/f5 [0,4194304] 0 2024-04-20T21:31:59.536 INFO:tasks.workunit.client.1.smithi189.stdout:5/30: mknod d7/c8 0 2024-04-20T21:31:59.537 INFO:tasks.workunit.client.1.smithi189.stdout:0/168: readlink d3/df/d28/d2b/d2c/l32 0 2024-04-20T21:31:59.540 INFO:tasks.workunit.client.1.smithi189.stdout:0/169: stat d3/c39 0 2024-04-20T21:31:59.541 INFO:tasks.workunit.client.0.smithi062.stdout:9/425: creat d16/d2f/f8f x:0 0 0 2024-04-20T21:31:59.543 INFO:tasks.workunit.client.0.smithi062.stdout:0/417: fsync d7/d1c/d15/d17/d40/d79/f57 0 2024-04-20T21:31:59.544 INFO:tasks.workunit.client.1.smithi189.stdout:5/31: mkdir d7/d9 0 2024-04-20T21:31:59.547 INFO:tasks.workunit.client.0.smithi062.stdout:5/438: rmdir d8/d1e/d80 0 2024-04-20T21:31:59.548 INFO:tasks.workunit.client.1.smithi189.stdout:9/52: dread f5 [0,4194304] 0 2024-04-20T21:31:59.548 INFO:tasks.workunit.client.1.smithi189.stdout:2/41: getdents d1 0 2024-04-20T21:31:59.549 INFO:tasks.workunit.client.1.smithi189.stdout:5/32: mknod d7/ca 0 2024-04-20T21:31:59.550 INFO:tasks.workunit.client.0.smithi062.stdout:8/457: unlink d7/d9/f89 0 2024-04-20T21:31:59.550 INFO:tasks.workunit.client.0.smithi062.stdout:1/468: write dc/d15/d75/d94/f1e [410963,81531] 0 2024-04-20T21:31:59.551 INFO:tasks.workunit.client.0.smithi062.stdout:8/458: stat d7/d9 0 2024-04-20T21:31:59.551 INFO:tasks.workunit.client.1.smithi189.stdout:5/33: dread - f6 zero size 2024-04-20T21:31:59.552 INFO:tasks.workunit.client.1.smithi189.stdout:9/53: write f8 [1041418,13138] 0 2024-04-20T21:31:59.552 INFO:tasks.workunit.client.1.smithi189.stdout:2/42: write d1/f5 [1340391,33544] 0 2024-04-20T21:31:59.552 INFO:tasks.workunit.client.1.smithi189.stdout:6/50: fsync d8/ff 0 2024-04-20T21:31:59.552 INFO:tasks.workunit.client.0.smithi062.stdout:1/469: fsync dc/d15/d75/d94/f34 0 2024-04-20T21:31:59.554 INFO:tasks.workunit.client.1.smithi189.stdout:8/211: dread f9 [0,4194304] 0 2024-04-20T21:31:59.556 INFO:tasks.workunit.client.0.smithi062.stdout:3/441: creat d0/f98 x:0 0 0 2024-04-20T21:31:59.556 INFO:tasks.workunit.client.0.smithi062.stdout:6/327: dwrite d0/d2c/f30 [4194304,4194304] 0 2024-04-20T21:31:59.558 INFO:tasks.workunit.client.1.smithi189.stdout:6/51: fsync f7 0 2024-04-20T21:31:59.560 INFO:tasks.workunit.client.1.smithi189.stdout:5/34: truncate f6 562700 0 2024-04-20T21:31:59.560 INFO:tasks.workunit.client.1.smithi189.stdout:1/43: dread d8/f9 [0,4194304] 0 2024-04-20T21:31:59.561 INFO:tasks.workunit.client.1.smithi189.stdout:1/44: chown d8/ld 48327 1 2024-04-20T21:31:59.561 INFO:tasks.workunit.client.1.smithi189.stdout:9/54: unlink d9/dc/lf 0 2024-04-20T21:31:59.563 INFO:tasks.workunit.client.1.smithi189.stdout:2/43: getdents d1 0 2024-04-20T21:31:59.564 INFO:tasks.workunit.client.1.smithi189.stdout:1/45: chown d8/lb 218093145 1 2024-04-20T21:31:59.565 INFO:tasks.workunit.client.1.smithi189.stdout:9/55: fsync f6 0 2024-04-20T21:31:59.566 INFO:tasks.workunit.client.1.smithi189.stdout:5/35: creat d7/fb x:0 0 0 2024-04-20T21:31:59.569 INFO:tasks.workunit.client.1.smithi189.stdout:8/212: rmdir da/df/d3a/d46 0 2024-04-20T21:31:59.570 INFO:tasks.workunit.client.1.smithi189.stdout:5/36: write d7/fb [788075,69870] 0 2024-04-20T21:31:59.571 INFO:tasks.workunit.client.1.smithi189.stdout:2/44: link d1/f5 d1/f8 0 2024-04-20T21:31:59.574 INFO:tasks.workunit.client.1.smithi189.stdout:8/213: creat da/d1f/d2a/f4b x:0 0 0 2024-04-20T21:31:59.576 INFO:tasks.workunit.client.1.smithi189.stdout:2/45: chown d1/f6 591382082 1 2024-04-20T21:31:59.579 INFO:tasks.workunit.client.0.smithi062.stdout:9/426: creat d16/d4e/f90 x:0 0 0 2024-04-20T21:31:59.579 INFO:tasks.workunit.client.0.smithi062.stdout:3/442: readlink d0/d1c/d63/d42/d5b/l81 0 2024-04-20T21:31:59.579 INFO:tasks.workunit.client.1.smithi189.stdout:1/46: getdents d8 0 2024-04-20T21:31:59.580 INFO:tasks.workunit.client.0.smithi062.stdout:5/439: creat d8/d38/d7a/f91 x:0 0 0 2024-04-20T21:31:59.580 INFO:tasks.workunit.client.1.smithi189.stdout:5/37: getdents d7/d9 0 2024-04-20T21:31:59.581 INFO:tasks.workunit.client.0.smithi062.stdout:8/459: mknod d7/d9/d29/c9a 0 2024-04-20T21:31:59.581 INFO:tasks.workunit.client.0.smithi062.stdout:1/470: mknod dc/d70/d7f/ca1 0 2024-04-20T21:31:59.581 INFO:tasks.workunit.client.1.smithi189.stdout:6/52: dread f4 [0,4194304] 0 2024-04-20T21:31:59.582 INFO:tasks.workunit.client.1.smithi189.stdout:2/46: read d1/f8 [30383,49776] 0 2024-04-20T21:31:59.585 INFO:tasks.workunit.client.1.smithi189.stdout:5/38: write f6 [612988,36908] 0 2024-04-20T21:31:59.585 INFO:tasks.workunit.client.0.smithi062.stdout:9/427: fdatasync d16/d2c/d5e/d75/d76/f81 0 2024-04-20T21:31:59.585 INFO:tasks.workunit.client.0.smithi062.stdout:7/424: dwrite d7/ff [0,4194304] 0 2024-04-20T21:31:59.587 INFO:tasks.workunit.client.0.smithi062.stdout:2/439: dread - d7/da/d15/d2f/d38/d6a/d7c/f8c zero size 2024-04-20T21:31:59.588 INFO:tasks.workunit.client.0.smithi062.stdout:7/425: chown d7/d26/c50 59630 1 2024-04-20T21:31:59.592 INFO:tasks.workunit.client.0.smithi062.stdout:4/468: getdents da/dd/d15/d4b 0 2024-04-20T21:31:59.592 INFO:tasks.workunit.client.1.smithi189.stdout:2/47: dread d1/f4 [0,4194304] 0 2024-04-20T21:31:59.596 INFO:tasks.workunit.client.1.smithi189.stdout:2/48: chown d1 117 1 2024-04-20T21:31:59.596 INFO:tasks.workunit.client.0.smithi062.stdout:8/460: write d7/d25/d5c/f72 [361639,24246] 0 2024-04-20T21:31:59.598 INFO:tasks.workunit.client.0.smithi062.stdout:6/328: write d0/f3 [3329367,45571] 0 2024-04-20T21:31:59.601 INFO:tasks.workunit.client.1.smithi189.stdout:8/214: truncate da/de/f10 1133081 0 2024-04-20T21:31:59.603 INFO:tasks.workunit.client.0.smithi062.stdout:1/471: mkdir dc/d15/d75/d94/d36/d3a/da2 0 2024-04-20T21:31:59.603 INFO:tasks.workunit.client.1.smithi189.stdout:2/49: write d1/f7 [548251,70006] 0 2024-04-20T21:31:59.606 INFO:tasks.workunit.client.0.smithi062.stdout:3/443: mkdir d0/d1/d10/d99 0 2024-04-20T21:31:59.607 INFO:tasks.workunit.client.0.smithi062.stdout:9/428: creat d16/d3d/d55/d6e/d7d/f91 x:0 0 0 2024-04-20T21:31:59.607 INFO:tasks.workunit.client.0.smithi062.stdout:2/440: creat d7/d39/f8e x:0 0 0 2024-04-20T21:31:59.609 INFO:tasks.workunit.client.0.smithi062.stdout:4/469: creat da/dd/d15/fa7 x:0 0 0 2024-04-20T21:31:59.612 INFO:tasks.workunit.client.0.smithi062.stdout:9/429: write d16/d3d/f43 [3237504,2113] 0 2024-04-20T21:31:59.614 INFO:tasks.workunit.client.0.smithi062.stdout:4/470: chown da/dd/df/f17 378942944 1 2024-04-20T21:31:59.615 INFO:tasks.workunit.client.1.smithi189.stdout:9/56: dwrite d9/dc/fe [0,4194304] 0 2024-04-20T21:31:59.617 INFO:tasks.workunit.client.0.smithi062.stdout:3/444: symlink d0/d5c/l9a 0 2024-04-20T21:31:59.618 INFO:tasks.workunit.client.0.smithi062.stdout:4/471: chown da/dd/d15/d50/f86 1367 1 2024-04-20T21:31:59.618 INFO:tasks.workunit.client.1.smithi189.stdout:1/47: dread d8/f9 [0,4194304] 0 2024-04-20T21:31:59.620 INFO:tasks.workunit.client.0.smithi062.stdout:2/441: read d7/da/d28/d4b/f73 [2380828,47417] 0 2024-04-20T21:31:59.623 INFO:tasks.workunit.client.1.smithi189.stdout:2/50: creat d1/f9 x:0 0 0 2024-04-20T21:31:59.624 INFO:tasks.workunit.client.0.smithi062.stdout:5/440: creat d8/d1e/d2c/d6c/d56/f92 x:0 0 0 2024-04-20T21:31:59.625 INFO:tasks.workunit.client.0.smithi062.stdout:4/472: dread da/dd/f45 [0,4194304] 0 2024-04-20T21:31:59.626 INFO:tasks.workunit.client.0.smithi062.stdout:1/472: getdents dc/d27/d32 0 2024-04-20T21:31:59.626 INFO:tasks.workunit.client.1.smithi189.stdout:9/57: rename d9/da to d9/d12 0 2024-04-20T21:31:59.627 INFO:tasks.workunit.client.1.smithi189.stdout:1/48: readlink d8/lc 0 2024-04-20T21:31:59.629 INFO:tasks.workunit.client.0.smithi062.stdout:2/442: truncate d7/d39/f8e 18759 0 2024-04-20T21:31:59.631 INFO:tasks.workunit.client.0.smithi062.stdout:7/426: rename d7/fe to d7/d3e/d33/d3d/d5e/d7e/f93 0 2024-04-20T21:31:59.631 INFO:tasks.workunit.client.1.smithi189.stdout:8/215: getdents da/df 0 2024-04-20T21:31:59.633 INFO:tasks.workunit.client.1.smithi189.stdout:9/58: truncate f4 1746521 0 2024-04-20T21:31:59.636 INFO:tasks.workunit.client.0.smithi062.stdout:6/329: getdents d0/d2c 0 2024-04-20T21:31:59.637 INFO:tasks.workunit.client.1.smithi189.stdout:1/49: creat d8/f11 x:0 0 0 2024-04-20T21:31:59.637 INFO:tasks.workunit.client.1.smithi189.stdout:6/53: dwrite d8/da/dd/fe [0,4194304] 0 2024-04-20T21:31:59.637 INFO:tasks.workunit.client.0.smithi062.stdout:4/473: fsync da/dd/d15/d50/d64/d3d/f7e 0 2024-04-20T21:31:59.638 INFO:tasks.workunit.client.0.smithi062.stdout:9/430: rmdir d16/d2f/d5b/d61 0 2024-04-20T21:31:59.640 INFO:tasks.workunit.client.1.smithi189.stdout:5/39: dwrite f6 [0,4194304] 0 2024-04-20T21:31:59.642 INFO:tasks.workunit.client.0.smithi062.stdout:6/330: chown d0/l49 3 1 2024-04-20T21:31:59.645 INFO:tasks.workunit.client.0.smithi062.stdout:1/473: link dc/d70/d7f/c80 dc/d15/d20/d28/d83/ca3 0 2024-04-20T21:31:59.648 INFO:tasks.workunit.client.1.smithi189.stdout:9/59: symlink d9/d12/l13 0 2024-04-20T21:31:59.650 INFO:tasks.workunit.client.1.smithi189.stdout:9/60: chown d9 58922 1 2024-04-20T21:31:59.650 INFO:tasks.workunit.client.0.smithi062.stdout:3/445: dwrite d0/d1/d10/d3a/d5e/f7d [0,4194304] 0 2024-04-20T21:31:59.650 INFO:tasks.workunit.client.1.smithi189.stdout:2/51: dread d1/f8 [0,4194304] 0 2024-04-20T21:31:59.657 INFO:tasks.workunit.client.0.smithi062.stdout:6/331: truncate d0/d9/d11/f65 478343 0 2024-04-20T21:31:59.658 INFO:tasks.workunit.client.1.smithi189.stdout:5/40: creat d7/d9/fc x:0 0 0 2024-04-20T21:31:59.659 INFO:tasks.workunit.client.0.smithi062.stdout:3/446: chown d0/l13 40 1 2024-04-20T21:31:59.660 INFO:tasks.workunit.client.0.smithi062.stdout:8/461: dwrite d7/d30/f5d [0,4194304] 0 2024-04-20T21:31:59.662 INFO:tasks.workunit.client.1.smithi189.stdout:2/52: mknod d1/ca 0 2024-04-20T21:31:59.662 INFO:tasks.workunit.client.0.smithi062.stdout:1/474: chown dc/d15/d75/d94/d36/d78 192974 1 2024-04-20T21:31:59.666 INFO:tasks.workunit.client.1.smithi189.stdout:1/50: dread f1 [0,4194304] 0 2024-04-20T21:31:59.669 INFO:tasks.workunit.client.0.smithi062.stdout:2/443: dwrite d7/da/d15/d1b/d3d/f52 [0,4194304] 0 2024-04-20T21:31:59.672 INFO:tasks.workunit.client.0.smithi062.stdout:3/447: dread - d0/d1/d10/d31/f92 zero size 2024-04-20T21:31:59.676 INFO:tasks.workunit.client.1.smithi189.stdout:1/51: readlink d8/ld 0 2024-04-20T21:31:59.679 INFO:tasks.workunit.client.0.smithi062.stdout:5/441: dwrite d8/d1e/d2c/f86 [0,4194304] 0 2024-04-20T21:31:59.681 INFO:tasks.workunit.client.0.smithi062.stdout:5/442: chown d8/d1e/d2c/d6c/d56/d50/d88/f8d 2613 1 2024-04-20T21:31:59.686 INFO:tasks.workunit.client.1.smithi189.stdout:6/54: dwrite d8/da/dd/fe [0,4194304] 0 2024-04-20T21:31:59.689 INFO:tasks.workunit.client.1.smithi189.stdout:1/52: creat d8/f12 x:0 0 0 2024-04-20T21:31:59.691 INFO:tasks.workunit.client.1.smithi189.stdout:7/57: sync 2024-04-20T21:31:59.691 INFO:tasks.workunit.client.1.smithi189.stdout:4/22: sync 2024-04-20T21:31:59.693 INFO:tasks.workunit.client.0.smithi062.stdout:5/443: dread - d8/d1e/d2c/d6c/d56/d50/d88/f8d zero size 2024-04-20T21:31:59.693 INFO:tasks.workunit.client.1.smithi189.stdout:3/40: sync 2024-04-20T21:31:59.693 INFO:tasks.workunit.client.1.smithi189.stdout:8/216: truncate da/de/f28 241690 0 2024-04-20T21:31:59.696 INFO:tasks.workunit.client.1.smithi189.stdout:4/23: write d2/f4 [1049409,116059] 0 2024-04-20T21:31:59.699 INFO:tasks.workunit.client.0.smithi062.stdout:2/444: dwrite d7/f56 [0,4194304] 0 2024-04-20T21:31:59.703 INFO:tasks.workunit.client.1.smithi189.stdout:7/58: mkdir d5/d15 0 2024-04-20T21:31:59.707 INFO:tasks.workunit.client.1.smithi189.stdout:3/41: symlink d2/d8/le 0 2024-04-20T21:31:59.708 INFO:tasks.workunit.client.1.smithi189.stdout:5/41: fsync d7/d9/fc 0 2024-04-20T21:31:59.708 INFO:tasks.workunit.client.1.smithi189.stdout:4/24: mknod d2/c7 0 2024-04-20T21:31:59.708 INFO:tasks.workunit.client.1.smithi189.stdout:0/170: sync 2024-04-20T21:31:59.709 INFO:tasks.workunit.client.1.smithi189.stdout:6/55: link f7 d8/d10/f11 0 2024-04-20T21:31:59.711 INFO:tasks.workunit.client.0.smithi062.stdout:1/475: dread f1 [0,4194304] 0 2024-04-20T21:31:59.713 INFO:tasks.workunit.client.0.smithi062.stdout:9/431: dread d16/f19 [0,4194304] 0 2024-04-20T21:31:59.724 INFO:tasks.workunit.client.0.smithi062.stdout:2/445: unlink d7/da/d15/d1b/d24/l4f 0 2024-04-20T21:31:59.725 INFO:tasks.workunit.client.0.smithi062.stdout:5/444: dwrite d8/d38/d7a/f36 [0,4194304] 0 2024-04-20T21:31:59.727 INFO:tasks.workunit.client.0.smithi062.stdout:9/432: creat d16/d3d/d55/d6e/f92 x:0 0 0 2024-04-20T21:31:59.728 INFO:tasks.workunit.client.1.smithi189.stdout:7/59: symlink d5/l16 0 2024-04-20T21:31:59.729 INFO:tasks.workunit.client.0.smithi062.stdout:1/476: unlink dc/d70/f8c 0 2024-04-20T21:31:59.731 INFO:tasks.workunit.client.1.smithi189.stdout:1/53: creat d8/df/f13 x:0 0 0 2024-04-20T21:31:59.738 INFO:tasks.workunit.client.0.smithi062.stdout:3/448: write d0/d1b/f57 [1337832,6499] 0 2024-04-20T21:31:59.742 INFO:tasks.workunit.client.0.smithi062.stdout:0/418: sync 2024-04-20T21:31:59.745 INFO:tasks.workunit.client.0.smithi062.stdout:2/446: mkdir d7/da/d28/d4b/d8f 0 2024-04-20T21:31:59.746 INFO:tasks.workunit.client.1.smithi189.stdout:7/60: fsync f2 0 2024-04-20T21:31:59.748 INFO:tasks.workunit.client.0.smithi062.stdout:7/427: dwrite d7/d3e/d33/f37 [0,4194304] 0 2024-04-20T21:31:59.749 INFO:tasks.workunit.client.0.smithi062.stdout:9/433: symlink d16/d2c/d42/l93 0 2024-04-20T21:31:59.751 INFO:tasks.workunit.client.1.smithi189.stdout:5/42: mknod d7/d9/cd 0 2024-04-20T21:31:59.754 INFO:tasks.workunit.client.0.smithi062.stdout:6/332: dwrite d0/d9/d11/f63 [0,4194304] 0 2024-04-20T21:31:59.762 INFO:tasks.workunit.client.0.smithi062.stdout:6/333: readlink d0/d9/d11/l64 0 2024-04-20T21:31:59.763 INFO:tasks.workunit.client.0.smithi062.stdout:4/474: dread da/f7b [0,4194304] 0 2024-04-20T21:31:59.765 INFO:tasks.workunit.client.0.smithi062.stdout:2/447: symlink d7/l90 0 2024-04-20T21:31:59.766 INFO:tasks.workunit.client.0.smithi062.stdout:5/445: dwrite d8/f44 [0,4194304] 0 2024-04-20T21:31:59.767 INFO:tasks.workunit.client.0.smithi062.stdout:4/475: fdatasync da/dd/f73 0 2024-04-20T21:31:59.768 INFO:tasks.workunit.client.0.smithi062.stdout:7/428: mkdir d7/d3e/d33/d3d/d4d/d83/d94 0 2024-04-20T21:31:59.770 INFO:tasks.workunit.client.0.smithi062.stdout:2/448: stat d7/da/d15/d1b/l5c 0 2024-04-20T21:31:59.775 INFO:tasks.workunit.client.0.smithi062.stdout:3/449: getdents d0/d1/d10/d31/d41 0 2024-04-20T21:31:59.775 INFO:tasks.workunit.client.0.smithi062.stdout:2/449: stat d7/da/d15/d2f/d38/c43 0 2024-04-20T21:31:59.776 INFO:tasks.workunit.client.1.smithi189.stdout:8/217: dwrite da/d1f/f2f [0,4194304] 0 2024-04-20T21:31:59.776 INFO:tasks.workunit.client.1.smithi189.stdout:1/54: creat d8/f14 x:0 0 0 2024-04-20T21:31:59.777 INFO:tasks.workunit.client.0.smithi062.stdout:0/419: rename d7/d1c/d15/d17/d37/f5d to d7/d1c/d15/d17/d40/f91 0 2024-04-20T21:31:59.789 INFO:tasks.workunit.client.1.smithi189.stdout:2/53: dwrite d1/f8 [0,4194304] 0 2024-04-20T21:31:59.794 INFO:tasks.workunit.client.0.smithi062.stdout:0/420: symlink d7/d1c/d15/d53/l92 0 2024-04-20T21:31:59.801 INFO:tasks.workunit.client.1.smithi189.stdout:0/171: creat d3/f3b x:0 0 0 2024-04-20T21:31:59.801 INFO:tasks.workunit.client.1.smithi189.stdout:9/61: dwrite f3 [0,4194304] 0 2024-04-20T21:31:59.801 INFO:tasks.workunit.client.1.smithi189.stdout:5/43: write d7/d9/fc [693160,298] 0 2024-04-20T21:31:59.803 INFO:tasks.workunit.client.0.smithi062.stdout:6/334: dread d0/f5e [0,4194304] 0 2024-04-20T21:31:59.806 INFO:tasks.workunit.client.0.smithi062.stdout:6/335: read d0/d9/f15 [820673,54057] 0 2024-04-20T21:31:59.809 INFO:tasks.workunit.client.0.smithi062.stdout:7/429: dread f4 [0,4194304] 0 2024-04-20T21:31:59.813 INFO:tasks.workunit.client.0.smithi062.stdout:0/421: rename d7/d1c/d15/d17/f1d to d7/d1e/d6e/f93 0 2024-04-20T21:31:59.814 INFO:tasks.workunit.client.0.smithi062.stdout:7/430: stat d7/d3e/d33/f49 0 2024-04-20T21:31:59.816 INFO:tasks.workunit.client.1.smithi189.stdout:9/62: fdatasync f7 0 2024-04-20T21:31:59.820 INFO:tasks.workunit.client.1.smithi189.stdout:3/42: dwrite d2/d8/fb [0,4194304] 0 2024-04-20T21:31:59.823 INFO:tasks.workunit.client.0.smithi062.stdout:5/446: dwrite d8/d1e/d2c/d6c/f65 [0,4194304] 0 2024-04-20T21:31:59.824 INFO:tasks.workunit.client.1.smithi189.stdout:5/44: chown d7/c8 6217841 1 2024-04-20T21:31:59.825 INFO:tasks.workunit.client.1.smithi189.stdout:6/56: dwrite d8/da/fb [0,4194304] 0 2024-04-20T21:31:59.827 INFO:tasks.workunit.client.0.smithi062.stdout:7/431: mkdir d7/d3e/d33/d3d/d4d/d83/d95 0 2024-04-20T21:31:59.827 INFO:tasks.workunit.client.0.smithi062.stdout:8/462: dwrite d7/d25/d3b/d58/d8b/f1c [0,4194304] 0 2024-04-20T21:31:59.828 INFO:tasks.workunit.client.1.smithi189.stdout:9/63: write f7 [131646,85522] 0 2024-04-20T21:31:59.831 INFO:tasks.workunit.client.0.smithi062.stdout:5/447: unlink d8/d1e/d2c/f71 0 2024-04-20T21:31:59.834 INFO:tasks.workunit.client.0.smithi062.stdout:7/432: write d7/d3e/d33/f76 [637290,30289] 0 2024-04-20T21:31:59.835 INFO:tasks.workunit.client.1.smithi189.stdout:4/25: dwrite d2/f4 [0,4194304] 0 2024-04-20T21:31:59.836 INFO:tasks.workunit.client.0.smithi062.stdout:1/477: truncate dc/d15/d75/d94/d36/f5d 461305 0 2024-04-20T21:31:59.850 INFO:tasks.workunit.client.0.smithi062.stdout:6/336: symlink d0/d9/l6b 0 2024-04-20T21:31:59.851 INFO:tasks.workunit.client.0.smithi062.stdout:9/434: symlink d16/d2c/l94 0 2024-04-20T21:31:59.853 INFO:tasks.workunit.client.0.smithi062.stdout:7/433: symlink d7/d3e/d33/d3d/d5e/d7e/d80/l96 0 2024-04-20T21:31:59.854 INFO:tasks.workunit.client.1.smithi189.stdout:0/172: dwrite d3/df/d30/d37/f31 [0,4194304] 0 2024-04-20T21:31:59.855 INFO:tasks.workunit.client.0.smithi062.stdout:9/435: read fd [6150202,87138] 0 2024-04-20T21:31:59.856 INFO:tasks.workunit.client.0.smithi062.stdout:1/478: symlink dc/d15/d20/la4 0 2024-04-20T21:31:59.856 INFO:tasks.workunit.client.0.smithi062.stdout:8/463: link d7/d25/d5c/l80 d7/d25/d3b/d58/d8b/d96/l9b 0 2024-04-20T21:31:59.859 INFO:tasks.workunit.client.0.smithi062.stdout:6/337: symlink d0/l6c 0 2024-04-20T21:31:59.861 INFO:tasks.workunit.client.1.smithi189.stdout:1/55: dwrite d8/fa [0,4194304] 0 2024-04-20T21:31:59.863 INFO:tasks.workunit.client.0.smithi062.stdout:5/448: dwrite d8/d1e/f25 [0,4194304] 0 2024-04-20T21:31:59.863 INFO:tasks.workunit.client.0.smithi062.stdout:8/464: rmdir d7/d25/d3b 39 2024-04-20T21:31:59.868 INFO:tasks.workunit.client.1.smithi189.stdout:0/173: truncate d3/df/d30/d37/f31 5074687 0 2024-04-20T21:31:59.874 INFO:tasks.workunit.client.0.smithi062.stdout:5/449: creat d8/d1e/f93 x:0 0 0 2024-04-20T21:31:59.879 INFO:tasks.workunit.client.0.smithi062.stdout:2/450: dwrite d7/da/f13 [0,4194304] 0 2024-04-20T21:31:59.879 INFO:tasks.workunit.client.1.smithi189.stdout:0/174: write d3/fb [682582,86275] 0 2024-04-20T21:31:59.882 INFO:tasks.workunit.client.0.smithi062.stdout:8/465: dread - d7/d25/d3b/d58/f6b zero size 2024-04-20T21:31:59.882 INFO:tasks.workunit.client.0.smithi062.stdout:5/450: creat d8/d10/d3e/f94 x:0 0 0 2024-04-20T21:31:59.884 INFO:tasks.workunit.client.0.smithi062.stdout:8/466: fsync d7/d25/f93 0 2024-04-20T21:31:59.885 INFO:tasks.workunit.client.1.smithi189.stdout:6/57: dwrite f5 [0,4194304] 0 2024-04-20T21:31:59.888 INFO:tasks.workunit.client.0.smithi062.stdout:2/451: mknod d7/da/c91 0 2024-04-20T21:31:59.889 INFO:tasks.workunit.client.0.smithi062.stdout:3/450: dread d0/d1/f20 [0,4194304] 0 2024-04-20T21:31:59.890 INFO:tasks.workunit.client.1.smithi189.stdout:6/58: fsync d8/da/dd/fe 0 2024-04-20T21:31:59.893 INFO:tasks.workunit.client.0.smithi062.stdout:8/467: rmdir d7/d30 39 2024-04-20T21:31:59.900 INFO:tasks.workunit.client.1.smithi189.stdout:1/56: dwrite d8/f14 [0,4194304] 0 2024-04-20T21:31:59.901 INFO:tasks.workunit.client.1.smithi189.stdout:1/57: fsync d8/f12 0 2024-04-20T21:31:59.901 INFO:tasks.workunit.client.0.smithi062.stdout:3/451: stat d0/d1c/d52/l5d 0 2024-04-20T21:31:59.903 INFO:tasks.workunit.client.0.smithi062.stdout:1/479: dwrite dc/f35 [0,4194304] 0 2024-04-20T21:31:59.906 INFO:tasks.workunit.client.0.smithi062.stdout:8/468: creat d7/d25/d76/f9c x:0 0 0 2024-04-20T21:31:59.916 INFO:tasks.workunit.client.0.smithi062.stdout:2/452: getdents d7 0 2024-04-20T21:31:59.916 INFO:tasks.workunit.client.0.smithi062.stdout:1/480: creat dc/d15/d20/d28/d83/fa5 x:0 0 0 2024-04-20T21:31:59.920 INFO:tasks.workunit.client.0.smithi062.stdout:0/422: truncate d7/d1c/d15/d17/d37/f5b 1330412 0 2024-04-20T21:31:59.920 INFO:tasks.workunit.client.0.smithi062.stdout:8/469: creat d7/d25/f9d x:0 0 0 2024-04-20T21:31:59.922 INFO:tasks.workunit.client.0.smithi062.stdout:9/436: dread d16/d3d/d55/f73 [0,4194304] 0 2024-04-20T21:31:59.926 INFO:tasks.workunit.client.0.smithi062.stdout:2/453: mkdir d7/da/d15/d1b/d92 0 2024-04-20T21:31:59.931 INFO:tasks.workunit.client.0.smithi062.stdout:8/470: mknod d7/d25/d70/c9e 0 2024-04-20T21:31:59.935 INFO:tasks.workunit.client.0.smithi062.stdout:8/471: fdatasync d7/d25/f9d 0 2024-04-20T21:31:59.935 INFO:tasks.workunit.client.1.smithi189.stdout:1/58: dwrite f2 [0,4194304] 0 2024-04-20T21:31:59.937 INFO:tasks.workunit.client.1.smithi189.stdout:8/218: creat da/df/d3a/f4c x:0 0 0 2024-04-20T21:31:59.937 INFO:tasks.workunit.client.1.smithi189.stdout:2/54: mknod d1/cb 0 2024-04-20T21:31:59.938 INFO:tasks.workunit.client.0.smithi062.stdout:0/423: write d7/d20/f5c [430571,104163] 0 2024-04-20T21:31:59.940 INFO:tasks.workunit.client.1.smithi189.stdout:2/55: write d1/f7 [1087959,95308] 0 2024-04-20T21:31:59.941 INFO:tasks.workunit.client.1.smithi189.stdout:5/45: unlink c5 0 2024-04-20T21:31:59.942 INFO:tasks.workunit.client.0.smithi062.stdout:2/454: mkdir d7/da/d15/d2f/d93 0 2024-04-20T21:31:59.943 INFO:tasks.workunit.client.0.smithi062.stdout:8/472: rename d7/d25/c4a to d7/d25/d5c/c9f 0 2024-04-20T21:31:59.943 INFO:tasks.workunit.client.1.smithi189.stdout:2/56: write d1/f8 [2303405,91894] 0 2024-04-20T21:31:59.945 INFO:tasks.workunit.client.1.smithi189.stdout:3/43: dread d2/f5 [0,4194304] 0 2024-04-20T21:31:59.945 INFO:tasks.workunit.client.1.smithi189.stdout:5/46: write d7/d9/fc [759647,18289] 0 2024-04-20T21:31:59.945 INFO:tasks.workunit.client.0.smithi062.stdout:5/451: truncate d8/d38/d7a/f40 851776 0 2024-04-20T21:31:59.946 INFO:tasks.workunit.client.1.smithi189.stdout:4/26: creat d2/f8 x:0 0 0 2024-04-20T21:31:59.948 INFO:tasks.workunit.client.1.smithi189.stdout:5/47: write d7/d9/fc [1530420,67697] 0 2024-04-20T21:31:59.949 INFO:tasks.workunit.client.0.smithi062.stdout:0/424: write d7/d1c/d15/d17/d1b/f1a [3704565,21491] 0 2024-04-20T21:31:59.950 INFO:tasks.workunit.client.0.smithi062.stdout:2/455: creat d7/da/d15/d1b/d3d/f94 x:0 0 0 2024-04-20T21:31:59.950 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:31:59 smithi062 ceph-mon[16221]: pgmap v14: 65 pgs: 65 active+clean; 1.1 GiB data, 3.4 GiB used, 533 GiB / 536 GiB avail; 31 MiB/s rd, 135 MiB/s wr, 365 op/s 2024-04-20T21:31:59.950 INFO:tasks.workunit.client.1.smithi189.stdout:5/48: read f6 [2130306,32904] 0 2024-04-20T21:31:59.951 INFO:tasks.workunit.client.1.smithi189.stdout:0/175: symlink d3/df/d30/l3c 0 2024-04-20T21:31:59.952 INFO:tasks.workunit.client.0.smithi062.stdout:9/437: dread f4 [4194304,4194304] 0 2024-04-20T21:31:59.953 INFO:tasks.workunit.client.0.smithi062.stdout:0/425: write d7/d1c/f3e [205493,81465] 0 2024-04-20T21:31:59.954 INFO:tasks.workunit.client.0.smithi062.stdout:9/438: fsync d16/f36 0 2024-04-20T21:31:59.956 INFO:tasks.workunit.client.0.smithi062.stdout:8/473: creat d7/d25/d70/fa0 x:0 0 0 2024-04-20T21:31:59.957 INFO:tasks.workunit.client.0.smithi062.stdout:9/439: chown d16/d3d/d55/f73 468829 1 2024-04-20T21:31:59.957 INFO:tasks.workunit.client.0.smithi062.stdout:2/456: creat d7/da/d28/d4b/f95 x:0 0 0 2024-04-20T21:31:59.959 INFO:tasks.workunit.client.0.smithi062.stdout:5/452: unlink d8/d1e/d2c/d6c/d56/l5a 0 2024-04-20T21:31:59.960 INFO:tasks.workunit.client.0.smithi062.stdout:9/440: chown d16/d2c/d5e/d75/d76/f81 3081194 1 2024-04-20T21:31:59.961 INFO:tasks.workunit.client.1.smithi189.stdout:7/61: rmdir d5/d15 0 2024-04-20T21:31:59.961 INFO:tasks.workunit.client.1.smithi189.stdout:8/219: rmdir da/de 39 2024-04-20T21:31:59.962 INFO:tasks.workunit.client.0.smithi062.stdout:9/441: stat d16/d2f/l51 0 2024-04-20T21:31:59.962 INFO:tasks.workunit.client.1.smithi189.stdout:1/59: creat d8/df/f15 x:0 0 0 2024-04-20T21:31:59.964 INFO:tasks.workunit.client.0.smithi062.stdout:8/474: creat d7/d25/d70/fa1 x:0 0 0 2024-04-20T21:31:59.967 INFO:tasks.workunit.client.1.smithi189.stdout:0/176: dread d3/df/d28/d2b/f36 [0,4194304] 0 2024-04-20T21:31:59.969 INFO:tasks.workunit.client.1.smithi189.stdout:0/177: truncate d3/f3b 966771 0 2024-04-20T21:31:59.970 INFO:tasks.workunit.client.0.smithi062.stdout:9/442: unlink d16/l1a 0 2024-04-20T21:31:59.970 INFO:tasks.workunit.client.0.smithi062.stdout:0/426: dread d7/d1c/d15/f2d [0,4194304] 0 2024-04-20T21:31:59.972 INFO:tasks.workunit.client.0.smithi062.stdout:2/457: creat d7/da/d15/d2f/d38/d6a/d7c/f96 x:0 0 0 2024-04-20T21:31:59.972 INFO:tasks.workunit.client.1.smithi189.stdout:2/57: unlink d1/ca 0 2024-04-20T21:31:59.974 INFO:tasks.workunit.client.0.smithi062.stdout:3/452: dread d0/d1b/f23 [0,4194304] 0 2024-04-20T21:31:59.975 INFO:tasks.workunit.client.1.smithi189.stdout:4/27: symlink d2/l9 0 2024-04-20T21:31:59.976 INFO:tasks.workunit.client.0.smithi062.stdout:8/475: link d7/d25/d3b/d58/d8b/f4e d7/d25/d3b/fa2 0 2024-04-20T21:31:59.977 INFO:tasks.workunit.client.0.smithi062.stdout:2/458: dread d7/da/d15/d1b/f2c [0,4194304] 0 2024-04-20T21:31:59.977 INFO:tasks.workunit.client.1.smithi189.stdout:9/64: link d9/dc/fe d9/d12/f14 0 2024-04-20T21:31:59.978 INFO:tasks.workunit.client.0.smithi062.stdout:3/453: write d0/d1/d10/f79 [288274,111078] 0 2024-04-20T21:31:59.982 INFO:tasks.workunit.client.1.smithi189.stdout:5/49: mknod d7/d9/ce 0 2024-04-20T21:31:59.985 INFO:tasks.workunit.client.1.smithi189.stdout:8/220: symlink da/df/d20/l4d 0 2024-04-20T21:31:59.986 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:31:59 smithi189 ceph-mon[21562]: pgmap v14: 65 pgs: 65 active+clean; 1.1 GiB data, 3.4 GiB used, 533 GiB / 536 GiB avail; 31 MiB/s rd, 135 MiB/s wr, 365 op/s 2024-04-20T21:31:59.987 INFO:tasks.workunit.client.0.smithi062.stdout:8/476: dread d7/d25/d5c/f98 [0,4194304] 0 2024-04-20T21:31:59.987 INFO:tasks.workunit.client.1.smithi189.stdout:1/60: symlink d8/df/l16 0 2024-04-20T21:31:59.987 INFO:tasks.workunit.client.1.smithi189.stdout:1/61: read - d8/fe zero size 2024-04-20T21:31:59.988 INFO:tasks.workunit.client.0.smithi062.stdout:5/453: dwrite d8/d38/d7a/f91 [0,4194304] 0 2024-04-20T21:31:59.989 INFO:tasks.workunit.client.0.smithi062.stdout:2/459: rename d7/da/d15/d1b/d92 to d7/da/d15/d2f/d97 0 2024-04-20T21:31:59.989 INFO:tasks.workunit.client.1.smithi189.stdout:0/178: creat d3/df/d30/f3d x:0 0 0 2024-04-20T21:31:59.991 INFO:tasks.workunit.client.0.smithi062.stdout:3/454: truncate d0/d11/f56 747408 0 2024-04-20T21:31:59.992 INFO:tasks.workunit.client.0.smithi062.stdout:8/477: creat d7/d9/d29/fa3 x:0 0 0 2024-04-20T21:31:59.992 INFO:tasks.workunit.client.1.smithi189.stdout:4/28: symlink d2/la 0 2024-04-20T21:31:59.993 INFO:tasks.workunit.client.1.smithi189.stdout:1/62: write d8/fe [124708,37498] 0 2024-04-20T21:31:59.994 INFO:tasks.workunit.client.0.smithi062.stdout:8/478: dread - d7/d25/d5c/f94 zero size 2024-04-20T21:31:59.994 INFO:tasks.workunit.client.0.smithi062.stdout:3/455: dread - d0/d1c/d52/d7b/f85 zero size 2024-04-20T21:31:59.995 INFO:tasks.workunit.client.1.smithi189.stdout:9/65: mkdir d9/d12/d15 0 2024-04-20T21:32:00.002 INFO:tasks.workunit.client.0.smithi062.stdout:5/454: dread d8/d1e/d2c/f86 [0,4194304] 0 2024-04-20T21:32:00.003 INFO:tasks.workunit.client.1.smithi189.stdout:9/66: dread f7 [0,4194304] 0 2024-04-20T21:32:00.006 INFO:tasks.workunit.client.1.smithi189.stdout:9/67: fsync f6 0 2024-04-20T21:32:00.006 INFO:tasks.workunit.client.0.smithi062.stdout:3/456: mknod d0/d1/d10/d3a/d5e/c9b 0 2024-04-20T21:32:00.009 INFO:tasks.workunit.client.1.smithi189.stdout:5/50: dwrite d7/d9/fc [0,4194304] 0 2024-04-20T21:32:00.009 INFO:tasks.workunit.client.0.smithi062.stdout:5/455: write d8/d1e/d2c/d6c/d56/d50/d88/f8d [962811,40595] 0 2024-04-20T21:32:00.016 INFO:tasks.workunit.client.0.smithi062.stdout:5/456: dread - d8/d1e/d2c/d6c/d56/f92 zero size 2024-04-20T21:32:00.016 INFO:tasks.workunit.client.0.smithi062.stdout:3/457: symlink d0/d1c/d63/l9c 0 2024-04-20T21:32:00.016 INFO:tasks.workunit.client.0.smithi062.stdout:2/460: dwrite d7/da/d28/d4b/f73 [0,4194304] 0 2024-04-20T21:32:00.017 INFO:tasks.workunit.client.0.smithi062.stdout:5/457: chown d8/d1e/d2c/d6c/d56/d2f/d8e/l90 11 1 2024-04-20T21:32:00.017 INFO:tasks.workunit.client.1.smithi189.stdout:4/29: creat d2/fb x:0 0 0 2024-04-20T21:32:00.017 INFO:tasks.workunit.client.1.smithi189.stdout:4/30: dread - d2/fb zero size 2024-04-20T21:32:00.017 INFO:tasks.workunit.client.1.smithi189.stdout:9/68: dread f4 [0,4194304] 0 2024-04-20T21:32:00.017 INFO:tasks.workunit.client.1.smithi189.stdout:1/63: dread d8/fa [0,4194304] 0 2024-04-20T21:32:00.017 INFO:tasks.workunit.client.1.smithi189.stdout:4/31: write d2/f8 [394517,92330] 0 2024-04-20T21:32:00.017 INFO:tasks.workunit.client.0.smithi062.stdout:3/458: write d0/d1/d10/d31/f92 [1039275,11664] 0 2024-04-20T21:32:00.018 INFO:tasks.workunit.client.1.smithi189.stdout:3/44: link d2/d3/l4 d2/d8/lf 0 2024-04-20T21:32:00.019 INFO:tasks.workunit.client.1.smithi189.stdout:4/32: write f0 [672821,100321] 0 2024-04-20T21:32:00.026 INFO:tasks.workunit.client.1.smithi189.stdout:0/179: rename d3/df/d30/d37/d29/f38 to d3/df/f3e 0 2024-04-20T21:32:00.028 INFO:tasks.workunit.client.1.smithi189.stdout:7/62: link d5/f11 d5/f17 0 2024-04-20T21:32:00.029 INFO:tasks.workunit.client.1.smithi189.stdout:0/180: chown d3/df/f3e 255 1 2024-04-20T21:32:00.034 INFO:tasks.workunit.client.1.smithi189.stdout:5/51: unlink d7/c8 0 2024-04-20T21:32:00.035 INFO:tasks.workunit.client.1.smithi189.stdout:9/69: mknod d9/dc/c16 0 2024-04-20T21:32:00.037 INFO:tasks.workunit.client.1.smithi189.stdout:1/64: symlink d8/df/l17 0 2024-04-20T21:32:00.040 INFO:tasks.workunit.client.1.smithi189.stdout:4/33: symlink d2/lc 0 2024-04-20T21:32:00.041 INFO:tasks.workunit.client.1.smithi189.stdout:1/65: write d8/f12 [89630,91094] 0 2024-04-20T21:32:00.042 INFO:tasks.workunit.client.1.smithi189.stdout:7/63: chown d5/c8 15 1 2024-04-20T21:32:00.043 INFO:tasks.workunit.client.1.smithi189.stdout:0/181: symlink d3/df/d28/l3f 0 2024-04-20T21:32:00.043 INFO:tasks.workunit.client.1.smithi189.stdout:4/34: write d2/fb [997282,116145] 0 2024-04-20T21:32:00.045 INFO:tasks.workunit.client.1.smithi189.stdout:8/221: rename da/df/d20/d2c/l42 to da/de/d1e/d21/l4e 0 2024-04-20T21:32:00.047 INFO:tasks.workunit.client.1.smithi189.stdout:5/52: write f6 [1863070,116717] 0 2024-04-20T21:32:00.049 INFO:tasks.workunit.client.1.smithi189.stdout:4/35: mkdir d2/dd 0 2024-04-20T21:32:00.049 INFO:tasks.workunit.client.1.smithi189.stdout:1/66: write f1 [2821562,65071] 0 2024-04-20T21:32:00.051 INFO:tasks.workunit.client.1.smithi189.stdout:7/64: rename d5/f13 to d5/f18 0 2024-04-20T21:32:00.051 INFO:tasks.workunit.client.1.smithi189.stdout:0/182: rmdir d3/df/d28/d2b/d2c 39 2024-04-20T21:32:00.051 INFO:tasks.workunit.client.1.smithi189.stdout:1/67: rename d8 to d8/df/d18 22 2024-04-20T21:32:00.053 INFO:tasks.workunit.client.1.smithi189.stdout:5/53: mknod d7/cf 0 2024-04-20T21:32:00.056 INFO:tasks.workunit.client.1.smithi189.stdout:7/65: write d5/f18 [49368,71775] 0 2024-04-20T21:32:00.056 INFO:tasks.workunit.client.1.smithi189.stdout:4/36: creat d2/dd/fe x:0 0 0 2024-04-20T21:32:00.057 INFO:tasks.workunit.client.1.smithi189.stdout:0/183: symlink d3/df/d28/l40 0 2024-04-20T21:32:00.064 INFO:tasks.workunit.client.1.smithi189.stdout:7/66: rename d5/l12 to d5/l19 0 2024-04-20T21:32:00.065 INFO:tasks.workunit.client.1.smithi189.stdout:7/67: chown f2 2764158 1 2024-04-20T21:32:00.066 INFO:tasks.workunit.client.1.smithi189.stdout:0/184: creat d3/df/d30/d37/d29/f41 x:0 0 0 2024-04-20T21:32:00.066 INFO:tasks.workunit.client.1.smithi189.stdout:8/222: dwrite f3 [4194304,4194304] 0 2024-04-20T21:32:00.067 INFO:tasks.workunit.client.1.smithi189.stdout:0/185: write d3/fb [1055988,1040] 0 2024-04-20T21:32:00.068 INFO:tasks.workunit.client.1.smithi189.stdout:7/68: creat d5/f1a x:0 0 0 2024-04-20T21:32:00.069 INFO:tasks.workunit.client.1.smithi189.stdout:0/186: chown d3 1843 1 2024-04-20T21:32:00.070 INFO:tasks.workunit.client.1.smithi189.stdout:5/54: dread f6 [0,4194304] 0 2024-04-20T21:32:00.071 INFO:tasks.workunit.client.1.smithi189.stdout:7/69: symlink d5/l1b 0 2024-04-20T21:32:00.076 INFO:tasks.workunit.client.1.smithi189.stdout:5/55: mknod d7/c10 0 2024-04-20T21:32:00.076 INFO:tasks.workunit.client.1.smithi189.stdout:7/70: write d5/fe [541145,6595] 0 2024-04-20T21:32:00.079 INFO:tasks.workunit.client.1.smithi189.stdout:8/223: link da/df/d13/l3f da/d30/l4f 0 2024-04-20T21:32:00.081 INFO:tasks.workunit.client.1.smithi189.stdout:8/224: readlink l2 0 2024-04-20T21:32:00.082 INFO:tasks.workunit.client.1.smithi189.stdout:5/56: chown d7/d9/fc 7 1 2024-04-20T21:32:00.085 INFO:tasks.workunit.client.1.smithi189.stdout:6/59: dwrite f4 [4194304,4194304] 0 2024-04-20T21:32:00.086 INFO:tasks.workunit.client.1.smithi189.stdout:5/57: chown f6 0 1 2024-04-20T21:32:00.087 INFO:tasks.workunit.client.1.smithi189.stdout:4/37: dwrite d2/f4 [0,4194304] 0 2024-04-20T21:32:00.087 INFO:tasks.workunit.client.1.smithi189.stdout:8/225: mkdir da/df/d3c/d50 0 2024-04-20T21:32:00.092 INFO:tasks.workunit.client.0.smithi062.stdout:4/476: sync 2024-04-20T21:32:00.098 INFO:tasks.workunit.client.0.smithi062.stdout:5/458: rmdir d8/d38/d7a/d4c 39 2024-04-20T21:32:00.101 INFO:tasks.workunit.client.1.smithi189.stdout:2/58: truncate d1/f8 1363907 0 2024-04-20T21:32:00.102 INFO:tasks.workunit.client.1.smithi189.stdout:4/38: rename d2/c7 to d2/cf 0 2024-04-20T21:32:00.103 INFO:tasks.workunit.client.1.smithi189.stdout:6/60: mkdir d8/d12 0 2024-04-20T21:32:00.104 INFO:tasks.workunit.client.1.smithi189.stdout:0/187: dwrite d3/df/d30/d37/f31 [0,4194304] 0 2024-04-20T21:32:00.105 INFO:tasks.workunit.client.1.smithi189.stdout:2/59: rename d1 to d1/dc 22 2024-04-20T21:32:00.109 INFO:tasks.workunit.client.1.smithi189.stdout:4/39: creat d2/f10 x:0 0 0 2024-04-20T21:32:00.109 INFO:tasks.workunit.client.0.smithi062.stdout:5/459: mknod d8/d1e/d2c/d6c/d56/d2f/d8e/c95 0 2024-04-20T21:32:00.110 INFO:tasks.workunit.client.0.smithi062.stdout:4/477: rename da/d4f/l66 to da/dd/d15/d50/d64/la8 0 2024-04-20T21:32:00.113 INFO:tasks.workunit.client.1.smithi189.stdout:1/68: getdents d8/df 0 2024-04-20T21:32:00.115 INFO:tasks.workunit.client.1.smithi189.stdout:1/69: read d8/f9 [858179,93575] 0 2024-04-20T21:32:00.115 INFO:tasks.workunit.client.1.smithi189.stdout:6/61: unlink d8/da/fb 0 2024-04-20T21:32:00.115 INFO:tasks.workunit.client.1.smithi189.stdout:3/45: write d2/f5 [12674,28866] 0 2024-04-20T21:32:00.118 INFO:tasks.workunit.client.1.smithi189.stdout:6/62: readlink d8/da/lc 0 2024-04-20T21:32:00.118 INFO:tasks.workunit.client.1.smithi189.stdout:2/60: creat d1/fd x:0 0 0 2024-04-20T21:32:00.119 INFO:tasks.workunit.client.0.smithi062.stdout:5/460: chown d8/d10/c46 0 1 2024-04-20T21:32:00.119 INFO:tasks.workunit.client.0.smithi062.stdout:2/461: dwrite d7/da/d15/d2f/d38/d51/f68 [0,4194304] 0 2024-04-20T21:32:00.120 INFO:tasks.workunit.client.1.smithi189.stdout:6/63: fsync d8/ff 0 2024-04-20T21:32:00.121 INFO:tasks.workunit.client.1.smithi189.stdout:2/61: chown d1/f4 20554 1 2024-04-20T21:32:00.121 INFO:tasks.workunit.client.1.smithi189.stdout:6/64: chown f7 587674 1 2024-04-20T21:32:00.121 INFO:tasks.workunit.client.1.smithi189.stdout:9/70: truncate f7 1209057 0 2024-04-20T21:32:00.124 INFO:tasks.workunit.client.0.smithi062.stdout:5/461: stat d8/d10 0 2024-04-20T21:32:00.125 INFO:tasks.workunit.client.1.smithi189.stdout:9/71: chown d9/d12 0 1 2024-04-20T21:32:00.125 INFO:tasks.workunit.client.0.smithi062.stdout:4/478: truncate da/dd/d15/f20 2009992 0 2024-04-20T21:32:00.125 INFO:tasks.workunit.client.1.smithi189.stdout:6/65: rename d8/d10 to d8/d10/d13 22 2024-04-20T21:32:00.127 INFO:tasks.workunit.client.1.smithi189.stdout:4/40: mkdir d2/dd/d11 0 2024-04-20T21:32:00.129 INFO:tasks.workunit.client.0.smithi062.stdout:2/462: mknod d7/da/d15/d1b/d24/c98 0 2024-04-20T21:32:00.129 INFO:tasks.workunit.client.0.smithi062.stdout:5/462: write d8/d10/f74 [3697138,115364] 0 2024-04-20T21:32:00.129 INFO:tasks.workunit.client.1.smithi189.stdout:0/188: dwrite d3/df/d30/d37/f23 [0,4194304] 0 2024-04-20T21:32:00.129 INFO:tasks.workunit.client.1.smithi189.stdout:6/66: truncate f4 9282941 0 2024-04-20T21:32:00.130 INFO:tasks.workunit.client.0.smithi062.stdout:4/479: fsync da/dd/d15/d50/d64/d3d/f7e 0 2024-04-20T21:32:00.132 INFO:tasks.workunit.client.1.smithi189.stdout:1/70: mknod d8/df/c19 0 2024-04-20T21:32:00.133 INFO:tasks.workunit.client.0.smithi062.stdout:0/427: dread d7/d1c/d15/d17/d1b/f1a [0,4194304] 0 2024-04-20T21:32:00.134 INFO:tasks.workunit.client.1.smithi189.stdout:1/71: truncate d8/fe 1055562 0 2024-04-20T21:32:00.134 INFO:tasks.workunit.client.0.smithi062.stdout:4/480: unlink da/cc 0 2024-04-20T21:32:00.135 INFO:tasks.workunit.client.1.smithi189.stdout:5/58: dwrite f6 [0,4194304] 0 2024-04-20T21:32:00.136 INFO:tasks.workunit.client.0.smithi062.stdout:5/463: write d8/d1e/d2c/d6c/d56/d50/d88/f8c [649317,85254] 0 2024-04-20T21:32:00.136 INFO:tasks.workunit.client.1.smithi189.stdout:1/72: write d8/f11 [557273,109546] 0 2024-04-20T21:32:00.137 INFO:tasks.workunit.client.1.smithi189.stdout:2/62: creat d1/fe x:0 0 0 2024-04-20T21:32:00.137 INFO:tasks.workunit.client.0.smithi062.stdout:2/463: rename d7/da/d15 to d7/da/d15/d1b/d99 22 2024-04-20T21:32:00.138 INFO:tasks.workunit.client.1.smithi189.stdout:9/72: mkdir d9/dc/d17 0 2024-04-20T21:32:00.139 INFO:tasks.workunit.client.1.smithi189.stdout:2/63: chown d1/f5 5085006 1 2024-04-20T21:32:00.141 INFO:tasks.workunit.client.1.smithi189.stdout:2/64: chown d1 67349750 1 2024-04-20T21:32:00.142 INFO:tasks.workunit.client.1.smithi189.stdout:4/41: read d2/fb [128925,117451] 0 2024-04-20T21:32:00.143 INFO:tasks.workunit.client.0.smithi062.stdout:0/428: creat d7/d1c/d15/d17/d40/d79/d71/f94 x:0 0 0 2024-04-20T21:32:00.145 INFO:tasks.workunit.client.0.smithi062.stdout:0/429: read d7/d20/f5c [137121,107370] 0 2024-04-20T21:32:00.146 INFO:tasks.workunit.client.0.smithi062.stdout:2/464: creat d7/da/d15/d1b/f9a x:0 0 0 2024-04-20T21:32:00.148 INFO:tasks.workunit.client.1.smithi189.stdout:7/71: getdents d5 0 2024-04-20T21:32:00.149 INFO:tasks.workunit.client.0.smithi062.stdout:5/464: symlink d8/d1e/d2c/d6c/l96 0 2024-04-20T21:32:00.152 INFO:tasks.workunit.client.0.smithi062.stdout:0/430: write d7/f9 [8774824,121917] 0 2024-04-20T21:32:00.152 INFO:tasks.workunit.client.0.smithi062.stdout:4/481: rename da/dd/l2b to da/dd/df/la9 0 2024-04-20T21:32:00.153 INFO:tasks.workunit.client.1.smithi189.stdout:7/72: truncate d5/f7 443696 0 2024-04-20T21:32:00.156 INFO:tasks.workunit.client.0.smithi062.stdout:2/465: mkdir d7/da/d15/d1b/d3d/d84/d9b 0 2024-04-20T21:32:00.158 INFO:tasks.workunit.client.0.smithi062.stdout:4/482: truncate da/dd/d15/d50/d64/d57/d5b/f94 661536 0 2024-04-20T21:32:00.159 INFO:tasks.workunit.client.1.smithi189.stdout:3/46: link d2/f6 d2/d8/f10 0 2024-04-20T21:32:00.160 INFO:tasks.workunit.client.1.smithi189.stdout:5/59: rename d7/c10 to d7/c11 0 2024-04-20T21:32:00.160 INFO:tasks.workunit.client.1.smithi189.stdout:9/73: dwrite f8 [0,4194304] 0 2024-04-20T21:32:00.160 INFO:tasks.workunit.client.0.smithi062.stdout:2/466: mkdir d7/da/d15/d2f/d38/d6b/d9c 0 2024-04-20T21:32:00.161 INFO:tasks.workunit.client.1.smithi189.stdout:4/42: dread d2/f4 [0,4194304] 0 2024-04-20T21:32:00.163 INFO:tasks.workunit.client.0.smithi062.stdout:4/483: creat da/dd/d15/d50/faa x:0 0 0 2024-04-20T21:32:00.166 INFO:tasks.workunit.client.0.smithi062.stdout:2/467: mkdir d7/da/d28/d4b/d9d 0 2024-04-20T21:32:00.167 INFO:tasks.workunit.client.0.smithi062.stdout:4/484: write da/dd/d15/d50/d64/d57/d5b/f6f [123363,53686] 0 2024-04-20T21:32:00.167 INFO:tasks.workunit.client.1.smithi189.stdout:0/189: rename d3 to d3/d26/d42 22 2024-04-20T21:32:00.173 INFO:tasks.workunit.client.1.smithi189.stdout:6/67: creat d8/d12/f14 x:0 0 0 2024-04-20T21:32:00.173 INFO:tasks.workunit.client.0.smithi062.stdout:4/485: chown da/dd/d15/d50/d64/d57/d5b/f8d 5 1 2024-04-20T21:32:00.174 INFO:tasks.workunit.client.1.smithi189.stdout:5/60: symlink d7/l12 0 2024-04-20T21:32:00.174 INFO:tasks.workunit.client.1.smithi189.stdout:2/65: rename d1/f8 to d1/ff 0 2024-04-20T21:32:00.177 INFO:tasks.workunit.client.0.smithi062.stdout:4/486: rmdir da/dd/df 39 2024-04-20T21:32:00.177 INFO:tasks.workunit.client.1.smithi189.stdout:4/43: symlink d2/l12 0 2024-04-20T21:32:00.178 INFO:tasks.workunit.client.1.smithi189.stdout:2/66: read - d1/fe zero size 2024-04-20T21:32:00.179 INFO:tasks.workunit.client.1.smithi189.stdout:6/68: rmdir d8/d10 39 2024-04-20T21:32:00.179 INFO:tasks.workunit.client.1.smithi189.stdout:0/190: symlink d3/df/d30/d37/l43 0 2024-04-20T21:32:00.181 INFO:tasks.workunit.client.1.smithi189.stdout:7/73: getdents d5 0 2024-04-20T21:32:00.181 INFO:tasks.workunit.client.1.smithi189.stdout:5/61: rmdir d7/d9 39 2024-04-20T21:32:00.186 INFO:tasks.workunit.client.1.smithi189.stdout:2/67: chown d1/fd 110211 1 2024-04-20T21:32:00.188 INFO:tasks.workunit.client.0.smithi062.stdout:2/468: dwrite d7/da/d15/d2f/d38/d6b/f88 [0,4194304] 0 2024-04-20T21:32:00.189 INFO:tasks.workunit.client.1.smithi189.stdout:6/69: mknod d8/da/c15 0 2024-04-20T21:32:00.189 INFO:tasks.workunit.client.0.smithi062.stdout:4/487: rename da/dd/d15/d6d/f8e to da/dd/df/d27/d32/fab 0 2024-04-20T21:32:00.195 INFO:tasks.workunit.client.0.smithi062.stdout:4/488: mkdir da/dd/d3c/dac 0 2024-04-20T21:32:00.197 INFO:tasks.workunit.client.0.smithi062.stdout:5/465: truncate d8/d38/f70 436522 0 2024-04-20T21:32:00.201 INFO:tasks.workunit.client.1.smithi189.stdout:3/47: dwrite d2/f7 [0,4194304] 0 2024-04-20T21:32:00.201 INFO:tasks.workunit.client.1.smithi189.stdout:6/70: mkdir d8/d12/d16 0 2024-04-20T21:32:00.202 INFO:tasks.workunit.client.0.smithi062.stdout:4/489: unlink da/f13 0 2024-04-20T21:32:00.202 INFO:tasks.workunit.client.0.smithi062.stdout:2/469: getdents d7/da/d28/d4b/d9d 0 2024-04-20T21:32:00.203 INFO:tasks.workunit.client.0.smithi062.stdout:5/466: fdatasync d8/d1e/d2c/f3a 0 2024-04-20T21:32:00.207 INFO:tasks.workunit.client.1.smithi189.stdout:6/71: write d8/da/dd/fe [2775335,27457] 0 2024-04-20T21:32:00.207 INFO:tasks.workunit.client.1.smithi189.stdout:9/74: dwrite d9/fb [0,4194304] 0 2024-04-20T21:32:00.208 INFO:tasks.workunit.client.1.smithi189.stdout:2/68: rename d1/ff to d1/f10 0 2024-04-20T21:32:00.208 INFO:tasks.workunit.client.0.smithi062.stdout:5/467: readlink d8/d38/l4d 0 2024-04-20T21:32:00.210 INFO:tasks.workunit.client.1.smithi189.stdout:2/69: chown d1 4 1 2024-04-20T21:32:00.210 INFO:tasks.workunit.client.0.smithi062.stdout:2/470: rename d7/da/d15/d1b/f9a to d7/da/d15/d2f/d97/f9e 0 2024-04-20T21:32:00.211 INFO:tasks.workunit.client.1.smithi189.stdout:9/75: chown d9/d12/d15 538 1 2024-04-20T21:32:00.212 INFO:tasks.workunit.client.0.smithi062.stdout:5/468: chown d8/d1e/d2c/d6c/d56/d2f/d8e/l90 3 1 2024-04-20T21:32:00.214 INFO:tasks.workunit.client.0.smithi062.stdout:2/471: write d7/da/d15/d2f/d38/f54 [491502,105594] 0 2024-04-20T21:32:00.215 INFO:tasks.workunit.client.1.smithi189.stdout:7/74: link d5/c8 d5/c1c 0 2024-04-20T21:32:00.218 INFO:tasks.workunit.client.1.smithi189.stdout:9/76: write f6 [2929514,126573] 0 2024-04-20T21:32:00.219 INFO:tasks.workunit.client.0.smithi062.stdout:5/469: truncate d8/d1e/d2c/d6c/d56/d50/f5f 877090 0 2024-04-20T21:32:00.220 INFO:tasks.workunit.client.0.smithi062.stdout:2/472: mknod d7/da/d15/d2f/d97/c9f 0 2024-04-20T21:32:00.221 INFO:tasks.workunit.client.1.smithi189.stdout:6/72: creat d8/d12/f17 x:0 0 0 2024-04-20T21:32:00.223 INFO:tasks.workunit.client.0.smithi062.stdout:5/470: write d8/d1e/d2c/d6c/d56/d50/d88/f8d [1985427,41495] 0 2024-04-20T21:32:00.226 INFO:tasks.workunit.client.0.smithi062.stdout:5/471: dread - d8/d1e/f93 zero size 2024-04-20T21:32:00.227 INFO:tasks.workunit.client.1.smithi189.stdout:6/73: chown d8/ff 7913 1 2024-04-20T21:32:00.228 INFO:tasks.workunit.client.0.smithi062.stdout:2/473: creat d7/da/d15/d2f/d97/fa0 x:0 0 0 2024-04-20T21:32:00.230 INFO:tasks.workunit.client.1.smithi189.stdout:7/75: mkdir d5/d1d 0 2024-04-20T21:32:00.233 INFO:tasks.workunit.client.0.smithi062.stdout:2/474: write d7/da/d15/d1b/d24/f7a [268858,31294] 0 2024-04-20T21:32:00.233 INFO:tasks.workunit.client.0.smithi062.stdout:4/490: getdents da 0 2024-04-20T21:32:00.235 INFO:tasks.workunit.client.0.smithi062.stdout:5/472: stat d8/d1e/d2c/d6c/f3c 0 2024-04-20T21:32:00.237 INFO:tasks.workunit.client.1.smithi189.stdout:9/77: rename d9/d12/l10 to d9/dc/d17/l18 0 2024-04-20T21:32:00.239 INFO:tasks.workunit.client.1.smithi189.stdout:6/74: chown d8/da/c15 78 1 2024-04-20T21:32:00.241 INFO:tasks.workunit.client.1.smithi189.stdout:4/44: dwrite d2/f4 [0,4194304] 0 2024-04-20T21:32:00.242 INFO:tasks.workunit.client.1.smithi189.stdout:0/191: dwrite d3/df/d30/f3d [0,4194304] 0 2024-04-20T21:32:00.243 INFO:tasks.workunit.client.1.smithi189.stdout:6/75: fsync f5 0 2024-04-20T21:32:00.244 INFO:tasks.workunit.client.1.smithi189.stdout:5/62: dwrite d7/fb [0,4194304] 0 2024-04-20T21:32:00.245 INFO:tasks.workunit.client.0.smithi062.stdout:5/473: rename d8/d10/f1b to d8/d1e/d2c/d6c/d56/d2f/d8e/f97 0 2024-04-20T21:32:00.247 INFO:tasks.workunit.client.1.smithi189.stdout:7/76: chown d5/c1c 5919710 1 2024-04-20T21:32:00.250 INFO:tasks.workunit.client.1.smithi189.stdout:3/48: dwrite d2/f5 [0,4194304] 0 2024-04-20T21:32:00.257 INFO:tasks.workunit.client.0.smithi062.stdout:4/491: dread da/dd/df/d27/f2c [0,4194304] 0 2024-04-20T21:32:00.260 INFO:tasks.workunit.client.1.smithi189.stdout:9/78: mkdir d9/dc/d17/d19 0 2024-04-20T21:32:00.260 INFO:tasks.workunit.client.0.smithi062.stdout:4/492: write da/f83 [1833924,112573] 0 2024-04-20T21:32:00.263 INFO:tasks.workunit.client.0.smithi062.stdout:5/474: creat d8/d38/d7a/d4c/d7d/f98 x:0 0 0 2024-04-20T21:32:00.264 INFO:tasks.workunit.client.0.smithi062.stdout:2/475: dwrite d7/d39/f45 [0,4194304] 0 2024-04-20T21:32:00.266 INFO:tasks.workunit.client.0.smithi062.stdout:2/476: dread - d7/d39/f86 zero size 2024-04-20T21:32:00.267 INFO:tasks.workunit.client.1.smithi189.stdout:4/45: mknod d2/dd/c13 0 2024-04-20T21:32:00.268 INFO:tasks.workunit.client.1.smithi189.stdout:5/63: stat d7/c11 0 2024-04-20T21:32:00.272 INFO:tasks.workunit.client.0.smithi062.stdout:5/475: symlink d8/d1e/d2c/d6c/d56/d2f/d8e/l99 0 2024-04-20T21:32:00.273 INFO:tasks.workunit.client.1.smithi189.stdout:6/76: symlink d8/l18 0 2024-04-20T21:32:00.274 INFO:tasks.workunit.client.1.smithi189.stdout:5/64: fdatasync f6 0 2024-04-20T21:32:00.280 INFO:tasks.workunit.client.0.smithi062.stdout:2/477: link d7/da/d15/d2f/d97/c9f d7/da/d28/d4b/d9d/ca1 0 2024-04-20T21:32:00.282 INFO:tasks.workunit.client.0.smithi062.stdout:2/478: dread d7/d39/f8e [0,4194304] 0 2024-04-20T21:32:00.286 INFO:tasks.workunit.client.0.smithi062.stdout:2/479: readlink d7/da/d15/d1b/l5c 0 2024-04-20T21:32:00.287 INFO:tasks.workunit.client.1.smithi189.stdout:4/46: write d2/fb [610484,81886] 0 2024-04-20T21:32:00.287 INFO:tasks.workunit.client.1.smithi189.stdout:7/77: write d5/f17 [658710,68036] 0 2024-04-20T21:32:00.289 INFO:tasks.workunit.client.1.smithi189.stdout:6/77: mknod d8/c19 0 2024-04-20T21:32:00.290 INFO:tasks.workunit.client.1.smithi189.stdout:3/49: dwrite d2/fd [0,4194304] 0 2024-04-20T21:32:00.292 INFO:tasks.workunit.client.1.smithi189.stdout:0/192: dwrite d3/df/f3a [0,4194304] 0 2024-04-20T21:32:00.294 INFO:tasks.workunit.client.0.smithi062.stdout:2/480: read d7/da/d15/d1b/f80 [260301,33128] 0 2024-04-20T21:32:00.295 INFO:tasks.workunit.client.1.smithi189.stdout:4/47: chown d2/dd/c13 31376892 1 2024-04-20T21:32:00.296 INFO:tasks.workunit.client.0.smithi062.stdout:2/481: fsync d7/da/f13 0 2024-04-20T21:32:00.299 INFO:tasks.workunit.client.0.smithi062.stdout:2/482: rename d7 to d7/da/d15/d1b/da2 22 2024-04-20T21:32:00.299 INFO:tasks.workunit.client.0.smithi062.stdout:4/493: dwrite da/dd/d15/d50/fa0 [0,4194304] 0 2024-04-20T21:32:00.302 INFO:tasks.workunit.client.1.smithi189.stdout:0/193: write d3/df/d30/d37/f31 [6003004,20472] 0 2024-04-20T21:32:00.302 INFO:tasks.workunit.client.0.smithi062.stdout:4/494: chown da/dd/df/d27/d32/l9a 54132 1 2024-04-20T21:32:00.303 INFO:tasks.workunit.client.1.smithi189.stdout:9/79: dwrite f3 [0,4194304] 0 2024-04-20T21:32:00.304 INFO:tasks.workunit.client.0.smithi062.stdout:2/483: creat d7/da/d15/d2f/d38/d6b/fa3 x:0 0 0 2024-04-20T21:32:00.305 INFO:tasks.workunit.client.0.smithi062.stdout:4/495: symlink da/dd/d15/lad 0 2024-04-20T21:32:00.306 INFO:tasks.workunit.client.0.smithi062.stdout:6/338: sync 2024-04-20T21:32:00.306 INFO:tasks.workunit.client.0.smithi062.stdout:7/434: sync 2024-04-20T21:32:00.308 INFO:tasks.workunit.client.1.smithi189.stdout:7/78: getdents d5 0 2024-04-20T21:32:00.310 INFO:tasks.workunit.client.0.smithi062.stdout:6/339: rmdir d0/d2c 39 2024-04-20T21:32:00.313 INFO:tasks.workunit.client.1.smithi189.stdout:9/80: chown d9/dc/c16 172 1 2024-04-20T21:32:00.315 INFO:tasks.workunit.client.1.smithi189.stdout:4/48: mkdir d2/dd/d11/d14 0 2024-04-20T21:32:00.315 INFO:tasks.workunit.client.1.smithi189.stdout:0/194: chown d3/df/f3a 5 1 2024-04-20T21:32:00.315 INFO:tasks.workunit.client.0.smithi062.stdout:7/435: symlink d7/d3e/d33/d3d/d4d/d83/d94/l97 0 2024-04-20T21:32:00.317 INFO:tasks.workunit.client.1.smithi189.stdout:7/79: truncate d5/f1a 532771 0 2024-04-20T21:32:00.318 INFO:tasks.workunit.client.1.smithi189.stdout:9/81: read d9/dc/fe [3498969,127798] 0 2024-04-20T21:32:00.322 INFO:tasks.workunit.client.0.smithi062.stdout:4/496: link da/dd/df/d27/f2c da/dd/fae 0 2024-04-20T21:32:00.323 INFO:tasks.workunit.client.0.smithi062.stdout:6/340: mknod d0/d9/d11/d5c/c6d 0 2024-04-20T21:32:00.323 INFO:tasks.workunit.client.0.smithi062.stdout:2/484: dwrite d7/da/f22 [4194304,4194304] 0 2024-04-20T21:32:00.326 INFO:tasks.workunit.client.1.smithi189.stdout:7/80: write d5/f11 [941390,62337] 0 2024-04-20T21:32:00.330 INFO:tasks.workunit.client.0.smithi062.stdout:6/341: unlink d0/f17 0 2024-04-20T21:32:00.332 INFO:tasks.workunit.client.1.smithi189.stdout:5/65: dwrite d7/d9/fc [0,4194304] 0 2024-04-20T21:32:00.332 INFO:tasks.workunit.client.0.smithi062.stdout:4/497: write da/dd/df/d27/d32/fab [355928,122737] 0 2024-04-20T21:32:00.334 INFO:tasks.workunit.client.0.smithi062.stdout:4/498: dread - da/dd/d3c/f95 zero size 2024-04-20T21:32:00.335 INFO:tasks.workunit.client.0.smithi062.stdout:4/499: chown da/dd/d15 0 1 2024-04-20T21:32:00.337 INFO:tasks.workunit.client.0.smithi062.stdout:6/342: rename d0/d9/d11/l28 to d0/d52/l6e 0 2024-04-20T21:32:00.338 INFO:tasks.workunit.client.0.smithi062.stdout:6/343: chown d0/f3e 647 1 2024-04-20T21:32:00.341 INFO:tasks.workunit.client.1.smithi189.stdout:0/195: creat d3/df/d30/d37/d29/d35/f44 x:0 0 0 2024-04-20T21:32:00.342 INFO:tasks.workunit.client.0.smithi062.stdout:2/485: link d7/da/d15/d2f/f30 d7/da/d15/d2f/fa4 0 2024-04-20T21:32:00.344 INFO:tasks.workunit.client.0.smithi062.stdout:2/486: read - d7/da/d28/d4b/f95 zero size 2024-04-20T21:32:00.344 INFO:tasks.workunit.client.1.smithi189.stdout:9/82: rename d9/dc/d17/d19 to d9/d12/d15/d1a 0 2024-04-20T21:32:00.345 INFO:tasks.workunit.client.0.smithi062.stdout:6/344: symlink d0/d9/d11/d22/l6f 0 2024-04-20T21:32:00.346 INFO:tasks.workunit.client.0.smithi062.stdout:2/487: write d7/da/d28/f2a [2305384,106894] 0 2024-04-20T21:32:00.353 INFO:tasks.workunit.client.1.smithi189.stdout:5/66: unlink f6 0 2024-04-20T21:32:00.356 INFO:tasks.workunit.client.1.smithi189.stdout:6/78: dwrite d8/d12/f17 [0,4194304] 0 2024-04-20T21:32:00.356 INFO:tasks.workunit.client.1.smithi189.stdout:7/81: link d5/f18 d5/f1e 0 2024-04-20T21:32:00.358 INFO:tasks.workunit.client.1.smithi189.stdout:5/67: chown d7/ca 51874 1 2024-04-20T21:32:00.361 INFO:tasks.workunit.client.0.smithi062.stdout:6/345: dwrite d0/d2c/f2e [0,4194304] 0 2024-04-20T21:32:00.362 INFO:tasks.workunit.client.0.smithi062.stdout:6/346: stat d0/d9/l6b 0 2024-04-20T21:32:00.363 INFO:tasks.workunit.client.1.smithi189.stdout:6/79: chown d8/c19 223 1 2024-04-20T21:32:00.364 INFO:tasks.workunit.client.0.smithi062.stdout:7/436: dwrite d7/d3e/f7b [0,4194304] 0 2024-04-20T21:32:00.367 INFO:tasks.workunit.client.1.smithi189.stdout:9/83: fdatasync d9/dc/fe 0 2024-04-20T21:32:00.368 INFO:tasks.workunit.client.0.smithi062.stdout:6/347: unlink d0/d1a/d44/l4e 0 2024-04-20T21:32:00.370 INFO:tasks.workunit.client.1.smithi189.stdout:7/82: creat d5/f1f x:0 0 0 2024-04-20T21:32:00.371 INFO:tasks.workunit.client.0.smithi062.stdout:7/437: write d7/f48 [3788010,119167] 0 2024-04-20T21:32:00.372 INFO:tasks.workunit.client.1.smithi189.stdout:9/84: chown d9 158997292 1 2024-04-20T21:32:00.376 INFO:tasks.workunit.client.1.smithi189.stdout:6/80: write f2 [2097336,41798] 0 2024-04-20T21:32:00.376 INFO:tasks.workunit.client.0.smithi062.stdout:7/438: dread - d7/d3e/d33/d3d/d5e/d7e/d80/f8c zero size 2024-04-20T21:32:00.376 INFO:tasks.workunit.client.0.smithi062.stdout:6/348: mknod d0/d2c/d67/c70 0 2024-04-20T21:32:00.379 INFO:tasks.workunit.client.0.smithi062.stdout:6/349: chown d0/d1a/c3d 0 1 2024-04-20T21:32:00.379 INFO:tasks.workunit.client.1.smithi189.stdout:9/85: symlink d9/l1b 0 2024-04-20T21:32:00.380 INFO:tasks.workunit.client.1.smithi189.stdout:4/49: dwrite d2/f10 [0,4194304] 0 2024-04-20T21:32:00.382 INFO:tasks.workunit.client.1.smithi189.stdout:7/83: creat d5/f20 x:0 0 0 2024-04-20T21:32:00.384 INFO:tasks.workunit.client.1.smithi189.stdout:0/196: getdents d3/d26 0 2024-04-20T21:32:00.387 INFO:tasks.workunit.client.1.smithi189.stdout:4/50: fdatasync d2/f4 0 2024-04-20T21:32:00.388 INFO:tasks.workunit.client.1.smithi189.stdout:3/50: dwrite d2/f6 [0,4194304] 0 2024-04-20T21:32:00.393 INFO:tasks.workunit.client.1.smithi189.stdout:7/84: mkdir d5/d21 0 2024-04-20T21:32:00.397 INFO:tasks.workunit.client.1.smithi189.stdout:7/85: truncate d5/f14 236720 0 2024-04-20T21:32:00.397 INFO:tasks.workunit.client.1.smithi189.stdout:5/68: dread d7/d9/fc [0,4194304] 0 2024-04-20T21:32:00.398 INFO:tasks.workunit.client.0.smithi062.stdout:7/439: rename d7/d3e/d33/d3d/d4d/d54/f2b to d7/d3e/d33/d3d/d5e/d72/f98 0 2024-04-20T21:32:00.398 INFO:tasks.workunit.client.1.smithi189.stdout:6/81: link d8/ff d8/f1a 0 2024-04-20T21:32:00.399 INFO:tasks.workunit.client.1.smithi189.stdout:7/86: readlink d5/l16 0 2024-04-20T21:32:00.400 INFO:tasks.workunit.client.1.smithi189.stdout:6/82: write f5 [980563,101628] 0 2024-04-20T21:32:00.401 INFO:tasks.workunit.client.1.smithi189.stdout:3/51: mknod d2/c11 0 2024-04-20T21:32:00.404 INFO:tasks.workunit.client.0.smithi062.stdout:6/350: creat d0/d9/f71 x:0 0 0 2024-04-20T21:32:00.404 INFO:tasks.workunit.client.0.smithi062.stdout:2/488: write d7/da/d15/d1b/f80 [117943,42177] 0 2024-04-20T21:32:00.406 INFO:tasks.workunit.client.1.smithi189.stdout:5/69: creat d7/f13 x:0 0 0 2024-04-20T21:32:00.406 INFO:tasks.workunit.client.1.smithi189.stdout:4/51: dread d2/f8 [0,4194304] 0 2024-04-20T21:32:00.407 INFO:tasks.workunit.client.0.smithi062.stdout:2/489: chown d7/da/d15/d1b/f80 178 1 2024-04-20T21:32:00.408 INFO:tasks.workunit.client.0.smithi062.stdout:6/351: readlink d0/d52/l6e 0 2024-04-20T21:32:00.411 INFO:tasks.workunit.client.0.smithi062.stdout:2/490: chown d7/da/d15/d2f/d38/d6a/d7c/f8c 105819746 1 2024-04-20T21:32:00.411 INFO:tasks.workunit.client.1.smithi189.stdout:0/197: rmdir d3 39 2024-04-20T21:32:00.415 INFO:tasks.workunit.client.1.smithi189.stdout:5/70: mkdir d7/d14 0 2024-04-20T21:32:00.415 INFO:tasks.workunit.client.0.smithi062.stdout:2/491: rmdir d7/da/d15/d2f/d38 39 2024-04-20T21:32:00.417 INFO:tasks.workunit.client.1.smithi189.stdout:3/52: rename d2/d8/cc to d2/d8/c12 0 2024-04-20T21:32:00.417 INFO:tasks.workunit.client.0.smithi062.stdout:7/440: dwrite d7/d3e/d33/f52 [0,4194304] 0 2024-04-20T21:32:00.418 INFO:tasks.workunit.client.0.smithi062.stdout:2/492: write d7/da/f13 [1558251,122286] 0 2024-04-20T21:32:00.420 INFO:tasks.workunit.client.0.smithi062.stdout:7/441: stat d7/d3e/d33/d3d/c7c 0 2024-04-20T21:32:00.420 INFO:tasks.workunit.client.0.smithi062.stdout:2/493: chown d7/da/d15/d1b/d24 378002 1 2024-04-20T21:32:00.422 INFO:tasks.workunit.client.1.smithi189.stdout:4/52: mknod d2/dd/d11/d14/c15 0 2024-04-20T21:32:00.423 INFO:tasks.workunit.client.1.smithi189.stdout:5/71: chown d7/d9/cd 188497 1 2024-04-20T21:32:00.426 INFO:tasks.workunit.client.1.smithi189.stdout:3/53: mknod d2/c13 0 2024-04-20T21:32:00.427 INFO:tasks.workunit.client.0.smithi062.stdout:6/352: rename d0/c51 to d0/d9/c72 0 2024-04-20T21:32:00.429 INFO:tasks.workunit.client.0.smithi062.stdout:2/494: readlink d7/da/d15/d2f/l63 0 2024-04-20T21:32:00.431 INFO:tasks.workunit.client.1.smithi189.stdout:9/86: getdents d9/dc/d17 0 2024-04-20T21:32:00.432 INFO:tasks.workunit.client.0.smithi062.stdout:6/353: creat d0/d4f/f73 x:0 0 0 2024-04-20T21:32:00.437 INFO:tasks.workunit.client.0.smithi062.stdout:7/442: creat d7/d3e/f99 x:0 0 0 2024-04-20T21:32:00.437 INFO:tasks.workunit.client.0.smithi062.stdout:7/443: readlink d7/d3e/l21 0 2024-04-20T21:32:00.438 INFO:tasks.workunit.client.0.smithi062.stdout:7/444: stat d7/d3e 0 2024-04-20T21:32:00.442 INFO:tasks.workunit.client.0.smithi062.stdout:2/495: creat d7/da/d15/d2f/d38/d6a/d7c/fa5 x:0 0 0 2024-04-20T21:32:00.442 INFO:tasks.workunit.client.1.smithi189.stdout:4/53: symlink d2/dd/d11/d14/l16 0 2024-04-20T21:32:00.442 INFO:tasks.workunit.client.1.smithi189.stdout:0/198: unlink d3/df/d30/d37/c1a 0 2024-04-20T21:32:00.442 INFO:tasks.workunit.client.1.smithi189.stdout:7/87: dwrite d5/f14 [0,4194304] 0 2024-04-20T21:32:00.443 INFO:tasks.workunit.client.1.smithi189.stdout:9/87: write f8 [1208241,79248] 0 2024-04-20T21:32:00.443 INFO:tasks.workunit.client.0.smithi062.stdout:7/445: creat d7/d26/f9a x:0 0 0 2024-04-20T21:32:00.444 INFO:tasks.workunit.client.1.smithi189.stdout:3/54: symlink d2/l14 0 2024-04-20T21:32:00.445 INFO:tasks.workunit.client.0.smithi062.stdout:7/446: write d7/f6c [674970,105971] 0 2024-04-20T21:32:00.448 INFO:tasks.workunit.client.1.smithi189.stdout:7/88: dread - d5/f9 zero size 2024-04-20T21:32:00.452 INFO:tasks.workunit.client.1.smithi189.stdout:6/83: dwrite d8/d12/f14 [0,4194304] 0 2024-04-20T21:32:00.454 INFO:tasks.workunit.client.0.smithi062.stdout:7/447: read d7/d26/f47 [434127,110761] 0 2024-04-20T21:32:00.456 INFO:tasks.workunit.client.0.smithi062.stdout:6/354: write d0/d9/d11/d22/f24 [1353609,100300] 0 2024-04-20T21:32:00.456 INFO:tasks.workunit.client.1.smithi189.stdout:0/199: write d3/df/d30/d37/f22 [301226,54080] 0 2024-04-20T21:32:00.460 INFO:tasks.workunit.client.1.smithi189.stdout:4/54: write d2/f8 [1005530,116160] 0 2024-04-20T21:32:00.462 INFO:tasks.workunit.client.1.smithi189.stdout:9/88: unlink d9/d12/l11 0 2024-04-20T21:32:00.464 INFO:tasks.workunit.client.1.smithi189.stdout:0/200: mkdir d3/df/d30/d45 0 2024-04-20T21:32:00.467 INFO:tasks.workunit.client.0.smithi062.stdout:6/355: mknod d0/c74 0 2024-04-20T21:32:00.469 INFO:tasks.workunit.client.0.smithi062.stdout:6/356: stat d0/c74 0 2024-04-20T21:32:00.469 INFO:tasks.workunit.client.1.smithi189.stdout:4/55: write f0 [1958739,70418] 0 2024-04-20T21:32:00.470 INFO:tasks.workunit.client.0.smithi062.stdout:7/448: getdents d7/d3e/d33/d3d 0 2024-04-20T21:32:00.471 INFO:tasks.workunit.client.0.smithi062.stdout:7/449: fdatasync d7/d26/f9a 0 2024-04-20T21:32:00.474 INFO:tasks.workunit.client.1.smithi189.stdout:6/84: symlink d8/d12/d16/l1b 0 2024-04-20T21:32:00.475 INFO:tasks.workunit.client.0.smithi062.stdout:7/450: symlink d7/d3e/d33/d3d/d7f/l9b 0 2024-04-20T21:32:00.479 INFO:tasks.workunit.client.1.smithi189.stdout:0/201: symlink d3/d26/l46 0 2024-04-20T21:32:00.482 INFO:tasks.workunit.client.1.smithi189.stdout:9/89: creat d9/dc/f1c x:0 0 0 2024-04-20T21:32:00.483 INFO:tasks.workunit.client.1.smithi189.stdout:0/202: stat d3/df 0 2024-04-20T21:32:00.485 INFO:tasks.workunit.client.1.smithi189.stdout:4/56: truncate d2/dd/fe 1040842 0 2024-04-20T21:32:00.486 INFO:tasks.workunit.client.0.smithi062.stdout:2/496: dread d7/da/d15/d1b/d3d/f49 [0,4194304] 0 2024-04-20T21:32:00.488 INFO:tasks.workunit.client.1.smithi189.stdout:5/72: dwrite d7/d9/fc [0,4194304] 0 2024-04-20T21:32:00.491 INFO:tasks.workunit.client.0.smithi062.stdout:1/481: sync 2024-04-20T21:32:00.491 INFO:tasks.workunit.client.0.smithi062.stdout:9/443: sync 2024-04-20T21:32:00.493 INFO:tasks.workunit.client.0.smithi062.stdout:1/482: stat dc/d15/d75/l77 0 2024-04-20T21:32:00.493 INFO:tasks.workunit.client.1.smithi189.stdout:7/89: dwrite d5/f6 [0,4194304] 0 2024-04-20T21:32:00.495 INFO:tasks.workunit.client.1.smithi189.stdout:0/203: mknod d3/df/c47 0 2024-04-20T21:32:00.497 INFO:tasks.workunit.client.0.smithi062.stdout:7/451: mknod d7/c9c 0 2024-04-20T21:32:00.498 INFO:tasks.workunit.client.1.smithi189.stdout:4/57: mknod d2/dd/c17 0 2024-04-20T21:32:00.500 INFO:tasks.workunit.client.0.smithi062.stdout:9/444: symlink d16/d2c/d42/l95 0 2024-04-20T21:32:00.501 INFO:tasks.workunit.client.0.smithi062.stdout:1/483: mknod dc/d2d/ca6 0 2024-04-20T21:32:00.502 INFO:tasks.workunit.client.0.smithi062.stdout:6/357: dread d0/fe [4194304,4194304] 0 2024-04-20T21:32:00.502 INFO:tasks.workunit.client.1.smithi189.stdout:3/55: dwrite d2/f5 [4194304,4194304] 0 2024-04-20T21:32:00.502 INFO:tasks.workunit.client.1.smithi189.stdout:7/90: mknod d5/d1d/c22 0 2024-04-20T21:32:00.503 INFO:tasks.workunit.client.0.smithi062.stdout:2/497: truncate d7/da/d15/d2f/f42 314154 0 2024-04-20T21:32:00.504 INFO:tasks.workunit.client.0.smithi062.stdout:9/445: write d16/d2f/d69/f7c [151762,123888] 0 2024-04-20T21:32:00.505 INFO:tasks.workunit.client.1.smithi189.stdout:4/58: fsync d2/dd/fe 0 2024-04-20T21:32:00.509 INFO:tasks.workunit.client.0.smithi062.stdout:2/498: dread - d7/da/d15/d2f/d38/d6a/d7c/fa5 zero size 2024-04-20T21:32:00.510 INFO:tasks.workunit.client.0.smithi062.stdout:1/484: chown dc/d15/d75/d94/d36/c72 26023442 1 2024-04-20T21:32:00.510 INFO:tasks.workunit.client.1.smithi189.stdout:3/56: stat l1 0 2024-04-20T21:32:00.512 INFO:tasks.workunit.client.1.smithi189.stdout:7/91: creat d5/d21/f23 x:0 0 0 2024-04-20T21:32:00.512 INFO:tasks.workunit.client.0.smithi062.stdout:2/499: unlink d7/f12 0 2024-04-20T21:32:00.513 INFO:tasks.workunit.client.1.smithi189.stdout:0/204: write d3/df/d30/d37/f31 [1966416,18600] 0 2024-04-20T21:32:00.516 INFO:tasks.workunit.client.1.smithi189.stdout:7/92: write d5/f9 [997762,76477] 0 2024-04-20T21:32:00.516 INFO:tasks.workunit.client.1.smithi189.stdout:8/226: sync 2024-04-20T21:32:00.518 INFO:tasks.workunit.client.0.smithi062.stdout:9/446: getdents d16/d2c/d5e/d75/d76 0 2024-04-20T21:32:00.518 INFO:tasks.workunit.client.1.smithi189.stdout:4/59: creat d2/f18 x:0 0 0 2024-04-20T21:32:00.518 INFO:tasks.workunit.client.1.smithi189.stdout:6/85: getdents d8/d12/d16 0 2024-04-20T21:32:00.519 INFO:tasks.workunit.client.0.smithi062.stdout:7/452: dwrite d7/f36 [4194304,4194304] 0 2024-04-20T21:32:00.521 INFO:tasks.workunit.client.1.smithi189.stdout:4/60: dread - d2/f18 zero size 2024-04-20T21:32:00.523 INFO:tasks.workunit.client.1.smithi189.stdout:4/61: dread - d2/f18 zero size 2024-04-20T21:32:00.523 INFO:tasks.workunit.client.0.smithi062.stdout:6/358: dwrite d0/d9/d11/d22/f68 [0,4194304] 0 2024-04-20T21:32:00.525 INFO:tasks.workunit.client.0.smithi062.stdout:6/359: write d0/d2c/f30 [4061029,57258] 0 2024-04-20T21:32:00.527 INFO:tasks.workunit.client.0.smithi062.stdout:6/360: chown d0/d9/d11/f63 1 1 2024-04-20T21:32:00.528 INFO:tasks.workunit.client.1.smithi189.stdout:4/62: stat d2/dd/c13 0 2024-04-20T21:32:00.528 INFO:tasks.workunit.client.1.smithi189.stdout:9/90: dwrite f3 [0,4194304] 0 2024-04-20T21:32:00.530 INFO:tasks.workunit.client.0.smithi062.stdout:7/453: mkdir d7/d3e/d33/d3d/d4d/d54/d4f/d9d 0 2024-04-20T21:32:00.534 INFO:tasks.workunit.client.0.smithi062.stdout:6/361: mkdir d0/d2c/d67/d75 0 2024-04-20T21:32:00.535 INFO:tasks.workunit.client.1.smithi189.stdout:5/73: dwrite d7/fb [0,4194304] 0 2024-04-20T21:32:00.538 INFO:tasks.workunit.client.0.smithi062.stdout:9/447: getdents d16 0 2024-04-20T21:32:00.539 INFO:tasks.workunit.client.0.smithi062.stdout:9/448: dread - d16/d2f/f7f zero size 2024-04-20T21:32:00.545 INFO:tasks.workunit.client.0.smithi062.stdout:7/454: link d7/f48 d7/d3e/d33/d3d/d7f/f9e 0 2024-04-20T21:32:00.546 INFO:tasks.workunit.client.0.smithi062.stdout:9/449: fsync fe 0 2024-04-20T21:32:00.562 INFO:tasks.workunit.client.0.smithi062.stdout:2/500: chown d7/da/d28/d4b/d8f 0 1 2024-04-20T21:32:00.564 INFO:tasks.workunit.client.0.smithi062.stdout:9/450: dwrite d16/f32 [4194304,4194304] 0 2024-04-20T21:32:00.566 INFO:tasks.workunit.client.0.smithi062.stdout:2/501: write d7/da/d28/f53 [4016270,122554] 0 2024-04-20T21:32:00.566 INFO:tasks.workunit.client.0.smithi062.stdout:7/455: dwrite d7/d3e/d33/d3d/d7f/f9e [0,4194304] 0 2024-04-20T21:32:00.569 INFO:tasks.workunit.client.0.smithi062.stdout:9/451: mkdir d16/d2c/d5e/d96 0 2024-04-20T21:32:00.572 INFO:tasks.workunit.client.0.smithi062.stdout:2/502: stat d7/da/d15/d1b/f34 0 2024-04-20T21:32:00.573 INFO:tasks.workunit.client.0.smithi062.stdout:6/362: write d0/d1a/d44/f4c [4179505,14288] 0 2024-04-20T21:32:00.573 INFO:tasks.workunit.client.0.smithi062.stdout:7/456: chown d7/c59 44 1 2024-04-20T21:32:00.573 INFO:tasks.workunit.client.0.smithi062.stdout:9/452: symlink d16/d3e/d77/d84/l97 0 2024-04-20T21:32:00.573 INFO:tasks.workunit.client.0.smithi062.stdout:2/503: chown d7/da/d15/d2f/d97 26579048 1 2024-04-20T21:32:00.576 INFO:tasks.workunit.client.0.smithi062.stdout:6/363: write d0/d2c/f2e [5156807,21547] 0 2024-04-20T21:32:00.576 INFO:tasks.workunit.client.0.smithi062.stdout:7/457: mknod d7/d3e/d33/d3d/d7f/c9f 0 2024-04-20T21:32:00.576 INFO:tasks.workunit.client.0.smithi062.stdout:9/453: rmdir d16/d3d 39 2024-04-20T21:32:00.581 INFO:tasks.workunit.client.0.smithi062.stdout:2/504: symlink d7/da/d15/d2f/d38/d6a/la6 0 2024-04-20T21:32:00.589 INFO:tasks.workunit.client.0.smithi062.stdout:2/505: rename d7/da/d28/f53 to d7/da/d15/d2f/d38/d51/fa7 0 2024-04-20T21:32:00.591 INFO:tasks.workunit.client.0.smithi062.stdout:6/364: dread d0/d1a/d44/f4c [0,4194304] 0 2024-04-20T21:32:00.593 INFO:tasks.workunit.client.0.smithi062.stdout:9/454: rename d16/d3e/l5c to d16/d3d/d55/d6e/d7d/l98 0 2024-04-20T21:32:00.593 INFO:tasks.workunit.client.1.smithi189.stdout:9/91: dread f4 [0,4194304] 0 2024-04-20T21:32:00.593 INFO:tasks.workunit.client.0.smithi062.stdout:2/506: mkdir d7/da/d15/d1b/da8 0 2024-04-20T21:32:00.594 INFO:tasks.workunit.client.0.smithi062.stdout:6/365: stat d0/d9/d11/d22/d26/c50 0 2024-04-20T21:32:00.595 INFO:tasks.workunit.client.0.smithi062.stdout:2/507: chown d7/da/d15/f23 2096 1 2024-04-20T21:32:00.596 INFO:tasks.workunit.client.1.smithi189.stdout:9/92: read f8 [517418,3002] 0 2024-04-20T21:32:00.597 INFO:tasks.workunit.client.0.smithi062.stdout:6/366: truncate d0/d9/d11/f25 1448151 0 2024-04-20T21:32:00.601 INFO:tasks.workunit.client.0.smithi062.stdout:6/367: symlink d0/d9/d11/d5c/l76 0 2024-04-20T21:32:00.602 INFO:tasks.workunit.client.1.smithi189.stdout:4/63: dwrite d2/fb [0,4194304] 0 2024-04-20T21:32:00.602 INFO:tasks.workunit.client.0.smithi062.stdout:7/458: fsync d7/f48 0 2024-04-20T21:32:00.604 INFO:tasks.workunit.client.1.smithi189.stdout:0/205: dwrite d3/df/f15 [0,4194304] 0 2024-04-20T21:32:00.610 INFO:tasks.workunit.client.0.smithi062.stdout:7/459: dread - d7/d3e/d33/d3d/d4d/d54/f3f zero size 2024-04-20T21:32:00.616 INFO:tasks.workunit.client.0.smithi062.stdout:6/368: truncate d0/d9/f16 2997623 0 2024-04-20T21:32:00.618 INFO:tasks.workunit.client.1.smithi189.stdout:9/93: dwrite f8 [0,4194304] 0 2024-04-20T21:32:00.625 INFO:tasks.workunit.client.1.smithi189.stdout:4/64: dwrite f0 [0,4194304] 0 2024-04-20T21:32:00.626 INFO:tasks.workunit.client.0.smithi062.stdout:2/508: dwrite d7/da/d15/d1b/f2c [0,4194304] 0 2024-04-20T21:32:00.634 INFO:tasks.workunit.client.0.smithi062.stdout:7/460: truncate d7/d3e/d33/f35 2804218 0 2024-04-20T21:32:00.637 INFO:tasks.workunit.client.0.smithi062.stdout:6/369: dread d0/d1a/d44/f4c [0,4194304] 0 2024-04-20T21:32:00.637 INFO:tasks.workunit.client.0.smithi062.stdout:2/509: rmdir d7/da/d15/d2f/d38/d6b/d9c 0 2024-04-20T21:32:00.637 INFO:tasks.workunit.client.0.smithi062.stdout:7/461: dread - d7/d3e/f71 zero size 2024-04-20T21:32:00.638 INFO:tasks.workunit.client.0.smithi062.stdout:9/455: dread d16/f1d [0,4194304] 0 2024-04-20T21:32:00.640 INFO:tasks.workunit.client.0.smithi062.stdout:7/462: chown d7/f24 1155310 1 2024-04-20T21:32:00.641 INFO:tasks.workunit.client.1.smithi189.stdout:8/227: mknod da/de/d1e/d21/c51 0 2024-04-20T21:32:00.642 INFO:tasks.workunit.client.0.smithi062.stdout:6/370: unlink d0/c46 0 2024-04-20T21:32:00.643 INFO:tasks.workunit.client.0.smithi062.stdout:7/463: chown d7/d3e/d33/d3d/d5e/l67 840948 1 2024-04-20T21:32:00.644 INFO:tasks.workunit.client.0.smithi062.stdout:9/456: creat d16/d3e/d77/d84/f99 x:0 0 0 2024-04-20T21:32:00.645 INFO:tasks.workunit.client.1.smithi189.stdout:1/73: sync 2024-04-20T21:32:00.645 INFO:tasks.workunit.client.1.smithi189.stdout:2/70: sync 2024-04-20T21:32:00.646 INFO:tasks.workunit.client.0.smithi062.stdout:6/371: chown d0/d4f/c53 1665 1 2024-04-20T21:32:00.646 INFO:tasks.workunit.client.1.smithi189.stdout:5/74: mkdir d7/d9/d15 0 2024-04-20T21:32:00.649 INFO:tasks.workunit.client.1.smithi189.stdout:4/65: creat d2/f19 x:0 0 0 2024-04-20T21:32:00.649 INFO:tasks.workunit.client.1.smithi189.stdout:5/75: write d7/fb [3734819,124510] 0 2024-04-20T21:32:00.651 INFO:tasks.workunit.client.0.smithi062.stdout:6/372: truncate d0/d1a/d44/f4b 287754 0 2024-04-20T21:32:00.651 INFO:tasks.workunit.client.1.smithi189.stdout:3/57: link d2/d8/f10 d2/f15 0 2024-04-20T21:32:00.651 INFO:tasks.workunit.client.1.smithi189.stdout:4/66: read d2/f4 [453738,89984] 0 2024-04-20T21:32:00.652 INFO:tasks.workunit.client.1.smithi189.stdout:5/76: dread - d7/f13 zero size 2024-04-20T21:32:00.653 INFO:tasks.workunit.client.1.smithi189.stdout:1/74: creat d8/f1a x:0 0 0 2024-04-20T21:32:00.653 INFO:tasks.workunit.client.1.smithi189.stdout:5/77: readlink l0 0 2024-04-20T21:32:00.655 INFO:tasks.workunit.client.1.smithi189.stdout:2/71: mkdir d1/d11 0 2024-04-20T21:32:00.656 INFO:tasks.workunit.client.0.smithi062.stdout:6/373: mknod d0/d2c/c77 0 2024-04-20T21:32:00.656 INFO:tasks.workunit.client.0.smithi062.stdout:7/464: rename d7/d3e/f20 to d7/fa0 0 2024-04-20T21:32:00.656 INFO:tasks.workunit.client.1.smithi189.stdout:2/72: chown d1/cb 261716435 1 2024-04-20T21:32:00.659 INFO:tasks.workunit.client.0.smithi062.stdout:6/374: fdatasync d0/d9/d11/f48 0 2024-04-20T21:32:00.661 INFO:tasks.workunit.client.0.smithi062.stdout:6/375: chown d0/l3f 206649 1 2024-04-20T21:32:00.662 INFO:tasks.workunit.client.0.smithi062.stdout:2/510: getdents d7/da/d15/d2f/d38/d6b 0 2024-04-20T21:32:00.665 INFO:tasks.workunit.client.0.smithi062.stdout:9/457: write d16/d2c/d42/f50 [870944,40588] 0 2024-04-20T21:32:00.666 INFO:tasks.workunit.client.0.smithi062.stdout:2/511: write d7/d39/f5f [4145042,87215] 0 2024-04-20T21:32:00.669 INFO:tasks.workunit.client.0.smithi062.stdout:6/376: getdents d0/d52 0 2024-04-20T21:32:00.670 INFO:tasks.workunit.client.0.smithi062.stdout:2/512: fdatasync d7/da/d15/d1b/f2c 0 2024-04-20T21:32:00.671 INFO:tasks.workunit.client.0.smithi062.stdout:8/479: sync 2024-04-20T21:32:00.671 INFO:tasks.workunit.client.0.smithi062.stdout:7/465: write d7/f2a [528226,122270] 0 2024-04-20T21:32:00.673 INFO:tasks.workunit.client.0.smithi062.stdout:7/466: dread - d7/d26/f9a zero size 2024-04-20T21:32:00.674 INFO:tasks.workunit.client.0.smithi062.stdout:9/458: write d16/f3b [1301539,82881] 0 2024-04-20T21:32:00.675 INFO:tasks.workunit.client.0.smithi062.stdout:6/377: mknod d0/d9/c78 0 2024-04-20T21:32:00.681 INFO:tasks.workunit.client.0.smithi062.stdout:7/467: link d7/d26/l4b d7/d3e/d33/d3d/d4d/d83/d94/la1 0 2024-04-20T21:32:00.682 INFO:tasks.workunit.client.0.smithi062.stdout:7/468: readlink d7/l1d 0 2024-04-20T21:32:00.683 INFO:tasks.workunit.client.0.smithi062.stdout:6/378: link d0/d1a/f59 d0/f79 0 2024-04-20T21:32:00.683 INFO:tasks.workunit.client.0.smithi062.stdout:8/480: truncate d7/d25/f42 3059084 0 2024-04-20T21:32:00.684 INFO:tasks.workunit.client.0.smithi062.stdout:6/379: stat d0/d4f 0 2024-04-20T21:32:00.687 INFO:tasks.workunit.client.0.smithi062.stdout:7/469: rmdir d7/d3e/d33/d3d/d4d 39 2024-04-20T21:32:00.687 INFO:tasks.workunit.client.0.smithi062.stdout:8/481: read - d7/d25/d70/fa0 zero size 2024-04-20T21:32:00.690 INFO:tasks.workunit.client.0.smithi062.stdout:2/513: dwrite d7/da/d15/f85 [0,4194304] 0 2024-04-20T21:32:00.692 INFO:tasks.workunit.client.0.smithi062.stdout:9/459: dwrite d16/f3b [0,4194304] 0 2024-04-20T21:32:00.697 INFO:tasks.workunit.client.0.smithi062.stdout:6/380: truncate d0/d9/d11/f25 647964 0 2024-04-20T21:32:00.699 INFO:tasks.workunit.client.0.smithi062.stdout:9/460: write fe [4117896,128626] 0 2024-04-20T21:32:00.699 INFO:tasks.workunit.client.0.smithi062.stdout:2/514: creat d7/da/d15/d1b/d3d/d84/fa9 x:0 0 0 2024-04-20T21:32:00.700 INFO:tasks.workunit.client.0.smithi062.stdout:6/381: fsync d0/d4f/f73 0 2024-04-20T21:32:00.702 INFO:tasks.workunit.client.0.smithi062.stdout:6/382: write d0/d2c/f2e [5175279,11798] 0 2024-04-20T21:32:00.705 INFO:tasks.workunit.client.0.smithi062.stdout:9/461: write d16/d2f/f7f [636209,9412] 0 2024-04-20T21:32:00.711 INFO:tasks.workunit.client.0.smithi062.stdout:9/462: unlink d16/d2f/f7f 0 2024-04-20T21:32:00.714 INFO:tasks.workunit.client.0.smithi062.stdout:2/515: dread d7/da/d15/d2f/fa4 [0,4194304] 0 2024-04-20T21:32:00.715 INFO:tasks.workunit.client.0.smithi062.stdout:7/470: dwrite d7/d3e/f41 [0,4194304] 0 2024-04-20T21:32:00.718 INFO:tasks.workunit.client.0.smithi062.stdout:9/463: read d16/f3a [2998301,19433] 0 2024-04-20T21:32:00.722 INFO:tasks.workunit.client.0.smithi062.stdout:7/471: symlink d7/d26/la2 0 2024-04-20T21:32:00.727 INFO:tasks.workunit.client.0.smithi062.stdout:9/464: truncate d16/d2f/f8f 70067 0 2024-04-20T21:32:00.727 INFO:tasks.workunit.client.0.smithi062.stdout:8/482: dwrite d7/d25/f93 [0,4194304] 0 2024-04-20T21:32:00.730 INFO:tasks.workunit.client.0.smithi062.stdout:7/472: chown d7/c23 127656 1 2024-04-20T21:32:00.731 INFO:tasks.workunit.client.0.smithi062.stdout:2/516: symlink d7/da/d15/d1b/d3d/d84/d9b/laa 0 2024-04-20T21:32:00.733 INFO:tasks.workunit.client.0.smithi062.stdout:6/383: dwrite d0/fe [0,4194304] 0 2024-04-20T21:32:00.733 INFO:tasks.workunit.client.0.smithi062.stdout:9/465: chown d16/l37 46108243 1 2024-04-20T21:32:00.736 INFO:tasks.workunit.client.0.smithi062.stdout:3/459: sync 2024-04-20T21:32:00.736 INFO:tasks.workunit.client.0.smithi062.stdout:2/517: creat d7/da/fab x:0 0 0 2024-04-20T21:32:00.737 INFO:tasks.workunit.client.0.smithi062.stdout:7/473: fdatasync d7/d3e/d33/d3d/d5e/f61 0 2024-04-20T21:32:00.740 INFO:tasks.workunit.client.0.smithi062.stdout:9/466: creat d16/d3e/d77/d84/f9a x:0 0 0 2024-04-20T21:32:00.743 INFO:tasks.workunit.client.0.smithi062.stdout:6/384: creat d0/d2c/d67/d75/f7a x:0 0 0 2024-04-20T21:32:00.746 INFO:tasks.workunit.client.0.smithi062.stdout:2/518: mkdir d7/da/d15/d2f/d93/dac 0 2024-04-20T21:32:00.748 INFO:tasks.workunit.client.0.smithi062.stdout:9/467: mkdir d16/d3d/d55/d6e/d9b 0 2024-04-20T21:32:00.748 INFO:tasks.workunit.client.1.smithi189.stdout:3/58: symlink d2/d3/l16 0 2024-04-20T21:32:00.749 INFO:tasks.workunit.client.1.smithi189.stdout:4/67: creat d2/f1a x:0 0 0 2024-04-20T21:32:00.749 INFO:tasks.workunit.client.1.smithi189.stdout:0/206: rename d3/df/d30/d37/f23 to d3/f48 0 2024-04-20T21:32:00.751 INFO:tasks.workunit.client.0.smithi062.stdout:8/483: dwrite d7/d25/d76/f9c [0,4194304] 0 2024-04-20T21:32:00.751 INFO:tasks.workunit.client.1.smithi189.stdout:3/59: chown d2/c11 9105286 1 2024-04-20T21:32:00.753 INFO:tasks.workunit.client.0.smithi062.stdout:7/474: truncate d7/d3e/d33/d3d/d4d/d83/f81 285491 0 2024-04-20T21:32:00.754 INFO:tasks.workunit.client.0.smithi062.stdout:9/468: creat d16/d2c/d62/f9c x:0 0 0 2024-04-20T21:32:00.756 INFO:tasks.workunit.client.1.smithi189.stdout:1/75: rmdir d8 39 2024-04-20T21:32:00.757 INFO:tasks.workunit.client.0.smithi062.stdout:3/460: dread d0/d1/d10/f79 [0,4194304] 0 2024-04-20T21:32:00.758 INFO:tasks.workunit.client.0.smithi062.stdout:9/469: stat d16/d3d/l82 0 2024-04-20T21:32:00.758 INFO:tasks.workunit.client.0.smithi062.stdout:8/484: chown d7/d9/d29/c7a 299 1 2024-04-20T21:32:00.759 INFO:tasks.workunit.client.0.smithi062.stdout:7/475: creat d7/d3e/d33/d3d/d4d/fa3 x:0 0 0 2024-04-20T21:32:00.763 INFO:tasks.workunit.client.0.smithi062.stdout:3/461: read d0/d1c/f51 [378342,22895] 0 2024-04-20T21:32:00.766 INFO:tasks.workunit.client.1.smithi189.stdout:5/78: rename d7/d9/cd to d7/c16 0 2024-04-20T21:32:00.766 INFO:tasks.workunit.client.1.smithi189.stdout:9/94: dwrite f4 [0,4194304] 0 2024-04-20T21:32:00.767 INFO:tasks.workunit.client.1.smithi189.stdout:5/79: chown d7/d9 0 1 2024-04-20T21:32:00.770 INFO:tasks.workunit.client.0.smithi062.stdout:9/470: creat d16/d3d/d55/d6e/f9d x:0 0 0 2024-04-20T21:32:00.771 INFO:tasks.workunit.client.0.smithi062.stdout:7/476: dread d7/d26/f2c [0,4194304] 0 2024-04-20T21:32:00.773 INFO:tasks.workunit.client.1.smithi189.stdout:3/60: getdents d2/d8 0 2024-04-20T21:32:00.777 INFO:tasks.workunit.client.0.smithi062.stdout:3/462: unlink d0/d1/c58 0 2024-04-20T21:32:00.779 INFO:tasks.workunit.client.1.smithi189.stdout:1/76: write d8/f9 [1782151,110158] 0 2024-04-20T21:32:00.780 INFO:tasks.workunit.client.0.smithi062.stdout:8/485: dwrite d7/d25/d3b/d58/d8b/d24/f75 [0,4194304] 0 2024-04-20T21:32:00.780 INFO:tasks.workunit.client.0.smithi062.stdout:3/463: stat d0/d1c/d52/c96 0 2024-04-20T21:32:00.780 INFO:tasks.workunit.client.0.smithi062.stdout:7/477: rename d7/f5f to d7/d3e/d33/d3d/d4d/fa4 0 2024-04-20T21:32:00.782 INFO:tasks.workunit.client.0.smithi062.stdout:9/471: dread d16/d2f/d69/f7c [0,4194304] 0 2024-04-20T21:32:00.783 INFO:tasks.workunit.client.1.smithi189.stdout:5/80: rename d7/f13 to d7/d9/d15/f17 0 2024-04-20T21:32:00.784 INFO:tasks.workunit.client.0.smithi062.stdout:3/464: chown d0/l13 23537 1 2024-04-20T21:32:00.784 INFO:tasks.workunit.client.0.smithi062.stdout:7/478: truncate d7/d26/f9a 152796 0 2024-04-20T21:32:00.787 INFO:tasks.workunit.client.0.smithi062.stdout:9/472: read d16/d2c/d42/f50 [3101775,11893] 0 2024-04-20T21:32:00.790 INFO:tasks.workunit.client.1.smithi189.stdout:1/77: creat d8/f1b x:0 0 0 2024-04-20T21:32:00.790 INFO:tasks.workunit.client.0.smithi062.stdout:8/486: creat d7/d25/d3b/d58/d8b/d96/fa4 x:0 0 0 2024-04-20T21:32:00.792 INFO:tasks.workunit.client.1.smithi189.stdout:5/81: chown d7/fb 10 1 2024-04-20T21:32:00.796 INFO:tasks.workunit.client.1.smithi189.stdout:3/61: write d2/f6 [250241,56534] 0 2024-04-20T21:32:00.796 INFO:tasks.workunit.client.1.smithi189.stdout:1/78: mknod d8/c1c 0 2024-04-20T21:32:00.798 INFO:tasks.workunit.client.1.smithi189.stdout:0/207: dwrite d3/df/d30/d37/d29/f41 [0,4194304] 0 2024-04-20T21:32:00.799 INFO:tasks.workunit.client.1.smithi189.stdout:9/95: dread f7 [0,4194304] 0 2024-04-20T21:32:00.803 INFO:tasks.workunit.client.0.smithi062.stdout:8/487: link d7/d25/d5c/f94 d7/fa5 0 2024-04-20T21:32:00.807 INFO:tasks.workunit.client.1.smithi189.stdout:9/96: chown f7 358125978 1 2024-04-20T21:32:00.808 INFO:tasks.workunit.client.1.smithi189.stdout:3/62: write d2/fd [4251562,34017] 0 2024-04-20T21:32:00.808 INFO:tasks.workunit.client.1.smithi189.stdout:4/68: dwrite d2/dd/fe [0,4194304] 0 2024-04-20T21:32:00.809 INFO:tasks.workunit.client.0.smithi062.stdout:9/473: dwrite d16/d3d/d55/d6e/f92 [0,4194304] 0 2024-04-20T21:32:00.809 INFO:tasks.workunit.client.0.smithi062.stdout:8/488: chown d7/d30/f95 12342 1 2024-04-20T21:32:00.811 INFO:tasks.workunit.client.0.smithi062.stdout:7/479: dread d7/d3e/d33/f35 [0,4194304] 0 2024-04-20T21:32:00.812 INFO:tasks.workunit.client.1.smithi189.stdout:1/79: mknod d8/c1d 0 2024-04-20T21:32:00.812 INFO:tasks.workunit.client.1.smithi189.stdout:2/73: dwrite d1/f4 [0,4194304] 0 2024-04-20T21:32:00.814 INFO:tasks.workunit.client.1.smithi189.stdout:0/208: write d3/f1f [1683478,39464] 0 2024-04-20T21:32:00.821 INFO:tasks.workunit.client.0.smithi062.stdout:8/489: creat d7/d25/d3b/d58/fa6 x:0 0 0 2024-04-20T21:32:00.821 INFO:tasks.workunit.client.0.smithi062.stdout:9/474: creat d16/f9e x:0 0 0 2024-04-20T21:32:00.821 INFO:tasks.workunit.client.1.smithi189.stdout:9/97: creat d9/dc/f1d x:0 0 0 2024-04-20T21:32:00.821 INFO:tasks.workunit.client.1.smithi189.stdout:4/69: mknod d2/dd/d11/c1b 0 2024-04-20T21:32:00.821 INFO:tasks.workunit.client.1.smithi189.stdout:1/80: mkdir d8/d1e 0 2024-04-20T21:32:00.823 INFO:tasks.workunit.client.1.smithi189.stdout:3/63: mknod d2/d8/c17 0 2024-04-20T21:32:00.827 INFO:tasks.workunit.client.1.smithi189.stdout:7/93: sync 2024-04-20T21:32:00.827 INFO:tasks.workunit.client.1.smithi189.stdout:6/86: sync 2024-04-20T21:32:00.827 INFO:tasks.workunit.client.1.smithi189.stdout:5/82: dread d7/fb [0,4194304] 0 2024-04-20T21:32:00.828 INFO:tasks.workunit.client.1.smithi189.stdout:3/64: write d2/f15 [1289507,104329] 0 2024-04-20T21:32:00.828 INFO:tasks.workunit.client.1.smithi189.stdout:2/74: fdatasync d1/f6 0 2024-04-20T21:32:00.830 INFO:tasks.workunit.client.0.smithi062.stdout:9/475: chown d16/d2c/d42/l45 420 1 2024-04-20T21:32:00.830 INFO:tasks.workunit.client.1.smithi189.stdout:1/81: truncate d8/f9 4729278 0 2024-04-20T21:32:00.831 INFO:tasks.workunit.client.1.smithi189.stdout:7/94: read d5/fd [705724,7750] 0 2024-04-20T21:32:00.836 INFO:tasks.workunit.client.0.smithi062.stdout:8/490: getdents d7/d25/d70 0 2024-04-20T21:32:00.837 INFO:tasks.workunit.client.0.smithi062.stdout:9/476: symlink d16/d3d/d55/d6e/l9f 0 2024-04-20T21:32:00.838 INFO:tasks.workunit.client.1.smithi189.stdout:6/87: creat d8/d12/d16/f1c x:0 0 0 2024-04-20T21:32:00.839 INFO:tasks.workunit.client.1.smithi189.stdout:0/209: creat d3/df/d28/d2b/f49 x:0 0 0 2024-04-20T21:32:00.840 INFO:tasks.workunit.client.1.smithi189.stdout:2/75: creat d1/f12 x:0 0 0 2024-04-20T21:32:00.840 INFO:tasks.workunit.client.0.smithi062.stdout:7/480: dwrite d7/f55 [4194304,4194304] 0 2024-04-20T21:32:00.841 INFO:tasks.workunit.client.0.smithi062.stdout:3/465: dread d0/d1/fd [0,4194304] 0 2024-04-20T21:32:00.842 INFO:tasks.workunit.client.0.smithi062.stdout:8/491: chown d7/d25/d3b/d58/d8b/f3f 91267099 1 2024-04-20T21:32:00.842 INFO:tasks.workunit.client.1.smithi189.stdout:5/83: rename d7/d9/d15 to d7/d14/d18 0 2024-04-20T21:32:00.844 INFO:tasks.workunit.client.1.smithi189.stdout:2/76: chown d1/fd 27830 1 2024-04-20T21:32:00.845 INFO:tasks.workunit.client.1.smithi189.stdout:5/84: readlink d7/l12 0 2024-04-20T21:32:00.847 INFO:tasks.workunit.client.0.smithi062.stdout:7/481: write d7/d3e/d33/d3d/d4d/d54/f2e [3809878,84432] 0 2024-04-20T21:32:00.847 INFO:tasks.workunit.client.0.smithi062.stdout:8/492: write d7/d25/d5c/f72 [604467,85733] 0 2024-04-20T21:32:00.851 INFO:tasks.workunit.client.1.smithi189.stdout:1/82: mknod d8/d1e/c1f 0 2024-04-20T21:32:00.852 INFO:tasks.workunit.client.1.smithi189.stdout:2/77: truncate d1/f9 501134 0 2024-04-20T21:32:00.853 INFO:tasks.workunit.client.0.smithi062.stdout:7/482: rmdir d7/d3e/d33/d3d/d4d/d54 39 2024-04-20T21:32:00.857 INFO:tasks.workunit.client.1.smithi189.stdout:3/65: rename d2/l14 to d2/l18 0 2024-04-20T21:32:00.861 INFO:tasks.workunit.client.1.smithi189.stdout:9/98: dread f8 [0,4194304] 0 2024-04-20T21:32:00.862 INFO:tasks.workunit.client.1.smithi189.stdout:4/70: dwrite d2/f10 [0,4194304] 0 2024-04-20T21:32:00.864 INFO:tasks.workunit.client.1.smithi189.stdout:5/85: mkdir d7/d19 0 2024-04-20T21:32:00.865 INFO:tasks.workunit.client.1.smithi189.stdout:6/88: write d8/f1a [3655521,37646] 0 2024-04-20T21:32:00.869 INFO:tasks.workunit.client.1.smithi189.stdout:0/210: link d3/df/d28/l40 d3/df/d30/d45/l4a 0 2024-04-20T21:32:00.872 INFO:tasks.workunit.client.1.smithi189.stdout:7/95: dwrite f2 [0,4194304] 0 2024-04-20T21:32:00.877 INFO:tasks.workunit.client.1.smithi189.stdout:2/78: mknod d1/d11/c13 0 2024-04-20T21:32:00.877 INFO:tasks.workunit.client.1.smithi189.stdout:4/71: write f0 [4995175,89733] 0 2024-04-20T21:32:00.878 INFO:tasks.workunit.client.1.smithi189.stdout:9/99: chown d9/dc/c16 378896 1 2024-04-20T21:32:00.878 INFO:tasks.workunit.client.1.smithi189.stdout:8/228: sync 2024-04-20T21:32:00.878 INFO:tasks.workunit.client.1.smithi189.stdout:1/83: truncate d8/f11 601475 0 2024-04-20T21:32:00.879 INFO:tasks.workunit.client.1.smithi189.stdout:6/89: write d8/d12/d16/f1c [534374,10228] 0 2024-04-20T21:32:00.879 INFO:tasks.workunit.client.0.smithi062.stdout:5/476: sync 2024-04-20T21:32:00.879 INFO:tasks.workunit.client.0.smithi062.stdout:4/500: sync 2024-04-20T21:32:00.879 INFO:tasks.workunit.client.0.smithi062.stdout:0/431: sync 2024-04-20T21:32:00.879 INFO:tasks.workunit.client.0.smithi062.stdout:1/485: sync 2024-04-20T21:32:00.881 INFO:tasks.workunit.client.1.smithi189.stdout:4/72: write d2/f8 [536792,43284] 0 2024-04-20T21:32:00.883 INFO:tasks.workunit.client.1.smithi189.stdout:7/96: creat d5/f24 x:0 0 0 2024-04-20T21:32:00.885 INFO:tasks.workunit.client.0.smithi062.stdout:7/483: mknod d7/d63/ca5 0 2024-04-20T21:32:00.887 INFO:tasks.workunit.client.0.smithi062.stdout:5/477: readlink d8/l27 0 2024-04-20T21:32:00.888 INFO:tasks.workunit.client.1.smithi189.stdout:2/79: write d1/f4 [5220415,8896] 0 2024-04-20T21:32:00.889 INFO:tasks.workunit.client.1.smithi189.stdout:9/100: readlink d9/dc/d17/l18 0 2024-04-20T21:32:00.890 INFO:tasks.workunit.client.0.smithi062.stdout:8/493: creat d7/d25/d3b/d58/d8b/d24/fa7 x:0 0 0 2024-04-20T21:32:00.893 INFO:tasks.workunit.client.1.smithi189.stdout:4/73: mknod d2/dd/c1c 0 2024-04-20T21:32:00.894 INFO:tasks.workunit.client.1.smithi189.stdout:9/101: dread - d9/dc/f1c zero size 2024-04-20T21:32:00.895 INFO:tasks.workunit.client.0.smithi062.stdout:0/432: mkdir d7/d1c/d15/d17/d37/d95 0 2024-04-20T21:32:00.895 INFO:tasks.workunit.client.0.smithi062.stdout:5/478: truncate d8/d10/d3e/f94 423229 0 2024-04-20T21:32:00.895 INFO:tasks.workunit.client.1.smithi189.stdout:1/84: mknod d8/d1e/c20 0 2024-04-20T21:32:00.896 INFO:tasks.workunit.client.0.smithi062.stdout:8/494: write d7/d25/f9d [266766,118214] 0 2024-04-20T21:32:00.898 INFO:tasks.workunit.client.0.smithi062.stdout:3/466: dwrite d0/d1b/f57 [0,4194304] 0 2024-04-20T21:32:00.898 INFO:tasks.workunit.client.1.smithi189.stdout:2/80: mknod d1/d11/c14 0 2024-04-20T21:32:00.899 INFO:tasks.workunit.client.0.smithi062.stdout:1/486: chown dc/d15/d75/d94/c25 1 1 2024-04-20T21:32:00.900 INFO:tasks.workunit.client.1.smithi189.stdout:0/211: getdents d3/df/d28/d2b/d2c 0 2024-04-20T21:32:00.901 INFO:tasks.workunit.client.0.smithi062.stdout:3/467: read - d0/d1b/f87 zero size 2024-04-20T21:32:00.901 INFO:tasks.workunit.client.1.smithi189.stdout:1/85: dread - d8/f1b zero size 2024-04-20T21:32:00.901 INFO:tasks.workunit.client.1.smithi189.stdout:9/102: chown d9/dc/c16 27914573 1 2024-04-20T21:32:00.902 INFO:tasks.workunit.client.1.smithi189.stdout:4/74: symlink d2/dd/d11/d14/l1d 0 2024-04-20T21:32:00.903 INFO:tasks.workunit.client.0.smithi062.stdout:3/468: stat d0/d1b/f4d 0 2024-04-20T21:32:00.907 INFO:tasks.workunit.client.1.smithi189.stdout:0/212: creat d3/df/d30/d37/f4b x:0 0 0 2024-04-20T21:32:00.907 INFO:tasks.workunit.client.1.smithi189.stdout:5/86: dread d7/d9/fc [0,4194304] 0 2024-04-20T21:32:00.908 INFO:tasks.workunit.client.0.smithi062.stdout:5/479: mkdir d8/d10/d3e/d8a/d9a 0 2024-04-20T21:32:00.909 INFO:tasks.workunit.client.1.smithi189.stdout:3/66: dwrite d2/f7 [0,4194304] 0 2024-04-20T21:32:00.909 INFO:tasks.workunit.client.0.smithi062.stdout:4/501: dwrite da/fb [0,4194304] 0 2024-04-20T21:32:00.910 INFO:tasks.workunit.client.1.smithi189.stdout:2/81: creat d1/d11/f15 x:0 0 0 2024-04-20T21:32:00.911 INFO:tasks.workunit.client.1.smithi189.stdout:0/213: write d3/f27 [730675,107345] 0 2024-04-20T21:32:00.913 INFO:tasks.workunit.client.0.smithi062.stdout:1/487: rename dc/c44 to dc/d27/d9a/da0/ca7 0 2024-04-20T21:32:00.917 INFO:tasks.workunit.client.1.smithi189.stdout:2/82: fdatasync d1/f9 0 2024-04-20T21:32:00.919 INFO:tasks.workunit.client.0.smithi062.stdout:8/495: creat d7/fa8 x:0 0 0 2024-04-20T21:32:00.921 INFO:tasks.workunit.client.1.smithi189.stdout:3/67: fdatasync d2/f15 0 2024-04-20T21:32:00.924 INFO:tasks.workunit.client.1.smithi189.stdout:0/214: read d3/f1f [696367,65981] 0 2024-04-20T21:32:00.925 INFO:tasks.workunit.client.0.smithi062.stdout:4/502: mkdir da/dd/d15/daf 0 2024-04-20T21:32:00.927 INFO:tasks.workunit.client.1.smithi189.stdout:9/103: link d9/dc/d17/l18 d9/l1e 0 2024-04-20T21:32:00.930 INFO:tasks.workunit.client.1.smithi189.stdout:9/104: stat f7 0 2024-04-20T21:32:00.931 INFO:tasks.workunit.client.1.smithi189.stdout:3/68: rmdir d2 39 2024-04-20T21:32:00.932 INFO:tasks.workunit.client.0.smithi062.stdout:3/469: symlink d0/d1c/d52/d97/l9d 0 2024-04-20T21:32:00.933 INFO:tasks.workunit.client.1.smithi189.stdout:4/75: write d2/fb [4949123,53369] 0 2024-04-20T21:32:00.935 INFO:tasks.workunit.client.1.smithi189.stdout:1/86: truncate d8/f11 645192 0 2024-04-20T21:32:00.938 INFO:tasks.workunit.client.1.smithi189.stdout:4/76: write d2/f1a [779808,18277] 0 2024-04-20T21:32:00.939 INFO:tasks.workunit.client.1.smithi189.stdout:9/105: write d9/fb [2285285,94334] 0 2024-04-20T21:32:00.940 INFO:tasks.workunit.client.1.smithi189.stdout:6/90: dwrite f5 [0,4194304] 0 2024-04-20T21:32:00.943 INFO:tasks.workunit.client.0.smithi062.stdout:7/484: dwrite d7/d26/f2f [0,4194304] 0 2024-04-20T21:32:00.943 INFO:tasks.workunit.client.1.smithi189.stdout:1/87: mkdir d8/d21 0 2024-04-20T21:32:00.944 INFO:tasks.workunit.client.1.smithi189.stdout:3/69: write d2/f15 [3558204,7821] 0 2024-04-20T21:32:00.944 INFO:tasks.workunit.client.0.smithi062.stdout:8/496: fsync d7/d25/d3b/d58/d8b/d24/f20 0 2024-04-20T21:32:00.945 INFO:tasks.workunit.client.0.smithi062.stdout:0/433: write d7/d1c/f63 [585194,92198] 0 2024-04-20T21:32:00.946 INFO:tasks.workunit.client.1.smithi189.stdout:7/97: dwrite d5/f1e [0,4194304] 0 2024-04-20T21:32:00.947 INFO:tasks.workunit.client.0.smithi062.stdout:4/503: dread da/dd/df/d27/d3f/d82/f8b [0,4194304] 0 2024-04-20T21:32:00.948 INFO:tasks.workunit.client.0.smithi062.stdout:4/504: fsync da/d56/d71/f93 0 2024-04-20T21:32:00.952 INFO:tasks.workunit.client.0.smithi062.stdout:0/434: symlink d7/d1c/d15/d17/d1b/d74/l96 0 2024-04-20T21:32:00.952 INFO:tasks.workunit.client.0.smithi062.stdout:1/488: getdents dc/d27/d9a/da0 0 2024-04-20T21:32:00.953 INFO:tasks.workunit.client.1.smithi189.stdout:3/70: unlink d2/d8/la 0 2024-04-20T21:32:00.953 INFO:tasks.workunit.client.1.smithi189.stdout:1/88: dread - d8/df/f15 zero size 2024-04-20T21:32:00.953 INFO:tasks.workunit.client.1.smithi189.stdout:5/87: dread d7/fb [0,4194304] 0 2024-04-20T21:32:00.956 INFO:tasks.workunit.client.0.smithi062.stdout:8/497: mkdir d7/da9 0 2024-04-20T21:32:00.960 INFO:tasks.workunit.client.1.smithi189.stdout:7/98: write d5/f24 [768201,91104] 0 2024-04-20T21:32:00.960 INFO:tasks.workunit.client.1.smithi189.stdout:3/71: rmdir d2/d3 39 2024-04-20T21:32:00.962 INFO:tasks.workunit.client.0.smithi062.stdout:1/489: mknod dc/d2d/ca8 0 2024-04-20T21:32:00.969 INFO:tasks.workunit.client.1.smithi189.stdout:2/83: getdents d1/d11 0 2024-04-20T21:32:00.969 INFO:tasks.workunit.client.1.smithi189.stdout:3/72: chown d2/c13 6 1 2024-04-20T21:32:00.970 INFO:tasks.workunit.client.0.smithi062.stdout:5/480: write d8/d38/d7a/d4c/f7f [572608,35740] 0 2024-04-20T21:32:00.976 INFO:tasks.workunit.client.0.smithi062.stdout:0/435: creat d7/f97 x:0 0 0 2024-04-20T21:32:00.977 INFO:tasks.workunit.client.0.smithi062.stdout:5/481: truncate d8/d1e/d2c/d6c/d56/f92 149628 0 2024-04-20T21:32:00.977 INFO:tasks.workunit.client.0.smithi062.stdout:8/498: rename d7/d9/d29/f60 to d7/d9/faa 0 2024-04-20T21:32:00.979 INFO:tasks.workunit.client.1.smithi189.stdout:7/99: mkdir d5/d21/d25 0 2024-04-20T21:32:00.979 INFO:tasks.workunit.client.1.smithi189.stdout:9/106: dwrite d9/dc/f1d [0,4194304] 0 2024-04-20T21:32:00.981 INFO:tasks.workunit.client.0.smithi062.stdout:8/499: chown d7/d25/d3b/d58/d8b/f1e 164 1 2024-04-20T21:32:00.982 INFO:tasks.workunit.client.1.smithi189.stdout:0/215: dwrite d3/fd [0,4194304] 0 2024-04-20T21:32:00.982 INFO:tasks.workunit.client.1.smithi189.stdout:2/84: creat d1/f16 x:0 0 0 2024-04-20T21:32:00.982 INFO:tasks.workunit.client.1.smithi189.stdout:5/88: write d7/d14/d18/f17 [666922,106154] 0 2024-04-20T21:32:00.983 INFO:tasks.workunit.client.0.smithi062.stdout:0/436: read d7/d1c/d15/d17/d40/f91 [2068719,34278] 0 2024-04-20T21:32:00.985 INFO:tasks.workunit.client.0.smithi062.stdout:7/485: dread d7/d3e/d33/d3d/f75 [0,4194304] 0 2024-04-20T21:32:00.985 INFO:tasks.workunit.client.1.smithi189.stdout:0/216: chown d3/fd 15 1 2024-04-20T21:32:00.985 INFO:tasks.workunit.client.0.smithi062.stdout:0/437: dread - d7/d1c/d15/d17/f8d zero size 2024-04-20T21:32:00.986 INFO:tasks.workunit.client.0.smithi062.stdout:1/490: dwrite dc/f53 [0,4194304] 0 2024-04-20T21:32:00.986 INFO:tasks.workunit.client.1.smithi189.stdout:4/77: dread d2/f4 [0,4194304] 0 2024-04-20T21:32:00.991 INFO:tasks.workunit.client.0.smithi062.stdout:5/482: getdents d8/d38/d7a 0 2024-04-20T21:32:00.994 INFO:tasks.workunit.client.1.smithi189.stdout:0/217: read - d3/df/d30/d37/d29/d35/f44 zero size 2024-04-20T21:32:00.994 INFO:tasks.workunit.client.1.smithi189.stdout:1/89: dread f2 [0,4194304] 0 2024-04-20T21:32:01.003 INFO:tasks.workunit.client.1.smithi189.stdout:8/229: sync 2024-04-20T21:32:01.005 INFO:tasks.workunit.client.0.smithi062.stdout:7/486: chown d7/d3e/d33/d3d/d4d/d54/f1f 61931338 1 2024-04-20T21:32:01.008 INFO:tasks.workunit.client.0.smithi062.stdout:7/487: chown d7/d26/la2 384 1 2024-04-20T21:32:01.012 INFO:tasks.workunit.client.1.smithi189.stdout:0/218: dread d3/fe [0,4194304] 0 2024-04-20T21:32:01.013 INFO:tasks.workunit.client.0.smithi062.stdout:8/500: dwrite d7/d30/d90/f92 [0,4194304] 0 2024-04-20T21:32:01.015 INFO:tasks.workunit.client.0.smithi062.stdout:0/438: dread d7/d1c/d15/f22 [0,4194304] 0 2024-04-20T21:32:01.015 INFO:tasks.workunit.client.1.smithi189.stdout:9/107: dread f1 [0,4194304] 0 2024-04-20T21:32:01.019 INFO:tasks.workunit.client.0.smithi062.stdout:8/501: dread - d7/d30/f95 zero size 2024-04-20T21:32:01.019 INFO:tasks.workunit.client.0.smithi062.stdout:7/488: rename d7/l16 to d7/d3e/d33/d3d/d5e/d7e/la6 0 2024-04-20T21:32:01.025 INFO:tasks.workunit.client.0.smithi062.stdout:0/439: read d7/f47 [18430,24815] 0 2024-04-20T21:32:01.027 INFO:tasks.workunit.client.1.smithi189.stdout:4/78: mknod d2/dd/d11/c1e 0 2024-04-20T21:32:01.029 INFO:tasks.workunit.client.1.smithi189.stdout:4/79: stat d2/dd/d11/c1b 0 2024-04-20T21:32:01.029 INFO:tasks.workunit.client.0.smithi062.stdout:5/483: dwrite d8/fb [0,4194304] 0 2024-04-20T21:32:01.031 INFO:tasks.workunit.client.0.smithi062.stdout:3/470: dread d0/d5c/f5a [4194304,4194304] 0 2024-04-20T21:32:01.034 INFO:tasks.workunit.client.1.smithi189.stdout:8/230: creat da/de/d1e/f52 x:0 0 0 2024-04-20T21:32:01.034 INFO:tasks.workunit.client.0.smithi062.stdout:1/491: truncate dc/d15/d75/d94/f1e 115176 0 2024-04-20T21:32:01.036 INFO:tasks.workunit.client.1.smithi189.stdout:0/219: write d3/df/d28/d2b/f36 [2526915,17261] 0 2024-04-20T21:32:01.036 INFO:tasks.workunit.client.1.smithi189.stdout:8/231: write da/df/d13/f14 [4177010,13908] 0 2024-04-20T21:32:01.038 INFO:tasks.workunit.client.1.smithi189.stdout:9/108: mkdir d9/d12/d15/d1a/d1f 0 2024-04-20T21:32:01.038 INFO:tasks.workunit.client.1.smithi189.stdout:6/91: truncate d8/da/dd/fe 1311366 0 2024-04-20T21:32:01.042 INFO:tasks.workunit.client.1.smithi189.stdout:3/73: truncate d2/fd 1275270 0 2024-04-20T21:32:01.042 INFO:tasks.workunit.client.1.smithi189.stdout:5/89: symlink d7/l1a 0 2024-04-20T21:32:01.043 INFO:tasks.workunit.client.1.smithi189.stdout:7/100: write d5/fd [4108929,73234] 0 2024-04-20T21:32:01.046 INFO:tasks.workunit.client.0.smithi062.stdout:5/484: creat d8/d1e/d2c/d6c/f9b x:0 0 0 2024-04-20T21:32:01.046 INFO:tasks.workunit.client.0.smithi062.stdout:3/471: rename d0/d1/d10/f64 to d0/d1/d10/d31/f9e 0 2024-04-20T21:32:01.050 INFO:tasks.workunit.client.0.smithi062.stdout:0/440: dread d7/d20/d3a/f87 [0,4194304] 0 2024-04-20T21:32:01.050 INFO:tasks.workunit.client.0.smithi062.stdout:3/472: fdatasync d0/d1/f29 0 2024-04-20T21:32:01.054 INFO:tasks.workunit.client.1.smithi189.stdout:1/90: creat d8/d21/f22 x:0 0 0 2024-04-20T21:32:01.055 INFO:tasks.workunit.client.0.smithi062.stdout:5/485: write d8/d1e/f25 [670744,26664] 0 2024-04-20T21:32:01.055 INFO:tasks.workunit.client.0.smithi062.stdout:8/502: dwrite d7/fa5 [0,4194304] 0 2024-04-20T21:32:01.064 INFO:tasks.workunit.client.1.smithi189.stdout:9/109: dwrite f6 [0,4194304] 0 2024-04-20T21:32:01.067 INFO:tasks.workunit.client.1.smithi189.stdout:8/232: mknod da/df/d20/d2c/c53 0 2024-04-20T21:32:01.071 INFO:tasks.workunit.client.0.smithi062.stdout:1/492: dwrite dc/d2d/d7c/f88 [0,4194304] 0 2024-04-20T21:32:01.071 INFO:tasks.workunit.client.0.smithi062.stdout:0/441: getdents d7/d1c 0 2024-04-20T21:32:01.072 INFO:tasks.workunit.client.0.smithi062.stdout:3/473: mknod d0/d1/d10/c9f 0 2024-04-20T21:32:01.075 INFO:tasks.workunit.client.0.smithi062.stdout:1/493: readlink dc/d15/d20/l3b 0 2024-04-20T21:32:01.076 INFO:tasks.workunit.client.0.smithi062.stdout:8/503: unlink d7/d25/d3b/l99 0 2024-04-20T21:32:01.079 INFO:tasks.workunit.client.1.smithi189.stdout:0/220: rename d3/df/d30/d37/d29 to d3/df/d4c 0 2024-04-20T21:32:01.081 INFO:tasks.workunit.client.0.smithi062.stdout:1/494: mknod dc/d2d/d84/ca9 0 2024-04-20T21:32:01.082 INFO:tasks.workunit.client.0.smithi062.stdout:1/495: readlink dc/d15/d75/l8e 0 2024-04-20T21:32:01.085 INFO:tasks.workunit.client.1.smithi189.stdout:5/90: chown d7/c16 182832 1 2024-04-20T21:32:01.087 INFO:tasks.workunit.client.1.smithi189.stdout:7/101: creat d5/d1d/f26 x:0 0 0 2024-04-20T21:32:01.087 INFO:tasks.workunit.client.0.smithi062.stdout:3/474: getdents d0/d1/d3c/d43/d72 0 2024-04-20T21:32:01.088 INFO:tasks.workunit.client.1.smithi189.stdout:7/102: write d5/f18 [4952933,68710] 0 2024-04-20T21:32:01.090 INFO:tasks.workunit.client.1.smithi189.stdout:7/103: fdatasync d5/d21/f23 0 2024-04-20T21:32:01.091 INFO:tasks.workunit.client.0.smithi062.stdout:0/442: link d7/d1c/d15/d17/c69 d7/d1c/c98 0 2024-04-20T21:32:01.093 INFO:tasks.workunit.client.0.smithi062.stdout:5/486: dwrite d8/d1e/d2c/d6c/d56/d50/d88/f8d [0,4194304] 0 2024-04-20T21:32:01.093 INFO:tasks.workunit.client.0.smithi062.stdout:0/443: chown d7/f97 1268 1 2024-04-20T21:32:01.095 INFO:tasks.workunit.client.1.smithi189.stdout:7/104: write d5/f24 [77296,26842] 0 2024-04-20T21:32:01.096 INFO:tasks.workunit.client.0.smithi062.stdout:1/496: creat dc/d15/d75/d94/d36/d78/faa x:0 0 0 2024-04-20T21:32:01.096 INFO:tasks.workunit.client.0.smithi062.stdout:7/489: dread d7/d3e/d33/d3d/d4d/d54/f28 [4194304,4194304] 0 2024-04-20T21:32:01.099 INFO:tasks.workunit.client.1.smithi189.stdout:9/110: symlink d9/dc/d17/l20 0 2024-04-20T21:32:01.099 INFO:tasks.workunit.client.1.smithi189.stdout:0/221: dwrite d3/df/d4c/d35/f44 [0,4194304] 0 2024-04-20T21:32:01.100 INFO:tasks.workunit.client.0.smithi062.stdout:1/497: chown dc/d15/d75/d94/d36/f9c 4 1 2024-04-20T21:32:01.100 INFO:tasks.workunit.client.1.smithi189.stdout:9/111: write d9/fb [495831,91655] 0 2024-04-20T21:32:01.101 INFO:tasks.workunit.client.0.smithi062.stdout:8/504: rmdir d7 39 2024-04-20T21:32:01.102 INFO:tasks.workunit.client.1.smithi189.stdout:8/233: rmdir da/df/d13/d31 39 2024-04-20T21:32:01.106 INFO:tasks.workunit.client.0.smithi062.stdout:5/487: chown d8/d1e/d2c/d6c/c4a 88650 1 2024-04-20T21:32:01.107 INFO:tasks.workunit.client.1.smithi189.stdout:9/112: dread f1 [0,4194304] 0 2024-04-20T21:32:01.112 INFO:tasks.workunit.client.0.smithi062.stdout:1/498: mkdir dc/d15/d75/d94/d36/d78/dab 0 2024-04-20T21:32:01.114 INFO:tasks.workunit.client.1.smithi189.stdout:3/74: chown d2/d3/l4 1 1 2024-04-20T21:32:01.120 INFO:tasks.workunit.client.1.smithi189.stdout:4/80: link d2/l12 d2/dd/l1f 0 2024-04-20T21:32:01.120 INFO:tasks.workunit.client.1.smithi189.stdout:5/91: fdatasync d7/d9/fc 0 2024-04-20T21:32:01.121 INFO:tasks.workunit.client.1.smithi189.stdout:1/91: unlink c7 0 2024-04-20T21:32:01.122 INFO:tasks.workunit.client.0.smithi062.stdout:3/475: dwrite d0/d1/d10/f4e [0,4194304] 0 2024-04-20T21:32:01.123 INFO:tasks.workunit.client.1.smithi189.stdout:9/113: dread f3 [0,4194304] 0 2024-04-20T21:32:01.123 INFO:tasks.workunit.client.0.smithi062.stdout:8/505: creat d7/fab x:0 0 0 2024-04-20T21:32:01.123 INFO:tasks.workunit.client.0.smithi062.stdout:1/499: dread dc/d15/d75/d94/f1e [0,4194304] 0 2024-04-20T21:32:01.126 INFO:tasks.workunit.client.1.smithi189.stdout:7/105: unlink d5/f17 0 2024-04-20T21:32:01.130 INFO:tasks.workunit.client.1.smithi189.stdout:5/92: chown d7/d9/ce 1 1 2024-04-20T21:32:01.131 INFO:tasks.workunit.client.0.smithi062.stdout:3/476: rmdir d0/d1c/d52/d7b 39 2024-04-20T21:32:01.131 INFO:tasks.workunit.client.1.smithi189.stdout:8/234: creat da/df/d20/d27/f54 x:0 0 0 2024-04-20T21:32:01.133 INFO:tasks.workunit.client.1.smithi189.stdout:6/92: read d8/da/dd/fe [891164,130418] 0 2024-04-20T21:32:01.134 INFO:tasks.workunit.client.1.smithi189.stdout:2/85: dwrite d1/f5 [0,4194304] 0 2024-04-20T21:32:01.135 INFO:tasks.workunit.client.1.smithi189.stdout:6/93: chown f7 1414858 1 2024-04-20T21:32:01.136 INFO:tasks.workunit.client.1.smithi189.stdout:3/75: creat d2/d3/f19 x:0 0 0 2024-04-20T21:32:01.137 INFO:tasks.workunit.client.0.smithi062.stdout:5/488: dwrite d8/d1e/d2c/d6c/f60 [0,4194304] 0 2024-04-20T21:32:01.137 INFO:tasks.workunit.client.0.smithi062.stdout:8/506: mkdir d7/d25/d3b/d58/dac 0 2024-04-20T21:32:01.138 INFO:tasks.workunit.client.0.smithi062.stdout:6/385: sync 2024-04-20T21:32:01.139 INFO:tasks.workunit.client.1.smithi189.stdout:2/86: write d1/f7 [1144665,5125] 0 2024-04-20T21:32:01.140 INFO:tasks.workunit.client.0.smithi062.stdout:3/477: fdatasync d0/d1c/d52/d7b/f85 0 2024-04-20T21:32:01.143 INFO:tasks.workunit.client.1.smithi189.stdout:3/76: chown d2/d3/l9 46 1 2024-04-20T21:32:01.143 INFO:tasks.workunit.client.0.smithi062.stdout:1/500: symlink dc/d15/d75/d94/d6f/lac 0 2024-04-20T21:32:01.147 INFO:tasks.workunit.client.0.smithi062.stdout:7/490: truncate d7/d3e/d33/d3d/d5e/d72/f86 2804992 0 2024-04-20T21:32:01.150 INFO:tasks.workunit.client.0.smithi062.stdout:1/501: readlink dc/d15/d75/d94/d36/d78/l95 0 2024-04-20T21:32:01.150 INFO:tasks.workunit.client.0.smithi062.stdout:0/444: readlink d7/d1c/l8c 0 2024-04-20T21:32:01.151 INFO:tasks.workunit.client.1.smithi189.stdout:5/93: mkdir d7/d14/d1b 0 2024-04-20T21:32:01.153 INFO:tasks.workunit.client.0.smithi062.stdout:7/491: fsync d7/d3e/d33/d3d/f75 0 2024-04-20T21:32:01.154 INFO:tasks.workunit.client.1.smithi189.stdout:9/114: dread f5 [0,4194304] 0 2024-04-20T21:32:01.154 INFO:tasks.workunit.client.0.smithi062.stdout:8/507: creat d7/fad x:0 0 0 2024-04-20T21:32:01.155 INFO:tasks.workunit.client.0.smithi062.stdout:5/489: getdents d8/d1e/d2c/d6c/d56/d66 0 2024-04-20T21:32:01.159 INFO:tasks.workunit.client.0.smithi062.stdout:7/492: creat d7/d63/fa7 x:0 0 0 2024-04-20T21:32:01.160 INFO:tasks.workunit.client.0.smithi062.stdout:5/490: write d8/d1e/f24 [10085,32987] 0 2024-04-20T21:32:01.160 INFO:tasks.workunit.client.0.smithi062.stdout:8/508: write d7/d30/f3c [5399868,79394] 0 2024-04-20T21:32:01.160 INFO:tasks.workunit.client.1.smithi189.stdout:8/235: chown da/df/d13/d31 48258099 1 2024-04-20T21:32:01.160 INFO:tasks.workunit.client.1.smithi189.stdout:6/94: rmdir d8/d10 39 2024-04-20T21:32:01.161 INFO:tasks.workunit.client.1.smithi189.stdout:8/236: chown da/d1f 140226280 1 2024-04-20T21:32:01.161 INFO:tasks.workunit.client.1.smithi189.stdout:3/77: mknod d2/d8/c1a 0 2024-04-20T21:32:01.162 INFO:tasks.workunit.client.0.smithi062.stdout:1/502: rename dc/c8a to dc/d27/d9a/cad 0 2024-04-20T21:32:01.163 INFO:tasks.workunit.client.1.smithi189.stdout:4/81: dwrite d2/f8 [0,4194304] 0 2024-04-20T21:32:01.165 INFO:tasks.workunit.client.1.smithi189.stdout:9/115: mknod d9/d12/c21 0 2024-04-20T21:32:01.166 INFO:tasks.workunit.client.0.smithi062.stdout:3/478: dread d0/d1b/f23 [0,4194304] 0 2024-04-20T21:32:01.166 INFO:tasks.workunit.client.1.smithi189.stdout:5/94: truncate d7/d9/fc 4574068 0 2024-04-20T21:32:01.167 INFO:tasks.workunit.client.0.smithi062.stdout:5/491: write d8/d1e/d2c/d6c/d56/d50/f5f [1591000,123354] 0 2024-04-20T21:32:01.169 INFO:tasks.workunit.client.1.smithi189.stdout:1/92: dwrite d8/f1b [0,4194304] 0 2024-04-20T21:32:01.170 INFO:tasks.workunit.client.0.smithi062.stdout:7/493: chown d7/d26/l91 0 1 2024-04-20T21:32:01.171 INFO:tasks.workunit.client.0.smithi062.stdout:0/445: dread d7/d1c/d15/f22 [0,4194304] 0 2024-04-20T21:32:01.173 INFO:tasks.workunit.client.0.smithi062.stdout:9/477: sync 2024-04-20T21:32:01.173 INFO:tasks.workunit.client.0.smithi062.stdout:4/505: sync 2024-04-20T21:32:01.174 INFO:tasks.workunit.client.1.smithi189.stdout:7/106: getdents d5 0 2024-04-20T21:32:01.174 INFO:tasks.workunit.client.1.smithi189.stdout:4/82: stat d2/dd/c13 0 2024-04-20T21:32:01.175 INFO:tasks.workunit.client.0.smithi062.stdout:1/503: rename dc/d15/d75/d96/l97 to dc/d27/lae 0 2024-04-20T21:32:01.176 INFO:tasks.workunit.client.0.smithi062.stdout:2/519: sync 2024-04-20T21:32:01.180 INFO:tasks.workunit.client.1.smithi189.stdout:4/83: fdatasync d2/f10 0 2024-04-20T21:32:01.180 INFO:tasks.workunit.client.0.smithi062.stdout:8/509: getdents d7/d30/d90 0 2024-04-20T21:32:01.183 INFO:tasks.workunit.client.1.smithi189.stdout:0/222: truncate d3/f3b 42465 0 2024-04-20T21:32:01.183 INFO:tasks.workunit.client.1.smithi189.stdout:6/95: unlink d8/d10/f11 0 2024-04-20T21:32:01.183 INFO:tasks.workunit.client.1.smithi189.stdout:8/237: mknod da/de/d1e/d21/c55 0 2024-04-20T21:32:01.185 INFO:tasks.workunit.client.1.smithi189.stdout:9/116: dread - d9/dc/f1c zero size 2024-04-20T21:32:01.186 INFO:tasks.workunit.client.0.smithi062.stdout:8/510: readlink d7/d25/l36 0 2024-04-20T21:32:01.186 INFO:tasks.workunit.client.0.smithi062.stdout:9/478: creat d16/d2f/d5b/fa0 x:0 0 0 2024-04-20T21:32:01.188 INFO:tasks.workunit.client.0.smithi062.stdout:1/504: write dc/d15/d20/d28/d83/fa5 [467181,60374] 0 2024-04-20T21:32:01.193 INFO:tasks.workunit.client.1.smithi189.stdout:4/84: write d2/f10 [3449223,25908] 0 2024-04-20T21:32:01.193 INFO:tasks.workunit.client.0.smithi062.stdout:4/506: truncate da/dd/d15/d50/faa 351533 0 2024-04-20T21:32:01.199 INFO:tasks.workunit.client.0.smithi062.stdout:5/492: rename d8/d1e/d2c/d6c/d56/d50/d88/f8d to d8/d1e/d2c/d6c/d56/d50/f9c 0 2024-04-20T21:32:01.200 INFO:tasks.workunit.client.1.smithi189.stdout:2/87: dwrite d1/d11/f15 [0,4194304] 0 2024-04-20T21:32:01.202 INFO:tasks.workunit.client.0.smithi062.stdout:4/507: truncate da/dd/d15/d50/d64/d3d/f5e 1164565 0 2024-04-20T21:32:01.203 INFO:tasks.workunit.client.0.smithi062.stdout:8/511: creat d7/d25/d4f/fae x:0 0 0 2024-04-20T21:32:01.205 INFO:tasks.workunit.client.0.smithi062.stdout:9/479: symlink d16/d3e/d77/la1 0 2024-04-20T21:32:01.207 INFO:tasks.workunit.client.0.smithi062.stdout:5/493: fsync d8/f4f 0 2024-04-20T21:32:01.208 INFO:tasks.workunit.client.0.smithi062.stdout:4/508: mknod da/cb0 0 2024-04-20T21:32:01.208 INFO:tasks.workunit.client.0.smithi062.stdout:9/480: stat d16/d3d/d55/d6e/f9d 0 2024-04-20T21:32:01.210 INFO:tasks.workunit.client.1.smithi189.stdout:0/223: write d3/f3b [91278,64011] 0 2024-04-20T21:32:01.212 INFO:tasks.workunit.client.1.smithi189.stdout:2/88: chown d1/f9 485984 1 2024-04-20T21:32:01.212 INFO:tasks.workunit.client.0.smithi062.stdout:7/494: getdents d7/d3e/d33/d3d/d5e/d7e/d80 0 2024-04-20T21:32:01.214 INFO:tasks.workunit.client.0.smithi062.stdout:7/495: readlink d7/d3e/d33/l3b 0 2024-04-20T21:32:01.216 INFO:tasks.workunit.client.0.smithi062.stdout:5/494: unlink d8/d38/d7a/l3b 0 2024-04-20T21:32:01.217 INFO:tasks.workunit.client.0.smithi062.stdout:7/496: stat d7 0 2024-04-20T21:32:01.218 INFO:tasks.workunit.client.1.smithi189.stdout:0/224: readlink d3/df/d28/l3f 0 2024-04-20T21:32:01.224 INFO:tasks.workunit.client.0.smithi062.stdout:4/509: read da/dd/df/d27/d3f/f4c [1182307,129528] 0 2024-04-20T21:32:01.229 INFO:tasks.workunit.client.0.smithi062.stdout:7/497: mkdir d7/d26/da8 0 2024-04-20T21:32:01.231 INFO:tasks.workunit.client.0.smithi062.stdout:0/446: dwrite d7/d1c/f32 [0,4194304] 0 2024-04-20T21:32:01.234 INFO:tasks.workunit.client.0.smithi062.stdout:1/505: dwrite dc/d27/d32/f89 [0,4194304] 0 2024-04-20T21:32:01.234 INFO:tasks.workunit.client.0.smithi062.stdout:4/510: symlink da/lb1 0 2024-04-20T21:32:01.234 INFO:tasks.workunit.client.1.smithi189.stdout:1/93: dwrite f1 [4194304,4194304] 0 2024-04-20T21:32:01.235 INFO:tasks.workunit.client.0.smithi062.stdout:0/447: chown d7/d20 218393276 1 2024-04-20T21:32:01.239 INFO:tasks.workunit.client.0.smithi062.stdout:3/479: dread d0/d1/f19 [0,4194304] 0 2024-04-20T21:32:01.242 INFO:tasks.workunit.client.1.smithi189.stdout:5/95: dwrite d7/d9/fc [0,4194304] 0 2024-04-20T21:32:01.245 INFO:tasks.workunit.client.1.smithi189.stdout:6/96: dwrite d8/d12/f14 [0,4194304] 0 2024-04-20T21:32:01.249 INFO:tasks.workunit.client.0.smithi062.stdout:7/498: fsync d7/d3e/f5d 0 2024-04-20T21:32:01.252 INFO:tasks.workunit.client.0.smithi062.stdout:7/499: write d7/ff [4665817,88316] 0 2024-04-20T21:32:01.253 INFO:tasks.workunit.client.0.smithi062.stdout:4/511: creat da/dd/df/d27/d32/fb2 x:0 0 0 2024-04-20T21:32:01.255 INFO:tasks.workunit.client.1.smithi189.stdout:7/107: dwrite d5/f1f [0,4194304] 0 2024-04-20T21:32:01.255 INFO:tasks.workunit.client.0.smithi062.stdout:8/512: getdents d7/d25/d3b/d58/d8b/d24/d46 0 2024-04-20T21:32:01.256 INFO:tasks.workunit.client.1.smithi189.stdout:4/85: dread d2/f8 [0,4194304] 0 2024-04-20T21:32:01.256 INFO:tasks.workunit.client.0.smithi062.stdout:7/500: write d7/f2a [1412401,70308] 0 2024-04-20T21:32:01.260 INFO:tasks.workunit.client.0.smithi062.stdout:7/501: stat d7/d3e/f41 0 2024-04-20T21:32:01.261 INFO:tasks.workunit.client.1.smithi189.stdout:1/94: chown d8/lb 733 1 2024-04-20T21:32:01.263 INFO:tasks.workunit.client.1.smithi189.stdout:6/97: mknod d8/d12/c1d 0 2024-04-20T21:32:01.264 INFO:tasks.workunit.client.0.smithi062.stdout:2/520: write d7/f1a [5011200,34382] 0 2024-04-20T21:32:01.265 INFO:tasks.workunit.client.0.smithi062.stdout:0/448: creat d7/d1c/d15/d17/d1b/d74/f99 x:0 0 0 2024-04-20T21:32:01.266 INFO:tasks.workunit.client.1.smithi189.stdout:9/117: dwrite d9/dc/fe [0,4194304] 0 2024-04-20T21:32:01.268 INFO:tasks.workunit.client.1.smithi189.stdout:0/225: link d3/df/c47 d3/d26/c4d 0 2024-04-20T21:32:01.269 INFO:tasks.workunit.client.0.smithi062.stdout:7/502: truncate d7/d3e/d33/d3d/d5e/d7e/d80/f8c 503769 0 2024-04-20T21:32:01.270 INFO:tasks.workunit.client.1.smithi189.stdout:5/96: creat d7/d19/f1c x:0 0 0 2024-04-20T21:32:01.271 INFO:tasks.workunit.client.1.smithi189.stdout:4/86: mkdir d2/d20 0 2024-04-20T21:32:01.272 INFO:tasks.workunit.client.0.smithi062.stdout:2/521: write d7/da/f13 [4523979,94760] 0 2024-04-20T21:32:01.273 INFO:tasks.workunit.client.1.smithi189.stdout:3/78: dwrite d2/fd [0,4194304] 0 2024-04-20T21:32:01.278 INFO:tasks.workunit.client.1.smithi189.stdout:5/97: write d7/d14/d18/f17 [371967,21041] 0 2024-04-20T21:32:01.278 INFO:tasks.workunit.client.0.smithi062.stdout:2/522: write d7/da/d15/d1b/f80 [1538267,26991] 0 2024-04-20T21:32:01.279 INFO:tasks.workunit.client.1.smithi189.stdout:3/79: write d2/f6 [4638485,60600] 0 2024-04-20T21:32:01.281 INFO:tasks.workunit.client.1.smithi189.stdout:6/98: rename d8/d12/f14 to d8/da/f1e 0 2024-04-20T21:32:01.282 INFO:tasks.workunit.client.1.smithi189.stdout:1/95: creat d8/d1e/f23 x:0 0 0 2024-04-20T21:32:01.282 INFO:tasks.workunit.client.0.smithi062.stdout:4/512: dwrite da/d56/d71/f79 [0,4194304] 0 2024-04-20T21:32:01.285 INFO:tasks.workunit.client.0.smithi062.stdout:4/513: read da/dd/d15/fa6 [3206520,108051] 0 2024-04-20T21:32:01.286 INFO:tasks.workunit.client.1.smithi189.stdout:6/99: stat d8/ff 0 2024-04-20T21:32:01.287 INFO:tasks.workunit.client.1.smithi189.stdout:6/100: chown d8/da/dd/fe 39 1 2024-04-20T21:32:01.287 INFO:tasks.workunit.client.1.smithi189.stdout:6/101: readlink d8/da/lc 0 2024-04-20T21:32:01.287 INFO:tasks.workunit.client.0.smithi062.stdout:4/514: readlink da/dd/d15/d50/l6b 0 2024-04-20T21:32:01.289 INFO:tasks.workunit.client.0.smithi062.stdout:1/506: rename dc/c2c to dc/d2d/d7c/caf 0 2024-04-20T21:32:01.290 INFO:tasks.workunit.client.1.smithi189.stdout:6/102: write f5 [1826030,29424] 0 2024-04-20T21:32:01.295 INFO:tasks.workunit.client.1.smithi189.stdout:1/96: creat d8/f24 x:0 0 0 2024-04-20T21:32:01.296 INFO:tasks.workunit.client.1.smithi189.stdout:1/97: stat d8 0 2024-04-20T21:32:01.297 INFO:tasks.workunit.client.0.smithi062.stdout:3/480: mknod d0/d5c/d76/ca0 0 2024-04-20T21:32:01.299 INFO:tasks.workunit.client.1.smithi189.stdout:1/98: write d8/fe [924976,97753] 0 2024-04-20T21:32:01.302 INFO:tasks.workunit.client.1.smithi189.stdout:4/87: getdents d2/d20 0 2024-04-20T21:32:01.304 INFO:tasks.workunit.client.1.smithi189.stdout:7/108: dread d5/ff [0,4194304] 0 2024-04-20T21:32:01.305 INFO:tasks.workunit.client.0.smithi062.stdout:9/481: dwrite d16/f1e [0,4194304] 0 2024-04-20T21:32:01.307 INFO:tasks.workunit.client.1.smithi189.stdout:7/109: truncate d5/f24 1611732 0 2024-04-20T21:32:01.309 INFO:tasks.workunit.client.1.smithi189.stdout:9/118: rmdir d9/d12/d15/d1a/d1f 0 2024-04-20T21:32:01.310 INFO:tasks.workunit.client.1.smithi189.stdout:1/99: mknod d8/c25 0 2024-04-20T21:32:01.310 INFO:tasks.workunit.client.0.smithi062.stdout:1/507: unlink dc/d15/d75/d94/c73 0 2024-04-20T21:32:01.312 INFO:tasks.workunit.client.0.smithi062.stdout:1/508: chown dc/d70/f90 738992524 1 2024-04-20T21:32:01.312 INFO:tasks.workunit.client.0.smithi062.stdout:8/513: mknod d7/d25/caf 0 2024-04-20T21:32:01.316 INFO:tasks.workunit.client.0.smithi062.stdout:5/495: chown d8/d38/d7a/d4c/f7b 31782 1 2024-04-20T21:32:01.316 INFO:tasks.workunit.client.0.smithi062.stdout:7/503: unlink d7/d63/c65 0 2024-04-20T21:32:01.317 INFO:tasks.workunit.client.1.smithi189.stdout:7/110: creat d5/d21/d25/f27 x:0 0 0 2024-04-20T21:32:01.317 INFO:tasks.workunit.client.1.smithi189.stdout:9/119: mknod d9/d12/c22 0 2024-04-20T21:32:01.317 INFO:tasks.workunit.client.0.smithi062.stdout:8/514: write d7/d30/f3c [5812274,8646] 0 2024-04-20T21:32:01.318 INFO:tasks.workunit.client.0.smithi062.stdout:2/523: symlink d7/da/d15/d2f/lad 0 2024-04-20T21:32:01.319 INFO:tasks.workunit.client.0.smithi062.stdout:9/482: creat d16/d3e/fa2 x:0 0 0 2024-04-20T21:32:01.319 INFO:tasks.workunit.client.0.smithi062.stdout:5/496: chown d8/d38/c81 2 1 2024-04-20T21:32:01.319 INFO:tasks.workunit.client.1.smithi189.stdout:4/88: dread d2/f4 [0,4194304] 0 2024-04-20T21:32:01.320 INFO:tasks.workunit.client.0.smithi062.stdout:7/504: stat d7/d3e/d33/d3d/d5e/c84 0 2024-04-20T21:32:01.322 INFO:tasks.workunit.client.1.smithi189.stdout:1/100: symlink d8/df/l26 0 2024-04-20T21:32:01.324 INFO:tasks.workunit.client.0.smithi062.stdout:1/509: creat dc/d2d/d84/fb0 x:0 0 0 2024-04-20T21:32:01.325 INFO:tasks.workunit.client.1.smithi189.stdout:7/111: mkdir d5/d1d/d28 0 2024-04-20T21:32:01.327 INFO:tasks.workunit.client.0.smithi062.stdout:3/481: rename d0/d1/d10/d31/d41/l90 to d0/d1/la1 0 2024-04-20T21:32:01.329 INFO:tasks.workunit.client.0.smithi062.stdout:5/497: mkdir d8/d38/d7a/d4c/d7d/d9d 0 2024-04-20T21:32:01.332 INFO:tasks.workunit.client.1.smithi189.stdout:0/226: unlink d3/d26/l46 0 2024-04-20T21:32:01.332 INFO:tasks.workunit.client.1.smithi189.stdout:5/98: fsync d7/d19/f1c 0 2024-04-20T21:32:01.332 INFO:tasks.workunit.client.1.smithi189.stdout:6/103: rmdir d8/da 39 2024-04-20T21:32:01.332 INFO:tasks.workunit.client.0.smithi062.stdout:7/505: unlink d7/d3e/d33/d3d/d5e/d7e/l87 0 2024-04-20T21:32:01.334 INFO:tasks.workunit.client.0.smithi062.stdout:0/449: getdents d7/d1c/d15/d17/d40/d79 0 2024-04-20T21:32:01.335 INFO:tasks.workunit.client.0.smithi062.stdout:9/483: creat d16/d4e/fa3 x:0 0 0 2024-04-20T21:32:01.339 INFO:tasks.workunit.client.1.smithi189.stdout:5/99: mknod d7/d19/c1d 0 2024-04-20T21:32:01.341 INFO:tasks.workunit.client.0.smithi062.stdout:1/510: rename dc/d15/d20/c2e to dc/d27/d9a/cb1 0 2024-04-20T21:32:01.341 INFO:tasks.workunit.client.0.smithi062.stdout:3/482: dread - d0/d1/d10/d31/d41/f7c zero size 2024-04-20T21:32:01.342 INFO:tasks.workunit.client.0.smithi062.stdout:1/511: fdatasync dc/d15/d20/f45 0 2024-04-20T21:32:01.342 INFO:tasks.workunit.client.0.smithi062.stdout:8/515: getdents d7/d30/d90 0 2024-04-20T21:32:01.343 INFO:tasks.workunit.client.1.smithi189.stdout:3/80: truncate d2/f6 192528 0 2024-04-20T21:32:01.343 INFO:tasks.workunit.client.1.smithi189.stdout:6/104: read f7 [3930972,9870] 0 2024-04-20T21:32:01.343 INFO:tasks.workunit.client.0.smithi062.stdout:5/498: mkdir d8/d10/d3e/d8a/d9e 0 2024-04-20T21:32:01.345 INFO:tasks.workunit.client.0.smithi062.stdout:9/484: stat d16/l20 0 2024-04-20T21:32:01.346 INFO:tasks.workunit.client.1.smithi189.stdout:7/112: dwrite d5/f1f [0,4194304] 0 2024-04-20T21:32:01.347 INFO:tasks.workunit.client.1.smithi189.stdout:7/113: chown d5/l1b 5489 1 2024-04-20T21:32:01.348 INFO:tasks.workunit.client.0.smithi062.stdout:9/485: dread - d16/d3e/d77/d84/f9a zero size 2024-04-20T21:32:01.349 INFO:tasks.workunit.client.1.smithi189.stdout:0/227: symlink d3/l4e 0 2024-04-20T21:32:01.350 INFO:tasks.workunit.client.0.smithi062.stdout:7/506: rename d7/d3e/d33/d3d/d4d/f5a to d7/d3e/d33/d3d/d5e/d7e/fa9 0 2024-04-20T21:32:01.351 INFO:tasks.workunit.client.1.smithi189.stdout:0/228: readlink d3/l4e 0 2024-04-20T21:32:01.352 INFO:tasks.workunit.client.1.smithi189.stdout:5/100: rename d7/l1a to d7/d14/d18/l1e 0 2024-04-20T21:32:01.353 INFO:tasks.workunit.client.1.smithi189.stdout:9/120: getdents d9/d12/d15 0 2024-04-20T21:32:01.353 INFO:tasks.workunit.client.1.smithi189.stdout:5/101: chown d7/d14/d1b 33236737 1 2024-04-20T21:32:01.355 INFO:tasks.workunit.client.1.smithi189.stdout:6/105: mknod d8/d10/c1f 0 2024-04-20T21:32:01.357 INFO:tasks.workunit.client.1.smithi189.stdout:5/102: write d7/d19/f1c [295200,84834] 0 2024-04-20T21:32:01.357 INFO:tasks.workunit.client.1.smithi189.stdout:7/114: symlink d5/d21/d25/l29 0 2024-04-20T21:32:01.357 INFO:tasks.workunit.client.0.smithi062.stdout:4/515: truncate da/fb 503390 0 2024-04-20T21:32:01.358 INFO:tasks.workunit.client.1.smithi189.stdout:1/101: getdents d8/d21 0 2024-04-20T21:32:01.362 INFO:tasks.workunit.client.0.smithi062.stdout:4/516: dread da/dd/f78 [0,4194304] 0 2024-04-20T21:32:01.362 INFO:tasks.workunit.client.0.smithi062.stdout:2/524: link d7/l90 d7/da/d15/d2f/d97/lae 0 2024-04-20T21:32:01.363 INFO:tasks.workunit.client.1.smithi189.stdout:9/121: chown d9/dc/d17/l18 32 1 2024-04-20T21:32:01.363 INFO:tasks.workunit.client.0.smithi062.stdout:5/499: mknod d8/d38/d7a/d4c/d7d/c9f 0 2024-04-20T21:32:01.365 INFO:tasks.workunit.client.0.smithi062.stdout:9/486: mkdir d16/d2f/d5b/da4 0 2024-04-20T21:32:01.366 INFO:tasks.workunit.client.0.smithi062.stdout:1/512: dwrite dc/d70/f90 [0,4194304] 0 2024-04-20T21:32:01.367 INFO:tasks.workunit.client.1.smithi189.stdout:4/89: truncate d2/fb 3410541 0 2024-04-20T21:32:01.370 INFO:tasks.workunit.client.1.smithi189.stdout:0/229: fsync d3/d26/f1d 0 2024-04-20T21:32:01.372 INFO:tasks.workunit.client.0.smithi062.stdout:6/386: sync 2024-04-20T21:32:01.373 INFO:tasks.workunit.client.1.smithi189.stdout:3/81: dread d2/f6 [0,4194304] 0 2024-04-20T21:32:01.374 INFO:tasks.workunit.client.0.smithi062.stdout:0/450: rename d7/d1c/l8c to d7/d1c/d15/l9a 0 2024-04-20T21:32:01.378 INFO:tasks.workunit.client.1.smithi189.stdout:5/103: mkdir d7/d14/d1b/d1f 0 2024-04-20T21:32:01.378 INFO:tasks.workunit.client.0.smithi062.stdout:1/513: chown dc/d27/d32/f74 173408586 1 2024-04-20T21:32:01.379 INFO:tasks.workunit.client.1.smithi189.stdout:5/104: stat d7/d9/ce 0 2024-04-20T21:32:01.380 INFO:tasks.workunit.client.1.smithi189.stdout:4/90: creat d2/f21 x:0 0 0 2024-04-20T21:32:01.386 INFO:tasks.workunit.client.0.smithi062.stdout:3/483: dread d0/f8 [0,4194304] 0 2024-04-20T21:32:01.387 INFO:tasks.workunit.client.1.smithi189.stdout:8/238: sync 2024-04-20T21:32:01.387 INFO:tasks.workunit.client.1.smithi189.stdout:3/82: write d2/f5 [1286430,128042] 0 2024-04-20T21:32:01.388 INFO:tasks.workunit.client.0.smithi062.stdout:7/507: dwrite d7/d3e/d33/d3d/d7f/f9e [0,4194304] 0 2024-04-20T21:32:01.388 INFO:tasks.workunit.client.0.smithi062.stdout:6/387: mkdir d0/d9/d11/d7b 0 2024-04-20T21:32:01.393 INFO:tasks.workunit.client.1.smithi189.stdout:1/102: dwrite d8/f24 [0,4194304] 0 2024-04-20T21:32:01.394 INFO:tasks.workunit.client.0.smithi062.stdout:0/451: creat d7/d20/d62/d6f/f9b x:0 0 0 2024-04-20T21:32:01.395 INFO:tasks.workunit.client.1.smithi189.stdout:7/115: dwrite d5/f9 [0,4194304] 0 2024-04-20T21:32:01.396 INFO:tasks.workunit.client.1.smithi189.stdout:9/122: dread f4 [0,4194304] 0 2024-04-20T21:32:01.405 INFO:tasks.workunit.client.0.smithi062.stdout:8/516: dwrite d7/d25/d5c/f98 [0,4194304] 0 2024-04-20T21:32:01.407 INFO:tasks.workunit.client.1.smithi189.stdout:7/116: dread - d5/d1d/f26 zero size 2024-04-20T21:32:01.407 INFO:tasks.workunit.client.1.smithi189.stdout:1/103: stat d8/d21/f22 0 2024-04-20T21:32:01.407 INFO:tasks.workunit.client.1.smithi189.stdout:4/91: getdents d2/dd 0 2024-04-20T21:32:01.407 INFO:tasks.workunit.client.1.smithi189.stdout:1/104: dread - d8/d21/f22 zero size 2024-04-20T21:32:01.407 INFO:tasks.workunit.client.1.smithi189.stdout:9/123: fsync d9/dc/f1d 0 2024-04-20T21:32:01.409 INFO:tasks.workunit.client.1.smithi189.stdout:8/239: mknod da/df/d13/d31/c56 0 2024-04-20T21:32:01.409 INFO:tasks.workunit.client.0.smithi062.stdout:8/517: chown d7/d25/c86 2209031 1 2024-04-20T21:32:01.410 INFO:tasks.workunit.client.1.smithi189.stdout:7/117: unlink d5/c8 0 2024-04-20T21:32:01.413 INFO:tasks.workunit.client.1.smithi189.stdout:4/92: mkdir d2/d22 0 2024-04-20T21:32:01.414 INFO:tasks.workunit.client.1.smithi189.stdout:9/124: symlink d9/d12/l23 0 2024-04-20T21:32:01.419 INFO:tasks.workunit.client.0.smithi062.stdout:3/484: creat d0/d11/fa2 x:0 0 0 2024-04-20T21:32:01.420 INFO:tasks.workunit.client.0.smithi062.stdout:1/514: mknod dc/d15/d75/d94/d36/d3a/da2/cb2 0 2024-04-20T21:32:01.422 INFO:tasks.workunit.client.1.smithi189.stdout:9/125: write d9/dc/f1c [839048,64516] 0 2024-04-20T21:32:01.422 INFO:tasks.workunit.client.1.smithi189.stdout:7/118: rename d5/l1b to d5/d21/d25/l2a 0 2024-04-20T21:32:01.422 INFO:tasks.workunit.client.1.smithi189.stdout:1/105: truncate d8/f11 113034 0 2024-04-20T21:32:01.422 INFO:tasks.workunit.client.1.smithi189.stdout:4/93: mknod d2/dd/d11/d14/c23 0 2024-04-20T21:32:01.423 INFO:tasks.workunit.client.1.smithi189.stdout:7/119: write d5/f9 [2874589,37814] 0 2024-04-20T21:32:01.424 INFO:tasks.workunit.client.1.smithi189.stdout:4/94: chown d2/dd/c13 0 1 2024-04-20T21:32:01.425 INFO:tasks.workunit.client.0.smithi062.stdout:1/515: write dc/d15/f6e [70356,46203] 0 2024-04-20T21:32:01.429 INFO:tasks.workunit.client.0.smithi062.stdout:5/500: dread d8/d1e/f53 [4194304,4194304] 0 2024-04-20T21:32:01.429 INFO:tasks.workunit.client.1.smithi189.stdout:9/126: creat d9/dc/d17/f24 x:0 0 0 2024-04-20T21:32:01.430 INFO:tasks.workunit.client.1.smithi189.stdout:6/106: dread d8/ff [0,4194304] 0 2024-04-20T21:32:01.431 INFO:tasks.workunit.client.1.smithi189.stdout:3/83: dread d2/f5 [0,4194304] 0 2024-04-20T21:32:01.433 INFO:tasks.workunit.client.0.smithi062.stdout:2/525: getdents d7/da/d15/d2f/d38 0 2024-04-20T21:32:01.434 INFO:tasks.workunit.client.1.smithi189.stdout:7/120: unlink d5/f18 0 2024-04-20T21:32:01.434 INFO:tasks.workunit.client.1.smithi189.stdout:2/89: sync 2024-04-20T21:32:01.434 INFO:tasks.workunit.client.1.smithi189.stdout:1/106: rename d8/df/c19 to d8/d1e/c27 0 2024-04-20T21:32:01.434 INFO:tasks.workunit.client.1.smithi189.stdout:9/127: rename d9 to d9/dc/d17/d25 22 2024-04-20T21:32:01.435 INFO:tasks.workunit.client.0.smithi062.stdout:4/517: link da/dd/df/d27/f36 da/dd/df/fb3 0 2024-04-20T21:32:01.437 INFO:tasks.workunit.client.0.smithi062.stdout:2/526: chown d7/da/d28/d4b/f95 111 1 2024-04-20T21:32:01.437 INFO:tasks.workunit.client.0.smithi062.stdout:7/508: link d7/d63/f8a d7/d3e/d33/d3d/d4d/d83/d95/faa 0 2024-04-20T21:32:01.437 INFO:tasks.workunit.client.0.smithi062.stdout:3/485: creat d0/d1c/d52/d97/fa3 x:0 0 0 2024-04-20T21:32:01.438 INFO:tasks.workunit.client.1.smithi189.stdout:0/230: truncate d3/df/d4c/d35/f44 1388885 0 2024-04-20T21:32:01.438 INFO:tasks.workunit.client.1.smithi189.stdout:5/105: truncate d7/d9/fc 508298 0 2024-04-20T21:32:01.440 INFO:tasks.workunit.client.0.smithi062.stdout:4/518: truncate da/dd/f78 982280 0 2024-04-20T21:32:01.440 INFO:tasks.workunit.client.0.smithi062.stdout:2/527: chown d7/da/d28/d4b/d9d 331700733 1 2024-04-20T21:32:01.441 INFO:tasks.workunit.client.1.smithi189.stdout:4/95: creat d2/d20/f24 x:0 0 0 2024-04-20T21:32:01.442 INFO:tasks.workunit.client.1.smithi189.stdout:2/90: mkdir d1/d17 0 2024-04-20T21:32:01.445 INFO:tasks.workunit.client.0.smithi062.stdout:4/519: symlink da/d56/lb4 0 2024-04-20T21:32:01.445 INFO:tasks.workunit.client.1.smithi189.stdout:7/121: creat d5/d1d/f2b x:0 0 0 2024-04-20T21:32:01.445 INFO:tasks.workunit.client.0.smithi062.stdout:0/452: getdents d7/d1c/d15/d6b 0 2024-04-20T21:32:01.448 INFO:tasks.workunit.client.1.smithi189.stdout:0/231: rmdir d3/df/d4c/d35 39 2024-04-20T21:32:01.448 INFO:tasks.workunit.client.1.smithi189.stdout:4/96: write d2/f18 [566631,19092] 0 2024-04-20T21:32:01.456 INFO:tasks.workunit.client.0.smithi062.stdout:8/518: dread d7/d9/d29/f40 [0,4194304] 0 2024-04-20T21:32:01.457 INFO:tasks.workunit.client.1.smithi189.stdout:0/232: readlink d3/df/d30/d37/l25 0 2024-04-20T21:32:01.457 INFO:tasks.workunit.client.1.smithi189.stdout:4/97: dread - d2/f21 zero size 2024-04-20T21:32:01.458 INFO:tasks.workunit.client.1.smithi189.stdout:6/107: getdents d8 0 2024-04-20T21:32:01.460 INFO:tasks.workunit.client.1.smithi189.stdout:2/91: mknod d1/d17/c18 0 2024-04-20T21:32:01.460 INFO:tasks.workunit.client.0.smithi062.stdout:4/520: creat da/dd/d15/d50/d64/d3d/fb5 x:0 0 0 2024-04-20T21:32:01.460 INFO:tasks.workunit.client.0.smithi062.stdout:7/509: getdents d7/d3e/d33/d3d/d5e/d7e/d80 0 2024-04-20T21:32:01.461 INFO:tasks.workunit.client.0.smithi062.stdout:1/516: dread dc/d15/d75/d94/d36/d3a/f43 [0,4194304] 0 2024-04-20T21:32:01.463 INFO:tasks.workunit.client.0.smithi062.stdout:6/388: write d0/f3e [328296,101013] 0 2024-04-20T21:32:01.467 INFO:tasks.workunit.client.0.smithi062.stdout:9/487: dread d16/d2c/d5e/d75/d76/f81 [0,4194304] 0 2024-04-20T21:32:01.467 INFO:tasks.workunit.client.0.smithi062.stdout:3/486: dread d0/d1c/f51 [0,4194304] 0 2024-04-20T21:32:01.467 INFO:tasks.workunit.client.0.smithi062.stdout:2/528: write d7/d39/f8e [469023,74224] 0 2024-04-20T21:32:01.471 INFO:tasks.workunit.client.1.smithi189.stdout:2/92: write d1/d11/f15 [3189367,31767] 0 2024-04-20T21:32:01.471 INFO:tasks.workunit.client.0.smithi062.stdout:8/519: read d7/d25/d3b/d58/d8b/d24/d46/f49 [334084,53980] 0 2024-04-20T21:32:01.471 INFO:tasks.workunit.client.0.smithi062.stdout:0/453: write d7/d1e/f45 [183735,19171] 0 2024-04-20T21:32:01.472 INFO:tasks.workunit.client.1.smithi189.stdout:1/107: getdents d8/df 0 2024-04-20T21:32:01.472 INFO:tasks.workunit.client.0.smithi062.stdout:9/488: fsync d16/d2c/d5e/f7b 0 2024-04-20T21:32:01.473 INFO:tasks.workunit.client.1.smithi189.stdout:3/84: dwrite d2/f15 [0,4194304] 0 2024-04-20T21:32:01.473 INFO:tasks.workunit.client.1.smithi189.stdout:2/93: truncate d1/fd 119001 0 2024-04-20T21:32:01.480 INFO:tasks.workunit.client.1.smithi189.stdout:1/108: write d8/f1a [455777,60448] 0 2024-04-20T21:32:01.480 INFO:tasks.workunit.client.1.smithi189.stdout:7/122: dread d5/f14 [0,4194304] 0 2024-04-20T21:32:01.480 INFO:tasks.workunit.client.0.smithi062.stdout:4/521: rename da/d56/f8c to da/dd/df/d27/d7a/d91/fb6 0 2024-04-20T21:32:01.481 INFO:tasks.workunit.client.0.smithi062.stdout:8/520: write f1 [1217399,90504] 0 2024-04-20T21:32:01.481 INFO:tasks.workunit.client.0.smithi062.stdout:3/487: fsync d0/d1c/d52/d7b/f68 0 2024-04-20T21:32:01.486 INFO:tasks.workunit.client.1.smithi189.stdout:9/128: dread f3 [0,4194304] 0 2024-04-20T21:32:01.488 INFO:tasks.workunit.client.1.smithi189.stdout:5/106: truncate d7/d9/fc 1262732 0 2024-04-20T21:32:01.490 INFO:tasks.workunit.client.1.smithi189.stdout:1/109: write d8/df/f15 [918838,118336] 0 2024-04-20T21:32:01.490 INFO:tasks.workunit.client.0.smithi062.stdout:4/522: write f3 [2709422,77384] 0 2024-04-20T21:32:01.492 INFO:tasks.workunit.client.0.smithi062.stdout:2/529: dread d7/da/d15/d1b/d24/f66 [0,4194304] 0 2024-04-20T21:32:01.492 INFO:tasks.workunit.client.0.smithi062.stdout:0/454: rename d7/d1c/d15/d17/d1b/f1a to d7/d1c/d15/d53/f9c 0 2024-04-20T21:32:01.496 INFO:tasks.workunit.client.0.smithi062.stdout:7/510: getdents d7/d3e/d33/d3d/d4d/d83/d94 0 2024-04-20T21:32:01.496 INFO:tasks.workunit.client.1.smithi189.stdout:5/107: write d7/d14/d18/f17 [1024033,24046] 0 2024-04-20T21:32:01.498 INFO:tasks.workunit.client.0.smithi062.stdout:4/523: write da/dd/d15/fa7 [889512,112803] 0 2024-04-20T21:32:01.500 INFO:tasks.workunit.client.0.smithi062.stdout:2/530: write d7/da/f3c [3464658,64091] 0 2024-04-20T21:32:01.502 INFO:tasks.workunit.client.1.smithi189.stdout:3/85: rename d2/d8/fb to d2/d3/f1b 0 2024-04-20T21:32:01.502 INFO:tasks.workunit.client.1.smithi189.stdout:9/129: truncate f8 633417 0 2024-04-20T21:32:01.502 INFO:tasks.workunit.client.1.smithi189.stdout:8/240: dread f9 [0,4194304] 0 2024-04-20T21:32:01.507 INFO:tasks.workunit.client.1.smithi189.stdout:9/130: readlink d9/dc/d17/l20 0 2024-04-20T21:32:01.507 INFO:tasks.workunit.client.0.smithi062.stdout:1/517: dwrite dc/d2d/d84/fb0 [0,4194304] 0 2024-04-20T21:32:01.507 INFO:tasks.workunit.client.0.smithi062.stdout:6/389: dwrite d0/d9/d11/d22/f68 [0,4194304] 0 2024-04-20T21:32:01.508 INFO:tasks.workunit.client.0.smithi062.stdout:1/518: chown dc/d27/d32/l37 0 1 2024-04-20T21:32:01.510 INFO:tasks.workunit.client.1.smithi189.stdout:0/233: dwrite d3/f27 [0,4194304] 0 2024-04-20T21:32:01.511 INFO:tasks.workunit.client.0.smithi062.stdout:6/390: write d0/d2c/f2e [3255465,13071] 0 2024-04-20T21:32:01.511 INFO:tasks.workunit.client.0.smithi062.stdout:1/519: chown dc/d15/d75/d94/d36/l85 426686105 1 2024-04-20T21:32:01.513 INFO:tasks.workunit.client.1.smithi189.stdout:2/94: getdents d1/d17 0 2024-04-20T21:32:01.515 INFO:tasks.workunit.client.1.smithi189.stdout:0/234: fdatasync d3/df/f15 0 2024-04-20T21:32:01.516 INFO:tasks.workunit.client.0.smithi062.stdout:1/520: write dc/d27/d32/f74 [365022,40409] 0 2024-04-20T21:32:01.518 INFO:tasks.workunit.client.1.smithi189.stdout:4/98: dwrite f0 [0,4194304] 0 2024-04-20T21:32:01.519 INFO:tasks.workunit.client.0.smithi062.stdout:1/521: chown dc/d27/c30 1558806 1 2024-04-20T21:32:01.525 INFO:tasks.workunit.client.0.smithi062.stdout:0/455: mkdir d7/d1c/d15/d17/d1b/d9d 0 2024-04-20T21:32:01.527 INFO:tasks.workunit.client.0.smithi062.stdout:7/511: mknod d7/d63/cab 0 2024-04-20T21:32:01.527 INFO:tasks.workunit.client.0.smithi062.stdout:8/521: write d7/d25/d3b/d58/d8b/f1f [1849275,115182] 0 2024-04-20T21:32:01.530 INFO:tasks.workunit.client.1.smithi189.stdout:9/131: creat d9/d12/d15/f26 x:0 0 0 2024-04-20T21:32:01.531 INFO:tasks.workunit.client.0.smithi062.stdout:1/522: mkdir dc/d27/d9a/da0/db3 0 2024-04-20T21:32:01.531 INFO:tasks.workunit.client.1.smithi189.stdout:8/241: symlink da/de/d1e/d21/l57 0 2024-04-20T21:32:01.533 INFO:tasks.workunit.client.0.smithi062.stdout:4/524: mknod da/dd/d15/daf/cb7 0 2024-04-20T21:32:01.533 INFO:tasks.workunit.client.1.smithi189.stdout:2/95: symlink d1/d11/l19 0 2024-04-20T21:32:01.533 INFO:tasks.workunit.client.1.smithi189.stdout:8/242: chown da/df/d22 18 1 2024-04-20T21:32:01.535 INFO:tasks.workunit.client.0.smithi062.stdout:0/456: unlink d7/d1c/d15/d17/f8d 0 2024-04-20T21:32:01.535 INFO:tasks.workunit.client.0.smithi062.stdout:5/501: dread d8/d38/d7a/f91 [0,4194304] 0 2024-04-20T21:32:01.536 INFO:tasks.workunit.client.1.smithi189.stdout:2/96: readlink d1/d11/l19 0 2024-04-20T21:32:01.536 INFO:tasks.workunit.client.0.smithi062.stdout:3/488: creat d0/d1/d10/fa4 x:0 0 0 2024-04-20T21:32:01.538 INFO:tasks.workunit.client.0.smithi062.stdout:3/489: readlink d0/d5c/l9a 0 2024-04-20T21:32:01.538 INFO:tasks.workunit.client.0.smithi062.stdout:0/457: readlink d7/d1c/d15/d17/l3f 0 2024-04-20T21:32:01.539 INFO:tasks.workunit.client.0.smithi062.stdout:9/489: rename fc to d16/fa5 0 2024-04-20T21:32:01.540 INFO:tasks.workunit.client.0.smithi062.stdout:3/490: chown d0/d1/d10/d31/f5f 130049 1 2024-04-20T21:32:01.542 INFO:tasks.workunit.client.0.smithi062.stdout:9/490: dread d16/d2f/f8f [0,4194304] 0 2024-04-20T21:32:01.542 INFO:tasks.workunit.client.0.smithi062.stdout:5/502: write d8/d10/f4e [1199305,111804] 0 2024-04-20T21:32:01.544 INFO:tasks.workunit.client.0.smithi062.stdout:8/522: unlink d7/d25/d70/fa0 0 2024-04-20T21:32:01.544 INFO:tasks.workunit.client.0.smithi062.stdout:6/391: read d0/d2c/f3b [1778037,130956] 0 2024-04-20T21:32:01.546 INFO:tasks.workunit.client.1.smithi189.stdout:6/108: dwrite d8/f1a [0,4194304] 0 2024-04-20T21:32:01.546 INFO:tasks.workunit.client.0.smithi062.stdout:0/458: readlink d7/d1c/d15/d17/d1b/l49 0 2024-04-20T21:32:01.547 INFO:tasks.workunit.client.1.smithi189.stdout:4/99: symlink d2/dd/d11/l25 0 2024-04-20T21:32:01.551 INFO:tasks.workunit.client.1.smithi189.stdout:3/86: dwrite d2/fd [0,4194304] 0 2024-04-20T21:32:01.551 INFO:tasks.workunit.client.0.smithi062.stdout:0/459: chown d7/d1c/d15/d17/d40/f91 81245 1 2024-04-20T21:32:01.551 INFO:tasks.workunit.client.1.smithi189.stdout:9/132: unlink l0 0 2024-04-20T21:32:01.557 INFO:tasks.workunit.client.0.smithi062.stdout:2/531: link d7/d39/c3b d7/da/d15/d2f/caf 0 2024-04-20T21:32:01.561 INFO:tasks.workunit.client.0.smithi062.stdout:7/512: dwrite d7/d3e/d33/d3d/d5e/f61 [0,4194304] 0 2024-04-20T21:32:01.561 INFO:tasks.workunit.client.1.smithi189.stdout:0/235: symlink d3/df/d4c/l4f 0 2024-04-20T21:32:01.565 INFO:tasks.workunit.client.0.smithi062.stdout:9/491: chown d16/d2c/c3c 2097272195 1 2024-04-20T21:32:01.566 INFO:tasks.workunit.client.0.smithi062.stdout:6/392: rmdir d0/d9/d11/d5c 39 2024-04-20T21:32:01.567 INFO:tasks.workunit.client.1.smithi189.stdout:4/100: creat d2/dd/f26 x:0 0 0 2024-04-20T21:32:01.568 INFO:tasks.workunit.client.0.smithi062.stdout:1/523: dwrite dc/d15/d75/d94/d36/f54 [0,4194304] 0 2024-04-20T21:32:01.570 INFO:tasks.workunit.client.1.smithi189.stdout:1/110: dread d8/f11 [0,4194304] 0 2024-04-20T21:32:01.572 INFO:tasks.workunit.client.1.smithi189.stdout:1/111: readlink l4 0 2024-04-20T21:32:01.572 INFO:tasks.workunit.client.0.smithi062.stdout:5/503: creat d8/d10/d3e/d8a/fa0 x:0 0 0 2024-04-20T21:32:01.573 INFO:tasks.workunit.client.0.smithi062.stdout:9/492: dread - d16/d3e/f5f zero size 2024-04-20T21:32:01.576 INFO:tasks.workunit.client.1.smithi189.stdout:6/109: write d8/da/dd/fe [2324788,27328] 0 2024-04-20T21:32:01.576 INFO:tasks.workunit.client.0.smithi062.stdout:4/525: dwrite da/dd/f78 [0,4194304] 0 2024-04-20T21:32:01.577 INFO:tasks.workunit.client.1.smithi189.stdout:9/133: creat d9/d12/d15/d1a/f27 x:0 0 0 2024-04-20T21:32:01.581 INFO:tasks.workunit.client.1.smithi189.stdout:2/97: dwrite d1/f4 [0,4194304] 0 2024-04-20T21:32:01.583 INFO:tasks.workunit.client.1.smithi189.stdout:2/98: stat d1/f9 0 2024-04-20T21:32:01.588 INFO:tasks.workunit.client.0.smithi062.stdout:8/523: mkdir d7/da9/db0 0 2024-04-20T21:32:01.589 INFO:tasks.workunit.client.0.smithi062.stdout:7/513: mkdir d7/d26/dac 0 2024-04-20T21:32:01.589 INFO:tasks.workunit.client.1.smithi189.stdout:3/87: dread d2/fd [0,4194304] 0 2024-04-20T21:32:01.594 INFO:tasks.workunit.client.1.smithi189.stdout:9/134: symlink d9/dc/d17/l28 0 2024-04-20T21:32:01.599 INFO:tasks.workunit.client.0.smithi062.stdout:0/460: dread d7/d1c/f32 [0,4194304] 0 2024-04-20T21:32:01.604 INFO:tasks.workunit.client.1.smithi189.stdout:4/101: creat d2/d22/f27 x:0 0 0 2024-04-20T21:32:01.604 INFO:tasks.workunit.client.0.smithi062.stdout:6/393: mkdir d0/d9/d11/d5c/d7c 0 2024-04-20T21:32:01.604 INFO:tasks.workunit.client.1.smithi189.stdout:0/236: mknod d3/c50 0 2024-04-20T21:32:01.606 INFO:tasks.workunit.client.0.smithi062.stdout:4/526: mkdir da/dd/db8 0 2024-04-20T21:32:01.609 INFO:tasks.workunit.client.0.smithi062.stdout:1/524: dread dc/d15/f18 [0,4194304] 0 2024-04-20T21:32:01.609 INFO:tasks.workunit.client.0.smithi062.stdout:6/394: chown d0/d52/l6e 3 1 2024-04-20T21:32:01.611 INFO:tasks.workunit.client.1.smithi189.stdout:2/99: rmdir d1/d11 39 2024-04-20T21:32:01.612 INFO:tasks.workunit.client.1.smithi189.stdout:7/123: creat d5/d1d/d28/f2c x:0 0 0 2024-04-20T21:32:01.613 INFO:tasks.workunit.client.1.smithi189.stdout:4/102: mknod d2/d22/c28 0 2024-04-20T21:32:01.616 INFO:tasks.workunit.client.1.smithi189.stdout:3/88: mkdir d2/d1c 0 2024-04-20T21:32:01.617 INFO:tasks.workunit.client.0.smithi062.stdout:7/514: dread d7/d3e/f41 [0,4194304] 0 2024-04-20T21:32:01.619 INFO:tasks.workunit.client.0.smithi062.stdout:7/515: dread - d7/d3e/f5d zero size 2024-04-20T21:32:01.624 INFO:tasks.workunit.client.1.smithi189.stdout:7/124: creat d5/d21/d25/f2d x:0 0 0 2024-04-20T21:32:01.625 INFO:tasks.workunit.client.0.smithi062.stdout:5/504: rename d8/d10/c13 to d8/d38/d7a/d4c/ca1 0 2024-04-20T21:32:01.625 INFO:tasks.workunit.client.0.smithi062.stdout:7/516: read d7/d3e/f7b [1832339,37519] 0 2024-04-20T21:32:01.626 INFO:tasks.workunit.client.1.smithi189.stdout:4/103: rmdir d2/dd/d11/d14 39 2024-04-20T21:32:01.626 INFO:tasks.workunit.client.0.smithi062.stdout:3/491: dread d0/d1c/d63/f30 [0,4194304] 0 2024-04-20T21:32:01.627 INFO:tasks.workunit.client.1.smithi189.stdout:0/237: rename d3/d26/c18 to d3/df/d4c/d35/c51 0 2024-04-20T21:32:01.631 INFO:tasks.workunit.client.1.smithi189.stdout:7/125: creat d5/d1d/d28/f2e x:0 0 0 2024-04-20T21:32:01.632 INFO:tasks.workunit.client.1.smithi189.stdout:3/89: rename d2 to d2/d3/d1d 22 2024-04-20T21:32:01.633 INFO:tasks.workunit.client.0.smithi062.stdout:8/524: dread d7/d25/d3b/d58/d8b/f1e [0,4194304] 0 2024-04-20T21:32:01.633 INFO:tasks.workunit.client.1.smithi189.stdout:9/135: dwrite d9/d12/f14 [0,4194304] 0 2024-04-20T21:32:01.637 INFO:tasks.workunit.client.1.smithi189.stdout:0/238: write d3/df/d30/f3d [1518593,76490] 0 2024-04-20T21:32:01.638 INFO:tasks.workunit.client.1.smithi189.stdout:1/112: dwrite d8/fa [0,4194304] 0 2024-04-20T21:32:01.638 INFO:tasks.workunit.client.1.smithi189.stdout:8/243: dwrite da/df/f23 [0,4194304] 0 2024-04-20T21:32:01.639 INFO:tasks.workunit.client.0.smithi062.stdout:6/395: fdatasync d0/d9/d11/f63 0 2024-04-20T21:32:01.639 INFO:tasks.workunit.client.0.smithi062.stdout:8/525: chown d7/d25/d5c/f72 20 1 2024-04-20T21:32:01.641 INFO:tasks.workunit.client.0.smithi062.stdout:5/505: creat d8/d1e/d2c/fa2 x:0 0 0 2024-04-20T21:32:01.641 INFO:tasks.workunit.client.0.smithi062.stdout:8/526: dread - d7/d9/d29/f2b zero size 2024-04-20T21:32:01.641 INFO:tasks.workunit.client.0.smithi062.stdout:4/527: dread da/dd/f12 [0,4194304] 0 2024-04-20T21:32:01.643 INFO:tasks.workunit.client.0.smithi062.stdout:3/492: symlink d0/d1c/la5 0 2024-04-20T21:32:01.649 INFO:tasks.workunit.client.1.smithi189.stdout:6/110: dread f2 [0,4194304] 0 2024-04-20T21:32:01.649 INFO:tasks.workunit.client.1.smithi189.stdout:4/104: mknod d2/c29 0 2024-04-20T21:32:01.649 INFO:tasks.workunit.client.1.smithi189.stdout:3/90: rmdir d2/d8 39 2024-04-20T21:32:01.650 INFO:tasks.workunit.client.1.smithi189.stdout:9/136: fsync f5 0 2024-04-20T21:32:01.651 INFO:tasks.workunit.client.1.smithi189.stdout:1/113: mkdir d8/d1e/d28 0 2024-04-20T21:32:01.651 INFO:tasks.workunit.client.1.smithi189.stdout:8/244: creat da/d1f/d2a/f58 x:0 0 0 2024-04-20T21:32:01.654 INFO:tasks.workunit.client.0.smithi062.stdout:2/532: write d7/da/d15/d2f/f42 [222891,23279] 0 2024-04-20T21:32:01.655 INFO:tasks.workunit.client.1.smithi189.stdout:6/111: creat d8/d10/f20 x:0 0 0 2024-04-20T21:32:01.655 INFO:tasks.workunit.client.1.smithi189.stdout:7/126: link d5/d1d/f26 d5/d1d/d28/f2f 0 2024-04-20T21:32:01.657 INFO:tasks.workunit.client.0.smithi062.stdout:9/493: dread fe [0,4194304] 0 2024-04-20T21:32:01.657 INFO:tasks.workunit.client.1.smithi189.stdout:1/114: write d8/df/f15 [1080367,46326] 0 2024-04-20T21:32:01.657 INFO:tasks.workunit.client.1.smithi189.stdout:4/105: symlink d2/dd/d11/l2a 0 2024-04-20T21:32:01.659 INFO:tasks.workunit.client.1.smithi189.stdout:3/91: symlink d2/l1e 0 2024-04-20T21:32:01.663 INFO:tasks.workunit.client.1.smithi189.stdout:6/112: chown d8/da/c15 12810 1 2024-04-20T21:32:01.663 INFO:tasks.workunit.client.0.smithi062.stdout:6/396: dread d0/d9/d11/d22/f68 [0,4194304] 0 2024-04-20T21:32:01.668 INFO:tasks.workunit.client.1.smithi189.stdout:7/127: creat d5/d21/d25/f30 x:0 0 0 2024-04-20T21:32:01.669 INFO:tasks.workunit.client.1.smithi189.stdout:6/113: fsync d8/d12/f17 0 2024-04-20T21:32:01.672 INFO:tasks.workunit.client.1.smithi189.stdout:3/92: mknod d2/c1f 0 2024-04-20T21:32:01.672 INFO:tasks.workunit.client.0.smithi062.stdout:0/461: link d7/d1c/d15/d17/d40/d79/c70 d7/d1c/d15/d17/d26/c9e 0 2024-04-20T21:32:01.672 INFO:tasks.workunit.client.0.smithi062.stdout:5/506: creat d8/d1e/d2c/d6c/d56/d50/fa3 x:0 0 0 2024-04-20T21:32:01.673 INFO:tasks.workunit.client.1.smithi189.stdout:3/93: stat d2 0 2024-04-20T21:32:01.673 INFO:tasks.workunit.client.1.smithi189.stdout:7/128: truncate d5/d21/f23 848737 0 2024-04-20T21:32:01.675 INFO:tasks.workunit.client.1.smithi189.stdout:3/94: chown d2/d3/l16 253251305 1 2024-04-20T21:32:01.675 INFO:tasks.workunit.client.1.smithi189.stdout:6/114: write d8/da/dd/fe [807136,60707] 0 2024-04-20T21:32:01.677 INFO:tasks.workunit.client.0.smithi062.stdout:2/533: mkdir d7/da/d15/d2f/d38/db0 0 2024-04-20T21:32:01.678 INFO:tasks.workunit.client.1.smithi189.stdout:9/137: dread f3 [0,4194304] 0 2024-04-20T21:32:01.678 INFO:tasks.workunit.client.1.smithi189.stdout:4/106: link d2/l6 d2/dd/d11/l2b 0 2024-04-20T21:32:01.678 INFO:tasks.workunit.client.1.smithi189.stdout:7/129: chown d5/f1e 1 1 2024-04-20T21:32:01.679 INFO:tasks.workunit.client.0.smithi062.stdout:7/517: write d7/d3e/f5d [470734,114483] 0 2024-04-20T21:32:01.679 INFO:tasks.workunit.client.0.smithi062.stdout:9/494: mkdir d16/d2c/d42/da6 0 2024-04-20T21:32:01.682 INFO:tasks.workunit.client.0.smithi062.stdout:1/525: truncate dc/d15/d75/d94/f19 1353665 0 2024-04-20T21:32:01.683 INFO:tasks.workunit.client.0.smithi062.stdout:9/495: write d16/d3e/f64 [207924,61606] 0 2024-04-20T21:32:01.684 INFO:tasks.workunit.client.1.smithi189.stdout:4/107: creat d2/f2c x:0 0 0 2024-04-20T21:32:01.684 INFO:tasks.workunit.client.0.smithi062.stdout:6/397: rename d0/d9/d11/d22/d26 to d0/d1a/d7d 0 2024-04-20T21:32:01.689 INFO:tasks.workunit.client.0.smithi062.stdout:8/527: dwrite d7/d25/d76/f84 [0,4194304] 0 2024-04-20T21:32:01.690 INFO:tasks.workunit.client.1.smithi189.stdout:0/239: dwrite d3/d26/f1d [4194304,4194304] 0 2024-04-20T21:32:01.692 INFO:tasks.workunit.client.1.smithi189.stdout:6/115: link d8/d12/f17 d8/f21 0 2024-04-20T21:32:01.693 INFO:tasks.workunit.client.0.smithi062.stdout:3/493: write d0/d1b/f35 [2194330,27034] 0 2024-04-20T21:32:01.694 INFO:tasks.workunit.client.0.smithi062.stdout:1/526: creat dc/d15/d75/d94/d36/d3a/da2/fb4 x:0 0 0 2024-04-20T21:32:01.694 INFO:tasks.workunit.client.1.smithi189.stdout:9/138: rename d9/fb to d9/d12/f29 0 2024-04-20T21:32:01.696 INFO:tasks.workunit.client.1.smithi189.stdout:9/139: chown d9/d12/d15/d1a 8359788 1 2024-04-20T21:32:01.698 INFO:tasks.workunit.client.1.smithi189.stdout:9/140: chown d9/d12/d15/d1a 0 1 2024-04-20T21:32:01.700 INFO:tasks.workunit.client.0.smithi062.stdout:7/518: fsync d7/d3e/d33/d3d/d5e/d72/f98 0 2024-04-20T21:32:01.706 INFO:tasks.workunit.client.1.smithi189.stdout:0/240: write d3/df/f3e [153283,85283] 0 2024-04-20T21:32:01.706 INFO:tasks.workunit.client.0.smithi062.stdout:1/527: symlink dc/d15/d75/d94/d36/d3a/da2/lb5 0 2024-04-20T21:32:01.706 INFO:tasks.workunit.client.0.smithi062.stdout:8/528: mknod d7/d9/cb1 0 2024-04-20T21:32:01.710 INFO:tasks.workunit.client.0.smithi062.stdout:3/494: symlink d0/d1/d10/d3a/la6 0 2024-04-20T21:32:01.712 INFO:tasks.workunit.client.1.smithi189.stdout:0/241: chown d3/f48 0 1 2024-04-20T21:32:01.712 INFO:tasks.workunit.client.0.smithi062.stdout:2/534: creat d7/da/d15/d2f/d38/d6b/fb1 x:0 0 0 2024-04-20T21:32:01.713 INFO:tasks.workunit.client.1.smithi189.stdout:4/108: link d2/f19 d2/dd/f2d 0 2024-04-20T21:32:01.714 INFO:tasks.workunit.client.1.smithi189.stdout:9/141: rename d9/d12/c21 to d9/dc/c2a 0 2024-04-20T21:32:01.714 INFO:tasks.workunit.client.1.smithi189.stdout:8/245: dwrite da/f39 [0,4194304] 0 2024-04-20T21:32:01.717 INFO:tasks.workunit.client.0.smithi062.stdout:4/528: dread da/f83 [0,4194304] 0 2024-04-20T21:32:01.718 INFO:tasks.workunit.client.0.smithi062.stdout:7/519: mkdir d7/d3e/d33/d3d/dad 0 2024-04-20T21:32:01.719 INFO:tasks.workunit.client.1.smithi189.stdout:4/109: symlink d2/dd/l2e 0 2024-04-20T21:32:01.720 INFO:tasks.workunit.client.0.smithi062.stdout:1/528: mknod dc/cb6 0 2024-04-20T21:32:01.721 INFO:tasks.workunit.client.1.smithi189.stdout:1/115: dwrite d8/f11 [0,4194304] 0 2024-04-20T21:32:01.722 INFO:tasks.workunit.client.1.smithi189.stdout:7/130: dwrite d5/f6 [0,4194304] 0 2024-04-20T21:32:01.724 INFO:tasks.workunit.client.1.smithi189.stdout:0/242: symlink d3/df/d30/l52 0 2024-04-20T21:32:01.729 INFO:tasks.workunit.client.1.smithi189.stdout:9/142: mkdir d9/dc/d17/d2b 0 2024-04-20T21:32:01.729 INFO:tasks.workunit.client.1.smithi189.stdout:8/246: mkdir da/df/d22/d33/d59 0 2024-04-20T21:32:01.732 INFO:tasks.workunit.client.0.smithi062.stdout:0/462: dwrite d7/f47 [0,4194304] 0 2024-04-20T21:32:01.734 INFO:tasks.workunit.client.1.smithi189.stdout:7/131: truncate d5/d1d/d28/f2f 43122 0 2024-04-20T21:32:01.737 INFO:tasks.workunit.client.0.smithi062.stdout:5/507: dwrite d8/d1e/f5d [0,4194304] 0 2024-04-20T21:32:01.738 INFO:tasks.workunit.client.0.smithi062.stdout:7/520: symlink d7/d3e/lae 0 2024-04-20T21:32:01.739 INFO:tasks.workunit.client.0.smithi062.stdout:9/496: dread d16/f1e [0,4194304] 0 2024-04-20T21:32:01.741 INFO:tasks.workunit.client.1.smithi189.stdout:4/110: rename d2/c29 to d2/dd/c2f 0 2024-04-20T21:32:01.743 INFO:tasks.workunit.client.1.smithi189.stdout:4/111: dread - d2/dd/f2d zero size 2024-04-20T21:32:01.747 INFO:tasks.workunit.client.1.smithi189.stdout:4/112: dread - d2/f19 zero size 2024-04-20T21:32:01.747 INFO:tasks.workunit.client.1.smithi189.stdout:3/95: dread d2/d3/f1b [0,4194304] 0 2024-04-20T21:32:01.748 INFO:tasks.workunit.client.1.smithi189.stdout:5/108: sync 2024-04-20T21:32:01.750 INFO:tasks.workunit.client.0.smithi062.stdout:8/529: rename d7/d9/l71 to d7/lb2 0 2024-04-20T21:32:01.750 INFO:tasks.workunit.client.0.smithi062.stdout:0/463: creat d7/d20/d62/d6f/f9f x:0 0 0 2024-04-20T21:32:01.753 INFO:tasks.workunit.client.1.smithi189.stdout:1/116: dread d8/f1b [0,4194304] 0 2024-04-20T21:32:01.756 INFO:tasks.workunit.client.1.smithi189.stdout:4/113: fdatasync d2/f8 0 2024-04-20T21:32:01.756 INFO:tasks.workunit.client.1.smithi189.stdout:9/143: dread f5 [0,4194304] 0 2024-04-20T21:32:01.757 INFO:tasks.workunit.client.1.smithi189.stdout:7/132: dread d5/f6 [0,4194304] 0 2024-04-20T21:32:01.761 INFO:tasks.workunit.client.1.smithi189.stdout:4/114: write d2/f18 [888448,34414] 0 2024-04-20T21:32:01.762 INFO:tasks.workunit.client.1.smithi189.stdout:9/144: chown d9/dc/f1c 12363 1 2024-04-20T21:32:01.763 INFO:tasks.workunit.client.1.smithi189.stdout:5/109: dread d7/d19/f1c [0,4194304] 0 2024-04-20T21:32:01.763 INFO:tasks.workunit.client.1.smithi189.stdout:4/115: chown d2/d22/f27 184009549 1 2024-04-20T21:32:01.764 INFO:tasks.workunit.client.1.smithi189.stdout:3/96: mknod d2/c20 0 2024-04-20T21:32:01.765 INFO:tasks.workunit.client.0.smithi062.stdout:7/521: creat d7/d3e/d33/d3d/d5e/faf x:0 0 0 2024-04-20T21:32:01.767 INFO:tasks.workunit.client.1.smithi189.stdout:4/116: write d2/f21 [617299,82465] 0 2024-04-20T21:32:01.770 INFO:tasks.workunit.client.1.smithi189.stdout:2/100: sync 2024-04-20T21:32:01.771 INFO:tasks.workunit.client.0.smithi062.stdout:3/495: write d0/d5c/f5a [7902527,80915] 0 2024-04-20T21:32:01.772 INFO:tasks.workunit.client.1.smithi189.stdout:9/145: write d9/d12/d15/f26 [135261,90368] 0 2024-04-20T21:32:01.775 INFO:tasks.workunit.client.1.smithi189.stdout:9/146: chown f4 0 1 2024-04-20T21:32:01.775 INFO:tasks.workunit.client.1.smithi189.stdout:1/117: mknod d8/df/c29 0 2024-04-20T21:32:01.776 INFO:tasks.workunit.client.1.smithi189.stdout:6/116: mknod d8/da/dd/c22 0 2024-04-20T21:32:01.777 INFO:tasks.workunit.client.1.smithi189.stdout:1/118: readlink d8/lc 0 2024-04-20T21:32:01.778 INFO:tasks.workunit.client.1.smithi189.stdout:9/147: write d9/d12/f14 [2397497,91968] 0 2024-04-20T21:32:01.779 INFO:tasks.workunit.client.0.smithi062.stdout:2/535: dwrite d7/da/d15/d2f/f7e [0,4194304] 0 2024-04-20T21:32:01.780 INFO:tasks.workunit.client.1.smithi189.stdout:4/117: creat d2/d22/f30 x:0 0 0 2024-04-20T21:32:01.783 INFO:tasks.workunit.client.1.smithi189.stdout:8/247: dread da/de/f10 [0,4194304] 0 2024-04-20T21:32:01.784 INFO:tasks.workunit.client.0.smithi062.stdout:1/529: dwrite dc/d15/d20/d28/d83/fa5 [0,4194304] 0 2024-04-20T21:32:01.785 INFO:tasks.workunit.client.0.smithi062.stdout:3/496: rmdir d0/d5c 39 2024-04-20T21:32:01.787 INFO:tasks.workunit.client.1.smithi189.stdout:1/119: symlink d8/d1e/l2a 0 2024-04-20T21:32:01.787 INFO:tasks.workunit.client.1.smithi189.stdout:9/148: creat d9/dc/f2c x:0 0 0 2024-04-20T21:32:01.788 INFO:tasks.workunit.client.1.smithi189.stdout:7/133: dread d5/fe [0,4194304] 0 2024-04-20T21:32:01.792 INFO:tasks.workunit.client.1.smithi189.stdout:6/117: mkdir d8/d12/d16/d23 0 2024-04-20T21:32:01.792 INFO:tasks.workunit.client.1.smithi189.stdout:8/248: rmdir da/df/d20/d2c 39 2024-04-20T21:32:01.795 INFO:tasks.workunit.client.1.smithi189.stdout:0/243: truncate d3/df/d30/f3d 59071 0 2024-04-20T21:32:01.796 INFO:tasks.workunit.client.1.smithi189.stdout:1/120: fdatasync f2 0 2024-04-20T21:32:01.797 INFO:tasks.workunit.client.1.smithi189.stdout:3/97: dwrite d2/d3/f1b [0,4194304] 0 2024-04-20T21:32:01.798 INFO:tasks.workunit.client.0.smithi062.stdout:3/497: rename d0/d1/d10/d31/f70 to d0/d11/fa7 0 2024-04-20T21:32:01.798 INFO:tasks.workunit.client.1.smithi189.stdout:9/149: symlink d9/d12/l2d 0 2024-04-20T21:32:01.798 INFO:tasks.workunit.client.1.smithi189.stdout:7/134: stat d5/l10 0 2024-04-20T21:32:01.798 INFO:tasks.workunit.client.0.smithi062.stdout:4/529: dread da/dd/d15/d50/fa0 [0,4194304] 0 2024-04-20T21:32:01.800 INFO:tasks.workunit.client.1.smithi189.stdout:6/118: rmdir d8/da 39 2024-04-20T21:32:01.801 INFO:tasks.workunit.client.1.smithi189.stdout:0/244: fsync d3/df/f3e 0 2024-04-20T21:32:01.805 INFO:tasks.workunit.client.1.smithi189.stdout:7/135: unlink f2 0 2024-04-20T21:32:01.806 INFO:tasks.workunit.client.1.smithi189.stdout:3/98: truncate d2/f7 9064735 0 2024-04-20T21:32:01.806 INFO:tasks.workunit.client.1.smithi189.stdout:6/119: write f5 [2069568,1994] 0 2024-04-20T21:32:01.806 INFO:tasks.workunit.client.1.smithi189.stdout:5/110: dread d7/d14/d18/f17 [0,4194304] 0 2024-04-20T21:32:01.806 INFO:tasks.workunit.client.1.smithi189.stdout:6/120: stat f5 0 2024-04-20T21:32:01.810 INFO:tasks.workunit.client.1.smithi189.stdout:3/99: write d2/f15 [1427049,59590] 0 2024-04-20T21:32:01.817 INFO:tasks.workunit.client.0.smithi062.stdout:5/508: dwrite d8/d10/f4e [0,4194304] 0 2024-04-20T21:32:01.820 INFO:tasks.workunit.client.0.smithi062.stdout:9/497: dwrite d16/f66 [0,4194304] 0 2024-04-20T21:32:01.823 INFO:tasks.workunit.client.1.smithi189.stdout:4/118: dwrite d2/f19 [0,4194304] 0 2024-04-20T21:32:01.827 INFO:tasks.workunit.client.0.smithi062.stdout:0/464: dwrite d7/d1e/f67 [0,4194304] 0 2024-04-20T21:32:01.828 INFO:tasks.workunit.client.0.smithi062.stdout:3/498: symlink d0/d1c/d63/la8 0 2024-04-20T21:32:01.829 INFO:tasks.workunit.client.0.smithi062.stdout:1/530: rename dc/d15/d75/d94/d36/d3a/f5a to dc/d15/d75/d94/d36/d78/dab/fb7 0 2024-04-20T21:32:01.832 INFO:tasks.workunit.client.1.smithi189.stdout:2/101: truncate d1/d11/f15 1823959 0 2024-04-20T21:32:01.833 INFO:tasks.workunit.client.1.smithi189.stdout:9/150: mknod d9/dc/d17/c2e 0 2024-04-20T21:32:01.834 INFO:tasks.workunit.client.0.smithi062.stdout:6/398: dread d0/d9/d11/d22/f24 [0,4194304] 0 2024-04-20T21:32:01.834 INFO:tasks.workunit.client.0.smithi062.stdout:5/509: fdatasync d8/fb 0 2024-04-20T21:32:01.834 INFO:tasks.workunit.client.0.smithi062.stdout:3/499: stat d0/d1b/f2b 0 2024-04-20T21:32:01.836 INFO:tasks.workunit.client.0.smithi062.stdout:9/498: symlink d16/la7 0 2024-04-20T21:32:01.836 INFO:tasks.workunit.client.1.smithi189.stdout:2/102: fdatasync d1/f4 0 2024-04-20T21:32:01.838 INFO:tasks.workunit.client.1.smithi189.stdout:0/245: rename d3/df/d30/d37/l43 to d3/df/l53 0 2024-04-20T21:32:01.839 INFO:tasks.workunit.client.1.smithi189.stdout:5/111: mknod d7/d9/c20 0 2024-04-20T21:32:01.839 INFO:tasks.workunit.client.1.smithi189.stdout:6/121: dwrite f5 [4194304,4194304] 0 2024-04-20T21:32:01.841 INFO:tasks.workunit.client.0.smithi062.stdout:5/510: dread - d8/d1e/d2c/f79 zero size 2024-04-20T21:32:01.845 INFO:tasks.workunit.client.0.smithi062.stdout:1/531: creat dc/d15/d75/d96/fb8 x:0 0 0 2024-04-20T21:32:01.846 INFO:tasks.workunit.client.0.smithi062.stdout:7/522: dwrite d7/d3e/d33/d3d/d4d/fa4 [0,4194304] 0 2024-04-20T21:32:01.846 INFO:tasks.workunit.client.0.smithi062.stdout:3/500: fdatasync d0/d1b/f23 0 2024-04-20T21:32:01.847 INFO:tasks.workunit.client.0.smithi062.stdout:9/499: truncate d16/d2c/d62/f9c 996370 0 2024-04-20T21:32:01.848 INFO:tasks.workunit.client.1.smithi189.stdout:1/121: getdents d8/d1e/d28 0 2024-04-20T21:32:01.850 INFO:tasks.workunit.client.1.smithi189.stdout:5/112: fsync d7/d19/f1c 0 2024-04-20T21:32:01.851 INFO:tasks.workunit.client.1.smithi189.stdout:3/100: write d2/f5 [2026947,101487] 0 2024-04-20T21:32:01.852 INFO:tasks.workunit.client.0.smithi062.stdout:0/465: creat d7/d1c/d15/d17/d1b/d9d/fa0 x:0 0 0 2024-04-20T21:32:01.853 INFO:tasks.workunit.client.0.smithi062.stdout:2/536: dwrite d7/da/d28/f79 [0,4194304] 0 2024-04-20T21:32:01.855 INFO:tasks.workunit.client.0.smithi062.stdout:1/532: unlink f9 0 2024-04-20T21:32:01.857 INFO:tasks.workunit.client.1.smithi189.stdout:3/101: read d2/fd [3007109,106922] 0 2024-04-20T21:32:01.858 INFO:tasks.workunit.client.1.smithi189.stdout:2/103: rename d1/f10 to d1/d11/f1a 0 2024-04-20T21:32:01.864 INFO:tasks.workunit.client.1.smithi189.stdout:2/104: dread - d1/f12 zero size 2024-04-20T21:32:01.865 INFO:tasks.workunit.client.0.smithi062.stdout:5/511: mknod d8/d10/ca4 0 2024-04-20T21:32:01.866 INFO:tasks.workunit.client.0.smithi062.stdout:0/466: mknod d7/d20/d62/d6f/ca1 0 2024-04-20T21:32:01.868 INFO:tasks.workunit.client.0.smithi062.stdout:3/501: mkdir d0/d1/d3c/d43/d72/da9 0 2024-04-20T21:32:01.868 INFO:tasks.workunit.client.1.smithi189.stdout:4/119: dwrite d2/d20/f24 [0,4194304] 0 2024-04-20T21:32:01.869 INFO:tasks.workunit.client.1.smithi189.stdout:6/122: dread - d8/d10/f20 zero size 2024-04-20T21:32:01.872 INFO:tasks.workunit.client.0.smithi062.stdout:7/523: link d7/d3e/f99 d7/d3e/d33/d3d/d5e/d7e/fb0 0 2024-04-20T21:32:01.873 INFO:tasks.workunit.client.1.smithi189.stdout:5/113: mknod d7/d14/d18/c21 0 2024-04-20T21:32:01.873 INFO:tasks.workunit.client.1.smithi189.stdout:1/122: mkdir d8/d1e/d28/d2b 0 2024-04-20T21:32:01.874 INFO:tasks.workunit.client.0.smithi062.stdout:3/502: fsync d0/d1/d10/d31/f92 0 2024-04-20T21:32:01.875 INFO:tasks.workunit.client.0.smithi062.stdout:5/512: write d8/d10/f74 [3362732,15120] 0 2024-04-20T21:32:01.877 INFO:tasks.workunit.client.1.smithi189.stdout:2/105: write d1/f16 [104828,2660] 0 2024-04-20T21:32:01.878 INFO:tasks.workunit.client.1.smithi189.stdout:9/151: dwrite f4 [0,4194304] 0 2024-04-20T21:32:01.879 INFO:tasks.workunit.client.0.smithi062.stdout:0/467: write d7/d1e/f45 [1159748,31309] 0 2024-04-20T21:32:01.880 INFO:tasks.workunit.client.0.smithi062.stdout:1/533: mknod dc/d70/cb9 0 2024-04-20T21:32:01.882 INFO:tasks.workunit.client.0.smithi062.stdout:1/534: stat dc/d2d/d84/ca9 0 2024-04-20T21:32:01.884 INFO:tasks.workunit.client.0.smithi062.stdout:5/513: chown d8/d1e/d2c/d6c/d56/d2f/d8e/f97 88820703 1 2024-04-20T21:32:01.886 INFO:tasks.workunit.client.0.smithi062.stdout:2/537: link d7/da/d15/d2f/d97/fa0 d7/da/d15/d2f/d38/db0/fb2 0 2024-04-20T21:32:01.889 INFO:tasks.workunit.client.1.smithi189.stdout:1/123: write d8/d21/f22 [391361,102920] 0 2024-04-20T21:32:01.891 INFO:tasks.workunit.client.0.smithi062.stdout:0/468: unlink d7/d1c/f32 0 2024-04-20T21:32:01.892 INFO:tasks.workunit.client.1.smithi189.stdout:0/246: dwrite d3/df/f15 [0,4194304] 0 2024-04-20T21:32:01.893 INFO:tasks.workunit.client.1.smithi189.stdout:2/106: mknod d1/d11/c1b 0 2024-04-20T21:32:01.896 INFO:tasks.workunit.client.0.smithi062.stdout:7/524: creat d7/d3e/d33/d3d/fb1 x:0 0 0 2024-04-20T21:32:01.897 INFO:tasks.workunit.client.1.smithi189.stdout:7/136: dread d5/f24 [0,4194304] 0 2024-04-20T21:32:01.900 INFO:tasks.workunit.client.1.smithi189.stdout:7/137: chown d5/d21/f23 17 1 2024-04-20T21:32:01.902 INFO:tasks.workunit.client.0.smithi062.stdout:4/530: dread da/dd/d15/d50/d64/d3d/f40 [4194304,4194304] 0 2024-04-20T21:32:01.902 INFO:tasks.workunit.client.0.smithi062.stdout:1/535: getdents dc/d70/d7f 0 2024-04-20T21:32:01.905 INFO:tasks.workunit.client.0.smithi062.stdout:7/525: creat d7/d3e/d33/d3d/d5e/d72/fb2 x:0 0 0 2024-04-20T21:32:01.909 INFO:tasks.workunit.client.1.smithi189.stdout:3/102: dwrite d2/fd [4194304,4194304] 0 2024-04-20T21:32:01.911 INFO:tasks.workunit.client.1.smithi189.stdout:9/152: creat d9/dc/d17/f2f x:0 0 0 2024-04-20T21:32:01.913 INFO:tasks.workunit.client.1.smithi189.stdout:4/120: dread d2/dd/f2d [0,4194304] 0 2024-04-20T21:32:01.916 INFO:tasks.workunit.client.1.smithi189.stdout:9/153: fsync d9/dc/f2c 0 2024-04-20T21:32:01.917 INFO:tasks.workunit.client.0.smithi062.stdout:4/531: getdents da/dd/df/d27/da5 0 2024-04-20T21:32:01.923 INFO:tasks.workunit.client.1.smithi189.stdout:0/247: dwrite d3/d26/f1d [0,4194304] 0 2024-04-20T21:32:01.927 INFO:tasks.workunit.client.0.smithi062.stdout:5/514: dwrite d8/d1e/d2c/d6c/d56/d50/fa3 [0,4194304] 0 2024-04-20T21:32:01.932 INFO:tasks.workunit.client.0.smithi062.stdout:7/526: link d7/d3e/d33/f35 d7/d3e/d33/d3d/d4d/d54/d4f/fb3 0 2024-04-20T21:32:01.936 INFO:tasks.workunit.client.0.smithi062.stdout:1/536: link dc/d15/d75/d94/d36/l85 dc/d15/d75/d94/lba 0 2024-04-20T21:32:01.938 INFO:tasks.workunit.client.1.smithi189.stdout:4/121: dwrite d2/f10 [0,4194304] 0 2024-04-20T21:32:01.938 INFO:tasks.workunit.client.1.smithi189.stdout:2/107: fdatasync d1/d11/f15 0 2024-04-20T21:32:01.939 INFO:tasks.workunit.client.1.smithi189.stdout:2/108: truncate d1/f9 1174034 0 2024-04-20T21:32:01.941 INFO:tasks.workunit.client.1.smithi189.stdout:2/109: fdatasync d1/f7 0 2024-04-20T21:32:01.941 INFO:tasks.workunit.client.0.smithi062.stdout:4/532: creat da/dd/d15/d50/d64/d3d/da1/fb9 x:0 0 0 2024-04-20T21:32:01.943 INFO:tasks.workunit.client.0.smithi062.stdout:1/537: chown dc/d15/d75/d94/d36/d78 721 1 2024-04-20T21:32:01.948 INFO:tasks.workunit.client.0.smithi062.stdout:1/538: readlink dc/d2d/l86 0 2024-04-20T21:32:01.948 INFO:tasks.workunit.client.0.smithi062.stdout:2/538: truncate d7/f5d 474630 0 2024-04-20T21:32:01.950 INFO:tasks.workunit.client.0.smithi062.stdout:7/527: creat d7/d3e/d33/d3d/d5e/fb4 x:0 0 0 2024-04-20T21:32:01.951 INFO:tasks.workunit.client.0.smithi062.stdout:0/469: dwrite f1 [0,4194304] 0 2024-04-20T21:32:01.953 INFO:tasks.workunit.client.1.smithi189.stdout:9/154: creat d9/d12/d15/f30 x:0 0 0 2024-04-20T21:32:01.954 INFO:tasks.workunit.client.1.smithi189.stdout:7/138: truncate d5/f11 908188 0 2024-04-20T21:32:01.956 INFO:tasks.workunit.client.0.smithi062.stdout:3/503: dwrite d0/d1/d10/d31/f5f [0,4194304] 0 2024-04-20T21:32:01.966 INFO:tasks.workunit.client.0.smithi062.stdout:7/528: readlink d7/d3e/d33/d3d/d5e/d72/l6f 0 2024-04-20T21:32:01.966 INFO:tasks.workunit.client.1.smithi189.stdout:0/248: symlink d3/df/d28/d2b/d2c/l54 0 2024-04-20T21:32:01.970 INFO:tasks.workunit.client.1.smithi189.stdout:9/155: dread d9/dc/fe [0,4194304] 0 2024-04-20T21:32:01.972 INFO:tasks.workunit.client.1.smithi189.stdout:9/156: dread - d9/dc/d17/f24 zero size 2024-04-20T21:32:01.972 INFO:tasks.workunit.client.0.smithi062.stdout:0/470: readlink d7/d1c/l3b 0 2024-04-20T21:32:01.976 INFO:tasks.workunit.client.0.smithi062.stdout:7/529: readlink d7/d3e/d33/d3d/d4d/d83/d94/l97 0 2024-04-20T21:32:01.977 INFO:tasks.workunit.client.1.smithi189.stdout:6/123: getdents d8/da 0 2024-04-20T21:32:01.979 INFO:tasks.workunit.client.1.smithi189.stdout:6/124: stat d8 0 2024-04-20T21:32:01.979 INFO:tasks.workunit.client.0.smithi062.stdout:7/530: chown d7/f36 94 1 2024-04-20T21:32:01.980 INFO:tasks.workunit.client.1.smithi189.stdout:2/110: dread d1/d11/f15 [0,4194304] 0 2024-04-20T21:32:01.982 INFO:tasks.workunit.client.0.smithi062.stdout:3/504: rename d0/d1b/f23 to d0/d5c/d76/faa 0 2024-04-20T21:32:01.984 INFO:tasks.workunit.client.1.smithi189.stdout:3/103: rename d2/f7 to d2/f21 0 2024-04-20T21:32:01.985 INFO:tasks.workunit.client.0.smithi062.stdout:1/539: dwrite dc/d15/d75/d94/d36/d3a/f59 [0,4194304] 0 2024-04-20T21:32:01.985 INFO:tasks.workunit.client.0.smithi062.stdout:4/533: dwrite da/f7b [0,4194304] 0 2024-04-20T21:32:01.986 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:01 smithi189 ceph-mon[21562]: pgmap v15: 65 pgs: 65 active+clean; 1.1 GiB data, 3.4 GiB used, 533 GiB / 536 GiB avail; 30 MiB/s rd, 133 MiB/s wr, 285 op/s 2024-04-20T21:32:01.987 INFO:tasks.workunit.client.1.smithi189.stdout:1/124: getdents d8/d1e/d28 0 2024-04-20T21:32:01.987 INFO:tasks.workunit.client.1.smithi189.stdout:1/125: readlink d8/ld 0 2024-04-20T21:32:01.988 INFO:tasks.workunit.client.0.smithi062.stdout:0/471: truncate d7/d20/d62/d6f/f73 1612347 0 2024-04-20T21:32:01.996 INFO:tasks.workunit.client.1.smithi189.stdout:5/114: sync 2024-04-20T21:32:01.996 INFO:tasks.workunit.client.1.smithi189.stdout:8/249: sync 2024-04-20T21:32:01.996 INFO:tasks.workunit.client.1.smithi189.stdout:7/139: unlink d5/f9 0 2024-04-20T21:32:01.998 INFO:tasks.workunit.client.1.smithi189.stdout:4/122: creat d2/dd/d11/f31 x:0 0 0 2024-04-20T21:32:01.999 INFO:tasks.workunit.client.0.smithi062.stdout:1/540: symlink dc/d70/d7f/lbb 0 2024-04-20T21:32:02.000 INFO:tasks.workunit.client.1.smithi189.stdout:9/157: getdents d9/dc/d17/d2b 0 2024-04-20T21:32:02.001 INFO:tasks.workunit.client.1.smithi189.stdout:2/111: dwrite d1/f12 [0,4194304] 0 2024-04-20T21:32:02.003 INFO:tasks.workunit.client.0.smithi062.stdout:0/472: fsync d7/d1c/d15/d17/f38 0 2024-04-20T21:32:02.006 INFO:tasks.workunit.client.0.smithi062.stdout:7/531: dwrite d7/d3e/f41 [4194304,4194304] 0 2024-04-20T21:32:02.009 INFO:tasks.workunit.client.0.smithi062.stdout:1/541: mknod dc/d27/d9a/cbc 0 2024-04-20T21:32:02.011 INFO:tasks.workunit.client.0.smithi062.stdout:4/534: dread da/f7b [0,4194304] 0 2024-04-20T21:32:02.013 INFO:tasks.workunit.client.1.smithi189.stdout:5/115: creat d7/d9/f22 x:0 0 0 2024-04-20T21:32:02.016 INFO:tasks.workunit.client.1.smithi189.stdout:2/112: read d1/f4 [5010191,115618] 0 2024-04-20T21:32:02.017 INFO:tasks.workunit.client.0.smithi062.stdout:0/473: mknod d7/d1c/d15/d17/d40/d3c/ca2 0 2024-04-20T21:32:02.018 INFO:tasks.workunit.client.1.smithi189.stdout:5/116: truncate d7/d14/d18/f17 1074082 0 2024-04-20T21:32:02.018 INFO:tasks.workunit.client.1.smithi189.stdout:7/140: mkdir d5/d1d/d31 0 2024-04-20T21:32:02.019 INFO:tasks.workunit.client.1.smithi189.stdout:2/113: chown d1/d11/c13 417 1 2024-04-20T21:32:02.020 INFO:tasks.workunit.client.0.smithi062.stdout:3/505: write d0/d1/d10/f7a [179019,32973] 0 2024-04-20T21:32:02.021 INFO:tasks.workunit.client.1.smithi189.stdout:3/104: creat d2/d8/f22 x:0 0 0 2024-04-20T21:32:02.022 INFO:tasks.workunit.client.0.smithi062.stdout:4/535: write da/dd/df/d27/d3f/f4c [2500910,57675] 0 2024-04-20T21:32:02.024 INFO:tasks.workunit.client.0.smithi062.stdout:1/542: mknod dc/d15/d20/cbd 0 2024-04-20T21:32:02.024 INFO:tasks.workunit.client.1.smithi189.stdout:0/249: dread d3/f1f [0,4194304] 0 2024-04-20T21:32:02.024 INFO:tasks.workunit.client.1.smithi189.stdout:7/141: truncate d5/d1d/d28/f2f 97252 0 2024-04-20T21:32:02.025 INFO:tasks.workunit.client.1.smithi189.stdout:4/123: creat d2/f32 x:0 0 0 2024-04-20T21:32:02.025 INFO:tasks.workunit.client.1.smithi189.stdout:9/158: creat d9/d12/d15/f31 x:0 0 0 2024-04-20T21:32:02.029 INFO:tasks.workunit.client.0.smithi062.stdout:4/536: write da/dd/f45 [232637,2355] 0 2024-04-20T21:32:02.032 INFO:tasks.workunit.client.1.smithi189.stdout:0/250: read d3/df/d4c/f41 [2689270,39341] 0 2024-04-20T21:32:02.033 INFO:tasks.workunit.client.0.smithi062.stdout:2/539: dwrite d7/da/d15/d2f/d97/fa0 [0,4194304] 0 2024-04-20T21:32:02.033 INFO:tasks.workunit.client.0.smithi062.stdout:1/543: mkdir dc/d2d/d84/dbe 0 2024-04-20T21:32:02.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:01 smithi062 ceph-mon[16221]: pgmap v15: 65 pgs: 65 active+clean; 1.1 GiB data, 3.4 GiB used, 533 GiB / 536 GiB avail; 30 MiB/s rd, 133 MiB/s wr, 285 op/s 2024-04-20T21:32:02.036 INFO:tasks.workunit.client.0.smithi062.stdout:5/515: dread d8/f44 [0,4194304] 0 2024-04-20T21:32:02.037 INFO:tasks.workunit.client.0.smithi062.stdout:0/474: rename d7/d1c/f11 to d7/d1c/d15/d17/d37/d64/fa3 0 2024-04-20T21:32:02.037 INFO:tasks.workunit.client.1.smithi189.stdout:6/125: rmdir d8/d12/d16/d23 0 2024-04-20T21:32:02.039 INFO:tasks.workunit.client.1.smithi189.stdout:5/117: dwrite d7/d19/f1c [0,4194304] 0 2024-04-20T21:32:02.042 INFO:tasks.workunit.client.1.smithi189.stdout:5/118: chown d7/d14/d18/c21 1 1 2024-04-20T21:32:02.042 INFO:tasks.workunit.client.1.smithi189.stdout:6/126: chown d8/da/lc 980530 1 2024-04-20T21:32:02.044 INFO:tasks.workunit.client.0.smithi062.stdout:0/475: fsync d7/d1c/d15/d17/d1b/d9d/fa0 0 2024-04-20T21:32:02.046 INFO:tasks.workunit.client.1.smithi189.stdout:4/124: creat d2/d20/f33 x:0 0 0 2024-04-20T21:32:02.047 INFO:tasks.workunit.client.1.smithi189.stdout:5/119: rename d7/d9 to d7/d9/d23 22 2024-04-20T21:32:02.047 INFO:tasks.workunit.client.1.smithi189.stdout:4/125: readlink l1 0 2024-04-20T21:32:02.050 INFO:tasks.workunit.client.0.smithi062.stdout:0/476: write d7/d20/d62/d6f/f73 [224078,89409] 0 2024-04-20T21:32:02.055 INFO:tasks.workunit.client.0.smithi062.stdout:4/537: dwrite da/d56/d71/f93 [0,4194304] 0 2024-04-20T21:32:02.058 INFO:tasks.workunit.client.1.smithi189.stdout:5/120: mknod d7/d14/c24 0 2024-04-20T21:32:02.061 INFO:tasks.workunit.client.1.smithi189.stdout:2/114: dwrite d1/f5 [0,4194304] 0 2024-04-20T21:32:02.063 INFO:tasks.workunit.client.1.smithi189.stdout:1/126: truncate d8/f1b 3345162 0 2024-04-20T21:32:02.063 INFO:tasks.workunit.client.0.smithi062.stdout:5/516: unlink d8/d10/l15 0 2024-04-20T21:32:02.067 INFO:tasks.workunit.client.1.smithi189.stdout:1/127: write d8/f1a [701401,37928] 0 2024-04-20T21:32:02.068 INFO:tasks.workunit.client.1.smithi189.stdout:9/159: getdents d9 0 2024-04-20T21:32:02.070 INFO:tasks.workunit.client.1.smithi189.stdout:1/128: chown d8/df/f15 7247 1 2024-04-20T21:32:02.073 INFO:tasks.workunit.client.1.smithi189.stdout:0/251: creat d3/df/d28/d2b/f55 x:0 0 0 2024-04-20T21:32:02.076 INFO:tasks.workunit.client.0.smithi062.stdout:7/532: dwrite d7/d3e/f45 [0,4194304] 0 2024-04-20T21:32:02.077 INFO:tasks.workunit.client.1.smithi189.stdout:1/129: write f1 [3173581,110306] 0 2024-04-20T21:32:02.081 INFO:tasks.workunit.client.0.smithi062.stdout:7/533: chown d7/f24 106243764 1 2024-04-20T21:32:02.084 INFO:tasks.workunit.client.0.smithi062.stdout:0/477: getdents d7/d20/d62 0 2024-04-20T21:32:02.088 INFO:tasks.workunit.client.1.smithi189.stdout:3/105: link l1 d2/l23 0 2024-04-20T21:32:02.089 INFO:tasks.workunit.client.0.smithi062.stdout:1/544: write dc/d15/d75/d94/d36/d3a/f43 [1989461,10692] 0 2024-04-20T21:32:02.090 INFO:tasks.workunit.client.1.smithi189.stdout:2/115: dwrite d1/f16 [0,4194304] 0 2024-04-20T21:32:02.091 INFO:tasks.workunit.client.1.smithi189.stdout:3/106: dread - d2/d3/f19 zero size 2024-04-20T21:32:02.097 INFO:tasks.workunit.client.1.smithi189.stdout:1/130: mkdir d8/d21/d2c 0 2024-04-20T21:32:02.098 INFO:tasks.workunit.client.0.smithi062.stdout:1/545: fsync dc/d15/d20/f45 0 2024-04-20T21:32:02.102 INFO:tasks.workunit.client.0.smithi062.stdout:0/478: mkdir d7/d1c/d15/d17/d40/d3c/da4 0 2024-04-20T21:32:02.103 INFO:tasks.workunit.client.1.smithi189.stdout:1/131: chown f6 188982 1 2024-04-20T21:32:02.106 INFO:tasks.workunit.client.0.smithi062.stdout:5/517: dwrite d8/d38/f89 [0,4194304] 0 2024-04-20T21:32:02.107 INFO:tasks.workunit.client.1.smithi189.stdout:9/160: dread f5 [0,4194304] 0 2024-04-20T21:32:02.107 INFO:tasks.workunit.client.1.smithi189.stdout:4/126: rename d2/l12 to d2/dd/d11/d14/l34 0 2024-04-20T21:32:02.108 INFO:tasks.workunit.client.0.smithi062.stdout:2/540: dwrite d7/da/d15/d2f/f33 [0,4194304] 0 2024-04-20T21:32:02.111 INFO:tasks.workunit.client.0.smithi062.stdout:1/546: symlink dc/d15/d75/d94/d6f/lbf 0 2024-04-20T21:32:02.114 INFO:tasks.workunit.client.0.smithi062.stdout:7/534: write d7/d3e/f99 [341261,76057] 0 2024-04-20T21:32:02.121 INFO:tasks.workunit.client.0.smithi062.stdout:2/541: truncate d7/da/d15/d1b/d3d/f94 99549 0 2024-04-20T21:32:02.126 INFO:tasks.workunit.client.1.smithi189.stdout:6/127: dread d8/f21 [0,4194304] 0 2024-04-20T21:32:02.127 INFO:tasks.workunit.client.0.smithi062.stdout:4/538: dread da/dd/df/d27/f36 [0,4194304] 0 2024-04-20T21:32:02.127 INFO:tasks.workunit.client.1.smithi189.stdout:0/252: dwrite d3/df/d30/d37/f31 [0,4194304] 0 2024-04-20T21:32:02.131 INFO:tasks.workunit.client.0.smithi062.stdout:2/542: symlink d7/da/d15/d1b/d24/lb3 0 2024-04-20T21:32:02.133 INFO:tasks.workunit.client.1.smithi189.stdout:1/132: link c5 d8/d1e/c2d 0 2024-04-20T21:32:02.136 INFO:tasks.workunit.client.1.smithi189.stdout:7/142: write d5/ff [1309904,47956] 0 2024-04-20T21:32:02.137 INFO:tasks.workunit.client.1.smithi189.stdout:8/250: sync 2024-04-20T21:32:02.137 INFO:tasks.workunit.client.1.smithi189.stdout:4/127: rmdir d2/dd 39 2024-04-20T21:32:02.137 INFO:tasks.workunit.client.0.smithi062.stdout:7/535: creat d7/d3e/d33/d3d/dad/fb5 x:0 0 0 2024-04-20T21:32:02.140 INFO:tasks.workunit.client.0.smithi062.stdout:0/479: dread d7/d20/d62/d6f/f73 [0,4194304] 0 2024-04-20T21:32:02.140 INFO:tasks.workunit.client.0.smithi062.stdout:7/536: read d7/f6c [55135,20643] 0 2024-04-20T21:32:02.142 INFO:tasks.workunit.client.1.smithi189.stdout:1/133: write f6 [930510,100852] 0 2024-04-20T21:32:02.143 INFO:tasks.workunit.client.0.smithi062.stdout:5/518: creat d8/d38/d7a/fa5 x:0 0 0 2024-04-20T21:32:02.146 INFO:tasks.workunit.client.1.smithi189.stdout:7/143: dread - d5/d1d/f2b zero size 2024-04-20T21:32:02.153 INFO:tasks.workunit.client.1.smithi189.stdout:1/134: symlink d8/d21/d2c/l2e 0 2024-04-20T21:32:02.153 INFO:tasks.workunit.client.1.smithi189.stdout:7/144: read d5/fe [530651,121751] 0 2024-04-20T21:32:02.155 INFO:tasks.workunit.client.0.smithi062.stdout:4/539: mknod da/dd/cba 0 2024-04-20T21:32:02.157 INFO:tasks.workunit.client.1.smithi189.stdout:8/251: creat da/df/d22/d33/f5a x:0 0 0 2024-04-20T21:32:02.159 INFO:tasks.workunit.client.1.smithi189.stdout:4/128: rename d2/dd/c13 to d2/dd/c35 0 2024-04-20T21:32:02.159 INFO:tasks.workunit.client.0.smithi062.stdout:1/547: getdents dc/d15/d75 0 2024-04-20T21:32:02.160 INFO:tasks.workunit.client.1.smithi189.stdout:9/161: write d9/d12/f14 [73990,92547] 0 2024-04-20T21:32:02.161 INFO:tasks.workunit.client.1.smithi189.stdout:8/252: chown l7 0 1 2024-04-20T21:32:02.161 INFO:tasks.workunit.client.1.smithi189.stdout:5/121: dwrite d7/d9/fc [0,4194304] 0 2024-04-20T21:32:02.161 INFO:tasks.workunit.client.1.smithi189.stdout:7/145: creat d5/d21/f32 x:0 0 0 2024-04-20T21:32:02.161 INFO:tasks.workunit.client.1.smithi189.stdout:3/107: dwrite d2/f21 [0,4194304] 0 2024-04-20T21:32:02.162 INFO:tasks.workunit.client.0.smithi062.stdout:5/519: symlink d8/d1e/d2c/d6c/la6 0 2024-04-20T21:32:02.167 INFO:tasks.workunit.client.1.smithi189.stdout:1/135: symlink d8/d1e/l2f 0 2024-04-20T21:32:02.167 INFO:tasks.workunit.client.0.smithi062.stdout:1/548: mkdir dc/d15/d20/d28/d83/dc0 0 2024-04-20T21:32:02.168 INFO:tasks.workunit.client.1.smithi189.stdout:4/129: stat d2/dd/d11/d14/c23 0 2024-04-20T21:32:02.170 INFO:tasks.workunit.client.0.smithi062.stdout:0/480: dwrite d7/d1c/d15/d17/d1b/d74/f88 [0,4194304] 0 2024-04-20T21:32:02.171 INFO:tasks.workunit.client.0.smithi062.stdout:2/543: getdents d7/da/d15/d2f/d38/d6a 0 2024-04-20T21:32:02.173 INFO:tasks.workunit.client.0.smithi062.stdout:4/540: creat da/dd/df/d27/d7a/fbb x:0 0 0 2024-04-20T21:32:02.173 INFO:tasks.workunit.client.0.smithi062.stdout:5/520: rmdir d8/d1e/d2c/d6c/d56/d2f 39 2024-04-20T21:32:02.175 INFO:tasks.workunit.client.1.smithi189.stdout:6/128: truncate d8/f1a 708528 0 2024-04-20T21:32:02.175 INFO:tasks.workunit.client.1.smithi189.stdout:9/162: mknod d9/dc/d17/c32 0 2024-04-20T21:32:02.178 INFO:tasks.workunit.client.0.smithi062.stdout:5/521: chown d8/d1e/d2c/d6c/d56/d50/d88/f8c 45486 1 2024-04-20T21:32:02.178 INFO:tasks.workunit.client.0.smithi062.stdout:4/541: read f4 [1237529,123276] 0 2024-04-20T21:32:02.179 INFO:tasks.workunit.client.1.smithi189.stdout:8/253: symlink da/df/d20/l5b 0 2024-04-20T21:32:02.183 INFO:tasks.workunit.client.1.smithi189.stdout:2/116: dread d1/f6 [0,4194304] 0 2024-04-20T21:32:02.183 INFO:tasks.workunit.client.1.smithi189.stdout:8/254: write da/de/d1e/f52 [354742,41099] 0 2024-04-20T21:32:02.185 INFO:tasks.workunit.client.1.smithi189.stdout:7/146: creat d5/f33 x:0 0 0 2024-04-20T21:32:02.186 INFO:tasks.workunit.client.0.smithi062.stdout:0/481: stat d7/d1c/c8b 0 2024-04-20T21:32:02.186 INFO:tasks.workunit.client.1.smithi189.stdout:6/129: fdatasync f7 0 2024-04-20T21:32:02.187 INFO:tasks.workunit.client.1.smithi189.stdout:3/108: getdents d2/d1c 0 2024-04-20T21:32:02.191 INFO:tasks.workunit.client.1.smithi189.stdout:0/253: dwrite d3/df/d4c/d35/f44 [0,4194304] 0 2024-04-20T21:32:02.193 INFO:tasks.workunit.client.0.smithi062.stdout:2/544: rmdir d7/da/d15/d1b/d24 39 2024-04-20T21:32:02.196 INFO:tasks.workunit.client.1.smithi189.stdout:5/122: link d7/d14/c24 d7/d14/d1b/d1f/c25 0 2024-04-20T21:32:02.196 INFO:tasks.workunit.client.1.smithi189.stdout:3/109: chown d2/d8/le 40013 1 2024-04-20T21:32:02.197 INFO:tasks.workunit.client.1.smithi189.stdout:8/255: mknod da/df/d13/c5c 0 2024-04-20T21:32:02.198 INFO:tasks.workunit.client.0.smithi062.stdout:4/542: rmdir da/dd/d15/d50/d64/d57/d5b 39 2024-04-20T21:32:02.202 INFO:tasks.workunit.client.1.smithi189.stdout:7/147: mkdir d5/d21/d25/d34 0 2024-04-20T21:32:02.202 INFO:tasks.workunit.client.1.smithi189.stdout:2/117: dread d1/f9 [0,4194304] 0 2024-04-20T21:32:02.205 INFO:tasks.workunit.client.1.smithi189.stdout:3/110: fsync d2/d3/f1b 0 2024-04-20T21:32:02.205 INFO:tasks.workunit.client.0.smithi062.stdout:0/482: mkdir d7/d1c/d15/d17/d1b/d74/da5 0 2024-04-20T21:32:02.206 INFO:tasks.workunit.client.1.smithi189.stdout:6/130: rename d8/da/dd/fe to d8/d12/f24 0 2024-04-20T21:32:02.210 INFO:tasks.workunit.client.0.smithi062.stdout:5/522: rmdir d8/d10/d3e/d8a/d9a 0 2024-04-20T21:32:02.213 INFO:tasks.workunit.client.1.smithi189.stdout:2/118: fsync d1/f4 0 2024-04-20T21:32:02.213 INFO:tasks.workunit.client.1.smithi189.stdout:5/123: creat d7/d14/f26 x:0 0 0 2024-04-20T21:32:02.217 INFO:tasks.workunit.client.0.smithi062.stdout:2/545: getdents d7/da/d28/d4b 0 2024-04-20T21:32:02.217 INFO:tasks.workunit.client.1.smithi189.stdout:1/136: dwrite d8/f1a [0,4194304] 0 2024-04-20T21:32:02.218 INFO:tasks.workunit.client.1.smithi189.stdout:7/148: mkdir d5/d21/d35 0 2024-04-20T21:32:02.219 INFO:tasks.workunit.client.1.smithi189.stdout:3/111: mknod d2/d3/c24 0 2024-04-20T21:32:02.221 INFO:tasks.workunit.client.1.smithi189.stdout:4/130: dwrite d2/d20/f33 [0,4194304] 0 2024-04-20T21:32:02.222 INFO:tasks.workunit.client.0.smithi062.stdout:0/483: creat d7/d1c/d15/d17/d26/d76/fa6 x:0 0 0 2024-04-20T21:32:02.222 INFO:tasks.workunit.client.1.smithi189.stdout:8/256: link da/df/f11 da/df/d3a/f5d 0 2024-04-20T21:32:02.226 INFO:tasks.workunit.client.1.smithi189.stdout:8/257: dread - da/df/d3a/f4c zero size 2024-04-20T21:32:02.227 INFO:tasks.workunit.client.1.smithi189.stdout:2/119: mknod d1/d17/c1c 0 2024-04-20T21:32:02.228 INFO:tasks.workunit.client.1.smithi189.stdout:6/131: rename d8/da/dd/c22 to d8/d10/c25 0 2024-04-20T21:32:02.228 INFO:tasks.workunit.client.0.smithi062.stdout:4/543: dwrite da/dd/f45 [0,4194304] 0 2024-04-20T21:32:02.230 INFO:tasks.workunit.client.1.smithi189.stdout:2/120: fdatasync d1/d11/f1a 0 2024-04-20T21:32:02.233 INFO:tasks.workunit.client.1.smithi189.stdout:2/121: truncate d1/f7 1237679 0 2024-04-20T21:32:02.233 INFO:tasks.workunit.client.1.smithi189.stdout:7/149: mkdir d5/d1d/d28/d36 0 2024-04-20T21:32:02.234 INFO:tasks.workunit.client.0.smithi062.stdout:1/549: dread dc/d15/d20/d28/f4e [0,4194304] 0 2024-04-20T21:32:02.234 INFO:tasks.workunit.client.0.smithi062.stdout:5/523: mkdir d8/d38/d7a/d4c/da7 0 2024-04-20T21:32:02.235 INFO:tasks.workunit.client.0.smithi062.stdout:2/546: readlink d7/da/d15/d2f/d38/l3e 0 2024-04-20T21:32:02.236 INFO:tasks.workunit.client.0.smithi062.stdout:1/550: readlink dc/l9d 0 2024-04-20T21:32:02.239 INFO:tasks.workunit.client.1.smithi189.stdout:5/124: write d7/d14/d18/f17 [653926,112612] 0 2024-04-20T21:32:02.243 INFO:tasks.workunit.client.1.smithi189.stdout:0/254: link d3/df/d30/l52 d3/df/d30/d45/l56 0 2024-04-20T21:32:02.247 INFO:tasks.workunit.client.1.smithi189.stdout:8/258: dread da/f39 [0,4194304] 0 2024-04-20T21:32:02.250 INFO:tasks.workunit.client.1.smithi189.stdout:8/259: chown da/df/d3a/f5d 14357207 1 2024-04-20T21:32:02.254 INFO:tasks.workunit.client.1.smithi189.stdout:5/125: mknod d7/d14/d1b/c27 0 2024-04-20T21:32:02.257 INFO:tasks.workunit.client.0.smithi062.stdout:0/484: dwrite d7/d1e/d6e/f7d [0,4194304] 0 2024-04-20T21:32:02.257 INFO:tasks.workunit.client.0.smithi062.stdout:5/524: link d8/f4f d8/d1e/d2c/d6c/d56/d66/fa8 0 2024-04-20T21:32:02.259 INFO:tasks.workunit.client.1.smithi189.stdout:1/137: link d8/lb d8/d1e/d28/l30 0 2024-04-20T21:32:02.261 INFO:tasks.workunit.client.0.smithi062.stdout:4/544: dwrite da/dd/f73 [0,4194304] 0 2024-04-20T21:32:02.265 INFO:tasks.workunit.client.0.smithi062.stdout:5/525: fsync d8/d1e/d2c/f3a 0 2024-04-20T21:32:02.267 INFO:tasks.workunit.client.1.smithi189.stdout:8/260: link da/f39 da/df/d3c/d50/f5e 0 2024-04-20T21:32:02.268 INFO:tasks.workunit.client.1.smithi189.stdout:1/138: mknod d8/df/c31 0 2024-04-20T21:32:02.271 INFO:tasks.workunit.client.0.smithi062.stdout:4/545: creat da/dd/d15/d50/d64/d3d/da1/fbc x:0 0 0 2024-04-20T21:32:02.275 INFO:tasks.workunit.client.0.smithi062.stdout:1/551: write dc/d15/f18 [1559549,61723] 0 2024-04-20T21:32:02.289 INFO:tasks.workunit.client.1.smithi189.stdout:7/150: dwrite d5/d1d/d28/f2f [0,4194304] 0 2024-04-20T21:32:02.292 INFO:tasks.workunit.client.0.smithi062.stdout:6/399: sync 2024-04-20T21:32:02.292 INFO:tasks.workunit.client.0.smithi062.stdout:8/530: sync 2024-04-20T21:32:02.293 INFO:tasks.workunit.client.1.smithi189.stdout:1/139: dread d8/f1b [0,4194304] 0 2024-04-20T21:32:02.295 INFO:tasks.workunit.client.0.smithi062.stdout:1/552: symlink dc/d15/d75/lc1 0 2024-04-20T21:32:02.295 INFO:tasks.workunit.client.1.smithi189.stdout:1/140: fsync d8/f12 0 2024-04-20T21:32:02.297 INFO:tasks.workunit.client.0.smithi062.stdout:2/547: dwrite d7/f83 [0,4194304] 0 2024-04-20T21:32:02.299 INFO:tasks.workunit.client.1.smithi189.stdout:8/261: link da/f39 da/df/d13/f5f 0 2024-04-20T21:32:02.300 INFO:tasks.workunit.client.1.smithi189.stdout:3/112: truncate d2/fd 2443138 0 2024-04-20T21:32:02.308 INFO:tasks.workunit.client.1.smithi189.stdout:6/132: truncate d8/d12/f17 3914589 0 2024-04-20T21:32:02.309 INFO:tasks.workunit.client.1.smithi189.stdout:0/255: dread d3/fd [0,4194304] 0 2024-04-20T21:32:02.310 INFO:tasks.workunit.client.1.smithi189.stdout:3/113: mkdir d2/d3/d25 0 2024-04-20T21:32:02.311 INFO:tasks.workunit.client.1.smithi189.stdout:2/122: truncate d1/f16 141354 0 2024-04-20T21:32:02.312 INFO:tasks.workunit.client.0.smithi062.stdout:0/485: truncate d7/d1c/d15/d17/d37/f61 1199713 0 2024-04-20T21:32:02.312 INFO:tasks.workunit.client.1.smithi189.stdout:8/262: symlink da/de/d1e/d21/d44/l60 0 2024-04-20T21:32:02.313 INFO:tasks.workunit.client.0.smithi062.stdout:2/548: rmdir d7/da/d15/d1b/d24 39 2024-04-20T21:32:02.315 INFO:tasks.workunit.client.1.smithi189.stdout:7/151: mknod d5/c37 0 2024-04-20T21:32:02.318 INFO:tasks.workunit.client.1.smithi189.stdout:6/133: mknod d8/d12/d16/c26 0 2024-04-20T21:32:02.324 INFO:tasks.workunit.client.0.smithi062.stdout:1/553: dread dc/d15/d75/d94/d36/f54 [0,4194304] 0 2024-04-20T21:32:02.327 INFO:tasks.workunit.client.0.smithi062.stdout:8/531: dwrite d7/d25/d3b/d58/d8b/d24/fa7 [0,4194304] 0 2024-04-20T21:32:02.328 INFO:tasks.workunit.client.1.smithi189.stdout:2/123: rename d1/d17/c1c to d1/d11/c1d 0 2024-04-20T21:32:02.330 INFO:tasks.workunit.client.0.smithi062.stdout:2/549: mknod d7/da/d15/d1b/d3d/d84/d9b/cb4 0 2024-04-20T21:32:02.334 INFO:tasks.workunit.client.1.smithi189.stdout:1/141: link d8/d1e/c1f d8/df/c32 0 2024-04-20T21:32:02.336 INFO:tasks.workunit.client.0.smithi062.stdout:1/554: creat dc/d15/d20/d28/d83/dc0/fc2 x:0 0 0 2024-04-20T21:32:02.338 INFO:tasks.workunit.client.1.smithi189.stdout:0/256: symlink d3/df/d28/d2b/l57 0 2024-04-20T21:32:02.339 INFO:tasks.workunit.client.1.smithi189.stdout:8/263: creat da/df/d3c/d50/f61 x:0 0 0 2024-04-20T21:32:02.340 INFO:tasks.workunit.client.0.smithi062.stdout:8/532: readlink d7/d25/l33 0 2024-04-20T21:32:02.340 INFO:tasks.workunit.client.1.smithi189.stdout:1/142: fdatasync d8/f1a 0 2024-04-20T21:32:02.343 INFO:tasks.workunit.client.1.smithi189.stdout:3/114: dwrite d2/d8/f22 [0,4194304] 0 2024-04-20T21:32:02.343 INFO:tasks.workunit.client.1.smithi189.stdout:0/257: read - d3/df/d30/d37/f4b zero size 2024-04-20T21:32:02.345 INFO:tasks.workunit.client.1.smithi189.stdout:8/264: stat da/d1f/d2a/f4b 0 2024-04-20T21:32:02.346 INFO:tasks.workunit.client.1.smithi189.stdout:6/134: dread f2 [0,4194304] 0 2024-04-20T21:32:02.346 INFO:tasks.workunit.client.1.smithi189.stdout:1/143: write f2 [1863742,104170] 0 2024-04-20T21:32:02.350 INFO:tasks.workunit.client.1.smithi189.stdout:8/265: truncate da/df/d13/f18 4523399 0 2024-04-20T21:32:02.353 INFO:tasks.workunit.client.1.smithi189.stdout:1/144: symlink d8/df/l33 0 2024-04-20T21:32:02.355 INFO:tasks.workunit.client.0.smithi062.stdout:0/486: truncate d7/d1c/d15/d17/d40/f85 245843 0 2024-04-20T21:32:02.355 INFO:tasks.workunit.client.0.smithi062.stdout:1/555: creat dc/d15/d20/fc3 x:0 0 0 2024-04-20T21:32:02.364 INFO:tasks.workunit.client.1.smithi189.stdout:7/152: dwrite d5/f7 [0,4194304] 0 2024-04-20T21:32:02.365 INFO:tasks.workunit.client.1.smithi189.stdout:9/163: sync 2024-04-20T21:32:02.366 INFO:tasks.workunit.client.1.smithi189.stdout:0/258: rename d3/f3b to d3/df/d28/d2b/f58 0 2024-04-20T21:32:02.367 INFO:tasks.workunit.client.0.smithi062.stdout:0/487: fdatasync d7/d1c/d15/d53/f9c 0 2024-04-20T21:32:02.368 INFO:tasks.workunit.client.0.smithi062.stdout:6/400: dread d0/f5e [0,4194304] 0 2024-04-20T21:32:02.368 INFO:tasks.workunit.client.0.smithi062.stdout:2/550: dwrite d7/da/d15/d2f/d38/d6b/f88 [0,4194304] 0 2024-04-20T21:32:02.369 INFO:tasks.workunit.client.1.smithi189.stdout:8/266: dread da/de/d1e/f52 [0,4194304] 0 2024-04-20T21:32:02.372 INFO:tasks.workunit.client.1.smithi189.stdout:8/267: stat da/de/d1e/d21/d44/l60 0 2024-04-20T21:32:02.374 INFO:tasks.workunit.client.0.smithi062.stdout:2/551: write d7/da/d15/d1b/d3d/f94 [565216,26416] 0 2024-04-20T21:32:02.374 INFO:tasks.workunit.client.1.smithi189.stdout:9/164: rmdir d9/d12 39 2024-04-20T21:32:02.376 INFO:tasks.workunit.client.1.smithi189.stdout:1/145: dwrite d8/df/f15 [0,4194304] 0 2024-04-20T21:32:02.377 INFO:tasks.workunit.client.1.smithi189.stdout:3/115: link d2/c11 d2/c26 0 2024-04-20T21:32:02.381 INFO:tasks.workunit.client.1.smithi189.stdout:7/153: dread d5/f6 [0,4194304] 0 2024-04-20T21:32:02.381 INFO:tasks.workunit.client.1.smithi189.stdout:2/124: chown d1/d11/c1d 266726206 1 2024-04-20T21:32:02.382 INFO:tasks.workunit.client.0.smithi062.stdout:4/546: dread da/dd/d3c/f53 [0,4194304] 0 2024-04-20T21:32:02.383 INFO:tasks.workunit.client.0.smithi062.stdout:0/488: creat d7/d1c/d15/d17/d26/fa7 x:0 0 0 2024-04-20T21:32:02.383 INFO:tasks.workunit.client.0.smithi062.stdout:6/401: symlink d0/d9/d11/d5c/l7e 0 2024-04-20T21:32:02.384 INFO:tasks.workunit.client.0.smithi062.stdout:8/533: dread d7/d9/f51 [4194304,4194304] 0 2024-04-20T21:32:02.384 INFO:tasks.workunit.client.1.smithi189.stdout:0/259: rename d3/df/d30/f3d to d3/df/d30/d45/f59 0 2024-04-20T21:32:02.386 INFO:tasks.workunit.client.1.smithi189.stdout:2/125: write d1/f5 [3266983,19199] 0 2024-04-20T21:32:02.387 INFO:tasks.workunit.client.0.smithi062.stdout:4/547: mkdir da/d56/d71/dbd 0 2024-04-20T21:32:02.390 INFO:tasks.workunit.client.1.smithi189.stdout:9/165: fdatasync d9/d12/f29 0 2024-04-20T21:32:02.390 INFO:tasks.workunit.client.1.smithi189.stdout:3/116: fsync d2/fd 0 2024-04-20T21:32:02.390 INFO:tasks.workunit.client.1.smithi189.stdout:2/126: chown d1/d11/c14 1542743 1 2024-04-20T21:32:02.390 INFO:tasks.workunit.client.0.smithi062.stdout:6/402: write d0/d1a/d44/f4b [1159025,75373] 0 2024-04-20T21:32:02.391 INFO:tasks.workunit.client.0.smithi062.stdout:0/489: fdatasync d7/d1c/d15/d17/d40/d79/d71/f94 0 2024-04-20T21:32:02.391 INFO:tasks.workunit.client.0.smithi062.stdout:4/548: stat da/dd/d15/d50/d64/c48 0 2024-04-20T21:32:02.393 INFO:tasks.workunit.client.1.smithi189.stdout:1/146: rename d8/fe to d8/df/f34 0 2024-04-20T21:32:02.394 INFO:tasks.workunit.client.0.smithi062.stdout:8/534: mkdir d7/d30/db3 0 2024-04-20T21:32:02.399 INFO:tasks.workunit.client.1.smithi189.stdout:0/260: mkdir d3/d26/d5a 0 2024-04-20T21:32:02.400 INFO:tasks.workunit.client.1.smithi189.stdout:9/166: truncate f3 2663989 0 2024-04-20T21:32:02.400 INFO:tasks.workunit.client.0.smithi062.stdout:1/556: dwrite dc/d15/d20/d28/f66 [0,4194304] 0 2024-04-20T21:32:02.401 INFO:tasks.workunit.client.0.smithi062.stdout:4/549: symlink da/dd/d15/d4b/lbe 0 2024-04-20T21:32:02.401 INFO:tasks.workunit.client.0.smithi062.stdout:8/535: mknod d7/da9/cb4 0 2024-04-20T21:32:02.402 INFO:tasks.workunit.client.1.smithi189.stdout:2/127: mkdir d1/d11/d1e 0 2024-04-20T21:32:02.406 INFO:tasks.workunit.client.0.smithi062.stdout:0/490: rename d7/d1c/d15/d17/l50 to d7/d1c/d15/d17/d40/la8 0 2024-04-20T21:32:02.406 INFO:tasks.workunit.client.1.smithi189.stdout:1/147: rename d8/d1e/c20 to d8/c35 0 2024-04-20T21:32:02.407 INFO:tasks.workunit.client.1.smithi189.stdout:0/261: read - d3/df/d30/d37/f4b zero size 2024-04-20T21:32:02.407 INFO:tasks.workunit.client.1.smithi189.stdout:9/167: fsync d9/dc/f1d 0 2024-04-20T21:32:02.407 INFO:tasks.workunit.client.1.smithi189.stdout:6/135: dwrite f7 [0,4194304] 0 2024-04-20T21:32:02.407 INFO:tasks.workunit.client.1.smithi189.stdout:7/154: link d5/fd d5/d1d/d28/f38 0 2024-04-20T21:32:02.409 INFO:tasks.workunit.client.0.smithi062.stdout:4/550: creat da/dd/df/d27/d3f/d82/fbf x:0 0 0 2024-04-20T21:32:02.411 INFO:tasks.workunit.client.1.smithi189.stdout:2/128: write d1/f7 [1313058,31885] 0 2024-04-20T21:32:02.411 INFO:tasks.workunit.client.1.smithi189.stdout:1/148: write d8/d1e/f23 [415186,114603] 0 2024-04-20T21:32:02.415 INFO:tasks.workunit.client.1.smithi189.stdout:0/262: chown d3/f27 0 1 2024-04-20T21:32:02.415 INFO:tasks.workunit.client.1.smithi189.stdout:7/155: write d5/f1a [679749,45210] 0 2024-04-20T21:32:02.417 INFO:tasks.workunit.client.1.smithi189.stdout:6/136: creat d8/da/f27 x:0 0 0 2024-04-20T21:32:02.425 INFO:tasks.workunit.client.0.smithi062.stdout:6/403: fdatasync d0/d1a/d44/f4b 0 2024-04-20T21:32:02.425 INFO:tasks.workunit.client.1.smithi189.stdout:1/149: truncate d8/f14 4875648 0 2024-04-20T21:32:02.427 INFO:tasks.workunit.client.0.smithi062.stdout:0/491: rmdir d7/d1c/d15/d17/d1b/d74/da5 0 2024-04-20T21:32:02.428 INFO:tasks.workunit.client.1.smithi189.stdout:3/117: dwrite d2/d8/f10 [4194304,4194304] 0 2024-04-20T21:32:02.434 INFO:tasks.workunit.client.1.smithi189.stdout:6/137: creat d8/da/dd/f28 x:0 0 0 2024-04-20T21:32:02.435 INFO:tasks.workunit.client.0.smithi062.stdout:8/536: dwrite d7/d25/d4f/fae [0,4194304] 0 2024-04-20T21:32:02.442 INFO:tasks.workunit.client.0.smithi062.stdout:8/537: chown d7/fa8 2 1 2024-04-20T21:32:02.442 INFO:tasks.workunit.client.1.smithi189.stdout:9/168: dwrite d9/dc/d17/f2f [0,4194304] 0 2024-04-20T21:32:02.443 INFO:tasks.workunit.client.1.smithi189.stdout:7/156: creat d5/f39 x:0 0 0 2024-04-20T21:32:02.445 INFO:tasks.workunit.client.1.smithi189.stdout:9/169: write d9/d12/f14 [3536803,116717] 0 2024-04-20T21:32:02.446 INFO:tasks.workunit.client.0.smithi062.stdout:8/538: chown d7/d25/d3b 77364 1 2024-04-20T21:32:02.449 INFO:tasks.workunit.client.1.smithi189.stdout:1/150: mkdir d8/d21/d2c/d36 0 2024-04-20T21:32:02.450 INFO:tasks.workunit.client.0.smithi062.stdout:8/539: creat d7/d25/d4f/fb5 x:0 0 0 2024-04-20T21:32:02.452 INFO:tasks.workunit.client.1.smithi189.stdout:1/151: truncate d8/f11 5082762 0 2024-04-20T21:32:02.455 INFO:tasks.workunit.client.1.smithi189.stdout:6/138: mknod d8/da/dd/c29 0 2024-04-20T21:32:02.457 INFO:tasks.workunit.client.0.smithi062.stdout:1/557: dread dc/ff [0,4194304] 0 2024-04-20T21:32:02.458 INFO:tasks.workunit.client.1.smithi189.stdout:9/170: dread d9/dc/fe [0,4194304] 0 2024-04-20T21:32:02.459 INFO:tasks.workunit.client.1.smithi189.stdout:9/171: stat d9/d12/f14 0 2024-04-20T21:32:02.459 INFO:tasks.workunit.client.1.smithi189.stdout:7/157: write d5/f20 [271233,41587] 0 2024-04-20T21:32:02.459 INFO:tasks.workunit.client.0.smithi062.stdout:6/404: dread d0/d9/d11/d22/f24 [0,4194304] 0 2024-04-20T21:32:02.462 INFO:tasks.workunit.client.1.smithi189.stdout:7/158: write d5/f7 [4462226,56448] 0 2024-04-20T21:32:02.464 INFO:tasks.workunit.client.0.smithi062.stdout:4/551: dwrite da/dd/d15/d50/d64/d57/d5b/f94 [0,4194304] 0 2024-04-20T21:32:02.465 INFO:tasks.workunit.client.1.smithi189.stdout:8/268: dread da/df/d20/d27/f38 [0,4194304] 0 2024-04-20T21:32:02.465 INFO:tasks.workunit.client.1.smithi189.stdout:7/159: read d5/d21/f23 [271971,83266] 0 2024-04-20T21:32:02.469 INFO:tasks.workunit.client.1.smithi189.stdout:7/160: rename d5 to d5/d1d/d3a 22 2024-04-20T21:32:02.470 INFO:tasks.workunit.client.1.smithi189.stdout:7/161: chown d5/c37 4787 1 2024-04-20T21:32:02.471 INFO:tasks.workunit.client.1.smithi189.stdout:0/263: getdents d3/d26 0 2024-04-20T21:32:02.472 INFO:tasks.workunit.client.1.smithi189.stdout:4/131: sync 2024-04-20T21:32:02.473 INFO:tasks.workunit.client.1.smithi189.stdout:6/139: mkdir d8/da/dd/d2a 0 2024-04-20T21:32:02.474 INFO:tasks.workunit.client.0.smithi062.stdout:4/552: mknod da/dd/d15/d50/d64/d57/d5b/cc0 0 2024-04-20T21:32:02.474 INFO:tasks.workunit.client.1.smithi189.stdout:0/264: stat d3/df/d4c/d35/f44 0 2024-04-20T21:32:02.475 INFO:tasks.workunit.client.1.smithi189.stdout:7/162: truncate d5/d1d/d28/f2c 708248 0 2024-04-20T21:32:02.476 INFO:tasks.workunit.client.0.smithi062.stdout:8/540: creat d7/d25/d3b/fb6 x:0 0 0 2024-04-20T21:32:02.476 INFO:tasks.workunit.client.0.smithi062.stdout:6/405: creat d0/d9/d11/d5c/d7c/f7f x:0 0 0 2024-04-20T21:32:02.482 INFO:tasks.workunit.client.0.smithi062.stdout:0/492: write d7/d1c/d15/d17/f38 [503366,115676] 0 2024-04-20T21:32:02.482 INFO:tasks.workunit.client.0.smithi062.stdout:4/553: fsync da/dd/d15/d50/d64/d3d/f5e 0 2024-04-20T21:32:02.491 INFO:tasks.workunit.client.1.smithi189.stdout:6/140: rmdir d8 39 2024-04-20T21:32:02.497 INFO:tasks.workunit.client.1.smithi189.stdout:7/163: symlink d5/d1d/d28/l3b 0 2024-04-20T21:32:02.498 INFO:tasks.workunit.client.1.smithi189.stdout:0/265: dwrite d3/df/f15 [0,4194304] 0 2024-04-20T21:32:02.500 INFO:tasks.workunit.client.1.smithi189.stdout:7/164: truncate d5/d21/f32 909461 0 2024-04-20T21:32:02.500 INFO:tasks.workunit.client.1.smithi189.stdout:0/266: chown d3/d26/d5a 61315 1 2024-04-20T21:32:02.502 INFO:tasks.workunit.client.0.smithi062.stdout:4/554: mknod da/dd/d15/cc1 0 2024-04-20T21:32:02.502 INFO:tasks.workunit.client.0.smithi062.stdout:6/406: creat d0/f80 x:0 0 0 2024-04-20T21:32:02.503 INFO:tasks.workunit.client.1.smithi189.stdout:3/118: fdatasync d2/f15 0 2024-04-20T21:32:02.506 INFO:tasks.workunit.client.1.smithi189.stdout:1/152: truncate d8/f1a 3999827 0 2024-04-20T21:32:02.507 INFO:tasks.workunit.client.1.smithi189.stdout:2/129: truncate d1/d11/f1a 1204744 0 2024-04-20T21:32:02.507 INFO:tasks.workunit.client.0.smithi062.stdout:1/558: dwrite dc/d15/f16 [4194304,4194304] 0 2024-04-20T21:32:02.507 INFO:tasks.workunit.client.0.smithi062.stdout:8/541: link d7/d25/d5c/c9f d7/d25/d70/cb7 0 2024-04-20T21:32:02.509 INFO:tasks.workunit.client.1.smithi189.stdout:4/132: mkdir d2/d36 0 2024-04-20T21:32:02.512 INFO:tasks.workunit.client.0.smithi062.stdout:1/559: read dc/d15/d75/d94/d36/f54 [4592659,21413] 0 2024-04-20T21:32:02.517 INFO:tasks.workunit.client.0.smithi062.stdout:8/542: creat d7/d25/d4f/fb8 x:0 0 0 2024-04-20T21:32:02.517 INFO:tasks.workunit.client.0.smithi062.stdout:4/555: creat da/dd/df/d27/fc2 x:0 0 0 2024-04-20T21:32:02.517 INFO:tasks.workunit.client.1.smithi189.stdout:5/126: sync 2024-04-20T21:32:02.520 INFO:tasks.workunit.client.1.smithi189.stdout:5/127: stat d7/d14/f26 0 2024-04-20T21:32:02.524 INFO:tasks.workunit.client.1.smithi189.stdout:7/165: dwrite d5/f20 [0,4194304] 0 2024-04-20T21:32:02.527 INFO:tasks.workunit.client.1.smithi189.stdout:7/166: fdatasync d5/d21/d25/f27 0 2024-04-20T21:32:02.528 INFO:tasks.workunit.client.0.smithi062.stdout:0/493: dread d7/d1c/ff [0,4194304] 0 2024-04-20T21:32:02.528 INFO:tasks.workunit.client.1.smithi189.stdout:2/130: symlink d1/d11/l1f 0 2024-04-20T21:32:02.529 INFO:tasks.workunit.client.1.smithi189.stdout:4/133: dread d2/dd/f2d [0,4194304] 0 2024-04-20T21:32:02.530 INFO:tasks.workunit.client.0.smithi062.stdout:8/543: unlink d7/d25/d3b/d58/d8b/d24/d46/f49 0 2024-04-20T21:32:02.533 INFO:tasks.workunit.client.0.smithi062.stdout:8/544: write d7/fab [257744,38211] 0 2024-04-20T21:32:02.535 INFO:tasks.workunit.client.1.smithi189.stdout:2/131: write d1/f4 [5495057,80244] 0 2024-04-20T21:32:02.540 INFO:tasks.workunit.client.1.smithi189.stdout:2/132: chown d1/cb 233424560 1 2024-04-20T21:32:02.541 INFO:tasks.workunit.client.1.smithi189.stdout:9/172: write f8 [894849,11096] 0 2024-04-20T21:32:02.543 INFO:tasks.workunit.client.1.smithi189.stdout:2/133: write d1/f7 [2245630,16683] 0 2024-04-20T21:32:02.543 INFO:tasks.workunit.client.1.smithi189.stdout:5/128: unlink d7/d9/f22 0 2024-04-20T21:32:02.546 INFO:tasks.workunit.client.0.smithi062.stdout:4/556: dwrite da/dd/d15/fa7 [0,4194304] 0 2024-04-20T21:32:02.550 INFO:tasks.workunit.client.1.smithi189.stdout:8/269: dwrite da/de/f10 [0,4194304] 0 2024-04-20T21:32:02.550 INFO:tasks.workunit.client.0.smithi062.stdout:9/500: sync 2024-04-20T21:32:02.551 INFO:tasks.workunit.client.0.smithi062.stdout:3/506: sync 2024-04-20T21:32:02.555 INFO:tasks.workunit.client.1.smithi189.stdout:4/134: dread d2/d20/f24 [0,4194304] 0 2024-04-20T21:32:02.556 INFO:tasks.workunit.client.1.smithi189.stdout:0/267: rmdir d3/d26/d5a 0 2024-04-20T21:32:02.556 INFO:tasks.workunit.client.1.smithi189.stdout:4/135: chown d2/f2c 1769824 1 2024-04-20T21:32:02.558 INFO:tasks.workunit.client.0.smithi062.stdout:4/557: creat da/dd/df/d27/d3f/fc3 x:0 0 0 2024-04-20T21:32:02.564 INFO:tasks.workunit.client.1.smithi189.stdout:6/141: mkdir d8/d2b 0 2024-04-20T21:32:02.565 INFO:tasks.workunit.client.0.smithi062.stdout:9/501: write d16/d2c/d42/f50 [3161749,74980] 0 2024-04-20T21:32:02.566 INFO:tasks.workunit.client.1.smithi189.stdout:5/129: readlink d7/d14/d18/l1e 0 2024-04-20T21:32:02.568 INFO:tasks.workunit.client.1.smithi189.stdout:8/270: symlink da/df/d22/d33/l62 0 2024-04-20T21:32:02.568 INFO:tasks.workunit.client.1.smithi189.stdout:1/153: link d8/d1e/l2a d8/d1e/d28/d2b/l37 0 2024-04-20T21:32:02.568 INFO:tasks.workunit.client.1.smithi189.stdout:5/130: readlink d7/l12 0 2024-04-20T21:32:02.572 INFO:tasks.workunit.client.0.smithi062.stdout:4/558: symlink da/dd/d15/d50/d64/d3d/da1/lc4 0 2024-04-20T21:32:02.575 INFO:tasks.workunit.client.1.smithi189.stdout:4/136: mkdir d2/dd/d37 0 2024-04-20T21:32:02.576 INFO:tasks.workunit.client.0.smithi062.stdout:1/560: write dc/d15/f48 [4749160,117871] 0 2024-04-20T21:32:02.578 INFO:tasks.workunit.client.1.smithi189.stdout:7/167: rmdir d5/d1d/d28/d36 0 2024-04-20T21:32:02.579 INFO:tasks.workunit.client.0.smithi062.stdout:9/502: symlink d16/d2c/d5e/la8 0 2024-04-20T21:32:02.580 INFO:tasks.workunit.client.0.smithi062.stdout:1/561: fsync dc/d27/d32/f89 0 2024-04-20T21:32:02.581 INFO:tasks.workunit.client.1.smithi189.stdout:1/154: creat d8/d21/f38 x:0 0 0 2024-04-20T21:32:02.581 INFO:tasks.workunit.client.0.smithi062.stdout:0/494: dread d7/d1c/d15/d53/f9c [0,4194304] 0 2024-04-20T21:32:02.583 INFO:tasks.workunit.client.1.smithi189.stdout:1/155: truncate d8/d1e/f23 718586 0 2024-04-20T21:32:02.584 INFO:tasks.workunit.client.1.smithi189.stdout:9/173: creat d9/f33 x:0 0 0 2024-04-20T21:32:02.584 INFO:tasks.workunit.client.1.smithi189.stdout:6/142: mknod d8/d2b/c2c 0 2024-04-20T21:32:02.585 INFO:tasks.workunit.client.0.smithi062.stdout:8/545: getdents d7/d25/d3b/d58/d8b/d24/d46 0 2024-04-20T21:32:02.587 INFO:tasks.workunit.client.1.smithi189.stdout:4/137: write d2/dd/f2d [1215230,76819] 0 2024-04-20T21:32:02.588 INFO:tasks.workunit.client.1.smithi189.stdout:8/271: mkdir da/df/d13/d63 0 2024-04-20T21:32:02.588 INFO:tasks.workunit.client.1.smithi189.stdout:3/119: truncate d2/f6 8299204 0 2024-04-20T21:32:02.589 INFO:tasks.workunit.client.0.smithi062.stdout:0/495: fdatasync d7/d1c/d15/d17/d40/d79/d71/f94 0 2024-04-20T21:32:02.590 INFO:tasks.workunit.client.0.smithi062.stdout:6/407: dread d0/d9/d11/f1f [0,4194304] 0 2024-04-20T21:32:02.590 INFO:tasks.workunit.client.0.smithi062.stdout:0/496: chown d7 0 1 2024-04-20T21:32:02.591 INFO:tasks.workunit.client.0.smithi062.stdout:4/559: symlink da/dd/df/d27/d3f/lc5 0 2024-04-20T21:32:02.592 INFO:tasks.workunit.client.1.smithi189.stdout:4/138: dread - d2/dd/d11/f31 zero size 2024-04-20T21:32:02.593 INFO:tasks.workunit.client.1.smithi189.stdout:3/120: readlink d2/d3/l4 0 2024-04-20T21:32:02.593 INFO:tasks.workunit.client.0.smithi062.stdout:6/408: dread - d0/d2c/d67/f6a zero size 2024-04-20T21:32:02.596 INFO:tasks.workunit.client.0.smithi062.stdout:8/546: mkdir d7/d25/d3b/d58/db9 0 2024-04-20T21:32:02.596 INFO:tasks.workunit.client.0.smithi062.stdout:9/503: mkdir d16/d2c/d42/da6/da9 0 2024-04-20T21:32:02.597 INFO:tasks.workunit.client.1.smithi189.stdout:6/143: creat d8/d10/f2d x:0 0 0 2024-04-20T21:32:02.602 INFO:tasks.workunit.client.0.smithi062.stdout:9/504: rmdir d16/d2c/d5e/d75/d76 39 2024-04-20T21:32:02.603 INFO:tasks.workunit.client.1.smithi189.stdout:9/174: dread d9/dc/fe [0,4194304] 0 2024-04-20T21:32:02.603 INFO:tasks.workunit.client.0.smithi062.stdout:6/409: creat d0/d9/f81 x:0 0 0 2024-04-20T21:32:02.606 INFO:tasks.workunit.client.1.smithi189.stdout:3/121: symlink d2/d8/l27 0 2024-04-20T21:32:02.608 INFO:tasks.workunit.client.1.smithi189.stdout:3/122: fdatasync d2/d3/f1b 0 2024-04-20T21:32:02.613 INFO:tasks.workunit.client.1.smithi189.stdout:6/144: read d8/f21 [3777805,93509] 0 2024-04-20T21:32:02.614 INFO:tasks.workunit.client.1.smithi189.stdout:6/145: chown d8/da/f27 14608 1 2024-04-20T21:32:02.616 INFO:tasks.workunit.client.1.smithi189.stdout:2/134: truncate d1/f7 1782181 0 2024-04-20T21:32:02.617 INFO:tasks.workunit.client.1.smithi189.stdout:9/175: stat f3 0 2024-04-20T21:32:02.619 INFO:tasks.workunit.client.1.smithi189.stdout:6/146: readlink d8/l18 0 2024-04-20T21:32:02.619 INFO:tasks.workunit.client.1.smithi189.stdout:4/139: creat d2/f38 x:0 0 0 2024-04-20T21:32:02.620 INFO:tasks.workunit.client.1.smithi189.stdout:9/176: stat d9/dc/d17/l20 0 2024-04-20T21:32:02.621 INFO:tasks.workunit.client.0.smithi062.stdout:0/497: dread d7/d1c/d15/d53/f9c [0,4194304] 0 2024-04-20T21:32:02.625 INFO:tasks.workunit.client.1.smithi189.stdout:3/123: dwrite d2/f5 [0,4194304] 0 2024-04-20T21:32:02.626 INFO:tasks.workunit.client.1.smithi189.stdout:1/156: dread d8/f9 [0,4194304] 0 2024-04-20T21:32:02.626 INFO:tasks.workunit.client.0.smithi062.stdout:3/507: dwrite d0/d1/d10/d3a/d5e/d65/d88/f8a [0,4194304] 0 2024-04-20T21:32:02.629 INFO:tasks.workunit.client.1.smithi189.stdout:6/147: chown d8/f1a 1020025 1 2024-04-20T21:32:02.631 INFO:tasks.workunit.client.0.smithi062.stdout:4/560: dread da/dd/d15/fa6 [0,4194304] 0 2024-04-20T21:32:02.631 INFO:tasks.workunit.client.0.smithi062.stdout:0/498: mkdir d7/d1e/da9 0 2024-04-20T21:32:02.633 INFO:tasks.workunit.client.1.smithi189.stdout:3/124: rename d2/f21 to d2/d1c/f28 0 2024-04-20T21:32:02.634 INFO:tasks.workunit.client.0.smithi062.stdout:9/505: dwrite d16/f32 [0,4194304] 0 2024-04-20T21:32:02.635 INFO:tasks.workunit.client.1.smithi189.stdout:6/148: truncate d8/d10/f2d 811181 0 2024-04-20T21:32:02.637 INFO:tasks.workunit.client.1.smithi189.stdout:9/177: creat d9/f34 x:0 0 0 2024-04-20T21:32:02.637 INFO:tasks.workunit.client.1.smithi189.stdout:8/272: dread da/df/d13/f14 [0,4194304] 0 2024-04-20T21:32:02.638 INFO:tasks.workunit.client.1.smithi189.stdout:2/135: dread d1/f4 [0,4194304] 0 2024-04-20T21:32:02.639 INFO:tasks.workunit.client.0.smithi062.stdout:0/499: chown d7/d20/c21 11 1 2024-04-20T21:32:02.642 INFO:tasks.workunit.client.1.smithi189.stdout:4/140: creat d2/dd/d37/f39 x:0 0 0 2024-04-20T21:32:02.644 INFO:tasks.workunit.client.1.smithi189.stdout:4/141: fdatasync f0 0 2024-04-20T21:32:02.644 INFO:tasks.workunit.client.0.smithi062.stdout:3/508: creat d0/d1/d3c/d43/d72/da9/fab x:0 0 0 2024-04-20T21:32:02.646 INFO:tasks.workunit.client.1.smithi189.stdout:1/157: rename d8/d1e/c1f to d8/d1e/d28/d2b/c39 0 2024-04-20T21:32:02.648 INFO:tasks.workunit.client.0.smithi062.stdout:6/410: dwrite d0/d1a/f3a [0,4194304] 0 2024-04-20T21:32:02.649 INFO:tasks.workunit.client.0.smithi062.stdout:8/547: dwrite d7/d25/d5c/f69 [0,4194304] 0 2024-04-20T21:32:02.651 INFO:tasks.workunit.client.0.smithi062.stdout:9/506: dread d16/d2f/f8f [0,4194304] 0 2024-04-20T21:32:02.652 INFO:tasks.workunit.client.1.smithi189.stdout:6/149: mknod d8/d12/d16/c2e 0 2024-04-20T21:32:02.653 INFO:tasks.workunit.client.1.smithi189.stdout:9/178: symlink d9/l35 0 2024-04-20T21:32:02.653 INFO:tasks.workunit.client.1.smithi189.stdout:6/150: chown d8/l18 1651624 1 2024-04-20T21:32:02.654 INFO:tasks.workunit.client.1.smithi189.stdout:3/125: creat d2/f29 x:0 0 0 2024-04-20T21:32:02.655 INFO:tasks.workunit.client.1.smithi189.stdout:6/151: stat d8/d12/f24 0 2024-04-20T21:32:02.658 INFO:tasks.workunit.client.1.smithi189.stdout:2/136: symlink d1/d11/d1e/l20 0 2024-04-20T21:32:02.658 INFO:tasks.workunit.client.1.smithi189.stdout:1/158: mkdir d8/df/d3a 0 2024-04-20T21:32:02.658 INFO:tasks.workunit.client.1.smithi189.stdout:4/142: mkdir d2/d3a 0 2024-04-20T21:32:02.659 INFO:tasks.workunit.client.1.smithi189.stdout:9/179: mkdir d9/dc/d17/d36 0 2024-04-20T21:32:02.661 INFO:tasks.workunit.client.1.smithi189.stdout:6/152: fsync d8/d12/f17 0 2024-04-20T21:32:02.664 INFO:tasks.workunit.client.0.smithi062.stdout:6/411: symlink d0/d4f/l82 0 2024-04-20T21:32:02.667 INFO:tasks.workunit.client.0.smithi062.stdout:8/548: link d7/d25/d3b/d58/d8b/d24/f50 d7/d30/fba 0 2024-04-20T21:32:02.667 INFO:tasks.workunit.client.1.smithi189.stdout:2/137: dread d1/f9 [0,4194304] 0 2024-04-20T21:32:02.669 INFO:tasks.workunit.client.1.smithi189.stdout:1/159: mknod d8/d21/d2c/d36/c3b 0 2024-04-20T21:32:02.670 INFO:tasks.workunit.client.0.smithi062.stdout:4/561: dread da/dd/d15/d50/d64/d57/d5b/f94 [0,4194304] 0 2024-04-20T21:32:02.671 INFO:tasks.workunit.client.1.smithi189.stdout:2/138: chown d1/d17 5 1 2024-04-20T21:32:02.674 INFO:tasks.workunit.client.1.smithi189.stdout:3/126: creat d2/f2a x:0 0 0 2024-04-20T21:32:02.676 INFO:tasks.workunit.client.1.smithi189.stdout:6/153: dread d8/ff [0,4194304] 0 2024-04-20T21:32:02.676 INFO:tasks.workunit.client.1.smithi189.stdout:1/160: fsync d8/f9 0 2024-04-20T21:32:02.679 INFO:tasks.workunit.client.1.smithi189.stdout:2/139: write d1/f5 [1548437,120163] 0 2024-04-20T21:32:02.681 INFO:tasks.workunit.client.0.smithi062.stdout:3/509: truncate d0/f4c 3972482 0 2024-04-20T21:32:02.685 INFO:tasks.workunit.client.1.smithi189.stdout:4/143: dwrite d2/f8 [0,4194304] 0 2024-04-20T21:32:02.685 INFO:tasks.workunit.client.1.smithi189.stdout:9/180: dwrite d9/dc/f1d [0,4194304] 0 2024-04-20T21:32:02.686 INFO:tasks.workunit.client.1.smithi189.stdout:3/127: creat d2/d1c/f2b x:0 0 0 2024-04-20T21:32:02.686 INFO:tasks.workunit.client.1.smithi189.stdout:6/154: fsync d8/f21 0 2024-04-20T21:32:02.687 INFO:tasks.workunit.client.0.smithi062.stdout:9/507: link d16/f1c d16/d2c/d5e/d75/faa 0 2024-04-20T21:32:02.687 INFO:tasks.workunit.client.1.smithi189.stdout:1/161: creat d8/d1e/f3c x:0 0 0 2024-04-20T21:32:02.691 INFO:tasks.workunit.client.0.smithi062.stdout:3/510: write d0/d1/d3c/d43/d67/f86 [941873,22716] 0 2024-04-20T21:32:02.698 INFO:tasks.workunit.client.0.smithi062.stdout:6/412: dwrite d0/fe [0,4194304] 0 2024-04-20T21:32:02.699 INFO:tasks.workunit.client.1.smithi189.stdout:6/155: unlink f4 0 2024-04-20T21:32:02.701 INFO:tasks.workunit.client.0.smithi062.stdout:3/511: creat d0/d1b/fac x:0 0 0 2024-04-20T21:32:02.704 INFO:tasks.workunit.client.0.smithi062.stdout:6/413: creat d0/d52/f83 x:0 0 0 2024-04-20T21:32:02.704 INFO:tasks.workunit.client.1.smithi189.stdout:3/128: mkdir d2/d3/d2c 0 2024-04-20T21:32:02.707 INFO:tasks.workunit.client.0.smithi062.stdout:0/500: dwrite d7/d1c/d15/d17/d26/f68 [0,4194304] 0 2024-04-20T21:32:02.708 INFO:tasks.workunit.client.0.smithi062.stdout:6/414: chown d0/d2c/f3b 833 1 2024-04-20T21:32:02.713 INFO:tasks.workunit.client.0.smithi062.stdout:0/501: chown d7/d1c/d15/d53 122594 1 2024-04-20T21:32:02.715 INFO:tasks.workunit.client.1.smithi189.stdout:9/181: rename d9/dc/d17/f24 to d9/d12/d15/f37 0 2024-04-20T21:32:02.716 INFO:tasks.workunit.client.0.smithi062.stdout:3/512: rename d0/d5c/c28 to d0/d1/cad 0 2024-04-20T21:32:02.717 INFO:tasks.workunit.client.1.smithi189.stdout:4/144: creat d2/d3a/f3b x:0 0 0 2024-04-20T21:32:02.721 INFO:tasks.workunit.client.0.smithi062.stdout:3/513: chown d0/d1/f19 19162 1 2024-04-20T21:32:02.721 INFO:tasks.workunit.client.1.smithi189.stdout:8/273: dread da/df/d3a/f5d [0,4194304] 0 2024-04-20T21:32:02.727 INFO:tasks.workunit.client.0.smithi062.stdout:3/514: readlink d0/d1/d10/l26 0 2024-04-20T21:32:02.730 INFO:tasks.workunit.client.1.smithi189.stdout:1/162: getdents d8/d1e/d28 0 2024-04-20T21:32:02.734 INFO:tasks.workunit.client.0.smithi062.stdout:0/502: rename d7/d1c/d15/d17/d40/d3c/ca2 to d7/d1c/d15/d17/d1b/d9d/caa 0 2024-04-20T21:32:02.736 INFO:tasks.workunit.client.0.smithi062.stdout:4/562: dwrite da/dd/df/f10 [0,4194304] 0 2024-04-20T21:32:02.737 INFO:tasks.workunit.client.0.smithi062.stdout:3/515: symlink d0/d1/d3c/d43/lae 0 2024-04-20T21:32:02.740 INFO:tasks.workunit.client.1.smithi189.stdout:1/163: creat d8/df/f3d x:0 0 0 2024-04-20T21:32:02.741 INFO:tasks.workunit.client.1.smithi189.stdout:6/156: link d8/d10/c25 d8/da/dd/c2f 0 2024-04-20T21:32:02.741 INFO:tasks.workunit.client.0.smithi062.stdout:9/508: dwrite d16/d4e/f6f [0,4194304] 0 2024-04-20T21:32:02.744 INFO:tasks.workunit.client.0.smithi062.stdout:9/509: chown d16/d2c/l59 767302 1 2024-04-20T21:32:02.744 INFO:tasks.workunit.client.1.smithi189.stdout:8/274: rename l7 to da/df/d22/l64 0 2024-04-20T21:32:02.745 INFO:tasks.workunit.client.0.smithi062.stdout:4/563: creat da/dd/d15/d50/d64/d3d/fc6 x:0 0 0 2024-04-20T21:32:02.749 INFO:tasks.workunit.client.0.smithi062.stdout:9/510: mkdir d16/d2f/d5b/dab 0 2024-04-20T21:32:02.749 INFO:tasks.workunit.client.1.smithi189.stdout:6/157: mkdir d8/d12/d30 0 2024-04-20T21:32:02.752 INFO:tasks.workunit.client.1.smithi189.stdout:9/182: dwrite f4 [0,4194304] 0 2024-04-20T21:32:02.752 INFO:tasks.workunit.client.1.smithi189.stdout:1/164: rename d8/d1e/c27 to d8/df/d3a/c3e 0 2024-04-20T21:32:02.753 INFO:tasks.workunit.client.1.smithi189.stdout:8/275: chown da/df/d20/d2c 942 1 2024-04-20T21:32:02.755 INFO:tasks.workunit.client.0.smithi062.stdout:9/511: symlink d16/d2f/d69/lac 0 2024-04-20T21:32:02.762 INFO:tasks.workunit.client.1.smithi189.stdout:6/158: rename d8/ff to d8/d12/d16/f31 0 2024-04-20T21:32:02.762 INFO:tasks.workunit.client.0.smithi062.stdout:9/512: write d16/d2f/d69/f7c [1238159,52117] 0 2024-04-20T21:32:02.769 INFO:tasks.workunit.client.0.smithi062.stdout:4/564: creat da/dd/df/fc7 x:0 0 0 2024-04-20T21:32:02.772 INFO:tasks.workunit.client.0.smithi062.stdout:7/537: sync 2024-04-20T21:32:02.774 INFO:tasks.workunit.client.1.smithi189.stdout:2/140: dwrite d1/f9 [0,4194304] 0 2024-04-20T21:32:02.778 INFO:tasks.workunit.client.0.smithi062.stdout:0/503: dwrite d7/d1c/d15/d17/f72 [0,4194304] 0 2024-04-20T21:32:02.780 INFO:tasks.workunit.client.0.smithi062.stdout:7/538: mkdir d7/d26/da8/db6 0 2024-04-20T21:32:02.784 INFO:tasks.workunit.client.0.smithi062.stdout:7/539: truncate d7/d3e/d33/d3d/dad/fb5 71295 0 2024-04-20T21:32:02.785 INFO:tasks.workunit.client.0.smithi062.stdout:9/513: link d16/c2a d16/d2f/d5b/dab/cad 0 2024-04-20T21:32:02.787 INFO:tasks.workunit.client.0.smithi062.stdout:0/504: mknod d7/d1c/d15/d17/d40/d3c/da4/cab 0 2024-04-20T21:32:02.789 INFO:tasks.workunit.client.0.smithi062.stdout:9/514: fdatasync d16/fa5 0 2024-04-20T21:32:02.789 INFO:tasks.workunit.client.0.smithi062.stdout:0/505: readlink d7/d1c/d15/d17/d1b/d74/l8e 0 2024-04-20T21:32:02.791 INFO:tasks.workunit.client.1.smithi189.stdout:0/268: sync 2024-04-20T21:32:02.791 INFO:tasks.workunit.client.1.smithi189.stdout:9/183: dread d9/dc/f1d [0,4194304] 0 2024-04-20T21:32:02.794 INFO:tasks.workunit.client.1.smithi189.stdout:6/159: dread d8/d12/f17 [0,4194304] 0 2024-04-20T21:32:02.795 INFO:tasks.workunit.client.1.smithi189.stdout:4/145: dread d2/f4 [0,4194304] 0 2024-04-20T21:32:02.795 INFO:tasks.workunit.client.1.smithi189.stdout:9/184: truncate d9/d12/d15/f31 841811 0 2024-04-20T21:32:02.796 INFO:tasks.workunit.client.0.smithi062.stdout:4/565: write da/dd/df/f3e [990846,98893] 0 2024-04-20T21:32:02.797 INFO:tasks.workunit.client.1.smithi189.stdout:4/146: dread - d2/d22/f27 zero size 2024-04-20T21:32:02.798 INFO:tasks.workunit.client.0.smithi062.stdout:9/515: write d16/f36 [3660918,131032] 0 2024-04-20T21:32:02.798 INFO:tasks.workunit.client.0.smithi062.stdout:0/506: creat d7/d1e/da9/fac x:0 0 0 2024-04-20T21:32:02.801 INFO:tasks.workunit.client.1.smithi189.stdout:4/147: dread - d2/d22/f30 zero size 2024-04-20T21:32:02.806 INFO:tasks.workunit.client.0.smithi062.stdout:9/516: truncate d16/d2f/d69/f7c 2017311 0 2024-04-20T21:32:02.809 INFO:tasks.workunit.client.1.smithi189.stdout:9/185: symlink d9/d12/d15/d1a/l38 0 2024-04-20T21:32:02.811 INFO:tasks.workunit.client.0.smithi062.stdout:9/517: symlink d16/d4e/d80/lae 0 2024-04-20T21:32:02.812 INFO:tasks.workunit.client.0.smithi062.stdout:4/566: rename da/dd/d15/f20 to da/d56/d71/dbd/fc8 0 2024-04-20T21:32:02.814 INFO:tasks.workunit.client.0.smithi062.stdout:7/540: dwrite d7/f55 [4194304,4194304] 0 2024-04-20T21:32:02.815 INFO:tasks.workunit.client.0.smithi062.stdout:4/567: stat da/dd/df 0 2024-04-20T21:32:02.821 INFO:tasks.workunit.client.0.smithi062.stdout:0/507: dread d7/d1e/f67 [0,4194304] 0 2024-04-20T21:32:02.821 INFO:tasks.workunit.client.1.smithi189.stdout:9/186: rename d9/dc/d17/l28 to d9/dc/d17/d36/l39 0 2024-04-20T21:32:02.824 INFO:tasks.workunit.client.0.smithi062.stdout:0/508: read - d7/d1c/d15/d17/d1b/d74/f99 zero size 2024-04-20T21:32:02.824 INFO:tasks.workunit.client.0.smithi062.stdout:7/541: rename d7/d3e/c73 to d7/d3e/d33/d3d/d5e/d7e/d80/cb7 0 2024-04-20T21:32:02.825 INFO:tasks.workunit.client.0.smithi062.stdout:0/509: stat c4 0 2024-04-20T21:32:02.827 INFO:tasks.workunit.client.0.smithi062.stdout:9/518: link d16/d2f/f8f d16/d3d/d55/faf 0 2024-04-20T21:32:02.832 INFO:tasks.workunit.client.1.smithi189.stdout:0/269: dwrite d3/fd [0,4194304] 0 2024-04-20T21:32:02.833 INFO:tasks.workunit.client.0.smithi062.stdout:4/568: creat da/dd/df/d27/fc9 x:0 0 0 2024-04-20T21:32:02.833 INFO:tasks.workunit.client.0.smithi062.stdout:7/542: mkdir d7/d3e/d33/d3d/d5e/db8 0 2024-04-20T21:32:02.833 INFO:tasks.workunit.client.0.smithi062.stdout:0/510: creat d7/d1c/d15/d17/fad x:0 0 0 2024-04-20T21:32:02.838 INFO:tasks.workunit.client.1.smithi189.stdout:6/160: dwrite f5 [4194304,4194304] 0 2024-04-20T21:32:02.839 INFO:tasks.workunit.client.1.smithi189.stdout:8/276: dwrite da/df/d3c/d50/f5e [4194304,4194304] 0 2024-04-20T21:32:02.841 INFO:tasks.workunit.client.0.smithi062.stdout:0/511: chown d7/d1c/d15/d17/d40/la8 14538117 1 2024-04-20T21:32:02.849 INFO:tasks.workunit.client.1.smithi189.stdout:9/187: getdents d9/dc/d17 0 2024-04-20T21:32:02.850 INFO:tasks.workunit.client.1.smithi189.stdout:5/131: sync 2024-04-20T21:32:02.850 INFO:tasks.workunit.client.1.smithi189.stdout:7/168: sync 2024-04-20T21:32:02.850 INFO:tasks.workunit.client.1.smithi189.stdout:3/129: sync 2024-04-20T21:32:02.851 INFO:tasks.workunit.client.0.smithi062.stdout:9/519: link d16/l37 d16/d4e/lb0 0 2024-04-20T21:32:02.852 INFO:tasks.workunit.client.0.smithi062.stdout:0/512: creat d7/d20/d62/d6f/fae x:0 0 0 2024-04-20T21:32:02.853 INFO:tasks.workunit.client.1.smithi189.stdout:4/148: dread d2/d20/f24 [0,4194304] 0 2024-04-20T21:32:02.857 INFO:tasks.workunit.client.1.smithi189.stdout:3/130: stat l1 0 2024-04-20T21:32:02.857 INFO:tasks.workunit.client.1.smithi189.stdout:2/141: dwrite d1/f16 [0,4194304] 0 2024-04-20T21:32:02.859 INFO:tasks.workunit.client.1.smithi189.stdout:6/161: creat d8/da/f32 x:0 0 0 2024-04-20T21:32:02.860 INFO:tasks.workunit.client.1.smithi189.stdout:3/131: dread - d2/f29 zero size 2024-04-20T21:32:02.861 INFO:tasks.workunit.client.1.smithi189.stdout:2/142: write d1/fe [792827,71075] 0 2024-04-20T21:32:02.861 INFO:tasks.workunit.client.1.smithi189.stdout:4/149: dread - d2/dd/d37/f39 zero size 2024-04-20T21:32:02.862 INFO:tasks.workunit.client.1.smithi189.stdout:6/162: stat d8/da/c15 0 2024-04-20T21:32:02.864 INFO:tasks.workunit.client.0.smithi062.stdout:4/569: link da/dd/df/d27/d32/f37 da/dd/d15/d50/d64/fca 0 2024-04-20T21:32:02.870 INFO:tasks.workunit.client.1.smithi189.stdout:7/169: mknod d5/d1d/d28/c3c 0 2024-04-20T21:32:02.872 INFO:tasks.workunit.client.1.smithi189.stdout:5/132: mkdir d7/d9/d28 0 2024-04-20T21:32:02.875 INFO:tasks.workunit.client.1.smithi189.stdout:6/163: symlink d8/da/dd/l33 0 2024-04-20T21:32:02.876 INFO:tasks.workunit.client.1.smithi189.stdout:9/188: mknod d9/dc/d17/d2b/c3a 0 2024-04-20T21:32:02.877 INFO:tasks.workunit.client.0.smithi062.stdout:7/543: truncate d7/f55 400632 0 2024-04-20T21:32:02.878 INFO:tasks.workunit.client.1.smithi189.stdout:3/132: unlink d2/c26 0 2024-04-20T21:32:02.879 INFO:tasks.workunit.client.1.smithi189.stdout:7/170: symlink d5/d21/d35/l3d 0 2024-04-20T21:32:02.879 INFO:tasks.workunit.client.1.smithi189.stdout:9/189: fdatasync f8 0 2024-04-20T21:32:02.880 INFO:tasks.workunit.client.1.smithi189.stdout:0/270: getdents d3/df/d30 0 2024-04-20T21:32:02.881 INFO:tasks.workunit.client.0.smithi062.stdout:0/513: truncate d7/d1c/d15/f46 1964570 0 2024-04-20T21:32:02.883 INFO:tasks.workunit.client.0.smithi062.stdout:7/544: creat d7/d3e/d33/d3d/dad/fb9 x:0 0 0 2024-04-20T21:32:02.885 INFO:tasks.workunit.client.0.smithi062.stdout:9/520: write d16/f19 [3280349,397] 0 2024-04-20T21:32:02.886 INFO:tasks.workunit.client.0.smithi062.stdout:0/514: write d7/d20/d62/d6f/fae [904237,25051] 0 2024-04-20T21:32:02.888 INFO:tasks.workunit.client.0.smithi062.stdout:0/515: dread - d7/d1c/d15/d17/d1b/d9d/fa0 zero size 2024-04-20T21:32:02.889 INFO:tasks.workunit.client.0.smithi062.stdout:7/545: truncate d7/d3e/d33/d3d/dad/fb9 60985 0 2024-04-20T21:32:02.891 INFO:tasks.workunit.client.1.smithi189.stdout:3/133: mknod d2/c2d 0 2024-04-20T21:32:02.891 INFO:tasks.workunit.client.0.smithi062.stdout:7/546: truncate d7/ff 5028926 0 2024-04-20T21:32:02.891 INFO:tasks.workunit.client.0.smithi062.stdout:7/547: chown d7/c23 773 1 2024-04-20T21:32:02.892 INFO:tasks.workunit.client.0.smithi062.stdout:9/521: rename d16/d2f/d69/f7c to d16/d3d/d55/d6e/d9b/fb1 0 2024-04-20T21:32:02.894 INFO:tasks.workunit.client.0.smithi062.stdout:7/548: dread - d7/d3e/d33/d3d/d7f/f89 zero size 2024-04-20T21:32:02.894 INFO:tasks.workunit.client.1.smithi189.stdout:1/165: dread d8/f24 [0,4194304] 0 2024-04-20T21:32:02.895 INFO:tasks.workunit.client.0.smithi062.stdout:0/516: symlink d7/d20/d62/d6f/laf 0 2024-04-20T21:32:02.896 INFO:tasks.workunit.client.1.smithi189.stdout:7/171: symlink d5/d21/d35/l3e 0 2024-04-20T21:32:02.898 INFO:tasks.workunit.client.0.smithi062.stdout:0/517: write d7/d1c/d15/d17/d26/d76/fa6 [626698,20929] 0 2024-04-20T21:32:02.899 INFO:tasks.workunit.client.0.smithi062.stdout:7/549: symlink d7/d63/lba 0 2024-04-20T21:32:02.901 INFO:tasks.workunit.client.1.smithi189.stdout:0/271: mknod d3/df/d4c/d35/c5b 0 2024-04-20T21:32:02.902 INFO:tasks.workunit.client.1.smithi189.stdout:1/166: chown d8/d21/d2c/d36 5585737 1 2024-04-20T21:32:02.905 INFO:tasks.workunit.client.1.smithi189.stdout:0/272: chown d3 1314187 1 2024-04-20T21:32:02.906 INFO:tasks.workunit.client.1.smithi189.stdout:8/277: dwrite da/df/f11 [0,4194304] 0 2024-04-20T21:32:02.910 INFO:tasks.workunit.client.1.smithi189.stdout:2/143: dread d1/f12 [0,4194304] 0 2024-04-20T21:32:02.911 INFO:tasks.workunit.client.0.smithi062.stdout:0/518: creat d7/d1c/d15/d17/d40/d79/fb0 x:0 0 0 2024-04-20T21:32:02.912 INFO:tasks.workunit.client.1.smithi189.stdout:1/167: fdatasync d8/fa 0 2024-04-20T21:32:02.914 INFO:tasks.workunit.client.1.smithi189.stdout:7/172: write d5/d1d/d28/f2e [373591,77260] 0 2024-04-20T21:32:02.915 INFO:tasks.workunit.client.0.smithi062.stdout:7/550: symlink d7/d3e/d33/d3d/d7f/lbb 0 2024-04-20T21:32:02.917 INFO:tasks.workunit.client.0.smithi062.stdout:9/522: truncate d16/d3d/f43 2344356 0 2024-04-20T21:32:02.919 INFO:tasks.workunit.client.1.smithi189.stdout:5/133: dread d7/d9/fc [0,4194304] 0 2024-04-20T21:32:02.922 INFO:tasks.workunit.client.0.smithi062.stdout:0/519: rename d7/d1c/d15/d17/d26/d83 to d7/d1c/d15/d17/d1b/d9d/db1 0 2024-04-20T21:32:02.923 INFO:tasks.workunit.client.1.smithi189.stdout:6/164: dwrite d8/d10/f2d [0,4194304] 0 2024-04-20T21:32:02.924 INFO:tasks.workunit.client.0.smithi062.stdout:7/551: creat d7/d3e/d33/d3d/dad/fbc x:0 0 0 2024-04-20T21:32:02.924 INFO:tasks.workunit.client.1.smithi189.stdout:4/150: dwrite d2/d3a/f3b [0,4194304] 0 2024-04-20T21:32:02.924 INFO:tasks.workunit.client.1.smithi189.stdout:0/273: stat d3/d26/f2f 0 2024-04-20T21:32:02.927 INFO:tasks.workunit.client.1.smithi189.stdout:5/134: chown d7/c16 247099 1 2024-04-20T21:32:02.930 INFO:tasks.workunit.client.0.smithi062.stdout:7/552: rmdir d7/d3e/d33/d3d/d5e 39 2024-04-20T21:32:02.930 INFO:tasks.workunit.client.1.smithi189.stdout:6/165: chown c3 0 1 2024-04-20T21:32:02.930 INFO:tasks.workunit.client.1.smithi189.stdout:0/274: chown d3/df/d28/d2b 163 1 2024-04-20T21:32:02.930 INFO:tasks.workunit.client.0.smithi062.stdout:9/523: read d16/d2c/d42/f70 [3276506,88303] 0 2024-04-20T21:32:02.931 INFO:tasks.workunit.client.1.smithi189.stdout:4/151: write d2/f32 [662773,19484] 0 2024-04-20T21:32:02.933 INFO:tasks.workunit.client.1.smithi189.stdout:9/190: dwrite d9/f34 [0,4194304] 0 2024-04-20T21:32:02.934 INFO:tasks.workunit.client.1.smithi189.stdout:8/278: truncate da/d1f/d2a/f58 218443 0 2024-04-20T21:32:02.935 INFO:tasks.workunit.client.1.smithi189.stdout:3/134: write d2/d8/f10 [5386087,58680] 0 2024-04-20T21:32:02.938 INFO:tasks.workunit.client.0.smithi062.stdout:7/553: mknod d7/d3e/d33/d3d/d5e/db8/cbd 0 2024-04-20T21:32:02.939 INFO:tasks.workunit.client.1.smithi189.stdout:0/275: write d3/df/f15 [830772,84358] 0 2024-04-20T21:32:02.940 INFO:tasks.workunit.client.1.smithi189.stdout:7/173: symlink d5/d21/d35/l3f 0 2024-04-20T21:32:02.949 INFO:tasks.workunit.client.1.smithi189.stdout:4/152: creat d2/d3a/f3c x:0 0 0 2024-04-20T21:32:02.949 INFO:tasks.workunit.client.1.smithi189.stdout:0/276: rmdir d3/df/d4c 39 2024-04-20T21:32:02.950 INFO:tasks.workunit.client.0.smithi062.stdout:9/524: dwrite d16/d4e/f6f [0,4194304] 0 2024-04-20T21:32:02.953 INFO:tasks.workunit.client.1.smithi189.stdout:9/191: mkdir d9/d12/d3b 0 2024-04-20T21:32:02.953 INFO:tasks.workunit.client.0.smithi062.stdout:7/554: link d7/d26/l91 d7/d3e/d33/d3d/d5e/d7e/lbe 0 2024-04-20T21:32:02.955 INFO:tasks.workunit.client.1.smithi189.stdout:3/135: creat d2/d3/d25/f2e x:0 0 0 2024-04-20T21:32:02.955 INFO:tasks.workunit.client.0.smithi062.stdout:7/555: readlink d7/d3e/d33/l3b 0 2024-04-20T21:32:02.958 INFO:tasks.workunit.client.1.smithi189.stdout:0/277: symlink d3/df/d30/l5c 0 2024-04-20T21:32:02.960 INFO:tasks.workunit.client.1.smithi189.stdout:8/279: write da/de/f16 [912758,13787] 0 2024-04-20T21:32:02.961 INFO:tasks.workunit.client.1.smithi189.stdout:4/153: symlink d2/d36/l3d 0 2024-04-20T21:32:02.961 INFO:tasks.workunit.client.1.smithi189.stdout:3/136: fdatasync d2/d3/f19 0 2024-04-20T21:32:02.962 INFO:tasks.workunit.client.0.smithi062.stdout:0/520: unlink d7/d1c/d15/f46 0 2024-04-20T21:32:02.963 INFO:tasks.workunit.client.1.smithi189.stdout:9/192: mknod d9/d12/d3b/c3c 0 2024-04-20T21:32:02.964 INFO:tasks.workunit.client.0.smithi062.stdout:0/521: truncate d7/d1c/d15/d17/fad 528525 0 2024-04-20T21:32:02.964 INFO:tasks.workunit.client.1.smithi189.stdout:6/166: write d8/d12/d16/f31 [1527050,53371] 0 2024-04-20T21:32:02.964 INFO:tasks.workunit.client.1.smithi189.stdout:2/144: dwrite d1/d11/f15 [0,4194304] 0 2024-04-20T21:32:02.966 INFO:tasks.workunit.client.0.smithi062.stdout:9/525: rmdir d16/d3d/d55/d6e/d7d 39 2024-04-20T21:32:02.968 INFO:tasks.workunit.client.1.smithi189.stdout:1/168: dread d8/d1e/f23 [0,4194304] 0 2024-04-20T21:32:02.974 INFO:tasks.workunit.client.0.smithi062.stdout:5/526: sync 2024-04-20T21:32:02.975 INFO:tasks.workunit.client.0.smithi062.stdout:0/522: link d7/d1e/f67 d7/d1c/d15/d80/fb2 0 2024-04-20T21:32:02.976 INFO:tasks.workunit.client.0.smithi062.stdout:2/552: sync 2024-04-20T21:32:02.978 INFO:tasks.workunit.client.1.smithi189.stdout:9/193: creat d9/dc/f3d x:0 0 0 2024-04-20T21:32:02.978 INFO:tasks.workunit.client.1.smithi189.stdout:4/154: write d2/f8 [4783977,24863] 0 2024-04-20T21:32:02.979 INFO:tasks.workunit.client.1.smithi189.stdout:8/280: mkdir da/df/d20/d65 0 2024-04-20T21:32:02.979 INFO:tasks.workunit.client.1.smithi189.stdout:6/167: mknod d8/d10/c34 0 2024-04-20T21:32:02.981 INFO:tasks.workunit.client.0.smithi062.stdout:0/523: read d7/d1c/d15/d17/d1b/d74/f88 [4131912,54849] 0 2024-04-20T21:32:02.985 INFO:tasks.workunit.client.1.smithi189.stdout:4/155: fdatasync d2/d22/f30 0 2024-04-20T21:32:02.986 INFO:tasks.workunit.client.1.smithi189.stdout:5/135: truncate d7/d14/d18/f17 566077 0 2024-04-20T21:32:02.986 INFO:tasks.workunit.client.1.smithi189.stdout:2/145: symlink d1/l21 0 2024-04-20T21:32:02.987 INFO:tasks.workunit.client.0.smithi062.stdout:2/553: rename d7/da/d28/d4b/d9d to d7/db5 0 2024-04-20T21:32:02.989 INFO:tasks.workunit.client.0.smithi062.stdout:9/526: dwrite d16/f3b [0,4194304] 0 2024-04-20T21:32:02.994 INFO:tasks.workunit.client.1.smithi189.stdout:2/146: dread d1/fd [0,4194304] 0 2024-04-20T21:32:02.994 INFO:tasks.workunit.client.1.smithi189.stdout:8/281: truncate da/de/d1e/d21/d44/f43 986213 0 2024-04-20T21:32:02.994 INFO:tasks.workunit.client.1.smithi189.stdout:7/174: dwrite d5/d21/f32 [0,4194304] 0 2024-04-20T21:32:02.994 INFO:tasks.workunit.client.1.smithi189.stdout:9/194: symlink d9/d12/d15/d1a/l3e 0 2024-04-20T21:32:02.995 INFO:tasks.workunit.client.0.smithi062.stdout:2/554: write d7/da/d15/d1b/f7d [127033,27566] 0 2024-04-20T21:32:02.997 INFO:tasks.workunit.client.1.smithi189.stdout:0/278: read d3/fe [2762325,12864] 0 2024-04-20T21:32:02.997 INFO:tasks.workunit.client.1.smithi189.stdout:7/175: stat d5/f7 0 2024-04-20T21:32:02.997 INFO:tasks.workunit.client.1.smithi189.stdout:1/169: link d8/c25 d8/df/d3a/c3f 0 2024-04-20T21:32:03.002 INFO:tasks.workunit.client.0.smithi062.stdout:9/527: creat d16/d2f/d5b/dab/fb2 x:0 0 0 2024-04-20T21:32:03.003 INFO:tasks.workunit.client.0.smithi062.stdout:5/527: dwrite d8/fb [0,4194304] 0 2024-04-20T21:32:03.003 INFO:tasks.workunit.client.1.smithi189.stdout:5/136: mknod d7/d19/c29 0 2024-04-20T21:32:03.007 INFO:tasks.workunit.client.0.smithi062.stdout:5/528: fsync d8/d38/d7a/d4c/d7d/f98 0 2024-04-20T21:32:03.007 INFO:tasks.workunit.client.0.smithi062.stdout:9/528: chown d16/d2c/d5e/la8 55602 1 2024-04-20T21:32:03.008 INFO:tasks.workunit.client.1.smithi189.stdout:9/195: chown d9/dc/c2a 538 1 2024-04-20T21:32:03.010 INFO:tasks.workunit.client.1.smithi189.stdout:9/196: chown d9/d12/d15/d1a 211 1 2024-04-20T21:32:03.010 INFO:tasks.workunit.client.1.smithi189.stdout:0/279: fsync d3/df/f3a 0 2024-04-20T21:32:03.013 INFO:tasks.workunit.client.0.smithi062.stdout:2/555: symlink d7/da/d28/d4b/d8f/lb6 0 2024-04-20T21:32:03.013 INFO:tasks.workunit.client.0.smithi062.stdout:0/524: rename d7/d1c/d15/d17/d40/d79/c70 to d7/d1c/d15/cb3 0 2024-04-20T21:32:03.013 INFO:tasks.workunit.client.0.smithi062.stdout:5/529: creat d8/d1e/fa9 x:0 0 0 2024-04-20T21:32:03.014 INFO:tasks.workunit.client.1.smithi189.stdout:9/197: read - d9/d12/d15/f37 zero size 2024-04-20T21:32:03.015 INFO:tasks.workunit.client.0.smithi062.stdout:0/525: fsync d7/d20/d62/d6f/fae 0 2024-04-20T21:32:03.021 INFO:tasks.workunit.client.1.smithi189.stdout:4/156: creat d2/f3e x:0 0 0 2024-04-20T21:32:03.025 INFO:tasks.workunit.client.1.smithi189.stdout:7/176: mknod d5/d21/d25/d34/c40 0 2024-04-20T21:32:03.026 INFO:tasks.workunit.client.1.smithi189.stdout:6/168: dwrite d8/da/f32 [0,4194304] 0 2024-04-20T21:32:03.030 INFO:tasks.workunit.client.0.smithi062.stdout:2/556: rename d7/da/d15/d2f/l44 to d7/da/d15/d1b/d24/lb7 0 2024-04-20T21:32:03.031 INFO:tasks.workunit.client.0.smithi062.stdout:9/529: dwrite d16/d4e/f6f [0,4194304] 0 2024-04-20T21:32:03.037 INFO:tasks.workunit.client.0.smithi062.stdout:2/557: stat d7/c1f 0 2024-04-20T21:32:03.037 INFO:tasks.workunit.client.0.smithi062.stdout:9/530: write d16/f32 [4990671,84896] 0 2024-04-20T21:32:03.038 INFO:tasks.workunit.client.1.smithi189.stdout:6/169: read d8/d12/f24 [1548276,81467] 0 2024-04-20T21:32:03.039 INFO:tasks.workunit.client.1.smithi189.stdout:0/280: dwrite d3/fd [4194304,4194304] 0 2024-04-20T21:32:03.039 INFO:tasks.workunit.client.1.smithi189.stdout:2/147: truncate d1/f9 3435566 0 2024-04-20T21:32:03.039 INFO:tasks.workunit.client.1.smithi189.stdout:8/282: write da/d1f/f25 [437295,130840] 0 2024-04-20T21:32:03.045 INFO:tasks.workunit.client.1.smithi189.stdout:7/177: mknod d5/d21/c41 0 2024-04-20T21:32:03.045 INFO:tasks.workunit.client.0.smithi062.stdout:9/531: readlink d16/d2c/d42/l4f 0 2024-04-20T21:32:03.047 INFO:tasks.workunit.client.1.smithi189.stdout:1/170: getdents d8/d21/d2c 0 2024-04-20T21:32:03.047 INFO:tasks.workunit.client.0.smithi062.stdout:5/530: dwrite d8/d1e/d2c/d6c/d56/f39 [0,4194304] 0 2024-04-20T21:32:03.053 INFO:tasks.workunit.client.0.smithi062.stdout:2/558: creat d7/da/d15/d2f/d93/dac/fb8 x:0 0 0 2024-04-20T21:32:03.054 INFO:tasks.workunit.client.0.smithi062.stdout:0/526: dwrite d7/d1c/d15/d17/d1b/d74/f88 [0,4194304] 0 2024-04-20T21:32:03.055 INFO:tasks.workunit.client.1.smithi189.stdout:4/157: link d2/d3a/f3b d2/dd/d11/d14/f3f 0 2024-04-20T21:32:03.057 INFO:tasks.workunit.client.1.smithi189.stdout:8/283: creat da/df/d3c/d50/f66 x:0 0 0 2024-04-20T21:32:03.063 INFO:tasks.workunit.client.1.smithi189.stdout:3/137: dread d2/f6 [0,4194304] 0 2024-04-20T21:32:03.069 INFO:tasks.workunit.client.1.smithi189.stdout:0/281: mkdir d3/df/d5d 0 2024-04-20T21:32:03.069 INFO:tasks.workunit.client.1.smithi189.stdout:8/284: fsync da/de/d1e/f52 0 2024-04-20T21:32:03.070 INFO:tasks.workunit.client.1.smithi189.stdout:1/171: link d8/d21/f22 d8/f40 0 2024-04-20T21:32:03.072 INFO:tasks.workunit.client.0.smithi062.stdout:5/531: getdents d8/d1e/d2c/d6c 0 2024-04-20T21:32:03.074 INFO:tasks.workunit.client.1.smithi189.stdout:1/172: write d8/f14 [4766817,14790] 0 2024-04-20T21:32:03.076 INFO:tasks.workunit.client.1.smithi189.stdout:6/170: getdents d8/d12 0 2024-04-20T21:32:03.080 INFO:tasks.workunit.client.0.smithi062.stdout:9/532: dwrite d16/f36 [0,4194304] 0 2024-04-20T21:32:03.084 INFO:tasks.workunit.client.0.smithi062.stdout:5/532: chown d8/d1e/l78 134094 1 2024-04-20T21:32:03.089 INFO:tasks.workunit.client.1.smithi189.stdout:1/173: unlink d8/df/f34 0 2024-04-20T21:32:03.091 INFO:tasks.workunit.client.0.smithi062.stdout:2/559: dwrite d7/da/d15/d1b/f34 [0,4194304] 0 2024-04-20T21:32:03.093 INFO:tasks.workunit.client.1.smithi189.stdout:9/198: truncate f6 387297 0 2024-04-20T21:32:03.096 INFO:tasks.workunit.client.1.smithi189.stdout:2/148: dwrite d1/fe [0,4194304] 0 2024-04-20T21:32:03.100 INFO:tasks.workunit.client.0.smithi062.stdout:5/533: mkdir d8/d1e/d2c/d6c/d56/d66/daa 0 2024-04-20T21:32:03.102 INFO:tasks.workunit.client.0.smithi062.stdout:0/527: dwrite d7/d1c/d15/d17/d37/f59 [0,4194304] 0 2024-04-20T21:32:03.102 INFO:tasks.workunit.client.1.smithi189.stdout:9/199: mkdir d9/d12/d3f 0 2024-04-20T21:32:03.102 INFO:tasks.workunit.client.1.smithi189.stdout:4/158: dwrite d2/f18 [0,4194304] 0 2024-04-20T21:32:03.103 INFO:tasks.workunit.client.1.smithi189.stdout:6/171: dread d8/d12/d16/f1c [0,4194304] 0 2024-04-20T21:32:03.105 INFO:tasks.workunit.client.1.smithi189.stdout:2/149: chown d1/d11 1263644 1 2024-04-20T21:32:03.105 INFO:tasks.workunit.client.1.smithi189.stdout:2/150: stat d1/cb 0 2024-04-20T21:32:03.109 INFO:tasks.workunit.client.0.smithi062.stdout:0/528: write d7/d1c/d15/d17/d26/fa7 [312973,68483] 0 2024-04-20T21:32:03.110 INFO:tasks.workunit.client.1.smithi189.stdout:2/151: stat d1/f4 0 2024-04-20T21:32:03.113 INFO:tasks.workunit.client.0.smithi062.stdout:0/529: chown d7/d1e/c2b 1039 1 2024-04-20T21:32:03.122 INFO:tasks.workunit.client.0.smithi062.stdout:2/560: dread d7/da/d28/f6d [0,4194304] 0 2024-04-20T21:32:03.123 INFO:tasks.workunit.client.1.smithi189.stdout:2/152: write d1/f6 [2682932,120302] 0 2024-04-20T21:32:03.124 INFO:tasks.workunit.client.1.smithi189.stdout:9/200: truncate f7 270547 0 2024-04-20T21:32:03.126 INFO:tasks.workunit.client.0.smithi062.stdout:2/561: dread - d7/da/d15/d1b/d3d/f78 zero size 2024-04-20T21:32:03.137 INFO:tasks.workunit.client.0.smithi062.stdout:5/534: creat d8/d1e/d2c/d6c/d56/d66/daa/fab x:0 0 0 2024-04-20T21:32:03.140 INFO:tasks.workunit.client.1.smithi189.stdout:9/201: rename d9/l1e to d9/dc/d17/d36/l40 0 2024-04-20T21:32:03.145 INFO:tasks.workunit.client.1.smithi189.stdout:9/202: symlink d9/dc/d17/l41 0 2024-04-20T21:32:03.148 INFO:tasks.workunit.client.1.smithi189.stdout:7/178: dwrite d5/f6 [0,4194304] 0 2024-04-20T21:32:03.150 INFO:tasks.workunit.client.0.smithi062.stdout:0/530: rename d7/d1c/d15/d17/c7f to d7/d1c/d15/d17/d37/d95/cb4 0 2024-04-20T21:32:03.153 INFO:tasks.workunit.client.0.smithi062.stdout:5/535: dread d8/f44 [0,4194304] 0 2024-04-20T21:32:03.160 INFO:tasks.workunit.client.0.smithi062.stdout:1/562: sync 2024-04-20T21:32:03.162 INFO:tasks.workunit.client.0.smithi062.stdout:5/536: write d8/d1e/d2c/d6c/d56/d50/f9c [3382022,101167] 0 2024-04-20T21:32:03.163 INFO:tasks.workunit.client.1.smithi189.stdout:2/153: dread d1/f12 [0,4194304] 0 2024-04-20T21:32:03.163 INFO:tasks.workunit.client.1.smithi189.stdout:4/159: truncate d2/dd/f2d 2807027 0 2024-04-20T21:32:03.163 INFO:tasks.workunit.client.0.smithi062.stdout:0/531: truncate d7/d1c/f41 1505524 0 2024-04-20T21:32:03.165 INFO:tasks.workunit.client.0.smithi062.stdout:1/563: readlink dc/d15/d75/d94/d36/d78/l95 0 2024-04-20T21:32:03.165 INFO:tasks.workunit.client.1.smithi189.stdout:8/285: dread da/d1f/f2f [0,4194304] 0 2024-04-20T21:32:03.168 INFO:tasks.workunit.client.1.smithi189.stdout:4/160: readlink d2/dd/d11/l2a 0 2024-04-20T21:32:03.171 INFO:tasks.workunit.client.1.smithi189.stdout:9/203: truncate f4 1292299 0 2024-04-20T21:32:03.174 INFO:tasks.workunit.client.1.smithi189.stdout:8/286: dread - da/df/d3c/d50/f66 zero size 2024-04-20T21:32:03.174 INFO:tasks.workunit.client.1.smithi189.stdout:1/174: dwrite d8/f1a [4194304,4194304] 0 2024-04-20T21:32:03.174 INFO:tasks.workunit.client.1.smithi189.stdout:2/154: mknod d1/d11/d1e/c22 0 2024-04-20T21:32:03.175 INFO:tasks.workunit.client.1.smithi189.stdout:4/161: symlink d2/d36/l40 0 2024-04-20T21:32:03.177 INFO:tasks.workunit.client.0.smithi062.stdout:9/533: dwrite d16/d3d/d55/faf [0,4194304] 0 2024-04-20T21:32:03.178 INFO:tasks.workunit.client.1.smithi189.stdout:6/172: dwrite d8/f21 [0,4194304] 0 2024-04-20T21:32:03.179 INFO:tasks.workunit.client.0.smithi062.stdout:2/562: dwrite d7/da/f4d [0,4194304] 0 2024-04-20T21:32:03.181 INFO:tasks.workunit.client.1.smithi189.stdout:8/287: read da/d1f/f2f [1631630,74904] 0 2024-04-20T21:32:03.184 INFO:tasks.workunit.client.0.smithi062.stdout:5/537: rename d8/d1e/d2c/d6c/d56 to d8/d38/d7a/d4c/d7d/d9d/dac 0 2024-04-20T21:32:03.184 INFO:tasks.workunit.client.1.smithi189.stdout:8/288: chown da/d1f/f25 5 1 2024-04-20T21:32:03.185 INFO:tasks.workunit.client.1.smithi189.stdout:2/155: unlink d1/f16 0 2024-04-20T21:32:03.186 INFO:tasks.workunit.client.0.smithi062.stdout:1/564: creat dc/d15/fc4 x:0 0 0 2024-04-20T21:32:03.187 INFO:tasks.workunit.client.1.smithi189.stdout:4/162: unlink f0 0 2024-04-20T21:32:03.188 INFO:tasks.workunit.client.1.smithi189.stdout:1/175: unlink d8/df/c31 0 2024-04-20T21:32:03.190 INFO:tasks.workunit.client.1.smithi189.stdout:3/138: sync 2024-04-20T21:32:03.190 INFO:tasks.workunit.client.1.smithi189.stdout:5/137: sync 2024-04-20T21:32:03.194 INFO:tasks.workunit.client.0.smithi062.stdout:9/534: fdatasync d16/f1e 0 2024-04-20T21:32:03.194 INFO:tasks.workunit.client.0.smithi062.stdout:0/532: dwrite d7/d1c/d15/d17/d1b/d74/f99 [0,4194304] 0 2024-04-20T21:32:03.197 INFO:tasks.workunit.client.1.smithi189.stdout:2/156: chown d1/d11/f15 229304700 1 2024-04-20T21:32:03.200 INFO:tasks.workunit.client.1.smithi189.stdout:0/282: dread d3/df/d28/d2b/f36 [0,4194304] 0 2024-04-20T21:32:03.203 INFO:tasks.workunit.client.0.smithi062.stdout:5/538: unlink d8/d1e/f25 0 2024-04-20T21:32:03.203 INFO:tasks.workunit.client.1.smithi189.stdout:0/283: stat d3/df/d30/d37 0 2024-04-20T21:32:03.203 INFO:tasks.workunit.client.1.smithi189.stdout:1/176: creat d8/f41 x:0 0 0 2024-04-20T21:32:03.213 INFO:tasks.workunit.client.1.smithi189.stdout:0/284: stat d3/df/f3a 0 2024-04-20T21:32:03.213 INFO:tasks.workunit.client.1.smithi189.stdout:4/163: fsync d2/dd/f26 0 2024-04-20T21:32:03.216 INFO:tasks.workunit.client.0.smithi062.stdout:1/565: dread dc/d27/d32/f51 [0,4194304] 0 2024-04-20T21:32:03.224 INFO:tasks.workunit.client.1.smithi189.stdout:5/138: fsync d7/fb 0 2024-04-20T21:32:03.227 INFO:tasks.workunit.client.1.smithi189.stdout:8/289: dwrite da/df/d22/d33/f5a [0,4194304] 0 2024-04-20T21:32:03.227 INFO:tasks.workunit.client.1.smithi189.stdout:9/204: link d9/dc/d17/d36/l39 d9/dc/d17/d2b/l42 0 2024-04-20T21:32:03.228 INFO:tasks.workunit.client.1.smithi189.stdout:0/285: write d3/f1f [3051382,11570] 0 2024-04-20T21:32:03.230 INFO:tasks.workunit.client.1.smithi189.stdout:2/157: creat d1/f23 x:0 0 0 2024-04-20T21:32:03.231 INFO:tasks.workunit.client.1.smithi189.stdout:4/164: creat d2/dd/d11/d14/f41 x:0 0 0 2024-04-20T21:32:03.231 INFO:tasks.workunit.client.0.smithi062.stdout:1/566: fdatasync dc/d15/d20/d28/f49 0 2024-04-20T21:32:03.232 INFO:tasks.workunit.client.1.smithi189.stdout:5/139: mkdir d7/d14/d1b/d1f/d2a 0 2024-04-20T21:32:03.233 INFO:tasks.workunit.client.1.smithi189.stdout:6/173: dread d8/f21 [0,4194304] 0 2024-04-20T21:32:03.235 INFO:tasks.workunit.client.1.smithi189.stdout:2/158: read d1/d11/f1a [1571063,93332] 0 2024-04-20T21:32:03.235 INFO:tasks.workunit.client.1.smithi189.stdout:9/205: readlink d9/dc/d17/d2b/l42 0 2024-04-20T21:32:03.235 INFO:tasks.workunit.client.1.smithi189.stdout:6/174: stat d8/d10/c34 0 2024-04-20T21:32:03.236 INFO:tasks.workunit.client.0.smithi062.stdout:1/567: creat dc/d2d/fc5 x:0 0 0 2024-04-20T21:32:03.239 INFO:tasks.workunit.client.1.smithi189.stdout:8/290: creat da/de/d1e/d21/f67 x:0 0 0 2024-04-20T21:32:03.241 INFO:tasks.workunit.client.0.smithi062.stdout:1/568: truncate dc/d2d/d7c/f8d 291010 0 2024-04-20T21:32:03.244 INFO:tasks.workunit.client.0.smithi062.stdout:1/569: write dc/d15/d75/f91 [4079989,105232] 0 2024-04-20T21:32:03.250 INFO:tasks.workunit.client.1.smithi189.stdout:8/291: write da/df/d13/f18 [2445358,19810] 0 2024-04-20T21:32:03.250 INFO:tasks.workunit.client.1.smithi189.stdout:2/159: read d1/f9 [3050443,31821] 0 2024-04-20T21:32:03.251 INFO:tasks.workunit.client.0.smithi062.stdout:5/539: fsync d8/d1e/d2c/d6c/f3c 0 2024-04-20T21:32:03.253 INFO:tasks.workunit.client.0.smithi062.stdout:1/570: mknod dc/d27/cc6 0 2024-04-20T21:32:03.253 INFO:tasks.workunit.client.1.smithi189.stdout:9/206: creat d9/dc/d17/d36/f43 x:0 0 0 2024-04-20T21:32:03.254 INFO:tasks.workunit.client.0.smithi062.stdout:5/540: write d8/d1e/f93 [526434,22754] 0 2024-04-20T21:32:03.255 INFO:tasks.workunit.client.1.smithi189.stdout:3/139: dwrite d2/f2a [0,4194304] 0 2024-04-20T21:32:03.255 INFO:tasks.workunit.client.1.smithi189.stdout:7/179: dread d5/f1e [0,4194304] 0 2024-04-20T21:32:03.255 INFO:tasks.workunit.client.1.smithi189.stdout:1/177: write d8/df/f15 [5160285,52944] 0 2024-04-20T21:32:03.257 INFO:tasks.workunit.client.1.smithi189.stdout:7/180: chown d5/f7 1359 1 2024-04-20T21:32:03.258 INFO:tasks.workunit.client.0.smithi062.stdout:9/535: dread d16/f1c [0,4194304] 0 2024-04-20T21:32:03.261 INFO:tasks.workunit.client.0.smithi062.stdout:1/571: creat dc/d15/d75/d96/fc7 x:0 0 0 2024-04-20T21:32:03.263 INFO:tasks.workunit.client.1.smithi189.stdout:8/292: rmdir da/df/d22 39 2024-04-20T21:32:03.265 INFO:tasks.workunit.client.0.smithi062.stdout:5/541: mknod d8/d38/d7a/d4c/cad 0 2024-04-20T21:32:03.268 INFO:tasks.workunit.client.1.smithi189.stdout:2/160: symlink d1/d17/l24 0 2024-04-20T21:32:03.271 INFO:tasks.workunit.client.0.smithi062.stdout:1/572: unlink dc/d15/d20/f3e 0 2024-04-20T21:32:03.271 INFO:tasks.workunit.client.1.smithi189.stdout:6/175: dwrite d8/d12/f24 [0,4194304] 0 2024-04-20T21:32:03.272 INFO:tasks.workunit.client.1.smithi189.stdout:3/140: mknod d2/d1c/c2f 0 2024-04-20T21:32:03.273 INFO:tasks.workunit.client.1.smithi189.stdout:8/293: mkdir da/df/d3c/d68 0 2024-04-20T21:32:03.273 INFO:tasks.workunit.client.0.smithi062.stdout:2/563: dwrite d7/d39/f5f [0,4194304] 0 2024-04-20T21:32:03.274 INFO:tasks.workunit.client.1.smithi189.stdout:3/141: fsync d2/f29 0 2024-04-20T21:32:03.274 INFO:tasks.workunit.client.1.smithi189.stdout:1/178: unlink d8/d1e/d28/d2b/c39 0 2024-04-20T21:32:03.278 INFO:tasks.workunit.client.0.smithi062.stdout:1/573: chown dc/d15/d75/d94/c9b 4515115 1 2024-04-20T21:32:03.281 INFO:tasks.workunit.client.1.smithi189.stdout:0/286: rename d3/df/d28/d2b/f55 to d3/f5e 0 2024-04-20T21:32:03.282 INFO:tasks.workunit.client.1.smithi189.stdout:9/207: creat d9/f44 x:0 0 0 2024-04-20T21:32:03.286 INFO:tasks.workunit.client.1.smithi189.stdout:6/176: rmdir d8/da 39 2024-04-20T21:32:03.288 INFO:tasks.workunit.client.1.smithi189.stdout:5/140: truncate d7/d9/fc 2650804 0 2024-04-20T21:32:03.289 INFO:tasks.workunit.client.0.smithi062.stdout:9/536: dwrite d16/d3d/d55/d6e/f92 [0,4194304] 0 2024-04-20T21:32:03.291 INFO:tasks.workunit.client.0.smithi062.stdout:1/574: creat dc/d15/d75/d94/d36/d78/fc8 x:0 0 0 2024-04-20T21:32:03.292 INFO:tasks.workunit.client.1.smithi189.stdout:5/141: read d7/d19/f1c [3880318,5403] 0 2024-04-20T21:32:03.295 INFO:tasks.workunit.client.0.smithi062.stdout:9/537: chown d16/d4e/d80/lae 85089 1 2024-04-20T21:32:03.296 INFO:tasks.workunit.client.1.smithi189.stdout:1/179: mknod d8/df/d3a/c42 0 2024-04-20T21:32:03.296 INFO:tasks.workunit.client.1.smithi189.stdout:2/161: dwrite d1/f4 [0,4194304] 0 2024-04-20T21:32:03.297 INFO:tasks.workunit.client.1.smithi189.stdout:7/181: dwrite d5/f14 [4194304,4194304] 0 2024-04-20T21:32:03.304 INFO:tasks.workunit.client.0.smithi062.stdout:9/538: write d16/d4e/fa3 [786241,104667] 0 2024-04-20T21:32:03.304 INFO:tasks.workunit.client.0.smithi062.stdout:2/564: write d7/f5d [4943961,77706] 0 2024-04-20T21:32:03.309 INFO:tasks.workunit.client.0.smithi062.stdout:2/565: chown d7/da/d28/d4b/d8f 2116046634 1 2024-04-20T21:32:03.311 INFO:tasks.workunit.client.0.smithi062.stdout:2/566: fsync d7/d39/f8e 0 2024-04-20T21:32:03.316 INFO:tasks.workunit.client.1.smithi189.stdout:4/165: dread d2/dd/fe [0,4194304] 0 2024-04-20T21:32:03.316 INFO:tasks.workunit.client.0.smithi062.stdout:2/567: chown d7/da/d28/f79 0 1 2024-04-20T21:32:03.318 INFO:tasks.workunit.client.1.smithi189.stdout:8/294: dwrite da/df/d13/f14 [0,4194304] 0 2024-04-20T21:32:03.318 INFO:tasks.workunit.client.1.smithi189.stdout:1/180: rmdir d8/d21/d2c 39 2024-04-20T21:32:03.319 INFO:tasks.workunit.client.0.smithi062.stdout:9/539: unlink d16/d2c/d42/f50 0 2024-04-20T21:32:03.320 INFO:tasks.workunit.client.0.smithi062.stdout:5/542: dwrite d8/d38/d7a/d4c/d7d/d9d/dac/f73 [0,4194304] 0 2024-04-20T21:32:03.331 INFO:tasks.workunit.client.0.smithi062.stdout:9/540: creat d16/d2c/d5e/d96/fb3 x:0 0 0 2024-04-20T21:32:03.336 INFO:tasks.workunit.client.0.smithi062.stdout:5/543: getdents d8/d10/d3e/d8a 0 2024-04-20T21:32:03.340 INFO:tasks.workunit.client.0.smithi062.stdout:9/541: getdents d16/d2f/d5b/da4 0 2024-04-20T21:32:03.342 INFO:tasks.workunit.client.1.smithi189.stdout:6/177: rename d8/d12/d16 to d8/da/dd/d35 0 2024-04-20T21:32:03.345 INFO:tasks.workunit.client.1.smithi189.stdout:6/178: chown d8/d10/c34 317488 1 2024-04-20T21:32:03.346 INFO:tasks.workunit.client.0.smithi062.stdout:2/568: dwrite d7/da/f4d [0,4194304] 0 2024-04-20T21:32:03.347 INFO:tasks.workunit.client.0.smithi062.stdout:5/544: write d8/d38/d7a/d4c/d7d/d9d/dac/d50/f9c [1735810,29485] 0 2024-04-20T21:32:03.348 INFO:tasks.workunit.client.0.smithi062.stdout:9/542: chown d16/d3d/d55/d6e/l9f 3 1 2024-04-20T21:32:03.348 INFO:tasks.workunit.client.1.smithi189.stdout:3/142: getdents d2/d8 0 2024-04-20T21:32:03.351 INFO:tasks.workunit.client.1.smithi189.stdout:5/142: creat d7/f2b x:0 0 0 2024-04-20T21:32:03.353 INFO:tasks.workunit.client.1.smithi189.stdout:5/143: chown d7/d19/c1d 429103354 1 2024-04-20T21:32:03.355 INFO:tasks.workunit.client.1.smithi189.stdout:2/162: getdents d1 0 2024-04-20T21:32:03.355 INFO:tasks.workunit.client.1.smithi189.stdout:4/166: link d2/dd/d11/l25 d2/d22/l42 0 2024-04-20T21:32:03.356 INFO:tasks.workunit.client.0.smithi062.stdout:6/415: sync 2024-04-20T21:32:03.357 INFO:tasks.workunit.client.0.smithi062.stdout:7/556: sync 2024-04-20T21:32:03.357 INFO:tasks.workunit.client.0.smithi062.stdout:8/549: sync 2024-04-20T21:32:03.357 INFO:tasks.workunit.client.0.smithi062.stdout:3/516: sync 2024-04-20T21:32:03.357 INFO:tasks.workunit.client.0.smithi062.stdout:4/570: sync 2024-04-20T21:32:03.359 INFO:tasks.workunit.client.1.smithi189.stdout:3/143: mkdir d2/d3/d2c/d30 0 2024-04-20T21:32:03.359 INFO:tasks.workunit.client.1.smithi189.stdout:4/167: write d2/d20/f33 [3131401,92186] 0 2024-04-20T21:32:03.360 INFO:tasks.workunit.client.0.smithi062.stdout:9/543: write fe [2001103,31111] 0 2024-04-20T21:32:03.361 INFO:tasks.workunit.client.0.smithi062.stdout:2/569: mkdir d7/da/d28/db9 0 2024-04-20T21:32:03.364 INFO:tasks.workunit.client.1.smithi189.stdout:6/179: dread f2 [0,4194304] 0 2024-04-20T21:32:03.364 INFO:tasks.workunit.client.0.smithi062.stdout:7/557: stat d7/d3e/d33/d3d/d5e/db8/cbd 0 2024-04-20T21:32:03.368 INFO:tasks.workunit.client.1.smithi189.stdout:8/295: dread da/df/d22/d33/f5a [0,4194304] 0 2024-04-20T21:32:03.369 INFO:tasks.workunit.client.1.smithi189.stdout:9/208: truncate d9/dc/d17/f2f 3673756 0 2024-04-20T21:32:03.373 INFO:tasks.workunit.client.0.smithi062.stdout:2/570: read d7/da/d15/f85 [3975959,28361] 0 2024-04-20T21:32:03.373 INFO:tasks.workunit.client.1.smithi189.stdout:7/182: dwrite d5/f1e [4194304,4194304] 0 2024-04-20T21:32:03.374 INFO:tasks.workunit.client.1.smithi189.stdout:1/181: dread - d8/df/f13 zero size 2024-04-20T21:32:03.378 INFO:tasks.workunit.client.1.smithi189.stdout:6/180: unlink d8/d12/c1d 0 2024-04-20T21:32:03.378 INFO:tasks.workunit.client.1.smithi189.stdout:8/296: mkdir da/de/d1e/d69 0 2024-04-20T21:32:03.379 INFO:tasks.workunit.client.0.smithi062.stdout:7/558: symlink d7/d3e/d33/d3d/lbf 0 2024-04-20T21:32:03.379 INFO:tasks.workunit.client.0.smithi062.stdout:1/575: dread dc/d15/d20/f3c [0,4194304] 0 2024-04-20T21:32:03.383 INFO:tasks.workunit.client.0.smithi062.stdout:8/550: rename d7/d9/f51 to d7/fbb 0 2024-04-20T21:32:03.385 INFO:tasks.workunit.client.0.smithi062.stdout:5/545: dwrite d8/d1e/d2c/f79 [0,4194304] 0 2024-04-20T21:32:03.386 INFO:tasks.workunit.client.1.smithi189.stdout:8/297: fdatasync da/df/d20/d27/f54 0 2024-04-20T21:32:03.386 INFO:tasks.workunit.client.1.smithi189.stdout:9/209: symlink d9/d12/d15/d1a/l45 0 2024-04-20T21:32:03.386 INFO:tasks.workunit.client.1.smithi189.stdout:7/183: mkdir d5/d1d/d42 0 2024-04-20T21:32:03.387 INFO:tasks.workunit.client.1.smithi189.stdout:6/181: creat d8/d10/f36 x:0 0 0 2024-04-20T21:32:03.391 INFO:tasks.workunit.client.0.smithi062.stdout:6/416: rename d0/d2c/f2e to d0/d9/d11/d7b/f84 0 2024-04-20T21:32:03.394 INFO:tasks.workunit.client.1.smithi189.stdout:2/163: dwrite d1/f7 [0,4194304] 0 2024-04-20T21:32:03.396 INFO:tasks.workunit.client.1.smithi189.stdout:5/144: fsync d7/f2b 0 2024-04-20T21:32:03.398 INFO:tasks.workunit.client.0.smithi062.stdout:9/544: dread d16/d3d/d55/d6e/d9b/fb1 [0,4194304] 0 2024-04-20T21:32:03.399 INFO:tasks.workunit.client.1.smithi189.stdout:6/182: creat d8/da/f37 x:0 0 0 2024-04-20T21:32:03.403 INFO:tasks.workunit.client.1.smithi189.stdout:7/184: creat d5/f43 x:0 0 0 2024-04-20T21:32:03.405 INFO:tasks.workunit.client.0.smithi062.stdout:9/545: dread d16/d2c/d5e/f7b [0,4194304] 0 2024-04-20T21:32:03.407 INFO:tasks.workunit.client.0.smithi062.stdout:3/517: truncate d0/d1b/f75 2496148 0 2024-04-20T21:32:03.411 INFO:tasks.workunit.client.1.smithi189.stdout:7/185: chown d5/f14 149 1 2024-04-20T21:32:03.413 INFO:tasks.workunit.client.1.smithi189.stdout:1/182: dread d8/f9 [0,4194304] 0 2024-04-20T21:32:03.414 INFO:tasks.workunit.client.1.smithi189.stdout:3/144: dwrite d2/d1c/f28 [8388608,4194304] 0 2024-04-20T21:32:03.415 INFO:tasks.workunit.client.0.smithi062.stdout:6/417: creat d0/d1a/f85 x:0 0 0 2024-04-20T21:32:03.420 INFO:tasks.workunit.client.1.smithi189.stdout:9/210: rename d9/dc/fe to d9/d12/d3b/f46 0 2024-04-20T21:32:03.420 INFO:tasks.workunit.client.0.smithi062.stdout:2/571: creat d7/da/d15/d1b/fba x:0 0 0 2024-04-20T21:32:03.422 INFO:tasks.workunit.client.0.smithi062.stdout:7/559: creat d7/d3e/d33/d3d/d4d/fc0 x:0 0 0 2024-04-20T21:32:03.423 INFO:tasks.workunit.client.1.smithi189.stdout:1/183: dread - d8/df/f13 zero size 2024-04-20T21:32:03.424 INFO:tasks.workunit.client.1.smithi189.stdout:3/145: read d2/d1c/f28 [2283421,42166] 0 2024-04-20T21:32:03.426 INFO:tasks.workunit.client.1.smithi189.stdout:2/164: mknod d1/c25 0 2024-04-20T21:32:03.427 INFO:tasks.workunit.client.1.smithi189.stdout:5/145: symlink d7/l2c 0 2024-04-20T21:32:03.428 INFO:tasks.workunit.client.1.smithi189.stdout:7/186: rmdir d5/d1d/d28 39 2024-04-20T21:32:03.431 INFO:tasks.workunit.client.0.smithi062.stdout:3/518: readlink d0/d1/d3c/l83 0 2024-04-20T21:32:03.431 INFO:tasks.workunit.client.1.smithi189.stdout:4/168: dwrite d2/d3a/f3b [4194304,4194304] 0 2024-04-20T21:32:03.431 INFO:tasks.workunit.client.1.smithi189.stdout:1/184: write f6 [727429,45711] 0 2024-04-20T21:32:03.432 INFO:tasks.workunit.client.0.smithi062.stdout:9/546: chown d16/d3d/d55/c6c 46 1 2024-04-20T21:32:03.434 INFO:tasks.workunit.client.1.smithi189.stdout:0/287: dread d3/d26/f1d [0,4194304] 0 2024-04-20T21:32:03.434 INFO:tasks.workunit.client.1.smithi189.stdout:5/146: truncate d7/d14/f26 915162 0 2024-04-20T21:32:03.434 INFO:tasks.workunit.client.0.smithi062.stdout:4/571: rename da/dd/d15/d50/d64/fca to da/dd/d15/d50/fcb 0 2024-04-20T21:32:03.438 INFO:tasks.workunit.client.1.smithi189.stdout:8/298: dread da/d1f/f2f [0,4194304] 0 2024-04-20T21:32:03.438 INFO:tasks.workunit.client.1.smithi189.stdout:7/187: read - d5/d21/d25/f27 zero size 2024-04-20T21:32:03.440 INFO:tasks.workunit.client.1.smithi189.stdout:3/146: mkdir d2/d3/d2c/d31 0 2024-04-20T21:32:03.440 INFO:tasks.workunit.client.0.smithi062.stdout:6/418: creat d0/d9/d11/d5c/f86 x:0 0 0 2024-04-20T21:32:03.444 INFO:tasks.workunit.client.0.smithi062.stdout:2/572: dwrite d7/da/d28/f79 [0,4194304] 0 2024-04-20T21:32:03.445 INFO:tasks.workunit.client.1.smithi189.stdout:5/147: fsync d7/fb 0 2024-04-20T21:32:03.447 INFO:tasks.workunit.client.1.smithi189.stdout:4/169: write d2/f38 [841937,13616] 0 2024-04-20T21:32:03.450 INFO:tasks.workunit.client.0.smithi062.stdout:3/519: symlink d0/d1/d3c/d43/laf 0 2024-04-20T21:32:03.453 INFO:tasks.workunit.client.1.smithi189.stdout:6/183: dwrite d8/da/f37 [0,4194304] 0 2024-04-20T21:32:03.461 INFO:tasks.workunit.client.0.smithi062.stdout:1/576: rename dc/d2d to dc/d15/d20/d28/dc9 0 2024-04-20T21:32:03.461 INFO:tasks.workunit.client.1.smithi189.stdout:8/299: mknod da/df/d3a/c6a 0 2024-04-20T21:32:03.466 INFO:tasks.workunit.client.1.smithi189.stdout:4/170: rmdir d2/d3a 39 2024-04-20T21:32:03.466 INFO:tasks.workunit.client.1.smithi189.stdout:9/211: dwrite d9/f33 [0,4194304] 0 2024-04-20T21:32:03.467 INFO:tasks.workunit.client.1.smithi189.stdout:5/148: write d7/d19/f1c [2058933,27889] 0 2024-04-20T21:32:03.469 INFO:tasks.workunit.client.1.smithi189.stdout:3/147: rename d2/c20 to d2/d3/d2c/d30/c32 0 2024-04-20T21:32:03.471 INFO:tasks.workunit.client.1.smithi189.stdout:4/171: chown d2/dd/d11/l2a 3 1 2024-04-20T21:32:03.471 INFO:tasks.workunit.client.0.smithi062.stdout:6/419: write d0/f5e [2404141,25994] 0 2024-04-20T21:32:03.473 INFO:tasks.workunit.client.0.smithi062.stdout:5/546: dwrite d8/d38/d7a/d4c/d7d/d9d/dac/d66/fa8 [0,4194304] 0 2024-04-20T21:32:03.476 INFO:tasks.workunit.client.0.smithi062.stdout:7/560: rename d7/d3e/lae to d7/d3e/d33/d3d/d5e/d7e/lc1 0 2024-04-20T21:32:03.479 INFO:tasks.workunit.client.0.smithi062.stdout:9/547: link d16/d2c/d42/l4a d16/d2c/lb4 0 2024-04-20T21:32:03.481 INFO:tasks.workunit.client.0.smithi062.stdout:1/577: symlink dc/d15/d75/d94/d36/d78/dab/lca 0 2024-04-20T21:32:03.482 INFO:tasks.workunit.client.0.smithi062.stdout:6/420: dread d0/d9/d11/f65 [0,4194304] 0 2024-04-20T21:32:03.482 INFO:tasks.workunit.client.0.smithi062.stdout:7/561: write d7/d26/f2f [1656685,94690] 0 2024-04-20T21:32:03.484 INFO:tasks.workunit.client.1.smithi189.stdout:0/288: dwrite d3/d26/f1d [4194304,4194304] 0 2024-04-20T21:32:03.486 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:03 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:32:03.486 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:03 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:32:03.486 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:03 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:32:03.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:03 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T21:32:03.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:03 smithi189 ceph-mon[21562]: Updating smithi062:/etc/ceph/ceph.conf 2024-04-20T21:32:03.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:03 smithi189 ceph-mon[21562]: Updating smithi189:/etc/ceph/ceph.conf 2024-04-20T21:32:03.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:03 smithi189 ceph-mon[21562]: pgmap v16: 65 pgs: 65 active+clean; 1.2 GiB data, 3.6 GiB used, 533 GiB / 536 GiB avail; 35 MiB/s rd, 156 MiB/s wr, 308 op/s 2024-04-20T21:32:03.495 INFO:tasks.workunit.client.0.smithi062.stdout:3/520: getdents d0/d1/d10/d3a 0 2024-04-20T21:32:03.495 INFO:tasks.workunit.client.0.smithi062.stdout:0/533: sync 2024-04-20T21:32:03.499 INFO:tasks.workunit.client.1.smithi189.stdout:9/212: mkdir d9/d12/d15/d1a/d47 0 2024-04-20T21:32:03.503 INFO:tasks.workunit.client.1.smithi189.stdout:6/184: dwrite d8/f21 [0,4194304] 0 2024-04-20T21:32:03.503 INFO:tasks.workunit.client.1.smithi189.stdout:7/188: creat d5/d1d/f44 x:0 0 0 2024-04-20T21:32:03.503 INFO:tasks.workunit.client.1.smithi189.stdout:1/185: getdents d8/d1e/d28/d2b 0 2024-04-20T21:32:03.505 INFO:tasks.workunit.client.1.smithi189.stdout:0/289: write d3/f5e [468016,114915] 0 2024-04-20T21:32:03.508 INFO:tasks.workunit.client.1.smithi189.stdout:3/148: mkdir d2/d33 0 2024-04-20T21:32:03.511 INFO:tasks.workunit.client.1.smithi189.stdout:2/165: write d1/d11/f15 [428708,49019] 0 2024-04-20T21:32:03.513 INFO:tasks.workunit.client.1.smithi189.stdout:8/300: dwrite da/d1f/f2f [0,4194304] 0 2024-04-20T21:32:03.513 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:03 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:32:03.513 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:03 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:32:03.513 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:03 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:32:03.513 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:03 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T21:32:03.513 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:03 smithi062 ceph-mon[16221]: Updating smithi062:/etc/ceph/ceph.conf 2024-04-20T21:32:03.513 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:03 smithi062 ceph-mon[16221]: Updating smithi189:/etc/ceph/ceph.conf 2024-04-20T21:32:03.513 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:03 smithi062 ceph-mon[16221]: pgmap v16: 65 pgs: 65 active+clean; 1.2 GiB data, 3.6 GiB used, 533 GiB / 536 GiB avail; 35 MiB/s rd, 156 MiB/s wr, 308 op/s 2024-04-20T21:32:03.514 INFO:tasks.workunit.client.0.smithi062.stdout:1/578: creat dc/d15/d20/d28/d83/dc0/fcb x:0 0 0 2024-04-20T21:32:03.514 INFO:tasks.workunit.client.0.smithi062.stdout:4/572: rename da/dd/d15/d50/d64/d3d/da1/fbc to da/dd/fcc 0 2024-04-20T21:32:03.521 INFO:tasks.workunit.client.1.smithi189.stdout:4/172: creat d2/d3a/f43 x:0 0 0 2024-04-20T21:32:03.521 INFO:tasks.workunit.client.0.smithi062.stdout:2/573: dread d7/da/d28/d4b/f73 [0,4194304] 0 2024-04-20T21:32:03.521 INFO:tasks.workunit.client.1.smithi189.stdout:9/213: dread f8 [0,4194304] 0 2024-04-20T21:32:03.522 INFO:tasks.workunit.client.0.smithi062.stdout:4/573: dread - da/dd/df/d27/d7a/fbb zero size 2024-04-20T21:32:03.524 INFO:tasks.workunit.client.0.smithi062.stdout:3/521: dread d0/d1c/f51 [0,4194304] 0 2024-04-20T21:32:03.526 INFO:tasks.workunit.client.1.smithi189.stdout:1/186: mkdir d8/d1e/d43 0 2024-04-20T21:32:03.526 INFO:tasks.workunit.client.1.smithi189.stdout:4/173: read - d2/dd/d37/f39 zero size 2024-04-20T21:32:03.529 INFO:tasks.workunit.client.0.smithi062.stdout:7/562: write d7/d3e/d33/f35 [2917011,34740] 0 2024-04-20T21:32:03.530 INFO:tasks.workunit.client.1.smithi189.stdout:3/149: creat d2/d8/f34 x:0 0 0 2024-04-20T21:32:03.532 INFO:tasks.workunit.client.1.smithi189.stdout:0/290: write d3/f27 [693817,8742] 0 2024-04-20T21:32:03.532 INFO:tasks.workunit.client.1.smithi189.stdout:1/187: write d8/f41 [444989,49994] 0 2024-04-20T21:32:03.534 INFO:tasks.workunit.client.1.smithi189.stdout:2/166: dread d1/fd [0,4194304] 0 2024-04-20T21:32:03.534 INFO:tasks.workunit.client.0.smithi062.stdout:5/547: rename d8/d38/d7a/d4c/d7d/d9d/dac/d50 to d8/d38/d7a/d4c/d7d/d9d/dac/d66/dae 0 2024-04-20T21:32:03.536 INFO:tasks.workunit.client.0.smithi062.stdout:1/579: dread dc/d27/d32/f74 [0,4194304] 0 2024-04-20T21:32:03.537 INFO:tasks.workunit.client.1.smithi189.stdout:3/150: chown d2/d8/f22 13 1 2024-04-20T21:32:03.542 INFO:tasks.workunit.client.0.smithi062.stdout:3/522: symlink d0/d1/d10/d3a/lb0 0 2024-04-20T21:32:03.542 INFO:tasks.workunit.client.1.smithi189.stdout:6/185: dwrite d8/d10/f20 [0,4194304] 0 2024-04-20T21:32:03.545 INFO:tasks.workunit.client.1.smithi189.stdout:6/186: write f5 [5743392,36768] 0 2024-04-20T21:32:03.546 INFO:tasks.workunit.client.1.smithi189.stdout:9/214: rename d9/dc/d17/d36/l40 to d9/d12/l48 0 2024-04-20T21:32:03.547 INFO:tasks.workunit.client.1.smithi189.stdout:7/189: dread d5/d21/f32 [0,4194304] 0 2024-04-20T21:32:03.547 INFO:tasks.workunit.client.0.smithi062.stdout:3/523: dread - d0/d1c/d52/d7b/f85 zero size 2024-04-20T21:32:03.547 INFO:tasks.workunit.client.0.smithi062.stdout:5/548: truncate d8/d38/d7a/fa5 153384 0 2024-04-20T21:32:03.547 INFO:tasks.workunit.client.0.smithi062.stdout:9/548: dwrite fd [0,4194304] 0 2024-04-20T21:32:03.551 INFO:tasks.workunit.client.0.smithi062.stdout:4/574: getdents da/dd/df/d27/d32/d97 0 2024-04-20T21:32:03.554 INFO:tasks.workunit.client.0.smithi062.stdout:9/549: chown d16/d2c/l30 914517 1 2024-04-20T21:32:03.556 INFO:tasks.workunit.client.0.smithi062.stdout:9/550: chown d16/d2c/d5e/d75 3 1 2024-04-20T21:32:03.558 INFO:tasks.workunit.client.0.smithi062.stdout:4/575: rename da/d56/f72 to da/dd/d3c/dac/fcd 0 2024-04-20T21:32:03.561 INFO:tasks.workunit.client.0.smithi062.stdout:6/421: dread d0/d9/d11/f25 [0,4194304] 0 2024-04-20T21:32:03.562 INFO:tasks.workunit.client.0.smithi062.stdout:3/524: creat d0/d1/fb1 x:0 0 0 2024-04-20T21:32:03.564 INFO:tasks.workunit.client.0.smithi062.stdout:9/551: mknod d16/d2c/d42/da6/da9/cb5 0 2024-04-20T21:32:03.566 INFO:tasks.workunit.client.0.smithi062.stdout:5/549: rename d8/d38/d7a/fa5 to d8/d38/d7a/d4c/d7d/d9d/dac/d66/dae/faf 0 2024-04-20T21:32:03.568 INFO:tasks.workunit.client.1.smithi189.stdout:4/174: getdents d2/d22 0 2024-04-20T21:32:03.569 INFO:tasks.workunit.client.0.smithi062.stdout:4/576: mkdir da/dd/d3c/dac/dce 0 2024-04-20T21:32:03.571 INFO:tasks.workunit.client.1.smithi189.stdout:3/151: rename d2/c13 to d2/c35 0 2024-04-20T21:32:03.575 INFO:tasks.workunit.client.1.smithi189.stdout:9/215: dwrite d9/dc/f3d [0,4194304] 0 2024-04-20T21:32:03.575 INFO:tasks.workunit.client.1.smithi189.stdout:1/188: read f2 [1026316,14276] 0 2024-04-20T21:32:03.582 INFO:tasks.workunit.client.0.smithi062.stdout:5/550: read d8/d10/d3e/f62 [2829573,103004] 0 2024-04-20T21:32:03.584 INFO:tasks.workunit.client.1.smithi189.stdout:2/167: dwrite d1/d11/f15 [0,4194304] 0 2024-04-20T21:32:03.584 INFO:tasks.workunit.client.0.smithi062.stdout:9/552: creat d16/d2c/d5e/d75/d76/fb6 x:0 0 0 2024-04-20T21:32:03.586 INFO:tasks.workunit.client.0.smithi062.stdout:5/551: chown d8/d38/f89 3997 1 2024-04-20T21:32:03.587 INFO:tasks.workunit.client.1.smithi189.stdout:4/175: rename d2/d22/f27 to d2/dd/f44 0 2024-04-20T21:32:03.588 INFO:tasks.workunit.client.1.smithi189.stdout:4/176: fdatasync d2/f21 0 2024-04-20T21:32:03.589 INFO:tasks.workunit.client.1.smithi189.stdout:6/187: dwrite d8/d10/f36 [0,4194304] 0 2024-04-20T21:32:03.589 INFO:tasks.workunit.client.1.smithi189.stdout:1/189: write f2 [1171892,109861] 0 2024-04-20T21:32:03.591 INFO:tasks.workunit.client.0.smithi062.stdout:9/553: creat d16/d3d/d55/fb7 x:0 0 0 2024-04-20T21:32:03.595 INFO:tasks.workunit.client.1.smithi189.stdout:4/177: fsync d2/dd/fe 0 2024-04-20T21:32:03.595 INFO:tasks.workunit.client.0.smithi062.stdout:6/422: dwrite d0/d9/d11/f48 [0,4194304] 0 2024-04-20T21:32:03.597 INFO:tasks.workunit.client.1.smithi189.stdout:2/168: write d1/fe [4582055,120866] 0 2024-04-20T21:32:03.598 INFO:tasks.workunit.client.0.smithi062.stdout:0/534: truncate d7/d1c/d15/d17/d40/f91 1013941 0 2024-04-20T21:32:03.601 INFO:tasks.workunit.client.0.smithi062.stdout:9/554: readlink d16/l52 0 2024-04-20T21:32:03.604 INFO:tasks.workunit.client.1.smithi189.stdout:7/190: write d5/d21/f23 [1808647,71955] 0 2024-04-20T21:32:03.605 INFO:tasks.workunit.client.1.smithi189.stdout:6/188: chown d8/da/dd/f28 33219722 1 2024-04-20T21:32:03.608 INFO:tasks.workunit.client.0.smithi062.stdout:7/563: write d7/d3e/d33/d3d/d5e/d72/f86 [37336,12879] 0 2024-04-20T21:32:03.608 INFO:tasks.workunit.client.0.smithi062.stdout:1/580: write dc/ff [168030,99991] 0 2024-04-20T21:32:03.608 INFO:tasks.workunit.client.0.smithi062.stdout:3/525: fdatasync d0/d1/d10/d3a/d5e/f7d 0 2024-04-20T21:32:03.612 INFO:tasks.workunit.client.1.smithi189.stdout:1/190: dread d8/f41 [0,4194304] 0 2024-04-20T21:32:03.613 INFO:tasks.workunit.client.1.smithi189.stdout:4/178: truncate d2/dd/f44 448584 0 2024-04-20T21:32:03.615 INFO:tasks.workunit.client.1.smithi189.stdout:2/169: rename d1/d11/d1e to d1/d11/d26 0 2024-04-20T21:32:03.619 INFO:tasks.workunit.client.0.smithi062.stdout:5/552: getdents d8/d10/d3e 0 2024-04-20T21:32:03.620 INFO:tasks.workunit.client.1.smithi189.stdout:3/152: dwrite d2/f6 [8388608,4194304] 0 2024-04-20T21:32:03.621 INFO:tasks.workunit.client.1.smithi189.stdout:9/216: dwrite d9/dc/d17/d36/f43 [0,4194304] 0 2024-04-20T21:32:03.622 INFO:tasks.workunit.client.0.smithi062.stdout:7/564: fdatasync d7/d3e/d33/f57 0 2024-04-20T21:32:03.624 INFO:tasks.workunit.client.1.smithi189.stdout:4/179: creat d2/d22/f45 x:0 0 0 2024-04-20T21:32:03.628 INFO:tasks.workunit.client.1.smithi189.stdout:3/153: dread - d2/d1c/f2b zero size 2024-04-20T21:32:03.636 INFO:tasks.workunit.client.0.smithi062.stdout:1/581: symlink dc/d27/d9a/da0/db3/lcc 0 2024-04-20T21:32:03.636 INFO:tasks.workunit.client.0.smithi062.stdout:5/553: symlink d8/d10/d3e/lb0 0 2024-04-20T21:32:03.637 INFO:tasks.workunit.client.1.smithi189.stdout:6/189: creat d8/d12/d30/f38 x:0 0 0 2024-04-20T21:32:03.640 INFO:tasks.workunit.client.0.smithi062.stdout:7/565: write d7/d3e/d33/d3d/d4d/d54/f2e [4593780,130820] 0 2024-04-20T21:32:03.640 INFO:tasks.workunit.client.1.smithi189.stdout:9/217: symlink d9/d12/d15/l49 0 2024-04-20T21:32:03.641 INFO:tasks.workunit.client.0.smithi062.stdout:5/554: stat d8/d1e 0 2024-04-20T21:32:03.645 INFO:tasks.workunit.client.0.smithi062.stdout:7/566: write d7/d63/fa7 [941135,110092] 0 2024-04-20T21:32:03.645 INFO:tasks.workunit.client.0.smithi062.stdout:4/577: dread da/dd/f78 [0,4194304] 0 2024-04-20T21:32:03.648 INFO:tasks.workunit.client.0.smithi062.stdout:0/535: dwrite d7/d1c/d15/d17/fad [0,4194304] 0 2024-04-20T21:32:03.648 INFO:tasks.workunit.client.1.smithi189.stdout:7/191: dwrite d5/d1d/d28/f38 [0,4194304] 0 2024-04-20T21:32:03.649 INFO:tasks.workunit.client.1.smithi189.stdout:1/191: mkdir d8/d1e/d43/d44 0 2024-04-20T21:32:03.650 INFO:tasks.workunit.client.0.smithi062.stdout:1/582: dread dc/d15/f6e [0,4194304] 0 2024-04-20T21:32:03.652 INFO:tasks.workunit.client.1.smithi189.stdout:9/218: fdatasync d9/f34 0 2024-04-20T21:32:03.652 INFO:tasks.workunit.client.0.smithi062.stdout:7/567: dread - d7/d3e/f71 zero size 2024-04-20T21:32:03.657 INFO:tasks.workunit.client.0.smithi062.stdout:7/568: fdatasync d7/d3e/d33/d3d/dad/fb9 0 2024-04-20T21:32:03.657 INFO:tasks.workunit.client.0.smithi062.stdout:5/555: chown d8/d1e/d2c/d6c/l96 485902859 1 2024-04-20T21:32:03.659 INFO:tasks.workunit.client.0.smithi062.stdout:0/536: mknod d7/d1c/d15/d17/d40/d79/d71/cb5 0 2024-04-20T21:32:03.662 INFO:tasks.workunit.client.1.smithi189.stdout:4/180: dwrite d2/f4 [0,4194304] 0 2024-04-20T21:32:03.663 INFO:tasks.workunit.client.0.smithi062.stdout:7/569: dread - d7/d3e/d33/d3d/d4d/fc0 zero size 2024-04-20T21:32:03.665 INFO:tasks.workunit.client.1.smithi189.stdout:9/219: write d9/d12/d15/f31 [1679779,48285] 0 2024-04-20T21:32:03.665 INFO:tasks.workunit.client.1.smithi189.stdout:7/192: write d5/f1f [271231,103933] 0 2024-04-20T21:32:03.670 INFO:tasks.workunit.client.1.smithi189.stdout:2/170: dwrite d1/f4 [0,4194304] 0 2024-04-20T21:32:03.671 INFO:tasks.workunit.client.0.smithi062.stdout:0/537: rename d7/d1c/d15/d17/d37/d64 to d7/d1e/db6 0 2024-04-20T21:32:03.673 INFO:tasks.workunit.client.0.smithi062.stdout:5/556: truncate d8/d1e/d2c/d6c/f9b 873406 0 2024-04-20T21:32:03.675 INFO:tasks.workunit.client.0.smithi062.stdout:8/551: sync 2024-04-20T21:32:03.676 INFO:tasks.workunit.client.1.smithi189.stdout:6/190: dwrite f7 [0,4194304] 0 2024-04-20T21:32:03.678 INFO:tasks.workunit.client.0.smithi062.stdout:7/570: creat d7/d3e/d33/d3d/d4d/d54/d4f/fc2 x:0 0 0 2024-04-20T21:32:03.680 INFO:tasks.workunit.client.1.smithi189.stdout:4/181: mknod d2/d22/c46 0 2024-04-20T21:32:03.684 INFO:tasks.workunit.client.0.smithi062.stdout:7/571: fdatasync d7/d26/f9a 0 2024-04-20T21:32:03.685 INFO:tasks.workunit.client.1.smithi189.stdout:9/220: creat d9/dc/d17/d36/f4a x:0 0 0 2024-04-20T21:32:03.686 INFO:tasks.workunit.client.1.smithi189.stdout:5/149: sync 2024-04-20T21:32:03.689 INFO:tasks.workunit.client.0.smithi062.stdout:0/538: write d7/d1c/d15/d17/f38 [160083,19226] 0 2024-04-20T21:32:03.690 INFO:tasks.workunit.client.1.smithi189.stdout:5/150: write d7/d14/f26 [1621656,21864] 0 2024-04-20T21:32:03.692 INFO:tasks.workunit.client.0.smithi062.stdout:8/552: truncate d7/d25/d3b/d58/f63 1152432 0 2024-04-20T21:32:03.693 INFO:tasks.workunit.client.1.smithi189.stdout:2/171: dread d1/fe [0,4194304] 0 2024-04-20T21:32:03.693 INFO:tasks.workunit.client.1.smithi189.stdout:1/192: rename d8/df/d3a/c3f to d8/d1e/d28/d2b/c45 0 2024-04-20T21:32:03.694 INFO:tasks.workunit.client.0.smithi062.stdout:0/539: dread - d7/d20/d62/d6f/f9f zero size 2024-04-20T21:32:03.695 INFO:tasks.workunit.client.0.smithi062.stdout:6/423: dwrite d0/d9/d11/f63 [0,4194304] 0 2024-04-20T21:32:03.695 INFO:tasks.workunit.client.0.smithi062.stdout:7/572: unlink d7/d3e/d33/d3d/dad/fb5 0 2024-04-20T21:32:03.698 INFO:tasks.workunit.client.0.smithi062.stdout:5/557: fsync d8/d38/d7a/f91 0 2024-04-20T21:32:03.700 INFO:tasks.workunit.client.0.smithi062.stdout:6/424: chown d0/d2c/d67/d75 2094264 1 2024-04-20T21:32:03.702 INFO:tasks.workunit.client.0.smithi062.stdout:2/574: sync 2024-04-20T21:32:03.702 INFO:tasks.workunit.client.0.smithi062.stdout:6/425: fsync d0/d1a/d44/f4b 0 2024-04-20T21:32:03.703 INFO:tasks.workunit.client.1.smithi189.stdout:6/191: creat d8/d2b/f39 x:0 0 0 2024-04-20T21:32:03.705 INFO:tasks.workunit.client.0.smithi062.stdout:0/540: dread - d7/d20/d62/d6f/f9b zero size 2024-04-20T21:32:03.710 INFO:tasks.workunit.client.1.smithi189.stdout:7/193: unlink d5/l19 0 2024-04-20T21:32:03.710 INFO:tasks.workunit.client.1.smithi189.stdout:7/194: chown d5/f7 52179599 1 2024-04-20T21:32:03.712 INFO:tasks.workunit.client.0.smithi062.stdout:9/555: dread d16/f2e [0,4194304] 0 2024-04-20T21:32:03.713 INFO:tasks.workunit.client.0.smithi062.stdout:6/426: truncate d0/d9/f71 1017867 0 2024-04-20T21:32:03.715 INFO:tasks.workunit.client.1.smithi189.stdout:9/221: dread d9/dc/f1d [0,4194304] 0 2024-04-20T21:32:03.716 INFO:tasks.workunit.client.0.smithi062.stdout:0/541: stat d7/d1e/c30 0 2024-04-20T21:32:03.717 INFO:tasks.workunit.client.0.smithi062.stdout:6/427: fsync d0/d1a/f85 0 2024-04-20T21:32:03.718 INFO:tasks.workunit.client.1.smithi189.stdout:4/182: symlink d2/dd/d11/l47 0 2024-04-20T21:32:03.719 INFO:tasks.workunit.client.1.smithi189.stdout:2/172: write d1/f7 [5029067,52357] 0 2024-04-20T21:32:03.720 INFO:tasks.workunit.client.0.smithi062.stdout:6/428: stat d0/d9/l12 0 2024-04-20T21:32:03.721 INFO:tasks.workunit.client.0.smithi062.stdout:5/558: mkdir d8/d38/d7a/db1 0 2024-04-20T21:32:03.722 INFO:tasks.workunit.client.1.smithi189.stdout:9/222: fsync d9/d12/d3b/f46 0 2024-04-20T21:32:03.727 INFO:tasks.workunit.client.0.smithi062.stdout:7/573: rename d7/d3e/d33/d3d/d7f/f89 to d7/d3e/d33/d3d/d4d/d83/d95/fc3 0 2024-04-20T21:32:03.728 INFO:tasks.workunit.client.0.smithi062.stdout:3/526: dwrite d0/d1c/d63/d42/f73 [0,4194304] 0 2024-04-20T21:32:03.728 INFO:tasks.workunit.client.0.smithi062.stdout:5/559: write d8/d38/d7a/d4c/d7d/d9d/dac/d66/daa/fab [641107,16419] 0 2024-04-20T21:32:03.731 INFO:tasks.workunit.client.0.smithi062.stdout:4/578: write da/f16 [4907501,53879] 0 2024-04-20T21:32:03.732 INFO:tasks.workunit.client.1.smithi189.stdout:4/183: mknod d2/dd/d11/c48 0 2024-04-20T21:32:03.735 INFO:tasks.workunit.client.1.smithi189.stdout:5/151: dread d7/d14/d18/f17 [0,4194304] 0 2024-04-20T21:32:03.736 INFO:tasks.workunit.client.0.smithi062.stdout:9/556: dread - d16/d3d/d55/d6e/d7d/f91 zero size 2024-04-20T21:32:03.736 INFO:tasks.workunit.client.0.smithi062.stdout:3/527: chown d0/d1c/la5 16 1 2024-04-20T21:32:03.738 INFO:tasks.workunit.client.0.smithi062.stdout:5/560: chown d8/d38/d7a/d4c/d7d/d9d/dac 473813629 1 2024-04-20T21:32:03.742 INFO:tasks.workunit.client.0.smithi062.stdout:6/429: unlink d0/d1a/d7d/l5a 0 2024-04-20T21:32:03.743 INFO:tasks.workunit.client.1.smithi189.stdout:9/223: rename d9/dc/d17/l41 to d9/d12/d15/d1a/d47/l4b 0 2024-04-20T21:32:03.743 INFO:tasks.workunit.client.1.smithi189.stdout:4/184: unlink d2/d3a/f43 0 2024-04-20T21:32:03.746 INFO:tasks.workunit.client.0.smithi062.stdout:2/575: link d7/da/d15/d2f/d38/d6a/d7c/fa5 d7/da/d15/d2f/d97/fbb 0 2024-04-20T21:32:03.749 INFO:tasks.workunit.client.0.smithi062.stdout:1/583: dwrite dc/d15/d75/d94/d36/f5f [0,4194304] 0 2024-04-20T21:32:03.749 INFO:tasks.workunit.client.1.smithi189.stdout:9/224: truncate d9/d12/d15/f31 2072973 0 2024-04-20T21:32:03.751 INFO:tasks.workunit.client.1.smithi189.stdout:1/193: dread d8/f1b [0,4194304] 0 2024-04-20T21:32:03.751 INFO:tasks.workunit.client.0.smithi062.stdout:7/574: symlink d7/d3e/d33/d3d/d5e/lc4 0 2024-04-20T21:32:03.755 INFO:tasks.workunit.client.0.smithi062.stdout:8/553: truncate d7/d25/d76/f9c 514262 0 2024-04-20T21:32:03.755 INFO:tasks.workunit.client.0.smithi062.stdout:9/557: mknod d16/d2f/d69/cb8 0 2024-04-20T21:32:03.755 INFO:tasks.workunit.client.0.smithi062.stdout:3/528: rmdir d0/d1/d3c/d43/d72/da9 39 2024-04-20T21:32:03.756 INFO:tasks.workunit.client.1.smithi189.stdout:1/194: dread - d8/d1e/f3c zero size 2024-04-20T21:32:03.756 INFO:tasks.workunit.client.1.smithi189.stdout:2/173: fsync d1/f4 0 2024-04-20T21:32:03.757 INFO:tasks.workunit.client.1.smithi189.stdout:1/195: fdatasync f6 0 2024-04-20T21:32:03.757 INFO:tasks.workunit.client.1.smithi189.stdout:0/291: sync 2024-04-20T21:32:03.757 INFO:tasks.workunit.client.1.smithi189.stdout:8/301: sync 2024-04-20T21:32:03.758 INFO:tasks.workunit.client.0.smithi062.stdout:9/558: write d16/d3e/f64 [792747,21176] 0 2024-04-20T21:32:03.759 INFO:tasks.workunit.client.0.smithi062.stdout:6/430: symlink d0/d2c/d67/d75/l87 0 2024-04-20T21:32:03.761 INFO:tasks.workunit.client.1.smithi189.stdout:1/196: write d8/d1e/f3c [341811,15606] 0 2024-04-20T21:32:03.763 INFO:tasks.workunit.client.1.smithi189.stdout:7/195: rmdir d5 39 2024-04-20T21:32:03.767 INFO:tasks.workunit.client.1.smithi189.stdout:1/197: write d8/f11 [5496343,24962] 0 2024-04-20T21:32:03.771 INFO:tasks.workunit.client.1.smithi189.stdout:3/154: dread d2/d8/f22 [0,4194304] 0 2024-04-20T21:32:03.771 INFO:tasks.workunit.client.1.smithi189.stdout:8/302: mkdir da/de/d1e/d6b 0 2024-04-20T21:32:03.773 INFO:tasks.workunit.client.0.smithi062.stdout:6/431: read d0/fe [8935273,54957] 0 2024-04-20T21:32:03.774 INFO:tasks.workunit.client.0.smithi062.stdout:9/559: write d16/d2f/d5b/dab/fb2 [1042495,88491] 0 2024-04-20T21:32:03.777 INFO:tasks.workunit.client.1.smithi189.stdout:6/192: truncate d8/da/f32 2373515 0 2024-04-20T21:32:03.777 INFO:tasks.workunit.client.1.smithi189.stdout:8/303: chown da/de/d1e/d21 32596236 1 2024-04-20T21:32:03.778 INFO:tasks.workunit.client.0.smithi062.stdout:2/576: mkdir d7/da/d15/d2f/d38/d51/dbc 0 2024-04-20T21:32:03.781 INFO:tasks.workunit.client.1.smithi189.stdout:0/292: symlink d3/df/d5d/l5f 0 2024-04-20T21:32:03.782 INFO:tasks.workunit.client.0.smithi062.stdout:5/561: dwrite d8/d10/d3e/d8a/fa0 [0,4194304] 0 2024-04-20T21:32:03.783 INFO:tasks.workunit.client.1.smithi189.stdout:2/174: dread d1/f4 [4194304,4194304] 0 2024-04-20T21:32:03.791 INFO:tasks.workunit.client.1.smithi189.stdout:3/155: mkdir d2/d3/d36 0 2024-04-20T21:32:03.795 INFO:tasks.workunit.client.1.smithi189.stdout:6/193: creat d8/d2b/f3a x:0 0 0 2024-04-20T21:32:03.795 INFO:tasks.workunit.client.1.smithi189.stdout:7/196: rmdir d5/d1d/d28 39 2024-04-20T21:32:03.799 INFO:tasks.workunit.client.1.smithi189.stdout:0/293: readlink d3/df/d28/l40 0 2024-04-20T21:32:03.799 INFO:tasks.workunit.client.1.smithi189.stdout:7/197: read - d5/f33 zero size 2024-04-20T21:32:03.800 INFO:tasks.workunit.client.0.smithi062.stdout:5/562: unlink d8/d38/f6f 0 2024-04-20T21:32:03.800 INFO:tasks.workunit.client.0.smithi062.stdout:0/542: dwrite d7/d1e/d6e/f75 [0,4194304] 0 2024-04-20T21:32:03.803 INFO:tasks.workunit.client.1.smithi189.stdout:8/304: write da/df/d22/d33/f5a [5206589,114494] 0 2024-04-20T21:32:03.805 INFO:tasks.workunit.client.1.smithi189.stdout:2/175: rename d1/d11/d26/l20 to d1/d11/d26/l27 0 2024-04-20T21:32:03.806 INFO:tasks.workunit.client.1.smithi189.stdout:9/225: getdents d9/dc/d17/d2b 0 2024-04-20T21:32:03.807 INFO:tasks.workunit.client.0.smithi062.stdout:3/529: creat d0/d1c/d63/fb2 x:0 0 0 2024-04-20T21:32:03.807 INFO:tasks.workunit.client.1.smithi189.stdout:8/305: write da/fb [3558504,38348] 0 2024-04-20T21:32:03.815 INFO:tasks.workunit.client.0.smithi062.stdout:6/432: truncate d0/f5e 4249851 0 2024-04-20T21:32:03.819 INFO:tasks.workunit.client.0.smithi062.stdout:5/563: mknod d8/d10/d3e/d8a/cb2 0 2024-04-20T21:32:03.819 INFO:tasks.workunit.client.1.smithi189.stdout:7/198: write d5/fe [494222,23301] 0 2024-04-20T21:32:03.821 INFO:tasks.workunit.client.1.smithi189.stdout:3/156: mknod d2/d3/d2c/d31/c37 0 2024-04-20T21:32:03.822 INFO:tasks.workunit.client.1.smithi189.stdout:5/152: dwrite d7/d9/fc [0,4194304] 0 2024-04-20T21:32:03.822 INFO:tasks.workunit.client.1.smithi189.stdout:7/199: chown d5/d21/c41 5910762 1 2024-04-20T21:32:03.822 INFO:tasks.workunit.client.1.smithi189.stdout:6/194: mkdir d8/da/dd/d2a/d3b 0 2024-04-20T21:32:03.824 INFO:tasks.workunit.client.1.smithi189.stdout:1/198: truncate f1 7577516 0 2024-04-20T21:32:03.826 INFO:tasks.workunit.client.1.smithi189.stdout:2/176: creat d1/d11/f28 x:0 0 0 2024-04-20T21:32:03.828 INFO:tasks.workunit.client.0.smithi062.stdout:3/530: fdatasync d0/d11/f56 0 2024-04-20T21:32:03.828 INFO:tasks.workunit.client.1.smithi189.stdout:3/157: fdatasync d2/d3/f19 0 2024-04-20T21:32:03.828 INFO:tasks.workunit.client.1.smithi189.stdout:4/185: dread d2/f8 [0,4194304] 0 2024-04-20T21:32:03.831 INFO:tasks.workunit.client.0.smithi062.stdout:1/584: dwrite dc/d15/d75/d94/d6f/f7d [0,4194304] 0 2024-04-20T21:32:03.833 INFO:tasks.workunit.client.1.smithi189.stdout:8/306: fdatasync da/df/d20/d27/f38 0 2024-04-20T21:32:03.837 INFO:tasks.workunit.client.0.smithi062.stdout:6/433: unlink d0/d1a/d44/l4a 0 2024-04-20T21:32:03.840 INFO:tasks.workunit.client.1.smithi189.stdout:9/226: dread d9/dc/f3d [0,4194304] 0 2024-04-20T21:32:03.840 INFO:tasks.workunit.client.0.smithi062.stdout:2/577: dread d7/da/d28/d4b/f73 [0,4194304] 0 2024-04-20T21:32:03.844 INFO:tasks.workunit.client.1.smithi189.stdout:7/200: write d5/d1d/d28/f2c [1411979,123743] 0 2024-04-20T21:32:03.846 INFO:tasks.workunit.client.1.smithi189.stdout:2/177: chown d1/d17/l24 37564520 1 2024-04-20T21:32:03.846 INFO:tasks.workunit.client.0.smithi062.stdout:4/579: dwrite da/dd/df/fa2 [0,4194304] 0 2024-04-20T21:32:03.847 INFO:tasks.workunit.client.0.smithi062.stdout:7/575: dwrite d7/d3e/d33/d3d/d4d/d83/d95/faa [0,4194304] 0 2024-04-20T21:32:03.847 INFO:tasks.workunit.client.1.smithi189.stdout:5/153: creat d7/d19/f2d x:0 0 0 2024-04-20T21:32:03.847 INFO:tasks.workunit.client.1.smithi189.stdout:0/294: rename d3/df/d4c/d35 to d3/df/d30/d45/d60 0 2024-04-20T21:32:03.851 INFO:tasks.workunit.client.1.smithi189.stdout:1/199: creat d8/f46 x:0 0 0 2024-04-20T21:32:03.855 INFO:tasks.workunit.client.0.smithi062.stdout:6/434: truncate d0/d4f/f73 672926 0 2024-04-20T21:32:03.855 INFO:tasks.workunit.client.1.smithi189.stdout:4/186: mknod d2/d36/c49 0 2024-04-20T21:32:03.856 INFO:tasks.workunit.client.0.smithi062.stdout:7/576: fdatasync d7/d3e/d33/d3d/d5e/fb4 0 2024-04-20T21:32:03.856 INFO:tasks.workunit.client.0.smithi062.stdout:5/564: dread d8/d1e/d2c/f69 [0,4194304] 0 2024-04-20T21:32:03.857 INFO:tasks.workunit.client.1.smithi189.stdout:8/307: mkdir da/df/d3c/d50/d6c 0 2024-04-20T21:32:03.861 INFO:tasks.workunit.client.1.smithi189.stdout:8/308: chown da/df/d13/f5f 113 1 2024-04-20T21:32:03.861 INFO:tasks.workunit.client.0.smithi062.stdout:2/578: write d7/da/d15/d2f/d38/d51/f60 [508453,106017] 0 2024-04-20T21:32:03.861 INFO:tasks.workunit.client.0.smithi062.stdout:1/585: symlink dc/d15/d75/d94/d36/d3a/lcd 0 2024-04-20T21:32:03.861 INFO:tasks.workunit.client.0.smithi062.stdout:9/560: dwrite d16/d3d/d55/d6e/d7d/f91 [0,4194304] 0 2024-04-20T21:32:03.864 INFO:tasks.workunit.client.1.smithi189.stdout:3/158: dwrite d2/d1c/f2b [0,4194304] 0 2024-04-20T21:32:03.867 INFO:tasks.workunit.client.0.smithi062.stdout:1/586: chown dc/d27/l71 293 1 2024-04-20T21:32:03.868 INFO:tasks.workunit.client.0.smithi062.stdout:0/543: dwrite d7/d20/d62/d6f/f9f [0,4194304] 0 2024-04-20T21:32:03.869 INFO:tasks.workunit.client.1.smithi189.stdout:2/178: symlink d1/d11/d26/l29 0 2024-04-20T21:32:03.869 INFO:tasks.workunit.client.1.smithi189.stdout:6/195: rename d8/d2b/c2c to d8/da/dd/d2a/d3b/c3c 0 2024-04-20T21:32:03.870 INFO:tasks.workunit.client.1.smithi189.stdout:5/154: mknod d7/d14/d18/c2e 0 2024-04-20T21:32:03.874 INFO:tasks.workunit.client.0.smithi062.stdout:4/580: getdents da/d56/d71 0 2024-04-20T21:32:03.875 INFO:tasks.workunit.client.1.smithi189.stdout:0/295: creat d3/df/d28/f61 x:0 0 0 2024-04-20T21:32:03.875 INFO:tasks.workunit.client.1.smithi189.stdout:4/187: mknod d2/d22/c4a 0 2024-04-20T21:32:03.875 INFO:tasks.workunit.client.1.smithi189.stdout:2/179: write d1/f4 [6327528,47085] 0 2024-04-20T21:32:03.875 INFO:tasks.workunit.client.1.smithi189.stdout:4/188: read d2/f8 [4552490,50330] 0 2024-04-20T21:32:03.876 INFO:tasks.workunit.client.0.smithi062.stdout:6/435: mknod d0/c88 0 2024-04-20T21:32:03.876 INFO:tasks.workunit.client.1.smithi189.stdout:7/201: mkdir d5/d1d/d42/d45 0 2024-04-20T21:32:03.878 INFO:tasks.workunit.client.0.smithi062.stdout:8/554: sync 2024-04-20T21:32:03.882 INFO:tasks.workunit.client.0.smithi062.stdout:5/565: chown d8/d38/d7a/d4c/d7d/d9d/dac/c6e 190 1 2024-04-20T21:32:03.882 INFO:tasks.workunit.client.0.smithi062.stdout:8/555: chown d7/d9/l2e 40 1 2024-04-20T21:32:03.883 INFO:tasks.workunit.client.0.smithi062.stdout:9/561: fsync d16/f3a 0 2024-04-20T21:32:03.883 INFO:tasks.workunit.client.0.smithi062.stdout:6/436: fdatasync d0/d52/f83 0 2024-04-20T21:32:03.883 INFO:tasks.workunit.client.1.smithi189.stdout:9/227: unlink d9/d12/d15/d1a/d47/l4b 0 2024-04-20T21:32:03.883 INFO:tasks.workunit.client.0.smithi062.stdout:3/531: rename d0/d1/d10/c80 to d0/d1/d3c/cb3 0 2024-04-20T21:32:03.884 INFO:tasks.workunit.client.0.smithi062.stdout:2/579: creat d7/da/fbd x:0 0 0 2024-04-20T21:32:03.885 INFO:tasks.workunit.client.0.smithi062.stdout:1/587: mkdir dc/d15/d20/d28/d83/dce 0 2024-04-20T21:32:03.886 INFO:tasks.workunit.client.1.smithi189.stdout:3/159: symlink d2/d3/d2c/d30/l38 0 2024-04-20T21:32:03.886 INFO:tasks.workunit.client.1.smithi189.stdout:6/196: mknod d8/da/dd/c3d 0 2024-04-20T21:32:03.887 INFO:tasks.workunit.client.0.smithi062.stdout:8/556: creat d7/d25/d4f/fbc x:0 0 0 2024-04-20T21:32:03.893 INFO:tasks.workunit.client.1.smithi189.stdout:5/155: mknod d7/d19/c2f 0 2024-04-20T21:32:03.894 INFO:tasks.workunit.client.1.smithi189.stdout:8/309: dwrite da/de/d1e/d21/d44/f43 [0,4194304] 0 2024-04-20T21:32:03.894 INFO:tasks.workunit.client.1.smithi189.stdout:6/197: truncate d8/d10/f36 5238853 0 2024-04-20T21:32:03.894 INFO:tasks.workunit.client.1.smithi189.stdout:1/200: creat d8/d1e/d43/d44/f47 x:0 0 0 2024-04-20T21:32:03.895 INFO:tasks.workunit.client.0.smithi062.stdout:9/562: rename d16/d2c/d42/f48 to d16/d2c/fb9 0 2024-04-20T21:32:03.896 INFO:tasks.workunit.client.1.smithi189.stdout:5/156: stat d7/d14/d18/f17 0 2024-04-20T21:32:03.896 INFO:tasks.workunit.client.0.smithi062.stdout:6/437: creat d0/d2c/f89 x:0 0 0 2024-04-20T21:32:03.900 INFO:tasks.workunit.client.0.smithi062.stdout:2/580: creat d7/da/d28/d4b/fbe x:0 0 0 2024-04-20T21:32:03.900 INFO:tasks.workunit.client.0.smithi062.stdout:1/588: symlink dc/d15/d75/d94/d6f/lcf 0 2024-04-20T21:32:03.902 INFO:tasks.workunit.client.1.smithi189.stdout:0/296: rename d3/fb to d3/df/d28/d2b/d2c/f62 0 2024-04-20T21:32:03.910 INFO:tasks.workunit.client.0.smithi062.stdout:3/532: mknod d0/d1/d10/cb4 0 2024-04-20T21:32:03.911 INFO:tasks.workunit.client.1.smithi189.stdout:0/297: read d3/df/d28/d2b/f58 [63995,20928] 0 2024-04-20T21:32:03.911 INFO:tasks.workunit.client.1.smithi189.stdout:6/198: mknod d8/d12/d30/c3e 0 2024-04-20T21:32:03.913 INFO:tasks.workunit.client.0.smithi062.stdout:8/557: mkdir d7/d25/d3b/d58/d8b/d24/dbd 0 2024-04-20T21:32:03.913 INFO:tasks.workunit.client.0.smithi062.stdout:1/589: mkdir dc/d15/d75/dd0 0 2024-04-20T21:32:03.917 INFO:tasks.workunit.client.1.smithi189.stdout:7/202: dwrite d5/d1d/d28/f38 [4194304,4194304] 0 2024-04-20T21:32:03.917 INFO:tasks.workunit.client.1.smithi189.stdout:3/160: rename d2/l23 to d2/d3/d25/l39 0 2024-04-20T21:32:03.917 INFO:tasks.workunit.client.0.smithi062.stdout:0/544: dwrite d7/d1c/d15/d17/f38 [0,4194304] 0 2024-04-20T21:32:03.918 INFO:tasks.workunit.client.1.smithi189.stdout:9/228: fdatasync d9/dc/d17/f2f 0 2024-04-20T21:32:03.922 INFO:tasks.workunit.client.1.smithi189.stdout:0/298: chown d3/df/d30/d45/d60 0 1 2024-04-20T21:32:03.922 INFO:tasks.workunit.client.1.smithi189.stdout:8/310: link da/df/f19 da/df/d3c/d50/f6d 0 2024-04-20T21:32:03.925 INFO:tasks.workunit.client.0.smithi062.stdout:3/533: symlink d0/d1/d10/d3a/d5e/d65/lb5 0 2024-04-20T21:32:03.927 INFO:tasks.workunit.client.0.smithi062.stdout:1/590: chown dc/d15/d20/d28/dc9/c40 0 1 2024-04-20T21:32:03.927 INFO:tasks.workunit.client.0.smithi062.stdout:7/577: dwrite d7/f24 [0,4194304] 0 2024-04-20T21:32:03.928 INFO:tasks.workunit.client.1.smithi189.stdout:0/299: chown d3/df/d28/d2b/f49 126352029 1 2024-04-20T21:32:03.928 INFO:tasks.workunit.client.1.smithi189.stdout:1/201: read d8/df/f15 [2534111,49909] 0 2024-04-20T21:32:03.930 INFO:tasks.workunit.client.1.smithi189.stdout:3/161: truncate d2/d8/f22 4346095 0 2024-04-20T21:32:03.932 INFO:tasks.workunit.client.1.smithi189.stdout:3/162: readlink d2/d3/l9 0 2024-04-20T21:32:03.932 INFO:tasks.workunit.client.1.smithi189.stdout:4/189: dwrite d2/f8 [0,4194304] 0 2024-04-20T21:32:03.935 INFO:tasks.workunit.client.1.smithi189.stdout:2/180: creat d1/f2a x:0 0 0 2024-04-20T21:32:03.935 INFO:tasks.workunit.client.0.smithi062.stdout:9/563: dwrite d16/d3d/d55/d6e/f72 [0,4194304] 0 2024-04-20T21:32:03.937 INFO:tasks.workunit.client.1.smithi189.stdout:4/190: stat d2/f32 0 2024-04-20T21:32:03.940 INFO:tasks.workunit.client.1.smithi189.stdout:1/202: dread d8/d1e/f23 [0,4194304] 0 2024-04-20T21:32:03.941 INFO:tasks.workunit.client.0.smithi062.stdout:3/534: chown d0/d1c/d63/la8 0 1 2024-04-20T21:32:03.941 INFO:tasks.workunit.client.0.smithi062.stdout:0/545: symlink d7/d1c/d15/d17/d1b/lb7 0 2024-04-20T21:32:03.941 INFO:tasks.workunit.client.0.smithi062.stdout:6/438: dread d0/d1a/d44/f4c [0,4194304] 0 2024-04-20T21:32:03.943 INFO:tasks.workunit.client.1.smithi189.stdout:4/191: dread - d2/dd/d11/d14/f41 zero size 2024-04-20T21:32:03.943 INFO:tasks.workunit.client.0.smithi062.stdout:1/591: unlink dc/d70/d7f/ca1 0 2024-04-20T21:32:03.945 INFO:tasks.workunit.client.0.smithi062.stdout:4/581: truncate da/dd/df/d27/d32/f89 830292 0 2024-04-20T21:32:03.947 INFO:tasks.workunit.client.0.smithi062.stdout:0/546: write d7/d20/d3a/f44 [1392047,80025] 0 2024-04-20T21:32:03.948 INFO:tasks.workunit.client.1.smithi189.stdout:6/199: dwrite d8/da/f1e [0,4194304] 0 2024-04-20T21:32:03.950 INFO:tasks.workunit.client.1.smithi189.stdout:8/311: mkdir da/df/d20/d27/d6e 0 2024-04-20T21:32:03.950 INFO:tasks.workunit.client.1.smithi189.stdout:6/200: readlink d8/da/lc 0 2024-04-20T21:32:03.951 INFO:tasks.workunit.client.0.smithi062.stdout:2/581: dwrite d7/da/d15/d2f/d38/d51/f60 [0,4194304] 0 2024-04-20T21:32:03.951 INFO:tasks.workunit.client.0.smithi062.stdout:3/535: truncate d0/d1/fb 759454 0 2024-04-20T21:32:03.952 INFO:tasks.workunit.client.1.smithi189.stdout:8/312: chown da/df/f23 1072411153 1 2024-04-20T21:32:03.957 INFO:tasks.workunit.client.0.smithi062.stdout:3/536: fsync d0/d1/d10/f4e 0 2024-04-20T21:32:03.957 INFO:tasks.workunit.client.0.smithi062.stdout:2/582: chown d7/da/d15/d2f/d97/fbb 401672990 1 2024-04-20T21:32:03.960 INFO:tasks.workunit.client.0.smithi062.stdout:0/547: stat d7/d1c/d15/d6b 0 2024-04-20T21:32:03.961 INFO:tasks.workunit.client.0.smithi062.stdout:4/582: mknod da/dd/df/d27/d3f/ccf 0 2024-04-20T21:32:03.962 INFO:tasks.workunit.client.1.smithi189.stdout:3/163: symlink d2/d3/d2c/d30/l3a 0 2024-04-20T21:32:03.964 INFO:tasks.workunit.client.1.smithi189.stdout:5/157: rmdir d7/d14 39 2024-04-20T21:32:03.967 INFO:tasks.workunit.client.1.smithi189.stdout:3/164: write d2/d8/f34 [822693,41995] 0 2024-04-20T21:32:03.972 INFO:tasks.workunit.client.0.smithi062.stdout:7/578: dread d7/f6c [0,4194304] 0 2024-04-20T21:32:03.974 INFO:tasks.workunit.client.1.smithi189.stdout:7/203: mkdir d5/d46 0 2024-04-20T21:32:03.976 INFO:tasks.workunit.client.1.smithi189.stdout:6/201: creat d8/d10/f3f x:0 0 0 2024-04-20T21:32:03.979 INFO:tasks.workunit.client.1.smithi189.stdout:9/229: dread f4 [0,4194304] 0 2024-04-20T21:32:03.980 INFO:tasks.workunit.client.1.smithi189.stdout:0/300: dwrite d3/df/d28/d2b/f36 [0,4194304] 0 2024-04-20T21:32:03.989 INFO:tasks.workunit.client.1.smithi189.stdout:8/313: rename da/df/c12 to da/df/d22/d33/c6f 0 2024-04-20T21:32:03.991 INFO:tasks.workunit.client.0.smithi062.stdout:2/583: dread d7/da/d15/d1b/f80 [0,4194304] 0 2024-04-20T21:32:03.991 INFO:tasks.workunit.client.1.smithi189.stdout:3/165: dread d2/d8/f10 [8388608,4194304] 0 2024-04-20T21:32:03.999 INFO:tasks.workunit.client.0.smithi062.stdout:1/592: write dc/f6d [45937,1059] 0 2024-04-20T21:32:04.011 INFO:tasks.workunit.client.1.smithi189.stdout:9/230: dwrite d9/d12/d15/f26 [0,4194304] 0 2024-04-20T21:32:04.012 INFO:tasks.workunit.client.0.smithi062.stdout:8/558: dwrite d7/d25/f5a [0,4194304] 0 2024-04-20T21:32:04.013 INFO:tasks.workunit.client.1.smithi189.stdout:5/158: dread d7/d14/f26 [0,4194304] 0 2024-04-20T21:32:04.015 INFO:tasks.workunit.client.1.smithi189.stdout:1/203: link d8/f40 d8/d1e/d28/d2b/f48 0 2024-04-20T21:32:04.016 INFO:tasks.workunit.client.0.smithi062.stdout:1/593: stat dc/d15/d75/d94/d36/f5f 0 2024-04-20T21:32:04.016 INFO:tasks.workunit.client.0.smithi062.stdout:2/584: link d7/d39/f86 d7/da/d15/d2f/d38/fbf 0 2024-04-20T21:32:04.020 INFO:tasks.workunit.client.1.smithi189.stdout:9/231: fsync d9/d12/d15/f31 0 2024-04-20T21:32:04.020 INFO:tasks.workunit.client.1.smithi189.stdout:3/166: getdents d2/d3/d36 0 2024-04-20T21:32:04.023 INFO:tasks.workunit.client.1.smithi189.stdout:9/232: chown d9/d12/d15/d1a/l3e 14 1 2024-04-20T21:32:04.025 INFO:tasks.workunit.client.1.smithi189.stdout:9/233: read - d9/dc/f2c zero size 2024-04-20T21:32:04.025 INFO:tasks.workunit.client.0.smithi062.stdout:4/583: dwrite da/d56/f74 [0,4194304] 0 2024-04-20T21:32:04.029 INFO:tasks.workunit.client.1.smithi189.stdout:8/314: dwrite da/df/d3c/d50/f61 [0,4194304] 0 2024-04-20T21:32:04.030 INFO:tasks.workunit.client.0.smithi062.stdout:9/564: dwrite d16/d2c/d5e/f87 [0,4194304] 0 2024-04-20T21:32:04.032 INFO:tasks.workunit.client.0.smithi062.stdout:1/594: mknod dc/d15/d75/d94/d36/d3a/cd1 0 2024-04-20T21:32:04.033 INFO:tasks.workunit.client.1.smithi189.stdout:7/204: creat d5/f47 x:0 0 0 2024-04-20T21:32:04.037 INFO:tasks.workunit.client.1.smithi189.stdout:8/315: fsync da/df/d13/f18 0 2024-04-20T21:32:04.040 INFO:tasks.workunit.client.0.smithi062.stdout:9/565: write d16/d2f/d5b/dab/fb2 [1050104,47277] 0 2024-04-20T21:32:04.040 INFO:tasks.workunit.client.0.smithi062.stdout:4/584: mknod da/dd/d15/d50/d64/cd0 0 2024-04-20T21:32:04.043 INFO:tasks.workunit.client.0.smithi062.stdout:3/537: dwrite d0/d5c/d76/faa [4194304,4194304] 0 2024-04-20T21:32:04.050 INFO:tasks.workunit.client.0.smithi062.stdout:2/585: rename d7/da/d15/d1b/c4c to d7/da/d15/d2f/d38/d51/cc0 0 2024-04-20T21:32:04.050 INFO:tasks.workunit.client.1.smithi189.stdout:0/301: creat d3/d26/f63 x:0 0 0 2024-04-20T21:32:04.051 INFO:tasks.workunit.client.1.smithi189.stdout:6/202: rmdir d8/d10 39 2024-04-20T21:32:04.053 INFO:tasks.workunit.client.0.smithi062.stdout:2/586: fsync d7/d39/f55 0 2024-04-20T21:32:04.055 INFO:tasks.workunit.client.1.smithi189.stdout:2/181: write d1/f5 [2544493,40147] 0 2024-04-20T21:32:04.055 INFO:tasks.workunit.client.0.smithi062.stdout:1/595: mkdir dc/d15/d75/d94/d6f/dd2 0 2024-04-20T21:32:04.058 INFO:tasks.workunit.client.1.smithi189.stdout:5/159: dwrite d7/fb [0,4194304] 0 2024-04-20T21:32:04.062 INFO:tasks.workunit.client.0.smithi062.stdout:4/585: creat da/dd/d15/d50/fd1 x:0 0 0 2024-04-20T21:32:04.063 INFO:tasks.workunit.client.0.smithi062.stdout:9/566: symlink d16/d3d/d55/d6e/d7d/lba 0 2024-04-20T21:32:04.065 INFO:tasks.workunit.client.0.smithi062.stdout:7/579: dread d7/f48 [0,4194304] 0 2024-04-20T21:32:04.065 INFO:tasks.workunit.client.0.smithi062.stdout:3/538: creat d0/d1c/d52/d7b/fb6 x:0 0 0 2024-04-20T21:32:04.067 INFO:tasks.workunit.client.1.smithi189.stdout:9/234: dread f1 [0,4194304] 0 2024-04-20T21:32:04.069 INFO:tasks.workunit.client.1.smithi189.stdout:3/167: dwrite d2/d3/d25/f2e [0,4194304] 0 2024-04-20T21:32:04.069 INFO:tasks.workunit.client.1.smithi189.stdout:6/203: write f2 [3198079,52425] 0 2024-04-20T21:32:04.070 INFO:tasks.workunit.client.0.smithi062.stdout:1/596: creat dc/d15/d20/d28/dc9/fd3 x:0 0 0 2024-04-20T21:32:04.071 INFO:tasks.workunit.client.1.smithi189.stdout:7/205: creat d5/d1d/d42/d45/f48 x:0 0 0 2024-04-20T21:32:04.073 INFO:tasks.workunit.client.0.smithi062.stdout:4/586: mknod da/cd2 0 2024-04-20T21:32:04.074 INFO:tasks.workunit.client.0.smithi062.stdout:9/567: symlink d16/d3d/lbb 0 2024-04-20T21:32:04.077 INFO:tasks.workunit.client.1.smithi189.stdout:1/204: getdents d8/d21 0 2024-04-20T21:32:04.077 INFO:tasks.workunit.client.0.smithi062.stdout:2/587: unlink d7/da/d15/d2f/d38/d6b/l6f 0 2024-04-20T21:32:04.079 INFO:tasks.workunit.client.0.smithi062.stdout:7/580: mknod d7/d3e/d33/d3d/d5e/d7e/cc5 0 2024-04-20T21:32:04.081 INFO:tasks.workunit.client.1.smithi189.stdout:9/235: creat d9/d12/d3b/f4c x:0 0 0 2024-04-20T21:32:04.081 INFO:tasks.workunit.client.0.smithi062.stdout:4/587: chown da/fb 226 1 2024-04-20T21:32:04.084 INFO:tasks.workunit.client.0.smithi062.stdout:9/568: creat d16/d2c/d62/fbc x:0 0 0 2024-04-20T21:32:04.089 INFO:tasks.workunit.client.0.smithi062.stdout:7/581: mknod d7/d26/cc6 0 2024-04-20T21:32:04.092 INFO:tasks.workunit.client.1.smithi189.stdout:8/316: dwrite da/fb [0,4194304] 0 2024-04-20T21:32:04.092 INFO:tasks.workunit.client.1.smithi189.stdout:4/192: dwrite d2/dd/f2d [0,4194304] 0 2024-04-20T21:32:04.095 INFO:tasks.workunit.client.0.smithi062.stdout:8/559: dread d7/fa5 [0,4194304] 0 2024-04-20T21:32:04.095 INFO:tasks.workunit.client.0.smithi062.stdout:1/597: rename dc/d15/d75/d94/d36 to dc/d15/d75/d96/dd4 0 2024-04-20T21:32:04.098 INFO:tasks.workunit.client.0.smithi062.stdout:8/560: chown d7/d9/d29/f40 2202 1 2024-04-20T21:32:04.099 INFO:tasks.workunit.client.1.smithi189.stdout:9/236: creat d9/dc/f4d x:0 0 0 2024-04-20T21:32:04.100 INFO:tasks.workunit.client.0.smithi062.stdout:9/569: dread d16/d2c/fb9 [0,4194304] 0 2024-04-20T21:32:04.100 INFO:tasks.workunit.client.1.smithi189.stdout:4/193: chown d2/dd/d11/d14/c15 441196 1 2024-04-20T21:32:04.100 INFO:tasks.workunit.client.1.smithi189.stdout:8/317: fdatasync da/de/f10 0 2024-04-20T21:32:04.103 INFO:tasks.workunit.client.1.smithi189.stdout:3/168: read d2/d8/f10 [1483250,92870] 0 2024-04-20T21:32:04.108 INFO:tasks.workunit.client.0.smithi062.stdout:8/561: mknod d7/d30/d90/cbe 0 2024-04-20T21:32:04.109 INFO:tasks.workunit.client.1.smithi189.stdout:5/160: dread d7/fb [0,4194304] 0 2024-04-20T21:32:04.111 INFO:tasks.workunit.client.0.smithi062.stdout:9/570: creat d16/d4e/fbd x:0 0 0 2024-04-20T21:32:04.111 INFO:tasks.workunit.client.1.smithi189.stdout:9/237: symlink d9/d12/l4e 0 2024-04-20T21:32:04.114 INFO:tasks.workunit.client.1.smithi189.stdout:8/318: truncate da/de/f16 5071466 0 2024-04-20T21:32:04.116 INFO:tasks.workunit.client.0.smithi062.stdout:1/598: creat dc/d70/fd5 x:0 0 0 2024-04-20T21:32:04.117 INFO:tasks.workunit.client.1.smithi189.stdout:0/302: dwrite d3/df/d28/d2b/f49 [0,4194304] 0 2024-04-20T21:32:04.119 INFO:tasks.workunit.client.0.smithi062.stdout:1/599: readlink dc/d15/d75/d94/d6f/lbf 0 2024-04-20T21:32:04.122 INFO:tasks.workunit.client.0.smithi062.stdout:1/600: write dc/d15/d75/d96/fb8 [113536,49078] 0 2024-04-20T21:32:04.126 INFO:tasks.workunit.client.0.smithi062.stdout:2/588: rename d7/f1a to d7/da/d15/d2f/fc1 0 2024-04-20T21:32:04.126 INFO:tasks.workunit.client.1.smithi189.stdout:7/206: dwrite d5/d1d/f44 [0,4194304] 0 2024-04-20T21:32:04.129 INFO:tasks.workunit.client.1.smithi189.stdout:7/207: chown d5/d1d 647 1 2024-04-20T21:32:04.129 INFO:tasks.workunit.client.0.smithi062.stdout:8/562: mkdir d7/d30/db3/dbf 0 2024-04-20T21:32:04.133 INFO:tasks.workunit.client.0.smithi062.stdout:9/571: mkdir d16/dbe 0 2024-04-20T21:32:04.133 INFO:tasks.workunit.client.0.smithi062.stdout:7/582: rename d7/d3e/d33/f57 to d7/d26/da8/fc7 0 2024-04-20T21:32:04.134 INFO:tasks.workunit.client.0.smithi062.stdout:8/563: truncate d7/d30/d90/f92 4558759 0 2024-04-20T21:32:04.137 INFO:tasks.workunit.client.1.smithi189.stdout:5/161: dread d7/fb [0,4194304] 0 2024-04-20T21:32:04.138 INFO:tasks.workunit.client.0.smithi062.stdout:7/583: write d7/d3e/d33/d3d/f75 [3329945,89762] 0 2024-04-20T21:32:04.138 INFO:tasks.workunit.client.0.smithi062.stdout:8/564: readlink d7/d25/d3b/d58/d8b/d24/l47 0 2024-04-20T21:32:04.140 INFO:tasks.workunit.client.0.smithi062.stdout:1/601: rename dc/d27/d9a/cb1 to dc/d27/d9a/da0/cd6 0 2024-04-20T21:32:04.140 INFO:tasks.workunit.client.1.smithi189.stdout:8/319: dread da/df/d20/d27/f38 [4194304,4194304] 0 2024-04-20T21:32:04.141 INFO:tasks.workunit.client.0.smithi062.stdout:4/588: dwrite da/dd/d15/f1e [0,4194304] 0 2024-04-20T21:32:04.141 INFO:tasks.workunit.client.1.smithi189.stdout:9/238: dwrite d9/d12/d3b/f4c [0,4194304] 0 2024-04-20T21:32:04.144 INFO:tasks.workunit.client.0.smithi062.stdout:8/565: chown d7/da9/cb4 68334423 1 2024-04-20T21:32:04.148 INFO:tasks.workunit.client.0.smithi062.stdout:7/584: creat d7/d3e/d33/d3d/d4d/d83/fc8 x:0 0 0 2024-04-20T21:32:04.153 INFO:tasks.workunit.client.0.smithi062.stdout:4/589: rename da/ce to da/dd/cd3 0 2024-04-20T21:32:04.153 INFO:tasks.workunit.client.0.smithi062.stdout:7/585: symlink d7/d3e/d33/d3d/d5e/d72/lc9 0 2024-04-20T21:32:04.156 INFO:tasks.workunit.client.0.smithi062.stdout:8/566: fsync d7/d25/d3b/d58/d8b/d24/f20 0 2024-04-20T21:32:04.159 INFO:tasks.workunit.client.0.smithi062.stdout:7/586: truncate d7/d3e/d33/d3d/d4d/fa3 51527 0 2024-04-20T21:32:04.163 INFO:tasks.workunit.client.0.smithi062.stdout:4/590: truncate da/dd/df/d27/d3f/d82/fbf 870090 0 2024-04-20T21:32:04.163 INFO:tasks.workunit.client.1.smithi189.stdout:1/205: dread d8/df/f15 [0,4194304] 0 2024-04-20T21:32:04.164 INFO:tasks.workunit.client.0.smithi062.stdout:9/572: write d16/f1d [1719151,76629] 0 2024-04-20T21:32:04.165 INFO:tasks.workunit.client.1.smithi189.stdout:1/206: fdatasync d8/f14 0 2024-04-20T21:32:04.165 INFO:tasks.workunit.client.0.smithi062.stdout:7/587: fsync d7/d3e/d33/d3d/d4d/d54/f3f 0 2024-04-20T21:32:04.166 INFO:tasks.workunit.client.0.smithi062.stdout:9/573: readlink d16/d3d/lbb 0 2024-04-20T21:32:04.166 INFO:tasks.workunit.client.1.smithi189.stdout:1/207: fdatasync f6 0 2024-04-20T21:32:04.167 INFO:tasks.workunit.client.0.smithi062.stdout:1/602: dwrite dc/d15/d75/d96/dd4/d3a/f46 [0,4194304] 0 2024-04-20T21:32:04.168 INFO:tasks.workunit.client.0.smithi062.stdout:8/567: mknod d7/d25/d3b/d58/d8b/cc0 0 2024-04-20T21:32:04.172 INFO:tasks.workunit.client.1.smithi189.stdout:8/320: dwrite da/de/f10 [4194304,4194304] 0 2024-04-20T21:32:04.183 INFO:tasks.workunit.client.0.smithi062.stdout:7/588: stat d7/d3e/d33/d3d/d4d/d54/d4f/fb3 0 2024-04-20T21:32:04.183 INFO:tasks.workunit.client.0.smithi062.stdout:9/574: creat d16/d3d/d55/fbf x:0 0 0 2024-04-20T21:32:04.184 INFO:tasks.workunit.client.1.smithi189.stdout:2/182: link d1/d11/f28 d1/d17/f2b 0 2024-04-20T21:32:04.184 INFO:tasks.workunit.client.0.smithi062.stdout:4/591: mknod da/dd/df/d27/d7a/cd4 0 2024-04-20T21:32:04.186 INFO:tasks.workunit.client.1.smithi189.stdout:6/204: getdents d8/da 0 2024-04-20T21:32:04.186 INFO:tasks.workunit.client.0.smithi062.stdout:8/568: write d7/d25/d3b/d58/d8b/f3f [3951054,113399] 0 2024-04-20T21:32:04.187 INFO:tasks.workunit.client.1.smithi189.stdout:2/183: write d1/f2a [722681,114872] 0 2024-04-20T21:32:04.188 INFO:tasks.workunit.client.0.smithi062.stdout:9/575: creat d16/d4e/d80/fc0 x:0 0 0 2024-04-20T21:32:04.188 INFO:tasks.workunit.client.0.smithi062.stdout:1/603: link dc/d15/d20/c47 dc/d15/d75/cd7 0 2024-04-20T21:32:04.190 INFO:tasks.workunit.client.0.smithi062.stdout:7/589: rmdir d7/d3e/d33/d3d/d5e/d7e 39 2024-04-20T21:32:04.191 INFO:tasks.workunit.client.0.smithi062.stdout:1/604: write dc/ff [2610551,28238] 0 2024-04-20T21:32:04.194 INFO:tasks.workunit.client.0.smithi062.stdout:4/592: mknod da/dd/d15/daf/cd5 0 2024-04-20T21:32:04.194 INFO:tasks.workunit.client.0.smithi062.stdout:9/576: symlink d16/d2c/lc1 0 2024-04-20T21:32:04.196 INFO:tasks.workunit.client.1.smithi189.stdout:5/162: rmdir d7 39 2024-04-20T21:32:04.202 INFO:tasks.workunit.client.1.smithi189.stdout:9/239: readlink d9/dc/d17/l18 0 2024-04-20T21:32:04.203 INFO:tasks.workunit.client.0.smithi062.stdout:9/577: dread d16/d4e/fa3 [0,4194304] 0 2024-04-20T21:32:04.205 INFO:tasks.workunit.client.1.smithi189.stdout:6/205: dwrite f5 [4194304,4194304] 0 2024-04-20T21:32:04.206 INFO:tasks.workunit.client.0.smithi062.stdout:1/605: rename dc/d15/d75/d96/dd4/f7e to dc/d27/fd8 0 2024-04-20T21:32:04.206 INFO:tasks.workunit.client.0.smithi062.stdout:9/578: write d16/d3e/f64 [352963,13702] 0 2024-04-20T21:32:04.207 INFO:tasks.workunit.client.0.smithi062.stdout:4/593: rename da/dd/d15/d50/d64/d57/d5b to da/dd/d15/d50/d64/d57/d5b/da3/dd6 22 2024-04-20T21:32:04.207 INFO:tasks.workunit.client.1.smithi189.stdout:8/321: rename da/de/d1e/d6b to da/de/d1e/d69/d70 0 2024-04-20T21:32:04.207 INFO:tasks.workunit.client.1.smithi189.stdout:1/208: write d8/f24 [3880735,54198] 0 2024-04-20T21:32:04.210 INFO:tasks.workunit.client.0.smithi062.stdout:7/590: link d7/fa0 d7/d3e/d33/d3d/fca 0 2024-04-20T21:32:04.211 INFO:tasks.workunit.client.0.smithi062.stdout:9/579: symlink d16/d2f/d5b/dab/lc2 0 2024-04-20T21:32:04.212 INFO:tasks.workunit.client.0.smithi062.stdout:7/591: read d7/d3e/f5d [471322,2031] 0 2024-04-20T21:32:04.213 INFO:tasks.workunit.client.0.smithi062.stdout:1/606: chown dc/d15/d20/d28/dc9/d84/l8f 1 1 2024-04-20T21:32:04.213 INFO:tasks.workunit.client.1.smithi189.stdout:2/184: symlink d1/d11/l2c 0 2024-04-20T21:32:04.214 INFO:tasks.workunit.client.0.smithi062.stdout:9/580: write d16/d4e/fbd [816317,1059] 0 2024-04-20T21:32:04.215 INFO:tasks.workunit.client.1.smithi189.stdout:8/322: write da/d1f/d2a/f4b [671600,115673] 0 2024-04-20T21:32:04.216 INFO:tasks.workunit.client.0.smithi062.stdout:1/607: stat dc/d15/d20/d28/dc9/d84/fb0 0 2024-04-20T21:32:04.218 INFO:tasks.workunit.client.1.smithi189.stdout:3/169: dwrite d2/f15 [0,4194304] 0 2024-04-20T21:32:04.218 INFO:tasks.workunit.client.1.smithi189.stdout:4/194: link d2/dd/d37/f39 d2/dd/d11/f4b 0 2024-04-20T21:32:04.219 INFO:tasks.workunit.client.1.smithi189.stdout:9/240: rename d9/dc/f1d to d9/d12/d15/d1a/d47/f4f 0 2024-04-20T21:32:04.220 INFO:tasks.workunit.client.1.smithi189.stdout:8/323: fdatasync da/df/d3c/d50/f5e 0 2024-04-20T21:32:04.221 INFO:tasks.workunit.client.0.smithi062.stdout:7/592: symlink d7/d3e/lcb 0 2024-04-20T21:32:04.224 INFO:tasks.workunit.client.0.smithi062.stdout:8/569: write d7/d25/d3b/d58/d8b/d24/f74 [505093,113427] 0 2024-04-20T21:32:04.225 INFO:tasks.workunit.client.1.smithi189.stdout:2/185: readlink d1/d11/d26/l27 0 2024-04-20T21:32:04.229 INFO:tasks.workunit.client.1.smithi189.stdout:4/195: write d2/d3a/f3b [8470409,26349] 0 2024-04-20T21:32:04.229 INFO:tasks.workunit.client.1.smithi189.stdout:0/303: getdents d3/df/d28/d2b/d2c 0 2024-04-20T21:32:04.233 INFO:tasks.workunit.client.1.smithi189.stdout:4/196: mknod d2/d20/c4c 0 2024-04-20T21:32:04.234 INFO:tasks.workunit.client.1.smithi189.stdout:2/186: mkdir d1/d11/d2d 0 2024-04-20T21:32:04.234 INFO:tasks.workunit.client.0.smithi062.stdout:8/570: creat d7/d25/d4f/fc1 x:0 0 0 2024-04-20T21:32:04.234 INFO:tasks.workunit.client.0.smithi062.stdout:9/581: creat d16/fc3 x:0 0 0 2024-04-20T21:32:04.238 INFO:tasks.workunit.client.0.smithi062.stdout:8/571: chown d7/d25/d4f/fbc 33 1 2024-04-20T21:32:04.238 INFO:tasks.workunit.client.0.smithi062.stdout:7/593: mkdir d7/d3e/d33/d3d/d5e/d7e/dcc 0 2024-04-20T21:32:04.238 INFO:tasks.workunit.client.1.smithi189.stdout:0/304: fsync d3/df/d28/d2b/f36 0 2024-04-20T21:32:04.238 INFO:tasks.workunit.client.1.smithi189.stdout:3/170: rmdir d2/d33 0 2024-04-20T21:32:04.239 INFO:tasks.workunit.client.1.smithi189.stdout:8/324: write da/df/d22/d33/f5a [2412992,26247] 0 2024-04-20T21:32:04.242 INFO:tasks.workunit.client.0.smithi062.stdout:4/594: dwrite da/dd/d15/d50/fd1 [0,4194304] 0 2024-04-20T21:32:04.242 INFO:tasks.workunit.client.1.smithi189.stdout:7/208: dwrite d5/f24 [0,4194304] 0 2024-04-20T21:32:04.243 INFO:tasks.workunit.client.0.smithi062.stdout:9/582: creat d16/d2c/d42/da6/da9/fc4 x:0 0 0 2024-04-20T21:32:04.244 INFO:tasks.workunit.client.0.smithi062.stdout:1/608: getdents dc/d70/d7f 0 2024-04-20T21:32:04.250 INFO:tasks.workunit.client.1.smithi189.stdout:2/187: creat d1/d11/f2e x:0 0 0 2024-04-20T21:32:04.250 INFO:tasks.workunit.client.0.smithi062.stdout:4/595: fdatasync da/dd/f73 0 2024-04-20T21:32:04.252 INFO:tasks.workunit.client.0.smithi062.stdout:4/596: chown f4 481055 1 2024-04-20T21:32:04.253 INFO:tasks.workunit.client.0.smithi062.stdout:9/583: write d16/d3d/d55/fb7 [599494,98021] 0 2024-04-20T21:32:04.255 INFO:tasks.workunit.client.1.smithi189.stdout:6/206: dwrite d8/d2b/f3a [0,4194304] 0 2024-04-20T21:32:04.259 INFO:tasks.workunit.client.1.smithi189.stdout:3/171: symlink d2/d3/d25/l3b 0 2024-04-20T21:32:04.260 INFO:tasks.workunit.client.0.smithi062.stdout:7/594: unlink d7/d3e/d33/d3d/d5e/l67 0 2024-04-20T21:32:04.261 INFO:tasks.workunit.client.0.smithi062.stdout:1/609: mknod dc/d15/d20/d28/d83/dc0/cd9 0 2024-04-20T21:32:04.263 INFO:tasks.workunit.client.1.smithi189.stdout:0/305: mkdir d3/df/d5d/d64 0 2024-04-20T21:32:04.266 INFO:tasks.workunit.client.1.smithi189.stdout:2/188: mkdir d1/d17/d2f 0 2024-04-20T21:32:04.268 INFO:tasks.workunit.client.0.smithi062.stdout:9/584: creat d16/d2f/d69/fc5 x:0 0 0 2024-04-20T21:32:04.271 INFO:tasks.workunit.client.0.smithi062.stdout:9/585: dread - d16/d2c/d5e/d96/fb3 zero size 2024-04-20T21:32:04.273 INFO:tasks.workunit.client.1.smithi189.stdout:7/209: unlink d5/d21/d25/f2d 0 2024-04-20T21:32:04.278 INFO:tasks.workunit.client.1.smithi189.stdout:1/209: dwrite d8/d1e/f23 [0,4194304] 0 2024-04-20T21:32:04.278 INFO:tasks.workunit.client.1.smithi189.stdout:6/207: symlink d8/l40 0 2024-04-20T21:32:04.279 INFO:tasks.workunit.client.0.smithi062.stdout:8/572: getdents d7/d25/d70 0 2024-04-20T21:32:04.280 INFO:tasks.workunit.client.0.smithi062.stdout:4/597: dwrite da/dd/df/fc7 [0,4194304] 0 2024-04-20T21:32:04.284 INFO:tasks.workunit.client.1.smithi189.stdout:5/163: dwrite d7/d19/f2d [0,4194304] 0 2024-04-20T21:32:04.284 INFO:tasks.workunit.client.0.smithi062.stdout:9/586: dread d16/d4e/fa3 [0,4194304] 0 2024-04-20T21:32:04.285 INFO:tasks.workunit.client.1.smithi189.stdout:2/189: write d1/f4 [4660180,110531] 0 2024-04-20T21:32:04.287 INFO:tasks.workunit.client.1.smithi189.stdout:0/306: creat d3/df/d30/d37/f65 x:0 0 0 2024-04-20T21:32:04.288 INFO:tasks.workunit.client.0.smithi062.stdout:4/598: symlink da/d56/d71/ld7 0 2024-04-20T21:32:04.291 INFO:tasks.workunit.client.1.smithi189.stdout:3/172: creat d2/d3/d2c/d31/f3c x:0 0 0 2024-04-20T21:32:04.292 INFO:tasks.workunit.client.1.smithi189.stdout:9/241: dwrite d9/dc/d17/d36/f43 [0,4194304] 0 2024-04-20T21:32:04.292 INFO:tasks.workunit.client.1.smithi189.stdout:7/210: truncate d5/f1a 785599 0 2024-04-20T21:32:04.293 INFO:tasks.workunit.client.0.smithi062.stdout:8/573: rename d7/d25/f7d to d7/d25/d3b/d58/d8b/d24/d46/fc2 0 2024-04-20T21:32:04.294 INFO:tasks.workunit.client.0.smithi062.stdout:4/599: write da/dd/f12 [953130,57083] 0 2024-04-20T21:32:04.300 INFO:tasks.workunit.client.1.smithi189.stdout:6/208: unlink d8/d12/f24 0 2024-04-20T21:32:04.302 INFO:tasks.workunit.client.1.smithi189.stdout:4/197: dwrite d2/f4 [4194304,4194304] 0 2024-04-20T21:32:04.302 INFO:tasks.workunit.client.1.smithi189.stdout:0/307: readlink d3/df/d28/l40 0 2024-04-20T21:32:04.304 INFO:tasks.workunit.client.1.smithi189.stdout:1/210: mkdir d8/df/d3a/d49 0 2024-04-20T21:32:04.304 INFO:tasks.workunit.client.1.smithi189.stdout:9/242: chown c2 362651762 1 2024-04-20T21:32:04.305 INFO:tasks.workunit.client.0.smithi062.stdout:7/595: write d7/d3e/d33/d3d/fca [463430,88385] 0 2024-04-20T21:32:04.307 INFO:tasks.workunit.client.0.smithi062.stdout:4/600: truncate da/dd/d15/d50/d64/d3d/f40 260591 0 2024-04-20T21:32:04.309 INFO:tasks.workunit.client.1.smithi189.stdout:0/308: chown d3/df/d28/l3f 14271 1 2024-04-20T21:32:04.309 INFO:tasks.workunit.client.1.smithi189.stdout:9/243: readlink d9/d12/l13 0 2024-04-20T21:32:04.310 INFO:tasks.workunit.client.1.smithi189.stdout:4/198: stat d2/dd/d11/f4b 0 2024-04-20T21:32:04.313 INFO:tasks.workunit.client.1.smithi189.stdout:1/211: truncate d8/df/f13 686575 0 2024-04-20T21:32:04.315 INFO:tasks.workunit.client.1.smithi189.stdout:8/325: dread da/de/d1e/d21/d44/f43 [0,4194304] 0 2024-04-20T21:32:04.320 INFO:tasks.workunit.client.0.smithi062.stdout:7/596: mkdir d7/dcd 0 2024-04-20T21:32:04.320 INFO:tasks.workunit.client.0.smithi062.stdout:8/574: dwrite d7/d25/d4f/fae [0,4194304] 0 2024-04-20T21:32:04.321 INFO:tasks.workunit.client.0.smithi062.stdout:4/601: unlink da/dd/df/la9 0 2024-04-20T21:32:04.321 INFO:tasks.workunit.client.1.smithi189.stdout:5/164: rename d7/d19 to d7/d9/d30 0 2024-04-20T21:32:04.323 INFO:tasks.workunit.client.1.smithi189.stdout:5/165: stat d7/cf 0 2024-04-20T21:32:04.330 INFO:tasks.workunit.client.0.smithi062.stdout:8/575: fdatasync d7/d25/d3b/d58/d8b/f1f 0 2024-04-20T21:32:04.332 INFO:tasks.workunit.client.1.smithi189.stdout:3/173: symlink d2/d3/d36/l3d 0 2024-04-20T21:32:04.334 INFO:tasks.workunit.client.0.smithi062.stdout:1/610: dwrite dc/d15/d20/d28/f60 [0,4194304] 0 2024-04-20T21:32:04.334 INFO:tasks.workunit.client.0.smithi062.stdout:4/602: symlink da/dd/db8/ld8 0 2024-04-20T21:32:04.334 INFO:tasks.workunit.client.1.smithi189.stdout:6/209: dread - d8/d10/f3f zero size 2024-04-20T21:32:04.337 INFO:tasks.workunit.client.0.smithi062.stdout:5/566: sync 2024-04-20T21:32:04.339 INFO:tasks.workunit.client.1.smithi189.stdout:2/190: dwrite d1/f7 [4194304,4194304] 0 2024-04-20T21:32:04.343 INFO:tasks.workunit.client.1.smithi189.stdout:5/166: mkdir d7/d9/d31 0 2024-04-20T21:32:04.343 INFO:tasks.workunit.client.0.smithi062.stdout:1/611: symlink dc/d15/d20/d28/d83/dc0/lda 0 2024-04-20T21:32:04.344 INFO:tasks.workunit.client.1.smithi189.stdout:4/199: dwrite d2/dd/f2d [4194304,4194304] 0 2024-04-20T21:32:04.344 INFO:tasks.workunit.client.1.smithi189.stdout:7/211: creat d5/f49 x:0 0 0 2024-04-20T21:32:04.347 INFO:tasks.workunit.client.1.smithi189.stdout:3/174: creat d2/d3/f3e x:0 0 0 2024-04-20T21:32:04.349 INFO:tasks.workunit.client.1.smithi189.stdout:8/326: dwrite da/df/d3c/d50/f5e [0,4194304] 0 2024-04-20T21:32:04.354 INFO:tasks.workunit.client.0.smithi062.stdout:7/597: rename d7/d3e/d33/d3d/d4d/d83/d94 to d7/d3e/d33/d3d/d4d/dce 0 2024-04-20T21:32:04.359 INFO:tasks.workunit.client.1.smithi189.stdout:0/309: rename d3/d26/f63 to d3/f66 0 2024-04-20T21:32:04.359 INFO:tasks.workunit.client.1.smithi189.stdout:4/200: creat d2/d3a/f4d x:0 0 0 2024-04-20T21:32:04.359 INFO:tasks.workunit.client.0.smithi062.stdout:5/567: link d8/f4f d8/d10/d3e/fb3 0 2024-04-20T21:32:04.363 INFO:tasks.workunit.client.1.smithi189.stdout:8/327: read f3 [6561560,43871] 0 2024-04-20T21:32:04.364 INFO:tasks.workunit.client.0.smithi062.stdout:1/612: rmdir dc/d15/d75/d96/dd4/d3a 39 2024-04-20T21:32:04.364 INFO:tasks.workunit.client.0.smithi062.stdout:7/598: symlink d7/d26/da8/lcf 0 2024-04-20T21:32:04.367 INFO:tasks.workunit.client.1.smithi189.stdout:7/212: mkdir d5/d21/d4a 0 2024-04-20T21:32:04.370 INFO:tasks.workunit.client.1.smithi189.stdout:1/212: dread d8/d21/f22 [0,4194304] 0 2024-04-20T21:32:04.370 INFO:tasks.workunit.client.1.smithi189.stdout:7/213: write d5/f11 [1862301,127635] 0 2024-04-20T21:32:04.372 INFO:tasks.workunit.client.1.smithi189.stdout:3/175: truncate d2/fd 1024756 0 2024-04-20T21:32:04.374 INFO:tasks.workunit.client.1.smithi189.stdout:3/176: write d2/d8/f10 [5156236,34437] 0 2024-04-20T21:32:04.376 INFO:tasks.workunit.client.0.smithi062.stdout:9/587: dread d16/d2c/d5e/d75/d76/f81 [0,4194304] 0 2024-04-20T21:32:04.377 INFO:tasks.workunit.client.1.smithi189.stdout:4/201: mknod d2/dd/c4e 0 2024-04-20T21:32:04.381 INFO:tasks.workunit.client.0.smithi062.stdout:9/588: fsync d16/d3d/d55/fbf 0 2024-04-20T21:32:04.386 INFO:tasks.workunit.client.1.smithi189.stdout:9/244: rename d9/d12/d15/d1a/d47 to d9/d12/d50 0 2024-04-20T21:32:04.386 INFO:tasks.workunit.client.0.smithi062.stdout:5/568: link d8/d38/d7a/d4c/d7d/f98 d8/d38/d7a/d4c/fb4 0 2024-04-20T21:32:04.388 INFO:tasks.workunit.client.1.smithi189.stdout:2/191: symlink d1/d17/d2f/l30 0 2024-04-20T21:32:04.392 INFO:tasks.workunit.client.1.smithi189.stdout:3/177: mkdir d2/d3/d2c/d30/d3f 0 2024-04-20T21:32:04.395 INFO:tasks.workunit.client.0.smithi062.stdout:5/569: mknod d8/d1e/cb5 0 2024-04-20T21:32:04.396 INFO:tasks.workunit.client.1.smithi189.stdout:6/210: rename d8/da/dd/d35/f1c to d8/d10/f41 0 2024-04-20T21:32:04.397 INFO:tasks.workunit.client.1.smithi189.stdout:0/310: dwrite d3/f48 [4194304,4194304] 0 2024-04-20T21:32:04.398 INFO:tasks.workunit.client.1.smithi189.stdout:9/245: symlink d9/d12/d15/d1a/l51 0 2024-04-20T21:32:04.399 INFO:tasks.workunit.client.0.smithi062.stdout:1/613: dwrite dc/f53 [12582912,4194304] 0 2024-04-20T21:32:04.400 INFO:tasks.workunit.client.0.smithi062.stdout:9/589: truncate d16/d2c/d42/da6/da9/fc4 308315 0 2024-04-20T21:32:04.402 INFO:tasks.workunit.client.0.smithi062.stdout:9/590: chown fd 43 1 2024-04-20T21:32:04.404 INFO:tasks.workunit.client.0.smithi062.stdout:1/614: dread - dc/d70/fd5 zero size 2024-04-20T21:32:04.406 INFO:tasks.workunit.client.0.smithi062.stdout:9/591: fdatasync d16/d3e/f5f 0 2024-04-20T21:32:04.406 INFO:tasks.workunit.client.1.smithi189.stdout:5/167: getdents d7/d14/d1b/d1f 0 2024-04-20T21:32:04.409 INFO:tasks.workunit.client.0.smithi062.stdout:4/603: dwrite da/dd/d15/d50/d64/d57/f80 [4194304,4194304] 0 2024-04-20T21:32:04.413 INFO:tasks.workunit.client.1.smithi189.stdout:6/211: symlink d8/da/l42 0 2024-04-20T21:32:04.414 INFO:tasks.workunit.client.1.smithi189.stdout:2/192: link d1/f2a d1/d17/d2f/f31 0 2024-04-20T21:32:04.414 INFO:tasks.workunit.client.0.smithi062.stdout:5/570: readlink d8/d1e/d2c/d6c/l45 0 2024-04-20T21:32:04.418 INFO:tasks.workunit.client.0.smithi062.stdout:6/439: sync 2024-04-20T21:32:04.419 INFO:tasks.workunit.client.0.smithi062.stdout:8/576: dwrite d7/d25/d3b/d58/d8b/d24/f85 [0,4194304] 0 2024-04-20T21:32:04.421 INFO:tasks.workunit.client.0.smithi062.stdout:6/440: read - d0/d9/f81 zero size 2024-04-20T21:32:04.424 INFO:tasks.workunit.client.1.smithi189.stdout:9/246: stat d9/dc/d17/d36/l39 0 2024-04-20T21:32:04.427 INFO:tasks.workunit.client.1.smithi189.stdout:0/311: write d3/df/d28/d2b/d2c/f62 [67653,67750] 0 2024-04-20T21:32:04.428 INFO:tasks.workunit.client.0.smithi062.stdout:7/599: dread d7/d3e/d33/d3d/d5e/d72/f86 [0,4194304] 0 2024-04-20T21:32:04.430 INFO:tasks.workunit.client.1.smithi189.stdout:1/213: write d8/f40 [416306,95945] 0 2024-04-20T21:32:04.435 INFO:tasks.workunit.client.1.smithi189.stdout:6/212: symlink d8/da/dd/d2a/d3b/l43 0 2024-04-20T21:32:04.435 INFO:tasks.workunit.client.0.smithi062.stdout:8/577: creat d7/d30/fc3 x:0 0 0 2024-04-20T21:32:04.435 INFO:tasks.workunit.client.0.smithi062.stdout:6/441: symlink d0/d4f/l8a 0 2024-04-20T21:32:04.436 INFO:tasks.workunit.client.0.smithi062.stdout:1/615: truncate dc/d15/d75/d96/dd4/f5d 933665 0 2024-04-20T21:32:04.437 INFO:tasks.workunit.client.1.smithi189.stdout:4/202: dread d2/d20/f33 [0,4194304] 0 2024-04-20T21:32:04.438 INFO:tasks.workunit.client.1.smithi189.stdout:6/213: chown d8/d2b/f39 8 1 2024-04-20T21:32:04.439 INFO:tasks.workunit.client.1.smithi189.stdout:2/193: chown d1/f23 6641 1 2024-04-20T21:32:04.441 INFO:tasks.workunit.client.0.smithi062.stdout:7/600: creat d7/d26/da8/fd0 x:0 0 0 2024-04-20T21:32:04.444 INFO:tasks.workunit.client.0.smithi062.stdout:6/442: creat d0/d2c/f8b x:0 0 0 2024-04-20T21:32:04.446 INFO:tasks.workunit.client.0.smithi062.stdout:1/616: readlink dc/d15/d75/d94/d6f/lbf 0 2024-04-20T21:32:04.451 INFO:tasks.workunit.client.0.smithi062.stdout:9/592: dwrite d16/d2f/d5b/fa0 [0,4194304] 0 2024-04-20T21:32:04.453 INFO:tasks.workunit.client.1.smithi189.stdout:8/328: rename da/l29 to da/df/l71 0 2024-04-20T21:32:04.453 INFO:tasks.workunit.client.1.smithi189.stdout:9/247: dwrite d9/dc/d17/d36/f4a [0,4194304] 0 2024-04-20T21:32:04.454 INFO:tasks.workunit.client.1.smithi189.stdout:4/203: creat d2/d20/f4f x:0 0 0 2024-04-20T21:32:04.456 INFO:tasks.workunit.client.0.smithi062.stdout:8/578: creat d7/d25/d3b/d58/fc4 x:0 0 0 2024-04-20T21:32:04.457 INFO:tasks.workunit.client.1.smithi189.stdout:8/329: chown da/df 27560 1 2024-04-20T21:32:04.458 INFO:tasks.workunit.client.1.smithi189.stdout:2/194: link d1/f23 d1/d11/d2d/f32 0 2024-04-20T21:32:04.459 INFO:tasks.workunit.client.0.smithi062.stdout:0/548: sync 2024-04-20T21:32:04.459 INFO:tasks.workunit.client.0.smithi062.stdout:3/539: sync 2024-04-20T21:32:04.459 INFO:tasks.workunit.client.0.smithi062.stdout:2/589: sync 2024-04-20T21:32:04.460 INFO:tasks.workunit.client.1.smithi189.stdout:9/248: write d9/d12/d15/f26 [1424796,64370] 0 2024-04-20T21:32:04.461 INFO:tasks.workunit.client.1.smithi189.stdout:7/214: rename d5/d1d/d28/c3c to d5/d21/d35/c4b 0 2024-04-20T21:32:04.463 INFO:tasks.workunit.client.0.smithi062.stdout:1/617: rename dc/d15/d20/cbd to dc/d70/d7f/cdb 0 2024-04-20T21:32:04.464 INFO:tasks.workunit.client.1.smithi189.stdout:0/312: creat d3/df/d28/d2b/f67 x:0 0 0 2024-04-20T21:32:04.466 INFO:tasks.workunit.client.0.smithi062.stdout:4/604: truncate da/dd/f19 951275 0 2024-04-20T21:32:04.466 INFO:tasks.workunit.client.1.smithi189.stdout:4/204: creat d2/dd/d11/d14/f50 x:0 0 0 2024-04-20T21:32:04.475 INFO:tasks.workunit.client.0.smithi062.stdout:8/579: write d7/d9/d29/f4b [569272,84139] 0 2024-04-20T21:32:04.477 INFO:tasks.workunit.client.1.smithi189.stdout:9/249: mkdir d9/dc/d17/d2b/d52 0 2024-04-20T21:32:04.480 INFO:tasks.workunit.client.0.smithi062.stdout:5/571: dwrite d8/d38/f70 [0,4194304] 0 2024-04-20T21:32:04.481 INFO:tasks.workunit.client.1.smithi189.stdout:3/178: dread d2/fd [0,4194304] 0 2024-04-20T21:32:04.481 INFO:tasks.workunit.client.1.smithi189.stdout:7/215: dwrite d5/d1d/d42/d45/f48 [0,4194304] 0 2024-04-20T21:32:04.481 INFO:tasks.workunit.client.0.smithi062.stdout:9/593: rename d16/d3d/d55/d6e/l9f to d16/d2f/d5b/lc6 0 2024-04-20T21:32:04.485 INFO:tasks.workunit.client.0.smithi062.stdout:2/590: write d7/d39/f8e [568941,57459] 0 2024-04-20T21:32:04.485 INFO:tasks.workunit.client.1.smithi189.stdout:0/313: creat d3/df/d30/d45/d60/f68 x:0 0 0 2024-04-20T21:32:04.487 INFO:tasks.workunit.client.0.smithi062.stdout:3/540: mkdir d0/d1/db7 0 2024-04-20T21:32:04.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:04 smithi189 ceph-mon[21562]: Updating smithi062:/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-20T21:32:04.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:04 smithi189 ceph-mon[21562]: Updating smithi189:/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-20T21:32:04.488 INFO:tasks.workunit.client.0.smithi062.stdout:1/618: symlink dc/ldc 0 2024-04-20T21:32:04.488 INFO:tasks.workunit.client.0.smithi062.stdout:5/572: dread - d8/d38/d7a/d4c/fb4 zero size 2024-04-20T21:32:04.491 INFO:tasks.workunit.client.1.smithi189.stdout:1/214: rename d8/d1e/l2a to d8/d21/d2c/l4a 0 2024-04-20T21:32:04.492 INFO:tasks.workunit.client.0.smithi062.stdout:7/601: dread d7/d63/f7a [0,4194304] 0 2024-04-20T21:32:04.492 INFO:tasks.workunit.client.0.smithi062.stdout:6/443: dwrite d0/d4f/f73 [0,4194304] 0 2024-04-20T21:32:04.494 INFO:tasks.workunit.client.1.smithi189.stdout:9/250: mknod d9/d12/c53 0 2024-04-20T21:32:04.495 INFO:tasks.workunit.client.1.smithi189.stdout:7/216: readlink d5/l16 0 2024-04-20T21:32:04.496 INFO:tasks.workunit.client.1.smithi189.stdout:5/168: dwrite d7/d14/f26 [0,4194304] 0 2024-04-20T21:32:04.497 INFO:tasks.workunit.client.0.smithi062.stdout:7/602: fsync d7/d3e/f99 0 2024-04-20T21:32:04.497 INFO:tasks.workunit.client.1.smithi189.stdout:2/195: creat d1/f33 x:0 0 0 2024-04-20T21:32:04.498 INFO:tasks.workunit.client.0.smithi062.stdout:6/444: chown d0/d9/f71 23669884 1 2024-04-20T21:32:04.503 INFO:tasks.workunit.client.1.smithi189.stdout:1/215: symlink d8/d1e/d43/d44/l4b 0 2024-04-20T21:32:04.509 INFO:tasks.workunit.client.1.smithi189.stdout:5/169: symlink d7/d9/d30/l32 0 2024-04-20T21:32:04.513 INFO:tasks.workunit.client.0.smithi062.stdout:2/591: rmdir d7/da/d15/d2f/d93 39 2024-04-20T21:32:04.515 INFO:tasks.workunit.client.0.smithi062.stdout:8/580: mkdir d7/d25/dc5 0 2024-04-20T21:32:04.517 INFO:tasks.workunit.client.1.smithi189.stdout:9/251: symlink d9/d12/d50/l54 0 2024-04-20T21:32:04.518 INFO:tasks.workunit.client.1.smithi189.stdout:5/170: read d7/fb [175019,125700] 0 2024-04-20T21:32:04.521 INFO:tasks.workunit.client.1.smithi189.stdout:2/196: rename d1/f4 to d1/d17/f34 0 2024-04-20T21:32:04.522 INFO:tasks.workunit.client.0.smithi062.stdout:7/603: dread - d7/d3e/f6a zero size 2024-04-20T21:32:04.522 INFO:tasks.workunit.client.0.smithi062.stdout:1/619: chown dc/d15/d75/d96/dd4/d3a/f59 5 1 2024-04-20T21:32:04.522 INFO:tasks.workunit.client.0.smithi062.stdout:6/445: mkdir d0/d2c/d8c 0 2024-04-20T21:32:04.522 INFO:tasks.workunit.client.0.smithi062.stdout:2/592: read - d7/da/d15/d2f/d38/d6b/fb1 zero size 2024-04-20T21:32:04.524 INFO:tasks.workunit.client.0.smithi062.stdout:1/620: dread - dc/d15/d20/d28/dc9/fd3 zero size 2024-04-20T21:32:04.524 INFO:tasks.workunit.client.1.smithi189.stdout:3/179: dwrite d2/d1c/f2b [4194304,4194304] 0 2024-04-20T21:32:04.526 INFO:tasks.workunit.client.0.smithi062.stdout:2/593: stat d7/da/d15/d2f/d38/db0 0 2024-04-20T21:32:04.530 INFO:tasks.workunit.client.1.smithi189.stdout:2/197: rename d1/f33 to d1/d17/f35 0 2024-04-20T21:32:04.530 INFO:tasks.workunit.client.1.smithi189.stdout:0/314: dwrite d3/f27 [0,4194304] 0 2024-04-20T21:32:04.531 INFO:tasks.workunit.client.1.smithi189.stdout:5/171: unlink d7/d9/fc 0 2024-04-20T21:32:04.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:04 smithi062 ceph-mon[16221]: Updating smithi062:/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-20T21:32:04.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:04 smithi062 ceph-mon[16221]: Updating smithi189:/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-20T21:32:04.538 INFO:tasks.workunit.client.1.smithi189.stdout:9/252: mkdir d9/dc/d17/d2b/d55 0 2024-04-20T21:32:04.539 INFO:tasks.workunit.client.1.smithi189.stdout:7/217: dwrite d5/f24 [4194304,4194304] 0 2024-04-20T21:32:04.542 INFO:tasks.workunit.client.1.smithi189.stdout:8/330: truncate f3 6989300 0 2024-04-20T21:32:04.543 INFO:tasks.workunit.client.0.smithi062.stdout:3/541: getdents d0/d1/db7 0 2024-04-20T21:32:04.547 INFO:tasks.workunit.client.0.smithi062.stdout:8/581: creat d7/d30/d90/fc6 x:0 0 0 2024-04-20T21:32:04.547 INFO:tasks.workunit.client.0.smithi062.stdout:5/573: mknod d8/d38/d7a/d4c/d7d/d9d/dac/d2f/d8e/cb6 0 2024-04-20T21:32:04.551 INFO:tasks.workunit.client.1.smithi189.stdout:4/205: dread d2/dd/fe [0,4194304] 0 2024-04-20T21:32:04.553 INFO:tasks.workunit.client.0.smithi062.stdout:2/594: mknod d7/da/d28/cc2 0 2024-04-20T21:32:04.554 INFO:tasks.workunit.client.1.smithi189.stdout:3/180: dwrite d2/d3/f19 [0,4194304] 0 2024-04-20T21:32:04.555 INFO:tasks.workunit.client.1.smithi189.stdout:7/218: unlink d5/f33 0 2024-04-20T21:32:04.557 INFO:tasks.workunit.client.1.smithi189.stdout:8/331: dread da/df/d3c/d50/f61 [0,4194304] 0 2024-04-20T21:32:04.558 INFO:tasks.workunit.client.0.smithi062.stdout:0/549: dwrite d7/d1e/db6/fa3 [0,4194304] 0 2024-04-20T21:32:04.558 INFO:tasks.workunit.client.0.smithi062.stdout:5/574: readlink d8/d1e/d2c/d6c/l5c 0 2024-04-20T21:32:04.562 INFO:tasks.workunit.client.1.smithi189.stdout:9/253: read d9/dc/f1c [715307,57623] 0 2024-04-20T21:32:04.566 INFO:tasks.workunit.client.0.smithi062.stdout:6/446: creat d0/d9/d11/f8d x:0 0 0 2024-04-20T21:32:04.567 INFO:tasks.workunit.client.0.smithi062.stdout:9/594: getdents d16/d2f/d5b/dab 0 2024-04-20T21:32:04.568 INFO:tasks.workunit.client.0.smithi062.stdout:5/575: chown d8/d38/f70 126818314 1 2024-04-20T21:32:04.570 INFO:tasks.workunit.client.1.smithi189.stdout:4/206: symlink d2/dd/d11/d14/l51 0 2024-04-20T21:32:04.571 INFO:tasks.workunit.client.0.smithi062.stdout:1/621: creat dc/d15/fdd x:0 0 0 2024-04-20T21:32:04.575 INFO:tasks.workunit.client.1.smithi189.stdout:7/219: dwrite d5/f1a [0,4194304] 0 2024-04-20T21:32:04.578 INFO:tasks.workunit.client.1.smithi189.stdout:7/220: truncate d5/f43 1008882 0 2024-04-20T21:32:04.581 INFO:tasks.workunit.client.1.smithi189.stdout:3/181: creat d2/d3/d25/f40 x:0 0 0 2024-04-20T21:32:04.581 INFO:tasks.workunit.client.0.smithi062.stdout:3/542: dwrite d0/d1b/f35 [0,4194304] 0 2024-04-20T21:32:04.588 INFO:tasks.workunit.client.0.smithi062.stdout:2/595: symlink d7/da/d15/d2f/d38/d51/dbc/lc3 0 2024-04-20T21:32:04.588 INFO:tasks.workunit.client.1.smithi189.stdout:3/182: write d2/f29 [436256,23918] 0 2024-04-20T21:32:04.589 INFO:tasks.workunit.client.0.smithi062.stdout:6/447: truncate d0/fe 6196216 0 2024-04-20T21:32:04.589 INFO:tasks.workunit.client.0.smithi062.stdout:9/595: chown d16/d3d/d55/d6e/d7d/lba 17648 1 2024-04-20T21:32:04.591 INFO:tasks.workunit.client.1.smithi189.stdout:0/315: creat d3/df/d4c/f69 x:0 0 0 2024-04-20T21:32:04.591 INFO:tasks.workunit.client.0.smithi062.stdout:3/543: read - d0/d1/d10/d31/d41/f7c zero size 2024-04-20T21:32:04.592 INFO:tasks.workunit.client.0.smithi062.stdout:9/596: stat d16/d3d 0 2024-04-20T21:32:04.592 INFO:tasks.workunit.client.0.smithi062.stdout:0/550: truncate d7/d1c/d15/d17/d1b/d74/f99 4530031 0 2024-04-20T21:32:04.594 INFO:tasks.workunit.client.1.smithi189.stdout:0/316: fsync d3/df/f3a 0 2024-04-20T21:32:04.595 INFO:tasks.workunit.client.1.smithi189.stdout:0/317: readlink d3/df/d28/d2b/l57 0 2024-04-20T21:32:04.600 INFO:tasks.workunit.client.1.smithi189.stdout:0/318: fdatasync d3/df/d28/d2b/d2c/f62 0 2024-04-20T21:32:04.601 INFO:tasks.workunit.client.0.smithi062.stdout:3/544: write d0/d1c/d52/d7b/f68 [3561833,25636] 0 2024-04-20T21:32:04.601 INFO:tasks.workunit.client.0.smithi062.stdout:7/604: dread d7/ff [0,4194304] 0 2024-04-20T21:32:04.604 INFO:tasks.workunit.client.0.smithi062.stdout:1/622: mkdir dc/d15/d75/d94/d6f/dd2/dde 0 2024-04-20T21:32:04.604 INFO:tasks.workunit.client.0.smithi062.stdout:5/576: mknod d8/d38/d7a/cb7 0 2024-04-20T21:32:04.605 INFO:tasks.workunit.client.0.smithi062.stdout:2/596: dread - d7/da/d15/d2f/d38/d6b/fa3 zero size 2024-04-20T21:32:04.613 INFO:tasks.workunit.client.0.smithi062.stdout:3/545: read d0/d1b/f2b [743374,121791] 0 2024-04-20T21:32:04.614 INFO:tasks.workunit.client.0.smithi062.stdout:5/577: rmdir d8 39 2024-04-20T21:32:04.617 INFO:tasks.workunit.client.1.smithi189.stdout:0/319: rmdir d3/df/d30/d45 39 2024-04-20T21:32:04.621 INFO:tasks.workunit.client.1.smithi189.stdout:7/221: creat d5/d46/f4c x:0 0 0 2024-04-20T21:32:04.622 INFO:tasks.workunit.client.0.smithi062.stdout:8/582: dwrite d7/d9/d29/f40 [0,4194304] 0 2024-04-20T21:32:04.624 INFO:tasks.workunit.client.0.smithi062.stdout:7/605: rename d7/l11 to d7/d3e/d33/d3d/d5e/d7e/ld1 0 2024-04-20T21:32:04.626 INFO:tasks.workunit.client.0.smithi062.stdout:1/623: creat dc/d15/d75/d94/d6f/dd2/fdf x:0 0 0 2024-04-20T21:32:04.626 INFO:tasks.workunit.client.1.smithi189.stdout:4/207: truncate d2/fb 4364578 0 2024-04-20T21:32:04.628 INFO:tasks.workunit.client.0.smithi062.stdout:3/546: unlink d0/d1c/d63/l9c 0 2024-04-20T21:32:04.636 INFO:tasks.workunit.client.0.smithi062.stdout:9/597: dwrite d16/f3a [0,4194304] 0 2024-04-20T21:32:04.640 INFO:tasks.workunit.client.1.smithi189.stdout:0/320: dwrite d3/f5e [0,4194304] 0 2024-04-20T21:32:04.642 INFO:tasks.workunit.client.0.smithi062.stdout:8/583: mkdir d7/d25/d3b/d58/dac/dc7 0 2024-04-20T21:32:04.643 INFO:tasks.workunit.client.1.smithi189.stdout:4/208: symlink d2/d22/l52 0 2024-04-20T21:32:04.644 INFO:tasks.workunit.client.0.smithi062.stdout:3/547: getdents d0/d1/d10/d3a 0 2024-04-20T21:32:04.644 INFO:tasks.workunit.client.1.smithi189.stdout:8/332: getdents da/df/d20/d27 0 2024-04-20T21:32:04.646 INFO:tasks.workunit.client.1.smithi189.stdout:4/209: readlink d2/dd/l2e 0 2024-04-20T21:32:04.647 INFO:tasks.workunit.client.0.smithi062.stdout:9/598: read d16/f2e [1081410,105832] 0 2024-04-20T21:32:04.650 INFO:tasks.workunit.client.1.smithi189.stdout:2/198: dwrite d1/f9 [4194304,4194304] 0 2024-04-20T21:32:04.650 INFO:tasks.workunit.client.1.smithi189.stdout:3/183: link d2/d3/d36/l3d d2/d3/d2c/l41 0 2024-04-20T21:32:04.653 INFO:tasks.workunit.client.0.smithi062.stdout:4/605: sync 2024-04-20T21:32:04.653 INFO:tasks.workunit.client.1.smithi189.stdout:2/199: chown d1/d11/l19 428499 1 2024-04-20T21:32:04.655 INFO:tasks.workunit.client.1.smithi189.stdout:9/254: getdents d9 0 2024-04-20T21:32:04.659 INFO:tasks.workunit.client.1.smithi189.stdout:7/222: creat d5/d21/d4a/f4d x:0 0 0 2024-04-20T21:32:04.659 INFO:tasks.workunit.client.1.smithi189.stdout:9/255: fsync d9/d12/d3b/f4c 0 2024-04-20T21:32:04.662 INFO:tasks.workunit.client.1.smithi189.stdout:0/321: creat d3/df/d28/f6a x:0 0 0 2024-04-20T21:32:04.665 INFO:tasks.workunit.client.1.smithi189.stdout:7/223: dread - d5/d21/d25/f30 zero size 2024-04-20T21:32:04.667 INFO:tasks.workunit.client.1.smithi189.stdout:4/210: readlink d2/dd/d11/l25 0 2024-04-20T21:32:04.668 INFO:tasks.workunit.client.1.smithi189.stdout:8/333: creat da/de/d1e/d69/d70/f72 x:0 0 0 2024-04-20T21:32:04.671 INFO:tasks.workunit.client.1.smithi189.stdout:8/334: chown da/de/d1e/d21/d44/l60 4 1 2024-04-20T21:32:04.674 INFO:tasks.workunit.client.0.smithi062.stdout:2/597: dwrite d7/da/d15/d1b/d3d/f78 [0,4194304] 0 2024-04-20T21:32:04.676 INFO:tasks.workunit.client.0.smithi062.stdout:0/551: dwrite d7/d1c/d15/d17/d37/f84 [0,4194304] 0 2024-04-20T21:32:04.679 INFO:tasks.workunit.client.0.smithi062.stdout:6/448: dread d0/d2c/f30 [4194304,4194304] 0 2024-04-20T21:32:04.680 INFO:tasks.workunit.client.1.smithi189.stdout:9/256: mkdir d9/dc/d17/d2b/d56 0 2024-04-20T21:32:04.684 INFO:tasks.workunit.client.1.smithi189.stdout:7/224: unlink d5/d1d/f2b 0 2024-04-20T21:32:04.685 INFO:tasks.workunit.client.1.smithi189.stdout:7/225: write d5/f6 [7115863,88815] 0 2024-04-20T21:32:04.689 INFO:tasks.workunit.client.1.smithi189.stdout:3/184: mkdir d2/d3/d2c/d30/d3f/d42 0 2024-04-20T21:32:04.691 INFO:tasks.workunit.client.1.smithi189.stdout:9/257: creat d9/dc/d17/d36/f57 x:0 0 0 2024-04-20T21:32:04.692 INFO:tasks.workunit.client.0.smithi062.stdout:1/624: dread dc/d15/d20/d28/f63 [0,4194304] 0 2024-04-20T21:32:04.692 INFO:tasks.workunit.client.1.smithi189.stdout:0/322: creat d3/df/d5d/d64/f6b x:0 0 0 2024-04-20T21:32:04.693 INFO:tasks.workunit.client.0.smithi062.stdout:9/599: dwrite d16/d2c/d5e/d75/d76/f81 [0,4194304] 0 2024-04-20T21:32:04.694 INFO:tasks.workunit.client.1.smithi189.stdout:9/258: write d9/dc/d17/d36/f57 [208678,72708] 0 2024-04-20T21:32:04.695 INFO:tasks.workunit.client.0.smithi062.stdout:1/625: chown dc/d15/d75/d94/d6f 1002926 1 2024-04-20T21:32:04.696 INFO:tasks.workunit.client.0.smithi062.stdout:9/600: chown d16/d4e/f6f 101589569 1 2024-04-20T21:32:04.698 INFO:tasks.workunit.client.1.smithi189.stdout:4/211: mknod d2/c53 0 2024-04-20T21:32:04.698 INFO:tasks.workunit.client.1.smithi189.stdout:4/212: stat d2/dd/d37 0 2024-04-20T21:32:04.703 INFO:tasks.workunit.client.1.smithi189.stdout:8/335: creat da/df/d13/d63/f73 x:0 0 0 2024-04-20T21:32:04.704 INFO:tasks.workunit.client.0.smithi062.stdout:9/601: dread d16/d2c/d62/f9c [0,4194304] 0 2024-04-20T21:32:04.706 INFO:tasks.workunit.client.1.smithi189.stdout:2/200: link d1/d11/d26/l29 d1/d11/d2d/l36 0 2024-04-20T21:32:04.709 INFO:tasks.workunit.client.1.smithi189.stdout:3/185: chown d2/d3/d2c/l41 4898379 1 2024-04-20T21:32:04.710 INFO:tasks.workunit.client.0.smithi062.stdout:9/602: write d16/d2c/d42/da6/da9/fc4 [1275426,1142] 0 2024-04-20T21:32:04.710 INFO:tasks.workunit.client.1.smithi189.stdout:7/226: mknod d5/c4e 0 2024-04-20T21:32:04.712 INFO:tasks.workunit.client.0.smithi062.stdout:9/603: readlink d16/d2c/d42/l93 0 2024-04-20T21:32:04.714 INFO:tasks.workunit.client.0.smithi062.stdout:9/604: readlink d16/d2c/d42/l95 0 2024-04-20T21:32:04.717 INFO:tasks.workunit.client.1.smithi189.stdout:7/227: truncate d5/d21/d25/f30 451385 0 2024-04-20T21:32:04.718 INFO:tasks.workunit.client.1.smithi189.stdout:9/259: mknod d9/dc/d17/d2b/d52/c58 0 2024-04-20T21:32:04.719 INFO:tasks.workunit.client.1.smithi189.stdout:3/186: mknod d2/d3/d2c/c43 0 2024-04-20T21:32:04.719 INFO:tasks.workunit.client.1.smithi189.stdout:0/323: creat d3/df/d30/d45/f6c x:0 0 0 2024-04-20T21:32:04.724 INFO:tasks.workunit.client.1.smithi189.stdout:7/228: write d5/d1d/d28/f38 [8179456,56276] 0 2024-04-20T21:32:04.731 INFO:tasks.workunit.client.1.smithi189.stdout:8/336: symlink da/df/l74 0 2024-04-20T21:32:04.738 INFO:tasks.workunit.client.1.smithi189.stdout:7/229: unlink d5/f1e 0 2024-04-20T21:32:04.740 INFO:tasks.workunit.client.1.smithi189.stdout:0/324: creat d3/df/d5d/d64/f6d x:0 0 0 2024-04-20T21:32:04.747 INFO:tasks.workunit.client.1.smithi189.stdout:7/230: symlink d5/d21/d35/l4f 0 2024-04-20T21:32:04.748 INFO:tasks.workunit.client.1.smithi189.stdout:7/231: fsync d5/f7 0 2024-04-20T21:32:04.748 INFO:tasks.workunit.client.0.smithi062.stdout:4/606: creat da/dd/d15/d4b/fd9 x:0 0 0 2024-04-20T21:32:04.749 INFO:tasks.workunit.client.0.smithi062.stdout:7/606: write d7/d26/f29 [675485,92920] 0 2024-04-20T21:32:04.750 INFO:tasks.workunit.client.0.smithi062.stdout:7/607: chown d7/d26 4574 1 2024-04-20T21:32:04.751 INFO:tasks.workunit.client.1.smithi189.stdout:8/337: creat da/df/d20/d27/f75 x:0 0 0 2024-04-20T21:32:04.751 INFO:tasks.workunit.client.0.smithi062.stdout:4/607: write da/dd/d15/fa7 [2257545,51541] 0 2024-04-20T21:32:04.752 INFO:tasks.workunit.client.0.smithi062.stdout:2/598: unlink d7/da/d15/d2f/d97/fbb 0 2024-04-20T21:32:04.755 INFO:tasks.workunit.client.0.smithi062.stdout:4/608: readlink da/dd/d15/d4b/l75 0 2024-04-20T21:32:04.757 INFO:tasks.workunit.client.0.smithi062.stdout:0/552: symlink d7/d1c/d15/d17/d1b/d74/lb8 0 2024-04-20T21:32:04.758 INFO:tasks.workunit.client.1.smithi189.stdout:7/232: write d5/d1d/d42/d45/f48 [3686944,7721] 0 2024-04-20T21:32:04.758 INFO:tasks.workunit.client.1.smithi189.stdout:3/187: dwrite d2/f15 [4194304,4194304] 0 2024-04-20T21:32:04.761 INFO:tasks.workunit.client.0.smithi062.stdout:0/553: truncate d7/d1c/d15/d17/d26/f90 567584 0 2024-04-20T21:32:04.765 INFO:tasks.workunit.client.1.smithi189.stdout:7/233: chown d5/d46/f4c 369131 1 2024-04-20T21:32:04.766 INFO:tasks.workunit.client.1.smithi189.stdout:2/201: dwrite d1/f12 [0,4194304] 0 2024-04-20T21:32:04.769 INFO:tasks.workunit.client.0.smithi062.stdout:9/605: mknod d16/d2f/cc7 0 2024-04-20T21:32:04.771 INFO:tasks.workunit.client.0.smithi062.stdout:9/606: truncate d16/d4e/d80/fc0 650191 0 2024-04-20T21:32:04.772 INFO:tasks.workunit.client.0.smithi062.stdout:5/578: link d8/d38/d7a/d4c/d7d/d9d/dac/d66/dae/faf d8/d38/d7a/d4c/d7d/d9d/fb8 0 2024-04-20T21:32:04.777 INFO:tasks.workunit.client.1.smithi189.stdout:3/188: mknod d2/d3/d25/c44 0 2024-04-20T21:32:04.778 INFO:tasks.workunit.client.0.smithi062.stdout:8/584: unlink d7/d25/d3b/l83 0 2024-04-20T21:32:04.778 INFO:tasks.workunit.client.1.smithi189.stdout:2/202: mkdir d1/d17/d37 0 2024-04-20T21:32:04.780 INFO:tasks.workunit.client.1.smithi189.stdout:4/213: dread d2/f10 [0,4194304] 0 2024-04-20T21:32:04.780 INFO:tasks.workunit.client.1.smithi189.stdout:8/338: rename da/df/d20/d27/f75 to da/df/d20/d2c/f76 0 2024-04-20T21:32:04.781 INFO:tasks.workunit.client.0.smithi062.stdout:8/585: write d7/d25/d4f/fbc [403146,67267] 0 2024-04-20T21:32:04.788 INFO:tasks.workunit.client.1.smithi189.stdout:9/260: dwrite d9/dc/f3d [4194304,4194304] 0 2024-04-20T21:32:04.790 INFO:tasks.workunit.client.1.smithi189.stdout:3/189: creat d2/d8/f45 x:0 0 0 2024-04-20T21:32:04.790 INFO:tasks.workunit.client.1.smithi189.stdout:7/234: dread d5/fe [0,4194304] 0 2024-04-20T21:32:04.791 INFO:tasks.workunit.client.1.smithi189.stdout:0/325: dwrite d3/df/d4c/f69 [0,4194304] 0 2024-04-20T21:32:04.791 INFO:tasks.workunit.client.1.smithi189.stdout:8/339: rmdir da/df/d20/d2c 39 2024-04-20T21:32:04.795 INFO:tasks.workunit.client.0.smithi062.stdout:6/449: mknod d0/d9/c8e 0 2024-04-20T21:32:04.797 INFO:tasks.workunit.client.0.smithi062.stdout:6/450: dread - d0/d9/d11/d5c/f86 zero size 2024-04-20T21:32:04.798 INFO:tasks.workunit.client.0.smithi062.stdout:4/609: rename da/dd/d15/d4b/f96 to da/dd/d15/daf/fda 0 2024-04-20T21:32:04.801 INFO:tasks.workunit.client.1.smithi189.stdout:8/340: dread - da/df/d20/d27/f54 zero size 2024-04-20T21:32:04.801 INFO:tasks.workunit.client.1.smithi189.stdout:0/326: fdatasync d3/df/d30/d37/f65 0 2024-04-20T21:32:04.802 INFO:tasks.workunit.client.1.smithi189.stdout:7/235: truncate d5/ff 2860579 0 2024-04-20T21:32:04.802 INFO:tasks.workunit.client.0.smithi062.stdout:0/554: mknod d7/d1c/d15/d17/d1b/d9d/cb9 0 2024-04-20T21:32:04.804 INFO:tasks.workunit.client.1.smithi189.stdout:3/190: rename d2/d3/d2c/d31/c37 to d2/d3/c46 0 2024-04-20T21:32:04.804 INFO:tasks.workunit.client.0.smithi062.stdout:9/607: creat d16/d2f/fc8 x:0 0 0 2024-04-20T21:32:04.805 INFO:tasks.workunit.client.1.smithi189.stdout:7/236: fsync d5/ff 0 2024-04-20T21:32:04.809 INFO:tasks.workunit.client.1.smithi189.stdout:0/327: read - d3/df/d30/d37/f65 zero size 2024-04-20T21:32:04.809 INFO:tasks.workunit.client.1.smithi189.stdout:4/214: link d2/f18 d2/f54 0 2024-04-20T21:32:04.809 INFO:tasks.workunit.client.0.smithi062.stdout:7/608: creat d7/dcd/fd2 x:0 0 0 2024-04-20T21:32:04.812 INFO:tasks.workunit.client.0.smithi062.stdout:2/599: mkdir d7/da/d15/d2f/d38/db0/dc4 0 2024-04-20T21:32:04.815 INFO:tasks.workunit.client.0.smithi062.stdout:5/579: dread d8/d10/f74 [0,4194304] 0 2024-04-20T21:32:04.817 INFO:tasks.workunit.client.1.smithi189.stdout:4/215: fdatasync d2/f1a 0 2024-04-20T21:32:04.817 INFO:tasks.workunit.client.1.smithi189.stdout:3/191: symlink d2/d3/d2c/l47 0 2024-04-20T21:32:04.817 INFO:tasks.workunit.client.0.smithi062.stdout:5/580: stat d8/d38/c7c 0 2024-04-20T21:32:04.818 INFO:tasks.workunit.client.0.smithi062.stdout:1/626: rmdir dc/d15/d75/d94/d6f/dd2/dde 0 2024-04-20T21:32:04.820 INFO:tasks.workunit.client.1.smithi189.stdout:7/237: mkdir d5/d1d/d42/d50 0 2024-04-20T21:32:04.821 INFO:tasks.workunit.client.1.smithi189.stdout:9/261: dread d9/dc/f1c [0,4194304] 0 2024-04-20T21:32:04.828 INFO:tasks.workunit.client.1.smithi189.stdout:8/341: write da/df/d3c/d50/f5e [9232354,4719] 0 2024-04-20T21:32:04.828 INFO:tasks.workunit.client.1.smithi189.stdout:7/238: read - d5/f47 zero size 2024-04-20T21:32:04.829 INFO:tasks.workunit.client.1.smithi189.stdout:4/216: stat d2/dd/d11/d14/c15 0 2024-04-20T21:32:04.829 INFO:tasks.workunit.client.0.smithi062.stdout:7/609: chown d7/d26/l91 20158 1 2024-04-20T21:32:04.830 INFO:tasks.workunit.client.1.smithi189.stdout:0/328: fsync d3/df/d28/d2b/f67 0 2024-04-20T21:32:04.830 INFO:tasks.workunit.client.0.smithi062.stdout:2/600: creat d7/da/d15/d2f/d38/d51/fc5 x:0 0 0 2024-04-20T21:32:04.830 INFO:tasks.workunit.client.0.smithi062.stdout:0/555: truncate d7/d1c/f41 2254188 0 2024-04-20T21:32:04.831 INFO:tasks.workunit.client.1.smithi189.stdout:2/203: dwrite d1/f9 [8388608,4194304] 0 2024-04-20T21:32:04.835 INFO:tasks.workunit.client.0.smithi062.stdout:9/608: dwrite d16/d3e/d77/d84/f9a [0,4194304] 0 2024-04-20T21:32:04.838 INFO:tasks.workunit.client.0.smithi062.stdout:2/601: chown d7/da/d15/d1b/l5e 40055625 1 2024-04-20T21:32:04.840 INFO:tasks.workunit.client.0.smithi062.stdout:1/627: dread f1 [0,4194304] 0 2024-04-20T21:32:04.843 INFO:tasks.workunit.client.0.smithi062.stdout:0/556: creat d7/d1c/d15/d17/d1b/d9d/fba x:0 0 0 2024-04-20T21:32:04.851 INFO:tasks.workunit.client.0.smithi062.stdout:7/610: creat d7/d3e/d33/d3d/d5e/d72/fd3 x:0 0 0 2024-04-20T21:32:04.854 INFO:tasks.workunit.client.1.smithi189.stdout:3/192: rmdir d2/d3/d2c/d30/d3f/d42 0 2024-04-20T21:32:04.854 INFO:tasks.workunit.client.1.smithi189.stdout:7/239: link d5/c4e d5/d46/c51 0 2024-04-20T21:32:04.855 INFO:tasks.workunit.client.1.smithi189.stdout:3/193: stat d2/d3/d2c 0 2024-04-20T21:32:04.856 INFO:tasks.workunit.client.0.smithi062.stdout:9/609: stat d16/c60 0 2024-04-20T21:32:04.861 INFO:tasks.workunit.client.0.smithi062.stdout:2/602: symlink d7/da/d15/d2f/d38/db0/lc6 0 2024-04-20T21:32:04.863 INFO:tasks.workunit.client.0.smithi062.stdout:1/628: rename dc/d27/l71 to dc/d15/d20/d28/dc9/d84/dbe/le0 0 2024-04-20T21:32:04.864 INFO:tasks.workunit.client.1.smithi189.stdout:0/329: write d3/fe [722908,110307] 0 2024-04-20T21:32:04.868 INFO:tasks.workunit.client.0.smithi062.stdout:0/557: chown d7/d1c/d15/d17/d37/f61 251266702 1 2024-04-20T21:32:04.870 INFO:tasks.workunit.client.1.smithi189.stdout:7/240: write d5/d21/f23 [1026201,21212] 0 2024-04-20T21:32:04.874 INFO:tasks.workunit.client.1.smithi189.stdout:3/194: write d2/d1c/f2b [206649,109162] 0 2024-04-20T21:32:04.877 INFO:tasks.workunit.client.1.smithi189.stdout:7/241: dread - d5/d46/f4c zero size 2024-04-20T21:32:04.883 INFO:tasks.workunit.client.1.smithi189.stdout:0/330: mknod d3/df/d28/d2b/d2c/c6e 0 2024-04-20T21:32:04.883 INFO:tasks.workunit.client.0.smithi062.stdout:1/629: rmdir dc/d15/d75/d96/dd4/d78 39 2024-04-20T21:32:04.883 INFO:tasks.workunit.client.1.smithi189.stdout:8/342: dwrite da/df/f11 [0,4194304] 0 2024-04-20T21:32:04.885 INFO:tasks.workunit.client.0.smithi062.stdout:5/581: getdents d8/d38/d7a/d4c/d7d/d9d/dac/d66/dae 0 2024-04-20T21:32:04.886 INFO:tasks.workunit.client.0.smithi062.stdout:0/558: creat d7/d1c/d15/d53/fbb x:0 0 0 2024-04-20T21:32:04.886 INFO:tasks.workunit.client.1.smithi189.stdout:0/331: read - d3/d26/f2f zero size 2024-04-20T21:32:04.890 INFO:tasks.workunit.client.0.smithi062.stdout:5/582: chown d8/d10/l63 12 1 2024-04-20T21:32:04.891 INFO:tasks.workunit.client.0.smithi062.stdout:6/451: link d0/d1a/d7d/c50 d0/d2c/c8f 0 2024-04-20T21:32:04.891 INFO:tasks.workunit.client.1.smithi189.stdout:3/195: link d2/d8/f22 d2/d3/f48 0 2024-04-20T21:32:04.892 INFO:tasks.workunit.client.1.smithi189.stdout:4/217: dwrite d2/f18 [0,4194304] 0 2024-04-20T21:32:04.895 INFO:tasks.workunit.client.1.smithi189.stdout:7/242: rename d5/d1d/d42/d50 to d5/d21/d4a/d52 0 2024-04-20T21:32:04.896 INFO:tasks.workunit.client.1.smithi189.stdout:8/343: write da/de/f28 [1656849,52492] 0 2024-04-20T21:32:04.898 INFO:tasks.workunit.client.0.smithi062.stdout:2/603: creat d7/da/d15/d2f/d38/db0/dc4/fc7 x:0 0 0 2024-04-20T21:32:04.899 INFO:tasks.workunit.client.1.smithi189.stdout:3/196: write d2/f5 [2818915,113734] 0 2024-04-20T21:32:04.900 INFO:tasks.workunit.client.1.smithi189.stdout:9/262: dwrite d9/dc/d17/f2f [0,4194304] 0 2024-04-20T21:32:04.902 INFO:tasks.workunit.client.0.smithi062.stdout:1/630: creat dc/d27/d9a/da0/db3/fe1 x:0 0 0 2024-04-20T21:32:04.903 INFO:tasks.workunit.client.1.smithi189.stdout:8/344: fsync da/df/d13/f5f 0 2024-04-20T21:32:04.904 INFO:tasks.workunit.client.1.smithi189.stdout:3/197: read d2/d1c/f2b [6864377,103699] 0 2024-04-20T21:32:04.905 INFO:tasks.workunit.client.1.smithi189.stdout:9/263: stat d9/d12/d15/d1a/l45 0 2024-04-20T21:32:04.906 INFO:tasks.workunit.client.0.smithi062.stdout:3/548: dwrite d0/d11/f56 [0,4194304] 0 2024-04-20T21:32:04.911 INFO:tasks.workunit.client.1.smithi189.stdout:0/332: rename d3/df/d28/l40 to d3/df/d28/d2b/d2c/l6f 0 2024-04-20T21:32:04.912 INFO:tasks.workunit.client.1.smithi189.stdout:4/218: symlink d2/dd/d11/l55 0 2024-04-20T21:32:04.913 INFO:tasks.workunit.client.0.smithi062.stdout:5/583: rename d8/d1e/f24 to d8/d10/d3e/d8a/fb9 0 2024-04-20T21:32:04.915 INFO:tasks.workunit.client.0.smithi062.stdout:2/604: rmdir d7/da/d15/d1b/d3d/d84/d9b 39 2024-04-20T21:32:04.918 INFO:tasks.workunit.client.0.smithi062.stdout:7/611: dread d7/d3e/d33/f52 [0,4194304] 0 2024-04-20T21:32:04.925 INFO:tasks.workunit.client.1.smithi189.stdout:3/198: symlink d2/d3/d2c/d30/d3f/l49 0 2024-04-20T21:32:04.927 INFO:tasks.workunit.client.1.smithi189.stdout:0/333: mknod d3/df/d5d/d64/c70 0 2024-04-20T21:32:04.927 INFO:tasks.workunit.client.0.smithi062.stdout:1/631: symlink dc/d15/d75/d96/dd4/d3a/le2 0 2024-04-20T21:32:04.927 INFO:tasks.workunit.client.0.smithi062.stdout:3/549: mknod d0/d1c/d63/d42/d5b/cb8 0 2024-04-20T21:32:04.928 INFO:tasks.workunit.client.1.smithi189.stdout:4/219: mkdir d2/d36/d56 0 2024-04-20T21:32:04.931 INFO:tasks.workunit.client.1.smithi189.stdout:7/243: getdents d5 0 2024-04-20T21:32:04.932 INFO:tasks.workunit.client.0.smithi062.stdout:7/612: creat d7/d26/dac/fd4 x:0 0 0 2024-04-20T21:32:04.934 INFO:tasks.workunit.client.1.smithi189.stdout:8/345: symlink da/l77 0 2024-04-20T21:32:04.936 INFO:tasks.workunit.client.1.smithi189.stdout:7/244: write d5/d1d/f26 [3735390,97972] 0 2024-04-20T21:32:04.940 INFO:tasks.workunit.client.1.smithi189.stdout:3/199: creat d2/d8/f4a x:0 0 0 2024-04-20T21:32:04.940 INFO:tasks.workunit.client.1.smithi189.stdout:4/220: mkdir d2/d22/d57 0 2024-04-20T21:32:04.941 INFO:tasks.workunit.client.0.smithi062.stdout:6/452: rename d0/d1a/c41 to d0/d9/d11/d5c/d7c/c90 0 2024-04-20T21:32:04.942 INFO:tasks.workunit.client.0.smithi062.stdout:1/632: mkdir dc/d15/d75/d94/d6f/dd2/de3 0 2024-04-20T21:32:04.943 INFO:tasks.workunit.client.0.smithi062.stdout:7/613: read d7/d3e/d33/d3d/d5e/d7e/f93 [8073311,7311] 0 2024-04-20T21:32:04.943 INFO:tasks.workunit.client.1.smithi189.stdout:4/221: chown d2/f21 853484487 1 2024-04-20T21:32:04.944 INFO:tasks.workunit.client.1.smithi189.stdout:7/245: mkdir d5/d21/d35/d53 0 2024-04-20T21:32:04.945 INFO:tasks.workunit.client.0.smithi062.stdout:3/550: mkdir d0/d1/db7/db9 0 2024-04-20T21:32:04.946 INFO:tasks.workunit.client.1.smithi189.stdout:8/346: rename da/df/d13/d31/c49 to da/de/d1e/d21/d44/c78 0 2024-04-20T21:32:04.948 INFO:tasks.workunit.client.1.smithi189.stdout:4/222: chown d2/dd/d11/f4b 79760819 1 2024-04-20T21:32:04.949 INFO:tasks.workunit.client.0.smithi062.stdout:3/551: chown d0/d1/f77 1228 1 2024-04-20T21:32:04.949 INFO:tasks.workunit.client.0.smithi062.stdout:7/614: write d7/d3e/f99 [1319954,119704] 0 2024-04-20T21:32:04.956 INFO:tasks.workunit.client.1.smithi189.stdout:0/334: dwrite d3/f48 [4194304,4194304] 0 2024-04-20T21:32:04.961 INFO:tasks.workunit.client.1.smithi189.stdout:7/246: rename d5/d1d/d28/l3b to d5/d21/d4a/l54 0 2024-04-20T21:32:04.963 INFO:tasks.workunit.client.0.smithi062.stdout:6/453: creat d0/f91 x:0 0 0 2024-04-20T21:32:04.965 INFO:tasks.workunit.client.0.smithi062.stdout:3/552: dread d0/d1/d10/f7a [0,4194304] 0 2024-04-20T21:32:04.965 INFO:tasks.workunit.client.1.smithi189.stdout:3/200: link d2/fd d2/d8/f4b 0 2024-04-20T21:32:04.967 INFO:tasks.workunit.client.0.smithi062.stdout:4/610: unlink da/dd/f1c 0 2024-04-20T21:32:04.969 INFO:tasks.workunit.client.1.smithi189.stdout:7/247: symlink d5/d46/l55 0 2024-04-20T21:32:04.970 INFO:tasks.workunit.client.0.smithi062.stdout:0/559: dread d7/d1c/d15/d17/d1b/d74/f88 [0,4194304] 0 2024-04-20T21:32:04.972 INFO:tasks.workunit.client.1.smithi189.stdout:4/223: creat d2/dd/d11/f58 x:0 0 0 2024-04-20T21:32:04.976 INFO:tasks.workunit.client.1.smithi189.stdout:9/264: dwrite f6 [0,4194304] 0 2024-04-20T21:32:04.977 INFO:tasks.workunit.client.1.smithi189.stdout:4/224: creat d2/dd/d37/f59 x:0 0 0 2024-04-20T21:32:04.977 INFO:tasks.workunit.client.1.smithi189.stdout:9/265: chown f5 116 1 2024-04-20T21:32:04.978 INFO:tasks.workunit.client.0.smithi062.stdout:1/633: rename dc/d15/d20/d28/dc9 to dc/d15/d75/d96/dd4/d3a/da2/de4 0 2024-04-20T21:32:04.981 INFO:tasks.workunit.client.0.smithi062.stdout:1/634: write dc/f53 [16144147,57032] 0 2024-04-20T21:32:04.982 INFO:tasks.workunit.client.1.smithi189.stdout:3/201: getdents d2/d3/d2c/d30 0 2024-04-20T21:32:04.986 INFO:tasks.workunit.client.1.smithi189.stdout:3/202: write d2/f5 [9479316,13248] 0 2024-04-20T21:32:04.988 INFO:tasks.workunit.client.1.smithi189.stdout:7/248: creat d5/d1d/f56 x:0 0 0 2024-04-20T21:32:04.990 INFO:tasks.workunit.client.0.smithi062.stdout:5/584: dread d8/d10/d3e/fb3 [0,4194304] 0 2024-04-20T21:32:04.991 INFO:tasks.workunit.client.0.smithi062.stdout:7/615: getdents d7/d3e/d33/d3d/d4d/d83/d95 0 2024-04-20T21:32:04.992 INFO:tasks.workunit.client.1.smithi189.stdout:7/249: write d5/d1d/d28/f2f [4306973,125157] 0 2024-04-20T21:32:05.000 INFO:tasks.workunit.client.1.smithi189.stdout:4/225: dwrite d2/dd/f26 [0,4194304] 0 2024-04-20T21:32:05.001 INFO:tasks.workunit.client.1.smithi189.stdout:4/226: chown d2/dd 5625734 1 2024-04-20T21:32:05.004 INFO:tasks.workunit.client.1.smithi189.stdout:4/227: write d2/d3a/f3b [5966265,8387] 0 2024-04-20T21:32:05.004 INFO:tasks.workunit.client.0.smithi062.stdout:1/635: mkdir dc/d15/d75/d94/d6f/dd2/de3/de5 0 2024-04-20T21:32:05.006 INFO:tasks.workunit.client.0.smithi062.stdout:0/560: getdents d7/d1e/db6 0 2024-04-20T21:32:05.006 INFO:tasks.workunit.client.0.smithi062.stdout:9/610: dwrite d16/d2c/d42/f54 [0,4194304] 0 2024-04-20T21:32:05.008 INFO:tasks.workunit.client.1.smithi189.stdout:3/203: getdents d2/d1c 0 2024-04-20T21:32:05.010 INFO:tasks.workunit.client.0.smithi062.stdout:5/585: rename d8/d1e/f93 to d8/d1e/fba 0 2024-04-20T21:32:05.010 INFO:tasks.workunit.client.0.smithi062.stdout:8/586: sync 2024-04-20T21:32:05.010 INFO:tasks.workunit.client.0.smithi062.stdout:9/611: fdatasync d16/f32 0 2024-04-20T21:32:05.018 INFO:tasks.workunit.client.0.smithi062.stdout:8/587: chown d7/d30/f5d 103216 1 2024-04-20T21:32:05.021 INFO:tasks.workunit.client.1.smithi189.stdout:7/250: dwrite d5/d1d/f26 [4194304,4194304] 0 2024-04-20T21:32:05.021 INFO:tasks.workunit.client.0.smithi062.stdout:2/605: dread d7/da/d15/d1b/f2e [0,4194304] 0 2024-04-20T21:32:05.021 INFO:tasks.workunit.client.0.smithi062.stdout:6/454: dwrite d0/d9/d11/d5c/d7c/f7f [0,4194304] 0 2024-04-20T21:32:05.022 INFO:tasks.workunit.client.1.smithi189.stdout:4/228: creat d2/f5a x:0 0 0 2024-04-20T21:32:05.022 INFO:tasks.workunit.client.1.smithi189.stdout:7/251: stat d5/f47 0 2024-04-20T21:32:05.028 INFO:tasks.workunit.client.0.smithi062.stdout:5/586: write d8/d1e/d2c/f86 [726667,19665] 0 2024-04-20T21:32:05.028 INFO:tasks.workunit.client.1.smithi189.stdout:7/252: creat d5/f57 x:0 0 0 2024-04-20T21:32:05.029 INFO:tasks.workunit.client.1.smithi189.stdout:9/266: dwrite f4 [0,4194304] 0 2024-04-20T21:32:05.039 INFO:tasks.workunit.client.1.smithi189.stdout:8/347: dread f9 [0,4194304] 0 2024-04-20T21:32:05.042 INFO:tasks.workunit.client.0.smithi062.stdout:4/611: dwrite da/dd/df/d27/d32/f81 [0,4194304] 0 2024-04-20T21:32:05.042 INFO:tasks.workunit.client.0.smithi062.stdout:8/588: rename d7/d9/d29/c9a to d7/da9/cc8 0 2024-04-20T21:32:05.043 INFO:tasks.workunit.client.1.smithi189.stdout:0/335: dread d3/df/d4c/f41 [0,4194304] 0 2024-04-20T21:32:05.043 INFO:tasks.workunit.client.1.smithi189.stdout:9/267: read d9/f33 [3059660,66190] 0 2024-04-20T21:32:05.044 INFO:tasks.workunit.client.0.smithi062.stdout:2/606: symlink d7/da/d28/d4b/d8f/lc8 0 2024-04-20T21:32:05.046 INFO:tasks.workunit.client.0.smithi062.stdout:3/553: dwrite d0/f2f [0,4194304] 0 2024-04-20T21:32:05.048 INFO:tasks.workunit.client.1.smithi189.stdout:0/336: write d3/df/d30/d45/f6c [945477,130518] 0 2024-04-20T21:32:05.053 INFO:tasks.workunit.client.0.smithi062.stdout:1/636: dwrite dc/d15/d20/d28/f66 [0,4194304] 0 2024-04-20T21:32:05.054 INFO:tasks.workunit.client.0.smithi062.stdout:9/612: link d16/d2c/d42/da6/da9/fc4 d16/d3e/fc9 0 2024-04-20T21:32:05.055 INFO:tasks.workunit.client.1.smithi189.stdout:9/268: symlink d9/dc/d17/d2b/d52/l59 0 2024-04-20T21:32:05.057 INFO:tasks.workunit.client.1.smithi189.stdout:6/214: sync 2024-04-20T21:32:05.060 INFO:tasks.workunit.client.1.smithi189.stdout:0/337: fdatasync d3/df/d4c/f41 0 2024-04-20T21:32:05.061 INFO:tasks.workunit.client.1.smithi189.stdout:6/215: write d8/f1a [2276302,101819] 0 2024-04-20T21:32:05.062 INFO:tasks.workunit.client.0.smithi062.stdout:2/607: creat d7/d39/fc9 x:0 0 0 2024-04-20T21:32:05.063 INFO:tasks.workunit.client.1.smithi189.stdout:8/348: dread da/df/f19 [4194304,4194304] 0 2024-04-20T21:32:05.063 INFO:tasks.workunit.client.0.smithi062.stdout:6/455: symlink d0/d9/d11/d22/l92 0 2024-04-20T21:32:05.064 INFO:tasks.workunit.client.0.smithi062.stdout:5/587: mknod d8/d38/d7a/cbb 0 2024-04-20T21:32:05.066 INFO:tasks.workunit.client.1.smithi189.stdout:6/216: fsync d8/d2b/f3a 0 2024-04-20T21:32:05.067 INFO:tasks.workunit.client.0.smithi062.stdout:1/637: truncate dc/fe 4953254 0 2024-04-20T21:32:05.067 INFO:tasks.workunit.client.0.smithi062.stdout:0/561: dwrite d7/d1c/d15/d17/d40/d79/fb0 [0,4194304] 0 2024-04-20T21:32:05.067 INFO:tasks.workunit.client.0.smithi062.stdout:3/554: unlink d0/d1c/d63/f30 0 2024-04-20T21:32:05.070 INFO:tasks.workunit.client.1.smithi189.stdout:6/217: chown d8/da/dd/c29 3443776 1 2024-04-20T21:32:05.073 INFO:tasks.workunit.client.0.smithi062.stdout:9/613: creat d16/dbe/fca x:0 0 0 2024-04-20T21:32:05.078 INFO:tasks.workunit.client.0.smithi062.stdout:3/555: symlink d0/d1c/lba 0 2024-04-20T21:32:05.079 INFO:tasks.workunit.client.0.smithi062.stdout:0/562: chown d7/d1c/f23 51 1 2024-04-20T21:32:05.079 INFO:tasks.workunit.client.1.smithi189.stdout:0/338: unlink d3/d26/f2f 0 2024-04-20T21:32:05.081 INFO:tasks.workunit.client.1.smithi189.stdout:6/218: creat d8/d12/f44 x:0 0 0 2024-04-20T21:32:05.082 INFO:tasks.workunit.client.0.smithi062.stdout:3/556: chown d0/d1b/f57 82393 1 2024-04-20T21:32:05.084 INFO:tasks.workunit.client.0.smithi062.stdout:6/456: rename d0/d52/l6e to d0/d9/d11/d5c/l93 0 2024-04-20T21:32:05.087 INFO:tasks.workunit.client.1.smithi189.stdout:0/339: creat d3/df/d30/d45/d60/f71 x:0 0 0 2024-04-20T21:32:05.088 INFO:tasks.workunit.client.0.smithi062.stdout:2/608: mkdir d7/da/d15/d2f/d93/dac/dca 0 2024-04-20T21:32:05.090 INFO:tasks.workunit.client.1.smithi189.stdout:3/204: dread d2/d1c/f28 [4194304,4194304] 0 2024-04-20T21:32:05.091 INFO:tasks.workunit.client.0.smithi062.stdout:9/614: symlink d16/d2c/d62/lcb 0 2024-04-20T21:32:05.093 INFO:tasks.workunit.client.0.smithi062.stdout:3/557: fsync d0/d1/f8b 0 2024-04-20T21:32:05.094 INFO:tasks.workunit.client.0.smithi062.stdout:5/588: mknod d8/d38/d7a/d4c/cbc 0 2024-04-20T21:32:05.097 INFO:tasks.workunit.client.1.smithi189.stdout:7/253: dwrite d5/f14 [4194304,4194304] 0 2024-04-20T21:32:05.099 INFO:tasks.workunit.client.0.smithi062.stdout:9/615: symlink d16/d2c/d42/da6/lcc 0 2024-04-20T21:32:05.101 INFO:tasks.workunit.client.1.smithi189.stdout:8/349: dwrite da/df/d13/d63/f73 [0,4194304] 0 2024-04-20T21:32:05.104 INFO:tasks.workunit.client.0.smithi062.stdout:0/563: mknod d7/d1c/d15/d17/d1b/d9d/db1/cbc 0 2024-04-20T21:32:05.104 INFO:tasks.workunit.client.1.smithi189.stdout:3/205: mkdir d2/d3/d2c/d31/d4c 0 2024-04-20T21:32:05.104 INFO:tasks.workunit.client.0.smithi062.stdout:6/457: rename d0/fe to d0/d9/d11/d5c/f94 0 2024-04-20T21:32:05.106 INFO:tasks.workunit.client.0.smithi062.stdout:9/616: readlink d16/d2c/l94 0 2024-04-20T21:32:05.106 INFO:tasks.workunit.client.0.smithi062.stdout:8/589: dwrite d7/d25/d3b/d58/d8b/d24/f75 [0,4194304] 0 2024-04-20T21:32:05.108 INFO:tasks.workunit.client.0.smithi062.stdout:3/558: unlink d0/d1/d10/d31/c34 0 2024-04-20T21:32:05.111 INFO:tasks.workunit.client.0.smithi062.stdout:6/458: truncate d0/d52/f83 642129 0 2024-04-20T21:32:05.118 INFO:tasks.workunit.client.0.smithi062.stdout:3/559: rmdir d0/d11 39 2024-04-20T21:32:05.125 INFO:tasks.workunit.client.1.smithi189.stdout:9/269: getdents d9/dc/d17/d2b/d52 0 2024-04-20T21:32:05.125 INFO:tasks.workunit.client.0.smithi062.stdout:9/617: mkdir d16/d2f/d5b/dab/dcd 0 2024-04-20T21:32:05.127 INFO:tasks.workunit.client.0.smithi062.stdout:0/564: unlink d7/d1c/d15/d17/d26/f90 0 2024-04-20T21:32:05.129 INFO:tasks.workunit.client.1.smithi189.stdout:6/219: getdents d8 0 2024-04-20T21:32:05.132 INFO:tasks.workunit.client.1.smithi189.stdout:7/254: fdatasync d5/f6 0 2024-04-20T21:32:05.134 INFO:tasks.workunit.client.0.smithi062.stdout:6/459: mknod d0/d9/d11/c95 0 2024-04-20T21:32:05.136 INFO:tasks.workunit.client.1.smithi189.stdout:9/270: unlink d9/d12/d15/f37 0 2024-04-20T21:32:05.139 INFO:tasks.workunit.client.1.smithi189.stdout:6/220: creat d8/da/dd/d35/f45 x:0 0 0 2024-04-20T21:32:05.143 INFO:tasks.workunit.client.0.smithi062.stdout:4/612: dread da/dd/df/f17 [0,4194304] 0 2024-04-20T21:32:05.145 INFO:tasks.workunit.client.0.smithi062.stdout:9/618: write d16/d4e/d80/fc0 [791440,90081] 0 2024-04-20T21:32:05.146 INFO:tasks.workunit.client.0.smithi062.stdout:3/560: fdatasync d0/d11/fa7 0 2024-04-20T21:32:05.146 INFO:tasks.workunit.client.0.smithi062.stdout:2/609: dwrite d7/da/d15/d2f/d38/d6a/d7c/f96 [0,4194304] 0 2024-04-20T21:32:05.148 INFO:tasks.workunit.client.0.smithi062.stdout:0/565: symlink d7/d1c/d15/d80/lbd 0 2024-04-20T21:32:05.151 INFO:tasks.workunit.client.0.smithi062.stdout:6/460: creat d0/d1a/d7d/f96 x:0 0 0 2024-04-20T21:32:05.155 INFO:tasks.workunit.client.1.smithi189.stdout:0/340: rename d3/df/d28/d2b/d2c/l6f to d3/df/d28/d2b/d2c/l72 0 2024-04-20T21:32:05.157 INFO:tasks.workunit.client.0.smithi062.stdout:7/616: dread d7/d3e/d33/d3d/d4d/d83/d95/faa [4194304,4194304] 0 2024-04-20T21:32:05.157 INFO:tasks.workunit.client.0.smithi062.stdout:3/561: write d0/d1c/d52/d7b/f68 [1045654,36471] 0 2024-04-20T21:32:05.165 INFO:tasks.workunit.client.1.smithi189.stdout:8/350: creat da/df/d20/f79 x:0 0 0 2024-04-20T21:32:05.167 INFO:tasks.workunit.client.0.smithi062.stdout:0/566: read d7/d1c/d15/d80/fb2 [3109918,53431] 0 2024-04-20T21:32:05.167 INFO:tasks.workunit.client.0.smithi062.stdout:6/461: unlink d0/d9/d11/d22/l58 0 2024-04-20T21:32:05.168 INFO:tasks.workunit.client.1.smithi189.stdout:7/255: dwrite d5/d1d/f44 [4194304,4194304] 0 2024-04-20T21:32:05.168 INFO:tasks.workunit.client.1.smithi189.stdout:9/271: creat d9/dc/d17/d2b/d55/f5a x:0 0 0 2024-04-20T21:32:05.168 INFO:tasks.workunit.client.0.smithi062.stdout:6/462: read - d0/d2c/f89 zero size 2024-04-20T21:32:05.170 INFO:tasks.workunit.client.0.smithi062.stdout:7/617: chown d7/d3e/d33/d3d/d7f 3112207 1 2024-04-20T21:32:05.171 INFO:tasks.workunit.client.0.smithi062.stdout:0/567: chown d7/d1c/d15/d17/d40/d3c 2120 1 2024-04-20T21:32:05.171 INFO:tasks.workunit.client.0.smithi062.stdout:5/589: dwrite d8/d38/d7a/d4c/f7b [0,4194304] 0 2024-04-20T21:32:05.172 INFO:tasks.workunit.client.0.smithi062.stdout:2/610: dread d7/da/d15/d1b/d24/f66 [0,4194304] 0 2024-04-20T21:32:05.174 INFO:tasks.workunit.client.1.smithi189.stdout:0/341: write d3/df/d28/d2b/f67 [427,111055] 0 2024-04-20T21:32:05.174 INFO:tasks.workunit.client.0.smithi062.stdout:4/613: getdents da/dd 0 2024-04-20T21:32:05.174 INFO:tasks.workunit.client.1.smithi189.stdout:8/351: mknod da/df/d13/d63/c7a 0 2024-04-20T21:32:05.177 INFO:tasks.workunit.client.0.smithi062.stdout:0/568: mkdir d7/d1e/da9/dbe 0 2024-04-20T21:32:05.178 INFO:tasks.workunit.client.1.smithi189.stdout:9/272: symlink d9/l5b 0 2024-04-20T21:32:05.178 INFO:tasks.workunit.client.0.smithi062.stdout:1/638: dread dc/d70/f90 [0,4194304] 0 2024-04-20T21:32:05.180 INFO:tasks.workunit.client.0.smithi062.stdout:7/618: chown d7/d3e/d33/d3d/d4d/d54/f3f 209457 1 2024-04-20T21:32:05.181 INFO:tasks.workunit.client.1.smithi189.stdout:6/221: write d8/da/f32 [1033007,68157] 0 2024-04-20T21:32:05.184 INFO:tasks.workunit.client.0.smithi062.stdout:3/562: rename d0/d11/fa7 to d0/d1c/d63/fbb 0 2024-04-20T21:32:05.184 INFO:tasks.workunit.client.0.smithi062.stdout:2/611: symlink d7/da/d15/d1b/d24/lcb 0 2024-04-20T21:32:05.184 INFO:tasks.workunit.client.1.smithi189.stdout:6/222: chown d8/da/f1e 92 1 2024-04-20T21:32:05.186 INFO:tasks.workunit.client.1.smithi189.stdout:8/352: mkdir da/df/d13/d63/d7b 0 2024-04-20T21:32:05.189 INFO:tasks.workunit.client.0.smithi062.stdout:5/590: mkdir d8/dbd 0 2024-04-20T21:32:05.190 INFO:tasks.workunit.client.0.smithi062.stdout:0/569: creat d7/d1c/d15/d17/d37/fbf x:0 0 0 2024-04-20T21:32:05.190 INFO:tasks.workunit.client.1.smithi189.stdout:0/342: mkdir d3/df/d28/d2b/d2c/d73 0 2024-04-20T21:32:05.190 INFO:tasks.workunit.client.0.smithi062.stdout:1/639: write f1 [1467182,12081] 0 2024-04-20T21:32:05.194 INFO:tasks.workunit.client.1.smithi189.stdout:6/223: mknod d8/da/dd/c46 0 2024-04-20T21:32:05.195 INFO:tasks.workunit.client.1.smithi189.stdout:3/206: dwrite d2/fd [0,4194304] 0 2024-04-20T21:32:05.196 INFO:tasks.workunit.client.0.smithi062.stdout:2/612: link d7/da/d15/d1b/d24/c98 d7/da/d15/d2f/d38/d6a/ccc 0 2024-04-20T21:32:05.198 INFO:tasks.workunit.client.0.smithi062.stdout:0/570: creat d7/d1c/d15/d17/d26/fc0 x:0 0 0 2024-04-20T21:32:05.198 INFO:tasks.workunit.client.0.smithi062.stdout:6/463: rename d0/d9/d11/d22/l92 to d0/d9/d11/l97 0 2024-04-20T21:32:05.199 INFO:tasks.workunit.client.0.smithi062.stdout:2/613: chown d7/da/d28/d4b/l8b 5 1 2024-04-20T21:32:05.201 INFO:tasks.workunit.client.0.smithi062.stdout:0/571: stat d7/d1e/f67 0 2024-04-20T21:32:05.203 INFO:tasks.workunit.client.1.smithi189.stdout:7/256: getdents d5/d1d 0 2024-04-20T21:32:05.204 INFO:tasks.workunit.client.1.smithi189.stdout:3/207: rmdir d2/d3 39 2024-04-20T21:32:05.204 INFO:tasks.workunit.client.1.smithi189.stdout:0/343: chown d3/df/d30/d45/d60/c51 10 1 2024-04-20T21:32:05.207 INFO:tasks.workunit.client.0.smithi062.stdout:6/464: stat d0/d9/d11/d5c/d7c/c90 0 2024-04-20T21:32:05.207 INFO:tasks.workunit.client.1.smithi189.stdout:8/353: getdents da/df/d20/d27 0 2024-04-20T21:32:05.208 INFO:tasks.workunit.client.1.smithi189.stdout:3/208: stat d2/d8/f34 0 2024-04-20T21:32:05.209 INFO:tasks.workunit.client.1.smithi189.stdout:5/172: sync 2024-04-20T21:32:05.209 INFO:tasks.workunit.client.1.smithi189.stdout:1/216: sync 2024-04-20T21:32:05.210 INFO:tasks.workunit.client.1.smithi189.stdout:8/354: dread - da/de/d1e/d69/d70/f72 zero size 2024-04-20T21:32:05.210 INFO:tasks.workunit.client.0.smithi062.stdout:0/572: unlink d7/f9 0 2024-04-20T21:32:05.211 INFO:tasks.workunit.client.1.smithi189.stdout:7/257: readlink d5/d21/d25/l2a 0 2024-04-20T21:32:05.212 INFO:tasks.workunit.client.0.smithi062.stdout:2/614: creat d7/da/d15/d2f/d38/d51/dbc/fcd x:0 0 0 2024-04-20T21:32:05.213 INFO:tasks.workunit.client.1.smithi189.stdout:5/173: write d7/d14/f26 [492396,835] 0 2024-04-20T21:32:05.216 INFO:tasks.workunit.client.0.smithi062.stdout:3/563: rename d0/d1/f29 to d0/d1c/d52/fbc 0 2024-04-20T21:32:05.216 INFO:tasks.workunit.client.0.smithi062.stdout:1/640: fsync f1 0 2024-04-20T21:32:05.219 INFO:tasks.workunit.client.0.smithi062.stdout:2/615: fdatasync d7/da/d15/d2f/d38/d51/dbc/fcd 0 2024-04-20T21:32:05.220 INFO:tasks.workunit.client.1.smithi189.stdout:6/224: dread d8/d2b/f3a [0,4194304] 0 2024-04-20T21:32:05.220 INFO:tasks.workunit.client.1.smithi189.stdout:1/217: dread d8/d1e/d28/d2b/f48 [0,4194304] 0 2024-04-20T21:32:05.220 INFO:tasks.workunit.client.1.smithi189.stdout:3/209: symlink d2/d3/d2c/d30/l4d 0 2024-04-20T21:32:05.227 INFO:tasks.workunit.client.1.smithi189.stdout:1/218: rmdir d8/d1e/d43/d44 39 2024-04-20T21:32:05.229 INFO:tasks.workunit.client.1.smithi189.stdout:0/344: dwrite d3/df/d4c/f69 [0,4194304] 0 2024-04-20T21:32:05.231 INFO:tasks.workunit.client.1.smithi189.stdout:7/258: rename d5/l16 to d5/d1d/d42/d45/l58 0 2024-04-20T21:32:05.232 INFO:tasks.workunit.client.1.smithi189.stdout:9/273: creat d9/d12/d15/d1a/f5c x:0 0 0 2024-04-20T21:32:05.233 INFO:tasks.workunit.client.1.smithi189.stdout:0/345: read - d3/f66 zero size 2024-04-20T21:32:05.233 INFO:tasks.workunit.client.1.smithi189.stdout:1/219: dread d8/f9 [4194304,4194304] 0 2024-04-20T21:32:05.238 INFO:tasks.workunit.client.1.smithi189.stdout:9/274: read - d9/dc/f2c zero size 2024-04-20T21:32:05.239 INFO:tasks.workunit.client.1.smithi189.stdout:5/174: dread d7/d9/d30/f2d [0,4194304] 0 2024-04-20T21:32:05.240 INFO:tasks.workunit.client.1.smithi189.stdout:8/355: truncate da/de/f28 2706889 0 2024-04-20T21:32:05.242 INFO:tasks.workunit.client.1.smithi189.stdout:9/275: stat d9/d12/d50/f4f 0 2024-04-20T21:32:05.245 INFO:tasks.workunit.client.1.smithi189.stdout:7/259: mkdir d5/d59 0 2024-04-20T21:32:05.248 INFO:tasks.workunit.client.1.smithi189.stdout:9/276: rmdir d9 39 2024-04-20T21:32:05.248 INFO:tasks.workunit.client.1.smithi189.stdout:6/225: dwrite d8/d2b/f3a [4194304,4194304] 0 2024-04-20T21:32:05.252 INFO:tasks.workunit.client.1.smithi189.stdout:6/226: fdatasync d8/d12/f17 0 2024-04-20T21:32:05.254 INFO:tasks.workunit.client.1.smithi189.stdout:2/204: sync 2024-04-20T21:32:05.254 INFO:tasks.workunit.client.1.smithi189.stdout:4/229: sync 2024-04-20T21:32:05.260 INFO:tasks.workunit.client.1.smithi189.stdout:1/220: link d8/d21/d2c/l4a d8/d21/d2c/d36/l4c 0 2024-04-20T21:32:05.260 INFO:tasks.workunit.client.1.smithi189.stdout:4/230: truncate d2/dd/fe 4337599 0 2024-04-20T21:32:05.263 INFO:tasks.workunit.client.1.smithi189.stdout:6/227: mkdir d8/d12/d30/d47 0 2024-04-20T21:32:05.263 INFO:tasks.workunit.client.1.smithi189.stdout:8/356: truncate f9 3637968 0 2024-04-20T21:32:05.265 INFO:tasks.workunit.client.1.smithi189.stdout:6/228: readlink d8/l40 0 2024-04-20T21:32:05.266 INFO:tasks.workunit.client.1.smithi189.stdout:9/277: mkdir d9/dc/d17/d36/d5d 0 2024-04-20T21:32:05.268 INFO:tasks.workunit.client.1.smithi189.stdout:6/229: chown d8/l40 5504515 1 2024-04-20T21:32:05.270 INFO:tasks.workunit.client.1.smithi189.stdout:5/175: getdents d7/d9/d30 0 2024-04-20T21:32:05.271 INFO:tasks.workunit.client.1.smithi189.stdout:3/210: dread d2/d8/f22 [0,4194304] 0 2024-04-20T21:32:05.272 INFO:tasks.workunit.client.1.smithi189.stdout:8/357: dread - da/df/d20/d2c/f76 zero size 2024-04-20T21:32:05.272 INFO:tasks.workunit.client.1.smithi189.stdout:7/260: link d5/d46/l55 d5/d1d/d42/d45/l5a 0 2024-04-20T21:32:05.278 INFO:tasks.workunit.client.1.smithi189.stdout:5/176: mkdir d7/d14/d1b/d1f/d33 0 2024-04-20T21:32:05.278 INFO:tasks.workunit.client.1.smithi189.stdout:9/278: write d9/f33 [4467588,82733] 0 2024-04-20T21:32:05.281 INFO:tasks.workunit.client.1.smithi189.stdout:3/211: chown d2/d3/d25/f40 1 1 2024-04-20T21:32:05.281 INFO:tasks.workunit.client.1.smithi189.stdout:1/221: dread d8/f14 [4194304,4194304] 0 2024-04-20T21:32:05.284 INFO:tasks.workunit.client.1.smithi189.stdout:5/177: unlink d7/d9/c20 0 2024-04-20T21:32:05.288 INFO:tasks.workunit.client.1.smithi189.stdout:5/178: chown d7/l12 20722519 1 2024-04-20T21:32:05.289 INFO:tasks.workunit.client.1.smithi189.stdout:9/279: write d9/dc/d17/f2f [517375,114216] 0 2024-04-20T21:32:05.291 INFO:tasks.workunit.client.1.smithi189.stdout:0/346: dread d3/df/d30/d37/f31 [0,4194304] 0 2024-04-20T21:32:05.291 INFO:tasks.workunit.client.1.smithi189.stdout:7/261: rename d5/d1d/d28 to d5/d21/d35/d53/d5b 0 2024-04-20T21:32:05.293 INFO:tasks.workunit.client.1.smithi189.stdout:1/222: read - d8/d1e/d43/d44/f47 zero size 2024-04-20T21:32:05.295 INFO:tasks.workunit.client.1.smithi189.stdout:0/347: dread - d3/f66 zero size 2024-04-20T21:32:05.296 INFO:tasks.workunit.client.1.smithi189.stdout:1/223: readlink d8/lc 0 2024-04-20T21:32:05.308 INFO:tasks.workunit.client.1.smithi189.stdout:8/358: link da/df/d20/l2e da/df/d3a/l7c 0 2024-04-20T21:32:05.310 INFO:tasks.workunit.client.1.smithi189.stdout:6/230: dwrite d8/d12/d30/f38 [0,4194304] 0 2024-04-20T21:32:05.310 INFO:tasks.workunit.client.1.smithi189.stdout:7/262: creat d5/d21/d4a/d52/f5c x:0 0 0 2024-04-20T21:32:05.313 INFO:tasks.workunit.client.1.smithi189.stdout:5/179: rename d7/d14/c24 to d7/d14/d1b/d1f/c34 0 2024-04-20T21:32:05.314 INFO:tasks.workunit.client.1.smithi189.stdout:0/348: mkdir d3/df/d28/d2b/d2c/d74 0 2024-04-20T21:32:05.316 INFO:tasks.workunit.client.1.smithi189.stdout:0/349: chown d3/df/d28 3227 1 2024-04-20T21:32:05.316 INFO:tasks.workunit.client.1.smithi189.stdout:5/180: stat d7/d9/d30 0 2024-04-20T21:32:05.316 INFO:tasks.workunit.client.1.smithi189.stdout:2/205: dwrite d1/fe [0,4194304] 0 2024-04-20T21:32:05.316 INFO:tasks.workunit.client.1.smithi189.stdout:6/231: unlink d8/d12/d30/c3e 0 2024-04-20T21:32:05.317 INFO:tasks.workunit.client.1.smithi189.stdout:7/263: chown d5/c1c 17073 1 2024-04-20T21:32:05.322 INFO:tasks.workunit.client.1.smithi189.stdout:6/232: truncate d8/d10/f3f 699191 0 2024-04-20T21:32:05.322 INFO:tasks.workunit.client.1.smithi189.stdout:3/212: dwrite d2/d3/f19 [0,4194304] 0 2024-04-20T21:32:05.323 INFO:tasks.workunit.client.1.smithi189.stdout:2/206: unlink d1/d11/d2d/f32 0 2024-04-20T21:32:05.324 INFO:tasks.workunit.client.1.smithi189.stdout:0/350: truncate d3/df/d4c/f69 4470808 0 2024-04-20T21:32:05.326 INFO:tasks.workunit.client.1.smithi189.stdout:1/224: write f1 [6530847,56461] 0 2024-04-20T21:32:05.328 INFO:tasks.workunit.client.1.smithi189.stdout:8/359: write da/df/d3c/d50/f61 [4672200,111497] 0 2024-04-20T21:32:05.331 INFO:tasks.workunit.client.1.smithi189.stdout:0/351: dread - d3/df/d5d/d64/f6d zero size 2024-04-20T21:32:05.337 INFO:tasks.workunit.client.1.smithi189.stdout:6/233: mknod d8/d12/d30/d47/c48 0 2024-04-20T21:32:05.339 INFO:tasks.workunit.client.1.smithi189.stdout:1/225: mknod d8/df/d3a/c4d 0 2024-04-20T21:32:05.345 INFO:tasks.workunit.client.1.smithi189.stdout:0/352: mknod d3/df/d28/d2b/d2c/c75 0 2024-04-20T21:32:05.347 INFO:tasks.workunit.client.1.smithi189.stdout:0/353: fsync d3/df/f15 0 2024-04-20T21:32:05.348 INFO:tasks.workunit.client.1.smithi189.stdout:8/360: creat da/df/d20/f7d x:0 0 0 2024-04-20T21:32:05.351 INFO:tasks.workunit.client.1.smithi189.stdout:1/226: mkdir d8/df/d4e 0 2024-04-20T21:32:05.351 INFO:tasks.workunit.client.1.smithi189.stdout:7/264: dread d5/d21/d35/d53/d5b/f2c [0,4194304] 0 2024-04-20T21:32:05.352 INFO:tasks.workunit.client.1.smithi189.stdout:2/207: getdents d1/d11/d26 0 2024-04-20T21:32:05.353 INFO:tasks.workunit.client.1.smithi189.stdout:3/213: truncate d2/f5 7469358 0 2024-04-20T21:32:05.357 INFO:tasks.workunit.client.1.smithi189.stdout:7/265: truncate d5/fe 1443516 0 2024-04-20T21:32:05.359 INFO:tasks.workunit.client.1.smithi189.stdout:3/214: write d2/d3/d25/f2e [2961283,74433] 0 2024-04-20T21:32:05.361 INFO:tasks.workunit.client.1.smithi189.stdout:8/361: truncate da/de/d1e/d69/d70/f72 68243 0 2024-04-20T21:32:05.361 INFO:tasks.workunit.client.1.smithi189.stdout:0/354: creat d3/df/f76 x:0 0 0 2024-04-20T21:32:05.361 INFO:tasks.workunit.client.1.smithi189.stdout:2/208: fsync d1/fd 0 2024-04-20T21:32:05.364 INFO:tasks.workunit.client.1.smithi189.stdout:0/355: readlink d3/df/d28/d2b/l57 0 2024-04-20T21:32:05.365 INFO:tasks.workunit.client.1.smithi189.stdout:3/215: readlink d2/d3/d36/l3d 0 2024-04-20T21:32:05.371 INFO:tasks.workunit.client.0.smithi062.stdout:0/573: rename d7/d1c/d15/d17/c2a to d7/d1c/d15/d17/d40/d79/d71/cc1 0 2024-04-20T21:32:05.371 INFO:tasks.workunit.client.1.smithi189.stdout:8/362: rmdir da/de/d1e/d69/d70 39 2024-04-20T21:32:05.373 INFO:tasks.workunit.client.1.smithi189.stdout:0/356: fdatasync d3/df/d28/f61 0 2024-04-20T21:32:05.374 INFO:tasks.workunit.client.1.smithi189.stdout:6/234: dwrite d8/d12/f44 [0,4194304] 0 2024-04-20T21:32:05.375 INFO:tasks.workunit.client.1.smithi189.stdout:2/209: rename d1/fd to d1/d11/d2d/f38 0 2024-04-20T21:32:05.375 INFO:tasks.workunit.client.0.smithi062.stdout:4/614: dwrite da/dd/df/f28 [0,4194304] 0 2024-04-20T21:32:05.379 INFO:tasks.workunit.client.0.smithi062.stdout:2/616: mknod d7/da/d15/d2f/d38/db0/dc4/cce 0 2024-04-20T21:32:05.379 INFO:tasks.workunit.client.1.smithi189.stdout:5/181: dread d7/d9/d30/f1c [0,4194304] 0 2024-04-20T21:32:05.381 INFO:tasks.workunit.client.1.smithi189.stdout:8/363: write da/df/d22/d33/f5a [4047566,104223] 0 2024-04-20T21:32:05.381 INFO:tasks.workunit.client.0.smithi062.stdout:0/574: mknod d7/d1c/d15/d17/d40/cc2 0 2024-04-20T21:32:05.383 INFO:tasks.workunit.client.0.smithi062.stdout:4/615: mkdir da/dd/d15/d4b/ddb 0 2024-04-20T21:32:05.383 INFO:tasks.workunit.client.1.smithi189.stdout:3/216: write d2/d8/f22 [4405233,45643] 0 2024-04-20T21:32:05.387 INFO:tasks.workunit.client.0.smithi062.stdout:2/617: write d7/da/d28/f6d [4135734,2672] 0 2024-04-20T21:32:05.389 INFO:tasks.workunit.client.0.smithi062.stdout:1/641: dwrite dc/d15/f18 [0,4194304] 0 2024-04-20T21:32:05.390 INFO:tasks.workunit.client.1.smithi189.stdout:0/357: mknod d3/df/d28/d2b/d2c/d74/c77 0 2024-04-20T21:32:05.390 INFO:tasks.workunit.client.0.smithi062.stdout:2/618: fdatasync d7/da/d28/f2a 0 2024-04-20T21:32:05.395 INFO:tasks.workunit.client.1.smithi189.stdout:3/217: chown d2/d3/d2c/d31/f3c 0 1 2024-04-20T21:32:05.396 INFO:tasks.workunit.client.1.smithi189.stdout:1/227: dwrite d8/df/f3d [0,4194304] 0 2024-04-20T21:32:05.396 INFO:tasks.workunit.client.1.smithi189.stdout:8/364: chown da/df/l74 28205 1 2024-04-20T21:32:05.396 INFO:tasks.workunit.client.0.smithi062.stdout:0/575: symlink d7/d20/d62/d6f/lc3 0 2024-04-20T21:32:05.398 INFO:tasks.workunit.client.0.smithi062.stdout:0/576: write d7/d1c/d15/d17/d37/fbf [813935,90391] 0 2024-04-20T21:32:05.403 INFO:tasks.workunit.client.1.smithi189.stdout:5/182: truncate d7/d14/d18/f17 977215 0 2024-04-20T21:32:05.403 INFO:tasks.workunit.client.0.smithi062.stdout:0/577: readlink d7/d1c/d15/d17/d1b/d74/l8e 0 2024-04-20T21:32:05.406 INFO:tasks.workunit.client.0.smithi062.stdout:1/642: getdents dc/d15/d75/dd0 0 2024-04-20T21:32:05.407 INFO:tasks.workunit.client.1.smithi189.stdout:0/358: creat d3/df/d4c/f78 x:0 0 0 2024-04-20T21:32:05.408 INFO:tasks.workunit.client.1.smithi189.stdout:8/365: mknod da/df/d13/d63/c7e 0 2024-04-20T21:32:05.409 INFO:tasks.workunit.client.1.smithi189.stdout:1/228: mkdir d8/d1e/d28/d4f 0 2024-04-20T21:32:05.410 INFO:tasks.workunit.client.1.smithi189.stdout:8/366: stat da/df/d20/l4d 0 2024-04-20T21:32:05.410 INFO:tasks.workunit.client.1.smithi189.stdout:6/235: dwrite f5 [0,4194304] 0 2024-04-20T21:32:05.411 INFO:tasks.workunit.client.0.smithi062.stdout:0/578: fsync d7/d1c/f39 0 2024-04-20T21:32:05.411 INFO:tasks.workunit.client.0.smithi062.stdout:4/616: dwrite f3 [0,4194304] 0 2024-04-20T21:32:05.421 INFO:tasks.workunit.client.1.smithi189.stdout:3/218: dwrite d2/f29 [0,4194304] 0 2024-04-20T21:32:05.421 INFO:tasks.workunit.client.1.smithi189.stdout:5/183: unlink d7/d9/d30/c1d 0 2024-04-20T21:32:05.422 INFO:tasks.workunit.client.0.smithi062.stdout:2/619: dwrite d7/d39/f8e [0,4194304] 0 2024-04-20T21:32:05.422 INFO:tasks.workunit.client.0.smithi062.stdout:4/617: fdatasync da/dd/f19 0 2024-04-20T21:32:05.424 INFO:tasks.workunit.client.1.smithi189.stdout:2/210: truncate d1/f6 3385119 0 2024-04-20T21:32:05.425 INFO:tasks.workunit.client.0.smithi062.stdout:3/564: truncate d0/d1/d10/f79 207334 0 2024-04-20T21:32:05.425 INFO:tasks.workunit.client.0.smithi062.stdout:5/591: write d8/f1a [2333707,71495] 0 2024-04-20T21:32:05.426 INFO:tasks.workunit.client.1.smithi189.stdout:6/236: mkdir d8/d2b/d49 0 2024-04-20T21:32:05.431 INFO:tasks.workunit.client.0.smithi062.stdout:1/643: creat dc/d15/d20/fe6 x:0 0 0 2024-04-20T21:32:05.431 INFO:tasks.workunit.client.1.smithi189.stdout:0/359: unlink d3/df/d30/d45/d60/c51 0 2024-04-20T21:32:05.431 INFO:tasks.workunit.client.1.smithi189.stdout:6/237: readlink d8/l40 0 2024-04-20T21:32:05.432 INFO:tasks.workunit.client.1.smithi189.stdout:8/367: creat da/df/d13/d63/f7f x:0 0 0 2024-04-20T21:32:05.434 INFO:tasks.workunit.client.0.smithi062.stdout:2/620: unlink d7/da/l5b 0 2024-04-20T21:32:05.435 INFO:tasks.workunit.client.1.smithi189.stdout:3/219: mknod d2/c4e 0 2024-04-20T21:32:05.436 INFO:tasks.workunit.client.0.smithi062.stdout:1/644: read dc/d15/d75/d94/f19 [2342602,83278] 0 2024-04-20T21:32:05.446 INFO:tasks.workunit.client.0.smithi062.stdout:3/565: creat d0/d1/d3c/d43/d72/fbd x:0 0 0 2024-04-20T21:32:05.447 INFO:tasks.workunit.client.0.smithi062.stdout:3/566: dread - d0/d1c/d52/d97/fa3 zero size 2024-04-20T21:32:05.449 INFO:tasks.workunit.client.1.smithi189.stdout:0/360: mkdir d3/df/d30/d45/d60/d79 0 2024-04-20T21:32:05.449 INFO:tasks.workunit.client.1.smithi189.stdout:6/238: symlink d8/d12/d30/l4a 0 2024-04-20T21:32:05.451 INFO:tasks.workunit.client.1.smithi189.stdout:2/211: dwrite d1/d11/f15 [0,4194304] 0 2024-04-20T21:32:05.452 INFO:tasks.workunit.client.1.smithi189.stdout:6/239: mknod d8/da/c4b 0 2024-04-20T21:32:05.452 INFO:tasks.workunit.client.0.smithi062.stdout:9/619: sync 2024-04-20T21:32:05.452 INFO:tasks.workunit.client.0.smithi062.stdout:8/590: sync 2024-04-20T21:32:05.452 INFO:tasks.workunit.client.0.smithi062.stdout:7/619: sync 2024-04-20T21:32:05.453 INFO:tasks.workunit.client.0.smithi062.stdout:1/645: read - dc/d15/d75/d96/dd4/d78/dab/fb7 zero size 2024-04-20T21:32:05.454 INFO:tasks.workunit.client.0.smithi062.stdout:9/620: stat d16/d2c/d5e/f7b 0 2024-04-20T21:32:05.459 INFO:tasks.workunit.client.1.smithi189.stdout:6/240: dread - d8/d2b/f39 zero size 2024-04-20T21:32:05.459 INFO:tasks.workunit.client.0.smithi062.stdout:7/620: mknod d7/dcd/cd5 0 2024-04-20T21:32:05.460 INFO:tasks.workunit.client.1.smithi189.stdout:0/361: creat d3/df/d30/d45/d60/d79/f7a x:0 0 0 2024-04-20T21:32:05.461 INFO:tasks.workunit.client.0.smithi062.stdout:8/591: symlink d7/da9/lc9 0 2024-04-20T21:32:05.462 INFO:tasks.workunit.client.0.smithi062.stdout:2/621: dwrite d7/da/fbd [0,4194304] 0 2024-04-20T21:32:05.463 INFO:tasks.workunit.client.1.smithi189.stdout:2/212: mknod d1/d17/d2f/c39 0 2024-04-20T21:32:05.465 INFO:tasks.workunit.client.0.smithi062.stdout:1/646: mknod dc/d15/d20/d28/d83/dce/ce7 0 2024-04-20T21:32:05.466 INFO:tasks.workunit.client.0.smithi062.stdout:7/621: read d7/d3e/d33/d3d/d4d/d83/f81 [79914,62866] 0 2024-04-20T21:32:05.468 INFO:tasks.workunit.client.0.smithi062.stdout:9/621: rename d16/d3d/d55/d6e/d9b/fb1 to d16/d2c/d42/da6/fce 0 2024-04-20T21:32:05.472 INFO:tasks.workunit.client.0.smithi062.stdout:1/647: fdatasync dc/d15/d75/d94/d6f/dd2/fdf 0 2024-04-20T21:32:05.474 INFO:tasks.workunit.client.1.smithi189.stdout:0/362: creat d3/df/d5d/f7b x:0 0 0 2024-04-20T21:32:05.476 INFO:tasks.workunit.client.1.smithi189.stdout:5/184: truncate d7/fb 3507483 0 2024-04-20T21:32:05.477 INFO:tasks.workunit.client.0.smithi062.stdout:8/592: symlink d7/d9/lca 0 2024-04-20T21:32:05.477 INFO:tasks.workunit.client.0.smithi062.stdout:7/622: symlink d7/dcd/ld6 0 2024-04-20T21:32:05.477 INFO:tasks.workunit.client.0.smithi062.stdout:0/579: write d7/d1c/d15/d17/d37/f61 [871899,61833] 0 2024-04-20T21:32:05.477 INFO:tasks.workunit.client.0.smithi062.stdout:2/622: truncate d7/da/d15/d1b/d24/f66 609762 0 2024-04-20T21:32:05.478 INFO:tasks.workunit.client.0.smithi062.stdout:9/622: symlink d16/d2c/d62/lcf 0 2024-04-20T21:32:05.485 INFO:tasks.workunit.client.0.smithi062.stdout:1/648: rename dc/c13 to dc/d15/d75/d96/ce8 0 2024-04-20T21:32:05.486 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:05 smithi189 ceph-mon[21562]: Updating smithi062:/etc/ceph/ceph.client.admin.keyring 2024-04-20T21:32:05.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:05 smithi189 ceph-mon[21562]: Updating smithi189:/etc/ceph/ceph.client.admin.keyring 2024-04-20T21:32:05.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:05 smithi189 ceph-mon[21562]: pgmap v17: 65 pgs: 65 active+clean; 1.7 GiB data, 5.0 GiB used, 531 GiB / 536 GiB avail; 55 MiB/s rd, 229 MiB/s wr, 452 op/s 2024-04-20T21:32:05.487 INFO:tasks.workunit.client.0.smithi062.stdout:6/465: sync 2024-04-20T21:32:05.487 INFO:tasks.workunit.client.0.smithi062.stdout:3/567: write d0/d1/d3c/d43/d72/f89 [588907,51027] 0 2024-04-20T21:32:05.489 INFO:tasks.workunit.client.0.smithi062.stdout:7/623: creat d7/d3e/d33/d3d/d4d/d83/fd7 x:0 0 0 2024-04-20T21:32:05.491 INFO:tasks.workunit.client.0.smithi062.stdout:2/623: write d7/da/d15/d1b/d3d/f49 [3174249,73843] 0 2024-04-20T21:32:05.496 INFO:tasks.workunit.client.0.smithi062.stdout:4/618: dwrite da/dd/d15/d50/fcb [0,4194304] 0 2024-04-20T21:32:05.497 INFO:tasks.workunit.client.0.smithi062.stdout:2/624: chown d7/da 2633629 1 2024-04-20T21:32:05.497 INFO:tasks.workunit.client.0.smithi062.stdout:0/580: truncate d7/d1c/f41 98975 0 2024-04-20T21:32:05.499 INFO:tasks.workunit.client.0.smithi062.stdout:7/624: fdatasync d7/d3e/d33/d3d/d4d/fc0 0 2024-04-20T21:32:05.501 INFO:tasks.workunit.client.0.smithi062.stdout:1/649: truncate dc/d15/d75/d96/dd4/d78/faa 815084 0 2024-04-20T21:32:05.502 INFO:tasks.workunit.client.0.smithi062.stdout:2/625: stat d7/da/d28/d4b/l8a 0 2024-04-20T21:32:05.505 INFO:tasks.workunit.client.0.smithi062.stdout:8/593: write d7/d9/f3a [2925914,38740] 0 2024-04-20T21:32:05.511 INFO:tasks.workunit.client.0.smithi062.stdout:0/581: read d7/d1c/d15/d17/d37/f59 [152127,128490] 0 2024-04-20T21:32:05.511 INFO:tasks.workunit.client.0.smithi062.stdout:9/623: dwrite d16/d2c/d5e/f7b [0,4194304] 0 2024-04-20T21:32:05.521 INFO:tasks.workunit.client.0.smithi062.stdout:2/626: mknod d7/ccf 0 2024-04-20T21:32:05.524 INFO:tasks.workunit.client.0.smithi062.stdout:8/594: mknod d7/d25/d3b/d58/ccb 0 2024-04-20T21:32:05.524 INFO:tasks.workunit.client.1.smithi189.stdout:4/231: sync 2024-04-20T21:32:05.527 INFO:tasks.workunit.client.0.smithi062.stdout:7/625: creat d7/d26/da8/db6/fd8 x:0 0 0 2024-04-20T21:32:05.529 INFO:tasks.workunit.client.0.smithi062.stdout:2/627: stat d7/da/d15/d2f/d38/d51/f68 0 2024-04-20T21:32:05.530 INFO:tasks.workunit.client.0.smithi062.stdout:5/592: dread d8/d10/d3e/f62 [0,4194304] 0 2024-04-20T21:32:05.533 INFO:tasks.workunit.client.0.smithi062.stdout:7/626: readlink d7/d3e/d33/l56 0 2024-04-20T21:32:05.534 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:05 smithi062 ceph-mon[16221]: Updating smithi062:/etc/ceph/ceph.client.admin.keyring 2024-04-20T21:32:05.534 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:05 smithi062 ceph-mon[16221]: Updating smithi189:/etc/ceph/ceph.client.admin.keyring 2024-04-20T21:32:05.534 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:05 smithi062 ceph-mon[16221]: pgmap v17: 65 pgs: 65 active+clean; 1.7 GiB data, 5.0 GiB used, 531 GiB / 536 GiB avail; 55 MiB/s rd, 229 MiB/s wr, 452 op/s 2024-04-20T21:32:05.535 INFO:tasks.workunit.client.1.smithi189.stdout:0/363: readlink d3/df/d30/d45/l4a 0 2024-04-20T21:32:05.536 INFO:tasks.workunit.client.1.smithi189.stdout:4/232: dread - d2/dd/d37/f59 zero size 2024-04-20T21:32:05.538 INFO:tasks.workunit.client.1.smithi189.stdout:6/241: dwrite d8/d10/f36 [0,4194304] 0 2024-04-20T21:32:05.539 INFO:tasks.workunit.client.1.smithi189.stdout:1/229: dread d8/d1e/f23 [0,4194304] 0 2024-04-20T21:32:05.540 INFO:tasks.workunit.client.0.smithi062.stdout:3/568: dwrite d0/d1c/f45 [0,4194304] 0 2024-04-20T21:32:05.541 INFO:tasks.workunit.client.1.smithi189.stdout:5/185: mkdir d7/d35 0 2024-04-20T21:32:05.543 INFO:tasks.workunit.client.1.smithi189.stdout:0/364: creat d3/df/d5d/f7c x:0 0 0 2024-04-20T21:32:05.545 INFO:tasks.workunit.client.1.smithi189.stdout:2/213: link d1/cb d1/d11/d2d/c3a 0 2024-04-20T21:32:05.546 INFO:tasks.workunit.client.0.smithi062.stdout:4/619: dread da/dd/df/d27/d32/f37 [0,4194304] 0 2024-04-20T21:32:05.546 INFO:tasks.workunit.client.0.smithi062.stdout:0/582: mknod d7/d1c/d15/d17/d1b/d9d/cc4 0 2024-04-20T21:32:05.546 INFO:tasks.workunit.client.1.smithi189.stdout:1/230: write d8/d21/f22 [354978,20210] 0 2024-04-20T21:32:05.549 INFO:tasks.workunit.client.1.smithi189.stdout:0/365: write d3/df/d30/d45/d60/f68 [494140,33153] 0 2024-04-20T21:32:05.551 INFO:tasks.workunit.client.1.smithi189.stdout:6/242: rename d8/da/f1e to d8/d12/d30/d47/f4c 0 2024-04-20T21:32:05.553 INFO:tasks.workunit.client.1.smithi189.stdout:0/366: chown d3/df/d28/d2b/f67 14282 1 2024-04-20T21:32:05.554 INFO:tasks.workunit.client.0.smithi062.stdout:7/627: symlink d7/d26/da8/ld9 0 2024-04-20T21:32:05.555 INFO:tasks.workunit.client.0.smithi062.stdout:8/595: unlink d7/da9/cc8 0 2024-04-20T21:32:05.556 INFO:tasks.workunit.client.1.smithi189.stdout:8/368: dread f3 [0,4194304] 0 2024-04-20T21:32:05.557 INFO:tasks.workunit.client.0.smithi062.stdout:1/650: getdents dc/d70 0 2024-04-20T21:32:05.559 INFO:tasks.workunit.client.0.smithi062.stdout:6/466: dread d0/d9/f13 [0,4194304] 0 2024-04-20T21:32:05.562 INFO:tasks.workunit.client.1.smithi189.stdout:5/186: fdatasync d7/d14/d18/f17 0 2024-04-20T21:32:05.563 INFO:tasks.workunit.client.0.smithi062.stdout:8/596: symlink d7/d30/lcc 0 2024-04-20T21:32:05.569 INFO:tasks.workunit.client.0.smithi062.stdout:4/620: read da/dd/df/d27/d3f/f4c [3351162,12713] 0 2024-04-20T21:32:05.571 INFO:tasks.workunit.client.0.smithi062.stdout:0/583: rename d7/d1c/d15/d17/d1b/d74 to d7/d1c/d15/d17/d40/d3c/dc5 0 2024-04-20T21:32:05.571 INFO:tasks.workunit.client.0.smithi062.stdout:6/467: chown d0/d1a/d7d 273 1 2024-04-20T21:32:05.573 INFO:tasks.workunit.client.0.smithi062.stdout:7/628: symlink d7/d26/dac/lda 0 2024-04-20T21:32:05.573 INFO:tasks.workunit.client.0.smithi062.stdout:6/468: stat d0/c88 0 2024-04-20T21:32:05.576 INFO:tasks.workunit.client.0.smithi062.stdout:5/593: fdatasync d8/d38/d7a/d4c/d7d/d9d/dac/d66/dae/f64 0 2024-04-20T21:32:05.579 INFO:tasks.workunit.client.0.smithi062.stdout:1/651: creat dc/fe9 x:0 0 0 2024-04-20T21:32:05.580 INFO:tasks.workunit.client.0.smithi062.stdout:7/629: mkdir d7/d26/dac/ddb 0 2024-04-20T21:32:05.581 INFO:tasks.workunit.client.0.smithi062.stdout:6/469: creat d0/d52/f98 x:0 0 0 2024-04-20T21:32:05.582 INFO:tasks.workunit.client.0.smithi062.stdout:5/594: creat d8/d38/fbe x:0 0 0 2024-04-20T21:32:05.589 INFO:tasks.workunit.client.0.smithi062.stdout:4/621: getdents da/dd/d15 0 2024-04-20T21:32:05.591 INFO:tasks.workunit.client.1.smithi189.stdout:8/369: creat da/df/d13/d31/f80 x:0 0 0 2024-04-20T21:32:05.597 INFO:tasks.workunit.client.0.smithi062.stdout:0/584: getdents d7/d1c/d15/d17/d37 0 2024-04-20T21:32:05.598 INFO:tasks.workunit.client.0.smithi062.stdout:7/630: mkdir d7/d3e/d33/d3d/d5e/ddc 0 2024-04-20T21:32:05.598 INFO:tasks.workunit.client.0.smithi062.stdout:1/652: creat dc/d15/d20/fea x:0 0 0 2024-04-20T21:32:05.598 INFO:tasks.workunit.client.1.smithi189.stdout:0/367: rmdir d3/df 39 2024-04-20T21:32:05.598 INFO:tasks.workunit.client.1.smithi189.stdout:5/187: write d7/d9/d30/f2d [5101325,125339] 0 2024-04-20T21:32:05.600 INFO:tasks.workunit.client.1.smithi189.stdout:5/188: readlink l0 0 2024-04-20T21:32:05.600 INFO:tasks.workunit.client.0.smithi062.stdout:4/622: mknod da/cdc 0 2024-04-20T21:32:05.600 INFO:tasks.workunit.client.1.smithi189.stdout:5/189: read - d7/f2b zero size 2024-04-20T21:32:05.601 INFO:tasks.workunit.client.0.smithi062.stdout:9/624: dread d16/d3d/d55/faf [0,4194304] 0 2024-04-20T21:32:05.604 INFO:tasks.workunit.client.1.smithi189.stdout:1/231: mkdir d8/df/d50 0 2024-04-20T21:32:05.605 INFO:tasks.workunit.client.0.smithi062.stdout:7/631: write d7/d26/f9a [64816,75843] 0 2024-04-20T21:32:05.607 INFO:tasks.workunit.client.0.smithi062.stdout:9/625: dread - d16/d3e/d77/d84/f99 zero size 2024-04-20T21:32:05.607 INFO:tasks.workunit.client.0.smithi062.stdout:8/597: write d7/d9/f12 [5111384,48752] 0 2024-04-20T21:32:05.607 INFO:tasks.workunit.client.1.smithi189.stdout:4/233: truncate d2/f8 1184092 0 2024-04-20T21:32:05.607 INFO:tasks.workunit.client.1.smithi189.stdout:5/190: read d7/d14/d18/f17 [786377,96408] 0 2024-04-20T21:32:05.608 INFO:tasks.workunit.client.1.smithi189.stdout:4/234: chown d2/f32 13 1 2024-04-20T21:32:05.611 INFO:tasks.workunit.client.0.smithi062.stdout:8/598: readlink d7/d30/l53 0 2024-04-20T21:32:05.611 INFO:tasks.workunit.client.1.smithi189.stdout:4/235: chown d2/dd/c4e 1 1 2024-04-20T21:32:05.611 INFO:tasks.workunit.client.0.smithi062.stdout:0/585: rename d7/d1c/d15/d17/d40/d3c/dc5/lb8 to d7/d1c/d15/d17/lc6 0 2024-04-20T21:32:05.612 INFO:tasks.workunit.client.1.smithi189.stdout:4/236: fdatasync d2/f5a 0 2024-04-20T21:32:05.613 INFO:tasks.workunit.client.1.smithi189.stdout:6/243: dread d8/da/f37 [0,4194304] 0 2024-04-20T21:32:05.614 INFO:tasks.workunit.client.1.smithi189.stdout:9/280: sync 2024-04-20T21:32:05.615 INFO:tasks.workunit.client.0.smithi062.stdout:1/653: write dc/d15/d20/d28/d83/dc0/fc2 [992410,31857] 0 2024-04-20T21:32:05.616 INFO:tasks.workunit.client.1.smithi189.stdout:9/281: stat d9/dc/d17/d2b/d55 0 2024-04-20T21:32:05.616 INFO:tasks.workunit.client.0.smithi062.stdout:0/586: dread - d7/d1c/d15/d53/fbb zero size 2024-04-20T21:32:05.617 INFO:tasks.workunit.client.1.smithi189.stdout:2/214: dread d1/f5 [0,4194304] 0 2024-04-20T21:32:05.618 INFO:tasks.workunit.client.0.smithi062.stdout:4/623: rename da/dd/d15/d50/d64/d57/f80 to da/dd/d3c/dac/dce/fdd 0 2024-04-20T21:32:05.619 INFO:tasks.workunit.client.1.smithi189.stdout:1/232: symlink d8/d21/d2c/l51 0 2024-04-20T21:32:05.619 INFO:tasks.workunit.client.1.smithi189.stdout:8/370: mkdir da/d81 0 2024-04-20T21:32:05.621 INFO:tasks.workunit.client.1.smithi189.stdout:6/244: mknod d8/c4d 0 2024-04-20T21:32:05.621 INFO:tasks.workunit.client.0.smithi062.stdout:8/599: chown d7/d25/d3b/d58/d8b/cc0 15872 1 2024-04-20T21:32:05.621 INFO:tasks.workunit.client.0.smithi062.stdout:2/628: dread d7/d39/f45 [0,4194304] 0 2024-04-20T21:32:05.621 INFO:tasks.workunit.client.0.smithi062.stdout:9/626: chown d16/d2f/d5b/dab/cad 1 1 2024-04-20T21:32:05.622 INFO:tasks.workunit.client.1.smithi189.stdout:1/233: fsync d8/d21/f22 0 2024-04-20T21:32:05.623 INFO:tasks.workunit.client.1.smithi189.stdout:1/234: stat d8/f11 0 2024-04-20T21:32:05.623 INFO:tasks.workunit.client.1.smithi189.stdout:2/215: truncate d1/d11/d2d/f38 961590 0 2024-04-20T21:32:05.624 INFO:tasks.workunit.client.1.smithi189.stdout:4/237: symlink d2/d22/d57/l5b 0 2024-04-20T21:32:05.627 INFO:tasks.workunit.client.1.smithi189.stdout:8/371: symlink da/df/d13/l82 0 2024-04-20T21:32:05.630 INFO:tasks.workunit.client.1.smithi189.stdout:4/238: dread d2/dd/f44 [0,4194304] 0 2024-04-20T21:32:05.630 INFO:tasks.workunit.client.0.smithi062.stdout:2/629: creat d7/da/d28/d4b/d8f/fd0 x:0 0 0 2024-04-20T21:32:05.633 INFO:tasks.workunit.client.0.smithi062.stdout:5/595: dwrite d8/d10/f26 [0,4194304] 0 2024-04-20T21:32:05.633 INFO:tasks.workunit.client.0.smithi062.stdout:2/630: chown d7/da/fbd 223 1 2024-04-20T21:32:05.634 INFO:tasks.workunit.client.0.smithi062.stdout:0/587: write d7/d20/f5c [1110972,5280] 0 2024-04-20T21:32:05.636 INFO:tasks.workunit.client.1.smithi189.stdout:8/372: fdatasync da/de/f16 0 2024-04-20T21:32:05.637 INFO:tasks.workunit.client.0.smithi062.stdout:8/600: truncate d7/d25/d3b/d58/d8b/f1c 428709 0 2024-04-20T21:32:05.637 INFO:tasks.workunit.client.1.smithi189.stdout:2/216: rename d1/d17/l24 to d1/d17/d37/l3b 0 2024-04-20T21:32:05.640 INFO:tasks.workunit.client.0.smithi062.stdout:1/654: write dc/d15/d75/d96/dd4/d3a/da2/de4/f6c [2863412,47036] 0 2024-04-20T21:32:05.640 INFO:tasks.workunit.client.1.smithi189.stdout:0/368: dwrite d3/df/d30/d37/f4b [0,4194304] 0 2024-04-20T21:32:05.644 INFO:tasks.workunit.client.1.smithi189.stdout:2/217: chown d1/f2a 0 1 2024-04-20T21:32:05.647 INFO:tasks.workunit.client.0.smithi062.stdout:7/632: dwrite d7/d3e/d33/d3d/d5e/d7e/fa9 [0,4194304] 0 2024-04-20T21:32:05.650 INFO:tasks.workunit.client.0.smithi062.stdout:7/633: dread - d7/dcd/fd2 zero size 2024-04-20T21:32:05.650 INFO:tasks.workunit.client.0.smithi062.stdout:8/601: unlink d7/d25/d3b/d58/d8b/d24/fa7 0 2024-04-20T21:32:05.654 INFO:tasks.workunit.client.1.smithi189.stdout:9/282: dwrite d9/dc/d17/d2b/d55/f5a [0,4194304] 0 2024-04-20T21:32:05.656 INFO:tasks.workunit.client.0.smithi062.stdout:5/596: mknod d8/d38/d7a/db1/cbf 0 2024-04-20T21:32:05.658 INFO:tasks.workunit.client.1.smithi189.stdout:0/369: symlink d3/df/d28/d2b/d2c/d74/l7d 0 2024-04-20T21:32:05.666 INFO:tasks.workunit.client.1.smithi189.stdout:4/239: dwrite d2/d22/f30 [0,4194304] 0 2024-04-20T21:32:05.669 INFO:tasks.workunit.client.0.smithi062.stdout:1/655: symlink dc/d15/leb 0 2024-04-20T21:32:05.669 INFO:tasks.workunit.client.1.smithi189.stdout:6/245: stat d8/d10 0 2024-04-20T21:32:05.670 INFO:tasks.workunit.client.0.smithi062.stdout:7/634: creat d7/d3e/d33/d3d/d5e/d7e/d80/fdd x:0 0 0 2024-04-20T21:32:05.672 INFO:tasks.workunit.client.0.smithi062.stdout:8/602: creat d7/d25/d3b/d58/dac/fcd x:0 0 0 2024-04-20T21:32:05.675 INFO:tasks.workunit.client.0.smithi062.stdout:9/627: link d16/l8a d16/d3d/d55/d6e/ld0 0 2024-04-20T21:32:05.680 INFO:tasks.workunit.client.0.smithi062.stdout:8/603: write d7/d9/d29/f2b [216682,107873] 0 2024-04-20T21:32:05.681 INFO:tasks.workunit.client.1.smithi189.stdout:8/373: dwrite da/df/d3a/f4c [0,4194304] 0 2024-04-20T21:32:05.685 INFO:tasks.workunit.client.1.smithi189.stdout:1/235: dread d8/f1b [0,4194304] 0 2024-04-20T21:32:05.685 INFO:tasks.workunit.client.0.smithi062.stdout:7/635: chown d7/d3e/d33/d3d/d4d/d54/d4f/fc2 12047 1 2024-04-20T21:32:05.685 INFO:tasks.workunit.client.0.smithi062.stdout:2/631: dwrite d7/d39/f86 [0,4194304] 0 2024-04-20T21:32:05.686 INFO:tasks.workunit.client.0.smithi062.stdout:5/597: readlink d8/d1e/d2c/d6c/l96 0 2024-04-20T21:32:05.690 INFO:tasks.workunit.client.1.smithi189.stdout:2/218: rename d1/d17/d2f/l30 to d1/l3c 0 2024-04-20T21:32:05.693 INFO:tasks.workunit.client.1.smithi189.stdout:4/240: symlink d2/dd/l5c 0 2024-04-20T21:32:05.693 INFO:tasks.workunit.client.1.smithi189.stdout:1/236: fdatasync d8/f9 0 2024-04-20T21:32:05.694 INFO:tasks.workunit.client.0.smithi062.stdout:7/636: write d7/d3e/d33/d3d/d4d/d83/fc8 [428544,89756] 0 2024-04-20T21:32:05.696 INFO:tasks.workunit.client.1.smithi189.stdout:0/370: rename d3/df/d4c/f41 to d3/df/d4c/f7e 0 2024-04-20T21:32:05.696 INFO:tasks.workunit.client.1.smithi189.stdout:6/246: mkdir d8/d12/d4e 0 2024-04-20T21:32:05.699 INFO:tasks.workunit.client.1.smithi189.stdout:2/219: mknod d1/d11/c3d 0 2024-04-20T21:32:05.700 INFO:tasks.workunit.client.1.smithi189.stdout:0/371: fsync d3/df/f15 0 2024-04-20T21:32:05.701 INFO:tasks.workunit.client.0.smithi062.stdout:5/598: rmdir d8/d1e/d2c/d6c 39 2024-04-20T21:32:05.702 INFO:tasks.workunit.client.1.smithi189.stdout:6/247: rename d8/d12 to d8/d12/d30/d47/d4f 22 2024-04-20T21:32:05.703 INFO:tasks.workunit.client.1.smithi189.stdout:8/374: symlink da/de/d1e/d69/l83 0 2024-04-20T21:32:05.704 INFO:tasks.workunit.client.0.smithi062.stdout:7/637: rmdir d7/d26/da8 39 2024-04-20T21:32:05.705 INFO:tasks.workunit.client.0.smithi062.stdout:4/624: dread da/dd/d3c/f53 [0,4194304] 0 2024-04-20T21:32:05.706 INFO:tasks.workunit.client.1.smithi189.stdout:1/237: creat d8/d21/d2c/d36/f52 x:0 0 0 2024-04-20T21:32:05.706 INFO:tasks.workunit.client.1.smithi189.stdout:4/241: creat d2/d22/f5d x:0 0 0 2024-04-20T21:32:05.709 INFO:tasks.workunit.client.1.smithi189.stdout:6/248: chown d8/l40 0 1 2024-04-20T21:32:05.709 INFO:tasks.workunit.client.1.smithi189.stdout:1/238: fdatasync d8/d21/f38 0 2024-04-20T21:32:05.712 INFO:tasks.workunit.client.1.smithi189.stdout:7/266: sync 2024-04-20T21:32:05.712 INFO:tasks.workunit.client.1.smithi189.stdout:3/220: sync 2024-04-20T21:32:05.713 INFO:tasks.workunit.client.1.smithi189.stdout:8/375: mkdir da/de/d84 0 2024-04-20T21:32:05.713 INFO:tasks.workunit.client.1.smithi189.stdout:0/372: write d3/f66 [664494,54002] 0 2024-04-20T21:32:05.714 INFO:tasks.workunit.client.0.smithi062.stdout:7/638: readlink d7/d3e/d33/d3d/d5e/d7e/la6 0 2024-04-20T21:32:05.715 INFO:tasks.workunit.client.0.smithi062.stdout:9/628: dwrite d16/f19 [0,4194304] 0 2024-04-20T21:32:05.721 INFO:tasks.workunit.client.1.smithi189.stdout:9/283: dwrite d9/d12/d50/f4f [0,4194304] 0 2024-04-20T21:32:05.721 INFO:tasks.workunit.client.0.smithi062.stdout:1/656: dread dc/f6d [0,4194304] 0 2024-04-20T21:32:05.722 INFO:tasks.workunit.client.1.smithi189.stdout:4/242: creat d2/f5e x:0 0 0 2024-04-20T21:32:05.724 INFO:tasks.workunit.client.1.smithi189.stdout:6/249: creat d8/d12/d4e/f50 x:0 0 0 2024-04-20T21:32:05.724 INFO:tasks.workunit.client.0.smithi062.stdout:0/588: dwrite d7/d1c/d15/d17/d40/f85 [0,4194304] 0 2024-04-20T21:32:05.726 INFO:tasks.workunit.client.1.smithi189.stdout:5/191: dwrite d7/fb [0,4194304] 0 2024-04-20T21:32:05.728 INFO:tasks.workunit.client.1.smithi189.stdout:0/373: symlink d3/l7f 0 2024-04-20T21:32:05.729 INFO:tasks.workunit.client.1.smithi189.stdout:4/243: write d2/f1a [1697477,110701] 0 2024-04-20T21:32:05.731 INFO:tasks.workunit.client.0.smithi062.stdout:9/629: rmdir d16/d2c/d62 39 2024-04-20T21:32:05.733 INFO:tasks.workunit.client.0.smithi062.stdout:3/569: sync 2024-04-20T21:32:05.740 INFO:tasks.workunit.client.0.smithi062.stdout:2/632: dwrite d7/da/d15/d1b/d3d/f49 [0,4194304] 0 2024-04-20T21:32:05.742 INFO:tasks.workunit.client.1.smithi189.stdout:2/220: rmdir d1 39 2024-04-20T21:32:05.745 INFO:tasks.workunit.client.0.smithi062.stdout:7/639: rename d7/d3e/d33/f35 to d7/d3e/d33/d3d/d4d/dce/fde 0 2024-04-20T21:32:05.746 INFO:tasks.workunit.client.1.smithi189.stdout:9/284: creat d9/dc/d17/f5e x:0 0 0 2024-04-20T21:32:05.748 INFO:tasks.workunit.client.0.smithi062.stdout:7/640: write d7/d3e/f41 [1514192,3617] 0 2024-04-20T21:32:05.749 INFO:tasks.workunit.client.1.smithi189.stdout:4/244: readlink d2/l6 0 2024-04-20T21:32:05.750 INFO:tasks.workunit.client.0.smithi062.stdout:3/570: read d0/d1c/f51 [561973,82718] 0 2024-04-20T21:32:05.752 INFO:tasks.workunit.client.1.smithi189.stdout:8/376: mkdir da/df/d3c/d50/d6c/d85 0 2024-04-20T21:32:05.753 INFO:tasks.workunit.client.0.smithi062.stdout:7/641: chown d7/d3e/d33/l56 5 1 2024-04-20T21:32:05.754 INFO:tasks.workunit.client.1.smithi189.stdout:5/192: mkdir d7/d9/d31/d36 0 2024-04-20T21:32:05.757 INFO:tasks.workunit.client.0.smithi062.stdout:0/589: dwrite d7/f47 [0,4194304] 0 2024-04-20T21:32:05.759 INFO:tasks.workunit.client.1.smithi189.stdout:3/221: getdents d2/d3/d2c/d31 0 2024-04-20T21:32:05.760 INFO:tasks.workunit.client.0.smithi062.stdout:2/633: creat d7/da/d15/d2f/d38/d6b/fd1 x:0 0 0 2024-04-20T21:32:05.760 INFO:tasks.workunit.client.0.smithi062.stdout:9/630: rename d16/d3d/d55/d6e/d9b to d16/d2c/d62/dd1 0 2024-04-20T21:32:05.761 INFO:tasks.workunit.client.1.smithi189.stdout:7/267: dwrite d5/f43 [0,4194304] 0 2024-04-20T21:32:05.765 INFO:tasks.workunit.client.1.smithi189.stdout:1/239: dwrite f2 [0,4194304] 0 2024-04-20T21:32:05.765 INFO:tasks.workunit.client.0.smithi062.stdout:1/657: creat dc/d15/d75/d94/fec x:0 0 0 2024-04-20T21:32:05.766 INFO:tasks.workunit.client.0.smithi062.stdout:3/571: creat d0/d1b/d74/fbe x:0 0 0 2024-04-20T21:32:05.766 INFO:tasks.workunit.client.0.smithi062.stdout:0/590: chown d7/d1c/d15/d17/d40/d3c/da4 239623089 1 2024-04-20T21:32:05.766 INFO:tasks.workunit.client.0.smithi062.stdout:4/625: truncate da/dd/df/d27/d3f/d82/fbf 560795 0 2024-04-20T21:32:05.767 INFO:tasks.workunit.client.1.smithi189.stdout:4/245: symlink d2/d3a/l5f 0 2024-04-20T21:32:05.767 INFO:tasks.workunit.client.1.smithi189.stdout:2/221: unlink d1/fe 0 2024-04-20T21:32:05.767 INFO:tasks.workunit.client.0.smithi062.stdout:2/634: symlink d7/da/d15/d2f/d93/ld2 0 2024-04-20T21:32:05.769 INFO:tasks.workunit.client.1.smithi189.stdout:4/246: chown d2/f19 87 1 2024-04-20T21:32:05.770 INFO:tasks.workunit.client.1.smithi189.stdout:5/193: write d7/d9/d30/f1c [1918648,49907] 0 2024-04-20T21:32:05.771 INFO:tasks.workunit.client.1.smithi189.stdout:4/247: readlink d2/l6 0 2024-04-20T21:32:05.773 INFO:tasks.workunit.client.1.smithi189.stdout:3/222: mknod d2/d3/d2c/c4f 0 2024-04-20T21:32:05.773 INFO:tasks.workunit.client.1.smithi189.stdout:4/248: readlink d2/lc 0 2024-04-20T21:32:05.775 INFO:tasks.workunit.client.0.smithi062.stdout:2/635: fdatasync d7/da/d28/f6d 0 2024-04-20T21:32:05.776 INFO:tasks.workunit.client.1.smithi189.stdout:8/377: mkdir da/df/d3c/d68/d86 0 2024-04-20T21:32:05.777 INFO:tasks.workunit.client.1.smithi189.stdout:4/249: truncate d2/d3a/f4d 913857 0 2024-04-20T21:32:05.780 INFO:tasks.workunit.client.0.smithi062.stdout:3/572: mkdir d0/d1c/d52/dbf 0 2024-04-20T21:32:05.781 INFO:tasks.workunit.client.1.smithi189.stdout:7/268: rename d5/fd to d5/d46/f5d 0 2024-04-20T21:32:05.786 INFO:tasks.workunit.client.0.smithi062.stdout:2/636: fsync d7/da/d28/f6d 0 2024-04-20T21:32:05.788 INFO:tasks.workunit.client.0.smithi062.stdout:4/626: truncate da/dd/df/d27/d3f/f4c 4343306 0 2024-04-20T21:32:05.794 INFO:tasks.workunit.client.1.smithi189.stdout:2/222: symlink d1/l3e 0 2024-04-20T21:32:05.794 INFO:tasks.workunit.client.1.smithi189.stdout:2/223: write d1/f12 [4884391,72959] 0 2024-04-20T21:32:05.795 INFO:tasks.workunit.client.0.smithi062.stdout:8/604: dread d7/d30/f3c [0,4194304] 0 2024-04-20T21:32:05.798 INFO:tasks.workunit.client.1.smithi189.stdout:8/378: mkdir da/de/d1e/d21/d87 0 2024-04-20T21:32:05.799 INFO:tasks.workunit.client.0.smithi062.stdout:1/658: rmdir dc/d15/d75/d94/d6f/dd2/de3/de5 0 2024-04-20T21:32:05.800 INFO:tasks.workunit.client.0.smithi062.stdout:3/573: unlink d0/d1/d10/d3a/lb0 0 2024-04-20T21:32:05.803 INFO:tasks.workunit.client.0.smithi062.stdout:4/627: mknod da/dd/d15/d50/d64/d57/d5b/cde 0 2024-04-20T21:32:05.805 INFO:tasks.workunit.client.0.smithi062.stdout:6/470: sync 2024-04-20T21:32:05.807 INFO:tasks.workunit.client.1.smithi189.stdout:6/250: dread d8/d12/f17 [0,4194304] 0 2024-04-20T21:32:05.808 INFO:tasks.workunit.client.1.smithi189.stdout:5/194: rename d7/d9/ce to d7/d9/d31/d36/c37 0 2024-04-20T21:32:05.810 INFO:tasks.workunit.client.1.smithi189.stdout:6/251: fdatasync d8/f21 0 2024-04-20T21:32:05.811 INFO:tasks.workunit.client.1.smithi189.stdout:5/195: chown d7/d14/d1b/d1f/d33 0 1 2024-04-20T21:32:05.815 INFO:tasks.workunit.client.0.smithi062.stdout:9/631: dwrite d16/d2c/d5e/d75/d76/fb6 [0,4194304] 0 2024-04-20T21:32:05.818 INFO:tasks.workunit.client.0.smithi062.stdout:6/471: fdatasync d0/d9/f16 0 2024-04-20T21:32:05.820 INFO:tasks.workunit.client.0.smithi062.stdout:1/659: read - dc/d15/d75/d96/dd4/d78/dab/fb7 zero size 2024-04-20T21:32:05.821 INFO:tasks.workunit.client.0.smithi062.stdout:9/632: dread - d16/d3e/fa2 zero size 2024-04-20T21:32:05.822 INFO:tasks.workunit.client.0.smithi062.stdout:2/637: rename d7/da/fab to d7/da/d15/fd3 0 2024-04-20T21:32:05.822 INFO:tasks.workunit.client.0.smithi062.stdout:6/472: chown d0/d4f/f73 1 1 2024-04-20T21:32:05.823 INFO:tasks.workunit.client.1.smithi189.stdout:2/224: mknod d1/d11/c3f 0 2024-04-20T21:32:05.825 INFO:tasks.workunit.client.1.smithi189.stdout:2/225: chown d1/d11/c14 9508 1 2024-04-20T21:32:05.829 INFO:tasks.workunit.client.0.smithi062.stdout:5/599: dread d8/d10/d3e/f52 [0,4194304] 0 2024-04-20T21:32:05.829 INFO:tasks.workunit.client.0.smithi062.stdout:2/638: write d7/da/d15/d1b/f34 [2110101,70400] 0 2024-04-20T21:32:05.830 INFO:tasks.workunit.client.0.smithi062.stdout:9/633: write d16/d2f/d5b/dab/fb2 [2051728,129379] 0 2024-04-20T21:32:05.834 INFO:tasks.workunit.client.1.smithi189.stdout:6/252: fdatasync d8/da/dd/f28 0 2024-04-20T21:32:05.835 INFO:tasks.workunit.client.1.smithi189.stdout:5/196: dwrite d7/d14/f26 [0,4194304] 0 2024-04-20T21:32:05.838 INFO:tasks.workunit.client.1.smithi189.stdout:5/197: stat d7/d14/d1b/d1f/d2a 0 2024-04-20T21:32:05.839 INFO:tasks.workunit.client.0.smithi062.stdout:8/605: link d7/c43 d7/d25/d3b/d58/d8b/d24/dbd/cce 0 2024-04-20T21:32:05.844 INFO:tasks.workunit.client.1.smithi189.stdout:2/226: symlink d1/d11/l40 0 2024-04-20T21:32:05.844 INFO:tasks.workunit.client.1.smithi189.stdout:3/223: rename d2/f5 to d2/d3/d2c/d31/f50 0 2024-04-20T21:32:05.844 INFO:tasks.workunit.client.0.smithi062.stdout:6/473: mkdir d0/d9/d11/d7b/d99 0 2024-04-20T21:32:05.845 INFO:tasks.workunit.client.1.smithi189.stdout:1/240: write d8/f1b [4319328,84595] 0 2024-04-20T21:32:05.846 INFO:tasks.workunit.client.0.smithi062.stdout:6/474: fdatasync d0/d52/f83 0 2024-04-20T21:32:05.848 INFO:tasks.workunit.client.0.smithi062.stdout:0/591: dread d7/d1e/f45 [0,4194304] 0 2024-04-20T21:32:05.849 INFO:tasks.workunit.client.1.smithi189.stdout:5/198: getdents d7/d14/d1b/d1f/d2a 0 2024-04-20T21:32:05.850 INFO:tasks.workunit.client.0.smithi062.stdout:0/592: chown d7/d1e/c6a 3056 1 2024-04-20T21:32:05.851 INFO:tasks.workunit.client.1.smithi189.stdout:2/227: creat d1/d11/d2d/f41 x:0 0 0 2024-04-20T21:32:05.851 INFO:tasks.workunit.client.0.smithi062.stdout:9/634: creat d16/d2c/d42/da6/fd2 x:0 0 0 2024-04-20T21:32:05.855 INFO:tasks.workunit.client.1.smithi189.stdout:6/253: rename d8/d10/f41 to d8/d12/d4e/f51 0 2024-04-20T21:32:05.857 INFO:tasks.workunit.client.0.smithi062.stdout:1/660: rename dc/d15/d75/d96/dd4/d78/dab/lca to dc/d15/d75/d96/dd4/d3a/led 0 2024-04-20T21:32:05.858 INFO:tasks.workunit.client.1.smithi189.stdout:3/224: symlink d2/d3/d2c/l51 0 2024-04-20T21:32:05.858 INFO:tasks.workunit.client.1.smithi189.stdout:6/254: read - d8/da/f27 zero size 2024-04-20T21:32:05.862 INFO:tasks.workunit.client.0.smithi062.stdout:5/600: truncate d8/d38/d7a/d4c/d7d/f98 981590 0 2024-04-20T21:32:05.866 INFO:tasks.workunit.client.1.smithi189.stdout:2/228: dread d1/f5 [0,4194304] 0 2024-04-20T21:32:05.866 INFO:tasks.workunit.client.1.smithi189.stdout:1/241: symlink d8/df/d3a/d49/l53 0 2024-04-20T21:32:05.866 INFO:tasks.workunit.client.1.smithi189.stdout:7/269: truncate d5/d21/d35/d53/d5b/f2f 1582749 0 2024-04-20T21:32:05.866 INFO:tasks.workunit.client.1.smithi189.stdout:5/199: creat d7/d9/d31/d36/f38 x:0 0 0 2024-04-20T21:32:05.866 INFO:tasks.workunit.client.1.smithi189.stdout:3/225: rename d2/d8/f4a to d2/d3/f52 0 2024-04-20T21:32:05.869 INFO:tasks.workunit.client.0.smithi062.stdout:5/601: chown d8/d38/d7a/d4c/d7d/d9d/dac/d66/dae/fa3 17 1 2024-04-20T21:32:05.869 INFO:tasks.workunit.client.1.smithi189.stdout:8/379: truncate da/df/d3c/d50/f61 4089887 0 2024-04-20T21:32:05.869 INFO:tasks.workunit.client.1.smithi189.stdout:1/242: mkdir d8/d1e/d54 0 2024-04-20T21:32:05.870 INFO:tasks.workunit.client.1.smithi189.stdout:8/380: read - da/df/d20/f7d zero size 2024-04-20T21:32:05.871 INFO:tasks.workunit.client.0.smithi062.stdout:8/606: dread d7/d30/fba [0,4194304] 0 2024-04-20T21:32:05.874 INFO:tasks.workunit.client.0.smithi062.stdout:0/593: chown d7/d1c/d15/d17/d37/d95/cb4 9 1 2024-04-20T21:32:05.874 INFO:tasks.workunit.client.1.smithi189.stdout:5/200: creat d7/d14/d1b/d1f/d2a/f39 x:0 0 0 2024-04-20T21:32:05.876 INFO:tasks.workunit.client.0.smithi062.stdout:1/661: mknod dc/d15/d75/d94/d6f/dd2/de3/cee 0 2024-04-20T21:32:05.876 INFO:tasks.workunit.client.1.smithi189.stdout:5/201: fsync d7/f2b 0 2024-04-20T21:32:05.876 INFO:tasks.workunit.client.0.smithi062.stdout:9/635: write d16/d2c/d42/da6/fce [2650186,52695] 0 2024-04-20T21:32:05.877 INFO:tasks.workunit.client.1.smithi189.stdout:3/226: rename d2/d8/c17 to d2/c53 0 2024-04-20T21:32:05.878 INFO:tasks.workunit.client.0.smithi062.stdout:9/636: fdatasync d16/fa5 0 2024-04-20T21:32:05.879 INFO:tasks.workunit.client.0.smithi062.stdout:4/628: write da/dd/df/d27/f2c [5020172,125985] 0 2024-04-20T21:32:05.880 INFO:tasks.workunit.client.0.smithi062.stdout:6/475: mkdir d0/d9a 0 2024-04-20T21:32:05.882 INFO:tasks.workunit.client.1.smithi189.stdout:8/381: mknod da/df/d13/c88 0 2024-04-20T21:32:05.887 INFO:tasks.workunit.client.1.smithi189.stdout:6/255: dwrite d8/d10/f3f [0,4194304] 0 2024-04-20T21:32:05.887 INFO:tasks.workunit.client.1.smithi189.stdout:3/227: creat d2/d3/d25/f54 x:0 0 0 2024-04-20T21:32:05.890 INFO:tasks.workunit.client.1.smithi189.stdout:1/243: truncate d8/f14 2715897 0 2024-04-20T21:32:05.892 INFO:tasks.workunit.client.1.smithi189.stdout:3/228: chown d2/d3/d2c/d31 8 1 2024-04-20T21:32:05.892 INFO:tasks.workunit.client.0.smithi062.stdout:5/602: chown d8/c20 3081183 1 2024-04-20T21:32:05.893 INFO:tasks.workunit.client.1.smithi189.stdout:1/244: dread - d8/d21/d2c/d36/f52 zero size 2024-04-20T21:32:05.895 INFO:tasks.workunit.client.1.smithi189.stdout:3/229: fsync d2/fd 0 2024-04-20T21:32:05.895 INFO:tasks.workunit.client.0.smithi062.stdout:2/639: dread d7/da/d28/f6d [0,4194304] 0 2024-04-20T21:32:05.896 INFO:tasks.workunit.client.1.smithi189.stdout:4/250: dread d2/dd/f2d [0,4194304] 0 2024-04-20T21:32:05.900 INFO:tasks.workunit.client.0.smithi062.stdout:1/662: chown dc/d15/d75/d94/f34 13691 1 2024-04-20T21:32:05.906 INFO:tasks.workunit.client.0.smithi062.stdout:9/637: rmdir d16/d2c/d42/da6/da9 39 2024-04-20T21:32:05.907 INFO:tasks.workunit.client.0.smithi062.stdout:8/607: truncate d7/f23 3052279 0 2024-04-20T21:32:05.908 INFO:tasks.workunit.client.0.smithi062.stdout:3/574: dwrite d0/d1c/d63/d42/f60 [0,4194304] 0 2024-04-20T21:32:05.908 INFO:tasks.workunit.client.0.smithi062.stdout:6/476: truncate d0/d9/d11/f1f 4624352 0 2024-04-20T21:32:05.913 INFO:tasks.workunit.client.1.smithi189.stdout:7/270: dread d5/d21/d35/d53/d5b/f2f [0,4194304] 0 2024-04-20T21:32:05.916 INFO:tasks.workunit.client.1.smithi189.stdout:8/382: creat da/f89 x:0 0 0 2024-04-20T21:32:05.917 INFO:tasks.workunit.client.1.smithi189.stdout:1/245: mknod d8/df/c55 0 2024-04-20T21:32:05.919 INFO:tasks.workunit.client.0.smithi062.stdout:8/608: chown d7/d25/d3b/d58/d8b/d24/d46/c81 5 1 2024-04-20T21:32:05.919 INFO:tasks.workunit.client.0.smithi062.stdout:5/603: dread d8/d1e/d2c/d6c/f9b [0,4194304] 0 2024-04-20T21:32:05.920 INFO:tasks.workunit.client.0.smithi062.stdout:1/663: symlink dc/d15/d75/d96/dd4/d3a/lef 0 2024-04-20T21:32:05.922 INFO:tasks.workunit.client.1.smithi189.stdout:4/251: rename d2/d3a/l5f to d2/dd/d11/d14/l60 0 2024-04-20T21:32:05.924 INFO:tasks.workunit.client.0.smithi062.stdout:6/477: symlink d0/d4f/l9b 0 2024-04-20T21:32:05.926 INFO:tasks.workunit.client.0.smithi062.stdout:4/629: dwrite da/dd/d15/d4b/fd9 [0,4194304] 0 2024-04-20T21:32:05.927 INFO:tasks.workunit.client.1.smithi189.stdout:7/271: unlink d5/d46/f5d 0 2024-04-20T21:32:05.928 INFO:tasks.workunit.client.1.smithi189.stdout:1/246: symlink d8/d1e/d43/d44/l56 0 2024-04-20T21:32:05.930 INFO:tasks.workunit.client.1.smithi189.stdout:4/252: write d2/d3a/f4d [1205373,90166] 0 2024-04-20T21:32:05.931 INFO:tasks.workunit.client.0.smithi062.stdout:9/638: dread d16/f2e [0,4194304] 0 2024-04-20T21:32:05.934 INFO:tasks.workunit.client.0.smithi062.stdout:9/639: dread - d16/d3d/d55/d6e/f9d zero size 2024-04-20T21:32:05.935 INFO:tasks.workunit.client.1.smithi189.stdout:5/202: dwrite d7/d14/d18/f17 [0,4194304] 0 2024-04-20T21:32:05.936 INFO:tasks.workunit.client.0.smithi062.stdout:9/640: chown d16/f66 24 1 2024-04-20T21:32:05.937 INFO:tasks.workunit.client.1.smithi189.stdout:6/256: getdents d8/d12 0 2024-04-20T21:32:05.937 INFO:tasks.workunit.client.1.smithi189.stdout:3/230: link d2/d3/d2c/d30/l38 d2/d8/l55 0 2024-04-20T21:32:05.937 INFO:tasks.workunit.client.0.smithi062.stdout:3/575: truncate d0/d1/d3c/d43/d72/da9/fab 950982 0 2024-04-20T21:32:05.938 INFO:tasks.workunit.client.0.smithi062.stdout:4/630: creat da/dd/df/d27/d3f/fdf x:0 0 0 2024-04-20T21:32:05.939 INFO:tasks.workunit.client.1.smithi189.stdout:6/257: readlink d8/d12/d30/l4a 0 2024-04-20T21:32:05.940 INFO:tasks.workunit.client.1.smithi189.stdout:4/253: chown d2/dd/d11/f31 497458971 1 2024-04-20T21:32:05.941 INFO:tasks.workunit.client.1.smithi189.stdout:4/254: readlink d2/l5 0 2024-04-20T21:32:05.942 INFO:tasks.workunit.client.1.smithi189.stdout:7/272: stat d5/ff 0 2024-04-20T21:32:05.943 INFO:tasks.workunit.client.1.smithi189.stdout:1/247: readlink d8/d1e/d28/l30 0 2024-04-20T21:32:05.944 INFO:tasks.workunit.client.1.smithi189.stdout:3/231: stat d2/c53 0 2024-04-20T21:32:05.945 INFO:tasks.workunit.client.0.smithi062.stdout:6/478: creat d0/d9/f9c x:0 0 0 2024-04-20T21:32:05.946 INFO:tasks.workunit.client.1.smithi189.stdout:6/258: creat d8/d12/d30/d47/f52 x:0 0 0 2024-04-20T21:32:05.950 INFO:tasks.workunit.client.1.smithi189.stdout:4/255: mknod d2/dd/d37/c61 0 2024-04-20T21:32:05.950 INFO:tasks.workunit.client.1.smithi189.stdout:3/232: write d2/d3/f48 [909221,126946] 0 2024-04-20T21:32:05.950 INFO:tasks.workunit.client.0.smithi062.stdout:0/594: truncate d7/d1c/d15/d17/d40/d79/f57 3294150 0 2024-04-20T21:32:05.951 INFO:tasks.workunit.client.1.smithi189.stdout:5/203: symlink d7/l3a 0 2024-04-20T21:32:05.952 INFO:tasks.workunit.client.1.smithi189.stdout:1/248: mknod d8/d21/d2c/c57 0 2024-04-20T21:32:05.954 INFO:tasks.workunit.client.1.smithi189.stdout:5/204: write d7/f2b [897668,52631] 0 2024-04-20T21:32:05.954 INFO:tasks.workunit.client.0.smithi062.stdout:6/479: mknod d0/d9/d11/d7b/c9d 0 2024-04-20T21:32:05.955 INFO:tasks.workunit.client.1.smithi189.stdout:7/273: dread - d5/d1d/f56 zero size 2024-04-20T21:32:05.956 INFO:tasks.workunit.client.0.smithi062.stdout:0/595: fsync d7/d1c/d15/d17/d26/fc0 0 2024-04-20T21:32:05.958 INFO:tasks.workunit.client.0.smithi062.stdout:9/641: truncate d16/f2e 1621837 0 2024-04-20T21:32:05.961 INFO:tasks.workunit.client.1.smithi189.stdout:1/249: rmdir d8/df 39 2024-04-20T21:32:05.962 INFO:tasks.workunit.client.0.smithi062.stdout:5/604: dwrite d8/d1e/d2c/fa2 [0,4194304] 0 2024-04-20T21:32:05.963 INFO:tasks.workunit.client.0.smithi062.stdout:1/664: getdents dc/d15/d75/d96/dd4/d78/dab 0 2024-04-20T21:32:05.965 INFO:tasks.workunit.client.0.smithi062.stdout:9/642: chown d16/d3d/d55/d6e/d7d/lba 228161 1 2024-04-20T21:32:05.966 INFO:tasks.workunit.client.0.smithi062.stdout:4/631: link da/dd/d15/d4b/l75 da/dd/d15/d50/d64/d57/d5b/d9d/le0 0 2024-04-20T21:32:05.967 INFO:tasks.workunit.client.1.smithi189.stdout:4/256: getdents d2/d36/d56 0 2024-04-20T21:32:05.967 INFO:tasks.workunit.client.1.smithi189.stdout:6/259: creat d8/d2b/f53 x:0 0 0 2024-04-20T21:32:05.968 INFO:tasks.workunit.client.1.smithi189.stdout:3/233: fdatasync d2/d3/d2c/d31/f50 0 2024-04-20T21:32:05.969 INFO:tasks.workunit.client.0.smithi062.stdout:8/609: dwrite d7/d9/d29/f2b [0,4194304] 0 2024-04-20T21:32:05.969 INFO:tasks.workunit.client.1.smithi189.stdout:6/260: write d8/d12/f44 [4519001,89640] 0 2024-04-20T21:32:05.973 INFO:tasks.workunit.client.1.smithi189.stdout:6/261: chown d8/da/dd/c46 217568 1 2024-04-20T21:32:05.979 INFO:tasks.workunit.client.1.smithi189.stdout:5/205: dwrite d7/d14/d1b/d1f/d2a/f39 [0,4194304] 0 2024-04-20T21:32:05.979 INFO:tasks.workunit.client.1.smithi189.stdout:6/262: write d8/d12/f44 [4512842,78256] 0 2024-04-20T21:32:05.980 INFO:tasks.workunit.client.0.smithi062.stdout:0/596: creat d7/d1c/d15/d17/d40/d3c/dc5/fc7 x:0 0 0 2024-04-20T21:32:05.986 INFO:tasks.workunit.client.1.smithi189.stdout:2/229: sync 2024-04-20T21:32:05.987 INFO:tasks.workunit.client.1.smithi189.stdout:0/374: sync 2024-04-20T21:32:05.987 INFO:tasks.workunit.client.1.smithi189.stdout:9/285: sync 2024-04-20T21:32:05.988 INFO:tasks.workunit.client.1.smithi189.stdout:1/250: dwrite f2 [0,4194304] 0 2024-04-20T21:32:05.992 INFO:tasks.workunit.client.0.smithi062.stdout:9/643: rename d16/d3d/f7a to d16/d3d/d55/d6e/fd3 0 2024-04-20T21:32:05.994 INFO:tasks.workunit.client.0.smithi062.stdout:1/665: dread dc/d15/d20/d28/d83/dc0/fc2 [0,4194304] 0 2024-04-20T21:32:05.999 INFO:tasks.workunit.client.0.smithi062.stdout:0/597: rmdir d7/d1e/db6 39 2024-04-20T21:32:06.000 INFO:tasks.workunit.client.0.smithi062.stdout:3/576: dwrite d0/d1c/f45 [0,4194304] 0 2024-04-20T21:32:06.001 INFO:tasks.workunit.client.0.smithi062.stdout:6/480: dwrite d0/d9/f47 [0,4194304] 0 2024-04-20T21:32:06.002 INFO:tasks.workunit.client.1.smithi189.stdout:8/383: dread da/de/d1e/d21/d44/f4a [0,4194304] 0 2024-04-20T21:32:06.002 INFO:tasks.workunit.client.1.smithi189.stdout:9/286: write d9/d12/d15/d1a/f5c [181213,45685] 0 2024-04-20T21:32:06.011 INFO:tasks.workunit.client.1.smithi189.stdout:9/287: chown d9/d12/d50 62880 1 2024-04-20T21:32:06.013 INFO:tasks.workunit.client.0.smithi062.stdout:9/644: mknod d16/d2c/d62/dd1/cd4 0 2024-04-20T21:32:06.013 INFO:tasks.workunit.client.0.smithi062.stdout:3/577: chown d0/d1/d3c/d43/d72/da9 7725939 1 2024-04-20T21:32:06.015 INFO:tasks.workunit.client.1.smithi189.stdout:3/234: dwrite d2/d3/f52 [0,4194304] 0 2024-04-20T21:32:06.016 INFO:tasks.workunit.client.0.smithi062.stdout:4/632: dread da/dd/d3c/f53 [0,4194304] 0 2024-04-20T21:32:06.019 INFO:tasks.workunit.client.0.smithi062.stdout:5/605: dread d8/d38/d7a/d4c/d7d/d9d/dac/d2f/d8e/f97 [0,4194304] 0 2024-04-20T21:32:06.026 INFO:tasks.workunit.client.1.smithi189.stdout:2/230: read d1/d17/f34 [4195614,125754] 0 2024-04-20T21:32:06.027 INFO:tasks.workunit.client.1.smithi189.stdout:0/375: dwrite d3/fe [0,4194304] 0 2024-04-20T21:32:06.028 INFO:tasks.workunit.client.1.smithi189.stdout:2/231: write d1/f9 [4333470,57452] 0 2024-04-20T21:32:06.029 INFO:tasks.workunit.client.0.smithi062.stdout:9/645: creat d16/d3e/fd5 x:0 0 0 2024-04-20T21:32:06.031 INFO:tasks.workunit.client.1.smithi189.stdout:2/232: dread - d1/d11/f28 zero size 2024-04-20T21:32:06.032 INFO:tasks.workunit.client.0.smithi062.stdout:3/578: symlink d0/d5c/lc0 0 2024-04-20T21:32:06.036 INFO:tasks.workunit.client.0.smithi062.stdout:0/598: dwrite d7/d1e/d6e/f75 [4194304,4194304] 0 2024-04-20T21:32:06.038 INFO:tasks.workunit.client.1.smithi189.stdout:5/206: mknod d7/d14/d18/c3b 0 2024-04-20T21:32:06.040 INFO:tasks.workunit.client.0.smithi062.stdout:6/481: dread d0/d9/f47 [0,4194304] 0 2024-04-20T21:32:06.040 INFO:tasks.workunit.client.0.smithi062.stdout:7/642: sync 2024-04-20T21:32:06.040 INFO:tasks.workunit.client.0.smithi062.stdout:3/579: dread - d0/d1/f82 zero size 2024-04-20T21:32:06.041 INFO:tasks.workunit.client.1.smithi189.stdout:7/274: dread d5/d21/f32 [0,4194304] 0 2024-04-20T21:32:06.041 INFO:tasks.workunit.client.1.smithi189.stdout:5/207: read d7/d14/f26 [2401109,43662] 0 2024-04-20T21:32:06.046 INFO:tasks.workunit.client.1.smithi189.stdout:5/208: write d7/d14/d18/f17 [1099747,12247] 0 2024-04-20T21:32:06.047 INFO:tasks.workunit.client.0.smithi062.stdout:4/633: rename da/dd/d15/d4b/l75 to da/dd/df/d27/d3f/le1 0 2024-04-20T21:32:06.048 INFO:tasks.workunit.client.1.smithi189.stdout:5/209: chown d7/d9/d30/f1c 170297 1 2024-04-20T21:32:06.050 INFO:tasks.workunit.client.1.smithi189.stdout:5/210: write d7/d14/f26 [1244531,117113] 0 2024-04-20T21:32:06.051 INFO:tasks.workunit.client.1.smithi189.stdout:5/211: fdatasync d7/d9/d30/f2d 0 2024-04-20T21:32:06.052 INFO:tasks.workunit.client.0.smithi062.stdout:1/666: write dc/d15/d20/d28/f63 [486224,122977] 0 2024-04-20T21:32:06.054 INFO:tasks.workunit.client.1.smithi189.stdout:6/263: mkdir d8/d2b/d49/d54 0 2024-04-20T21:32:06.056 INFO:tasks.workunit.client.1.smithi189.stdout:5/212: write d7/d14/d1b/d1f/d2a/f39 [3972105,17927] 0 2024-04-20T21:32:06.056 INFO:tasks.workunit.client.0.smithi062.stdout:5/606: creat d8/d38/d7a/d4c/d7d/d9d/dac/d2f/fc0 x:0 0 0 2024-04-20T21:32:06.059 INFO:tasks.workunit.client.1.smithi189.stdout:9/288: write d9/f34 [2764424,21441] 0 2024-04-20T21:32:06.061 INFO:tasks.workunit.client.0.smithi062.stdout:3/580: mknod d0/d1/d10/d31/d41/cc1 0 2024-04-20T21:32:06.064 INFO:tasks.workunit.client.1.smithi189.stdout:3/235: symlink d2/d3/d2c/l56 0 2024-04-20T21:32:06.065 INFO:tasks.workunit.client.0.smithi062.stdout:1/667: write dc/d27/d32/f89 [4220293,115652] 0 2024-04-20T21:32:06.068 INFO:tasks.workunit.client.0.smithi062.stdout:4/634: write da/dd/d3c/dac/dce/fdd [1528680,64626] 0 2024-04-20T21:32:06.070 INFO:tasks.workunit.client.0.smithi062.stdout:2/640: sync 2024-04-20T21:32:06.076 INFO:tasks.workunit.client.1.smithi189.stdout:6/264: dwrite d8/da/f27 [0,4194304] 0 2024-04-20T21:32:06.079 INFO:tasks.workunit.client.0.smithi062.stdout:1/668: read dc/d15/d75/d96/dd4/d78/faa [788841,86714] 0 2024-04-20T21:32:06.082 INFO:tasks.workunit.client.0.smithi062.stdout:7/643: creat d7/d3e/d33/d3d/d4d/fdf x:0 0 0 2024-04-20T21:32:06.084 INFO:tasks.workunit.client.1.smithi189.stdout:7/275: write d5/f49 [896022,9268] 0 2024-04-20T21:32:06.085 INFO:tasks.workunit.client.0.smithi062.stdout:3/581: dread - d0/d1c/d63/f91 zero size 2024-04-20T21:32:06.089 INFO:tasks.workunit.client.0.smithi062.stdout:9/646: dwrite d16/d3e/d77/d84/f9a [0,4194304] 0 2024-04-20T21:32:06.089 INFO:tasks.workunit.client.1.smithi189.stdout:8/384: creat da/df/d3c/d50/d6c/d85/f8a x:0 0 0 2024-04-20T21:32:06.093 INFO:tasks.workunit.client.1.smithi189.stdout:8/385: chown da/de/d1e/d21/d44/f47 310 1 2024-04-20T21:32:06.094 INFO:tasks.workunit.client.1.smithi189.stdout:3/236: creat d2/d8/f57 x:0 0 0 2024-04-20T21:32:06.095 INFO:tasks.workunit.client.0.smithi062.stdout:2/641: symlink d7/da/d15/d2f/d38/db0/dc4/ld4 0 2024-04-20T21:32:06.096 INFO:tasks.workunit.client.0.smithi062.stdout:7/644: rmdir d7 39 2024-04-20T21:32:06.097 INFO:tasks.workunit.client.1.smithi189.stdout:6/265: mknod d8/da/dd/d2a/d3b/c55 0 2024-04-20T21:32:06.098 INFO:tasks.workunit.client.1.smithi189.stdout:2/233: dwrite d1/d11/f1a [0,4194304] 0 2024-04-20T21:32:06.101 INFO:tasks.workunit.client.1.smithi189.stdout:4/257: dwrite d2/d20/f24 [0,4194304] 0 2024-04-20T21:32:06.101 INFO:tasks.workunit.client.1.smithi189.stdout:2/234: chown d1/f12 534167 1 2024-04-20T21:32:06.102 INFO:tasks.workunit.client.0.smithi062.stdout:8/610: dread d7/f23 [0,4194304] 0 2024-04-20T21:32:06.103 INFO:tasks.workunit.client.0.smithi062.stdout:3/582: creat d0/d1c/d52/d7b/fc2 x:0 0 0 2024-04-20T21:32:06.103 INFO:tasks.workunit.client.0.smithi062.stdout:6/482: dwrite d0/d9/f16 [0,4194304] 0 2024-04-20T21:32:06.104 INFO:tasks.workunit.client.1.smithi189.stdout:7/276: creat d5/d21/d35/d53/d5b/f5e x:0 0 0 2024-04-20T21:32:06.104 INFO:tasks.workunit.client.1.smithi189.stdout:9/289: mknod d9/dc/d17/d36/d5d/c5f 0 2024-04-20T21:32:06.104 INFO:tasks.workunit.client.1.smithi189.stdout:2/235: write d1/f7 [3891658,127650] 0 2024-04-20T21:32:06.105 INFO:tasks.workunit.client.1.smithi189.stdout:7/277: chown d5 100421910 1 2024-04-20T21:32:06.109 INFO:tasks.workunit.client.1.smithi189.stdout:6/266: symlink d8/da/dd/l56 0 2024-04-20T21:32:06.109 INFO:tasks.workunit.client.0.smithi062.stdout:1/669: write dc/d15/d20/d28/f4e [1283556,333] 0 2024-04-20T21:32:06.110 INFO:tasks.workunit.client.0.smithi062.stdout:3/583: fsync d0/d1/d10/f4e 0 2024-04-20T21:32:06.110 INFO:tasks.workunit.client.1.smithi189.stdout:6/267: read d8/d12/d30/f38 [1078824,20006] 0 2024-04-20T21:32:06.114 INFO:tasks.workunit.client.1.smithi189.stdout:3/237: dread d2/f29 [0,4194304] 0 2024-04-20T21:32:06.114 INFO:tasks.workunit.client.1.smithi189.stdout:6/268: chown d8/d12/d4e/f51 4256342 1 2024-04-20T21:32:06.115 INFO:tasks.workunit.client.1.smithi189.stdout:4/258: unlink d2/d3a/f3b 0 2024-04-20T21:32:06.116 INFO:tasks.workunit.client.0.smithi062.stdout:9/647: creat d16/d3d/d55/d6e/d7d/fd6 x:0 0 0 2024-04-20T21:32:06.117 INFO:tasks.workunit.client.0.smithi062.stdout:5/607: rename d8/d1e/d2c/d6c/f3c to d8/d38/d7a/d4c/fc1 0 2024-04-20T21:32:06.118 INFO:tasks.workunit.client.0.smithi062.stdout:2/642: fdatasync d7/da/d15/d2f/fa4 0 2024-04-20T21:32:06.119 INFO:tasks.workunit.client.0.smithi062.stdout:8/611: fsync d7/d30/f78 0 2024-04-20T21:32:06.120 INFO:tasks.workunit.client.0.smithi062.stdout:9/648: dread - d16/d3d/d55/fbf zero size 2024-04-20T21:32:06.121 INFO:tasks.workunit.client.1.smithi189.stdout:1/251: rename d8/df/d4e to d8/d1e/d43/d58 0 2024-04-20T21:32:06.121 INFO:tasks.workunit.client.0.smithi062.stdout:1/670: write dc/fe9 [1040869,112926] 0 2024-04-20T21:32:06.128 INFO:tasks.workunit.client.1.smithi189.stdout:7/278: dwrite d5/ff [0,4194304] 0 2024-04-20T21:32:06.133 INFO:tasks.workunit.client.1.smithi189.stdout:2/236: fsync d1/d17/f34 0 2024-04-20T21:32:06.136 INFO:tasks.workunit.client.0.smithi062.stdout:9/649: write d16/d2c/d5e/d75/d76/fb6 [5016423,44507] 0 2024-04-20T21:32:06.138 INFO:tasks.workunit.client.0.smithi062.stdout:7/645: mknod d7/d3e/d33/d3d/ce0 0 2024-04-20T21:32:06.138 INFO:tasks.workunit.client.0.smithi062.stdout:8/612: rmdir d7/d25/d3b 39 2024-04-20T21:32:06.139 INFO:tasks.workunit.client.0.smithi062.stdout:3/584: fdatasync d0/d1/d10/d3a/d5e/d65/d88/f8a 0 2024-04-20T21:32:06.139 INFO:tasks.workunit.client.1.smithi189.stdout:8/386: truncate da/df/d13/f18 3472995 0 2024-04-20T21:32:06.141 INFO:tasks.workunit.client.1.smithi189.stdout:8/387: fsync da/df/d13/d31/f80 0 2024-04-20T21:32:06.143 INFO:tasks.workunit.client.0.smithi062.stdout:0/599: dwrite d7/d1c/f23 [4194304,4194304] 0 2024-04-20T21:32:06.147 INFO:tasks.workunit.client.1.smithi189.stdout:1/252: dwrite f1 [4194304,4194304] 0 2024-04-20T21:32:06.152 INFO:tasks.workunit.client.1.smithi189.stdout:5/213: rename d7/d14/d18/c21 to d7/d9/d31/c3c 0 2024-04-20T21:32:06.153 INFO:tasks.workunit.client.1.smithi189.stdout:5/214: write d7/f2b [1125824,120062] 0 2024-04-20T21:32:06.156 INFO:tasks.workunit.client.1.smithi189.stdout:5/215: readlink d7/l3a 0 2024-04-20T21:32:06.156 INFO:tasks.workunit.client.0.smithi062.stdout:3/585: creat d0/d1/db7/fc3 x:0 0 0 2024-04-20T21:32:06.158 INFO:tasks.workunit.client.0.smithi062.stdout:9/650: rename d16/d3d/c3f to d16/d3d/d55/cd7 0 2024-04-20T21:32:06.159 INFO:tasks.workunit.client.0.smithi062.stdout:6/483: dread d0/d9/d11/f48 [0,4194304] 0 2024-04-20T21:32:06.159 INFO:tasks.workunit.client.1.smithi189.stdout:5/216: readlink d7/d14/d18/l1e 0 2024-04-20T21:32:06.162 INFO:tasks.workunit.client.1.smithi189.stdout:5/217: chown d7/d14/d1b/d1f/d33 709835501 1 2024-04-20T21:32:06.165 INFO:tasks.workunit.client.1.smithi189.stdout:6/269: dwrite d8/d12/d4e/f51 [0,4194304] 0 2024-04-20T21:32:06.165 INFO:tasks.workunit.client.1.smithi189.stdout:5/218: chown d7/d14/d1b/d1f 208 1 2024-04-20T21:32:06.167 INFO:tasks.workunit.client.1.smithi189.stdout:6/270: fdatasync d8/f21 0 2024-04-20T21:32:06.171 INFO:tasks.workunit.client.0.smithi062.stdout:9/651: write fe [538417,120087] 0 2024-04-20T21:32:06.171 INFO:tasks.workunit.client.0.smithi062.stdout:3/586: fdatasync d0/d1/d10/d31/f9e 0 2024-04-20T21:32:06.172 INFO:tasks.workunit.client.1.smithi189.stdout:2/237: creat d1/d17/d2f/f42 x:0 0 0 2024-04-20T21:32:06.174 INFO:tasks.workunit.client.1.smithi189.stdout:2/238: write d1/f7 [6008904,59548] 0 2024-04-20T21:32:06.177 INFO:tasks.workunit.client.1.smithi189.stdout:2/239: stat d1/d11/c1b 0 2024-04-20T21:32:06.178 INFO:tasks.workunit.client.1.smithi189.stdout:2/240: chown d1/d17/d2f/f42 662 1 2024-04-20T21:32:06.179 INFO:tasks.workunit.client.1.smithi189.stdout:0/376: sync 2024-04-20T21:32:06.180 INFO:tasks.workunit.client.1.smithi189.stdout:3/238: link d2/d3/d2c/d31/f3c d2/d8/f58 0 2024-04-20T21:32:06.182 INFO:tasks.workunit.client.0.smithi062.stdout:3/587: write d0/d1b/fac [986717,45789] 0 2024-04-20T21:32:06.183 INFO:tasks.workunit.client.1.smithi189.stdout:3/239: dread - d2/d3/d2c/d31/f3c zero size 2024-04-20T21:32:06.184 INFO:tasks.workunit.client.1.smithi189.stdout:0/377: readlink d3/df/d28/d2b/l57 0 2024-04-20T21:32:06.185 INFO:tasks.workunit.client.1.smithi189.stdout:1/253: mknod d8/d21/d2c/c59 0 2024-04-20T21:32:06.186 INFO:tasks.workunit.client.1.smithi189.stdout:3/240: fsync d2/d3/f19 0 2024-04-20T21:32:06.186 INFO:tasks.workunit.client.0.smithi062.stdout:6/484: fsync d0/d9/f13 0 2024-04-20T21:32:06.187 INFO:tasks.workunit.client.0.smithi062.stdout:5/608: dwrite d8/d38/d7a/d4c/d7d/d9d/dac/d66/daa/fab [0,4194304] 0 2024-04-20T21:32:06.188 INFO:tasks.workunit.client.1.smithi189.stdout:0/378: write d3/df/d30/d37/f4b [5082689,20160] 0 2024-04-20T21:32:06.189 INFO:tasks.workunit.client.0.smithi062.stdout:7/646: dwrite d7/d26/f2f [0,4194304] 0 2024-04-20T21:32:06.191 INFO:tasks.workunit.client.1.smithi189.stdout:0/379: write d3/df/d28/d2b/f67 [245076,72744] 0 2024-04-20T21:32:06.193 INFO:tasks.workunit.client.1.smithi189.stdout:0/380: write d3/df/f3a [4180045,66504] 0 2024-04-20T21:32:06.194 INFO:tasks.workunit.client.0.smithi062.stdout:9/652: fdatasync d16/d3e/fc9 0 2024-04-20T21:32:06.197 INFO:tasks.workunit.client.0.smithi062.stdout:1/671: dread dc/d15/d75/d94/f19 [0,4194304] 0 2024-04-20T21:32:06.199 INFO:tasks.workunit.client.0.smithi062.stdout:0/600: dwrite d7/d1c/d15/d17/d40/d3c/dc5/f99 [0,4194304] 0 2024-04-20T21:32:06.202 INFO:tasks.workunit.client.0.smithi062.stdout:4/635: dread da/f2f [0,4194304] 0 2024-04-20T21:32:06.202 INFO:tasks.workunit.client.1.smithi189.stdout:7/279: dread d5/fe [0,4194304] 0 2024-04-20T21:32:06.204 INFO:tasks.workunit.client.1.smithi189.stdout:5/219: fsync d7/f2b 0 2024-04-20T21:32:06.210 INFO:tasks.workunit.client.0.smithi062.stdout:2/643: dread d7/da/d15/d1b/f80 [0,4194304] 0 2024-04-20T21:32:06.213 INFO:tasks.workunit.client.1.smithi189.stdout:7/280: truncate d5/d21/d35/d53/d5b/f2e 642050 0 2024-04-20T21:32:06.215 INFO:tasks.workunit.client.0.smithi062.stdout:8/613: rename d7/d25/d3b/c8a to d7/d9/ccf 0 2024-04-20T21:32:06.215 INFO:tasks.workunit.client.0.smithi062.stdout:6/485: creat d0/d2c/d67/f9e x:0 0 0 2024-04-20T21:32:06.218 INFO:tasks.workunit.client.1.smithi189.stdout:9/290: getdents d9/dc/d17/d2b/d55 0 2024-04-20T21:32:06.229 INFO:tasks.workunit.client.0.smithi062.stdout:1/672: mknod dc/d27/d9a/da0/db3/cf0 0 2024-04-20T21:32:06.233 INFO:tasks.workunit.client.1.smithi189.stdout:2/241: dwrite d1/d11/d2d/f41 [0,4194304] 0 2024-04-20T21:32:06.243 INFO:tasks.workunit.client.1.smithi189.stdout:6/271: link d8/d12/d4e/f50 d8/d2b/f57 0 2024-04-20T21:32:06.244 INFO:tasks.workunit.client.0.smithi062.stdout:4/636: dread da/dd/d15/d50/f86 [0,4194304] 0 2024-04-20T21:32:06.244 INFO:tasks.workunit.client.0.smithi062.stdout:9/653: dwrite d16/d3d/d55/fb7 [0,4194304] 0 2024-04-20T21:32:06.244 INFO:tasks.workunit.client.0.smithi062.stdout:3/588: symlink d0/d1/lc4 0 2024-04-20T21:32:06.246 INFO:tasks.workunit.client.1.smithi189.stdout:5/220: symlink d7/d14/l3d 0 2024-04-20T21:32:06.249 INFO:tasks.workunit.client.1.smithi189.stdout:0/381: dread d3/df/d30/d45/f59 [0,4194304] 0 2024-04-20T21:32:06.250 INFO:tasks.workunit.client.0.smithi062.stdout:0/601: dwrite d7/d1e/d6e/f7d [0,4194304] 0 2024-04-20T21:32:06.251 INFO:tasks.workunit.client.1.smithi189.stdout:1/254: dread d8/f24 [0,4194304] 0 2024-04-20T21:32:06.253 INFO:tasks.workunit.client.1.smithi189.stdout:1/255: fdatasync d8/f46 0 2024-04-20T21:32:06.254 INFO:tasks.workunit.client.1.smithi189.stdout:1/256: chown d8 340374604 1 2024-04-20T21:32:06.255 INFO:tasks.workunit.client.1.smithi189.stdout:1/257: readlink d8/d21/d2c/l2e 0 2024-04-20T21:32:06.260 INFO:tasks.workunit.client.0.smithi062.stdout:9/654: mknod d16/d2f/d69/cd8 0 2024-04-20T21:32:06.263 INFO:tasks.workunit.client.1.smithi189.stdout:2/242: mknod d1/d17/d37/c43 0 2024-04-20T21:32:06.263 INFO:tasks.workunit.client.0.smithi062.stdout:6/486: symlink d0/d2c/d8c/l9f 0 2024-04-20T21:32:06.265 INFO:tasks.workunit.client.1.smithi189.stdout:8/388: getdents da/de/d1e 0 2024-04-20T21:32:06.268 INFO:tasks.workunit.client.0.smithi062.stdout:4/637: creat da/d56/d71/fe2 x:0 0 0 2024-04-20T21:32:06.271 INFO:tasks.workunit.client.1.smithi189.stdout:0/382: symlink d3/df/d30/d37/l80 0 2024-04-20T21:32:06.271 INFO:tasks.workunit.client.0.smithi062.stdout:3/589: mkdir d0/d1c/d52/dc5 0 2024-04-20T21:32:06.273 INFO:tasks.workunit.client.0.smithi062.stdout:6/487: readlink d0/d9/d11/d22/l6f 0 2024-04-20T21:32:06.274 INFO:tasks.workunit.client.1.smithi189.stdout:3/241: link d2/d3/f1b d2/f59 0 2024-04-20T21:32:06.279 INFO:tasks.workunit.client.0.smithi062.stdout:9/655: mkdir d16/d2c/d5e/d75/d76/dd9 0 2024-04-20T21:32:06.281 INFO:tasks.workunit.client.0.smithi062.stdout:9/656: chown d16/d2f 15 1 2024-04-20T21:32:06.281 INFO:tasks.workunit.client.1.smithi189.stdout:8/389: chown da/df/d22/l64 959578355 1 2024-04-20T21:32:06.285 INFO:tasks.workunit.client.1.smithi189.stdout:7/281: rename d5/d21/d35/d53/d5b/f2c to d5/d1d/f5f 0 2024-04-20T21:32:06.287 INFO:tasks.workunit.client.0.smithi062.stdout:3/590: mknod d0/d1/d10/cc6 0 2024-04-20T21:32:06.288 INFO:tasks.workunit.client.1.smithi189.stdout:1/258: creat d8/df/d50/f5a x:0 0 0 2024-04-20T21:32:06.290 INFO:tasks.workunit.client.1.smithi189.stdout:2/243: dwrite d1/f5 [4194304,4194304] 0 2024-04-20T21:32:06.294 INFO:tasks.workunit.client.1.smithi189.stdout:8/390: chown da/de/d1e/d69/d70 5875119 1 2024-04-20T21:32:06.296 INFO:tasks.workunit.client.0.smithi062.stdout:6/488: unlink d0/d9/c72 0 2024-04-20T21:32:06.296 INFO:tasks.workunit.client.0.smithi062.stdout:0/602: dwrite d7/d20/d62/d6f/f9f [0,4194304] 0 2024-04-20T21:32:06.298 INFO:tasks.workunit.client.0.smithi062.stdout:9/657: symlink d16/d4e/d80/lda 0 2024-04-20T21:32:06.299 INFO:tasks.workunit.client.1.smithi189.stdout:6/272: rename d8/d12/d4e/f51 to d8/d12/d30/f58 0 2024-04-20T21:32:06.300 INFO:tasks.workunit.client.0.smithi062.stdout:7/647: dread d7/ff [0,4194304] 0 2024-04-20T21:32:06.302 INFO:tasks.workunit.client.0.smithi062.stdout:6/489: fdatasync d0/d9/f16 0 2024-04-20T21:32:06.303 INFO:tasks.workunit.client.0.smithi062.stdout:2/644: dread d7/da/d15/d2f/d38/d6b/f88 [0,4194304] 0 2024-04-20T21:32:06.310 INFO:tasks.workunit.client.1.smithi189.stdout:2/244: dread d1/d11/d2d/f41 [0,4194304] 0 2024-04-20T21:32:06.310 INFO:tasks.workunit.client.0.smithi062.stdout:6/490: stat d0/f3e 0 2024-04-20T21:32:06.310 INFO:tasks.workunit.client.0.smithi062.stdout:4/638: link da/dd/d15/fa6 da/dd/d3c/dac/fe3 0 2024-04-20T21:32:06.314 INFO:tasks.workunit.client.0.smithi062.stdout:9/658: symlink d16/d2c/d42/ldb 0 2024-04-20T21:32:06.314 INFO:tasks.workunit.client.0.smithi062.stdout:7/648: stat d7/d63/ca5 0 2024-04-20T21:32:06.318 INFO:tasks.workunit.client.1.smithi189.stdout:8/391: mkdir da/df/d3c/d68/d8b 0 2024-04-20T21:32:06.319 INFO:tasks.workunit.client.0.smithi062.stdout:4/639: write da/d56/d71/f93 [4127843,72390] 0 2024-04-20T21:32:06.323 INFO:tasks.workunit.client.0.smithi062.stdout:9/659: stat d16/d2f/d69/fc5 0 2024-04-20T21:32:06.325 INFO:tasks.workunit.client.1.smithi189.stdout:2/245: creat d1/d17/d37/f44 x:0 0 0 2024-04-20T21:32:06.332 INFO:tasks.workunit.client.0.smithi062.stdout:9/660: write d16/d2c/d5e/d96/fb3 [251278,30862] 0 2024-04-20T21:32:06.332 INFO:tasks.workunit.client.0.smithi062.stdout:1/673: dread dc/d15/d75/d94/f34 [0,4194304] 0 2024-04-20T21:32:06.334 INFO:tasks.workunit.client.0.smithi062.stdout:4/640: rmdir da/d56/d71 39 2024-04-20T21:32:06.338 INFO:tasks.workunit.client.0.smithi062.stdout:4/641: dread - da/dd/df/d27/fc2 zero size 2024-04-20T21:32:06.338 INFO:tasks.workunit.client.0.smithi062.stdout:0/603: getdents d7/d1c/d15/d80 0 2024-04-20T21:32:06.340 INFO:tasks.workunit.client.1.smithi189.stdout:0/383: dread d3/df/d28/d2b/f49 [0,4194304] 0 2024-04-20T21:32:06.344 INFO:tasks.workunit.client.1.smithi189.stdout:3/242: rename d2/d3/d2c/d31/f3c to d2/f5a 0 2024-04-20T21:32:06.345 INFO:tasks.workunit.client.0.smithi062.stdout:4/642: stat da/dd/d15/d50/d64/d3d/fb5 0 2024-04-20T21:32:06.345 INFO:tasks.workunit.client.0.smithi062.stdout:1/674: readlink dc/d15/d75/d96/dd4/d3a/da2/de4/d84/l8f 0 2024-04-20T21:32:06.346 INFO:tasks.workunit.client.1.smithi189.stdout:2/246: truncate d1/f6 2935009 0 2024-04-20T21:32:06.348 INFO:tasks.workunit.client.1.smithi189.stdout:2/247: write d1/d11/f1a [619820,25882] 0 2024-04-20T21:32:06.348 INFO:tasks.workunit.client.0.smithi062.stdout:9/661: rename d16/d2c/l59 to d16/d3d/ldc 0 2024-04-20T21:32:06.355 INFO:tasks.workunit.client.0.smithi062.stdout:0/604: mknod d7/d1c/d15/d80/cc8 0 2024-04-20T21:32:06.356 INFO:tasks.workunit.client.1.smithi189.stdout:6/273: rename d8/da/f37 to d8/da/dd/d35/f59 0 2024-04-20T21:32:06.357 INFO:tasks.workunit.client.1.smithi189.stdout:6/274: chown d8/d10/f3f 19 1 2024-04-20T21:32:06.358 INFO:tasks.workunit.client.0.smithi062.stdout:3/591: dwrite d0/d1/f77 [0,4194304] 0 2024-04-20T21:32:06.358 INFO:tasks.workunit.client.0.smithi062.stdout:4/643: chown da/dd/d15/d50/d64/d57/la4 1 1 2024-04-20T21:32:06.359 INFO:tasks.workunit.client.1.smithi189.stdout:6/275: chown f5 4094 1 2024-04-20T21:32:06.360 INFO:tasks.workunit.client.1.smithi189.stdout:3/243: dwrite d2/d8/f22 [0,4194304] 0 2024-04-20T21:32:06.360 INFO:tasks.workunit.client.0.smithi062.stdout:9/662: chown d16/d2c/l34 17252 1 2024-04-20T21:32:06.364 INFO:tasks.workunit.client.1.smithi189.stdout:2/248: mknod d1/c45 0 2024-04-20T21:32:06.366 INFO:tasks.workunit.client.0.smithi062.stdout:3/592: stat d0/d1/f20 0 2024-04-20T21:32:06.366 INFO:tasks.workunit.client.1.smithi189.stdout:3/244: creat d2/d3/d25/f5b x:0 0 0 2024-04-20T21:32:06.368 INFO:tasks.workunit.client.0.smithi062.stdout:9/663: fdatasync f15 0 2024-04-20T21:32:06.371 INFO:tasks.workunit.client.0.smithi062.stdout:1/675: creat dc/d15/d75/d96/dd4/d3a/da2/de4/ff1 x:0 0 0 2024-04-20T21:32:06.372 INFO:tasks.workunit.client.0.smithi062.stdout:0/605: write d7/d1e/d6e/f75 [8502535,82047] 0 2024-04-20T21:32:06.377 INFO:tasks.workunit.client.0.smithi062.stdout:9/664: fsync f4 0 2024-04-20T21:32:06.378 INFO:tasks.workunit.client.1.smithi189.stdout:5/221: truncate d7/fb 2243299 0 2024-04-20T21:32:06.381 INFO:tasks.workunit.client.1.smithi189.stdout:7/282: rmdir d5/d21/d35/d53/d5b 39 2024-04-20T21:32:06.382 INFO:tasks.workunit.client.0.smithi062.stdout:8/614: dread d7/d25/f5a [0,4194304] 0 2024-04-20T21:32:06.383 INFO:tasks.workunit.client.1.smithi189.stdout:6/276: dwrite d8/d12/f17 [4194304,4194304] 0 2024-04-20T21:32:06.384 INFO:tasks.workunit.client.0.smithi062.stdout:2/645: dwrite d7/da/d15/d2f/fc1 [0,4194304] 0 2024-04-20T21:32:06.386 INFO:tasks.workunit.client.1.smithi189.stdout:7/283: chown d5/f14 114798103 1 2024-04-20T21:32:06.387 INFO:tasks.workunit.client.1.smithi189.stdout:1/259: write d8/f14 [3003175,61542] 0 2024-04-20T21:32:06.388 INFO:tasks.workunit.client.0.smithi062.stdout:7/649: dwrite d7/d3e/d33/d3d/d4d/d54/f3f [0,4194304] 0 2024-04-20T21:32:06.390 INFO:tasks.workunit.client.1.smithi189.stdout:6/277: dread - d8/d2b/f39 zero size 2024-04-20T21:32:06.391 INFO:tasks.workunit.client.1.smithi189.stdout:5/222: link d7/d14/d18/f17 d7/d14/d18/f3e 0 2024-04-20T21:32:06.392 INFO:tasks.workunit.client.1.smithi189.stdout:4/259: write d2/dd/d11/d14/f3f [2410088,30793] 0 2024-04-20T21:32:06.395 INFO:tasks.workunit.client.1.smithi189.stdout:3/245: dwrite d2/f15 [4194304,4194304] 0 2024-04-20T21:32:06.395 INFO:tasks.workunit.client.1.smithi189.stdout:5/223: readlink d7/d9/d30/l32 0 2024-04-20T21:32:06.400 INFO:tasks.workunit.client.0.smithi062.stdout:2/646: write d7/da/d15/d1b/f7d [1114057,24604] 0 2024-04-20T21:32:06.401 INFO:tasks.workunit.client.1.smithi189.stdout:1/260: creat d8/d21/d2c/d36/f5b x:0 0 0 2024-04-20T21:32:06.406 INFO:tasks.workunit.client.0.smithi062.stdout:8/615: symlink d7/d25/d3b/d58/d8b/d24/d46/ld0 0 2024-04-20T21:32:06.408 INFO:tasks.workunit.client.0.smithi062.stdout:7/650: fdatasync d7/d3e/d33/d3d/f4a 0 2024-04-20T21:32:06.409 INFO:tasks.workunit.client.1.smithi189.stdout:7/284: rename d5/c4e to d5/d21/c60 0 2024-04-20T21:32:06.410 INFO:tasks.workunit.client.1.smithi189.stdout:0/384: write d3/df/d30/d45/f59 [628671,106910] 0 2024-04-20T21:32:06.412 INFO:tasks.workunit.client.1.smithi189.stdout:4/260: write d2/dd/f44 [1000652,118878] 0 2024-04-20T21:32:06.414 INFO:tasks.workunit.client.1.smithi189.stdout:4/261: chown d2/f54 68505 1 2024-04-20T21:32:06.418 INFO:tasks.workunit.client.0.smithi062.stdout:7/651: fdatasync d7/f2d 0 2024-04-20T21:32:06.419 INFO:tasks.workunit.client.1.smithi189.stdout:4/262: readlink d2/dd/d11/l2b 0 2024-04-20T21:32:06.419 INFO:tasks.workunit.client.1.smithi189.stdout:6/278: mknod d8/d2b/d49/d54/c5a 0 2024-04-20T21:32:06.419 INFO:tasks.workunit.client.1.smithi189.stdout:2/249: truncate d1/f12 132027 0 2024-04-20T21:32:06.420 INFO:tasks.workunit.client.1.smithi189.stdout:6/279: chown d8/d12/d4e 11805920 1 2024-04-20T21:32:06.420 INFO:tasks.workunit.client.1.smithi189.stdout:2/250: fdatasync d1/f9 0 2024-04-20T21:32:06.421 INFO:tasks.workunit.client.1.smithi189.stdout:1/261: mknod d8/d1e/d43/d58/c5c 0 2024-04-20T21:32:06.421 INFO:tasks.workunit.client.0.smithi062.stdout:0/606: link d7/d1c/d15/d17/d37/f59 d7/d1c/fc9 0 2024-04-20T21:32:06.427 INFO:tasks.workunit.client.1.smithi189.stdout:9/291: sync 2024-04-20T21:32:06.428 INFO:tasks.workunit.client.0.smithi062.stdout:3/593: dwrite d0/d1c/f45 [0,4194304] 0 2024-04-20T21:32:06.431 INFO:tasks.workunit.client.1.smithi189.stdout:1/262: write d8/d21/d2c/d36/f5b [628175,47329] 0 2024-04-20T21:32:06.432 INFO:tasks.workunit.client.1.smithi189.stdout:0/385: creat d3/df/d30/d45/d60/f81 x:0 0 0 2024-04-20T21:32:06.432 INFO:tasks.workunit.client.1.smithi189.stdout:5/224: fsync d7/fb 0 2024-04-20T21:32:06.433 INFO:tasks.workunit.client.0.smithi062.stdout:9/665: dwrite d16/d4e/fa3 [0,4194304] 0 2024-04-20T21:32:06.433 INFO:tasks.workunit.client.0.smithi062.stdout:8/616: mknod d7/d25/d3b/d58/d8b/d24/cd1 0 2024-04-20T21:32:06.434 INFO:tasks.workunit.client.0.smithi062.stdout:0/607: creat d7/d1c/d15/fca x:0 0 0 2024-04-20T21:32:06.437 INFO:tasks.workunit.client.0.smithi062.stdout:7/652: rename d7/d26 to d7/d26/de1 22 2024-04-20T21:32:06.438 INFO:tasks.workunit.client.1.smithi189.stdout:9/292: read d9/f34 [957621,72058] 0 2024-04-20T21:32:06.440 INFO:tasks.workunit.client.0.smithi062.stdout:9/666: write d16/d2f/fc8 [352326,29492] 0 2024-04-20T21:32:06.442 INFO:tasks.workunit.client.1.smithi189.stdout:0/386: read d3/df/d30/d45/f59 [343377,124800] 0 2024-04-20T21:32:06.444 INFO:tasks.workunit.client.1.smithi189.stdout:5/225: write d7/d14/d18/f3e [3079064,121504] 0 2024-04-20T21:32:06.445 INFO:tasks.workunit.client.0.smithi062.stdout:2/647: dwrite d7/da/d15/d1b/d3d/d84/fa9 [0,4194304] 0 2024-04-20T21:32:06.448 INFO:tasks.workunit.client.0.smithi062.stdout:8/617: creat d7/d25/d3b/d58/d8b/d24/d46/fd2 x:0 0 0 2024-04-20T21:32:06.448 INFO:tasks.workunit.client.0.smithi062.stdout:1/676: dwrite dc/d27/d32/f51 [0,4194304] 0 2024-04-20T21:32:06.451 INFO:tasks.workunit.client.1.smithi189.stdout:6/280: dwrite f7 [4194304,4194304] 0 2024-04-20T21:32:06.452 INFO:tasks.workunit.client.1.smithi189.stdout:0/387: rmdir d3/df/d30/d45/d60/d79 39 2024-04-20T21:32:06.453 INFO:tasks.workunit.client.1.smithi189.stdout:9/293: rmdir d9/dc/d17/d36/d5d 39 2024-04-20T21:32:06.456 INFO:tasks.workunit.client.1.smithi189.stdout:3/246: truncate d2/d8/f10 7583131 0 2024-04-20T21:32:06.457 INFO:tasks.workunit.client.1.smithi189.stdout:2/251: dwrite d1/d17/d2f/f42 [0,4194304] 0 2024-04-20T21:32:06.463 INFO:tasks.workunit.client.1.smithi189.stdout:6/281: write d8/d2b/f3a [5437453,130677] 0 2024-04-20T21:32:06.464 INFO:tasks.workunit.client.1.smithi189.stdout:3/247: mkdir d2/d1c/d5c 0 2024-04-20T21:32:06.465 INFO:tasks.workunit.client.1.smithi189.stdout:9/294: truncate d9/f44 103235 0 2024-04-20T21:32:06.465 INFO:tasks.workunit.client.1.smithi189.stdout:2/252: chown d1/d11/d26/l27 833119 1 2024-04-20T21:32:06.467 INFO:tasks.workunit.client.0.smithi062.stdout:4/644: truncate da/dd/d3c/f53 2806000 0 2024-04-20T21:32:06.469 INFO:tasks.workunit.client.1.smithi189.stdout:0/388: mknod d3/df/d30/d45/d60/d79/c82 0 2024-04-20T21:32:06.470 INFO:tasks.workunit.client.1.smithi189.stdout:3/248: stat d2/d8/c12 0 2024-04-20T21:32:06.470 INFO:tasks.workunit.client.1.smithi189.stdout:2/253: dread - d1/d17/d37/f44 zero size 2024-04-20T21:32:06.471 INFO:tasks.workunit.client.0.smithi062.stdout:0/608: truncate d7/d1c/d15/d17/d40/d79/f57 2314038 0 2024-04-20T21:32:06.473 INFO:tasks.workunit.client.0.smithi062.stdout:7/653: link d7/l17 d7/d63/le2 0 2024-04-20T21:32:06.475 INFO:tasks.workunit.client.0.smithi062.stdout:1/677: symlink dc/d15/d20/d28/d76/lf2 0 2024-04-20T21:32:06.480 INFO:tasks.workunit.client.1.smithi189.stdout:5/226: dwrite d7/d9/d30/f2d [0,4194304] 0 2024-04-20T21:32:06.485 INFO:tasks.workunit.client.0.smithi062.stdout:6/491: sync 2024-04-20T21:32:06.485 INFO:tasks.workunit.client.0.smithi062.stdout:5/609: sync 2024-04-20T21:32:06.487 INFO:tasks.workunit.client.1.smithi189.stdout:9/295: mknod d9/dc/d17/c60 0 2024-04-20T21:32:06.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:06 smithi189 ceph-mon[21562]: Updating smithi062:/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-20T21:32:06.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:06 smithi189 ceph-mon[21562]: Updating smithi189:/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-20T21:32:06.489 INFO:tasks.workunit.client.0.smithi062.stdout:8/618: dwrite d7/d25/d3b/d58/d8b/f1f [0,4194304] 0 2024-04-20T21:32:06.491 INFO:tasks.workunit.client.1.smithi189.stdout:6/282: mkdir d8/da/dd/d2a/d3b/d5b 0 2024-04-20T21:32:06.492 INFO:tasks.workunit.client.0.smithi062.stdout:7/654: unlink d7/d3e/d33/d3d/d4d/d54/f28 0 2024-04-20T21:32:06.493 INFO:tasks.workunit.client.0.smithi062.stdout:5/610: chown d8/d1e/d2c/f34 1 1 2024-04-20T21:32:06.494 INFO:tasks.workunit.client.0.smithi062.stdout:9/667: dwrite d16/f9e [0,4194304] 0 2024-04-20T21:32:06.501 INFO:tasks.workunit.client.0.smithi062.stdout:5/611: chown d8/d10/f26 5916 1 2024-04-20T21:32:06.502 INFO:tasks.workunit.client.1.smithi189.stdout:5/227: write d7/d14/d18/f17 [287985,42974] 0 2024-04-20T21:32:06.502 INFO:tasks.workunit.client.1.smithi189.stdout:1/263: truncate d8/df/f3d 38795 0 2024-04-20T21:32:06.503 INFO:tasks.workunit.client.0.smithi062.stdout:6/492: mkdir d0/d52/da0 0 2024-04-20T21:32:06.504 INFO:tasks.workunit.client.1.smithi189.stdout:0/389: dwrite d3/df/d30/d45/d60/f81 [0,4194304] 0 2024-04-20T21:32:06.509 INFO:tasks.workunit.client.1.smithi189.stdout:5/228: chown d7/d9/d30/l32 438400473 1 2024-04-20T21:32:06.509 INFO:tasks.workunit.client.1.smithi189.stdout:9/296: rmdir d9/dc 39 2024-04-20T21:32:06.511 INFO:tasks.workunit.client.1.smithi189.stdout:9/297: write f4 [843136,107821] 0 2024-04-20T21:32:06.511 INFO:tasks.workunit.client.1.smithi189.stdout:0/390: write d3/fe [3414133,83543] 0 2024-04-20T21:32:06.512 INFO:tasks.workunit.client.0.smithi062.stdout:5/612: dread d8/d38/d7a/d4c/d7d/d9d/fb8 [0,4194304] 0 2024-04-20T21:32:06.512 INFO:tasks.workunit.client.0.smithi062.stdout:3/594: write d0/d1/f20 [3241506,121378] 0 2024-04-20T21:32:06.516 INFO:tasks.workunit.client.1.smithi189.stdout:5/229: mknod d7/d35/c3f 0 2024-04-20T21:32:06.516 INFO:tasks.workunit.client.1.smithi189.stdout:3/249: dwrite d2/d3/d25/f2e [0,4194304] 0 2024-04-20T21:32:06.528 INFO:tasks.workunit.client.0.smithi062.stdout:1/678: rename dc/d15/d75/d94/c9b to dc/d15/cf3 0 2024-04-20T21:32:06.530 INFO:tasks.workunit.client.0.smithi062.stdout:3/595: write d0/d1c/d52/d97/fa3 [178316,116934] 0 2024-04-20T21:32:06.532 INFO:tasks.workunit.client.1.smithi189.stdout:0/391: rmdir d3/df/d30/d45/d60/d79 39 2024-04-20T21:32:06.534 INFO:tasks.workunit.client.1.smithi189.stdout:5/230: mknod d7/d14/d1b/c40 0 2024-04-20T21:32:06.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:06 smithi062 ceph-mon[16221]: Updating smithi062:/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-20T21:32:06.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:06 smithi062 ceph-mon[16221]: Updating smithi189:/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-20T21:32:06.536 INFO:tasks.workunit.client.0.smithi062.stdout:8/619: dwrite d7/d30/d90/f92 [0,4194304] 0 2024-04-20T21:32:06.537 INFO:tasks.workunit.client.1.smithi189.stdout:6/283: dread d8/d10/f36 [0,4194304] 0 2024-04-20T21:32:06.537 INFO:tasks.workunit.client.1.smithi189.stdout:9/298: mknod d9/dc/d17/d2b/d56/c61 0 2024-04-20T21:32:06.542 INFO:tasks.workunit.client.1.smithi189.stdout:5/231: rmdir d7/d9 39 2024-04-20T21:32:06.544 INFO:tasks.workunit.client.1.smithi189.stdout:1/264: dwrite d8/f14 [0,4194304] 0 2024-04-20T21:32:06.546 INFO:tasks.workunit.client.0.smithi062.stdout:7/655: dwrite d7/d3e/f71 [0,4194304] 0 2024-04-20T21:32:06.550 INFO:tasks.workunit.client.1.smithi189.stdout:8/392: sync 2024-04-20T21:32:06.550 INFO:tasks.workunit.client.1.smithi189.stdout:9/299: mkdir d9/d12/d62 0 2024-04-20T21:32:06.551 INFO:tasks.workunit.client.1.smithi189.stdout:6/284: mkdir d8/d2b/d5c 0 2024-04-20T21:32:06.551 INFO:tasks.workunit.client.0.smithi062.stdout:6/493: dread d0/d9/f13 [0,4194304] 0 2024-04-20T21:32:06.552 INFO:tasks.workunit.client.0.smithi062.stdout:7/656: write d7/d3e/d33/d3d/dad/fbc [904217,22725] 0 2024-04-20T21:32:06.554 INFO:tasks.workunit.client.0.smithi062.stdout:3/596: fdatasync d0/d1/f20 0 2024-04-20T21:32:06.556 INFO:tasks.workunit.client.1.smithi189.stdout:3/250: dwrite d2/d3/d25/f40 [0,4194304] 0 2024-04-20T21:32:06.558 INFO:tasks.workunit.client.1.smithi189.stdout:9/300: chown d9/d12/d3b/f4c 232068514 1 2024-04-20T21:32:06.558 INFO:tasks.workunit.client.1.smithi189.stdout:1/265: write d8/f1a [8018112,113026] 0 2024-04-20T21:32:06.559 INFO:tasks.workunit.client.0.smithi062.stdout:1/679: dread dc/d27/d32/f51 [0,4194304] 0 2024-04-20T21:32:06.560 INFO:tasks.workunit.client.0.smithi062.stdout:7/657: dread d7/d26/f2c [0,4194304] 0 2024-04-20T21:32:06.561 INFO:tasks.workunit.client.1.smithi189.stdout:7/285: dread d5/f20 [0,4194304] 0 2024-04-20T21:32:06.567 INFO:tasks.workunit.client.1.smithi189.stdout:9/301: write d9/dc/f1c [1513121,118841] 0 2024-04-20T21:32:06.573 INFO:tasks.workunit.client.0.smithi062.stdout:2/648: dread d7/ff [0,4194304] 0 2024-04-20T21:32:06.578 INFO:tasks.workunit.client.1.smithi189.stdout:7/286: mknod d5/d21/d25/d34/c61 0 2024-04-20T21:32:06.584 INFO:tasks.workunit.client.1.smithi189.stdout:3/251: dwrite d2/d8/f4b [0,4194304] 0 2024-04-20T21:32:06.593 INFO:tasks.workunit.client.1.smithi189.stdout:6/285: link d8/da/dd/d2a/d3b/l43 d8/d2b/d49/l5d 0 2024-04-20T21:32:06.595 INFO:tasks.workunit.client.1.smithi189.stdout:8/393: link da/df/f23 da/de/f8c 0 2024-04-20T21:32:06.600 INFO:tasks.workunit.client.1.smithi189.stdout:5/232: dwrite d7/f2b [0,4194304] 0 2024-04-20T21:32:06.600 INFO:tasks.workunit.client.1.smithi189.stdout:2/254: write d1/f23 [877983,16907] 0 2024-04-20T21:32:06.600 INFO:tasks.workunit.client.1.smithi189.stdout:9/302: dwrite f4 [0,4194304] 0 2024-04-20T21:32:06.600 INFO:tasks.workunit.client.1.smithi189.stdout:6/286: write f7 [3283330,44613] 0 2024-04-20T21:32:06.605 INFO:tasks.workunit.client.1.smithi189.stdout:7/287: creat d5/d1d/f62 x:0 0 0 2024-04-20T21:32:06.606 INFO:tasks.workunit.client.1.smithi189.stdout:0/392: truncate d3/f66 312110 0 2024-04-20T21:32:06.611 INFO:tasks.workunit.client.1.smithi189.stdout:6/287: fdatasync d8/d10/f36 0 2024-04-20T21:32:06.613 INFO:tasks.workunit.client.1.smithi189.stdout:4/263: sync 2024-04-20T21:32:06.613 INFO:tasks.workunit.client.1.smithi189.stdout:8/394: mkdir da/df/d3c/d68/d8b/d8d 0 2024-04-20T21:32:06.615 INFO:tasks.workunit.client.1.smithi189.stdout:9/303: mknod d9/d12/d15/d1a/c63 0 2024-04-20T21:32:06.617 INFO:tasks.workunit.client.1.smithi189.stdout:7/288: rename d5/d21/d25/d34/c61 to d5/d21/d25/c63 0 2024-04-20T21:32:06.622 INFO:tasks.workunit.client.0.smithi062.stdout:5/613: unlink d8/d38/d7a/d4c/d7d/d9d/dac/d2f/d8e/l90 0 2024-04-20T21:32:06.623 INFO:tasks.workunit.client.1.smithi189.stdout:6/288: stat d8/da/dd/d2a 0 2024-04-20T21:32:06.624 INFO:tasks.workunit.client.1.smithi189.stdout:4/264: stat d2/cf 0 2024-04-20T21:32:06.628 INFO:tasks.workunit.client.1.smithi189.stdout:0/393: rename d3/df/f15 to d3/df/d28/d2b/d2c/f83 0 2024-04-20T21:32:06.629 INFO:tasks.workunit.client.1.smithi189.stdout:6/289: stat d8/d12/d4e 0 2024-04-20T21:32:06.631 INFO:tasks.workunit.client.1.smithi189.stdout:8/395: fsync da/df/d3a/f4c 0 2024-04-20T21:32:06.634 INFO:tasks.workunit.client.0.smithi062.stdout:6/494: rename d0/f3 to d0/d2c/d67/fa1 0 2024-04-20T21:32:06.634 INFO:tasks.workunit.client.1.smithi189.stdout:7/289: mkdir d5/d1d/d64 0 2024-04-20T21:32:06.635 INFO:tasks.workunit.client.1.smithi189.stdout:9/304: write d9/d12/d15/d1a/f27 [958480,4506] 0 2024-04-20T21:32:06.643 INFO:tasks.workunit.client.1.smithi189.stdout:9/305: chown d9/dc/d17/d36/f57 1106678 1 2024-04-20T21:32:06.646 INFO:tasks.workunit.client.1.smithi189.stdout:0/394: creat d3/df/d4c/f84 x:0 0 0 2024-04-20T21:32:06.646 INFO:tasks.workunit.client.0.smithi062.stdout:0/609: write d7/d1e/db6/fa3 [3247112,47998] 0 2024-04-20T21:32:06.647 INFO:tasks.workunit.client.0.smithi062.stdout:9/668: getdents d16/d3e 0 2024-04-20T21:32:06.647 INFO:tasks.workunit.client.1.smithi189.stdout:2/255: dwrite d1/f9 [4194304,4194304] 0 2024-04-20T21:32:06.648 INFO:tasks.workunit.client.1.smithi189.stdout:5/233: dwrite d7/d14/d18/f3e [0,4194304] 0 2024-04-20T21:32:06.648 INFO:tasks.workunit.client.1.smithi189.stdout:4/265: rmdir d2/d36/d56 0 2024-04-20T21:32:06.649 INFO:tasks.workunit.client.0.smithi062.stdout:4/645: sync 2024-04-20T21:32:06.651 INFO:tasks.workunit.client.1.smithi189.stdout:7/290: unlink d5/d21/d35/l3e 0 2024-04-20T21:32:06.654 INFO:tasks.workunit.client.1.smithi189.stdout:6/290: creat d8/d2b/d5c/f5e x:0 0 0 2024-04-20T21:32:06.654 INFO:tasks.workunit.client.0.smithi062.stdout:9/669: chown d16/d3d/d55/d6e/f72 5444001 1 2024-04-20T21:32:06.654 INFO:tasks.workunit.client.0.smithi062.stdout:5/614: creat d8/d38/d7a/d4c/d7d/d9d/dac/d66/fc2 x:0 0 0 2024-04-20T21:32:06.655 INFO:tasks.workunit.client.1.smithi189.stdout:2/256: mknod d1/d17/d2f/c46 0 2024-04-20T21:32:06.656 INFO:tasks.workunit.client.1.smithi189.stdout:7/291: read d5/ff [4086970,28819] 0 2024-04-20T21:32:06.657 INFO:tasks.workunit.client.1.smithi189.stdout:0/395: mknod d3/df/d5d/d64/c85 0 2024-04-20T21:32:06.660 INFO:tasks.workunit.client.0.smithi062.stdout:7/658: link d7/d3e/f45 d7/d3e/d33/d3d/d4d/dce/fe3 0 2024-04-20T21:32:06.661 INFO:tasks.workunit.client.0.smithi062.stdout:1/680: dwrite dc/d70/f90 [0,4194304] 0 2024-04-20T21:32:06.662 INFO:tasks.workunit.client.1.smithi189.stdout:7/292: chown d5/d21/d25/d34 117845219 1 2024-04-20T21:32:06.664 INFO:tasks.workunit.client.0.smithi062.stdout:0/610: creat d7/d1e/da9/fcb x:0 0 0 2024-04-20T21:32:06.664 INFO:tasks.workunit.client.1.smithi189.stdout:6/291: rename d8/d10/c1f to d8/d2b/c5f 0 2024-04-20T21:32:06.668 INFO:tasks.workunit.client.0.smithi062.stdout:7/659: truncate d7/d3e/d33/d3d/dad/fb9 910300 0 2024-04-20T21:32:06.668 INFO:tasks.workunit.client.0.smithi062.stdout:4/646: mkdir da/dd/df/d27/d7a/d91/de4 0 2024-04-20T21:32:06.669 INFO:tasks.workunit.client.1.smithi189.stdout:7/293: creat d5/d46/f65 x:0 0 0 2024-04-20T21:32:06.670 INFO:tasks.workunit.client.0.smithi062.stdout:5/615: fdatasync d8/d1e/d2c/f69 0 2024-04-20T21:32:06.672 INFO:tasks.workunit.client.1.smithi189.stdout:9/306: dread f4 [0,4194304] 0 2024-04-20T21:32:06.674 INFO:tasks.workunit.client.1.smithi189.stdout:4/266: getdents d2/d20 0 2024-04-20T21:32:06.676 INFO:tasks.workunit.client.1.smithi189.stdout:9/307: stat f4 0 2024-04-20T21:32:06.678 INFO:tasks.workunit.client.1.smithi189.stdout:3/252: dread d2/d1c/f28 [0,4194304] 0 2024-04-20T21:32:06.680 INFO:tasks.workunit.client.1.smithi189.stdout:9/308: fdatasync d9/dc/f2c 0 2024-04-20T21:32:06.684 INFO:tasks.workunit.client.1.smithi189.stdout:9/309: fsync d9/d12/d15/d1a/f5c 0 2024-04-20T21:32:06.688 INFO:tasks.workunit.client.1.smithi189.stdout:6/292: creat d8/d2b/d49/f60 x:0 0 0 2024-04-20T21:32:06.689 INFO:tasks.workunit.client.0.smithi062.stdout:9/670: fdatasync d16/d3d/f43 0 2024-04-20T21:32:06.691 INFO:tasks.workunit.client.0.smithi062.stdout:6/495: rename d0/d9/d11/d7b/f84 to d0/d9/d11/fa2 0 2024-04-20T21:32:06.691 INFO:tasks.workunit.client.1.smithi189.stdout:4/267: mknod d2/d22/c62 0 2024-04-20T21:32:06.692 INFO:tasks.workunit.client.1.smithi189.stdout:1/266: dread d8/fa [0,4194304] 0 2024-04-20T21:32:06.694 INFO:tasks.workunit.client.1.smithi189.stdout:4/268: fsync d2/dd/fe 0 2024-04-20T21:32:06.698 INFO:tasks.workunit.client.0.smithi062.stdout:1/681: mkdir dc/df4 0 2024-04-20T21:32:06.699 INFO:tasks.workunit.client.0.smithi062.stdout:8/620: write d7/d30/f5f [620715,69075] 0 2024-04-20T21:32:06.701 INFO:tasks.workunit.client.1.smithi189.stdout:2/257: link d1/d17/d2f/c46 d1/d17/d37/c47 0 2024-04-20T21:32:06.701 INFO:tasks.workunit.client.1.smithi189.stdout:5/234: getdents d7/d14/d1b/d1f/d2a 0 2024-04-20T21:32:06.702 INFO:tasks.workunit.client.0.smithi062.stdout:9/671: getdents d16/d2f/d5b/dab/dcd 0 2024-04-20T21:32:06.703 INFO:tasks.workunit.client.1.smithi189.stdout:9/310: rename d9/d12/f14 to d9/d12/d15/d1a/f64 0 2024-04-20T21:32:06.706 INFO:tasks.workunit.client.0.smithi062.stdout:5/616: getdents d8/d38/d7a/d4c 0 2024-04-20T21:32:06.706 INFO:tasks.workunit.client.0.smithi062.stdout:3/597: write d0/d1/f82 [848188,114125] 0 2024-04-20T21:32:06.708 INFO:tasks.workunit.client.1.smithi189.stdout:8/396: dread da/df/f11 [0,4194304] 0 2024-04-20T21:32:06.708 INFO:tasks.workunit.client.1.smithi189.stdout:8/397: chown da/d81 164369 1 2024-04-20T21:32:06.710 INFO:tasks.workunit.client.1.smithi189.stdout:8/398: write da/df/d20/f7d [391362,1443] 0 2024-04-20T21:32:06.711 INFO:tasks.workunit.client.0.smithi062.stdout:0/611: write d7/d1c/f41 [338145,24812] 0 2024-04-20T21:32:06.711 INFO:tasks.workunit.client.0.smithi062.stdout:8/621: chown d7/l6c 876269 1 2024-04-20T21:32:06.715 INFO:tasks.workunit.client.1.smithi189.stdout:4/269: creat d2/d3a/f63 x:0 0 0 2024-04-20T21:32:06.716 INFO:tasks.workunit.client.1.smithi189.stdout:2/258: symlink d1/d17/l48 0 2024-04-20T21:32:06.717 INFO:tasks.workunit.client.0.smithi062.stdout:3/598: mkdir d0/d1b/d74/dc7 0 2024-04-20T21:32:06.717 INFO:tasks.workunit.client.0.smithi062.stdout:9/672: creat d16/d2c/d5e/d75/fdd x:0 0 0 2024-04-20T21:32:06.717 INFO:tasks.workunit.client.1.smithi189.stdout:0/396: dread d3/f5e [0,4194304] 0 2024-04-20T21:32:06.719 INFO:tasks.workunit.client.1.smithi189.stdout:0/397: write d3/df/d5d/f7c [525850,34117] 0 2024-04-20T21:32:06.720 INFO:tasks.workunit.client.0.smithi062.stdout:6/496: dread d0/d9/d11/f63 [0,4194304] 0 2024-04-20T21:32:06.721 INFO:tasks.workunit.client.0.smithi062.stdout:2/649: dread d7/da/d15/d1b/f2c [0,4194304] 0 2024-04-20T21:32:06.721 INFO:tasks.workunit.client.1.smithi189.stdout:5/235: symlink d7/d14/l41 0 2024-04-20T21:32:06.721 INFO:tasks.workunit.client.1.smithi189.stdout:0/398: chown d3/df/d28/f6a 0 1 2024-04-20T21:32:06.723 INFO:tasks.workunit.client.1.smithi189.stdout:0/399: write d3/f27 [4013734,8398] 0 2024-04-20T21:32:06.724 INFO:tasks.workunit.client.1.smithi189.stdout:9/311: symlink d9/dc/d17/d2b/l65 0 2024-04-20T21:32:06.727 INFO:tasks.workunit.client.0.smithi062.stdout:8/622: fdatasync d7/d25/f93 0 2024-04-20T21:32:06.728 INFO:tasks.workunit.client.0.smithi062.stdout:3/599: unlink d0/f4f 0 2024-04-20T21:32:06.730 INFO:tasks.workunit.client.0.smithi062.stdout:0/612: creat d7/d1c/d15/d17/d40/fcc x:0 0 0 2024-04-20T21:32:06.731 INFO:tasks.workunit.client.0.smithi062.stdout:6/497: chown d0/d1a/f29 7821593 1 2024-04-20T21:32:06.734 INFO:tasks.workunit.client.0.smithi062.stdout:8/623: dread - d7/d25/d3b/f79 zero size 2024-04-20T21:32:06.735 INFO:tasks.workunit.client.1.smithi189.stdout:3/253: creat d2/f5d x:0 0 0 2024-04-20T21:32:06.735 INFO:tasks.workunit.client.1.smithi189.stdout:0/400: dwrite d3/df/d28/d2b/f36 [0,4194304] 0 2024-04-20T21:32:06.736 INFO:tasks.workunit.client.0.smithi062.stdout:4/647: dwrite da/dd/df/d27/d32/f37 [4194304,4194304] 0 2024-04-20T21:32:06.736 INFO:tasks.workunit.client.0.smithi062.stdout:7/660: dwrite d7/f18 [4194304,4194304] 0 2024-04-20T21:32:06.739 INFO:tasks.workunit.client.0.smithi062.stdout:6/498: stat d0/d1a/d7d/c45 0 2024-04-20T21:32:06.739 INFO:tasks.workunit.client.1.smithi189.stdout:8/399: rmdir da/d1f 39 2024-04-20T21:32:06.740 INFO:tasks.workunit.client.0.smithi062.stdout:9/673: creat d16/d2c/d42/fde x:0 0 0 2024-04-20T21:32:06.743 INFO:tasks.workunit.client.1.smithi189.stdout:2/259: creat d1/d17/d2f/f49 x:0 0 0 2024-04-20T21:32:06.744 INFO:tasks.workunit.client.0.smithi062.stdout:2/650: creat d7/da/d15/d2f/d93/fd5 x:0 0 0 2024-04-20T21:32:06.745 INFO:tasks.workunit.client.0.smithi062.stdout:9/674: truncate d16/f3a 4532821 0 2024-04-20T21:32:06.751 INFO:tasks.workunit.client.1.smithi189.stdout:5/236: fsync d7/d14/f26 0 2024-04-20T21:32:06.751 INFO:tasks.workunit.client.0.smithi062.stdout:0/613: unlink d7/d1c/d15/d17/d1b/lb7 0 2024-04-20T21:32:06.752 INFO:tasks.workunit.client.1.smithi189.stdout:5/237: chown l0 914 1 2024-04-20T21:32:06.753 INFO:tasks.workunit.client.0.smithi062.stdout:9/675: creat d16/d3e/fdf x:0 0 0 2024-04-20T21:32:06.755 INFO:tasks.workunit.client.0.smithi062.stdout:7/661: dread - d7/d3e/d33/d3d/d5e/d7e/d80/fdd zero size 2024-04-20T21:32:06.757 INFO:tasks.workunit.client.0.smithi062.stdout:4/648: symlink da/dd/df/d27/d32/d97/le5 0 2024-04-20T21:32:06.759 INFO:tasks.workunit.client.0.smithi062.stdout:6/499: link d0/d2c/d67/f9e d0/d1a/d44/fa3 0 2024-04-20T21:32:06.759 INFO:tasks.workunit.client.0.smithi062.stdout:0/614: stat d7/d1c/d15/d17/d37 0 2024-04-20T21:32:06.759 INFO:tasks.workunit.client.1.smithi189.stdout:9/312: symlink d9/dc/d17/d2b/d52/l66 0 2024-04-20T21:32:06.761 INFO:tasks.workunit.client.1.smithi189.stdout:3/254: creat d2/d3/d2c/d31/f5e x:0 0 0 2024-04-20T21:32:06.762 INFO:tasks.workunit.client.0.smithi062.stdout:2/651: mknod d7/da/d15/d1b/da8/cd6 0 2024-04-20T21:32:06.763 INFO:tasks.workunit.client.1.smithi189.stdout:3/255: rename d2 to d2/d3/d2c/d31/d4c/d5f 22 2024-04-20T21:32:06.767 INFO:tasks.workunit.client.0.smithi062.stdout:6/500: dread - d0/d2c/d67/f9e zero size 2024-04-20T21:32:06.768 INFO:tasks.workunit.client.1.smithi189.stdout:6/293: truncate d8/d10/f2d 3685364 0 2024-04-20T21:32:06.770 INFO:tasks.workunit.client.1.smithi189.stdout:6/294: chown f5 299500906 1 2024-04-20T21:32:06.770 INFO:tasks.workunit.client.0.smithi062.stdout:2/652: fsync d7/da/d28/f6d 0 2024-04-20T21:32:06.770 INFO:tasks.workunit.client.0.smithi062.stdout:5/617: dwrite d8/d38/d7a/f36 [0,4194304] 0 2024-04-20T21:32:06.771 INFO:tasks.workunit.client.0.smithi062.stdout:0/615: mknod d7/d1e/db6/ccd 0 2024-04-20T21:32:06.773 INFO:tasks.workunit.client.1.smithi189.stdout:6/295: write d8/d2b/d5c/f5e [696808,398] 0 2024-04-20T21:32:06.775 INFO:tasks.workunit.client.0.smithi062.stdout:8/624: dread d7/d25/f9d [0,4194304] 0 2024-04-20T21:32:06.775 INFO:tasks.workunit.client.0.smithi062.stdout:4/649: fsync da/dd/d15/d50/d64/d3d/da1/fb9 0 2024-04-20T21:32:06.775 INFO:tasks.workunit.client.0.smithi062.stdout:1/682: dread dc/d15/d20/d28/f63 [0,4194304] 0 2024-04-20T21:32:06.775 INFO:tasks.workunit.client.1.smithi189.stdout:2/260: stat d1/f6 0 2024-04-20T21:32:06.776 INFO:tasks.workunit.client.1.smithi189.stdout:2/261: readlink d1/d17/l48 0 2024-04-20T21:32:06.777 INFO:tasks.workunit.client.1.smithi189.stdout:3/256: dread d2/fd [0,4194304] 0 2024-04-20T21:32:06.778 INFO:tasks.workunit.client.0.smithi062.stdout:2/653: dread - d7/da/d15/d2f/d38/d6b/fd1 zero size 2024-04-20T21:32:06.779 INFO:tasks.workunit.client.1.smithi189.stdout:9/313: dwrite d9/dc/f1c [0,4194304] 0 2024-04-20T21:32:06.780 INFO:tasks.workunit.client.1.smithi189.stdout:3/257: read d2/d3/f48 [797225,29838] 0 2024-04-20T21:32:06.781 INFO:tasks.workunit.client.0.smithi062.stdout:6/501: mkdir d0/d9/d11/d7b/da4 0 2024-04-20T21:32:06.785 INFO:tasks.workunit.client.0.smithi062.stdout:9/676: dread d16/d4e/fa3 [0,4194304] 0 2024-04-20T21:32:06.785 INFO:tasks.workunit.client.1.smithi189.stdout:0/401: getdents d3/df/d28/d2b/d2c/d73 0 2024-04-20T21:32:06.786 INFO:tasks.workunit.client.1.smithi189.stdout:8/400: truncate da/de/f8c 2807443 0 2024-04-20T21:32:06.787 INFO:tasks.workunit.client.1.smithi189.stdout:1/267: link d8/d21/d2c/d36/l4c d8/d1e/d54/l5d 0 2024-04-20T21:32:06.788 INFO:tasks.workunit.client.1.smithi189.stdout:0/402: chown d3/df/d30/d45/f6c 792 1 2024-04-20T21:32:06.789 INFO:tasks.workunit.client.0.smithi062.stdout:8/625: mkdir d7/d25/dc5/dd3 0 2024-04-20T21:32:06.789 INFO:tasks.workunit.client.0.smithi062.stdout:6/502: read d0/d52/f83 [30926,70421] 0 2024-04-20T21:32:06.790 INFO:tasks.workunit.client.0.smithi062.stdout:2/654: chown d7/da/d15/d2f/lad 180432 1 2024-04-20T21:32:06.792 INFO:tasks.workunit.client.1.smithi189.stdout:5/238: creat d7/d9/f42 x:0 0 0 2024-04-20T21:32:06.793 INFO:tasks.workunit.client.0.smithi062.stdout:9/677: write d16/d3d/d55/d6e/d7d/f91 [3853371,77880] 0 2024-04-20T21:32:06.793 INFO:tasks.workunit.client.0.smithi062.stdout:2/655: chown d7/da/d15 1975334 1 2024-04-20T21:32:06.796 INFO:tasks.workunit.client.0.smithi062.stdout:2/656: read d7/da/d28/d4b/f6e [1692242,55319] 0 2024-04-20T21:32:06.798 INFO:tasks.workunit.client.1.smithi189.stdout:2/262: fdatasync d1/d11/f28 0 2024-04-20T21:32:06.799 INFO:tasks.workunit.client.0.smithi062.stdout:0/616: dread d7/d20/d62/d6f/fae [0,4194304] 0 2024-04-20T21:32:06.802 INFO:tasks.workunit.client.0.smithi062.stdout:1/683: creat dc/d15/d75/d96/dd4/ff5 x:0 0 0 2024-04-20T21:32:06.803 INFO:tasks.workunit.client.0.smithi062.stdout:9/678: chown d16/f2e 322947403 1 2024-04-20T21:32:06.804 INFO:tasks.workunit.client.1.smithi189.stdout:9/314: mknod d9/dc/c67 0 2024-04-20T21:32:06.804 INFO:tasks.workunit.client.1.smithi189.stdout:1/268: dwrite d8/f9 [0,4194304] 0 2024-04-20T21:32:06.805 INFO:tasks.workunit.client.0.smithi062.stdout:4/650: getdents da/d56/d71/d9e 0 2024-04-20T21:32:06.808 INFO:tasks.workunit.client.0.smithi062.stdout:1/684: fdatasync dc/d27/d32/f89 0 2024-04-20T21:32:06.810 INFO:tasks.workunit.client.1.smithi189.stdout:3/258: rename d2/d3/l9 to d2/d3/d2c/d31/l60 0 2024-04-20T21:32:06.811 INFO:tasks.workunit.client.0.smithi062.stdout:2/657: mknod d7/da/d15/d1b/da8/cd7 0 2024-04-20T21:32:06.811 INFO:tasks.workunit.client.1.smithi189.stdout:3/259: dread - d2/f5a zero size 2024-04-20T21:32:06.813 INFO:tasks.workunit.client.0.smithi062.stdout:5/618: link d8/d1e/c48 d8/d10/d3e/cc3 0 2024-04-20T21:32:06.813 INFO:tasks.workunit.client.0.smithi062.stdout:2/658: fdatasync d7/da/fbd 0 2024-04-20T21:32:06.817 INFO:tasks.workunit.client.0.smithi062.stdout:0/617: unlink d7/d1c/c98 0 2024-04-20T21:32:06.819 INFO:tasks.workunit.client.0.smithi062.stdout:9/679: rename d16/d3e/fc9 to d16/d2f/d5b/fe0 0 2024-04-20T21:32:06.820 INFO:tasks.workunit.client.0.smithi062.stdout:4/651: symlink da/dd/df/d27/d3f/d82/le6 0 2024-04-20T21:32:06.823 INFO:tasks.workunit.client.1.smithi189.stdout:8/401: unlink da/de/d1e/d21/l32 0 2024-04-20T21:32:06.825 INFO:tasks.workunit.client.0.smithi062.stdout:0/618: stat d7/d1c/d15/d17/d1b/d9d 0 2024-04-20T21:32:06.825 INFO:tasks.workunit.client.1.smithi189.stdout:5/239: creat d7/d35/f43 x:0 0 0 2024-04-20T21:32:06.828 INFO:tasks.workunit.client.0.smithi062.stdout:1/685: mkdir dc/d15/df6 0 2024-04-20T21:32:06.830 INFO:tasks.workunit.client.1.smithi189.stdout:7/294: sync 2024-04-20T21:32:06.831 INFO:tasks.workunit.client.0.smithi062.stdout:2/659: mkdir d7/da/d15/d2f/d93/dac/dca/dd8 0 2024-04-20T21:32:06.832 INFO:tasks.workunit.client.1.smithi189.stdout:6/296: link d8/da/dd/d35/f59 d8/d2b/f61 0 2024-04-20T21:32:06.834 INFO:tasks.workunit.client.0.smithi062.stdout:4/652: chown c8 1968651380 1 2024-04-20T21:32:06.838 INFO:tasks.workunit.client.0.smithi062.stdout:1/686: mknod dc/d15/d20/d28/d76/cf7 0 2024-04-20T21:32:06.838 INFO:tasks.workunit.client.1.smithi189.stdout:2/263: mkdir d1/d17/d2f/d4a 0 2024-04-20T21:32:06.840 INFO:tasks.workunit.client.0.smithi062.stdout:9/680: symlink d16/d2f/d5b/da4/le1 0 2024-04-20T21:32:06.840 INFO:tasks.workunit.client.0.smithi062.stdout:0/619: creat d7/d1c/d15/d17/d40/d3c/dc5/fce x:0 0 0 2024-04-20T21:32:06.840 INFO:tasks.workunit.client.1.smithi189.stdout:9/315: chown d9/dc/d17/d36/d5d/c5f 23 1 2024-04-20T21:32:06.842 INFO:tasks.workunit.client.1.smithi189.stdout:3/260: creat d2/d3/d2c/d30/f61 x:0 0 0 2024-04-20T21:32:06.843 INFO:tasks.workunit.client.1.smithi189.stdout:5/240: dwrite d7/d9/d31/d36/f38 [0,4194304] 0 2024-04-20T21:32:06.843 INFO:tasks.workunit.client.1.smithi189.stdout:4/270: getdents d2/dd/d11/d14 0 2024-04-20T21:32:06.844 INFO:tasks.workunit.client.1.smithi189.stdout:9/316: write d9/dc/d17/f2f [998086,29721] 0 2024-04-20T21:32:06.847 INFO:tasks.workunit.client.1.smithi189.stdout:7/295: creat d5/d21/d25/f66 x:0 0 0 2024-04-20T21:32:06.848 INFO:tasks.workunit.client.1.smithi189.stdout:3/261: chown d2/d3/d2c/c43 579994576 1 2024-04-20T21:32:06.848 INFO:tasks.workunit.client.1.smithi189.stdout:6/297: rmdir d8/d12/d30 39 2024-04-20T21:32:06.849 INFO:tasks.workunit.client.1.smithi189.stdout:1/269: mknod d8/df/c5e 0 2024-04-20T21:32:06.851 INFO:tasks.workunit.client.0.smithi062.stdout:6/503: dwrite d0/d9/f56 [0,4194304] 0 2024-04-20T21:32:06.853 INFO:tasks.workunit.client.0.smithi062.stdout:7/662: dwrite d7/d3e/d33/d3d/d4d/d54/f1f [0,4194304] 0 2024-04-20T21:32:06.856 INFO:tasks.workunit.client.0.smithi062.stdout:1/687: write dc/d15/f18 [3650241,125125] 0 2024-04-20T21:32:06.856 INFO:tasks.workunit.client.0.smithi062.stdout:5/619: write d8/d38/d7a/d4c/f7f [1441333,10443] 0 2024-04-20T21:32:06.861 INFO:tasks.workunit.client.0.smithi062.stdout:4/653: write da/dd/f35 [4818266,90219] 0 2024-04-20T21:32:06.863 INFO:tasks.workunit.client.0.smithi062.stdout:1/688: chown dc/c10 1 1 2024-04-20T21:32:06.865 INFO:tasks.workunit.client.0.smithi062.stdout:6/504: creat d0/d1a/d44/fa5 x:0 0 0 2024-04-20T21:32:06.866 INFO:tasks.workunit.client.1.smithi189.stdout:5/241: dread d7/d14/d18/f3e [0,4194304] 0 2024-04-20T21:32:06.870 INFO:tasks.workunit.client.1.smithi189.stdout:6/298: creat d8/da/dd/d35/f62 x:0 0 0 2024-04-20T21:32:06.872 INFO:tasks.workunit.client.0.smithi062.stdout:5/620: mknod d8/d1e/d2c/cc4 0 2024-04-20T21:32:06.874 INFO:tasks.workunit.client.1.smithi189.stdout:6/299: chown d8/d2b/d5c/f5e 4113 1 2024-04-20T21:32:06.874 INFO:tasks.workunit.client.0.smithi062.stdout:8/626: dread f1 [0,4194304] 0 2024-04-20T21:32:06.875 INFO:tasks.workunit.client.0.smithi062.stdout:2/660: dread d7/ff [0,4194304] 0 2024-04-20T21:32:06.875 INFO:tasks.workunit.client.0.smithi062.stdout:7/663: truncate d7/d3e/d33/d3d/d5e/d7e/d80/fdd 846619 0 2024-04-20T21:32:06.876 INFO:tasks.workunit.client.1.smithi189.stdout:0/403: rename d3/df/d4c/f78 to d3/df/d28/d2b/f86 0 2024-04-20T21:32:06.877 INFO:tasks.workunit.client.0.smithi062.stdout:4/654: unlink da/dd/d15/d50/d64/d57/d5b/f94 0 2024-04-20T21:32:06.877 INFO:tasks.workunit.client.0.smithi062.stdout:3/600: sync 2024-04-20T21:32:06.882 INFO:tasks.workunit.client.0.smithi062.stdout:6/505: mknod d0/d9/d11/ca6 0 2024-04-20T21:32:06.883 INFO:tasks.workunit.client.1.smithi189.stdout:5/242: dread d7/f2b [0,4194304] 0 2024-04-20T21:32:06.885 INFO:tasks.workunit.client.0.smithi062.stdout:1/689: fdatasync dc/d15/d75/d96/dd4/d3a/f4d 0 2024-04-20T21:32:06.889 INFO:tasks.workunit.client.1.smithi189.stdout:3/262: mknod d2/d3/d2c/d31/d4c/c62 0 2024-04-20T21:32:06.889 INFO:tasks.workunit.client.0.smithi062.stdout:3/601: mkdir d0/d1c/d63/d42/dc8 0 2024-04-20T21:32:06.891 INFO:tasks.workunit.client.0.smithi062.stdout:2/661: creat d7/da/d15/d2f/d97/fd9 x:0 0 0 2024-04-20T21:32:06.892 INFO:tasks.workunit.client.1.smithi189.stdout:3/263: chown d2/d8/f34 534414522 1 2024-04-20T21:32:06.893 INFO:tasks.workunit.client.0.smithi062.stdout:1/690: fsync dc/d15/d20/fea 0 2024-04-20T21:32:06.893 INFO:tasks.workunit.client.0.smithi062.stdout:3/602: stat d0/d5c/l9a 0 2024-04-20T21:32:06.894 INFO:tasks.workunit.client.0.smithi062.stdout:4/655: symlink da/dd/d15/d4b/ddb/le7 0 2024-04-20T21:32:06.894 INFO:tasks.workunit.client.1.smithi189.stdout:3/264: read - d2/d8/f58 zero size 2024-04-20T21:32:06.896 INFO:tasks.workunit.client.0.smithi062.stdout:7/664: symlink d7/d3e/d33/le4 0 2024-04-20T21:32:06.897 INFO:tasks.workunit.client.1.smithi189.stdout:5/243: creat d7/d35/f44 x:0 0 0 2024-04-20T21:32:06.899 INFO:tasks.workunit.client.1.smithi189.stdout:2/264: write d1/f6 [719962,109118] 0 2024-04-20T21:32:06.900 INFO:tasks.workunit.client.0.smithi062.stdout:6/506: rename d0/d9/d11/d7b/da4 to d0/d1a/d44/da7 0 2024-04-20T21:32:06.901 INFO:tasks.workunit.client.0.smithi062.stdout:0/620: dwrite d7/d20/d62/d6f/f9b [0,4194304] 0 2024-04-20T21:32:06.901 INFO:tasks.workunit.client.1.smithi189.stdout:0/404: mkdir d3/df/d30/d45/d60/d79/d87 0 2024-04-20T21:32:06.903 INFO:tasks.workunit.client.1.smithi189.stdout:0/405: read d3/fe [3762150,84087] 0 2024-04-20T21:32:06.903 INFO:tasks.workunit.client.0.smithi062.stdout:1/691: readlink dc/d15/d20/l3b 0 2024-04-20T21:32:06.904 INFO:tasks.workunit.client.1.smithi189.stdout:7/296: rmdir d5/d1d/d31 0 2024-04-20T21:32:06.904 INFO:tasks.workunit.client.1.smithi189.stdout:2/265: rename d1/d17/d2f to d1/d17/d2f/d4a/d4b 22 2024-04-20T21:32:06.907 INFO:tasks.workunit.client.1.smithi189.stdout:3/265: creat d2/d3/d2c/d31/d4c/f63 x:0 0 0 2024-04-20T21:32:06.907 INFO:tasks.workunit.client.0.smithi062.stdout:7/665: chown d7/d3e/d33/d3d/c51 26974704 1 2024-04-20T21:32:06.908 INFO:tasks.workunit.client.1.smithi189.stdout:1/270: getdents d8/df/d3a/d49 0 2024-04-20T21:32:06.908 INFO:tasks.workunit.client.1.smithi189.stdout:9/317: getdents d9/dc/d17/d36 0 2024-04-20T21:32:06.909 INFO:tasks.workunit.client.0.smithi062.stdout:4/656: chown da/dd/d15/d50/d64/d57/d5b/cde 404 1 2024-04-20T21:32:06.910 INFO:tasks.workunit.client.1.smithi189.stdout:6/300: creat d8/f63 x:0 0 0 2024-04-20T21:32:06.910 INFO:tasks.workunit.client.1.smithi189.stdout:8/402: chown da/de/f8c 7 1 2024-04-20T21:32:06.912 INFO:tasks.workunit.client.1.smithi189.stdout:4/271: truncate d2/dd/f2d 5419379 0 2024-04-20T21:32:06.913 INFO:tasks.workunit.client.0.smithi062.stdout:7/666: fsync d7/d3e/d33/d3d/d4d/fa4 0 2024-04-20T21:32:06.916 INFO:tasks.workunit.client.1.smithi189.stdout:8/403: readlink da/de/d1e/d21/d44/l60 0 2024-04-20T21:32:06.918 INFO:tasks.workunit.client.0.smithi062.stdout:6/507: dread d0/d9/d11/f65 [0,4194304] 0 2024-04-20T21:32:06.918 INFO:tasks.workunit.client.0.smithi062.stdout:1/692: dread - dc/d15/d20/d28/d83/dc0/fcb zero size 2024-04-20T21:32:06.918 INFO:tasks.workunit.client.0.smithi062.stdout:2/662: creat d7/da/d15/d1b/d3d/d84/d9b/fda x:0 0 0 2024-04-20T21:32:06.919 INFO:tasks.workunit.client.1.smithi189.stdout:3/266: unlink d2/f29 0 2024-04-20T21:32:06.920 INFO:tasks.workunit.client.1.smithi189.stdout:5/244: dwrite d7/d35/f44 [0,4194304] 0 2024-04-20T21:32:06.921 INFO:tasks.workunit.client.0.smithi062.stdout:3/603: mkdir d0/d1c/d63/d42/dc8/dc9 0 2024-04-20T21:32:06.924 INFO:tasks.workunit.client.1.smithi189.stdout:1/271: creat d8/d1e/d28/d4f/f5f x:0 0 0 2024-04-20T21:32:06.925 INFO:tasks.workunit.client.1.smithi189.stdout:7/297: mknod d5/d21/d35/d53/d5b/c67 0 2024-04-20T21:32:06.925 INFO:tasks.workunit.client.1.smithi189.stdout:8/404: dread da/df/d20/f7d [0,4194304] 0 2024-04-20T21:32:06.927 INFO:tasks.workunit.client.0.smithi062.stdout:5/621: link d8/d38/d7a/d4c/d7d/d9d/dac/d2f/f4b d8/d38/fc5 0 2024-04-20T21:32:06.928 INFO:tasks.workunit.client.0.smithi062.stdout:7/667: unlink d7/d26/f2c 0 2024-04-20T21:32:06.931 INFO:tasks.workunit.client.1.smithi189.stdout:5/245: symlink d7/d9/d31/l45 0 2024-04-20T21:32:06.931 INFO:tasks.workunit.client.1.smithi189.stdout:0/406: dwrite d3/f1f [0,4194304] 0 2024-04-20T21:32:06.931 INFO:tasks.workunit.client.1.smithi189.stdout:6/301: read d8/da/f32 [2096602,127129] 0 2024-04-20T21:32:06.932 INFO:tasks.workunit.client.0.smithi062.stdout:1/693: fsync dc/d15/d75/d96/dd4/d3a/f46 0 2024-04-20T21:32:06.933 INFO:tasks.workunit.client.1.smithi189.stdout:7/298: write d5/d21/d25/f66 [754795,57892] 0 2024-04-20T21:32:06.938 INFO:tasks.workunit.client.0.smithi062.stdout:5/622: rmdir d8/d38/d7a/db1 39 2024-04-20T21:32:06.938 INFO:tasks.workunit.client.1.smithi189.stdout:4/272: mkdir d2/d64 0 2024-04-20T21:32:06.941 INFO:tasks.workunit.client.1.smithi189.stdout:4/273: chown d2/dd/d11/d14/c23 694629209 1 2024-04-20T21:32:06.941 INFO:tasks.workunit.client.0.smithi062.stdout:9/681: dread d16/d2c/d5e/d75/d76/fb6 [0,4194304] 0 2024-04-20T21:32:06.941 INFO:tasks.workunit.client.0.smithi062.stdout:2/663: chown d7/da/d15/d2f/d38/d6a/ccc 0 1 2024-04-20T21:32:06.943 INFO:tasks.workunit.client.0.smithi062.stdout:1/694: mkdir dc/d27/d32/df8 0 2024-04-20T21:32:06.943 INFO:tasks.workunit.client.1.smithi189.stdout:3/267: symlink d2/l64 0 2024-04-20T21:32:06.946 INFO:tasks.workunit.client.0.smithi062.stdout:3/604: mknod d0/d1/d10/d3a/cca 0 2024-04-20T21:32:06.948 INFO:tasks.workunit.client.1.smithi189.stdout:5/246: creat d7/d35/f46 x:0 0 0 2024-04-20T21:32:06.948 INFO:tasks.workunit.client.0.smithi062.stdout:8/627: dwrite d7/fad [0,4194304] 0 2024-04-20T21:32:06.948 INFO:tasks.workunit.client.1.smithi189.stdout:3/268: chown d2/d3/d2c/d31/d4c 265031834 1 2024-04-20T21:32:06.950 INFO:tasks.workunit.client.1.smithi189.stdout:2/266: write d1/d17/f2b [778988,88953] 0 2024-04-20T21:32:06.952 INFO:tasks.workunit.client.1.smithi189.stdout:1/272: dwrite f6 [0,4194304] 0 2024-04-20T21:32:06.953 INFO:tasks.workunit.client.1.smithi189.stdout:2/267: write d1/d17/d2f/f42 [2677530,3057] 0 2024-04-20T21:32:06.955 INFO:tasks.workunit.client.1.smithi189.stdout:0/407: dread d3/df/d30/d45/d60/f68 [0,4194304] 0 2024-04-20T21:32:06.957 INFO:tasks.workunit.client.0.smithi062.stdout:4/657: dread f3 [0,4194304] 0 2024-04-20T21:32:06.957 INFO:tasks.workunit.client.0.smithi062.stdout:3/605: creat d0/d1c/d63/d42/fcb x:0 0 0 2024-04-20T21:32:06.960 INFO:tasks.workunit.client.1.smithi189.stdout:0/408: stat d3/df/d30/d45/d60/f81 0 2024-04-20T21:32:06.960 INFO:tasks.workunit.client.1.smithi189.stdout:6/302: mknod d8/da/c64 0 2024-04-20T21:32:06.960 INFO:tasks.workunit.client.0.smithi062.stdout:5/623: creat d8/d38/d7a/fc6 x:0 0 0 2024-04-20T21:32:06.964 INFO:tasks.workunit.client.1.smithi189.stdout:6/303: write d8/da/dd/d35/f45 [134861,105395] 0 2024-04-20T21:32:06.966 INFO:tasks.workunit.client.0.smithi062.stdout:3/606: unlink d0/d1/d10/d31/f5f 0 2024-04-20T21:32:06.972 INFO:tasks.workunit.client.0.smithi062.stdout:0/621: dread d7/d20/d3a/f44 [0,4194304] 0 2024-04-20T21:32:06.977 INFO:tasks.workunit.client.0.smithi062.stdout:3/607: truncate d0/d1c/d52/d7b/f85 205626 0 2024-04-20T21:32:06.981 INFO:tasks.workunit.client.1.smithi189.stdout:2/268: dread d1/f2a [0,4194304] 0 2024-04-20T21:32:06.981 INFO:tasks.workunit.client.1.smithi189.stdout:4/274: mknod d2/d22/d57/c65 0 2024-04-20T21:32:06.981 INFO:tasks.workunit.client.1.smithi189.stdout:9/318: truncate d9/d12/d15/f26 3792648 0 2024-04-20T21:32:06.981 INFO:tasks.workunit.client.1.smithi189.stdout:6/304: write f7 [5101568,90590] 0 2024-04-20T21:32:06.982 INFO:tasks.workunit.client.0.smithi062.stdout:5/624: rename d8/d38/d7a/d4c/d7d/d9d/dac/f73 to d8/d38/d7a/d4c/fc7 0 2024-04-20T21:32:06.986 INFO:tasks.workunit.client.1.smithi189.stdout:6/305: readlink d8/da/dd/l56 0 2024-04-20T21:32:06.989 INFO:tasks.workunit.client.1.smithi189.stdout:5/247: dread d7/d14/d18/f3e [0,4194304] 0 2024-04-20T21:32:06.989 INFO:tasks.workunit.client.0.smithi062.stdout:0/622: symlink d7/d1c/d15/d17/d40/d3c/dc5/lcf 0 2024-04-20T21:32:06.990 INFO:tasks.workunit.client.1.smithi189.stdout:9/319: write d9/d12/d15/f31 [1974122,39299] 0 2024-04-20T21:32:06.991 INFO:tasks.workunit.client.1.smithi189.stdout:7/299: dwrite d5/f39 [0,4194304] 0 2024-04-20T21:32:06.992 INFO:tasks.workunit.client.0.smithi062.stdout:3/608: getdents d0/d5c 0 2024-04-20T21:32:07.003 INFO:tasks.workunit.client.1.smithi189.stdout:6/306: symlink d8/da/dd/d35/l65 0 2024-04-20T21:32:07.003 INFO:tasks.workunit.client.1.smithi189.stdout:8/405: link da/df/d13/l82 da/df/d22/l8e 0 2024-04-20T21:32:07.003 INFO:tasks.workunit.client.0.smithi062.stdout:0/623: rename d7/d20/d62/d6f/f9f to d7/d1c/d15/d17/fd0 0 2024-04-20T21:32:07.007 INFO:tasks.workunit.client.1.smithi189.stdout:7/300: readlink d5/d21/d4a/l54 0 2024-04-20T21:32:07.009 INFO:tasks.workunit.client.1.smithi189.stdout:0/409: dwrite d3/df/d30/d45/d60/f44 [0,4194304] 0 2024-04-20T21:32:07.011 INFO:tasks.workunit.client.0.smithi062.stdout:6/508: dread d0/d1a/f3a [0,4194304] 0 2024-04-20T21:32:07.011 INFO:tasks.workunit.client.0.smithi062.stdout:4/658: dwrite da/dd/d15/d50/d64/d3d/da1/fb9 [0,4194304] 0 2024-04-20T21:32:07.011 INFO:tasks.workunit.client.0.smithi062.stdout:2/664: dread d7/da/d15/d1b/d3d/f49 [0,4194304] 0 2024-04-20T21:32:07.012 INFO:tasks.workunit.client.0.smithi062.stdout:0/624: creat d7/d20/d62/fd1 x:0 0 0 2024-04-20T21:32:07.012 INFO:tasks.workunit.client.1.smithi189.stdout:1/273: dwrite d8/f46 [0,4194304] 0 2024-04-20T21:32:07.015 INFO:tasks.workunit.client.0.smithi062.stdout:5/625: symlink d8/d38/d7a/d4c/d7d/d9d/dac/lc8 0 2024-04-20T21:32:07.015 INFO:tasks.workunit.client.1.smithi189.stdout:2/269: write d1/d11/d2d/f41 [4131400,7572] 0 2024-04-20T21:32:07.018 INFO:tasks.workunit.client.1.smithi189.stdout:0/410: readlink d3/df/d30/d37/l80 0 2024-04-20T21:32:07.019 INFO:tasks.workunit.client.1.smithi189.stdout:1/274: write d8/f14 [3352056,28919] 0 2024-04-20T21:32:07.019 INFO:tasks.workunit.client.1.smithi189.stdout:6/307: creat d8/da/dd/d2a/f66 x:0 0 0 2024-04-20T21:32:07.020 INFO:tasks.workunit.client.0.smithi062.stdout:2/665: chown d7/da/d15/d2f/d38/d6a/d7c/f8c 125350 1 2024-04-20T21:32:07.021 INFO:tasks.workunit.client.1.smithi189.stdout:7/301: mknod d5/d46/c68 0 2024-04-20T21:32:07.021 INFO:tasks.workunit.client.1.smithi189.stdout:4/275: link d2/dd/d11/d14/l34 d2/l66 0 2024-04-20T21:32:07.022 INFO:tasks.workunit.client.1.smithi189.stdout:2/270: mknod d1/d17/d37/c4c 0 2024-04-20T21:32:07.024 INFO:tasks.workunit.client.1.smithi189.stdout:4/276: chown d2/d22/d57 692408 1 2024-04-20T21:32:07.026 INFO:tasks.workunit.client.0.smithi062.stdout:1/695: dread dc/d15/d75/f91 [0,4194304] 0 2024-04-20T21:32:07.026 INFO:tasks.workunit.client.0.smithi062.stdout:0/625: fsync d7/d1c/f18 0 2024-04-20T21:32:07.027 INFO:tasks.workunit.client.1.smithi189.stdout:0/411: mkdir d3/d26/d88 0 2024-04-20T21:32:07.027 INFO:tasks.workunit.client.0.smithi062.stdout:8/628: dread d7/d9/d29/f32 [0,4194304] 0 2024-04-20T21:32:07.030 INFO:tasks.workunit.client.1.smithi189.stdout:2/271: truncate d1/d17/d2f/f49 975587 0 2024-04-20T21:32:07.030 INFO:tasks.workunit.client.1.smithi189.stdout:7/302: write d5/d1d/f44 [3203510,96127] 0 2024-04-20T21:32:07.031 INFO:tasks.workunit.client.1.smithi189.stdout:5/248: dread d7/d14/d18/f17 [0,4194304] 0 2024-04-20T21:32:07.031 INFO:tasks.workunit.client.1.smithi189.stdout:9/320: dwrite d9/d12/d15/d1a/f5c [0,4194304] 0 2024-04-20T21:32:07.031 INFO:tasks.workunit.client.0.smithi062.stdout:5/626: creat d8/d10/d3e/d8a/d9e/fc9 x:0 0 0 2024-04-20T21:32:07.031 INFO:tasks.workunit.client.0.smithi062.stdout:1/696: creat dc/d15/d75/ff9 x:0 0 0 2024-04-20T21:32:07.034 INFO:tasks.workunit.client.1.smithi189.stdout:1/275: mkdir d8/d1e/d43/d58/d60 0 2024-04-20T21:32:07.035 INFO:tasks.workunit.client.0.smithi062.stdout:6/509: rename d0/d1a/d44/fa5 to d0/d52/da0/fa8 0 2024-04-20T21:32:07.036 INFO:tasks.workunit.client.1.smithi189.stdout:0/412: fsync d3/df/d28/d2b/f67 0 2024-04-20T21:32:07.036 INFO:tasks.workunit.client.0.smithi062.stdout:2/666: fdatasync d7/da/d15/d2f/d38/f4e 0 2024-04-20T21:32:07.036 INFO:tasks.workunit.client.1.smithi189.stdout:8/406: link da/df/d3a/l7c da/df/d3c/d50/d6c/l8f 0 2024-04-20T21:32:07.040 INFO:tasks.workunit.client.1.smithi189.stdout:2/272: truncate d1/d17/f2b 1426047 0 2024-04-20T21:32:07.040 INFO:tasks.workunit.client.1.smithi189.stdout:0/413: stat d3/df/d5d/d64 0 2024-04-20T21:32:07.041 INFO:tasks.workunit.client.0.smithi062.stdout:1/697: truncate dc/d15/d75/ff9 1001598 0 2024-04-20T21:32:07.042 INFO:tasks.workunit.client.0.smithi062.stdout:4/659: rename da/dd/d15/d50/d64/d57 to da/dd/d15/d50/d64/d57/d5b/d9d/de8 22 2024-04-20T21:32:07.043 INFO:tasks.workunit.client.1.smithi189.stdout:5/249: stat d7/d14/d1b/c40 0 2024-04-20T21:32:07.044 INFO:tasks.workunit.client.1.smithi189.stdout:9/321: truncate d9/d12/d15/d1a/f27 1062801 0 2024-04-20T21:32:07.045 INFO:tasks.workunit.client.1.smithi189.stdout:0/414: dread - d3/df/d5d/d64/f6d zero size 2024-04-20T21:32:07.046 INFO:tasks.workunit.client.1.smithi189.stdout:5/250: chown d7/d9/d31 9885070 1 2024-04-20T21:32:07.048 INFO:tasks.workunit.client.1.smithi189.stdout:6/308: rmdir d8/da/dd/d2a/d3b/d5b 0 2024-04-20T21:32:07.049 INFO:tasks.workunit.client.0.smithi062.stdout:5/627: unlink d8/d10/c49 0 2024-04-20T21:32:07.049 INFO:tasks.workunit.client.0.smithi062.stdout:8/629: unlink d7/d25/d3b/d58/d8b/d24/f20 0 2024-04-20T21:32:07.049 INFO:tasks.workunit.client.0.smithi062.stdout:3/609: dwrite d0/d1/d10/d31/d41/f7c [0,4194304] 0 2024-04-20T21:32:07.053 INFO:tasks.workunit.client.1.smithi189.stdout:7/303: mkdir d5/d21/d35/d53/d69 0 2024-04-20T21:32:07.058 INFO:tasks.workunit.client.1.smithi189.stdout:7/304: read d5/fe [354002,83879] 0 2024-04-20T21:32:07.058 INFO:tasks.workunit.client.1.smithi189.stdout:4/277: link d2/f38 d2/dd/d11/f67 0 2024-04-20T21:32:07.059 INFO:tasks.workunit.client.1.smithi189.stdout:8/407: read da/df/f19 [1901155,37832] 0 2024-04-20T21:32:07.061 INFO:tasks.workunit.client.0.smithi062.stdout:3/610: mknod d0/d1/d10/ccc 0 2024-04-20T21:32:07.062 INFO:tasks.workunit.client.0.smithi062.stdout:8/630: truncate d7/d30/d90/fc6 7768 0 2024-04-20T21:32:07.062 INFO:tasks.workunit.client.0.smithi062.stdout:5/628: mknod d8/d38/cca 0 2024-04-20T21:32:07.065 INFO:tasks.workunit.client.1.smithi189.stdout:1/276: rename d8/df/f3d to d8/d1e/d28/d2b/f61 0 2024-04-20T21:32:07.068 INFO:tasks.workunit.client.0.smithi062.stdout:4/660: rename da/dd/f11 to da/dd/df/fe9 0 2024-04-20T21:32:07.069 INFO:tasks.workunit.client.0.smithi062.stdout:6/510: dread d0/d9/f13 [0,4194304] 0 2024-04-20T21:32:07.069 INFO:tasks.workunit.client.1.smithi189.stdout:6/309: unlink f7 0 2024-04-20T21:32:07.069 INFO:tasks.workunit.client.1.smithi189.stdout:1/277: write f2 [244035,53058] 0 2024-04-20T21:32:07.071 INFO:tasks.workunit.client.1.smithi189.stdout:1/278: stat d8/ld 0 2024-04-20T21:32:07.072 INFO:tasks.workunit.client.0.smithi062.stdout:3/611: mknod d0/d1/d10/d99/ccd 0 2024-04-20T21:32:07.073 INFO:tasks.workunit.client.1.smithi189.stdout:5/251: truncate d7/f2b 817255 0 2024-04-20T21:32:07.073 INFO:tasks.workunit.client.1.smithi189.stdout:1/279: stat d8/d1e/d43 0 2024-04-20T21:32:07.073 INFO:tasks.workunit.client.1.smithi189.stdout:0/415: symlink d3/df/d28/d2b/d2c/d74/l89 0 2024-04-20T21:32:07.076 INFO:tasks.workunit.client.1.smithi189.stdout:0/416: truncate d3/df/d5d/d64/f6b 760998 0 2024-04-20T21:32:07.077 INFO:tasks.workunit.client.0.smithi062.stdout:5/629: write d8/d38/d7a/d4c/fc7 [4580810,66721] 0 2024-04-20T21:32:07.079 INFO:tasks.workunit.client.1.smithi189.stdout:4/278: stat d2/l66 0 2024-04-20T21:32:07.079 INFO:tasks.workunit.client.0.smithi062.stdout:3/612: creat d0/d1c/fce x:0 0 0 2024-04-20T21:32:07.080 INFO:tasks.workunit.client.0.smithi062.stdout:8/631: dread d7/d25/d4f/fbc [0,4194304] 0 2024-04-20T21:32:07.081 INFO:tasks.workunit.client.1.smithi189.stdout:7/305: fsync d5/f1f 0 2024-04-20T21:32:07.081 INFO:tasks.workunit.client.0.smithi062.stdout:1/698: dread dc/d15/d20/d28/f63 [0,4194304] 0 2024-04-20T21:32:07.086 INFO:tasks.workunit.client.0.smithi062.stdout:8/632: stat d7/d25/d3b/d58/dac/dc7 0 2024-04-20T21:32:07.087 INFO:tasks.workunit.client.0.smithi062.stdout:3/613: chown d0/d1c/l6d 15 1 2024-04-20T21:32:07.087 INFO:tasks.workunit.client.0.smithi062.stdout:6/511: rename d0/d2c/d67/c70 to d0/d1a/d7d/ca9 0 2024-04-20T21:32:07.088 INFO:tasks.workunit.client.0.smithi062.stdout:5/630: creat d8/d38/d7a/d4c/d7d/d9d/dac/d66/fcb x:0 0 0 2024-04-20T21:32:07.089 INFO:tasks.workunit.client.1.smithi189.stdout:9/322: rename f8 to d9/d12/d3f/f68 0 2024-04-20T21:32:07.089 INFO:tasks.workunit.client.0.smithi062.stdout:0/626: dwrite d7/d1c/f39 [0,4194304] 0 2024-04-20T21:32:07.092 INFO:tasks.workunit.client.0.smithi062.stdout:3/614: stat d0/d1c/d63/f91 0 2024-04-20T21:32:07.095 INFO:tasks.workunit.client.0.smithi062.stdout:0/627: symlink d7/d1c/d15/d17/d37/ld2 0 2024-04-20T21:32:07.096 INFO:tasks.workunit.client.0.smithi062.stdout:8/633: chown d7/da9/cb4 35290201 1 2024-04-20T21:32:07.097 INFO:tasks.workunit.client.1.smithi189.stdout:2/273: link d1/d17/d37/c43 d1/d17/d2f/d4a/c4d 0 2024-04-20T21:32:07.098 INFO:tasks.workunit.client.0.smithi062.stdout:1/699: link dc/d15/d75/d96/dd4/d3a/f6a dc/d15/d75/d96/dd4/d3a/da2/de4/d7c/ffa 0 2024-04-20T21:32:07.100 INFO:tasks.workunit.client.1.smithi189.stdout:8/408: creat da/df/d3c/d68/d8b/d8d/f90 x:0 0 0 2024-04-20T21:32:07.101 INFO:tasks.workunit.client.1.smithi189.stdout:1/280: chown d8/df/l33 179645833 1 2024-04-20T21:32:07.104 INFO:tasks.workunit.client.0.smithi062.stdout:2/667: dread d7/da/d28/f79 [0,4194304] 0 2024-04-20T21:32:07.105 INFO:tasks.workunit.client.1.smithi189.stdout:1/281: write d8/d21/d2c/d36/f52 [707335,83854] 0 2024-04-20T21:32:07.105 INFO:tasks.workunit.client.1.smithi189.stdout:6/310: mkdir d8/d2b/d49/d54/d67 0 2024-04-20T21:32:07.107 INFO:tasks.workunit.client.0.smithi062.stdout:5/631: fsync d8/d38/d7a/d4c/d7d/d9d/dac/d2f/d8e/f97 0 2024-04-20T21:32:07.108 INFO:tasks.workunit.client.1.smithi189.stdout:1/282: chown d8/d1e/d54 3077209 1 2024-04-20T21:32:07.109 INFO:tasks.workunit.client.0.smithi062.stdout:0/628: creat d7/d1e/db6/fd3 x:0 0 0 2024-04-20T21:32:07.112 INFO:tasks.workunit.client.0.smithi062.stdout:3/615: getdents d0/d1/d10/d31/d41 0 2024-04-20T21:32:07.113 INFO:tasks.workunit.client.1.smithi189.stdout:9/323: dwrite d9/dc/d17/d36/f4a [0,4194304] 0 2024-04-20T21:32:07.114 INFO:tasks.workunit.client.1.smithi189.stdout:8/409: mkdir da/d81/d91 0 2024-04-20T21:32:07.114 INFO:tasks.workunit.client.1.smithi189.stdout:7/306: rmdir d5/d21/d35/d53/d69 0 2024-04-20T21:32:07.114 INFO:tasks.workunit.client.0.smithi062.stdout:2/668: fsync d7/da/d15/d2f/f7e 0 2024-04-20T21:32:07.116 INFO:tasks.workunit.client.1.smithi189.stdout:6/311: creat d8/da/dd/d2a/d3b/f68 x:0 0 0 2024-04-20T21:32:07.117 INFO:tasks.workunit.client.0.smithi062.stdout:6/512: dwrite d0/d9/f56 [4194304,4194304] 0 2024-04-20T21:32:07.119 INFO:tasks.workunit.client.0.smithi062.stdout:4/661: dwrite da/dd/d15/d50/d64/d3d/fc6 [0,4194304] 0 2024-04-20T21:32:07.119 INFO:tasks.workunit.client.1.smithi189.stdout:4/279: rename d2/f18 to d2/dd/d11/f68 0 2024-04-20T21:32:07.122 INFO:tasks.workunit.client.0.smithi062.stdout:3/616: fdatasync d0/d1/d10/d31/f92 0 2024-04-20T21:32:07.124 INFO:tasks.workunit.client.0.smithi062.stdout:1/700: truncate dc/d15/d75/d96/dd4/d3a/f4f 4384221 0 2024-04-20T21:32:07.125 INFO:tasks.workunit.client.1.smithi189.stdout:1/283: creat d8/d21/d2c/f62 x:0 0 0 2024-04-20T21:32:07.126 INFO:tasks.workunit.client.0.smithi062.stdout:3/617: chown d0/d5c/d76/faa 0 1 2024-04-20T21:32:07.126 INFO:tasks.workunit.client.0.smithi062.stdout:5/632: creat d8/d38/d7a/d4c/d7d/d9d/fcc x:0 0 0 2024-04-20T21:32:07.128 INFO:tasks.workunit.client.1.smithi189.stdout:9/324: rmdir d9/d12 39 2024-04-20T21:32:07.134 INFO:tasks.workunit.client.0.smithi062.stdout:6/513: mkdir d0/d2c/d67/daa 0 2024-04-20T21:32:07.136 INFO:tasks.workunit.client.0.smithi062.stdout:0/629: dread d7/d1e/f45 [0,4194304] 0 2024-04-20T21:32:07.137 INFO:tasks.workunit.client.0.smithi062.stdout:1/701: creat dc/d15/df6/ffb x:0 0 0 2024-04-20T21:32:07.138 INFO:tasks.workunit.client.1.smithi189.stdout:8/410: creat da/df/d20/d2c/f92 x:0 0 0 2024-04-20T21:32:07.138 INFO:tasks.workunit.client.0.smithi062.stdout:2/669: mknod d7/da/d15/d1b/d3d/d84/cdb 0 2024-04-20T21:32:07.138 INFO:tasks.workunit.client.0.smithi062.stdout:6/514: stat d0/d9/d11/ca6 0 2024-04-20T21:32:07.140 INFO:tasks.workunit.client.1.smithi189.stdout:8/411: write da/df/d13/f5f [2998703,16376] 0 2024-04-20T21:32:07.140 INFO:tasks.workunit.client.0.smithi062.stdout:8/634: link d7/d25/d3b/d58/d8b/d24/l47 d7/d25/d76/ld4 0 2024-04-20T21:32:07.141 INFO:tasks.workunit.client.0.smithi062.stdout:5/633: creat d8/d38/d7a/d4c/fcd x:0 0 0 2024-04-20T21:32:07.143 INFO:tasks.workunit.client.1.smithi189.stdout:8/412: stat da/f89 0 2024-04-20T21:32:07.143 INFO:tasks.workunit.client.0.smithi062.stdout:0/630: chown d7/d1c/d15/d17/d1b/d9d/db1 438 1 2024-04-20T21:32:07.145 INFO:tasks.workunit.client.1.smithi189.stdout:4/280: mkdir d2/dd/d11/d14/d69 0 2024-04-20T21:32:07.148 INFO:tasks.workunit.client.1.smithi189.stdout:1/284: unlink d8/df/d50/f5a 0 2024-04-20T21:32:07.149 INFO:tasks.workunit.client.0.smithi062.stdout:6/515: unlink d0/ca 0 2024-04-20T21:32:07.150 INFO:tasks.workunit.client.0.smithi062.stdout:3/618: rename d0/d1/d10/d3a/d5e/f78 to d0/d1c/d63/d42/dc8/dc9/fcf 0 2024-04-20T21:32:07.151 INFO:tasks.workunit.client.0.smithi062.stdout:5/634: mkdir d8/d10/d3e/d8a/dce 0 2024-04-20T21:32:07.151 INFO:tasks.workunit.client.1.smithi189.stdout:8/413: creat da/d30/f93 x:0 0 0 2024-04-20T21:32:07.154 INFO:tasks.workunit.client.1.smithi189.stdout:7/307: rename d5/cc to d5/d21/d4a/d52/c6a 0 2024-04-20T21:32:07.154 INFO:tasks.workunit.client.1.smithi189.stdout:8/414: write da/d30/f93 [18433,104037] 0 2024-04-20T21:32:07.158 INFO:tasks.workunit.client.0.smithi062.stdout:2/670: mkdir d7/da/d15/d2f/d93/dac/dca/dd8/ddc 0 2024-04-20T21:32:07.159 INFO:tasks.workunit.client.0.smithi062.stdout:5/635: chown d8/d1e/c48 414627090 1 2024-04-20T21:32:07.161 INFO:tasks.workunit.client.0.smithi062.stdout:4/662: dwrite da/dd/df/f41 [4194304,4194304] 0 2024-04-20T21:32:07.161 INFO:tasks.workunit.client.0.smithi062.stdout:8/635: symlink d7/d25/ld5 0 2024-04-20T21:32:07.161 INFO:tasks.workunit.client.0.smithi062.stdout:3/619: mknod d0/d1c/d52/d7b/cd0 0 2024-04-20T21:32:07.161 INFO:tasks.workunit.client.1.smithi189.stdout:6/312: write d8/d12/d30/d47/f4c [5093272,21354] 0 2024-04-20T21:32:07.162 INFO:tasks.workunit.client.0.smithi062.stdout:6/516: write d0/d9/d11/f48 [1389999,90974] 0 2024-04-20T21:32:07.163 INFO:tasks.workunit.client.1.smithi189.stdout:6/313: chown d8/da/c15 518577 1 2024-04-20T21:32:07.164 INFO:tasks.workunit.client.0.smithi062.stdout:2/671: mknod d7/db5/cdd 0 2024-04-20T21:32:07.165 INFO:tasks.workunit.client.1.smithi189.stdout:7/308: chown d5/d1d/d42/d45/f48 0 1 2024-04-20T21:32:07.169 INFO:tasks.workunit.client.0.smithi062.stdout:9/682: sync 2024-04-20T21:32:07.169 INFO:tasks.workunit.client.1.smithi189.stdout:0/417: dwrite d3/df/d4c/f7e [0,4194304] 0 2024-04-20T21:32:07.170 INFO:tasks.workunit.client.1.smithi189.stdout:8/415: symlink da/df/d20/d2c/l94 0 2024-04-20T21:32:07.170 INFO:tasks.workunit.client.0.smithi062.stdout:2/672: readlink d7/l8 0 2024-04-20T21:32:07.181 INFO:tasks.workunit.client.0.smithi062.stdout:4/663: rename da/dd/d3c to da/dd/d15/d50/d64/d57/d5b/d9d/dea 0 2024-04-20T21:32:07.183 INFO:tasks.workunit.client.0.smithi062.stdout:5/636: mkdir d8/d38/d7a/d4c/dcf 0 2024-04-20T21:32:07.189 INFO:tasks.workunit.client.1.smithi189.stdout:6/314: mknod d8/d10/c69 0 2024-04-20T21:32:07.190 INFO:tasks.workunit.client.1.smithi189.stdout:7/309: dwrite d5/d1d/f56 [0,4194304] 0 2024-04-20T21:32:07.190 INFO:tasks.workunit.client.1.smithi189.stdout:3/269: sync 2024-04-20T21:32:07.191 INFO:tasks.workunit.client.0.smithi062.stdout:5/637: dread - d8/d38/d7a/fc6 zero size 2024-04-20T21:32:07.192 INFO:tasks.workunit.client.0.smithi062.stdout:2/673: symlink d7/da/d15/d1b/d3d/lde 0 2024-04-20T21:32:07.194 INFO:tasks.workunit.client.1.smithi189.stdout:0/418: dread d3/df/d4c/f7e [0,4194304] 0 2024-04-20T21:32:07.195 INFO:tasks.workunit.client.0.smithi062.stdout:4/664: creat da/dd/d15/d50/feb x:0 0 0 2024-04-20T21:32:07.196 INFO:tasks.workunit.client.1.smithi189.stdout:4/281: write d2/f2c [953546,99093] 0 2024-04-20T21:32:07.197 INFO:tasks.workunit.client.0.smithi062.stdout:6/517: link d0/d9/d11/d5c/l76 d0/d2c/d67/d75/lab 0 2024-04-20T21:32:07.197 INFO:tasks.workunit.client.0.smithi062.stdout:5/638: chown d8/d10/f26 47931 1 2024-04-20T21:32:07.199 INFO:tasks.workunit.client.1.smithi189.stdout:6/315: mknod d8/d12/d4e/c6a 0 2024-04-20T21:32:07.200 INFO:tasks.workunit.client.1.smithi189.stdout:7/310: truncate d5/f47 352628 0 2024-04-20T21:32:07.201 INFO:tasks.workunit.client.1.smithi189.stdout:9/325: truncate d9/d12/d15/f31 1193055 0 2024-04-20T21:32:07.201 INFO:tasks.workunit.client.0.smithi062.stdout:3/620: dwrite d0/d1/d10/f4e [4194304,4194304] 0 2024-04-20T21:32:07.203 INFO:tasks.workunit.client.1.smithi189.stdout:9/326: fsync f6 0 2024-04-20T21:32:07.203 INFO:tasks.workunit.client.1.smithi189.stdout:6/316: write d8/d2b/d5c/f5e [590067,52155] 0 2024-04-20T21:32:07.203 INFO:tasks.workunit.client.1.smithi189.stdout:4/282: unlink d2/dd/d11/d14/f41 0 2024-04-20T21:32:07.204 INFO:tasks.workunit.client.1.smithi189.stdout:8/416: write da/de/f28 [259806,71695] 0 2024-04-20T21:32:07.205 INFO:tasks.workunit.client.0.smithi062.stdout:2/674: chown d7/da/d15/d2f/c8d 8 1 2024-04-20T21:32:07.209 INFO:tasks.workunit.client.0.smithi062.stdout:1/702: dwrite dc/f35 [0,4194304] 0 2024-04-20T21:32:07.209 INFO:tasks.workunit.client.0.smithi062.stdout:0/631: dread d7/d1c/d15/d17/d37/f84 [0,4194304] 0 2024-04-20T21:32:07.215 INFO:tasks.workunit.client.0.smithi062.stdout:5/639: mknod d8/d10/d3e/cd0 0 2024-04-20T21:32:07.216 INFO:tasks.workunit.client.1.smithi189.stdout:8/417: dread - da/df/d13/d31/f80 zero size 2024-04-20T21:32:07.216 INFO:tasks.workunit.client.1.smithi189.stdout:6/317: creat d8/da/f6b x:0 0 0 2024-04-20T21:32:07.217 INFO:tasks.workunit.client.0.smithi062.stdout:6/518: read d0/d1a/d44/f4c [3637838,65408] 0 2024-04-20T21:32:07.219 INFO:tasks.workunit.client.1.smithi189.stdout:6/318: fsync d8/da/dd/f28 0 2024-04-20T21:32:07.219 INFO:tasks.workunit.client.1.smithi189.stdout:1/285: dwrite d8/d1e/d28/d2b/f61 [0,4194304] 0 2024-04-20T21:32:07.222 INFO:tasks.workunit.client.1.smithi189.stdout:9/327: mkdir d9/dc/d69 0 2024-04-20T21:32:07.225 INFO:tasks.workunit.client.0.smithi062.stdout:2/675: readlink d7/da/d15/d2f/lad 0 2024-04-20T21:32:07.226 INFO:tasks.workunit.client.1.smithi189.stdout:4/283: mkdir d2/dd/d11/d14/d69/d6a 0 2024-04-20T21:32:07.228 INFO:tasks.workunit.client.1.smithi189.stdout:3/270: dread d2/fd [0,4194304] 0 2024-04-20T21:32:07.230 INFO:tasks.workunit.client.0.smithi062.stdout:8/636: dwrite d7/d9/d29/f4c [0,4194304] 0 2024-04-20T21:32:07.233 INFO:tasks.workunit.client.1.smithi189.stdout:0/419: truncate d3/df/d30/d45/d60/f81 1820345 0 2024-04-20T21:32:07.233 INFO:tasks.workunit.client.1.smithi189.stdout:4/284: read d2/dd/d11/f67 [54593,22621] 0 2024-04-20T21:32:07.234 INFO:tasks.workunit.client.1.smithi189.stdout:1/286: unlink d8/d21/d2c/l51 0 2024-04-20T21:32:07.236 INFO:tasks.workunit.client.0.smithi062.stdout:3/621: mknod d0/d1/cd1 0 2024-04-20T21:32:07.237 INFO:tasks.workunit.client.1.smithi189.stdout:0/420: chown d3/d26/f1d 6 1 2024-04-20T21:32:07.238 INFO:tasks.workunit.client.1.smithi189.stdout:1/287: rename d8 to d8/d1e/d28/d4f/d63 22 2024-04-20T21:32:07.240 INFO:tasks.workunit.client.0.smithi062.stdout:5/640: mkdir d8/d1e/d2c/dd1 0 2024-04-20T21:32:07.241 INFO:tasks.workunit.client.1.smithi189.stdout:6/319: symlink d8/da/dd/d2a/d3b/l6c 0 2024-04-20T21:32:07.242 INFO:tasks.workunit.client.1.smithi189.stdout:7/311: dwrite d5/f1a [0,4194304] 0 2024-04-20T21:32:07.244 INFO:tasks.workunit.client.0.smithi062.stdout:9/683: dread d16/d3d/d55/d6e/d7d/f91 [0,4194304] 0 2024-04-20T21:32:07.245 INFO:tasks.workunit.client.0.smithi062.stdout:7/668: sync 2024-04-20T21:32:07.245 INFO:tasks.workunit.client.1.smithi189.stdout:6/320: readlink d8/da/dd/d35/l65 0 2024-04-20T21:32:07.246 INFO:tasks.workunit.client.0.smithi062.stdout:0/632: unlink d7/d1c/d15/d17/d26/fc0 0 2024-04-20T21:32:07.247 INFO:tasks.workunit.client.0.smithi062.stdout:6/519: rmdir d0/d52/da0 39 2024-04-20T21:32:07.249 INFO:tasks.workunit.client.1.smithi189.stdout:4/285: mkdir d2/dd/d6b 0 2024-04-20T21:32:07.249 INFO:tasks.workunit.client.1.smithi189.stdout:6/321: chown d8/d10 278632788 1 2024-04-20T21:32:07.250 INFO:tasks.workunit.client.1.smithi189.stdout:3/271: mkdir d2/d3/d65 0 2024-04-20T21:32:07.252 INFO:tasks.workunit.client.1.smithi189.stdout:7/312: mknod d5/d46/c6b 0 2024-04-20T21:32:07.253 INFO:tasks.workunit.client.1.smithi189.stdout:0/421: dread d3/df/d28/d2b/f67 [0,4194304] 0 2024-04-20T21:32:07.254 INFO:tasks.workunit.client.0.smithi062.stdout:4/665: link da/dd/df/d27/d7a/l7c da/dd/df/d27/d3f/lec 0 2024-04-20T21:32:07.256 INFO:tasks.workunit.client.1.smithi189.stdout:8/418: creat da/de/d1e/d69/f95 x:0 0 0 2024-04-20T21:32:07.257 INFO:tasks.workunit.client.1.smithi189.stdout:1/288: creat d8/d1e/d43/d58/d60/f64 x:0 0 0 2024-04-20T21:32:07.258 INFO:tasks.workunit.client.1.smithi189.stdout:4/286: symlink d2/d20/l6c 0 2024-04-20T21:32:07.259 INFO:tasks.workunit.client.0.smithi062.stdout:8/637: fdatasync d7/d25/d3b/d58/d8b/f1c 0 2024-04-20T21:32:07.259 INFO:tasks.workunit.client.0.smithi062.stdout:4/666: dread - da/dd/df/d27/d3f/fc3 zero size 2024-04-20T21:32:07.261 INFO:tasks.workunit.client.0.smithi062.stdout:4/667: fdatasync da/dd/d15/d50/d64/d3d/da1/fb9 0 2024-04-20T21:32:07.263 INFO:tasks.workunit.client.0.smithi062.stdout:6/520: rename d0/d1a/f59 to d0/d1a/d44/fac 0 2024-04-20T21:32:07.265 INFO:tasks.workunit.client.1.smithi189.stdout:3/272: rename d2/d8/l55 to d2/d3/d2c/d30/l66 0 2024-04-20T21:32:07.265 INFO:tasks.workunit.client.0.smithi062.stdout:7/669: mknod d7/d3e/d33/d3d/d4d/d54/d4f/ce5 0 2024-04-20T21:32:07.268 INFO:tasks.workunit.client.1.smithi189.stdout:3/273: write d2/f5d [450821,103743] 0 2024-04-20T21:32:07.270 INFO:tasks.workunit.client.0.smithi062.stdout:2/676: creat d7/da/fdf x:0 0 0 2024-04-20T21:32:07.271 INFO:tasks.workunit.client.1.smithi189.stdout:4/287: unlink d2/f3e 0 2024-04-20T21:32:07.273 INFO:tasks.workunit.client.0.smithi062.stdout:5/641: dread d8/d38/d7a/d4c/d7d/f98 [0,4194304] 0 2024-04-20T21:32:07.275 INFO:tasks.workunit.client.0.smithi062.stdout:9/684: link d16/f1e d16/d2c/d42/fe2 0 2024-04-20T21:32:07.276 INFO:tasks.workunit.client.1.smithi189.stdout:0/422: symlink d3/d26/l8a 0 2024-04-20T21:32:07.276 INFO:tasks.workunit.client.0.smithi062.stdout:8/638: rmdir d7/d25/d5c 39 2024-04-20T21:32:07.277 INFO:tasks.workunit.client.1.smithi189.stdout:6/322: dwrite d8/d12/d30/d47/f52 [0,4194304] 0 2024-04-20T21:32:07.277 INFO:tasks.workunit.client.1.smithi189.stdout:1/289: mknod d8/c65 0 2024-04-20T21:32:07.278 INFO:tasks.workunit.client.0.smithi062.stdout:4/668: fsync da/dd/d15/d50/d64/d57/d5b/f68 0 2024-04-20T21:32:07.280 INFO:tasks.workunit.client.1.smithi189.stdout:6/323: readlink d8/da/l42 0 2024-04-20T21:32:07.280 INFO:tasks.workunit.client.1.smithi189.stdout:4/288: dread d2/f38 [0,4194304] 0 2024-04-20T21:32:07.283 INFO:tasks.workunit.client.0.smithi062.stdout:5/642: mknod d8/d38/d7a/d4c/dcf/cd2 0 2024-04-20T21:32:07.283 INFO:tasks.workunit.client.1.smithi189.stdout:6/324: dread - d8/da/dd/d2a/d3b/f68 zero size 2024-04-20T21:32:07.285 INFO:tasks.workunit.client.0.smithi062.stdout:4/669: mkdir da/dd/df/d27/d32/ded 0 2024-04-20T21:32:07.288 INFO:tasks.workunit.client.1.smithi189.stdout:3/274: fsync d2/f5a 0 2024-04-20T21:32:07.290 INFO:tasks.workunit.client.0.smithi062.stdout:6/521: rename d0/d4f/l8a to d0/d9/d11/lad 0 2024-04-20T21:32:07.294 INFO:tasks.workunit.client.0.smithi062.stdout:8/639: dread d7/d25/d3b/d58/d8b/f1c [0,4194304] 0 2024-04-20T21:32:07.295 INFO:tasks.workunit.client.0.smithi062.stdout:2/677: creat d7/fe0 x:0 0 0 2024-04-20T21:32:07.297 INFO:tasks.workunit.client.0.smithi062.stdout:1/703: truncate dc/f53 5464779 0 2024-04-20T21:32:07.298 INFO:tasks.workunit.client.1.smithi189.stdout:1/290: creat d8/d1e/f66 x:0 0 0 2024-04-20T21:32:07.299 INFO:tasks.workunit.client.1.smithi189.stdout:7/313: getdents d5/d21/d4a 0 2024-04-20T21:32:07.299 INFO:tasks.workunit.client.0.smithi062.stdout:4/670: chown da/dd/df/d27/l38 544793 1 2024-04-20T21:32:07.299 INFO:tasks.workunit.client.0.smithi062.stdout:7/670: getdents d7/d3e/d33/d3d/d4d/dce 0 2024-04-20T21:32:07.300 INFO:tasks.workunit.client.1.smithi189.stdout:7/314: readlink d5/d21/d35/l3d 0 2024-04-20T21:32:07.303 INFO:tasks.workunit.client.0.smithi062.stdout:9/685: creat d16/d2c/d42/da6/fe3 x:0 0 0 2024-04-20T21:32:07.303 INFO:tasks.workunit.client.0.smithi062.stdout:3/622: truncate d0/f2f 4029138 0 2024-04-20T21:32:07.305 INFO:tasks.workunit.client.0.smithi062.stdout:9/686: stat d16/d2c/d5e/la8 0 2024-04-20T21:32:07.309 INFO:tasks.workunit.client.1.smithi189.stdout:6/325: mkdir d8/da/dd/d35/d6d 0 2024-04-20T21:32:07.309 INFO:tasks.workunit.client.0.smithi062.stdout:6/522: creat d0/d9a/fae x:0 0 0 2024-04-20T21:32:07.309 INFO:tasks.workunit.client.0.smithi062.stdout:3/623: write d0/d1c/d52/d7b/fb6 [519278,19038] 0 2024-04-20T21:32:07.310 INFO:tasks.workunit.client.0.smithi062.stdout:4/671: mkdir da/dd/d15/daf/dee 0 2024-04-20T21:32:07.310 INFO:tasks.workunit.client.0.smithi062.stdout:2/678: creat d7/da/d15/d2f/d93/dac/fe1 x:0 0 0 2024-04-20T21:32:07.313 INFO:tasks.workunit.client.0.smithi062.stdout:5/643: dread d8/d1e/d2c/d6c/f9b [0,4194304] 0 2024-04-20T21:32:07.313 INFO:tasks.workunit.client.0.smithi062.stdout:1/704: mknod dc/d15/d75/d96/dd4/d3a/da2/de4/d84/cfc 0 2024-04-20T21:32:07.318 INFO:tasks.workunit.client.0.smithi062.stdout:1/705: write f1 [484027,127364] 0 2024-04-20T21:32:07.318 INFO:tasks.workunit.client.0.smithi062.stdout:9/687: rename d16/d3e/d77/la1 to d16/d2f/le4 0 2024-04-20T21:32:07.320 INFO:tasks.workunit.client.0.smithi062.stdout:0/633: dread d7/d1e/d6e/f75 [4194304,4194304] 0 2024-04-20T21:32:07.321 INFO:tasks.workunit.client.0.smithi062.stdout:5/644: chown d8/d1e/d2c/f79 248751 1 2024-04-20T21:32:07.322 INFO:tasks.workunit.client.0.smithi062.stdout:1/706: chown dc/d15/d20/fc3 0 1 2024-04-20T21:32:07.324 INFO:tasks.workunit.client.1.smithi189.stdout:4/289: dread d2/dd/d11/f68 [0,4194304] 0 2024-04-20T21:32:07.325 INFO:tasks.workunit.client.1.smithi189.stdout:7/315: creat d5/d21/d25/f6c x:0 0 0 2024-04-20T21:32:07.326 INFO:tasks.workunit.client.0.smithi062.stdout:2/679: rename d7/db5/cdd to d7/da/d28/db9/ce2 0 2024-04-20T21:32:07.326 INFO:tasks.workunit.client.0.smithi062.stdout:9/688: write d16/d2c/d42/da6/fe3 [277262,120264] 0 2024-04-20T21:32:07.329 INFO:tasks.workunit.client.1.smithi189.stdout:6/326: mkdir d8/d2b/d49/d54/d6e 0 2024-04-20T21:32:07.335 INFO:tasks.workunit.client.0.smithi062.stdout:9/689: write d16/d2c/d42/fde [428072,119312] 0 2024-04-20T21:32:07.335 INFO:tasks.workunit.client.1.smithi189.stdout:0/423: link d3/df/d28/f6a d3/df/d30/d45/d60/f8b 0 2024-04-20T21:32:07.338 INFO:tasks.workunit.client.1.smithi189.stdout:1/291: symlink d8/df/l67 0 2024-04-20T21:32:07.342 INFO:tasks.workunit.client.0.smithi062.stdout:8/640: dwrite d7/d25/d3b/d58/d8b/d24/f85 [0,4194304] 0 2024-04-20T21:32:07.343 INFO:tasks.workunit.client.0.smithi062.stdout:6/523: getdents d0/d9/d11/d22 0 2024-04-20T21:32:07.344 INFO:tasks.workunit.client.1.smithi189.stdout:8/419: truncate da/df/d13/d63/f73 3035868 0 2024-04-20T21:32:07.344 INFO:tasks.workunit.client.0.smithi062.stdout:2/680: mknod d7/da/d15/d1b/ce3 0 2024-04-20T21:32:07.345 INFO:tasks.workunit.client.1.smithi189.stdout:7/316: write d5/fe [942072,93809] 0 2024-04-20T21:32:07.346 INFO:tasks.workunit.client.0.smithi062.stdout:9/690: fdatasync d16/d2f/d5b/fa0 0 2024-04-20T21:32:07.346 INFO:tasks.workunit.client.0.smithi062.stdout:0/634: link d7/d1c/f18 d7/d1c/d15/d17/d1b/d9d/db1/fd4 0 2024-04-20T21:32:07.349 INFO:tasks.workunit.client.1.smithi189.stdout:6/327: unlink d8/da/c15 0 2024-04-20T21:32:07.350 INFO:tasks.workunit.client.0.smithi062.stdout:1/707: truncate dc/ff 1565080 0 2024-04-20T21:32:07.352 INFO:tasks.workunit.client.0.smithi062.stdout:5/645: write d8/d38/d7a/d4c/d7d/d9d/dac/d66/dae/d88/f8c [1684192,115909] 0 2024-04-20T21:32:07.356 INFO:tasks.workunit.client.1.smithi189.stdout:1/292: creat d8/d1e/d43/d58/d60/f68 x:0 0 0 2024-04-20T21:32:07.356 INFO:tasks.workunit.client.0.smithi062.stdout:7/671: dwrite d7/d3e/d33/d3d/f4a [4194304,4194304] 0 2024-04-20T21:32:07.357 INFO:tasks.workunit.client.1.smithi189.stdout:8/420: creat da/df/d20/d2c/f96 x:0 0 0 2024-04-20T21:32:07.359 INFO:tasks.workunit.client.1.smithi189.stdout:1/293: read - d8/d1e/d43/d58/d60/f68 zero size 2024-04-20T21:32:07.359 INFO:tasks.workunit.client.1.smithi189.stdout:3/275: getdents d2/d3/d2c/d31/d4c 0 2024-04-20T21:32:07.359 INFO:tasks.workunit.client.1.smithi189.stdout:7/317: creat d5/d21/d25/d34/f6d x:0 0 0 2024-04-20T21:32:07.360 INFO:tasks.workunit.client.1.smithi189.stdout:3/276: stat d2/d8/f4b 0 2024-04-20T21:32:07.362 INFO:tasks.workunit.client.1.smithi189.stdout:5/252: sync 2024-04-20T21:32:07.362 INFO:tasks.workunit.client.1.smithi189.stdout:2/274: sync 2024-04-20T21:32:07.362 INFO:tasks.workunit.client.1.smithi189.stdout:1/294: write d8/d1e/f66 [1022850,81018] 0 2024-04-20T21:32:07.363 INFO:tasks.workunit.client.1.smithi189.stdout:2/275: fsync d1/f7 0 2024-04-20T21:32:07.365 INFO:tasks.workunit.client.0.smithi062.stdout:0/635: readlink d7/d1c/l5e 0 2024-04-20T21:32:07.367 INFO:tasks.workunit.client.1.smithi189.stdout:6/328: creat d8/d12/d4e/f6f x:0 0 0 2024-04-20T21:32:07.367 INFO:tasks.workunit.client.0.smithi062.stdout:9/691: creat d16/d3e/fe5 x:0 0 0 2024-04-20T21:32:07.368 INFO:tasks.workunit.client.0.smithi062.stdout:8/641: dread d7/d25/d3b/d58/d8b/f1c [0,4194304] 0 2024-04-20T21:32:07.368 INFO:tasks.workunit.client.0.smithi062.stdout:7/672: symlink d7/d26/da8/le6 0 2024-04-20T21:32:07.368 INFO:tasks.workunit.client.1.smithi189.stdout:6/329: chown f2 1066 1 2024-04-20T21:32:07.369 INFO:tasks.workunit.client.0.smithi062.stdout:3/624: dwrite d0/d11/f56 [0,4194304] 0 2024-04-20T21:32:07.377 INFO:tasks.workunit.client.1.smithi189.stdout:4/290: creat d2/f6d x:0 0 0 2024-04-20T21:32:07.377 INFO:tasks.workunit.client.1.smithi189.stdout:8/421: creat da/de/d1e/d21/f97 x:0 0 0 2024-04-20T21:32:07.379 INFO:tasks.workunit.client.0.smithi062.stdout:0/636: symlink d7/d1c/ld5 0 2024-04-20T21:32:07.379 INFO:tasks.workunit.client.0.smithi062.stdout:4/672: dwrite da/dd/df/d27/d7a/fbb [0,4194304] 0 2024-04-20T21:32:07.381 INFO:tasks.workunit.client.1.smithi189.stdout:9/328: sync 2024-04-20T21:32:07.382 INFO:tasks.workunit.client.1.smithi189.stdout:7/318: write d5/f24 [4022328,111236] 0 2024-04-20T21:32:07.384 INFO:tasks.workunit.client.0.smithi062.stdout:5/646: write d8/d10/d3e/f52 [2910022,10983] 0 2024-04-20T21:32:07.385 INFO:tasks.workunit.client.0.smithi062.stdout:9/692: symlink d16/d3d/d55/d6e/d7d/le6 0 2024-04-20T21:32:07.390 INFO:tasks.workunit.client.0.smithi062.stdout:8/642: readlink d7/d25/d3b/d58/d8b/d24/d46/l5b 0 2024-04-20T21:32:07.391 INFO:tasks.workunit.client.1.smithi189.stdout:5/253: mknod d7/d9/d30/c47 0 2024-04-20T21:32:07.391 INFO:tasks.workunit.client.1.smithi189.stdout:2/276: symlink d1/d11/d26/l4e 0 2024-04-20T21:32:07.392 INFO:tasks.workunit.client.0.smithi062.stdout:0/637: unlink d7/d1e/db6/fa3 0 2024-04-20T21:32:07.394 INFO:tasks.workunit.client.0.smithi062.stdout:4/673: truncate da/dd/fcc 558165 0 2024-04-20T21:32:07.394 INFO:tasks.workunit.client.1.smithi189.stdout:0/424: rename d3/df/d30/d45/d60/d79 to d3/d8c 0 2024-04-20T21:32:07.394 INFO:tasks.workunit.client.1.smithi189.stdout:5/254: write d7/d35/f44 [2152537,49708] 0 2024-04-20T21:32:07.396 INFO:tasks.workunit.client.0.smithi062.stdout:3/625: chown d0/d1/d10/f79 3254 1 2024-04-20T21:32:07.397 INFO:tasks.workunit.client.1.smithi189.stdout:5/255: truncate d7/d9/f42 684944 0 2024-04-20T21:32:07.400 INFO:tasks.workunit.client.0.smithi062.stdout:2/681: dread d7/da/d28/f6d [0,4194304] 0 2024-04-20T21:32:07.402 INFO:tasks.workunit.client.1.smithi189.stdout:7/319: readlink d5/l10 0 2024-04-20T21:32:07.403 INFO:tasks.workunit.client.0.smithi062.stdout:3/626: readlink d0/d1/d10/d3a/d5e/d65/lb5 0 2024-04-20T21:32:07.406 INFO:tasks.workunit.client.0.smithi062.stdout:9/693: symlink d16/d2f/d5b/da4/le7 0 2024-04-20T21:32:07.406 INFO:tasks.workunit.client.0.smithi062.stdout:0/638: readlink d7/d1c/d15/d17/d40/d3c/l89 0 2024-04-20T21:32:07.409 INFO:tasks.workunit.client.1.smithi189.stdout:0/425: dread d3/df/d28/d2b/f49 [0,4194304] 0 2024-04-20T21:32:07.410 INFO:tasks.workunit.client.0.smithi062.stdout:8/643: unlink d7/d9/d29/fa3 0 2024-04-20T21:32:07.410 INFO:tasks.workunit.client.1.smithi189.stdout:2/277: mkdir d1/d11/d2d/d4f 0 2024-04-20T21:32:07.412 INFO:tasks.workunit.client.0.smithi062.stdout:6/524: dwrite d0/d2c/f8b [0,4194304] 0 2024-04-20T21:32:07.412 INFO:tasks.workunit.client.0.smithi062.stdout:4/674: dread - da/dd/d15/d4b/f90 zero size 2024-04-20T21:32:07.415 INFO:tasks.workunit.client.1.smithi189.stdout:0/426: write d3/df/d28/f61 [561629,76705] 0 2024-04-20T21:32:07.416 INFO:tasks.workunit.client.0.smithi062.stdout:9/694: write d16/f19 [2942106,130145] 0 2024-04-20T21:32:07.417 INFO:tasks.workunit.client.1.smithi189.stdout:0/427: stat d3/df 0 2024-04-20T21:32:07.419 INFO:tasks.workunit.client.0.smithi062.stdout:7/673: dwrite d7/d26/f9a [0,4194304] 0 2024-04-20T21:32:07.419 INFO:tasks.workunit.client.0.smithi062.stdout:3/627: creat d0/d1b/d74/dc7/fd2 x:0 0 0 2024-04-20T21:32:07.420 INFO:tasks.workunit.client.0.smithi062.stdout:8/644: creat d7/d25/d70/fd6 x:0 0 0 2024-04-20T21:32:07.420 INFO:tasks.workunit.client.1.smithi189.stdout:8/422: link da/df/f23 da/df/d3c/d50/d6c/d85/f98 0 2024-04-20T21:32:07.422 INFO:tasks.workunit.client.0.smithi062.stdout:4/675: mknod da/dd/df/d27/d3f/cef 0 2024-04-20T21:32:07.422 INFO:tasks.workunit.client.1.smithi189.stdout:5/256: dwrite d7/d9/d30/f1c [0,4194304] 0 2024-04-20T21:32:07.427 INFO:tasks.workunit.client.0.smithi062.stdout:2/682: dread d7/da/d15/d1b/d3d/f94 [0,4194304] 0 2024-04-20T21:32:07.428 INFO:tasks.workunit.client.0.smithi062.stdout:8/645: read d7/fa5 [3731802,65178] 0 2024-04-20T21:32:07.430 INFO:tasks.workunit.client.0.smithi062.stdout:7/674: chown d7/d3e/d33/d3d/d5e/d7e/f93 4 1 2024-04-20T21:32:07.431 INFO:tasks.workunit.client.0.smithi062.stdout:6/525: rename d0/d2c/d67/d75/l87 to d0/d2c/d67/d75/laf 0 2024-04-20T21:32:07.433 INFO:tasks.workunit.client.0.smithi062.stdout:8/646: chown d7/d25/c55 3 1 2024-04-20T21:32:07.434 INFO:tasks.workunit.client.1.smithi189.stdout:9/329: link d9/d12/d3f/f68 d9/dc/d17/d2b/f6a 0 2024-04-20T21:32:07.436 INFO:tasks.workunit.client.1.smithi189.stdout:6/330: truncate d8/d10/f36 163015 0 2024-04-20T21:32:07.436 INFO:tasks.workunit.client.0.smithi062.stdout:9/695: symlink d16/d2c/d5e/d75/d76/dd9/le8 0 2024-04-20T21:32:07.437 INFO:tasks.workunit.client.1.smithi189.stdout:1/295: rename d8/df/c32 to d8/d21/d2c/d36/c69 0 2024-04-20T21:32:07.443 INFO:tasks.workunit.client.1.smithi189.stdout:4/291: write d2/dd/d11/f67 [1230345,42167] 0 2024-04-20T21:32:07.443 INFO:tasks.workunit.client.0.smithi062.stdout:2/683: symlink d7/da/d15/d2f/d38/d51/dbc/le4 0 2024-04-20T21:32:07.446 INFO:tasks.workunit.client.0.smithi062.stdout:6/526: write d0/d1a/f3a [2807753,73186] 0 2024-04-20T21:32:07.448 INFO:tasks.workunit.client.1.smithi189.stdout:9/330: rename d9/d12/d15/d1a/c63 to d9/dc/d17/c6b 0 2024-04-20T21:32:07.450 INFO:tasks.workunit.client.1.smithi189.stdout:7/320: write d5/d1d/f26 [1682436,78552] 0 2024-04-20T21:32:07.450 INFO:tasks.workunit.client.1.smithi189.stdout:6/331: mknod d8/da/dd/d2a/c70 0 2024-04-20T21:32:07.452 INFO:tasks.workunit.client.0.smithi062.stdout:2/684: readlink d7/da/d15/d1b/d24/lcb 0 2024-04-20T21:32:07.456 INFO:tasks.workunit.client.1.smithi189.stdout:9/331: write d9/d12/d15/d1a/f5c [2995574,97972] 0 2024-04-20T21:32:07.456 INFO:tasks.workunit.client.0.smithi062.stdout:5/647: truncate d8/d1e/f5d 4063123 0 2024-04-20T21:32:07.458 INFO:tasks.workunit.client.1.smithi189.stdout:8/423: creat da/df/d20/f99 x:0 0 0 2024-04-20T21:32:07.460 INFO:tasks.workunit.client.1.smithi189.stdout:1/296: rename d8/d1e/d28/d2b/l37 to d8/df/d50/l6a 0 2024-04-20T21:32:07.461 INFO:tasks.workunit.client.1.smithi189.stdout:2/278: dread d1/d17/f34 [0,4194304] 0 2024-04-20T21:32:07.463 INFO:tasks.workunit.client.0.smithi062.stdout:5/648: write d8/d38/d7a/d4c/d7d/d9d/dac/f39 [3732225,32331] 0 2024-04-20T21:32:07.463 INFO:tasks.workunit.client.0.smithi062.stdout:4/676: dwrite da/f7b [0,4194304] 0 2024-04-20T21:32:07.464 INFO:tasks.workunit.client.1.smithi189.stdout:3/277: dwrite d2/f5a [0,4194304] 0 2024-04-20T21:32:07.464 INFO:tasks.workunit.client.1.smithi189.stdout:6/332: creat d8/da/dd/f71 x:0 0 0 2024-04-20T21:32:07.464 INFO:tasks.workunit.client.0.smithi062.stdout:5/649: readlink d8/d1e/l78 0 2024-04-20T21:32:07.467 INFO:tasks.workunit.client.0.smithi062.stdout:8/647: chown d7/d25/d5c/c9f 20236 1 2024-04-20T21:32:07.467 INFO:tasks.workunit.client.0.smithi062.stdout:4/677: dread - da/dd/df/d27/d32/f49 zero size 2024-04-20T21:32:07.467 INFO:tasks.workunit.client.0.smithi062.stdout:2/685: mkdir d7/da/d15/d1b/d3d/de5 0 2024-04-20T21:32:07.470 INFO:tasks.workunit.client.1.smithi189.stdout:9/332: read d9/f34 [3013465,26311] 0 2024-04-20T21:32:07.472 INFO:tasks.workunit.client.0.smithi062.stdout:0/639: write d7/d1c/d15/d17/d37/f59 [2945782,114354] 0 2024-04-20T21:32:07.472 INFO:tasks.workunit.client.1.smithi189.stdout:4/292: rename d2/dd/d11/f68 to d2/d22/f6e 0 2024-04-20T21:32:07.474 INFO:tasks.workunit.client.0.smithi062.stdout:3/628: rename d0/d1c/d52 to d0/d1/d3c/d43/d72/dd3 0 2024-04-20T21:32:07.475 INFO:tasks.workunit.client.1.smithi189.stdout:0/428: dread d3/df/d30/d37/f4b [0,4194304] 0 2024-04-20T21:32:07.476 INFO:tasks.workunit.client.1.smithi189.stdout:8/424: mkdir da/df/d20/d2c/d9a 0 2024-04-20T21:32:07.477 INFO:tasks.workunit.client.0.smithi062.stdout:9/696: creat d16/d2c/d42/da6/da9/fe9 x:0 0 0 2024-04-20T21:32:07.478 INFO:tasks.workunit.client.1.smithi189.stdout:4/293: write d2/dd/d11/f31 [273577,103211] 0 2024-04-20T21:32:07.479 INFO:tasks.workunit.client.1.smithi189.stdout:3/278: readlink d2/l18 0 2024-04-20T21:32:07.481 INFO:tasks.workunit.client.0.smithi062.stdout:8/648: stat d7/f22 0 2024-04-20T21:32:07.482 INFO:tasks.workunit.client.0.smithi062.stdout:4/678: mknod da/dd/d15/d4b/ddb/cf0 0 2024-04-20T21:32:07.483 INFO:tasks.workunit.client.1.smithi189.stdout:3/279: chown d2/d3/f48 759913347 1 2024-04-20T21:32:07.485 INFO:tasks.workunit.client.1.smithi189.stdout:4/294: fdatasync d2/dd/d11/d14/f50 0 2024-04-20T21:32:07.486 INFO:tasks.workunit.client.0.smithi062.stdout:6/527: creat d0/d52/fb0 x:0 0 0 2024-04-20T21:32:07.486 INFO:tasks.workunit.client.0.smithi062.stdout:4/679: chown da/dd/df/d27/d3f/ccf 57851 1 2024-04-20T21:32:07.486 INFO:tasks.workunit.client.1.smithi189.stdout:6/333: creat d8/d2b/d49/f72 x:0 0 0 2024-04-20T21:32:07.487 INFO:tasks.workunit.client.0.smithi062.stdout:3/629: dread - d0/d11/fa2 zero size 2024-04-20T21:32:07.488 INFO:tasks.workunit.client.1.smithi189.stdout:1/297: dwrite d8/d21/d2c/d36/f52 [0,4194304] 0 2024-04-20T21:32:07.489 INFO:tasks.workunit.client.0.smithi062.stdout:7/675: getdents d7/d3e/d33/d3d/d5e 0 2024-04-20T21:32:07.489 INFO:tasks.workunit.client.0.smithi062.stdout:4/680: chown da/f2f 108 1 2024-04-20T21:32:07.490 INFO:tasks.workunit.client.1.smithi189.stdout:6/334: write d8/da/dd/f28 [760676,39790] 0 2024-04-20T21:32:07.493 INFO:tasks.workunit.client.1.smithi189.stdout:7/321: link d5/d46/l55 d5/d21/l6e 0 2024-04-20T21:32:07.494 INFO:tasks.workunit.client.1.smithi189.stdout:7/322: stat d5/d21/d35/l4f 0 2024-04-20T21:32:07.494 INFO:tasks.workunit.client.1.smithi189.stdout:8/425: rmdir da/de/d1e 39 2024-04-20T21:32:07.497 INFO:tasks.workunit.client.1.smithi189.stdout:7/323: readlink d5/d21/d35/l4f 0 2024-04-20T21:32:07.497 INFO:tasks.workunit.client.1.smithi189.stdout:2/279: link d1/d11/d2d/f41 d1/d11/d2d/f50 0 2024-04-20T21:32:07.497 INFO:tasks.workunit.client.0.smithi062.stdout:3/630: readlink d0/d1/d3c/d43/d72/dd3/l61 0 2024-04-20T21:32:07.499 INFO:tasks.workunit.client.1.smithi189.stdout:0/429: write d3/df/d28/d2b/f49 [2347516,125] 0 2024-04-20T21:32:07.500 INFO:tasks.workunit.client.0.smithi062.stdout:5/650: dwrite d8/d1e/fa9 [0,4194304] 0 2024-04-20T21:32:07.502 INFO:tasks.workunit.client.0.smithi062.stdout:7/676: write d7/d3e/d33/f52 [288388,35711] 0 2024-04-20T21:32:07.504 INFO:tasks.workunit.client.1.smithi189.stdout:9/333: getdents d9/dc/d17/d2b/d55 0 2024-04-20T21:32:07.504 INFO:tasks.workunit.client.0.smithi062.stdout:4/681: mkdir da/dd/df/d27/da5/df1 0 2024-04-20T21:32:07.506 INFO:tasks.workunit.client.1.smithi189.stdout:5/257: truncate d7/d9/d30/f1c 1899566 0 2024-04-20T21:32:07.506 INFO:tasks.workunit.client.1.smithi189.stdout:9/334: chown d9/d12/d15/d1a 103720151 1 2024-04-20T21:32:07.508 INFO:tasks.workunit.client.0.smithi062.stdout:9/697: getdents d16/d2c/d42 0 2024-04-20T21:32:07.508 INFO:tasks.workunit.client.1.smithi189.stdout:9/335: chown d9/d12/d15/d1a/f27 41 1 2024-04-20T21:32:07.509 INFO:tasks.workunit.client.0.smithi062.stdout:4/682: read - da/dd/d15/d50/feb zero size 2024-04-20T21:32:07.509 INFO:tasks.workunit.client.0.smithi062.stdout:5/651: fdatasync d8/d1e/d2c/d6c/f9b 0 2024-04-20T21:32:07.510 INFO:tasks.workunit.client.0.smithi062.stdout:8/649: rename d7/f23 to d7/d9/fd7 0 2024-04-20T21:32:07.512 INFO:tasks.workunit.client.0.smithi062.stdout:9/698: write d16/dbe/fca [321574,50783] 0 2024-04-20T21:32:07.513 INFO:tasks.workunit.client.0.smithi062.stdout:7/677: mknod d7/d3e/d33/d3d/d5e/d7e/dcc/ce7 0 2024-04-20T21:32:07.513 INFO:tasks.workunit.client.0.smithi062.stdout:8/650: readlink d7/d25/d70/l73 0 2024-04-20T21:32:07.515 INFO:tasks.workunit.client.0.smithi062.stdout:3/631: rename d0/d1/d10/f7a to d0/d1/d3c/d43/fd4 0 2024-04-20T21:32:07.517 INFO:tasks.workunit.client.1.smithi189.stdout:0/430: creat d3/d8c/f8d x:0 0 0 2024-04-20T21:32:07.518 INFO:tasks.workunit.client.0.smithi062.stdout:9/699: dread - d16/d3e/fa2 zero size 2024-04-20T21:32:07.522 INFO:tasks.workunit.client.0.smithi062.stdout:5/652: rmdir d8/d38/d7a/d4c/d7d 39 2024-04-20T21:32:07.523 INFO:tasks.workunit.client.1.smithi189.stdout:5/258: mkdir d7/d35/d48 0 2024-04-20T21:32:07.524 INFO:tasks.workunit.client.1.smithi189.stdout:4/295: getdents d2/dd/d11/d14/d69/d6a 0 2024-04-20T21:32:07.525 INFO:tasks.workunit.client.1.smithi189.stdout:2/280: mkdir d1/d11/d2d/d4f/d51 0 2024-04-20T21:32:07.526 INFO:tasks.workunit.client.0.smithi062.stdout:7/678: rename d7/d3e/d33/d3d/d5e/ddc to d7/d26/dac/ddb/de8 0 2024-04-20T21:32:07.530 INFO:tasks.workunit.client.1.smithi189.stdout:7/324: rename d5/f11 to d5/d21/d25/d34/f6f 0 2024-04-20T21:32:07.532 INFO:tasks.workunit.client.1.smithi189.stdout:4/296: creat d2/d3a/f6f x:0 0 0 2024-04-20T21:32:07.533 INFO:tasks.workunit.client.0.smithi062.stdout:3/632: mkdir d0/d1/d3c/d43/d72/dd3/dc5/dd5 0 2024-04-20T21:32:07.533 INFO:tasks.workunit.client.1.smithi189.stdout:2/281: mkdir d1/d11/d52 0 2024-04-20T21:32:07.537 INFO:tasks.workunit.client.0.smithi062.stdout:3/633: stat d0/d11/f56 0 2024-04-20T21:32:07.537 INFO:tasks.workunit.client.0.smithi062.stdout:5/653: dread d8/d38/d7a/d4c/d7d/d9d/fb8 [0,4194304] 0 2024-04-20T21:32:07.538 INFO:tasks.workunit.client.1.smithi189.stdout:8/426: rename da/df/d13/d31/c56 to da/df/d3a/c9b 0 2024-04-20T21:32:07.539 INFO:tasks.workunit.client.1.smithi189.stdout:2/282: unlink d1/f6 0 2024-04-20T21:32:07.540 INFO:tasks.workunit.client.1.smithi189.stdout:8/427: fsync da/df/d20/f99 0 2024-04-20T21:32:07.542 INFO:tasks.workunit.client.1.smithi189.stdout:2/283: truncate d1/d17/f35 85550 0 2024-04-20T21:32:07.543 INFO:tasks.workunit.client.1.smithi189.stdout:7/325: symlink d5/d21/d4a/l70 0 2024-04-20T21:32:07.547 INFO:tasks.workunit.client.0.smithi062.stdout:4/683: truncate da/f83 3502401 0 2024-04-20T21:32:07.549 INFO:tasks.workunit.client.0.smithi062.stdout:6/528: dwrite d0/d4f/f57 [0,4194304] 0 2024-04-20T21:32:07.551 INFO:tasks.workunit.client.1.smithi189.stdout:2/284: creat d1/d11/d26/f53 x:0 0 0 2024-04-20T21:32:07.552 INFO:tasks.workunit.client.0.smithi062.stdout:9/700: dread d16/d2c/d5e/d75/d76/fb6 [0,4194304] 0 2024-04-20T21:32:07.553 INFO:tasks.workunit.client.0.smithi062.stdout:0/640: dread d7/d1c/d15/d17/f72 [0,4194304] 0 2024-04-20T21:32:07.555 INFO:tasks.workunit.client.1.smithi189.stdout:7/326: mkdir d5/d21/d35/d53/d5b/d71 0 2024-04-20T21:32:07.556 INFO:tasks.workunit.client.1.smithi189.stdout:6/335: truncate d8/da/dd/f28 99110 0 2024-04-20T21:32:07.558 INFO:tasks.workunit.client.1.smithi189.stdout:7/327: write d5/d1d/f44 [7820146,113223] 0 2024-04-20T21:32:07.560 INFO:tasks.workunit.client.0.smithi062.stdout:9/701: read d16/d2f/d5b/fa0 [2286836,47967] 0 2024-04-20T21:32:07.561 INFO:tasks.workunit.client.1.smithi189.stdout:2/285: mknod d1/d11/d2d/c54 0 2024-04-20T21:32:07.562 INFO:tasks.workunit.client.1.smithi189.stdout:3/280: dwrite d2/d3/f48 [0,4194304] 0 2024-04-20T21:32:07.562 INFO:tasks.workunit.client.1.smithi189.stdout:6/336: chown d8/da/dd/d2a/d3b/c55 58 1 2024-04-20T21:32:07.563 INFO:tasks.workunit.client.0.smithi062.stdout:4/684: symlink da/dd/df/d27/d7a/d91/lf2 0 2024-04-20T21:32:07.564 INFO:tasks.workunit.client.1.smithi189.stdout:0/431: write d3/df/d28/f6a [973646,13792] 0 2024-04-20T21:32:07.567 INFO:tasks.workunit.client.1.smithi189.stdout:0/432: chown d3/df/d5d/d64/c85 1013447574 1 2024-04-20T21:32:07.569 INFO:tasks.workunit.client.0.smithi062.stdout:6/529: dread - d0/d52/da0/fa8 zero size 2024-04-20T21:32:07.570 INFO:tasks.workunit.client.1.smithi189.stdout:7/328: rename d5/d21/d35/d53/d5b/f2f to d5/d21/d35/d53/d5b/f72 0 2024-04-20T21:32:07.570 INFO:tasks.workunit.client.1.smithi189.stdout:6/337: mknod d8/c73 0 2024-04-20T21:32:07.577 INFO:tasks.workunit.client.0.smithi062.stdout:6/530: mkdir d0/d1a/d7d/db1 0 2024-04-20T21:32:07.578 INFO:tasks.workunit.client.1.smithi189.stdout:2/286: dread d1/f5 [4194304,4194304] 0 2024-04-20T21:32:07.579 INFO:tasks.workunit.client.1.smithi189.stdout:2/287: write d1/f9 [1691190,101820] 0 2024-04-20T21:32:07.581 INFO:tasks.workunit.client.0.smithi062.stdout:6/531: rmdir d0/d2c/d8c 39 2024-04-20T21:32:07.581 INFO:tasks.workunit.client.1.smithi189.stdout:2/288: chown d1/d17/f34 115096730 1 2024-04-20T21:32:07.586 INFO:tasks.workunit.client.1.smithi189.stdout:5/259: dread d7/d14/d1b/d1f/d2a/f39 [0,4194304] 0 2024-04-20T21:32:07.588 INFO:tasks.workunit.client.0.smithi062.stdout:5/654: write d8/d10/f17 [6074013,114347] 0 2024-04-20T21:32:07.589 INFO:tasks.workunit.client.0.smithi062.stdout:0/641: dread d7/d20/d62/d6f/f73 [0,4194304] 0 2024-04-20T21:32:07.593 INFO:tasks.workunit.client.0.smithi062.stdout:4/685: dwrite da/dd/f45 [0,4194304] 0 2024-04-20T21:32:07.595 INFO:tasks.workunit.client.0.smithi062.stdout:6/532: truncate d0/d9/f16 3729970 0 2024-04-20T21:32:07.597 INFO:tasks.workunit.client.0.smithi062.stdout:0/642: creat d7/d1c/d15/d17/d37/d95/fd6 x:0 0 0 2024-04-20T21:32:07.598 INFO:tasks.workunit.client.0.smithi062.stdout:4/686: fdatasync f3 0 2024-04-20T21:32:07.599 INFO:tasks.workunit.client.0.smithi062.stdout:5/655: mknod d8/d38/d7a/d4c/d7d/d9d/dac/d2f/d8e/cd3 0 2024-04-20T21:32:07.603 INFO:tasks.workunit.client.0.smithi062.stdout:0/643: mknod d7/d1c/d15/d17/d40/d3c/cd7 0 2024-04-20T21:32:07.603 INFO:tasks.workunit.client.0.smithi062.stdout:9/702: dread d16/d2c/d42/f70 [0,4194304] 0 2024-04-20T21:32:07.606 INFO:tasks.workunit.client.0.smithi062.stdout:5/656: symlink d8/d38/d7a/d4c/d7d/d9d/dac/d2f/d8e/ld4 0 2024-04-20T21:32:07.608 INFO:tasks.workunit.client.0.smithi062.stdout:9/703: mkdir d16/d2c/d62/dea 0 2024-04-20T21:32:07.608 INFO:tasks.workunit.client.0.smithi062.stdout:4/687: read - da/dd/df/d27/fc2 zero size 2024-04-20T21:32:07.610 INFO:tasks.workunit.client.0.smithi062.stdout:0/644: creat d7/d1c/d15/d17/d40/d3c/fd8 x:0 0 0 2024-04-20T21:32:07.613 INFO:tasks.workunit.client.0.smithi062.stdout:5/657: fsync d8/d38/d7a/f36 0 2024-04-20T21:32:07.613 INFO:tasks.workunit.client.0.smithi062.stdout:8/651: dread d7/d25/d5c/f98 [0,4194304] 0 2024-04-20T21:32:07.615 INFO:tasks.workunit.client.0.smithi062.stdout:4/688: creat da/dd/df/d27/da5/ff3 x:0 0 0 2024-04-20T21:32:07.615 INFO:tasks.workunit.client.0.smithi062.stdout:6/533: truncate d0/d9/d11/f1f 1133703 0 2024-04-20T21:32:07.618 INFO:tasks.workunit.client.0.smithi062.stdout:4/689: chown da/cb0 786318727 1 2024-04-20T21:32:07.619 INFO:tasks.workunit.client.0.smithi062.stdout:8/652: rmdir d7/d25/d3b/d58/dac 39 2024-04-20T21:32:07.621 INFO:tasks.workunit.client.0.smithi062.stdout:6/534: creat d0/d9/d11/d7b/fb2 x:0 0 0 2024-04-20T21:32:07.624 INFO:tasks.workunit.client.0.smithi062.stdout:0/645: link d7/d1c/d15/d17/d40/d3c/dc5/f99 d7/d1c/d15/d17/d1b/fd9 0 2024-04-20T21:32:07.626 INFO:tasks.workunit.client.0.smithi062.stdout:6/535: mkdir d0/d1a/d44/db3 0 2024-04-20T21:32:07.628 INFO:tasks.workunit.client.0.smithi062.stdout:1/708: sync 2024-04-20T21:32:07.630 INFO:tasks.workunit.client.0.smithi062.stdout:4/690: rename da/dd/fcc to da/dd/df/ff4 0 2024-04-20T21:32:07.632 INFO:tasks.workunit.client.0.smithi062.stdout:1/709: symlink dc/d15/d20/d28/d83/lfd 0 2024-04-20T21:32:07.634 INFO:tasks.workunit.client.0.smithi062.stdout:6/536: mknod d0/d1a/d44/db3/cb4 0 2024-04-20T21:32:07.635 INFO:tasks.workunit.client.0.smithi062.stdout:1/710: read f1 [944622,98913] 0 2024-04-20T21:32:07.643 INFO:tasks.workunit.client.1.smithi189.stdout:3/281: rename d2/d3/d2c/d30/f61 to d2/d3/d25/f67 0 2024-04-20T21:32:07.644 INFO:tasks.workunit.client.0.smithi062.stdout:5/658: dread d8/d38/d7a/d4c/d7d/d9d/dac/d66/dae/d88/f8c [0,4194304] 0 2024-04-20T21:32:07.647 INFO:tasks.workunit.client.0.smithi062.stdout:1/711: rename dc/d70/d7f/lbb to dc/d15/d75/d96/dd4/d3a/da2/lfe 0 2024-04-20T21:32:07.648 INFO:tasks.workunit.client.0.smithi062.stdout:8/653: dread d7/d9/fd7 [0,4194304] 0 2024-04-20T21:32:07.650 INFO:tasks.workunit.client.0.smithi062.stdout:9/704: dread f4 [0,4194304] 0 2024-04-20T21:32:07.650 INFO:tasks.workunit.client.0.smithi062.stdout:6/537: creat d0/fb5 x:0 0 0 2024-04-20T21:32:07.652 INFO:tasks.workunit.client.0.smithi062.stdout:5/659: symlink d8/d10/d3e/d8a/d9e/ld5 0 2024-04-20T21:32:07.653 INFO:tasks.workunit.client.0.smithi062.stdout:8/654: fdatasync d7/d9/d29/f4b 0 2024-04-20T21:32:07.653 INFO:tasks.workunit.client.0.smithi062.stdout:4/691: getdents da/dd/d15/d50/d64/d57/d5b/d9d/dea 0 2024-04-20T21:32:07.657 INFO:tasks.workunit.client.1.smithi189.stdout:2/289: rename d1/d11/d2d/d4f/d51 to d1/d17/d2f/d55 0 2024-04-20T21:32:07.658 INFO:tasks.workunit.client.1.smithi189.stdout:2/290: write d1/f9 [7265426,32393] 0 2024-04-20T21:32:07.661 INFO:tasks.workunit.client.1.smithi189.stdout:4/297: truncate d2/d20/f33 800737 0 2024-04-20T21:32:07.661 INFO:tasks.workunit.client.1.smithi189.stdout:2/291: read d1/d11/f15 [1913320,26452] 0 2024-04-20T21:32:07.663 INFO:tasks.workunit.client.1.smithi189.stdout:3/282: creat d2/d3/d2c/d30/f68 x:0 0 0 2024-04-20T21:32:07.665 INFO:tasks.workunit.client.1.smithi189.stdout:3/283: write d2/d8/f34 [505778,88241] 0 2024-04-20T21:32:07.665 INFO:tasks.workunit.client.0.smithi062.stdout:1/712: creat dc/d15/d20/d28/fff x:0 0 0 2024-04-20T21:32:07.667 INFO:tasks.workunit.client.1.smithi189.stdout:8/428: write da/df/d13/d63/f73 [1578309,3656] 0 2024-04-20T21:32:07.667 INFO:tasks.workunit.client.0.smithi062.stdout:9/705: unlink d16/d2c/d42/da6/fce 0 2024-04-20T21:32:07.668 INFO:tasks.workunit.client.0.smithi062.stdout:5/660: mkdir d8/d38/d7a/d4c/d7d/d9d/dac/d2f/d8e/dd6 0 2024-04-20T21:32:07.669 INFO:tasks.workunit.client.0.smithi062.stdout:4/692: mkdir da/dd/d15/d6d/df5 0 2024-04-20T21:32:07.669 INFO:tasks.workunit.client.1.smithi189.stdout:3/284: chown d2/d3/d36/l3d 1 1 2024-04-20T21:32:07.670 INFO:tasks.workunit.client.1.smithi189.stdout:0/433: creat d3/df/d30/d45/d60/f8e x:0 0 0 2024-04-20T21:32:07.672 INFO:tasks.workunit.client.1.smithi189.stdout:0/434: write d3/df/d5d/f7c [126506,114181] 0 2024-04-20T21:32:07.675 INFO:tasks.workunit.client.1.smithi189.stdout:2/292: symlink d1/d11/l56 0 2024-04-20T21:32:07.677 INFO:tasks.workunit.client.1.smithi189.stdout:2/293: chown d1/d17/d2f/f49 21725 1 2024-04-20T21:32:07.678 INFO:tasks.workunit.client.0.smithi062.stdout:9/706: chown d16/d2c/c2d 14125378 1 2024-04-20T21:32:07.678 INFO:tasks.workunit.client.0.smithi062.stdout:1/713: mkdir dc/d15/d75/d94/d100 0 2024-04-20T21:32:07.679 INFO:tasks.workunit.client.1.smithi189.stdout:4/298: symlink d2/d64/l70 0 2024-04-20T21:32:07.680 INFO:tasks.workunit.client.0.smithi062.stdout:8/655: rename d7/d25/d3b/d58/d8b/d24/d46/c91 to d7/d9/cd8 0 2024-04-20T21:32:07.681 INFO:tasks.workunit.client.0.smithi062.stdout:4/693: dread da/fb [0,4194304] 0 2024-04-20T21:32:07.681 INFO:tasks.workunit.client.1.smithi189.stdout:4/299: write d2/d22/f45 [1023317,32376] 0 2024-04-20T21:32:07.683 INFO:tasks.workunit.client.1.smithi189.stdout:3/285: mknod d2/d3/d2c/d31/c69 0 2024-04-20T21:32:07.684 INFO:tasks.workunit.client.1.smithi189.stdout:5/260: write d7/d14/f26 [151160,41448] 0 2024-04-20T21:32:07.685 INFO:tasks.workunit.client.0.smithi062.stdout:0/646: dread d7/d1c/d15/d17/fad [0,4194304] 0 2024-04-20T21:32:07.686 INFO:tasks.workunit.client.0.smithi062.stdout:1/714: rmdir dc/d15/d75/d96/dd4/d3a/da2 39 2024-04-20T21:32:07.686 INFO:tasks.workunit.client.1.smithi189.stdout:5/261: stat d7/d14/f26 0 2024-04-20T21:32:07.687 INFO:tasks.workunit.client.0.smithi062.stdout:9/707: read d16/f1e [957039,130647] 0 2024-04-20T21:32:07.688 INFO:tasks.workunit.client.1.smithi189.stdout:6/338: dwrite d8/da/dd/d35/f59 [0,4194304] 0 2024-04-20T21:32:07.688 INFO:tasks.workunit.client.1.smithi189.stdout:8/429: rename da/df/d13/d63/c7e to da/df/d20/d2c/d9a/c9c 0 2024-04-20T21:32:07.693 INFO:tasks.workunit.client.1.smithi189.stdout:0/435: mknod d3/d8c/c8f 0 2024-04-20T21:32:07.696 INFO:tasks.workunit.client.0.smithi062.stdout:4/694: rename da/dd/df/d27/d32/l9a to da/d56/d71/d9e/lf6 0 2024-04-20T21:32:07.697 INFO:tasks.workunit.client.1.smithi189.stdout:2/294: stat d1/d17/d37/l3b 0 2024-04-20T21:32:07.699 INFO:tasks.workunit.client.0.smithi062.stdout:6/538: dwrite d0/d9/d11/d22/f68 [0,4194304] 0 2024-04-20T21:32:07.699 INFO:tasks.workunit.client.0.smithi062.stdout:1/715: read - dc/d15/d75/d96/dd4/d3a/da2/fb4 zero size 2024-04-20T21:32:07.703 INFO:tasks.workunit.client.1.smithi189.stdout:4/300: symlink d2/d22/l71 0 2024-04-20T21:32:07.704 INFO:tasks.workunit.client.0.smithi062.stdout:4/695: chown da/d56/f74 60 1 2024-04-20T21:32:07.704 INFO:tasks.workunit.client.0.smithi062.stdout:8/656: getdents d7/d25/d3b/d58/d8b/d24/d46 0 2024-04-20T21:32:07.705 INFO:tasks.workunit.client.0.smithi062.stdout:9/708: rename d16/fa5 to d16/d2c/d42/da6/da9/feb 0 2024-04-20T21:32:07.705 INFO:tasks.workunit.client.0.smithi062.stdout:0/647: link d7/d20/c7b d7/d1c/d15/d17/d1b/cda 0 2024-04-20T21:32:07.707 INFO:tasks.workunit.client.1.smithi189.stdout:6/339: creat d8/d2b/f74 x:0 0 0 2024-04-20T21:32:07.708 INFO:tasks.workunit.client.0.smithi062.stdout:9/709: dread - d16/d3e/fd5 zero size 2024-04-20T21:32:07.711 INFO:tasks.workunit.client.0.smithi062.stdout:4/696: chown da/dd/df/d27/d3f/d82/f8b 1933312 1 2024-04-20T21:32:07.713 INFO:tasks.workunit.client.0.smithi062.stdout:5/661: dwrite d8/d38/d7a/d4c/d7d/d9d/dac/d66/dae/f9c [0,4194304] 0 2024-04-20T21:32:07.713 INFO:tasks.workunit.client.1.smithi189.stdout:2/295: unlink d1/f23 0 2024-04-20T21:32:07.716 INFO:tasks.workunit.client.0.smithi062.stdout:1/716: mknod dc/d15/d75/d96/dd4/d78/dab/c101 0 2024-04-20T21:32:07.717 INFO:tasks.workunit.client.0.smithi062.stdout:4/697: stat da/dd/d15/d50/d64/d57/f65 0 2024-04-20T21:32:07.717 INFO:tasks.workunit.client.0.smithi062.stdout:9/710: mknod d16/d2c/d62/dd1/cec 0 2024-04-20T21:32:07.718 INFO:tasks.workunit.client.1.smithi189.stdout:3/286: rename d2/c11 to d2/d3/d65/c6a 0 2024-04-20T21:32:07.719 INFO:tasks.workunit.client.0.smithi062.stdout:1/717: chown dc/l9d 0 1 2024-04-20T21:32:07.721 INFO:tasks.workunit.client.0.smithi062.stdout:1/718: stat dc/d15/d20/d28/c38 0 2024-04-20T21:32:07.721 INFO:tasks.workunit.client.1.smithi189.stdout:3/287: readlink d2/l1e 0 2024-04-20T21:32:07.722 INFO:tasks.workunit.client.1.smithi189.stdout:2/296: rmdir d1/d11/d26 39 2024-04-20T21:32:07.723 INFO:tasks.workunit.client.0.smithi062.stdout:8/657: creat d7/d30/db3/fd9 x:0 0 0 2024-04-20T21:32:07.724 INFO:tasks.workunit.client.0.smithi062.stdout:6/539: rename d0/d4f/f57 to d0/fb6 0 2024-04-20T21:32:07.725 INFO:tasks.workunit.client.0.smithi062.stdout:5/662: mkdir d8/d10/d3e/d8a/d9e/dd7 0 2024-04-20T21:32:07.725 INFO:tasks.workunit.client.1.smithi189.stdout:3/288: dread - d2/d3/d25/f67 zero size 2024-04-20T21:32:07.728 INFO:tasks.workunit.client.0.smithi062.stdout:9/711: mknod d16/d4e/d80/ced 0 2024-04-20T21:32:07.731 INFO:tasks.workunit.client.0.smithi062.stdout:9/712: fdatasync d16/d4e/fbd 0 2024-04-20T21:32:07.731 INFO:tasks.workunit.client.0.smithi062.stdout:4/698: mkdir da/dd/d15/d50/d64/d57/d5b/df7 0 2024-04-20T21:32:07.732 INFO:tasks.workunit.client.0.smithi062.stdout:6/540: fdatasync d0/f80 0 2024-04-20T21:32:07.736 INFO:tasks.workunit.client.1.smithi189.stdout:0/436: truncate d3/df/d28/d2b/d2c/f83 221939 0 2024-04-20T21:32:07.736 INFO:tasks.workunit.client.1.smithi189.stdout:4/301: dwrite d2/d20/f4f [0,4194304] 0 2024-04-20T21:32:07.737 INFO:tasks.workunit.client.0.smithi062.stdout:0/648: creat d7/d1e/fdb x:0 0 0 2024-04-20T21:32:07.737 INFO:tasks.workunit.client.0.smithi062.stdout:6/541: read - d0/d9/d11/d5c/f86 zero size 2024-04-20T21:32:07.738 INFO:tasks.workunit.client.0.smithi062.stdout:1/719: link dc/d15/d75/d96/fc7 dc/d15/d75/d96/dd4/d3a/da2/de4/d84/f102 0 2024-04-20T21:32:07.740 INFO:tasks.workunit.client.1.smithi189.stdout:5/262: dwrite d7/d9/d30/f1c [0,4194304] 0 2024-04-20T21:32:07.740 INFO:tasks.workunit.client.0.smithi062.stdout:8/658: rename d7/d9/cd8 to d7/d25/d3b/d58/d8b/d24/d46/cda 0 2024-04-20T21:32:07.741 INFO:tasks.workunit.client.1.smithi189.stdout:2/297: dread d1/d11/d2d/f38 [0,4194304] 0 2024-04-20T21:32:07.741 INFO:tasks.workunit.client.0.smithi062.stdout:9/713: write d16/d3e/d77/d84/f9a [577391,66485] 0 2024-04-20T21:32:07.741 INFO:tasks.workunit.client.0.smithi062.stdout:4/699: write da/dd/d15/fa7 [3656136,106757] 0 2024-04-20T21:32:07.742 INFO:tasks.workunit.client.0.smithi062.stdout:0/649: fsync d7/d1c/d15/d17/d1b/d9d/fa0 0 2024-04-20T21:32:07.745 INFO:tasks.workunit.client.1.smithi189.stdout:6/340: dwrite d8/da/dd/d2a/f66 [0,4194304] 0 2024-04-20T21:32:07.745 INFO:tasks.workunit.client.1.smithi189.stdout:0/437: unlink d3/f1f 0 2024-04-20T21:32:07.749 INFO:tasks.workunit.client.1.smithi189.stdout:6/341: chown d8/d2b/d49/d54/d67 249503660 1 2024-04-20T21:32:07.751 INFO:tasks.workunit.client.1.smithi189.stdout:4/302: unlink d2/d20/l6c 0 2024-04-20T21:32:07.751 INFO:tasks.workunit.client.1.smithi189.stdout:0/438: readlink d3/df/d30/l3c 0 2024-04-20T21:32:07.753 INFO:tasks.workunit.client.0.smithi062.stdout:8/659: creat d7/d25/d4f/fdb x:0 0 0 2024-04-20T21:32:07.753 INFO:tasks.workunit.client.0.smithi062.stdout:6/542: symlink d0/d9/d11/d7b/d99/lb7 0 2024-04-20T21:32:07.754 INFO:tasks.workunit.client.0.smithi062.stdout:5/663: rename d8/d38/d7a/d4c/cad to d8/d38/d7a/d4c/d7d/cd8 0 2024-04-20T21:32:07.754 INFO:tasks.workunit.client.1.smithi189.stdout:2/298: write d1/d17/d2f/f49 [1792764,89579] 0 2024-04-20T21:32:07.757 INFO:tasks.workunit.client.1.smithi189.stdout:5/263: rmdir d7/d14/d1b/d1f/d2a 39 2024-04-20T21:32:07.759 INFO:tasks.workunit.client.1.smithi189.stdout:3/289: dwrite d2/d8/f10 [0,4194304] 0 2024-04-20T21:32:07.759 INFO:tasks.workunit.client.0.smithi062.stdout:6/543: read d0/d2c/f8b [2210770,125750] 0 2024-04-20T21:32:07.760 INFO:tasks.workunit.client.0.smithi062.stdout:4/700: readlink da/dd/df/d27/d3f/le1 0 2024-04-20T21:32:07.761 INFO:tasks.workunit.client.0.smithi062.stdout:7/679: sync 2024-04-20T21:32:07.761 INFO:tasks.workunit.client.0.smithi062.stdout:2/686: sync 2024-04-20T21:32:07.761 INFO:tasks.workunit.client.0.smithi062.stdout:3/634: sync 2024-04-20T21:32:07.762 INFO:tasks.workunit.client.1.smithi189.stdout:3/290: chown d2/d8/f58 83077 1 2024-04-20T21:32:07.770 INFO:tasks.workunit.client.1.smithi189.stdout:0/439: mkdir d3/df/d30/d45/d90 0 2024-04-20T21:32:07.771 INFO:tasks.workunit.client.0.smithi062.stdout:6/544: rename d0/d9/d11/fa2 to d0/d1a/d44/fb8 0 2024-04-20T21:32:07.772 INFO:tasks.workunit.client.0.smithi062.stdout:5/664: chown d8/d38/d7a/d4c/f7b 11245421 1 2024-04-20T21:32:07.772 INFO:tasks.workunit.client.1.smithi189.stdout:5/264: rename d7/fb to d7/d14/f49 0 2024-04-20T21:32:07.773 INFO:tasks.workunit.client.0.smithi062.stdout:4/701: creat da/dd/d15/d6d/df5/ff8 x:0 0 0 2024-04-20T21:32:07.774 INFO:tasks.workunit.client.0.smithi062.stdout:3/635: readlink d0/d1/d10/l66 0 2024-04-20T21:32:07.774 INFO:tasks.workunit.client.1.smithi189.stdout:4/303: creat d2/dd/d6b/f72 x:0 0 0 2024-04-20T21:32:07.778 INFO:tasks.workunit.client.0.smithi062.stdout:2/687: fdatasync d7/da/d15/f23 0 2024-04-20T21:32:07.778 INFO:tasks.workunit.client.1.smithi189.stdout:3/291: creat d2/d3/d2c/f6b x:0 0 0 2024-04-20T21:32:07.779 INFO:tasks.workunit.client.1.smithi189.stdout:6/342: dread d8/d12/d30/d47/f52 [0,4194304] 0 2024-04-20T21:32:07.786 INFO:tasks.workunit.client.0.smithi062.stdout:7/680: mkdir d7/d3e/d33/d3d/d4d/d54/d4f/de9 0 2024-04-20T21:32:07.786 INFO:tasks.workunit.client.0.smithi062.stdout:4/702: creat da/dd/df/d27/d32/ff9 x:0 0 0 2024-04-20T21:32:07.787 INFO:tasks.workunit.client.0.smithi062.stdout:6/545: mkdir d0/d1a/d7d/db1/db9 0 2024-04-20T21:32:07.787 INFO:tasks.workunit.client.0.smithi062.stdout:8/660: dread d7/d25/d3b/d58/d8b/d24/f74 [0,4194304] 0 2024-04-20T21:32:07.787 INFO:tasks.workunit.client.1.smithi189.stdout:2/299: rmdir d1/d11/d26 39 2024-04-20T21:32:07.789 INFO:tasks.workunit.client.1.smithi189.stdout:3/292: mknod d2/d3/d65/c6c 0 2024-04-20T21:32:07.790 INFO:tasks.workunit.client.0.smithi062.stdout:4/703: unlink da/d56/d71/f79 0 2024-04-20T21:32:07.791 INFO:tasks.workunit.client.1.smithi189.stdout:3/293: stat d2/d3/d65/c6c 0 2024-04-20T21:32:07.791 INFO:tasks.workunit.client.0.smithi062.stdout:2/688: mkdir d7/da/d15/d2f/d38/de6 0 2024-04-20T21:32:07.792 INFO:tasks.workunit.client.1.smithi189.stdout:5/265: dread d7/d14/d18/f17 [0,4194304] 0 2024-04-20T21:32:07.793 INFO:tasks.workunit.client.1.smithi189.stdout:2/300: read d1/f7 [4997324,94616] 0 2024-04-20T21:32:07.794 INFO:tasks.workunit.client.0.smithi062.stdout:5/665: dread d8/d1e/d2c/f69 [0,4194304] 0 2024-04-20T21:32:07.794 INFO:tasks.workunit.client.0.smithi062.stdout:8/661: readlink d7/da9/lc9 0 2024-04-20T21:32:07.796 INFO:tasks.workunit.client.0.smithi062.stdout:6/546: symlink d0/d1a/d44/da7/lba 0 2024-04-20T21:32:07.796 INFO:tasks.workunit.client.1.smithi189.stdout:4/304: creat d2/dd/d11/f73 x:0 0 0 2024-04-20T21:32:07.797 INFO:tasks.workunit.client.1.smithi189.stdout:3/294: creat d2/d3/d2c/f6d x:0 0 0 2024-04-20T21:32:07.797 INFO:tasks.workunit.client.0.smithi062.stdout:4/704: rmdir da/dd/df/d27/d32/d97 39 2024-04-20T21:32:07.799 INFO:tasks.workunit.client.0.smithi062.stdout:3/636: link d0/d1c/d63/d42/fcb d0/d1/d10/d3a/d5e/d65/d88/fd6 0 2024-04-20T21:32:07.799 INFO:tasks.workunit.client.1.smithi189.stdout:2/301: write d1/d17/f35 [467527,56640] 0 2024-04-20T21:32:07.800 INFO:tasks.workunit.client.1.smithi189.stdout:3/295: write d2/f6 [849020,104547] 0 2024-04-20T21:32:07.802 INFO:tasks.workunit.client.0.smithi062.stdout:8/662: chown d7/d25/d3b/d58/d8b/d24/f74 59583 1 2024-04-20T21:32:07.802 INFO:tasks.workunit.client.0.smithi062.stdout:1/720: truncate dc/d15/d20/f3c 2861544 0 2024-04-20T21:32:07.803 INFO:tasks.workunit.client.0.smithi062.stdout:2/689: chown d7/da/d15/d2f/d38/d51/dbc/le4 278513369 1 2024-04-20T21:32:07.803 INFO:tasks.workunit.client.1.smithi189.stdout:3/296: dread - d2/d3/d2c/d30/f68 zero size 2024-04-20T21:32:07.804 INFO:tasks.workunit.client.0.smithi062.stdout:0/650: write d7/d1c/d15/d80/fb2 [4329740,128023] 0 2024-04-20T21:32:07.804 INFO:tasks.workunit.client.0.smithi062.stdout:5/666: rmdir d8/d38/d7a/db1 39 2024-04-20T21:32:07.807 INFO:tasks.workunit.client.1.smithi189.stdout:5/266: truncate d7/f2b 1804296 0 2024-04-20T21:32:07.809 INFO:tasks.workunit.client.1.smithi189.stdout:5/267: chown d7/d14/d1b/d1f/d33 28689 1 2024-04-20T21:32:07.810 INFO:tasks.workunit.client.1.smithi189.stdout:2/302: unlink d1/d17/l48 0 2024-04-20T21:32:07.810 INFO:tasks.workunit.client.0.smithi062.stdout:4/705: mkdir da/dd/df/dfa 0 2024-04-20T21:32:07.813 INFO:tasks.workunit.client.0.smithi062.stdout:6/547: rmdir d0/d1a/d7d 39 2024-04-20T21:32:07.814 INFO:tasks.workunit.client.0.smithi062.stdout:8/663: unlink d7/fab 0 2024-04-20T21:32:07.816 INFO:tasks.workunit.client.1.smithi189.stdout:4/305: creat d2/dd/d11/f74 x:0 0 0 2024-04-20T21:32:07.817 INFO:tasks.workunit.client.1.smithi189.stdout:2/303: symlink d1/d11/d2d/l57 0 2024-04-20T21:32:07.817 INFO:tasks.workunit.client.1.smithi189.stdout:6/343: dread d8/d12/d30/f38 [0,4194304] 0 2024-04-20T21:32:07.817 INFO:tasks.workunit.client.1.smithi189.stdout:5/268: write d7/d14/d18/f3e [4347736,69185] 0 2024-04-20T21:32:07.819 INFO:tasks.workunit.client.0.smithi062.stdout:9/714: dread d16/d2c/d5e/f7b [0,4194304] 0 2024-04-20T21:32:07.819 INFO:tasks.workunit.client.0.smithi062.stdout:5/667: truncate d8/d1e/f53 1313254 0 2024-04-20T21:32:07.820 INFO:tasks.workunit.client.0.smithi062.stdout:0/651: dread d7/d1c/f3e [0,4194304] 0 2024-04-20T21:32:07.822 INFO:tasks.workunit.client.1.smithi189.stdout:6/344: creat d8/d2b/f75 x:0 0 0 2024-04-20T21:32:07.823 INFO:tasks.workunit.client.1.smithi189.stdout:6/345: chown d8/da/l42 12074013 1 2024-04-20T21:32:07.825 INFO:tasks.workunit.client.0.smithi062.stdout:1/721: creat dc/f103 x:0 0 0 2024-04-20T21:32:07.825 INFO:tasks.workunit.client.0.smithi062.stdout:9/715: read d16/f3b [1320770,130257] 0 2024-04-20T21:32:07.827 INFO:tasks.workunit.client.1.smithi189.stdout:6/346: chown d8/d10/f3f 431247 1 2024-04-20T21:32:07.828 INFO:tasks.workunit.client.1.smithi189.stdout:5/269: rename d7/d14/d1b/c40 to d7/d9/d30/c4a 0 2024-04-20T21:32:07.829 INFO:tasks.workunit.client.0.smithi062.stdout:3/637: getdents d0/d1/d3c/d43/d72/dd3/dc5 0 2024-04-20T21:32:07.829 INFO:tasks.workunit.client.0.smithi062.stdout:8/664: unlink d7/d25/d3b/fa2 0 2024-04-20T21:32:07.829 INFO:tasks.workunit.client.1.smithi189.stdout:6/347: fdatasync d8/d10/f3f 0 2024-04-20T21:32:07.832 INFO:tasks.workunit.client.1.smithi189.stdout:5/270: write d7/d9/d30/f1c [3926695,39995] 0 2024-04-20T21:32:07.832 INFO:tasks.workunit.client.0.smithi062.stdout:5/668: chown d8/d38/d7a/d4c/d7d/d9d/dac/d66/fa8 432546687 1 2024-04-20T21:32:07.834 INFO:tasks.workunit.client.1.smithi189.stdout:3/297: rmdir d2/d3/d2c 39 2024-04-20T21:32:07.834 INFO:tasks.workunit.client.0.smithi062.stdout:6/548: mknod d0/d52/cbb 0 2024-04-20T21:32:07.836 INFO:tasks.workunit.client.1.smithi189.stdout:5/271: write d7/d14/d18/f17 [1932436,115181] 0 2024-04-20T21:32:07.837 INFO:tasks.workunit.client.0.smithi062.stdout:5/669: write d8/d38/fbe [1021109,95859] 0 2024-04-20T21:32:07.837 INFO:tasks.workunit.client.1.smithi189.stdout:2/304: dwrite d1/d17/f35 [0,4194304] 0 2024-04-20T21:32:07.841 INFO:tasks.workunit.client.0.smithi062.stdout:9/716: symlink d16/d2c/d5e/d75/d76/lee 0 2024-04-20T21:32:07.841 INFO:tasks.workunit.client.0.smithi062.stdout:3/638: rmdir d0/d1 39 2024-04-20T21:32:07.844 INFO:tasks.workunit.client.0.smithi062.stdout:8/665: creat d7/d25/d4f/fdc x:0 0 0 2024-04-20T21:32:07.846 INFO:tasks.workunit.client.1.smithi189.stdout:4/306: creat d2/f75 x:0 0 0 2024-04-20T21:32:07.851 INFO:tasks.workunit.client.0.smithi062.stdout:4/706: write da/dd/d15/d50/d64/d57/d5b/d9d/dea/dac/fe3 [3432856,77708] 0 2024-04-20T21:32:07.853 INFO:tasks.workunit.client.1.smithi189.stdout:3/298: dwrite d2/f2a [4194304,4194304] 0 2024-04-20T21:32:07.854 INFO:tasks.workunit.client.0.smithi062.stdout:2/690: dwrite d7/da/d15/d1b/f6c [0,4194304] 0 2024-04-20T21:32:07.855 INFO:tasks.workunit.client.0.smithi062.stdout:5/670: symlink d8/d10/ld9 0 2024-04-20T21:32:07.856 INFO:tasks.workunit.client.1.smithi189.stdout:9/336: sync 2024-04-20T21:32:07.856 INFO:tasks.workunit.client.1.smithi189.stdout:1/298: sync 2024-04-20T21:32:07.857 INFO:tasks.workunit.client.0.smithi062.stdout:6/549: creat d0/d9/d11/fbc x:0 0 0 2024-04-20T21:32:07.859 INFO:tasks.workunit.client.1.smithi189.stdout:9/337: dread d9/f44 [0,4194304] 0 2024-04-20T21:32:07.860 INFO:tasks.workunit.client.1.smithi189.stdout:3/299: fdatasync d2/d3/d25/f67 0 2024-04-20T21:32:07.860 INFO:tasks.workunit.client.0.smithi062.stdout:1/722: getdents dc/d27/d9a 0 2024-04-20T21:32:07.862 INFO:tasks.workunit.client.1.smithi189.stdout:2/305: chown d1/d17/d37/c47 0 1 2024-04-20T21:32:07.867 INFO:tasks.workunit.client.1.smithi189.stdout:4/307: dwrite d2/d20/f24 [4194304,4194304] 0 2024-04-20T21:32:07.868 INFO:tasks.workunit.client.0.smithi062.stdout:2/691: readlink d7/da/d15/d1b/d24/l35 0 2024-04-20T21:32:07.872 INFO:tasks.workunit.client.0.smithi062.stdout:0/652: dwrite d7/d1c/d15/d17/d37/f61 [0,4194304] 0 2024-04-20T21:32:07.878 INFO:tasks.workunit.client.1.smithi189.stdout:3/300: readlink d2/d3/d2c/d30/d3f/l49 0 2024-04-20T21:32:07.880 INFO:tasks.workunit.client.1.smithi189.stdout:5/272: dread d7/f2b [0,4194304] 0 2024-04-20T21:32:07.881 INFO:tasks.workunit.client.0.smithi062.stdout:2/692: rmdir d7/da/d15/d2f 39 2024-04-20T21:32:07.884 INFO:tasks.workunit.client.0.smithi062.stdout:9/717: dwrite d16/d2c/d62/fbc [0,4194304] 0 2024-04-20T21:32:07.884 INFO:tasks.workunit.client.1.smithi189.stdout:2/306: mkdir d1/d11/d58 0 2024-04-20T21:32:07.886 INFO:tasks.workunit.client.1.smithi189.stdout:2/307: stat d1/d11/d2d/c54 0 2024-04-20T21:32:07.886 INFO:tasks.workunit.client.0.smithi062.stdout:4/707: getdents da/dd/d15/daf 0 2024-04-20T21:32:07.888 INFO:tasks.workunit.client.0.smithi062.stdout:6/550: rename d0/d9/d11/d22/f24 to d0/d9/fbd 0 2024-04-20T21:32:07.894 INFO:tasks.workunit.client.1.smithi189.stdout:5/273: rename d7/d9/d31/l45 to d7/d9/l4b 0 2024-04-20T21:32:07.894 INFO:tasks.workunit.client.0.smithi062.stdout:3/639: getdents d0/d1/d3c/d43/d72/dd3/d97 0 2024-04-20T21:32:07.897 INFO:tasks.workunit.client.0.smithi062.stdout:8/666: dread d7/d30/fba [0,4194304] 0 2024-04-20T21:32:07.899 INFO:tasks.workunit.client.0.smithi062.stdout:9/718: write d16/d2c/d5e/d96/fb3 [961705,31790] 0 2024-04-20T21:32:07.899 INFO:tasks.workunit.client.0.smithi062.stdout:5/671: rename d8/d38/d7a/db1/cbf to d8/d38/d7a/d4c/d7d/d9d/dac/d66/cda 0 2024-04-20T21:32:07.900 INFO:tasks.workunit.client.1.smithi189.stdout:3/301: dwrite d2/d3/d25/f40 [4194304,4194304] 0 2024-04-20T21:32:07.904 INFO:tasks.workunit.client.1.smithi189.stdout:2/308: dwrite d1/d11/f28 [0,4194304] 0 2024-04-20T21:32:07.905 INFO:tasks.workunit.client.0.smithi062.stdout:2/693: symlink d7/da/d15/d2f/d93/dac/dca/dd8/le7 0 2024-04-20T21:32:07.908 INFO:tasks.workunit.client.0.smithi062.stdout:4/708: creat da/dd/df/d27/da5/df1/ffb x:0 0 0 2024-04-20T21:32:07.910 INFO:tasks.workunit.client.0.smithi062.stdout:3/640: symlink d0/d1/d10/d31/d41/ld7 0 2024-04-20T21:32:07.911 INFO:tasks.workunit.client.1.smithi189.stdout:9/338: getdents d9/d12/d3f 0 2024-04-20T21:32:07.917 INFO:tasks.workunit.client.0.smithi062.stdout:5/672: mknod d8/d10/cdb 0 2024-04-20T21:32:07.918 INFO:tasks.workunit.client.1.smithi189.stdout:5/274: creat d7/d14/f4c x:0 0 0 2024-04-20T21:32:07.920 INFO:tasks.workunit.client.0.smithi062.stdout:3/641: rename d0/d1/d10/f47 to d0/d1b/d74/fd8 0 2024-04-20T21:32:07.922 INFO:tasks.workunit.client.0.smithi062.stdout:8/667: link d7/d25/d4f/fdc d7/d25/d3b/d58/d8b/d96/fdd 0 2024-04-20T21:32:07.923 INFO:tasks.workunit.client.0.smithi062.stdout:4/709: mknod da/dd/d15/d50/d64/d57/cfc 0 2024-04-20T21:32:07.927 INFO:tasks.workunit.client.1.smithi189.stdout:6/348: dwrite d8/d10/f2d [0,4194304] 0 2024-04-20T21:32:07.928 INFO:tasks.workunit.client.1.smithi189.stdout:6/349: fdatasync d8/d2b/d49/f60 0 2024-04-20T21:32:07.929 INFO:tasks.workunit.client.1.smithi189.stdout:4/308: link d2/dd/d11/l2a d2/l76 0 2024-04-20T21:32:07.930 INFO:tasks.workunit.client.0.smithi062.stdout:4/710: chown da/dd/f4d 2033267 1 2024-04-20T21:32:07.933 INFO:tasks.workunit.client.0.smithi062.stdout:3/642: mknod d0/d1/d10/d3a/d5e/d65/cd9 0 2024-04-20T21:32:07.933 INFO:tasks.workunit.client.0.smithi062.stdout:8/668: write d7/d25/d5c/f69 [3196782,72638] 0 2024-04-20T21:32:07.935 INFO:tasks.workunit.client.1.smithi189.stdout:9/339: fsync f7 0 2024-04-20T21:32:07.936 INFO:tasks.workunit.client.1.smithi189.stdout:6/350: symlink d8/d12/d30/l76 0 2024-04-20T21:32:07.940 INFO:tasks.workunit.client.0.smithi062.stdout:1/723: dwrite dc/d15/d75/d96/dd4/d78/faa [0,4194304] 0 2024-04-20T21:32:07.942 INFO:tasks.workunit.client.0.smithi062.stdout:4/711: creat da/d56/d71/d9e/ffd x:0 0 0 2024-04-20T21:32:07.942 INFO:tasks.workunit.client.0.smithi062.stdout:8/669: read - d7/d25/d3b/d58/f6b zero size 2024-04-20T21:32:07.942 INFO:tasks.workunit.client.0.smithi062.stdout:5/673: link d8/d38/d7a/d4c/d7d/d9d/dac/d66/fa8 d8/dbd/fdc 0 2024-04-20T21:32:07.942 INFO:tasks.workunit.client.1.smithi189.stdout:3/302: rename d2/d3/d2c/d31/d4c/c62 to d2/c6e 0 2024-04-20T21:32:07.943 INFO:tasks.workunit.client.0.smithi062.stdout:2/694: dwrite d7/da/d15/d2f/d38/d6a/d7c/f96 [4194304,4194304] 0 2024-04-20T21:32:07.950 INFO:tasks.workunit.client.1.smithi189.stdout:6/351: rmdir d8/d2b/d5c 39 2024-04-20T21:32:07.952 INFO:tasks.workunit.client.0.smithi062.stdout:3/643: unlink d0/d1/d3c/d43/d72/dd3/d7b/cd0 0 2024-04-20T21:32:07.952 INFO:tasks.workunit.client.1.smithi189.stdout:2/309: link d1/d17/c18 d1/d11/d52/c59 0 2024-04-20T21:32:07.952 INFO:tasks.workunit.client.1.smithi189.stdout:6/352: chown d8/d2b/d49/d54/d67 78007 1 2024-04-20T21:32:07.953 INFO:tasks.workunit.client.0.smithi062.stdout:4/712: dread - da/dd/df/d27/d3f/fdf zero size 2024-04-20T21:32:07.953 INFO:tasks.workunit.client.0.smithi062.stdout:7/681: sync 2024-04-20T21:32:07.956 INFO:tasks.workunit.client.0.smithi062.stdout:8/670: rmdir d7/d25/dc5 39 2024-04-20T21:32:07.957 INFO:tasks.workunit.client.0.smithi062.stdout:5/674: symlink d8/d1e/d2c/dd1/ldd 0 2024-04-20T21:32:07.960 INFO:tasks.workunit.client.0.smithi062.stdout:2/695: write d7/da/d15/d1b/f80 [12619,19460] 0 2024-04-20T21:32:07.964 INFO:tasks.workunit.client.0.smithi062.stdout:1/724: creat dc/d15/d75/d96/dd4/d3a/da2/de4/d84/f104 x:0 0 0 2024-04-20T21:32:07.965 INFO:tasks.workunit.client.1.smithi189.stdout:5/275: rename d7/d14/d1b/d1f/d2a to d7/d14/d18/d4d 0 2024-04-20T21:32:07.965 INFO:tasks.workunit.client.0.smithi062.stdout:0/653: write d7/d1c/d15/f22 [718709,8850] 0 2024-04-20T21:32:07.966 INFO:tasks.workunit.client.0.smithi062.stdout:3/644: rmdir d0/d1/d3c/d43/d72/dd3 39 2024-04-20T21:32:07.967 INFO:tasks.workunit.client.1.smithi189.stdout:4/309: getdents d2/d22 0 2024-04-20T21:32:07.969 INFO:tasks.workunit.client.1.smithi189.stdout:4/310: fsync d2/d22/f45 0 2024-04-20T21:32:07.970 INFO:tasks.workunit.client.0.smithi062.stdout:8/671: creat d7/d25/dc5/fde x:0 0 0 2024-04-20T21:32:07.971 INFO:tasks.workunit.client.1.smithi189.stdout:2/310: creat d1/d11/d26/f5a x:0 0 0 2024-04-20T21:32:07.973 INFO:tasks.workunit.client.0.smithi062.stdout:5/675: creat d8/d38/d7a/d4c/d7d/fde x:0 0 0 2024-04-20T21:32:07.973 INFO:tasks.workunit.client.1.smithi189.stdout:4/311: dread - d2/dd/d11/f74 zero size 2024-04-20T21:32:07.974 INFO:tasks.workunit.client.1.smithi189.stdout:4/312: readlink d2/dd/d11/d14/l34 0 2024-04-20T21:32:07.975 INFO:tasks.workunit.client.1.smithi189.stdout:4/313: dread - d2/d3a/f3c zero size 2024-04-20T21:32:07.976 INFO:tasks.workunit.client.0.smithi062.stdout:6/551: write d0/d9/d11/f65 [503461,95294] 0 2024-04-20T21:32:07.977 INFO:tasks.workunit.client.1.smithi189.stdout:2/311: symlink d1/d17/d2f/d55/l5b 0 2024-04-20T21:32:07.978 INFO:tasks.workunit.client.0.smithi062.stdout:4/713: link da/dd/d15/d50/d64/d3d/da1/fb9 da/d4f/ffe 0 2024-04-20T21:32:07.982 INFO:tasks.workunit.client.1.smithi189.stdout:4/314: unlink d2/dd/c35 0 2024-04-20T21:32:07.984 INFO:tasks.workunit.client.0.smithi062.stdout:8/672: write d7/d25/d3b/d58/d8b/d24/d46/fd2 [881030,52838] 0 2024-04-20T21:32:07.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:07 smithi189 ceph-mon[21562]: pgmap v18: 65 pgs: 65 active+clean; 1.7 GiB data, 5.0 GiB used, 531 GiB / 536 GiB avail; 48 MiB/s rd, 176 MiB/s wr, 341 op/s 2024-04-20T21:32:07.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:07 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:32:07.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:07 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:32:07.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:07 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:32:07.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:07 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:32:07.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:07 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:32:07.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:07 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:32:07.987 INFO:tasks.workunit.client.0.smithi062.stdout:2/696: link d7/da/d15/f26 d7/db5/fe8 0 2024-04-20T21:32:07.988 INFO:tasks.workunit.client.0.smithi062.stdout:6/552: rmdir d0/d1a/d44 39 2024-04-20T21:32:07.988 INFO:tasks.workunit.client.1.smithi189.stdout:4/315: fdatasync d2/f54 0 2024-04-20T21:32:07.990 INFO:tasks.workunit.client.1.smithi189.stdout:4/316: chown d2/d36/l40 1579907798 1 2024-04-20T21:32:07.990 INFO:tasks.workunit.client.1.smithi189.stdout:5/276: dwrite d7/d14/d18/d4d/f39 [4194304,4194304] 0 2024-04-20T21:32:07.990 INFO:tasks.workunit.client.0.smithi062.stdout:1/725: rename dc/d27/d32/c6b to dc/d27/d9a/da0/db3/c105 0 2024-04-20T21:32:07.994 INFO:tasks.workunit.client.0.smithi062.stdout:4/714: fsync da/dd/d15/d50/d64/d3d/f5e 0 2024-04-20T21:32:07.995 INFO:tasks.workunit.client.1.smithi189.stdout:3/303: write d2/d3/d2c/d31/f50 [400545,25393] 0 2024-04-20T21:32:07.997 INFO:tasks.workunit.client.1.smithi189.stdout:8/430: sync 2024-04-20T21:32:07.997 INFO:tasks.workunit.client.1.smithi189.stdout:7/329: sync 2024-04-20T21:32:07.998 INFO:tasks.workunit.client.0.smithi062.stdout:8/673: creat d7/d25/d4f/fdf x:0 0 0 2024-04-20T21:32:08.001 INFO:tasks.workunit.client.1.smithi189.stdout:4/317: mknod d2/dd/d11/c77 0 2024-04-20T21:32:08.002 INFO:tasks.workunit.client.1.smithi189.stdout:2/312: dwrite d1/d17/d2f/f42 [0,4194304] 0 2024-04-20T21:32:08.003 INFO:tasks.workunit.client.1.smithi189.stdout:7/330: write d5/d1d/f56 [2869957,97182] 0 2024-04-20T21:32:08.006 INFO:tasks.workunit.client.0.smithi062.stdout:6/553: unlink d0/d1a/f3a 0 2024-04-20T21:32:08.008 INFO:tasks.workunit.client.0.smithi062.stdout:5/676: truncate d8/d38/d7a/d4c/d7d/d9d/dac/f6b 882502 0 2024-04-20T21:32:08.009 INFO:tasks.workunit.client.1.smithi189.stdout:2/313: chown d1/d11/c1d 93 1 2024-04-20T21:32:08.009 INFO:tasks.workunit.client.0.smithi062.stdout:9/719: dread d16/d3d/d55/d6e/f92 [0,4194304] 0 2024-04-20T21:32:08.012 INFO:tasks.workunit.client.1.smithi189.stdout:3/304: readlink d2/d3/d2c/d30/l66 0 2024-04-20T21:32:08.012 INFO:tasks.workunit.client.1.smithi189.stdout:2/314: dread - d1/d11/d26/f53 zero size 2024-04-20T21:32:08.012 INFO:tasks.workunit.client.1.smithi189.stdout:7/331: truncate d5/d21/d25/f27 55360 0 2024-04-20T21:32:08.013 INFO:tasks.workunit.client.0.smithi062.stdout:9/720: chown d16/d3e/fa2 3286101 1 2024-04-20T21:32:08.013 INFO:tasks.workunit.client.1.smithi189.stdout:8/431: mkdir da/df/d3c/d9d 0 2024-04-20T21:32:08.014 INFO:tasks.workunit.client.0.smithi062.stdout:0/654: dwrite d7/d1c/d15/d17/d37/f59 [0,4194304] 0 2024-04-20T21:32:08.016 INFO:tasks.workunit.client.0.smithi062.stdout:4/715: truncate da/fb 1015513 0 2024-04-20T21:32:08.016 INFO:tasks.workunit.client.0.smithi062.stdout:9/721: fsync d16/f36 0 2024-04-20T21:32:08.018 INFO:tasks.workunit.client.0.smithi062.stdout:3/645: dread d0/d1b/f35 [0,4194304] 0 2024-04-20T21:32:08.018 INFO:tasks.workunit.client.1.smithi189.stdout:2/315: chown d1/d11/d58 1 1 2024-04-20T21:32:08.019 INFO:tasks.workunit.client.1.smithi189.stdout:5/277: rename d7/d14/d18/d4d to d7/d14/d4e 0 2024-04-20T21:32:08.020 INFO:tasks.workunit.client.1.smithi189.stdout:7/332: chown d5/d21/d35/d53/d5b/d71 1604028 1 2024-04-20T21:32:08.021 INFO:tasks.workunit.client.0.smithi062.stdout:0/655: fsync d7/d1c/d15/d17/d1b/d9d/fba 0 2024-04-20T21:32:08.021 INFO:tasks.workunit.client.1.smithi189.stdout:4/318: creat d2/dd/f78 x:0 0 0 2024-04-20T21:32:08.022 INFO:tasks.workunit.client.1.smithi189.stdout:3/305: write d2/d3/d25/f54 [489206,61939] 0 2024-04-20T21:32:08.024 INFO:tasks.workunit.client.1.smithi189.stdout:9/340: dread d9/d12/d15/f26 [0,4194304] 0 2024-04-20T21:32:08.024 INFO:tasks.workunit.client.0.smithi062.stdout:0/656: chown d7/d1c/d15/d17/d1b/d9d/fba 674675987 1 2024-04-20T21:32:08.028 INFO:tasks.workunit.client.1.smithi189.stdout:3/306: fsync d2/f5d 0 2024-04-20T21:32:08.030 INFO:tasks.workunit.client.1.smithi189.stdout:2/316: write d1/d11/f1a [335737,99925] 0 2024-04-20T21:32:08.031 INFO:tasks.workunit.client.0.smithi062.stdout:3/646: unlink d0/d1/d10/d3a/d5e/c9b 0 2024-04-20T21:32:08.031 INFO:tasks.workunit.client.1.smithi189.stdout:4/319: dread - d2/dd/d37/f39 zero size 2024-04-20T21:32:08.034 INFO:tasks.workunit.client.0.smithi062.stdout:2/697: write d7/da/d28/d4b/f6e [1361118,120473] 0 2024-04-20T21:32:08.034 INFO:tasks.workunit.client.1.smithi189.stdout:9/341: chown d9/dc/d17/d2b/d52/l59 2523848 1 2024-04-20T21:32:08.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:07 smithi062 ceph-mon[16221]: pgmap v18: 65 pgs: 65 active+clean; 1.7 GiB data, 5.0 GiB used, 531 GiB / 536 GiB avail; 48 MiB/s rd, 176 MiB/s wr, 341 op/s 2024-04-20T21:32:08.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:07 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:32:08.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:07 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:32:08.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:07 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:32:08.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:07 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:32:08.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:07 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:32:08.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:07 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:32:08.036 INFO:tasks.workunit.client.1.smithi189.stdout:8/432: unlink da/df/d22/l8e 0 2024-04-20T21:32:08.038 INFO:tasks.workunit.client.0.smithi062.stdout:7/682: rename d7/d3e/d33/d3d/d5e/d7e to d7/d3e/d33/d3d/d4d/d54/dea 0 2024-04-20T21:32:08.040 INFO:tasks.workunit.client.0.smithi062.stdout:6/554: creat d0/d9/d11/d22/fbe x:0 0 0 2024-04-20T21:32:08.041 INFO:tasks.workunit.client.1.smithi189.stdout:4/320: creat d2/dd/d11/d14/f79 x:0 0 0 2024-04-20T21:32:08.042 INFO:tasks.workunit.client.0.smithi062.stdout:4/716: truncate da/dd/df/d27/d3f/d82/fbf 736889 0 2024-04-20T21:32:08.044 INFO:tasks.workunit.client.1.smithi189.stdout:9/342: creat d9/dc/d69/f6c x:0 0 0 2024-04-20T21:32:08.045 INFO:tasks.workunit.client.1.smithi189.stdout:9/343: chown d9/d12 1 1 2024-04-20T21:32:08.047 INFO:tasks.workunit.client.0.smithi062.stdout:6/555: rename d0/d52 to d0/d52/dbf 22 2024-04-20T21:32:08.048 INFO:tasks.workunit.client.0.smithi062.stdout:0/657: creat d7/d1e/fdc x:0 0 0 2024-04-20T21:32:08.049 INFO:tasks.workunit.client.0.smithi062.stdout:4/717: rmdir da/d56/d71 39 2024-04-20T21:32:08.051 INFO:tasks.workunit.client.0.smithi062.stdout:4/718: readlink da/dd/df/d27/d3f/lc5 0 2024-04-20T21:32:08.053 INFO:tasks.workunit.client.1.smithi189.stdout:9/344: symlink d9/dc/d17/d36/d5d/l6d 0 2024-04-20T21:32:08.055 INFO:tasks.workunit.client.0.smithi062.stdout:0/658: mkdir d7/d1c/d15/d6b/ddd 0 2024-04-20T21:32:08.055 INFO:tasks.workunit.client.1.smithi189.stdout:3/307: dread d2/d8/f58 [0,4194304] 0 2024-04-20T21:32:08.058 INFO:tasks.workunit.client.1.smithi189.stdout:9/345: chown d9/dc/d17/d2b/f6a 16 1 2024-04-20T21:32:08.059 INFO:tasks.workunit.client.0.smithi062.stdout:4/719: mknod da/dd/df/dfa/cff 0 2024-04-20T21:32:08.060 INFO:tasks.workunit.client.0.smithi062.stdout:0/659: truncate d7/d1c/d15/f22 1179065 0 2024-04-20T21:32:08.061 INFO:tasks.workunit.client.0.smithi062.stdout:0/660: chown d7/d1e/db6/l7a 3658859 1 2024-04-20T21:32:08.062 INFO:tasks.workunit.client.1.smithi189.stdout:7/333: dwrite d5/d46/f65 [0,4194304] 0 2024-04-20T21:32:08.062 INFO:tasks.workunit.client.1.smithi189.stdout:3/308: dread - d2/d3/d2c/d31/d4c/f63 zero size 2024-04-20T21:32:08.066 INFO:tasks.workunit.client.0.smithi062.stdout:1/726: write dc/d15/d75/d94/f1e [659869,9374] 0 2024-04-20T21:32:08.067 INFO:tasks.workunit.client.1.smithi189.stdout:7/334: fsync d5/d1d/f56 0 2024-04-20T21:32:08.069 INFO:tasks.workunit.client.1.smithi189.stdout:9/346: mknod d9/d12/d3b/c6e 0 2024-04-20T21:32:08.070 INFO:tasks.workunit.client.0.smithi062.stdout:7/683: dread d7/d3e/d33/d3d/d4d/d54/d4f/fb3 [0,4194304] 0 2024-04-20T21:32:08.071 INFO:tasks.workunit.client.0.smithi062.stdout:4/720: unlink da/dd/df/d27/f36 0 2024-04-20T21:32:08.073 INFO:tasks.workunit.client.1.smithi189.stdout:5/278: dwrite d7/f2b [0,4194304] 0 2024-04-20T21:32:08.077 INFO:tasks.workunit.client.1.smithi189.stdout:3/309: creat d2/d3/d2c/d31/f6f x:0 0 0 2024-04-20T21:32:08.080 INFO:tasks.workunit.client.1.smithi189.stdout:9/347: mknod d9/dc/d17/d2b/d52/c6f 0 2024-04-20T21:32:08.083 INFO:tasks.workunit.client.1.smithi189.stdout:5/279: mknod d7/d9/c4f 0 2024-04-20T21:32:08.088 INFO:tasks.workunit.client.1.smithi189.stdout:4/321: dwrite d2/dd/d11/d14/f79 [0,4194304] 0 2024-04-20T21:32:08.090 INFO:tasks.workunit.client.1.smithi189.stdout:0/440: sync 2024-04-20T21:32:08.090 INFO:tasks.workunit.client.0.smithi062.stdout:9/722: dwrite d16/d2c/d5e/d75/faa [0,4194304] 0 2024-04-20T21:32:08.093 INFO:tasks.workunit.client.1.smithi189.stdout:3/310: symlink d2/d3/d2c/d31/l70 0 2024-04-20T21:32:08.098 INFO:tasks.workunit.client.1.smithi189.stdout:9/348: rename d9/dc/d17/c32 to d9/dc/d17/d2b/d52/c70 0 2024-04-20T21:32:08.099 INFO:tasks.workunit.client.0.smithi062.stdout:4/721: dwrite da/dd/df/fc7 [0,4194304] 0 2024-04-20T21:32:08.100 INFO:tasks.workunit.client.1.smithi189.stdout:2/317: dwrite d1/d17/f2b [4194304,4194304] 0 2024-04-20T21:32:08.101 INFO:tasks.workunit.client.1.smithi189.stdout:8/433: dread da/df/d3c/d50/f6d [0,4194304] 0 2024-04-20T21:32:08.102 INFO:tasks.workunit.client.1.smithi189.stdout:7/335: dread d5/f1a [0,4194304] 0 2024-04-20T21:32:08.104 INFO:tasks.workunit.client.0.smithi062.stdout:5/677: dread d8/d38/d7a/d4c/f55 [0,4194304] 0 2024-04-20T21:32:08.106 INFO:tasks.workunit.client.1.smithi189.stdout:4/322: mkdir d2/dd/d11/d14/d69/d6a/d7a 0 2024-04-20T21:32:08.108 INFO:tasks.workunit.client.1.smithi189.stdout:7/336: fsync d5/d21/d4a/d52/f5c 0 2024-04-20T21:32:08.113 INFO:tasks.workunit.client.1.smithi189.stdout:3/311: creat d2/d1c/d5c/f71 x:0 0 0 2024-04-20T21:32:08.115 INFO:tasks.workunit.client.0.smithi062.stdout:1/727: truncate dc/d15/d75/d96/dd4/d3a/f4f 5299186 0 2024-04-20T21:32:08.115 INFO:tasks.workunit.client.1.smithi189.stdout:3/312: chown d2/d3/d2c/c4f 56566 1 2024-04-20T21:32:08.115 INFO:tasks.workunit.client.0.smithi062.stdout:2/698: write d7/da/f13 [3867274,25553] 0 2024-04-20T21:32:08.116 INFO:tasks.workunit.client.1.smithi189.stdout:0/441: rename d3/df/f3a to d3/df/f91 0 2024-04-20T21:32:08.117 INFO:tasks.workunit.client.1.smithi189.stdout:9/349: symlink d9/dc/d17/d2b/d52/l71 0 2024-04-20T21:32:08.119 INFO:tasks.workunit.client.1.smithi189.stdout:5/280: dwrite d7/d35/f43 [0,4194304] 0 2024-04-20T21:32:08.119 INFO:tasks.workunit.client.1.smithi189.stdout:9/350: read d9/dc/d17/d36/f4a [4126579,90487] 0 2024-04-20T21:32:08.121 INFO:tasks.workunit.client.0.smithi062.stdout:2/699: dread - d7/da/fdf zero size 2024-04-20T21:32:08.124 INFO:tasks.workunit.client.1.smithi189.stdout:0/442: fdatasync d3/df/d30/d45/d60/f71 0 2024-04-20T21:32:08.125 INFO:tasks.workunit.client.1.smithi189.stdout:4/323: rename d2/d22/l52 to d2/dd/d37/l7b 0 2024-04-20T21:32:08.125 INFO:tasks.workunit.client.1.smithi189.stdout:0/443: stat d3/fe 0 2024-04-20T21:32:08.127 INFO:tasks.workunit.client.1.smithi189.stdout:9/351: mkdir d9/d12/d3f/d72 0 2024-04-20T21:32:08.128 INFO:tasks.workunit.client.1.smithi189.stdout:9/352: stat f6 0 2024-04-20T21:32:08.128 INFO:tasks.workunit.client.1.smithi189.stdout:7/337: mknod d5/d59/c73 0 2024-04-20T21:32:08.129 INFO:tasks.workunit.client.0.smithi062.stdout:0/661: rename d7/d20/c7b to d7/cde 0 2024-04-20T21:32:08.129 INFO:tasks.workunit.client.0.smithi062.stdout:3/647: dwrite d0/d1b/f4d [0,4194304] 0 2024-04-20T21:32:08.132 INFO:tasks.workunit.client.1.smithi189.stdout:8/434: read da/df/d22/d33/f5a [4386048,122878] 0 2024-04-20T21:32:08.133 INFO:tasks.workunit.client.0.smithi062.stdout:7/684: link d7/d3e/d33/d3d/d4d/d54/dea/dcc/ce7 d7/d3e/d33/d3d/dad/ceb 0 2024-04-20T21:32:08.133 INFO:tasks.workunit.client.1.smithi189.stdout:4/324: mknod d2/dd/d37/c7c 0 2024-04-20T21:32:08.134 INFO:tasks.workunit.client.1.smithi189.stdout:8/435: dread - da/df/d20/f99 zero size 2024-04-20T21:32:08.136 INFO:tasks.workunit.client.1.smithi189.stdout:5/281: symlink d7/d35/d48/l50 0 2024-04-20T21:32:08.138 INFO:tasks.workunit.client.1.smithi189.stdout:7/338: mkdir d5/d21/d4a/d52/d74 0 2024-04-20T21:32:08.138 INFO:tasks.workunit.client.1.smithi189.stdout:5/282: fsync d7/f2b 0 2024-04-20T21:32:08.138 INFO:tasks.workunit.client.1.smithi189.stdout:2/318: dread d1/f12 [0,4194304] 0 2024-04-20T21:32:08.139 INFO:tasks.workunit.client.0.smithi062.stdout:4/722: unlink da/f83 0 2024-04-20T21:32:08.139 INFO:tasks.workunit.client.1.smithi189.stdout:8/436: mkdir da/df/d3c/d50/d9e 0 2024-04-20T21:32:08.140 INFO:tasks.workunit.client.1.smithi189.stdout:3/313: link d2/d8/l27 d2/d3/d2c/d30/l72 0 2024-04-20T21:32:08.141 INFO:tasks.workunit.client.1.smithi189.stdout:1/299: sync 2024-04-20T21:32:08.141 INFO:tasks.workunit.client.1.smithi189.stdout:6/353: sync 2024-04-20T21:32:08.142 INFO:tasks.workunit.client.1.smithi189.stdout:7/339: dread - d5/f57 zero size 2024-04-20T21:32:08.146 INFO:tasks.workunit.client.1.smithi189.stdout:4/325: creat d2/dd/d11/d14/f7d x:0 0 0 2024-04-20T21:32:08.149 INFO:tasks.workunit.client.0.smithi062.stdout:6/556: dwrite d0/d2c/f3b [0,4194304] 0 2024-04-20T21:32:08.149 INFO:tasks.workunit.client.0.smithi062.stdout:5/678: mknod d8/d38/d7a/cdf 0 2024-04-20T21:32:08.149 INFO:tasks.workunit.client.1.smithi189.stdout:0/444: link d3/d26/l8a d3/d26/l92 0 2024-04-20T21:32:08.149 INFO:tasks.workunit.client.1.smithi189.stdout:5/283: mkdir d7/d9/d51 0 2024-04-20T21:32:08.151 INFO:tasks.workunit.client.1.smithi189.stdout:2/319: mknod d1/d11/d26/c5c 0 2024-04-20T21:32:08.151 INFO:tasks.workunit.client.1.smithi189.stdout:4/326: stat d2/dd/d6b 0 2024-04-20T21:32:08.151 INFO:tasks.workunit.client.1.smithi189.stdout:1/300: truncate d8/d1e/d28/d4f/f5f 559521 0 2024-04-20T21:32:08.152 INFO:tasks.workunit.client.0.smithi062.stdout:1/728: creat dc/d15/d75/d94/d6f/dd2/f106 x:0 0 0 2024-04-20T21:32:08.153 INFO:tasks.workunit.client.0.smithi062.stdout:2/700: chown d7/da/d15/d2f/d97/lae 0 1 2024-04-20T21:32:08.154 INFO:tasks.workunit.client.1.smithi189.stdout:7/340: dread - d5/d1d/f62 zero size 2024-04-20T21:32:08.156 INFO:tasks.workunit.client.1.smithi189.stdout:1/301: stat d8/d21/d2c/l2e 0 2024-04-20T21:32:08.159 INFO:tasks.workunit.client.0.smithi062.stdout:0/662: dwrite d7/d1c/d15/d17/d26/f68 [0,4194304] 0 2024-04-20T21:32:08.162 INFO:tasks.workunit.client.0.smithi062.stdout:0/663: chown d7/d1c/d15/d17/d40/f85 108818 1 2024-04-20T21:32:08.168 INFO:tasks.workunit.client.1.smithi189.stdout:0/445: mknod d3/df/c93 0 2024-04-20T21:32:08.169 INFO:tasks.workunit.client.1.smithi189.stdout:3/314: symlink d2/d3/l73 0 2024-04-20T21:32:08.171 INFO:tasks.workunit.client.1.smithi189.stdout:7/341: unlink d5/d1d/f44 0 2024-04-20T21:32:08.171 INFO:tasks.workunit.client.0.smithi062.stdout:0/664: write d7/d1c/d15/d17/d26/fa7 [570933,9640] 0 2024-04-20T21:32:08.172 INFO:tasks.workunit.client.1.smithi189.stdout:9/353: dread d9/dc/d17/d36/f4a [0,4194304] 0 2024-04-20T21:32:08.174 INFO:tasks.workunit.client.1.smithi189.stdout:8/437: dwrite da/df/f11 [0,4194304] 0 2024-04-20T21:32:08.174 INFO:tasks.workunit.client.1.smithi189.stdout:6/354: dwrite d8/d12/f44 [0,4194304] 0 2024-04-20T21:32:08.174 INFO:tasks.workunit.client.1.smithi189.stdout:7/342: fsync d5/f43 0 2024-04-20T21:32:08.175 INFO:tasks.workunit.client.0.smithi062.stdout:4/723: mknod da/c100 0 2024-04-20T21:32:08.176 INFO:tasks.workunit.client.0.smithi062.stdout:1/729: dwrite dc/d15/fc4 [0,4194304] 0 2024-04-20T21:32:08.177 INFO:tasks.workunit.client.1.smithi189.stdout:3/315: mkdir d2/d3/d2c/d31/d74 0 2024-04-20T21:32:08.180 INFO:tasks.workunit.client.0.smithi062.stdout:7/685: rmdir d7/d3e/d33/d3d/d4d/d83 39 2024-04-20T21:32:08.183 INFO:tasks.workunit.client.0.smithi062.stdout:9/723: dread d16/d4e/f6f [0,4194304] 0 2024-04-20T21:32:08.184 INFO:tasks.workunit.client.1.smithi189.stdout:2/320: link d1/l21 d1/d17/l5d 0 2024-04-20T21:32:08.184 INFO:tasks.workunit.client.1.smithi189.stdout:4/327: rename d2/d20/f33 to d2/dd/d11/f7e 0 2024-04-20T21:32:08.185 INFO:tasks.workunit.client.0.smithi062.stdout:5/679: mknod d8/d1e/d2c/dd1/ce0 0 2024-04-20T21:32:08.186 INFO:tasks.workunit.client.1.smithi189.stdout:4/328: fsync d2/f5e 0 2024-04-20T21:32:08.187 INFO:tasks.workunit.client.0.smithi062.stdout:9/724: chown d16/d3e/c6b 764 1 2024-04-20T21:32:08.193 INFO:tasks.workunit.client.1.smithi189.stdout:0/446: rmdir d3/df/d28/d2b/d2c 39 2024-04-20T21:32:08.197 INFO:tasks.workunit.client.0.smithi062.stdout:8/674: sync 2024-04-20T21:32:08.203 INFO:tasks.workunit.client.0.smithi062.stdout:8/675: chown d7/d9/lca 2033 1 2024-04-20T21:32:08.204 INFO:tasks.workunit.client.1.smithi189.stdout:5/284: dread d7/d14/d4e/f39 [4194304,4194304] 0 2024-04-20T21:32:08.212 INFO:tasks.workunit.client.1.smithi189.stdout:8/438: dread f3 [4194304,4194304] 0 2024-04-20T21:32:08.212 INFO:tasks.workunit.client.0.smithi062.stdout:4/724: creat da/f101 x:0 0 0 2024-04-20T21:32:08.215 INFO:tasks.workunit.client.1.smithi189.stdout:7/343: dwrite d5/f47 [0,4194304] 0 2024-04-20T21:32:08.217 INFO:tasks.workunit.client.1.smithi189.stdout:4/329: mkdir d2/d7f 0 2024-04-20T21:32:08.218 INFO:tasks.workunit.client.0.smithi062.stdout:0/665: truncate d7/f4c 816219 0 2024-04-20T21:32:08.218 INFO:tasks.workunit.client.0.smithi062.stdout:6/557: chown d0/d9/d11/c19 50580732 1 2024-04-20T21:32:08.220 INFO:tasks.workunit.client.0.smithi062.stdout:1/730: mkdir dc/d15/d75/d96/dd4/d107 0 2024-04-20T21:32:08.221 INFO:tasks.workunit.client.0.smithi062.stdout:9/725: creat d16/d2c/d62/dea/fef x:0 0 0 2024-04-20T21:32:08.223 INFO:tasks.workunit.client.0.smithi062.stdout:3/648: getdents d0/d1/d3c/d43/d72 0 2024-04-20T21:32:08.226 INFO:tasks.workunit.client.1.smithi189.stdout:0/447: rmdir d3/d8c/d87 0 2024-04-20T21:32:08.229 INFO:tasks.workunit.client.0.smithi062.stdout:7/686: getdents d7/d3e/d33 0 2024-04-20T21:32:08.232 INFO:tasks.workunit.client.1.smithi189.stdout:2/321: getdents d1/d17/d2f 0 2024-04-20T21:32:08.232 INFO:tasks.workunit.client.0.smithi062.stdout:0/666: mkdir d7/ddf 0 2024-04-20T21:32:08.234 INFO:tasks.workunit.client.0.smithi062.stdout:1/731: rename dc/c12 to dc/d15/d75/d94/d100/c108 0 2024-04-20T21:32:08.235 INFO:tasks.workunit.client.1.smithi189.stdout:2/322: chown d1/d17/d37/c4c 78081920 1 2024-04-20T21:32:08.237 INFO:tasks.workunit.client.1.smithi189.stdout:4/330: dwrite d2/dd/d11/f67 [0,4194304] 0 2024-04-20T21:32:08.237 INFO:tasks.workunit.client.0.smithi062.stdout:9/726: mkdir d16/d2c/d62/dd1/df0 0 2024-04-20T21:32:08.249 INFO:tasks.workunit.client.0.smithi062.stdout:7/687: mkdir d7/d26/dac/dec 0 2024-04-20T21:32:08.249 INFO:tasks.workunit.client.0.smithi062.stdout:1/732: symlink dc/d15/d75/d96/dd4/d3a/da2/l109 0 2024-04-20T21:32:08.249 INFO:tasks.workunit.client.1.smithi189.stdout:4/331: chown d2/l76 24866592 1 2024-04-20T21:32:08.251 INFO:tasks.workunit.client.1.smithi189.stdout:2/323: rename d1/d11/d2d/l57 to d1/d17/d2f/d55/l5e 0 2024-04-20T21:32:08.251 INFO:tasks.workunit.client.0.smithi062.stdout:0/667: link d7/d20/c6c d7/d20/d3a/ce0 0 2024-04-20T21:32:08.252 INFO:tasks.workunit.client.1.smithi189.stdout:4/332: truncate d2/f2c 2086301 0 2024-04-20T21:32:08.255 INFO:tasks.workunit.client.0.smithi062.stdout:7/688: chown d7/d63/fa7 1 1 2024-04-20T21:32:08.257 INFO:tasks.workunit.client.1.smithi189.stdout:7/344: truncate d5/f43 3177975 0 2024-04-20T21:32:08.259 INFO:tasks.workunit.client.0.smithi062.stdout:8/676: dwrite d7/d9/d29/f4c [4194304,4194304] 0 2024-04-20T21:32:08.260 INFO:tasks.workunit.client.1.smithi189.stdout:3/316: dread d2/f59 [0,4194304] 0 2024-04-20T21:32:08.262 INFO:tasks.workunit.client.1.smithi189.stdout:7/345: write d5/f47 [3593835,51151] 0 2024-04-20T21:32:08.262 INFO:tasks.workunit.client.1.smithi189.stdout:2/324: creat d1/d17/d2f/d55/f5f x:0 0 0 2024-04-20T21:32:08.266 INFO:tasks.workunit.client.0.smithi062.stdout:1/733: mkdir dc/d15/d75/d10a 0 2024-04-20T21:32:08.268 INFO:tasks.workunit.client.1.smithi189.stdout:2/325: stat d1/d11/d26/c22 0 2024-04-20T21:32:08.268 INFO:tasks.workunit.client.1.smithi189.stdout:8/439: link da/de/d1e/d21/d44/f43 da/de/f9f 0 2024-04-20T21:32:08.269 INFO:tasks.workunit.client.1.smithi189.stdout:3/317: write d2/d8/f10 [7413897,70085] 0 2024-04-20T21:32:08.270 INFO:tasks.workunit.client.1.smithi189.stdout:6/355: dwrite d8/d10/f36 [0,4194304] 0 2024-04-20T21:32:08.270 INFO:tasks.workunit.client.1.smithi189.stdout:7/346: dread - d5/d21/d4a/d52/f5c zero size 2024-04-20T21:32:08.270 INFO:tasks.workunit.client.0.smithi062.stdout:9/727: write fb [4299596,79223] 0 2024-04-20T21:32:08.271 INFO:tasks.workunit.client.0.smithi062.stdout:0/668: dread d7/d1c/f3e [0,4194304] 0 2024-04-20T21:32:08.272 INFO:tasks.workunit.client.0.smithi062.stdout:2/701: dwrite d7/da/d15/d2f/d38/f4e [0,4194304] 0 2024-04-20T21:32:08.276 INFO:tasks.workunit.client.1.smithi189.stdout:0/448: creat d3/df/d28/d2b/d2c/d73/f94 x:0 0 0 2024-04-20T21:32:08.277 INFO:tasks.workunit.client.1.smithi189.stdout:6/356: write d8/d2b/f53 [335433,11991] 0 2024-04-20T21:32:08.278 INFO:tasks.workunit.client.1.smithi189.stdout:4/333: symlink d2/l80 0 2024-04-20T21:32:08.280 INFO:tasks.workunit.client.0.smithi062.stdout:5/680: dwrite d8/d10/f4e [0,4194304] 0 2024-04-20T21:32:08.281 INFO:tasks.workunit.client.1.smithi189.stdout:9/354: dwrite d9/d12/d15/f31 [0,4194304] 0 2024-04-20T21:32:08.283 INFO:tasks.workunit.client.1.smithi189.stdout:2/326: dread d1/d11/d2d/f38 [0,4194304] 0 2024-04-20T21:32:08.284 INFO:tasks.workunit.client.0.smithi062.stdout:2/702: truncate d7/fe0 502718 0 2024-04-20T21:32:08.285 INFO:tasks.workunit.client.0.smithi062.stdout:0/669: fdatasync d7/d1e/d6e/f75 0 2024-04-20T21:32:08.289 INFO:tasks.workunit.client.0.smithi062.stdout:3/649: dwrite d0/d1c/d63/d42/f73 [0,4194304] 0 2024-04-20T21:32:08.289 INFO:tasks.workunit.client.0.smithi062.stdout:4/725: dwrite da/dd/df/f41 [0,4194304] 0 2024-04-20T21:32:08.297 INFO:tasks.workunit.client.0.smithi062.stdout:2/703: symlink d7/da/d15/le9 0 2024-04-20T21:32:08.301 INFO:tasks.workunit.client.0.smithi062.stdout:0/670: creat d7/d1c/d15/d17/d40/d3c/da4/fe1 x:0 0 0 2024-04-20T21:32:08.302 INFO:tasks.workunit.client.1.smithi189.stdout:7/347: dread d5/f1a [0,4194304] 0 2024-04-20T21:32:08.307 INFO:tasks.workunit.client.1.smithi189.stdout:3/318: dwrite d2/d3/d2c/d30/f68 [0,4194304] 0 2024-04-20T21:32:08.308 INFO:tasks.workunit.client.0.smithi062.stdout:2/704: write d7/da/d28/d4b/f73 [3131982,114566] 0 2024-04-20T21:32:08.308 INFO:tasks.workunit.client.0.smithi062.stdout:0/671: fdatasync d7/d1e/fdc 0 2024-04-20T21:32:08.309 INFO:tasks.workunit.client.1.smithi189.stdout:9/355: dwrite d9/dc/d17/d36/f57 [0,4194304] 0 2024-04-20T21:32:08.310 INFO:tasks.workunit.client.1.smithi189.stdout:6/357: mkdir d8/da/dd/d35/d6d/d77 0 2024-04-20T21:32:08.311 INFO:tasks.workunit.client.0.smithi062.stdout:3/650: read d0/d1/d3c/d43/d72/dd3/d7b/fb6 [312652,41301] 0 2024-04-20T21:32:08.311 INFO:tasks.workunit.client.0.smithi062.stdout:6/558: dread d0/d2c/d67/fa1 [0,4194304] 0 2024-04-20T21:32:08.314 INFO:tasks.workunit.client.1.smithi189.stdout:3/319: chown d2/d3/d2c/l56 1253949 1 2024-04-20T21:32:08.318 INFO:tasks.workunit.client.1.smithi189.stdout:6/358: mkdir d8/d10/d78 0 2024-04-20T21:32:08.319 INFO:tasks.workunit.client.1.smithi189.stdout:9/356: creat d9/dc/d17/d2b/d56/f73 x:0 0 0 2024-04-20T21:32:08.320 INFO:tasks.workunit.client.0.smithi062.stdout:3/651: chown d0/d1/d10/d31/f92 2652262 1 2024-04-20T21:32:08.324 INFO:tasks.workunit.client.0.smithi062.stdout:4/726: getdents da/dd/df/d27/d3f 0 2024-04-20T21:32:08.325 INFO:tasks.workunit.client.0.smithi062.stdout:0/672: read d7/d1c/d15/d17/fd0 [236375,66764] 0 2024-04-20T21:32:08.326 INFO:tasks.workunit.client.0.smithi062.stdout:2/705: unlink d7/da/d15/d1b/f7d 0 2024-04-20T21:32:08.326 INFO:tasks.workunit.client.1.smithi189.stdout:3/320: rename d2/d3/d25/c44 to d2/d3/d2c/c75 0 2024-04-20T21:32:08.328 INFO:tasks.workunit.client.0.smithi062.stdout:9/728: dwrite d16/f19 [0,4194304] 0 2024-04-20T21:32:08.333 INFO:tasks.workunit.client.0.smithi062.stdout:8/677: dread d7/d25/d3b/d58/f63 [0,4194304] 0 2024-04-20T21:32:08.333 INFO:tasks.workunit.client.1.smithi189.stdout:5/285: sync 2024-04-20T21:32:08.334 INFO:tasks.workunit.client.1.smithi189.stdout:1/302: sync 2024-04-20T21:32:08.334 INFO:tasks.workunit.client.0.smithi062.stdout:6/559: read - d0/d1a/d7d/f96 zero size 2024-04-20T21:32:08.336 INFO:tasks.workunit.client.0.smithi062.stdout:4/727: write da/dd/d15/d6d/df5/ff8 [286468,8544] 0 2024-04-20T21:32:08.338 INFO:tasks.workunit.client.1.smithi189.stdout:5/286: fsync d7/d35/f44 0 2024-04-20T21:32:08.338 INFO:tasks.workunit.client.0.smithi062.stdout:5/681: dwrite d8/d38/d7a/d4c/d7d/d9d/dac/d66/fc2 [0,4194304] 0 2024-04-20T21:32:08.339 INFO:tasks.workunit.client.0.smithi062.stdout:8/678: fdatasync d7/d30/fba 0 2024-04-20T21:32:08.340 INFO:tasks.workunit.client.0.smithi062.stdout:9/729: chown d16/d3e/c6b 15882 1 2024-04-20T21:32:08.345 INFO:tasks.workunit.client.1.smithi189.stdout:2/327: dread d1/f9 [0,4194304] 0 2024-04-20T21:32:08.348 INFO:tasks.workunit.client.1.smithi189.stdout:2/328: chown d1/f5 76779 1 2024-04-20T21:32:08.348 INFO:tasks.workunit.client.1.smithi189.stdout:5/287: write d7/d35/f44 [208930,45427] 0 2024-04-20T21:32:08.352 INFO:tasks.workunit.client.0.smithi062.stdout:2/706: rename d7/da/d28/c59 to d7/da/d28/d4b/d8f/cea 0 2024-04-20T21:32:08.353 INFO:tasks.workunit.client.0.smithi062.stdout:4/728: truncate da/dd/df/d27/d32/fab 267544 0 2024-04-20T21:32:08.356 INFO:tasks.workunit.client.1.smithi189.stdout:9/357: dwrite d9/dc/f4d [0,4194304] 0 2024-04-20T21:32:08.356 INFO:tasks.workunit.client.1.smithi189.stdout:2/329: write d1/d17/d2f/d55/f5f [528096,78309] 0 2024-04-20T21:32:08.356 INFO:tasks.workunit.client.1.smithi189.stdout:5/288: write d7/d35/f44 [1655970,51661] 0 2024-04-20T21:32:08.359 INFO:tasks.workunit.client.0.smithi062.stdout:8/679: readlink d7/d25/d3b/d58/d8b/d24/d46/ld0 0 2024-04-20T21:32:08.363 INFO:tasks.workunit.client.1.smithi189.stdout:4/334: dread d2/f8 [0,4194304] 0 2024-04-20T21:32:08.367 INFO:tasks.workunit.client.0.smithi062.stdout:7/689: dwrite d7/d3e/d33/d3d/d4d/d83/fd7 [0,4194304] 0 2024-04-20T21:32:08.367 INFO:tasks.workunit.client.0.smithi062.stdout:6/560: symlink d0/d9a/lc0 0 2024-04-20T21:32:08.370 INFO:tasks.workunit.client.1.smithi189.stdout:2/330: mknod d1/d11/d2d/d4f/c60 0 2024-04-20T21:32:08.370 INFO:tasks.workunit.client.1.smithi189.stdout:5/289: mknod d7/d14/d1b/c52 0 2024-04-20T21:32:08.379 INFO:tasks.workunit.client.1.smithi189.stdout:9/358: mknod d9/dc/d17/d2b/d52/c74 0 2024-04-20T21:32:08.379 INFO:tasks.workunit.client.1.smithi189.stdout:1/303: dread d8/f12 [0,4194304] 0 2024-04-20T21:32:08.379 INFO:tasks.workunit.client.0.smithi062.stdout:8/680: mknod d7/d25/d3b/d58/ce0 0 2024-04-20T21:32:08.380 INFO:tasks.workunit.client.0.smithi062.stdout:6/561: truncate d0/d52/da0/fa8 797778 0 2024-04-20T21:32:08.380 INFO:tasks.workunit.client.0.smithi062.stdout:9/730: dwrite fe [0,4194304] 0 2024-04-20T21:32:08.384 INFO:tasks.workunit.client.1.smithi189.stdout:2/331: creat d1/d17/d2f/d55/f61 x:0 0 0 2024-04-20T21:32:08.384 INFO:tasks.workunit.client.0.smithi062.stdout:2/707: rename d7/da/d15/d1b/d3d/d84/d9b/cb4 to d7/ceb 0 2024-04-20T21:32:08.385 INFO:tasks.workunit.client.0.smithi062.stdout:8/681: chown d7/d30/d90/cbe 41 1 2024-04-20T21:32:08.387 INFO:tasks.workunit.client.0.smithi062.stdout:9/731: write d16/d2c/d42/da6/fe3 [322139,41609] 0 2024-04-20T21:32:08.388 INFO:tasks.workunit.client.1.smithi189.stdout:5/290: dread d7/d9/d30/f1c [0,4194304] 0 2024-04-20T21:32:08.389 INFO:tasks.workunit.client.0.smithi062.stdout:1/734: dread dc/d15/d75/f91 [0,4194304] 0 2024-04-20T21:32:08.390 INFO:tasks.workunit.client.0.smithi062.stdout:2/708: mknod d7/d39/cec 0 2024-04-20T21:32:08.391 INFO:tasks.workunit.client.0.smithi062.stdout:6/562: symlink d0/d52/lc1 0 2024-04-20T21:32:08.392 INFO:tasks.workunit.client.0.smithi062.stdout:9/732: chown d16/d2c/lc1 12903154 1 2024-04-20T21:32:08.393 INFO:tasks.workunit.client.1.smithi189.stdout:1/304: symlink d8/d1e/d28/d2b/l6b 0 2024-04-20T21:32:08.395 INFO:tasks.workunit.client.1.smithi189.stdout:5/291: truncate d7/d35/f46 353690 0 2024-04-20T21:32:08.397 INFO:tasks.workunit.client.1.smithi189.stdout:1/305: fsync d8/d1e/f3c 0 2024-04-20T21:32:08.397 INFO:tasks.workunit.client.0.smithi062.stdout:8/682: unlink d7/d30/c7f 0 2024-04-20T21:32:08.401 INFO:tasks.workunit.client.0.smithi062.stdout:3/652: dread d0/d1/d10/d31/f9e [0,4194304] 0 2024-04-20T21:32:08.402 INFO:tasks.workunit.client.1.smithi189.stdout:1/306: rmdir d8/d1e/d28/d2b 39 2024-04-20T21:32:08.403 INFO:tasks.workunit.client.1.smithi189.stdout:7/348: dread d5/d21/f32 [0,4194304] 0 2024-04-20T21:32:08.404 INFO:tasks.workunit.client.0.smithi062.stdout:1/735: symlink dc/d15/d75/d96/dd4/d3a/da2/de4/d7c/l10b 0 2024-04-20T21:32:08.404 INFO:tasks.workunit.client.0.smithi062.stdout:8/683: mkdir d7/de1 0 2024-04-20T21:32:08.404 INFO:tasks.workunit.client.1.smithi189.stdout:0/449: dread d3/f5e [0,4194304] 0 2024-04-20T21:32:08.405 INFO:tasks.workunit.client.1.smithi189.stdout:1/307: write d8/f46 [3187596,10813] 0 2024-04-20T21:32:08.409 INFO:tasks.workunit.client.1.smithi189.stdout:5/292: link d7/d35/f46 d7/d14/d18/f53 0 2024-04-20T21:32:08.411 INFO:tasks.workunit.client.0.smithi062.stdout:1/736: write dc/f6d [336439,102657] 0 2024-04-20T21:32:08.412 INFO:tasks.workunit.client.1.smithi189.stdout:3/321: fdatasync d2/d3/d2c/d30/f68 0 2024-04-20T21:32:08.412 INFO:tasks.workunit.client.1.smithi189.stdout:5/293: write d7/d14/d18/f17 [2575368,91727] 0 2024-04-20T21:32:08.413 INFO:tasks.workunit.client.1.smithi189.stdout:3/322: chown d2/d8/f58 1016551 1 2024-04-20T21:32:08.417 INFO:tasks.workunit.client.1.smithi189.stdout:7/349: unlink d5/f24 0 2024-04-20T21:32:08.421 INFO:tasks.workunit.client.0.smithi062.stdout:3/653: dwrite d0/d1b/d74/fbe [0,4194304] 0 2024-04-20T21:32:08.430 INFO:tasks.workunit.client.1.smithi189.stdout:8/440: write da/de/d1e/d21/d44/f47 [565446,90785] 0 2024-04-20T21:32:08.431 INFO:tasks.workunit.client.1.smithi189.stdout:3/323: link d2/d8/f34 d2/d1c/f76 0 2024-04-20T21:32:08.433 INFO:tasks.workunit.client.1.smithi189.stdout:3/324: stat d2/d3/l4 0 2024-04-20T21:32:08.436 INFO:tasks.workunit.client.1.smithi189.stdout:8/441: fdatasync da/df/f11 0 2024-04-20T21:32:08.437 INFO:tasks.workunit.client.1.smithi189.stdout:4/335: dread d2/dd/d11/f31 [0,4194304] 0 2024-04-20T21:32:08.442 INFO:tasks.workunit.client.0.smithi062.stdout:3/654: mkdir d0/d1c/d63/dda 0 2024-04-20T21:32:08.444 INFO:tasks.workunit.client.0.smithi062.stdout:1/737: link dc/d15/d75/d96/dd4/d3a/f59 dc/d15/d20/d28/d83/f10c 0 2024-04-20T21:32:08.446 INFO:tasks.workunit.client.1.smithi189.stdout:0/450: dwrite d3/df/d30/d37/f4b [0,4194304] 0 2024-04-20T21:32:08.447 INFO:tasks.workunit.client.1.smithi189.stdout:3/325: rename d2/d8/c12 to d2/d3/d2c/d31/d74/c77 0 2024-04-20T21:32:08.448 INFO:tasks.workunit.client.1.smithi189.stdout:8/442: fdatasync da/df/d20/d2c/f76 0 2024-04-20T21:32:08.450 INFO:tasks.workunit.client.1.smithi189.stdout:5/294: dwrite d7/d14/d4e/f39 [0,4194304] 0 2024-04-20T21:32:08.450 INFO:tasks.workunit.client.1.smithi189.stdout:4/336: symlink d2/d22/d57/l81 0 2024-04-20T21:32:08.453 INFO:tasks.workunit.client.1.smithi189.stdout:7/350: dwrite d5/d21/d25/f27 [0,4194304] 0 2024-04-20T21:32:08.453 INFO:tasks.workunit.client.1.smithi189.stdout:1/308: dread d8/df/f15 [0,4194304] 0 2024-04-20T21:32:08.453 INFO:tasks.workunit.client.0.smithi062.stdout:4/729: write da/dd/df/fe9 [4403309,88524] 0 2024-04-20T21:32:08.456 INFO:tasks.workunit.client.1.smithi189.stdout:0/451: symlink d3/df/d28/l95 0 2024-04-20T21:32:08.456 INFO:tasks.workunit.client.1.smithi189.stdout:9/359: write d9/dc/d17/d36/f4a [1544894,45642] 0 2024-04-20T21:32:08.457 INFO:tasks.workunit.client.0.smithi062.stdout:1/738: mkdir dc/d15/d75/d96/dd4/d3a/da2/de4/d84/dbe/d10d 0 2024-04-20T21:32:08.462 INFO:tasks.workunit.client.1.smithi189.stdout:7/351: mkdir d5/d21/d4a/d52/d75 0 2024-04-20T21:32:08.463 INFO:tasks.workunit.client.0.smithi062.stdout:4/730: rename da/dd/f69 to da/dd/d15/d6d/df5/f102 0 2024-04-20T21:32:08.463 INFO:tasks.workunit.client.1.smithi189.stdout:7/352: chown d5/l10 104850572 1 2024-04-20T21:32:08.466 INFO:tasks.workunit.client.0.smithi062.stdout:1/739: unlink dc/d15/d75/d94/f67 0 2024-04-20T21:32:08.467 INFO:tasks.workunit.client.0.smithi062.stdout:1/740: stat dc/cb6 0 2024-04-20T21:32:08.469 INFO:tasks.workunit.client.1.smithi189.stdout:7/353: readlink d5/d21/d25/l29 0 2024-04-20T21:32:08.470 INFO:tasks.workunit.client.1.smithi189.stdout:7/354: readlink d5/d21/d35/l3f 0 2024-04-20T21:32:08.471 INFO:tasks.workunit.client.1.smithi189.stdout:1/309: creat d8/d21/d2c/f6c x:0 0 0 2024-04-20T21:32:08.472 INFO:tasks.workunit.client.1.smithi189.stdout:6/359: dread d8/da/f27 [0,4194304] 0 2024-04-20T21:32:08.477 INFO:tasks.workunit.client.0.smithi062.stdout:6/563: write d0/f79 [4682211,127976] 0 2024-04-20T21:32:08.479 INFO:tasks.workunit.client.1.smithi189.stdout:6/360: rename d8/da to d8/da/dd/d2a/d3b/d79 22 2024-04-20T21:32:08.488 INFO:tasks.workunit.client.0.smithi062.stdout:6/564: chown d0/d1a/d7d/c45 13 1 2024-04-20T21:32:08.489 INFO:tasks.workunit.client.0.smithi062.stdout:1/741: rmdir dc/d15/d75/dd0 0 2024-04-20T21:32:08.491 INFO:tasks.workunit.client.1.smithi189.stdout:9/360: dread d9/d12/d15/f31 [0,4194304] 0 2024-04-20T21:32:08.492 INFO:tasks.workunit.client.1.smithi189.stdout:2/332: sync 2024-04-20T21:32:08.493 INFO:tasks.workunit.client.1.smithi189.stdout:0/452: link d3/df/d30/d45/d60/f81 d3/df/d5d/d64/f96 0 2024-04-20T21:32:08.496 INFO:tasks.workunit.client.1.smithi189.stdout:6/361: creat d8/d2b/d49/d54/d67/f7a x:0 0 0 2024-04-20T21:32:08.497 INFO:tasks.workunit.client.0.smithi062.stdout:6/565: readlink d0/d2c/d67/d75/lab 0 2024-04-20T21:32:08.498 INFO:tasks.workunit.client.0.smithi062.stdout:2/709: dread d7/ff [0,4194304] 0 2024-04-20T21:32:08.498 INFO:tasks.workunit.client.0.smithi062.stdout:1/742: truncate dc/d15/d75/ff9 1811384 0 2024-04-20T21:32:08.499 INFO:tasks.workunit.client.1.smithi189.stdout:8/443: dwrite da/df/d20/d27/f38 [0,4194304] 0 2024-04-20T21:32:08.500 INFO:tasks.workunit.client.0.smithi062.stdout:4/731: dwrite da/dd/df/d27/fc9 [0,4194304] 0 2024-04-20T21:32:08.500 INFO:tasks.workunit.client.1.smithi189.stdout:9/361: read d9/f44 [62989,9194] 0 2024-04-20T21:32:08.500 INFO:tasks.workunit.client.1.smithi189.stdout:2/333: symlink d1/d11/d52/l62 0 2024-04-20T21:32:08.502 INFO:tasks.workunit.client.1.smithi189.stdout:7/355: getdents d5 0 2024-04-20T21:32:08.503 INFO:tasks.workunit.client.1.smithi189.stdout:4/337: dwrite d2/d3a/f3c [0,4194304] 0 2024-04-20T21:32:08.507 INFO:tasks.workunit.client.0.smithi062.stdout:0/673: dwrite d7/d20/d3a/f87 [0,4194304] 0 2024-04-20T21:32:08.510 INFO:tasks.workunit.client.1.smithi189.stdout:0/453: truncate d3/df/d4c/f7e 5151073 0 2024-04-20T21:32:08.513 INFO:tasks.workunit.client.1.smithi189.stdout:7/356: creat d5/d1d/d42/d45/f76 x:0 0 0 2024-04-20T21:32:08.516 INFO:tasks.workunit.client.1.smithi189.stdout:3/326: dwrite d2/d3/d2c/f6d [0,4194304] 0 2024-04-20T21:32:08.516 INFO:tasks.workunit.client.1.smithi189.stdout:1/310: dwrite d8/f1a [4194304,4194304] 0 2024-04-20T21:32:08.526 INFO:tasks.workunit.client.1.smithi189.stdout:0/454: fdatasync d3/df/d4c/f7e 0 2024-04-20T21:32:08.527 INFO:tasks.workunit.client.1.smithi189.stdout:2/334: fsync d1/d17/f35 0 2024-04-20T21:32:08.531 INFO:tasks.workunit.client.1.smithi189.stdout:5/295: truncate d7/d9/d31/d36/f38 3203421 0 2024-04-20T21:32:08.533 INFO:tasks.workunit.client.1.smithi189.stdout:1/311: truncate d8/d21/d2c/f6c 949214 0 2024-04-20T21:32:08.533 INFO:tasks.workunit.client.1.smithi189.stdout:8/444: truncate da/de/f28 755887 0 2024-04-20T21:32:08.537 INFO:tasks.workunit.client.1.smithi189.stdout:3/327: creat d2/d3/d65/f78 x:0 0 0 2024-04-20T21:32:08.538 INFO:tasks.workunit.client.1.smithi189.stdout:8/445: read da/df/d3c/d50/f6d [568231,50878] 0 2024-04-20T21:32:08.539 INFO:tasks.workunit.client.1.smithi189.stdout:4/338: creat d2/dd/d11/d14/d69/f82 x:0 0 0 2024-04-20T21:32:08.540 INFO:tasks.workunit.client.0.smithi062.stdout:1/743: dread dc/d15/fc4 [0,4194304] 0 2024-04-20T21:32:08.541 INFO:tasks.workunit.client.1.smithi189.stdout:3/328: fsync d2/d3/d2c/d31/f50 0 2024-04-20T21:32:08.541 INFO:tasks.workunit.client.1.smithi189.stdout:7/357: unlink d5/d21/c60 0 2024-04-20T21:32:08.545 INFO:tasks.workunit.client.1.smithi189.stdout:4/339: read - d2/dd/d6b/f72 zero size 2024-04-20T21:32:08.545 INFO:tasks.workunit.client.1.smithi189.stdout:5/296: creat d7/d9/d31/d36/f54 x:0 0 0 2024-04-20T21:32:08.545 INFO:tasks.workunit.client.0.smithi062.stdout:4/732: dread da/dd/d15/d50/d64/d3d/f40 [0,4194304] 0 2024-04-20T21:32:08.546 INFO:tasks.workunit.client.1.smithi189.stdout:1/312: unlink d8/d21/d2c/d36/c3b 0 2024-04-20T21:32:08.547 INFO:tasks.workunit.client.1.smithi189.stdout:8/446: mknod da/de/d1e/d21/ca0 0 2024-04-20T21:32:08.547 INFO:tasks.workunit.client.1.smithi189.stdout:9/362: dread d9/d12/d3f/f68 [0,4194304] 0 2024-04-20T21:32:08.548 INFO:tasks.workunit.client.0.smithi062.stdout:2/710: write d7/da/d28/d4b/f95 [234417,129760] 0 2024-04-20T21:32:08.548 INFO:tasks.workunit.client.1.smithi189.stdout:0/455: symlink d3/df/l97 0 2024-04-20T21:32:08.549 INFO:tasks.workunit.client.0.smithi062.stdout:6/566: write d0/d9/d11/f63 [3424107,120167] 0 2024-04-20T21:32:08.549 INFO:tasks.workunit.client.0.smithi062.stdout:0/674: write f1 [3798236,115571] 0 2024-04-20T21:32:08.550 INFO:tasks.workunit.client.1.smithi189.stdout:1/313: write f1 [3785165,30386] 0 2024-04-20T21:32:08.550 INFO:tasks.workunit.client.1.smithi189.stdout:3/329: mknod d2/d3/d25/c79 0 2024-04-20T21:32:08.550 INFO:tasks.workunit.client.1.smithi189.stdout:7/358: chown d5/d21/l6e 1021 1 2024-04-20T21:32:08.551 INFO:tasks.workunit.client.1.smithi189.stdout:6/362: truncate d8/da/f27 3717912 0 2024-04-20T21:32:08.551 INFO:tasks.workunit.client.1.smithi189.stdout:5/297: write d7/d9/d30/f2d [3362519,15564] 0 2024-04-20T21:32:08.554 INFO:tasks.workunit.client.0.smithi062.stdout:0/675: fdatasync d7/d1c/d15/d53/fbb 0 2024-04-20T21:32:08.558 INFO:tasks.workunit.client.1.smithi189.stdout:3/330: fdatasync d2/d3/d25/f5b 0 2024-04-20T21:32:08.559 INFO:tasks.workunit.client.1.smithi189.stdout:8/447: creat da/d1f/fa1 x:0 0 0 2024-04-20T21:32:08.561 INFO:tasks.workunit.client.1.smithi189.stdout:1/314: mknod d8/d1e/d28/d4f/c6d 0 2024-04-20T21:32:08.563 INFO:tasks.workunit.client.0.smithi062.stdout:1/744: rmdir dc/d15/d20/d28/d83/dc0 39 2024-04-20T21:32:08.569 INFO:tasks.workunit.client.1.smithi189.stdout:7/359: rename d5/d21/d35/d53 to d5/d1d/d77 0 2024-04-20T21:32:08.572 INFO:tasks.workunit.client.1.smithi189.stdout:3/331: mkdir d2/d3/d2c/d31/d74/d7a 0 2024-04-20T21:32:08.573 INFO:tasks.workunit.client.0.smithi062.stdout:1/745: unlink dc/d15/d20/fc3 0 2024-04-20T21:32:08.573 INFO:tasks.workunit.client.1.smithi189.stdout:0/456: dread d3/df/d30/d45/d60/f68 [0,4194304] 0 2024-04-20T21:32:08.573 INFO:tasks.workunit.client.1.smithi189.stdout:8/448: symlink da/de/d1e/d69/la2 0 2024-04-20T21:32:08.573 INFO:tasks.workunit.client.1.smithi189.stdout:2/335: dwrite d1/d17/d37/f44 [0,4194304] 0 2024-04-20T21:32:08.578 INFO:tasks.workunit.client.0.smithi062.stdout:0/676: dwrite d7/d1c/d15/d17/d37/f59 [0,4194304] 0 2024-04-20T21:32:08.578 INFO:tasks.workunit.client.0.smithi062.stdout:6/567: creat d0/d2c/d67/daa/fc2 x:0 0 0 2024-04-20T21:32:08.580 INFO:tasks.workunit.client.1.smithi189.stdout:7/360: write d5/f1a [187223,96686] 0 2024-04-20T21:32:08.580 INFO:tasks.workunit.client.1.smithi189.stdout:2/336: dread - d1/d11/d26/f5a zero size 2024-04-20T21:32:08.581 INFO:tasks.workunit.client.0.smithi062.stdout:1/746: creat dc/d27/d9a/da0/db3/f10e x:0 0 0 2024-04-20T21:32:08.581 INFO:tasks.workunit.client.1.smithi189.stdout:9/363: getdents d9/d12/d15/d1a 0 2024-04-20T21:32:08.584 INFO:tasks.workunit.client.0.smithi062.stdout:6/568: stat d0/d9/d11/d22/f27 0 2024-04-20T21:32:08.592 INFO:tasks.workunit.client.1.smithi189.stdout:8/449: mkdir da/de/d1e/d69/da3 0 2024-04-20T21:32:08.592 INFO:tasks.workunit.client.0.smithi062.stdout:1/747: creat dc/d15/d20/f10f x:0 0 0 2024-04-20T21:32:08.597 INFO:tasks.workunit.client.0.smithi062.stdout:1/748: truncate fb 2651611 0 2024-04-20T21:32:08.599 INFO:tasks.workunit.client.1.smithi189.stdout:9/364: getdents d9/dc/d17/d2b 0 2024-04-20T21:32:08.601 INFO:tasks.workunit.client.0.smithi062.stdout:4/733: fsync da/dd/df/ff4 0 2024-04-20T21:32:08.602 INFO:tasks.workunit.client.1.smithi189.stdout:6/363: dwrite f5 [8388608,4194304] 0 2024-04-20T21:32:08.605 INFO:tasks.workunit.client.1.smithi189.stdout:7/361: chown d5/d46 182 1 2024-04-20T21:32:08.606 INFO:tasks.workunit.client.0.smithi062.stdout:0/677: dwrite d7/d1c/d15/d17/d37/f61 [0,4194304] 0 2024-04-20T21:32:08.607 INFO:tasks.workunit.client.1.smithi189.stdout:1/315: dread d8/f24 [0,4194304] 0 2024-04-20T21:32:08.609 INFO:tasks.workunit.client.1.smithi189.stdout:3/332: dread d2/f5a [0,4194304] 0 2024-04-20T21:32:08.611 INFO:tasks.workunit.client.1.smithi189.stdout:4/340: dwrite d2/fb [0,4194304] 0 2024-04-20T21:32:08.612 INFO:tasks.workunit.client.0.smithi062.stdout:6/569: dwrite d0/d9/d11/f8d [0,4194304] 0 2024-04-20T21:32:08.612 INFO:tasks.workunit.client.1.smithi189.stdout:0/457: dwrite d3/df/d5d/f7b [0,4194304] 0 2024-04-20T21:32:08.613 INFO:tasks.workunit.client.0.smithi062.stdout:4/734: dread da/dd/d15/d50/d64/d57/d5b/f68 [0,4194304] 0 2024-04-20T21:32:08.613 INFO:tasks.workunit.client.1.smithi189.stdout:2/337: write d1/f2a [1856341,118005] 0 2024-04-20T21:32:08.613 INFO:tasks.workunit.client.0.smithi062.stdout:0/678: fsync d7/d1c/d15/d17/d37/d95/fd6 0 2024-04-20T21:32:08.614 INFO:tasks.workunit.client.1.smithi189.stdout:5/298: write d7/d9/d31/d36/f38 [527739,112173] 0 2024-04-20T21:32:08.616 INFO:tasks.workunit.client.0.smithi062.stdout:0/679: stat d7/d1c 0 2024-04-20T21:32:08.616 INFO:tasks.workunit.client.1.smithi189.stdout:5/299: read d7/f2b [1660917,53556] 0 2024-04-20T21:32:08.621 INFO:tasks.workunit.client.0.smithi062.stdout:6/570: truncate d0/d1a/f85 1045806 0 2024-04-20T21:32:08.623 INFO:tasks.workunit.client.1.smithi189.stdout:9/365: rename d9/dc/d17/d36/f4a to d9/dc/d17/d2b/d52/f75 0 2024-04-20T21:32:08.627 INFO:tasks.workunit.client.0.smithi062.stdout:0/680: stat d7/d1c/d15/d17/d37/fbf 0 2024-04-20T21:32:08.628 INFO:tasks.workunit.client.1.smithi189.stdout:1/316: creat d8/d1e/f6e x:0 0 0 2024-04-20T21:32:08.629 INFO:tasks.workunit.client.0.smithi062.stdout:7/690: sync 2024-04-20T21:32:08.629 INFO:tasks.workunit.client.0.smithi062.stdout:6/571: dread - d0/d52/fb0 zero size 2024-04-20T21:32:08.630 INFO:tasks.workunit.client.1.smithi189.stdout:3/333: creat d2/d1c/f7b x:0 0 0 2024-04-20T21:32:08.631 INFO:tasks.workunit.client.1.smithi189.stdout:2/338: creat d1/d11/d2d/f63 x:0 0 0 2024-04-20T21:32:08.631 INFO:tasks.workunit.client.1.smithi189.stdout:9/366: fsync d9/dc/d17/f2f 0 2024-04-20T21:32:08.631 INFO:tasks.workunit.client.1.smithi189.stdout:4/341: creat d2/d22/d57/f83 x:0 0 0 2024-04-20T21:32:08.634 INFO:tasks.workunit.client.1.smithi189.stdout:1/317: fsync d8/d21/d2c/d36/f52 0 2024-04-20T21:32:08.635 INFO:tasks.workunit.client.0.smithi062.stdout:0/681: rmdir d7/d1c 39 2024-04-20T21:32:08.641 INFO:tasks.workunit.client.0.smithi062.stdout:1/749: dwrite dc/d15/d75/d96/dd4/d3a/f4d [0,4194304] 0 2024-04-20T21:32:08.642 INFO:tasks.workunit.client.1.smithi189.stdout:0/458: creat d3/df/d30/d45/f98 x:0 0 0 2024-04-20T21:32:08.647 INFO:tasks.workunit.client.1.smithi189.stdout:8/450: dread da/de/d1e/d21/d44/f47 [0,4194304] 0 2024-04-20T21:32:08.649 INFO:tasks.workunit.client.1.smithi189.stdout:8/451: readlink l2 0 2024-04-20T21:32:08.652 INFO:tasks.workunit.client.0.smithi062.stdout:0/682: chown d7/d1c/f63 1700 1 2024-04-20T21:32:08.653 INFO:tasks.workunit.client.1.smithi189.stdout:4/342: mknod d2/d64/c84 0 2024-04-20T21:32:08.657 INFO:tasks.workunit.client.1.smithi189.stdout:3/334: dwrite d2/f2a [8388608,4194304] 0 2024-04-20T21:32:08.659 INFO:tasks.workunit.client.1.smithi189.stdout:0/459: rename d3/df/d30/d37/f22 to d3/df/d5d/f99 0 2024-04-20T21:32:08.662 INFO:tasks.workunit.client.1.smithi189.stdout:2/339: creat d1/d11/d58/f64 x:0 0 0 2024-04-20T21:32:08.663 INFO:tasks.workunit.client.0.smithi062.stdout:4/735: dwrite da/dd/d15/d4b/f90 [0,4194304] 0 2024-04-20T21:32:08.665 INFO:tasks.workunit.client.1.smithi189.stdout:7/362: dwrite d5/d1d/f26 [0,4194304] 0 2024-04-20T21:32:08.665 INFO:tasks.workunit.client.0.smithi062.stdout:0/683: getdents d7/d1e/da9/dbe 0 2024-04-20T21:32:08.665 INFO:tasks.workunit.client.1.smithi189.stdout:8/452: creat da/df/d13/d63/fa4 x:0 0 0 2024-04-20T21:32:08.666 INFO:tasks.workunit.client.1.smithi189.stdout:8/453: stat da/de 0 2024-04-20T21:32:08.668 INFO:tasks.workunit.client.1.smithi189.stdout:5/300: dwrite d7/d14/f49 [0,4194304] 0 2024-04-20T21:32:08.671 INFO:tasks.workunit.client.0.smithi062.stdout:4/736: readlink da/dd/d15/d50/d64/la8 0 2024-04-20T21:32:08.672 INFO:tasks.workunit.client.1.smithi189.stdout:5/301: write d7/d14/d4e/f39 [9085987,39450] 0 2024-04-20T21:32:08.676 INFO:tasks.workunit.client.1.smithi189.stdout:7/363: write d5/d1d/d77/d5b/f2e [1155700,2465] 0 2024-04-20T21:32:08.678 INFO:tasks.workunit.client.0.smithi062.stdout:4/737: mknod da/dd/d15/c103 0 2024-04-20T21:32:08.680 INFO:tasks.workunit.client.0.smithi062.stdout:0/684: rmdir d7/d1c/d15/d6b/ddd 0 2024-04-20T21:32:08.680 INFO:tasks.workunit.client.1.smithi189.stdout:4/343: rename d2/dd/d11/d14/l34 to d2/dd/d6b/l85 0 2024-04-20T21:32:08.680 INFO:tasks.workunit.client.1.smithi189.stdout:2/340: symlink d1/d17/l65 0 2024-04-20T21:32:08.681 INFO:tasks.workunit.client.1.smithi189.stdout:1/318: truncate d8/fa 583096 0 2024-04-20T21:32:08.682 INFO:tasks.workunit.client.1.smithi189.stdout:0/460: unlink d3/d26/l8a 0 2024-04-20T21:32:08.684 INFO:tasks.workunit.client.0.smithi062.stdout:0/685: dread - d7/d1c/d15/fca zero size 2024-04-20T21:32:08.685 INFO:tasks.workunit.client.1.smithi189.stdout:8/454: mknod da/df/d3c/d68/d8b/d8d/ca5 0 2024-04-20T21:32:08.690 INFO:tasks.workunit.client.0.smithi062.stdout:0/686: truncate d7/d1c/f3e 1211294 0 2024-04-20T21:32:08.691 INFO:tasks.workunit.client.1.smithi189.stdout:7/364: creat d5/d21/d25/f78 x:0 0 0 2024-04-20T21:32:08.691 INFO:tasks.workunit.client.1.smithi189.stdout:5/302: fdatasync d7/d14/d4e/f39 0 2024-04-20T21:32:08.693 INFO:tasks.workunit.client.1.smithi189.stdout:5/303: stat d7/d14/l41 0 2024-04-20T21:32:08.695 INFO:tasks.workunit.client.1.smithi189.stdout:4/344: symlink d2/d22/l86 0 2024-04-20T21:32:08.698 INFO:tasks.workunit.client.0.smithi062.stdout:0/687: symlink d7/d1c/le2 0 2024-04-20T21:32:08.699 INFO:tasks.workunit.client.1.smithi189.stdout:0/461: symlink d3/df/d30/d37/l9a 0 2024-04-20T21:32:08.699 INFO:tasks.workunit.client.0.smithi062.stdout:7/691: dread d7/d3e/f7b [0,4194304] 0 2024-04-20T21:32:08.703 INFO:tasks.workunit.client.1.smithi189.stdout:2/341: dread d1/f2a [0,4194304] 0 2024-04-20T21:32:08.707 INFO:tasks.workunit.client.0.smithi062.stdout:7/692: dread d7/d3e/d33/d3d/d4d/d54/dea/d80/f8c [0,4194304] 0 2024-04-20T21:32:08.708 INFO:tasks.workunit.client.1.smithi189.stdout:1/319: mknod d8/d1e/d43/c6f 0 2024-04-20T21:32:08.709 INFO:tasks.workunit.client.0.smithi062.stdout:7/693: stat d7/d3e/d33/d3d/d4d/d54/dea/fb0 0 2024-04-20T21:32:08.710 INFO:tasks.workunit.client.1.smithi189.stdout:2/342: read d1/d17/d37/f44 [821696,54288] 0 2024-04-20T21:32:08.711 INFO:tasks.workunit.client.0.smithi062.stdout:6/572: dread d0/d9/d11/f48 [0,4194304] 0 2024-04-20T21:32:08.712 INFO:tasks.workunit.client.1.smithi189.stdout:1/320: write f1 [6303463,12026] 0 2024-04-20T21:32:08.716 INFO:tasks.workunit.client.0.smithi062.stdout:6/573: write d0/d9/d11/d7b/fb2 [473238,50352] 0 2024-04-20T21:32:08.717 INFO:tasks.workunit.client.1.smithi189.stdout:6/364: truncate d8/d2b/f3a 2338943 0 2024-04-20T21:32:08.721 INFO:tasks.workunit.client.0.smithi062.stdout:6/574: fdatasync d0/d9/f71 0 2024-04-20T21:32:08.724 INFO:tasks.workunit.client.1.smithi189.stdout:6/365: write d8/d12/d30/d47/f4c [3469039,40102] 0 2024-04-20T21:32:08.727 INFO:tasks.workunit.client.1.smithi189.stdout:4/345: dwrite d2/fb [0,4194304] 0 2024-04-20T21:32:08.728 INFO:tasks.workunit.client.1.smithi189.stdout:2/343: mkdir d1/d17/d2f/d4a/d66 0 2024-04-20T21:32:08.730 INFO:tasks.workunit.client.1.smithi189.stdout:0/462: creat d3/df/f9b x:0 0 0 2024-04-20T21:32:08.730 INFO:tasks.workunit.client.0.smithi062.stdout:0/688: dwrite d7/d1c/d15/d17/fad [0,4194304] 0 2024-04-20T21:32:08.731 INFO:tasks.workunit.client.0.smithi062.stdout:6/575: write d0/d9/fbd [5179485,104612] 0 2024-04-20T21:32:08.732 INFO:tasks.workunit.client.1.smithi189.stdout:1/321: symlink d8/d1e/d28/l70 0 2024-04-20T21:32:08.734 INFO:tasks.workunit.client.1.smithi189.stdout:1/322: chown d8/d1e/d43/d44 247 1 2024-04-20T21:32:08.734 INFO:tasks.workunit.client.1.smithi189.stdout:5/304: dwrite d7/d9/d30/f1c [0,4194304] 0 2024-04-20T21:32:08.738 INFO:tasks.workunit.client.1.smithi189.stdout:1/323: write d8/df/f13 [945513,62729] 0 2024-04-20T21:32:08.741 INFO:tasks.workunit.client.1.smithi189.stdout:9/367: dread d9/d12/d3b/f4c [0,4194304] 0 2024-04-20T21:32:08.741 INFO:tasks.workunit.client.0.smithi062.stdout:4/738: dread da/dd/d15/d50/d64/d3d/f7e [0,4194304] 0 2024-04-20T21:32:08.741 INFO:tasks.workunit.client.1.smithi189.stdout:4/346: rename d2/d36 to d2/d22/d87 0 2024-04-20T21:32:08.743 INFO:tasks.workunit.client.0.smithi062.stdout:0/689: mkdir d7/d20/d62/de3 0 2024-04-20T21:32:08.743 INFO:tasks.workunit.client.1.smithi189.stdout:6/366: mkdir d8/da/dd/d35/d6d/d7b 0 2024-04-20T21:32:08.746 INFO:tasks.workunit.client.1.smithi189.stdout:9/368: write d9/dc/f2c [395923,4719] 0 2024-04-20T21:32:08.748 INFO:tasks.workunit.client.0.smithi062.stdout:0/690: fdatasync d7/d1c/d15/d17/d1b/d9d/fa0 0 2024-04-20T21:32:08.750 INFO:tasks.workunit.client.0.smithi062.stdout:6/576: rename d0/d9/d11/c19 to d0/d1a/cc3 0 2024-04-20T21:32:08.756 INFO:tasks.workunit.client.1.smithi189.stdout:7/365: getdents d5 0 2024-04-20T21:32:08.761 INFO:tasks.workunit.client.1.smithi189.stdout:6/367: readlink d8/da/dd/l33 0 2024-04-20T21:32:08.761 INFO:tasks.workunit.client.1.smithi189.stdout:9/369: chown d9/dc/d17/f2f 10558886 1 2024-04-20T21:32:08.761 INFO:tasks.workunit.client.0.smithi062.stdout:3/655: sync 2024-04-20T21:32:08.761 INFO:tasks.workunit.client.0.smithi062.stdout:8/684: sync 2024-04-20T21:32:08.761 INFO:tasks.workunit.client.0.smithi062.stdout:9/733: sync 2024-04-20T21:32:08.761 INFO:tasks.workunit.client.0.smithi062.stdout:5/682: sync 2024-04-20T21:32:08.762 INFO:tasks.workunit.client.0.smithi062.stdout:6/577: write d0/d9/d11/f48 [4975607,29318] 0 2024-04-20T21:32:08.762 INFO:tasks.workunit.client.1.smithi189.stdout:6/368: chown f2 250338901 1 2024-04-20T21:32:08.765 INFO:tasks.workunit.client.0.smithi062.stdout:3/656: write d0/d1c/d63/d42/f60 [4175420,33176] 0 2024-04-20T21:32:08.768 INFO:tasks.workunit.client.0.smithi062.stdout:0/691: symlink d7/d1c/d15/d17/d40/d3c/dc5/le4 0 2024-04-20T21:32:08.772 INFO:tasks.workunit.client.1.smithi189.stdout:1/324: dread d8/d1e/f66 [0,4194304] 0 2024-04-20T21:32:08.774 INFO:tasks.workunit.client.0.smithi062.stdout:9/734: symlink d16/d2c/d62/lf1 0 2024-04-20T21:32:08.775 INFO:tasks.workunit.client.0.smithi062.stdout:6/578: creat d0/d9/d11/d7b/fc4 x:0 0 0 2024-04-20T21:32:08.778 INFO:tasks.workunit.client.1.smithi189.stdout:4/347: dwrite d2/dd/d6b/f72 [0,4194304] 0 2024-04-20T21:32:08.779 INFO:tasks.workunit.client.0.smithi062.stdout:7/694: dwrite d7/d3e/d33/d3d/d4d/d83/f81 [0,4194304] 0 2024-04-20T21:32:08.779 INFO:tasks.workunit.client.0.smithi062.stdout:5/683: symlink d8/d38/d7a/d4c/d7d/d9d/dac/d66/dae/le1 0 2024-04-20T21:32:08.780 INFO:tasks.workunit.client.1.smithi189.stdout:3/335: dwrite d2/fd [0,4194304] 0 2024-04-20T21:32:08.780 INFO:tasks.workunit.client.0.smithi062.stdout:9/735: write d16/d2f/d5b/fa0 [2155430,105498] 0 2024-04-20T21:32:08.781 INFO:tasks.workunit.client.0.smithi062.stdout:2/711: sync 2024-04-20T21:32:08.781 INFO:tasks.workunit.client.0.smithi062.stdout:1/750: sync 2024-04-20T21:32:08.781 INFO:tasks.workunit.client.1.smithi189.stdout:4/348: dread - d2/d3a/f63 zero size 2024-04-20T21:32:08.783 INFO:tasks.workunit.client.1.smithi189.stdout:4/349: readlink d2/dd/d11/l2a 0 2024-04-20T21:32:08.784 INFO:tasks.workunit.client.0.smithi062.stdout:4/739: getdents da/dd/df/d27/da5/df1 0 2024-04-20T21:32:08.787 INFO:tasks.workunit.client.0.smithi062.stdout:9/736: chown d16/d2c/d62/dd1/df0 0 1 2024-04-20T21:32:08.791 INFO:tasks.workunit.client.0.smithi062.stdout:5/684: write d8/d38/d7a/d4c/d7d/fde [137862,4725] 0 2024-04-20T21:32:08.792 INFO:tasks.workunit.client.0.smithi062.stdout:3/657: mkdir d0/d1/d10/d3a/d5e/ddb 0 2024-04-20T21:32:08.792 INFO:tasks.workunit.client.0.smithi062.stdout:6/579: read - d0/d2c/d67/f6a zero size 2024-04-20T21:32:08.792 INFO:tasks.workunit.client.1.smithi189.stdout:9/370: read d9/d12/f29 [662026,50056] 0 2024-04-20T21:32:08.793 INFO:tasks.workunit.client.1.smithi189.stdout:7/366: creat d5/d1d/f79 x:0 0 0 2024-04-20T21:32:08.793 INFO:tasks.workunit.client.0.smithi062.stdout:8/685: symlink d7/d25/d3b/d58/d8b/d24/le2 0 2024-04-20T21:32:08.794 INFO:tasks.workunit.client.0.smithi062.stdout:7/695: rmdir d7/d3e/d33/d3d/d5e 39 2024-04-20T21:32:08.795 INFO:tasks.workunit.client.0.smithi062.stdout:1/751: stat dc/d15/d20/d28/d83/ca3 0 2024-04-20T21:32:08.796 INFO:tasks.workunit.client.1.smithi189.stdout:3/336: unlink d2/d1c/f28 0 2024-04-20T21:32:08.796 INFO:tasks.workunit.client.0.smithi062.stdout:4/740: dread - da/dd/d15/daf/fda zero size 2024-04-20T21:32:08.797 INFO:tasks.workunit.client.1.smithi189.stdout:5/305: write d7/d35/f46 [1223545,14462] 0 2024-04-20T21:32:08.805 INFO:tasks.workunit.client.0.smithi062.stdout:1/752: write dc/d15/d75/d96/dd4/f5f [2761967,15505] 0 2024-04-20T21:32:08.810 INFO:tasks.workunit.client.1.smithi189.stdout:1/325: symlink d8/d1e/d28/l71 0 2024-04-20T21:32:08.810 INFO:tasks.workunit.client.1.smithi189.stdout:6/369: getdents d8/da/dd/d35/d6d/d7b 0 2024-04-20T21:32:08.811 INFO:tasks.workunit.client.0.smithi062.stdout:8/686: creat d7/d30/db3/dbf/fe3 x:0 0 0 2024-04-20T21:32:08.815 INFO:tasks.workunit.client.0.smithi062.stdout:1/753: readlink dc/d15/d75/d96/dd4/d3a/led 0 2024-04-20T21:32:08.817 INFO:tasks.workunit.client.1.smithi189.stdout:9/371: creat d9/dc/d69/f76 x:0 0 0 2024-04-20T21:32:08.817 INFO:tasks.workunit.client.1.smithi189.stdout:1/326: readlink d8/d1e/l2f 0 2024-04-20T21:32:08.817 INFO:tasks.workunit.client.1.smithi189.stdout:8/455: dread da/de/d1e/d21/d44/f43 [0,4194304] 0 2024-04-20T21:32:08.818 INFO:tasks.workunit.client.1.smithi189.stdout:4/350: dwrite d2/d3a/f3c [4194304,4194304] 0 2024-04-20T21:32:08.823 INFO:tasks.workunit.client.0.smithi062.stdout:2/712: dread d7/da/d15/d2f/d38/f4e [0,4194304] 0 2024-04-20T21:32:08.824 INFO:tasks.workunit.client.1.smithi189.stdout:0/463: dread d3/df/f91 [0,4194304] 0 2024-04-20T21:32:08.824 INFO:tasks.workunit.client.1.smithi189.stdout:1/327: stat d8/f14 0 2024-04-20T21:32:08.824 INFO:tasks.workunit.client.1.smithi189.stdout:9/372: read d9/d12/d15/f31 [3948211,72761] 0 2024-04-20T21:32:08.825 INFO:tasks.workunit.client.1.smithi189.stdout:6/370: symlink d8/d12/l7c 0 2024-04-20T21:32:08.825 INFO:tasks.workunit.client.1.smithi189.stdout:2/344: dwrite d1/d11/d2d/f41 [0,4194304] 0 2024-04-20T21:32:08.826 INFO:tasks.workunit.client.0.smithi062.stdout:9/737: rename d16/d3d/ldc to d16/d2c/d42/da6/lf2 0 2024-04-20T21:32:08.827 INFO:tasks.workunit.client.0.smithi062.stdout:3/658: dwrite d0/d1/d3c/d43/d72/da9/fab [0,4194304] 0 2024-04-20T21:32:08.830 INFO:tasks.workunit.client.0.smithi062.stdout:9/738: write fb [1264879,76255] 0 2024-04-20T21:32:08.830 INFO:tasks.workunit.client.1.smithi189.stdout:1/328: chown d8/f11 5941296 1 2024-04-20T21:32:08.833 INFO:tasks.workunit.client.1.smithi189.stdout:8/456: readlink da/d30/l4f 0 2024-04-20T21:32:08.836 INFO:tasks.workunit.client.1.smithi189.stdout:4/351: read - d2/dd/d11/d14/d69/f82 zero size 2024-04-20T21:32:08.837 INFO:tasks.workunit.client.0.smithi062.stdout:4/741: dread da/dd/df/fc7 [0,4194304] 0 2024-04-20T21:32:08.841 INFO:tasks.workunit.client.1.smithi189.stdout:6/371: readlink d8/da/dd/d2a/d3b/l43 0 2024-04-20T21:32:08.842 INFO:tasks.workunit.client.0.smithi062.stdout:5/685: rename d8/d38/d7a/d4c/d7d/d9d/dac/d2f/c33 to d8/d38/d7a/d4c/d7d/ce2 0 2024-04-20T21:32:08.846 INFO:tasks.workunit.client.1.smithi189.stdout:1/329: write d8/d21/d2c/f62 [675245,98782] 0 2024-04-20T21:32:08.846 INFO:tasks.workunit.client.1.smithi189.stdout:2/345: truncate d1/d11/d58/f64 466906 0 2024-04-20T21:32:08.847 INFO:tasks.workunit.client.1.smithi189.stdout:9/373: chown d9/d12/d3b/c6e 509 1 2024-04-20T21:32:08.849 INFO:tasks.workunit.client.1.smithi189.stdout:4/352: write d2/d22/d57/f83 [343520,53971] 0 2024-04-20T21:32:08.850 INFO:tasks.workunit.client.0.smithi062.stdout:0/692: dread d7/d1e/d6e/f7d [0,4194304] 0 2024-04-20T21:32:08.851 INFO:tasks.workunit.client.0.smithi062.stdout:2/713: truncate d7/da/d15/f23 852389 0 2024-04-20T21:32:08.852 INFO:tasks.workunit.client.1.smithi189.stdout:2/346: readlink d1/d17/d37/l3b 0 2024-04-20T21:32:08.852 INFO:tasks.workunit.client.1.smithi189.stdout:5/306: dwrite d7/d35/f44 [0,4194304] 0 2024-04-20T21:32:08.853 INFO:tasks.workunit.client.1.smithi189.stdout:8/457: chown da/de/d1e/d21/d44/f47 3 1 2024-04-20T21:32:08.854 INFO:tasks.workunit.client.1.smithi189.stdout:1/330: write d8/d21/d2c/f6c [1726982,44488] 0 2024-04-20T21:32:08.855 INFO:tasks.workunit.client.0.smithi062.stdout:0/693: fdatasync d7/d1c/d15/d53/fbb 0 2024-04-20T21:32:08.857 INFO:tasks.workunit.client.0.smithi062.stdout:7/696: link d7/d3e/l21 d7/d3e/d33/d3d/d4d/d54/dea/dcc/led 0 2024-04-20T21:32:08.858 INFO:tasks.workunit.client.1.smithi189.stdout:5/307: fdatasync d7/d14/d18/f3e 0 2024-04-20T21:32:08.858 INFO:tasks.workunit.client.1.smithi189.stdout:9/374: mknod d9/d12/d3f/c77 0 2024-04-20T21:32:08.860 INFO:tasks.workunit.client.0.smithi062.stdout:1/754: mkdir dc/d15/d75/d96/dd4/d3a/da2/de4/d84/dbe/d10d/d110 0 2024-04-20T21:32:08.863 INFO:tasks.workunit.client.1.smithi189.stdout:6/372: truncate d8/da/dd/d2a/d3b/f68 1036854 0 2024-04-20T21:32:08.865 INFO:tasks.workunit.client.1.smithi189.stdout:2/347: creat d1/d17/d37/f67 x:0 0 0 2024-04-20T21:32:08.865 INFO:tasks.workunit.client.1.smithi189.stdout:1/331: write d8/d21/d2c/f6c [682267,3719] 0 2024-04-20T21:32:08.867 INFO:tasks.workunit.client.1.smithi189.stdout:7/367: dwrite d5/d21/d25/f30 [0,4194304] 0 2024-04-20T21:32:08.867 INFO:tasks.workunit.client.1.smithi189.stdout:8/458: write da/df/d3c/d50/f66 [946818,53024] 0 2024-04-20T21:32:08.868 INFO:tasks.workunit.client.0.smithi062.stdout:1/755: readlink dc/d15/d75/d96/dd4/d3a/le2 0 2024-04-20T21:32:08.869 INFO:tasks.workunit.client.0.smithi062.stdout:4/742: dwrite da/dd/d15/d50/f86 [0,4194304] 0 2024-04-20T21:32:08.873 INFO:tasks.workunit.client.1.smithi189.stdout:2/348: dread - d1/d17/d2f/d55/f61 zero size 2024-04-20T21:32:08.873 INFO:tasks.workunit.client.0.smithi062.stdout:9/739: rename d16/d2c/d5e/d75/d76/dd9/le8 to d16/d2c/d62/dea/lf3 0 2024-04-20T21:32:08.874 INFO:tasks.workunit.client.1.smithi189.stdout:5/308: rename d7/d14/d1b/d1f to d7/d9/d30/d55 0 2024-04-20T21:32:08.875 INFO:tasks.workunit.client.1.smithi189.stdout:1/332: chown d8/d1e/d43/d58/d60/f64 305991 1 2024-04-20T21:32:08.877 INFO:tasks.workunit.client.0.smithi062.stdout:6/580: dwrite d0/f80 [0,4194304] 0 2024-04-20T21:32:08.878 INFO:tasks.workunit.client.1.smithi189.stdout:2/349: readlink d1/d11/d52/l62 0 2024-04-20T21:32:08.879 INFO:tasks.workunit.client.1.smithi189.stdout:4/353: dread d2/f21 [0,4194304] 0 2024-04-20T21:32:08.881 INFO:tasks.workunit.client.1.smithi189.stdout:1/333: creat d8/d1e/d43/d58/f72 x:0 0 0 2024-04-20T21:32:08.883 INFO:tasks.workunit.client.1.smithi189.stdout:5/309: getdents d7/d9/d51 0 2024-04-20T21:32:08.885 INFO:tasks.workunit.client.1.smithi189.stdout:9/375: creat d9/dc/d17/d36/f78 x:0 0 0 2024-04-20T21:32:08.887 INFO:tasks.workunit.client.0.smithi062.stdout:5/686: symlink d8/d10/d3e/le3 0 2024-04-20T21:32:08.889 INFO:tasks.workunit.client.1.smithi189.stdout:1/334: write d8/d1e/d28/d2b/f48 [1207079,99183] 0 2024-04-20T21:32:08.890 INFO:tasks.workunit.client.0.smithi062.stdout:8/687: write d7/f57 [530867,89274] 0 2024-04-20T21:32:08.891 INFO:tasks.workunit.client.1.smithi189.stdout:4/354: rename d2/d22/l71 to d2/dd/d37/l88 0 2024-04-20T21:32:08.892 INFO:tasks.workunit.client.1.smithi189.stdout:9/376: chown d9/dc/d17/l18 0 1 2024-04-20T21:32:08.895 INFO:tasks.workunit.client.0.smithi062.stdout:2/714: write d7/ff [4418702,115303] 0 2024-04-20T21:32:08.895 INFO:tasks.workunit.client.0.smithi062.stdout:1/756: symlink dc/d15/df6/l111 0 2024-04-20T21:32:08.897 INFO:tasks.workunit.client.1.smithi189.stdout:4/355: chown d2/dd/d11/d14/f7d 28 1 2024-04-20T21:32:08.898 INFO:tasks.workunit.client.1.smithi189.stdout:1/335: unlink d8/d1e/d43/d44/l56 0 2024-04-20T21:32:08.899 INFO:tasks.workunit.client.1.smithi189.stdout:4/356: chown d2/dd/l5c 17817 1 2024-04-20T21:32:08.901 INFO:tasks.workunit.client.1.smithi189.stdout:4/357: fdatasync d2/f6d 0 2024-04-20T21:32:08.901 INFO:tasks.workunit.client.1.smithi189.stdout:2/350: rename d1/d17/d37/l3b to d1/d17/d2f/d4a/d66/l68 0 2024-04-20T21:32:08.902 INFO:tasks.workunit.client.0.smithi062.stdout:4/743: unlink da/dd/d15/d50/f86 0 2024-04-20T21:32:08.903 INFO:tasks.workunit.client.0.smithi062.stdout:3/659: dread d0/f4c [0,4194304] 0 2024-04-20T21:32:08.905 INFO:tasks.workunit.client.0.smithi062.stdout:2/715: fsync d7/da/d15/d2f/d38/d6a/d7c/f96 0 2024-04-20T21:32:08.905 INFO:tasks.workunit.client.1.smithi189.stdout:0/464: dread d3/fe [0,4194304] 0 2024-04-20T21:32:08.906 INFO:tasks.workunit.client.1.smithi189.stdout:9/377: symlink d9/d12/d15/l79 0 2024-04-20T21:32:08.907 INFO:tasks.workunit.client.0.smithi062.stdout:0/694: mknod d7/d20/d62/de3/ce5 0 2024-04-20T21:32:08.907 INFO:tasks.workunit.client.0.smithi062.stdout:8/688: write d7/d25/d3b/d58/d8b/d24/d46/fd2 [623010,50856] 0 2024-04-20T21:32:08.908 INFO:tasks.workunit.client.1.smithi189.stdout:4/358: mknod d2/d22/c89 0 2024-04-20T21:32:08.910 INFO:tasks.workunit.client.1.smithi189.stdout:0/465: symlink d3/d8c/l9c 0 2024-04-20T21:32:08.910 INFO:tasks.workunit.client.0.smithi062.stdout:6/581: dwrite d0/d52/f83 [0,4194304] 0 2024-04-20T21:32:08.912 INFO:tasks.workunit.client.1.smithi189.stdout:9/378: mkdir d9/d12/d3f/d7a 0 2024-04-20T21:32:08.915 INFO:tasks.workunit.client.0.smithi062.stdout:7/697: symlink d7/d26/dac/dec/lee 0 2024-04-20T21:32:08.915 INFO:tasks.workunit.client.0.smithi062.stdout:5/687: mkdir d8/d38/d7a/d4c/d7d/d9d/dac/d66/de4 0 2024-04-20T21:32:08.915 INFO:tasks.workunit.client.0.smithi062.stdout:1/757: mknod dc/d15/d20/d28/d83/dce/c112 0 2024-04-20T21:32:08.915 INFO:tasks.workunit.client.1.smithi189.stdout:4/359: unlink d2/dd/d11/l2b 0 2024-04-20T21:32:08.919 INFO:tasks.workunit.client.1.smithi189.stdout:5/310: dwrite d7/d9/d30/f2d [0,4194304] 0 2024-04-20T21:32:08.921 INFO:tasks.workunit.client.0.smithi062.stdout:7/698: write d7/d3e/f71 [1301961,77620] 0 2024-04-20T21:32:08.922 INFO:tasks.workunit.client.1.smithi189.stdout:5/311: readlink d7/d14/l3d 0 2024-04-20T21:32:08.925 INFO:tasks.workunit.client.1.smithi189.stdout:0/466: mknod d3/df/d30/d45/c9d 0 2024-04-20T21:32:08.928 INFO:tasks.workunit.client.1.smithi189.stdout:6/373: write d8/d2b/f3a [2127751,11380] 0 2024-04-20T21:32:08.928 INFO:tasks.workunit.client.0.smithi062.stdout:6/582: dread - d0/d2c/d67/f9e zero size 2024-04-20T21:32:08.929 INFO:tasks.workunit.client.0.smithi062.stdout:4/744: symlink da/d56/d71/d9e/l104 0 2024-04-20T21:32:08.929 INFO:tasks.workunit.client.0.smithi062.stdout:1/758: dread - dc/d15/d75/d96/dd4/d3a/da2/fb4 zero size 2024-04-20T21:32:08.930 INFO:tasks.workunit.client.1.smithi189.stdout:4/360: write d2/dd/d11/f31 [606921,102494] 0 2024-04-20T21:32:08.931 INFO:tasks.workunit.client.1.smithi189.stdout:4/361: stat d2/d3a/f63 0 2024-04-20T21:32:08.932 INFO:tasks.workunit.client.0.smithi062.stdout:5/688: dread d8/d38/d7a/d4c/d7d/fde [0,4194304] 0 2024-04-20T21:32:08.933 INFO:tasks.workunit.client.1.smithi189.stdout:9/379: fdatasync d9/d12/d3f/f68 0 2024-04-20T21:32:08.933 INFO:tasks.workunit.client.1.smithi189.stdout:1/336: link d8/d21/d2c/l2e d8/d1e/d28/l73 0 2024-04-20T21:32:08.934 INFO:tasks.workunit.client.1.smithi189.stdout:5/312: mknod d7/d9/d31/d36/c56 0 2024-04-20T21:32:08.936 INFO:tasks.workunit.client.1.smithi189.stdout:5/313: fdatasync d7/d9/f42 0 2024-04-20T21:32:08.937 INFO:tasks.workunit.client.1.smithi189.stdout:0/467: creat d3/df/d28/d2b/d2c/f9e x:0 0 0 2024-04-20T21:32:08.939 INFO:tasks.workunit.client.0.smithi062.stdout:0/695: creat d7/fe6 x:0 0 0 2024-04-20T21:32:08.946 INFO:tasks.workunit.client.0.smithi062.stdout:0/696: chown d7/d1c/d15/f2d 3990462 1 2024-04-20T21:32:08.946 INFO:tasks.workunit.client.1.smithi189.stdout:5/314: creat d7/d9/d31/f57 x:0 0 0 2024-04-20T21:32:08.946 INFO:tasks.workunit.client.0.smithi062.stdout:8/689: truncate d7/d25/d3b/d58/d8b/d24/f50 288230 0 2024-04-20T21:32:08.949 INFO:tasks.workunit.client.1.smithi189.stdout:5/315: write d7/d9/d31/d36/f54 [135781,91795] 0 2024-04-20T21:32:08.950 INFO:tasks.workunit.client.0.smithi062.stdout:4/745: creat da/dd/df/d27/d32/ded/f105 x:0 0 0 2024-04-20T21:32:08.951 INFO:tasks.workunit.client.1.smithi189.stdout:9/380: link d9/dc/f3d d9/d12/d3f/d72/f7b 0 2024-04-20T21:32:08.952 INFO:tasks.workunit.client.1.smithi189.stdout:8/459: dread da/f39 [4194304,4194304] 0 2024-04-20T21:32:08.952 INFO:tasks.workunit.client.1.smithi189.stdout:5/316: read d7/d14/d4e/f39 [1168664,39548] 0 2024-04-20T21:32:08.952 INFO:tasks.workunit.client.1.smithi189.stdout:1/337: getdents d8/d21 0 2024-04-20T21:32:08.955 INFO:tasks.workunit.client.1.smithi189.stdout:9/381: chown d9/dc/d17/d2b/d52/c6f 232832 1 2024-04-20T21:32:08.956 INFO:tasks.workunit.client.1.smithi189.stdout:1/338: write d8/d1e/d43/d44/f47 [192483,7694] 0 2024-04-20T21:32:08.957 INFO:tasks.workunit.client.1.smithi189.stdout:2/351: truncate d1/d17/f2b 1099902 0 2024-04-20T21:32:08.960 INFO:tasks.workunit.client.1.smithi189.stdout:1/339: write d8/d1e/f6e [861865,9017] 0 2024-04-20T21:32:08.960 INFO:tasks.workunit.client.0.smithi062.stdout:2/716: truncate d7/da/d15/f23 1042073 0 2024-04-20T21:32:08.960 INFO:tasks.workunit.client.0.smithi062.stdout:8/690: chown d7/d30/d90/f92 536630 1 2024-04-20T21:32:08.960 INFO:tasks.workunit.client.0.smithi062.stdout:7/699: getdents d7/d63 0 2024-04-20T21:32:08.960 INFO:tasks.workunit.client.0.smithi062.stdout:0/697: mknod d7/d1c/d15/d17/d40/d3c/da4/ce7 0 2024-04-20T21:32:08.961 INFO:tasks.workunit.client.1.smithi189.stdout:1/340: read d8/f24 [3698164,113556] 0 2024-04-20T21:32:08.963 INFO:tasks.workunit.client.1.smithi189.stdout:0/468: rename d3/df/d28/d2b/d2c/l54 to d3/df/l9f 0 2024-04-20T21:32:08.963 INFO:tasks.workunit.client.0.smithi062.stdout:4/746: mknod da/dd/d15/d4b/c106 0 2024-04-20T21:32:08.963 INFO:tasks.workunit.client.1.smithi189.stdout:5/317: creat d7/d9/d28/f58 x:0 0 0 2024-04-20T21:32:08.965 INFO:tasks.workunit.client.1.smithi189.stdout:9/382: creat d9/dc/d69/f7c x:0 0 0 2024-04-20T21:32:08.968 INFO:tasks.workunit.client.1.smithi189.stdout:8/460: creat da/df/d22/d33/d59/fa6 x:0 0 0 2024-04-20T21:32:08.969 INFO:tasks.workunit.client.0.smithi062.stdout:2/717: unlink d7/da/d15/d2f/f30 0 2024-04-20T21:32:08.972 INFO:tasks.workunit.client.0.smithi062.stdout:4/747: fsync da/dd/df/d27/da5/ff3 0 2024-04-20T21:32:08.973 INFO:tasks.workunit.client.0.smithi062.stdout:8/691: creat d7/d25/d70/fe4 x:0 0 0 2024-04-20T21:32:08.976 INFO:tasks.workunit.client.1.smithi189.stdout:1/341: getdents d8/d1e/d43/d44 0 2024-04-20T21:32:08.981 INFO:tasks.workunit.client.1.smithi189.stdout:0/469: link d3/df/d28/d2b/d2c/f9e d3/d26/d88/fa0 0 2024-04-20T21:32:08.982 INFO:tasks.workunit.client.1.smithi189.stdout:1/342: chown d8/c35 5 1 2024-04-20T21:32:08.984 INFO:tasks.workunit.client.1.smithi189.stdout:9/383: link d9/d12/d15/d1a/l38 d9/l7d 0 2024-04-20T21:32:08.986 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:08 smithi189 ceph-mon[21562]: Reconfiguring prometheus.smithi062 (dependencies changed)... 2024-04-20T21:32:08.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:08 smithi189 ceph-mon[21562]: Reconfiguring daemon prometheus.smithi062 on smithi062 2024-04-20T21:32:08.990 INFO:tasks.workunit.client.0.smithi062.stdout:5/689: dwrite d8/d10/d3e/d8a/d9e/fc9 [0,4194304] 0 2024-04-20T21:32:08.994 INFO:tasks.workunit.client.1.smithi189.stdout:0/470: creat d3/df/d30/d45/fa1 x:0 0 0 2024-04-20T21:32:08.994 INFO:tasks.workunit.client.0.smithi062.stdout:1/759: write dc/d15/d75/d96/dd4/d3a/f4f [521820,48500] 0 2024-04-20T21:32:08.997 INFO:tasks.workunit.client.1.smithi189.stdout:1/343: mknod d8/d1e/d43/d44/c74 0 2024-04-20T21:32:08.997 INFO:tasks.workunit.client.1.smithi189.stdout:0/471: chown d3/df/d30/d37/f31 155666 1 2024-04-20T21:32:09.002 INFO:tasks.workunit.client.0.smithi062.stdout:9/740: dwrite f15 [0,4194304] 0 2024-04-20T21:32:09.002 INFO:tasks.workunit.client.1.smithi189.stdout:0/472: fdatasync d3/df/d30/d37/f4b 0 2024-04-20T21:32:09.004 INFO:tasks.workunit.client.0.smithi062.stdout:7/700: mknod d7/d3e/d33/d3d/d4d/d54/d4f/cef 0 2024-04-20T21:32:09.004 INFO:tasks.workunit.client.1.smithi189.stdout:2/352: dwrite d1/d17/f2b [0,4194304] 0 2024-04-20T21:32:09.005 INFO:tasks.workunit.client.1.smithi189.stdout:8/461: truncate da/de/f9f 615161 0 2024-04-20T21:32:09.006 INFO:tasks.workunit.client.1.smithi189.stdout:0/473: chown d3/df/l13 1290 1 2024-04-20T21:32:09.007 INFO:tasks.workunit.client.1.smithi189.stdout:6/374: dread d8/d10/f20 [0,4194304] 0 2024-04-20T21:32:09.009 INFO:tasks.workunit.client.0.smithi062.stdout:3/660: dwrite d0/d11/f1a [4194304,4194304] 0 2024-04-20T21:32:09.009 INFO:tasks.workunit.client.1.smithi189.stdout:2/353: chown d1/d11/d26 4845138 1 2024-04-20T21:32:09.010 INFO:tasks.workunit.client.0.smithi062.stdout:0/698: dread d7/d1c/d15/d17/d26/fa7 [0,4194304] 0 2024-04-20T21:32:09.013 INFO:tasks.workunit.client.0.smithi062.stdout:6/583: dwrite d0/d1a/d44/f4b [0,4194304] 0 2024-04-20T21:32:09.015 INFO:tasks.workunit.client.1.smithi189.stdout:8/462: mknod da/d1f/ca7 0 2024-04-20T21:32:09.015 INFO:tasks.workunit.client.1.smithi189.stdout:0/474: mknod d3/df/d28/ca2 0 2024-04-20T21:32:09.022 INFO:tasks.workunit.client.0.smithi062.stdout:2/718: truncate d7/da/f13 3091305 0 2024-04-20T21:32:09.023 INFO:tasks.workunit.client.1.smithi189.stdout:8/463: truncate da/df/d22/d33/d59/fa6 7965 0 2024-04-20T21:32:09.025 INFO:tasks.workunit.client.1.smithi189.stdout:2/354: rename d1/d11/c1b to d1/d11/d52/c69 0 2024-04-20T21:32:09.028 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:08 smithi062 ceph-mon[16221]: Reconfiguring prometheus.smithi062 (dependencies changed)... 2024-04-20T21:32:09.028 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:08 smithi062 ceph-mon[16221]: Reconfiguring daemon prometheus.smithi062 on smithi062 2024-04-20T21:32:09.028 INFO:tasks.workunit.client.0.smithi062.stdout:6/584: dread - d0/d2c/d67/d75/f7a zero size 2024-04-20T21:32:09.028 INFO:tasks.workunit.client.1.smithi189.stdout:1/344: dwrite d8/f14 [0,4194304] 0 2024-04-20T21:32:09.029 INFO:tasks.workunit.client.0.smithi062.stdout:6/585: readlink d0/d4f/l9b 0 2024-04-20T21:32:09.031 INFO:tasks.workunit.client.1.smithi189.stdout:6/375: creat d8/d10/d78/f7d x:0 0 0 2024-04-20T21:32:09.032 INFO:tasks.workunit.client.0.smithi062.stdout:3/661: creat d0/fdc x:0 0 0 2024-04-20T21:32:09.033 INFO:tasks.workunit.client.1.smithi189.stdout:2/355: dread - d1/d17/d37/f67 zero size 2024-04-20T21:32:09.033 INFO:tasks.workunit.client.0.smithi062.stdout:7/701: write d7/d3e/d33/d3d/d5e/fb4 [611990,57275] 0 2024-04-20T21:32:09.033 INFO:tasks.workunit.client.0.smithi062.stdout:0/699: truncate d7/d1c/d15/d17/d40/d3c/dc5/f99 5188703 0 2024-04-20T21:32:09.035 INFO:tasks.workunit.client.0.smithi062.stdout:4/748: dread da/dd/df/d27/d32/f89 [0,4194304] 0 2024-04-20T21:32:09.036 INFO:tasks.workunit.client.1.smithi189.stdout:1/345: chown d8/df/d3a/c42 48950 1 2024-04-20T21:32:09.037 INFO:tasks.workunit.client.1.smithi189.stdout:6/376: write d8/d10/d78/f7d [759353,94932] 0 2024-04-20T21:32:09.041 INFO:tasks.workunit.client.1.smithi189.stdout:6/377: read - d8/d2b/f57 zero size 2024-04-20T21:32:09.046 INFO:tasks.workunit.client.0.smithi062.stdout:0/700: creat d7/ddf/fe8 x:0 0 0 2024-04-20T21:32:09.046 INFO:tasks.workunit.client.1.smithi189.stdout:1/346: creat d8/d21/d2c/f75 x:0 0 0 2024-04-20T21:32:09.047 INFO:tasks.workunit.client.1.smithi189.stdout:6/378: fsync d8/f1a 0 2024-04-20T21:32:09.049 INFO:tasks.workunit.client.1.smithi189.stdout:8/464: dread da/de/d1e/f52 [0,4194304] 0 2024-04-20T21:32:09.049 INFO:tasks.workunit.client.1.smithi189.stdout:9/384: dread d9/f33 [0,4194304] 0 2024-04-20T21:32:09.053 INFO:tasks.workunit.client.0.smithi062.stdout:0/701: creat d7/d1c/d15/d80/fe9 x:0 0 0 2024-04-20T21:32:09.053 INFO:tasks.workunit.client.1.smithi189.stdout:1/347: readlink d8/d1e/d54/l5d 0 2024-04-20T21:32:09.061 INFO:tasks.workunit.client.0.smithi062.stdout:8/692: truncate d7/d30/f3c 3875183 0 2024-04-20T21:32:09.062 INFO:tasks.workunit.client.0.smithi062.stdout:0/702: chown d7/d1c/d15/d17/d40/d3c/da4/ce7 422 1 2024-04-20T21:32:09.065 INFO:tasks.workunit.client.1.smithi189.stdout:6/379: mkdir d8/d10/d78/d7e 0 2024-04-20T21:32:09.067 INFO:tasks.workunit.client.1.smithi189.stdout:6/380: stat d8/d10/f36 0 2024-04-20T21:32:09.068 INFO:tasks.workunit.client.0.smithi062.stdout:0/703: dread - d7/d1c/d15/d53/fbb zero size 2024-04-20T21:32:09.068 INFO:tasks.workunit.client.0.smithi062.stdout:6/586: dwrite d0/f80 [4194304,4194304] 0 2024-04-20T21:32:09.068 INFO:tasks.workunit.client.0.smithi062.stdout:9/741: dwrite d16/f3a [0,4194304] 0 2024-04-20T21:32:09.072 INFO:tasks.workunit.client.0.smithi062.stdout:8/693: mkdir d7/de1/de5 0 2024-04-20T21:32:09.075 INFO:tasks.workunit.client.0.smithi062.stdout:6/587: write d0/d2c/f89 [598942,122713] 0 2024-04-20T21:32:09.076 INFO:tasks.workunit.client.1.smithi189.stdout:2/356: dread d1/d11/f1a [4194304,4194304] 0 2024-04-20T21:32:09.076 INFO:tasks.workunit.client.1.smithi189.stdout:9/385: mkdir d9/dc/d17/d7e 0 2024-04-20T21:32:09.077 INFO:tasks.workunit.client.0.smithi062.stdout:0/704: chown d7/d1c/d15/d17/d1b/d9d/db1/cbc 3984 1 2024-04-20T21:32:09.077 INFO:tasks.workunit.client.0.smithi062.stdout:8/694: chown d7/d9/d29/c7a 15437 1 2024-04-20T21:32:09.078 INFO:tasks.workunit.client.0.smithi062.stdout:6/588: chown d0/d1a/d7d/l36 122718391 1 2024-04-20T21:32:09.082 INFO:tasks.workunit.client.0.smithi062.stdout:8/695: write d7/d25/d4f/fdf [1001053,129580] 0 2024-04-20T21:32:09.082 INFO:tasks.workunit.client.0.smithi062.stdout:5/690: write d8/d38/fc5 [1469317,104782] 0 2024-04-20T21:32:09.082 INFO:tasks.workunit.client.1.smithi189.stdout:1/348: creat d8/d1e/d28/d2b/f76 x:0 0 0 2024-04-20T21:32:09.085 INFO:tasks.workunit.client.0.smithi062.stdout:0/705: creat d7/d1c/d15/d17/d26/fea x:0 0 0 2024-04-20T21:32:09.085 INFO:tasks.workunit.client.0.smithi062.stdout:6/589: write d0/d9/d11/d7b/fb2 [1556241,60339] 0 2024-04-20T21:32:09.087 INFO:tasks.workunit.client.0.smithi062.stdout:9/742: creat d16/d2c/d42/da6/da9/ff4 x:0 0 0 2024-04-20T21:32:09.088 INFO:tasks.workunit.client.1.smithi189.stdout:8/465: mkdir da/d81/d91/da8 0 2024-04-20T21:32:09.091 INFO:tasks.workunit.client.1.smithi189.stdout:2/357: creat d1/d11/d52/f6a x:0 0 0 2024-04-20T21:32:09.097 INFO:tasks.workunit.client.0.smithi062.stdout:0/706: creat d7/d1c/d15/d17/d1b/d9d/db1/feb x:0 0 0 2024-04-20T21:32:09.097 INFO:tasks.workunit.client.0.smithi062.stdout:7/702: dread d7/d26/f29 [0,4194304] 0 2024-04-20T21:32:09.101 INFO:tasks.workunit.client.1.smithi189.stdout:0/475: truncate d3/f48 1795760 0 2024-04-20T21:32:09.102 INFO:tasks.workunit.client.1.smithi189.stdout:6/381: mknod d8/da/dd/d35/d6d/d7b/c7f 0 2024-04-20T21:32:09.103 INFO:tasks.workunit.client.0.smithi062.stdout:3/662: dwrite d0/f98 [0,4194304] 0 2024-04-20T21:32:09.106 INFO:tasks.workunit.client.0.smithi062.stdout:3/663: chown d0/d1/f8b 0 1 2024-04-20T21:32:09.107 INFO:tasks.workunit.client.1.smithi189.stdout:0/476: truncate d3/df/d4c/f7e 5434679 0 2024-04-20T21:32:09.107 INFO:tasks.workunit.client.0.smithi062.stdout:7/703: symlink d7/d26/dac/ddb/lf0 0 2024-04-20T21:32:09.108 INFO:tasks.workunit.client.0.smithi062.stdout:4/749: dread da/dd/df/fa2 [0,4194304] 0 2024-04-20T21:32:09.109 INFO:tasks.workunit.client.0.smithi062.stdout:0/707: fsync d7/d1c/d15/d17/d37/f84 0 2024-04-20T21:32:09.110 INFO:tasks.workunit.client.0.smithi062.stdout:9/743: rename d16/d3d/l8e to d16/d3d/d55/lf5 0 2024-04-20T21:32:09.113 INFO:tasks.workunit.client.1.smithi189.stdout:8/466: rename da/de/d1e/d21/l57 to da/de/d1e/d21/d44/la9 0 2024-04-20T21:32:09.113 INFO:tasks.workunit.client.0.smithi062.stdout:0/708: truncate d7/d1e/fdc 834349 0 2024-04-20T21:32:09.114 INFO:tasks.workunit.client.0.smithi062.stdout:1/760: dread dc/d15/d20/d28/d83/fa5 [0,4194304] 0 2024-04-20T21:32:09.117 INFO:tasks.workunit.client.1.smithi189.stdout:6/382: creat d8/d10/d78/f80 x:0 0 0 2024-04-20T21:32:09.119 INFO:tasks.workunit.client.1.smithi189.stdout:6/383: fdatasync d8/d12/f17 0 2024-04-20T21:32:09.119 INFO:tasks.workunit.client.1.smithi189.stdout:3/337: sync 2024-04-20T21:32:09.119 INFO:tasks.workunit.client.1.smithi189.stdout:5/318: sync 2024-04-20T21:32:09.119 INFO:tasks.workunit.client.1.smithi189.stdout:7/368: sync 2024-04-20T21:32:09.120 INFO:tasks.workunit.client.0.smithi062.stdout:7/704: write d7/d26/f9a [3589470,46451] 0 2024-04-20T21:32:09.120 INFO:tasks.workunit.client.0.smithi062.stdout:0/709: chown d7/d1c/d15/d17/d40/d79/l78 0 1 2024-04-20T21:32:09.120 INFO:tasks.workunit.client.1.smithi189.stdout:4/362: sync 2024-04-20T21:32:09.120 INFO:tasks.workunit.client.1.smithi189.stdout:2/358: dwrite d1/d17/f34 [0,4194304] 0 2024-04-20T21:32:09.121 INFO:tasks.workunit.client.0.smithi062.stdout:7/705: dread - d7/d3e/d33/d3d/d4d/fc0 zero size 2024-04-20T21:32:09.123 INFO:tasks.workunit.client.0.smithi062.stdout:9/744: creat d16/d3d/ff6 x:0 0 0 2024-04-20T21:32:09.124 INFO:tasks.workunit.client.1.smithi189.stdout:2/359: write d1/d17/d37/f67 [599234,96410] 0 2024-04-20T21:32:09.125 INFO:tasks.workunit.client.0.smithi062.stdout:5/691: link d8/d1e/d2c/d6c/f65 d8/d38/d7a/d4c/d7d/d9d/dac/fe5 0 2024-04-20T21:32:09.127 INFO:tasks.workunit.client.0.smithi062.stdout:8/696: write d7/d25/d70/fa1 [320690,14297] 0 2024-04-20T21:32:09.128 INFO:tasks.workunit.client.1.smithi189.stdout:8/467: rename da/df/d22/l26 to da/df/d20/d2c/laa 0 2024-04-20T21:32:09.132 INFO:tasks.workunit.client.1.smithi189.stdout:9/386: dread d9/d12/d15/d1a/f5c [0,4194304] 0 2024-04-20T21:32:09.132 INFO:tasks.workunit.client.0.smithi062.stdout:7/706: stat d7/d3e/d33/d3d/d4d/d54/dea/dcc/ce7 0 2024-04-20T21:32:09.134 INFO:tasks.workunit.client.1.smithi189.stdout:4/363: chown d2/dd/d6b/l85 72312 1 2024-04-20T21:32:09.136 INFO:tasks.workunit.client.1.smithi189.stdout:3/338: unlink d2/d3/d2c/d31/c69 0 2024-04-20T21:32:09.136 INFO:tasks.workunit.client.0.smithi062.stdout:0/710: dread d7/d20/d3a/f87 [0,4194304] 0 2024-04-20T21:32:09.141 INFO:tasks.workunit.client.1.smithi189.stdout:3/339: write d2/d8/f22 [5155321,59463] 0 2024-04-20T21:32:09.143 INFO:tasks.workunit.client.0.smithi062.stdout:9/745: rename d16/d2c/d42/l4a to d16/d3d/lf7 0 2024-04-20T21:32:09.144 INFO:tasks.workunit.client.1.smithi189.stdout:4/364: write d2/dd/d11/d14/f79 [3344974,39558] 0 2024-04-20T21:32:09.144 INFO:tasks.workunit.client.1.smithi189.stdout:2/360: stat d1/d11/d52/c69 0 2024-04-20T21:32:09.145 INFO:tasks.workunit.client.1.smithi189.stdout:7/369: dread d5/d21/d25/f30 [0,4194304] 0 2024-04-20T21:32:09.148 INFO:tasks.workunit.client.1.smithi189.stdout:5/319: creat d7/d9/d30/d55/d33/f59 x:0 0 0 2024-04-20T21:32:09.149 INFO:tasks.workunit.client.0.smithi062.stdout:9/746: chown d16/d2c/d42/fde 138 1 2024-04-20T21:32:09.149 INFO:tasks.workunit.client.1.smithi189.stdout:9/387: truncate d9/d12/d15/f30 752954 0 2024-04-20T21:32:09.152 INFO:tasks.workunit.client.1.smithi189.stdout:0/477: link d3/d26/f1d d3/df/d28/d2b/d2c/d73/fa3 0 2024-04-20T21:32:09.153 INFO:tasks.workunit.client.1.smithi189.stdout:8/468: truncate f9 956065 0 2024-04-20T21:32:09.154 INFO:tasks.workunit.client.1.smithi189.stdout:5/320: chown d7/d35/f43 80833088 1 2024-04-20T21:32:09.154 INFO:tasks.workunit.client.0.smithi062.stdout:5/692: creat d8/d38/d7a/d4c/fe6 x:0 0 0 2024-04-20T21:32:09.154 INFO:tasks.workunit.client.0.smithi062.stdout:7/707: mkdir d7/d3e/d33/d3d/d4d/d54/d4f/de9/df1 0 2024-04-20T21:32:09.155 INFO:tasks.workunit.client.1.smithi189.stdout:2/361: mkdir d1/d11/d58/d6b 0 2024-04-20T21:32:09.155 INFO:tasks.workunit.client.0.smithi062.stdout:8/697: link d7/d25/l33 d7/d25/d76/le6 0 2024-04-20T21:32:09.158 INFO:tasks.workunit.client.1.smithi189.stdout:4/365: dread d2/dd/d11/f31 [0,4194304] 0 2024-04-20T21:32:09.159 INFO:tasks.workunit.client.1.smithi189.stdout:9/388: creat d9/dc/d69/f7f x:0 0 0 2024-04-20T21:32:09.161 INFO:tasks.workunit.client.0.smithi062.stdout:6/590: dread d0/d4f/f73 [0,4194304] 0 2024-04-20T21:32:09.161 INFO:tasks.workunit.client.0.smithi062.stdout:0/711: dread d7/d1c/f3e [0,4194304] 0 2024-04-20T21:32:09.164 INFO:tasks.workunit.client.1.smithi189.stdout:7/370: mkdir d5/d1d/d77/d5b/d71/d7a 0 2024-04-20T21:32:09.164 INFO:tasks.workunit.client.0.smithi062.stdout:1/761: dread dc/d15/d20/f45 [0,4194304] 0 2024-04-20T21:32:09.165 INFO:tasks.workunit.client.0.smithi062.stdout:6/591: chown d0/d9/d11/f65 108220 1 2024-04-20T21:32:09.166 INFO:tasks.workunit.client.0.smithi062.stdout:7/708: mkdir d7/d26/dac/df2 0 2024-04-20T21:32:09.167 INFO:tasks.workunit.client.1.smithi189.stdout:4/366: fsync d2/d3a/f4d 0 2024-04-20T21:32:09.167 INFO:tasks.workunit.client.1.smithi189.stdout:5/321: mkdir d7/d9/d30/d5a 0 2024-04-20T21:32:09.169 INFO:tasks.workunit.client.0.smithi062.stdout:3/664: dread d0/d1c/d63/fbb [0,4194304] 0 2024-04-20T21:32:09.170 INFO:tasks.workunit.client.0.smithi062.stdout:1/762: mknod dc/d15/d20/d28/d83/dce/c113 0 2024-04-20T21:32:09.172 INFO:tasks.workunit.client.1.smithi189.stdout:1/349: dread d8/f9 [0,4194304] 0 2024-04-20T21:32:09.172 INFO:tasks.workunit.client.1.smithi189.stdout:7/371: write d5/d1d/d77/d5b/f2e [1730399,13004] 0 2024-04-20T21:32:09.173 INFO:tasks.workunit.client.0.smithi062.stdout:6/592: mkdir d0/d9/d11/d5c/d7c/dc5 0 2024-04-20T21:32:09.173 INFO:tasks.workunit.client.1.smithi189.stdout:8/469: symlink da/de/lab 0 2024-04-20T21:32:09.174 INFO:tasks.workunit.client.1.smithi189.stdout:0/478: truncate d3/d26/d88/fa0 681032 0 2024-04-20T21:32:09.176 INFO:tasks.workunit.client.0.smithi062.stdout:6/593: chown d0/d9/d11/c61 1730061 1 2024-04-20T21:32:09.177 INFO:tasks.workunit.client.1.smithi189.stdout:8/470: stat da/d81/d91/da8 0 2024-04-20T21:32:09.178 INFO:tasks.workunit.client.1.smithi189.stdout:4/367: symlink d2/dd/d11/d14/d69/d6a/l8a 0 2024-04-20T21:32:09.179 INFO:tasks.workunit.client.1.smithi189.stdout:5/322: getdents d7/d9/d51 0 2024-04-20T21:32:09.179 INFO:tasks.workunit.client.1.smithi189.stdout:1/350: chown d8/d1e/d28/l71 728 1 2024-04-20T21:32:09.180 INFO:tasks.workunit.client.1.smithi189.stdout:3/340: dwrite d2/d1c/f7b [0,4194304] 0 2024-04-20T21:32:09.183 INFO:tasks.workunit.client.0.smithi062.stdout:7/709: fdatasync d7/d3e/d33/d3d/d5e/d72/fb2 0 2024-04-20T21:32:09.184 INFO:tasks.workunit.client.1.smithi189.stdout:5/323: fsync d7/d9/d28/f58 0 2024-04-20T21:32:09.184 INFO:tasks.workunit.client.1.smithi189.stdout:6/384: dread d8/d2b/f3a [0,4194304] 0 2024-04-20T21:32:09.184 INFO:tasks.workunit.client.1.smithi189.stdout:7/372: rename d5/d1d/d77/d5b/f2e to d5/d1d/d64/f7b 0 2024-04-20T21:32:09.185 INFO:tasks.workunit.client.0.smithi062.stdout:4/750: dread da/dd/df/d27/f31 [4194304,4194304] 0 2024-04-20T21:32:09.185 INFO:tasks.workunit.client.0.smithi062.stdout:3/665: mknod d0/d1/d10/d31/d41/cdd 0 2024-04-20T21:32:09.186 INFO:tasks.workunit.client.1.smithi189.stdout:4/368: dread - d2/dd/d11/d14/f7d zero size 2024-04-20T21:32:09.187 INFO:tasks.workunit.client.1.smithi189.stdout:0/479: unlink d3/df/d28/c2e 0 2024-04-20T21:32:09.187 INFO:tasks.workunit.client.0.smithi062.stdout:1/763: creat dc/d15/d75/d96/dd4/d3a/da2/de4/d84/dbe/f114 x:0 0 0 2024-04-20T21:32:09.190 INFO:tasks.workunit.client.0.smithi062.stdout:9/747: getdents d16/d3d 0 2024-04-20T21:32:09.190 INFO:tasks.workunit.client.0.smithi062.stdout:6/594: write d0/d52/da0/fa8 [861258,55619] 0 2024-04-20T21:32:09.192 INFO:tasks.workunit.client.1.smithi189.stdout:5/324: write d7/d9/d31/f57 [299906,245] 0 2024-04-20T21:32:09.192 INFO:tasks.workunit.client.1.smithi189.stdout:4/369: write d2/dd/f44 [43142,13373] 0 2024-04-20T21:32:09.195 INFO:tasks.workunit.client.0.smithi062.stdout:1/764: dread - dc/d15/d75/d96/dd4/d78/dab/fb7 zero size 2024-04-20T21:32:09.196 INFO:tasks.workunit.client.1.smithi189.stdout:1/351: symlink d8/d1e/d43/l77 0 2024-04-20T21:32:09.198 INFO:tasks.workunit.client.1.smithi189.stdout:6/385: symlink d8/d2b/d49/l81 0 2024-04-20T21:32:09.199 INFO:tasks.workunit.client.0.smithi062.stdout:5/693: dread d8/d1e/d2c/f79 [0,4194304] 0 2024-04-20T21:32:09.200 INFO:tasks.workunit.client.1.smithi189.stdout:4/370: chown d2/d22/d87/l3d 979445 1 2024-04-20T21:32:09.200 INFO:tasks.workunit.client.0.smithi062.stdout:3/666: unlink d0/d1/d10/c9f 0 2024-04-20T21:32:09.201 INFO:tasks.workunit.client.1.smithi189.stdout:1/352: readlink d8/df/d3a/d49/l53 0 2024-04-20T21:32:09.204 INFO:tasks.workunit.client.1.smithi189.stdout:7/373: creat d5/d21/d4a/d52/f7c x:0 0 0 2024-04-20T21:32:09.204 INFO:tasks.workunit.client.1.smithi189.stdout:4/371: chown d2/d22/f30 368541 1 2024-04-20T21:32:09.204 INFO:tasks.workunit.client.1.smithi189.stdout:6/386: write d8/da/dd/d35/f31 [2866041,102997] 0 2024-04-20T21:32:09.206 INFO:tasks.workunit.client.1.smithi189.stdout:2/362: dwrite d1/d17/d2f/f49 [0,4194304] 0 2024-04-20T21:32:09.206 INFO:tasks.workunit.client.0.smithi062.stdout:9/748: dread f15 [0,4194304] 0 2024-04-20T21:32:09.207 INFO:tasks.workunit.client.0.smithi062.stdout:6/595: readlink d0/d2c/d8c/l9f 0 2024-04-20T21:32:09.209 INFO:tasks.workunit.client.1.smithi189.stdout:4/372: dread - d2/dd/d11/d14/f7d zero size 2024-04-20T21:32:09.209 INFO:tasks.workunit.client.1.smithi189.stdout:5/325: rename d7/d14/d1b to d7/d14/d4e/d5b 0 2024-04-20T21:32:09.209 INFO:tasks.workunit.client.0.smithi062.stdout:9/749: truncate d16/d3d/d55/d6e/f72 4258003 0 2024-04-20T21:32:09.210 INFO:tasks.workunit.client.1.smithi189.stdout:0/480: creat d3/df/d4c/fa4 x:0 0 0 2024-04-20T21:32:09.210 INFO:tasks.workunit.client.0.smithi062.stdout:4/751: symlink da/dd/df/l107 0 2024-04-20T21:32:09.213 INFO:tasks.workunit.client.1.smithi189.stdout:2/363: chown d1/f5 309 1 2024-04-20T21:32:09.214 INFO:tasks.workunit.client.1.smithi189.stdout:1/353: creat d8/df/d3a/f78 x:0 0 0 2024-04-20T21:32:09.215 INFO:tasks.workunit.client.0.smithi062.stdout:7/710: mkdir d7/d3e/d33/d3d/d4d/d54/d4f/de9/df1/df3 0 2024-04-20T21:32:09.217 INFO:tasks.workunit.client.0.smithi062.stdout:1/765: stat dc/d15/d20/f3c 0 2024-04-20T21:32:09.217 INFO:tasks.workunit.client.1.smithi189.stdout:6/387: symlink d8/da/dd/d2a/l82 0 2024-04-20T21:32:09.217 INFO:tasks.workunit.client.1.smithi189.stdout:8/471: dread da/df/f23 [0,4194304] 0 2024-04-20T21:32:09.218 INFO:tasks.workunit.client.0.smithi062.stdout:6/596: dread d0/d2c/d67/fa1 [0,4194304] 0 2024-04-20T21:32:09.220 INFO:tasks.workunit.client.1.smithi189.stdout:3/341: getdents d2/d3/d2c/d31/d74 0 2024-04-20T21:32:09.220 INFO:tasks.workunit.client.1.smithi189.stdout:4/373: write d2/f10 [4084754,100085] 0 2024-04-20T21:32:09.220 INFO:tasks.workunit.client.1.smithi189.stdout:0/481: mknod d3/df/d28/ca5 0 2024-04-20T21:32:09.221 INFO:tasks.workunit.client.0.smithi062.stdout:5/694: fdatasync d8/d10/d3e/f94 0 2024-04-20T21:32:09.223 INFO:tasks.workunit.client.0.smithi062.stdout:5/695: fdatasync d8/d38/d7a/f36 0 2024-04-20T21:32:09.226 INFO:tasks.workunit.client.1.smithi189.stdout:1/354: write d8/d21/d2c/d36/f52 [1269157,47823] 0 2024-04-20T21:32:09.226 INFO:tasks.workunit.client.0.smithi062.stdout:9/750: mkdir d16/d2c/d62/dd1/df8 0 2024-04-20T21:32:09.226 INFO:tasks.workunit.client.0.smithi062.stdout:7/711: mknod d7/d3e/d33/d3d/d4d/d83/d95/cf4 0 2024-04-20T21:32:09.227 INFO:tasks.workunit.client.0.smithi062.stdout:1/766: mknod dc/d15/d75/d94/d100/c115 0 2024-04-20T21:32:09.227 INFO:tasks.workunit.client.1.smithi189.stdout:7/374: rename d5/d1d/d42 to d5/d46/d7d 0 2024-04-20T21:32:09.229 INFO:tasks.workunit.client.1.smithi189.stdout:0/482: truncate d3/df/d28/d2b/f36 5006567 0 2024-04-20T21:32:09.230 INFO:tasks.workunit.client.0.smithi062.stdout:5/696: mknod d8/d38/d7a/db1/ce7 0 2024-04-20T21:32:09.231 INFO:tasks.workunit.client.0.smithi062.stdout:1/767: chown dc/d15/d20/d28/f49 69326275 1 2024-04-20T21:32:09.232 INFO:tasks.workunit.client.1.smithi189.stdout:2/364: creat d1/d11/d58/d6b/f6c x:0 0 0 2024-04-20T21:32:09.232 INFO:tasks.workunit.client.1.smithi189.stdout:0/483: chown d3/df/d5d/d64/c70 66861553 1 2024-04-20T21:32:09.233 INFO:tasks.workunit.client.1.smithi189.stdout:1/355: symlink d8/d21/d2c/l79 0 2024-04-20T21:32:09.233 INFO:tasks.workunit.client.0.smithi062.stdout:4/752: symlink da/dd/d15/d50/d64/d57/l108 0 2024-04-20T21:32:09.234 INFO:tasks.workunit.client.1.smithi189.stdout:7/375: creat d5/d21/d35/f7e x:0 0 0 2024-04-20T21:32:09.236 INFO:tasks.workunit.client.0.smithi062.stdout:4/753: write da/d56/f74 [1658035,3066] 0 2024-04-20T21:32:09.237 INFO:tasks.workunit.client.1.smithi189.stdout:5/326: link d7/d9/d30/l32 d7/d14/d18/l5c 0 2024-04-20T21:32:09.239 INFO:tasks.workunit.client.1.smithi189.stdout:5/327: fdatasync d7/d9/d30/f2d 0 2024-04-20T21:32:09.239 INFO:tasks.workunit.client.1.smithi189.stdout:7/376: unlink d5/d1d/f26 0 2024-04-20T21:32:09.241 INFO:tasks.workunit.client.0.smithi062.stdout:1/768: creat dc/d15/d75/d96/dd4/d3a/da2/de4/d7c/f116 x:0 0 0 2024-04-20T21:32:09.242 INFO:tasks.workunit.client.1.smithi189.stdout:0/484: mknod d3/df/d28/d2b/d2c/d74/ca6 0 2024-04-20T21:32:09.243 INFO:tasks.workunit.client.1.smithi189.stdout:2/365: symlink d1/d17/d37/l6d 0 2024-04-20T21:32:09.244 INFO:tasks.workunit.client.0.smithi062.stdout:3/667: creat d0/d1/d10/d3a/d5e/d65/d88/fde x:0 0 0 2024-04-20T21:32:09.244 INFO:tasks.workunit.client.1.smithi189.stdout:7/377: read - d5/d21/d4a/d52/f5c zero size 2024-04-20T21:32:09.248 INFO:tasks.workunit.client.1.smithi189.stdout:7/378: dread - d5/d1d/f62 zero size 2024-04-20T21:32:09.248 INFO:tasks.workunit.client.1.smithi189.stdout:8/472: getdents da/de/d1e/d21 0 2024-04-20T21:32:09.248 INFO:tasks.workunit.client.0.smithi062.stdout:7/712: creat d7/d3e/d33/d3d/d4d/d54/d4f/de9/df1/df3/ff5 x:0 0 0 2024-04-20T21:32:09.250 INFO:tasks.workunit.client.0.smithi062.stdout:1/769: creat dc/d15/d75/d94/d100/f117 x:0 0 0 2024-04-20T21:32:09.250 INFO:tasks.workunit.client.1.smithi189.stdout:3/342: dwrite d2/d3/d65/f78 [0,4194304] 0 2024-04-20T21:32:09.250 INFO:tasks.workunit.client.0.smithi062.stdout:9/751: link d16/d2c/d42/l57 d16/d2c/d62/dd1/df0/lf9 0 2024-04-20T21:32:09.251 INFO:tasks.workunit.client.1.smithi189.stdout:8/473: chown da/fb 845 1 2024-04-20T21:32:09.253 INFO:tasks.workunit.client.1.smithi189.stdout:5/328: unlink d7/d14/f26 0 2024-04-20T21:32:09.254 INFO:tasks.workunit.client.1.smithi189.stdout:7/379: truncate d5/d21/d25/d34/f6d 824106 0 2024-04-20T21:32:09.254 INFO:tasks.workunit.client.1.smithi189.stdout:5/329: chown d7/d14 815 1 2024-04-20T21:32:09.255 INFO:tasks.workunit.client.0.smithi062.stdout:5/697: mknod d8/d38/d7a/d4c/d7d/d9d/dac/d66/de4/ce8 0 2024-04-20T21:32:09.255 INFO:tasks.workunit.client.1.smithi189.stdout:1/356: link d8/c65 d8/df/d3a/c7a 0 2024-04-20T21:32:09.256 INFO:tasks.workunit.client.0.smithi062.stdout:3/668: symlink d0/d1/d10/d31/d41/ldf 0 2024-04-20T21:32:09.256 INFO:tasks.workunit.client.1.smithi189.stdout:2/366: truncate d1/d17/d37/f67 1256905 0 2024-04-20T21:32:09.257 INFO:tasks.workunit.client.0.smithi062.stdout:4/754: mkdir da/dd/df/d109 0 2024-04-20T21:32:09.261 INFO:tasks.workunit.client.0.smithi062.stdout:6/597: dread d0/d9/d11/d22/f68 [0,4194304] 0 2024-04-20T21:32:09.261 INFO:tasks.workunit.client.1.smithi189.stdout:3/343: creat d2/d3/d2c/d31/d74/d7a/f7c x:0 0 0 2024-04-20T21:32:09.263 INFO:tasks.workunit.client.0.smithi062.stdout:1/770: rename dc/d15/d75/d94/d6f/lbf to dc/d15/d20/d28/d76/l118 0 2024-04-20T21:32:09.264 INFO:tasks.workunit.client.1.smithi189.stdout:2/367: unlink d1/d11/c1d 0 2024-04-20T21:32:09.264 INFO:tasks.workunit.client.1.smithi189.stdout:0/485: rename d3/df/d28/d2b/d2c/d74/c77 to d3/d26/ca7 0 2024-04-20T21:32:09.265 INFO:tasks.workunit.client.1.smithi189.stdout:7/380: rename d5/d46 to d5/d46/d7d/d7f 22 2024-04-20T21:32:09.265 INFO:tasks.workunit.client.1.smithi189.stdout:1/357: creat d8/df/d3a/f7b x:0 0 0 2024-04-20T21:32:09.266 INFO:tasks.workunit.client.0.smithi062.stdout:5/698: chown d8/d1e/d2c/d6c/c4a 1 1 2024-04-20T21:32:09.268 INFO:tasks.workunit.client.0.smithi062.stdout:4/755: creat da/dd/df/d27/d32/ded/f10a x:0 0 0 2024-04-20T21:32:09.268 INFO:tasks.workunit.client.1.smithi189.stdout:3/344: creat d2/d1c/f7d x:0 0 0 2024-04-20T21:32:09.268 INFO:tasks.workunit.client.1.smithi189.stdout:6/388: dread f2 [0,4194304] 0 2024-04-20T21:32:09.269 INFO:tasks.workunit.client.1.smithi189.stdout:5/330: creat d7/f5d x:0 0 0 2024-04-20T21:32:09.271 INFO:tasks.workunit.client.1.smithi189.stdout:1/358: mknod d8/df/d3a/c7c 0 2024-04-20T21:32:09.271 INFO:tasks.workunit.client.0.smithi062.stdout:6/598: mknod d0/d1a/d7d/db1/db9/cc6 0 2024-04-20T21:32:09.272 INFO:tasks.workunit.client.1.smithi189.stdout:0/486: write d3/df/d28/d2b/f58 [878401,24565] 0 2024-04-20T21:32:09.273 INFO:tasks.workunit.client.1.smithi189.stdout:2/368: creat d1/d11/d58/f6e x:0 0 0 2024-04-20T21:32:09.274 INFO:tasks.workunit.client.0.smithi062.stdout:9/752: dwrite d16/d2c/d5e/f7b [0,4194304] 0 2024-04-20T21:32:09.274 INFO:tasks.workunit.client.0.smithi062.stdout:5/699: truncate d8/d1e/d2c/f79 3064052 0 2024-04-20T21:32:09.275 INFO:tasks.workunit.client.1.smithi189.stdout:1/359: write d8/d1e/d28/d2b/f61 [1561317,39718] 0 2024-04-20T21:32:09.277 INFO:tasks.workunit.client.1.smithi189.stdout:3/345: mknod d2/d1c/d5c/c7e 0 2024-04-20T21:32:09.277 INFO:tasks.workunit.client.1.smithi189.stdout:2/369: dread - d1/d11/d26/f5a zero size 2024-04-20T21:32:09.278 INFO:tasks.workunit.client.0.smithi062.stdout:1/771: write dc/d15/d20/d28/d83/dc0/fc2 [1348140,114618] 0 2024-04-20T21:32:09.279 INFO:tasks.workunit.client.1.smithi189.stdout:6/389: creat d8/d2b/d49/d54/f83 x:0 0 0 2024-04-20T21:32:09.280 INFO:tasks.workunit.client.0.smithi062.stdout:3/669: write d0/d5c/f5a [9140272,120118] 0 2024-04-20T21:32:09.281 INFO:tasks.workunit.client.0.smithi062.stdout:1/772: chown dc/d27/d32/f89 1654 1 2024-04-20T21:32:09.283 INFO:tasks.workunit.client.0.smithi062.stdout:9/753: creat d16/d3e/d77/d84/ffa x:0 0 0 2024-04-20T21:32:09.283 INFO:tasks.workunit.client.0.smithi062.stdout:5/700: mknod d8/d1e/d2c/ce9 0 2024-04-20T21:32:09.285 INFO:tasks.workunit.client.1.smithi189.stdout:0/487: readlink d3/df/l9f 0 2024-04-20T21:32:09.288 INFO:tasks.workunit.client.1.smithi189.stdout:2/370: fdatasync d1/d17/d2f/f31 0 2024-04-20T21:32:09.288 INFO:tasks.workunit.client.0.smithi062.stdout:9/754: creat d16/d4e/ffb x:0 0 0 2024-04-20T21:32:09.290 INFO:tasks.workunit.client.0.smithi062.stdout:1/773: unlink dc/d15/d75/d96/dd4/f5f 0 2024-04-20T21:32:09.296 INFO:tasks.workunit.client.1.smithi189.stdout:5/331: dwrite d7/d9/d30/f2d [0,4194304] 0 2024-04-20T21:32:09.298 INFO:tasks.workunit.client.1.smithi189.stdout:8/474: dwrite da/df/d3c/d50/f6d [0,4194304] 0 2024-04-20T21:32:09.299 INFO:tasks.workunit.client.1.smithi189.stdout:5/332: truncate d7/f5d 99968 0 2024-04-20T21:32:09.300 INFO:tasks.workunit.client.1.smithi189.stdout:5/333: chown d7/d35/f46 45 1 2024-04-20T21:32:09.302 INFO:tasks.workunit.client.1.smithi189.stdout:5/334: write d7/d14/d18/f3e [3883316,38704] 0 2024-04-20T21:32:09.302 INFO:tasks.workunit.client.0.smithi062.stdout:9/755: symlink d16/d3e/lfc 0 2024-04-20T21:32:09.302 INFO:tasks.workunit.client.1.smithi189.stdout:2/371: read d1/d17/f35 [2764413,41117] 0 2024-04-20T21:32:09.303 INFO:tasks.workunit.client.0.smithi062.stdout:6/599: write d0/d9/f13 [4968083,97484] 0 2024-04-20T21:32:09.303 INFO:tasks.workunit.client.1.smithi189.stdout:5/335: read d7/f2b [2963929,51546] 0 2024-04-20T21:32:09.304 INFO:tasks.workunit.client.0.smithi062.stdout:9/756: fdatasync d16/d4e/fbd 0 2024-04-20T21:32:09.309 INFO:tasks.workunit.client.0.smithi062.stdout:6/600: write d0/d52/f98 [626753,113246] 0 2024-04-20T21:32:09.309 INFO:tasks.workunit.client.0.smithi062.stdout:4/756: dwrite da/dd/df/f70 [4194304,4194304] 0 2024-04-20T21:32:09.309 INFO:tasks.workunit.client.0.smithi062.stdout:3/670: link d0/d11/f56 d0/d1/d3c/d43/d72/dd3/fe0 0 2024-04-20T21:32:09.310 INFO:tasks.workunit.client.0.smithi062.stdout:5/701: link d8/d38/d7a/d4c/d7d/d9d/dac/d66/fcb d8/d38/d7a/d4c/d7d/d9d/dac/d2f/fea 0 2024-04-20T21:32:09.311 INFO:tasks.workunit.client.0.smithi062.stdout:1/774: mkdir dc/d15/d75/d96/dd4/d3a/da2/de4/d84/dbe/d119 0 2024-04-20T21:32:09.313 INFO:tasks.workunit.client.0.smithi062.stdout:9/757: mkdir d16/d2c/d42/da6/da9/dfd 0 2024-04-20T21:32:09.318 INFO:tasks.workunit.client.0.smithi062.stdout:3/671: mkdir d0/d1/d3c/d43/d67/de1 0 2024-04-20T21:32:09.318 INFO:tasks.workunit.client.0.smithi062.stdout:6/601: write d0/d9/d11/d5c/d7c/f7f [4102890,68787] 0 2024-04-20T21:32:09.319 INFO:tasks.workunit.client.1.smithi189.stdout:6/390: link f2 d8/d10/d78/d7e/f84 0 2024-04-20T21:32:09.320 INFO:tasks.workunit.client.0.smithi062.stdout:9/758: chown d16/d2c/d5e/d75/d76 73 1 2024-04-20T21:32:09.321 INFO:tasks.workunit.client.0.smithi062.stdout:1/775: creat dc/d15/d20/d28/d83/dce/f11a x:0 0 0 2024-04-20T21:32:09.324 INFO:tasks.workunit.client.0.smithi062.stdout:4/757: creat da/d4f/f10b x:0 0 0 2024-04-20T21:32:09.325 INFO:tasks.workunit.client.1.smithi189.stdout:3/346: link d2/d1c/c2f d2/d3/d2c/d31/d4c/c7f 0 2024-04-20T21:32:09.325 INFO:tasks.workunit.client.0.smithi062.stdout:9/759: write d16/f36 [1417196,22168] 0 2024-04-20T21:32:09.325 INFO:tasks.workunit.client.0.smithi062.stdout:1/776: fsync dc/d15/d20/fe6 0 2024-04-20T21:32:09.326 INFO:tasks.workunit.client.1.smithi189.stdout:0/488: write d3/df/d30/d45/f59 [1212663,68081] 0 2024-04-20T21:32:09.327 INFO:tasks.workunit.client.1.smithi189.stdout:3/347: write d2/d3/f48 [5495612,8629] 0 2024-04-20T21:32:09.328 INFO:tasks.workunit.client.0.smithi062.stdout:3/672: fsync d0/d1/d3c/d43/d72/dd3/d7b/f68 0 2024-04-20T21:32:09.333 INFO:tasks.workunit.client.1.smithi189.stdout:8/475: mkdir da/df/d3c/d68/d8b/dac 0 2024-04-20T21:32:09.336 INFO:tasks.workunit.client.1.smithi189.stdout:5/336: rename d7/d35/c3f to d7/d14/c5e 0 2024-04-20T21:32:09.336 INFO:tasks.workunit.client.0.smithi062.stdout:9/760: mkdir d16/d2c/d5e/d75/d76/dfe 0 2024-04-20T21:32:09.344 INFO:tasks.workunit.client.0.smithi062.stdout:6/602: rename d0/d9/d11/d5c to d0/dc7 0 2024-04-20T21:32:09.345 INFO:tasks.workunit.client.1.smithi189.stdout:5/337: read d7/d14/d18/f17 [2256811,84101] 0 2024-04-20T21:32:09.350 INFO:tasks.workunit.client.0.smithi062.stdout:5/702: write d8/d1e/d2c/d6c/f65 [4993274,90147] 0 2024-04-20T21:32:09.351 INFO:tasks.workunit.client.1.smithi189.stdout:5/338: mkdir d7/d9/d31/d36/d5f 0 2024-04-20T21:32:09.352 INFO:tasks.workunit.client.0.smithi062.stdout:4/758: rmdir da/d4f 39 2024-04-20T21:32:09.354 INFO:tasks.workunit.client.0.smithi062.stdout:6/603: symlink d0/d9/lc8 0 2024-04-20T21:32:09.355 INFO:tasks.workunit.client.0.smithi062.stdout:9/761: write d16/d3d/d55/fb7 [4180602,63539] 0 2024-04-20T21:32:09.355 INFO:tasks.workunit.client.1.smithi189.stdout:0/489: truncate d3/df/d30/d45/f6c 37788 0 2024-04-20T21:32:09.357 INFO:tasks.workunit.client.0.smithi062.stdout:1/777: dread dc/d15/d20/d28/d83/dc0/fc2 [0,4194304] 0 2024-04-20T21:32:09.357 INFO:tasks.workunit.client.0.smithi062.stdout:4/759: mknod da/d56/d71/c10c 0 2024-04-20T21:32:09.358 INFO:tasks.workunit.client.1.smithi189.stdout:5/339: unlink d7/cf 0 2024-04-20T21:32:09.358 INFO:tasks.workunit.client.0.smithi062.stdout:3/673: dwrite d0/d1b/d74/dc7/fd2 [0,4194304] 0 2024-04-20T21:32:09.361 INFO:tasks.workunit.client.1.smithi189.stdout:0/490: write d3/df/d30/d45/d60/f8b [1019812,96443] 0 2024-04-20T21:32:09.362 INFO:tasks.workunit.client.0.smithi062.stdout:9/762: fdatasync d16/d4e/fa3 0 2024-04-20T21:32:09.362 INFO:tasks.workunit.client.1.smithi189.stdout:7/381: dwrite d5/d21/d25/d34/f6f [0,4194304] 0 2024-04-20T21:32:09.366 INFO:tasks.workunit.client.1.smithi189.stdout:6/391: dread d8/da/f32 [0,4194304] 0 2024-04-20T21:32:09.367 INFO:tasks.workunit.client.1.smithi189.stdout:5/340: creat d7/d35/d48/f60 x:0 0 0 2024-04-20T21:32:09.368 INFO:tasks.workunit.client.1.smithi189.stdout:0/491: mkdir d3/da8 0 2024-04-20T21:32:09.371 INFO:tasks.workunit.client.0.smithi062.stdout:4/760: mknod da/d56/d71/c10d 0 2024-04-20T21:32:09.372 INFO:tasks.workunit.client.1.smithi189.stdout:8/476: dwrite da/df/d3c/d50/f61 [0,4194304] 0 2024-04-20T21:32:09.372 INFO:tasks.workunit.client.1.smithi189.stdout:6/392: creat d8/da/dd/d2a/f85 x:0 0 0 2024-04-20T21:32:09.373 INFO:tasks.workunit.client.1.smithi189.stdout:6/393: stat d8/f63 0 2024-04-20T21:32:09.374 INFO:tasks.workunit.client.0.smithi062.stdout:1/778: symlink dc/d15/d20/d28/d76/l11b 0 2024-04-20T21:32:09.374 INFO:tasks.workunit.client.0.smithi062.stdout:4/761: write da/dd/f35 [893573,108215] 0 2024-04-20T21:32:09.376 INFO:tasks.workunit.client.0.smithi062.stdout:9/763: symlink d16/d3e/d77/d84/lff 0 2024-04-20T21:32:09.378 INFO:tasks.workunit.client.1.smithi189.stdout:5/341: mkdir d7/d9/d30/d55/d61 0 2024-04-20T21:32:09.379 INFO:tasks.workunit.client.1.smithi189.stdout:5/342: fsync d7/f5d 0 2024-04-20T21:32:09.379 INFO:tasks.workunit.client.0.smithi062.stdout:3/674: unlink d0/d1b/d74/fd8 0 2024-04-20T21:32:09.380 INFO:tasks.workunit.client.1.smithi189.stdout:8/477: symlink da/d81/lad 0 2024-04-20T21:32:09.380 INFO:tasks.workunit.client.0.smithi062.stdout:1/779: write dc/d15/d75/d94/fec [73398,113637] 0 2024-04-20T21:32:09.382 INFO:tasks.workunit.client.1.smithi189.stdout:8/478: dread - da/df/d20/d27/f54 zero size 2024-04-20T21:32:09.382 INFO:tasks.workunit.client.1.smithi189.stdout:0/492: unlink d3/df/d30/l52 0 2024-04-20T21:32:09.383 INFO:tasks.workunit.client.0.smithi062.stdout:4/762: write da/dd/df/fc7 [4357595,67936] 0 2024-04-20T21:32:09.383 INFO:tasks.workunit.client.0.smithi062.stdout:6/604: dread d0/d4f/f73 [0,4194304] 0 2024-04-20T21:32:09.385 INFO:tasks.workunit.client.0.smithi062.stdout:3/675: rmdir d0/d1/d10/d31 39 2024-04-20T21:32:09.387 INFO:tasks.workunit.client.0.smithi062.stdout:4/763: creat da/dd/d15/d6d/f10e x:0 0 0 2024-04-20T21:32:09.389 INFO:tasks.workunit.client.1.smithi189.stdout:5/343: creat d7/d9/f62 x:0 0 0 2024-04-20T21:32:09.392 INFO:tasks.workunit.client.1.smithi189.stdout:9/389: sync 2024-04-20T21:32:09.394 INFO:tasks.workunit.client.0.smithi062.stdout:6/605: creat d0/d9/d11/d7b/fc9 x:0 0 0 2024-04-20T21:32:09.395 INFO:tasks.workunit.client.1.smithi189.stdout:6/394: dread d8/da/dd/d35/f31 [0,4194304] 0 2024-04-20T21:32:09.396 INFO:tasks.workunit.client.1.smithi189.stdout:6/395: chown d8/d2b/d49 30926 1 2024-04-20T21:32:09.398 INFO:tasks.workunit.client.0.smithi062.stdout:3/676: link d0/d1/fd d0/d1c/d63/d42/fe2 0 2024-04-20T21:32:09.401 INFO:tasks.workunit.client.1.smithi189.stdout:8/479: mknod da/df/d13/cae 0 2024-04-20T21:32:09.403 INFO:tasks.workunit.client.0.smithi062.stdout:6/606: link d0/f79 d0/d2c/d67/fca 0 2024-04-20T21:32:09.410 INFO:tasks.workunit.client.0.smithi062.stdout:9/764: dwrite d16/d2f/f8f [0,4194304] 0 2024-04-20T21:32:09.413 INFO:tasks.workunit.client.0.smithi062.stdout:9/765: write d16/d2f/fc8 [1355674,94376] 0 2024-04-20T21:32:09.413 INFO:tasks.workunit.client.0.smithi062.stdout:4/764: dwrite da/dd/f35 [0,4194304] 0 2024-04-20T21:32:09.414 INFO:tasks.workunit.client.1.smithi189.stdout:7/382: truncate d5/d21/d25/d34/f6f 2607814 0 2024-04-20T21:32:09.415 INFO:tasks.workunit.client.1.smithi189.stdout:5/344: symlink d7/d9/d51/l63 0 2024-04-20T21:32:09.417 INFO:tasks.workunit.client.0.smithi062.stdout:6/607: creat d0/d9/fcb x:0 0 0 2024-04-20T21:32:09.422 INFO:tasks.workunit.client.1.smithi189.stdout:7/383: creat d5/d59/f80 x:0 0 0 2024-04-20T21:32:09.422 INFO:tasks.workunit.client.1.smithi189.stdout:7/384: chown d5/d59 1374 1 2024-04-20T21:32:09.423 INFO:tasks.workunit.client.1.smithi189.stdout:0/493: write d3/f66 [549719,99042] 0 2024-04-20T21:32:09.424 INFO:tasks.workunit.client.0.smithi062.stdout:9/766: mkdir d16/d2c/d42/d100 0 2024-04-20T21:32:09.427 INFO:tasks.workunit.client.1.smithi189.stdout:7/385: rename d5/d21/d4a/d52 to d5/d21/d4a/d52/d81 22 2024-04-20T21:32:09.427 INFO:tasks.workunit.client.0.smithi062.stdout:3/677: dwrite d0/d1/d3c/d43/d72/dd3/fbc [0,4194304] 0 2024-04-20T21:32:09.431 INFO:tasks.workunit.client.0.smithi062.stdout:4/765: getdents da/d56/d71/d9e 0 2024-04-20T21:32:09.435 INFO:tasks.workunit.client.0.smithi062.stdout:9/767: mkdir d16/d2c/d5e/d75/d76/dfe/d101 0 2024-04-20T21:32:09.435 INFO:tasks.workunit.client.0.smithi062.stdout:3/678: read - d0/d1/d10/fa4 zero size 2024-04-20T21:32:09.435 INFO:tasks.workunit.client.1.smithi189.stdout:6/396: dwrite d8/d2b/d5c/f5e [0,4194304] 0 2024-04-20T21:32:09.439 INFO:tasks.workunit.client.0.smithi062.stdout:9/768: mknod d16/d2c/d5e/c102 0 2024-04-20T21:32:09.439 INFO:tasks.workunit.client.1.smithi189.stdout:6/397: fdatasync d8/da/dd/d35/f59 0 2024-04-20T21:32:09.441 INFO:tasks.workunit.client.0.smithi062.stdout:4/766: creat da/dd/df/f10f x:0 0 0 2024-04-20T21:32:09.443 INFO:tasks.workunit.client.0.smithi062.stdout:9/769: write d16/d2f/d5b/fa0 [626677,122358] 0 2024-04-20T21:32:09.448 INFO:tasks.workunit.client.1.smithi189.stdout:6/398: creat d8/d2b/d49/d54/d6e/f86 x:0 0 0 2024-04-20T21:32:09.452 INFO:tasks.workunit.client.0.smithi062.stdout:4/767: creat da/dd/df/d27/f110 x:0 0 0 2024-04-20T21:32:09.454 INFO:tasks.workunit.client.0.smithi062.stdout:9/770: rename d16/d3d/d55/d6e/d7d/l98 to d16/d2f/l103 0 2024-04-20T21:32:09.458 INFO:tasks.workunit.client.1.smithi189.stdout:7/386: dread d5/d1d/d64/f7b [0,4194304] 0 2024-04-20T21:32:09.459 INFO:tasks.workunit.client.1.smithi189.stdout:5/345: truncate d7/d14/f49 2190549 0 2024-04-20T21:32:09.459 INFO:tasks.workunit.client.1.smithi189.stdout:6/399: creat d8/d2b/f87 x:0 0 0 2024-04-20T21:32:09.459 INFO:tasks.workunit.client.0.smithi062.stdout:4/768: link da/d56/d71/fe2 da/dd/d15/d4b/ddb/f111 0 2024-04-20T21:32:09.459 INFO:tasks.workunit.client.0.smithi062.stdout:9/771: getdents d16/d2c/d42/da6/da9/dfd 0 2024-04-20T21:32:09.460 INFO:tasks.workunit.client.1.smithi189.stdout:5/346: stat d7/d14/d4e 0 2024-04-20T21:32:09.460 INFO:tasks.workunit.client.1.smithi189.stdout:6/400: write d8/f21 [553456,41424] 0 2024-04-20T21:32:09.462 INFO:tasks.workunit.client.1.smithi189.stdout:7/387: mknod d5/d21/d4a/d52/d74/c82 0 2024-04-20T21:32:09.464 INFO:tasks.workunit.client.0.smithi062.stdout:9/772: creat d16/d2c/d5e/d96/f104 x:0 0 0 2024-04-20T21:32:09.465 INFO:tasks.workunit.client.1.smithi189.stdout:7/388: write d5/fe [615574,94881] 0 2024-04-20T21:32:09.469 INFO:tasks.workunit.client.1.smithi189.stdout:7/389: chown d5/d46/d7d/d45 252 1 2024-04-20T21:32:09.469 INFO:tasks.workunit.client.1.smithi189.stdout:6/401: truncate d8/d2b/d49/d54/d6e/f86 302357 0 2024-04-20T21:32:09.471 INFO:tasks.workunit.client.1.smithi189.stdout:7/390: chown d5/f6 222 1 2024-04-20T21:32:09.475 INFO:tasks.workunit.client.0.smithi062.stdout:6/608: write d0/dc7/f94 [4725193,83328] 0 2024-04-20T21:32:09.475 INFO:tasks.workunit.client.1.smithi189.stdout:7/391: truncate d5/f49 1676594 0 2024-04-20T21:32:09.477 INFO:tasks.workunit.client.1.smithi189.stdout:9/390: dread d9/dc/d17/d36/f57 [0,4194304] 0 2024-04-20T21:32:09.478 INFO:tasks.workunit.client.1.smithi189.stdout:7/392: readlink d5/d21/d4a/l54 0 2024-04-20T21:32:09.480 INFO:tasks.workunit.client.1.smithi189.stdout:5/347: truncate d7/d14/d18/f17 3656797 0 2024-04-20T21:32:09.480 INFO:tasks.workunit.client.1.smithi189.stdout:7/393: write d5/f1a [4001556,68541] 0 2024-04-20T21:32:09.481 INFO:tasks.workunit.client.0.smithi062.stdout:6/609: creat d0/d52/da0/fcc x:0 0 0 2024-04-20T21:32:09.481 INFO:tasks.workunit.client.0.smithi062.stdout:9/773: dwrite d16/d4e/fbd [0,4194304] 0 2024-04-20T21:32:09.482 INFO:tasks.workunit.client.1.smithi189.stdout:0/494: rename d3/df/d30/d45/l4a to d3/df/d30/d45/d60/la9 0 2024-04-20T21:32:09.487 INFO:tasks.workunit.client.1.smithi189.stdout:9/391: creat d9/d12/d50/f80 x:0 0 0 2024-04-20T21:32:09.487 INFO:tasks.workunit.client.0.smithi062.stdout:4/769: truncate da/dd/df/f28 7509012 0 2024-04-20T21:32:09.488 INFO:tasks.workunit.client.1.smithi189.stdout:7/394: symlink d5/d59/l83 0 2024-04-20T21:32:09.488 INFO:tasks.workunit.client.0.smithi062.stdout:6/610: creat d0/dc7/fcd x:0 0 0 2024-04-20T21:32:09.489 INFO:tasks.workunit.client.0.smithi062.stdout:4/770: readlink da/dd/d15/l55 0 2024-04-20T21:32:09.489 INFO:tasks.workunit.client.0.smithi062.stdout:9/774: mkdir d16/d2c/d62/dd1/d105 0 2024-04-20T21:32:09.490 INFO:tasks.workunit.client.1.smithi189.stdout:5/348: creat d7/d9/d30/d5a/f64 x:0 0 0 2024-04-20T21:32:09.492 INFO:tasks.workunit.client.0.smithi062.stdout:6/611: rmdir d0/d9/d11/d7b 39 2024-04-20T21:32:09.492 INFO:tasks.workunit.client.1.smithi189.stdout:9/392: mknod d9/dc/d17/d2b/d52/c81 0 2024-04-20T21:32:09.494 INFO:tasks.workunit.client.1.smithi189.stdout:5/349: write d7/d14/f4c [772172,3640] 0 2024-04-20T21:32:09.494 INFO:tasks.workunit.client.1.smithi189.stdout:7/395: mknod d5/d21/d4a/d52/d74/c84 0 2024-04-20T21:32:09.498 INFO:tasks.workunit.client.1.smithi189.stdout:7/396: chown d5/d21/d4a/d52/d74/c82 780407992 1 2024-04-20T21:32:09.499 INFO:tasks.workunit.client.1.smithi189.stdout:5/350: write d7/d9/d28/f58 [678392,51369] 0 2024-04-20T21:32:09.499 INFO:tasks.workunit.client.1.smithi189.stdout:9/393: unlink d9/d12/d15/f31 0 2024-04-20T21:32:09.500 INFO:tasks.workunit.client.1.smithi189.stdout:7/397: dread - d5/d59/f80 zero size 2024-04-20T21:32:09.501 INFO:tasks.workunit.client.0.smithi062.stdout:4/771: dread da/dd/d15/d50/d64/d57/d5b/f6f [0,4194304] 0 2024-04-20T21:32:09.502 INFO:tasks.workunit.client.0.smithi062.stdout:6/612: truncate d0/d9/f71 572151 0 2024-04-20T21:32:09.504 INFO:tasks.workunit.client.1.smithi189.stdout:9/394: mkdir d9/dc/d17/d2b/d52/d82 0 2024-04-20T21:32:09.506 INFO:tasks.workunit.client.1.smithi189.stdout:7/398: creat d5/d1d/d64/f85 x:0 0 0 2024-04-20T21:32:09.507 INFO:tasks.workunit.client.0.smithi062.stdout:4/772: creat da/dd/df/d109/f112 x:0 0 0 2024-04-20T21:32:09.508 INFO:tasks.workunit.client.1.smithi189.stdout:9/395: fdatasync d9/d12/d3f/d72/f7b 0 2024-04-20T21:32:09.509 INFO:tasks.workunit.client.1.smithi189.stdout:7/399: rmdir d5/d21/d25/d34 39 2024-04-20T21:32:09.511 INFO:tasks.workunit.client.0.smithi062.stdout:4/773: readlink da/dd/df/d27/d3f/d82/le6 0 2024-04-20T21:32:09.512 INFO:tasks.workunit.client.1.smithi189.stdout:6/402: dread d8/d2b/f61 [0,4194304] 0 2024-04-20T21:32:09.516 INFO:tasks.workunit.client.1.smithi189.stdout:9/396: creat d9/dc/d17/d2b/d52/f83 x:0 0 0 2024-04-20T21:32:09.517 INFO:tasks.workunit.client.0.smithi062.stdout:4/774: mkdir da/d4f/d113 0 2024-04-20T21:32:09.518 INFO:tasks.workunit.client.1.smithi189.stdout:7/400: mkdir d5/d1d/d77/d5b/d86 0 2024-04-20T21:32:09.521 INFO:tasks.workunit.client.1.smithi189.stdout:9/397: unlink d9/dc/d69/f6c 0 2024-04-20T21:32:09.522 INFO:tasks.workunit.client.0.smithi062.stdout:9/775: dread fb [0,4194304] 0 2024-04-20T21:32:09.524 INFO:tasks.workunit.client.0.smithi062.stdout:6/613: dwrite d0/d9/f47 [4194304,4194304] 0 2024-04-20T21:32:09.527 INFO:tasks.workunit.client.0.smithi062.stdout:6/614: write d0/d52/f98 [931289,99283] 0 2024-04-20T21:32:09.529 INFO:tasks.workunit.client.0.smithi062.stdout:6/615: stat d0/d52/da0/fa8 0 2024-04-20T21:32:09.530 INFO:tasks.workunit.client.1.smithi189.stdout:9/398: truncate d9/dc/f3d 9349992 0 2024-04-20T21:32:09.530 INFO:tasks.workunit.client.1.smithi189.stdout:6/403: dwrite d8/d2b/d49/d54/f83 [0,4194304] 0 2024-04-20T21:32:09.532 INFO:tasks.workunit.client.1.smithi189.stdout:6/404: chown c3 466 1 2024-04-20T21:32:09.533 INFO:tasks.workunit.client.0.smithi062.stdout:9/776: link d16/f19 d16/d4e/f106 0 2024-04-20T21:32:09.538 INFO:tasks.workunit.client.1.smithi189.stdout:9/399: mknod d9/dc/d17/d7e/c84 0 2024-04-20T21:32:09.539 INFO:tasks.workunit.client.1.smithi189.stdout:6/405: chown d8/da/dd/d2a/d3b/c3c 8730305 1 2024-04-20T21:32:09.544 INFO:tasks.workunit.client.1.smithi189.stdout:7/401: dwrite d5/d21/f32 [4194304,4194304] 0 2024-04-20T21:32:09.550 INFO:tasks.workunit.client.1.smithi189.stdout:9/400: unlink d9/dc/d17/f5e 0 2024-04-20T21:32:09.551 INFO:tasks.workunit.client.0.smithi062.stdout:6/616: dwrite d0/d9a/fae [0,4194304] 0 2024-04-20T21:32:09.551 INFO:tasks.workunit.client.0.smithi062.stdout:9/777: rmdir d16/d4e 39 2024-04-20T21:32:09.556 INFO:tasks.workunit.client.0.smithi062.stdout:4/775: dread da/dd/d15/d4b/fd9 [0,4194304] 0 2024-04-20T21:32:09.556 INFO:tasks.workunit.client.1.smithi189.stdout:7/402: mknod d5/d21/d25/c87 0 2024-04-20T21:32:09.562 INFO:tasks.workunit.client.1.smithi189.stdout:9/401: creat d9/d12/d3f/d72/f85 x:0 0 0 2024-04-20T21:32:09.565 INFO:tasks.workunit.client.1.smithi189.stdout:7/403: link d5/d59/c73 d5/d21/c88 0 2024-04-20T21:32:09.569 INFO:tasks.workunit.client.1.smithi189.stdout:9/402: creat d9/dc/d17/d2b/d56/f86 x:0 0 0 2024-04-20T21:32:09.570 INFO:tasks.workunit.client.1.smithi189.stdout:6/406: dwrite d8/d10/f3f [0,4194304] 0 2024-04-20T21:32:09.572 INFO:tasks.workunit.client.1.smithi189.stdout:9/403: read d9/d12/d3b/f4c [1458591,97093] 0 2024-04-20T21:32:09.573 INFO:tasks.workunit.client.1.smithi189.stdout:7/404: symlink d5/d1d/d77/d5b/d86/l89 0 2024-04-20T21:32:09.573 INFO:tasks.workunit.client.1.smithi189.stdout:9/404: fdatasync d9/dc/f2c 0 2024-04-20T21:32:09.574 INFO:tasks.workunit.client.1.smithi189.stdout:6/407: creat d8/da/dd/f88 x:0 0 0 2024-04-20T21:32:09.575 INFO:tasks.workunit.client.1.smithi189.stdout:7/405: rmdir d5 39 2024-04-20T21:32:09.576 INFO:tasks.workunit.client.1.smithi189.stdout:9/405: unlink f5 0 2024-04-20T21:32:09.578 INFO:tasks.workunit.client.1.smithi189.stdout:9/406: write d9/dc/f1c [314129,111009] 0 2024-04-20T21:32:09.578 INFO:tasks.workunit.client.1.smithi189.stdout:6/408: creat d8/da/dd/d2a/f89 x:0 0 0 2024-04-20T21:32:09.581 INFO:tasks.workunit.client.1.smithi189.stdout:7/406: mknod d5/d21/c8a 0 2024-04-20T21:32:09.583 INFO:tasks.workunit.client.1.smithi189.stdout:6/409: creat d8/d12/d30/d47/f8a x:0 0 0 2024-04-20T21:32:09.584 INFO:tasks.workunit.client.0.smithi062.stdout:9/778: unlink d16/d3e/fdf 0 2024-04-20T21:32:09.584 INFO:tasks.workunit.client.0.smithi062.stdout:6/617: truncate d0/d2c/d67/f6a 860834 0 2024-04-20T21:32:09.585 INFO:tasks.workunit.client.0.smithi062.stdout:9/779: fsync d16/d2c/d5e/f7b 0 2024-04-20T21:32:09.587 INFO:tasks.workunit.client.1.smithi189.stdout:6/410: write d8/da/dd/d35/f62 [752735,94053] 0 2024-04-20T21:32:09.588 INFO:tasks.workunit.client.0.smithi062.stdout:6/618: creat d0/d52/fce x:0 0 0 2024-04-20T21:32:09.589 INFO:tasks.workunit.client.1.smithi189.stdout:7/407: rename d5/d1d/f62 to d5/d46/f8b 0 2024-04-20T21:32:09.590 INFO:tasks.workunit.client.0.smithi062.stdout:6/619: dread - d0/d9/d11/d22/fbe zero size 2024-04-20T21:32:09.591 INFO:tasks.workunit.client.0.smithi062.stdout:6/620: write d0/f91 [94240,24125] 0 2024-04-20T21:32:09.591 INFO:tasks.workunit.client.1.smithi189.stdout:6/411: write d8/da/f6b [973835,60898] 0 2024-04-20T21:32:09.594 INFO:tasks.workunit.client.1.smithi189.stdout:7/408: mknod d5/d21/d35/c8c 0 2024-04-20T21:32:09.597 INFO:tasks.workunit.client.1.smithi189.stdout:7/409: unlink d5/d46/d7d/d45/f76 0 2024-04-20T21:32:09.597 INFO:tasks.workunit.client.1.smithi189.stdout:6/412: dread d8/d2b/f3a [0,4194304] 0 2024-04-20T21:32:09.598 INFO:tasks.workunit.client.1.smithi189.stdout:7/410: write d5/f57 [183017,124279] 0 2024-04-20T21:32:09.599 INFO:tasks.workunit.client.0.smithi062.stdout:9/780: dwrite d16/d2f/d5b/fa0 [0,4194304] 0 2024-04-20T21:32:09.601 INFO:tasks.workunit.client.0.smithi062.stdout:9/781: stat d16/d2c/d5e/d75/faa 0 2024-04-20T21:32:09.602 INFO:tasks.workunit.client.1.smithi189.stdout:7/411: creat d5/d21/d4a/f8d x:0 0 0 2024-04-20T21:32:09.602 INFO:tasks.workunit.client.1.smithi189.stdout:6/413: symlink d8/d2b/l8b 0 2024-04-20T21:32:09.603 INFO:tasks.workunit.client.1.smithi189.stdout:4/374: sync 2024-04-20T21:32:09.604 INFO:tasks.workunit.client.1.smithi189.stdout:7/412: write d5/f7 [83198,90485] 0 2024-04-20T21:32:09.605 INFO:tasks.workunit.client.1.smithi189.stdout:6/414: fdatasync d8/d10/f36 0 2024-04-20T21:32:09.607 INFO:tasks.workunit.client.1.smithi189.stdout:4/375: read - d2/dd/d11/f4b zero size 2024-04-20T21:32:09.608 INFO:tasks.workunit.client.0.smithi062.stdout:9/782: read d16/d3d/d55/f73 [2171614,39787] 0 2024-04-20T21:32:09.612 INFO:tasks.workunit.client.0.smithi062.stdout:9/783: write d16/d2c/d5e/f87 [3177041,70460] 0 2024-04-20T21:32:09.616 INFO:tasks.workunit.client.0.smithi062.stdout:9/784: rmdir d16/d2c/d42 39 2024-04-20T21:32:09.616 INFO:tasks.workunit.client.1.smithi189.stdout:9/407: dwrite f3 [0,4194304] 0 2024-04-20T21:32:09.621 INFO:tasks.workunit.client.1.smithi189.stdout:9/408: truncate d9/dc/d17/d2b/d56/f86 519049 0 2024-04-20T21:32:09.628 INFO:tasks.workunit.client.0.smithi062.stdout:4/776: write da/dd/d15/d50/d64/d3d/f5e [1240918,4399] 0 2024-04-20T21:32:09.643 INFO:tasks.workunit.client.0.smithi062.stdout:4/777: dwrite da/dd/df/d27/d32/ded/f105 [0,4194304] 0 2024-04-20T21:32:09.645 INFO:tasks.workunit.client.0.smithi062.stdout:9/785: dread d16/d2c/d5e/d75/faa [0,4194304] 0 2024-04-20T21:32:09.645 INFO:tasks.workunit.client.0.smithi062.stdout:4/778: readlink da/dd/df/d27/d3f/d82/le6 0 2024-04-20T21:32:09.647 INFO:tasks.workunit.client.0.smithi062.stdout:9/786: stat d16/d3d/d55/f73 0 2024-04-20T21:32:09.677 INFO:tasks.workunit.client.0.smithi062.stdout:4/779: dread da/dd/d15/fa7 [0,4194304] 0 2024-04-20T21:32:09.701 INFO:tasks.workunit.client.0.smithi062.stdout:9/787: readlink d16/d2c/d42/l57 0 2024-04-20T21:32:09.703 INFO:tasks.workunit.client.0.smithi062.stdout:9/788: fsync d16/d3d/d55/d6e/f72 0 2024-04-20T21:32:09.711 INFO:tasks.workunit.client.0.smithi062.stdout:9/789: truncate d16/d4e/f90 430007 0 2024-04-20T21:32:09.713 INFO:tasks.workunit.client.0.smithi062.stdout:9/790: dread - d16/d2c/d42/da6/da9/ff4 zero size 2024-04-20T21:32:09.716 INFO:tasks.workunit.client.0.smithi062.stdout:9/791: mknod d16/d2f/d5b/da4/c107 0 2024-04-20T21:32:09.727 INFO:tasks.workunit.client.1.smithi189.stdout:1/360: sync 2024-04-20T21:32:09.728 INFO:tasks.workunit.client.1.smithi189.stdout:2/372: sync 2024-04-20T21:32:09.729 INFO:tasks.workunit.client.0.smithi062.stdout:4/780: rmdir da/dd/df/d27/d3f 39 2024-04-20T21:32:09.729 INFO:tasks.workunit.client.1.smithi189.stdout:2/373: read d1/f2a [210457,130938] 0 2024-04-20T21:32:09.730 INFO:tasks.workunit.client.0.smithi062.stdout:4/781: write da/f7b [5718075,17401] 0 2024-04-20T21:32:09.735 INFO:tasks.workunit.client.0.smithi062.stdout:4/782: fdatasync da/dd/df/d27/d32/fb2 0 2024-04-20T21:32:09.736 INFO:tasks.workunit.client.0.smithi062.stdout:4/783: stat da/c61 0 2024-04-20T21:32:09.743 INFO:tasks.workunit.client.1.smithi189.stdout:0/495: sync 2024-04-20T21:32:09.743 INFO:tasks.workunit.client.1.smithi189.stdout:8/480: sync 2024-04-20T21:32:09.743 INFO:tasks.workunit.client.1.smithi189.stdout:3/348: sync 2024-04-20T21:32:09.743 INFO:tasks.workunit.client.1.smithi189.stdout:5/351: sync 2024-04-20T21:32:09.745 INFO:tasks.workunit.client.0.smithi062.stdout:9/792: dread f4 [0,4194304] 0 2024-04-20T21:32:09.746 INFO:tasks.workunit.client.0.smithi062.stdout:4/784: getdents da/dd/d15/d50/d64/d57/d5b/d9d/dea/dac/dce 0 2024-04-20T21:32:09.750 INFO:tasks.workunit.client.1.smithi189.stdout:0/496: chown d3/df/d28/l95 1785525 1 2024-04-20T21:32:09.753 INFO:tasks.workunit.client.1.smithi189.stdout:0/497: read d3/df/d30/d45/d60/f68 [453180,37579] 0 2024-04-20T21:32:09.754 INFO:tasks.workunit.client.1.smithi189.stdout:2/374: dwrite d1/d17/d2f/d55/f61 [0,4194304] 0 2024-04-20T21:32:09.756 INFO:tasks.workunit.client.0.smithi062.stdout:9/793: rename d16/d3e/d77/d84/ffa to d16/d2f/d5b/dab/f108 0 2024-04-20T21:32:09.763 INFO:tasks.workunit.client.0.smithi062.stdout:9/794: truncate f4 5950763 0 2024-04-20T21:32:09.768 INFO:tasks.workunit.client.0.smithi062.stdout:4/785: dwrite da/dd/d15/d6d/df5/f102 [0,4194304] 0 2024-04-20T21:32:09.769 INFO:tasks.workunit.client.0.smithi062.stdout:9/795: unlink d16/c58 0 2024-04-20T21:32:09.769 INFO:tasks.workunit.client.0.smithi062.stdout:4/786: chown da/dd/d15/d4b 1 1 2024-04-20T21:32:09.771 INFO:tasks.workunit.client.0.smithi062.stdout:4/787: read - da/dd/df/d109/f112 zero size 2024-04-20T21:32:09.772 INFO:tasks.workunit.client.1.smithi189.stdout:0/498: dwrite d3/df/d28/f6a [0,4194304] 0 2024-04-20T21:32:09.774 INFO:tasks.workunit.client.0.smithi062.stdout:9/796: unlink d16/d3e/d77/d84/f9a 0 2024-04-20T21:32:09.778 INFO:tasks.workunit.client.1.smithi189.stdout:3/349: dread d2/d8/f10 [0,4194304] 0 2024-04-20T21:32:09.779 INFO:tasks.workunit.client.0.smithi062.stdout:4/788: unlink da/dd/d15/d50/d64/d57/d5b/d9d/dea/dac/dce/fdd 0 2024-04-20T21:32:09.781 INFO:tasks.workunit.client.0.smithi062.stdout:4/789: write da/dd/d15/d50/fcb [5385856,31571] 0 2024-04-20T21:32:09.782 INFO:tasks.workunit.client.0.smithi062.stdout:9/797: creat d16/d2c/d62/dd1/d105/f109 x:0 0 0 2024-04-20T21:32:09.785 INFO:tasks.workunit.client.0.smithi062.stdout:9/798: truncate d16/d2f/f8f 4556482 0 2024-04-20T21:32:09.794 INFO:tasks.workunit.client.0.smithi062.stdout:9/799: getdents d16/d2c/d42 0 2024-04-20T21:32:09.795 INFO:tasks.workunit.client.0.smithi062.stdout:4/790: rename da/dd/df/d27/d7a/l7c to da/dd/df/d27/d7a/l114 0 2024-04-20T21:32:09.797 INFO:tasks.workunit.client.1.smithi189.stdout:4/376: symlink d2/dd/d11/l8b 0 2024-04-20T21:32:09.798 INFO:tasks.workunit.client.1.smithi189.stdout:7/413: symlink d5/d1d/d77/d5b/d71/l8e 0 2024-04-20T21:32:09.800 INFO:tasks.workunit.client.0.smithi062.stdout:9/800: link d16/d4e/d80/lae d16/d2c/d42/d100/l10a 0 2024-04-20T21:32:09.801 INFO:tasks.workunit.client.1.smithi189.stdout:1/361: rename d8/ld to d8/d21/d2c/d36/l7d 0 2024-04-20T21:32:09.801 INFO:tasks.workunit.client.1.smithi189.stdout:4/377: chown d2/d22/f6e 68 1 2024-04-20T21:32:09.801 INFO:tasks.workunit.client.0.smithi062.stdout:9/801: stat d16/d4e/f6f 0 2024-04-20T21:32:09.802 INFO:tasks.workunit.client.1.smithi189.stdout:5/352: mknod d7/d14/d4e/d5b/c65 0 2024-04-20T21:32:09.802 INFO:tasks.workunit.client.1.smithi189.stdout:3/350: dwrite d2/d3/d25/f5b [0,4194304] 0 2024-04-20T21:32:09.803 INFO:tasks.workunit.client.0.smithi062.stdout:9/802: write d16/dbe/fca [1183239,27716] 0 2024-04-20T21:32:09.804 INFO:tasks.workunit.client.1.smithi189.stdout:7/414: write d5/f14 [972386,38692] 0 2024-04-20T21:32:09.805 INFO:tasks.workunit.client.0.smithi062.stdout:9/803: chown d16/d2f/d69 1 1 2024-04-20T21:32:09.806 INFO:tasks.workunit.client.1.smithi189.stdout:4/378: write d2/d22/f30 [4742213,44948] 0 2024-04-20T21:32:09.807 INFO:tasks.workunit.client.1.smithi189.stdout:2/375: readlink d1/d17/d2f/d55/l5e 0 2024-04-20T21:32:09.807 INFO:tasks.workunit.client.1.smithi189.stdout:5/353: write d7/d14/d4e/f39 [3334408,99021] 0 2024-04-20T21:32:09.812 INFO:tasks.workunit.client.1.smithi189.stdout:6/415: rename d8/d2b/f61 to d8/d2b/f8c 0 2024-04-20T21:32:09.814 INFO:tasks.workunit.client.0.smithi062.stdout:9/804: link d16/d3d/d55/d6e/d7d/f91 d16/d2c/d5e/f10b 0 2024-04-20T21:32:09.815 INFO:tasks.workunit.client.1.smithi189.stdout:1/362: truncate d8/f11 5339274 0 2024-04-20T21:32:09.817 INFO:tasks.workunit.client.1.smithi189.stdout:2/376: dread d1/d17/d37/f67 [0,4194304] 0 2024-04-20T21:32:09.818 INFO:tasks.workunit.client.1.smithi189.stdout:3/351: symlink d2/d3/d25/l80 0 2024-04-20T21:32:09.820 INFO:tasks.workunit.client.1.smithi189.stdout:4/379: unlink d2/d22/d87/c49 0 2024-04-20T21:32:09.821 INFO:tasks.workunit.client.0.smithi062.stdout:4/791: dread da/dd/df/f17 [0,4194304] 0 2024-04-20T21:32:09.823 INFO:tasks.workunit.client.1.smithi189.stdout:5/354: creat d7/d14/d18/f66 x:0 0 0 2024-04-20T21:32:09.824 INFO:tasks.workunit.client.1.smithi189.stdout:6/416: symlink d8/d10/l8d 0 2024-04-20T21:32:09.826 INFO:tasks.workunit.client.0.smithi062.stdout:4/792: read da/dd/df/f70 [3590574,59298] 0 2024-04-20T21:32:09.829 INFO:tasks.workunit.client.1.smithi189.stdout:8/481: getdents da/df/d20/d2c 0 2024-04-20T21:32:09.833 INFO:tasks.workunit.client.1.smithi189.stdout:9/409: link d9/dc/d17/d36/f43 d9/f87 0 2024-04-20T21:32:09.833 INFO:tasks.workunit.client.0.smithi062.stdout:9/805: dwrite f15 [0,4194304] 0 2024-04-20T21:32:09.833 INFO:tasks.workunit.client.1.smithi189.stdout:7/415: creat d5/d46/f8f x:0 0 0 2024-04-20T21:32:09.834 INFO:tasks.workunit.client.1.smithi189.stdout:8/482: chown da/d81/d91/da8 772 1 2024-04-20T21:32:09.834 INFO:tasks.workunit.client.1.smithi189.stdout:6/417: creat d8/da/dd/d2a/d3b/f8e x:0 0 0 2024-04-20T21:32:09.837 INFO:tasks.workunit.client.1.smithi189.stdout:4/380: creat d2/dd/d11/d14/d69/d6a/d7a/f8c x:0 0 0 2024-04-20T21:32:09.837 INFO:tasks.workunit.client.0.smithi062.stdout:9/806: creat d16/d4e/f10c x:0 0 0 2024-04-20T21:32:09.839 INFO:tasks.workunit.client.0.smithi062.stdout:4/793: link da/dd/df/d27/d3f/fdf da/dd/df/d27/d32/d97/f115 0 2024-04-20T21:32:09.839 INFO:tasks.workunit.client.1.smithi189.stdout:8/483: dread - da/df/d20/f99 zero size 2024-04-20T21:32:09.841 INFO:tasks.workunit.client.0.smithi062.stdout:9/807: read - d16/d2c/d42/da6/fd2 zero size 2024-04-20T21:32:09.842 INFO:tasks.workunit.client.1.smithi189.stdout:0/499: write d3/df/d28/d2b/d2c/f9e [684280,85793] 0 2024-04-20T21:32:09.844 INFO:tasks.workunit.client.1.smithi189.stdout:5/355: dwrite d7/d35/f44 [0,4194304] 0 2024-04-20T21:32:09.846 INFO:tasks.workunit.client.0.smithi062.stdout:4/794: getdents da/dd/d15/d50/d64/d57/d5b/da3 0 2024-04-20T21:32:09.848 INFO:tasks.workunit.client.0.smithi062.stdout:9/808: mkdir d16/d2c/d10d 0 2024-04-20T21:32:09.850 INFO:tasks.workunit.client.0.smithi062.stdout:4/795: mkdir da/dd/d15/daf/d116 0 2024-04-20T21:32:09.851 INFO:tasks.workunit.client.0.smithi062.stdout:9/809: write d16/d2c/d5e/d75/fdd [994354,124164] 0 2024-04-20T21:32:09.854 INFO:tasks.workunit.client.1.smithi189.stdout:1/363: rename d8/d1e/d43/d58/d60/f68 to d8/d1e/d28/f7e 0 2024-04-20T21:32:09.854 INFO:tasks.workunit.client.1.smithi189.stdout:3/352: creat d2/f81 x:0 0 0 2024-04-20T21:32:09.858 INFO:tasks.workunit.client.1.smithi189.stdout:8/484: rmdir da/df/d3c 39 2024-04-20T21:32:09.861 INFO:tasks.workunit.client.1.smithi189.stdout:0/500: fsync d3/df/d28/d2b/f86 0 2024-04-20T21:32:09.861 INFO:tasks.workunit.client.1.smithi189.stdout:5/356: creat d7/d14/f67 x:0 0 0 2024-04-20T21:32:09.862 INFO:tasks.workunit.client.1.smithi189.stdout:1/364: rmdir d8/df 39 2024-04-20T21:32:09.862 INFO:tasks.workunit.client.1.smithi189.stdout:3/353: dread - d2/d3/f3e zero size 2024-04-20T21:32:09.863 INFO:tasks.workunit.client.1.smithi189.stdout:0/501: chown d3/df/d30/d37 1220872 1 2024-04-20T21:32:09.865 INFO:tasks.workunit.client.1.smithi189.stdout:2/377: rename d1/d11/d26 to d1/d11/d52/d6f 0 2024-04-20T21:32:09.865 INFO:tasks.workunit.client.1.smithi189.stdout:1/365: read - d8/d21/f38 zero size 2024-04-20T21:32:09.866 INFO:tasks.workunit.client.1.smithi189.stdout:9/410: truncate d9/dc/d17/d36/f43 1708022 0 2024-04-20T21:32:09.867 INFO:tasks.workunit.client.1.smithi189.stdout:5/357: creat d7/d14/d4e/d5b/f68 x:0 0 0 2024-04-20T21:32:09.870 INFO:tasks.workunit.client.1.smithi189.stdout:6/418: rename d8/da/c64 to d8/d2b/d49/d54/d6e/c8f 0 2024-04-20T21:32:09.870 INFO:tasks.workunit.client.1.smithi189.stdout:3/354: write d2/d3/f3e [800200,108209] 0 2024-04-20T21:32:09.870 INFO:tasks.workunit.client.1.smithi189.stdout:4/381: truncate d2/f4 6629863 0 2024-04-20T21:32:09.871 INFO:tasks.workunit.client.1.smithi189.stdout:2/378: write d1/d11/d52/f6a [651006,6037] 0 2024-04-20T21:32:09.872 INFO:tasks.workunit.client.1.smithi189.stdout:0/502: chown d3/df/d28/d2b/d2c/d74/ca6 4761155 1 2024-04-20T21:32:09.872 INFO:tasks.workunit.client.1.smithi189.stdout:6/419: readlink d8/d12/d30/l4a 0 2024-04-20T21:32:09.873 INFO:tasks.workunit.client.1.smithi189.stdout:5/358: rmdir d7/d9/d30/d5a 39 2024-04-20T21:32:09.874 INFO:tasks.workunit.client.0.smithi062.stdout:4/796: dwrite da/dd/d15/f1e [0,4194304] 0 2024-04-20T21:32:09.874 INFO:tasks.workunit.client.1.smithi189.stdout:8/485: fdatasync da/df/d3c/d50/f5e 0 2024-04-20T21:32:09.876 INFO:tasks.workunit.client.1.smithi189.stdout:7/416: rename d5/d46/c6b to d5/d46/c90 0 2024-04-20T21:32:09.876 INFO:tasks.workunit.client.1.smithi189.stdout:5/359: dread d7/f5d [0,4194304] 0 2024-04-20T21:32:09.880 INFO:tasks.workunit.client.1.smithi189.stdout:3/355: mkdir d2/d3/d65/d82 0 2024-04-20T21:32:09.882 INFO:tasks.workunit.client.0.smithi062.stdout:4/797: mknod da/dd/df/d27/d32/d97/c117 0 2024-04-20T21:32:09.884 INFO:tasks.workunit.client.0.smithi062.stdout:9/810: dread d16/d3d/d55/d6e/f92 [0,4194304] 0 2024-04-20T21:32:09.885 INFO:tasks.workunit.client.1.smithi189.stdout:2/379: mknod d1/d11/d52/c70 0 2024-04-20T21:32:09.885 INFO:tasks.workunit.client.0.smithi062.stdout:9/811: dread - d16/d4e/f10c zero size 2024-04-20T21:32:09.886 INFO:tasks.workunit.client.1.smithi189.stdout:8/486: rmdir da/d81/d91 39 2024-04-20T21:32:09.886 INFO:tasks.workunit.client.1.smithi189.stdout:2/380: readlink d1/d11/l56 0 2024-04-20T21:32:09.887 INFO:tasks.workunit.client.0.smithi062.stdout:4/798: symlink da/dd/df/d109/l118 0 2024-04-20T21:32:09.887 INFO:tasks.workunit.client.1.smithi189.stdout:4/382: unlink d2/dd/d37/l88 0 2024-04-20T21:32:09.891 INFO:tasks.workunit.client.1.smithi189.stdout:4/383: stat d2/dd/d11/d14/l51 0 2024-04-20T21:32:09.891 INFO:tasks.workunit.client.0.smithi062.stdout:4/799: creat da/dd/df/d27/d3f/f119 x:0 0 0 2024-04-20T21:32:09.891 INFO:tasks.workunit.client.1.smithi189.stdout:8/487: unlink da/d1f/ca7 0 2024-04-20T21:32:09.895 INFO:tasks.workunit.client.1.smithi189.stdout:5/360: truncate d7/d9/d30/d5a/f64 129985 0 2024-04-20T21:32:09.896 INFO:tasks.workunit.client.1.smithi189.stdout:4/384: write d2/d20/f24 [2700206,71046] 0 2024-04-20T21:32:09.896 INFO:tasks.workunit.client.1.smithi189.stdout:3/356: mkdir d2/d3/d83 0 2024-04-20T21:32:09.896 INFO:tasks.workunit.client.1.smithi189.stdout:6/420: creat d8/d2b/f90 x:0 0 0 2024-04-20T21:32:09.898 INFO:tasks.workunit.client.1.smithi189.stdout:7/417: dwrite d5/d21/d4a/f8d [0,4194304] 0 2024-04-20T21:32:09.900 INFO:tasks.workunit.client.1.smithi189.stdout:0/503: write d3/fe [3250275,124714] 0 2024-04-20T21:32:09.902 INFO:tasks.workunit.client.1.smithi189.stdout:5/361: symlink d7/d14/l69 0 2024-04-20T21:32:09.905 INFO:tasks.workunit.client.1.smithi189.stdout:5/362: dread - d7/d9/f62 zero size 2024-04-20T21:32:09.906 INFO:tasks.workunit.client.1.smithi189.stdout:7/418: fsync d5/f6 0 2024-04-20T21:32:09.910 INFO:tasks.workunit.client.1.smithi189.stdout:8/488: creat da/df/d22/faf x:0 0 0 2024-04-20T21:32:09.912 INFO:tasks.workunit.client.1.smithi189.stdout:5/363: truncate d7/d9/d31/f57 1131449 0 2024-04-20T21:32:09.913 INFO:tasks.workunit.client.0.smithi062.stdout:4/800: dwrite da/dd/f4d [0,4194304] 0 2024-04-20T21:32:09.915 INFO:tasks.workunit.client.1.smithi189.stdout:0/504: rename d3/df/f9b to d3/df/d30/d45/d90/faa 0 2024-04-20T21:32:09.917 INFO:tasks.workunit.client.1.smithi189.stdout:3/357: link d2/c6e d2/d1c/c84 0 2024-04-20T21:32:09.918 INFO:tasks.workunit.client.1.smithi189.stdout:2/381: dwrite d1/d11/d2d/f41 [0,4194304] 0 2024-04-20T21:32:09.919 INFO:tasks.workunit.client.1.smithi189.stdout:0/505: dread - d3/df/d30/d45/d60/f71 zero size 2024-04-20T21:32:09.920 INFO:tasks.workunit.client.1.smithi189.stdout:5/364: unlink d7/d14/f67 0 2024-04-20T21:32:09.921 INFO:tasks.workunit.client.1.smithi189.stdout:4/385: getdents d2/d64 0 2024-04-20T21:32:09.922 INFO:tasks.workunit.client.1.smithi189.stdout:8/489: symlink da/lb0 0 2024-04-20T21:32:09.924 INFO:tasks.workunit.client.1.smithi189.stdout:2/382: rmdir d1/d17 39 2024-04-20T21:32:09.925 INFO:tasks.workunit.client.1.smithi189.stdout:4/386: chown d2/dd/d11/f58 460047 1 2024-04-20T21:32:09.926 INFO:tasks.workunit.client.1.smithi189.stdout:5/365: creat d7/d35/d48/f6a x:0 0 0 2024-04-20T21:32:09.928 INFO:tasks.workunit.client.1.smithi189.stdout:0/506: creat d3/d8c/fab x:0 0 0 2024-04-20T21:32:09.929 INFO:tasks.workunit.client.1.smithi189.stdout:2/383: mknod d1/d11/d58/d6b/c71 0 2024-04-20T21:32:09.931 INFO:tasks.workunit.client.0.smithi062.stdout:4/801: link da/dd/df/f41 da/dd/df/d27/d3f/f11a 0 2024-04-20T21:32:09.934 INFO:tasks.workunit.client.1.smithi189.stdout:5/366: dread d7/d9/d30/d5a/f64 [0,4194304] 0 2024-04-20T21:32:09.936 INFO:tasks.workunit.client.0.smithi062.stdout:4/802: mknod da/dd/df/d27/d32/ded/c11b 0 2024-04-20T21:32:09.940 INFO:tasks.workunit.client.0.smithi062.stdout:4/803: mknod da/dd/df/d27/d32/c11c 0 2024-04-20T21:32:09.940 INFO:tasks.workunit.client.1.smithi189.stdout:5/367: unlink d7/f2b 0 2024-04-20T21:32:09.942 INFO:tasks.workunit.client.1.smithi189.stdout:2/384: creat d1/d17/d2f/f72 x:0 0 0 2024-04-20T21:32:09.943 INFO:tasks.workunit.client.1.smithi189.stdout:0/507: rename d3/df/d30/d45/d60/f71 to d3/df/d5d/d64/fac 0 2024-04-20T21:32:09.944 INFO:tasks.workunit.client.1.smithi189.stdout:5/368: write d7/d14/d4e/f39 [3141903,19848] 0 2024-04-20T21:32:09.945 INFO:tasks.workunit.client.0.smithi062.stdout:4/804: creat da/dd/d15/d50/d64/d57/d5b/d9d/dea/dac/f11d x:0 0 0 2024-04-20T21:32:09.946 INFO:tasks.workunit.client.1.smithi189.stdout:2/385: fsync d1/d11/d2d/f63 0 2024-04-20T21:32:09.950 INFO:tasks.workunit.client.1.smithi189.stdout:3/358: dwrite d2/d3/d2c/d31/d74/d7a/f7c [0,4194304] 0 2024-04-20T21:32:09.950 INFO:tasks.workunit.client.1.smithi189.stdout:7/419: truncate d5/d21/f32 6498868 0 2024-04-20T21:32:09.954 INFO:tasks.workunit.client.1.smithi189.stdout:0/508: mknod d3/df/d28/cad 0 2024-04-20T21:32:09.954 INFO:tasks.workunit.client.1.smithi189.stdout:2/386: fdatasync d1/d11/d2d/f63 0 2024-04-20T21:32:09.959 INFO:tasks.workunit.client.1.smithi189.stdout:5/369: dread d7/d9/d31/d36/f54 [0,4194304] 0 2024-04-20T21:32:09.966 INFO:tasks.workunit.client.0.smithi062.stdout:4/805: rename da/dd/df/d27/d32/ded to da/dd/d15/d50/d64/d57/d11e 0 2024-04-20T21:32:09.967 INFO:tasks.workunit.client.1.smithi189.stdout:2/387: mkdir d1/d11/d2d/d73 0 2024-04-20T21:32:09.970 INFO:tasks.workunit.client.0.smithi062.stdout:4/806: dread - da/dd/df/d27/d3f/fdf zero size 2024-04-20T21:32:09.970 INFO:tasks.workunit.client.1.smithi189.stdout:4/387: dwrite d2/dd/d11/f58 [0,4194304] 0 2024-04-20T21:32:09.970 INFO:tasks.workunit.client.1.smithi189.stdout:5/370: truncate d7/d14/d18/f66 974467 0 2024-04-20T21:32:09.973 INFO:tasks.workunit.client.1.smithi189.stdout:3/359: link d2/d3/d2c/c43 d2/d3/d36/c85 0 2024-04-20T21:32:09.974 INFO:tasks.workunit.client.0.smithi062.stdout:4/807: symlink da/dd/df/d27/d32/l11f 0 2024-04-20T21:32:09.977 INFO:tasks.workunit.client.1.smithi189.stdout:5/371: rename d7/d14/d4e/f39 to d7/d9/f6b 0 2024-04-20T21:32:09.978 INFO:tasks.workunit.client.1.smithi189.stdout:6/421: dread d8/da/dd/d2a/f66 [0,4194304] 0 2024-04-20T21:32:09.980 INFO:tasks.workunit.client.0.smithi062.stdout:4/808: symlink da/dd/d15/d50/d64/d57/d5b/da3/l120 0 2024-04-20T21:32:09.981 INFO:tasks.workunit.client.1.smithi189.stdout:2/388: getdents d1/d11/d58 0 2024-04-20T21:32:09.982 INFO:tasks.workunit.client.1.smithi189.stdout:5/372: dread d7/f5d [0,4194304] 0 2024-04-20T21:32:09.983 INFO:tasks.workunit.client.1.smithi189.stdout:6/422: chown d8/da/dd/d2a/d3b/f68 471306 1 2024-04-20T21:32:09.984 INFO:tasks.workunit.client.1.smithi189.stdout:0/509: dwrite d3/df/d28/d2b/d2c/d73/f94 [0,4194304] 0 2024-04-20T21:32:09.984 INFO:tasks.workunit.client.1.smithi189.stdout:3/360: rename d2/d3/d2c/c4f to d2/d1c/d5c/c86 0 2024-04-20T21:32:09.986 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:09 smithi189 ceph-mon[21562]: pgmap v19: 65 pgs: 65 active+clean; 2.3 GiB data, 6.7 GiB used, 530 GiB / 536 GiB avail; 86 MiB/s rd, 285 MiB/s wr, 547 op/s 2024-04-20T21:32:09.991 INFO:tasks.workunit.client.1.smithi189.stdout:8/490: dread da/df/f11 [0,4194304] 0 2024-04-20T21:32:09.993 INFO:tasks.workunit.client.1.smithi189.stdout:6/423: truncate d8/d2b/f57 212409 0 2024-04-20T21:32:09.993 INFO:tasks.workunit.client.1.smithi189.stdout:8/491: fsync da/df/d13/d63/fa4 0 2024-04-20T21:32:09.997 INFO:tasks.workunit.client.1.smithi189.stdout:0/510: read d3/f48 [1232057,51396] 0 2024-04-20T21:32:09.998 INFO:tasks.workunit.client.0.smithi062.stdout:8/698: sync 2024-04-20T21:32:09.999 INFO:tasks.workunit.client.1.smithi189.stdout:6/424: creat d8/da/dd/d35/d6d/f91 x:0 0 0 2024-04-20T21:32:09.999 INFO:tasks.workunit.client.0.smithi062.stdout:2/719: sync 2024-04-20T21:32:10.001 INFO:tasks.workunit.client.0.smithi062.stdout:2/720: chown d7/da/d28/l77 227984319 1 2024-04-20T21:32:10.001 INFO:tasks.workunit.client.0.smithi062.stdout:8/699: symlink d7/d25/dc5/le7 0 2024-04-20T21:32:10.004 INFO:tasks.workunit.client.1.smithi189.stdout:6/425: creat d8/da/dd/f92 x:0 0 0 2024-04-20T21:32:10.005 INFO:tasks.workunit.client.1.smithi189.stdout:5/373: dwrite d7/d35/d48/f60 [0,4194304] 0 2024-04-20T21:32:10.005 INFO:tasks.workunit.client.1.smithi189.stdout:8/492: getdents da/d81/d91 0 2024-04-20T21:32:10.005 INFO:tasks.workunit.client.1.smithi189.stdout:0/511: dread d3/df/d30/d45/f6c [0,4194304] 0 2024-04-20T21:32:10.005 INFO:tasks.workunit.client.0.smithi062.stdout:8/700: fsync d7/d25/d5c/f72 0 2024-04-20T21:32:10.007 INFO:tasks.workunit.client.1.smithi189.stdout:3/361: dwrite d2/f2a [8388608,4194304] 0 2024-04-20T21:32:10.008 INFO:tasks.workunit.client.1.smithi189.stdout:6/426: write d8/d12/d30/d47/f4c [1144605,27] 0 2024-04-20T21:32:10.008 INFO:tasks.workunit.client.1.smithi189.stdout:5/374: read d7/f5d [25035,124098] 0 2024-04-20T21:32:10.010 INFO:tasks.workunit.client.1.smithi189.stdout:4/388: dread d2/dd/d11/f67 [0,4194304] 0 2024-04-20T21:32:10.012 INFO:tasks.workunit.client.1.smithi189.stdout:5/375: write d7/d14/d18/f53 [1960287,27691] 0 2024-04-20T21:32:10.014 INFO:tasks.workunit.client.1.smithi189.stdout:8/493: symlink da/df/d20/d2c/d9a/lb1 0 2024-04-20T21:32:10.015 INFO:tasks.workunit.client.1.smithi189.stdout:4/389: write d2/dd/d11/f74 [357681,35480] 0 2024-04-20T21:32:10.018 INFO:tasks.workunit.client.0.smithi062.stdout:8/701: mkdir d7/d25/d3b/d58/de8 0 2024-04-20T21:32:10.018 INFO:tasks.workunit.client.1.smithi189.stdout:6/427: creat d8/da/dd/d35/d6d/d77/f93 x:0 0 0 2024-04-20T21:32:10.023 INFO:tasks.workunit.client.1.smithi189.stdout:0/512: getdents d3/df/d28 0 2024-04-20T21:32:10.026 INFO:tasks.workunit.client.1.smithi189.stdout:0/513: chown d3/l7f 840059705 1 2024-04-20T21:32:10.026 INFO:tasks.workunit.client.1.smithi189.stdout:6/428: symlink d8/d2b/d49/d54/d67/l94 0 2024-04-20T21:32:10.027 INFO:tasks.workunit.client.1.smithi189.stdout:5/376: link d7/c11 d7/d9/d30/d55/c6c 0 2024-04-20T21:32:10.027 INFO:tasks.workunit.client.1.smithi189.stdout:0/514: chown d3/df 13610 1 2024-04-20T21:32:10.030 INFO:tasks.workunit.client.0.smithi062.stdout:2/721: dwrite d7/da/d15/d2f/d97/f9e [0,4194304] 0 2024-04-20T21:32:10.031 INFO:tasks.workunit.client.1.smithi189.stdout:5/377: write d7/d14/d18/f53 [2575620,106426] 0 2024-04-20T21:32:10.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:09 smithi062 ceph-mon[16221]: pgmap v19: 65 pgs: 65 active+clean; 2.3 GiB data, 6.7 GiB used, 530 GiB / 536 GiB avail; 86 MiB/s rd, 285 MiB/s wr, 547 op/s 2024-04-20T21:32:10.038 INFO:tasks.workunit.client.1.smithi189.stdout:6/429: creat d8/d12/f95 x:0 0 0 2024-04-20T21:32:10.039 INFO:tasks.workunit.client.1.smithi189.stdout:5/378: write d7/d9/d31/f57 [1211393,31083] 0 2024-04-20T21:32:10.047 INFO:tasks.workunit.client.1.smithi189.stdout:5/379: creat d7/d9/d51/f6d x:0 0 0 2024-04-20T21:32:10.052 INFO:tasks.workunit.client.0.smithi062.stdout:2/722: creat d7/da/d15/d2f/d38/d6a/d7c/fed x:0 0 0 2024-04-20T21:32:10.053 INFO:tasks.workunit.client.1.smithi189.stdout:6/430: getdents d8/d2b/d49 0 2024-04-20T21:32:10.054 INFO:tasks.workunit.client.1.smithi189.stdout:0/515: dwrite d3/d8c/f8d [0,4194304] 0 2024-04-20T21:32:10.059 INFO:tasks.workunit.client.1.smithi189.stdout:2/389: truncate d1/d17/d2f/d55/f5f 95818 0 2024-04-20T21:32:10.061 INFO:tasks.workunit.client.0.smithi062.stdout:2/723: mkdir d7/da/d15/d1b/d24/dee 0 2024-04-20T21:32:10.063 INFO:tasks.workunit.client.1.smithi189.stdout:0/516: write d3/df/f91 [5205187,106612] 0 2024-04-20T21:32:10.064 INFO:tasks.workunit.client.0.smithi062.stdout:4/809: dwrite da/fb [0,4194304] 0 2024-04-20T21:32:10.068 INFO:tasks.workunit.client.1.smithi189.stdout:3/362: dread d2/d3/d25/f40 [0,4194304] 0 2024-04-20T21:32:10.068 INFO:tasks.workunit.client.1.smithi189.stdout:0/517: rmdir d3/df/d30/d45 39 2024-04-20T21:32:10.082 INFO:tasks.workunit.client.1.smithi189.stdout:0/518: write d3/df/d30/d37/f31 [4989147,46754] 0 2024-04-20T21:32:10.086 INFO:tasks.workunit.client.1.smithi189.stdout:3/363: getdents d2/d3 0 2024-04-20T21:32:10.088 INFO:tasks.workunit.client.1.smithi189.stdout:1/366: sync 2024-04-20T21:32:10.089 INFO:tasks.workunit.client.1.smithi189.stdout:3/364: creat d2/d1c/d5c/f87 x:0 0 0 2024-04-20T21:32:10.090 INFO:tasks.workunit.client.0.smithi062.stdout:8/702: dwrite d7/d25/d3b/d58/d8b/d24/d46/fc2 [0,4194304] 0 2024-04-20T21:32:10.092 INFO:tasks.workunit.client.1.smithi189.stdout:3/365: truncate d2/d3/d25/f54 948962 0 2024-04-20T21:32:10.093 INFO:tasks.workunit.client.1.smithi189.stdout:3/366: fdatasync d2/d3/d2c/d31/f6f 0 2024-04-20T21:32:10.094 INFO:tasks.workunit.client.1.smithi189.stdout:3/367: chown d2/d1c/f76 6295 1 2024-04-20T21:32:10.095 INFO:tasks.workunit.client.1.smithi189.stdout:2/390: truncate d1/d17/d2f/d55/f5f 958488 0 2024-04-20T21:32:10.097 INFO:tasks.workunit.client.0.smithi062.stdout:8/703: dread - d7/d25/d3b/d58/fc4 zero size 2024-04-20T21:32:10.098 INFO:tasks.workunit.client.1.smithi189.stdout:1/367: symlink d8/d21/d2c/l7f 0 2024-04-20T21:32:10.098 INFO:tasks.workunit.client.1.smithi189.stdout:6/431: write d8/d10/d78/d7e/f84 [3258034,63717] 0 2024-04-20T21:32:10.098 INFO:tasks.workunit.client.1.smithi189.stdout:0/519: link d3/d26/l92 d3/lae 0 2024-04-20T21:32:10.098 INFO:tasks.workunit.client.1.smithi189.stdout:1/368: dread - d8/d1e/d43/d58/d60/f64 zero size 2024-04-20T21:32:10.098 INFO:tasks.workunit.client.1.smithi189.stdout:0/520: fsync d3/df/f76 0 2024-04-20T21:32:10.104 INFO:tasks.workunit.client.0.smithi062.stdout:8/704: creat d7/da9/db0/fe9 x:0 0 0 2024-04-20T21:32:10.108 INFO:tasks.workunit.client.0.smithi062.stdout:8/705: readlink d7/l6c 0 2024-04-20T21:32:10.109 INFO:tasks.workunit.client.1.smithi189.stdout:2/391: dread d1/d11/d52/f6a [0,4194304] 0 2024-04-20T21:32:10.110 INFO:tasks.workunit.client.1.smithi189.stdout:0/521: mknod d3/df/d30/d45/d60/caf 0 2024-04-20T21:32:10.110 INFO:tasks.workunit.client.0.smithi062.stdout:4/810: dwrite da/dd/d15/d50/d64/d57/d5b/d9d/dea/dac/fcd [0,4194304] 0 2024-04-20T21:32:10.120 INFO:tasks.workunit.client.0.smithi062.stdout:8/706: getdents d7/d25/d3b/d58/dac 0 2024-04-20T21:32:10.124 INFO:tasks.workunit.client.0.smithi062.stdout:8/707: read - d7/d25/d4f/fb8 zero size 2024-04-20T21:32:10.126 INFO:tasks.workunit.client.0.smithi062.stdout:8/708: readlink d7/d25/d3b/d58/d8b/l35 0 2024-04-20T21:32:10.129 INFO:tasks.workunit.client.0.smithi062.stdout:8/709: mknod d7/de1/de5/cea 0 2024-04-20T21:32:10.134 INFO:tasks.workunit.client.0.smithi062.stdout:8/710: write d7/d25/d3b/d58/f63 [1858035,77197] 0 2024-04-20T21:32:10.136 INFO:tasks.workunit.client.0.smithi062.stdout:8/711: chown d7/d30/f95 17874 1 2024-04-20T21:32:10.139 INFO:tasks.workunit.client.0.smithi062.stdout:8/712: mknod d7/d25/d4f/ceb 0 2024-04-20T21:32:10.139 INFO:tasks.workunit.client.0.smithi062.stdout:4/811: dread da/dd/fae [0,4194304] 0 2024-04-20T21:32:10.141 INFO:tasks.workunit.client.0.smithi062.stdout:4/812: chown da/d56/d71/d9e/lf6 257318118 1 2024-04-20T21:32:10.143 INFO:tasks.workunit.client.0.smithi062.stdout:8/713: creat d7/d25/d3b/d58/d8b/fec x:0 0 0 2024-04-20T21:32:10.145 INFO:tasks.workunit.client.0.smithi062.stdout:4/813: getdents da/dd/df/d27 0 2024-04-20T21:32:10.148 INFO:tasks.workunit.client.0.smithi062.stdout:8/714: truncate d7/d9/faa 969678 0 2024-04-20T21:32:10.152 INFO:tasks.workunit.client.0.smithi062.stdout:8/715: rename d7/d25/d3b/d58/d8b/d24/cd1 to d7/d30/ced 0 2024-04-20T21:32:10.152 INFO:tasks.workunit.client.0.smithi062.stdout:4/814: link da/dd/d15/d50/d64/d3d/f40 da/dd/df/d27/d32/d97/f121 0 2024-04-20T21:32:10.154 INFO:tasks.workunit.client.0.smithi062.stdout:8/716: symlink d7/d25/d4f/lee 0 2024-04-20T21:32:10.156 INFO:tasks.workunit.client.0.smithi062.stdout:0/712: sync 2024-04-20T21:32:10.158 INFO:tasks.workunit.client.0.smithi062.stdout:0/713: write d7/d1c/f23 [1832095,108379] 0 2024-04-20T21:32:10.160 INFO:tasks.workunit.client.0.smithi062.stdout:0/714: fdatasync d7/d1c/d15/d80/fb2 0 2024-04-20T21:32:10.165 INFO:tasks.workunit.client.0.smithi062.stdout:4/815: dread da/dd/d15/d50/d64/d57/d5b/d9d/dea/dac/fcd [0,4194304] 0 2024-04-20T21:32:10.166 INFO:tasks.workunit.client.0.smithi062.stdout:8/717: write d7/d25/d3b/d58/fc4 [892574,83419] 0 2024-04-20T21:32:10.171 INFO:tasks.workunit.client.0.smithi062.stdout:8/718: link d7/d9/d29/f4c d7/d25/fef 0 2024-04-20T21:32:10.172 INFO:tasks.workunit.client.0.smithi062.stdout:4/816: write da/dd/d15/d50/d64/d3d/fb5 [632638,10043] 0 2024-04-20T21:32:10.174 INFO:tasks.workunit.client.0.smithi062.stdout:8/719: creat d7/d25/dc5/ff0 x:0 0 0 2024-04-20T21:32:10.176 INFO:tasks.workunit.client.0.smithi062.stdout:8/720: chown d7/d25/d3b/d58/d8b/c61 13 1 2024-04-20T21:32:10.177 INFO:tasks.workunit.client.0.smithi062.stdout:8/721: readlink d7/d25/d4f/l52 0 2024-04-20T21:32:10.179 INFO:tasks.workunit.client.0.smithi062.stdout:8/722: write d7/d25/d70/fd6 [68825,114624] 0 2024-04-20T21:32:10.181 INFO:tasks.workunit.client.1.smithi189.stdout:3/368: write d2/d1c/f2b [2982344,47670] 0 2024-04-20T21:32:10.181 INFO:tasks.workunit.client.1.smithi189.stdout:6/432: truncate d8/d12/f17 6987046 0 2024-04-20T21:32:10.183 INFO:tasks.workunit.client.1.smithi189.stdout:2/392: write d1/d11/d58/f6e [422618,82145] 0 2024-04-20T21:32:10.189 INFO:tasks.workunit.client.1.smithi189.stdout:0/522: unlink d3/df/d30/d37/f4b 0 2024-04-20T21:32:10.189 INFO:tasks.workunit.client.1.smithi189.stdout:6/433: write d8/d10/f36 [3149375,32292] 0 2024-04-20T21:32:10.189 INFO:tasks.workunit.client.1.smithi189.stdout:1/369: dwrite d8/d21/d2c/d36/f52 [4194304,4194304] 0 2024-04-20T21:32:10.189 INFO:tasks.workunit.client.1.smithi189.stdout:6/434: dread - d8/f63 zero size 2024-04-20T21:32:10.190 INFO:tasks.workunit.client.0.smithi062.stdout:4/817: dwrite da/dd/f14 [0,4194304] 0 2024-04-20T21:32:10.191 INFO:tasks.workunit.client.1.smithi189.stdout:1/370: read d8/d1e/f66 [1020271,119910] 0 2024-04-20T21:32:10.192 INFO:tasks.workunit.client.1.smithi189.stdout:6/435: readlink d8/da/dd/l33 0 2024-04-20T21:32:10.194 INFO:tasks.workunit.client.1.smithi189.stdout:6/436: truncate d8/d2b/f39 904400 0 2024-04-20T21:32:10.194 INFO:tasks.workunit.client.1.smithi189.stdout:0/523: unlink d3/df/d28/d2b/d2c/d73/fa3 0 2024-04-20T21:32:10.197 INFO:tasks.workunit.client.1.smithi189.stdout:0/524: fsync d3/df/d5d/f7b 0 2024-04-20T21:32:10.197 INFO:tasks.workunit.client.0.smithi062.stdout:8/723: dwrite d7/d9/f12 [0,4194304] 0 2024-04-20T21:32:10.201 INFO:tasks.workunit.client.1.smithi189.stdout:6/437: creat d8/d10/d78/f96 x:0 0 0 2024-04-20T21:32:10.202 INFO:tasks.workunit.client.1.smithi189.stdout:3/369: dwrite d2/d1c/f2b [0,4194304] 0 2024-04-20T21:32:10.205 INFO:tasks.workunit.client.1.smithi189.stdout:3/370: mkdir d2/d3/d88 0 2024-04-20T21:32:10.206 INFO:tasks.workunit.client.1.smithi189.stdout:0/525: truncate d3/df/d30/d45/d60/f68 1567953 0 2024-04-20T21:32:10.207 INFO:tasks.workunit.client.0.smithi062.stdout:8/724: fsync d7/d25/d3b/d58/d8b/d24/f50 0 2024-04-20T21:32:10.208 INFO:tasks.workunit.client.0.smithi062.stdout:8/725: readlink d7/d9/lca 0 2024-04-20T21:32:10.211 INFO:tasks.workunit.client.1.smithi189.stdout:6/438: getdents d8/d12 0 2024-04-20T21:32:10.213 INFO:tasks.workunit.client.1.smithi189.stdout:0/526: truncate d3/df/d5d/d64/fac 882965 0 2024-04-20T21:32:10.214 INFO:tasks.workunit.client.0.smithi062.stdout:8/726: creat d7/d25/d3b/d58/d8b/ff1 x:0 0 0 2024-04-20T21:32:10.217 INFO:tasks.workunit.client.1.smithi189.stdout:6/439: rename d8/da/dd/d2a/f89 to d8/da/dd/d35/f97 0 2024-04-20T21:32:10.223 INFO:tasks.workunit.client.1.smithi189.stdout:6/440: creat d8/d12/d30/d47/f98 x:0 0 0 2024-04-20T21:32:10.225 INFO:tasks.workunit.client.1.smithi189.stdout:9/411: sync 2024-04-20T21:32:10.231 INFO:tasks.workunit.client.1.smithi189.stdout:6/441: write d8/da/dd/f88 [616779,60346] 0 2024-04-20T21:32:10.231 INFO:tasks.workunit.client.1.smithi189.stdout:9/412: readlink d9/dc/d17/d2b/d52/l66 0 2024-04-20T21:32:10.231 INFO:tasks.workunit.client.1.smithi189.stdout:3/371: fdatasync d2/d3/d25/f67 0 2024-04-20T21:32:10.234 INFO:tasks.workunit.client.1.smithi189.stdout:9/413: chown d9/d12/f29 689777 1 2024-04-20T21:32:10.238 INFO:tasks.workunit.client.1.smithi189.stdout:6/442: mkdir d8/d12/d99 0 2024-04-20T21:32:10.238 INFO:tasks.workunit.client.1.smithi189.stdout:0/527: dwrite d3/df/d28/d2b/d2c/f9e [0,4194304] 0 2024-04-20T21:32:10.241 INFO:tasks.workunit.client.1.smithi189.stdout:0/528: write d3/df/d4c/f7e [5210424,120086] 0 2024-04-20T21:32:10.243 INFO:tasks.workunit.client.1.smithi189.stdout:3/372: dread d2/d3/f3e [0,4194304] 0 2024-04-20T21:32:10.245 INFO:tasks.workunit.client.1.smithi189.stdout:0/529: symlink d3/df/lb0 0 2024-04-20T21:32:10.247 INFO:tasks.workunit.client.1.smithi189.stdout:3/373: write d2/f59 [3993505,98818] 0 2024-04-20T21:32:10.260 INFO:tasks.workunit.client.1.smithi189.stdout:0/530: unlink d3/df/d30/d45/f98 0 2024-04-20T21:32:10.260 INFO:tasks.workunit.client.1.smithi189.stdout:3/374: creat d2/d3/d2c/f89 x:0 0 0 2024-04-20T21:32:10.261 INFO:tasks.workunit.client.0.smithi062.stdout:8/727: dwrite d7/d9/f17 [0,4194304] 0 2024-04-20T21:32:10.261 INFO:tasks.workunit.client.0.smithi062.stdout:4/818: dwrite da/dd/df/d27/d3f/f11a [0,4194304] 0 2024-04-20T21:32:10.264 INFO:tasks.workunit.client.0.smithi062.stdout:8/728: fdatasync d7/d25/fef 0 2024-04-20T21:32:10.266 INFO:tasks.workunit.client.1.smithi189.stdout:0/531: mkdir d3/df/d28/d2b/db1 0 2024-04-20T21:32:10.270 INFO:tasks.workunit.client.1.smithi189.stdout:9/414: dread d9/d12/d15/d1a/f64 [0,4194304] 0 2024-04-20T21:32:10.270 INFO:tasks.workunit.client.1.smithi189.stdout:0/532: symlink d3/df/d28/lb2 0 2024-04-20T21:32:10.274 INFO:tasks.workunit.client.0.smithi062.stdout:8/729: unlink d7/d25/d3b/d58/d8b/fec 0 2024-04-20T21:32:10.275 INFO:tasks.workunit.client.1.smithi189.stdout:9/415: chown d9/dc/d17/c60 2105349 1 2024-04-20T21:32:10.275 INFO:tasks.workunit.client.1.smithi189.stdout:6/443: dwrite d8/d12/d30/d47/f4c [0,4194304] 0 2024-04-20T21:32:10.280 INFO:tasks.workunit.client.1.smithi189.stdout:6/444: read f2 [3308402,13248] 0 2024-04-20T21:32:10.281 INFO:tasks.workunit.client.1.smithi189.stdout:0/533: rename d3/df/d30/d45/d60/caf to d3/df/d30/d45/d90/cb3 0 2024-04-20T21:32:10.281 INFO:tasks.workunit.client.1.smithi189.stdout:3/375: write d2/d1c/f76 [108448,124696] 0 2024-04-20T21:32:10.282 INFO:tasks.workunit.client.0.smithi062.stdout:8/730: unlink d7/d25/d5c/f69 0 2024-04-20T21:32:10.283 INFO:tasks.workunit.client.1.smithi189.stdout:9/416: write f4 [1405476,98722] 0 2024-04-20T21:32:10.288 INFO:tasks.workunit.client.1.smithi189.stdout:6/445: write d8/d2b/d49/d54/d6e/f86 [1290816,5370] 0 2024-04-20T21:32:10.288 INFO:tasks.workunit.client.1.smithi189.stdout:0/534: dread - d3/df/d5d/d64/f6d zero size 2024-04-20T21:32:10.289 INFO:tasks.workunit.client.1.smithi189.stdout:3/376: rmdir d2/d3/d2c/d31 39 2024-04-20T21:32:10.290 INFO:tasks.workunit.client.1.smithi189.stdout:3/377: chown d2/d1c 56791584 1 2024-04-20T21:32:10.292 INFO:tasks.workunit.client.1.smithi189.stdout:3/378: chown d2/c4e 11229 1 2024-04-20T21:32:10.292 INFO:tasks.workunit.client.1.smithi189.stdout:0/535: dread - d3/df/d28/d2b/f86 zero size 2024-04-20T21:32:10.293 INFO:tasks.workunit.client.1.smithi189.stdout:6/446: symlink d8/d2b/d5c/l9a 0 2024-04-20T21:32:10.293 INFO:tasks.workunit.client.0.smithi062.stdout:8/731: rename d7/d30/d90/fc6 to d7/d25/d3b/d58/d8b/d24/ff2 0 2024-04-20T21:32:10.296 INFO:tasks.workunit.client.1.smithi189.stdout:9/417: rename d9/dc/d17/d2b/d52/l59 to d9/dc/d17/l88 0 2024-04-20T21:32:10.298 INFO:tasks.workunit.client.1.smithi189.stdout:3/379: creat d2/d3/d2c/d30/d3f/f8a x:0 0 0 2024-04-20T21:32:10.299 INFO:tasks.workunit.client.1.smithi189.stdout:0/536: symlink d3/df/d30/d45/d60/lb4 0 2024-04-20T21:32:10.304 INFO:tasks.workunit.client.0.smithi062.stdout:8/732: link d7/d30/fba d7/d25/d3b/d58/d8b/d24/ff3 0 2024-04-20T21:32:10.307 INFO:tasks.workunit.client.1.smithi189.stdout:9/418: creat d9/d12/f89 x:0 0 0 2024-04-20T21:32:10.307 INFO:tasks.workunit.client.0.smithi062.stdout:9/812: sync 2024-04-20T21:32:10.308 INFO:tasks.workunit.client.0.smithi062.stdout:1/780: sync 2024-04-20T21:32:10.308 INFO:tasks.workunit.client.0.smithi062.stdout:3/679: sync 2024-04-20T21:32:10.308 INFO:tasks.workunit.client.0.smithi062.stdout:6/621: sync 2024-04-20T21:32:10.308 INFO:tasks.workunit.client.0.smithi062.stdout:5/703: sync 2024-04-20T21:32:10.308 INFO:tasks.workunit.client.0.smithi062.stdout:7/713: sync 2024-04-20T21:32:10.310 INFO:tasks.workunit.client.1.smithi189.stdout:3/380: unlink d2/d3/d2c/l41 0 2024-04-20T21:32:10.310 INFO:tasks.workunit.client.0.smithi062.stdout:5/704: chown d8/d1e/c48 205394949 1 2024-04-20T21:32:10.311 INFO:tasks.workunit.client.1.smithi189.stdout:0/537: chown d3/d26/f1d 2838 1 2024-04-20T21:32:10.313 INFO:tasks.workunit.client.0.smithi062.stdout:3/680: truncate d0/d1/fb1 53404 0 2024-04-20T21:32:10.313 INFO:tasks.workunit.client.0.smithi062.stdout:8/733: write d7/d25/d4f/f7e [491293,114339] 0 2024-04-20T21:32:10.314 INFO:tasks.workunit.client.1.smithi189.stdout:6/447: link d8/d12/d30/d47/f4c d8/f9b 0 2024-04-20T21:32:10.315 INFO:tasks.workunit.client.1.smithi189.stdout:9/419: rmdir d9/d12/d3f/d72 39 2024-04-20T21:32:10.316 INFO:tasks.workunit.client.0.smithi062.stdout:9/813: write d16/d3e/d77/d84/f99 [124730,92593] 0 2024-04-20T21:32:10.316 INFO:tasks.workunit.client.1.smithi189.stdout:3/381: rmdir d2/d3 39 2024-04-20T21:32:10.318 INFO:tasks.workunit.client.1.smithi189.stdout:3/382: read d2/f5d [104018,64871] 0 2024-04-20T21:32:10.319 INFO:tasks.workunit.client.0.smithi062.stdout:6/622: link d0/d9/f9c d0/d2c/d67/daa/fcf 0 2024-04-20T21:32:10.320 INFO:tasks.workunit.client.1.smithi189.stdout:9/420: chown d9/l35 16707 1 2024-04-20T21:32:10.320 INFO:tasks.workunit.client.0.smithi062.stdout:8/734: read d7/d9/d29/f2b [2484555,50379] 0 2024-04-20T21:32:10.321 INFO:tasks.workunit.client.0.smithi062.stdout:1/781: write dc/d15/d75/d96/fb8 [654363,80400] 0 2024-04-20T21:32:10.321 INFO:tasks.workunit.client.0.smithi062.stdout:9/814: write d16/d2f/d5b/dab/fb2 [356036,23906] 0 2024-04-20T21:32:10.324 INFO:tasks.workunit.client.1.smithi189.stdout:6/448: link d8/d2b/d49/f60 d8/d2b/d49/d54/d67/f9c 0 2024-04-20T21:32:10.325 INFO:tasks.workunit.client.0.smithi062.stdout:3/681: getdents d0/d1/d3c/d43/d72/dd3/d97 0 2024-04-20T21:32:10.328 INFO:tasks.workunit.client.0.smithi062.stdout:6/623: stat d0/d9/d11/l64 0 2024-04-20T21:32:10.330 INFO:tasks.workunit.client.1.smithi189.stdout:9/421: chown d9/d12/d3f/d72/f85 156 1 2024-04-20T21:32:10.331 INFO:tasks.workunit.client.1.smithi189.stdout:9/422: chown d9/d12/l48 1906 1 2024-04-20T21:32:10.333 INFO:tasks.workunit.client.1.smithi189.stdout:9/423: chown d9/dc/c2a 920 1 2024-04-20T21:32:10.333 INFO:tasks.workunit.client.1.smithi189.stdout:6/449: dread d8/da/dd/d35/f31 [0,4194304] 0 2024-04-20T21:32:10.335 INFO:tasks.workunit.client.0.smithi062.stdout:1/782: fdatasync dc/d15/d75/d96/dd4/d3a/da2/fb4 0 2024-04-20T21:32:10.336 INFO:tasks.workunit.client.0.smithi062.stdout:9/815: creat d16/d2c/d42/da6/f10e x:0 0 0 2024-04-20T21:32:10.337 INFO:tasks.workunit.client.1.smithi189.stdout:9/424: fsync d9/d12/f29 0 2024-04-20T21:32:10.338 INFO:tasks.workunit.client.0.smithi062.stdout:1/783: readlink dc/d27/d9a/da0/db3/lcc 0 2024-04-20T21:32:10.340 INFO:tasks.workunit.client.0.smithi062.stdout:3/682: mknod d0/d1/d10/d99/ce3 0 2024-04-20T21:32:10.340 INFO:tasks.workunit.client.0.smithi062.stdout:6/624: dread d0/d2c/f89 [0,4194304] 0 2024-04-20T21:32:10.346 INFO:tasks.workunit.client.0.smithi062.stdout:1/784: mknod dc/d15/d20/d28/c11c 0 2024-04-20T21:32:10.347 INFO:tasks.workunit.client.1.smithi189.stdout:6/450: creat d8/d12/f9d x:0 0 0 2024-04-20T21:32:10.347 INFO:tasks.workunit.client.1.smithi189.stdout:3/383: write d2/d3/d2c/d31/f50 [1178783,99567] 0 2024-04-20T21:32:10.348 INFO:tasks.workunit.client.0.smithi062.stdout:6/625: mknod d0/d2c/d8c/cd0 0 2024-04-20T21:32:10.348 INFO:tasks.workunit.client.1.smithi189.stdout:9/425: truncate d9/f87 2177984 0 2024-04-20T21:32:10.350 INFO:tasks.workunit.client.0.smithi062.stdout:7/714: dread d7/d63/f8d [0,4194304] 0 2024-04-20T21:32:10.350 INFO:tasks.workunit.client.1.smithi189.stdout:9/426: write d9/dc/d17/d36/f78 [649135,117569] 0 2024-04-20T21:32:10.351 INFO:tasks.workunit.client.0.smithi062.stdout:9/816: getdents d16/d2c/d62/dd1/df0 0 2024-04-20T21:32:10.351 INFO:tasks.workunit.client.1.smithi189.stdout:6/451: symlink d8/da/dd/d35/l9e 0 2024-04-20T21:32:10.353 INFO:tasks.workunit.client.0.smithi062.stdout:6/626: creat d0/d9a/fd1 x:0 0 0 2024-04-20T21:32:10.354 INFO:tasks.workunit.client.0.smithi062.stdout:1/785: symlink dc/d15/d75/d96/dd4/d78/l11d 0 2024-04-20T21:32:10.355 INFO:tasks.workunit.client.1.smithi189.stdout:7/420: sync 2024-04-20T21:32:10.355 INFO:tasks.workunit.client.0.smithi062.stdout:8/735: dwrite d7/d25/d4f/fb8 [0,4194304] 0 2024-04-20T21:32:10.356 INFO:tasks.workunit.client.0.smithi062.stdout:7/715: write d7/f18 [8731120,125384] 0 2024-04-20T21:32:10.359 INFO:tasks.workunit.client.0.smithi062.stdout:9/817: write d16/d3e/d77/d84/f99 [621900,124748] 0 2024-04-20T21:32:10.361 INFO:tasks.workunit.client.0.smithi062.stdout:8/736: stat d7/d30/db3/dbf/fe3 0 2024-04-20T21:32:10.361 INFO:tasks.workunit.client.1.smithi189.stdout:7/421: creat d5/f91 x:0 0 0 2024-04-20T21:32:10.362 INFO:tasks.workunit.client.0.smithi062.stdout:6/627: mkdir d0/d2c/d67/d75/dd2 0 2024-04-20T21:32:10.362 INFO:tasks.workunit.client.0.smithi062.stdout:3/683: write d0/d1c/f51 [1291406,34782] 0 2024-04-20T21:32:10.365 INFO:tasks.workunit.client.0.smithi062.stdout:1/786: unlink dc/d27/lae 0 2024-04-20T21:32:10.367 INFO:tasks.workunit.client.0.smithi062.stdout:9/818: mkdir d16/d2c/d42/d10f 0 2024-04-20T21:32:10.367 INFO:tasks.workunit.client.0.smithi062.stdout:8/737: read d7/d25/d4f/fdf [167708,73953] 0 2024-04-20T21:32:10.369 INFO:tasks.workunit.client.1.smithi189.stdout:7/422: getdents d5/d21/d4a/d52/d75 0 2024-04-20T21:32:10.370 INFO:tasks.workunit.client.0.smithi062.stdout:8/738: chown d7/d25/f9d 1 1 2024-04-20T21:32:10.370 INFO:tasks.workunit.client.0.smithi062.stdout:3/684: mkdir d0/d1/de4 0 2024-04-20T21:32:10.371 INFO:tasks.workunit.client.0.smithi062.stdout:6/628: mknod d0/d4f/cd3 0 2024-04-20T21:32:10.372 INFO:tasks.workunit.client.1.smithi189.stdout:3/384: truncate d2/d1c/f7b 2772624 0 2024-04-20T21:32:10.372 INFO:tasks.workunit.client.0.smithi062.stdout:9/819: mknod d16/d3e/d77/c110 0 2024-04-20T21:32:10.372 INFO:tasks.workunit.client.0.smithi062.stdout:6/629: fsync d0/dc7/f94 0 2024-04-20T21:32:10.373 INFO:tasks.workunit.client.1.smithi189.stdout:9/427: truncate d9/f33 2486574 0 2024-04-20T21:32:10.373 INFO:tasks.workunit.client.0.smithi062.stdout:1/787: symlink dc/d15/d75/d96/dd4/d78/dab/l11e 0 2024-04-20T21:32:10.373 INFO:tasks.workunit.client.1.smithi189.stdout:6/452: dwrite d8/da/f32 [0,4194304] 0 2024-04-20T21:32:10.374 INFO:tasks.workunit.client.1.smithi189.stdout:7/423: write d5/d21/d4a/f8d [602307,107300] 0 2024-04-20T21:32:10.375 INFO:tasks.workunit.client.0.smithi062.stdout:3/685: creat d0/d1c/d63/d42/fe5 x:0 0 0 2024-04-20T21:32:10.375 INFO:tasks.workunit.client.0.smithi062.stdout:5/705: dread d8/d38/d7a/d4c/d7d/d9d/dac/d2f/d8e/f97 [0,4194304] 0 2024-04-20T21:32:10.377 INFO:tasks.workunit.client.0.smithi062.stdout:8/739: write d7/d25/d3b/d58/d8b/d24/f74 [835282,61246] 0 2024-04-20T21:32:10.377 INFO:tasks.workunit.client.0.smithi062.stdout:3/686: chown d0/d1/f20 1 1 2024-04-20T21:32:10.379 INFO:tasks.workunit.client.0.smithi062.stdout:9/820: mkdir d16/d2c/d5e/d75/d76/dd9/d111 0 2024-04-20T21:32:10.379 INFO:tasks.workunit.client.0.smithi062.stdout:5/706: chown d8/d10/d3e/d8a/fb9 671 1 2024-04-20T21:32:10.383 INFO:tasks.workunit.client.0.smithi062.stdout:9/821: dread - d16/d4e/f10c zero size 2024-04-20T21:32:10.383 INFO:tasks.workunit.client.0.smithi062.stdout:6/630: rename d0/d2c/c77 to d0/d2c/d67/daa/cd4 0 2024-04-20T21:32:10.384 INFO:tasks.workunit.client.0.smithi062.stdout:9/822: stat d16/d2c/d5e/d96/fb3 0 2024-04-20T21:32:10.385 INFO:tasks.workunit.client.1.smithi189.stdout:6/453: mknod d8/d2b/d49/d54/d67/c9f 0 2024-04-20T21:32:10.385 INFO:tasks.workunit.client.0.smithi062.stdout:5/707: truncate d8/d38/f89 3005245 0 2024-04-20T21:32:10.386 INFO:tasks.workunit.client.1.smithi189.stdout:3/385: creat d2/d3/d65/f8b x:0 0 0 2024-04-20T21:32:10.386 INFO:tasks.workunit.client.0.smithi062.stdout:8/740: mknod d7/d25/d3b/d58/d8b/d24/cf4 0 2024-04-20T21:32:10.388 INFO:tasks.workunit.client.0.smithi062.stdout:5/708: readlink d8/d10/d3e/d8a/l8f 0 2024-04-20T21:32:10.393 INFO:tasks.workunit.client.0.smithi062.stdout:9/823: symlink d16/d2c/d10d/l112 0 2024-04-20T21:32:10.395 INFO:tasks.workunit.client.0.smithi062.stdout:9/824: readlink d16/d2f/d5b/da4/le1 0 2024-04-20T21:32:10.397 INFO:tasks.workunit.client.1.smithi189.stdout:6/454: mkdir d8/d12/d30/d47/da0 0 2024-04-20T21:32:10.398 INFO:tasks.workunit.client.1.smithi189.stdout:7/424: dread d5/d21/d25/d34/f6f [0,4194304] 0 2024-04-20T21:32:10.398 INFO:tasks.workunit.client.0.smithi062.stdout:5/709: symlink d8/d38/d7a/d4c/d7d/d9d/dac/d66/de4/leb 0 2024-04-20T21:32:10.400 INFO:tasks.workunit.client.0.smithi062.stdout:9/825: truncate d16/d2c/d42/da6/da9/ff4 665910 0 2024-04-20T21:32:10.402 INFO:tasks.workunit.client.0.smithi062.stdout:9/826: write d16/d2f/fc8 [2160506,30339] 0 2024-04-20T21:32:10.402 INFO:tasks.workunit.client.0.smithi062.stdout:8/741: rename d7/d25/d3b/d58/d8b/d96/fdd to d7/d25/d3b/d58/d8b/d24/d46/ff5 0 2024-04-20T21:32:10.404 INFO:tasks.workunit.client.1.smithi189.stdout:6/455: mknod d8/d12/d4e/ca1 0 2024-04-20T21:32:10.404 INFO:tasks.workunit.client.1.smithi189.stdout:7/425: creat d5/d21/f92 x:0 0 0 2024-04-20T21:32:10.405 INFO:tasks.workunit.client.0.smithi062.stdout:8/742: write d7/d25/d70/fd6 [382484,105449] 0 2024-04-20T21:32:10.409 INFO:tasks.workunit.client.0.smithi062.stdout:9/827: symlink d16/d4e/d80/l113 0 2024-04-20T21:32:10.411 INFO:tasks.workunit.client.1.smithi189.stdout:9/428: dwrite d9/d12/d3f/f68 [0,4194304] 0 2024-04-20T21:32:10.415 INFO:tasks.workunit.client.1.smithi189.stdout:9/429: write d9/dc/f4d [1067136,109606] 0 2024-04-20T21:32:10.416 INFO:tasks.workunit.client.0.smithi062.stdout:9/828: fsync d16/f66 0 2024-04-20T21:32:10.416 INFO:tasks.workunit.client.0.smithi062.stdout:1/788: write dc/d15/d75/d96/dd4/d3a/da2/de4/d7c/ffa [5070167,121975] 0 2024-04-20T21:32:10.419 INFO:tasks.workunit.client.1.smithi189.stdout:9/430: chown d9/dc/d17/l18 305991098 1 2024-04-20T21:32:10.419 INFO:tasks.workunit.client.1.smithi189.stdout:7/426: mknod d5/d1d/d77/c93 0 2024-04-20T21:32:10.420 INFO:tasks.workunit.client.0.smithi062.stdout:9/829: write d16/d2f/d5b/fa0 [200062,39254] 0 2024-04-20T21:32:10.422 INFO:tasks.workunit.client.0.smithi062.stdout:5/710: link d8/d38/d7a/db1/ce7 d8/d1e/d2c/cec 0 2024-04-20T21:32:10.423 INFO:tasks.workunit.client.0.smithi062.stdout:6/631: dread d0/dc7/f94 [0,4194304] 0 2024-04-20T21:32:10.428 INFO:tasks.workunit.client.1.smithi189.stdout:6/456: link d8/da/l42 d8/da/dd/d35/la2 0 2024-04-20T21:32:10.428 INFO:tasks.workunit.client.0.smithi062.stdout:9/830: chown d16/d2f/d5b/da4/c107 144 1 2024-04-20T21:32:10.431 INFO:tasks.workunit.client.0.smithi062.stdout:3/687: dwrite d0/d1/f19 [0,4194304] 0 2024-04-20T21:32:10.432 INFO:tasks.workunit.client.0.smithi062.stdout:5/711: unlink d8/d1e/d2c/f69 0 2024-04-20T21:32:10.433 INFO:tasks.workunit.client.0.smithi062.stdout:5/712: fdatasync d8/d38/fc5 0 2024-04-20T21:32:10.434 INFO:tasks.workunit.client.1.smithi189.stdout:3/386: dread d2/d3/d2c/d31/f50 [4194304,4194304] 0 2024-04-20T21:32:10.438 INFO:tasks.workunit.client.0.smithi062.stdout:5/713: mkdir d8/d1e/d2c/dd1/ded 0 2024-04-20T21:32:10.440 INFO:tasks.workunit.client.0.smithi062.stdout:3/688: write d0/d1/d3c/d43/fd4 [694567,92491] 0 2024-04-20T21:32:10.443 INFO:tasks.workunit.client.1.smithi189.stdout:6/457: dread d8/d2b/d49/d54/f83 [0,4194304] 0 2024-04-20T21:32:10.444 INFO:tasks.workunit.client.0.smithi062.stdout:3/689: stat d0/d1/d3c/d43/d72/da9/fab 0 2024-04-20T21:32:10.446 INFO:tasks.workunit.client.1.smithi189.stdout:6/458: write d8/d12/d30/d47/f8a [891360,34748] 0 2024-04-20T21:32:10.447 INFO:tasks.workunit.client.0.smithi062.stdout:5/714: creat d8/d38/d7a/d4c/dcf/fee x:0 0 0 2024-04-20T21:32:10.449 INFO:tasks.workunit.client.1.smithi189.stdout:6/459: rmdir d8/d2b/d5c 39 2024-04-20T21:32:10.449 INFO:tasks.workunit.client.0.smithi062.stdout:8/743: dwrite d7/d9/d29/f32 [4194304,4194304] 0 2024-04-20T21:32:10.450 INFO:tasks.workunit.client.0.smithi062.stdout:3/690: chown d0/d1/d10/d3a/d5e/f8d 9632989 1 2024-04-20T21:32:10.452 INFO:tasks.workunit.client.1.smithi189.stdout:6/460: fsync d8/da/dd/d35/f62 0 2024-04-20T21:32:10.452 INFO:tasks.workunit.client.0.smithi062.stdout:8/744: write d7/d25/d3b/d58/f63 [553861,107871] 0 2024-04-20T21:32:10.454 INFO:tasks.workunit.client.1.smithi189.stdout:3/387: dwrite d2/d3/f52 [4194304,4194304] 0 2024-04-20T21:32:10.455 INFO:tasks.workunit.client.0.smithi062.stdout:3/691: chown d0/d1/d10/d3a/l6a 120 1 2024-04-20T21:32:10.456 INFO:tasks.workunit.client.1.smithi189.stdout:3/388: chown d2/d3/d25/f40 55 1 2024-04-20T21:32:10.457 INFO:tasks.workunit.client.0.smithi062.stdout:5/715: creat d8/d38/d7a/d4c/d7d/d9d/dac/fef x:0 0 0 2024-04-20T21:32:10.460 INFO:tasks.workunit.client.1.smithi189.stdout:3/389: mknod d2/d3/d88/c8c 0 2024-04-20T21:32:10.464 INFO:tasks.workunit.client.1.smithi189.stdout:3/390: mkdir d2/d8d 0 2024-04-20T21:32:10.468 INFO:tasks.workunit.client.1.smithi189.stdout:3/391: symlink d2/d3/d2c/d30/d3f/l8e 0 2024-04-20T21:32:10.470 INFO:tasks.workunit.client.1.smithi189.stdout:3/392: readlink l1 0 2024-04-20T21:32:10.472 INFO:tasks.workunit.client.1.smithi189.stdout:3/393: mknod d2/d3/d25/c8f 0 2024-04-20T21:32:10.475 INFO:tasks.workunit.client.1.smithi189.stdout:3/394: mkdir d2/d3/d2c/d31/d90 0 2024-04-20T21:32:10.484 INFO:tasks.workunit.client.1.smithi189.stdout:7/427: write d5/d21/d4a/f4d [638516,2674] 0 2024-04-20T21:32:10.488 INFO:tasks.workunit.client.1.smithi189.stdout:7/428: symlink d5/d46/d7d/l94 0 2024-04-20T21:32:10.489 INFO:tasks.workunit.client.1.smithi189.stdout:6/461: write d8/d12/f17 [1986364,8384] 0 2024-04-20T21:32:10.491 INFO:tasks.workunit.client.0.smithi062.stdout:6/632: truncate d0/d9/f16 551331 0 2024-04-20T21:32:10.492 INFO:tasks.workunit.client.1.smithi189.stdout:7/429: symlink d5/d46/l95 0 2024-04-20T21:32:10.493 INFO:tasks.workunit.client.0.smithi062.stdout:5/716: truncate d8/d1e/d2c/d6c/f65 4355417 0 2024-04-20T21:32:10.495 INFO:tasks.workunit.client.0.smithi062.stdout:8/745: dread d7/d9/f3a [0,4194304] 0 2024-04-20T21:32:10.497 INFO:tasks.workunit.client.0.smithi062.stdout:1/789: dwrite dc/d15/d75/d96/dd4/f5d [0,4194304] 0 2024-04-20T21:32:10.498 INFO:tasks.workunit.client.0.smithi062.stdout:2/724: sync 2024-04-20T21:32:10.498 INFO:tasks.workunit.client.0.smithi062.stdout:0/715: sync 2024-04-20T21:32:10.500 INFO:tasks.workunit.client.0.smithi062.stdout:8/746: write d7/fbb [3796314,126644] 0 2024-04-20T21:32:10.503 INFO:tasks.workunit.client.0.smithi062.stdout:2/725: chown d7/da/d15/d2f/d38/d51/dbc/fcd 3 1 2024-04-20T21:32:10.504 INFO:tasks.workunit.client.0.smithi062.stdout:5/717: write d8/d38/d7a/f91 [1034038,64671] 0 2024-04-20T21:32:10.505 INFO:tasks.workunit.client.1.smithi189.stdout:3/395: dread d2/d3/f48 [0,4194304] 0 2024-04-20T21:32:10.508 INFO:tasks.workunit.client.0.smithi062.stdout:1/790: mknod dc/d15/d75/d94/d6f/dd2/c11f 0 2024-04-20T21:32:10.510 INFO:tasks.workunit.client.0.smithi062.stdout:8/747: truncate d7/d25/f9d 33130 0 2024-04-20T21:32:10.511 INFO:tasks.workunit.client.1.smithi189.stdout:6/462: truncate d8/d12/f44 2500476 0 2024-04-20T21:32:10.511 INFO:tasks.workunit.client.0.smithi062.stdout:0/716: getdents d7/d1e/da9/dbe 0 2024-04-20T21:32:10.512 INFO:tasks.workunit.client.0.smithi062.stdout:2/726: dread - d7/da/d15/f1d zero size 2024-04-20T21:32:10.512 INFO:tasks.workunit.client.0.smithi062.stdout:5/718: chown d8/d38/d7a/d4c/d7d/d9d/dac/f59 284509 1 2024-04-20T21:32:10.513 INFO:tasks.workunit.client.1.smithi189.stdout:3/396: creat d2/d8d/f91 x:0 0 0 2024-04-20T21:32:10.514 INFO:tasks.workunit.client.1.smithi189.stdout:6/463: creat d8/da/fa3 x:0 0 0 2024-04-20T21:32:10.519 INFO:tasks.workunit.client.0.smithi062.stdout:6/633: dwrite d0/d9/d11/fbc [0,4194304] 0 2024-04-20T21:32:10.519 INFO:tasks.workunit.client.1.smithi189.stdout:7/430: dwrite d5/f43 [0,4194304] 0 2024-04-20T21:32:10.520 INFO:tasks.workunit.client.0.smithi062.stdout:0/717: symlink d7/d1c/d15/d17/d40/d3c/da4/lec 0 2024-04-20T21:32:10.521 INFO:tasks.workunit.client.1.smithi189.stdout:7/431: chown d5/f43 9355671 1 2024-04-20T21:32:10.522 INFO:tasks.workunit.client.0.smithi062.stdout:2/727: creat d7/da/d15/d2f/d38/d51/dbc/fef x:0 0 0 2024-04-20T21:32:10.525 INFO:tasks.workunit.client.1.smithi189.stdout:7/432: mknod d5/d21/d35/c96 0 2024-04-20T21:32:10.527 INFO:tasks.workunit.client.0.smithi062.stdout:2/728: chown d7/l8 1550465 1 2024-04-20T21:32:10.528 INFO:tasks.workunit.client.0.smithi062.stdout:8/748: rmdir d7 39 2024-04-20T21:32:10.530 INFO:tasks.workunit.client.0.smithi062.stdout:1/791: link dc/d15/d75/d94/f69 dc/d15/d75/d96/dd4/d3a/da2/f120 0 2024-04-20T21:32:10.532 INFO:tasks.workunit.client.0.smithi062.stdout:2/729: chown d7/da/d15/d1b/d24/lb7 712159 1 2024-04-20T21:32:10.533 INFO:tasks.workunit.client.1.smithi189.stdout:7/433: truncate d5/d21/d4a/d52/f5c 280608 0 2024-04-20T21:32:10.536 INFO:tasks.workunit.client.1.smithi189.stdout:7/434: fdatasync d5/d1d/d64/f7b 0 2024-04-20T21:32:10.536 INFO:tasks.workunit.client.1.smithi189.stdout:3/397: dwrite d2/f2a [0,4194304] 0 2024-04-20T21:32:10.536 INFO:tasks.workunit.client.1.smithi189.stdout:6/464: dwrite d8/d2b/d49/d54/d6e/f86 [0,4194304] 0 2024-04-20T21:32:10.538 INFO:tasks.workunit.client.0.smithi062.stdout:2/730: write d7/da/d15/d2f/d38/d6b/fd1 [956958,618] 0 2024-04-20T21:32:10.538 INFO:tasks.workunit.client.0.smithi062.stdout:6/634: rename d0/d2c to d0/d9/d11/d22/dd5 0 2024-04-20T21:32:10.540 INFO:tasks.workunit.client.0.smithi062.stdout:6/635: chown d0/d9/d11/f48 768411 1 2024-04-20T21:32:10.542 INFO:tasks.workunit.client.1.smithi189.stdout:7/435: mkdir d5/d21/d97 0 2024-04-20T21:32:10.543 INFO:tasks.workunit.client.0.smithi062.stdout:1/792: mkdir dc/d15/d75/d94/d6f/dd2/de3/d121 0 2024-04-20T21:32:10.544 INFO:tasks.workunit.client.1.smithi189.stdout:7/436: chown d5/d59/l83 365 1 2024-04-20T21:32:10.557 INFO:tasks.workunit.client.1.smithi189.stdout:6/465: dread d8/f9b [0,4194304] 0 2024-04-20T21:32:10.559 INFO:tasks.workunit.client.0.smithi062.stdout:5/719: dwrite d8/f44 [0,4194304] 0 2024-04-20T21:32:10.560 INFO:tasks.workunit.client.1.smithi189.stdout:6/466: symlink d8/da/dd/d35/la4 0 2024-04-20T21:32:10.562 INFO:tasks.workunit.client.1.smithi189.stdout:7/437: dwrite d5/f49 [0,4194304] 0 2024-04-20T21:32:10.562 INFO:tasks.workunit.client.1.smithi189.stdout:6/467: rename d8/da/dd to d8/da/dd/d2a/da5 22 2024-04-20T21:32:10.566 INFO:tasks.workunit.client.1.smithi189.stdout:8/494: sync 2024-04-20T21:32:10.570 INFO:tasks.workunit.client.1.smithi189.stdout:7/438: rename d5/d1d/d77/d5b/d86 to d5/d21/d4a/d52/d98 0 2024-04-20T21:32:10.574 INFO:tasks.workunit.client.1.smithi189.stdout:8/495: rename da/c15 to da/df/d3c/d68/cb2 0 2024-04-20T21:32:10.578 INFO:tasks.workunit.client.0.smithi062.stdout:6/636: rename d0/d9/d11/d22/dd5/d67/daa/fcf to d0/d9/d11/d22/dd5/d67/d75/fd6 0 2024-04-20T21:32:10.579 INFO:tasks.workunit.client.0.smithi062.stdout:5/720: unlink d8/d1e/d2c/l2e 0 2024-04-20T21:32:10.579 INFO:tasks.workunit.client.1.smithi189.stdout:8/496: mkdir da/df/d13/d63/db3 0 2024-04-20T21:32:10.579 INFO:tasks.workunit.client.0.smithi062.stdout:8/749: link c3 d7/d25/dc5/dd3/cf6 0 2024-04-20T21:32:10.580 INFO:tasks.workunit.client.0.smithi062.stdout:6/637: write d0/d52/da0/fcc [159472,80908] 0 2024-04-20T21:32:10.580 INFO:tasks.workunit.client.0.smithi062.stdout:5/721: readlink d8/d10/d3e/le3 0 2024-04-20T21:32:10.582 INFO:tasks.workunit.client.0.smithi062.stdout:6/638: fsync d0/f91 0 2024-04-20T21:32:10.582 INFO:tasks.workunit.client.1.smithi189.stdout:6/468: dwrite d8/f63 [0,4194304] 0 2024-04-20T21:32:10.585 INFO:tasks.workunit.client.0.smithi062.stdout:6/639: write d0/d9/d11/f8d [93636,44759] 0 2024-04-20T21:32:10.585 INFO:tasks.workunit.client.0.smithi062.stdout:1/793: rmdir dc/d15/d75/d10a 0 2024-04-20T21:32:10.586 INFO:tasks.workunit.client.1.smithi189.stdout:8/497: getdents da/df/d3c/d50/d6c/d85 0 2024-04-20T21:32:10.588 INFO:tasks.workunit.client.0.smithi062.stdout:8/750: unlink d7/de1/de5/cea 0 2024-04-20T21:32:10.591 INFO:tasks.workunit.client.1.smithi189.stdout:8/498: read da/de/d1e/d21/d44/f43 [45007,101374] 0 2024-04-20T21:32:10.596 INFO:tasks.workunit.client.0.smithi062.stdout:5/722: mkdir d8/d38/d7a/df0 0 2024-04-20T21:32:10.596 INFO:tasks.workunit.client.1.smithi189.stdout:7/439: write d5/d21/f32 [4831144,25644] 0 2024-04-20T21:32:10.599 INFO:tasks.workunit.client.0.smithi062.stdout:6/640: truncate d0/d9/d11/f63 206044 0 2024-04-20T21:32:10.603 INFO:tasks.workunit.client.0.smithi062.stdout:6/641: symlink d0/d1a/d44/ld7 0 2024-04-20T21:32:10.605 INFO:tasks.workunit.client.0.smithi062.stdout:6/642: read d0/d1a/f85 [130787,131030] 0 2024-04-20T21:32:10.606 INFO:tasks.workunit.client.0.smithi062.stdout:5/723: truncate d8/d1e/d2c/d6c/f65 643456 0 2024-04-20T21:32:10.609 INFO:tasks.workunit.client.0.smithi062.stdout:5/724: read d8/d38/d7a/d4c/d7d/d9d/dac/f92 [49683,2181] 0 2024-04-20T21:32:10.611 INFO:tasks.workunit.client.1.smithi189.stdout:7/440: truncate d5/d21/d25/f30 3722650 0 2024-04-20T21:32:10.612 INFO:tasks.workunit.client.0.smithi062.stdout:5/725: write d8/d10/f4e [3842692,13985] 0 2024-04-20T21:32:10.614 INFO:tasks.workunit.client.1.smithi189.stdout:8/499: dwrite da/df/d13/d63/f73 [0,4194304] 0 2024-04-20T21:32:10.615 INFO:tasks.workunit.client.0.smithi062.stdout:5/726: creat d8/d38/ff1 x:0 0 0 2024-04-20T21:32:10.617 INFO:tasks.workunit.client.1.smithi189.stdout:7/441: creat d5/d46/d7d/d45/f99 x:0 0 0 2024-04-20T21:32:10.619 INFO:tasks.workunit.client.1.smithi189.stdout:8/500: symlink da/df/d3c/d50/d6c/lb4 0 2024-04-20T21:32:10.619 INFO:tasks.workunit.client.0.smithi062.stdout:5/727: creat d8/d38/d7a/d4c/dcf/ff2 x:0 0 0 2024-04-20T21:32:10.623 INFO:tasks.workunit.client.0.smithi062.stdout:2/731: dwrite d7/da/d15/d2f/f33 [0,4194304] 0 2024-04-20T21:32:10.628 INFO:tasks.workunit.client.0.smithi062.stdout:5/728: mkdir d8/d38/d7a/d4c/d7d/d9d/dac/d2f/df3 0 2024-04-20T21:32:10.630 INFO:tasks.workunit.client.0.smithi062.stdout:1/794: dwrite dc/d70/fd5 [0,4194304] 0 2024-04-20T21:32:10.632 INFO:tasks.workunit.client.0.smithi062.stdout:4/819: sync 2024-04-20T21:32:10.633 INFO:tasks.workunit.client.0.smithi062.stdout:8/751: dread d7/d9/f41 [0,4194304] 0 2024-04-20T21:32:10.636 INFO:tasks.workunit.client.1.smithi189.stdout:7/442: dwrite d5/d1d/d64/f7b [0,4194304] 0 2024-04-20T21:32:10.637 INFO:tasks.workunit.client.0.smithi062.stdout:5/729: readlink d8/d38/d7a/l31 0 2024-04-20T21:32:10.639 INFO:tasks.workunit.client.0.smithi062.stdout:8/752: mkdir d7/d25/d70/df7 0 2024-04-20T21:32:10.639 INFO:tasks.workunit.client.1.smithi189.stdout:7/443: chown d5/d1d/d64/f7b 36437566 1 2024-04-20T21:32:10.648 INFO:tasks.workunit.client.0.smithi062.stdout:8/753: creat d7/ff8 x:0 0 0 2024-04-20T21:32:10.652 INFO:tasks.workunit.client.0.smithi062.stdout:5/730: link d8/d1e/d2c/d6c/l45 d8/d1e/d2c/lf4 0 2024-04-20T21:32:10.654 INFO:tasks.workunit.client.0.smithi062.stdout:2/732: write d7/da/d15/d2f/d38/db0/dc4/fc7 [519196,78641] 0 2024-04-20T21:32:10.657 INFO:tasks.workunit.client.0.smithi062.stdout:8/754: rename d7/d25/ld5 to d7/d25/d3b/d58/dac/lf9 0 2024-04-20T21:32:10.659 INFO:tasks.workunit.client.1.smithi189.stdout:7/444: symlink d5/d21/d4a/d52/d74/l9a 0 2024-04-20T21:32:10.660 INFO:tasks.workunit.client.0.smithi062.stdout:4/820: dwrite da/dd/d15/d50/d64/d3d/fb5 [0,4194304] 0 2024-04-20T21:32:10.664 INFO:tasks.workunit.client.0.smithi062.stdout:2/733: chown d7/da/f13 3 1 2024-04-20T21:32:10.667 INFO:tasks.workunit.client.1.smithi189.stdout:7/445: creat d5/d21/d4a/d52/d98/f9b x:0 0 0 2024-04-20T21:32:10.668 INFO:tasks.workunit.client.0.smithi062.stdout:4/821: mknod da/dd/d15/d50/d64/c122 0 2024-04-20T21:32:10.674 INFO:tasks.workunit.client.1.smithi189.stdout:8/501: link da/de/d1e/d21/d44/c78 da/df/d3c/cb5 0 2024-04-20T21:32:10.675 INFO:tasks.workunit.client.0.smithi062.stdout:2/734: creat d7/da/d15/d2f/d93/ff0 x:0 0 0 2024-04-20T21:32:10.675 INFO:tasks.workunit.client.1.smithi189.stdout:8/502: stat da/l77 0 2024-04-20T21:32:10.676 INFO:tasks.workunit.client.0.smithi062.stdout:4/822: mkdir da/dd/d15/d50/d123 0 2024-04-20T21:32:10.678 INFO:tasks.workunit.client.0.smithi062.stdout:1/795: dread dc/d15/d20/d28/f4e [0,4194304] 0 2024-04-20T21:32:10.681 INFO:tasks.workunit.client.0.smithi062.stdout:5/731: dwrite d8/f44 [0,4194304] 0 2024-04-20T21:32:10.683 INFO:tasks.workunit.client.0.smithi062.stdout:4/823: rename da/d56/lb4 to da/dd/d15/d50/d123/l124 0 2024-04-20T21:32:10.684 INFO:tasks.workunit.client.1.smithi189.stdout:7/446: dwrite d5/d21/d25/f66 [0,4194304] 0 2024-04-20T21:32:10.687 INFO:tasks.workunit.client.1.smithi189.stdout:8/503: rename da/df/d20/d2c/d9a/c9c to da/df/d13/d63/d7b/cb6 0 2024-04-20T21:32:10.687 INFO:tasks.workunit.client.0.smithi062.stdout:1/796: creat dc/d15/f122 x:0 0 0 2024-04-20T21:32:10.689 INFO:tasks.workunit.client.1.smithi189.stdout:7/447: mknod d5/d21/d4a/d52/c9c 0 2024-04-20T21:32:10.691 INFO:tasks.workunit.client.1.smithi189.stdout:8/504: stat da/de/d1e/d21/d87 0 2024-04-20T21:32:10.692 INFO:tasks.workunit.client.0.smithi062.stdout:1/797: creat dc/d15/d75/f123 x:0 0 0 2024-04-20T21:32:10.693 INFO:tasks.workunit.client.0.smithi062.stdout:5/732: truncate d8/dbd/fdc 3904763 0 2024-04-20T21:32:10.694 INFO:tasks.workunit.client.0.smithi062.stdout:4/824: mkdir da/dd/df/d27/d7a/d125 0 2024-04-20T21:32:10.698 INFO:tasks.workunit.client.1.smithi189.stdout:7/448: link d5/d21/d4a/d52/d98/l89 d5/d21/l9d 0 2024-04-20T21:32:10.700 INFO:tasks.workunit.client.1.smithi189.stdout:8/505: truncate da/df/d13/f5f 4210925 0 2024-04-20T21:32:10.702 INFO:tasks.workunit.client.0.smithi062.stdout:2/735: dwrite d7/da/d15/d2f/d38/d6b/f88 [0,4194304] 0 2024-04-20T21:32:10.702 INFO:tasks.workunit.client.0.smithi062.stdout:1/798: getdents dc/d15/d20/d28/d83/dc0 0 2024-04-20T21:32:10.705 INFO:tasks.workunit.client.0.smithi062.stdout:8/755: dread d7/d25/f42 [0,4194304] 0 2024-04-20T21:32:10.706 INFO:tasks.workunit.client.0.smithi062.stdout:4/825: creat da/dd/d15/d50/d64/d57/d5b/d9d/f126 x:0 0 0 2024-04-20T21:32:10.707 INFO:tasks.workunit.client.0.smithi062.stdout:1/799: chown dc/d15/d75/d94/d6f/lcf 26899529 1 2024-04-20T21:32:10.707 INFO:tasks.workunit.client.1.smithi189.stdout:7/449: unlink d5/d1d/f5f 0 2024-04-20T21:32:10.709 INFO:tasks.workunit.client.0.smithi062.stdout:2/736: mkdir d7/da/d15/d1b/df1 0 2024-04-20T21:32:10.709 INFO:tasks.workunit.client.1.smithi189.stdout:7/450: write d5/f7 [4420095,127281] 0 2024-04-20T21:32:10.710 INFO:tasks.workunit.client.0.smithi062.stdout:4/826: dread - da/dd/df/d109/f112 zero size 2024-04-20T21:32:10.711 INFO:tasks.workunit.client.1.smithi189.stdout:4/390: sync 2024-04-20T21:32:10.712 INFO:tasks.workunit.client.1.smithi189.stdout:7/451: stat d5/d21/d35/l3f 0 2024-04-20T21:32:10.716 INFO:tasks.workunit.client.0.smithi062.stdout:8/756: fdatasync d7/d25/d3b/d58/d8b/d24/d46/ff5 0 2024-04-20T21:32:10.718 INFO:tasks.workunit.client.1.smithi189.stdout:7/452: symlink d5/d46/l9e 0 2024-04-20T21:32:10.719 INFO:tasks.workunit.client.1.smithi189.stdout:8/506: dwrite da/df/d20/d27/f38 [4194304,4194304] 0 2024-04-20T21:32:10.720 INFO:tasks.workunit.client.0.smithi062.stdout:1/800: creat dc/d15/d20/d28/d83/dc0/f124 x:0 0 0 2024-04-20T21:32:10.721 INFO:tasks.workunit.client.0.smithi062.stdout:8/757: chown d7/d25/d5c/f72 31766 1 2024-04-20T21:32:10.723 INFO:tasks.workunit.client.1.smithi189.stdout:4/391: creat d2/dd/d11/d14/d69/d6a/d7a/f8d x:0 0 0 2024-04-20T21:32:10.724 INFO:tasks.workunit.client.1.smithi189.stdout:4/392: stat d2/f2c 0 2024-04-20T21:32:10.726 INFO:tasks.workunit.client.1.smithi189.stdout:8/507: creat da/de/d1e/fb7 x:0 0 0 2024-04-20T21:32:10.726 INFO:tasks.workunit.client.1.smithi189.stdout:7/453: symlink d5/d1d/d77/l9f 0 2024-04-20T21:32:10.727 INFO:tasks.workunit.client.1.smithi189.stdout:8/508: chown da 0 1 2024-04-20T21:32:10.728 INFO:tasks.workunit.client.0.smithi062.stdout:4/827: getdents da/dd/d15/d50/d64/d57 0 2024-04-20T21:32:10.729 INFO:tasks.workunit.client.0.smithi062.stdout:5/733: dwrite d8/d38/d7a/f40 [0,4194304] 0 2024-04-20T21:32:10.730 INFO:tasks.workunit.client.0.smithi062.stdout:8/758: link d7/d30/d90/cbe d7/d25/d76/cfa 0 2024-04-20T21:32:10.732 INFO:tasks.workunit.client.0.smithi062.stdout:4/828: mknod da/dd/c127 0 2024-04-20T21:32:10.734 INFO:tasks.workunit.client.0.smithi062.stdout:8/759: write d7/d9/d29/f4b [572236,12067] 0 2024-04-20T21:32:10.735 INFO:tasks.workunit.client.1.smithi189.stdout:7/454: mknod d5/d1d/d77/d5b/ca0 0 2024-04-20T21:32:10.736 INFO:tasks.workunit.client.0.smithi062.stdout:2/737: write d7/da/d15/f23 [1836085,2136] 0 2024-04-20T21:32:10.739 INFO:tasks.workunit.client.1.smithi189.stdout:7/455: chown d5/d21/c88 5 1 2024-04-20T21:32:10.740 INFO:tasks.workunit.client.1.smithi189.stdout:4/393: link d2/dd/d11/f7e d2/d7f/f8e 0 2024-04-20T21:32:10.741 INFO:tasks.workunit.client.0.smithi062.stdout:2/738: readlink d7/da/d15/d2f/d38/db0/dc4/ld4 0 2024-04-20T21:32:10.741 INFO:tasks.workunit.client.1.smithi189.stdout:7/456: stat d5/f6 0 2024-04-20T21:32:10.742 INFO:tasks.workunit.client.0.smithi062.stdout:5/734: mknod d8/d38/d7a/d4c/d7d/d9d/dac/d66/cf5 0 2024-04-20T21:32:10.742 INFO:tasks.workunit.client.1.smithi189.stdout:8/509: getdents da 0 2024-04-20T21:32:10.742 INFO:tasks.workunit.client.1.smithi189.stdout:4/394: chown d2/f5a 0 1 2024-04-20T21:32:10.745 INFO:tasks.workunit.client.1.smithi189.stdout:8/510: mknod da/df/d3c/d68/cb8 0 2024-04-20T21:32:10.746 INFO:tasks.workunit.client.1.smithi189.stdout:4/395: rmdir d2/d22/d87 39 2024-04-20T21:32:10.746 INFO:tasks.workunit.client.0.smithi062.stdout:5/735: fdatasync d8/d1e/d2c/f3a 0 2024-04-20T21:32:10.748 INFO:tasks.workunit.client.1.smithi189.stdout:8/511: readlink da/de/d1e/d21/d44/la9 0 2024-04-20T21:32:10.749 INFO:tasks.workunit.client.0.smithi062.stdout:1/801: dwrite dc/d15/d20/d28/d83/dce/f11a [0,4194304] 0 2024-04-20T21:32:10.750 INFO:tasks.workunit.client.1.smithi189.stdout:8/512: fsync da/df/d3c/d50/f66 0 2024-04-20T21:32:10.751 INFO:tasks.workunit.client.1.smithi189.stdout:8/513: chown da/d81/d91/da8 3 1 2024-04-20T21:32:10.751 INFO:tasks.workunit.client.1.smithi189.stdout:7/457: rename d5/d1d/d77/d5b/f72 to d5/d1d/d64/fa1 0 2024-04-20T21:32:10.752 INFO:tasks.workunit.client.1.smithi189.stdout:5/380: sync 2024-04-20T21:32:10.753 INFO:tasks.workunit.client.1.smithi189.stdout:4/396: creat d2/d7f/f8f x:0 0 0 2024-04-20T21:32:10.758 INFO:tasks.workunit.client.0.smithi062.stdout:5/736: rename d8/d38/d7a/d4c/d7d/d9d/dac/f59 to d8/d10/d3e/ff6 0 2024-04-20T21:32:10.759 INFO:tasks.workunit.client.1.smithi189.stdout:7/458: write d5/d1d/d77/d5b/f5e [1023790,14116] 0 2024-04-20T21:32:10.762 INFO:tasks.workunit.client.1.smithi189.stdout:5/381: mknod d7/d9/d30/d55/d61/c6e 0 2024-04-20T21:32:10.762 INFO:tasks.workunit.client.1.smithi189.stdout:7/459: write d5/d21/d25/f66 [1147661,45014] 0 2024-04-20T21:32:10.763 INFO:tasks.workunit.client.0.smithi062.stdout:1/802: truncate dc/d27/fd8 238818 0 2024-04-20T21:32:10.763 INFO:tasks.workunit.client.0.smithi062.stdout:4/829: dwrite da/dd/df/fa2 [0,4194304] 0 2024-04-20T21:32:10.765 INFO:tasks.workunit.client.1.smithi189.stdout:7/460: chown d5/d1d/d77/d5b/ca0 28 1 2024-04-20T21:32:10.768 INFO:tasks.workunit.client.1.smithi189.stdout:4/397: creat d2/d22/f90 x:0 0 0 2024-04-20T21:32:10.771 INFO:tasks.workunit.client.0.smithi062.stdout:2/739: dwrite d7/da/d15/d1b/d3d/f49 [0,4194304] 0 2024-04-20T21:32:10.774 INFO:tasks.workunit.client.1.smithi189.stdout:5/382: creat d7/d9/d28/f6f x:0 0 0 2024-04-20T21:32:10.775 INFO:tasks.workunit.client.1.smithi189.stdout:7/461: creat d5/d46/d7d/d45/fa2 x:0 0 0 2024-04-20T21:32:10.777 INFO:tasks.workunit.client.1.smithi189.stdout:4/398: chown d2/l6 280 1 2024-04-20T21:32:10.778 INFO:tasks.workunit.client.0.smithi062.stdout:2/740: creat d7/da/d15/d2f/d38/db0/ff2 x:0 0 0 2024-04-20T21:32:10.781 INFO:tasks.workunit.client.1.smithi189.stdout:7/462: mkdir d5/d21/d97/da3 0 2024-04-20T21:32:10.783 INFO:tasks.workunit.client.0.smithi062.stdout:4/830: creat da/dd/df/d27/d7a/f128 x:0 0 0 2024-04-20T21:32:10.784 INFO:tasks.workunit.client.0.smithi062.stdout:8/760: dread d7/d25/d3b/d58/d8b/d24/d46/f64 [0,4194304] 0 2024-04-20T21:32:10.784 INFO:tasks.workunit.client.1.smithi189.stdout:7/463: read d5/d21/d25/f66 [1990466,78914] 0 2024-04-20T21:32:10.784 INFO:tasks.workunit.client.0.smithi062.stdout:7/716: sync 2024-04-20T21:32:10.786 INFO:tasks.workunit.client.0.smithi062.stdout:1/803: write dc/d15/d20/f45 [4131703,125705] 0 2024-04-20T21:32:10.786 INFO:tasks.workunit.client.1.smithi189.stdout:4/399: read d2/d3a/f4d [1069085,76418] 0 2024-04-20T21:32:10.788 INFO:tasks.workunit.client.1.smithi189.stdout:8/514: dwrite da/de/f9f [0,4194304] 0 2024-04-20T21:32:10.789 INFO:tasks.workunit.client.0.smithi062.stdout:4/831: write da/dd/df/f10f [319621,26855] 0 2024-04-20T21:32:10.791 INFO:tasks.workunit.client.1.smithi189.stdout:7/464: readlink d5/d21/d4a/d52/d98/l89 0 2024-04-20T21:32:10.791 INFO:tasks.workunit.client.0.smithi062.stdout:2/741: creat d7/da/d15/d1b/ff3 x:0 0 0 2024-04-20T21:32:10.794 INFO:tasks.workunit.client.1.smithi189.stdout:4/400: rename d2/dd/f78 to d2/d3a/f91 0 2024-04-20T21:32:10.794 INFO:tasks.workunit.client.1.smithi189.stdout:1/371: sync 2024-04-20T21:32:10.794 INFO:tasks.workunit.client.1.smithi189.stdout:6/469: sync 2024-04-20T21:32:10.794 INFO:tasks.workunit.client.1.smithi189.stdout:3/398: sync 2024-04-20T21:32:10.794 INFO:tasks.workunit.client.1.smithi189.stdout:2/393: sync 2024-04-20T21:32:10.794 INFO:tasks.workunit.client.1.smithi189.stdout:0/538: sync 2024-04-20T21:32:10.794 INFO:tasks.workunit.client.1.smithi189.stdout:9/431: sync 2024-04-20T21:32:10.795 INFO:tasks.workunit.client.0.smithi062.stdout:8/761: rmdir d7/d25/dc5 39 2024-04-20T21:32:10.795 INFO:tasks.workunit.client.1.smithi189.stdout:8/515: rmdir da/df/d22/d33/d59 39 2024-04-20T21:32:10.796 INFO:tasks.workunit.client.1.smithi189.stdout:1/372: readlink d8/lc 0 2024-04-20T21:32:10.797 INFO:tasks.workunit.client.0.smithi062.stdout:1/804: mknod dc/d15/d75/d96/dd4/d3a/da2/c125 0 2024-04-20T21:32:10.798 INFO:tasks.workunit.client.0.smithi062.stdout:5/737: dwrite d8/d38/d7a/d4c/d7d/d9d/dac/d2f/d8e/f97 [0,4194304] 0 2024-04-20T21:32:10.802 INFO:tasks.workunit.client.1.smithi189.stdout:7/465: rename d5/f91 to d5/d1d/d77/d5b/d71/fa4 0 2024-04-20T21:32:10.803 INFO:tasks.workunit.client.0.smithi062.stdout:8/762: write d7/d25/d4f/fdf [128291,13422] 0 2024-04-20T21:32:10.805 INFO:tasks.workunit.client.0.smithi062.stdout:4/832: write da/dd/f35 [3093813,74067] 0 2024-04-20T21:32:10.806 INFO:tasks.workunit.client.1.smithi189.stdout:2/394: mkdir d1/d17/d2f/d74 0 2024-04-20T21:32:10.808 INFO:tasks.workunit.client.1.smithi189.stdout:5/383: write d7/d14/d18/f17 [218214,49802] 0 2024-04-20T21:32:10.809 INFO:tasks.workunit.client.1.smithi189.stdout:0/539: symlink d3/df/d28/d2b/d2c/d74/lb5 0 2024-04-20T21:32:10.811 INFO:tasks.workunit.client.0.smithi062.stdout:2/742: creat d7/da/d15/d1b/df1/ff4 x:0 0 0 2024-04-20T21:32:10.812 INFO:tasks.workunit.client.1.smithi189.stdout:4/401: creat d2/dd/d11/f92 x:0 0 0 2024-04-20T21:32:10.812 INFO:tasks.workunit.client.1.smithi189.stdout:2/395: fdatasync d1/d11/d58/d6b/f6c 0 2024-04-20T21:32:10.812 INFO:tasks.workunit.client.1.smithi189.stdout:3/399: chown d2/d3/c46 80675602 1 2024-04-20T21:32:10.813 INFO:tasks.workunit.client.0.smithi062.stdout:2/743: fdatasync d7/da/d15/d2f/d93/dac/fb8 0 2024-04-20T21:32:10.814 INFO:tasks.workunit.client.0.smithi062.stdout:4/833: truncate da/dd/d15/d50/d64/d57/d11e/f10a 495410 0 2024-04-20T21:32:10.818 INFO:tasks.workunit.client.1.smithi189.stdout:8/516: creat da/df/d22/d33/d59/fb9 x:0 0 0 2024-04-20T21:32:10.827 INFO:tasks.workunit.client.0.smithi062.stdout:5/738: truncate d8/d1e/fa9 1028422 0 2024-04-20T21:32:10.828 INFO:tasks.workunit.client.0.smithi062.stdout:8/763: write d7/d25/d76/f9c [1058708,111497] 0 2024-04-20T21:32:10.829 INFO:tasks.workunit.client.1.smithi189.stdout:5/384: dread d7/f5d [0,4194304] 0 2024-04-20T21:32:10.832 INFO:tasks.workunit.client.0.smithi062.stdout:5/739: chown d8/f4f 31706836 1 2024-04-20T21:32:10.833 INFO:tasks.workunit.client.1.smithi189.stdout:4/402: write d2/dd/d11/d14/f50 [602311,29240] 0 2024-04-20T21:32:10.837 INFO:tasks.workunit.client.0.smithi062.stdout:8/764: creat d7/d25/d3b/d58/db9/ffb x:0 0 0 2024-04-20T21:32:10.840 INFO:tasks.workunit.client.0.smithi062.stdout:5/740: chown d8/d38/d7a/d4c/dcf/ff2 56231574 1 2024-04-20T21:32:10.840 INFO:tasks.workunit.client.0.smithi062.stdout:7/717: dwrite d7/d3e/d33/d3d/d5e/d72/f98 [0,4194304] 0 2024-04-20T21:32:10.843 INFO:tasks.workunit.client.1.smithi189.stdout:9/432: dwrite d9/dc/f1c [0,4194304] 0 2024-04-20T21:32:10.844 INFO:tasks.workunit.client.0.smithi062.stdout:2/744: dwrite d7/d39/f5f [4194304,4194304] 0 2024-04-20T21:32:10.845 INFO:tasks.workunit.client.1.smithi189.stdout:6/470: dwrite d8/da/f6b [0,4194304] 0 2024-04-20T21:32:10.849 INFO:tasks.workunit.client.1.smithi189.stdout:1/373: dwrite d8/d21/d2c/f75 [0,4194304] 0 2024-04-20T21:32:10.849 INFO:tasks.workunit.client.1.smithi189.stdout:4/403: dread - d2/d7f/f8f zero size 2024-04-20T21:32:10.849 INFO:tasks.workunit.client.1.smithi189.stdout:5/385: write d7/d14/d18/f53 [2777889,12985] 0 2024-04-20T21:32:10.855 INFO:tasks.workunit.client.1.smithi189.stdout:7/466: dwrite d5/d21/d25/f27 [0,4194304] 0 2024-04-20T21:32:10.855 INFO:tasks.workunit.client.0.smithi062.stdout:4/834: dread da/dd/df/d27/d3f/f11a [4194304,4194304] 0 2024-04-20T21:32:10.856 INFO:tasks.workunit.client.0.smithi062.stdout:2/745: rmdir d7/da/d15/d2f/d38/db0/dc4 39 2024-04-20T21:32:10.859 INFO:tasks.workunit.client.1.smithi189.stdout:1/374: readlink d8/d1e/d28/l71 0 2024-04-20T21:32:10.861 INFO:tasks.workunit.client.0.smithi062.stdout:7/718: rmdir d7/d26/dac/df2 0 2024-04-20T21:32:10.861 INFO:tasks.workunit.client.1.smithi189.stdout:4/404: readlink d2/dd/d11/l8b 0 2024-04-20T21:32:10.864 INFO:tasks.workunit.client.1.smithi189.stdout:1/375: write d8/f14 [1449978,13839] 0 2024-04-20T21:32:10.866 INFO:tasks.workunit.client.1.smithi189.stdout:2/396: dwrite d1/d17/f34 [0,4194304] 0 2024-04-20T21:32:10.871 INFO:tasks.workunit.client.0.smithi062.stdout:8/765: dwrite d7/d25/d3b/d58/d8b/f1e [4194304,4194304] 0 2024-04-20T21:32:10.872 INFO:tasks.workunit.client.0.smithi062.stdout:2/746: creat d7/da/d15/d2f/d38/db0/ff5 x:0 0 0 2024-04-20T21:32:10.873 INFO:tasks.workunit.client.1.smithi189.stdout:8/517: mkdir da/df/d22/d33/dba 0 2024-04-20T21:32:10.874 INFO:tasks.workunit.client.1.smithi189.stdout:9/433: unlink d9/dc/d17/d2b/l65 0 2024-04-20T21:32:10.875 INFO:tasks.workunit.client.1.smithi189.stdout:7/467: creat d5/d59/fa5 x:0 0 0 2024-04-20T21:32:10.875 INFO:tasks.workunit.client.0.smithi062.stdout:7/719: symlink d7/d3e/d33/d3d/d4d/d54/d4f/d9d/lf6 0 2024-04-20T21:32:10.879 INFO:tasks.workunit.client.1.smithi189.stdout:2/397: creat d1/d17/d37/f75 x:0 0 0 2024-04-20T21:32:10.880 INFO:tasks.workunit.client.0.smithi062.stdout:4/835: dread da/dd/d15/d50/d64/d57/d5b/d9d/dea/dac/fcd [0,4194304] 0 2024-04-20T21:32:10.882 INFO:tasks.workunit.client.0.smithi062.stdout:5/741: dread d8/d38/fc5 [0,4194304] 0 2024-04-20T21:32:10.883 INFO:tasks.workunit.client.0.smithi062.stdout:2/747: mkdir d7/da/d15/d2f/d38/d6b/df6 0 2024-04-20T21:32:10.887 INFO:tasks.workunit.client.1.smithi189.stdout:3/400: mkdir d2/d3/d2c/d31/d90/d92 0 2024-04-20T21:32:10.889 INFO:tasks.workunit.client.1.smithi189.stdout:8/518: rmdir da/df/d20/d27 39 2024-04-20T21:32:10.889 INFO:tasks.workunit.client.1.smithi189.stdout:9/434: rmdir d9/dc/d17/d2b/d56 39 2024-04-20T21:32:10.890 INFO:tasks.workunit.client.0.smithi062.stdout:7/720: rmdir d7/d3e/d33/d3d/d4d/d54/d4f/de9 39 2024-04-20T21:32:10.891 INFO:tasks.workunit.client.1.smithi189.stdout:4/405: dread d2/dd/d11/f74 [0,4194304] 0 2024-04-20T21:32:10.891 INFO:tasks.workunit.client.1.smithi189.stdout:1/376: read d8/f1b [1585811,6302] 0 2024-04-20T21:32:10.893 INFO:tasks.workunit.client.1.smithi189.stdout:6/471: dwrite d8/da/dd/d2a/d3b/f8e [0,4194304] 0 2024-04-20T21:32:10.897 INFO:tasks.workunit.client.1.smithi189.stdout:0/540: link d3/df/d30/d37/l20 d3/lb6 0 2024-04-20T21:32:10.898 INFO:tasks.workunit.client.1.smithi189.stdout:9/435: chown d9/dc/d69/f7f 7475 1 2024-04-20T21:32:10.899 INFO:tasks.workunit.client.0.smithi062.stdout:2/748: creat d7/da/d15/d1b/d3d/ff7 x:0 0 0 2024-04-20T21:32:10.899 INFO:tasks.workunit.client.1.smithi189.stdout:4/406: chown d2/dd/d11/d14/f7d 14750 1 2024-04-20T21:32:10.899 INFO:tasks.workunit.client.1.smithi189.stdout:6/472: truncate d8/d2b/d49/f72 910628 0 2024-04-20T21:32:10.900 INFO:tasks.workunit.client.1.smithi189.stdout:7/468: rename d5/d21/d25/d34/f6d to d5/d1d/d77/d5b/d71/fa6 0 2024-04-20T21:32:10.901 INFO:tasks.workunit.client.1.smithi189.stdout:8/519: dread - da/df/d3c/d50/d6c/d85/f8a zero size 2024-04-20T21:32:10.902 INFO:tasks.workunit.client.0.smithi062.stdout:2/749: chown d7/da/d15/d1b/l5e 0 1 2024-04-20T21:32:10.902 INFO:tasks.workunit.client.1.smithi189.stdout:4/407: dread - d2/d3a/f91 zero size 2024-04-20T21:32:10.903 INFO:tasks.workunit.client.0.smithi062.stdout:7/721: mkdir d7/d3e/d33/d3d/d5e/df7 0 2024-04-20T21:32:10.903 INFO:tasks.workunit.client.1.smithi189.stdout:3/401: symlink d2/d3/d2c/d31/d74/d7a/l93 0 2024-04-20T21:32:10.904 INFO:tasks.workunit.client.1.smithi189.stdout:2/398: mknod d1/d11/c76 0 2024-04-20T21:32:10.906 INFO:tasks.workunit.client.0.smithi062.stdout:2/750: readlink d7/da/d15/d2f/d38/d51/dbc/le4 0 2024-04-20T21:32:10.907 INFO:tasks.workunit.client.1.smithi189.stdout:0/541: mknod d3/df/d28/d2b/d2c/d73/cb7 0 2024-04-20T21:32:10.909 INFO:tasks.workunit.client.1.smithi189.stdout:7/469: write d5/d21/d4a/f4d [723097,33226] 0 2024-04-20T21:32:10.909 INFO:tasks.workunit.client.0.smithi062.stdout:7/722: dread - d7/d3e/d33/d3d/d4d/d54/d4f/de9/df1/df3/ff5 zero size 2024-04-20T21:32:10.911 INFO:tasks.workunit.client.1.smithi189.stdout:4/408: dread - d2/dd/d37/f59 zero size 2024-04-20T21:32:10.913 INFO:tasks.workunit.client.0.smithi062.stdout:9/831: sync 2024-04-20T21:32:10.913 INFO:tasks.workunit.client.0.smithi062.stdout:3/692: sync 2024-04-20T21:32:10.913 INFO:tasks.workunit.client.0.smithi062.stdout:6/643: sync 2024-04-20T21:32:10.913 INFO:tasks.workunit.client.0.smithi062.stdout:0/718: sync 2024-04-20T21:32:10.915 INFO:tasks.workunit.client.1.smithi189.stdout:2/399: read d1/d17/f35 [3970626,27373] 0 2024-04-20T21:32:10.915 INFO:tasks.workunit.client.1.smithi189.stdout:8/520: creat da/df/d22/d33/d59/fbb x:0 0 0 2024-04-20T21:32:10.916 INFO:tasks.workunit.client.1.smithi189.stdout:3/402: mknod d2/d3/d2c/d30/d3f/c94 0 2024-04-20T21:32:10.917 INFO:tasks.workunit.client.1.smithi189.stdout:7/470: read d5/d1d/d64/f7b [1896586,3935] 0 2024-04-20T21:32:10.918 INFO:tasks.workunit.client.0.smithi062.stdout:9/832: fdatasync d16/d3d/d55/faf 0 2024-04-20T21:32:10.919 INFO:tasks.workunit.client.0.smithi062.stdout:3/693: read d0/d11/f1a [7323716,114430] 0 2024-04-20T21:32:10.920 INFO:tasks.workunit.client.1.smithi189.stdout:3/403: stat d2/d3/d2c/d31/l70 0 2024-04-20T21:32:10.921 INFO:tasks.workunit.client.1.smithi189.stdout:1/377: rename d8/df/d3a/f7b to d8/d1e/d28/d4f/f80 0 2024-04-20T21:32:10.921 INFO:tasks.workunit.client.1.smithi189.stdout:0/542: creat d3/df/d28/d2b/db1/fb8 x:0 0 0 2024-04-20T21:32:10.924 INFO:tasks.workunit.client.0.smithi062.stdout:6/644: dread - d0/d9/d11/d7b/fc9 zero size 2024-04-20T21:32:10.924 INFO:tasks.workunit.client.0.smithi062.stdout:0/719: rmdir d7/d1c/d15/d17/d40/d79/d71 39 2024-04-20T21:32:10.925 INFO:tasks.workunit.client.1.smithi189.stdout:2/400: write d1/d11/d58/f64 [1046272,2445] 0 2024-04-20T21:32:10.928 INFO:tasks.workunit.client.1.smithi189.stdout:3/404: chown d2/d3/f3e 10731 1 2024-04-20T21:32:10.930 INFO:tasks.workunit.client.0.smithi062.stdout:3/694: rmdir d0/d1c/d63/d42/d5b 39 2024-04-20T21:32:10.931 INFO:tasks.workunit.client.1.smithi189.stdout:7/471: rmdir d5/d1d/d64 39 2024-04-20T21:32:10.932 INFO:tasks.workunit.client.1.smithi189.stdout:2/401: chown d1/d17/d2f/f31 198022723 1 2024-04-20T21:32:10.933 INFO:tasks.workunit.client.0.smithi062.stdout:2/751: dwrite d7/da/d28/d4b/f95 [0,4194304] 0 2024-04-20T21:32:10.933 INFO:tasks.workunit.client.0.smithi062.stdout:0/720: creat d7/d1c/d15/d17/fed x:0 0 0 2024-04-20T21:32:10.934 INFO:tasks.workunit.client.1.smithi189.stdout:1/378: symlink d8/d1e/l81 0 2024-04-20T21:32:10.936 INFO:tasks.workunit.client.0.smithi062.stdout:5/742: dread d8/d1e/d2c/fa2 [0,4194304] 0 2024-04-20T21:32:10.941 INFO:tasks.workunit.client.1.smithi189.stdout:9/436: dread d9/dc/d17/d36/f43 [0,4194304] 0 2024-04-20T21:32:10.941 INFO:tasks.workunit.client.1.smithi189.stdout:8/521: creat da/df/d20/d27/fbc x:0 0 0 2024-04-20T21:32:10.945 INFO:tasks.workunit.client.0.smithi062.stdout:3/695: truncate d0/d1c/d63/f91 529155 0 2024-04-20T21:32:10.950 INFO:tasks.workunit.client.1.smithi189.stdout:2/402: creat d1/d11/d58/d6b/f77 x:0 0 0 2024-04-20T21:32:10.952 INFO:tasks.workunit.client.1.smithi189.stdout:7/472: unlink d5/d1d/d77/d5b/d71/l8e 0 2024-04-20T21:32:10.953 INFO:tasks.workunit.client.1.smithi189.stdout:0/543: mkdir d3/db9 0 2024-04-20T21:32:10.954 INFO:tasks.workunit.client.1.smithi189.stdout:4/409: dread d2/dd/d11/f7e [0,4194304] 0 2024-04-20T21:32:10.958 INFO:tasks.workunit.client.1.smithi189.stdout:1/379: creat d8/d21/d2c/d36/f82 x:0 0 0 2024-04-20T21:32:10.962 INFO:tasks.workunit.client.0.smithi062.stdout:0/721: truncate d7/d20/d62/d6f/fae 185161 0 2024-04-20T21:32:10.962 INFO:tasks.workunit.client.1.smithi189.stdout:9/437: dread d9/dc/d17/d36/f57 [0,4194304] 0 2024-04-20T21:32:10.965 INFO:tasks.workunit.client.0.smithi062.stdout:4/836: write da/f2f [5639075,14630] 0 2024-04-20T21:32:10.965 INFO:tasks.workunit.client.1.smithi189.stdout:0/544: write d3/f5e [4515515,1446] 0 2024-04-20T21:32:10.965 INFO:tasks.workunit.client.1.smithi189.stdout:9/438: chown d9/f34 27808 1 2024-04-20T21:32:10.965 INFO:tasks.workunit.client.0.smithi062.stdout:0/722: dread - d7/d1e/da9/fcb zero size 2024-04-20T21:32:10.967 INFO:tasks.workunit.client.0.smithi062.stdout:3/696: mkdir d0/d1/d10/d31/d41/de6 0 2024-04-20T21:32:10.967 INFO:tasks.workunit.client.1.smithi189.stdout:7/473: unlink d5/d21/f92 0 2024-04-20T21:32:10.969 INFO:tasks.workunit.client.0.smithi062.stdout:2/752: dwrite d7/da/d15/d2f/d38/d6b/fd1 [0,4194304] 0 2024-04-20T21:32:10.970 INFO:tasks.workunit.client.1.smithi189.stdout:7/474: write d5/d21/d4a/f8d [4693790,117427] 0 2024-04-20T21:32:10.970 INFO:tasks.workunit.client.0.smithi062.stdout:5/743: rename d8/d38/d7a/cdf to d8/d38/d7a/cf7 0 2024-04-20T21:32:10.972 INFO:tasks.workunit.client.0.smithi062.stdout:9/833: write d16/d2c/d42/f54 [2545104,109953] 0 2024-04-20T21:32:10.972 INFO:tasks.workunit.client.0.smithi062.stdout:0/723: fdatasync d7/d20/d62/d6f/f9b 0 2024-04-20T21:32:10.974 INFO:tasks.workunit.client.0.smithi062.stdout:7/723: dread d7/d3e/d33/f37 [0,4194304] 0 2024-04-20T21:32:10.974 INFO:tasks.workunit.client.0.smithi062.stdout:4/837: creat da/dd/d15/d6d/f129 x:0 0 0 2024-04-20T21:32:10.975 INFO:tasks.workunit.client.0.smithi062.stdout:0/724: dread - d7/ddf/fe8 zero size 2024-04-20T21:32:10.977 INFO:tasks.workunit.client.0.smithi062.stdout:9/834: stat d16/d3d/d55/f73 0 2024-04-20T21:32:10.977 INFO:tasks.workunit.client.0.smithi062.stdout:7/724: readlink d7/d26/da8/le6 0 2024-04-20T21:32:10.979 INFO:tasks.workunit.client.1.smithi189.stdout:0/545: rmdir d3/df/d4c 39 2024-04-20T21:32:10.981 INFO:tasks.workunit.client.1.smithi189.stdout:0/546: dread - d3/df/f76 zero size 2024-04-20T21:32:10.983 INFO:tasks.workunit.client.0.smithi062.stdout:5/744: read d8/d10/d3e/fb3 [2842108,46304] 0 2024-04-20T21:32:10.986 INFO:tasks.workunit.client.0.smithi062.stdout:0/725: write d7/d1c/d15/d53/fbb [670462,1609] 0 2024-04-20T21:32:10.991 INFO:tasks.workunit.client.0.smithi062.stdout:0/726: fdatasync f1 0 2024-04-20T21:32:10.994 INFO:tasks.workunit.client.0.smithi062.stdout:6/645: dwrite d0/d9/d11/f63 [0,4194304] 0 2024-04-20T21:32:10.999 INFO:tasks.workunit.client.0.smithi062.stdout:5/745: creat d8/d10/d3e/ff8 x:0 0 0 2024-04-20T21:32:11.000 INFO:tasks.workunit.client.0.smithi062.stdout:7/725: creat d7/d3e/d33/d3d/d4d/dce/ff8 x:0 0 0 2024-04-20T21:32:11.002 INFO:tasks.workunit.client.0.smithi062.stdout:2/753: link d7/da/f22 d7/da/d15/d2f/d93/dac/dca/ff8 0 2024-04-20T21:32:11.008 INFO:tasks.workunit.client.0.smithi062.stdout:3/697: dwrite d0/fdc [0,4194304] 0 2024-04-20T21:32:11.009 INFO:tasks.workunit.client.0.smithi062.stdout:6/646: fsync d0/d1a/d44/fb8 0 2024-04-20T21:32:11.019 INFO:tasks.workunit.client.0.smithi062.stdout:3/698: truncate d0/d1c/fce 843654 0 2024-04-20T21:32:11.023 INFO:tasks.workunit.client.0.smithi062.stdout:5/746: fdatasync d8/d38/d7a/d4c/d7d/d9d/dac/fef 0 2024-04-20T21:32:11.023 INFO:tasks.workunit.client.0.smithi062.stdout:4/838: dwrite da/d56/d71/f93 [4194304,4194304] 0 2024-04-20T21:32:11.026 INFO:tasks.workunit.client.0.smithi062.stdout:2/754: chown d7/da/d15/d1b/d3d/d84/d9b/laa 977761 1 2024-04-20T21:32:11.026 INFO:tasks.workunit.client.0.smithi062.stdout:3/699: creat d0/d1/d3c/d43/fe7 x:0 0 0 2024-04-20T21:32:11.032 INFO:tasks.workunit.client.0.smithi062.stdout:5/747: mkdir d8/d1e/d2c/d6c/df9 0 2024-04-20T21:32:11.033 INFO:tasks.workunit.client.0.smithi062.stdout:9/835: dwrite d16/d2c/d42/da6/f10e [0,4194304] 0 2024-04-20T21:32:11.035 INFO:tasks.workunit.client.0.smithi062.stdout:6/647: symlink d0/d9/d11/d22/dd5/d67/d75/dd2/ld8 0 2024-04-20T21:32:11.037 INFO:tasks.workunit.client.0.smithi062.stdout:7/726: link d7/d3e/d33/d3d/d5e/f61 d7/d3e/d33/d3d/d5e/ff9 0 2024-04-20T21:32:11.041 INFO:tasks.workunit.client.0.smithi062.stdout:3/700: fsync d0/d1c/d63/f91 0 2024-04-20T21:32:11.043 INFO:tasks.workunit.client.0.smithi062.stdout:2/755: creat d7/da/d15/d2f/d38/d6a/ff9 x:0 0 0 2024-04-20T21:32:11.043 INFO:tasks.workunit.client.0.smithi062.stdout:5/748: truncate d8/d1e/f53 1688507 0 2024-04-20T21:32:11.044 INFO:tasks.workunit.client.1.smithi189.stdout:3/405: truncate d2/d3/f19 3992912 0 2024-04-20T21:32:11.046 INFO:tasks.workunit.client.1.smithi189.stdout:3/406: stat d2/d3/d88/c8c 0 2024-04-20T21:32:11.046 INFO:tasks.workunit.client.0.smithi062.stdout:7/727: mknod d7/cfa 0 2024-04-20T21:32:11.053 INFO:tasks.workunit.client.0.smithi062.stdout:9/836: creat d16/d2c/d62/dd1/df8/f114 x:0 0 0 2024-04-20T21:32:11.057 INFO:tasks.workunit.client.1.smithi189.stdout:3/407: fdatasync d2/d3/d2c/d31/f50 0 2024-04-20T21:32:11.058 INFO:tasks.workunit.client.1.smithi189.stdout:4/410: getdents d2/d7f 0 2024-04-20T21:32:11.059 INFO:tasks.workunit.client.1.smithi189.stdout:3/408: write d2/d3/d2c/d31/f6f [139859,110294] 0 2024-04-20T21:32:11.061 INFO:tasks.workunit.client.1.smithi189.stdout:6/473: write d8/d2b/f57 [1231642,5299] 0 2024-04-20T21:32:11.064 INFO:tasks.workunit.client.0.smithi062.stdout:3/701: readlink d0/d1/d3c/d43/d72/dd3/l5d 0 2024-04-20T21:32:11.066 INFO:tasks.workunit.client.1.smithi189.stdout:4/411: mknod d2/dd/d11/d14/d69/d6a/c93 0 2024-04-20T21:32:11.066 INFO:tasks.workunit.client.0.smithi062.stdout:6/648: link d0/f80 d0/d9/d11/d22/dd5/d67/d75/fd9 0 2024-04-20T21:32:11.068 INFO:tasks.workunit.client.1.smithi189.stdout:3/409: creat d2/d3/d25/f95 x:0 0 0 2024-04-20T21:32:11.069 INFO:tasks.workunit.client.1.smithi189.stdout:6/474: chown d8/da/f27 65 1 2024-04-20T21:32:11.069 INFO:tasks.workunit.client.1.smithi189.stdout:5/386: sync 2024-04-20T21:32:11.069 INFO:tasks.workunit.client.0.smithi062.stdout:9/837: readlink d16/d4e/d80/lae 0 2024-04-20T21:32:11.070 INFO:tasks.workunit.client.1.smithi189.stdout:4/412: mkdir d2/dd/d11/d14/d69/d94 0 2024-04-20T21:32:11.075 INFO:tasks.workunit.client.1.smithi189.stdout:4/413: dread d2/d7f/f8e [0,4194304] 0 2024-04-20T21:32:11.076 INFO:tasks.workunit.client.0.smithi062.stdout:0/727: dwrite d7/d1c/d15/d17/d1b/d9d/db1/fd4 [0,4194304] 0 2024-04-20T21:32:11.078 INFO:tasks.workunit.client.1.smithi189.stdout:5/387: chown d7/d14/f49 1945168 1 2024-04-20T21:32:11.078 INFO:tasks.workunit.client.0.smithi062.stdout:2/756: link d7/da/d15/d2f/d38/d6a/d7c/f96 d7/da/d15/d2f/d38/db0/ffa 0 2024-04-20T21:32:11.080 INFO:tasks.workunit.client.0.smithi062.stdout:0/728: chown d7/d1e/c2b 4357812 1 2024-04-20T21:32:11.081 INFO:tasks.workunit.client.0.smithi062.stdout:3/702: mkdir d0/d1/d3c/d43/d72/dd3/dbf/de8 0 2024-04-20T21:32:11.081 INFO:tasks.workunit.client.0.smithi062.stdout:2/757: stat d7/da/d15/d1b/d3d/f94 0 2024-04-20T21:32:11.084 INFO:tasks.workunit.client.0.smithi062.stdout:3/703: fsync d0/d1b/d74/fbe 0 2024-04-20T21:32:11.084 INFO:tasks.workunit.client.0.smithi062.stdout:0/729: write d7/d1c/f23 [3465895,92510] 0 2024-04-20T21:32:11.085 INFO:tasks.workunit.client.0.smithi062.stdout:2/758: fdatasync d7/d39/f45 0 2024-04-20T21:32:11.089 INFO:tasks.workunit.client.0.smithi062.stdout:2/759: fdatasync d7/da/d28/d4b/f73 0 2024-04-20T21:32:11.091 INFO:tasks.workunit.client.0.smithi062.stdout:0/730: dread - d7/d1c/d15/d17/d26/fea zero size 2024-04-20T21:32:11.091 INFO:tasks.workunit.client.1.smithi189.stdout:5/388: readlink d7/d9/d30/l32 0 2024-04-20T21:32:11.092 INFO:tasks.workunit.client.0.smithi062.stdout:5/749: truncate d8/d38/d7a/d4c/d7d/d9d/dac/f39 127857 0 2024-04-20T21:32:11.094 INFO:tasks.workunit.client.0.smithi062.stdout:3/704: fsync d0/d1/d3c/d43/d72/fbd 0 2024-04-20T21:32:11.100 INFO:tasks.workunit.client.0.smithi062.stdout:0/731: truncate d7/d1c/d15/d17/d40/d3c/fd8 325174 0 2024-04-20T21:32:11.100 INFO:tasks.workunit.client.1.smithi189.stdout:6/475: dread d8/d12/d30/f38 [0,4194304] 0 2024-04-20T21:32:11.103 INFO:tasks.workunit.client.1.smithi189.stdout:8/522: write da/de/f10 [7644004,106610] 0 2024-04-20T21:32:11.103 INFO:tasks.workunit.client.0.smithi062.stdout:5/750: truncate d8/d38/d7a/d4c/d7d/f98 1083126 0 2024-04-20T21:32:11.105 INFO:tasks.workunit.client.0.smithi062.stdout:3/705: mkdir d0/d1/d10/d3a/de9 0 2024-04-20T21:32:11.106 INFO:tasks.workunit.client.0.smithi062.stdout:9/838: dread d16/d2c/d5e/f7b [0,4194304] 0 2024-04-20T21:32:11.109 INFO:tasks.workunit.client.0.smithi062.stdout:9/839: fdatasync d16/d3d/d55/fb7 0 2024-04-20T21:32:11.112 INFO:tasks.workunit.client.0.smithi062.stdout:7/728: dread d7/d26/f29 [0,4194304] 0 2024-04-20T21:32:11.113 INFO:tasks.workunit.client.1.smithi189.stdout:4/414: dread d2/d20/f4f [0,4194304] 0 2024-04-20T21:32:11.115 INFO:tasks.workunit.client.1.smithi189.stdout:2/403: dwrite d1/d11/f2e [0,4194304] 0 2024-04-20T21:32:11.116 INFO:tasks.workunit.client.0.smithi062.stdout:3/706: symlink d0/d1/d10/d3a/lea 0 2024-04-20T21:32:11.120 INFO:tasks.workunit.client.0.smithi062.stdout:7/729: dread - d7/d26/dac/fd4 zero size 2024-04-20T21:32:11.123 INFO:tasks.workunit.client.1.smithi189.stdout:6/476: getdents d8/d2b/d49 0 2024-04-20T21:32:11.126 INFO:tasks.workunit.client.0.smithi062.stdout:9/840: getdents d16/d2f/d5b 0 2024-04-20T21:32:11.127 INFO:tasks.workunit.client.1.smithi189.stdout:8/523: dwrite da/de/d1e/d21/f97 [0,4194304] 0 2024-04-20T21:32:11.128 INFO:tasks.workunit.client.0.smithi062.stdout:3/707: unlink d0/d1/d3c/l50 0 2024-04-20T21:32:11.133 INFO:tasks.workunit.client.0.smithi062.stdout:7/730: symlink d7/lfb 0 2024-04-20T21:32:11.134 INFO:tasks.workunit.client.1.smithi189.stdout:8/524: truncate da/df/d22/d33/d59/fb9 562655 0 2024-04-20T21:32:11.137 INFO:tasks.workunit.client.0.smithi062.stdout:7/731: chown d7/c23 470 1 2024-04-20T21:32:11.138 INFO:tasks.workunit.client.0.smithi062.stdout:5/751: write d8/d38/d7a/d4c/d7d/d9d/dac/f6b [1851591,62353] 0 2024-04-20T21:32:11.144 INFO:tasks.workunit.client.0.smithi062.stdout:0/732: dwrite d7/d1c/d15/d80/fe9 [0,4194304] 0 2024-04-20T21:32:11.145 INFO:tasks.workunit.client.1.smithi189.stdout:2/404: mkdir d1/d17/d78 0 2024-04-20T21:32:11.146 INFO:tasks.workunit.client.1.smithi189.stdout:4/415: link d2/l9 d2/l95 0 2024-04-20T21:32:11.150 INFO:tasks.workunit.client.1.smithi189.stdout:4/416: dread d2/dd/d11/f74 [0,4194304] 0 2024-04-20T21:32:11.150 INFO:tasks.workunit.client.0.smithi062.stdout:0/733: unlink d7/d1c/f3e 0 2024-04-20T21:32:11.153 INFO:tasks.workunit.client.0.smithi062.stdout:9/841: dwrite d16/d2c/d5e/f7b [0,4194304] 0 2024-04-20T21:32:11.155 INFO:tasks.workunit.client.1.smithi189.stdout:8/525: dwrite da/df/d20/d27/f54 [0,4194304] 0 2024-04-20T21:32:11.156 INFO:tasks.workunit.client.0.smithi062.stdout:7/732: rename d7/c23 to d7/d3e/d33/d3d/d5e/cfc 0 2024-04-20T21:32:11.157 INFO:tasks.workunit.client.0.smithi062.stdout:2/760: dread d7/da/d15/d2f/f7e [0,4194304] 0 2024-04-20T21:32:11.160 INFO:tasks.workunit.client.0.smithi062.stdout:0/734: creat d7/d1e/db6/fee x:0 0 0 2024-04-20T21:32:11.161 INFO:tasks.workunit.client.0.smithi062.stdout:9/842: symlink d16/d2c/d62/l115 0 2024-04-20T21:32:11.163 INFO:tasks.workunit.client.1.smithi189.stdout:4/417: symlink d2/d7f/l96 0 2024-04-20T21:32:11.164 INFO:tasks.workunit.client.0.smithi062.stdout:7/733: mknod d7/d3e/d33/d3d/cfd 0 2024-04-20T21:32:11.167 INFO:tasks.workunit.client.0.smithi062.stdout:8/766: sync 2024-04-20T21:32:11.167 INFO:tasks.workunit.client.0.smithi062.stdout:1/805: sync 2024-04-20T21:32:11.168 INFO:tasks.workunit.client.0.smithi062.stdout:3/708: rename d0/d1/d3c/d43/d67/f86 to d0/d1/d10/d3a/d5e/d65/d88/feb 0 2024-04-20T21:32:11.170 INFO:tasks.workunit.client.0.smithi062.stdout:0/735: creat d7/d1c/d15/d17/d1b/d9d/db1/fef x:0 0 0 2024-04-20T21:32:11.171 INFO:tasks.workunit.client.1.smithi189.stdout:8/526: rename da/df/f19 to da/de/d84/fbd 0 2024-04-20T21:32:11.173 INFO:tasks.workunit.client.1.smithi189.stdout:8/527: stat da/de/d1e/d21/d87 0 2024-04-20T21:32:11.174 INFO:tasks.workunit.client.0.smithi062.stdout:8/767: dread - d7/d25/d3b/d58/d8b/ff1 zero size 2024-04-20T21:32:11.175 INFO:tasks.workunit.client.1.smithi189.stdout:4/418: mknod d2/dd/d11/d14/c97 0 2024-04-20T21:32:11.177 INFO:tasks.workunit.client.0.smithi062.stdout:2/761: rename d7/da/d15/d1b/l5c to d7/da/d15/d2f/d38/d6a/d7c/lfb 0 2024-04-20T21:32:11.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:11.177+0000 7f2caa600700 1 -- 172.21.15.62:0/2893377135 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2ca40ff130 msgr2=0x7f2ca40ff540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:32:11.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:11.177+0000 7f2caa600700 1 --2- 172.21.15.62:0/2893377135 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2ca40ff130 0x7f2ca40ff540 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f2c94009910 tx=0x7f2c94009c20 comp rx=0 tx=0).stop 2024-04-20T21:32:11.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:11.177+0000 7f2caa600700 1 -- 172.21.15.62:0/2893377135 shutdown_connections 2024-04-20T21:32:11.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:11.177+0000 7f2caa600700 1 --2- 172.21.15.62:0/2893377135 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ca41003d0 0x7f2ca4100840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:32:11.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:11.177+0000 7f2caa600700 1 --2- 172.21.15.62:0/2893377135 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2ca40ff130 0x7f2ca40ff540 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:32:11.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:11.177+0000 7f2caa600700 1 -- 172.21.15.62:0/2893377135 >> 172.21.15.62:0/2893377135 conn(0x7f2ca40fa780 msgr2=0x7f2ca40fcbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:32:11.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:11.177+0000 7f2caa600700 1 -- 172.21.15.62:0/2893377135 shutdown_connections 2024-04-20T21:32:11.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:11.178+0000 7f2caa600700 1 -- 172.21.15.62:0/2893377135 wait complete. 2024-04-20T21:32:11.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:11.179+0000 7f2caa600700 1 Processor -- start 2024-04-20T21:32:11.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:11.179+0000 7f2caa600700 1 -- start start 2024-04-20T21:32:11.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:11.180+0000 7f2caa600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ca40ff130 0x7f2ca4071a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:32:11.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:11.180+0000 7f2caa600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2ca41003d0 0x7f2ca4071fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:32:11.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:11.180+0000 7f2caa600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ca40725f0 con 0x7f2ca41003d0 2024-04-20T21:32:11.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:11.180+0000 7f2caa600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ca41a5a00 con 0x7f2ca40ff130 2024-04-20T21:32:11.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:11.181+0000 7f2ca3400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ca40ff130 0x7f2ca4071a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:32:11.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:11.182+0000 7f2ca3400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ca40ff130 0x7f2ca4071a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.62:41814/0 (socket says 172.21.15.62:41814) 2024-04-20T21:32:11.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:11.182+0000 7f2ca3400700 1 -- 172.21.15.62:0/73400775 learned_addr learned my addr 172.21.15.62:0/73400775 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:32:11.181 INFO:tasks.workunit.client.0.smithi062.stdout:2/762: rmdir d7/da/d28/d4b 39 2024-04-20T21:32:11.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:11.182+0000 7f2ca2a00700 1 --2- 172.21.15.62:0/73400775 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2ca41003d0 0x7f2ca4071fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:32:11.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:11.183+0000 7f2ca2a00700 1 -- 172.21.15.62:0/73400775 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ca40ff130 msgr2=0x7f2ca4071a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:32:11.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:11.184+0000 7f2ca2a00700 1 --2- 172.21.15.62:0/73400775 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ca40ff130 0x7f2ca4071a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:32:11.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:11.184+0000 7f2ca2a00700 1 -- 172.21.15.62:0/73400775 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c90009520 con 0x7f2ca41003d0 2024-04-20T21:32:11.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:11.184+0000 7f2ca3400700 1 --2- 172.21.15.62:0/73400775 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ca40ff130 0x7f2ca4071a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-20T21:32:11.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:11.185+0000 7f2ca2a00700 1 --2- 172.21.15.62:0/73400775 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2ca41003d0 0x7f2ca4071fb0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f2c9000e970 tx=0x7f2c9000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:32:11.185 INFO:tasks.workunit.client.0.smithi062.stdout:1/806: symlink dc/d15/d75/d94/d100/l126 0 2024-04-20T21:32:11.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:11.186+0000 7f2c9be00700 1 -- 172.21.15.62:0/73400775 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2c9000cb40 con 0x7f2ca41003d0 2024-04-20T21:32:11.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:11.186+0000 7f2c9be00700 1 -- 172.21.15.62:0/73400775 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f2c9000eec0 con 0x7f2ca41003d0 2024-04-20T21:32:11.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:11.186+0000 7f2c9be00700 1 -- 172.21.15.62:0/73400775 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2c90003d30 con 0x7f2ca41003d0 2024-04-20T21:32:11.186 INFO:tasks.workunit.client.1.smithi189.stdout:4/419: dread d2/f2c [0,4194304] 0 2024-04-20T21:32:11.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:11.188+0000 7f2caa600700 1 -- 172.21.15.62:0/73400775 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c940095f0 con 0x7f2ca41003d0 2024-04-20T21:32:11.187 INFO:tasks.workunit.client.1.smithi189.stdout:8/528: unlink da/f39 0 2024-04-20T21:32:11.188 INFO:tasks.workunit.client.0.smithi062.stdout:8/768: creat d7/d25/d3b/d58/de8/ffc x:0 0 0 2024-04-20T21:32:11.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:11.189+0000 7f2caa600700 1 -- 172.21.15.62:0/73400775 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ca41a5f30 con 0x7f2ca41003d0 2024-04-20T21:32:11.188 INFO:tasks.workunit.client.1.smithi189.stdout:4/420: truncate d2/d22/f45 1716824 0 2024-04-20T21:32:11.189 INFO:tasks.workunit.client.0.smithi062.stdout:0/736: creat d7/d1c/d15/d17/d37/d95/ff0 x:0 0 0 2024-04-20T21:32:11.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:11.190+0000 7f2caa600700 1 -- 172.21.15.62:0/73400775 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ca4102b10 con 0x7f2ca41003d0 2024-04-20T21:32:11.189 INFO:tasks.workunit.client.1.smithi189.stdout:8/529: chown da/df/d22/d33/l62 67 1 2024-04-20T21:32:11.194 INFO:tasks.workunit.client.0.smithi062.stdout:1/807: unlink dc/f6d 0 2024-04-20T21:32:11.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:11.196+0000 7f2c9be00700 1 -- 172.21.15.62:0/73400775 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 29) v1 ==== 92700+0+0 (secure 0 0 0) 0x7f2c90003e90 con 0x7f2ca41003d0 2024-04-20T21:32:11.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:11.199+0000 7f2c9be00700 1 --2- 172.21.15.62:0/73400775 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f2c8806eb00 0x7f2c88070fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:32:11.199 INFO:tasks.workunit.client.0.smithi062.stdout:9/843: dwrite d16/d2c/d42/da6/f10e [0,4194304] 0 2024-04-20T21:32:11.199 INFO:tasks.workunit.client.0.smithi062.stdout:5/752: dread d8/d38/f89 [0,4194304] 0 2024-04-20T21:32:11.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:11.199+0000 7f2c9be00700 1 -- 172.21.15.62:0/73400775 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f2c90014070 con 0x7f2ca41003d0 2024-04-20T21:32:11.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:11.201+0000 7f2ca3400700 1 --2- 172.21.15.62:0/73400775 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f2c8806eb00 0x7f2c88070fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:32:11.200 INFO:tasks.workunit.client.0.smithi062.stdout:1/808: readlink dc/d15/d20/d28/l9e 0 2024-04-20T21:32:11.200 INFO:tasks.workunit.client.0.smithi062.stdout:2/763: getdents d7/da/d15/d2f/d38/d6a 0 2024-04-20T21:32:11.201 INFO:tasks.workunit.client.1.smithi189.stdout:4/421: mknod d2/d22/d57/c98 0 2024-04-20T21:32:11.202 INFO:tasks.workunit.client.0.smithi062.stdout:8/769: creat d7/d25/d3b/d58/d8b/ffd x:0 0 0 2024-04-20T21:32:11.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:11.203+0000 7f2ca3400700 1 --2- 172.21.15.62:0/73400775 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f2c8806eb00 0x7f2c88070fb0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f2c9400b6f0 tx=0x7f2c94009f90 comp rx=0 tx=0).ready entity=mgr.14658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:32:11.202 INFO:tasks.workunit.client.1.smithi189.stdout:4/422: write d2/f54 [2214839,92969] 0 2024-04-20T21:32:11.203 INFO:tasks.workunit.client.1.smithi189.stdout:4/423: chown d2/cf 3 1 2024-04-20T21:32:11.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:11.205+0000 7f2c9be00700 1 -- 172.21.15.62:0/73400775 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187527 (secure 0 0 0) 0x7f2c9005bc60 con 0x7f2ca41003d0 2024-04-20T21:32:11.204 INFO:tasks.workunit.client.0.smithi062.stdout:1/809: rmdir dc/d27 39 2024-04-20T21:32:11.204 INFO:tasks.workunit.client.1.smithi189.stdout:7/475: truncate d5/f49 1206940 0 2024-04-20T21:32:11.205 INFO:tasks.workunit.client.1.smithi189.stdout:4/424: dread - d2/dd/d11/f4b zero size 2024-04-20T21:32:11.205 INFO:tasks.workunit.client.1.smithi189.stdout:0/547: link d3/d26/ca7 d3/df/d30/d45/d60/cba 0 2024-04-20T21:32:11.207 INFO:tasks.workunit.client.0.smithi062.stdout:2/764: creat d7/da/d15/d1b/d3d/ffc x:0 0 0 2024-04-20T21:32:11.207 INFO:tasks.workunit.client.1.smithi189.stdout:7/476: chown d5/d21/d25/f78 3 1 2024-04-20T21:32:11.207 INFO:tasks.workunit.client.1.smithi189.stdout:0/548: stat d3/df/d30/d45/d60 0 2024-04-20T21:32:11.208 INFO:tasks.workunit.client.0.smithi062.stdout:9/844: truncate d16/d4e/f106 4866935 0 2024-04-20T21:32:11.208 INFO:tasks.workunit.client.1.smithi189.stdout:8/530: creat da/df/d3c/d50/d6c/d85/fbe x:0 0 0 2024-04-20T21:32:11.208 INFO:tasks.workunit.client.1.smithi189.stdout:7/477: readlink d5/d21/d35/l3d 0 2024-04-20T21:32:11.209 INFO:tasks.workunit.client.1.smithi189.stdout:0/549: read d3/df/d30/d45/f6c [730,47949] 0 2024-04-20T21:32:11.210 INFO:tasks.workunit.client.0.smithi062.stdout:5/753: link d8/d38/d7a/d4c/d7d/d9d/dac/f6b d8/d38/d7a/d4c/d7d/d9d/dac/d66/daa/ffa 0 2024-04-20T21:32:11.218 INFO:tasks.workunit.client.0.smithi062.stdout:9/845: read d16/d2c/d5e/d75/fdd [765301,42753] 0 2024-04-20T21:32:11.218 INFO:tasks.workunit.client.0.smithi062.stdout:5/754: mknod d8/d38/d7a/d4c/d7d/d9d/cfb 0 2024-04-20T21:32:11.218 INFO:tasks.workunit.client.1.smithi189.stdout:0/550: dread - d3/d8c/f7a zero size 2024-04-20T21:32:11.219 INFO:tasks.workunit.client.1.smithi189.stdout:3/410: truncate d2/d3/d2c/f6d 168135 0 2024-04-20T21:32:11.220 INFO:tasks.workunit.client.0.smithi062.stdout:2/765: rename d7/da/d15/d1b/d3d/f78 to d7/da/d28/d4b/d8f/ffd 0 2024-04-20T21:32:11.220 INFO:tasks.workunit.client.1.smithi189.stdout:3/411: read - d2/d3/d65/f8b zero size 2024-04-20T21:32:11.223 INFO:tasks.workunit.client.1.smithi189.stdout:3/412: chown d2/d3/d2c/d31/d90/d92 116 1 2024-04-20T21:32:11.225 INFO:tasks.workunit.client.0.smithi062.stdout:2/766: chown d7/da/d15/d1b/f2e 27781 1 2024-04-20T21:32:11.234 INFO:tasks.workunit.client.1.smithi189.stdout:0/551: link d3/df/d28/f61 d3/df/d30/d37/fbb 0 2024-04-20T21:32:11.235 INFO:tasks.workunit.client.1.smithi189.stdout:0/552: stat d3/df/d5d/f99 0 2024-04-20T21:32:11.237 INFO:tasks.workunit.client.1.smithi189.stdout:6/477: write d8/da/f27 [1261203,87081] 0 2024-04-20T21:32:11.239 INFO:tasks.workunit.client.1.smithi189.stdout:0/553: unlink d3/f5e 0 2024-04-20T21:32:11.241 INFO:tasks.workunit.client.0.smithi062.stdout:5/755: dread d8/f44 [0,4194304] 0 2024-04-20T21:32:11.244 INFO:tasks.workunit.client.1.smithi189.stdout:6/478: creat d8/d12/d4e/fa6 x:0 0 0 2024-04-20T21:32:11.244 INFO:tasks.workunit.client.1.smithi189.stdout:1/380: sync 2024-04-20T21:32:11.248 INFO:tasks.workunit.client.0.smithi062.stdout:2/767: dwrite d7/da/d15/d2f/d38/d6b/fd1 [0,4194304] 0 2024-04-20T21:32:11.250 INFO:tasks.workunit.client.1.smithi189.stdout:6/479: mkdir d8/da/da7 0 2024-04-20T21:32:11.252 INFO:tasks.workunit.client.0.smithi062.stdout:8/770: dwrite d7/d25/dc5/ff0 [0,4194304] 0 2024-04-20T21:32:11.254 INFO:tasks.workunit.client.0.smithi062.stdout:7/734: write d7/d26/dac/fd4 [356968,52510] 0 2024-04-20T21:32:11.254 INFO:tasks.workunit.client.0.smithi062.stdout:3/709: write d0/d1c/d63/d42/dc8/dc9/fcf [453814,119803] 0 2024-04-20T21:32:11.254 INFO:tasks.workunit.client.0.smithi062.stdout:5/756: truncate d8/d10/d3e/ff6 246063 0 2024-04-20T21:32:11.255 INFO:tasks.workunit.client.0.smithi062.stdout:2/768: symlink d7/da/d15/d2f/d38/db0/lfe 0 2024-04-20T21:32:11.257 INFO:tasks.workunit.client.1.smithi189.stdout:2/405: dwrite d1/d17/d2f/d55/f5f [0,4194304] 0 2024-04-20T21:32:11.261 INFO:tasks.workunit.client.0.smithi062.stdout:0/737: write d7/d1c/d15/d17/d37/fbf [1874640,4686] 0 2024-04-20T21:32:11.262 INFO:tasks.workunit.client.1.smithi189.stdout:2/406: chown d1/f5 3 1 2024-04-20T21:32:11.264 INFO:tasks.workunit.client.0.smithi062.stdout:8/771: fdatasync d7/d25/d5c/f72 0 2024-04-20T21:32:11.265 INFO:tasks.workunit.client.0.smithi062.stdout:7/735: fdatasync d7/d3e/d33/d3d/dad/fbc 0 2024-04-20T21:32:11.267 INFO:tasks.workunit.client.1.smithi189.stdout:7/478: read d5/f49 [177853,124254] 0 2024-04-20T21:32:11.267 INFO:tasks.workunit.client.0.smithi062.stdout:3/710: fdatasync d0/d1c/d63/d42/dc8/dc9/fcf 0 2024-04-20T21:32:11.269 INFO:tasks.workunit.client.0.smithi062.stdout:7/736: chown d7/d26/l91 73 1 2024-04-20T21:32:11.270 INFO:tasks.workunit.client.1.smithi189.stdout:2/407: creat d1/d17/d2f/d4a/f79 x:0 0 0 2024-04-20T21:32:11.274 INFO:tasks.workunit.client.0.smithi062.stdout:1/810: dwrite dc/d15/fdd [0,4194304] 0 2024-04-20T21:32:11.275 INFO:tasks.workunit.client.1.smithi189.stdout:1/381: creat d8/df/f83 x:0 0 0 2024-04-20T21:32:11.275 INFO:tasks.workunit.client.1.smithi189.stdout:7/479: truncate d5/fe 2061162 0 2024-04-20T21:32:11.275 INFO:tasks.workunit.client.0.smithi062.stdout:3/711: chown d0/d1/d3c/d43/d72/dd3/fbc 7073053 1 2024-04-20T21:32:11.277 INFO:tasks.workunit.client.1.smithi189.stdout:2/408: truncate d1/d11/d58/f6e 1000429 0 2024-04-20T21:32:11.277 INFO:tasks.workunit.client.0.smithi062.stdout:9/846: truncate d16/d2c/d42/da6/f10e 2298335 0 2024-04-20T21:32:11.277 INFO:tasks.workunit.client.1.smithi189.stdout:7/480: fsync d5/d21/d25/f6c 0 2024-04-20T21:32:11.279 INFO:tasks.workunit.client.0.smithi062.stdout:1/811: write dc/d15/d75/d94/fec [1007622,123429] 0 2024-04-20T21:32:11.279 INFO:tasks.workunit.client.0.smithi062.stdout:4/839: sync 2024-04-20T21:32:11.281 INFO:tasks.workunit.client.1.smithi189.stdout:2/409: stat d1/d11/d58/d6b/f77 0 2024-04-20T21:32:11.283 INFO:tasks.workunit.client.1.smithi189.stdout:3/413: rmdir d2/d3/d2c 39 2024-04-20T21:32:11.288 INFO:tasks.workunit.client.1.smithi189.stdout:0/554: truncate d3/df/d28/d2b/f49 1761163 0 2024-04-20T21:32:11.288 INFO:tasks.workunit.client.0.smithi062.stdout:0/738: mkdir d7/d1c/d15/d17/d1b/d9d/df1 0 2024-04-20T21:32:11.288 INFO:tasks.workunit.client.0.smithi062.stdout:5/757: dwrite d8/d38/d7a/f40 [4194304,4194304] 0 2024-04-20T21:32:11.289 INFO:tasks.workunit.client.0.smithi062.stdout:4/840: truncate da/dd/d15/d50/d64/d3d/fc6 4260688 0 2024-04-20T21:32:11.289 INFO:tasks.workunit.client.1.smithi189.stdout:6/480: dwrite d8/da/dd/d35/f31 [0,4194304] 0 2024-04-20T21:32:11.290 INFO:tasks.workunit.client.1.smithi189.stdout:7/481: creat d5/d21/d4a/d52/fa7 x:0 0 0 2024-04-20T21:32:11.290 INFO:tasks.workunit.client.1.smithi189.stdout:3/414: dread - d2/d1c/f7d zero size 2024-04-20T21:32:11.295 INFO:tasks.workunit.client.1.smithi189.stdout:4/425: dwrite d2/dd/d37/f59 [0,4194304] 0 2024-04-20T21:32:11.298 INFO:tasks.workunit.client.0.smithi062.stdout:2/769: truncate d7/d39/f45 2202616 0 2024-04-20T21:32:11.298 INFO:tasks.workunit.client.1.smithi189.stdout:6/481: mknod d8/da/ca8 0 2024-04-20T21:32:11.300 INFO:tasks.workunit.client.0.smithi062.stdout:3/712: fdatasync d0/d1/d10/fa4 0 2024-04-20T21:32:11.300 INFO:tasks.workunit.client.0.smithi062.stdout:2/770: chown d7/da/d15/d2f/d38/d6a 0 1 2024-04-20T21:32:11.303 INFO:tasks.workunit.client.0.smithi062.stdout:1/812: read - dc/d15/d75/d94/d6f/f99 zero size 2024-04-20T21:32:11.304 INFO:tasks.workunit.client.1.smithi189.stdout:2/410: getdents d1/d11/d2d 0 2024-04-20T21:32:11.309 INFO:tasks.workunit.client.0.smithi062.stdout:0/739: mknod d7/d20/d3a/cf2 0 2024-04-20T21:32:11.313 INFO:tasks.workunit.client.0.smithi062.stdout:4/841: symlink da/d56/d71/dbd/l12a 0 2024-04-20T21:32:11.315 INFO:tasks.workunit.client.0.smithi062.stdout:7/737: mknod d7/cfe 0 2024-04-20T21:32:11.315 INFO:tasks.workunit.client.1.smithi189.stdout:1/382: dwrite d8/d1e/f3c [0,4194304] 0 2024-04-20T21:32:11.315 INFO:tasks.workunit.client.1.smithi189.stdout:4/426: creat d2/d22/d57/f99 x:0 0 0 2024-04-20T21:32:11.318 INFO:tasks.workunit.client.1.smithi189.stdout:7/482: write d5/d1d/d77/d5b/f38 [8944847,116303] 0 2024-04-20T21:32:11.318 INFO:tasks.workunit.client.0.smithi062.stdout:3/713: mknod d0/d1/d10/d31/d41/cec 0 2024-04-20T21:32:11.320 INFO:tasks.workunit.client.1.smithi189.stdout:1/383: readlink d8/d21/d2c/l7f 0 2024-04-20T21:32:11.321 INFO:tasks.workunit.client.1.smithi189.stdout:0/555: link d3/df/d28/lb2 d3/df/d5d/d64/lbc 0 2024-04-20T21:32:11.323 INFO:tasks.workunit.client.0.smithi062.stdout:5/758: rename d8/d1e/f53 to d8/d38/ffc 0 2024-04-20T21:32:11.323 INFO:tasks.workunit.client.0.smithi062.stdout:1/813: read dc/d15/d75/d96/dd4/d3a/da2/de4/d84/fb0 [1630609,38414] 0 2024-04-20T21:32:11.325 INFO:tasks.workunit.client.1.smithi189.stdout:1/384: dread - d8/d1e/d28/f7e zero size 2024-04-20T21:32:11.327 INFO:tasks.workunit.client.1.smithi189.stdout:3/415: symlink d2/d3/d2c/d31/d4c/l96 0 2024-04-20T21:32:11.330 INFO:tasks.workunit.client.0.smithi062.stdout:3/714: write d0/d1/d3c/d43/d72/dd3/d7b/f85 [881215,51450] 0 2024-04-20T21:32:11.330 INFO:tasks.workunit.client.0.smithi062.stdout:0/740: rename d7/d1c/d15/d17/d40/d79/l78 to d7/d1c/d15/d17/d40/d3c/lf3 0 2024-04-20T21:32:11.334 INFO:tasks.workunit.client.0.smithi062.stdout:9/847: dwrite d16/d2c/d42/da6/da9/feb [0,4194304] 0 2024-04-20T21:32:11.335 INFO:tasks.workunit.client.0.smithi062.stdout:4/842: rmdir da/dd/d15/d50/d64/d57/d5b/df7 0 2024-04-20T21:32:11.337 INFO:tasks.workunit.client.1.smithi189.stdout:7/483: rmdir d5/d1d 39 2024-04-20T21:32:11.340 INFO:tasks.workunit.client.0.smithi062.stdout:3/715: dread - d0/d1c/d63/d42/fcb zero size 2024-04-20T21:32:11.340 INFO:tasks.workunit.client.0.smithi062.stdout:0/741: readlink d7/d1c/d15/d17/d1b/l49 0 2024-04-20T21:32:11.344 INFO:tasks.workunit.client.0.smithi062.stdout:9/848: fdatasync d16/d2c/d62/fbc 0 2024-04-20T21:32:11.344 INFO:tasks.workunit.client.0.smithi062.stdout:0/742: chown d7/d20/d3a 3524 1 2024-04-20T21:32:11.346 INFO:tasks.workunit.client.1.smithi189.stdout:9/439: sync 2024-04-20T21:32:11.346 INFO:tasks.workunit.client.1.smithi189.stdout:5/389: sync 2024-04-20T21:32:11.346 INFO:tasks.workunit.client.1.smithi189.stdout:6/482: truncate d8/d10/d78/d7e/f84 2330705 0 2024-04-20T21:32:11.348 INFO:tasks.workunit.client.1.smithi189.stdout:0/556: mkdir d3/df/d30/d37/dbd 0 2024-04-20T21:32:11.350 INFO:tasks.workunit.client.0.smithi062.stdout:1/814: getdents dc/d15/d75/d94/d6f/dd2 0 2024-04-20T21:32:11.351 INFO:tasks.workunit.client.0.smithi062.stdout:9/849: fsync d16/d3d/d55/faf 0 2024-04-20T21:32:11.351 INFO:tasks.workunit.client.0.smithi062.stdout:1/815: chown dc/d15/d75/f123 0 1 2024-04-20T21:32:11.351 INFO:tasks.workunit.client.1.smithi189.stdout:7/484: chown d5/d21/d4a/d52/d74/l9a 79 1 2024-04-20T21:32:11.352 INFO:tasks.workunit.client.1.smithi189.stdout:3/416: creat d2/d3/d25/f97 x:0 0 0 2024-04-20T21:32:11.354 INFO:tasks.workunit.client.1.smithi189.stdout:0/557: fsync d3/df/d5d/f99 0 2024-04-20T21:32:11.356 INFO:tasks.workunit.client.1.smithi189.stdout:5/390: write d7/d9/f42 [1684760,108419] 0 2024-04-20T21:32:11.357 INFO:tasks.workunit.client.0.smithi062.stdout:9/850: truncate fd 2179468 0 2024-04-20T21:32:11.357 INFO:tasks.workunit.client.1.smithi189.stdout:3/417: write d2/d3/d25/f5b [469482,28889] 0 2024-04-20T21:32:11.359 INFO:tasks.workunit.client.1.smithi189.stdout:0/558: chown d3/c39 0 1 2024-04-20T21:32:11.359 INFO:tasks.workunit.client.1.smithi189.stdout:1/385: dwrite d8/df/f13 [0,4194304] 0 2024-04-20T21:32:11.360 INFO:tasks.workunit.client.0.smithi062.stdout:1/816: mknod dc/d15/d75/d96/c127 0 2024-04-20T21:32:11.365 INFO:tasks.workunit.client.1.smithi189.stdout:4/427: creat d2/dd/d11/f9a x:0 0 0 2024-04-20T21:32:11.367 INFO:tasks.workunit.client.1.smithi189.stdout:1/386: write d8/d21/d2c/f62 [1148508,72156] 0 2024-04-20T21:32:11.371 INFO:tasks.workunit.client.1.smithi189.stdout:1/387: write d8/d21/d2c/f6c [1752247,66644] 0 2024-04-20T21:32:11.371 INFO:tasks.workunit.client.0.smithi062.stdout:3/716: dwrite d0/d1b/fac [0,4194304] 0 2024-04-20T21:32:11.373 INFO:tasks.workunit.client.1.smithi189.stdout:1/388: chown d8/df/c55 7744 1 2024-04-20T21:32:11.374 INFO:tasks.workunit.client.0.smithi062.stdout:1/817: creat dc/d15/d20/d28/d83/f128 x:0 0 0 2024-04-20T21:32:11.377 INFO:tasks.workunit.client.0.smithi062.stdout:7/738: dread d7/ff [0,4194304] 0 2024-04-20T21:32:11.378 INFO:tasks.workunit.client.0.smithi062.stdout:6/649: sync 2024-04-20T21:32:11.379 INFO:tasks.workunit.client.1.smithi189.stdout:5/391: mkdir d7/d9/d51/d70 0 2024-04-20T21:32:11.379 INFO:tasks.workunit.client.1.smithi189.stdout:3/418: mknod d2/d1c/d5c/c98 0 2024-04-20T21:32:11.384 INFO:tasks.workunit.client.0.smithi062.stdout:1/818: creat dc/d15/df6/f129 x:0 0 0 2024-04-20T21:32:11.385 INFO:tasks.workunit.client.1.smithi189.stdout:6/483: mknod d8/d10/d78/d7e/ca9 0 2024-04-20T21:32:11.385 INFO:tasks.workunit.client.0.smithi062.stdout:3/717: write d0/d1c/d63/fbb [5051055,129767] 0 2024-04-20T21:32:11.385 INFO:tasks.workunit.client.1.smithi189.stdout:3/419: creat d2/d3/d88/f99 x:0 0 0 2024-04-20T21:32:11.386 INFO:tasks.workunit.client.1.smithi189.stdout:5/392: mkdir d7/d35/d48/d71 0 2024-04-20T21:32:11.386 INFO:tasks.workunit.client.0.smithi062.stdout:7/739: creat d7/d3e/d33/d3d/d4d/d54/d4f/de9/df1/fff x:0 0 0 2024-04-20T21:32:11.388 INFO:tasks.workunit.client.0.smithi062.stdout:6/650: unlink d0/d4f/cd3 0 2024-04-20T21:32:11.389 INFO:tasks.workunit.client.1.smithi189.stdout:1/389: link d8/d1e/d43/d44/f47 d8/d21/d2c/d36/f84 0 2024-04-20T21:32:11.390 INFO:tasks.workunit.client.1.smithi189.stdout:5/393: write d7/d14/d18/f66 [940559,37660] 0 2024-04-20T21:32:11.401 INFO:tasks.workunit.client.1.smithi189.stdout:6/484: dread d8/d10/d78/f7d [0,4194304] 0 2024-04-20T21:32:11.401 INFO:tasks.workunit.client.0.smithi062.stdout:3/718: creat d0/d1c/d63/dda/fed x:0 0 0 2024-04-20T21:32:11.403 INFO:tasks.workunit.client.0.smithi062.stdout:8/772: dwrite d7/d9/d29/f4c [0,4194304] 0 2024-04-20T21:32:11.404 INFO:tasks.workunit.client.1.smithi189.stdout:9/440: link d9/f33 d9/dc/d17/d2b/d56/f8a 0 2024-04-20T21:32:11.405 INFO:tasks.workunit.client.0.smithi062.stdout:8/773: readlink d7/da9/lc9 0 2024-04-20T21:32:11.407 INFO:tasks.workunit.client.1.smithi189.stdout:0/559: dwrite d3/df/d4c/fa4 [0,4194304] 0 2024-04-20T21:32:11.409 INFO:tasks.workunit.client.1.smithi189.stdout:2/411: dwrite d1/d17/d37/f67 [0,4194304] 0 2024-04-20T21:32:11.409 INFO:tasks.workunit.client.0.smithi062.stdout:6/651: truncate d0/d1a/d44/fa3 256596 0 2024-04-20T21:32:11.409 INFO:tasks.workunit.client.0.smithi062.stdout:1/819: truncate dc/d15/d75/d94/f34 3910694 0 2024-04-20T21:32:11.413 INFO:tasks.workunit.client.1.smithi189.stdout:5/394: creat d7/d9/d51/f72 x:0 0 0 2024-04-20T21:32:11.414 INFO:tasks.workunit.client.1.smithi189.stdout:6/485: chown d8/da/dd/d35/d6d/d77/f93 1507354252 1 2024-04-20T21:32:11.415 INFO:tasks.workunit.client.0.smithi062.stdout:3/719: creat d0/d1/d10/d3a/d5e/ddb/fee x:0 0 0 2024-04-20T21:32:11.415 INFO:tasks.workunit.client.0.smithi062.stdout:7/740: link d7/d26/l4b d7/d3e/d33/d3d/d4d/d54/d4f/de9/df1/df3/l100 0 2024-04-20T21:32:11.420 INFO:tasks.workunit.client.0.smithi062.stdout:2/771: write d7/da/d28/d4b/d8f/ffd [4456302,94373] 0 2024-04-20T21:32:11.421 INFO:tasks.workunit.client.1.smithi189.stdout:4/428: getdents d2/dd/d11/d14 0 2024-04-20T21:32:11.422 INFO:tasks.workunit.client.1.smithi189.stdout:1/390: truncate d8/f11 3560702 0 2024-04-20T21:32:11.422 INFO:tasks.workunit.client.0.smithi062.stdout:7/741: truncate d7/f48 837200 0 2024-04-20T21:32:11.427 INFO:tasks.workunit.client.1.smithi189.stdout:6/486: write d8/da/fa3 [827367,44891] 0 2024-04-20T21:32:11.427 INFO:tasks.workunit.client.1.smithi189.stdout:1/391: fdatasync d8/d1e/d43/d58/d60/f64 0 2024-04-20T21:32:11.427 INFO:tasks.workunit.client.0.smithi062.stdout:4/843: write da/dd/df/d27/d3f/f6e [992186,44044] 0 2024-04-20T21:32:11.429 INFO:tasks.workunit.client.1.smithi189.stdout:5/395: rename d7/d9/f6b to d7/d9/d30/d5a/f73 0 2024-04-20T21:32:11.431 INFO:tasks.workunit.client.1.smithi189.stdout:6/487: write d8/f21 [4632479,24101] 0 2024-04-20T21:32:11.435 INFO:tasks.workunit.client.0.smithi062.stdout:3/720: creat d0/d1c/d63/fef x:0 0 0 2024-04-20T21:32:11.436 INFO:tasks.workunit.client.0.smithi062.stdout:8/774: dread d7/d25/dc5/ff0 [0,4194304] 0 2024-04-20T21:32:11.437 INFO:tasks.workunit.client.0.smithi062.stdout:5/759: dwrite d8/d10/d3e/d8a/fa0 [0,4194304] 0 2024-04-20T21:32:11.437 INFO:tasks.workunit.client.1.smithi189.stdout:1/392: unlink d8/d1e/d28/f7e 0 2024-04-20T21:32:11.442 INFO:tasks.workunit.client.1.smithi189.stdout:0/560: dwrite d3/df/d4c/f7e [4194304,4194304] 0 2024-04-20T21:32:11.442 INFO:tasks.workunit.client.1.smithi189.stdout:9/441: dwrite d9/dc/d17/d2b/f6a [0,4194304] 0 2024-04-20T21:32:11.443 INFO:tasks.workunit.client.1.smithi189.stdout:1/393: chown d8/df/d3a/c7c 13 1 2024-04-20T21:32:11.445 INFO:tasks.workunit.client.1.smithi189.stdout:5/396: read d7/d14/d18/f17 [816180,121043] 0 2024-04-20T21:32:11.446 INFO:tasks.workunit.client.0.smithi062.stdout:5/760: write d8/d38/d7a/d4c/dcf/ff2 [161743,81415] 0 2024-04-20T21:32:11.449 INFO:tasks.workunit.client.1.smithi189.stdout:6/488: mknod d8/da/caa 0 2024-04-20T21:32:11.449 INFO:tasks.workunit.client.1.smithi189.stdout:1/394: chown d8/d1e/d43/d44 771731 1 2024-04-20T21:32:11.450 INFO:tasks.workunit.client.1.smithi189.stdout:2/412: dwrite d1/d11/d2d/f38 [0,4194304] 0 2024-04-20T21:32:11.450 INFO:tasks.workunit.client.1.smithi189.stdout:4/429: link d2/d7f/f8e d2/f9b 0 2024-04-20T21:32:11.451 INFO:tasks.workunit.client.1.smithi189.stdout:1/395: stat d8/d1e/l81 0 2024-04-20T21:32:11.451 INFO:tasks.workunit.client.1.smithi189.stdout:5/397: fdatasync d7/d9/d28/f58 0 2024-04-20T21:32:11.452 INFO:tasks.workunit.client.0.smithi062.stdout:5/761: read d8/f44 [1804197,70954] 0 2024-04-20T21:32:11.452 INFO:tasks.workunit.client.1.smithi189.stdout:6/489: stat d8/d12/d30/f58 0 2024-04-20T21:32:11.453 INFO:tasks.workunit.client.0.smithi062.stdout:5/762: readlink d8/d38/l87 0 2024-04-20T21:32:11.453 INFO:tasks.workunit.client.1.smithi189.stdout:0/561: mkdir d3/db9/dbe 0 2024-04-20T21:32:11.454 INFO:tasks.workunit.client.0.smithi062.stdout:5/763: fdatasync d8/d38/fbe 0 2024-04-20T21:32:11.457 INFO:tasks.workunit.client.1.smithi189.stdout:5/398: stat d7/d9/d30/d55/d33 0 2024-04-20T21:32:11.458 INFO:tasks.workunit.client.1.smithi189.stdout:2/413: mkdir d1/d11/d2d/d7a 0 2024-04-20T21:32:11.459 INFO:tasks.workunit.client.1.smithi189.stdout:6/490: write d8/d2b/f90 [885350,79002] 0 2024-04-20T21:32:11.461 INFO:tasks.workunit.client.1.smithi189.stdout:8/531: sync 2024-04-20T21:32:11.462 INFO:tasks.workunit.client.1.smithi189.stdout:2/414: chown d1/d17/d2f/d4a/f79 28606 1 2024-04-20T21:32:11.464 INFO:tasks.workunit.client.0.smithi062.stdout:8/775: rename d7/f57 to d7/d25/d3b/d58/d8b/ffe 0 2024-04-20T21:32:11.465 INFO:tasks.workunit.client.1.smithi189.stdout:1/396: unlink d8/d1e/d28/d2b/f61 0 2024-04-20T21:32:11.469 INFO:tasks.workunit.client.1.smithi189.stdout:5/399: rename d7/d9/d51/f6d to d7/d35/f74 0 2024-04-20T21:32:11.470 INFO:tasks.workunit.client.1.smithi189.stdout:6/491: read d8/d12/f17 [2193019,77560] 0 2024-04-20T21:32:11.471 INFO:tasks.workunit.client.1.smithi189.stdout:8/532: rmdir da/df 39 2024-04-20T21:32:11.472 INFO:tasks.workunit.client.1.smithi189.stdout:9/442: dwrite f4 [0,4194304] 0 2024-04-20T21:32:11.472 INFO:tasks.workunit.client.1.smithi189.stdout:2/415: creat d1/d17/d2f/d4a/f7b x:0 0 0 2024-04-20T21:32:11.479 INFO:tasks.workunit.client.1.smithi189.stdout:6/492: dread - d8/d12/f95 zero size 2024-04-20T21:32:11.482 INFO:tasks.workunit.client.1.smithi189.stdout:1/397: link d8/d1e/d43/c6f d8/df/d3a/d49/c85 0 2024-04-20T21:32:11.483 INFO:tasks.workunit.client.0.smithi062.stdout:4/844: rename da/dd/d15/d50/d64/d57/d5b/f8d to da/dd/d15/d50/d64/d57/d5b/d9d/f12b 0 2024-04-20T21:32:11.486 INFO:tasks.workunit.client.0.smithi062.stdout:4/845: read - da/dd/df/d27/d3f/f119 zero size 2024-04-20T21:32:11.487 INFO:tasks.workunit.client.1.smithi189.stdout:6/493: dread - d8/da/dd/d35/d6d/d77/f93 zero size 2024-04-20T21:32:11.491 INFO:tasks.workunit.client.1.smithi189.stdout:8/533: unlink da/df/d3c/d50/f5e 0 2024-04-20T21:32:11.492 INFO:tasks.workunit.client.0.smithi062.stdout:5/764: mknod d8/d38/d7a/d4c/cfd 0 2024-04-20T21:32:11.495 INFO:tasks.workunit.client.0.smithi062.stdout:8/776: dread d7/d25/d3b/d58/d8b/d24/ff3 [0,4194304] 0 2024-04-20T21:32:11.496 INFO:tasks.workunit.client.1.smithi189.stdout:7/485: sync 2024-04-20T21:32:11.498 INFO:tasks.workunit.client.0.smithi062.stdout:4/846: mknod da/dd/df/d27/d32/d97/c12c 0 2024-04-20T21:32:11.500 INFO:tasks.workunit.client.0.smithi062.stdout:8/777: truncate d7/d30/db3/fd9 5090 0 2024-04-20T21:32:11.500 INFO:tasks.workunit.client.1.smithi189.stdout:1/398: creat d8/f86 x:0 0 0 2024-04-20T21:32:11.501 INFO:tasks.workunit.client.1.smithi189.stdout:6/494: rmdir d8/d2b 39 2024-04-20T21:32:11.503 INFO:tasks.workunit.client.1.smithi189.stdout:4/430: dwrite d2/dd/d11/d14/d69/d6a/d7a/f8c [0,4194304] 0 2024-04-20T21:32:11.504 INFO:tasks.workunit.client.0.smithi062.stdout:5/765: chown d8/d1e/d2c/d6c/l45 97 1 2024-04-20T21:32:11.504 INFO:tasks.workunit.client.0.smithi062.stdout:0/743: truncate d7/d1c/d15/d17/d37/f59 1447645 0 2024-04-20T21:32:11.514 INFO:tasks.workunit.client.1.smithi189.stdout:8/534: rmdir da/df/d22/d33/d59 39 2024-04-20T21:32:11.514 INFO:tasks.workunit.client.1.smithi189.stdout:5/400: dwrite d7/d14/f4c [0,4194304] 0 2024-04-20T21:32:11.515 INFO:tasks.workunit.client.1.smithi189.stdout:2/416: dwrite d1/d17/d2f/f72 [0,4194304] 0 2024-04-20T21:32:11.521 INFO:tasks.workunit.client.1.smithi189.stdout:4/431: symlink d2/dd/l9c 0 2024-04-20T21:32:11.521 INFO:tasks.workunit.client.1.smithi189.stdout:7/486: symlink d5/d1d/d77/d5b/d71/la8 0 2024-04-20T21:32:11.523 INFO:tasks.workunit.client.0.smithi062.stdout:0/744: dwrite d7/fe6 [0,4194304] 0 2024-04-20T21:32:11.524 INFO:tasks.workunit.client.0.smithi062.stdout:4/847: rename da/dd/d15/d50/fcb to da/dd/d15/d50/d64/d57/d5b/d9d/dea/dac/f12d 0 2024-04-20T21:32:11.525 INFO:tasks.workunit.client.1.smithi189.stdout:6/495: rename d8/da/dd/d35/f97 to d8/d2b/d49/d54/d67/fab 0 2024-04-20T21:32:11.529 INFO:tasks.workunit.client.1.smithi189.stdout:8/535: read da/de/f16 [4301893,114878] 0 2024-04-20T21:32:11.531 INFO:tasks.workunit.client.1.smithi189.stdout:5/401: creat d7/d9/d31/d36/d5f/f75 x:0 0 0 2024-04-20T21:32:11.531 INFO:tasks.workunit.client.1.smithi189.stdout:6/496: stat d8/d12/d30/f58 0 2024-04-20T21:32:11.534 INFO:tasks.workunit.client.1.smithi189.stdout:7/487: creat d5/d46/fa9 x:0 0 0 2024-04-20T21:32:11.535 INFO:tasks.workunit.client.1.smithi189.stdout:8/536: rmdir da/df/d3c/d68 39 2024-04-20T21:32:11.536 INFO:tasks.workunit.client.1.smithi189.stdout:6/497: truncate d8/d12/d4e/f6f 180582 0 2024-04-20T21:32:11.539 INFO:tasks.workunit.client.1.smithi189.stdout:5/402: creat d7/d14/d18/f76 x:0 0 0 2024-04-20T21:32:11.543 INFO:tasks.workunit.client.1.smithi189.stdout:8/537: creat da/df/d13/d31/fbf x:0 0 0 2024-04-20T21:32:11.544 INFO:tasks.workunit.client.0.smithi062.stdout:5/766: mkdir d8/d38/d7a/d4c/d7d/d9d/dac/d2f/d8e/dd6/dfe 0 2024-04-20T21:32:11.548 INFO:tasks.workunit.client.1.smithi189.stdout:4/432: dread d2/d20/f4f [0,4194304] 0 2024-04-20T21:32:11.548 INFO:tasks.workunit.client.1.smithi189.stdout:5/403: creat d7/d14/d18/f77 x:0 0 0 2024-04-20T21:32:11.552 INFO:tasks.workunit.client.1.smithi189.stdout:4/433: fdatasync d2/dd/d11/d14/d69/f82 0 2024-04-20T21:32:11.556 INFO:tasks.workunit.client.0.smithi062.stdout:1/820: truncate dc/d15/d75/d96/dd4/f9c 898317 0 2024-04-20T21:32:11.557 INFO:tasks.workunit.client.0.smithi062.stdout:2/772: write d7/d39/f8e [3041007,25468] 0 2024-04-20T21:32:11.557 INFO:tasks.workunit.client.1.smithi189.stdout:8/538: rmdir da/df/d13/d63/db3 0 2024-04-20T21:32:11.559 INFO:tasks.workunit.client.0.smithi062.stdout:4/848: write f3 [5239105,56185] 0 2024-04-20T21:32:11.561 INFO:tasks.workunit.client.1.smithi189.stdout:6/498: dread d8/d12/d30/f38 [0,4194304] 0 2024-04-20T21:32:11.561 INFO:tasks.workunit.client.1.smithi189.stdout:5/404: creat d7/f78 x:0 0 0 2024-04-20T21:32:11.563 INFO:tasks.workunit.client.1.smithi189.stdout:5/405: chown d7/f5d 0 1 2024-04-20T21:32:11.564 INFO:tasks.workunit.client.1.smithi189.stdout:6/499: write d8/d10/d78/f96 [618707,98970] 0 2024-04-20T21:32:11.571 INFO:tasks.workunit.client.1.smithi189.stdout:5/406: symlink d7/d14/d4e/d5b/l79 0 2024-04-20T21:32:11.577 INFO:tasks.workunit.client.1.smithi189.stdout:8/539: link da/de/l48 da/df/d3c/d68/d8b/dac/lc0 0 2024-04-20T21:32:11.584 INFO:tasks.workunit.client.1.smithi189.stdout:3/420: dwrite d2/d8/f10 [4194304,4194304] 0 2024-04-20T21:32:11.584 INFO:tasks.workunit.client.1.smithi189.stdout:8/540: write da/df/d13/d63/f73 [2243813,90195] 0 2024-04-20T21:32:11.586 INFO:tasks.workunit.client.0.smithi062.stdout:4/849: dread da/dd/df/d27/f2c [0,4194304] 0 2024-04-20T21:32:11.588 INFO:tasks.workunit.client.1.smithi189.stdout:5/407: getdents d7/d9/d30/d5a 0 2024-04-20T21:32:11.588 INFO:tasks.workunit.client.0.smithi062.stdout:7/742: dwrite d7/d26/da8/db6/fd8 [0,4194304] 0 2024-04-20T21:32:11.589 INFO:tasks.workunit.client.0.smithi062.stdout:3/721: truncate d0/d1/d10/d31/d41/f7c 2074404 0 2024-04-20T21:32:11.592 INFO:tasks.workunit.client.1.smithi189.stdout:9/443: dread d9/f33 [0,4194304] 0 2024-04-20T21:32:11.593 INFO:tasks.workunit.client.0.smithi062.stdout:7/743: write d7/d3e/d33/d3d/dad/fb9 [1099712,84234] 0 2024-04-20T21:32:11.593 INFO:tasks.workunit.client.1.smithi189.stdout:1/399: getdents d8/d1e/d28 0 2024-04-20T21:32:11.595 INFO:tasks.workunit.client.1.smithi189.stdout:3/421: readlink d2/d3/d2c/d30/l72 0 2024-04-20T21:32:11.596 INFO:tasks.workunit.client.1.smithi189.stdout:5/408: chown d7/d9/d30/d55/d61 9834759 1 2024-04-20T21:32:11.597 INFO:tasks.workunit.client.1.smithi189.stdout:8/541: write da/de/d1e/d21/d44/f43 [1376535,73202] 0 2024-04-20T21:32:11.597 INFO:tasks.workunit.client.0.smithi062.stdout:3/722: dread d0/d1c/d63/d42/dc8/dc9/fcf [0,4194304] 0 2024-04-20T21:32:11.600 INFO:tasks.workunit.client.1.smithi189.stdout:9/444: mknod d9/d12/c8b 0 2024-04-20T21:32:11.605 INFO:tasks.workunit.client.1.smithi189.stdout:6/500: dwrite d8/da/fa3 [0,4194304] 0 2024-04-20T21:32:11.613 INFO:tasks.workunit.client.1.smithi189.stdout:3/422: mkdir d2/d3/d2c/d30/d3f/d9a 0 2024-04-20T21:32:11.614 INFO:tasks.workunit.client.0.smithi062.stdout:9/851: dread d16/f3a [0,4194304] 0 2024-04-20T21:32:11.616 INFO:tasks.workunit.client.0.smithi062.stdout:9/852: readlink d16/d2c/d42/da6/lcc 0 2024-04-20T21:32:11.617 INFO:tasks.workunit.client.1.smithi189.stdout:1/400: fsync d8/d1e/d28/d4f/f80 0 2024-04-20T21:32:11.619 INFO:tasks.workunit.client.1.smithi189.stdout:2/417: truncate d1/d11/d52/f6a 454819 0 2024-04-20T21:32:11.620 INFO:tasks.workunit.client.0.smithi062.stdout:8/778: truncate d7/d25/d76/f84 486660 0 2024-04-20T21:32:11.621 INFO:tasks.workunit.client.1.smithi189.stdout:8/542: read da/de/f10 [480777,59165] 0 2024-04-20T21:32:11.621 INFO:tasks.workunit.client.1.smithi189.stdout:0/562: dwrite d3/f48 [0,4194304] 0 2024-04-20T21:32:11.623 INFO:tasks.workunit.client.1.smithi189.stdout:9/445: rename d9/d12/c22 to d9/d12/d62/c8c 0 2024-04-20T21:32:11.623 INFO:tasks.workunit.client.1.smithi189.stdout:3/423: dread - d2/d3/d25/f95 zero size 2024-04-20T21:32:11.629 INFO:tasks.workunit.client.1.smithi189.stdout:7/488: truncate d5/d21/f32 4041318 0 2024-04-20T21:32:11.633 INFO:tasks.workunit.client.1.smithi189.stdout:4/434: dread d2/dd/f44 [0,4194304] 0 2024-04-20T21:32:11.634 INFO:tasks.workunit.client.1.smithi189.stdout:9/446: chown d9/dc/d17/l88 20 1 2024-04-20T21:32:11.636 INFO:tasks.workunit.client.1.smithi189.stdout:0/563: dread - d3/df/d30/d37/f65 zero size 2024-04-20T21:32:11.636 INFO:tasks.workunit.client.1.smithi189.stdout:8/543: creat da/df/d3c/d68/d8b/fc1 x:0 0 0 2024-04-20T21:32:11.642 INFO:tasks.workunit.client.1.smithi189.stdout:8/544: dread - da/df/d3c/d68/d8b/d8d/f90 zero size 2024-04-20T21:32:11.642 INFO:tasks.workunit.client.1.smithi189.stdout:6/501: link d8/d2b/f57 d8/da/dd/d35/d6d/d77/fac 0 2024-04-20T21:32:11.645 INFO:tasks.workunit.client.0.smithi062.stdout:0/745: truncate d7/d1c/d15/d17/d40/d79/d71/f94 673220 0 2024-04-20T21:32:11.647 INFO:tasks.workunit.client.0.smithi062.stdout:5/767: truncate d8/d38/d7a/d4c/d7d/d9d/dac/f39 109710 0 2024-04-20T21:32:11.647 INFO:tasks.workunit.client.1.smithi189.stdout:3/424: creat d2/d3/d2c/d30/d3f/d9a/f9b x:0 0 0 2024-04-20T21:32:11.650 INFO:tasks.workunit.client.0.smithi062.stdout:5/768: write d8/d10/d3e/f52 [4340088,42460] 0 2024-04-20T21:32:11.651 INFO:tasks.workunit.client.0.smithi062.stdout:7/744: dread - d7/d3e/d33/d3d/d4d/d54/d4f/fc2 zero size 2024-04-20T21:32:11.652 INFO:tasks.workunit.client.1.smithi189.stdout:4/435: mkdir d2/dd/d37/d9d 0 2024-04-20T21:32:11.654 INFO:tasks.workunit.client.0.smithi062.stdout:8/779: symlink d7/d25/d5c/lff 0 2024-04-20T21:32:11.656 INFO:tasks.workunit.client.1.smithi189.stdout:9/447: creat d9/dc/d17/d7e/f8d x:0 0 0 2024-04-20T21:32:11.658 INFO:tasks.workunit.client.0.smithi062.stdout:3/723: symlink d0/d1/d10/lf0 0 2024-04-20T21:32:11.659 INFO:tasks.workunit.client.0.smithi062.stdout:7/745: symlink d7/d3e/d33/d3d/d4d/d83/l101 0 2024-04-20T21:32:11.659 INFO:tasks.workunit.client.1.smithi189.stdout:0/564: rmdir d3/df/d30 39 2024-04-20T21:32:11.661 INFO:tasks.workunit.client.0.smithi062.stdout:3/724: readlink d0/d1/d10/d3a/d5e/l84 0 2024-04-20T21:32:11.662 INFO:tasks.workunit.client.1.smithi189.stdout:7/489: creat d5/d46/faa x:0 0 0 2024-04-20T21:32:11.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:11.664+0000 7f2caa600700 1 -- 172.21.15.62:0/73400775 --> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2ca4106c30 con 0x7f2c8806eb00 2024-04-20T21:32:11.664 INFO:tasks.workunit.client.1.smithi189.stdout:1/401: dwrite d8/f14 [0,4194304] 0 2024-04-20T21:32:11.664 INFO:tasks.workunit.client.1.smithi189.stdout:8/545: symlink da/df/d20/d27/lc2 0 2024-04-20T21:32:11.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:11.665+0000 7f2c9be00700 1 -- 172.21.15.62:0/73400775 <== mgr.14658 v2:172.21.15.62:6800/1585760929 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+329 (secure 0 0 0) 0x7f2ca4106c30 con 0x7f2c8806eb00 2024-04-20T21:32:11.665 INFO:tasks.workunit.client.0.smithi062.stdout:9/853: unlink d16/c5d 0 2024-04-20T21:32:11.665 INFO:tasks.workunit.client.1.smithi189.stdout:2/418: dwrite d1/d11/d52/d6f/f53 [0,4194304] 0 2024-04-20T21:32:11.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:11.668+0000 7f2caa600700 1 -- 172.21.15.62:0/73400775 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f2c8806eb00 msgr2=0x7f2c88070fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:32:11.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:11.668+0000 7f2caa600700 1 --2- 172.21.15.62:0/73400775 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f2c8806eb00 0x7f2c88070fb0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f2c9400b6f0 tx=0x7f2c94009f90 comp rx=0 tx=0).stop 2024-04-20T21:32:11.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:11.668+0000 7f2caa600700 1 -- 172.21.15.62:0/73400775 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2ca41003d0 msgr2=0x7f2ca4071fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:32:11.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:11.668+0000 7f2caa600700 1 --2- 172.21.15.62:0/73400775 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2ca41003d0 0x7f2ca4071fb0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f2c9000e970 tx=0x7f2c9000ec80 comp rx=0 tx=0).stop 2024-04-20T21:32:11.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:11.669+0000 7f2caa600700 1 -- 172.21.15.62:0/73400775 shutdown_connections 2024-04-20T21:32:11.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:11.669+0000 7f2caa600700 1 --2- 172.21.15.62:0/73400775 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f2c8806eb00 0x7f2c88070fb0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:32:11.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:11.669+0000 7f2caa600700 1 --2- 172.21.15.62:0/73400775 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ca40ff130 0x7f2ca4071a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:32:11.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:11.669+0000 7f2caa600700 1 --2- 172.21.15.62:0/73400775 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2ca41003d0 0x7f2ca4071fb0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:32:11.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:11.669+0000 7f2caa600700 1 -- 172.21.15.62:0/73400775 >> 172.21.15.62:0/73400775 conn(0x7f2ca40fa780 msgr2=0x7f2ca4105510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:32:11.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:11.669+0000 7f2caa600700 1 -- 172.21.15.62:0/73400775 shutdown_connections 2024-04-20T21:32:11.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:11.669+0000 7f2caa600700 1 -- 172.21.15.62:0/73400775 wait complete. 2024-04-20T21:32:11.671 INFO:tasks.workunit.client.0.smithi062.stdout:1/821: rename dc/d15/d20/d28/c8b to dc/d15/d75/d96/dd4/c12a 0 2024-04-20T21:32:11.671 INFO:tasks.workunit.client.1.smithi189.stdout:1/402: stat d8/d1e/d43/d58/f72 0 2024-04-20T21:32:11.671 INFO:tasks.workunit.client.1.smithi189.stdout:5/409: dread d7/d35/d48/f60 [0,4194304] 0 2024-04-20T21:32:11.671 INFO:tasks.workunit.client.1.smithi189.stdout:3/425: unlink d2/d3/d2c/d30/l3a 0 2024-04-20T21:32:11.673 INFO:tasks.workunit.client.1.smithi189.stdout:9/448: creat d9/dc/d17/f8e x:0 0 0 2024-04-20T21:32:11.674 INFO:tasks.workunit.client.0.smithi062.stdout:6/652: sync 2024-04-20T21:32:11.680 INFO:tasks.workunit.client.0.smithi062.stdout:8/780: creat d7/de1/f100 x:0 0 0 2024-04-20T21:32:11.681 INFO:tasks.workunit.client.1.smithi189.stdout:7/490: creat d5/d21/d4a/d52/d75/fab x:0 0 0 2024-04-20T21:32:11.689 INFO:tasks.workunit.client.0.smithi062.stdout:2/773: write d7/da/d15/f85 [3986992,22717] 0 2024-04-20T21:32:11.694 INFO:tasks.workunit.client.0.smithi062.stdout:1/822: symlink dc/d15/d75/d96/dd4/d3a/da2/de4/d84/dbe/l12b 0 2024-04-20T21:32:11.695 INFO:tasks.workunit.client.1.smithi189.stdout:6/502: dwrite d8/d2b/d49/d54/d67/f7a [0,4194304] 0 2024-04-20T21:32:11.696 INFO:tasks.workunit.client.1.smithi189.stdout:0/565: chown d3/df/d30/d37/c19 24495 1 2024-04-20T21:32:11.697 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T21:32:11.699 INFO:tasks.workunit.client.1.smithi189.stdout:6/503: chown d8/d2b/f3a 1402129495 1 2024-04-20T21:32:11.702 INFO:tasks.workunit.client.0.smithi062.stdout:2/774: creat d7/db5/fff x:0 0 0 2024-04-20T21:32:11.703 INFO:tasks.workunit.client.0.smithi062.stdout:6/653: dwrite d0/d52/da0/fcc [0,4194304] 0 2024-04-20T21:32:11.710 INFO:tasks.workunit.client.0.smithi062.stdout:0/746: rename d7/d1c/d15/d17/d26/c9e to d7/d1e/db6/cf4 0 2024-04-20T21:32:11.714 INFO:tasks.workunit.client.1.smithi189.stdout:0/566: rename d3/df/d5d/d64/f6d to d3/df/d30/d45/d90/fbf 0 2024-04-20T21:32:11.714 INFO:tasks.workunit.client.1.smithi189.stdout:4/436: dwrite d2/f54 [0,4194304] 0 2024-04-20T21:32:11.716 INFO:tasks.workunit.client.0.smithi062.stdout:1/823: creat dc/d15/d75/d96/dd4/d3a/da2/de4/d84/dbe/d10d/d110/f12c x:0 0 0 2024-04-20T21:32:11.719 INFO:tasks.workunit.client.1.smithi189.stdout:6/504: fdatasync d8/d2b/d49/d54/d67/fab 0 2024-04-20T21:32:11.720 INFO:tasks.workunit.client.1.smithi189.stdout:2/419: dwrite d1/d11/d58/d6b/f6c [0,4194304] 0 2024-04-20T21:32:11.722 INFO:tasks.workunit.client.1.smithi189.stdout:5/410: dwrite d7/d14/d4e/d5b/f68 [0,4194304] 0 2024-04-20T21:32:11.722 INFO:tasks.workunit.client.1.smithi189.stdout:3/426: creat d2/d3/d2c/d31/d90/d92/f9c x:0 0 0 2024-04-20T21:32:11.722 INFO:tasks.workunit.client.0.smithi062.stdout:4/850: rename da/dd/d15/d50/faa to da/dd/df/dfa/f12e 0 2024-04-20T21:32:11.722 INFO:tasks.workunit.client.1.smithi189.stdout:9/449: truncate d9/f34 3842765 0 2024-04-20T21:32:11.728 INFO:tasks.workunit.client.1.smithi189.stdout:8/546: link da/df/d13/f14 da/de/d1e/d69/da3/fc3 0 2024-04-20T21:32:11.730 INFO:tasks.workunit.client.1.smithi189.stdout:6/505: fdatasync d8/d10/d78/f80 0 2024-04-20T21:32:11.732 INFO:tasks.workunit.client.0.smithi062.stdout:3/725: write d0/d1/fd [2992983,85503] 0 2024-04-20T21:32:11.732 INFO:tasks.workunit.client.0.smithi062.stdout:9/854: truncate d16/d2c/d5e/d96/fb3 683973 0 2024-04-20T21:32:11.734 INFO:tasks.workunit.client.1.smithi189.stdout:7/491: dwrite d5/d21/d4a/d52/d75/fab [0,4194304] 0 2024-04-20T21:32:11.735 INFO:tasks.workunit.client.1.smithi189.stdout:4/437: creat d2/dd/d11/d14/d69/f9e x:0 0 0 2024-04-20T21:32:11.736 INFO:tasks.workunit.client.0.smithi062.stdout:6/654: dwrite d0/d9/fbd [4194304,4194304] 0 2024-04-20T21:32:11.741 INFO:tasks.workunit.client.1.smithi189.stdout:9/450: rmdir d9/dc/d69 39 2024-04-20T21:32:11.742 INFO:tasks.workunit.client.1.smithi189.stdout:6/506: write d8/d12/d30/d47/f98 [553941,46476] 0 2024-04-20T21:32:11.742 INFO:tasks.workunit.client.1.smithi189.stdout:3/427: readlink d2/d3/d2c/d31/l60 0 2024-04-20T21:32:11.742 INFO:tasks.workunit.client.0.smithi062.stdout:1/824: creat dc/d15/d75/d96/dd4/d3a/da2/de4/d84/dbe/d10d/f12d x:0 0 0 2024-04-20T21:32:11.743 INFO:tasks.workunit.client.0.smithi062.stdout:5/769: rename d8/d1e/d2c/d6c/f65 to d8/d10/d3e/fff 0 2024-04-20T21:32:11.751 INFO:tasks.workunit.client.1.smithi189.stdout:9/451: fdatasync d9/d12/d3f/d72/f85 0 2024-04-20T21:32:11.752 INFO:tasks.workunit.client.0.smithi062.stdout:7/746: dwrite d7/d3e/d33/d3d/d4d/d54/f2e [0,4194304] 0 2024-04-20T21:32:11.754 INFO:tasks.workunit.client.1.smithi189.stdout:3/428: truncate d2/d3/d2c/f6b 901231 0 2024-04-20T21:32:11.757 INFO:tasks.workunit.client.1.smithi189.stdout:2/420: rmdir d1/d11/d2d/d7a 0 2024-04-20T21:32:11.758 INFO:tasks.workunit.client.0.smithi062.stdout:1/825: creat dc/d15/d75/f12e x:0 0 0 2024-04-20T21:32:11.758 INFO:tasks.workunit.client.0.smithi062.stdout:0/747: dwrite d7/d1c/d15/f22 [0,4194304] 0 2024-04-20T21:32:11.758 INFO:tasks.workunit.client.1.smithi189.stdout:4/438: creat d2/d22/d57/f9f x:0 0 0 2024-04-20T21:32:11.759 INFO:tasks.workunit.client.0.smithi062.stdout:5/770: readlink d8/d38/d7a/l75 0 2024-04-20T21:32:11.759 INFO:tasks.workunit.client.1.smithi189.stdout:6/507: creat d8/d12/d30/d47/fad x:0 0 0 2024-04-20T21:32:11.759 INFO:tasks.workunit.client.1.smithi189.stdout:5/411: link d7/d14/d18/f17 d7/d35/f7a 0 2024-04-20T21:32:11.761 INFO:tasks.workunit.client.0.smithi062.stdout:8/781: dread d7/fad [0,4194304] 0 2024-04-20T21:32:11.765 INFO:tasks.workunit.client.1.smithi189.stdout:9/452: creat d9/d12/d15/f8f x:0 0 0 2024-04-20T21:32:11.765 INFO:tasks.workunit.client.1.smithi189.stdout:5/412: write d7/d9/d28/f58 [678190,50398] 0 2024-04-20T21:32:11.773 INFO:tasks.workunit.client.0.smithi062.stdout:9/855: rmdir d16/d2c/d5e/d75/d76/dd9/d111 0 2024-04-20T21:32:11.773 INFO:tasks.workunit.client.0.smithi062.stdout:1/826: write dc/d15/d20/d28/d83/dc0/f124 [68173,104847] 0 2024-04-20T21:32:11.777 INFO:tasks.workunit.client.1.smithi189.stdout:5/413: creat d7/d9/d28/f7b x:0 0 0 2024-04-20T21:32:11.780 INFO:tasks.workunit.client.0.smithi062.stdout:9/856: stat d16/d3e/d77/d84/l97 0 2024-04-20T21:32:11.786 INFO:tasks.workunit.client.0.smithi062.stdout:9/857: symlink d16/d2c/d42/l116 0 2024-04-20T21:32:11.791 INFO:tasks.workunit.client.0.smithi062.stdout:0/748: dread d7/d1c/d15/f2d [0,4194304] 0 2024-04-20T21:32:11.792 INFO:tasks.workunit.client.0.smithi062.stdout:4/851: dwrite da/dd/df/d27/d32/ff9 [0,4194304] 0 2024-04-20T21:32:11.795 INFO:tasks.workunit.client.0.smithi062.stdout:6/655: rename d0/d1a/d44/fa3 to d0/d9/d11/fda 0 2024-04-20T21:32:11.799 INFO:tasks.workunit.client.0.smithi062.stdout:4/852: stat da/dd/df/f41 0 2024-04-20T21:32:11.800 INFO:tasks.workunit.client.1.smithi189.stdout:6/508: dwrite d8/d12/d30/f38 [0,4194304] 0 2024-04-20T21:32:11.800 INFO:tasks.workunit.client.0.smithi062.stdout:5/771: dwrite d8/d38/d7a/d4c/d7d/d9d/dac/d2f/f4b [0,4194304] 0 2024-04-20T21:32:11.800 INFO:tasks.workunit.client.0.smithi062.stdout:9/858: creat d16/d3d/f117 x:0 0 0 2024-04-20T21:32:11.805 INFO:tasks.workunit.client.1.smithi189.stdout:5/414: dread d7/d35/d48/f60 [0,4194304] 0 2024-04-20T21:32:11.805 INFO:tasks.workunit.client.0.smithi062.stdout:9/859: fsync d16/d3e/fd5 0 2024-04-20T21:32:11.816 INFO:tasks.workunit.client.1.smithi189.stdout:9/453: dwrite d9/dc/d17/d2b/d52/f83 [0,4194304] 0 2024-04-20T21:32:11.816 INFO:tasks.workunit.client.1.smithi189.stdout:0/567: dread d3/df/d28/d2b/f49 [0,4194304] 0 2024-04-20T21:32:11.817 INFO:tasks.workunit.client.1.smithi189.stdout:1/403: dread d8/d21/d2c/d36/f52 [0,4194304] 0 2024-04-20T21:32:11.817 INFO:tasks.workunit.client.0.smithi062.stdout:6/656: creat d0/d9/d11/d22/dd5/d67/d75/dd2/fdb x:0 0 0 2024-04-20T21:32:11.819 INFO:tasks.workunit.client.1.smithi189.stdout:5/415: rename d7/d9/d30/c29 to d7/d9/c7c 0 2024-04-20T21:32:11.823 INFO:tasks.workunit.client.1.smithi189.stdout:9/454: write d9/dc/f2c [1209580,60679] 0 2024-04-20T21:32:11.824 INFO:tasks.workunit.client.0.smithi062.stdout:7/747: rename d7/d3e/d33/d3d/d4d/d83/f81 to d7/d3e/d33/d3d/d5e/f102 0 2024-04-20T21:32:11.825 INFO:tasks.workunit.client.1.smithi189.stdout:0/568: chown d3/df/d28/d2b/f58 4096776 1 2024-04-20T21:32:11.826 INFO:tasks.workunit.client.1.smithi189.stdout:8/547: write da/d1f/d2a/f58 [408207,119660] 0 2024-04-20T21:32:11.831 INFO:tasks.workunit.client.1.smithi189.stdout:5/416: creat d7/d35/d48/f7d x:0 0 0 2024-04-20T21:32:11.831 INFO:tasks.workunit.client.1.smithi189.stdout:0/569: write d3/df/d28/d2b/f67 [1161831,129022] 0 2024-04-20T21:32:11.836 INFO:tasks.workunit.client.0.smithi062.stdout:7/748: rmdir d7/d3e/d33 39 2024-04-20T21:32:11.846 INFO:tasks.workunit.client.1.smithi189.stdout:6/509: getdents d8/d10/d78/d7e 0 2024-04-20T21:32:11.850 INFO:tasks.workunit.client.0.smithi062.stdout:2/775: dwrite d7/da/d15/d2f/d93/dac/dca/ff8 [0,4194304] 0 2024-04-20T21:32:11.850 INFO:tasks.workunit.client.1.smithi189.stdout:2/421: truncate d1/d11/f28 324114 0 2024-04-20T21:32:11.851 INFO:tasks.workunit.client.1.smithi189.stdout:8/548: write da/df/d13/d31/f80 [886260,5180] 0 2024-04-20T21:32:11.851 INFO:tasks.workunit.client.1.smithi189.stdout:0/570: read d3/d8c/f8d [1661907,103739] 0 2024-04-20T21:32:11.860 INFO:tasks.workunit.client.1.smithi189.stdout:3/429: dwrite d2/d8/f45 [0,4194304] 0 2024-04-20T21:32:11.862 INFO:tasks.workunit.client.1.smithi189.stdout:1/404: truncate d8/f24 501118 0 2024-04-20T21:32:11.863 INFO:tasks.workunit.client.0.smithi062.stdout:8/782: dwrite d7/d30/f5d [0,4194304] 0 2024-04-20T21:32:11.866 INFO:tasks.workunit.client.1.smithi189.stdout:9/455: dwrite d9/dc/d17/d2b/d52/f75 [0,4194304] 0 2024-04-20T21:32:11.867 INFO:tasks.workunit.client.1.smithi189.stdout:1/405: fdatasync d8/d21/d2c/d36/f82 0 2024-04-20T21:32:11.869 INFO:tasks.workunit.client.1.smithi189.stdout:1/406: chown d8/c35 1283 1 2024-04-20T21:32:11.870 INFO:tasks.workunit.client.0.smithi062.stdout:2/776: creat d7/da/d15/d1b/d3d/d84/f100 x:0 0 0 2024-04-20T21:32:11.870 INFO:tasks.workunit.client.1.smithi189.stdout:9/456: dread - d9/d12/f89 zero size 2024-04-20T21:32:11.871 INFO:tasks.workunit.client.1.smithi189.stdout:2/422: creat d1/d17/f7c x:0 0 0 2024-04-20T21:32:11.874 INFO:tasks.workunit.client.0.smithi062.stdout:4/853: dwrite f4 [0,4194304] 0 2024-04-20T21:32:11.874 INFO:tasks.workunit.client.1.smithi189.stdout:1/407: chown d8/d1e/f66 1578 1 2024-04-20T21:32:11.876 INFO:tasks.workunit.client.1.smithi189.stdout:0/571: creat d3/d26/d88/fc0 x:0 0 0 2024-04-20T21:32:11.876 INFO:tasks.workunit.client.1.smithi189.stdout:5/417: dread d7/d14/d4e/d5b/f68 [0,4194304] 0 2024-04-20T21:32:11.876 INFO:tasks.workunit.client.0.smithi062.stdout:8/783: rmdir d7/d25/d3b/d58/d8b/d24/d46 39 2024-04-20T21:32:11.878 INFO:tasks.workunit.client.1.smithi189.stdout:6/510: creat d8/d12/fae x:0 0 0 2024-04-20T21:32:11.881 INFO:tasks.workunit.client.0.smithi062.stdout:5/772: dwrite d8/d38/d7a/d4c/d7d/d9d/dac/d66/dae/f64 [0,4194304] 0 2024-04-20T21:32:11.881 INFO:tasks.workunit.client.1.smithi189.stdout:2/423: chown d1/d11/d52/c59 32628 1 2024-04-20T21:32:11.882 INFO:tasks.workunit.client.1.smithi189.stdout:1/408: creat d8/d1e/f87 x:0 0 0 2024-04-20T21:32:11.886 INFO:tasks.workunit.client.1.smithi189.stdout:8/549: getdents da 0 2024-04-20T21:32:11.886 INFO:tasks.workunit.client.0.smithi062.stdout:9/860: dwrite d16/d2c/d42/da6/da9/fc4 [0,4194304] 0 2024-04-20T21:32:11.887 INFO:tasks.workunit.client.1.smithi189.stdout:5/418: fsync d7/d14/d4e/d5b/f68 0 2024-04-20T21:32:11.888 INFO:tasks.workunit.client.1.smithi189.stdout:6/511: chown d8/d2b/d49/d54/d67/c9f 0 1 2024-04-20T21:32:11.889 INFO:tasks.workunit.client.1.smithi189.stdout:0/572: dread - d3/df/d4c/f84 zero size 2024-04-20T21:32:11.890 INFO:tasks.workunit.client.1.smithi189.stdout:1/409: creat d8/df/d50/f88 x:0 0 0 2024-04-20T21:32:11.890 INFO:tasks.workunit.client.0.smithi062.stdout:0/749: dread d7/d1c/d15/d17/f72 [0,4194304] 0 2024-04-20T21:32:11.890 INFO:tasks.workunit.client.0.smithi062.stdout:2/777: truncate d7/da/f3c 2612678 0 2024-04-20T21:32:11.891 INFO:tasks.workunit.client.1.smithi189.stdout:3/430: dwrite d2/d1c/f2b [4194304,4194304] 0 2024-04-20T21:32:11.892 INFO:tasks.workunit.client.0.smithi062.stdout:8/784: unlink d7/d9/lca 0 2024-04-20T21:32:11.894 INFO:tasks.workunit.client.0.smithi062.stdout:7/749: dwrite d7/d3e/d33/d3d/d4d/fc0 [0,4194304] 0 2024-04-20T21:32:11.897 INFO:tasks.workunit.client.1.smithi189.stdout:5/419: dread - d7/d14/d18/f77 zero size 2024-04-20T21:32:11.898 INFO:tasks.workunit.client.0.smithi062.stdout:0/750: write d7/d1c/d15/d17/d37/f84 [3360907,4396] 0 2024-04-20T21:32:11.899 INFO:tasks.workunit.client.1.smithi189.stdout:5/420: readlink d7/d9/d30/l32 0 2024-04-20T21:32:11.901 INFO:tasks.workunit.client.0.smithi062.stdout:2/778: mknod d7/da/d15/d1b/d24/c101 0 2024-04-20T21:32:11.903 INFO:tasks.workunit.client.1.smithi189.stdout:5/421: write d7/f5d [620622,48202] 0 2024-04-20T21:32:11.906 INFO:tasks.workunit.client.0.smithi062.stdout:9/861: symlink d16/d3e/d77/l118 0 2024-04-20T21:32:11.906 INFO:tasks.workunit.client.0.smithi062.stdout:0/751: dread - d7/d1e/da9/fcb zero size 2024-04-20T21:32:11.906 INFO:tasks.workunit.client.1.smithi189.stdout:1/410: link d8/f41 d8/d1e/f89 0 2024-04-20T21:32:11.907 INFO:tasks.workunit.client.0.smithi062.stdout:8/785: mkdir d7/d25/d70/d101 0 2024-04-20T21:32:11.908 INFO:tasks.workunit.client.1.smithi189.stdout:5/422: mknod d7/d14/d18/c7e 0 2024-04-20T21:32:11.910 INFO:tasks.workunit.client.0.smithi062.stdout:2/779: chown d7/da/d15/d1b/d3d/de5 282722 1 2024-04-20T21:32:11.912 INFO:tasks.workunit.client.0.smithi062.stdout:9/862: write d16/dbe/fca [954088,59540] 0 2024-04-20T21:32:11.912 INFO:tasks.workunit.client.0.smithi062.stdout:5/773: read d8/d10/d3e/ff6 [211510,32834] 0 2024-04-20T21:32:11.912 INFO:tasks.workunit.client.0.smithi062.stdout:2/780: fdatasync d7/d39/f8e 0 2024-04-20T21:32:11.914 INFO:tasks.workunit.client.0.smithi062.stdout:0/752: stat d7/d1c/d15/d17/d1b/l4a 0 2024-04-20T21:32:11.917 INFO:tasks.workunit.client.0.smithi062.stdout:6/657: dread d0/f80 [4194304,4194304] 0 2024-04-20T21:32:11.918 INFO:tasks.workunit.client.0.smithi062.stdout:8/786: rename d7/fa8 to d7/d25/d70/f102 0 2024-04-20T21:32:11.920 INFO:tasks.workunit.client.0.smithi062.stdout:0/753: chown d7/d1c/d15/d17/d26 5128 1 2024-04-20T21:32:11.921 INFO:tasks.workunit.client.1.smithi189.stdout:5/423: mknod d7/d9/d31/d36/c7f 0 2024-04-20T21:32:11.922 INFO:tasks.workunit.client.0.smithi062.stdout:5/774: chown d8/d38/d7a/d4c/d7d/d9d/dac/d66/dae/le1 28966 1 2024-04-20T21:32:11.923 INFO:tasks.workunit.client.1.smithi189.stdout:5/424: dread - d7/d9/f62 zero size 2024-04-20T21:32:11.923 INFO:tasks.workunit.client.0.smithi062.stdout:9/863: write d16/d3d/ff6 [105359,71160] 0 2024-04-20T21:32:11.927 INFO:tasks.workunit.client.0.smithi062.stdout:2/781: truncate d7/f56 2304266 0 2024-04-20T21:32:11.933 INFO:tasks.workunit.client.1.smithi189.stdout:5/425: rename d7/d14/c5e to d7/d9/c80 0 2024-04-20T21:32:11.936 INFO:tasks.workunit.client.0.smithi062.stdout:9/864: truncate d16/d3e/f5f 610422 0 2024-04-20T21:32:11.936 INFO:tasks.workunit.client.0.smithi062.stdout:7/750: read d7/d3e/d33/d3d/d4d/d54/d4f/f68 [2586836,19906] 0 2024-04-20T21:32:11.937 INFO:tasks.workunit.client.1.smithi189.stdout:6/512: rmdir d8/d12 39 2024-04-20T21:32:11.937 INFO:tasks.workunit.client.1.smithi189.stdout:2/424: dread d1/f9 [8388608,4194304] 0 2024-04-20T21:32:11.941 INFO:tasks.workunit.client.1.smithi189.stdout:6/513: dread - d8/d2b/d49/d54/d67/f9c zero size 2024-04-20T21:32:11.942 INFO:tasks.workunit.client.0.smithi062.stdout:9/865: rmdir d16/d2f/d5b/dab 39 2024-04-20T21:32:11.945 INFO:tasks.workunit.client.0.smithi062.stdout:2/782: rename d7/da/d28/d4b/f73 to d7/da/d15/d2f/d38/db0/dc4/f102 0 2024-04-20T21:32:11.946 INFO:tasks.workunit.client.1.smithi189.stdout:6/514: truncate d8/d2b/d49/d54/d67/fab 949915 0 2024-04-20T21:32:11.951 INFO:tasks.workunit.client.1.smithi189.stdout:6/515: write d8/d2b/f74 [31918,93807] 0 2024-04-20T21:32:11.951 INFO:tasks.workunit.client.1.smithi189.stdout:3/431: dread d2/fd [0,4194304] 0 2024-04-20T21:32:11.953 INFO:tasks.workunit.client.1.smithi189.stdout:0/573: write d3/df/d30/d45/d60/f68 [2064134,35201] 0 2024-04-20T21:32:11.953 INFO:tasks.workunit.client.1.smithi189.stdout:3/432: chown d2/d3/d2c/d30/d3f/c94 259015 1 2024-04-20T21:32:11.953 INFO:tasks.workunit.client.1.smithi189.stdout:6/516: write d8/da/dd/d2a/d3b/f8e [1843212,91276] 0 2024-04-20T21:32:11.954 INFO:tasks.workunit.client.0.smithi062.stdout:7/751: creat d7/d3e/d33/d3d/d4d/d54/f103 x:0 0 0 2024-04-20T21:32:11.956 INFO:tasks.workunit.client.1.smithi189.stdout:3/433: stat d2/d3/d2c/d31/d90 0 2024-04-20T21:32:11.957 INFO:tasks.workunit.client.1.smithi189.stdout:5/426: rename d7/d35/d48/d71 to d7/d9/d30/d55/d81 0 2024-04-20T21:32:11.960 INFO:tasks.workunit.client.1.smithi189.stdout:5/427: fdatasync d7/d9/d31/d36/d5f/f75 0 2024-04-20T21:32:11.960 INFO:tasks.workunit.client.0.smithi062.stdout:7/752: fsync d7/d3e/f45 0 2024-04-20T21:32:11.962 INFO:tasks.workunit.client.1.smithi189.stdout:1/411: write d8/d1e/f23 [1156043,54578] 0 2024-04-20T21:32:11.965 INFO:tasks.workunit.client.0.smithi062.stdout:2/783: link d7/da/d15/d2f/d97/lae d7/da/d15/d2f/d38/d51/dbc/l103 0 2024-04-20T21:32:11.965 INFO:tasks.workunit.client.1.smithi189.stdout:1/412: fdatasync d8/f1a 0 2024-04-20T21:32:11.965 INFO:tasks.workunit.client.1.smithi189.stdout:6/517: write d8/d12/d30/d47/f8a [461385,11576] 0 2024-04-20T21:32:11.966 INFO:tasks.workunit.client.1.smithi189.stdout:0/574: symlink d3/df/d28/d2b/lc1 0 2024-04-20T21:32:11.967 INFO:tasks.workunit.client.1.smithi189.stdout:0/575: chown d3/db9 3 1 2024-04-20T21:32:11.969 INFO:tasks.workunit.client.0.smithi062.stdout:7/753: truncate d7/d3e/d33/f76 1174129 0 2024-04-20T21:32:11.971 INFO:tasks.workunit.client.1.smithi189.stdout:5/428: creat d7/d9/d30/d55/f82 x:0 0 0 2024-04-20T21:32:11.972 INFO:tasks.workunit.client.1.smithi189.stdout:3/434: truncate d2/d3/d2c/f6d 1116631 0 2024-04-20T21:32:11.975 INFO:tasks.workunit.client.0.smithi062.stdout:2/784: mknod d7/da/d15/d2f/d93/dac/dca/dd8/c104 0 2024-04-20T21:32:11.975 INFO:tasks.workunit.client.0.smithi062.stdout:9/866: dwrite d16/d2f/d5b/dab/fb2 [0,4194304] 0 2024-04-20T21:32:11.975 INFO:tasks.workunit.client.1.smithi189.stdout:1/413: rmdir d8/d1e/d28/d2b 39 2024-04-20T21:32:11.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:11.973+0000 7f3a70c00700 1 -- 172.21.15.62:0/271102500 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3a6c100290 msgr2=0x7f3a6c1006a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:32:11.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:11.973+0000 7f3a70c00700 1 --2- 172.21.15.62:0/271102500 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3a6c100290 0x7f3a6c1006a0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f3a5c00b0a0 tx=0x7f3a5c00b3b0 comp rx=0 tx=0).stop 2024-04-20T21:32:11.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:11.973+0000 7f3a70c00700 1 -- 172.21.15.62:0/271102500 shutdown_connections 2024-04-20T21:32:11.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:11.973+0000 7f3a70c00700 1 --2- 172.21.15.62:0/271102500 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a6c101490 0x7f3a6c1018e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:32:11.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:11.973+0000 7f3a70c00700 1 --2- 172.21.15.62:0/271102500 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3a6c100290 0x7f3a6c1006a0 secure :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f3a5c00b0a0 tx=0x7f3a5c00b3b0 comp rx=0 tx=0).stop 2024-04-20T21:32:11.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:11.973+0000 7f3a70c00700 1 -- 172.21.15.62:0/271102500 >> 172.21.15.62:0/271102500 conn(0x7f3a6c0fb860 msgr2=0x7f3a6c0fdc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:32:11.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:11.976+0000 7f3a70c00700 1 -- 172.21.15.62:0/271102500 shutdown_connections 2024-04-20T21:32:11.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:11.976+0000 7f3a70c00700 1 -- 172.21.15.62:0/271102500 wait complete. 2024-04-20T21:32:11.977 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:11 smithi062 ceph-mon[16221]: pgmap v20: 65 pgs: 65 active+clean; 2.3 GiB data, 6.7 GiB used, 530 GiB / 536 GiB avail; 62 MiB/s rd, 213 MiB/s wr, 386 op/s 2024-04-20T21:32:11.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:11.978+0000 7f3a70c00700 1 Processor -- start 2024-04-20T21:32:11.979 INFO:tasks.workunit.client.0.smithi062.stdout:2/785: fsync d7/da/d15/d2f/d97/f9e 0 2024-04-20T21:32:11.980 INFO:tasks.workunit.client.0.smithi062.stdout:9/867: stat d16/d3e/d77/d84 0 2024-04-20T21:32:11.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:11.979+0000 7f3a70c00700 1 -- start start 2024-04-20T21:32:11.982 INFO:tasks.workunit.client.1.smithi189.stdout:5/429: unlink d7/d14/d18/f76 0 2024-04-20T21:32:11.982 INFO:tasks.workunit.client.1.smithi189.stdout:2/425: truncate d1/f9 1832506 0 2024-04-20T21:32:11.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:11.982+0000 7f3a70c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3a6c101490 0x7f3a6c19c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:32:11.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:11.982+0000 7f3a70c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a6c19c9c0 0x7f3a6c1a1a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:32:11.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:11.982+0000 7f3a70c00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a6c19cec0 con 0x7f3a6c101490 2024-04-20T21:32:11.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:11.982+0000 7f3a70c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a6c19d030 con 0x7f3a6c19c9c0 2024-04-20T21:32:11.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:11.983+0000 7f3a6aa00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a6c19c9c0 0x7f3a6c1a1a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:32:11.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:11.983+0000 7f3a6aa00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a6c19c9c0 0x7f3a6c1a1a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.62:43478/0 (socket says 172.21.15.62:43478) 2024-04-20T21:32:11.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:11.983+0000 7f3a6aa00700 1 -- 172.21.15.62:0/2801018281 learned_addr learned my addr 172.21.15.62:0/2801018281 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:32:11.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:11.983+0000 7f3a6b400700 1 --2- 172.21.15.62:0/2801018281 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3a6c101490 0x7f3a6c19c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:32:11.984 INFO:tasks.workunit.client.1.smithi189.stdout:5/430: read d7/d14/f4c [2286737,23686] 0 2024-04-20T21:32:11.984 INFO:tasks.workunit.client.1.smithi189.stdout:6/518: mkdir d8/d12/daf 0 2024-04-20T21:32:11.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:11.983+0000 7f3a6b400700 1 -- 172.21.15.62:0/2801018281 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a6c19c9c0 msgr2=0x7f3a6c1a1a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:32:11.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:11.983+0000 7f3a6b400700 1 --2- 172.21.15.62:0/2801018281 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a6c19c9c0 0x7f3a6c1a1a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:32:11.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:11.984+0000 7f3a6b400700 1 -- 172.21.15.62:0/2801018281 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a5c009d00 con 0x7f3a6c101490 2024-04-20T21:32:11.985 INFO:tasks.workunit.client.1.smithi189.stdout:5/431: chown d7/d9/d30/l32 1 1 2024-04-20T21:32:11.985 INFO:tasks.workunit.client.1.smithi189.stdout:6/519: read d8/d2b/f39 [254856,125463] 0 2024-04-20T21:32:11.986 INFO:tasks.workunit.client.1.smithi189.stdout:5/432: chown d7/d35/d48 145 1 2024-04-20T21:32:11.986 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:11 smithi189 ceph-mon[21562]: pgmap v20: 65 pgs: 65 active+clean; 2.3 GiB data, 6.7 GiB used, 530 GiB / 536 GiB avail; 62 MiB/s rd, 213 MiB/s wr, 386 op/s 2024-04-20T21:32:11.986 INFO:tasks.workunit.client.1.smithi189.stdout:6/520: chown d8/da/lc 591930 1 2024-04-20T21:32:11.988 INFO:tasks.workunit.client.0.smithi062.stdout:7/754: read d7/f18 [5764902,84512] 0 2024-04-20T21:32:11.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:11.988+0000 7f3a6b400700 1 --2- 172.21.15.62:0/2801018281 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3a6c101490 0x7f3a6c19c480 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c0ffde0 tx=0x7f3a5c003c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:32:11.989 INFO:tasks.workunit.client.0.smithi062.stdout:9/868: mkdir d16/d2c/d42/d100/d119 0 2024-04-20T21:32:11.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:11.988+0000 7f3a63e00700 1 -- 172.21.15.62:0/2801018281 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3a5c021070 con 0x7f3a6c101490 2024-04-20T21:32:11.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:11.990+0000 7f3a63e00700 1 -- 172.21.15.62:0/2801018281 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f3a5c0075c0 con 0x7f3a6c101490 2024-04-20T21:32:11.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:11.990+0000 7f3a63e00700 1 -- 172.21.15.62:0/2801018281 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3a5c00e040 con 0x7f3a6c101490 2024-04-20T21:32:11.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:11.991+0000 7f3a70c00700 1 -- 172.21.15.62:0/2801018281 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a6c1a1f70 con 0x7f3a6c101490 2024-04-20T21:32:11.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:11.991+0000 7f3a70c00700 1 -- 172.21.15.62:0/2801018281 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a6c1a2400 con 0x7f3a6c101490 2024-04-20T21:32:11.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:11.994+0000 7f3a70c00700 1 -- 172.21.15.62:0/2801018281 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a6c06a550 con 0x7f3a6c101490 2024-04-20T21:32:11.995 INFO:tasks.workunit.client.1.smithi189.stdout:0/576: rename d3/df/d30/d37/l20 to d3/df/d28/d2b/d2c/d73/lc2 0 2024-04-20T21:32:11.995 INFO:tasks.workunit.client.1.smithi189.stdout:1/414: dread d8/d1e/f6e [0,4194304] 0 2024-04-20T21:32:11.998 INFO:tasks.workunit.client.1.smithi189.stdout:7/492: sync 2024-04-20T21:32:11.998 INFO:tasks.workunit.client.1.smithi189.stdout:4/439: sync 2024-04-20T21:32:12.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:11.999+0000 7f3a63e00700 1 -- 172.21.15.62:0/2801018281 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 29) v1 ==== 92700+0+0 (secure 0 0 0) 0x7f3a5c019070 con 0x7f3a6c101490 2024-04-20T21:32:12.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:12.001+0000 7f3a63e00700 1 --2- 172.21.15.62:0/2801018281 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f3a5006ebd0 0x7f3a50071080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:32:12.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:12.001+0000 7f3a63e00700 1 -- 172.21.15.62:0/2801018281 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f3a5c08fdd0 con 0x7f3a6c101490 2024-04-20T21:32:12.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:12.001+0000 7f3a6aa00700 1 --2- 172.21.15.62:0/2801018281 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f3a5006ebd0 0x7f3a50071080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:32:12.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:12.003+0000 7f3a6aa00700 1 --2- 172.21.15.62:0/2801018281 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f3a5006ebd0 0x7f3a50071080 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f3a54009ad0 tx=0x7f3a54009290 comp rx=0 tx=0).ready entity=mgr.14658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:32:12.004 INFO:tasks.workunit.client.0.smithi062.stdout:5/775: dread d8/dbd/fdc [0,4194304] 0 2024-04-20T21:32:12.007 INFO:tasks.workunit.client.1.smithi189.stdout:1/415: fdatasync d8/d21/d2c/d36/f52 0 2024-04-20T21:32:12.007 INFO:tasks.workunit.client.1.smithi189.stdout:0/577: fdatasync d3/df/d28/d2b/f49 0 2024-04-20T21:32:12.008 INFO:tasks.workunit.client.1.smithi189.stdout:4/440: creat d2/d7f/fa0 x:0 0 0 2024-04-20T21:32:12.013 INFO:tasks.workunit.client.1.smithi189.stdout:7/493: symlink d5/d21/d25/d34/lac 0 2024-04-20T21:32:12.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:12.014+0000 7f3a63e00700 1 -- 172.21.15.62:0/2801018281 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187527 (secure 0 0 0) 0x7f3a5c05ca20 con 0x7f3a6c101490 2024-04-20T21:32:12.014 INFO:tasks.workunit.client.0.smithi062.stdout:8/787: dread d7/d25/d3b/d58/d8b/d24/f48 [0,4194304] 0 2024-04-20T21:32:12.015 INFO:tasks.workunit.client.1.smithi189.stdout:5/433: dwrite d7/d9/d31/d36/f38 [0,4194304] 0 2024-04-20T21:32:12.016 INFO:tasks.workunit.client.1.smithi189.stdout:6/521: dread d8/d10/f2d [0,4194304] 0 2024-04-20T21:32:12.016 INFO:tasks.workunit.client.1.smithi189.stdout:4/441: mknod d2/d20/ca1 0 2024-04-20T21:32:12.017 INFO:tasks.workunit.client.0.smithi062.stdout:5/776: creat d8/d38/d7a/df0/f100 x:0 0 0 2024-04-20T21:32:12.022 INFO:tasks.workunit.client.1.smithi189.stdout:0/578: symlink d3/df/d28/lc3 0 2024-04-20T21:32:12.024 INFO:tasks.workunit.client.1.smithi189.stdout:5/434: chown d7/d9/d28/f6f 7586345 1 2024-04-20T21:32:12.025 INFO:tasks.workunit.client.1.smithi189.stdout:9/457: dread d9/d12/d15/d1a/f5c [0,4194304] 0 2024-04-20T21:32:12.026 INFO:tasks.workunit.client.0.smithi062.stdout:0/754: write d7/d1c/ff [966413,38656] 0 2024-04-20T21:32:12.026 INFO:tasks.workunit.client.0.smithi062.stdout:8/788: mknod d7/d9/d29/c103 0 2024-04-20T21:32:12.027 INFO:tasks.workunit.client.0.smithi062.stdout:4/854: dwrite da/dd/df/d27/d3f/fdf [0,4194304] 0 2024-04-20T21:32:12.030 INFO:tasks.workunit.client.1.smithi189.stdout:8/550: dread da/de/d1e/d69/da3/fc3 [0,4194304] 0 2024-04-20T21:32:12.032 INFO:tasks.workunit.client.0.smithi062.stdout:9/869: dread f4 [0,4194304] 0 2024-04-20T21:32:12.032 INFO:tasks.workunit.client.0.smithi062.stdout:5/777: fdatasync d8/d10/d3e/f62 0 2024-04-20T21:32:12.039 INFO:tasks.workunit.client.1.smithi189.stdout:3/435: dread d2/d3/d2c/f6d [0,4194304] 0 2024-04-20T21:32:12.041 INFO:tasks.workunit.client.1.smithi189.stdout:2/426: dwrite d1/d11/f1a [0,4194304] 0 2024-04-20T21:32:12.041 INFO:tasks.workunit.client.1.smithi189.stdout:1/416: dwrite d8/f40 [0,4194304] 0 2024-04-20T21:32:12.041 INFO:tasks.workunit.client.1.smithi189.stdout:2/427: chown d1/d17/d78 4 1 2024-04-20T21:32:12.044 INFO:tasks.workunit.client.1.smithi189.stdout:4/442: creat d2/d22/d87/fa2 x:0 0 0 2024-04-20T21:32:12.048 INFO:tasks.workunit.client.0.smithi062.stdout:9/870: read d16/d3e/d77/d84/f99 [668177,110092] 0 2024-04-20T21:32:12.049 INFO:tasks.workunit.client.1.smithi189.stdout:5/435: mknod d7/d14/d18/c83 0 2024-04-20T21:32:12.050 INFO:tasks.workunit.client.0.smithi062.stdout:7/755: write d7/d3e/d33/f37 [2958497,51356] 0 2024-04-20T21:32:12.052 INFO:tasks.workunit.client.1.smithi189.stdout:5/436: write d7/d9/d31/d36/d5f/f75 [509519,28510] 0 2024-04-20T21:32:12.055 INFO:tasks.workunit.client.1.smithi189.stdout:8/551: creat da/d81/d91/fc4 x:0 0 0 2024-04-20T21:32:12.058 INFO:tasks.workunit.client.1.smithi189.stdout:3/436: mknod d2/d3/d2c/d30/c9d 0 2024-04-20T21:32:12.058 INFO:tasks.workunit.client.0.smithi062.stdout:9/871: dread - d16/d2c/d62/dd1/df8/f114 zero size 2024-04-20T21:32:12.066 INFO:tasks.workunit.client.1.smithi189.stdout:1/417: mkdir d8/d21/d2c/d36/d8a 0 2024-04-20T21:32:12.066 INFO:tasks.workunit.client.1.smithi189.stdout:2/428: unlink d1/d17/d37/l6d 0 2024-04-20T21:32:12.067 INFO:tasks.workunit.client.1.smithi189.stdout:1/418: chown d8/d1e/d43/l77 543 1 2024-04-20T21:32:12.069 INFO:tasks.workunit.client.1.smithi189.stdout:9/458: symlink d9/dc/l90 0 2024-04-20T21:32:12.070 INFO:tasks.workunit.client.0.smithi062.stdout:9/872: read d16/d3d/d55/d6e/f92 [1662428,106568] 0 2024-04-20T21:32:12.072 INFO:tasks.workunit.client.1.smithi189.stdout:0/579: write d3/df/f3e [876767,30891] 0 2024-04-20T21:32:12.073 INFO:tasks.workunit.client.0.smithi062.stdout:2/786: dwrite d7/da/d28/d4b/fbe [0,4194304] 0 2024-04-20T21:32:12.074 INFO:tasks.workunit.client.0.smithi062.stdout:9/873: write d16/d2c/d5e/f7b [4202695,8455] 0 2024-04-20T21:32:12.074 INFO:tasks.workunit.client.0.smithi062.stdout:0/755: write d7/d1c/f63 [3441345,48843] 0 2024-04-20T21:32:12.080 INFO:tasks.workunit.client.1.smithi189.stdout:5/437: mknod d7/d14/d4e/d5b/c84 0 2024-04-20T21:32:12.081 INFO:tasks.workunit.client.0.smithi062.stdout:5/778: rename d8/d1e/d2c/d6c/f60 to d8/f101 0 2024-04-20T21:32:12.081 INFO:tasks.workunit.client.0.smithi062.stdout:9/874: readlink d16/d2c/d42/da6/lcc 0 2024-04-20T21:32:12.081 INFO:tasks.workunit.client.1.smithi189.stdout:6/522: truncate d8/f9b 4977592 0 2024-04-20T21:32:12.082 INFO:tasks.workunit.client.1.smithi189.stdout:1/419: mknod d8/d21/c8b 0 2024-04-20T21:32:12.082 INFO:tasks.workunit.client.1.smithi189.stdout:5/438: readlink d7/l12 0 2024-04-20T21:32:12.083 INFO:tasks.workunit.client.0.smithi062.stdout:0/756: stat d7/d1c/ld5 0 2024-04-20T21:32:12.083 INFO:tasks.workunit.client.0.smithi062.stdout:2/787: readlink d7/da/d15/d1b/d24/l75 0 2024-04-20T21:32:12.084 INFO:tasks.workunit.client.1.smithi189.stdout:8/552: write da/df/d3c/d50/d6c/d85/fbe [505301,72170] 0 2024-04-20T21:32:12.086 INFO:tasks.workunit.client.0.smithi062.stdout:9/875: mknod d16/d2c/d62/dea/c11a 0 2024-04-20T21:32:12.087 INFO:tasks.workunit.client.1.smithi189.stdout:7/494: link d5/d59/l83 d5/lad 0 2024-04-20T21:32:12.088 INFO:tasks.workunit.client.0.smithi062.stdout:2/788: write d7/da/d15/d2f/f33 [2657422,81862] 0 2024-04-20T21:32:12.089 INFO:tasks.workunit.client.1.smithi189.stdout:5/439: chown d7/d35/f46 763848 1 2024-04-20T21:32:12.092 INFO:tasks.workunit.client.1.smithi189.stdout:2/429: dwrite d1/d17/d2f/d55/f61 [0,4194304] 0 2024-04-20T21:32:12.092 INFO:tasks.workunit.client.0.smithi062.stdout:0/757: mknod d7/d1c/d15/d53/cf5 0 2024-04-20T21:32:12.095 INFO:tasks.workunit.client.1.smithi189.stdout:6/523: fsync d8/d2b/f3a 0 2024-04-20T21:32:12.097 INFO:tasks.workunit.client.1.smithi189.stdout:5/440: chown d7/d9/d30/d55/d81 7 1 2024-04-20T21:32:12.097 INFO:tasks.workunit.client.0.smithi062.stdout:2/789: fsync d7/da/d15/d2f/d38/fbf 0 2024-04-20T21:32:12.100 INFO:tasks.workunit.client.1.smithi189.stdout:7/495: symlink d5/d21/d25/d34/lae 0 2024-04-20T21:32:12.100 INFO:tasks.workunit.client.1.smithi189.stdout:5/441: readlink d7/d14/l3d 0 2024-04-20T21:32:12.100 INFO:tasks.workunit.client.0.smithi062.stdout:9/876: creat d16/d2c/d62/f11b x:0 0 0 2024-04-20T21:32:12.101 INFO:tasks.workunit.client.0.smithi062.stdout:8/789: dwrite d7/d9/d29/f3e [0,4194304] 0 2024-04-20T21:32:12.101 INFO:tasks.workunit.client.1.smithi189.stdout:1/420: getdents d8/d1e 0 2024-04-20T21:32:12.102 INFO:tasks.workunit.client.1.smithi189.stdout:5/442: chown d7/d9/d30/d5a 47240068 1 2024-04-20T21:32:12.103 INFO:tasks.workunit.client.1.smithi189.stdout:2/430: read d1/f12 [3408,118450] 0 2024-04-20T21:32:12.103 INFO:tasks.workunit.client.1.smithi189.stdout:5/443: dread - d7/f78 zero size 2024-04-20T21:32:12.109 INFO:tasks.workunit.client.1.smithi189.stdout:6/524: creat d8/d2b/d49/d54/d6e/fb0 x:0 0 0 2024-04-20T21:32:12.110 INFO:tasks.workunit.client.1.smithi189.stdout:2/431: read d1/d17/d2f/f31 [778161,111540] 0 2024-04-20T21:32:12.111 INFO:tasks.workunit.client.0.smithi062.stdout:5/779: rename d8/d38/d7a/d4c/d7d/d9d/dac/d66/c6a to d8/d38/d7a/db1/c102 0 2024-04-20T21:32:12.115 INFO:tasks.workunit.client.1.smithi189.stdout:1/421: mkdir d8/df/d3a/d49/d8c 0 2024-04-20T21:32:12.119 INFO:tasks.workunit.client.0.smithi062.stdout:7/756: dwrite d7/d3e/f5d [0,4194304] 0 2024-04-20T21:32:12.121 INFO:tasks.workunit.client.1.smithi189.stdout:7/496: mknod d5/d1d/caf 0 2024-04-20T21:32:12.123 INFO:tasks.workunit.client.1.smithi189.stdout:2/432: mknod d1/d11/d52/c7d 0 2024-04-20T21:32:12.124 INFO:tasks.workunit.client.0.smithi062.stdout:2/790: fsync d7/da/d15/d2f/fa4 0 2024-04-20T21:32:12.126 INFO:tasks.workunit.client.1.smithi189.stdout:5/444: mknod d7/d9/d51/d70/c85 0 2024-04-20T21:32:12.127 INFO:tasks.workunit.client.0.smithi062.stdout:8/790: creat d7/d25/d4f/f104 x:0 0 0 2024-04-20T21:32:12.129 INFO:tasks.workunit.client.0.smithi062.stdout:5/780: dread - d8/d10/d3e/ff8 zero size 2024-04-20T21:32:12.129 INFO:tasks.workunit.client.0.smithi062.stdout:0/758: dwrite d7/d20/d62/fd1 [0,4194304] 0 2024-04-20T21:32:12.131 INFO:tasks.workunit.client.1.smithi189.stdout:1/422: unlink d8/d1e/f89 0 2024-04-20T21:32:12.132 INFO:tasks.workunit.client.1.smithi189.stdout:6/525: truncate d8/d2b/f3a 981544 0 2024-04-20T21:32:12.133 INFO:tasks.workunit.client.0.smithi062.stdout:7/757: mknod d7/d26/dac/dec/c104 0 2024-04-20T21:32:12.136 INFO:tasks.workunit.client.1.smithi189.stdout:9/459: dread d9/d12/d15/d1a/f64 [0,4194304] 0 2024-04-20T21:32:12.136 INFO:tasks.workunit.client.1.smithi189.stdout:0/580: dwrite d3/d8c/f7a [0,4194304] 0 2024-04-20T21:32:12.136 INFO:tasks.workunit.client.1.smithi189.stdout:1/423: readlink d8/d21/d2c/l4a 0 2024-04-20T21:32:12.144 INFO:tasks.workunit.client.1.smithi189.stdout:8/553: dwrite da/d1f/f25 [0,4194304] 0 2024-04-20T21:32:12.147 INFO:tasks.workunit.client.0.smithi062.stdout:0/759: dread d7/d1c/d15/d80/fe9 [0,4194304] 0 2024-04-20T21:32:12.147 INFO:tasks.workunit.client.1.smithi189.stdout:9/460: mkdir d9/d12/d3f/d91 0 2024-04-20T21:32:12.148 INFO:tasks.workunit.client.0.smithi062.stdout:5/781: creat d8/d38/d7a/d4c/d7d/d9d/dac/d66/daa/f103 x:0 0 0 2024-04-20T21:32:12.153 INFO:tasks.workunit.client.1.smithi189.stdout:0/581: write d3/df/d28/d2b/f36 [4401411,58198] 0 2024-04-20T21:32:12.153 INFO:tasks.workunit.client.1.smithi189.stdout:1/424: read d8/f41 [83085,15967] 0 2024-04-20T21:32:12.153 INFO:tasks.workunit.client.1.smithi189.stdout:8/554: write da/df/d13/d31/f80 [301800,17463] 0 2024-04-20T21:32:12.158 INFO:tasks.workunit.client.1.smithi189.stdout:9/461: read d9/d12/d15/f30 [429235,24161] 0 2024-04-20T21:32:12.159 INFO:tasks.workunit.client.1.smithi189.stdout:7/497: dwrite d5/d1d/d77/d5b/f5e [0,4194304] 0 2024-04-20T21:32:12.161 INFO:tasks.workunit.client.0.smithi062.stdout:8/791: link d7/d30/ced d7/d25/d3b/d58/de8/c105 0 2024-04-20T21:32:12.161 INFO:tasks.workunit.client.0.smithi062.stdout:9/877: dwrite d16/d2c/d62/f9c [0,4194304] 0 2024-04-20T21:32:12.165 INFO:tasks.workunit.client.0.smithi062.stdout:0/760: creat d7/d1e/da9/dbe/ff6 x:0 0 0 2024-04-20T21:32:12.166 INFO:tasks.workunit.client.0.smithi062.stdout:2/791: write d7/d39/f55 [1748215,4864] 0 2024-04-20T21:32:12.168 INFO:tasks.workunit.client.1.smithi189.stdout:9/462: read - d9/d12/d3f/d72/f85 zero size 2024-04-20T21:32:12.169 INFO:tasks.workunit.client.0.smithi062.stdout:0/761: fsync d7/d1e/db6/fee 0 2024-04-20T21:32:12.169 INFO:tasks.workunit.client.1.smithi189.stdout:7/498: write d5/f6 [2834397,90992] 0 2024-04-20T21:32:12.170 INFO:tasks.workunit.client.1.smithi189.stdout:1/425: creat d8/df/d50/f8d x:0 0 0 2024-04-20T21:32:12.173 INFO:tasks.workunit.client.0.smithi062.stdout:8/792: symlink d7/d25/dc5/dd3/l106 0 2024-04-20T21:32:12.178 INFO:tasks.workunit.client.1.smithi189.stdout:6/526: dwrite d8/f1a [4194304,4194304] 0 2024-04-20T21:32:12.180 INFO:tasks.workunit.client.1.smithi189.stdout:1/426: read f6 [2184655,110692] 0 2024-04-20T21:32:12.183 INFO:tasks.workunit.client.1.smithi189.stdout:6/527: stat d8/d2b/d49/d54/d6e/fb0 0 2024-04-20T21:32:12.184 INFO:tasks.workunit.client.0.smithi062.stdout:8/793: mknod d7/d25/d3b/d58/de8/c107 0 2024-04-20T21:32:12.189 INFO:tasks.workunit.client.1.smithi189.stdout:0/582: creat d3/df/d30/d45/fc4 x:0 0 0 2024-04-20T21:32:12.189 INFO:tasks.workunit.client.1.smithi189.stdout:7/499: mkdir d5/d1d/d77/d5b/d71/d7a/db0 0 2024-04-20T21:32:12.190 INFO:tasks.workunit.client.0.smithi062.stdout:9/878: dread d16/d2c/d62/f9c [0,4194304] 0 2024-04-20T21:32:12.190 INFO:tasks.workunit.client.1.smithi189.stdout:9/463: truncate d9/dc/d69/f7c 677045 0 2024-04-20T21:32:12.191 INFO:tasks.workunit.client.0.smithi062.stdout:5/782: dread d8/d38/d7a/d4c/d7d/d9d/dac/d66/dae/f5f [0,4194304] 0 2024-04-20T21:32:12.192 INFO:tasks.workunit.client.1.smithi189.stdout:9/464: readlink d9/d12/l2d 0 2024-04-20T21:32:12.192 INFO:tasks.workunit.client.0.smithi062.stdout:8/794: write d7/d25/d70/fe4 [291219,130916] 0 2024-04-20T21:32:12.193 INFO:tasks.workunit.client.0.smithi062.stdout:9/879: dread - d16/d2c/d42/da6/fd2 zero size 2024-04-20T21:32:12.198 INFO:tasks.workunit.client.0.smithi062.stdout:3/726: sync 2024-04-20T21:32:12.199 INFO:tasks.workunit.client.0.smithi062.stdout:9/880: mknod d16/d2c/d5e/d96/c11c 0 2024-04-20T21:32:12.201 INFO:tasks.workunit.client.1.smithi189.stdout:1/427: link d8/df/d3a/c42 d8/d21/d2c/c8e 0 2024-04-20T21:32:12.201 INFO:tasks.workunit.client.0.smithi062.stdout:5/783: mknod d8/d1e/d2c/d6c/df9/c104 0 2024-04-20T21:32:12.204 INFO:tasks.workunit.client.1.smithi189.stdout:3/437: dread d2/d8/f58 [0,4194304] 0 2024-04-20T21:32:12.206 INFO:tasks.workunit.client.0.smithi062.stdout:9/881: creat d16/d2c/d62/dd1/d105/f11d x:0 0 0 2024-04-20T21:32:12.207 INFO:tasks.workunit.client.1.smithi189.stdout:3/438: fdatasync d2/f6 0 2024-04-20T21:32:12.208 INFO:tasks.workunit.client.1.smithi189.stdout:7/500: dwrite d5/f14 [0,4194304] 0 2024-04-20T21:32:12.212 INFO:tasks.workunit.client.0.smithi062.stdout:5/784: mkdir d8/d38/d7a/d4c/d7d/d9d/dac/d105 0 2024-04-20T21:32:12.212 INFO:tasks.workunit.client.1.smithi189.stdout:0/583: symlink d3/df/lc5 0 2024-04-20T21:32:12.216 INFO:tasks.workunit.client.0.smithi062.stdout:3/727: link d0/d1/d10/d3a/l8e d0/d1/d3c/d43/d72/dd3/dc5/lf1 0 2024-04-20T21:32:12.218 INFO:tasks.workunit.client.0.smithi062.stdout:9/882: creat d16/d2f/d5b/dab/dcd/f11e x:0 0 0 2024-04-20T21:32:12.219 INFO:tasks.workunit.client.1.smithi189.stdout:3/439: write d2/d3/d25/f40 [7343918,126722] 0 2024-04-20T21:32:12.221 INFO:tasks.workunit.client.1.smithi189.stdout:3/440: readlink d2/d3/d2c/d30/l66 0 2024-04-20T21:32:12.223 INFO:tasks.workunit.client.0.smithi062.stdout:9/883: chown d16/f1c 25 1 2024-04-20T21:32:12.223 INFO:tasks.workunit.client.0.smithi062.stdout:2/792: dwrite d7/d39/f45 [0,4194304] 0 2024-04-20T21:32:12.224 INFO:tasks.workunit.client.0.smithi062.stdout:5/785: symlink d8/d38/d7a/d4c/d7d/d9d/dac/d2f/d8e/dd6/dfe/l106 0 2024-04-20T21:32:12.224 INFO:tasks.workunit.client.1.smithi189.stdout:7/501: creat d5/d46/d7d/fb1 x:0 0 0 2024-04-20T21:32:12.225 INFO:tasks.workunit.client.1.smithi189.stdout:8/555: dwrite da/de/d1e/d21/f67 [0,4194304] 0 2024-04-20T21:32:12.226 INFO:tasks.workunit.client.1.smithi189.stdout:8/556: stat da/df 0 2024-04-20T21:32:12.228 INFO:tasks.workunit.client.0.smithi062.stdout:5/786: chown d8/d10/ld9 726 1 2024-04-20T21:32:12.229 INFO:tasks.workunit.client.0.smithi062.stdout:2/793: read - d7/da/d15/d2f/d93/dac/fb8 zero size 2024-04-20T21:32:12.230 INFO:tasks.workunit.client.0.smithi062.stdout:3/728: fsync d0/d1/d10/d31/d41/f7c 0 2024-04-20T21:32:12.236 INFO:tasks.workunit.client.1.smithi189.stdout:4/443: sync 2024-04-20T21:32:12.236 INFO:tasks.workunit.client.1.smithi189.stdout:9/465: rename d9/l1b to d9/dc/d17/d7e/l92 0 2024-04-20T21:32:12.236 INFO:tasks.workunit.client.0.smithi062.stdout:2/794: chown d7/da/d15/d2f/d38/d6b/fd1 1445306478 1 2024-04-20T21:32:12.237 INFO:tasks.workunit.client.0.smithi062.stdout:5/787: mknod d8/d38/d7a/c107 0 2024-04-20T21:32:12.238 INFO:tasks.workunit.client.0.smithi062.stdout:3/729: mkdir d0/d1/d3c/d43/d72/df2 0 2024-04-20T21:32:12.240 INFO:tasks.workunit.client.1.smithi189.stdout:6/528: getdents d8/d10 0 2024-04-20T21:32:12.242 INFO:tasks.workunit.client.0.smithi062.stdout:5/788: write d8/d38/fc5 [4618298,77997] 0 2024-04-20T21:32:12.242 INFO:tasks.workunit.client.1.smithi189.stdout:9/466: read d9/d12/d3b/f4c [3216532,34881] 0 2024-04-20T21:32:12.243 INFO:tasks.workunit.client.0.smithi062.stdout:2/795: chown d7/da/d15/d1b/d24/f7a 25 1 2024-04-20T21:32:12.243 INFO:tasks.workunit.client.1.smithi189.stdout:3/441: unlink d2/d3/d2c/d31/f5e 0 2024-04-20T21:32:12.245 INFO:tasks.workunit.client.1.smithi189.stdout:9/467: read d9/dc/f2c [1246405,63730] 0 2024-04-20T21:32:12.247 INFO:tasks.workunit.client.0.smithi062.stdout:0/762: dread d7/d1c/f63 [0,4194304] 0 2024-04-20T21:32:12.248 INFO:tasks.workunit.client.1.smithi189.stdout:9/468: stat d9/d12/d3f/d72/f85 0 2024-04-20T21:32:12.250 INFO:tasks.workunit.client.1.smithi189.stdout:0/584: mknod d3/df/d30/cc6 0 2024-04-20T21:32:12.254 INFO:tasks.workunit.client.0.smithi062.stdout:5/789: truncate d8/d38/d7a/d4c/fcd 731252 0 2024-04-20T21:32:12.254 INFO:tasks.workunit.client.0.smithi062.stdout:9/884: write d16/f9e [4388819,16294] 0 2024-04-20T21:32:12.256 INFO:tasks.workunit.client.1.smithi189.stdout:8/557: fsync da/df/d13/f5f 0 2024-04-20T21:32:12.258 INFO:tasks.workunit.client.0.smithi062.stdout:0/763: mknod d7/d1c/d15/cf7 0 2024-04-20T21:32:12.259 INFO:tasks.workunit.client.0.smithi062.stdout:9/885: mknod d16/d3e/d77/d84/c11f 0 2024-04-20T21:32:12.262 INFO:tasks.workunit.client.0.smithi062.stdout:0/764: fsync d7/d1c/d15/d17/fad 0 2024-04-20T21:32:12.262 INFO:tasks.workunit.client.1.smithi189.stdout:4/444: dread d2/f2c [0,4194304] 0 2024-04-20T21:32:12.264 INFO:tasks.workunit.client.1.smithi189.stdout:7/502: truncate d5/d21/d4a/f4d 211790 0 2024-04-20T21:32:12.265 INFO:tasks.workunit.client.0.smithi062.stdout:9/886: fsync d16/d2c/d42/da6/fe3 0 2024-04-20T21:32:12.266 INFO:tasks.workunit.client.0.smithi062.stdout:2/796: write d7/da/f13 [746669,117132] 0 2024-04-20T21:32:12.268 INFO:tasks.workunit.client.1.smithi189.stdout:4/445: symlink d2/dd/la3 0 2024-04-20T21:32:12.268 INFO:tasks.workunit.client.1.smithi189.stdout:1/428: rename d8/d21/d2c/d36/c69 to d8/df/c8f 0 2024-04-20T21:32:12.269 INFO:tasks.workunit.client.0.smithi062.stdout:2/797: stat d7/d39/cec 0 2024-04-20T21:32:12.270 INFO:tasks.workunit.client.0.smithi062.stdout:0/765: mkdir d7/d1c/d15/d17/d37/df8 0 2024-04-20T21:32:12.272 INFO:tasks.workunit.client.0.smithi062.stdout:3/730: dwrite d0/d1c/d63/d42/fe2 [0,4194304] 0 2024-04-20T21:32:12.274 INFO:tasks.workunit.client.0.smithi062.stdout:2/798: write d7/da/d15/f23 [122362,128375] 0 2024-04-20T21:32:12.275 INFO:tasks.workunit.client.1.smithi189.stdout:7/503: chown d5/d21/d4a/d52/c6a 101015018 1 2024-04-20T21:32:12.275 INFO:tasks.workunit.client.1.smithi189.stdout:4/446: mkdir d2/dd/d11/d14/d69/da4 0 2024-04-20T21:32:12.275 INFO:tasks.workunit.client.1.smithi189.stdout:0/585: truncate d3/df/f91 623894 0 2024-04-20T21:32:12.277 INFO:tasks.workunit.client.1.smithi189.stdout:1/429: write d8/f1b [4878255,101660] 0 2024-04-20T21:32:12.277 INFO:tasks.workunit.client.1.smithi189.stdout:8/558: getdents da/df/d3c/d50 0 2024-04-20T21:32:12.286 INFO:tasks.workunit.client.1.smithi189.stdout:4/447: symlink d2/d20/la5 0 2024-04-20T21:32:12.287 INFO:tasks.workunit.client.1.smithi189.stdout:7/504: write d5/d46/faa [60553,111065] 0 2024-04-20T21:32:12.291 INFO:tasks.workunit.client.0.smithi062.stdout:2/799: link d7/da/d15/d2f/d38/db0/fb2 d7/da/d15/d2f/d38/d51/f105 0 2024-04-20T21:32:12.291 INFO:tasks.workunit.client.1.smithi189.stdout:4/448: mkdir d2/dd/d6b/da6 0 2024-04-20T21:32:12.292 INFO:tasks.workunit.client.1.smithi189.stdout:3/442: dread d2/d3/d65/f78 [0,4194304] 0 2024-04-20T21:32:12.293 INFO:tasks.workunit.client.1.smithi189.stdout:7/505: creat d5/d21/d4a/d52/d75/fb2 x:0 0 0 2024-04-20T21:32:12.294 INFO:tasks.workunit.client.1.smithi189.stdout:9/469: dwrite d9/dc/f3d [0,4194304] 0 2024-04-20T21:32:12.295 INFO:tasks.workunit.client.1.smithi189.stdout:8/559: mkdir da/df/d22/d33/dba/dc5 0 2024-04-20T21:32:12.297 INFO:tasks.workunit.client.1.smithi189.stdout:3/443: write d2/d1c/f7d [926167,40896] 0 2024-04-20T21:32:12.299 INFO:tasks.workunit.client.1.smithi189.stdout:7/506: fsync d5/fe 0 2024-04-20T21:32:12.301 INFO:tasks.workunit.client.0.smithi062.stdout:3/731: write d0/d1/d10/d3a/d5e/d65/d88/feb [2221836,8379] 0 2024-04-20T21:32:12.301 INFO:tasks.workunit.client.1.smithi189.stdout:8/560: creat da/df/d3c/d68/d8b/fc6 x:0 0 0 2024-04-20T21:32:12.301 INFO:tasks.workunit.client.1.smithi189.stdout:4/449: write d2/dd/f44 [1014491,72378] 0 2024-04-20T21:32:12.303 INFO:tasks.workunit.client.1.smithi189.stdout:7/507: chown d5/d1d/d77/d5b/d71/d7a/db0 292 1 2024-04-20T21:32:12.303 INFO:tasks.workunit.client.1.smithi189.stdout:3/444: unlink d2/d8/f57 0 2024-04-20T21:32:12.304 INFO:tasks.workunit.client.0.smithi062.stdout:9/887: dwrite d16/d3d/d55/d6e/d7d/f91 [0,4194304] 0 2024-04-20T21:32:12.307 INFO:tasks.workunit.client.1.smithi189.stdout:1/430: rename d8/df/f13 to d8/d1e/d28/f90 0 2024-04-20T21:32:12.308 INFO:tasks.workunit.client.0.smithi062.stdout:2/800: rename d7/da/d28/l77 to d7/da/d15/d2f/d93/dac/dca/dd8/ddc/l106 0 2024-04-20T21:32:12.309 INFO:tasks.workunit.client.1.smithi189.stdout:7/508: stat d5/d21/d25/d34/lae 0 2024-04-20T21:32:12.310 INFO:tasks.workunit.client.1.smithi189.stdout:8/561: mkdir da/df/d3c/d50/d6c/dc7 0 2024-04-20T21:32:12.311 INFO:tasks.workunit.client.1.smithi189.stdout:0/586: dwrite d3/df/d4c/fa4 [0,4194304] 0 2024-04-20T21:32:12.312 INFO:tasks.workunit.client.1.smithi189.stdout:4/450: symlink d2/dd/d11/la7 0 2024-04-20T21:32:12.314 INFO:tasks.workunit.client.1.smithi189.stdout:4/451: read - d2/dd/d37/f39 zero size 2024-04-20T21:32:12.314 INFO:tasks.workunit.client.1.smithi189.stdout:3/445: mknod d2/d3/d2c/d31/d90/c9e 0 2024-04-20T21:32:12.322 INFO:tasks.workunit.client.1.smithi189.stdout:0/587: creat d3/d8c/fc7 x:0 0 0 2024-04-20T21:32:12.323 INFO:tasks.workunit.client.1.smithi189.stdout:1/431: creat d8/d1e/d28/f91 x:0 0 0 2024-04-20T21:32:12.324 INFO:tasks.workunit.client.1.smithi189.stdout:1/432: dread - d8/df/d3a/f78 zero size 2024-04-20T21:32:12.327 INFO:tasks.workunit.client.1.smithi189.stdout:1/433: truncate d8/f14 5163667 0 2024-04-20T21:32:12.327 INFO:tasks.workunit.client.1.smithi189.stdout:1/434: chown d8/df/c55 116105 1 2024-04-20T21:32:12.328 INFO:tasks.workunit.client.1.smithi189.stdout:4/452: mknod d2/d22/d87/ca8 0 2024-04-20T21:32:12.329 INFO:tasks.workunit.client.1.smithi189.stdout:3/446: symlink d2/d3/d88/l9f 0 2024-04-20T21:32:12.333 INFO:tasks.workunit.client.1.smithi189.stdout:9/470: dread d9/d12/f29 [0,4194304] 0 2024-04-20T21:32:12.334 INFO:tasks.workunit.client.1.smithi189.stdout:8/562: write da/de/f28 [98263,94023] 0 2024-04-20T21:32:12.337 INFO:tasks.workunit.client.1.smithi189.stdout:4/453: rmdir d2/dd 39 2024-04-20T21:32:12.339 INFO:tasks.workunit.client.1.smithi189.stdout:4/454: write d2/d22/f30 [1600624,37600] 0 2024-04-20T21:32:12.340 INFO:tasks.workunit.client.1.smithi189.stdout:7/509: getdents d5/d21/d25 0 2024-04-20T21:32:12.342 INFO:tasks.workunit.client.1.smithi189.stdout:7/510: chown d5/d21/d4a/d52/d74/l9a 10749733 1 2024-04-20T21:32:12.343 INFO:tasks.workunit.client.1.smithi189.stdout:9/471: unlink d9/d12/d50/f80 0 2024-04-20T21:32:12.345 INFO:tasks.workunit.client.1.smithi189.stdout:8/563: dread da/de/d1e/d21/d44/f47 [0,4194304] 0 2024-04-20T21:32:12.347 INFO:tasks.workunit.client.1.smithi189.stdout:9/472: truncate d9/dc/d17/d2b/d56/f86 843871 0 2024-04-20T21:32:12.349 INFO:tasks.workunit.client.1.smithi189.stdout:1/435: creat d8/df/f92 x:0 0 0 2024-04-20T21:32:12.350 INFO:tasks.workunit.client.1.smithi189.stdout:0/588: getdents d3/df/d28 0 2024-04-20T21:32:12.350 INFO:tasks.workunit.client.1.smithi189.stdout:7/511: symlink d5/d46/d7d/lb3 0 2024-04-20T21:32:12.350 INFO:tasks.workunit.client.1.smithi189.stdout:9/473: symlink d9/dc/d17/d2b/d56/l93 0 2024-04-20T21:32:12.353 INFO:tasks.workunit.client.1.smithi189.stdout:0/589: mkdir d3/df/d28/dc8 0 2024-04-20T21:32:12.358 INFO:tasks.workunit.client.1.smithi189.stdout:4/455: write d2/dd/d11/d14/d69/d6a/d7a/f8c [3175041,101320] 0 2024-04-20T21:32:12.358 INFO:tasks.workunit.client.1.smithi189.stdout:1/436: mkdir d8/d1e/d54/d93 0 2024-04-20T21:32:12.361 INFO:tasks.workunit.client.1.smithi189.stdout:4/456: fsync d2/dd/d11/d14/d69/f9e 0 2024-04-20T21:32:12.364 INFO:tasks.workunit.client.1.smithi189.stdout:1/437: mknod d8/d1e/d54/d93/c94 0 2024-04-20T21:32:12.367 INFO:tasks.workunit.client.1.smithi189.stdout:3/447: dwrite d2/d1c/f7b [0,4194304] 0 2024-04-20T21:32:12.368 INFO:tasks.workunit.client.1.smithi189.stdout:4/457: mkdir d2/d22/da9 0 2024-04-20T21:32:12.373 INFO:tasks.workunit.client.1.smithi189.stdout:1/438: creat d8/f95 x:0 0 0 2024-04-20T21:32:12.375 INFO:tasks.workunit.client.1.smithi189.stdout:8/564: dread da/de/d1e/d21/d44/f4a [0,4194304] 0 2024-04-20T21:32:12.376 INFO:tasks.workunit.client.1.smithi189.stdout:4/458: symlink d2/laa 0 2024-04-20T21:32:12.379 INFO:tasks.workunit.client.0.smithi062.stdout:0/766: dwrite d7/d1c/d15/d17/d1b/d9d/db1/feb [0,4194304] 0 2024-04-20T21:32:12.379 INFO:tasks.workunit.client.1.smithi189.stdout:7/512: dwrite d5/d46/d7d/d45/f99 [0,4194304] 0 2024-04-20T21:32:12.379 INFO:tasks.workunit.client.1.smithi189.stdout:3/448: creat d2/d3/d2c/d30/d3f/d9a/fa0 x:0 0 0 2024-04-20T21:32:12.381 INFO:tasks.workunit.client.0.smithi062.stdout:3/732: truncate d0/d1/fb 577622 0 2024-04-20T21:32:12.382 INFO:tasks.workunit.client.1.smithi189.stdout:9/474: dwrite f4 [0,4194304] 0 2024-04-20T21:32:12.382 INFO:tasks.workunit.client.1.smithi189.stdout:4/459: symlink d2/d20/lab 0 2024-04-20T21:32:12.382 INFO:tasks.workunit.client.1.smithi189.stdout:1/439: rename d8/c1d to d8/d21/d2c/d36/d8a/c96 0 2024-04-20T21:32:12.383 INFO:tasks.workunit.client.1.smithi189.stdout:5/445: sync 2024-04-20T21:32:12.385 INFO:tasks.workunit.client.0.smithi062.stdout:1/827: sync 2024-04-20T21:32:12.385 INFO:tasks.workunit.client.0.smithi062.stdout:6/658: sync 2024-04-20T21:32:12.385 INFO:tasks.workunit.client.1.smithi189.stdout:6/529: sync 2024-04-20T21:32:12.385 INFO:tasks.workunit.client.1.smithi189.stdout:2/433: sync 2024-04-20T21:32:12.387 INFO:tasks.workunit.client.0.smithi062.stdout:6/659: chown d0/d1a 0 1 2024-04-20T21:32:12.387 INFO:tasks.workunit.client.1.smithi189.stdout:8/565: write da/df/d13/d31/fbf [854072,95550] 0 2024-04-20T21:32:12.391 INFO:tasks.workunit.client.0.smithi062.stdout:9/888: dwrite d16/d3d/d55/fb7 [0,4194304] 0 2024-04-20T21:32:12.391 INFO:tasks.workunit.client.0.smithi062.stdout:3/733: dread - d0/d1c/d63/d42/fe5 zero size 2024-04-20T21:32:12.394 INFO:tasks.workunit.client.1.smithi189.stdout:9/475: write d9/d12/d15/f8f [483865,111332] 0 2024-04-20T21:32:12.395 INFO:tasks.workunit.client.1.smithi189.stdout:5/446: read - d7/d14/d18/f77 zero size 2024-04-20T21:32:12.396 INFO:tasks.workunit.client.1.smithi189.stdout:1/440: readlink d8/d21/d2c/d36/l7d 0 2024-04-20T21:32:12.398 INFO:tasks.workunit.client.1.smithi189.stdout:2/434: creat d1/d17/d2f/d4a/f7e x:0 0 0 2024-04-20T21:32:12.398 INFO:tasks.workunit.client.1.smithi189.stdout:9/476: dread - d9/dc/d69/f7f zero size 2024-04-20T21:32:12.399 INFO:tasks.workunit.client.0.smithi062.stdout:2/801: rename d7/da/d15/d2f/d93/dac to d7/da/d28/d107 0 2024-04-20T21:32:12.400 INFO:tasks.workunit.client.1.smithi189.stdout:6/530: unlink d8/da/caa 0 2024-04-20T21:32:12.400 INFO:tasks.workunit.client.1.smithi189.stdout:7/513: write d5/d1d/d77/d5b/d71/fa4 [207457,106649] 0 2024-04-20T21:32:12.402 INFO:tasks.workunit.client.1.smithi189.stdout:1/441: fdatasync d8/d1e/d43/d58/d60/f64 0 2024-04-20T21:32:12.402 INFO:tasks.workunit.client.0.smithi062.stdout:2/802: write d7/da/d15/d2f/f33 [4577687,106997] 0 2024-04-20T21:32:12.403 INFO:tasks.workunit.client.1.smithi189.stdout:2/435: read d1/d11/d58/d6b/f6c [2614002,21128] 0 2024-04-20T21:32:12.403 INFO:tasks.workunit.client.0.smithi062.stdout:0/767: mkdir d7/d1e/df9 0 2024-04-20T21:32:12.404 INFO:tasks.workunit.client.1.smithi189.stdout:3/449: mkdir d2/d3/d36/da1 0 2024-04-20T21:32:12.406 INFO:tasks.workunit.client.1.smithi189.stdout:9/477: readlink d9/d12/d15/d1a/l3e 0 2024-04-20T21:32:12.411 INFO:tasks.workunit.client.1.smithi189.stdout:1/442: fdatasync d8/d21/d2c/f75 0 2024-04-20T21:32:12.413 INFO:tasks.workunit.client.1.smithi189.stdout:0/590: dread d3/df/d28/d2b/d2c/f62 [0,4194304] 0 2024-04-20T21:32:12.414 INFO:tasks.workunit.client.0.smithi062.stdout:9/889: symlink d16/d2c/d62/dd1/l120 0 2024-04-20T21:32:12.415 INFO:tasks.workunit.client.0.smithi062.stdout:1/828: fdatasync dc/d15/d75/d96/dd4/d3a/da2/f120 0 2024-04-20T21:32:12.417 INFO:tasks.workunit.client.0.smithi062.stdout:6/660: read d0/d9/d11/d22/dd5/d67/f6a [25458,39308] 0 2024-04-20T21:32:12.418 INFO:tasks.workunit.client.0.smithi062.stdout:2/803: fsync d7/da/d15/d2f/d38/d6b/fa3 0 2024-04-20T21:32:12.420 INFO:tasks.workunit.client.1.smithi189.stdout:8/566: rename da/df/d13/d31/fbf to da/de/d1e/d69/fc8 0 2024-04-20T21:32:12.420 INFO:tasks.workunit.client.0.smithi062.stdout:1/829: read dc/d15/d75/ff9 [865032,56733] 0 2024-04-20T21:32:12.420 INFO:tasks.workunit.client.0.smithi062.stdout:0/768: creat d7/d1c/d15/d17/d40/ffa x:0 0 0 2024-04-20T21:32:12.421 INFO:tasks.workunit.client.1.smithi189.stdout:1/443: mknod d8/d21/c97 0 2024-04-20T21:32:12.422 INFO:tasks.workunit.client.1.smithi189.stdout:5/447: dread d7/d9/d28/f58 [0,4194304] 0 2024-04-20T21:32:12.423 INFO:tasks.workunit.client.0.smithi062.stdout:1/830: readlink dc/d15/d75/d96/dd4/d3a/lcd 0 2024-04-20T21:32:12.425 INFO:tasks.workunit.client.1.smithi189.stdout:9/478: mkdir d9/dc/d17/d2b/d52/d82/d94 0 2024-04-20T21:32:12.426 INFO:tasks.workunit.client.1.smithi189.stdout:1/444: write d8/df/f92 [356238,118050] 0 2024-04-20T21:32:12.431 INFO:tasks.workunit.client.1.smithi189.stdout:1/445: dread - d8/f86 zero size 2024-04-20T21:32:12.432 INFO:tasks.workunit.client.1.smithi189.stdout:0/591: creat d3/df/d30/d45/d90/fc9 x:0 0 0 2024-04-20T21:32:12.436 INFO:tasks.workunit.client.0.smithi062.stdout:6/661: unlink d0/l49 0 2024-04-20T21:32:12.436 INFO:tasks.workunit.client.1.smithi189.stdout:0/592: fsync d3/d26/d88/fa0 0 2024-04-20T21:32:12.437 INFO:tasks.workunit.client.1.smithi189.stdout:5/448: symlink d7/d9/d31/l86 0 2024-04-20T21:32:12.437 INFO:tasks.workunit.client.1.smithi189.stdout:9/479: write d9/d12/d15/f8f [390139,6765] 0 2024-04-20T21:32:12.438 INFO:tasks.workunit.client.0.smithi062.stdout:0/769: readlink d7/l4f 0 2024-04-20T21:32:12.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:12.442+0000 7f3a70c00700 1 -- 172.21.15.62:0/2801018281 --> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3a6c105ab0 con 0x7f3a5006ebd0 2024-04-20T21:32:12.443 INFO:tasks.workunit.client.1.smithi189.stdout:6/531: dwrite d8/d12/f9d [0,4194304] 0 2024-04-20T21:32:12.444 INFO:tasks.workunit.client.1.smithi189.stdout:7/514: rename d5/d1d/d77/d5b to d5/db4 0 2024-04-20T21:32:12.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:12.447+0000 7f3a63e00700 1 -- 172.21.15.62:0/2801018281 <== mgr.14658 v2:172.21.15.62:6800/1585760929 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+329 (secure 0 0 0) 0x7f3a6c105ab0 con 0x7f3a5006ebd0 2024-04-20T21:32:12.446 INFO:tasks.workunit.client.1.smithi189.stdout:0/593: dread - d3/df/d30/d45/fc4 zero size 2024-04-20T21:32:12.447 INFO:tasks.workunit.client.1.smithi189.stdout:7/515: write d5/f43 [4962873,30794] 0 2024-04-20T21:32:12.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:12.450+0000 7f3a70c00700 1 -- 172.21.15.62:0/2801018281 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f3a5006ebd0 msgr2=0x7f3a50071080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:32:12.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:12.450+0000 7f3a70c00700 1 --2- 172.21.15.62:0/2801018281 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f3a5006ebd0 0x7f3a50071080 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f3a54009ad0 tx=0x7f3a54009290 comp rx=0 tx=0).stop 2024-04-20T21:32:12.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:12.450+0000 7f3a70c00700 1 -- 172.21.15.62:0/2801018281 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3a6c101490 msgr2=0x7f3a6c19c480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:32:12.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:12.450+0000 7f3a70c00700 1 --2- 172.21.15.62:0/2801018281 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3a6c101490 0x7f3a6c19c480 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c0ffde0 tx=0x7f3a5c003c30 comp rx=0 tx=0).stop 2024-04-20T21:32:12.450 INFO:tasks.workunit.client.1.smithi189.stdout:7/516: chown d5/d21/d25 331820800 1 2024-04-20T21:32:12.450 INFO:tasks.workunit.client.1.smithi189.stdout:3/450: dwrite d2/d8/f45 [0,4194304] 0 2024-04-20T21:32:12.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:12.451+0000 7f3a70c00700 1 -- 172.21.15.62:0/2801018281 shutdown_connections 2024-04-20T21:32:12.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:12.451+0000 7f3a70c00700 1 --2- 172.21.15.62:0/2801018281 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f3a5006ebd0 0x7f3a50071080 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:32:12.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:12.451+0000 7f3a70c00700 1 --2- 172.21.15.62:0/2801018281 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3a6c101490 0x7f3a6c19c480 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:32:12.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:12.451+0000 7f3a70c00700 1 --2- 172.21.15.62:0/2801018281 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a6c19c9c0 0x7f3a6c1a1a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:32:12.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:12.451+0000 7f3a70c00700 1 -- 172.21.15.62:0/2801018281 >> 172.21.15.62:0/2801018281 conn(0x7f3a6c0fb860 msgr2=0x7f3a6c0fdb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:32:12.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:12.451+0000 7f3a70c00700 1 -- 172.21.15.62:0/2801018281 shutdown_connections 2024-04-20T21:32:12.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:12.452+0000 7f3a70c00700 1 -- 172.21.15.62:0/2801018281 wait complete. 2024-04-20T21:32:12.451 INFO:tasks.workunit.client.0.smithi062.stdout:1/831: rename dc/d15/d75/d96/c127 to dc/d15/d75/d94/d100/c12f 0 2024-04-20T21:32:12.452 INFO:tasks.workunit.client.1.smithi189.stdout:6/532: fsync d8/d12/d30/d47/f8a 0 2024-04-20T21:32:12.453 INFO:tasks.workunit.client.0.smithi062.stdout:1/832: chown dc/c41 162962 1 2024-04-20T21:32:12.455 INFO:tasks.workunit.client.0.smithi062.stdout:3/734: fsync d0/d1/fb 0 2024-04-20T21:32:12.459 INFO:tasks.workunit.client.1.smithi189.stdout:6/533: fsync d8/d2b/d49/d54/d6e/fb0 0 2024-04-20T21:32:12.459 INFO:tasks.workunit.client.1.smithi189.stdout:3/451: chown d2/d3/d2c/d30/l72 50 1 2024-04-20T21:32:12.462 INFO:tasks.workunit.client.1.smithi189.stdout:2/436: rename d1/f9 to d1/d17/f7f 0 2024-04-20T21:32:12.462 INFO:tasks.workunit.client.1.smithi189.stdout:3/452: fsync d2/d3/d2c/d30/d3f/d9a/fa0 0 2024-04-20T21:32:12.466 INFO:tasks.workunit.client.0.smithi062.stdout:9/890: dread d16/d2c/d5e/d75/faa [0,4194304] 0 2024-04-20T21:32:12.467 INFO:tasks.workunit.client.1.smithi189.stdout:4/460: dread d2/f19 [0,4194304] 0 2024-04-20T21:32:12.469 INFO:tasks.workunit.client.0.smithi062.stdout:3/735: symlink d0/d1b/d74/lf3 0 2024-04-20T21:32:12.471 INFO:tasks.workunit.client.1.smithi189.stdout:4/461: dread - d2/dd/d37/f39 zero size 2024-04-20T21:32:12.471 INFO:tasks.workunit.client.1.smithi189.stdout:3/453: creat d2/d3/d2c/d31/d90/d92/fa2 x:0 0 0 2024-04-20T21:32:12.478 INFO:tasks.workunit.client.1.smithi189.stdout:1/446: rename d8/fa to d8/d1e/d28/d4f/f98 0 2024-04-20T21:32:12.481 INFO:tasks.workunit.client.1.smithi189.stdout:8/567: truncate da/de/d1e/d21/d44/f43 1102751 0 2024-04-20T21:32:12.483 INFO:tasks.workunit.client.1.smithi189.stdout:1/447: creat d8/df/d50/f99 x:0 0 0 2024-04-20T21:32:12.483 INFO:tasks.workunit.client.1.smithi189.stdout:0/594: write d3/fd [4052619,71797] 0 2024-04-20T21:32:12.485 INFO:tasks.workunit.client.1.smithi189.stdout:7/517: rename d5/d59/c73 to d5/db4/d71/d7a/db0/cb5 0 2024-04-20T21:32:12.489 INFO:tasks.workunit.client.1.smithi189.stdout:4/462: dread d2/dd/d11/f31 [0,4194304] 0 2024-04-20T21:32:12.489 INFO:tasks.workunit.client.0.smithi062.stdout:6/662: creat d0/d52/fdc x:0 0 0 2024-04-20T21:32:12.490 INFO:tasks.workunit.client.0.smithi062.stdout:6/663: stat d0/d1a 0 2024-04-20T21:32:12.490 INFO:tasks.workunit.client.1.smithi189.stdout:1/448: write f6 [4106582,120103] 0 2024-04-20T21:32:12.496 INFO:tasks.workunit.client.0.smithi062.stdout:3/736: chown d0/d1/d3c/d43/d72/fbd 120 1 2024-04-20T21:32:12.496 INFO:tasks.workunit.client.1.smithi189.stdout:0/595: write d3/df/d30/d37/f31 [5557322,17342] 0 2024-04-20T21:32:12.497 INFO:tasks.workunit.client.0.smithi062.stdout:3/737: readlink d0/l6 0 2024-04-20T21:32:12.497 INFO:tasks.workunit.client.0.smithi062.stdout:1/833: getdents dc/d15/d20 0 2024-04-20T21:32:12.498 INFO:tasks.workunit.client.0.smithi062.stdout:2/804: write d7/da/d15/f26 [233413,4660] 0 2024-04-20T21:32:12.502 INFO:tasks.workunit.client.0.smithi062.stdout:6/664: mkdir d0/d9/d11/d7b/d99/ddd 0 2024-04-20T21:32:12.503 INFO:tasks.workunit.client.1.smithi189.stdout:1/449: mknod d8/d1e/c9a 0 2024-04-20T21:32:12.504 INFO:tasks.workunit.client.0.smithi062.stdout:9/891: creat d16/f121 x:0 0 0 2024-04-20T21:32:12.504 INFO:tasks.workunit.client.1.smithi189.stdout:8/568: dread da/d1f/d2a/f4b [0,4194304] 0 2024-04-20T21:32:12.506 INFO:tasks.workunit.client.1.smithi189.stdout:1/450: fdatasync d8/df/d50/f99 0 2024-04-20T21:32:12.507 INFO:tasks.workunit.client.1.smithi189.stdout:5/449: dread d7/d9/d30/f1c [0,4194304] 0 2024-04-20T21:32:12.508 INFO:tasks.workunit.client.1.smithi189.stdout:3/454: dwrite d2/d3/d2c/f6b [0,4194304] 0 2024-04-20T21:32:12.508 INFO:tasks.workunit.client.0.smithi062.stdout:4/855: sync 2024-04-20T21:32:12.513 INFO:tasks.workunit.client.0.smithi062.stdout:0/770: dread d7/d1c/d15/d17/d40/d3c/dc5/f88 [0,4194304] 0 2024-04-20T21:32:12.515 INFO:tasks.workunit.client.1.smithi189.stdout:8/569: truncate da/df/d20/d2c/f92 694429 0 2024-04-20T21:32:12.517 INFO:tasks.workunit.client.1.smithi189.stdout:6/534: dread d8/f63 [0,4194304] 0 2024-04-20T21:32:12.519 INFO:tasks.workunit.client.0.smithi062.stdout:1/834: mknod dc/d27/c130 0 2024-04-20T21:32:12.519 INFO:tasks.workunit.client.0.smithi062.stdout:9/892: write d16/d2c/d5e/d75/fdd [1261068,13626] 0 2024-04-20T21:32:12.520 INFO:tasks.workunit.client.0.smithi062.stdout:3/738: read d0/d1/d3c/d43/d72/dd3/d97/fa3 [205942,87228] 0 2024-04-20T21:32:12.520 INFO:tasks.workunit.client.0.smithi062.stdout:6/665: dwrite d0/d9/f16 [0,4194304] 0 2024-04-20T21:32:12.520 INFO:tasks.workunit.client.1.smithi189.stdout:1/451: mkdir d8/d9b 0 2024-04-20T21:32:12.521 INFO:tasks.workunit.client.1.smithi189.stdout:3/455: chown d2/d3/d2c/d30/f68 46492 1 2024-04-20T21:32:12.521 INFO:tasks.workunit.client.0.smithi062.stdout:0/771: mkdir d7/d1c/d15/d17/d37/dfb 0 2024-04-20T21:32:12.525 INFO:tasks.workunit.client.1.smithi189.stdout:5/450: mkdir d7/d9/d30/d55/d61/d87 0 2024-04-20T21:32:12.526 INFO:tasks.workunit.client.1.smithi189.stdout:2/437: dread d1/d17/d2f/f49 [0,4194304] 0 2024-04-20T21:32:12.527 INFO:tasks.workunit.client.0.smithi062.stdout:2/805: link d7/da/d15/f1d d7/da/d15/d1b/d24/dee/f108 0 2024-04-20T21:32:12.528 INFO:tasks.workunit.client.1.smithi189.stdout:0/596: write d3/d26/f1d [4304717,29066] 0 2024-04-20T21:32:12.530 INFO:tasks.workunit.client.1.smithi189.stdout:8/570: mkdir da/df/d20/dc9 0 2024-04-20T21:32:12.530 INFO:tasks.workunit.client.1.smithi189.stdout:7/518: dwrite d5/d1d/f56 [0,4194304] 0 2024-04-20T21:32:12.531 INFO:tasks.workunit.client.0.smithi062.stdout:0/772: dread d7/d1e/fdc [0,4194304] 0 2024-04-20T21:32:12.534 INFO:tasks.workunit.client.0.smithi062.stdout:2/806: write d7/da/f22 [7274654,110232] 0 2024-04-20T21:32:12.535 INFO:tasks.workunit.client.0.smithi062.stdout:6/666: creat d0/d1a/d44/db3/fde x:0 0 0 2024-04-20T21:32:12.537 INFO:tasks.workunit.client.0.smithi062.stdout:0/773: write d7/d1e/da9/dbe/ff6 [1006609,52839] 0 2024-04-20T21:32:12.538 INFO:tasks.workunit.client.1.smithi189.stdout:6/535: mknod d8/d12/daf/cb1 0 2024-04-20T21:32:12.538 INFO:tasks.workunit.client.1.smithi189.stdout:3/456: rename d2/d3/d2c/d30/d3f/d9a/fa0 to d2/d3/d2c/d31/d4c/fa3 0 2024-04-20T21:32:12.538 INFO:tasks.workunit.client.1.smithi189.stdout:2/438: write d1/d17/d2f/f31 [1770872,87360] 0 2024-04-20T21:32:12.540 INFO:tasks.workunit.client.0.smithi062.stdout:2/807: dread - d7/da/d15/d2f/d93/fd5 zero size 2024-04-20T21:32:12.546 INFO:tasks.workunit.client.1.smithi189.stdout:6/536: stat d8/d2b/f3a 0 2024-04-20T21:32:12.546 INFO:tasks.workunit.client.1.smithi189.stdout:8/571: dread - da/df/d22/d33/d59/fbb zero size 2024-04-20T21:32:12.548 INFO:tasks.workunit.client.0.smithi062.stdout:3/739: rename d0/d1c/d63/d42/d5b/cb8 to d0/d1/d3c/d43/d72/df2/cf4 0 2024-04-20T21:32:12.548 INFO:tasks.workunit.client.1.smithi189.stdout:2/439: rmdir d1/d11/d58/d6b 39 2024-04-20T21:32:12.551 INFO:tasks.workunit.client.1.smithi189.stdout:1/452: dread d8/f24 [0,4194304] 0 2024-04-20T21:32:12.551 INFO:tasks.workunit.client.1.smithi189.stdout:3/457: creat d2/d3/d36/da1/fa4 x:0 0 0 2024-04-20T21:32:12.551 INFO:tasks.workunit.client.0.smithi062.stdout:1/835: dread dc/fe9 [0,4194304] 0 2024-04-20T21:32:12.554 INFO:tasks.workunit.client.0.smithi062.stdout:1/836: chown dc/d15/d75/d96/dd4/f54 191 1 2024-04-20T21:32:12.555 INFO:tasks.workunit.client.1.smithi189.stdout:8/572: chown da/df/d20/d2c/f96 47 1 2024-04-20T21:32:12.556 INFO:tasks.workunit.client.0.smithi062.stdout:0/774: symlink d7/d1c/d15/d17/d40/d3c/lfc 0 2024-04-20T21:32:12.560 INFO:tasks.workunit.client.1.smithi189.stdout:3/458: write d2/d1c/d5c/f87 [872396,128015] 0 2024-04-20T21:32:12.562 INFO:tasks.workunit.client.1.smithi189.stdout:6/537: rename d8/d12/d30/d47/f98 to d8/d12/d99/fb2 0 2024-04-20T21:32:12.564 INFO:tasks.workunit.client.1.smithi189.stdout:3/459: chown d2/d1c/f7d 1154560 1 2024-04-20T21:32:12.565 INFO:tasks.workunit.client.1.smithi189.stdout:8/573: symlink da/de/d1e/d21/lca 0 2024-04-20T21:32:12.568 INFO:tasks.workunit.client.1.smithi189.stdout:3/460: chown d2/d3/d2c/d30/l38 2019764615 1 2024-04-20T21:32:12.568 INFO:tasks.workunit.client.1.smithi189.stdout:6/538: write d8/d10/d78/f80 [442210,1464] 0 2024-04-20T21:32:12.572 INFO:tasks.workunit.client.0.smithi062.stdout:2/808: creat d7/db5/f109 x:0 0 0 2024-04-20T21:32:12.572 INFO:tasks.workunit.client.0.smithi062.stdout:3/740: symlink d0/d1b/d74/dc7/lf5 0 2024-04-20T21:32:12.573 INFO:tasks.workunit.client.0.smithi062.stdout:0/775: chown d7/d1c 25043 1 2024-04-20T21:32:12.574 INFO:tasks.workunit.client.1.smithi189.stdout:6/539: fdatasync d8/d10/f20 0 2024-04-20T21:32:12.578 INFO:tasks.workunit.client.1.smithi189.stdout:8/574: rename da/df/l71 to da/de/d1e/d69/lcb 0 2024-04-20T21:32:12.579 INFO:tasks.workunit.client.0.smithi062.stdout:2/809: chown d7/da/d15/d2f/d38/d51/dbc/le4 15626 1 2024-04-20T21:32:12.579 INFO:tasks.workunit.client.1.smithi189.stdout:6/540: truncate d8/d12/d30/d47/f8a 1345896 0 2024-04-20T21:32:12.580 INFO:tasks.workunit.client.0.smithi062.stdout:4/856: dwrite da/dd/df/f3e [0,4194304] 0 2024-04-20T21:32:12.581 INFO:tasks.workunit.client.1.smithi189.stdout:1/453: dread d8/d21/d2c/f6c [0,4194304] 0 2024-04-20T21:32:12.582 INFO:tasks.workunit.client.0.smithi062.stdout:2/810: dread - d7/da/d15/d1b/d3d/ffc zero size 2024-04-20T21:32:12.583 INFO:tasks.workunit.client.0.smithi062.stdout:4/857: stat da/dd/df/d27/d3f/fc3 0 2024-04-20T21:32:12.585 INFO:tasks.workunit.client.0.smithi062.stdout:9/893: dread d16/d2c/d42/f70 [0,4194304] 0 2024-04-20T21:32:12.592 INFO:tasks.workunit.client.1.smithi189.stdout:3/461: dread d2/d1c/f7b [0,4194304] 0 2024-04-20T21:32:12.594 INFO:tasks.workunit.client.1.smithi189.stdout:1/454: chown d8/d1e/f3c 33 1 2024-04-20T21:32:12.594 INFO:tasks.workunit.client.1.smithi189.stdout:8/575: rename da/df/d3c/d68/d8b/fc6 to da/df/d20/d2c/d9a/fcc 0 2024-04-20T21:32:12.597 INFO:tasks.workunit.client.1.smithi189.stdout:5/451: dread d7/d9/d30/f2d [0,4194304] 0 2024-04-20T21:32:12.599 INFO:tasks.workunit.client.0.smithi062.stdout:4/858: dwrite da/dd/df/ff4 [0,4194304] 0 2024-04-20T21:32:12.605 INFO:tasks.workunit.client.1.smithi189.stdout:8/576: read da/fb [1334265,125319] 0 2024-04-20T21:32:12.606 INFO:tasks.workunit.client.1.smithi189.stdout:1/455: mknod d8/d1e/c9c 0 2024-04-20T21:32:12.610 INFO:tasks.workunit.client.1.smithi189.stdout:7/519: dwrite d5/d46/f4c [0,4194304] 0 2024-04-20T21:32:12.610 INFO:tasks.workunit.client.1.smithi189.stdout:6/541: creat d8/da/dd/d35/d6d/d7b/fb3 x:0 0 0 2024-04-20T21:32:12.611 INFO:tasks.workunit.client.1.smithi189.stdout:2/440: dwrite d1/d11/d52/f6a [0,4194304] 0 2024-04-20T21:32:12.617 INFO:tasks.workunit.client.1.smithi189.stdout:3/462: rename d2/f5a to d2/d3/d2c/d30/d3f/fa5 0 2024-04-20T21:32:12.622 INFO:tasks.workunit.client.1.smithi189.stdout:5/452: unlink d7/d14/d18/f77 0 2024-04-20T21:32:12.625 INFO:tasks.workunit.client.1.smithi189.stdout:2/441: symlink d1/d17/d2f/d4a/d66/l80 0 2024-04-20T21:32:12.625 INFO:tasks.workunit.client.1.smithi189.stdout:3/463: write d2/d3/d2c/d30/d3f/d9a/f9b [502291,13599] 0 2024-04-20T21:32:12.626 INFO:tasks.workunit.client.1.smithi189.stdout:6/542: creat d8/d2b/d49/d54/d6e/fb4 x:0 0 0 2024-04-20T21:32:12.627 INFO:tasks.workunit.client.0.smithi062.stdout:2/811: rmdir d7/da/d15 39 2024-04-20T21:32:12.628 INFO:tasks.workunit.client.1.smithi189.stdout:5/453: truncate d7/d35/f44 4676554 0 2024-04-20T21:32:12.633 INFO:tasks.workunit.client.1.smithi189.stdout:8/577: link da/df/d20/f7d da/df/d3c/d50/d6c/dc7/fcd 0 2024-04-20T21:32:12.633 INFO:tasks.workunit.client.0.smithi062.stdout:9/894: rmdir d16/d3d/d55 39 2024-04-20T21:32:12.633 INFO:tasks.workunit.client.0.smithi062.stdout:4/859: mknod da/dd/df/d27/d3f/d82/c12f 0 2024-04-20T21:32:12.634 INFO:tasks.workunit.client.1.smithi189.stdout:5/454: creat d7/d9/d31/f88 x:0 0 0 2024-04-20T21:32:12.637 INFO:tasks.workunit.client.0.smithi062.stdout:4/860: symlink da/dd/d15/d50/d123/l130 0 2024-04-20T21:32:12.637 INFO:tasks.workunit.client.1.smithi189.stdout:5/455: fdatasync d7/d9/d28/f7b 0 2024-04-20T21:32:12.637 INFO:tasks.workunit.client.1.smithi189.stdout:3/464: mknod d2/d3/d2c/d30/ca6 0 2024-04-20T21:32:12.639 INFO:tasks.workunit.client.1.smithi189.stdout:2/442: rename d1/d17/d2f/d4a/f79 to d1/d17/d2f/d55/f81 0 2024-04-20T21:32:12.640 INFO:tasks.workunit.client.1.smithi189.stdout:5/456: chown d7/ca 22 1 2024-04-20T21:32:12.641 INFO:tasks.workunit.client.0.smithi062.stdout:3/741: rmdir d0/d1c/d63/d42 39 2024-04-20T21:32:12.642 INFO:tasks.workunit.client.1.smithi189.stdout:1/456: dwrite d8/d1e/f66 [0,4194304] 0 2024-04-20T21:32:12.642 INFO:tasks.workunit.client.1.smithi189.stdout:8/578: getdents da/d81/d91/da8 0 2024-04-20T21:32:12.644 INFO:tasks.workunit.client.0.smithi062.stdout:2/812: creat d7/da/d15/d1b/d24/dee/f10a x:0 0 0 2024-04-20T21:32:12.645 INFO:tasks.workunit.client.1.smithi189.stdout:2/443: dread - d1/d11/d2d/f63 zero size 2024-04-20T21:32:12.645 INFO:tasks.workunit.client.1.smithi189.stdout:3/465: creat d2/d3/d2c/d31/fa7 x:0 0 0 2024-04-20T21:32:12.646 INFO:tasks.workunit.client.0.smithi062.stdout:4/861: mkdir da/dd/d15/d4b/ddb/d131 0 2024-04-20T21:32:12.646 INFO:tasks.workunit.client.1.smithi189.stdout:5/457: write d7/d14/d4e/d5b/f68 [3842671,122824] 0 2024-04-20T21:32:12.648 INFO:tasks.workunit.client.1.smithi189.stdout:4/463: sync 2024-04-20T21:32:12.649 INFO:tasks.workunit.client.1.smithi189.stdout:9/480: sync 2024-04-20T21:32:12.652 INFO:tasks.workunit.client.0.smithi062.stdout:2/813: write d7/da/f3c [2835605,3360] 0 2024-04-20T21:32:12.652 INFO:tasks.workunit.client.0.smithi062.stdout:4/862: fdatasync da/dd/df/d27/d3f/f6e 0 2024-04-20T21:32:12.652 INFO:tasks.workunit.client.1.smithi189.stdout:1/457: creat d8/d1e/f9d x:0 0 0 2024-04-20T21:32:12.655 INFO:tasks.workunit.client.1.smithi189.stdout:5/458: mknod d7/d9/d51/c89 0 2024-04-20T21:32:12.656 INFO:tasks.workunit.client.0.smithi062.stdout:2/814: read - d7/da/d15/d1b/d3d/d84/f100 zero size 2024-04-20T21:32:12.659 INFO:tasks.workunit.client.0.smithi062.stdout:4/863: readlink da/dd/df/d27/d3f/le1 0 2024-04-20T21:32:12.659 INFO:tasks.workunit.client.1.smithi189.stdout:9/481: truncate d9/dc/d17/d36/f78 1475560 0 2024-04-20T21:32:12.659 INFO:tasks.workunit.client.1.smithi189.stdout:3/466: symlink d2/d3/d25/la8 0 2024-04-20T21:32:12.660 INFO:tasks.workunit.client.0.smithi062.stdout:3/742: mkdir d0/d1c/d63/d42/dc8/df6 0 2024-04-20T21:32:12.662 INFO:tasks.workunit.client.0.smithi062.stdout:3/743: chown d0/d1/d3c/d43/laf 129365096 1 2024-04-20T21:32:12.665 INFO:tasks.workunit.client.1.smithi189.stdout:4/464: mknod d2/d3a/cac 0 2024-04-20T21:32:12.667 INFO:tasks.workunit.client.1.smithi189.stdout:5/459: mkdir d7/d9/d51/d70/d8a 0 2024-04-20T21:32:12.670 INFO:tasks.workunit.client.1.smithi189.stdout:9/482: mkdir d9/dc/d17/d95 0 2024-04-20T21:32:12.672 INFO:tasks.workunit.client.1.smithi189.stdout:6/543: write d8/d2b/d49/d54/d67/f9c [219856,57705] 0 2024-04-20T21:32:12.674 INFO:tasks.workunit.client.1.smithi189.stdout:3/467: write d2/fd [2930787,68801] 0 2024-04-20T21:32:12.674 INFO:tasks.workunit.client.1.smithi189.stdout:4/465: creat d2/d3a/fad x:0 0 0 2024-04-20T21:32:12.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:12.677+0000 7f0174000700 1 -- 172.21.15.62:0/2859054693 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f016c103680 msgr2=0x7f016c105a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:32:12.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:12.677+0000 7f0174000700 1 --2- 172.21.15.62:0/2859054693 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f016c103680 0x7f016c105a60 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f0158009910 tx=0x7f0158009c20 comp rx=0 tx=0).stop 2024-04-20T21:32:12.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:12.677+0000 7f0174000700 1 -- 172.21.15.62:0/2859054693 shutdown_connections 2024-04-20T21:32:12.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:12.677+0000 7f0174000700 1 --2- 172.21.15.62:0/2859054693 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f016c103680 0x7f016c105a60 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:32:12.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:12.677+0000 7f0174000700 1 --2- 172.21.15.62:0/2859054693 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f016c100d60 0x7f016c103140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:32:12.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:12.677+0000 7f0174000700 1 -- 172.21.15.62:0/2859054693 >> 172.21.15.62:0/2859054693 conn(0x7f016c0fa780 msgr2=0x7f016c0fcbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:32:12.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:12.677+0000 7f0174000700 1 -- 172.21.15.62:0/2859054693 shutdown_connections 2024-04-20T21:32:12.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:12.677+0000 7f0174000700 1 -- 172.21.15.62:0/2859054693 wait complete. 2024-04-20T21:32:12.677 INFO:tasks.workunit.client.0.smithi062.stdout:3/744: read d0/d1/f77 [1006180,32949] 0 2024-04-20T21:32:12.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:12.678+0000 7f0174000700 1 Processor -- start 2024-04-20T21:32:12.678 INFO:tasks.workunit.client.1.smithi189.stdout:7/520: dread d5/f43 [0,4194304] 0 2024-04-20T21:32:12.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:12.679+0000 7f0174000700 1 -- start start 2024-04-20T21:32:12.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:12.679+0000 7f0174000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f016c100d60 0x7f016c197cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:32:12.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:12.679+0000 7f0174000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f016c103680 0x7f016c198230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:32:12.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:12.679+0000 7f0174000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f016c198850 con 0x7f016c103680 2024-04-20T21:32:12.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:12.679+0000 7f0174000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f016c19d260 con 0x7f016c100d60 2024-04-20T21:32:12.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:12.679+0000 7f0170c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f016c103680 0x7f016c198230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:32:12.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:12.679+0000 7f0170c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f016c103680 0x7f016c198230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:57392/0 (socket says 172.21.15.62:57392) 2024-04-20T21:32:12.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:12.679+0000 7f0170c00700 1 -- 172.21.15.62:0/4222545926 learned_addr learned my addr 172.21.15.62:0/4222545926 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:32:12.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:12.680+0000 7f0171600700 1 --2- 172.21.15.62:0/4222545926 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f016c100d60 0x7f016c197cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:32:12.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:12.681+0000 7f0171600700 1 -- 172.21.15.62:0/4222545926 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f016c103680 msgr2=0x7f016c198230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:32:12.680 INFO:tasks.workunit.client.1.smithi189.stdout:4/466: readlink d2/dd/l2e 0 2024-04-20T21:32:12.682 INFO:tasks.workunit.client.1.smithi189.stdout:1/458: dread d8/f40 [0,4194304] 0 2024-04-20T21:32:12.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:12.683+0000 7f0171600700 1 --2- 172.21.15.62:0/4222545926 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f016c103680 0x7f016c198230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:32:12.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:12.683+0000 7f0171600700 1 -- 172.21.15.62:0/4222545926 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01580095f0 con 0x7f016c100d60 2024-04-20T21:32:12.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:12.683+0000 7f0170c00700 1 --2- 172.21.15.62:0/4222545926 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f016c103680 0x7f016c198230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-20T21:32:12.683 INFO:tasks.workunit.client.1.smithi189.stdout:5/460: rmdir d7/d9/d30/d55/d33 39 2024-04-20T21:32:12.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:12.684+0000 7f0171600700 1 --2- 172.21.15.62:0/4222545926 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f016c100d60 0x7f016c197cf0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f016000e940 tx=0x7f016000ec50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:32:12.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:12.685+0000 7f016a000700 1 -- 172.21.15.62:0/4222545926 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f016000c990 con 0x7f016c100d60 2024-04-20T21:32:12.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:12.685+0000 7f016a000700 1 -- 172.21.15.62:0/4222545926 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f0160010430 con 0x7f016c100d60 2024-04-20T21:32:12.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:12.685+0000 7f016a000700 1 -- 172.21.15.62:0/4222545926 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0160018630 con 0x7f016c100d60 2024-04-20T21:32:12.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:12.685+0000 7f0174000700 1 -- 172.21.15.62:0/4222545926 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f016c06a830 con 0x7f016c100d60 2024-04-20T21:32:12.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:12.686+0000 7f0174000700 1 -- 172.21.15.62:0/4222545926 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f016c06ac40 con 0x7f016c100d60 2024-04-20T21:32:12.686 INFO:tasks.workunit.client.0.smithi062.stdout:3/745: fsync d0/d1c/d63/fb2 0 2024-04-20T21:32:12.686 INFO:tasks.workunit.client.1.smithi189.stdout:3/468: read - d2/d3/d2c/d31/d90/d92/f9c zero size 2024-04-20T21:32:12.686 INFO:tasks.workunit.client.1.smithi189.stdout:8/579: dread da/de/d1e/d21/f67 [0,4194304] 0 2024-04-20T21:32:12.686 INFO:tasks.workunit.client.1.smithi189.stdout:6/544: creat d8/d12/daf/fb5 x:0 0 0 2024-04-20T21:32:12.689 INFO:tasks.workunit.client.0.smithi062.stdout:8/795: sync 2024-04-20T21:32:12.689 INFO:tasks.workunit.client.0.smithi062.stdout:5/790: sync 2024-04-20T21:32:12.689 INFO:tasks.workunit.client.0.smithi062.stdout:2/815: write d7/d39/fc9 [282232,38689] 0 2024-04-20T21:32:12.689 INFO:tasks.workunit.client.0.smithi062.stdout:7/758: sync 2024-04-20T21:32:12.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:12.688+0000 7f0174000700 1 -- 172.21.15.62:0/4222545926 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f016c0fc340 con 0x7f016c100d60 2024-04-20T21:32:12.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:12.691+0000 7f016a000700 1 -- 172.21.15.62:0/4222545926 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 29) v1 ==== 92700+0+0 (secure 0 0 0) 0x7f01600105a0 con 0x7f016c100d60 2024-04-20T21:32:12.690 INFO:tasks.workunit.client.1.smithi189.stdout:9/483: creat d9/dc/d17/d7e/f96 x:0 0 0 2024-04-20T21:32:12.691 INFO:tasks.workunit.client.1.smithi189.stdout:4/467: truncate d2/dd/d11/d14/f50 1660046 0 2024-04-20T21:32:12.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:12.692+0000 7f016a000700 1 --2- 172.21.15.62:0/4222545926 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f015406eac0 0x7f0154070f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:32:12.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:12.692+0000 7f016a000700 1 -- 172.21.15.62:0/4222545926 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f0160014070 con 0x7f016c100d60 2024-04-20T21:32:12.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:12.692+0000 7f0170c00700 1 --2- 172.21.15.62:0/4222545926 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f015406eac0 0x7f0154070f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:32:12.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:12.693+0000 7f0170c00700 1 --2- 172.21.15.62:0/4222545926 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f015406eac0 0x7f0154070f70 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0158006010 tx=0x7f0158005a90 comp rx=0 tx=0).ready entity=mgr.14658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:32:12.695 INFO:tasks.workunit.client.0.smithi062.stdout:7/759: chown d7/f18 8235 1 2024-04-20T21:32:12.695 INFO:tasks.workunit.client.1.smithi189.stdout:2/444: dwrite d1/d11/d58/f6e [0,4194304] 0 2024-04-20T21:32:12.696 INFO:tasks.workunit.client.1.smithi189.stdout:0/597: sync 2024-04-20T21:32:12.699 INFO:tasks.workunit.client.0.smithi062.stdout:2/816: mknod d7/da/d15/d1b/d24/dee/c10b 0 2024-04-20T21:32:12.699 INFO:tasks.workunit.client.0.smithi062.stdout:1/837: dwrite dc/d15/d20/d28/d83/f10c [4194304,4194304] 0 2024-04-20T21:32:12.701 INFO:tasks.workunit.client.0.smithi062.stdout:0/776: dwrite d7/d1c/d15/d17/d40/d79/fb0 [4194304,4194304] 0 2024-04-20T21:32:12.702 INFO:tasks.workunit.client.1.smithi189.stdout:4/468: chown d2/f8 17274054 1 2024-04-20T21:32:12.704 INFO:tasks.workunit.client.1.smithi189.stdout:9/484: write d9/d12/d15/d1a/f27 [716365,96730] 0 2024-04-20T21:32:12.704 INFO:tasks.workunit.client.1.smithi189.stdout:0/598: fdatasync d3/df/d28/d2b/f58 0 2024-04-20T21:32:12.706 INFO:tasks.workunit.client.1.smithi189.stdout:8/580: write da/df/d3c/d50/f66 [240548,36546] 0 2024-04-20T21:32:12.707 INFO:tasks.workunit.client.1.smithi189.stdout:3/469: mkdir d2/d3/d2c/d30/d3f/d9a/da9 0 2024-04-20T21:32:12.708 INFO:tasks.workunit.client.0.smithi062.stdout:2/817: chown d7/da/d15/c20 25 1 2024-04-20T21:32:12.710 INFO:tasks.workunit.client.0.smithi062.stdout:4/864: dwrite da/dd/d15/daf/fda [0,4194304] 0 2024-04-20T21:32:12.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:12.711+0000 7f016a000700 1 -- 172.21.15.62:0/4222545926 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187527 (secure 0 0 0) 0x7f0160057f80 con 0x7f016c100d60 2024-04-20T21:32:12.715 INFO:tasks.workunit.client.1.smithi189.stdout:4/469: stat d2/dd/d11/d14/d69/d6a/d7a/f8c 0 2024-04-20T21:32:12.722 INFO:tasks.workunit.client.1.smithi189.stdout:7/521: getdents d5/db4/d71/d7a 0 2024-04-20T21:32:12.724 INFO:tasks.workunit.client.1.smithi189.stdout:4/470: write d2/d3a/f3c [9082638,33543] 0 2024-04-20T21:32:12.725 INFO:tasks.workunit.client.1.smithi189.stdout:0/599: rmdir d3/df/d5d/d64 39 2024-04-20T21:32:12.725 INFO:tasks.workunit.client.1.smithi189.stdout:5/461: dwrite d7/f78 [0,4194304] 0 2024-04-20T21:32:12.729 INFO:tasks.workunit.client.1.smithi189.stdout:3/470: mknod d2/d3/d2c/d31/caa 0 2024-04-20T21:32:12.729 INFO:tasks.workunit.client.1.smithi189.stdout:0/600: stat d3/df/d28/d2b/d2c/d73/cb7 0 2024-04-20T21:32:12.729 INFO:tasks.workunit.client.1.smithi189.stdout:7/522: write d5/d21/d4a/f8d [1464571,67558] 0 2024-04-20T21:32:12.730 INFO:tasks.workunit.client.1.smithi189.stdout:9/485: creat d9/dc/d17/d36/d5d/f97 x:0 0 0 2024-04-20T21:32:12.731 INFO:tasks.workunit.client.1.smithi189.stdout:2/445: dwrite d1/d11/f1a [4194304,4194304] 0 2024-04-20T21:32:12.733 INFO:tasks.workunit.client.1.smithi189.stdout:4/471: write d2/dd/d11/d14/d69/d6a/d7a/f8c [2761414,82503] 0 2024-04-20T21:32:12.735 INFO:tasks.workunit.client.1.smithi189.stdout:5/462: creat d7/d14/f8b x:0 0 0 2024-04-20T21:32:12.738 INFO:tasks.workunit.client.1.smithi189.stdout:9/486: rmdir d9/dc/d17/d2b 39 2024-04-20T21:32:12.738 INFO:tasks.workunit.client.1.smithi189.stdout:5/463: fdatasync d7/d9/f42 0 2024-04-20T21:32:12.739 INFO:tasks.workunit.client.1.smithi189.stdout:7/523: stat d5/d21/d4a/d52/d98/l89 0 2024-04-20T21:32:12.739 INFO:tasks.workunit.client.1.smithi189.stdout:3/471: readlink d2/d3/d2c/d31/l70 0 2024-04-20T21:32:12.747 INFO:tasks.workunit.client.1.smithi189.stdout:3/472: rmdir d2/d8d 39 2024-04-20T21:32:12.747 INFO:tasks.workunit.client.1.smithi189.stdout:7/524: chown d5/db4/d71/d7a 31 1 2024-04-20T21:32:12.749 INFO:tasks.workunit.client.1.smithi189.stdout:5/464: read - d7/d9/d28/f7b zero size 2024-04-20T21:32:12.749 INFO:tasks.workunit.client.1.smithi189.stdout:0/601: rename d3/df/d28/ca2 to d3/df/cca 0 2024-04-20T21:32:12.752 INFO:tasks.workunit.client.1.smithi189.stdout:1/459: dread d8/d1e/f3c [0,4194304] 0 2024-04-20T21:32:12.752 INFO:tasks.workunit.client.1.smithi189.stdout:4/472: mkdir d2/dd/d11/dae 0 2024-04-20T21:32:12.753 INFO:tasks.workunit.client.1.smithi189.stdout:9/487: readlink d9/dc/d17/d2b/l42 0 2024-04-20T21:32:12.757 INFO:tasks.workunit.client.1.smithi189.stdout:8/581: write da/f89 [802132,91297] 0 2024-04-20T21:32:12.759 INFO:tasks.workunit.client.1.smithi189.stdout:2/446: symlink d1/d17/d2f/d4a/l82 0 2024-04-20T21:32:12.761 INFO:tasks.workunit.client.1.smithi189.stdout:5/465: symlink d7/d14/d4e/d5b/l8c 0 2024-04-20T21:32:12.765 INFO:tasks.workunit.client.1.smithi189.stdout:4/473: creat d2/d20/faf x:0 0 0 2024-04-20T21:32:12.766 INFO:tasks.workunit.client.1.smithi189.stdout:0/602: symlink d3/df/lcb 0 2024-04-20T21:32:12.768 INFO:tasks.workunit.client.1.smithi189.stdout:8/582: mkdir da/df/d20/d2c/d9a/dce 0 2024-04-20T21:32:12.770 INFO:tasks.workunit.client.1.smithi189.stdout:1/460: fsync d8/f11 0 2024-04-20T21:32:12.771 INFO:tasks.workunit.client.1.smithi189.stdout:5/466: creat d7/d9/d30/d55/d61/f8d x:0 0 0 2024-04-20T21:32:12.772 INFO:tasks.workunit.client.1.smithi189.stdout:3/473: link d2/c53 d2/d3/d2c/d30/d3f/cab 0 2024-04-20T21:32:12.773 INFO:tasks.workunit.client.1.smithi189.stdout:4/474: chown d2/dd/d37/l7b 11217 1 2024-04-20T21:32:12.775 INFO:tasks.workunit.client.1.smithi189.stdout:8/583: creat da/df/d3c/d68/d8b/fcf x:0 0 0 2024-04-20T21:32:12.775 INFO:tasks.workunit.client.1.smithi189.stdout:0/603: symlink d3/df/d5d/lcc 0 2024-04-20T21:32:12.776 INFO:tasks.workunit.client.1.smithi189.stdout:5/467: rmdir d7/d14/d4e/d5b 39 2024-04-20T21:32:12.776 INFO:tasks.workunit.client.1.smithi189.stdout:3/474: chown d2/d1c/d5c/c7e 461335 1 2024-04-20T21:32:12.777 INFO:tasks.workunit.client.1.smithi189.stdout:4/475: symlink d2/d64/lb0 0 2024-04-20T21:32:12.777 INFO:tasks.workunit.client.1.smithi189.stdout:8/584: readlink da/de/d1e/d69/la2 0 2024-04-20T21:32:12.778 INFO:tasks.workunit.client.1.smithi189.stdout:2/447: write d1/d11/f15 [4415463,102033] 0 2024-04-20T21:32:12.781 INFO:tasks.workunit.client.1.smithi189.stdout:5/468: dread - d7/d9/f62 zero size 2024-04-20T21:32:12.781 INFO:tasks.workunit.client.1.smithi189.stdout:3/475: write d2/d8d/f91 [862525,35503] 0 2024-04-20T21:32:12.782 INFO:tasks.workunit.client.1.smithi189.stdout:0/604: creat d3/df/d28/d2b/d2c/d73/fcd x:0 0 0 2024-04-20T21:32:12.783 INFO:tasks.workunit.client.1.smithi189.stdout:7/525: dwrite d5/d46/f65 [0,4194304] 0 2024-04-20T21:32:12.784 INFO:tasks.workunit.client.1.smithi189.stdout:4/476: mknod d2/d22/d57/cb1 0 2024-04-20T21:32:12.784 INFO:tasks.workunit.client.1.smithi189.stdout:2/448: dread - d1/d17/d2f/d4a/f7b zero size 2024-04-20T21:32:12.787 INFO:tasks.workunit.client.1.smithi189.stdout:7/526: dread - d5/d46/f8f zero size 2024-04-20T21:32:12.788 INFO:tasks.workunit.client.1.smithi189.stdout:2/449: chown d1/d11/l40 12766399 1 2024-04-20T21:32:12.788 INFO:tasks.workunit.client.0.smithi062.stdout:0/777: mkdir d7/d1c/d15/d17/dfd 0 2024-04-20T21:32:12.793 INFO:tasks.workunit.client.1.smithi189.stdout:7/527: fsync d5/d46/d7d/d45/fa2 0 2024-04-20T21:32:12.796 INFO:tasks.workunit.client.1.smithi189.stdout:4/477: mknod d2/dd/d11/d14/cb2 0 2024-04-20T21:32:12.797 INFO:tasks.workunit.client.1.smithi189.stdout:1/461: dwrite d8/df/d50/f99 [0,4194304] 0 2024-04-20T21:32:12.798 INFO:tasks.workunit.client.0.smithi062.stdout:1/838: getdents dc/d15/df6 0 2024-04-20T21:32:12.801 INFO:tasks.workunit.client.1.smithi189.stdout:0/605: unlink d3/df/d30/d45/c9d 0 2024-04-20T21:32:12.802 INFO:tasks.workunit.client.0.smithi062.stdout:8/796: dwrite d7/d25/d3b/d58/d8b/d24/d46/ff5 [0,4194304] 0 2024-04-20T21:32:12.802 INFO:tasks.workunit.client.0.smithi062.stdout:7/760: dwrite d7/d3e/d33/d3d/d4d/d83/d95/faa [4194304,4194304] 0 2024-04-20T21:32:12.803 INFO:tasks.workunit.client.0.smithi062.stdout:0/778: rename d7/d1c/d15/d6b to d7/d1c/d15/d17/d37/dfb/dfe 0 2024-04-20T21:32:12.804 INFO:tasks.workunit.client.1.smithi189.stdout:5/469: creat d7/d9/d30/f8e x:0 0 0 2024-04-20T21:32:12.804 INFO:tasks.workunit.client.1.smithi189.stdout:8/585: link da/d1f/d2a/l2d da/d1f/d2a/ld0 0 2024-04-20T21:32:12.807 INFO:tasks.workunit.client.1.smithi189.stdout:3/476: rename d2/d3/d36/l3d to d2/d3/d2c/d31/d90/d92/lac 0 2024-04-20T21:32:12.807 INFO:tasks.workunit.client.0.smithi062.stdout:0/779: dread - d7/d1c/d15/d17/d1b/d9d/fba zero size 2024-04-20T21:32:12.808 INFO:tasks.workunit.client.0.smithi062.stdout:1/839: mkdir dc/d70/d131 0 2024-04-20T21:32:12.809 INFO:tasks.workunit.client.1.smithi189.stdout:1/462: mknod d8/df/d3a/d49/d8c/c9e 0 2024-04-20T21:32:12.809 INFO:tasks.workunit.client.1.smithi189.stdout:5/470: dread - d7/d9/d51/f72 zero size 2024-04-20T21:32:12.810 INFO:tasks.workunit.client.1.smithi189.stdout:4/478: creat d2/dd/d11/d14/d69/fb3 x:0 0 0 2024-04-20T21:32:12.811 INFO:tasks.workunit.client.0.smithi062.stdout:2/818: getdents d7/da/d28/d4b 0 2024-04-20T21:32:12.812 INFO:tasks.workunit.client.1.smithi189.stdout:0/606: chown d3/df/d30/d45/d90/fc9 2635926 1 2024-04-20T21:32:12.814 INFO:tasks.workunit.client.1.smithi189.stdout:8/586: write da/df/d20/d27/f54 [4038140,45492] 0 2024-04-20T21:32:12.814 INFO:tasks.workunit.client.1.smithi189.stdout:1/463: fdatasync d8/df/f83 0 2024-04-20T21:32:12.815 INFO:tasks.workunit.client.0.smithi062.stdout:7/761: mkdir d7/d3e/d33/d3d/d4d/d54/d4f/de9/df1/d105 0 2024-04-20T21:32:12.819 INFO:tasks.workunit.client.1.smithi189.stdout:2/450: dwrite d1/d11/f1a [0,4194304] 0 2024-04-20T21:32:12.822 INFO:tasks.workunit.client.1.smithi189.stdout:7/528: rename d5/d1d/c22 to d5/d21/d4a/d52/cb6 0 2024-04-20T21:32:12.822 INFO:tasks.workunit.client.1.smithi189.stdout:3/477: creat d2/d3/d2c/d30/fad x:0 0 0 2024-04-20T21:32:12.824 INFO:tasks.workunit.client.0.smithi062.stdout:2/819: symlink d7/da/d15/d1b/d24/dee/l10c 0 2024-04-20T21:32:12.824 INFO:tasks.workunit.client.0.smithi062.stdout:1/840: getdents dc/d15/df6 0 2024-04-20T21:32:12.825 INFO:tasks.workunit.client.0.smithi062.stdout:7/762: symlink d7/d3e/d33/d3d/d4d/d83/d95/l106 0 2024-04-20T21:32:12.826 INFO:tasks.workunit.client.1.smithi189.stdout:0/607: mknod d3/cce 0 2024-04-20T21:32:12.828 INFO:tasks.workunit.client.0.smithi062.stdout:8/797: rename d7/d25/c55 to d7/d9/c108 0 2024-04-20T21:32:12.831 INFO:tasks.workunit.client.0.smithi062.stdout:7/763: mknod d7/d63/c107 0 2024-04-20T21:32:12.831 INFO:tasks.workunit.client.0.smithi062.stdout:2/820: rmdir d7/da/d15/d2f/d38/d51/dbc 39 2024-04-20T21:32:12.834 INFO:tasks.workunit.client.0.smithi062.stdout:2/821: write d7/f5d [4895224,23781] 0 2024-04-20T21:32:12.834 INFO:tasks.workunit.client.0.smithi062.stdout:4/865: write da/dd/f19 [398274,9838] 0 2024-04-20T21:32:12.834 INFO:tasks.workunit.client.1.smithi189.stdout:7/529: write d5/d46/fa9 [960002,54633] 0 2024-04-20T21:32:12.835 INFO:tasks.workunit.client.1.smithi189.stdout:5/471: rename d7/d14/d4e/d5b/c52 to d7/d14/c8f 0 2024-04-20T21:32:12.838 INFO:tasks.workunit.client.0.smithi062.stdout:7/764: read d7/d3e/d33/d3d/d5e/f102 [2891576,56033] 0 2024-04-20T21:32:12.839 INFO:tasks.workunit.client.1.smithi189.stdout:7/530: chown d5/d46/d7d/d45/f99 0 1 2024-04-20T21:32:12.841 INFO:tasks.workunit.client.1.smithi189.stdout:2/451: creat d1/d17/d2f/d55/f83 x:0 0 0 2024-04-20T21:32:12.842 INFO:tasks.workunit.client.1.smithi189.stdout:3/478: fsync d2/d3/d25/f2e 0 2024-04-20T21:32:12.844 INFO:tasks.workunit.client.1.smithi189.stdout:7/531: chown d5/d46/l9e 5514710 1 2024-04-20T21:32:12.845 INFO:tasks.workunit.client.0.smithi062.stdout:2/822: symlink d7/da/d28/l10d 0 2024-04-20T21:32:12.845 INFO:tasks.workunit.client.1.smithi189.stdout:3/479: fsync d2/d1c/f76 0 2024-04-20T21:32:12.850 INFO:tasks.workunit.client.0.smithi062.stdout:2/823: readlink d7/da/d15/d1b/d24/l75 0 2024-04-20T21:32:12.850 INFO:tasks.workunit.client.0.smithi062.stdout:1/841: rename dc/d70/d131 to dc/d15/d20/d28/d83/dc0/d132 0 2024-04-20T21:32:12.851 INFO:tasks.workunit.client.0.smithi062.stdout:8/798: write d7/d25/d3b/d58/dac/fcd [300338,41183] 0 2024-04-20T21:32:12.851 INFO:tasks.workunit.client.1.smithi189.stdout:0/608: unlink d3/df/d30/d37/f31 0 2024-04-20T21:32:12.852 INFO:tasks.workunit.client.1.smithi189.stdout:8/587: creat da/de/d1e/d21/d87/fd1 x:0 0 0 2024-04-20T21:32:12.852 INFO:tasks.workunit.client.1.smithi189.stdout:3/480: dread - d2/f81 zero size 2024-04-20T21:32:12.853 INFO:tasks.workunit.client.1.smithi189.stdout:1/464: rename d8/f41 to d8/d21/d2c/f9f 0 2024-04-20T21:32:12.857 INFO:tasks.workunit.client.1.smithi189.stdout:2/452: unlink d1/d17/l65 0 2024-04-20T21:32:12.857 INFO:tasks.workunit.client.1.smithi189.stdout:4/479: dwrite d2/d7f/f8f [0,4194304] 0 2024-04-20T21:32:12.858 INFO:tasks.workunit.client.1.smithi189.stdout:0/609: chown d3/df/d30/d37/dbd 15603 1 2024-04-20T21:32:12.858 INFO:tasks.workunit.client.0.smithi062.stdout:4/866: dread da/dd/d15/d50/d64/d3d/f5e [0,4194304] 0 2024-04-20T21:32:12.858 INFO:tasks.workunit.client.0.smithi062.stdout:2/824: write d7/da/d15/d2f/d97/f9e [1037380,55485] 0 2024-04-20T21:32:12.864 INFO:tasks.workunit.client.0.smithi062.stdout:8/799: fdatasync d7/d25/d3b/d58/d8b/f4e 0 2024-04-20T21:32:12.865 INFO:tasks.workunit.client.0.smithi062.stdout:4/867: stat da/dd/d15/d50/d64/d57/d11e/c11b 0 2024-04-20T21:32:12.869 INFO:tasks.workunit.client.0.smithi062.stdout:7/765: dwrite d7/d3e/d33/d3d/d4d/d83/d95/fc3 [0,4194304] 0 2024-04-20T21:32:12.873 INFO:tasks.workunit.client.0.smithi062.stdout:1/842: rename dc/d27/f62 to dc/d15/d20/d28/d83/f133 0 2024-04-20T21:32:12.875 INFO:tasks.workunit.client.1.smithi189.stdout:3/481: dwrite d2/f6 [4194304,4194304] 0 2024-04-20T21:32:12.877 INFO:tasks.workunit.client.0.smithi062.stdout:2/825: rmdir d7/da/d15/d1b/d3d/de5 0 2024-04-20T21:32:12.879 INFO:tasks.workunit.client.1.smithi189.stdout:2/453: dwrite d1/d17/f35 [4194304,4194304] 0 2024-04-20T21:32:12.879 INFO:tasks.workunit.client.1.smithi189.stdout:4/480: dwrite d2/dd/d11/f92 [0,4194304] 0 2024-04-20T21:32:12.881 INFO:tasks.workunit.client.1.smithi189.stdout:2/454: chown d1 467775474 1 2024-04-20T21:32:12.885 INFO:tasks.workunit.client.0.smithi062.stdout:2/826: mkdir d7/da/d15/d1b/df1/d10e 0 2024-04-20T21:32:12.893 INFO:tasks.workunit.client.1.smithi189.stdout:3/482: dread d2/d3/d2c/d31/f6f [0,4194304] 0 2024-04-20T21:32:12.894 INFO:tasks.workunit.client.0.smithi062.stdout:4/868: link da/dd/d15/d50/d64/d57/d5b/d9d/dea/l4e da/l132 0 2024-04-20T21:32:12.894 INFO:tasks.workunit.client.0.smithi062.stdout:7/766: creat d7/d3e/d33/d3d/d5e/df7/f108 x:0 0 0 2024-04-20T21:32:12.894 INFO:tasks.workunit.client.1.smithi189.stdout:0/610: mknod d3/df/d28/d2b/d2c/ccf 0 2024-04-20T21:32:12.896 INFO:tasks.workunit.client.1.smithi189.stdout:5/472: creat d7/d9/d30/d55/d61/d87/f90 x:0 0 0 2024-04-20T21:32:12.898 INFO:tasks.workunit.client.1.smithi189.stdout:4/481: creat d2/d7f/fb4 x:0 0 0 2024-04-20T21:32:12.899 INFO:tasks.workunit.client.0.smithi062.stdout:1/843: creat dc/d15/d75/d96/dd4/d3a/f134 x:0 0 0 2024-04-20T21:32:12.901 INFO:tasks.workunit.client.0.smithi062.stdout:4/869: chown da/dd/df/d27/d7a/d91/lf2 4225 1 2024-04-20T21:32:12.904 INFO:tasks.workunit.client.1.smithi189.stdout:5/473: fdatasync d7/d9/d28/f6f 0 2024-04-20T21:32:12.904 INFO:tasks.workunit.client.1.smithi189.stdout:2/455: link d1/d11/d2d/f50 d1/d17/d2f/d4a/f84 0 2024-04-20T21:32:12.905 INFO:tasks.workunit.client.0.smithi062.stdout:4/870: chown da/dd/d15/d50/d64/cd0 3 1 2024-04-20T21:32:12.905 INFO:tasks.workunit.client.0.smithi062.stdout:1/844: read dc/fe [3584179,87615] 0 2024-04-20T21:32:12.909 INFO:tasks.workunit.client.1.smithi189.stdout:1/465: rename d8/df/d3a/d49/c85 to d8/df/d50/ca0 0 2024-04-20T21:32:12.911 INFO:tasks.workunit.client.0.smithi062.stdout:2/827: dwrite d7/da/d15/d2f/f7e [0,4194304] 0 2024-04-20T21:32:12.916 INFO:tasks.workunit.client.0.smithi062.stdout:1/845: symlink dc/d15/df6/l135 0 2024-04-20T21:32:12.917 INFO:tasks.workunit.client.1.smithi189.stdout:7/532: dwrite d5/d46/f8b [0,4194304] 0 2024-04-20T21:32:12.917 INFO:tasks.workunit.client.1.smithi189.stdout:5/474: mknod d7/d9/c91 0 2024-04-20T21:32:12.918 INFO:tasks.workunit.client.0.smithi062.stdout:7/767: link d7/d3e/f6a d7/d3e/d33/d3d/d7f/f109 0 2024-04-20T21:32:12.918 INFO:tasks.workunit.client.0.smithi062.stdout:2/828: readlink d7/da/d28/d107/dca/dd8/le7 0 2024-04-20T21:32:12.920 INFO:tasks.workunit.client.1.smithi189.stdout:2/456: mknod d1/d11/d58/d6b/c85 0 2024-04-20T21:32:12.921 INFO:tasks.workunit.client.1.smithi189.stdout:4/482: dwrite d2/fb [0,4194304] 0 2024-04-20T21:32:12.923 INFO:tasks.workunit.client.1.smithi189.stdout:7/533: chown d5/db4/d71/fa4 2 1 2024-04-20T21:32:12.925 INFO:tasks.workunit.client.1.smithi189.stdout:1/466: symlink d8/d9b/la1 0 2024-04-20T21:32:12.925 INFO:tasks.workunit.client.0.smithi062.stdout:4/871: mkdir da/dd/d15/d50/d64/d57/d5b/d9d/dea/dac/d133 0 2024-04-20T21:32:12.928 INFO:tasks.workunit.client.0.smithi062.stdout:7/768: mkdir d7/d26/dac/d10a 0 2024-04-20T21:32:12.928 INFO:tasks.workunit.client.1.smithi189.stdout:1/467: stat d8/f1a 0 2024-04-20T21:32:12.928 INFO:tasks.workunit.client.0.smithi062.stdout:1/846: mknod dc/d15/df6/c136 0 2024-04-20T21:32:12.929 INFO:tasks.workunit.client.1.smithi189.stdout:3/483: dwrite d2/d3/f48 [4194304,4194304] 0 2024-04-20T21:32:12.930 INFO:tasks.workunit.client.1.smithi189.stdout:2/457: mknod d1/d11/d2d/d4f/c86 0 2024-04-20T21:32:12.930 INFO:tasks.workunit.client.1.smithi189.stdout:5/475: fdatasync d7/d14/d18/f17 0 2024-04-20T21:32:12.931 INFO:tasks.workunit.client.1.smithi189.stdout:7/534: stat d5/db4/f38 0 2024-04-20T21:32:12.932 INFO:tasks.workunit.client.0.smithi062.stdout:8/800: dwrite d7/d25/d76/f84 [0,4194304] 0 2024-04-20T21:32:12.935 INFO:tasks.workunit.client.1.smithi189.stdout:4/483: unlink d2/dd/d11/d14/l60 0 2024-04-20T21:32:12.937 INFO:tasks.workunit.client.1.smithi189.stdout:5/476: fsync d7/d35/f46 0 2024-04-20T21:32:12.939 INFO:tasks.workunit.client.1.smithi189.stdout:1/468: truncate d8/d21/f38 785086 0 2024-04-20T21:32:12.943 INFO:tasks.workunit.client.1.smithi189.stdout:0/611: dwrite d3/df/d30/d45/d60/f44 [4194304,4194304] 0 2024-04-20T21:32:12.944 INFO:tasks.workunit.client.0.smithi062.stdout:2/829: rename d7/da/d28 to d7/da/d15/d2f/d38/d10f 0 2024-04-20T21:32:12.946 INFO:tasks.workunit.client.1.smithi189.stdout:4/484: read d2/d20/f4f [3626789,4722] 0 2024-04-20T21:32:12.948 INFO:tasks.workunit.client.1.smithi189.stdout:1/469: write f2 [4932962,29489] 0 2024-04-20T21:32:12.949 INFO:tasks.workunit.client.1.smithi189.stdout:3/484: mknod d2/d3/d2c/d30/d3f/d9a/cae 0 2024-04-20T21:32:12.950 INFO:tasks.workunit.client.1.smithi189.stdout:6/545: sync 2024-04-20T21:32:12.951 INFO:tasks.workunit.client.1.smithi189.stdout:7/535: creat d5/d46/fb7 x:0 0 0 2024-04-20T21:32:12.956 INFO:tasks.workunit.client.1.smithi189.stdout:0/612: creat d3/df/d28/d2b/d2c/fd0 x:0 0 0 2024-04-20T21:32:12.958 INFO:tasks.workunit.client.0.smithi062.stdout:4/872: dwrite da/dd/df/d27/d3f/f11a [4194304,4194304] 0 2024-04-20T21:32:12.962 INFO:tasks.workunit.client.0.smithi062.stdout:4/873: write da/dd/df/d27/fc2 [930662,122078] 0 2024-04-20T21:32:12.963 INFO:tasks.workunit.client.1.smithi189.stdout:3/485: rmdir d2 39 2024-04-20T21:32:12.964 INFO:tasks.workunit.client.0.smithi062.stdout:1/847: mkdir dc/d15/d20/d28/d76/d137 0 2024-04-20T21:32:12.966 INFO:tasks.workunit.client.1.smithi189.stdout:7/536: unlink d5/c37 0 2024-04-20T21:32:12.968 INFO:tasks.workunit.client.1.smithi189.stdout:7/537: read - d5/d59/f80 zero size 2024-04-20T21:32:12.970 INFO:tasks.workunit.client.1.smithi189.stdout:6/546: mknod d8/da/dd/d2a/cb6 0 2024-04-20T21:32:12.971 INFO:tasks.workunit.client.0.smithi062.stdout:2/830: creat d7/da/d15/d2f/d38/d6a/f110 x:0 0 0 2024-04-20T21:32:12.977 INFO:tasks.workunit.client.0.smithi062.stdout:8/801: creat d7/d25/d3b/d58/d8b/d24/dbd/f109 x:0 0 0 2024-04-20T21:32:12.978 INFO:tasks.workunit.client.0.smithi062.stdout:1/848: rename dc/d27/c30 to dc/d27/d32/df8/c138 0 2024-04-20T21:32:12.979 INFO:tasks.workunit.client.1.smithi189.stdout:4/485: mknod d2/dd/d11/dae/cb5 0 2024-04-20T21:32:12.982 INFO:tasks.workunit.client.1.smithi189.stdout:2/458: link d1/d11/d2d/l36 d1/d11/d58/l87 0 2024-04-20T21:32:12.984 INFO:tasks.workunit.client.1.smithi189.stdout:7/538: mkdir d5/d1d/d77/db8 0 2024-04-20T21:32:12.984 INFO:tasks.workunit.client.1.smithi189.stdout:1/470: dwrite d8/d21/d2c/f62 [0,4194304] 0 2024-04-20T21:32:12.985 INFO:tasks.workunit.client.0.smithi062.stdout:7/769: write d7/d3e/d33/d3d/d4d/fa3 [199304,13908] 0 2024-04-20T21:32:12.985 INFO:tasks.workunit.client.0.smithi062.stdout:4/874: rename da/dd/df/d27/d3f/d82/le6 to da/dd/d15/d50/d64/d57/d11e/l134 0 2024-04-20T21:32:12.986 INFO:tasks.workunit.client.1.smithi189.stdout:5/477: dwrite d7/d9/d30/f1c [0,4194304] 0 2024-04-20T21:32:12.986 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:12 smithi189 ceph-mon[21562]: from='client.14708 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:32:12.987 INFO:tasks.workunit.client.0.smithi062.stdout:8/802: write d7/d25/dc5/ff0 [2300132,54177] 0 2024-04-20T21:32:12.988 INFO:tasks.workunit.client.0.smithi062.stdout:4/875: chown da/d56/d71/fe2 0 1 2024-04-20T21:32:12.994 INFO:tasks.workunit.client.0.smithi062.stdout:7/770: unlink d7/d3e/d33/d3d/f8e 0 2024-04-20T21:32:12.994 INFO:tasks.workunit.client.0.smithi062.stdout:2/831: link d7/da/d15/d2f/d38/db0/fb2 d7/da/d15/d2f/d38/d6b/f111 0 2024-04-20T21:32:12.994 INFO:tasks.workunit.client.1.smithi189.stdout:1/471: write d8/d1e/f87 [225156,30536] 0 2024-04-20T21:32:13.000 INFO:tasks.workunit.client.1.smithi189.stdout:5/478: mknod d7/d9/d30/d55/d81/c92 0 2024-04-20T21:32:13.000 INFO:tasks.workunit.client.0.smithi062.stdout:1/849: rename dc/f35 to dc/d70/f139 0 2024-04-20T21:32:13.001 INFO:tasks.workunit.client.1.smithi189.stdout:8/588: dwrite da/df/d22/d33/f36 [0,4194304] 0 2024-04-20T21:32:13.004 INFO:tasks.workunit.client.0.smithi062.stdout:7/771: dread - d7/d26/da8/fd0 zero size 2024-04-20T21:32:13.007 INFO:tasks.workunit.client.1.smithi189.stdout:3/486: dread d2/f5d [0,4194304] 0 2024-04-20T21:32:13.010 INFO:tasks.workunit.client.0.smithi062.stdout:8/803: creat d7/d25/d3b/d58/f10a x:0 0 0 2024-04-20T21:32:13.018 INFO:tasks.workunit.client.0.smithi062.stdout:6/667: sync 2024-04-20T21:32:13.019 INFO:tasks.workunit.client.0.smithi062.stdout:7/772: creat d7/d3e/d33/d3d/d4d/d54/dea/f10b x:0 0 0 2024-04-20T21:32:13.020 INFO:tasks.workunit.client.1.smithi189.stdout:4/486: dwrite d2/dd/d11/f9a [0,4194304] 0 2024-04-20T21:32:13.021 INFO:tasks.workunit.client.0.smithi062.stdout:4/876: dwrite f4 [0,4194304] 0 2024-04-20T21:32:13.022 INFO:tasks.workunit.client.0.smithi062.stdout:4/877: stat da/dd/cba 0 2024-04-20T21:32:13.026 INFO:tasks.workunit.client.0.smithi062.stdout:7/773: readlink d7/d26/dac/lda 0 2024-04-20T21:32:13.027 INFO:tasks.workunit.client.0.smithi062.stdout:4/878: truncate da/d56/d71/d9e/ffd 992110 0 2024-04-20T21:32:13.027 INFO:tasks.workunit.client.1.smithi189.stdout:4/487: readlink d2/d64/l70 0 2024-04-20T21:32:13.028 INFO:tasks.workunit.client.1.smithi189.stdout:3/487: creat d2/d3/d2c/d31/d74/d7a/faf x:0 0 0 2024-04-20T21:32:13.031 INFO:tasks.workunit.client.1.smithi189.stdout:1/472: dread d8/f1b [4194304,4194304] 0 2024-04-20T21:32:13.031 INFO:tasks.workunit.client.1.smithi189.stdout:8/589: rename da/df/d3c/d50/d6c/d85 to da/df/d13/dd2 0 2024-04-20T21:32:13.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:12 smithi062 ceph-mon[16221]: from='client.14708 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:32:13.036 INFO:tasks.workunit.client.1.smithi189.stdout:7/539: dwrite d5/d46/d7d/d45/f99 [0,4194304] 0 2024-04-20T21:32:13.037 INFO:tasks.workunit.client.0.smithi062.stdout:7/774: mknod d7/d26/da8/c10c 0 2024-04-20T21:32:13.040 INFO:tasks.workunit.client.1.smithi189.stdout:6/547: dread d8/da/dd/d35/f59 [0,4194304] 0 2024-04-20T21:32:13.042 INFO:tasks.workunit.client.0.smithi062.stdout:8/804: rmdir d7/d25/d3b/d58/dac/dc7 0 2024-04-20T21:32:13.044 INFO:tasks.workunit.client.1.smithi189.stdout:3/488: mkdir d2/d3/d2c/d30/d3f/d9a/db0 0 2024-04-20T21:32:13.047 INFO:tasks.workunit.client.1.smithi189.stdout:1/473: unlink d8/d1e/d43/d44/l4b 0 2024-04-20T21:32:13.048 INFO:tasks.workunit.client.1.smithi189.stdout:7/540: write d5/db4/f5e [897613,44087] 0 2024-04-20T21:32:13.051 INFO:tasks.workunit.client.0.smithi062.stdout:4/879: rename da/d4f/ffe to da/dd/d15/d50/d64/d57/d5b/f135 0 2024-04-20T21:32:13.054 INFO:tasks.workunit.client.1.smithi189.stdout:5/479: dwrite d7/d9/d30/f8e [0,4194304] 0 2024-04-20T21:32:13.055 INFO:tasks.workunit.client.0.smithi062.stdout:6/668: dwrite d0/dc7/f94 [0,4194304] 0 2024-04-20T21:32:13.057 INFO:tasks.workunit.client.0.smithi062.stdout:8/805: creat d7/d25/d70/f10b x:0 0 0 2024-04-20T21:32:13.061 INFO:tasks.workunit.client.1.smithi189.stdout:2/459: dwrite d1/d11/f28 [0,4194304] 0 2024-04-20T21:32:13.063 INFO:tasks.workunit.client.1.smithi189.stdout:3/489: rename d2/d3/d2c/d31/d74/d7a to d2/d3/d2c/d31/d74/d7a/db1 22 2024-04-20T21:32:13.064 INFO:tasks.workunit.client.0.smithi062.stdout:6/669: rmdir d0/dc7 39 2024-04-20T21:32:13.065 INFO:tasks.workunit.client.0.smithi062.stdout:8/806: write d7/d9/ff [4334078,66407] 0 2024-04-20T21:32:13.065 INFO:tasks.workunit.client.1.smithi189.stdout:3/490: chown d2/d3/f52 13235040 1 2024-04-20T21:32:13.068 INFO:tasks.workunit.client.1.smithi189.stdout:8/590: dwrite da/de/d1e/fb7 [0,4194304] 0 2024-04-20T21:32:13.069 INFO:tasks.workunit.client.0.smithi062.stdout:1/850: dread dc/d15/d20/d28/d83/f133 [0,4194304] 0 2024-04-20T21:32:13.069 INFO:tasks.workunit.client.0.smithi062.stdout:2/832: dwrite d7/da/d15/d2f/fa4 [0,4194304] 0 2024-04-20T21:32:13.072 INFO:tasks.workunit.client.0.smithi062.stdout:6/670: creat d0/d4f/fdf x:0 0 0 2024-04-20T21:32:13.074 INFO:tasks.workunit.client.1.smithi189.stdout:6/548: creat d8/d12/d99/fb7 x:0 0 0 2024-04-20T21:32:13.077 INFO:tasks.workunit.client.1.smithi189.stdout:5/480: unlink d7/d35/f44 0 2024-04-20T21:32:13.077 INFO:tasks.workunit.client.1.smithi189.stdout:4/488: link d2/d22/d57/l5b d2/dd/d37/lb6 0 2024-04-20T21:32:13.080 INFO:tasks.workunit.client.1.smithi189.stdout:7/541: fsync d5/d21/d25/f30 0 2024-04-20T21:32:13.082 INFO:tasks.workunit.client.1.smithi189.stdout:2/460: mknod d1/d11/d52/c88 0 2024-04-20T21:32:13.083 INFO:tasks.workunit.client.1.smithi189.stdout:7/542: chown d5/db4/ca0 63034146 1 2024-04-20T21:32:13.083 INFO:tasks.workunit.client.1.smithi189.stdout:1/474: unlink d8/df/c8f 0 2024-04-20T21:32:13.085 INFO:tasks.workunit.client.1.smithi189.stdout:8/591: symlink da/df/d3c/d50/ld3 0 2024-04-20T21:32:13.087 INFO:tasks.workunit.client.0.smithi062.stdout:9/895: sync 2024-04-20T21:32:13.088 INFO:tasks.workunit.client.1.smithi189.stdout:7/543: read d5/d46/fa9 [875506,61961] 0 2024-04-20T21:32:13.091 INFO:tasks.workunit.client.1.smithi189.stdout:7/544: dread - d5/d21/d35/f7e zero size 2024-04-20T21:32:13.092 INFO:tasks.workunit.client.0.smithi062.stdout:1/851: mkdir dc/d27/d32/d13a 0 2024-04-20T21:32:13.095 INFO:tasks.workunit.client.1.smithi189.stdout:4/489: mkdir d2/dd/d11/d14/d69/d6a/d7a/db7 0 2024-04-20T21:32:13.097 INFO:tasks.workunit.client.0.smithi062.stdout:2/833: mknod d7/da/d15/d2f/d38/db0/c112 0 2024-04-20T21:32:13.098 INFO:tasks.workunit.client.1.smithi189.stdout:9/488: sync 2024-04-20T21:32:13.099 INFO:tasks.workunit.client.1.smithi189.stdout:8/592: truncate da/de/d1e/d69/f95 725921 0 2024-04-20T21:32:13.099 INFO:tasks.workunit.client.1.smithi189.stdout:1/475: dwrite f6 [0,4194304] 0 2024-04-20T21:32:13.102 INFO:tasks.workunit.client.1.smithi189.stdout:4/490: symlink d2/d20/lb8 0 2024-04-20T21:32:13.103 INFO:tasks.workunit.client.0.smithi062.stdout:6/671: truncate d0/d9/d11/d22/dd5/d67/f9e 892441 0 2024-04-20T21:32:13.104 INFO:tasks.workunit.client.0.smithi062.stdout:2/834: write d7/da/d15/d2f/d38/d10f/d4b/f95 [2298083,60102] 0 2024-04-20T21:32:13.105 INFO:tasks.workunit.client.0.smithi062.stdout:8/807: creat d7/d25/d3b/d58/d8b/d24/f10c x:0 0 0 2024-04-20T21:32:13.106 INFO:tasks.workunit.client.1.smithi189.stdout:7/545: mkdir d5/d1d/d64/db9 0 2024-04-20T21:32:13.108 INFO:tasks.workunit.client.0.smithi062.stdout:9/896: link d16/d3d/ff6 d16/d2f/d5b/da4/f122 0 2024-04-20T21:32:13.110 INFO:tasks.workunit.client.1.smithi189.stdout:5/481: link d7/c11 d7/d9/d51/d70/c93 0 2024-04-20T21:32:13.113 INFO:tasks.workunit.client.0.smithi062.stdout:6/672: mkdir d0/d1a/d7d/db1/de0 0 2024-04-20T21:32:13.114 INFO:tasks.workunit.client.0.smithi062.stdout:2/835: mknod d7/da/d15/d2f/d38/d6b/c113 0 2024-04-20T21:32:13.115 INFO:tasks.workunit.client.0.smithi062.stdout:9/897: chown d16/d2c/d42/da6/da9/fc4 15535 1 2024-04-20T21:32:13.116 INFO:tasks.workunit.client.1.smithi189.stdout:8/593: truncate da/df/d13/f14 4895220 0 2024-04-20T21:32:13.116 INFO:tasks.workunit.client.1.smithi189.stdout:5/482: fsync d7/d9/d30/d55/d61/f8d 0 2024-04-20T21:32:13.116 INFO:tasks.workunit.client.0.smithi062.stdout:8/808: chown d7/d25/f9d 1 1 2024-04-20T21:32:13.117 INFO:tasks.workunit.client.0.smithi062.stdout:2/836: chown d7/da/d15/l58 128510 1 2024-04-20T21:32:13.118 INFO:tasks.workunit.client.0.smithi062.stdout:9/898: write d16/d3e/fd5 [162566,247] 0 2024-04-20T21:32:13.120 INFO:tasks.workunit.client.0.smithi062.stdout:1/852: mkdir dc/d70/d7f/d13b 0 2024-04-20T21:32:13.121 INFO:tasks.workunit.client.1.smithi189.stdout:9/489: creat d9/dc/f98 x:0 0 0 2024-04-20T21:32:13.122 INFO:tasks.workunit.client.1.smithi189.stdout:7/546: symlink d5/db4/lba 0 2024-04-20T21:32:13.124 INFO:tasks.workunit.client.1.smithi189.stdout:9/490: write d9/dc/f3d [6795503,75884] 0 2024-04-20T21:32:13.129 INFO:tasks.workunit.client.1.smithi189.stdout:7/547: creat d5/d21/d4a/d52/d74/fbb x:0 0 0 2024-04-20T21:32:13.129 INFO:tasks.workunit.client.0.smithi062.stdout:1/853: unlink dc/d27/c130 0 2024-04-20T21:32:13.130 INFO:tasks.workunit.client.1.smithi189.stdout:7/548: chown d5/d1d/d64/db9 1003712 1 2024-04-20T21:32:13.130 INFO:tasks.workunit.client.1.smithi189.stdout:8/594: mknod da/df/d13/d63/cd4 0 2024-04-20T21:32:13.131 INFO:tasks.workunit.client.0.smithi062.stdout:7/775: truncate d7/d3e/d33/d3d/d4d/d54/dea/f93 66152 0 2024-04-20T21:32:13.132 INFO:tasks.workunit.client.1.smithi189.stdout:7/549: chown d5/d21/d4a/d52/d74/c84 16 1 2024-04-20T21:32:13.133 INFO:tasks.workunit.client.0.smithi062.stdout:9/899: creat d16/d2c/d42/d10f/f123 x:0 0 0 2024-04-20T21:32:13.133 INFO:tasks.workunit.client.0.smithi062.stdout:1/854: chown dc/d15/df6/l135 111 1 2024-04-20T21:32:13.135 INFO:tasks.workunit.client.1.smithi189.stdout:7/550: stat d5/d21/d4a/d52/d98/f9b 0 2024-04-20T21:32:13.135 INFO:tasks.workunit.client.0.smithi062.stdout:8/809: getdents d7/d30/db3 0 2024-04-20T21:32:13.137 INFO:tasks.workunit.client.0.smithi062.stdout:7/776: symlink d7/d26/l10d 0 2024-04-20T21:32:13.138 INFO:tasks.workunit.client.0.smithi062.stdout:9/900: write d16/d2c/d42/da6/da9/ff4 [700738,38590] 0 2024-04-20T21:32:13.139 INFO:tasks.workunit.client.0.smithi062.stdout:1/855: mknod dc/d15/d20/d28/c13c 0 2024-04-20T21:32:13.140 INFO:tasks.workunit.client.0.smithi062.stdout:6/673: dwrite d0/f80 [0,4194304] 0 2024-04-20T21:32:13.141 INFO:tasks.workunit.client.1.smithi189.stdout:9/491: link d9/dc/d17/d2b/d52/c70 d9/dc/d17/d95/c99 0 2024-04-20T21:32:13.142 INFO:tasks.workunit.client.0.smithi062.stdout:8/810: rmdir d7/da9 39 2024-04-20T21:32:13.146 INFO:tasks.workunit.client.1.smithi189.stdout:9/492: chown d9/dc/d17/d95 3 1 2024-04-20T21:32:13.146 INFO:tasks.workunit.client.0.smithi062.stdout:4/880: write da/dd/d15/d50/d64/d3d/da1/fb9 [4502616,94631] 0 2024-04-20T21:32:13.147 INFO:tasks.workunit.client.1.smithi189.stdout:3/491: truncate d2/d1c/d5c/f87 404752 0 2024-04-20T21:32:13.148 INFO:tasks.workunit.client.0.smithi062.stdout:6/674: mknod d0/d1a/d44/ce1 0 2024-04-20T21:32:13.150 INFO:tasks.workunit.client.0.smithi062.stdout:1/856: fdatasync dc/d15/d75/d96/dd4/d3a/f43 0 2024-04-20T21:32:13.155 INFO:tasks.workunit.client.1.smithi189.stdout:6/549: dwrite d8/da/dd/d35/d6d/d77/fac [0,4194304] 0 2024-04-20T21:32:13.155 INFO:tasks.workunit.client.0.smithi062.stdout:8/811: dread d7/d25/d70/fa1 [0,4194304] 0 2024-04-20T21:32:13.158 INFO:tasks.workunit.client.1.smithi189.stdout:9/493: readlink d9/dc/d17/d2b/d52/l66 0 2024-04-20T21:32:13.159 INFO:tasks.workunit.client.0.smithi062.stdout:6/675: readlink d0/dc7/l7e 0 2024-04-20T21:32:13.160 INFO:tasks.workunit.client.1.smithi189.stdout:1/476: truncate d8/d1e/d28/f90 638330 0 2024-04-20T21:32:13.161 INFO:tasks.workunit.client.0.smithi062.stdout:8/812: chown d7/d25/f9d 10964646 1 2024-04-20T21:32:13.162 INFO:tasks.workunit.client.1.smithi189.stdout:4/491: truncate d2/dd/d11/d14/f79 371426 0 2024-04-20T21:32:13.164 INFO:tasks.workunit.client.0.smithi062.stdout:8/813: stat d7/d25/d5c/lff 0 2024-04-20T21:32:13.165 INFO:tasks.workunit.client.0.smithi062.stdout:4/881: unlink da/dd/d15/daf/cd5 0 2024-04-20T21:32:13.165 INFO:tasks.workunit.client.1.smithi189.stdout:3/492: creat d2/d3/d36/da1/fb2 x:0 0 0 2024-04-20T21:32:13.166 INFO:tasks.workunit.client.1.smithi189.stdout:9/494: unlink d9/d12/d3b/c6e 0 2024-04-20T21:32:13.166 INFO:tasks.workunit.client.1.smithi189.stdout:1/477: write d8/f86 [810268,85669] 0 2024-04-20T21:32:13.166 INFO:tasks.workunit.client.1.smithi189.stdout:5/483: truncate d7/d9/d28/f58 495924 0 2024-04-20T21:32:13.168 INFO:tasks.workunit.client.1.smithi189.stdout:6/550: creat d8/d2b/d49/d54/d67/fb8 x:0 0 0 2024-04-20T21:32:13.169 INFO:tasks.workunit.client.0.smithi062.stdout:8/814: write d7/d25/d3b/d58/fc4 [555763,12776] 0 2024-04-20T21:32:13.174 INFO:tasks.workunit.client.1.smithi189.stdout:1/478: mkdir d8/df/d3a/d49/d8c/da2 0 2024-04-20T21:32:13.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:13.175+0000 7f0174000700 1 -- 172.21.15.62:0/4222545926 --> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f016c061190 con 0x7f015406eac0 2024-04-20T21:32:13.176 INFO:tasks.workunit.client.0.smithi062.stdout:2/837: write d7/da/d15/d1b/f2c [4663995,23395] 0 2024-04-20T21:32:13.178 INFO:tasks.workunit.client.0.smithi062.stdout:1/857: dread dc/d15/d20/d28/d83/dc0/fc2 [0,4194304] 0 2024-04-20T21:32:13.180 INFO:tasks.workunit.client.1.smithi189.stdout:4/492: mknod d2/dd/d6b/da6/cb9 0 2024-04-20T21:32:13.181 INFO:tasks.workunit.client.1.smithi189.stdout:8/595: dread da/df/d13/d63/f73 [0,4194304] 0 2024-04-20T21:32:13.182 INFO:tasks.workunit.client.1.smithi189.stdout:9/495: creat d9/dc/d17/d7e/f9a x:0 0 0 2024-04-20T21:32:13.187 INFO:tasks.workunit.client.0.smithi062.stdout:2/838: readlink d7/da/d15/d2f/d38/d10f/d4b/d8f/lc8 0 2024-04-20T21:32:13.189 INFO:tasks.workunit.client.0.smithi062.stdout:1/858: dread - dc/d15/d75/d96/dd4/d78/fc8 zero size 2024-04-20T21:32:13.190 INFO:tasks.workunit.client.1.smithi189.stdout:7/551: dread d5/f39 [0,4194304] 0 2024-04-20T21:32:13.194 INFO:tasks.workunit.client.0.smithi062.stdout:2/839: unlink d7/da/f3c 0 2024-04-20T21:32:13.194 INFO:tasks.workunit.client.1.smithi189.stdout:7/552: rename d5 to d5/d21/d97/dbc 22 2024-04-20T21:32:13.197 INFO:tasks.workunit.client.1.smithi189.stdout:6/551: dwrite d8/d12/d30/d47/fad [0,4194304] 0 2024-04-20T21:32:13.197 INFO:tasks.workunit.client.1.smithi189.stdout:9/496: symlink d9/d12/d15/d1a/l9b 0 2024-04-20T21:32:13.198 INFO:tasks.workunit.client.0.smithi062.stdout:7/777: dwrite d7/d3e/d33/d3d/d4d/d54/f3f [0,4194304] 0 2024-04-20T21:32:13.199 INFO:tasks.workunit.client.1.smithi189.stdout:7/553: dread - d5/d1d/d64/f85 zero size 2024-04-20T21:32:13.202 INFO:tasks.workunit.client.0.smithi062.stdout:5/791: sync 2024-04-20T21:32:13.202 INFO:tasks.workunit.client.0.smithi062.stdout:3/746: sync 2024-04-20T21:32:13.202 INFO:tasks.workunit.client.0.smithi062.stdout:0/780: sync 2024-04-20T21:32:13.205 INFO:tasks.workunit.client.0.smithi062.stdout:8/815: dread d7/d25/d3b/d58/f63 [0,4194304] 0 2024-04-20T21:32:13.205 INFO:tasks.workunit.client.0.smithi062.stdout:9/901: dwrite d16/d2c/d42/da6/da9/fe9 [0,4194304] 0 2024-04-20T21:32:13.205 INFO:tasks.workunit.client.0.smithi062.stdout:2/840: dread - d7/db5/fff zero size 2024-04-20T21:32:13.207 INFO:tasks.workunit.client.1.smithi189.stdout:4/493: mknod d2/cba 0 2024-04-20T21:32:13.209 INFO:tasks.workunit.client.1.smithi189.stdout:4/494: fsync d2/d20/f24 0 2024-04-20T21:32:13.210 INFO:tasks.workunit.client.1.smithi189.stdout:4/495: chown d2/f8 12831347 1 2024-04-20T21:32:13.210 INFO:tasks.workunit.client.0.smithi062.stdout:0/781: fsync f1 0 2024-04-20T21:32:13.212 INFO:tasks.workunit.client.1.smithi189.stdout:4/496: write d2/dd/d11/f92 [3576806,111679] 0 2024-04-20T21:32:13.213 INFO:tasks.workunit.client.0.smithi062.stdout:1/859: creat dc/d15/d75/d96/dd4/d3a/da2/de4/d84/dbe/d119/f13d x:0 0 0 2024-04-20T21:32:13.216 INFO:tasks.workunit.client.1.smithi189.stdout:3/493: dread d2/d3/d2c/d31/f50 [0,4194304] 0 2024-04-20T21:32:13.217 INFO:tasks.workunit.client.0.smithi062.stdout:3/747: write d0/d5c/f5a [7133819,130037] 0 2024-04-20T21:32:13.223 INFO:tasks.workunit.client.0.smithi062.stdout:6/676: dwrite d0/d1a/f85 [0,4194304] 0 2024-04-20T21:32:13.223 INFO:tasks.workunit.client.0.smithi062.stdout:1/860: readlink dc/d15/d75/d96/dd4/d3a/da2/de4/d7c/l10b 0 2024-04-20T21:32:13.227 INFO:tasks.workunit.client.0.smithi062.stdout:9/902: mkdir d16/d3d/d55/d6e/d7d/d124 0 2024-04-20T21:32:13.228 INFO:tasks.workunit.client.0.smithi062.stdout:2/841: unlink d7/da/d15/d2f/d38/f54 0 2024-04-20T21:32:13.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:13.229+0000 7f016a000700 1 -- 172.21.15.62:0/4222545926 <== mgr.14658 v2:172.21.15.62:6800/1585760929 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7f016c061190 con 0x7f015406eac0 2024-04-20T21:32:13.230 INFO:tasks.workunit.client.1.smithi189.stdout:5/484: dread d7/d9/d31/d36/f38 [0,4194304] 0 2024-04-20T21:32:13.231 INFO:tasks.workunit.client.1.smithi189.stdout:9/497: mknod d9/dc/d17/d7e/c9c 0 2024-04-20T21:32:13.231 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T21:32:13.231 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (10m) 27s ago 11m 14.0M - 0.25.0 c8568f914cd2 957f663a9cd7 2024-04-20T21:32:13.231 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (12m) 27s ago 12m 7926k - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T21:32:13.231 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (10m) 31s ago 10m 7755k - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T21:32:13.231 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (12m) 27s ago 12m 6656k - 18.2.0 dc2bc1663786 36e10ef5be39 2024-04-20T21:32:13.232 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (10m) 31s ago 10m 6665k - 18.2.0 dc2bc1663786 4101f9953e3d 2024-04-20T21:32:13.232 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (9m) 27s ago 11m 74.9M - 9.4.7 954c08fa6188 99363bb095d8 2024-04-20T21:32:13.232 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (4m) 27s ago 4m 143M - 18.2.0 dc2bc1663786 5e6460aa6159 2024-04-20T21:32:13.232 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (4m) 27s ago 4m 138M - 18.2.0 dc2bc1663786 e65f2267544e 2024-04-20T21:32:13.232 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (4m) 31s ago 4m 93.3M - 18.2.0 dc2bc1663786 fb2a8a223db8 2024-04-20T21:32:13.232 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (4m) 31s ago 4m 87.6M - 18.2.0 dc2bc1663786 f2eec2ee8951 2024-04-20T21:32:13.232 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (56s) 27s ago 13m 477M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T21:32:13.232 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (116s) 31s ago 10m 76.1M - 19.0.0-3133-ga8c84f59 76bc6b91b726 4cde2bc4de24 2024-04-20T21:32:13.233 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (13m) 27s ago 13m 53.4M 2048M 18.2.0 dc2bc1663786 bd6efbd5c380 2024-04-20T21:32:13.233 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (10m) 31s ago 10m 43.9M 2048M 18.2.0 dc2bc1663786 3735c8e5db9a 2024-04-20T21:32:13.233 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (12m) 27s ago 12m 7616k - 1.5.0 0da6a335fe13 fc746d8ae72e 2024-04-20T21:32:13.233 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (10m) 31s ago 10m 7956k - 1.5.0 0da6a335fe13 db03af85b286 2024-04-20T21:32:13.233 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (9m) 27s ago 9m 86.2M 1216M 18.2.0 dc2bc1663786 3f315c3afc1f 2024-04-20T21:32:13.233 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (8m) 27s ago 8m 98.5M 1216M 18.2.0 dc2bc1663786 44e4d07416ad 2024-04-20T21:32:13.233 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (8m) 27s ago 8m 78.6M 1216M 18.2.0 dc2bc1663786 31c5d9586e98 2024-04-20T21:32:13.233 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (7m) 31s ago 7m 94.5M 2240M 18.2.0 dc2bc1663786 611c1e242693 2024-04-20T21:32:13.233 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (7m) 31s ago 7m 90.3M 2240M 18.2.0 dc2bc1663786 6c8fce91d45d 2024-04-20T21:32:13.233 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (6m) 31s ago 6m 86.0M 2240M 18.2.0 dc2bc1663786 8e4574bae8db 2024-04-20T21:32:13.233 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (65s) 27s ago 11m 49.7M - 2.43.0 a07b618ecd1d 6777f15d9248 2024-04-20T21:32:13.234 INFO:tasks.workunit.client.1.smithi189.stdout:7/554: fdatasync d5/d21/d25/d34/f6f 0 2024-04-20T21:32:13.234 INFO:tasks.workunit.client.0.smithi062.stdout:4/882: dwrite da/dd/df/d27/d3f/f119 [0,4194304] 0 2024-04-20T21:32:13.235 INFO:tasks.workunit.client.1.smithi189.stdout:3/494: unlink d2/d3/d2c/d31/f50 0 2024-04-20T21:32:13.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:13.236+0000 7f0174000700 1 -- 172.21.15.62:0/4222545926 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f015406eac0 msgr2=0x7f0154070f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:32:13.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:13.236+0000 7f0174000700 1 --2- 172.21.15.62:0/4222545926 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f015406eac0 0x7f0154070f70 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0158006010 tx=0x7f0158005a90 comp rx=0 tx=0).stop 2024-04-20T21:32:13.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:13.236+0000 7f0174000700 1 -- 172.21.15.62:0/4222545926 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f016c100d60 msgr2=0x7f016c197cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:32:13.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:13.236+0000 7f0174000700 1 --2- 172.21.15.62:0/4222545926 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f016c100d60 0x7f016c197cf0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f016000e940 tx=0x7f016000ec50 comp rx=0 tx=0).stop 2024-04-20T21:32:13.237 INFO:tasks.workunit.client.1.smithi189.stdout:5/485: truncate d7/d9/d28/f7b 742291 0 2024-04-20T21:32:13.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:13.237+0000 7f0174000700 1 -- 172.21.15.62:0/4222545926 shutdown_connections 2024-04-20T21:32:13.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:13.237+0000 7f0174000700 1 --2- 172.21.15.62:0/4222545926 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f015406eac0 0x7f0154070f70 secure :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0158006010 tx=0x7f0158005a90 comp rx=0 tx=0).stop 2024-04-20T21:32:13.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:13.237+0000 7f0174000700 1 --2- 172.21.15.62:0/4222545926 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f016c100d60 0x7f016c197cf0 secure :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f016000e940 tx=0x7f016000ec50 comp rx=0 tx=0).stop 2024-04-20T21:32:13.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:13.237+0000 7f0174000700 1 --2- 172.21.15.62:0/4222545926 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f016c103680 0x7f016c198230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:32:13.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:13.237+0000 7f0174000700 1 -- 172.21.15.62:0/4222545926 >> 172.21.15.62:0/4222545926 conn(0x7f016c0fa780 msgr2=0x7f016c0ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:32:13.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:13.237+0000 7f0174000700 1 -- 172.21.15.62:0/4222545926 shutdown_connections 2024-04-20T21:32:13.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:13.238+0000 7f0174000700 1 -- 172.21.15.62:0/4222545926 wait complete. 2024-04-20T21:32:13.239 INFO:tasks.workunit.client.1.smithi189.stdout:2/461: sync 2024-04-20T21:32:13.239 INFO:tasks.workunit.client.1.smithi189.stdout:0/613: sync 2024-04-20T21:32:13.245 INFO:tasks.workunit.client.0.smithi062.stdout:6/677: creat d0/d9/d11/d22/dd5/d67/d75/fe2 x:0 0 0 2024-04-20T21:32:13.246 INFO:tasks.workunit.client.1.smithi189.stdout:9/498: creat d9/d12/d15/d1a/f9d x:0 0 0 2024-04-20T21:32:13.246 INFO:tasks.workunit.client.0.smithi062.stdout:5/792: write d8/d1e/d2c/f86 [5172811,74479] 0 2024-04-20T21:32:13.248 INFO:tasks.workunit.client.1.smithi189.stdout:8/596: dwrite da/fb [0,4194304] 0 2024-04-20T21:32:13.252 INFO:tasks.workunit.client.1.smithi189.stdout:5/486: creat d7/d14/d18/f94 x:0 0 0 2024-04-20T21:32:13.253 INFO:tasks.workunit.client.1.smithi189.stdout:0/614: rmdir d3/d26/d88 39 2024-04-20T21:32:13.255 INFO:tasks.workunit.client.1.smithi189.stdout:0/615: write d3/d8c/f7a [2157215,80381] 0 2024-04-20T21:32:13.256 INFO:tasks.workunit.client.1.smithi189.stdout:7/555: read d5/db4/f38 [870691,22084] 0 2024-04-20T21:32:13.257 INFO:tasks.workunit.client.1.smithi189.stdout:2/462: mkdir d1/d17/d37/d89 0 2024-04-20T21:32:13.258 INFO:tasks.workunit.client.1.smithi189.stdout:6/552: write d8/da/dd/f71 [483983,107343] 0 2024-04-20T21:32:13.258 INFO:tasks.workunit.client.0.smithi062.stdout:9/903: creat d16/d3d/d55/f125 x:0 0 0 2024-04-20T21:32:13.258 INFO:tasks.workunit.client.0.smithi062.stdout:8/816: dwrite d7/d25/d3b/d58/f63 [0,4194304] 0 2024-04-20T21:32:13.258 INFO:tasks.workunit.client.0.smithi062.stdout:7/778: dread d7/d63/f7a [0,4194304] 0 2024-04-20T21:32:13.260 INFO:tasks.workunit.client.0.smithi062.stdout:1/861: dread dc/d15/d75/d96/dd4/d3a/da2/de4/f65 [0,4194304] 0 2024-04-20T21:32:13.263 INFO:tasks.workunit.client.1.smithi189.stdout:1/479: link d8/d1e/d28/f90 d8/d21/d2c/fa3 0 2024-04-20T21:32:13.263 INFO:tasks.workunit.client.0.smithi062.stdout:9/904: readlink d16/d2c/d62/l115 0 2024-04-20T21:32:13.263 INFO:tasks.workunit.client.0.smithi062.stdout:6/678: symlink d0/d4f/le3 0 2024-04-20T21:32:13.264 INFO:tasks.workunit.client.1.smithi189.stdout:4/497: link d2/dd/d11/dae/cb5 d2/dd/d11/d14/d69/d6a/d7a/db7/cbb 0 2024-04-20T21:32:13.266 INFO:tasks.workunit.client.1.smithi189.stdout:6/553: stat d8/da/f6b 0 2024-04-20T21:32:13.267 INFO:tasks.workunit.client.1.smithi189.stdout:1/480: readlink d8/d21/d2c/d36/l7d 0 2024-04-20T21:32:13.268 INFO:tasks.workunit.client.0.smithi062.stdout:8/817: write d7/d25/d3b/d58/d8b/d24/d46/fd2 [1660906,124511] 0 2024-04-20T21:32:13.268 INFO:tasks.workunit.client.1.smithi189.stdout:1/481: fsync d8/df/d50/f88 0 2024-04-20T21:32:13.270 INFO:tasks.workunit.client.1.smithi189.stdout:1/482: chown d8/df/d3a/c4d 1900 1 2024-04-20T21:32:13.272 INFO:tasks.workunit.client.1.smithi189.stdout:1/483: read d8/f24 [245722,55862] 0 2024-04-20T21:32:13.275 INFO:tasks.workunit.client.0.smithi062.stdout:7/779: mknod d7/d26/dac/ddb/c10e 0 2024-04-20T21:32:13.275 INFO:tasks.workunit.client.0.smithi062.stdout:2/842: rename d7/da/d15/d2f/d38/d10f/d4b/d8f to d7/da/d15/d2f/d38/d10f/d4b/d114 0 2024-04-20T21:32:13.281 INFO:tasks.workunit.client.1.smithi189.stdout:7/556: dread d5/d1d/f56 [0,4194304] 0 2024-04-20T21:32:13.284 INFO:tasks.workunit.client.1.smithi189.stdout:5/487: mkdir d7/d9/d51/d70/d95 0 2024-04-20T21:32:13.285 INFO:tasks.workunit.client.0.smithi062.stdout:1/862: creat dc/d27/d32/df8/f13e x:0 0 0 2024-04-20T21:32:13.285 INFO:tasks.workunit.client.0.smithi062.stdout:6/679: read d0/d52/da0/fa8 [89415,121314] 0 2024-04-20T21:32:13.285 INFO:tasks.workunit.client.0.smithi062.stdout:4/883: write da/dd/d15/d50/d64/d3d/f40 [660120,101734] 0 2024-04-20T21:32:13.285 INFO:tasks.workunit.client.1.smithi189.stdout:2/463: creat d1/d11/d58/d6b/f8a x:0 0 0 2024-04-20T21:32:13.288 INFO:tasks.workunit.client.1.smithi189.stdout:2/464: stat d1/d17/d2f/d4a/f7b 0 2024-04-20T21:32:13.291 INFO:tasks.workunit.client.0.smithi062.stdout:8/818: creat d7/f10d x:0 0 0 2024-04-20T21:32:13.293 INFO:tasks.workunit.client.1.smithi189.stdout:0/616: creat d3/df/d5d/d64/fd1 x:0 0 0 2024-04-20T21:32:13.296 INFO:tasks.workunit.client.1.smithi189.stdout:0/617: fdatasync d3/df/d28/d2b/d2c/fd0 0 2024-04-20T21:32:13.298 INFO:tasks.workunit.client.0.smithi062.stdout:4/884: rename da/dd/d15/d50/d64/d57/d5b/cde to da/dd/db8/c136 0 2024-04-20T21:32:13.302 INFO:tasks.workunit.client.1.smithi189.stdout:3/495: getdents d2/d3/d25 0 2024-04-20T21:32:13.306 INFO:tasks.workunit.client.1.smithi189.stdout:9/499: truncate d9/d12/d3f/d72/f7b 5708912 0 2024-04-20T21:32:13.306 INFO:tasks.workunit.client.1.smithi189.stdout:6/554: truncate f2 1253484 0 2024-04-20T21:32:13.306 INFO:tasks.workunit.client.1.smithi189.stdout:7/557: write d5/f39 [3687469,61100] 0 2024-04-20T21:32:13.306 INFO:tasks.workunit.client.1.smithi189.stdout:5/488: mkdir d7/d9/d30/d55/d61/d96 0 2024-04-20T21:32:13.307 INFO:tasks.workunit.client.1.smithi189.stdout:4/498: truncate d2/d22/f45 702030 0 2024-04-20T21:32:13.307 INFO:tasks.workunit.client.0.smithi062.stdout:7/780: creat d7/d3e/d33/d3d/d4d/d54/d4f/de9/df1/d105/f10f x:0 0 0 2024-04-20T21:32:13.308 INFO:tasks.workunit.client.0.smithi062.stdout:9/905: dread d16/d2c/d42/da6/da9/fc4 [0,4194304] 0 2024-04-20T21:32:13.308 INFO:tasks.workunit.client.0.smithi062.stdout:4/885: unlink da/dd/d15/d50/d64/d57/f65 0 2024-04-20T21:32:13.309 INFO:tasks.workunit.client.1.smithi189.stdout:1/484: rename d8/df/c5e to d8/df/ca4 0 2024-04-20T21:32:13.309 INFO:tasks.workunit.client.1.smithi189.stdout:3/496: read d2/d3/f52 [6874355,59101] 0 2024-04-20T21:32:13.310 INFO:tasks.workunit.client.0.smithi062.stdout:0/782: dread d7/d1c/d15/d17/fad [0,4194304] 0 2024-04-20T21:32:13.311 INFO:tasks.workunit.client.1.smithi189.stdout:6/555: symlink d8/d10/lb9 0 2024-04-20T21:32:13.314 INFO:tasks.workunit.client.1.smithi189.stdout:9/500: mknod d9/dc/d17/d2b/d52/c9e 0 2024-04-20T21:32:13.314 INFO:tasks.workunit.client.1.smithi189.stdout:5/489: creat d7/d35/d48/f97 x:0 0 0 2024-04-20T21:32:13.315 INFO:tasks.workunit.client.1.smithi189.stdout:7/558: write d5/d46/f8f [187587,84847] 0 2024-04-20T21:32:13.317 INFO:tasks.workunit.client.1.smithi189.stdout:0/618: creat d3/df/d30/fd2 x:0 0 0 2024-04-20T21:32:13.319 INFO:tasks.workunit.client.0.smithi062.stdout:9/906: write d16/d2c/d42/da6/da9/fc4 [1047088,68551] 0 2024-04-20T21:32:13.320 INFO:tasks.workunit.client.1.smithi189.stdout:1/485: fdatasync d8/d1e/d43/d44/f47 0 2024-04-20T21:32:13.324 INFO:tasks.workunit.client.1.smithi189.stdout:8/597: dwrite da/df/d3a/f4c [4194304,4194304] 0 2024-04-20T21:32:13.325 INFO:tasks.workunit.client.1.smithi189.stdout:5/490: write d7/d9/d30/f8e [43999,62941] 0 2024-04-20T21:32:13.325 INFO:tasks.workunit.client.1.smithi189.stdout:6/556: write d8/d2b/d49/d54/f83 [4009991,127629] 0 2024-04-20T21:32:13.325 INFO:tasks.workunit.client.0.smithi062.stdout:3/748: dread d0/f98 [0,4194304] 0 2024-04-20T21:32:13.326 INFO:tasks.workunit.client.0.smithi062.stdout:5/793: dwrite d8/d38/d7a/d4c/d7d/d9d/dac/d2f/fc0 [0,4194304] 0 2024-04-20T21:32:13.326 INFO:tasks.workunit.client.1.smithi189.stdout:0/619: unlink d3/f27 0 2024-04-20T21:32:13.328 INFO:tasks.workunit.client.1.smithi189.stdout:1/486: read d8/d21/f38 [98394,60181] 0 2024-04-20T21:32:13.330 INFO:tasks.workunit.client.0.smithi062.stdout:4/886: symlink da/dd/df/l137 0 2024-04-20T21:32:13.331 INFO:tasks.workunit.client.0.smithi062.stdout:5/794: stat d8/d1e/d2c/l7e 0 2024-04-20T21:32:13.333 INFO:tasks.workunit.client.0.smithi062.stdout:7/781: dread d7/d63/fa7 [0,4194304] 0 2024-04-20T21:32:13.333 INFO:tasks.workunit.client.0.smithi062.stdout:0/783: link d7/d1c/d15/d17/d40/ffa d7/d1c/d15/d17/d40/fff 0 2024-04-20T21:32:13.343 INFO:tasks.workunit.client.0.smithi062.stdout:2/843: dread d7/da/fbd [0,4194304] 0 2024-04-20T21:32:13.344 INFO:tasks.workunit.client.1.smithi189.stdout:2/465: write d1/d11/d58/d6b/f6c [1071072,64651] 0 2024-04-20T21:32:13.346 INFO:tasks.workunit.client.0.smithi062.stdout:7/782: chown d7/d3e/l21 19 1 2024-04-20T21:32:13.348 INFO:tasks.workunit.client.1.smithi189.stdout:3/497: dwrite d2/f2a [0,4194304] 0 2024-04-20T21:32:13.349 INFO:tasks.workunit.client.0.smithi062.stdout:8/819: dwrite d7/d25/d3b/d58/d8b/d24/d46/f64 [0,4194304] 0 2024-04-20T21:32:13.350 INFO:tasks.workunit.client.0.smithi062.stdout:5/795: write d8/d10/f26 [3978301,53301] 0 2024-04-20T21:32:13.352 INFO:tasks.workunit.client.1.smithi189.stdout:3/498: fdatasync d2/d3/d2c/d31/d4c/f63 0 2024-04-20T21:32:13.355 INFO:tasks.workunit.client.0.smithi062.stdout:9/907: getdents d16/d3d/d55 0 2024-04-20T21:32:13.361 INFO:tasks.workunit.client.0.smithi062.stdout:9/908: mknod d16/d2c/d5e/d96/c126 0 2024-04-20T21:32:13.364 INFO:tasks.workunit.client.0.smithi062.stdout:5/796: fsync d8/d38/d7a/d4c/dcf/fee 0 2024-04-20T21:32:13.367 INFO:tasks.workunit.client.1.smithi189.stdout:5/491: symlink d7/d9/d51/d70/l98 0 2024-04-20T21:32:13.368 INFO:tasks.workunit.client.0.smithi062.stdout:8/820: chown d7/d25/d3b/d58/d8b/l82 4976 1 2024-04-20T21:32:13.369 INFO:tasks.workunit.client.0.smithi062.stdout:9/909: creat d16/d2f/f127 x:0 0 0 2024-04-20T21:32:13.372 INFO:tasks.workunit.client.0.smithi062.stdout:7/783: creat d7/d3e/d33/d3d/d4d/f110 x:0 0 0 2024-04-20T21:32:13.373 INFO:tasks.workunit.client.1.smithi189.stdout:4/499: dread d2/f4 [0,4194304] 0 2024-04-20T21:32:13.380 INFO:tasks.workunit.client.0.smithi062.stdout:3/749: dwrite d0/d1/d10/d3a/d5e/d65/d88/fde [0,4194304] 0 2024-04-20T21:32:13.381 INFO:tasks.workunit.client.1.smithi189.stdout:1/487: creat d8/d21/fa5 x:0 0 0 2024-04-20T21:32:13.386 INFO:tasks.workunit.client.1.smithi189.stdout:9/501: getdents d9/d12/d3f/d91 0 2024-04-20T21:32:13.389 INFO:tasks.workunit.client.1.smithi189.stdout:2/466: mkdir d1/d11/d52/d8b 0 2024-04-20T21:32:13.389 INFO:tasks.workunit.client.0.smithi062.stdout:7/784: write f4 [1009296,94849] 0 2024-04-20T21:32:13.392 INFO:tasks.workunit.client.0.smithi062.stdout:4/887: dread da/dd/d15/daf/fda [0,4194304] 0 2024-04-20T21:32:13.392 INFO:tasks.workunit.client.0.smithi062.stdout:8/821: unlink d7/d25/d4f/f7e 0 2024-04-20T21:32:13.396 INFO:tasks.workunit.client.0.smithi062.stdout:5/797: getdents d8/d38/d7a/d4c/d7d/d9d/dac/d66/dae/d88 0 2024-04-20T21:32:13.398 INFO:tasks.workunit.client.0.smithi062.stdout:7/785: write d7/d3e/d33/d3d/dad/fb9 [1811534,53869] 0 2024-04-20T21:32:13.399 INFO:tasks.workunit.client.1.smithi189.stdout:1/488: dwrite d8/d21/d2c/d36/f52 [0,4194304] 0 2024-04-20T21:32:13.400 INFO:tasks.workunit.client.0.smithi062.stdout:3/750: symlink d0/d1c/d63/lf7 0 2024-04-20T21:32:13.404 INFO:tasks.workunit.client.0.smithi062.stdout:5/798: chown d8/f4f 299 1 2024-04-20T21:32:13.405 INFO:tasks.workunit.client.0.smithi062.stdout:7/786: write d7/f55 [836791,84665] 0 2024-04-20T21:32:13.405 INFO:tasks.workunit.client.0.smithi062.stdout:9/910: link d16/d3d/c71 d16/d2f/d5b/dab/c128 0 2024-04-20T21:32:13.409 INFO:tasks.workunit.client.0.smithi062.stdout:9/911: write d16/d2c/d62/dd1/d105/f109 [570821,98] 0 2024-04-20T21:32:13.412 INFO:tasks.workunit.client.1.smithi189.stdout:1/489: dwrite f6 [0,4194304] 0 2024-04-20T21:32:13.417 INFO:tasks.workunit.client.1.smithi189.stdout:3/499: rename d2/d3/d2c/d31/d90/d92/f9c to d2/d3/d2c/d30/d3f/fb3 0 2024-04-20T21:32:13.419 INFO:tasks.workunit.client.1.smithi189.stdout:8/598: mknod da/df/d22/d33/cd5 0 2024-04-20T21:32:13.427 INFO:tasks.workunit.client.1.smithi189.stdout:4/500: symlink d2/dd/d11/d14/lbc 0 2024-04-20T21:32:13.440 INFO:tasks.workunit.client.0.smithi062.stdout:7/787: creat d7/d26/da8/f111 x:0 0 0 2024-04-20T21:32:13.441 INFO:tasks.workunit.client.0.smithi062.stdout:8/822: getdents d7/da9 0 2024-04-20T21:32:13.442 INFO:tasks.workunit.client.1.smithi189.stdout:6/557: link d8/d2b/f87 d8/d12/daf/fba 0 2024-04-20T21:32:13.444 INFO:tasks.workunit.client.1.smithi189.stdout:6/558: read d8/da/dd/d35/d6d/d77/fac [3866399,105710] 0 2024-04-20T21:32:13.449 INFO:tasks.workunit.client.0.smithi062.stdout:8/823: symlink d7/d25/d3b/l10e 0 2024-04-20T21:32:13.453 INFO:tasks.workunit.client.0.smithi062.stdout:8/824: stat d7/d25/d3b/d58/d8b/d24/ff3 0 2024-04-20T21:32:13.455 INFO:tasks.workunit.client.0.smithi062.stdout:0/784: dread d7/d1c/d15/d17/d37/f5b [0,4194304] 0 2024-04-20T21:32:13.456 INFO:tasks.workunit.client.0.smithi062.stdout:7/788: write d7/d3e/d33/d3d/d4d/d54/dea/fa9 [3087153,105872] 0 2024-04-20T21:32:13.458 INFO:tasks.workunit.client.0.smithi062.stdout:8/825: unlink d7/d25/c86 0 2024-04-20T21:32:13.458 INFO:tasks.workunit.client.0.smithi062.stdout:2/844: dread d7/da/d15/d2f/d38/fbf [0,4194304] 0 2024-04-20T21:32:13.464 INFO:tasks.workunit.client.0.smithi062.stdout:7/789: write d7/d3e/d33/d3d/d4d/dce/ff8 [397936,69447] 0 2024-04-20T21:32:13.465 INFO:tasks.workunit.client.1.smithi189.stdout:7/559: truncate d5/f6 6453384 0 2024-04-20T21:32:13.469 INFO:tasks.workunit.client.1.smithi189.stdout:3/500: creat d2/d8/fb4 x:0 0 0 2024-04-20T21:32:13.469 INFO:tasks.workunit.client.1.smithi189.stdout:3/501: read d2/d8/f10 [5556428,124676] 0 2024-04-20T21:32:13.470 INFO:tasks.workunit.client.0.smithi062.stdout:2/845: rmdir d7/da/d15/d1b 39 2024-04-20T21:32:13.472 INFO:tasks.workunit.client.0.smithi062.stdout:7/790: fsync d7/d3e/d33/d3d/d4d/fc0 0 2024-04-20T21:32:13.473 INFO:tasks.workunit.client.0.smithi062.stdout:8/826: truncate d7/d25/d3b/d58/d8b/f3f 1375083 0 2024-04-20T21:32:13.474 INFO:tasks.workunit.client.1.smithi189.stdout:5/492: unlink d7/d9/d31/d36/c37 0 2024-04-20T21:32:13.475 INFO:tasks.workunit.client.1.smithi189.stdout:4/501: creat d2/dd/fbd x:0 0 0 2024-04-20T21:32:13.477 INFO:tasks.workunit.client.0.smithi062.stdout:9/912: dwrite d16/f32 [4194304,4194304] 0 2024-04-20T21:32:13.483 INFO:tasks.workunit.client.1.smithi189.stdout:0/620: write d3/df/d28/d2b/d2c/f83 [998806,111327] 0 2024-04-20T21:32:13.484 INFO:tasks.workunit.client.0.smithi062.stdout:8/827: write d7/d25/d3b/d58/d8b/d24/dbd/f109 [7675,92660] 0 2024-04-20T21:32:13.484 INFO:tasks.workunit.client.1.smithi189.stdout:7/560: dwrite d5/ff [0,4194304] 0 2024-04-20T21:32:13.486 INFO:tasks.workunit.client.1.smithi189.stdout:3/502: creat d2/d3/d2c/d30/d3f/d9a/fb5 x:0 0 0 2024-04-20T21:32:13.490 INFO:tasks.workunit.client.1.smithi189.stdout:5/493: mkdir d7/d9/d51/d70/d99 0 2024-04-20T21:32:13.497 INFO:tasks.workunit.client.0.smithi062.stdout:9/913: creat d16/d4e/f129 x:0 0 0 2024-04-20T21:32:13.498 INFO:tasks.workunit.client.0.smithi062.stdout:7/791: rename d7/d3e/d33/d3d/d4d/d83/c92 to d7/d26/dac/ddb/de8/c112 0 2024-04-20T21:32:13.499 INFO:tasks.workunit.client.1.smithi189.stdout:2/467: creat d1/f8c x:0 0 0 2024-04-20T21:32:13.501 INFO:tasks.workunit.client.1.smithi189.stdout:3/503: dread d2/d3/f48 [4194304,4194304] 0 2024-04-20T21:32:13.503 INFO:tasks.workunit.client.0.smithi062.stdout:9/914: fsync d16/d2c/d42/f70 0 2024-04-20T21:32:13.505 INFO:tasks.workunit.client.1.smithi189.stdout:1/490: write d8/d21/d2c/fa3 [443730,110114] 0 2024-04-20T21:32:13.505 INFO:tasks.workunit.client.0.smithi062.stdout:7/792: chown d7/d3e/d33/d3d/d4d/d54/d4f/l79 178240875 1 2024-04-20T21:32:13.506 INFO:tasks.workunit.client.1.smithi189.stdout:7/561: mknod d5/d59/cbd 0 2024-04-20T21:32:13.509 INFO:tasks.workunit.client.1.smithi189.stdout:9/502: link d9/d12/l23 d9/l9f 0 2024-04-20T21:32:13.509 INFO:tasks.workunit.client.1.smithi189.stdout:6/559: creat d8/d2b/fbb x:0 0 0 2024-04-20T21:32:13.510 INFO:tasks.workunit.client.0.smithi062.stdout:2/846: link d7/da/d15/d1b/d24/dee/f108 d7/da/d15/d2f/f115 0 2024-04-20T21:32:13.510 INFO:tasks.workunit.client.0.smithi062.stdout:9/915: write d16/d2c/d42/da6/da9/fc4 [3084171,70249] 0 2024-04-20T21:32:13.514 INFO:tasks.workunit.client.0.smithi062.stdout:9/916: creat d16/d2c/d62/f12a x:0 0 0 2024-04-20T21:32:13.517 INFO:tasks.workunit.client.1.smithi189.stdout:2/468: rename d1/d17/d2f/d55/f83 to d1/d11/d52/d8b/f8d 0 2024-04-20T21:32:13.519 INFO:tasks.workunit.client.0.smithi062.stdout:9/917: symlink d16/d4e/l12b 0 2024-04-20T21:32:13.520 INFO:tasks.workunit.client.0.smithi062.stdout:8/828: dwrite d7/d9/f18 [0,4194304] 0 2024-04-20T21:32:13.521 INFO:tasks.workunit.client.0.smithi062.stdout:2/847: creat d7/da/d15/d2f/d38/d10f/d107/dca/dd8/ddc/f116 x:0 0 0 2024-04-20T21:32:13.522 INFO:tasks.workunit.client.1.smithi189.stdout:0/621: dread d3/df/d5d/f7c [0,4194304] 0 2024-04-20T21:32:13.524 INFO:tasks.workunit.client.0.smithi062.stdout:8/829: readlink d7/d30/l8c 0 2024-04-20T21:32:13.524 INFO:tasks.workunit.client.0.smithi062.stdout:9/918: chown d16/d4e/d80/l89 14 1 2024-04-20T21:32:13.527 INFO:tasks.workunit.client.0.smithi062.stdout:2/848: mkdir d7/da/d15/d1b/d24/dee/d117 0 2024-04-20T21:32:13.527 INFO:tasks.workunit.client.1.smithi189.stdout:3/504: creat d2/d3/d65/fb6 x:0 0 0 2024-04-20T21:32:13.529 INFO:tasks.workunit.client.1.smithi189.stdout:1/491: creat d8/d9b/fa6 x:0 0 0 2024-04-20T21:32:13.530 INFO:tasks.workunit.client.0.smithi062.stdout:9/919: chown d16/d2c/d42/l57 93 1 2024-04-20T21:32:13.530 INFO:tasks.workunit.client.1.smithi189.stdout:3/505: chown d2/d3/d2c/d30/c9d 6 1 2024-04-20T21:32:13.531 INFO:tasks.workunit.client.1.smithi189.stdout:4/502: dwrite d2/f5a [0,4194304] 0 2024-04-20T21:32:13.531 INFO:tasks.workunit.client.1.smithi189.stdout:3/506: stat d2/d8d 0 2024-04-20T21:32:13.533 INFO:tasks.workunit.client.0.smithi062.stdout:2/849: symlink d7/da/d15/d2f/d38/d6a/l118 0 2024-04-20T21:32:13.533 INFO:tasks.workunit.client.1.smithi189.stdout:4/503: chown d2/d20 1809762 1 2024-04-20T21:32:13.533 INFO:tasks.workunit.client.1.smithi189.stdout:5/494: creat d7/d9/d51/d70/d99/f9a x:0 0 0 2024-04-20T21:32:13.535 INFO:tasks.workunit.client.1.smithi189.stdout:4/504: write d2/f5e [346670,66068] 0 2024-04-20T21:32:13.535 INFO:tasks.workunit.client.0.smithi062.stdout:9/920: fsync d16/d2f/d5b/fe0 0 2024-04-20T21:32:13.536 INFO:tasks.workunit.client.1.smithi189.stdout:7/562: stat d5/d59/l83 0 2024-04-20T21:32:13.542 INFO:tasks.workunit.client.0.smithi062.stdout:5/799: dread d8/d38/d7a/d4c/fc1 [0,4194304] 0 2024-04-20T21:32:13.547 INFO:tasks.workunit.client.1.smithi189.stdout:9/503: rename d9/dc/d17/d2b to d9/dc/d17/d95/da0 0 2024-04-20T21:32:13.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:13.546+0000 7f8fd3000700 1 -- 172.21.15.62:0/3305528671 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8fcc101250 msgr2=0x7f8fcc1016a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:32:13.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:13.546+0000 7f8fd3000700 1 --2- 172.21.15.62:0/3305528671 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8fcc101250 0x7f8fcc1016a0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f8fb8009910 tx=0x7f8fb8009c20 comp rx=0 tx=0).stop 2024-04-20T21:32:13.547 INFO:tasks.workunit.client.0.smithi062.stdout:2/850: rename d7/da/d15/d1b/d24/l35 to d7/da/d15/d1b/d24/l119 0 2024-04-20T21:32:13.547 INFO:tasks.workunit.client.0.smithi062.stdout:8/830: truncate d7/d9/f3a 2503910 0 2024-04-20T21:32:13.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:13.547+0000 7f8fd3000700 1 -- 172.21.15.62:0/3305528671 shutdown_connections 2024-04-20T21:32:13.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:13.547+0000 7f8fd3000700 1 --2- 172.21.15.62:0/3305528671 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8fcc101250 0x7f8fcc1016a0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:32:13.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:13.547+0000 7f8fd3000700 1 --2- 172.21.15.62:0/3305528671 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8fcc100050 0x7f8fcc100460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:32:13.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:13.547+0000 7f8fd3000700 1 -- 172.21.15.62:0/3305528671 >> 172.21.15.62:0/3305528671 conn(0x7f8fcc0fb5e0 msgr2=0x7f8fcc0fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:32:13.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:13.548+0000 7f8fd3000700 1 -- 172.21.15.62:0/3305528671 shutdown_connections 2024-04-20T21:32:13.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:13.549+0000 7f8fd3000700 1 -- 172.21.15.62:0/3305528671 wait complete. 2024-04-20T21:32:13.551 INFO:tasks.workunit.client.0.smithi062.stdout:2/851: stat d7/da/d15/d2f/d38/d10f/d107/fb8 0 2024-04-20T21:32:13.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:13.551+0000 7f8fd3000700 1 Processor -- start 2024-04-20T21:32:13.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:13.551+0000 7f8fd3000700 1 -- start start 2024-04-20T21:32:13.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:13.551+0000 7f8fd3000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8fcc100050 0x7f8fcc197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:32:13.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:13.551+0000 7f8fd3000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8fcc101250 0x7f8fcc198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:32:13.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:13.551+0000 7f8fd3000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fcc198670 con 0x7f8fcc100050 2024-04-20T21:32:13.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:13.551+0000 7f8fd3000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fcc1987b0 con 0x7f8fcc101250 2024-04-20T21:32:13.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:13.553+0000 7f8fd1c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8fcc100050 0x7f8fcc197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:32:13.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:13.553+0000 7f8fd1200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8fcc101250 0x7f8fcc198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:32:13.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:13.553+0000 7f8fd1c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8fcc100050 0x7f8fcc197b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:57414/0 (socket says 172.21.15.62:57414) 2024-04-20T21:32:13.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:13.553+0000 7f8fd1c00700 1 -- 172.21.15.62:0/2500135782 learned_addr learned my addr 172.21.15.62:0/2500135782 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:32:13.553 INFO:tasks.workunit.client.1.smithi189.stdout:2/469: mkdir d1/d11/d2d/d4f/d8e 0 2024-04-20T21:32:13.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:13.554+0000 7f8fd1c00700 1 -- 172.21.15.62:0/2500135782 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8fcc101250 msgr2=0x7f8fcc198050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:32:13.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:13.554+0000 7f8fd1c00700 1 --2- 172.21.15.62:0/2500135782 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8fcc101250 0x7f8fcc198050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:32:13.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:13.554+0000 7f8fd1c00700 1 -- 172.21.15.62:0/2500135782 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fb80095f0 con 0x7f8fcc100050 2024-04-20T21:32:13.554 INFO:tasks.workunit.client.1.smithi189.stdout:0/622: fdatasync d3/df/d30/d45/d90/fbf 0 2024-04-20T21:32:13.554 INFO:tasks.workunit.client.0.smithi062.stdout:5/800: symlink d8/d1e/d2c/l108 0 2024-04-20T21:32:13.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:13.555+0000 7f8fd1c00700 1 --2- 172.21.15.62:0/2500135782 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8fcc100050 0x7f8fcc197b10 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f8fc000b480 tx=0x7f8fc000b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:32:13.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:13.556+0000 7f8fca000700 1 -- 172.21.15.62:0/2500135782 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8fc00118c0 con 0x7f8fcc100050 2024-04-20T21:32:13.556 INFO:tasks.workunit.client.1.smithi189.stdout:1/492: dwrite d8/d9b/fa6 [0,4194304] 0 2024-04-20T21:32:13.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:13.556+0000 7f8fca000700 1 -- 172.21.15.62:0/2500135782 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f8fc0010380 con 0x7f8fcc100050 2024-04-20T21:32:13.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:13.556+0000 7f8fca000700 1 -- 172.21.15.62:0/2500135782 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8fc000f4f0 con 0x7f8fcc100050 2024-04-20T21:32:13.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:13.556+0000 7f8fd3000700 1 -- 172.21.15.62:0/2500135782 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fcc19d260 con 0x7f8fcc100050 2024-04-20T21:32:13.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:13.556+0000 7f8fd3000700 1 -- 172.21.15.62:0/2500135782 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fcc19d830 con 0x7f8fcc100050 2024-04-20T21:32:13.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:13.558+0000 7f8fd3000700 1 -- 172.21.15.62:0/2500135782 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fcc105330 con 0x7f8fcc100050 2024-04-20T21:32:13.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:13.560+0000 7f8fca000700 1 -- 172.21.15.62:0/2500135782 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 29) v1 ==== 92700+0+0 (secure 0 0 0) 0x7f8fc00104f0 con 0x7f8fcc100050 2024-04-20T21:32:13.560 INFO:tasks.workunit.client.1.smithi189.stdout:1/493: write d8/f86 [1751754,91188] 0 2024-04-20T21:32:13.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:13.561+0000 7f8fca000700 1 --2- 172.21.15.62:0/2500135782 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f8fb406ea70 0x7f8fb4070f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:32:13.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:13.561+0000 7f8fca000700 1 -- 172.21.15.62:0/2500135782 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f8fc008e970 con 0x7f8fcc100050 2024-04-20T21:32:13.563 INFO:tasks.workunit.client.1.smithi189.stdout:5/495: unlink d7/d14/d18/f66 0 2024-04-20T21:32:13.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:13.564+0000 7f8fd1200700 1 --2- 172.21.15.62:0/2500135782 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f8fb406ea70 0x7f8fb4070f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:32:13.564 INFO:tasks.workunit.client.0.smithi062.stdout:8/831: getdents d7/d25/d3b/d58/d8b/d24/d46 0 2024-04-20T21:32:13.564 INFO:tasks.workunit.client.0.smithi062.stdout:5/801: mkdir d8/d38/d7a/d109 0 2024-04-20T21:32:13.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:13.566+0000 7f8fd1200700 1 --2- 172.21.15.62:0/2500135782 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f8fb406ea70 0x7f8fb4070f20 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f8fb8006010 tx=0x7f8fb800b540 comp rx=0 tx=0).ready entity=mgr.14658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:32:13.565 INFO:tasks.workunit.client.1.smithi189.stdout:9/504: rename d9/d12/f29 to d9/d12/fa1 0 2024-04-20T21:32:13.567 INFO:tasks.workunit.client.1.smithi189.stdout:2/470: creat d1/d17/d2f/d4a/f8f x:0 0 0 2024-04-20T21:32:13.567 INFO:tasks.workunit.client.1.smithi189.stdout:5/496: chown d7/d9/c91 435262816 1 2024-04-20T21:32:13.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:13.569+0000 7f8fca000700 1 -- 172.21.15.62:0/2500135782 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187527 (secure 0 0 0) 0x7f8fc005b790 con 0x7f8fcc100050 2024-04-20T21:32:13.569 INFO:tasks.workunit.client.0.smithi062.stdout:9/921: dwrite d16/d2c/d62/dd1/d105/f11d [0,4194304] 0 2024-04-20T21:32:13.572 INFO:tasks.workunit.client.1.smithi189.stdout:1/494: mkdir d8/d1e/d54/da7 0 2024-04-20T21:32:13.572 INFO:tasks.workunit.client.1.smithi189.stdout:0/623: chown d3/df/d30/d45/fa1 4819 1 2024-04-20T21:32:13.576 INFO:tasks.workunit.client.1.smithi189.stdout:7/563: rename d5/d46/c90 to d5/d21/d4a/d52/d98/cbe 0 2024-04-20T21:32:13.576 INFO:tasks.workunit.client.1.smithi189.stdout:5/497: creat d7/d9/d28/f9b x:0 0 0 2024-04-20T21:32:13.577 INFO:tasks.workunit.client.0.smithi062.stdout:5/802: creat d8/d38/d7a/d4c/dcf/f10a x:0 0 0 2024-04-20T21:32:13.578 INFO:tasks.workunit.client.1.smithi189.stdout:0/624: chown d3/df/d28/d2b/d2c 194 1 2024-04-20T21:32:13.580 INFO:tasks.workunit.client.1.smithi189.stdout:4/505: creat d2/fbe x:0 0 0 2024-04-20T21:32:13.581 INFO:tasks.workunit.client.1.smithi189.stdout:2/471: mknod d1/d17/d2f/d74/c90 0 2024-04-20T21:32:13.582 INFO:tasks.workunit.client.1.smithi189.stdout:6/560: dwrite d8/d12/d4e/f50 [0,4194304] 0 2024-04-20T21:32:13.586 INFO:tasks.workunit.client.0.smithi062.stdout:5/803: chown d8/d38/d7a/d4c/d7d/d9d/dac/d2f/df3 1069838 1 2024-04-20T21:32:13.587 INFO:tasks.workunit.client.1.smithi189.stdout:2/472: truncate d1/d11/d58/f6e 4645304 0 2024-04-20T21:32:13.587 INFO:tasks.workunit.client.1.smithi189.stdout:3/507: write d2/d3/d2c/f6d [1278121,88944] 0 2024-04-20T21:32:13.588 INFO:tasks.workunit.client.1.smithi189.stdout:7/564: truncate d5/d21/d4a/d52/f7c 171022 0 2024-04-20T21:32:13.592 INFO:tasks.workunit.client.1.smithi189.stdout:6/561: fdatasync d8/d12/d30/d47/f52 0 2024-04-20T21:32:13.592 INFO:tasks.workunit.client.1.smithi189.stdout:4/506: creat d2/d22/da9/fbf x:0 0 0 2024-04-20T21:32:13.593 INFO:tasks.workunit.client.1.smithi189.stdout:8/599: sync 2024-04-20T21:32:13.594 INFO:tasks.workunit.client.1.smithi189.stdout:0/625: dread d3/df/d5d/d64/f6b [0,4194304] 0 2024-04-20T21:32:13.595 INFO:tasks.workunit.client.1.smithi189.stdout:9/505: rename d9/dc/d17/d95/da0/d55/f5a to d9/dc/fa2 0 2024-04-20T21:32:13.595 INFO:tasks.workunit.client.1.smithi189.stdout:5/498: link d7/d14/d18/c7e d7/c9c 0 2024-04-20T21:32:13.596 INFO:tasks.workunit.client.1.smithi189.stdout:3/508: dread - d2/d3/d2c/d31/d74/d7a/faf zero size 2024-04-20T21:32:13.600 INFO:tasks.workunit.client.1.smithi189.stdout:6/562: fsync d8/d2b/d49/f60 0 2024-04-20T21:32:13.602 INFO:tasks.workunit.client.1.smithi189.stdout:3/509: chown d2/d3/d2c/d31/d4c 3 1 2024-04-20T21:32:13.602 INFO:tasks.workunit.client.1.smithi189.stdout:6/563: read - d8/d12/d4e/fa6 zero size 2024-04-20T21:32:13.602 INFO:tasks.workunit.client.1.smithi189.stdout:1/495: dread d8/d1e/d28/d4f/f98 [0,4194304] 0 2024-04-20T21:32:13.603 INFO:tasks.workunit.client.1.smithi189.stdout:5/499: write d7/d9/d51/d70/d99/f9a [16540,25166] 0 2024-04-20T21:32:13.606 INFO:tasks.workunit.client.0.smithi062.stdout:9/922: dwrite d16/d3e/fa2 [0,4194304] 0 2024-04-20T21:32:13.609 INFO:tasks.workunit.client.1.smithi189.stdout:0/626: mkdir d3/df/d30/d45/dd3 0 2024-04-20T21:32:13.609 INFO:tasks.workunit.client.0.smithi062.stdout:9/923: fsync d16/d2c/d42/f70 0 2024-04-20T21:32:13.611 INFO:tasks.workunit.client.0.smithi062.stdout:9/924: write d16/d2c/d42/da6/fd2 [662480,84650] 0 2024-04-20T21:32:13.619 INFO:tasks.workunit.client.1.smithi189.stdout:8/600: rename da/df/d22/d33 to da/df/d13/dd6 0 2024-04-20T21:32:13.620 INFO:tasks.workunit.client.1.smithi189.stdout:0/627: fsync d3/df/d4c/fa4 0 2024-04-20T21:32:13.622 INFO:tasks.workunit.client.1.smithi189.stdout:3/510: mkdir d2/d3/d2c/d31/d74/db7 0 2024-04-20T21:32:13.624 INFO:tasks.workunit.client.1.smithi189.stdout:5/500: mkdir d7/d9/d30/d5a/d9d 0 2024-04-20T21:32:13.627 INFO:tasks.workunit.client.1.smithi189.stdout:8/601: stat da/d81/d91/fc4 0 2024-04-20T21:32:13.631 INFO:tasks.workunit.client.1.smithi189.stdout:2/473: dwrite d1/d11/d2d/f63 [0,4194304] 0 2024-04-20T21:32:13.631 INFO:tasks.workunit.client.1.smithi189.stdout:7/565: dwrite d5/d21/d4a/d52/fa7 [0,4194304] 0 2024-04-20T21:32:13.636 INFO:tasks.workunit.client.0.smithi062.stdout:5/804: unlink d8/d1e/d2c/d6c/f9b 0 2024-04-20T21:32:13.638 INFO:tasks.workunit.client.1.smithi189.stdout:3/511: truncate d2/d3/d2c/d30/d3f/f8a 902025 0 2024-04-20T21:32:13.638 INFO:tasks.workunit.client.0.smithi062.stdout:9/925: creat d16/d3d/d55/f12c x:0 0 0 2024-04-20T21:32:13.643 INFO:tasks.workunit.client.1.smithi189.stdout:1/496: creat d8/d1e/d28/fa8 x:0 0 0 2024-04-20T21:32:13.644 INFO:tasks.workunit.client.0.smithi062.stdout:9/926: mknod d16/d2c/d5e/c12d 0 2024-04-20T21:32:13.645 INFO:tasks.workunit.client.1.smithi189.stdout:5/501: mknod d7/d9/d30/d5a/c9e 0 2024-04-20T21:32:13.645 INFO:tasks.workunit.client.1.smithi189.stdout:1/497: chown f1 46 1 2024-04-20T21:32:13.647 INFO:tasks.workunit.client.1.smithi189.stdout:5/502: write d7/d9/f42 [543530,86787] 0 2024-04-20T21:32:13.649 INFO:tasks.workunit.client.0.smithi062.stdout:9/927: mknod d16/d2c/d5e/c12e 0 2024-04-20T21:32:13.649 INFO:tasks.workunit.client.1.smithi189.stdout:3/512: unlink d2/d3/d2c/d31/d74/d7a/l93 0 2024-04-20T21:32:13.650 INFO:tasks.workunit.client.1.smithi189.stdout:7/566: creat d5/db4/d71/d7a/db0/fbf x:0 0 0 2024-04-20T21:32:13.658 INFO:tasks.workunit.client.1.smithi189.stdout:0/628: rename d3/df/d28/d2b/d2c/d73 to d3/df/d4c/dd4 0 2024-04-20T21:32:13.659 INFO:tasks.workunit.client.1.smithi189.stdout:9/506: dread d9/dc/d17/d36/f57 [0,4194304] 0 2024-04-20T21:32:13.661 INFO:tasks.workunit.client.1.smithi189.stdout:5/503: unlink d7/d9/d31/d36/d5f/f75 0 2024-04-20T21:32:13.663 INFO:tasks.workunit.client.1.smithi189.stdout:6/564: dwrite d8/f9b [0,4194304] 0 2024-04-20T21:32:13.666 INFO:tasks.workunit.client.1.smithi189.stdout:7/567: getdents d5/d21/d4a/d52/d98 0 2024-04-20T21:32:13.667 INFO:tasks.workunit.client.0.smithi062.stdout:9/928: dwrite d16/d2f/fc8 [0,4194304] 0 2024-04-20T21:32:13.674 INFO:tasks.workunit.client.1.smithi189.stdout:3/513: link d2/d3/d2c/d30/d3f/d9a/fb5 d2/d3/d2c/d30/d3f/d9a/db0/fb8 0 2024-04-20T21:32:13.675 INFO:tasks.workunit.client.0.smithi062.stdout:9/929: creat d16/d2f/d69/f12f x:0 0 0 2024-04-20T21:32:13.675 INFO:tasks.workunit.client.1.smithi189.stdout:1/498: dwrite d8/d1e/d43/d58/f72 [0,4194304] 0 2024-04-20T21:32:13.677 INFO:tasks.workunit.client.1.smithi189.stdout:3/514: read d2/f5d [180120,124140] 0 2024-04-20T21:32:13.681 INFO:tasks.workunit.client.1.smithi189.stdout:3/515: dread - d2/d3/d25/f95 zero size 2024-04-20T21:32:13.687 INFO:tasks.workunit.client.1.smithi189.stdout:6/565: truncate d8/d2b/f3a 1479556 0 2024-04-20T21:32:13.688 INFO:tasks.workunit.client.0.smithi062.stdout:9/930: symlink d16/d3d/d55/d6e/d7d/l130 0 2024-04-20T21:32:13.688 INFO:tasks.workunit.client.1.smithi189.stdout:1/499: mknod d8/d1e/d54/ca9 0 2024-04-20T21:32:13.690 INFO:tasks.workunit.client.0.smithi062.stdout:9/931: write d16/d4e/ffb [483670,116613] 0 2024-04-20T21:32:13.692 INFO:tasks.workunit.client.1.smithi189.stdout:8/602: write da/df/d13/d63/f7f [33292,22499] 0 2024-04-20T21:32:13.699 INFO:tasks.workunit.client.0.smithi062.stdout:8/832: link d7/d9/f3a d7/d25/d76/f10f 0 2024-04-20T21:32:13.701 INFO:tasks.workunit.client.1.smithi189.stdout:8/603: rename da/df/d13/f14 to da/df/d20/dc9/fd7 0 2024-04-20T21:32:13.703 INFO:tasks.workunit.client.1.smithi189.stdout:7/568: dwrite d5/db4/d71/fa4 [0,4194304] 0 2024-04-20T21:32:13.705 INFO:tasks.workunit.client.0.smithi062.stdout:5/805: write f4 [1154627,29301] 0 2024-04-20T21:32:13.710 INFO:tasks.workunit.client.0.smithi062.stdout:5/806: mknod d8/d38/d7a/d4c/dcf/c10b 0 2024-04-20T21:32:13.711 INFO:tasks.workunit.client.0.smithi062.stdout:5/807: chown d8/d38/d7a/d4c/d7d 25762322 1 2024-04-20T21:32:13.715 INFO:tasks.workunit.client.1.smithi189.stdout:7/569: truncate d5/d21/d4a/f4d 114 0 2024-04-20T21:32:13.717 INFO:tasks.workunit.client.1.smithi189.stdout:7/570: readlink d5/d46/l95 0 2024-04-20T21:32:13.721 INFO:tasks.workunit.client.1.smithi189.stdout:2/474: dread d1/f7 [4194304,4194304] 0 2024-04-20T21:32:13.722 INFO:tasks.workunit.client.0.smithi062.stdout:5/808: read d8/d38/d7a/d4c/d7d/d9d/dac/f39 [25219,42491] 0 2024-04-20T21:32:13.724 INFO:tasks.workunit.client.1.smithi189.stdout:7/571: mknod d5/d21/d97/da3/cc0 0 2024-04-20T21:32:13.725 INFO:tasks.workunit.client.1.smithi189.stdout:1/500: dwrite d8/df/d50/f8d [0,4194304] 0 2024-04-20T21:32:13.731 INFO:tasks.workunit.client.0.smithi062.stdout:1/863: sync 2024-04-20T21:32:13.734 INFO:tasks.workunit.client.0.smithi062.stdout:1/864: write dc/d15/d75/d94/d100/f117 [392179,67392] 0 2024-04-20T21:32:13.735 INFO:tasks.workunit.client.1.smithi189.stdout:9/507: dwrite d9/d12/d15/f30 [0,4194304] 0 2024-04-20T21:32:13.738 INFO:tasks.workunit.client.1.smithi189.stdout:7/572: unlink d5/d46/c68 0 2024-04-20T21:32:13.741 INFO:tasks.workunit.client.1.smithi189.stdout:8/604: dwrite da/df/d13/d63/f7f [0,4194304] 0 2024-04-20T21:32:13.741 INFO:tasks.workunit.client.0.smithi062.stdout:5/809: link d8/d38/d7a/d4c/d7d/d9d/dac/d66/dae/d88/f8c d8/d38/d7a/d4c/da7/f10c 0 2024-04-20T21:32:13.744 INFO:tasks.workunit.client.0.smithi062.stdout:1/865: truncate dc/d15/d75/d96/dd4/d3a/da2/de4/d84/dbe/d10d/f12d 551613 0 2024-04-20T21:32:13.746 INFO:tasks.workunit.client.0.smithi062.stdout:1/866: stat dc/d15/c68 0 2024-04-20T21:32:13.747 INFO:tasks.workunit.client.1.smithi189.stdout:6/566: write d8/d2b/f87 [441203,57973] 0 2024-04-20T21:32:13.751 INFO:tasks.workunit.client.1.smithi189.stdout:6/567: stat d8/d12/d30/d47/f52 0 2024-04-20T21:32:13.753 INFO:tasks.workunit.client.1.smithi189.stdout:0/629: dwrite d3/df/d30/d45/d60/f81 [0,4194304] 0 2024-04-20T21:32:13.757 INFO:tasks.workunit.client.1.smithi189.stdout:2/475: dwrite d1/d17/f2b [0,4194304] 0 2024-04-20T21:32:13.760 INFO:tasks.workunit.client.1.smithi189.stdout:8/605: creat da/d1f/fd8 x:0 0 0 2024-04-20T21:32:13.760 INFO:tasks.workunit.client.1.smithi189.stdout:9/508: creat d9/d12/d3f/d91/fa3 x:0 0 0 2024-04-20T21:32:13.761 INFO:tasks.workunit.client.1.smithi189.stdout:1/501: link d8/d21/d2c/l79 d8/d1e/d28/laa 0 2024-04-20T21:32:13.765 INFO:tasks.workunit.client.1.smithi189.stdout:6/568: symlink d8/d2b/d49/d54/d6e/lbc 0 2024-04-20T21:32:13.765 INFO:tasks.workunit.client.1.smithi189.stdout:3/516: dread d2/d3/d25/f5b [0,4194304] 0 2024-04-20T21:32:13.768 INFO:tasks.workunit.client.1.smithi189.stdout:7/573: creat d5/d1d/d77/fc1 x:0 0 0 2024-04-20T21:32:13.770 INFO:tasks.workunit.client.1.smithi189.stdout:6/569: write d8/d10/d78/f80 [983880,1020] 0 2024-04-20T21:32:13.772 INFO:tasks.workunit.client.1.smithi189.stdout:1/502: symlink d8/d21/d2c/lab 0 2024-04-20T21:32:13.775 INFO:tasks.workunit.client.1.smithi189.stdout:3/517: write d2/d3/d2c/d30/d3f/fa5 [4899972,36224] 0 2024-04-20T21:32:13.776 INFO:tasks.workunit.client.1.smithi189.stdout:8/606: truncate da/df/d3c/d68/d8b/fcf 550863 0 2024-04-20T21:32:13.776 INFO:tasks.workunit.client.1.smithi189.stdout:2/476: link d1/d17/c18 d1/d17/d2f/d4a/c91 0 2024-04-20T21:32:13.780 INFO:tasks.workunit.client.0.smithi062.stdout:1/867: dwrite dc/d15/d75/d96/dd4/d3a/f46 [0,4194304] 0 2024-04-20T21:32:13.783 INFO:tasks.workunit.client.1.smithi189.stdout:1/503: write d8/d21/d2c/d36/f52 [1531927,42663] 0 2024-04-20T21:32:13.786 INFO:tasks.workunit.client.0.smithi062.stdout:1/868: mknod dc/d15/d75/d96/dd4/d78/dab/c13f 0 2024-04-20T21:32:13.787 INFO:tasks.workunit.client.1.smithi189.stdout:8/607: truncate da/d1f/d2a/f58 1008899 0 2024-04-20T21:32:13.790 INFO:tasks.workunit.client.1.smithi189.stdout:7/574: link d5/db4/ca0 d5/d59/cc2 0 2024-04-20T21:32:13.792 INFO:tasks.workunit.client.1.smithi189.stdout:8/608: write da/d1f/fa1 [218584,98981] 0 2024-04-20T21:32:13.800 INFO:tasks.workunit.client.0.smithi062.stdout:5/810: dread d8/d38/d7a/d4c/d7d/d9d/dac/d66/daa/fab [0,4194304] 0 2024-04-20T21:32:13.801 INFO:tasks.workunit.client.0.smithi062.stdout:1/869: dread dc/d15/d20/d28/d83/dc0/f124 [0,4194304] 0 2024-04-20T21:32:13.803 INFO:tasks.workunit.client.1.smithi189.stdout:3/518: getdents d2/d3/d88 0 2024-04-20T21:32:13.803 INFO:tasks.workunit.client.1.smithi189.stdout:9/509: dwrite d9/d12/d3b/f46 [0,4194304] 0 2024-04-20T21:32:13.804 INFO:tasks.workunit.client.1.smithi189.stdout:7/575: rename d5/d21/d25/f66 to d5/d1d/d77/db8/fc3 0 2024-04-20T21:32:13.805 INFO:tasks.workunit.client.0.smithi062.stdout:5/811: dread - d8/d38/d7a/fc6 zero size 2024-04-20T21:32:13.806 INFO:tasks.workunit.client.1.smithi189.stdout:6/570: dread d8/d2b/f57 [0,4194304] 0 2024-04-20T21:32:13.806 INFO:tasks.workunit.client.1.smithi189.stdout:3/519: dread - d2/d3/d2c/d31/d4c/fa3 zero size 2024-04-20T21:32:13.807 INFO:tasks.workunit.client.0.smithi062.stdout:1/870: stat dc/d15/d75/d94/d100/c115 0 2024-04-20T21:32:13.815 INFO:tasks.workunit.client.1.smithi189.stdout:1/504: link d8/df/l16 d8/d21/d2c/d36/lac 0 2024-04-20T21:32:13.818 INFO:tasks.workunit.client.1.smithi189.stdout:0/630: write d3/df/d4c/f69 [2483709,30229] 0 2024-04-20T21:32:13.819 INFO:tasks.workunit.client.1.smithi189.stdout:5/504: sync 2024-04-20T21:32:13.819 INFO:tasks.workunit.client.0.smithi062.stdout:5/812: dread d8/d38/d7a/d4c/dcf/ff2 [0,4194304] 0 2024-04-20T21:32:13.820 INFO:tasks.workunit.client.1.smithi189.stdout:4/507: sync 2024-04-20T21:32:13.823 INFO:tasks.workunit.client.0.smithi062.stdout:1/871: rename dc/d15/cf3 to dc/d15/d75/d96/dd4/d3a/da2/de4/d84/dbe/d10d/d110/c140 0 2024-04-20T21:32:13.824 INFO:tasks.workunit.client.1.smithi189.stdout:3/520: rename d2/d3/d2c/d31/d74/d7a/faf to d2/d3/d2c/d30/d3f/fb9 0 2024-04-20T21:32:13.828 INFO:tasks.workunit.client.0.smithi062.stdout:5/813: creat d8/f10d x:0 0 0 2024-04-20T21:32:13.829 INFO:tasks.workunit.client.1.smithi189.stdout:4/508: creat d2/d64/fc0 x:0 0 0 2024-04-20T21:32:13.831 INFO:tasks.workunit.client.1.smithi189.stdout:2/477: truncate d1/d17/d2f/f72 1004260 0 2024-04-20T21:32:13.831 INFO:tasks.workunit.client.1.smithi189.stdout:7/576: dwrite d5/db4/d71/fa4 [0,4194304] 0 2024-04-20T21:32:13.831 INFO:tasks.workunit.client.1.smithi189.stdout:9/510: dwrite d9/d12/d3f/d91/fa3 [0,4194304] 0 2024-04-20T21:32:13.836 INFO:tasks.workunit.client.1.smithi189.stdout:8/609: write da/df/d20/d2c/f96 [284437,53126] 0 2024-04-20T21:32:13.838 INFO:tasks.workunit.client.1.smithi189.stdout:9/511: chown d9/d12/l48 128354 1 2024-04-20T21:32:13.839 INFO:tasks.workunit.client.1.smithi189.stdout:6/571: dwrite d8/d10/f36 [0,4194304] 0 2024-04-20T21:32:13.841 INFO:tasks.workunit.client.0.smithi062.stdout:1/872: getdents dc/d15/d75/d96/dd4/d3a/da2/de4/d84/dbe/d10d 0 2024-04-20T21:32:13.845 INFO:tasks.workunit.client.1.smithi189.stdout:2/478: readlink d1/d17/d2f/d55/l5b 0 2024-04-20T21:32:13.848 INFO:tasks.workunit.client.1.smithi189.stdout:6/572: fsync d8/d10/f2d 0 2024-04-20T21:32:13.857 INFO:tasks.workunit.client.0.smithi062.stdout:1/873: getdents dc/d15/d75/d96/dd4/d3a/da2 0 2024-04-20T21:32:13.858 INFO:tasks.workunit.client.1.smithi189.stdout:1/505: dwrite d8/d21/d2c/d36/f52 [0,4194304] 0 2024-04-20T21:32:13.859 INFO:tasks.workunit.client.1.smithi189.stdout:9/512: mkdir d9/dc/d17/d95/da0/d52/d82/da4 0 2024-04-20T21:32:13.860 INFO:tasks.workunit.client.1.smithi189.stdout:5/505: truncate d7/d9/d31/f57 290294 0 2024-04-20T21:32:13.861 INFO:tasks.workunit.client.1.smithi189.stdout:2/479: mkdir d1/d11/d2d/d4f/d92 0 2024-04-20T21:32:13.861 INFO:tasks.workunit.client.1.smithi189.stdout:4/509: write d2/dd/d11/d14/f3f [1142174,72000] 0 2024-04-20T21:32:13.864 INFO:tasks.workunit.client.1.smithi189.stdout:5/506: fsync d7/d35/d48/f97 0 2024-04-20T21:32:13.878 INFO:tasks.workunit.client.1.smithi189.stdout:7/577: dread d5/f1f [0,4194304] 0 2024-04-20T21:32:13.880 INFO:tasks.workunit.client.1.smithi189.stdout:3/521: dread d2/d3/d2c/d30/f68 [0,4194304] 0 2024-04-20T21:32:13.881 INFO:tasks.workunit.client.1.smithi189.stdout:3/522: chown d2/d3/d2c/f6b 41875854 1 2024-04-20T21:32:13.885 INFO:tasks.workunit.client.1.smithi189.stdout:1/506: write d8/d1e/d28/d2b/f76 [216766,107956] 0 2024-04-20T21:32:13.886 INFO:tasks.workunit.client.0.smithi062.stdout:1/874: dwrite dc/d15/d75/d96/dd4/d3a/f4d [0,4194304] 0 2024-04-20T21:32:13.886 INFO:tasks.workunit.client.0.smithi062.stdout:5/814: dread d8/d38/d7a/d4c/d7d/d9d/dac/d66/dae/f64 [0,4194304] 0 2024-04-20T21:32:13.886 INFO:tasks.workunit.client.1.smithi189.stdout:2/480: stat d1/d17/c18 0 2024-04-20T21:32:13.889 INFO:tasks.workunit.client.1.smithi189.stdout:9/513: creat d9/dc/d17/d36/d5d/fa5 x:0 0 0 2024-04-20T21:32:13.894 INFO:tasks.workunit.client.0.smithi062.stdout:1/875: chown dc/c11 1608 1 2024-04-20T21:32:13.896 INFO:tasks.workunit.client.1.smithi189.stdout:7/578: dwrite d5/ff [4194304,4194304] 0 2024-04-20T21:32:13.896 INFO:tasks.workunit.client.1.smithi189.stdout:3/523: creat d2/d8d/fba x:0 0 0 2024-04-20T21:32:13.897 INFO:tasks.workunit.client.0.smithi062.stdout:5/815: rename d8/d38/d7a/d4c/d7d/d9d/dac/d2f/d8e/dd6/dfe/l106 to d8/d38/d7a/d4c/dcf/l10e 0 2024-04-20T21:32:13.898 INFO:tasks.workunit.client.1.smithi189.stdout:3/524: stat d2/fd 0 2024-04-20T21:32:13.899 INFO:tasks.workunit.client.0.smithi062.stdout:1/876: creat dc/d15/d20/d28/d76/f141 x:0 0 0 2024-04-20T21:32:13.900 INFO:tasks.workunit.client.1.smithi189.stdout:8/610: link da/df/d13/f5f da/df/d20/d2c/d9a/dce/fd9 0 2024-04-20T21:32:13.901 INFO:tasks.workunit.client.1.smithi189.stdout:1/507: rmdir d8/df 39 2024-04-20T21:32:13.901 INFO:tasks.workunit.client.1.smithi189.stdout:2/481: unlink d1/d11/d58/d6b/f8a 0 2024-04-20T21:32:13.902 INFO:tasks.workunit.client.0.smithi062.stdout:1/877: write dc/d15/d75/d94/d100/f117 [680691,89791] 0 2024-04-20T21:32:13.902 INFO:tasks.workunit.client.1.smithi189.stdout:9/514: mkdir d9/dc/d17/d95/da6 0 2024-04-20T21:32:13.903 INFO:tasks.workunit.client.1.smithi189.stdout:2/482: truncate d1/f8c 119189 0 2024-04-20T21:32:13.907 INFO:tasks.workunit.client.1.smithi189.stdout:9/515: fsync d9/dc/d17/d95/da0/d56/f86 0 2024-04-20T21:32:13.907 INFO:tasks.workunit.client.1.smithi189.stdout:6/573: link d8/d12/d4e/ca1 d8/d12/d99/cbd 0 2024-04-20T21:32:13.907 INFO:tasks.workunit.client.1.smithi189.stdout:3/525: creat d2/d3/d65/fbb x:0 0 0 2024-04-20T21:32:13.911 INFO:tasks.workunit.client.1.smithi189.stdout:6/574: chown d8/da/dd/f71 7496970 1 2024-04-20T21:32:13.911 INFO:tasks.workunit.client.1.smithi189.stdout:8/611: truncate da/de/d1e/d21/d87/fd1 770368 0 2024-04-20T21:32:13.911 INFO:tasks.workunit.client.1.smithi189.stdout:3/526: readlink d2/d3/d2c/d30/l66 0 2024-04-20T21:32:13.912 INFO:tasks.workunit.client.1.smithi189.stdout:2/483: mknod d1/d11/c93 0 2024-04-20T21:32:13.913 INFO:tasks.workunit.client.1.smithi189.stdout:8/612: stat da/de/f28 0 2024-04-20T21:32:13.914 INFO:tasks.workunit.client.1.smithi189.stdout:5/507: getdents d7 0 2024-04-20T21:32:13.917 INFO:tasks.workunit.client.1.smithi189.stdout:2/484: write d1/d17/f35 [6121358,103523] 0 2024-04-20T21:32:13.917 INFO:tasks.workunit.client.1.smithi189.stdout:6/575: dread - d8/d2b/d49/d54/d6e/fb4 zero size 2024-04-20T21:32:13.917 INFO:tasks.workunit.client.1.smithi189.stdout:9/516: creat d9/dc/d17/d95/da0/d52/d82/fa7 x:0 0 0 2024-04-20T21:32:13.922 INFO:tasks.workunit.client.0.smithi062.stdout:5/816: truncate d8/d10/d3e/d8a/d9e/fc9 728894 0 2024-04-20T21:32:13.922 INFO:tasks.workunit.client.1.smithi189.stdout:5/508: creat d7/d9/d28/f9f x:0 0 0 2024-04-20T21:32:13.923 INFO:tasks.workunit.client.1.smithi189.stdout:2/485: write d1/d17/d37/f75 [130341,29138] 0 2024-04-20T21:32:13.924 INFO:tasks.workunit.client.1.smithi189.stdout:9/517: truncate d9/d12/f89 406591 0 2024-04-20T21:32:13.925 INFO:tasks.workunit.client.1.smithi189.stdout:6/576: mknod d8/da/dd/d35/d6d/cbe 0 2024-04-20T21:32:13.925 INFO:tasks.workunit.client.0.smithi062.stdout:5/817: write d8/d10/d3e/d8a/fb9 [5774437,119799] 0 2024-04-20T21:32:13.927 INFO:tasks.workunit.client.1.smithi189.stdout:2/486: dread - d1/d17/d2f/d55/f81 zero size 2024-04-20T21:32:13.927 INFO:tasks.workunit.client.0.smithi062.stdout:1/878: dread dc/d15/d20/d28/d83/f10c [0,4194304] 0 2024-04-20T21:32:13.928 INFO:tasks.workunit.client.1.smithi189.stdout:1/508: dwrite d8/f14 [0,4194304] 0 2024-04-20T21:32:13.928 INFO:tasks.workunit.client.1.smithi189.stdout:8/613: creat da/df/d20/d2c/d9a/dce/fda x:0 0 0 2024-04-20T21:32:13.931 INFO:tasks.workunit.client.0.smithi062.stdout:1/879: mkdir dc/d15/df6/d142 0 2024-04-20T21:32:13.936 INFO:tasks.workunit.client.0.smithi062.stdout:5/818: fdatasync d8/d38/d7a/d4c/d7d/d9d/dac/d66/dae/fa3 0 2024-04-20T21:32:13.940 INFO:tasks.workunit.client.0.smithi062.stdout:5/819: readlink d8/l58 0 2024-04-20T21:32:13.940 INFO:tasks.workunit.client.1.smithi189.stdout:2/487: getdents d1/d11/d58 0 2024-04-20T21:32:13.941 INFO:tasks.workunit.client.1.smithi189.stdout:8/614: creat da/df/d13/dd6/d59/fdb x:0 0 0 2024-04-20T21:32:13.941 INFO:tasks.workunit.client.0.smithi062.stdout:1/880: mkdir dc/d15/d20/d28/d83/d143 0 2024-04-20T21:32:13.943 INFO:tasks.workunit.client.1.smithi189.stdout:9/518: dread d9/d12/d15/d1a/f27 [0,4194304] 0 2024-04-20T21:32:13.944 INFO:tasks.workunit.client.0.smithi062.stdout:5/820: write d8/d1e/d2c/fa2 [720458,59616] 0 2024-04-20T21:32:13.948 INFO:tasks.workunit.client.0.smithi062.stdout:1/881: unlink dc/d15/d20/d28/d83/fa5 0 2024-04-20T21:32:13.952 INFO:tasks.workunit.client.1.smithi189.stdout:2/488: creat d1/d17/d2f/d55/f94 x:0 0 0 2024-04-20T21:32:13.953 INFO:tasks.workunit.client.1.smithi189.stdout:3/527: dwrite d2/d3/d2c/d31/fa7 [0,4194304] 0 2024-04-20T21:32:13.954 INFO:tasks.workunit.client.1.smithi189.stdout:9/519: fsync d9/dc/fa2 0 2024-04-20T21:32:13.958 INFO:tasks.workunit.client.1.smithi189.stdout:8/615: mkdir da/df/d3c/d50/ddc 0 2024-04-20T21:32:13.961 INFO:tasks.workunit.client.1.smithi189.stdout:2/489: dread - d1/d17/d2f/d55/f94 zero size 2024-04-20T21:32:13.961 INFO:tasks.workunit.client.1.smithi189.stdout:1/509: dwrite d8/d1e/d28/fa8 [0,4194304] 0 2024-04-20T21:32:13.962 INFO:tasks.workunit.client.1.smithi189.stdout:6/577: dwrite d8/da/dd/d35/f31 [4194304,4194304] 0 2024-04-20T21:32:13.967 INFO:tasks.workunit.client.1.smithi189.stdout:3/528: rmdir d2/d3/d65/d82 0 2024-04-20T21:32:13.968 INFO:tasks.workunit.client.0.smithi062.stdout:5/821: dwrite d8/d1e/d2c/fa2 [0,4194304] 0 2024-04-20T21:32:13.968 INFO:tasks.workunit.client.1.smithi189.stdout:8/616: mkdir da/df/d20/d2c/d9a/ddd 0 2024-04-20T21:32:13.974 INFO:tasks.workunit.client.0.smithi062.stdout:1/882: dwrite dc/d15/d75/d94/d100/f117 [0,4194304] 0 2024-04-20T21:32:13.974 INFO:tasks.workunit.client.0.smithi062.stdout:1/883: chown dc/d15/d75 2 1 2024-04-20T21:32:13.974 INFO:tasks.workunit.client.1.smithi189.stdout:1/510: chown d8/df/f92 1 1 2024-04-20T21:32:13.975 INFO:tasks.workunit.client.1.smithi189.stdout:8/617: chown da/df/d20/d2c/c53 14939913 1 2024-04-20T21:32:13.977 INFO:tasks.workunit.client.1.smithi189.stdout:6/578: rename d8/da/dd/d2a/l82 to d8/d10/lbf 0 2024-04-20T21:32:13.978 INFO:tasks.workunit.client.0.smithi062.stdout:5/822: mkdir d8/d38/d7a/d4c/d7d/d9d/d10f 0 2024-04-20T21:32:13.978 INFO:tasks.workunit.client.1.smithi189.stdout:1/511: write d8/d21/d2c/f62 [332634,4445] 0 2024-04-20T21:32:13.979 INFO:tasks.workunit.client.1.smithi189.stdout:0/631: sync 2024-04-20T21:32:13.982 INFO:tasks.workunit.client.1.smithi189.stdout:5/509: dread d7/d14/d18/f53 [0,4194304] 0 2024-04-20T21:32:13.982 INFO:tasks.workunit.client.1.smithi189.stdout:2/490: creat d1/d17/d37/d89/f95 x:0 0 0 2024-04-20T21:32:13.982 INFO:tasks.workunit.client.0.smithi062.stdout:1/884: getdents dc/d15/d20/d28/d83/dc0/d132 0 2024-04-20T21:32:13.983 INFO:tasks.workunit.client.1.smithi189.stdout:7/579: truncate d5/d46/f4c 2166318 0 2024-04-20T21:32:13.987 INFO:tasks.workunit.client.1.smithi189.stdout:4/510: dwrite d2/dd/d11/f73 [0,4194304] 0 2024-04-20T21:32:13.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:13 smithi189 ceph-mon[21562]: from='client.14712 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:32:13.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:13 smithi189 ceph-mon[21562]: pgmap v21: 65 pgs: 65 active+clean; 2.3 GiB data, 6.7 GiB used, 530 GiB / 536 GiB avail; 62 MiB/s rd, 213 MiB/s wr, 386 op/s 2024-04-20T21:32:13.988 INFO:tasks.workunit.client.0.smithi062.stdout:5/823: unlink d8/d38/d7a/d4c/d7d/d9d/dac/d66/dae/faf 0 2024-04-20T21:32:13.991 INFO:tasks.workunit.client.1.smithi189.stdout:2/491: fdatasync d1/d17/d2f/d55/f61 0 2024-04-20T21:32:13.991 INFO:tasks.workunit.client.1.smithi189.stdout:7/580: rmdir d5/d1d/d77/db8 39 2024-04-20T21:32:13.993 INFO:tasks.workunit.client.0.smithi062.stdout:5/824: creat d8/d38/f110 x:0 0 0 2024-04-20T21:32:13.994 INFO:tasks.workunit.client.1.smithi189.stdout:0/632: mknod d3/df/d5d/cd5 0 2024-04-20T21:32:13.995 INFO:tasks.workunit.client.0.smithi062.stdout:1/885: getdents dc/d27/d32/df8 0 2024-04-20T21:32:13.998 INFO:tasks.workunit.client.1.smithi189.stdout:6/579: rename d8/d12/f44 to d8/d12/d30/fc0 0 2024-04-20T21:32:14.000 INFO:tasks.workunit.client.1.smithi189.stdout:0/633: fsync d3/df/d5d/d64/f6b 0 2024-04-20T21:32:14.002 INFO:tasks.workunit.client.1.smithi189.stdout:1/512: rename d8/d9b/la1 to d8/d1e/d28/d4f/lad 0 2024-04-20T21:32:14.002 INFO:tasks.workunit.client.0.smithi062.stdout:1/886: creat dc/d15/d75/d94/d100/f144 x:0 0 0 2024-04-20T21:32:14.003 INFO:tasks.workunit.client.1.smithi189.stdout:2/492: truncate d1/d11/d52/d6f/f5a 831254 0 2024-04-20T21:32:14.005 INFO:tasks.workunit.client.0.smithi062.stdout:5/825: creat d8/d1e/d2c/f111 x:0 0 0 2024-04-20T21:32:14.008 INFO:tasks.workunit.client.1.smithi189.stdout:3/529: dwrite d2/d8/fb4 [0,4194304] 0 2024-04-20T21:32:14.009 INFO:tasks.workunit.client.0.smithi062.stdout:1/887: mknod dc/d70/c145 0 2024-04-20T21:32:14.013 INFO:tasks.workunit.client.0.smithi062.stdout:5/826: creat d8/d38/d7a/d4c/d7d/d9d/dac/f112 x:0 0 0 2024-04-20T21:32:14.014 INFO:tasks.workunit.client.1.smithi189.stdout:7/581: truncate d5/d21/f32 2566412 0 2024-04-20T21:32:14.014 INFO:tasks.workunit.client.1.smithi189.stdout:9/520: truncate d9/d12/d15/f30 849136 0 2024-04-20T21:32:14.019 INFO:tasks.workunit.client.1.smithi189.stdout:6/580: symlink d8/d10/d78/d7e/lc1 0 2024-04-20T21:32:14.019 INFO:tasks.workunit.client.0.smithi062.stdout:5/827: stat d8/d38/d7a/d4c/da7/f10c 0 2024-04-20T21:32:14.024 INFO:tasks.workunit.client.1.smithi189.stdout:5/510: dwrite d7/d35/f74 [0,4194304] 0 2024-04-20T21:32:14.024 INFO:tasks.workunit.client.1.smithi189.stdout:9/521: dread - d9/dc/d69/f7f zero size 2024-04-20T21:32:14.025 INFO:tasks.workunit.client.1.smithi189.stdout:3/530: rmdir d2/d1c/d5c 39 2024-04-20T21:32:14.026 INFO:tasks.workunit.client.1.smithi189.stdout:7/582: rename d5/d21/d35/l4f to d5/d59/lc4 0 2024-04-20T21:32:14.026 INFO:tasks.workunit.client.0.smithi062.stdout:5/828: rename d8/d38/ff1 to d8/d10/d3e/f113 0 2024-04-20T21:32:14.031 INFO:tasks.workunit.client.0.smithi062.stdout:6/680: sync 2024-04-20T21:32:14.031 INFO:tasks.workunit.client.0.smithi062.stdout:4/888: sync 2024-04-20T21:32:14.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:13 smithi062 ceph-mon[16221]: from='client.14712 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:32:14.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:13 smithi062 ceph-mon[16221]: pgmap v21: 65 pgs: 65 active+clean; 2.3 GiB data, 6.7 GiB used, 530 GiB / 536 GiB avail; 62 MiB/s rd, 213 MiB/s wr, 386 op/s 2024-04-20T21:32:14.035 INFO:tasks.workunit.client.0.smithi062.stdout:4/889: dread - da/dd/d15/d50/feb zero size 2024-04-20T21:32:14.037 INFO:tasks.workunit.client.0.smithi062.stdout:4/890: readlink da/dd/df/d27/d32/l11f 0 2024-04-20T21:32:14.038 INFO:tasks.workunit.client.1.smithi189.stdout:7/583: chown d5/d21/d4a/d52 83 1 2024-04-20T21:32:14.038 INFO:tasks.workunit.client.1.smithi189.stdout:1/513: dwrite d8/f1a [0,4194304] 0 2024-04-20T21:32:14.038 INFO:tasks.workunit.client.1.smithi189.stdout:6/581: dread d8/da/dd/f28 [0,4194304] 0 2024-04-20T21:32:14.045 INFO:tasks.workunit.client.1.smithi189.stdout:5/511: getdents d7/d14/d18 0 2024-04-20T21:32:14.049 INFO:tasks.workunit.client.1.smithi189.stdout:8/618: dread da/df/d20/f7d [0,4194304] 0 2024-04-20T21:32:14.049 INFO:tasks.workunit.client.0.smithi062.stdout:6/681: dwrite d0/d52/fb0 [0,4194304] 0 2024-04-20T21:32:14.054 INFO:tasks.workunit.client.1.smithi189.stdout:0/634: dwrite d3/df/d5d/d64/fac [0,4194304] 0 2024-04-20T21:32:14.055 INFO:tasks.workunit.client.1.smithi189.stdout:1/514: rename d8/d1e/d28/d2b/f76 to d8/d1e/d54/fae 0 2024-04-20T21:32:14.056 INFO:tasks.workunit.client.0.smithi062.stdout:6/682: unlink d0/d1a/d44/f4b 0 2024-04-20T21:32:14.061 INFO:tasks.workunit.client.1.smithi189.stdout:4/511: dread d2/d20/f4f [0,4194304] 0 2024-04-20T21:32:14.061 INFO:tasks.workunit.client.1.smithi189.stdout:5/512: fdatasync d7/d9/d31/f57 0 2024-04-20T21:32:14.061 INFO:tasks.workunit.client.1.smithi189.stdout:8/619: creat da/df/d3a/fde x:0 0 0 2024-04-20T21:32:14.066 INFO:tasks.workunit.client.0.smithi062.stdout:6/683: dread d0/d9/d11/fda [0,4194304] 0 2024-04-20T21:32:14.067 INFO:tasks.workunit.client.0.smithi062.stdout:4/891: dwrite da/dd/df/d27/d7a/d91/fb6 [0,4194304] 0 2024-04-20T21:32:14.068 INFO:tasks.workunit.client.1.smithi189.stdout:5/513: rmdir d7/d9/d30/d55/d61 39 2024-04-20T21:32:14.069 INFO:tasks.workunit.client.1.smithi189.stdout:8/620: dread - da/df/d20/d2c/d9a/fcc zero size 2024-04-20T21:32:14.073 INFO:tasks.workunit.client.1.smithi189.stdout:6/582: dwrite d8/da/f32 [4194304,4194304] 0 2024-04-20T21:32:14.073 INFO:tasks.workunit.client.0.smithi062.stdout:6/684: write d0/d9/d11/d22/f68 [2843399,3088] 0 2024-04-20T21:32:14.075 INFO:tasks.workunit.client.1.smithi189.stdout:1/515: symlink d8/laf 0 2024-04-20T21:32:14.079 INFO:tasks.workunit.client.0.smithi062.stdout:4/892: creat da/dd/df/d27/d7a/d91/de4/f138 x:0 0 0 2024-04-20T21:32:14.080 INFO:tasks.workunit.client.1.smithi189.stdout:8/621: creat da/d1f/d2a/fdf x:0 0 0 2024-04-20T21:32:14.084 INFO:tasks.workunit.client.1.smithi189.stdout:8/622: fdatasync da/d1f/fd8 0 2024-04-20T21:32:14.085 INFO:tasks.workunit.client.0.smithi062.stdout:6/685: rename d0/d9/d11/d22/dd5/d8c to d0/dc7/de4 0 2024-04-20T21:32:14.093 INFO:tasks.workunit.client.1.smithi189.stdout:1/516: rename d8/d21/d2c/f9f to d8/d1e/d54/fb0 0 2024-04-20T21:32:14.094 INFO:tasks.workunit.client.1.smithi189.stdout:2/493: write d1/d17/f7f [2807620,79000] 0 2024-04-20T21:32:14.098 INFO:tasks.workunit.client.1.smithi189.stdout:1/517: write d8/d21/d2c/f62 [3929395,79404] 0 2024-04-20T21:32:14.098 INFO:tasks.workunit.client.1.smithi189.stdout:6/583: rename d8/d12/f95 to d8/d12/d30/d47/da0/fc2 0 2024-04-20T21:32:14.099 INFO:tasks.workunit.client.0.smithi062.stdout:3/751: sync 2024-04-20T21:32:14.099 INFO:tasks.workunit.client.0.smithi062.stdout:0/785: sync 2024-04-20T21:32:14.099 INFO:tasks.workunit.client.0.smithi062.stdout:2/852: sync 2024-04-20T21:32:14.099 INFO:tasks.workunit.client.0.smithi062.stdout:7/793: sync 2024-04-20T21:32:14.099 INFO:tasks.workunit.client.0.smithi062.stdout:9/932: sync 2024-04-20T21:32:14.099 INFO:tasks.workunit.client.0.smithi062.stdout:8/833: sync 2024-04-20T21:32:14.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:14.099+0000 7f8fd3000700 1 -- 172.21.15.62:0/2500135782 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f8fcc02cc30 con 0x7f8fcc100050 2024-04-20T21:32:14.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:14.100+0000 7f8fca000700 1 -- 172.21.15.62:0/2500135782 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+675 (secure 0 0 0) 0x7f8fc005b320 con 0x7f8fcc100050 2024-04-20T21:32:14.101 INFO:tasks.workunit.client.0.smithi062.stdout:1/888: truncate dc/d15/d20/f45 524505 0 2024-04-20T21:32:14.104 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:32:14.104 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T21:32:14.104 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 2 2024-04-20T21:32:14.104 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:32:14.104 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T21:32:14.104 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:32:14.105 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:32:14.105 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T21:32:14.105 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 6 2024-04-20T21:32:14.105 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:32:14.105 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T21:32:14.105 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 4 2024-04-20T21:32:14.105 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:32:14.105 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T21:32:14.105 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 12, 2024-04-20T21:32:14.105 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:32:14.105 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T21:32:14.105 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:32:14.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:14.103+0000 7f8fd3000700 1 -- 172.21.15.62:0/2500135782 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f8fb406ea70 msgr2=0x7f8fb4070f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:32:14.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:14.104+0000 7f8fd3000700 1 --2- 172.21.15.62:0/2500135782 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f8fb406ea70 0x7f8fb4070f20 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f8fb8006010 tx=0x7f8fb800b540 comp rx=0 tx=0).stop 2024-04-20T21:32:14.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:14.104+0000 7f8fd3000700 1 -- 172.21.15.62:0/2500135782 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8fcc100050 msgr2=0x7f8fcc197b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:32:14.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:14.104+0000 7f8fd3000700 1 --2- 172.21.15.62:0/2500135782 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8fcc100050 0x7f8fcc197b10 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f8fc000b480 tx=0x7f8fc000b840 comp rx=0 tx=0).stop 2024-04-20T21:32:14.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:14.104+0000 7f8fd3000700 1 -- 172.21.15.62:0/2500135782 shutdown_connections 2024-04-20T21:32:14.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:14.104+0000 7f8fd3000700 1 --2- 172.21.15.62:0/2500135782 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f8fb406ea70 0x7f8fb4070f20 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:32:14.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:14.104+0000 7f8fd3000700 1 --2- 172.21.15.62:0/2500135782 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8fcc100050 0x7f8fcc197b10 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:32:14.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:14.104+0000 7f8fd3000700 1 --2- 172.21.15.62:0/2500135782 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8fcc101250 0x7f8fcc198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:32:14.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:14.104+0000 7f8fd3000700 1 -- 172.21.15.62:0/2500135782 >> 172.21.15.62:0/2500135782 conn(0x7f8fcc0fb5e0 msgr2=0x7f8fcc102470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:32:14.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:14.104+0000 7f8fd3000700 1 -- 172.21.15.62:0/2500135782 shutdown_connections 2024-04-20T21:32:14.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:14.104+0000 7f8fd3000700 1 -- 172.21.15.62:0/2500135782 wait complete. 2024-04-20T21:32:14.107 INFO:tasks.workunit.client.1.smithi189.stdout:3/531: truncate d2/d3/d2c/d30/d3f/fa5 1393766 0 2024-04-20T21:32:14.107 INFO:tasks.workunit.client.1.smithi189.stdout:8/623: dread da/df/d3c/d50/d6c/dc7/fcd [0,4194304] 0 2024-04-20T21:32:14.108 INFO:tasks.workunit.client.0.smithi062.stdout:3/752: read - d0/d1/f8b zero size 2024-04-20T21:32:14.109 INFO:tasks.workunit.client.0.smithi062.stdout:4/893: dread da/dd/df/d27/d7a/d91/f9f [0,4194304] 0 2024-04-20T21:32:14.110 INFO:tasks.workunit.client.0.smithi062.stdout:5/829: write d8/d38/d7a/d4c/f55 [1988197,63508] 0 2024-04-20T21:32:14.111 INFO:tasks.workunit.client.0.smithi062.stdout:9/933: mknod d16/d3e/c131 0 2024-04-20T21:32:14.112 INFO:tasks.workunit.client.0.smithi062.stdout:3/753: readlink d0/d5c/l9a 0 2024-04-20T21:32:14.116 INFO:tasks.workunit.client.0.smithi062.stdout:6/686: dwrite d0/dc7/fcd [0,4194304] 0 2024-04-20T21:32:14.116 INFO:tasks.workunit.client.0.smithi062.stdout:0/786: truncate d7/d1e/f45 457600 0 2024-04-20T21:32:14.116 INFO:tasks.workunit.client.1.smithi189.stdout:5/514: dwrite d7/d9/d30/d5a/f73 [0,4194304] 0 2024-04-20T21:32:14.118 INFO:tasks.workunit.client.1.smithi189.stdout:6/584: read d8/d2b/f3a [911021,51499] 0 2024-04-20T21:32:14.122 INFO:tasks.workunit.client.0.smithi062.stdout:6/687: truncate d0/d9/f9c 666877 0 2024-04-20T21:32:14.123 INFO:tasks.workunit.client.0.smithi062.stdout:2/853: symlink d7/da/d15/d1b/d3d/l11a 0 2024-04-20T21:32:14.127 INFO:tasks.workunit.client.1.smithi189.stdout:9/522: dwrite d9/d12/d50/f4f [4194304,4194304] 0 2024-04-20T21:32:14.128 INFO:tasks.workunit.client.1.smithi189.stdout:7/584: dwrite d5/f20 [0,4194304] 0 2024-04-20T21:32:14.128 INFO:tasks.workunit.client.1.smithi189.stdout:1/518: unlink d8/lc 0 2024-04-20T21:32:14.129 INFO:tasks.workunit.client.1.smithi189.stdout:6/585: dread - d8/da/dd/d2a/f85 zero size 2024-04-20T21:32:14.132 INFO:tasks.workunit.client.0.smithi062.stdout:4/894: mkdir da/d4f/d139 0 2024-04-20T21:32:14.134 INFO:tasks.workunit.client.1.smithi189.stdout:3/532: symlink d2/d3/d2c/d31/lbc 0 2024-04-20T21:32:14.134 INFO:tasks.workunit.client.1.smithi189.stdout:5/515: creat d7/d9/d28/fa0 x:0 0 0 2024-04-20T21:32:14.135 INFO:tasks.workunit.client.0.smithi062.stdout:7/794: dread - d7/d3e/d33/d3d/fb1 zero size 2024-04-20T21:32:14.135 INFO:tasks.workunit.client.0.smithi062.stdout:8/834: write d7/d25/d4f/fbc [376697,19135] 0 2024-04-20T21:32:14.137 INFO:tasks.workunit.client.1.smithi189.stdout:2/494: write d1/d11/d2d/f41 [2402814,5337] 0 2024-04-20T21:32:14.138 INFO:tasks.workunit.client.1.smithi189.stdout:6/586: stat d8/d12/d30/d47/f8a 0 2024-04-20T21:32:14.139 INFO:tasks.workunit.client.1.smithi189.stdout:1/519: mknod d8/d21/d2c/d36/d8a/cb1 0 2024-04-20T21:32:14.140 INFO:tasks.workunit.client.1.smithi189.stdout:4/512: dwrite d2/d3a/f6f [0,4194304] 0 2024-04-20T21:32:14.140 INFO:tasks.workunit.client.0.smithi062.stdout:1/889: mknod dc/d15/d75/d96/dd4/d107/c146 0 2024-04-20T21:32:14.140 INFO:tasks.workunit.client.0.smithi062.stdout:0/787: creat d7/d1c/d15/f100 x:0 0 0 2024-04-20T21:32:14.144 INFO:tasks.workunit.client.1.smithi189.stdout:5/516: chown d7/d14/d4e/d5b 196 1 2024-04-20T21:32:14.146 INFO:tasks.workunit.client.0.smithi062.stdout:9/934: link d16/d2c/d42/d10f/f123 d16/d2f/d5b/da4/f132 0 2024-04-20T21:32:14.148 INFO:tasks.workunit.client.1.smithi189.stdout:6/587: fsync d8/d12/d99/fb2 0 2024-04-20T21:32:14.151 INFO:tasks.workunit.client.1.smithi189.stdout:0/635: dwrite d3/df/d30/d37/f65 [0,4194304] 0 2024-04-20T21:32:14.160 INFO:tasks.workunit.client.1.smithi189.stdout:0/636: symlink d3/da8/ld6 0 2024-04-20T21:32:14.163 INFO:tasks.workunit.client.1.smithi189.stdout:6/588: mknod d8/d12/cc3 0 2024-04-20T21:32:14.163 INFO:tasks.workunit.client.0.smithi062.stdout:2/854: dwrite d7/da/d15/d2f/d97/f9e [0,4194304] 0 2024-04-20T21:32:14.164 INFO:tasks.workunit.client.1.smithi189.stdout:0/637: fdatasync d3/df/d5d/f7b 0 2024-04-20T21:32:14.169 INFO:tasks.workunit.client.1.smithi189.stdout:3/533: dread d2/d3/d2c/d30/d3f/d9a/f9b [0,4194304] 0 2024-04-20T21:32:14.173 INFO:tasks.workunit.client.1.smithi189.stdout:0/638: read - d3/d8c/fab zero size 2024-04-20T21:32:14.174 INFO:tasks.workunit.client.0.smithi062.stdout:3/754: rename d0/d1c/d63/lf7 to d0/d1/d3c/lf8 0 2024-04-20T21:32:14.178 INFO:tasks.workunit.client.0.smithi062.stdout:0/788: mknod d7/d20/d62/de3/c101 0 2024-04-20T21:32:14.178 INFO:tasks.workunit.client.1.smithi189.stdout:6/589: getdents d8/d2b/d49/d54 0 2024-04-20T21:32:14.179 INFO:tasks.workunit.client.0.smithi062.stdout:1/890: truncate dc/d15/d75/d94/d6f/dd2/fdf 600782 0 2024-04-20T21:32:14.181 INFO:tasks.workunit.client.1.smithi189.stdout:5/517: dwrite d7/d9/d28/fa0 [0,4194304] 0 2024-04-20T21:32:14.183 INFO:tasks.workunit.client.0.smithi062.stdout:1/891: write dc/d15/d20/fe6 [413113,130687] 0 2024-04-20T21:32:14.185 INFO:tasks.workunit.client.0.smithi062.stdout:1/892: stat dc/d15/d20/d28/d83/dc0/lda 0 2024-04-20T21:32:14.189 INFO:tasks.workunit.client.0.smithi062.stdout:1/893: chown dc/d15/d75/d96/dd4/d3a/da2/lb5 20 1 2024-04-20T21:32:14.190 INFO:tasks.workunit.client.0.smithi062.stdout:0/789: write d7/d1c/f18 [4947262,123356] 0 2024-04-20T21:32:14.190 INFO:tasks.workunit.client.0.smithi062.stdout:3/755: creat d0/d1c/d63/d42/dc8/dc9/ff9 x:0 0 0 2024-04-20T21:32:14.193 INFO:tasks.workunit.client.1.smithi189.stdout:5/518: read d7/d9/d28/f7b [178847,113855] 0 2024-04-20T21:32:14.194 INFO:tasks.workunit.client.1.smithi189.stdout:3/534: rename d2/d1c/f7d to d2/d1c/d5c/fbd 0 2024-04-20T21:32:14.194 INFO:tasks.workunit.client.0.smithi062.stdout:4/895: getdents da/dd 0 2024-04-20T21:32:14.195 INFO:tasks.workunit.client.0.smithi062.stdout:0/790: write d7/d1c/d15/d17/d37/f84 [2285293,59929] 0 2024-04-20T21:32:14.199 INFO:tasks.workunit.client.1.smithi189.stdout:6/590: mkdir d8/d12/d4e/dc4 0 2024-04-20T21:32:14.200 INFO:tasks.workunit.client.0.smithi062.stdout:5/830: rename d8/d38/d7a/d4c/d7d/d9d/dac/c6e to d8/d38/d7a/d4c/d7d/d9d/dac/c114 0 2024-04-20T21:32:14.200 INFO:tasks.workunit.client.0.smithi062.stdout:4/896: readlink da/dd/df/d27/l29 0 2024-04-20T21:32:14.202 INFO:tasks.workunit.client.0.smithi062.stdout:7/795: link d7/d3e/d33/d3d/d4d/d54/dea/c88 d7/d26/dac/d10a/c113 0 2024-04-20T21:32:14.205 INFO:tasks.workunit.client.1.smithi189.stdout:3/535: chown d2/d3/d2c/c43 5025 1 2024-04-20T21:32:14.206 INFO:tasks.workunit.client.0.smithi062.stdout:3/756: fdatasync d0/d1/d10/f79 0 2024-04-20T21:32:14.207 INFO:tasks.workunit.client.1.smithi189.stdout:2/495: dread d1/d17/f7f [0,4194304] 0 2024-04-20T21:32:14.207 INFO:tasks.workunit.client.0.smithi062.stdout:9/935: link d16/d3d/d55/d6e/d7d/lba d16/d2c/d5e/d75/l133 0 2024-04-20T21:32:14.207 INFO:tasks.workunit.client.1.smithi189.stdout:3/536: stat d2/d1c 0 2024-04-20T21:32:14.209 INFO:tasks.workunit.client.1.smithi189.stdout:2/496: fsync d1/d11/f2e 0 2024-04-20T21:32:14.213 INFO:tasks.workunit.client.0.smithi062.stdout:4/897: creat da/dd/d15/d6d/df5/f13a x:0 0 0 2024-04-20T21:32:14.214 INFO:tasks.workunit.client.1.smithi189.stdout:8/624: write da/de/d1e/d69/f95 [1667026,81294] 0 2024-04-20T21:32:14.214 INFO:tasks.workunit.client.1.smithi189.stdout:2/497: stat d1/d17/d37/f44 0 2024-04-20T21:32:14.218 INFO:tasks.workunit.client.0.smithi062.stdout:7/796: creat d7/d3e/d33/d3d/d4d/d54/d4f/d9d/f114 x:0 0 0 2024-04-20T21:32:14.221 INFO:tasks.workunit.client.1.smithi189.stdout:5/519: dwrite d7/d35/f74 [0,4194304] 0 2024-04-20T21:32:14.221 INFO:tasks.workunit.client.1.smithi189.stdout:3/537: symlink d2/d3/d2c/d31/d90/lbe 0 2024-04-20T21:32:14.223 INFO:tasks.workunit.client.1.smithi189.stdout:2/498: chown d1/d17/d2f/f42 9 1 2024-04-20T21:32:14.230 INFO:tasks.workunit.client.0.smithi062.stdout:9/936: dwrite d16/d2c/d5e/d75/fdd [0,4194304] 0 2024-04-20T21:32:14.232 INFO:tasks.workunit.client.1.smithi189.stdout:0/639: dread d3/df/d30/d45/d60/f8b [0,4194304] 0 2024-04-20T21:32:14.232 INFO:tasks.workunit.client.1.smithi189.stdout:3/538: creat d2/d3/d2c/d30/d3f/fbf x:0 0 0 2024-04-20T21:32:14.233 INFO:tasks.workunit.client.1.smithi189.stdout:8/625: mknod da/df/d3c/d9d/ce0 0 2024-04-20T21:32:14.233 INFO:tasks.workunit.client.0.smithi062.stdout:7/797: creat d7/d26/dac/ddb/de8/f115 x:0 0 0 2024-04-20T21:32:14.235 INFO:tasks.workunit.client.1.smithi189.stdout:5/520: creat d7/d9/d51/d70/d95/fa1 x:0 0 0 2024-04-20T21:32:14.237 INFO:tasks.workunit.client.1.smithi189.stdout:1/520: write d8/df/f15 [4556011,119069] 0 2024-04-20T21:32:14.240 INFO:tasks.workunit.client.0.smithi062.stdout:5/831: link d8/d38/d7a/d4c/d7d/d9d/dac/d66/cf5 d8/d1e/d2c/dd1/ded/c115 0 2024-04-20T21:32:14.243 INFO:tasks.workunit.client.1.smithi189.stdout:7/585: write d5/d21/f32 [1062969,51905] 0 2024-04-20T21:32:14.246 INFO:tasks.workunit.client.1.smithi189.stdout:6/591: truncate d8/da/dd/f88 63602 0 2024-04-20T21:32:14.247 INFO:tasks.workunit.client.0.smithi062.stdout:8/835: write d7/d25/d5c/f98 [1521748,87058] 0 2024-04-20T21:32:14.247 INFO:tasks.workunit.client.0.smithi062.stdout:7/798: mkdir d7/d3e/d33/d3d/d7f/d116 0 2024-04-20T21:32:14.249 INFO:tasks.workunit.client.0.smithi062.stdout:2/855: dread d7/da/d15/d2f/d38/d10f/d4b/fbe [0,4194304] 0 2024-04-20T21:32:14.250 INFO:tasks.workunit.client.1.smithi189.stdout:3/539: getdents d2 0 2024-04-20T21:32:14.254 INFO:tasks.workunit.client.0.smithi062.stdout:8/836: read d7/d25/d3b/d58/d8b/d24/dbd/f109 [13474,76660] 0 2024-04-20T21:32:14.255 INFO:tasks.workunit.client.0.smithi062.stdout:3/757: link d0/d1/d3c/d43/d72/dd3/l61 d0/d1/d3c/d43/d72/da9/lfa 0 2024-04-20T21:32:14.255 INFO:tasks.workunit.client.1.smithi189.stdout:8/626: rename da/d30 to da/df/d3c/d68/d8b/dac/de1 0 2024-04-20T21:32:14.256 INFO:tasks.workunit.client.1.smithi189.stdout:3/540: dread - d2/d3/d65/fbb zero size 2024-04-20T21:32:14.258 INFO:tasks.workunit.client.0.smithi062.stdout:5/832: mkdir d8/d38/d7a/d116 0 2024-04-20T21:32:14.263 INFO:tasks.workunit.client.1.smithi189.stdout:7/586: symlink d5/db4/d71/lc5 0 2024-04-20T21:32:14.264 INFO:tasks.workunit.client.1.smithi189.stdout:4/513: dwrite d2/d3a/f63 [0,4194304] 0 2024-04-20T21:32:14.265 INFO:tasks.workunit.client.0.smithi062.stdout:4/898: dread da/dd/df/f28 [0,4194304] 0 2024-04-20T21:32:14.265 INFO:tasks.workunit.client.0.smithi062.stdout:3/758: symlink d0/d1b/d74/lfb 0 2024-04-20T21:32:14.266 INFO:tasks.workunit.client.1.smithi189.stdout:3/541: mknod d2/d8/cc0 0 2024-04-20T21:32:14.267 INFO:tasks.workunit.client.1.smithi189.stdout:5/521: link d7/ca d7/d9/d31/d36/d5f/ca2 0 2024-04-20T21:32:14.270 INFO:tasks.workunit.client.0.smithi062.stdout:2/856: rename d7/da/d15/d1b/l5e to d7/da/d15/d1b/df1/d10e/l11b 0 2024-04-20T21:32:14.272 INFO:tasks.workunit.client.1.smithi189.stdout:2/499: dwrite d1/d11/d58/d6b/f77 [0,4194304] 0 2024-04-20T21:32:14.272 INFO:tasks.workunit.client.0.smithi062.stdout:9/937: dread d16/d2c/d42/fe2 [0,4194304] 0 2024-04-20T21:32:14.276 INFO:tasks.workunit.client.1.smithi189.stdout:9/523: dread d9/d12/d3b/f4c [0,4194304] 0 2024-04-20T21:32:14.277 INFO:tasks.workunit.client.1.smithi189.stdout:7/587: dread - d5/d21/d4a/d52/d75/fb2 zero size 2024-04-20T21:32:14.278 INFO:tasks.workunit.client.0.smithi062.stdout:0/791: write d7/d20/d62/d6f/f73 [2556125,48895] 0 2024-04-20T21:32:14.279 INFO:tasks.workunit.client.1.smithi189.stdout:5/522: mkdir d7/d9/d28/da3 0 2024-04-20T21:32:14.279 INFO:tasks.workunit.client.1.smithi189.stdout:3/542: chown d2/d3/d2c/c43 3923545 1 2024-04-20T21:32:14.280 INFO:tasks.workunit.client.0.smithi062.stdout:9/938: rmdir d16/d2c/d62/dd1/d105 39 2024-04-20T21:32:14.283 INFO:tasks.workunit.client.1.smithi189.stdout:1/521: truncate d8/d1e/d28/fa8 1606332 0 2024-04-20T21:32:14.287 INFO:tasks.workunit.client.0.smithi062.stdout:7/799: creat d7/d3e/d33/d3d/d4d/d54/d4f/f117 x:0 0 0 2024-04-20T21:32:14.287 INFO:tasks.workunit.client.0.smithi062.stdout:8/837: creat d7/d25/d3b/d58/f110 x:0 0 0 2024-04-20T21:32:14.287 INFO:tasks.workunit.client.0.smithi062.stdout:1/894: dwrite dc/d15/d75/d96/dd4/f54 [4194304,4194304] 0 2024-04-20T21:32:14.288 INFO:tasks.workunit.client.0.smithi062.stdout:0/792: mkdir d7/d1e/df9/d102 0 2024-04-20T21:32:14.289 INFO:tasks.workunit.client.1.smithi189.stdout:2/500: write d1/d11/d2d/f50 [2670837,44019] 0 2024-04-20T21:32:14.290 INFO:tasks.workunit.client.0.smithi062.stdout:7/800: write d7/d26/f9a [3982173,99454] 0 2024-04-20T21:32:14.290 INFO:tasks.workunit.client.0.smithi062.stdout:8/838: write d7/d9/f12 [3115974,36123] 0 2024-04-20T21:32:14.292 INFO:tasks.workunit.client.1.smithi189.stdout:4/514: link d2/dd/d37/f59 d2/d7f/fc1 0 2024-04-20T21:32:14.294 INFO:tasks.workunit.client.0.smithi062.stdout:9/939: truncate d16/d2f/f8f 343571 0 2024-04-20T21:32:14.300 INFO:tasks.workunit.client.1.smithi189.stdout:0/640: dwrite d3/df/d30/d45/f6c [0,4194304] 0 2024-04-20T21:32:14.300 INFO:tasks.workunit.client.1.smithi189.stdout:9/524: unlink d9/dc/d17/d95/da0/d52/f75 0 2024-04-20T21:32:14.303 INFO:tasks.workunit.client.0.smithi062.stdout:9/940: creat d16/dbe/f134 x:0 0 0 2024-04-20T21:32:14.306 INFO:tasks.workunit.client.1.smithi189.stdout:7/588: fsync d5/d21/d25/d34/f6f 0 2024-04-20T21:32:14.307 INFO:tasks.workunit.client.0.smithi062.stdout:9/941: symlink d16/d3e/d77/l135 0 2024-04-20T21:32:14.308 INFO:tasks.workunit.client.1.smithi189.stdout:5/523: rename d7/d9/f42 to d7/d9/d30/d55/d61/fa4 0 2024-04-20T21:32:14.310 INFO:tasks.workunit.client.1.smithi189.stdout:2/501: symlink d1/d17/d2f/d4a/d66/l96 0 2024-04-20T21:32:14.311 INFO:tasks.workunit.client.1.smithi189.stdout:9/525: creat d9/d12/fa8 x:0 0 0 2024-04-20T21:32:14.314 INFO:tasks.workunit.client.1.smithi189.stdout:6/592: dwrite d8/d2b/f75 [0,4194304] 0 2024-04-20T21:32:14.316 INFO:tasks.workunit.client.1.smithi189.stdout:7/589: mkdir d5/d21/d97/da3/dc6 0 2024-04-20T21:32:14.317 INFO:tasks.workunit.client.1.smithi189.stdout:0/641: mkdir d3/db9/dbe/dd7 0 2024-04-20T21:32:14.319 INFO:tasks.workunit.client.1.smithi189.stdout:2/502: fsync d1/f12 0 2024-04-20T21:32:14.320 INFO:tasks.workunit.client.0.smithi062.stdout:4/899: write da/dd/d15/d6d/df5/ff8 [592852,35122] 0 2024-04-20T21:32:14.321 INFO:tasks.workunit.client.0.smithi062.stdout:2/857: write d7/da/d15/d1b/f34 [2932897,77289] 0 2024-04-20T21:32:14.322 INFO:tasks.workunit.client.1.smithi189.stdout:8/627: dread da/d1f/d2a/f58 [0,4194304] 0 2024-04-20T21:32:14.324 INFO:tasks.workunit.client.1.smithi189.stdout:3/543: dread d2/d3/d2c/d30/f68 [0,4194304] 0 2024-04-20T21:32:14.325 INFO:tasks.workunit.client.1.smithi189.stdout:9/526: rmdir d9/d12/d15/d1a 39 2024-04-20T21:32:14.326 INFO:tasks.workunit.client.0.smithi062.stdout:5/833: write d8/d38/d7a/d4c/d7d/d9d/fcc [355131,33232] 0 2024-04-20T21:32:14.327 INFO:tasks.workunit.client.1.smithi189.stdout:5/524: symlink d7/d9/d31/la5 0 2024-04-20T21:32:14.329 INFO:tasks.workunit.client.1.smithi189.stdout:3/544: dread - d2/d3/d36/da1/fb2 zero size 2024-04-20T21:32:14.334 INFO:tasks.workunit.client.1.smithi189.stdout:5/525: read d7/f5d [127816,21523] 0 2024-04-20T21:32:14.334 INFO:tasks.workunit.client.1.smithi189.stdout:7/590: creat d5/db4/d71/fc7 x:0 0 0 2024-04-20T21:32:14.334 INFO:tasks.workunit.client.0.smithi062.stdout:1/895: write dc/d15/f6e [453188,116312] 0 2024-04-20T21:32:14.336 INFO:tasks.workunit.client.0.smithi062.stdout:2/858: rmdir d7/da/d15/d2f/d38/d6b 39 2024-04-20T21:32:14.338 INFO:tasks.workunit.client.0.smithi062.stdout:8/839: write d7/d25/d3b/d58/d8b/d24/f48 [6057565,59125] 0 2024-04-20T21:32:14.338 INFO:tasks.workunit.client.0.smithi062.stdout:2/859: fsync d7/da/d15/d1b/f2c 0 2024-04-20T21:32:14.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:14.341+0000 7f3277400700 1 -- 172.21.15.62:0/3489292994 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3270072750 msgr2=0x7f3270107d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:32:14.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:14.341+0000 7f3277400700 1 --2- 172.21.15.62:0/3489292994 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3270072750 0x7f3270107d50 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f325c009910 tx=0x7f325c009c20 comp rx=0 tx=0).stop 2024-04-20T21:32:14.341 INFO:tasks.workunit.client.0.smithi062.stdout:8/840: chown d7/de1/de5 7559740 1 2024-04-20T21:32:14.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:14.341+0000 7f3277400700 1 -- 172.21.15.62:0/3489292994 shutdown_connections 2024-04-20T21:32:14.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:14.341+0000 7f3277400700 1 --2- 172.21.15.62:0/3489292994 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3270072750 0x7f3270107d50 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:32:14.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:14.341+0000 7f3277400700 1 --2- 172.21.15.62:0/3489292994 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3270071e00 0x7f3270072210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:32:14.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:14.341+0000 7f3277400700 1 -- 172.21.15.62:0/3489292994 >> 172.21.15.62:0/3489292994 conn(0x7f327006d110 msgr2=0x7f327006f560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:32:14.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:14.341+0000 7f3277400700 1 -- 172.21.15.62:0/3489292994 shutdown_connections 2024-04-20T21:32:14.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:14.341+0000 7f3277400700 1 -- 172.21.15.62:0/3489292994 wait complete. 2024-04-20T21:32:14.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:14.342+0000 7f3277400700 1 Processor -- start 2024-04-20T21:32:14.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:14.342+0000 7f3277400700 1 -- start start 2024-04-20T21:32:14.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:14.343+0000 7f3277400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3270071e00 0x7f3270117410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:32:14.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:14.343+0000 7f3277400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3270072750 0x7f3270117950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:32:14.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:14.343+0000 7f3277400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3270117f70 con 0x7f3270071e00 2024-04-20T21:32:14.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:14.343+0000 7f3277400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32701180b0 con 0x7f3270072750 2024-04-20T21:32:14.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:14.343+0000 7f3274a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3270071e00 0x7f3270117410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:32:14.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:14.343+0000 7f3274a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3270071e00 0x7f3270117410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:57428/0 (socket says 172.21.15.62:57428) 2024-04-20T21:32:14.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:14.343+0000 7f3274a00700 1 -- 172.21.15.62:0/4170028537 learned_addr learned my addr 172.21.15.62:0/4170028537 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:32:14.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:14.344+0000 7f3274a00700 1 -- 172.21.15.62:0/4170028537 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3270072750 msgr2=0x7f3270117950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:32:14.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:14.344+0000 7f3274a00700 1 --2- 172.21.15.62:0/4170028537 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3270072750 0x7f3270117950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:32:14.346 INFO:tasks.workunit.client.1.smithi189.stdout:2/503: mknod d1/d17/d37/c97 0 2024-04-20T21:32:14.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:14.344+0000 7f3274a00700 1 -- 172.21.15.62:0/4170028537 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f325c0095f0 con 0x7f3270071e00 2024-04-20T21:32:14.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:14.345+0000 7f3274a00700 1 --2- 172.21.15.62:0/4170028537 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3270071e00 0x7f3270117410 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f326400b4b0 tx=0x7f326400b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:32:14.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:14.346+0000 7f326d600700 1 -- 172.21.15.62:0/4170028537 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3264011800 con 0x7f3270071e00 2024-04-20T21:32:14.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:14.347+0000 7f326d600700 1 -- 172.21.15.62:0/4170028537 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f3264011e10 con 0x7f3270071e00 2024-04-20T21:32:14.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:14.347+0000 7f3277400700 1 -- 172.21.15.62:0/4170028537 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f327011ab50 con 0x7f3270071e00 2024-04-20T21:32:14.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:14.347+0000 7f3277400700 1 -- 172.21.15.62:0/4170028537 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f327011b0a0 con 0x7f3270071e00 2024-04-20T21:32:14.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:14.347+0000 7f326d600700 1 -- 172.21.15.62:0/4170028537 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f326400f2e0 con 0x7f3270071e00 2024-04-20T21:32:14.348 INFO:tasks.workunit.client.0.smithi062.stdout:5/834: write d8/d38/d7a/d4c/d7d/fde [634974,96114] 0 2024-04-20T21:32:14.348 INFO:tasks.workunit.client.0.smithi062.stdout:1/896: symlink dc/d15/d20/d28/d76/d137/l147 0 2024-04-20T21:32:14.349 INFO:tasks.workunit.client.0.smithi062.stdout:8/841: unlink d7/d25/d3b/d58/f10a 0 2024-04-20T21:32:14.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:14.349+0000 7f3277400700 1 -- 172.21.15.62:0/4170028537 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3250005320 con 0x7f3270071e00 2024-04-20T21:32:14.351 INFO:tasks.workunit.client.1.smithi189.stdout:3/545: mkdir d2/d1c/dc1 0 2024-04-20T21:32:14.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:14.351+0000 7f326d600700 1 -- 172.21.15.62:0/4170028537 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 29) v1 ==== 92700+0+0 (secure 0 0 0) 0x7f3264011960 con 0x7f3270071e00 2024-04-20T21:32:14.354 INFO:tasks.workunit.client.0.smithi062.stdout:5/835: chown d8/d38/fbe 15912 1 2024-04-20T21:32:14.354 INFO:tasks.workunit.client.1.smithi189.stdout:3/546: dread - d2/d3/d65/fbb zero size 2024-04-20T21:32:14.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:14.355+0000 7f326d600700 1 --2- 172.21.15.62:0/4170028537 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f325806eac0 0x7f3258070f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:32:14.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:14.356+0000 7f326d600700 1 -- 172.21.15.62:0/4170028537 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f326408e5b0 con 0x7f3270071e00 2024-04-20T21:32:14.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:14.356+0000 7f326fe00700 1 --2- 172.21.15.62:0/4170028537 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f325806eac0 0x7f3258070f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:32:14.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:14.357+0000 7f326fe00700 1 --2- 172.21.15.62:0/4170028537 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f325806eac0 0x7f3258070f70 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f325c005090 tx=0x7f325c005fb0 comp rx=0 tx=0).ready entity=mgr.14658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:32:14.357 INFO:tasks.workunit.client.0.smithi062.stdout:2/860: rename d7/da/d15/d2f/d97/lae to d7/da/d15/d1b/d3d/l11c 0 2024-04-20T21:32:14.358 INFO:tasks.workunit.client.1.smithi189.stdout:8/628: truncate da/df/d20/f79 304568 0 2024-04-20T21:32:14.361 INFO:tasks.workunit.client.1.smithi189.stdout:4/515: dread d2/f10 [0,4194304] 0 2024-04-20T21:32:14.363 INFO:tasks.workunit.client.1.smithi189.stdout:2/504: rename d1/l3e to d1/d11/d52/d8b/l98 0 2024-04-20T21:32:14.364 INFO:tasks.workunit.client.0.smithi062.stdout:1/897: mkdir dc/d15/d75/d96/dd4/d3a/da2/de4/d84/dbe/d10d/d148 0 2024-04-20T21:32:14.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:14.365+0000 7f326d600700 1 -- 172.21.15.62:0/4170028537 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187527 (secure 0 0 0) 0x7f326405b2a0 con 0x7f3270071e00 2024-04-20T21:32:14.365 INFO:tasks.workunit.client.0.smithi062.stdout:5/836: stat d8/d38/d7a/d4c/d7d/d9d/dac/d2f/d8e/l99 0 2024-04-20T21:32:14.366 INFO:tasks.workunit.client.0.smithi062.stdout:8/842: read d7/d25/d3b/d58/d8b/d24/f75 [88032,56155] 0 2024-04-20T21:32:14.370 INFO:tasks.workunit.client.1.smithi189.stdout:3/547: symlink d2/d3/d2c/d31/d4c/lc2 0 2024-04-20T21:32:14.372 INFO:tasks.workunit.client.1.smithi189.stdout:6/593: dwrite d8/d2b/f87 [0,4194304] 0 2024-04-20T21:32:14.375 INFO:tasks.workunit.client.1.smithi189.stdout:1/522: dwrite d8/f24 [0,4194304] 0 2024-04-20T21:32:14.375 INFO:tasks.workunit.client.1.smithi189.stdout:8/629: mknod da/ce2 0 2024-04-20T21:32:14.376 INFO:tasks.workunit.client.0.smithi062.stdout:8/843: creat d7/de1/f111 x:0 0 0 2024-04-20T21:32:14.379 INFO:tasks.workunit.client.1.smithi189.stdout:2/505: mkdir d1/d11/d99 0 2024-04-20T21:32:14.380 INFO:tasks.workunit.client.0.smithi062.stdout:4/900: dwrite da/dd/df/d27/d7a/fbb [0,4194304] 0 2024-04-20T21:32:14.381 INFO:tasks.workunit.client.0.smithi062.stdout:7/801: dwrite d7/d3e/d33/d3d/d5e/d72/fb2 [0,4194304] 0 2024-04-20T21:32:14.381 INFO:tasks.workunit.client.0.smithi062.stdout:5/837: creat d8/d38/d7a/f117 x:0 0 0 2024-04-20T21:32:14.381 INFO:tasks.workunit.client.1.smithi189.stdout:1/523: stat d8/d1e/l81 0 2024-04-20T21:32:14.386 INFO:tasks.workunit.client.0.smithi062.stdout:1/898: rename dc/d15/df6/d142 to dc/d15/d20/d28/d83/dc0/d132/d149 0 2024-04-20T21:32:14.388 INFO:tasks.workunit.client.0.smithi062.stdout:5/838: dread - d8/d38/f110 zero size 2024-04-20T21:32:14.388 INFO:tasks.workunit.client.0.smithi062.stdout:3/759: dread d0/d5c/d76/faa [4194304,4194304] 0 2024-04-20T21:32:14.389 INFO:tasks.workunit.client.0.smithi062.stdout:9/942: dwrite d16/d2f/d69/f7e [0,4194304] 0 2024-04-20T21:32:14.390 INFO:tasks.workunit.client.1.smithi189.stdout:7/591: dwrite d5/d21/d4a/d52/f5c [0,4194304] 0 2024-04-20T21:32:14.390 INFO:tasks.workunit.client.0.smithi062.stdout:8/844: creat d7/d30/f112 x:0 0 0 2024-04-20T21:32:14.392 INFO:tasks.workunit.client.0.smithi062.stdout:2/861: truncate d7/da/d15/d1b/d24/dee/f108 872506 0 2024-04-20T21:32:14.395 INFO:tasks.workunit.client.1.smithi189.stdout:9/527: dwrite d9/dc/f1c [4194304,4194304] 0 2024-04-20T21:32:14.396 INFO:tasks.workunit.client.0.smithi062.stdout:4/901: chown da/dd/df/d27/l38 846316 1 2024-04-20T21:32:14.397 INFO:tasks.workunit.client.0.smithi062.stdout:1/899: creat dc/d15/d20/d28/d83/dc0/f14a x:0 0 0 2024-04-20T21:32:14.398 INFO:tasks.workunit.client.1.smithi189.stdout:0/642: truncate d3/df/d28/d2b/f67 38559 0 2024-04-20T21:32:14.399 INFO:tasks.workunit.client.1.smithi189.stdout:1/524: mkdir d8/d1e/d43/d44/db2 0 2024-04-20T21:32:14.410 INFO:tasks.workunit.client.1.smithi189.stdout:6/594: dwrite d8/d2b/f90 [0,4194304] 0 2024-04-20T21:32:14.411 INFO:tasks.workunit.client.1.smithi189.stdout:9/528: mknod d9/d12/d62/ca9 0 2024-04-20T21:32:14.411 INFO:tasks.workunit.client.0.smithi062.stdout:8/845: creat d7/d25/d3b/d58/db9/f113 x:0 0 0 2024-04-20T21:32:14.412 INFO:tasks.workunit.client.0.smithi062.stdout:9/943: mknod d16/d2f/d5b/c136 0 2024-04-20T21:32:14.413 INFO:tasks.workunit.client.1.smithi189.stdout:1/525: creat d8/d21/fb3 x:0 0 0 2024-04-20T21:32:14.414 INFO:tasks.workunit.client.0.smithi062.stdout:2/862: write d7/da/d15/d2f/d38/d6a/d7c/fed [187265,80028] 0 2024-04-20T21:32:14.419 INFO:tasks.workunit.client.0.smithi062.stdout:4/902: creat da/dd/f13b x:0 0 0 2024-04-20T21:32:14.420 INFO:tasks.workunit.client.0.smithi062.stdout:7/802: link d7/d3e/d33/d3d/f75 d7/d26/dac/d10a/f118 0 2024-04-20T21:32:14.420 INFO:tasks.workunit.client.0.smithi062.stdout:6/688: sync 2024-04-20T21:32:14.422 INFO:tasks.workunit.client.1.smithi189.stdout:2/506: read d1/d17/d37/f44 [3472248,129990] 0 2024-04-20T21:32:14.424 INFO:tasks.workunit.client.1.smithi189.stdout:6/595: dread - d8/d2b/d49/d54/d6e/fb0 zero size 2024-04-20T21:32:14.424 INFO:tasks.workunit.client.1.smithi189.stdout:9/529: chown d9/dc/d17/d7e/f8d 0 1 2024-04-20T21:32:14.428 INFO:tasks.workunit.client.1.smithi189.stdout:8/630: dwrite da/fb [4194304,4194304] 0 2024-04-20T21:32:14.428 INFO:tasks.workunit.client.1.smithi189.stdout:1/526: fdatasync d8/d1e/d43/d44/f47 0 2024-04-20T21:32:14.430 INFO:tasks.workunit.client.1.smithi189.stdout:6/596: dread - d8/d12/fae zero size 2024-04-20T21:32:14.434 INFO:tasks.workunit.client.0.smithi062.stdout:6/689: mkdir d0/dc7/de5 0 2024-04-20T21:32:14.438 INFO:tasks.workunit.client.1.smithi189.stdout:1/527: truncate d8/d21/fa5 741247 0 2024-04-20T21:32:14.439 INFO:tasks.workunit.client.0.smithi062.stdout:9/944: creat d16/d2c/d5e/d75/d76/dfe/f137 x:0 0 0 2024-04-20T21:32:14.440 INFO:tasks.workunit.client.0.smithi062.stdout:3/760: dread d0/d1/f82 [0,4194304] 0 2024-04-20T21:32:14.441 INFO:tasks.workunit.client.1.smithi189.stdout:8/631: chown da/df/d20/l4d 5326892 1 2024-04-20T21:32:14.442 INFO:tasks.workunit.client.1.smithi189.stdout:8/632: chown l8 582995117 1 2024-04-20T21:32:14.446 INFO:tasks.workunit.client.1.smithi189.stdout:2/507: link d1/d17/d37/f75 d1/d11/d2d/d4f/d8e/f9a 0 2024-04-20T21:32:14.448 INFO:tasks.workunit.client.0.smithi062.stdout:3/761: chown d0/d1c/d63/dda 199592162 1 2024-04-20T21:32:14.449 INFO:tasks.workunit.client.0.smithi062.stdout:6/690: stat d0/d9/d11/d22/c66 0 2024-04-20T21:32:14.449 INFO:tasks.workunit.client.0.smithi062.stdout:9/945: mknod d16/c138 0 2024-04-20T21:32:14.451 INFO:tasks.workunit.client.0.smithi062.stdout:1/900: dwrite dc/d15/f18 [4194304,4194304] 0 2024-04-20T21:32:14.452 INFO:tasks.workunit.client.0.smithi062.stdout:7/803: rename d7/d3e/d33/d3d/d5e/cfc to d7/d3e/d33/d3d/d4d/d54/d4f/de9/df1/c119 0 2024-04-20T21:32:14.454 INFO:tasks.workunit.client.0.smithi062.stdout:9/946: chown d16/d4e/c65 1252933242 1 2024-04-20T21:32:14.457 INFO:tasks.workunit.client.1.smithi189.stdout:8/633: creat da/df/d20/d27/fe3 x:0 0 0 2024-04-20T21:32:14.462 INFO:tasks.workunit.client.1.smithi189.stdout:1/528: dwrite d8/d21/d2c/d36/f52 [0,4194304] 0 2024-04-20T21:32:14.464 INFO:tasks.workunit.client.0.smithi062.stdout:6/691: fdatasync d0/fb5 0 2024-04-20T21:32:14.466 INFO:tasks.workunit.client.1.smithi189.stdout:3/548: dread d2/d3/f1b [0,4194304] 0 2024-04-20T21:32:14.469 INFO:tasks.workunit.client.0.smithi062.stdout:9/947: mknod d16/d2c/d62/c139 0 2024-04-20T21:32:14.470 INFO:tasks.workunit.client.0.smithi062.stdout:3/762: mkdir d0/d1/d3c/d43/d72/dd3/dbf/de8/dfc 0 2024-04-20T21:32:14.473 INFO:tasks.workunit.client.1.smithi189.stdout:3/549: fdatasync d2/f2a 0 2024-04-20T21:32:14.476 INFO:tasks.workunit.client.1.smithi189.stdout:9/530: truncate d9/dc/d17/d36/f57 2816796 0 2024-04-20T21:32:14.478 INFO:tasks.workunit.client.0.smithi062.stdout:3/763: fsync d0/d1/d10/d3a/d5e/d65/d88/feb 0 2024-04-20T21:32:14.479 INFO:tasks.workunit.client.1.smithi189.stdout:7/592: dread d5/d46/d7d/d45/f48 [0,4194304] 0 2024-04-20T21:32:14.480 INFO:tasks.workunit.client.1.smithi189.stdout:8/634: read da/de/d1e/d21/d44/f4a [2764743,10537] 0 2024-04-20T21:32:14.481 INFO:tasks.workunit.client.0.smithi062.stdout:6/692: creat d0/d1a/d7d/db1/de0/fe6 x:0 0 0 2024-04-20T21:32:14.482 INFO:tasks.workunit.client.1.smithi189.stdout:1/529: dread d8/d1e/d28/d4f/f5f [0,4194304] 0 2024-04-20T21:32:14.482 INFO:tasks.workunit.client.1.smithi189.stdout:0/643: dwrite d3/df/d28/d2b/d2c/f62 [0,4194304] 0 2024-04-20T21:32:14.482 INFO:tasks.workunit.client.0.smithi062.stdout:1/901: getdents dc/d27/d9a/da0 0 2024-04-20T21:32:14.485 INFO:tasks.workunit.client.0.smithi062.stdout:9/948: mkdir d16/d3e/d77/d84/d13a 0 2024-04-20T21:32:14.486 INFO:tasks.workunit.client.1.smithi189.stdout:3/550: write d2/d1c/d5c/fbd [596855,87515] 0 2024-04-20T21:32:14.487 INFO:tasks.workunit.client.1.smithi189.stdout:3/551: readlink d2/d3/d2c/l51 0 2024-04-20T21:32:14.487 INFO:tasks.workunit.client.1.smithi189.stdout:7/593: truncate d5/d46/faa 814406 0 2024-04-20T21:32:14.488 INFO:tasks.workunit.client.0.smithi062.stdout:8/846: dwrite d7/d30/d90/f92 [4194304,4194304] 0 2024-04-20T21:32:14.490 INFO:tasks.workunit.client.1.smithi189.stdout:8/635: mkdir da/df/d13/dd6/d59/de4 0 2024-04-20T21:32:14.490 INFO:tasks.workunit.client.1.smithi189.stdout:1/530: symlink d8/d1e/d28/d4f/lb4 0 2024-04-20T21:32:14.491 INFO:tasks.workunit.client.0.smithi062.stdout:2/863: dwrite d7/da/d15/d2f/d38/d10f/f2a [4194304,4194304] 0 2024-04-20T21:32:14.498 INFO:tasks.workunit.client.1.smithi189.stdout:7/594: mkdir d5/d21/d4a/d52/d75/dc8 0 2024-04-20T21:32:14.501 INFO:tasks.workunit.client.0.smithi062.stdout:4/903: truncate da/dd/df/d27/d32/f37 1263412 0 2024-04-20T21:32:14.501 INFO:tasks.workunit.client.1.smithi189.stdout:6/597: truncate d8/d12/d4e/f6f 90279 0 2024-04-20T21:32:14.502 INFO:tasks.workunit.client.1.smithi189.stdout:3/552: dread d2/d8/f34 [0,4194304] 0 2024-04-20T21:32:14.504 INFO:tasks.workunit.client.1.smithi189.stdout:8/636: chown da/d81/d91/da8 4 1 2024-04-20T21:32:14.506 INFO:tasks.workunit.client.1.smithi189.stdout:1/531: rename d8/laf to d8/d21/d2c/d36/d8a/lb5 0 2024-04-20T21:32:14.509 INFO:tasks.workunit.client.0.smithi062.stdout:1/902: truncate dc/d15/d20/d28/d76/f141 780743 0 2024-04-20T21:32:14.509 INFO:tasks.workunit.client.0.smithi062.stdout:2/864: write d7/da/d15/d2f/f7e [3563515,127329] 0 2024-04-20T21:32:14.509 INFO:tasks.workunit.client.0.smithi062.stdout:7/804: getdents d7/d3e/d33/d3d/d5e 0 2024-04-20T21:32:14.511 INFO:tasks.workunit.client.1.smithi189.stdout:7/595: creat d5/d46/d7d/fc9 x:0 0 0 2024-04-20T21:32:14.513 INFO:tasks.workunit.client.1.smithi189.stdout:0/644: creat d3/df/d30/d45/d60/fd8 x:0 0 0 2024-04-20T21:32:14.514 INFO:tasks.workunit.client.1.smithi189.stdout:8/637: dread da/df/d20/d2c/f96 [0,4194304] 0 2024-04-20T21:32:14.516 INFO:tasks.workunit.client.0.smithi062.stdout:7/805: chown d7/d3e/d33/d3d/d4d/d54/dea 5341 1 2024-04-20T21:32:14.516 INFO:tasks.workunit.client.0.smithi062.stdout:9/949: getdents d16/d3e 0 2024-04-20T21:32:14.516 INFO:tasks.workunit.client.0.smithi062.stdout:4/904: getdents da/d4f/d139 0 2024-04-20T21:32:14.517 INFO:tasks.workunit.client.1.smithi189.stdout:8/638: readlink da/de/d1e/d21/lca 0 2024-04-20T21:32:14.517 INFO:tasks.workunit.client.1.smithi189.stdout:0/645: write d3/df/d30/d45/fc4 [746316,49611] 0 2024-04-20T21:32:14.518 INFO:tasks.workunit.client.1.smithi189.stdout:2/508: write d1/f7 [8578356,64790] 0 2024-04-20T21:32:14.520 INFO:tasks.workunit.client.1.smithi189.stdout:2/509: stat d1/d11/d99 0 2024-04-20T21:32:14.522 INFO:tasks.workunit.client.0.smithi062.stdout:9/950: stat d16/d2f/d69/cd8 0 2024-04-20T21:32:14.522 INFO:tasks.workunit.client.1.smithi189.stdout:6/598: rename d8/da/dd/d35/d6d/d7b/c7f to d8/da/dd/d2a/cc5 0 2024-04-20T21:32:14.522 INFO:tasks.workunit.client.1.smithi189.stdout:1/532: mknod d8/d21/cb6 0 2024-04-20T21:32:14.523 INFO:tasks.workunit.client.1.smithi189.stdout:3/553: link d2/d3/f52 d2/d1c/d5c/fc3 0 2024-04-20T21:32:14.525 INFO:tasks.workunit.client.1.smithi189.stdout:2/510: write d1/d11/d52/f6a [2273163,73192] 0 2024-04-20T21:32:14.526 INFO:tasks.workunit.client.0.smithi062.stdout:8/847: truncate d7/d9/f17 86044 0 2024-04-20T21:32:14.526 INFO:tasks.workunit.client.0.smithi062.stdout:0/793: sync 2024-04-20T21:32:14.530 INFO:tasks.workunit.client.1.smithi189.stdout:8/639: chown da/d1f 31345896 1 2024-04-20T21:32:14.530 INFO:tasks.workunit.client.1.smithi189.stdout:1/533: chown d8/df/d50/l6a 162514729 1 2024-04-20T21:32:14.530 INFO:tasks.workunit.client.0.smithi062.stdout:1/903: dread - dc/d15/d20/d28/d83/dc0/fcb zero size 2024-04-20T21:32:14.530 INFO:tasks.workunit.client.0.smithi062.stdout:8/848: dread - d7/d30/f112 zero size 2024-04-20T21:32:14.531 INFO:tasks.workunit.client.1.smithi189.stdout:9/531: dread d9/d12/fa1 [0,4194304] 0 2024-04-20T21:32:14.533 INFO:tasks.workunit.client.1.smithi189.stdout:6/599: rename d8/d2b/f57 to d8/d10/fc6 0 2024-04-20T21:32:14.533 INFO:tasks.workunit.client.1.smithi189.stdout:9/532: chown d9/d12/d3b/c3c 211 1 2024-04-20T21:32:14.534 INFO:tasks.workunit.client.0.smithi062.stdout:9/951: rmdir d16/dbe 39 2024-04-20T21:32:14.535 INFO:tasks.workunit.client.1.smithi189.stdout:0/646: symlink d3/df/d4c/ld9 0 2024-04-20T21:32:14.535 INFO:tasks.workunit.client.1.smithi189.stdout:6/600: dread - d8/d12/daf/fb5 zero size 2024-04-20T21:32:14.537 INFO:tasks.workunit.client.0.smithi062.stdout:7/806: read d7/d26/dac/d10a/f118 [252969,50871] 0 2024-04-20T21:32:14.539 INFO:tasks.workunit.client.1.smithi189.stdout:7/596: truncate d5/fe 1285738 0 2024-04-20T21:32:14.541 INFO:tasks.workunit.client.0.smithi062.stdout:3/764: dwrite d0/d11/fa2 [0,4194304] 0 2024-04-20T21:32:14.541 INFO:tasks.workunit.client.1.smithi189.stdout:3/554: rename d2/d3/d2c/d30/c32 to d2/d3/d2c/d30/cc4 0 2024-04-20T21:32:14.545 INFO:tasks.workunit.client.0.smithi062.stdout:0/794: fdatasync d7/d1c/d15/d17/d40/d79/fb0 0 2024-04-20T21:32:14.545 INFO:tasks.workunit.client.1.smithi189.stdout:8/640: symlink da/le5 0 2024-04-20T21:32:14.546 INFO:tasks.workunit.client.0.smithi062.stdout:5/839: dread d8/d10/f26 [0,4194304] 0 2024-04-20T21:32:14.546 INFO:tasks.workunit.client.0.smithi062.stdout:1/904: write dc/d15/d20/f10f [417252,74884] 0 2024-04-20T21:32:14.548 INFO:tasks.workunit.client.1.smithi189.stdout:9/533: unlink d9/dc/d17/d95/da0/d52/c58 0 2024-04-20T21:32:14.550 INFO:tasks.workunit.client.0.smithi062.stdout:2/865: dread - d7/da/d15/d2f/d38/d51/dbc/fef zero size 2024-04-20T21:32:14.552 INFO:tasks.workunit.client.1.smithi189.stdout:0/647: symlink d3/df/d28/lda 0 2024-04-20T21:32:14.554 INFO:tasks.workunit.client.1.smithi189.stdout:7/597: rmdir d5/d21/d25/d34 39 2024-04-20T21:32:14.556 INFO:tasks.workunit.client.1.smithi189.stdout:1/534: read f6 [3644940,40592] 0 2024-04-20T21:32:14.557 INFO:tasks.workunit.client.0.smithi062.stdout:4/905: chown da/dd/d15/d50/d64/d57/d11e/l134 3 1 2024-04-20T21:32:14.557 INFO:tasks.workunit.client.1.smithi189.stdout:8/641: mkdir da/df/d3c/d68/de6 0 2024-04-20T21:32:14.559 INFO:tasks.workunit.client.0.smithi062.stdout:9/952: mkdir d16/d2f/d13b 0 2024-04-20T21:32:14.562 INFO:tasks.workunit.client.1.smithi189.stdout:1/535: write d8/df/d50/f8d [238658,57253] 0 2024-04-20T21:32:14.562 INFO:tasks.workunit.client.1.smithi189.stdout:8/642: read da/df/d13/d31/f80 [138843,111061] 0 2024-04-20T21:32:14.564 INFO:tasks.workunit.client.0.smithi062.stdout:0/795: dread d7/d1c/d15/d17/d40/d3c/fd8 [0,4194304] 0 2024-04-20T21:32:14.564 INFO:tasks.workunit.client.0.smithi062.stdout:3/765: chown d0/d1/d3c/d43/d72/dd3/d7b/f85 197486713 1 2024-04-20T21:32:14.564 INFO:tasks.workunit.client.1.smithi189.stdout:1/536: readlink d8/df/d3a/d49/l53 0 2024-04-20T21:32:14.566 INFO:tasks.workunit.client.0.smithi062.stdout:6/693: dwrite d0/d9/d11/d22/f68 [0,4194304] 0 2024-04-20T21:32:14.568 INFO:tasks.workunit.client.1.smithi189.stdout:0/648: rmdir d3/d8c 39 2024-04-20T21:32:14.569 INFO:tasks.workunit.client.1.smithi189.stdout:0/649: fdatasync d3/fe 0 2024-04-20T21:32:14.570 INFO:tasks.workunit.client.0.smithi062.stdout:7/807: mknod d7/d3e/d33/d3d/d5e/d72/c11a 0 2024-04-20T21:32:14.577 INFO:tasks.workunit.client.0.smithi062.stdout:9/953: creat d16/d2c/d42/da6/da9/f13c x:0 0 0 2024-04-20T21:32:14.580 INFO:tasks.workunit.client.1.smithi189.stdout:3/555: mkdir d2/d1c/dc1/dc5 0 2024-04-20T21:32:14.591 INFO:tasks.workunit.client.0.smithi062.stdout:6/694: creat d0/d9/d11/d22/dd5/d67/fe7 x:0 0 0 2024-04-20T21:32:14.593 INFO:tasks.workunit.client.0.smithi062.stdout:2/866: getdents d7/d39 0 2024-04-20T21:32:14.602 INFO:tasks.workunit.client.1.smithi189.stdout:8/643: creat da/df/d3c/d68/d8b/dac/fe7 x:0 0 0 2024-04-20T21:32:14.604 INFO:tasks.workunit.client.1.smithi189.stdout:5/526: sync 2024-04-20T21:32:14.604 INFO:tasks.workunit.client.1.smithi189.stdout:4/516: sync 2024-04-20T21:32:14.604 INFO:tasks.workunit.client.0.smithi062.stdout:1/905: dwrite dc/fe9 [0,4194304] 0 2024-04-20T21:32:14.608 INFO:tasks.workunit.client.1.smithi189.stdout:6/601: write d8/d2b/f3a [1719571,28744] 0 2024-04-20T21:32:14.608 INFO:tasks.workunit.client.1.smithi189.stdout:2/511: dread d1/d17/d2f/d55/f5f [0,4194304] 0 2024-04-20T21:32:14.609 INFO:tasks.workunit.client.1.smithi189.stdout:7/598: symlink d5/db4/lca 0 2024-04-20T21:32:14.609 INFO:tasks.workunit.client.1.smithi189.stdout:5/527: write d7/d9/d28/f9f [965754,54281] 0 2024-04-20T21:32:14.610 INFO:tasks.workunit.client.0.smithi062.stdout:3/766: dwrite d0/d1c/f51 [0,4194304] 0 2024-04-20T21:32:14.614 INFO:tasks.workunit.client.1.smithi189.stdout:2/512: readlink d1/d17/d2f/d4a/l82 0 2024-04-20T21:32:14.614 INFO:tasks.workunit.client.1.smithi189.stdout:6/602: readlink d8/d2b/l8b 0 2024-04-20T21:32:14.615 INFO:tasks.workunit.client.1.smithi189.stdout:5/528: stat d7/d14/d4e/d5b/l8c 0 2024-04-20T21:32:14.615 INFO:tasks.workunit.client.1.smithi189.stdout:3/556: rename d2/d3/c46 to d2/d3/d2c/d30/d3f/d9a/da9/cc6 0 2024-04-20T21:32:14.616 INFO:tasks.workunit.client.0.smithi062.stdout:6/695: readlink d0/d9/d11/l97 0 2024-04-20T21:32:14.616 INFO:tasks.workunit.client.1.smithi189.stdout:2/513: read d1/d17/f34 [245672,61547] 0 2024-04-20T21:32:14.620 INFO:tasks.workunit.client.0.smithi062.stdout:9/954: mkdir d16/d13d 0 2024-04-20T21:32:14.622 INFO:tasks.workunit.client.1.smithi189.stdout:2/514: chown d1/d11/d52/c69 27618 1 2024-04-20T21:32:14.624 INFO:tasks.workunit.client.1.smithi189.stdout:2/515: fsync d1/f5 0 2024-04-20T21:32:14.627 INFO:tasks.workunit.client.0.smithi062.stdout:9/955: write d16/d2f/d69/f12f [669009,98458] 0 2024-04-20T21:32:14.627 INFO:tasks.workunit.client.1.smithi189.stdout:6/603: mknod d8/da/dd/d35/cc7 0 2024-04-20T21:32:14.627 INFO:tasks.workunit.client.0.smithi062.stdout:4/906: dwrite da/dd/df/d27/d3f/d82/fbf [0,4194304] 0 2024-04-20T21:32:14.628 INFO:tasks.workunit.client.1.smithi189.stdout:3/557: symlink d2/d8/lc7 0 2024-04-20T21:32:14.631 INFO:tasks.workunit.client.0.smithi062.stdout:5/840: dread d8/d1e/d2c/f86 [0,4194304] 0 2024-04-20T21:32:14.631 INFO:tasks.workunit.client.0.smithi062.stdout:7/808: truncate d7/d3e/d33/d3d/dad/fbc 391922 0 2024-04-20T21:32:14.632 INFO:tasks.workunit.client.0.smithi062.stdout:3/767: mknod d0/d1c/d63/cfd 0 2024-04-20T21:32:14.634 INFO:tasks.workunit.client.1.smithi189.stdout:8/644: symlink da/de/d1e/d69/le8 0 2024-04-20T21:32:14.635 INFO:tasks.workunit.client.1.smithi189.stdout:2/516: dread - d1/d17/d2f/d55/f81 zero size 2024-04-20T21:32:14.636 INFO:tasks.workunit.client.0.smithi062.stdout:5/841: chown d8/d38/f70 88676355 1 2024-04-20T21:32:14.639 INFO:tasks.workunit.client.1.smithi189.stdout:4/517: dwrite d2/f5e [0,4194304] 0 2024-04-20T21:32:14.639 INFO:tasks.workunit.client.1.smithi189.stdout:3/558: chown d2/d8d/fba 3 1 2024-04-20T21:32:14.641 INFO:tasks.workunit.client.0.smithi062.stdout:5/842: write d8/d38/d7a/d4c/d7d/d9d/dac/d2f/d8e/f97 [272090,102095] 0 2024-04-20T21:32:14.645 INFO:tasks.workunit.client.1.smithi189.stdout:2/517: write d1/d11/d58/d6b/f77 [4502040,50703] 0 2024-04-20T21:32:14.647 INFO:tasks.workunit.client.0.smithi062.stdout:9/956: unlink d16/d2f/l51 0 2024-04-20T21:32:14.648 INFO:tasks.workunit.client.1.smithi189.stdout:3/559: mkdir d2/d3/d2c/d30/d3f/d9a/db0/dc8 0 2024-04-20T21:32:14.648 INFO:tasks.workunit.client.1.smithi189.stdout:4/518: unlink d2/d22/f90 0 2024-04-20T21:32:14.651 INFO:tasks.workunit.client.1.smithi189.stdout:4/519: stat d2/d22/f30 0 2024-04-20T21:32:14.651 INFO:tasks.workunit.client.0.smithi062.stdout:6/696: dwrite d0/fb6 [0,4194304] 0 2024-04-20T21:32:14.655 INFO:tasks.workunit.client.1.smithi189.stdout:4/520: write d2/d22/f6e [2931361,56552] 0 2024-04-20T21:32:14.662 INFO:tasks.workunit.client.1.smithi189.stdout:8/645: dread da/df/d13/dd2/fbe [0,4194304] 0 2024-04-20T21:32:14.662 INFO:tasks.workunit.client.1.smithi189.stdout:2/518: link d1/d17/d2f/f42 d1/d17/d2f/d74/f9b 0 2024-04-20T21:32:14.662 INFO:tasks.workunit.client.1.smithi189.stdout:4/521: readlink d2/l6 0 2024-04-20T21:32:14.664 INFO:tasks.workunit.client.1.smithi189.stdout:1/537: truncate d8/d21/f22 3837057 0 2024-04-20T21:32:14.664 INFO:tasks.workunit.client.1.smithi189.stdout:3/560: read d2/d8/f10 [3493975,48961] 0 2024-04-20T21:32:14.665 INFO:tasks.workunit.client.1.smithi189.stdout:4/522: write d2/d64/fc0 [616311,2164] 0 2024-04-20T21:32:14.670 INFO:tasks.workunit.client.1.smithi189.stdout:5/529: dread d7/d9/d31/d36/f38 [0,4194304] 0 2024-04-20T21:32:14.672 INFO:tasks.workunit.client.1.smithi189.stdout:3/561: chown d2/d3/f1b 81 1 2024-04-20T21:32:14.672 INFO:tasks.workunit.client.1.smithi189.stdout:1/538: mkdir d8/d21/d2c/d36/db7 0 2024-04-20T21:32:14.677 INFO:tasks.workunit.client.1.smithi189.stdout:1/539: stat d8/d1e/d43/d44/f47 0 2024-04-20T21:32:14.679 INFO:tasks.workunit.client.1.smithi189.stdout:1/540: fdatasync f1 0 2024-04-20T21:32:14.681 INFO:tasks.workunit.client.0.smithi062.stdout:2/867: dread d7/da/d15/d1b/f34 [0,4194304] 0 2024-04-20T21:32:14.681 INFO:tasks.workunit.client.1.smithi189.stdout:1/541: readlink d8/df/d3a/d49/l53 0 2024-04-20T21:32:14.682 INFO:tasks.workunit.client.1.smithi189.stdout:3/562: mkdir d2/d3/d2c/d30/d3f/d9a/db0/dc8/dc9 0 2024-04-20T21:32:14.685 INFO:tasks.workunit.client.1.smithi189.stdout:4/523: read d2/dd/d6b/f72 [2456787,34684] 0 2024-04-20T21:32:14.686 INFO:tasks.workunit.client.1.smithi189.stdout:3/563: mkdir d2/d1c/d5c/dca 0 2024-04-20T21:32:14.688 INFO:tasks.workunit.client.1.smithi189.stdout:0/650: dwrite d3/d8c/f8d [4194304,4194304] 0 2024-04-20T21:32:14.694 INFO:tasks.workunit.client.1.smithi189.stdout:1/542: rename d8/d1e/d28/d4f/lad to d8/d21/d2c/d36/d8a/lb8 0 2024-04-20T21:32:14.694 INFO:tasks.workunit.client.1.smithi189.stdout:3/564: dread - d2/d3/d2c/d30/d3f/fb3 zero size 2024-04-20T21:32:14.695 INFO:tasks.workunit.client.1.smithi189.stdout:9/534: write d9/dc/d17/d36/f57 [3442560,26679] 0 2024-04-20T21:32:14.696 INFO:tasks.workunit.client.1.smithi189.stdout:8/646: write da/df/d20/f7d [644836,72897] 0 2024-04-20T21:32:14.696 INFO:tasks.workunit.client.1.smithi189.stdout:1/543: fdatasync d8/df/d50/f8d 0 2024-04-20T21:32:14.698 INFO:tasks.workunit.client.1.smithi189.stdout:4/524: creat d2/dd/d6b/da6/fc2 x:0 0 0 2024-04-20T21:32:14.699 INFO:tasks.workunit.client.1.smithi189.stdout:2/519: truncate d1/d11/d2d/f38 3686865 0 2024-04-20T21:32:14.702 INFO:tasks.workunit.client.1.smithi189.stdout:8/647: chown da/de/l1d 6 1 2024-04-20T21:32:14.704 INFO:tasks.workunit.client.1.smithi189.stdout:3/565: fdatasync d2/d1c/d5c/f87 0 2024-04-20T21:32:14.711 INFO:tasks.workunit.client.1.smithi189.stdout:4/525: creat d2/dd/d37/fc3 x:0 0 0 2024-04-20T21:32:14.717 INFO:tasks.workunit.client.1.smithi189.stdout:3/566: creat d2/d3/d2c/d31/d74/d7a/fcb x:0 0 0 2024-04-20T21:32:14.721 INFO:tasks.workunit.client.1.smithi189.stdout:0/651: dread d3/f66 [0,4194304] 0 2024-04-20T21:32:14.721 INFO:tasks.workunit.client.1.smithi189.stdout:8/648: link da/df/d20/d2c/f76 da/df/d3c/d50/fe9 0 2024-04-20T21:32:14.725 INFO:tasks.workunit.client.1.smithi189.stdout:4/526: rename d2/f32 to d2/d20/fc4 0 2024-04-20T21:32:14.734 INFO:tasks.workunit.client.1.smithi189.stdout:4/527: fdatasync d2/dd/f26 0 2024-04-20T21:32:14.735 INFO:tasks.workunit.client.1.smithi189.stdout:1/544: dwrite d8/d21/fb3 [0,4194304] 0 2024-04-20T21:32:14.738 INFO:tasks.workunit.client.1.smithi189.stdout:2/520: dwrite d1/f5 [0,4194304] 0 2024-04-20T21:32:14.738 INFO:tasks.workunit.client.1.smithi189.stdout:7/599: dread d5/d21/d25/f27 [0,4194304] 0 2024-04-20T21:32:14.742 INFO:tasks.workunit.client.1.smithi189.stdout:9/535: dwrite d9/dc/d17/d36/d5d/f97 [0,4194304] 0 2024-04-20T21:32:14.743 INFO:tasks.workunit.client.1.smithi189.stdout:8/649: mknod da/df/d3c/d50/d9e/cea 0 2024-04-20T21:32:14.745 INFO:tasks.workunit.client.1.smithi189.stdout:4/528: write d2/dd/d11/d14/d69/fb3 [373254,1888] 0 2024-04-20T21:32:14.745 INFO:tasks.workunit.client.1.smithi189.stdout:1/545: creat d8/df/d3a/d49/d8c/fb9 x:0 0 0 2024-04-20T21:32:14.748 INFO:tasks.workunit.client.1.smithi189.stdout:2/521: truncate d1/d17/d2f/d55/f81 900138 0 2024-04-20T21:32:14.748 INFO:tasks.workunit.client.1.smithi189.stdout:7/600: creat d5/d21/d97/da3/dc6/fcb x:0 0 0 2024-04-20T21:32:14.748 INFO:tasks.workunit.client.1.smithi189.stdout:5/530: dread d7/d35/f43 [0,4194304] 0 2024-04-20T21:32:14.749 INFO:tasks.workunit.client.0.smithi062.stdout:4/907: symlink da/dd/d15/d4b/ddb/l13c 0 2024-04-20T21:32:14.749 INFO:tasks.workunit.client.1.smithi189.stdout:0/652: getdents d3/df/d28/d2b/db1 0 2024-04-20T21:32:14.750 INFO:tasks.workunit.client.1.smithi189.stdout:4/529: mknod d2/d22/cc5 0 2024-04-20T21:32:14.752 INFO:tasks.workunit.client.1.smithi189.stdout:8/650: chown da/de/d1e/d69/la2 351 1 2024-04-20T21:32:14.752 INFO:tasks.workunit.client.1.smithi189.stdout:7/601: stat d5/d21/d25 0 2024-04-20T21:32:14.754 INFO:tasks.workunit.client.1.smithi189.stdout:2/522: chown d1/d17/d2f/d55/l5b 372 1 2024-04-20T21:32:14.757 INFO:tasks.workunit.client.1.smithi189.stdout:7/602: write d5/db4/d71/fa4 [2719516,89958] 0 2024-04-20T21:32:14.757 INFO:tasks.workunit.client.0.smithi062.stdout:3/768: mkdir d0/d1/d10/d31/dfe 0 2024-04-20T21:32:14.758 INFO:tasks.workunit.client.0.smithi062.stdout:3/769: readlink d0/d1c/l6d 0 2024-04-20T21:32:14.759 INFO:tasks.workunit.client.0.smithi062.stdout:3/770: chown d0/d1c 29499 1 2024-04-20T21:32:14.760 INFO:tasks.workunit.client.1.smithi189.stdout:2/523: creat d1/d11/d52/d6f/f9c x:0 0 0 2024-04-20T21:32:14.762 INFO:tasks.workunit.client.1.smithi189.stdout:5/531: symlink d7/d9/d30/la6 0 2024-04-20T21:32:14.763 INFO:tasks.workunit.client.0.smithi062.stdout:3/771: read d0/d1/f19 [2452977,7323] 0 2024-04-20T21:32:14.764 INFO:tasks.workunit.client.1.smithi189.stdout:8/651: mkdir da/df/d3c/deb 0 2024-04-20T21:32:14.765 INFO:tasks.workunit.client.1.smithi189.stdout:1/546: link d8/df/f92 d8/d21/d2c/fba 0 2024-04-20T21:32:14.768 INFO:tasks.workunit.client.1.smithi189.stdout:4/530: rmdir d2/dd/d37/d9d 0 2024-04-20T21:32:14.770 INFO:tasks.workunit.client.1.smithi189.stdout:1/547: mkdir d8/d9b/dbb 0 2024-04-20T21:32:14.771 INFO:tasks.workunit.client.1.smithi189.stdout:4/531: read - d2/dd/d11/d14/d69/f9e zero size 2024-04-20T21:32:14.771 INFO:tasks.workunit.client.1.smithi189.stdout:0/653: dread d3/df/d30/d37/f65 [0,4194304] 0 2024-04-20T21:32:14.773 INFO:tasks.workunit.client.1.smithi189.stdout:4/532: fsync d2/d7f/fa0 0 2024-04-20T21:32:14.775 INFO:tasks.workunit.client.1.smithi189.stdout:8/652: unlink da/df/d13/dd6/f5a 0 2024-04-20T21:32:14.776 INFO:tasks.workunit.client.1.smithi189.stdout:0/654: chown d3/df/d28/lda 1260043503 1 2024-04-20T21:32:14.776 INFO:tasks.workunit.client.1.smithi189.stdout:4/533: read d2/f5e [673835,5836] 0 2024-04-20T21:32:14.778 INFO:tasks.workunit.client.1.smithi189.stdout:1/548: symlink d8/d1e/d54/da7/lbc 0 2024-04-20T21:32:14.780 INFO:tasks.workunit.client.1.smithi189.stdout:8/653: write da/df/f23 [719577,113972] 0 2024-04-20T21:32:14.781 INFO:tasks.workunit.client.1.smithi189.stdout:0/655: truncate d3/f66 990417 0 2024-04-20T21:32:14.785 INFO:tasks.workunit.client.1.smithi189.stdout:7/603: dwrite d5/d21/d4a/d52/fa7 [0,4194304] 0 2024-04-20T21:32:14.788 INFO:tasks.workunit.client.1.smithi189.stdout:8/654: creat da/df/d3c/d9d/fec x:0 0 0 2024-04-20T21:32:14.789 INFO:tasks.workunit.client.1.smithi189.stdout:0/656: mknod d3/df/d30/d45/cdb 0 2024-04-20T21:32:14.792 INFO:tasks.workunit.client.1.smithi189.stdout:7/604: symlink d5/d59/lcc 0 2024-04-20T21:32:14.793 INFO:tasks.workunit.client.0.smithi062.stdout:9/957: rename d16/d2c/d42/ldb to d16/d2c/d42/da6/l13e 0 2024-04-20T21:32:14.793 INFO:tasks.workunit.client.0.smithi062.stdout:8/849: sync 2024-04-20T21:32:14.794 INFO:tasks.workunit.client.1.smithi189.stdout:5/532: dread d7/d14/d18/f3e [0,4194304] 0 2024-04-20T21:32:14.795 INFO:tasks.workunit.client.1.smithi189.stdout:0/657: chown d3/d26/d88 75682 1 2024-04-20T21:32:14.796 INFO:tasks.workunit.client.1.smithi189.stdout:0/658: chown d3/df/d30/cc6 1 1 2024-04-20T21:32:14.797 INFO:tasks.workunit.client.0.smithi062.stdout:4/908: link da/dd/df/d27/d3f/cef da/dd/d15/daf/d116/c13d 0 2024-04-20T21:32:14.797 INFO:tasks.workunit.client.1.smithi189.stdout:8/655: unlink da/de/f9f 0 2024-04-20T21:32:14.798 INFO:tasks.workunit.client.0.smithi062.stdout:6/697: rename d0/d9/d11/d22/dd5/d67/daa to d0/d9/d11/d7b/d99/ddd/de8 0 2024-04-20T21:32:14.798 INFO:tasks.workunit.client.1.smithi189.stdout:5/533: symlink d7/d9/d31/d36/la7 0 2024-04-20T21:32:14.800 INFO:tasks.workunit.client.0.smithi062.stdout:2/868: getdents d7/da/d15/d1b/d3d 0 2024-04-20T21:32:14.801 INFO:tasks.workunit.client.1.smithi189.stdout:0/659: mknod d3/df/d28/d2b/cdc 0 2024-04-20T21:32:14.804 INFO:tasks.workunit.client.0.smithi062.stdout:9/958: mkdir d16/d2f/d5b/dab/dcd/d13f 0 2024-04-20T21:32:14.804 INFO:tasks.workunit.client.1.smithi189.stdout:5/534: creat d7/d9/d28/da3/fa8 x:0 0 0 2024-04-20T21:32:14.805 INFO:tasks.workunit.client.1.smithi189.stdout:0/660: rmdir d3/db9/dbe 39 2024-04-20T21:32:14.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:14.808+0000 7f3277400700 1 -- 172.21.15.62:0/4170028537 --> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3250000bf0 con 0x7f325806eac0 2024-04-20T21:32:14.809 INFO:tasks.workunit.client.1.smithi189.stdout:0/661: truncate d3/df/d28/f61 654837 0 2024-04-20T21:32:14.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:14.810+0000 7f326d600700 1 -- 172.21.15.62:0/4170028537 <== mgr.14658 v2:172.21.15.62:6800/1585760929 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+329 (secure 0 0 0) 0x7f3250000bf0 con 0x7f325806eac0 2024-04-20T21:32:14.810 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:32:14.810 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T21:32:14.810 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T21:32:14.810 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-20T21:32:14.810 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T21:32:14.810 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr" 2024-04-20T21:32:14.810 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T21:32:14.810 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "2/2 daemons upgraded", 2024-04-20T21:32:14.810 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "", 2024-04-20T21:32:14.810 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T21:32:14.810 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:32:14.811 INFO:tasks.workunit.client.1.smithi189.stdout:7/605: dwrite d5/d46/faa [0,4194304] 0 2024-04-20T21:32:14.812 INFO:tasks.workunit.client.0.smithi062.stdout:6/698: mkdir d0/dc7/d7c/dc5/de9 0 2024-04-20T21:32:14.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:14.813+0000 7f3277400700 1 -- 172.21.15.62:0/4170028537 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f325806eac0 msgr2=0x7f3258070f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:32:14.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:14.813+0000 7f3277400700 1 --2- 172.21.15.62:0/4170028537 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f325806eac0 0x7f3258070f70 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f325c005090 tx=0x7f325c005fb0 comp rx=0 tx=0).stop 2024-04-20T21:32:14.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:14.813+0000 7f3277400700 1 -- 172.21.15.62:0/4170028537 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3270071e00 msgr2=0x7f3270117410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:32:14.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:14.813+0000 7f3277400700 1 --2- 172.21.15.62:0/4170028537 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3270071e00 0x7f3270117410 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f326400b4b0 tx=0x7f326400b870 comp rx=0 tx=0).stop 2024-04-20T21:32:14.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:14.814+0000 7f3277400700 1 -- 172.21.15.62:0/4170028537 shutdown_connections 2024-04-20T21:32:14.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:14.814+0000 7f3277400700 1 --2- 172.21.15.62:0/4170028537 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f325806eac0 0x7f3258070f70 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:32:14.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:14.814+0000 7f3277400700 1 --2- 172.21.15.62:0/4170028537 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3270071e00 0x7f3270117410 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:32:14.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:14.814+0000 7f3277400700 1 --2- 172.21.15.62:0/4170028537 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3270072750 0x7f3270117950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:32:14.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:14.814+0000 7f3277400700 1 -- 172.21.15.62:0/4170028537 >> 172.21.15.62:0/4170028537 conn(0x7f327006d110 msgr2=0x7f327010d080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:32:14.813 INFO:tasks.workunit.client.1.smithi189.stdout:5/535: creat d7/d14/d4e/d5b/fa9 x:0 0 0 2024-04-20T21:32:14.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:14.814+0000 7f3277400700 1 -- 172.21.15.62:0/4170028537 shutdown_connections 2024-04-20T21:32:14.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:14.814+0000 7f3277400700 1 -- 172.21.15.62:0/4170028537 wait complete. 2024-04-20T21:32:14.814 INFO:tasks.workunit.client.0.smithi062.stdout:2/869: fsync d7/da/d15/d2f/f42 0 2024-04-20T21:32:14.815 INFO:tasks.workunit.client.1.smithi189.stdout:0/662: symlink d3/df/d28/d2b/d2c/ldd 0 2024-04-20T21:32:14.817 INFO:tasks.workunit.client.1.smithi189.stdout:5/536: chown d7/d9/d30/d55/d81/c92 55497 1 2024-04-20T21:32:14.818 INFO:tasks.workunit.client.1.smithi189.stdout:8/656: dwrite da/de/f28 [0,4194304] 0 2024-04-20T21:32:14.821 INFO:tasks.workunit.client.1.smithi189.stdout:0/663: chown d3/df/d5d/d64/lbc 412 1 2024-04-20T21:32:14.824 INFO:tasks.workunit.client.1.smithi189.stdout:5/537: mkdir d7/d9/d30/d55/d61/daa 0 2024-04-20T21:32:14.825 INFO:tasks.workunit.client.0.smithi062.stdout:4/909: dwrite da/dd/df/f28 [4194304,4194304] 0 2024-04-20T21:32:14.826 INFO:tasks.workunit.client.0.smithi062.stdout:2/870: truncate d7/da/f13 3585009 0 2024-04-20T21:32:14.826 INFO:tasks.workunit.client.1.smithi189.stdout:8/657: chown da/df/d13/l82 1 1 2024-04-20T21:32:14.828 INFO:tasks.workunit.client.1.smithi189.stdout:5/538: mkdir d7/d9/d31/d36/d5f/dab 0 2024-04-20T21:32:14.829 INFO:tasks.workunit.client.0.smithi062.stdout:6/699: link d0/d9/d11/c95 d0/d9/d11/d22/cea 0 2024-04-20T21:32:14.830 INFO:tasks.workunit.client.1.smithi189.stdout:0/664: creat d3/db9/dbe/dd7/fde x:0 0 0 2024-04-20T21:32:14.832 INFO:tasks.workunit.client.0.smithi062.stdout:6/700: fdatasync d0/d52/fce 0 2024-04-20T21:32:14.832 INFO:tasks.workunit.client.1.smithi189.stdout:5/539: creat d7/d9/d30/d5a/d9d/fac x:0 0 0 2024-04-20T21:32:14.836 INFO:tasks.workunit.client.0.smithi062.stdout:1/906: write dc/d15/f5b [2329800,43560] 0 2024-04-20T21:32:14.838 INFO:tasks.workunit.client.0.smithi062.stdout:9/959: dwrite d16/d2c/d5e/d75/faa [0,4194304] 0 2024-04-20T21:32:14.838 INFO:tasks.workunit.client.0.smithi062.stdout:4/910: truncate da/d56/d71/d9e/ffd 1548237 0 2024-04-20T21:32:14.842 INFO:tasks.workunit.client.1.smithi189.stdout:0/665: rename d3/df/d30/d45/d60/f8b to d3/d26/fdf 0 2024-04-20T21:32:14.843 INFO:tasks.workunit.client.1.smithi189.stdout:0/666: chown d3/df/l9f 9356 1 2024-04-20T21:32:14.843 INFO:tasks.workunit.client.0.smithi062.stdout:4/911: write da/dd/f13b [540765,95617] 0 2024-04-20T21:32:14.847 INFO:tasks.workunit.client.1.smithi189.stdout:5/540: dwrite d7/d9/d51/f72 [0,4194304] 0 2024-04-20T21:32:14.850 INFO:tasks.workunit.client.0.smithi062.stdout:0/796: dwrite d7/d1c/d15/d17/d40/d79/d71/f94 [0,4194304] 0 2024-04-20T21:32:14.858 INFO:tasks.workunit.client.0.smithi062.stdout:1/907: getdents dc 0 2024-04-20T21:32:14.865 INFO:tasks.workunit.client.0.smithi062.stdout:4/912: getdents da/dd/df/d109 0 2024-04-20T21:32:14.866 INFO:tasks.workunit.client.0.smithi062.stdout:9/960: dwrite d16/d2c/d62/f12a [0,4194304] 0 2024-04-20T21:32:14.868 INFO:tasks.workunit.client.1.smithi189.stdout:0/667: link d3/df/d28/d2b/d2c/d74/l7d d3/df/d30/d45/dd3/le0 0 2024-04-20T21:32:14.873 INFO:tasks.workunit.client.1.smithi189.stdout:5/541: creat d7/d9/d30/d55/d61/daa/fad x:0 0 0 2024-04-20T21:32:14.874 INFO:tasks.workunit.client.1.smithi189.stdout:0/668: mkdir d3/df/d30/d37/dbd/de1 0 2024-04-20T21:32:14.882 INFO:tasks.workunit.client.0.smithi062.stdout:9/961: rmdir d16/d2c/d5e/d75/d76/dfe/d101 0 2024-04-20T21:32:14.882 INFO:tasks.workunit.client.0.smithi062.stdout:9/962: readlink d16/d4e/d80/l89 0 2024-04-20T21:32:14.883 INFO:tasks.workunit.client.1.smithi189.stdout:5/542: creat d7/d9/d30/d55/d61/daa/fae x:0 0 0 2024-04-20T21:32:14.888 INFO:tasks.workunit.client.1.smithi189.stdout:0/669: rename d3/df/d30/d37/fbb to d3/df/d28/d2b/fe2 0 2024-04-20T21:32:14.889 INFO:tasks.workunit.client.0.smithi062.stdout:9/963: mknod d16/d2c/d42/da6/da9/c140 0 2024-04-20T21:32:14.894 INFO:tasks.workunit.client.1.smithi189.stdout:8/658: dread da/de/f16 [4194304,4194304] 0 2024-04-20T21:32:14.897 INFO:tasks.workunit.client.0.smithi062.stdout:9/964: read fd [653168,63630] 0 2024-04-20T21:32:14.898 INFO:tasks.workunit.client.1.smithi189.stdout:5/543: rename d7/d9/d30/d55/d61/c6e to d7/d14/d18/caf 0 2024-04-20T21:32:14.898 INFO:tasks.workunit.client.0.smithi062.stdout:9/965: chown d16/d2c/d5e/f10b 0 1 2024-04-20T21:32:14.899 INFO:tasks.workunit.client.0.smithi062.stdout:3/772: creat d0/d1/d10/d3a/d5e/d65/fff x:0 0 0 2024-04-20T21:32:14.900 INFO:tasks.workunit.client.1.smithi189.stdout:9/536: truncate d9/d12/d3b/f46 3102085 0 2024-04-20T21:32:14.902 INFO:tasks.workunit.client.1.smithi189.stdout:8/659: creat da/de/fed x:0 0 0 2024-04-20T21:32:14.904 INFO:tasks.workunit.client.0.smithi062.stdout:3/773: truncate d0/d1/f8b 11083 0 2024-04-20T21:32:14.906 INFO:tasks.workunit.client.1.smithi189.stdout:9/537: creat d9/dc/d17/faa x:0 0 0 2024-04-20T21:32:14.909 INFO:tasks.workunit.client.1.smithi189.stdout:8/660: rename da/df/d3a/fde to da/d81/d91/fee 0 2024-04-20T21:32:14.913 INFO:tasks.workunit.client.1.smithi189.stdout:8/661: truncate da/d1f/d2a/fdf 161478 0 2024-04-20T21:32:14.914 INFO:tasks.workunit.client.1.smithi189.stdout:9/538: mkdir d9/dc/dab 0 2024-04-20T21:32:14.918 INFO:tasks.workunit.client.1.smithi189.stdout:7/606: write d5/fe [2061870,67469] 0 2024-04-20T21:32:14.922 INFO:tasks.workunit.client.1.smithi189.stdout:4/534: dwrite d2/f1a [0,4194304] 0 2024-04-20T21:32:14.922 INFO:tasks.workunit.client.1.smithi189.stdout:7/607: mknod d5/d21/d4a/d52/ccd 0 2024-04-20T21:32:14.923 INFO:tasks.workunit.client.1.smithi189.stdout:8/662: creat da/df/d3c/d50/d6c/dc7/fef x:0 0 0 2024-04-20T21:32:14.925 INFO:tasks.workunit.client.1.smithi189.stdout:7/608: read d5/f43 [213794,90623] 0 2024-04-20T21:32:14.926 INFO:tasks.workunit.client.1.smithi189.stdout:4/535: write d2/d3a/f63 [2340295,49258] 0 2024-04-20T21:32:14.930 INFO:tasks.workunit.client.1.smithi189.stdout:8/663: fsync da/df/d20/d27/f38 0 2024-04-20T21:32:14.932 INFO:tasks.workunit.client.1.smithi189.stdout:7/609: creat d5/d21/d4a/d52/d75/dc8/fce x:0 0 0 2024-04-20T21:32:14.935 INFO:tasks.workunit.client.1.smithi189.stdout:7/610: write d5/d46/f8b [1303653,109792] 0 2024-04-20T21:32:14.935 INFO:tasks.workunit.client.1.smithi189.stdout:8/664: dread - da/df/d3c/d68/d8b/fc1 zero size 2024-04-20T21:32:14.939 INFO:tasks.workunit.client.1.smithi189.stdout:6/604: dread d8/d2b/f3a [0,4194304] 0 2024-04-20T21:32:14.944 INFO:tasks.workunit.client.1.smithi189.stdout:6/605: fdatasync d8/d12/fae 0 2024-04-20T21:32:14.958 INFO:tasks.workunit.client.1.smithi189.stdout:4/536: dwrite d2/d3a/f91 [0,4194304] 0 2024-04-20T21:32:14.962 INFO:tasks.workunit.client.1.smithi189.stdout:6/606: write d8/da/dd/d35/f45 [530509,48689] 0 2024-04-20T21:32:14.970 INFO:tasks.workunit.client.1.smithi189.stdout:8/665: dwrite da/de/d1e/d69/fc8 [0,4194304] 0 2024-04-20T21:32:14.971 INFO:tasks.workunit.client.1.smithi189.stdout:4/537: mkdir d2/d20/dc6 0 2024-04-20T21:32:14.976 INFO:tasks.workunit.client.1.smithi189.stdout:8/666: fsync da/df/d13/dd6/f36 0 2024-04-20T21:32:14.978 INFO:tasks.workunit.client.1.smithi189.stdout:7/611: dwrite d5/d21/d25/f6c [0,4194304] 0 2024-04-20T21:32:14.981 INFO:tasks.workunit.client.1.smithi189.stdout:8/667: chown da/de 2 1 2024-04-20T21:32:14.981 INFO:tasks.workunit.client.1.smithi189.stdout:4/538: creat d2/dd/d6b/fc7 x:0 0 0 2024-04-20T21:32:14.983 INFO:tasks.workunit.client.1.smithi189.stdout:7/612: write d5/db4/d71/fc7 [646713,21622] 0 2024-04-20T21:32:14.991 INFO:tasks.workunit.client.1.smithi189.stdout:8/668: mkdir da/df/d13/dd6/df0 0 2024-04-20T21:32:15.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:14 smithi062 ceph-mon[16221]: from='client.24491 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:32:15.143 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:14 smithi062 ceph-mon[16221]: from='client.? 172.21.15.62:0/2500135782' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:32:15.152 INFO:tasks.workunit.client.0.smithi062.stdout:8/850: truncate d7/d25/d3b/d58/d8b/d24/f67 1066217 0 2024-04-20T21:32:15.152 INFO:tasks.workunit.client.0.smithi062.stdout:8/851: fdatasync d7/d25/d3b/d58/db9/f113 0 2024-04-20T21:32:15.152 INFO:tasks.workunit.client.0.smithi062.stdout:8/852: creat d7/d30/f114 x:0 0 0 2024-04-20T21:32:15.153 INFO:tasks.workunit.client.0.smithi062.stdout:2/871: write d7/da/fdf [798718,44641] 0 2024-04-20T21:32:15.153 INFO:tasks.workunit.client.0.smithi062.stdout:8/853: fsync d7/d25/d4f/fb8 0 2024-04-20T21:32:15.153 INFO:tasks.workunit.client.0.smithi062.stdout:4/913: sync 2024-04-20T21:32:15.153 INFO:tasks.workunit.client.0.smithi062.stdout:5/843: mkdir d8/d118 0 2024-04-20T21:32:15.153 INFO:tasks.workunit.client.0.smithi062.stdout:9/966: truncate d16/d2c/d42/f54 2796010 0 2024-04-20T21:32:15.153 INFO:tasks.workunit.client.0.smithi062.stdout:3/774: write d0/d1/d10/d3a/d5e/f7d [748665,14610] 0 2024-04-20T21:32:15.153 INFO:tasks.workunit.client.0.smithi062.stdout:7/809: creat d7/d3e/d33/d3d/d4d/f11b x:0 0 0 2024-04-20T21:32:15.154 INFO:tasks.workunit.client.0.smithi062.stdout:3/775: fsync d0/fdc 0 2024-04-20T21:32:15.154 INFO:tasks.workunit.client.0.smithi062.stdout:7/810: fdatasync f4 0 2024-04-20T21:32:15.154 INFO:tasks.workunit.client.0.smithi062.stdout:9/967: creat d16/d2c/d5e/d96/f141 x:0 0 0 2024-04-20T21:32:15.154 INFO:tasks.workunit.client.0.smithi062.stdout:7/811: chown d7/d3e/d33/d3d/d5e/d72/l6f 32687 1 2024-04-20T21:32:15.154 INFO:tasks.workunit.client.0.smithi062.stdout:9/968: symlink d16/d2f/d5b/dab/dcd/l142 0 2024-04-20T21:32:15.154 INFO:tasks.workunit.client.0.smithi062.stdout:0/797: rename d7/d20/d3a to d7/d1c/d103 0 2024-04-20T21:32:15.154 INFO:tasks.workunit.client.0.smithi062.stdout:0/798: creat d7/d1e/da9/dbe/f104 x:0 0 0 2024-04-20T21:32:15.155 INFO:tasks.workunit.client.0.smithi062.stdout:6/701: creat d0/d9/d11/d7b/feb x:0 0 0 2024-04-20T21:32:15.155 INFO:tasks.workunit.client.0.smithi062.stdout:0/799: stat d7/d1e/d6e/f93 0 2024-04-20T21:32:15.155 INFO:tasks.workunit.client.0.smithi062.stdout:1/908: creat dc/f14b x:0 0 0 2024-04-20T21:32:15.155 INFO:tasks.workunit.client.0.smithi062.stdout:8/854: write d7/d9/fd7 [2741455,126275] 0 2024-04-20T21:32:15.155 INFO:tasks.workunit.client.0.smithi062.stdout:6/702: chown d0/d9/d11/c14 38455 1 2024-04-20T21:32:15.155 INFO:tasks.workunit.client.0.smithi062.stdout:2/872: dread d7/da/d15/d2f/d38/d10f/d4b/f95 [0,4194304] 0 2024-04-20T21:32:15.155 INFO:tasks.workunit.client.0.smithi062.stdout:9/969: dwrite d16/d2c/d42/da6/da9/ff4 [0,4194304] 0 2024-04-20T21:32:15.156 INFO:tasks.workunit.client.1.smithi189.stdout:3/567: sync 2024-04-20T21:32:15.156 INFO:tasks.workunit.client.1.smithi189.stdout:7/613: mkdir d5/d1d/d64/db9/dcf 0 2024-04-20T21:32:15.156 INFO:tasks.workunit.client.1.smithi189.stdout:4/539: read d2/dd/d11/f31 [478072,22556] 0 2024-04-20T21:32:15.156 INFO:tasks.workunit.client.1.smithi189.stdout:4/540: dread - d2/dd/d6b/fc7 zero size 2024-04-20T21:32:15.157 INFO:tasks.workunit.client.1.smithi189.stdout:8/669: rename da/le5 to da/df/d3c/d68/lf1 0 2024-04-20T21:32:15.157 INFO:tasks.workunit.client.1.smithi189.stdout:7/614: unlink d5/d46/d7d/lb3 0 2024-04-20T21:32:15.157 INFO:tasks.workunit.client.1.smithi189.stdout:3/568: chown d2/d3/d2c/d30/d3f/d9a/da9 199456343 1 2024-04-20T21:32:15.157 INFO:tasks.workunit.client.1.smithi189.stdout:6/607: dwrite d8/d2b/f75 [4194304,4194304] 0 2024-04-20T21:32:15.157 INFO:tasks.workunit.client.1.smithi189.stdout:8/670: mkdir da/df/d3c/d50/ddc/df2 0 2024-04-20T21:32:15.157 INFO:tasks.workunit.client.1.smithi189.stdout:6/608: write d8/d12/daf/fb5 [895419,55925] 0 2024-04-20T21:32:15.158 INFO:tasks.workunit.client.1.smithi189.stdout:8/671: dread - da/df/d20/d27/fe3 zero size 2024-04-20T21:32:15.158 INFO:tasks.workunit.client.1.smithi189.stdout:6/609: rename d8/da/dd/d35/d6d/d7b/fb3 to d8/da/dd/d35/fc8 0 2024-04-20T21:32:15.158 INFO:tasks.workunit.client.1.smithi189.stdout:6/610: write d8/d2b/f75 [7299592,30629] 0 2024-04-20T21:32:15.158 INFO:tasks.workunit.client.1.smithi189.stdout:6/611: creat d8/d12/d30/d47/fc9 x:0 0 0 2024-04-20T21:32:15.158 INFO:tasks.workunit.client.1.smithi189.stdout:6/612: read d8/f63 [2575546,89164] 0 2024-04-20T21:32:15.158 INFO:tasks.workunit.client.1.smithi189.stdout:6/613: write d8/da/dd/d35/f59 [3034889,68126] 0 2024-04-20T21:32:15.159 INFO:tasks.workunit.client.1.smithi189.stdout:4/541: truncate d2/dd/d11/f58 3596286 0 2024-04-20T21:32:15.159 INFO:tasks.workunit.client.1.smithi189.stdout:6/614: fsync d8/d12/d4e/f50 0 2024-04-20T21:32:15.159 INFO:tasks.workunit.client.1.smithi189.stdout:4/542: rename d2/l76 to d2/d22/d57/lc8 0 2024-04-20T21:32:15.159 INFO:tasks.workunit.client.1.smithi189.stdout:3/569: dwrite d2/d3/d2c/d30/d3f/d9a/db0/fb8 [0,4194304] 0 2024-04-20T21:32:15.159 INFO:tasks.workunit.client.1.smithi189.stdout:2/524: sync 2024-04-20T21:32:15.159 INFO:tasks.workunit.client.1.smithi189.stdout:4/543: chown d2/f5e 192819 1 2024-04-20T21:32:15.159 INFO:tasks.workunit.client.1.smithi189.stdout:8/672: dwrite da/d1f/d2a/f58 [0,4194304] 0 2024-04-20T21:32:15.160 INFO:tasks.workunit.client.1.smithi189.stdout:2/525: write d1/d11/d52/d8b/f8d [957741,43514] 0 2024-04-20T21:32:15.160 INFO:tasks.workunit.client.1.smithi189.stdout:2/526: stat d1/d11/d2d/d4f/d8e 0 2024-04-20T21:32:15.160 INFO:tasks.workunit.client.1.smithi189.stdout:7/615: dread d5/d21/d4a/f8d [0,4194304] 0 2024-04-20T21:32:15.160 INFO:tasks.workunit.client.1.smithi189.stdout:7/616: fsync d5/d46/f8b 0 2024-04-20T21:32:15.161 INFO:tasks.workunit.client.0.smithi062.stdout:8/855: chown d7/d9/c108 7920306 1 2024-04-20T21:32:15.161 INFO:tasks.workunit.client.0.smithi062.stdout:6/703: mknod d0/cec 0 2024-04-20T21:32:15.161 INFO:tasks.workunit.client.0.smithi062.stdout:0/800: truncate d7/d1c/d15/d17/d40/d3c/dc5/f88 4256881 0 2024-04-20T21:32:15.163 INFO:tasks.workunit.client.0.smithi062.stdout:3/776: dwrite d0/d1/d3c/d43/d72/dd3/d7b/fc2 [0,4194304] 0 2024-04-20T21:32:15.164 INFO:tasks.workunit.client.0.smithi062.stdout:9/970: symlink d16/dbe/l143 0 2024-04-20T21:32:15.165 INFO:tasks.workunit.client.0.smithi062.stdout:7/812: dwrite d7/d3e/d33/d3d/d4d/d54/d4f/fb3 [0,4194304] 0 2024-04-20T21:32:15.167 INFO:tasks.workunit.client.0.smithi062.stdout:0/801: creat d7/d20/d62/d6f/f105 x:0 0 0 2024-04-20T21:32:15.167 INFO:tasks.workunit.client.0.smithi062.stdout:8/856: mknod d7/d25/d3b/d58/d8b/c115 0 2024-04-20T21:32:15.169 INFO:tasks.workunit.client.1.smithi189.stdout:9/539: sync 2024-04-20T21:32:15.169 INFO:tasks.workunit.client.1.smithi189.stdout:5/544: sync 2024-04-20T21:32:15.169 INFO:tasks.workunit.client.1.smithi189.stdout:1/549: sync 2024-04-20T21:32:15.169 INFO:tasks.workunit.client.1.smithi189.stdout:0/670: sync 2024-04-20T21:32:15.169 INFO:tasks.workunit.client.0.smithi062.stdout:0/802: stat d7/d1c/d15/cf7 0 2024-04-20T21:32:15.171 INFO:tasks.workunit.client.0.smithi062.stdout:6/704: dread d0/d9/f71 [0,4194304] 0 2024-04-20T21:32:15.172 INFO:tasks.workunit.client.0.smithi062.stdout:8/857: unlink d7/de1/f100 0 2024-04-20T21:32:15.172 INFO:tasks.workunit.client.0.smithi062.stdout:7/813: fsync d7/d3e/d33/f37 0 2024-04-20T21:32:15.175 INFO:tasks.workunit.client.0.smithi062.stdout:2/873: write d7/da/d15/f1d [1789745,119482] 0 2024-04-20T21:32:15.178 INFO:tasks.workunit.client.0.smithi062.stdout:1/909: rename dc/ff to dc/d15/d75/d94/d6f/dd2/de3/d121/f14c 0 2024-04-20T21:32:15.180 INFO:tasks.workunit.client.0.smithi062.stdout:6/705: mknod d0/d1a/d7d/ced 0 2024-04-20T21:32:15.182 INFO:tasks.workunit.client.0.smithi062.stdout:6/706: write d0/d52/da0/fcc [391538,45465] 0 2024-04-20T21:32:15.184 INFO:tasks.workunit.client.0.smithi062.stdout:0/803: rename d7/d1c/d15/d17/d1b/d9d to d7/d1c/d15/d17/d1b/d106 0 2024-04-20T21:32:15.185 INFO:tasks.workunit.client.0.smithi062.stdout:7/814: fdatasync d7/d3e/d33/d3d/d4d/dce/fe3 0 2024-04-20T21:32:15.187 INFO:tasks.workunit.client.0.smithi062.stdout:2/874: fdatasync d7/da/d15/d1b/d3d/d84/fa9 0 2024-04-20T21:32:15.187 INFO:tasks.workunit.client.0.smithi062.stdout:8/858: creat d7/d25/d70/d101/f116 x:0 0 0 2024-04-20T21:32:15.188 INFO:tasks.workunit.client.0.smithi062.stdout:6/707: mknod d0/d9/d11/d7b/d99/ddd/de8/cee 0 2024-04-20T21:32:15.188 INFO:tasks.workunit.client.0.smithi062.stdout:0/804: chown d7/d1c/d15/d17/d37/df8 8083 1 2024-04-20T21:32:15.195 INFO:tasks.workunit.client.0.smithi062.stdout:7/815: read d7/d3e/d33/d3d/d4d/d54/f1f [28154,46950] 0 2024-04-20T21:32:15.196 INFO:tasks.workunit.client.0.smithi062.stdout:0/805: readlink d7/d1c/d15/d17/d40/d3c/lf3 0 2024-04-20T21:32:15.197 INFO:tasks.workunit.client.0.smithi062.stdout:8/859: creat d7/d25/d3b/d58/d8b/d24/d46/f117 x:0 0 0 2024-04-20T21:32:15.198 INFO:tasks.workunit.client.1.smithi189.stdout:3/570: creat d2/d1c/dc1/dc5/fcc x:0 0 0 2024-04-20T21:32:15.198 INFO:tasks.workunit.client.1.smithi189.stdout:7/617: rmdir d5/db4/d71/d7a 39 2024-04-20T21:32:15.200 INFO:tasks.workunit.client.0.smithi062.stdout:3/777: rename d0/d1/d10/d31/dfe to d0/d1/d100 0 2024-04-20T21:32:15.201 INFO:tasks.workunit.client.0.smithi062.stdout:0/806: creat d7/d20/d62/d6f/f107 x:0 0 0 2024-04-20T21:32:15.203 INFO:tasks.workunit.client.1.smithi189.stdout:9/540: dread d9/d12/d3f/f68 [0,4194304] 0 2024-04-20T21:32:15.205 INFO:tasks.workunit.client.0.smithi062.stdout:0/807: write d7/d1c/d15/f100 [161616,32470] 0 2024-04-20T21:32:15.205 INFO:tasks.workunit.client.1.smithi189.stdout:7/618: dread - d5/d21/d4a/d52/d75/fb2 zero size 2024-04-20T21:32:15.206 INFO:tasks.workunit.client.0.smithi062.stdout:6/708: rename d0/dc7/de4 to d0/dc7/d7c/dc5/def 0 2024-04-20T21:32:15.209 INFO:tasks.workunit.client.0.smithi062.stdout:3/778: symlink d0/d1/d3c/d43/d72/dd3/dbf/de8/dfc/l101 0 2024-04-20T21:32:15.210 INFO:tasks.workunit.client.0.smithi062.stdout:0/808: rmdir d7/d1c/d15/d17/d40/d79 39 2024-04-20T21:32:15.214 INFO:tasks.workunit.client.1.smithi189.stdout:8/673: dread da/df/d13/f18 [0,4194304] 0 2024-04-20T21:32:15.214 INFO:tasks.workunit.client.0.smithi062.stdout:6/709: symlink d0/d9/d11/lf0 0 2024-04-20T21:32:15.215 INFO:tasks.workunit.client.0.smithi062.stdout:8/860: dwrite d7/d25/d76/f9c [0,4194304] 0 2024-04-20T21:32:15.216 INFO:tasks.workunit.client.1.smithi189.stdout:8/674: dread - da/df/d22/faf zero size 2024-04-20T21:32:15.216 INFO:tasks.workunit.client.0.smithi062.stdout:6/710: readlink d0/l6c 0 2024-04-20T21:32:15.216 INFO:tasks.workunit.client.0.smithi062.stdout:0/809: creat d7/d20/d62/d6f/f108 x:0 0 0 2024-04-20T21:32:15.217 INFO:tasks.workunit.client.0.smithi062.stdout:7/816: dwrite d7/d3e/d33/d3d/d5e/d72/f98 [4194304,4194304] 0 2024-04-20T21:32:15.219 INFO:tasks.workunit.client.0.smithi062.stdout:3/779: mkdir d0/d1/d10/d31/d41/de6/d102 0 2024-04-20T21:32:15.220 INFO:tasks.workunit.client.0.smithi062.stdout:8/861: creat d7/d30/f118 x:0 0 0 2024-04-20T21:32:15.220 INFO:tasks.workunit.client.0.smithi062.stdout:7/817: dread - d7/d3e/d33/d3d/d4d/d54/dea/f10b zero size 2024-04-20T21:32:15.221 INFO:tasks.workunit.client.0.smithi062.stdout:1/910: mkdir dc/d15/d75/d94/d6f/dd2/de3/d121/d14d 0 2024-04-20T21:32:15.226 INFO:tasks.workunit.client.0.smithi062.stdout:3/780: creat d0/d1c/d63/d42/dc8/f103 x:0 0 0 2024-04-20T21:32:15.232 INFO:tasks.workunit.client.0.smithi062.stdout:7/818: read d7/d3e/d33/d3d/fca [1847227,37275] 0 2024-04-20T21:32:15.233 INFO:tasks.workunit.client.0.smithi062.stdout:1/911: mkdir dc/d15/d75/d94/d6f/dd2/de3/d121/d14d/d14e 0 2024-04-20T21:32:15.234 INFO:tasks.workunit.client.0.smithi062.stdout:2/875: dread d7/da/d15/d2f/fa4 [0,4194304] 0 2024-04-20T21:32:15.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:14 smithi189 ceph-mon[21562]: from='client.24491 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:32:15.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:14 smithi189 ceph-mon[21562]: from='client.? 172.21.15.62:0/2500135782' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:32:15.238 INFO:tasks.workunit.client.0.smithi062.stdout:3/781: getdents d0/d1/d3c/d43/d67 0 2024-04-20T21:32:15.240 INFO:tasks.workunit.client.0.smithi062.stdout:7/819: truncate d7/f2d 847444 0 2024-04-20T21:32:15.241 INFO:tasks.workunit.client.0.smithi062.stdout:2/876: creat d7/da/d15/d2f/d97/f11d x:0 0 0 2024-04-20T21:32:15.242 INFO:tasks.workunit.client.0.smithi062.stdout:3/782: readlink d0/d1/d10/l26 0 2024-04-20T21:32:15.243 INFO:tasks.workunit.client.0.smithi062.stdout:1/912: dread - dc/d15/d75/d96/dd4/d3a/da2/de4/d7c/f116 zero size 2024-04-20T21:32:15.246 INFO:tasks.workunit.client.0.smithi062.stdout:8/862: dwrite d7/d25/d70/fe4 [0,4194304] 0 2024-04-20T21:32:15.247 INFO:tasks.workunit.client.0.smithi062.stdout:1/913: symlink dc/d15/l14f 0 2024-04-20T21:32:15.248 INFO:tasks.workunit.client.0.smithi062.stdout:6/711: dwrite d0/d9/f15 [0,4194304] 0 2024-04-20T21:32:15.249 INFO:tasks.workunit.client.0.smithi062.stdout:2/877: readlink d7/da/d15/d2f/d38/d51/dbc/l103 0 2024-04-20T21:32:15.250 INFO:tasks.workunit.client.0.smithi062.stdout:3/783: symlink d0/d1c/d63/dda/l104 0 2024-04-20T21:32:15.251 INFO:tasks.workunit.client.0.smithi062.stdout:7/820: chown d7/d3e/d33/d3d/d4d/d54/d4f/de9/df1/c119 0 1 2024-04-20T21:32:15.254 INFO:tasks.workunit.client.0.smithi062.stdout:6/712: symlink d0/d1a/d7d/db1/de0/lf1 0 2024-04-20T21:32:15.255 INFO:tasks.workunit.client.0.smithi062.stdout:7/821: write d7/d3e/f5d [500443,2453] 0 2024-04-20T21:32:15.257 INFO:tasks.workunit.client.0.smithi062.stdout:8/863: creat d7/f119 x:0 0 0 2024-04-20T21:32:15.259 INFO:tasks.workunit.client.0.smithi062.stdout:1/914: mkdir dc/d15/d75/d96/dd4/d3a/da2/de4/d84/dbe/d10d/d150 0 2024-04-20T21:32:15.260 INFO:tasks.workunit.client.0.smithi062.stdout:0/810: dread d7/d1e/d6e/f93 [0,4194304] 0 2024-04-20T21:32:15.261 INFO:tasks.workunit.client.0.smithi062.stdout:2/878: unlink d7/da/d15/d2f/c36 0 2024-04-20T21:32:15.264 INFO:tasks.workunit.client.0.smithi062.stdout:2/879: chown d7/da/d15/d2f/d38/d10f/d4b/f6e 1953 1 2024-04-20T21:32:15.264 INFO:tasks.workunit.client.0.smithi062.stdout:8/864: unlink d7/d25/dc5/dd3/l106 0 2024-04-20T21:32:15.265 INFO:tasks.workunit.client.0.smithi062.stdout:2/880: fdatasync d7/da/d15/d1b/ff3 0 2024-04-20T21:32:15.269 INFO:tasks.workunit.client.0.smithi062.stdout:7/822: truncate d7/d3e/d33/d3d/d4d/d54/d4f/fc2 98727 0 2024-04-20T21:32:15.269 INFO:tasks.workunit.client.0.smithi062.stdout:3/784: write d0/d1b/f4d [795283,4762] 0 2024-04-20T21:32:15.269 INFO:tasks.workunit.client.1.smithi189.stdout:6/615: truncate d8/d12/f17 3350319 0 2024-04-20T21:32:15.272 INFO:tasks.workunit.client.0.smithi062.stdout:1/915: mkdir dc/d15/d20/d28/d83/d151 0 2024-04-20T21:32:15.272 INFO:tasks.workunit.client.1.smithi189.stdout:4/544: write d2/f2c [249582,130855] 0 2024-04-20T21:32:15.272 INFO:tasks.workunit.client.1.smithi189.stdout:6/616: fdatasync d8/d2b/f74 0 2024-04-20T21:32:15.274 INFO:tasks.workunit.client.0.smithi062.stdout:6/713: link d0/d9/c8e d0/dc7/cf2 0 2024-04-20T21:32:15.274 INFO:tasks.workunit.client.1.smithi189.stdout:5/545: creat d7/d9/d28/fb0 x:0 0 0 2024-04-20T21:32:15.275 INFO:tasks.workunit.client.1.smithi189.stdout:4/545: write d2/f54 [1487644,45194] 0 2024-04-20T21:32:15.275 INFO:tasks.workunit.client.0.smithi062.stdout:3/785: stat d0/d11/c2a 0 2024-04-20T21:32:15.278 INFO:tasks.workunit.client.1.smithi189.stdout:3/571: unlink d2/d3/d2c/d31/d74/d7a/fcb 0 2024-04-20T21:32:15.281 INFO:tasks.workunit.client.1.smithi189.stdout:8/675: readlink da/lb0 0 2024-04-20T21:32:15.281 INFO:tasks.workunit.client.1.smithi189.stdout:3/572: fdatasync d2/d3/d25/f95 0 2024-04-20T21:32:15.283 INFO:tasks.workunit.client.0.smithi062.stdout:0/811: getdents d7/d1c/d15/d17/d26/d76 0 2024-04-20T21:32:15.283 INFO:tasks.workunit.client.1.smithi189.stdout:8/676: fsync da/df/d20/d2c/f92 0 2024-04-20T21:32:15.289 INFO:tasks.workunit.client.0.smithi062.stdout:7/823: write d7/d3e/d33/d3d/fb1 [275002,56556] 0 2024-04-20T21:32:15.290 INFO:tasks.workunit.client.1.smithi189.stdout:9/541: mkdir d9/dc/d17/dac 0 2024-04-20T21:32:15.291 INFO:tasks.workunit.client.1.smithi189.stdout:7/619: symlink d5/db4/ld0 0 2024-04-20T21:32:15.293 INFO:tasks.workunit.client.1.smithi189.stdout:7/620: fdatasync d5/f7 0 2024-04-20T21:32:15.293 INFO:tasks.workunit.client.0.smithi062.stdout:8/865: dwrite d7/d25/d4f/fdf [0,4194304] 0 2024-04-20T21:32:15.294 INFO:tasks.workunit.client.1.smithi189.stdout:0/671: rename d3/df/d5d/f99 to d3/d26/fe3 0 2024-04-20T21:32:15.298 INFO:tasks.workunit.client.1.smithi189.stdout:2/527: truncate d1/d17/f2b 170896 0 2024-04-20T21:32:15.301 INFO:tasks.workunit.client.0.smithi062.stdout:7/824: readlink d7/d3e/d33/d3d/d4d/d54/l3a 0 2024-04-20T21:32:15.302 INFO:tasks.workunit.client.1.smithi189.stdout:7/621: chown d5/d21/c88 1988 1 2024-04-20T21:32:15.302 INFO:tasks.workunit.client.0.smithi062.stdout:2/881: dwrite d7/da/d15/d2f/d38/d6a/d7c/f8c [0,4194304] 0 2024-04-20T21:32:15.303 INFO:tasks.workunit.client.0.smithi062.stdout:3/786: write d0/d1/d10/d31/f92 [651713,61339] 0 2024-04-20T21:32:15.304 INFO:tasks.workunit.client.1.smithi189.stdout:6/617: rename d8/d10/d78/f7d to d8/d12/d99/fca 0 2024-04-20T21:32:15.306 INFO:tasks.workunit.client.0.smithi062.stdout:6/714: dwrite d0/d9/f16 [0,4194304] 0 2024-04-20T21:32:15.308 INFO:tasks.workunit.client.1.smithi189.stdout:1/550: truncate d8/f86 1358354 0 2024-04-20T21:32:15.308 INFO:tasks.workunit.client.0.smithi062.stdout:8/866: symlink d7/d30/db3/dbf/l11a 0 2024-04-20T21:32:15.309 INFO:tasks.workunit.client.1.smithi189.stdout:7/622: truncate d5/d59/f80 250157 0 2024-04-20T21:32:15.309 INFO:tasks.workunit.client.1.smithi189.stdout:0/672: symlink d3/df/d28/d2b/d2c/le4 0 2024-04-20T21:32:15.314 INFO:tasks.workunit.client.0.smithi062.stdout:1/916: dwrite dc/d15/d75/d96/dd4/f54 [4194304,4194304] 0 2024-04-20T21:32:15.314 INFO:tasks.workunit.client.1.smithi189.stdout:4/546: getdents d2/dd/d11 0 2024-04-20T21:32:15.314 INFO:tasks.workunit.client.0.smithi062.stdout:7/825: creat d7/d26/dac/d10a/f11c x:0 0 0 2024-04-20T21:32:15.318 INFO:tasks.workunit.client.1.smithi189.stdout:0/673: read - d3/d8c/fc7 zero size 2024-04-20T21:32:15.319 INFO:tasks.workunit.client.1.smithi189.stdout:1/551: chown d8/d1e/d28/f90 494 1 2024-04-20T21:32:15.320 INFO:tasks.workunit.client.0.smithi062.stdout:8/867: creat d7/d25/d3b/d58/dac/f11b x:0 0 0 2024-04-20T21:32:15.321 INFO:tasks.workunit.client.0.smithi062.stdout:0/812: dwrite d7/d1c/d15/f2d [0,4194304] 0 2024-04-20T21:32:15.327 INFO:tasks.workunit.client.0.smithi062.stdout:7/826: creat d7/d26/dac/ddb/de8/f11d x:0 0 0 2024-04-20T21:32:15.327 INFO:tasks.workunit.client.1.smithi189.stdout:9/542: dwrite d9/d12/d3b/f4c [0,4194304] 0 2024-04-20T21:32:15.327 INFO:tasks.workunit.client.1.smithi189.stdout:0/674: read d3/df/d5d/d64/f6b [159142,130086] 0 2024-04-20T21:32:15.329 INFO:tasks.workunit.client.1.smithi189.stdout:4/547: unlink d2/dd/la3 0 2024-04-20T21:32:15.330 INFO:tasks.workunit.client.0.smithi062.stdout:3/787: rename d0/d1c/f45 to d0/d1c/d63/d42/dc8/df6/f105 0 2024-04-20T21:32:15.331 INFO:tasks.workunit.client.1.smithi189.stdout:8/677: dread da/de/d1e/fb7 [0,4194304] 0 2024-04-20T21:32:15.334 INFO:tasks.workunit.client.0.smithi062.stdout:2/882: dwrite d7/d39/f45 [4194304,4194304] 0 2024-04-20T21:32:15.335 INFO:tasks.workunit.client.1.smithi189.stdout:5/546: dwrite d7/d35/f43 [0,4194304] 0 2024-04-20T21:32:15.339 INFO:tasks.workunit.client.1.smithi189.stdout:3/573: rename d2/d8/fb4 to d2/d1c/fcd 0 2024-04-20T21:32:15.341 INFO:tasks.workunit.client.0.smithi062.stdout:1/917: truncate dc/d15/d20/d28/f63 2123849 0 2024-04-20T21:32:15.342 INFO:tasks.workunit.client.0.smithi062.stdout:7/827: creat d7/d3e/d33/d3d/dad/f11e x:0 0 0 2024-04-20T21:32:15.347 INFO:tasks.workunit.client.1.smithi189.stdout:5/547: truncate d7/d9/d28/f9b 956931 0 2024-04-20T21:32:15.347 INFO:tasks.workunit.client.0.smithi062.stdout:3/788: mknod d0/d1b/c106 0 2024-04-20T21:32:15.348 INFO:tasks.workunit.client.1.smithi189.stdout:6/618: dwrite d8/d2b/d49/d54/d67/f7a [4194304,4194304] 0 2024-04-20T21:32:15.348 INFO:tasks.workunit.client.0.smithi062.stdout:1/918: dread - dc/d15/d75/d96/dd4/d3a/da2/de4/d84/dbe/d10d/d110/f12c zero size 2024-04-20T21:32:15.349 INFO:tasks.workunit.client.0.smithi062.stdout:2/883: rename d7/da/d15/d1b/d3d/l11a to d7/da/d15/d2f/d97/l11e 0 2024-04-20T21:32:15.352 INFO:tasks.workunit.client.0.smithi062.stdout:2/884: chown d7/da/d15/f23 78221 1 2024-04-20T21:32:15.354 INFO:tasks.workunit.client.0.smithi062.stdout:1/919: stat dc/d15/d20/f45 0 2024-04-20T21:32:15.355 INFO:tasks.workunit.client.0.smithi062.stdout:8/868: getdents d7/d9/d29 0 2024-04-20T21:32:15.357 INFO:tasks.workunit.client.1.smithi189.stdout:8/678: read da/df/d13/f18 [4162762,66179] 0 2024-04-20T21:32:15.360 INFO:tasks.workunit.client.1.smithi189.stdout:5/548: mkdir d7/d9/d28/db1 0 2024-04-20T21:32:15.364 INFO:tasks.workunit.client.0.smithi062.stdout:0/813: write d7/d1c/d15/fca [764064,88265] 0 2024-04-20T21:32:15.365 INFO:tasks.workunit.client.0.smithi062.stdout:7/828: link d7/c13 d7/d3e/d33/d3d/d4d/d83/c11f 0 2024-04-20T21:32:15.365 INFO:tasks.workunit.client.0.smithi062.stdout:2/885: mkdir d7/da/d15/d1b/d3d/d11f 0 2024-04-20T21:32:15.366 INFO:tasks.workunit.client.0.smithi062.stdout:8/869: mkdir d7/d9/d29/d11c 0 2024-04-20T21:32:15.371 INFO:tasks.workunit.client.0.smithi062.stdout:3/789: creat d0/d1/d10/d3a/f107 x:0 0 0 2024-04-20T21:32:15.372 INFO:tasks.workunit.client.1.smithi189.stdout:8/679: write da/de/f10 [1325462,98523] 0 2024-04-20T21:32:15.372 INFO:tasks.workunit.client.1.smithi189.stdout:2/528: rename d1/d17/d2f/f31 to d1/d11/f9d 0 2024-04-20T21:32:15.373 INFO:tasks.workunit.client.0.smithi062.stdout:7/829: fdatasync d7/d26/f47 0 2024-04-20T21:32:15.373 INFO:tasks.workunit.client.0.smithi062.stdout:1/920: creat dc/d15/d75/d94/d6f/dd2/de3/d121/d14d/d14e/f152 x:0 0 0 2024-04-20T21:32:15.374 INFO:tasks.workunit.client.1.smithi189.stdout:7/623: dread d5/f43 [0,4194304] 0 2024-04-20T21:32:15.376 INFO:tasks.workunit.client.0.smithi062.stdout:0/814: readlink d7/d1c/d15/d17/d40/d3c/lfc 0 2024-04-20T21:32:15.377 INFO:tasks.workunit.client.1.smithi189.stdout:0/675: getdents d3/df/d28/d2b 0 2024-04-20T21:32:15.378 INFO:tasks.workunit.client.0.smithi062.stdout:6/715: dwrite d0/d1a/f29 [0,4194304] 0 2024-04-20T21:32:15.381 INFO:tasks.workunit.client.0.smithi062.stdout:0/815: write d7/d20/d62/d6f/f105 [427571,83790] 0 2024-04-20T21:32:15.381 INFO:tasks.workunit.client.1.smithi189.stdout:3/574: getdents d2/d3/d2c/d30/d3f/d9a/db0 0 2024-04-20T21:32:15.382 INFO:tasks.workunit.client.1.smithi189.stdout:8/680: chown da/d1f/d2a/fdf 111959 1 2024-04-20T21:32:15.382 INFO:tasks.workunit.client.1.smithi189.stdout:0/676: chown d3/d8c/f8d 1995620136 1 2024-04-20T21:32:15.383 INFO:tasks.workunit.client.0.smithi062.stdout:3/790: stat d0/d1/cad 0 2024-04-20T21:32:15.384 INFO:tasks.workunit.client.0.smithi062.stdout:6/716: fsync d0/d52/da0/fcc 0 2024-04-20T21:32:15.385 INFO:tasks.workunit.client.1.smithi189.stdout:2/529: symlink d1/d17/d2f/d55/l9e 0 2024-04-20T21:32:15.387 INFO:tasks.workunit.client.0.smithi062.stdout:3/791: chown d0/d1/d10/d31/d41 1621698380 1 2024-04-20T21:32:15.388 INFO:tasks.workunit.client.1.smithi189.stdout:6/619: dwrite d8/d12/d4e/f50 [0,4194304] 0 2024-04-20T21:32:15.389 INFO:tasks.workunit.client.0.smithi062.stdout:7/830: symlink d7/d3e/d33/d3d/d4d/d54/dea/d80/l120 0 2024-04-20T21:32:15.392 INFO:tasks.workunit.client.1.smithi189.stdout:7/624: mkdir d5/d1d/d77/db8/dd1 0 2024-04-20T21:32:15.392 INFO:tasks.workunit.client.1.smithi189.stdout:9/543: write d9/f87 [732987,128965] 0 2024-04-20T21:32:15.395 INFO:tasks.workunit.client.0.smithi062.stdout:1/921: creat dc/d15/d75/d96/dd4/d3a/da2/f153 x:0 0 0 2024-04-20T21:32:15.396 INFO:tasks.workunit.client.1.smithi189.stdout:1/552: truncate d8/d1e/f66 372431 0 2024-04-20T21:32:15.396 INFO:tasks.workunit.client.0.smithi062.stdout:4/914: sync 2024-04-20T21:32:15.396 INFO:tasks.workunit.client.0.smithi062.stdout:9/971: sync 2024-04-20T21:32:15.396 INFO:tasks.workunit.client.0.smithi062.stdout:5/844: sync 2024-04-20T21:32:15.397 INFO:tasks.workunit.client.1.smithi189.stdout:2/530: write d1/d11/d58/f64 [132540,22625] 0 2024-04-20T21:32:15.400 INFO:tasks.workunit.client.0.smithi062.stdout:0/816: mkdir d7/d1c/d15/d17/d37/d109 0 2024-04-20T21:32:15.401 INFO:tasks.workunit.client.0.smithi062.stdout:6/717: rmdir d0/d9/d11/d22/dd5/d67/d75/dd2 39 2024-04-20T21:32:15.402 INFO:tasks.workunit.client.0.smithi062.stdout:9/972: write d16/d2c/d62/fbc [104980,61248] 0 2024-04-20T21:32:15.405 INFO:tasks.workunit.client.1.smithi189.stdout:4/548: write d2/dd/d11/d14/f79 [1139843,44888] 0 2024-04-20T21:32:15.405 INFO:tasks.workunit.client.0.smithi062.stdout:4/915: truncate da/dd/d15/d6d/f129 476198 0 2024-04-20T21:32:15.407 INFO:tasks.workunit.client.0.smithi062.stdout:5/845: unlink d8/d1e/l1f 0 2024-04-20T21:32:15.412 INFO:tasks.workunit.client.1.smithi189.stdout:0/677: link d3/df/d5d/d64/lbc d3/d26/le5 0 2024-04-20T21:32:15.413 INFO:tasks.workunit.client.1.smithi189.stdout:7/625: rename d5/d21/d4a/d52/d98/l89 to d5/d21/d35/ld2 0 2024-04-20T21:32:15.415 INFO:tasks.workunit.client.0.smithi062.stdout:7/831: dwrite f4 [0,4194304] 0 2024-04-20T21:32:15.419 INFO:tasks.workunit.client.1.smithi189.stdout:1/553: creat d8/d1e/d54/d93/fbd x:0 0 0 2024-04-20T21:32:15.421 INFO:tasks.workunit.client.0.smithi062.stdout:7/832: chown d7/d26/c50 4 1 2024-04-20T21:32:15.421 INFO:tasks.workunit.client.0.smithi062.stdout:6/718: rmdir d0/d4f 39 2024-04-20T21:32:15.423 INFO:tasks.workunit.client.0.smithi062.stdout:2/886: getdents d7/da/d15/d2f/d97 0 2024-04-20T21:32:15.426 INFO:tasks.workunit.client.1.smithi189.stdout:3/575: dwrite d2/d3/d25/f95 [0,4194304] 0 2024-04-20T21:32:15.430 INFO:tasks.workunit.client.0.smithi062.stdout:1/922: creat dc/d15/d75/d96/dd4/d3a/da2/de4/d84/dbe/d10d/f154 x:0 0 0 2024-04-20T21:32:15.430 INFO:tasks.workunit.client.1.smithi189.stdout:0/678: write d3/df/d30/d45/fc4 [904169,39140] 0 2024-04-20T21:32:15.433 INFO:tasks.workunit.client.0.smithi062.stdout:9/973: dwrite d16/d2f/d5b/fe0 [0,4194304] 0 2024-04-20T21:32:15.435 INFO:tasks.workunit.client.1.smithi189.stdout:2/531: rename d1/d17/c18 to d1/d17/d2f/d74/c9f 0 2024-04-20T21:32:15.435 INFO:tasks.workunit.client.0.smithi062.stdout:2/887: truncate d7/da/d15/d2f/d93/fd5 96123 0 2024-04-20T21:32:15.438 INFO:tasks.workunit.client.0.smithi062.stdout:5/846: unlink d8/d10/d3e/ff6 0 2024-04-20T21:32:15.439 INFO:tasks.workunit.client.1.smithi189.stdout:0/679: stat d3/df/d5d/cd5 0 2024-04-20T21:32:15.443 INFO:tasks.workunit.client.1.smithi189.stdout:2/532: fsync d1/d11/d2d/f50 0 2024-04-20T21:32:15.445 INFO:tasks.workunit.client.1.smithi189.stdout:3/576: read d2/d3/d2c/d30/f68 [75609,56021] 0 2024-04-20T21:32:15.448 INFO:tasks.workunit.client.0.smithi062.stdout:7/833: truncate d7/d3e/d33/d3d/d5e/f90 521377 0 2024-04-20T21:32:15.453 INFO:tasks.workunit.client.1.smithi189.stdout:8/681: dwrite da/de/d1e/d21/d87/fd1 [0,4194304] 0 2024-04-20T21:32:15.457 INFO:tasks.workunit.client.0.smithi062.stdout:0/817: fdatasync d7/d1c/d15/d17/d40/d3c/dc5/f88 0 2024-04-20T21:32:15.460 INFO:tasks.workunit.client.1.smithi189.stdout:6/620: dwrite d8/d2b/d49/f60 [0,4194304] 0 2024-04-20T21:32:15.460 INFO:tasks.workunit.client.1.smithi189.stdout:5/549: dwrite d7/d9/d30/d5a/f64 [0,4194304] 0 2024-04-20T21:32:15.460 INFO:tasks.workunit.client.0.smithi062.stdout:0/818: fsync d7/d1c/d15/f100 0 2024-04-20T21:32:15.461 INFO:tasks.workunit.client.0.smithi062.stdout:4/916: unlink da/dd/d15/d50/d64/d57/d5b/f6f 0 2024-04-20T21:32:15.461 INFO:tasks.workunit.client.1.smithi189.stdout:9/544: rename d9/d12/d15/l79 to d9/dc/d17/d95/da0/lad 0 2024-04-20T21:32:15.463 INFO:tasks.workunit.client.1.smithi189.stdout:7/626: dread d5/d46/f8f [0,4194304] 0 2024-04-20T21:32:15.465 INFO:tasks.workunit.client.1.smithi189.stdout:3/577: dread - d2/d3/d2c/d31/d90/d92/fa2 zero size 2024-04-20T21:32:15.465 INFO:tasks.workunit.client.1.smithi189.stdout:0/680: rmdir d3/df/d30/d37/dbd 39 2024-04-20T21:32:15.467 INFO:tasks.workunit.client.0.smithi062.stdout:9/974: symlink d16/d2c/d62/dd1/df8/l144 0 2024-04-20T21:32:15.469 INFO:tasks.workunit.client.0.smithi062.stdout:9/975: readlink d16/d2c/d42/l45 0 2024-04-20T21:32:15.469 INFO:tasks.workunit.client.0.smithi062.stdout:8/870: write d7/d25/d76/f10f [340239,111370] 0 2024-04-20T21:32:15.470 INFO:tasks.workunit.client.0.smithi062.stdout:9/976: readlink d16/d2c/d42/l45 0 2024-04-20T21:32:15.474 INFO:tasks.workunit.client.1.smithi189.stdout:5/550: dread - d7/d9/d28/fb0 zero size 2024-04-20T21:32:15.475 INFO:tasks.workunit.client.1.smithi189.stdout:9/545: mkdir d9/d12/d62/dae 0 2024-04-20T21:32:15.475 INFO:tasks.workunit.client.1.smithi189.stdout:8/682: mknod da/df/d3c/d68/d8b/d8d/cf3 0 2024-04-20T21:32:15.480 INFO:tasks.workunit.client.0.smithi062.stdout:3/792: getdents d0/d1/d3c/d43/d72/df2 0 2024-04-20T21:32:15.482 INFO:tasks.workunit.client.1.smithi189.stdout:7/627: chown d5/d21/d25/l29 9 1 2024-04-20T21:32:15.483 INFO:tasks.workunit.client.0.smithi062.stdout:6/719: dread d0/d9/d11/fbc [0,4194304] 0 2024-04-20T21:32:15.488 INFO:tasks.workunit.client.1.smithi189.stdout:3/578: symlink d2/d3/d83/lce 0 2024-04-20T21:32:15.488 INFO:tasks.workunit.client.1.smithi189.stdout:5/551: fsync d7/d9/d28/fa0 0 2024-04-20T21:32:15.490 INFO:tasks.workunit.client.1.smithi189.stdout:9/546: rename d9/dc/d17/d7e/f9a to d9/dc/d69/faf 0 2024-04-20T21:32:15.496 INFO:tasks.workunit.client.1.smithi189.stdout:5/552: readlink d7/d14/l69 0 2024-04-20T21:32:15.497 INFO:tasks.workunit.client.1.smithi189.stdout:8/683: truncate da/df/d13/dd6/d59/fb9 1114787 0 2024-04-20T21:32:15.499 INFO:tasks.workunit.client.1.smithi189.stdout:8/684: stat l2 0 2024-04-20T21:32:15.502 INFO:tasks.workunit.client.1.smithi189.stdout:4/549: dread d2/dd/fe [0,4194304] 0 2024-04-20T21:32:15.505 INFO:tasks.workunit.client.1.smithi189.stdout:5/553: fdatasync d7/d9/d28/f6f 0 2024-04-20T21:32:15.506 INFO:tasks.workunit.client.0.smithi062.stdout:9/977: mknod d16/d2c/d62/dd1/df8/c145 0 2024-04-20T21:32:15.506 INFO:tasks.workunit.client.1.smithi189.stdout:3/579: creat d2/d3/d2c/d31/d74/db7/fcf x:0 0 0 2024-04-20T21:32:15.509 INFO:tasks.workunit.client.0.smithi062.stdout:3/793: fsync d0/d11/f1a 0 2024-04-20T21:32:15.509 INFO:tasks.workunit.client.1.smithi189.stdout:4/550: dread - d2/d20/faf zero size 2024-04-20T21:32:15.509 INFO:tasks.workunit.client.1.smithi189.stdout:1/554: dwrite d8/d21/f22 [0,4194304] 0 2024-04-20T21:32:15.510 INFO:tasks.workunit.client.1.smithi189.stdout:8/685: write da/d1f/fa1 [362765,54436] 0 2024-04-20T21:32:15.512 INFO:tasks.workunit.client.0.smithi062.stdout:7/834: mknod d7/c121 0 2024-04-20T21:32:15.513 INFO:tasks.workunit.client.1.smithi189.stdout:0/681: dwrite d3/df/d28/d2b/d2c/f9e [0,4194304] 0 2024-04-20T21:32:15.515 INFO:tasks.workunit.client.1.smithi189.stdout:4/551: readlink d2/d64/l70 0 2024-04-20T21:32:15.517 INFO:tasks.workunit.client.1.smithi189.stdout:6/621: truncate d8/d10/fc6 3714861 0 2024-04-20T21:32:15.520 INFO:tasks.workunit.client.1.smithi189.stdout:3/580: creat d2/d3/d2c/d31/d74/d7a/fd0 x:0 0 0 2024-04-20T21:32:15.521 INFO:tasks.workunit.client.1.smithi189.stdout:7/628: truncate d5/d21/d25/f6c 2129849 0 2024-04-20T21:32:15.522 INFO:tasks.workunit.client.1.smithi189.stdout:4/552: chown d2/fbe 58 1 2024-04-20T21:32:15.526 INFO:tasks.workunit.client.1.smithi189.stdout:8/686: mknod da/df/d3c/d68/d8b/d8d/cf4 0 2024-04-20T21:32:15.528 INFO:tasks.workunit.client.0.smithi062.stdout:4/917: creat da/d4f/d113/f13e x:0 0 0 2024-04-20T21:32:15.529 INFO:tasks.workunit.client.0.smithi062.stdout:1/923: write dc/d15/d75/d94/f69 [408580,66868] 0 2024-04-20T21:32:15.530 INFO:tasks.workunit.client.1.smithi189.stdout:1/555: mkdir d8/df/d3a/d49/d8c/da2/dbe 0 2024-04-20T21:32:15.531 INFO:tasks.workunit.client.0.smithi062.stdout:4/918: write da/dd/f19 [1576197,49578] 0 2024-04-20T21:32:15.531 INFO:tasks.workunit.client.1.smithi189.stdout:7/629: dread d5/d21/d4a/d52/f7c [0,4194304] 0 2024-04-20T21:32:15.533 INFO:tasks.workunit.client.0.smithi062.stdout:5/847: creat d8/f119 x:0 0 0 2024-04-20T21:32:15.536 INFO:tasks.workunit.client.1.smithi189.stdout:6/622: rename d8/l40 to d8/da/da7/lcb 0 2024-04-20T21:32:15.538 INFO:tasks.workunit.client.1.smithi189.stdout:1/556: mkdir d8/d21/dbf 0 2024-04-20T21:32:15.538 INFO:tasks.workunit.client.0.smithi062.stdout:3/794: mknod d0/d1/d10/d31/d41/c108 0 2024-04-20T21:32:15.539 INFO:tasks.workunit.client.0.smithi062.stdout:5/848: dread d8/d38/d7a/d4c/d7d/d9d/dac/f39 [0,4194304] 0 2024-04-20T21:32:15.539 INFO:tasks.workunit.client.1.smithi189.stdout:4/553: truncate d2/dd/d11/d14/d69/f9e 730156 0 2024-04-20T21:32:15.539 INFO:tasks.workunit.client.1.smithi189.stdout:2/533: dread d1/d11/f15 [0,4194304] 0 2024-04-20T21:32:15.542 INFO:tasks.workunit.client.1.smithi189.stdout:6/623: dread - d8/d12/d99/fb7 zero size 2024-04-20T21:32:15.543 INFO:tasks.workunit.client.1.smithi189.stdout:2/534: chown d1/d17/d78 256293 1 2024-04-20T21:32:15.543 INFO:tasks.workunit.client.1.smithi189.stdout:1/557: chown f2 139010241 1 2024-04-20T21:32:15.544 INFO:tasks.workunit.client.0.smithi062.stdout:8/871: write d7/d25/f5a [1219289,111664] 0 2024-04-20T21:32:15.545 INFO:tasks.workunit.client.0.smithi062.stdout:7/835: creat d7/d3e/d33/d3d/dad/f122 x:0 0 0 2024-04-20T21:32:15.545 INFO:tasks.workunit.client.1.smithi189.stdout:0/682: dwrite d3/fe [4194304,4194304] 0 2024-04-20T21:32:15.547 INFO:tasks.workunit.client.0.smithi062.stdout:7/836: dread - d7/d3e/d33/d3d/d4d/f11b zero size 2024-04-20T21:32:15.550 INFO:tasks.workunit.client.1.smithi189.stdout:8/687: rename da/df/d3c/d50/d9e to da/df/d3c/d68/df5 0 2024-04-20T21:32:15.550 INFO:tasks.workunit.client.1.smithi189.stdout:4/554: symlink d2/dd/d37/lc9 0 2024-04-20T21:32:15.550 INFO:tasks.workunit.client.1.smithi189.stdout:7/630: fdatasync d5/d21/d4a/d52/fa7 0 2024-04-20T21:32:15.551 INFO:tasks.workunit.client.0.smithi062.stdout:7/837: write d7/d3e/d33/d3d/d4d/d54/d4f/de9/df1/d105/f10f [4474,1546] 0 2024-04-20T21:32:15.559 INFO:tasks.workunit.client.1.smithi189.stdout:6/624: write d8/d12/d30/d47/f8a [866277,22972] 0 2024-04-20T21:32:15.562 INFO:tasks.workunit.client.0.smithi062.stdout:6/720: link d0/d9/f13 d0/d1a/d7d/ff3 0 2024-04-20T21:32:15.563 INFO:tasks.workunit.client.0.smithi062.stdout:8/872: fdatasync d7/d25/d3b/d58/d8b/d24/ff3 0 2024-04-20T21:32:15.563 INFO:tasks.workunit.client.1.smithi189.stdout:7/631: chown d5/db4/lca 1060352 1 2024-04-20T21:32:15.563 INFO:tasks.workunit.client.1.smithi189.stdout:2/535: dread - d1/d17/d2f/d4a/f7b zero size 2024-04-20T21:32:15.564 INFO:tasks.workunit.client.0.smithi062.stdout:0/819: getdents d7/d1c/d15/d17 0 2024-04-20T21:32:15.565 INFO:tasks.workunit.client.0.smithi062.stdout:7/838: fsync d7/d26/da8/fd0 0 2024-04-20T21:32:15.565 INFO:tasks.workunit.client.1.smithi189.stdout:8/688: creat da/de/d1e/d21/d87/ff6 x:0 0 0 2024-04-20T21:32:15.577 INFO:tasks.workunit.client.1.smithi189.stdout:3/581: dwrite d2/d3/d2c/f6d [0,4194304] 0 2024-04-20T21:32:15.577 INFO:tasks.workunit.client.1.smithi189.stdout:2/536: creat d1/d11/d58/fa0 x:0 0 0 2024-04-20T21:32:15.579 INFO:tasks.workunit.client.1.smithi189.stdout:8/689: write da/df/d20/d2c/d9a/dce/fda [817884,74410] 0 2024-04-20T21:32:15.580 INFO:tasks.workunit.client.1.smithi189.stdout:6/625: rename d8/d12/d30/d47/f4c to d8/d12/d30/d47/da0/fcc 0 2024-04-20T21:32:15.580 INFO:tasks.workunit.client.1.smithi189.stdout:5/554: dread d7/d14/f49 [0,4194304] 0 2024-04-20T21:32:15.580 INFO:tasks.workunit.client.1.smithi189.stdout:2/537: write d1/f8c [1027246,4210] 0 2024-04-20T21:32:15.580 INFO:tasks.workunit.client.1.smithi189.stdout:3/582: chown d2/d8/f22 7 1 2024-04-20T21:32:15.583 INFO:tasks.workunit.client.0.smithi062.stdout:5/849: mkdir d8/d38/d7a/d4c/d11a 0 2024-04-20T21:32:15.591 INFO:tasks.workunit.client.1.smithi189.stdout:8/690: creat da/df/d20/d2c/d9a/ff7 x:0 0 0 2024-04-20T21:32:15.592 INFO:tasks.workunit.client.1.smithi189.stdout:3/583: dread - d2/d3/d2c/d30/d3f/fbf zero size 2024-04-20T21:32:15.592 INFO:tasks.workunit.client.1.smithi189.stdout:6/626: read d8/d2b/f75 [4389832,90187] 0 2024-04-20T21:32:15.592 INFO:tasks.workunit.client.1.smithi189.stdout:2/538: write d1/d17/f7f [3926836,36741] 0 2024-04-20T21:32:15.593 INFO:tasks.workunit.client.0.smithi062.stdout:8/873: creat d7/d25/d3b/d58/d8b/d96/f11d x:0 0 0 2024-04-20T21:32:15.594 INFO:tasks.workunit.client.0.smithi062.stdout:3/795: write d0/f4c [1981258,86476] 0 2024-04-20T21:32:15.595 INFO:tasks.workunit.client.0.smithi062.stdout:7/839: mknod d7/d63/c123 0 2024-04-20T21:32:15.596 INFO:tasks.workunit.client.1.smithi189.stdout:1/558: dwrite d8/d21/d2c/d36/f52 [0,4194304] 0 2024-04-20T21:32:15.598 INFO:tasks.workunit.client.0.smithi062.stdout:1/924: write dc/d15/d75/d94/d6f/dd2/fdf [1483957,79219] 0 2024-04-20T21:32:15.600 INFO:tasks.workunit.client.0.smithi062.stdout:0/820: truncate d7/d1c/d15/d17/d37/d95/fd6 767666 0 2024-04-20T21:32:15.601 INFO:tasks.workunit.client.0.smithi062.stdout:4/919: dwrite da/dd/d15/fa6 [0,4194304] 0 2024-04-20T21:32:15.602 INFO:tasks.workunit.client.0.smithi062.stdout:7/840: chown d7/d3e/d33/d3d/d4d/d54/d4f/ce5 2 1 2024-04-20T21:32:15.602 INFO:tasks.workunit.client.0.smithi062.stdout:9/978: link d16/d3d/l63 d16/d2c/d62/dd1/l146 0 2024-04-20T21:32:15.604 INFO:tasks.workunit.client.1.smithi189.stdout:8/691: dread - da/de/d1e/d21/d87/ff6 zero size 2024-04-20T21:32:15.605 INFO:tasks.workunit.client.1.smithi189.stdout:4/555: dwrite d2/dd/d11/f58 [0,4194304] 0 2024-04-20T21:32:15.611 INFO:tasks.workunit.client.1.smithi189.stdout:0/683: dwrite d3/df/d5d/f7c [0,4194304] 0 2024-04-20T21:32:15.617 INFO:tasks.workunit.client.0.smithi062.stdout:3/796: dread d0/d1b/f2b [4194304,4194304] 0 2024-04-20T21:32:15.617 INFO:tasks.workunit.client.1.smithi189.stdout:5/555: getdents d7/d14 0 2024-04-20T21:32:15.620 INFO:tasks.workunit.client.1.smithi189.stdout:6/627: truncate d8/d12/d30/d47/fc9 471480 0 2024-04-20T21:32:15.623 INFO:tasks.workunit.client.1.smithi189.stdout:5/556: chown d7/l2c 1971 1 2024-04-20T21:32:15.623 INFO:tasks.workunit.client.1.smithi189.stdout:0/684: dread - d3/df/d28/d2b/d2c/fd0 zero size 2024-04-20T21:32:15.623 INFO:tasks.workunit.client.1.smithi189.stdout:7/632: dwrite d5/f57 [0,4194304] 0 2024-04-20T21:32:15.628 INFO:tasks.workunit.client.0.smithi062.stdout:2/888: sync 2024-04-20T21:32:15.629 INFO:tasks.workunit.client.1.smithi189.stdout:0/685: write d3/df/f3e [1903693,128406] 0 2024-04-20T21:32:15.629 INFO:tasks.workunit.client.1.smithi189.stdout:3/584: write d2/d1c/d5c/f87 [762270,68625] 0 2024-04-20T21:32:15.632 INFO:tasks.workunit.client.1.smithi189.stdout:4/556: mknod d2/dd/d11/d14/d69/d6a/d7a/db7/cca 0 2024-04-20T21:32:15.633 INFO:tasks.workunit.client.1.smithi189.stdout:7/633: chown d5/d21/d35/l3d 490141 1 2024-04-20T21:32:15.639 INFO:tasks.workunit.client.1.smithi189.stdout:6/628: write d8/f63 [2787936,69892] 0 2024-04-20T21:32:15.641 INFO:tasks.workunit.client.1.smithi189.stdout:5/557: creat d7/d9/d30/d55/d61/fb2 x:0 0 0 2024-04-20T21:32:15.641 INFO:tasks.workunit.client.1.smithi189.stdout:1/559: dwrite d8/d21/d2c/fa3 [0,4194304] 0 2024-04-20T21:32:15.641 INFO:tasks.workunit.client.1.smithi189.stdout:4/557: mknod d2/d22/da9/ccb 0 2024-04-20T21:32:15.642 INFO:tasks.workunit.client.0.smithi062.stdout:5/850: mknod d8/d38/d7a/d4c/d7d/d9d/dac/d66/de4/c11b 0 2024-04-20T21:32:15.643 INFO:tasks.workunit.client.1.smithi189.stdout:7/634: write d5/d46/f8b [5072967,69120] 0 2024-04-20T21:32:15.644 INFO:tasks.workunit.client.1.smithi189.stdout:0/686: creat d3/df/d4c/dd4/fe6 x:0 0 0 2024-04-20T21:32:15.644 INFO:tasks.workunit.client.0.smithi062.stdout:8/874: creat d7/d25/d70/f11e x:0 0 0 2024-04-20T21:32:15.646 INFO:tasks.workunit.client.1.smithi189.stdout:6/629: readlink d8/d2b/d5c/l9a 0 2024-04-20T21:32:15.647 INFO:tasks.workunit.client.1.smithi189.stdout:5/558: dread - d7/d9/d30/d55/d61/daa/fad zero size 2024-04-20T21:32:15.647 INFO:tasks.workunit.client.1.smithi189.stdout:0/687: stat d3/d8c/c8f 0 2024-04-20T21:32:15.654 INFO:tasks.workunit.client.0.smithi062.stdout:4/920: unlink da/cb0 0 2024-04-20T21:32:15.654 INFO:tasks.workunit.client.0.smithi062.stdout:0/821: rmdir d7/d20 39 2024-04-20T21:32:15.658 INFO:tasks.workunit.client.1.smithi189.stdout:0/688: read d3/df/d28/d2b/f49 [767383,2633] 0 2024-04-20T21:32:15.662 INFO:tasks.workunit.client.0.smithi062.stdout:3/797: dread d0/f98 [0,4194304] 0 2024-04-20T21:32:15.663 INFO:tasks.workunit.client.0.smithi062.stdout:7/841: rmdir d7/d3e/d33 39 2024-04-20T21:32:15.664 INFO:tasks.workunit.client.1.smithi189.stdout:7/635: creat d5/d1d/d77/fd3 x:0 0 0 2024-04-20T21:32:15.666 INFO:tasks.workunit.client.0.smithi062.stdout:1/925: read dc/d15/f16 [2987014,87757] 0 2024-04-20T21:32:15.668 INFO:tasks.workunit.client.1.smithi189.stdout:0/689: write d3/df/d5d/d64/fac [2682037,73809] 0 2024-04-20T21:32:15.671 INFO:tasks.workunit.client.0.smithi062.stdout:8/875: dwrite d7/d25/d4f/fdc [0,4194304] 0 2024-04-20T21:32:15.674 INFO:tasks.workunit.client.1.smithi189.stdout:3/585: dwrite d2/d3/f3e [0,4194304] 0 2024-04-20T21:32:15.677 INFO:tasks.workunit.client.1.smithi189.stdout:0/690: write d3/df/d28/d2b/d2c/f83 [1982914,49294] 0 2024-04-20T21:32:15.679 INFO:tasks.workunit.client.1.smithi189.stdout:6/630: link d8/d2b/d49/d54/d67/l94 d8/da/da7/lcd 0 2024-04-20T21:32:15.681 INFO:tasks.workunit.client.0.smithi062.stdout:4/921: rename da/dd/df/d27/d7a/l8f to da/dd/d15/daf/dee/l13f 0 2024-04-20T21:32:15.684 INFO:tasks.workunit.client.1.smithi189.stdout:1/560: fdatasync d8/d1e/d28/f90 0 2024-04-20T21:32:15.684 INFO:tasks.workunit.client.1.smithi189.stdout:3/586: fdatasync d2/f6 0 2024-04-20T21:32:15.685 INFO:tasks.workunit.client.0.smithi062.stdout:3/798: dread d0/d11/fa2 [0,4194304] 0 2024-04-20T21:32:15.687 INFO:tasks.workunit.client.1.smithi189.stdout:5/559: truncate d7/d9/d28/f9f 149303 0 2024-04-20T21:32:15.688 INFO:tasks.workunit.client.0.smithi062.stdout:7/842: symlink d7/d63/l124 0 2024-04-20T21:32:15.688 INFO:tasks.workunit.client.1.smithi189.stdout:5/560: readlink l0 0 2024-04-20T21:32:15.691 INFO:tasks.workunit.client.0.smithi062.stdout:7/843: chown d7/f2a 25371 1 2024-04-20T21:32:15.692 INFO:tasks.workunit.client.0.smithi062.stdout:1/926: creat dc/d15/df6/f155 x:0 0 0 2024-04-20T21:32:15.692 INFO:tasks.workunit.client.0.smithi062.stdout:5/851: dread d8/d10/d3e/fff [0,4194304] 0 2024-04-20T21:32:15.694 INFO:tasks.workunit.client.1.smithi189.stdout:7/636: dread d5/d46/fa9 [0,4194304] 0 2024-04-20T21:32:15.694 INFO:tasks.workunit.client.1.smithi189.stdout:5/561: unlink d7/d35/d48/f97 0 2024-04-20T21:32:15.696 INFO:tasks.workunit.client.1.smithi189.stdout:6/631: dread d8/d2b/f39 [0,4194304] 0 2024-04-20T21:32:15.696 INFO:tasks.workunit.client.0.smithi062.stdout:0/822: creat d7/d20/d62/de3/f10a x:0 0 0 2024-04-20T21:32:15.697 INFO:tasks.workunit.client.1.smithi189.stdout:2/539: dread d1/d11/d58/d6b/f6c [0,4194304] 0 2024-04-20T21:32:15.699 INFO:tasks.workunit.client.1.smithi189.stdout:5/562: write d7/d9/d30/d55/d61/daa/fad [35930,56172] 0 2024-04-20T21:32:15.700 INFO:tasks.workunit.client.1.smithi189.stdout:1/561: link d8/df/d50/f8d d8/df/d3a/d49/d8c/da2/fc0 0 2024-04-20T21:32:15.702 INFO:tasks.workunit.client.0.smithi062.stdout:9/979: getdents d16/d3e 0 2024-04-20T21:32:15.703 INFO:tasks.workunit.client.1.smithi189.stdout:2/540: stat d1/d17/d2f/d74/c9f 0 2024-04-20T21:32:15.706 INFO:tasks.workunit.client.1.smithi189.stdout:2/541: chown d1/d11/d2d/f63 385 1 2024-04-20T21:32:15.708 INFO:tasks.workunit.client.1.smithi189.stdout:5/563: mkdir d7/d9/d30/d55/d61/db3 0 2024-04-20T21:32:15.709 INFO:tasks.workunit.client.0.smithi062.stdout:6/721: truncate d0/d9/d11/f8d 1476432 0 2024-04-20T21:32:15.711 INFO:tasks.workunit.client.1.smithi189.stdout:1/562: mkdir d8/df/d3a/d49/d8c/da2/dbe/dc1 0 2024-04-20T21:32:15.711 INFO:tasks.workunit.client.1.smithi189.stdout:2/542: fdatasync d1/d11/d52/d8b/f8d 0 2024-04-20T21:32:15.713 INFO:tasks.workunit.client.0.smithi062.stdout:3/799: unlink d0/d1c/d63/fef 0 2024-04-20T21:32:15.714 INFO:tasks.workunit.client.1.smithi189.stdout:5/564: creat d7/d9/d51/fb4 x:0 0 0 2024-04-20T21:32:15.714 INFO:tasks.workunit.client.1.smithi189.stdout:4/558: dread d2/d22/f30 [0,4194304] 0 2024-04-20T21:32:15.716 INFO:tasks.workunit.client.1.smithi189.stdout:5/565: dread - d7/d35/d48/f6a zero size 2024-04-20T21:32:15.716 INFO:tasks.workunit.client.0.smithi062.stdout:2/889: write d7/da/d15/d2f/d38/f4e [3109127,113832] 0 2024-04-20T21:32:15.718 INFO:tasks.workunit.client.1.smithi189.stdout:2/543: creat d1/d11/d52/d8b/fa1 x:0 0 0 2024-04-20T21:32:15.719 INFO:tasks.workunit.client.1.smithi189.stdout:1/563: write d8/d1e/f6e [1273545,113984] 0 2024-04-20T21:32:15.724 INFO:tasks.workunit.client.1.smithi189.stdout:4/559: mkdir d2/d22/d87/dcc 0 2024-04-20T21:32:15.724 INFO:tasks.workunit.client.1.smithi189.stdout:5/566: fdatasync d7/d9/d30/d55/d61/daa/fae 0 2024-04-20T21:32:15.725 INFO:tasks.workunit.client.1.smithi189.stdout:3/587: truncate d2/d3/d2c/d30/d3f/d9a/db0/fb8 3309917 0 2024-04-20T21:32:15.726 INFO:tasks.workunit.client.0.smithi062.stdout:0/823: mkdir d7/d1c/d15/d53/d10b 0 2024-04-20T21:32:15.726 INFO:tasks.workunit.client.1.smithi189.stdout:7/637: truncate d5/d21/d4a/d52/d75/fab 513928 0 2024-04-20T21:32:15.727 INFO:tasks.workunit.client.1.smithi189.stdout:0/691: dread d3/df/d28/f6a [0,4194304] 0 2024-04-20T21:32:15.730 INFO:tasks.workunit.client.1.smithi189.stdout:6/632: dwrite d8/da/dd/f71 [0,4194304] 0 2024-04-20T21:32:15.731 INFO:tasks.workunit.client.1.smithi189.stdout:5/567: readlink d7/d9/l4b 0 2024-04-20T21:32:15.732 INFO:tasks.workunit.client.0.smithi062.stdout:4/922: fsync da/d56/d71/dbd/fc8 0 2024-04-20T21:32:15.733 INFO:tasks.workunit.client.0.smithi062.stdout:8/876: write d7/d25/d3b/d58/d8b/f1c [1166843,95651] 0 2024-04-20T21:32:15.733 INFO:tasks.workunit.client.1.smithi189.stdout:0/692: dread - d3/df/f76 zero size 2024-04-20T21:32:15.735 INFO:tasks.workunit.client.0.smithi062.stdout:4/923: chown da/dd/df/d27/d7a/d91/f9f 967 1 2024-04-20T21:32:15.737 INFO:tasks.workunit.client.0.smithi062.stdout:3/800: fsync d0/d1/d3c/d43/fd4 0 2024-04-20T21:32:15.737 INFO:tasks.workunit.client.0.smithi062.stdout:4/924: fdatasync da/dd/d15/d6d/f129 0 2024-04-20T21:32:15.739 INFO:tasks.workunit.client.0.smithi062.stdout:4/925: stat da/dd/f4d 0 2024-04-20T21:32:15.740 INFO:tasks.workunit.client.0.smithi062.stdout:2/890: symlink d7/da/d15/d1b/df1/l120 0 2024-04-20T21:32:15.741 INFO:tasks.workunit.client.0.smithi062.stdout:0/824: rmdir d7/d1c/d15/d17/d40/d3c 39 2024-04-20T21:32:15.742 INFO:tasks.workunit.client.1.smithi189.stdout:5/568: write d7/d9/d30/d55/d61/f8d [676985,80478] 0 2024-04-20T21:32:15.743 INFO:tasks.workunit.client.1.smithi189.stdout:0/693: chown d3/df/d30/d45/d90 7444 1 2024-04-20T21:32:15.744 INFO:tasks.workunit.client.1.smithi189.stdout:7/638: dread d5/d21/d4a/d52/f7c [0,4194304] 0 2024-04-20T21:32:15.745 INFO:tasks.workunit.client.0.smithi062.stdout:9/980: unlink d16/l25 0 2024-04-20T21:32:15.747 INFO:tasks.workunit.client.1.smithi189.stdout:6/633: creat d8/d2b/d49/d54/d67/fce x:0 0 0 2024-04-20T21:32:15.747 INFO:tasks.workunit.client.1.smithi189.stdout:3/588: rename d2/d3/d2c/d31/d4c/lc2 to d2/d3/d2c/d30/d3f/ld1 0 2024-04-20T21:32:15.748 INFO:tasks.workunit.client.1.smithi189.stdout:2/544: write d1/d17/d2f/d74/f9b [1149694,101109] 0 2024-04-20T21:32:15.749 INFO:tasks.workunit.client.1.smithi189.stdout:1/564: truncate d8/d1e/f3c 97431 0 2024-04-20T21:32:15.753 INFO:tasks.workunit.client.0.smithi062.stdout:6/722: mkdir d0/d9/d11/d22/dd5/d67/d75/df4 0 2024-04-20T21:32:15.755 INFO:tasks.workunit.client.0.smithi062.stdout:8/877: mknod d7/da9/db0/c11f 0 2024-04-20T21:32:15.757 INFO:tasks.workunit.client.0.smithi062.stdout:2/891: write d7/da/d15/d1b/df1/ff4 [1379,6062] 0 2024-04-20T21:32:15.758 INFO:tasks.workunit.client.1.smithi189.stdout:3/589: dread - d2/d3/d2c/d31/d74/db7/fcf zero size 2024-04-20T21:32:15.759 INFO:tasks.workunit.client.0.smithi062.stdout:3/801: unlink d0/d1/d10/c8f 0 2024-04-20T21:32:15.759 INFO:tasks.workunit.client.0.smithi062.stdout:5/852: rmdir d8/d38/d7a/d4c/d11a 0 2024-04-20T21:32:15.760 INFO:tasks.workunit.client.1.smithi189.stdout:5/569: getdents d7/d9/d28/da3 0 2024-04-20T21:32:15.763 INFO:tasks.workunit.client.0.smithi062.stdout:1/927: dwrite dc/d15/d75/ff9 [0,4194304] 0 2024-04-20T21:32:15.766 INFO:tasks.workunit.client.1.smithi189.stdout:4/560: dwrite d2/f8 [0,4194304] 0 2024-04-20T21:32:15.766 INFO:tasks.workunit.client.1.smithi189.stdout:5/570: dread - d7/d9/d30/d5a/d9d/fac zero size 2024-04-20T21:32:15.769 INFO:tasks.workunit.client.1.smithi189.stdout:4/561: write d2/f1a [2470815,62487] 0 2024-04-20T21:32:15.776 INFO:tasks.workunit.client.0.smithi062.stdout:3/802: fsync d0/d1c/d63/d42/fcb 0 2024-04-20T21:32:15.776 INFO:tasks.workunit.client.1.smithi189.stdout:3/590: rename d2/d3/d2c/c75 to d2/d3/d2c/d30/d3f/d9a/cd2 0 2024-04-20T21:32:15.778 INFO:tasks.workunit.client.1.smithi189.stdout:7/639: truncate d5/f43 4175494 0 2024-04-20T21:32:15.778 INFO:tasks.workunit.client.0.smithi062.stdout:0/825: unlink d7/d1c/d15/d17/d40/d3c/dc5/f88 0 2024-04-20T21:32:15.779 INFO:tasks.workunit.client.1.smithi189.stdout:5/571: unlink d7/d9/d30/d55/c25 0 2024-04-20T21:32:15.782 INFO:tasks.workunit.client.0.smithi062.stdout:1/928: chown dc/d15/d75/d96/dd4/d3a/da2/de4/d7c/caf 1554879 1 2024-04-20T21:32:15.784 INFO:tasks.workunit.client.1.smithi189.stdout:4/562: rename d2/dd/d11/d14/d69 to d2/dd/d37/dcd 0 2024-04-20T21:32:15.785 INFO:tasks.workunit.client.1.smithi189.stdout:0/694: dread d3/df/f3e [0,4194304] 0 2024-04-20T21:32:15.785 INFO:tasks.workunit.client.1.smithi189.stdout:7/640: dread d5/db4/d71/fa6 [0,4194304] 0 2024-04-20T21:32:15.785 INFO:tasks.workunit.client.1.smithi189.stdout:6/634: truncate d8/d2b/f8c 1404672 0 2024-04-20T21:32:15.786 INFO:tasks.workunit.client.0.smithi062.stdout:7/844: dread d7/d3e/d33/d3d/d5e/d72/f86 [0,4194304] 0 2024-04-20T21:32:15.787 INFO:tasks.workunit.client.1.smithi189.stdout:5/572: dread - d7/d9/d28/fb0 zero size 2024-04-20T21:32:15.789 INFO:tasks.workunit.client.0.smithi062.stdout:3/803: symlink d0/d1c/d63/dda/l109 0 2024-04-20T21:32:15.791 INFO:tasks.workunit.client.0.smithi062.stdout:5/853: rmdir d8/d10/d3e/d8a/dce 0 2024-04-20T21:32:15.792 INFO:tasks.workunit.client.1.smithi189.stdout:2/545: truncate d1/d17/d2f/d74/f9b 449505 0 2024-04-20T21:32:15.796 INFO:tasks.workunit.client.1.smithi189.stdout:7/641: write d5/d21/d35/f7e [585456,56113] 0 2024-04-20T21:32:15.797 INFO:tasks.workunit.client.0.smithi062.stdout:0/826: creat d7/d1c/d15/d17/d37/df8/f10c x:0 0 0 2024-04-20T21:32:15.798 INFO:tasks.workunit.client.1.smithi189.stdout:0/695: mknod d3/d8c/ce7 0 2024-04-20T21:32:15.799 INFO:tasks.workunit.client.0.smithi062.stdout:0/827: chown d7/d1e/c2b 32177542 1 2024-04-20T21:32:15.799 INFO:tasks.workunit.client.1.smithi189.stdout:4/563: mkdir d2/dd/d37/dcd/d6a/d7a/dce 0 2024-04-20T21:32:15.800 INFO:tasks.workunit.client.1.smithi189.stdout:6/635: mkdir d8/d10/d78/d7e/dcf 0 2024-04-20T21:32:15.801 INFO:tasks.workunit.client.1.smithi189.stdout:0/696: readlink d3/df/d5d/l5f 0 2024-04-20T21:32:15.802 INFO:tasks.workunit.client.0.smithi062.stdout:8/878: getdents d7/d25/dc5 0 2024-04-20T21:32:15.803 INFO:tasks.workunit.client.1.smithi189.stdout:5/573: mkdir d7/d9/d30/d55/d61/daa/db5 0 2024-04-20T21:32:15.807 INFO:tasks.workunit.client.0.smithi062.stdout:4/926: dread da/dd/f4d [0,4194304] 0 2024-04-20T21:32:15.807 INFO:tasks.workunit.client.1.smithi189.stdout:5/574: chown d7/d9/d30/d55/d61/daa/db5 82671 1 2024-04-20T21:32:15.808 INFO:tasks.workunit.client.1.smithi189.stdout:2/546: getdents d1/d11 0 2024-04-20T21:32:15.808 INFO:tasks.workunit.client.1.smithi189.stdout:0/697: dread - d3/d8c/fc7 zero size 2024-04-20T21:32:15.811 INFO:tasks.workunit.client.1.smithi189.stdout:6/636: symlink d8/d12/d4e/dc4/ld0 0 2024-04-20T21:32:15.813 INFO:tasks.workunit.client.0.smithi062.stdout:0/828: creat d7/d20/f10d x:0 0 0 2024-04-20T21:32:15.813 INFO:tasks.workunit.client.0.smithi062.stdout:4/927: dread da/d56/d71/d9e/ffd [0,4194304] 0 2024-04-20T21:32:15.814 INFO:tasks.workunit.client.1.smithi189.stdout:4/564: creat d2/dd/d37/dcd/d6a/d7a/db7/fcf x:0 0 0 2024-04-20T21:32:15.815 INFO:tasks.workunit.client.1.smithi189.stdout:6/637: stat d8/d2b/d49/d54/d6e/fb4 0 2024-04-20T21:32:15.816 INFO:tasks.workunit.client.0.smithi062.stdout:9/981: truncate d16/d2c/d42/da6/da9/fe9 574251 0 2024-04-20T21:32:15.819 INFO:tasks.workunit.client.1.smithi189.stdout:0/698: symlink d3/df/d28/d2b/d2c/le8 0 2024-04-20T21:32:15.819 INFO:tasks.workunit.client.1.smithi189.stdout:5/575: rename d7/d9/d31/d36/c56 to d7/d9/d30/d55/d61/daa/db5/cb6 0 2024-04-20T21:32:15.822 INFO:tasks.workunit.client.0.smithi062.stdout:2/892: truncate d7/da/d15/f26 158455 0 2024-04-20T21:32:15.823 INFO:tasks.workunit.client.1.smithi189.stdout:6/638: creat d8/d12/daf/fd1 x:0 0 0 2024-04-20T21:32:15.826 INFO:tasks.workunit.client.1.smithi189.stdout:1/565: dread d8/f24 [0,4194304] 0 2024-04-20T21:32:15.827 INFO:tasks.workunit.client.0.smithi062.stdout:3/804: rename d0/d1/d3c/d43/d72/da9/lfa to d0/d1/d10/d3a/d5e/d65/l10a 0 2024-04-20T21:32:15.828 INFO:tasks.workunit.client.1.smithi189.stdout:6/639: dread - d8/d12/d30/d47/da0/fc2 zero size 2024-04-20T21:32:15.829 INFO:tasks.workunit.client.0.smithi062.stdout:1/929: write dc/d15/d75/d96/dd4/d3a/da2/de4/fc5 [971524,17221] 0 2024-04-20T21:32:15.833 INFO:tasks.workunit.client.0.smithi062.stdout:7/845: dread d7/d3e/d33/d3d/d4d/d54/f3f [0,4194304] 0 2024-04-20T21:32:15.837 INFO:tasks.workunit.client.1.smithi189.stdout:9/547: sync 2024-04-20T21:32:15.839 INFO:tasks.workunit.client.0.smithi062.stdout:7/846: dread d7/f6c [0,4194304] 0 2024-04-20T21:32:15.840 INFO:tasks.workunit.client.0.smithi062.stdout:4/928: dread da/dd/df/d27/fc2 [0,4194304] 0 2024-04-20T21:32:15.842 INFO:tasks.workunit.client.0.smithi062.stdout:5/854: creat d8/d1e/d2c/dd1/ded/f11c x:0 0 0 2024-04-20T21:32:15.844 INFO:tasks.workunit.client.1.smithi189.stdout:2/547: write d1/d17/d37/f44 [3980064,110874] 0 2024-04-20T21:32:15.846 INFO:tasks.workunit.client.1.smithi189.stdout:7/642: dwrite d5/d21/f32 [0,4194304] 0 2024-04-20T21:32:15.847 INFO:tasks.workunit.client.1.smithi189.stdout:1/566: mkdir d8/d1e/d43/d58/dc2 0 2024-04-20T21:32:15.847 INFO:tasks.workunit.client.1.smithi189.stdout:9/548: stat d9/dc/d17/d95/da0/f6a 0 2024-04-20T21:32:15.848 INFO:tasks.workunit.client.0.smithi062.stdout:9/982: creat d16/dbe/f147 x:0 0 0 2024-04-20T21:32:15.858 INFO:tasks.workunit.client.1.smithi189.stdout:0/699: truncate d3/df/d4c/f7e 2339673 0 2024-04-20T21:32:15.858 INFO:tasks.workunit.client.1.smithi189.stdout:3/591: dread d2/d3/d25/f40 [4194304,4194304] 0 2024-04-20T21:32:15.858 INFO:tasks.workunit.client.1.smithi189.stdout:9/549: symlink d9/dc/d17/lb0 0 2024-04-20T21:32:15.858 INFO:tasks.workunit.client.1.smithi189.stdout:5/576: dread d7/d14/d18/f53 [0,4194304] 0 2024-04-20T21:32:15.858 INFO:tasks.workunit.client.1.smithi189.stdout:7/643: getdents d5/d21/d97/da3 0 2024-04-20T21:32:15.859 INFO:tasks.workunit.client.0.smithi062.stdout:4/929: creat da/dd/df/d109/f140 x:0 0 0 2024-04-20T21:32:15.859 INFO:tasks.workunit.client.1.smithi189.stdout:3/592: fdatasync d2/d3/d25/f54 0 2024-04-20T21:32:15.860 INFO:tasks.workunit.client.1.smithi189.stdout:0/700: creat d3/da8/fe9 x:0 0 0 2024-04-20T21:32:15.861 INFO:tasks.workunit.client.0.smithi062.stdout:7/847: rename d7/d3e/d33/d3d/dad/f122 to d7/d3e/d33/d3d/d4d/d83/f125 0 2024-04-20T21:32:15.862 INFO:tasks.workunit.client.1.smithi189.stdout:5/577: chown d7/l12 10877354 1 2024-04-20T21:32:15.862 INFO:tasks.workunit.client.1.smithi189.stdout:7/644: dread - d5/d46/d7d/fc9 zero size 2024-04-20T21:32:15.865 INFO:tasks.workunit.client.1.smithi189.stdout:5/578: chown d7/d35/d48/f7d 683241646 1 2024-04-20T21:32:15.871 INFO:tasks.workunit.client.0.smithi062.stdout:9/983: fsync d16/d2f/d69/fc5 0 2024-04-20T21:32:15.873 INFO:tasks.workunit.client.0.smithi062.stdout:8/879: creat d7/f120 x:0 0 0 2024-04-20T21:32:15.873 INFO:tasks.workunit.client.0.smithi062.stdout:9/984: chown d16/d3d/lbb 1 1 2024-04-20T21:32:15.874 INFO:tasks.workunit.client.0.smithi062.stdout:2/893: mkdir d7/da/d15/d2f/d38/d10f/d121 0 2024-04-20T21:32:15.874 INFO:tasks.workunit.client.1.smithi189.stdout:6/640: dwrite d8/d12/d99/fb7 [0,4194304] 0 2024-04-20T21:32:15.874 INFO:tasks.workunit.client.1.smithi189.stdout:3/593: truncate d2/f15 8761982 0 2024-04-20T21:32:15.879 INFO:tasks.workunit.client.1.smithi189.stdout:3/594: fsync d2/d3/d2c/d31/d74/d7a/fd0 0 2024-04-20T21:32:15.881 INFO:tasks.workunit.client.1.smithi189.stdout:2/548: dwrite d1/d11/d52/d8b/fa1 [0,4194304] 0 2024-04-20T21:32:15.881 INFO:tasks.workunit.client.1.smithi189.stdout:0/701: symlink d3/df/d30/d45/lea 0 2024-04-20T21:32:15.881 INFO:tasks.workunit.client.0.smithi062.stdout:3/805: mkdir d0/d1/d10/d10b 0 2024-04-20T21:32:15.881 INFO:tasks.workunit.client.0.smithi062.stdout:9/985: chown d16/d2f/d5b/dab/dcd/f11e 113879840 1 2024-04-20T21:32:15.882 INFO:tasks.workunit.client.0.smithi062.stdout:7/848: fsync d7/d3e/d33/d3d/d4d/d54/f1f 0 2024-04-20T21:32:15.883 INFO:tasks.workunit.client.1.smithi189.stdout:0/702: dread - d3/df/d28/d2b/db1/fb8 zero size 2024-04-20T21:32:15.884 INFO:tasks.workunit.client.1.smithi189.stdout:0/703: fsync d3/fd 0 2024-04-20T21:32:15.885 INFO:tasks.workunit.client.0.smithi062.stdout:5/855: dwrite d8/d38/d7a/d4c/dcf/ff2 [0,4194304] 0 2024-04-20T21:32:15.886 INFO:tasks.workunit.client.1.smithi189.stdout:3/595: dread - d2/d3/d65/f8b zero size 2024-04-20T21:32:15.889 INFO:tasks.workunit.client.0.smithi062.stdout:8/880: fsync d7/d25/d3b/fb6 0 2024-04-20T21:32:15.890 INFO:tasks.workunit.client.0.smithi062.stdout:2/894: creat d7/da/d15/d1b/da8/f122 x:0 0 0 2024-04-20T21:32:15.890 INFO:tasks.workunit.client.0.smithi062.stdout:3/806: creat d0/d1/d3c/d43/d72/dd3/dbf/f10c x:0 0 0 2024-04-20T21:32:15.900 INFO:tasks.workunit.client.1.smithi189.stdout:0/704: dwrite d3/df/d30/d45/d90/fc9 [0,4194304] 0 2024-04-20T21:32:15.902 INFO:tasks.workunit.client.1.smithi189.stdout:0/705: fsync d3/df/d28/d2b/d2c/fd0 0 2024-04-20T21:32:15.905 INFO:tasks.workunit.client.1.smithi189.stdout:0/706: write d3/df/d28/d2b/d2c/f83 [380877,535] 0 2024-04-20T21:32:15.917 INFO:tasks.workunit.client.1.smithi189.stdout:5/579: unlink d7/d14/f49 0 2024-04-20T21:32:15.918 INFO:tasks.workunit.client.1.smithi189.stdout:5/580: chown d7/d35/d48/l50 1121 1 2024-04-20T21:32:15.919 INFO:tasks.workunit.client.1.smithi189.stdout:7/645: symlink d5/db4/d71/d7a/ld4 0 2024-04-20T21:32:15.919 INFO:tasks.workunit.client.1.smithi189.stdout:6/641: read - d8/da/dd/f92 zero size 2024-04-20T21:32:15.921 INFO:tasks.workunit.client.1.smithi189.stdout:7/646: write d5/fe [2613492,8615] 0 2024-04-20T21:32:15.922 INFO:tasks.workunit.client.1.smithi189.stdout:9/550: truncate d9/d12/d15/d1a/f64 570492 0 2024-04-20T21:32:15.922 INFO:tasks.workunit.client.1.smithi189.stdout:2/549: rename d1/d11/d52/c88 to d1/d17/d78/ca2 0 2024-04-20T21:32:15.922 INFO:tasks.workunit.client.1.smithi189.stdout:7/647: stat d5/d46/d7d/fb1 0 2024-04-20T21:32:15.924 INFO:tasks.workunit.client.1.smithi189.stdout:6/642: truncate d8/da/dd/d35/d6d/d77/f93 220054 0 2024-04-20T21:32:15.931 INFO:tasks.workunit.client.0.smithi062.stdout:0/829: getdents d7/d1c/d15/d17/d40/d3c/da4 0 2024-04-20T21:32:15.932 INFO:tasks.workunit.client.0.smithi062.stdout:5/856: chown d8/d10/c21 14 1 2024-04-20T21:32:15.933 INFO:tasks.workunit.client.0.smithi062.stdout:0/830: readlink d7/d1c/d103/l54 0 2024-04-20T21:32:15.934 INFO:tasks.workunit.client.1.smithi189.stdout:0/707: symlink d3/d26/d88/leb 0 2024-04-20T21:32:15.936 INFO:tasks.workunit.client.0.smithi062.stdout:5/857: chown d8/d38/d7a/d4c/fe6 3203947 1 2024-04-20T21:32:15.937 INFO:tasks.workunit.client.0.smithi062.stdout:0/831: readlink d7/d1c/d15/d17/d40/d3c/lf3 0 2024-04-20T21:32:15.939 INFO:tasks.workunit.client.1.smithi189.stdout:2/550: mknod d1/d11/d58/d6b/ca3 0 2024-04-20T21:32:15.941 INFO:tasks.workunit.client.1.smithi189.stdout:9/551: rename d9/d12/d15 to d9/d12/d3b/db1 0 2024-04-20T21:32:15.943 INFO:tasks.workunit.client.0.smithi062.stdout:3/807: write d0/d1c/d63/d42/f60 [2547924,129621] 0 2024-04-20T21:32:15.943 INFO:tasks.workunit.client.0.smithi062.stdout:9/986: creat d16/d2c/d42/da6/da9/dfd/f148 x:0 0 0 2024-04-20T21:32:15.944 INFO:tasks.workunit.client.0.smithi062.stdout:7/849: symlink d7/d3e/l126 0 2024-04-20T21:32:15.945 INFO:tasks.workunit.client.1.smithi189.stdout:5/581: mknod d7/d9/d51/d70/d8a/cb7 0 2024-04-20T21:32:15.945 INFO:tasks.workunit.client.1.smithi189.stdout:9/552: chown d9/d12/d3b/db1/d1a/l9b 518688296 1 2024-04-20T21:32:15.948 INFO:tasks.workunit.client.0.smithi062.stdout:1/930: getdents dc/d70 0 2024-04-20T21:32:15.948 INFO:tasks.workunit.client.0.smithi062.stdout:5/858: chown d8/d10/d3e/d8a/fa0 6152459 1 2024-04-20T21:32:15.948 INFO:tasks.workunit.client.0.smithi062.stdout:4/930: getdents da/dd/d15/daf/d116 0 2024-04-20T21:32:15.949 INFO:tasks.workunit.client.0.smithi062.stdout:2/895: getdents d7/d39 0 2024-04-20T21:32:15.950 INFO:tasks.workunit.client.1.smithi189.stdout:2/551: rmdir d1/d17/d78 39 2024-04-20T21:32:15.952 INFO:tasks.workunit.client.1.smithi189.stdout:6/643: mknod d8/cd2 0 2024-04-20T21:32:15.954 INFO:tasks.workunit.client.0.smithi062.stdout:7/850: mknod d7/d26/dac/ddb/de8/c127 0 2024-04-20T21:32:15.954 INFO:tasks.workunit.client.0.smithi062.stdout:9/987: creat d16/d2c/d62/dd1/df8/f149 x:0 0 0 2024-04-20T21:32:15.959 INFO:tasks.workunit.client.1.smithi189.stdout:5/582: unlink d7/d35/d48/f60 0 2024-04-20T21:32:15.959 INFO:tasks.workunit.client.1.smithi189.stdout:5/583: chown d7/l12 27 1 2024-04-20T21:32:15.959 INFO:tasks.workunit.client.1.smithi189.stdout:0/708: mkdir d3/df/d30/d45/d60/dec 0 2024-04-20T21:32:15.960 INFO:tasks.workunit.client.1.smithi189.stdout:2/552: mknod d1/d11/d52/d6f/ca4 0 2024-04-20T21:32:15.960 INFO:tasks.workunit.client.0.smithi062.stdout:3/808: creat d0/d1c/d63/d42/f10d x:0 0 0 2024-04-20T21:32:15.960 INFO:tasks.workunit.client.0.smithi062.stdout:4/931: fsync da/dd/d15/d6d/f10e 0 2024-04-20T21:32:15.961 INFO:tasks.workunit.client.0.smithi062.stdout:8/881: dwrite d7/d25/d4f/fdb [0,4194304] 0 2024-04-20T21:32:15.961 INFO:tasks.workunit.client.0.smithi062.stdout:5/859: dread - d8/d38/d7a/d4c/d7d/d9d/dac/f112 zero size 2024-04-20T21:32:15.963 INFO:tasks.workunit.client.0.smithi062.stdout:3/809: write d0/d1/d10/d3a/d5e/f7d [3744933,79096] 0 2024-04-20T21:32:15.967 INFO:tasks.workunit.client.1.smithi189.stdout:3/596: write d2/d3/d2c/d30/d3f/d9a/fb5 [3448508,76508] 0 2024-04-20T21:32:15.967 INFO:tasks.workunit.client.0.smithi062.stdout:2/896: dread - d7/da/d15/d2f/d38/d51/dbc/fcd zero size 2024-04-20T21:32:15.968 INFO:tasks.workunit.client.0.smithi062.stdout:1/931: creat dc/d15/d20/d28/d76/d137/f156 x:0 0 0 2024-04-20T21:32:15.969 INFO:tasks.workunit.client.1.smithi189.stdout:5/584: rename d7/d9/d30/d55/d61/fa4 to d7/d9/fb8 0 2024-04-20T21:32:15.970 INFO:tasks.workunit.client.1.smithi189.stdout:3/597: read d2/d3/d25/f5b [2446396,50678] 0 2024-04-20T21:32:15.972 INFO:tasks.workunit.client.1.smithi189.stdout:3/598: stat d2/d3/d25 0 2024-04-20T21:32:15.975 INFO:tasks.workunit.client.1.smithi189.stdout:0/709: dread d3/df/d28/d2b/fe2 [0,4194304] 0 2024-04-20T21:32:15.977 INFO:tasks.workunit.client.0.smithi062.stdout:9/988: rename d16/d2c/l35 to d16/d2c/d10d/l14a 0 2024-04-20T21:32:15.978 INFO:tasks.workunit.client.0.smithi062.stdout:8/882: mknod d7/d9/c121 0 2024-04-20T21:32:15.978 INFO:tasks.workunit.client.1.smithi189.stdout:5/585: truncate d7/d9/d31/d36/f54 732978 0 2024-04-20T21:32:15.980 INFO:tasks.workunit.client.0.smithi062.stdout:5/860: mkdir d8/d38/d7a/d4c/d11d 0 2024-04-20T21:32:15.982 INFO:tasks.workunit.client.0.smithi062.stdout:7/851: dwrite d7/d26/da8/db6/fd8 [0,4194304] 0 2024-04-20T21:32:15.983 INFO:tasks.workunit.client.0.smithi062.stdout:4/932: write da/dd/df/d27/fc9 [526674,69833] 0 2024-04-20T21:32:15.984 INFO:tasks.workunit.client.0.smithi062.stdout:9/989: mkdir d16/d2c/d62/dd1/df0/d14b 0 2024-04-20T21:32:15.986 INFO:tasks.workunit.client.0.smithi062.stdout:3/810: rename d0/d1c/d63/d42/dc8/dc9/fcf to d0/d1c/d63/d42/d5b/f10e 0 2024-04-20T21:32:15.987 INFO:tasks.workunit.client.0.smithi062.stdout:4/933: rmdir da/d4f 39 2024-04-20T21:32:15.992 INFO:tasks.workunit.client.0.smithi062.stdout:2/897: getdents d7/da/d15/d2f/d38/d10f/db9 0 2024-04-20T21:32:15.993 INFO:tasks.workunit.client.0.smithi062.stdout:5/861: creat d8/f11e x:0 0 0 2024-04-20T21:32:15.995 INFO:tasks.workunit.client.0.smithi062.stdout:7/852: symlink d7/d3e/d33/d3d/d4d/l128 0 2024-04-20T21:32:15.998 INFO:tasks.workunit.client.0.smithi062.stdout:1/932: rename dc/c31 to dc/d15/d75/d96/dd4/d3a/da2/de4/d84/dbe/d10d/d150/c157 0 2024-04-20T21:32:16.002 INFO:tasks.workunit.client.0.smithi062.stdout:7/853: creat d7/d3e/d33/d3d/d4d/d54/dea/d80/f129 x:0 0 0 2024-04-20T21:32:16.005 INFO:tasks.workunit.client.0.smithi062.stdout:9/990: dwrite d16/dbe/f147 [0,4194304] 0 2024-04-20T21:32:16.005 INFO:tasks.workunit.client.0.smithi062.stdout:5/862: getdents d8/d1e/d2c/d6c/df9 0 2024-04-20T21:32:16.006 INFO:tasks.workunit.client.0.smithi062.stdout:4/934: write da/d56/d71/dbd/fc8 [727085,6205] 0 2024-04-20T21:32:16.024 INFO:tasks.workunit.client.1.smithi189.stdout:8/692: sync 2024-04-20T21:32:16.026 INFO:tasks.workunit.client.0.smithi062.stdout:7/854: symlink d7/d3e/d33/d3d/d7f/d116/l12a 0 2024-04-20T21:32:16.029 INFO:tasks.workunit.client.1.smithi189.stdout:3/599: unlink d2/d3/d2c/d30/d3f/d9a/db0/fb8 0 2024-04-20T21:32:16.030 INFO:tasks.workunit.client.0.smithi062.stdout:7/855: write d7/d3e/d33/d3d/d4d/d54/d4f/de9/df1/df3/ff5 [470720,111639] 0 2024-04-20T21:32:16.030 INFO:tasks.workunit.client.1.smithi189.stdout:0/710: write d3/df/d30/d45/d90/fc9 [4411472,45575] 0 2024-04-20T21:32:16.031 INFO:tasks.workunit.client.0.smithi062.stdout:5/863: fsync d8/d38/d7a/d4c/d7d/d9d/dac/d66/dae/d88/f8c 0 2024-04-20T21:32:16.032 INFO:tasks.workunit.client.1.smithi189.stdout:2/553: truncate d1/d11/f2e 1360245 0 2024-04-20T21:32:16.032 INFO:tasks.workunit.client.1.smithi189.stdout:6/644: write d8/f21 [3096070,109905] 0 2024-04-20T21:32:16.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:15 smithi062 ceph-mon[16221]: pgmap v22: 65 pgs: 65 active+clean; 2.7 GiB data, 8.0 GiB used, 528 GiB / 536 GiB avail; 104 MiB/s rd, 288 MiB/s wr, 590 op/s 2024-04-20T21:32:16.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:15 smithi062 ceph-mon[16221]: from='client.14724 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:32:16.035 INFO:tasks.workunit.client.0.smithi062.stdout:9/991: rmdir d16/d2c/d42/d100/d119 0 2024-04-20T21:32:16.037 INFO:tasks.workunit.client.0.smithi062.stdout:5/864: mknod d8/d1e/d2c/dd1/ded/c11f 0 2024-04-20T21:32:16.038 INFO:tasks.workunit.client.0.smithi062.stdout:7/856: truncate d7/d3e/d33/d3d/d4d/d83/fc8 46046 0 2024-04-20T21:32:16.039 INFO:tasks.workunit.client.0.smithi062.stdout:5/865: chown d8/d10/d3e/cd0 178 1 2024-04-20T21:32:16.039 INFO:tasks.workunit.client.0.smithi062.stdout:9/992: rmdir d16/d2c/d62/dd1/df0 39 2024-04-20T21:32:16.039 INFO:tasks.workunit.client.0.smithi062.stdout:7/857: chown d7/ff 55 1 2024-04-20T21:32:16.041 INFO:tasks.workunit.client.0.smithi062.stdout:8/883: dread d7/d25/dc5/ff0 [0,4194304] 0 2024-04-20T21:32:16.042 INFO:tasks.workunit.client.1.smithi189.stdout:9/553: dread d9/dc/f1c [0,4194304] 0 2024-04-20T21:32:16.042 INFO:tasks.workunit.client.1.smithi189.stdout:3/600: write d2/d3/d2c/d31/fa7 [1805716,110045] 0 2024-04-20T21:32:16.043 INFO:tasks.workunit.client.0.smithi062.stdout:8/884: dread - d7/de1/f111 zero size 2024-04-20T21:32:16.048 INFO:tasks.workunit.client.1.smithi189.stdout:9/554: dread - d9/dc/d17/f8e zero size 2024-04-20T21:32:16.055 INFO:tasks.workunit.client.0.smithi062.stdout:8/885: creat d7/d25/dc5/f122 x:0 0 0 2024-04-20T21:32:16.064 INFO:tasks.workunit.client.0.smithi062.stdout:8/886: creat d7/d25/d70/df7/f123 x:0 0 0 2024-04-20T21:32:16.065 INFO:tasks.workunit.client.1.smithi189.stdout:6/645: dwrite d8/d12/fae [0,4194304] 0 2024-04-20T21:32:16.069 INFO:tasks.workunit.client.0.smithi062.stdout:8/887: symlink d7/da9/db0/l124 0 2024-04-20T21:32:16.078 INFO:tasks.workunit.client.1.smithi189.stdout:0/711: dread d3/df/d5d/d64/f96 [0,4194304] 0 2024-04-20T21:32:16.080 INFO:tasks.workunit.client.1.smithi189.stdout:0/712: chown d3/df/d28/d2b/lc1 13 1 2024-04-20T21:32:16.082 INFO:tasks.workunit.client.0.smithi062.stdout:5/866: rmdir d8 39 2024-04-20T21:32:16.085 INFO:tasks.workunit.client.0.smithi062.stdout:8/888: dwrite d7/d30/f5d [4194304,4194304] 0 2024-04-20T21:32:16.091 INFO:tasks.workunit.client.0.smithi062.stdout:8/889: rmdir d7/d25/d3b/d58 39 2024-04-20T21:32:16.091 INFO:tasks.workunit.client.0.smithi062.stdout:5/867: unlink d8/d10/d3e/f62 0 2024-04-20T21:32:16.094 INFO:tasks.workunit.client.0.smithi062.stdout:8/890: chown d7/d25/d70/df7 132786245 1 2024-04-20T21:32:16.094 INFO:tasks.workunit.client.0.smithi062.stdout:5/868: write d8/d10/f74 [2493832,100349] 0 2024-04-20T21:32:16.097 INFO:tasks.workunit.client.0.smithi062.stdout:8/891: mknod d7/de1/de5/c125 0 2024-04-20T21:32:16.098 INFO:tasks.workunit.client.0.smithi062.stdout:7/858: dwrite d7/d3e/d33/d3d/d7f/f109 [0,4194304] 0 2024-04-20T21:32:16.099 INFO:tasks.workunit.client.1.smithi189.stdout:0/713: dwrite d3/fe [4194304,4194304] 0 2024-04-20T21:32:16.100 INFO:tasks.workunit.client.0.smithi062.stdout:5/869: rmdir d8/d38/d7a 39 2024-04-20T21:32:16.104 INFO:tasks.workunit.client.0.smithi062.stdout:8/892: dread d7/d9/d29/f4b [0,4194304] 0 2024-04-20T21:32:16.109 INFO:tasks.workunit.client.0.smithi062.stdout:7/859: mknod d7/d3e/d33/d3d/d4d/d54/dea/d80/c12b 0 2024-04-20T21:32:16.113 INFO:tasks.workunit.client.0.smithi062.stdout:5/870: mknod d8/d38/d7a/d4c/dcf/c120 0 2024-04-20T21:32:16.118 INFO:tasks.workunit.client.0.smithi062.stdout:8/893: rename d7/d25/d3b/d58/d8b/d24/f74 to d7/d25/d76/f126 0 2024-04-20T21:32:16.120 INFO:tasks.workunit.client.1.smithi189.stdout:4/565: sync 2024-04-20T21:32:16.121 INFO:tasks.workunit.client.0.smithi062.stdout:5/871: rename d8/d1e/d2c/dd1/ded to d8/d38/d7a/db1/d121 0 2024-04-20T21:32:16.123 INFO:tasks.workunit.client.0.smithi062.stdout:5/872: read d8/d38/f89 [1530756,52712] 0 2024-04-20T21:32:16.128 INFO:tasks.workunit.client.0.smithi062.stdout:5/873: creat d8/d38/d7a/d4c/f122 x:0 0 0 2024-04-20T21:32:16.130 INFO:tasks.workunit.client.0.smithi062.stdout:5/874: fsync d8/d38/d7a/d4c/dcf/fee 0 2024-04-20T21:32:16.132 INFO:tasks.workunit.client.0.smithi062.stdout:7/860: write d7/d3e/d33/d3d/d4d/d54/dea/f93 [984220,103456] 0 2024-04-20T21:32:16.134 INFO:tasks.workunit.client.0.smithi062.stdout:7/861: chown d7/d26/dac/ddb/de8/f11d 14413567 1 2024-04-20T21:32:16.137 INFO:tasks.workunit.client.1.smithi189.stdout:1/567: sync 2024-04-20T21:32:16.137 INFO:tasks.workunit.client.1.smithi189.stdout:7/648: sync 2024-04-20T21:32:16.139 INFO:tasks.workunit.client.0.smithi062.stdout:7/862: rmdir d7/d26/dac/ddb/de8 39 2024-04-20T21:32:16.139 INFO:tasks.workunit.client.1.smithi189.stdout:1/568: stat d8/d21/fa5 0 2024-04-20T21:32:16.142 INFO:tasks.workunit.client.1.smithi189.stdout:1/569: write d8/d21/d2c/fa3 [4505580,86918] 0 2024-04-20T21:32:16.144 INFO:tasks.workunit.client.1.smithi189.stdout:3/601: mkdir d2/d3/d2c/d31/d74/db7/dd3 0 2024-04-20T21:32:16.147 INFO:tasks.workunit.client.1.smithi189.stdout:2/554: dread d1/d17/f2b [0,4194304] 0 2024-04-20T21:32:16.148 INFO:tasks.workunit.client.1.smithi189.stdout:6/646: truncate d8/da/f6b 2572215 0 2024-04-20T21:32:16.150 INFO:tasks.workunit.client.0.smithi062.stdout:8/894: dread d7/d25/d3b/d58/d8b/f1f [0,4194304] 0 2024-04-20T21:32:16.150 INFO:tasks.workunit.client.0.smithi062.stdout:7/863: link d7/d26/dac/ddb/de8/f115 d7/d3e/d33/d3d/d4d/d54/d4f/d9d/f12c 0 2024-04-20T21:32:16.152 INFO:tasks.workunit.client.0.smithi062.stdout:7/864: fdatasync d7/d3e/d33/d3d/d4d/fa3 0 2024-04-20T21:32:16.153 INFO:tasks.workunit.client.0.smithi062.stdout:8/895: fsync d7/d25/dc5/ff0 0 2024-04-20T21:32:16.156 INFO:tasks.workunit.client.0.smithi062.stdout:7/865: write d7/d3e/d33/d3d/d4d/d54/f3f [2040248,130968] 0 2024-04-20T21:32:16.159 INFO:tasks.workunit.client.0.smithi062.stdout:7/866: dread - d7/d26/dac/ddb/de8/f11d zero size 2024-04-20T21:32:16.159 INFO:tasks.workunit.client.1.smithi189.stdout:7/649: mknod d5/db4/d71/d7a/cd5 0 2024-04-20T21:32:16.161 INFO:tasks.workunit.client.1.smithi189.stdout:4/566: dread d2/d20/fc4 [0,4194304] 0 2024-04-20T21:32:16.162 INFO:tasks.workunit.client.0.smithi062.stdout:7/867: stat d7/d3e/d33/d3d/d5e/d72/lc9 0 2024-04-20T21:32:16.166 INFO:tasks.workunit.client.1.smithi189.stdout:4/567: dread - d2/d7f/fa0 zero size 2024-04-20T21:32:16.166 INFO:tasks.workunit.client.1.smithi189.stdout:8/693: mkdir da/df/d13/dd6/df0/df8 0 2024-04-20T21:32:16.166 INFO:tasks.workunit.client.1.smithi189.stdout:1/570: symlink d8/d21/lc3 0 2024-04-20T21:32:16.167 INFO:tasks.workunit.client.0.smithi062.stdout:7/868: creat d7/d3e/d33/d3d/d4d/d54/d4f/de9/df1/df3/f12d x:0 0 0 2024-04-20T21:32:16.168 INFO:tasks.workunit.client.1.smithi189.stdout:3/602: write d2/f59 [2186498,53801] 0 2024-04-20T21:32:16.168 INFO:tasks.workunit.client.1.smithi189.stdout:9/555: creat d9/d12/d62/dae/fb2 x:0 0 0 2024-04-20T21:32:16.172 INFO:tasks.workunit.client.0.smithi062.stdout:7/869: unlink d7/d3e/d33/d3d/d4d/d54/f1f 0 2024-04-20T21:32:16.172 INFO:tasks.workunit.client.1.smithi189.stdout:0/714: mkdir d3/db9/ded 0 2024-04-20T21:32:16.175 INFO:tasks.workunit.client.1.smithi189.stdout:4/568: chown d2/l9 6 1 2024-04-20T21:32:16.175 INFO:tasks.workunit.client.0.smithi062.stdout:8/896: dread d7/d25/d3b/d58/d8b/f1e [0,4194304] 0 2024-04-20T21:32:16.177 INFO:tasks.workunit.client.1.smithi189.stdout:1/571: mknod d8/df/d3a/d49/d8c/da2/dbe/cc4 0 2024-04-20T21:32:16.178 INFO:tasks.workunit.client.1.smithi189.stdout:4/569: chown d2/dd/d11/c77 65295 1 2024-04-20T21:32:16.179 INFO:tasks.workunit.client.0.smithi062.stdout:8/897: rmdir d7/d25/d70 39 2024-04-20T21:32:16.185 INFO:tasks.workunit.client.0.smithi062.stdout:7/870: dwrite d7/d3e/d33/d3d/d5e/d72/f98 [0,4194304] 0 2024-04-20T21:32:16.189 INFO:tasks.workunit.client.1.smithi189.stdout:4/570: creat d2/dd/d37/dcd/d6a/fd0 x:0 0 0 2024-04-20T21:32:16.190 INFO:tasks.workunit.client.1.smithi189.stdout:7/650: link d5/d46/l95 d5/d21/d97/ld6 0 2024-04-20T21:32:16.191 INFO:tasks.workunit.client.1.smithi189.stdout:3/603: dwrite d2/d3/f1b [0,4194304] 0 2024-04-20T21:32:16.194 INFO:tasks.workunit.client.1.smithi189.stdout:2/555: link d1/l3c d1/d11/d99/la5 0 2024-04-20T21:32:16.198 INFO:tasks.workunit.client.0.smithi062.stdout:8/898: dwrite d7/d9/d29/f32 [0,4194304] 0 2024-04-20T21:32:16.199 INFO:tasks.workunit.client.1.smithi189.stdout:7/651: fdatasync d5/d21/d25/d34/f6f 0 2024-04-20T21:32:16.199 INFO:tasks.workunit.client.1.smithi189.stdout:3/604: symlink d2/d3/d36/da1/ld4 0 2024-04-20T21:32:16.205 INFO:tasks.workunit.client.0.smithi062.stdout:7/871: write d7/d3e/d33/d3d/f4a [7889056,41162] 0 2024-04-20T21:32:16.206 INFO:tasks.workunit.client.0.smithi062.stdout:8/899: mkdir d7/d25/d3b/d58/d8b/d24/dbd/d127 0 2024-04-20T21:32:16.208 INFO:tasks.workunit.client.1.smithi189.stdout:7/652: write d5/db4/d71/fa6 [1809044,64572] 0 2024-04-20T21:32:16.210 INFO:tasks.workunit.client.1.smithi189.stdout:0/715: dwrite d3/df/f3e [0,4194304] 0 2024-04-20T21:32:16.211 INFO:tasks.workunit.client.0.smithi062.stdout:7/872: symlink d7/d3e/d33/d3d/d5e/l12e 0 2024-04-20T21:32:16.216 INFO:tasks.workunit.client.1.smithi189.stdout:7/653: creat d5/d1d/d64/fd7 x:0 0 0 2024-04-20T21:32:16.221 INFO:tasks.workunit.client.1.smithi189.stdout:2/556: dwrite d1/d11/d58/fa0 [0,4194304] 0 2024-04-20T21:32:16.223 INFO:tasks.workunit.client.1.smithi189.stdout:0/716: truncate d3/df/d4c/dd4/f94 5079401 0 2024-04-20T21:32:16.223 INFO:tasks.workunit.client.1.smithi189.stdout:5/586: write d7/d9/d28/f7b [1031254,112597] 0 2024-04-20T21:32:16.228 INFO:tasks.workunit.client.1.smithi189.stdout:9/556: rmdir d9/dc/d69 39 2024-04-20T21:32:16.228 INFO:tasks.workunit.client.1.smithi189.stdout:5/587: dread - d7/d9/d28/fb0 zero size 2024-04-20T21:32:16.231 INFO:tasks.workunit.client.1.smithi189.stdout:2/557: rename d1/d11/d2d/f41 to d1/d11/d2d/fa6 0 2024-04-20T21:32:16.235 INFO:tasks.workunit.client.1.smithi189.stdout:5/588: creat d7/d35/d48/fb9 x:0 0 0 2024-04-20T21:32:16.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:15 smithi189 ceph-mon[21562]: pgmap v22: 65 pgs: 65 active+clean; 2.7 GiB data, 8.0 GiB used, 528 GiB / 536 GiB avail; 104 MiB/s rd, 288 MiB/s wr, 590 op/s 2024-04-20T21:32:16.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:15 smithi189 ceph-mon[21562]: from='client.14724 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:32:16.243 INFO:tasks.workunit.client.1.smithi189.stdout:7/654: dwrite d5/d21/d97/da3/dc6/fcb [0,4194304] 0 2024-04-20T21:32:16.245 INFO:tasks.workunit.client.1.smithi189.stdout:5/589: getdents d7/d9/d31/d36 0 2024-04-20T21:32:16.246 INFO:tasks.workunit.client.1.smithi189.stdout:7/655: rmdir d5/d21/d25/d34 39 2024-04-20T21:32:16.252 INFO:tasks.workunit.client.1.smithi189.stdout:7/656: truncate d5/d1d/d77/fd3 770172 0 2024-04-20T21:32:16.253 INFO:tasks.workunit.client.1.smithi189.stdout:1/572: write d8/d21/f38 [1482470,78156] 0 2024-04-20T21:32:16.253 INFO:tasks.workunit.client.1.smithi189.stdout:7/657: chown d5/d46 141 1 2024-04-20T21:32:16.258 INFO:tasks.workunit.client.1.smithi189.stdout:1/573: creat d8/d1e/fc5 x:0 0 0 2024-04-20T21:32:16.261 INFO:tasks.workunit.client.1.smithi189.stdout:7/658: unlink d5/d1d/d77/db8/fc3 0 2024-04-20T21:32:16.262 INFO:tasks.workunit.client.1.smithi189.stdout:3/605: creat d2/d3/d2c/d31/fd5 x:0 0 0 2024-04-20T21:32:16.265 INFO:tasks.workunit.client.1.smithi189.stdout:1/574: creat d8/d1e/fc6 x:0 0 0 2024-04-20T21:32:16.266 INFO:tasks.workunit.client.1.smithi189.stdout:8/694: rmdir da/df/d20/dc9 39 2024-04-20T21:32:16.266 INFO:tasks.workunit.client.1.smithi189.stdout:6/647: mknod d8/da/dd/d35/cd3 0 2024-04-20T21:32:16.270 INFO:tasks.workunit.client.1.smithi189.stdout:0/717: write d3/df/d5d/d64/f6b [836573,67415] 0 2024-04-20T21:32:16.272 INFO:tasks.workunit.client.1.smithi189.stdout:9/557: write d9/dc/d17/f2f [5064936,100060] 0 2024-04-20T21:32:16.274 INFO:tasks.workunit.client.1.smithi189.stdout:8/695: fdatasync da/df/d3c/d68/d8b/fcf 0 2024-04-20T21:32:16.275 INFO:tasks.workunit.client.1.smithi189.stdout:3/606: creat d2/d1c/d5c/fd6 x:0 0 0 2024-04-20T21:32:16.276 INFO:tasks.workunit.client.1.smithi189.stdout:1/575: creat d8/d21/d2c/d36/fc7 x:0 0 0 2024-04-20T21:32:16.276 INFO:tasks.workunit.client.1.smithi189.stdout:5/590: dwrite d7/d9/d30/f1c [0,4194304] 0 2024-04-20T21:32:16.277 INFO:tasks.workunit.client.0.smithi062.stdout:6/723: sync 2024-04-20T21:32:16.278 INFO:tasks.workunit.client.0.smithi062.stdout:7/873: dread d7/f36 [4194304,4194304] 0 2024-04-20T21:32:16.281 INFO:tasks.workunit.client.0.smithi062.stdout:6/724: fdatasync d0/d1a/d44/fac 0 2024-04-20T21:32:16.282 INFO:tasks.workunit.client.1.smithi189.stdout:2/558: dread d1/d17/d2f/f49 [0,4194304] 0 2024-04-20T21:32:16.282 INFO:tasks.workunit.client.1.smithi189.stdout:0/718: rmdir d3/df/d30 39 2024-04-20T21:32:16.284 INFO:tasks.workunit.client.1.smithi189.stdout:9/558: unlink d9/d12/d3b/db1/l49 0 2024-04-20T21:32:16.284 INFO:tasks.workunit.client.1.smithi189.stdout:6/648: truncate d8/d10/fc6 4090114 0 2024-04-20T21:32:16.288 INFO:tasks.workunit.client.1.smithi189.stdout:8/696: creat da/d1f/d2a/ff9 x:0 0 0 2024-04-20T21:32:16.289 INFO:tasks.workunit.client.0.smithi062.stdout:6/725: dread - d0/d4f/fdf zero size 2024-04-20T21:32:16.291 INFO:tasks.workunit.client.1.smithi189.stdout:2/559: read d1/d11/d2d/f50 [1809809,117536] 0 2024-04-20T21:32:16.292 INFO:tasks.workunit.client.1.smithi189.stdout:1/576: rename d8/d1e/l2f to d8/d1e/d28/d4f/lc8 0 2024-04-20T21:32:16.294 INFO:tasks.workunit.client.1.smithi189.stdout:7/659: dwrite d5/d1d/d64/f85 [0,4194304] 0 2024-04-20T21:32:16.294 INFO:tasks.workunit.client.1.smithi189.stdout:6/649: chown d8/d2b/d49/l5d 4 1 2024-04-20T21:32:16.294 INFO:tasks.workunit.client.1.smithi189.stdout:1/577: fsync f2 0 2024-04-20T21:32:16.296 INFO:tasks.workunit.client.0.smithi062.stdout:6/726: mkdir d0/d9/df5 0 2024-04-20T21:32:16.297 INFO:tasks.workunit.client.1.smithi189.stdout:9/559: getdents d9/dc/d17/d95/da0/d52/d82/d94 0 2024-04-20T21:32:16.299 INFO:tasks.workunit.client.1.smithi189.stdout:6/650: fsync d8/d2b/d49/d54/d6e/fb4 0 2024-04-20T21:32:16.300 INFO:tasks.workunit.client.0.smithi062.stdout:6/727: creat d0/d1a/d44/ff6 x:0 0 0 2024-04-20T21:32:16.303 INFO:tasks.workunit.client.1.smithi189.stdout:1/578: creat d8/d1e/d43/d44/fc9 x:0 0 0 2024-04-20T21:32:16.303 INFO:tasks.workunit.client.1.smithi189.stdout:8/697: read - da/df/d3c/d68/d8b/d8d/f90 zero size 2024-04-20T21:32:16.303 INFO:tasks.workunit.client.0.smithi062.stdout:7/874: truncate d7/d3e/f6a 1303995 0 2024-04-20T21:32:16.306 INFO:tasks.workunit.client.0.smithi062.stdout:6/728: getdents d0/d9/d11/d22/dd5/d67/d75/df4 0 2024-04-20T21:32:16.306 INFO:tasks.workunit.client.1.smithi189.stdout:6/651: creat d8/d10/d78/d7e/fd4 x:0 0 0 2024-04-20T21:32:16.306 INFO:tasks.workunit.client.1.smithi189.stdout:2/560: truncate d1/d11/d52/d8b/fa1 5198121 0 2024-04-20T21:32:16.308 INFO:tasks.workunit.client.1.smithi189.stdout:0/719: creat d3/df/d30/d45/dd3/fee x:0 0 0 2024-04-20T21:32:16.309 INFO:tasks.workunit.client.1.smithi189.stdout:8/698: chown da/de/d1e/d21/f97 98957 1 2024-04-20T21:32:16.310 INFO:tasks.workunit.client.0.smithi062.stdout:7/875: mknod d7/d3e/d33/d3d/d4d/d54/d4f/de9/c12f 0 2024-04-20T21:32:16.312 INFO:tasks.workunit.client.1.smithi189.stdout:3/607: dread d2/d3/d25/f40 [4194304,4194304] 0 2024-04-20T21:32:16.312 INFO:tasks.workunit.client.0.smithi062.stdout:6/729: creat d0/d9/d11/ff7 x:0 0 0 2024-04-20T21:32:16.313 INFO:tasks.workunit.client.0.smithi062.stdout:7/876: rmdir d7/d3e 39 2024-04-20T21:32:16.313 INFO:tasks.workunit.client.1.smithi189.stdout:9/560: creat d9/dc/d17/d95/da6/fb3 x:0 0 0 2024-04-20T21:32:16.313 INFO:tasks.workunit.client.1.smithi189.stdout:2/561: symlink d1/d11/d2d/la7 0 2024-04-20T21:32:16.316 INFO:tasks.workunit.client.1.smithi189.stdout:7/660: link d5/d1d/d77/l9f d5/d1d/d64/ld8 0 2024-04-20T21:32:16.320 INFO:tasks.workunit.client.1.smithi189.stdout:0/720: creat d3/df/d30/fef x:0 0 0 2024-04-20T21:32:16.322 INFO:tasks.workunit.client.1.smithi189.stdout:6/652: truncate d8/d2b/d49/d54/d67/fce 847445 0 2024-04-20T21:32:16.322 INFO:tasks.workunit.client.0.smithi062.stdout:7/877: creat d7/d3e/d33/d3d/d4d/d54/dea/f130 x:0 0 0 2024-04-20T21:32:16.326 INFO:tasks.workunit.client.1.smithi189.stdout:5/591: dwrite d7/d9/d31/f57 [0,4194304] 0 2024-04-20T21:32:16.326 INFO:tasks.workunit.client.0.smithi062.stdout:7/878: chown d7/d3e/d33/d3d/d4d/d54/d4f/fc2 6891384 1 2024-04-20T21:32:16.327 INFO:tasks.workunit.client.1.smithi189.stdout:5/592: readlink d7/d14/l3d 0 2024-04-20T21:32:16.329 INFO:tasks.workunit.client.0.smithi062.stdout:6/730: dwrite d0/d9/d11/fbc [4194304,4194304] 0 2024-04-20T21:32:16.334 INFO:tasks.workunit.client.0.smithi062.stdout:6/731: chown d0/cc 691 1 2024-04-20T21:32:16.337 INFO:tasks.workunit.client.1.smithi189.stdout:8/699: dwrite da/df/d20/d27/f38 [0,4194304] 0 2024-04-20T21:32:16.341 INFO:tasks.workunit.client.0.smithi062.stdout:6/732: mknod d0/d9/d11/d7b/cf8 0 2024-04-20T21:32:16.342 INFO:tasks.workunit.client.1.smithi189.stdout:1/579: dread d8/df/f15 [0,4194304] 0 2024-04-20T21:32:16.358 INFO:tasks.workunit.client.1.smithi189.stdout:2/562: dread d1/d17/f34 [4194304,4194304] 0 2024-04-20T21:32:16.359 INFO:tasks.workunit.client.0.smithi062.stdout:7/879: dwrite d7/f24 [4194304,4194304] 0 2024-04-20T21:32:16.366 INFO:tasks.workunit.client.0.smithi062.stdout:7/880: mkdir d7/d26/da8/d131 0 2024-04-20T21:32:16.371 INFO:tasks.workunit.client.0.smithi062.stdout:7/881: fdatasync d7/d3e/d33/d3d/d5e/faf 0 2024-04-20T21:32:16.375 INFO:tasks.workunit.client.1.smithi189.stdout:7/661: dread d5/f6 [0,4194304] 0 2024-04-20T21:32:16.375 INFO:tasks.workunit.client.0.smithi062.stdout:7/882: write d7/d3e/d33/d3d/d4d/d54/dea/fa9 [3760518,110750] 0 2024-04-20T21:32:16.378 INFO:tasks.workunit.client.1.smithi189.stdout:2/563: dwrite d1/d17/d2f/d4a/f8f [0,4194304] 0 2024-04-20T21:32:16.380 INFO:tasks.workunit.client.0.smithi062.stdout:7/883: mkdir d7/d26/dac/ddb/d132 0 2024-04-20T21:32:16.387 INFO:tasks.workunit.client.0.smithi062.stdout:7/884: rename d7/d3e/d33/d3d/d4d/f110 to d7/d3e/d33/d3d/d4d/d83/d95/f133 0 2024-04-20T21:32:16.391 INFO:tasks.workunit.client.0.smithi062.stdout:7/885: readlink d7/d3e/d33/d3d/d4d/d54/d4f/de9/df1/df3/l100 0 2024-04-20T21:32:16.396 INFO:tasks.workunit.client.0.smithi062.stdout:7/886: creat d7/d3e/d33/d3d/d4d/d54/d4f/de9/df1/d105/f134 x:0 0 0 2024-04-20T21:32:16.401 INFO:tasks.workunit.client.0.smithi062.stdout:7/887: creat d7/d3e/d33/d3d/d5e/db8/f135 x:0 0 0 2024-04-20T21:32:16.425 INFO:tasks.workunit.client.0.smithi062.stdout:7/888: dwrite d7/d3e/d33/d3d/d4d/d54/dea/d80/f129 [0,4194304] 0 2024-04-20T21:32:16.429 INFO:tasks.workunit.client.1.smithi189.stdout:4/571: sync 2024-04-20T21:32:16.430 INFO:tasks.workunit.client.1.smithi189.stdout:9/561: rename d9/dc/d17/c60 to d9/d12/d62/dae/cb4 0 2024-04-20T21:32:16.433 INFO:tasks.workunit.client.1.smithi189.stdout:0/721: read d3/df/f91 [48942,68594] 0 2024-04-20T21:32:16.436 INFO:tasks.workunit.client.1.smithi189.stdout:8/700: mkdir da/df/d13/d63/d7b/dfa 0 2024-04-20T21:32:16.437 INFO:tasks.workunit.client.1.smithi189.stdout:0/722: fsync d3/df/d28/d2b/f36 0 2024-04-20T21:32:16.438 INFO:tasks.workunit.client.1.smithi189.stdout:1/580: creat d8/d1e/fca x:0 0 0 2024-04-20T21:32:16.441 INFO:tasks.workunit.client.1.smithi189.stdout:1/581: truncate d8/d1e/d54/fae 406119 0 2024-04-20T21:32:16.441 INFO:tasks.workunit.client.1.smithi189.stdout:7/662: creat d5/d46/d7d/fd9 x:0 0 0 2024-04-20T21:32:16.444 INFO:tasks.workunit.client.0.smithi062.stdout:7/889: truncate d7/d3e/d33/d3d/d4d/d54/f2e 2754899 0 2024-04-20T21:32:16.449 INFO:tasks.workunit.client.0.smithi062.stdout:7/890: mknod d7/d3e/d33/d3d/d4d/d54/dea/dcc/c136 0 2024-04-20T21:32:16.458 INFO:tasks.workunit.client.0.smithi062.stdout:7/891: dread d7/d3e/d33/d3d/d4d/d83/d95/faa [0,4194304] 0 2024-04-20T21:32:16.462 INFO:tasks.workunit.client.0.smithi062.stdout:7/892: symlink d7/d3e/d33/d3d/d5e/d72/l137 0 2024-04-20T21:32:16.466 INFO:tasks.workunit.client.0.smithi062.stdout:7/893: mkdir d7/d3e/d33/d3d/d4d/d138 0 2024-04-20T21:32:16.467 INFO:tasks.workunit.client.0.smithi062.stdout:7/894: chown d7/d3e/lcb 46326 1 2024-04-20T21:32:16.470 INFO:tasks.workunit.client.0.smithi062.stdout:7/895: mkdir d7/d3e/d33/d3d/d7f/d116/d139 0 2024-04-20T21:32:16.483 INFO:tasks.workunit.client.1.smithi189.stdout:4/572: rename d2/f38 to d2/d22/d87/fd1 0 2024-04-20T21:32:16.483 INFO:tasks.workunit.client.1.smithi189.stdout:2/564: link d1/d17/d2f/d55/f94 d1/d11/fa8 0 2024-04-20T21:32:16.484 INFO:tasks.workunit.client.1.smithi189.stdout:7/663: creat d5/d21/d4a/d52/d75/fda x:0 0 0 2024-04-20T21:32:16.484 INFO:tasks.workunit.client.1.smithi189.stdout:3/608: write d2/d3/d2c/d30/d3f/d9a/fb5 [1892405,79292] 0 2024-04-20T21:32:16.485 INFO:tasks.workunit.client.1.smithi189.stdout:5/593: truncate d7/d9/d30/d55/d61/f8d 689266 0 2024-04-20T21:32:16.486 INFO:tasks.workunit.client.1.smithi189.stdout:4/573: chown d2/dd/d11/d14/l16 297293 1 2024-04-20T21:32:16.486 INFO:tasks.workunit.client.0.smithi062.stdout:7/896: dwrite d7/f18 [0,4194304] 0 2024-04-20T21:32:16.487 INFO:tasks.workunit.client.1.smithi189.stdout:1/582: write d8/d1e/d43/d44/f47 [305595,20435] 0 2024-04-20T21:32:16.488 INFO:tasks.workunit.client.1.smithi189.stdout:3/609: chown d2/d8/lc7 29 1 2024-04-20T21:32:16.491 INFO:tasks.workunit.client.1.smithi189.stdout:7/664: creat d5/d21/d97/da3/fdb x:0 0 0 2024-04-20T21:32:16.491 INFO:tasks.workunit.client.1.smithi189.stdout:4/574: chown d2/dd/d37/dcd/d6a/c93 82836 1 2024-04-20T21:32:16.491 INFO:tasks.workunit.client.1.smithi189.stdout:1/583: chown d8/d1e/d43/d58/d60/f64 12 1 2024-04-20T21:32:16.492 INFO:tasks.workunit.client.1.smithi189.stdout:0/723: rename d3/df/d5d/d64/lbc to d3/df/d28/lf0 0 2024-04-20T21:32:16.493 INFO:tasks.workunit.client.0.smithi062.stdout:7/897: truncate d7/d26/dac/d10a/f11c 573363 0 2024-04-20T21:32:16.493 INFO:tasks.workunit.client.1.smithi189.stdout:5/594: mknod d7/d14/d4e/d5b/cba 0 2024-04-20T21:32:16.494 INFO:tasks.workunit.client.1.smithi189.stdout:2/565: creat d1/fa9 x:0 0 0 2024-04-20T21:32:16.495 INFO:tasks.workunit.client.0.smithi062.stdout:0/832: sync 2024-04-20T21:32:16.498 INFO:tasks.workunit.client.1.smithi189.stdout:3/610: creat d2/d3/d36/fd7 x:0 0 0 2024-04-20T21:32:16.498 INFO:tasks.workunit.client.1.smithi189.stdout:4/575: symlink d2/dd/d37/dcd/d6a/ld2 0 2024-04-20T21:32:16.501 INFO:tasks.workunit.client.1.smithi189.stdout:5/595: stat d7/c9c 0 2024-04-20T21:32:16.501 INFO:tasks.workunit.client.0.smithi062.stdout:7/898: link d7/dcd/cd5 d7/d3e/d33/d3d/d4d/d54/dea/dcc/c13a 0 2024-04-20T21:32:16.503 INFO:tasks.workunit.client.1.smithi189.stdout:7/665: mkdir d5/db4/d71/ddc 0 2024-04-20T21:32:16.505 INFO:tasks.workunit.client.0.smithi062.stdout:0/833: getdents d7/d1c/d15/d53 0 2024-04-20T21:32:16.508 INFO:tasks.workunit.client.1.smithi189.stdout:4/576: symlink d2/dd/d37/dcd/d6a/ld3 0 2024-04-20T21:32:16.510 INFO:tasks.workunit.client.1.smithi189.stdout:4/577: stat d2/d20/lab 0 2024-04-20T21:32:16.513 INFO:tasks.workunit.client.1.smithi189.stdout:7/666: mkdir d5/d1d/d64/db9/ddd 0 2024-04-20T21:32:16.515 INFO:tasks.workunit.client.0.smithi062.stdout:7/899: dread d7/d3e/d33/d3d/d4d/d54/dea/d80/fdd [0,4194304] 0 2024-04-20T21:32:16.516 INFO:tasks.workunit.client.1.smithi189.stdout:3/611: truncate d2/d3/d2c/d30/d3f/fa5 502287 0 2024-04-20T21:32:16.518 INFO:tasks.workunit.client.1.smithi189.stdout:5/596: mknod d7/d9/d30/d55/d33/cbb 0 2024-04-20T21:32:16.520 INFO:tasks.workunit.client.1.smithi189.stdout:1/584: link d8/df/d3a/d49/l53 d8/d1e/d43/lcb 0 2024-04-20T21:32:16.522 INFO:tasks.workunit.client.1.smithi189.stdout:4/578: creat d2/dd/d11/dae/fd4 x:0 0 0 2024-04-20T21:32:16.523 INFO:tasks.workunit.client.1.smithi189.stdout:0/724: write d3/df/d4c/dd4/f94 [5136860,72095] 0 2024-04-20T21:32:16.524 INFO:tasks.workunit.client.1.smithi189.stdout:3/612: creat d2/d3/d2c/d31/d90/fd8 x:0 0 0 2024-04-20T21:32:16.526 INFO:tasks.workunit.client.0.smithi062.stdout:0/834: dwrite d7/d1e/d6e/f7d [0,4194304] 0 2024-04-20T21:32:16.528 INFO:tasks.workunit.client.1.smithi189.stdout:1/585: symlink d8/d1e/d43/d44/db2/lcc 0 2024-04-20T21:32:16.529 INFO:tasks.workunit.client.1.smithi189.stdout:7/667: dwrite d5/ff [4194304,4194304] 0 2024-04-20T21:32:16.532 INFO:tasks.workunit.client.0.smithi062.stdout:7/900: dwrite d7/d3e/d33/d3d/d4d/fa3 [0,4194304] 0 2024-04-20T21:32:16.533 INFO:tasks.workunit.client.1.smithi189.stdout:7/668: stat d5/d21/d35/l3f 0 2024-04-20T21:32:16.534 INFO:tasks.workunit.client.1.smithi189.stdout:4/579: symlink d2/d22/d87/ld5 0 2024-04-20T21:32:16.538 INFO:tasks.workunit.client.1.smithi189.stdout:1/586: unlink d8/df/d50/f88 0 2024-04-20T21:32:16.539 INFO:tasks.workunit.client.1.smithi189.stdout:5/597: dwrite d7/d9/d28/f7b [0,4194304] 0 2024-04-20T21:32:16.545 INFO:tasks.workunit.client.0.smithi062.stdout:0/835: dwrite d7/d20/d62/d6f/f9b [0,4194304] 0 2024-04-20T21:32:16.546 INFO:tasks.workunit.client.1.smithi189.stdout:5/598: mknod d7/d9/d30/d55/d61/d96/cbc 0 2024-04-20T21:32:16.548 INFO:tasks.workunit.client.1.smithi189.stdout:3/613: getdents d2/d3/d65 0 2024-04-20T21:32:16.551 INFO:tasks.workunit.client.1.smithi189.stdout:2/566: dwrite d1/d11/d2d/d4f/d8e/f9a [0,4194304] 0 2024-04-20T21:32:16.558 INFO:tasks.workunit.client.1.smithi189.stdout:7/669: rename d5/d46/d7d/fc9 to d5/d21/d25/fde 0 2024-04-20T21:32:16.564 INFO:tasks.workunit.client.1.smithi189.stdout:3/614: truncate d2/d1c/dc1/dc5/fcc 801070 0 2024-04-20T21:32:16.564 INFO:tasks.workunit.client.1.smithi189.stdout:5/599: chown d7/d9/d51/d70/d8a/cb7 0 1 2024-04-20T21:32:16.564 INFO:tasks.workunit.client.1.smithi189.stdout:1/587: getdents d8/d9b/dbb 0 2024-04-20T21:32:16.564 INFO:tasks.workunit.client.1.smithi189.stdout:7/670: stat d5/d21/d4a/d52/d75/fb2 0 2024-04-20T21:32:16.566 INFO:tasks.workunit.client.1.smithi189.stdout:5/600: chown d7/d9/d31/f88 1601 1 2024-04-20T21:32:16.575 INFO:tasks.workunit.client.1.smithi189.stdout:4/580: dwrite d2/d22/da9/fbf [0,4194304] 0 2024-04-20T21:32:16.577 INFO:tasks.workunit.client.1.smithi189.stdout:3/615: truncate d2/d3/d25/f67 644281 0 2024-04-20T21:32:16.579 INFO:tasks.workunit.client.0.smithi062.stdout:0/836: dread d7/d1c/d15/d80/fb2 [0,4194304] 0 2024-04-20T21:32:16.581 INFO:tasks.workunit.client.1.smithi189.stdout:1/588: getdents d8/d21/d2c 0 2024-04-20T21:32:16.582 INFO:tasks.workunit.client.1.smithi189.stdout:4/581: readlink d2/dd/l2e 0 2024-04-20T21:32:16.583 INFO:tasks.workunit.client.0.smithi062.stdout:0/837: readlink d7/d20/d62/l8a 0 2024-04-20T21:32:16.585 INFO:tasks.workunit.client.1.smithi189.stdout:1/589: readlink d8/d21/lc3 0 2024-04-20T21:32:16.592 INFO:tasks.workunit.client.1.smithi189.stdout:7/671: dwrite d5/d1d/d77/fd3 [0,4194304] 0 2024-04-20T21:32:16.594 INFO:tasks.workunit.client.1.smithi189.stdout:5/601: link d7/d9/d30/d55/c6c d7/cbd 0 2024-04-20T21:32:16.597 INFO:tasks.workunit.client.1.smithi189.stdout:0/725: truncate d3/df/d28/f6a 3826766 0 2024-04-20T21:32:16.597 INFO:tasks.workunit.client.1.smithi189.stdout:1/590: rename d8/d21/f38 to d8/df/d3a/d49/d8c/da2/dbe/dc1/fcd 0 2024-04-20T21:32:16.600 INFO:tasks.workunit.client.1.smithi189.stdout:5/602: unlink d7/d9/d31/f57 0 2024-04-20T21:32:16.601 INFO:tasks.workunit.client.0.smithi062.stdout:7/901: dwrite d7/d3e/f45 [4194304,4194304] 0 2024-04-20T21:32:16.607 INFO:tasks.workunit.client.1.smithi189.stdout:1/591: unlink d8/d1e/d54/ca9 0 2024-04-20T21:32:16.609 INFO:tasks.workunit.client.1.smithi189.stdout:7/672: creat d5/db4/d71/fdf x:0 0 0 2024-04-20T21:32:16.609 INFO:tasks.workunit.client.0.smithi062.stdout:0/838: dread d7/d1c/d15/d17/d1b/d106/db1/fd4 [4194304,4194304] 0 2024-04-20T21:32:16.612 INFO:tasks.workunit.client.1.smithi189.stdout:0/726: rename d3/df/d30/d45/d60 to d3/df/d28/dc8/df1 0 2024-04-20T21:32:16.617 INFO:tasks.workunit.client.1.smithi189.stdout:7/673: creat d5/d1d/d64/db9/fe0 x:0 0 0 2024-04-20T21:32:16.618 INFO:tasks.workunit.client.1.smithi189.stdout:3/616: dwrite d2/fd [0,4194304] 0 2024-04-20T21:32:16.619 INFO:tasks.workunit.client.1.smithi189.stdout:5/603: getdents d7 0 2024-04-20T21:32:16.621 INFO:tasks.workunit.client.1.smithi189.stdout:0/727: chown d3/df/d28/d2b/d2c/f62 8874 1 2024-04-20T21:32:16.622 INFO:tasks.workunit.client.1.smithi189.stdout:7/674: truncate d5/d46/fb7 184050 0 2024-04-20T21:32:16.623 INFO:tasks.workunit.client.1.smithi189.stdout:2/567: dwrite d1/d11/fa8 [0,4194304] 0 2024-04-20T21:32:16.625 INFO:tasks.workunit.client.1.smithi189.stdout:1/592: getdents d8/d1e/d28/d2b 0 2024-04-20T21:32:16.627 INFO:tasks.workunit.client.1.smithi189.stdout:5/604: creat d7/d9/d31/fbe x:0 0 0 2024-04-20T21:32:16.629 INFO:tasks.workunit.client.1.smithi189.stdout:3/617: truncate d2/d3/f48 2616118 0 2024-04-20T21:32:16.631 INFO:tasks.workunit.client.1.smithi189.stdout:6/653: sync 2024-04-20T21:32:16.632 INFO:tasks.workunit.client.1.smithi189.stdout:5/605: dread - d7/d9/d28/da3/fa8 zero size 2024-04-20T21:32:16.635 INFO:tasks.workunit.client.0.smithi062.stdout:3/811: sync 2024-04-20T21:32:16.635 INFO:tasks.workunit.client.0.smithi062.stdout:1/933: sync 2024-04-20T21:32:16.635 INFO:tasks.workunit.client.0.smithi062.stdout:2/898: sync 2024-04-20T21:32:16.641 INFO:tasks.workunit.client.1.smithi189.stdout:1/593: symlink d8/d1e/d54/d93/lce 0 2024-04-20T21:32:16.643 INFO:tasks.workunit.client.1.smithi189.stdout:5/606: creat d7/d9/fbf x:0 0 0 2024-04-20T21:32:16.643 INFO:tasks.workunit.client.1.smithi189.stdout:0/728: rename d3/f66 to d3/df/d30/d37/ff2 0 2024-04-20T21:32:16.644 INFO:tasks.workunit.client.1.smithi189.stdout:1/594: write d8/f14 [3756346,78631] 0 2024-04-20T21:32:16.649 INFO:tasks.workunit.client.1.smithi189.stdout:7/675: link d5/d21/d25/f78 d5/d21/d25/fe1 0 2024-04-20T21:32:16.651 INFO:tasks.workunit.client.1.smithi189.stdout:0/729: mknod d3/db9/cf3 0 2024-04-20T21:32:16.652 INFO:tasks.workunit.client.1.smithi189.stdout:4/582: dread d2/dd/f2d [0,4194304] 0 2024-04-20T21:32:16.653 INFO:tasks.workunit.client.1.smithi189.stdout:1/595: mkdir d8/d21/d2c/d36/db7/dcf 0 2024-04-20T21:32:16.654 INFO:tasks.workunit.client.1.smithi189.stdout:5/607: link d7/d9/d30/d55/c6c d7/d35/cc0 0 2024-04-20T21:32:16.655 INFO:tasks.workunit.client.1.smithi189.stdout:6/654: dwrite d8/d12/f17 [0,4194304] 0 2024-04-20T21:32:16.656 INFO:tasks.workunit.client.1.smithi189.stdout:4/583: write d2/f5a [2492330,111391] 0 2024-04-20T21:32:16.658 INFO:tasks.workunit.client.1.smithi189.stdout:7/676: creat d5/db4/d71/fe2 x:0 0 0 2024-04-20T21:32:16.662 INFO:tasks.workunit.client.1.smithi189.stdout:6/655: write d8/d2b/d49/f60 [871596,79459] 0 2024-04-20T21:32:16.663 INFO:tasks.workunit.client.1.smithi189.stdout:7/677: stat d5/d1d/d77/fd3 0 2024-04-20T21:32:16.674 INFO:tasks.workunit.client.1.smithi189.stdout:5/608: link d7/d14/f8b d7/d9/d31/fc1 0 2024-04-20T21:32:16.675 INFO:tasks.workunit.client.1.smithi189.stdout:7/678: read d5/d1d/d64/fa1 [4175362,20483] 0 2024-04-20T21:32:16.676 INFO:tasks.workunit.client.1.smithi189.stdout:3/618: truncate d2/d8d/f91 303409 0 2024-04-20T21:32:16.680 INFO:tasks.workunit.client.1.smithi189.stdout:5/609: stat d7/d9/d30/d55/d61/db3 0 2024-04-20T21:32:16.680 INFO:tasks.workunit.client.1.smithi189.stdout:9/562: sync 2024-04-20T21:32:16.680 INFO:tasks.workunit.client.1.smithi189.stdout:8/701: sync 2024-04-20T21:32:16.680 INFO:tasks.workunit.client.1.smithi189.stdout:2/568: truncate d1/d11/d52/d8b/fa1 600473 0 2024-04-20T21:32:16.681 INFO:tasks.workunit.client.1.smithi189.stdout:0/730: dwrite d3/df/d4c/fa4 [0,4194304] 0 2024-04-20T21:32:16.683 INFO:tasks.workunit.client.0.smithi062.stdout:5/875: sync 2024-04-20T21:32:16.683 INFO:tasks.workunit.client.0.smithi062.stdout:4/935: sync 2024-04-20T21:32:16.683 INFO:tasks.workunit.client.0.smithi062.stdout:6/733: sync 2024-04-20T21:32:16.683 INFO:tasks.workunit.client.0.smithi062.stdout:8/900: sync 2024-04-20T21:32:16.683 INFO:tasks.workunit.client.0.smithi062.stdout:9/993: sync 2024-04-20T21:32:16.684 INFO:tasks.workunit.client.1.smithi189.stdout:9/563: chown d9/l35 19679 1 2024-04-20T21:32:16.684 INFO:tasks.workunit.client.0.smithi062.stdout:7/902: mkdir d7/d3e/d33/d3d/d4d/d54/d4f/de9/df1/d105/d13b 0 2024-04-20T21:32:16.686 INFO:tasks.workunit.client.1.smithi189.stdout:5/610: chown d7/d9/d30/d5a/d9d 817293 1 2024-04-20T21:32:16.687 INFO:tasks.workunit.client.0.smithi062.stdout:9/994: chown d16/f32 0 1 2024-04-20T21:32:16.687 INFO:tasks.workunit.client.0.smithi062.stdout:6/734: chown d0/d1a/c3d 190325306 1 2024-04-20T21:32:16.692 INFO:tasks.workunit.client.1.smithi189.stdout:3/619: truncate d2/d3/d25/f2e 3532326 0 2024-04-20T21:32:16.695 INFO:tasks.workunit.client.0.smithi062.stdout:2/899: mkdir d7/da/d123 0 2024-04-20T21:32:16.695 INFO:tasks.workunit.client.0.smithi062.stdout:3/812: truncate d0/d11/f56 2426152 0 2024-04-20T21:32:16.695 INFO:tasks.workunit.client.0.smithi062.stdout:5/876: symlink d8/d10/l123 0 2024-04-20T21:32:16.697 INFO:tasks.workunit.client.1.smithi189.stdout:4/584: dwrite d2/dd/f44 [0,4194304] 0 2024-04-20T21:32:16.700 INFO:tasks.workunit.client.1.smithi189.stdout:3/620: write d2/d3/d65/fb6 [235435,45050] 0 2024-04-20T21:32:16.703 INFO:tasks.workunit.client.1.smithi189.stdout:8/702: truncate da/de/d1e/d69/d70/f72 633384 0 2024-04-20T21:32:16.703 INFO:tasks.workunit.client.1.smithi189.stdout:3/621: readlink d2/d3/d2c/d30/l38 0 2024-04-20T21:32:16.704 INFO:tasks.workunit.client.1.smithi189.stdout:5/611: creat d7/d14/d4e/d5b/fc2 x:0 0 0 2024-04-20T21:32:16.705 INFO:tasks.workunit.client.1.smithi189.stdout:0/731: rename d3/df/d28/d2b/d2c/d74 to d3/df/d30/d45/d90/df4 0 2024-04-20T21:32:16.705 INFO:tasks.workunit.client.0.smithi062.stdout:1/934: dread dc/d27/d32/f74 [0,4194304] 0 2024-04-20T21:32:16.707 INFO:tasks.workunit.client.0.smithi062.stdout:6/735: symlink d0/d9/d11/d22/lf9 0 2024-04-20T21:32:16.715 INFO:tasks.workunit.client.1.smithi189.stdout:4/585: read d2/d3a/f3c [1820980,11316] 0 2024-04-20T21:32:16.715 INFO:tasks.workunit.client.0.smithi062.stdout:4/936: dwrite da/dd/d15/d50/d64/d57/d5b/d9d/dea/dac/fcd [4194304,4194304] 0 2024-04-20T21:32:16.717 INFO:tasks.workunit.client.0.smithi062.stdout:7/903: rename d7/d3e/d33/d3d/d4d/d54/dea/c88 to d7/d26/da8/d131/c13c 0 2024-04-20T21:32:16.718 INFO:tasks.workunit.client.1.smithi189.stdout:1/596: dwrite d8/d1e/d28/fa8 [0,4194304] 0 2024-04-20T21:32:16.718 INFO:tasks.workunit.client.1.smithi189.stdout:5/612: creat d7/d9/d28/da3/fc3 x:0 0 0 2024-04-20T21:32:16.721 INFO:tasks.workunit.client.1.smithi189.stdout:0/732: read d3/df/d28/d2b/f36 [3539123,92163] 0 2024-04-20T21:32:16.721 INFO:tasks.workunit.client.0.smithi062.stdout:3/813: truncate d0/d1c/d63/d42/f73 4958886 0 2024-04-20T21:32:16.725 INFO:tasks.workunit.client.1.smithi189.stdout:3/622: fdatasync d2/d3/f19 0 2024-04-20T21:32:16.725 INFO:tasks.workunit.client.0.smithi062.stdout:3/814: write d0/d1c/d63/d42/dc8/dc9/ff9 [472473,122856] 0 2024-04-20T21:32:16.729 INFO:tasks.workunit.client.0.smithi062.stdout:5/877: truncate d8/d38/d7a/d4c/da7/f10c 2539697 0 2024-04-20T21:32:16.729 INFO:tasks.workunit.client.0.smithi062.stdout:3/815: dread d0/d1c/d63/f91 [0,4194304] 0 2024-04-20T21:32:16.731 INFO:tasks.workunit.client.1.smithi189.stdout:8/703: getdents da/df/d3c/d50/d6c/dc7 0 2024-04-20T21:32:16.732 INFO:tasks.workunit.client.0.smithi062.stdout:3/816: chown d0/d1/d10/fa4 187019 1 2024-04-20T21:32:16.733 INFO:tasks.workunit.client.1.smithi189.stdout:4/586: rmdir d2/dd/d37/dcd 39 2024-04-20T21:32:16.734 INFO:tasks.workunit.client.0.smithi062.stdout:5/878: dread - d8/d38/d7a/db1/d121/f11c zero size 2024-04-20T21:32:16.735 INFO:tasks.workunit.client.0.smithi062.stdout:3/817: write d0/fdc [2757111,71358] 0 2024-04-20T21:32:16.737 INFO:tasks.workunit.client.1.smithi189.stdout:1/597: chown d8/d1e/fc5 3980800 1 2024-04-20T21:32:16.738 INFO:tasks.workunit.client.1.smithi189.stdout:0/733: unlink d3/df/d5d/f7b 0 2024-04-20T21:32:16.739 INFO:tasks.workunit.client.1.smithi189.stdout:6/656: dwrite d8/d10/d78/d7e/f84 [0,4194304] 0 2024-04-20T21:32:16.742 INFO:tasks.workunit.client.0.smithi062.stdout:8/901: getdents d7 0 2024-04-20T21:32:16.745 INFO:tasks.workunit.client.1.smithi189.stdout:5/613: symlink d7/d9/d30/d55/d61/daa/db5/lc4 0 2024-04-20T21:32:16.748 INFO:tasks.workunit.client.1.smithi189.stdout:7/679: dwrite d5/d1d/f79 [0,4194304] 0 2024-04-20T21:32:16.748 INFO:tasks.workunit.client.1.smithi189.stdout:6/657: stat d8/d12/daf/fba 0 2024-04-20T21:32:16.751 INFO:tasks.workunit.client.1.smithi189.stdout:4/587: rename d2/l95 to d2/d3a/ld6 0 2024-04-20T21:32:16.754 INFO:tasks.workunit.client.1.smithi189.stdout:2/569: dwrite d1/d17/d2f/f72 [0,4194304] 0 2024-04-20T21:32:16.754 INFO:tasks.workunit.client.1.smithi189.stdout:1/598: creat d8/d21/d2c/d36/d8a/fd0 x:0 0 0 2024-04-20T21:32:16.756 INFO:tasks.workunit.client.1.smithi189.stdout:8/704: creat da/df/d13/dd6/d59/ffb x:0 0 0 2024-04-20T21:32:16.758 INFO:tasks.workunit.client.1.smithi189.stdout:3/623: truncate d2/d3/d2c/d31/d74/d7a/fd0 486862 0 2024-04-20T21:32:16.761 INFO:tasks.workunit.client.1.smithi189.stdout:5/614: mknod d7/d9/d30/d55/d61/daa/db5/cc5 0 2024-04-20T21:32:16.762 INFO:tasks.workunit.client.0.smithi062.stdout:9/995: getdents d16/d2f 0 2024-04-20T21:32:16.763 INFO:tasks.workunit.client.1.smithi189.stdout:3/624: chown d2/d3/d2c/d30/c9d 12742976 1 2024-04-20T21:32:16.764 INFO:tasks.workunit.client.0.smithi062.stdout:1/935: mkdir dc/d27/d32/d13a/d158 0 2024-04-20T21:32:16.764 INFO:tasks.workunit.client.1.smithi189.stdout:7/680: rmdir d5/d1d/d77 39 2024-04-20T21:32:16.768 INFO:tasks.workunit.client.1.smithi189.stdout:3/625: chown d2/d3/d2c/d31/d90 109619 1 2024-04-20T21:32:16.768 INFO:tasks.workunit.client.1.smithi189.stdout:5/615: write d7/d9/d30/d55/d61/daa/fae [114663,52806] 0 2024-04-20T21:32:16.768 INFO:tasks.workunit.client.0.smithi062.stdout:2/900: creat d7/f124 x:0 0 0 2024-04-20T21:32:16.770 INFO:tasks.workunit.client.1.smithi189.stdout:6/658: creat d8/da/dd/d35/d6d/d77/fd5 x:0 0 0 2024-04-20T21:32:16.770 INFO:tasks.workunit.client.1.smithi189.stdout:9/564: dwrite d9/d12/d3f/d72/f7b [0,4194304] 0 2024-04-20T21:32:16.772 INFO:tasks.workunit.client.0.smithi062.stdout:3/818: fdatasync d0/d1/f20 0 2024-04-20T21:32:16.774 INFO:tasks.workunit.client.0.smithi062.stdout:5/879: creat d8/d38/d7a/d4c/d7d/d9d/dac/d2f/d8e/dd6/f124 x:0 0 0 2024-04-20T21:32:16.776 INFO:tasks.workunit.client.1.smithi189.stdout:7/681: truncate d5/d21/d4a/d52/d74/fbb 172785 0 2024-04-20T21:32:16.776 INFO:tasks.workunit.client.1.smithi189.stdout:0/734: creat d3/df/d28/dc8/df1/ff5 x:0 0 0 2024-04-20T21:32:16.777 INFO:tasks.workunit.client.0.smithi062.stdout:3/819: chown d0/d1/d3c/d43/d72/dd3/d97/l9d 33 1 2024-04-20T21:32:16.778 INFO:tasks.workunit.client.1.smithi189.stdout:8/705: mkdir da/df/d20/d2c/d9a/ddd/dfc 0 2024-04-20T21:32:16.779 INFO:tasks.workunit.client.0.smithi062.stdout:5/880: readlink d8/d1e/l78 0 2024-04-20T21:32:16.781 INFO:tasks.workunit.client.1.smithi189.stdout:2/570: dread d1/d17/d2f/d55/f94 [0,4194304] 0 2024-04-20T21:32:16.781 INFO:tasks.workunit.client.0.smithi062.stdout:7/904: creat d7/d26/dac/ddb/d132/f13d x:0 0 0 2024-04-20T21:32:16.781 INFO:tasks.workunit.client.0.smithi062.stdout:8/902: truncate d7/d25/d3b/fb6 1037847 0 2024-04-20T21:32:16.781 INFO:tasks.workunit.client.0.smithi062.stdout:3/820: truncate d0/d1/fb 691320 0 2024-04-20T21:32:16.782 INFO:tasks.workunit.client.1.smithi189.stdout:3/626: unlink d2/d3/d65/fb6 0 2024-04-20T21:32:16.783 INFO:tasks.workunit.client.0.smithi062.stdout:9/996: rmdir d16/d2c/d62/dd1/df8 39 2024-04-20T21:32:16.787 INFO:tasks.workunit.client.1.smithi189.stdout:9/565: rename d9/dc/d17/d95/da0/d52/c9e to d9/d12/cb5 0 2024-04-20T21:32:16.788 INFO:tasks.workunit.client.0.smithi062.stdout:8/903: chown d7/d25/dc5/fde 4 1 2024-04-20T21:32:16.789 INFO:tasks.workunit.client.1.smithi189.stdout:5/616: chown d7/d35/cc0 8527 1 2024-04-20T21:32:16.794 INFO:tasks.workunit.client.0.smithi062.stdout:4/937: getdents da/dd/df/d109 0 2024-04-20T21:32:16.794 INFO:tasks.workunit.client.0.smithi062.stdout:5/881: creat d8/d1e/d2c/dd1/f125 x:0 0 0 2024-04-20T21:32:16.795 INFO:tasks.workunit.client.1.smithi189.stdout:4/588: dwrite d2/d20/f24 [4194304,4194304] 0 2024-04-20T21:32:16.797 INFO:tasks.workunit.client.0.smithi062.stdout:3/821: unlink d0/d1c/d63/d42/f10d 0 2024-04-20T21:32:16.798 INFO:tasks.workunit.client.1.smithi189.stdout:0/735: symlink d3/df/d30/d37/dbd/lf6 0 2024-04-20T21:32:16.799 INFO:tasks.workunit.client.0.smithi062.stdout:1/936: fdatasync dc/d15/d75/d96/dd4/d3a/da2/de4/d7c/f88 0 2024-04-20T21:32:16.802 INFO:tasks.workunit.client.1.smithi189.stdout:2/571: mkdir d1/d11/d52/d6f/daa 0 2024-04-20T21:32:16.802 INFO:tasks.workunit.client.1.smithi189.stdout:0/736: write d3/fd [3214730,22372] 0 2024-04-20T21:32:16.804 INFO:tasks.workunit.client.0.smithi062.stdout:8/904: creat d7/d25/d3b/d58/dac/f128 x:0 0 0 2024-04-20T21:32:16.805 INFO:tasks.workunit.client.1.smithi189.stdout:0/737: readlink d3/df/d28/lc3 0 2024-04-20T21:32:16.805 INFO:tasks.workunit.client.1.smithi189.stdout:5/617: mknod d7/d9/d30/d5a/d9d/cc6 0 2024-04-20T21:32:16.805 INFO:tasks.workunit.client.1.smithi189.stdout:4/589: mknod d2/dd/d11/d14/cd7 0 2024-04-20T21:32:16.806 INFO:tasks.workunit.client.0.smithi062.stdout:4/938: mknod da/d56/d71/dbd/c141 0 2024-04-20T21:32:16.808 INFO:tasks.workunit.client.1.smithi189.stdout:8/706: rmdir da/df/d3c/d68/d86 0 2024-04-20T21:32:16.808 INFO:tasks.workunit.client.1.smithi189.stdout:2/572: truncate d1/fa9 1045502 0 2024-04-20T21:32:16.809 INFO:tasks.workunit.client.1.smithi189.stdout:0/738: fdatasync d3/df/d30/d45/f6c 0 2024-04-20T21:32:16.810 INFO:tasks.workunit.client.0.smithi062.stdout:4/939: chown da/dd/f45 2 1 2024-04-20T21:32:16.810 INFO:tasks.workunit.client.0.smithi062.stdout:9/997: mknod d16/d2c/d5e/d75/c14c 0 2024-04-20T21:32:16.813 INFO:tasks.workunit.client.0.smithi062.stdout:9/998: readlink d16/d2c/d10d/l112 0 2024-04-20T21:32:16.813 INFO:tasks.workunit.client.1.smithi189.stdout:3/627: dwrite d2/d8/f4b [0,4194304] 0 2024-04-20T21:32:16.815 INFO:tasks.workunit.client.0.smithi062.stdout:5/882: mkdir d8/d38/d7a/d4c/d7d/d9d/dac/d66/de4/d126 0 2024-04-20T21:32:16.815 INFO:tasks.workunit.client.1.smithi189.stdout:0/739: fsync d3/d8c/fc7 0 2024-04-20T21:32:16.816 INFO:tasks.workunit.client.0.smithi062.stdout:7/905: dwrite d7/d3e/d33/d3d/d5e/db8/f135 [0,4194304] 0 2024-04-20T21:32:16.818 INFO:tasks.workunit.client.1.smithi189.stdout:0/740: stat d3/df/d28/ca5 0 2024-04-20T21:32:16.818 INFO:tasks.workunit.client.1.smithi189.stdout:3/628: readlink d2/d3/d25/l3b 0 2024-04-20T21:32:16.818 INFO:tasks.workunit.client.0.smithi062.stdout:0/839: write d7/d1c/d15/d17/d37/f59 [1268160,99095] 0 2024-04-20T21:32:16.822 INFO:tasks.workunit.client.1.smithi189.stdout:7/682: dread d5/d46/faa [0,4194304] 0 2024-04-20T21:32:16.825 INFO:tasks.workunit.client.1.smithi189.stdout:7/683: write d5/db4/d71/fa6 [305999,111308] 0 2024-04-20T21:32:16.825 INFO:tasks.workunit.client.1.smithi189.stdout:6/659: truncate d8/f9b 4602290 0 2024-04-20T21:32:16.826 INFO:tasks.workunit.client.0.smithi062.stdout:4/940: mkdir da/dd/df/d27/d142 0 2024-04-20T21:32:16.829 INFO:tasks.workunit.client.1.smithi189.stdout:8/707: symlink da/df/d3c/d68/d8b/dac/lfd 0 2024-04-20T21:32:16.829 INFO:tasks.workunit.client.1.smithi189.stdout:9/566: getdents d9/d12/d3b 0 2024-04-20T21:32:16.829 INFO:tasks.workunit.client.0.smithi062.stdout:2/901: dread d7/da/d15/d1b/f2c [0,4194304] 0 2024-04-20T21:32:16.831 INFO:tasks.workunit.client.0.smithi062.stdout:0/840: chown d7/d20/c86 35892 1 2024-04-20T21:32:16.831 INFO:tasks.workunit.client.0.smithi062.stdout:8/905: mknod d7/d25/d70/c129 0 2024-04-20T21:32:16.832 INFO:tasks.workunit.client.0.smithi062.stdout:6/736: write d0/d9/d11/d22/dd5/d67/d75/f7a [44031,117588] 0 2024-04-20T21:32:16.832 INFO:tasks.workunit.client.1.smithi189.stdout:5/618: symlink d7/d9/d30/d55/d81/lc7 0 2024-04-20T21:32:16.833 INFO:tasks.workunit.client.1.smithi189.stdout:8/708: read - da/d1f/fd8 zero size 2024-04-20T21:32:16.839 INFO:tasks.workunit.client.0.smithi062.stdout:9/999: read d16/d2c/d42/da6/f10e [2089670,74110] 0 2024-04-20T21:32:16.840 INFO:tasks.workunit.client.0.smithi062.stdout:1/937: dwrite dc/d15/f5b [0,4194304] 0 2024-04-20T21:32:16.841 INFO:tasks.workunit.client.0.smithi062.stdout:7/906: read d7/d26/f2f [3206697,51702] 0 2024-04-20T21:32:16.842 INFO:tasks.workunit.client.1.smithi189.stdout:2/573: dread d1/d11/d52/d8b/f8d [0,4194304] 0 2024-04-20T21:32:16.842 INFO:tasks.workunit.client.1.smithi189.stdout:4/590: rename d2/dd/c17 to d2/dd/cd8 0 2024-04-20T21:32:16.843 INFO:tasks.workunit.client.1.smithi189.stdout:3/629: symlink d2/d3/d2c/d30/d3f/d9a/ld9 0 2024-04-20T21:32:16.844 INFO:tasks.workunit.client.1.smithi189.stdout:7/684: symlink d5/d21/d4a/le3 0 2024-04-20T21:32:16.846 INFO:tasks.workunit.client.1.smithi189.stdout:9/567: chown d9/d12/d3b/db1/d1a/f64 16341971 1 2024-04-20T21:32:16.848 INFO:tasks.workunit.client.1.smithi189.stdout:8/709: mkdir da/df/d3a/dfe 0 2024-04-20T21:32:16.848 INFO:tasks.workunit.client.0.smithi062.stdout:2/902: creat d7/da/d15/d2f/d38/db0/dc4/f125 x:0 0 0 2024-04-20T21:32:16.848 INFO:tasks.workunit.client.0.smithi062.stdout:6/737: fdatasync d0/d52/f98 0 2024-04-20T21:32:16.849 INFO:tasks.workunit.client.1.smithi189.stdout:9/568: readlink d9/d12/d3b/db1/d1a/l9b 0 2024-04-20T21:32:16.852 INFO:tasks.workunit.client.0.smithi062.stdout:6/738: fsync d0/d9/d11/ff7 0 2024-04-20T21:32:16.854 INFO:tasks.workunit.client.1.smithi189.stdout:8/710: truncate da/df/d20/d2c/f92 1463667 0 2024-04-20T21:32:16.856 INFO:tasks.workunit.client.0.smithi062.stdout:1/938: truncate dc/d27/d32/f89 2818573 0 2024-04-20T21:32:16.857 INFO:tasks.workunit.client.1.smithi189.stdout:9/569: write d9/dc/d17/d95/da6/fb3 [678469,56074] 0 2024-04-20T21:32:16.858 INFO:tasks.workunit.client.1.smithi189.stdout:6/660: dwrite d8/d2b/d49/d54/d6e/fb4 [0,4194304] 0 2024-04-20T21:32:16.860 INFO:tasks.workunit.client.1.smithi189.stdout:8/711: stat da/df/d13/dd6/d59/ffb 0 2024-04-20T21:32:16.861 INFO:tasks.workunit.client.0.smithi062.stdout:2/903: unlink d7/da/d15/d1b/f80 0 2024-04-20T21:32:16.861 INFO:tasks.workunit.client.1.smithi189.stdout:5/619: mkdir d7/d9/d31/d36/d5f/dab/dc8 0 2024-04-20T21:32:16.863 INFO:tasks.workunit.client.0.smithi062.stdout:6/739: creat d0/d9/d11/d22/dd5/d67/ffa x:0 0 0 2024-04-20T21:32:16.864 INFO:tasks.workunit.client.1.smithi189.stdout:5/620: readlink d7/d14/d18/l1e 0 2024-04-20T21:32:16.865 INFO:tasks.workunit.client.0.smithi062.stdout:6/740: chown d0/d9/d11 10677 1 2024-04-20T21:32:16.865 INFO:tasks.workunit.client.1.smithi189.stdout:9/570: chown d9/d12/f89 0 1 2024-04-20T21:32:16.866 INFO:tasks.workunit.client.1.smithi189.stdout:8/712: symlink da/de/d1e/lff 0 2024-04-20T21:32:16.866 INFO:tasks.workunit.client.1.smithi189.stdout:0/741: getdents d3/d26/d88 0 2024-04-20T21:32:16.868 INFO:tasks.workunit.client.1.smithi189.stdout:7/685: mkdir d5/db4/d71/de4 0 2024-04-20T21:32:16.868 INFO:tasks.workunit.client.1.smithi189.stdout:8/713: readlink l8 0 2024-04-20T21:32:16.869 INFO:tasks.workunit.client.0.smithi062.stdout:1/939: mknod dc/d15/d20/d28/d83/dc0/d132/d149/c159 0 2024-04-20T21:32:16.871 INFO:tasks.workunit.client.1.smithi189.stdout:1/599: dread d8/d21/fb3 [0,4194304] 0 2024-04-20T21:32:16.871 INFO:tasks.workunit.client.0.smithi062.stdout:2/904: symlink d7/da/d15/d2f/d93/l126 0 2024-04-20T21:32:16.871 INFO:tasks.workunit.client.1.smithi189.stdout:6/661: read d8/d10/f3f [3702050,1743] 0 2024-04-20T21:32:16.872 INFO:tasks.workunit.client.1.smithi189.stdout:7/686: chown d5/f47 1319999 1 2024-04-20T21:32:16.872 INFO:tasks.workunit.client.1.smithi189.stdout:8/714: readlink da/df/d3a/l7c 0 2024-04-20T21:32:16.873 INFO:tasks.workunit.client.0.smithi062.stdout:6/741: creat d0/d9/d11/d22/dd5/d67/ffb x:0 0 0 2024-04-20T21:32:16.878 INFO:tasks.workunit.client.1.smithi189.stdout:3/630: creat d2/d3/d2c/fda x:0 0 0 2024-04-20T21:32:16.881 INFO:tasks.workunit.client.0.smithi062.stdout:2/905: write d7/da/d15/d2f/d38/d10f/d107/dca/dd8/ddc/f116 [1025822,90859] 0 2024-04-20T21:32:16.881 INFO:tasks.workunit.client.0.smithi062.stdout:7/907: write d7/d63/f7a [380391,67725] 0 2024-04-20T21:32:16.882 INFO:tasks.workunit.client.1.smithi189.stdout:3/631: stat d2/d8/le 0 2024-04-20T21:32:16.885 INFO:tasks.workunit.client.0.smithi062.stdout:8/906: link d7/d25/d3b/d58/d8b/d96/l9b d7/d25/d3b/d58/d8b/d24/d46/l12a 0 2024-04-20T21:32:16.888 INFO:tasks.workunit.client.0.smithi062.stdout:6/742: creat d0/dc7/d7c/dc5/ffc x:0 0 0 2024-04-20T21:32:16.889 INFO:tasks.workunit.client.0.smithi062.stdout:0/841: getdents d7/d1c/d15/d17 0 2024-04-20T21:32:16.893 INFO:tasks.workunit.client.0.smithi062.stdout:5/883: dwrite d8/d38/d7a/d4c/d7d/d9d/dac/d2f/fea [0,4194304] 0 2024-04-20T21:32:16.895 INFO:tasks.workunit.client.0.smithi062.stdout:7/908: creat d7/d3e/d33/d3d/d4d/f13e x:0 0 0 2024-04-20T21:32:16.897 INFO:tasks.workunit.client.0.smithi062.stdout:2/906: truncate d7/da/d15/d2f/d97/fd9 1026765 0 2024-04-20T21:32:16.898 INFO:tasks.workunit.client.0.smithi062.stdout:0/842: read - d7/d20/d62/de3/f10a zero size 2024-04-20T21:32:16.899 INFO:tasks.workunit.client.1.smithi189.stdout:4/591: dwrite d2/d22/f30 [0,4194304] 0 2024-04-20T21:32:16.903 INFO:tasks.workunit.client.0.smithi062.stdout:8/907: dread d7/d25/d3b/d58/d8b/d24/f50 [0,4194304] 0 2024-04-20T21:32:16.908 INFO:tasks.workunit.client.0.smithi062.stdout:3/822: dread d0/d11/f56 [0,4194304] 0 2024-04-20T21:32:16.913 INFO:tasks.workunit.client.0.smithi062.stdout:0/843: truncate d7/d1c/d15/d17/d40/fff 80098 0 2024-04-20T21:32:16.913 INFO:tasks.workunit.client.0.smithi062.stdout:7/909: read d7/f2a [633965,77938] 0 2024-04-20T21:32:16.914 INFO:tasks.workunit.client.0.smithi062.stdout:4/941: dread da/dd/d15/d6d/df5/f102 [0,4194304] 0 2024-04-20T21:32:16.916 INFO:tasks.workunit.client.0.smithi062.stdout:0/844: chown d7/ddf 225 1 2024-04-20T21:32:16.921 INFO:tasks.workunit.client.1.smithi189.stdout:2/574: dread d1/d11/d58/f6e [0,4194304] 0 2024-04-20T21:32:16.922 INFO:tasks.workunit.client.0.smithi062.stdout:2/907: link d7/da/d15/d2f/f7e d7/da/d15/d2f/d38/d10f/db9/f127 0 2024-04-20T21:32:16.923 INFO:tasks.workunit.client.0.smithi062.stdout:6/743: dwrite d0/d52/da0/fcc [4194304,4194304] 0 2024-04-20T21:32:16.926 INFO:tasks.workunit.client.1.smithi189.stdout:1/600: mknod d8/df/d3a/cd1 0 2024-04-20T21:32:16.926 INFO:tasks.workunit.client.1.smithi189.stdout:9/571: mknod d9/dc/d17/cb6 0 2024-04-20T21:32:16.927 INFO:tasks.workunit.client.1.smithi189.stdout:5/621: dread d7/d9/d51/f72 [0,4194304] 0 2024-04-20T21:32:16.928 INFO:tasks.workunit.client.1.smithi189.stdout:7/687: fsync d5/d46/faa 0 2024-04-20T21:32:16.930 INFO:tasks.workunit.client.1.smithi189.stdout:6/662: mkdir d8/d10/d78/dd6 0 2024-04-20T21:32:16.931 INFO:tasks.workunit.client.1.smithi189.stdout:8/715: write da/df/d13/dd2/f8a [467128,32261] 0 2024-04-20T21:32:16.933 INFO:tasks.workunit.client.1.smithi189.stdout:0/742: dwrite d3/df/d28/d2b/f49 [0,4194304] 0 2024-04-20T21:32:16.933 INFO:tasks.workunit.client.1.smithi189.stdout:3/632: creat d2/d3/d36/da1/fdb x:0 0 0 2024-04-20T21:32:16.936 INFO:tasks.workunit.client.1.smithi189.stdout:2/575: rename d1/d17/d2f to d1/d11/d2d/dab 0 2024-04-20T21:32:16.937 INFO:tasks.workunit.client.1.smithi189.stdout:5/622: creat d7/fc9 x:0 0 0 2024-04-20T21:32:16.943 INFO:tasks.workunit.client.1.smithi189.stdout:3/633: mknod d2/d3/d36/cdc 0 2024-04-20T21:32:16.943 INFO:tasks.workunit.client.1.smithi189.stdout:9/572: symlink d9/dc/lb7 0 2024-04-20T21:32:16.943 INFO:tasks.workunit.client.1.smithi189.stdout:6/663: mkdir d8/d12/dd7 0 2024-04-20T21:32:16.944 INFO:tasks.workunit.client.1.smithi189.stdout:2/576: chown d1/d11/d2d/dab/d74/c9f 737711 1 2024-04-20T21:32:16.945 INFO:tasks.workunit.client.1.smithi189.stdout:0/743: truncate d3/df/d30/fd2 297505 0 2024-04-20T21:32:16.953 INFO:tasks.workunit.client.1.smithi189.stdout:9/573: readlink d9/d12/l23 0 2024-04-20T21:32:16.954 INFO:tasks.workunit.client.1.smithi189.stdout:5/623: write d7/d9/d31/d36/f38 [2718229,68982] 0 2024-04-20T21:32:16.954 INFO:tasks.workunit.client.1.smithi189.stdout:4/592: dread d2/d3a/f6f [0,4194304] 0 2024-04-20T21:32:16.956 INFO:tasks.workunit.client.1.smithi189.stdout:2/577: chown d1/d11/d2d/dab/d4a/d66/l68 81569509 1 2024-04-20T21:32:16.956 INFO:tasks.workunit.client.1.smithi189.stdout:9/574: truncate f4 4259484 0 2024-04-20T21:32:16.957 INFO:tasks.workunit.client.1.smithi189.stdout:9/575: stat d9/d12 0 2024-04-20T21:32:16.959 INFO:tasks.workunit.client.1.smithi189.stdout:0/744: rename d3/df/d5d/d64/fac to d3/db9/dbe/dd7/ff7 0 2024-04-20T21:32:16.964 INFO:tasks.workunit.client.1.smithi189.stdout:7/688: write d5/d46/d7d/d45/f48 [3619412,113689] 0 2024-04-20T21:32:16.965 INFO:tasks.workunit.client.1.smithi189.stdout:5/624: read - d7/d9/d51/d70/d95/fa1 zero size 2024-04-20T21:32:16.965 INFO:tasks.workunit.client.1.smithi189.stdout:8/716: truncate da/df/d20/f7d 258290 0 2024-04-20T21:32:16.966 INFO:tasks.workunit.client.1.smithi189.stdout:1/601: dwrite d8/f14 [0,4194304] 0 2024-04-20T21:32:16.966 INFO:tasks.workunit.client.1.smithi189.stdout:6/664: write d8/d12/d4e/f6f [337165,59393] 0 2024-04-20T21:32:16.972 INFO:tasks.workunit.client.1.smithi189.stdout:4/593: link d2/f8 d2/fd9 0 2024-04-20T21:32:16.973 INFO:tasks.workunit.client.1.smithi189.stdout:8/717: symlink da/d81/d91/l100 0 2024-04-20T21:32:16.974 INFO:tasks.workunit.client.1.smithi189.stdout:1/602: rmdir d8/d1e/d43/d58/d60 39 2024-04-20T21:32:16.977 INFO:tasks.workunit.client.1.smithi189.stdout:9/576: dread d9/dc/d17/f2f [4194304,4194304] 0 2024-04-20T21:32:16.978 INFO:tasks.workunit.client.1.smithi189.stdout:0/745: creat d3/df/d28/dc8/df1/ff8 x:0 0 0 2024-04-20T21:32:16.979 INFO:tasks.workunit.client.1.smithi189.stdout:6/665: mkdir d8/d12/d99/dd8 0 2024-04-20T21:32:16.979 INFO:tasks.workunit.client.1.smithi189.stdout:7/689: truncate d5/d21/f23 1879876 0 2024-04-20T21:32:16.980 INFO:tasks.workunit.client.1.smithi189.stdout:3/634: dwrite d2/d3/d2c/d31/d90/fd8 [0,4194304] 0 2024-04-20T21:32:16.980 INFO:tasks.workunit.client.1.smithi189.stdout:5/625: rename d7/d9/d30/l32 to d7/d9/d51/d70/lca 0 2024-04-20T21:32:16.982 INFO:tasks.workunit.client.1.smithi189.stdout:0/746: fsync d3/df/d4c/f69 0 2024-04-20T21:32:16.982 INFO:tasks.workunit.client.1.smithi189.stdout:6/666: dread - d8/da/dd/d35/fc8 zero size 2024-04-20T21:32:16.982 INFO:tasks.workunit.client.1.smithi189.stdout:4/594: mkdir d2/d20/dc6/dda 0 2024-04-20T21:32:16.983 INFO:tasks.workunit.client.1.smithi189.stdout:8/718: creat da/de/d1e/d21/d87/f101 x:0 0 0 2024-04-20T21:32:16.986 INFO:tasks.workunit.client.1.smithi189.stdout:1/603: dread d8/d1e/d43/d44/f47 [0,4194304] 0 2024-04-20T21:32:16.991 INFO:tasks.workunit.client.1.smithi189.stdout:2/578: truncate d1/d17/d37/f75 3061041 0 2024-04-20T21:32:16.992 INFO:tasks.workunit.client.1.smithi189.stdout:5/626: mkdir d7/d9/d31/d36/d5f/dab/dcb 0 2024-04-20T21:32:16.993 INFO:tasks.workunit.client.1.smithi189.stdout:4/595: mknod d2/d22/da9/cdb 0 2024-04-20T21:32:16.994 INFO:tasks.workunit.client.1.smithi189.stdout:1/604: rmdir d8/df 39 2024-04-20T21:32:16.996 INFO:tasks.workunit.client.1.smithi189.stdout:2/579: write d1/d11/d52/d6f/f5a [1204195,21489] 0 2024-04-20T21:32:16.997 INFO:tasks.workunit.client.1.smithi189.stdout:7/690: creat d5/db4/fe5 x:0 0 0 2024-04-20T21:32:16.999 INFO:tasks.workunit.client.1.smithi189.stdout:8/719: write da/df/d13/d63/f73 [1586185,6023] 0 2024-04-20T21:32:17.001 INFO:tasks.workunit.client.1.smithi189.stdout:2/580: chown d1/d11/d52/d8b/fa1 4216015 1 2024-04-20T21:32:17.002 INFO:tasks.workunit.client.1.smithi189.stdout:5/627: mknod d7/d9/d51/d70/d95/ccc 0 2024-04-20T21:32:17.003 INFO:tasks.workunit.client.1.smithi189.stdout:7/691: write d5/f7 [1764001,33555] 0 2024-04-20T21:32:17.004 INFO:tasks.workunit.client.1.smithi189.stdout:6/667: getdents d8/d2b/d49/d54/d67 0 2024-04-20T21:32:17.004 INFO:tasks.workunit.client.1.smithi189.stdout:4/596: unlink d2/d22/d87/ld5 0 2024-04-20T21:32:17.007 INFO:tasks.workunit.client.1.smithi189.stdout:6/668: rmdir d8/da/dd/d35 39 2024-04-20T21:32:17.008 INFO:tasks.workunit.client.1.smithi189.stdout:5/628: mknod d7/d9/d31/d36/ccd 0 2024-04-20T21:32:17.012 INFO:tasks.workunit.client.1.smithi189.stdout:2/581: creat d1/d17/d78/fac x:0 0 0 2024-04-20T21:32:17.014 INFO:tasks.workunit.client.1.smithi189.stdout:7/692: symlink d5/db4/d71/le6 0 2024-04-20T21:32:17.014 INFO:tasks.workunit.client.1.smithi189.stdout:8/720: dread da/d1f/fa1 [0,4194304] 0 2024-04-20T21:32:17.017 INFO:tasks.workunit.client.1.smithi189.stdout:6/669: fdatasync d8/d12/d30/d47/da0/fcc 0 2024-04-20T21:32:17.017 INFO:tasks.workunit.client.1.smithi189.stdout:2/582: symlink d1/d11/d52/lad 0 2024-04-20T21:32:17.023 INFO:tasks.workunit.client.1.smithi189.stdout:8/721: stat da/df/d3c/d50/d6c/dc7 0 2024-04-20T21:32:17.025 INFO:tasks.workunit.client.1.smithi189.stdout:5/629: dread d7/d9/d31/d36/f54 [0,4194304] 0 2024-04-20T21:32:17.027 INFO:tasks.workunit.client.1.smithi189.stdout:2/583: stat d1/d11/d2d/dab/d74/f9b 0 2024-04-20T21:32:17.030 INFO:tasks.workunit.client.1.smithi189.stdout:0/747: dread d3/df/d4c/f69 [0,4194304] 0 2024-04-20T21:32:17.030 INFO:tasks.workunit.client.1.smithi189.stdout:7/693: mkdir d5/d1d/d64/db9/dcf/de7 0 2024-04-20T21:32:17.034 INFO:tasks.workunit.client.1.smithi189.stdout:9/577: dwrite d9/d12/d3b/f46 [0,4194304] 0 2024-04-20T21:32:17.035 INFO:tasks.workunit.client.1.smithi189.stdout:6/670: dread d8/da/dd/d35/d6d/d77/f93 [0,4194304] 0 2024-04-20T21:32:17.035 INFO:tasks.workunit.client.1.smithi189.stdout:0/748: rmdir d3/df 39 2024-04-20T21:32:17.036 INFO:tasks.workunit.client.1.smithi189.stdout:3/635: dwrite d2/d3/f52 [0,4194304] 0 2024-04-20T21:32:17.037 INFO:tasks.workunit.client.1.smithi189.stdout:2/584: symlink d1/d11/d2d/dab/lae 0 2024-04-20T21:32:17.037 INFO:tasks.workunit.client.1.smithi189.stdout:4/597: dwrite d2/d22/d87/fa2 [0,4194304] 0 2024-04-20T21:32:17.037 INFO:tasks.workunit.client.1.smithi189.stdout:8/722: write da/df/d3c/d68/d8b/fc1 [657383,23493] 0 2024-04-20T21:32:17.039 INFO:tasks.workunit.client.1.smithi189.stdout:6/671: write d8/d12/f9d [2639321,56770] 0 2024-04-20T21:32:17.043 INFO:tasks.workunit.client.1.smithi189.stdout:5/630: getdents d7/d9/d30/d55 0 2024-04-20T21:32:17.044 INFO:tasks.workunit.client.1.smithi189.stdout:0/749: chown d3/df/d28/l3f 1493 1 2024-04-20T21:32:17.044 INFO:tasks.workunit.client.1.smithi189.stdout:3/636: chown d2/d3/d36/da1/fa4 3535832 1 2024-04-20T21:32:17.049 INFO:tasks.workunit.client.1.smithi189.stdout:4/598: write d2/d20/faf [714624,113768] 0 2024-04-20T21:32:17.050 INFO:tasks.workunit.client.1.smithi189.stdout:5/631: truncate d7/d9/d28/da3/fc3 951063 0 2024-04-20T21:32:17.052 INFO:tasks.workunit.client.1.smithi189.stdout:2/585: mkdir d1/d11/d58/d6b/daf 0 2024-04-20T21:32:17.058 INFO:tasks.workunit.client.1.smithi189.stdout:8/723: creat da/df/d3c/d68/f102 x:0 0 0 2024-04-20T21:32:17.059 INFO:tasks.workunit.client.1.smithi189.stdout:0/750: mkdir d3/db9/dbe/df9 0 2024-04-20T21:32:17.061 INFO:tasks.workunit.client.1.smithi189.stdout:3/637: symlink d2/d1c/d5c/dca/ldd 0 2024-04-20T21:32:17.069 INFO:tasks.workunit.client.1.smithi189.stdout:7/694: dwrite d5/d21/d25/fde [0,4194304] 0 2024-04-20T21:32:17.071 INFO:tasks.workunit.client.1.smithi189.stdout:2/586: rename d1/d11/d58/fa0 to d1/d17/d37/d89/fb0 0 2024-04-20T21:32:17.072 INFO:tasks.workunit.client.1.smithi189.stdout:3/638: dread - d2/d3/d65/f8b zero size 2024-04-20T21:32:17.072 INFO:tasks.workunit.client.0.smithi062.stdout:5/884: write d8/f4f [1229400,19666] 0 2024-04-20T21:32:17.073 INFO:tasks.workunit.client.0.smithi062.stdout:1/940: dwrite dc/d15/d75/d96/dd4/d3a/da2/de4/f65 [0,4194304] 0 2024-04-20T21:32:17.074 INFO:tasks.workunit.client.0.smithi062.stdout:7/910: write d7/d3e/d33/d3d/d4d/fa4 [4944992,58809] 0 2024-04-20T21:32:17.077 INFO:tasks.workunit.client.0.smithi062.stdout:8/908: getdents d7/d25/d3b/d58/d8b 0 2024-04-20T21:32:17.079 INFO:tasks.workunit.client.1.smithi189.stdout:7/695: rmdir d5/d21/d4a/d52 39 2024-04-20T21:32:17.081 INFO:tasks.workunit.client.0.smithi062.stdout:4/942: truncate da/dd/df/f17 2908297 0 2024-04-20T21:32:17.082 INFO:tasks.workunit.client.1.smithi189.stdout:4/599: dwrite d2/f2c [0,4194304] 0 2024-04-20T21:32:17.085 INFO:tasks.workunit.client.1.smithi189.stdout:6/672: dwrite d8/d2b/d49/f60 [0,4194304] 0 2024-04-20T21:32:17.086 INFO:tasks.workunit.client.1.smithi189.stdout:7/696: stat d5/f57 0 2024-04-20T21:32:17.088 INFO:tasks.workunit.client.1.smithi189.stdout:2/587: write d1/d11/d52/d8b/f8d [1793730,111994] 0 2024-04-20T21:32:17.088 INFO:tasks.workunit.client.0.smithi062.stdout:0/845: mkdir d7/d1c/d15/d17/d1b/d106/df1/d10e 0 2024-04-20T21:32:17.089 INFO:tasks.workunit.client.0.smithi062.stdout:2/908: creat d7/da/d15/d2f/d97/f128 x:0 0 0 2024-04-20T21:32:17.089 INFO:tasks.workunit.client.1.smithi189.stdout:5/632: dread d7/d9/d51/f72 [0,4194304] 0 2024-04-20T21:32:17.093 INFO:tasks.workunit.client.0.smithi062.stdout:0/846: stat d7/d1c/d15/d17/d1b/d106/db1 0 2024-04-20T21:32:17.097 INFO:tasks.workunit.client.1.smithi189.stdout:3/639: write d2/d1c/fcd [1197731,74399] 0 2024-04-20T21:32:17.101 INFO:tasks.workunit.client.1.smithi189.stdout:4/600: mkdir d2/d3a/ddc 0 2024-04-20T21:32:17.104 INFO:tasks.workunit.client.0.smithi062.stdout:4/943: creat da/dd/d15/daf/f143 x:0 0 0 2024-04-20T21:32:17.104 INFO:tasks.workunit.client.0.smithi062.stdout:6/744: truncate d0/d9/d11/d22/dd5/d67/fca 2089871 0 2024-04-20T21:32:17.108 INFO:tasks.workunit.client.1.smithi189.stdout:5/633: creat d7/d35/fce x:0 0 0 2024-04-20T21:32:17.108 INFO:tasks.workunit.client.0.smithi062.stdout:2/909: symlink d7/da/d15/d1b/d3d/d84/l129 0 2024-04-20T21:32:17.110 INFO:tasks.workunit.client.1.smithi189.stdout:0/751: dwrite d3/df/d30/fef [0,4194304] 0 2024-04-20T21:32:17.111 INFO:tasks.workunit.client.1.smithi189.stdout:2/588: truncate d1/d11/d2d/fa6 4655628 0 2024-04-20T21:32:17.111 INFO:tasks.workunit.client.0.smithi062.stdout:5/885: link d8/d38/f70 d8/d38/d7a/d4c/d7d/f127 0 2024-04-20T21:32:17.113 INFO:tasks.workunit.client.1.smithi189.stdout:4/601: creat d2/dd/d37/dcd/d6a/fdd x:0 0 0 2024-04-20T21:32:17.113 INFO:tasks.workunit.client.1.smithi189.stdout:5/634: read - d7/fc9 zero size 2024-04-20T21:32:17.115 INFO:tasks.workunit.client.1.smithi189.stdout:0/752: dread - d3/d26/d88/fc0 zero size 2024-04-20T21:32:17.118 INFO:tasks.workunit.client.1.smithi189.stdout:9/578: dread d9/d12/d3f/f68 [0,4194304] 0 2024-04-20T21:32:17.118 INFO:tasks.workunit.client.1.smithi189.stdout:4/602: chown d2/laa 0 1 2024-04-20T21:32:17.119 INFO:tasks.workunit.client.0.smithi062.stdout:6/745: truncate d0/d9/d11/d22/dd5/f30 230665 0 2024-04-20T21:32:17.120 INFO:tasks.workunit.client.0.smithi062.stdout:2/910: mknod d7/da/d15/d1b/df1/d10e/c12a 0 2024-04-20T21:32:17.120 INFO:tasks.workunit.client.1.smithi189.stdout:4/603: readlink d2/la 0 2024-04-20T21:32:17.131 INFO:tasks.workunit.client.1.smithi189.stdout:8/724: write da/df/d20/d2c/d9a/dce/fd9 [4215195,49260] 0 2024-04-20T21:32:17.131 INFO:tasks.workunit.client.1.smithi189.stdout:0/753: truncate d3/df/d28/d2b/f86 327451 0 2024-04-20T21:32:17.132 INFO:tasks.workunit.client.0.smithi062.stdout:4/944: link da/dd/df/d109/l118 da/dd/d15/d50/d64/d3d/da1/l144 0 2024-04-20T21:32:17.135 INFO:tasks.workunit.client.1.smithi189.stdout:4/604: symlink d2/dd/d37/dcd/d94/lde 0 2024-04-20T21:32:17.136 INFO:tasks.workunit.client.0.smithi062.stdout:2/911: getdents d7/da/d15/d2f/d38/d10f/d121 0 2024-04-20T21:32:17.140 INFO:tasks.workunit.client.0.smithi062.stdout:6/746: creat d0/d52/ffd x:0 0 0 2024-04-20T21:32:17.143 INFO:tasks.workunit.client.1.smithi189.stdout:2/589: dwrite d1/d17/d78/fac [0,4194304] 0 2024-04-20T21:32:17.143 INFO:tasks.workunit.client.1.smithi189.stdout:5/635: getdents d7/d9/d31/d36/d5f 0 2024-04-20T21:32:17.147 INFO:tasks.workunit.client.1.smithi189.stdout:4/605: rename d2/d64/c84 to d2/d20/cdf 0 2024-04-20T21:32:17.149 INFO:tasks.workunit.client.1.smithi189.stdout:8/725: creat da/df/d3c/deb/f103 x:0 0 0 2024-04-20T21:32:17.153 INFO:tasks.workunit.client.0.smithi062.stdout:6/747: stat d0/d9/d11/d22/dd5/d67/d75/dd2/ld8 0 2024-04-20T21:32:17.155 INFO:tasks.workunit.client.1.smithi189.stdout:3/640: dread d2/d3/d2c/f6d [0,4194304] 0 2024-04-20T21:32:17.156 INFO:tasks.workunit.client.1.smithi189.stdout:9/579: getdents d9/dc/d17/d95/da0/d52 0 2024-04-20T21:32:17.157 INFO:tasks.workunit.client.1.smithi189.stdout:2/590: rename d1/d11/fa8 to d1/d11/d52/d6f/fb1 0 2024-04-20T21:32:17.157 INFO:tasks.workunit.client.0.smithi062.stdout:7/911: dwrite d7/d63/fa7 [0,4194304] 0 2024-04-20T21:32:17.158 INFO:tasks.workunit.client.1.smithi189.stdout:5/636: creat d7/d9/d28/db1/fcf x:0 0 0 2024-04-20T21:32:17.159 INFO:tasks.workunit.client.0.smithi062.stdout:1/941: dwrite dc/d15/df6/f129 [0,4194304] 0 2024-04-20T21:32:17.161 INFO:tasks.workunit.client.1.smithi189.stdout:3/641: readlink d2/d3/d2c/d30/d3f/l49 0 2024-04-20T21:32:17.162 INFO:tasks.workunit.client.1.smithi189.stdout:0/754: link d3/df/d4c/dd4/lc2 d3/db9/dbe/lfa 0 2024-04-20T21:32:17.162 INFO:tasks.workunit.client.0.smithi062.stdout:2/912: getdents d7/da/d15/d1b/d24/dee/d117 0 2024-04-20T21:32:17.163 INFO:tasks.workunit.client.1.smithi189.stdout:5/637: chown d7/d9/d51/d70/c85 113435395 1 2024-04-20T21:32:17.166 INFO:tasks.workunit.client.1.smithi189.stdout:8/726: creat da/df/d3c/d50/ddc/df2/f104 x:0 0 0 2024-04-20T21:32:17.166 INFO:tasks.workunit.client.1.smithi189.stdout:9/580: write d9/dc/d69/f7f [354935,102710] 0 2024-04-20T21:32:17.167 INFO:tasks.workunit.client.0.smithi062.stdout:8/909: dwrite d7/d25/d5c/f72 [0,4194304] 0 2024-04-20T21:32:17.170 INFO:tasks.workunit.client.0.smithi062.stdout:2/913: chown d7/da/d15/d1b/d3d/d84/l129 620648 1 2024-04-20T21:32:17.172 INFO:tasks.workunit.client.0.smithi062.stdout:4/945: dread - da/dd/df/d27/d7a/f128 zero size 2024-04-20T21:32:17.177 INFO:tasks.workunit.client.0.smithi062.stdout:5/886: dwrite d8/d38/d7a/f36 [0,4194304] 0 2024-04-20T21:32:17.183 INFO:tasks.workunit.client.0.smithi062.stdout:0/847: dread d7/d1c/d15/d17/fd0 [0,4194304] 0 2024-04-20T21:32:17.183 INFO:tasks.workunit.client.1.smithi189.stdout:0/755: read d3/df/d4c/f7e [653723,10985] 0 2024-04-20T21:32:17.185 INFO:tasks.workunit.client.1.smithi189.stdout:5/638: truncate d7/d9/d28/f9f 421625 0 2024-04-20T21:32:17.185 INFO:tasks.workunit.client.0.smithi062.stdout:6/748: rename d0/d52/ffd to d0/d9/d11/d7b/d99/ddd/ffe 0 2024-04-20T21:32:17.186 INFO:tasks.workunit.client.1.smithi189.stdout:3/642: link l1 d2/d1c/dc1/lde 0 2024-04-20T21:32:17.190 INFO:tasks.workunit.client.0.smithi062.stdout:7/912: mkdir d7/d3e/d33/d3d/d4d/d54/d13f 0 2024-04-20T21:32:17.192 INFO:tasks.workunit.client.0.smithi062.stdout:8/910: creat d7/d25/d5c/f12b x:0 0 0 2024-04-20T21:32:17.192 INFO:tasks.workunit.client.1.smithi189.stdout:2/591: dwrite d1/d11/d2d/dab/d55/f81 [0,4194304] 0 2024-04-20T21:32:17.195 INFO:tasks.workunit.client.1.smithi189.stdout:5/639: write d7/d9/d31/f88 [867702,29401] 0 2024-04-20T21:32:17.196 INFO:tasks.workunit.client.0.smithi062.stdout:5/887: mknod d8/d38/d7a/d4c/dcf/c128 0 2024-04-20T21:32:17.198 INFO:tasks.workunit.client.1.smithi189.stdout:5/640: readlink l0 0 2024-04-20T21:32:17.199 INFO:tasks.workunit.client.1.smithi189.stdout:3/643: mkdir d2/d1c/dc1/dc5/ddf 0 2024-04-20T21:32:17.201 INFO:tasks.workunit.client.0.smithi062.stdout:2/914: symlink d7/da/d15/d1b/l12b 0 2024-04-20T21:32:17.201 INFO:tasks.workunit.client.1.smithi189.stdout:5/641: dread - d7/d14/d18/f94 zero size 2024-04-20T21:32:17.205 INFO:tasks.workunit.client.1.smithi189.stdout:8/727: dwrite da/df/d3c/d68/d8b/dac/fe7 [0,4194304] 0 2024-04-20T21:32:17.205 INFO:tasks.workunit.client.1.smithi189.stdout:2/592: truncate d1/d11/d2d/f63 4427207 0 2024-04-20T21:32:17.207 INFO:tasks.workunit.client.1.smithi189.stdout:2/593: fdatasync d1/f8c 0 2024-04-20T21:32:17.208 INFO:tasks.workunit.client.0.smithi062.stdout:2/915: chown d7/da/d15/d1b/d3d/d84/fa9 306729 1 2024-04-20T21:32:17.209 INFO:tasks.workunit.client.0.smithi062.stdout:7/913: dread - d7/d3e/d33/d3d/d5e/d72/fd3 zero size 2024-04-20T21:32:17.214 INFO:tasks.workunit.client.1.smithi189.stdout:5/642: unlink d7/d9/d51/d70/d99/f9a 0 2024-04-20T21:32:17.215 INFO:tasks.workunit.client.0.smithi062.stdout:6/749: dwrite d0/fb6 [4194304,4194304] 0 2024-04-20T21:32:17.215 INFO:tasks.workunit.client.0.smithi062.stdout:5/888: creat d8/d1e/d2c/d6c/df9/f129 x:0 0 0 2024-04-20T21:32:17.217 INFO:tasks.workunit.client.1.smithi189.stdout:4/606: write d2/f5e [1747169,55643] 0 2024-04-20T21:32:17.221 INFO:tasks.workunit.client.1.smithi189.stdout:2/594: creat d1/d11/d2d/d4f/fb2 x:0 0 0 2024-04-20T21:32:17.223 INFO:tasks.workunit.client.0.smithi062.stdout:3/823: sync 2024-04-20T21:32:17.223 INFO:tasks.workunit.client.0.smithi062.stdout:1/942: write dc/d27/d32/f51 [2954456,44220] 0 2024-04-20T21:32:17.227 INFO:tasks.workunit.client.0.smithi062.stdout:3/824: chown d0/d1/cad 115348941 1 2024-04-20T21:32:17.230 INFO:tasks.workunit.client.1.smithi189.stdout:5/643: fsync d7/d9/d30/d5a/d9d/fac 0 2024-04-20T21:32:17.230 INFO:tasks.workunit.client.1.smithi189.stdout:4/607: symlink d2/d7f/le0 0 2024-04-20T21:32:17.230 INFO:tasks.workunit.client.1.smithi189.stdout:2/595: rmdir d1/d17/d78 39 2024-04-20T21:32:17.230 INFO:tasks.workunit.client.1.smithi189.stdout:9/581: write d9/dc/d17/d7e/f8d [342560,18146] 0 2024-04-20T21:32:17.230 INFO:tasks.workunit.client.0.smithi062.stdout:7/914: mkdir d7/d140 0 2024-04-20T21:32:17.231 INFO:tasks.workunit.client.1.smithi189.stdout:5/644: mknod d7/d14/d4e/d5b/cd0 0 2024-04-20T21:32:17.232 INFO:tasks.workunit.client.0.smithi062.stdout:3/825: chown d0/d1b/f87 0 1 2024-04-20T21:32:17.232 INFO:tasks.workunit.client.0.smithi062.stdout:0/848: dwrite d7/d20/d62/d6f/f107 [0,4194304] 0 2024-04-20T21:32:17.234 INFO:tasks.workunit.client.1.smithi189.stdout:2/596: write d1/d11/d2d/dab/d4a/f7b [713138,49841] 0 2024-04-20T21:32:17.239 INFO:tasks.workunit.client.1.smithi189.stdout:9/582: chown d9/d12/d3f/d91/fa3 3998852 1 2024-04-20T21:32:17.240 INFO:tasks.workunit.client.0.smithi062.stdout:1/943: mknod dc/d15/d75/d94/d6f/dd2/de3/d121/d14d/d14e/c15a 0 2024-04-20T21:32:17.240 INFO:tasks.workunit.client.1.smithi189.stdout:5/645: mknod d7/d9/d30/d55/d33/cd1 0 2024-04-20T21:32:17.243 INFO:tasks.workunit.client.0.smithi062.stdout:2/916: link d7/da/d15/d2f/d97/f128 d7/da/d15/d2f/d38/d51/f12c 0 2024-04-20T21:32:17.243 INFO:tasks.workunit.client.0.smithi062.stdout:3/826: write d0/d1/d3c/d43/d72/dd3/d7b/fc2 [938652,17955] 0 2024-04-20T21:32:17.244 INFO:tasks.workunit.client.1.smithi189.stdout:0/756: dread d3/fd [4194304,4194304] 0 2024-04-20T21:32:17.244 INFO:tasks.workunit.client.0.smithi062.stdout:7/915: truncate d7/fa0 5294704 0 2024-04-20T21:32:17.247 INFO:tasks.workunit.client.1.smithi189.stdout:9/583: mknod d9/d12/d50/cb8 0 2024-04-20T21:32:17.248 INFO:tasks.workunit.client.1.smithi189.stdout:2/597: creat d1/d11/d2d/dab/d74/fb3 x:0 0 0 2024-04-20T21:32:17.250 INFO:tasks.workunit.client.0.smithi062.stdout:0/849: truncate d7/d1c/d15/d17/d26/d76/fa6 1277456 0 2024-04-20T21:32:17.251 INFO:tasks.workunit.client.1.smithi189.stdout:5/646: symlink d7/d9/d30/d55/d61/d96/ld2 0 2024-04-20T21:32:17.251 INFO:tasks.workunit.client.0.smithi062.stdout:2/917: fsync d7/da/d15/d2f/d38/d10f/f2a 0 2024-04-20T21:32:17.251 INFO:tasks.workunit.client.0.smithi062.stdout:3/827: mknod d0/d1/db7/c10f 0 2024-04-20T21:32:17.254 INFO:tasks.workunit.client.0.smithi062.stdout:1/944: fdatasync dc/d70/f90 0 2024-04-20T21:32:17.254 INFO:tasks.workunit.client.1.smithi189.stdout:5/647: read - d7/d35/d48/f6a zero size 2024-04-20T21:32:17.256 INFO:tasks.workunit.client.1.smithi189.stdout:2/598: rename d1/d11/d52/d6f to d1/d11/d2d/d4f/db4 0 2024-04-20T21:32:17.257 INFO:tasks.workunit.client.1.smithi189.stdout:9/584: mknod d9/d12/d3f/cb9 0 2024-04-20T21:32:17.259 INFO:tasks.workunit.client.0.smithi062.stdout:3/828: fdatasync d0/f53 0 2024-04-20T21:32:17.264 INFO:tasks.workunit.client.1.smithi189.stdout:5/648: creat d7/d9/d30/d55/d61/d96/fd3 x:0 0 0 2024-04-20T21:32:17.265 INFO:tasks.workunit.client.0.smithi062.stdout:1/945: rename dc/d15/d20/d28/d83/f10c to dc/d15/d20/d28/d83/dce/f15b 0 2024-04-20T21:32:17.266 INFO:tasks.workunit.client.0.smithi062.stdout:7/916: dwrite d7/d3e/d33/d3d/d5e/d72/fb2 [4194304,4194304] 0 2024-04-20T21:32:17.269 INFO:tasks.workunit.client.1.smithi189.stdout:2/599: creat d1/d11/d2d/d4f/d8e/fb5 x:0 0 0 2024-04-20T21:32:17.284 INFO:tasks.workunit.client.0.smithi062.stdout:8/911: write d7/d9/f41 [955575,95681] 0 2024-04-20T21:32:17.286 INFO:tasks.workunit.client.0.smithi062.stdout:4/946: dwrite da/dd/d15/d50/fd1 [0,4194304] 0 2024-04-20T21:32:17.289 INFO:tasks.workunit.client.0.smithi062.stdout:1/946: link dc/d15/d75/d96/dd4/d3a/f4f dc/d15/d75/d96/dd4/d3a/da2/de4/d84/dbe/d10d/d110/f15c 0 2024-04-20T21:32:17.293 INFO:tasks.workunit.client.1.smithi189.stdout:9/585: dwrite d9/d12/d3f/d72/f85 [0,4194304] 0 2024-04-20T21:32:17.293 INFO:tasks.workunit.client.1.smithi189.stdout:8/728: dread da/df/d13/d63/f73 [0,4194304] 0 2024-04-20T21:32:17.294 INFO:tasks.workunit.client.1.smithi189.stdout:0/757: dwrite d3/df/d4c/dd4/fcd [0,4194304] 0 2024-04-20T21:32:17.294 INFO:tasks.workunit.client.0.smithi062.stdout:1/947: chown dc/f53 910177437 1 2024-04-20T21:32:17.296 INFO:tasks.workunit.client.1.smithi189.stdout:3/644: dwrite d2/d3/d25/f67 [0,4194304] 0 2024-04-20T21:32:17.306 INFO:tasks.workunit.client.0.smithi062.stdout:8/912: dwrite d7/d9/f12 [0,4194304] 0 2024-04-20T21:32:17.307 INFO:tasks.workunit.client.1.smithi189.stdout:9/586: mknod d9/dc/d17/d7e/cba 0 2024-04-20T21:32:17.309 INFO:tasks.workunit.client.0.smithi062.stdout:1/948: rename dc/d15/d75/d96/dd4/f9c to dc/d15/d20/d28/d76/f15d 0 2024-04-20T21:32:17.315 INFO:tasks.workunit.client.1.smithi189.stdout:3/645: rename d2/d3/d2c/d30/d3f/d9a to d2/d3/d25/de0 0 2024-04-20T21:32:17.316 INFO:tasks.workunit.client.1.smithi189.stdout:4/608: write d2/dd/d11/f7e [1027095,105354] 0 2024-04-20T21:32:17.316 INFO:tasks.workunit.client.0.smithi062.stdout:6/750: truncate d0/d52/fb0 188773 0 2024-04-20T21:32:17.317 INFO:tasks.workunit.client.1.smithi189.stdout:3/646: stat d2/l1e 0 2024-04-20T21:32:17.317 INFO:tasks.workunit.client.0.smithi062.stdout:2/918: dread d7/f83 [0,4194304] 0 2024-04-20T21:32:17.319 INFO:tasks.workunit.client.1.smithi189.stdout:4/609: readlink d2/d22/d87/l40 0 2024-04-20T21:32:17.321 INFO:tasks.workunit.client.0.smithi062.stdout:1/949: write dc/d27/d32/f51 [3799606,6249] 0 2024-04-20T21:32:17.324 INFO:tasks.workunit.client.0.smithi062.stdout:2/919: chown d7/da/d15/c50 948502 1 2024-04-20T21:32:17.327 INFO:tasks.workunit.client.0.smithi062.stdout:5/889: dwrite d8/d38/d7a/d4c/fc1 [0,4194304] 0 2024-04-20T21:32:17.328 INFO:tasks.workunit.client.1.smithi189.stdout:3/647: read d2/d3/d2c/f6b [1533907,88991] 0 2024-04-20T21:32:17.330 INFO:tasks.workunit.client.0.smithi062.stdout:1/950: truncate dc/d15/d20/d28/d83/dc0/fc2 1775815 0 2024-04-20T21:32:17.333 INFO:tasks.workunit.client.1.smithi189.stdout:4/610: dread - d2/d22/d57/f99 zero size 2024-04-20T21:32:17.334 INFO:tasks.workunit.client.0.smithi062.stdout:0/850: write d7/d1c/d15/d17/d26/fea [374898,99860] 0 2024-04-20T21:32:17.337 INFO:tasks.workunit.client.1.smithi189.stdout:3/648: symlink d2/d3/d25/de0/db0/le1 0 2024-04-20T21:32:17.339 INFO:tasks.workunit.client.1.smithi189.stdout:3/649: readlink d2/d8/lc7 0 2024-04-20T21:32:17.341 INFO:tasks.workunit.client.0.smithi062.stdout:8/913: rename d7/d25/dc5/dd3 to d7/d25/d3b/d12c 0 2024-04-20T21:32:17.341 INFO:tasks.workunit.client.1.smithi189.stdout:2/600: getdents d1/d11/d2d/d4f/d8e 0 2024-04-20T21:32:17.342 INFO:tasks.workunit.client.1.smithi189.stdout:4/611: mkdir d2/d22/d87/de1 0 2024-04-20T21:32:17.342 INFO:tasks.workunit.client.0.smithi062.stdout:7/917: write d7/d3e/d33/d3d/d4d/d54/dea/f93 [2117512,60239] 0 2024-04-20T21:32:17.346 INFO:tasks.workunit.client.0.smithi062.stdout:2/920: mknod d7/c12d 0 2024-04-20T21:32:17.348 INFO:tasks.workunit.client.0.smithi062.stdout:7/918: chown d7/d3e/d33/d3d/d4d/d54/d4f 96522862 1 2024-04-20T21:32:17.348 INFO:tasks.workunit.client.1.smithi189.stdout:3/650: unlink d2/d3/d2c/d30/fad 0 2024-04-20T21:32:17.350 INFO:tasks.workunit.client.0.smithi062.stdout:1/951: rename dc/d15/d75/d94/d6f to dc/d15/d75/d96/dd4/d3a/da2/de4/d84/dbe/d119/d15e 0 2024-04-20T21:32:17.352 INFO:tasks.workunit.client.1.smithi189.stdout:8/729: write da/df/d20/dc9/fd7 [2549684,58806] 0 2024-04-20T21:32:17.354 INFO:tasks.workunit.client.0.smithi062.stdout:8/914: unlink d7/f22 0 2024-04-20T21:32:17.354 INFO:tasks.workunit.client.0.smithi062.stdout:3/829: dwrite d0/d1/d10/f4e [0,4194304] 0 2024-04-20T21:32:17.354 INFO:tasks.workunit.client.0.smithi062.stdout:6/751: getdents d0/d9/d11/d7b/d99/ddd 0 2024-04-20T21:32:17.361 INFO:tasks.workunit.client.1.smithi189.stdout:1/605: sync 2024-04-20T21:32:17.361 INFO:tasks.workunit.client.1.smithi189.stdout:7/697: sync 2024-04-20T21:32:17.361 INFO:tasks.workunit.client.0.smithi062.stdout:4/947: dwrite da/d56/f74 [0,4194304] 0 2024-04-20T21:32:17.361 INFO:tasks.workunit.client.0.smithi062.stdout:1/952: read - dc/d15/d75/d96/dd4/d3a/f134 zero size 2024-04-20T21:32:17.362 INFO:tasks.workunit.client.1.smithi189.stdout:6/673: sync 2024-04-20T21:32:17.365 INFO:tasks.workunit.client.1.smithi189.stdout:0/758: dwrite d3/df/f91 [0,4194304] 0 2024-04-20T21:32:17.365 INFO:tasks.workunit.client.0.smithi062.stdout:3/830: rmdir d0/d1b/d74 39 2024-04-20T21:32:17.366 INFO:tasks.workunit.client.0.smithi062.stdout:1/953: dread - dc/d15/d75/d96/dd4/ff5 zero size 2024-04-20T21:32:17.377 INFO:tasks.workunit.client.1.smithi189.stdout:6/674: symlink d8/d2b/d49/ld9 0 2024-04-20T21:32:17.377 INFO:tasks.workunit.client.0.smithi062.stdout:5/890: dread d8/d1e/d2c/fa2 [0,4194304] 0 2024-04-20T21:32:17.380 INFO:tasks.workunit.client.0.smithi062.stdout:7/919: dwrite d7/d3e/d33/d3d/dad/f11e [0,4194304] 0 2024-04-20T21:32:17.382 INFO:tasks.workunit.client.1.smithi189.stdout:9/587: dwrite f7 [0,4194304] 0 2024-04-20T21:32:17.383 INFO:tasks.workunit.client.1.smithi189.stdout:0/759: symlink d3/db9/dbe/lfb 0 2024-04-20T21:32:17.384 INFO:tasks.workunit.client.0.smithi062.stdout:1/954: rename dc/d15/d75/d96/dd4/d3a/da2/de4/d7c/f116 to dc/df4/f15f 0 2024-04-20T21:32:17.384 INFO:tasks.workunit.client.0.smithi062.stdout:7/920: chown d7/d3e/d33/d3d/d5e/d72/l137 53253195 1 2024-04-20T21:32:17.385 INFO:tasks.workunit.client.1.smithi189.stdout:0/760: chown d3/df/d28/f61 0 1 2024-04-20T21:32:17.385 INFO:tasks.workunit.client.0.smithi062.stdout:5/891: dread - d8/d38/d7a/d4c/d7d/d9d/dac/fef zero size 2024-04-20T21:32:17.386 INFO:tasks.workunit.client.0.smithi062.stdout:3/831: getdents d0/d1/d10/d3a/d5e/ddb 0 2024-04-20T21:32:17.387 INFO:tasks.workunit.client.1.smithi189.stdout:6/675: write d8/f1a [1326470,92135] 0 2024-04-20T21:32:17.390 INFO:tasks.workunit.client.1.smithi189.stdout:1/606: rename d8/d21/d2c/d36 to d8/d1e/d43/d44/dd2 0 2024-04-20T21:32:17.392 INFO:tasks.workunit.client.1.smithi189.stdout:0/761: symlink d3/df/d4c/dd4/lfc 0 2024-04-20T21:32:17.393 INFO:tasks.workunit.client.1.smithi189.stdout:6/676: readlink d8/l18 0 2024-04-20T21:32:17.396 INFO:tasks.workunit.client.1.smithi189.stdout:1/607: symlink d8/d1e/d54/ld3 0 2024-04-20T21:32:17.398 INFO:tasks.workunit.client.0.smithi062.stdout:1/955: creat dc/d15/d75/d96/dd4/d3a/da2/de4/d84/dbe/d10d/f160 x:0 0 0 2024-04-20T21:32:17.398 INFO:tasks.workunit.client.1.smithi189.stdout:0/762: write d3/fe [5891482,111] 0 2024-04-20T21:32:17.398 INFO:tasks.workunit.client.1.smithi189.stdout:4/612: dwrite d2/dd/d37/dcd/fb3 [0,4194304] 0 2024-04-20T21:32:17.400 INFO:tasks.workunit.client.0.smithi062.stdout:7/921: rename d7/d3e/d33/d3d/d4d/d54/dea/f93 to d7/d3e/d33/d3d/d5e/db8/f141 0 2024-04-20T21:32:17.405 INFO:tasks.workunit.client.0.smithi062.stdout:7/922: write d7/d26/dac/d10a/f11c [927395,39932] 0 2024-04-20T21:32:17.406 INFO:tasks.workunit.client.1.smithi189.stdout:5/649: sync 2024-04-20T21:32:17.408 INFO:tasks.workunit.client.1.smithi189.stdout:1/608: write f6 [2882341,130363] 0 2024-04-20T21:32:17.409 INFO:tasks.workunit.client.0.smithi062.stdout:3/832: dwrite d0/fdc [0,4194304] 0 2024-04-20T21:32:17.410 INFO:tasks.workunit.client.0.smithi062.stdout:7/923: read - d7/d26/da8/f111 zero size 2024-04-20T21:32:17.412 INFO:tasks.workunit.client.1.smithi189.stdout:0/763: creat d3/df/d30/d45/dd3/ffd x:0 0 0 2024-04-20T21:32:17.420 INFO:tasks.workunit.client.1.smithi189.stdout:5/650: dread - d7/d14/d4e/d5b/fc2 zero size 2024-04-20T21:32:17.421 INFO:tasks.workunit.client.0.smithi062.stdout:7/924: mknod d7/d26/da8/c142 0 2024-04-20T21:32:17.421 INFO:tasks.workunit.client.0.smithi062.stdout:1/956: dwrite dc/d15/d20/fe6 [0,4194304] 0 2024-04-20T21:32:17.428 INFO:tasks.workunit.client.1.smithi189.stdout:1/609: fsync d8/f86 0 2024-04-20T21:32:17.429 INFO:tasks.workunit.client.1.smithi189.stdout:0/764: mkdir d3/df/d28/d2b/db1/dfe 0 2024-04-20T21:32:17.430 INFO:tasks.workunit.client.0.smithi062.stdout:0/851: dread d7/d1c/d15/d17/d1b/d106/db1/fd4 [0,4194304] 0 2024-04-20T21:32:17.432 INFO:tasks.workunit.client.1.smithi189.stdout:6/677: link d8/d2b/d49/l5d d8/da/dd/d2a/d3b/lda 0 2024-04-20T21:32:17.433 INFO:tasks.workunit.client.1.smithi189.stdout:0/765: write d3/d8c/f8d [6701173,27109] 0 2024-04-20T21:32:17.434 INFO:tasks.workunit.client.0.smithi062.stdout:1/957: read dc/d15/fdd [1969906,106668] 0 2024-04-20T21:32:17.436 INFO:tasks.workunit.client.1.smithi189.stdout:3/651: dwrite d2/f6 [4194304,4194304] 0 2024-04-20T21:32:17.440 INFO:tasks.workunit.client.1.smithi189.stdout:5/651: mkdir d7/d9/d51/d70/d99/dd4 0 2024-04-20T21:32:17.441 INFO:tasks.workunit.client.0.smithi062.stdout:7/925: rename d7/d63/lba to d7/d3e/d33/d3d/l143 0 2024-04-20T21:32:17.446 INFO:tasks.workunit.client.0.smithi062.stdout:4/948: write da/dd/df/d109/f112 [835006,108700] 0 2024-04-20T21:32:17.450 INFO:tasks.workunit.client.0.smithi062.stdout:1/958: dread - dc/d15/d75/d96/dd4/d3a/da2/de4/fd3 zero size 2024-04-20T21:32:17.452 INFO:tasks.workunit.client.0.smithi062.stdout:6/752: write d0/d9/d11/d22/dd5/d67/fca [1429925,126569] 0 2024-04-20T21:32:17.453 INFO:tasks.workunit.client.1.smithi189.stdout:5/652: fsync d7/d14/d18/f17 0 2024-04-20T21:32:17.456 INFO:tasks.workunit.client.1.smithi189.stdout:2/601: dread d1/d11/d2d/f63 [0,4194304] 0 2024-04-20T21:32:17.457 INFO:tasks.workunit.client.0.smithi062.stdout:8/915: read - d7/d25/d3b/d58/de8/ffc zero size 2024-04-20T21:32:17.458 INFO:tasks.workunit.client.0.smithi062.stdout:1/959: read - dc/d15/d75/d96/dd4/d78/dab/fb7 zero size 2024-04-20T21:32:17.458 INFO:tasks.workunit.client.0.smithi062.stdout:7/926: rmdir d7/d3e/d33/d3d/d4d/d54/dea/d80 39 2024-04-20T21:32:17.459 INFO:tasks.workunit.client.1.smithi189.stdout:8/730: dwrite da/df/d20/d2c/f76 [0,4194304] 0 2024-04-20T21:32:17.459 INFO:tasks.workunit.client.1.smithi189.stdout:9/588: dread f6 [0,4194304] 0 2024-04-20T21:32:17.459 INFO:tasks.workunit.client.0.smithi062.stdout:2/921: dwrite d7/da/d15/d2f/f7e [4194304,4194304] 0 2024-04-20T21:32:17.460 INFO:tasks.workunit.client.1.smithi189.stdout:3/652: mkdir d2/d3/d2c/d31/d74/db7/dd3/de2 0 2024-04-20T21:32:17.461 INFO:tasks.workunit.client.1.smithi189.stdout:0/766: mknod d3/db9/cff 0 2024-04-20T21:32:17.463 INFO:tasks.workunit.client.1.smithi189.stdout:3/653: chown d2/d1c/d5c/dca 35617 1 2024-04-20T21:32:17.465 INFO:tasks.workunit.client.1.smithi189.stdout:7/698: truncate d5/d46/f8b 1634926 0 2024-04-20T21:32:17.468 INFO:tasks.workunit.client.1.smithi189.stdout:3/654: write d2/d3/d36/fd7 [1047976,88485] 0 2024-04-20T21:32:17.470 INFO:tasks.workunit.client.1.smithi189.stdout:2/602: truncate d1/d11/d2d/dab/d55/f94 4421334 0 2024-04-20T21:32:17.471 INFO:tasks.workunit.client.1.smithi189.stdout:3/655: chown d2/d3/d2c/d31/lbc 886534 1 2024-04-20T21:32:17.473 INFO:tasks.workunit.client.0.smithi062.stdout:0/852: dwrite d7/d1c/d15/d17/d1b/d106/db1/feb [0,4194304] 0 2024-04-20T21:32:17.475 INFO:tasks.workunit.client.1.smithi189.stdout:3/656: write d2/d3/d2c/d31/fd5 [686053,71980] 0 2024-04-20T21:32:17.477 INFO:tasks.workunit.client.1.smithi189.stdout:1/610: dread d8/d1e/d54/fb0 [0,4194304] 0 2024-04-20T21:32:17.478 INFO:tasks.workunit.client.1.smithi189.stdout:5/653: mknod d7/d9/d30/d55/d81/cd5 0 2024-04-20T21:32:17.488 INFO:tasks.workunit.client.0.smithi062.stdout:7/927: mkdir d7/d3e/d33/d3d/d7f/d116/d139/d144 0 2024-04-20T21:32:17.491 INFO:tasks.workunit.client.0.smithi062.stdout:6/753: dwrite d0/d9/d11/d22/dd5/d67/d75/fd9 [0,4194304] 0 2024-04-20T21:32:17.492 INFO:tasks.workunit.client.1.smithi189.stdout:2/603: dwrite d1/d11/d2d/dab/f72 [0,4194304] 0 2024-04-20T21:32:17.495 INFO:tasks.workunit.client.0.smithi062.stdout:5/892: dread d8/d38/d7a/f91 [0,4194304] 0 2024-04-20T21:32:17.497 INFO:tasks.workunit.client.0.smithi062.stdout:2/922: dread d7/da/d15/d2f/f7e [4194304,4194304] 0 2024-04-20T21:32:17.501 INFO:tasks.workunit.client.1.smithi189.stdout:0/767: symlink d3/df/d30/d45/l100 0 2024-04-20T21:32:17.502 INFO:tasks.workunit.client.0.smithi062.stdout:2/923: chown d7/f124 19197 1 2024-04-20T21:32:17.505 INFO:tasks.workunit.client.0.smithi062.stdout:0/853: dwrite d7/d20/d62/fd1 [4194304,4194304] 0 2024-04-20T21:32:17.507 INFO:tasks.workunit.client.0.smithi062.stdout:7/928: mkdir d7/d3e/d33/d3d/d4d/d54/d4f/de9/df1/d145 0 2024-04-20T21:32:17.507 INFO:tasks.workunit.client.0.smithi062.stdout:5/893: write d8/d38/d7a/d4c/d7d/d9d/dac/d2f/d8e/dd6/f124 [654061,33088] 0 2024-04-20T21:32:17.514 INFO:tasks.workunit.client.0.smithi062.stdout:0/854: chown d7/d1c/d15/d17/d37/d109 920506 1 2024-04-20T21:32:17.514 INFO:tasks.workunit.client.0.smithi062.stdout:6/754: mkdir d0/d9/d11/d22/dd5/dff 0 2024-04-20T21:32:17.514 INFO:tasks.workunit.client.0.smithi062.stdout:3/833: dwrite d0/d1b/f57 [4194304,4194304] 0 2024-04-20T21:32:17.515 INFO:tasks.workunit.client.0.smithi062.stdout:6/755: chown d0/d9/d11/ff7 2 1 2024-04-20T21:32:17.517 INFO:tasks.workunit.client.0.smithi062.stdout:4/949: dwrite da/dd/d15/fa7 [0,4194304] 0 2024-04-20T21:32:17.525 INFO:tasks.workunit.client.0.smithi062.stdout:5/894: mkdir d8/d1e/d2c/d6c/d12a 0 2024-04-20T21:32:17.525 INFO:tasks.workunit.client.0.smithi062.stdout:3/834: chown d0/d1c/d63/dda 640551487 1 2024-04-20T21:32:17.527 INFO:tasks.workunit.client.1.smithi189.stdout:0/768: dwrite d3/da8/fe9 [0,4194304] 0 2024-04-20T21:32:17.530 INFO:tasks.workunit.client.1.smithi189.stdout:4/613: dwrite d2/dd/d37/f59 [0,4194304] 0 2024-04-20T21:32:17.530 INFO:tasks.workunit.client.1.smithi189.stdout:9/589: rename d9/d12/d62/c8c to d9/dc/d17/d95/da0/d52/d82/da4/cbb 0 2024-04-20T21:32:17.530 INFO:tasks.workunit.client.0.smithi062.stdout:2/924: getdents d7/db5 0 2024-04-20T21:32:17.531 INFO:tasks.workunit.client.0.smithi062.stdout:6/756: truncate d0/d52/f98 31644 0 2024-04-20T21:32:17.532 INFO:tasks.workunit.client.1.smithi189.stdout:4/614: chown d2/d3a/ddc 294799710 1 2024-04-20T21:32:17.536 INFO:tasks.workunit.client.0.smithi062.stdout:3/835: creat d0/d1/d3c/d43/d72/da9/f110 x:0 0 0 2024-04-20T21:32:17.537 INFO:tasks.workunit.client.1.smithi189.stdout:3/657: mkdir d2/d3/d2c/de3 0 2024-04-20T21:32:17.538 INFO:tasks.workunit.client.0.smithi062.stdout:5/895: unlink d8/d38/d7a/d4c/d7d/d9d/dac/d2f/fea 0 2024-04-20T21:32:17.539 INFO:tasks.workunit.client.1.smithi189.stdout:9/590: dread d9/dc/d17/d95/da0/d56/f86 [0,4194304] 0 2024-04-20T21:32:17.539 INFO:tasks.workunit.client.0.smithi062.stdout:4/950: mknod da/c145 0 2024-04-20T21:32:17.539 INFO:tasks.workunit.client.1.smithi189.stdout:5/654: dread d7/d14/d18/f3e [0,4194304] 0 2024-04-20T21:32:17.544 INFO:tasks.workunit.client.1.smithi189.stdout:7/699: unlink d5/d1d/d77/l9f 0 2024-04-20T21:32:17.545 INFO:tasks.workunit.client.0.smithi062.stdout:6/757: rename d0/d9/d11/d22/dd5/d67/f6a to d0/d1a/d44/da7/f100 0 2024-04-20T21:32:17.546 INFO:tasks.workunit.client.1.smithi189.stdout:7/700: readlink d5/d21/d4a/l54 0 2024-04-20T21:32:17.551 INFO:tasks.workunit.client.1.smithi189.stdout:3/658: creat d2/d3/d25/de0/fe4 x:0 0 0 2024-04-20T21:32:17.555 INFO:tasks.workunit.client.0.smithi062.stdout:4/951: creat da/dd/df/dfa/f146 x:0 0 0 2024-04-20T21:32:17.555 INFO:tasks.workunit.client.0.smithi062.stdout:6/758: rmdir d0/d9 39 2024-04-20T21:32:17.560 INFO:tasks.workunit.client.1.smithi189.stdout:5/655: mknod d7/d14/d18/cd6 0 2024-04-20T21:32:17.564 INFO:tasks.workunit.client.1.smithi189.stdout:9/591: rename d9/d12/fa1 to d9/dc/d17/d7e/fbc 0 2024-04-20T21:32:17.564 INFO:tasks.workunit.client.1.smithi189.stdout:0/769: dwrite d3/df/d30/d45/d90/fbf [0,4194304] 0 2024-04-20T21:32:17.567 INFO:tasks.workunit.client.1.smithi189.stdout:4/615: symlink d2/dd/d37/le2 0 2024-04-20T21:32:17.567 INFO:tasks.workunit.client.0.smithi062.stdout:0/855: write d7/d1c/d15/f22 [5163489,122154] 0 2024-04-20T21:32:17.568 INFO:tasks.workunit.client.1.smithi189.stdout:7/701: chown d5/d21/d4a/d52/f7c 117907 1 2024-04-20T21:32:17.569 INFO:tasks.workunit.client.0.smithi062.stdout:8/916: dread d7/d25/f93 [0,4194304] 0 2024-04-20T21:32:17.571 INFO:tasks.workunit.client.0.smithi062.stdout:7/929: dwrite d7/d3e/f7b [0,4194304] 0 2024-04-20T21:32:17.571 INFO:tasks.workunit.client.1.smithi189.stdout:5/656: rename d7/d14/d4e/d5b/c27 to d7/d9/d51/d70/d95/cd7 0 2024-04-20T21:32:17.572 INFO:tasks.workunit.client.1.smithi189.stdout:6/678: dwrite d8/d12/d30/fc0 [0,4194304] 0 2024-04-20T21:32:17.578 INFO:tasks.workunit.client.1.smithi189.stdout:2/604: write d1/d11/d52/d8b/fa1 [951813,25658] 0 2024-04-20T21:32:17.583 INFO:tasks.workunit.client.1.smithi189.stdout:0/770: write d3/df/d28/f61 [679218,119630] 0 2024-04-20T21:32:17.584 INFO:tasks.workunit.client.1.smithi189.stdout:6/679: symlink d8/d2b/d49/d54/d67/ldb 0 2024-04-20T21:32:17.587 INFO:tasks.workunit.client.0.smithi062.stdout:3/836: write d0/d1/f77 [981366,118943] 0 2024-04-20T21:32:17.588 INFO:tasks.workunit.client.0.smithi062.stdout:8/917: stat d7/d9/f17 0 2024-04-20T21:32:17.588 INFO:tasks.workunit.client.1.smithi189.stdout:2/605: chown d1/d17/d78 10 1 2024-04-20T21:32:17.589 INFO:tasks.workunit.client.0.smithi062.stdout:7/930: creat d7/d26/dac/d10a/f146 x:0 0 0 2024-04-20T21:32:17.589 INFO:tasks.workunit.client.1.smithi189.stdout:9/592: dwrite d9/dc/d17/d36/f78 [0,4194304] 0 2024-04-20T21:32:17.591 INFO:tasks.workunit.client.0.smithi062.stdout:2/925: write d7/da/d15/d2f/d38/d51/f12c [767080,98408] 0 2024-04-20T21:32:17.591 INFO:tasks.workunit.client.0.smithi062.stdout:5/896: write d8/d38/fbe [605434,90508] 0 2024-04-20T21:32:17.594 INFO:tasks.workunit.client.0.smithi062.stdout:6/759: truncate d0/d1a/f29 1788507 0 2024-04-20T21:32:17.594 INFO:tasks.workunit.client.0.smithi062.stdout:8/918: chown d7/d25/d3b/d58/d8b/ffd 16 1 2024-04-20T21:32:17.609 INFO:tasks.workunit.client.0.smithi062.stdout:6/760: write d0/d1a/d44/f4c [1032274,7309] 0 2024-04-20T21:32:17.616 INFO:tasks.workunit.client.1.smithi189.stdout:6/680: mkdir d8/d12/d30/d47/ddc 0 2024-04-20T21:32:17.622 INFO:tasks.workunit.client.1.smithi189.stdout:8/731: sync 2024-04-20T21:32:17.622 INFO:tasks.workunit.client.1.smithi189.stdout:1/611: sync 2024-04-20T21:32:17.622 INFO:tasks.workunit.client.1.smithi189.stdout:7/702: creat d5/d21/d25/fe8 x:0 0 0 2024-04-20T21:32:17.623 INFO:tasks.workunit.client.0.smithi062.stdout:3/837: write d0/d1/d10/d31/d41/f7c [531341,76361] 0 2024-04-20T21:32:17.625 INFO:tasks.workunit.client.1.smithi189.stdout:4/616: dread d2/dd/f26 [0,4194304] 0 2024-04-20T21:32:17.625 INFO:tasks.workunit.client.1.smithi189.stdout:1/612: chown d8/d1e/c9a 9750259 1 2024-04-20T21:32:17.626 INFO:tasks.workunit.client.1.smithi189.stdout:0/771: link d3/df/d28/d2b/d2c/f83 d3/df/d28/d2b/db1/dfe/f101 0 2024-04-20T21:32:17.631 INFO:tasks.workunit.client.0.smithi062.stdout:4/952: dwrite da/dd/d15/d50/d64/d57/d5b/d9d/dea/dac/f11d [0,4194304] 0 2024-04-20T21:32:17.634 INFO:tasks.workunit.client.1.smithi189.stdout:8/732: rmdir da/df/d3c/d9d 39 2024-04-20T21:32:17.636 INFO:tasks.workunit.client.1.smithi189.stdout:5/657: write d7/d9/d28/f58 [67054,25747] 0 2024-04-20T21:32:17.641 INFO:tasks.workunit.client.0.smithi062.stdout:0/856: dwrite d7/ddf/fe8 [0,4194304] 0 2024-04-20T21:32:17.643 INFO:tasks.workunit.client.1.smithi189.stdout:6/681: getdents d8 0 2024-04-20T21:32:17.643 INFO:tasks.workunit.client.1.smithi189.stdout:2/606: dwrite d1/d11/d2d/dab/d4a/f84 [0,4194304] 0 2024-04-20T21:32:17.648 INFO:tasks.workunit.client.0.smithi062.stdout:5/897: dread d8/f101 [0,4194304] 0 2024-04-20T21:32:17.648 INFO:tasks.workunit.client.0.smithi062.stdout:0/857: chown d7/d1c/f63 19297 1 2024-04-20T21:32:17.651 INFO:tasks.workunit.client.1.smithi189.stdout:1/613: mkdir d8/d1e/d28/dd4 0 2024-04-20T21:32:17.653 INFO:tasks.workunit.client.1.smithi189.stdout:7/703: creat d5/d21/d4a/fe9 x:0 0 0 2024-04-20T21:32:17.656 INFO:tasks.workunit.client.0.smithi062.stdout:6/761: dwrite d0/f80 [8388608,4194304] 0 2024-04-20T21:32:17.659 INFO:tasks.workunit.client.1.smithi189.stdout:0/772: read d3/fd [1161033,90172] 0 2024-04-20T21:32:17.662 INFO:tasks.workunit.client.1.smithi189.stdout:6/682: rename d8/da/dd/d35/c26 to d8/da/dd/d35/d6d/cdd 0 2024-04-20T21:32:17.666 INFO:tasks.workunit.client.1.smithi189.stdout:3/659: sync 2024-04-20T21:32:17.666 INFO:tasks.workunit.client.1.smithi189.stdout:0/773: dread - d3/df/d30/d45/dd3/fee zero size 2024-04-20T21:32:17.672 INFO:tasks.workunit.client.1.smithi189.stdout:8/733: dwrite da/df/d22/faf [0,4194304] 0 2024-04-20T21:32:17.675 INFO:tasks.workunit.client.1.smithi189.stdout:7/704: getdents d5/db4/d71/d7a/db0 0 2024-04-20T21:32:17.678 INFO:tasks.workunit.client.1.smithi189.stdout:1/614: mkdir d8/df/d3a/d49/d8c/da2/dbe/dd5 0 2024-04-20T21:32:17.685 INFO:tasks.workunit.client.1.smithi189.stdout:3/660: rename d2/d1c/d5c/fd6 to d2/d3/d25/de0/da9/fe5 0 2024-04-20T21:32:17.688 INFO:tasks.workunit.client.0.smithi062.stdout:7/931: mknod d7/d3e/d33/d3d/d4d/c147 0 2024-04-20T21:32:17.689 INFO:tasks.workunit.client.1.smithi189.stdout:2/607: dwrite d1/f8c [0,4194304] 0 2024-04-20T21:32:17.689 INFO:tasks.workunit.client.1.smithi189.stdout:8/734: fsync da/df/d3a/f5d 0 2024-04-20T21:32:17.691 INFO:tasks.workunit.client.1.smithi189.stdout:1/615: chown d8/d1e/d28/d4f/lb4 484 1 2024-04-20T21:32:17.701 INFO:tasks.workunit.client.0.smithi062.stdout:4/953: mknod da/dd/df/d109/c147 0 2024-04-20T21:32:17.702 INFO:tasks.workunit.client.1.smithi189.stdout:8/735: dread - da/df/d3c/d50/ddc/df2/f104 zero size 2024-04-20T21:32:17.702 INFO:tasks.workunit.client.1.smithi189.stdout:3/661: symlink d2/d3/d2c/d31/d74/le6 0 2024-04-20T21:32:17.702 INFO:tasks.workunit.client.1.smithi189.stdout:0/774: rename d3/df/d28/d2b/f58 to d3/f102 0 2024-04-20T21:32:17.705 INFO:tasks.workunit.client.0.smithi062.stdout:0/858: unlink d7/d1c/d15/fca 0 2024-04-20T21:32:17.706 INFO:tasks.workunit.client.1.smithi189.stdout:4/617: dwrite d2/dd/d11/d14/f7d [0,4194304] 0 2024-04-20T21:32:17.707 INFO:tasks.workunit.client.0.smithi062.stdout:0/859: write d7/ddf/fe8 [659058,27298] 0 2024-04-20T21:32:17.707 INFO:tasks.workunit.client.0.smithi062.stdout:6/762: creat d0/d9a/f101 x:0 0 0 2024-04-20T21:32:17.709 INFO:tasks.workunit.client.0.smithi062.stdout:0/860: write d7/d1e/d6e/f7d [3279803,128616] 0 2024-04-20T21:32:17.710 INFO:tasks.workunit.client.1.smithi189.stdout:1/616: truncate f2 4583746 0 2024-04-20T21:32:17.711 INFO:tasks.workunit.client.0.smithi062.stdout:8/919: mkdir d7/d9/d29/d11c/d12d 0 2024-04-20T21:32:17.713 INFO:tasks.workunit.client.1.smithi189.stdout:9/593: dread d9/d12/d3f/d91/fa3 [0,4194304] 0 2024-04-20T21:32:17.714 INFO:tasks.workunit.client.0.smithi062.stdout:7/932: chown d7/d3e/f6a 6 1 2024-04-20T21:32:17.715 INFO:tasks.workunit.client.0.smithi062.stdout:7/933: readlink d7/d26/la2 0 2024-04-20T21:32:17.715 INFO:tasks.workunit.client.1.smithi189.stdout:6/683: dwrite d8/d12/d30/fc0 [0,4194304] 0 2024-04-20T21:32:17.716 INFO:tasks.workunit.client.1.smithi189.stdout:1/617: dread - d8/d1e/d28/f91 zero size 2024-04-20T21:32:17.718 INFO:tasks.workunit.client.1.smithi189.stdout:0/775: creat d3/d26/d88/f103 x:0 0 0 2024-04-20T21:32:17.723 INFO:tasks.workunit.client.1.smithi189.stdout:3/662: creat d2/d3/d2c/d31/d74/d7a/fe7 x:0 0 0 2024-04-20T21:32:17.723 INFO:tasks.workunit.client.0.smithi062.stdout:6/763: creat d0/dc7/d7c/dc5/def/f102 x:0 0 0 2024-04-20T21:32:17.727 INFO:tasks.workunit.client.1.smithi189.stdout:2/608: rename d1/d17/d37/c43 to d1/d11/d2d/d4f/d8e/cb6 0 2024-04-20T21:32:17.727 INFO:tasks.workunit.client.1.smithi189.stdout:1/618: unlink d8/d21/c8b 0 2024-04-20T21:32:17.730 INFO:tasks.workunit.client.0.smithi062.stdout:7/934: rmdir d7/d3e/d33/d3d/d4d/d54/d4f/de9 39 2024-04-20T21:32:17.730 INFO:tasks.workunit.client.0.smithi062.stdout:4/954: creat da/dd/df/d27/da5/df1/f148 x:0 0 0 2024-04-20T21:32:17.733 INFO:tasks.workunit.client.1.smithi189.stdout:9/594: mknod d9/d12/d3b/db1/d1a/cbd 0 2024-04-20T21:32:17.734 INFO:tasks.workunit.client.0.smithi062.stdout:0/861: creat d7/d1c/d15/d17/dfd/f10f x:0 0 0 2024-04-20T21:32:17.735 INFO:tasks.workunit.client.1.smithi189.stdout:6/684: unlink d8/d12/d30/fc0 0 2024-04-20T21:32:17.738 INFO:tasks.workunit.client.0.smithi062.stdout:7/935: rmdir d7/d3e/d33/d3d/d4d/d54/d4f 39 2024-04-20T21:32:17.739 INFO:tasks.workunit.client.0.smithi062.stdout:4/955: chown da/dd/df/d27/d3f/d82/c12f 95 1 2024-04-20T21:32:17.746 INFO:tasks.workunit.client.1.smithi189.stdout:7/705: dwrite d5/f6 [0,4194304] 0 2024-04-20T21:32:17.746 INFO:tasks.workunit.client.0.smithi062.stdout:6/764: link d0/d52/da0/fcc d0/d9/d11/d7b/f103 0 2024-04-20T21:32:17.747 INFO:tasks.workunit.client.1.smithi189.stdout:1/619: mknod d8/df/d3a/d49/d8c/da2/dbe/cd6 0 2024-04-20T21:32:17.748 INFO:tasks.workunit.client.1.smithi189.stdout:2/609: mkdir d1/d11/d99/db7 0 2024-04-20T21:32:17.748 INFO:tasks.workunit.client.0.smithi062.stdout:6/765: dread - d0/d1a/d7d/db1/de0/fe6 zero size 2024-04-20T21:32:17.749 INFO:tasks.workunit.client.1.smithi189.stdout:9/595: symlink d9/dc/d17/d95/da0/d52/d82/d94/lbe 0 2024-04-20T21:32:17.749 INFO:tasks.workunit.client.0.smithi062.stdout:0/862: creat d7/d1c/d15/d17/d1b/d106/df1/d10e/f110 x:0 0 0 2024-04-20T21:32:17.751 INFO:tasks.workunit.client.0.smithi062.stdout:6/766: truncate d0/d9/d11/f1f 4596958 0 2024-04-20T21:32:17.753 INFO:tasks.workunit.client.1.smithi189.stdout:5/658: dread d7/d9/d30/d5a/f64 [0,4194304] 0 2024-04-20T21:32:17.753 INFO:tasks.workunit.client.1.smithi189.stdout:7/706: read - d5/d46/d7d/fb1 zero size 2024-04-20T21:32:17.755 INFO:tasks.workunit.client.1.smithi189.stdout:2/610: write d1/d11/d2d/dab/d4a/f84 [2362479,41771] 0 2024-04-20T21:32:17.756 INFO:tasks.workunit.client.1.smithi189.stdout:9/596: chown d9/dc/d17/faa 0 1 2024-04-20T21:32:17.766 INFO:tasks.workunit.client.0.smithi062.stdout:0/863: rmdir d7/d1c/d15/d53 39 2024-04-20T21:32:17.769 INFO:tasks.workunit.client.1.smithi189.stdout:7/707: mknod d5/d1d/d64/db9/ddd/cea 0 2024-04-20T21:32:17.771 INFO:tasks.workunit.client.0.smithi062.stdout:4/956: symlink da/d4f/l149 0 2024-04-20T21:32:17.771 INFO:tasks.workunit.client.1.smithi189.stdout:9/597: rename d9/dc/c2a to d9/d12/d3f/cbf 0 2024-04-20T21:32:17.774 INFO:tasks.workunit.client.1.smithi189.stdout:4/618: dread d2/f8 [0,4194304] 0 2024-04-20T21:32:17.775 INFO:tasks.workunit.client.1.smithi189.stdout:3/663: dwrite d2/d3/d25/de0/fe4 [0,4194304] 0 2024-04-20T21:32:17.775 INFO:tasks.workunit.client.1.smithi189.stdout:7/708: write d5/d1d/d77/fc1 [1002681,65758] 0 2024-04-20T21:32:17.777 INFO:tasks.workunit.client.0.smithi062.stdout:6/767: symlink d0/d9/d11/d22/dd5/d67/d75/l104 0 2024-04-20T21:32:17.778 INFO:tasks.workunit.client.1.smithi189.stdout:9/598: dread - d9/d12/fa8 zero size 2024-04-20T21:32:17.779 INFO:tasks.workunit.client.0.smithi062.stdout:0/864: symlink d7/d1c/d15/d17/d37/l111 0 2024-04-20T21:32:17.781 INFO:tasks.workunit.client.1.smithi189.stdout:0/776: truncate d3/df/d30/fef 2761890 0 2024-04-20T21:32:17.783 INFO:tasks.workunit.client.0.smithi062.stdout:7/936: read - d7/d3e/d33/d3d/d4d/d54/d4f/de9/df1/fff zero size 2024-04-20T21:32:17.785 INFO:tasks.workunit.client.0.smithi062.stdout:0/865: symlink d7/d1c/d15/d17/d1b/l112 0 2024-04-20T21:32:17.787 INFO:tasks.workunit.client.1.smithi189.stdout:9/599: write d9/d12/d50/f4f [5309805,2388] 0 2024-04-20T21:32:17.788 INFO:tasks.workunit.client.1.smithi189.stdout:3/664: chown d2/d3/d83/lce 28 1 2024-04-20T21:32:17.790 INFO:tasks.workunit.client.1.smithi189.stdout:1/620: write f1 [7158732,126489] 0 2024-04-20T21:32:17.791 INFO:tasks.workunit.client.0.smithi062.stdout:7/937: creat d7/d26/da8/d131/f148 x:0 0 0 2024-04-20T21:32:17.792 INFO:tasks.workunit.client.1.smithi189.stdout:4/619: unlink d2/d20/faf 0 2024-04-20T21:32:17.793 INFO:tasks.workunit.client.1.smithi189.stdout:5/659: dwrite d7/d9/d31/f88 [0,4194304] 0 2024-04-20T21:32:17.793 INFO:tasks.workunit.client.1.smithi189.stdout:8/736: dwrite da/de/f8c [0,4194304] 0 2024-04-20T21:32:17.799 INFO:tasks.workunit.client.1.smithi189.stdout:3/665: dread - d2/d3/d25/de0/da9/fe5 zero size 2024-04-20T21:32:17.802 INFO:tasks.workunit.client.1.smithi189.stdout:3/666: fsync d2/d3/d2c/d31/fd5 0 2024-04-20T21:32:17.802 INFO:tasks.workunit.client.0.smithi062.stdout:6/768: creat d0/d1a/f105 x:0 0 0 2024-04-20T21:32:17.803 INFO:tasks.workunit.client.1.smithi189.stdout:1/621: chown d8/df/d50/ca0 676336025 1 2024-04-20T21:32:17.804 INFO:tasks.workunit.client.1.smithi189.stdout:7/709: rename d5/d1d/d77/fd3 to d5/d1d/d64/feb 0 2024-04-20T21:32:17.806 INFO:tasks.workunit.client.0.smithi062.stdout:7/938: fdatasync d7/d3e/d33/d3d/f75 0 2024-04-20T21:32:17.807 INFO:tasks.workunit.client.1.smithi189.stdout:8/737: unlink da/de/d1e/d69/le8 0 2024-04-20T21:32:17.807 INFO:tasks.workunit.client.1.smithi189.stdout:3/667: stat d2/d3/d2c/d30/d3f/f8a 0 2024-04-20T21:32:17.808 INFO:tasks.workunit.client.0.smithi062.stdout:7/939: fdatasync d7/d3e/d33/d3d/d5e/d72/fb2 0 2024-04-20T21:32:17.811 INFO:tasks.workunit.client.1.smithi189.stdout:7/710: dread - d5/d1d/d64/db9/fe0 zero size 2024-04-20T21:32:17.812 INFO:tasks.workunit.client.1.smithi189.stdout:5/660: unlink d7/f5d 0 2024-04-20T21:32:17.814 INFO:tasks.workunit.client.1.smithi189.stdout:1/622: rmdir d8/df/d3a/d49 39 2024-04-20T21:32:17.814 INFO:tasks.workunit.client.1.smithi189.stdout:9/600: mknod d9/dc/d17/d95/da0/d52/d82/d94/cc0 0 2024-04-20T21:32:17.815 INFO:tasks.workunit.client.1.smithi189.stdout:7/711: chown d5/d21/d25/f6c 757921 1 2024-04-20T21:32:17.816 INFO:tasks.workunit.client.0.smithi062.stdout:6/769: creat d0/d1a/d44/db3/f106 x:0 0 0 2024-04-20T21:32:17.818 INFO:tasks.workunit.client.1.smithi189.stdout:7/712: dread - d5/db4/fe5 zero size 2024-04-20T21:32:17.818 INFO:tasks.workunit.client.1.smithi189.stdout:2/611: dread d1/d11/d2d/d4f/d8e/f9a [0,4194304] 0 2024-04-20T21:32:17.818 INFO:tasks.workunit.client.1.smithi189.stdout:9/601: rmdir d9/d12/d3f 39 2024-04-20T21:32:17.823 INFO:tasks.workunit.client.1.smithi189.stdout:1/623: symlink d8/d1e/d43/d58/d60/ld7 0 2024-04-20T21:32:17.824 INFO:tasks.workunit.client.0.smithi062.stdout:0/866: link d7/d1c/d15/d17/d40/fcc d7/d1c/d15/d17/d37/f113 0 2024-04-20T21:32:17.825 INFO:tasks.workunit.client.1.smithi189.stdout:2/612: rmdir d1/d11/d2d/d4f/db4 39 2024-04-20T21:32:17.825 INFO:tasks.workunit.client.1.smithi189.stdout:9/602: symlink d9/dc/d17/d95/lc1 0 2024-04-20T21:32:17.827 INFO:tasks.workunit.client.1.smithi189.stdout:1/624: write d8/f14 [4751450,82576] 0 2024-04-20T21:32:17.828 INFO:tasks.workunit.client.1.smithi189.stdout:6/685: dread d8/da/dd/d2a/d3b/f8e [0,4194304] 0 2024-04-20T21:32:17.830 INFO:tasks.workunit.client.1.smithi189.stdout:9/603: rmdir d9/dc/d17/d95/da0/d52/d82/d94 39 2024-04-20T21:32:17.831 INFO:tasks.workunit.client.0.smithi062.stdout:7/940: fdatasync d7/d3e/d33/d3d/d7f/f9e 0 2024-04-20T21:32:17.833 INFO:tasks.workunit.client.1.smithi189.stdout:1/625: dread - d8/df/d3a/d49/d8c/fb9 zero size 2024-04-20T21:32:17.834 INFO:tasks.workunit.client.1.smithi189.stdout:6/686: write d8/d12/d30/f58 [3977727,37868] 0 2024-04-20T21:32:17.836 INFO:tasks.workunit.client.1.smithi189.stdout:2/613: creat d1/d11/d99/db7/fb8 x:0 0 0 2024-04-20T21:32:17.837 INFO:tasks.workunit.client.1.smithi189.stdout:9/604: fsync d9/dc/f1c 0 2024-04-20T21:32:17.841 INFO:tasks.workunit.client.1.smithi189.stdout:0/777: write d3/d26/fdf [82578,67140] 0 2024-04-20T21:32:17.842 INFO:tasks.workunit.client.1.smithi189.stdout:6/687: chown d8/d2b/d49/d54/d67/f9c 104693 1 2024-04-20T21:32:17.843 INFO:tasks.workunit.client.1.smithi189.stdout:9/605: readlink d9/d12/d3b/db1/d1a/l9b 0 2024-04-20T21:32:17.845 INFO:tasks.workunit.client.1.smithi189.stdout:2/614: mkdir d1/d17/d78/db9 0 2024-04-20T21:32:17.848 INFO:tasks.workunit.client.0.smithi062.stdout:4/957: getdents da/dd/df/d27/da5/df1 0 2024-04-20T21:32:17.851 INFO:tasks.workunit.client.0.smithi062.stdout:6/770: fsync d0/d52/da0/fcc 0 2024-04-20T21:32:17.851 INFO:tasks.workunit.client.0.smithi062.stdout:4/958: readlink da/lb1 0 2024-04-20T21:32:17.856 INFO:tasks.workunit.client.1.smithi189.stdout:4/620: write d2/dd/d37/dcd/d6a/d7a/f8d [856985,127521] 0 2024-04-20T21:32:17.858 INFO:tasks.workunit.client.1.smithi189.stdout:4/621: truncate d2/dd/fbd 238649 0 2024-04-20T21:32:17.860 INFO:tasks.workunit.client.1.smithi189.stdout:6/688: mknod d8/da/dd/d2a/cde 0 2024-04-20T21:32:17.862 INFO:tasks.workunit.client.1.smithi189.stdout:6/689: chown d8/d12/d4e/dc4/ld0 35952 1 2024-04-20T21:32:17.862 INFO:tasks.workunit.client.1.smithi189.stdout:5/661: dread d7/d35/f43 [0,4194304] 0 2024-04-20T21:32:17.863 INFO:tasks.workunit.client.0.smithi062.stdout:3/838: dwrite d0/d1b/f35 [0,4194304] 0 2024-04-20T21:32:17.865 INFO:tasks.workunit.client.0.smithi062.stdout:7/941: dwrite d7/d3e/d33/d3d/d4d/dce/fde [0,4194304] 0 2024-04-20T21:32:17.866 INFO:tasks.workunit.client.1.smithi189.stdout:5/662: dread d7/d9/d28/da3/fc3 [0,4194304] 0 2024-04-20T21:32:17.868 INFO:tasks.workunit.client.1.smithi189.stdout:1/626: creat d8/d1e/d28/fd8 x:0 0 0 2024-04-20T21:32:17.868 INFO:tasks.workunit.client.1.smithi189.stdout:7/713: write d5/d46/fa9 [1205335,118453] 0 2024-04-20T21:32:17.870 INFO:tasks.workunit.client.1.smithi189.stdout:1/627: readlink d8/df/d50/l6a 0 2024-04-20T21:32:17.872 INFO:tasks.workunit.client.1.smithi189.stdout:9/606: creat d9/dc/d17/dac/fc2 x:0 0 0 2024-04-20T21:32:17.873 INFO:tasks.workunit.client.1.smithi189.stdout:4/622: creat d2/d22/d57/fe3 x:0 0 0 2024-04-20T21:32:17.873 INFO:tasks.workunit.client.1.smithi189.stdout:7/714: write d5/d46/fa9 [2202037,89804] 0 2024-04-20T21:32:17.874 INFO:tasks.workunit.client.1.smithi189.stdout:2/615: unlink d1/d11/f28 0 2024-04-20T21:32:17.874 INFO:tasks.workunit.client.0.smithi062.stdout:6/771: mkdir d0/d9/d11/d22/d107 0 2024-04-20T21:32:17.880 INFO:tasks.workunit.client.0.smithi062.stdout:2/926: dwrite d7/da/d15/d2f/d97/fa0 [0,4194304] 0 2024-04-20T21:32:17.882 INFO:tasks.workunit.client.0.smithi062.stdout:1/960: sync 2024-04-20T21:32:17.883 INFO:tasks.workunit.client.0.smithi062.stdout:5/898: sync 2024-04-20T21:32:17.884 INFO:tasks.workunit.client.0.smithi062.stdout:8/920: dwrite d7/d9/f17 [0,4194304] 0 2024-04-20T21:32:17.885 INFO:tasks.workunit.client.1.smithi189.stdout:6/690: rename d8/d12/d30/f38 to d8/da/dd/d2a/fdf 0 2024-04-20T21:32:17.886 INFO:tasks.workunit.client.1.smithi189.stdout:0/778: fdatasync d3/df/d28/d2b/db1/dfe/f101 0 2024-04-20T21:32:17.889 INFO:tasks.workunit.client.0.smithi062.stdout:4/959: truncate da/dd/d15/d50/d64/d3d/fb5 4064830 0 2024-04-20T21:32:17.889 INFO:tasks.workunit.client.0.smithi062.stdout:0/867: dread d7/d1c/f39 [0,4194304] 0 2024-04-20T21:32:17.894 INFO:tasks.workunit.client.0.smithi062.stdout:3/839: mknod d0/d1/d3c/d43/d72/dd3/dbf/de8/c111 0 2024-04-20T21:32:17.894 INFO:tasks.workunit.client.0.smithi062.stdout:8/921: rmdir d7/d9/d29/d11c 39 2024-04-20T21:32:17.895 INFO:tasks.workunit.client.1.smithi189.stdout:5/663: dwrite d7/d9/d51/fb4 [0,4194304] 0 2024-04-20T21:32:17.897 INFO:tasks.workunit.client.1.smithi189.stdout:7/715: dwrite d5/d46/fa9 [0,4194304] 0 2024-04-20T21:32:17.899 INFO:tasks.workunit.client.0.smithi062.stdout:5/899: chown d8/d38/d7a/d4c/d7d/d9d/dac/d2f/df3 45363990 1 2024-04-20T21:32:17.908 INFO:tasks.workunit.client.0.smithi062.stdout:2/927: dread d7/da/d15/d1b/f2c [0,4194304] 0 2024-04-20T21:32:17.909 INFO:tasks.workunit.client.0.smithi062.stdout:0/868: chown d7/d1c/d15/d17/d40/d79/fb0 0 1 2024-04-20T21:32:17.911 INFO:tasks.workunit.client.1.smithi189.stdout:4/623: creat d2/d64/fe4 x:0 0 0 2024-04-20T21:32:17.912 INFO:tasks.workunit.client.1.smithi189.stdout:9/607: symlink d9/d12/d62/dae/lc3 0 2024-04-20T21:32:17.912 INFO:tasks.workunit.client.1.smithi189.stdout:4/624: chown d2/d22/c4a 2752 1 2024-04-20T21:32:17.915 INFO:tasks.workunit.client.1.smithi189.stdout:4/625: write d2/f54 [5179071,49557] 0 2024-04-20T21:32:17.916 INFO:tasks.workunit.client.0.smithi062.stdout:3/840: rename d0/d1/d3c/d43/d67 to d0/d1/d10/d112 0 2024-04-20T21:32:17.916 INFO:tasks.workunit.client.1.smithi189.stdout:4/626: write d2/d22/d57/fe3 [35224,37234] 0 2024-04-20T21:32:17.920 INFO:tasks.workunit.client.1.smithi189.stdout:2/616: creat d1/d11/d2d/dab/d55/fba x:0 0 0 2024-04-20T21:32:17.920 INFO:tasks.workunit.client.0.smithi062.stdout:5/900: rename d8/d38/d7a/d4c/d7d/d9d/dac/d66/dae/f64 to d8/d38/d7a/d4c/d7d/d9d/dac/d105/f12b 0 2024-04-20T21:32:17.924 INFO:tasks.workunit.client.0.smithi062.stdout:3/841: chown d0/d1/d10/d112 50840 1 2024-04-20T21:32:17.927 INFO:tasks.workunit.client.0.smithi062.stdout:2/928: fdatasync d7/da/d15/d2f/d38/d10f/d4b/d114/ffd 0 2024-04-20T21:32:17.928 INFO:tasks.workunit.client.0.smithi062.stdout:5/901: chown d8/d1e/d2c/f86 1806003519 1 2024-04-20T21:32:17.935 INFO:tasks.workunit.client.0.smithi062.stdout:5/902: creat d8/d10/d3e/d8a/d9e/dd7/f12c x:0 0 0 2024-04-20T21:32:17.936 INFO:tasks.workunit.client.0.smithi062.stdout:4/960: dwrite da/dd/f14 [0,4194304] 0 2024-04-20T21:32:17.950 INFO:tasks.workunit.client.0.smithi062.stdout:6/772: dread d0/d1a/f85 [0,4194304] 0 2024-04-20T21:32:17.961 INFO:tasks.workunit.client.0.smithi062.stdout:0/869: dwrite d7/d1c/d15/d17/fd0 [0,4194304] 0 2024-04-20T21:32:17.972 INFO:tasks.workunit.client.0.smithi062.stdout:3/842: dwrite d0/d1/d10/d3a/d5e/f7d [0,4194304] 0 2024-04-20T21:32:17.973 INFO:tasks.workunit.client.0.smithi062.stdout:6/773: creat d0/dc7/d7c/dc5/def/f108 x:0 0 0 2024-04-20T21:32:17.974 INFO:tasks.workunit.client.0.smithi062.stdout:2/929: dwrite d7/da/d15/d2f/d38/db0/dc4/fc7 [0,4194304] 0 2024-04-20T21:32:17.975 INFO:tasks.workunit.client.0.smithi062.stdout:6/774: readlink d0/d4f/le3 0 2024-04-20T21:32:17.982 INFO:tasks.workunit.client.0.smithi062.stdout:4/961: link da/dd/df/d27/f110 da/dd/df/d27/f14a 0 2024-04-20T21:32:17.987 INFO:tasks.workunit.client.0.smithi062.stdout:2/930: mknod d7/da/d15/d1b/da8/c12e 0 2024-04-20T21:32:18.008 INFO:tasks.workunit.client.0.smithi062.stdout:0/870: dwrite d7/d1e/db6/fee [0,4194304] 0 2024-04-20T21:32:18.010 INFO:tasks.workunit.client.1.smithi189.stdout:8/738: sync 2024-04-20T21:32:18.010 INFO:tasks.workunit.client.1.smithi189.stdout:3/668: sync 2024-04-20T21:32:18.010 INFO:tasks.workunit.client.0.smithi062.stdout:3/843: dwrite d0/d1b/f4d [0,4194304] 0 2024-04-20T21:32:18.012 INFO:tasks.workunit.client.0.smithi062.stdout:7/942: dread d7/d3e/d33/d3d/d4d/d83/d95/faa [0,4194304] 0 2024-04-20T21:32:18.018 INFO:tasks.workunit.client.0.smithi062.stdout:0/871: stat d7/d1c/d15/d17/d1b/d106/caa 0 2024-04-20T21:32:18.020 INFO:tasks.workunit.client.0.smithi062.stdout:7/943: chown d7/d26/da8/lcf 416045669 1 2024-04-20T21:32:18.023 INFO:tasks.workunit.client.0.smithi062.stdout:3/844: mknod d0/d1c/d63/d42/dc8/dc9/c113 0 2024-04-20T21:32:18.024 INFO:tasks.workunit.client.0.smithi062.stdout:0/872: mknod d7/d20/d62/d6f/c114 0 2024-04-20T21:32:18.029 INFO:tasks.workunit.client.0.smithi062.stdout:2/931: dwrite d7/da/d15/d2f/d38/d10f/d4b/f95 [0,4194304] 0 2024-04-20T21:32:18.029 INFO:tasks.workunit.client.0.smithi062.stdout:3/845: fdatasync d0/d1c/d63/d42/dc8/dc9/ff9 0 2024-04-20T21:32:18.029 INFO:tasks.workunit.client.0.smithi062.stdout:7/944: dread - d7/d3e/d33/d3d/d4d/fdf zero size 2024-04-20T21:32:18.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:17 smithi062 ceph-mon[16221]: pgmap v23: 65 pgs: 65 active+clean; 2.7 GiB data, 8.0 GiB used, 528 GiB / 536 GiB avail; 84 MiB/s rd, 207 MiB/s wr, 433 op/s 2024-04-20T21:32:18.035 INFO:tasks.workunit.client.0.smithi062.stdout:7/945: rmdir d7/d26/da8/db6 39 2024-04-20T21:32:18.036 INFO:tasks.workunit.client.0.smithi062.stdout:3/846: symlink d0/d1/d3c/d43/l114 0 2024-04-20T21:32:18.038 INFO:tasks.workunit.client.0.smithi062.stdout:5/903: dread d8/f4f [0,4194304] 0 2024-04-20T21:32:18.045 INFO:tasks.workunit.client.1.smithi189.stdout:6/691: mknod d8/da/dd/d35/d6d/d7b/ce0 0 2024-04-20T21:32:18.051 INFO:tasks.workunit.client.1.smithi189.stdout:5/664: rmdir d7/d9/d28 39 2024-04-20T21:32:18.054 INFO:tasks.workunit.client.0.smithi062.stdout:6/775: dread d0/d9/d11/f8d [0,4194304] 0 2024-04-20T21:32:18.055 INFO:tasks.workunit.client.1.smithi189.stdout:3/669: dread d2/d3/d2c/d31/fa7 [0,4194304] 0 2024-04-20T21:32:18.065 INFO:tasks.workunit.client.1.smithi189.stdout:2/617: rmdir d1/d17/d37/d89 39 2024-04-20T21:32:18.073 INFO:tasks.workunit.client.0.smithi062.stdout:6/776: write d0/dc7/d7c/dc5/def/f102 [740928,98107] 0 2024-04-20T21:32:18.074 INFO:tasks.workunit.client.1.smithi189.stdout:8/739: mknod da/de/d1e/d69/d70/c105 0 2024-04-20T21:32:18.075 INFO:tasks.workunit.client.0.smithi062.stdout:2/932: dwrite d7/da/d15/d2f/d97/f128 [0,4194304] 0 2024-04-20T21:32:18.076 INFO:tasks.workunit.client.0.smithi062.stdout:0/873: dwrite d7/d1c/f18 [4194304,4194304] 0 2024-04-20T21:32:18.082 INFO:tasks.workunit.client.1.smithi189.stdout:0/779: mknod d3/df/d28/d2b/db1/dfe/c104 0 2024-04-20T21:32:18.087 INFO:tasks.workunit.client.0.smithi062.stdout:7/946: dwrite d7/f55 [0,4194304] 0 2024-04-20T21:32:18.091 INFO:tasks.workunit.client.1.smithi189.stdout:5/665: rmdir d7/d9/d31/d36/d5f 39 2024-04-20T21:32:18.092 INFO:tasks.workunit.client.0.smithi062.stdout:0/874: fsync d7/d1e/fdc 0 2024-04-20T21:32:18.092 INFO:tasks.workunit.client.0.smithi062.stdout:5/904: dread d8/d38/d7a/d4c/d7d/d9d/dac/d66/fcb [0,4194304] 0 2024-04-20T21:32:18.094 INFO:tasks.workunit.client.0.smithi062.stdout:0/875: fdatasync d7/d1c/fc9 0 2024-04-20T21:32:18.099 INFO:tasks.workunit.client.0.smithi062.stdout:7/947: mkdir d7/d3e/d33/d149 0 2024-04-20T21:32:18.099 INFO:tasks.workunit.client.0.smithi062.stdout:6/777: creat d0/dc7/d7c/dc5/de9/f109 x:0 0 0 2024-04-20T21:32:18.101 INFO:tasks.workunit.client.0.smithi062.stdout:7/948: chown d7/d63/c123 61 1 2024-04-20T21:32:18.104 INFO:tasks.workunit.client.1.smithi189.stdout:3/670: creat d2/d3/d2c/d31/d74/db7/dd3/fe8 x:0 0 0 2024-04-20T21:32:18.106 INFO:tasks.workunit.client.0.smithi062.stdout:1/961: dread dc/d70/f90 [0,4194304] 0 2024-04-20T21:32:18.109 INFO:tasks.workunit.client.1.smithi189.stdout:2/618: symlink d1/d11/d2d/dab/lbb 0 2024-04-20T21:32:18.110 INFO:tasks.workunit.client.0.smithi062.stdout:4/962: dread da/dd/df/d27/d3f/d82/fbf [0,4194304] 0 2024-04-20T21:32:18.113 INFO:tasks.workunit.client.0.smithi062.stdout:2/933: dread d7/da/d15/d2f/d38/d10f/d107/dca/dd8/ddc/f116 [0,4194304] 0 2024-04-20T21:32:18.118 INFO:tasks.workunit.client.0.smithi062.stdout:1/962: creat dc/d15/d75/d96/dd4/d3a/da2/de4/d84/dbe/f161 x:0 0 0 2024-04-20T21:32:18.120 INFO:tasks.workunit.client.0.smithi062.stdout:0/876: dwrite d7/d1c/d15/d17/d40/f85 [0,4194304] 0 2024-04-20T21:32:18.120 INFO:tasks.workunit.client.1.smithi189.stdout:6/692: creat d8/d10/d78/d7e/dcf/fe1 x:0 0 0 2024-04-20T21:32:18.121 INFO:tasks.workunit.client.0.smithi062.stdout:2/934: chown d7/da/d15/d2f/d38/d51/f60 109102 1 2024-04-20T21:32:18.122 INFO:tasks.workunit.client.0.smithi062.stdout:4/963: write da/dd/d15/d50/d64/d3d/f5e [1802544,607] 0 2024-04-20T21:32:18.124 INFO:tasks.workunit.client.1.smithi189.stdout:5/666: creat d7/d9/d51/d70/d99/fd8 x:0 0 0 2024-04-20T21:32:18.125 INFO:tasks.workunit.client.1.smithi189.stdout:5/667: fsync d7/d35/d48/fb9 0 2024-04-20T21:32:18.127 INFO:tasks.workunit.client.0.smithi062.stdout:1/963: creat dc/d70/f162 x:0 0 0 2024-04-20T21:32:18.128 INFO:tasks.workunit.client.1.smithi189.stdout:7/716: rmdir d5/db4/d71/ddc 0 2024-04-20T21:32:18.129 INFO:tasks.workunit.client.0.smithi062.stdout:2/935: rmdir d7/da/d15/d2f/d38/d10f/d4b/d114 39 2024-04-20T21:32:18.129 INFO:tasks.workunit.client.1.smithi189.stdout:1/628: getdents d8/df/d50 0 2024-04-20T21:32:18.131 INFO:tasks.workunit.client.1.smithi189.stdout:3/671: symlink d2/d3/d2c/d31/d74/db7/dd3/le9 0 2024-04-20T21:32:18.132 INFO:tasks.workunit.client.1.smithi189.stdout:1/629: chown d8/d21/c97 5752 1 2024-04-20T21:32:18.133 INFO:tasks.workunit.client.0.smithi062.stdout:6/778: dwrite d0/d1a/d44/db3/fde [0,4194304] 0 2024-04-20T21:32:18.133 INFO:tasks.workunit.client.0.smithi062.stdout:2/936: readlink d7/da/d15/d1b/d24/l75 0 2024-04-20T21:32:18.133 INFO:tasks.workunit.client.0.smithi062.stdout:1/964: fsync dc/d15/f42 0 2024-04-20T21:32:18.135 INFO:tasks.workunit.client.1.smithi189.stdout:1/630: write d8/df/d3a/d49/d8c/fb9 [345332,3927] 0 2024-04-20T21:32:18.135 INFO:tasks.workunit.client.1.smithi189.stdout:9/608: rename d9/d12/d3b/db1/f30 to d9/dc/d17/d95/da0/d52/d82/d94/fc4 0 2024-04-20T21:32:18.137 INFO:tasks.workunit.client.1.smithi189.stdout:9/609: chown d9/dc/d17 10292 1 2024-04-20T21:32:18.137 INFO:tasks.workunit.client.0.smithi062.stdout:5/905: dread d8/d38/d7a/d4c/d7d/d9d/dac/d66/daa/ffa [0,4194304] 0 2024-04-20T21:32:18.139 INFO:tasks.workunit.client.0.smithi062.stdout:6/779: write d0/d1a/f105 [914223,34230] 0 2024-04-20T21:32:18.141 INFO:tasks.workunit.client.0.smithi062.stdout:2/937: unlink d7/f56 0 2024-04-20T21:32:18.142 INFO:tasks.workunit.client.0.smithi062.stdout:0/877: write d7/d1c/d15/d17/d26/f68 [319918,111892] 0 2024-04-20T21:32:18.142 INFO:tasks.workunit.client.1.smithi189.stdout:0/780: mkdir d3/df/d28/d2b/d2c/d105 0 2024-04-20T21:32:18.145 INFO:tasks.workunit.client.1.smithi189.stdout:6/693: mknod d8/da/ce2 0 2024-04-20T21:32:18.146 INFO:tasks.workunit.client.0.smithi062.stdout:1/965: truncate dc/df4/f15f 630336 0 2024-04-20T21:32:18.146 INFO:tasks.workunit.client.1.smithi189.stdout:5/668: creat d7/d35/fd9 x:0 0 0 2024-04-20T21:32:18.147 INFO:tasks.workunit.client.0.smithi062.stdout:5/906: dread f4 [0,4194304] 0 2024-04-20T21:32:18.148 INFO:tasks.workunit.client.0.smithi062.stdout:6/780: mknod d0/d9/df5/c10a 0 2024-04-20T21:32:18.148 INFO:tasks.workunit.client.1.smithi189.stdout:5/669: stat d7/d35/fd9 0 2024-04-20T21:32:18.149 INFO:tasks.workunit.client.0.smithi062.stdout:2/938: mkdir d7/da/d15/d2f/d97/d12f 0 2024-04-20T21:32:18.150 INFO:tasks.workunit.client.1.smithi189.stdout:5/670: chown d7/d9/c91 7 1 2024-04-20T21:32:18.153 INFO:tasks.workunit.client.1.smithi189.stdout:3/672: truncate d2/d3/d65/f78 2354023 0 2024-04-20T21:32:18.154 INFO:tasks.workunit.client.1.smithi189.stdout:1/631: mkdir d8/df/d3a/d49/d8c/da2/dbe/dc1/dd9 0 2024-04-20T21:32:18.156 INFO:tasks.workunit.client.0.smithi062.stdout:0/878: rename d7/d1c/f63 to d7/d1c/d15/d80/f115 0 2024-04-20T21:32:18.157 INFO:tasks.workunit.client.1.smithi189.stdout:4/627: write d2/d22/d57/f99 [136295,96623] 0 2024-04-20T21:32:18.157 INFO:tasks.workunit.client.1.smithi189.stdout:3/673: readlink d2/d3/d2c/d31/lbc 0 2024-04-20T21:32:18.159 INFO:tasks.workunit.client.0.smithi062.stdout:0/879: chown d7/d1c/d15/d17/d1b/c6d 18622 1 2024-04-20T21:32:18.159 INFO:tasks.workunit.client.0.smithi062.stdout:8/922: sync 2024-04-20T21:32:18.160 INFO:tasks.workunit.client.1.smithi189.stdout:5/671: mkdir d7/d9/d51/d70/d99/dd4/dda 0 2024-04-20T21:32:18.160 INFO:tasks.workunit.client.1.smithi189.stdout:4/628: dread - d2/d7f/fb4 zero size 2024-04-20T21:32:18.160 INFO:tasks.workunit.client.1.smithi189.stdout:1/632: truncate d8/d21/fb3 4957758 0 2024-04-20T21:32:18.161 INFO:tasks.workunit.client.1.smithi189.stdout:2/619: getdents d1/d11/d58 0 2024-04-20T21:32:18.162 INFO:tasks.workunit.client.0.smithi062.stdout:0/880: readlink d7/d20/d62/d6f/lc3 0 2024-04-20T21:32:18.163 INFO:tasks.workunit.client.1.smithi189.stdout:9/610: creat d9/dc/d17/d95/da0/fc5 x:0 0 0 2024-04-20T21:32:18.164 INFO:tasks.workunit.client.0.smithi062.stdout:6/781: rmdir d0/d1a/d7d/db1/db9 39 2024-04-20T21:32:18.165 INFO:tasks.workunit.client.0.smithi062.stdout:1/966: mknod dc/d70/d7f/d13b/c163 0 2024-04-20T21:32:18.166 INFO:tasks.workunit.client.0.smithi062.stdout:8/923: write d7/d25/d3b/d58/d8b/d24/d46/fd2 [814597,74283] 0 2024-04-20T21:32:18.170 INFO:tasks.workunit.client.0.smithi062.stdout:5/907: getdents d8/d1e/d2c 0 2024-04-20T21:32:18.171 INFO:tasks.workunit.client.1.smithi189.stdout:8/740: dwrite da/de/d1e/d21/d44/f4a [0,4194304] 0 2024-04-20T21:32:18.172 INFO:tasks.workunit.client.0.smithi062.stdout:0/881: truncate d7/d1e/fdc 1420201 0 2024-04-20T21:32:18.174 INFO:tasks.workunit.client.0.smithi062.stdout:8/924: truncate d7/f120 475311 0 2024-04-20T21:32:18.175 INFO:tasks.workunit.client.1.smithi189.stdout:6/694: read d8/d10/f3f [3500744,102061] 0 2024-04-20T21:32:18.176 INFO:tasks.workunit.client.1.smithi189.stdout:3/674: unlink d2/d3/d2c/d31/d74/le6 0 2024-04-20T21:32:18.176 INFO:tasks.workunit.client.0.smithi062.stdout:1/967: creat dc/d27/d32/df8/f164 x:0 0 0 2024-04-20T21:32:18.177 INFO:tasks.workunit.client.1.smithi189.stdout:4/629: symlink d2/d20/dc6/le5 0 2024-04-20T21:32:18.178 INFO:tasks.workunit.client.0.smithi062.stdout:8/925: fsync d7/d9/f17 0 2024-04-20T21:32:18.180 INFO:tasks.workunit.client.1.smithi189.stdout:5/672: read d7/d9/d28/f7b [2625678,50769] 0 2024-04-20T21:32:18.181 INFO:tasks.workunit.client.0.smithi062.stdout:0/882: unlink d7/d1c/d15/d17/lc6 0 2024-04-20T21:32:18.189 INFO:tasks.workunit.client.0.smithi062.stdout:4/964: dread da/dd/d15/d4b/fd9 [0,4194304] 0 2024-04-20T21:32:18.189 INFO:tasks.workunit.client.0.smithi062.stdout:1/968: truncate fb 2121169 0 2024-04-20T21:32:18.189 INFO:tasks.workunit.client.0.smithi062.stdout:0/883: rmdir d7/d1c/d15/d17/d40/d79/d71 39 2024-04-20T21:32:18.192 INFO:tasks.workunit.client.1.smithi189.stdout:9/611: dwrite d9/dc/f1c [4194304,4194304] 0 2024-04-20T21:32:18.193 INFO:tasks.workunit.client.1.smithi189.stdout:0/781: dread d3/df/d30/d37/f65 [0,4194304] 0 2024-04-20T21:32:18.197 INFO:tasks.workunit.client.0.smithi062.stdout:6/782: dwrite d0/d9/d11/d22/dd5/f8b [0,4194304] 0 2024-04-20T21:32:18.197 INFO:tasks.workunit.client.0.smithi062.stdout:0/884: chown d7/d1c/d15/d17/d37/f61 2931 1 2024-04-20T21:32:18.199 INFO:tasks.workunit.client.1.smithi189.stdout:9/612: readlink d9/dc/d17/d95/da0/d56/l93 0 2024-04-20T21:32:18.204 INFO:tasks.workunit.client.0.smithi062.stdout:0/885: fsync d7/d1c/d15/d17/d37/f59 0 2024-04-20T21:32:18.206 INFO:tasks.workunit.client.0.smithi062.stdout:2/939: dread d7/f5d [0,4194304] 0 2024-04-20T21:32:18.207 INFO:tasks.workunit.client.1.smithi189.stdout:8/741: dwrite da/df/d3c/d68/d8b/fc1 [0,4194304] 0 2024-04-20T21:32:18.213 INFO:tasks.workunit.client.0.smithi062.stdout:6/783: mknod d0/d1a/d7d/db1/db9/c10b 0 2024-04-20T21:32:18.217 INFO:tasks.workunit.client.1.smithi189.stdout:4/630: dwrite d2/dd/d11/d14/f79 [0,4194304] 0 2024-04-20T21:32:18.221 INFO:tasks.workunit.client.1.smithi189.stdout:2/620: fsync d1/d17/d37/f75 0 2024-04-20T21:32:18.222 INFO:tasks.workunit.client.0.smithi062.stdout:0/886: fsync d7/d1c/d103/f44 0 2024-04-20T21:32:18.222 INFO:tasks.workunit.client.1.smithi189.stdout:3/675: creat d2/d3/d2c/d31/d74/db7/dd3/de2/fea x:0 0 0 2024-04-20T21:32:18.222 INFO:tasks.workunit.client.1.smithi189.stdout:8/742: creat da/df/d13/dd2/f106 x:0 0 0 2024-04-20T21:32:18.226 INFO:tasks.workunit.client.1.smithi189.stdout:1/633: getdents d8/d1e/d43/d44/dd2/db7 0 2024-04-20T21:32:18.230 INFO:tasks.workunit.client.1.smithi189.stdout:7/717: dread d5/f43 [0,4194304] 0 2024-04-20T21:32:18.231 INFO:tasks.workunit.client.1.smithi189.stdout:4/631: creat d2/d22/d57/fe6 x:0 0 0 2024-04-20T21:32:18.233 INFO:tasks.workunit.client.0.smithi062.stdout:6/784: mkdir d0/dc7/d7c/dc5/de9/d10c 0 2024-04-20T21:32:18.233 INFO:tasks.workunit.client.1.smithi189.stdout:3/676: stat d2/d8 0 2024-04-20T21:32:18.234 INFO:tasks.workunit.client.1.smithi189.stdout:3/677: fdatasync d2/d3/f1b 0 2024-04-20T21:32:18.235 INFO:tasks.workunit.client.1.smithi189.stdout:9/613: creat d9/d12/d3f/d72/fc6 x:0 0 0 2024-04-20T21:32:18.236 INFO:tasks.workunit.client.1.smithi189.stdout:5/673: write d7/d14/f4c [2882879,108715] 0 2024-04-20T21:32:18.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:17 smithi189 ceph-mon[21562]: pgmap v23: 65 pgs: 65 active+clean; 2.7 GiB data, 8.0 GiB used, 528 GiB / 536 GiB avail; 84 MiB/s rd, 207 MiB/s wr, 433 op/s 2024-04-20T21:32:18.238 INFO:tasks.workunit.client.1.smithi189.stdout:8/743: fsync f9 0 2024-04-20T21:32:18.238 INFO:tasks.workunit.client.1.smithi189.stdout:0/782: dread d3/df/d30/d45/d90/fbf [0,4194304] 0 2024-04-20T21:32:18.238 INFO:tasks.workunit.client.1.smithi189.stdout:7/718: chown d5/d21/d97/da3 71133 1 2024-04-20T21:32:18.244 INFO:tasks.workunit.client.1.smithi189.stdout:1/634: creat d8/d1e/d43/d44/dd2/d8a/fda x:0 0 0 2024-04-20T21:32:18.244 INFO:tasks.workunit.client.1.smithi189.stdout:7/719: write d5/d46/d7d/d45/f48 [1054806,63637] 0 2024-04-20T21:32:18.247 INFO:tasks.workunit.client.1.smithi189.stdout:3/678: read - d2/d3/d2c/d31/d74/db7/dd3/fe8 zero size 2024-04-20T21:32:18.247 INFO:tasks.workunit.client.0.smithi062.stdout:5/908: dread d8/d10/f74 [0,4194304] 0 2024-04-20T21:32:18.249 INFO:tasks.workunit.client.1.smithi189.stdout:0/783: unlink d3/df/d28/f6a 0 2024-04-20T21:32:18.251 INFO:tasks.workunit.client.0.smithi062.stdout:5/909: stat d8/d10/d3e/c67 0 2024-04-20T21:32:18.251 INFO:tasks.workunit.client.1.smithi189.stdout:7/720: readlink d5/d21/d4a/d52/d74/l9a 0 2024-04-20T21:32:18.252 INFO:tasks.workunit.client.1.smithi189.stdout:8/744: truncate da/df/d13/dd6/d59/fdb 777400 0 2024-04-20T21:32:18.253 INFO:tasks.workunit.client.1.smithi189.stdout:9/614: truncate d9/dc/d69/f76 212552 0 2024-04-20T21:32:18.254 INFO:tasks.workunit.client.1.smithi189.stdout:9/615: fsync d9/dc/f3d 0 2024-04-20T21:32:18.255 INFO:tasks.workunit.client.0.smithi062.stdout:6/785: dread d0/d1a/d44/da7/f100 [0,4194304] 0 2024-04-20T21:32:18.255 INFO:tasks.workunit.client.1.smithi189.stdout:5/674: creat d7/d9/d31/d36/d5f/dab/fdb x:0 0 0 2024-04-20T21:32:18.257 INFO:tasks.workunit.client.0.smithi062.stdout:8/926: dwrite d7/d25/d3b/d58/db9/ffb [0,4194304] 0 2024-04-20T21:32:18.259 INFO:tasks.workunit.client.0.smithi062.stdout:0/887: dread d7/d1e/f45 [0,4194304] 0 2024-04-20T21:32:18.260 INFO:tasks.workunit.client.1.smithi189.stdout:5/675: write d7/d9/d28/f9b [1889429,104719] 0 2024-04-20T21:32:18.263 INFO:tasks.workunit.client.0.smithi062.stdout:6/786: stat d0/d1a/d44/ce1 0 2024-04-20T21:32:18.263 INFO:tasks.workunit.client.1.smithi189.stdout:7/721: dread d5/d1d/d77/fc1 [0,4194304] 0 2024-04-20T21:32:18.265 INFO:tasks.workunit.client.1.smithi189.stdout:8/745: dread da/df/d13/dd2/f8a [0,4194304] 0 2024-04-20T21:32:18.266 INFO:tasks.workunit.client.1.smithi189.stdout:5/676: dread - d7/d9/d31/d36/d5f/dab/fdb zero size 2024-04-20T21:32:18.269 INFO:tasks.workunit.client.1.smithi189.stdout:5/677: chown d7/d35/d48/f6a 108557350 1 2024-04-20T21:32:18.269 INFO:tasks.workunit.client.0.smithi062.stdout:6/787: readlink d0/d1a/d44/ld7 0 2024-04-20T21:32:18.273 INFO:tasks.workunit.client.0.smithi062.stdout:1/969: dwrite dc/d15/d75/d96/fc7 [0,4194304] 0 2024-04-20T21:32:18.273 INFO:tasks.workunit.client.0.smithi062.stdout:2/940: dwrite d7/da/d15/d2f/d38/d51/fa7 [0,4194304] 0 2024-04-20T21:32:18.274 INFO:tasks.workunit.client.1.smithi189.stdout:8/746: mknod da/df/d3c/d68/d8b/d8d/c107 0 2024-04-20T21:32:18.274 INFO:tasks.workunit.client.1.smithi189.stdout:5/678: creat d7/d9/d51/fdc x:0 0 0 2024-04-20T21:32:18.274 INFO:tasks.workunit.client.1.smithi189.stdout:8/747: fdatasync da/df/d22/faf 0 2024-04-20T21:32:18.278 INFO:tasks.workunit.client.1.smithi189.stdout:7/722: creat d5/d21/d25/d34/fec x:0 0 0 2024-04-20T21:32:18.278 INFO:tasks.workunit.client.0.smithi062.stdout:3/847: sync 2024-04-20T21:32:18.284 INFO:tasks.workunit.client.1.smithi189.stdout:7/723: write d5/db4/d71/fc7 [1248528,78325] 0 2024-04-20T21:32:18.286 INFO:tasks.workunit.client.0.smithi062.stdout:6/788: creat d0/d9/d11/d7b/d99/ddd/f10d x:0 0 0 2024-04-20T21:32:18.286 INFO:tasks.workunit.client.0.smithi062.stdout:4/965: dwrite da/dd/d15/d50/feb [0,4194304] 0 2024-04-20T21:32:18.287 INFO:tasks.workunit.client.1.smithi189.stdout:8/748: symlink da/df/d3c/d50/l108 0 2024-04-20T21:32:18.287 INFO:tasks.workunit.client.1.smithi189.stdout:5/679: mknod d7/d9/d31/d36/cdd 0 2024-04-20T21:32:18.294 INFO:tasks.workunit.client.0.smithi062.stdout:4/966: readlink da/dd/d15/d50/d64/d57/d5b/da3/l120 0 2024-04-20T21:32:18.299 INFO:tasks.workunit.client.0.smithi062.stdout:3/848: mkdir d0/d1/d115 0 2024-04-20T21:32:18.299 INFO:tasks.workunit.client.1.smithi189.stdout:7/724: mkdir d5/d1d/ded 0 2024-04-20T21:32:18.302 INFO:tasks.workunit.client.0.smithi062.stdout:4/967: mknod da/dd/d15/c14b 0 2024-04-20T21:32:18.305 INFO:tasks.workunit.client.0.smithi062.stdout:2/941: creat d7/da/d15/d2f/d38/d10f/f130 x:0 0 0 2024-04-20T21:32:18.305 INFO:tasks.workunit.client.1.smithi189.stdout:4/632: dread d2/d7f/f8f [0,4194304] 0 2024-04-20T21:32:18.311 INFO:tasks.workunit.client.1.smithi189.stdout:7/725: creat d5/d21/d97/fee x:0 0 0 2024-04-20T21:32:18.312 INFO:tasks.workunit.client.0.smithi062.stdout:3/849: mknod d0/d1/d10/d112/de1/c116 0 2024-04-20T21:32:18.313 INFO:tasks.workunit.client.1.smithi189.stdout:0/784: dread d3/fe [0,4194304] 0 2024-04-20T21:32:18.313 INFO:tasks.workunit.client.0.smithi062.stdout:8/927: truncate d7/d25/d4f/fdb 199614 0 2024-04-20T21:32:18.315 INFO:tasks.workunit.client.1.smithi189.stdout:5/680: creat d7/d9/d31/d36/fde x:0 0 0 2024-04-20T21:32:18.318 INFO:tasks.workunit.client.0.smithi062.stdout:3/850: fsync d0/f53 0 2024-04-20T21:32:18.318 INFO:tasks.workunit.client.0.smithi062.stdout:8/928: chown d7/d25/d4f 81 1 2024-04-20T21:32:18.318 INFO:tasks.workunit.client.1.smithi189.stdout:5/681: write d7/d9/d30/d55/d61/daa/fae [59651,15453] 0 2024-04-20T21:32:18.319 INFO:tasks.workunit.client.0.smithi062.stdout:2/942: symlink d7/da/d15/d2f/d38/d10f/d107/dca/dd8/l131 0 2024-04-20T21:32:18.320 INFO:tasks.workunit.client.0.smithi062.stdout:4/968: symlink da/dd/df/d27/d7a/d125/l14c 0 2024-04-20T21:32:18.324 INFO:tasks.workunit.client.1.smithi189.stdout:5/682: write d7/d9/d30/f1c [59181,80567] 0 2024-04-20T21:32:18.326 INFO:tasks.workunit.client.0.smithi062.stdout:6/789: dwrite d0/d9/d11/d22/dd5/d67/ffa [0,4194304] 0 2024-04-20T21:32:18.329 INFO:tasks.workunit.client.1.smithi189.stdout:1/635: dread d8/d21/d2c/f75 [0,4194304] 0 2024-04-20T21:32:18.331 INFO:tasks.workunit.client.1.smithi189.stdout:8/749: rename da/de/l1d to da/de/l109 0 2024-04-20T21:32:18.332 INFO:tasks.workunit.client.0.smithi062.stdout:4/969: creat da/dd/df/d27/d32/d97/f14d x:0 0 0 2024-04-20T21:32:18.334 INFO:tasks.workunit.client.0.smithi062.stdout:6/790: truncate d0/d9/d11/d7b/fb2 1049230 0 2024-04-20T21:32:18.334 INFO:tasks.workunit.client.0.smithi062.stdout:8/929: getdents d7/d30/db3/dbf 0 2024-04-20T21:32:18.335 INFO:tasks.workunit.client.1.smithi189.stdout:6/695: dread d8/da/f27 [0,4194304] 0 2024-04-20T21:32:18.336 INFO:tasks.workunit.client.1.smithi189.stdout:7/726: creat d5/d21/d97/da3/fef x:0 0 0 2024-04-20T21:32:18.339 INFO:tasks.workunit.client.1.smithi189.stdout:7/727: chown d5/db4/ld0 6 1 2024-04-20T21:32:18.339 INFO:tasks.workunit.client.1.smithi189.stdout:7/728: stat d5/d21 0 2024-04-20T21:32:18.350 INFO:tasks.workunit.client.1.smithi189.stdout:6/696: symlink d8/le3 0 2024-04-20T21:32:18.350 INFO:tasks.workunit.client.0.smithi062.stdout:6/791: rename d0/dc7/l7e to d0/d1a/d44/l10e 0 2024-04-20T21:32:18.352 INFO:tasks.workunit.client.1.smithi189.stdout:5/683: dwrite d7/d9/d31/d36/d5f/dab/fdb [0,4194304] 0 2024-04-20T21:32:18.353 INFO:tasks.workunit.client.1.smithi189.stdout:8/750: rename da/df/d13/dd6/d59/ffb to da/df/d3c/d68/d8b/f10a 0 2024-04-20T21:32:18.354 INFO:tasks.workunit.client.0.smithi062.stdout:2/943: dwrite d7/da/d15/d2f/d38/d6a/f110 [0,4194304] 0 2024-04-20T21:32:18.354 INFO:tasks.workunit.client.0.smithi062.stdout:5/910: dread d8/d10/d3e/f52 [0,4194304] 0 2024-04-20T21:32:18.356 INFO:tasks.workunit.client.1.smithi189.stdout:5/684: chown d7/d9/d30 41398 1 2024-04-20T21:32:18.357 INFO:tasks.workunit.client.0.smithi062.stdout:1/970: dwrite dc/d15/d75/d96/dd4/d78/dab/fb7 [0,4194304] 0 2024-04-20T21:32:18.357 INFO:tasks.workunit.client.0.smithi062.stdout:6/792: unlink d0/d9a/f101 0 2024-04-20T21:32:18.360 INFO:tasks.workunit.client.1.smithi189.stdout:0/785: unlink d3/df/d28/d2b/d2c/l72 0 2024-04-20T21:32:18.362 INFO:tasks.workunit.client.0.smithi062.stdout:4/970: getdents da/dd/d15/d6d/df5 0 2024-04-20T21:32:18.364 INFO:tasks.workunit.client.1.smithi189.stdout:7/729: dwrite d5/d21/d25/fde [0,4194304] 0 2024-04-20T21:32:18.368 INFO:tasks.workunit.client.1.smithi189.stdout:1/636: mkdir d8/d1e/d28/dd4/ddb 0 2024-04-20T21:32:18.370 INFO:tasks.workunit.client.0.smithi062.stdout:6/793: unlink d0/d9/d11/l97 0 2024-04-20T21:32:18.381 INFO:tasks.workunit.client.1.smithi189.stdout:5/685: dread d7/d9/d28/da3/fc3 [0,4194304] 0 2024-04-20T21:32:18.383 INFO:tasks.workunit.client.0.smithi062.stdout:1/971: getdents dc/d15/d75/d96 0 2024-04-20T21:32:18.384 INFO:tasks.workunit.client.1.smithi189.stdout:5/686: write d7/d35/d48/f6a [826939,42745] 0 2024-04-20T21:32:18.388 INFO:tasks.workunit.client.1.smithi189.stdout:0/786: rename d3/df/d5d/l5f to d3/df/d30/d37/dbd/l106 0 2024-04-20T21:32:18.392 INFO:tasks.workunit.client.0.smithi062.stdout:8/930: dwrite d7/d25/d3b/d58/d8b/d24/dbd/f109 [0,4194304] 0 2024-04-20T21:32:18.392 INFO:tasks.workunit.client.1.smithi189.stdout:1/637: creat d8/d21/fdc x:0 0 0 2024-04-20T21:32:18.392 INFO:tasks.workunit.client.1.smithi189.stdout:1/638: stat d8/d21 0 2024-04-20T21:32:18.395 INFO:tasks.workunit.client.0.smithi062.stdout:1/972: stat dc/d15/d75/d96/dd4/d3a/da2/de4/d84/dbe/d119/d15e/dd2/de3/d121/d14d/d14e/f152 0 2024-04-20T21:32:18.397 INFO:tasks.workunit.client.1.smithi189.stdout:5/687: rename d7/d9/d30/d55/d61/daa/db5 to d7/d9/d51/ddf 0 2024-04-20T21:32:18.397 INFO:tasks.workunit.client.1.smithi189.stdout:4/633: write d2/f19 [4672656,32597] 0 2024-04-20T21:32:18.398 INFO:tasks.workunit.client.1.smithi189.stdout:5/688: readlink l0 0 2024-04-20T21:32:18.402 INFO:tasks.workunit.client.0.smithi062.stdout:8/931: unlink d7/d9/f18 0 2024-04-20T21:32:18.403 INFO:tasks.workunit.client.1.smithi189.stdout:7/730: creat d5/d1d/d64/db9/dcf/de7/ff0 x:0 0 0 2024-04-20T21:32:18.404 INFO:tasks.workunit.client.1.smithi189.stdout:0/787: creat d3/df/d28/d2b/db1/f107 x:0 0 0 2024-04-20T21:32:18.405 INFO:tasks.workunit.client.0.smithi062.stdout:3/851: dread d0/d11/fa2 [0,4194304] 0 2024-04-20T21:32:18.405 INFO:tasks.workunit.client.1.smithi189.stdout:1/639: read - d8/df/d3a/f78 zero size 2024-04-20T21:32:18.413 INFO:tasks.workunit.client.1.smithi189.stdout:5/689: mkdir d7/d9/d31/d36/de0 0 2024-04-20T21:32:18.413 INFO:tasks.workunit.client.1.smithi189.stdout:7/731: chown d5/d21/d25/f30 204209334 1 2024-04-20T21:32:18.414 INFO:tasks.workunit.client.1.smithi189.stdout:0/788: creat d3/df/d28/d2b/d2c/f108 x:0 0 0 2024-04-20T21:32:18.417 INFO:tasks.workunit.client.0.smithi062.stdout:5/911: dwrite d8/d10/d3e/d8a/fa0 [0,4194304] 0 2024-04-20T21:32:18.419 INFO:tasks.workunit.client.1.smithi189.stdout:1/640: mknod d8/d1e/d54/cdd 0 2024-04-20T21:32:18.421 INFO:tasks.workunit.client.1.smithi189.stdout:8/751: write da/df/d3a/f5d [206280,90354] 0 2024-04-20T21:32:18.422 INFO:tasks.workunit.client.0.smithi062.stdout:4/971: dwrite da/dd/d15/d50/d64/d57/d5b/d9d/dea/f53 [0,4194304] 0 2024-04-20T21:32:18.422 INFO:tasks.workunit.client.0.smithi062.stdout:6/794: dwrite d0/d9/d11/d22/dd5/d67/ffb [0,4194304] 0 2024-04-20T21:32:18.436 INFO:tasks.workunit.client.0.smithi062.stdout:1/973: truncate dc/d15/d75/d96/dd4/d3a/da2/de4/d84/fb0 167936 0 2024-04-20T21:32:18.438 INFO:tasks.workunit.client.0.smithi062.stdout:1/974: chown dc/d15/d20/d28/d83/dc0/d132/d149 7192753 1 2024-04-20T21:32:18.440 INFO:tasks.workunit.client.1.smithi189.stdout:5/690: dwrite d7/d9/d30/f8e [0,4194304] 0 2024-04-20T21:32:18.445 INFO:tasks.workunit.client.1.smithi189.stdout:7/732: getdents d5/d21/d97/da3/dc6 0 2024-04-20T21:32:18.446 INFO:tasks.workunit.client.0.smithi062.stdout:1/975: read - dc/d15/d75/d96/dd4/d3a/da2/de4/d84/dbe/d119/d15e/f99 zero size 2024-04-20T21:32:18.449 INFO:tasks.workunit.client.1.smithi189.stdout:5/691: unlink d7/d9/d30/d55/d61/d96/cbc 0 2024-04-20T21:32:18.449 INFO:tasks.workunit.client.0.smithi062.stdout:1/976: mkdir dc/d15/d75/d165 0 2024-04-20T21:32:18.450 INFO:tasks.workunit.client.1.smithi189.stdout:7/733: stat d5/d21/d35/ld2 0 2024-04-20T21:32:18.452 INFO:tasks.workunit.client.1.smithi189.stdout:7/734: stat d5/db4/fe5 0 2024-04-20T21:32:18.457 INFO:tasks.workunit.client.1.smithi189.stdout:6/697: dwrite d8/d12/daf/fba [0,4194304] 0 2024-04-20T21:32:18.460 INFO:tasks.workunit.client.1.smithi189.stdout:0/789: dwrite d3/fe [0,4194304] 0 2024-04-20T21:32:18.461 INFO:tasks.workunit.client.0.smithi062.stdout:8/932: dwrite d7/d9/d29/f4b [0,4194304] 0 2024-04-20T21:32:18.462 INFO:tasks.workunit.client.1.smithi189.stdout:7/735: creat d5/d21/d4a/d52/ff1 x:0 0 0 2024-04-20T21:32:18.470 INFO:tasks.workunit.client.0.smithi062.stdout:8/933: symlink d7/d25/dc5/l12e 0 2024-04-20T21:32:18.473 INFO:tasks.workunit.client.1.smithi189.stdout:6/698: fdatasync d8/d10/f36 0 2024-04-20T21:32:18.475 INFO:tasks.workunit.client.1.smithi189.stdout:4/634: dread d2/dd/d6b/f72 [0,4194304] 0 2024-04-20T21:32:18.476 INFO:tasks.workunit.client.0.smithi062.stdout:1/977: dwrite dc/d15/d20/d28/f4e [0,4194304] 0 2024-04-20T21:32:18.478 INFO:tasks.workunit.client.1.smithi189.stdout:4/635: stat d2/dd/d37/dcd/d6a/c93 0 2024-04-20T21:32:18.485 INFO:tasks.workunit.client.0.smithi062.stdout:1/978: creat dc/d15/d75/d96/dd4/d3a/da2/f166 x:0 0 0 2024-04-20T21:32:18.489 INFO:tasks.workunit.client.1.smithi189.stdout:0/790: creat d3/df/d30/d37/dbd/de1/f109 x:0 0 0 2024-04-20T21:32:18.490 INFO:tasks.workunit.client.1.smithi189.stdout:0/791: fdatasync d3/df/d30/d45/d90/fc9 0 2024-04-20T21:32:18.491 INFO:tasks.workunit.client.0.smithi062.stdout:5/912: dwrite d8/d10/f17 [4194304,4194304] 0 2024-04-20T21:32:18.493 INFO:tasks.workunit.client.1.smithi189.stdout:4/636: symlink d2/dd/d37/dcd/le7 0 2024-04-20T21:32:18.493 INFO:tasks.workunit.client.0.smithi062.stdout:1/979: read - dc/d15/d75/d96/dd4/d3a/da2/de4/d84/dbe/d119/d15e/dd2/de3/d121/d14d/d14e/f152 zero size 2024-04-20T21:32:18.494 INFO:tasks.workunit.client.1.smithi189.stdout:7/736: mknod d5/d1d/ded/cf2 0 2024-04-20T21:32:18.494 INFO:tasks.workunit.client.0.smithi062.stdout:5/913: readlink d8/d10/d3e/lb0 0 2024-04-20T21:32:18.495 INFO:tasks.workunit.client.1.smithi189.stdout:0/792: mkdir d3/df/d28/dc8/d10a 0 2024-04-20T21:32:18.496 INFO:tasks.workunit.client.1.smithi189.stdout:6/699: rename d8/da/dd/d35/d6d/cbe to d8/d10/d78/ce4 0 2024-04-20T21:32:18.501 INFO:tasks.workunit.client.1.smithi189.stdout:4/637: mkdir d2/d22/d87/dcc/de8 0 2024-04-20T21:32:18.505 INFO:tasks.workunit.client.1.smithi189.stdout:0/793: rmdir d3/df/d5d 39 2024-04-20T21:32:18.507 INFO:tasks.workunit.client.0.smithi062.stdout:1/980: getdents dc/d15/d75/d96/dd4/d3a/da2/de4/d84/dbe/d119/d15e 0 2024-04-20T21:32:18.519 INFO:tasks.workunit.client.0.smithi062.stdout:4/972: dread da/dd/d15/d50/d64/d3d/fc6 [0,4194304] 0 2024-04-20T21:32:18.538 INFO:tasks.workunit.client.1.smithi189.stdout:2/621: sync 2024-04-20T21:32:18.540 INFO:tasks.workunit.client.0.smithi062.stdout:3/852: truncate d0/d1/d10/d3a/d5e/f8d 3246728 0 2024-04-20T21:32:18.542 INFO:tasks.workunit.client.1.smithi189.stdout:2/622: chown d1/d11/d2d/dab/d55/f81 57431738 1 2024-04-20T21:32:18.542 INFO:tasks.workunit.client.1.smithi189.stdout:8/752: write da/df/d20/f79 [831154,83217] 0 2024-04-20T21:32:18.544 INFO:tasks.workunit.client.1.smithi189.stdout:2/623: stat d1/d17/f35 0 2024-04-20T21:32:18.548 INFO:tasks.workunit.client.1.smithi189.stdout:8/753: write da/df/d13/dd2/fbe [222240,11009] 0 2024-04-20T21:32:18.553 INFO:tasks.workunit.client.0.smithi062.stdout:6/795: dwrite d0/d9/d11/d22/f27 [0,4194304] 0 2024-04-20T21:32:18.554 INFO:tasks.workunit.client.1.smithi189.stdout:8/754: creat da/df/d20/dc9/f10b x:0 0 0 2024-04-20T21:32:18.555 INFO:tasks.workunit.client.1.smithi189.stdout:7/737: dread d5/f20 [0,4194304] 0 2024-04-20T21:32:18.557 INFO:tasks.workunit.client.1.smithi189.stdout:7/738: chown d5/d46 1603 1 2024-04-20T21:32:18.558 INFO:tasks.workunit.client.1.smithi189.stdout:2/624: creat d1/d11/d2d/d4f/fbc x:0 0 0 2024-04-20T21:32:18.558 INFO:tasks.workunit.client.1.smithi189.stdout:9/616: sync 2024-04-20T21:32:18.559 INFO:tasks.workunit.client.1.smithi189.stdout:3/679: sync 2024-04-20T21:32:18.559 INFO:tasks.workunit.client.1.smithi189.stdout:1/641: sync 2024-04-20T21:32:18.559 INFO:tasks.workunit.client.0.smithi062.stdout:8/934: write d7/d9/ff [2363195,72358] 0 2024-04-20T21:32:18.560 INFO:tasks.workunit.client.0.smithi062.stdout:4/973: write da/dd/d15/d50/d64/d57/d5b/f68 [1374740,70598] 0 2024-04-20T21:32:18.561 INFO:tasks.workunit.client.1.smithi189.stdout:5/692: write d7/d14/d18/f3e [3752806,64140] 0 2024-04-20T21:32:18.566 INFO:tasks.workunit.client.0.smithi062.stdout:3/853: write d0/d1c/d63/dda/fed [710351,104516] 0 2024-04-20T21:32:18.572 INFO:tasks.workunit.client.1.smithi189.stdout:4/638: write d2/dd/d11/f74 [126293,38444] 0 2024-04-20T21:32:18.573 INFO:tasks.workunit.client.0.smithi062.stdout:8/935: dread - d7/d25/d3b/d58/d8b/d24/d46/f117 zero size 2024-04-20T21:32:18.573 INFO:tasks.workunit.client.0.smithi062.stdout:5/914: dread d8/d38/d7a/d4c/d7d/d9d/dac/d2f/f4b [0,4194304] 0 2024-04-20T21:32:18.575 INFO:tasks.workunit.client.1.smithi189.stdout:3/680: mknod d2/d1c/ceb 0 2024-04-20T21:32:18.576 INFO:tasks.workunit.client.1.smithi189.stdout:6/700: truncate f2 867968 0 2024-04-20T21:32:18.578 INFO:tasks.workunit.client.1.smithi189.stdout:5/693: creat d7/d9/d30/d55/d61/d87/fe1 x:0 0 0 2024-04-20T21:32:18.579 INFO:tasks.workunit.client.1.smithi189.stdout:3/681: chown d2/d3/d25/la8 86941 1 2024-04-20T21:32:18.579 INFO:tasks.workunit.client.0.smithi062.stdout:1/981: dwrite dc/d15/d75/d96/dd4/d78/fc8 [0,4194304] 0 2024-04-20T21:32:18.580 INFO:tasks.workunit.client.1.smithi189.stdout:2/625: dread d1/d11/d2d/fa6 [0,4194304] 0 2024-04-20T21:32:18.580 INFO:tasks.workunit.client.0.smithi062.stdout:4/974: rename da/dd/d15/d50/d64/d57/d5b/d9d/le0 to da/d4f/l14e 0 2024-04-20T21:32:18.582 INFO:tasks.workunit.client.1.smithi189.stdout:3/682: fsync d2/d1c/d5c/fbd 0 2024-04-20T21:32:18.584 INFO:tasks.workunit.client.0.smithi062.stdout:8/936: mknod d7/d25/d3b/d12c/c12f 0 2024-04-20T21:32:18.584 INFO:tasks.workunit.client.1.smithi189.stdout:0/794: dread d3/df/d28/dc8/df1/f44 [4194304,4194304] 0 2024-04-20T21:32:18.586 INFO:tasks.workunit.client.0.smithi062.stdout:8/937: chown d7/d25/d76 1 1 2024-04-20T21:32:18.588 INFO:tasks.workunit.client.1.smithi189.stdout:8/755: rename da/de/fed to da/de/d1e/d69/f10c 0 2024-04-20T21:32:18.590 INFO:tasks.workunit.client.0.smithi062.stdout:6/796: dwrite d0/d1a/d44/da7/f100 [0,4194304] 0 2024-04-20T21:32:18.593 INFO:tasks.workunit.client.1.smithi189.stdout:2/626: truncate d1/d11/d99/db7/fb8 953851 0 2024-04-20T21:32:18.593 INFO:tasks.workunit.client.1.smithi189.stdout:1/642: dwrite d8/f9 [0,4194304] 0 2024-04-20T21:32:18.594 INFO:tasks.workunit.client.0.smithi062.stdout:5/915: creat d8/d38/d7a/f12d x:0 0 0 2024-04-20T21:32:18.597 INFO:tasks.workunit.client.0.smithi062.stdout:8/938: symlink d7/d25/d70/l130 0 2024-04-20T21:32:18.598 INFO:tasks.workunit.client.1.smithi189.stdout:2/627: dread - d1/d11/d2d/dab/d74/fb3 zero size 2024-04-20T21:32:18.601 INFO:tasks.workunit.client.0.smithi062.stdout:7/949: sync 2024-04-20T21:32:18.601 INFO:tasks.workunit.client.0.smithi062.stdout:1/982: link dc/d27/d32/df8/c138 dc/d15/d20/d28/c167 0 2024-04-20T21:32:18.603 INFO:tasks.workunit.client.0.smithi062.stdout:8/939: chown d7/d25/d3b/d58/dac/f128 44327920 1 2024-04-20T21:32:18.603 INFO:tasks.workunit.client.1.smithi189.stdout:3/683: dread d2/d3/d2c/d30/d3f/f8a [0,4194304] 0 2024-04-20T21:32:18.606 INFO:tasks.workunit.client.1.smithi189.stdout:4/639: rename d2/d64 to d2/dd/d37/dcd/d6a/d7a/de9 0 2024-04-20T21:32:18.606 INFO:tasks.workunit.client.1.smithi189.stdout:0/795: creat d3/df/d28/dc8/d10a/f10b x:0 0 0 2024-04-20T21:32:18.608 INFO:tasks.workunit.client.1.smithi189.stdout:6/701: creat d8/d2b/fe5 x:0 0 0 2024-04-20T21:32:18.608 INFO:tasks.workunit.client.1.smithi189.stdout:8/756: symlink da/df/d20/d2c/d9a/ddd/l10d 0 2024-04-20T21:32:18.612 INFO:tasks.workunit.client.1.smithi189.stdout:3/684: symlink d2/d3/d65/lec 0 2024-04-20T21:32:18.613 INFO:tasks.workunit.client.1.smithi189.stdout:8/757: write da/d1f/d2a/ff9 [740487,72957] 0 2024-04-20T21:32:18.613 INFO:tasks.workunit.client.0.smithi062.stdout:5/916: symlink d8/d38/d7a/d4c/d7d/d9d/dac/d2f/df3/l12e 0 2024-04-20T21:32:18.617 INFO:tasks.workunit.client.1.smithi189.stdout:5/694: dwrite d7/d9/d28/fa0 [0,4194304] 0 2024-04-20T21:32:18.618 INFO:tasks.workunit.client.0.smithi062.stdout:3/854: truncate d0/d1b/f35 3450786 0 2024-04-20T21:32:18.618 INFO:tasks.workunit.client.1.smithi189.stdout:2/628: rename d1/fa9 to d1/d11/d52/fbd 0 2024-04-20T21:32:18.618 INFO:tasks.workunit.client.1.smithi189.stdout:4/640: write d2/d22/d57/fe6 [553342,90677] 0 2024-04-20T21:32:18.625 INFO:tasks.workunit.client.1.smithi189.stdout:8/758: mknod da/df/d3c/d68/df5/c10e 0 2024-04-20T21:32:18.627 INFO:tasks.workunit.client.1.smithi189.stdout:5/695: fdatasync d7/d35/f43 0 2024-04-20T21:32:18.627 INFO:tasks.workunit.client.1.smithi189.stdout:4/641: chown d2/d3a/f63 8023302 1 2024-04-20T21:32:18.628 INFO:tasks.workunit.client.1.smithi189.stdout:1/643: dwrite d8/d1e/f87 [0,4194304] 0 2024-04-20T21:32:18.629 INFO:tasks.workunit.client.0.smithi062.stdout:6/797: dwrite d0/d1a/d44/db3/f106 [0,4194304] 0 2024-04-20T21:32:18.629 INFO:tasks.workunit.client.0.smithi062.stdout:0/888: sync 2024-04-20T21:32:18.631 INFO:tasks.workunit.client.0.smithi062.stdout:6/798: chown d0/d4f/l9b 2814509 1 2024-04-20T21:32:18.633 INFO:tasks.workunit.client.0.smithi062.stdout:0/889: readlink d7/d1c/d15/d17/d40/d3c/dc5/l7c 0 2024-04-20T21:32:18.634 INFO:tasks.workunit.client.1.smithi189.stdout:0/796: getdents d3/df/d30 0 2024-04-20T21:32:18.636 INFO:tasks.workunit.client.1.smithi189.stdout:7/739: dread d5/d21/d25/f30 [0,4194304] 0 2024-04-20T21:32:18.637 INFO:tasks.workunit.client.1.smithi189.stdout:5/696: chown d7/d9/d30/d55/d61/d96/fd3 1 1 2024-04-20T21:32:18.637 INFO:tasks.workunit.client.0.smithi062.stdout:0/890: chown d7/d20/d62/l8a 5 1 2024-04-20T21:32:18.643 INFO:tasks.workunit.client.0.smithi062.stdout:3/855: unlink d0/d1/la1 0 2024-04-20T21:32:18.646 INFO:tasks.workunit.client.0.smithi062.stdout:7/950: mknod d7/d3e/d33/d3d/d4d/d54/d4f/de9/df1/df3/c14a 0 2024-04-20T21:32:18.647 INFO:tasks.workunit.client.1.smithi189.stdout:2/629: dread d1/d17/f2b [0,4194304] 0 2024-04-20T21:32:18.647 INFO:tasks.workunit.client.0.smithi062.stdout:0/891: creat d7/d20/d62/d6f/f116 x:0 0 0 2024-04-20T21:32:18.650 INFO:tasks.workunit.client.1.smithi189.stdout:2/630: chown d1/d11/d2d/dab/d4a/c91 10 1 2024-04-20T21:32:18.651 INFO:tasks.workunit.client.1.smithi189.stdout:4/642: mkdir d2/dd/d37/dcd/d6a/d7a/db7/dea 0 2024-04-20T21:32:18.653 INFO:tasks.workunit.client.0.smithi062.stdout:0/892: fsync d7/d1e/db6/fd3 0 2024-04-20T21:32:18.656 INFO:tasks.workunit.client.1.smithi189.stdout:5/697: truncate d7/d9/d28/f6f 937357 0 2024-04-20T21:32:18.656 INFO:tasks.workunit.client.1.smithi189.stdout:0/797: creat d3/da8/f10c x:0 0 0 2024-04-20T21:32:18.660 INFO:tasks.workunit.client.1.smithi189.stdout:0/798: stat d3/df/d30/d37/l80 0 2024-04-20T21:32:18.662 INFO:tasks.workunit.client.1.smithi189.stdout:3/685: getdents d2/d3/d25/de0 0 2024-04-20T21:32:18.667 INFO:tasks.workunit.client.0.smithi062.stdout:7/951: getdents d7/d26/dac/ddb/d132 0 2024-04-20T21:32:18.670 INFO:tasks.workunit.client.0.smithi062.stdout:0/893: getdents d7/d1c/d15/d17/d40/d79 0 2024-04-20T21:32:18.671 INFO:tasks.workunit.client.1.smithi189.stdout:1/644: mknod d8/cde 0 2024-04-20T21:32:18.672 INFO:tasks.workunit.client.0.smithi062.stdout:6/799: dwrite d0/d9/d11/d22/dd5/d67/ffb [0,4194304] 0 2024-04-20T21:32:18.673 INFO:tasks.workunit.client.1.smithi189.stdout:7/740: dwrite d5/db4/fe5 [0,4194304] 0 2024-04-20T21:32:18.673 INFO:tasks.workunit.client.1.smithi189.stdout:3/686: truncate d2/d3/d2c/d30/d3f/fb3 885083 0 2024-04-20T21:32:18.673 INFO:tasks.workunit.client.1.smithi189.stdout:5/698: mknod d7/d9/d31/d36/d5f/ce2 0 2024-04-20T21:32:18.679 INFO:tasks.workunit.client.1.smithi189.stdout:0/799: mknod d3/db9/ded/c10d 0 2024-04-20T21:32:18.680 INFO:tasks.workunit.client.1.smithi189.stdout:2/631: creat d1/d11/d2d/d4f/fbe x:0 0 0 2024-04-20T21:32:18.681 INFO:tasks.workunit.client.0.smithi062.stdout:4/975: dread da/dd/d15/d50/d64/d3d/da1/fb9 [0,4194304] 0 2024-04-20T21:32:18.689 INFO:tasks.workunit.client.0.smithi062.stdout:6/800: creat d0/d9/d11/d22/dd5/f10f x:0 0 0 2024-04-20T21:32:18.690 INFO:tasks.workunit.client.0.smithi062.stdout:3/856: dread d0/d1/d10/d31/f9e [0,4194304] 0 2024-04-20T21:32:18.690 INFO:tasks.workunit.client.1.smithi189.stdout:6/702: dread d8/d12/f9d [0,4194304] 0 2024-04-20T21:32:18.694 INFO:tasks.workunit.client.1.smithi189.stdout:3/687: rename d2/d3/d2c/d30/ca6 to d2/d3/d2c/d31/d90/d92/ced 0 2024-04-20T21:32:18.696 INFO:tasks.workunit.client.0.smithi062.stdout:4/976: write da/dd/df/d27/d7a/d91/f9f [801218,77548] 0 2024-04-20T21:32:18.696 INFO:tasks.workunit.client.0.smithi062.stdout:3/857: write d0/fdc [1466423,39636] 0 2024-04-20T21:32:18.699 INFO:tasks.workunit.client.1.smithi189.stdout:3/688: chown d2/d3/d2c/d30/l72 362415 1 2024-04-20T21:32:18.699 INFO:tasks.workunit.client.1.smithi189.stdout:7/741: dread d5/d21/f23 [0,4194304] 0 2024-04-20T21:32:18.700 INFO:tasks.workunit.client.1.smithi189.stdout:5/699: dwrite d7/d14/d4e/d5b/fc2 [0,4194304] 0 2024-04-20T21:32:18.705 INFO:tasks.workunit.client.0.smithi062.stdout:8/940: dwrite d7/d30/f95 [0,4194304] 0 2024-04-20T21:32:18.705 INFO:tasks.workunit.client.0.smithi062.stdout:4/977: creat da/dd/f14f x:0 0 0 2024-04-20T21:32:18.707 INFO:tasks.workunit.client.1.smithi189.stdout:1/645: link d8/d1e/d54/d93/fbd d8/d1e/fdf 0 2024-04-20T21:32:18.708 INFO:tasks.workunit.client.0.smithi062.stdout:1/983: dwrite dc/d15/d75/d96/dd4/d3a/da2/de4/d84/dbe/d119/d15e/dd2/de3/d121/f14c [0,4194304] 0 2024-04-20T21:32:18.709 INFO:tasks.workunit.client.0.smithi062.stdout:0/894: dwrite d7/d20/d62/d6f/f105 [0,4194304] 0 2024-04-20T21:32:18.710 INFO:tasks.workunit.client.0.smithi062.stdout:8/941: chown d7/d25/d3b/d58/f6b 0 1 2024-04-20T21:32:18.713 INFO:tasks.workunit.client.1.smithi189.stdout:6/703: chown d8/d10/d78/ce4 7898809 1 2024-04-20T21:32:18.718 INFO:tasks.workunit.client.0.smithi062.stdout:1/984: mkdir dc/d70/d7f/d168 0 2024-04-20T21:32:18.718 INFO:tasks.workunit.client.0.smithi062.stdout:8/942: chown d7/c7b 19689467 1 2024-04-20T21:32:18.718 INFO:tasks.workunit.client.1.smithi189.stdout:2/632: symlink d1/d17/d37/lbf 0 2024-04-20T21:32:18.719 INFO:tasks.workunit.client.1.smithi189.stdout:8/759: dwrite da/df/d3c/d50/d6c/dc7/fcd [0,4194304] 0 2024-04-20T21:32:18.719 INFO:tasks.workunit.client.1.smithi189.stdout:9/617: sync 2024-04-20T21:32:18.728 INFO:tasks.workunit.client.0.smithi062.stdout:0/895: creat d7/d1c/d15/d80/f117 x:0 0 0 2024-04-20T21:32:18.729 INFO:tasks.workunit.client.1.smithi189.stdout:1/646: unlink l4 0 2024-04-20T21:32:18.729 INFO:tasks.workunit.client.1.smithi189.stdout:7/742: truncate d5/d1d/f56 5064452 0 2024-04-20T21:32:18.733 INFO:tasks.workunit.client.1.smithi189.stdout:6/704: mknod d8/d2b/d49/d54/ce6 0 2024-04-20T21:32:18.733 INFO:tasks.workunit.client.0.smithi062.stdout:8/943: fsync d7/d30/fba 0 2024-04-20T21:32:18.734 INFO:tasks.workunit.client.0.smithi062.stdout:1/985: mknod dc/d27/d9a/da0/db3/c169 0 2024-04-20T21:32:18.735 INFO:tasks.workunit.client.0.smithi062.stdout:6/801: write d0/d1a/f29 [1955536,63011] 0 2024-04-20T21:32:18.736 INFO:tasks.workunit.client.0.smithi062.stdout:0/896: chown d7/d1e/df9 876 1 2024-04-20T21:32:18.736 INFO:tasks.workunit.client.1.smithi189.stdout:9/618: unlink d9/dc/d17/d95/da0/d56/f73 0 2024-04-20T21:32:18.737 INFO:tasks.workunit.client.1.smithi189.stdout:1/647: write d8/d1e/d43/d44/dd2/fc7 [247518,20298] 0 2024-04-20T21:32:18.738 INFO:tasks.workunit.client.1.smithi189.stdout:2/633: chown d1/d17/d78/ca2 951786 1 2024-04-20T21:32:18.739 INFO:tasks.workunit.client.0.smithi062.stdout:3/858: dread d0/d1/d3c/d43/d72/dd3/d7b/fb6 [0,4194304] 0 2024-04-20T21:32:18.739 INFO:tasks.workunit.client.1.smithi189.stdout:3/689: rename d2/f5d to d2/d3/d2c/fee 0 2024-04-20T21:32:18.741 INFO:tasks.workunit.client.1.smithi189.stdout:8/760: creat da/df/d22/f10f x:0 0 0 2024-04-20T21:32:18.741 INFO:tasks.workunit.client.1.smithi189.stdout:3/690: write d2/d8d/fba [332843,124937] 0 2024-04-20T21:32:18.745 INFO:tasks.workunit.client.0.smithi062.stdout:1/986: mkdir dc/d27/d9a/da0/d16a 0 2024-04-20T21:32:18.747 INFO:tasks.workunit.client.1.smithi189.stdout:1/648: creat d8/d9b/fe0 x:0 0 0 2024-04-20T21:32:18.748 INFO:tasks.workunit.client.1.smithi189.stdout:2/634: write d1/d11/d2d/d4f/db4/f9c [202231,24265] 0 2024-04-20T21:32:18.748 INFO:tasks.workunit.client.1.smithi189.stdout:3/691: read - d2/d3/d88/f99 zero size 2024-04-20T21:32:18.749 INFO:tasks.workunit.client.0.smithi062.stdout:4/978: dwrite da/dd/f13b [0,4194304] 0 2024-04-20T21:32:18.749 INFO:tasks.workunit.client.1.smithi189.stdout:7/743: rename d5/d1d/caf to d5/db4/d71/d7a/db0/cf3 0 2024-04-20T21:32:18.751 INFO:tasks.workunit.client.0.smithi062.stdout:0/897: mknod d7/d1c/d15/d17/d40/d79/c118 0 2024-04-20T21:32:18.752 INFO:tasks.workunit.client.1.smithi189.stdout:1/649: unlink d8/f40 0 2024-04-20T21:32:18.753 INFO:tasks.workunit.client.1.smithi189.stdout:3/692: write d2/d1c/d5c/f87 [797600,130276] 0 2024-04-20T21:32:18.753 INFO:tasks.workunit.client.0.smithi062.stdout:6/802: symlink d0/d1a/l110 0 2024-04-20T21:32:18.755 INFO:tasks.workunit.client.1.smithi189.stdout:1/650: fdatasync d8/d1e/d28/f91 0 2024-04-20T21:32:18.756 INFO:tasks.workunit.client.0.smithi062.stdout:3/859: creat d0/d1/d3c/d43/d72/da9/f117 x:0 0 0 2024-04-20T21:32:18.759 INFO:tasks.workunit.client.0.smithi062.stdout:8/944: truncate d7/d25/d3b/d58/d8b/d24/d46/fd2 2404683 0 2024-04-20T21:32:18.760 INFO:tasks.workunit.client.0.smithi062.stdout:7/952: dread d7/d63/fa7 [0,4194304] 0 2024-04-20T21:32:18.761 INFO:tasks.workunit.client.1.smithi189.stdout:6/705: rename d8/d10/d78/f96 to d8/d2b/d49/d54/fe7 0 2024-04-20T21:32:18.761 INFO:tasks.workunit.client.0.smithi062.stdout:6/803: dread - d0/d9/d11/d22/fbe zero size 2024-04-20T21:32:18.763 INFO:tasks.workunit.client.0.smithi062.stdout:3/860: fdatasync d0/d1c/d63/f91 0 2024-04-20T21:32:18.763 INFO:tasks.workunit.client.0.smithi062.stdout:0/898: dread d7/d1c/d15/d17/d37/d95/fd6 [0,4194304] 0 2024-04-20T21:32:18.765 INFO:tasks.workunit.client.0.smithi062.stdout:6/804: read - d0/dc7/d7c/dc5/def/f108 zero size 2024-04-20T21:32:18.766 INFO:tasks.workunit.client.1.smithi189.stdout:3/693: mknod d2/d3/d2c/d30/d3f/cef 0 2024-04-20T21:32:18.772 INFO:tasks.workunit.client.0.smithi062.stdout:6/805: truncate d0/f3e 947212 0 2024-04-20T21:32:18.772 INFO:tasks.workunit.client.1.smithi189.stdout:7/744: rename d5/d21/d97/da3/dc6/fcb to d5/d59/ff4 0 2024-04-20T21:32:18.773 INFO:tasks.workunit.client.0.smithi062.stdout:8/945: creat d7/d25/d3b/f131 x:0 0 0 2024-04-20T21:32:18.777 INFO:tasks.workunit.client.0.smithi062.stdout:8/946: write d7/d25/d70/fe4 [2008,53108] 0 2024-04-20T21:32:18.782 INFO:tasks.workunit.client.0.smithi062.stdout:3/861: getdents d0/d1/d10 0 2024-04-20T21:32:18.783 INFO:tasks.workunit.client.0.smithi062.stdout:6/806: symlink d0/d9/d11/d22/dd5/d67/l111 0 2024-04-20T21:32:18.783 INFO:tasks.workunit.client.0.smithi062.stdout:3/862: readlink d0/d1c/la5 0 2024-04-20T21:32:18.784 INFO:tasks.workunit.client.0.smithi062.stdout:4/979: dwrite da/dd/d15/d6d/df5/ff8 [0,4194304] 0 2024-04-20T21:32:18.787 INFO:tasks.workunit.client.1.smithi189.stdout:1/651: symlink d8/d21/dbf/le1 0 2024-04-20T21:32:18.788 INFO:tasks.workunit.client.1.smithi189.stdout:5/700: dwrite d7/d14/f8b [0,4194304] 0 2024-04-20T21:32:18.788 INFO:tasks.workunit.client.1.smithi189.stdout:3/694: mknod d2/d3/d2c/d31/d74/cf0 0 2024-04-20T21:32:18.793 INFO:tasks.workunit.client.0.smithi062.stdout:0/899: write d7/d1c/d15/d17/d37/d95/ff0 [908658,63326] 0 2024-04-20T21:32:18.794 INFO:tasks.workunit.client.1.smithi189.stdout:8/761: dread da/d1f/d2a/f58 [0,4194304] 0 2024-04-20T21:32:18.798 INFO:tasks.workunit.client.1.smithi189.stdout:8/762: chown da/de/lab 44 1 2024-04-20T21:32:18.798 INFO:tasks.workunit.client.1.smithi189.stdout:2/635: getdents d1 0 2024-04-20T21:32:18.799 INFO:tasks.workunit.client.1.smithi189.stdout:2/636: fdatasync d1/d11/d2d/d4f/fbe 0 2024-04-20T21:32:18.800 INFO:tasks.workunit.client.0.smithi062.stdout:0/900: creat d7/d20/f119 x:0 0 0 2024-04-20T21:32:18.804 INFO:tasks.workunit.client.0.smithi062.stdout:7/953: dwrite d7/ff [0,4194304] 0 2024-04-20T21:32:18.807 INFO:tasks.workunit.client.1.smithi189.stdout:5/701: creat d7/d9/d51/d70/d99/dd4/fe3 x:0 0 0 2024-04-20T21:32:18.807 INFO:tasks.workunit.client.1.smithi189.stdout:5/702: stat l0 0 2024-04-20T21:32:18.811 INFO:tasks.workunit.client.1.smithi189.stdout:3/695: mkdir d2/d3/d2c/d31/d74/d7a/df1 0 2024-04-20T21:32:18.813 INFO:tasks.workunit.client.0.smithi062.stdout:7/954: creat d7/d26/dac/d10a/f14b x:0 0 0 2024-04-20T21:32:18.813 INFO:tasks.workunit.client.1.smithi189.stdout:8/763: readlink da/de/l109 0 2024-04-20T21:32:18.815 INFO:tasks.workunit.client.1.smithi189.stdout:9/619: dwrite d9/dc/f2c [0,4194304] 0 2024-04-20T21:32:18.818 INFO:tasks.workunit.client.1.smithi189.stdout:1/652: symlink d8/df/le2 0 2024-04-20T21:32:18.819 INFO:tasks.workunit.client.1.smithi189.stdout:9/620: readlink d9/d12/l48 0 2024-04-20T21:32:18.821 INFO:tasks.workunit.client.0.smithi062.stdout:2/944: sync 2024-04-20T21:32:18.823 INFO:tasks.workunit.client.0.smithi062.stdout:3/863: dwrite d0/d1/db7/fc3 [0,4194304] 0 2024-04-20T21:32:18.825 INFO:tasks.workunit.client.1.smithi189.stdout:7/745: creat d5/d21/d25/ff5 x:0 0 0 2024-04-20T21:32:18.825 INFO:tasks.workunit.client.1.smithi189.stdout:9/621: truncate d9/d12/d3b/db1/d1a/f27 1114927 0 2024-04-20T21:32:18.827 INFO:tasks.workunit.client.1.smithi189.stdout:8/764: dread - da/d81/d91/fc4 zero size 2024-04-20T21:32:18.828 INFO:tasks.workunit.client.1.smithi189.stdout:7/746: write d5/d1d/f79 [3897937,121010] 0 2024-04-20T21:32:18.829 INFO:tasks.workunit.client.0.smithi062.stdout:2/945: creat d7/da/d15/d2f/d97/f132 x:0 0 0 2024-04-20T21:32:18.830 INFO:tasks.workunit.client.1.smithi189.stdout:9/622: chown d9/dc/d17/cb6 7 1 2024-04-20T21:32:18.831 INFO:tasks.workunit.client.0.smithi062.stdout:0/901: write d7/d1c/d15/d17/d40/fff [127331,8043] 0 2024-04-20T21:32:18.834 INFO:tasks.workunit.client.1.smithi189.stdout:5/703: dwrite d7/d9/d30/d5a/d9d/fac [0,4194304] 0 2024-04-20T21:32:18.834 INFO:tasks.workunit.client.0.smithi062.stdout:3/864: mkdir d0/d1/de4/d118 0 2024-04-20T21:32:18.837 INFO:tasks.workunit.client.1.smithi189.stdout:5/704: fsync d7/d9/d30/d55/d61/daa/fad 0 2024-04-20T21:32:18.838 INFO:tasks.workunit.client.1.smithi189.stdout:3/696: symlink d2/d3/d25/de0/db0/dc8/dc9/lf2 0 2024-04-20T21:32:18.841 INFO:tasks.workunit.client.1.smithi189.stdout:3/697: fsync d2/d1c/d5c/fc3 0 2024-04-20T21:32:18.841 INFO:tasks.workunit.client.1.smithi189.stdout:8/765: mkdir da/df/d13/d63/d110 0 2024-04-20T21:32:18.841 INFO:tasks.workunit.client.0.smithi062.stdout:0/902: chown d7/d1c/d15/l9a 1527608 1 2024-04-20T21:32:18.845 INFO:tasks.workunit.client.0.smithi062.stdout:2/946: creat d7/f133 x:0 0 0 2024-04-20T21:32:18.848 INFO:tasks.workunit.client.1.smithi189.stdout:3/698: stat d2/d3/d2c/d31/d74/db7/dd3/fe8 0 2024-04-20T21:32:18.849 INFO:tasks.workunit.client.0.smithi062.stdout:7/955: dwrite d7/d3e/d33/d3d/d5e/ff9 [0,4194304] 0 2024-04-20T21:32:18.850 INFO:tasks.workunit.client.1.smithi189.stdout:1/653: dread d8/f14 [0,4194304] 0 2024-04-20T21:32:18.852 INFO:tasks.workunit.client.1.smithi189.stdout:3/699: write d2/d3/f3e [1795112,126463] 0 2024-04-20T21:32:18.853 INFO:tasks.workunit.client.0.smithi062.stdout:3/865: link d0/d1/d10/d3a/d5e/d65/fff d0/d1c/d63/d42/dc8/dc9/f119 0 2024-04-20T21:32:18.854 INFO:tasks.workunit.client.1.smithi189.stdout:8/766: symlink da/d1f/d2a/l111 0 2024-04-20T21:32:18.854 INFO:tasks.workunit.client.1.smithi189.stdout:5/705: creat d7/d9/d31/d36/d5f/dab/dcb/fe4 x:0 0 0 2024-04-20T21:32:18.855 INFO:tasks.workunit.client.1.smithi189.stdout:6/706: dwrite d8/f9b [0,4194304] 0 2024-04-20T21:32:18.860 INFO:tasks.workunit.client.0.smithi062.stdout:2/947: mknod d7/da/c134 0 2024-04-20T21:32:18.861 INFO:tasks.workunit.client.1.smithi189.stdout:1/654: unlink d8/d1e/d43/d44/dd2/l7d 0 2024-04-20T21:32:18.863 INFO:tasks.workunit.client.0.smithi062.stdout:7/956: write d7/d3e/d33/d3d/d5e/db8/f141 [1180315,20101] 0 2024-04-20T21:32:18.868 INFO:tasks.workunit.client.0.smithi062.stdout:2/948: unlink d7/da/d15/d2f/d38/d51/f105 0 2024-04-20T21:32:18.868 INFO:tasks.workunit.client.0.smithi062.stdout:3/866: rename d0/d1/d10/d3a/d5e/d65/d88 to d0/d1/d10/d3a/d5e/d65/d88/d11a 22 2024-04-20T21:32:18.870 INFO:tasks.workunit.client.0.smithi062.stdout:3/867: readlink d0/d1b/l8c 0 2024-04-20T21:32:18.871 INFO:tasks.workunit.client.0.smithi062.stdout:6/807: dread d0/d9/f13 [0,4194304] 0 2024-04-20T21:32:18.874 INFO:tasks.workunit.client.0.smithi062.stdout:7/957: creat d7/d3e/d33/d3d/d4d/f14c x:0 0 0 2024-04-20T21:32:18.876 INFO:tasks.workunit.client.0.smithi062.stdout:3/868: symlink d0/d1/de4/d118/l11b 0 2024-04-20T21:32:18.879 INFO:tasks.workunit.client.0.smithi062.stdout:7/958: chown d7/c121 3 1 2024-04-20T21:32:18.881 INFO:tasks.workunit.client.1.smithi189.stdout:5/706: truncate d7/d9/d31/d36/f54 320208 0 2024-04-20T21:32:18.882 INFO:tasks.workunit.client.1.smithi189.stdout:5/707: fdatasync d7/d9/d28/f6f 0 2024-04-20T21:32:18.882 INFO:tasks.workunit.client.1.smithi189.stdout:6/707: dread d8/da/dd/d35/f62 [0,4194304] 0 2024-04-20T21:32:18.887 INFO:tasks.workunit.client.0.smithi062.stdout:3/869: rename d0/d1/d3c/d43/d72/dd3/d7b to d0/d1/d115/d11c 0 2024-04-20T21:32:18.889 INFO:tasks.workunit.client.0.smithi062.stdout:6/808: link d0/d52/da0/fa8 d0/d9/d11/f112 0 2024-04-20T21:32:18.890 INFO:tasks.workunit.client.1.smithi189.stdout:8/767: creat da/df/d20/d65/f112 x:0 0 0 2024-04-20T21:32:18.891 INFO:tasks.workunit.client.1.smithi189.stdout:7/747: dread d5/d21/f32 [0,4194304] 0 2024-04-20T21:32:18.892 INFO:tasks.workunit.client.1.smithi189.stdout:8/768: write da/df/d3c/deb/f103 [1034584,43005] 0 2024-04-20T21:32:18.893 INFO:tasks.workunit.client.1.smithi189.stdout:5/708: creat d7/d9/d28/db1/fe5 x:0 0 0 2024-04-20T21:32:18.894 INFO:tasks.workunit.client.0.smithi062.stdout:3/870: mkdir d0/d1/d10/d112/d11d 0 2024-04-20T21:32:18.896 INFO:tasks.workunit.client.0.smithi062.stdout:2/949: dread d7/da/d15/d1b/d24/dee/f108 [0,4194304] 0 2024-04-20T21:32:18.901 INFO:tasks.workunit.client.0.smithi062.stdout:7/959: dread d7/d3e/d33/d3d/d4d/dce/ff8 [0,4194304] 0 2024-04-20T21:32:18.903 INFO:tasks.workunit.client.1.smithi189.stdout:1/655: dwrite d8/f24 [0,4194304] 0 2024-04-20T21:32:18.915 INFO:tasks.workunit.client.0.smithi062.stdout:3/871: rename d0/d1/d10/d3a/d5e/d65/d88 to d0/d1c/d63/d42/d11e 0 2024-04-20T21:32:18.915 INFO:tasks.workunit.client.0.smithi062.stdout:2/950: mknod d7/da/d15/d2f/d38/c135 0 2024-04-20T21:32:18.917 INFO:tasks.workunit.client.1.smithi189.stdout:4/643: sync 2024-04-20T21:32:18.917 INFO:tasks.workunit.client.1.smithi189.stdout:0/800: sync 2024-04-20T21:32:18.919 INFO:tasks.workunit.client.0.smithi062.stdout:0/903: write d7/d1c/d15/d17/d37/f113 [317673,115671] 0 2024-04-20T21:32:18.922 INFO:tasks.workunit.client.0.smithi062.stdout:6/809: dwrite d0/d9/d11/d22/dd5/d67/d75/fd9 [8388608,4194304] 0 2024-04-20T21:32:18.928 INFO:tasks.workunit.client.0.smithi062.stdout:2/951: getdents d7/da/d15/d2f/d38/d10f/d107 0 2024-04-20T21:32:18.928 INFO:tasks.workunit.client.0.smithi062.stdout:6/810: symlink d0/dc7/d7c/dc5/def/l113 0 2024-04-20T21:32:18.933 INFO:tasks.workunit.client.0.smithi062.stdout:6/811: read d0/d1a/d44/db3/f106 [3189340,103698] 0 2024-04-20T21:32:18.934 INFO:tasks.workunit.client.0.smithi062.stdout:2/952: mkdir d7/da/d15/d2f/d38/d136 0 2024-04-20T21:32:18.934 INFO:tasks.workunit.client.1.smithi189.stdout:6/708: dread d8/da/f6b [0,4194304] 0 2024-04-20T21:32:18.938 INFO:tasks.workunit.client.0.smithi062.stdout:5/917: sync 2024-04-20T21:32:18.940 INFO:tasks.workunit.client.1.smithi189.stdout:3/700: creat d2/d3/ff3 x:0 0 0 2024-04-20T21:32:18.941 INFO:tasks.workunit.client.0.smithi062.stdout:0/904: write d7/d1c/ff [517078,71556] 0 2024-04-20T21:32:18.943 INFO:tasks.workunit.client.1.smithi189.stdout:6/709: write d8/da/dd/d35/d6d/d77/fd5 [974703,39471] 0 2024-04-20T21:32:18.943 INFO:tasks.workunit.client.1.smithi189.stdout:3/701: dread - d2/d3/d88/f99 zero size 2024-04-20T21:32:18.943 INFO:tasks.workunit.client.1.smithi189.stdout:1/656: chown d8/d1e/d28/l71 14212641 1 2024-04-20T21:32:18.944 INFO:tasks.workunit.client.0.smithi062.stdout:2/953: mkdir d7/da/d15/d2f/d38/de6/d137 0 2024-04-20T21:32:18.945 INFO:tasks.workunit.client.1.smithi189.stdout:0/801: fsync d3/df/d28/d2b/fe2 0 2024-04-20T21:32:18.946 INFO:tasks.workunit.client.0.smithi062.stdout:0/905: mkdir d7/d1c/d15/d80/d11a 0 2024-04-20T21:32:18.948 INFO:tasks.workunit.client.0.smithi062.stdout:2/954: fsync d7/da/d15/d2f/d97/f9e 0 2024-04-20T21:32:18.949 INFO:tasks.workunit.client.0.smithi062.stdout:5/918: getdents d8/d38/d7a/d4c/d7d 0 2024-04-20T21:32:18.951 INFO:tasks.workunit.client.1.smithi189.stdout:0/802: symlink d3/df/d28/d2b/d2c/l10e 0 2024-04-20T21:32:18.951 INFO:tasks.workunit.client.0.smithi062.stdout:7/960: dwrite d7/d3e/d33/d3d/d5e/f90 [0,4194304] 0 2024-04-20T21:32:18.952 INFO:tasks.workunit.client.0.smithi062.stdout:2/955: dread - d7/db5/fff zero size 2024-04-20T21:32:18.954 INFO:tasks.workunit.client.1.smithi189.stdout:6/710: truncate d8/d10/f20 1626698 0 2024-04-20T21:32:18.956 INFO:tasks.workunit.client.0.smithi062.stdout:6/812: write d0/d9a/fae [2958908,117151] 0 2024-04-20T21:32:18.957 INFO:tasks.workunit.client.0.smithi062.stdout:7/961: chown d7/d3e/d33/d3d/d4d/dce 142135706 1 2024-04-20T21:32:18.958 INFO:tasks.workunit.client.1.smithi189.stdout:6/711: dread - d8/d12/d30/d47/da0/fc2 zero size 2024-04-20T21:32:18.960 INFO:tasks.workunit.client.0.smithi062.stdout:6/813: readlink d0/d9/d11/lf0 0 2024-04-20T21:32:18.965 INFO:tasks.workunit.client.1.smithi189.stdout:0/803: rmdir d3/df/d28/d2b/d2c/d105 0 2024-04-20T21:32:18.967 INFO:tasks.workunit.client.1.smithi189.stdout:1/657: dwrite d8/d1e/d43/d44/fc9 [0,4194304] 0 2024-04-20T21:32:18.968 INFO:tasks.workunit.client.1.smithi189.stdout:0/804: write d3/df/d28/d2b/d2c/f9e [4995589,110901] 0 2024-04-20T21:32:18.971 INFO:tasks.workunit.client.1.smithi189.stdout:6/712: creat d8/d12/d30/d47/ddc/fe8 x:0 0 0 2024-04-20T21:32:18.974 INFO:tasks.workunit.client.1.smithi189.stdout:0/805: creat d3/df/d30/d37/dbd/f10f x:0 0 0 2024-04-20T21:32:18.974 INFO:tasks.workunit.client.1.smithi189.stdout:1/658: symlink d8/le3 0 2024-04-20T21:32:18.978 INFO:tasks.workunit.client.1.smithi189.stdout:6/713: symlink d8/d12/d30/d47/da0/le9 0 2024-04-20T21:32:18.979 INFO:tasks.workunit.client.1.smithi189.stdout:1/659: unlink d8/df/d50/l6a 0 2024-04-20T21:32:18.982 INFO:tasks.workunit.client.1.smithi189.stdout:6/714: creat d8/da/dd/d35/d6d/d7b/fea x:0 0 0 2024-04-20T21:32:18.984 INFO:tasks.workunit.client.1.smithi189.stdout:3/702: getdents d2/d3 0 2024-04-20T21:32:18.985 INFO:tasks.workunit.client.1.smithi189.stdout:4/644: truncate d2/d22/d87/fa2 4022671 0 2024-04-20T21:32:18.986 INFO:tasks.workunit.client.1.smithi189.stdout:7/748: dwrite d5/d46/d7d/d45/f99 [4194304,4194304] 0 2024-04-20T21:32:18.987 INFO:tasks.workunit.client.1.smithi189.stdout:6/715: chown d8/da/dd/d35/cd3 2036 1 2024-04-20T21:32:18.996 INFO:tasks.workunit.client.1.smithi189.stdout:6/716: chown d8/da/dd/l33 23893 1 2024-04-20T21:32:18.997 INFO:tasks.workunit.client.1.smithi189.stdout:7/749: dread - d5/d1d/d64/fd7 zero size 2024-04-20T21:32:19.000 INFO:tasks.workunit.client.1.smithi189.stdout:0/806: dwrite d3/df/d5d/f7c [4194304,4194304] 0 2024-04-20T21:32:19.000 INFO:tasks.workunit.client.1.smithi189.stdout:3/703: truncate d2/d3/d2c/d31/d4c/f63 683857 0 2024-04-20T21:32:19.002 INFO:tasks.workunit.client.1.smithi189.stdout:4/645: write d2/dd/d6b/f72 [3024333,59010] 0 2024-04-20T21:32:19.004 INFO:tasks.workunit.client.1.smithi189.stdout:6/717: chown d8/d2b/d49/d54/ce6 217 1 2024-04-20T21:32:19.007 INFO:tasks.workunit.client.1.smithi189.stdout:1/660: dwrite d8/d1e/d43/d44/dd2/d8a/fda [0,4194304] 0 2024-04-20T21:32:19.019 INFO:tasks.workunit.client.1.smithi189.stdout:7/750: mknod d5/d21/d4a/cf6 0 2024-04-20T21:32:19.019 INFO:tasks.workunit.client.1.smithi189.stdout:3/704: creat d2/d3/d2c/ff4 x:0 0 0 2024-04-20T21:32:19.023 INFO:tasks.workunit.client.1.smithi189.stdout:6/718: dread d8/d2b/d49/d54/d67/fab [0,4194304] 0 2024-04-20T21:32:19.025 INFO:tasks.workunit.client.1.smithi189.stdout:7/751: write d5/d21/d4a/d52/d74/fbb [1085445,103308] 0 2024-04-20T21:32:19.025 INFO:tasks.workunit.client.1.smithi189.stdout:4/646: read d2/d3a/f4d [365980,107008] 0 2024-04-20T21:32:19.025 INFO:tasks.workunit.client.1.smithi189.stdout:3/705: creat d2/d3/d88/ff5 x:0 0 0 2024-04-20T21:32:19.030 INFO:tasks.workunit.client.1.smithi189.stdout:3/706: chown d2/d3/d2c/d31/f6f 7 1 2024-04-20T21:32:19.031 INFO:tasks.workunit.client.1.smithi189.stdout:7/752: symlink d5/d21/d4a/d52/d74/lf7 0 2024-04-20T21:32:19.032 INFO:tasks.workunit.client.1.smithi189.stdout:6/719: truncate d8/d2b/d49/f72 710146 0 2024-04-20T21:32:19.032 INFO:tasks.workunit.client.0.smithi062.stdout:5/919: write d8/d38/d7a/d4c/f7f [931397,24120] 0 2024-04-20T21:32:19.033 INFO:tasks.workunit.client.1.smithi189.stdout:2/637: sync 2024-04-20T21:32:19.034 INFO:tasks.workunit.client.0.smithi062.stdout:2/956: creat d7/da/d15/d2f/d38/db0/f138 x:0 0 0 2024-04-20T21:32:19.034 INFO:tasks.workunit.client.1.smithi189.stdout:3/707: creat d2/d3/d88/ff6 x:0 0 0 2024-04-20T21:32:19.034 INFO:tasks.workunit.client.1.smithi189.stdout:7/753: write d5/d21/d25/fde [2563648,3225] 0 2024-04-20T21:32:19.041 INFO:tasks.workunit.client.1.smithi189.stdout:6/720: creat d8/d12/d99/dd8/feb x:0 0 0 2024-04-20T21:32:19.042 INFO:tasks.workunit.client.0.smithi062.stdout:2/957: rename d7/da/fdf to d7/da/d15/d1b/df1/d10e/f139 0 2024-04-20T21:32:19.042 INFO:tasks.workunit.client.1.smithi189.stdout:3/708: creat d2/d3/d2c/d31/d74/db7/dd3/ff7 x:0 0 0 2024-04-20T21:32:19.045 INFO:tasks.workunit.client.0.smithi062.stdout:6/814: write d0/d9/f71 [314635,67095] 0 2024-04-20T21:32:19.045 INFO:tasks.workunit.client.1.smithi189.stdout:3/709: stat d2/d3/d65/c6c 0 2024-04-20T21:32:19.046 INFO:tasks.workunit.client.1.smithi189.stdout:6/721: chown d8/cd2 5 1 2024-04-20T21:32:19.048 INFO:tasks.workunit.client.1.smithi189.stdout:3/710: chown d2/d3/l16 11484948 1 2024-04-20T21:32:19.050 INFO:tasks.workunit.client.0.smithi062.stdout:6/815: read d0/d9/d11/f8d [342840,873] 0 2024-04-20T21:32:19.052 INFO:tasks.workunit.client.0.smithi062.stdout:6/816: readlink d0/d1a/d44/ld7 0 2024-04-20T21:32:19.053 INFO:tasks.workunit.client.1.smithi189.stdout:9/623: sync 2024-04-20T21:32:19.053 INFO:tasks.workunit.client.1.smithi189.stdout:5/709: sync 2024-04-20T21:32:19.054 INFO:tasks.workunit.client.1.smithi189.stdout:8/769: sync 2024-04-20T21:32:19.056 INFO:tasks.workunit.client.0.smithi062.stdout:5/920: dwrite d8/d10/d3e/d8a/d9e/dd7/f12c [0,4194304] 0 2024-04-20T21:32:19.057 INFO:tasks.workunit.client.1.smithi189.stdout:7/754: write d5/d21/d25/f6c [2852259,131026] 0 2024-04-20T21:32:19.060 INFO:tasks.workunit.client.1.smithi189.stdout:3/711: creat d2/d3/d2c/d31/d74/db7/dd3/de2/ff8 x:0 0 0 2024-04-20T21:32:19.060 INFO:tasks.workunit.client.1.smithi189.stdout:8/770: chown da/df/d13/d63/f7f 2943819 1 2024-04-20T21:32:19.060 INFO:tasks.workunit.client.1.smithi189.stdout:0/807: dwrite d3/d8c/f7a [0,4194304] 0 2024-04-20T21:32:19.061 INFO:tasks.workunit.client.0.smithi062.stdout:5/921: write d8/d38/d7a/d4c/f7f [1698772,123967] 0 2024-04-20T21:32:19.065 INFO:tasks.workunit.client.0.smithi062.stdout:5/922: chown d8/f101 5898 1 2024-04-20T21:32:19.065 INFO:tasks.workunit.client.1.smithi189.stdout:5/710: creat d7/d9/d30/d55/d33/fe6 x:0 0 0 2024-04-20T21:32:19.065 INFO:tasks.workunit.client.1.smithi189.stdout:0/808: chown d3/df/d28/d2b/d2c/l32 976 1 2024-04-20T21:32:19.066 INFO:tasks.workunit.client.0.smithi062.stdout:2/958: write d7/da/d15/d1b/d3d/d84/d9b/fda [397488,85406] 0 2024-04-20T21:32:19.067 INFO:tasks.workunit.client.1.smithi189.stdout:3/712: chown d2/d3/d2c/ff4 50 1 2024-04-20T21:32:19.067 INFO:tasks.workunit.client.1.smithi189.stdout:1/661: dwrite d8/d21/f22 [0,4194304] 0 2024-04-20T21:32:19.069 INFO:tasks.workunit.client.1.smithi189.stdout:0/809: chown d3/df/d28/f61 228 1 2024-04-20T21:32:19.073 INFO:tasks.workunit.client.1.smithi189.stdout:9/624: write d9/d12/d3f/d72/f7b [1453910,105305] 0 2024-04-20T21:32:19.075 INFO:tasks.workunit.client.0.smithi062.stdout:5/923: chown d8/d1e/d2c/d6c/la6 0 1 2024-04-20T21:32:19.075 INFO:tasks.workunit.client.1.smithi189.stdout:8/771: creat da/df/d3c/d9d/f113 x:0 0 0 2024-04-20T21:32:19.077 INFO:tasks.workunit.client.0.smithi062.stdout:6/817: dwrite d0/d9a/fae [0,4194304] 0 2024-04-20T21:32:19.080 INFO:tasks.workunit.client.1.smithi189.stdout:9/625: fsync d9/dc/d17/d7e/f8d 0 2024-04-20T21:32:19.081 INFO:tasks.workunit.client.1.smithi189.stdout:3/713: creat d2/d3/d2c/d31/ff9 x:0 0 0 2024-04-20T21:32:19.083 INFO:tasks.workunit.client.1.smithi189.stdout:2/638: dread d1/d11/d2d/f63 [0,4194304] 0 2024-04-20T21:32:19.084 INFO:tasks.workunit.client.1.smithi189.stdout:0/810: fsync d3/df/d30/d45/d90/fbf 0 2024-04-20T21:32:19.086 INFO:tasks.workunit.client.0.smithi062.stdout:2/959: rmdir d7/da/d15/d2f/d38/d10f/d107/dca/dd8/ddc 39 2024-04-20T21:32:19.088 INFO:tasks.workunit.client.0.smithi062.stdout:8/947: sync 2024-04-20T21:32:19.088 INFO:tasks.workunit.client.0.smithi062.stdout:1/987: sync 2024-04-20T21:32:19.089 INFO:tasks.workunit.client.0.smithi062.stdout:5/924: symlink d8/d38/d7a/d4c/dcf/l12f 0 2024-04-20T21:32:19.090 INFO:tasks.workunit.client.1.smithi189.stdout:2/639: chown d1/d11/d2d/d4f/db4 13151583 1 2024-04-20T21:32:19.090 INFO:tasks.workunit.client.0.smithi062.stdout:6/818: mknod d0/d4f/c114 0 2024-04-20T21:32:19.091 INFO:tasks.workunit.client.0.smithi062.stdout:4/980: sync 2024-04-20T21:32:19.092 INFO:tasks.workunit.client.1.smithi189.stdout:5/711: creat d7/d9/fe7 x:0 0 0 2024-04-20T21:32:19.093 INFO:tasks.workunit.client.0.smithi062.stdout:2/960: stat d7/da/d15/d2f/d38/d6b/c113 0 2024-04-20T21:32:19.094 INFO:tasks.workunit.client.0.smithi062.stdout:4/981: stat da/dd/cba 0 2024-04-20T21:32:19.094 INFO:tasks.workunit.client.1.smithi189.stdout:8/772: write da/df/d3c/d50/d6c/dc7/fcd [2242952,57174] 0 2024-04-20T21:32:19.095 INFO:tasks.workunit.client.1.smithi189.stdout:3/714: write d2/d1c/d5c/f87 [436902,62712] 0 2024-04-20T21:32:19.099 INFO:tasks.workunit.client.0.smithi062.stdout:2/961: stat d7/da/d15/d2f/d38/d6b/fa3 0 2024-04-20T21:32:19.100 INFO:tasks.workunit.client.1.smithi189.stdout:7/755: read d5/d21/d4a/d52/d75/fab [183452,112171] 0 2024-04-20T21:32:19.100 INFO:tasks.workunit.client.1.smithi189.stdout:1/662: rename d8/d1e/d54/da7/lbc to d8/df/le4 0 2024-04-20T21:32:19.101 INFO:tasks.workunit.client.0.smithi062.stdout:4/982: chown l9 1305 1 2024-04-20T21:32:19.104 INFO:tasks.workunit.client.0.smithi062.stdout:4/983: chown da/dd/cd3 21470867 1 2024-04-20T21:32:19.106 INFO:tasks.workunit.client.1.smithi189.stdout:4/647: dread d2/d7f/fc1 [0,4194304] 0 2024-04-20T21:32:19.108 INFO:tasks.workunit.client.0.smithi062.stdout:1/988: creat dc/d15/d75/d96/dd4/d3a/da2/de4/d84/dbe/d10d/d148/f16b x:0 0 0 2024-04-20T21:32:19.110 INFO:tasks.workunit.client.1.smithi189.stdout:7/756: write d5/d46/f8f [270624,101085] 0 2024-04-20T21:32:19.110 INFO:tasks.workunit.client.1.smithi189.stdout:4/648: fsync d2/f5e 0 2024-04-20T21:32:19.111 INFO:tasks.workunit.client.1.smithi189.stdout:3/715: dread - d2/d3/d2c/d31/d74/db7/fcf zero size 2024-04-20T21:32:19.112 INFO:tasks.workunit.client.0.smithi062.stdout:6/819: creat d0/d9/d11/d22/dd5/f115 x:0 0 0 2024-04-20T21:32:19.112 INFO:tasks.workunit.client.1.smithi189.stdout:5/712: unlink d7/d14/d4e/d5b/c65 0 2024-04-20T21:32:19.112 INFO:tasks.workunit.client.0.smithi062.stdout:2/962: getdents d7/da/d15/d2f/d38/de6/d137 0 2024-04-20T21:32:19.114 INFO:tasks.workunit.client.0.smithi062.stdout:2/963: readlink d7/da/d15/d2f/d93/l126 0 2024-04-20T21:32:19.114 INFO:tasks.workunit.client.1.smithi189.stdout:1/663: truncate d8/f12 1228665 0 2024-04-20T21:32:19.115 INFO:tasks.workunit.client.0.smithi062.stdout:5/925: link d8/d38/d7a/d4c/d7d/d9d/dac/c35 d8/d10/d3e/d8a/d9e/c130 0 2024-04-20T21:32:19.116 INFO:tasks.workunit.client.1.smithi189.stdout:8/773: mkdir da/df/d13/dd6/d114 0 2024-04-20T21:32:19.117 INFO:tasks.workunit.client.1.smithi189.stdout:6/722: dread d8/d2b/d49/d54/d67/f9c [0,4194304] 0 2024-04-20T21:32:19.118 INFO:tasks.workunit.client.0.smithi062.stdout:2/964: chown d7/da/d15/d1b/f34 330036409 1 2024-04-20T21:32:19.119 INFO:tasks.workunit.client.1.smithi189.stdout:7/757: unlink d5/d46/faa 0 2024-04-20T21:32:19.122 INFO:tasks.workunit.client.1.smithi189.stdout:8/774: chown da/lb0 389 1 2024-04-20T21:32:19.124 INFO:tasks.workunit.client.1.smithi189.stdout:5/713: read d7/d9/d31/f88 [2213897,63826] 0 2024-04-20T21:32:19.126 INFO:tasks.workunit.client.0.smithi062.stdout:5/926: rename d8/d38/d7a/f12d to d8/f131 0 2024-04-20T21:32:19.126 INFO:tasks.workunit.client.0.smithi062.stdout:8/948: dwrite d7/d9/d29/f3e [0,4194304] 0 2024-04-20T21:32:19.127 INFO:tasks.workunit.client.1.smithi189.stdout:3/716: creat d2/d3/d25/de0/db0/dc8/ffa x:0 0 0 2024-04-20T21:32:19.130 INFO:tasks.workunit.client.0.smithi062.stdout:8/949: write d7/d25/d3b/d58/d8b/d96/f11d [1042305,48717] 0 2024-04-20T21:32:19.132 INFO:tasks.workunit.client.1.smithi189.stdout:7/758: unlink d5/f6 0 2024-04-20T21:32:19.133 INFO:tasks.workunit.client.1.smithi189.stdout:0/811: rename d3/df/d30/d45/d90/cb3 to d3/df/d28/dc8/df1/c110 0 2024-04-20T21:32:19.134 INFO:tasks.workunit.client.0.smithi062.stdout:8/950: chown d7/d25/d3b/d58/d8b/ffe 38439 1 2024-04-20T21:32:19.134 INFO:tasks.workunit.client.0.smithi062.stdout:1/989: write dc/d15/f48 [2846969,48734] 0 2024-04-20T21:32:19.135 INFO:tasks.workunit.client.1.smithi189.stdout:2/640: dwrite d1/d11/d2d/dab/d55/f5f [0,4194304] 0 2024-04-20T21:32:19.136 INFO:tasks.workunit.client.0.smithi062.stdout:8/951: fsync d7/d25/d3b/d58/f110 0 2024-04-20T21:32:19.141 INFO:tasks.workunit.client.1.smithi189.stdout:0/812: stat d3/d8c/fc7 0 2024-04-20T21:32:19.142 INFO:tasks.workunit.client.1.smithi189.stdout:6/723: creat d8/d10/fec x:0 0 0 2024-04-20T21:32:19.142 INFO:tasks.workunit.client.1.smithi189.stdout:5/714: rmdir d7/d9/d31 39 2024-04-20T21:32:19.143 INFO:tasks.workunit.client.0.smithi062.stdout:5/927: rename d8/d38/d7a/d4c/l57 to d8/d38/d7a/d4c/d7d/d9d/dac/d66/dae/l132 0 2024-04-20T21:32:19.144 INFO:tasks.workunit.client.0.smithi062.stdout:2/965: getdents d7/da/d15/d2f/d38/db0/dc4 0 2024-04-20T21:32:19.145 INFO:tasks.workunit.client.0.smithi062.stdout:8/952: mkdir d7/d25/d5c/d132 0 2024-04-20T21:32:19.146 INFO:tasks.workunit.client.0.smithi062.stdout:1/990: mkdir dc/d15/d20/d28/d76/d137/d16c 0 2024-04-20T21:32:19.148 INFO:tasks.workunit.client.1.smithi189.stdout:3/717: rename d2/c1f to d2/d3/d2c/d31/d74/db7/dd3/de2/cfb 0 2024-04-20T21:32:19.149 INFO:tasks.workunit.client.1.smithi189.stdout:7/759: stat d5/d21/d25/c87 0 2024-04-20T21:32:19.150 INFO:tasks.workunit.client.1.smithi189.stdout:8/775: mknod da/df/d13/dd6/dba/dc5/c115 0 2024-04-20T21:32:19.154 INFO:tasks.workunit.client.1.smithi189.stdout:1/664: dwrite d8/d1e/f6e [0,4194304] 0 2024-04-20T21:32:19.154 INFO:tasks.workunit.client.1.smithi189.stdout:2/641: truncate d1/d11/d2d/dab/d55/fba 346303 0 2024-04-20T21:32:19.155 INFO:tasks.workunit.client.1.smithi189.stdout:6/724: mknod d8/d12/d30/d47/ddc/ced 0 2024-04-20T21:32:19.156 INFO:tasks.workunit.client.0.smithi062.stdout:2/966: rmdir d7/da/d15/d2f/d38/d136 0 2024-04-20T21:32:19.157 INFO:tasks.workunit.client.1.smithi189.stdout:9/626: dwrite d9/dc/d17/d95/da0/d52/f83 [0,4194304] 0 2024-04-20T21:32:19.161 INFO:tasks.workunit.client.1.smithi189.stdout:7/760: chown d5/d21/d4a/d52/d74/lf7 5704 1 2024-04-20T21:32:19.171 INFO:tasks.workunit.client.0.smithi062.stdout:5/928: dread d8/d1e/fba [0,4194304] 0 2024-04-20T21:32:19.172 INFO:tasks.workunit.client.1.smithi189.stdout:8/776: truncate da/df/d3c/d68/d8b/fc1 5080311 0 2024-04-20T21:32:19.173 INFO:tasks.workunit.client.1.smithi189.stdout:8/777: stat da/df/d13/d63/f7f 0 2024-04-20T21:32:19.175 INFO:tasks.workunit.client.0.smithi062.stdout:2/967: getdents d7/da/d15/d2f/d38/d10f/d107 0 2024-04-20T21:32:19.175 INFO:tasks.workunit.client.0.smithi062.stdout:8/953: dwrite d7/d9/d29/f4b [0,4194304] 0 2024-04-20T21:32:19.177 INFO:tasks.workunit.client.1.smithi189.stdout:0/813: write d3/df/d28/d2b/f86 [1267774,50969] 0 2024-04-20T21:32:19.179 INFO:tasks.workunit.client.1.smithi189.stdout:2/642: creat d1/d11/d2d/d4f/d8e/fc0 x:0 0 0 2024-04-20T21:32:19.185 INFO:tasks.workunit.client.0.smithi062.stdout:2/968: rename d7/da/c19 to d7/da/d15/d2f/d38/d6b/df6/c13a 0 2024-04-20T21:32:19.185 INFO:tasks.workunit.client.1.smithi189.stdout:1/665: creat d8/d1e/d28/d4f/fe5 x:0 0 0 2024-04-20T21:32:19.185 INFO:tasks.workunit.client.1.smithi189.stdout:9/627: creat d9/dc/d17/d95/da0/d52/d82/da4/fc7 x:0 0 0 2024-04-20T21:32:19.189 INFO:tasks.workunit.client.1.smithi189.stdout:8/778: mkdir da/d1f/d116 0 2024-04-20T21:32:19.190 INFO:tasks.workunit.client.0.smithi062.stdout:2/969: chown d7/da/d15/d2f/d93/fd5 7767181 1 2024-04-20T21:32:19.193 INFO:tasks.workunit.client.0.smithi062.stdout:5/929: mkdir d8/d1e/d2c/d6c/d133 0 2024-04-20T21:32:19.194 INFO:tasks.workunit.client.0.smithi062.stdout:1/991: dread dc/d15/d20/d28/d83/dce/f15b [0,4194304] 0 2024-04-20T21:32:19.194 INFO:tasks.workunit.client.0.smithi062.stdout:4/984: dread da/dd/df/fc7 [0,4194304] 0 2024-04-20T21:32:19.197 INFO:tasks.workunit.client.1.smithi189.stdout:2/643: creat d1/d11/d2d/dab/d55/fc1 x:0 0 0 2024-04-20T21:32:19.198 INFO:tasks.workunit.client.1.smithi189.stdout:0/814: stat d3/df/d30/d45/f59 0 2024-04-20T21:32:19.199 INFO:tasks.workunit.client.0.smithi062.stdout:2/970: mkdir d7/da/d15/d2f/d38/d10f/d107/d13b 0 2024-04-20T21:32:19.202 INFO:tasks.workunit.client.1.smithi189.stdout:7/761: dread d5/d46/f8b [0,4194304] 0 2024-04-20T21:32:19.202 INFO:tasks.workunit.client.1.smithi189.stdout:8/779: mknod da/df/d3c/d68/d8b/dac/c117 0 2024-04-20T21:32:19.203 INFO:tasks.workunit.client.1.smithi189.stdout:6/725: dwrite f5 [8388608,4194304] 0 2024-04-20T21:32:19.204 INFO:tasks.workunit.client.0.smithi062.stdout:5/930: creat d8/d38/d7a/d4c/d7d/d9d/dac/d66/de4/f134 x:0 0 0 2024-04-20T21:32:19.206 INFO:tasks.workunit.client.1.smithi189.stdout:7/762: chown d5/d1d/ded/cf2 1 1 2024-04-20T21:32:19.208 INFO:tasks.workunit.client.0.smithi062.stdout:2/971: mkdir d7/db5/d13c 0 2024-04-20T21:32:19.209 INFO:tasks.workunit.client.1.smithi189.stdout:2/644: readlink d1/d11/l19 0 2024-04-20T21:32:19.212 INFO:tasks.workunit.client.1.smithi189.stdout:5/715: dread d7/d9/d31/d36/f54 [0,4194304] 0 2024-04-20T21:32:19.215 INFO:tasks.workunit.client.0.smithi062.stdout:4/985: dread da/dd/d15/d50/feb [0,4194304] 0 2024-04-20T21:32:19.217 INFO:tasks.workunit.client.0.smithi062.stdout:1/992: rename dc/d15/d75/d96/dd4/d3a/f46 to dc/d27/d32/d13a/d158/f16d 0 2024-04-20T21:32:19.217 INFO:tasks.workunit.client.0.smithi062.stdout:5/931: rmdir d8/d38 39 2024-04-20T21:32:19.218 INFO:tasks.workunit.client.1.smithi189.stdout:2/645: write d1/d11/d2d/dab/d4a/f8f [1480323,35086] 0 2024-04-20T21:32:19.218 INFO:tasks.workunit.client.0.smithi062.stdout:2/972: mkdir d7/db5/d13d 0 2024-04-20T21:32:19.219 INFO:tasks.workunit.client.1.smithi189.stdout:4/649: dread d2/f8 [0,4194304] 0 2024-04-20T21:32:19.222 INFO:tasks.workunit.client.1.smithi189.stdout:2/646: stat d1/d11/d2d/dab/c39 0 2024-04-20T21:32:19.223 INFO:tasks.workunit.client.1.smithi189.stdout:6/726: read d8/d2b/d49/d54/fe7 [369217,91138] 0 2024-04-20T21:32:19.224 INFO:tasks.workunit.client.1.smithi189.stdout:5/716: rmdir d7/d9/d51/d70 39 2024-04-20T21:32:19.226 INFO:tasks.workunit.client.1.smithi189.stdout:6/727: chown d8/da/dd/d35/l1b 2174793 1 2024-04-20T21:32:19.227 INFO:tasks.workunit.client.0.smithi062.stdout:1/993: mkdir dc/d15/d20/d28/d83/dce/d16e 0 2024-04-20T21:32:19.229 INFO:tasks.workunit.client.1.smithi189.stdout:4/650: mknod d2/dd/ceb 0 2024-04-20T21:32:19.230 INFO:tasks.workunit.client.0.smithi062.stdout:1/994: stat dc/d15/d75/d94/f19 0 2024-04-20T21:32:19.230 INFO:tasks.workunit.client.1.smithi189.stdout:5/717: truncate d7/d9/d31/fbe 959548 0 2024-04-20T21:32:19.235 INFO:tasks.workunit.client.1.smithi189.stdout:6/728: creat d8/d10/fee x:0 0 0 2024-04-20T21:32:19.235 INFO:tasks.workunit.client.1.smithi189.stdout:2/647: fdatasync d1/d11/f15 0 2024-04-20T21:32:19.235 INFO:tasks.workunit.client.1.smithi189.stdout:8/780: dread da/df/d3c/d50/d6c/dc7/fcd [0,4194304] 0 2024-04-20T21:32:19.235 INFO:tasks.workunit.client.0.smithi062.stdout:4/986: mknod da/dd/d15/d4b/ddb/d131/c150 0 2024-04-20T21:32:19.235 INFO:tasks.workunit.client.0.smithi062.stdout:2/973: mkdir d7/da/d15/d2f/d13e 0 2024-04-20T21:32:19.236 INFO:tasks.workunit.client.0.smithi062.stdout:8/954: dwrite d7/d25/d3b/d58/d8b/d96/fa4 [0,4194304] 0 2024-04-20T21:32:19.245 INFO:tasks.workunit.client.1.smithi189.stdout:7/763: getdents d5/d1d/d77 0 2024-04-20T21:32:19.245 INFO:tasks.workunit.client.1.smithi189.stdout:5/718: creat d7/d35/fe8 x:0 0 0 2024-04-20T21:32:19.245 INFO:tasks.workunit.client.1.smithi189.stdout:2/648: write d1/d11/f15 [3525776,102419] 0 2024-04-20T21:32:19.245 INFO:tasks.workunit.client.1.smithi189.stdout:6/729: creat d8/d12/d4e/fef x:0 0 0 2024-04-20T21:32:19.245 INFO:tasks.workunit.client.1.smithi189.stdout:4/651: symlink d2/dd/d37/dcd/d6a/d7a/de9/lec 0 2024-04-20T21:32:19.245 INFO:tasks.workunit.client.1.smithi189.stdout:5/719: readlink d7/d9/d30/la6 0 2024-04-20T21:32:19.245 INFO:tasks.workunit.client.1.smithi189.stdout:6/730: chown d8/d12/cc3 522 1 2024-04-20T21:32:19.246 INFO:tasks.workunit.client.0.smithi062.stdout:1/995: unlink dc/d27/d9a/da0/ca7 0 2024-04-20T21:32:19.247 INFO:tasks.workunit.client.1.smithi189.stdout:4/652: readlink d2/l6 0 2024-04-20T21:32:19.248 INFO:tasks.workunit.client.0.smithi062.stdout:4/987: creat da/dd/df/dfa/f151 x:0 0 0 2024-04-20T21:32:19.250 INFO:tasks.workunit.client.0.smithi062.stdout:1/996: fsync dc/d15/d75/d96/dd4/d3a/da2/de4/d84/dbe/d119/d15e/dd2/fdf 0 2024-04-20T21:32:19.250 INFO:tasks.workunit.client.0.smithi062.stdout:4/988: stat da/dd/d15/daf/d116 0 2024-04-20T21:32:19.253 INFO:tasks.workunit.client.1.smithi189.stdout:4/653: readlink d2/la 0 2024-04-20T21:32:19.254 INFO:tasks.workunit.client.1.smithi189.stdout:1/666: dwrite d8/d1e/d28/d4f/f80 [0,4194304] 0 2024-04-20T21:32:19.254 INFO:tasks.workunit.client.1.smithi189.stdout:8/781: rename da/df/d13/l1a to da/df/d13/d63/l118 0 2024-04-20T21:32:19.256 INFO:tasks.workunit.client.1.smithi189.stdout:9/628: dread d9/f33 [0,4194304] 0 2024-04-20T21:32:19.256 INFO:tasks.workunit.client.0.smithi062.stdout:8/955: symlink d7/d25/d3b/d58/d8b/d24/d46/l133 0 2024-04-20T21:32:19.260 INFO:tasks.workunit.client.1.smithi189.stdout:0/815: dwrite d3/d26/fdf [4194304,4194304] 0 2024-04-20T21:32:19.263 INFO:tasks.workunit.client.1.smithi189.stdout:3/718: dread d2/d3/d2c/f6b [0,4194304] 0 2024-04-20T21:32:19.263 INFO:tasks.workunit.client.0.smithi062.stdout:5/932: write d8/d10/d3e/f52 [3379960,9252] 0 2024-04-20T21:32:19.264 INFO:tasks.workunit.client.0.smithi062.stdout:3/872: sync 2024-04-20T21:32:19.264 INFO:tasks.workunit.client.1.smithi189.stdout:2/649: getdents d1/d11/d2d 0 2024-04-20T21:32:19.266 INFO:tasks.workunit.client.1.smithi189.stdout:8/782: read da/de/d1e/d69/da3/fc3 [3879570,28316] 0 2024-04-20T21:32:19.266 INFO:tasks.workunit.client.0.smithi062.stdout:3/873: read - d0/d1c/d63/fb2 zero size 2024-04-20T21:32:19.267 INFO:tasks.workunit.client.1.smithi189.stdout:6/731: chown d8/da/dd/d35/d6d/d77/fac 530644 1 2024-04-20T21:32:19.270 INFO:tasks.workunit.client.1.smithi189.stdout:2/650: write d1/d17/d37/f44 [1012128,74326] 0 2024-04-20T21:32:19.271 INFO:tasks.workunit.client.1.smithi189.stdout:1/667: mkdir d8/df/d50/de6 0 2024-04-20T21:32:19.271 INFO:tasks.workunit.client.1.smithi189.stdout:8/783: dread - da/df/d20/dc9/f10b zero size 2024-04-20T21:32:19.271 INFO:tasks.workunit.client.0.smithi062.stdout:1/997: dread dc/d15/d75/d96/dd4/d3a/da2/de4/d84/dbe/d119/d15e/dd2/fdf [0,4194304] 0 2024-04-20T21:32:19.272 INFO:tasks.workunit.client.1.smithi189.stdout:9/629: truncate d9/dc/d17/d95/da0/d52/d82/fa7 838080 0 2024-04-20T21:32:19.274 INFO:tasks.workunit.client.0.smithi062.stdout:2/974: link d7/da/d15/d2f/d97/f128 d7/da/d15/d2f/d38/db0/f13f 0 2024-04-20T21:32:19.275 INFO:tasks.workunit.client.1.smithi189.stdout:4/654: creat d2/dd/d37/dcd/d6a/d7a/db7/fed x:0 0 0 2024-04-20T21:32:19.276 INFO:tasks.workunit.client.1.smithi189.stdout:6/732: mknod d8/d2b/d5c/cf0 0 2024-04-20T21:32:19.282 INFO:tasks.workunit.client.0.smithi062.stdout:3/874: mknod d0/d1c/c11f 0 2024-04-20T21:32:19.282 INFO:tasks.workunit.client.1.smithi189.stdout:2/651: write d1/d11/d52/d8b/f8d [2499845,115637] 0 2024-04-20T21:32:19.283 INFO:tasks.workunit.client.1.smithi189.stdout:4/655: fsync d2/dd/d37/dcd/d6a/fdd 0 2024-04-20T21:32:19.284 INFO:tasks.workunit.client.1.smithi189.stdout:9/630: fdatasync d9/dc/d17/f8e 0 2024-04-20T21:32:19.285 INFO:tasks.workunit.client.1.smithi189.stdout:3/719: getdents d2/d1c/d5c 0 2024-04-20T21:32:19.287 INFO:tasks.workunit.client.0.smithi062.stdout:4/989: write da/dd/f35 [1576929,115112] 0 2024-04-20T21:32:19.287 INFO:tasks.workunit.client.0.smithi062.stdout:2/975: mknod d7/da/d15/d1b/d3d/d84/d9b/c140 0 2024-04-20T21:32:19.287 INFO:tasks.workunit.client.1.smithi189.stdout:8/784: link da/d81/d91/fc4 da/df/d13/dd6/dba/f119 0 2024-04-20T21:32:19.289 INFO:tasks.workunit.client.0.smithi062.stdout:5/933: creat d8/d38/d7a/d4c/d7d/d9d/dac/d66/dae/f135 x:0 0 0 2024-04-20T21:32:19.289 INFO:tasks.workunit.client.1.smithi189.stdout:9/631: stat d9/dc/l90 0 2024-04-20T21:32:19.292 INFO:tasks.workunit.client.1.smithi189.stdout:6/733: dread d8/da/dd/f88 [0,4194304] 0 2024-04-20T21:32:19.294 INFO:tasks.workunit.client.0.smithi062.stdout:4/990: mkdir da/d56/d71/d9e/d152 0 2024-04-20T21:32:19.294 INFO:tasks.workunit.client.1.smithi189.stdout:2/652: dread - d1/d17/d37/d89/f95 zero size 2024-04-20T21:32:19.295 INFO:tasks.workunit.client.1.smithi189.stdout:6/734: chown d8/d10 1 1 2024-04-20T21:32:19.295 INFO:tasks.workunit.client.0.smithi062.stdout:5/934: write d8/f101 [944898,50515] 0 2024-04-20T21:32:19.296 INFO:tasks.workunit.client.0.smithi062.stdout:1/998: creat dc/d70/f16f x:0 0 0 2024-04-20T21:32:19.296 INFO:tasks.workunit.client.1.smithi189.stdout:4/656: symlink d2/dd/d11/d14/lee 0 2024-04-20T21:32:19.296 INFO:tasks.workunit.client.0.smithi062.stdout:2/976: mknod d7/da/d15/d2f/d38/d51/c141 0 2024-04-20T21:32:19.300 INFO:tasks.workunit.client.1.smithi189.stdout:8/785: mknod da/d81/c11a 0 2024-04-20T21:32:19.301 INFO:tasks.workunit.client.0.smithi062.stdout:4/991: rmdir da/dd/d15/daf/dee 39 2024-04-20T21:32:19.302 INFO:tasks.workunit.client.1.smithi189.stdout:5/720: dwrite d7/d9/d30/d55/d61/d87/f90 [0,4194304] 0 2024-04-20T21:32:19.305 INFO:tasks.workunit.client.0.smithi062.stdout:1/999: fdatasync dc/d15/d20/d28/d83/f128 0 2024-04-20T21:32:19.306 INFO:tasks.workunit.client.1.smithi189.stdout:6/735: unlink d8/d2b/f90 0 2024-04-20T21:32:19.306 INFO:tasks.workunit.client.1.smithi189.stdout:9/632: dread - d9/d12/d3b/db1/d1a/f9d zero size 2024-04-20T21:32:19.310 INFO:tasks.workunit.client.1.smithi189.stdout:5/721: fsync d7/d9/d30/d5a/f73 0 2024-04-20T21:32:19.311 INFO:tasks.workunit.client.1.smithi189.stdout:6/736: mkdir d8/d12/df1 0 2024-04-20T21:32:19.312 INFO:tasks.workunit.client.1.smithi189.stdout:8/786: truncate da/df/d3c/d50/d6c/dc7/fef 409450 0 2024-04-20T21:32:19.313 INFO:tasks.workunit.client.1.smithi189.stdout:5/722: fsync d7/d35/f46 0 2024-04-20T21:32:19.313 INFO:tasks.workunit.client.1.smithi189.stdout:4/657: rename d2/d22/d87/dcc to d2/dd/d37/dcd/d6a/d7a/def 0 2024-04-20T21:32:19.316 INFO:tasks.workunit.client.1.smithi189.stdout:3/720: dread d2/d3/d2c/d30/d3f/f8a [0,4194304] 0 2024-04-20T21:32:19.318 INFO:tasks.workunit.client.1.smithi189.stdout:8/787: chown da/df/d3c/d68/d8b/f10a 3076625 1 2024-04-20T21:32:19.319 INFO:tasks.workunit.client.1.smithi189.stdout:6/737: creat d8/da/dd/d35/d6d/d77/ff2 x:0 0 0 2024-04-20T21:32:19.319 INFO:tasks.workunit.client.0.smithi062.stdout:3/875: dwrite d0/d1/d3c/d43/d72/dd3/d97/fa3 [0,4194304] 0 2024-04-20T21:32:19.320 INFO:tasks.workunit.client.0.smithi062.stdout:5/935: dread d8/d38/d7a/d4c/da7/f10c [0,4194304] 0 2024-04-20T21:32:19.323 INFO:tasks.workunit.client.1.smithi189.stdout:5/723: symlink d7/d9/d31/d36/d5f/dab/le9 0 2024-04-20T21:32:19.323 INFO:tasks.workunit.client.1.smithi189.stdout:1/668: dread d8/d1e/d28/d2b/f48 [0,4194304] 0 2024-04-20T21:32:19.325 INFO:tasks.workunit.client.0.smithi062.stdout:4/992: link da/dd/df/c47 da/d56/d71/d9e/c153 0 2024-04-20T21:32:19.326 INFO:tasks.workunit.client.1.smithi189.stdout:6/738: mknod d8/d2b/d49/d54/d6e/cf3 0 2024-04-20T21:32:19.326 INFO:tasks.workunit.client.1.smithi189.stdout:4/658: write d2/dd/d37/dcd/fb3 [1895818,19831] 0 2024-04-20T21:32:19.329 INFO:tasks.workunit.client.1.smithi189.stdout:9/633: rename d9/dc/d17/d95/da0/d52/c70 to d9/dc/d17/cc8 0 2024-04-20T21:32:19.329 INFO:tasks.workunit.client.1.smithi189.stdout:1/669: write d8/d1e/d28/fa8 [2335797,129999] 0 2024-04-20T21:32:19.332 INFO:tasks.workunit.client.1.smithi189.stdout:6/739: creat d8/d10/d78/d7e/dcf/ff4 x:0 0 0 2024-04-20T21:32:19.333 INFO:tasks.workunit.client.0.smithi062.stdout:2/977: write d7/da/d15/d1b/d24/f66 [1192191,83264] 0 2024-04-20T21:32:19.333 INFO:tasks.workunit.client.1.smithi189.stdout:5/724: fdatasync d7/d9/d51/d70/d95/fa1 0 2024-04-20T21:32:19.334 INFO:tasks.workunit.client.1.smithi189.stdout:2/653: dread d1/d11/d52/d8b/f8d [0,4194304] 0 2024-04-20T21:32:19.336 INFO:tasks.workunit.client.1.smithi189.stdout:9/634: mknod d9/dc/d17/d95/cc9 0 2024-04-20T21:32:19.339 INFO:tasks.workunit.client.1.smithi189.stdout:4/659: mknod d2/dd/d37/dcd/d6a/d7a/dce/cf0 0 2024-04-20T21:32:19.339 INFO:tasks.workunit.client.0.smithi062.stdout:3/876: getdents d0/d1/d10/d3a/de9 0 2024-04-20T21:32:19.339 INFO:tasks.workunit.client.1.smithi189.stdout:8/788: rename da/d1f/d2a/ld0 to da/df/d20/d65/l11b 0 2024-04-20T21:32:19.344 INFO:tasks.workunit.client.0.smithi062.stdout:2/978: mknod d7/da/d15/d1b/df1/c142 0 2024-04-20T21:32:19.345 INFO:tasks.workunit.client.1.smithi189.stdout:1/670: creat d8/d1e/d43/d58/fe7 x:0 0 0 2024-04-20T21:32:19.346 INFO:tasks.workunit.client.1.smithi189.stdout:9/635: rename d9/d12/l23 to d9/dc/d17/dac/lca 0 2024-04-20T21:32:19.347 INFO:tasks.workunit.client.0.smithi062.stdout:3/877: symlink d0/d1/d10/d31/d41/l120 0 2024-04-20T21:32:19.348 INFO:tasks.workunit.client.1.smithi189.stdout:4/660: mknod d2/d20/dc6/dda/cf1 0 2024-04-20T21:32:19.350 INFO:tasks.workunit.client.0.smithi062.stdout:2/979: mknod d7/da/d15/d1b/d3d/d11f/c143 0 2024-04-20T21:32:19.353 INFO:tasks.workunit.client.0.smithi062.stdout:3/878: rmdir d0/d1c/d63/d42/d5b 39 2024-04-20T21:32:19.354 INFO:tasks.workunit.client.1.smithi189.stdout:2/654: dread d1/d11/d2d/dab/f42 [0,4194304] 0 2024-04-20T21:32:19.355 INFO:tasks.workunit.client.1.smithi189.stdout:9/636: symlink d9/dc/d17/d95/da0/d52/d82/d94/lcb 0 2024-04-20T21:32:19.355 INFO:tasks.workunit.client.0.smithi062.stdout:2/980: readlink d7/da/d15/d2f/d38/d51/dbc/lc3 0 2024-04-20T21:32:19.355 INFO:tasks.workunit.client.0.smithi062.stdout:5/936: dwrite d8/f4f [0,4194304] 0 2024-04-20T21:32:19.357 INFO:tasks.workunit.client.1.smithi189.stdout:9/637: chown d9/d12/l48 720393 1 2024-04-20T21:32:19.359 INFO:tasks.workunit.client.1.smithi189.stdout:5/725: dwrite d7/d9/d28/db1/fe5 [0,4194304] 0 2024-04-20T21:32:19.359 INFO:tasks.workunit.client.0.smithi062.stdout:2/981: chown d7/da/d15/d2f/d38/db0/c112 294561 1 2024-04-20T21:32:19.363 INFO:tasks.workunit.client.1.smithi189.stdout:9/638: symlink d9/dc/lcc 0 2024-04-20T21:32:19.365 INFO:tasks.workunit.client.1.smithi189.stdout:1/671: dread d8/d1e/d43/d44/fc9 [0,4194304] 0 2024-04-20T21:32:19.365 INFO:tasks.workunit.client.0.smithi062.stdout:5/937: rmdir d8/d38/d7a/d4c/d7d/d9d/dac/d66/de4 39 2024-04-20T21:32:19.368 INFO:tasks.workunit.client.1.smithi189.stdout:9/639: write d9/d12/fa8 [659706,10937] 0 2024-04-20T21:32:19.368 INFO:tasks.workunit.client.1.smithi189.stdout:5/726: write d7/d9/d28/db1/fe5 [721406,93946] 0 2024-04-20T21:32:19.370 INFO:tasks.workunit.client.1.smithi189.stdout:3/721: truncate d2/d3/d2c/d31/d90/fd8 1499437 0 2024-04-20T21:32:19.371 INFO:tasks.workunit.client.1.smithi189.stdout:5/727: chown d7/d9/d51/d70/d95 572396341 1 2024-04-20T21:32:19.372 INFO:tasks.workunit.client.1.smithi189.stdout:9/640: write d9/dc/d69/faf [29449,63118] 0 2024-04-20T21:32:19.374 INFO:tasks.workunit.client.1.smithi189.stdout:3/722: write d2/d3/d36/fd7 [1675537,44290] 0 2024-04-20T21:32:19.377 INFO:tasks.workunit.client.0.smithi062.stdout:6/820: sync 2024-04-20T21:32:19.377 INFO:tasks.workunit.client.0.smithi062.stdout:0/906: sync 2024-04-20T21:32:19.377 INFO:tasks.workunit.client.0.smithi062.stdout:7/962: sync 2024-04-20T21:32:19.378 INFO:tasks.workunit.client.1.smithi189.stdout:6/740: truncate d8/d12/d30/d47/da0/fcc 785393 0 2024-04-20T21:32:19.380 INFO:tasks.workunit.client.1.smithi189.stdout:5/728: creat d7/d9/d30/d5a/fea x:0 0 0 2024-04-20T21:32:19.380 INFO:tasks.workunit.client.0.smithi062.stdout:7/963: readlink d7/d3e/d33/d3d/d4d/d83/l101 0 2024-04-20T21:32:19.380 INFO:tasks.workunit.client.0.smithi062.stdout:5/938: symlink d8/d38/d7a/d4c/d7d/d9d/dac/d66/dae/l136 0 2024-04-20T21:32:19.381 INFO:tasks.workunit.client.1.smithi189.stdout:8/789: truncate da/de/d1e/d69/f95 592511 0 2024-04-20T21:32:19.381 INFO:tasks.workunit.client.1.smithi189.stdout:5/729: read d7/d14/d18/f17 [3769281,49444] 0 2024-04-20T21:32:19.384 INFO:tasks.workunit.client.1.smithi189.stdout:5/730: dread - d7/d9/fbf zero size 2024-04-20T21:32:19.384 INFO:tasks.workunit.client.1.smithi189.stdout:8/790: write da/df/d3c/d68/d8b/fcf [45210,35801] 0 2024-04-20T21:32:19.384 INFO:tasks.workunit.client.1.smithi189.stdout:3/723: creat d2/d3/d2c/d31/d4c/ffc x:0 0 0 2024-04-20T21:32:19.385 INFO:tasks.workunit.client.0.smithi062.stdout:7/964: mkdir d7/d3e/d33/d3d/dad/d14d 0 2024-04-20T21:32:19.386 INFO:tasks.workunit.client.1.smithi189.stdout:4/661: write d2/f6d [1040515,44974] 0 2024-04-20T21:32:19.386 INFO:tasks.workunit.client.1.smithi189.stdout:2/655: truncate d1/d11/d99/db7/fb8 555022 0 2024-04-20T21:32:19.388 INFO:tasks.workunit.client.0.smithi062.stdout:5/939: mknod d8/d38/d7a/d4c/d7d/d9d/dac/d2f/c137 0 2024-04-20T21:32:19.389 INFO:tasks.workunit.client.1.smithi189.stdout:9/641: rename d9/dc/d17/d36/d5d/c5f to d9/dc/d17/d95/da0/d52/ccd 0 2024-04-20T21:32:19.389 INFO:tasks.workunit.client.1.smithi189.stdout:8/791: write da/df/d3c/d9d/f113 [740632,4271] 0 2024-04-20T21:32:19.390 INFO:tasks.workunit.client.0.smithi062.stdout:5/940: stat d8/d1e/d2c/d6c/la6 0 2024-04-20T21:32:19.391 INFO:tasks.workunit.client.1.smithi189.stdout:3/724: rmdir d2 39 2024-04-20T21:32:19.392 INFO:tasks.workunit.client.1.smithi189.stdout:1/672: write d8/df/d3a/f78 [4851,112658] 0 2024-04-20T21:32:19.393 INFO:tasks.workunit.client.1.smithi189.stdout:5/731: rmdir d7/d35 39 2024-04-20T21:32:19.394 INFO:tasks.workunit.client.1.smithi189.stdout:9/642: chown d9/dc/d17/d36/d5d/f97 1768987 1 2024-04-20T21:32:19.395 INFO:tasks.workunit.client.0.smithi062.stdout:4/993: dread da/f2f [0,4194304] 0 2024-04-20T21:32:19.397 INFO:tasks.workunit.client.1.smithi189.stdout:6/741: mknod d8/d12/cf5 0 2024-04-20T21:32:19.399 INFO:tasks.workunit.client.1.smithi189.stdout:6/742: stat d8/d12/d30/d47/c48 0 2024-04-20T21:32:19.399 INFO:tasks.workunit.client.1.smithi189.stdout:8/792: read da/f89 [20290,7005] 0 2024-04-20T21:32:19.400 INFO:tasks.workunit.client.1.smithi189.stdout:9/643: rmdir d9/d12/d3b/db1 39 2024-04-20T21:32:19.401 INFO:tasks.workunit.client.1.smithi189.stdout:4/662: mknod d2/d22/cf2 0 2024-04-20T21:32:19.403 INFO:tasks.workunit.client.1.smithi189.stdout:5/732: symlink d7/d9/d31/d36/d5f/dab/dcb/leb 0 2024-04-20T21:32:19.403 INFO:tasks.workunit.client.1.smithi189.stdout:4/663: readlink d2/dd/d37/dcd/le7 0 2024-04-20T21:32:19.408 INFO:tasks.workunit.client.1.smithi189.stdout:8/793: dread da/df/d20/f79 [0,4194304] 0 2024-04-20T21:32:19.410 INFO:tasks.workunit.client.1.smithi189.stdout:3/725: mkdir d2/d3/dfd 0 2024-04-20T21:32:19.412 INFO:tasks.workunit.client.1.smithi189.stdout:6/743: write d8/da/dd/d35/fc8 [958311,57313] 0 2024-04-20T21:32:19.416 INFO:tasks.workunit.client.1.smithi189.stdout:5/733: fdatasync d7/d35/f46 0 2024-04-20T21:32:19.418 INFO:tasks.workunit.client.1.smithi189.stdout:9/644: dwrite d9/dc/d17/d95/da6/fb3 [0,4194304] 0 2024-04-20T21:32:19.419 INFO:tasks.workunit.client.1.smithi189.stdout:4/664: rename d2/dd/d37/dcd/d6a/d7a/f8c to d2/d3a/ddc/ff3 0 2024-04-20T21:32:19.420 INFO:tasks.workunit.client.0.smithi062.stdout:3/879: truncate d0/d11/fa2 1763830 0 2024-04-20T21:32:19.420 INFO:tasks.workunit.client.1.smithi189.stdout:2/656: getdents d1/d11/d52/d8b 0 2024-04-20T21:32:19.421 INFO:tasks.workunit.client.0.smithi062.stdout:0/907: write d7/d1c/d15/d17/d26/d76/fa6 [2168083,37591] 0 2024-04-20T21:32:19.422 INFO:tasks.workunit.client.1.smithi189.stdout:9/645: chown d9/dc/d17/d95/da0/d52/d82/d94/cc0 1 1 2024-04-20T21:32:19.422 INFO:tasks.workunit.client.1.smithi189.stdout:1/673: creat d8/d1e/d43/d58/fe8 x:0 0 0 2024-04-20T21:32:19.423 INFO:tasks.workunit.client.0.smithi062.stdout:7/965: truncate d7/d63/f7a 357158 0 2024-04-20T21:32:19.423 INFO:tasks.workunit.client.0.smithi062.stdout:5/941: rename d8/d10/l29 to d8/d38/d7a/d4c/d7d/l138 0 2024-04-20T21:32:19.425 INFO:tasks.workunit.client.1.smithi189.stdout:8/794: write da/d1f/d2a/f58 [2386474,34000] 0 2024-04-20T21:32:19.429 INFO:tasks.workunit.client.0.smithi062.stdout:2/982: dwrite d7/da/d15/d1b/d3d/d84/f100 [0,4194304] 0 2024-04-20T21:32:19.434 INFO:tasks.workunit.client.1.smithi189.stdout:3/726: unlink d2/d3/d2c/d31/d4c/ffc 0 2024-04-20T21:32:19.438 INFO:tasks.workunit.client.0.smithi062.stdout:2/983: chown d7/da/d15/d2f/d38/d10f 9 1 2024-04-20T21:32:19.438 INFO:tasks.workunit.client.0.smithi062.stdout:0/908: rename d7/d1c/d15/d17/d40/d3c/dc5 to d7/d1c/d15/d17/d37/d109/d11b 0 2024-04-20T21:32:19.438 INFO:tasks.workunit.client.1.smithi189.stdout:4/665: truncate d2/f5e 3535668 0 2024-04-20T21:32:19.440 INFO:tasks.workunit.client.1.smithi189.stdout:8/795: rmdir da/d1f/d116 0 2024-04-20T21:32:19.442 INFO:tasks.workunit.client.1.smithi189.stdout:6/744: rename d8/d2b/d5c/cf0 to d8/d12/cf6 0 2024-04-20T21:32:19.442 INFO:tasks.workunit.client.0.smithi062.stdout:4/994: readlink da/dd/d15/daf/dee/l13f 0 2024-04-20T21:32:19.443 INFO:tasks.workunit.client.1.smithi189.stdout:9/646: creat d9/dc/d17/d36/fce x:0 0 0 2024-04-20T21:32:19.444 INFO:tasks.workunit.client.1.smithi189.stdout:8/796: write da/df/d3c/d68/d8b/d8d/f90 [127972,67135] 0 2024-04-20T21:32:19.445 INFO:tasks.workunit.client.1.smithi189.stdout:7/764: sync 2024-04-20T21:32:19.448 INFO:tasks.workunit.client.1.smithi189.stdout:3/727: rmdir d2/d3/d2c/d30/d3f 39 2024-04-20T21:32:19.448 INFO:tasks.workunit.client.0.smithi062.stdout:5/942: dread d8/d38/d7a/d4c/d7d/d9d/dac/d2f/f4b [0,4194304] 0 2024-04-20T21:32:19.450 INFO:tasks.workunit.client.0.smithi062.stdout:3/880: creat d0/d1c/d63/d42/f121 x:0 0 0 2024-04-20T21:32:19.453 INFO:tasks.workunit.client.1.smithi189.stdout:4/666: chown d2/l9 21 1 2024-04-20T21:32:19.454 INFO:tasks.workunit.client.0.smithi062.stdout:0/909: mknod d7/d1c/c11c 0 2024-04-20T21:32:19.456 INFO:tasks.workunit.client.0.smithi062.stdout:4/995: truncate da/dd/df/d27/d32/d97/f121 191854 0 2024-04-20T21:32:19.456 INFO:tasks.workunit.client.1.smithi189.stdout:9/647: symlink d9/dc/d17/d7e/lcf 0 2024-04-20T21:32:19.457 INFO:tasks.workunit.client.1.smithi189.stdout:2/657: dread d1/f2a [0,4194304] 0 2024-04-20T21:32:19.458 INFO:tasks.workunit.client.0.smithi062.stdout:7/966: link d7/d3e/d33/le4 d7/d3e/d33/d3d/d4d/d54/d4f/de9/df1/d105/d13b/l14e 0 2024-04-20T21:32:19.460 INFO:tasks.workunit.client.0.smithi062.stdout:5/943: mkdir d8/d38/d7a/d4c/d7d/d9d/dac/d66/dae/d88/d139 0 2024-04-20T21:32:19.462 INFO:tasks.workunit.client.1.smithi189.stdout:3/728: chown l1 1651012 1 2024-04-20T21:32:19.463 INFO:tasks.workunit.client.1.smithi189.stdout:2/658: chown d1/d11/d2d/fa6 0 1 2024-04-20T21:32:19.467 INFO:tasks.workunit.client.1.smithi189.stdout:7/765: rmdir d5/d1d/d64/db9 39 2024-04-20T21:32:19.467 INFO:tasks.workunit.client.1.smithi189.stdout:4/667: rename d2/f19 to d2/dd/d37/dcd/d6a/d7a/dce/ff4 0 2024-04-20T21:32:19.467 INFO:tasks.workunit.client.0.smithi062.stdout:4/996: mkdir da/dd/d15/d50/d64/d57/d5b/d9d/d154 0 2024-04-20T21:32:19.468 INFO:tasks.workunit.client.0.smithi062.stdout:7/967: mkdir d7/d3e/d33/d3d/d4d/d14f 0 2024-04-20T21:32:19.469 INFO:tasks.workunit.client.1.smithi189.stdout:6/745: dwrite d8/d12/d99/fb7 [0,4194304] 0 2024-04-20T21:32:19.472 INFO:tasks.workunit.client.1.smithi189.stdout:3/729: read d2/f59 [2808079,7580] 0 2024-04-20T21:32:19.474 INFO:tasks.workunit.client.1.smithi189.stdout:7/766: fsync d5/d46/fa9 0 2024-04-20T21:32:19.481 INFO:tasks.workunit.client.1.smithi189.stdout:4/668: dread - d2/fbe zero size 2024-04-20T21:32:19.481 INFO:tasks.workunit.client.0.smithi062.stdout:4/997: mkdir da/dd/df/d27/d3f/d82/d155 0 2024-04-20T21:32:19.486 INFO:tasks.workunit.client.1.smithi189.stdout:8/797: dread da/df/d13/d31/f80 [0,4194304] 0 2024-04-20T21:32:19.488 INFO:tasks.workunit.client.1.smithi189.stdout:4/669: chown d2/fbe 49094391 1 2024-04-20T21:32:19.488 INFO:tasks.workunit.client.1.smithi189.stdout:7/767: fsync d5/f39 0 2024-04-20T21:32:19.489 INFO:tasks.workunit.client.0.smithi062.stdout:5/944: rename d8/d38/d7a/d4c/d7d/f127 to d8/d38/d7a/d4c/d7d/d9d/dac/d66/dae/d88/f13a 0 2024-04-20T21:32:19.490 INFO:tasks.workunit.client.1.smithi189.stdout:2/659: link d1/d11/d2d/d4f/db4/c22 d1/d11/d58/cc2 0 2024-04-20T21:32:19.492 INFO:tasks.workunit.client.1.smithi189.stdout:6/746: rename d8/da/dd/d35/d6d/d77 to d8/d10/d78/df7 0 2024-04-20T21:32:19.493 INFO:tasks.workunit.client.0.smithi062.stdout:8/956: sync 2024-04-20T21:32:19.493 INFO:tasks.workunit.client.0.smithi062.stdout:4/998: mkdir da/dd/d15/d50/d64/d156 0 2024-04-20T21:32:19.494 INFO:tasks.workunit.client.1.smithi189.stdout:0/816: sync 2024-04-20T21:32:19.495 INFO:tasks.workunit.client.1.smithi189.stdout:1/674: dwrite d8/d1e/d43/d44/dd2/f82 [0,4194304] 0 2024-04-20T21:32:19.496 INFO:tasks.workunit.client.0.smithi062.stdout:8/957: write d7/d9/f12 [4194273,74598] 0 2024-04-20T21:32:19.500 INFO:tasks.workunit.client.1.smithi189.stdout:3/730: creat d2/d3/d25/de0/db0/ffe x:0 0 0 2024-04-20T21:32:19.505 INFO:tasks.workunit.client.1.smithi189.stdout:1/675: dread - d8/d1e/fc5 zero size 2024-04-20T21:32:19.505 INFO:tasks.workunit.client.1.smithi189.stdout:9/648: dwrite d9/dc/d17/d95/da0/d52/f83 [4194304,4194304] 0 2024-04-20T21:32:19.505 INFO:tasks.workunit.client.1.smithi189.stdout:5/734: dwrite d7/d9/f62 [0,4194304] 0 2024-04-20T21:32:19.507 INFO:tasks.workunit.client.0.smithi062.stdout:5/945: creat d8/d38/d7a/d4c/d11d/f13b x:0 0 0 2024-04-20T21:32:19.508 INFO:tasks.workunit.client.1.smithi189.stdout:7/768: fdatasync d5/d21/d4a/d52/d75/dc8/fce 0 2024-04-20T21:32:19.509 INFO:tasks.workunit.client.1.smithi189.stdout:4/670: truncate d2/d22/d57/f9f 535726 0 2024-04-20T21:32:19.509 INFO:tasks.workunit.client.0.smithi062.stdout:8/958: truncate d7/d25/d3b/d58/d8b/d24/d46/f64 7968012 0 2024-04-20T21:32:19.510 INFO:tasks.workunit.client.1.smithi189.stdout:0/817: write d3/df/d28/d2b/f49 [313869,100829] 0 2024-04-20T21:32:19.511 INFO:tasks.workunit.client.0.smithi062.stdout:5/946: chown d8/d38/d7a/df0 15 1 2024-04-20T21:32:19.513 INFO:tasks.workunit.client.1.smithi189.stdout:3/731: mknod d2/d8d/cff 0 2024-04-20T21:32:19.516 INFO:tasks.workunit.client.1.smithi189.stdout:1/676: unlink d8/d21/f22 0 2024-04-20T21:32:19.519 INFO:tasks.workunit.client.0.smithi062.stdout:8/959: write d7/d25/d3b/d58/db9/f113 [584788,105119] 0 2024-04-20T21:32:19.519 INFO:tasks.workunit.client.1.smithi189.stdout:7/769: creat d5/d1d/d77/ff8 x:0 0 0 2024-04-20T21:32:19.521 INFO:tasks.workunit.client.1.smithi189.stdout:9/649: mkdir d9/dc/d17/d95/da0/d55/dd0 0 2024-04-20T21:32:19.521 INFO:tasks.workunit.client.1.smithi189.stdout:6/747: truncate d8/d10/d78/d7e/f84 1153995 0 2024-04-20T21:32:19.523 INFO:tasks.workunit.client.1.smithi189.stdout:6/748: chown d8/da/da7 60 1 2024-04-20T21:32:19.525 INFO:tasks.workunit.client.0.smithi062.stdout:3/881: truncate d0/d11/fa2 1209562 0 2024-04-20T21:32:19.525 INFO:tasks.workunit.client.1.smithi189.stdout:4/671: dread d2/f8 [0,4194304] 0 2024-04-20T21:32:19.526 INFO:tasks.workunit.client.1.smithi189.stdout:0/818: rename d3/db9/dbe/df9 to d3/df/d28/d2b/db1/d111 0 2024-04-20T21:32:19.526 INFO:tasks.workunit.client.0.smithi062.stdout:4/999: dwrite da/dd/df/d27/d7a/d91/de4/f138 [0,4194304] 0 2024-04-20T21:32:19.527 INFO:tasks.workunit.client.1.smithi189.stdout:3/732: mkdir d2/d1c/d5c/dca/d100 0 2024-04-20T21:32:19.528 INFO:tasks.workunit.client.0.smithi062.stdout:0/910: dwrite d7/d1c/d15/d17/d1b/d106/fa0 [0,4194304] 0 2024-04-20T21:32:19.534 INFO:tasks.workunit.client.1.smithi189.stdout:8/798: link da/d81/d91/l100 da/df/d3c/d50/ddc/l11c 0 2024-04-20T21:32:19.534 INFO:tasks.workunit.client.1.smithi189.stdout:1/677: mkdir d8/df/d3a/d49/d8c/da2/dbe/dc1/de9 0 2024-04-20T21:32:19.534 INFO:tasks.workunit.client.1.smithi189.stdout:2/660: dwrite d1/d11/d2d/fa6 [0,4194304] 0 2024-04-20T21:32:19.537 INFO:tasks.workunit.client.0.smithi062.stdout:8/960: mkdir d7/d30/d134 0 2024-04-20T21:32:19.537 INFO:tasks.workunit.client.1.smithi189.stdout:1/678: chown d8/d1e/f87 2322555 1 2024-04-20T21:32:19.539 INFO:tasks.workunit.client.1.smithi189.stdout:5/735: link d7/d14/d4e/d5b/fc2 d7/d9/d31/fec 0 2024-04-20T21:32:19.542 INFO:tasks.workunit.client.0.smithi062.stdout:7/968: dwrite d7/f2a [0,4194304] 0 2024-04-20T21:32:19.545 INFO:tasks.workunit.client.1.smithi189.stdout:5/736: write d7/d9/d51/d70/d95/fa1 [897188,37840] 0 2024-04-20T21:32:19.545 INFO:tasks.workunit.client.1.smithi189.stdout:8/799: write da/de/d1e/d21/d87/ff6 [331958,130072] 0 2024-04-20T21:32:19.552 INFO:tasks.workunit.client.1.smithi189.stdout:6/749: creat d8/d2b/d49/d54/d6e/ff8 x:0 0 0 2024-04-20T21:32:19.554 INFO:tasks.workunit.client.1.smithi189.stdout:0/819: dwrite d3/d26/fdf [4194304,4194304] 0 2024-04-20T21:32:19.558 INFO:tasks.workunit.client.0.smithi062.stdout:7/969: mkdir d7/d3e/d33/d3d/d4d/d54/d4f/d9d/d150 0 2024-04-20T21:32:19.559 INFO:tasks.workunit.client.1.smithi189.stdout:4/672: rename d2/f9b to d2/dd/ff5 0 2024-04-20T21:32:19.561 INFO:tasks.workunit.client.1.smithi189.stdout:3/733: creat d2/d8d/f101 x:0 0 0 2024-04-20T21:32:19.565 INFO:tasks.workunit.client.1.smithi189.stdout:4/673: readlink d2/dd/d37/dcd/d6a/ld2 0 2024-04-20T21:32:19.568 INFO:tasks.workunit.client.0.smithi062.stdout:5/947: dwrite d8/d38/d7a/d4c/d7d/d9d/dac/f92 [0,4194304] 0 2024-04-20T21:32:19.569 INFO:tasks.workunit.client.1.smithi189.stdout:2/661: fdatasync d1/d11/d2d/f63 0 2024-04-20T21:32:19.570 INFO:tasks.workunit.client.1.smithi189.stdout:9/650: write d9/dc/d17/d95/da0/f6a [13814,24558] 0 2024-04-20T21:32:19.574 INFO:tasks.workunit.client.0.smithi062.stdout:5/948: read - d8/d38/d7a/d4c/dcf/fee zero size 2024-04-20T21:32:19.575 INFO:tasks.workunit.client.1.smithi189.stdout:6/750: mknod d8/d2b/d49/d54/cf9 0 2024-04-20T21:32:19.578 INFO:tasks.workunit.client.1.smithi189.stdout:1/679: mkdir d8/df/d3a/d49/d8c/da2/dbe/dc1/dd9/dea 0 2024-04-20T21:32:19.579 INFO:tasks.workunit.client.1.smithi189.stdout:3/734: chown d2/d3/d65 66765603 1 2024-04-20T21:32:19.583 INFO:tasks.workunit.client.1.smithi189.stdout:9/651: chown d9/dc/d17/d95/da0/d56/l93 3 1 2024-04-20T21:32:19.584 INFO:tasks.workunit.client.1.smithi189.stdout:7/770: creat d5/db4/d71/de4/ff9 x:0 0 0 2024-04-20T21:32:19.584 INFO:tasks.workunit.client.1.smithi189.stdout:5/737: link d7/d14/d18/l1e d7/d9/d30/d55/led 0 2024-04-20T21:32:19.590 INFO:tasks.workunit.client.0.smithi062.stdout:7/970: rename d7/d3e/d33/d3d/d4d/d54/d4f/l66 to d7/d26/da8/db6/l151 0 2024-04-20T21:32:19.590 INFO:tasks.workunit.client.1.smithi189.stdout:1/680: readlink d8/d1e/d28/laa 0 2024-04-20T21:32:19.591 INFO:tasks.workunit.client.0.smithi062.stdout:3/882: write d0/d1/d3c/d43/d72/da9/fab [4604847,9192] 0 2024-04-20T21:32:19.592 INFO:tasks.workunit.client.1.smithi189.stdout:5/738: chown d7/d9/d31/la5 67296566 1 2024-04-20T21:32:19.593 INFO:tasks.workunit.client.0.smithi062.stdout:8/961: truncate d7/d9/f12 2442210 0 2024-04-20T21:32:19.594 INFO:tasks.workunit.client.1.smithi189.stdout:7/771: creat d5/d59/ffa x:0 0 0 2024-04-20T21:32:19.594 INFO:tasks.workunit.client.1.smithi189.stdout:0/820: dwrite d3/df/d4c/f69 [4194304,4194304] 0 2024-04-20T21:32:19.594 INFO:tasks.workunit.client.1.smithi189.stdout:9/652: symlink d9/dc/d17/d95/da0/d55/ld1 0 2024-04-20T21:32:19.595 INFO:tasks.workunit.client.1.smithi189.stdout:3/735: write d2/d3/d25/de0/f9b [218781,102427] 0 2024-04-20T21:32:19.597 INFO:tasks.workunit.client.0.smithi062.stdout:8/962: chown d7/d25/d3b/d58/d8b/d24/ff3 100603864 1 2024-04-20T21:32:19.599 INFO:tasks.workunit.client.1.smithi189.stdout:7/772: rmdir d5/d46 39 2024-04-20T21:32:19.602 INFO:tasks.workunit.client.0.smithi062.stdout:5/949: rename d8/d38/d7a/d4c/d7d/d9d/dac/d66/daa/f103 to d8/d118/f13c 0 2024-04-20T21:32:19.604 INFO:tasks.workunit.client.1.smithi189.stdout:0/821: write d3/fe [3360298,61664] 0 2024-04-20T21:32:19.605 INFO:tasks.workunit.client.1.smithi189.stdout:5/739: unlink d7/d9/d51/fdc 0 2024-04-20T21:32:19.605 INFO:tasks.workunit.client.0.smithi062.stdout:5/950: fdatasync d8/d38/fbe 0 2024-04-20T21:32:19.605 INFO:tasks.workunit.client.0.smithi062.stdout:7/971: creat d7/d3e/d33/d3d/dad/f152 x:0 0 0 2024-04-20T21:32:19.605 INFO:tasks.workunit.client.0.smithi062.stdout:8/963: creat d7/d25/d3b/d58/d8b/d24/d46/f135 x:0 0 0 2024-04-20T21:32:19.606 INFO:tasks.workunit.client.1.smithi189.stdout:1/681: rename d8/c25 to d8/df/d50/ceb 0 2024-04-20T21:32:19.609 INFO:tasks.workunit.client.0.smithi062.stdout:7/972: dread - d7/d3e/d33/d3d/d4d/d54/d4f/d9d/f12c zero size 2024-04-20T21:32:19.611 INFO:tasks.workunit.client.1.smithi189.stdout:2/662: dwrite d1/d11/d52/d8b/fa1 [0,4194304] 0 2024-04-20T21:32:19.615 INFO:tasks.workunit.client.0.smithi062.stdout:8/964: creat d7/d30/db3/f136 x:0 0 0 2024-04-20T21:32:19.616 INFO:tasks.workunit.client.1.smithi189.stdout:6/751: dwrite d8/d12/daf/fba [0,4194304] 0 2024-04-20T21:32:19.616 INFO:tasks.workunit.client.1.smithi189.stdout:7/773: unlink d5/d21/c41 0 2024-04-20T21:32:19.619 INFO:tasks.workunit.client.1.smithi189.stdout:8/800: write da/df/d20/d2c/f96 [1034248,28272] 0 2024-04-20T21:32:19.619 INFO:tasks.workunit.client.1.smithi189.stdout:6/752: fdatasync d8/d2b/d49/d54/d67/f7a 0 2024-04-20T21:32:19.621 INFO:tasks.workunit.client.1.smithi189.stdout:5/740: symlink d7/d9/d31/lee 0 2024-04-20T21:32:19.622 INFO:tasks.workunit.client.1.smithi189.stdout:1/682: mknod d8/d21/d2c/cec 0 2024-04-20T21:32:19.632 INFO:tasks.workunit.client.1.smithi189.stdout:8/801: symlink da/de/d1e/d21/d44/l11d 0 2024-04-20T21:32:19.632 INFO:tasks.workunit.client.1.smithi189.stdout:0/822: rename d3/df/d28/dc8/df1/dec to d3/db9/dbe/dd7/d112 0 2024-04-20T21:32:19.632 INFO:tasks.workunit.client.1.smithi189.stdout:7/774: truncate d5/d21/d4a/d52/f7c 761555 0 2024-04-20T21:32:19.635 INFO:tasks.workunit.client.1.smithi189.stdout:5/741: unlink d7/d14/d18/caf 0 2024-04-20T21:32:19.636 INFO:tasks.workunit.client.1.smithi189.stdout:8/802: chown da/df/d3c/d68/d8b/dac/de1/c35 1970116 1 2024-04-20T21:32:19.643 INFO:tasks.workunit.client.1.smithi189.stdout:8/803: mkdir da/df/d13/dd6/dba/d11e 0 2024-04-20T21:32:19.645 INFO:tasks.workunit.client.1.smithi189.stdout:8/804: readlink da/de/d1e/d21/d44/la9 0 2024-04-20T21:32:19.646 INFO:tasks.workunit.client.1.smithi189.stdout:7/775: creat d5/d21/d4a/ffb x:0 0 0 2024-04-20T21:32:19.648 INFO:tasks.workunit.client.1.smithi189.stdout:8/805: fsync da/df/d20/f7d 0 2024-04-20T21:32:19.650 INFO:tasks.workunit.client.1.smithi189.stdout:7/776: chown d5/d21/d97/da3/dc6 8590632 1 2024-04-20T21:32:19.656 INFO:tasks.workunit.client.1.smithi189.stdout:7/777: mkdir d5/d21/d97/dfc 0 2024-04-20T21:32:19.659 INFO:tasks.workunit.client.1.smithi189.stdout:5/742: truncate d7/d9/d30/d55/d61/daa/fae 927475 0 2024-04-20T21:32:19.660 INFO:tasks.workunit.client.1.smithi189.stdout:7/778: truncate d5/d21/d25/ff5 896095 0 2024-04-20T21:32:19.660 INFO:tasks.workunit.client.1.smithi189.stdout:8/806: link da/de/d1e/d21/d44/la9 da/df/d13/d31/l11f 0 2024-04-20T21:32:19.662 INFO:tasks.workunit.client.1.smithi189.stdout:5/743: readlink d7/d9/d31/lee 0 2024-04-20T21:32:19.664 INFO:tasks.workunit.client.1.smithi189.stdout:8/807: chown da/de/d1e/d21/d87/fd1 1135 1 2024-04-20T21:32:19.664 INFO:tasks.workunit.client.1.smithi189.stdout:7/779: read d5/d21/f23 [232683,55320] 0 2024-04-20T21:32:19.664 INFO:tasks.workunit.client.1.smithi189.stdout:3/736: write d2/d3/d88/f99 [613118,77292] 0 2024-04-20T21:32:19.669 INFO:tasks.workunit.client.1.smithi189.stdout:3/737: mknod d2/d8/c102 0 2024-04-20T21:32:19.673 INFO:tasks.workunit.client.1.smithi189.stdout:0/823: dread d3/df/d30/fef [0,4194304] 0 2024-04-20T21:32:19.676 INFO:tasks.workunit.client.1.smithi189.stdout:3/738: write d2/d3/d25/f54 [1201856,14336] 0 2024-04-20T21:32:19.676 INFO:tasks.workunit.client.1.smithi189.stdout:5/744: creat d7/d9/d31/d36/fef x:0 0 0 2024-04-20T21:32:19.679 INFO:tasks.workunit.client.1.smithi189.stdout:9/653: truncate d9/d12/d3b/f4c 1241036 0 2024-04-20T21:32:19.680 INFO:tasks.workunit.client.0.smithi062.stdout:6/821: sync 2024-04-20T21:32:19.680 INFO:tasks.workunit.client.1.smithi189.stdout:0/824: mknod d3/df/d28/dc8/c113 0 2024-04-20T21:32:19.680 INFO:tasks.workunit.client.1.smithi189.stdout:5/745: write d7/d9/d28/f9b [2585974,103192] 0 2024-04-20T21:32:19.683 INFO:tasks.workunit.client.1.smithi189.stdout:8/808: getdents da/df/d3c 0 2024-04-20T21:32:19.688 INFO:tasks.workunit.client.1.smithi189.stdout:1/683: truncate d8/d21/fb3 2631783 0 2024-04-20T21:32:19.691 INFO:tasks.workunit.client.1.smithi189.stdout:5/746: chown d7/d35/fe8 59 1 2024-04-20T21:32:19.694 INFO:tasks.workunit.client.1.smithi189.stdout:9/654: rename d9/f33 to d9/dc/d17/d95/da0/d56/fd2 0 2024-04-20T21:32:19.697 INFO:tasks.workunit.client.1.smithi189.stdout:8/809: symlink da/df/d13/dd6/dba/l120 0 2024-04-20T21:32:19.701 INFO:tasks.workunit.client.1.smithi189.stdout:7/780: dwrite d5/f7 [0,4194304] 0 2024-04-20T21:32:19.702 INFO:tasks.workunit.client.1.smithi189.stdout:3/739: rename d2/f81 to d2/d3/d2c/d30/d3f/f103 0 2024-04-20T21:32:19.702 INFO:tasks.workunit.client.1.smithi189.stdout:6/753: dread d8/da/dd/d2a/d3b/f8e [0,4194304] 0 2024-04-20T21:32:19.706 INFO:tasks.workunit.client.1.smithi189.stdout:7/781: fdatasync d5/d21/d25/f6c 0 2024-04-20T21:32:19.709 INFO:tasks.workunit.client.1.smithi189.stdout:5/747: creat d7/d9/d51/d70/d95/ff0 x:0 0 0 2024-04-20T21:32:19.710 INFO:tasks.workunit.client.1.smithi189.stdout:2/663: dwrite d1/d17/d37/f67 [0,4194304] 0 2024-04-20T21:32:19.719 INFO:tasks.workunit.client.1.smithi189.stdout:9/655: link d9/d12/d3f/d72/fc6 d9/dc/d17/d95/da0/d52/fd3 0 2024-04-20T21:32:19.719 INFO:tasks.workunit.client.1.smithi189.stdout:6/754: dread d8/f9b [0,4194304] 0 2024-04-20T21:32:19.720 INFO:tasks.workunit.client.1.smithi189.stdout:3/740: creat d2/d3/d25/de0/db0/dc8/dc9/f104 x:0 0 0 2024-04-20T21:32:19.724 INFO:tasks.workunit.client.1.smithi189.stdout:0/825: dwrite d3/da8/f10c [0,4194304] 0 2024-04-20T21:32:19.727 INFO:tasks.workunit.client.1.smithi189.stdout:7/782: mkdir d5/d21/d4a/dfd 0 2024-04-20T21:32:19.732 INFO:tasks.workunit.client.1.smithi189.stdout:5/748: rmdir d7/d9/d31/d36/d5f/dab/dc8 0 2024-04-20T21:32:19.733 INFO:tasks.workunit.client.1.smithi189.stdout:1/684: dwrite d8/d21/fdc [0,4194304] 0 2024-04-20T21:32:19.735 INFO:tasks.workunit.client.1.smithi189.stdout:3/741: mknod d2/d3/d25/de0/db0/dc8/c105 0 2024-04-20T21:32:19.737 INFO:tasks.workunit.client.1.smithi189.stdout:0/826: mkdir d3/df/d30/d45/d90/df4/d114 0 2024-04-20T21:32:19.737 INFO:tasks.workunit.client.1.smithi189.stdout:9/656: dread d9/d12/fa8 [0,4194304] 0 2024-04-20T21:32:19.743 INFO:tasks.workunit.client.1.smithi189.stdout:9/657: write d9/dc/d17/d95/da0/d52/d82/fa7 [1436277,73793] 0 2024-04-20T21:32:19.747 INFO:tasks.workunit.client.1.smithi189.stdout:6/755: rename d8/d12/d99/fb2 to d8/d2b/ffa 0 2024-04-20T21:32:19.748 INFO:tasks.workunit.client.1.smithi189.stdout:3/742: rmdir d2/d3/d2c/d31/d90/d92 39 2024-04-20T21:32:19.748 INFO:tasks.workunit.client.1.smithi189.stdout:0/827: creat d3/da8/f115 x:0 0 0 2024-04-20T21:32:19.749 INFO:tasks.workunit.client.1.smithi189.stdout:1/685: dread d8/d1e/d43/d44/dd2/fc7 [0,4194304] 0 2024-04-20T21:32:19.753 INFO:tasks.workunit.client.1.smithi189.stdout:3/743: mknod d2/d3/d25/c106 0 2024-04-20T21:32:19.758 INFO:tasks.workunit.client.1.smithi189.stdout:1/686: creat d8/df/d3a/d49/fed x:0 0 0 2024-04-20T21:32:19.761 INFO:tasks.workunit.client.1.smithi189.stdout:6/756: write d8/d2b/d49/d54/d67/fce [280851,84901] 0 2024-04-20T21:32:19.761 INFO:tasks.workunit.client.0.smithi062.stdout:2/984: sync 2024-04-20T21:32:19.767 INFO:tasks.workunit.client.1.smithi189.stdout:3/744: mkdir d2/d3/d2c/d31/d74/d7a/d107 0 2024-04-20T21:32:19.768 INFO:tasks.workunit.client.1.smithi189.stdout:7/783: dwrite d5/d1d/d64/fd7 [0,4194304] 0 2024-04-20T21:32:19.769 INFO:tasks.workunit.client.1.smithi189.stdout:6/757: creat d8/d12/daf/ffb x:0 0 0 2024-04-20T21:32:19.773 INFO:tasks.workunit.client.1.smithi189.stdout:7/784: write d5/d21/d4a/d52/d74/fbb [1371005,125831] 0 2024-04-20T21:32:19.777 INFO:tasks.workunit.client.1.smithi189.stdout:3/745: mkdir d2/d3/d25/d108 0 2024-04-20T21:32:19.777 INFO:tasks.workunit.client.1.smithi189.stdout:6/758: fsync d8/d2b/d49/d54/fe7 0 2024-04-20T21:32:19.782 INFO:tasks.workunit.client.1.smithi189.stdout:7/785: rmdir d5/d46/d7d 39 2024-04-20T21:32:19.785 INFO:tasks.workunit.client.1.smithi189.stdout:3/746: mkdir d2/d1c/dc1/d109 0 2024-04-20T21:32:19.785 INFO:tasks.workunit.client.1.smithi189.stdout:1/687: dwrite d8/d21/fdc [0,4194304] 0 2024-04-20T21:32:19.787 INFO:tasks.workunit.client.1.smithi189.stdout:6/759: creat d8/d2b/d49/d54/d67/ffc x:0 0 0 2024-04-20T21:32:19.788 INFO:tasks.workunit.client.1.smithi189.stdout:8/810: truncate da/de/d1e/d69/fc8 1522169 0 2024-04-20T21:32:19.788 INFO:tasks.workunit.client.1.smithi189.stdout:7/786: creat d5/d21/d4a/d52/d75/ffe x:0 0 0 2024-04-20T21:32:19.790 INFO:tasks.workunit.client.1.smithi189.stdout:6/760: write d8/d12/fae [4445157,2031] 0 2024-04-20T21:32:19.790 INFO:tasks.workunit.client.1.smithi189.stdout:7/787: chown d5/f43 2434 1 2024-04-20T21:32:19.793 INFO:tasks.workunit.client.1.smithi189.stdout:5/749: rename d7/d9/d31/d36/d5f to d7/d14/d4e/d5b/df1 0 2024-04-20T21:32:19.794 INFO:tasks.workunit.client.1.smithi189.stdout:7/788: dread - d5/d21/d4a/d52/ff1 zero size 2024-04-20T21:32:19.797 INFO:tasks.workunit.client.1.smithi189.stdout:6/761: dread d8/d12/d30/d47/fc9 [0,4194304] 0 2024-04-20T21:32:19.799 INFO:tasks.workunit.client.1.smithi189.stdout:3/747: rename d2/d1c/ceb to d2/d3/d2c/de3/c10a 0 2024-04-20T21:32:19.803 INFO:tasks.workunit.client.0.smithi062.stdout:0/911: sync 2024-04-20T21:32:19.803 INFO:tasks.workunit.client.0.smithi062.stdout:5/951: sync 2024-04-20T21:32:19.805 INFO:tasks.workunit.client.1.smithi189.stdout:5/750: dread d7/d9/d51/d70/d95/fa1 [0,4194304] 0 2024-04-20T21:32:19.807 INFO:tasks.workunit.client.1.smithi189.stdout:6/762: read d8/d2b/f87 [4158121,81854] 0 2024-04-20T21:32:19.810 INFO:tasks.workunit.client.1.smithi189.stdout:0/828: write d3/df/d4c/f7e [452934,2037] 0 2024-04-20T21:32:19.811 INFO:tasks.workunit.client.1.smithi189.stdout:1/688: link d8/d1e/d43/d44/dd2/d8a/lb5 d8/df/d50/lee 0 2024-04-20T21:32:19.812 INFO:tasks.workunit.client.1.smithi189.stdout:3/748: creat d2/d3/d25/de0/db0/dc8/f10b x:0 0 0 2024-04-20T21:32:19.819 INFO:tasks.workunit.client.1.smithi189.stdout:8/811: dwrite da/df/d20/d2c/d9a/ff7 [0,4194304] 0 2024-04-20T21:32:19.820 INFO:tasks.workunit.client.1.smithi189.stdout:2/664: dread d1/d11/d2d/dab/d55/f81 [0,4194304] 0 2024-04-20T21:32:19.821 INFO:tasks.workunit.client.1.smithi189.stdout:5/751: truncate d7/d9/d51/d70/d95/ff0 813539 0 2024-04-20T21:32:19.823 INFO:tasks.workunit.client.0.smithi062.stdout:0/912: dwrite d7/d20/d62/d6f/f108 [0,4194304] 0 2024-04-20T21:32:19.823 INFO:tasks.workunit.client.1.smithi189.stdout:6/763: rename d8/d12/d30/d47/ddc/fe8 to d8/d2b/d5c/ffd 0 2024-04-20T21:32:19.824 INFO:tasks.workunit.client.1.smithi189.stdout:0/829: symlink d3/df/d28/d2b/db1/d111/l116 0 2024-04-20T21:32:19.828 INFO:tasks.workunit.client.1.smithi189.stdout:5/752: stat d7/d14/d4e/d5b/cba 0 2024-04-20T21:32:19.830 INFO:tasks.workunit.client.1.smithi189.stdout:7/789: dwrite d5/d21/d4a/d52/d75/fda [0,4194304] 0 2024-04-20T21:32:19.831 INFO:tasks.workunit.client.1.smithi189.stdout:0/830: chown d3/df/d28/d2b/db1/dfe/f101 3375 1 2024-04-20T21:32:19.832 INFO:tasks.workunit.client.1.smithi189.stdout:6/764: fsync d8/d12/d30/d47/fad 0 2024-04-20T21:32:19.838 INFO:tasks.workunit.client.0.smithi062.stdout:3/883: link d0/d5c/l9a d0/d1/de4/d118/l122 0 2024-04-20T21:32:19.838 INFO:tasks.workunit.client.1.smithi189.stdout:3/749: rename d2/d3/d2c/d31/d74/db7/dd3/de2 to d2/d3/d2c/d30/d10c 0 2024-04-20T21:32:19.839 INFO:tasks.workunit.client.0.smithi062.stdout:3/884: chown d0/d1/d115 13 1 2024-04-20T21:32:19.840 INFO:tasks.workunit.client.1.smithi189.stdout:0/831: symlink d3/df/d30/d45/dd3/l117 0 2024-04-20T21:32:19.842 INFO:tasks.workunit.client.1.smithi189.stdout:6/765: symlink d8/d12/d99/lfe 0 2024-04-20T21:32:19.842 INFO:tasks.workunit.client.1.smithi189.stdout:3/750: dread - d2/d3/d2c/d31/d74/db7/fcf zero size 2024-04-20T21:32:19.846 INFO:tasks.workunit.client.1.smithi189.stdout:7/790: truncate d5/d21/d25/f78 265 0 2024-04-20T21:32:19.846 INFO:tasks.workunit.client.1.smithi189.stdout:5/753: getdents d7/d9/d31/d36 0 2024-04-20T21:32:19.847 INFO:tasks.workunit.client.1.smithi189.stdout:8/812: rename da/df/d3c/d68/df5/cea to da/df/d3c/d68/c121 0 2024-04-20T21:32:19.848 INFO:tasks.workunit.client.1.smithi189.stdout:5/754: readlink d7/l3a 0 2024-04-20T21:32:19.852 INFO:tasks.workunit.client.1.smithi189.stdout:2/665: rename d1/d11/d2d/dab/c46 to d1/d11/d58/d6b/cc3 0 2024-04-20T21:32:19.854 INFO:tasks.workunit.client.1.smithi189.stdout:8/813: creat da/df/d13/dd2/f122 x:0 0 0 2024-04-20T21:32:19.854 INFO:tasks.workunit.client.1.smithi189.stdout:0/832: write d3/d26/d88/fc0 [158416,103247] 0 2024-04-20T21:32:19.854 INFO:tasks.workunit.client.0.smithi062.stdout:2/985: write d7/da/d15/d2f/d38/d6a/d7c/fed [601539,22246] 0 2024-04-20T21:32:19.855 INFO:tasks.workunit.client.1.smithi189.stdout:5/755: write d7/d35/f43 [4911326,59917] 0 2024-04-20T21:32:19.856 INFO:tasks.workunit.client.1.smithi189.stdout:2/666: chown d1/d11/d2d/d4f 127003078 1 2024-04-20T21:32:19.858 INFO:tasks.workunit.client.1.smithi189.stdout:3/751: dread d2/d3/d2c/d31/d4c/f63 [0,4194304] 0 2024-04-20T21:32:19.858 INFO:tasks.workunit.client.1.smithi189.stdout:7/791: creat d5/d1d/d64/db9/fff x:0 0 0 2024-04-20T21:32:19.859 INFO:tasks.workunit.client.1.smithi189.stdout:0/833: read - d3/d8c/fab zero size 2024-04-20T21:32:19.862 INFO:tasks.workunit.client.1.smithi189.stdout:2/667: dread - d1/d17/d37/d89/f95 zero size 2024-04-20T21:32:19.863 INFO:tasks.workunit.client.1.smithi189.stdout:7/792: write d5/d21/d25/ff5 [124707,74521] 0 2024-04-20T21:32:19.865 INFO:tasks.workunit.client.1.smithi189.stdout:2/668: read d1/f2a [1045000,1621] 0 2024-04-20T21:32:19.866 INFO:tasks.workunit.client.1.smithi189.stdout:6/766: getdents d8/d10/d78/df7 0 2024-04-20T21:32:19.868 INFO:tasks.workunit.client.0.smithi062.stdout:3/885: mknod d0/d1/d10/d3a/d5e/c123 0 2024-04-20T21:32:19.870 INFO:tasks.workunit.client.1.smithi189.stdout:0/834: symlink d3/df/d30/d37/dbd/l118 0 2024-04-20T21:32:19.871 INFO:tasks.workunit.client.1.smithi189.stdout:3/752: read d2/d8d/f91 [92193,28805] 0 2024-04-20T21:32:19.873 INFO:tasks.workunit.client.1.smithi189.stdout:4/674: sync 2024-04-20T21:32:19.876 INFO:tasks.workunit.client.0.smithi062.stdout:2/986: mknod d7/da/d15/d2f/d38/d10f/d121/c144 0 2024-04-20T21:32:19.878 INFO:tasks.workunit.client.1.smithi189.stdout:5/756: rename d7/d9/d51/l63 to d7/d9/d30/lf2 0 2024-04-20T21:32:19.881 INFO:tasks.workunit.client.1.smithi189.stdout:0/835: mkdir d3/df/d28/d2b/db1/dfe/d119 0 2024-04-20T21:32:19.883 INFO:tasks.workunit.client.1.smithi189.stdout:4/675: write d2/d3a/f63 [2863877,45599] 0 2024-04-20T21:32:19.887 INFO:tasks.workunit.client.1.smithi189.stdout:2/669: rename d1/f5 to d1/d11/d2d/d4f/d8e/fc4 0 2024-04-20T21:32:19.887 INFO:tasks.workunit.client.0.smithi062.stdout:3/886: dwrite d0/d1c/d63/d42/f121 [0,4194304] 0 2024-04-20T21:32:19.889 INFO:tasks.workunit.client.0.smithi062.stdout:6/822: getdents d0/d1a/d7d/db1/de0 0 2024-04-20T21:32:19.889 INFO:tasks.workunit.client.0.smithi062.stdout:3/887: fsync d0/d1/fb 0 2024-04-20T21:32:19.894 INFO:tasks.workunit.client.0.smithi062.stdout:6/823: truncate d0/d9/d11/d7b/d99/ddd/f10d 380875 0 2024-04-20T21:32:19.896 INFO:tasks.workunit.client.0.smithi062.stdout:2/987: symlink d7/da/d15/d2f/d38/d51/dbc/l145 0 2024-04-20T21:32:19.897 INFO:tasks.workunit.client.1.smithi189.stdout:5/757: truncate d7/d14/d4e/d5b/f68 295154 0 2024-04-20T21:32:19.901 INFO:tasks.workunit.client.1.smithi189.stdout:6/767: write d8/d12/d30/d47/fad [1023095,87828] 0 2024-04-20T21:32:19.905 INFO:tasks.workunit.client.1.smithi189.stdout:5/758: write d7/d14/d18/f53 [3336189,6312] 0 2024-04-20T21:32:19.908 INFO:tasks.workunit.client.0.smithi062.stdout:3/888: creat d0/d1/d3c/d43/d72/df2/f124 x:0 0 0 2024-04-20T21:32:19.912 INFO:tasks.workunit.client.0.smithi062.stdout:5/952: link d8/d38/f70 d8/d38/d7a/d4c/d7d/d9d/dac/d2f/df3/f13d 0 2024-04-20T21:32:19.916 INFO:tasks.workunit.client.0.smithi062.stdout:6/824: unlink d0/d9/df5/c10a 0 2024-04-20T21:32:19.920 INFO:tasks.workunit.client.1.smithi189.stdout:8/814: dwrite da/de/d1e/d21/f67 [0,4194304] 0 2024-04-20T21:32:19.920 INFO:tasks.workunit.client.1.smithi189.stdout:4/676: dwrite d2/dd/d37/dcd/d6a/fd0 [0,4194304] 0 2024-04-20T21:32:19.921 INFO:tasks.workunit.client.1.smithi189.stdout:3/753: dwrite d2/d3/d2c/d30/d3f/f8a [0,4194304] 0 2024-04-20T21:32:19.924 INFO:tasks.workunit.client.1.smithi189.stdout:5/759: dread d7/d35/d48/f6a [0,4194304] 0 2024-04-20T21:32:19.929 INFO:tasks.workunit.client.1.smithi189.stdout:0/836: rename d3/df/d5d/d64/f6b to d3/f11a 0 2024-04-20T21:32:19.931 INFO:tasks.workunit.client.1.smithi189.stdout:6/768: write d8/d12/d30/d47/da0/fc2 [797711,53583] 0 2024-04-20T21:32:19.932 INFO:tasks.workunit.client.1.smithi189.stdout:5/760: rmdir d7/d9/d51/ddf 39 2024-04-20T21:32:19.935 INFO:tasks.workunit.client.1.smithi189.stdout:0/837: fsync d3/d26/d88/fc0 0 2024-04-20T21:32:19.937 INFO:tasks.workunit.client.1.smithi189.stdout:4/677: truncate d2/dd/d37/dcd/f9e 1432188 0 2024-04-20T21:32:19.940 INFO:tasks.workunit.client.1.smithi189.stdout:2/670: write d1/d11/d52/f6a [1709175,57931] 0 2024-04-20T21:32:19.940 INFO:tasks.workunit.client.1.smithi189.stdout:5/761: fsync d7/d14/d4e/d5b/df1/dab/dcb/fe4 0 2024-04-20T21:32:19.943 INFO:tasks.workunit.client.1.smithi189.stdout:5/762: dread - d7/d9/d28/da3/fa8 zero size 2024-04-20T21:32:19.948 INFO:tasks.workunit.client.1.smithi189.stdout:8/815: rename da/df to da/d81/d123 0 2024-04-20T21:32:19.953 INFO:tasks.workunit.client.1.smithi189.stdout:2/671: rmdir d1/d11/d2d/dab/d55 39 2024-04-20T21:32:19.956 INFO:tasks.workunit.client.1.smithi189.stdout:0/838: creat d3/df/d4c/f11b x:0 0 0 2024-04-20T21:32:19.963 INFO:tasks.workunit.client.1.smithi189.stdout:0/839: write d3/fd [3979855,54862] 0 2024-04-20T21:32:19.968 INFO:tasks.workunit.client.1.smithi189.stdout:3/754: write d2/d8/f58 [378118,28697] 0 2024-04-20T21:32:19.969 INFO:tasks.workunit.client.1.smithi189.stdout:0/840: chown d3/df/d28/d2b/d2c/f62 3580796 1 2024-04-20T21:32:19.970 INFO:tasks.workunit.client.0.smithi062.stdout:7/973: rename d7/d3e/d33/d3d/d4d/d54/dea/d80 to d7/d26/da8/d153 0 2024-04-20T21:32:19.976 INFO:tasks.workunit.client.1.smithi189.stdout:8/816: dwrite da/d81/d123/d13/dd2/f98 [0,4194304] 0 2024-04-20T21:32:19.978 INFO:tasks.workunit.client.1.smithi189.stdout:4/678: dwrite d2/dd/d37/f59 [0,4194304] 0 2024-04-20T21:32:19.978 INFO:tasks.workunit.client.0.smithi062.stdout:3/889: rename d0/d1c/d63/l2e to d0/d1b/d74/dc7/l125 0 2024-04-20T21:32:19.982 INFO:tasks.workunit.client.0.smithi062.stdout:3/890: dread d0/d1c/d63/f91 [0,4194304] 0 2024-04-20T21:32:19.984 INFO:tasks.workunit.client.0.smithi062.stdout:3/891: dread - d0/d1/d3c/d43/d72/da9/f117 zero size 2024-04-20T21:32:19.991 INFO:tasks.workunit.client.1.smithi189.stdout:6/769: dwrite d8/d12/daf/fba [0,4194304] 0 2024-04-20T21:32:19.992 INFO:tasks.workunit.client.1.smithi189.stdout:4/679: creat d2/dd/d37/ff6 x:0 0 0 2024-04-20T21:32:19.993 INFO:tasks.workunit.client.0.smithi062.stdout:3/892: truncate d0/d1/d3c/d43/fe7 487427 0 2024-04-20T21:32:19.997 INFO:tasks.workunit.client.1.smithi189.stdout:2/672: dwrite d1/d11/d2d/dab/d4a/f7b [0,4194304] 0 2024-04-20T21:32:19.997 INFO:tasks.workunit.client.1.smithi189.stdout:3/755: dread d2/d3/d2c/d31/f6f [0,4194304] 0 2024-04-20T21:32:19.998 INFO:tasks.workunit.client.0.smithi062.stdout:7/974: symlink d7/d3e/d33/d3d/d4d/d54/d4f/l154 0 2024-04-20T21:32:20.001 INFO:tasks.workunit.client.0.smithi062.stdout:7/975: write d7/d26/da8/f111 [76659,58646] 0 2024-04-20T21:32:20.001 INFO:tasks.workunit.client.1.smithi189.stdout:6/770: creat d8/d12/df1/fff x:0 0 0 2024-04-20T21:32:20.001 INFO:tasks.workunit.client.1.smithi189.stdout:5/763: dwrite d7/d14/d4e/d5b/fc2 [0,4194304] 0 2024-04-20T21:32:20.002 INFO:tasks.workunit.client.0.smithi062.stdout:7/976: readlink d7/d26/dac/ddb/lf0 0 2024-04-20T21:32:20.004 INFO:tasks.workunit.client.1.smithi189.stdout:2/673: chown d1/d11/d58/d6b/f6c 3 1 2024-04-20T21:32:20.006 INFO:tasks.workunit.client.1.smithi189.stdout:6/771: write d8/d2b/fe5 [771018,75901] 0 2024-04-20T21:32:20.009 INFO:tasks.workunit.client.0.smithi062.stdout:7/977: symlink d7/d26/dac/d10a/l155 0 2024-04-20T21:32:20.010 INFO:tasks.workunit.client.0.smithi062.stdout:6/825: dread d0/fb6 [4194304,4194304] 0 2024-04-20T21:32:20.011 INFO:tasks.workunit.client.1.smithi189.stdout:5/764: creat d7/d9/d28/da3/ff3 x:0 0 0 2024-04-20T21:32:20.012 INFO:tasks.workunit.client.1.smithi189.stdout:0/841: dread d3/df/d28/d2b/f86 [0,4194304] 0 2024-04-20T21:32:20.013 INFO:tasks.workunit.client.1.smithi189.stdout:6/772: creat d8/d12/d30/d47/da0/f100 x:0 0 0 2024-04-20T21:32:20.015 INFO:tasks.workunit.client.1.smithi189.stdout:5/765: write d7/d35/f43 [5419321,81722] 0 2024-04-20T21:32:20.017 INFO:tasks.workunit.client.1.smithi189.stdout:2/674: truncate d1/d11/d2d/d4f/d8e/fc0 427441 0 2024-04-20T21:32:20.020 INFO:tasks.workunit.client.1.smithi189.stdout:5/766: chown d7/d9/d30/d55/d61/daa 0 1 2024-04-20T21:32:20.021 INFO:tasks.workunit.client.1.smithi189.stdout:0/842: fdatasync d3/df/d30/d45/dd3/fee 0 2024-04-20T21:32:20.022 INFO:tasks.workunit.client.0.smithi062.stdout:6/826: unlink d0/d1a/l31 0 2024-04-20T21:32:20.023 INFO:tasks.workunit.client.1.smithi189.stdout:4/680: dwrite d2/dd/d11/f58 [4194304,4194304] 0 2024-04-20T21:32:20.023 INFO:tasks.workunit.client.0.smithi062.stdout:6/827: chown d0/dc7/l76 7605 1 2024-04-20T21:32:20.027 INFO:tasks.workunit.client.1.smithi189.stdout:5/767: fdatasync d7/d9/d30/f8e 0 2024-04-20T21:32:20.031 INFO:tasks.workunit.client.0.smithi062.stdout:6/828: chown d0/d9/d11/d22/l37 1 1 2024-04-20T21:32:20.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:19 smithi062 ceph-mon[16221]: pgmap v24: 65 pgs: 65 active+clean; 3.3 GiB data, 9.6 GiB used, 527 GiB / 536 GiB avail; 121 MiB/s rd, 305 MiB/s wr, 648 op/s 2024-04-20T21:32:20.036 INFO:tasks.workunit.client.1.smithi189.stdout:2/675: mkdir d1/d17/d37/d89/dc5 0 2024-04-20T21:32:20.036 INFO:tasks.workunit.client.1.smithi189.stdout:5/768: chown d7/d9/d30/d55/d33 2 1 2024-04-20T21:32:20.038 INFO:tasks.workunit.client.1.smithi189.stdout:9/658: sync 2024-04-20T21:32:20.039 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:19 smithi189 ceph-mon[21562]: pgmap v24: 65 pgs: 65 active+clean; 3.3 GiB data, 9.6 GiB used, 527 GiB / 536 GiB avail; 121 MiB/s rd, 305 MiB/s wr, 648 op/s 2024-04-20T21:32:20.043 INFO:tasks.workunit.client.1.smithi189.stdout:5/769: symlink d7/d9/d30/d55/d33/lf4 0 2024-04-20T21:32:20.043 INFO:tasks.workunit.client.1.smithi189.stdout:9/659: symlink d9/d12/d3f/ld4 0 2024-04-20T21:32:20.045 INFO:tasks.workunit.client.1.smithi189.stdout:2/676: dread d1/d11/d2d/d4f/db4/f9c [0,4194304] 0 2024-04-20T21:32:20.049 INFO:tasks.workunit.client.1.smithi189.stdout:5/770: creat d7/d9/d28/da3/ff5 x:0 0 0 2024-04-20T21:32:20.055 INFO:tasks.workunit.client.1.smithi189.stdout:2/677: rmdir d1/d11/d52/d8b 39 2024-04-20T21:32:20.060 INFO:tasks.workunit.client.1.smithi189.stdout:2/678: fdatasync d1/d17/f2b 0 2024-04-20T21:32:20.061 INFO:tasks.workunit.client.1.smithi189.stdout:0/843: dwrite d3/df/d30/d45/d90/faa [0,4194304] 0 2024-04-20T21:32:20.065 INFO:tasks.workunit.client.1.smithi189.stdout:2/679: mkdir d1/d11/d2d/d73/dc6 0 2024-04-20T21:32:20.067 INFO:tasks.workunit.client.1.smithi189.stdout:0/844: read - d3/df/d28/dc8/df1/ff5 zero size 2024-04-20T21:32:20.071 INFO:tasks.workunit.client.1.smithi189.stdout:2/680: read d1/d11/d99/db7/fb8 [449215,56074] 0 2024-04-20T21:32:20.078 INFO:tasks.workunit.client.1.smithi189.stdout:2/681: creat d1/d17/d78/db9/fc7 x:0 0 0 2024-04-20T21:32:20.079 INFO:tasks.workunit.client.0.smithi062.stdout:8/965: dwrite d7/d25/d3b/d58/f63 [0,4194304] 0 2024-04-20T21:32:20.080 INFO:tasks.workunit.client.0.smithi062.stdout:0/913: dwrite d7/f97 [0,4194304] 0 2024-04-20T21:32:20.085 INFO:tasks.workunit.client.1.smithi189.stdout:2/682: creat d1/d11/d52/d8b/fc8 x:0 0 0 2024-04-20T21:32:20.086 INFO:tasks.workunit.client.1.smithi189.stdout:1/689: sync 2024-04-20T21:32:20.092 INFO:tasks.workunit.client.1.smithi189.stdout:2/683: mkdir d1/d11/d2d/dab/d4a/d66/dc9 0 2024-04-20T21:32:20.093 INFO:tasks.workunit.client.0.smithi062.stdout:0/914: mknod d7/d1c/d15/d17/d37/dfb/dfe/c11d 0 2024-04-20T21:32:20.095 INFO:tasks.workunit.client.1.smithi189.stdout:4/681: dread d2/dd/d11/f73 [0,4194304] 0 2024-04-20T21:32:20.096 INFO:tasks.workunit.client.1.smithi189.stdout:2/684: write d1/d11/d2d/fa6 [635444,129433] 0 2024-04-20T21:32:20.098 INFO:tasks.workunit.client.0.smithi062.stdout:5/953: truncate d8/f101 1143582 0 2024-04-20T21:32:20.102 INFO:tasks.workunit.client.1.smithi189.stdout:2/685: creat d1/d17/d78/fca x:0 0 0 2024-04-20T21:32:20.102 INFO:tasks.workunit.client.0.smithi062.stdout:0/915: creat d7/d1c/d15/d17/d1b/d106/df1/d10e/f11e x:0 0 0 2024-04-20T21:32:20.104 INFO:tasks.workunit.client.1.smithi189.stdout:2/686: chown d1/d17 150 1 2024-04-20T21:32:20.113 INFO:tasks.workunit.client.0.smithi062.stdout:6/829: dread d0/d9/d11/d22/dd5/f3b [0,4194304] 0 2024-04-20T21:32:20.113 INFO:tasks.workunit.client.0.smithi062.stdout:8/966: rmdir d7/d25/d3b/d58/d8b/d24/dbd 39 2024-04-20T21:32:20.113 INFO:tasks.workunit.client.0.smithi062.stdout:8/967: chown d7/d30/db3 1019 1 2024-04-20T21:32:20.113 INFO:tasks.workunit.client.0.smithi062.stdout:5/954: unlink d8/d38/d7a/d4c/d7d/d9d/dac/d66/daa/fab 0 2024-04-20T21:32:20.113 INFO:tasks.workunit.client.0.smithi062.stdout:2/988: dwrite d7/db5/fff [0,4194304] 0 2024-04-20T21:32:20.114 INFO:tasks.workunit.client.1.smithi189.stdout:2/687: mknod d1/d11/d2d/dab/d4a/d66/ccb 0 2024-04-20T21:32:20.116 INFO:tasks.workunit.client.1.smithi189.stdout:4/682: dwrite d2/dd/d11/f7e [0,4194304] 0 2024-04-20T21:32:20.126 INFO:tasks.workunit.client.0.smithi062.stdout:2/989: creat d7/da/d15/d2f/d97/f146 x:0 0 0 2024-04-20T21:32:20.135 INFO:tasks.workunit.client.1.smithi189.stdout:2/688: dwrite d1/d11/d2d/dab/d74/fb3 [0,4194304] 0 2024-04-20T21:32:20.136 INFO:tasks.workunit.client.0.smithi062.stdout:7/978: write d7/d26/da8/d153/fdd [1117245,36804] 0 2024-04-20T21:32:20.137 INFO:tasks.workunit.client.0.smithi062.stdout:3/893: dwrite d0/d1/d10/d3a/d5e/ddb/fee [0,4194304] 0 2024-04-20T21:32:20.138 INFO:tasks.workunit.client.0.smithi062.stdout:7/979: chown d7/d26 5 1 2024-04-20T21:32:20.139 INFO:tasks.workunit.client.0.smithi062.stdout:5/955: dwrite d8/d38/d7a/d4c/d7d/d9d/dac/d66/dae/f135 [0,4194304] 0 2024-04-20T21:32:20.141 INFO:tasks.workunit.client.0.smithi062.stdout:6/830: creat d0/d9/d11/d22/dd5/d67/d75/f116 x:0 0 0 2024-04-20T21:32:20.142 INFO:tasks.workunit.client.1.smithi189.stdout:2/689: fsync d1/d11/d2d/d4f/fbe 0 2024-04-20T21:32:20.143 INFO:tasks.workunit.client.0.smithi062.stdout:6/831: fdatasync d0/d4f/fdf 0 2024-04-20T21:32:20.145 INFO:tasks.workunit.client.1.smithi189.stdout:3/756: write d2/d3/d2c/d30/d3f/f103 [110473,37475] 0 2024-04-20T21:32:20.146 INFO:tasks.workunit.client.1.smithi189.stdout:7/793: sync 2024-04-20T21:32:20.148 INFO:tasks.workunit.client.1.smithi189.stdout:4/683: dread d2/dd/d37/dcd/d6a/d7a/dce/ff4 [4194304,4194304] 0 2024-04-20T21:32:20.149 INFO:tasks.workunit.client.1.smithi189.stdout:6/773: write d8/d12/d99/fca [146622,91505] 0 2024-04-20T21:32:20.151 INFO:tasks.workunit.client.1.smithi189.stdout:5/771: truncate d7/d9/d31/fec 4141282 0 2024-04-20T21:32:20.152 INFO:tasks.workunit.client.1.smithi189.stdout:9/660: write d9/dc/d17/d36/f57 [3560352,21316] 0 2024-04-20T21:32:20.152 INFO:tasks.workunit.client.1.smithi189.stdout:7/794: dread - d5/d21/d25/d34/fec zero size 2024-04-20T21:32:20.154 INFO:tasks.workunit.client.1.smithi189.stdout:0/845: write d3/df/d28/d2b/d2c/f62 [1783968,108081] 0 2024-04-20T21:32:20.156 INFO:tasks.workunit.client.0.smithi062.stdout:7/980: mkdir d7/d3e/d33/d3d/d7f/d116/d156 0 2024-04-20T21:32:20.157 INFO:tasks.workunit.client.1.smithi189.stdout:8/817: dwrite da/d81/d123/d20/d2c/d9a/fcc [0,4194304] 0 2024-04-20T21:32:20.161 INFO:tasks.workunit.client.0.smithi062.stdout:8/968: getdents d7/d25/d70/df7 0 2024-04-20T21:32:20.161 INFO:tasks.workunit.client.0.smithi062.stdout:8/969: chown d7 2 1 2024-04-20T21:32:20.163 INFO:tasks.workunit.client.1.smithi189.stdout:2/690: truncate d1/d11/d2d/fa6 5279753 0 2024-04-20T21:32:20.163 INFO:tasks.workunit.client.1.smithi189.stdout:4/684: dread - d2/dd/d11/f4b zero size 2024-04-20T21:32:20.164 INFO:tasks.workunit.client.0.smithi062.stdout:2/990: read d7/d39/fc9 [185871,87416] 0 2024-04-20T21:32:20.166 INFO:tasks.workunit.client.1.smithi189.stdout:3/757: write d2/d3/d2c/fee [1483733,87589] 0 2024-04-20T21:32:20.166 INFO:tasks.workunit.client.1.smithi189.stdout:9/661: write d9/d12/d50/f4f [6900338,14478] 0 2024-04-20T21:32:20.169 INFO:tasks.workunit.client.0.smithi062.stdout:6/832: chown d0/d9/d11/d7b/fb2 77985 1 2024-04-20T21:32:20.170 INFO:tasks.workunit.client.0.smithi062.stdout:7/981: symlink d7/d26/dac/ddb/d132/l157 0 2024-04-20T21:32:20.171 INFO:tasks.workunit.client.1.smithi189.stdout:2/691: creat d1/d17/d37/d89/fcc x:0 0 0 2024-04-20T21:32:20.173 INFO:tasks.workunit.client.1.smithi189.stdout:9/662: stat d9/dc/d17/d36/f57 0 2024-04-20T21:32:20.176 INFO:tasks.workunit.client.1.smithi189.stdout:7/795: rename d5/d46/l9e to d5/d59/l100 0 2024-04-20T21:32:20.177 INFO:tasks.workunit.client.0.smithi062.stdout:7/982: chown d7/d26/da8/d153 3997 1 2024-04-20T21:32:20.177 INFO:tasks.workunit.client.0.smithi062.stdout:8/970: rename d7/d25/d3b/d58/d8b/f1e to d7/d30/f137 0 2024-04-20T21:32:20.178 INFO:tasks.workunit.client.0.smithi062.stdout:8/971: stat d7/d25/l37 0 2024-04-20T21:32:20.181 INFO:tasks.workunit.client.1.smithi189.stdout:1/690: dread d8/d9b/fa6 [0,4194304] 0 2024-04-20T21:32:20.181 INFO:tasks.workunit.client.1.smithi189.stdout:8/818: creat da/d81/d123/d20/d2c/d9a/ddd/f124 x:0 0 0 2024-04-20T21:32:20.181 INFO:tasks.workunit.client.0.smithi062.stdout:7/983: truncate d7/d3e/d33/d3d/d4d/d54/dea/f10b 656961 0 2024-04-20T21:32:20.181 INFO:tasks.workunit.client.0.smithi062.stdout:2/991: read d7/da/d15/d2f/d38/fbf [2797680,47333] 0 2024-04-20T21:32:20.182 INFO:tasks.workunit.client.1.smithi189.stdout:9/663: fdatasync f1 0 2024-04-20T21:32:20.183 INFO:tasks.workunit.client.1.smithi189.stdout:5/772: rename d7/d9/d28/fb0 to d7/d9/d30/d55/d61/daa/ff6 0 2024-04-20T21:32:20.183 INFO:tasks.workunit.client.0.smithi062.stdout:6/833: creat d0/d9/d11/d7b/d99/ddd/de8/f117 x:0 0 0 2024-04-20T21:32:20.186 INFO:tasks.workunit.client.0.smithi062.stdout:8/972: fsync d7/d25/d4f/f104 0 2024-04-20T21:32:20.192 INFO:tasks.workunit.client.0.smithi062.stdout:5/956: dwrite d8/f101 [0,4194304] 0 2024-04-20T21:32:20.192 INFO:tasks.workunit.client.1.smithi189.stdout:7/796: getdents d5/d1d/d64/db9 0 2024-04-20T21:32:20.193 INFO:tasks.workunit.client.0.smithi062.stdout:5/957: chown d8/d38 124818071 1 2024-04-20T21:32:20.194 INFO:tasks.workunit.client.1.smithi189.stdout:2/692: getdents d1/d11/d2d/d4f 0 2024-04-20T21:32:20.194 INFO:tasks.workunit.client.1.smithi189.stdout:0/846: rename d3/df/d30/d45/d90/faa to d3/df/d5d/f11c 0 2024-04-20T21:32:20.194 INFO:tasks.workunit.client.1.smithi189.stdout:3/758: rename d2 to d2/d3/d25/de0/d10d 22 2024-04-20T21:32:20.197 INFO:tasks.workunit.client.0.smithi062.stdout:7/984: rename d7/d63/f8d to d7/d3e/d33/d3d/d4d/d54/dea/dcc/f158 0 2024-04-20T21:32:20.198 INFO:tasks.workunit.client.1.smithi189.stdout:5/773: rename d7/d9 to d7/d9/d31/df7 22 2024-04-20T21:32:20.199 INFO:tasks.workunit.client.0.smithi062.stdout:7/985: write d7/d3e/d33/d3d/d4d/f13e [366605,9237] 0 2024-04-20T21:32:20.200 INFO:tasks.workunit.client.0.smithi062.stdout:6/834: mknod d0/d9/d11/d22/dd5/d67/c118 0 2024-04-20T21:32:20.204 INFO:tasks.workunit.client.1.smithi189.stdout:4/685: dwrite d2/f54 [0,4194304] 0 2024-04-20T21:32:20.206 INFO:tasks.workunit.client.0.smithi062.stdout:8/973: rename d7/d25/d5c/lff to d7/d25/d70/df7/l138 0 2024-04-20T21:32:20.207 INFO:tasks.workunit.client.1.smithi189.stdout:7/797: mkdir d5/d21/d4a/dfd/d101 0 2024-04-20T21:32:20.208 INFO:tasks.workunit.client.1.smithi189.stdout:2/693: mknod d1/d11/d2d/dab/ccd 0 2024-04-20T21:32:20.214 INFO:tasks.workunit.client.1.smithi189.stdout:6/774: dread d8/f63 [0,4194304] 0 2024-04-20T21:32:20.216 INFO:tasks.workunit.client.1.smithi189.stdout:4/686: symlink d2/d20/dc6/dda/lf7 0 2024-04-20T21:32:20.220 INFO:tasks.workunit.client.1.smithi189.stdout:6/775: write d8/da/dd/d35/fc8 [15960,26474] 0 2024-04-20T21:32:20.221 INFO:tasks.workunit.client.0.smithi062.stdout:6/835: mknod d0/d1a/d44/da7/c119 0 2024-04-20T21:32:20.222 INFO:tasks.workunit.client.0.smithi062.stdout:3/894: write d0/d5c/f5a [1226235,98378] 0 2024-04-20T21:32:20.225 INFO:tasks.workunit.client.0.smithi062.stdout:3/895: dread - d0/d1/d10/d3a/f107 zero size 2024-04-20T21:32:20.226 INFO:tasks.workunit.client.1.smithi189.stdout:1/691: dread d8/d21/d2c/f75 [0,4194304] 0 2024-04-20T21:32:20.227 INFO:tasks.workunit.client.1.smithi189.stdout:4/687: link d2/d3a/f3c d2/d22/ff8 0 2024-04-20T21:32:20.228 INFO:tasks.workunit.client.1.smithi189.stdout:6/776: symlink d8/d12/l101 0 2024-04-20T21:32:20.229 INFO:tasks.workunit.client.0.smithi062.stdout:5/958: rename d8/d10/f74 to d8/d1e/d2c/f13e 0 2024-04-20T21:32:20.230 INFO:tasks.workunit.client.1.smithi189.stdout:5/774: dwrite d7/d9/d30/d55/d61/fb2 [0,4194304] 0 2024-04-20T21:32:20.236 INFO:tasks.workunit.client.0.smithi062.stdout:7/986: mknod d7/d3e/d33/d3d/d4d/d54/d4f/d9d/d150/c159 0 2024-04-20T21:32:20.238 INFO:tasks.workunit.client.1.smithi189.stdout:4/688: mkdir d2/dd/d37/dcd/df9 0 2024-04-20T21:32:20.239 INFO:tasks.workunit.client.1.smithi189.stdout:5/775: rmdir d7/d35 39 2024-04-20T21:32:20.239 INFO:tasks.workunit.client.1.smithi189.stdout:6/777: rmdir d8/da/dd/d35/d6d 39 2024-04-20T21:32:20.240 INFO:tasks.workunit.client.0.smithi062.stdout:8/974: mkdir d7/da9/d139 0 2024-04-20T21:32:20.241 INFO:tasks.workunit.client.1.smithi189.stdout:7/798: dwrite d5/d21/d25/ff5 [0,4194304] 0 2024-04-20T21:32:20.242 INFO:tasks.workunit.client.0.smithi062.stdout:8/975: write d7/d9/ff [3143494,47441] 0 2024-04-20T21:32:20.244 INFO:tasks.workunit.client.1.smithi189.stdout:6/778: chown d8/d12/fae 160557 1 2024-04-20T21:32:20.246 INFO:tasks.workunit.client.1.smithi189.stdout:4/689: creat d2/dd/ffa x:0 0 0 2024-04-20T21:32:20.247 INFO:tasks.workunit.client.1.smithi189.stdout:5/776: read d7/d9/d30/d5a/f64 [3798398,30553] 0 2024-04-20T21:32:20.247 INFO:tasks.workunit.client.1.smithi189.stdout:1/692: link d8/d1e/d28/d4f/lc8 d8/d21/d2c/lef 0 2024-04-20T21:32:20.249 INFO:tasks.workunit.client.1.smithi189.stdout:7/799: write d5/d21/f32 [3699143,44896] 0 2024-04-20T21:32:20.254 INFO:tasks.workunit.client.1.smithi189.stdout:1/693: chown d8/d1e/d54/fae 1034149826 1 2024-04-20T21:32:20.254 INFO:tasks.workunit.client.0.smithi062.stdout:6/836: rename d0/d1a/d44/ff6 to d0/d52/da0/f11a 0 2024-04-20T21:32:20.255 INFO:tasks.workunit.client.1.smithi189.stdout:5/777: mknod d7/d9/d30/d5a/d9d/cf8 0 2024-04-20T21:32:20.255 INFO:tasks.workunit.client.0.smithi062.stdout:6/837: stat d0/d52/fce 0 2024-04-20T21:32:20.258 INFO:tasks.workunit.client.0.smithi062.stdout:5/959: creat d8/d38/d7a/d4c/d7d/d9d/dac/d105/f13f x:0 0 0 2024-04-20T21:32:20.259 INFO:tasks.workunit.client.1.smithi189.stdout:4/690: mkdir d2/dd/d37/dcd/d6a/d7a/db7/dfb 0 2024-04-20T21:32:20.261 INFO:tasks.workunit.client.1.smithi189.stdout:7/800: truncate d5/f1f 3974233 0 2024-04-20T21:32:20.261 INFO:tasks.workunit.client.0.smithi062.stdout:6/838: readlink d0/dc7/d7c/dc5/def/l9f 0 2024-04-20T21:32:20.263 INFO:tasks.workunit.client.1.smithi189.stdout:7/801: readlink d5/d21/d25/l29 0 2024-04-20T21:32:20.266 INFO:tasks.workunit.client.1.smithi189.stdout:7/802: write d5/d59/ffa [409153,114510] 0 2024-04-20T21:32:20.267 INFO:tasks.workunit.client.1.smithi189.stdout:5/778: write d7/d9/d51/d70/d95/fa1 [966309,108526] 0 2024-04-20T21:32:20.269 INFO:tasks.workunit.client.1.smithi189.stdout:4/691: dread d2/d22/d57/f99 [0,4194304] 0 2024-04-20T21:32:20.273 INFO:tasks.workunit.client.1.smithi189.stdout:8/819: write da/de/d1e/d21/d44/f47 [1682924,75356] 0 2024-04-20T21:32:20.276 INFO:tasks.workunit.client.0.smithi062.stdout:2/992: truncate d7/da/d15/d2f/d38/d6a/d7c/f8c 2399118 0 2024-04-20T21:32:20.277 INFO:tasks.workunit.client.0.smithi062.stdout:2/993: chown d7/d39/f55 328981236 1 2024-04-20T21:32:20.279 INFO:tasks.workunit.client.0.smithi062.stdout:8/976: dread d7/d25/f42 [0,4194304] 0 2024-04-20T21:32:20.279 INFO:tasks.workunit.client.0.smithi062.stdout:6/839: unlink d0/d9/d11/d22/c39 0 2024-04-20T21:32:20.282 INFO:tasks.workunit.client.1.smithi189.stdout:8/820: write da/de/d1e/d21/d87/ff6 [1296677,43398] 0 2024-04-20T21:32:20.284 INFO:tasks.workunit.client.1.smithi189.stdout:5/779: dread d7/d9/d51/d70/d95/fa1 [0,4194304] 0 2024-04-20T21:32:20.288 INFO:tasks.workunit.client.0.smithi062.stdout:2/994: truncate d7/da/d15/d2f/d38/d51/f60 3117541 0 2024-04-20T21:32:20.289 INFO:tasks.workunit.client.0.smithi062.stdout:3/896: write d0/d1/d115/d11c/f68 [864817,79392] 0 2024-04-20T21:32:20.289 INFO:tasks.workunit.client.1.smithi189.stdout:3/759: dwrite d2/d8/f45 [0,4194304] 0 2024-04-20T21:32:20.289 INFO:tasks.workunit.client.1.smithi189.stdout:8/821: chown da/d81/d123/d3c/deb 62675136 1 2024-04-20T21:32:20.296 INFO:tasks.workunit.client.1.smithi189.stdout:0/847: dwrite d3/df/d28/d2b/f67 [0,4194304] 0 2024-04-20T21:32:20.296 INFO:tasks.workunit.client.1.smithi189.stdout:8/822: stat da/de/d1e/d21/d87/ff6 0 2024-04-20T21:32:20.298 INFO:tasks.workunit.client.1.smithi189.stdout:5/780: rmdir d7/d9/d28/db1 39 2024-04-20T21:32:20.299 INFO:tasks.workunit.client.1.smithi189.stdout:9/664: dwrite d9/dc/d17/d95/da0/d52/fd3 [0,4194304] 0 2024-04-20T21:32:20.300 INFO:tasks.workunit.client.1.smithi189.stdout:3/760: chown d2/d3/d25/de0 1 1 2024-04-20T21:32:20.301 INFO:tasks.workunit.client.1.smithi189.stdout:2/694: dread d1/d11/d2d/f63 [0,4194304] 0 2024-04-20T21:32:20.302 INFO:tasks.workunit.client.0.smithi062.stdout:2/995: stat d7/da/d15/d2f/d38/d6a/d7c/fa5 0 2024-04-20T21:32:20.304 INFO:tasks.workunit.client.0.smithi062.stdout:5/960: link d8/f10d d8/d1e/d2c/d6c/df9/f140 0 2024-04-20T21:32:20.304 INFO:tasks.workunit.client.1.smithi189.stdout:6/779: dread d8/da/dd/d2a/f66 [0,4194304] 0 2024-04-20T21:32:20.306 INFO:tasks.workunit.client.1.smithi189.stdout:2/695: chown d1/d11/d99/db7 1 1 2024-04-20T21:32:20.307 INFO:tasks.workunit.client.1.smithi189.stdout:0/848: fsync d3/df/d30/d45/dd3/ffd 0 2024-04-20T21:32:20.310 INFO:tasks.workunit.client.0.smithi062.stdout:7/987: dwrite d7/d3e/d33/d3d/d4d/fc0 [4194304,4194304] 0 2024-04-20T21:32:20.311 INFO:tasks.workunit.client.1.smithi189.stdout:1/694: write d8/df/f83 [896526,101367] 0 2024-04-20T21:32:20.312 INFO:tasks.workunit.client.1.smithi189.stdout:4/692: write d2/dd/d11/f4b [266684,62099] 0 2024-04-20T21:32:20.317 INFO:tasks.workunit.client.1.smithi189.stdout:4/693: stat d2/d22/da9 0 2024-04-20T21:32:20.317 INFO:tasks.workunit.client.1.smithi189.stdout:7/803: dread d5/d46/f4c [0,4194304] 0 2024-04-20T21:32:20.317 INFO:tasks.workunit.client.0.smithi062.stdout:7/988: creat d7/d3e/d33/d3d/d5e/db8/f15a x:0 0 0 2024-04-20T21:32:20.318 INFO:tasks.workunit.client.1.smithi189.stdout:4/694: fdatasync d2/dd/d37/f59 0 2024-04-20T21:32:20.320 INFO:tasks.workunit.client.1.smithi189.stdout:5/781: rmdir d7/d9/d51/d70/d99/dd4 39 2024-04-20T21:32:20.323 INFO:tasks.workunit.client.1.smithi189.stdout:4/695: write d2/d7f/fc1 [568664,29820] 0 2024-04-20T21:32:20.324 INFO:tasks.workunit.client.1.smithi189.stdout:9/665: creat d9/dc/d17/d95/fd5 x:0 0 0 2024-04-20T21:32:20.325 INFO:tasks.workunit.client.0.smithi062.stdout:6/840: dwrite d0/dc7/f94 [0,4194304] 0 2024-04-20T21:32:20.329 INFO:tasks.workunit.client.0.smithi062.stdout:5/961: rename d8/d1e/d2c/dd1 to d8/d38/d7a/d141 0 2024-04-20T21:32:20.329 INFO:tasks.workunit.client.1.smithi189.stdout:0/849: chown d3/d26/c4d 83325 1 2024-04-20T21:32:20.329 INFO:tasks.workunit.client.1.smithi189.stdout:8/823: mknod da/de/d1e/d69/c125 0 2024-04-20T21:32:20.332 INFO:tasks.workunit.client.1.smithi189.stdout:1/695: mkdir d8/d1e/d54/df0 0 2024-04-20T21:32:20.332 INFO:tasks.workunit.client.1.smithi189.stdout:2/696: dread d1/f7 [8388608,4194304] 0 2024-04-20T21:32:20.336 INFO:tasks.workunit.client.1.smithi189.stdout:3/761: mknod d2/d3/c10e 0 2024-04-20T21:32:20.337 INFO:tasks.workunit.client.1.smithi189.stdout:5/782: mknod d7/d9/d51/d70/d95/cf9 0 2024-04-20T21:32:20.338 INFO:tasks.workunit.client.1.smithi189.stdout:6/780: dread - d8/da/dd/d35/d6d/f91 zero size 2024-04-20T21:32:20.339 INFO:tasks.workunit.client.1.smithi189.stdout:2/697: stat d1/d11/d2d/dab/d4a/d66/dc9 0 2024-04-20T21:32:20.341 INFO:tasks.workunit.client.0.smithi062.stdout:7/989: rename d7/d3e/d33/d3d/d4d/d83/d95/faa to d7/d26/dac/ddb/d132/f15b 0 2024-04-20T21:32:20.342 INFO:tasks.workunit.client.0.smithi062.stdout:6/841: fsync d0/d9/d11/d7b/f103 0 2024-04-20T21:32:20.344 INFO:tasks.workunit.client.0.smithi062.stdout:8/977: dwrite d7/d25/d4f/fae [4194304,4194304] 0 2024-04-20T21:32:20.345 INFO:tasks.workunit.client.0.smithi062.stdout:2/996: dwrite d7/da/d15/d2f/d38/d10f/db9/f127 [0,4194304] 0 2024-04-20T21:32:20.345 INFO:tasks.workunit.client.1.smithi189.stdout:6/781: unlink d8/d2b/f75 0 2024-04-20T21:32:20.345 INFO:tasks.workunit.client.0.smithi062.stdout:5/962: mkdir d8/d142 0 2024-04-20T21:32:20.347 INFO:tasks.workunit.client.1.smithi189.stdout:3/762: chown d2/d3/d2c/d30/cc4 119 1 2024-04-20T21:32:20.347 INFO:tasks.workunit.client.1.smithi189.stdout:1/696: creat d8/df/d3a/d49/d8c/da2/dbe/dc1/de9/ff1 x:0 0 0 2024-04-20T21:32:20.348 INFO:tasks.workunit.client.0.smithi062.stdout:6/842: read - d0/fb5 zero size 2024-04-20T21:32:20.349 INFO:tasks.workunit.client.1.smithi189.stdout:0/850: rename d3/df/d28/d2b/cdc to d3/d8c/c11d 0 2024-04-20T21:32:20.351 INFO:tasks.workunit.client.0.smithi062.stdout:6/843: stat d0/d9/d11/d22/dd5/f10f 0 2024-04-20T21:32:20.351 INFO:tasks.workunit.client.0.smithi062.stdout:2/997: chown d7/da/d15/d1b/d3d/d84/d9b/fda 5109805 1 2024-04-20T21:32:20.351 INFO:tasks.workunit.client.0.smithi062.stdout:5/963: readlink d8/ld 0 2024-04-20T21:32:20.354 INFO:tasks.workunit.client.1.smithi189.stdout:8/824: creat da/d81/d123/d22/f126 x:0 0 0 2024-04-20T21:32:20.354 INFO:tasks.workunit.client.1.smithi189.stdout:0/851: chown d3/da8/fe9 19 1 2024-04-20T21:32:20.355 INFO:tasks.workunit.client.1.smithi189.stdout:7/804: getdents d5/db4/d71/d7a/db0 0 2024-04-20T21:32:20.356 INFO:tasks.workunit.client.0.smithi062.stdout:8/978: mknod d7/d25/dc5/c13a 0 2024-04-20T21:32:20.356 INFO:tasks.workunit.client.0.smithi062.stdout:7/990: creat d7/d3e/d33/d3d/d4d/d14f/f15c x:0 0 0 2024-04-20T21:32:20.356 INFO:tasks.workunit.client.1.smithi189.stdout:6/782: symlink d8/d12/d30/d47/l102 0 2024-04-20T21:32:20.357 INFO:tasks.workunit.client.0.smithi062.stdout:3/897: dwrite d0/d1/d3c/d43/d72/dd3/fbc [0,4194304] 0 2024-04-20T21:32:20.359 INFO:tasks.workunit.client.1.smithi189.stdout:5/783: getdents d7/d9/d31/d36/de0 0 2024-04-20T21:32:20.360 INFO:tasks.workunit.client.0.smithi062.stdout:2/998: chown d7/da/d15/d2f/d38/d6a/f110 1777 1 2024-04-20T21:32:20.361 INFO:tasks.workunit.client.1.smithi189.stdout:2/698: write d1/d11/d2d/d4f/db4/fb1 [4410333,104447] 0 2024-04-20T21:32:20.365 INFO:tasks.workunit.client.0.smithi062.stdout:0/916: sync 2024-04-20T21:32:20.369 INFO:tasks.workunit.client.1.smithi189.stdout:4/696: dwrite d2/dd/d37/dcd/d6a/d7a/de9/fe4 [0,4194304] 0 2024-04-20T21:32:20.369 INFO:tasks.workunit.client.1.smithi189.stdout:9/666: dwrite d9/dc/d17/d95/da0/f6a [4194304,4194304] 0 2024-04-20T21:32:20.372 INFO:tasks.workunit.client.0.smithi062.stdout:6/844: mknod d0/d9/d11/d7b/d99/ddd/de8/c11b 0 2024-04-20T21:32:20.373 INFO:tasks.workunit.client.1.smithi189.stdout:4/697: readlink d2/d22/l86 0 2024-04-20T21:32:20.374 INFO:tasks.workunit.client.1.smithi189.stdout:1/697: rmdir d8/d1e/d43/d44/db2 39 2024-04-20T21:32:20.374 INFO:tasks.workunit.client.0.smithi062.stdout:3/898: truncate d0/f53 5060478 0 2024-04-20T21:32:20.375 INFO:tasks.workunit.client.1.smithi189.stdout:0/852: mkdir d3/df/d30/d37/dbd/d11e 0 2024-04-20T21:32:20.376 INFO:tasks.workunit.client.1.smithi189.stdout:6/783: rmdir d8/da/dd/d35/d6d/d7b 39 2024-04-20T21:32:20.377 INFO:tasks.workunit.client.0.smithi062.stdout:3/899: stat d0/d1c/d63/cfd 0 2024-04-20T21:32:20.378 INFO:tasks.workunit.client.0.smithi062.stdout:6/845: write d0/d1a/d44/fac [2007363,124381] 0 2024-04-20T21:32:20.381 INFO:tasks.workunit.client.1.smithi189.stdout:5/784: chown d7/d9/d51/ddf 14961 1 2024-04-20T21:32:20.381 INFO:tasks.workunit.client.0.smithi062.stdout:2/999: mknod d7/da/d15/d2f/d97/c147 0 2024-04-20T21:32:20.385 INFO:tasks.workunit.client.1.smithi189.stdout:2/699: link d1/d11/d2d/dab/d4a/f8f d1/d17/d78/fce 0 2024-04-20T21:32:20.385 INFO:tasks.workunit.client.1.smithi189.stdout:4/698: write d2/dd/d37/dcd/d6a/d7a/f8d [643002,55332] 0 2024-04-20T21:32:20.386 INFO:tasks.workunit.client.1.smithi189.stdout:3/763: getdents d2/d3/d2c/d30/d10c 0 2024-04-20T21:32:20.387 INFO:tasks.workunit.client.1.smithi189.stdout:0/853: truncate d3/df/d30/d45/fa1 6065 0 2024-04-20T21:32:20.387 INFO:tasks.workunit.client.1.smithi189.stdout:1/698: mknod d8/d1e/d54/d93/cf2 0 2024-04-20T21:32:20.390 INFO:tasks.workunit.client.1.smithi189.stdout:4/699: fsync d2/d7f/fa0 0 2024-04-20T21:32:20.390 INFO:tasks.workunit.client.1.smithi189.stdout:6/784: symlink d8/da/dd/d2a/d3b/l103 0 2024-04-20T21:32:20.392 INFO:tasks.workunit.client.1.smithi189.stdout:0/854: chown d3/df/d4c/f69 17 1 2024-04-20T21:32:20.392 INFO:tasks.workunit.client.0.smithi062.stdout:0/917: dwrite d7/d1c/d15/f2d [0,4194304] 0 2024-04-20T21:32:20.403 INFO:tasks.workunit.client.1.smithi189.stdout:3/764: creat d2/f10f x:0 0 0 2024-04-20T21:32:20.407 INFO:tasks.workunit.client.0.smithi062.stdout:3/900: link d0/d1/f77 d0/d1/d10/d31/d41/de6/f126 0 2024-04-20T21:32:20.408 INFO:tasks.workunit.client.1.smithi189.stdout:2/700: symlink d1/d17/lcf 0 2024-04-20T21:32:20.409 INFO:tasks.workunit.client.0.smithi062.stdout:6/846: creat d0/d9/d11/f11c x:0 0 0 2024-04-20T21:32:20.411 INFO:tasks.workunit.client.0.smithi062.stdout:8/979: dwrite d7/d25/f9d [0,4194304] 0 2024-04-20T21:32:20.412 INFO:tasks.workunit.client.0.smithi062.stdout:0/918: symlink d7/d1c/d15/d17/d40/d79/l11f 0 2024-04-20T21:32:20.418 INFO:tasks.workunit.client.0.smithi062.stdout:6/847: mkdir d0/d1a/d7d/db1/db9/d11d 0 2024-04-20T21:32:20.421 INFO:tasks.workunit.client.1.smithi189.stdout:6/785: getdents d8/d12/d4e/dc4 0 2024-04-20T21:32:20.422 INFO:tasks.workunit.client.1.smithi189.stdout:3/765: mkdir d2/d3/d2c/d31/d90/d110 0 2024-04-20T21:32:20.424 INFO:tasks.workunit.client.0.smithi062.stdout:0/919: truncate d7/d1c/d15/d17/fed 829785 0 2024-04-20T21:32:20.426 INFO:tasks.workunit.client.0.smithi062.stdout:0/920: chown d7/d20/d62/de3/f10a 8547837 1 2024-04-20T21:32:20.427 INFO:tasks.workunit.client.1.smithi189.stdout:5/785: dread d7/d9/fb8 [0,4194304] 0 2024-04-20T21:32:20.428 INFO:tasks.workunit.client.0.smithi062.stdout:0/921: write d7/d20/d62/d6f/f116 [601143,1180] 0 2024-04-20T21:32:20.428 INFO:tasks.workunit.client.1.smithi189.stdout:9/667: dwrite d9/d12/d3b/f4c [0,4194304] 0 2024-04-20T21:32:20.433 INFO:tasks.workunit.client.1.smithi189.stdout:7/805: dwrite d5/f47 [0,4194304] 0 2024-04-20T21:32:20.435 INFO:tasks.workunit.client.0.smithi062.stdout:0/922: rmdir d7/d1c/d15/d17/d37/df8 39 2024-04-20T21:32:20.439 INFO:tasks.workunit.client.1.smithi189.stdout:9/668: rmdir d9/dc/d17/d95/da0/d56 39 2024-04-20T21:32:20.439 INFO:tasks.workunit.client.1.smithi189.stdout:6/786: creat d8/da/da7/f104 x:0 0 0 2024-04-20T21:32:20.440 INFO:tasks.workunit.client.0.smithi062.stdout:5/964: dwrite d8/d38/d7a/d4c/d7d/d9d/dac/f39 [0,4194304] 0 2024-04-20T21:32:20.445 INFO:tasks.workunit.client.1.smithi189.stdout:9/669: chown d9/dc/d17/d95/fd5 14 1 2024-04-20T21:32:20.446 INFO:tasks.workunit.client.1.smithi189.stdout:8/825: dread da/d81/d123/d22/faf [0,4194304] 0 2024-04-20T21:32:20.447 INFO:tasks.workunit.client.1.smithi189.stdout:6/787: fdatasync d8/d12/d30/d47/da0/fcc 0 2024-04-20T21:32:20.450 INFO:tasks.workunit.client.1.smithi189.stdout:3/766: dread d2/d8d/f91 [0,4194304] 0 2024-04-20T21:32:20.453 INFO:tasks.workunit.client.1.smithi189.stdout:7/806: rename d5/db4/f5e to d5/d46/d7d/f102 0 2024-04-20T21:32:20.458 INFO:tasks.workunit.client.1.smithi189.stdout:9/670: symlink d9/d12/d3f/ld6 0 2024-04-20T21:32:20.462 INFO:tasks.workunit.client.1.smithi189.stdout:1/699: dwrite d8/df/d3a/d49/d8c/da2/dbe/dc1/fcd [0,4194304] 0 2024-04-20T21:32:20.464 INFO:tasks.workunit.client.0.smithi062.stdout:6/848: dread d0/d9/d11/d22/dd5/f30 [0,4194304] 0 2024-04-20T21:32:20.469 INFO:tasks.workunit.client.0.smithi062.stdout:5/965: getdents d8/d38/d7a/d4c/dcf 0 2024-04-20T21:32:20.470 INFO:tasks.workunit.client.1.smithi189.stdout:6/788: rename d8/da/dd/d2a/d3b/l43 to d8/d12/d30/d47/l105 0 2024-04-20T21:32:20.472 INFO:tasks.workunit.client.0.smithi062.stdout:0/923: dread d7/d20/d62/d6f/fae [0,4194304] 0 2024-04-20T21:32:20.473 INFO:tasks.workunit.client.0.smithi062.stdout:5/966: dread d8/d1e/fba [0,4194304] 0 2024-04-20T21:32:20.475 INFO:tasks.workunit.client.0.smithi062.stdout:6/849: rename d0/d9/d11/d22/fbe to d0/d9/d11/d22/dd5/dff/f11e 0 2024-04-20T21:32:20.476 INFO:tasks.workunit.client.1.smithi189.stdout:6/789: creat d8/da/da7/f106 x:0 0 0 2024-04-20T21:32:20.479 INFO:tasks.workunit.client.0.smithi062.stdout:0/924: unlink d7/d1c/d15/d17/d1b/d106/cb9 0 2024-04-20T21:32:20.480 INFO:tasks.workunit.client.1.smithi189.stdout:7/807: dwrite d5/d21/d35/f7e [0,4194304] 0 2024-04-20T21:32:20.482 INFO:tasks.workunit.client.1.smithi189.stdout:9/671: rename d9/d12/d3b/db1/d1a/l3e to d9/dc/d17/d36/ld7 0 2024-04-20T21:32:20.483 INFO:tasks.workunit.client.0.smithi062.stdout:6/850: rmdir d0/d9a 39 2024-04-20T21:32:20.487 INFO:tasks.workunit.client.1.smithi189.stdout:9/672: chown d9/dc/d17/d95/da6/fb3 1798381 1 2024-04-20T21:32:20.490 INFO:tasks.workunit.client.0.smithi062.stdout:6/851: unlink d0/d4f/c53 0 2024-04-20T21:32:20.491 INFO:tasks.workunit.client.1.smithi189.stdout:5/786: truncate d7/d35/f43 1181152 0 2024-04-20T21:32:20.493 INFO:tasks.workunit.client.0.smithi062.stdout:6/852: write d0/d9/d11/f11c [464381,63145] 0 2024-04-20T21:32:20.493 INFO:tasks.workunit.client.0.smithi062.stdout:0/925: stat d7/d1c/d15/d17/d40/d79/c55 0 2024-04-20T21:32:20.494 INFO:tasks.workunit.client.1.smithi189.stdout:7/808: creat d5/d1d/d64/db9/ddd/f103 x:0 0 0 2024-04-20T21:32:20.495 INFO:tasks.workunit.client.1.smithi189.stdout:6/790: dread d8/d2b/d49/f72 [0,4194304] 0 2024-04-20T21:32:20.496 INFO:tasks.workunit.client.0.smithi062.stdout:0/926: chown d7/d1e/da9/dbe 7 1 2024-04-20T21:32:20.496 INFO:tasks.workunit.client.1.smithi189.stdout:1/700: truncate d8/d1e/f87 4110488 0 2024-04-20T21:32:20.499 INFO:tasks.workunit.client.0.smithi062.stdout:6/853: fsync d0/d9/d11/d7b/d99/ddd/ffe 0 2024-04-20T21:32:20.502 INFO:tasks.workunit.client.0.smithi062.stdout:6/854: write d0/d1a/d44/db3/fde [646833,65573] 0 2024-04-20T21:32:20.503 INFO:tasks.workunit.client.0.smithi062.stdout:5/967: dwrite d8/d38/d7a/d4c/d7d/d9d/dac/d66/daa/ffa [0,4194304] 0 2024-04-20T21:32:20.506 INFO:tasks.workunit.client.0.smithi062.stdout:6/855: chown d0/d1a/d7d/db1/de0 0 1 2024-04-20T21:32:20.508 INFO:tasks.workunit.client.1.smithi189.stdout:9/673: read d9/dc/d69/f7f [346038,118417] 0 2024-04-20T21:32:20.510 INFO:tasks.workunit.client.1.smithi189.stdout:7/809: creat d5/d21/d4a/dfd/f104 x:0 0 0 2024-04-20T21:32:20.512 INFO:tasks.workunit.client.0.smithi062.stdout:5/968: readlink d8/d1e/d2c/lf4 0 2024-04-20T21:32:20.513 INFO:tasks.workunit.client.1.smithi189.stdout:4/700: dwrite d2/dd/d11/f67 [0,4194304] 0 2024-04-20T21:32:20.513 INFO:tasks.workunit.client.1.smithi189.stdout:6/791: unlink d8/d2b/d5c/ffd 0 2024-04-20T21:32:20.514 INFO:tasks.workunit.client.1.smithi189.stdout:8/826: dwrite da/d81/d123/d3c/d9d/fec [0,4194304] 0 2024-04-20T21:32:20.516 INFO:tasks.workunit.client.0.smithi062.stdout:0/927: write d7/d1c/d15/d17/d37/f61 [520400,31139] 0 2024-04-20T21:32:20.518 INFO:tasks.workunit.client.1.smithi189.stdout:2/701: dread d1/d11/d2d/dab/f49 [0,4194304] 0 2024-04-20T21:32:20.518 INFO:tasks.workunit.client.0.smithi062.stdout:5/969: truncate d8/fb 227544 0 2024-04-20T21:32:20.524 INFO:tasks.workunit.client.1.smithi189.stdout:1/701: rename d8/df/d50/lee to d8/d1e/d43/d44/dd2/db7/lf3 0 2024-04-20T21:32:20.524 INFO:tasks.workunit.client.0.smithi062.stdout:6/856: dwrite d0/dc7/d7c/dc5/de9/f109 [0,4194304] 0 2024-04-20T21:32:20.525 INFO:tasks.workunit.client.1.smithi189.stdout:4/701: unlink d2/dd/fbd 0 2024-04-20T21:32:20.526 INFO:tasks.workunit.client.1.smithi189.stdout:2/702: write d1/d11/f15 [5482323,70756] 0 2024-04-20T21:32:20.528 INFO:tasks.workunit.client.0.smithi062.stdout:6/857: mknod d0/d1a/d44/c11f 0 2024-04-20T21:32:20.528 INFO:tasks.workunit.client.1.smithi189.stdout:8/827: stat da/d81/d123/d3c/cb5 0 2024-04-20T21:32:20.529 INFO:tasks.workunit.client.1.smithi189.stdout:6/792: link d8/d12/fae d8/d12/daf/f107 0 2024-04-20T21:32:20.532 INFO:tasks.workunit.client.1.smithi189.stdout:3/767: dread d2/d1c/f7b [0,4194304] 0 2024-04-20T21:32:20.532 INFO:tasks.workunit.client.1.smithi189.stdout:7/810: creat d5/d21/d4a/d52/d75/f105 x:0 0 0 2024-04-20T21:32:20.535 INFO:tasks.workunit.client.1.smithi189.stdout:6/793: chown d8/da/ca8 5 1 2024-04-20T21:32:20.537 INFO:tasks.workunit.client.1.smithi189.stdout:8/828: unlink da/d81/d123/d20/d27/fbc 0 2024-04-20T21:32:20.537 INFO:tasks.workunit.client.0.smithi062.stdout:5/970: dwrite d8/d38/d7a/d4c/d7d/d9d/dac/d2f/d8e/f97 [0,4194304] 0 2024-04-20T21:32:20.538 INFO:tasks.workunit.client.1.smithi189.stdout:4/702: mkdir d2/d3a/ddc/dfc 0 2024-04-20T21:32:20.539 INFO:tasks.workunit.client.1.smithi189.stdout:6/794: read d8/d12/daf/fb5 [418117,108207] 0 2024-04-20T21:32:20.545 INFO:tasks.workunit.client.1.smithi189.stdout:3/768: rename d2/d3/d2c/d30/d3f/fbf to d2/d8/f111 0 2024-04-20T21:32:20.547 INFO:tasks.workunit.client.1.smithi189.stdout:7/811: mknod d5/d21/d4a/d52/d75/c106 0 2024-04-20T21:32:20.547 INFO:tasks.workunit.client.0.smithi062.stdout:5/971: mkdir d8/d38/d7a/df0/d143 0 2024-04-20T21:32:20.551 INFO:tasks.workunit.client.1.smithi189.stdout:6/795: unlink d8/da/dd/d35/f62 0 2024-04-20T21:32:20.551 INFO:tasks.workunit.client.1.smithi189.stdout:8/829: mkdir da/d81/d123/d20/d27/d6e/d127 0 2024-04-20T21:32:20.552 INFO:tasks.workunit.client.0.smithi062.stdout:6/858: dwrite d0/d1a/f85 [0,4194304] 0 2024-04-20T21:32:20.554 INFO:tasks.workunit.client.1.smithi189.stdout:8/830: read da/d1f/d2a/fdf [9182,80145] 0 2024-04-20T21:32:20.555 INFO:tasks.workunit.client.1.smithi189.stdout:6/796: mkdir d8/d2b/d49/d108 0 2024-04-20T21:32:20.555 INFO:tasks.workunit.client.1.smithi189.stdout:3/769: write d2/d3/d2c/d31/d90/d92/fa2 [1023019,84834] 0 2024-04-20T21:32:20.556 INFO:tasks.workunit.client.0.smithi062.stdout:5/972: readlink d8/d10/l16 0 2024-04-20T21:32:20.561 INFO:tasks.workunit.client.1.smithi189.stdout:8/831: stat da/de/d1e/d69/d70/c105 0 2024-04-20T21:32:20.564 INFO:tasks.workunit.client.1.smithi189.stdout:7/812: creat d5/f107 x:0 0 0 2024-04-20T21:32:20.568 INFO:tasks.workunit.client.1.smithi189.stdout:4/703: dwrite d2/dd/ff5 [0,4194304] 0 2024-04-20T21:32:20.570 INFO:tasks.workunit.client.1.smithi189.stdout:8/832: write da/de/d1e/d21/d87/fd1 [509104,96906] 0 2024-04-20T21:32:20.577 INFO:tasks.workunit.client.0.smithi062.stdout:5/973: dwrite d8/d38/d7a/d4c/d7d/d9d/dac/f6b [4194304,4194304] 0 2024-04-20T21:32:20.579 INFO:tasks.workunit.client.1.smithi189.stdout:5/787: truncate d7/d9/d51/d70/d95/fa1 328734 0 2024-04-20T21:32:20.582 INFO:tasks.workunit.client.1.smithi189.stdout:4/704: chown d2/d22/d57/c98 125766330 1 2024-04-20T21:32:20.585 INFO:tasks.workunit.client.1.smithi189.stdout:1/702: write d8/d21/d2c/f62 [639703,40474] 0 2024-04-20T21:32:20.593 INFO:tasks.workunit.client.1.smithi189.stdout:9/674: dwrite f1 [0,4194304] 0 2024-04-20T21:32:20.597 INFO:tasks.workunit.client.1.smithi189.stdout:5/788: unlink d7/d14/d4e/d5b/df1/ca2 0 2024-04-20T21:32:20.605 INFO:tasks.workunit.client.0.smithi062.stdout:6/859: write d0/d9a/fd1 [473199,101391] 0 2024-04-20T21:32:20.609 INFO:tasks.workunit.client.1.smithi189.stdout:5/789: read d7/d9/d51/d70/d95/ff0 [571836,83816] 0 2024-04-20T21:32:20.613 INFO:tasks.workunit.client.1.smithi189.stdout:2/703: dwrite d1/d11/d2d/dab/d55/f61 [0,4194304] 0 2024-04-20T21:32:20.614 INFO:tasks.workunit.client.0.smithi062.stdout:6/860: symlink d0/d9a/l120 0 2024-04-20T21:32:20.622 INFO:tasks.workunit.client.1.smithi189.stdout:7/813: write d5/db4/d71/fa4 [993573,98467] 0 2024-04-20T21:32:20.624 INFO:tasks.workunit.client.1.smithi189.stdout:5/790: creat d7/d9/d51/d70/d8a/ffa x:0 0 0 2024-04-20T21:32:20.629 INFO:tasks.workunit.client.1.smithi189.stdout:2/704: symlink d1/d11/d2d/dab/d74/ld0 0 2024-04-20T21:32:20.629 INFO:tasks.workunit.client.1.smithi189.stdout:9/675: getdents d9/dc/d17/d7e 0 2024-04-20T21:32:20.637 INFO:tasks.workunit.client.1.smithi189.stdout:6/797: dwrite d8/d2b/f53 [0,4194304] 0 2024-04-20T21:32:20.643 INFO:tasks.workunit.client.1.smithi189.stdout:2/705: dread - d1/d11/d52/d8b/fc8 zero size 2024-04-20T21:32:20.644 INFO:tasks.workunit.client.1.smithi189.stdout:7/814: read d5/d1d/f79 [3193711,10236] 0 2024-04-20T21:32:20.645 INFO:tasks.workunit.client.1.smithi189.stdout:1/703: dwrite d8/d21/d2c/f62 [0,4194304] 0 2024-04-20T21:32:20.649 INFO:tasks.workunit.client.1.smithi189.stdout:0/855: sync 2024-04-20T21:32:20.650 INFO:tasks.workunit.client.1.smithi189.stdout:3/770: dwrite d2/d1c/f2b [4194304,4194304] 0 2024-04-20T21:32:20.655 INFO:tasks.workunit.client.1.smithi189.stdout:9/676: write d9/dc/d17/d95/da0/fc5 [990326,64531] 0 2024-04-20T21:32:20.656 INFO:tasks.workunit.client.1.smithi189.stdout:6/798: creat d8/da/dd/d35/d6d/d7b/f109 x:0 0 0 2024-04-20T21:32:20.657 INFO:tasks.workunit.client.1.smithi189.stdout:3/771: chown d2/d3/d88/f99 14110944 1 2024-04-20T21:32:20.661 INFO:tasks.workunit.client.1.smithi189.stdout:1/704: readlink d8/d1e/d28/laa 0 2024-04-20T21:32:20.665 INFO:tasks.workunit.client.1.smithi189.stdout:8/833: dread da/d81/d123/d20/f7d [0,4194304] 0 2024-04-20T21:32:20.668 INFO:tasks.workunit.client.1.smithi189.stdout:2/706: creat d1/d11/d2d/d4f/d92/fd1 x:0 0 0 2024-04-20T21:32:20.674 INFO:tasks.workunit.client.1.smithi189.stdout:4/705: dwrite d2/f4 [4194304,4194304] 0 2024-04-20T21:32:20.675 INFO:tasks.workunit.client.1.smithi189.stdout:5/791: write d7/d14/d4e/d5b/fa9 [411873,47692] 0 2024-04-20T21:32:20.678 INFO:tasks.workunit.client.1.smithi189.stdout:1/705: unlink d8/d1e/d54/l5d 0 2024-04-20T21:32:20.681 INFO:tasks.workunit.client.1.smithi189.stdout:1/706: read d8/d21/d2c/fba [44521,14905] 0 2024-04-20T21:32:20.681 INFO:tasks.workunit.client.1.smithi189.stdout:2/707: truncate d1/d11/d58/d6b/f77 3510486 0 2024-04-20T21:32:20.684 INFO:tasks.workunit.client.1.smithi189.stdout:4/706: dread d2/d22/d57/f99 [0,4194304] 0 2024-04-20T21:32:20.690 INFO:tasks.workunit.client.1.smithi189.stdout:8/834: link da/d81/d123/d13/d63/d7b/cb6 da/d81/d123/d3a/c128 0 2024-04-20T21:32:20.691 INFO:tasks.workunit.client.1.smithi189.stdout:2/708: write d1/d17/d37/d89/f95 [348518,126016] 0 2024-04-20T21:32:20.692 INFO:tasks.workunit.client.1.smithi189.stdout:1/707: chown d8/f1a 85987178 1 2024-04-20T21:32:20.692 INFO:tasks.workunit.client.0.smithi062.stdout:0/928: dread d7/d20/d62/d6f/f9b [0,4194304] 0 2024-04-20T21:32:20.696 INFO:tasks.workunit.client.1.smithi189.stdout:5/792: dread d7/d9/d28/f9f [0,4194304] 0 2024-04-20T21:32:20.700 INFO:tasks.workunit.client.1.smithi189.stdout:4/707: mkdir d2/dd/d37/dcd/d6a/d7a/def/de8/dfd 0 2024-04-20T21:32:20.702 INFO:tasks.workunit.client.1.smithi189.stdout:0/856: dwrite d3/df/d28/dc8/df1/f68 [0,4194304] 0 2024-04-20T21:32:20.702 INFO:tasks.workunit.client.1.smithi189.stdout:5/793: stat d7/d9/l4b 0 2024-04-20T21:32:20.705 INFO:tasks.workunit.client.1.smithi189.stdout:8/835: mknod da/d81/d123/d13/dd6/d114/c129 0 2024-04-20T21:32:20.710 INFO:tasks.workunit.client.1.smithi189.stdout:2/709: write d1/d11/d52/d8b/f8d [2830499,97953] 0 2024-04-20T21:32:20.713 INFO:tasks.workunit.client.1.smithi189.stdout:3/772: dwrite d2/d3/d2c/f89 [0,4194304] 0 2024-04-20T21:32:20.714 INFO:tasks.workunit.client.0.smithi062.stdout:0/929: dread - d7/d1e/da9/fac zero size 2024-04-20T21:32:20.717 INFO:tasks.workunit.client.1.smithi189.stdout:5/794: write d7/d9/d31/d36/f38 [673247,79024] 0 2024-04-20T21:32:20.718 INFO:tasks.workunit.client.0.smithi062.stdout:5/974: write d8/fb [779064,25898] 0 2024-04-20T21:32:20.722 INFO:tasks.workunit.client.1.smithi189.stdout:1/708: link d8/d1e/d54/fb0 d8/df/d50/ff4 0 2024-04-20T21:32:20.725 INFO:tasks.workunit.client.1.smithi189.stdout:0/857: symlink d3/df/d28/d2b/l11f 0 2024-04-20T21:32:20.726 INFO:tasks.workunit.client.1.smithi189.stdout:3/773: chown d2/d3/d25/de0/cd2 39708154 1 2024-04-20T21:32:20.730 INFO:tasks.workunit.client.1.smithi189.stdout:5/795: symlink d7/d9/d30/d55/d81/lfb 0 2024-04-20T21:32:20.730 INFO:tasks.workunit.client.1.smithi189.stdout:6/799: dwrite d8/d2b/f39 [0,4194304] 0 2024-04-20T21:32:20.731 INFO:tasks.workunit.client.1.smithi189.stdout:3/774: stat d2/d3/d25/f54 0 2024-04-20T21:32:20.732 INFO:tasks.workunit.client.1.smithi189.stdout:4/708: link d2/d20/f24 d2/dd/d11/d14/ffe 0 2024-04-20T21:32:20.733 INFO:tasks.workunit.client.1.smithi189.stdout:3/775: readlink d2/d8/l27 0 2024-04-20T21:32:20.734 INFO:tasks.workunit.client.1.smithi189.stdout:1/709: creat d8/d1e/ff5 x:0 0 0 2024-04-20T21:32:20.736 INFO:tasks.workunit.client.1.smithi189.stdout:5/796: fdatasync d7/d9/d28/f58 0 2024-04-20T21:32:20.736 INFO:tasks.workunit.client.1.smithi189.stdout:9/677: dread d9/d12/d3f/d72/f85 [0,4194304] 0 2024-04-20T21:32:20.738 INFO:tasks.workunit.client.1.smithi189.stdout:1/710: chown d8/df/d50 165087052 1 2024-04-20T21:32:20.738 INFO:tasks.workunit.client.1.smithi189.stdout:7/815: dread d5/d1d/f79 [0,4194304] 0 2024-04-20T21:32:20.744 INFO:tasks.workunit.client.1.smithi189.stdout:3/776: readlink d2/d3/d2c/d31/d74/db7/dd3/le9 0 2024-04-20T21:32:20.744 INFO:tasks.workunit.client.1.smithi189.stdout:8/836: rename da/d81/d123/d13/dd6/d114 to da/de/d84/d12a 0 2024-04-20T21:32:20.745 INFO:tasks.workunit.client.0.smithi062.stdout:6/861: write d0/d9/f16 [1905734,10875] 0 2024-04-20T21:32:20.745 INFO:tasks.workunit.client.1.smithi189.stdout:6/800: symlink d8/d2b/d49/d54/d6e/l10a 0 2024-04-20T21:32:20.746 INFO:tasks.workunit.client.0.smithi062.stdout:0/930: write d7/d1e/d6e/f75 [3495385,120824] 0 2024-04-20T21:32:20.749 INFO:tasks.workunit.client.0.smithi062.stdout:5/975: dread d8/d38/d7a/d4c/d7d/d9d/dac/d2f/df3/f13d [0,4194304] 0 2024-04-20T21:32:20.753 INFO:tasks.workunit.client.1.smithi189.stdout:6/801: write d8/d12/d4e/f6f [1084026,81841] 0 2024-04-20T21:32:20.753 INFO:tasks.workunit.client.1.smithi189.stdout:5/797: creat d7/ffc x:0 0 0 2024-04-20T21:32:20.753 INFO:tasks.workunit.client.1.smithi189.stdout:9/678: fsync d9/dc/d69/f76 0 2024-04-20T21:32:20.755 INFO:tasks.workunit.client.0.smithi062.stdout:6/862: rmdir d0/d9/df5 0 2024-04-20T21:32:20.757 INFO:tasks.workunit.client.1.smithi189.stdout:1/711: link c5 d8/d21/cf6 0 2024-04-20T21:32:20.762 INFO:tasks.workunit.client.0.smithi062.stdout:5/976: rename d8/d38/d7a/d4c/f54 to d8/d118/f144 0 2024-04-20T21:32:20.763 INFO:tasks.workunit.client.1.smithi189.stdout:9/679: chown d9/dc/d17/d36/l39 66 1 2024-04-20T21:32:20.763 INFO:tasks.workunit.client.1.smithi189.stdout:5/798: rename d7/d9/fe7 to d7/d9/d30/d5a/d9d/ffd 0 2024-04-20T21:32:20.764 INFO:tasks.workunit.client.0.smithi062.stdout:5/977: dread - d8/d38/d7a/d141/f125 zero size 2024-04-20T21:32:20.764 INFO:tasks.workunit.client.0.smithi062.stdout:6/863: truncate d0/d9/d11/d22/dd5/d67/fa1 825501 0 2024-04-20T21:32:20.766 INFO:tasks.workunit.client.1.smithi189.stdout:3/777: dread d2/d3/d88/f99 [0,4194304] 0 2024-04-20T21:32:20.768 INFO:tasks.workunit.client.1.smithi189.stdout:6/802: link d8/d2b/d49/d54/d6e/c8f d8/d12/d4e/dc4/c10b 0 2024-04-20T21:32:20.769 INFO:tasks.workunit.client.0.smithi062.stdout:6/864: creat d0/d1a/d7d/f121 x:0 0 0 2024-04-20T21:32:20.771 INFO:tasks.workunit.client.1.smithi189.stdout:0/858: truncate d3/f11a 350878 0 2024-04-20T21:32:20.772 INFO:tasks.workunit.client.0.smithi062.stdout:5/978: mknod d8/d38/d7a/d4c/d7d/d9d/dac/d66/dae/d88/d139/c145 0 2024-04-20T21:32:20.775 INFO:tasks.workunit.client.0.smithi062.stdout:6/865: dread d0/d9/f71 [0,4194304] 0 2024-04-20T21:32:20.776 INFO:tasks.workunit.client.1.smithi189.stdout:7/816: write d5/d21/d25/f27 [4389552,34200] 0 2024-04-20T21:32:20.776 INFO:tasks.workunit.client.1.smithi189.stdout:9/680: creat d9/dc/d17/d95/da0/d56/fd8 x:0 0 0 2024-04-20T21:32:20.777 INFO:tasks.workunit.client.1.smithi189.stdout:2/710: dread d1/d11/d2d/d4f/db4/fb1 [0,4194304] 0 2024-04-20T21:32:20.778 INFO:tasks.workunit.client.0.smithi062.stdout:6/866: unlink d0/f91 0 2024-04-20T21:32:20.780 INFO:tasks.workunit.client.1.smithi189.stdout:0/859: symlink d3/db9/dbe/dd7/l120 0 2024-04-20T21:32:20.781 INFO:tasks.workunit.client.1.smithi189.stdout:8/837: dwrite da/d81/d123/d20/d2c/d9a/fcc [0,4194304] 0 2024-04-20T21:32:20.785 INFO:tasks.workunit.client.1.smithi189.stdout:5/799: rename d7/d9/d31/d36/cdd to d7/d9/d28/cfe 0 2024-04-20T21:32:20.788 INFO:tasks.workunit.client.1.smithi189.stdout:0/860: mkdir d3/df/d28/dc8/d121 0 2024-04-20T21:32:20.788 INFO:tasks.workunit.client.1.smithi189.stdout:4/709: dread d2/d22/d87/fa2 [0,4194304] 0 2024-04-20T21:32:20.790 INFO:tasks.workunit.client.1.smithi189.stdout:2/711: mkdir d1/d11/d2d/d4f/db4/daa/dd2 0 2024-04-20T21:32:20.791 INFO:tasks.workunit.client.0.smithi062.stdout:0/931: dread d7/f47 [0,4194304] 0 2024-04-20T21:32:20.792 INFO:tasks.workunit.client.1.smithi189.stdout:1/712: truncate d8/d1e/d28/d4f/f80 1570710 0 2024-04-20T21:32:20.795 INFO:tasks.workunit.client.1.smithi189.stdout:1/713: chown d8/c35 28012 1 2024-04-20T21:32:20.795 INFO:tasks.workunit.client.1.smithi189.stdout:5/800: fdatasync d7/d14/d18/f3e 0 2024-04-20T21:32:20.796 INFO:tasks.workunit.client.0.smithi062.stdout:0/932: fsync d7/f4c 0 2024-04-20T21:32:20.796 INFO:tasks.workunit.client.1.smithi189.stdout:8/838: symlink da/d81/d123/d13/d63/l12b 0 2024-04-20T21:32:20.798 INFO:tasks.workunit.client.0.smithi062.stdout:0/933: chown d7/d1e/db6/l7a 325318035 1 2024-04-20T21:32:20.801 INFO:tasks.workunit.client.1.smithi189.stdout:6/803: dread d8/da/dd/d35/f59 [0,4194304] 0 2024-04-20T21:32:20.804 INFO:tasks.workunit.client.1.smithi189.stdout:1/714: fsync d8/df/d50/f99 0 2024-04-20T21:32:20.805 INFO:tasks.workunit.client.1.smithi189.stdout:3/778: dwrite d2/f10f [0,4194304] 0 2024-04-20T21:32:20.807 INFO:tasks.workunit.client.1.smithi189.stdout:0/861: rename d3/df/d4c/dd4/f94 to d3/df/d5d/d64/f122 0 2024-04-20T21:32:20.808 INFO:tasks.workunit.client.0.smithi062.stdout:5/979: dwrite d8/d38/d7a/d4c/d7d/d9d/dac/d66/dae/f9c [0,4194304] 0 2024-04-20T21:32:20.812 INFO:tasks.workunit.client.1.smithi189.stdout:0/862: chown d3/df/d4c/dd4 15009366 1 2024-04-20T21:32:20.815 INFO:tasks.workunit.client.0.smithi062.stdout:0/934: dwrite d7/d1c/d15/f22 [4194304,4194304] 0 2024-04-20T21:32:20.820 INFO:tasks.workunit.client.1.smithi189.stdout:7/817: write d5/d21/d4a/d52/d75/fab [1091491,11531] 0 2024-04-20T21:32:20.820 INFO:tasks.workunit.client.1.smithi189.stdout:6/804: rename d8/c4d to d8/d2b/d49/d54/d67/c10c 0 2024-04-20T21:32:20.820 INFO:tasks.workunit.client.0.smithi062.stdout:5/980: rmdir d8/d38/d7a/d4c/d7d 39 2024-04-20T21:32:20.821 INFO:tasks.workunit.client.0.smithi062.stdout:0/935: creat d7/d1e/da9/f120 x:0 0 0 2024-04-20T21:32:20.823 INFO:tasks.workunit.client.1.smithi189.stdout:8/839: dread f9 [0,4194304] 0 2024-04-20T21:32:20.823 INFO:tasks.workunit.client.1.smithi189.stdout:3/779: mkdir d2/d3/d2c/d30/d10c/d112 0 2024-04-20T21:32:20.824 INFO:tasks.workunit.client.1.smithi189.stdout:9/681: dwrite d9/d12/d3b/f4c [0,4194304] 0 2024-04-20T21:32:20.824 INFO:tasks.workunit.client.1.smithi189.stdout:4/710: truncate d2/dd/d37/dcd/d6a/d7a/de9/fc0 534177 0 2024-04-20T21:32:20.831 INFO:tasks.workunit.client.0.smithi062.stdout:5/981: creat d8/d38/d7a/d4c/d7d/d9d/dac/d2f/df3/f146 x:0 0 0 2024-04-20T21:32:20.832 INFO:tasks.workunit.client.1.smithi189.stdout:3/780: readlink d2/d3/d88/l9f 0 2024-04-20T21:32:20.834 INFO:tasks.workunit.client.1.smithi189.stdout:9/682: fdatasync d9/d12/d3b/f46 0 2024-04-20T21:32:20.837 INFO:tasks.workunit.client.1.smithi189.stdout:1/715: dwrite f6 [0,4194304] 0 2024-04-20T21:32:20.837 INFO:tasks.workunit.client.1.smithi189.stdout:8/840: fdatasync da/d81/d123/d3c/d68/d8b/fcf 0 2024-04-20T21:32:20.837 INFO:tasks.workunit.client.1.smithi189.stdout:4/711: write d2/d22/d57/fe3 [388762,49310] 0 2024-04-20T21:32:20.849 INFO:tasks.workunit.client.1.smithi189.stdout:8/841: readlink da/d81/d123/d13/dd6/dba/l120 0 2024-04-20T21:32:20.850 INFO:tasks.workunit.client.1.smithi189.stdout:3/781: rmdir d2/d3/d2c/d31/d90 39 2024-04-20T21:32:20.851 INFO:tasks.workunit.client.1.smithi189.stdout:5/801: dwrite d7/d14/d4e/d5b/df1/dab/dcb/fe4 [0,4194304] 0 2024-04-20T21:32:20.853 INFO:tasks.workunit.client.1.smithi189.stdout:8/842: chown da/de/d1e/d69 3822 1 2024-04-20T21:32:20.857 INFO:tasks.workunit.client.1.smithi189.stdout:0/863: dwrite d3/da8/fe9 [0,4194304] 0 2024-04-20T21:32:20.857 INFO:tasks.workunit.client.1.smithi189.stdout:8/843: dread - da/d1f/fd8 zero size 2024-04-20T21:32:20.861 INFO:tasks.workunit.client.0.smithi062.stdout:7/991: sync 2024-04-20T21:32:20.861 INFO:tasks.workunit.client.0.smithi062.stdout:8/980: sync 2024-04-20T21:32:20.861 INFO:tasks.workunit.client.0.smithi062.stdout:3/901: sync 2024-04-20T21:32:20.870 INFO:tasks.workunit.client.0.smithi062.stdout:3/902: chown d0/d1/cd1 911 1 2024-04-20T21:32:20.870 INFO:tasks.workunit.client.0.smithi062.stdout:7/992: fdatasync d7/d3e/d33/f76 0 2024-04-20T21:32:20.874 INFO:tasks.workunit.client.1.smithi189.stdout:2/712: dwrite d1/d17/f2b [0,4194304] 0 2024-04-20T21:32:20.875 INFO:tasks.workunit.client.1.smithi189.stdout:7/818: dwrite d5/d1d/d64/db9/ddd/f103 [0,4194304] 0 2024-04-20T21:32:20.880 INFO:tasks.workunit.client.1.smithi189.stdout:2/713: read - d1/d17/d78/db9/fc7 zero size 2024-04-20T21:32:20.882 INFO:tasks.workunit.client.0.smithi062.stdout:3/903: readlink d0/d1c/d63/l3b 0 2024-04-20T21:32:20.891 INFO:tasks.workunit.client.1.smithi189.stdout:5/802: mkdir d7/d14/d4e/d5b/df1/dff 0 2024-04-20T21:32:20.894 INFO:tasks.workunit.client.0.smithi062.stdout:6/867: dread d0/d9/d11/d22/dd5/d67/ffb [0,4194304] 0 2024-04-20T21:32:20.898 INFO:tasks.workunit.client.0.smithi062.stdout:5/982: write d8/d10/d3e/f94 [333117,11333] 0 2024-04-20T21:32:20.902 INFO:tasks.workunit.client.0.smithi062.stdout:7/993: dwrite d7/d3e/d33/d3d/d5e/d72/fb2 [0,4194304] 0 2024-04-20T21:32:20.905 INFO:tasks.workunit.client.1.smithi189.stdout:1/716: getdents d8/d1e/d54 0 2024-04-20T21:32:20.906 INFO:tasks.workunit.client.1.smithi189.stdout:3/782: unlink d2/d3/f1b 0 2024-04-20T21:32:20.909 INFO:tasks.workunit.client.1.smithi189.stdout:9/683: write d9/dc/d17/d95/da0/d56/f8a [3154109,18131] 0 2024-04-20T21:32:20.913 INFO:tasks.workunit.client.0.smithi062.stdout:7/994: mkdir d7/d3e/d15d 0 2024-04-20T21:32:20.915 INFO:tasks.workunit.client.1.smithi189.stdout:0/864: rename d3/db9/dbe/lfa to d3/df/d28/d2b/db1/d111/l123 0 2024-04-20T21:32:20.916 INFO:tasks.workunit.client.0.smithi062.stdout:0/936: dwrite d7/d1e/fdc [0,4194304] 0 2024-04-20T21:32:20.918 INFO:tasks.workunit.client.1.smithi189.stdout:8/844: dwrite da/d81/d123/d13/f5f [0,4194304] 0 2024-04-20T21:32:20.922 INFO:tasks.workunit.client.1.smithi189.stdout:6/805: write d8/d10/d78/df7/fd5 [1093366,43238] 0 2024-04-20T21:32:20.926 INFO:tasks.workunit.client.0.smithi062.stdout:5/983: dread d8/d38/d7a/d4c/fc7 [4194304,4194304] 0 2024-04-20T21:32:20.935 INFO:tasks.workunit.client.1.smithi189.stdout:7/819: dwrite d5/d21/d4a/ffb [0,4194304] 0 2024-04-20T21:32:20.935 INFO:tasks.workunit.client.1.smithi189.stdout:5/803: dwrite d7/d9/d31/d36/fde [0,4194304] 0 2024-04-20T21:32:20.935 INFO:tasks.workunit.client.0.smithi062.stdout:8/981: dread d7/d30/f137 [0,4194304] 0 2024-04-20T21:32:20.942 INFO:tasks.workunit.client.1.smithi189.stdout:4/712: dwrite d2/f5e [0,4194304] 0 2024-04-20T21:32:20.942 INFO:tasks.workunit.client.1.smithi189.stdout:1/717: dwrite f6 [0,4194304] 0 2024-04-20T21:32:20.950 INFO:tasks.workunit.client.1.smithi189.stdout:1/718: read d8/d1e/d54/fae [179963,33949] 0 2024-04-20T21:32:20.951 INFO:tasks.workunit.client.0.smithi062.stdout:0/937: write d7/d20/d62/d6f/f73 [2764203,70814] 0 2024-04-20T21:32:20.951 INFO:tasks.workunit.client.1.smithi189.stdout:1/719: stat d8 0 2024-04-20T21:32:20.955 INFO:tasks.workunit.client.1.smithi189.stdout:9/684: dread d9/d12/d3b/db1/d1a/f27 [0,4194304] 0 2024-04-20T21:32:20.955 INFO:tasks.workunit.client.0.smithi062.stdout:0/938: fsync d7/d1c/ff 0 2024-04-20T21:32:20.957 INFO:tasks.workunit.client.0.smithi062.stdout:8/982: fsync d7/d25/d3b/d58/d8b/d24/dbd/f109 0 2024-04-20T21:32:20.957 INFO:tasks.workunit.client.0.smithi062.stdout:6/868: dwrite d0/d9/fcb [0,4194304] 0 2024-04-20T21:32:20.962 INFO:tasks.workunit.client.1.smithi189.stdout:8/845: dwrite da/de/d1e/d21/f67 [0,4194304] 0 2024-04-20T21:32:20.964 INFO:tasks.workunit.client.1.smithi189.stdout:8/846: chown da/d1f/d2a/l111 0 1 2024-04-20T21:32:20.970 INFO:tasks.workunit.client.0.smithi062.stdout:0/939: creat d7/d1c/d15/d17/d1b/d106/df1/f121 x:0 0 0 2024-04-20T21:32:20.973 INFO:tasks.workunit.client.0.smithi062.stdout:7/995: dwrite d7/d26/dac/fd4 [0,4194304] 0 2024-04-20T21:32:20.974 INFO:tasks.workunit.client.1.smithi189.stdout:3/783: rmdir d2/d3/d25/de0/db0/dc8/dc9 39 2024-04-20T21:32:20.974 INFO:tasks.workunit.client.0.smithi062.stdout:8/983: creat d7/d9/f13b x:0 0 0 2024-04-20T21:32:20.975 INFO:tasks.workunit.client.0.smithi062.stdout:3/904: dread d0/d1b/f2b [0,4194304] 0 2024-04-20T21:32:20.979 INFO:tasks.workunit.client.1.smithi189.stdout:0/865: mkdir d3/df/d30/d37/dbd/de1/d124 0 2024-04-20T21:32:20.987 INFO:tasks.workunit.client.0.smithi062.stdout:7/996: dread d7/d26/da8/d153/f8c [0,4194304] 0 2024-04-20T21:32:20.990 INFO:tasks.workunit.client.0.smithi062.stdout:0/940: mkdir d7/d1c/d15/d17/d37/d109/d11b/d122 0 2024-04-20T21:32:20.991 INFO:tasks.workunit.client.1.smithi189.stdout:7/820: rename d5/f43 to d5/d46/d7d/f108 0 2024-04-20T21:32:20.994 INFO:tasks.workunit.client.0.smithi062.stdout:0/941: stat d7/d20/d62/d6f/lc3 0 2024-04-20T21:32:20.994 INFO:tasks.workunit.client.0.smithi062.stdout:6/869: truncate d0/d9/d11/d22/dd5/dff/f11e 1034945 0 2024-04-20T21:32:20.996 INFO:tasks.workunit.client.1.smithi189.stdout:2/714: dread d1/d11/d58/f6e [0,4194304] 0 2024-04-20T21:32:21.001 INFO:tasks.workunit.client.0.smithi062.stdout:8/984: symlink d7/d9/d29/d11c/d12d/l13c 0 2024-04-20T21:32:21.004 INFO:tasks.workunit.client.0.smithi062.stdout:8/985: stat d7/d25/d3b/d58/db9/ffb 0 2024-04-20T21:32:21.004 INFO:tasks.workunit.client.0.smithi062.stdout:3/905: rename d0/d1c/d63/d42/d11e/fde to d0/f127 0 2024-04-20T21:32:21.009 INFO:tasks.workunit.client.0.smithi062.stdout:7/997: link d7/d3e/d33/d3d/d5e/f102 d7/f15e 0 2024-04-20T21:32:21.010 INFO:tasks.workunit.client.0.smithi062.stdout:0/942: read f1 [2707725,26554] 0 2024-04-20T21:32:21.014 INFO:tasks.workunit.client.0.smithi062.stdout:5/984: dread d8/d10/d3e/d8a/fa0 [0,4194304] 0 2024-04-20T21:32:21.018 INFO:tasks.workunit.client.0.smithi062.stdout:0/943: fdatasync d7/d1c/d15/d80/f117 0 2024-04-20T21:32:21.020 INFO:tasks.workunit.client.1.smithi189.stdout:4/713: creat d2/d20/fff x:0 0 0 2024-04-20T21:32:21.036 INFO:tasks.workunit.client.0.smithi062.stdout:8/986: dwrite d7/d25/d3b/d58/f63 [0,4194304] 0 2024-04-20T21:32:21.041 INFO:tasks.workunit.client.0.smithi062.stdout:6/870: dwrite d0/d9/d11/d22/dd5/f30 [0,4194304] 0 2024-04-20T21:32:21.041 INFO:tasks.workunit.client.0.smithi062.stdout:8/987: rmdir d7/d9/d29/d11c 39 2024-04-20T21:32:21.043 INFO:tasks.workunit.client.0.smithi062.stdout:7/998: dwrite d7/fa0 [4194304,4194304] 0 2024-04-20T21:32:21.053 INFO:tasks.workunit.client.0.smithi062.stdout:8/988: mknod d7/d25/d3b/d12c/c13d 0 2024-04-20T21:32:21.053 INFO:tasks.workunit.client.0.smithi062.stdout:6/871: getdents d0/d1a/d44/da7 0 2024-04-20T21:32:21.064 INFO:tasks.workunit.client.1.smithi189.stdout:9/685: unlink d9/d12/d3b/db1/d1a/l51 0 2024-04-20T21:32:21.064 INFO:tasks.workunit.client.1.smithi189.stdout:9/686: chown d9/d12/d3b/db1/f8f 3368 1 2024-04-20T21:32:21.067 INFO:tasks.workunit.client.0.smithi062.stdout:6/872: getdents d0/d52/da0 0 2024-04-20T21:32:21.067 INFO:tasks.workunit.client.1.smithi189.stdout:3/784: mknod d2/d1c/d5c/c113 0 2024-04-20T21:32:21.071 INFO:tasks.workunit.client.0.smithi062.stdout:5/985: dread d8/d1e/f5d [0,4194304] 0 2024-04-20T21:32:21.072 INFO:tasks.workunit.client.1.smithi189.stdout:8/847: write da/fb [8825853,34239] 0 2024-04-20T21:32:21.072 INFO:tasks.workunit.client.0.smithi062.stdout:6/873: dread d0/d9/d11/fda [0,4194304] 0 2024-04-20T21:32:21.073 INFO:tasks.workunit.client.1.smithi189.stdout:0/866: symlink d3/df/d30/d45/d90/l125 0 2024-04-20T21:32:21.074 INFO:tasks.workunit.client.0.smithi062.stdout:7/999: dwrite d7/d3e/d33/d3d/d5e/db8/f15a [0,4194304] 0 2024-04-20T21:32:21.076 INFO:tasks.workunit.client.1.smithi189.stdout:7/821: fdatasync d5/f57 0 2024-04-20T21:32:21.078 INFO:tasks.workunit.client.1.smithi189.stdout:4/714: unlink d2/d3a/f3c 0 2024-04-20T21:32:21.080 INFO:tasks.workunit.client.1.smithi189.stdout:0/867: write d3/df/d5d/f11c [3700228,57491] 0 2024-04-20T21:32:21.081 INFO:tasks.workunit.client.1.smithi189.stdout:4/715: readlink d2/dd/d37/dcd/d6a/d7a/de9/lec 0 2024-04-20T21:32:21.083 INFO:tasks.workunit.client.1.smithi189.stdout:4/716: readlink d2/dd/d11/l47 0 2024-04-20T21:32:21.087 INFO:tasks.workunit.client.0.smithi062.stdout:5/986: mkdir d8/d38/d7a/d4c/d7d/d9d/dac/d2f/d8e/dd6/dfe/d147 0 2024-04-20T21:32:21.093 INFO:tasks.workunit.client.0.smithi062.stdout:6/874: rename d0/d9/d11/d22/dd5/d67/d75/dd2/fdb to d0/d1a/d7d/f122 0 2024-04-20T21:32:21.095 INFO:tasks.workunit.client.1.smithi189.stdout:9/687: read d9/dc/f4d [3659601,130813] 0 2024-04-20T21:32:21.095 INFO:tasks.workunit.client.0.smithi062.stdout:8/989: dwrite d7/d25/d76/f9c [0,4194304] 0 2024-04-20T21:32:21.098 INFO:tasks.workunit.client.1.smithi189.stdout:9/688: readlink d9/dc/d17/d95/da0/d52/d82/d94/lcb 0 2024-04-20T21:32:21.099 INFO:tasks.workunit.client.0.smithi062.stdout:5/987: mknod d8/d10/c148 0 2024-04-20T21:32:21.102 INFO:tasks.workunit.client.0.smithi062.stdout:6/875: rename d0/d52/cbb to d0/d9/d11/d7b/d99/c123 0 2024-04-20T21:32:21.103 INFO:tasks.workunit.client.1.smithi189.stdout:9/689: mkdir d9/d12/d62/dd9 0 2024-04-20T21:32:21.104 INFO:tasks.workunit.client.0.smithi062.stdout:6/876: readlink d0/d9a/lc0 0 2024-04-20T21:32:21.107 INFO:tasks.workunit.client.1.smithi189.stdout:9/690: fdatasync d9/f44 0 2024-04-20T21:32:21.112 INFO:tasks.workunit.client.1.smithi189.stdout:9/691: symlink d9/d12/d62/lda 0 2024-04-20T21:32:21.112 INFO:tasks.workunit.client.0.smithi062.stdout:5/988: dread d8/d38/d7a/d4c/d7d/d9d/dac/f6b [4194304,4194304] 0 2024-04-20T21:32:21.113 INFO:tasks.workunit.client.0.smithi062.stdout:6/877: unlink d0/d9/d11/d22/c66 0 2024-04-20T21:32:21.116 INFO:tasks.workunit.client.1.smithi189.stdout:9/692: creat d9/dc/d69/fdb x:0 0 0 2024-04-20T21:32:21.117 INFO:tasks.workunit.client.1.smithi189.stdout:7/822: dread d5/d21/d4a/d52/f5c [0,4194304] 0 2024-04-20T21:32:21.117 INFO:tasks.workunit.client.0.smithi062.stdout:6/878: symlink d0/d1a/d7d/db1/db9/l124 0 2024-04-20T21:32:21.119 INFO:tasks.workunit.client.1.smithi189.stdout:2/715: mknod d1/d11/d52/cd3 0 2024-04-20T21:32:21.125 INFO:tasks.workunit.client.1.smithi189.stdout:2/716: mknod d1/d11/d2d/dab/cd4 0 2024-04-20T21:32:21.134 INFO:tasks.workunit.client.1.smithi189.stdout:4/717: creat d2/dd/d6b/f100 x:0 0 0 2024-04-20T21:32:21.134 INFO:tasks.workunit.client.0.smithi062.stdout:8/990: write d7/d25/d4f/fdb [326182,110908] 0 2024-04-20T21:32:21.136 INFO:tasks.workunit.client.0.smithi062.stdout:5/989: dread d8/d10/f17 [0,4194304] 0 2024-04-20T21:32:21.139 INFO:tasks.workunit.client.0.smithi062.stdout:5/990: readlink d8/d1e/d2c/d6c/la6 0 2024-04-20T21:32:21.143 INFO:tasks.workunit.client.1.smithi189.stdout:5/804: dwrite d7/d14/d4e/d5b/fc2 [0,4194304] 0 2024-04-20T21:32:21.144 INFO:tasks.workunit.client.0.smithi062.stdout:8/991: mkdir d7/de1/de5/d13e 0 2024-04-20T21:32:21.149 INFO:tasks.workunit.client.0.smithi062.stdout:6/879: dwrite d0/d52/da0/fcc [4194304,4194304] 0 2024-04-20T21:32:21.151 INFO:tasks.workunit.client.0.smithi062.stdout:6/880: dread - d0/dc7/d7c/dc5/ffc zero size 2024-04-20T21:32:21.154 INFO:tasks.workunit.client.0.smithi062.stdout:6/881: write d0/d9/d11/ff7 [211417,36776] 0 2024-04-20T21:32:21.156 INFO:tasks.workunit.client.0.smithi062.stdout:5/991: dwrite d8/d10/d3e/f94 [0,4194304] 0 2024-04-20T21:32:21.157 INFO:tasks.workunit.client.0.smithi062.stdout:6/882: chown d0/d1a/d44/db3/fde 1 1 2024-04-20T21:32:21.160 INFO:tasks.workunit.client.0.smithi062.stdout:6/883: unlink d0/d4f/le3 0 2024-04-20T21:32:21.162 INFO:tasks.workunit.client.0.smithi062.stdout:5/992: truncate d8/d38/d7a/d4c/d7d/d9d/dac/f92 4643939 0 2024-04-20T21:32:21.162 INFO:tasks.workunit.client.1.smithi189.stdout:1/720: rename d8/df/d3a/d49/d8c/da2/dbe/dc1/de9/ff1 to d8/d1e/d54/ff7 0 2024-04-20T21:32:21.163 INFO:tasks.workunit.client.1.smithi189.stdout:1/721: dread - d8/d1e/d43/d58/fe8 zero size 2024-04-20T21:32:21.166 INFO:tasks.workunit.client.1.smithi189.stdout:6/806: rename d8/d2b/d49/d54/d6e/c8f to d8/d2b/d49/d54/d67/c10d 0 2024-04-20T21:32:21.167 INFO:tasks.workunit.client.1.smithi189.stdout:5/805: dwrite d7/d9/d31/d36/f38 [0,4194304] 0 2024-04-20T21:32:21.167 INFO:tasks.workunit.client.1.smithi189.stdout:1/722: write d8/d21/d2c/f62 [1009684,106545] 0 2024-04-20T21:32:21.168 INFO:tasks.workunit.client.0.smithi062.stdout:5/993: dread d8/d38/f3d [0,4194304] 0 2024-04-20T21:32:21.171 INFO:tasks.workunit.client.1.smithi189.stdout:8/848: rename da/d81/d123/d20/d2c/c53 to da/d81/d123/d3c/d68/df5/c12c 0 2024-04-20T21:32:21.172 INFO:tasks.workunit.client.0.smithi062.stdout:6/884: rename d0/d9/d11/d22/dd5/d67/d75/lab to d0/d9/d11/d22/d107/l125 0 2024-04-20T21:32:21.174 INFO:tasks.workunit.client.0.smithi062.stdout:6/885: chown d0/d1a/l110 10450400 1 2024-04-20T21:32:21.175 INFO:tasks.workunit.client.1.smithi189.stdout:5/806: fdatasync d7/d9/d30/d5a/d9d/ffd 0 2024-04-20T21:32:21.176 INFO:tasks.workunit.client.0.smithi062.stdout:8/992: write d7/fa5 [4130343,110016] 0 2024-04-20T21:32:21.177 INFO:tasks.workunit.client.1.smithi189.stdout:1/723: readlink d8/d21/d2c/l4a 0 2024-04-20T21:32:21.179 INFO:tasks.workunit.client.1.smithi189.stdout:1/724: chown d8/d1e/d28/dd4/ddb 107746531 1 2024-04-20T21:32:21.180 INFO:tasks.workunit.client.0.smithi062.stdout:8/993: mkdir d7/d25/d3b/d58/de8/d13f 0 2024-04-20T21:32:21.182 INFO:tasks.workunit.client.1.smithi189.stdout:2/717: rename d1/d11/d52/d8b/fc8 to d1/d11/d58/fd5 0 2024-04-20T21:32:21.182 INFO:tasks.workunit.client.0.smithi062.stdout:8/994: stat d7/d25/l33 0 2024-04-20T21:32:21.187 INFO:tasks.workunit.client.1.smithi189.stdout:9/693: dread d9/dc/d17/d95/da6/fb3 [0,4194304] 0 2024-04-20T21:32:21.189 INFO:tasks.workunit.client.1.smithi189.stdout:1/725: read d8/f9 [2066698,43015] 0 2024-04-20T21:32:21.189 INFO:tasks.workunit.client.1.smithi189.stdout:4/718: rename d2/dd/d11/f31 to d2/dd/d37/dcd/d6a/d7a/def/f101 0 2024-04-20T21:32:21.189 INFO:tasks.workunit.client.1.smithi189.stdout:9/694: dread - d9/dc/d17/d7e/f96 zero size 2024-04-20T21:32:21.190 INFO:tasks.workunit.client.0.smithi062.stdout:5/994: rename d8/d1e/d2c/d6c/df9/f140 to d8/d38/d7a/d4c/d7d/d9d/dac/d66/f149 0 2024-04-20T21:32:21.191 INFO:tasks.workunit.client.0.smithi062.stdout:8/995: dread - d7/d25/d3b/d58/d8b/ff1 zero size 2024-04-20T21:32:21.192 INFO:tasks.workunit.client.0.smithi062.stdout:8/996: readlink d7/d25/l37 0 2024-04-20T21:32:21.196 INFO:tasks.workunit.client.1.smithi189.stdout:9/695: dread d9/dc/d69/f76 [0,4194304] 0 2024-04-20T21:32:21.197 INFO:tasks.workunit.client.1.smithi189.stdout:1/726: symlink d8/df/d50/de6/lf8 0 2024-04-20T21:32:21.197 INFO:tasks.workunit.client.0.smithi062.stdout:8/997: creat d7/d25/d3b/d58/d8b/d24/dbd/d127/f140 x:0 0 0 2024-04-20T21:32:21.198 INFO:tasks.workunit.client.1.smithi189.stdout:4/719: getdents d2/d7f 0 2024-04-20T21:32:21.205 INFO:tasks.workunit.client.1.smithi189.stdout:9/696: getdents d9/dc/d17/d95/da0/d55/dd0 0 2024-04-20T21:32:21.206 INFO:tasks.workunit.client.1.smithi189.stdout:2/718: dwrite d1/d17/d37/f44 [0,4194304] 0 2024-04-20T21:32:21.213 INFO:tasks.workunit.client.0.smithi062.stdout:6/886: write d0/d52/fdc [830143,127165] 0 2024-04-20T21:32:21.213 INFO:tasks.workunit.client.1.smithi189.stdout:7/823: creat d5/d21/f109 x:0 0 0 2024-04-20T21:32:21.214 INFO:tasks.workunit.client.0.smithi062.stdout:5/995: dread d8/d38/d7a/d4c/d7d/d9d/dac/d66/fcb [0,4194304] 0 2024-04-20T21:32:21.215 INFO:tasks.workunit.client.1.smithi189.stdout:2/719: fsync d1/d11/d2d/d4f/fbc 0 2024-04-20T21:32:21.217 INFO:tasks.workunit.client.1.smithi189.stdout:9/697: mknod d9/d12/d3b/db1/d1a/cdc 0 2024-04-20T21:32:21.220 INFO:tasks.workunit.client.1.smithi189.stdout:5/807: rename d7/d14 to d7/d9/d31/d100 0 2024-04-20T21:32:21.220 INFO:tasks.workunit.client.1.smithi189.stdout:4/720: creat d2/d22/d87/f102 x:0 0 0 2024-04-20T21:32:21.222 INFO:tasks.workunit.client.0.smithi062.stdout:6/887: unlink d0/d9/d11/d7b/fc9 0 2024-04-20T21:32:21.224 INFO:tasks.workunit.client.1.smithi189.stdout:5/808: chown d7/l2c 985063 1 2024-04-20T21:32:21.225 INFO:tasks.workunit.client.1.smithi189.stdout:8/849: dread da/d1f/d2a/f4b [0,4194304] 0 2024-04-20T21:32:21.226 INFO:tasks.workunit.client.1.smithi189.stdout:7/824: rename d5/d21/d4a/cf6 to d5/d21/d4a/d52/d75/dc8/c10a 0 2024-04-20T21:32:21.227 INFO:tasks.workunit.client.1.smithi189.stdout:5/809: read - d7/d9/d31/d36/fef zero size 2024-04-20T21:32:21.229 INFO:tasks.workunit.client.1.smithi189.stdout:8/850: stat da/d81/d123/d20/l4d 0 2024-04-20T21:32:21.230 INFO:tasks.workunit.client.1.smithi189.stdout:1/727: dwrite d8/d1e/f6e [0,4194304] 0 2024-04-20T21:32:21.231 INFO:tasks.workunit.client.1.smithi189.stdout:9/698: creat d9/d12/d3b/db1/fdd x:0 0 0 2024-04-20T21:32:21.234 INFO:tasks.workunit.client.1.smithi189.stdout:2/720: dread d1/d11/d2d/dab/d74/f9b [0,4194304] 0 2024-04-20T21:32:21.241 INFO:tasks.workunit.client.1.smithi189.stdout:7/825: mknod d5/d1d/d77/db8/c10b 0 2024-04-20T21:32:21.243 INFO:tasks.workunit.client.0.smithi062.stdout:5/996: read - d8/d38/d7a/d4c/d7d/d9d/dac/d66/f149 zero size 2024-04-20T21:32:21.244 INFO:tasks.workunit.client.1.smithi189.stdout:9/699: creat d9/d12/d3f/d72/fde x:0 0 0 2024-04-20T21:32:21.247 INFO:tasks.workunit.client.1.smithi189.stdout:9/700: chown d9/dc/d17/l20 53177 1 2024-04-20T21:32:21.252 INFO:tasks.workunit.client.1.smithi189.stdout:9/701: fdatasync d9/dc/d17/d36/f78 0 2024-04-20T21:32:21.253 INFO:tasks.workunit.client.1.smithi189.stdout:2/721: dwrite d1/d11/f15 [4194304,4194304] 0 2024-04-20T21:32:21.257 INFO:tasks.workunit.client.1.smithi189.stdout:7/826: mknod d5/c10c 0 2024-04-20T21:32:21.259 INFO:tasks.workunit.client.1.smithi189.stdout:7/827: dread - d5/d46/d7d/fb1 zero size 2024-04-20T21:32:21.260 INFO:tasks.workunit.client.1.smithi189.stdout:7/828: stat d5/f14 0 2024-04-20T21:32:21.262 INFO:tasks.workunit.client.1.smithi189.stdout:7/829: chown d5/ff 608873 1 2024-04-20T21:32:21.264 INFO:tasks.workunit.client.0.smithi062.stdout:5/997: dwrite d8/d38/fbe [0,4194304] 0 2024-04-20T21:32:21.267 INFO:tasks.workunit.client.0.smithi062.stdout:5/998: readlink d8/d10/ld9 0 2024-04-20T21:32:21.268 INFO:tasks.workunit.client.1.smithi189.stdout:2/722: creat d1/d17/fd6 x:0 0 0 2024-04-20T21:32:21.270 INFO:tasks.workunit.client.1.smithi189.stdout:8/851: write da/de/d1e/d21/d44/f43 [1370726,129644] 0 2024-04-20T21:32:21.271 INFO:tasks.workunit.client.1.smithi189.stdout:5/810: dread d7/d9/d30/f8e [0,4194304] 0 2024-04-20T21:32:21.272 INFO:tasks.workunit.client.0.smithi062.stdout:6/888: dread d0/d9/f13 [0,4194304] 0 2024-04-20T21:32:21.272 INFO:tasks.workunit.client.1.smithi189.stdout:7/830: rmdir d5/d21/d4a/d52/d98 39 2024-04-20T21:32:21.273 INFO:tasks.workunit.client.1.smithi189.stdout:9/702: link d9/d12/d3f/ld6 d9/dc/d17/d95/da0/d52/d82/d94/ldf 0 2024-04-20T21:32:21.279 INFO:tasks.workunit.client.1.smithi189.stdout:2/723: rmdir d1/d17/d37/d89 39 2024-04-20T21:32:21.279 INFO:tasks.workunit.client.1.smithi189.stdout:1/728: truncate f6 454361 0 2024-04-20T21:32:21.280 INFO:tasks.workunit.client.1.smithi189.stdout:9/703: readlink d9/d12/d3f/ld6 0 2024-04-20T21:32:21.280 INFO:tasks.workunit.client.1.smithi189.stdout:7/831: symlink d5/d59/l10d 0 2024-04-20T21:32:21.281 INFO:tasks.workunit.client.0.smithi062.stdout:5/999: symlink d8/d38/d7a/d4c/d7d/d9d/dac/d66/l14a 0 2024-04-20T21:32:21.283 INFO:tasks.workunit.client.1.smithi189.stdout:9/704: stat d9/d12/d50 0 2024-04-20T21:32:21.284 INFO:tasks.workunit.client.1.smithi189.stdout:5/811: creat d7/d9/d30/d55/d61/db3/f101 x:0 0 0 2024-04-20T21:32:21.291 INFO:tasks.workunit.client.1.smithi189.stdout:2/724: truncate d1/d17/f7f 2383121 0 2024-04-20T21:32:21.293 INFO:tasks.workunit.client.1.smithi189.stdout:7/832: getdents d5/d21/d25 0 2024-04-20T21:32:21.295 INFO:tasks.workunit.client.1.smithi189.stdout:8/852: dwrite da/d81/d123/d3c/d9d/fec [0,4194304] 0 2024-04-20T21:32:21.296 INFO:tasks.workunit.client.1.smithi189.stdout:4/721: dread d2/dd/f2d [0,4194304] 0 2024-04-20T21:32:21.298 INFO:tasks.workunit.client.1.smithi189.stdout:7/833: truncate d5/d21/d4a/dfd/f104 1014155 0 2024-04-20T21:32:21.300 INFO:tasks.workunit.client.1.smithi189.stdout:7/834: write d5/d21/f32 [1521244,74982] 0 2024-04-20T21:32:21.304 INFO:tasks.workunit.client.1.smithi189.stdout:7/835: truncate d5/d46/d7d/fb1 812973 0 2024-04-20T21:32:21.309 INFO:tasks.workunit.client.1.smithi189.stdout:7/836: creat d5/d21/d97/da3/f10e x:0 0 0 2024-04-20T21:32:21.316 INFO:tasks.workunit.client.1.smithi189.stdout:7/837: mknod d5/d1d/d64/db9/ddd/c10f 0 2024-04-20T21:32:21.320 INFO:tasks.workunit.client.1.smithi189.stdout:2/725: dwrite d1/d11/d52/d8b/fa1 [0,4194304] 0 2024-04-20T21:32:21.321 INFO:tasks.workunit.client.1.smithi189.stdout:7/838: unlink d5/d59/lc4 0 2024-04-20T21:32:21.323 INFO:tasks.workunit.client.1.smithi189.stdout:7/839: stat d5/f7 0 2024-04-20T21:32:21.329 INFO:tasks.workunit.client.1.smithi189.stdout:2/726: chown d1/d11/d2d/dab/d4a/d66/ccb 15 1 2024-04-20T21:32:21.330 INFO:tasks.workunit.client.0.smithi062.stdout:6/889: write d0/d9/d11/d22/dd5/f89 [9492,10384] 0 2024-04-20T21:32:21.330 INFO:tasks.workunit.client.1.smithi189.stdout:5/812: write d7/d9/d28/f7b [2896685,77937] 0 2024-04-20T21:32:21.331 INFO:tasks.workunit.client.1.smithi189.stdout:1/729: truncate d8/d21/d2c/fa3 60317 0 2024-04-20T21:32:21.333 INFO:tasks.workunit.client.1.smithi189.stdout:2/727: write d1/d11/d2d/dab/d74/fb3 [2775803,85596] 0 2024-04-20T21:32:21.334 INFO:tasks.workunit.client.1.smithi189.stdout:1/730: stat d8/d1e/d28/fd8 0 2024-04-20T21:32:21.338 INFO:tasks.workunit.client.0.smithi062.stdout:6/890: creat d0/dc7/d7c/dc5/f126 x:0 0 0 2024-04-20T21:32:21.341 INFO:tasks.workunit.client.1.smithi189.stdout:1/731: symlink d8/d1e/d43/d44/dd2/d8a/lf9 0 2024-04-20T21:32:21.342 INFO:tasks.workunit.client.0.smithi062.stdout:6/891: truncate d0/dc7/f86 1019437 0 2024-04-20T21:32:21.345 INFO:tasks.workunit.client.0.smithi062.stdout:6/892: mknod d0/d1a/d44/db3/c127 0 2024-04-20T21:32:21.346 INFO:tasks.workunit.client.1.smithi189.stdout:5/813: getdents d7/d9/d51/d70/d99/dd4 0 2024-04-20T21:32:21.347 INFO:tasks.workunit.client.1.smithi189.stdout:7/840: dwrite d5/f7 [0,4194304] 0 2024-04-20T21:32:21.347 INFO:tasks.workunit.client.1.smithi189.stdout:1/732: creat d8/d1e/d54/d93/ffa x:0 0 0 2024-04-20T21:32:21.351 INFO:tasks.workunit.client.1.smithi189.stdout:1/733: fsync d8/d1e/fc5 0 2024-04-20T21:32:21.352 INFO:tasks.workunit.client.1.smithi189.stdout:9/705: dread d9/f34 [0,4194304] 0 2024-04-20T21:32:21.355 INFO:tasks.workunit.client.1.smithi189.stdout:7/841: dread - d5/d21/d97/da3/fef zero size 2024-04-20T21:32:21.357 INFO:tasks.workunit.client.1.smithi189.stdout:1/734: dread - d8/d1e/d43/d44/dd2/d8a/fd0 zero size 2024-04-20T21:32:21.357 INFO:tasks.workunit.client.1.smithi189.stdout:7/842: rename d5/d1d to d5/d1d/d110 22 2024-04-20T21:32:21.360 INFO:tasks.workunit.client.1.smithi189.stdout:1/735: chown d8/d1e/d28/f91 3346 1 2024-04-20T21:32:21.363 INFO:tasks.workunit.client.1.smithi189.stdout:9/706: creat d9/d12/d3b/fe0 x:0 0 0 2024-04-20T21:32:21.364 INFO:tasks.workunit.client.1.smithi189.stdout:7/843: mkdir d5/d46/d7d/d111 0 2024-04-20T21:32:21.369 INFO:tasks.workunit.client.0.smithi062.stdout:6/893: write d0/d1a/d44/db3/f106 [36267,104273] 0 2024-04-20T21:32:21.373 INFO:tasks.workunit.client.0.smithi062.stdout:6/894: mkdir d0/d9a/d128 0 2024-04-20T21:32:21.376 INFO:tasks.workunit.client.1.smithi189.stdout:4/722: dread d2/dd/d11/f9a [0,4194304] 0 2024-04-20T21:32:21.380 INFO:tasks.workunit.client.1.smithi189.stdout:2/728: dwrite d1/f7 [0,4194304] 0 2024-04-20T21:32:21.380 INFO:tasks.workunit.client.1.smithi189.stdout:5/814: write d7/d35/d48/f7d [1026496,14776] 0 2024-04-20T21:32:21.380 INFO:tasks.workunit.client.0.smithi062.stdout:6/895: rename d0/dc7/d7c/dc5/de9/f109 to d0/d1a/d7d/f129 0 2024-04-20T21:32:21.385 INFO:tasks.workunit.client.1.smithi189.stdout:1/736: write d8/d21/d2c/f75 [2439817,70529] 0 2024-04-20T21:32:21.386 INFO:tasks.workunit.client.1.smithi189.stdout:9/707: write d9/d12/d3b/db1/d1a/f9d [544182,129280] 0 2024-04-20T21:32:21.387 INFO:tasks.workunit.client.1.smithi189.stdout:2/729: read d1/d11/d2d/f38 [1729398,117516] 0 2024-04-20T21:32:21.387 INFO:tasks.workunit.client.1.smithi189.stdout:4/723: chown d2/d22/d57/fe6 23 1 2024-04-20T21:32:21.393 INFO:tasks.workunit.client.1.smithi189.stdout:2/730: chown d1/d11/d2d/la7 7516 1 2024-04-20T21:32:21.394 INFO:tasks.workunit.client.1.smithi189.stdout:7/844: dwrite d5/d21/f109 [0,4194304] 0 2024-04-20T21:32:21.398 INFO:tasks.workunit.client.1.smithi189.stdout:7/845: dread - d5/d1d/d77/ff8 zero size 2024-04-20T21:32:21.399 INFO:tasks.workunit.client.1.smithi189.stdout:4/724: write d2/dd/d37/dcd/d6a/d7a/f8d [883890,98860] 0 2024-04-20T21:32:21.401 INFO:tasks.workunit.client.1.smithi189.stdout:9/708: truncate d9/dc/d17/faa 186343 0 2024-04-20T21:32:21.403 INFO:tasks.workunit.client.1.smithi189.stdout:2/731: mknod d1/d11/d58/cd7 0 2024-04-20T21:32:21.409 INFO:tasks.workunit.client.1.smithi189.stdout:4/725: mknod d2/dd/d37/dcd/c103 0 2024-04-20T21:32:21.411 INFO:tasks.workunit.client.0.smithi062.stdout:6/896: dwrite d0/d9/d11/d22/dd5/dff/f11e [0,4194304] 0 2024-04-20T21:32:21.411 INFO:tasks.workunit.client.1.smithi189.stdout:5/815: dwrite d7/d9/d28/fa0 [0,4194304] 0 2024-04-20T21:32:21.416 INFO:tasks.workunit.client.1.smithi189.stdout:2/732: chown d1/f12 59 1 2024-04-20T21:32:21.419 INFO:tasks.workunit.client.0.smithi062.stdout:3/906: sync 2024-04-20T21:32:21.419 INFO:tasks.workunit.client.0.smithi062.stdout:8/998: sync 2024-04-20T21:32:21.420 INFO:tasks.workunit.client.0.smithi062.stdout:0/944: sync 2024-04-20T21:32:21.420 INFO:tasks.workunit.client.1.smithi189.stdout:4/726: symlink d2/d3a/ddc/l104 0 2024-04-20T21:32:21.422 INFO:tasks.workunit.client.1.smithi189.stdout:4/727: fsync d2/dd/ffa 0 2024-04-20T21:32:21.425 INFO:tasks.workunit.client.0.smithi062.stdout:6/897: unlink d0/d9/d11/d7b/d99/ddd/de8/cd4 0 2024-04-20T21:32:21.426 INFO:tasks.workunit.client.1.smithi189.stdout:4/728: fsync d2/d7f/fc1 0 2024-04-20T21:32:21.428 INFO:tasks.workunit.client.1.smithi189.stdout:2/733: mknod d1/d11/d2d/d4f/db4/daa/dd2/cd8 0 2024-04-20T21:32:21.431 INFO:tasks.workunit.client.0.smithi062.stdout:0/945: mknod d7/d20/d62/d6f/c123 0 2024-04-20T21:32:21.432 INFO:tasks.workunit.client.1.smithi189.stdout:1/737: dwrite d8/f12 [0,4194304] 0 2024-04-20T21:32:21.432 INFO:tasks.workunit.client.1.smithi189.stdout:4/729: truncate d2/d7f/fb4 377535 0 2024-04-20T21:32:21.435 INFO:tasks.workunit.client.0.smithi062.stdout:3/907: mknod d0/d1c/d63/dda/c128 0 2024-04-20T21:32:21.437 INFO:tasks.workunit.client.0.smithi062.stdout:8/999: mkdir d7/d25/d3b/d58/d8b/d24/dbd/d127/d141 0 2024-04-20T21:32:21.437 INFO:tasks.workunit.client.1.smithi189.stdout:7/846: dread d5/d21/f109 [0,4194304] 0 2024-04-20T21:32:21.437 INFO:tasks.workunit.client.1.smithi189.stdout:2/734: creat d1/d11/d2d/d4f/fd9 x:0 0 0 2024-04-20T21:32:21.445 INFO:tasks.workunit.client.1.smithi189.stdout:7/847: write d5/d1d/d64/fd7 [240621,39538] 0 2024-04-20T21:32:21.445 INFO:tasks.workunit.client.1.smithi189.stdout:7/848: readlink d5/db4/d71/le6 0 2024-04-20T21:32:21.453 INFO:tasks.workunit.client.0.smithi062.stdout:6/898: link d0/d1a/d7d/c55 d0/d9/d11/d22/dd5/dff/c12a 0 2024-04-20T21:32:21.455 INFO:tasks.workunit.client.0.smithi062.stdout:6/899: write d0/d9/d11/d7b/d99/ddd/f10d [179335,85822] 0 2024-04-20T21:32:21.456 INFO:tasks.workunit.client.1.smithi189.stdout:2/735: dread d1/d11/d2d/dab/d55/f61 [0,4194304] 0 2024-04-20T21:32:21.457 INFO:tasks.workunit.client.1.smithi189.stdout:9/709: dread f4 [0,4194304] 0 2024-04-20T21:32:21.464 INFO:tasks.workunit.client.0.smithi062.stdout:0/946: creat d7/d1c/d15/d17/d37/f124 x:0 0 0 2024-04-20T21:32:21.467 INFO:tasks.workunit.client.0.smithi062.stdout:0/947: write d7/d1c/d15/d17/d40/fff [875852,54714] 0 2024-04-20T21:32:21.467 INFO:tasks.workunit.client.1.smithi189.stdout:9/710: getdents d9/d12/d3b/db1 0 2024-04-20T21:32:21.474 INFO:tasks.workunit.client.1.smithi189.stdout:2/736: dwrite d1/d17/f2b [0,4194304] 0 2024-04-20T21:32:21.482 INFO:tasks.workunit.client.1.smithi189.stdout:2/737: creat d1/d11/d2d/dab/d74/fda x:0 0 0 2024-04-20T21:32:21.484 INFO:tasks.workunit.client.1.smithi189.stdout:5/816: dread d7/d9/d30/d5a/d9d/fac [0,4194304] 0 2024-04-20T21:32:21.485 INFO:tasks.workunit.client.1.smithi189.stdout:2/738: stat d1/d11/d52/f6a 0 2024-04-20T21:32:21.492 INFO:tasks.workunit.client.1.smithi189.stdout:5/817: readlink d7/d9/d31/d100/d4e/d5b/l79 0 2024-04-20T21:32:21.493 INFO:tasks.workunit.client.1.smithi189.stdout:9/711: dwrite d9/d12/fa8 [0,4194304] 0 2024-04-20T21:32:21.499 INFO:tasks.workunit.client.0.smithi062.stdout:6/900: fsync d0/d52/fb0 0 2024-04-20T21:32:21.507 INFO:tasks.workunit.client.0.smithi062.stdout:3/908: dwrite d0/d1/d115/d11c/f85 [0,4194304] 0 2024-04-20T21:32:21.510 INFO:tasks.workunit.client.1.smithi189.stdout:4/730: dwrite d2/f10 [4194304,4194304] 0 2024-04-20T21:32:21.519 INFO:tasks.workunit.client.1.smithi189.stdout:4/731: chown d2/dd/d11/la7 850 1 2024-04-20T21:32:21.519 INFO:tasks.workunit.client.0.smithi062.stdout:3/909: getdents d0/d1/d3c/d43/d72/dd3/d97 0 2024-04-20T21:32:21.524 INFO:tasks.workunit.client.1.smithi189.stdout:6/807: sync 2024-04-20T21:32:21.524 INFO:tasks.workunit.client.1.smithi189.stdout:8/853: sync 2024-04-20T21:32:21.524 INFO:tasks.workunit.client.1.smithi189.stdout:3/785: sync 2024-04-20T21:32:21.525 INFO:tasks.workunit.client.0.smithi062.stdout:3/910: rename d0/f98 to d0/d1/d10/f129 0 2024-04-20T21:32:21.525 INFO:tasks.workunit.client.1.smithi189.stdout:1/738: dread d8/d1e/d43/d44/dd2/d8a/fda [0,4194304] 0 2024-04-20T21:32:21.525 INFO:tasks.workunit.client.1.smithi189.stdout:0/868: sync 2024-04-20T21:32:21.528 INFO:tasks.workunit.client.1.smithi189.stdout:1/739: dread - d8/d1e/fca zero size 2024-04-20T21:32:21.551 INFO:tasks.workunit.client.0.smithi062.stdout:3/911: rename d0/d1c/d63/d42/dc8/df6 to d0/d1/d10/d3a/de9/d12a 0 2024-04-20T21:32:21.560 INFO:tasks.workunit.client.0.smithi062.stdout:3/912: rename d0/d1/d3c/d43/d72/df2/cf4 to d0/d1/d3c/d43/d72/dd3/dbf/c12b 0 2024-04-20T21:32:21.567 INFO:tasks.workunit.client.0.smithi062.stdout:3/913: rename d0/d1/f8b to d0/f12c 0 2024-04-20T21:32:21.571 INFO:tasks.workunit.client.0.smithi062.stdout:3/914: read d0/d1b/f75 [2310172,92347] 0 2024-04-20T21:32:21.573 INFO:tasks.workunit.client.0.smithi062.stdout:3/915: rmdir d0/d1/de4/d118 39 2024-04-20T21:32:21.576 INFO:tasks.workunit.client.1.smithi189.stdout:8/854: mknod da/d81/d123/d3c/deb/c12d 0 2024-04-20T21:32:21.578 INFO:tasks.workunit.client.1.smithi189.stdout:0/869: rename d3/df/d30/d45/dd3/l117 to d3/df/d28/d2b/db1/dfe/l126 0 2024-04-20T21:32:21.580 INFO:tasks.workunit.client.1.smithi189.stdout:1/740: mknod d8/df/d3a/d49/d8c/da2/dbe/cfb 0 2024-04-20T21:32:21.580 INFO:tasks.workunit.client.0.smithi062.stdout:3/916: link d0/d1b/f57 d0/d1c/d63/d42/f12d 0 2024-04-20T21:32:21.581 INFO:tasks.workunit.client.1.smithi189.stdout:2/739: getdents d1/d11/d2d/d4f/d8e 0 2024-04-20T21:32:21.582 INFO:tasks.workunit.client.1.smithi189.stdout:2/740: dread - d1/d11/d2d/d4f/d8e/fb5 zero size 2024-04-20T21:32:21.583 INFO:tasks.workunit.client.1.smithi189.stdout:5/818: link d7/d9/d30/f8e d7/d9/d51/d70/f102 0 2024-04-20T21:32:21.584 INFO:tasks.workunit.client.0.smithi062.stdout:3/917: chown d0/d5c/l33 1 1 2024-04-20T21:32:21.586 INFO:tasks.workunit.client.1.smithi189.stdout:5/819: write d7/d35/fd9 [507767,20128] 0 2024-04-20T21:32:21.586 INFO:tasks.workunit.client.1.smithi189.stdout:4/732: dwrite d2/f8 [0,4194304] 0 2024-04-20T21:32:21.587 INFO:tasks.workunit.client.1.smithi189.stdout:6/808: symlink d8/da/dd/d2a/l10e 0 2024-04-20T21:32:21.589 INFO:tasks.workunit.client.0.smithi062.stdout:3/918: chown d0/d5c/lc0 198 1 2024-04-20T21:32:21.591 INFO:tasks.workunit.client.1.smithi189.stdout:8/855: unlink da/d81/d123/d3c/deb/f103 0 2024-04-20T21:32:21.594 INFO:tasks.workunit.client.1.smithi189.stdout:1/741: symlink d8/df/d3a/d49/d8c/da2/dbe/dc1/de9/lfc 0 2024-04-20T21:32:21.596 INFO:tasks.workunit.client.1.smithi189.stdout:2/741: truncate d1/d17/f35 7565653 0 2024-04-20T21:32:21.597 INFO:tasks.workunit.client.1.smithi189.stdout:1/742: chown d8/df/d50/ca0 1941 1 2024-04-20T21:32:21.598 INFO:tasks.workunit.client.0.smithi062.stdout:3/919: symlink d0/d1/d10/d31/d41/l12e 0 2024-04-20T21:32:21.601 INFO:tasks.workunit.client.1.smithi189.stdout:3/786: fsync d2/d3/d2c/d31/d74/d7a/f7c 0 2024-04-20T21:32:21.604 INFO:tasks.workunit.client.1.smithi189.stdout:5/820: mknod d7/d9/d31/d36/c103 0 2024-04-20T21:32:21.607 INFO:tasks.workunit.client.1.smithi189.stdout:9/712: write d9/d12/d3b/db1/f26 [288152,103847] 0 2024-04-20T21:32:21.615 INFO:tasks.workunit.client.1.smithi189.stdout:7/849: dwrite d5/d46/f65 [4194304,4194304] 0 2024-04-20T21:32:21.615 INFO:tasks.workunit.client.1.smithi189.stdout:4/733: dwrite d2/d7f/fc1 [4194304,4194304] 0 2024-04-20T21:32:21.621 INFO:tasks.workunit.client.1.smithi189.stdout:2/742: write d1/d11/d2d/d4f/db4/f5a [358972,64914] 0 2024-04-20T21:32:21.623 INFO:tasks.workunit.client.1.smithi189.stdout:8/856: dwrite da/d81/d123/d3c/d68/d8b/d8d/f90 [0,4194304] 0 2024-04-20T21:32:21.629 INFO:tasks.workunit.client.1.smithi189.stdout:6/809: mknod d8/d12/dd7/c10f 0 2024-04-20T21:32:21.629 INFO:tasks.workunit.client.0.smithi062.stdout:3/920: dwrite d0/d1c/d63/d42/dc8/dc9/ff9 [0,4194304] 0 2024-04-20T21:32:21.632 INFO:tasks.workunit.client.1.smithi189.stdout:5/821: truncate d7/d9/d30/d55/d61/d87/fe1 980313 0 2024-04-20T21:32:21.632 INFO:tasks.workunit.client.1.smithi189.stdout:8/857: readlink da/d81/d123/d20/l5b 0 2024-04-20T21:32:21.632 INFO:tasks.workunit.client.1.smithi189.stdout:9/713: creat d9/dc/d17/d95/da6/fe1 x:0 0 0 2024-04-20T21:32:21.634 INFO:tasks.workunit.client.1.smithi189.stdout:2/743: rmdir d1 39 2024-04-20T21:32:21.635 INFO:tasks.workunit.client.1.smithi189.stdout:6/810: stat d8/d12/d30/d47 0 2024-04-20T21:32:21.639 INFO:tasks.workunit.client.1.smithi189.stdout:4/734: creat d2/dd/d37/dcd/d6a/f105 x:0 0 0 2024-04-20T21:32:21.642 INFO:tasks.workunit.client.1.smithi189.stdout:7/850: truncate d5/d21/d97/da3/fef 354529 0 2024-04-20T21:32:21.645 INFO:tasks.workunit.client.0.smithi062.stdout:3/921: dread d0/d1b/f87 [0,4194304] 0 2024-04-20T21:32:21.645 INFO:tasks.workunit.client.1.smithi189.stdout:5/822: mkdir d7/d9/d30/d55/d104 0 2024-04-20T21:32:21.647 INFO:tasks.workunit.client.1.smithi189.stdout:3/787: dwrite d2/d3/d25/f67 [0,4194304] 0 2024-04-20T21:32:21.656 INFO:tasks.workunit.client.1.smithi189.stdout:7/851: rmdir d5/d21/d4a/d52/d75/dc8 39 2024-04-20T21:32:21.660 INFO:tasks.workunit.client.1.smithi189.stdout:3/788: write d2/d1c/f2b [2830434,116932] 0 2024-04-20T21:32:21.661 INFO:tasks.workunit.client.1.smithi189.stdout:7/852: dread d5/d21/d25/f78 [0,4194304] 0 2024-04-20T21:32:21.663 INFO:tasks.workunit.client.1.smithi189.stdout:6/811: truncate d8/d12/d30/d47/f8a 476596 0 2024-04-20T21:32:21.666 INFO:tasks.workunit.client.1.smithi189.stdout:5/823: link d7/d9/fbf d7/d9/d30/d55/d81/f105 0 2024-04-20T21:32:21.669 INFO:tasks.workunit.client.1.smithi189.stdout:7/853: creat d5/d1d/ded/f112 x:0 0 0 2024-04-20T21:32:21.671 INFO:tasks.workunit.client.1.smithi189.stdout:0/870: dwrite d3/d26/f1d [4194304,4194304] 0 2024-04-20T21:32:21.674 INFO:tasks.workunit.client.1.smithi189.stdout:7/854: chown d5/d21/f109 23332 1 2024-04-20T21:32:21.678 INFO:tasks.workunit.client.1.smithi189.stdout:7/855: truncate d5/f39 3416220 0 2024-04-20T21:32:21.681 INFO:tasks.workunit.client.1.smithi189.stdout:9/714: dwrite d9/dc/d17/d36/fce [0,4194304] 0 2024-04-20T21:32:21.687 INFO:tasks.workunit.client.1.smithi189.stdout:0/871: creat d3/df/d30/d45/d90/f127 x:0 0 0 2024-04-20T21:32:21.691 INFO:tasks.workunit.client.1.smithi189.stdout:9/715: truncate d9/dc/d17/d95/da6/fe1 1010637 0 2024-04-20T21:32:21.693 INFO:tasks.workunit.client.1.smithi189.stdout:7/856: creat d5/db4/d71/d7a/f113 x:0 0 0 2024-04-20T21:32:21.695 INFO:tasks.workunit.client.1.smithi189.stdout:7/857: fdatasync d5/db4/d71/fa4 0 2024-04-20T21:32:21.696 INFO:tasks.workunit.client.1.smithi189.stdout:9/716: readlink d9/dc/d17/dac/lca 0 2024-04-20T21:32:21.700 INFO:tasks.workunit.client.1.smithi189.stdout:0/872: rmdir d3/df/d30/d37/dbd/d11e 0 2024-04-20T21:32:21.713 INFO:tasks.workunit.client.1.smithi189.stdout:8/858: dread da/de/d1e/d69/da3/fc3 [0,4194304] 0 2024-04-20T21:32:21.716 INFO:tasks.workunit.client.1.smithi189.stdout:0/873: dread - d3/df/d28/d2b/db1/f107 zero size 2024-04-20T21:32:21.718 INFO:tasks.workunit.client.1.smithi189.stdout:2/744: dwrite d1/d11/d2d/dab/d4a/f7e [0,4194304] 0 2024-04-20T21:32:21.720 INFO:tasks.workunit.client.1.smithi189.stdout:8/859: mkdir da/de/d1e/d69/d70/d12e 0 2024-04-20T21:32:21.728 INFO:tasks.workunit.client.0.smithi062.stdout:6/901: sync 2024-04-20T21:32:21.728 INFO:tasks.workunit.client.0.smithi062.stdout:0/948: sync 2024-04-20T21:32:21.728 INFO:tasks.workunit.client.1.smithi189.stdout:7/858: dread d5/d21/f23 [0,4194304] 0 2024-04-20T21:32:21.729 INFO:tasks.workunit.client.1.smithi189.stdout:7/859: stat d5/d21/d4a/ffb 0 2024-04-20T21:32:21.730 INFO:tasks.workunit.client.1.smithi189.stdout:8/860: fsync da/de/d1e/d69/f95 0 2024-04-20T21:32:21.731 INFO:tasks.workunit.client.1.smithi189.stdout:6/812: dread d8/f63 [0,4194304] 0 2024-04-20T21:32:21.731 INFO:tasks.workunit.client.0.smithi062.stdout:6/902: chown d0/d9/d11/d22/dd5/f115 1220334 1 2024-04-20T21:32:21.732 INFO:tasks.workunit.client.0.smithi062.stdout:6/903: chown d0/d1a/f29 732307687 1 2024-04-20T21:32:21.735 INFO:tasks.workunit.client.1.smithi189.stdout:7/860: symlink d5/d1d/d64/db9/dcf/l114 0 2024-04-20T21:32:21.737 INFO:tasks.workunit.client.1.smithi189.stdout:4/735: dread d2/dd/d11/d14/f3f [0,4194304] 0 2024-04-20T21:32:21.744 INFO:tasks.workunit.client.1.smithi189.stdout:4/736: mknod d2/dd/d6b/c106 0 2024-04-20T21:32:21.747 INFO:tasks.workunit.client.1.smithi189.stdout:9/717: dwrite d9/f87 [0,4194304] 0 2024-04-20T21:32:21.747 INFO:tasks.workunit.client.1.smithi189.stdout:7/861: mkdir d5/d21/d4a/d52/d74/d115 0 2024-04-20T21:32:21.750 INFO:tasks.workunit.client.1.smithi189.stdout:4/737: read - d2/dd/d37/dcd/d6a/f105 zero size 2024-04-20T21:32:21.751 INFO:tasks.workunit.client.1.smithi189.stdout:8/861: link da/d81/d123/d20/d2c/d9a/ddd/f124 da/de/d1e/d69/d70/d12e/f12f 0 2024-04-20T21:32:21.751 INFO:tasks.workunit.client.1.smithi189.stdout:7/862: unlink d5/c1c 0 2024-04-20T21:32:21.752 INFO:tasks.workunit.client.1.smithi189.stdout:2/745: dwrite d1/d17/d37/d89/f95 [0,4194304] 0 2024-04-20T21:32:21.754 INFO:tasks.workunit.client.1.smithi189.stdout:6/813: truncate d8/d12/d99/fb7 2414850 0 2024-04-20T21:32:21.754 INFO:tasks.workunit.client.1.smithi189.stdout:2/746: fsync d1/f7 0 2024-04-20T21:32:21.757 INFO:tasks.workunit.client.1.smithi189.stdout:7/863: read - d5/d21/d97/da3/fdb zero size 2024-04-20T21:32:21.759 INFO:tasks.workunit.client.1.smithi189.stdout:9/718: creat d9/d12/d62/dae/fe2 x:0 0 0 2024-04-20T21:32:21.759 INFO:tasks.workunit.client.1.smithi189.stdout:4/738: chown d2/d22/c46 9582 1 2024-04-20T21:32:21.761 INFO:tasks.workunit.client.1.smithi189.stdout:8/862: symlink da/d81/d123/d13/dd6/dba/d11e/l130 0 2024-04-20T21:32:21.762 INFO:tasks.workunit.client.1.smithi189.stdout:2/747: chown d1/d11/d2d/c3a 40576 1 2024-04-20T21:32:21.762 INFO:tasks.workunit.client.1.smithi189.stdout:9/719: write d9/dc/d69/faf [228554,128807] 0 2024-04-20T21:32:21.765 INFO:tasks.workunit.client.0.smithi062.stdout:0/949: creat d7/d1c/d15/d17/d37/d109/f125 x:0 0 0 2024-04-20T21:32:21.767 INFO:tasks.workunit.client.1.smithi189.stdout:8/863: creat da/d81/d123/d3c/d68/d8b/dac/de1/f131 x:0 0 0 2024-04-20T21:32:21.767 INFO:tasks.workunit.client.0.smithi062.stdout:6/904: creat d0/d9/d11/d7b/d99/ddd/f12b x:0 0 0 2024-04-20T21:32:21.769 INFO:tasks.workunit.client.1.smithi189.stdout:9/720: creat d9/d12/d50/fe3 x:0 0 0 2024-04-20T21:32:21.772 INFO:tasks.workunit.client.1.smithi189.stdout:8/864: truncate da/de/d1e/d69/f10c 467040 0 2024-04-20T21:32:21.774 INFO:tasks.workunit.client.1.smithi189.stdout:2/748: creat d1/d11/d2d/d4f/db4/daa/dd2/fdb x:0 0 0 2024-04-20T21:32:21.775 INFO:tasks.workunit.client.0.smithi062.stdout:0/950: rename d7/d20/d62/d6f/f108 to d7/d1c/f126 0 2024-04-20T21:32:21.777 INFO:tasks.workunit.client.1.smithi189.stdout:6/814: write d8/d10/d78/df7/fac [1376273,72296] 0 2024-04-20T21:32:21.781 INFO:tasks.workunit.client.1.smithi189.stdout:9/721: getdents d9 0 2024-04-20T21:32:21.782 INFO:tasks.workunit.client.0.smithi062.stdout:6/905: rename d0/d9/d11/d7b/cf8 to d0/dc7/d7c/dc5/def/c12c 0 2024-04-20T21:32:21.785 INFO:tasks.workunit.client.1.smithi189.stdout:9/722: truncate d9/d12/d3f/d72/fde 99081 0 2024-04-20T21:32:21.786 INFO:tasks.workunit.client.1.smithi189.stdout:6/815: rename d8/da/dd/c29 to d8/d10/d78/dd6/c110 0 2024-04-20T21:32:21.791 INFO:tasks.workunit.client.0.smithi062.stdout:6/906: rename d0/d9/d11/d22/dd5/c8f to d0/d9/d11/d22/dd5/dff/c12d 0 2024-04-20T21:32:21.805 INFO:tasks.workunit.client.0.smithi062.stdout:6/907: dwrite d0/fb6 [0,4194304] 0 2024-04-20T21:32:21.814 INFO:tasks.workunit.client.0.smithi062.stdout:3/922: write d0/d1/d115/d11c/fb6 [1117845,40456] 0 2024-04-20T21:32:21.815 INFO:tasks.workunit.client.0.smithi062.stdout:3/923: stat d0/d1/d10/d31/c59 0 2024-04-20T21:32:21.815 INFO:tasks.workunit.client.0.smithi062.stdout:6/908: unlink d0/d1a/d7d/c55 0 2024-04-20T21:32:21.818 INFO:tasks.workunit.client.0.smithi062.stdout:3/924: link d0/d1/d10/d31/d41/c108 d0/d1/c12f 0 2024-04-20T21:32:21.821 INFO:tasks.workunit.client.0.smithi062.stdout:6/909: getdents d0/dc7/d7c/dc5 0 2024-04-20T21:32:21.823 INFO:tasks.workunit.client.0.smithi062.stdout:6/910: chown d0/d9/f15 5143 1 2024-04-20T21:32:21.825 INFO:tasks.workunit.client.0.smithi062.stdout:6/911: fsync d0/d4f/fdf 0 2024-04-20T21:32:21.826 INFO:tasks.workunit.client.0.smithi062.stdout:3/925: dread d0/d1c/d63/dda/fed [0,4194304] 0 2024-04-20T21:32:21.827 INFO:tasks.workunit.client.0.smithi062.stdout:6/912: write d0/d9/d11/f11c [360202,2919] 0 2024-04-20T21:32:21.830 INFO:tasks.workunit.client.0.smithi062.stdout:3/926: creat d0/d1c/f130 x:0 0 0 2024-04-20T21:32:21.831 INFO:tasks.workunit.client.0.smithi062.stdout:6/913: read d0/d9/d11/d22/dd5/dff/f11e [2999498,97436] 0 2024-04-20T21:32:21.834 INFO:tasks.workunit.client.0.smithi062.stdout:6/914: read d0/f79 [692214,23404] 0 2024-04-20T21:32:21.836 INFO:tasks.workunit.client.0.smithi062.stdout:3/927: chown d0/d1/d3c/d43/d72/dd3/dbf/f10c 1252694 1 2024-04-20T21:32:21.838 INFO:tasks.workunit.client.0.smithi062.stdout:3/928: write d0/fdc [1706064,21142] 0 2024-04-20T21:32:21.839 INFO:tasks.workunit.client.1.smithi189.stdout:9/723: creat d9/dc/d17/d95/fe4 x:0 0 0 2024-04-20T21:32:21.840 INFO:tasks.workunit.client.0.smithi062.stdout:6/915: rmdir d0/dc7/de5 0 2024-04-20T21:32:21.841 INFO:tasks.workunit.client.0.smithi062.stdout:3/929: symlink d0/d5c/l131 0 2024-04-20T21:32:21.844 INFO:tasks.workunit.client.1.smithi189.stdout:6/816: mkdir d8/d2b/d49/d108/d111 0 2024-04-20T21:32:21.847 INFO:tasks.workunit.client.1.smithi189.stdout:6/817: dread - d8/d10/d78/df7/ff2 zero size 2024-04-20T21:32:21.847 INFO:tasks.workunit.client.1.smithi189.stdout:8/865: dwrite da/d81/d123/d3c/d68/d8b/f10a [0,4194304] 0 2024-04-20T21:32:21.860 INFO:tasks.workunit.client.1.smithi189.stdout:8/866: truncate da/d81/d123/d3c/d50/f66 1215909 0 2024-04-20T21:32:21.865 INFO:tasks.workunit.client.1.smithi189.stdout:6/818: getdents d8/d2b/d49/d108 0 2024-04-20T21:32:21.866 INFO:tasks.workunit.client.1.smithi189.stdout:9/724: dwrite d9/dc/d69/f7f [0,4194304] 0 2024-04-20T21:32:21.866 INFO:tasks.workunit.client.1.smithi189.stdout:8/867: symlink da/d81/d123/d3c/d68/d8b/dac/l132 0 2024-04-20T21:32:21.868 INFO:tasks.workunit.client.1.smithi189.stdout:2/749: dread d1/d11/d2d/d4f/db4/fb1 [0,4194304] 0 2024-04-20T21:32:21.871 INFO:tasks.workunit.client.1.smithi189.stdout:6/819: mknod d8/da/c112 0 2024-04-20T21:32:21.880 INFO:tasks.workunit.client.0.smithi062.stdout:3/930: dread d0/d1/d10/d31/d41/de6/f126 [0,4194304] 0 2024-04-20T21:32:21.882 INFO:tasks.workunit.client.1.smithi189.stdout:6/820: creat d8/da/dd/f113 x:0 0 0 2024-04-20T21:32:21.884 INFO:tasks.workunit.client.0.smithi062.stdout:3/931: creat d0/d1c/f132 x:0 0 0 2024-04-20T21:32:21.891 INFO:tasks.workunit.client.1.smithi189.stdout:6/821: rename d8/da/dd/d2a/d3b/f8e to d8/d12/d4e/dc4/f114 0 2024-04-20T21:32:21.893 INFO:tasks.workunit.client.1.smithi189.stdout:2/750: dwrite d1/d11/d52/f6a [4194304,4194304] 0 2024-04-20T21:32:21.897 INFO:tasks.workunit.client.1.smithi189.stdout:9/725: dwrite d9/dc/d17/d36/d5d/fa5 [0,4194304] 0 2024-04-20T21:32:21.900 INFO:tasks.workunit.client.1.smithi189.stdout:8/868: dwrite da/d81/d123/d13/d63/f7f [0,4194304] 0 2024-04-20T21:32:21.903 INFO:tasks.workunit.client.1.smithi189.stdout:8/869: chown da/d81/d123/d13/dd6/dba/dc5 41234786 1 2024-04-20T21:32:21.908 INFO:tasks.workunit.client.1.smithi189.stdout:1/743: sync 2024-04-20T21:32:21.908 INFO:tasks.workunit.client.1.smithi189.stdout:3/789: sync 2024-04-20T21:32:21.909 INFO:tasks.workunit.client.1.smithi189.stdout:2/751: creat d1/d11/d2d/d73/dc6/fdc x:0 0 0 2024-04-20T21:32:21.909 INFO:tasks.workunit.client.1.smithi189.stdout:6/822: rename d8/d12/l7c to d8/d2b/d49/d54/d6e/l115 0 2024-04-20T21:32:21.910 INFO:tasks.workunit.client.1.smithi189.stdout:9/726: mknod d9/d12/d3b/ce5 0 2024-04-20T21:32:21.912 INFO:tasks.workunit.client.1.smithi189.stdout:8/870: write da/d81/d123/d20/d2c/d9a/ff7 [703359,54921] 0 2024-04-20T21:32:21.937 INFO:tasks.workunit.client.1.smithi189.stdout:9/727: chown d9/dc/d17/d36/ld7 104859 1 2024-04-20T21:32:21.937 INFO:tasks.workunit.client.1.smithi189.stdout:9/728: stat c2 0 2024-04-20T21:32:21.941 INFO:tasks.workunit.client.1.smithi189.stdout:9/729: rmdir d9/dc/d17/d7e 39 2024-04-20T21:32:21.941 INFO:tasks.workunit.client.1.smithi189.stdout:6/823: write d8/d2b/d49/d54/d6e/fb4 [1323690,90091] 0 2024-04-20T21:32:21.942 INFO:tasks.workunit.client.1.smithi189.stdout:3/790: creat d2/d3/f114 x:0 0 0 2024-04-20T21:32:21.946 INFO:tasks.workunit.client.1.smithi189.stdout:9/730: mkdir d9/dc/d17/d95/da0/d55/dd0/de6 0 2024-04-20T21:32:21.948 INFO:tasks.workunit.client.1.smithi189.stdout:6/824: truncate d8/d2b/f74 556259 0 2024-04-20T21:32:21.949 INFO:tasks.workunit.client.1.smithi189.stdout:9/731: creat d9/d12/d3f/d91/fe7 x:0 0 0 2024-04-20T21:32:21.951 INFO:tasks.workunit.client.1.smithi189.stdout:3/791: rename d2/d3/d2c/d31/lbc to d2/d3/d36/da1/l115 0 2024-04-20T21:32:21.952 INFO:tasks.workunit.client.1.smithi189.stdout:2/752: dwrite d1/d11/d2d/dab/f49 [0,4194304] 0 2024-04-20T21:32:21.954 INFO:tasks.workunit.client.1.smithi189.stdout:3/792: chown d2/d3/d2c/ff4 22989480 1 2024-04-20T21:32:21.955 INFO:tasks.workunit.client.1.smithi189.stdout:2/753: chown d1/d11/d2d/dab/d4a/c91 43009 1 2024-04-20T21:32:21.955 INFO:tasks.workunit.client.1.smithi189.stdout:9/732: mkdir d9/dc/d17/d7e/de8 0 2024-04-20T21:32:21.956 INFO:tasks.workunit.client.1.smithi189.stdout:6/825: creat d8/d12/f116 x:0 0 0 2024-04-20T21:32:21.958 INFO:tasks.workunit.client.1.smithi189.stdout:3/793: symlink d2/d3/d25/de0/da9/l116 0 2024-04-20T21:32:21.961 INFO:tasks.workunit.client.1.smithi189.stdout:9/733: mkdir d9/dc/de9 0 2024-04-20T21:32:21.962 INFO:tasks.workunit.client.1.smithi189.stdout:6/826: unlink d8/da/dd/d35/d6d/d7b/ce0 0 2024-04-20T21:32:21.964 INFO:tasks.workunit.client.1.smithi189.stdout:3/794: creat d2/d1c/dc1/dc5/ddf/f117 x:0 0 0 2024-04-20T21:32:21.966 INFO:tasks.workunit.client.1.smithi189.stdout:9/734: mkdir d9/dc/d17/d95/da6/dea 0 2024-04-20T21:32:21.971 INFO:tasks.workunit.client.1.smithi189.stdout:8/871: dread da/d81/d123/d3c/d50/f6d [0,4194304] 0 2024-04-20T21:32:21.972 INFO:tasks.workunit.client.1.smithi189.stdout:3/795: readlink d2/d3/d2c/d30/d3f/ld1 0 2024-04-20T21:32:21.975 INFO:tasks.workunit.client.1.smithi189.stdout:5/824: sync 2024-04-20T21:32:21.975 INFO:tasks.workunit.client.1.smithi189.stdout:7/864: sync 2024-04-20T21:32:21.975 INFO:tasks.workunit.client.1.smithi189.stdout:0/874: sync 2024-04-20T21:32:21.975 INFO:tasks.workunit.client.1.smithi189.stdout:4/739: sync 2024-04-20T21:32:21.980 INFO:tasks.workunit.client.1.smithi189.stdout:0/875: mknod d3/db9/dbe/c128 0 2024-04-20T21:32:21.982 INFO:tasks.workunit.client.1.smithi189.stdout:8/872: creat da/d81/d123/d13/d63/d110/f133 x:0 0 0 2024-04-20T21:32:21.982 INFO:tasks.workunit.client.1.smithi189.stdout:1/744: dwrite d8/d21/fa5 [0,4194304] 0 2024-04-20T21:32:21.989 INFO:tasks.workunit.client.1.smithi189.stdout:4/740: mkdir d2/dd/d37/dcd/d94/d107 0 2024-04-20T21:32:21.991 INFO:tasks.workunit.client.1.smithi189.stdout:0/876: mkdir d3/df/d30/d37/dbd/de1/d129 0 2024-04-20T21:32:21.991 INFO:tasks.workunit.client.1.smithi189.stdout:1/745: fsync d8/d1e/f23 0 2024-04-20T21:32:21.992 INFO:tasks.workunit.client.1.smithi189.stdout:3/796: truncate d2/d3/f48 589065 0 2024-04-20T21:32:21.997 INFO:tasks.workunit.client.1.smithi189.stdout:4/741: creat d2/d3a/ddc/f108 x:0 0 0 2024-04-20T21:32:22.000 INFO:tasks.workunit.client.1.smithi189.stdout:7/865: rename d5/d21/d4a/d52/ccd to d5/d1d/d64/c116 0 2024-04-20T21:32:22.002 INFO:tasks.workunit.client.1.smithi189.stdout:4/742: chown d2/d3a/ddc/dfc 53922190 1 2024-04-20T21:32:22.004 INFO:tasks.workunit.client.1.smithi189.stdout:0/877: symlink d3/db9/dbe/l12a 0 2024-04-20T21:32:22.005 INFO:tasks.workunit.client.1.smithi189.stdout:2/754: dread d1/d11/d2d/d4f/db4/f53 [0,4194304] 0 2024-04-20T21:32:22.006 INFO:tasks.workunit.client.1.smithi189.stdout:9/735: dwrite d9/d12/d3b/db1/d1a/f27 [0,4194304] 0 2024-04-20T21:32:22.007 INFO:tasks.workunit.client.1.smithi189.stdout:7/866: write d5/d1d/ded/f112 [272501,58339] 0 2024-04-20T21:32:22.011 INFO:tasks.workunit.client.1.smithi189.stdout:1/746: write d8/d9b/fa6 [2620700,61455] 0 2024-04-20T21:32:22.012 INFO:tasks.workunit.client.1.smithi189.stdout:4/743: symlink d2/dd/d37/dcd/l109 0 2024-04-20T21:32:22.014 INFO:tasks.workunit.client.1.smithi189.stdout:6/827: dwrite d8/d2b/d49/d54/fe7 [0,4194304] 0 2024-04-20T21:32:22.017 INFO:tasks.workunit.client.1.smithi189.stdout:9/736: symlink d9/d12/d62/dae/leb 0 2024-04-20T21:32:22.027 INFO:tasks.workunit.client.1.smithi189.stdout:5/825: dwrite d7/d9/d30/f2d [4194304,4194304] 0 2024-04-20T21:32:22.027 INFO:tasks.workunit.client.1.smithi189.stdout:0/878: dread d3/f102 [0,4194304] 0 2024-04-20T21:32:22.028 INFO:tasks.workunit.client.1.smithi189.stdout:2/755: symlink d1/d11/d2d/dab/d4a/d66/ldd 0 2024-04-20T21:32:22.029 INFO:tasks.workunit.client.1.smithi189.stdout:8/873: dwrite da/d81/d123/d3c/d9d/fec [0,4194304] 0 2024-04-20T21:32:22.030 INFO:tasks.workunit.client.1.smithi189.stdout:9/737: stat d9/d12/d62/dae/fb2 0 2024-04-20T21:32:22.030 INFO:tasks.workunit.client.0.smithi062.stdout:0/951: sync 2024-04-20T21:32:22.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:21 smithi062 ceph-mon[16221]: pgmap v25: 65 pgs: 65 active+clean; 3.3 GiB data, 9.6 GiB used, 527 GiB / 536 GiB avail; 84 MiB/s rd, 196 MiB/s wr, 442 op/s 2024-04-20T21:32:22.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:21 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:32:22.038 INFO:tasks.workunit.client.1.smithi189.stdout:9/738: write d9/dc/f1c [7715728,116142] 0 2024-04-20T21:32:22.042 INFO:tasks.workunit.client.1.smithi189.stdout:8/874: chown da/d81/d123/d13/d31/f80 96775554 1 2024-04-20T21:32:22.042 INFO:tasks.workunit.client.1.smithi189.stdout:7/867: dread d5/d46/f8f [0,4194304] 0 2024-04-20T21:32:22.047 INFO:tasks.workunit.client.1.smithi189.stdout:9/739: symlink d9/dc/d17/d95/da0/d55/dd0/lec 0 2024-04-20T21:32:22.047 INFO:tasks.workunit.client.1.smithi189.stdout:2/756: write d1/d11/d2d/d4f/d92/fd1 [386347,34868] 0 2024-04-20T21:32:22.054 INFO:tasks.workunit.client.1.smithi189.stdout:0/879: getdents d3/d8c 0 2024-04-20T21:32:22.059 INFO:tasks.workunit.client.1.smithi189.stdout:2/757: symlink d1/d11/lde 0 2024-04-20T21:32:22.059 INFO:tasks.workunit.client.1.smithi189.stdout:8/875: read f3 [2622269,103271] 0 2024-04-20T21:32:22.060 INFO:tasks.workunit.client.1.smithi189.stdout:9/740: dread d9/dc/d69/f7c [0,4194304] 0 2024-04-20T21:32:22.062 INFO:tasks.workunit.client.1.smithi189.stdout:6/828: dwrite d8/d2b/f87 [4194304,4194304] 0 2024-04-20T21:32:22.062 INFO:tasks.workunit.client.1.smithi189.stdout:0/880: stat d3/df/d28/d2b/d2c/f62 0 2024-04-20T21:32:22.065 INFO:tasks.workunit.client.1.smithi189.stdout:1/747: write d8/d1e/d54/d93/fbd [138031,68684] 0 2024-04-20T21:32:22.068 INFO:tasks.workunit.client.1.smithi189.stdout:0/881: write d3/d8c/f8d [2831005,98871] 0 2024-04-20T21:32:22.068 INFO:tasks.workunit.client.0.smithi062.stdout:0/952: write d7/d1e/da9/dbe/f104 [749800,62289] 0 2024-04-20T21:32:22.070 INFO:tasks.workunit.client.1.smithi189.stdout:6/829: truncate d8/d10/d78/d7e/dcf/fe1 218145 0 2024-04-20T21:32:22.071 INFO:tasks.workunit.client.1.smithi189.stdout:8/876: symlink da/d1f/l134 0 2024-04-20T21:32:22.080 INFO:tasks.workunit.client.0.smithi062.stdout:6/916: sync 2024-04-20T21:32:22.080 INFO:tasks.workunit.client.0.smithi062.stdout:3/932: sync 2024-04-20T21:32:22.083 INFO:tasks.workunit.client.1.smithi189.stdout:5/826: write d7/d9/d28/f9f [625457,88870] 0 2024-04-20T21:32:22.085 INFO:tasks.workunit.client.1.smithi189.stdout:7/868: truncate d5/d1d/d77/fc1 5422 0 2024-04-20T21:32:22.088 INFO:tasks.workunit.client.1.smithi189.stdout:8/877: write da/d81/d123/d22/f126 [167065,128658] 0 2024-04-20T21:32:22.090 INFO:tasks.workunit.client.1.smithi189.stdout:6/830: truncate d8/d10/f2d 3043268 0 2024-04-20T21:32:22.091 INFO:tasks.workunit.client.0.smithi062.stdout:0/953: link d7/d1e/da9/fac d7/d1c/d15/d17/d1b/d106/df1/f127 0 2024-04-20T21:32:22.094 INFO:tasks.workunit.client.1.smithi189.stdout:4/744: dwrite d2/d22/f45 [0,4194304] 0 2024-04-20T21:32:22.094 INFO:tasks.workunit.client.0.smithi062.stdout:6/917: truncate d0/d9/d11/d7b/feb 1012010 0 2024-04-20T21:32:22.097 INFO:tasks.workunit.client.0.smithi062.stdout:6/918: mknod d0/d9a/c12e 0 2024-04-20T21:32:22.098 INFO:tasks.workunit.client.0.smithi062.stdout:0/954: truncate d7/d1c/d15/d17/d37/d95/fd6 1593191 0 2024-04-20T21:32:22.102 INFO:tasks.workunit.client.0.smithi062.stdout:0/955: fdatasync d7/d20/d62/de3/f10a 0 2024-04-20T21:32:22.103 INFO:tasks.workunit.client.1.smithi189.stdout:7/869: creat d5/db4/d71/de4/f117 x:0 0 0 2024-04-20T21:32:22.103 INFO:tasks.workunit.client.0.smithi062.stdout:6/919: symlink d0/dc7/d7c/dc5/def/l12f 0 2024-04-20T21:32:22.104 INFO:tasks.workunit.client.0.smithi062.stdout:3/933: write d0/d1c/fce [1550635,7234] 0 2024-04-20T21:32:22.105 INFO:tasks.workunit.client.1.smithi189.stdout:1/748: dwrite d8/df/f15 [4194304,4194304] 0 2024-04-20T21:32:22.106 INFO:tasks.workunit.client.0.smithi062.stdout:6/920: write d0/d1a/d44/da7/f100 [3228913,12974] 0 2024-04-20T21:32:22.108 INFO:tasks.workunit.client.0.smithi062.stdout:0/956: symlink d7/d1e/db6/l128 0 2024-04-20T21:32:22.114 INFO:tasks.workunit.client.0.smithi062.stdout:3/934: mkdir d0/d1c/d133 0 2024-04-20T21:32:22.116 INFO:tasks.workunit.client.1.smithi189.stdout:6/831: mknod d8/da/dd/d35/c117 0 2024-04-20T21:32:22.116 INFO:tasks.workunit.client.1.smithi189.stdout:7/870: write d5/d1d/d64/db9/ddd/f103 [3556750,65933] 0 2024-04-20T21:32:22.117 INFO:tasks.workunit.client.1.smithi189.stdout:2/758: dwrite d1/d11/d52/fbd [0,4194304] 0 2024-04-20T21:32:22.118 INFO:tasks.workunit.client.0.smithi062.stdout:0/957: symlink d7/d1c/d15/d17/d37/d95/l129 0 2024-04-20T21:32:22.120 INFO:tasks.workunit.client.1.smithi189.stdout:0/882: dwrite d3/fe [0,4194304] 0 2024-04-20T21:32:22.124 INFO:tasks.workunit.client.0.smithi062.stdout:6/921: mknod d0/d9/d11/d22/dd5/d67/d75/df4/c130 0 2024-04-20T21:32:22.124 INFO:tasks.workunit.client.1.smithi189.stdout:8/878: mknod da/d81/d123/d3c/d68/d8b/dac/de1/c135 0 2024-04-20T21:32:22.127 INFO:tasks.workunit.client.1.smithi189.stdout:9/741: dread d9/dc/f3d [0,4194304] 0 2024-04-20T21:32:22.128 INFO:tasks.workunit.client.1.smithi189.stdout:7/871: write d5/d1d/d64/fd7 [2173653,76582] 0 2024-04-20T21:32:22.129 INFO:tasks.workunit.client.1.smithi189.stdout:4/745: mkdir d2/d22/d87/de1/d10a 0 2024-04-20T21:32:22.134 INFO:tasks.workunit.client.1.smithi189.stdout:3/797: sync 2024-04-20T21:32:22.136 INFO:tasks.workunit.client.1.smithi189.stdout:2/759: unlink d1/d11/d2d/d4f/fd9 0 2024-04-20T21:32:22.136 INFO:tasks.workunit.client.1.smithi189.stdout:8/879: write da/d81/d123/d13/dd2/fbe [863244,20480] 0 2024-04-20T21:32:22.138 INFO:tasks.workunit.client.0.smithi062.stdout:3/935: mknod d0/c134 0 2024-04-20T21:32:22.140 INFO:tasks.workunit.client.1.smithi189.stdout:7/872: creat d5/db4/d71/d7a/db0/f118 x:0 0 0 2024-04-20T21:32:22.142 INFO:tasks.workunit.client.0.smithi062.stdout:0/958: unlink d7/d1c/d15/d53/l92 0 2024-04-20T21:32:22.144 INFO:tasks.workunit.client.1.smithi189.stdout:4/746: chown d2/dd/d37/dcd/d6a/ld2 195 1 2024-04-20T21:32:22.144 INFO:tasks.workunit.client.1.smithi189.stdout:1/749: dwrite d8/d1e/fc5 [0,4194304] 0 2024-04-20T21:32:22.144 INFO:tasks.workunit.client.0.smithi062.stdout:3/936: chown d0/d1/d10/d99 6802014 1 2024-04-20T21:32:22.144 INFO:tasks.workunit.client.1.smithi189.stdout:9/742: symlink d9/dc/d69/led 0 2024-04-20T21:32:22.145 INFO:tasks.workunit.client.0.smithi062.stdout:6/922: dwrite d0/d1a/d44/f4c [0,4194304] 0 2024-04-20T21:32:22.145 INFO:tasks.workunit.client.1.smithi189.stdout:6/832: rename d8/d12 to d8/da/dd/d118 0 2024-04-20T21:32:22.147 INFO:tasks.workunit.client.0.smithi062.stdout:0/959: mknod d7/d1c/d103/c12a 0 2024-04-20T21:32:22.148 INFO:tasks.workunit.client.1.smithi189.stdout:8/880: symlink da/d81/d123/d3c/deb/l136 0 2024-04-20T21:32:22.155 INFO:tasks.workunit.client.1.smithi189.stdout:7/873: symlink d5/l119 0 2024-04-20T21:32:22.156 INFO:tasks.workunit.client.1.smithi189.stdout:4/747: symlink d2/d7f/l10b 0 2024-04-20T21:32:22.159 INFO:tasks.workunit.client.1.smithi189.stdout:2/760: creat d1/d17/d37/d89/dc5/fdf x:0 0 0 2024-04-20T21:32:22.160 INFO:tasks.workunit.client.1.smithi189.stdout:5/827: dwrite d7/d9/d30/d5a/f64 [0,4194304] 0 2024-04-20T21:32:22.160 INFO:tasks.workunit.client.1.smithi189.stdout:2/761: fsync d1/d17/fd6 0 2024-04-20T21:32:22.161 INFO:tasks.workunit.client.0.smithi062.stdout:6/923: rename d0/d9/d11/d22/dd5/d67/d75/df4/c130 to d0/d1a/d7d/db1/db9/c131 0 2024-04-20T21:32:22.166 INFO:tasks.workunit.client.0.smithi062.stdout:0/960: creat d7/d1c/d15/d17/d1b/d106/db1/f12b x:0 0 0 2024-04-20T21:32:22.167 INFO:tasks.workunit.client.1.smithi189.stdout:8/881: dread da/de/d1e/d69/f10c [0,4194304] 0 2024-04-20T21:32:22.167 INFO:tasks.workunit.client.1.smithi189.stdout:1/750: mkdir d8/df/d3a/dfd 0 2024-04-20T21:32:22.168 INFO:tasks.workunit.client.0.smithi062.stdout:3/937: creat d0/d1/de4/d118/f135 x:0 0 0 2024-04-20T21:32:22.169 INFO:tasks.workunit.client.0.smithi062.stdout:0/961: read d7/f97 [1530941,5136] 0 2024-04-20T21:32:22.169 INFO:tasks.workunit.client.1.smithi189.stdout:2/762: fsync d1/d11/d2d/fa6 0 2024-04-20T21:32:22.172 INFO:tasks.workunit.client.1.smithi189.stdout:0/883: dwrite d3/df/d28/dc8/df1/ff5 [0,4194304] 0 2024-04-20T21:32:22.174 INFO:tasks.workunit.client.1.smithi189.stdout:2/763: chown d1/d11/f15 7125654 1 2024-04-20T21:32:22.176 INFO:tasks.workunit.client.1.smithi189.stdout:9/743: creat d9/dc/fee x:0 0 0 2024-04-20T21:32:22.179 INFO:tasks.workunit.client.1.smithi189.stdout:3/798: link d2/d3/d2c/d31/caa d2/d3/d2c/d31/d4c/c118 0 2024-04-20T21:32:22.181 INFO:tasks.workunit.client.0.smithi062.stdout:0/962: fdatasync d7/d1c/d15/d17/f38 0 2024-04-20T21:32:22.183 INFO:tasks.workunit.client.1.smithi189.stdout:0/884: creat d3/df/d28/d2b/db1/d111/f12b x:0 0 0 2024-04-20T21:32:22.188 INFO:tasks.workunit.client.1.smithi189.stdout:9/744: fdatasync d9/dc/f4d 0 2024-04-20T21:32:22.188 INFO:tasks.workunit.client.0.smithi062.stdout:3/938: dwrite d0/d1c/d63/d42/f121 [0,4194304] 0 2024-04-20T21:32:22.199 INFO:tasks.workunit.client.1.smithi189.stdout:2/764: rename d1/d11/d52/c7d to d1/d11/d2d/d4f/d8e/ce0 0 2024-04-20T21:32:22.200 INFO:tasks.workunit.client.1.smithi189.stdout:7/874: link d5/d21/d35/c4b d5/c11a 0 2024-04-20T21:32:22.202 INFO:tasks.workunit.client.0.smithi062.stdout:0/963: rename d7/d20/d62/d6f/c114 to d7/c12c 0 2024-04-20T21:32:22.208 INFO:tasks.workunit.client.0.smithi062.stdout:0/964: truncate d7/d1c/d15/d17/d26/fea 1334893 0 2024-04-20T21:32:22.209 INFO:tasks.workunit.client.1.smithi189.stdout:9/745: link d9/d12/cb5 d9/dc/d69/cef 0 2024-04-20T21:32:22.211 INFO:tasks.workunit.client.1.smithi189.stdout:8/882: dwrite da/de/d84/fbd [4194304,4194304] 0 2024-04-20T21:32:22.212 INFO:tasks.workunit.client.0.smithi062.stdout:3/939: rename d0/d1/d3c/d43/l114 to d0/d1c/d63/d42/d11e/l136 0 2024-04-20T21:32:22.218 INFO:tasks.workunit.client.1.smithi189.stdout:0/885: dwrite d3/df/d4c/dd4/fcd [0,4194304] 0 2024-04-20T21:32:22.218 INFO:tasks.workunit.client.1.smithi189.stdout:6/833: dread - d8/d2b/d49/d54/d67/fb8 zero size 2024-04-20T21:32:22.219 INFO:tasks.workunit.client.0.smithi062.stdout:3/940: rmdir d0/d1/d3c/d43/d72/dd3 39 2024-04-20T21:32:22.220 INFO:tasks.workunit.client.1.smithi189.stdout:8/883: mkdir da/de/d1e/d137 0 2024-04-20T21:32:22.220 INFO:tasks.workunit.client.1.smithi189.stdout:9/746: mkdir d9/d12/d62/dd9/df0 0 2024-04-20T21:32:22.222 INFO:tasks.workunit.client.1.smithi189.stdout:6/834: chown d8/d2b/l8b 1200199 1 2024-04-20T21:32:22.223 INFO:tasks.workunit.client.1.smithi189.stdout:2/765: getdents d1/d17/d37/d89 0 2024-04-20T21:32:22.224 INFO:tasks.workunit.client.1.smithi189.stdout:9/747: chown d9/d12/d62/dd9 3417977 1 2024-04-20T21:32:22.225 INFO:tasks.workunit.client.1.smithi189.stdout:8/884: mknod da/d81/d123/d13/dd6/dba/d11e/c138 0 2024-04-20T21:32:22.226 INFO:tasks.workunit.client.1.smithi189.stdout:0/886: write d3/df/d28/dc8/df1/ff5 [4749658,18008] 0 2024-04-20T21:32:22.228 INFO:tasks.workunit.client.0.smithi062.stdout:3/941: truncate d0/d1/d10/f79 13914 0 2024-04-20T21:32:22.232 INFO:tasks.workunit.client.1.smithi189.stdout:2/766: dread - d1/d11/d2d/d73/dc6/fdc zero size 2024-04-20T21:32:22.233 INFO:tasks.workunit.client.1.smithi189.stdout:9/748: unlink d9/d12/d62/dae/fe2 0 2024-04-20T21:32:22.235 INFO:tasks.workunit.client.1.smithi189.stdout:0/887: symlink d3/df/d30/d37/l12c 0 2024-04-20T21:32:22.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:21 smithi189 ceph-mon[21562]: pgmap v25: 65 pgs: 65 active+clean; 3.3 GiB data, 9.6 GiB used, 527 GiB / 536 GiB avail; 84 MiB/s rd, 196 MiB/s wr, 442 op/s 2024-04-20T21:32:22.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:21 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:32:22.238 INFO:tasks.workunit.client.1.smithi189.stdout:2/767: chown d1/d11/d2d/dab/d55/f94 23 1 2024-04-20T21:32:22.239 INFO:tasks.workunit.client.1.smithi189.stdout:0/888: readlink d3/df/d30/d37/l80 0 2024-04-20T21:32:22.240 INFO:tasks.workunit.client.1.smithi189.stdout:3/799: getdents d2/d3/d2c/d31/d4c 0 2024-04-20T21:32:22.243 INFO:tasks.workunit.client.1.smithi189.stdout:0/889: readlink d3/df/d5d/lcc 0 2024-04-20T21:32:22.243 INFO:tasks.workunit.client.1.smithi189.stdout:9/749: creat d9/d12/d62/ff1 x:0 0 0 2024-04-20T21:32:22.246 INFO:tasks.workunit.client.1.smithi189.stdout:1/751: dread d8/d1e/f87 [0,4194304] 0 2024-04-20T21:32:22.249 INFO:tasks.workunit.client.1.smithi189.stdout:0/890: write d3/df/d4c/f69 [6694823,126408] 0 2024-04-20T21:32:22.249 INFO:tasks.workunit.client.1.smithi189.stdout:8/885: link da/d81/d123/d20/d65/f112 da/d81/d123/d22/f139 0 2024-04-20T21:32:22.250 INFO:tasks.workunit.client.1.smithi189.stdout:9/750: rmdir d9/dc/d17/d95/da0/d55/dd0 39 2024-04-20T21:32:22.252 INFO:tasks.workunit.client.1.smithi189.stdout:0/891: stat d3/df/d28/d2b/db1/f107 0 2024-04-20T21:32:22.255 INFO:tasks.workunit.client.1.smithi189.stdout:8/886: fdatasync da/d81/d123/d3c/d50/d6c/dc7/fcd 0 2024-04-20T21:32:22.257 INFO:tasks.workunit.client.1.smithi189.stdout:9/751: mkdir d9/dc/d17/d95/da0/df2 0 2024-04-20T21:32:22.261 INFO:tasks.workunit.client.1.smithi189.stdout:8/887: write da/d81/d123/d3c/d68/d8b/fcf [1141988,43047] 0 2024-04-20T21:32:22.265 INFO:tasks.workunit.client.1.smithi189.stdout:9/752: write d9/dc/d17/d95/da0/d52/d82/fa7 [216989,61772] 0 2024-04-20T21:32:22.271 INFO:tasks.workunit.client.0.smithi062.stdout:0/965: write d7/d1c/d15/d17/f72 [3888745,43067] 0 2024-04-20T21:32:22.273 INFO:tasks.workunit.client.1.smithi189.stdout:9/753: write d9/d12/d3f/d72/fde [232754,18003] 0 2024-04-20T21:32:22.275 INFO:tasks.workunit.client.1.smithi189.stdout:9/754: fdatasync f7 0 2024-04-20T21:32:22.278 INFO:tasks.workunit.client.0.smithi062.stdout:0/966: mkdir d7/d1c/d15/d53/d12d 0 2024-04-20T21:32:22.279 INFO:tasks.workunit.client.1.smithi189.stdout:9/755: read - d9/d12/d3b/fe0 zero size 2024-04-20T21:32:22.281 INFO:tasks.workunit.client.1.smithi189.stdout:6/835: write d8/d10/d78/f80 [1842948,109823] 0 2024-04-20T21:32:22.281 INFO:tasks.workunit.client.0.smithi062.stdout:3/942: dwrite d0/d1b/f57 [0,4194304] 0 2024-04-20T21:32:22.288 INFO:tasks.workunit.client.0.smithi062.stdout:0/967: symlink d7/d1c/d15/d53/d10b/l12e 0 2024-04-20T21:32:22.291 INFO:tasks.workunit.client.1.smithi189.stdout:0/892: dwrite d3/df/d30/d45/dd3/fee [0,4194304] 0 2024-04-20T21:32:22.293 INFO:tasks.workunit.client.1.smithi189.stdout:7/875: dwrite d5/d46/f8b [0,4194304] 0 2024-04-20T21:32:22.299 INFO:tasks.workunit.client.1.smithi189.stdout:6/836: unlink d8/da/dd/d118/d4e/c6a 0 2024-04-20T21:32:22.306 INFO:tasks.workunit.client.1.smithi189.stdout:3/800: dread d2/d3/f19 [0,4194304] 0 2024-04-20T21:32:22.306 INFO:tasks.workunit.client.1.smithi189.stdout:0/893: mknod d3/df/d28/dc8/d10a/c12d 0 2024-04-20T21:32:22.306 INFO:tasks.workunit.client.1.smithi189.stdout:6/837: stat d8/d2b/d49/d54/d67/fb8 0 2024-04-20T21:32:22.306 INFO:tasks.workunit.client.0.smithi062.stdout:0/968: mkdir d7/d1c/d12f 0 2024-04-20T21:32:22.309 INFO:tasks.workunit.client.1.smithi189.stdout:8/888: dwrite da/d1f/f2f [0,4194304] 0 2024-04-20T21:32:22.312 INFO:tasks.workunit.client.0.smithi062.stdout:0/969: fsync d7/d1c/d103/f87 0 2024-04-20T21:32:22.315 INFO:tasks.workunit.client.1.smithi189.stdout:9/756: rename d9/d12/d3f/d7a to d9/dc/d17/d95/da0/d52/d82/da4/df3 0 2024-04-20T21:32:22.316 INFO:tasks.workunit.client.0.smithi062.stdout:0/970: truncate d7/d1e/da9/f120 534398 0 2024-04-20T21:32:22.320 INFO:tasks.workunit.client.1.smithi189.stdout:0/894: chown d3/df/d4c/dd4/fe6 5557 1 2024-04-20T21:32:22.320 INFO:tasks.workunit.client.1.smithi189.stdout:8/889: dread - da/d81/d123/d20/d65/f112 zero size 2024-04-20T21:32:22.320 INFO:tasks.workunit.client.1.smithi189.stdout:7/876: mkdir d5/d46/d7d/d111/d11b 0 2024-04-20T21:32:22.323 INFO:tasks.workunit.client.1.smithi189.stdout:1/752: dread d8/d1e/d43/d44/fc9 [0,4194304] 0 2024-04-20T21:32:22.326 INFO:tasks.workunit.client.1.smithi189.stdout:6/838: mkdir d8/da/dd/d118/d119 0 2024-04-20T21:32:22.329 INFO:tasks.workunit.client.1.smithi189.stdout:8/890: creat da/de/d1e/d21/f13a x:0 0 0 2024-04-20T21:32:22.330 INFO:tasks.workunit.client.1.smithi189.stdout:9/757: creat d9/d12/d62/dd9/df0/ff4 x:0 0 0 2024-04-20T21:32:22.332 INFO:tasks.workunit.client.1.smithi189.stdout:7/877: symlink d5/d1d/l11c 0 2024-04-20T21:32:22.334 INFO:tasks.workunit.client.1.smithi189.stdout:6/839: dread - d8/d2b/fbb zero size 2024-04-20T21:32:22.335 INFO:tasks.workunit.client.1.smithi189.stdout:1/753: fdatasync d8/d21/fb3 0 2024-04-20T21:32:22.340 INFO:tasks.workunit.client.0.smithi062.stdout:0/971: dwrite d7/d1c/fc9 [0,4194304] 0 2024-04-20T21:32:22.342 INFO:tasks.workunit.client.1.smithi189.stdout:3/801: dwrite d2/d3/d88/ff6 [0,4194304] 0 2024-04-20T21:32:22.343 INFO:tasks.workunit.client.1.smithi189.stdout:1/754: fsync d8/d21/d2c/f75 0 2024-04-20T21:32:22.343 INFO:tasks.workunit.client.1.smithi189.stdout:0/895: dread d3/df/d28/d2b/d2c/f9e [4194304,4194304] 0 2024-04-20T21:32:22.344 INFO:tasks.workunit.client.1.smithi189.stdout:6/840: chown d8/da/dd/d2a/f66 1128593 1 2024-04-20T21:32:22.349 INFO:tasks.workunit.client.1.smithi189.stdout:4/748: sync 2024-04-20T21:32:22.350 INFO:tasks.workunit.client.1.smithi189.stdout:8/891: mkdir da/d81/d123/d3a/dfe/d13b 0 2024-04-20T21:32:22.355 INFO:tasks.workunit.client.0.smithi062.stdout:0/972: dwrite d7/d1e/db6/fee [0,4194304] 0 2024-04-20T21:32:22.355 INFO:tasks.workunit.client.1.smithi189.stdout:7/878: write d5/d21/d25/d34/f6f [3270630,71128] 0 2024-04-20T21:32:22.357 INFO:tasks.workunit.client.1.smithi189.stdout:4/749: chown d2/f4 827 1 2024-04-20T21:32:22.359 INFO:tasks.workunit.client.1.smithi189.stdout:8/892: stat da/d81/d123/d13/dd6 0 2024-04-20T21:32:22.362 INFO:tasks.workunit.client.1.smithi189.stdout:6/841: creat d8/da/dd/d2a/d3b/f11a x:0 0 0 2024-04-20T21:32:22.365 INFO:tasks.workunit.client.1.smithi189.stdout:0/896: symlink d3/df/d28/d2b/db1/dfe/d119/l12e 0 2024-04-20T21:32:22.368 INFO:tasks.workunit.client.1.smithi189.stdout:1/755: rename d8/d1e/d28/d4f to d8/df/d3a/d49/d8c/da2/dfe 0 2024-04-20T21:32:22.372 INFO:tasks.workunit.client.0.smithi062.stdout:0/973: write d7/d20/f10d [614122,66872] 0 2024-04-20T21:32:22.373 INFO:tasks.workunit.client.1.smithi189.stdout:8/893: symlink da/d81/d123/d13/d63/l13c 0 2024-04-20T21:32:22.374 INFO:tasks.workunit.client.0.smithi062.stdout:6/924: sync 2024-04-20T21:32:22.374 INFO:tasks.workunit.client.1.smithi189.stdout:7/879: rename d5/d21/d25/l29 to d5/d21/d4a/dfd/l11d 0 2024-04-20T21:32:22.377 INFO:tasks.workunit.client.0.smithi062.stdout:0/974: fsync d7/d1e/fdb 0 2024-04-20T21:32:22.380 INFO:tasks.workunit.client.1.smithi189.stdout:8/894: truncate da/de/d1e/d21/d44/f4a 5020534 0 2024-04-20T21:32:22.381 INFO:tasks.workunit.client.1.smithi189.stdout:6/842: mkdir d8/d10/d78/d7e/d11b 0 2024-04-20T21:32:22.381 INFO:tasks.workunit.client.0.smithi062.stdout:6/925: fdatasync d0/d9/d11/d22/dd5/d67/ffb 0 2024-04-20T21:32:22.384 INFO:tasks.workunit.client.1.smithi189.stdout:2/768: sync 2024-04-20T21:32:22.384 INFO:tasks.workunit.client.1.smithi189.stdout:5/828: sync 2024-04-20T21:32:22.386 INFO:tasks.workunit.client.0.smithi062.stdout:0/975: truncate d7/d1e/da9/dbe/ff6 1045839 0 2024-04-20T21:32:22.386 INFO:tasks.workunit.client.1.smithi189.stdout:8/895: chown da/d81/d123/d3c/d68/d8b/dac/l132 30514905 1 2024-04-20T21:32:22.386 INFO:tasks.workunit.client.1.smithi189.stdout:1/756: chown d8/d1e/d43/d44/db2/lcc 65624021 1 2024-04-20T21:32:22.387 INFO:tasks.workunit.client.0.smithi062.stdout:6/926: dread d0/d9/d11/ff7 [0,4194304] 0 2024-04-20T21:32:22.387 INFO:tasks.workunit.client.1.smithi189.stdout:5/829: fdatasync d7/d9/d30/f1c 0 2024-04-20T21:32:22.392 INFO:tasks.workunit.client.1.smithi189.stdout:4/750: symlink d2/dd/d37/dcd/d6a/d7a/db7/dfb/l10c 0 2024-04-20T21:32:22.392 INFO:tasks.workunit.client.1.smithi189.stdout:6/843: chown d8/da/dd/d118/dd7/c10f 1486091 1 2024-04-20T21:32:22.394 INFO:tasks.workunit.client.0.smithi062.stdout:0/976: fdatasync d7/d20/f5c 0 2024-04-20T21:32:22.397 INFO:tasks.workunit.client.1.smithi189.stdout:1/757: stat d8/df/d3a/dfd 0 2024-04-20T21:32:22.398 INFO:tasks.workunit.client.0.smithi062.stdout:6/927: dread d0/d9/f13 [0,4194304] 0 2024-04-20T21:32:22.399 INFO:tasks.workunit.client.0.smithi062.stdout:6/928: fsync d0/d1a/d44/db3/f106 0 2024-04-20T21:32:22.400 INFO:tasks.workunit.client.1.smithi189.stdout:0/897: dwrite d3/df/d30/fd2 [0,4194304] 0 2024-04-20T21:32:22.400 INFO:tasks.workunit.client.0.smithi062.stdout:0/977: mknod d7/d1c/d15/d53/d12d/c130 0 2024-04-20T21:32:22.405 INFO:tasks.workunit.client.1.smithi189.stdout:6/844: fdatasync d8/da/dd/d35/f45 0 2024-04-20T21:32:22.407 INFO:tasks.workunit.client.0.smithi062.stdout:0/978: symlink d7/ddf/l131 0 2024-04-20T21:32:22.409 INFO:tasks.workunit.client.0.smithi062.stdout:0/979: chown d7/d1e/c30 184864 1 2024-04-20T21:32:22.411 INFO:tasks.workunit.client.1.smithi189.stdout:5/830: rename d7/d9/d51/d70/d8a to d7/d9/d30/d55/d104/d106 0 2024-04-20T21:32:22.413 INFO:tasks.workunit.client.1.smithi189.stdout:9/758: dread d9/dc/d17/d95/da0/d56/f8a [0,4194304] 0 2024-04-20T21:32:22.418 INFO:tasks.workunit.client.1.smithi189.stdout:6/845: creat d8/d2b/d49/d108/f11c x:0 0 0 2024-04-20T21:32:22.420 INFO:tasks.workunit.client.1.smithi189.stdout:3/802: dread d2/d3/d2c/d31/d74/d7a/f7c [0,4194304] 0 2024-04-20T21:32:22.420 INFO:tasks.workunit.client.1.smithi189.stdout:0/898: truncate d3/df/d30/d37/ff2 256473 0 2024-04-20T21:32:22.422 INFO:tasks.workunit.client.0.smithi062.stdout:0/980: link d7/d20/d62/de3/ce5 d7/d1e/df9/d102/c132 0 2024-04-20T21:32:22.425 INFO:tasks.workunit.client.0.smithi062.stdout:0/981: truncate d7/ddf/fe8 4428671 0 2024-04-20T21:32:22.425 INFO:tasks.workunit.client.1.smithi189.stdout:5/831: mknod d7/c107 0 2024-04-20T21:32:22.425 INFO:tasks.workunit.client.1.smithi189.stdout:9/759: symlink d9/dc/d17/d95/da0/lf5 0 2024-04-20T21:32:22.425 INFO:tasks.workunit.client.1.smithi189.stdout:3/803: fsync d2/d1c/f2b 0 2024-04-20T21:32:22.427 INFO:tasks.workunit.client.1.smithi189.stdout:6/846: mknod d8/da/dd/d118/daf/c11d 0 2024-04-20T21:32:22.428 INFO:tasks.workunit.client.0.smithi062.stdout:6/929: write d0/d9/d11/d22/dd5/d67/fca [766962,14280] 0 2024-04-20T21:32:22.432 INFO:tasks.workunit.client.0.smithi062.stdout:0/982: mknod d7/d1e/db6/c133 0 2024-04-20T21:32:22.433 INFO:tasks.workunit.client.1.smithi189.stdout:5/832: creat d7/d9/d51/d70/f108 x:0 0 0 2024-04-20T21:32:22.434 INFO:tasks.workunit.client.1.smithi189.stdout:0/899: read d3/d26/f1d [1858376,106902] 0 2024-04-20T21:32:22.437 INFO:tasks.workunit.client.1.smithi189.stdout:8/896: dread f3 [4194304,4194304] 0 2024-04-20T21:32:22.437 INFO:tasks.workunit.client.0.smithi062.stdout:0/983: unlink d7/d1e/db6/ccd 0 2024-04-20T21:32:22.438 INFO:tasks.workunit.client.1.smithi189.stdout:9/760: getdents d9/dc/de9 0 2024-04-20T21:32:22.438 INFO:tasks.workunit.client.1.smithi189.stdout:7/880: dwrite d5/d21/d4a/d52/f5c [0,4194304] 0 2024-04-20T21:32:22.439 INFO:tasks.workunit.client.1.smithi189.stdout:3/804: rmdir d2/d3/d2c/d31/d90 39 2024-04-20T21:32:22.447 INFO:tasks.workunit.client.0.smithi062.stdout:0/984: creat d7/d1c/d103/f134 x:0 0 0 2024-04-20T21:32:22.447 INFO:tasks.workunit.client.0.smithi062.stdout:3/943: sync 2024-04-20T21:32:22.452 INFO:tasks.workunit.client.1.smithi189.stdout:9/761: rmdir d9/dc/d17/d95/da0/df2 0 2024-04-20T21:32:22.454 INFO:tasks.workunit.client.0.smithi062.stdout:0/985: dread - d7/d1c/d15/d17/d1b/d106/fba zero size 2024-04-20T21:32:22.454 INFO:tasks.workunit.client.0.smithi062.stdout:3/944: creat d0/d1/d10/d99/f137 x:0 0 0 2024-04-20T21:32:22.455 INFO:tasks.workunit.client.1.smithi189.stdout:8/897: getdents da/d81/d123/d20/d2c/d9a/dce 0 2024-04-20T21:32:22.457 INFO:tasks.workunit.client.1.smithi189.stdout:9/762: chown d9/dc/d17/d7e/de8 1028 1 2024-04-20T21:32:22.460 INFO:tasks.workunit.client.0.smithi062.stdout:0/986: mknod d7/d1c/d15/d17/d37/d109/c135 0 2024-04-20T21:32:22.461 INFO:tasks.workunit.client.1.smithi189.stdout:9/763: readlink d9/l5b 0 2024-04-20T21:32:22.462 INFO:tasks.workunit.client.0.smithi062.stdout:3/945: mkdir d0/d1/d10/d10b/d138 0 2024-04-20T21:32:22.464 INFO:tasks.workunit.client.0.smithi062.stdout:3/946: fsync d0/d1/d3c/d43/d72/da9/fab 0 2024-04-20T21:32:22.468 INFO:tasks.workunit.client.1.smithi189.stdout:6/847: dwrite d8/d10/d78/d7e/dcf/ff4 [0,4194304] 0 2024-04-20T21:32:22.470 INFO:tasks.workunit.client.1.smithi189.stdout:9/764: chown d9/dc/d17/d95/da6 850022 1 2024-04-20T21:32:22.470 INFO:tasks.workunit.client.0.smithi062.stdout:0/987: rename d7/d1c/f126 to d7/d1c/d15/d17/d37/d95/f136 0 2024-04-20T21:32:22.472 INFO:tasks.workunit.client.0.smithi062.stdout:3/947: getdents d0/d1/d10/d31/d41/de6/d102 0 2024-04-20T21:32:22.473 INFO:tasks.workunit.client.0.smithi062.stdout:0/988: chown d7/d1c/d15/d17/d37/d109/d11b/fc7 6929 1 2024-04-20T21:32:22.474 INFO:tasks.workunit.client.1.smithi189.stdout:8/898: mknod da/d81/d123/d13/dd6/dba/dc5/c13d 0 2024-04-20T21:32:22.475 INFO:tasks.workunit.client.0.smithi062.stdout:3/948: write d0/d5c/f5a [6017157,100395] 0 2024-04-20T21:32:22.476 INFO:tasks.workunit.client.1.smithi189.stdout:4/751: dread d2/dd/d11/f92 [0,4194304] 0 2024-04-20T21:32:22.477 INFO:tasks.workunit.client.0.smithi062.stdout:0/989: chown d7/d20/l2f 2215310 1 2024-04-20T21:32:22.480 INFO:tasks.workunit.client.1.smithi189.stdout:0/900: dwrite d3/df/d28/dc8/df1/ff5 [4194304,4194304] 0 2024-04-20T21:32:22.482 INFO:tasks.workunit.client.1.smithi189.stdout:1/758: dread d8/d1e/d28/fa8 [0,4194304] 0 2024-04-20T21:32:22.489 INFO:tasks.workunit.client.0.smithi062.stdout:3/949: fdatasync d0/f2f 0 2024-04-20T21:32:22.491 INFO:tasks.workunit.client.1.smithi189.stdout:8/899: symlink da/d81/d123/d13/d63/d7b/l13e 0 2024-04-20T21:32:22.493 INFO:tasks.workunit.client.1.smithi189.stdout:7/881: dwrite d5/db4/d71/de4/f117 [0,4194304] 0 2024-04-20T21:32:22.497 INFO:tasks.workunit.client.1.smithi189.stdout:6/848: mkdir d8/da/dd/d118/dd7/d11e 0 2024-04-20T21:32:22.497 INFO:tasks.workunit.client.1.smithi189.stdout:0/901: creat d3/d8c/f12f x:0 0 0 2024-04-20T21:32:22.497 INFO:tasks.workunit.client.0.smithi062.stdout:0/990: dwrite d7/d1c/f18 [4194304,4194304] 0 2024-04-20T21:32:22.500 INFO:tasks.workunit.client.1.smithi189.stdout:4/752: getdents d2/dd/d37/dcd/df9 0 2024-04-20T21:32:22.502 INFO:tasks.workunit.client.0.smithi062.stdout:3/950: rename d0/d5c/d76 to d0/d1/d10/d31/d41/de6/d139 0 2024-04-20T21:32:22.504 INFO:tasks.workunit.client.1.smithi189.stdout:0/902: fdatasync d3/d8c/fc7 0 2024-04-20T21:32:22.505 INFO:tasks.workunit.client.1.smithi189.stdout:1/759: creat d8/d1e/d43/d44/dd2/db7/dcf/fff x:0 0 0 2024-04-20T21:32:22.508 INFO:tasks.workunit.client.1.smithi189.stdout:7/882: fdatasync d5/d21/f109 0 2024-04-20T21:32:22.511 INFO:tasks.workunit.client.0.smithi062.stdout:3/951: creat d0/d1/d10/d31/d41/f13a x:0 0 0 2024-04-20T21:32:22.513 INFO:tasks.workunit.client.1.smithi189.stdout:8/900: write da/d1f/fa1 [1245091,102110] 0 2024-04-20T21:32:22.514 INFO:tasks.workunit.client.1.smithi189.stdout:6/849: mkdir d8/d2b/d49/d54/d6e/d11f 0 2024-04-20T21:32:22.517 INFO:tasks.workunit.client.1.smithi189.stdout:3/805: dwrite d2/d8/f4b [0,4194304] 0 2024-04-20T21:32:22.519 INFO:tasks.workunit.client.1.smithi189.stdout:5/833: dread d7/d9/d30/d55/d61/d87/f90 [0,4194304] 0 2024-04-20T21:32:22.521 INFO:tasks.workunit.client.1.smithi189.stdout:7/883: creat d5/d1d/d64/db9/dcf/f11e x:0 0 0 2024-04-20T21:32:22.526 INFO:tasks.workunit.client.1.smithi189.stdout:3/806: read d2/d3/d2c/d31/d4c/f63 [184985,108309] 0 2024-04-20T21:32:22.527 INFO:tasks.workunit.client.1.smithi189.stdout:9/765: dwrite d9/d12/d3b/db1/d1a/f9d [0,4194304] 0 2024-04-20T21:32:22.531 INFO:tasks.workunit.client.0.smithi062.stdout:3/952: stat d0/d1/d3c/d43/d72/dd3/dc5/lf1 0 2024-04-20T21:32:22.533 INFO:tasks.workunit.client.1.smithi189.stdout:3/807: fsync d2/d3/d2c/d31/d74/db7/dd3/fe8 0 2024-04-20T21:32:22.534 INFO:tasks.workunit.client.1.smithi189.stdout:8/901: truncate da/d81/d123/d20/f99 471096 0 2024-04-20T21:32:22.539 INFO:tasks.workunit.client.1.smithi189.stdout:9/766: symlink d9/dc/d17/d36/d5d/lf6 0 2024-04-20T21:32:22.540 INFO:tasks.workunit.client.1.smithi189.stdout:2/769: sync 2024-04-20T21:32:22.543 INFO:tasks.workunit.client.1.smithi189.stdout:5/834: symlink d7/d9/d30/l109 0 2024-04-20T21:32:22.543 INFO:tasks.workunit.client.1.smithi189.stdout:1/760: dread d8/d1e/fc5 [0,4194304] 0 2024-04-20T21:32:22.545 INFO:tasks.workunit.client.1.smithi189.stdout:9/767: chown d9/d12/d3b/ce5 10761564 1 2024-04-20T21:32:22.551 INFO:tasks.workunit.client.1.smithi189.stdout:2/770: write d1/d11/d2d/dab/d55/fba [774847,123908] 0 2024-04-20T21:32:22.551 INFO:tasks.workunit.client.1.smithi189.stdout:8/902: getdents da/d81/d91 0 2024-04-20T21:32:22.556 INFO:tasks.workunit.client.1.smithi189.stdout:9/768: fdatasync f4 0 2024-04-20T21:32:22.558 INFO:tasks.workunit.client.1.smithi189.stdout:3/808: dread d2/d1c/dc1/dc5/fcc [0,4194304] 0 2024-04-20T21:32:22.558 INFO:tasks.workunit.client.0.smithi062.stdout:0/991: write d7/d1c/d15/d17/d40/d79/fb0 [8966504,114616] 0 2024-04-20T21:32:22.562 INFO:tasks.workunit.client.0.smithi062.stdout:3/953: dread d0/d1/d115/d11c/fb6 [0,4194304] 0 2024-04-20T21:32:22.563 INFO:tasks.workunit.client.0.smithi062.stdout:3/954: chown d0/d1/de4/d118 40661 1 2024-04-20T21:32:22.565 INFO:tasks.workunit.client.0.smithi062.stdout:0/992: rmdir d7/d1c/d15/d17/d40/d79/d71 39 2024-04-20T21:32:22.566 INFO:tasks.workunit.client.1.smithi189.stdout:8/903: creat da/de/d1e/d69/f13f x:0 0 0 2024-04-20T21:32:22.566 INFO:tasks.workunit.client.0.smithi062.stdout:3/955: stat d0/d1/d3c/d43/d72/dd3/dc5/dd5 0 2024-04-20T21:32:22.567 INFO:tasks.workunit.client.1.smithi189.stdout:3/809: chown d2/d3/d65/f8b 51 1 2024-04-20T21:32:22.568 INFO:tasks.workunit.client.1.smithi189.stdout:2/771: fdatasync d1/d11/d58/f64 0 2024-04-20T21:32:22.569 INFO:tasks.workunit.client.1.smithi189.stdout:4/753: dread d2/d7f/fc1 [4194304,4194304] 0 2024-04-20T21:32:22.569 INFO:tasks.workunit.client.0.smithi062.stdout:0/993: read d7/d1e/da9/dbe/f104 [148545,64149] 0 2024-04-20T21:32:22.570 INFO:tasks.workunit.client.1.smithi189.stdout:9/769: chown d9/dc/d17/cc8 30 1 2024-04-20T21:32:22.573 INFO:tasks.workunit.client.0.smithi062.stdout:3/956: mkdir d0/d1/d10/d3a/d13b 0 2024-04-20T21:32:22.574 INFO:tasks.workunit.client.1.smithi189.stdout:8/904: dread - da/d81/d91/fc4 zero size 2024-04-20T21:32:22.575 INFO:tasks.workunit.client.1.smithi189.stdout:0/903: dwrite d3/d8c/fc7 [0,4194304] 0 2024-04-20T21:32:22.579 INFO:tasks.workunit.client.1.smithi189.stdout:3/810: mknod d2/d3/d2c/d31/d74/c119 0 2024-04-20T21:32:22.581 INFO:tasks.workunit.client.0.smithi062.stdout:6/930: sync 2024-04-20T21:32:22.581 INFO:tasks.workunit.client.0.smithi062.stdout:0/994: truncate d7/d1c/d103/f44 3310712 0 2024-04-20T21:32:22.583 INFO:tasks.workunit.client.0.smithi062.stdout:6/931: fsync d0/d1a/d7d/f121 0 2024-04-20T21:32:22.587 INFO:tasks.workunit.client.1.smithi189.stdout:4/754: rmdir d2/dd/d37/dcd/d6a/d7a/dce 39 2024-04-20T21:32:22.588 INFO:tasks.workunit.client.1.smithi189.stdout:0/904: dread - d3/df/d30/d37/dbd/f10f zero size 2024-04-20T21:32:22.590 INFO:tasks.workunit.client.1.smithi189.stdout:7/884: dread d5/f49 [0,4194304] 0 2024-04-20T21:32:22.597 INFO:tasks.workunit.client.1.smithi189.stdout:4/755: fdatasync d2/fd9 0 2024-04-20T21:32:22.600 INFO:tasks.workunit.client.1.smithi189.stdout:0/905: readlink d3/df/d30/l5c 0 2024-04-20T21:32:22.600 INFO:tasks.workunit.client.1.smithi189.stdout:5/835: truncate d7/d9/d31/d36/f38 2119364 0 2024-04-20T21:32:22.601 INFO:tasks.workunit.client.1.smithi189.stdout:3/811: creat d2/d3/d2c/f11a x:0 0 0 2024-04-20T21:32:22.601 INFO:tasks.workunit.client.1.smithi189.stdout:7/885: chown d5/d59/fa5 18 1 2024-04-20T21:32:22.603 INFO:tasks.workunit.client.0.smithi062.stdout:0/995: dwrite d7/d1e/d6e/f75 [4194304,4194304] 0 2024-04-20T21:32:22.605 INFO:tasks.workunit.client.1.smithi189.stdout:0/906: fdatasync d3/df/d4c/f7e 0 2024-04-20T21:32:22.607 INFO:tasks.workunit.client.1.smithi189.stdout:3/812: write d2/d1c/f2b [8011685,2337] 0 2024-04-20T21:32:22.608 INFO:tasks.workunit.client.0.smithi062.stdout:6/932: dread d0/d9/d11/d22/dd5/f30 [0,4194304] 0 2024-04-20T21:32:22.610 INFO:tasks.workunit.client.1.smithi189.stdout:7/886: mkdir d5/d21/d97/da3/dc6/d11f 0 2024-04-20T21:32:22.611 INFO:tasks.workunit.client.1.smithi189.stdout:6/850: dwrite d8/da/f32 [8388608,4194304] 0 2024-04-20T21:32:22.611 INFO:tasks.workunit.client.1.smithi189.stdout:5/836: creat d7/d9/d30/d5a/f10a x:0 0 0 2024-04-20T21:32:22.612 INFO:tasks.workunit.client.0.smithi062.stdout:3/957: dwrite d0/f2f [0,4194304] 0 2024-04-20T21:32:22.613 INFO:tasks.workunit.client.1.smithi189.stdout:4/756: truncate d2/dd/d11/f92 4667375 0 2024-04-20T21:32:22.618 INFO:tasks.workunit.client.1.smithi189.stdout:7/887: readlink d5/d46/l55 0 2024-04-20T21:32:22.620 INFO:tasks.workunit.client.1.smithi189.stdout:8/905: dread da/d81/d123/d20/d2c/f96 [0,4194304] 0 2024-04-20T21:32:22.622 INFO:tasks.workunit.client.0.smithi062.stdout:3/958: creat d0/d1/db7/db9/f13c x:0 0 0 2024-04-20T21:32:22.623 INFO:tasks.workunit.client.1.smithi189.stdout:3/813: rmdir d2/d3/d2c/d31/d74 39 2024-04-20T21:32:22.625 INFO:tasks.workunit.client.1.smithi189.stdout:6/851: mknod d8/da/dd/d35/c120 0 2024-04-20T21:32:22.626 INFO:tasks.workunit.client.1.smithi189.stdout:1/761: dread d8/df/d3a/d49/d8c/da2/dbe/dc1/fcd [0,4194304] 0 2024-04-20T21:32:22.633 INFO:tasks.workunit.client.1.smithi189.stdout:2/772: dread d1/d11/d2d/d4f/db4/fb1 [0,4194304] 0 2024-04-20T21:32:22.633 INFO:tasks.workunit.client.1.smithi189.stdout:6/852: fdatasync d8/da/dd/d118/d30/d47/da0/fc2 0 2024-04-20T21:32:22.634 INFO:tasks.workunit.client.0.smithi062.stdout:6/933: truncate d0/dc7/f94 3062610 0 2024-04-20T21:32:22.634 INFO:tasks.workunit.client.1.smithi189.stdout:8/906: chown da/d81/d123/d13/c88 99 1 2024-04-20T21:32:22.635 INFO:tasks.workunit.client.1.smithi189.stdout:0/907: dread d3/df/d30/d45/fc4 [0,4194304] 0 2024-04-20T21:32:22.635 INFO:tasks.workunit.client.1.smithi189.stdout:9/770: truncate d9/d12/fa8 2890608 0 2024-04-20T21:32:22.639 INFO:tasks.workunit.client.1.smithi189.stdout:7/888: creat d5/d21/d97/da3/dc6/d11f/f120 x:0 0 0 2024-04-20T21:32:22.639 INFO:tasks.workunit.client.1.smithi189.stdout:9/771: write f1 [3743947,4967] 0 2024-04-20T21:32:22.641 INFO:tasks.workunit.client.1.smithi189.stdout:1/762: dread - d8/d1e/fc6 zero size 2024-04-20T21:32:22.643 INFO:tasks.workunit.client.0.smithi062.stdout:6/934: read d0/d9a/fae [2116411,50119] 0 2024-04-20T21:32:22.643 INFO:tasks.workunit.client.1.smithi189.stdout:8/907: chown da/d81/d123/d3c/d68/d8b/dac/fe7 6328362 1 2024-04-20T21:32:22.644 INFO:tasks.workunit.client.1.smithi189.stdout:4/757: link d2/la d2/dd/d37/dcd/d6a/d7a/def/de8/dfd/l10d 0 2024-04-20T21:32:22.645 INFO:tasks.workunit.client.0.smithi062.stdout:0/996: dwrite d7/d1c/d15/d17/d1b/d106/db1/fef [0,4194304] 0 2024-04-20T21:32:22.647 INFO:tasks.workunit.client.1.smithi189.stdout:1/763: write d8/d1e/ff5 [552520,12120] 0 2024-04-20T21:32:22.647 INFO:tasks.workunit.client.0.smithi062.stdout:0/997: stat d7/d1c/d15/d17/d26/d76 0 2024-04-20T21:32:22.647 INFO:tasks.workunit.client.1.smithi189.stdout:9/772: write d9/d12/d3f/d91/fe7 [751317,11713] 0 2024-04-20T21:32:22.648 INFO:tasks.workunit.client.0.smithi062.stdout:0/998: fsync d7/ddf/fe8 0 2024-04-20T21:32:22.650 INFO:tasks.workunit.client.0.smithi062.stdout:0/999: chown d7/d1c/d15/d17/d1b/d106/df1 6343599 1 2024-04-20T21:32:22.653 INFO:tasks.workunit.client.1.smithi189.stdout:8/908: write da/de/d1e/d21/d87/ff6 [606859,75674] 0 2024-04-20T21:32:22.655 INFO:tasks.workunit.client.1.smithi189.stdout:0/908: link d3/db9/cf3 d3/df/d30/d45/dd3/c130 0 2024-04-20T21:32:22.658 INFO:tasks.workunit.client.1.smithi189.stdout:4/758: mkdir d2/dd/d11/dae/d10e 0 2024-04-20T21:32:22.660 INFO:tasks.workunit.client.1.smithi189.stdout:6/853: link d8/da/dd/d118/d99/cbd d8/d2b/d49/d108/d111/c121 0 2024-04-20T21:32:22.661 INFO:tasks.workunit.client.1.smithi189.stdout:0/909: fdatasync d3/d26/d88/f103 0 2024-04-20T21:32:22.665 INFO:tasks.workunit.client.1.smithi189.stdout:1/764: truncate d8/d1e/d28/f90 324157 0 2024-04-20T21:32:22.674 INFO:tasks.workunit.client.1.smithi189.stdout:3/814: write d2/d3/d36/da1/fb2 [899197,105418] 0 2024-04-20T21:32:22.680 INFO:tasks.workunit.client.1.smithi189.stdout:0/910: dread - d3/db9/dbe/dd7/fde zero size 2024-04-20T21:32:22.680 INFO:tasks.workunit.client.1.smithi189.stdout:4/759: getdents d2/dd/d37/dcd/d6a/d7a/db7/dea 0 2024-04-20T21:32:22.681 INFO:tasks.workunit.client.1.smithi189.stdout:2/773: dwrite d1/d11/d2d/d4f/fbc [0,4194304] 0 2024-04-20T21:32:22.687 INFO:tasks.workunit.client.1.smithi189.stdout:1/765: mknod d8/df/d3a/d49/d8c/da2/dbe/dc1/dd9/c100 0 2024-04-20T21:32:22.687 INFO:tasks.workunit.client.1.smithi189.stdout:5/837: dwrite d7/d9/d31/d100/d4e/d5b/f68 [0,4194304] 0 2024-04-20T21:32:22.692 INFO:tasks.workunit.client.1.smithi189.stdout:7/889: dread d5/d46/d7d/f108 [0,4194304] 0 2024-04-20T21:32:22.699 INFO:tasks.workunit.client.1.smithi189.stdout:1/766: creat d8/d1e/d43/d44/dd2/db7/f101 x:0 0 0 2024-04-20T21:32:22.699 INFO:tasks.workunit.client.1.smithi189.stdout:5/838: write d7/d9/fb8 [750165,107884] 0 2024-04-20T21:32:22.700 INFO:tasks.workunit.client.1.smithi189.stdout:1/767: chown d8/d1e/fc5 346132755 1 2024-04-20T21:32:22.705 INFO:tasks.workunit.client.1.smithi189.stdout:1/768: readlink d8/d21/d2c/lef 0 2024-04-20T21:32:22.707 INFO:tasks.workunit.client.1.smithi189.stdout:5/839: creat d7/d9/d30/d55/d104/d106/f10b x:0 0 0 2024-04-20T21:32:22.708 INFO:tasks.workunit.client.1.smithi189.stdout:1/769: chown d8/d21/dbf/le1 329 1 2024-04-20T21:32:22.713 INFO:tasks.workunit.client.1.smithi189.stdout:5/840: unlink d7/d9/d31/d36/fef 0 2024-04-20T21:32:22.713 INFO:tasks.workunit.client.1.smithi189.stdout:3/815: dwrite d2/d3/d25/f5b [0,4194304] 0 2024-04-20T21:32:22.718 INFO:tasks.workunit.client.1.smithi189.stdout:1/770: dread f6 [0,4194304] 0 2024-04-20T21:32:22.721 INFO:tasks.workunit.client.1.smithi189.stdout:4/760: write d2/fbe [200951,40271] 0 2024-04-20T21:32:22.724 INFO:tasks.workunit.client.1.smithi189.stdout:5/841: rmdir d7/d9/d30 39 2024-04-20T21:32:22.725 INFO:tasks.workunit.client.1.smithi189.stdout:3/816: unlink d2/d3/d2c/d30/c9d 0 2024-04-20T21:32:22.725 INFO:tasks.workunit.client.1.smithi189.stdout:7/890: write d5/db4/d71/fdf [29570,95760] 0 2024-04-20T21:32:22.726 INFO:tasks.workunit.client.1.smithi189.stdout:0/911: truncate d3/df/d4c/f69 3152534 0 2024-04-20T21:32:22.729 INFO:tasks.workunit.client.1.smithi189.stdout:2/774: dread d1/d11/d2d/d4f/db4/f5a [0,4194304] 0 2024-04-20T21:32:22.731 INFO:tasks.workunit.client.1.smithi189.stdout:5/842: write d7/d9/d31/d100/f8b [4071403,71219] 0 2024-04-20T21:32:22.732 INFO:tasks.workunit.client.1.smithi189.stdout:7/891: dread d5/d59/f80 [0,4194304] 0 2024-04-20T21:32:22.734 INFO:tasks.workunit.client.1.smithi189.stdout:4/761: mkdir d2/dd/d37/dcd/d6a/d7a/db7/d10f 0 2024-04-20T21:32:22.734 INFO:tasks.workunit.client.1.smithi189.stdout:0/912: chown d3/df/d5d/d64/f122 8321438 1 2024-04-20T21:32:22.735 INFO:tasks.workunit.client.1.smithi189.stdout:2/775: chown d1/d17/f35 281334 1 2024-04-20T21:32:22.735 INFO:tasks.workunit.client.0.smithi062.stdout:6/935: creat d0/d1a/d7d/db1/db9/f132 x:0 0 0 2024-04-20T21:32:22.736 INFO:tasks.workunit.client.1.smithi189.stdout:3/817: creat d2/d3/d2c/f11b x:0 0 0 2024-04-20T21:32:22.736 INFO:tasks.workunit.client.1.smithi189.stdout:5/843: mknod d7/d9/d31/d100/c10c 0 2024-04-20T21:32:22.740 INFO:tasks.workunit.client.0.smithi062.stdout:3/959: creat d0/d1c/d63/f13d x:0 0 0 2024-04-20T21:32:22.741 INFO:tasks.workunit.client.1.smithi189.stdout:0/913: write d3/df/d28/d2b/d2c/f108 [25524,67138] 0 2024-04-20T21:32:22.741 INFO:tasks.workunit.client.1.smithi189.stdout:4/762: rmdir d2/d20/dc6/dda 39 2024-04-20T21:32:22.745 INFO:tasks.workunit.client.1.smithi189.stdout:0/914: write d3/df/d30/d37/dbd/de1/f109 [631758,106861] 0 2024-04-20T21:32:22.746 INFO:tasks.workunit.client.1.smithi189.stdout:2/776: mkdir d1/d11/de1 0 2024-04-20T21:32:22.746 INFO:tasks.workunit.client.1.smithi189.stdout:6/854: dread d8/d10/d78/df7/fac [0,4194304] 0 2024-04-20T21:32:22.747 INFO:tasks.workunit.client.0.smithi062.stdout:3/960: symlink d0/d1/d10/d3a/d13b/l13e 0 2024-04-20T21:32:22.749 INFO:tasks.workunit.client.0.smithi062.stdout:3/961: chown d0/d1b/d74 14 1 2024-04-20T21:32:22.750 INFO:tasks.workunit.client.1.smithi189.stdout:2/777: write d1/f7 [1440729,38269] 0 2024-04-20T21:32:22.752 INFO:tasks.workunit.client.1.smithi189.stdout:1/771: write d8/df/d50/f8d [308928,16402] 0 2024-04-20T21:32:22.752 INFO:tasks.workunit.client.0.smithi062.stdout:3/962: rmdir d0/d1/d3c/d43/d72/dd3/d97 39 2024-04-20T21:32:22.755 INFO:tasks.workunit.client.1.smithi189.stdout:0/915: chown d3/df/d30/d37/l9a 32283481 1 2024-04-20T21:32:22.756 INFO:tasks.workunit.client.1.smithi189.stdout:6/855: chown d8/d10/d78/d7e/dcf/fe1 1 1 2024-04-20T21:32:22.756 INFO:tasks.workunit.client.1.smithi189.stdout:3/818: mkdir d2/d3/d2c/d31/d74/d11c 0 2024-04-20T21:32:22.756 INFO:tasks.workunit.client.1.smithi189.stdout:7/892: getdents d5/d1d/d64/db9 0 2024-04-20T21:32:22.760 INFO:tasks.workunit.client.0.smithi062.stdout:3/963: mkdir d0/d1/d3c/d43/d72/df2/d13f 0 2024-04-20T21:32:22.760 INFO:tasks.workunit.client.1.smithi189.stdout:6/856: rmdir d8/da/da7 39 2024-04-20T21:32:22.762 INFO:tasks.workunit.client.1.smithi189.stdout:1/772: write d8/df/f83 [1246481,48066] 0 2024-04-20T21:32:22.765 INFO:tasks.workunit.client.1.smithi189.stdout:7/893: fdatasync d5/db4/d71/fa6 0 2024-04-20T21:32:22.768 INFO:tasks.workunit.client.1.smithi189.stdout:3/819: creat d2/d3/d2c/d31/d90/d110/f11d x:0 0 0 2024-04-20T21:32:22.770 INFO:tasks.workunit.client.1.smithi189.stdout:2/778: write d1/d11/d2d/d4f/fbe [435,16087] 0 2024-04-20T21:32:22.775 INFO:tasks.workunit.client.1.smithi189.stdout:4/763: dwrite d2/d22/f6e [0,4194304] 0 2024-04-20T21:32:22.775 INFO:tasks.workunit.client.1.smithi189.stdout:1/773: write d8/f14 [2557878,7224] 0 2024-04-20T21:32:22.779 INFO:tasks.workunit.client.1.smithi189.stdout:7/894: truncate d5/d21/d25/fe1 377089 0 2024-04-20T21:32:22.783 INFO:tasks.workunit.client.1.smithi189.stdout:3/820: mkdir d2/d1c/dc1/d11e 0 2024-04-20T21:32:22.786 INFO:tasks.workunit.client.1.smithi189.stdout:5/844: dread d7/d9/d31/d100/d4e/d5b/fc2 [0,4194304] 0 2024-04-20T21:32:22.787 INFO:tasks.workunit.client.1.smithi189.stdout:6/857: dread d8/d2b/f8c [0,4194304] 0 2024-04-20T21:32:22.789 INFO:tasks.workunit.client.1.smithi189.stdout:2/779: write d1/d17/f7f [1101768,73504] 0 2024-04-20T21:32:22.793 INFO:tasks.workunit.client.1.smithi189.stdout:0/916: dwrite d3/df/d4c/f84 [0,4194304] 0 2024-04-20T21:32:22.797 INFO:tasks.workunit.client.1.smithi189.stdout:4/764: rmdir d2/d22/d87/de1/d10a 0 2024-04-20T21:32:22.797 INFO:tasks.workunit.client.1.smithi189.stdout:3/821: truncate d2/d3/d25/de0/db0/dc8/dc9/f104 133701 0 2024-04-20T21:32:22.798 INFO:tasks.workunit.client.0.smithi062.stdout:3/964: dread d0/d1b/f35 [0,4194304] 0 2024-04-20T21:32:22.800 INFO:tasks.workunit.client.1.smithi189.stdout:2/780: creat d1/d11/d2d/d4f/d8e/fe2 x:0 0 0 2024-04-20T21:32:22.801 INFO:tasks.workunit.client.1.smithi189.stdout:5/845: symlink d7/d9/d30/d55/d104/l10d 0 2024-04-20T21:32:22.803 INFO:tasks.workunit.client.0.smithi062.stdout:3/965: rmdir d0/d1/d115 39 2024-04-20T21:32:22.804 INFO:tasks.workunit.client.1.smithi189.stdout:4/765: symlink d2/dd/d37/dcd/d6a/l110 0 2024-04-20T21:32:22.810 INFO:tasks.workunit.client.1.smithi189.stdout:0/917: write d3/df/d30/d37/f65 [4881615,35595] 0 2024-04-20T21:32:22.811 INFO:tasks.workunit.client.1.smithi189.stdout:3/822: creat d2/d1c/dc1/dc5/ddf/f11f x:0 0 0 2024-04-20T21:32:22.811 INFO:tasks.workunit.client.1.smithi189.stdout:4/766: write d2/f54 [5709073,110706] 0 2024-04-20T21:32:22.811 INFO:tasks.workunit.client.1.smithi189.stdout:6/858: link d8/da/dd/d118/f17 d8/da/f122 0 2024-04-20T21:32:22.811 INFO:tasks.workunit.client.1.smithi189.stdout:7/895: dwrite d5/db4/d71/fdf [0,4194304] 0 2024-04-20T21:32:22.812 INFO:tasks.workunit.client.1.smithi189.stdout:5/846: creat d7/d9/d30/d55/d61/d96/f10e x:0 0 0 2024-04-20T21:32:22.816 INFO:tasks.workunit.client.1.smithi189.stdout:7/896: write d5/d46/fa9 [373159,51342] 0 2024-04-20T21:32:22.816 INFO:tasks.workunit.client.0.smithi062.stdout:3/966: fsync d0/f127 0 2024-04-20T21:32:22.818 INFO:tasks.workunit.client.1.smithi189.stdout:3/823: chown d2/d3/d25/f5b 7569 1 2024-04-20T21:32:22.818 INFO:tasks.workunit.client.1.smithi189.stdout:6/859: symlink d8/da/dd/d118/dd7/l123 0 2024-04-20T21:32:22.821 INFO:tasks.workunit.client.1.smithi189.stdout:4/767: creat d2/d22/d87/de1/f111 x:0 0 0 2024-04-20T21:32:22.821 INFO:tasks.workunit.client.1.smithi189.stdout:3/824: stat d2/d3/d25/de0/da9/fe5 0 2024-04-20T21:32:22.822 INFO:tasks.workunit.client.1.smithi189.stdout:7/897: chown d5/d1d/d64/db9/ddd/c10f 125 1 2024-04-20T21:32:22.823 INFO:tasks.workunit.client.1.smithi189.stdout:4/768: chown d2/dd/d37/lc9 6847429 1 2024-04-20T21:32:22.824 INFO:tasks.workunit.client.0.smithi062.stdout:3/967: creat d0/d5c/f140 x:0 0 0 2024-04-20T21:32:22.826 INFO:tasks.workunit.client.0.smithi062.stdout:3/968: read - d0/d1c/d63/f13d zero size 2024-04-20T21:32:22.827 INFO:tasks.workunit.client.1.smithi189.stdout:0/918: rename d3/df/lcb to d3/df/d4c/l131 0 2024-04-20T21:32:22.832 INFO:tasks.workunit.client.1.smithi189.stdout:7/898: unlink d5/d1d/ded/cf2 0 2024-04-20T21:32:22.835 INFO:tasks.workunit.client.1.smithi189.stdout:0/919: symlink d3/db9/dbe/dd7/l132 0 2024-04-20T21:32:22.835 INFO:tasks.workunit.client.1.smithi189.stdout:2/781: dwrite d1/f8c [4194304,4194304] 0 2024-04-20T21:32:22.839 INFO:tasks.workunit.client.1.smithi189.stdout:7/899: mknod d5/db4/d71/d7a/db0/c121 0 2024-04-20T21:32:22.839 INFO:tasks.workunit.client.1.smithi189.stdout:0/920: rmdir d3/df 39 2024-04-20T21:32:22.841 INFO:tasks.workunit.client.1.smithi189.stdout:2/782: readlink d1/d11/l2c 0 2024-04-20T21:32:22.846 INFO:tasks.workunit.client.1.smithi189.stdout:0/921: rmdir d3/d8c 39 2024-04-20T21:32:22.851 INFO:tasks.workunit.client.1.smithi189.stdout:7/900: mknod d5/d1d/d64/db9/dcf/c122 0 2024-04-20T21:32:22.851 INFO:tasks.workunit.client.1.smithi189.stdout:4/769: link d2/d7f/l10b d2/l112 0 2024-04-20T21:32:22.851 INFO:tasks.workunit.client.1.smithi189.stdout:2/783: truncate d1/d11/d2d/f38 2574951 0 2024-04-20T21:32:22.852 INFO:tasks.workunit.client.1.smithi189.stdout:3/825: write d2/d3/d2c/d31/d90/fd8 [1418043,80532] 0 2024-04-20T21:32:22.854 INFO:tasks.workunit.client.1.smithi189.stdout:0/922: write d3/df/d4c/f84 [3038905,47630] 0 2024-04-20T21:32:22.855 INFO:tasks.workunit.client.1.smithi189.stdout:4/770: dread - d2/dd/d6b/f100 zero size 2024-04-20T21:32:22.858 INFO:tasks.workunit.client.1.smithi189.stdout:3/826: mknod d2/d3/d36/c120 0 2024-04-20T21:32:22.858 INFO:tasks.workunit.client.1.smithi189.stdout:2/784: readlink d1/d11/d2d/d4f/db4/l29 0 2024-04-20T21:32:22.863 INFO:tasks.workunit.client.1.smithi189.stdout:2/785: mknod d1/d11/d2d/ce3 0 2024-04-20T21:32:22.868 INFO:tasks.workunit.client.1.smithi189.stdout:3/827: link d2/d3/d36/da1/fb2 d2/d3/d83/f121 0 2024-04-20T21:32:22.869 INFO:tasks.workunit.client.0.smithi062.stdout:3/969: dread d0/d1/d10/d3a/de9/d12a/f105 [0,4194304] 0 2024-04-20T21:32:22.869 INFO:tasks.workunit.client.1.smithi189.stdout:7/901: write d5/d46/d7d/f108 [4559124,98952] 0 2024-04-20T21:32:22.870 INFO:tasks.workunit.client.0.smithi062.stdout:3/970: chown d0/d1/d10/d3a/cca 36 1 2024-04-20T21:32:22.874 INFO:tasks.workunit.client.1.smithi189.stdout:7/902: creat d5/d21/d97/dfc/f123 x:0 0 0 2024-04-20T21:32:22.875 INFO:tasks.workunit.client.1.smithi189.stdout:3/828: rename d2/d3/d2c/d31/d74/d7a/fe7 to d2/d8/f122 0 2024-04-20T21:32:22.880 INFO:tasks.workunit.client.1.smithi189.stdout:4/771: dwrite d2/dd/d11/dae/fd4 [0,4194304] 0 2024-04-20T21:32:22.885 INFO:tasks.workunit.client.1.smithi189.stdout:2/786: dwrite d1/d11/d99/db7/fb8 [0,4194304] 0 2024-04-20T21:32:22.889 INFO:tasks.workunit.client.1.smithi189.stdout:4/772: dread - d2/dd/d37/dcd/d6a/d7a/db7/fcf zero size 2024-04-20T21:32:22.890 INFO:tasks.workunit.client.1.smithi189.stdout:2/787: readlink d1/d17/d37/lbf 0 2024-04-20T21:32:22.895 INFO:tasks.workunit.client.1.smithi189.stdout:3/829: truncate d2/f10f 220816 0 2024-04-20T21:32:22.895 INFO:tasks.workunit.client.1.smithi189.stdout:9/773: sync 2024-04-20T21:32:22.899 INFO:tasks.workunit.client.1.smithi189.stdout:7/903: dwrite d5/d1d/d64/db9/fff [0,4194304] 0 2024-04-20T21:32:22.903 INFO:tasks.workunit.client.1.smithi189.stdout:9/774: creat d9/dc/d17/d95/ff7 x:0 0 0 2024-04-20T21:32:22.910 INFO:tasks.workunit.client.0.smithi062.stdout:6/936: sync 2024-04-20T21:32:22.911 INFO:tasks.workunit.client.0.smithi062.stdout:3/971: getdents d0/d1c/d63/d42/d5b 0 2024-04-20T21:32:22.912 INFO:tasks.workunit.client.0.smithi062.stdout:6/937: stat d0/d52/f83 0 2024-04-20T21:32:22.915 INFO:tasks.workunit.client.0.smithi062.stdout:3/972: truncate d0/d1/db7/db9/f13c 51686 0 2024-04-20T21:32:22.917 INFO:tasks.workunit.client.0.smithi062.stdout:6/938: mknod d0/d52/c133 0 2024-04-20T21:32:22.918 INFO:tasks.workunit.client.1.smithi189.stdout:7/904: rename d5/d59/cc2 to d5/d1d/d64/db9/dcf/c124 0 2024-04-20T21:32:22.919 INFO:tasks.workunit.client.0.smithi062.stdout:3/973: fdatasync d0/d1/d3c/d43/d72/dd3/fe0 0 2024-04-20T21:32:22.919 INFO:tasks.workunit.client.0.smithi062.stdout:6/939: chown d0/d1a/d44/f4c 54896850 1 2024-04-20T21:32:22.920 INFO:tasks.workunit.client.0.smithi062.stdout:3/974: stat d0/d1/db7/c10f 0 2024-04-20T21:32:22.920 INFO:tasks.workunit.client.1.smithi189.stdout:9/775: mkdir d9/dc/de9/df8 0 2024-04-20T21:32:22.922 INFO:tasks.workunit.client.0.smithi062.stdout:6/940: chown d0/d9/d11/d22/dd5/d67/d75/fe2 169227 1 2024-04-20T21:32:22.923 INFO:tasks.workunit.client.0.smithi062.stdout:6/941: chown d0/c74 93911653 1 2024-04-20T21:32:22.926 INFO:tasks.workunit.client.1.smithi189.stdout:4/773: dwrite d2/dd/d37/ff6 [0,4194304] 0 2024-04-20T21:32:22.928 INFO:tasks.workunit.client.0.smithi062.stdout:3/975: rename d0/d1/d10/d31/d41/cc1 to d0/c141 0 2024-04-20T21:32:22.931 INFO:tasks.workunit.client.0.smithi062.stdout:6/942: read d0/dc7/d7c/f7f [1843240,111059] 0 2024-04-20T21:32:22.934 INFO:tasks.workunit.client.1.smithi189.stdout:3/830: dwrite d2/d3/d2c/fda [0,4194304] 0 2024-04-20T21:32:22.941 INFO:tasks.workunit.client.0.smithi062.stdout:6/943: link d0/d1a/d7d/db1/de0/lf1 d0/d1a/l134 0 2024-04-20T21:32:22.941 INFO:tasks.workunit.client.1.smithi189.stdout:7/905: chown d5/d1d/f56 1 1 2024-04-20T21:32:22.943 INFO:tasks.workunit.client.1.smithi189.stdout:9/776: creat d9/dc/d17/d7e/ff9 x:0 0 0 2024-04-20T21:32:22.944 INFO:tasks.workunit.client.1.smithi189.stdout:9/777: fsync f7 0 2024-04-20T21:32:22.945 INFO:tasks.workunit.client.1.smithi189.stdout:2/788: dread d1/d11/d52/d8b/fa1 [0,4194304] 0 2024-04-20T21:32:22.946 INFO:tasks.workunit.client.0.smithi062.stdout:3/976: dwrite d0/d1/d10/d3a/d5e/ddb/fee [4194304,4194304] 0 2024-04-20T21:32:22.948 INFO:tasks.workunit.client.1.smithi189.stdout:3/831: creat d2/d3/d2c/d31/d74/d11c/f123 x:0 0 0 2024-04-20T21:32:22.950 INFO:tasks.workunit.client.1.smithi189.stdout:7/906: chown d5/db4/d71/d7a/db0/cf3 27937 1 2024-04-20T21:32:22.950 INFO:tasks.workunit.client.1.smithi189.stdout:9/778: mknod d9/dc/d17/dac/cfa 0 2024-04-20T21:32:22.952 INFO:tasks.workunit.client.0.smithi062.stdout:3/977: mknod d0/d1/db7/db9/c142 0 2024-04-20T21:32:22.953 INFO:tasks.workunit.client.1.smithi189.stdout:7/907: write d5/d21/d97/da3/f10e [1036851,21427] 0 2024-04-20T21:32:22.955 INFO:tasks.workunit.client.1.smithi189.stdout:9/779: fdatasync d9/dc/f3d 0 2024-04-20T21:32:22.958 INFO:tasks.workunit.client.0.smithi062.stdout:3/978: write d0/d1/d3c/d43/d72/dd3/d97/fa3 [551690,34384] 0 2024-04-20T21:32:22.960 INFO:tasks.workunit.client.1.smithi189.stdout:3/832: rename d2/c4e to d2/d1c/dc1/dc5/ddf/c124 0 2024-04-20T21:32:22.961 INFO:tasks.workunit.client.0.smithi062.stdout:6/944: dwrite d0/d9/d11/d22/dd5/d67/ffb [0,4194304] 0 2024-04-20T21:32:22.967 INFO:tasks.workunit.client.1.smithi189.stdout:3/833: fdatasync d2/d3/d2c/d31/d90/d110/f11d 0 2024-04-20T21:32:22.968 INFO:tasks.workunit.client.0.smithi062.stdout:3/979: creat d0/d1/d100/f143 x:0 0 0 2024-04-20T21:32:22.971 INFO:tasks.workunit.client.1.smithi189.stdout:3/834: dread - d2/d3/d2c/d31/d74/db7/dd3/fe8 zero size 2024-04-20T21:32:22.971 INFO:tasks.workunit.client.1.smithi189.stdout:9/780: rename d9/dc/d17/d36/f43 to d9/dc/ffb 0 2024-04-20T21:32:22.975 INFO:tasks.workunit.client.1.smithi189.stdout:3/835: dread - d2/d3/d25/f97 zero size 2024-04-20T21:32:22.975 INFO:tasks.workunit.client.0.smithi062.stdout:3/980: creat d0/d1/d10/d31/d41/de6/d102/f144 x:0 0 0 2024-04-20T21:32:22.977 INFO:tasks.workunit.client.1.smithi189.stdout:9/781: mkdir d9/dc/dfc 0 2024-04-20T21:32:22.979 INFO:tasks.workunit.client.0.smithi062.stdout:3/981: readlink d0/d1/d10/l17 0 2024-04-20T21:32:22.979 INFO:tasks.workunit.client.1.smithi189.stdout:3/836: read d2/d3/d83/f121 [283019,38168] 0 2024-04-20T21:32:22.980 INFO:tasks.workunit.client.0.smithi062.stdout:3/982: fsync d0/d1/db7/db9/f13c 0 2024-04-20T21:32:22.981 INFO:tasks.workunit.client.1.smithi189.stdout:9/782: rmdir d9/dc/d17/d95/da6 39 2024-04-20T21:32:22.986 INFO:tasks.workunit.client.1.smithi189.stdout:9/783: chown d9/dc/d17/cc8 12164 1 2024-04-20T21:32:22.986 INFO:tasks.workunit.client.1.smithi189.stdout:3/837: mknod d2/d1c/dc1/dc5/c125 0 2024-04-20T21:32:22.989 INFO:tasks.workunit.client.1.smithi189.stdout:3/838: rmdir d2/d1c 39 2024-04-20T21:32:22.991 INFO:tasks.workunit.client.1.smithi189.stdout:2/789: write d1/d11/d2d/f38 [2485459,4473] 0 2024-04-20T21:32:22.992 INFO:tasks.workunit.client.1.smithi189.stdout:2/790: stat d1/d11/d2d 0 2024-04-20T21:32:22.995 INFO:tasks.workunit.client.1.smithi189.stdout:2/791: chown d1/d11/d58/fd5 1481 1 2024-04-20T21:32:22.998 INFO:tasks.workunit.client.1.smithi189.stdout:4/774: dwrite d2/dd/d6b/fc7 [0,4194304] 0 2024-04-20T21:32:22.999 INFO:tasks.workunit.client.0.smithi062.stdout:3/983: link d0/d1c/d63/f91 d0/d5c/f145 0 2024-04-20T21:32:23.000 INFO:tasks.workunit.client.0.smithi062.stdout:3/984: stat d0/d1b/f35 0 2024-04-20T21:32:23.004 INFO:tasks.workunit.client.1.smithi189.stdout:9/784: getdents d9/dc 0 2024-04-20T21:32:23.006 INFO:tasks.workunit.client.0.smithi062.stdout:3/985: mkdir d0/d1/d10/d3a/d146 0 2024-04-20T21:32:23.007 INFO:tasks.workunit.client.1.smithi189.stdout:8/909: sync 2024-04-20T21:32:23.008 INFO:tasks.workunit.client.1.smithi189.stdout:7/908: dwrite d5/fe [0,4194304] 0 2024-04-20T21:32:23.011 INFO:tasks.workunit.client.1.smithi189.stdout:8/910: write da/de/d1e/d21/d44/f4a [3107343,71900] 0 2024-04-20T21:32:23.019 INFO:tasks.workunit.client.1.smithi189.stdout:8/911: write da/d81/d91/fee [586068,26226] 0 2024-04-20T21:32:23.021 INFO:tasks.workunit.client.1.smithi189.stdout:4/775: truncate d2/dd/d11/dae/fd4 580895 0 2024-04-20T21:32:23.022 INFO:tasks.workunit.client.1.smithi189.stdout:7/909: symlink d5/d1d/d64/db9/ddd/l125 0 2024-04-20T21:32:23.026 INFO:tasks.workunit.client.1.smithi189.stdout:7/910: readlink d5/d21/d25/d34/lac 0 2024-04-20T21:32:23.026 INFO:tasks.workunit.client.1.smithi189.stdout:9/785: rmdir d9/dc/d17/d95/da0/d55/dd0/de6 0 2024-04-20T21:32:23.030 INFO:tasks.workunit.client.1.smithi189.stdout:2/792: dwrite d1/d11/d58/f64 [0,4194304] 0 2024-04-20T21:32:23.030 INFO:tasks.workunit.client.1.smithi189.stdout:7/911: stat d5/d21/d25/f27 0 2024-04-20T21:32:23.031 INFO:tasks.workunit.client.1.smithi189.stdout:4/776: creat d2/dd/d37/f113 x:0 0 0 2024-04-20T21:32:23.032 INFO:tasks.workunit.client.0.smithi062.stdout:3/986: dwrite d0/d1/d10/d31/f9e [0,4194304] 0 2024-04-20T21:32:23.037 INFO:tasks.workunit.client.0.smithi062.stdout:3/987: fdatasync d0/d1/d115/d11c/fb6 0 2024-04-20T21:32:23.040 INFO:tasks.workunit.client.0.smithi062.stdout:3/988: chown d0/d1/d10/d99/f137 3703 1 2024-04-20T21:32:23.043 INFO:tasks.workunit.client.1.smithi189.stdout:9/786: mknod d9/dc/d17/d95/da0/d52/d82/d94/cfd 0 2024-04-20T21:32:23.047 INFO:tasks.workunit.client.0.smithi062.stdout:3/989: rename d0/d5c/lc0 to d0/d1/d3c/d43/d72/dd3/dc5/l147 0 2024-04-20T21:32:23.048 INFO:tasks.workunit.client.1.smithi189.stdout:2/793: getdents d1/d11/d2d 0 2024-04-20T21:32:23.049 INFO:tasks.workunit.client.0.smithi062.stdout:3/990: write d0/d1c/f132 [712270,53683] 0 2024-04-20T21:32:23.056 INFO:tasks.workunit.client.0.smithi062.stdout:3/991: truncate d0/d1c/d63/d42/fcb 920899 0 2024-04-20T21:32:23.058 INFO:tasks.workunit.client.1.smithi189.stdout:9/787: rename d9/dc/fee to d9/dc/d17/d95/ffe 0 2024-04-20T21:32:23.060 INFO:tasks.workunit.client.0.smithi062.stdout:3/992: creat d0/d1/d10/d31/d41/de6/f148 x:0 0 0 2024-04-20T21:32:23.064 INFO:tasks.workunit.client.0.smithi062.stdout:3/993: creat d0/d5c/f149 x:0 0 0 2024-04-20T21:32:23.066 INFO:tasks.workunit.client.1.smithi189.stdout:8/912: dwrite da/d81/d123/d22/f139 [0,4194304] 0 2024-04-20T21:32:23.071 INFO:tasks.workunit.client.1.smithi189.stdout:4/777: dwrite d2/dd/d37/f39 [0,4194304] 0 2024-04-20T21:32:23.075 INFO:tasks.workunit.client.1.smithi189.stdout:7/912: dwrite d5/d46/fb7 [0,4194304] 0 2024-04-20T21:32:23.079 INFO:tasks.workunit.client.0.smithi062.stdout:3/994: write d0/d11/f1a [7850478,126560] 0 2024-04-20T21:32:23.083 INFO:tasks.workunit.client.1.smithi189.stdout:2/794: dwrite d1/d11/d52/fbd [0,4194304] 0 2024-04-20T21:32:23.084 INFO:tasks.workunit.client.0.smithi062.stdout:3/995: fsync d0/d1b/f75 0 2024-04-20T21:32:23.093 INFO:tasks.workunit.client.1.smithi189.stdout:4/778: creat d2/dd/d37/dcd/d6a/f114 x:0 0 0 2024-04-20T21:32:23.094 INFO:tasks.workunit.client.0.smithi062.stdout:3/996: unlink d0/d11/fa2 0 2024-04-20T21:32:23.094 INFO:tasks.workunit.client.1.smithi189.stdout:9/788: fdatasync d9/dc/d17/d95/da6/fe1 0 2024-04-20T21:32:23.094 INFO:tasks.workunit.client.1.smithi189.stdout:2/795: mkdir d1/d11/d52/d8b/de4 0 2024-04-20T21:32:23.097 INFO:tasks.workunit.client.1.smithi189.stdout:7/913: mknod d5/d21/d4a/d52/d75/c126 0 2024-04-20T21:32:23.099 INFO:tasks.workunit.client.0.smithi062.stdout:3/997: mkdir d0/d1/d10/d99/d14a 0 2024-04-20T21:32:23.101 INFO:tasks.workunit.client.0.smithi062.stdout:3/998: fsync d0/d1/d100/f143 0 2024-04-20T21:32:23.101 INFO:tasks.workunit.client.1.smithi189.stdout:8/913: dwrite da/de/d1e/d21/f13a [0,4194304] 0 2024-04-20T21:32:23.103 INFO:tasks.workunit.client.1.smithi189.stdout:7/914: mkdir d5/d21/d4a/dfd/d127 0 2024-04-20T21:32:23.105 INFO:tasks.workunit.client.0.smithi062.stdout:3/999: readlink d0/d1b/l38 0 2024-04-20T21:32:23.108 INFO:tasks.workunit.client.1.smithi189.stdout:7/915: fsync d5/d1d/d64/fa1 0 2024-04-20T21:32:23.109 INFO:tasks.workunit.client.1.smithi189.stdout:8/914: rename da/d81/d123/d3c/d68/lf1 to da/d81/d123/d3c/d68/d8b/l140 0 2024-04-20T21:32:23.118 INFO:tasks.workunit.client.1.smithi189.stdout:7/916: creat d5/d46/d7d/d111/d11b/f128 x:0 0 0 2024-04-20T21:32:23.121 INFO:tasks.workunit.client.1.smithi189.stdout:1/774: sync 2024-04-20T21:32:23.122 INFO:tasks.workunit.client.1.smithi189.stdout:9/789: dwrite d9/dc/f3d [0,4194304] 0 2024-04-20T21:32:23.130 INFO:tasks.workunit.client.1.smithi189.stdout:9/790: write d9/d12/d3b/db1/d1a/f27 [1032966,56790] 0 2024-04-20T21:32:23.130 INFO:tasks.workunit.client.1.smithi189.stdout:2/796: truncate d1/d11/f9d 1021248 0 2024-04-20T21:32:23.132 INFO:tasks.workunit.client.1.smithi189.stdout:9/791: mknod d9/d12/d3f/d91/cff 0 2024-04-20T21:32:23.135 INFO:tasks.workunit.client.1.smithi189.stdout:2/797: mknod d1/d11/d2d/d4f/ce5 0 2024-04-20T21:32:23.136 INFO:tasks.workunit.client.1.smithi189.stdout:9/792: creat d9/dc/d17/d95/da0/d55/dd0/f100 x:0 0 0 2024-04-20T21:32:23.139 INFO:tasks.workunit.client.1.smithi189.stdout:8/915: write da/d81/d123/d3c/d68/d8b/d8d/f90 [4133121,126135] 0 2024-04-20T21:32:23.141 INFO:tasks.workunit.client.1.smithi189.stdout:8/916: readlink da/de/lab 0 2024-04-20T21:32:23.145 INFO:tasks.workunit.client.1.smithi189.stdout:8/917: truncate da/d81/d123/d13/f5f 4699859 0 2024-04-20T21:32:23.147 INFO:tasks.workunit.client.1.smithi189.stdout:9/793: dread d9/d12/d3f/d91/fe7 [0,4194304] 0 2024-04-20T21:32:23.147 INFO:tasks.workunit.client.1.smithi189.stdout:2/798: link d1/d11/l40 d1/d11/d2d/d4f/le6 0 2024-04-20T21:32:23.150 INFO:tasks.workunit.client.1.smithi189.stdout:8/918: creat da/d81/d123/d20/d2c/f141 x:0 0 0 2024-04-20T21:32:23.151 INFO:tasks.workunit.client.1.smithi189.stdout:2/799: dread - d1/d11/d2d/d4f/db4/daa/dd2/fdb zero size 2024-04-20T21:32:23.155 INFO:tasks.workunit.client.1.smithi189.stdout:4/779: dread d2/f2c [0,4194304] 0 2024-04-20T21:32:23.157 INFO:tasks.workunit.client.1.smithi189.stdout:8/919: fdatasync da/d81/d123/d13/dd6/d59/fa6 0 2024-04-20T21:32:23.157 INFO:tasks.workunit.client.1.smithi189.stdout:1/775: dread d8/f24 [0,4194304] 0 2024-04-20T21:32:23.160 INFO:tasks.workunit.client.1.smithi189.stdout:1/776: stat d8/d1e/c9a 0 2024-04-20T21:32:23.161 INFO:tasks.workunit.client.1.smithi189.stdout:8/920: chown da/d81/d123/d13/d63/d7b/dfa 12 1 2024-04-20T21:32:23.162 INFO:tasks.workunit.client.1.smithi189.stdout:2/800: read d1/d11/f9d [855645,86395] 0 2024-04-20T21:32:23.163 INFO:tasks.workunit.client.1.smithi189.stdout:9/794: write d9/dc/d69/f7c [246586,58462] 0 2024-04-20T21:32:23.171 INFO:tasks.workunit.client.1.smithi189.stdout:4/780: write d2/dd/d37/fc3 [1036693,92622] 0 2024-04-20T21:32:23.172 INFO:tasks.workunit.client.1.smithi189.stdout:2/801: mkdir d1/d11/d2d/dab/d4a/d66/dc9/de7 0 2024-04-20T21:32:23.174 INFO:tasks.workunit.client.1.smithi189.stdout:4/781: chown d2/dd/d11/f92 5 1 2024-04-20T21:32:23.176 INFO:tasks.workunit.client.0.smithi062.stdout:6/945: sync 2024-04-20T21:32:23.179 INFO:tasks.workunit.client.1.smithi189.stdout:2/802: dread d1/d11/d2d/d4f/db4/f5a [0,4194304] 0 2024-04-20T21:32:23.180 INFO:tasks.workunit.client.1.smithi189.stdout:9/795: dread d9/dc/d69/faf [0,4194304] 0 2024-04-20T21:32:23.180 INFO:tasks.workunit.client.0.smithi062.stdout:6/946: creat d0/d1a/d44/db3/f135 x:0 0 0 2024-04-20T21:32:23.181 INFO:tasks.workunit.client.1.smithi189.stdout:8/921: dwrite da/d81/d123/d3c/d9d/fec [0,4194304] 0 2024-04-20T21:32:23.183 INFO:tasks.workunit.client.1.smithi189.stdout:8/922: fdatasync da/d81/d123/d22/f139 0 2024-04-20T21:32:23.185 INFO:tasks.workunit.client.0.smithi062.stdout:6/947: creat d0/d4f/f136 x:0 0 0 2024-04-20T21:32:23.189 INFO:tasks.workunit.client.1.smithi189.stdout:9/796: creat d9/dc/d17/d95/da6/f101 x:0 0 0 2024-04-20T21:32:23.189 INFO:tasks.workunit.client.1.smithi189.stdout:2/803: creat d1/d11/fe8 x:0 0 0 2024-04-20T21:32:23.191 INFO:tasks.workunit.client.1.smithi189.stdout:8/923: creat da/d81/d123/d13/d63/d7b/f142 x:0 0 0 2024-04-20T21:32:23.196 INFO:tasks.workunit.client.1.smithi189.stdout:1/777: dread d8/d21/fa5 [0,4194304] 0 2024-04-20T21:32:23.198 INFO:tasks.workunit.client.0.smithi062.stdout:6/948: rename d0/d1a/d44/da7/lba to d0/d1a/d7d/l137 0 2024-04-20T21:32:23.199 INFO:tasks.workunit.client.1.smithi189.stdout:8/924: mknod da/d81/d123/d3c/d68/c143 0 2024-04-20T21:32:23.210 INFO:tasks.workunit.client.1.smithi189.stdout:9/797: dwrite d9/dc/d17/d95/da0/d52/fd3 [0,4194304] 0 2024-04-20T21:32:23.211 INFO:tasks.workunit.client.1.smithi189.stdout:8/925: link da/d81/d123/d20/d2c/d9a/dce/fd9 da/d81/d123/d3c/d50/d6c/f144 0 2024-04-20T21:32:23.218 INFO:tasks.workunit.client.1.smithi189.stdout:1/778: truncate d8/f14 681533 0 2024-04-20T21:32:23.221 INFO:tasks.workunit.client.1.smithi189.stdout:8/926: creat da/d81/d123/d3a/dfe/d13b/f145 x:0 0 0 2024-04-20T21:32:23.223 INFO:tasks.workunit.client.1.smithi189.stdout:1/779: symlink d8/df/d3a/d49/d8c/da2/dbe/dc1/l102 0 2024-04-20T21:32:23.228 INFO:tasks.workunit.client.1.smithi189.stdout:1/780: rmdir d8/d9b 39 2024-04-20T21:32:23.229 INFO:tasks.workunit.client.0.smithi062.stdout:6/949: dwrite d0/d1a/f29 [0,4194304] 0 2024-04-20T21:32:23.230 INFO:tasks.workunit.client.1.smithi189.stdout:2/804: dread d1/d11/d58/d6b/f6c [0,4194304] 0 2024-04-20T21:32:23.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:22 smithi189 ceph-mon[21562]: pgmap v26: 65 pgs: 65 active+clean; 3.3 GiB data, 9.6 GiB used, 527 GiB / 536 GiB avail; 84 MiB/s rd, 196 MiB/s wr, 442 op/s 2024-04-20T21:32:23.239 INFO:tasks.workunit.client.1.smithi189.stdout:9/798: dwrite d9/d12/d62/dae/fb2 [0,4194304] 0 2024-04-20T21:32:23.239 INFO:tasks.workunit.client.0.smithi062.stdout:6/950: creat d0/d9/f138 x:0 0 0 2024-04-20T21:32:23.241 INFO:tasks.workunit.client.1.smithi189.stdout:9/799: write d9/dc/d69/f7c [737294,80671] 0 2024-04-20T21:32:23.246 INFO:tasks.workunit.client.1.smithi189.stdout:9/800: write d9/dc/d17/d95/da0/d52/d82/fa7 [1753642,110194] 0 2024-04-20T21:32:23.260 INFO:tasks.workunit.client.1.smithi189.stdout:8/927: dwrite da/de/d1e/d69/d70/f72 [0,4194304] 0 2024-04-20T21:32:23.261 INFO:tasks.workunit.client.1.smithi189.stdout:5/847: sync 2024-04-20T21:32:23.262 INFO:tasks.workunit.client.1.smithi189.stdout:3/839: sync 2024-04-20T21:32:23.262 INFO:tasks.workunit.client.1.smithi189.stdout:0/923: sync 2024-04-20T21:32:23.262 INFO:tasks.workunit.client.1.smithi189.stdout:2/805: write d1/d17/d37/d89/f95 [4324866,65073] 0 2024-04-20T21:32:23.265 INFO:tasks.workunit.client.1.smithi189.stdout:6/860: sync 2024-04-20T21:32:23.265 INFO:tasks.workunit.client.1.smithi189.stdout:1/781: truncate d8/d1e/d43/d44/fc9 2697953 0 2024-04-20T21:32:23.266 INFO:tasks.workunit.client.1.smithi189.stdout:0/924: fdatasync d3/df/d4c/f11b 0 2024-04-20T21:32:23.270 INFO:tasks.workunit.client.1.smithi189.stdout:5/848: creat d7/d9/f10f x:0 0 0 2024-04-20T21:32:23.271 INFO:tasks.workunit.client.1.smithi189.stdout:1/782: mkdir d8/df/d3a/d49/d8c/d103 0 2024-04-20T21:32:23.272 INFO:tasks.workunit.client.0.smithi062.stdout:6/951: truncate d0/d9/d11/d7b/f103 3047249 0 2024-04-20T21:32:23.274 INFO:tasks.workunit.client.1.smithi189.stdout:6/861: fdatasync d8/da/dd/d118/daf/fd1 0 2024-04-20T21:32:23.275 INFO:tasks.workunit.client.1.smithi189.stdout:9/801: write d9/dc/d17/d36/d5d/f97 [3538775,46485] 0 2024-04-20T21:32:23.276 INFO:tasks.workunit.client.1.smithi189.stdout:5/849: creat d7/d9/d31/d100/f110 x:0 0 0 2024-04-20T21:32:23.280 INFO:tasks.workunit.client.1.smithi189.stdout:5/850: truncate d7/d9/f10f 875174 0 2024-04-20T21:32:23.281 INFO:tasks.workunit.client.1.smithi189.stdout:6/862: rmdir d8/da/dd/d35/d6d/d7b 39 2024-04-20T21:32:23.284 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:22 smithi062 ceph-mon[16221]: pgmap v26: 65 pgs: 65 active+clean; 3.3 GiB data, 9.6 GiB used, 527 GiB / 536 GiB avail; 84 MiB/s rd, 196 MiB/s wr, 442 op/s 2024-04-20T21:32:23.286 INFO:tasks.workunit.client.0.smithi062.stdout:6/952: creat d0/d1a/d7d/db1/f139 x:0 0 0 2024-04-20T21:32:23.286 INFO:tasks.workunit.client.1.smithi189.stdout:0/925: creat d3/df/d5d/f133 x:0 0 0 2024-04-20T21:32:23.286 INFO:tasks.workunit.client.1.smithi189.stdout:5/851: chown d7/d9/d31/fc1 203368969 1 2024-04-20T21:32:23.299 INFO:tasks.workunit.client.1.smithi189.stdout:0/926: mknod d3/df/d30/d37/dbd/de1/c134 0 2024-04-20T21:32:23.299 INFO:tasks.workunit.client.1.smithi189.stdout:8/928: dwrite da/d81/d123/d13/dd2/fbe [0,4194304] 0 2024-04-20T21:32:23.303 INFO:tasks.workunit.client.1.smithi189.stdout:6/863: dread d8/f9b [0,4194304] 0 2024-04-20T21:32:23.305 INFO:tasks.workunit.client.1.smithi189.stdout:8/929: chown da/de/d1e/d69 3807095 1 2024-04-20T21:32:23.305 INFO:tasks.workunit.client.1.smithi189.stdout:5/852: getdents d7/d9/d51/d70/d99/dd4 0 2024-04-20T21:32:23.307 INFO:tasks.workunit.client.1.smithi189.stdout:2/806: dread d1/d11/d2d/fa6 [0,4194304] 0 2024-04-20T21:32:23.309 INFO:tasks.workunit.client.1.smithi189.stdout:1/783: dwrite d8/d1e/d54/d93/ffa [0,4194304] 0 2024-04-20T21:32:23.313 INFO:tasks.workunit.client.1.smithi189.stdout:6/864: rmdir d8/d10/d78/d7e/dcf 39 2024-04-20T21:32:23.319 INFO:tasks.workunit.client.0.smithi062.stdout:6/953: dwrite d0/d9/d11/d22/dd5/f3b [4194304,4194304] 0 2024-04-20T21:32:23.320 INFO:tasks.workunit.client.1.smithi189.stdout:1/784: write d8/d1e/d43/d44/dd2/d8a/fd0 [72444,82154] 0 2024-04-20T21:32:23.321 INFO:tasks.workunit.client.1.smithi189.stdout:2/807: dread d1/d11/d2d/d4f/db4/f9c [0,4194304] 0 2024-04-20T21:32:23.329 INFO:tasks.workunit.client.1.smithi189.stdout:5/853: getdents d7/d9/d51/d70 0 2024-04-20T21:32:23.331 INFO:tasks.workunit.client.1.smithi189.stdout:9/802: dwrite d9/d12/fa8 [0,4194304] 0 2024-04-20T21:32:23.336 INFO:tasks.workunit.client.1.smithi189.stdout:8/930: rename da/d81/d123/d3c/d68/d8b/dac/de1/f93 to da/d81/d123/d3c/deb/f146 0 2024-04-20T21:32:23.341 INFO:tasks.workunit.client.1.smithi189.stdout:2/808: creat d1/d11/d2d/d4f/fe9 x:0 0 0 2024-04-20T21:32:23.341 INFO:tasks.workunit.client.1.smithi189.stdout:8/931: unlink da/d1f/l134 0 2024-04-20T21:32:23.348 INFO:tasks.workunit.client.1.smithi189.stdout:2/809: dread - d1/d17/d78/db9/fc7 zero size 2024-04-20T21:32:23.351 INFO:tasks.workunit.client.1.smithi189.stdout:9/803: getdents d9/dc/d17/d95/da0/d52/d82/da4/df3 0 2024-04-20T21:32:23.351 INFO:tasks.workunit.client.1.smithi189.stdout:0/927: dwrite d3/db9/dbe/dd7/fde [0,4194304] 0 2024-04-20T21:32:23.352 INFO:tasks.workunit.client.1.smithi189.stdout:8/932: chown da/d81/d123/d13/dd6/dba/d11e/c138 0 1 2024-04-20T21:32:23.354 INFO:tasks.workunit.client.0.smithi062.stdout:6/954: dread d0/dc7/fcd [0,4194304] 0 2024-04-20T21:32:23.356 INFO:tasks.workunit.client.0.smithi062.stdout:6/955: chown d0/d9a/l120 341374 1 2024-04-20T21:32:23.358 INFO:tasks.workunit.client.1.smithi189.stdout:0/928: rmdir d3/d26/d88 39 2024-04-20T21:32:23.360 INFO:tasks.workunit.client.1.smithi189.stdout:8/933: mknod da/d81/d123/d3c/d50/d6c/c147 0 2024-04-20T21:32:23.361 INFO:tasks.workunit.client.0.smithi062.stdout:6/956: dread d0/d9/f71 [0,4194304] 0 2024-04-20T21:32:23.361 INFO:tasks.workunit.client.1.smithi189.stdout:9/804: mkdir d9/dc/d17/d95/d102 0 2024-04-20T21:32:23.368 INFO:tasks.workunit.client.1.smithi189.stdout:0/929: mknod d3/df/d30/d45/d90/df4/d114/c135 0 2024-04-20T21:32:23.369 INFO:tasks.workunit.client.1.smithi189.stdout:9/805: mknod d9/d12/d62/dae/c103 0 2024-04-20T21:32:23.374 INFO:tasks.workunit.client.1.smithi189.stdout:5/854: truncate d7/d9/fb8 453794 0 2024-04-20T21:32:23.376 INFO:tasks.workunit.client.1.smithi189.stdout:1/785: dread d8/d1e/f6e [0,4194304] 0 2024-04-20T21:32:23.377 INFO:tasks.workunit.client.1.smithi189.stdout:8/934: rename da/de/d1e/fb7 to da/de/d1e/d69/f148 0 2024-04-20T21:32:23.379 INFO:tasks.workunit.client.1.smithi189.stdout:5/855: chown d7/d9/d30/f8e 2833 1 2024-04-20T21:32:23.381 INFO:tasks.workunit.client.1.smithi189.stdout:9/806: creat d9/d12/d3f/f104 x:0 0 0 2024-04-20T21:32:23.382 INFO:tasks.workunit.client.0.smithi062.stdout:6/957: dwrite d0/d1a/d7d/db1/f139 [0,4194304] 0 2024-04-20T21:32:23.384 INFO:tasks.workunit.client.1.smithi189.stdout:9/807: chown d9/dc/d69 10254 1 2024-04-20T21:32:23.385 INFO:tasks.workunit.client.0.smithi062.stdout:6/958: chown d0/d1a/d44/fac 0 1 2024-04-20T21:32:23.386 INFO:tasks.workunit.client.1.smithi189.stdout:6/865: dread d8/da/dd/f71 [0,4194304] 0 2024-04-20T21:32:23.386 INFO:tasks.workunit.client.1.smithi189.stdout:5/856: chown d7/d35/fce 125076413 1 2024-04-20T21:32:23.388 INFO:tasks.workunit.client.1.smithi189.stdout:1/786: rename d8/df/d3a/d49/l53 to d8/d1e/l104 0 2024-04-20T21:32:23.391 INFO:tasks.workunit.client.1.smithi189.stdout:4/782: sync 2024-04-20T21:32:23.391 INFO:tasks.workunit.client.1.smithi189.stdout:7/917: sync 2024-04-20T21:32:23.395 INFO:tasks.workunit.client.0.smithi062.stdout:6/959: mknod d0/d9/d11/d22/dd5/d67/d75/c13a 0 2024-04-20T21:32:23.395 INFO:tasks.workunit.client.1.smithi189.stdout:8/935: symlink da/d81/d123/d13/dd6/df0/df8/l149 0 2024-04-20T21:32:23.395 INFO:tasks.workunit.client.1.smithi189.stdout:6/866: symlink d8/d2b/l124 0 2024-04-20T21:32:23.395 INFO:tasks.workunit.client.1.smithi189.stdout:4/783: write d2/f6d [317199,117118] 0 2024-04-20T21:32:23.395 INFO:tasks.workunit.client.1.smithi189.stdout:2/810: dwrite d1/d17/d78/fac [4194304,4194304] 0 2024-04-20T21:32:23.396 INFO:tasks.workunit.client.1.smithi189.stdout:1/787: truncate d8/f1a 2766226 0 2024-04-20T21:32:23.396 INFO:tasks.workunit.client.1.smithi189.stdout:0/930: dwrite d3/da8/f115 [0,4194304] 0 2024-04-20T21:32:23.405 INFO:tasks.workunit.client.1.smithi189.stdout:5/857: creat d7/d9/d51/d70/d95/f111 x:0 0 0 2024-04-20T21:32:23.407 INFO:tasks.workunit.client.1.smithi189.stdout:9/808: mkdir d9/dc/d17/d95/da6/dea/d105 0 2024-04-20T21:32:23.408 INFO:tasks.workunit.client.1.smithi189.stdout:4/784: fdatasync d2/dd/d11/f9a 0 2024-04-20T21:32:23.412 INFO:tasks.workunit.client.0.smithi062.stdout:6/960: link d0/d9a/fae d0/dc7/d7c/dc5/de9/f13b 0 2024-04-20T21:32:23.413 INFO:tasks.workunit.client.1.smithi189.stdout:6/867: creat d8/d2b/d49/d108/f125 x:0 0 0 2024-04-20T21:32:23.417 INFO:tasks.workunit.client.1.smithi189.stdout:8/936: creat da/d81/d123/d20/d27/d6e/f14a x:0 0 0 2024-04-20T21:32:23.417 INFO:tasks.workunit.client.0.smithi062.stdout:6/961: rename d0/d9/f13 to d0/d4f/f13c 0 2024-04-20T21:32:23.420 INFO:tasks.workunit.client.1.smithi189.stdout:7/918: rename d5/c11a to d5/d59/c129 0 2024-04-20T21:32:23.420 INFO:tasks.workunit.client.1.smithi189.stdout:8/937: chown da/d81/d123/d3c/d68/d8b/d8d 71358512 1 2024-04-20T21:32:23.422 INFO:tasks.workunit.client.1.smithi189.stdout:6/868: truncate d8/da/dd/d118/f116 297061 0 2024-04-20T21:32:23.425 INFO:tasks.workunit.client.1.smithi189.stdout:6/869: stat d8/da/dd/d118/l101 0 2024-04-20T21:32:23.426 INFO:tasks.workunit.client.0.smithi062.stdout:6/962: mknod d0/d9/d11/d22/dd5/dff/c13d 0 2024-04-20T21:32:23.426 INFO:tasks.workunit.client.1.smithi189.stdout:5/858: creat d7/d9/d30/d55/d81/f112 x:0 0 0 2024-04-20T21:32:23.426 INFO:tasks.workunit.client.1.smithi189.stdout:2/811: dwrite d1/d11/d2d/f50 [0,4194304] 0 2024-04-20T21:32:23.429 INFO:tasks.workunit.client.1.smithi189.stdout:4/785: unlink d2/dd/d11/d14/c15 0 2024-04-20T21:32:23.432 INFO:tasks.workunit.client.1.smithi189.stdout:2/812: write d1/d11/d2d/d4f/d8e/fc0 [846459,2069] 0 2024-04-20T21:32:23.432 INFO:tasks.workunit.client.1.smithi189.stdout:9/809: rename d9/d12/d50/l54 to d9/dc/d17/d95/da0/d52/d82/da4/df3/l106 0 2024-04-20T21:32:23.433 INFO:tasks.workunit.client.0.smithi062.stdout:6/963: fdatasync d0/d1a/d44/fb8 0 2024-04-20T21:32:23.433 INFO:tasks.workunit.client.1.smithi189.stdout:1/788: mknod d8/df/d3a/d49/d8c/da2/dbe/dd5/c105 0 2024-04-20T21:32:23.437 INFO:tasks.workunit.client.1.smithi189.stdout:7/919: chown d5/d46/d7d/d45/l58 1 1 2024-04-20T21:32:23.440 INFO:tasks.workunit.client.1.smithi189.stdout:6/870: mkdir d8/d2b/d49/d108/d126 0 2024-04-20T21:32:23.440 INFO:tasks.workunit.client.1.smithi189.stdout:7/920: stat d5/d1d/d64/db9/ddd/f103 0 2024-04-20T21:32:23.443 INFO:tasks.workunit.client.1.smithi189.stdout:0/931: dread d3/df/d4c/dd4/fcd [0,4194304] 0 2024-04-20T21:32:23.443 INFO:tasks.workunit.client.1.smithi189.stdout:8/938: mkdir da/de/d1e/d137/d14b 0 2024-04-20T21:32:23.448 INFO:tasks.workunit.client.1.smithi189.stdout:4/786: rename d2/dd/d37 to d2/d20/d115 0 2024-04-20T21:32:23.451 INFO:tasks.workunit.client.1.smithi189.stdout:3/840: sync 2024-04-20T21:32:23.451 INFO:tasks.workunit.client.1.smithi189.stdout:7/921: mkdir d5/d1d/ded/d12a 0 2024-04-20T21:32:23.458 INFO:tasks.workunit.client.1.smithi189.stdout:8/939: rmdir da/de/d1e/d21 39 2024-04-20T21:32:23.459 INFO:tasks.workunit.client.1.smithi189.stdout:9/810: fdatasync d9/dc/f4d 0 2024-04-20T21:32:23.462 INFO:tasks.workunit.client.1.smithi189.stdout:8/940: write da/d81/d123/d3c/d50/d6c/dc7/fcd [3932127,102979] 0 2024-04-20T21:32:23.467 INFO:tasks.workunit.client.1.smithi189.stdout:5/859: rename d7/d9/d51/c89 to d7/d9/d31/d100/d4e/d5b/df1/dab/dcb/c113 0 2024-04-20T21:32:23.468 INFO:tasks.workunit.client.1.smithi189.stdout:5/860: dread - d7/d9/d28/da3/fa8 zero size 2024-04-20T21:32:23.470 INFO:tasks.workunit.client.1.smithi189.stdout:5/861: write d7/d35/fe8 [602239,68171] 0 2024-04-20T21:32:23.472 INFO:tasks.workunit.client.0.smithi062.stdout:6/964: dread d0/fb6 [4194304,4194304] 0 2024-04-20T21:32:23.472 INFO:tasks.workunit.client.1.smithi189.stdout:7/922: mknod d5/d1d/d64/db9/dcf/de7/c12b 0 2024-04-20T21:32:23.476 INFO:tasks.workunit.client.0.smithi062.stdout:6/965: mkdir d0/dc7/d7c/dc5/def/d13e 0 2024-04-20T21:32:23.477 INFO:tasks.workunit.client.1.smithi189.stdout:9/811: symlink d9/dc/d17/d95/da0/d52/d82/da4/df3/l107 0 2024-04-20T21:32:23.477 INFO:tasks.workunit.client.1.smithi189.stdout:5/862: truncate d7/d9/d30/f8e 5006517 0 2024-04-20T21:32:23.477 INFO:tasks.workunit.client.1.smithi189.stdout:8/941: creat da/d81/d123/d3a/dfe/d13b/f14c x:0 0 0 2024-04-20T21:32:23.478 INFO:tasks.workunit.client.1.smithi189.stdout:2/813: link d1/d17/d37/f44 d1/fea 0 2024-04-20T21:32:23.479 INFO:tasks.workunit.client.1.smithi189.stdout:9/812: chown d9/d12/d3b 3 1 2024-04-20T21:32:23.479 INFO:tasks.workunit.client.1.smithi189.stdout:3/841: creat d2/d3/d2c/d30/d10c/d112/f126 x:0 0 0 2024-04-20T21:32:23.480 INFO:tasks.workunit.client.0.smithi062.stdout:6/966: chown d0/d1a/d7d/db1/db9/c131 530 1 2024-04-20T21:32:23.482 INFO:tasks.workunit.client.1.smithi189.stdout:0/932: dwrite d3/df/d4c/dd4/fcd [0,4194304] 0 2024-04-20T21:32:23.486 INFO:tasks.workunit.client.1.smithi189.stdout:8/942: read da/fb [8003239,106925] 0 2024-04-20T21:32:23.488 INFO:tasks.workunit.client.1.smithi189.stdout:2/814: write d1/d11/d2d/d4f/d8e/fe2 [1000320,87979] 0 2024-04-20T21:32:23.491 INFO:tasks.workunit.client.1.smithi189.stdout:6/871: dwrite d8/da/dd/d118/d30/d47/fc9 [0,4194304] 0 2024-04-20T21:32:23.491 INFO:tasks.workunit.client.1.smithi189.stdout:4/787: dread d2/dd/fe [0,4194304] 0 2024-04-20T21:32:23.493 INFO:tasks.workunit.client.1.smithi189.stdout:0/933: creat d3/df/d28/d2b/d2c/f136 x:0 0 0 2024-04-20T21:32:23.496 INFO:tasks.workunit.client.1.smithi189.stdout:3/842: unlink d2/d3/d65/f78 0 2024-04-20T21:32:23.498 INFO:tasks.workunit.client.0.smithi062.stdout:6/967: dwrite d0/d9/d11/f1f [0,4194304] 0 2024-04-20T21:32:23.498 INFO:tasks.workunit.client.1.smithi189.stdout:0/934: write d3/d26/fe3 [700167,87174] 0 2024-04-20T21:32:23.502 INFO:tasks.workunit.client.1.smithi189.stdout:3/843: write d2/d3/d25/de0/db0/dc8/ffa [251812,28792] 0 2024-04-20T21:32:23.503 INFO:tasks.workunit.client.0.smithi062.stdout:6/968: symlink d0/d52/da0/l13f 0 2024-04-20T21:32:23.505 INFO:tasks.workunit.client.1.smithi189.stdout:8/943: rename da/d81/d123/d3c/d9d/fec to da/d81/d123/d20/d65/f14d 0 2024-04-20T21:32:23.505 INFO:tasks.workunit.client.1.smithi189.stdout:6/872: mkdir d8/d127 0 2024-04-20T21:32:23.512 INFO:tasks.workunit.client.1.smithi189.stdout:9/813: write d9/dc/d17/d95/da0/d52/d82/d94/fc4 [202217,17633] 0 2024-04-20T21:32:23.514 INFO:tasks.workunit.client.1.smithi189.stdout:9/814: stat d9/d12/d3f/d91 0 2024-04-20T21:32:23.517 INFO:tasks.workunit.client.1.smithi189.stdout:0/935: mknod d3/df/d28/d2b/db1/dfe/d119/c137 0 2024-04-20T21:32:23.517 INFO:tasks.workunit.client.1.smithi189.stdout:2/815: rename d1 to d1/d11/d52/deb 22 2024-04-20T21:32:23.518 INFO:tasks.workunit.client.1.smithi189.stdout:3/844: read - d2/d1c/dc1/dc5/ddf/f117 zero size 2024-04-20T21:32:23.521 INFO:tasks.workunit.client.1.smithi189.stdout:4/788: creat d2/d20/d115/dcd/d6a/d7a/db7/d10f/f116 x:0 0 0 2024-04-20T21:32:23.523 INFO:tasks.workunit.client.1.smithi189.stdout:2/816: fsync d1/d11/d58/fd5 0 2024-04-20T21:32:23.525 INFO:tasks.workunit.client.1.smithi189.stdout:2/817: fdatasync d1/d11/d2d/d4f/d8e/fc0 0 2024-04-20T21:32:23.525 INFO:tasks.workunit.client.1.smithi189.stdout:3/845: creat d2/d3/d25/de0/db0/f127 x:0 0 0 2024-04-20T21:32:23.526 INFO:tasks.workunit.client.1.smithi189.stdout:4/789: symlink d2/d3a/l117 0 2024-04-20T21:32:23.529 INFO:tasks.workunit.client.0.smithi062.stdout:6/969: dwrite d0/d9/d11/d7b/fc4 [0,4194304] 0 2024-04-20T21:32:23.529 INFO:tasks.workunit.client.1.smithi189.stdout:9/815: creat d9/dc/d17/d95/da0/d52/d82/f108 x:0 0 0 2024-04-20T21:32:23.533 INFO:tasks.workunit.client.1.smithi189.stdout:6/873: mknod d8/da/da7/c128 0 2024-04-20T21:32:23.535 INFO:tasks.workunit.client.1.smithi189.stdout:3/846: fsync d2/d3/d2c/f6b 0 2024-04-20T21:32:23.535 INFO:tasks.workunit.client.1.smithi189.stdout:9/816: stat d9/dc/d17/d95/da0/d55/dd0/lec 0 2024-04-20T21:32:23.535 INFO:tasks.workunit.client.1.smithi189.stdout:0/936: rename d3/df/d30/d45/lea to d3/df/d5d/l138 0 2024-04-20T21:32:23.537 INFO:tasks.workunit.client.1.smithi189.stdout:1/789: dread d8/f1a [0,4194304] 0 2024-04-20T21:32:23.538 INFO:tasks.workunit.client.0.smithi062.stdout:6/970: symlink d0/d9/d11/d22/dd5/dff/l140 0 2024-04-20T21:32:23.542 INFO:tasks.workunit.client.1.smithi189.stdout:8/944: dwrite da/de/d1e/d69/f10c [0,4194304] 0 2024-04-20T21:32:23.550 INFO:tasks.workunit.client.1.smithi189.stdout:3/847: dread - d2/d3/d36/da1/fa4 zero size 2024-04-20T21:32:23.551 INFO:tasks.workunit.client.1.smithi189.stdout:6/874: fdatasync d8/d2b/d49/d54/d6e/fb0 0 2024-04-20T21:32:23.553 INFO:tasks.workunit.client.1.smithi189.stdout:8/945: write da/d1f/f2f [1285270,19540] 0 2024-04-20T21:32:23.559 INFO:tasks.workunit.client.1.smithi189.stdout:9/817: read d9/dc/d17/d95/da0/d52/f83 [97369,68721] 0 2024-04-20T21:32:23.561 INFO:tasks.workunit.client.1.smithi189.stdout:6/875: dread - d8/da/dd/d118/d99/dd8/feb zero size 2024-04-20T21:32:23.564 INFO:tasks.workunit.client.0.smithi062.stdout:6/971: dwrite d0/d9/d11/d22/dd5/d67/fe7 [0,4194304] 0 2024-04-20T21:32:23.566 INFO:tasks.workunit.client.1.smithi189.stdout:4/790: dread d2/dd/d11/d14/f3f [4194304,4194304] 0 2024-04-20T21:32:23.567 INFO:tasks.workunit.client.0.smithi062.stdout:6/972: truncate d0/f5e 5167439 0 2024-04-20T21:32:23.571 INFO:tasks.workunit.client.0.smithi062.stdout:6/973: write d0/d52/fdc [1550450,112404] 0 2024-04-20T21:32:23.571 INFO:tasks.workunit.client.1.smithi189.stdout:2/818: dread d1/d11/d2d/d4f/d8e/fc4 [0,4194304] 0 2024-04-20T21:32:23.578 INFO:tasks.workunit.client.1.smithi189.stdout:9/818: mknod d9/dc/d17/d95/c109 0 2024-04-20T21:32:23.584 INFO:tasks.workunit.client.1.smithi189.stdout:0/937: dwrite d3/df/f76 [0,4194304] 0 2024-04-20T21:32:23.591 INFO:tasks.workunit.client.1.smithi189.stdout:1/790: dwrite d8/df/d3a/d49/d8c/da2/dfe/f80 [0,4194304] 0 2024-04-20T21:32:23.591 INFO:tasks.workunit.client.1.smithi189.stdout:0/938: fdatasync d3/df/d30/d37/dbd/de1/f109 0 2024-04-20T21:32:23.591 INFO:tasks.workunit.client.1.smithi189.stdout:6/876: write d8/da/dd/d118/daf/f107 [3010482,89706] 0 2024-04-20T21:32:23.594 INFO:tasks.workunit.client.1.smithi189.stdout:7/923: sync 2024-04-20T21:32:23.594 INFO:tasks.workunit.client.1.smithi189.stdout:5/863: sync 2024-04-20T21:32:23.595 INFO:tasks.workunit.client.1.smithi189.stdout:9/819: read d9/dc/d17/d36/d5d/fa5 [2438619,97503] 0 2024-04-20T21:32:23.597 INFO:tasks.workunit.client.1.smithi189.stdout:0/939: rmdir d3/df/d28/dc8/df1 39 2024-04-20T21:32:23.598 INFO:tasks.workunit.client.1.smithi189.stdout:5/864: stat d7/d9/d51/d70/d99/fd8 0 2024-04-20T21:32:23.598 INFO:tasks.workunit.client.1.smithi189.stdout:1/791: truncate d8/d1e/fca 899835 0 2024-04-20T21:32:23.599 INFO:tasks.workunit.client.1.smithi189.stdout:3/848: dwrite d2/f2a [4194304,4194304] 0 2024-04-20T21:32:23.603 INFO:tasks.workunit.client.1.smithi189.stdout:0/940: chown d3/db9/dbe/dd7/d112 23 1 2024-04-20T21:32:23.603 INFO:tasks.workunit.client.1.smithi189.stdout:7/924: creat d5/d21/d4a/d52/d98/f12c x:0 0 0 2024-04-20T21:32:23.604 INFO:tasks.workunit.client.1.smithi189.stdout:9/820: symlink d9/d12/d3f/d91/l10a 0 2024-04-20T21:32:23.605 INFO:tasks.workunit.client.1.smithi189.stdout:1/792: truncate d8/d1e/d54/ff7 71935 0 2024-04-20T21:32:23.608 INFO:tasks.workunit.client.1.smithi189.stdout:5/865: read - d7/d9/d51/d70/f108 zero size 2024-04-20T21:32:23.609 INFO:tasks.workunit.client.1.smithi189.stdout:3/849: mknod d2/d1c/dc1/dc5/ddf/c128 0 2024-04-20T21:32:23.613 INFO:tasks.workunit.client.1.smithi189.stdout:0/941: mknod d3/df/d28/d2b/c139 0 2024-04-20T21:32:23.616 INFO:tasks.workunit.client.1.smithi189.stdout:1/793: fdatasync d8/d1e/d43/d44/dd2/db7/f101 0 2024-04-20T21:32:23.619 INFO:tasks.workunit.client.1.smithi189.stdout:9/821: creat d9/dc/d17/d95/da0/d52/f10b x:0 0 0 2024-04-20T21:32:23.619 INFO:tasks.workunit.client.1.smithi189.stdout:5/866: creat d7/d9/d30/d55/d104/d106/f114 x:0 0 0 2024-04-20T21:32:23.621 INFO:tasks.workunit.client.1.smithi189.stdout:1/794: write d8/d21/fdc [4761134,88951] 0 2024-04-20T21:32:23.621 INFO:tasks.workunit.client.1.smithi189.stdout:0/942: write d3/df/d28/d2b/db1/f107 [4782,78994] 0 2024-04-20T21:32:23.622 INFO:tasks.workunit.client.1.smithi189.stdout:3/850: read d2/d3/d2c/d31/fd5 [111304,8102] 0 2024-04-20T21:32:23.628 INFO:tasks.workunit.client.0.smithi062.stdout:6/974: dread d0/dc7/d7c/dc5/de9/f13b [0,4194304] 0 2024-04-20T21:32:23.628 INFO:tasks.workunit.client.1.smithi189.stdout:9/822: rmdir d9/dc/d17 39 2024-04-20T21:32:23.630 INFO:tasks.workunit.client.1.smithi189.stdout:7/925: dread d5/f49 [0,4194304] 0 2024-04-20T21:32:23.633 INFO:tasks.workunit.client.0.smithi062.stdout:6/975: stat d0/d9/d11/d22/lf9 0 2024-04-20T21:32:23.635 INFO:tasks.workunit.client.1.smithi189.stdout:2/819: dread d1/d11/d58/d6b/f77 [0,4194304] 0 2024-04-20T21:32:23.635 INFO:tasks.workunit.client.1.smithi189.stdout:5/867: write d7/d9/d31/f88 [4275653,130289] 0 2024-04-20T21:32:23.636 INFO:tasks.workunit.client.1.smithi189.stdout:1/795: fdatasync d8/d1e/d43/d58/fe7 0 2024-04-20T21:32:23.640 INFO:tasks.workunit.client.1.smithi189.stdout:8/946: dread da/d81/d123/d20/d2c/d9a/fcc [0,4194304] 0 2024-04-20T21:32:23.640 INFO:tasks.workunit.client.1.smithi189.stdout:3/851: fdatasync d2/d3/d2c/d31/d74/d11c/f123 0 2024-04-20T21:32:23.641 INFO:tasks.workunit.client.1.smithi189.stdout:6/877: dwrite d8/d2b/d49/d54/d6e/fb4 [4194304,4194304] 0 2024-04-20T21:32:23.643 INFO:tasks.workunit.client.1.smithi189.stdout:0/943: rename d3/df/d28/lf0 to d3/db9/dbe/l13a 0 2024-04-20T21:32:23.644 INFO:tasks.workunit.client.1.smithi189.stdout:4/791: dread d2/fd9 [0,4194304] 0 2024-04-20T21:32:23.647 INFO:tasks.workunit.client.1.smithi189.stdout:6/878: chown d8/d2b/d49/d54/d6e/ff8 79309931 1 2024-04-20T21:32:23.647 INFO:tasks.workunit.client.1.smithi189.stdout:5/868: readlink d7/d9/d30/d55/d81/lfb 0 2024-04-20T21:32:23.647 INFO:tasks.workunit.client.1.smithi189.stdout:2/820: creat d1/d11/d2d/dab/d4a/fec x:0 0 0 2024-04-20T21:32:23.649 INFO:tasks.workunit.client.1.smithi189.stdout:8/947: chown da/d81/d123/d13/dd6/dba/dc5/c115 324151 1 2024-04-20T21:32:23.655 INFO:tasks.workunit.client.1.smithi189.stdout:2/821: stat d1/d11/d2d/d4f/db4/l27 0 2024-04-20T21:32:23.657 INFO:tasks.workunit.client.1.smithi189.stdout:1/796: mkdir d8/d1e/d43/d44/dd2/d106 0 2024-04-20T21:32:23.661 INFO:tasks.workunit.client.1.smithi189.stdout:4/792: rename d2/d20/fc4 to d2/d20/dc6/dda/f118 0 2024-04-20T21:32:23.664 INFO:tasks.workunit.client.1.smithi189.stdout:3/852: getdents d2/d1c/dc1/d11e 0 2024-04-20T21:32:23.664 INFO:tasks.workunit.client.1.smithi189.stdout:2/822: chown d1/d11/d58/cc2 3874 1 2024-04-20T21:32:23.664 INFO:tasks.workunit.client.1.smithi189.stdout:5/869: mkdir d7/d9/d31/d36/de0/d115 0 2024-04-20T21:32:23.668 INFO:tasks.workunit.client.1.smithi189.stdout:6/879: write d8/da/dd/d35/d6d/d7b/f109 [598656,63580] 0 2024-04-20T21:32:23.668 INFO:tasks.workunit.client.1.smithi189.stdout:1/797: creat d8/d1e/d43/d44/db2/f107 x:0 0 0 2024-04-20T21:32:23.669 INFO:tasks.workunit.client.1.smithi189.stdout:9/823: dwrite d9/dc/d17/d95/da0/d52/f10b [0,4194304] 0 2024-04-20T21:32:23.672 INFO:tasks.workunit.client.1.smithi189.stdout:8/948: write da/d1f/d2a/fdf [130578,63764] 0 2024-04-20T21:32:23.672 INFO:tasks.workunit.client.1.smithi189.stdout:4/793: mknod d2/c119 0 2024-04-20T21:32:23.672 INFO:tasks.workunit.client.1.smithi189.stdout:0/944: write d3/df/d28/d2b/db1/dfe/f101 [2722529,79547] 0 2024-04-20T21:32:23.674 INFO:tasks.workunit.client.1.smithi189.stdout:1/798: chown d8/df/c55 18 1 2024-04-20T21:32:23.683 INFO:tasks.workunit.client.1.smithi189.stdout:6/880: unlink d8/d2b/f8c 0 2024-04-20T21:32:23.683 INFO:tasks.workunit.client.1.smithi189.stdout:3/853: mknod d2/d1c/d5c/dca/c129 0 2024-04-20T21:32:23.683 INFO:tasks.workunit.client.1.smithi189.stdout:2/823: rename d1/d17/d37/f75 to d1/d11/d52/d8b/fed 0 2024-04-20T21:32:23.684 INFO:tasks.workunit.client.1.smithi189.stdout:9/824: truncate d9/dc/d17/d95/fe4 491147 0 2024-04-20T21:32:23.687 INFO:tasks.workunit.client.1.smithi189.stdout:0/945: mkdir d3/db9/d13b 0 2024-04-20T21:32:23.687 INFO:tasks.workunit.client.1.smithi189.stdout:2/824: dread - d1/d11/d2d/d4f/fe9 zero size 2024-04-20T21:32:23.692 INFO:tasks.workunit.client.1.smithi189.stdout:6/881: truncate d8/da/dd/d118/d30/d47/da0/f100 489317 0 2024-04-20T21:32:23.694 INFO:tasks.workunit.client.1.smithi189.stdout:2/825: read d1/d11/d2d/d4f/fbc [2070301,9069] 0 2024-04-20T21:32:23.697 INFO:tasks.workunit.client.1.smithi189.stdout:0/946: symlink d3/db9/d13b/l13c 0 2024-04-20T21:32:23.697 INFO:tasks.workunit.client.1.smithi189.stdout:7/926: dread d5/db4/fe5 [0,4194304] 0 2024-04-20T21:32:23.700 INFO:tasks.workunit.client.1.smithi189.stdout:3/854: write d2/d3/d25/de0/fb5 [561263,8670] 0 2024-04-20T21:32:23.704 INFO:tasks.workunit.client.1.smithi189.stdout:4/794: truncate d2/dd/d11/f7e 1369869 0 2024-04-20T21:32:23.707 INFO:tasks.workunit.client.1.smithi189.stdout:8/949: dread da/de/d1e/d21/d87/ff6 [0,4194304] 0 2024-04-20T21:32:23.708 INFO:tasks.workunit.client.1.smithi189.stdout:3/855: mkdir d2/d3/d25/de0/da9/d12a 0 2024-04-20T21:32:23.710 INFO:tasks.workunit.client.1.smithi189.stdout:9/825: dwrite d9/d12/d50/f4f [4194304,4194304] 0 2024-04-20T21:32:23.710 INFO:tasks.workunit.client.1.smithi189.stdout:7/927: write d5/d46/d7d/fd9 [177869,89795] 0 2024-04-20T21:32:23.715 INFO:tasks.workunit.client.1.smithi189.stdout:0/947: getdents d3/df/d5d/d64 0 2024-04-20T21:32:23.717 INFO:tasks.workunit.client.1.smithi189.stdout:8/950: mkdir da/d81/d123/d13/d63/d7b/d14e 0 2024-04-20T21:32:23.718 INFO:tasks.workunit.client.1.smithi189.stdout:9/826: creat d9/d12/d50/f10c x:0 0 0 2024-04-20T21:32:23.719 INFO:tasks.workunit.client.0.smithi062.stdout:6/976: write d0/d52/fce [807669,34033] 0 2024-04-20T21:32:23.720 INFO:tasks.workunit.client.1.smithi189.stdout:6/882: dwrite d8/da/dd/d2a/f85 [0,4194304] 0 2024-04-20T21:32:23.722 INFO:tasks.workunit.client.1.smithi189.stdout:3/856: getdents d2/d1c/dc1/d11e 0 2024-04-20T21:32:23.723 INFO:tasks.workunit.client.1.smithi189.stdout:7/928: unlink d5/d21/d4a/f8d 0 2024-04-20T21:32:23.724 INFO:tasks.workunit.client.1.smithi189.stdout:9/827: dread - d9/dc/d17/d95/fd5 zero size 2024-04-20T21:32:23.726 INFO:tasks.workunit.client.1.smithi189.stdout:3/857: write d2/d8/f4b [3365629,60187] 0 2024-04-20T21:32:23.726 INFO:tasks.workunit.client.1.smithi189.stdout:7/929: chown d5/lad 6634 1 2024-04-20T21:32:23.729 INFO:tasks.workunit.client.0.smithi062.stdout:6/977: rename d0/d9/d11/d22/dd5/dff/c13d to d0/d9/d11/d7b/d99/ddd/c141 0 2024-04-20T21:32:23.731 INFO:tasks.workunit.client.1.smithi189.stdout:6/883: chown d8/da/dd/d118/daf 41 1 2024-04-20T21:32:23.731 INFO:tasks.workunit.client.1.smithi189.stdout:8/951: rmdir da/d81/d123/d3c/d50/d6c/dc7 39 2024-04-20T21:32:23.733 INFO:tasks.workunit.client.1.smithi189.stdout:7/930: chown d5/db4/d71/d7a/db0/cf3 73877346 1 2024-04-20T21:32:23.734 INFO:tasks.workunit.client.1.smithi189.stdout:4/795: dread d2/d20/d115/ff6 [0,4194304] 0 2024-04-20T21:32:23.734 INFO:tasks.workunit.client.0.smithi062.stdout:6/978: read d0/d9/d11/d7b/fb2 [748041,50550] 0 2024-04-20T21:32:23.737 INFO:tasks.workunit.client.1.smithi189.stdout:9/828: creat d9/d12/d62/dd9/f10d x:0 0 0 2024-04-20T21:32:23.739 INFO:tasks.workunit.client.1.smithi189.stdout:3/858: truncate d2/d3/d25/f97 953069 0 2024-04-20T21:32:23.739 INFO:tasks.workunit.client.0.smithi062.stdout:6/979: mknod d0/d9/d11/d7b/d99/ddd/c142 0 2024-04-20T21:32:23.739 INFO:tasks.workunit.client.1.smithi189.stdout:9/829: readlink d9/dc/d17/d36/d5d/l6d 0 2024-04-20T21:32:23.739 INFO:tasks.workunit.client.1.smithi189.stdout:7/931: write d5/db4/d71/d7a/f113 [531093,74438] 0 2024-04-20T21:32:23.739 INFO:tasks.workunit.client.1.smithi189.stdout:6/884: creat d8/da/dd/d118/d4e/dc4/f129 x:0 0 0 2024-04-20T21:32:23.743 INFO:tasks.workunit.client.1.smithi189.stdout:3/859: chown d2/d1c/dc1 2 1 2024-04-20T21:32:23.743 INFO:tasks.workunit.client.0.smithi062.stdout:6/980: mknod d0/d1a/d7d/db1/db9/c143 0 2024-04-20T21:32:23.744 INFO:tasks.workunit.client.1.smithi189.stdout:6/885: chown d8/da/dd/d2a/cb6 525 1 2024-04-20T21:32:23.744 INFO:tasks.workunit.client.1.smithi189.stdout:4/796: mkdir d2/d20/d115/dcd/d94/d107/d11a 0 2024-04-20T21:32:23.746 INFO:tasks.workunit.client.1.smithi189.stdout:7/932: mknod d5/d21/d97/da3/c12d 0 2024-04-20T21:32:23.749 INFO:tasks.workunit.client.1.smithi189.stdout:3/860: mkdir d2/d1c/d12b 0 2024-04-20T21:32:23.749 INFO:tasks.workunit.client.1.smithi189.stdout:4/797: truncate d2/d22/d87/de1/f111 161645 0 2024-04-20T21:32:23.751 INFO:tasks.workunit.client.1.smithi189.stdout:6/886: fsync d8/f63 0 2024-04-20T21:32:23.751 INFO:tasks.workunit.client.1.smithi189.stdout:7/933: dread - d5/d21/d4a/d52/d98/f9b zero size 2024-04-20T21:32:23.753 INFO:tasks.workunit.client.0.smithi062.stdout:6/981: link d0/d1a/d7d/f122 d0/d9/d11/d7b/d99/ddd/f144 0 2024-04-20T21:32:23.756 INFO:tasks.workunit.client.0.smithi062.stdout:6/982: chown d0/d9/f81 98000308 1 2024-04-20T21:32:23.757 INFO:tasks.workunit.client.1.smithi189.stdout:0/948: dread d3/df/d28/d2b/db1/dfe/f101 [0,4194304] 0 2024-04-20T21:32:23.757 INFO:tasks.workunit.client.0.smithi062.stdout:6/983: read - d0/d9/d11/d7b/d99/ddd/f12b zero size 2024-04-20T21:32:23.759 INFO:tasks.workunit.client.1.smithi189.stdout:7/934: creat d5/d1d/d64/db9/dcf/f12e x:0 0 0 2024-04-20T21:32:23.760 INFO:tasks.workunit.client.1.smithi189.stdout:8/952: dwrite da/d81/d123/d20/dc9/f10b [0,4194304] 0 2024-04-20T21:32:23.761 INFO:tasks.workunit.client.1.smithi189.stdout:4/798: write d2/d20/d115/dcd/d6a/d7a/dce/ff4 [5357101,53266] 0 2024-04-20T21:32:23.764 INFO:tasks.workunit.client.1.smithi189.stdout:6/887: creat d8/da/dd/d118/dd7/f12a x:0 0 0 2024-04-20T21:32:23.764 INFO:tasks.workunit.client.1.smithi189.stdout:0/949: mkdir d3/df/d30/d45/dd3/d13d 0 2024-04-20T21:32:23.766 INFO:tasks.workunit.client.0.smithi062.stdout:6/984: link d0/fb5 d0/d9/d11/d7b/f145 0 2024-04-20T21:32:23.766 INFO:tasks.workunit.client.1.smithi189.stdout:8/953: fsync da/d1f/fd8 0 2024-04-20T21:32:23.770 INFO:tasks.workunit.client.1.smithi189.stdout:9/830: dwrite d9/dc/ffb [0,4194304] 0 2024-04-20T21:32:23.771 INFO:tasks.workunit.client.1.smithi189.stdout:0/950: mknod d3/df/d28/d2b/c13e 0 2024-04-20T21:32:23.774 INFO:tasks.workunit.client.1.smithi189.stdout:8/954: write da/de/d1e/d69/f10c [4644844,10287] 0 2024-04-20T21:32:23.779 INFO:tasks.workunit.client.1.smithi189.stdout:0/951: mknod d3/df/d30/d45/d90/df4/c13f 0 2024-04-20T21:32:23.781 INFO:tasks.workunit.client.0.smithi062.stdout:6/985: write d0/d9/d11/fda [917607,121628] 0 2024-04-20T21:32:23.782 INFO:tasks.workunit.client.1.smithi189.stdout:0/952: chown d3/df/d5d/f133 5 1 2024-04-20T21:32:23.782 INFO:tasks.workunit.client.1.smithi189.stdout:8/955: rename l2 to da/d81/d123/d20/d27/l14f 0 2024-04-20T21:32:23.786 INFO:tasks.workunit.client.1.smithi189.stdout:9/831: link d9/dc/d17/d36/f78 d9/dc/d17/d95/d102/f10e 0 2024-04-20T21:32:23.788 INFO:tasks.workunit.client.1.smithi189.stdout:3/861: dwrite d2/d3/d25/f97 [0,4194304] 0 2024-04-20T21:32:23.789 INFO:tasks.workunit.client.0.smithi062.stdout:6/986: mknod d0/d52/c146 0 2024-04-20T21:32:23.793 INFO:tasks.workunit.client.0.smithi062.stdout:6/987: dread - d0/fb5 zero size 2024-04-20T21:32:23.799 INFO:tasks.workunit.client.0.smithi062.stdout:6/988: truncate d0/d9/d11/fbc 4829757 0 2024-04-20T21:32:23.800 INFO:tasks.workunit.client.1.smithi189.stdout:5/870: sync 2024-04-20T21:32:23.800 INFO:tasks.workunit.client.1.smithi189.stdout:6/888: write d8/da/dd/d2a/fdf [724251,97914] 0 2024-04-20T21:32:23.802 INFO:tasks.workunit.client.1.smithi189.stdout:4/799: dwrite d2/d3a/f91 [0,4194304] 0 2024-04-20T21:32:23.809 INFO:tasks.workunit.client.0.smithi062.stdout:6/989: creat d0/d1a/d7d/db1/db9/d11d/f147 x:0 0 0 2024-04-20T21:32:23.812 INFO:tasks.workunit.client.1.smithi189.stdout:5/871: mkdir d7/d9/d30/d55/d61/daa/d116 0 2024-04-20T21:32:23.815 INFO:tasks.workunit.client.1.smithi189.stdout:4/800: symlink d2/d20/d115/dcd/d6a/d7a/def/de8/l11b 0 2024-04-20T21:32:23.817 INFO:tasks.workunit.client.0.smithi062.stdout:6/990: getdents d0/dc7/d7c 0 2024-04-20T21:32:23.817 INFO:tasks.workunit.client.1.smithi189.stdout:7/935: dread d5/d21/d4a/d52/f5c [0,4194304] 0 2024-04-20T21:32:23.820 INFO:tasks.workunit.client.0.smithi062.stdout:6/991: write d0/d9/f138 [482171,35285] 0 2024-04-20T21:32:23.822 INFO:tasks.workunit.client.1.smithi189.stdout:3/862: write d2/d8d/f91 [1052353,88846] 0 2024-04-20T21:32:23.824 INFO:tasks.workunit.client.1.smithi189.stdout:4/801: write d2/dd/d11/d14/f79 [2691978,9940] 0 2024-04-20T21:32:23.830 INFO:tasks.workunit.client.1.smithi189.stdout:4/802: rmdir d2/d20/dc6 39 2024-04-20T21:32:23.834 INFO:tasks.workunit.client.1.smithi189.stdout:6/889: dwrite d8/d2b/f39 [0,4194304] 0 2024-04-20T21:32:23.834 INFO:tasks.workunit.client.1.smithi189.stdout:7/936: mknod d5/d46/d7d/d111/d11b/c12f 0 2024-04-20T21:32:23.836 INFO:tasks.workunit.client.1.smithi189.stdout:0/953: dwrite d3/df/d4c/dd4/fe6 [0,4194304] 0 2024-04-20T21:32:23.839 INFO:tasks.workunit.client.1.smithi189.stdout:3/863: rename d2/f6 to d2/d1c/d5c/f12c 0 2024-04-20T21:32:23.839 INFO:tasks.workunit.client.1.smithi189.stdout:0/954: readlink d3/df/l13 0 2024-04-20T21:32:23.840 INFO:tasks.workunit.client.1.smithi189.stdout:9/832: dwrite d9/dc/fa2 [0,4194304] 0 2024-04-20T21:32:23.841 INFO:tasks.workunit.client.0.smithi062.stdout:6/992: write d0/d52/f83 [217302,94244] 0 2024-04-20T21:32:23.842 INFO:tasks.workunit.client.1.smithi189.stdout:7/937: readlink d5/d46/l95 0 2024-04-20T21:32:23.842 INFO:tasks.workunit.client.1.smithi189.stdout:4/803: unlink d2/d20/dc6/dda/cf1 0 2024-04-20T21:32:23.844 INFO:tasks.workunit.client.1.smithi189.stdout:6/890: symlink d8/d2b/d49/d108/d111/l12b 0 2024-04-20T21:32:23.844 INFO:tasks.workunit.client.1.smithi189.stdout:1/799: sync 2024-04-20T21:32:23.844 INFO:tasks.workunit.client.1.smithi189.stdout:2/826: sync 2024-04-20T21:32:23.849 INFO:tasks.workunit.client.0.smithi062.stdout:6/993: rename d0/d9/d11/d7b/fb2 to d0/d9a/f148 0 2024-04-20T21:32:23.851 INFO:tasks.workunit.client.1.smithi189.stdout:2/827: stat d1/d11/d2d/d4f/db4/ca4 0 2024-04-20T21:32:23.852 INFO:tasks.workunit.client.1.smithi189.stdout:0/955: fsync d3/d26/d88/fa0 0 2024-04-20T21:32:23.852 INFO:tasks.workunit.client.1.smithi189.stdout:8/956: dread da/d1f/f25 [0,4194304] 0 2024-04-20T21:32:23.852 INFO:tasks.workunit.client.0.smithi062.stdout:6/994: read - d0/d9/d11/d7b/f145 zero size 2024-04-20T21:32:23.854 INFO:tasks.workunit.client.1.smithi189.stdout:9/833: creat d9/dc/d17/d36/d5d/f10f x:0 0 0 2024-04-20T21:32:23.855 INFO:tasks.workunit.client.1.smithi189.stdout:6/891: stat d8/d2b/d49/d108/d111 0 2024-04-20T21:32:23.862 INFO:tasks.workunit.client.1.smithi189.stdout:9/834: write d9/dc/f2c [2762518,87107] 0 2024-04-20T21:32:23.863 INFO:tasks.workunit.client.1.smithi189.stdout:5/872: dwrite d7/d9/d51/f72 [0,4194304] 0 2024-04-20T21:32:23.865 INFO:tasks.workunit.client.1.smithi189.stdout:2/828: rename d1/d17/fd6 to d1/d11/d58/d6b/fee 0 2024-04-20T21:32:23.867 INFO:tasks.workunit.client.1.smithi189.stdout:1/800: creat d8/d9b/dbb/f108 x:0 0 0 2024-04-20T21:32:23.870 INFO:tasks.workunit.client.1.smithi189.stdout:8/957: fdatasync da/d81/d123/d20/d2c/f141 0 2024-04-20T21:32:23.871 INFO:tasks.workunit.client.1.smithi189.stdout:9/835: readlink d9/dc/d17/d7e/l92 0 2024-04-20T21:32:23.873 INFO:tasks.workunit.client.1.smithi189.stdout:3/864: creat d2/d3/d25/de0/db0/dc8/f12d x:0 0 0 2024-04-20T21:32:23.873 INFO:tasks.workunit.client.1.smithi189.stdout:2/829: chown d1/f8c 87198 1 2024-04-20T21:32:23.877 INFO:tasks.workunit.client.1.smithi189.stdout:3/865: chown d2/d3/d2c/fda 2886 1 2024-04-20T21:32:23.877 INFO:tasks.workunit.client.1.smithi189.stdout:2/830: fdatasync d1/d17/d78/db9/fc7 0 2024-04-20T21:32:23.878 INFO:tasks.workunit.client.1.smithi189.stdout:8/958: symlink da/d81/d123/d3c/l150 0 2024-04-20T21:32:23.879 INFO:tasks.workunit.client.1.smithi189.stdout:1/801: chown d8/d1e/d28/d2b/c45 3931 1 2024-04-20T21:32:23.881 INFO:tasks.workunit.client.1.smithi189.stdout:8/959: dread - da/d1f/fd8 zero size 2024-04-20T21:32:23.882 INFO:tasks.workunit.client.1.smithi189.stdout:0/956: dwrite d3/df/d30/d45/d90/fbf [0,4194304] 0 2024-04-20T21:32:23.884 INFO:tasks.workunit.client.1.smithi189.stdout:3/866: fsync d2/d1c/f76 0 2024-04-20T21:32:23.887 INFO:tasks.workunit.client.1.smithi189.stdout:1/802: fsync d8/d1e/d43/d44/dd2/fc7 0 2024-04-20T21:32:23.888 INFO:tasks.workunit.client.1.smithi189.stdout:9/836: link d9/dc/d17/d7e/ff9 d9/dc/d17/d7e/de8/f110 0 2024-04-20T21:32:23.891 INFO:tasks.workunit.client.1.smithi189.stdout:2/831: rename d1/d11/d52/d8b/fed to d1/d11/d52/d8b/fef 0 2024-04-20T21:32:23.892 INFO:tasks.workunit.client.1.smithi189.stdout:1/803: chown d8/df/d3a/d49/d8c/da2/dfe/f5f 382756411 1 2024-04-20T21:32:23.893 INFO:tasks.workunit.client.1.smithi189.stdout:2/832: chown d1/d17/d37/d89/dc5 221533994 1 2024-04-20T21:32:23.896 INFO:tasks.workunit.client.1.smithi189.stdout:2/833: write d1/d11/d2d/d4f/d8e/fc0 [249978,16531] 0 2024-04-20T21:32:23.898 INFO:tasks.workunit.client.1.smithi189.stdout:0/957: rename d3/df/d28/dc8/d10a to d3/df/d28/d2b/db1/dfe/d119/d140 0 2024-04-20T21:32:23.905 INFO:tasks.workunit.client.1.smithi189.stdout:8/960: rename da/de/d1e/d69/f13f to da/d81/d123/d13/dd6/df0/df8/f151 0 2024-04-20T21:32:23.906 INFO:tasks.workunit.client.1.smithi189.stdout:1/804: mknod d8/c109 0 2024-04-20T21:32:23.907 INFO:tasks.workunit.client.1.smithi189.stdout:2/834: mkdir d1/d11/de1/df0 0 2024-04-20T21:32:23.908 INFO:tasks.workunit.client.1.smithi189.stdout:4/804: truncate d2/d20/d115/f59 4935816 0 2024-04-20T21:32:23.910 INFO:tasks.workunit.client.1.smithi189.stdout:6/892: truncate d8/d2b/d49/d54/d67/f9c 606348 0 2024-04-20T21:32:23.910 INFO:tasks.workunit.client.1.smithi189.stdout:4/805: chown d2/dd/d11/f92 389476416 1 2024-04-20T21:32:23.912 INFO:tasks.workunit.client.1.smithi189.stdout:9/837: rename d9/dc/d17/d7e/de8/f110 to d9/dc/d17/d95/da6/dea/d105/f111 0 2024-04-20T21:32:23.913 INFO:tasks.workunit.client.1.smithi189.stdout:0/958: mkdir d3/df/d28/dc8/d121/d141 0 2024-04-20T21:32:23.921 INFO:tasks.workunit.client.0.smithi062.stdout:6/995: dread d0/d9/d11/d22/dd5/f8b [0,4194304] 0 2024-04-20T21:32:23.923 INFO:tasks.workunit.client.0.smithi062.stdout:6/996: mkdir d0/d1a/d7d/db1/db9/d149 0 2024-04-20T21:32:23.926 INFO:tasks.workunit.client.0.smithi062.stdout:6/997: creat d0/d1a/d7d/db1/db9/f14a x:0 0 0 2024-04-20T21:32:23.928 INFO:tasks.workunit.client.0.smithi062.stdout:6/998: chown d0/d1a/f105 4562544 1 2024-04-20T21:32:23.932 INFO:tasks.workunit.client.0.smithi062.stdout:6/999: rename d0/d1a/d7d/db1/de0 to d0/d52/da0/d14b 0 2024-04-20T21:32:23.941 INFO:tasks.workunit.client.0.smithi062.stderr:+ rm -rf -- ./tmp.uS6ubwsDVL 2024-04-20T21:32:23.973 INFO:tasks.workunit.client.1.smithi189.stdout:9/838: creat d9/d12/d3b/db1/d1a/f112 x:0 0 0 2024-04-20T21:32:23.974 INFO:tasks.workunit.client.1.smithi189.stdout:6/893: chown d8/d10/d78/d7e/dcf 8328 1 2024-04-20T21:32:23.976 INFO:tasks.workunit.client.1.smithi189.stdout:7/938: dread d5/d21/d4a/d52/d75/fda [0,4194304] 0 2024-04-20T21:32:23.978 INFO:tasks.workunit.client.1.smithi189.stdout:8/961: link da/de/l48 da/d81/d123/d3a/dfe/l152 0 2024-04-20T21:32:23.981 INFO:tasks.workunit.client.1.smithi189.stdout:3/867: dread d2/d3/d25/f67 [0,4194304] 0 2024-04-20T21:32:23.981 INFO:tasks.workunit.client.1.smithi189.stdout:1/805: link d8/d1e/d43/d44/dd2/d8a/lb5 d8/d21/d2c/l10a 0 2024-04-20T21:32:23.982 INFO:tasks.workunit.client.1.smithi189.stdout:6/894: creat d8/d127/f12c x:0 0 0 2024-04-20T21:32:23.983 INFO:tasks.workunit.client.1.smithi189.stdout:0/959: getdents d3/df/d30/d45/dd3 0 2024-04-20T21:32:23.984 INFO:tasks.workunit.client.1.smithi189.stdout:7/939: mknod d5/d21/d4a/d52/d75/dc8/c130 0 2024-04-20T21:32:23.990 INFO:tasks.workunit.client.1.smithi189.stdout:8/962: creat da/de/d1e/d137/d14b/f153 x:0 0 0 2024-04-20T21:32:23.992 INFO:tasks.workunit.client.1.smithi189.stdout:6/895: readlink d8/da/dd/d118/d30/d47/l105 0 2024-04-20T21:32:23.992 INFO:tasks.workunit.client.1.smithi189.stdout:7/940: truncate d5/db4/d71/de4/ff9 625901 0 2024-04-20T21:32:23.994 INFO:tasks.workunit.client.1.smithi189.stdout:0/960: fdatasync d3/df/d5d/d64/f122 0 2024-04-20T21:32:23.999 INFO:tasks.workunit.client.1.smithi189.stdout:9/839: dwrite d9/dc/d17/d95/da6/f101 [0,4194304] 0 2024-04-20T21:32:23.999 INFO:tasks.workunit.client.1.smithi189.stdout:7/941: write d5/d1d/ded/f112 [1080651,63260] 0 2024-04-20T21:32:24.000 INFO:tasks.workunit.client.1.smithi189.stdout:0/961: stat d3/df/d28/d2b/f49 0 2024-04-20T21:32:24.003 INFO:tasks.workunit.client.1.smithi189.stdout:7/942: chown d5/d21/d25/f30 1329373 1 2024-04-20T21:32:24.004 INFO:tasks.workunit.client.1.smithi189.stdout:6/896: read d8/d2b/d49/d54/d6e/f86 [1217382,93126] 0 2024-04-20T21:32:24.006 INFO:tasks.workunit.client.1.smithi189.stdout:8/963: link da/d81/d123/d22/f139 da/d81/d123/d13/dd6/df0/f154 0 2024-04-20T21:32:24.006 INFO:tasks.workunit.client.1.smithi189.stdout:9/840: creat d9/dc/d17/d7e/de8/f113 x:0 0 0 2024-04-20T21:32:24.010 INFO:tasks.workunit.client.1.smithi189.stdout:5/873: sync 2024-04-20T21:32:24.011 INFO:tasks.workunit.client.1.smithi189.stdout:0/962: chown d3/df/d30/d45/d90/df4/l7d 483131623 1 2024-04-20T21:32:24.011 INFO:tasks.workunit.client.1.smithi189.stdout:8/964: fsync da/d81/d123/d3c/d68/d8b/f10a 0 2024-04-20T21:32:24.016 INFO:tasks.workunit.client.1.smithi189.stdout:2/835: truncate d1/d11/d2d/dab/d4a/f7b 3523349 0 2024-04-20T21:32:24.017 INFO:tasks.workunit.client.1.smithi189.stdout:9/841: mkdir d9/dc/d114 0 2024-04-20T21:32:24.019 INFO:tasks.workunit.client.1.smithi189.stdout:5/874: mkdir d7/d9/d31/d100/d4e/d117 0 2024-04-20T21:32:24.019 INFO:tasks.workunit.client.1.smithi189.stdout:2/836: dread - d1/d11/d2d/dab/d74/fda zero size 2024-04-20T21:32:24.021 INFO:tasks.workunit.client.1.smithi189.stdout:1/806: write d8/f1a [3035604,73507] 0 2024-04-20T21:32:24.022 INFO:tasks.workunit.client.1.smithi189.stdout:9/842: creat d9/dc/d17/d95/da0/d55/dd0/f115 x:0 0 0 2024-04-20T21:32:24.023 INFO:tasks.workunit.client.1.smithi189.stdout:5/875: dread - d7/d9/d30/d5a/fea zero size 2024-04-20T21:32:24.026 INFO:tasks.workunit.client.1.smithi189.stdout:2/837: chown d1/d17/d37/d89 30407848 1 2024-04-20T21:32:24.027 INFO:tasks.workunit.client.1.smithi189.stdout:0/963: link d3/df/d30/cc6 d3/df/d30/d45/d90/df4/d114/c142 0 2024-04-20T21:32:24.030 INFO:tasks.workunit.client.1.smithi189.stdout:5/876: read d7/d9/d30/d55/d61/d87/f90 [3827140,33333] 0 2024-04-20T21:32:24.031 INFO:tasks.workunit.client.1.smithi189.stdout:9/843: creat d9/dc/f116 x:0 0 0 2024-04-20T21:32:24.032 INFO:tasks.workunit.client.1.smithi189.stdout:7/943: write d5/d21/d25/f78 [12710,36689] 0 2024-04-20T21:32:24.033 INFO:tasks.workunit.client.1.smithi189.stdout:6/897: write d8/da/fa3 [1367433,20887] 0 2024-04-20T21:32:24.036 INFO:tasks.workunit.client.1.smithi189.stdout:2/838: truncate d1/d17/d78/fce 2563956 0 2024-04-20T21:32:24.037 INFO:tasks.workunit.client.1.smithi189.stdout:3/868: dread d2/d3/d2c/d30/f68 [0,4194304] 0 2024-04-20T21:32:24.040 INFO:tasks.workunit.client.1.smithi189.stdout:0/964: getdents d3/df/d30/d37/dbd/de1 0 2024-04-20T21:32:24.042 INFO:tasks.workunit.client.1.smithi189.stdout:9/844: chown d9/d12/d62/dae/lc3 236567308 1 2024-04-20T21:32:24.043 INFO:tasks.workunit.client.1.smithi189.stdout:5/877: symlink d7/d9/d51/d70/d99/dd4/dda/l118 0 2024-04-20T21:32:24.046 INFO:tasks.workunit.client.1.smithi189.stdout:2/839: write d1/d11/d52/d8b/f8d [1800720,3379] 0 2024-04-20T21:32:24.049 INFO:tasks.workunit.client.1.smithi189.stdout:5/878: readlink d7/d9/d30/d55/d81/lfb 0 2024-04-20T21:32:24.050 INFO:tasks.workunit.client.1.smithi189.stdout:8/965: dwrite da/de/d1e/d21/d87/ff6 [0,4194304] 0 2024-04-20T21:32:24.051 INFO:tasks.workunit.client.1.smithi189.stdout:6/898: creat d8/f12d x:0 0 0 2024-04-20T21:32:24.051 INFO:tasks.workunit.client.1.smithi189.stdout:9/845: read - d9/dc/d17/d95/da6/dea/d105/f111 zero size 2024-04-20T21:32:24.056 INFO:tasks.workunit.client.1.smithi189.stdout:7/944: creat d5/d46/d7d/d45/f131 x:0 0 0 2024-04-20T21:32:24.056 INFO:tasks.workunit.client.1.smithi189.stdout:1/807: dwrite d8/d9b/dbb/f108 [0,4194304] 0 2024-04-20T21:32:24.057 INFO:tasks.workunit.client.1.smithi189.stdout:0/965: symlink d3/df/d30/d37/dbd/de1/l143 0 2024-04-20T21:32:24.058 INFO:tasks.workunit.client.1.smithi189.stdout:6/899: unlink c3 0 2024-04-20T21:32:24.064 INFO:tasks.workunit.client.1.smithi189.stdout:2/840: mkdir d1/d11/d2d/dab/d4a/d66/df1 0 2024-04-20T21:32:24.064 INFO:tasks.workunit.client.1.smithi189.stdout:7/945: mknod d5/d46/d7d/d45/c132 0 2024-04-20T21:32:24.067 INFO:tasks.workunit.client.1.smithi189.stdout:3/869: rename d2/f59 to d2/d3/d2c/d31/d74/d7a/f12e 0 2024-04-20T21:32:24.068 INFO:tasks.workunit.client.1.smithi189.stdout:4/806: sync 2024-04-20T21:32:24.075 INFO:tasks.workunit.client.1.smithi189.stdout:7/946: truncate d5/db4/d71/fe2 948013 0 2024-04-20T21:32:24.079 INFO:tasks.workunit.client.1.smithi189.stdout:0/966: unlink d3/df/d28/d2b/lc1 0 2024-04-20T21:32:24.082 INFO:tasks.workunit.client.1.smithi189.stdout:6/900: truncate d8/d2b/d49/f60 1204695 0 2024-04-20T21:32:24.083 INFO:tasks.workunit.client.1.smithi189.stdout:0/967: creat d3/df/d28/f144 x:0 0 0 2024-04-20T21:32:24.085 INFO:tasks.workunit.client.1.smithi189.stdout:4/807: mkdir d2/dd/d11/dae/d10e/d11c 0 2024-04-20T21:32:24.085 INFO:tasks.workunit.client.1.smithi189.stdout:3/870: creat d2/d3/d25/de0/db0/dc8/f12f x:0 0 0 2024-04-20T21:32:24.086 INFO:tasks.workunit.client.1.smithi189.stdout:2/841: getdents d1/d11/d99/db7 0 2024-04-20T21:32:24.087 INFO:tasks.workunit.client.1.smithi189.stdout:6/901: fdatasync d8/da/dd/d118/d4e/f50 0 2024-04-20T21:32:24.088 INFO:tasks.workunit.client.1.smithi189.stdout:0/968: fsync d3/db9/dbe/dd7/ff7 0 2024-04-20T21:32:24.089 INFO:tasks.workunit.client.1.smithi189.stdout:2/842: mknod d1/d11/d2d/d73/cf2 0 2024-04-20T21:32:24.091 INFO:tasks.workunit.client.1.smithi189.stdout:3/871: read d2/d8/f10 [4751485,64655] 0 2024-04-20T21:32:24.092 INFO:tasks.workunit.client.1.smithi189.stdout:4/808: creat d2/d20/d115/dcd/da4/f11d x:0 0 0 2024-04-20T21:32:24.093 INFO:tasks.workunit.client.1.smithi189.stdout:0/969: unlink d3/df/d5d/lcc 0 2024-04-20T21:32:24.096 INFO:tasks.workunit.client.1.smithi189.stdout:4/809: truncate d2/d20/fff 32663 0 2024-04-20T21:32:24.098 INFO:tasks.workunit.client.1.smithi189.stdout:4/810: chown d2/cf 211866803 1 2024-04-20T21:32:24.100 INFO:tasks.workunit.client.1.smithi189.stdout:6/902: getdents d8/d10 0 2024-04-20T21:32:24.106 INFO:tasks.workunit.client.1.smithi189.stdout:9/846: write d9/dc/f98 [1035256,100958] 0 2024-04-20T21:32:24.107 INFO:tasks.workunit.client.1.smithi189.stdout:4/811: mknod d2/d3a/ddc/c11e 0 2024-04-20T21:32:24.117 INFO:tasks.workunit.client.1.smithi189.stdout:2/843: dwrite d1/d11/f15 [0,4194304] 0 2024-04-20T21:32:24.117 INFO:tasks.workunit.client.1.smithi189.stdout:6/903: fdatasync d8/d10/d78/d7e/fd4 0 2024-04-20T21:32:24.117 INFO:tasks.workunit.client.1.smithi189.stdout:9/847: dread d9/f44 [0,4194304] 0 2024-04-20T21:32:24.120 INFO:tasks.workunit.client.1.smithi189.stdout:4/812: mkdir d2/dd/d11/d14/d11f 0 2024-04-20T21:32:24.126 INFO:tasks.workunit.client.1.smithi189.stdout:5/879: dwrite d7/d9/d31/d100/d4e/d5b/df1/dab/fdb [0,4194304] 0 2024-04-20T21:32:24.126 INFO:tasks.workunit.client.1.smithi189.stdout:4/813: readlink d2/dd/d11/la7 0 2024-04-20T21:32:24.130 INFO:tasks.workunit.client.1.smithi189.stdout:1/808: dwrite d8/d21/fb3 [0,4194304] 0 2024-04-20T21:32:24.135 INFO:tasks.workunit.client.1.smithi189.stdout:9/848: chown d9/dc/d17/d7e/c9c 1787 1 2024-04-20T21:32:24.143 INFO:tasks.workunit.client.1.smithi189.stdout:2/844: creat d1/d11/d58/ff3 x:0 0 0 2024-04-20T21:32:24.144 INFO:tasks.workunit.client.1.smithi189.stdout:4/814: mknod d2/d20/d115/dcd/d94/c120 0 2024-04-20T21:32:24.144 INFO:tasks.workunit.client.1.smithi189.stdout:8/966: dwrite da/de/d1e/d21/f97 [0,4194304] 0 2024-04-20T21:32:24.144 INFO:tasks.workunit.client.1.smithi189.stdout:6/904: mkdir d8/da/dd/d118/d12e 0 2024-04-20T21:32:24.145 INFO:tasks.workunit.client.1.smithi189.stdout:9/849: stat d9/dc/d17/d95/da6/fb3 0 2024-04-20T21:32:24.147 INFO:tasks.workunit.client.1.smithi189.stdout:5/880: truncate d7/d9/d30/d5a/d9d/ffd 800496 0 2024-04-20T21:32:24.150 INFO:tasks.workunit.client.1.smithi189.stdout:3/872: dwrite d2/d1c/d5c/f12c [4194304,4194304] 0 2024-04-20T21:32:24.157 INFO:tasks.workunit.client.1.smithi189.stdout:3/873: fsync d2/d8/f45 0 2024-04-20T21:32:24.157 INFO:tasks.workunit.client.1.smithi189.stdout:0/970: dwrite d3/df/d30/d37/ff2 [0,4194304] 0 2024-04-20T21:32:24.157 INFO:tasks.workunit.client.1.smithi189.stdout:8/967: chown da/d81/d123/d20/d2c/d9a/ddd/f124 16 1 2024-04-20T21:32:24.157 INFO:tasks.workunit.client.1.smithi189.stdout:2/845: dread - d1/d17/d37/d89/dc5/fdf zero size 2024-04-20T21:32:24.158 INFO:tasks.workunit.client.1.smithi189.stdout:5/881: rmdir d7/d9/d31/d36/de0 39 2024-04-20T21:32:24.168 INFO:tasks.workunit.client.1.smithi189.stdout:6/905: dread d8/d2b/d49/f72 [0,4194304] 0 2024-04-20T21:32:24.168 INFO:tasks.workunit.client.1.smithi189.stdout:8/968: symlink da/de/d1e/d137/l155 0 2024-04-20T21:32:24.168 INFO:tasks.workunit.client.1.smithi189.stdout:4/815: creat d2/d20/d115/dcd/d94/d107/d11a/f121 x:0 0 0 2024-04-20T21:32:24.173 INFO:tasks.workunit.client.1.smithi189.stdout:0/971: creat d3/df/d30/d45/dd3/f145 x:0 0 0 2024-04-20T21:32:24.176 INFO:tasks.workunit.client.1.smithi189.stdout:6/906: fsync d8/d2b/ffa 0 2024-04-20T21:32:24.178 INFO:tasks.workunit.client.1.smithi189.stdout:1/809: write d8/df/d50/f99 [245678,56550] 0 2024-04-20T21:32:24.185 INFO:tasks.workunit.client.1.smithi189.stdout:0/972: mknod d3/df/d28/c146 0 2024-04-20T21:32:24.199 INFO:tasks.workunit.client.1.smithi189.stdout:4/816: rename d2/d20/d115/dcd/f82 to d2/d20/d115/dcd/d94/d107/d11a/f122 0 2024-04-20T21:32:24.201 INFO:tasks.workunit.client.1.smithi189.stdout:0/973: creat d3/df/d30/d37/dbd/de1/d129/f147 x:0 0 0 2024-04-20T21:32:24.203 INFO:tasks.workunit.client.1.smithi189.stdout:2/846: dwrite d1/d11/d2d/d4f/fe9 [0,4194304] 0 2024-04-20T21:32:24.204 INFO:tasks.workunit.client.1.smithi189.stdout:3/874: dwrite d2/d3/d25/de0/db0/dc8/f12d [0,4194304] 0 2024-04-20T21:32:24.204 INFO:tasks.workunit.client.1.smithi189.stdout:5/882: dread d7/d9/d51/f72 [0,4194304] 0 2024-04-20T21:32:24.211 INFO:tasks.workunit.client.1.smithi189.stdout:9/850: dread d9/dc/f1c [0,4194304] 0 2024-04-20T21:32:24.216 INFO:tasks.workunit.client.1.smithi189.stdout:4/817: stat d2/d20/d115/dcd/d6a/d7a/def/f101 0 2024-04-20T21:32:24.225 INFO:tasks.workunit.client.1.smithi189.stdout:2/847: fsync d1/d11/d2d/d4f/fb2 0 2024-04-20T21:32:24.239 INFO:tasks.workunit.client.1.smithi189.stdout:9/851: symlink d9/dc/d17/d36/l117 0 2024-04-20T21:32:24.243 INFO:tasks.workunit.client.1.smithi189.stdout:1/810: dwrite d8/f14 [0,4194304] 0 2024-04-20T21:32:24.245 INFO:tasks.workunit.client.1.smithi189.stdout:2/848: chown d1/d11/d2d/d4f/d8e 115844276 1 2024-04-20T21:32:24.247 INFO:tasks.workunit.client.1.smithi189.stdout:7/947: sync 2024-04-20T21:32:24.252 INFO:tasks.workunit.client.1.smithi189.stdout:6/907: dwrite d8/d10/d78/d7e/fd4 [0,4194304] 0 2024-04-20T21:32:24.253 INFO:tasks.workunit.client.1.smithi189.stdout:0/974: dwrite d3/fe [0,4194304] 0 2024-04-20T21:32:24.259 INFO:tasks.workunit.client.1.smithi189.stdout:6/908: write d8/f63 [444107,66246] 0 2024-04-20T21:32:24.264 INFO:tasks.workunit.client.1.smithi189.stdout:7/948: write d5/d21/d35/f7e [4105500,55765] 0 2024-04-20T21:32:24.267 INFO:tasks.workunit.client.1.smithi189.stdout:5/883: dwrite d7/fc9 [0,4194304] 0 2024-04-20T21:32:24.270 INFO:tasks.workunit.client.1.smithi189.stdout:9/852: truncate d9/d12/d3b/db1/d1a/f112 27536 0 2024-04-20T21:32:24.272 INFO:tasks.workunit.client.1.smithi189.stdout:1/811: chown d8/d1e/d43/d44/dd2/d8a/c96 15 1 2024-04-20T21:32:24.274 INFO:tasks.workunit.client.1.smithi189.stdout:4/818: link d2/d20/d115/dcd/d6a/d7a/de9/lb0 d2/d20/dc6/dda/l123 0 2024-04-20T21:32:24.274 INFO:tasks.workunit.client.1.smithi189.stdout:8/969: dread da/d81/d123/d20/d2c/f76 [0,4194304] 0 2024-04-20T21:32:24.276 INFO:tasks.workunit.client.1.smithi189.stdout:5/884: fsync d7/d9/d31/d100/d18/f53 0 2024-04-20T21:32:24.277 INFO:tasks.workunit.client.1.smithi189.stdout:7/949: fdatasync d5/d1d/d64/db9/dcf/de7/ff0 0 2024-04-20T21:32:24.277 INFO:tasks.workunit.client.1.smithi189.stdout:3/875: dwrite d2/d3/f52 [0,4194304] 0 2024-04-20T21:32:24.282 INFO:tasks.workunit.client.1.smithi189.stdout:2/849: link d1/d11/l2c d1/d11/d2d/dab/d4a/d66/df1/lf4 0 2024-04-20T21:32:24.282 INFO:tasks.workunit.client.1.smithi189.stdout:6/909: rename d8/da/dd/d35/f45 to d8/da/da7/f12f 0 2024-04-20T21:32:24.283 INFO:tasks.workunit.client.1.smithi189.stdout:1/812: unlink d8/f9 0 2024-04-20T21:32:24.285 INFO:tasks.workunit.client.1.smithi189.stdout:5/885: rmdir d7/d9/d51 39 2024-04-20T21:32:24.289 INFO:tasks.workunit.client.1.smithi189.stdout:0/975: rename d3/df/d28/lda to d3/df/d28/d2b/db1/dfe/d119/d140/l148 0 2024-04-20T21:32:24.289 INFO:tasks.workunit.client.1.smithi189.stdout:7/950: rmdir d5/d1d/d77/db8 39 2024-04-20T21:32:24.290 INFO:tasks.workunit.client.1.smithi189.stdout:6/910: mknod d8/da/dd/d118/dd7/c130 0 2024-04-20T21:32:24.293 INFO:tasks.workunit.client.1.smithi189.stdout:3/876: truncate d2/d3/d36/da1/fb2 309092 0 2024-04-20T21:32:24.298 INFO:tasks.workunit.client.1.smithi189.stdout:7/951: symlink d5/d46/d7d/l133 0 2024-04-20T21:32:24.299 INFO:tasks.workunit.client.1.smithi189.stdout:6/911: readlink d8/da/da7/lcb 0 2024-04-20T21:32:24.303 INFO:tasks.workunit.client.1.smithi189.stdout:6/912: fdatasync d8/da/dd/d118/fae 0 2024-04-20T21:32:24.307 INFO:tasks.workunit.client.1.smithi189.stdout:4/819: dwrite d2/f54 [4194304,4194304] 0 2024-04-20T21:32:24.311 INFO:tasks.workunit.client.1.smithi189.stdout:6/913: chown d8/da/dd/d118/d99/lfe 21087122 1 2024-04-20T21:32:24.311 INFO:tasks.workunit.client.1.smithi189.stdout:3/877: rmdir d2/d1c/dc1/dc5/ddf 39 2024-04-20T21:32:24.311 INFO:tasks.workunit.client.1.smithi189.stdout:9/853: dwrite d9/dc/f1c [8388608,4194304] 0 2024-04-20T21:32:24.314 INFO:tasks.workunit.client.1.smithi189.stdout:8/970: rename da/d81/d123/d20/f99 to da/d81/d123/d3c/d50/d6c/dc7/f156 0 2024-04-20T21:32:24.317 INFO:tasks.workunit.client.1.smithi189.stdout:5/886: getdents d7/d9/d30/d55/d61/d87 0 2024-04-20T21:32:24.318 INFO:tasks.workunit.client.1.smithi189.stdout:7/952: mkdir d5/d1d/ded/d12a/d134 0 2024-04-20T21:32:24.323 INFO:tasks.workunit.client.1.smithi189.stdout:3/878: creat d2/d3/d25/f130 x:0 0 0 2024-04-20T21:32:24.326 INFO:tasks.workunit.client.1.smithi189.stdout:3/879: stat d2/d3/d65/lec 0 2024-04-20T21:32:24.329 INFO:tasks.workunit.client.1.smithi189.stdout:0/976: link d3/df/d28/dc8/c113 d3/db9/dbe/dd7/c149 0 2024-04-20T21:32:24.331 INFO:tasks.workunit.client.1.smithi189.stdout:5/887: symlink d7/d9/d30/d55/d61/d96/l119 0 2024-04-20T21:32:24.331 INFO:tasks.workunit.client.1.smithi189.stdout:0/977: write d3/df/d30/d45/d90/f127 [521779,4716] 0 2024-04-20T21:32:24.332 INFO:tasks.workunit.client.1.smithi189.stdout:8/971: creat da/d81/d123/d20/d27/d6e/f157 x:0 0 0 2024-04-20T21:32:24.333 INFO:tasks.workunit.client.1.smithi189.stdout:7/953: creat d5/d21/d4a/d52/d98/f135 x:0 0 0 2024-04-20T21:32:24.334 INFO:tasks.workunit.client.1.smithi189.stdout:1/813: dread d8/d1e/d54/d93/ffa [0,4194304] 0 2024-04-20T21:32:24.335 INFO:tasks.workunit.client.1.smithi189.stdout:4/820: creat d2/d20/d115/dcd/df9/f124 x:0 0 0 2024-04-20T21:32:24.338 INFO:tasks.workunit.client.1.smithi189.stdout:7/954: chown d5/f49 41085408 1 2024-04-20T21:32:24.339 INFO:tasks.workunit.client.1.smithi189.stdout:6/914: getdents d8/d127 0 2024-04-20T21:32:24.340 INFO:tasks.workunit.client.1.smithi189.stdout:1/814: truncate d8/f1a 3191399 0 2024-04-20T21:32:24.341 INFO:tasks.workunit.client.1.smithi189.stdout:3/880: creat d2/d1c/dc1/dc5/ddf/f131 x:0 0 0 2024-04-20T21:32:24.343 INFO:tasks.workunit.client.1.smithi189.stdout:8/972: rmdir da/d81/d123/d13/d63/d7b 39 2024-04-20T21:32:24.343 INFO:tasks.workunit.client.1.smithi189.stdout:7/955: stat d5/d21/d97/da3/dc6 0 2024-04-20T21:32:24.345 INFO:tasks.workunit.client.1.smithi189.stdout:3/881: readlink d2/d3/d25/l3b 0 2024-04-20T21:32:24.347 INFO:tasks.workunit.client.1.smithi189.stdout:4/821: rename d2/f21 to d2/d20/d115/dcd/f125 0 2024-04-20T21:32:24.348 INFO:tasks.workunit.client.1.smithi189.stdout:8/973: chown da/d81/d123/d20/d2c/f141 27895 1 2024-04-20T21:32:24.349 INFO:tasks.workunit.client.1.smithi189.stdout:1/815: readlink d8/d21/d2c/l10a 0 2024-04-20T21:32:24.350 INFO:tasks.workunit.client.1.smithi189.stdout:6/915: read d8/da/dd/d35/f59 [1081206,27146] 0 2024-04-20T21:32:24.351 INFO:tasks.workunit.client.1.smithi189.stdout:7/956: mknod d5/d1d/ded/c136 0 2024-04-20T21:32:24.354 INFO:tasks.workunit.client.1.smithi189.stdout:3/882: symlink d2/d3/d65/l132 0 2024-04-20T21:32:24.355 INFO:tasks.workunit.client.1.smithi189.stdout:6/916: chown d8/da/dd/d118/f116 679230805 1 2024-04-20T21:32:24.356 INFO:tasks.workunit.client.1.smithi189.stdout:9/854: dread d9/d12/fa8 [0,4194304] 0 2024-04-20T21:32:24.356 INFO:tasks.workunit.client.1.smithi189.stdout:4/822: mkdir d2/dd/d6b/d126 0 2024-04-20T21:32:24.359 INFO:tasks.workunit.client.1.smithi189.stdout:9/855: fsync d9/dc/d17/d7e/f8d 0 2024-04-20T21:32:24.360 INFO:tasks.workunit.client.1.smithi189.stdout:6/917: fsync d8/d2b/d49/d54/d67/ffc 0 2024-04-20T21:32:24.361 INFO:tasks.workunit.client.1.smithi189.stdout:3/883: chown d2/d3/d25/de0/db0/le1 338 1 2024-04-20T21:32:24.364 INFO:tasks.workunit.client.1.smithi189.stdout:0/978: dwrite d3/df/d28/dc8/df1/ff5 [0,4194304] 0 2024-04-20T21:32:24.365 INFO:tasks.workunit.client.1.smithi189.stdout:5/888: rename d7/d9/d30/d55/d33/f59 to d7/d9/d30/f11a 0 2024-04-20T21:32:24.368 INFO:tasks.workunit.client.1.smithi189.stdout:7/957: creat d5/d21/d4a/dfd/d101/f137 x:0 0 0 2024-04-20T21:32:24.373 INFO:tasks.workunit.client.1.smithi189.stdout:9/856: creat d9/d12/d50/f118 x:0 0 0 2024-04-20T21:32:24.375 INFO:tasks.workunit.client.1.smithi189.stdout:3/884: creat d2/d3/d2c/d31/d4c/f133 x:0 0 0 2024-04-20T21:32:24.377 INFO:tasks.workunit.client.1.smithi189.stdout:0/979: mkdir d3/df/d28/d2b/d2c/d14a 0 2024-04-20T21:32:24.379 INFO:tasks.workunit.client.1.smithi189.stdout:0/980: write d3/df/d4c/dd4/fe6 [1727411,93837] 0 2024-04-20T21:32:24.381 INFO:tasks.workunit.client.1.smithi189.stdout:1/816: truncate d8/d1e/d43/d44/dd2/fc7 209026 0 2024-04-20T21:32:24.382 INFO:tasks.workunit.client.1.smithi189.stdout:8/974: write da/d81/d123/d13/dd6/dba/f119 [680922,50068] 0 2024-04-20T21:32:24.382 INFO:tasks.workunit.client.1.smithi189.stdout:0/981: chown d3/df/f3e 125369185 1 2024-04-20T21:32:24.385 INFO:tasks.workunit.client.1.smithi189.stdout:6/918: rename d8/d2b/d49/d54/d67/c10d to d8/d10/d78/d7e/dcf/c131 0 2024-04-20T21:32:24.387 INFO:tasks.workunit.client.1.smithi189.stdout:9/857: truncate d9/d12/d3f/d91/fa3 4632749 0 2024-04-20T21:32:24.391 INFO:tasks.workunit.client.1.smithi189.stdout:5/889: unlink d7/d9/d51/d70/d95/ff0 0 2024-04-20T21:32:24.393 INFO:tasks.workunit.client.1.smithi189.stdout:7/958: link d5/d21/d4a/d52/d98/f12c d5/d46/d7d/d45/f138 0 2024-04-20T21:32:24.395 INFO:tasks.workunit.client.1.smithi189.stdout:8/975: fsync da/fb 0 2024-04-20T21:32:24.397 INFO:tasks.workunit.client.1.smithi189.stdout:8/976: chown da/de/d1e/d69/f95 8784 1 2024-04-20T21:32:24.398 INFO:tasks.workunit.client.1.smithi189.stdout:3/885: mkdir d2/d1c/dc1/dc5/d134 0 2024-04-20T21:32:24.402 INFO:tasks.workunit.client.1.smithi189.stdout:9/858: write d9/d12/fa8 [1928984,116376] 0 2024-04-20T21:32:24.405 INFO:tasks.workunit.client.1.smithi189.stdout:1/817: mknod d8/d1e/d43/d58/dc2/c10b 0 2024-04-20T21:32:24.407 INFO:tasks.workunit.client.1.smithi189.stdout:6/919: creat d8/d2b/d49/d54/d6e/d11f/f132 x:0 0 0 2024-04-20T21:32:24.411 INFO:tasks.workunit.client.1.smithi189.stdout:1/818: dread - d8/d1e/d28/fd8 zero size 2024-04-20T21:32:24.412 INFO:tasks.workunit.client.1.smithi189.stdout:3/886: symlink d2/d1c/dc1/dc5/l135 0 2024-04-20T21:32:24.413 INFO:tasks.workunit.client.1.smithi189.stdout:3/887: readlink d2/d3/d2c/l56 0 2024-04-20T21:32:24.415 INFO:tasks.workunit.client.1.smithi189.stdout:3/888: readlink d2/d3/d2c/d30/l72 0 2024-04-20T21:32:24.415 INFO:tasks.workunit.client.1.smithi189.stdout:8/977: rename da/d81/d123/d3c/d50/f61 to da/d81/d123/d13/d31/f158 0 2024-04-20T21:32:24.416 INFO:tasks.workunit.client.1.smithi189.stdout:5/890: creat d7/d9/d28/f11b x:0 0 0 2024-04-20T21:32:24.419 INFO:tasks.workunit.client.1.smithi189.stdout:7/959: write d5/f39 [1288071,18373] 0 2024-04-20T21:32:24.419 INFO:tasks.workunit.client.1.smithi189.stdout:0/982: dread d3/db9/dbe/dd7/fde [0,4194304] 0 2024-04-20T21:32:24.421 INFO:tasks.workunit.client.1.smithi189.stdout:9/859: creat d9/d12/f119 x:0 0 0 2024-04-20T21:32:24.423 INFO:tasks.workunit.client.1.smithi189.stdout:1/819: write d8/d1e/d43/d58/f72 [4658211,57971] 0 2024-04-20T21:32:24.429 INFO:tasks.workunit.client.1.smithi189.stdout:8/978: creat da/d81/d123/d3c/d50/f159 x:0 0 0 2024-04-20T21:32:24.430 INFO:tasks.workunit.client.1.smithi189.stdout:5/891: symlink d7/d9/d51/d70/d95/l11c 0 2024-04-20T21:32:24.433 INFO:tasks.workunit.client.1.smithi189.stdout:6/920: dwrite d8/d2b/d49/d108/f125 [0,4194304] 0 2024-04-20T21:32:24.435 INFO:tasks.workunit.client.1.smithi189.stdout:3/889: rename d2/d3/d2c/d31/d74/db7/dd3/le9 to d2/l136 0 2024-04-20T21:32:24.441 INFO:tasks.workunit.client.1.smithi189.stdout:0/983: mknod d3/df/d28/dc8/d121/d141/c14b 0 2024-04-20T21:32:24.443 INFO:tasks.workunit.client.1.smithi189.stdout:1/820: truncate d8/d1e/d28/f91 82949 0 2024-04-20T21:32:24.445 INFO:tasks.workunit.client.1.smithi189.stdout:8/979: mknod da/d81/d123/d3c/d68/de6/c15a 0 2024-04-20T21:32:24.448 INFO:tasks.workunit.client.1.smithi189.stdout:3/890: chown d2/d3/d25/de0/da9/cc6 3518 1 2024-04-20T21:32:24.454 INFO:tasks.workunit.client.1.smithi189.stdout:1/821: rmdir d8/df/d3a/d49/d8c/da2/dbe/dc1 39 2024-04-20T21:32:24.458 INFO:tasks.workunit.client.1.smithi189.stdout:5/892: dread d7/fc9 [0,4194304] 0 2024-04-20T21:32:24.460 INFO:tasks.workunit.client.1.smithi189.stdout:7/960: dwrite d5/d46/f8b [4194304,4194304] 0 2024-04-20T21:32:24.462 INFO:tasks.workunit.client.1.smithi189.stdout:9/860: dwrite d9/d12/d3b/fe0 [0,4194304] 0 2024-04-20T21:32:24.467 INFO:tasks.workunit.client.1.smithi189.stdout:8/980: rename da/d81/d123/d22 to da/d81/d123/d13/d63/d7b/d15b 0 2024-04-20T21:32:24.468 INFO:tasks.workunit.client.1.smithi189.stdout:6/921: dwrite d8/d10/d78/df7/fac [4194304,4194304] 0 2024-04-20T21:32:24.474 INFO:tasks.workunit.client.1.smithi189.stdout:5/893: rename l0 to d7/d9/d31/d100/l11d 0 2024-04-20T21:32:24.475 INFO:tasks.workunit.client.1.smithi189.stdout:9/861: creat d9/dc/d114/f11a x:0 0 0 2024-04-20T21:32:24.478 INFO:tasks.workunit.client.1.smithi189.stdout:2/850: sync 2024-04-20T21:32:24.480 INFO:tasks.workunit.client.1.smithi189.stdout:7/961: link d5/d46/c51 d5/d21/d97/c139 0 2024-04-20T21:32:24.480 INFO:tasks.workunit.client.1.smithi189.stdout:9/862: mkdir d9/d12/d3f/d72/d11b 0 2024-04-20T21:32:24.481 INFO:tasks.workunit.client.1.smithi189.stdout:3/891: dread d2/d3/d25/de0/db0/dc8/f12d [0,4194304] 0 2024-04-20T21:32:24.483 INFO:tasks.workunit.client.1.smithi189.stdout:9/863: dread - d9/dc/d17/d95/ffe zero size 2024-04-20T21:32:24.485 INFO:tasks.workunit.client.1.smithi189.stdout:5/894: creat d7/d9/d51/d70/d99/dd4/f11e x:0 0 0 2024-04-20T21:32:24.486 INFO:tasks.workunit.client.1.smithi189.stdout:7/962: mkdir d5/db4/d13a 0 2024-04-20T21:32:24.494 INFO:tasks.workunit.client.1.smithi189.stdout:3/892: creat d2/d3/d2c/d31/d74/d7a/df1/f137 x:0 0 0 2024-04-20T21:32:24.494 INFO:tasks.workunit.client.1.smithi189.stdout:5/895: read d7/d9/d30/d55/d61/fb2 [1248332,47597] 0 2024-04-20T21:32:24.496 INFO:tasks.workunit.client.1.smithi189.stdout:9/864: mknod d9/dc/d17/d95/da0/d52/d82/c11c 0 2024-04-20T21:32:24.498 INFO:tasks.workunit.client.1.smithi189.stdout:2/851: truncate d1/d11/d2d/d4f/d8e/f9a 3944004 0 2024-04-20T21:32:24.499 INFO:tasks.workunit.client.1.smithi189.stdout:3/893: creat d2/d3/d25/de0/db0/f138 x:0 0 0 2024-04-20T21:32:24.499 INFO:tasks.workunit.client.1.smithi189.stdout:1/822: dread d8/d21/fdc [0,4194304] 0 2024-04-20T21:32:24.500 INFO:tasks.workunit.client.1.smithi189.stdout:9/865: fsync d9/dc/d17/f2f 0 2024-04-20T21:32:24.501 INFO:tasks.workunit.client.1.smithi189.stdout:2/852: stat d1/l21 0 2024-04-20T21:32:24.506 INFO:tasks.workunit.client.1.smithi189.stdout:1/823: creat d8/df/d3a/d49/d8c/da2/f10c x:0 0 0 2024-04-20T21:32:24.510 INFO:tasks.workunit.client.1.smithi189.stdout:2/853: write d1/d11/d2d/d4f/d92/fd1 [1282601,90010] 0 2024-04-20T21:32:24.515 INFO:tasks.workunit.client.1.smithi189.stdout:7/963: write d5/d46/d7d/d45/f48 [3058871,89072] 0 2024-04-20T21:32:24.515 INFO:tasks.workunit.client.1.smithi189.stdout:5/896: dwrite d7/d9/d51/d70/d95/f111 [0,4194304] 0 2024-04-20T21:32:24.518 INFO:tasks.workunit.client.1.smithi189.stdout:0/984: dread d3/df/d4c/fa4 [0,4194304] 0 2024-04-20T21:32:24.519 INFO:tasks.workunit.client.1.smithi189.stdout:3/894: truncate d2/f2a 663087 0 2024-04-20T21:32:24.522 INFO:tasks.workunit.client.1.smithi189.stdout:3/895: chown d2/d8/f22 125861 1 2024-04-20T21:32:24.525 INFO:tasks.workunit.client.1.smithi189.stdout:0/985: symlink d3/l14c 0 2024-04-20T21:32:24.531 INFO:tasks.workunit.client.1.smithi189.stdout:0/986: creat d3/df/d30/d45/f14d x:0 0 0 2024-04-20T21:32:24.531 INFO:tasks.workunit.client.1.smithi189.stdout:1/824: dwrite d8/d1e/d43/d44/f47 [0,4194304] 0 2024-04-20T21:32:24.532 INFO:tasks.workunit.client.1.smithi189.stdout:0/987: fsync d3/fe 0 2024-04-20T21:32:24.535 INFO:tasks.workunit.client.1.smithi189.stdout:1/825: chown d8/d1e/d43/d58/d60/f64 1128208727 1 2024-04-20T21:32:24.537 INFO:tasks.workunit.client.1.smithi189.stdout:3/896: dread d2/d3/d25/de0/f9b [0,4194304] 0 2024-04-20T21:32:24.545 INFO:tasks.workunit.client.1.smithi189.stdout:0/988: creat d3/df/d30/d45/d90/df4/f14e x:0 0 0 2024-04-20T21:32:24.545 INFO:tasks.workunit.client.1.smithi189.stdout:5/897: truncate d7/d9/f62 3054994 0 2024-04-20T21:32:24.547 INFO:tasks.workunit.client.1.smithi189.stdout:7/964: truncate d5/d21/d4a/d52/d75/fda 2484528 0 2024-04-20T21:32:24.548 INFO:tasks.workunit.client.1.smithi189.stdout:3/897: rename d2/d3/d2c/d31/d74/d11c/f123 to d2/d1c/dc1/dc5/d134/f139 0 2024-04-20T21:32:24.549 INFO:tasks.workunit.client.1.smithi189.stdout:0/989: readlink d3/df/d30/d37/l25 0 2024-04-20T21:32:24.552 INFO:tasks.workunit.client.1.smithi189.stdout:5/898: chown d7/d9/d31/d36/f38 3 1 2024-04-20T21:32:24.553 INFO:tasks.workunit.client.1.smithi189.stdout:3/898: creat d2/d1c/d5c/f13a x:0 0 0 2024-04-20T21:32:24.564 INFO:tasks.workunit.client.1.smithi189.stdout:3/899: getdents d2/d3/d2c/d31 0 2024-04-20T21:32:24.574 INFO:tasks.workunit.client.1.smithi189.stdout:3/900: write d2/d3/f19 [3069550,110926] 0 2024-04-20T21:32:24.575 INFO:tasks.workunit.client.1.smithi189.stdout:7/965: dwrite d5/f14 [0,4194304] 0 2024-04-20T21:32:24.577 INFO:tasks.workunit.client.1.smithi189.stdout:5/899: dwrite d7/d9/d30/d55/d81/f112 [0,4194304] 0 2024-04-20T21:32:24.577 INFO:tasks.workunit.client.1.smithi189.stdout:0/990: dwrite d3/db9/dbe/dd7/fde [0,4194304] 0 2024-04-20T21:32:24.587 INFO:tasks.workunit.client.1.smithi189.stdout:5/900: rmdir d7 39 2024-04-20T21:32:24.592 INFO:tasks.workunit.client.1.smithi189.stdout:3/901: unlink d2/d3/d2c/de3/c10a 0 2024-04-20T21:32:24.596 INFO:tasks.workunit.client.1.smithi189.stdout:0/991: creat d3/df/d28/d2b/db1/f14f x:0 0 0 2024-04-20T21:32:24.597 INFO:tasks.workunit.client.1.smithi189.stdout:0/992: chown d3/cce 1 1 2024-04-20T21:32:24.602 INFO:tasks.workunit.client.1.smithi189.stdout:7/966: link d5/d46/d7d/d45/l5a d5/d59/l13b 0 2024-04-20T21:32:24.604 INFO:tasks.workunit.client.1.smithi189.stdout:3/902: mknod d2/d3/d25/de0/db0/dc8/c13b 0 2024-04-20T21:32:24.606 INFO:tasks.workunit.client.1.smithi189.stdout:0/993: creat d3/df/d28/d2b/f150 x:0 0 0 2024-04-20T21:32:24.609 INFO:tasks.workunit.client.1.smithi189.stdout:5/901: rename d7/d9/d30/d5a/d9d/cc6 to d7/d9/d51/c11f 0 2024-04-20T21:32:24.611 INFO:tasks.workunit.client.1.smithi189.stdout:3/903: unlink d2/d3/d2c/ff4 0 2024-04-20T21:32:24.611 INFO:tasks.workunit.client.1.smithi189.stdout:5/902: write d7/d9/d30/d55/d61/db3/f101 [368612,47907] 0 2024-04-20T21:32:24.611 INFO:tasks.workunit.client.1.smithi189.stdout:7/967: mknod d5/d21/d4a/d52/d74/d115/c13c 0 2024-04-20T21:32:24.616 INFO:tasks.workunit.client.1.smithi189.stdout:0/994: link d3/df/d28/d2b/d2c/le8 d3/df/d28/l151 0 2024-04-20T21:32:24.618 INFO:tasks.workunit.client.1.smithi189.stdout:3/904: mkdir d2/d3/d2c/d13c 0 2024-04-20T21:32:24.619 INFO:tasks.workunit.client.1.smithi189.stdout:3/905: dread - d2/d3/d2c/f11a zero size 2024-04-20T21:32:24.619 INFO:tasks.workunit.client.1.smithi189.stdout:5/903: creat d7/d9/d31/d36/de0/f120 x:0 0 0 2024-04-20T21:32:24.622 INFO:tasks.workunit.client.1.smithi189.stdout:5/904: read - d7/d9/d30/d55/d61/d96/fd3 zero size 2024-04-20T21:32:24.626 INFO:tasks.workunit.client.1.smithi189.stdout:5/905: write d7/d9/d30/d55/d33/fe6 [127468,94439] 0 2024-04-20T21:32:24.627 INFO:tasks.workunit.client.1.smithi189.stdout:7/968: dwrite d5/d21/d25/d34/fec [0,4194304] 0 2024-04-20T21:32:24.633 INFO:tasks.workunit.client.1.smithi189.stdout:7/969: write d5/d1d/d64/feb [4988846,119348] 0 2024-04-20T21:32:24.633 INFO:tasks.workunit.client.1.smithi189.stdout:5/906: dread d7/d9/d30/d55/d61/daa/fae [0,4194304] 0 2024-04-20T21:32:24.635 INFO:tasks.workunit.client.1.smithi189.stdout:7/970: chown d5/d59/ff4 271703 1 2024-04-20T21:32:24.636 INFO:tasks.workunit.client.1.smithi189.stdout:7/971: chown d5/d46/d7d/d111/d11b 2 1 2024-04-20T21:32:24.639 INFO:tasks.workunit.client.1.smithi189.stdout:5/907: mkdir d7/d9/d31/d36/de0/d121 0 2024-04-20T21:32:24.640 INFO:tasks.workunit.client.1.smithi189.stdout:7/972: fsync d5/d59/fa5 0 2024-04-20T21:32:24.641 INFO:tasks.workunit.client.1.smithi189.stdout:4/823: sync 2024-04-20T21:32:24.644 INFO:tasks.workunit.client.1.smithi189.stdout:5/908: write d7/fc9 [2963935,105487] 0 2024-04-20T21:32:24.647 INFO:tasks.workunit.client.1.smithi189.stdout:4/824: creat d2/dd/d11/d14/f127 x:0 0 0 2024-04-20T21:32:24.649 INFO:tasks.workunit.client.1.smithi189.stdout:5/909: unlink d7/d9/d31/fbe 0 2024-04-20T21:32:24.659 INFO:tasks.workunit.client.1.smithi189.stdout:4/825: creat d2/d20/d115/dcd/d6a/d7a/f128 x:0 0 0 2024-04-20T21:32:24.675 INFO:tasks.workunit.client.1.smithi189.stdout:5/910: dwrite d7/d35/fce [0,4194304] 0 2024-04-20T21:32:24.680 INFO:tasks.workunit.client.1.smithi189.stdout:5/911: mknod d7/d9/d31/d100/c122 0 2024-04-20T21:32:24.685 INFO:tasks.workunit.client.1.smithi189.stdout:5/912: unlink d7/d9/d51/d70/d99/dd4/fe3 0 2024-04-20T21:32:24.690 INFO:tasks.workunit.client.1.smithi189.stdout:5/913: creat d7/d9/d31/d100/d4e/d5b/df1/dab/f123 x:0 0 0 2024-04-20T21:32:24.694 INFO:tasks.workunit.client.1.smithi189.stdout:5/914: readlink d7/l3a 0 2024-04-20T21:32:24.694 INFO:tasks.workunit.client.1.smithi189.stdout:4/826: dread d2/f54 [0,4194304] 0 2024-04-20T21:32:24.695 INFO:tasks.workunit.client.1.smithi189.stdout:4/827: chown d2/f1a 1055 1 2024-04-20T21:32:24.708 INFO:tasks.workunit.client.1.smithi189.stdout:4/828: truncate d2/f4 6351434 0 2024-04-20T21:32:24.711 INFO:tasks.workunit.client.1.smithi189.stdout:7/973: dread d5/d46/fb7 [0,4194304] 0 2024-04-20T21:32:24.711 INFO:tasks.workunit.client.1.smithi189.stdout:4/829: mknod d2/d22/d57/c129 0 2024-04-20T21:32:24.724 INFO:tasks.workunit.client.1.smithi189.stdout:7/974: rmdir d5/d21/d4a/dfd 39 2024-04-20T21:32:24.724 INFO:tasks.workunit.client.1.smithi189.stdout:4/830: mknod d2/d22/c12a 0 2024-04-20T21:32:24.729 INFO:tasks.workunit.client.1.smithi189.stdout:4/831: mkdir d2/d20/d12b 0 2024-04-20T21:32:24.730 INFO:tasks.workunit.client.1.smithi189.stdout:7/975: mknod d5/db4/d71/c13d 0 2024-04-20T21:32:24.738 INFO:tasks.workunit.client.1.smithi189.stdout:7/976: creat d5/d21/d4a/dfd/f13e x:0 0 0 2024-04-20T21:32:24.741 INFO:tasks.workunit.client.1.smithi189.stdout:7/977: creat d5/d21/d35/f13f x:0 0 0 2024-04-20T21:32:24.756 INFO:tasks.workunit.client.1.smithi189.stdout:4/832: dwrite d2/d3a/f4d [0,4194304] 0 2024-04-20T21:32:24.763 INFO:tasks.workunit.client.1.smithi189.stdout:4/833: fdatasync d2/d3a/ddc/ff3 0 2024-04-20T21:32:24.777 INFO:tasks.workunit.client.1.smithi189.stdout:4/834: link d2/dd/d6b/da6/fc2 d2/d20/d115/dcd/df9/f12c 0 2024-04-20T21:32:24.802 INFO:tasks.workunit.client.1.smithi189.stdout:4/835: dwrite d2/d20/d115/dcd/d94/d107/d11a/f121 [0,4194304] 0 2024-04-20T21:32:24.809 INFO:tasks.workunit.client.1.smithi189.stdout:4/836: creat d2/d22/f12d x:0 0 0 2024-04-20T21:32:24.811 INFO:tasks.workunit.client.1.smithi189.stdout:8/981: sync 2024-04-20T21:32:24.812 INFO:tasks.workunit.client.1.smithi189.stdout:4/837: chown d2/d20/d115/l7b 208912 1 2024-04-20T21:32:24.828 INFO:tasks.workunit.client.1.smithi189.stdout:4/838: write d2/dd/d11/f9a [905760,46330] 0 2024-04-20T21:32:24.830 INFO:tasks.workunit.client.1.smithi189.stdout:4/839: readlink d2/d20/d115/dcd/d6a/l8a 0 2024-04-20T21:32:24.834 INFO:tasks.workunit.client.1.smithi189.stdout:8/982: dwrite da/de/d1e/d21/d87/ff6 [0,4194304] 0 2024-04-20T21:32:24.834 INFO:tasks.workunit.client.1.smithi189.stdout:4/840: mkdir d2/d20/dc6/dda/d12e 0 2024-04-20T21:32:24.836 INFO:tasks.workunit.client.1.smithi189.stdout:4/841: readlink d2/l5 0 2024-04-20T21:32:24.841 INFO:tasks.workunit.client.1.smithi189.stdout:8/983: chown da/d81/d123/d13/dd6/c6f 1925489 1 2024-04-20T21:32:24.843 INFO:tasks.workunit.client.1.smithi189.stdout:8/984: fdatasync da/d81/d123/d20/f7d 0 2024-04-20T21:32:24.847 INFO:tasks.workunit.client.1.smithi189.stdout:4/842: write d2/dd/d11/d14/f3f [8821257,25893] 0 2024-04-20T21:32:24.852 INFO:tasks.workunit.client.1.smithi189.stdout:8/985: rename da/d81/d123/d3c/d50/d6c/lb4 to da/d81/d123/d3c/d68/d8b/dac/l15c 0 2024-04-20T21:32:24.857 INFO:tasks.workunit.client.1.smithi189.stdout:4/843: getdents d2/d3a/ddc 0 2024-04-20T21:32:24.859 INFO:tasks.workunit.client.1.smithi189.stdout:4/844: readlink d2/dd/d11/l25 0 2024-04-20T21:32:24.861 INFO:tasks.workunit.client.1.smithi189.stdout:4/845: dread - d2/f75 zero size 2024-04-20T21:32:24.866 INFO:tasks.workunit.client.1.smithi189.stdout:8/986: write da/de/f16 [3144907,8610] 0 2024-04-20T21:32:24.867 INFO:tasks.workunit.client.1.smithi189.stdout:4/846: creat d2/d20/d12b/f12f x:0 0 0 2024-04-20T21:32:24.870 INFO:tasks.workunit.client.1.smithi189.stdout:8/987: chown da/d81/d123/d3c/d68/d8b/fc1 1 1 2024-04-20T21:32:24.873 INFO:tasks.workunit.client.1.smithi189.stdout:4/847: rmdir d2/d3a/ddc/dfc 0 2024-04-20T21:32:24.873 INFO:tasks.workunit.client.1.smithi189.stdout:8/988: chown da/de/d1e/d21/d44/l60 7832 1 2024-04-20T21:32:24.875 INFO:tasks.workunit.client.1.smithi189.stdout:4/848: write d2/f6d [1486926,65763] 0 2024-04-20T21:32:24.877 INFO:tasks.workunit.client.1.smithi189.stdout:8/989: write da/de/d1e/d21/d44/f4a [610039,121482] 0 2024-04-20T21:32:24.881 INFO:tasks.workunit.client.1.smithi189.stdout:8/990: write da/d81/d123/d13/d63/d7b/f142 [1040858,79829] 0 2024-04-20T21:32:24.882 INFO:tasks.workunit.client.1.smithi189.stdout:4/849: write d2/f5a [58213,24048] 0 2024-04-20T21:32:24.883 INFO:tasks.workunit.client.1.smithi189.stdout:4/850: chown d2/laa 452906 1 2024-04-20T21:32:24.886 INFO:tasks.workunit.client.1.smithi189.stdout:8/991: chown da/de/l48 1 1 2024-04-20T21:32:24.888 INFO:tasks.workunit.client.1.smithi189.stdout:4/851: creat d2/d22/d87/f130 x:0 0 0 2024-04-20T21:32:24.893 INFO:tasks.workunit.client.1.smithi189.stdout:4/852: unlink d2/d3a/fad 0 2024-04-20T21:32:24.894 INFO:tasks.workunit.client.1.smithi189.stdout:4/853: stat d2/dd/f44 0 2024-04-20T21:32:24.897 INFO:tasks.workunit.client.1.smithi189.stdout:4/854: readlink d2/d20/dc6/dda/l123 0 2024-04-20T21:32:24.903 INFO:tasks.workunit.client.1.smithi189.stdout:4/855: creat d2/d20/dc6/f131 x:0 0 0 2024-04-20T21:32:24.907 INFO:tasks.workunit.client.1.smithi189.stdout:8/992: dwrite da/d81/d123/d3c/d68/d8b/f10a [0,4194304] 0 2024-04-20T21:32:24.929 INFO:tasks.workunit.client.1.smithi189.stdout:8/993: truncate da/d81/d123/d3c/d68/d8b/d8d/f90 3094282 0 2024-04-20T21:32:24.930 INFO:tasks.workunit.client.1.smithi189.stdout:4/856: dread d2/d22/d87/fd1 [0,4194304] 0 2024-04-20T21:32:24.935 INFO:tasks.workunit.client.1.smithi189.stdout:8/994: dread da/d81/d123/d3c/deb/f146 [0,4194304] 0 2024-04-20T21:32:24.938 INFO:tasks.workunit.client.1.smithi189.stdout:8/995: read da/d81/d123/d3c/deb/f146 [69022,19216] 0 2024-04-20T21:32:25.007 INFO:tasks.workunit.client.1.smithi189.stdout:6/922: sync 2024-04-20T21:32:25.018 INFO:tasks.workunit.client.1.smithi189.stdout:4/857: dwrite d2/d22/d87/f130 [0,4194304] 0 2024-04-20T21:32:25.089 INFO:tasks.workunit.client.1.smithi189.stdout:8/996: fdatasync da/d81/d123/d3c/d50/ddc/df2/f104 0 2024-04-20T21:32:25.091 INFO:tasks.workunit.client.1.smithi189.stdout:6/923: mkdir d8/d2b/d49/d108/d111/d133 0 2024-04-20T21:32:25.093 INFO:tasks.workunit.client.1.smithi189.stdout:4/858: fdatasync d2/dd/f44 0 2024-04-20T21:32:25.094 INFO:tasks.workunit.client.1.smithi189.stdout:8/997: creat da/d81/d123/d20/d2c/f15d x:0 0 0 2024-04-20T21:32:25.095 INFO:tasks.workunit.client.1.smithi189.stdout:4/859: chown d2/dd/d6b 101247 1 2024-04-20T21:32:25.121 INFO:tasks.workunit.client.1.smithi189.stdout:6/924: dread d8/da/dd/d35/f31 [0,4194304] 0 2024-04-20T21:32:25.129 INFO:tasks.workunit.client.1.smithi189.stdout:4/860: write d2/dd/d6b/da6/fc2 [134355,3696] 0 2024-04-20T21:32:25.129 INFO:tasks.workunit.client.1.smithi189.stdout:8/998: dread da/d81/d123/d13/dd2/fbe [0,4194304] 0 2024-04-20T21:32:25.131 INFO:tasks.workunit.client.1.smithi189.stdout:2/854: sync 2024-04-20T21:32:25.131 INFO:tasks.workunit.client.1.smithi189.stdout:9/866: sync 2024-04-20T21:32:25.132 INFO:tasks.workunit.client.1.smithi189.stdout:4/861: chown d2/d20/d115/dcd/d6a/d7a/def/f101 94857 1 2024-04-20T21:32:25.135 INFO:tasks.workunit.client.1.smithi189.stdout:8/999: creat da/d81/d123/d13/dd6/df0/f15e x:0 0 0 2024-04-20T21:32:25.137 INFO:tasks.workunit.client.1.smithi189.stdout:2/855: rename d1/d11/d2d/d4f/d8e/fc4 to d1/d11/d99/db7/ff5 0 2024-04-20T21:32:25.141 INFO:tasks.workunit.client.1.smithi189.stdout:9/867: rename d9/dc/d17/d95/da0/d55/dd0/f115 to d9/d12/d62/dae/f11d 0 2024-04-20T21:32:25.143 INFO:tasks.workunit.client.1.smithi189.stdout:2/856: creat d1/d11/d2d/d73/dc6/ff6 x:0 0 0 2024-04-20T21:32:25.147 INFO:tasks.workunit.client.1.smithi189.stdout:9/868: mkdir d9/d12/d3f/d11e 0 2024-04-20T21:32:25.147 INFO:tasks.workunit.client.1.smithi189.stdout:2/857: write d1/d17/d37/d89/f95 [2279684,25017] 0 2024-04-20T21:32:25.150 INFO:tasks.workunit.client.1.smithi189.stdout:6/925: dwrite f5 [12582912,4194304] 0 2024-04-20T21:32:25.152 INFO:tasks.workunit.client.1.smithi189.stdout:9/869: creat d9/dc/d17/dac/f11f x:0 0 0 2024-04-20T21:32:25.154 INFO:tasks.workunit.client.1.smithi189.stdout:2/858: fdatasync d1/d11/d2d/dab/d4a/f8f 0 2024-04-20T21:32:25.155 INFO:tasks.workunit.client.1.smithi189.stdout:6/926: fdatasync d8/d2b/d49/d54/d6e/f86 0 2024-04-20T21:32:25.156 INFO:tasks.workunit.client.1.smithi189.stdout:4/862: dread d2/f1a [0,4194304] 0 2024-04-20T21:32:25.159 INFO:tasks.workunit.client.1.smithi189.stdout:6/927: creat d8/da/dd/d35/d6d/d7b/f134 x:0 0 0 2024-04-20T21:32:25.160 INFO:tasks.workunit.client.1.smithi189.stdout:2/859: creat d1/d11/d2d/dab/d4a/d66/dc9/ff7 x:0 0 0 2024-04-20T21:32:25.165 INFO:tasks.workunit.client.1.smithi189.stdout:4/863: mkdir d2/dd/d11/dae/d10e/d11c/d132 0 2024-04-20T21:32:25.170 INFO:tasks.workunit.client.1.smithi189.stdout:4/864: creat d2/dd/d6b/d126/f133 x:0 0 0 2024-04-20T21:32:25.171 INFO:tasks.workunit.client.1.smithi189.stdout:4/865: fsync d2/dd/f2d 0 2024-04-20T21:32:25.175 INFO:tasks.workunit.client.1.smithi189.stdout:6/928: truncate d8/d10/d78/df7/fac 1044047 0 2024-04-20T21:32:25.179 INFO:tasks.workunit.client.1.smithi189.stdout:4/866: link d2/d3a/f63 d2/d22/d57/f134 0 2024-04-20T21:32:25.182 INFO:tasks.workunit.client.1.smithi189.stdout:9/870: dwrite d9/dc/d17/dac/fc2 [0,4194304] 0 2024-04-20T21:32:25.182 INFO:tasks.workunit.client.1.smithi189.stdout:4/867: write d2/dd/d6b/f100 [724452,122960] 0 2024-04-20T21:32:25.186 INFO:tasks.workunit.client.1.smithi189.stdout:6/929: link d8/da/dd/d118/d4e/dc4/ld0 d8/d10/l135 0 2024-04-20T21:32:25.188 INFO:tasks.workunit.client.1.smithi189.stdout:9/871: fdatasync d9/dc/d17/f8e 0 2024-04-20T21:32:25.193 INFO:tasks.workunit.client.1.smithi189.stdout:9/872: mknod d9/dc/d17/d36/d5d/c120 0 2024-04-20T21:32:25.194 INFO:tasks.workunit.client.1.smithi189.stdout:4/868: write d2/f1a [3382287,92352] 0 2024-04-20T21:32:25.195 INFO:tasks.workunit.client.1.smithi189.stdout:9/873: read - d9/dc/d17/d95/fd5 zero size 2024-04-20T21:32:25.198 INFO:tasks.workunit.client.1.smithi189.stdout:4/869: symlink d2/d20/d115/dcd/da4/l135 0 2024-04-20T21:32:25.259 INFO:tasks.workunit.client.1.smithi189.stdout:0/995: sync 2024-04-20T21:32:25.259 INFO:tasks.workunit.client.1.smithi189.stdout:3/906: sync 2024-04-20T21:32:25.259 INFO:tasks.workunit.client.1.smithi189.stdout:7/978: sync 2024-04-20T21:32:25.259 INFO:tasks.workunit.client.1.smithi189.stdout:5/915: sync 2024-04-20T21:32:25.259 INFO:tasks.workunit.client.1.smithi189.stdout:1/826: sync 2024-04-20T21:32:25.263 INFO:tasks.workunit.client.1.smithi189.stdout:0/996: write d3/df/d28/d2b/d2c/f136 [988445,109388] 0 2024-04-20T21:32:25.266 INFO:tasks.workunit.client.1.smithi189.stdout:7/979: dread - d5/d21/d4a/d52/d75/fb2 zero size 2024-04-20T21:32:25.272 INFO:tasks.workunit.client.1.smithi189.stdout:5/916: unlink d7/d35/d48/f7d 0 2024-04-20T21:32:25.275 INFO:tasks.workunit.client.1.smithi189.stdout:5/917: read d7/d9/d30/d55/d61/d87/f90 [2193355,11958] 0 2024-04-20T21:32:25.277 INFO:tasks.workunit.client.1.smithi189.stdout:0/997: rename d3/df/d4c/dd4/lfc to d3/db9/dbe/dd7/l152 0 2024-04-20T21:32:25.283 INFO:tasks.workunit.client.1.smithi189.stdout:7/980: rename d5/d21/d35/f7e to d5/db4/d71/de4/f140 0 2024-04-20T21:32:25.283 INFO:tasks.workunit.client.1.smithi189.stdout:0/998: mknod d3/db9/dbe/dd7/d112/c153 0 2024-04-20T21:32:25.284 INFO:tasks.workunit.client.1.smithi189.stdout:5/918: creat d7/d9/f124 x:0 0 0 2024-04-20T21:32:25.289 INFO:tasks.workunit.client.1.smithi189.stdout:3/907: dread d2/d3/d2c/fda [0,4194304] 0 2024-04-20T21:32:25.289 INFO:tasks.workunit.client.1.smithi189.stdout:1/827: dwrite d8/d1e/d43/d58/d60/f64 [0,4194304] 0 2024-04-20T21:32:25.290 INFO:tasks.workunit.client.1.smithi189.stdout:7/981: rename d5/db4/d71/de4/ff9 to d5/d46/d7d/f141 0 2024-04-20T21:32:25.292 INFO:tasks.workunit.client.1.smithi189.stdout:0/999: creat d3/df/d28/d2b/db1/dfe/d119/d140/f154 x:0 0 0 2024-04-20T21:32:25.292 INFO:tasks.workunit.client.1.smithi189.stdout:7/982: readlink d5/d46/d7d/l133 0 2024-04-20T21:32:25.296 INFO:tasks.workunit.client.1.smithi189.stdout:5/919: unlink d7/d9/d30/d55/d104/d106/f10b 0 2024-04-20T21:32:25.301 INFO:tasks.workunit.client.1.smithi189.stdout:1/828: mknod d8/df/d3a/d49/c10d 0 2024-04-20T21:32:25.303 INFO:tasks.workunit.client.1.smithi189.stdout:7/983: unlink d5/d21/d97/da3/fef 0 2024-04-20T21:32:25.316 INFO:tasks.workunit.client.1.smithi189.stdout:1/829: mkdir d8/df/d3a/d49/d8c/da2/dbe/dc1/de9/d10e 0 2024-04-20T21:32:25.318 INFO:tasks.workunit.client.1.smithi189.stdout:5/920: getdents d7/d9/d30/d55/d61/db3 0 2024-04-20T21:32:25.320 INFO:tasks.workunit.client.1.smithi189.stdout:7/984: getdents d5/d1d/ded/d12a 0 2024-04-20T21:32:25.320 INFO:tasks.workunit.client.1.smithi189.stdout:3/908: dwrite d2/d3/d2c/d30/d10c/ff8 [0,4194304] 0 2024-04-20T21:32:25.326 INFO:tasks.workunit.client.1.smithi189.stdout:3/909: creat d2/d1c/dc1/dc5/ddf/f13d x:0 0 0 2024-04-20T21:32:25.326 INFO:tasks.workunit.client.1.smithi189.stdout:1/830: dread d8/df/d3a/d49/d8c/da2/dfe/f98 [0,4194304] 0 2024-04-20T21:32:25.327 INFO:tasks.workunit.client.1.smithi189.stdout:1/831: chown d8/df/l67 0 1 2024-04-20T21:32:25.328 INFO:tasks.workunit.client.1.smithi189.stdout:5/921: write d7/d9/d30/f8e [686983,91237] 0 2024-04-20T21:32:25.328 INFO:tasks.workunit.client.1.smithi189.stdout:7/985: getdents d5/d1d/d64 0 2024-04-20T21:32:25.333 INFO:tasks.workunit.client.1.smithi189.stdout:3/910: write d2/d3/d2c/fda [2323377,59604] 0 2024-04-20T21:32:25.333 INFO:tasks.workunit.client.1.smithi189.stdout:1/832: write d8/d1e/d54/d93/ffa [3329666,98388] 0 2024-04-20T21:32:25.337 INFO:tasks.workunit.client.1.smithi189.stdout:3/911: dread - d2/d3/d2c/d31/d74/db7/fcf zero size 2024-04-20T21:32:25.338 INFO:tasks.workunit.client.1.smithi189.stdout:5/922: rename d7/d9/d51/d70/f102 to d7/d9/f125 0 2024-04-20T21:32:25.339 INFO:tasks.workunit.client.1.smithi189.stdout:1/833: truncate d8/d21/d2c/fa3 905333 0 2024-04-20T21:32:25.339 INFO:tasks.workunit.client.1.smithi189.stdout:3/912: write d2/d3/d88/ff6 [1207558,106102] 0 2024-04-20T21:32:25.343 INFO:tasks.workunit.client.1.smithi189.stdout:7/986: read d5/f1f [2583054,37901] 0 2024-04-20T21:32:25.345 INFO:tasks.workunit.client.1.smithi189.stdout:1/834: creat d8/d1e/d43/d58/d60/f10f x:0 0 0 2024-04-20T21:32:25.346 INFO:tasks.workunit.client.1.smithi189.stdout:7/987: fsync d5/d46/d7d/d45/f48 0 2024-04-20T21:32:25.350 INFO:tasks.workunit.client.1.smithi189.stdout:5/923: write d7/d9/d28/db1/fe5 [1384172,7683] 0 2024-04-20T21:32:25.356 INFO:tasks.workunit.client.1.smithi189.stdout:7/988: write d5/f1f [621514,38096] 0 2024-04-20T21:32:25.356 INFO:tasks.workunit.client.1.smithi189.stdout:1/835: write d8/d1e/d43/d44/dd2/fc7 [167973,3124] 0 2024-04-20T21:32:25.356 INFO:tasks.workunit.client.1.smithi189.stdout:5/924: write d7/d9/d31/d36/f54 [1235581,36486] 0 2024-04-20T21:32:25.374 INFO:tasks.workunit.client.1.smithi189.stdout:3/913: dread d2/d3/d2c/f89 [0,4194304] 0 2024-04-20T21:32:25.376 INFO:tasks.workunit.client.1.smithi189.stdout:3/914: stat d2/d1c/dc1/dc5/ddf/f117 0 2024-04-20T21:32:25.379 INFO:tasks.workunit.client.1.smithi189.stdout:3/915: write d2/d3/d2c/f89 [2974725,130223] 0 2024-04-20T21:32:25.380 INFO:tasks.workunit.client.1.smithi189.stdout:5/925: dread d7/d9/d30/d55/d61/fb2 [0,4194304] 0 2024-04-20T21:32:25.382 INFO:tasks.workunit.client.1.smithi189.stdout:3/916: mknod d2/d3/d25/de0/c13e 0 2024-04-20T21:32:25.386 INFO:tasks.workunit.client.1.smithi189.stdout:5/926: rename d7/d35/f7a to d7/d9/d30/d55/d104/f126 0 2024-04-20T21:32:25.386 INFO:tasks.workunit.client.1.smithi189.stdout:3/917: mknod d2/d3/c13f 0 2024-04-20T21:32:25.388 INFO:tasks.workunit.client.1.smithi189.stdout:5/927: write d7/d9/f125 [3886120,56904] 0 2024-04-20T21:32:25.391 INFO:tasks.workunit.client.1.smithi189.stdout:5/928: dread - d7/d9/fbf zero size 2024-04-20T21:32:25.393 INFO:tasks.workunit.client.1.smithi189.stdout:3/918: dread d2/d3/d88/f99 [0,4194304] 0 2024-04-20T21:32:25.395 INFO:tasks.workunit.client.1.smithi189.stdout:5/929: fsync d7/d35/d48/fb9 0 2024-04-20T21:32:25.395 INFO:tasks.workunit.client.1.smithi189.stdout:3/919: chown d2/d3/d2c/d31/d74/c77 0 1 2024-04-20T21:32:25.396 INFO:tasks.workunit.client.1.smithi189.stdout:5/930: fdatasync d7/d9/d28/fa0 0 2024-04-20T21:32:25.399 INFO:tasks.workunit.client.1.smithi189.stdout:5/931: creat d7/d9/d28/da3/f127 x:0 0 0 2024-04-20T21:32:25.399 INFO:tasks.workunit.client.1.smithi189.stdout:3/920: truncate d2/d3/d65/fbb 655951 0 2024-04-20T21:32:25.400 INFO:tasks.workunit.client.1.smithi189.stdout:5/932: stat d7/d9/d28/db1/fe5 0 2024-04-20T21:32:25.401 INFO:tasks.workunit.client.1.smithi189.stdout:3/921: dread - d2/d3/d2c/d31/d4c/f133 zero size 2024-04-20T21:32:25.403 INFO:tasks.workunit.client.1.smithi189.stdout:5/933: write d7/d9/d30/d5a/d9d/ffd [675135,121845] 0 2024-04-20T21:32:25.405 INFO:tasks.workunit.client.1.smithi189.stdout:3/922: truncate d2/d1c/dc1/dc5/ddf/f13d 342225 0 2024-04-20T21:32:25.406 INFO:tasks.workunit.client.1.smithi189.stdout:5/934: mkdir d7/d128 0 2024-04-20T21:32:25.406 INFO:tasks.workunit.client.1.smithi189.stdout:3/923: chown d2/d1c/dc1/dc5/ddf 182773 1 2024-04-20T21:32:25.407 INFO:tasks.workunit.client.1.smithi189.stdout:5/935: stat d7/d9/d30/d55/d61/d87 0 2024-04-20T21:32:25.409 INFO:tasks.workunit.client.1.smithi189.stdout:3/924: unlink d2/d3/d25/f67 0 2024-04-20T21:32:25.410 INFO:tasks.workunit.client.1.smithi189.stdout:5/936: creat d7/d9/d30/d55/d81/f129 x:0 0 0 2024-04-20T21:32:25.410 INFO:tasks.workunit.client.1.smithi189.stdout:3/925: chown d2/d3/d2c/d30/d10c 95 1 2024-04-20T21:32:25.412 INFO:tasks.workunit.client.1.smithi189.stdout:5/937: chown d7/d9/d31/d36/de0/f120 3214 1 2024-04-20T21:32:25.413 INFO:tasks.workunit.client.1.smithi189.stdout:3/926: symlink d2/d3/d83/l140 0 2024-04-20T21:32:25.416 INFO:tasks.workunit.client.1.smithi189.stdout:3/927: fsync d2/d3/d2c/d31/d4c/fa3 0 2024-04-20T21:32:25.418 INFO:tasks.workunit.client.1.smithi189.stdout:5/938: getdents d7/d9/d31/d36 0 2024-04-20T21:32:25.420 INFO:tasks.workunit.client.1.smithi189.stdout:5/939: chown d7/d9/d51/ddf/cc5 239843 1 2024-04-20T21:32:25.421 INFO:tasks.workunit.client.1.smithi189.stdout:3/928: rename d2/d8/f4b to d2/d3/d25/f141 0 2024-04-20T21:32:25.436 INFO:tasks.workunit.client.1.smithi189.stdout:5/940: dwrite d7/d9/d30/d55/d81/f112 [4194304,4194304] 0 2024-04-20T21:32:25.438 INFO:tasks.workunit.client.1.smithi189.stdout:3/929: dwrite d2/d1c/f2b [8388608,4194304] 0 2024-04-20T21:32:25.441 INFO:tasks.workunit.client.1.smithi189.stdout:3/930: chown d2/d1c/d5c/c113 6 1 2024-04-20T21:32:25.441 INFO:tasks.workunit.client.1.smithi189.stdout:5/941: unlink d7/d9/d30/d55/d81/lfb 0 2024-04-20T21:32:25.444 INFO:tasks.workunit.client.1.smithi189.stdout:3/931: symlink d2/d3/d36/da1/l142 0 2024-04-20T21:32:25.446 INFO:tasks.workunit.client.1.smithi189.stdout:5/942: truncate d7/d9/d28/db1/fcf 427023 0 2024-04-20T21:32:25.449 INFO:tasks.workunit.client.1.smithi189.stdout:3/932: truncate d2/d3/d2c/d31/d74/d7a/df1/f137 676944 0 2024-04-20T21:32:25.449 INFO:tasks.workunit.client.1.smithi189.stdout:4/870: sync 2024-04-20T21:32:25.449 INFO:tasks.workunit.client.1.smithi189.stdout:9/874: sync 2024-04-20T21:32:25.450 INFO:tasks.workunit.client.1.smithi189.stdout:2/860: sync 2024-04-20T21:32:25.450 INFO:tasks.workunit.client.1.smithi189.stdout:6/930: sync 2024-04-20T21:32:25.453 INFO:tasks.workunit.client.1.smithi189.stdout:5/943: creat d7/d9/d28/da3/f12a x:0 0 0 2024-04-20T21:32:25.456 INFO:tasks.workunit.client.1.smithi189.stdout:5/944: chown d7/d9/d31/c3c 38394881 1 2024-04-20T21:32:25.457 INFO:tasks.workunit.client.1.smithi189.stdout:6/931: unlink d8/da/dd/d118/dd7/l123 0 2024-04-20T21:32:25.460 INFO:tasks.workunit.client.1.smithi189.stdout:2/861: rename d1/d11/d2d/dab/d4a/d66/l96 to d1/d11/d2d/dab/d74/lf8 0 2024-04-20T21:32:25.462 INFO:tasks.workunit.client.1.smithi189.stdout:9/875: link d9/dc/d17/d36/d5d/l6d d9/dc/d114/l121 0 2024-04-20T21:32:25.464 INFO:tasks.workunit.client.1.smithi189.stdout:6/932: symlink d8/d2b/l136 0 2024-04-20T21:32:25.464 INFO:tasks.workunit.client.1.smithi189.stdout:9/876: stat d9/d12/c8b 0 2024-04-20T21:32:25.466 INFO:tasks.workunit.client.1.smithi189.stdout:2/862: mknod d1/d11/d2d/d4f/db4/daa/cf9 0 2024-04-20T21:32:25.467 INFO:tasks.workunit.client.1.smithi189.stdout:9/877: chown d9/l5b 49 1 2024-04-20T21:32:25.474 INFO:tasks.workunit.client.1.smithi189.stdout:4/871: dwrite d2/d20/d115/fc3 [0,4194304] 0 2024-04-20T21:32:25.475 INFO:tasks.workunit.client.1.smithi189.stdout:6/933: truncate d8/d2b/d49/d54/f83 4956758 0 2024-04-20T21:32:25.479 INFO:tasks.workunit.client.1.smithi189.stdout:9/878: rename d9/d12/d3b/fe0 to d9/dc/d17/d36/f122 0 2024-04-20T21:32:25.483 INFO:tasks.workunit.client.1.smithi189.stdout:4/872: dread d2/d22/d57/fe3 [0,4194304] 0 2024-04-20T21:32:25.491 INFO:tasks.workunit.client.1.smithi189.stdout:2/863: dwrite d1/d11/d2d/d4f/d8e/fe2 [0,4194304] 0 2024-04-20T21:32:25.504 INFO:tasks.workunit.client.1.smithi189.stdout:5/945: dread d7/f78 [0,4194304] 0 2024-04-20T21:32:25.505 INFO:tasks.workunit.client.1.smithi189.stdout:9/879: dwrite d9/dc/d17/d95/da0/d52/f83 [0,4194304] 0 2024-04-20T21:32:25.507 INFO:tasks.workunit.client.1.smithi189.stdout:4/873: write d2/d22/d57/f9f [418187,128478] 0 2024-04-20T21:32:25.507 INFO:tasks.workunit.client.1.smithi189.stdout:6/934: dread d8/da/f6b [0,4194304] 0 2024-04-20T21:32:25.510 INFO:tasks.workunit.client.1.smithi189.stdout:9/880: write d9/dc/d17/d95/da0/d52/fd3 [758484,24351] 0 2024-04-20T21:32:25.511 INFO:tasks.workunit.client.1.smithi189.stdout:2/864: link d1/d11/d2d/dab/d4a/d66/l68 d1/d11/d2d/d73/dc6/lfa 0 2024-04-20T21:32:25.513 INFO:tasks.workunit.client.1.smithi189.stdout:4/874: chown d2/d22/d57/cb1 12085217 1 2024-04-20T21:32:25.515 INFO:tasks.workunit.client.1.smithi189.stdout:6/935: symlink d8/d10/d78/d7e/dcf/l137 0 2024-04-20T21:32:25.515 INFO:tasks.workunit.client.1.smithi189.stdout:2/865: chown d1/d11/d2d/d4f/db4/daa/dd2 5015713 1 2024-04-20T21:32:25.520 INFO:tasks.workunit.client.1.smithi189.stdout:4/875: creat d2/d20/d115/dcd/d6a/d7a/db7/f136 x:0 0 0 2024-04-20T21:32:25.521 INFO:tasks.workunit.client.1.smithi189.stdout:9/881: write d9/dc/d17/d95/da0/d56/fd2 [3259225,63704] 0 2024-04-20T21:32:25.521 INFO:tasks.workunit.client.1.smithi189.stdout:6/936: dread d8/d10/d78/d7e/dcf/fe1 [0,4194304] 0 2024-04-20T21:32:25.525 INFO:tasks.workunit.client.1.smithi189.stdout:4/876: write d2/d20/d115/dcd/d6a/f105 [322701,91795] 0 2024-04-20T21:32:25.527 INFO:tasks.workunit.client.1.smithi189.stdout:6/937: mkdir d8/d10/d78/df7/d138 0 2024-04-20T21:32:25.529 INFO:tasks.workunit.client.1.smithi189.stdout:9/882: getdents d9/dc/d114 0 2024-04-20T21:32:25.532 INFO:tasks.workunit.client.1.smithi189.stdout:9/883: chown d9/dc/d17/d95/da0/d55/dd0/lec 936 1 2024-04-20T21:32:25.536 INFO:tasks.workunit.client.1.smithi189.stdout:9/884: rmdir d9/dc/d17/d95/da6/dea/d105 39 2024-04-20T21:32:25.537 INFO:tasks.workunit.client.1.smithi189.stdout:2/866: truncate d1/d11/d2d/d4f/fe9 4117168 0 2024-04-20T21:32:25.539 INFO:tasks.workunit.client.1.smithi189.stdout:9/885: symlink d9/d12/d3f/d11e/l123 0 2024-04-20T21:32:25.540 INFO:tasks.workunit.client.1.smithi189.stdout:2/867: dread - d1/d11/d2d/dab/d74/fda zero size 2024-04-20T21:32:25.544 INFO:tasks.workunit.client.1.smithi189.stdout:9/886: truncate d9/d12/d3b/db1/d1a/f112 590256 0 2024-04-20T21:32:25.544 INFO:tasks.workunit.client.1.smithi189.stdout:2/868: truncate d1/d11/fe8 620097 0 2024-04-20T21:32:25.545 INFO:tasks.workunit.client.1.smithi189.stdout:6/938: dwrite d8/d2b/d49/d54/d6e/fb4 [0,4194304] 0 2024-04-20T21:32:25.547 INFO:tasks.workunit.client.1.smithi189.stdout:9/887: creat d9/dc/d17/dac/f124 x:0 0 0 2024-04-20T21:32:25.548 INFO:tasks.workunit.client.1.smithi189.stdout:6/939: chown d8/da/dd/d35/l1b 11875474 1 2024-04-20T21:32:25.550 INFO:tasks.workunit.client.1.smithi189.stdout:2/869: rename d1/d11/d2d/dab/f42 to d1/d11/d2d/dab/d74/ffb 0 2024-04-20T21:32:25.560 INFO:tasks.workunit.client.1.smithi189.stdout:6/940: symlink d8/da/dd/l139 0 2024-04-20T21:32:25.562 INFO:tasks.workunit.client.1.smithi189.stdout:6/941: read d8/d10/d78/df7/f93 [96208,28146] 0 2024-04-20T21:32:25.562 INFO:tasks.workunit.client.1.smithi189.stdout:2/870: mknod d1/d11/d2d/dab/cfc 0 2024-04-20T21:32:25.565 INFO:tasks.workunit.client.1.smithi189.stdout:9/888: rename d9/dc/d17 to d9/dc/d69/d125 0 2024-04-20T21:32:25.566 INFO:tasks.workunit.client.1.smithi189.stdout:6/942: mkdir d8/da/d13a 0 2024-04-20T21:32:25.569 INFO:tasks.workunit.client.1.smithi189.stdout:6/943: write d8/d2b/f87 [2569511,9647] 0 2024-04-20T21:32:25.572 INFO:tasks.workunit.client.1.smithi189.stdout:6/944: readlink d8/d10/l135 0 2024-04-20T21:32:25.576 INFO:tasks.workunit.client.1.smithi189.stdout:9/889: write d9/d12/d3b/db1/d1a/f5c [5085231,121136] 0 2024-04-20T21:32:25.582 INFO:tasks.workunit.client.1.smithi189.stdout:9/890: chown d9/dc/d69/d125/d95/da0/d52/d82/da4/df3 2322624 1 2024-04-20T21:32:25.582 INFO:tasks.workunit.client.1.smithi189.stdout:2/871: dread d1/d11/d2d/f63 [0,4194304] 0 2024-04-20T21:32:25.584 INFO:tasks.workunit.client.1.smithi189.stdout:9/891: rename d9/dc/d69/d125/d7e/lcf to d9/dc/d69/d125/d95/da0/d52/l126 0 2024-04-20T21:32:25.586 INFO:tasks.workunit.client.1.smithi189.stdout:9/892: chown d9/dc/d69/d125 4845419 1 2024-04-20T21:32:25.590 INFO:tasks.workunit.client.1.smithi189.stdout:2/872: write d1/d11/f2e [2075057,7832] 0 2024-04-20T21:32:25.592 INFO:tasks.workunit.client.1.smithi189.stdout:6/945: dwrite d8/da/dd/d118/df1/fff [0,4194304] 0 2024-04-20T21:32:25.595 INFO:tasks.workunit.client.1.smithi189.stdout:2/873: creat d1/d11/d2d/ffd x:0 0 0 2024-04-20T21:32:25.597 INFO:tasks.workunit.client.1.smithi189.stdout:6/946: write d8/d10/d78/d7e/fd4 [4150414,96553] 0 2024-04-20T21:32:25.600 INFO:tasks.workunit.client.1.smithi189.stdout:6/947: readlink d8/da/dd/d118/d30/d47/l102 0 2024-04-20T21:32:25.600 INFO:tasks.workunit.client.1.smithi189.stdout:2/874: creat d1/d11/d2d/d4f/d8e/ffe x:0 0 0 2024-04-20T21:32:25.604 INFO:tasks.workunit.client.1.smithi189.stdout:6/948: read - d8/d10/d78/df7/ff2 zero size 2024-04-20T21:32:25.607 INFO:tasks.workunit.client.1.smithi189.stdout:6/949: chown d8/d10/f20 148623 1 2024-04-20T21:32:25.609 INFO:tasks.workunit.client.1.smithi189.stdout:2/875: creat d1/d11/d2d/d73/dc6/fff x:0 0 0 2024-04-20T21:32:25.611 INFO:tasks.workunit.client.1.smithi189.stdout:9/893: dwrite d9/dc/d69/d125/f8e [0,4194304] 0 2024-04-20T21:32:25.615 INFO:tasks.workunit.client.1.smithi189.stdout:6/950: symlink d8/da/dd/d118/d12e/l13b 0 2024-04-20T21:32:25.616 INFO:tasks.workunit.client.1.smithi189.stdout:9/894: chown d9/dc/d69/d125/d95/da0/d52/l126 2211 1 2024-04-20T21:32:25.618 INFO:tasks.workunit.client.1.smithi189.stdout:9/895: write d9/d12/f119 [353222,130374] 0 2024-04-20T21:32:25.624 INFO:tasks.workunit.client.1.smithi189.stdout:6/951: rename d8/da/dd/d118/df1 to d8/da/dd/d35/d13c 0 2024-04-20T21:32:25.627 INFO:tasks.workunit.client.1.smithi189.stdout:9/896: symlink d9/dc/d69/d125/d95/da0/d52/d82/da4/l127 0 2024-04-20T21:32:25.633 INFO:tasks.workunit.client.1.smithi189.stdout:9/897: creat d9/dc/d69/d125/d95/da0/d56/f128 x:0 0 0 2024-04-20T21:32:25.637 INFO:tasks.workunit.client.1.smithi189.stdout:9/898: mkdir d9/d12/d129 0 2024-04-20T21:32:25.639 INFO:tasks.workunit.client.1.smithi189.stdout:9/899: chown d9/d12/d3b/c3c 7695 1 2024-04-20T21:32:25.643 INFO:tasks.workunit.client.1.smithi189.stdout:6/952: dwrite d8/d10/f36 [0,4194304] 0 2024-04-20T21:32:25.644 INFO:tasks.workunit.client.1.smithi189.stdout:2/876: dread d1/d17/f2b [0,4194304] 0 2024-04-20T21:32:25.644 INFO:tasks.workunit.client.1.smithi189.stdout:9/900: creat d9/d12/d62/dd9/df0/f12a x:0 0 0 2024-04-20T21:32:25.655 INFO:tasks.workunit.client.1.smithi189.stdout:9/901: getdents d9/d12/d62/dd9 0 2024-04-20T21:32:25.657 INFO:tasks.workunit.client.1.smithi189.stdout:9/902: write f1 [4572819,69140] 0 2024-04-20T21:32:25.660 INFO:tasks.workunit.client.1.smithi189.stdout:9/903: readlink d9/dc/d69/d125/d95/da0/d52/d82/d94/lcb 0 2024-04-20T21:32:25.663 INFO:tasks.workunit.client.1.smithi189.stdout:9/904: fsync d9/d12/d3b/db1/d1a/f112 0 2024-04-20T21:32:25.666 INFO:tasks.workunit.client.1.smithi189.stdout:2/877: dwrite d1/d11/d2d/dab/d74/ffb [0,4194304] 0 2024-04-20T21:32:25.669 INFO:tasks.workunit.client.1.smithi189.stdout:9/905: creat d9/dc/d69/d125/d95/da0/d55/dd0/f12b x:0 0 0 2024-04-20T21:32:25.673 INFO:tasks.workunit.client.1.smithi189.stdout:2/878: creat d1/d11/d2d/dab/d4a/d66/df1/f100 x:0 0 0 2024-04-20T21:32:25.679 INFO:tasks.workunit.client.1.smithi189.stdout:9/906: write d9/dc/d69/d125/d36/f78 [3438615,112680] 0 2024-04-20T21:32:25.683 INFO:tasks.workunit.client.1.smithi189.stdout:9/907: symlink d9/d12/d3f/d72/d11b/l12c 0 2024-04-20T21:32:25.687 INFO:tasks.workunit.client.1.smithi189.stdout:2/879: dwrite d1/f7 [4194304,4194304] 0 2024-04-20T21:32:25.690 INFO:tasks.workunit.client.1.smithi189.stdout:9/908: unlink d9/dc/d69/d125/d95/fd5 0 2024-04-20T21:32:25.691 INFO:tasks.workunit.client.1.smithi189.stdout:2/880: rmdir d1/d11/d99/db7 39 2024-04-20T21:32:25.695 INFO:tasks.workunit.client.1.smithi189.stdout:9/909: dread - d9/d12/d62/dd9/df0/f12a zero size 2024-04-20T21:32:25.696 INFO:tasks.workunit.client.1.smithi189.stdout:2/881: fsync d1/d11/d2d/dab/d4a/d66/dc9/ff7 0 2024-04-20T21:32:25.703 INFO:tasks.workunit.client.1.smithi189.stdout:9/910: rename d9/d12/d62/dae/f11d to d9/dc/d69/d125/d36/f12d 0 2024-04-20T21:32:25.706 INFO:tasks.workunit.client.1.smithi189.stdout:9/911: creat d9/d12/d3b/db1/f12e x:0 0 0 2024-04-20T21:32:25.713 INFO:tasks.workunit.client.1.smithi189.stdout:2/882: dwrite d1/d11/f15 [4194304,4194304] 0 2024-04-20T21:32:25.734 INFO:tasks.workunit.client.1.smithi189.stdout:9/912: dread d9/dc/ffb [0,4194304] 0 2024-04-20T21:32:25.737 INFO:tasks.workunit.client.1.smithi189.stdout:9/913: mkdir d9/d12/d129/d12f 0 2024-04-20T21:32:25.743 INFO:tasks.workunit.client.1.smithi189.stdout:2/883: dread d1/d11/d2d/f50 [0,4194304] 0 2024-04-20T21:32:25.745 INFO:tasks.workunit.client.1.smithi189.stdout:2/884: write d1/d17/d78/fca [67337,88332] 0 2024-04-20T21:32:25.749 INFO:tasks.workunit.client.1.smithi189.stdout:1/836: sync 2024-04-20T21:32:25.749 INFO:tasks.workunit.client.1.smithi189.stdout:7/989: sync 2024-04-20T21:32:25.755 INFO:tasks.workunit.client.1.smithi189.stdout:9/914: dwrite d9/d12/d3f/f68 [4194304,4194304] 0 2024-04-20T21:32:25.765 INFO:tasks.workunit.client.1.smithi189.stdout:9/915: mknod d9/dc/dab/c130 0 2024-04-20T21:32:25.779 INFO:tasks.workunit.client.1.smithi189.stdout:1/837: dread d8/f1a [0,4194304] 0 2024-04-20T21:32:25.779 INFO:tasks.workunit.client.1.smithi189.stdout:9/916: truncate d9/dc/d69/d125/d95/da0/d56/f86 1456132 0 2024-04-20T21:32:25.781 INFO:tasks.workunit.client.1.smithi189.stdout:3/933: sync 2024-04-20T21:32:25.781 INFO:tasks.workunit.client.1.smithi189.stdout:5/946: sync 2024-04-20T21:32:25.786 INFO:tasks.workunit.client.1.smithi189.stdout:1/838: creat d8/df/d3a/dfd/f110 x:0 0 0 2024-04-20T21:32:25.787 INFO:tasks.workunit.client.1.smithi189.stdout:5/947: truncate d7/d9/d30/f8e 5307157 0 2024-04-20T21:32:25.787 INFO:tasks.workunit.client.1.smithi189.stdout:2/885: dread d1/d11/d99/db7/fb8 [0,4194304] 0 2024-04-20T21:32:25.787 INFO:tasks.workunit.client.1.smithi189.stdout:9/917: write f6 [5086449,91683] 0 2024-04-20T21:32:25.789 INFO:tasks.workunit.client.1.smithi189.stdout:5/948: chown d7/d9/fbf 186601921 1 2024-04-20T21:32:25.789 INFO:tasks.workunit.client.1.smithi189.stdout:3/934: rename d2/d3/d36/da1/fa4 to d2/d1c/d5c/dca/f143 0 2024-04-20T21:32:25.792 INFO:tasks.workunit.client.1.smithi189.stdout:2/886: dread - d1/d17/d37/d89/fcc zero size 2024-04-20T21:32:25.793 INFO:tasks.workunit.client.1.smithi189.stdout:5/949: readlink d7/d9/d51/d70/d95/l11c 0 2024-04-20T21:32:25.796 INFO:tasks.workunit.client.1.smithi189.stdout:3/935: mkdir d2/d3/d83/d144 0 2024-04-20T21:32:25.798 INFO:tasks.workunit.client.1.smithi189.stdout:2/887: creat d1/d17/d37/d89/dc5/f101 x:0 0 0 2024-04-20T21:32:25.798 INFO:tasks.workunit.client.1.smithi189.stdout:9/918: write d9/d12/d3b/db1/d1a/f64 [3704069,89002] 0 2024-04-20T21:32:25.802 INFO:tasks.workunit.client.1.smithi189.stdout:5/950: truncate d7/d9/d30/d55/d61/daa/fad 439514 0 2024-04-20T21:32:25.805 INFO:tasks.workunit.client.1.smithi189.stdout:2/888: symlink d1/d11/de1/df0/l102 0 2024-04-20T21:32:25.806 INFO:tasks.workunit.client.1.smithi189.stdout:7/990: dread d5/d1d/d64/feb [0,4194304] 0 2024-04-20T21:32:25.809 INFO:tasks.workunit.client.1.smithi189.stdout:9/919: rename d9/d12/d3b/db1/d1a/l45 to d9/d12/d62/dae/l131 0 2024-04-20T21:32:25.811 INFO:tasks.workunit.client.1.smithi189.stdout:5/951: symlink d7/d9/d30/d55/d61/db3/l12b 0 2024-04-20T21:32:25.814 INFO:tasks.workunit.client.1.smithi189.stdout:1/839: dwrite d8/df/d50/ff4 [0,4194304] 0 2024-04-20T21:32:25.821 INFO:tasks.workunit.client.1.smithi189.stdout:1/840: chown c5 20966317 1 2024-04-20T21:32:25.822 INFO:tasks.workunit.client.1.smithi189.stdout:2/889: link d1/d11/c3d d1/d11/d2d/dab/d4a/d66/dc9/c103 0 2024-04-20T21:32:25.825 INFO:tasks.workunit.client.1.smithi189.stdout:1/841: chown d8/le3 2 1 2024-04-20T21:32:25.832 INFO:tasks.workunit.client.1.smithi189.stdout:3/936: dread d2/f15 [0,4194304] 0 2024-04-20T21:32:25.832 INFO:tasks.workunit.client.1.smithi189.stdout:9/920: truncate d9/dc/d69/d125/d36/d5d/f97 1285218 0 2024-04-20T21:32:25.834 INFO:tasks.workunit.client.1.smithi189.stdout:1/842: creat d8/df/d3a/d49/d8c/da2/dbe/dc1/de9/f111 x:0 0 0 2024-04-20T21:32:25.836 INFO:tasks.workunit.client.1.smithi189.stdout:3/937: creat d2/d3/d2c/d30/f145 x:0 0 0 2024-04-20T21:32:25.839 INFO:tasks.workunit.client.1.smithi189.stdout:1/843: fdatasync d8/f11 0 2024-04-20T21:32:25.840 INFO:tasks.workunit.client.1.smithi189.stdout:9/921: rename d9/dc/d69/d125/dac to d9/dc/d69/d125/d95/da0/d52/d132 0 2024-04-20T21:32:25.840 INFO:tasks.workunit.client.1.smithi189.stdout:3/938: chown d2/d3/d2c/d30/f145 6 1 2024-04-20T21:32:25.840 INFO:tasks.workunit.client.1.smithi189.stdout:7/991: dread d5/d1d/d64/f85 [0,4194304] 0 2024-04-20T21:32:25.845 INFO:tasks.workunit.client.1.smithi189.stdout:9/922: write d9/dc/d69/d125/d95/da0/d52/d82/fa7 [421853,123093] 0 2024-04-20T21:32:25.846 INFO:tasks.workunit.client.1.smithi189.stdout:1/844: mkdir d8/d1e/d43/d58/d112 0 2024-04-20T21:32:25.848 INFO:tasks.workunit.client.1.smithi189.stdout:7/992: symlink d5/d1d/ded/d12a/l142 0 2024-04-20T21:32:25.849 INFO:tasks.workunit.client.1.smithi189.stdout:1/845: chown d8/df/d3a/d49/d8c/da2/dbe/dc1/de9 280485 1 2024-04-20T21:32:25.849 INFO:tasks.workunit.client.1.smithi189.stdout:9/923: mknod d9/c133 0 2024-04-20T21:32:25.851 INFO:tasks.workunit.client.1.smithi189.stdout:9/924: chown f4 804496 1 2024-04-20T21:32:25.851 INFO:tasks.workunit.client.1.smithi189.stdout:7/993: write d5/d46/fb7 [3453943,78262] 0 2024-04-20T21:32:25.851 INFO:tasks.workunit.client.1.smithi189.stdout:3/939: write d2/d3/d88/ff5 [434089,70896] 0 2024-04-20T21:32:25.852 INFO:tasks.workunit.client.1.smithi189.stdout:5/952: dread d7/d9/d31/d100/f8b [0,4194304] 0 2024-04-20T21:32:25.852 INFO:tasks.workunit.client.1.smithi189.stdout:1/846: mkdir d8/d21/d2c/d113 0 2024-04-20T21:32:25.857 INFO:tasks.workunit.client.1.smithi189.stdout:1/847: creat d8/df/d3a/dfd/f114 x:0 0 0 2024-04-20T21:32:25.862 INFO:tasks.workunit.client.1.smithi189.stdout:1/848: truncate d8/d1e/d43/d58/fe7 482835 0 2024-04-20T21:32:25.862 INFO:tasks.workunit.client.1.smithi189.stdout:5/953: rename d7/d9/d31/d36/de0/d115 to d7/d9/d51/d70/d99/d12c 0 2024-04-20T21:32:25.865 INFO:tasks.workunit.client.1.smithi189.stdout:3/940: link d2/d3/d65/c6a d2/d3/d2c/d30/d10c/d112/c146 0 2024-04-20T21:32:25.867 INFO:tasks.workunit.client.1.smithi189.stdout:5/954: symlink d7/d9/d30/d5a/l12d 0 2024-04-20T21:32:25.869 INFO:tasks.workunit.client.1.smithi189.stdout:9/925: truncate d9/dc/d69/d125/d36/f57 1719000 0 2024-04-20T21:32:25.870 INFO:tasks.workunit.client.1.smithi189.stdout:1/849: link d8/d1e/f87 d8/d1e/d43/d58/d60/f115 0 2024-04-20T21:32:25.871 INFO:tasks.workunit.client.1.smithi189.stdout:5/955: mknod d7/d35/d48/c12e 0 2024-04-20T21:32:25.873 INFO:tasks.workunit.client.1.smithi189.stdout:3/941: link d2/d3/d2c/d31/d90/d110/f11d d2/d3/dfd/f147 0 2024-04-20T21:32:25.874 INFO:tasks.workunit.client.1.smithi189.stdout:7/994: dread d5/d46/fa9 [0,4194304] 0 2024-04-20T21:32:25.874 INFO:tasks.workunit.client.1.smithi189.stdout:9/926: write d9/dc/d69/faf [1242514,127258] 0 2024-04-20T21:32:25.875 INFO:tasks.workunit.client.1.smithi189.stdout:1/850: readlink d8/d1e/d43/d44/dd2/lac 0 2024-04-20T21:32:25.877 INFO:tasks.workunit.client.1.smithi189.stdout:5/956: creat d7/d35/d48/f12f x:0 0 0 2024-04-20T21:32:25.877 INFO:tasks.workunit.client.1.smithi189.stdout:1/851: chown d8/d21/d2c/c59 89018598 1 2024-04-20T21:32:25.880 INFO:tasks.workunit.client.1.smithi189.stdout:9/927: mknod d9/dc/d69/d125/d95/da0/d52/c134 0 2024-04-20T21:32:25.882 INFO:tasks.workunit.client.1.smithi189.stdout:5/957: symlink d7/d35/l130 0 2024-04-20T21:32:25.882 INFO:tasks.workunit.client.1.smithi189.stdout:9/928: chown d9/dc/d69/d125/d7e/c84 26403 1 2024-04-20T21:32:25.887 INFO:tasks.workunit.client.1.smithi189.stdout:5/958: symlink d7/d9/d51/ddf/l131 0 2024-04-20T21:32:25.891 INFO:tasks.workunit.client.1.smithi189.stdout:1/852: link d8/df/d3a/d49/d8c/da2/fc0 d8/df/d3a/f116 0 2024-04-20T21:32:25.894 INFO:tasks.workunit.client.1.smithi189.stdout:5/959: creat d7/d9/d31/f132 x:0 0 0 2024-04-20T21:32:25.897 INFO:tasks.workunit.client.1.smithi189.stdout:9/929: creat d9/dc/d69/d125/f135 x:0 0 0 2024-04-20T21:32:25.899 INFO:tasks.workunit.client.1.smithi189.stdout:1/853: getdents d8/df/d3a/d49/d8c/da2/dbe/dc1/dd9/dea 0 2024-04-20T21:32:25.899 INFO:tasks.workunit.client.1.smithi189.stdout:3/942: write d2/f2a [781238,83359] 0 2024-04-20T21:32:25.902 INFO:tasks.workunit.client.1.smithi189.stdout:1/854: chown d8/df/d3a/d49/d8c/da2/dfe/lc8 465 1 2024-04-20T21:32:25.903 INFO:tasks.workunit.client.1.smithi189.stdout:9/930: creat d9/dc/d69/d125/d36/f136 x:0 0 0 2024-04-20T21:32:25.945 INFO:tasks.workunit.client.1.smithi189.stdout:4/877: sync 2024-04-20T21:32:25.945 INFO:tasks.workunit.client.1.smithi189.stdout:6/953: sync 2024-04-20T21:32:25.972 INFO:tasks.workunit.client.1.smithi189.stdout:3/943: mkdir d2/d3/d2c/d30/d10c/d112/d148 0 2024-04-20T21:32:25.974 INFO:tasks.workunit.client.1.smithi189.stdout:7/995: dwrite d5/d21/d25/f30 [0,4194304] 0 2024-04-20T21:32:25.975 INFO:tasks.workunit.client.1.smithi189.stdout:6/954: chown d8/d10/f3f 1434 1 2024-04-20T21:32:25.976 INFO:tasks.workunit.client.1.smithi189.stdout:9/931: write d9/d12/d50/f4f [8538764,96845] 0 2024-04-20T21:32:25.982 INFO:tasks.workunit.client.1.smithi189.stdout:3/944: stat d2/d1c/d5c/dca/f143 0 2024-04-20T21:32:25.985 INFO:tasks.workunit.client.1.smithi189.stdout:7/996: symlink d5/d21/d97/da3/l143 0 2024-04-20T21:32:25.985 INFO:tasks.workunit.client.1.smithi189.stdout:6/955: read d8/d10/d78/df7/fd5 [964536,6973] 0 2024-04-20T21:32:25.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:25 smithi189 ceph-mon[21562]: pgmap v27: 65 pgs: 65 active+clean; 3.9 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 126 MiB/s rd, 293 MiB/s wr, 655 op/s 2024-04-20T21:32:25.987 INFO:tasks.workunit.client.1.smithi189.stdout:1/855: dread d8/df/d3a/d49/d8c/fb9 [0,4194304] 0 2024-04-20T21:32:25.988 INFO:tasks.workunit.client.1.smithi189.stdout:9/932: symlink d9/dc/l137 0 2024-04-20T21:32:25.989 INFO:tasks.workunit.client.1.smithi189.stdout:7/997: write d5/db4/d71/de4/f140 [4124993,47695] 0 2024-04-20T21:32:25.989 INFO:tasks.workunit.client.1.smithi189.stdout:6/956: symlink d8/da/dd/d118/d30/d47/ddc/l13d 0 2024-04-20T21:32:26.006 INFO:tasks.workunit.client.1.smithi189.stdout:7/998: dwrite d5/d21/d4a/d52/d98/f135 [0,4194304] 0 2024-04-20T21:32:26.008 INFO:tasks.workunit.client.1.smithi189.stdout:7/999: dread - d5/d1d/d64/db9/dcf/f11e zero size 2024-04-20T21:32:26.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:25 smithi062 ceph-mon[16221]: pgmap v27: 65 pgs: 65 active+clean; 3.9 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 126 MiB/s rd, 293 MiB/s wr, 655 op/s 2024-04-20T21:32:26.056 INFO:tasks.workunit.client.1.smithi189.stdout:1/856: mkdir d8/d1e/d43/d44/dd2/d117 0 2024-04-20T21:32:26.056 INFO:tasks.workunit.client.1.smithi189.stdout:3/945: link d2/d3/d65/fbb d2/d1c/dc1/f149 0 2024-04-20T21:32:26.059 INFO:tasks.workunit.client.1.smithi189.stdout:9/933: mkdir d9/d12/d129/d138 0 2024-04-20T21:32:26.067 INFO:tasks.workunit.client.1.smithi189.stdout:2/890: sync 2024-04-20T21:32:26.131 INFO:tasks.workunit.client.1.smithi189.stdout:5/960: sync 2024-04-20T21:32:26.131 INFO:tasks.workunit.client.1.smithi189.stdout:4/878: sync 2024-04-20T21:32:26.141 INFO:tasks.workunit.client.1.smithi189.stdout:1/857: symlink d8/d9b/l118 0 2024-04-20T21:32:26.147 INFO:tasks.workunit.client.1.smithi189.stdout:9/934: creat d9/dc/d69/d125/d95/da0/d56/f139 x:0 0 0 2024-04-20T21:32:26.147 INFO:tasks.workunit.client.1.smithi189.stdout:2/891: unlink d1/d11/d2d/dab/d74/ld0 0 2024-04-20T21:32:26.153 INFO:tasks.workunit.client.1.smithi189.stdout:3/946: unlink d2/c35 0 2024-04-20T21:32:26.155 INFO:tasks.workunit.client.1.smithi189.stdout:4/879: getdents d2/d20/d115/dcd/d6a/d7a/db7/dea 0 2024-04-20T21:32:26.156 INFO:tasks.workunit.client.1.smithi189.stdout:5/961: dread d7/d9/d30/d55/d61/f8d [0,4194304] 0 2024-04-20T21:32:26.158 INFO:tasks.workunit.client.1.smithi189.stdout:9/935: dread d9/dc/d69/d125/d7e/f8d [0,4194304] 0 2024-04-20T21:32:26.160 INFO:tasks.workunit.client.1.smithi189.stdout:9/936: write f1 [5371516,85103] 0 2024-04-20T21:32:26.168 INFO:tasks.workunit.client.1.smithi189.stdout:9/937: mknod d9/d12/d3f/d72/c13a 0 2024-04-20T21:32:26.170 INFO:tasks.workunit.client.1.smithi189.stdout:1/858: dread d8/df/f15 [0,4194304] 0 2024-04-20T21:32:26.171 INFO:tasks.workunit.client.1.smithi189.stdout:5/962: creat d7/d9/d28/f133 x:0 0 0 2024-04-20T21:32:26.174 INFO:tasks.workunit.client.1.smithi189.stdout:9/938: mkdir d9/dc/d69/d125/d95/da0/d52/d82/da4/d13b 0 2024-04-20T21:32:26.174 INFO:tasks.workunit.client.1.smithi189.stdout:5/963: chown d7/d9/d31/d100/d4e/d5b/df1/dab/le9 310 1 2024-04-20T21:32:26.176 INFO:tasks.workunit.client.1.smithi189.stdout:9/939: stat d9/d12/d3f/d72/f7b 0 2024-04-20T21:32:26.181 INFO:tasks.workunit.client.1.smithi189.stdout:1/859: rmdir d8/df/d3a/d49/d8c/da2/dbe/dc1/dd9 39 2024-04-20T21:32:26.182 INFO:tasks.workunit.client.1.smithi189.stdout:3/947: truncate d2/d3/d2c/d30/d3f/f103 466356 0 2024-04-20T21:32:26.184 INFO:tasks.workunit.client.1.smithi189.stdout:9/940: chown d9/dc/f3d 521 1 2024-04-20T21:32:26.186 INFO:tasks.workunit.client.1.smithi189.stdout:1/860: symlink d8/d1e/d43/d44/dd2/d8a/l119 0 2024-04-20T21:32:26.186 INFO:tasks.workunit.client.1.smithi189.stdout:9/941: read - d9/dc/d69/fdb zero size 2024-04-20T21:32:26.187 INFO:tasks.workunit.client.1.smithi189.stdout:3/948: mknod d2/d3/d2c/d30/d10c/d112/c14a 0 2024-04-20T21:32:26.194 INFO:tasks.workunit.client.1.smithi189.stdout:5/964: dwrite d7/d9/d31/d100/d4e/d5b/f68 [0,4194304] 0 2024-04-20T21:32:26.194 INFO:tasks.workunit.client.1.smithi189.stdout:9/942: mkdir d9/dc/d13c 0 2024-04-20T21:32:26.195 INFO:tasks.workunit.client.1.smithi189.stdout:9/943: dread d9/dc/d69/f76 [0,4194304] 0 2024-04-20T21:32:26.195 INFO:tasks.workunit.client.1.smithi189.stdout:1/861: mkdir d8/df/d3a/d49/d8c/da2/dbe/dc1/de9/d11a 0 2024-04-20T21:32:26.195 INFO:tasks.workunit.client.1.smithi189.stdout:5/965: chown d7/d9/d31/d100/d4e/d5b/df1/ce2 168042004 1 2024-04-20T21:32:26.195 INFO:tasks.workunit.client.1.smithi189.stdout:3/949: rename d2/d3/d88/ff6 to d2/d3/d2c/d30/d3f/f14b 0 2024-04-20T21:32:26.197 INFO:tasks.workunit.client.1.smithi189.stdout:1/862: write d8/d1e/d43/d44/dd2/db7/f101 [586532,99554] 0 2024-04-20T21:32:26.197 INFO:tasks.workunit.client.1.smithi189.stdout:3/950: mkdir d2/d1c/dc1/d14c 0 2024-04-20T21:32:26.204 INFO:tasks.workunit.client.1.smithi189.stdout:9/944: mkdir d9/dc/d69/d125/d95/da0/d52/d82/d13d 0 2024-04-20T21:32:26.210 INFO:tasks.workunit.client.1.smithi189.stdout:5/966: link d7/d9/d51/d70/d95/cd7 d7/d9/d30/d55/d61/d87/c134 0 2024-04-20T21:32:26.211 INFO:tasks.workunit.client.1.smithi189.stdout:3/951: link d2/d3/l73 d2/d1c/d5c/dca/l14d 0 2024-04-20T21:32:26.214 INFO:tasks.workunit.client.1.smithi189.stdout:9/945: rename d9/dc/de9 to d9/dc/d69/d125/d7e/d13e 0 2024-04-20T21:32:26.214 INFO:tasks.workunit.client.1.smithi189.stdout:3/952: write d2/d1c/dc1/dc5/ddf/f11f [51356,31443] 0 2024-04-20T21:32:26.216 INFO:tasks.workunit.client.1.smithi189.stdout:9/946: chown d9/dc/d69 9908443 1 2024-04-20T21:32:26.219 INFO:tasks.workunit.client.1.smithi189.stdout:9/947: write d9/dc/d69/d125/d95/da0/d52/fd3 [1299300,59170] 0 2024-04-20T21:32:26.226 INFO:tasks.workunit.client.1.smithi189.stdout:9/948: stat d9/d12/d3b/db1/f8f 0 2024-04-20T21:32:26.233 INFO:tasks.workunit.client.1.smithi189.stdout:9/949: link d9/dc/d69/d125/d95/da0/d52/d82/d94/fc4 d9/d12/d3b/db1/d1a/f13f 0 2024-04-20T21:32:26.235 INFO:tasks.workunit.client.1.smithi189.stdout:9/950: chown d9/d12/d62/dae 1 1 2024-04-20T21:32:26.241 INFO:tasks.workunit.client.1.smithi189.stdout:9/951: getdents d9/dc/d69/d125 0 2024-04-20T21:32:26.246 INFO:tasks.workunit.client.1.smithi189.stdout:9/952: creat d9/dc/d69/d125/d95/da0/d52/d82/d13d/f140 x:0 0 0 2024-04-20T21:32:26.247 INFO:tasks.workunit.client.1.smithi189.stdout:9/953: chown c2 2825537 1 2024-04-20T21:32:26.254 INFO:tasks.workunit.client.1.smithi189.stdout:9/954: getdents d9/dc/d69/d125/d7e/de8 0 2024-04-20T21:32:26.257 INFO:tasks.workunit.client.1.smithi189.stdout:6/957: dwrite d8/da/dd/d118/d4e/f50 [0,4194304] 0 2024-04-20T21:32:26.261 INFO:tasks.workunit.client.1.smithi189.stdout:6/958: rmdir d8/da/dd/d118/d4e 39 2024-04-20T21:32:26.273 INFO:tasks.workunit.client.1.smithi189.stdout:5/967: write d7/d9/d30/d55/d104/f126 [2236508,97694] 0 2024-04-20T21:32:26.274 INFO:tasks.workunit.client.1.smithi189.stdout:4/880: creat d2/dd/f137 x:0 0 0 2024-04-20T21:32:26.280 INFO:tasks.workunit.client.1.smithi189.stdout:1/863: dwrite d8/f46 [0,4194304] 0 2024-04-20T21:32:26.283 INFO:tasks.workunit.client.1.smithi189.stdout:2/892: sync 2024-04-20T21:32:26.284 INFO:tasks.workunit.client.1.smithi189.stdout:5/968: rename d7/c9c to d7/d9/d51/c135 0 2024-04-20T21:32:26.288 INFO:tasks.workunit.client.1.smithi189.stdout:3/953: dwrite d2/d1c/d5c/f87 [0,4194304] 0 2024-04-20T21:32:26.291 INFO:tasks.workunit.client.1.smithi189.stdout:2/893: unlink d1/d11/d2d/d4f/fb2 0 2024-04-20T21:32:26.293 INFO:tasks.workunit.client.1.smithi189.stdout:5/969: readlink d7/d9/d31/d100/d4e/d5b/df1/dab/le9 0 2024-04-20T21:32:26.294 INFO:tasks.workunit.client.1.smithi189.stdout:6/959: dwrite d8/d2b/d49/d54/d67/fab [0,4194304] 0 2024-04-20T21:32:26.294 INFO:tasks.workunit.client.1.smithi189.stdout:9/955: dread d9/d12/d3f/d91/fa3 [0,4194304] 0 2024-04-20T21:32:26.295 INFO:tasks.workunit.client.1.smithi189.stdout:1/864: chown d8/df/d3a/d49/d8c/da2/dbe/cd6 867 1 2024-04-20T21:32:26.299 INFO:tasks.workunit.client.1.smithi189.stdout:2/894: rmdir d1/d11/d2d/dab/d74 39 2024-04-20T21:32:26.304 INFO:tasks.workunit.client.1.smithi189.stdout:4/881: dwrite d2/dd/d11/d14/f50 [0,4194304] 0 2024-04-20T21:32:26.311 INFO:tasks.workunit.client.1.smithi189.stdout:2/895: read d1/d11/d2d/d4f/db4/f5a [770470,69254] 0 2024-04-20T21:32:26.316 INFO:tasks.workunit.client.1.smithi189.stdout:1/865: creat d8/d1e/d43/d44/dd2/db7/f11b x:0 0 0 2024-04-20T21:32:26.318 INFO:tasks.workunit.client.1.smithi189.stdout:4/882: creat d2/dd/d11/d14/d11f/f138 x:0 0 0 2024-04-20T21:32:26.321 INFO:tasks.workunit.client.1.smithi189.stdout:5/970: rename d7/d9/d30/d55 to d7/d136 0 2024-04-20T21:32:26.324 INFO:tasks.workunit.client.1.smithi189.stdout:1/866: stat d8/d1e/d43/d44/dd2/l4c 0 2024-04-20T21:32:26.326 INFO:tasks.workunit.client.1.smithi189.stdout:4/883: rename d2/d22/d57/c129 to d2/dd/d6b/c139 0 2024-04-20T21:32:26.329 INFO:tasks.workunit.client.1.smithi189.stdout:5/971: symlink d7/d136/d33/l137 0 2024-04-20T21:32:26.331 INFO:tasks.workunit.client.1.smithi189.stdout:3/954: truncate d2/d3/d25/f97 3822572 0 2024-04-20T21:32:26.332 INFO:tasks.workunit.client.1.smithi189.stdout:9/956: dwrite d9/d12/d3b/db1/f12e [0,4194304] 0 2024-04-20T21:32:26.336 INFO:tasks.workunit.client.1.smithi189.stdout:5/972: fdatasync d7/d35/d48/f12f 0 2024-04-20T21:32:26.337 INFO:tasks.workunit.client.1.smithi189.stdout:9/957: chown d9/d12/d50/f4f 162810419 1 2024-04-20T21:32:26.341 INFO:tasks.workunit.client.1.smithi189.stdout:3/955: unlink d2/d1c/d5c/c86 0 2024-04-20T21:32:26.345 INFO:tasks.workunit.client.1.smithi189.stdout:5/973: write d7/d136/d61/d87/f90 [3696913,39148] 0 2024-04-20T21:32:26.348 INFO:tasks.workunit.client.1.smithi189.stdout:6/960: dwrite d8/d10/f20 [0,4194304] 0 2024-04-20T21:32:26.348 INFO:tasks.workunit.client.1.smithi189.stdout:9/958: chown d9/dc/d69/d125/d95/da0/d56/f86 584273186 1 2024-04-20T21:32:26.349 INFO:tasks.workunit.client.1.smithi189.stdout:3/956: unlink d2/d3/d25/de0/da9/cc6 0 2024-04-20T21:32:26.352 INFO:tasks.workunit.client.1.smithi189.stdout:1/867: dwrite d8/d1e/ff5 [0,4194304] 0 2024-04-20T21:32:26.355 INFO:tasks.workunit.client.1.smithi189.stdout:2/896: dread d1/d11/d52/d8b/f8d [0,4194304] 0 2024-04-20T21:32:26.357 INFO:tasks.workunit.client.1.smithi189.stdout:1/868: readlink d8/d1e/l81 0 2024-04-20T21:32:26.360 INFO:tasks.workunit.client.1.smithi189.stdout:9/959: rename d9/dc/d69/d125/d95/da0/d56/c61 to d9/c141 0 2024-04-20T21:32:26.360 INFO:tasks.workunit.client.1.smithi189.stdout:2/897: fdatasync d1/d11/d2d/d4f/d8e/ffe 0 2024-04-20T21:32:26.362 INFO:tasks.workunit.client.1.smithi189.stdout:3/957: creat d2/d1c/d5c/dca/d100/f14e x:0 0 0 2024-04-20T21:32:26.365 INFO:tasks.workunit.client.1.smithi189.stdout:5/974: rename d7/d136/d61/daa/d116 to d7/d9/d51/d70/d95/d138 0 2024-04-20T21:32:26.371 INFO:tasks.workunit.client.1.smithi189.stdout:9/960: truncate d9/d12/d3f/d91/fa3 266948 0 2024-04-20T21:32:26.372 INFO:tasks.workunit.client.1.smithi189.stdout:5/975: mknod d7/d9/d28/db1/c139 0 2024-04-20T21:32:26.375 INFO:tasks.workunit.client.1.smithi189.stdout:3/958: rename d2/d1c/d5c/dca/c129 to d2/c14f 0 2024-04-20T21:32:26.377 INFO:tasks.workunit.client.1.smithi189.stdout:9/961: creat d9/dc/d69/d125/d95/da6/f142 x:0 0 0 2024-04-20T21:32:26.379 INFO:tasks.workunit.client.1.smithi189.stdout:3/959: chown d2/d8d/cff 12109 1 2024-04-20T21:32:26.380 INFO:tasks.workunit.client.1.smithi189.stdout:6/961: dwrite d8/f12d [0,4194304] 0 2024-04-20T21:32:26.381 INFO:tasks.workunit.client.1.smithi189.stdout:5/976: chown d7/d9/d31/d100/d4e/d5b/c84 18 1 2024-04-20T21:32:26.386 INFO:tasks.workunit.client.1.smithi189.stdout:5/977: rmdir d7/d136/d81 39 2024-04-20T21:32:26.388 INFO:tasks.workunit.client.1.smithi189.stdout:3/960: truncate d2/d1c/f76 1673078 0 2024-04-20T21:32:26.389 INFO:tasks.workunit.client.1.smithi189.stdout:6/962: dread - d8/d2b/d49/d54/d6e/d11f/f132 zero size 2024-04-20T21:32:26.390 INFO:tasks.workunit.client.1.smithi189.stdout:1/869: dwrite d8/d1e/d43/d58/d60/f64 [0,4194304] 0 2024-04-20T21:32:26.394 INFO:tasks.workunit.client.1.smithi189.stdout:3/961: dread - d2/d8d/f101 zero size 2024-04-20T21:32:26.396 INFO:tasks.workunit.client.1.smithi189.stdout:6/963: rmdir d8/da 39 2024-04-20T21:32:26.399 INFO:tasks.workunit.client.1.smithi189.stdout:3/962: unlink d2/d3/f48 0 2024-04-20T21:32:26.404 INFO:tasks.workunit.client.1.smithi189.stdout:5/978: dread d7/d136/d33/fe6 [0,4194304] 0 2024-04-20T21:32:26.405 INFO:tasks.workunit.client.1.smithi189.stdout:6/964: symlink d8/da/dd/d118/d99/l13e 0 2024-04-20T21:32:26.408 INFO:tasks.workunit.client.1.smithi189.stdout:2/898: dread d1/fea [0,4194304] 0 2024-04-20T21:32:26.412 INFO:tasks.workunit.client.1.smithi189.stdout:6/965: chown d8/da/dd/d2a/d3b/l103 34567 1 2024-04-20T21:32:26.412 INFO:tasks.workunit.client.1.smithi189.stdout:5/979: read - d7/d9/d28/da3/ff3 zero size 2024-04-20T21:32:26.417 INFO:tasks.workunit.client.1.smithi189.stdout:5/980: symlink d7/d9/d31/d100/d4e/d5b/df1/l13a 0 2024-04-20T21:32:26.422 INFO:tasks.workunit.client.1.smithi189.stdout:3/963: dwrite d2/d3/d2c/fda [0,4194304] 0 2024-04-20T21:32:26.423 INFO:tasks.workunit.client.1.smithi189.stdout:6/966: dread d8/d10/d78/df7/fd5 [0,4194304] 0 2024-04-20T21:32:26.425 INFO:tasks.workunit.client.1.smithi189.stdout:5/981: mknod d7/d9/d31/d100/d4e/d5b/df1/dff/c13b 0 2024-04-20T21:32:26.426 INFO:tasks.workunit.client.1.smithi189.stdout:2/899: truncate d1/d11/d99/db7/fb8 2973219 0 2024-04-20T21:32:26.427 INFO:tasks.workunit.client.1.smithi189.stdout:1/870: dread d8/d1e/d28/d2b/f48 [0,4194304] 0 2024-04-20T21:32:26.429 INFO:tasks.workunit.client.1.smithi189.stdout:3/964: creat d2/d1c/dc1/d109/f150 x:0 0 0 2024-04-20T21:32:26.429 INFO:tasks.workunit.client.1.smithi189.stdout:2/900: chown d1/l21 244288 1 2024-04-20T21:32:26.435 INFO:tasks.workunit.client.1.smithi189.stdout:2/901: dread d1/d11/d58/f6e [4194304,4194304] 0 2024-04-20T21:32:26.437 INFO:tasks.workunit.client.1.smithi189.stdout:1/871: truncate d8/df/d3a/d49/d8c/da2/dfe/f5f 775483 0 2024-04-20T21:32:26.439 INFO:tasks.workunit.client.1.smithi189.stdout:1/872: fsync d8/d21/d2c/f75 0 2024-04-20T21:32:26.441 INFO:tasks.workunit.client.1.smithi189.stdout:6/967: dread d8/d10/fc6 [0,4194304] 0 2024-04-20T21:32:26.442 INFO:tasks.workunit.client.1.smithi189.stdout:3/965: write d2/d3/d25/f95 [4596188,27790] 0 2024-04-20T21:32:26.444 INFO:tasks.workunit.client.1.smithi189.stdout:2/902: unlink d1/d11/d2d/dab/d74/c90 0 2024-04-20T21:32:26.451 INFO:tasks.workunit.client.1.smithi189.stdout:6/968: rmdir d8/d2b/d49/d108 39 2024-04-20T21:32:26.455 INFO:tasks.workunit.client.1.smithi189.stdout:3/966: rmdir d2/d3/dfd 39 2024-04-20T21:32:26.455 INFO:tasks.workunit.client.1.smithi189.stdout:1/873: mknod d8/df/d3a/d49/d8c/d103/c11c 0 2024-04-20T21:32:26.457 INFO:tasks.workunit.client.1.smithi189.stdout:5/982: dwrite d7/d136/d33/fe6 [0,4194304] 0 2024-04-20T21:32:26.460 INFO:tasks.workunit.client.1.smithi189.stdout:5/983: fdatasync d7/d136/d104/d106/f114 0 2024-04-20T21:32:26.463 INFO:tasks.workunit.client.1.smithi189.stdout:3/967: fdatasync d2/d8/f10 0 2024-04-20T21:32:26.466 INFO:tasks.workunit.client.1.smithi189.stdout:6/969: mkdir d8/da/d13f 0 2024-04-20T21:32:26.468 INFO:tasks.workunit.client.1.smithi189.stdout:5/984: mkdir d7/d9/d31/d100/d18/d13c 0 2024-04-20T21:32:26.470 INFO:tasks.workunit.client.1.smithi189.stdout:1/874: getdents d8/d1e/d28/dd4/ddb 0 2024-04-20T21:32:26.472 INFO:tasks.workunit.client.1.smithi189.stdout:2/903: dwrite d1/d11/d2d/f38 [0,4194304] 0 2024-04-20T21:32:26.473 INFO:tasks.workunit.client.1.smithi189.stdout:6/970: symlink d8/d10/d78/d7e/dcf/l140 0 2024-04-20T21:32:26.474 INFO:tasks.workunit.client.1.smithi189.stdout:5/985: mkdir d7/d136/d61/d87/d13d 0 2024-04-20T21:32:26.476 INFO:tasks.workunit.client.1.smithi189.stdout:5/986: readlink d7/d136/d33/l137 0 2024-04-20T21:32:26.478 INFO:tasks.workunit.client.1.smithi189.stdout:6/971: creat d8/d2b/d49/d54/d6e/f141 x:0 0 0 2024-04-20T21:32:26.480 INFO:tasks.workunit.client.1.smithi189.stdout:6/972: chown d8/d2b/d49/d54/d67/fb8 125816 1 2024-04-20T21:32:26.483 INFO:tasks.workunit.client.1.smithi189.stdout:1/875: unlink d8/d1e/d43/d44/dd2/db7/dcf/fff 0 2024-04-20T21:32:26.484 INFO:tasks.workunit.client.1.smithi189.stdout:2/904: symlink d1/d11/d2d/d4f/db4/l104 0 2024-04-20T21:32:26.486 INFO:tasks.workunit.client.1.smithi189.stdout:5/987: symlink d7/d9/d51/d70/d95/l13e 0 2024-04-20T21:32:26.487 INFO:tasks.workunit.client.1.smithi189.stdout:2/905: write d1/d17/d78/fac [6954191,32898] 0 2024-04-20T21:32:26.491 INFO:tasks.workunit.client.1.smithi189.stdout:6/973: write d8/da/dd/d118/d4e/dc4/f129 [23805,94799] 0 2024-04-20T21:32:26.495 INFO:tasks.workunit.client.1.smithi189.stdout:5/988: fsync d7/d136/d81/f129 0 2024-04-20T21:32:26.500 INFO:tasks.workunit.client.1.smithi189.stdout:4/884: sync 2024-04-20T21:32:26.505 INFO:tasks.workunit.client.1.smithi189.stdout:5/989: rmdir d7/d9/d51 39 2024-04-20T21:32:26.508 INFO:tasks.workunit.client.1.smithi189.stdout:2/906: symlink d1/d11/d52/l105 0 2024-04-20T21:32:26.511 INFO:tasks.workunit.client.1.smithi189.stdout:4/885: creat d2/dd/d11/dae/f13a x:0 0 0 2024-04-20T21:32:26.514 INFO:tasks.workunit.client.1.smithi189.stdout:4/886: write d2/d20/d115/dcd/d6a/f105 [422690,38578] 0 2024-04-20T21:32:26.514 INFO:tasks.workunit.client.1.smithi189.stdout:3/968: write d2/d3/d25/f97 [4430175,39390] 0 2024-04-20T21:32:26.515 INFO:tasks.workunit.client.1.smithi189.stdout:2/907: mknod d1/d11/d2d/d4f/db4/daa/c106 0 2024-04-20T21:32:26.520 INFO:tasks.workunit.client.1.smithi189.stdout:5/990: mknod d7/d9/c13f 0 2024-04-20T21:32:26.523 INFO:tasks.workunit.client.1.smithi189.stdout:2/908: symlink d1/d11/d2d/dab/d4a/d66/df1/l107 0 2024-04-20T21:32:26.524 INFO:tasks.workunit.client.1.smithi189.stdout:4/887: mkdir d2/d13b 0 2024-04-20T21:32:26.525 INFO:tasks.workunit.client.1.smithi189.stdout:5/991: truncate d7/d9/d31/d100/f110 225770 0 2024-04-20T21:32:26.526 INFO:tasks.workunit.client.1.smithi189.stdout:6/974: write d8/da/dd/d118/d99/fca [171840,124912] 0 2024-04-20T21:32:26.526 INFO:tasks.workunit.client.1.smithi189.stdout:1/876: write d8/d1e/d28/f90 [99422,46222] 0 2024-04-20T21:32:26.527 INFO:tasks.workunit.client.1.smithi189.stdout:2/909: dread d1/f12 [0,4194304] 0 2024-04-20T21:32:26.531 INFO:tasks.workunit.client.1.smithi189.stdout:5/992: getdents d7/d128 0 2024-04-20T21:32:26.532 INFO:tasks.workunit.client.1.smithi189.stdout:1/877: write d8/df/d3a/dfd/f110 [591680,86864] 0 2024-04-20T21:32:26.533 INFO:tasks.workunit.client.1.smithi189.stdout:2/910: dread - d1/d17/d37/d89/dc5/fdf zero size 2024-04-20T21:32:26.535 INFO:tasks.workunit.client.1.smithi189.stdout:6/975: getdents d8/da/dd/d118/d99/dd8 0 2024-04-20T21:32:26.536 INFO:tasks.workunit.client.1.smithi189.stdout:3/969: dwrite d2/d1c/dc1/d109/f150 [0,4194304] 0 2024-04-20T21:32:26.538 INFO:tasks.workunit.client.1.smithi189.stdout:3/970: chown d2/d3/d25/d108 348993 1 2024-04-20T21:32:26.538 INFO:tasks.workunit.client.1.smithi189.stdout:2/911: rmdir d1/d11/d2d/d4f/d8e 39 2024-04-20T21:32:26.541 INFO:tasks.workunit.client.1.smithi189.stdout:3/971: write d2/d1c/dc1/dc5/ddf/f11f [156433,4521] 0 2024-04-20T21:32:26.551 INFO:tasks.workunit.client.1.smithi189.stdout:5/993: dwrite d7/d136/d61/d96/f10e [0,4194304] 0 2024-04-20T21:32:26.556 INFO:tasks.workunit.client.1.smithi189.stdout:5/994: mkdir d7/d136/d104/d106/d140 0 2024-04-20T21:32:26.563 INFO:tasks.workunit.client.1.smithi189.stdout:3/972: write d2/d3/d83/f121 [1100369,62818] 0 2024-04-20T21:32:26.569 INFO:tasks.workunit.client.1.smithi189.stdout:1/878: dwrite d8/d21/d2c/f6c [0,4194304] 0 2024-04-20T21:32:26.571 INFO:tasks.workunit.client.1.smithi189.stdout:2/912: dread d1/d11/d2d/d4f/fbc [0,4194304] 0 2024-04-20T21:32:26.575 INFO:tasks.workunit.client.1.smithi189.stdout:2/913: chown d1/d17/d37/d89/dc5/fdf 262941683 1 2024-04-20T21:32:26.575 INFO:tasks.workunit.client.1.smithi189.stdout:3/973: creat d2/d3/d25/de0/da9/f151 x:0 0 0 2024-04-20T21:32:26.578 INFO:tasks.workunit.client.1.smithi189.stdout:5/995: dwrite d7/d136/d33/fe6 [0,4194304] 0 2024-04-20T21:32:26.579 INFO:tasks.workunit.client.1.smithi189.stdout:1/879: unlink d8/d1e/f87 0 2024-04-20T21:32:26.583 INFO:tasks.workunit.client.1.smithi189.stdout:3/974: mkdir d2/d3/d2c/d31/d4c/d152 0 2024-04-20T21:32:26.586 INFO:tasks.workunit.client.1.smithi189.stdout:4/888: dread d2/d22/f6e [0,4194304] 0 2024-04-20T21:32:26.593 INFO:tasks.workunit.client.1.smithi189.stdout:1/880: rmdir d8/df/d3a/d49/d8c/da2/dbe/dc1/dd9 39 2024-04-20T21:32:26.596 INFO:tasks.workunit.client.1.smithi189.stdout:5/996: creat d7/d9/f141 x:0 0 0 2024-04-20T21:32:26.597 INFO:tasks.workunit.client.1.smithi189.stdout:4/889: chown d2/d22/c62 214224332 1 2024-04-20T21:32:26.597 INFO:tasks.workunit.client.1.smithi189.stdout:3/975: read d2/d1c/f7b [3131773,90408] 0 2024-04-20T21:32:26.599 INFO:tasks.workunit.client.1.smithi189.stdout:3/976: readlink d2/d3/d25/de0/ld9 0 2024-04-20T21:32:26.600 INFO:tasks.workunit.client.1.smithi189.stdout:2/914: dwrite d1/d17/d37/f44 [0,4194304] 0 2024-04-20T21:32:26.606 INFO:tasks.workunit.client.1.smithi189.stdout:5/997: write d7/d9/d51/d70/d95/f111 [3019879,126934] 0 2024-04-20T21:32:26.610 INFO:tasks.workunit.client.1.smithi189.stdout:3/977: rename d2/d3/d2c/d30/d10c/d112/c14a to d2/d3/d2c/d31/d74/d7a/c153 0 2024-04-20T21:32:26.612 INFO:tasks.workunit.client.1.smithi189.stdout:2/915: creat d1/d17/f108 x:0 0 0 2024-04-20T21:32:26.618 INFO:tasks.workunit.client.1.smithi189.stdout:2/916: getdents d1/d17/d37/d89 0 2024-04-20T21:32:26.622 INFO:tasks.workunit.client.1.smithi189.stdout:5/998: dread d7/d9/d31/d100/f8b [0,4194304] 0 2024-04-20T21:32:26.624 INFO:tasks.workunit.client.1.smithi189.stdout:5/999: stat d7/d136/d104/d106 0 2024-04-20T21:32:26.629 INFO:tasks.workunit.client.1.smithi189.stdout:1/881: write d8/d1e/d43/d44/dd2/f82 [248991,123364] 0 2024-04-20T21:32:26.633 INFO:tasks.workunit.client.1.smithi189.stdout:4/890: write d2/d20/d115/dcd/fb3 [4505176,3762] 0 2024-04-20T21:32:26.638 INFO:tasks.workunit.client.1.smithi189.stdout:3/978: write d2/d8d/fba [1362457,76449] 0 2024-04-20T21:32:26.639 INFO:tasks.workunit.client.1.smithi189.stdout:2/917: dwrite d1/d11/d2d/dab/d4a/d66/df1/f100 [0,4194304] 0 2024-04-20T21:32:26.646 INFO:tasks.workunit.client.1.smithi189.stdout:3/979: chown d2/d3/l73 28 1 2024-04-20T21:32:26.651 INFO:tasks.workunit.client.1.smithi189.stdout:2/918: creat d1/d11/d2d/dab/d74/f109 x:0 0 0 2024-04-20T21:32:26.651 INFO:tasks.workunit.client.1.smithi189.stdout:2/919: fdatasync d1/d11/d52/f6a 0 2024-04-20T21:32:26.651 INFO:tasks.workunit.client.1.smithi189.stdout:1/882: write d8/d1e/d43/d44/dd2/f5b [354343,130509] 0 2024-04-20T21:32:26.651 INFO:tasks.workunit.client.1.smithi189.stdout:4/891: write d2/d22/f5d [466035,130142] 0 2024-04-20T21:32:26.654 INFO:tasks.workunit.client.1.smithi189.stdout:4/892: stat d2/dd/d11/d14/l16 0 2024-04-20T21:32:26.657 INFO:tasks.workunit.client.1.smithi189.stdout:2/920: creat d1/d11/d2d/dab/f10a x:0 0 0 2024-04-20T21:32:26.661 INFO:tasks.workunit.client.1.smithi189.stdout:2/921: mknod d1/d11/d52/d8b/de4/c10b 0 2024-04-20T21:32:26.664 INFO:tasks.workunit.client.1.smithi189.stdout:4/893: read d2/f5a [2715358,115933] 0 2024-04-20T21:32:26.668 INFO:tasks.workunit.client.1.smithi189.stdout:2/922: rename d1/d11/d2d/dab/d55 to d1/d11/d58/d6b/d10c 0 2024-04-20T21:32:26.672 INFO:tasks.workunit.client.1.smithi189.stdout:4/894: rename d2/d20/la5 to d2/d20/dc6/dda/l13c 0 2024-04-20T21:32:26.679 INFO:tasks.workunit.client.1.smithi189.stdout:4/895: creat d2/d20/d115/dcd/d6a/d7a/def/f13d x:0 0 0 2024-04-20T21:32:26.680 INFO:tasks.workunit.client.1.smithi189.stdout:1/883: dwrite d8/d1e/d43/d58/d60/f10f [0,4194304] 0 2024-04-20T21:32:26.682 INFO:tasks.workunit.client.1.smithi189.stdout:4/896: fsync d2/dd/d6b/fc7 0 2024-04-20T21:32:26.685 INFO:tasks.workunit.client.1.smithi189.stdout:1/884: creat d8/d21/f11d x:0 0 0 2024-04-20T21:32:26.688 INFO:tasks.workunit.client.1.smithi189.stdout:4/897: symlink d2/d20/d115/l13e 0 2024-04-20T21:32:26.689 INFO:tasks.workunit.client.1.smithi189.stdout:1/885: stat d8/df/d3a/d49/d8c/da2/f10c 0 2024-04-20T21:32:26.691 INFO:tasks.workunit.client.1.smithi189.stdout:2/923: dread d1/d17/f2b [0,4194304] 0 2024-04-20T21:32:26.693 INFO:tasks.workunit.client.1.smithi189.stdout:1/886: read d8/df/d3a/f78 [97174,126952] 0 2024-04-20T21:32:26.699 INFO:tasks.workunit.client.1.smithi189.stdout:2/924: link d1/d11/d2d/dab/d4a/d66/dc9/ff7 d1/d11/d58/f10d 0 2024-04-20T21:32:26.702 INFO:tasks.workunit.client.1.smithi189.stdout:2/925: write d1/d11/d2d/d4f/db4/f9c [1145036,89045] 0 2024-04-20T21:32:26.706 INFO:tasks.workunit.client.1.smithi189.stdout:2/926: link d1/d11/d2d/dab/f10a d1/d17/d78/db9/f10e 0 2024-04-20T21:32:26.708 INFO:tasks.workunit.client.1.smithi189.stdout:2/927: fdatasync d1/d11/d2d/f38 0 2024-04-20T21:32:26.714 INFO:tasks.workunit.client.1.smithi189.stdout:2/928: getdents d1/d11/d2d/dab/d4a 0 2024-04-20T21:32:26.719 INFO:tasks.workunit.client.1.smithi189.stdout:2/929: write d1/d11/d2d/d4f/db4/f53 [2704645,42779] 0 2024-04-20T21:32:26.722 INFO:tasks.workunit.client.1.smithi189.stdout:2/930: creat d1/d17/d37/d89/f10f x:0 0 0 2024-04-20T21:32:26.725 INFO:tasks.workunit.client.1.smithi189.stdout:2/931: creat d1/d11/de1/f110 x:0 0 0 2024-04-20T21:32:26.735 INFO:tasks.workunit.client.1.smithi189.stdout:2/932: dread d1/d17/f2b [0,4194304] 0 2024-04-20T21:32:26.739 INFO:tasks.workunit.client.1.smithi189.stdout:2/933: creat d1/d11/d2d/dab/f111 x:0 0 0 2024-04-20T21:32:26.740 INFO:tasks.workunit.client.1.smithi189.stdout:9/962: sync 2024-04-20T21:32:26.740 INFO:tasks.workunit.client.1.smithi189.stdout:2/934: readlink d1/d11/d2d/dab/lae 0 2024-04-20T21:32:26.741 INFO:tasks.workunit.client.1.smithi189.stdout:9/963: readlink d9/dc/d69/d125/l88 0 2024-04-20T21:32:26.742 INFO:tasks.workunit.client.1.smithi189.stdout:2/935: stat d1/d17/f34 0 2024-04-20T21:32:26.744 INFO:tasks.workunit.client.1.smithi189.stdout:9/964: unlink d9/dc/dab/c130 0 2024-04-20T21:32:26.745 INFO:tasks.workunit.client.1.smithi189.stdout:9/965: chown d9/d12/d3b/db1/d1a/f64 247 1 2024-04-20T21:32:26.746 INFO:tasks.workunit.client.1.smithi189.stdout:2/936: rename d1/d11/d58/f6e to d1/d11/d2d/f112 0 2024-04-20T21:32:26.750 INFO:tasks.workunit.client.1.smithi189.stdout:2/937: fdatasync d1/d11/d2d/d4f/d8e/fb5 0 2024-04-20T21:32:26.754 INFO:tasks.workunit.client.1.smithi189.stdout:2/938: mkdir d1/d11/d2d/d4f/d113 0 2024-04-20T21:32:26.758 INFO:tasks.workunit.client.1.smithi189.stdout:2/939: symlink d1/l114 0 2024-04-20T21:32:26.760 INFO:tasks.workunit.client.1.smithi189.stdout:2/940: chown d1/c45 59 1 2024-04-20T21:32:26.760 INFO:tasks.workunit.client.1.smithi189.stdout:9/966: dwrite d9/dc/f1c [0,4194304] 0 2024-04-20T21:32:26.763 INFO:tasks.workunit.client.1.smithi189.stdout:9/967: chown d9/dc/d69/d125/d95/ff7 6 1 2024-04-20T21:32:26.768 INFO:tasks.workunit.client.1.smithi189.stdout:2/941: link d1/d11/d2d/dab/d4a/f84 d1/f115 0 2024-04-20T21:32:26.772 INFO:tasks.workunit.client.1.smithi189.stdout:2/942: mkdir d1/d11/d116 0 2024-04-20T21:32:26.774 INFO:tasks.workunit.client.1.smithi189.stdout:2/943: chown d1/d11/d58/d6b/d10c/f61 2 1 2024-04-20T21:32:26.776 INFO:tasks.workunit.client.1.smithi189.stdout:2/944: chown d1/l3c 379 1 2024-04-20T21:32:26.778 INFO:tasks.workunit.client.1.smithi189.stdout:9/968: dwrite d9/dc/d69/d125/d36/f78 [0,4194304] 0 2024-04-20T21:32:26.779 INFO:tasks.workunit.client.1.smithi189.stdout:9/969: readlink d9/d12/l2d 0 2024-04-20T21:32:26.782 INFO:tasks.workunit.client.1.smithi189.stdout:9/970: chown d9/d12/d62/ff1 109 1 2024-04-20T21:32:26.786 INFO:tasks.workunit.client.1.smithi189.stdout:2/945: rename d1/d17/d37/c47 to d1/d11/de1/c117 0 2024-04-20T21:32:26.790 INFO:tasks.workunit.client.1.smithi189.stdout:9/971: link d9/dc/d69/d125/d95/da0/d52/fd3 d9/d12/d62/f143 0 2024-04-20T21:32:26.791 INFO:tasks.workunit.client.1.smithi189.stdout:2/946: readlink d1/d11/l2c 0 2024-04-20T21:32:26.796 INFO:tasks.workunit.client.1.smithi189.stdout:2/947: mkdir d1/d11/d2d/dab/d118 0 2024-04-20T21:32:26.805 INFO:tasks.workunit.client.1.smithi189.stdout:9/972: truncate d9/dc/f2c 2831994 0 2024-04-20T21:32:26.809 INFO:tasks.workunit.client.1.smithi189.stdout:9/973: creat d9/dc/d69/d125/f144 x:0 0 0 2024-04-20T21:32:26.816 INFO:tasks.workunit.client.1.smithi189.stdout:9/974: rmdir d9/dc/dfc 0 2024-04-20T21:32:26.821 INFO:tasks.workunit.client.1.smithi189.stdout:2/948: dread d1/d11/d52/fbd [0,4194304] 0 2024-04-20T21:32:26.826 INFO:tasks.workunit.client.1.smithi189.stdout:2/949: write d1/d11/d52/d8b/fa1 [3544852,87294] 0 2024-04-20T21:32:26.831 INFO:tasks.workunit.client.1.smithi189.stdout:2/950: creat d1/d11/d52/f119 x:0 0 0 2024-04-20T21:32:26.833 INFO:tasks.workunit.client.1.smithi189.stdout:2/951: write d1/d11/d58/f64 [2183735,102121] 0 2024-04-20T21:32:26.841 INFO:tasks.workunit.client.1.smithi189.stdout:2/952: truncate d1/d11/d2d/dab/d4a/f8f 2868775 0 2024-04-20T21:32:26.843 INFO:tasks.workunit.client.1.smithi189.stdout:9/975: dwrite d9/dc/d69/d125/d95/da0/d52/d82/fa7 [0,4194304] 0 2024-04-20T21:32:26.848 INFO:tasks.workunit.client.1.smithi189.stdout:9/976: mkdir d9/dc/d69/d125/d95/da0/d52/d82/da4/d13b/d145 0 2024-04-20T21:32:26.849 INFO:tasks.workunit.client.1.smithi189.stdout:6/976: sync 2024-04-20T21:32:26.849 INFO:tasks.workunit.client.1.smithi189.stdout:3/980: sync 2024-04-20T21:32:26.852 INFO:tasks.workunit.client.1.smithi189.stdout:9/977: rmdir d9/d12/d3f 39 2024-04-20T21:32:26.855 INFO:tasks.workunit.client.1.smithi189.stdout:3/981: fsync d2/d1c/d5c/fbd 0 2024-04-20T21:32:26.858 INFO:tasks.workunit.client.1.smithi189.stdout:9/978: stat d9/dc/d69/d125/d7e/d13e 0 2024-04-20T21:32:26.861 INFO:tasks.workunit.client.1.smithi189.stdout:2/953: dwrite d1/d11/f2e [0,4194304] 0 2024-04-20T21:32:26.863 INFO:tasks.workunit.client.1.smithi189.stdout:3/982: creat d2/d3/d2c/d30/f154 x:0 0 0 2024-04-20T21:32:26.863 INFO:tasks.workunit.client.1.smithi189.stdout:9/979: truncate d9/d12/d3b/db1/f26 3986941 0 2024-04-20T21:32:26.867 INFO:tasks.workunit.client.1.smithi189.stdout:6/977: write d8/da/f122 [1755998,3496] 0 2024-04-20T21:32:26.868 INFO:tasks.workunit.client.1.smithi189.stdout:9/980: write d9/dc/d69/d125/d95/da6/fe1 [678884,121266] 0 2024-04-20T21:32:26.882 INFO:tasks.workunit.client.1.smithi189.stdout:2/954: dread d1/d17/f34 [4194304,4194304] 0 2024-04-20T21:32:26.882 INFO:tasks.workunit.client.1.smithi189.stdout:3/983: dread d2/d1c/dc1/d109/f150 [0,4194304] 0 2024-04-20T21:32:26.882 INFO:tasks.workunit.client.1.smithi189.stdout:6/978: link d8/da/dd/d118/d4e/fa6 d8/da/dd/d118/d99/dd8/f142 0 2024-04-20T21:32:26.882 INFO:tasks.workunit.client.1.smithi189.stdout:3/984: dread - d2/d1c/d5c/f13a zero size 2024-04-20T21:32:26.889 INFO:tasks.workunit.client.1.smithi189.stdout:3/985: link d2/d3/d25/c8f d2/d8/c155 0 2024-04-20T21:32:26.890 INFO:tasks.workunit.client.1.smithi189.stdout:6/979: write d8/da/da7/f12f [1539695,53307] 0 2024-04-20T21:32:26.891 INFO:tasks.workunit.client.1.smithi189.stdout:2/955: write d1/d11/d58/d6b/d10c/fc1 [397320,76885] 0 2024-04-20T21:32:26.892 INFO:tasks.workunit.client.1.smithi189.stdout:3/986: creat d2/d3/d25/de0/da9/f156 x:0 0 0 2024-04-20T21:32:26.895 INFO:tasks.workunit.client.1.smithi189.stdout:6/980: rmdir d8/da/dd/d118/d99/dd8 39 2024-04-20T21:32:26.898 INFO:tasks.workunit.client.1.smithi189.stdout:6/981: write d8/da/dd/f92 [717852,47355] 0 2024-04-20T21:32:26.900 INFO:tasks.workunit.client.1.smithi189.stdout:3/987: symlink d2/d3/d25/de0/da9/d12a/l157 0 2024-04-20T21:32:26.900 INFO:tasks.workunit.client.1.smithi189.stdout:4/898: sync 2024-04-20T21:32:26.901 INFO:tasks.workunit.client.1.smithi189.stdout:1/887: sync 2024-04-20T21:32:26.905 INFO:tasks.workunit.client.1.smithi189.stdout:4/899: symlink d2/dd/d11/d14/l13f 0 2024-04-20T21:32:26.907 INFO:tasks.workunit.client.1.smithi189.stdout:9/981: dread d9/d12/d3b/db1/f26 [0,4194304] 0 2024-04-20T21:32:26.909 INFO:tasks.workunit.client.1.smithi189.stdout:6/982: link d8/d10/d78/df7/fd5 d8/da/dd/d118/d30/d47/f143 0 2024-04-20T21:32:26.913 INFO:tasks.workunit.client.1.smithi189.stdout:3/988: link d2/d3/d36/da1/l115 d2/d3/l158 0 2024-04-20T21:32:26.915 INFO:tasks.workunit.client.1.smithi189.stdout:2/956: dwrite d1/d11/d2d/dab/d74/f109 [0,4194304] 0 2024-04-20T21:32:26.916 INFO:tasks.workunit.client.1.smithi189.stdout:9/982: mknod d9/dc/d69/d125/d95/da0/d55/dd0/c146 0 2024-04-20T21:32:26.920 INFO:tasks.workunit.client.1.smithi189.stdout:3/989: write d2/d3/d25/f5b [4947168,73744] 0 2024-04-20T21:32:26.924 INFO:tasks.workunit.client.1.smithi189.stdout:2/957: mkdir d1/d11/d2d/d4f/d11a 0 2024-04-20T21:32:26.928 INFO:tasks.workunit.client.1.smithi189.stdout:1/888: dwrite d8/df/d3a/f116 [0,4194304] 0 2024-04-20T21:32:26.930 INFO:tasks.workunit.client.1.smithi189.stdout:3/990: creat d2/d1c/f159 x:0 0 0 2024-04-20T21:32:26.930 INFO:tasks.workunit.client.1.smithi189.stdout:2/958: dread - d1/d17/d78/db9/f10e zero size 2024-04-20T21:32:26.935 INFO:tasks.workunit.client.1.smithi189.stdout:9/983: rename d9/d12/d3b/db1/d1a/f27 to d9/dc/d69/d125/d95/da0/d52/d82/da4/d13b/d145/f147 0 2024-04-20T21:32:26.937 INFO:tasks.workunit.client.1.smithi189.stdout:3/991: dread - d2/d3/d2c/d30/f145 zero size 2024-04-20T21:32:26.941 INFO:tasks.workunit.client.1.smithi189.stdout:4/900: dwrite d2/d20/d115/dcd/d6a/d7a/db7/f136 [0,4194304] 0 2024-04-20T21:32:26.946 INFO:tasks.workunit.client.1.smithi189.stdout:6/983: dwrite d8/da/dd/f88 [0,4194304] 0 2024-04-20T21:32:26.954 INFO:tasks.workunit.client.1.smithi189.stdout:3/992: mkdir d2/d1c/dc1/dc5/d15a 0 2024-04-20T21:32:26.955 INFO:tasks.workunit.client.1.smithi189.stdout:4/901: mknod d2/dd/d11/dae/d10e/c140 0 2024-04-20T21:32:26.965 INFO:tasks.workunit.client.1.smithi189.stdout:6/984: truncate d8/da/dd/d118/d99/fb7 3046767 0 2024-04-20T21:32:26.966 INFO:tasks.workunit.client.1.smithi189.stdout:9/984: write d9/dc/d69/d125/d36/d5d/f97 [1170635,106021] 0 2024-04-20T21:32:26.970 INFO:tasks.workunit.client.1.smithi189.stdout:2/959: dwrite d1/d11/d58/d6b/fee [0,4194304] 0 2024-04-20T21:32:26.971 INFO:tasks.workunit.client.1.smithi189.stdout:3/993: read d2/d3/d2c/d31/f6f [212407,90950] 0 2024-04-20T21:32:26.973 INFO:tasks.workunit.client.1.smithi189.stdout:9/985: readlink d9/d12/d3f/d72/d11b/l12c 0 2024-04-20T21:32:26.974 INFO:tasks.workunit.client.1.smithi189.stdout:6/985: mkdir d8/da/dd/d118/d30/d47/ddc/d144 0 2024-04-20T21:32:26.975 INFO:tasks.workunit.client.1.smithi189.stdout:4/902: link d2/d22/d87/f102 d2/d20/d115/dcd/d6a/d7a/def/f141 0 2024-04-20T21:32:26.976 INFO:tasks.workunit.client.1.smithi189.stdout:2/960: unlink d1/d11/d2d/d4f/db4/l104 0 2024-04-20T21:32:26.978 INFO:tasks.workunit.client.1.smithi189.stdout:2/961: chown d1/d11/de1/df0 2 1 2024-04-20T21:32:26.979 INFO:tasks.workunit.client.1.smithi189.stdout:3/994: mkdir d2/d3/d2c/d15b 0 2024-04-20T21:32:26.985 INFO:tasks.workunit.client.1.smithi189.stdout:2/962: creat d1/d11/d2d/d4f/db4/f11b x:0 0 0 2024-04-20T21:32:26.988 INFO:tasks.workunit.client.1.smithi189.stdout:9/986: creat d9/dc/d69/d125/d36/f148 x:0 0 0 2024-04-20T21:32:26.989 INFO:tasks.workunit.client.1.smithi189.stdout:6/986: getdents d8/d10/d78 0 2024-04-20T21:32:27.002 INFO:tasks.workunit.client.1.smithi189.stdout:4/903: dwrite d2/d22/f6e [4194304,4194304] 0 2024-04-20T21:32:27.005 INFO:tasks.workunit.client.1.smithi189.stdout:2/963: truncate d1/d11/d58/f64 603549 0 2024-04-20T21:32:27.008 INFO:tasks.workunit.client.1.smithi189.stdout:1/889: dread d8/d1e/d28/fa8 [0,4194304] 0 2024-04-20T21:32:27.016 INFO:tasks.workunit.client.1.smithi189.stdout:2/964: mkdir d1/d11/d2d/d4f/db4/daa/dd2/d11c 0 2024-04-20T21:32:27.017 INFO:tasks.workunit.client.1.smithi189.stdout:4/904: truncate d2/d22/da9/fbf 960891 0 2024-04-20T21:32:27.019 INFO:tasks.workunit.client.1.smithi189.stdout:3/995: dwrite d2/d3/d2c/d31/d74/db7/fcf [0,4194304] 0 2024-04-20T21:32:27.019 INFO:tasks.workunit.client.1.smithi189.stdout:1/890: symlink d8/df/d3a/d49/l11e 0 2024-04-20T21:32:27.039 INFO:tasks.workunit.client.1.smithi189.stdout:3/996: read d2/d3/d25/f2e [2362877,71836] 0 2024-04-20T21:32:27.041 INFO:tasks.workunit.client.1.smithi189.stdout:9/987: dread d9/dc/f4d [0,4194304] 0 2024-04-20T21:32:27.041 INFO:tasks.workunit.client.1.smithi189.stdout:6/987: dread d8/da/dd/d118/daf/fba [4194304,4194304] 0 2024-04-20T21:32:27.055 INFO:tasks.workunit.client.1.smithi189.stdout:1/891: unlink d8/d1e/d43/d58/d60/f115 0 2024-04-20T21:32:27.056 INFO:tasks.workunit.client.1.smithi189.stdout:3/997: symlink d2/d3/d2c/d30/d10c/d112/d148/l15c 0 2024-04-20T21:32:27.061 INFO:tasks.workunit.client.1.smithi189.stdout:6/988: creat d8/da/f145 x:0 0 0 2024-04-20T21:32:27.062 INFO:tasks.workunit.client.1.smithi189.stdout:9/988: link d9/d12/d62/ca9 d9/dc/d69/d125/d95/da0/d52/d82/da4/df3/c149 0 2024-04-20T21:32:27.063 INFO:tasks.workunit.client.1.smithi189.stdout:4/905: dread d2/d20/d115/dcd/fb3 [0,4194304] 0 2024-04-20T21:32:27.066 INFO:tasks.workunit.client.1.smithi189.stdout:6/989: mknod d8/c146 0 2024-04-20T21:32:27.071 INFO:tasks.workunit.client.1.smithi189.stdout:3/998: getdents d2/d3/d25/de0/db0/dc8/dc9 0 2024-04-20T21:32:27.074 INFO:tasks.workunit.client.1.smithi189.stdout:3/999: chown d2/d8/f22 459133 1 2024-04-20T21:32:27.075 INFO:tasks.workunit.client.1.smithi189.stdout:1/892: dwrite d8/d1e/d54/d93/fbd [0,4194304] 0 2024-04-20T21:32:27.076 INFO:tasks.workunit.client.1.smithi189.stdout:6/990: getdents d8/d2b/d49/d108/d126 0 2024-04-20T21:32:27.076 INFO:tasks.workunit.client.1.smithi189.stdout:9/989: link d9/d12/d3b/db1/d1a/l9b d9/dc/d69/d125/d95/da0/d55/l14a 0 2024-04-20T21:32:27.078 INFO:tasks.workunit.client.1.smithi189.stdout:1/893: rename d8 to d8/df/d3a/d49/d8c/d103/d11f 22 2024-04-20T21:32:27.082 INFO:tasks.workunit.client.1.smithi189.stdout:6/991: rename d8/da/dd/d118/daf/ffb to d8/d2b/d49/d54/d6e/f147 0 2024-04-20T21:32:27.085 INFO:tasks.workunit.client.1.smithi189.stdout:1/894: creat d8/d1e/d43/d44/f120 x:0 0 0 2024-04-20T21:32:27.088 INFO:tasks.workunit.client.1.smithi189.stdout:6/992: mknod d8/da/dd/d118/d99/c148 0 2024-04-20T21:32:27.094 INFO:tasks.workunit.client.1.smithi189.stdout:2/965: truncate d1/d11/d2d/d4f/db4/f53 3787971 0 2024-04-20T21:32:27.101 INFO:tasks.workunit.client.1.smithi189.stdout:4/906: write d2/fb [4798215,123094] 0 2024-04-20T21:32:27.103 INFO:tasks.workunit.client.1.smithi189.stdout:1/895: dread d8/d1e/d43/d58/d60/f64 [0,4194304] 0 2024-04-20T21:32:27.106 INFO:tasks.workunit.client.1.smithi189.stdout:4/907: readlink d2/dd/d11/d14/l13f 0 2024-04-20T21:32:27.107 INFO:tasks.workunit.client.1.smithi189.stdout:6/993: dread d8/d10/d78/d7e/f84 [0,4194304] 0 2024-04-20T21:32:27.111 INFO:tasks.workunit.client.1.smithi189.stdout:2/966: dread d1/d11/d58/d6b/fee [0,4194304] 0 2024-04-20T21:32:27.114 INFO:tasks.workunit.client.1.smithi189.stdout:9/990: dwrite d9/dc/d69/f76 [0,4194304] 0 2024-04-20T21:32:27.117 INFO:tasks.workunit.client.1.smithi189.stdout:1/896: getdents d8/df/d3a/d49/d8c/da2/dbe/dd5 0 2024-04-20T21:32:27.119 INFO:tasks.workunit.client.1.smithi189.stdout:2/967: mkdir d1/d11/d2d/d4f/db4/daa/dd2/d11c/d11d 0 2024-04-20T21:32:27.119 INFO:tasks.workunit.client.1.smithi189.stdout:9/991: symlink d9/d12/d3f/l14b 0 2024-04-20T21:32:27.121 INFO:tasks.workunit.client.1.smithi189.stdout:1/897: write d8/f14 [3298158,55379] 0 2024-04-20T21:32:27.123 INFO:tasks.workunit.client.1.smithi189.stdout:1/898: stat d8/df/d3a/dfd/f114 0 2024-04-20T21:32:27.126 INFO:tasks.workunit.client.1.smithi189.stdout:2/968: write d1/d11/f9d [97738,59882] 0 2024-04-20T21:32:27.127 INFO:tasks.workunit.client.1.smithi189.stdout:1/899: dread - d8/d9b/fe0 zero size 2024-04-20T21:32:27.129 INFO:tasks.workunit.client.1.smithi189.stdout:9/992: creat d9/dc/d69/d125/d95/da0/d52/d82/da4/df3/f14c x:0 0 0 2024-04-20T21:32:27.133 INFO:tasks.workunit.client.1.smithi189.stdout:2/969: mknod d1/d11/d58/d6b/d10c/c11e 0 2024-04-20T21:32:27.134 INFO:tasks.workunit.client.1.smithi189.stdout:4/908: dwrite d2/d22/f30 [0,4194304] 0 2024-04-20T21:32:27.140 INFO:tasks.workunit.client.1.smithi189.stdout:6/994: dwrite d8/da/dd/d118/d4e/dc4/f114 [0,4194304] 0 2024-04-20T21:32:27.141 INFO:tasks.workunit.client.1.smithi189.stdout:1/900: write d8/d1e/d43/d44/fc9 [1920375,112255] 0 2024-04-20T21:32:27.143 INFO:tasks.workunit.client.1.smithi189.stdout:4/909: mknod d2/c142 0 2024-04-20T21:32:27.150 INFO:tasks.workunit.client.1.smithi189.stdout:6/995: mkdir d8/da/dd/d118/d99/dd8/d149 0 2024-04-20T21:32:27.157 INFO:tasks.workunit.client.1.smithi189.stdout:6/996: creat d8/d2b/d49/f14a x:0 0 0 2024-04-20T21:32:27.162 INFO:tasks.workunit.client.1.smithi189.stdout:6/997: stat d8/da/l42 0 2024-04-20T21:32:27.168 INFO:tasks.workunit.client.1.smithi189.stdout:4/910: dwrite d2/d22/f12d [0,4194304] 0 2024-04-20T21:32:27.175 INFO:tasks.workunit.client.1.smithi189.stdout:1/901: dwrite d8/df/d3a/d49/d8c/da2/dfe/f80 [0,4194304] 0 2024-04-20T21:32:27.177 INFO:tasks.workunit.client.1.smithi189.stdout:6/998: write d8/d2b/d49/f60 [1846667,101807] 0 2024-04-20T21:32:27.190 INFO:tasks.workunit.client.1.smithi189.stdout:2/970: dread d1/d11/d2d/d4f/d8e/fe2 [0,4194304] 0 2024-04-20T21:32:27.190 INFO:tasks.workunit.client.1.smithi189.stdout:6/999: rmdir d8/d2b/d49/d108/d126 0 2024-04-20T21:32:27.197 INFO:tasks.workunit.client.1.smithi189.stdout:4/911: dwrite d2/d3a/f4d [4194304,4194304] 0 2024-04-20T21:32:27.201 INFO:tasks.workunit.client.1.smithi189.stdout:2/971: mkdir d1/d11/d2d/d4f/d113/d11f 0 2024-04-20T21:32:27.204 INFO:tasks.workunit.client.1.smithi189.stdout:4/912: mkdir d2/d20/d143 0 2024-04-20T21:32:27.209 INFO:tasks.workunit.client.1.smithi189.stdout:4/913: mkdir d2/dd/d11/d14/d11f/d144 0 2024-04-20T21:32:27.216 INFO:tasks.workunit.client.1.smithi189.stdout:1/902: dwrite d8/d21/d2c/fba [0,4194304] 0 2024-04-20T21:32:27.218 INFO:tasks.workunit.client.1.smithi189.stdout:4/914: link d2/dd/f137 d2/dd/d11/d14/d11f/f145 0 2024-04-20T21:32:27.225 INFO:tasks.workunit.client.1.smithi189.stdout:4/915: creat d2/d20/d115/dcd/d94/f146 x:0 0 0 2024-04-20T21:32:27.229 INFO:tasks.workunit.client.1.smithi189.stdout:4/916: chown d2/dd/d11/d14/l1d 17240512 1 2024-04-20T21:32:27.233 INFO:tasks.workunit.client.1.smithi189.stdout:2/972: dwrite d1/d11/d58/d6b/d10c/f61 [0,4194304] 0 2024-04-20T21:32:27.234 INFO:tasks.workunit.client.1.smithi189.stdout:4/917: dread - d2/d20/d115/dcd/df9/f124 zero size 2024-04-20T21:32:27.239 INFO:tasks.workunit.client.1.smithi189.stdout:1/903: dwrite d8/d1e/d43/d58/d60/f10f [0,4194304] 0 2024-04-20T21:32:27.242 INFO:tasks.workunit.client.1.smithi189.stdout:2/973: rmdir d1/d11/d2d/dab/d118 0 2024-04-20T21:32:27.250 INFO:tasks.workunit.client.1.smithi189.stdout:2/974: creat d1/d11/d2d/d4f/f120 x:0 0 0 2024-04-20T21:32:27.253 INFO:tasks.workunit.client.1.smithi189.stdout:2/975: dread - d1/d11/d2d/dab/f10a zero size 2024-04-20T21:32:27.256 INFO:tasks.workunit.client.1.smithi189.stdout:2/976: chown d1/d11/d2d/dab/f72 5317 1 2024-04-20T21:32:27.259 INFO:tasks.workunit.client.1.smithi189.stdout:2/977: chown d1/d11/d2d/d4f/db4/daa/dd2 165 1 2024-04-20T21:32:27.263 INFO:tasks.workunit.client.1.smithi189.stdout:1/904: dwrite d8/df/d3a/f116 [0,4194304] 0 2024-04-20T21:32:27.266 INFO:tasks.workunit.client.1.smithi189.stdout:2/978: symlink d1/d11/d2d/d4f/d11a/l121 0 2024-04-20T21:32:27.269 INFO:tasks.workunit.client.1.smithi189.stdout:2/979: chown d1/d11/d2d/d4f/db4/daa/cf9 25132 1 2024-04-20T21:32:27.271 INFO:tasks.workunit.client.1.smithi189.stdout:2/980: write d1/d11/d2d/dab/f111 [178615,72724] 0 2024-04-20T21:32:27.275 INFO:tasks.workunit.client.1.smithi189.stdout:1/905: write d8/f1b [4624929,128783] 0 2024-04-20T21:32:27.281 INFO:tasks.workunit.client.1.smithi189.stdout:1/906: mknod d8/d1e/d28/d2b/c121 0 2024-04-20T21:32:27.283 INFO:tasks.workunit.client.1.smithi189.stdout:4/918: dread d2/dd/f44 [0,4194304] 0 2024-04-20T21:32:27.283 INFO:tasks.workunit.client.1.smithi189.stdout:1/907: dread - d8/df/d3a/dfd/f114 zero size 2024-04-20T21:32:27.288 INFO:tasks.workunit.client.1.smithi189.stdout:1/908: mkdir d8/df/d3a/d49/d8c/da2/dbe/dc1/de9/d122 0 2024-04-20T21:32:27.293 INFO:tasks.workunit.client.1.smithi189.stdout:1/909: rename d8/d1e/d43/d44/dd2/d8a/c96 to d8/d9b/c123 0 2024-04-20T21:32:27.294 INFO:tasks.workunit.client.1.smithi189.stdout:4/919: link d2/d22/d57/l81 d2/dd/d11/dae/d10e/d11c/l147 0 2024-04-20T21:32:27.295 INFO:tasks.workunit.client.1.smithi189.stdout:1/910: stat d8/df/d3a/d49/d8c/da2 0 2024-04-20T21:32:27.314 INFO:tasks.workunit.client.1.smithi189.stdout:4/920: dwrite d2/d22/d57/f134 [0,4194304] 0 2024-04-20T21:32:27.317 INFO:tasks.workunit.client.1.smithi189.stdout:1/911: dread d8/d1e/d43/d44/f47 [0,4194304] 0 2024-04-20T21:32:27.338 INFO:tasks.workunit.client.1.smithi189.stdout:1/912: truncate d8/d1e/d43/d44/dd2/f84 2552224 0 2024-04-20T21:32:27.338 INFO:tasks.workunit.client.1.smithi189.stdout:4/921: dread d2/dd/ff5 [0,4194304] 0 2024-04-20T21:32:27.356 INFO:tasks.workunit.client.1.smithi189.stdout:4/922: dwrite d2/dd/d11/f74 [0,4194304] 0 2024-04-20T21:32:27.359 INFO:tasks.workunit.client.1.smithi189.stdout:4/923: write d2/dd/d6b/fc7 [3697814,34814] 0 2024-04-20T21:32:27.369 INFO:tasks.workunit.client.1.smithi189.stdout:1/913: dread d8/f12 [0,4194304] 0 2024-04-20T21:32:27.370 INFO:tasks.workunit.client.1.smithi189.stdout:1/914: chown d8 3845721 1 2024-04-20T21:32:27.370 INFO:tasks.workunit.client.1.smithi189.stdout:4/924: link d2/cba d2/d20/dc6/c148 0 2024-04-20T21:32:27.370 INFO:tasks.workunit.client.1.smithi189.stdout:1/915: chown d8/df/l33 798 1 2024-04-20T21:32:27.374 INFO:tasks.workunit.client.1.smithi189.stdout:4/925: fsync d2/d20/d115/dcd/d6a/d7a/de9/fe4 0 2024-04-20T21:32:27.378 INFO:tasks.workunit.client.1.smithi189.stdout:1/916: creat d8/df/d3a/f124 x:0 0 0 2024-04-20T21:32:27.380 INFO:tasks.workunit.client.1.smithi189.stdout:4/926: creat d2/d13b/f149 x:0 0 0 2024-04-20T21:32:27.383 INFO:tasks.workunit.client.1.smithi189.stdout:1/917: creat d8/d1e/f125 x:0 0 0 2024-04-20T21:32:27.385 INFO:tasks.workunit.client.1.smithi189.stdout:4/927: chown d2/d22/l86 2 1 2024-04-20T21:32:27.388 INFO:tasks.workunit.client.1.smithi189.stdout:1/918: symlink d8/d1e/d43/d44/dd2/d117/l126 0 2024-04-20T21:32:27.392 INFO:tasks.workunit.client.1.smithi189.stdout:4/928: rename d2/d3a/ddc/ff3 to d2/dd/d6b/d126/f14a 0 2024-04-20T21:32:27.411 INFO:tasks.workunit.client.1.smithi189.stdout:1/919: dread d8/d21/d2c/f62 [0,4194304] 0 2024-04-20T21:32:27.412 INFO:tasks.workunit.client.1.smithi189.stdout:4/929: dwrite d2/d20/d115/dcd/d6a/d7a/def/f141 [0,4194304] 0 2024-04-20T21:32:27.413 INFO:tasks.workunit.client.1.smithi189.stdout:1/920: readlink d8/d1e/d43/d44/db2/lcc 0 2024-04-20T21:32:27.422 INFO:tasks.workunit.client.1.smithi189.stdout:4/930: write d2/dd/d11/f92 [5139884,61533] 0 2024-04-20T21:32:27.424 INFO:tasks.workunit.client.1.smithi189.stdout:1/921: creat d8/df/d3a/d49/d8c/da2/dbe/dc1/f127 x:0 0 0 2024-04-20T21:32:27.426 INFO:tasks.workunit.client.1.smithi189.stdout:4/931: creat d2/d22/d57/f14b x:0 0 0 2024-04-20T21:32:27.427 INFO:tasks.workunit.client.1.smithi189.stdout:4/932: dread - d2/d13b/f149 zero size 2024-04-20T21:32:27.431 INFO:tasks.workunit.client.1.smithi189.stdout:1/922: creat d8/df/d3a/d49/d8c/da2/dbe/dd5/f128 x:0 0 0 2024-04-20T21:32:27.433 INFO:tasks.workunit.client.1.smithi189.stdout:9/993: sync 2024-04-20T21:32:27.435 INFO:tasks.workunit.client.1.smithi189.stdout:4/933: rename d2/d20/d115/dcd/d6a/fdd to d2/d20/d115/dcd/d6a/f14c 0 2024-04-20T21:32:27.436 INFO:tasks.workunit.client.1.smithi189.stdout:1/923: creat d8/d9b/f129 x:0 0 0 2024-04-20T21:32:27.438 INFO:tasks.workunit.client.1.smithi189.stdout:9/994: creat d9/dc/d69/d125/d95/da0/d56/f14d x:0 0 0 2024-04-20T21:32:27.442 INFO:tasks.workunit.client.1.smithi189.stdout:4/934: creat d2/d20/d115/dcd/d6a/d7a/f14d x:0 0 0 2024-04-20T21:32:27.444 INFO:tasks.workunit.client.1.smithi189.stdout:4/935: write d2/dd/d6b/fc7 [4397950,16635] 0 2024-04-20T21:32:27.453 INFO:tasks.workunit.client.1.smithi189.stdout:9/995: link d9/dc/d69/d125/d95/da0/d52/d82/d13d/f140 d9/d12/d3b/db1/d1a/f14e 0 2024-04-20T21:32:27.453 INFO:tasks.workunit.client.1.smithi189.stdout:4/936: fsync d2/d20/d115/f59 0 2024-04-20T21:32:27.455 INFO:tasks.workunit.client.1.smithi189.stdout:1/924: dwrite d8/d1e/d43/d58/d60/f10f [0,4194304] 0 2024-04-20T21:32:27.460 INFO:tasks.workunit.client.1.smithi189.stdout:9/996: fdatasync d9/d12/d3f/d72/fc6 0 2024-04-20T21:32:27.464 INFO:tasks.workunit.client.1.smithi189.stdout:4/937: unlink d2/d20/d115/dcd/d6a/d7a/dce/ff4 0 2024-04-20T21:32:27.473 INFO:tasks.workunit.client.1.smithi189.stdout:4/938: symlink d2/dd/d6b/d126/l14e 0 2024-04-20T21:32:27.478 INFO:tasks.workunit.client.1.smithi189.stdout:9/997: mkdir d9/dc/d69/d125/d95/da0/d52/d82/d14f 0 2024-04-20T21:32:27.479 INFO:tasks.workunit.client.1.smithi189.stdout:1/925: link d8/d9b/l118 d8/d1e/l12a 0 2024-04-20T21:32:27.480 INFO:tasks.workunit.client.1.smithi189.stdout:4/939: truncate d2/f75 21661 0 2024-04-20T21:32:27.482 INFO:tasks.workunit.client.1.smithi189.stdout:1/926: creat d8/d21/dbf/f12b x:0 0 0 2024-04-20T21:32:27.485 INFO:tasks.workunit.client.1.smithi189.stdout:9/998: creat d9/dc/d69/d125/d95/da6/f150 x:0 0 0 2024-04-20T21:32:27.486 INFO:tasks.workunit.client.1.smithi189.stdout:4/940: fdatasync d2/d20/d115/dcd/d6a/d7a/f128 0 2024-04-20T21:32:27.487 INFO:tasks.workunit.client.1.smithi189.stdout:1/927: chown d8/c65 1988579 1 2024-04-20T21:32:27.492 INFO:tasks.workunit.client.1.smithi189.stdout:4/941: mknod d2/dd/d6b/c14f 0 2024-04-20T21:32:27.493 INFO:tasks.workunit.client.1.smithi189.stdout:1/928: fsync d8/d9b/fa6 0 2024-04-20T21:32:27.495 INFO:tasks.workunit.client.1.smithi189.stdout:9/999: creat d9/d12/d129/d138/f151 x:0 0 0 2024-04-20T21:32:27.499 INFO:tasks.workunit.client.1.smithi189.stdout:4/942: getdents d2/d22 0 2024-04-20T21:32:27.514 INFO:tasks.workunit.client.1.smithi189.stdout:2/981: sync 2024-04-20T21:32:27.526 INFO:tasks.workunit.client.1.smithi189.stdout:4/943: truncate d2/d22/f6e 488441 0 2024-04-20T21:32:27.532 INFO:tasks.workunit.client.1.smithi189.stdout:1/929: dwrite d8/df/d3a/d49/d8c/fb9 [0,4194304] 0 2024-04-20T21:32:27.536 INFO:tasks.workunit.client.1.smithi189.stdout:4/944: read d2/f4 [767036,70275] 0 2024-04-20T21:32:27.538 INFO:tasks.workunit.client.1.smithi189.stdout:1/930: mkdir d8/d1e/d43/d44/d12c 0 2024-04-20T21:32:27.540 INFO:tasks.workunit.client.1.smithi189.stdout:4/945: fsync d2/d20/d115/dcd/d6a/d7a/f8d 0 2024-04-20T21:32:27.542 INFO:tasks.workunit.client.1.smithi189.stdout:1/931: rmdir d8/d9b 39 2024-04-20T21:32:27.545 INFO:tasks.workunit.client.1.smithi189.stdout:1/932: stat d8/d1e/d43/d44/dd2/d8a/lf9 0 2024-04-20T21:32:27.548 INFO:tasks.workunit.client.1.smithi189.stdout:4/946: symlink d2/d20/d115/dcd/d94/d107/d11a/l150 0 2024-04-20T21:32:27.549 INFO:tasks.workunit.client.1.smithi189.stdout:1/933: write d8/d1e/d43/d44/dd2/fc7 [323258,122830] 0 2024-04-20T21:32:27.562 INFO:tasks.workunit.client.1.smithi189.stdout:4/947: write d2/d22/da9/fbf [1935211,95148] 0 2024-04-20T21:32:27.566 INFO:tasks.workunit.client.1.smithi189.stdout:4/948: symlink d2/d20/d143/l151 0 2024-04-20T21:32:27.573 INFO:tasks.workunit.client.1.smithi189.stdout:4/949: rename d2/d3a/ld6 to d2/d20/d115/dcd/d6a/d7a/db7/dea/l152 0 2024-04-20T21:32:27.577 INFO:tasks.workunit.client.1.smithi189.stdout:4/950: symlink d2/dd/d11/d14/d11f/l153 0 2024-04-20T21:32:27.582 INFO:tasks.workunit.client.1.smithi189.stdout:4/951: getdents d2/d20/d115/dcd/d94 0 2024-04-20T21:32:27.584 INFO:tasks.workunit.client.1.smithi189.stdout:4/952: chown d2/d20/f24 1 1 2024-04-20T21:32:27.585 INFO:tasks.workunit.client.1.smithi189.stdout:4/953: fsync d2/d22/d57/f134 0 2024-04-20T21:32:27.587 INFO:tasks.workunit.client.1.smithi189.stdout:4/954: chown d2/d20/d115/dcd/d6a/d7a/def/de8/l11b 1692380047 1 2024-04-20T21:32:27.589 INFO:tasks.workunit.client.1.smithi189.stdout:4/955: chown d2/d20/d115/dcd/l109 39 1 2024-04-20T21:32:27.591 INFO:tasks.workunit.client.1.smithi189.stdout:4/956: write d2/d22/f30 [4851583,49890] 0 2024-04-20T21:32:27.594 INFO:tasks.workunit.client.1.smithi189.stdout:4/957: rmdir d2/d20/d115/dcd/d6a/d7a/def/de8 39 2024-04-20T21:32:27.598 INFO:tasks.workunit.client.1.smithi189.stdout:4/958: creat d2/dd/f154 x:0 0 0 2024-04-20T21:32:27.600 INFO:tasks.workunit.client.1.smithi189.stdout:4/959: write d2/d20/dc6/f131 [752053,77186] 0 2024-04-20T21:32:27.604 INFO:tasks.workunit.client.1.smithi189.stdout:4/960: unlink d2/d20/d115/dcd/f9e 0 2024-04-20T21:32:27.613 INFO:tasks.workunit.client.1.smithi189.stdout:4/961: dread d2/d20/d115/dcd/d6a/d7a/f8d [0,4194304] 0 2024-04-20T21:32:27.615 INFO:tasks.workunit.client.1.smithi189.stdout:4/962: chown d2/d20/d115/lc9 10121 1 2024-04-20T21:32:27.618 INFO:tasks.workunit.client.1.smithi189.stdout:4/963: mkdir d2/dd/d11/d155 0 2024-04-20T21:32:27.622 INFO:tasks.workunit.client.1.smithi189.stdout:4/964: rename d2/f8 to d2/dd/d11/f156 0 2024-04-20T21:32:27.636 INFO:tasks.workunit.client.1.smithi189.stdout:4/965: dwrite d2/d22/d87/f130 [0,4194304] 0 2024-04-20T21:32:27.640 INFO:tasks.workunit.client.1.smithi189.stdout:4/966: mkdir d2/d20/d115/dcd/d6a/d7a/def/d157 0 2024-04-20T21:32:27.647 INFO:tasks.workunit.client.1.smithi189.stdout:4/967: creat d2/d20/d115/dcd/d6a/d7a/def/f158 x:0 0 0 2024-04-20T21:32:27.652 INFO:tasks.workunit.client.1.smithi189.stdout:4/968: creat d2/d20/d115/dcd/f159 x:0 0 0 2024-04-20T21:32:27.655 INFO:tasks.workunit.client.1.smithi189.stdout:4/969: unlink d2/d20/d115/le2 0 2024-04-20T21:32:27.666 INFO:tasks.workunit.client.1.smithi189.stdout:4/970: dwrite d2/d22/f30 [0,4194304] 0 2024-04-20T21:32:27.674 INFO:tasks.workunit.client.1.smithi189.stdout:4/971: mknod d2/dd/d6b/d126/c15a 0 2024-04-20T21:32:27.683 INFO:tasks.workunit.client.1.smithi189.stdout:4/972: dread d2/d22/f5d [0,4194304] 0 2024-04-20T21:32:27.685 INFO:tasks.workunit.client.1.smithi189.stdout:2/982: sync 2024-04-20T21:32:27.685 INFO:tasks.workunit.client.1.smithi189.stdout:1/934: sync 2024-04-20T21:32:27.692 INFO:tasks.workunit.client.1.smithi189.stdout:4/973: rename d2/d20/dc6/le5 to d2/d20/d115/l15b 0 2024-04-20T21:32:27.694 INFO:tasks.workunit.client.1.smithi189.stdout:4/974: chown d2/d20/d115/dcd/d6a/ld3 437784020 1 2024-04-20T21:32:27.695 INFO:tasks.workunit.client.1.smithi189.stdout:1/935: unlink d8/d1e/d43/d44/dd2/d8a/lb8 0 2024-04-20T21:32:27.696 INFO:tasks.workunit.client.1.smithi189.stdout:4/975: dread - d2/d3a/ddc/f108 zero size 2024-04-20T21:32:27.701 INFO:tasks.workunit.client.1.smithi189.stdout:2/983: dwrite d1/d11/d99/db7/ff5 [4194304,4194304] 0 2024-04-20T21:32:27.702 INFO:tasks.workunit.client.1.smithi189.stdout:4/976: mknod d2/d20/d115/dcd/d6a/d7a/db7/dea/c15c 0 2024-04-20T21:32:27.705 INFO:tasks.workunit.client.1.smithi189.stdout:2/984: symlink d1/d11/d2d/d4f/d92/l122 0 2024-04-20T21:32:27.705 INFO:tasks.workunit.client.1.smithi189.stdout:1/936: link d8/d1e/d43/d58/d60/ld7 d8/d1e/d43/d44/d12c/l12d 0 2024-04-20T21:32:27.707 INFO:tasks.workunit.client.1.smithi189.stdout:4/977: mknod d2/d20/d115/dcd/d6a/d7a/dce/c15d 0 2024-04-20T21:32:27.713 INFO:tasks.workunit.client.1.smithi189.stdout:1/937: mknod d8/d1e/d43/d58/d112/c12e 0 2024-04-20T21:32:27.714 INFO:tasks.workunit.client.1.smithi189.stdout:4/978: mknod d2/d20/d115/dcd/d6a/d7a/db7/d10f/c15e 0 2024-04-20T21:32:27.716 INFO:tasks.workunit.client.1.smithi189.stdout:1/938: readlink d8/d1e/d43/d44/db2/lcc 0 2024-04-20T21:32:27.716 INFO:tasks.workunit.client.1.smithi189.stdout:4/979: readlink d2/d20/dc6/dda/lf7 0 2024-04-20T21:32:27.727 INFO:tasks.workunit.client.1.smithi189.stdout:4/980: symlink d2/d20/d115/l15f 0 2024-04-20T21:32:27.730 INFO:tasks.workunit.client.1.smithi189.stdout:1/939: mkdir d8/d1e/d43/d58/d12f 0 2024-04-20T21:32:27.740 INFO:tasks.workunit.client.1.smithi189.stdout:4/981: rename d2/dd/d11/d14/f50 to d2/d20/d115/dcd/d6a/d7a/def/f160 0 2024-04-20T21:32:27.745 INFO:tasks.workunit.client.1.smithi189.stdout:4/982: symlink d2/dd/d6b/l161 0 2024-04-20T21:32:27.749 INFO:tasks.workunit.client.1.smithi189.stdout:1/940: rename d8/d1e/d43/d58/fe7 to d8/d1e/d43/f130 0 2024-04-20T21:32:27.752 INFO:tasks.workunit.client.1.smithi189.stdout:2/985: dwrite d1/d17/f35 [4194304,4194304] 0 2024-04-20T21:32:27.754 INFO:tasks.workunit.client.1.smithi189.stdout:4/983: rename d2/dd to d2/dd/d162 22 2024-04-20T21:32:27.756 INFO:tasks.workunit.client.1.smithi189.stdout:2/986: read d1/d11/d2d/dab/d4a/d66/df1/f100 [1392485,77395] 0 2024-04-20T21:32:27.757 INFO:tasks.workunit.client.1.smithi189.stdout:2/987: stat d1/d11/d99/la5 0 2024-04-20T21:32:27.765 INFO:tasks.workunit.client.1.smithi189.stdout:4/984: fsync d2/dd/d11/f7e 0 2024-04-20T21:32:27.770 INFO:tasks.workunit.client.1.smithi189.stdout:1/941: getdents d8/df/d3a/d49/d8c/da2 0 2024-04-20T21:32:27.772 INFO:tasks.workunit.client.1.smithi189.stdout:2/988: getdents d1/d11/d2d/d4f/d8e 0 2024-04-20T21:32:27.774 INFO:tasks.workunit.client.1.smithi189.stdout:4/985: rename d2/dd/d6b/c106 to d2/d20/dc6/dda/d12e/c163 0 2024-04-20T21:32:27.777 INFO:tasks.workunit.client.1.smithi189.stdout:1/942: rename d8/d21/d2c/lab to d8/d1e/d43/d44/dd2/l131 0 2024-04-20T21:32:27.778 INFO:tasks.workunit.client.1.smithi189.stdout:4/986: rmdir d2/d20/d115/dcd/d6a 39 2024-04-20T21:32:27.780 INFO:tasks.workunit.client.1.smithi189.stdout:4/987: write d2/f6d [1175708,31021] 0 2024-04-20T21:32:27.784 INFO:tasks.workunit.client.1.smithi189.stdout:2/989: write d1/d17/f34 [6178155,130228] 0 2024-04-20T21:32:27.786 INFO:tasks.workunit.client.1.smithi189.stdout:4/988: creat d2/dd/d11/d155/f164 x:0 0 0 2024-04-20T21:32:27.790 INFO:tasks.workunit.client.1.smithi189.stdout:2/990: creat d1/d11/d99/db7/f123 x:0 0 0 2024-04-20T21:32:27.794 INFO:tasks.workunit.client.1.smithi189.stdout:1/943: dwrite d8/f14 [0,4194304] 0 2024-04-20T21:32:27.799 INFO:tasks.workunit.client.1.smithi189.stdout:1/944: read f6 [393227,90696] 0 2024-04-20T21:32:27.805 INFO:tasks.workunit.client.1.smithi189.stdout:2/991: dread d1/d11/d2d/dab/d74/f109 [0,4194304] 0 2024-04-20T21:32:27.807 INFO:tasks.workunit.client.1.smithi189.stdout:1/945: rename d8/d1e/d28/d2b to d8/d1e/d43/d44/dd2/d117/d132 0 2024-04-20T21:32:27.809 INFO:tasks.workunit.client.1.smithi189.stdout:1/946: dread - d8/d1e/d43/d44/db2/f107 zero size 2024-04-20T21:32:27.810 INFO:tasks.workunit.client.1.smithi189.stdout:2/992: rmdir d1/d11/d2d/d4f/d113/d11f 0 2024-04-20T21:32:27.811 INFO:tasks.workunit.client.1.smithi189.stdout:1/947: mknod d8/df/d3a/dfd/c133 0 2024-04-20T21:32:27.815 INFO:tasks.workunit.client.1.smithi189.stdout:2/993: creat d1/d17/d37/f124 x:0 0 0 2024-04-20T21:32:27.816 INFO:tasks.workunit.client.1.smithi189.stdout:2/994: read d1/d17/f2b [3768163,16166] 0 2024-04-20T21:32:27.816 INFO:tasks.workunit.client.1.smithi189.stdout:1/948: creat d8/df/d3a/d49/d8c/da2/dfe/f134 x:0 0 0 2024-04-20T21:32:27.818 INFO:tasks.workunit.client.1.smithi189.stdout:2/995: chown d1/d11/d2d/dab/d4a/d66/l80 2757 1 2024-04-20T21:32:27.822 INFO:tasks.workunit.client.1.smithi189.stdout:1/949: link d8/f24 d8/d1e/d43/d58/dc2/f135 0 2024-04-20T21:32:27.826 INFO:tasks.workunit.client.1.smithi189.stdout:1/950: rmdir d8/d1e/d28/dd4/ddb 0 2024-04-20T21:32:27.830 INFO:tasks.workunit.client.1.smithi189.stdout:2/996: dread d1/d11/d2d/d4f/fbc [0,4194304] 0 2024-04-20T21:32:27.830 INFO:tasks.workunit.client.1.smithi189.stdout:1/951: mknod d8/d9b/dbb/c136 0 2024-04-20T21:32:27.845 INFO:tasks.workunit.client.1.smithi189.stdout:1/952: dwrite d8/df/f83 [0,4194304] 0 2024-04-20T21:32:27.850 INFO:tasks.workunit.client.1.smithi189.stdout:1/953: unlink f6 0 2024-04-20T21:32:27.850 INFO:tasks.workunit.client.1.smithi189.stdout:2/997: dread d1/d17/d78/fac [0,4194304] 0 2024-04-20T21:32:27.851 INFO:tasks.workunit.client.1.smithi189.stdout:1/954: stat d8/d21/f11d 0 2024-04-20T21:32:27.852 INFO:tasks.workunit.client.1.smithi189.stdout:2/998: read - d1/d11/d2d/dab/d74/fda zero size 2024-04-20T21:32:27.860 INFO:tasks.workunit.client.1.smithi189.stdout:1/955: link d8/d1e/c9a d8/d21/d2c/d113/c137 0 2024-04-20T21:32:27.861 INFO:tasks.workunit.client.1.smithi189.stdout:2/999: creat d1/d17/f125 x:0 0 0 2024-04-20T21:32:27.878 INFO:tasks.workunit.client.1.smithi189.stdout:1/956: dwrite d8/df/d3a/d49/d8c/da2/fc0 [0,4194304] 0 2024-04-20T21:32:27.879 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:27 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:32:27.880 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:27 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:32:27.880 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:27 smithi189 ceph-mon[21562]: pgmap v28: 65 pgs: 65 active+clean; 3.9 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 80 MiB/s rd, 196 MiB/s wr, 429 op/s 2024-04-20T21:32:27.880 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:27 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-20T21:32:27.880 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:27 smithi189 ceph-mon[21562]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-20T21:32:27.880 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:27 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:32:27.884 INFO:tasks.workunit.client.1.smithi189.stdout:1/957: rmdir d8/d1e/d54 39 2024-04-20T21:32:27.887 INFO:tasks.workunit.client.1.smithi189.stdout:1/958: read d8/d1e/d43/d44/dd2/d8a/fd0 [103047,42933] 0 2024-04-20T21:32:27.894 INFO:tasks.workunit.client.1.smithi189.stdout:1/959: unlink d8/d1e/d54/ld3 0 2024-04-20T21:32:27.898 INFO:tasks.workunit.client.1.smithi189.stdout:1/960: creat d8/d21/d2c/d113/f138 x:0 0 0 2024-04-20T21:32:27.902 INFO:tasks.workunit.client.1.smithi189.stdout:1/961: symlink d8/df/d3a/d49/d8c/d103/l139 0 2024-04-20T21:32:27.903 INFO:tasks.workunit.client.1.smithi189.stdout:1/962: chown d8/f46 488364222 1 2024-04-20T21:32:27.907 INFO:tasks.workunit.client.1.smithi189.stdout:1/963: creat d8/d1e/d43/d44/dd2/db7/dcf/f13a x:0 0 0 2024-04-20T21:32:27.909 INFO:tasks.workunit.client.1.smithi189.stdout:1/964: fsync d8/d1e/d28/fa8 0 2024-04-20T21:32:27.911 INFO:tasks.workunit.client.1.smithi189.stdout:1/965: stat d8/df/c29 0 2024-04-20T21:32:27.941 INFO:tasks.workunit.client.1.smithi189.stdout:4/989: sync 2024-04-20T21:32:27.947 INFO:tasks.workunit.client.1.smithi189.stdout:4/990: creat d2/dd/d11/f165 x:0 0 0 2024-04-20T21:32:27.950 INFO:tasks.workunit.client.1.smithi189.stdout:4/991: chown d2/dd/d11/d14/ffe 86442598 1 2024-04-20T21:32:27.951 INFO:tasks.workunit.client.1.smithi189.stdout:1/966: write d8/df/d3a/d49/d8c/da2/dfe/f5f [1074417,7695] 0 2024-04-20T21:32:27.976 INFO:tasks.workunit.client.1.smithi189.stdout:1/967: dwrite d8/d1e/f125 [0,4194304] 0 2024-04-20T21:32:27.978 INFO:tasks.workunit.client.1.smithi189.stdout:4/992: dwrite d2/d20/d115/dcd/d6a/d7a/def/f160 [4194304,4194304] 0 2024-04-20T21:32:27.982 INFO:tasks.workunit.client.1.smithi189.stdout:4/993: dread d2/d22/f5d [0,4194304] 0 2024-04-20T21:32:27.988 INFO:tasks.workunit.client.1.smithi189.stdout:1/968: mknod d8/d21/dbf/c13b 0 2024-04-20T21:32:27.990 INFO:tasks.workunit.client.1.smithi189.stdout:4/994: read d2/f75 [16526,114235] 0 2024-04-20T21:32:27.991 INFO:tasks.workunit.client.1.smithi189.stdout:1/969: readlink d8/d1e/d43/d44/db2/lcc 0 2024-04-20T21:32:27.991 INFO:tasks.workunit.client.1.smithi189.stdout:4/995: chown d2/d20/dc6/dda 2834 1 2024-04-20T21:32:27.994 INFO:tasks.workunit.client.1.smithi189.stdout:4/996: readlink d2/d20/dc6/dda/l123 0 2024-04-20T21:32:28.001 INFO:tasks.workunit.client.1.smithi189.stdout:4/997: getdents d2/d20/d115/dcd/d94 0 2024-04-20T21:32:28.002 INFO:tasks.workunit.client.1.smithi189.stdout:4/998: stat d2/dd/d11/c77 0 2024-04-20T21:32:28.026 INFO:tasks.workunit.client.1.smithi189.stdout:1/970: dread d8/d9b/fa6 [0,4194304] 0 2024-04-20T21:32:28.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:27 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:32:28.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:27 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:32:28.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:27 smithi062 ceph-mon[16221]: pgmap v28: 65 pgs: 65 active+clean; 3.9 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 80 MiB/s rd, 196 MiB/s wr, 429 op/s 2024-04-20T21:32:28.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:27 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-20T21:32:28.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:27 smithi062 ceph-mon[16221]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-20T21:32:28.037 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:27 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:32:28.041 INFO:tasks.workunit.client.1.smithi189.stdout:4/999: dread d2/dd/d11/f92 [0,4194304] 0 2024-04-20T21:32:28.046 INFO:tasks.workunit.client.1.smithi189.stdout:1/971: write d8/d1e/f3c [283363,119013] 0 2024-04-20T21:32:28.102 INFO:tasks.workunit.client.1.smithi189.stdout:1/972: symlink d8/df/d3a/d49/d8c/da2/dfe/l13c 0 2024-04-20T21:32:28.112 INFO:tasks.workunit.client.1.smithi189.stdout:1/973: write d8/df/d3a/d49/d8c/da2/dbe/dc1/fcd [4956799,77201] 0 2024-04-20T21:32:28.118 INFO:tasks.workunit.client.1.smithi189.stdout:1/974: fdatasync d8/f24 0 2024-04-20T21:32:28.120 INFO:tasks.workunit.client.1.smithi189.stdout:1/975: write d8/df/d3a/f116 [2895732,31862] 0 2024-04-20T21:32:28.135 INFO:tasks.workunit.client.1.smithi189.stdout:1/976: dwrite d8/d1e/fdf [0,4194304] 0 2024-04-20T21:32:28.166 INFO:tasks.workunit.client.1.smithi189.stdout:1/977: write d8/f24 [1368079,36352] 0 2024-04-20T21:32:28.186 INFO:tasks.workunit.client.1.smithi189.stdout:1/978: dwrite d8/d21/f11d [0,4194304] 0 2024-04-20T21:32:28.200 INFO:tasks.workunit.client.1.smithi189.stdout:1/979: dread d8/d1e/d43/d44/dd2/f5b [0,4194304] 0 2024-04-20T21:32:28.207 INFO:tasks.workunit.client.1.smithi189.stdout:1/980: symlink d8/df/d3a/d49/d8c/da2/dbe/dc1/de9/d122/l13d 0 2024-04-20T21:32:28.211 INFO:tasks.workunit.client.1.smithi189.stdout:1/981: readlink d8/d1e/d43/lcb 0 2024-04-20T21:32:28.217 INFO:tasks.workunit.client.1.smithi189.stdout:1/982: mknod d8/d1e/d43/d44/dd2/db7/dcf/c13e 0 2024-04-20T21:32:28.219 INFO:tasks.workunit.client.1.smithi189.stdout:1/983: chown d8/d1e/d28/fd8 44790 1 2024-04-20T21:32:28.222 INFO:tasks.workunit.client.1.smithi189.stdout:1/984: chown d8/d21/d2c/l7f 0 1 2024-04-20T21:32:28.224 INFO:tasks.workunit.client.1.smithi189.stdout:1/985: fdatasync d8/d1e/d54/d93/fbd 0 2024-04-20T21:32:28.232 INFO:tasks.workunit.client.1.smithi189.stdout:1/986: creat d8/df/d50/de6/f13f x:0 0 0 2024-04-20T21:32:28.237 INFO:tasks.workunit.client.1.smithi189.stdout:1/987: mknod d8/df/d3a/dfd/c140 0 2024-04-20T21:32:28.261 INFO:tasks.workunit.client.1.smithi189.stdout:1/988: dwrite d8/d21/d2c/f62 [0,4194304] 0 2024-04-20T21:32:28.270 INFO:tasks.workunit.client.1.smithi189.stdout:1/989: mkdir d8/d1e/d43/d44/dd2/d8a/d141 0 2024-04-20T21:32:28.276 INFO:tasks.workunit.client.1.smithi189.stdout:1/990: truncate d8/df/d3a/f78 940441 0 2024-04-20T21:32:28.291 INFO:tasks.workunit.client.1.smithi189.stdout:1/991: link d8/df/c29 d8/df/d3a/d49/d8c/da2/dbe/dc1/dd9/c142 0 2024-04-20T21:32:28.296 INFO:tasks.workunit.client.1.smithi189.stdout:1/992: mkdir d8/d1e/d43/d58/d143 0 2024-04-20T21:32:28.310 INFO:tasks.workunit.client.1.smithi189.stdout:1/993: dread d8/f14 [0,4194304] 0 2024-04-20T21:32:28.634 INFO:tasks.workunit.client.1.smithi189.stdout:1/994: sync 2024-04-20T21:32:28.639 INFO:tasks.workunit.client.1.smithi189.stdout:1/995: mkdir d8/d1e/d43/d58/d143/d144 0 2024-04-20T21:32:28.646 INFO:tasks.workunit.client.1.smithi189.stdout:1/996: symlink d8/d1e/d43/d44/l145 0 2024-04-20T21:32:28.901 INFO:tasks.workunit.client.1.smithi189.stdout:1/997: sync 2024-04-20T21:32:28.911 INFO:tasks.workunit.client.1.smithi189.stdout:1/998: creat d8/df/d3a/d49/d8c/da2/f146 x:0 0 0 2024-04-20T21:32:28.920 INFO:tasks.workunit.client.1.smithi189.stdout:1/999: unlink d8/df/d3a/d49/d8c/da2/dbe/dc1/l102 0 2024-04-20T21:32:28.934 INFO:tasks.workunit.client.1.smithi189.stderr:+ rm -rf -- ./tmp.3tuvMuLnfy 2024-04-20T21:32:29.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:29 smithi189 ceph-mon[21562]: pgmap v29: 65 pgs: 65 active+clean; 4.1 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 110 MiB/s rd, 252 MiB/s wr, 598 op/s 2024-04-20T21:32:30.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:29 smithi062 ceph-mon[16221]: pgmap v29: 65 pgs: 65 active+clean; 4.1 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 110 MiB/s rd, 252 MiB/s wr, 598 op/s 2024-04-20T21:32:30.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:30 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi189.tawavz", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-20T21:32:30.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:30 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-20T21:32:30.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:30 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:32:31.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:30 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi189.tawavz", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-20T21:32:31.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:30 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-20T21:32:31.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:30 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:32:31.903 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:31 smithi189 ceph-mon[21562]: Upgrade: Updating mgr.smithi189.tawavz 2024-04-20T21:32:31.903 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:31 smithi189 ceph-mon[21562]: Deploying daemon mgr.smithi189.tawavz on smithi189 2024-04-20T21:32:31.903 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:31 smithi189 ceph-mon[21562]: pgmap v30: 65 pgs: 65 active+clean; 4.1 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 72 MiB/s rd, 154 MiB/s wr, 382 op/s 2024-04-20T21:32:32.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:31 smithi062 ceph-mon[16221]: Upgrade: Updating mgr.smithi189.tawavz 2024-04-20T21:32:32.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:31 smithi062 ceph-mon[16221]: Deploying daemon mgr.smithi189.tawavz on smithi189 2024-04-20T21:32:32.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:31 smithi062 ceph-mon[16221]: pgmap v30: 65 pgs: 65 active+clean; 4.1 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 72 MiB/s rd, 154 MiB/s wr, 382 op/s 2024-04-20T21:32:33.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:32 smithi189 ceph-mon[21562]: pgmap v31: 65 pgs: 65 active+clean; 4.1 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 72 MiB/s rd, 154 MiB/s wr, 382 op/s 2024-04-20T21:32:33.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:32 smithi062 ceph-mon[16221]: pgmap v31: 65 pgs: 65 active+clean; 4.1 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 72 MiB/s rd, 154 MiB/s wr, 382 op/s 2024-04-20T21:32:35.826 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:35 smithi189 ceph-mon[21562]: pgmap v32: 65 pgs: 65 active+clean; 3.6 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 74 MiB/s rd, 159 MiB/s wr, 451 op/s 2024-04-20T21:32:36.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:35 smithi062 ceph-mon[16221]: pgmap v32: 65 pgs: 65 active+clean; 3.6 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 74 MiB/s rd, 159 MiB/s wr, 451 op/s 2024-04-20T21:32:37.448 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:37 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:32:37.448 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:37 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:32:37.448 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:37 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:32:37.448 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:37 smithi189 ceph-mon[21562]: pgmap v33: 65 pgs: 65 active+clean; 3.6 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 32 MiB/s rd, 61 MiB/s wr, 238 op/s 2024-04-20T21:32:37.448 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:37 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:32:37.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:37 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:32:37.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:37 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:32:37.537 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:37 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:32:37.537 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:37 smithi062 ceph-mon[16221]: pgmap v33: 65 pgs: 65 active+clean; 3.6 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 32 MiB/s rd, 61 MiB/s wr, 238 op/s 2024-04-20T21:32:37.537 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:37 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:32:39.919 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:39 smithi189 ceph-mon[21562]: pgmap v34: 65 pgs: 65 active+clean; 2.9 GiB data, 8.9 GiB used, 528 GiB / 536 GiB avail; 32 MiB/s rd, 62 MiB/s wr, 306 op/s 2024-04-20T21:32:40.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:39 smithi062 ceph-mon[16221]: pgmap v34: 65 pgs: 65 active+clean; 2.9 GiB data, 8.9 GiB used, 528 GiB / 536 GiB avail; 32 MiB/s rd, 62 MiB/s wr, 306 op/s 2024-04-20T21:32:41.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:41 smithi189 ceph-mon[21562]: pgmap v35: 65 pgs: 65 active+clean; 2.9 GiB data, 8.9 GiB used, 528 GiB / 536 GiB avail; 2.4 MiB/s rd, 5.0 MiB/s wr, 136 op/s 2024-04-20T21:32:41.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:41 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:32:41.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:41 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:32:42.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:41 smithi062 ceph-mon[16221]: pgmap v35: 65 pgs: 65 active+clean; 2.9 GiB data, 8.9 GiB used, 528 GiB / 536 GiB avail; 2.4 MiB/s rd, 5.0 MiB/s wr, 136 op/s 2024-04-20T21:32:42.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:41 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:32:42.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:41 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:32:42.986 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:42 smithi189 ceph-mon[21562]: pgmap v36: 65 pgs: 65 active+clean; 2.9 GiB data, 8.9 GiB used, 528 GiB / 536 GiB avail; 2.4 MiB/s rd, 5.0 MiB/s wr, 136 op/s 2024-04-20T21:32:43.235 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:42 smithi062 ceph-mon[16221]: pgmap v36: 65 pgs: 65 active+clean; 2.9 GiB data, 8.9 GiB used, 528 GiB / 536 GiB avail; 2.4 MiB/s rd, 5.0 MiB/s wr, 136 op/s 2024-04-20T21:32:44.330 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:44 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:32:44.330 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:44 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:32:44.511 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:44 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:32:44.511 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:44 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:32:45.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:45.041+0000 7f3c27400700 1 -- 172.21.15.62:0/3809760669 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3c20101440 msgr2=0x7f3c20101890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:32:45.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:45.041+0000 7f3c27400700 1 --2- 172.21.15.62:0/3809760669 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3c20101440 0x7f3c20101890 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f3c0c009910 tx=0x7f3c0c009c20 comp rx=0 tx=0).stop 2024-04-20T21:32:45.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:45.042+0000 7f3c27400700 1 -- 172.21.15.62:0/3809760669 shutdown_connections 2024-04-20T21:32:45.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:45.042+0000 7f3c27400700 1 --2- 172.21.15.62:0/3809760669 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3c20101440 0x7f3c20101890 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:32:45.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:45.042+0000 7f3c27400700 1 --2- 172.21.15.62:0/3809760669 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3c20100240 0x7f3c20100650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:32:45.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:45.042+0000 7f3c27400700 1 -- 172.21.15.62:0/3809760669 >> 172.21.15.62:0/3809760669 conn(0x7f3c200fb7f0 msgr2=0x7f3c200fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:32:45.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:45.042+0000 7f3c27400700 1 -- 172.21.15.62:0/3809760669 shutdown_connections 2024-04-20T21:32:45.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:45.042+0000 7f3c27400700 1 -- 172.21.15.62:0/3809760669 wait complete. 2024-04-20T21:32:45.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:45.044+0000 7f3c27400700 1 Processor -- start 2024-04-20T21:32:45.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:45.044+0000 7f3c27400700 1 -- start start 2024-04-20T21:32:45.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:45.045+0000 7f3c27400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3c20100240 0x7f3c20195ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:32:45.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:45.045+0000 7f3c27400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3c20101440 0x7f3c20195ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:32:45.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:45.045+0000 7f3c27400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c20196610 con 0x7f3c20100240 2024-04-20T21:32:45.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:45.045+0000 7f3c27400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c20196750 con 0x7f3c20101440 2024-04-20T21:32:45.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:45.046+0000 7f3c1fe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3c20101440 0x7f3c20195ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:32:45.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:45.046+0000 7f3c1fe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3c20101440 0x7f3c20195ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.62:60388/0 (socket says 172.21.15.62:60388) 2024-04-20T21:32:45.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:45.046+0000 7f3c1fe00700 1 -- 172.21.15.62:0/1018018473 learned_addr learned my addr 172.21.15.62:0/1018018473 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:32:45.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:45.046+0000 7f3c24a00700 1 --2- 172.21.15.62:0/1018018473 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3c20100240 0x7f3c20195ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:32:45.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:45.046+0000 7f3c1fe00700 1 -- 172.21.15.62:0/1018018473 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3c20100240 msgr2=0x7f3c20195ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:32:45.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:45.046+0000 7f3c1fe00700 1 --2- 172.21.15.62:0/1018018473 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3c20100240 0x7f3c20195ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:32:45.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:45.046+0000 7f3c1fe00700 1 -- 172.21.15.62:0/1018018473 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c0c0095f0 con 0x7f3c20101440 2024-04-20T21:32:45.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:45.047+0000 7f3c1fe00700 1 --2- 172.21.15.62:0/1018018473 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3c20101440 0x7f3c20195ff0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f3c0c005b40 tx=0x7f3c0c005800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:32:45.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:45.047+0000 7f3c1d600700 1 -- 172.21.15.62:0/1018018473 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3c0c023cb0 con 0x7f3c20101440 2024-04-20T21:32:45.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:45.047+0000 7f3c27400700 1 -- 172.21.15.62:0/1018018473 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c20072f80 con 0x7f3c20101440 2024-04-20T21:32:45.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:45.047+0000 7f3c1d600700 1 -- 172.21.15.62:0/1018018473 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f3c0c023e10 con 0x7f3c20101440 2024-04-20T21:32:45.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:45.047+0000 7f3c1d600700 1 -- 172.21.15.62:0/1018018473 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3c0c00fae0 con 0x7f3c20101440 2024-04-20T21:32:45.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:45.048+0000 7f3c27400700 1 -- 172.21.15.62:0/1018018473 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c20073470 con 0x7f3c20101440 2024-04-20T21:32:45.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:45.051+0000 7f3c1d600700 1 -- 172.21.15.62:0/1018018473 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 29) v1 ==== 92700+0+0 (secure 0 0 0) 0x7f3c0c029070 con 0x7f3c20101440 2024-04-20T21:32:45.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:45.052+0000 7f3c27400700 1 -- 172.21.15.62:0/1018018473 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c20108d90 con 0x7f3c20101440 2024-04-20T21:32:45.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:45.053+0000 7f3c1d600700 1 --2- 172.21.15.62:0/1018018473 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f3c0806ea70 0x7f3c08070f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:32:45.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:45.053+0000 7f3c1d600700 1 -- 172.21.15.62:0/1018018473 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f3c0c098800 con 0x7f3c20101440 2024-04-20T21:32:45.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:45.054+0000 7f3c24a00700 1 --2- 172.21.15.62:0/1018018473 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f3c0806ea70 0x7f3c08070f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:32:45.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:45.055+0000 7f3c24a00700 1 --2- 172.21.15.62:0/1018018473 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f3c0806ea70 0x7f3c08070f20 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f3c201012a0 tx=0x7f3c14007400 comp rx=0 tx=0).ready entity=mgr.14658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:32:45.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:45.066+0000 7f3c1d600700 1 -- 172.21.15.62:0/1018018473 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187527 (secure 0 0 0) 0x7f3c0c065460 con 0x7f3c20101440 2024-04-20T21:32:45.343 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:45 smithi189 ceph-mon[21562]: pgmap v37: 65 pgs: 65 active+clean; 2.0 GiB data, 6.3 GiB used, 530 GiB / 536 GiB avail; 2.9 MiB/s rd, 5.4 MiB/s wr, 204 op/s 2024-04-20T21:32:45.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:45.417+0000 7f3c27400700 1 -- 172.21.15.62:0/1018018473 --> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3c20105d90 con 0x7f3c0806ea70 2024-04-20T21:32:45.417 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:45 smithi062 ceph-mon[16221]: pgmap v37: 65 pgs: 65 active+clean; 2.0 GiB data, 6.3 GiB used, 530 GiB / 536 GiB avail; 2.9 MiB/s rd, 5.4 MiB/s wr, 204 op/s 2024-04-20T21:32:45.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:45.419+0000 7f3c1d600700 1 -- 172.21.15.62:0/1018018473 <== mgr.14658 v2:172.21.15.62:6800/1585760929 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7f3c20105d90 con 0x7f3c0806ea70 2024-04-20T21:32:45.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:45.423+0000 7f3c27400700 1 -- 172.21.15.62:0/1018018473 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f3c0806ea70 msgr2=0x7f3c08070f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:32:45.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:45.423+0000 7f3c27400700 1 --2- 172.21.15.62:0/1018018473 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f3c0806ea70 0x7f3c08070f20 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f3c201012a0 tx=0x7f3c14007400 comp rx=0 tx=0).stop 2024-04-20T21:32:45.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:45.423+0000 7f3c27400700 1 -- 172.21.15.62:0/1018018473 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3c20101440 msgr2=0x7f3c20195ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:32:45.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:45.423+0000 7f3c27400700 1 --2- 172.21.15.62:0/1018018473 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3c20101440 0x7f3c20195ff0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f3c0c005b40 tx=0x7f3c0c005800 comp rx=0 tx=0).stop 2024-04-20T21:32:45.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:45.423+0000 7f3c27400700 1 -- 172.21.15.62:0/1018018473 shutdown_connections 2024-04-20T21:32:45.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:45.423+0000 7f3c27400700 1 --2- 172.21.15.62:0/1018018473 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f3c0806ea70 0x7f3c08070f20 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:32:45.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:45.423+0000 7f3c27400700 1 --2- 172.21.15.62:0/1018018473 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3c20100240 0x7f3c20195ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:32:45.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:45.423+0000 7f3c27400700 1 --2- 172.21.15.62:0/1018018473 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3c20101440 0x7f3c20195ff0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:32:45.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:45.423+0000 7f3c27400700 1 -- 172.21.15.62:0/1018018473 >> 172.21.15.62:0/1018018473 conn(0x7f3c200fb7f0 msgr2=0x7f3c20104670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:32:45.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:45.423+0000 7f3c27400700 1 -- 172.21.15.62:0/1018018473 shutdown_connections 2024-04-20T21:32:45.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:45.423+0000 7f3c27400700 1 -- 172.21.15.62:0/1018018473 wait complete. 2024-04-20T21:32:45.442 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T21:32:45.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:45.634+0000 7f8740200700 1 -- 172.21.15.62:0/1919986715 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f87380723e0 msgr2=0x7f873810be10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:32:45.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:45.634+0000 7f8740200700 1 --2- 172.21.15.62:0/1919986715 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f87380723e0 0x7f873810be10 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f872c009910 tx=0x7f872c009c20 comp rx=0 tx=0).stop 2024-04-20T21:32:45.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:45.634+0000 7f8740200700 1 -- 172.21.15.62:0/1919986715 shutdown_connections 2024-04-20T21:32:45.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:45.634+0000 7f8740200700 1 --2- 172.21.15.62:0/1919986715 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f87380723e0 0x7f873810be10 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:32:45.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:45.634+0000 7f8740200700 1 --2- 172.21.15.62:0/1919986715 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8738071a00 0x7f8738071e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:32:45.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:45.634+0000 7f8740200700 1 -- 172.21.15.62:0/1919986715 >> 172.21.15.62:0/1919986715 conn(0x7f873806d110 msgr2=0x7f873806f560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:32:45.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:45.635+0000 7f8740200700 1 -- 172.21.15.62:0/1919986715 shutdown_connections 2024-04-20T21:32:45.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:45.635+0000 7f8740200700 1 -- 172.21.15.62:0/1919986715 wait complete. 2024-04-20T21:32:45.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:45.636+0000 7f8740200700 1 Processor -- start 2024-04-20T21:32:45.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:45.638+0000 7f8740200700 1 -- start start 2024-04-20T21:32:45.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:45.639+0000 7f8740200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8738071a00 0x7f87381a4870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:32:45.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:45.639+0000 7f8740200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87380723e0 0x7f87381a4db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:32:45.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:45.639+0000 7f8740200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87381a53d0 con 0x7f8738071a00 2024-04-20T21:32:45.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:45.639+0000 7f8740200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87381a5510 con 0x7f87380723e0 2024-04-20T21:32:45.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:45.640+0000 7f873d800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8738071a00 0x7f87381a4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:32:45.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:45.640+0000 7f873ce00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87380723e0 0x7f87381a4db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:32:45.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:45.640+0000 7f873ce00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87380723e0 0x7f87381a4db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.62:60394/0 (socket says 172.21.15.62:60394) 2024-04-20T21:32:45.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:45.640+0000 7f873ce00700 1 -- 172.21.15.62:0/3050101459 learned_addr learned my addr 172.21.15.62:0/3050101459 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:32:45.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:45.640+0000 7f873ce00700 1 -- 172.21.15.62:0/3050101459 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8738071a00 msgr2=0x7f87381a4870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:32:45.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:45.640+0000 7f873ce00700 1 --2- 172.21.15.62:0/3050101459 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8738071a00 0x7f87381a4870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:32:45.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:45.640+0000 7f873ce00700 1 -- 172.21.15.62:0/3050101459 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f872c0095f0 con 0x7f87380723e0 2024-04-20T21:32:45.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:45.641+0000 7f873ce00700 1 --2- 172.21.15.62:0/3050101459 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87380723e0 0x7f87381a4db0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f872c00fa00 tx=0x7f872c00fae0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:32:45.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:45.642+0000 7f8736000700 1 -- 172.21.15.62:0/3050101459 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f872c01d070 con 0x7f87380723e0 2024-04-20T21:32:45.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:45.643+0000 7f8736000700 1 -- 172.21.15.62:0/3050101459 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f872c005640 con 0x7f87380723e0 2024-04-20T21:32:45.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:45.643+0000 7f8736000700 1 -- 172.21.15.62:0/3050101459 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f872c017630 con 0x7f87380723e0 2024-04-20T21:32:45.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:45.643+0000 7f8740200700 1 -- 172.21.15.62:0/3050101459 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87381a9f60 con 0x7f87380723e0 2024-04-20T21:32:45.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:45.644+0000 7f8740200700 1 -- 172.21.15.62:0/3050101459 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87381aa420 con 0x7f87380723e0 2024-04-20T21:32:45.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:45.645+0000 7f8740200700 1 -- 172.21.15.62:0/3050101459 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f873819ea80 con 0x7f87380723e0 2024-04-20T21:32:45.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:45.648+0000 7f8736000700 1 -- 172.21.15.62:0/3050101459 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 29) v1 ==== 92700+0+0 (secure 0 0 0) 0x7f872c003680 con 0x7f87380723e0 2024-04-20T21:32:45.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:45.649+0000 7f8736000700 1 --2- 172.21.15.62:0/3050101459 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f872006eac0 0x7f8720070f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:32:45.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:45.649+0000 7f8736000700 1 -- 172.21.15.62:0/3050101459 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f872c0900c0 con 0x7f87380723e0 2024-04-20T21:32:45.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:45.650+0000 7f873d800700 1 --2- 172.21.15.62:0/3050101459 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f872006eac0 0x7f8720070f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:32:45.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:45.651+0000 7f873d800700 1 --2- 172.21.15.62:0/3050101459 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f872006eac0 0x7f8720070f70 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f8724009bf0 tx=0x7f8724009260 comp rx=0 tx=0).ready entity=mgr.14658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:32:45.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:45.658+0000 7f8736000700 1 -- 172.21.15.62:0/3050101459 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187527 (secure 0 0 0) 0x7f872c05cdd0 con 0x7f87380723e0 2024-04-20T21:32:46.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:46.023+0000 7f8740200700 1 -- 172.21.15.62:0/3050101459 --> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8738061190 con 0x7f872006eac0 2024-04-20T21:32:46.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:46.026+0000 7f8736000700 1 -- 172.21.15.62:0/3050101459 <== mgr.14658 v2:172.21.15.62:6800/1585760929 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7f8738061190 con 0x7f872006eac0 2024-04-20T21:32:46.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:46.029+0000 7f8740200700 1 -- 172.21.15.62:0/3050101459 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f872006eac0 msgr2=0x7f8720070f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:32:46.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:46.029+0000 7f8740200700 1 --2- 172.21.15.62:0/3050101459 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f872006eac0 0x7f8720070f70 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f8724009bf0 tx=0x7f8724009260 comp rx=0 tx=0).stop 2024-04-20T21:32:46.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:46.029+0000 7f8740200700 1 -- 172.21.15.62:0/3050101459 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87380723e0 msgr2=0x7f87381a4db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:32:46.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:46.029+0000 7f8740200700 1 --2- 172.21.15.62:0/3050101459 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87380723e0 0x7f87381a4db0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f872c00fa00 tx=0x7f872c00fae0 comp rx=0 tx=0).stop 2024-04-20T21:32:46.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:46.029+0000 7f8740200700 1 -- 172.21.15.62:0/3050101459 shutdown_connections 2024-04-20T21:32:46.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:46.029+0000 7f8740200700 1 --2- 172.21.15.62:0/3050101459 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f872006eac0 0x7f8720070f70 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:32:46.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:46.029+0000 7f8740200700 1 --2- 172.21.15.62:0/3050101459 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8738071a00 0x7f87381a4870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:32:46.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:46.029+0000 7f8740200700 1 --2- 172.21.15.62:0/3050101459 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87380723e0 0x7f87381a4db0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:32:46.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:46.029+0000 7f8740200700 1 -- 172.21.15.62:0/3050101459 >> 172.21.15.62:0/3050101459 conn(0x7f873806d110 msgr2=0x7f873810b410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:32:46.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:46.030+0000 7f8740200700 1 -- 172.21.15.62:0/3050101459 shutdown_connections 2024-04-20T21:32:46.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:46.030+0000 7f8740200700 1 -- 172.21.15.62:0/3050101459 wait complete. 2024-04-20T21:32:46.145 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-20T21:32:46.145 DEBUG:teuthology.orchestra.run.smithi062:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-20T21:32:46.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:46.216+0000 7fb1c8a00700 1 -- 172.21.15.62:0/1160818340 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb1c4100ab0 msgr2=0x7fb1c4102e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:32:46.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:46.216+0000 7fb1c8a00700 1 --2- 172.21.15.62:0/1160818340 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb1c4100ab0 0x7fb1c4102e90 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fb1b4009910 tx=0x7fb1b4009c20 comp rx=0 tx=0).stop 2024-04-20T21:32:46.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:46.216+0000 7fb1c8a00700 1 -- 172.21.15.62:0/1160818340 shutdown_connections 2024-04-20T21:32:46.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:46.216+0000 7fb1c8a00700 1 --2- 172.21.15.62:0/1160818340 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb1c41033d0 0x7fb1c41057b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:32:46.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:46.216+0000 7fb1c8a00700 1 --2- 172.21.15.62:0/1160818340 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb1c4100ab0 0x7fb1c4102e90 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:32:46.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:46.216+0000 7fb1c8a00700 1 -- 172.21.15.62:0/1160818340 >> 172.21.15.62:0/1160818340 conn(0x7fb1c40fa4d0 msgr2=0x7fb1c40fc900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:32:46.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:46.216+0000 7fb1c8a00700 1 -- 172.21.15.62:0/1160818340 shutdown_connections 2024-04-20T21:32:46.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:46.216+0000 7fb1c8a00700 1 -- 172.21.15.62:0/1160818340 wait complete. 2024-04-20T21:32:46.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:46.217+0000 7fb1c8a00700 1 Processor -- start 2024-04-20T21:32:46.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:46.217+0000 7fb1c8a00700 1 -- start start 2024-04-20T21:32:46.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:46.218+0000 7fb1c8a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb1c4100ab0 0x7fb1c4100410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:32:46.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:46.218+0000 7fb1c8a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb1c41033d0 0x7fb1c40fea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:32:46.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:46.218+0000 7fb1c8a00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1c40fefa0 con 0x7fb1c4100ab0 2024-04-20T21:32:46.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:46.218+0000 7fb1c8a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1c40ff0e0 con 0x7fb1c41033d0 2024-04-20T21:32:46.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:46.218+0000 7fb1c3400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb1c4100ab0 0x7fb1c4100410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:32:46.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:46.218+0000 7fb1c3400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb1c4100ab0 0x7fb1c4100410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34372/0 (socket says 172.21.15.62:34372) 2024-04-20T21:32:46.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:46.218+0000 7fb1c3400700 1 -- 172.21.15.62:0/3027542871 learned_addr learned my addr 172.21.15.62:0/3027542871 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:32:46.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:46.218+0000 7fb1c2a00700 1 --2- 172.21.15.62:0/3027542871 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb1c41033d0 0x7fb1c40fea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:32:46.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:46.218+0000 7fb1c3400700 1 -- 172.21.15.62:0/3027542871 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb1c41033d0 msgr2=0x7fb1c40fea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:32:46.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:46.218+0000 7fb1c3400700 1 --2- 172.21.15.62:0/3027542871 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb1c41033d0 0x7fb1c40fea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:32:46.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:46.218+0000 7fb1c3400700 1 -- 172.21.15.62:0/3027542871 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1b40095f0 con 0x7fb1c4100ab0 2024-04-20T21:32:46.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:46.219+0000 7fb1c3400700 1 --2- 172.21.15.62:0/3027542871 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb1c4100ab0 0x7fb1c4100410 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fb1b4000c00 tx=0x7fb1b400f740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:32:46.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:46.219+0000 7fb1bbe00700 1 -- 172.21.15.62:0/3027542871 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb1b401d070 con 0x7fb1c4100ab0 2024-04-20T21:32:46.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:46.219+0000 7fb1bbe00700 1 -- 172.21.15.62:0/3027542871 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fb1b4005dc0 con 0x7fb1c4100ab0 2024-04-20T21:32:46.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:46.219+0000 7fb1bbe00700 1 -- 172.21.15.62:0/3027542871 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb1b40226a0 con 0x7fb1c4100ab0 2024-04-20T21:32:46.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:46.219+0000 7fb1c8a00700 1 -- 172.21.15.62:0/3027542871 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1c40ff360 con 0x7fb1c4100ab0 2024-04-20T21:32:46.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:46.219+0000 7fb1c8a00700 1 -- 172.21.15.62:0/3027542871 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1c40ff850 con 0x7fb1c4100ab0 2024-04-20T21:32:46.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:46.221+0000 7fb1c8a00700 1 -- 172.21.15.62:0/3027542871 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1c418fa40 con 0x7fb1c4100ab0 2024-04-20T21:32:46.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:46.222+0000 7fb1bbe00700 1 -- 172.21.15.62:0/3027542871 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 29) v1 ==== 92700+0+0 (secure 0 0 0) 0x7fb1b4003680 con 0x7fb1c4100ab0 2024-04-20T21:32:46.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:46.223+0000 7fb1bbe00700 1 --2- 172.21.15.62:0/3027542871 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7fb1a806ea70 0x7fb1a8070f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:32:46.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:46.223+0000 7fb1bbe00700 1 -- 172.21.15.62:0/3027542871 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6070+0+0 (secure 0 0 0) 0x7fb1b40900c0 con 0x7fb1c4100ab0 2024-04-20T21:32:46.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:46.223+0000 7fb1c2a00700 1 --2- 172.21.15.62:0/3027542871 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7fb1a806ea70 0x7fb1a8070f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:32:46.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:46.225+0000 7fb1c2a00700 1 --2- 172.21.15.62:0/3027542871 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7fb1a806ea70 0x7fb1a8070f20 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fb1b0009bf0 tx=0x7fb1b0009260 comp rx=0 tx=0).ready entity=mgr.14658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:32:46.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:46.229+0000 7fb1bbe00700 1 -- 172.21.15.62:0/3027542871 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187527 (secure 0 0 0) 0x7fb1b405cc90 con 0x7fb1c4100ab0 2024-04-20T21:32:46.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:46.533+0000 7fb1c8a00700 1 -- 172.21.15.62:0/3027542871 --> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fb1c4061190 con 0x7fb1a806ea70 2024-04-20T21:32:46.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:46.552+0000 7fb1bbe00700 1 -- 172.21.15.62:0/3027542871 <== mgr.14658 v2:172.21.15.62:6800/1585760929 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fb1c4061190 con 0x7fb1a806ea70 2024-04-20T21:32:46.551 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T21:32:46.551 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (10m) 1s ago 12m 14.1M - 0.25.0 c8568f914cd2 957f663a9cd7 2024-04-20T21:32:46.552 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (12m) 1s ago 12m 8028k - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T21:32:46.552 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (11m) 5s ago 11m 7834k - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T21:32:46.552 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (12m) 1s ago 12m 6656k - 18.2.0 dc2bc1663786 36e10ef5be39 2024-04-20T21:32:46.552 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (11m) 5s ago 11m 6665k - 18.2.0 dc2bc1663786 4101f9953e3d 2024-04-20T21:32:46.552 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (10m) 1s ago 12m 75.1M - 9.4.7 954c08fa6188 99363bb095d8 2024-04-20T21:32:46.552 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (5m) 1s ago 5m 182M - 18.2.0 dc2bc1663786 5e6460aa6159 2024-04-20T21:32:46.552 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (5m) 1s ago 5m 182M - 18.2.0 dc2bc1663786 e65f2267544e 2024-04-20T21:32:46.552 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (5m) 5s ago 5m 149M - 18.2.0 dc2bc1663786 fb2a8a223db8 2024-04-20T21:32:46.552 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (5m) 5s ago 5m 116M - 18.2.0 dc2bc1663786 f2eec2ee8951 2024-04-20T21:32:46.552 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (90s) 1s ago 14m 494M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T21:32:46.552 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (11s) 5s ago 11m 57.9M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T21:32:46.552 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (14m) 1s ago 14m 53.7M 2048M 18.2.0 dc2bc1663786 bd6efbd5c380 2024-04-20T21:32:46.552 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (11m) 5s ago 11m 45.1M 2048M 18.2.0 dc2bc1663786 3735c8e5db9a 2024-04-20T21:32:46.552 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (12m) 1s ago 12m 8342k - 1.5.0 0da6a335fe13 fc746d8ae72e 2024-04-20T21:32:46.553 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (11m) 5s ago 11m 7994k - 1.5.0 0da6a335fe13 db03af85b286 2024-04-20T21:32:46.553 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (10m) 1s ago 10m 194M 1216M 18.2.0 dc2bc1663786 3f315c3afc1f 2024-04-20T21:32:46.553 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (9m) 1s ago 9m 205M 1216M 18.2.0 dc2bc1663786 44e4d07416ad 2024-04-20T21:32:46.553 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (8m) 1s ago 8m 159M 1216M 18.2.0 dc2bc1663786 31c5d9586e98 2024-04-20T21:32:46.553 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (8m) 5s ago 8m 232M 2240M 18.2.0 dc2bc1663786 611c1e242693 2024-04-20T21:32:46.553 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (7m) 5s ago 7m 197M 2240M 18.2.0 dc2bc1663786 6c8fce91d45d 2024-04-20T21:32:46.553 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (7m) 5s ago 7m 212M 2240M 18.2.0 dc2bc1663786 8e4574bae8db 2024-04-20T21:32:46.553 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (20s) 1s ago 11m 55.2M - 2.43.0 a07b618ecd1d 3e50e774ad4e 2024-04-20T21:32:46.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:46.555+0000 7fb1c8a00700 1 -- 172.21.15.62:0/3027542871 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7fb1a806ea70 msgr2=0x7fb1a8070f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:32:46.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:46.555+0000 7fb1c8a00700 1 --2- 172.21.15.62:0/3027542871 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7fb1a806ea70 0x7fb1a8070f20 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fb1b0009bf0 tx=0x7fb1b0009260 comp rx=0 tx=0).stop 2024-04-20T21:32:46.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:46.555+0000 7fb1c8a00700 1 -- 172.21.15.62:0/3027542871 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb1c4100ab0 msgr2=0x7fb1c4100410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:32:46.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:46.555+0000 7fb1c8a00700 1 --2- 172.21.15.62:0/3027542871 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb1c4100ab0 0x7fb1c4100410 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fb1b4000c00 tx=0x7fb1b400f740 comp rx=0 tx=0).stop 2024-04-20T21:32:46.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:46.555+0000 7fb1c8a00700 1 -- 172.21.15.62:0/3027542871 shutdown_connections 2024-04-20T21:32:46.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:46.555+0000 7fb1c8a00700 1 --2- 172.21.15.62:0/3027542871 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7fb1a806ea70 0x7fb1a8070f20 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:32:46.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:46.555+0000 7fb1c8a00700 1 --2- 172.21.15.62:0/3027542871 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb1c4100ab0 0x7fb1c4100410 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:32:46.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:46.555+0000 7fb1c8a00700 1 --2- 172.21.15.62:0/3027542871 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb1c41033d0 0x7fb1c40fea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:32:46.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:46.555+0000 7fb1c8a00700 1 -- 172.21.15.62:0/3027542871 >> 172.21.15.62:0/3027542871 conn(0x7fb1c40fa4d0 msgr2=0x7fb1c40fc900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:32:46.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:46.555+0000 7fb1c8a00700 1 -- 172.21.15.62:0/3027542871 shutdown_connections 2024-04-20T21:32:46.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:46.556+0000 7fb1c8a00700 1 -- 172.21.15.62:0/3027542871 wait complete. 2024-04-20T21:32:46.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:46 smithi189 ceph-mon[21562]: from='client.24499 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:32:46.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:46 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:32:46.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:46 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:32:46.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:46 smithi062 ceph-mon[16221]: from='client.24499 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:32:46.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:46 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:32:46.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:46 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:32:46.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:46.790+0000 7f478ce00700 1 -- 172.21.15.62:0/3739243920 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4788071e00 msgr2=0x7f4788072210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:32:46.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:46.790+0000 7f478ce00700 1 --2- 172.21.15.62:0/3739243920 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4788071e00 0x7f4788072210 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f4778009910 tx=0x7f4778009c20 comp rx=0 tx=0).stop 2024-04-20T21:32:46.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:46.790+0000 7f478ce00700 1 -- 172.21.15.62:0/3739243920 shutdown_connections 2024-04-20T21:32:46.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:46.790+0000 7f478ce00700 1 --2- 172.21.15.62:0/3739243920 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4788072750 0x7f4788107d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:32:46.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:46.790+0000 7f478ce00700 1 --2- 172.21.15.62:0/3739243920 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4788071e00 0x7f4788072210 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:32:46.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:46.790+0000 7f478ce00700 1 -- 172.21.15.62:0/3739243920 >> 172.21.15.62:0/3739243920 conn(0x7f478806d110 msgr2=0x7f478806f560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:32:46.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:46.791+0000 7f478ce00700 1 -- 172.21.15.62:0/3739243920 shutdown_connections 2024-04-20T21:32:46.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:46.791+0000 7f478ce00700 1 -- 172.21.15.62:0/3739243920 wait complete. 2024-04-20T21:32:46.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:46.793+0000 7f478ce00700 1 Processor -- start 2024-04-20T21:32:46.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:46.793+0000 7f478ce00700 1 -- start start 2024-04-20T21:32:46.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:46.793+0000 7f478ce00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4788071e00 0x7f47881a04e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:32:46.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:46.793+0000 7f478ce00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4788072750 0x7f47881a0a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:32:46.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:46.793+0000 7f478ce00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47881a1040 con 0x7f4788071e00 2024-04-20T21:32:46.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:46.793+0000 7f478ce00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47881a1180 con 0x7f4788072750 2024-04-20T21:32:46.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:46.794+0000 7f4785400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4788072750 0x7f47881a0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:32:46.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:46.794+0000 7f4785e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4788071e00 0x7f47881a04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:32:46.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:46.794+0000 7f4785400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4788072750 0x7f47881a0a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.62:60418/0 (socket says 172.21.15.62:60418) 2024-04-20T21:32:46.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:46.794+0000 7f4785400700 1 -- 172.21.15.62:0/3746015701 learned_addr learned my addr 172.21.15.62:0/3746015701 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:32:46.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:46.795+0000 7f4785e00700 1 -- 172.21.15.62:0/3746015701 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4788072750 msgr2=0x7f47881a0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:32:46.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:46.795+0000 7f4785e00700 1 --2- 172.21.15.62:0/3746015701 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4788072750 0x7f47881a0a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:32:46.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:46.795+0000 7f4785e00700 1 -- 172.21.15.62:0/3746015701 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47780095f0 con 0x7f4788071e00 2024-04-20T21:32:46.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:46.795+0000 7f4785e00700 1 --2- 172.21.15.62:0/3746015701 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4788071e00 0x7f47881a04e0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f4778000c00 tx=0x7f477800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:32:46.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:46.796+0000 7f477ea00700 1 -- 172.21.15.62:0/3746015701 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f477801d070 con 0x7f4788071e00 2024-04-20T21:32:46.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:46.796+0000 7f477ea00700 1 -- 172.21.15.62:0/3746015701 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f4778005dc0 con 0x7f4788071e00 2024-04-20T21:32:46.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:46.797+0000 7f477ea00700 1 -- 172.21.15.62:0/3746015701 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4778021410 con 0x7f4788071e00 2024-04-20T21:32:46.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:46.797+0000 7f478ce00700 1 -- 172.21.15.62:0/3746015701 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47881a5bd0 con 0x7f4788071e00 2024-04-20T21:32:46.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:46.797+0000 7f478ce00700 1 -- 172.21.15.62:0/3746015701 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47881a60c0 con 0x7f4788071e00 2024-04-20T21:32:46.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:46.799+0000 7f478ce00700 1 -- 172.21.15.62:0/3746015701 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f478810dd30 con 0x7f4788071e00 2024-04-20T21:32:46.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:46.803+0000 7f477ea00700 1 -- 172.21.15.62:0/3746015701 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 29) v1 ==== 92700+0+0 (secure 0 0 0) 0x7f477800f7d0 con 0x7f4788071e00 2024-04-20T21:32:46.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:46.804+0000 7f477ea00700 1 --2- 172.21.15.62:0/3746015701 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f476c06ea70 0x7f476c070f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:32:46.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:46.805+0000 7f477ea00700 1 -- 172.21.15.62:0/3746015701 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f4778090040 con 0x7f4788071e00 2024-04-20T21:32:46.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:46.805+0000 7f4785400700 1 --2- 172.21.15.62:0/3746015701 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f476c06ea70 0x7f476c070f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:32:46.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:46.808+0000 7f4785400700 1 --2- 172.21.15.62:0/3746015701 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f476c06ea70 0x7f476c070f20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f477400bd10 tx=0x7f477400b480 comp rx=0 tx=0).ready entity=mgr.14658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:32:46.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:46.813+0000 7f477ea00700 1 -- 172.21.15.62:0/3746015701 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187527 (secure 0 0 0) 0x7f477805cd50 con 0x7f4788071e00 2024-04-20T21:32:47.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:47.217+0000 7f478ce00700 1 -- 172.21.15.62:0/3746015701 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f478802cea0 con 0x7f4788071e00 2024-04-20T21:32:47.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:47.218+0000 7f477ea00700 1 -- 172.21.15.62:0/3746015701 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+675 (secure 0 0 0) 0x7f477800f460 con 0x7f4788071e00 2024-04-20T21:32:47.217 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:32:47.217 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T21:32:47.217 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 2 2024-04-20T21:32:47.217 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:32:47.217 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T21:32:47.217 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:32:47.218 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:32:47.218 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T21:32:47.218 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 6 2024-04-20T21:32:47.218 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:32:47.218 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T21:32:47.218 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 4 2024-04-20T21:32:47.218 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:32:47.218 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T21:32:47.218 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 12, 2024-04-20T21:32:47.218 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:32:47.218 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T21:32:47.218 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:32:47.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:47.221+0000 7f478ce00700 1 -- 172.21.15.62:0/3746015701 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f476c06ea70 msgr2=0x7f476c070f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:32:47.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:47.221+0000 7f478ce00700 1 --2- 172.21.15.62:0/3746015701 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f476c06ea70 0x7f476c070f20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f477400bd10 tx=0x7f477400b480 comp rx=0 tx=0).stop 2024-04-20T21:32:47.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:47.221+0000 7f478ce00700 1 -- 172.21.15.62:0/3746015701 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4788071e00 msgr2=0x7f47881a04e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:32:47.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:47.221+0000 7f478ce00700 1 --2- 172.21.15.62:0/3746015701 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4788071e00 0x7f47881a04e0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f4778000c00 tx=0x7f477800bd70 comp rx=0 tx=0).stop 2024-04-20T21:32:47.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:47.221+0000 7f478ce00700 1 -- 172.21.15.62:0/3746015701 shutdown_connections 2024-04-20T21:32:47.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:47.221+0000 7f478ce00700 1 --2- 172.21.15.62:0/3746015701 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f476c06ea70 0x7f476c070f20 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:32:47.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:47.221+0000 7f478ce00700 1 --2- 172.21.15.62:0/3746015701 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4788071e00 0x7f47881a04e0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:32:47.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:47.221+0000 7f478ce00700 1 --2- 172.21.15.62:0/3746015701 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4788072750 0x7f47881a0a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:32:47.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:47.221+0000 7f478ce00700 1 -- 172.21.15.62:0/3746015701 >> 172.21.15.62:0/3746015701 conn(0x7f478806d110 msgr2=0x7f478810ae70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:32:47.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:47.222+0000 7f478ce00700 1 -- 172.21.15.62:0/3746015701 shutdown_connections 2024-04-20T21:32:47.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:47.222+0000 7f478ce00700 1 -- 172.21.15.62:0/3746015701 wait complete. 2024-04-20T21:32:47.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:47.412+0000 7ff824c00700 1 -- 172.21.15.62:0/658691022 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff820103480 msgr2=0x7ff8201038d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:32:47.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:47.412+0000 7ff824c00700 1 --2- 172.21.15.62:0/658691022 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff820103480 0x7ff8201038d0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7ff80800b0a0 tx=0x7ff80800b3b0 comp rx=0 tx=0).stop 2024-04-20T21:32:47.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:47.413+0000 7ff824c00700 1 -- 172.21.15.62:0/658691022 shutdown_connections 2024-04-20T21:32:47.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:47.413+0000 7ff824c00700 1 --2- 172.21.15.62:0/658691022 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff820103480 0x7ff8201038d0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:32:47.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:47.413+0000 7ff824c00700 1 --2- 172.21.15.62:0/658691022 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff820102280 0x7ff820102690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:32:47.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:47.413+0000 7ff824c00700 1 -- 172.21.15.62:0/658691022 >> 172.21.15.62:0/658691022 conn(0x7ff8200fd8d0 msgr2=0x7ff8200ffcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:32:47.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:47.413+0000 7ff824c00700 1 -- 172.21.15.62:0/658691022 shutdown_connections 2024-04-20T21:32:47.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:47.413+0000 7ff824c00700 1 -- 172.21.15.62:0/658691022 wait complete. 2024-04-20T21:32:47.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:47.414+0000 7ff824c00700 1 Processor -- start 2024-04-20T21:32:47.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:47.414+0000 7ff824c00700 1 -- start start 2024-04-20T21:32:47.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:47.415+0000 7ff824c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff820102280 0x7ff820197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:32:47.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:47.415+0000 7ff824c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff820103480 0x7ff820197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:32:47.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:47.415+0000 7ff824c00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff820198600 con 0x7ff820102280 2024-04-20T21:32:47.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:47.415+0000 7ff824c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff820198740 con 0x7ff820103480 2024-04-20T21:32:47.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:47.415+0000 7ff81f400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff820102280 0x7ff820197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:32:47.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:47.415+0000 7ff81f400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff820102280 0x7ff820197aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34414/0 (socket says 172.21.15.62:34414) 2024-04-20T21:32:47.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:47.415+0000 7ff81f400700 1 -- 172.21.15.62:0/846269783 learned_addr learned my addr 172.21.15.62:0/846269783 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:32:47.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:47.415+0000 7ff817e00700 1 --2- 172.21.15.62:0/846269783 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff820103480 0x7ff820197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:32:47.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:47.415+0000 7ff81f400700 1 -- 172.21.15.62:0/846269783 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff820103480 msgr2=0x7ff820197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:32:47.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:47.415+0000 7ff81f400700 1 --2- 172.21.15.62:0/846269783 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff820103480 0x7ff820197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:32:47.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:47.415+0000 7ff81f400700 1 -- 172.21.15.62:0/846269783 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff808009d00 con 0x7ff820102280 2024-04-20T21:32:47.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:47.416+0000 7ff81f400700 1 --2- 172.21.15.62:0/846269783 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff820102280 0x7ff820197aa0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7ff81000d900 tx=0x7ff81000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:32:47.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:47.416+0000 7ff81cc00700 1 -- 172.21.15.62:0/846269783 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff8100041f0 con 0x7ff820102280 2024-04-20T21:32:47.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:47.416+0000 7ff81cc00700 1 -- 172.21.15.62:0/846269783 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff810004350 con 0x7ff820102280 2024-04-20T21:32:47.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:47.416+0000 7ff81cc00700 1 -- 172.21.15.62:0/846269783 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff810003da0 con 0x7ff820102280 2024-04-20T21:32:47.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:47.416+0000 7ff824c00700 1 -- 172.21.15.62:0/846269783 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff82019d1f0 con 0x7ff820102280 2024-04-20T21:32:47.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:47.416+0000 7ff824c00700 1 -- 172.21.15.62:0/846269783 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff82019d7c0 con 0x7ff820102280 2024-04-20T21:32:47.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:47.418+0000 7ff824c00700 1 -- 172.21.15.62:0/846269783 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff82010add0 con 0x7ff820102280 2024-04-20T21:32:47.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:47.419+0000 7ff81cc00700 1 -- 172.21.15.62:0/846269783 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 29) v1 ==== 92700+0+0 (secure 0 0 0) 0x7ff810009730 con 0x7ff820102280 2024-04-20T21:32:47.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:47.420+0000 7ff81cc00700 1 --2- 172.21.15.62:0/846269783 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7ff80406ea70 0x7ff804070f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:32:47.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:47.420+0000 7ff81cc00700 1 -- 172.21.15.62:0/846269783 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6070+0+0 (secure 0 0 0) 0x7ff81008ec70 con 0x7ff820102280 2024-04-20T21:32:47.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:47.421+0000 7ff817e00700 1 --2- 172.21.15.62:0/846269783 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7ff80406ea70 0x7ff804070f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:32:47.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:47.421+0000 7ff817e00700 1 --2- 172.21.15.62:0/846269783 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7ff80406ea70 0x7ff804070f20 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7ff808000f80 tx=0x7ff80800b7f0 comp rx=0 tx=0).ready entity=mgr.14658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:32:47.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:47.425+0000 7ff81cc00700 1 -- 172.21.15.62:0/846269783 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187527 (secure 0 0 0) 0x7ff810095080 con 0x7ff820102280 2024-04-20T21:32:47.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:47 smithi062 ceph-mon[16221]: from='client.24503 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:32:47.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:47 smithi062 ceph-mon[16221]: from='client.14740 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:32:47.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:47 smithi062 ceph-mon[16221]: pgmap v38: 65 pgs: 65 active+clean; 2.0 GiB data, 6.3 GiB used, 530 GiB / 536 GiB avail; 977 KiB/s rd, 969 KiB/s wr, 136 op/s 2024-04-20T21:32:47.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:47 smithi062 ceph-mon[16221]: from='client.? 172.21.15.62:0/3746015701' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:32:47.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:47.822+0000 7ff824c00700 1 -- 172.21.15.62:0/846269783 --> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff820107dd0 con 0x7ff80406ea70 2024-04-20T21:32:47.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:47.824+0000 7ff81cc00700 1 -- 172.21.15.62:0/846269783 <== mgr.14658 v2:172.21.15.62:6800/1585760929 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7ff820107dd0 con 0x7ff80406ea70 2024-04-20T21:32:47.825 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:32:47.825 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T21:32:47.825 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T21:32:47.825 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-20T21:32:47.825 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T21:32:47.825 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr" 2024-04-20T21:32:47.825 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T21:32:47.825 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "2/2 daemons upgraded", 2024-04-20T21:32:47.825 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading mgr daemons", 2024-04-20T21:32:47.825 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T21:32:47.825 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:32:47.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:47.827+0000 7ff824c00700 1 -- 172.21.15.62:0/846269783 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7ff80406ea70 msgr2=0x7ff804070f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:32:47.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:47.827+0000 7ff824c00700 1 --2- 172.21.15.62:0/846269783 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7ff80406ea70 0x7ff804070f20 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7ff808000f80 tx=0x7ff80800b7f0 comp rx=0 tx=0).stop 2024-04-20T21:32:47.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:47.827+0000 7ff824c00700 1 -- 172.21.15.62:0/846269783 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff820102280 msgr2=0x7ff820197aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:32:47.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:47.827+0000 7ff824c00700 1 --2- 172.21.15.62:0/846269783 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff820102280 0x7ff820197aa0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7ff81000d900 tx=0x7ff81000dcc0 comp rx=0 tx=0).stop 2024-04-20T21:32:47.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:47.827+0000 7ff824c00700 1 -- 172.21.15.62:0/846269783 shutdown_connections 2024-04-20T21:32:47.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:47.827+0000 7ff824c00700 1 --2- 172.21.15.62:0/846269783 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7ff80406ea70 0x7ff804070f20 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:32:47.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:47.827+0000 7ff824c00700 1 --2- 172.21.15.62:0/846269783 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff820102280 0x7ff820197aa0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:32:47.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:47.827+0000 7ff824c00700 1 --2- 172.21.15.62:0/846269783 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff820103480 0x7ff820197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:32:47.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:47.827+0000 7ff824c00700 1 -- 172.21.15.62:0/846269783 >> 172.21.15.62:0/846269783 conn(0x7ff8200fd8d0 msgr2=0x7ff8201066b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:32:47.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:47.828+0000 7ff824c00700 1 -- 172.21.15.62:0/846269783 shutdown_connections 2024-04-20T21:32:47.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:32:47.828+0000 7ff824c00700 1 -- 172.21.15.62:0/846269783 wait complete. 2024-04-20T21:32:47.982 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:47 smithi189 ceph-mon[21562]: from='client.24503 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:32:47.982 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:47 smithi189 ceph-mon[21562]: from='client.14740 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:32:47.982 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:47 smithi189 ceph-mon[21562]: pgmap v38: 65 pgs: 65 active+clean; 2.0 GiB data, 6.3 GiB used, 530 GiB / 536 GiB avail; 977 KiB/s rd, 969 KiB/s wr, 136 op/s 2024-04-20T21:32:47.982 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:47 smithi189 ceph-mon[21562]: from='client.? 172.21.15.62:0/3746015701' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:32:49.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:49 smithi189 ceph-mon[21562]: from='client.14748 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:32:49.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:49 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:32:49.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:49 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:32:49.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:49 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:32:49.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:49 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T21:32:49.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:49 smithi189 ceph-mon[21562]: Standby manager daemon smithi189.tawavz restarted 2024-04-20T21:32:49.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:49 smithi189 ceph-mon[21562]: Standby manager daemon smithi189.tawavz started 2024-04-20T21:32:49.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:49 smithi189 ceph-mon[21562]: from='mgr.? 172.21.15.189:0/1460018821' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi189.tawavz/crt"}]: dispatch 2024-04-20T21:32:49.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:49 smithi189 ceph-mon[21562]: from='mgr.? 172.21.15.189:0/1460018821' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-20T21:32:49.489 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:49 smithi189 ceph-mon[21562]: from='mgr.? 172.21.15.189:0/1460018821' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi189.tawavz/key"}]: dispatch 2024-04-20T21:32:49.489 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:49 smithi189 ceph-mon[21562]: from='mgr.? 172.21.15.189:0/1460018821' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-20T21:32:49.489 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:49 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:32:49.489 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:49 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:32:49.489 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:49 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:32:49.489 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:49 smithi189 ceph-mon[21562]: Upgrade: Setting container_image for all mgr 2024-04-20T21:32:49.489 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:49 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:32:49.489 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:49 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi062.rqxuxi"}]: dispatch 2024-04-20T21:32:49.489 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:49 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi062.rqxuxi"}]': finished 2024-04-20T21:32:49.489 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:49 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi189.tawavz"}]: dispatch 2024-04-20T21:32:49.489 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:49 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi189.tawavz"}]': finished 2024-04-20T21:32:49.489 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:49 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:32:49.489 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:49 smithi189 ceph-mon[21562]: Upgrade: Setting container_image for all rgw 2024-04-20T21:32:49.489 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:49 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:32:49.490 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:49 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:32:49.490 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:49 smithi189 ceph-mon[21562]: Upgrade: Setting container_image for all rbd-mirror 2024-04-20T21:32:49.490 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:49 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:32:49.490 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:49 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:32:49.490 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:49 smithi189 ceph-mon[21562]: Upgrade: Setting container_image for all cephfs-mirror 2024-04-20T21:32:49.490 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:49 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:32:49.490 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:49 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:32:49.490 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:49 smithi189 ceph-mon[21562]: Upgrade: Setting container_image for all iscsi 2024-04-20T21:32:49.490 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:49 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:32:49.490 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:49 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:32:49.490 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:49 smithi189 ceph-mon[21562]: Upgrade: Setting container_image for all nfs 2024-04-20T21:32:49.490 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:49 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:32:49.490 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:49 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:32:49.490 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:49 smithi189 ceph-mon[21562]: Upgrade: Setting container_image for all nvmeof 2024-04-20T21:32:49.491 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:49 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:32:49.491 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:49 smithi189 ceph-mon[21562]: pgmap v39: 65 pgs: 65 active+clean; 1.1 GiB data, 3.6 GiB used, 533 GiB / 536 GiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 200 op/s 2024-04-20T21:32:49.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:49 smithi062 ceph-mon[16221]: from='client.14748 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:32:49.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:49 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:32:49.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:49 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:32:49.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:49 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:32:49.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:49 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T21:32:49.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:49 smithi062 ceph-mon[16221]: Standby manager daemon smithi189.tawavz restarted 2024-04-20T21:32:49.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:49 smithi062 ceph-mon[16221]: Standby manager daemon smithi189.tawavz started 2024-04-20T21:32:49.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:49 smithi062 ceph-mon[16221]: from='mgr.? 172.21.15.189:0/1460018821' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi189.tawavz/crt"}]: dispatch 2024-04-20T21:32:49.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:49 smithi062 ceph-mon[16221]: from='mgr.? 172.21.15.189:0/1460018821' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-20T21:32:49.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:49 smithi062 ceph-mon[16221]: from='mgr.? 172.21.15.189:0/1460018821' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi189.tawavz/key"}]: dispatch 2024-04-20T21:32:49.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:49 smithi062 ceph-mon[16221]: from='mgr.? 172.21.15.189:0/1460018821' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-20T21:32:49.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:49 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:32:49.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:49 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:32:49.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:49 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:32:49.537 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:49 smithi062 ceph-mon[16221]: Upgrade: Setting container_image for all mgr 2024-04-20T21:32:49.537 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:49 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:32:49.537 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:49 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi062.rqxuxi"}]: dispatch 2024-04-20T21:32:49.537 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:49 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi062.rqxuxi"}]': finished 2024-04-20T21:32:49.537 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:49 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi189.tawavz"}]: dispatch 2024-04-20T21:32:49.537 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:49 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi189.tawavz"}]': finished 2024-04-20T21:32:49.537 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:49 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:32:49.537 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:49 smithi062 ceph-mon[16221]: Upgrade: Setting container_image for all rgw 2024-04-20T21:32:49.537 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:49 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:32:49.537 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:49 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:32:49.537 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:49 smithi062 ceph-mon[16221]: Upgrade: Setting container_image for all rbd-mirror 2024-04-20T21:32:49.537 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:49 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:32:49.537 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:49 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:32:49.537 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:49 smithi062 ceph-mon[16221]: Upgrade: Setting container_image for all cephfs-mirror 2024-04-20T21:32:49.537 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:49 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:32:49.538 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:49 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:32:49.538 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:49 smithi062 ceph-mon[16221]: Upgrade: Setting container_image for all iscsi 2024-04-20T21:32:49.538 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:49 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:32:49.538 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:49 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:32:49.538 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:49 smithi062 ceph-mon[16221]: Upgrade: Setting container_image for all nfs 2024-04-20T21:32:49.538 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:49 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:32:49.538 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:49 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:32:49.538 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:49 smithi062 ceph-mon[16221]: Upgrade: Setting container_image for all nvmeof 2024-04-20T21:32:49.538 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:49 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:32:49.538 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:49 smithi062 ceph-mon[16221]: pgmap v39: 65 pgs: 65 active+clean; 1.1 GiB data, 3.6 GiB used, 533 GiB / 536 GiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 200 op/s 2024-04-20T21:32:50.389 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:50 smithi189 ceph-mon[21562]: mgrmap e30: smithi062.rqxuxi(active, since 72s), standbys: smithi189.tawavz 2024-04-20T21:32:50.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:50 smithi062 ceph-mon[16221]: mgrmap e30: smithi062.rqxuxi(active, since 72s), standbys: smithi189.tawavz 2024-04-20T21:32:51.410 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:51 smithi189 ceph-mon[21562]: pgmap v40: 65 pgs: 65 active+clean; 1.1 GiB data, 3.6 GiB used, 533 GiB / 536 GiB avail; 889 KiB/s rd, 964 KiB/s wr, 132 op/s 2024-04-20T21:32:51.534 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:51 smithi062 ceph-mon[16221]: pgmap v40: 65 pgs: 65 active+clean; 1.1 GiB data, 3.6 GiB used, 533 GiB / 536 GiB avail; 889 KiB/s rd, 964 KiB/s wr, 132 op/s 2024-04-20T21:32:52.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:52 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:32:52.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:52 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:32:52.830 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-20T21:32:52.830 DEBUG:teuthology.orchestra.run.smithi189:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.1/client.1/tmp 2024-04-20T21:32:53.213 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:53 smithi189 ceph-mon[21562]: pgmap v41: 65 pgs: 65 active+clean; 1.1 GiB data, 3.6 GiB used, 533 GiB / 536 GiB avail; 889 KiB/s rd, 964 KiB/s wr, 132 op/s 2024-04-20T21:32:53.363 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:53 smithi062 ceph-mon[16221]: pgmap v41: 65 pgs: 65 active+clean; 1.1 GiB data, 3.6 GiB used, 533 GiB / 536 GiB avail; 889 KiB/s rd, 964 KiB/s wr, 132 op/s 2024-04-20T21:32:55.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:55 smithi062 ceph-mon[16221]: pgmap v42: 65 pgs: 65 active+clean; 414 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 189 op/s 2024-04-20T21:32:55.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:55 smithi189 ceph-mon[21562]: pgmap v42: 65 pgs: 65 active+clean; 414 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 189 op/s 2024-04-20T21:32:56.972 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:56 smithi062 ceph-mon[16221]: Upgrade: Updating node-exporter.smithi062 (1/2) 2024-04-20T21:32:56.972 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:56 smithi062 ceph-mon[16221]: Deploying daemon node-exporter.smithi062 on smithi062 2024-04-20T21:32:56.986 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:56 smithi189 ceph-mon[21562]: Upgrade: Updating node-exporter.smithi062 (1/2) 2024-04-20T21:32:56.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:56 smithi189 ceph-mon[21562]: Deploying daemon node-exporter.smithi062 on smithi062 2024-04-20T21:32:57.986 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:57 smithi189 ceph-mon[21562]: pgmap v43: 65 pgs: 65 active+clean; 414 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 830 KiB/s rd, 894 KiB/s wr, 121 op/s 2024-04-20T21:32:58.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:57 smithi062 ceph-mon[16221]: pgmap v43: 65 pgs: 65 active+clean; 414 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 830 KiB/s rd, 894 KiB/s wr, 121 op/s 2024-04-20T21:32:59.986 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:59 smithi189 ceph-mon[21562]: pgmap v44: 65 pgs: 65 active+clean; 289 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 166 op/s 2024-04-20T21:32:59.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:59 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:32:59.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:32:59 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:33:00.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:59 smithi062 ceph-mon[16221]: pgmap v44: 65 pgs: 65 active+clean; 289 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 166 op/s 2024-04-20T21:33:00.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:59 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:33:00.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:32:59 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:33:01.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:01 smithi189 ceph-mon[21562]: pgmap v45: 65 pgs: 65 active+clean; 289 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 943 KiB/s rd, 928 KiB/s wr, 101 op/s 2024-04-20T21:33:02.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:01 smithi062 ceph-mon[16221]: pgmap v45: 65 pgs: 65 active+clean; 289 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 943 KiB/s rd, 928 KiB/s wr, 101 op/s 2024-04-20T21:33:03.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:02 smithi189 ceph-mon[21562]: Upgrade: Updating node-exporter.smithi189 (2/2) 2024-04-20T21:33:03.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:02 smithi189 ceph-mon[21562]: Deploying daemon node-exporter.smithi189 on smithi189 2024-04-20T21:33:03.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:02 smithi189 ceph-mon[21562]: pgmap v46: 65 pgs: 65 active+clean; 289 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 943 KiB/s rd, 928 KiB/s wr, 101 op/s 2024-04-20T21:33:03.284 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:02 smithi062 ceph-mon[16221]: Upgrade: Updating node-exporter.smithi189 (2/2) 2024-04-20T21:33:03.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:02 smithi062 ceph-mon[16221]: Deploying daemon node-exporter.smithi189 on smithi189 2024-04-20T21:33:03.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:02 smithi062 ceph-mon[16221]: pgmap v46: 65 pgs: 65 active+clean; 289 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 943 KiB/s rd, 928 KiB/s wr, 101 op/s 2024-04-20T21:33:05.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:05 smithi189 ceph-mon[21562]: pgmap v47: 65 pgs: 65 active+clean; 293 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 143 op/s 2024-04-20T21:33:06.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:05 smithi062 ceph-mon[16221]: pgmap v47: 65 pgs: 65 active+clean; 293 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 143 op/s 2024-04-20T21:33:07.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:07 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:33:07.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:07 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:33:07.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:07 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:33:07.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:07 smithi189 ceph-mon[21562]: pgmap v48: 65 pgs: 65 active+clean; 293 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 86 op/s 2024-04-20T21:33:07.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:07 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:33:07.784 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:07 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:33:07.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:07 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:33:07.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:07 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:33:07.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:07 smithi062 ceph-mon[16221]: pgmap v48: 65 pgs: 65 active+clean; 293 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 86 op/s 2024-04-20T21:33:07.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:07 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:33:09.650 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:09 smithi189 ceph-mon[21562]: pgmap v49: 65 pgs: 65 active+clean; 300 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 126 op/s 2024-04-20T21:33:10.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:09 smithi062 ceph-mon[16221]: pgmap v49: 65 pgs: 65 active+clean; 300 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 126 op/s 2024-04-20T21:33:10.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:10 smithi189 ceph-mon[21562]: mgrmap e31: smithi062.rqxuxi(active, since 93s), standbys: smithi189.tawavz 2024-04-20T21:33:11.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:10 smithi062 ceph-mon[16221]: mgrmap e31: smithi062.rqxuxi(active, since 93s), standbys: smithi189.tawavz 2024-04-20T21:33:11.986 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:11 smithi189 ceph-mon[21562]: pgmap v50: 65 pgs: 65 active+clean; 300 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 988 KiB/s rd, 1.2 MiB/s wr, 81 op/s 2024-04-20T21:33:11.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:11 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:33:11.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:11 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:33:12.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:11 smithi062 ceph-mon[16221]: pgmap v50: 65 pgs: 65 active+clean; 300 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 988 KiB/s rd, 1.2 MiB/s wr, 81 op/s 2024-04-20T21:33:12.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:11 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:33:12.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:11 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:33:13.236 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:12 smithi189 ceph-mon[21562]: pgmap v51: 65 pgs: 65 active+clean; 300 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 988 KiB/s rd, 1.2 MiB/s wr, 81 op/s 2024-04-20T21:33:13.284 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:12 smithi062 ceph-mon[16221]: pgmap v51: 65 pgs: 65 active+clean; 300 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 988 KiB/s rd, 1.2 MiB/s wr, 81 op/s 2024-04-20T21:33:14.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:14 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:33:14.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:14 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:33:14.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:14 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:33:14.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:14 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:33:15.937 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:15 smithi189 ceph-mon[21562]: pgmap v52: 65 pgs: 65 active+clean; 306 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.8 MiB/s rd, 1.8 MiB/s wr, 133 op/s 2024-04-20T21:33:16.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:15 smithi062 ceph-mon[16221]: pgmap v52: 65 pgs: 65 active+clean; 306 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.8 MiB/s rd, 1.8 MiB/s wr, 133 op/s 2024-04-20T21:33:17.146 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:16 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:33:17.146 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:16 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:33:17.146 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:16 smithi189 ceph-mon[21562]: pgmap v53: 65 pgs: 65 active+clean; 306 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 91 op/s 2024-04-20T21:33:17.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:16 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:33:17.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:16 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:33:17.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:16 smithi062 ceph-mon[16221]: pgmap v53: 65 pgs: 65 active+clean; 306 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 91 op/s 2024-04-20T21:33:18.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:18.030+0000 7f4479400700 1 -- 172.21.15.62:0/2402286066 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4474102490 msgr2=0x7f44741028a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:33:18.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:18.030+0000 7f4479400700 1 --2- 172.21.15.62:0/2402286066 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4474102490 0x7f44741028a0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f4464009870 tx=0x7f4464009b80 comp rx=0 tx=0).stop 2024-04-20T21:33:18.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:18.030+0000 7f4479400700 1 -- 172.21.15.62:0/2402286066 shutdown_connections 2024-04-20T21:33:18.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:18.030+0000 7f4479400700 1 --2- 172.21.15.62:0/2402286066 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4474103640 0x7f4474103a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:33:18.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:18.030+0000 7f4479400700 1 --2- 172.21.15.62:0/2402286066 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4474102490 0x7f44741028a0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:33:18.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:18.030+0000 7f4479400700 1 -- 172.21.15.62:0/2402286066 >> 172.21.15.62:0/2402286066 conn(0x7f44740fda60 msgr2=0x7f44740ffe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:33:18.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:18.030+0000 7f4479400700 1 -- 172.21.15.62:0/2402286066 shutdown_connections 2024-04-20T21:33:18.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:18.030+0000 7f4479400700 1 -- 172.21.15.62:0/2402286066 wait complete. 2024-04-20T21:33:18.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:18.032+0000 7f4479400700 1 Processor -- start 2024-04-20T21:33:18.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:18.032+0000 7f4479400700 1 -- start start 2024-04-20T21:33:18.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:18.032+0000 7f4479400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4474102490 0x7f4474197c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:33:18.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:18.033+0000 7f4479400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4474103640 0x7f44741981d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:33:18.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:18.033+0000 7f4479400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44741987d0 con 0x7f4474102490 2024-04-20T21:33:18.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:18.033+0000 7f4479400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4474198940 con 0x7f4474103640 2024-04-20T21:33:18.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:18.033+0000 7f4472800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4474102490 0x7f4474197c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:33:18.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:18.033+0000 7f4472800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4474102490 0x7f4474197c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:57860/0 (socket says 172.21.15.62:57860) 2024-04-20T21:33:18.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:18.033+0000 7f4472800700 1 -- 172.21.15.62:0/748431634 learned_addr learned my addr 172.21.15.62:0/748431634 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:33:18.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:18.034+0000 7f4472800700 1 -- 172.21.15.62:0/748431634 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4474103640 msgr2=0x7f44741981d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-20T21:33:18.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:18.034+0000 7f4472800700 1 --2- 172.21.15.62:0/748431634 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4474103640 0x7f44741981d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:33:18.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:18.034+0000 7f4472800700 1 -- 172.21.15.62:0/748431634 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4464009520 con 0x7f4474102490 2024-04-20T21:33:18.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:18.034+0000 7f4472800700 1 --2- 172.21.15.62:0/748431634 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4474102490 0x7f4474197c90 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f4464005950 tx=0x7f446400f740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:33:18.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:18.035+0000 7f446b400700 1 -- 172.21.15.62:0/748431634 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f446401c070 con 0x7f4474102490 2024-04-20T21:33:18.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:18.035+0000 7f446b400700 1 -- 172.21.15.62:0/748431634 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f446400fd90 con 0x7f4474102490 2024-04-20T21:33:18.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:18.035+0000 7f446b400700 1 -- 172.21.15.62:0/748431634 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f44640173f0 con 0x7f4474102490 2024-04-20T21:33:18.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:18.035+0000 7f4479400700 1 -- 172.21.15.62:0/748431634 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f447419d370 con 0x7f4474102490 2024-04-20T21:33:18.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:18.036+0000 7f4479400700 1 -- 172.21.15.62:0/748431634 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f447419d860 con 0x7f4474102490 2024-04-20T21:33:18.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:18.039+0000 7f4479400700 1 -- 172.21.15.62:0/748431634 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4454005320 con 0x7f4474102490 2024-04-20T21:33:18.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:18.043+0000 7f446b400700 1 -- 172.21.15.62:0/748431634 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 31) v1 ==== 92598+0+0 (secure 0 0 0) 0x7f4464020070 con 0x7f4474102490 2024-04-20T21:33:18.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:18.044+0000 7f446b400700 1 --2- 172.21.15.62:0/748431634 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f445806e9e0 0x7f4458070e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:33:18.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:18.045+0000 7f4471e00700 1 --2- 172.21.15.62:0/748431634 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f445806e9e0 0x7f4458070e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:33:18.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:18.046+0000 7f446b400700 1 -- 172.21.15.62:0/748431634 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f446408fb40 con 0x7f4474102490 2024-04-20T21:33:18.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:18.046+0000 7f4471e00700 1 --2- 172.21.15.62:0/748431634 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f445806e9e0 0x7f4458070e90 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f44741991a0 tx=0x7f4460009310 comp rx=0 tx=0).ready entity=mgr.14658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:33:18.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:18.063+0000 7f446b400700 1 -- 172.21.15.62:0/748431634 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187527 (secure 0 0 0) 0x7f4464058ec0 con 0x7f4474102490 2024-04-20T21:33:18.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:18.450+0000 7f4479400700 1 -- 172.21.15.62:0/748431634 --> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4454000bf0 con 0x7f445806e9e0 2024-04-20T21:33:18.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:18.452+0000 7f446b400700 1 -- 172.21.15.62:0/748431634 <== mgr.14658 v2:172.21.15.62:6800/1585760929 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+370 (secure 0 0 0) 0x7f4454000bf0 con 0x7f445806e9e0 2024-04-20T21:33:18.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:18.455+0000 7f4479400700 1 -- 172.21.15.62:0/748431634 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f445806e9e0 msgr2=0x7f4458070e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:33:18.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:18.455+0000 7f4479400700 1 --2- 172.21.15.62:0/748431634 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f445806e9e0 0x7f4458070e90 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f44741991a0 tx=0x7f4460009310 comp rx=0 tx=0).stop 2024-04-20T21:33:18.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:18.455+0000 7f4479400700 1 -- 172.21.15.62:0/748431634 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4474102490 msgr2=0x7f4474197c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:33:18.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:18.455+0000 7f4479400700 1 --2- 172.21.15.62:0/748431634 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4474102490 0x7f4474197c90 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f4464005950 tx=0x7f446400f740 comp rx=0 tx=0).stop 2024-04-20T21:33:18.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:18.456+0000 7f4479400700 1 -- 172.21.15.62:0/748431634 shutdown_connections 2024-04-20T21:33:18.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:18.456+0000 7f4479400700 1 --2- 172.21.15.62:0/748431634 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f445806e9e0 0x7f4458070e90 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:33:18.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:18.456+0000 7f4479400700 1 --2- 172.21.15.62:0/748431634 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4474102490 0x7f4474197c90 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:33:18.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:18.456+0000 7f4479400700 1 --2- 172.21.15.62:0/748431634 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4474103640 0x7f44741981d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:33:18.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:18.456+0000 7f4479400700 1 -- 172.21.15.62:0/748431634 >> 172.21.15.62:0/748431634 conn(0x7f44740fda60 msgr2=0x7f4474106870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:33:18.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:18.456+0000 7f4479400700 1 -- 172.21.15.62:0/748431634 shutdown_connections 2024-04-20T21:33:18.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:18.456+0000 7f4479400700 1 -- 172.21.15.62:0/748431634 wait complete. 2024-04-20T21:33:18.477 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T21:33:18.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:18.645+0000 7efffea00700 1 -- 172.21.15.62:0/1949762945 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efff80feb50 msgr2=0x7efff8100f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:33:18.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:18.645+0000 7efffea00700 1 --2- 172.21.15.62:0/1949762945 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efff80feb50 0x7efff8100f70 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7effe8009910 tx=0x7effe8009c20 comp rx=0 tx=0).stop 2024-04-20T21:33:18.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:18.646+0000 7efffea00700 1 -- 172.21.15.62:0/1949762945 shutdown_connections 2024-04-20T21:33:18.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:18.646+0000 7efffea00700 1 --2- 172.21.15.62:0/1949762945 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efff81014b0 0x7efff8103930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:33:18.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:18.646+0000 7efffea00700 1 --2- 172.21.15.62:0/1949762945 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efff80feb50 0x7efff8100f70 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:33:18.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:18.646+0000 7efffea00700 1 -- 172.21.15.62:0/1949762945 >> 172.21.15.62:0/1949762945 conn(0x7efff80fa780 msgr2=0x7efff80fcbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:33:18.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:18.646+0000 7efffea00700 1 -- 172.21.15.62:0/1949762945 shutdown_connections 2024-04-20T21:33:18.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:18.646+0000 7efffea00700 1 -- 172.21.15.62:0/1949762945 wait complete. 2024-04-20T21:33:18.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:18.647+0000 7efffea00700 1 Processor -- start 2024-04-20T21:33:18.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:18.647+0000 7efffea00700 1 -- start start 2024-04-20T21:33:18.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:18.648+0000 7efffea00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efff80feb50 0x7efff819c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:33:18.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:18.648+0000 7efffea00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efff81014b0 0x7efff819c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:33:18.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:18.648+0000 7efffea00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efff819cc10 con 0x7efff80feb50 2024-04-20T21:33:18.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:18.648+0000 7efffea00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efff819cd50 con 0x7efff81014b0 2024-04-20T21:33:18.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:18.648+0000 7efff7e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efff80feb50 0x7efff819c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:33:18.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:18.648+0000 7efff7e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efff80feb50 0x7efff819c0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:57878/0 (socket says 172.21.15.62:57878) 2024-04-20T21:33:18.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:18.648+0000 7efff7e00700 1 -- 172.21.15.62:0/4137303378 learned_addr learned my addr 172.21.15.62:0/4137303378 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:33:18.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:18.648+0000 7efff7400700 1 --2- 172.21.15.62:0/4137303378 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efff81014b0 0x7efff819c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:33:18.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:18.649+0000 7efff7e00700 1 -- 172.21.15.62:0/4137303378 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efff81014b0 msgr2=0x7efff819c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:33:18.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:18.649+0000 7efff7e00700 1 --2- 172.21.15.62:0/4137303378 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efff81014b0 0x7efff819c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:33:18.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:18.649+0000 7efff7e00700 1 -- 172.21.15.62:0/4137303378 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effe80095f0 con 0x7efff80feb50 2024-04-20T21:33:18.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:18.649+0000 7efff7e00700 1 --2- 172.21.15.62:0/4137303378 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efff80feb50 0x7efff819c0b0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7effe800bd70 tx=0x7effe800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:33:18.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:18.649+0000 7efff4c00700 1 -- 172.21.15.62:0/4137303378 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7effe801d070 con 0x7efff80feb50 2024-04-20T21:33:18.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:18.649+0000 7efff4c00700 1 -- 172.21.15.62:0/4137303378 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7effe800f460 con 0x7efff80feb50 2024-04-20T21:33:18.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:18.649+0000 7efffea00700 1 -- 172.21.15.62:0/4137303378 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efff81a17a0 con 0x7efff80feb50 2024-04-20T21:33:18.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:18.649+0000 7efff4c00700 1 -- 172.21.15.62:0/4137303378 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7effe8021620 con 0x7efff80feb50 2024-04-20T21:33:18.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:18.650+0000 7efffea00700 1 -- 172.21.15.62:0/4137303378 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efff81a1c90 con 0x7efff80feb50 2024-04-20T21:33:18.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:18.652+0000 7effee000700 1 -- 172.21.15.62:0/4137303378 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effd00052f0 con 0x7efff80feb50 2024-04-20T21:33:18.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:18.653+0000 7efff4c00700 1 -- 172.21.15.62:0/4137303378 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 31) v1 ==== 92598+0+0 (secure 0 0 0) 0x7effe800fa40 con 0x7efff80feb50 2024-04-20T21:33:18.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:18.654+0000 7efff4c00700 1 --2- 172.21.15.62:0/4137303378 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7effdc06e990 0x7effdc070e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:33:18.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:18.654+0000 7efff4c00700 1 -- 172.21.15.62:0/4137303378 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6070+0+0 (secure 0 0 0) 0x7effe808fc20 con 0x7efff80feb50 2024-04-20T21:33:18.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:18.655+0000 7efff7400700 1 --2- 172.21.15.62:0/4137303378 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7effdc06e990 0x7effdc070e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:33:18.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:18.656+0000 7efff7400700 1 --2- 172.21.15.62:0/4137303378 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7effdc06e990 0x7effdc070e40 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7effe4005fd0 tx=0x7effe4005f60 comp rx=0 tx=0).ready entity=mgr.14658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:33:18.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:18.664+0000 7efff4c00700 1 -- 172.21.15.62:0/4137303378 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187527 (secure 0 0 0) 0x7effe805c990 con 0x7efff80feb50 2024-04-20T21:33:18.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:18.989+0000 7effee000700 1 -- 172.21.15.62:0/4137303378 --> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7effd0000bc0 con 0x7effdc06e990 2024-04-20T21:33:18.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:18.991+0000 7efff4c00700 1 -- 172.21.15.62:0/4137303378 <== mgr.14658 v2:172.21.15.62:6800/1585760929 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+370 (secure 0 0 0) 0x7effd0000bc0 con 0x7effdc06e990 2024-04-20T21:33:18.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:18.994+0000 7effee000700 1 -- 172.21.15.62:0/4137303378 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7effdc06e990 msgr2=0x7effdc070e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:33:18.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:18.994+0000 7effee000700 1 --2- 172.21.15.62:0/4137303378 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7effdc06e990 0x7effdc070e40 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7effe4005fd0 tx=0x7effe4005f60 comp rx=0 tx=0).stop 2024-04-20T21:33:18.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:18.994+0000 7effee000700 1 -- 172.21.15.62:0/4137303378 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efff80feb50 msgr2=0x7efff819c0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:33:18.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:18.994+0000 7effee000700 1 --2- 172.21.15.62:0/4137303378 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efff80feb50 0x7efff819c0b0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7effe800bd70 tx=0x7effe800be50 comp rx=0 tx=0).stop 2024-04-20T21:33:18.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:18.994+0000 7effee000700 1 -- 172.21.15.62:0/4137303378 shutdown_connections 2024-04-20T21:33:18.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:18.994+0000 7effee000700 1 --2- 172.21.15.62:0/4137303378 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7effdc06e990 0x7effdc070e40 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:33:18.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:18.994+0000 7effee000700 1 --2- 172.21.15.62:0/4137303378 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efff80feb50 0x7efff819c0b0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:33:18.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:18.994+0000 7effee000700 1 --2- 172.21.15.62:0/4137303378 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efff81014b0 0x7efff819c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:33:18.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:18.994+0000 7effee000700 1 -- 172.21.15.62:0/4137303378 >> 172.21.15.62:0/4137303378 conn(0x7efff80fa780 msgr2=0x7efff80fcbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:33:18.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:18.995+0000 7effee000700 1 -- 172.21.15.62:0/4137303378 shutdown_connections 2024-04-20T21:33:18.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:18.995+0000 7effee000700 1 -- 172.21.15.62:0/4137303378 wait complete. 2024-04-20T21:33:19.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:19.208+0000 7f1347c00700 1 -- 172.21.15.62:0/1038233205 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1340102490 msgr2=0x7f13401028a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:33:19.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:19.208+0000 7f1347c00700 1 --2- 172.21.15.62:0/1038233205 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1340102490 0x7f13401028a0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f1334009910 tx=0x7f1334009c20 comp rx=0 tx=0).stop 2024-04-20T21:33:19.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:19.209+0000 7f1347c00700 1 -- 172.21.15.62:0/1038233205 shutdown_connections 2024-04-20T21:33:19.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:19.209+0000 7f1347c00700 1 --2- 172.21.15.62:0/1038233205 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1340103640 0x7f1340103a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:33:19.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:19.209+0000 7f1347c00700 1 --2- 172.21.15.62:0/1038233205 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1340102490 0x7f13401028a0 secure :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f1334009910 tx=0x7f1334009c20 comp rx=0 tx=0).stop 2024-04-20T21:33:19.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:19.209+0000 7f1347c00700 1 -- 172.21.15.62:0/1038233205 >> 172.21.15.62:0/1038233205 conn(0x7f13400fda60 msgr2=0x7f13400ffe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:33:19.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:19.212+0000 7f1347c00700 1 -- 172.21.15.62:0/1038233205 shutdown_connections 2024-04-20T21:33:19.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:19.212+0000 7f1347c00700 1 -- 172.21.15.62:0/1038233205 wait complete. 2024-04-20T21:33:19.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:19.213+0000 7f1347c00700 1 Processor -- start 2024-04-20T21:33:19.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:19.214+0000 7f1347c00700 1 -- start start 2024-04-20T21:33:19.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:19.214+0000 7f1347c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1340103640 0x7f1340197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:33:19.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:19.214+0000 7f1347c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1340198460 0x7f134019d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:33:19.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:19.214+0000 7f1347c00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1340198960 con 0x7f1340103640 2024-04-20T21:33:19.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:19.214+0000 7f1347c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1340198ad0 con 0x7f1340198460 2024-04-20T21:33:19.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:19.215+0000 7f133fe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1340198460 0x7f134019d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:33:19.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:19.215+0000 7f133fe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1340198460 0x7f134019d4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.62:39092/0 (socket says 172.21.15.62:39092) 2024-04-20T21:33:19.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:19.215+0000 7f133fe00700 1 -- 172.21.15.62:0/1716123930 learned_addr learned my addr 172.21.15.62:0/1716123930 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:33:19.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:19.215+0000 7f1345200700 1 --2- 172.21.15.62:0/1716123930 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1340103640 0x7f1340197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:33:19.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:19.216+0000 7f133fe00700 1 -- 172.21.15.62:0/1716123930 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1340103640 msgr2=0x7f1340197f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:33:19.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:19.216+0000 7f133fe00700 1 --2- 172.21.15.62:0/1716123930 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1340103640 0x7f1340197f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:33:19.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:19.217+0000 7f133fe00700 1 -- 172.21.15.62:0/1716123930 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13340095f0 con 0x7f1340198460 2024-04-20T21:33:19.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:19.217+0000 7f133fe00700 1 --2- 172.21.15.62:0/1716123930 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1340198460 0x7f134019d4d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f132c00d900 tx=0x7f132c00dc10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:33:19.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:19.217+0000 7f133d600700 1 -- 172.21.15.62:0/1716123930 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f132c0041f0 con 0x7f1340198460 2024-04-20T21:33:19.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:19.217+0000 7f1347c00700 1 -- 172.21.15.62:0/1716123930 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f134019da70 con 0x7f1340198460 2024-04-20T21:33:19.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:19.217+0000 7f133d600700 1 -- 172.21.15.62:0/1716123930 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f132c004ce0 con 0x7f1340198460 2024-04-20T21:33:19.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:19.217+0000 7f133d600700 1 -- 172.21.15.62:0/1716123930 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f132c00b750 con 0x7f1340198460 2024-04-20T21:33:19.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:19.218+0000 7f1347c00700 1 -- 172.21.15.62:0/1716123930 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f134019df90 con 0x7f1340198460 2024-04-20T21:33:19.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:19.219+0000 7f1347c00700 1 -- 172.21.15.62:0/1716123930 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f134010af90 con 0x7f1340198460 2024-04-20T21:33:19.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:19.222+0000 7f133d600700 1 -- 172.21.15.62:0/1716123930 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 31) v1 ==== 92598+0+0 (secure 0 0 0) 0x7f132c004e50 con 0x7f1340198460 2024-04-20T21:33:19.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:19.224+0000 7f133d600700 1 --2- 172.21.15.62:0/1716123930 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f132806e990 0x7f1328070e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:33:19.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:19.224+0000 7f133d600700 1 -- 172.21.15.62:0/1716123930 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f132c08ecc0 con 0x7f1340198460 2024-04-20T21:33:19.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:19.226+0000 7f1345200700 1 --2- 172.21.15.62:0/1716123930 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f132806e990 0x7f1328070e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:33:19.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:19.227+0000 7f1345200700 1 --2- 172.21.15.62:0/1716123930 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f132806e990 0x7f1328070e40 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f1334000c00 tx=0x7f1334005fb0 comp rx=0 tx=0).ready entity=mgr.14658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:33:19.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:19.230+0000 7f133d600700 1 -- 172.21.15.62:0/1716123930 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187527 (secure 0 0 0) 0x7f132c05bab0 con 0x7f1340198460 2024-04-20T21:33:19.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:19.587+0000 7f1347c00700 1 -- 172.21.15.62:0/1716123930 --> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f1340107f90 con 0x7f132806e990 2024-04-20T21:33:19.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:19.603+0000 7f133d600700 1 -- 172.21.15.62:0/1716123930 <== mgr.14658 v2:172.21.15.62:6800/1585760929 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f1340107f90 con 0x7f132806e990 2024-04-20T21:33:19.603 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T21:33:19.603 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (11m) 3s ago 13m 14.3M - 0.25.0 c8568f914cd2 957f663a9cd7 2024-04-20T21:33:19.603 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (13m) 3s ago 13m 8066k - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T21:33:19.603 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (11m) 8s ago 11m 7876k - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T21:33:19.603 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (13m) 3s ago 13m 6656k - 18.2.0 dc2bc1663786 36e10ef5be39 2024-04-20T21:33:19.604 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (11m) 8s ago 11m 6665k - 18.2.0 dc2bc1663786 4101f9953e3d 2024-04-20T21:33:19.604 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (10m) 3s ago 12m 73.6M - 9.4.7 954c08fa6188 99363bb095d8 2024-04-20T21:33:19.604 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (5m) 3s ago 5m 185M - 18.2.0 dc2bc1663786 5e6460aa6159 2024-04-20T21:33:19.604 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (5m) 3s ago 5m 186M - 18.2.0 dc2bc1663786 e65f2267544e 2024-04-20T21:33:19.604 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (5m) 8s ago 5m 151M - 18.2.0 dc2bc1663786 fb2a8a223db8 2024-04-20T21:33:19.604 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (5m) 8s ago 5m 114M - 18.2.0 dc2bc1663786 f2eec2ee8951 2024-04-20T21:33:19.604 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (2m) 3s ago 14m 499M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T21:33:19.604 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (44s) 8s ago 11m 448M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T21:33:19.604 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (14m) 3s ago 14m 55.3M 2048M 18.2.0 dc2bc1663786 bd6efbd5c380 2024-04-20T21:33:19.604 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (11m) 8s ago 11m 45.7M 2048M 18.2.0 dc2bc1663786 3735c8e5db9a 2024-04-20T21:33:19.604 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (20s) 3s ago 13m 2683k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T21:33:19.604 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (13s) 8s ago 11m 3003k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T21:33:19.604 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (10m) 3s ago 10m 179M 1216M 18.2.0 dc2bc1663786 3f315c3afc1f 2024-04-20T21:33:19.604 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (10m) 3s ago 10m 185M 1216M 18.2.0 dc2bc1663786 44e4d07416ad 2024-04-20T21:33:19.604 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (9m) 3s ago 9m 149M 1216M 18.2.0 dc2bc1663786 31c5d9586e98 2024-04-20T21:33:19.604 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (8m) 8s ago 8m 211M 2240M 18.2.0 dc2bc1663786 611c1e242693 2024-04-20T21:33:19.605 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (8m) 8s ago 8m 190M 2240M 18.2.0 dc2bc1663786 6c8fce91d45d 2024-04-20T21:33:19.605 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (7m) 8s ago 7m 189M 2240M 18.2.0 dc2bc1663786 8e4574bae8db 2024-04-20T21:33:19.605 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (53s) 3s ago 12m 57.5M - 2.43.0 a07b618ecd1d 3e50e774ad4e 2024-04-20T21:33:19.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:19.606+0000 7f1347c00700 1 -- 172.21.15.62:0/1716123930 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f132806e990 msgr2=0x7f1328070e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:33:19.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:19.606+0000 7f1347c00700 1 --2- 172.21.15.62:0/1716123930 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f132806e990 0x7f1328070e40 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f1334000c00 tx=0x7f1334005fb0 comp rx=0 tx=0).stop 2024-04-20T21:33:19.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:19.606+0000 7f1347c00700 1 -- 172.21.15.62:0/1716123930 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1340198460 msgr2=0x7f134019d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:33:19.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:19.606+0000 7f1347c00700 1 --2- 172.21.15.62:0/1716123930 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1340198460 0x7f134019d4d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f132c00d900 tx=0x7f132c00dc10 comp rx=0 tx=0).stop 2024-04-20T21:33:19.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:19.607+0000 7f1347c00700 1 -- 172.21.15.62:0/1716123930 shutdown_connections 2024-04-20T21:33:19.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:19.607+0000 7f1347c00700 1 --2- 172.21.15.62:0/1716123930 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f132806e990 0x7f1328070e40 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:33:19.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:19.607+0000 7f1347c00700 1 --2- 172.21.15.62:0/1716123930 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1340103640 0x7f1340197f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:33:19.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:19.607+0000 7f1347c00700 1 --2- 172.21.15.62:0/1716123930 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1340198460 0x7f134019d4d0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:33:19.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:19.607+0000 7f1347c00700 1 -- 172.21.15.62:0/1716123930 >> 172.21.15.62:0/1716123930 conn(0x7f13400fda60 msgr2=0x7f1340106870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:33:19.607 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:19 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:33:19.607 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:19 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:33:19.607 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:19 smithi062 ceph-mon[16221]: from='client.14752 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:33:19.607 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:19 smithi062 ceph-mon[16221]: pgmap v54: 65 pgs: 65 active+clean; 309 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.0 MiB/s rd, 2.0 MiB/s wr, 140 op/s 2024-04-20T21:33:19.607 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:19 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:33:19.607 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:19 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:33:19.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:19.610+0000 7f1347c00700 1 -- 172.21.15.62:0/1716123930 shutdown_connections 2024-04-20T21:33:19.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:19.610+0000 7f1347c00700 1 -- 172.21.15.62:0/1716123930 wait complete. 2024-04-20T21:33:19.736 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:19 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:33:19.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:19 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:33:19.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:19 smithi189 ceph-mon[21562]: from='client.14752 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:33:19.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:19 smithi189 ceph-mon[21562]: pgmap v54: 65 pgs: 65 active+clean; 309 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.0 MiB/s rd, 2.0 MiB/s wr, 140 op/s 2024-04-20T21:33:19.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:19 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:33:19.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:19 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:33:19.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:19.815+0000 7f7f47000700 1 -- 172.21.15.62:0/1580223879 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f40100080 msgr2=0x7f7f40100490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:33:19.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:19.815+0000 7f7f47000700 1 --2- 172.21.15.62:0/1580223879 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f40100080 0x7f7f40100490 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f7f34009910 tx=0x7f7f34009c20 comp rx=0 tx=0).stop 2024-04-20T21:33:19.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:19.815+0000 7f7f47000700 1 -- 172.21.15.62:0/1580223879 shutdown_connections 2024-04-20T21:33:19.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:19.815+0000 7f7f47000700 1 --2- 172.21.15.62:0/1580223879 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f40101280 0x7f7f401016d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:33:19.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:19.815+0000 7f7f47000700 1 --2- 172.21.15.62:0/1580223879 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f40100080 0x7f7f40100490 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:33:19.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:19.815+0000 7f7f47000700 1 -- 172.21.15.62:0/1580223879 >> 172.21.15.62:0/1580223879 conn(0x7f7f400fb610 msgr2=0x7f7f400fda60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:33:19.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:19.817+0000 7f7f47000700 1 -- 172.21.15.62:0/1580223879 shutdown_connections 2024-04-20T21:33:19.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:19.817+0000 7f7f47000700 1 -- 172.21.15.62:0/1580223879 wait complete. 2024-04-20T21:33:19.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:19.818+0000 7f7f47000700 1 Processor -- start 2024-04-20T21:33:19.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:19.818+0000 7f7f47000700 1 -- start start 2024-04-20T21:33:19.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:19.819+0000 7f7f47000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f40100080 0x7f7f40074af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:33:19.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:19.819+0000 7f7f47000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f40101280 0x7f7f40073140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:33:19.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:19.819+0000 7f7f47000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f40073680 con 0x7f7f40100080 2024-04-20T21:33:19.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:19.819+0000 7f7f47000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f400737c0 con 0x7f7f40101280 2024-04-20T21:33:19.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:19.819+0000 7f7f45c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f40100080 0x7f7f40074af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:33:19.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:19.819+0000 7f7f45c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f40100080 0x7f7f40074af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:57908/0 (socket says 172.21.15.62:57908) 2024-04-20T21:33:19.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:19.819+0000 7f7f45c00700 1 -- 172.21.15.62:0/1451756885 learned_addr learned my addr 172.21.15.62:0/1451756885 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:33:19.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:19.819+0000 7f7f45200700 1 --2- 172.21.15.62:0/1451756885 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f40101280 0x7f7f40073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:33:19.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:19.820+0000 7f7f45c00700 1 -- 172.21.15.62:0/1451756885 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f40101280 msgr2=0x7f7f40073140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:33:19.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:19.820+0000 7f7f45c00700 1 --2- 172.21.15.62:0/1451756885 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f40101280 0x7f7f40073140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:33:19.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:19.820+0000 7f7f45c00700 1 -- 172.21.15.62:0/1451756885 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f340095f0 con 0x7f7f40100080 2024-04-20T21:33:19.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:19.820+0000 7f7f45c00700 1 --2- 172.21.15.62:0/1451756885 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f40100080 0x7f7f40074af0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f7f3401cdb0 tx=0x7f7f3401cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:33:19.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:19.820+0000 7f7f3e000700 1 -- 172.21.15.62:0/1451756885 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7f34003a60 con 0x7f7f40100080 2024-04-20T21:33:19.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:19.820+0000 7f7f3e000700 1 -- 172.21.15.62:0/1451756885 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f7f34003bc0 con 0x7f7f40100080 2024-04-20T21:33:19.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:19.821+0000 7f7f47000700 1 -- 172.21.15.62:0/1451756885 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f40073a40 con 0x7f7f40100080 2024-04-20T21:33:19.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:19.821+0000 7f7f47000700 1 -- 172.21.15.62:0/1451756885 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f40073f30 con 0x7f7f40100080 2024-04-20T21:33:19.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:19.821+0000 7f7f3e000700 1 -- 172.21.15.62:0/1451756885 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7f3401f620 con 0x7f7f40100080 2024-04-20T21:33:19.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:19.822+0000 7f7f47000700 1 -- 172.21.15.62:0/1451756885 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f4006ce80 con 0x7f7f40100080 2024-04-20T21:33:19.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:19.824+0000 7f7f3e000700 1 -- 172.21.15.62:0/1451756885 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 31) v1 ==== 92598+0+0 (secure 0 0 0) 0x7f7f34010970 con 0x7f7f40100080 2024-04-20T21:33:19.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:19.825+0000 7f7f3e000700 1 --2- 172.21.15.62:0/1451756885 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f7f2806e940 0x7f7f28070df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:33:19.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:19.826+0000 7f7f3e000700 1 -- 172.21.15.62:0/1451756885 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f7f34090a70 con 0x7f7f40100080 2024-04-20T21:33:19.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:19.826+0000 7f7f45200700 1 --2- 172.21.15.62:0/1451756885 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f7f2806e940 0x7f7f28070df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:33:19.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:19.827+0000 7f7f45200700 1 --2- 172.21.15.62:0/1451756885 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f7f2806e940 0x7f7f28070df0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f7f2c005950 tx=0x7f7f2c00a300 comp rx=0 tx=0).ready entity=mgr.14658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:33:19.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:19.831+0000 7f7f3e000700 1 -- 172.21.15.62:0/1451756885 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187527 (secure 0 0 0) 0x7f7f3405d880 con 0x7f7f40100080 2024-04-20T21:33:20.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:20.321+0000 7f7f47000700 1 -- 172.21.15.62:0/1451756885 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f7f401b7590 con 0x7f7f40100080 2024-04-20T21:33:20.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:20.322+0000 7f7f3e000700 1 -- 172.21.15.62:0/1451756885 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+675 (secure 0 0 0) 0x7f7f34003d30 con 0x7f7f40100080 2024-04-20T21:33:20.322 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:33:20.322 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T21:33:20.322 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 2 2024-04-20T21:33:20.322 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:33:20.322 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T21:33:20.322 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:33:20.322 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:33:20.322 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T21:33:20.322 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 6 2024-04-20T21:33:20.322 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:33:20.322 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T21:33:20.323 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 4 2024-04-20T21:33:20.323 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:33:20.323 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T21:33:20.323 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 12, 2024-04-20T21:33:20.323 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:33:20.323 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T21:33:20.323 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:33:20.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:20.325+0000 7f7f47000700 1 -- 172.21.15.62:0/1451756885 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f7f2806e940 msgr2=0x7f7f28070df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:33:20.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:20.325+0000 7f7f47000700 1 --2- 172.21.15.62:0/1451756885 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f7f2806e940 0x7f7f28070df0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f7f2c005950 tx=0x7f7f2c00a300 comp rx=0 tx=0).stop 2024-04-20T21:33:20.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:20.325+0000 7f7f47000700 1 -- 172.21.15.62:0/1451756885 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f40100080 msgr2=0x7f7f40074af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:33:20.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:20.325+0000 7f7f47000700 1 --2- 172.21.15.62:0/1451756885 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f40100080 0x7f7f40074af0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f7f3401cdb0 tx=0x7f7f3401cde0 comp rx=0 tx=0).stop 2024-04-20T21:33:20.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:20.326+0000 7f7f47000700 1 -- 172.21.15.62:0/1451756885 shutdown_connections 2024-04-20T21:33:20.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:20.326+0000 7f7f47000700 1 --2- 172.21.15.62:0/1451756885 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f7f2806e940 0x7f7f28070df0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:33:20.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:20.326+0000 7f7f47000700 1 --2- 172.21.15.62:0/1451756885 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f40100080 0x7f7f40074af0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:33:20.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:20.326+0000 7f7f47000700 1 --2- 172.21.15.62:0/1451756885 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f40101280 0x7f7f40073140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:33:20.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:20.326+0000 7f7f47000700 1 -- 172.21.15.62:0/1451756885 >> 172.21.15.62:0/1451756885 conn(0x7f7f400fb610 msgr2=0x7f7f401044b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:33:20.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:20.326+0000 7f7f47000700 1 -- 172.21.15.62:0/1451756885 shutdown_connections 2024-04-20T21:33:20.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:20.326+0000 7f7f47000700 1 -- 172.21.15.62:0/1451756885 wait complete. 2024-04-20T21:33:20.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:20.511+0000 7f102ca00700 1 -- 172.21.15.62:0/1238232770 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1028074ce0 msgr2=0x7f10280731e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:33:20.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:20.511+0000 7f102ca00700 1 --2- 172.21.15.62:0/1238232770 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1028074ce0 0x7f10280731e0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f1018009940 tx=0x7f1018009c50 comp rx=0 tx=0).stop 2024-04-20T21:33:20.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:20.511+0000 7f102ca00700 1 -- 172.21.15.62:0/1238232770 shutdown_connections 2024-04-20T21:33:20.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:20.511+0000 7f102ca00700 1 --2- 172.21.15.62:0/1238232770 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1028073720 0x7f1028073b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:33:20.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:20.511+0000 7f102ca00700 1 --2- 172.21.15.62:0/1238232770 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1028074ce0 0x7f10280731e0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:33:20.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:20.511+0000 7f102ca00700 1 -- 172.21.15.62:0/1238232770 >> 172.21.15.62:0/1238232770 conn(0x7f10280fb650 msgr2=0x7f10280fdaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:33:20.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:20.511+0000 7f102ca00700 1 -- 172.21.15.62:0/1238232770 shutdown_connections 2024-04-20T21:33:20.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:20.512+0000 7f102ca00700 1 -- 172.21.15.62:0/1238232770 wait complete. 2024-04-20T21:33:20.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:20.513+0000 7f102ca00700 1 Processor -- start 2024-04-20T21:33:20.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:20.513+0000 7f102ca00700 1 -- start start 2024-04-20T21:33:20.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:20.513+0000 7f102ca00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1028073720 0x7f1028197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:33:20.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:20.513+0000 7f102ca00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1028074ce0 0x7f1028197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:33:20.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:20.513+0000 7f102ca00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1028198590 con 0x7f1028074ce0 2024-04-20T21:33:20.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:20.513+0000 7f102ca00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10281986d0 con 0x7f1028073720 2024-04-20T21:33:20.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:20.513+0000 7f1025e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1028074ce0 0x7f1028197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:33:20.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:20.514+0000 7f1025e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1028074ce0 0x7f1028197f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:57918/0 (socket says 172.21.15.62:57918) 2024-04-20T21:33:20.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:20.514+0000 7f1025e00700 1 -- 172.21.15.62:0/3328221617 learned_addr learned my addr 172.21.15.62:0/3328221617 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:33:20.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:20.514+0000 7f1026800700 1 --2- 172.21.15.62:0/3328221617 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1028073720 0x7f1028197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:33:20.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:20.514+0000 7f1025e00700 1 -- 172.21.15.62:0/3328221617 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1028073720 msgr2=0x7f1028197a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:33:20.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:20.514+0000 7f1025e00700 1 --2- 172.21.15.62:0/3328221617 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1028073720 0x7f1028197a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:33:20.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:20.514+0000 7f1025e00700 1 -- 172.21.15.62:0/3328221617 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10180095f0 con 0x7f1028074ce0 2024-04-20T21:33:20.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:20.514+0000 7f1025e00700 1 --2- 172.21.15.62:0/3328221617 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1028074ce0 0x7f1028197f70 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f101400d8d0 tx=0x7f101400dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:33:20.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:20.515+0000 7f101f400700 1 -- 172.21.15.62:0/3328221617 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f10140098a0 con 0x7f1028074ce0 2024-04-20T21:33:20.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:20.515+0000 7f102ca00700 1 -- 172.21.15.62:0/3328221617 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f102819d180 con 0x7f1028074ce0 2024-04-20T21:33:20.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:20.515+0000 7f101f400700 1 -- 172.21.15.62:0/3328221617 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f1014010460 con 0x7f1028074ce0 2024-04-20T21:33:20.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:20.515+0000 7f101f400700 1 -- 172.21.15.62:0/3328221617 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f101400f5f0 con 0x7f1028074ce0 2024-04-20T21:33:20.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:20.515+0000 7f102ca00700 1 -- 172.21.15.62:0/3328221617 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f102819d750 con 0x7f1028074ce0 2024-04-20T21:33:20.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:20.516+0000 7f102ca00700 1 -- 172.21.15.62:0/3328221617 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f102810ace0 con 0x7f1028074ce0 2024-04-20T21:33:20.516 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:20 smithi062 ceph-mon[16221]: from='client.14756 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:33:20.516 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:20 smithi062 ceph-mon[16221]: from='client.24525 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:33:20.516 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:20 smithi062 ceph-mon[16221]: from='client.? 172.21.15.62:0/1451756885' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:33:20.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:20.517+0000 7f101f400700 1 -- 172.21.15.62:0/3328221617 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 31) v1 ==== 92598+0+0 (secure 0 0 0) 0x7f1014009a00 con 0x7f1028074ce0 2024-04-20T21:33:20.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:20.518+0000 7f101f400700 1 --2- 172.21.15.62:0/3328221617 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f100c06e940 0x7f100c070df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:33:20.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:20.518+0000 7f101f400700 1 -- 172.21.15.62:0/3328221617 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f101408e710 con 0x7f1028074ce0 2024-04-20T21:33:20.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:20.519+0000 7f1026800700 1 --2- 172.21.15.62:0/3328221617 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f100c06e940 0x7f100c070df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:33:20.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:20.519+0000 7f1026800700 1 --2- 172.21.15.62:0/3328221617 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f100c06e940 0x7f100c070df0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1018006010 tx=0x7f101800b9a0 comp rx=0 tx=0).ready entity=mgr.14658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:33:20.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:20.525+0000 7f101f400700 1 -- 172.21.15.62:0/3328221617 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187527 (secure 0 0 0) 0x7f101405b500 con 0x7f1028074ce0 2024-04-20T21:33:20.640 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:20 smithi189 ceph-mon[21562]: from='client.14756 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:33:20.640 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:20 smithi189 ceph-mon[21562]: from='client.24525 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:33:20.640 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:20 smithi189 ceph-mon[21562]: from='client.? 172.21.15.62:0/1451756885' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:33:20.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:20.850+0000 7f102ca00700 1 -- 172.21.15.62:0/3328221617 --> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f10281075c0 con 0x7f100c06e940 2024-04-20T21:33:20.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:20.851+0000 7f101f400700 1 -- 172.21.15.62:0/3328221617 <== mgr.14658 v2:172.21.15.62:6800/1585760929 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+370 (secure 0 0 0) 0x7f10281075c0 con 0x7f100c06e940 2024-04-20T21:33:20.851 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:33:20.851 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T21:33:20.851 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T21:33:20.851 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-20T21:33:20.852 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T21:33:20.852 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr" 2024-04-20T21:33:20.852 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T21:33:20.852 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "2/2 daemons upgraded", 2024-04-20T21:33:20.852 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading node-exporter daemons", 2024-04-20T21:33:20.852 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T21:33:20.852 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:33:20.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:20.856+0000 7f102ca00700 1 -- 172.21.15.62:0/3328221617 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f100c06e940 msgr2=0x7f100c070df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:33:20.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:20.857+0000 7f102ca00700 1 --2- 172.21.15.62:0/3328221617 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f100c06e940 0x7f100c070df0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1018006010 tx=0x7f101800b9a0 comp rx=0 tx=0).stop 2024-04-20T21:33:20.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:20.857+0000 7f102ca00700 1 -- 172.21.15.62:0/3328221617 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1028074ce0 msgr2=0x7f1028197f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:33:20.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:20.857+0000 7f102ca00700 1 --2- 172.21.15.62:0/3328221617 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1028074ce0 0x7f1028197f70 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f101400d8d0 tx=0x7f101400dbe0 comp rx=0 tx=0).stop 2024-04-20T21:33:20.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:20.857+0000 7f102ca00700 1 -- 172.21.15.62:0/3328221617 shutdown_connections 2024-04-20T21:33:20.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:20.857+0000 7f102ca00700 1 --2- 172.21.15.62:0/3328221617 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f100c06e940 0x7f100c070df0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:33:20.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:20.857+0000 7f102ca00700 1 --2- 172.21.15.62:0/3328221617 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1028073720 0x7f1028197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:33:20.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:20.857+0000 7f102ca00700 1 --2- 172.21.15.62:0/3328221617 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1028074ce0 0x7f1028197f70 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:33:20.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:20.857+0000 7f102ca00700 1 -- 172.21.15.62:0/3328221617 >> 172.21.15.62:0/3328221617 conn(0x7f10280fb650 msgr2=0x7f1028105ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:33:20.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:20.857+0000 7f102ca00700 1 -- 172.21.15.62:0/3328221617 shutdown_connections 2024-04-20T21:33:20.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:20.858+0000 7f102ca00700 1 -- 172.21.15.62:0/3328221617 wait complete. 2024-04-20T21:33:21.514 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:21 smithi062 ceph-mon[16221]: pgmap v55: 65 pgs: 65 active+clean; 309 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.6 MiB/s rd, 1.4 MiB/s wr, 100 op/s 2024-04-20T21:33:21.666 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:21 smithi189 ceph-mon[21562]: pgmap v55: 65 pgs: 65 active+clean; 309 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.6 MiB/s rd, 1.4 MiB/s wr, 100 op/s 2024-04-20T21:33:22.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:22 smithi189 ceph-mon[21562]: from='client.14768 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:33:22.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:22 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:33:22.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:22 smithi062 ceph-mon[16221]: from='client.14768 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:33:22.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:22 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:33:23.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:23 smithi189 ceph-mon[21562]: pgmap v56: 65 pgs: 65 active+clean; 309 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.6 MiB/s rd, 1.4 MiB/s wr, 100 op/s 2024-04-20T21:33:23.784 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:23 smithi062 ceph-mon[16221]: pgmap v56: 65 pgs: 65 active+clean; 309 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.6 MiB/s rd, 1.4 MiB/s wr, 100 op/s 2024-04-20T21:33:24.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:24 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:33:24.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:24 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:33:24.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:24 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:33:24.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:24 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T21:33:24.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:24 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:33:24.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:24 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:33:24.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:24 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:33:24.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:24 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:33:24.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:24 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:33:24.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:24 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:33:24.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:24 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:33:24.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:24 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:33:24.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:24 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:33:24.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:24 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:33:25.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:24 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:33:25.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:24 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:33:25.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:24 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:33:25.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:24 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T21:33:25.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:24 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:33:25.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:24 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:33:25.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:24 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:33:25.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:24 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:33:25.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:24 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:33:25.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:24 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:33:25.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:24 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:33:25.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:24 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:33:25.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:24 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:33:25.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:24 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:33:25.964 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:25 smithi062 ceph-mon[16221]: pgmap v57: 65 pgs: 65 active+clean; 307 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.1 MiB/s rd, 2.1 MiB/s wr, 155 op/s 2024-04-20T21:33:25.986 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:25 smithi189 ceph-mon[21562]: pgmap v57: 65 pgs: 65 active+clean; 307 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.1 MiB/s rd, 2.1 MiB/s wr, 155 op/s 2024-04-20T21:33:27.986 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:27 smithi189 ceph-mon[21562]: pgmap v58: 65 pgs: 65 active+clean; 307 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 103 op/s 2024-04-20T21:33:27.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:27 smithi189 ceph-mon[21562]: Upgrade: Updating prometheus.smithi062 2024-04-20T21:33:28.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:27 smithi062 ceph-mon[16221]: pgmap v58: 65 pgs: 65 active+clean; 307 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 103 op/s 2024-04-20T21:33:28.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:27 smithi062 ceph-mon[16221]: Upgrade: Updating prometheus.smithi062 2024-04-20T21:33:28.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:28 smithi189 ceph-mon[21562]: Deploying daemon prometheus.smithi062 on smithi062 2024-04-20T21:33:29.003 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:28 smithi062 ceph-mon[16221]: Deploying daemon prometheus.smithi062 on smithi062 2024-04-20T21:33:29.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:29 smithi189 ceph-mon[21562]: pgmap v59: 65 pgs: 65 active+clean; 306 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.1 MiB/s rd, 2.1 MiB/s wr, 143 op/s 2024-04-20T21:33:29.993 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:29 smithi062 ceph-mon[16221]: pgmap v59: 65 pgs: 65 active+clean; 306 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.1 MiB/s rd, 2.1 MiB/s wr, 143 op/s 2024-04-20T21:33:31.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:31 smithi062 ceph-mon[16221]: pgmap v60: 65 pgs: 65 active+clean; 306 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 94 op/s 2024-04-20T21:33:31.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:31 smithi189 ceph-mon[21562]: pgmap v60: 65 pgs: 65 active+clean; 306 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 94 op/s 2024-04-20T21:33:33.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:32 smithi062 ceph-mon[16221]: pgmap v61: 65 pgs: 65 active+clean; 306 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 94 op/s 2024-04-20T21:33:33.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:32 smithi189 ceph-mon[21562]: pgmap v61: 65 pgs: 65 active+clean; 306 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 94 op/s 2024-04-20T21:33:35.773 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:35 smithi062 ceph-mon[16221]: pgmap v62: 65 pgs: 65 active+clean; 307 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.1 MiB/s rd, 2.1 MiB/s wr, 132 op/s 2024-04-20T21:33:35.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:35 smithi189 ceph-mon[21562]: pgmap v62: 65 pgs: 65 active+clean; 307 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.1 MiB/s rd, 2.1 MiB/s wr, 132 op/s 2024-04-20T21:33:37.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:37 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:33:37.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:37 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:33:37.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:37 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:33:37.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:37 smithi189 ceph-mon[21562]: pgmap v63: 65 pgs: 65 active+clean; 307 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 77 op/s 2024-04-20T21:33:37.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:37 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:33:37.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:37 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:33:37.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:37 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:33:37.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:37 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:33:37.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:37 smithi062 ceph-mon[16221]: pgmap v63: 65 pgs: 65 active+clean; 307 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 77 op/s 2024-04-20T21:33:37.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:37 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:33:39.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:39 smithi189 ceph-mon[21562]: pgmap v64: 65 pgs: 65 active+clean; 309 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.1 MiB/s rd, 2.0 MiB/s wr, 117 op/s 2024-04-20T21:33:40.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:39 smithi062 ceph-mon[16221]: pgmap v64: 65 pgs: 65 active+clean; 309 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.1 MiB/s rd, 2.0 MiB/s wr, 117 op/s 2024-04-20T21:33:41.878 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:41 smithi062 ceph-mon[16221]: pgmap v65: 65 pgs: 65 active+clean; 309 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 77 op/s 2024-04-20T21:33:41.986 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:41 smithi189 ceph-mon[21562]: pgmap v65: 65 pgs: 65 active+clean; 309 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 77 op/s 2024-04-20T21:33:43.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:42 smithi189 ceph-mon[21562]: pgmap v66: 65 pgs: 65 active+clean; 309 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 77 op/s 2024-04-20T21:33:43.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:42 smithi062 ceph-mon[16221]: pgmap v66: 65 pgs: 65 active+clean; 309 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 77 op/s 2024-04-20T21:33:45.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:45 smithi189 ceph-mon[21562]: pgmap v67: 65 pgs: 65 active+clean; 313 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 MiB/s rd, 1.7 MiB/s wr, 122 op/s 2024-04-20T21:33:46.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:45 smithi062 ceph-mon[16221]: pgmap v67: 65 pgs: 65 active+clean; 313 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 MiB/s rd, 1.7 MiB/s wr, 122 op/s 2024-04-20T21:33:47.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:47 smithi189 ceph-mon[21562]: pgmap v68: 65 pgs: 65 active+clean; 313 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1008 KiB/s rd, 1.0 MiB/s wr, 84 op/s 2024-04-20T21:33:48.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:47 smithi062 ceph-mon[16221]: pgmap v68: 65 pgs: 65 active+clean; 313 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1008 KiB/s rd, 1.0 MiB/s wr, 84 op/s 2024-04-20T21:33:49.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:49 smithi189 ceph-mon[21562]: pgmap v69: 65 pgs: 65 active+clean; 317 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 128 op/s 2024-04-20T21:33:49.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:49 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:33:49.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:49 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:33:50.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:49 smithi062 ceph-mon[16221]: pgmap v69: 65 pgs: 65 active+clean; 317 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 128 op/s 2024-04-20T21:33:50.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:49 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:33:50.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:49 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:33:51.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:51.108+0000 7f5564e00700 1 -- 172.21.15.62:0/1038041535 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55601033d0 msgr2=0x7f55601057b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:33:51.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:51.108+0000 7f5564e00700 1 --2- 172.21.15.62:0/1038041535 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55601033d0 0x7f55601057b0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f554c009910 tx=0x7f554c009c20 comp rx=0 tx=0).stop 2024-04-20T21:33:51.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:51.108+0000 7f5564e00700 1 -- 172.21.15.62:0/1038041535 shutdown_connections 2024-04-20T21:33:51.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:51.108+0000 7f5564e00700 1 --2- 172.21.15.62:0/1038041535 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55601033d0 0x7f55601057b0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:33:51.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:51.108+0000 7f5564e00700 1 --2- 172.21.15.62:0/1038041535 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5560100ab0 0x7f5560102e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:33:51.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:51.108+0000 7f5564e00700 1 -- 172.21.15.62:0/1038041535 >> 172.21.15.62:0/1038041535 conn(0x7f55600fa4d0 msgr2=0x7f55600fc900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:33:51.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:51.109+0000 7f5564e00700 1 -- 172.21.15.62:0/1038041535 shutdown_connections 2024-04-20T21:33:51.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:51.109+0000 7f5564e00700 1 -- 172.21.15.62:0/1038041535 wait complete. 2024-04-20T21:33:51.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:51.110+0000 7f5564e00700 1 Processor -- start 2024-04-20T21:33:51.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:51.110+0000 7f5564e00700 1 -- start start 2024-04-20T21:33:51.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:51.111+0000 7f5564e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5560100ab0 0x7f5560191c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:33:51.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:51.111+0000 7f5564e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55601033d0 0x7f5560192150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:33:51.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:51.111+0000 7f5564e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f556018e700 con 0x7f55601033d0 2024-04-20T21:33:51.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:51.111+0000 7f5564e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f556018e870 con 0x7f5560100ab0 2024-04-20T21:33:51.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:51.111+0000 7f555ea00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55601033d0 0x7f5560192150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:33:51.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:51.111+0000 7f555ea00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55601033d0 0x7f5560192150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:57434/0 (socket says 172.21.15.62:57434) 2024-04-20T21:33:51.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:51.112+0000 7f555ea00700 1 -- 172.21.15.62:0/2515609651 learned_addr learned my addr 172.21.15.62:0/2515609651 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:33:51.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:51.112+0000 7f555f400700 1 --2- 172.21.15.62:0/2515609651 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5560100ab0 0x7f5560191c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:33:51.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:51.112+0000 7f555ea00700 1 -- 172.21.15.62:0/2515609651 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5560100ab0 msgr2=0x7f5560191c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:33:51.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:51.112+0000 7f555ea00700 1 --2- 172.21.15.62:0/2515609651 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5560100ab0 0x7f5560191c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:33:51.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:51.112+0000 7f555ea00700 1 -- 172.21.15.62:0/2515609651 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f554c0095f0 con 0x7f55601033d0 2024-04-20T21:33:51.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:51.113+0000 7f555ea00700 1 --2- 172.21.15.62:0/2515609651 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55601033d0 0x7f5560192150 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f554c00b5c0 tx=0x7f554c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:33:51.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:51.113+0000 7f5557e00700 1 -- 172.21.15.62:0/2515609651 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f554c01d070 con 0x7f55601033d0 2024-04-20T21:33:51.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:51.113+0000 7f5557e00700 1 -- 172.21.15.62:0/2515609651 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f554c00f460 con 0x7f55601033d0 2024-04-20T21:33:51.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:51.113+0000 7f5557e00700 1 -- 172.21.15.62:0/2515609651 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f554c021640 con 0x7f55601033d0 2024-04-20T21:33:51.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:51.113+0000 7f5564e00700 1 -- 172.21.15.62:0/2515609651 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f556018eaf0 con 0x7f55601033d0 2024-04-20T21:33:51.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:51.113+0000 7f5564e00700 1 -- 172.21.15.62:0/2515609651 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f556018efe0 con 0x7f55601033d0 2024-04-20T21:33:51.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:51.115+0000 7f5564e00700 1 -- 172.21.15.62:0/2515609651 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55600fc090 con 0x7f55601033d0 2024-04-20T21:33:51.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:51.117+0000 7f5557e00700 1 -- 172.21.15.62:0/2515609651 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 31) v1 ==== 92598+0+0 (secure 0 0 0) 0x7f554c00f5d0 con 0x7f55601033d0 2024-04-20T21:33:51.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:51.118+0000 7f5557e00700 1 --2- 172.21.15.62:0/2515609651 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f554406e940 0x7f5544070df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:33:51.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:51.119+0000 7f5557e00700 1 -- 172.21.15.62:0/2515609651 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f554c090070 con 0x7f55601033d0 2024-04-20T21:33:51.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:51.119+0000 7f555f400700 1 --2- 172.21.15.62:0/2515609651 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f554406e940 0x7f5544070df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:33:51.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:51.120+0000 7f555f400700 1 --2- 172.21.15.62:0/2515609651 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f554406e940 0x7f5544070df0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5550005950 tx=0x7f55500058e0 comp rx=0 tx=0).ready entity=mgr.14658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:33:51.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:51.123+0000 7f5557e00700 1 -- 172.21.15.62:0/2515609651 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187527 (secure 0 0 0) 0x7f554c05cee0 con 0x7f55601033d0 2024-04-20T21:33:51.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:51.461+0000 7f5564e00700 1 -- 172.21.15.62:0/2515609651 --> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5560061190 con 0x7f554406e940 2024-04-20T21:33:51.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:51.463+0000 7f5557e00700 1 -- 172.21.15.62:0/2515609651 <== mgr.14658 v2:172.21.15.62:6800/1585760929 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+367 (secure 0 0 0) 0x7f5560061190 con 0x7f554406e940 2024-04-20T21:33:51.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:51.466+0000 7f5564e00700 1 -- 172.21.15.62:0/2515609651 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f554406e940 msgr2=0x7f5544070df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:33:51.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:51.466+0000 7f5564e00700 1 --2- 172.21.15.62:0/2515609651 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f554406e940 0x7f5544070df0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5550005950 tx=0x7f55500058e0 comp rx=0 tx=0).stop 2024-04-20T21:33:51.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:51.466+0000 7f5564e00700 1 -- 172.21.15.62:0/2515609651 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55601033d0 msgr2=0x7f5560192150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:33:51.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:51.466+0000 7f5564e00700 1 --2- 172.21.15.62:0/2515609651 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55601033d0 0x7f5560192150 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f554c00b5c0 tx=0x7f554c00be20 comp rx=0 tx=0).stop 2024-04-20T21:33:51.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:51.466+0000 7f5564e00700 1 -- 172.21.15.62:0/2515609651 shutdown_connections 2024-04-20T21:33:51.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:51.466+0000 7f5564e00700 1 --2- 172.21.15.62:0/2515609651 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f554406e940 0x7f5544070df0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:33:51.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:51.466+0000 7f5564e00700 1 --2- 172.21.15.62:0/2515609651 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5560100ab0 0x7f5560191c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:33:51.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:51.466+0000 7f5564e00700 1 --2- 172.21.15.62:0/2515609651 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55601033d0 0x7f5560192150 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:33:51.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:51.466+0000 7f5564e00700 1 -- 172.21.15.62:0/2515609651 >> 172.21.15.62:0/2515609651 conn(0x7f55600fa4d0 msgr2=0x7f5560100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:33:51.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:51.466+0000 7f5564e00700 1 -- 172.21.15.62:0/2515609651 shutdown_connections 2024-04-20T21:33:51.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:51.466+0000 7f5564e00700 1 -- 172.21.15.62:0/2515609651 wait complete. 2024-04-20T21:33:51.485 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T21:33:51.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:51.636+0000 7f60cc400700 1 -- 172.21.15.62:0/2983579411 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60c4100240 msgr2=0x7f60c4100650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:33:51.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:51.636+0000 7f60cc400700 1 --2- 172.21.15.62:0/2983579411 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60c4100240 0x7f60c4100650 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f60b8009910 tx=0x7f60b8009c20 comp rx=0 tx=0).stop 2024-04-20T21:33:51.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:51.636+0000 7f60cc400700 1 -- 172.21.15.62:0/2983579411 shutdown_connections 2024-04-20T21:33:51.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:51.636+0000 7f60cc400700 1 --2- 172.21.15.62:0/2983579411 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60c4101440 0x7f60c4101890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:33:51.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:51.636+0000 7f60cc400700 1 --2- 172.21.15.62:0/2983579411 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60c4100240 0x7f60c4100650 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:33:51.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:51.636+0000 7f60cc400700 1 -- 172.21.15.62:0/2983579411 >> 172.21.15.62:0/2983579411 conn(0x7f60c40fb7f0 msgr2=0x7f60c40fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:33:51.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:51.636+0000 7f60cc400700 1 -- 172.21.15.62:0/2983579411 shutdown_connections 2024-04-20T21:33:51.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:51.637+0000 7f60cc400700 1 -- 172.21.15.62:0/2983579411 wait complete. 2024-04-20T21:33:51.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:51.638+0000 7f60cc400700 1 Processor -- start 2024-04-20T21:33:51.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:51.638+0000 7f60cc400700 1 -- start start 2024-04-20T21:33:51.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:51.638+0000 7f60cc400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60c4100240 0x7f60c4074af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:33:51.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:51.638+0000 7f60cc400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60c4101440 0x7f60c4073140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:33:51.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:51.638+0000 7f60cc400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60c4073680 con 0x7f60c4100240 2024-04-20T21:33:51.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:51.638+0000 7f60cc400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60c40737c0 con 0x7f60c4101440 2024-04-20T21:33:51.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:51.639+0000 7f60c9a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60c4100240 0x7f60c4074af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:33:51.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:51.639+0000 7f60c9a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60c4100240 0x7f60c4074af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41590/0 (socket says 172.21.15.62:41590) 2024-04-20T21:33:51.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:51.639+0000 7f60c9a00700 1 -- 172.21.15.62:0/4029337526 learned_addr learned my addr 172.21.15.62:0/4029337526 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:33:51.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:51.639+0000 7f60c9000700 1 --2- 172.21.15.62:0/4029337526 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60c4101440 0x7f60c4073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:33:51.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:51.639+0000 7f60c9a00700 1 -- 172.21.15.62:0/4029337526 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60c4101440 msgr2=0x7f60c4073140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:33:51.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:51.639+0000 7f60c9a00700 1 --2- 172.21.15.62:0/4029337526 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60c4101440 0x7f60c4073140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:33:51.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:51.639+0000 7f60c9a00700 1 -- 172.21.15.62:0/4029337526 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60b80095f0 con 0x7f60c4100240 2024-04-20T21:33:51.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:51.640+0000 7f60c9a00700 1 --2- 172.21.15.62:0/4029337526 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60c4100240 0x7f60c4074af0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f60b800bd70 tx=0x7f60b800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:33:51.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:51.640+0000 7f60c2000700 1 -- 172.21.15.62:0/4029337526 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f60b801d070 con 0x7f60c4100240 2024-04-20T21:33:51.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:51.640+0000 7f60c2000700 1 -- 172.21.15.62:0/4029337526 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f60b800f460 con 0x7f60c4100240 2024-04-20T21:33:51.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:51.640+0000 7f60cc400700 1 -- 172.21.15.62:0/4029337526 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60c4073a40 con 0x7f60c4100240 2024-04-20T21:33:51.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:51.640+0000 7f60c2000700 1 -- 172.21.15.62:0/4029337526 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f60b8021640 con 0x7f60c4100240 2024-04-20T21:33:51.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:51.640+0000 7f60cc400700 1 -- 172.21.15.62:0/4029337526 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60c4073f30 con 0x7f60c4100240 2024-04-20T21:33:51.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:51.641+0000 7f60cc400700 1 -- 172.21.15.62:0/4029337526 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60c4108e10 con 0x7f60c4100240 2024-04-20T21:33:51.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:51.642+0000 7f60c2000700 1 -- 172.21.15.62:0/4029337526 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 31) v1 ==== 92598+0+0 (secure 0 0 0) 0x7f60b800fab0 con 0x7f60c4100240 2024-04-20T21:33:51.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:51.643+0000 7f60c2000700 1 --2- 172.21.15.62:0/4029337526 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f60ac06e940 0x7f60ac070df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:33:51.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:51.643+0000 7f60c2000700 1 -- 172.21.15.62:0/4029337526 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f60b808fd00 con 0x7f60c4100240 2024-04-20T21:33:51.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:51.644+0000 7f60c9000700 1 --2- 172.21.15.62:0/4029337526 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f60ac06e940 0x7f60ac070df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:33:51.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:51.644+0000 7f60c9000700 1 --2- 172.21.15.62:0/4029337526 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f60ac06e940 0x7f60ac070df0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f60b0006fd0 tx=0x7f60b0008040 comp rx=0 tx=0).ready entity=mgr.14658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:33:51.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:51.648+0000 7f60c2000700 1 -- 172.21.15.62:0/4029337526 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187527 (secure 0 0 0) 0x7f60b805caf0 con 0x7f60c4100240 2024-04-20T21:33:51.745 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:51 smithi062 ceph-mon[16221]: pgmap v70: 65 pgs: 65 active+clean; 317 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 839 KiB/s rd, 856 KiB/s wr, 89 op/s 2024-04-20T21:33:51.745 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:51 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:33:51.745 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:51 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:33:51.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:51 smithi189 ceph-mon[21562]: pgmap v70: 65 pgs: 65 active+clean; 317 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 839 KiB/s rd, 856 KiB/s wr, 89 op/s 2024-04-20T21:33:51.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:51 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:33:51.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:51 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:33:52.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:52.034+0000 7f60cc400700 1 -- 172.21.15.62:0/4029337526 --> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f60c419e040 con 0x7f60ac06e940 2024-04-20T21:33:52.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:52.035+0000 7f60c2000700 1 -- 172.21.15.62:0/4029337526 <== mgr.14658 v2:172.21.15.62:6800/1585760929 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+367 (secure 0 0 0) 0x7f60c419e040 con 0x7f60ac06e940 2024-04-20T21:33:52.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:52.038+0000 7f60cc400700 1 -- 172.21.15.62:0/4029337526 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f60ac06e940 msgr2=0x7f60ac070df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:33:52.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:52.038+0000 7f60cc400700 1 --2- 172.21.15.62:0/4029337526 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f60ac06e940 0x7f60ac070df0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f60b0006fd0 tx=0x7f60b0008040 comp rx=0 tx=0).stop 2024-04-20T21:33:52.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:52.038+0000 7f60cc400700 1 -- 172.21.15.62:0/4029337526 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60c4100240 msgr2=0x7f60c4074af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:33:52.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:52.038+0000 7f60cc400700 1 --2- 172.21.15.62:0/4029337526 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60c4100240 0x7f60c4074af0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f60b800bd70 tx=0x7f60b800be50 comp rx=0 tx=0).stop 2024-04-20T21:33:52.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:52.038+0000 7f60cc400700 1 -- 172.21.15.62:0/4029337526 shutdown_connections 2024-04-20T21:33:52.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:52.038+0000 7f60cc400700 1 --2- 172.21.15.62:0/4029337526 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f60ac06e940 0x7f60ac070df0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:33:52.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:52.038+0000 7f60cc400700 1 --2- 172.21.15.62:0/4029337526 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60c4100240 0x7f60c4074af0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:33:52.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:52.038+0000 7f60cc400700 1 --2- 172.21.15.62:0/4029337526 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60c4101440 0x7f60c4073140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:33:52.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:52.038+0000 7f60cc400700 1 -- 172.21.15.62:0/4029337526 >> 172.21.15.62:0/4029337526 conn(0x7f60c40fb7f0 msgr2=0x7f60c4104670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:33:52.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:52.038+0000 7f60cc400700 1 -- 172.21.15.62:0/4029337526 shutdown_connections 2024-04-20T21:33:52.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:52.038+0000 7f60cc400700 1 -- 172.21.15.62:0/4029337526 wait complete. 2024-04-20T21:33:52.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:52.224+0000 7f7bfbe00700 1 -- 172.21.15.62:0/4127349994 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7bf4074d80 msgr2=0x7f7bf40731e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:33:52.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:52.224+0000 7f7bfbe00700 1 --2- 172.21.15.62:0/4127349994 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7bf4074d80 0x7f7bf40731e0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f7be8009940 tx=0x7f7be8009c50 comp rx=0 tx=0).stop 2024-04-20T21:33:52.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:52.224+0000 7f7bfbe00700 1 -- 172.21.15.62:0/4127349994 shutdown_connections 2024-04-20T21:33:52.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:52.224+0000 7f7bfbe00700 1 --2- 172.21.15.62:0/4127349994 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7bf40737b0 0x7f7bf4073c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:33:52.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:52.224+0000 7f7bfbe00700 1 --2- 172.21.15.62:0/4127349994 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7bf4074d80 0x7f7bf40731e0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:33:52.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:52.224+0000 7f7bfbe00700 1 -- 172.21.15.62:0/4127349994 >> 172.21.15.62:0/4127349994 conn(0x7f7bf40fb7f0 msgr2=0x7f7bf40fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:33:52.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:52.225+0000 7f7bfbe00700 1 -- 172.21.15.62:0/4127349994 shutdown_connections 2024-04-20T21:33:52.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:52.226+0000 7f7bfbe00700 1 -- 172.21.15.62:0/4127349994 wait complete. 2024-04-20T21:33:52.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:52.227+0000 7f7bfbe00700 1 Processor -- start 2024-04-20T21:33:52.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:52.227+0000 7f7bfbe00700 1 -- start start 2024-04-20T21:33:52.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:52.227+0000 7f7bfbe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7bf40737b0 0x7f7bf4193890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:33:52.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:52.227+0000 7f7bfbe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7bf4074d80 0x7f7bf4193dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:33:52.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:52.227+0000 7f7bfbe00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf41943a0 con 0x7f7bf4074d80 2024-04-20T21:33:52.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:52.227+0000 7f7bfbe00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf41944e0 con 0x7f7bf40737b0 2024-04-20T21:33:52.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:52.228+0000 7f7bf8a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7bf4074d80 0x7f7bf4193dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:33:52.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:52.228+0000 7f7bf8a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7bf4074d80 0x7f7bf4193dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41612/0 (socket says 172.21.15.62:41612) 2024-04-20T21:33:52.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:52.228+0000 7f7bf8a00700 1 -- 172.21.15.62:0/2159546435 learned_addr learned my addr 172.21.15.62:0/2159546435 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:33:52.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:52.228+0000 7f7bf9400700 1 --2- 172.21.15.62:0/2159546435 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7bf40737b0 0x7f7bf4193890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:33:52.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:52.228+0000 7f7bf8a00700 1 -- 172.21.15.62:0/2159546435 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7bf40737b0 msgr2=0x7f7bf4193890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:33:52.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:52.228+0000 7f7bf8a00700 1 --2- 172.21.15.62:0/2159546435 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7bf40737b0 0x7f7bf4193890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:33:52.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:52.228+0000 7f7bf8a00700 1 -- 172.21.15.62:0/2159546435 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7be80095f0 con 0x7f7bf4074d80 2024-04-20T21:33:52.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:52.228+0000 7f7bf9400700 1 --2- 172.21.15.62:0/2159546435 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7bf40737b0 0x7f7bf4193890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-20T21:33:52.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:52.229+0000 7f7bf8a00700 1 --2- 172.21.15.62:0/2159546435 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7bf4074d80 0x7f7bf4193dd0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f7be000c8d0 tx=0x7f7be000cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:33:52.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:52.229+0000 7f7bf2000700 1 -- 172.21.15.62:0/2159546435 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7be0007a90 con 0x7f7bf4074d80 2024-04-20T21:33:52.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:52.229+0000 7f7bf2000700 1 -- 172.21.15.62:0/2159546435 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f7be0004ce0 con 0x7f7bf4074d80 2024-04-20T21:33:52.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:52.229+0000 7f7bf2000700 1 -- 172.21.15.62:0/2159546435 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7be00055f0 con 0x7f7bf4074d80 2024-04-20T21:33:52.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:52.229+0000 7f7bfbe00700 1 -- 172.21.15.62:0/2159546435 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bf41aa000 con 0x7f7bf4074d80 2024-04-20T21:33:52.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:52.230+0000 7f7bfbe00700 1 -- 172.21.15.62:0/2159546435 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bf41aa520 con 0x7f7bf4074d80 2024-04-20T21:33:52.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:52.231+0000 7f7bfbe00700 1 -- 172.21.15.62:0/2159546435 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bf4104a90 con 0x7f7bf4074d80 2024-04-20T21:33:52.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:52.233+0000 7f7bf2000700 1 -- 172.21.15.62:0/2159546435 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 31) v1 ==== 92598+0+0 (secure 0 0 0) 0x7f7be0004750 con 0x7f7bf4074d80 2024-04-20T21:33:52.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:52.235+0000 7f7bf2000700 1 --2- 172.21.15.62:0/2159546435 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f7bdc06e990 0x7f7bdc070e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:33:52.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:52.235+0000 7f7bf2000700 1 -- 172.21.15.62:0/2159546435 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f7be008e500 con 0x7f7bf4074d80 2024-04-20T21:33:52.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:52.235+0000 7f7bf9400700 1 --2- 172.21.15.62:0/2159546435 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f7bdc06e990 0x7f7bdc070e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:33:52.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:52.236+0000 7f7bf9400700 1 --2- 172.21.15.62:0/2159546435 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f7bdc06e990 0x7f7bdc070e40 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f7be8006010 tx=0x7f7be800b9c0 comp rx=0 tx=0).ready entity=mgr.14658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:33:52.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:52.240+0000 7f7bf2000700 1 -- 172.21.15.62:0/2159546435 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187527 (secure 0 0 0) 0x7f7be005b2f0 con 0x7f7bf4074d80 2024-04-20T21:33:52.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:52.557+0000 7f7bfbe00700 1 -- 172.21.15.62:0/2159546435 --> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f7bf4061190 con 0x7f7bdc06e990 2024-04-20T21:33:52.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:52.574+0000 7f7bf2000700 1 -- 172.21.15.62:0/2159546435 <== mgr.14658 v2:172.21.15.62:6800/1585760929 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f7bf4061190 con 0x7f7bdc06e990 2024-04-20T21:33:52.575 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T21:33:52.576 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (11m) 3s ago 13m 14.4M - 0.25.0 c8568f914cd2 957f663a9cd7 2024-04-20T21:33:52.576 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (13m) 3s ago 13m 8112k - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T21:33:52.576 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (12m) 41s ago 12m 7876k - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T21:33:52.576 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (13m) 3s ago 13m 6656k - 18.2.0 dc2bc1663786 36e10ef5be39 2024-04-20T21:33:52.576 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (12m) 41s ago 12m 6665k - 18.2.0 dc2bc1663786 4101f9953e3d 2024-04-20T21:33:52.576 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (11m) 3s ago 13m 74.1M - 9.4.7 954c08fa6188 99363bb095d8 2024-04-20T21:33:52.576 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (6m) 3s ago 6m 148M - 18.2.0 dc2bc1663786 5e6460aa6159 2024-04-20T21:33:52.576 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (6m) 3s ago 6m 178M - 18.2.0 dc2bc1663786 e65f2267544e 2024-04-20T21:33:52.576 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (6m) 41s ago 6m 151M - 18.2.0 dc2bc1663786 fb2a8a223db8 2024-04-20T21:33:52.576 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (6m) 41s ago 6m 114M - 18.2.0 dc2bc1663786 f2eec2ee8951 2024-04-20T21:33:52.576 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (2m) 3s ago 15m 501M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T21:33:52.576 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (77s) 41s ago 12m 448M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T21:33:52.576 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (15m) 3s ago 15m 56.6M 2048M 18.2.0 dc2bc1663786 bd6efbd5c380 2024-04-20T21:33:52.576 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (12m) 41s ago 12m 45.7M 2048M 18.2.0 dc2bc1663786 3735c8e5db9a 2024-04-20T21:33:52.577 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (53s) 3s ago 13m 7331k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T21:33:52.577 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (46s) 41s ago 12m 3003k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T21:33:52.577 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (11m) 3s ago 11m 190M 1216M 18.2.0 dc2bc1663786 3f315c3afc1f 2024-04-20T21:33:52.577 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (10m) 3s ago 10m 194M 1216M 18.2.0 dc2bc1663786 44e4d07416ad 2024-04-20T21:33:52.577 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (9m) 3s ago 9m 159M 1216M 18.2.0 dc2bc1663786 31c5d9586e98 2024-04-20T21:33:52.577 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (9m) 41s ago 9m 211M 2240M 18.2.0 dc2bc1663786 611c1e242693 2024-04-20T21:33:52.577 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (8m) 41s ago 8m 190M 2240M 18.2.0 dc2bc1663786 6c8fce91d45d 2024-04-20T21:33:52.577 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (8m) 41s ago 8m 189M 2240M 18.2.0 dc2bc1663786 8e4574bae8db 2024-04-20T21:33:52.577 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (17s) 3s ago 12m 49.7M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T21:33:52.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:52.578+0000 7f7bfbe00700 1 -- 172.21.15.62:0/2159546435 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f7bdc06e990 msgr2=0x7f7bdc070e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:33:52.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:52.578+0000 7f7bfbe00700 1 --2- 172.21.15.62:0/2159546435 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f7bdc06e990 0x7f7bdc070e40 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f7be8006010 tx=0x7f7be800b9c0 comp rx=0 tx=0).stop 2024-04-20T21:33:52.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:52.578+0000 7f7bfbe00700 1 -- 172.21.15.62:0/2159546435 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7bf4074d80 msgr2=0x7f7bf4193dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:33:52.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:52.578+0000 7f7bfbe00700 1 --2- 172.21.15.62:0/2159546435 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7bf4074d80 0x7f7bf4193dd0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f7be000c8d0 tx=0x7f7be000cc90 comp rx=0 tx=0).stop 2024-04-20T21:33:52.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:52.578+0000 7f7bfbe00700 1 -- 172.21.15.62:0/2159546435 shutdown_connections 2024-04-20T21:33:52.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:52.578+0000 7f7bfbe00700 1 --2- 172.21.15.62:0/2159546435 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f7bdc06e990 0x7f7bdc070e40 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:33:52.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:52.578+0000 7f7bfbe00700 1 --2- 172.21.15.62:0/2159546435 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7bf40737b0 0x7f7bf4193890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:33:52.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:52.578+0000 7f7bfbe00700 1 --2- 172.21.15.62:0/2159546435 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7bf4074d80 0x7f7bf4193dd0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:33:52.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:52.578+0000 7f7bfbe00700 1 -- 172.21.15.62:0/2159546435 >> 172.21.15.62:0/2159546435 conn(0x7f7bf40fb7f0 msgr2=0x7f7bf4101bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:33:52.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:52.578+0000 7f7bfbe00700 1 -- 172.21.15.62:0/2159546435 shutdown_connections 2024-04-20T21:33:52.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:52.578+0000 7f7bfbe00700 1 -- 172.21.15.62:0/2159546435 wait complete. 2024-04-20T21:33:52.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:52.748+0000 7f38d5400700 1 -- 172.21.15.62:0/422143019 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f38d0101440 msgr2=0x7f38d0101890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:33:52.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:52.748+0000 7f38d5400700 1 --2- 172.21.15.62:0/422143019 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f38d0101440 0x7f38d0101890 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f38bc009910 tx=0x7f38bc009c20 comp rx=0 tx=0).stop 2024-04-20T21:33:52.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:52.749+0000 7f38d5400700 1 -- 172.21.15.62:0/422143019 shutdown_connections 2024-04-20T21:33:52.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:52.749+0000 7f38d5400700 1 --2- 172.21.15.62:0/422143019 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f38d0101440 0x7f38d0101890 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:33:52.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:52.749+0000 7f38d5400700 1 --2- 172.21.15.62:0/422143019 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38d0100240 0x7f38d0100650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:33:52.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:52.749+0000 7f38d5400700 1 -- 172.21.15.62:0/422143019 >> 172.21.15.62:0/422143019 conn(0x7f38d00fb7f0 msgr2=0x7f38d00fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:33:52.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:52.749+0000 7f38d5400700 1 -- 172.21.15.62:0/422143019 shutdown_connections 2024-04-20T21:33:52.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:52.749+0000 7f38d5400700 1 -- 172.21.15.62:0/422143019 wait complete. 2024-04-20T21:33:52.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:52.750+0000 7f38d5400700 1 Processor -- start 2024-04-20T21:33:52.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:52.750+0000 7f38d5400700 1 -- start start 2024-04-20T21:33:52.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:52.750+0000 7f38d5400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38d0100240 0x7f38d0074af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:33:52.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:52.750+0000 7f38d5400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f38d0101440 0x7f38d0073140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:33:52.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:52.750+0000 7f38d5400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38d0073680 con 0x7f38d0101440 2024-04-20T21:33:52.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:52.750+0000 7f38d5400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38d00737c0 con 0x7f38d0100240 2024-04-20T21:33:52.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:52.751+0000 7f38cde00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f38d0101440 0x7f38d0073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:33:52.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:52.751+0000 7f38cde00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f38d0101440 0x7f38d0073140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41636/0 (socket says 172.21.15.62:41636) 2024-04-20T21:33:52.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:52.751+0000 7f38cde00700 1 -- 172.21.15.62:0/3927445739 learned_addr learned my addr 172.21.15.62:0/3927445739 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:33:52.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:52.751+0000 7f38ce800700 1 --2- 172.21.15.62:0/3927445739 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38d0100240 0x7f38d0074af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:33:52.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:52.751+0000 7f38cde00700 1 -- 172.21.15.62:0/3927445739 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38d0100240 msgr2=0x7f38d0074af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:33:52.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:52.751+0000 7f38cde00700 1 --2- 172.21.15.62:0/3927445739 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38d0100240 0x7f38d0074af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:33:52.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:52.751+0000 7f38cde00700 1 -- 172.21.15.62:0/3927445739 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38bc0095f0 con 0x7f38d0101440 2024-04-20T21:33:52.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:52.751+0000 7f38cde00700 1 --2- 172.21.15.62:0/3927445739 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f38d0101440 0x7f38d0073140 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f38bc005b40 tx=0x7f38bc00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:33:52.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:52.752+0000 7f38c7400700 1 -- 172.21.15.62:0/3927445739 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f38bc01d070 con 0x7f38d0101440 2024-04-20T21:33:52.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:52.752+0000 7f38c7400700 1 -- 172.21.15.62:0/3927445739 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f38bc00f460 con 0x7f38d0101440 2024-04-20T21:33:52.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:52.752+0000 7f38d5400700 1 -- 172.21.15.62:0/3927445739 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38d0073a40 con 0x7f38d0101440 2024-04-20T21:33:52.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:52.752+0000 7f38c7400700 1 -- 172.21.15.62:0/3927445739 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f38bc021640 con 0x7f38d0101440 2024-04-20T21:33:52.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:52.752+0000 7f38d5400700 1 -- 172.21.15.62:0/3927445739 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38d0073f30 con 0x7f38d0101440 2024-04-20T21:33:52.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:52.753+0000 7f38d5400700 1 -- 172.21.15.62:0/3927445739 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38d0108d90 con 0x7f38d0101440 2024-04-20T21:33:52.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:52.754+0000 7f38c7400700 1 -- 172.21.15.62:0/3927445739 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 31) v1 ==== 92598+0+0 (secure 0 0 0) 0x7f38bc00fa90 con 0x7f38d0101440 2024-04-20T21:33:52.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:52.755+0000 7f38c7400700 1 --2- 172.21.15.62:0/3927445739 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f38b406e940 0x7f38b4070df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:33:52.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:52.755+0000 7f38c7400700 1 -- 172.21.15.62:0/3927445739 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f38bc08fab0 con 0x7f38d0101440 2024-04-20T21:33:52.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:52.756+0000 7f38ce800700 1 --2- 172.21.15.62:0/3927445739 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f38b406e940 0x7f38b4070df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:33:52.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:52.757+0000 7f38ce800700 1 --2- 172.21.15.62:0/3927445739 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f38b406e940 0x7f38b4070df0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f38c0005950 tx=0x7f38c00058e0 comp rx=0 tx=0).ready entity=mgr.14658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:33:52.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:52.762+0000 7f38c7400700 1 -- 172.21.15.62:0/3927445739 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187527 (secure 0 0 0) 0x7f38bc05c940 con 0x7f38d0101440 2024-04-20T21:33:52.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:52 smithi189 ceph-mon[21562]: from='client.14772 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:33:52.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:52 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:33:53.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:52 smithi062 ceph-mon[16221]: from='client.14772 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:33:53.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:52 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:33:53.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:53.219+0000 7f38d5400700 1 -- 172.21.15.62:0/3927445739 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f38d019df30 con 0x7f38d0101440 2024-04-20T21:33:53.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:53.220+0000 7f38c7400700 1 -- 172.21.15.62:0/3927445739 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+675 (secure 0 0 0) 0x7f38bc05c4d0 con 0x7f38d0101440 2024-04-20T21:33:53.219 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:33:53.219 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T21:33:53.219 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 2 2024-04-20T21:33:53.220 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:33:53.220 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T21:33:53.220 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:33:53.220 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:33:53.220 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T21:33:53.220 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 6 2024-04-20T21:33:53.220 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:33:53.220 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T21:33:53.220 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 4 2024-04-20T21:33:53.220 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:33:53.220 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T21:33:53.220 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 12, 2024-04-20T21:33:53.220 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:33:53.220 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T21:33:53.220 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:33:53.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:53.222+0000 7f38d5400700 1 -- 172.21.15.62:0/3927445739 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f38b406e940 msgr2=0x7f38b4070df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:33:53.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:53.222+0000 7f38d5400700 1 --2- 172.21.15.62:0/3927445739 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f38b406e940 0x7f38b4070df0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f38c0005950 tx=0x7f38c00058e0 comp rx=0 tx=0).stop 2024-04-20T21:33:53.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:53.222+0000 7f38d5400700 1 -- 172.21.15.62:0/3927445739 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f38d0101440 msgr2=0x7f38d0073140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:33:53.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:53.222+0000 7f38d5400700 1 --2- 172.21.15.62:0/3927445739 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f38d0101440 0x7f38d0073140 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f38bc005b40 tx=0x7f38bc00be20 comp rx=0 tx=0).stop 2024-04-20T21:33:53.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:53.223+0000 7f38d5400700 1 -- 172.21.15.62:0/3927445739 shutdown_connections 2024-04-20T21:33:53.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:53.223+0000 7f38d5400700 1 --2- 172.21.15.62:0/3927445739 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f38b406e940 0x7f38b4070df0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:33:53.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:53.223+0000 7f38d5400700 1 --2- 172.21.15.62:0/3927445739 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38d0100240 0x7f38d0074af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:33:53.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:53.223+0000 7f38d5400700 1 --2- 172.21.15.62:0/3927445739 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f38d0101440 0x7f38d0073140 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:33:53.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:53.223+0000 7f38d5400700 1 -- 172.21.15.62:0/3927445739 >> 172.21.15.62:0/3927445739 conn(0x7f38d00fb7f0 msgr2=0x7f38d0104670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:33:53.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:53.223+0000 7f38d5400700 1 -- 172.21.15.62:0/3927445739 shutdown_connections 2024-04-20T21:33:53.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:53.223+0000 7f38d5400700 1 -- 172.21.15.62:0/3927445739 wait complete. 2024-04-20T21:33:53.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:53.402+0000 7feff4400700 1 -- 172.21.15.62:0/811482079 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fefec103310 msgr2=0x7fefec105790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:33:53.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:53.402+0000 7feff4400700 1 --2- 172.21.15.62:0/811482079 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fefec103310 0x7fefec105790 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fefd8009910 tx=0x7fefd8009c20 comp rx=0 tx=0).stop 2024-04-20T21:33:53.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:53.402+0000 7feff4400700 1 -- 172.21.15.62:0/811482079 shutdown_connections 2024-04-20T21:33:53.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:53.402+0000 7feff4400700 1 --2- 172.21.15.62:0/811482079 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fefec103310 0x7fefec105790 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:33:53.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:53.402+0000 7feff4400700 1 --2- 172.21.15.62:0/811482079 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fefec069270 0x7fefec102dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:33:53.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:53.402+0000 7feff4400700 1 -- 172.21.15.62:0/811482079 >> 172.21.15.62:0/811482079 conn(0x7fefec0fa780 msgr2=0x7fefec0fcbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:33:53.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:53.404+0000 7feff4400700 1 -- 172.21.15.62:0/811482079 shutdown_connections 2024-04-20T21:33:53.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:53.404+0000 7feff4400700 1 -- 172.21.15.62:0/811482079 wait complete. 2024-04-20T21:33:53.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:53.405+0000 7feff4400700 1 Processor -- start 2024-04-20T21:33:53.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:53.405+0000 7feff4400700 1 -- start start 2024-04-20T21:33:53.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:53.406+0000 7feff4400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fefec103310 0x7fefec071df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:33:53.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:53.406+0000 7feff4400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fefec072330 0x7fefec0fef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:33:53.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:53.406+0000 7feff4400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefec1a3510 con 0x7fefec103310 2024-04-20T21:33:53.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:53.406+0000 7feff4400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefec1a3680 con 0x7fefec072330 2024-04-20T21:33:53.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:53.407+0000 7feff1000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fefec072330 0x7fefec0fef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:33:53.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:53.407+0000 7feff1000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fefec072330 0x7fefec0fef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.62:38508/0 (socket says 172.21.15.62:38508) 2024-04-20T21:33:53.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:53.407+0000 7feff1000700 1 -- 172.21.15.62:0/959134623 learned_addr learned my addr 172.21.15.62:0/959134623 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:33:53.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:53.407+0000 7feff1a00700 1 --2- 172.21.15.62:0/959134623 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fefec103310 0x7fefec071df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:33:53.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:53.407+0000 7feff1000700 1 -- 172.21.15.62:0/959134623 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fefec103310 msgr2=0x7fefec071df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:33:53.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:53.408+0000 7feff1000700 1 --2- 172.21.15.62:0/959134623 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fefec103310 0x7fefec071df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:33:53.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:53.408+0000 7feff1000700 1 -- 172.21.15.62:0/959134623 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefd80095f0 con 0x7fefec072330 2024-04-20T21:33:53.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:53.408+0000 7feff1a00700 1 --2- 172.21.15.62:0/959134623 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fefec103310 0x7fefec071df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-20T21:33:53.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:53.408+0000 7feff1000700 1 --2- 172.21.15.62:0/959134623 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fefec072330 0x7fefec0fef80 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fefd8006010 tx=0x7fefd800bd70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:33:53.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:53.409+0000 7fefea000700 1 -- 172.21.15.62:0/959134623 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fefd801d070 con 0x7fefec072330 2024-04-20T21:33:53.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:53.409+0000 7fefea000700 1 -- 172.21.15.62:0/959134623 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fefd800f460 con 0x7fefec072330 2024-04-20T21:33:53.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:53.409+0000 7feff4400700 1 -- 172.21.15.62:0/959134623 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefec1a38a0 con 0x7fefec072330 2024-04-20T21:33:53.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:53.410+0000 7feff4400700 1 -- 172.21.15.62:0/959134623 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefec1a3d90 con 0x7fefec072330 2024-04-20T21:33:53.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:53.410+0000 7fefea000700 1 -- 172.21.15.62:0/959134623 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fefd8021620 con 0x7fefec072330 2024-04-20T21:33:53.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:53.412+0000 7feff4400700 1 -- 172.21.15.62:0/959134623 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefec069270 con 0x7fefec072330 2024-04-20T21:33:53.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:53.414+0000 7fefea000700 1 -- 172.21.15.62:0/959134623 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 31) v1 ==== 92598+0+0 (secure 0 0 0) 0x7fefd8003a60 con 0x7fefec072330 2024-04-20T21:33:53.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:53.415+0000 7fefea000700 1 --2- 172.21.15.62:0/959134623 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7fefd406e990 0x7fefd4070e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:33:53.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:53.415+0000 7fefea000700 1 -- 172.21.15.62:0/959134623 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6070+0+0 (secure 0 0 0) 0x7fefd808fc50 con 0x7fefec072330 2024-04-20T21:33:53.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:53.416+0000 7feff1a00700 1 --2- 172.21.15.62:0/959134623 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7fefd406e990 0x7fefd4070e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:33:53.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:53.416+0000 7feff1a00700 1 --2- 172.21.15.62:0/959134623 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7fefd406e990 0x7fefd4070e40 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fefe0005950 tx=0x7fefe000a300 comp rx=0 tx=0).ready entity=mgr.14658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:33:53.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:53.434+0000 7fefea000700 1 -- 172.21.15.62:0/959134623 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187527 (secure 0 0 0) 0x7fefd8058e90 con 0x7fefec072330 2024-04-20T21:33:53.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:53.802+0000 7feff4400700 1 -- 172.21.15.62:0/959134623 --> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fefec061190 con 0x7fefd406e990 2024-04-20T21:33:53.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:53.804+0000 7fefea000700 1 -- 172.21.15.62:0/959134623 <== mgr.14658 v2:172.21.15.62:6800/1585760929 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+367 (secure 0 0 0) 0x7fefec061190 con 0x7fefd406e990 2024-04-20T21:33:53.803 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:33:53.803 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T21:33:53.803 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T21:33:53.803 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-20T21:33:53.803 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T21:33:53.803 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr" 2024-04-20T21:33:53.803 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T21:33:53.803 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "2/2 daemons upgraded", 2024-04-20T21:33:53.804 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading prometheus daemons", 2024-04-20T21:33:53.804 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T21:33:53.804 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:33:53.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:53.808+0000 7feff4400700 1 -- 172.21.15.62:0/959134623 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7fefd406e990 msgr2=0x7fefd4070e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:33:53.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:53.808+0000 7feff4400700 1 --2- 172.21.15.62:0/959134623 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7fefd406e990 0x7fefd4070e40 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fefe0005950 tx=0x7fefe000a300 comp rx=0 tx=0).stop 2024-04-20T21:33:53.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:53.808+0000 7feff4400700 1 -- 172.21.15.62:0/959134623 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fefec072330 msgr2=0x7fefec0fef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:33:53.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:53.808+0000 7feff4400700 1 --2- 172.21.15.62:0/959134623 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fefec072330 0x7fefec0fef80 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fefd8006010 tx=0x7fefd800bd70 comp rx=0 tx=0).stop 2024-04-20T21:33:53.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:53.808+0000 7feff4400700 1 -- 172.21.15.62:0/959134623 shutdown_connections 2024-04-20T21:33:53.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:53.808+0000 7feff4400700 1 --2- 172.21.15.62:0/959134623 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7fefd406e990 0x7fefd4070e40 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:33:53.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:53.808+0000 7feff4400700 1 --2- 172.21.15.62:0/959134623 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fefec103310 0x7fefec071df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:33:53.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:53.808+0000 7feff4400700 1 --2- 172.21.15.62:0/959134623 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fefec072330 0x7fefec0fef80 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:33:53.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:53.808+0000 7feff4400700 1 -- 172.21.15.62:0/959134623 >> 172.21.15.62:0/959134623 conn(0x7fefec0fa780 msgr2=0x7fefec0fcbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:33:53.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:53.809+0000 7feff4400700 1 -- 172.21.15.62:0/959134623 shutdown_connections 2024-04-20T21:33:53.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:33:53.810+0000 7feff4400700 1 -- 172.21.15.62:0/959134623 wait complete. 2024-04-20T21:33:53.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:53 smithi189 ceph-mon[21562]: from='client.14776 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:33:53.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:53 smithi189 ceph-mon[21562]: from='client.14780 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:33:53.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:53 smithi189 ceph-mon[21562]: pgmap v71: 65 pgs: 65 active+clean; 317 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 839 KiB/s rd, 856 KiB/s wr, 89 op/s 2024-04-20T21:33:53.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:53 smithi189 ceph-mon[21562]: from='client.? 172.21.15.62:0/3927445739' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:33:54.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:53 smithi062 ceph-mon[16221]: from='client.14776 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:33:54.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:53 smithi062 ceph-mon[16221]: from='client.14780 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:33:54.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:53 smithi062 ceph-mon[16221]: pgmap v71: 65 pgs: 65 active+clean; 317 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 839 KiB/s rd, 856 KiB/s wr, 89 op/s 2024-04-20T21:33:54.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:53 smithi062 ceph-mon[16221]: from='client.? 172.21.15.62:0/3927445739' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:33:54.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:54 smithi189 ceph-mon[21562]: from='client.24545 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:33:55.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:54 smithi062 ceph-mon[16221]: from='client.24545 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:33:55.986 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:55 smithi189 ceph-mon[21562]: pgmap v72: 65 pgs: 65 active+clean; 319 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 135 op/s 2024-04-20T21:33:56.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:55 smithi062 ceph-mon[16221]: pgmap v72: 65 pgs: 65 active+clean; 319 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 135 op/s 2024-04-20T21:33:57.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:57 smithi062 ceph-mon[16221]: pgmap v73: 65 pgs: 65 active+clean; 319 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 849 KiB/s rd, 859 KiB/s wr, 91 op/s 2024-04-20T21:33:57.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:57 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:33:57.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:57 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:33:57.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:57 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:33:57.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:57 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T21:33:57.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:57 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:33:57.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:57 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-20T21:33:57.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:57 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:33:57.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:57 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:33:57.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:57 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:33:57.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:57 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:33:57.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:57 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:33:57.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:57 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:33:57.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:57 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:33:57.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:57 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:33:57.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:57 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:33:57.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:57 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:33:57.990 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:57 smithi189 ceph-mon[21562]: pgmap v73: 65 pgs: 65 active+clean; 319 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 849 KiB/s rd, 859 KiB/s wr, 91 op/s 2024-04-20T21:33:57.990 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:57 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:33:57.990 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:57 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:33:57.990 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:57 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:33:57.990 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:57 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T21:33:57.990 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:57 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:33:57.990 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:57 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-20T21:33:57.990 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:57 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:33:57.990 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:57 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:33:57.990 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:57 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:33:57.990 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:57 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:33:57.990 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:57 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:33:57.991 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:57 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:33:57.991 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:57 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:33:57.991 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:57 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:33:57.991 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:57 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:33:57.991 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:57 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:33:58.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:58 smithi189 ceph-mon[21562]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-20T21:33:59.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:58 smithi062 ceph-mon[16221]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-20T21:33:59.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:33:59 smithi189 ceph-mon[21562]: pgmap v74: 65 pgs: 65 active+clean; 322 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 142 op/s 2024-04-20T21:34:00.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:33:59 smithi062 ceph-mon[16221]: pgmap v74: 65 pgs: 65 active+clean; 322 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 142 op/s 2024-04-20T21:34:01.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:34:01 smithi189 ceph-mon[21562]: pgmap v75: 65 pgs: 65 active+clean; 322 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 870 KiB/s rd, 933 KiB/s wr, 97 op/s 2024-04-20T21:34:01.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:34:01 smithi189 ceph-mon[21562]: Upgrade: Updating alertmanager.smithi062 2024-04-20T21:34:01.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:34:01 smithi189 ceph-mon[21562]: Deploying daemon alertmanager.smithi062 on smithi062 2024-04-20T21:34:02.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:34:01 smithi062 ceph-mon[16221]: pgmap v75: 65 pgs: 65 active+clean; 322 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 870 KiB/s rd, 933 KiB/s wr, 97 op/s 2024-04-20T21:34:02.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:34:01 smithi062 ceph-mon[16221]: Upgrade: Updating alertmanager.smithi062 2024-04-20T21:34:02.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:34:01 smithi062 ceph-mon[16221]: Deploying daemon alertmanager.smithi062 on smithi062 2024-04-20T21:34:03.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:34:02 smithi062 ceph-mon[16221]: pgmap v76: 65 pgs: 65 active+clean; 322 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 870 KiB/s rd, 933 KiB/s wr, 97 op/s 2024-04-20T21:34:03.420 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:34:02 smithi189 ceph-mon[21562]: pgmap v76: 65 pgs: 65 active+clean; 322 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 870 KiB/s rd, 933 KiB/s wr, 97 op/s 2024-04-20T21:34:05.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:34:05 smithi189 ceph-mon[21562]: pgmap v77: 65 pgs: 65 active+clean; 324 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 136 op/s 2024-04-20T21:34:06.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:34:05 smithi062 ceph-mon[16221]: pgmap v77: 65 pgs: 65 active+clean; 324 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 136 op/s 2024-04-20T21:34:07.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:34:07 smithi189 ceph-mon[21562]: pgmap v78: 65 pgs: 65 active+clean; 324 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 907 KiB/s rd, 919 KiB/s wr, 89 op/s 2024-04-20T21:34:07.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:34:07 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:34:08.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:34:07 smithi062 ceph-mon[16221]: pgmap v78: 65 pgs: 65 active+clean; 324 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 907 KiB/s rd, 919 KiB/s wr, 89 op/s 2024-04-20T21:34:08.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:34:07 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:34:09.986 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:34:09 smithi189 ceph-mon[21562]: pgmap v79: 65 pgs: 65 active+clean; 322 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 139 op/s 2024-04-20T21:34:10.003 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:34:09 smithi062 ceph-mon[16221]: pgmap v79: 65 pgs: 65 active+clean; 322 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 139 op/s 2024-04-20T21:34:11.744 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:34:11 smithi062 ceph-mon[16221]: pgmap v80: 65 pgs: 65 active+clean; 322 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.0 MiB/s rd, 990 KiB/s wr, 88 op/s 2024-04-20T21:34:11.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:34:11 smithi189 ceph-mon[21562]: pgmap v80: 65 pgs: 65 active+clean; 322 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.0 MiB/s rd, 990 KiB/s wr, 88 op/s 2024-04-20T21:34:13.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:34:13 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:34:13.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:34:13 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:34:13.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:34:13 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:34:13.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:34:13 smithi189 ceph-mon[21562]: pgmap v81: 65 pgs: 65 active+clean; 322 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.0 MiB/s rd, 990 KiB/s wr, 88 op/s 2024-04-20T21:34:13.507 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:34:13 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:34:13.508 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:34:13 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:34:13.508 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:34:13 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:34:13.508 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:34:13 smithi062 ceph-mon[16221]: pgmap v81: 65 pgs: 65 active+clean; 322 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.0 MiB/s rd, 990 KiB/s wr, 88 op/s 2024-04-20T21:34:15.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:34:15 smithi189 ceph-mon[21562]: pgmap v82: 65 pgs: 65 active+clean; 322 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 149 op/s 2024-04-20T21:34:16.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:34:15 smithi062 ceph-mon[16221]: pgmap v82: 65 pgs: 65 active+clean; 322 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 149 op/s 2024-04-20T21:34:17.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:34:17 smithi189 ceph-mon[21562]: pgmap v83: 65 pgs: 65 active+clean; 322 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 898 KiB/s rd, 907 KiB/s wr, 111 op/s 2024-04-20T21:34:18.027 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:34:17 smithi062 ceph-mon[16221]: pgmap v83: 65 pgs: 65 active+clean; 322 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 898 KiB/s rd, 907 KiB/s wr, 111 op/s 2024-04-20T21:34:19.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:34:19 smithi062 ceph-mon[16221]: pgmap v84: 65 pgs: 65 active+clean; 322 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 181 op/s 2024-04-20T21:34:19.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:34:19 smithi189 ceph-mon[21562]: pgmap v84: 65 pgs: 65 active+clean; 322 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 181 op/s 2024-04-20T21:34:21.950 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:34:21 smithi062 ceph-mon[16221]: pgmap v85: 65 pgs: 65 active+clean; 322 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 729 KiB/s rd, 766 KiB/s wr, 132 op/s 2024-04-20T21:34:21.950 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:34:21 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:34:21.951 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:34:21 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:34:21.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:34:21 smithi189 ceph-mon[21562]: pgmap v85: 65 pgs: 65 active+clean; 322 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 729 KiB/s rd, 766 KiB/s wr, 132 op/s 2024-04-20T21:34:21.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:34:21 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:34:21.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:34:21 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:34:22.084 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.0... 2024-04-20T21:34:22.085 DEBUG:teuthology.orchestra.run.smithi062:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-04-20T21:34:22.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:34:22 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:34:23.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:34:22 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:34:23.353 DEBUG:teuthology.parallel:result is None 2024-04-20T21:34:23.986 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:34:23 smithi062 ceph-mon[16221]: pgmap v86: 65 pgs: 65 active+clean; 322 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 729 KiB/s rd, 766 KiB/s wr, 132 op/s 2024-04-20T21:34:23.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:23.987+0000 7f98ef400700 1 -- 172.21.15.62:0/3914383483 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f98e8069270 msgr2=0x7f98e8102dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:34:23.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:23.987+0000 7f98ef400700 1 --2- 172.21.15.62:0/3914383483 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f98e8069270 0x7f98e8102dd0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f98dc009910 tx=0x7f98dc009c20 comp rx=0 tx=0).stop 2024-04-20T21:34:23.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:23.987+0000 7f98ef400700 1 -- 172.21.15.62:0/3914383483 shutdown_connections 2024-04-20T21:34:23.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:23.987+0000 7f98ef400700 1 --2- 172.21.15.62:0/3914383483 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98e8103310 0x7f98e8105790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:34:23.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:23.987+0000 7f98ef400700 1 --2- 172.21.15.62:0/3914383483 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f98e8069270 0x7f98e8102dd0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:34:23.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:23.987+0000 7f98ef400700 1 -- 172.21.15.62:0/3914383483 >> 172.21.15.62:0/3914383483 conn(0x7f98e80fa780 msgr2=0x7f98e80fcbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:34:23.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:23.987+0000 7f98ef400700 1 -- 172.21.15.62:0/3914383483 shutdown_connections 2024-04-20T21:34:23.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:23.987+0000 7f98ef400700 1 -- 172.21.15.62:0/3914383483 wait complete. 2024-04-20T21:34:23.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:34:23 smithi189 ceph-mon[21562]: pgmap v86: 65 pgs: 65 active+clean; 322 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 729 KiB/s rd, 766 KiB/s wr, 132 op/s 2024-04-20T21:34:23.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:23.989+0000 7f98ef400700 1 Processor -- start 2024-04-20T21:34:23.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:23.989+0000 7f98ef400700 1 -- start start 2024-04-20T21:34:23.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:23.989+0000 7f98ef400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98e8103310 0x7f98e8100720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:34:23.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:23.989+0000 7f98ef400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f98e80fed70 0x7f98e80ff1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:34:23.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:23.989+0000 7f98ef400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98e80ff720 con 0x7f98e80fed70 2024-04-20T21:34:23.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:23.989+0000 7f98ef400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98e80ff890 con 0x7f98e8103310 2024-04-20T21:34:23.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:23.990+0000 7f98e7e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f98e80fed70 0x7f98e80ff1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:34:23.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:23.990+0000 7f98e7e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f98e80fed70 0x7f98e80ff1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48266/0 (socket says 172.21.15.62:48266) 2024-04-20T21:34:23.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:23.990+0000 7f98e7e00700 1 -- 172.21.15.62:0/1241018164 learned_addr learned my addr 172.21.15.62:0/1241018164 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:34:23.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:23.990+0000 7f98eca00700 1 --2- 172.21.15.62:0/1241018164 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98e8103310 0x7f98e8100720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:34:23.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:23.990+0000 7f98e7e00700 1 -- 172.21.15.62:0/1241018164 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98e8103310 msgr2=0x7f98e8100720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:34:23.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:23.990+0000 7f98e7e00700 1 --2- 172.21.15.62:0/1241018164 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98e8103310 0x7f98e8100720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:34:23.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:23.990+0000 7f98e7e00700 1 -- 172.21.15.62:0/1241018164 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98dc0095f0 con 0x7f98e80fed70 2024-04-20T21:34:23.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:23.991+0000 7f98e7e00700 1 --2- 172.21.15.62:0/1241018164 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f98e80fed70 0x7f98e80ff1e0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f98d400b4b0 tx=0x7f98d400b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:34:23.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:23.991+0000 7f98e5600700 1 -- 172.21.15.62:0/1241018164 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f98d4011800 con 0x7f98e80fed70 2024-04-20T21:34:23.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:23.991+0000 7f98ef400700 1 -- 172.21.15.62:0/1241018164 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98e80ffb70 con 0x7f98e80fed70 2024-04-20T21:34:23.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:23.991+0000 7f98e5600700 1 -- 172.21.15.62:0/1241018164 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f98d4011e10 con 0x7f98e80fed70 2024-04-20T21:34:23.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:23.991+0000 7f98e5600700 1 -- 172.21.15.62:0/1241018164 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f98d400f2e0 con 0x7f98e80fed70 2024-04-20T21:34:23.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:23.992+0000 7f98ef400700 1 -- 172.21.15.62:0/1241018164 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98e819de00 con 0x7f98e80fed70 2024-04-20T21:34:23.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:23.993+0000 7f98ef400700 1 -- 172.21.15.62:0/1241018164 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98e818fc50 con 0x7f98e80fed70 2024-04-20T21:34:23.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:23.994+0000 7f98e5600700 1 -- 172.21.15.62:0/1241018164 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 31) v1 ==== 92598+0+0 (secure 0 0 0) 0x7f98d4011960 con 0x7f98e80fed70 2024-04-20T21:34:23.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:23.995+0000 7f98e5600700 1 --2- 172.21.15.62:0/1241018164 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f98d006e940 0x7f98d0070df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:34:23.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:23.995+0000 7f98e5600700 1 -- 172.21.15.62:0/1241018164 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f98d408e430 con 0x7f98e80fed70 2024-04-20T21:34:23.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:23.995+0000 7f98eca00700 1 --2- 172.21.15.62:0/1241018164 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f98d006e940 0x7f98d0070df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:34:23.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:23.996+0000 7f98eca00700 1 --2- 172.21.15.62:0/1241018164 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f98d006e940 0x7f98d0070df0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f98dc006010 tx=0x7f98dc00b540 comp rx=0 tx=0).ready entity=mgr.14658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:34:24.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:24.001+0000 7f98e5600700 1 -- 172.21.15.62:0/1241018164 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187527 (secure 0 0 0) 0x7f98d405b220 con 0x7f98e80fed70 2024-04-20T21:34:24.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:24.328+0000 7f98ef400700 1 -- 172.21.15.62:0/1241018164 --> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f98e802ceb0 con 0x7f98d006e940 2024-04-20T21:34:24.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:24.330+0000 7f98e5600700 1 -- 172.21.15.62:0/1241018164 <== mgr.14658 v2:172.21.15.62:6800/1585760929 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+369 (secure 0 0 0) 0x7f98e802ceb0 con 0x7f98d006e940 2024-04-20T21:34:24.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:24.333+0000 7f98ef400700 1 -- 172.21.15.62:0/1241018164 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f98d006e940 msgr2=0x7f98d0070df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:34:24.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:24.333+0000 7f98ef400700 1 --2- 172.21.15.62:0/1241018164 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f98d006e940 0x7f98d0070df0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f98dc006010 tx=0x7f98dc00b540 comp rx=0 tx=0).stop 2024-04-20T21:34:24.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:24.333+0000 7f98ef400700 1 -- 172.21.15.62:0/1241018164 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f98e80fed70 msgr2=0x7f98e80ff1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:34:24.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:24.333+0000 7f98ef400700 1 --2- 172.21.15.62:0/1241018164 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f98e80fed70 0x7f98e80ff1e0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f98d400b4b0 tx=0x7f98d400b7c0 comp rx=0 tx=0).stop 2024-04-20T21:34:24.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:24.334+0000 7f98ef400700 1 -- 172.21.15.62:0/1241018164 shutdown_connections 2024-04-20T21:34:24.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:24.334+0000 7f98ef400700 1 --2- 172.21.15.62:0/1241018164 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f98d006e940 0x7f98d0070df0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:34:24.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:24.334+0000 7f98ef400700 1 --2- 172.21.15.62:0/1241018164 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98e8103310 0x7f98e8100720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:34:24.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:24.334+0000 7f98ef400700 1 --2- 172.21.15.62:0/1241018164 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f98e80fed70 0x7f98e80ff1e0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:34:24.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:24.334+0000 7f98ef400700 1 -- 172.21.15.62:0/1241018164 >> 172.21.15.62:0/1241018164 conn(0x7f98e80fa780 msgr2=0x7f98e80fcbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:34:24.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:24.334+0000 7f98ef400700 1 -- 172.21.15.62:0/1241018164 shutdown_connections 2024-04-20T21:34:24.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:24.334+0000 7f98ef400700 1 -- 172.21.15.62:0/1241018164 wait complete. 2024-04-20T21:34:24.348 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T21:34:24.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:24.491+0000 7f359ba00700 1 -- 172.21.15.62:0/1835157232 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3594101440 msgr2=0x7f3594101890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:34:24.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:24.491+0000 7f359ba00700 1 --2- 172.21.15.62:0/1835157232 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3594101440 0x7f3594101890 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f3580009910 tx=0x7f3580009c20 comp rx=0 tx=0).stop 2024-04-20T21:34:24.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:24.491+0000 7f359ba00700 1 -- 172.21.15.62:0/1835157232 shutdown_connections 2024-04-20T21:34:24.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:24.491+0000 7f359ba00700 1 --2- 172.21.15.62:0/1835157232 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3594101440 0x7f3594101890 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:34:24.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:24.491+0000 7f359ba00700 1 --2- 172.21.15.62:0/1835157232 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3594100240 0x7f3594100650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:34:24.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:24.491+0000 7f359ba00700 1 -- 172.21.15.62:0/1835157232 >> 172.21.15.62:0/1835157232 conn(0x7f35940fb7f0 msgr2=0x7f35940fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:34:24.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:24.491+0000 7f359ba00700 1 -- 172.21.15.62:0/1835157232 shutdown_connections 2024-04-20T21:34:24.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:24.491+0000 7f359ba00700 1 -- 172.21.15.62:0/1835157232 wait complete. 2024-04-20T21:34:24.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:24.492+0000 7f359ba00700 1 Processor -- start 2024-04-20T21:34:24.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:24.493+0000 7f359ba00700 1 -- start start 2024-04-20T21:34:24.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:24.493+0000 7f359ba00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3594100240 0x7f3594195ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:34:24.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:24.493+0000 7f359ba00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3594101440 0x7f3594195ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:34:24.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:24.493+0000 7f359ba00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3594196610 con 0x7f3594100240 2024-04-20T21:34:24.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:24.493+0000 7f359ba00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3594196750 con 0x7f3594101440 2024-04-20T21:34:24.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:24.493+0000 7f3599000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3594100240 0x7f3594195ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:34:24.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:24.493+0000 7f3599000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3594100240 0x7f3594195ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48284/0 (socket says 172.21.15.62:48284) 2024-04-20T21:34:24.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:24.493+0000 7f3599000700 1 -- 172.21.15.62:0/113161001 learned_addr learned my addr 172.21.15.62:0/113161001 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:34:24.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:24.494+0000 7f3593e00700 1 --2- 172.21.15.62:0/113161001 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3594101440 0x7f3594195ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:34:24.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:24.494+0000 7f3599000700 1 -- 172.21.15.62:0/113161001 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3594101440 msgr2=0x7f3594195ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:34:24.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:24.494+0000 7f3599000700 1 --2- 172.21.15.62:0/113161001 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3594101440 0x7f3594195ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:34:24.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:24.494+0000 7f3599000700 1 -- 172.21.15.62:0/113161001 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35800095f0 con 0x7f3594100240 2024-04-20T21:34:24.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:24.494+0000 7f3593e00700 1 --2- 172.21.15.62:0/113161001 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3594101440 0x7f3594195ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T21:34:24.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:24.494+0000 7f3599000700 1 --2- 172.21.15.62:0/113161001 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3594100240 0x7f3594195ab0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f358800c8f0 tx=0x7f358800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:34:24.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:24.494+0000 7f3591600700 1 -- 172.21.15.62:0/113161001 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3588007a70 con 0x7f3594100240 2024-04-20T21:34:24.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:24.494+0000 7f359ba00700 1 -- 172.21.15.62:0/113161001 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f359419b200 con 0x7f3594100240 2024-04-20T21:34:24.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:24.494+0000 7f3591600700 1 -- 172.21.15.62:0/113161001 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f3588007bd0 con 0x7f3594100240 2024-04-20T21:34:24.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:24.494+0000 7f3591600700 1 -- 172.21.15.62:0/113161001 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3588018630 con 0x7f3594100240 2024-04-20T21:34:24.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:24.495+0000 7f359ba00700 1 -- 172.21.15.62:0/113161001 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f359419b750 con 0x7f3594100240 2024-04-20T21:34:24.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:24.496+0000 7f359ba00700 1 -- 172.21.15.62:0/113161001 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3594108d90 con 0x7f3594100240 2024-04-20T21:34:24.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:24.497+0000 7f3591600700 1 -- 172.21.15.62:0/113161001 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 31) v1 ==== 92598+0+0 (secure 0 0 0) 0x7f358800f450 con 0x7f3594100240 2024-04-20T21:34:24.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:24.498+0000 7f3591600700 1 --2- 172.21.15.62:0/113161001 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f357c06e940 0x7f357c070df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:34:24.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:24.498+0000 7f3591600700 1 -- 172.21.15.62:0/113161001 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f358808e8a0 con 0x7f3594100240 2024-04-20T21:34:24.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:24.499+0000 7f3593e00700 1 --2- 172.21.15.62:0/113161001 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f357c06e940 0x7f357c070df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:34:24.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:24.499+0000 7f3593e00700 1 --2- 172.21.15.62:0/113161001 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f357c06e940 0x7f357c070df0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f3580006010 tx=0x7f358000b540 comp rx=0 tx=0).ready entity=mgr.14658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:34:24.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:24.503+0000 7f3591600700 1 -- 172.21.15.62:0/113161001 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187527 (secure 0 0 0) 0x7f358805b740 con 0x7f3594100240 2024-04-20T21:34:24.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:24.784+0000 7f359ba00700 1 -- 172.21.15.62:0/113161001 --> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3594105d90 con 0x7f357c06e940 2024-04-20T21:34:24.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:24.786+0000 7f3591600700 1 -- 172.21.15.62:0/113161001 <== mgr.14658 v2:172.21.15.62:6800/1585760929 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+369 (secure 0 0 0) 0x7f3594105d90 con 0x7f357c06e940 2024-04-20T21:34:24.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:24.789+0000 7f359ba00700 1 -- 172.21.15.62:0/113161001 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f357c06e940 msgr2=0x7f357c070df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:34:24.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:24.789+0000 7f359ba00700 1 --2- 172.21.15.62:0/113161001 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f357c06e940 0x7f357c070df0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f3580006010 tx=0x7f358000b540 comp rx=0 tx=0).stop 2024-04-20T21:34:24.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:24.789+0000 7f359ba00700 1 -- 172.21.15.62:0/113161001 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3594100240 msgr2=0x7f3594195ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:34:24.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:24.789+0000 7f359ba00700 1 --2- 172.21.15.62:0/113161001 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3594100240 0x7f3594195ab0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f358800c8f0 tx=0x7f358800ccb0 comp rx=0 tx=0).stop 2024-04-20T21:34:24.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:24.790+0000 7f359ba00700 1 -- 172.21.15.62:0/113161001 shutdown_connections 2024-04-20T21:34:24.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:24.790+0000 7f359ba00700 1 --2- 172.21.15.62:0/113161001 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f357c06e940 0x7f357c070df0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:34:24.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:24.790+0000 7f359ba00700 1 --2- 172.21.15.62:0/113161001 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3594100240 0x7f3594195ab0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:34:24.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:24.790+0000 7f359ba00700 1 --2- 172.21.15.62:0/113161001 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3594101440 0x7f3594195ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:34:24.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:24.790+0000 7f359ba00700 1 -- 172.21.15.62:0/113161001 >> 172.21.15.62:0/113161001 conn(0x7f35940fb7f0 msgr2=0x7f3594104670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:34:24.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:24.790+0000 7f359ba00700 1 -- 172.21.15.62:0/113161001 shutdown_connections 2024-04-20T21:34:24.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:24.790+0000 7f359ba00700 1 -- 172.21.15.62:0/113161001 wait complete. 2024-04-20T21:34:24.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:24.954+0000 7f783e600700 1 -- 172.21.15.62:0/3647882474 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7838100240 msgr2=0x7f7838100650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:34:24.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:24.954+0000 7f783e600700 1 --2- 172.21.15.62:0/3647882474 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7838100240 0x7f7838100650 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f7828009910 tx=0x7f7828009c20 comp rx=0 tx=0).stop 2024-04-20T21:34:24.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:24.954+0000 7f783e600700 1 -- 172.21.15.62:0/3647882474 shutdown_connections 2024-04-20T21:34:24.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:24.954+0000 7f783e600700 1 --2- 172.21.15.62:0/3647882474 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7838101440 0x7f7838101890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:34:24.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:24.954+0000 7f783e600700 1 --2- 172.21.15.62:0/3647882474 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7838100240 0x7f7838100650 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:34:24.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:24.954+0000 7f783e600700 1 -- 172.21.15.62:0/3647882474 >> 172.21.15.62:0/3647882474 conn(0x7f78380fb7f0 msgr2=0x7f78380fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:34:24.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:24.954+0000 7f783e600700 1 -- 172.21.15.62:0/3647882474 shutdown_connections 2024-04-20T21:34:24.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:24.954+0000 7f783e600700 1 -- 172.21.15.62:0/3647882474 wait complete. 2024-04-20T21:34:24.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:24.956+0000 7f783e600700 1 Processor -- start 2024-04-20T21:34:24.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:24.956+0000 7f783e600700 1 -- start start 2024-04-20T21:34:24.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:24.956+0000 7f783e600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7838100240 0x7f7838195ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:34:24.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:24.956+0000 7f783e600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7838101440 0x7f7838195ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:34:24.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:24.956+0000 7f783e600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7838196610 con 0x7f7838100240 2024-04-20T21:34:24.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:24.956+0000 7f783e600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7838196750 con 0x7f7838101440 2024-04-20T21:34:24.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:24.956+0000 7f7837400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7838100240 0x7f7838195ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:34:24.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:24.957+0000 7f7837400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7838100240 0x7f7838195ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48302/0 (socket says 172.21.15.62:48302) 2024-04-20T21:34:24.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:24.957+0000 7f7837400700 1 -- 172.21.15.62:0/3832670721 learned_addr learned my addr 172.21.15.62:0/3832670721 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:34:24.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:24.957+0000 7f782ea00700 1 --2- 172.21.15.62:0/3832670721 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7838101440 0x7f7838195ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:34:24.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:24.957+0000 7f7837400700 1 -- 172.21.15.62:0/3832670721 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7838101440 msgr2=0x7f7838195ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:34:24.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:24.957+0000 7f7837400700 1 --2- 172.21.15.62:0/3832670721 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7838101440 0x7f7838195ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:34:24.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:24.957+0000 7f7837400700 1 -- 172.21.15.62:0/3832670721 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78280095f0 con 0x7f7838100240 2024-04-20T21:34:24.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:24.957+0000 7f7837400700 1 --2- 172.21.15.62:0/3832670721 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7838100240 0x7f7838195ab0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f782800bd70 tx=0x7f782800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:34:24.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:24.958+0000 7f7834c00700 1 -- 172.21.15.62:0/3832670721 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f782801d070 con 0x7f7838100240 2024-04-20T21:34:24.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:24.958+0000 7f7834c00700 1 -- 172.21.15.62:0/3832670721 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f782800f460 con 0x7f7838100240 2024-04-20T21:34:24.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:24.958+0000 7f7834c00700 1 -- 172.21.15.62:0/3832670721 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7828021640 con 0x7f7838100240 2024-04-20T21:34:24.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:24.958+0000 7f783e600700 1 -- 172.21.15.62:0/3832670721 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f783819b1a0 con 0x7f7838100240 2024-04-20T21:34:24.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:24.958+0000 7f783e600700 1 -- 172.21.15.62:0/3832670721 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f783819b690 con 0x7f7838100240 2024-04-20T21:34:24.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:24.959+0000 7f783e600700 1 -- 172.21.15.62:0/3832670721 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7838108d90 con 0x7f7838100240 2024-04-20T21:34:24.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:24.960+0000 7f7834c00700 1 -- 172.21.15.62:0/3832670721 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 31) v1 ==== 92598+0+0 (secure 0 0 0) 0x7f782800fab0 con 0x7f7838100240 2024-04-20T21:34:24.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:24.961+0000 7f7834c00700 1 --2- 172.21.15.62:0/3832670721 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f781c06e940 0x7f781c070df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:34:24.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:24.961+0000 7f7834c00700 1 -- 172.21.15.62:0/3832670721 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f782808fbe0 con 0x7f7838100240 2024-04-20T21:34:24.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:24.961+0000 7f782ea00700 1 --2- 172.21.15.62:0/3832670721 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f781c06e940 0x7f781c070df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:34:24.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:24.962+0000 7f782ea00700 1 --2- 172.21.15.62:0/3832670721 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f781c06e940 0x7f781c070df0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f78200060b0 tx=0x7f7820007b80 comp rx=0 tx=0).ready entity=mgr.14658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:34:24.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:24.966+0000 7f7834c00700 1 -- 172.21.15.62:0/3832670721 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187527 (secure 0 0 0) 0x7f782805ca70 con 0x7f7838100240 2024-04-20T21:34:25.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:34:24 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:34:25.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:34:24 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:34:25.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:34:24 smithi062 ceph-mon[16221]: from='client.14792 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:34:25.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:34:24 smithi062 ceph-mon[16221]: pgmap v87: 65 pgs: 65 active+clean; 320 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 198 op/s 2024-04-20T21:34:25.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:34:24 smithi062 ceph-mon[16221]: from='client.14796 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:34:25.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:34:24 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:34:25.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:34:24 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:34:25.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:34:24 smithi189 ceph-mon[21562]: from='client.14792 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:34:25.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:34:24 smithi189 ceph-mon[21562]: pgmap v87: 65 pgs: 65 active+clean; 320 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 198 op/s 2024-04-20T21:34:25.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:34:24 smithi189 ceph-mon[21562]: from='client.14796 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:34:25.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:25.254+0000 7f783e600700 1 -- 172.21.15.62:0/3832670721 --> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f783819bde0 con 0x7f781c06e940 2024-04-20T21:34:25.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:25.287+0000 7f7834c00700 1 -- 172.21.15.62:0/3832670721 <== mgr.14658 v2:172.21.15.62:6800/1585760929 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7f783819bde0 con 0x7f781c06e940 2024-04-20T21:34:25.287 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T21:34:25.287 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (14s) 3s ago 14m 13.8M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T21:34:25.287 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (14m) 3s ago 14m 8158k - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T21:34:25.287 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (13m) 73s ago 13m 7876k - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T21:34:25.287 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (14m) 3s ago 14m 6656k - 18.2.0 dc2bc1663786 36e10ef5be39 2024-04-20T21:34:25.287 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (13m) 73s ago 13m 6665k - 18.2.0 dc2bc1663786 4101f9953e3d 2024-04-20T21:34:25.287 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (12m) 3s ago 13m 74.4M - 9.4.7 954c08fa6188 99363bb095d8 2024-04-20T21:34:25.287 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (6m) 3s ago 6m 87.0M - 18.2.0 dc2bc1663786 5e6460aa6159 2024-04-20T21:34:25.287 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (6m) 3s ago 6m 159M - 18.2.0 dc2bc1663786 e65f2267544e 2024-04-20T21:34:25.287 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (6m) 73s ago 6m 151M - 18.2.0 dc2bc1663786 fb2a8a223db8 2024-04-20T21:34:25.288 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (6m) 73s ago 6m 114M - 18.2.0 dc2bc1663786 f2eec2ee8951 2024-04-20T21:34:25.288 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (3m) 3s ago 15m 502M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T21:34:25.288 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (109s) 73s ago 12m 448M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T21:34:25.288 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (15m) 3s ago 16m 57.5M 2048M 18.2.0 dc2bc1663786 bd6efbd5c380 2024-04-20T21:34:25.288 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (12m) 73s ago 12m 45.7M 2048M 18.2.0 dc2bc1663786 3735c8e5db9a 2024-04-20T21:34:25.288 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (86s) 3s ago 14m 7218k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T21:34:25.288 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (79s) 73s ago 12m 3003k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T21:34:25.288 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (11m) 3s ago 11m 213M 1216M 18.2.0 dc2bc1663786 3f315c3afc1f 2024-04-20T21:34:25.288 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (11m) 3s ago 11m 203M 1216M 18.2.0 dc2bc1663786 44e4d07416ad 2024-04-20T21:34:25.288 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (10m) 3s ago 10m 167M 1216M 18.2.0 dc2bc1663786 31c5d9586e98 2024-04-20T21:34:25.288 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (9m) 73s ago 9m 211M 2240M 18.2.0 dc2bc1663786 611c1e242693 2024-04-20T21:34:25.288 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (9m) 73s ago 9m 190M 2240M 18.2.0 dc2bc1663786 6c8fce91d45d 2024-04-20T21:34:25.288 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (8m) 73s ago 8m 189M 2240M 18.2.0 dc2bc1663786 8e4574bae8db 2024-04-20T21:34:25.289 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (49s) 3s ago 13m 59.8M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T21:34:25.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:25.289+0000 7f783e600700 1 -- 172.21.15.62:0/3832670721 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f781c06e940 msgr2=0x7f781c070df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:34:25.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:25.289+0000 7f783e600700 1 --2- 172.21.15.62:0/3832670721 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f781c06e940 0x7f781c070df0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f78200060b0 tx=0x7f7820007b80 comp rx=0 tx=0).stop 2024-04-20T21:34:25.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:25.289+0000 7f783e600700 1 -- 172.21.15.62:0/3832670721 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7838100240 msgr2=0x7f7838195ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:34:25.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:25.289+0000 7f783e600700 1 --2- 172.21.15.62:0/3832670721 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7838100240 0x7f7838195ab0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f782800bd70 tx=0x7f782800be50 comp rx=0 tx=0).stop 2024-04-20T21:34:25.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:25.290+0000 7f783e600700 1 -- 172.21.15.62:0/3832670721 shutdown_connections 2024-04-20T21:34:25.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:25.290+0000 7f783e600700 1 --2- 172.21.15.62:0/3832670721 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f781c06e940 0x7f781c070df0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:34:25.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:25.290+0000 7f783e600700 1 --2- 172.21.15.62:0/3832670721 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7838100240 0x7f7838195ab0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:34:25.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:25.290+0000 7f783e600700 1 --2- 172.21.15.62:0/3832670721 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7838101440 0x7f7838195ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:34:25.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:25.290+0000 7f783e600700 1 -- 172.21.15.62:0/3832670721 >> 172.21.15.62:0/3832670721 conn(0x7f78380fb7f0 msgr2=0x7f7838104670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:34:25.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:25.290+0000 7f783e600700 1 -- 172.21.15.62:0/3832670721 shutdown_connections 2024-04-20T21:34:25.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:25.290+0000 7f783e600700 1 -- 172.21.15.62:0/3832670721 wait complete. 2024-04-20T21:34:25.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:25.439+0000 7f7393200700 1 -- 172.21.15.62:0/2451511482 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7394100020 msgr2=0x7f7394100430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:34:25.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:25.439+0000 7f7393200700 1 --2- 172.21.15.62:0/2451511482 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7394100020 0x7f7394100430 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f7384009910 tx=0x7f7384009c20 comp rx=0 tx=0).stop 2024-04-20T21:34:25.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:25.439+0000 7f7393200700 1 -- 172.21.15.62:0/2451511482 shutdown_connections 2024-04-20T21:34:25.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:25.439+0000 7f7393200700 1 --2- 172.21.15.62:0/2451511482 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f73941012c0 0x7f7394101730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:34:25.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:25.439+0000 7f7393200700 1 --2- 172.21.15.62:0/2451511482 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7394100020 0x7f7394100430 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:34:25.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:25.439+0000 7f7393200700 1 -- 172.21.15.62:0/2451511482 >> 172.21.15.62:0/2451511482 conn(0x7f73940fb650 msgr2=0x7f73940fdaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:34:25.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:25.439+0000 7f7393200700 1 -- 172.21.15.62:0/2451511482 shutdown_connections 2024-04-20T21:34:25.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:25.440+0000 7f7393200700 1 -- 172.21.15.62:0/2451511482 wait complete. 2024-04-20T21:34:25.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:25.441+0000 7f7393200700 1 Processor -- start 2024-04-20T21:34:25.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:25.441+0000 7f7393200700 1 -- start start 2024-04-20T21:34:25.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:25.441+0000 7f7393200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7394100020 0x7f7394197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:34:25.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:25.441+0000 7f7393200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f73941012c0 0x7f7394197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:34:25.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:25.441+0000 7f7393200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7394198560 con 0x7f73941012c0 2024-04-20T21:34:25.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:25.441+0000 7f7393200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73941986a0 con 0x7f7394100020 2024-04-20T21:34:25.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:25.441+0000 7f738be00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f73941012c0 0x7f7394197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:34:25.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:25.442+0000 7f738be00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f73941012c0 0x7f7394197f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48318/0 (socket says 172.21.15.62:48318) 2024-04-20T21:34:25.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:25.442+0000 7f738be00700 1 -- 172.21.15.62:0/3528227358 learned_addr learned my addr 172.21.15.62:0/3528227358 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:34:25.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:25.442+0000 7f7391e00700 1 --2- 172.21.15.62:0/3528227358 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7394100020 0x7f7394197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:34:25.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:25.442+0000 7f738be00700 1 -- 172.21.15.62:0/3528227358 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7394100020 msgr2=0x7f7394197a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:34:25.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:25.442+0000 7f738be00700 1 --2- 172.21.15.62:0/3528227358 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7394100020 0x7f7394197a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:34:25.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:25.442+0000 7f738be00700 1 -- 172.21.15.62:0/3528227358 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73840095f0 con 0x7f73941012c0 2024-04-20T21:34:25.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:25.442+0000 7f738be00700 1 --2- 172.21.15.62:0/3528227358 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f73941012c0 0x7f7394197f40 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f737c00c8a0 tx=0x7f737c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:34:25.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:25.443+0000 7f738aa00700 1 -- 172.21.15.62:0/3528227358 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f737c00cea0 con 0x7f73941012c0 2024-04-20T21:34:25.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:25.443+0000 7f738aa00700 1 -- 172.21.15.62:0/3528227358 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f737c004830 con 0x7f73941012c0 2024-04-20T21:34:25.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:25.443+0000 7f738aa00700 1 -- 172.21.15.62:0/3528227358 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f737c0055b0 con 0x7f73941012c0 2024-04-20T21:34:25.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:25.443+0000 7f7393200700 1 -- 172.21.15.62:0/3528227358 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f739419d150 con 0x7f73941012c0 2024-04-20T21:34:25.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:25.443+0000 7f7393200700 1 -- 172.21.15.62:0/3528227358 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f739419d6a0 con 0x7f73941012c0 2024-04-20T21:34:25.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:25.444+0000 7f7393200700 1 -- 172.21.15.62:0/3528227358 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7394108b00 con 0x7f73941012c0 2024-04-20T21:34:25.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:25.445+0000 7f738aa00700 1 -- 172.21.15.62:0/3528227358 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 31) v1 ==== 92598+0+0 (secure 0 0 0) 0x7f737c007ad0 con 0x7f73941012c0 2024-04-20T21:34:25.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:25.446+0000 7f738aa00700 1 --2- 172.21.15.62:0/3528227358 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f737806e940 0x7f7378070df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:34:25.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:25.446+0000 7f738aa00700 1 -- 172.21.15.62:0/3528227358 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f737c08e5b0 con 0x7f73941012c0 2024-04-20T21:34:25.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:25.446+0000 7f7391e00700 1 --2- 172.21.15.62:0/3528227358 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f737806e940 0x7f7378070df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:34:25.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:25.447+0000 7f7391e00700 1 --2- 172.21.15.62:0/3528227358 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f737806e940 0x7f7378070df0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f7384006010 tx=0x7f738400b540 comp rx=0 tx=0).ready entity=mgr.14658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:34:25.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:25.450+0000 7f738aa00700 1 -- 172.21.15.62:0/3528227358 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187527 (secure 0 0 0) 0x7f737c05b3a0 con 0x7f73941012c0 2024-04-20T21:34:25.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:25.790+0000 7f7393200700 1 -- 172.21.15.62:0/3528227358 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f7394108cc0 con 0x7f73941012c0 2024-04-20T21:34:25.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:25.791+0000 7f738aa00700 1 -- 172.21.15.62:0/3528227358 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+675 (secure 0 0 0) 0x7f737c05af30 con 0x7f73941012c0 2024-04-20T21:34:25.791 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:34:25.791 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T21:34:25.791 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 2 2024-04-20T21:34:25.791 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:34:25.791 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T21:34:25.791 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:34:25.791 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:34:25.791 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T21:34:25.791 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 6 2024-04-20T21:34:25.791 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:34:25.791 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T21:34:25.792 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 4 2024-04-20T21:34:25.792 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:34:25.792 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T21:34:25.792 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 12, 2024-04-20T21:34:25.792 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:34:25.792 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T21:34:25.792 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:34:25.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:25.795+0000 7f7393200700 1 -- 172.21.15.62:0/3528227358 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f737806e940 msgr2=0x7f7378070df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:34:25.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:25.795+0000 7f7393200700 1 --2- 172.21.15.62:0/3528227358 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f737806e940 0x7f7378070df0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f7384006010 tx=0x7f738400b540 comp rx=0 tx=0).stop 2024-04-20T21:34:25.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:25.795+0000 7f7393200700 1 -- 172.21.15.62:0/3528227358 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f73941012c0 msgr2=0x7f7394197f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:34:25.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:25.795+0000 7f7393200700 1 --2- 172.21.15.62:0/3528227358 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f73941012c0 0x7f7394197f40 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f737c00c8a0 tx=0x7f737c00cc60 comp rx=0 tx=0).stop 2024-04-20T21:34:25.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:25.795+0000 7f7393200700 1 -- 172.21.15.62:0/3528227358 shutdown_connections 2024-04-20T21:34:25.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:25.795+0000 7f7393200700 1 --2- 172.21.15.62:0/3528227358 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f737806e940 0x7f7378070df0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:34:25.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:25.795+0000 7f7393200700 1 --2- 172.21.15.62:0/3528227358 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7394100020 0x7f7394197a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:34:25.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:25.795+0000 7f7393200700 1 --2- 172.21.15.62:0/3528227358 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f73941012c0 0x7f7394197f40 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:34:25.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:25.795+0000 7f7393200700 1 -- 172.21.15.62:0/3528227358 >> 172.21.15.62:0/3528227358 conn(0x7f73940fb650 msgr2=0x7f73940fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:34:25.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:25.795+0000 7f7393200700 1 -- 172.21.15.62:0/3528227358 shutdown_connections 2024-04-20T21:34:25.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:25.795+0000 7f7393200700 1 -- 172.21.15.62:0/3528227358 wait complete. 2024-04-20T21:34:25.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:25.959+0000 7f39e2400700 1 -- 172.21.15.62:0/3346684924 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39dc0fe950 msgr2=0x7f39dc0fed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:34:25.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:25.959+0000 7f39e2400700 1 --2- 172.21.15.62:0/3346684924 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39dc0fe950 0x7f39dc0fed60 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f39d0009910 tx=0x7f39d0009c20 comp rx=0 tx=0).stop 2024-04-20T21:34:25.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:25.959+0000 7f39e2400700 1 -- 172.21.15.62:0/3346684924 shutdown_connections 2024-04-20T21:34:25.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:25.959+0000 7f39e2400700 1 --2- 172.21.15.62:0/3346684924 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39dc0ff2a0 0x7f39dc0ff710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:34:25.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:25.959+0000 7f39e2400700 1 --2- 172.21.15.62:0/3346684924 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39dc0fe950 0x7f39dc0fed60 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:34:25.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:25.959+0000 7f39e2400700 1 -- 172.21.15.62:0/3346684924 >> 172.21.15.62:0/3346684924 conn(0x7f39dc0fa580 msgr2=0x7f39dc0fc9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:34:25.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:25.959+0000 7f39e2400700 1 -- 172.21.15.62:0/3346684924 shutdown_connections 2024-04-20T21:34:25.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:25.959+0000 7f39e2400700 1 -- 172.21.15.62:0/3346684924 wait complete. 2024-04-20T21:34:25.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:25.960+0000 7f39e2400700 1 Processor -- start 2024-04-20T21:34:25.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:25.960+0000 7f39e2400700 1 -- start start 2024-04-20T21:34:25.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:25.961+0000 7f39e2400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39dc0fe950 0x7f39dc197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:34:25.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:25.961+0000 7f39e2400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39dc0ff2a0 0x7f39dc198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:34:25.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:25.961+0000 7f39e2400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39dc198640 con 0x7f39dc0fe950 2024-04-20T21:34:25.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:25.961+0000 7f39e2400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39dc198780 con 0x7f39dc0ff2a0 2024-04-20T21:34:25.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:25.961+0000 7f39e1000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39dc0fe950 0x7f39dc197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:34:25.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:25.961+0000 7f39e1000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39dc0fe950 0x7f39dc197ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48334/0 (socket says 172.21.15.62:48334) 2024-04-20T21:34:25.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:25.961+0000 7f39e1000700 1 -- 172.21.15.62:0/4188023603 learned_addr learned my addr 172.21.15.62:0/4188023603 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:34:25.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:25.961+0000 7f39dbe00700 1 --2- 172.21.15.62:0/4188023603 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39dc0ff2a0 0x7f39dc198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:34:25.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:25.962+0000 7f39e1000700 1 -- 172.21.15.62:0/4188023603 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39dc0ff2a0 msgr2=0x7f39dc198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:34:25.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:25.962+0000 7f39e1000700 1 --2- 172.21.15.62:0/4188023603 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39dc0ff2a0 0x7f39dc198020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:34:25.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:25.962+0000 7f39e1000700 1 -- 172.21.15.62:0/4188023603 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39d00095f0 con 0x7f39dc0fe950 2024-04-20T21:34:25.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:25.962+0000 7f39e1000700 1 --2- 172.21.15.62:0/4188023603 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39dc0fe950 0x7f39dc197ae0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f39d000bda0 tx=0x7f39d000be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:34:25.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:25.962+0000 7f39d9600700 1 -- 172.21.15.62:0/4188023603 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f39d001d070 con 0x7f39dc0fe950 2024-04-20T21:34:25.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:25.962+0000 7f39d9600700 1 -- 172.21.15.62:0/4188023603 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f39d0004c50 con 0x7f39dc0fe950 2024-04-20T21:34:25.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:25.962+0000 7f39e2400700 1 -- 172.21.15.62:0/4188023603 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39dc19d1d0 con 0x7f39dc0fe950 2024-04-20T21:34:25.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:25.962+0000 7f39d9600700 1 -- 172.21.15.62:0/4188023603 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f39d000f690 con 0x7f39dc0fe950 2024-04-20T21:34:25.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:25.963+0000 7f39e2400700 1 -- 172.21.15.62:0/4188023603 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39dc19d740 con 0x7f39dc0fe950 2024-04-20T21:34:25.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:25.964+0000 7f39e2400700 1 -- 172.21.15.62:0/4188023603 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39dc10ad70 con 0x7f39dc0fe950 2024-04-20T21:34:25.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:25.965+0000 7f39d9600700 1 -- 172.21.15.62:0/4188023603 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 31) v1 ==== 92598+0+0 (secure 0 0 0) 0x7f39d0022a20 con 0x7f39dc0fe950 2024-04-20T21:34:25.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:25.966+0000 7f39d9600700 1 --2- 172.21.15.62:0/4188023603 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f39c406e940 0x7f39c4070df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:34:25.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:25.966+0000 7f39d9600700 1 -- 172.21.15.62:0/4188023603 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f39d0090090 con 0x7f39dc0fe950 2024-04-20T21:34:25.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:25.966+0000 7f39dbe00700 1 --2- 172.21.15.62:0/4188023603 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f39c406e940 0x7f39c4070df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:34:25.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:25.967+0000 7f39dbe00700 1 --2- 172.21.15.62:0/4188023603 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f39c406e940 0x7f39c4070df0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f39cc006fd0 tx=0x7f39cc008040 comp rx=0 tx=0).ready entity=mgr.14658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:34:25.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:25.971+0000 7f39d9600700 1 -- 172.21.15.62:0/4188023603 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187527 (secure 0 0 0) 0x7f39d005ceb0 con 0x7f39dc0fe950 2024-04-20T21:34:26.193 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:34:25 smithi062 ceph-mon[16221]: from='client.14800 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:34:26.193 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:34:25 smithi062 ceph-mon[16221]: from='client.? 172.21.15.62:0/3528227358' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:34:26.236 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:34:25 smithi189 ceph-mon[21562]: from='client.14800 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:34:26.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:34:25 smithi189 ceph-mon[21562]: from='client.? 172.21.15.62:0/3528227358' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:34:26.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:26.273+0000 7f39e2400700 1 -- 172.21.15.62:0/4188023603 --> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f39dc1086a0 con 0x7f39c406e940 2024-04-20T21:34:26.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:26.277+0000 7f39d9600700 1 -- 172.21.15.62:0/4188023603 <== mgr.14658 v2:172.21.15.62:6800/1585760929 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+369 (secure 0 0 0) 0x7f39dc1086a0 con 0x7f39c406e940 2024-04-20T21:34:26.276 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:34:26.276 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T21:34:26.277 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T21:34:26.277 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-20T21:34:26.277 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T21:34:26.277 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr" 2024-04-20T21:34:26.277 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T21:34:26.277 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "2/2 daemons upgraded", 2024-04-20T21:34:26.277 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading alertmanager daemons", 2024-04-20T21:34:26.277 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T21:34:26.277 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:34:26.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:26.280+0000 7f39e2400700 1 -- 172.21.15.62:0/4188023603 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f39c406e940 msgr2=0x7f39c4070df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:34:26.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:26.280+0000 7f39e2400700 1 --2- 172.21.15.62:0/4188023603 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f39c406e940 0x7f39c4070df0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f39cc006fd0 tx=0x7f39cc008040 comp rx=0 tx=0).stop 2024-04-20T21:34:26.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:26.280+0000 7f39e2400700 1 -- 172.21.15.62:0/4188023603 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39dc0fe950 msgr2=0x7f39dc197ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:34:26.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:26.280+0000 7f39e2400700 1 --2- 172.21.15.62:0/4188023603 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39dc0fe950 0x7f39dc197ae0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f39d000bda0 tx=0x7f39d000be80 comp rx=0 tx=0).stop 2024-04-20T21:34:26.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:26.280+0000 7f39e2400700 1 -- 172.21.15.62:0/4188023603 shutdown_connections 2024-04-20T21:34:26.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:26.280+0000 7f39e2400700 1 --2- 172.21.15.62:0/4188023603 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f39c406e940 0x7f39c4070df0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:34:26.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:26.280+0000 7f39e2400700 1 --2- 172.21.15.62:0/4188023603 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39dc0fe950 0x7f39dc197ae0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:34:26.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:26.280+0000 7f39e2400700 1 --2- 172.21.15.62:0/4188023603 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39dc0ff2a0 0x7f39dc198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:34:26.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:26.280+0000 7f39e2400700 1 -- 172.21.15.62:0/4188023603 >> 172.21.15.62:0/4188023603 conn(0x7f39dc0fa580 msgr2=0x7f39dc106f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:34:26.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:26.281+0000 7f39e2400700 1 -- 172.21.15.62:0/4188023603 shutdown_connections 2024-04-20T21:34:26.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:26.281+0000 7f39e2400700 1 -- 172.21.15.62:0/4188023603 wait complete. 2024-04-20T21:34:27.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:34:26 smithi062 ceph-mon[16221]: from='client.14808 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:34:27.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:34:26 smithi062 ceph-mon[16221]: pgmap v88: 65 pgs: 65 active+clean; 320 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 758 KiB/s rd, 774 KiB/s wr, 136 op/s 2024-04-20T21:34:27.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:34:26 smithi189 ceph-mon[21562]: from='client.14808 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:34:27.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:34:26 smithi189 ceph-mon[21562]: pgmap v88: 65 pgs: 65 active+clean; 320 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 758 KiB/s rd, 774 KiB/s wr, 136 op/s 2024-04-20T21:34:30.095 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:34:29 smithi062 ceph-mon[16221]: pgmap v89: 65 pgs: 65 active+clean; 318 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 973 KiB/s rd, 971 KiB/s wr, 175 op/s 2024-04-20T21:34:30.095 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:34:29 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:34:30.095 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:34:29 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:34:30.095 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:34:29 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:34:30.095 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:34:29 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T21:34:30.095 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:34:29 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:34:30.095 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:34:29 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-20T21:34:30.095 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:34:29 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:34:30.095 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:34:29 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:34:30.095 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:34:29 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:34:30.095 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:34:29 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:34:30.096 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:34:29 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:34:30.096 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:34:29 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:34:30.096 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:34:29 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:34:30.096 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:34:29 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:34:30.096 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:34:29 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:34:30.096 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:34:29 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:34:30.096 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:34:29 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:34:30.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:34:29 smithi189 ceph-mon[21562]: pgmap v89: 65 pgs: 65 active+clean; 318 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 973 KiB/s rd, 971 KiB/s wr, 175 op/s 2024-04-20T21:34:30.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:34:29 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:34:30.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:34:29 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:34:30.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:34:29 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:34:30.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:34:29 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T21:34:30.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:34:29 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:34:30.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:34:29 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-20T21:34:30.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:34:29 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:34:30.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:34:29 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:34:30.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:34:29 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:34:30.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:34:29 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:34:30.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:34:29 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:34:30.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:34:29 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:34:30.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:34:29 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:34:30.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:34:29 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:34:30.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:34:29 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:34:30.239 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:34:29 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:34:30.239 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:34:29 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:34:31.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:34:30 smithi062 ceph-mon[16221]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-20T21:34:31.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:34:30 smithi189 ceph-mon[21562]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-20T21:34:31.941 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.1... 2024-04-20T21:34:31.941 DEBUG:teuthology.orchestra.run.smithi189:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.1 /home/ubuntu/cephtest/clone.client.1 2024-04-20T21:34:32.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:34:31 smithi062 ceph-mon[16221]: pgmap v90: 65 pgs: 65 active+clean; 318 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 583 KiB/s rd, 582 KiB/s wr, 104 op/s 2024-04-20T21:34:32.236 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:34:31 smithi189 ceph-mon[21562]: pgmap v90: 65 pgs: 65 active+clean; 318 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 583 KiB/s rd, 582 KiB/s wr, 104 op/s 2024-04-20T21:34:33.060 DEBUG:teuthology.parallel:result is None 2024-04-20T21:34:33.061 DEBUG:teuthology.orchestra.run.smithi062:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-04-20T21:34:33.153 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:34:32 smithi062 ceph-mon[16221]: Upgrade: Updating grafana.smithi062 2024-04-20T21:34:33.153 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:34:32 smithi062 ceph-mon[16221]: Deploying daemon grafana.smithi062 on smithi062 2024-04-20T21:34:33.190 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-04-20T21:34:33.191 DEBUG:teuthology.orchestra.run.smithi189:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.1/client.1 2024-04-20T21:34:33.236 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:34:32 smithi189 ceph-mon[21562]: Upgrade: Updating grafana.smithi062 2024-04-20T21:34:33.236 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:34:32 smithi189 ceph-mon[21562]: Deploying daemon grafana.smithi062 on smithi062 2024-04-20T21:34:33.301 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.1/client.1 2024-04-20T21:34:33.301 DEBUG:teuthology.parallel:result is None 2024-04-20T21:34:34.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:34:33 smithi189 ceph-mon[21562]: pgmap v91: 65 pgs: 65 active+clean; 318 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 583 KiB/s rd, 582 KiB/s wr, 104 op/s 2024-04-20T21:34:34.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:34:33 smithi062 ceph-mon[16221]: pgmap v91: 65 pgs: 65 active+clean; 318 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 583 KiB/s rd, 582 KiB/s wr, 104 op/s 2024-04-20T21:34:35.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:34:34 smithi189 ceph-mon[21562]: pgmap v92: 65 pgs: 65 active+clean; 316 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 802 KiB/s rd, 778 KiB/s wr, 135 op/s 2024-04-20T21:34:35.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:34:34 smithi062 ceph-mon[16221]: pgmap v92: 65 pgs: 65 active+clean; 316 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 802 KiB/s rd, 778 KiB/s wr, 135 op/s 2024-04-20T21:34:37.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:34:37 smithi189 ceph-mon[21562]: pgmap v93: 65 pgs: 65 active+clean; 316 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 435 KiB/s rd, 392 KiB/s wr, 69 op/s 2024-04-20T21:34:37.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:34:37 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:34:38.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:34:37 smithi062 ceph-mon[16221]: pgmap v93: 65 pgs: 65 active+clean; 316 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 435 KiB/s rd, 392 KiB/s wr, 69 op/s 2024-04-20T21:34:38.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:34:37 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:34:39.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:34:39 smithi189 ceph-mon[21562]: pgmap v94: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 451 KiB/s rd, 409 KiB/s wr, 78 op/s 2024-04-20T21:34:40.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:34:39 smithi062 ceph-mon[16221]: pgmap v94: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 451 KiB/s rd, 409 KiB/s wr, 78 op/s 2024-04-20T21:34:41.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:34:41 smithi189 ceph-mon[21562]: pgmap v95: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 236 KiB/s rd, 213 KiB/s wr, 39 op/s 2024-04-20T21:34:42.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:34:41 smithi062 ceph-mon[16221]: pgmap v95: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 236 KiB/s rd, 213 KiB/s wr, 39 op/s 2024-04-20T21:34:43.200 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:34:43 smithi062 ceph-mon[16221]: pgmap v96: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 236 KiB/s rd, 213 KiB/s wr, 39 op/s 2024-04-20T21:34:43.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:34:43 smithi189 ceph-mon[21562]: pgmap v96: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 236 KiB/s rd, 213 KiB/s wr, 39 op/s 2024-04-20T21:34:45.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:34:45 smithi062 ceph-mon[16221]: pgmap v97: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 237 KiB/s rd, 213 KiB/s wr, 41 op/s 2024-04-20T21:34:45.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:34:45 smithi189 ceph-mon[21562]: pgmap v97: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 237 KiB/s rd, 213 KiB/s wr, 41 op/s 2024-04-20T21:34:47.867 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:34:47 smithi062 ceph-mon[16221]: pgmap v98: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 18 KiB/s rd, 17 KiB/s wr, 11 op/s 2024-04-20T21:34:47.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:34:47 smithi189 ceph-mon[21562]: pgmap v98: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 18 KiB/s rd, 17 KiB/s wr, 11 op/s 2024-04-20T21:34:49.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:34:49 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:34:49.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:34:49 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:34:49.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:34:49 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:34:49.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:34:49 smithi189 ceph-mon[21562]: pgmap v99: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 19 KiB/s rd, 17 KiB/s wr, 12 op/s 2024-04-20T21:34:49.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:34:49 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:34:49.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:34:49 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:34:49.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:34:49 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:34:49.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:34:49 smithi062 ceph-mon[16221]: pgmap v99: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 19 KiB/s rd, 17 KiB/s wr, 12 op/s 2024-04-20T21:34:51.971 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:34:51 smithi062 ceph-mon[16221]: pgmap v100: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 341 B/s wr, 3 op/s 2024-04-20T21:34:51.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:34:51 smithi189 ceph-mon[21562]: pgmap v100: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 341 B/s wr, 3 op/s 2024-04-20T21:34:52.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:34:52 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:34:53.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:34:52 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:34:53.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:34:53 smithi062 ceph-mon[16221]: pgmap v101: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 341 B/s wr, 3 op/s 2024-04-20T21:34:53.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:34:53 smithi189 ceph-mon[21562]: pgmap v101: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 341 B/s wr, 3 op/s 2024-04-20T21:34:55.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:34:55 smithi189 ceph-mon[21562]: pgmap v102: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.8 KiB/s rd, 341 B/s wr, 5 op/s 2024-04-20T21:34:56.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:34:55 smithi062 ceph-mon[16221]: pgmap v102: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.8 KiB/s rd, 341 B/s wr, 5 op/s 2024-04-20T21:34:56.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:56.468+0000 7f6ea5600700 1 -- 172.21.15.62:0/3661570856 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6ea00feb50 msgr2=0x7f6ea00fef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:34:56.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:56.468+0000 7f6ea5600700 1 --2- 172.21.15.62:0/3661570856 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6ea00feb50 0x7f6ea00fef60 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f6e90009910 tx=0x7f6e90009c20 comp rx=0 tx=0).stop 2024-04-20T21:34:56.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:56.468+0000 7f6ea5600700 1 -- 172.21.15.62:0/3661570856 shutdown_connections 2024-04-20T21:34:56.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:56.468+0000 7f6ea5600700 1 --2- 172.21.15.62:0/3661570856 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ea00ff4a0 0x7f6ea00ff910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:34:56.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:56.468+0000 7f6ea5600700 1 --2- 172.21.15.62:0/3661570856 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6ea00feb50 0x7f6ea00fef60 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:34:56.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:56.468+0000 7f6ea5600700 1 -- 172.21.15.62:0/3661570856 >> 172.21.15.62:0/3661570856 conn(0x7f6ea00fa780 msgr2=0x7f6ea00fcbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:34:56.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:56.468+0000 7f6ea5600700 1 -- 172.21.15.62:0/3661570856 shutdown_connections 2024-04-20T21:34:56.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:56.468+0000 7f6ea5600700 1 -- 172.21.15.62:0/3661570856 wait complete. 2024-04-20T21:34:56.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:56.469+0000 7f6ea5600700 1 Processor -- start 2024-04-20T21:34:56.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:56.469+0000 7f6ea5600700 1 -- start start 2024-04-20T21:34:56.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:56.470+0000 7f6ea5600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ea00feb50 0x7f6ea0197ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:34:56.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:56.470+0000 7f6ea5600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6ea00ff4a0 0x7f6ea0198220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:34:56.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:56.470+0000 7f6ea5600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ea0198840 con 0x7f6ea00ff4a0 2024-04-20T21:34:56.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:56.470+0000 7f6ea5600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ea019d250 con 0x7f6ea00feb50 2024-04-20T21:34:56.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:56.470+0000 7f6e9e000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6ea00ff4a0 0x7f6ea0198220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:34:56.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:56.470+0000 7f6e9e000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6ea00ff4a0 0x7f6ea0198220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37398/0 (socket says 172.21.15.62:37398) 2024-04-20T21:34:56.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:56.470+0000 7f6e9e000700 1 -- 172.21.15.62:0/2407345603 learned_addr learned my addr 172.21.15.62:0/2407345603 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:34:56.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:56.470+0000 7f6e9ea00700 1 --2- 172.21.15.62:0/2407345603 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ea00feb50 0x7f6ea0197ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:34:56.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:56.470+0000 7f6e9e000700 1 -- 172.21.15.62:0/2407345603 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ea00feb50 msgr2=0x7f6ea0197ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:34:56.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:56.470+0000 7f6e9e000700 1 --2- 172.21.15.62:0/2407345603 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ea00feb50 0x7f6ea0197ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:34:56.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:56.470+0000 7f6e9e000700 1 -- 172.21.15.62:0/2407345603 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e900095f0 con 0x7f6ea00ff4a0 2024-04-20T21:34:56.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:56.471+0000 7f6e9e000700 1 --2- 172.21.15.62:0/2407345603 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6ea00ff4a0 0x7f6ea0198220 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f6e8c00c8a0 tx=0x7f6e8c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:34:56.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:56.471+0000 7f6e97400700 1 -- 172.21.15.62:0/2407345603 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6e8c00cea0 con 0x7f6ea00ff4a0 2024-04-20T21:34:56.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:56.471+0000 7f6e97400700 1 -- 172.21.15.62:0/2407345603 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f6e8c004830 con 0x7f6ea00ff4a0 2024-04-20T21:34:56.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:56.471+0000 7f6e97400700 1 -- 172.21.15.62:0/2407345603 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6e8c0055b0 con 0x7f6ea00ff4a0 2024-04-20T21:34:56.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:56.471+0000 7f6ea5600700 1 -- 172.21.15.62:0/2407345603 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ea019d450 con 0x7f6ea00ff4a0 2024-04-20T21:34:56.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:56.471+0000 7f6ea5600700 1 -- 172.21.15.62:0/2407345603 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ea019d970 con 0x7f6ea00ff4a0 2024-04-20T21:34:56.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:56.473+0000 7f6ea5600700 1 -- 172.21.15.62:0/2407345603 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ea010aef0 con 0x7f6ea00ff4a0 2024-04-20T21:34:56.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:56.474+0000 7f6e97400700 1 -- 172.21.15.62:0/2407345603 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 31) v1 ==== 92598+0+0 (secure 0 0 0) 0x7f6e8c007ad0 con 0x7f6ea00ff4a0 2024-04-20T21:34:56.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:56.475+0000 7f6e97400700 1 --2- 172.21.15.62:0/2407345603 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f6e8406e940 0x7f6e84070df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:34:56.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:56.475+0000 7f6e97400700 1 -- 172.21.15.62:0/2407345603 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f6e8c08e5f0 con 0x7f6ea00ff4a0 2024-04-20T21:34:56.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:56.475+0000 7f6e9ea00700 1 --2- 172.21.15.62:0/2407345603 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f6e8406e940 0x7f6e84070df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:34:56.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:56.476+0000 7f6e9ea00700 1 --2- 172.21.15.62:0/2407345603 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f6e8406e940 0x7f6e84070df0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6e90009fd0 tx=0x7f6e9000b540 comp rx=0 tx=0).ready entity=mgr.14658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:34:56.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:56.481+0000 7f6e97400700 1 -- 172.21.15.62:0/2407345603 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187527 (secure 0 0 0) 0x7f6e8c0578d0 con 0x7f6ea00ff4a0 2024-04-20T21:34:56.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:56.738+0000 7f6ea5600700 1 -- 172.21.15.62:0/2407345603 --> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6ea01088a0 con 0x7f6e8406e940 2024-04-20T21:34:56.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:56.740+0000 7f6e97400700 1 -- 172.21.15.62:0/2407345603 <== mgr.14658 v2:172.21.15.62:6800/1585760929 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+364 (secure 0 0 0) 0x7f6ea01088a0 con 0x7f6e8406e940 2024-04-20T21:34:56.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:56.742+0000 7f6ea5600700 1 -- 172.21.15.62:0/2407345603 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f6e8406e940 msgr2=0x7f6e84070df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:34:56.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:56.743+0000 7f6ea5600700 1 --2- 172.21.15.62:0/2407345603 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f6e8406e940 0x7f6e84070df0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6e90009fd0 tx=0x7f6e9000b540 comp rx=0 tx=0).stop 2024-04-20T21:34:56.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:56.743+0000 7f6ea5600700 1 -- 172.21.15.62:0/2407345603 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6ea00ff4a0 msgr2=0x7f6ea0198220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:34:56.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:56.743+0000 7f6ea5600700 1 --2- 172.21.15.62:0/2407345603 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6ea00ff4a0 0x7f6ea0198220 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f6e8c00c8a0 tx=0x7f6e8c00cc60 comp rx=0 tx=0).stop 2024-04-20T21:34:56.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:56.743+0000 7f6ea5600700 1 -- 172.21.15.62:0/2407345603 shutdown_connections 2024-04-20T21:34:56.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:56.743+0000 7f6ea5600700 1 --2- 172.21.15.62:0/2407345603 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f6e8406e940 0x7f6e84070df0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:34:56.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:56.743+0000 7f6ea5600700 1 --2- 172.21.15.62:0/2407345603 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ea00feb50 0x7f6ea0197ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:34:56.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:56.743+0000 7f6ea5600700 1 --2- 172.21.15.62:0/2407345603 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6ea00ff4a0 0x7f6ea0198220 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:34:56.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:56.743+0000 7f6ea5600700 1 -- 172.21.15.62:0/2407345603 >> 172.21.15.62:0/2407345603 conn(0x7f6ea00fa780 msgr2=0x7f6ea0107180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:34:56.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:56.743+0000 7f6ea5600700 1 -- 172.21.15.62:0/2407345603 shutdown_connections 2024-04-20T21:34:56.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:56.743+0000 7f6ea5600700 1 -- 172.21.15.62:0/2407345603 wait complete. 2024-04-20T21:34:56.757 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T21:34:56.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:56.903+0000 7fe98f200700 1 -- 172.21.15.62:0/2389292548 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe988101440 msgr2=0x7fe988101890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:34:56.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:56.903+0000 7fe98f200700 1 --2- 172.21.15.62:0/2389292548 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe988101440 0x7fe988101890 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fe974009910 tx=0x7fe974009c20 comp rx=0 tx=0).stop 2024-04-20T21:34:56.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:56.903+0000 7fe98f200700 1 -- 172.21.15.62:0/2389292548 shutdown_connections 2024-04-20T21:34:56.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:56.903+0000 7fe98f200700 1 --2- 172.21.15.62:0/2389292548 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe988101440 0x7fe988101890 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:34:56.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:56.903+0000 7fe98f200700 1 --2- 172.21.15.62:0/2389292548 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe988100240 0x7fe988100650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:34:56.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:56.903+0000 7fe98f200700 1 -- 172.21.15.62:0/2389292548 >> 172.21.15.62:0/2389292548 conn(0x7fe9880fb7f0 msgr2=0x7fe9880fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:34:56.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:56.903+0000 7fe98f200700 1 -- 172.21.15.62:0/2389292548 shutdown_connections 2024-04-20T21:34:56.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:56.903+0000 7fe98f200700 1 -- 172.21.15.62:0/2389292548 wait complete. 2024-04-20T21:34:56.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:56.904+0000 7fe98f200700 1 Processor -- start 2024-04-20T21:34:56.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:56.905+0000 7fe98f200700 1 -- start start 2024-04-20T21:34:56.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:56.905+0000 7fe98f200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe988100240 0x7fe988197c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:34:56.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:56.905+0000 7fe98f200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe988101440 0x7fe9881981d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:34:56.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:56.905+0000 7fe98f200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9881987d0 con 0x7fe988101440 2024-04-20T21:34:56.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:56.905+0000 7fe98f200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe988198940 con 0x7fe988100240 2024-04-20T21:34:56.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:56.905+0000 7fe987400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe988101440 0x7fe9881981d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:34:56.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:56.905+0000 7fe987400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe988101440 0x7fe9881981d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37422/0 (socket says 172.21.15.62:37422) 2024-04-20T21:34:56.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:56.905+0000 7fe987400700 1 -- 172.21.15.62:0/4281392354 learned_addr learned my addr 172.21.15.62:0/4281392354 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:34:56.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:56.906+0000 7fe987400700 1 -- 172.21.15.62:0/4281392354 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe988100240 msgr2=0x7fe988197c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:34:56.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:56.906+0000 7fe987400700 1 --2- 172.21.15.62:0/4281392354 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe988100240 0x7fe988197c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:34:56.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:56.906+0000 7fe987400700 1 -- 172.21.15.62:0/4281392354 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9740095f0 con 0x7fe988101440 2024-04-20T21:34:56.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:56.906+0000 7fe987400700 1 --2- 172.21.15.62:0/4281392354 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe988101440 0x7fe9881981d0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fe97400b5c0 tx=0x7fe97400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:34:56.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:56.906+0000 7fe984c00700 1 -- 172.21.15.62:0/4281392354 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe97401d070 con 0x7fe988101440 2024-04-20T21:34:56.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:56.906+0000 7fe984c00700 1 -- 172.21.15.62:0/4281392354 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fe974022470 con 0x7fe988101440 2024-04-20T21:34:56.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:56.906+0000 7fe98f200700 1 -- 172.21.15.62:0/4281392354 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe98819d370 con 0x7fe988101440 2024-04-20T21:34:56.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:56.906+0000 7fe984c00700 1 -- 172.21.15.62:0/4281392354 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe97400f690 con 0x7fe988101440 2024-04-20T21:34:56.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:56.906+0000 7fe98f200700 1 -- 172.21.15.62:0/4281392354 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe98819d860 con 0x7fe988101440 2024-04-20T21:34:56.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:56.908+0000 7fe98f200700 1 -- 172.21.15.62:0/4281392354 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe988105520 con 0x7fe988101440 2024-04-20T21:34:56.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:56.909+0000 7fe984c00700 1 -- 172.21.15.62:0/4281392354 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 31) v1 ==== 92598+0+0 (secure 0 0 0) 0x7fe974004b70 con 0x7fe988101440 2024-04-20T21:34:56.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:56.910+0000 7fe984c00700 1 --2- 172.21.15.62:0/4281392354 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7fe96c06e940 0x7fe96c070df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:34:56.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:56.910+0000 7fe984c00700 1 -- 172.21.15.62:0/4281392354 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6070+0+0 (secure 0 0 0) 0x7fe97408ff90 con 0x7fe988101440 2024-04-20T21:34:56.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:56.910+0000 7fe987e00700 1 --2- 172.21.15.62:0/4281392354 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7fe96c06e940 0x7fe96c070df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:34:56.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:56.911+0000 7fe987e00700 1 --2- 172.21.15.62:0/4281392354 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7fe96c06e940 0x7fe96c070df0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe988198b70 tx=0x7fe978005ca0 comp rx=0 tx=0).ready entity=mgr.14658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:34:56.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:56.915+0000 7fe984c00700 1 -- 172.21.15.62:0/4281392354 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187527 (secure 0 0 0) 0x7fe974059300 con 0x7fe988101440 2024-04-20T21:34:57.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:57.178+0000 7fe98f200700 1 -- 172.21.15.62:0/4281392354 --> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe988061190 con 0x7fe96c06e940 2024-04-20T21:34:57.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:57.180+0000 7fe984c00700 1 -- 172.21.15.62:0/4281392354 <== mgr.14658 v2:172.21.15.62:6800/1585760929 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+364 (secure 0 0 0) 0x7fe988061190 con 0x7fe96c06e940 2024-04-20T21:34:57.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:57.183+0000 7fe98f200700 1 -- 172.21.15.62:0/4281392354 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7fe96c06e940 msgr2=0x7fe96c070df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:34:57.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:57.183+0000 7fe98f200700 1 --2- 172.21.15.62:0/4281392354 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7fe96c06e940 0x7fe96c070df0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe988198b70 tx=0x7fe978005ca0 comp rx=0 tx=0).stop 2024-04-20T21:34:57.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:57.183+0000 7fe98f200700 1 -- 172.21.15.62:0/4281392354 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe988101440 msgr2=0x7fe9881981d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:34:57.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:57.183+0000 7fe98f200700 1 --2- 172.21.15.62:0/4281392354 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe988101440 0x7fe9881981d0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fe97400b5c0 tx=0x7fe97400be20 comp rx=0 tx=0).stop 2024-04-20T21:34:57.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:57.183+0000 7fe98f200700 1 -- 172.21.15.62:0/4281392354 shutdown_connections 2024-04-20T21:34:57.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:57.183+0000 7fe98f200700 1 --2- 172.21.15.62:0/4281392354 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7fe96c06e940 0x7fe96c070df0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:34:57.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:57.183+0000 7fe98f200700 1 --2- 172.21.15.62:0/4281392354 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe988100240 0x7fe988197c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:34:57.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:57.183+0000 7fe98f200700 1 --2- 172.21.15.62:0/4281392354 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe988101440 0x7fe9881981d0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:34:57.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:57.183+0000 7fe98f200700 1 -- 172.21.15.62:0/4281392354 >> 172.21.15.62:0/4281392354 conn(0x7fe9880fb7f0 msgr2=0x7fe988102660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:34:57.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:57.183+0000 7fe98f200700 1 -- 172.21.15.62:0/4281392354 shutdown_connections 2024-04-20T21:34:57.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:57.183+0000 7fe98f200700 1 -- 172.21.15.62:0/4281392354 wait complete. 2024-04-20T21:34:57.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:57.342+0000 7f3e6bc00700 1 -- 172.21.15.62:0/4182396475 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e64074d80 msgr2=0x7f3e640731e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:34:57.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:57.342+0000 7f3e6bc00700 1 --2- 172.21.15.62:0/4182396475 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e64074d80 0x7f3e640731e0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f3e58009940 tx=0x7f3e58009c50 comp rx=0 tx=0).stop 2024-04-20T21:34:57.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:57.342+0000 7f3e6bc00700 1 -- 172.21.15.62:0/4182396475 shutdown_connections 2024-04-20T21:34:57.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:57.342+0000 7f3e6bc00700 1 --2- 172.21.15.62:0/4182396475 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e640737b0 0x7f3e64073c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:34:57.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:57.342+0000 7f3e6bc00700 1 --2- 172.21.15.62:0/4182396475 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e64074d80 0x7f3e640731e0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:34:57.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:57.342+0000 7f3e6bc00700 1 -- 172.21.15.62:0/4182396475 >> 172.21.15.62:0/4182396475 conn(0x7f3e640fb7f0 msgr2=0x7f3e640fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:34:57.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:57.342+0000 7f3e6bc00700 1 -- 172.21.15.62:0/4182396475 shutdown_connections 2024-04-20T21:34:57.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:57.343+0000 7f3e6bc00700 1 -- 172.21.15.62:0/4182396475 wait complete. 2024-04-20T21:34:57.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:57.344+0000 7f3e6bc00700 1 Processor -- start 2024-04-20T21:34:57.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:57.344+0000 7f3e6bc00700 1 -- start start 2024-04-20T21:34:57.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:57.344+0000 7f3e6bc00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e640737b0 0x7f3e64197da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:34:57.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:57.344+0000 7f3e6bc00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e64074d80 0x7f3e641982e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:34:57.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:57.344+0000 7f3e6bc00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e64198900 con 0x7f3e640737b0 2024-04-20T21:34:57.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:57.344+0000 7f3e6bc00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e64198a40 con 0x7f3e64074d80 2024-04-20T21:34:57.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:57.345+0000 7f3e69200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e640737b0 0x7f3e64197da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:34:57.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:57.345+0000 7f3e69200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e640737b0 0x7f3e64197da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37440/0 (socket says 172.21.15.62:37440) 2024-04-20T21:34:57.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:57.345+0000 7f3e69200700 1 -- 172.21.15.62:0/3759189405 learned_addr learned my addr 172.21.15.62:0/3759189405 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:34:57.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:57.345+0000 7f3e63e00700 1 --2- 172.21.15.62:0/3759189405 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e64074d80 0x7f3e641982e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:34:57.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:57.345+0000 7f3e69200700 1 -- 172.21.15.62:0/3759189405 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e64074d80 msgr2=0x7f3e641982e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:34:57.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:57.345+0000 7f3e69200700 1 --2- 172.21.15.62:0/3759189405 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e64074d80 0x7f3e641982e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:34:57.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:57.345+0000 7f3e69200700 1 -- 172.21.15.62:0/3759189405 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e580095f0 con 0x7f3e640737b0 2024-04-20T21:34:57.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:57.345+0000 7f3e69200700 1 --2- 172.21.15.62:0/3759189405 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e640737b0 0x7f3e64197da0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f3e580055f0 tx=0x7f3e580056d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:34:57.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:57.346+0000 7f3e61600700 1 -- 172.21.15.62:0/3759189405 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3e5801d070 con 0x7f3e640737b0 2024-04-20T21:34:57.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:57.346+0000 7f3e61600700 1 -- 172.21.15.62:0/3759189405 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f3e58003ba0 con 0x7f3e640737b0 2024-04-20T21:34:57.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:57.346+0000 7f3e6bc00700 1 -- 172.21.15.62:0/3759189405 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e6419d490 con 0x7f3e640737b0 2024-04-20T21:34:57.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:57.346+0000 7f3e61600700 1 -- 172.21.15.62:0/3759189405 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3e58021710 con 0x7f3e640737b0 2024-04-20T21:34:57.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:57.346+0000 7f3e6bc00700 1 -- 172.21.15.62:0/3759189405 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e6419d980 con 0x7f3e640737b0 2024-04-20T21:34:57.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:57.347+0000 7f3e6bc00700 1 -- 172.21.15.62:0/3759189405 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e6410b000 con 0x7f3e640737b0 2024-04-20T21:34:57.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:57.348+0000 7f3e61600700 1 -- 172.21.15.62:0/3759189405 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 31) v1 ==== 92598+0+0 (secure 0 0 0) 0x7f3e5800faa0 con 0x7f3e640737b0 2024-04-20T21:34:57.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:57.349+0000 7f3e61600700 1 --2- 172.21.15.62:0/3759189405 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f3e4c06e940 0x7f3e4c070df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:34:57.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:57.349+0000 7f3e61600700 1 -- 172.21.15.62:0/3759189405 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f3e5808fc80 con 0x7f3e640737b0 2024-04-20T21:34:57.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:57.350+0000 7f3e63e00700 1 --2- 172.21.15.62:0/3759189405 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f3e4c06e940 0x7f3e4c070df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:34:57.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:57.350+0000 7f3e63e00700 1 --2- 172.21.15.62:0/3759189405 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f3e4c06e940 0x7f3e4c070df0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3e50006fd0 tx=0x7f3e50008040 comp rx=0 tx=0).ready entity=mgr.14658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:34:57.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:57.357+0000 7f3e61600700 1 -- 172.21.15.62:0/3759189405 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187527 (secure 0 0 0) 0x7f3e5805ca70 con 0x7f3e640737b0 2024-04-20T21:34:57.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:57.629+0000 7f3e6bc00700 1 -- 172.21.15.62:0/3759189405 --> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f3e64107740 con 0x7f3e4c06e940 2024-04-20T21:34:57.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:57.645+0000 7f3e61600700 1 -- 172.21.15.62:0/3759189405 <== mgr.14658 v2:172.21.15.62:6800/1585760929 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7f3e64107740 con 0x7f3e4c06e940 2024-04-20T21:34:57.645 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T21:34:57.645 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (47s) 36s ago 14m 13.8M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T21:34:57.645 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (15m) 36s ago 15m 8158k - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T21:34:57.645 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (13m) 106s ago 13m 7876k - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T21:34:57.645 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (14m) 36s ago 14m 6656k - 18.2.0 dc2bc1663786 36e10ef5be39 2024-04-20T21:34:57.645 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (13m) 106s ago 13m 6665k - 18.2.0 dc2bc1663786 4101f9953e3d 2024-04-20T21:34:57.645 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 starting - - - - 2024-04-20T21:34:57.645 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (7m) 36s ago 7m 87.0M - 18.2.0 dc2bc1663786 5e6460aa6159 2024-04-20T21:34:57.645 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (7m) 36s ago 7m 159M - 18.2.0 dc2bc1663786 e65f2267544e 2024-04-20T21:34:57.645 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (7m) 106s ago 7m 151M - 18.2.0 dc2bc1663786 fb2a8a223db8 2024-04-20T21:34:57.645 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (7m) 106s ago 7m 114M - 18.2.0 dc2bc1663786 f2eec2ee8951 2024-04-20T21:34:57.645 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (3m) 36s ago 16m 502M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T21:34:57.645 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (2m) 106s ago 13m 448M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T21:34:57.646 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (16m) 36s ago 16m 57.5M 2048M 18.2.0 dc2bc1663786 bd6efbd5c380 2024-04-20T21:34:57.646 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (13m) 106s ago 13m 45.7M 2048M 18.2.0 dc2bc1663786 3735c8e5db9a 2024-04-20T21:34:57.646 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (118s) 36s ago 14m 7218k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T21:34:57.646 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (111s) 106s ago 13m 3003k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T21:34:57.646 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (12m) 36s ago 12m 213M 1216M 18.2.0 dc2bc1663786 3f315c3afc1f 2024-04-20T21:34:57.646 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (11m) 36s ago 11m 203M 1216M 18.2.0 dc2bc1663786 44e4d07416ad 2024-04-20T21:34:57.646 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (11m) 36s ago 11m 167M 1216M 18.2.0 dc2bc1663786 31c5d9586e98 2024-04-20T21:34:57.646 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (10m) 106s ago 10m 211M 2240M 18.2.0 dc2bc1663786 611c1e242693 2024-04-20T21:34:57.646 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (9m) 106s ago 9m 190M 2240M 18.2.0 dc2bc1663786 6c8fce91d45d 2024-04-20T21:34:57.646 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (9m) 106s ago 9m 189M 2240M 18.2.0 dc2bc1663786 8e4574bae8db 2024-04-20T21:34:57.646 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (82s) 36s ago 14m 59.8M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T21:34:57.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:57.648+0000 7f3e6bc00700 1 -- 172.21.15.62:0/3759189405 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f3e4c06e940 msgr2=0x7f3e4c070df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:34:57.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:57.648+0000 7f3e6bc00700 1 --2- 172.21.15.62:0/3759189405 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f3e4c06e940 0x7f3e4c070df0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3e50006fd0 tx=0x7f3e50008040 comp rx=0 tx=0).stop 2024-04-20T21:34:57.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:57.648+0000 7f3e6bc00700 1 -- 172.21.15.62:0/3759189405 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e640737b0 msgr2=0x7f3e64197da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:34:57.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:57.648+0000 7f3e6bc00700 1 --2- 172.21.15.62:0/3759189405 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e640737b0 0x7f3e64197da0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f3e580055f0 tx=0x7f3e580056d0 comp rx=0 tx=0).stop 2024-04-20T21:34:57.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:57.649+0000 7f3e6bc00700 1 -- 172.21.15.62:0/3759189405 shutdown_connections 2024-04-20T21:34:57.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:57.649+0000 7f3e6bc00700 1 --2- 172.21.15.62:0/3759189405 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f3e4c06e940 0x7f3e4c070df0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:34:57.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:57.649+0000 7f3e6bc00700 1 --2- 172.21.15.62:0/3759189405 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e640737b0 0x7f3e64197da0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:34:57.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:57.649+0000 7f3e6bc00700 1 --2- 172.21.15.62:0/3759189405 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e64074d80 0x7f3e641982e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:34:57.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:57.649+0000 7f3e6bc00700 1 -- 172.21.15.62:0/3759189405 >> 172.21.15.62:0/3759189405 conn(0x7f3e640fb7f0 msgr2=0x7f3e64106020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:34:57.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:57.649+0000 7f3e6bc00700 1 -- 172.21.15.62:0/3759189405 shutdown_connections 2024-04-20T21:34:57.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:57.649+0000 7f3e6bc00700 1 -- 172.21.15.62:0/3759189405 wait complete. 2024-04-20T21:34:57.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:57.812+0000 7f568d600700 1 -- 172.21.15.62:0/3038662427 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f56881014b0 msgr2=0x7f5688103930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:34:57.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:57.812+0000 7f568d600700 1 --2- 172.21.15.62:0/3038662427 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f56881014b0 0x7f5688103930 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f5674009910 tx=0x7f5674009c20 comp rx=0 tx=0).stop 2024-04-20T21:34:57.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:57.813+0000 7f568d600700 1 -- 172.21.15.62:0/3038662427 shutdown_connections 2024-04-20T21:34:57.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:57.813+0000 7f568d600700 1 --2- 172.21.15.62:0/3038662427 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f56881014b0 0x7f5688103930 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:34:57.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:57.813+0000 7f568d600700 1 --2- 172.21.15.62:0/3038662427 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56880feb50 0x7f5688100f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:34:57.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:57.813+0000 7f568d600700 1 -- 172.21.15.62:0/3038662427 >> 172.21.15.62:0/3038662427 conn(0x7f56880fa780 msgr2=0x7f56880fcbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:34:57.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:57.813+0000 7f568d600700 1 -- 172.21.15.62:0/3038662427 shutdown_connections 2024-04-20T21:34:57.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:57.813+0000 7f568d600700 1 -- 172.21.15.62:0/3038662427 wait complete. 2024-04-20T21:34:57.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:57.814+0000 7f568d600700 1 Processor -- start 2024-04-20T21:34:57.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:57.814+0000 7f568d600700 1 -- start start 2024-04-20T21:34:57.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:57.814+0000 7f568d600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56880feb50 0x7f568819c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:34:57.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:57.815+0000 7f568d600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f56881014b0 0x7f568819c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:34:57.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:57.815+0000 7f568d600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f568819cc00 con 0x7f56881014b0 2024-04-20T21:34:57.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:57.815+0000 7f568d600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f568819cd40 con 0x7f56880feb50 2024-04-20T21:34:57.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:57.815+0000 7f5686000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f56881014b0 0x7f568819c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:34:57.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:57.815+0000 7f5686000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f56881014b0 0x7f568819c5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37472/0 (socket says 172.21.15.62:37472) 2024-04-20T21:34:57.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:57.815+0000 7f5686000700 1 -- 172.21.15.62:0/3996889697 learned_addr learned my addr 172.21.15.62:0/3996889697 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:34:57.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:57.815+0000 7f5686000700 1 -- 172.21.15.62:0/3996889697 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56880feb50 msgr2=0x7f568819c0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:34:57.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:57.815+0000 7f5686000700 1 --2- 172.21.15.62:0/3996889697 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56880feb50 0x7f568819c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:34:57.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:57.815+0000 7f5686000700 1 -- 172.21.15.62:0/3996889697 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56740095f0 con 0x7f56881014b0 2024-04-20T21:34:57.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:57.816+0000 7f5686000700 1 --2- 172.21.15.62:0/3996889697 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f56881014b0 0x7f568819c5e0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f567400b5c0 tx=0x7f567400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:34:57.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:57.816+0000 7f567f400700 1 -- 172.21.15.62:0/3996889697 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f567401d070 con 0x7f56881014b0 2024-04-20T21:34:57.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:57.816+0000 7f567f400700 1 -- 172.21.15.62:0/3996889697 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f5674022470 con 0x7f56881014b0 2024-04-20T21:34:57.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:57.816+0000 7f568d600700 1 -- 172.21.15.62:0/3996889697 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56881a1790 con 0x7f56881014b0 2024-04-20T21:34:57.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:57.816+0000 7f568d600700 1 -- 172.21.15.62:0/3996889697 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56881a1c80 con 0x7f56881014b0 2024-04-20T21:34:57.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:57.817+0000 7f567f400700 1 -- 172.21.15.62:0/3996889697 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f567400f690 con 0x7f56881014b0 2024-04-20T21:34:57.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:57.818+0000 7f568d600700 1 -- 172.21.15.62:0/3996889697 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5668005320 con 0x7f56881014b0 2024-04-20T21:34:57.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:57.819+0000 7f567f400700 1 -- 172.21.15.62:0/3996889697 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 31) v1 ==== 92598+0+0 (secure 0 0 0) 0x7f5674004b70 con 0x7f56881014b0 2024-04-20T21:34:57.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:57.821+0000 7f567f400700 1 --2- 172.21.15.62:0/3996889697 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f566c06e940 0x7f566c070df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:34:57.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:57.821+0000 7f567f400700 1 -- 172.21.15.62:0/3996889697 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f567408fe60 con 0x7f56881014b0 2024-04-20T21:34:57.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:57.821+0000 7f5686a00700 1 --2- 172.21.15.62:0/3996889697 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f566c06e940 0x7f566c070df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:34:57.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:57.822+0000 7f5686a00700 1 --2- 172.21.15.62:0/3996889697 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f566c06e940 0x7f566c070df0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5688069000 tx=0x7f5678005ca0 comp rx=0 tx=0).ready entity=mgr.14658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:34:57.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:57.825+0000 7f567f400700 1 -- 172.21.15.62:0/3996889697 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187527 (secure 0 0 0) 0x7f567405cda0 con 0x7f56881014b0 2024-04-20T21:34:57.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:34:57 smithi189 ceph-mon[21562]: pgmap v103: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:34:57.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:34:57 smithi189 ceph-mon[21562]: from='client.14812 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:34:58.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:34:57 smithi062 ceph-mon[16221]: pgmap v103: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:34:58.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:34:57 smithi062 ceph-mon[16221]: from='client.14812 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:34:58.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:58.181+0000 7f568d600700 1 -- 172.21.15.62:0/3996889697 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f5668006200 con 0x7f56881014b0 2024-04-20T21:34:58.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:58.182+0000 7f567f400700 1 -- 172.21.15.62:0/3996889697 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+675 (secure 0 0 0) 0x7f56740272c0 con 0x7f56881014b0 2024-04-20T21:34:58.182 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:34:58.182 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T21:34:58.182 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 2 2024-04-20T21:34:58.182 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:34:58.183 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T21:34:58.183 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:34:58.183 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:34:58.183 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T21:34:58.183 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 6 2024-04-20T21:34:58.183 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:34:58.183 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T21:34:58.183 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 4 2024-04-20T21:34:58.183 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:34:58.183 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T21:34:58.183 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 12, 2024-04-20T21:34:58.183 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:34:58.183 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T21:34:58.183 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:34:58.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:58.187+0000 7f568d600700 1 -- 172.21.15.62:0/3996889697 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f566c06e940 msgr2=0x7f566c070df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:34:58.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:58.187+0000 7f568d600700 1 --2- 172.21.15.62:0/3996889697 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f566c06e940 0x7f566c070df0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5688069000 tx=0x7f5678005ca0 comp rx=0 tx=0).stop 2024-04-20T21:34:58.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:58.187+0000 7f568d600700 1 -- 172.21.15.62:0/3996889697 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f56881014b0 msgr2=0x7f568819c5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:34:58.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:58.187+0000 7f568d600700 1 --2- 172.21.15.62:0/3996889697 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f56881014b0 0x7f568819c5e0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f567400b5c0 tx=0x7f567400be20 comp rx=0 tx=0).stop 2024-04-20T21:34:58.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:58.187+0000 7f568d600700 1 -- 172.21.15.62:0/3996889697 shutdown_connections 2024-04-20T21:34:58.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:58.187+0000 7f568d600700 1 --2- 172.21.15.62:0/3996889697 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f566c06e940 0x7f566c070df0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:34:58.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:58.187+0000 7f568d600700 1 --2- 172.21.15.62:0/3996889697 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56880feb50 0x7f568819c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:34:58.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:58.187+0000 7f568d600700 1 --2- 172.21.15.62:0/3996889697 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f56881014b0 0x7f568819c5e0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:34:58.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:58.187+0000 7f568d600700 1 -- 172.21.15.62:0/3996889697 >> 172.21.15.62:0/3996889697 conn(0x7f56880fa780 msgr2=0x7f56880fcbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:34:58.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:58.187+0000 7f568d600700 1 -- 172.21.15.62:0/3996889697 shutdown_connections 2024-04-20T21:34:58.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:58.187+0000 7f568d600700 1 -- 172.21.15.62:0/3996889697 wait complete. 2024-04-20T21:34:58.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:58.372+0000 7f6de0800700 1 -- 172.21.15.62:0/3640119776 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6dd8100240 msgr2=0x7f6dd8100650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:34:58.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:58.372+0000 7f6de0800700 1 --2- 172.21.15.62:0/3640119776 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6dd8100240 0x7f6dd8100650 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f6dcc009910 tx=0x7f6dcc009c20 comp rx=0 tx=0).stop 2024-04-20T21:34:58.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:58.373+0000 7f6de0800700 1 -- 172.21.15.62:0/3640119776 shutdown_connections 2024-04-20T21:34:58.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:58.373+0000 7f6de0800700 1 --2- 172.21.15.62:0/3640119776 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6dd8101440 0x7f6dd8101890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:34:58.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:58.373+0000 7f6de0800700 1 --2- 172.21.15.62:0/3640119776 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6dd8100240 0x7f6dd8100650 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:34:58.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:58.373+0000 7f6de0800700 1 -- 172.21.15.62:0/3640119776 >> 172.21.15.62:0/3640119776 conn(0x7f6dd80fb7f0 msgr2=0x7f6dd80fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:34:58.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:58.373+0000 7f6de0800700 1 -- 172.21.15.62:0/3640119776 shutdown_connections 2024-04-20T21:34:58.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:58.373+0000 7f6de0800700 1 -- 172.21.15.62:0/3640119776 wait complete. 2024-04-20T21:34:58.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:58.374+0000 7f6de0800700 1 Processor -- start 2024-04-20T21:34:58.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:58.374+0000 7f6de0800700 1 -- start start 2024-04-20T21:34:58.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:58.374+0000 7f6de0800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6dd8100240 0x7f6dd8074af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:34:58.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:58.374+0000 7f6de0800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6dd8101440 0x7f6dd8073140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:34:58.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:58.375+0000 7f6de0800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dd8073680 con 0x7f6dd8100240 2024-04-20T21:34:58.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:58.375+0000 7f6de0800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dd80737c0 con 0x7f6dd8101440 2024-04-20T21:34:58.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:58.375+0000 7f6ddde00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6dd8100240 0x7f6dd8074af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:34:58.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:58.375+0000 7f6ddde00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6dd8100240 0x7f6dd8074af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37486/0 (socket says 172.21.15.62:37486) 2024-04-20T21:34:58.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:58.375+0000 7f6ddde00700 1 -- 172.21.15.62:0/1849699902 learned_addr learned my addr 172.21.15.62:0/1849699902 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:34:58.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:58.375+0000 7f6ddde00700 1 -- 172.21.15.62:0/1849699902 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6dd8101440 msgr2=0x7f6dd8073140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:34:58.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:58.375+0000 7f6ddde00700 1 --2- 172.21.15.62:0/1849699902 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6dd8101440 0x7f6dd8073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:34:58.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:58.375+0000 7f6ddde00700 1 -- 172.21.15.62:0/1849699902 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dcc0095f0 con 0x7f6dd8100240 2024-04-20T21:34:58.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:58.376+0000 7f6ddde00700 1 --2- 172.21.15.62:0/1849699902 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6dd8100240 0x7f6dd8074af0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f6dcc00bda0 tx=0x7f6dcc00be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:34:58.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:58.376+0000 7f6dd2a00700 1 -- 172.21.15.62:0/1849699902 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6dcc01d070 con 0x7f6dd8100240 2024-04-20T21:34:58.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:58.376+0000 7f6dd2a00700 1 -- 172.21.15.62:0/1849699902 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f6dcc004c50 con 0x7f6dd8100240 2024-04-20T21:34:58.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:58.376+0000 7f6de0800700 1 -- 172.21.15.62:0/1849699902 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6dd8073a40 con 0x7f6dd8100240 2024-04-20T21:34:58.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:58.376+0000 7f6de0800700 1 -- 172.21.15.62:0/1849699902 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6dd8073f30 con 0x7f6dd8100240 2024-04-20T21:34:58.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:58.377+0000 7f6dd2a00700 1 -- 172.21.15.62:0/1849699902 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6dcc00f690 con 0x7f6dd8100240 2024-04-20T21:34:58.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:58.378+0000 7f6de0800700 1 -- 172.21.15.62:0/1849699902 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6dd8108e10 con 0x7f6dd8100240 2024-04-20T21:34:58.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:58.380+0000 7f6dd2a00700 1 -- 172.21.15.62:0/1849699902 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 31) v1 ==== 92598+0+0 (secure 0 0 0) 0x7f6dcc00f900 con 0x7f6dd8100240 2024-04-20T21:34:58.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:58.382+0000 7f6dd2a00700 1 --2- 172.21.15.62:0/1849699902 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f6dc006e940 0x7f6dc0070df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:34:58.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:58.382+0000 7f6dd2a00700 1 -- 172.21.15.62:0/1849699902 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f6dcc090a00 con 0x7f6dd8100240 2024-04-20T21:34:58.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:58.382+0000 7f6ddd400700 1 --2- 172.21.15.62:0/1849699902 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f6dc006e940 0x7f6dc0070df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:34:58.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:58.383+0000 7f6ddd400700 1 --2- 172.21.15.62:0/1849699902 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f6dc006e940 0x7f6dc0070df0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f6dc80097b0 tx=0x7f6dc8006d20 comp rx=0 tx=0).ready entity=mgr.14658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:34:58.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:58.387+0000 7f6dd2a00700 1 -- 172.21.15.62:0/1849699902 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187527 (secure 0 0 0) 0x7f6dcc05cec0 con 0x7f6dd8100240 2024-04-20T21:34:58.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:58.650+0000 7f6de0800700 1 -- 172.21.15.62:0/1849699902 --> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6dd8105d90 con 0x7f6dc006e940 2024-04-20T21:34:58.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:58.652+0000 7f6dd2a00700 1 -- 172.21.15.62:0/1849699902 <== mgr.14658 v2:172.21.15.62:6800/1585760929 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+364 (secure 0 0 0) 0x7f6dd8105d90 con 0x7f6dc006e940 2024-04-20T21:34:58.651 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:34:58.651 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T21:34:58.651 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T21:34:58.651 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-20T21:34:58.652 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T21:34:58.652 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr" 2024-04-20T21:34:58.652 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T21:34:58.652 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "2/2 daemons upgraded", 2024-04-20T21:34:58.652 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading grafana daemons", 2024-04-20T21:34:58.652 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T21:34:58.652 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:34:58.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:58.655+0000 7f6de0800700 1 -- 172.21.15.62:0/1849699902 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f6dc006e940 msgr2=0x7f6dc0070df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:34:58.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:58.655+0000 7f6de0800700 1 --2- 172.21.15.62:0/1849699902 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f6dc006e940 0x7f6dc0070df0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f6dc80097b0 tx=0x7f6dc8006d20 comp rx=0 tx=0).stop 2024-04-20T21:34:58.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:58.655+0000 7f6de0800700 1 -- 172.21.15.62:0/1849699902 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6dd8100240 msgr2=0x7f6dd8074af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:34:58.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:58.655+0000 7f6de0800700 1 --2- 172.21.15.62:0/1849699902 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6dd8100240 0x7f6dd8074af0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f6dcc00bda0 tx=0x7f6dcc00be80 comp rx=0 tx=0).stop 2024-04-20T21:34:58.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:58.655+0000 7f6de0800700 1 -- 172.21.15.62:0/1849699902 shutdown_connections 2024-04-20T21:34:58.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:58.655+0000 7f6de0800700 1 --2- 172.21.15.62:0/1849699902 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f6dc006e940 0x7f6dc0070df0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:34:58.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:58.655+0000 7f6de0800700 1 --2- 172.21.15.62:0/1849699902 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6dd8100240 0x7f6dd8074af0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:34:58.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:58.655+0000 7f6de0800700 1 --2- 172.21.15.62:0/1849699902 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6dd8101440 0x7f6dd8073140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:34:58.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:58.655+0000 7f6de0800700 1 -- 172.21.15.62:0/1849699902 >> 172.21.15.62:0/1849699902 conn(0x7f6dd80fb7f0 msgr2=0x7f6dd8104670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:34:58.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:58.655+0000 7f6de0800700 1 -- 172.21.15.62:0/1849699902 shutdown_connections 2024-04-20T21:34:58.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:34:58.655+0000 7f6de0800700 1 -- 172.21.15.62:0/1849699902 wait complete. 2024-04-20T21:34:58.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:34:58 smithi189 ceph-mon[21562]: from='client.14816 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:34:58.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:34:58 smithi189 ceph-mon[21562]: from='client.14820 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:34:58.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:34:58 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:34:58.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:34:58 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:34:58.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:34:58 smithi189 ceph-mon[21562]: from='client.? 172.21.15.62:0/3996889697' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:34:59.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:34:58 smithi062 ceph-mon[16221]: from='client.14816 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:34:59.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:34:58 smithi062 ceph-mon[16221]: from='client.14820 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:34:59.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:34:58 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:34:59.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:34:58 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:34:59.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:34:58 smithi062 ceph-mon[16221]: from='client.? 172.21.15.62:0/3996889697' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:34:59.971 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:34:59 smithi189 ceph-mon[21562]: pgmap v104: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:34:59.972 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:34:59 smithi189 ceph-mon[21562]: from='client.14828 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:35:00.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:34:59 smithi062 ceph-mon[16221]: pgmap v104: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:35:00.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:34:59 smithi062 ceph-mon[16221]: from='client.14828 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:35:01.154 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:00 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:35:01.155 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:00 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:35:01.155 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:00 smithi062 ceph-mon[16221]: pgmap v105: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:35:01.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:00 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:35:01.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:00 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:35:01.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:00 smithi189 ceph-mon[21562]: pgmap v105: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:35:03.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:03 smithi189 ceph-mon[21562]: pgmap v106: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:35:03.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:03 smithi062 ceph-mon[16221]: pgmap v106: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:35:05.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:05 smithi189 ceph-mon[21562]: pgmap v107: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:35:05.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:05 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:35:05.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:05 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:35:05.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:05 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:35:05.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:05 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T21:35:05.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:05 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:35:05.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:05 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-20T21:35:05.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:05 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:35:05.989 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:05 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:35:05.989 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:05 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:35:05.989 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:05 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:35:05.989 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:05 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:35:05.989 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:05 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:35:05.989 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:05 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:35:05.989 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:05 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:35:05.989 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:05 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:35:05.989 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:05 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:35:05.989 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:05 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:35:05.989 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:05 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:35:05.989 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:05 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:35:05.989 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:05 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:35:05.989 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:05 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:35:05.990 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:05 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-20T21:35:05.990 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:05 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-04-20T21:35:05.990 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:05 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-20T21:35:05.990 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:05 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-04-20T21:35:05.990 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:05 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-20T21:35:05.990 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:05 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-20T21:35:05.990 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:05 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-20T21:35:05.990 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:05 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-20T21:35:05.990 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:05 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-04-20T21:35:05.990 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:05 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-20T21:35:05.990 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:05 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-04-20T21:35:05.990 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:05 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-20T21:35:05.991 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:05 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-20T21:35:05.991 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:05 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-20T21:35:05.991 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:05 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-04-20T21:35:05.991 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:05 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-20T21:35:05.991 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:05 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-04-20T21:35:05.991 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:05 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-20T21:35:05.991 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:05 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-04-20T21:35:05.991 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:05 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-20T21:35:05.991 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:05 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-20T21:35:05.991 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:05 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-20T21:35:05.991 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:05 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-20T21:35:05.992 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:05 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-20T21:35:05.992 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:05 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-20T21:35:05.992 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:05 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-20T21:35:05.992 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:05 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-04-20T21:35:05.992 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:05 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:35:05.992 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:05 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:35:05.992 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:05 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T21:35:05.992 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:05 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:35:05.992 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:05 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:35:05.992 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:05 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:35:05.992 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:05 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T21:35:06.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:05 smithi062 ceph-mon[16221]: pgmap v107: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:35:06.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:05 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:35:06.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:05 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:35:06.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:05 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:35:06.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:05 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T21:35:06.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:05 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:35:06.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:05 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-20T21:35:06.037 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:05 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:35:06.037 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:05 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:35:06.037 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:05 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:35:06.037 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:05 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:35:06.037 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:05 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:35:06.037 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:05 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:35:06.037 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:05 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:35:06.037 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:05 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:35:06.037 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:05 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:35:06.037 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:05 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:35:06.037 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:05 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:35:06.037 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:05 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:35:06.037 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:05 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:35:06.037 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:05 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:35:06.038 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:05 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:35:06.038 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:05 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-20T21:35:06.038 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:05 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-04-20T21:35:06.038 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:05 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-20T21:35:06.038 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:05 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-04-20T21:35:06.038 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:05 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-20T21:35:06.038 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:05 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-20T21:35:06.039 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:05 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-20T21:35:06.039 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:05 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-20T21:35:06.039 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:05 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-04-20T21:35:06.039 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:05 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-20T21:35:06.039 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:05 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-04-20T21:35:06.039 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:05 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-20T21:35:06.039 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:05 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-20T21:35:06.039 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:05 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-20T21:35:06.039 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:05 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-04-20T21:35:06.039 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:05 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-20T21:35:06.039 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:05 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-04-20T21:35:06.039 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:05 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-20T21:35:06.039 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:05 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-04-20T21:35:06.040 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:05 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-20T21:35:06.040 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:05 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-20T21:35:06.040 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:05 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-20T21:35:06.040 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:05 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-20T21:35:06.040 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:05 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-20T21:35:06.040 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:05 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-20T21:35:06.040 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:05 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-20T21:35:06.040 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:05 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-04-20T21:35:06.040 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:05 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:35:06.040 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:05 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:35:06.040 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:05 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T21:35:06.040 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:05 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:35:06.040 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:05 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:35:06.040 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:05 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:35:06.040 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:05 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T21:35:06.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:06 smithi189 ceph-mon[21562]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-20T21:35:06.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:06 smithi189 ceph-mon[21562]: Upgrade: Finalizing container_image settings 2024-04-20T21:35:06.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:06 smithi189 ceph-mon[21562]: Upgrade: Complete! 2024-04-20T21:35:06.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:06 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:35:06.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:06 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:35:07.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:06 smithi062 ceph-mon[16221]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-20T21:35:07.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:06 smithi062 ceph-mon[16221]: Upgrade: Finalizing container_image settings 2024-04-20T21:35:07.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:06 smithi062 ceph-mon[16221]: Upgrade: Complete! 2024-04-20T21:35:07.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:06 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:35:07.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:06 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:35:08.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:07 smithi062 ceph-mon[16221]: pgmap v108: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:35:08.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:07 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:35:08.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:07 smithi189 ceph-mon[21562]: pgmap v108: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:35:08.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:07 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:35:10.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:09 smithi062 ceph-mon[16221]: pgmap v109: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:35:10.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:09 smithi189 ceph-mon[21562]: pgmap v109: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:35:12.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:11 smithi062 ceph-mon[16221]: pgmap v110: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:35:12.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:11 smithi189 ceph-mon[21562]: pgmap v110: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:35:13.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:13 smithi062 ceph-mon[16221]: pgmap v111: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:35:13.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:13 smithi189 ceph-mon[21562]: pgmap v111: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:35:15.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:15 smithi189 ceph-mon[21562]: pgmap v112: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:35:16.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:15 smithi062 ceph-mon[16221]: pgmap v112: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:35:17.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:17 smithi189 ceph-mon[21562]: pgmap v113: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:35:18.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:17 smithi062 ceph-mon[16221]: pgmap v113: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:35:19.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:19 smithi189 ceph-mon[21562]: pgmap v114: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:35:20.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:19 smithi062 ceph-mon[16221]: pgmap v114: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:35:21.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:21 smithi189 ceph-mon[21562]: pgmap v115: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:35:21.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:21 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:35:22.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:21 smithi062 ceph-mon[16221]: pgmap v115: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:35:22.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:21 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:35:23.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:23 smithi062 ceph-mon[16221]: pgmap v116: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-20T21:35:23.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:23 smithi189 ceph-mon[21562]: pgmap v116: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-20T21:35:25.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:25 smithi189 ceph-mon[21562]: pgmap v117: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:35:26.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:25 smithi062 ceph-mon[16221]: pgmap v117: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:35:27.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:27 smithi189 ceph-mon[21562]: pgmap v118: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:35:28.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:27 smithi062 ceph-mon[16221]: pgmap v118: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:35:28.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:28.824+0000 7fb50b600700 1 -- 172.21.15.62:0/284623941 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb5041036d0 msgr2=0x7fb504103b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:35:28.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:28.824+0000 7fb50b600700 1 --2- 172.21.15.62:0/284623941 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb5041036d0 0x7fb504103b40 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fb4f0009910 tx=0x7fb4f0009c20 comp rx=0 tx=0).stop 2024-04-20T21:35:28.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:28.824+0000 7fb50b600700 1 -- 172.21.15.62:0/284623941 shutdown_connections 2024-04-20T21:35:28.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:28.824+0000 7fb50b600700 1 --2- 172.21.15.62:0/284623941 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb5041036d0 0x7fb504103b40 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:35:28.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:28.824+0000 7fb50b600700 1 --2- 172.21.15.62:0/284623941 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb504102430 0x7fb504102840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:35:28.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:28.824+0000 7fb50b600700 1 -- 172.21.15.62:0/284623941 >> 172.21.15.62:0/284623941 conn(0x7fb5040fda80 msgr2=0x7fb5040ffeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:35:28.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:28.824+0000 7fb50b600700 1 -- 172.21.15.62:0/284623941 shutdown_connections 2024-04-20T21:35:28.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:28.825+0000 7fb50b600700 1 -- 172.21.15.62:0/284623941 wait complete. 2024-04-20T21:35:28.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:28.826+0000 7fb50b600700 1 Processor -- start 2024-04-20T21:35:28.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:28.826+0000 7fb50b600700 1 -- start start 2024-04-20T21:35:28.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:28.826+0000 7fb50b600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb504102430 0x7fb504197c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:35:28.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:28.826+0000 7fb50b600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb5041036d0 0x7fb5041981d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:35:28.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:28.826+0000 7fb50b600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5041987d0 con 0x7fb5041036d0 2024-04-20T21:35:28.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:28.826+0000 7fb50b600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb504198940 con 0x7fb504102430 2024-04-20T21:35:28.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:28.826+0000 7fb503e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb5041036d0 0x7fb5041981d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:35:28.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:28.827+0000 7fb503e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb5041036d0 0x7fb5041981d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:38412/0 (socket says 172.21.15.62:38412) 2024-04-20T21:35:28.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:28.827+0000 7fb503e00700 1 -- 172.21.15.62:0/1859073409 learned_addr learned my addr 172.21.15.62:0/1859073409 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:35:28.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:28.827+0000 7fb508c00700 1 --2- 172.21.15.62:0/1859073409 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb504102430 0x7fb504197c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:35:28.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:28.827+0000 7fb503e00700 1 -- 172.21.15.62:0/1859073409 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb504102430 msgr2=0x7fb504197c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:35:28.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:28.827+0000 7fb503e00700 1 --2- 172.21.15.62:0/1859073409 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb504102430 0x7fb504197c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:35:28.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:28.827+0000 7fb503e00700 1 -- 172.21.15.62:0/1859073409 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4f00095f0 con 0x7fb5041036d0 2024-04-20T21:35:28.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:28.827+0000 7fb503e00700 1 --2- 172.21.15.62:0/1859073409 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb5041036d0 0x7fb5041981d0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fb4f000bd70 tx=0x7fb4f000be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:35:28.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:28.828+0000 7fb501600700 1 -- 172.21.15.62:0/1859073409 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb4f001d070 con 0x7fb5041036d0 2024-04-20T21:35:28.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:28.828+0000 7fb501600700 1 -- 172.21.15.62:0/1859073409 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb4f000f460 con 0x7fb5041036d0 2024-04-20T21:35:28.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:28.828+0000 7fb50b600700 1 -- 172.21.15.62:0/1859073409 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb50419d370 con 0x7fb5041036d0 2024-04-20T21:35:28.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:28.828+0000 7fb501600700 1 -- 172.21.15.62:0/1859073409 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb4f0021640 con 0x7fb5041036d0 2024-04-20T21:35:28.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:28.828+0000 7fb50b600700 1 -- 172.21.15.62:0/1859073409 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb50419d860 con 0x7fb5041036d0 2024-04-20T21:35:28.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:28.829+0000 7fb50b600700 1 -- 172.21.15.62:0/1859073409 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb50410af10 con 0x7fb5041036d0 2024-04-20T21:35:28.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:28.830+0000 7fb501600700 1 -- 172.21.15.62:0/1859073409 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 31) v1 ==== 92598+0+0 (secure 0 0 0) 0x7fb4f000fab0 con 0x7fb5041036d0 2024-04-20T21:35:28.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:28.831+0000 7fb501600700 1 --2- 172.21.15.62:0/1859073409 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7fb4ec06e940 0x7fb4ec070df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:35:28.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:28.831+0000 7fb501600700 1 -- 172.21.15.62:0/1859073409 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6070+0+0 (secure 0 0 0) 0x7fb4f008fca0 con 0x7fb5041036d0 2024-04-20T21:35:28.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:28.831+0000 7fb508c00700 1 --2- 172.21.15.62:0/1859073409 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7fb4ec06e940 0x7fb4ec070df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:35:28.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:28.832+0000 7fb508c00700 1 --2- 172.21.15.62:0/1859073409 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7fb4ec06e940 0x7fb4ec070df0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb4f8005950 tx=0x7fb4f8005ed0 comp rx=0 tx=0).ready entity=mgr.14658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:35:28.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:28.836+0000 7fb501600700 1 -- 172.21.15.62:0/1859073409 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187527 (secure 0 0 0) 0x7fb4f0094070 con 0x7fb5041036d0 2024-04-20T21:35:29.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:29.104+0000 7fb50b600700 1 -- 172.21.15.62:0/1859073409 --> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb504108020 con 0x7fb4ec06e940 2024-04-20T21:35:29.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:29.107+0000 7fb501600700 1 -- 172.21.15.62:0/1859073409 <== mgr.14658 v2:172.21.15.62:6800/1585760929 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+175 (secure 0 0 0) 0x7fb504108020 con 0x7fb4ec06e940 2024-04-20T21:35:29.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:29.110+0000 7fb50b600700 1 -- 172.21.15.62:0/1859073409 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7fb4ec06e940 msgr2=0x7fb4ec070df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:35:29.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:29.110+0000 7fb50b600700 1 --2- 172.21.15.62:0/1859073409 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7fb4ec06e940 0x7fb4ec070df0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb4f8005950 tx=0x7fb4f8005ed0 comp rx=0 tx=0).stop 2024-04-20T21:35:29.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:29.110+0000 7fb50b600700 1 -- 172.21.15.62:0/1859073409 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb5041036d0 msgr2=0x7fb5041981d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:35:29.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:29.110+0000 7fb50b600700 1 --2- 172.21.15.62:0/1859073409 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb5041036d0 0x7fb5041981d0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fb4f000bd70 tx=0x7fb4f000be50 comp rx=0 tx=0).stop 2024-04-20T21:35:29.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:29.110+0000 7fb50b600700 1 -- 172.21.15.62:0/1859073409 shutdown_connections 2024-04-20T21:35:29.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:29.110+0000 7fb50b600700 1 --2- 172.21.15.62:0/1859073409 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7fb4ec06e940 0x7fb4ec070df0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:35:29.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:29.110+0000 7fb50b600700 1 --2- 172.21.15.62:0/1859073409 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb504102430 0x7fb504197c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:35:29.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:29.110+0000 7fb50b600700 1 --2- 172.21.15.62:0/1859073409 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb5041036d0 0x7fb5041981d0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:35:29.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:29.110+0000 7fb50b600700 1 -- 172.21.15.62:0/1859073409 >> 172.21.15.62:0/1859073409 conn(0x7fb5040fda80 msgr2=0x7fb504106900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:35:29.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:29.111+0000 7fb50b600700 1 -- 172.21.15.62:0/1859073409 shutdown_connections 2024-04-20T21:35:29.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:29.111+0000 7fb50b600700 1 -- 172.21.15.62:0/1859073409 wait complete. 2024-04-20T21:35:29.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:29 smithi062 ceph-mon[16221]: pgmap v119: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:35:29.952 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -e sha1=a8c84f5997950fd7d008204839253291d52e6eda -- bash -c 'ceph versions | jq -e '"'"'.mgr | length == 1'"'"'' 2024-04-20T21:35:29.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:29 smithi189 ceph-mon[21562]: pgmap v119: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:35:30.392 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T21:35:30.902 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:30 smithi062 ceph-mon[16221]: from='client.14832 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:35:30.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:30 smithi189 ceph-mon[21562]: from='client.14832 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:35:31.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:31 smithi189 ceph-mon[21562]: pgmap v120: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:35:32.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:31 smithi062 ceph-mon[16221]: pgmap v120: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:35:32.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:32.536+0000 7fc2e2000700 1 -- 172.21.15.62:0/3715663105 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc2dc1004e0 msgr2=0x7fc2dc1008b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:35:32.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:32.536+0000 7fc2e2000700 1 --2- 172.21.15.62:0/3715663105 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc2dc1004e0 0x7fc2dc1008b0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fc2cc009910 tx=0x7fc2cc009c20 comp rx=0 tx=0).stop 2024-04-20T21:35:32.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:32.536+0000 7fc2e2000700 1 -- 172.21.15.62:0/3715663105 shutdown_connections 2024-04-20T21:35:32.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:32.536+0000 7fc2e2000700 1 --2- 172.21.15.62:0/3715663105 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc2dc100e80 0x7fc2dc104e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:35:32.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:32.536+0000 7fc2e2000700 1 --2- 172.21.15.62:0/3715663105 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc2dc1004e0 0x7fc2dc1008b0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:35:32.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:32.536+0000 7fc2e2000700 1 -- 172.21.15.62:0/3715663105 >> 172.21.15.62:0/3715663105 conn(0x7fc2dc0fbda0 msgr2=0x7fc2dc0fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:35:32.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:32.536+0000 7fc2e2000700 1 -- 172.21.15.62:0/3715663105 shutdown_connections 2024-04-20T21:35:32.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:32.536+0000 7fc2e2000700 1 -- 172.21.15.62:0/3715663105 wait complete. 2024-04-20T21:35:32.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:32.537+0000 7fc2e2000700 1 Processor -- start 2024-04-20T21:35:32.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:32.538+0000 7fc2e2000700 1 -- start start 2024-04-20T21:35:32.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:32.538+0000 7fc2e2000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc2dc1004e0 0x7fc2dc195ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:35:32.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:32.538+0000 7fc2e2000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc2dc100e80 0x7fc2dc196430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:35:32.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:32.538+0000 7fc2e2000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2dc196b10 con 0x7fc2dc100e80 2024-04-20T21:35:32.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:32.538+0000 7fc2e2000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2dc19a8a0 con 0x7fc2dc1004e0 2024-04-20T21:35:32.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:32.538+0000 7fc2daa00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc2dc100e80 0x7fc2dc196430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:35:32.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:32.538+0000 7fc2daa00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc2dc100e80 0x7fc2dc196430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42864/0 (socket says 172.21.15.62:42864) 2024-04-20T21:35:32.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:32.538+0000 7fc2daa00700 1 -- 172.21.15.62:0/141023878 learned_addr learned my addr 172.21.15.62:0/141023878 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:35:32.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:32.539+0000 7fc2db400700 1 --2- 172.21.15.62:0/141023878 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc2dc1004e0 0x7fc2dc195ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:35:32.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:32.539+0000 7fc2daa00700 1 -- 172.21.15.62:0/141023878 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc2dc1004e0 msgr2=0x7fc2dc195ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:35:32.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:32.539+0000 7fc2daa00700 1 --2- 172.21.15.62:0/141023878 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc2dc1004e0 0x7fc2dc195ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:35:32.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:32.539+0000 7fc2daa00700 1 -- 172.21.15.62:0/141023878 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2cc0095f0 con 0x7fc2dc100e80 2024-04-20T21:35:32.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:32.539+0000 7fc2daa00700 1 --2- 172.21.15.62:0/141023878 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc2dc100e80 0x7fc2dc196430 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fc2c800d900 tx=0x7fc2c800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:35:32.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:32.539+0000 7fc2d3e00700 1 -- 172.21.15.62:0/141023878 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc2c8009900 con 0x7fc2dc100e80 2024-04-20T21:35:32.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:32.540+0000 7fc2d3e00700 1 -- 172.21.15.62:0/141023878 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc2c8010460 con 0x7fc2dc100e80 2024-04-20T21:35:32.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:32.540+0000 7fc2e2000700 1 -- 172.21.15.62:0/141023878 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2dc19ab80 con 0x7fc2dc100e80 2024-04-20T21:35:32.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:32.540+0000 7fc2d3e00700 1 -- 172.21.15.62:0/141023878 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc2c800f5f0 con 0x7fc2dc100e80 2024-04-20T21:35:32.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:32.540+0000 7fc2e2000700 1 -- 172.21.15.62:0/141023878 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2dc19b0d0 con 0x7fc2dc100e80 2024-04-20T21:35:32.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:32.541+0000 7fc2e2000700 1 -- 172.21.15.62:0/141023878 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2dc1087f0 con 0x7fc2dc100e80 2024-04-20T21:35:32.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:32.542+0000 7fc2d3e00700 1 -- 172.21.15.62:0/141023878 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 31) v1 ==== 92598+0+0 (secure 0 0 0) 0x7fc2c80105d0 con 0x7fc2dc100e80 2024-04-20T21:35:32.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:32.543+0000 7fc2d3e00700 1 --2- 172.21.15.62:0/141023878 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7fc2c006e940 0x7fc2c0070df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:35:32.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:32.543+0000 7fc2d3e00700 1 -- 172.21.15.62:0/141023878 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6070+0+0 (secure 0 0 0) 0x7fc2c808e700 con 0x7fc2dc100e80 2024-04-20T21:35:32.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:32.543+0000 7fc2db400700 1 --2- 172.21.15.62:0/141023878 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7fc2c006e940 0x7fc2c0070df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:35:32.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:32.544+0000 7fc2db400700 1 --2- 172.21.15.62:0/141023878 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7fc2c006e940 0x7fc2c0070df0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc2cc00b5c0 tx=0x7fc2cc01a040 comp rx=0 tx=0).ready entity=mgr.14658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:35:32.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:32.548+0000 7fc2d3e00700 1 -- 172.21.15.62:0/141023878 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187527 (secure 0 0 0) 0x7fc2c80579e0 con 0x7fc2dc100e80 2024-04-20T21:35:32.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:32.884+0000 7fc2e2000700 1 -- 172.21.15.62:0/141023878 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fc2dc04f2a0 con 0x7fc2dc100e80 2024-04-20T21:35:32.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:32.885+0000 7fc2d3e00700 1 -- 172.21.15.62:0/141023878 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+675 (secure 0 0 0) 0x7fc2c805b0a0 con 0x7fc2dc100e80 2024-04-20T21:35:32.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:32.888+0000 7fc2e2000700 1 -- 172.21.15.62:0/141023878 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7fc2c006e940 msgr2=0x7fc2c0070df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:35:32.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:32.888+0000 7fc2e2000700 1 --2- 172.21.15.62:0/141023878 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7fc2c006e940 0x7fc2c0070df0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc2cc00b5c0 tx=0x7fc2cc01a040 comp rx=0 tx=0).stop 2024-04-20T21:35:32.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:32.888+0000 7fc2e2000700 1 -- 172.21.15.62:0/141023878 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc2dc100e80 msgr2=0x7fc2dc196430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:35:32.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:32.888+0000 7fc2e2000700 1 --2- 172.21.15.62:0/141023878 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc2dc100e80 0x7fc2dc196430 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fc2c800d900 tx=0x7fc2c800dcc0 comp rx=0 tx=0).stop 2024-04-20T21:35:32.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:32.888+0000 7fc2e2000700 1 -- 172.21.15.62:0/141023878 shutdown_connections 2024-04-20T21:35:32.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:32.888+0000 7fc2e2000700 1 --2- 172.21.15.62:0/141023878 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7fc2c006e940 0x7fc2c0070df0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:35:32.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:32.888+0000 7fc2e2000700 1 --2- 172.21.15.62:0/141023878 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc2dc1004e0 0x7fc2dc195ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:35:32.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:32.888+0000 7fc2e2000700 1 --2- 172.21.15.62:0/141023878 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc2dc100e80 0x7fc2dc196430 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:35:32.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:32.888+0000 7fc2e2000700 1 -- 172.21.15.62:0/141023878 >> 172.21.15.62:0/141023878 conn(0x7fc2dc0fbda0 msgr2=0x7fc2dc0fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:35:32.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:32.888+0000 7fc2e2000700 1 -- 172.21.15.62:0/141023878 shutdown_connections 2024-04-20T21:35:32.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:32.888+0000 7fc2e2000700 1 -- 172.21.15.62:0/141023878 wait complete. 2024-04-20T21:35:32.906 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T21:35:33.219 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:33 smithi062 ceph-mon[16221]: pgmap v121: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:35:33.219 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:33 smithi062 ceph-mon[16221]: from='client.? 172.21.15.62:0/141023878' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:35:33.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:33 smithi189 ceph-mon[21562]: pgmap v121: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:35:33.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:33 smithi189 ceph-mon[21562]: from='client.? 172.21.15.62:0/141023878' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:35:33.773 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -e sha1=a8c84f5997950fd7d008204839253291d52e6eda -- bash -c 'ceph versions | jq -e '"'"'.mgr | keys'"'"' | grep $sha1' 2024-04-20T21:35:34.204 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T21:35:35.969 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:35 smithi062 ceph-mon[16221]: pgmap v122: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-20T21:35:35.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:35 smithi189 ceph-mon[21562]: pgmap v122: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-20T21:35:36.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:36.529+0000 7f1528e00700 1 -- 172.21.15.62:0/2133750544 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1524068540 msgr2=0x7f15240689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:35:36.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:36.529+0000 7f1528e00700 1 --2- 172.21.15.62:0/2133750544 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1524068540 0x7f15240689b0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f150c009910 tx=0x7f150c009c20 comp rx=0 tx=0).stop 2024-04-20T21:35:36.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:36.529+0000 7f1528e00700 1 -- 172.21.15.62:0/2133750544 shutdown_connections 2024-04-20T21:35:36.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:36.529+0000 7f1528e00700 1 --2- 172.21.15.62:0/2133750544 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1524068540 0x7f15240689b0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:35:36.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:36.529+0000 7f1528e00700 1 --2- 172.21.15.62:0/2133750544 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1524105230 0x7f1524105600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:35:36.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:36.529+0000 7f1528e00700 1 -- 172.21.15.62:0/2133750544 >> 172.21.15.62:0/2133750544 conn(0x7f1524075130 msgr2=0x7f1524075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:35:36.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:36.529+0000 7f1528e00700 1 -- 172.21.15.62:0/2133750544 shutdown_connections 2024-04-20T21:35:36.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:36.529+0000 7f1528e00700 1 -- 172.21.15.62:0/2133750544 wait complete. 2024-04-20T21:35:36.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:36.531+0000 7f1528e00700 1 Processor -- start 2024-04-20T21:35:36.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:36.531+0000 7f1528e00700 1 -- start start 2024-04-20T21:35:36.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:36.531+0000 7f1528e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1524068540 0x7f1524197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:35:36.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:36.531+0000 7f1528e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1524105230 0x7f1524198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:35:36.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:36.531+0000 7f1528e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1524198b60 con 0x7f1524105230 2024-04-20T21:35:36.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:36.531+0000 7f1528e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1524198ca0 con 0x7f1524068540 2024-04-20T21:35:36.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:36.532+0000 7f1519400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1524105230 0x7f1524198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:35:36.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:36.532+0000 7f1519400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1524105230 0x7f1524198510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42874/0 (socket says 172.21.15.62:42874) 2024-04-20T21:35:36.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:36.532+0000 7f1519400700 1 -- 172.21.15.62:0/3708477068 learned_addr learned my addr 172.21.15.62:0/3708477068 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:35:36.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:36.532+0000 7f1521e00700 1 --2- 172.21.15.62:0/3708477068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1524068540 0x7f1524197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:35:36.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:36.532+0000 7f1519400700 1 -- 172.21.15.62:0/3708477068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1524068540 msgr2=0x7f1524197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:35:36.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:36.532+0000 7f1519400700 1 --2- 172.21.15.62:0/3708477068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1524068540 0x7f1524197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:35:36.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:36.532+0000 7f1519400700 1 -- 172.21.15.62:0/3708477068 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f150c0095f0 con 0x7f1524105230 2024-04-20T21:35:36.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:36.533+0000 7f1519400700 1 --2- 172.21.15.62:0/3708477068 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1524105230 0x7f1524198510 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f150c006010 tx=0x7f150c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:35:36.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:36.533+0000 7f151b400700 1 -- 172.21.15.62:0/3708477068 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f150c01d070 con 0x7f1524105230 2024-04-20T21:35:36.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:36.533+0000 7f151b400700 1 -- 172.21.15.62:0/3708477068 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f150c004fb0 con 0x7f1524105230 2024-04-20T21:35:36.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:36.533+0000 7f151b400700 1 -- 172.21.15.62:0/3708477068 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f150c022ea0 con 0x7f1524105230 2024-04-20T21:35:36.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:36.533+0000 7f1528e00700 1 -- 172.21.15.62:0/3708477068 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f152419ca90 con 0x7f1524105230 2024-04-20T21:35:36.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:36.533+0000 7f1528e00700 1 -- 172.21.15.62:0/3708477068 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f152419cf80 con 0x7f1524105230 2024-04-20T21:35:36.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:36.535+0000 7f1528e00700 1 -- 172.21.15.62:0/3708477068 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1524103a80 con 0x7f1524105230 2024-04-20T21:35:36.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:36.536+0000 7f151b400700 1 -- 172.21.15.62:0/3708477068 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 31) v1 ==== 92598+0+0 (secure 0 0 0) 0x7f150c022470 con 0x7f1524105230 2024-04-20T21:35:36.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:36.537+0000 7f151b400700 1 --2- 172.21.15.62:0/3708477068 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f150806e940 0x7f1508070df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:35:36.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:36.537+0000 7f151b400700 1 -- 172.21.15.62:0/3708477068 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f150c08fe30 con 0x7f1524105230 2024-04-20T21:35:36.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:36.537+0000 7f1521e00700 1 --2- 172.21.15.62:0/3708477068 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f150806e940 0x7f1508070df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:35:36.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:36.538+0000 7f1521e00700 1 --2- 172.21.15.62:0/3708477068 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f150806e940 0x7f1508070df0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f151400e270 tx=0x7f1514009040 comp rx=0 tx=0).ready entity=mgr.14658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:35:36.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:36.542+0000 7f151b400700 1 -- 172.21.15.62:0/3708477068 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187527 (secure 0 0 0) 0x7f150c0591a0 con 0x7f1524105230 2024-04-20T21:35:36.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:36.881+0000 7f1528e00700 1 -- 172.21.15.62:0/3708477068 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f152404f2a0 con 0x7f1524105230 2024-04-20T21:35:36.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:36.882+0000 7f151b400700 1 -- 172.21.15.62:0/3708477068 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+675 (secure 0 0 0) 0x7f150c027070 con 0x7f1524105230 2024-04-20T21:35:36.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:36.885+0000 7f1528e00700 1 -- 172.21.15.62:0/3708477068 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f150806e940 msgr2=0x7f1508070df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:35:36.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:36.885+0000 7f1528e00700 1 --2- 172.21.15.62:0/3708477068 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f150806e940 0x7f1508070df0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f151400e270 tx=0x7f1514009040 comp rx=0 tx=0).stop 2024-04-20T21:35:36.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:36.885+0000 7f1528e00700 1 -- 172.21.15.62:0/3708477068 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1524105230 msgr2=0x7f1524198510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:35:36.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:36.885+0000 7f1528e00700 1 --2- 172.21.15.62:0/3708477068 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1524105230 0x7f1524198510 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f150c006010 tx=0x7f150c00be20 comp rx=0 tx=0).stop 2024-04-20T21:35:36.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:36.885+0000 7f1528e00700 1 -- 172.21.15.62:0/3708477068 shutdown_connections 2024-04-20T21:35:36.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:36.885+0000 7f1528e00700 1 --2- 172.21.15.62:0/3708477068 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f150806e940 0x7f1508070df0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:35:36.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:36.885+0000 7f1528e00700 1 --2- 172.21.15.62:0/3708477068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1524068540 0x7f1524197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:35:36.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:36.885+0000 7f1528e00700 1 --2- 172.21.15.62:0/3708477068 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1524105230 0x7f1524198510 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:35:36.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:36.885+0000 7f1528e00700 1 -- 172.21.15.62:0/3708477068 >> 172.21.15.62:0/3708477068 conn(0x7f1524075130 msgr2=0x7f152410a3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:35:36.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:36.885+0000 7f1528e00700 1 -- 172.21.15.62:0/3708477068 shutdown_connections 2024-04-20T21:35:36.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:36.886+0000 7f1528e00700 1 -- 172.21.15.62:0/3708477068 wait complete. 2024-04-20T21:35:36.902 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)" 2024-04-20T21:35:36.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:36 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:35:37.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:36 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:35:37.779 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -e sha1=a8c84f5997950fd7d008204839253291d52e6eda -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 2'"'"'' 2024-04-20T21:35:37.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:37 smithi189 ceph-mon[21562]: pgmap v123: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:35:37.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:37 smithi189 ceph-mon[21562]: from='client.? 172.21.15.62:0/3708477068' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:35:38.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:37 smithi062 ceph-mon[16221]: pgmap v123: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:35:38.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:37 smithi062 ceph-mon[16221]: from='client.? 172.21.15.62:0/3708477068' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:35:38.223 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T21:35:39.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:39 smithi189 ceph-mon[21562]: pgmap v124: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:35:39.999 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:39 smithi062 ceph-mon[16221]: pgmap v124: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:35:40.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:40.299+0000 7feb9bc00700 1 -- 172.21.15.62:0/2640013948 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb941004e0 msgr2=0x7feb941008b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:35:40.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:40.299+0000 7feb9bc00700 1 --2- 172.21.15.62:0/2640013948 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb941004e0 0x7feb941008b0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7feb8400b0a0 tx=0x7feb8400b3b0 comp rx=0 tx=0).stop 2024-04-20T21:35:40.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:40.299+0000 7feb9bc00700 1 -- 172.21.15.62:0/2640013948 shutdown_connections 2024-04-20T21:35:40.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:40.299+0000 7feb9bc00700 1 --2- 172.21.15.62:0/2640013948 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb94100e80 0x7feb94104e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:35:40.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:40.299+0000 7feb9bc00700 1 --2- 172.21.15.62:0/2640013948 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb941004e0 0x7feb941008b0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:35:40.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:40.299+0000 7feb9bc00700 1 -- 172.21.15.62:0/2640013948 >> 172.21.15.62:0/2640013948 conn(0x7feb940fbda0 msgr2=0x7feb940fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:35:40.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:40.299+0000 7feb9bc00700 1 -- 172.21.15.62:0/2640013948 shutdown_connections 2024-04-20T21:35:40.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:40.299+0000 7feb9bc00700 1 -- 172.21.15.62:0/2640013948 wait complete. 2024-04-20T21:35:40.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:40.300+0000 7feb9bc00700 1 Processor -- start 2024-04-20T21:35:40.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:40.300+0000 7feb9bc00700 1 -- start start 2024-04-20T21:35:40.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:40.301+0000 7feb9bc00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb941004e0 0x7feb94195f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:35:40.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:40.301+0000 7feb9bc00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb94100e80 0x7feb941964d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:35:40.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:40.301+0000 7feb9bc00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb94196bb0 con 0x7feb94100e80 2024-04-20T21:35:40.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:40.301+0000 7feb9bc00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb9419a940 con 0x7feb941004e0 2024-04-20T21:35:40.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:40.301+0000 7feb8be00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb94100e80 0x7feb941964d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:35:40.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:40.301+0000 7feb8be00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb94100e80 0x7feb941964d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42894/0 (socket says 172.21.15.62:42894) 2024-04-20T21:35:40.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:40.301+0000 7feb8be00700 1 -- 172.21.15.62:0/1097591198 learned_addr learned my addr 172.21.15.62:0/1097591198 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:35:40.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:40.301+0000 7feb99200700 1 --2- 172.21.15.62:0/1097591198 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb941004e0 0x7feb94195f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:35:40.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:40.302+0000 7feb8be00700 1 -- 172.21.15.62:0/1097591198 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb941004e0 msgr2=0x7feb94195f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:35:40.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:40.302+0000 7feb8be00700 1 --2- 172.21.15.62:0/1097591198 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb941004e0 0x7feb94195f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:35:40.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:40.302+0000 7feb8be00700 1 -- 172.21.15.62:0/1097591198 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb84009d00 con 0x7feb94100e80 2024-04-20T21:35:40.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:40.302+0000 7feb8be00700 1 --2- 172.21.15.62:0/1097591198 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb94100e80 0x7feb941964d0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7feb7c00c8f0 tx=0x7feb7c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:35:40.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:40.302+0000 7feb92000700 1 -- 172.21.15.62:0/1097591198 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feb7c007a50 con 0x7feb94100e80 2024-04-20T21:35:40.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:40.302+0000 7feb92000700 1 -- 172.21.15.62:0/1097591198 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7feb7c007bb0 con 0x7feb94100e80 2024-04-20T21:35:40.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:40.302+0000 7feb92000700 1 -- 172.21.15.62:0/1097591198 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feb7c018630 con 0x7feb94100e80 2024-04-20T21:35:40.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:40.302+0000 7feb9bc00700 1 -- 172.21.15.62:0/1097591198 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb9419ac20 con 0x7feb94100e80 2024-04-20T21:35:40.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:40.303+0000 7feb9bc00700 1 -- 172.21.15.62:0/1097591198 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb9419b120 con 0x7feb94100e80 2024-04-20T21:35:40.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:40.304+0000 7feb9bc00700 1 -- 172.21.15.62:0/1097591198 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb941087f0 con 0x7feb94100e80 2024-04-20T21:35:40.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:40.305+0000 7feb92000700 1 -- 172.21.15.62:0/1097591198 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 31) v1 ==== 92598+0+0 (secure 0 0 0) 0x7feb7c007d20 con 0x7feb94100e80 2024-04-20T21:35:40.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:40.306+0000 7feb92000700 1 --2- 172.21.15.62:0/1097591198 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7feb7806e940 0x7feb78070df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:35:40.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:40.307+0000 7feb92000700 1 -- 172.21.15.62:0/1097591198 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6070+0+0 (secure 0 0 0) 0x7feb7c08e930 con 0x7feb94100e80 2024-04-20T21:35:40.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:40.307+0000 7feb99200700 1 --2- 172.21.15.62:0/1097591198 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7feb7806e940 0x7feb78070df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:35:40.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:40.308+0000 7feb99200700 1 --2- 172.21.15.62:0/1097591198 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7feb7806e940 0x7feb78070df0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7feb84009480 tx=0x7feb8400b7f0 comp rx=0 tx=0).ready entity=mgr.14658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:35:40.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:40.312+0000 7feb92000700 1 -- 172.21.15.62:0/1097591198 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187527 (secure 0 0 0) 0x7feb7c057b70 con 0x7feb94100e80 2024-04-20T21:35:40.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:40.655+0000 7feb9bc00700 1 -- 172.21.15.62:0/1097591198 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7feb9419b3f0 con 0x7feb94100e80 2024-04-20T21:35:40.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:40.656+0000 7feb92000700 1 -- 172.21.15.62:0/1097591198 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+675 (secure 0 0 0) 0x7feb7c05b230 con 0x7feb94100e80 2024-04-20T21:35:40.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:40.659+0000 7feb9bc00700 1 -- 172.21.15.62:0/1097591198 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7feb7806e940 msgr2=0x7feb78070df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:35:40.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:40.659+0000 7feb9bc00700 1 --2- 172.21.15.62:0/1097591198 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7feb7806e940 0x7feb78070df0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7feb84009480 tx=0x7feb8400b7f0 comp rx=0 tx=0).stop 2024-04-20T21:35:40.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:40.659+0000 7feb9bc00700 1 -- 172.21.15.62:0/1097591198 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb94100e80 msgr2=0x7feb941964d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:35:40.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:40.659+0000 7feb9bc00700 1 --2- 172.21.15.62:0/1097591198 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb94100e80 0x7feb941964d0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7feb7c00c8f0 tx=0x7feb7c00ccb0 comp rx=0 tx=0).stop 2024-04-20T21:35:40.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:40.659+0000 7feb9bc00700 1 -- 172.21.15.62:0/1097591198 shutdown_connections 2024-04-20T21:35:40.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:40.659+0000 7feb9bc00700 1 --2- 172.21.15.62:0/1097591198 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7feb7806e940 0x7feb78070df0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:35:40.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:40.659+0000 7feb9bc00700 1 --2- 172.21.15.62:0/1097591198 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb941004e0 0x7feb94195f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:35:40.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:40.659+0000 7feb9bc00700 1 --2- 172.21.15.62:0/1097591198 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb94100e80 0x7feb941964d0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:35:40.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:40.659+0000 7feb9bc00700 1 -- 172.21.15.62:0/1097591198 >> 172.21.15.62:0/1097591198 conn(0x7feb940fbda0 msgr2=0x7feb940fd410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:35:40.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:40.659+0000 7feb9bc00700 1 -- 172.21.15.62:0/1097591198 shutdown_connections 2024-04-20T21:35:40.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:40.660+0000 7feb9bc00700 1 -- 172.21.15.62:0/1097591198 wait complete. 2024-04-20T21:35:40.674 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T21:35:40.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:40 smithi189 ceph-mon[21562]: from='client.? 172.21.15.62:0/1097591198' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:35:41.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:40 smithi062 ceph-mon[16221]: from='client.? 172.21.15.62:0/1097591198' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:35:41.471 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -e sha1=a8c84f5997950fd7d008204839253291d52e6eda -- bash -c 'ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '"'"'.up_to_date | length == 2'"'"'' 2024-04-20T21:35:41.900 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T21:35:42.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:41 smithi062 ceph-mon[16221]: pgmap v125: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:35:42.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:41 smithi189 ceph-mon[21562]: pgmap v125: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:35:43.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:43 smithi062 ceph-mon[16221]: pgmap v126: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:35:43.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:43 smithi189 ceph-mon[21562]: pgmap v126: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:35:44.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:44.293+0000 7f04e3200700 1 -- 172.21.15.62:0/3675551376 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04e40fecd0 msgr2=0x7f04e4104cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:35:44.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:44.293+0000 7f04e3200700 1 --2- 172.21.15.62:0/3675551376 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04e40fecd0 0x7f04e4104cb0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f04d0009910 tx=0x7f04d0009c20 comp rx=0 tx=0).stop 2024-04-20T21:35:44.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:44.293+0000 7f04e3200700 1 -- 172.21.15.62:0/3675551376 shutdown_connections 2024-04-20T21:35:44.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:44.293+0000 7f04e3200700 1 --2- 172.21.15.62:0/3675551376 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04e40fecd0 0x7f04e4104cb0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:35:44.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:44.293+0000 7f04e3200700 1 --2- 172.21.15.62:0/3675551376 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f04e40fe330 0x7f04e40fe700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:35:44.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:44.293+0000 7f04e3200700 1 -- 172.21.15.62:0/3675551376 >> 172.21.15.62:0/3675551376 conn(0x7f04e40f9d40 msgr2=0x7f04e40fc150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:35:44.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:44.293+0000 7f04e3200700 1 -- 172.21.15.62:0/3675551376 shutdown_connections 2024-04-20T21:35:44.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:44.293+0000 7f04e3200700 1 -- 172.21.15.62:0/3675551376 wait complete. 2024-04-20T21:35:44.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:44.294+0000 7f04e3200700 1 Processor -- start 2024-04-20T21:35:44.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:44.294+0000 7f04e3200700 1 -- start start 2024-04-20T21:35:44.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:44.295+0000 7f04e3200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04e40fe330 0x7f04e4197d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:35:44.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:44.295+0000 7f04e3200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f04e40fecd0 0x7f04e41982d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:35:44.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:44.295+0000 7f04e3200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04e41989b0 con 0x7f04e40fe330 2024-04-20T21:35:44.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:44.295+0000 7f04e3200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04e419c740 con 0x7f04e40fecd0 2024-04-20T21:35:44.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:44.295+0000 7f04e1e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04e40fe330 0x7f04e4197d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:35:44.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:44.295+0000 7f04e1e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04e40fe330 0x7f04e4197d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49890/0 (socket says 172.21.15.62:49890) 2024-04-20T21:35:44.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:44.295+0000 7f04e1e00700 1 -- 172.21.15.62:0/1373560299 learned_addr learned my addr 172.21.15.62:0/1373560299 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:35:44.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:44.295+0000 7f04e1400700 1 --2- 172.21.15.62:0/1373560299 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f04e40fecd0 0x7f04e41982d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:35:44.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:44.295+0000 7f04e1e00700 1 -- 172.21.15.62:0/1373560299 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f04e40fecd0 msgr2=0x7f04e41982d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:35:44.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:44.295+0000 7f04e1e00700 1 --2- 172.21.15.62:0/1373560299 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f04e40fecd0 0x7f04e41982d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:35:44.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:44.295+0000 7f04e1e00700 1 -- 172.21.15.62:0/1373560299 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04d00095f0 con 0x7f04e40fe330 2024-04-20T21:35:44.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:44.295+0000 7f04e1400700 1 --2- 172.21.15.62:0/1373560299 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f04e40fecd0 0x7f04e41982d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T21:35:44.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:44.296+0000 7f04e1e00700 1 --2- 172.21.15.62:0/1373560299 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04e40fe330 0x7f04e4197d90 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f04d400d8d0 tx=0x7f04d400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:35:44.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:44.296+0000 7f04daa00700 1 -- 172.21.15.62:0/1373560299 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f04d4009940 con 0x7f04e40fe330 2024-04-20T21:35:44.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:44.296+0000 7f04daa00700 1 -- 172.21.15.62:0/1373560299 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f04d4010460 con 0x7f04e40fe330 2024-04-20T21:35:44.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:44.296+0000 7f04daa00700 1 -- 172.21.15.62:0/1373560299 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f04d400f5f0 con 0x7f04e40fe330 2024-04-20T21:35:44.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:44.296+0000 7f04e3200700 1 -- 172.21.15.62:0/1373560299 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04e419ca20 con 0x7f04e40fe330 2024-04-20T21:35:44.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:44.296+0000 7f04e3200700 1 -- 172.21.15.62:0/1373560299 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04e419cff0 con 0x7f04e40fe330 2024-04-20T21:35:44.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:44.298+0000 7f04e3200700 1 -- 172.21.15.62:0/1373560299 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04e4062380 con 0x7f04e40fe330 2024-04-20T21:35:44.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:44.299+0000 7f04daa00700 1 -- 172.21.15.62:0/1373560299 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 31) v1 ==== 92598+0+0 (secure 0 0 0) 0x7f04d40105d0 con 0x7f04e40fe330 2024-04-20T21:35:44.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:44.299+0000 7f04daa00700 1 --2- 172.21.15.62:0/1373560299 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f04c806e940 0x7f04c8070df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:35:44.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:44.300+0000 7f04daa00700 1 -- 172.21.15.62:0/1373560299 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f04d408e850 con 0x7f04e40fe330 2024-04-20T21:35:44.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:44.300+0000 7f04e1400700 1 --2- 172.21.15.62:0/1373560299 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f04c806e940 0x7f04c8070df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:35:44.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:44.301+0000 7f04e1400700 1 --2- 172.21.15.62:0/1373560299 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f04c806e940 0x7f04c8070df0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f04d000b5c0 tx=0x7f04d0005960 comp rx=0 tx=0).ready entity=mgr.14658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:35:44.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:44.305+0000 7f04daa00700 1 -- 172.21.15.62:0/1373560299 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187527 (secure 0 0 0) 0x7f04d4057b10 con 0x7f04e40fe330 2024-04-20T21:35:44.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:44.565+0000 7f04e3200700 1 -- 172.21.15.62:0/1373560299 --> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] -- mgr_command(tid 0: {"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:a8c84f5997950fd7d008204839253291d52e6eda", "target": ["mon-mgr", ""]}) v1 -- 0x7f04e419d410 con 0x7f04c806e940 2024-04-20T21:35:45.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:45 smithi189 ceph-mon[21562]: from='client.14848 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:a8c84f5997950fd7d008204839253291d52e6eda", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:35:45.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:45 smithi189 ceph-mon[21562]: pgmap v127: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:35:46.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:45 smithi062 ceph-mon[16221]: from='client.14848 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:a8c84f5997950fd7d008204839253291d52e6eda", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:35:46.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:45 smithi062 ceph-mon[16221]: pgmap v127: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:35:47.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:47.591+0000 7f04daa00700 1 -- 172.21.15.62:0/1373560299 <== mgr.14658 v2:172.21.15.62:6800/1585760929 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+5391 (secure 0 0 0) 0x7f04e419d410 con 0x7f04c806e940 2024-04-20T21:35:47.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:47.595+0000 7f04e3200700 1 -- 172.21.15.62:0/1373560299 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f04c806e940 msgr2=0x7f04c8070df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:35:47.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:47.595+0000 7f04e3200700 1 --2- 172.21.15.62:0/1373560299 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f04c806e940 0x7f04c8070df0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f04d000b5c0 tx=0x7f04d0005960 comp rx=0 tx=0).stop 2024-04-20T21:35:47.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:47.595+0000 7f04e3200700 1 -- 172.21.15.62:0/1373560299 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04e40fe330 msgr2=0x7f04e4197d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:35:47.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:47.595+0000 7f04e3200700 1 --2- 172.21.15.62:0/1373560299 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04e40fe330 0x7f04e4197d90 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f04d400d8d0 tx=0x7f04d400dc90 comp rx=0 tx=0).stop 2024-04-20T21:35:47.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:47.595+0000 7f04e3200700 1 -- 172.21.15.62:0/1373560299 shutdown_connections 2024-04-20T21:35:47.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:47.595+0000 7f04e3200700 1 --2- 172.21.15.62:0/1373560299 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f04c806e940 0x7f04c8070df0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:35:47.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:47.595+0000 7f04e3200700 1 --2- 172.21.15.62:0/1373560299 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04e40fe330 0x7f04e4197d90 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:35:47.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:47.595+0000 7f04e3200700 1 --2- 172.21.15.62:0/1373560299 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f04e40fecd0 0x7f04e41982d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:35:47.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:47.595+0000 7f04e3200700 1 -- 172.21.15.62:0/1373560299 >> 172.21.15.62:0/1373560299 conn(0x7f04e40f9d40 msgr2=0x7f04e40fb2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:35:47.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:47.596+0000 7f04e3200700 1 -- 172.21.15.62:0/1373560299 shutdown_connections 2024-04-20T21:35:47.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:47.596+0000 7f04e3200700 1 -- 172.21.15.62:0/1373560299 wait complete. 2024-04-20T21:35:47.619 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T21:35:47.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:47 smithi062 ceph-mon[16221]: pgmap v128: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:35:47.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:47 smithi189 ceph-mon[21562]: pgmap v128: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:35:48.416 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -e sha1=a8c84f5997950fd7d008204839253291d52e6eda -- bash -c 'ceph orch ps' 2024-04-20T21:35:48.846 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T21:35:49.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:49 smithi189 ceph-mon[21562]: pgmap v129: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:35:50.011 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:49 smithi062 ceph-mon[16221]: pgmap v129: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:35:50.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:50.967+0000 7f006e200700 1 -- 172.21.15.62:0/1498890713 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0068108470 msgr2=0x7f0068108840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:35:50.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:50.967+0000 7f006e200700 1 --2- 172.21.15.62:0/1498890713 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0068108470 0x7f0068108840 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f0058009910 tx=0x7f0058009c20 comp rx=0 tx=0).stop 2024-04-20T21:35:50.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:50.967+0000 7f006e200700 1 -- 172.21.15.62:0/1498890713 shutdown_connections 2024-04-20T21:35:50.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:50.967+0000 7f006e200700 1 --2- 172.21.15.62:0/1498890713 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0068102470 0x7f00681028e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:35:50.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:50.967+0000 7f006e200700 1 --2- 172.21.15.62:0/1498890713 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0068108470 0x7f0068108840 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:35:50.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:50.967+0000 7f006e200700 1 -- 172.21.15.62:0/1498890713 >> 172.21.15.62:0/1498890713 conn(0x7f00680fdf90 msgr2=0x7f00681003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:35:50.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:50.967+0000 7f006e200700 1 -- 172.21.15.62:0/1498890713 shutdown_connections 2024-04-20T21:35:50.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:50.967+0000 7f006e200700 1 -- 172.21.15.62:0/1498890713 wait complete. 2024-04-20T21:35:50.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:50.968+0000 7f006e200700 1 Processor -- start 2024-04-20T21:35:50.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:50.969+0000 7f006e200700 1 -- start start 2024-04-20T21:35:50.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:50.969+0000 7f006e200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0068102470 0x7f00681981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:35:50.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:50.969+0000 7f006e200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0068108470 0x7f0068198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:35:50.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:50.969+0000 7f006e200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0068198e10 con 0x7f0068102470 2024-04-20T21:35:50.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:50.969+0000 7f006e200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f006819cba0 con 0x7f0068108470 2024-04-20T21:35:50.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:50.969+0000 7f0067400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0068102470 0x7f00681981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:35:50.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:50.969+0000 7f0067400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0068102470 0x7f00681981f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49914/0 (socket says 172.21.15.62:49914) 2024-04-20T21:35:50.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:50.969+0000 7f0067400700 1 -- 172.21.15.62:0/710946334 learned_addr learned my addr 172.21.15.62:0/710946334 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:35:50.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:50.970+0000 7f0067400700 1 -- 172.21.15.62:0/710946334 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0068108470 msgr2=0x7f0068198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:35:50.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:50.970+0000 7f0066a00700 1 --2- 172.21.15.62:0/710946334 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0068108470 0x7f0068198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:35:50.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:50.970+0000 7f0067400700 1 --2- 172.21.15.62:0/710946334 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0068108470 0x7f0068198730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:35:50.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:50.970+0000 7f0067400700 1 -- 172.21.15.62:0/710946334 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00580095f0 con 0x7f0068102470 2024-04-20T21:35:50.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:50.970+0000 7f0067400700 1 --2- 172.21.15.62:0/710946334 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0068102470 0x7f00681981f0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f0058006010 tx=0x7f005800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:35:50.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:50.970+0000 7f005fe00700 1 -- 172.21.15.62:0/710946334 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f005801d070 con 0x7f0068102470 2024-04-20T21:35:50.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:50.970+0000 7f005fe00700 1 -- 172.21.15.62:0/710946334 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f005800f460 con 0x7f0068102470 2024-04-20T21:35:50.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:50.970+0000 7f005fe00700 1 -- 172.21.15.62:0/710946334 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0058017630 con 0x7f0068102470 2024-04-20T21:35:50.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:50.970+0000 7f006e200700 1 -- 172.21.15.62:0/710946334 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f006819ce20 con 0x7f0068102470 2024-04-20T21:35:50.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:50.971+0000 7f006e200700 1 -- 172.21.15.62:0/710946334 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f006819d2c0 con 0x7f0068102470 2024-04-20T21:35:50.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:50.972+0000 7f006e200700 1 -- 172.21.15.62:0/710946334 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f006810a9e0 con 0x7f0068102470 2024-04-20T21:35:50.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:50.973+0000 7f005fe00700 1 -- 172.21.15.62:0/710946334 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 31) v1 ==== 92598+0+0 (secure 0 0 0) 0x7f005800f5d0 con 0x7f0068102470 2024-04-20T21:35:50.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:50.974+0000 7f005fe00700 1 --2- 172.21.15.62:0/710946334 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f004c06e940 0x7f004c070df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:35:50.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:50.974+0000 7f005fe00700 1 -- 172.21.15.62:0/710946334 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f005808fd70 con 0x7f0068102470 2024-04-20T21:35:50.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:50.974+0000 7f0066a00700 1 --2- 172.21.15.62:0/710946334 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f004c06e940 0x7f004c070df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:35:50.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:50.975+0000 7f0066a00700 1 --2- 172.21.15.62:0/710946334 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f004c06e940 0x7f004c070df0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f0054005950 tx=0x7f00540058e0 comp rx=0 tx=0).ready entity=mgr.14658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:35:50.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:50.979+0000 7f005fe00700 1 -- 172.21.15.62:0/710946334 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187527 (secure 0 0 0) 0x7f0058059030 con 0x7f0068102470 2024-04-20T21:35:51.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:51.224+0000 7f006e200700 1 -- 172.21.15.62:0/710946334 --> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f0068199690 con 0x7f004c06e940 2024-04-20T21:35:51.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:51.242+0000 7f005fe00700 1 -- 172.21.15.62:0/710946334 <== mgr.14658 v2:172.21.15.62:6800/1585760929 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7f00540097d0 con 0x7f004c06e940 2024-04-20T21:35:51.243 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T21:35:51.243 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (100s) 53s ago 15m 13.8M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T21:35:51.243 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (15m) 53s ago 15m 8204k - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T21:35:51.243 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (14m) 2m ago 14m 7876k - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T21:35:51.243 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (15m) 53s ago 15m 6656k - 18.2.0 dc2bc1663786 36e10ef5be39 2024-04-20T21:35:51.243 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (14m) 2m ago 14m 6665k - 18.2.0 dc2bc1663786 4101f9953e3d 2024-04-20T21:35:51.243 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (63s) 53s ago 15m 49.1M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T21:35:51.243 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (8m) 53s ago 8m 65.7M - 18.2.0 dc2bc1663786 5e6460aa6159 2024-04-20T21:35:51.243 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (8m) 53s ago 8m 132M - 18.2.0 dc2bc1663786 e65f2267544e 2024-04-20T21:35:51.243 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (8m) 2m ago 8m 151M - 18.2.0 dc2bc1663786 fb2a8a223db8 2024-04-20T21:35:51.243 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (8m) 2m ago 8m 114M - 18.2.0 dc2bc1663786 f2eec2ee8951 2024-04-20T21:35:51.243 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (4m) 53s ago 17m 510M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T21:35:51.244 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (3m) 2m ago 14m 448M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T21:35:51.244 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (17m) 53s ago 17m 55.0M 2048M 18.2.0 dc2bc1663786 bd6efbd5c380 2024-04-20T21:35:51.244 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (14m) 2m ago 14m 45.7M 2048M 18.2.0 dc2bc1663786 3735c8e5db9a 2024-04-20T21:35:51.244 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (2m) 53s ago 15m 7386k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T21:35:51.244 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (2m) 2m ago 14m 3003k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T21:35:51.244 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (13m) 53s ago 13m 216M 1216M 18.2.0 dc2bc1663786 3f315c3afc1f 2024-04-20T21:35:51.244 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (12m) 53s ago 12m 210M 1216M 18.2.0 dc2bc1663786 44e4d07416ad 2024-04-20T21:35:51.244 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (11m) 53s ago 11m 171M 1216M 18.2.0 dc2bc1663786 31c5d9586e98 2024-04-20T21:35:51.244 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (11m) 2m ago 11m 211M 2240M 18.2.0 dc2bc1663786 611c1e242693 2024-04-20T21:35:51.244 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (10m) 2m ago 10m 190M 2240M 18.2.0 dc2bc1663786 6c8fce91d45d 2024-04-20T21:35:51.244 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (10m) 2m ago 10m 189M 2240M 18.2.0 dc2bc1663786 8e4574bae8db 2024-04-20T21:35:51.244 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (2m) 53s ago 14m 61.7M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T21:35:51.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:51.245+0000 7f006e200700 1 -- 172.21.15.62:0/710946334 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f004c06e940 msgr2=0x7f004c070df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:35:51.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:51.245+0000 7f006e200700 1 --2- 172.21.15.62:0/710946334 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f004c06e940 0x7f004c070df0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f0054005950 tx=0x7f00540058e0 comp rx=0 tx=0).stop 2024-04-20T21:35:51.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:51.246+0000 7f006e200700 1 -- 172.21.15.62:0/710946334 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0068102470 msgr2=0x7f00681981f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:35:51.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:51.246+0000 7f006e200700 1 --2- 172.21.15.62:0/710946334 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0068102470 0x7f00681981f0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f0058006010 tx=0x7f005800be20 comp rx=0 tx=0).stop 2024-04-20T21:35:51.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:51.246+0000 7f006e200700 1 -- 172.21.15.62:0/710946334 shutdown_connections 2024-04-20T21:35:51.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:51.246+0000 7f006e200700 1 --2- 172.21.15.62:0/710946334 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f004c06e940 0x7f004c070df0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:35:51.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:51.246+0000 7f006e200700 1 --2- 172.21.15.62:0/710946334 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0068102470 0x7f00681981f0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:35:51.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:51.246+0000 7f006e200700 1 --2- 172.21.15.62:0/710946334 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0068108470 0x7f0068198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:35:51.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:51.246+0000 7f006e200700 1 -- 172.21.15.62:0/710946334 >> 172.21.15.62:0/710946334 conn(0x7f00680fdf90 msgr2=0x7f00681002c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:35:51.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:51.246+0000 7f006e200700 1 -- 172.21.15.62:0/710946334 shutdown_connections 2024-04-20T21:35:51.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:51.246+0000 7f006e200700 1 -- 172.21.15.62:0/710946334 wait complete. 2024-04-20T21:35:51.789 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:51 smithi189 ceph-mon[21562]: pgmap v130: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:35:51.789 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:51 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:35:52.017 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:51 smithi062 ceph-mon[16221]: pgmap v130: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:35:52.017 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:51 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:35:52.081 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-04-20T21:35:52.081 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi062.front.sepia.ceph.com 2024-04-20T21:35:52.082 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -e sha1=a8c84f5997950fd7d008204839253291d52e6eda -- bash -c 'ceph config set mgr mgr/orchestrator/fail_fs true' 2024-04-20T21:35:52.500 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T21:35:52.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:52 smithi189 ceph-mon[21562]: from='client.14852 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:35:52.995 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:52 smithi062 ceph-mon[16221]: from='client.14852 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:35:53.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:53 smithi062 ceph-mon[16221]: pgmap v131: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:35:53.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:53 smithi189 ceph-mon[21562]: pgmap v131: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:35:54.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:54.830+0000 7f4a68a00700 1 -- 172.21.15.62:0/1715879981 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4a64102470 msgr2=0x7f4a641028e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:35:54.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:54.830+0000 7f4a68a00700 1 --2- 172.21.15.62:0/1715879981 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4a64102470 0x7f4a641028e0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f4a50009910 tx=0x7f4a50009c20 comp rx=0 tx=0).stop 2024-04-20T21:35:54.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:54.830+0000 7f4a68a00700 1 -- 172.21.15.62:0/1715879981 shutdown_connections 2024-04-20T21:35:54.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:54.830+0000 7f4a68a00700 1 --2- 172.21.15.62:0/1715879981 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4a64102470 0x7f4a641028e0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:35:54.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:54.830+0000 7f4a68a00700 1 --2- 172.21.15.62:0/1715879981 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4a64108470 0x7f4a64108840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:35:54.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:54.830+0000 7f4a68a00700 1 -- 172.21.15.62:0/1715879981 >> 172.21.15.62:0/1715879981 conn(0x7f4a640fdf90 msgr2=0x7f4a641003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:35:54.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:54.830+0000 7f4a68a00700 1 -- 172.21.15.62:0/1715879981 shutdown_connections 2024-04-20T21:35:54.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:54.830+0000 7f4a68a00700 1 -- 172.21.15.62:0/1715879981 wait complete. 2024-04-20T21:35:54.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:54.832+0000 7f4a68a00700 1 Processor -- start 2024-04-20T21:35:54.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:54.832+0000 7f4a68a00700 1 -- start start 2024-04-20T21:35:54.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:54.833+0000 7f4a68a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4a64102470 0x7f4a641981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:35:54.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:54.833+0000 7f4a68a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4a64108470 0x7f4a64198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:35:54.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:54.833+0000 7f4a68a00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a64198d80 con 0x7f4a64108470 2024-04-20T21:35:54.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:54.833+0000 7f4a68a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a64198ec0 con 0x7f4a64102470 2024-04-20T21:35:54.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:54.833+0000 7f4a61400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4a64108470 0x7f4a64198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:35:54.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:54.834+0000 7f4a61400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4a64108470 0x7f4a64198730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34364/0 (socket says 172.21.15.62:34364) 2024-04-20T21:35:54.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:54.834+0000 7f4a61400700 1 -- 172.21.15.62:0/1341525584 learned_addr learned my addr 172.21.15.62:0/1341525584 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:35:54.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:54.834+0000 7f4a61e00700 1 --2- 172.21.15.62:0/1341525584 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4a64102470 0x7f4a641981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:35:54.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:54.834+0000 7f4a61400700 1 -- 172.21.15.62:0/1341525584 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4a64102470 msgr2=0x7f4a641981f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:35:54.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:54.834+0000 7f4a61400700 1 --2- 172.21.15.62:0/1341525584 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4a64102470 0x7f4a641981f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:35:54.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:54.834+0000 7f4a61400700 1 -- 172.21.15.62:0/1341525584 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a500095f0 con 0x7f4a64108470 2024-04-20T21:35:54.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:54.834+0000 7f4a61400700 1 --2- 172.21.15.62:0/1341525584 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4a64108470 0x7f4a64198730 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f4a50006010 tx=0x7f4a5000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:35:54.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:54.835+0000 7f4a5aa00700 1 -- 172.21.15.62:0/1341525584 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4a5001d070 con 0x7f4a64108470 2024-04-20T21:35:54.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:54.835+0000 7f4a5aa00700 1 -- 172.21.15.62:0/1341525584 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4a50004fb0 con 0x7f4a64108470 2024-04-20T21:35:54.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:54.835+0000 7f4a5aa00700 1 -- 172.21.15.62:0/1341525584 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4a50022ea0 con 0x7f4a64108470 2024-04-20T21:35:54.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:54.835+0000 7f4a68a00700 1 -- 172.21.15.62:0/1341525584 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a6419ccb0 con 0x7f4a64108470 2024-04-20T21:35:54.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:54.835+0000 7f4a68a00700 1 -- 172.21.15.62:0/1341525584 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a6419d1a0 con 0x7f4a64108470 2024-04-20T21:35:54.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:54.836+0000 7f4a68a00700 1 -- 172.21.15.62:0/1341525584 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a6410a970 con 0x7f4a64108470 2024-04-20T21:35:54.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:54.838+0000 7f4a5aa00700 1 -- 172.21.15.62:0/1341525584 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 31) v1 ==== 92598+0+0 (secure 0 0 0) 0x7f4a50022470 con 0x7f4a64108470 2024-04-20T21:35:54.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:54.839+0000 7f4a5aa00700 1 --2- 172.21.15.62:0/1341525584 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f4a4806e940 0x7f4a48070df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:35:54.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:54.839+0000 7f4a5aa00700 1 -- 172.21.15.62:0/1341525584 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f4a5008fd60 con 0x7f4a64108470 2024-04-20T21:35:54.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:54.839+0000 7f4a61e00700 1 --2- 172.21.15.62:0/1341525584 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f4a4806e940 0x7f4a48070df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:35:54.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:54.840+0000 7f4a61e00700 1 --2- 172.21.15.62:0/1341525584 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f4a4806e940 0x7f4a48070df0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4a54005ea0 tx=0x7f4a54005e30 comp rx=0 tx=0).ready entity=mgr.14658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:35:54.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:54.844+0000 7f4a5aa00700 1 -- 172.21.15.62:0/1341525584 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187527 (secure 0 0 0) 0x7f4a50059170 con 0x7f4a64108470 2024-04-20T21:35:55.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:55.078+0000 7f4a68a00700 1 -- 172.21.15.62:0/1341525584 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command([{prefix=config set, name=mgr/orchestrator/fail_fs}] v 0) v1 -- 0x7f4a6404f2a0 con 0x7f4a64108470 2024-04-20T21:35:55.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:55.089+0000 7f4a5aa00700 1 -- 172.21.15.62:0/1341525584 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/orchestrator/fail_fs}]=0 v44)=0 v44) v1 ==== 125+0+0 (secure 0 0 0) 0x7f4a500270c0 con 0x7f4a64108470 2024-04-20T21:35:55.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:55.093+0000 7f4a68a00700 1 -- 172.21.15.62:0/1341525584 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f4a4806e940 msgr2=0x7f4a48070df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:35:55.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:55.093+0000 7f4a68a00700 1 --2- 172.21.15.62:0/1341525584 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f4a4806e940 0x7f4a48070df0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4a54005ea0 tx=0x7f4a54005e30 comp rx=0 tx=0).stop 2024-04-20T21:35:55.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:55.093+0000 7f4a68a00700 1 -- 172.21.15.62:0/1341525584 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4a64108470 msgr2=0x7f4a64198730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:35:55.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:55.093+0000 7f4a68a00700 1 --2- 172.21.15.62:0/1341525584 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4a64108470 0x7f4a64198730 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f4a50006010 tx=0x7f4a5000be20 comp rx=0 tx=0).stop 2024-04-20T21:35:55.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:55.093+0000 7f4a68a00700 1 -- 172.21.15.62:0/1341525584 shutdown_connections 2024-04-20T21:35:55.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:55.093+0000 7f4a68a00700 1 --2- 172.21.15.62:0/1341525584 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f4a4806e940 0x7f4a48070df0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:35:55.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:55.093+0000 7f4a68a00700 1 --2- 172.21.15.62:0/1341525584 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4a64102470 0x7f4a641981f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:35:55.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:55.093+0000 7f4a68a00700 1 --2- 172.21.15.62:0/1341525584 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4a64108470 0x7f4a64198730 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:35:55.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:55.093+0000 7f4a68a00700 1 -- 172.21.15.62:0/1341525584 >> 172.21.15.62:0/1341525584 conn(0x7f4a640fdf90 msgr2=0x7f4a640ff470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:35:55.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:55.093+0000 7f4a68a00700 1 -- 172.21.15.62:0/1341525584 shutdown_connections 2024-04-20T21:35:55.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:55.094+0000 7f4a68a00700 1 -- 172.21.15.62:0/1341525584 wait complete. 2024-04-20T21:35:55.850 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-04-20T21:35:55.850 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi062.front.sepia.ceph.com 2024-04-20T21:35:55.850 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -e sha1=a8c84f5997950fd7d008204839253291d52e6eda -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-04-20T21:35:55.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:55 smithi189 ceph-mon[21562]: pgmap v132: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:35:55.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:55 smithi189 ceph-mon[21562]: from='client.? 172.21.15.62:0/1341525584' entity='client.admin' 2024-04-20T21:35:55.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:55 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:35:56.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:55 smithi062 ceph-mon[16221]: pgmap v132: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:35:56.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:55 smithi062 ceph-mon[16221]: from='client.? 172.21.15.62:0/1341525584' entity='client.admin' 2024-04-20T21:35:56.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:55 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:35:56.284 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T21:35:56.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:56 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:35:56.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:56 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T21:35:56.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:56 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:35:57.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:56 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:35:57.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:56 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T21:35:57.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:56 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:35:57.854 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:57 smithi062 ceph-mon[16221]: pgmap v133: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:35:58.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:57 smithi189 ceph-mon[21562]: pgmap v133: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:35:58.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:58.510+0000 7fa20e200700 1 -- 172.21.15.62:0/3668525171 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa2080ff9e0 msgr2=0x7fa2080ffe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:35:58.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:58.510+0000 7fa20e200700 1 --2- 172.21.15.62:0/3668525171 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa2080ff9e0 0x7fa2080ffe50 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fa1f4009910 tx=0x7fa1f4009c20 comp rx=0 tx=0).stop 2024-04-20T21:35:58.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:58.510+0000 7fa20e200700 1 -- 172.21.15.62:0/3668525171 shutdown_connections 2024-04-20T21:35:58.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:58.510+0000 7fa20e200700 1 --2- 172.21.15.62:0/3668525171 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa2080ff9e0 0x7fa2080ffe50 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:35:58.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:58.510+0000 7fa20e200700 1 --2- 172.21.15.62:0/3668525171 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa208102e30 0x7fa208103200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:35:58.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:58.510+0000 7fa20e200700 1 -- 172.21.15.62:0/3668525171 >> 172.21.15.62:0/3668525171 conn(0x7fa208074fa0 msgr2=0x7fa2080753a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:35:58.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:58.510+0000 7fa20e200700 1 -- 172.21.15.62:0/3668525171 shutdown_connections 2024-04-20T21:35:58.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:58.510+0000 7fa20e200700 1 -- 172.21.15.62:0/3668525171 wait complete. 2024-04-20T21:35:58.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:58.512+0000 7fa20e200700 1 Processor -- start 2024-04-20T21:35:58.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:58.512+0000 7fa20e200700 1 -- start start 2024-04-20T21:35:58.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:58.513+0000 7fa20e200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2080ff9e0 0x7fa208197d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:35:58.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:58.513+0000 7fa20e200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa208102e30 0x7fa2081982a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:35:58.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:58.513+0000 7fa20e200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa208198980 con 0x7fa208102e30 2024-04-20T21:35:58.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:58.513+0000 7fa20e200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa20819c710 con 0x7fa2080ff9e0 2024-04-20T21:35:58.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:58.513+0000 7fa207e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa208102e30 0x7fa2081982a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:35:58.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:58.513+0000 7fa207e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa208102e30 0x7fa2081982a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34386/0 (socket says 172.21.15.62:34386) 2024-04-20T21:35:58.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:58.513+0000 7fa207e00700 1 -- 172.21.15.62:0/3751182564 learned_addr learned my addr 172.21.15.62:0/3751182564 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:35:58.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:58.513+0000 7fa20ce00700 1 --2- 172.21.15.62:0/3751182564 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2080ff9e0 0x7fa208197d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:35:58.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:58.514+0000 7fa207e00700 1 -- 172.21.15.62:0/3751182564 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2080ff9e0 msgr2=0x7fa208197d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:35:58.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:58.514+0000 7fa207e00700 1 --2- 172.21.15.62:0/3751182564 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2080ff9e0 0x7fa208197d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:35:58.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:58.514+0000 7fa207e00700 1 -- 172.21.15.62:0/3751182564 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1f40095f0 con 0x7fa208102e30 2024-04-20T21:35:58.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:58.514+0000 7fa207e00700 1 --2- 172.21.15.62:0/3751182564 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa208102e30 0x7fa2081982a0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fa1f40092e0 tx=0x7fa1f400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:35:58.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:58.514+0000 7fa205600700 1 -- 172.21.15.62:0/3751182564 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa1f401d070 con 0x7fa208102e30 2024-04-20T21:35:58.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:58.514+0000 7fa205600700 1 -- 172.21.15.62:0/3751182564 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa1f4005e00 con 0x7fa208102e30 2024-04-20T21:35:58.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:58.514+0000 7fa20e200700 1 -- 172.21.15.62:0/3751182564 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa20819c990 con 0x7fa208102e30 2024-04-20T21:35:58.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:58.514+0000 7fa205600700 1 -- 172.21.15.62:0/3751182564 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa1f4017400 con 0x7fa208102e30 2024-04-20T21:35:58.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:58.515+0000 7fa20e200700 1 -- 172.21.15.62:0/3751182564 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa20819cf00 con 0x7fa208102e30 2024-04-20T21:35:58.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:58.516+0000 7fa20e200700 1 -- 172.21.15.62:0/3751182564 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa208062380 con 0x7fa208102e30 2024-04-20T21:35:58.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:58.517+0000 7fa205600700 1 -- 172.21.15.62:0/3751182564 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 31) v1 ==== 92598+0+0 (secure 0 0 0) 0x7fa1f400f460 con 0x7fa208102e30 2024-04-20T21:35:58.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:58.518+0000 7fa205600700 1 --2- 172.21.15.62:0/3751182564 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7fa1f006e940 0x7fa1f0070df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:35:58.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:58.518+0000 7fa205600700 1 -- 172.21.15.62:0/3751182564 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6070+0+0 (secure 0 0 0) 0x7fa1f408fdf0 con 0x7fa208102e30 2024-04-20T21:35:58.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:58.519+0000 7fa20ce00700 1 --2- 172.21.15.62:0/3751182564 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7fa1f006e940 0x7fa1f0070df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:35:58.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:58.519+0000 7fa20ce00700 1 --2- 172.21.15.62:0/3751182564 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7fa1f006e940 0x7fa1f0070df0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa1fc005950 tx=0x7fa1fc0058e0 comp rx=0 tx=0).ready entity=mgr.14658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:35:58.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:58.524+0000 7fa205600700 1 -- 172.21.15.62:0/3751182564 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187527 (secure 0 0 0) 0x7fa1f40590d0 con 0x7fa208102e30 2024-04-20T21:35:58.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:58.775+0000 7fa20e200700 1 -- 172.21.15.62:0/3751182564 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}] v 0) v1 -- 0x7fa2081990c0 con 0x7fa208102e30 2024-04-20T21:35:58.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:58.776+0000 7fa205600700 1 -- 172.21.15.62:0/3751182564 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}]=0 v44)=0 v44) v1 ==== 155+0+0 (secure 0 0 0) 0x7fa1f405c790 con 0x7fa208102e30 2024-04-20T21:35:58.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:58.779+0000 7fa20e200700 1 -- 172.21.15.62:0/3751182564 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7fa1f006e940 msgr2=0x7fa1f0070df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:35:58.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:58.779+0000 7fa20e200700 1 --2- 172.21.15.62:0/3751182564 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7fa1f006e940 0x7fa1f0070df0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa1fc005950 tx=0x7fa1fc0058e0 comp rx=0 tx=0).stop 2024-04-20T21:35:58.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:58.779+0000 7fa20e200700 1 -- 172.21.15.62:0/3751182564 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa208102e30 msgr2=0x7fa2081982a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:35:58.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:58.779+0000 7fa20e200700 1 --2- 172.21.15.62:0/3751182564 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa208102e30 0x7fa2081982a0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fa1f40092e0 tx=0x7fa1f400be20 comp rx=0 tx=0).stop 2024-04-20T21:35:58.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:58.779+0000 7fa20e200700 1 -- 172.21.15.62:0/3751182564 shutdown_connections 2024-04-20T21:35:58.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:58.779+0000 7fa20e200700 1 --2- 172.21.15.62:0/3751182564 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7fa1f006e940 0x7fa1f0070df0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:35:58.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:58.779+0000 7fa20e200700 1 --2- 172.21.15.62:0/3751182564 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2080ff9e0 0x7fa208197d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:35:58.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:58.779+0000 7fa20e200700 1 --2- 172.21.15.62:0/3751182564 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa208102e30 0x7fa2081982a0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:35:58.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:58.779+0000 7fa20e200700 1 -- 172.21.15.62:0/3751182564 >> 172.21.15.62:0/3751182564 conn(0x7fa208074fa0 msgr2=0x7fa2081095f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:35:58.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:58.779+0000 7fa20e200700 1 -- 172.21.15.62:0/3751182564 shutdown_connections 2024-04-20T21:35:58.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:35:58.779+0000 7fa20e200700 1 -- 172.21.15.62:0/3751182564 wait complete. 2024-04-20T21:35:59.837 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -e sha1=a8c84f5997950fd7d008204839253291d52e6eda -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-04-20T21:36:00.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:35:59 smithi062 ceph-mon[16221]: pgmap v134: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:36:00.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:35:59 smithi189 ceph-mon[21562]: pgmap v134: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:36:00.272 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T21:36:02.027 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:01 smithi062 ceph-mon[16221]: pgmap v135: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:36:02.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:01 smithi189 ceph-mon[21562]: pgmap v135: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:36:02.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:02.605+0000 7f11fa200700 1 -- 172.21.15.62:0/4137253429 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f11f41004e0 msgr2=0x7f11f41008b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:36:02.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:02.605+0000 7f11fa200700 1 --2- 172.21.15.62:0/4137253429 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f11f41004e0 0x7f11f41008b0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f11e4009910 tx=0x7f11e4009c20 comp rx=0 tx=0).stop 2024-04-20T21:36:02.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:02.605+0000 7f11fa200700 1 -- 172.21.15.62:0/4137253429 shutdown_connections 2024-04-20T21:36:02.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:02.605+0000 7f11fa200700 1 --2- 172.21.15.62:0/4137253429 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f11f4100e80 0x7f11f4104e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:02.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:02.605+0000 7f11fa200700 1 --2- 172.21.15.62:0/4137253429 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f11f41004e0 0x7f11f41008b0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:02.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:02.605+0000 7f11fa200700 1 -- 172.21.15.62:0/4137253429 >> 172.21.15.62:0/4137253429 conn(0x7f11f40fbda0 msgr2=0x7f11f40fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:36:02.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:02.605+0000 7f11fa200700 1 -- 172.21.15.62:0/4137253429 shutdown_connections 2024-04-20T21:36:02.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:02.605+0000 7f11fa200700 1 -- 172.21.15.62:0/4137253429 wait complete. 2024-04-20T21:36:02.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:02.606+0000 7f11fa200700 1 Processor -- start 2024-04-20T21:36:02.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:02.606+0000 7f11fa200700 1 -- start start 2024-04-20T21:36:02.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:02.607+0000 7f11fa200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f11f41004e0 0x7f11f4073050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:36:02.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:02.607+0000 7f11fa200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f11f4100e80 0x7f11f4073590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:36:02.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:02.607+0000 7f11fa200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11f4073ad0 con 0x7f11f4100e80 2024-04-20T21:36:02.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:02.607+0000 7f11fa200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11f4073c10 con 0x7f11f41004e0 2024-04-20T21:36:02.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:02.607+0000 7f11ebe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f11f4100e80 0x7f11f4073590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:36:02.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:02.607+0000 7f11ebe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f11f4100e80 0x7f11f4073590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:39542/0 (socket says 172.21.15.62:39542) 2024-04-20T21:36:02.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:02.607+0000 7f11ebe00700 1 -- 172.21.15.62:0/332402896 learned_addr learned my addr 172.21.15.62:0/332402896 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:36:02.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:02.608+0000 7f11f3400700 1 --2- 172.21.15.62:0/332402896 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f11f41004e0 0x7f11f4073050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:36:02.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:02.608+0000 7f11ebe00700 1 -- 172.21.15.62:0/332402896 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f11f41004e0 msgr2=0x7f11f4073050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:36:02.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:02.608+0000 7f11ebe00700 1 --2- 172.21.15.62:0/332402896 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f11f41004e0 0x7f11f4073050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:02.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:02.608+0000 7f11ebe00700 1 -- 172.21.15.62:0/332402896 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11e40095f0 con 0x7f11f4100e80 2024-04-20T21:36:02.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:02.608+0000 7f11ebe00700 1 --2- 172.21.15.62:0/332402896 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f11f4100e80 0x7f11f4073590 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f11dc00c8f0 tx=0x7f11dc00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:36:02.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:02.609+0000 7f11f0c00700 1 -- 172.21.15.62:0/332402896 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f11dc007a50 con 0x7f11f4100e80 2024-04-20T21:36:02.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:02.609+0000 7f11f0c00700 1 -- 172.21.15.62:0/332402896 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f11dc007bb0 con 0x7f11f4100e80 2024-04-20T21:36:02.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:02.609+0000 7f11f0c00700 1 -- 172.21.15.62:0/332402896 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f11dc018630 con 0x7f11f4100e80 2024-04-20T21:36:02.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:02.609+0000 7f11fa200700 1 -- 172.21.15.62:0/332402896 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11f41a20c0 con 0x7f11f4100e80 2024-04-20T21:36:02.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:02.609+0000 7f11fa200700 1 -- 172.21.15.62:0/332402896 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11f41a2470 con 0x7f11f4100e80 2024-04-20T21:36:02.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:02.610+0000 7f11fa200700 1 -- 172.21.15.62:0/332402896 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11f41087f0 con 0x7f11f4100e80 2024-04-20T21:36:02.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:02.611+0000 7f11f0c00700 1 -- 172.21.15.62:0/332402896 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 31) v1 ==== 92598+0+0 (secure 0 0 0) 0x7f11dc007d20 con 0x7f11f4100e80 2024-04-20T21:36:02.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:02.612+0000 7f11f0c00700 1 --2- 172.21.15.62:0/332402896 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f11d806e940 0x7f11d8070df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:36:02.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:02.612+0000 7f11f0c00700 1 -- 172.21.15.62:0/332402896 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f11dc08e930 con 0x7f11f4100e80 2024-04-20T21:36:02.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:02.613+0000 7f11f3400700 1 --2- 172.21.15.62:0/332402896 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f11d806e940 0x7f11d8070df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:36:02.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:02.613+0000 7f11f3400700 1 --2- 172.21.15.62:0/332402896 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f11d806e940 0x7f11d8070df0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f11e400b5c0 tx=0x7f11e401a040 comp rx=0 tx=0).ready entity=mgr.14658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:36:02.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:02.618+0000 7f11f0c00700 1 -- 172.21.15.62:0/332402896 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187527 (secure 0 0 0) 0x7f11dc057b70 con 0x7f11f4100e80 2024-04-20T21:36:02.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:02.856+0000 7f11fa200700 1 -- 172.21.15.62:0/332402896 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}] v 0) v1 -- 0x7f11f404f2a0 con 0x7f11f4100e80 2024-04-20T21:36:02.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:02.857+0000 7f11f0c00700 1 -- 172.21.15.62:0/332402896 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}]=0 v44)=0 v44) v1 ==== 163+0+0 (secure 0 0 0) 0x7f11dc05b230 con 0x7f11f4100e80 2024-04-20T21:36:02.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:02.860+0000 7f11fa200700 1 -- 172.21.15.62:0/332402896 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f11d806e940 msgr2=0x7f11d8070df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:36:02.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:02.860+0000 7f11fa200700 1 --2- 172.21.15.62:0/332402896 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f11d806e940 0x7f11d8070df0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f11e400b5c0 tx=0x7f11e401a040 comp rx=0 tx=0).stop 2024-04-20T21:36:02.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:02.860+0000 7f11fa200700 1 -- 172.21.15.62:0/332402896 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f11f4100e80 msgr2=0x7f11f4073590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:36:02.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:02.860+0000 7f11fa200700 1 --2- 172.21.15.62:0/332402896 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f11f4100e80 0x7f11f4073590 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f11dc00c8f0 tx=0x7f11dc00ccb0 comp rx=0 tx=0).stop 2024-04-20T21:36:02.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:02.860+0000 7f11fa200700 1 -- 172.21.15.62:0/332402896 shutdown_connections 2024-04-20T21:36:02.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:02.860+0000 7f11fa200700 1 --2- 172.21.15.62:0/332402896 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f11d806e940 0x7f11d8070df0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:02.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:02.860+0000 7f11fa200700 1 --2- 172.21.15.62:0/332402896 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f11f41004e0 0x7f11f4073050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:02.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:02.860+0000 7f11fa200700 1 --2- 172.21.15.62:0/332402896 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f11f4100e80 0x7f11f4073590 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:02.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:02.860+0000 7f11fa200700 1 -- 172.21.15.62:0/332402896 >> 172.21.15.62:0/332402896 conn(0x7f11f40fbda0 msgr2=0x7f11f40fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:36:02.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:02.861+0000 7f11fa200700 1 -- 172.21.15.62:0/332402896 shutdown_connections 2024-04-20T21:36:02.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:02.861+0000 7f11fa200700 1 -- 172.21.15.62:0/332402896 wait complete. 2024-04-20T21:36:03.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:03 smithi062 ceph-mon[16221]: pgmap v136: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:36:03.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:03 smithi189 ceph-mon[21562]: pgmap v136: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:36:03.831 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -e sha1=a8c84f5997950fd7d008204839253291d52e6eda -- bash -c 'ceph config set global log_to_journald false --force' 2024-04-20T21:36:04.269 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T21:36:05.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:05 smithi189 ceph-mon[21562]: pgmap v137: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:36:06.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:05 smithi062 ceph-mon[16221]: pgmap v137: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:36:06.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:06.533+0000 7fed23000700 1 -- 172.21.15.62:0/4000271772 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fed1c1004e0 msgr2=0x7fed1c1008b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:36:06.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:06.533+0000 7fed23000700 1 --2- 172.21.15.62:0/4000271772 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fed1c1004e0 0x7fed1c1008b0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fed0c009910 tx=0x7fed0c009c20 comp rx=0 tx=0).stop 2024-04-20T21:36:06.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:06.533+0000 7fed23000700 1 -- 172.21.15.62:0/4000271772 shutdown_connections 2024-04-20T21:36:06.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:06.533+0000 7fed23000700 1 --2- 172.21.15.62:0/4000271772 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fed1c100e80 0x7fed1c104e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:06.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:06.533+0000 7fed23000700 1 --2- 172.21.15.62:0/4000271772 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fed1c1004e0 0x7fed1c1008b0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:06.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:06.533+0000 7fed23000700 1 -- 172.21.15.62:0/4000271772 >> 172.21.15.62:0/4000271772 conn(0x7fed1c0fbda0 msgr2=0x7fed1c0fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:36:06.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:06.533+0000 7fed23000700 1 -- 172.21.15.62:0/4000271772 shutdown_connections 2024-04-20T21:36:06.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:06.533+0000 7fed23000700 1 -- 172.21.15.62:0/4000271772 wait complete. 2024-04-20T21:36:06.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:06.534+0000 7fed23000700 1 Processor -- start 2024-04-20T21:36:06.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:06.534+0000 7fed23000700 1 -- start start 2024-04-20T21:36:06.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:06.535+0000 7fed23000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fed1c1004e0 0x7fed1c195ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:36:06.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:06.535+0000 7fed23000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fed1c100e80 0x7fed1c196430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:36:06.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:06.535+0000 7fed23000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed1c196a80 con 0x7fed1c1004e0 2024-04-20T21:36:06.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:06.535+0000 7fed23000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed1c196bc0 con 0x7fed1c100e80 2024-04-20T21:36:06.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:06.535+0000 7fed1be00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fed1c1004e0 0x7fed1c195ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:36:06.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:06.535+0000 7fed1be00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fed1c1004e0 0x7fed1c195ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:39560/0 (socket says 172.21.15.62:39560) 2024-04-20T21:36:06.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:06.535+0000 7fed1be00700 1 -- 172.21.15.62:0/129002240 learned_addr learned my addr 172.21.15.62:0/129002240 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:36:06.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:06.536+0000 7fed1b400700 1 --2- 172.21.15.62:0/129002240 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fed1c100e80 0x7fed1c196430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:36:06.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:06.536+0000 7fed1be00700 1 -- 172.21.15.62:0/129002240 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fed1c100e80 msgr2=0x7fed1c196430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:36:06.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:06.536+0000 7fed1be00700 1 --2- 172.21.15.62:0/129002240 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fed1c100e80 0x7fed1c196430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:06.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:06.536+0000 7fed1be00700 1 -- 172.21.15.62:0/129002240 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed0c0095f0 con 0x7fed1c1004e0 2024-04-20T21:36:06.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:06.536+0000 7fed1b400700 1 --2- 172.21.15.62:0/129002240 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fed1c100e80 0x7fed1c196430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T21:36:06.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:06.536+0000 7fed1be00700 1 --2- 172.21.15.62:0/129002240 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fed1c1004e0 0x7fed1c195ef0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fed0c006010 tx=0x7fed0c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:36:06.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:06.536+0000 7fed18c00700 1 -- 172.21.15.62:0/129002240 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fed0c01d070 con 0x7fed1c1004e0 2024-04-20T21:36:06.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:06.536+0000 7fed18c00700 1 -- 172.21.15.62:0/129002240 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fed0c004fb0 con 0x7fed1c1004e0 2024-04-20T21:36:06.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:06.536+0000 7fed23000700 1 -- 172.21.15.62:0/129002240 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed1c19a9b0 con 0x7fed1c1004e0 2024-04-20T21:36:06.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:06.536+0000 7fed18c00700 1 -- 172.21.15.62:0/129002240 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fed0c022ea0 con 0x7fed1c1004e0 2024-04-20T21:36:06.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:06.537+0000 7fed23000700 1 -- 172.21.15.62:0/129002240 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed1c19aea0 con 0x7fed1c1004e0 2024-04-20T21:36:06.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:06.538+0000 7fed23000700 1 -- 172.21.15.62:0/129002240 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed1c1087f0 con 0x7fed1c1004e0 2024-04-20T21:36:06.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:06.539+0000 7fed18c00700 1 -- 172.21.15.62:0/129002240 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 31) v1 ==== 92598+0+0 (secure 0 0 0) 0x7fed0c022470 con 0x7fed1c1004e0 2024-04-20T21:36:06.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:06.541+0000 7fed18c00700 1 --2- 172.21.15.62:0/129002240 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7fed0006e940 0x7fed00070df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:36:06.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:06.541+0000 7fed18c00700 1 -- 172.21.15.62:0/129002240 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6070+0+0 (secure 0 0 0) 0x7fed0c08fd60 con 0x7fed1c1004e0 2024-04-20T21:36:06.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:06.541+0000 7fed1b400700 1 --2- 172.21.15.62:0/129002240 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7fed0006e940 0x7fed00070df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:36:06.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:06.542+0000 7fed1b400700 1 --2- 172.21.15.62:0/129002240 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7fed0006e940 0x7fed00070df0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fed08005fd0 tx=0x7fed08005e20 comp rx=0 tx=0).ready entity=mgr.14658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:36:06.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:06.546+0000 7fed18c00700 1 -- 172.21.15.62:0/129002240 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187527 (secure 0 0 0) 0x7fed0c059170 con 0x7fed1c1004e0 2024-04-20T21:36:06.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:06.784+0000 7fed23000700 1 -- 172.21.15.62:0/129002240 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command([{prefix=config set, name=log_to_journald}] v 0) v1 -- 0x7fed1c04f2a0 con 0x7fed1c1004e0 2024-04-20T21:36:06.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:06.784+0000 7fed18c00700 1 -- 172.21.15.62:0/129002240 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{prefix=config set, name=log_to_journald}]=0 v44)=0 v44) v1 ==== 135+0+0 (secure 0 0 0) 0x7fed0c0270c0 con 0x7fed1c1004e0 2024-04-20T21:36:06.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:06.787+0000 7fed23000700 1 -- 172.21.15.62:0/129002240 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7fed0006e940 msgr2=0x7fed00070df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:36:06.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:06.787+0000 7fed23000700 1 --2- 172.21.15.62:0/129002240 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7fed0006e940 0x7fed00070df0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fed08005fd0 tx=0x7fed08005e20 comp rx=0 tx=0).stop 2024-04-20T21:36:06.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:06.787+0000 7fed23000700 1 -- 172.21.15.62:0/129002240 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fed1c1004e0 msgr2=0x7fed1c195ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:36:06.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:06.787+0000 7fed23000700 1 --2- 172.21.15.62:0/129002240 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fed1c1004e0 0x7fed1c195ef0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fed0c006010 tx=0x7fed0c00be20 comp rx=0 tx=0).stop 2024-04-20T21:36:06.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:06.788+0000 7fed23000700 1 -- 172.21.15.62:0/129002240 shutdown_connections 2024-04-20T21:36:06.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:06.788+0000 7fed23000700 1 --2- 172.21.15.62:0/129002240 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7fed0006e940 0x7fed00070df0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:06.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:06.788+0000 7fed23000700 1 --2- 172.21.15.62:0/129002240 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fed1c1004e0 0x7fed1c195ef0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:06.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:06.788+0000 7fed23000700 1 --2- 172.21.15.62:0/129002240 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fed1c100e80 0x7fed1c196430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:06.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:06.788+0000 7fed23000700 1 -- 172.21.15.62:0/129002240 >> 172.21.15.62:0/129002240 conn(0x7fed1c0fbda0 msgr2=0x7fed1c0fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:36:06.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:06.788+0000 7fed23000700 1 -- 172.21.15.62:0/129002240 shutdown_connections 2024-04-20T21:36:06.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:06.788+0000 7fed23000700 1 -- 172.21.15.62:0/129002240 wait complete. 2024-04-20T21:36:06.974 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:06 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:36:07.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:06 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:36:07.502 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -e sha1=a8c84f5997950fd7d008204839253291d52e6eda -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-04-20T21:36:07.946 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T21:36:07.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:07 smithi189 ceph-mon[21562]: pgmap v138: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:36:08.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:07 smithi062 ceph-mon[16221]: pgmap v138: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:36:09.884 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:09 smithi062 ceph-mon[16221]: pgmap v139: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:36:09.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:09 smithi189 ceph-mon[21562]: pgmap v139: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:36:10.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:10.433+0000 7f5c1c400700 1 -- 172.21.15.62:0/4130286273 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c14075b90 msgr2=0x7f5c14110ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:36:10.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:10.433+0000 7f5c1c400700 1 --2- 172.21.15.62:0/4130286273 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c14075b90 0x7f5c14110ba0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f5c00009910 tx=0x7f5c00009c20 comp rx=0 tx=0).stop 2024-04-20T21:36:10.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:10.434+0000 7f5c1c400700 1 -- 172.21.15.62:0/4130286273 shutdown_connections 2024-04-20T21:36:10.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:10.434+0000 7f5c1c400700 1 --2- 172.21.15.62:0/4130286273 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c14075b90 0x7f5c14110ba0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:10.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:10.434+0000 7f5c1c400700 1 --2- 172.21.15.62:0/4130286273 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c14075280 0x7f5c14075650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:10.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:10.434+0000 7f5c1c400700 1 -- 172.21.15.62:0/4130286273 >> 172.21.15.62:0/4130286273 conn(0x7f5c140fdf40 msgr2=0x7f5c14100350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:36:10.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:10.434+0000 7f5c1c400700 1 -- 172.21.15.62:0/4130286273 shutdown_connections 2024-04-20T21:36:10.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:10.434+0000 7f5c1c400700 1 -- 172.21.15.62:0/4130286273 wait complete. 2024-04-20T21:36:10.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:10.435+0000 7f5c1c400700 1 Processor -- start 2024-04-20T21:36:10.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:10.435+0000 7f5c1c400700 1 -- start start 2024-04-20T21:36:10.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:10.435+0000 7f5c1c400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c14075280 0x7f5c1419c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:36:10.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:10.435+0000 7f5c1c400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c14075b90 0x7f5c1419ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:36:10.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:10.435+0000 7f5c1c400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c1419d150 con 0x7f5c14075b90 2024-04-20T21:36:10.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:10.435+0000 7f5c1c400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c141a0ee0 con 0x7f5c14075280 2024-04-20T21:36:10.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:10.436+0000 7f5c19000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c14075b90 0x7f5c1419ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:36:10.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:10.436+0000 7f5c19000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c14075b90 0x7f5c1419ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:39566/0 (socket says 172.21.15.62:39566) 2024-04-20T21:36:10.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:10.436+0000 7f5c19000700 1 -- 172.21.15.62:0/1854005915 learned_addr learned my addr 172.21.15.62:0/1854005915 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:36:10.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:10.436+0000 7f5c19a00700 1 --2- 172.21.15.62:0/1854005915 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c14075280 0x7f5c1419c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:36:10.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:10.436+0000 7f5c19000700 1 -- 172.21.15.62:0/1854005915 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c14075280 msgr2=0x7f5c1419c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:36:10.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:10.436+0000 7f5c19000700 1 --2- 172.21.15.62:0/1854005915 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c14075280 0x7f5c1419c530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:10.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:10.436+0000 7f5c19000700 1 -- 172.21.15.62:0/1854005915 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c000095f0 con 0x7f5c14075b90 2024-04-20T21:36:10.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:10.437+0000 7f5c19000700 1 --2- 172.21.15.62:0/1854005915 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c14075b90 0x7f5c1419ca70 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f5c00000c00 tx=0x7f5c000056a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:36:10.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:10.437+0000 7f5c12000700 1 -- 172.21.15.62:0/1854005915 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5c0001d070 con 0x7f5c14075b90 2024-04-20T21:36:10.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:10.437+0000 7f5c1c400700 1 -- 172.21.15.62:0/1854005915 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c141a1160 con 0x7f5c14075b90 2024-04-20T21:36:10.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:10.437+0000 7f5c12000700 1 -- 172.21.15.62:0/1854005915 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5c0000f460 con 0x7f5c14075b90 2024-04-20T21:36:10.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:10.437+0000 7f5c12000700 1 -- 172.21.15.62:0/1854005915 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5c00021640 con 0x7f5c14075b90 2024-04-20T21:36:10.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:10.437+0000 7f5c1c400700 1 -- 172.21.15.62:0/1854005915 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c141a16b0 con 0x7f5c14075b90 2024-04-20T21:36:10.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:10.438+0000 7f5c1c400700 1 -- 172.21.15.62:0/1854005915 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c1410e310 con 0x7f5c14075b90 2024-04-20T21:36:10.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:10.440+0000 7f5c12000700 1 -- 172.21.15.62:0/1854005915 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 31) v1 ==== 92598+0+0 (secure 0 0 0) 0x7f5c0000fa60 con 0x7f5c14075b90 2024-04-20T21:36:10.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:10.441+0000 7f5c12000700 1 --2- 172.21.15.62:0/1854005915 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f5bfc06e940 0x7f5bfc070df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:36:10.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:10.441+0000 7f5c12000700 1 -- 172.21.15.62:0/1854005915 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f5c0008fa90 con 0x7f5c14075b90 2024-04-20T21:36:10.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:10.441+0000 7f5c19a00700 1 --2- 172.21.15.62:0/1854005915 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f5bfc06e940 0x7f5bfc070df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:36:10.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:10.442+0000 7f5c19a00700 1 --2- 172.21.15.62:0/1854005915 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f5bfc06e940 0x7f5bfc070df0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5c08005950 tx=0x7f5c080058e0 comp rx=0 tx=0).ready entity=mgr.14658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:36:10.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:10.446+0000 7f5c12000700 1 -- 172.21.15.62:0/1854005915 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187527 (secure 0 0 0) 0x7f5c00058d50 con 0x7f5c14075b90 2024-04-20T21:36:10.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:10.710+0000 7f5c1c400700 1 -- 172.21.15.62:0/1854005915 --> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] -- mgr_command(tid 0: {"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:a8c84f5997950fd7d008204839253291d52e6eda", "target": ["mon-mgr", ""]}) v1 -- 0x7f5c141a19e0 con 0x7f5bfc06e940 2024-04-20T21:36:10.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:10.718+0000 7f5c12000700 1 -- 172.21.15.62:0/1854005915 <== mgr.14658 v2:172.21.15.62:6800/1585760929 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+89 (secure 0 0 0) 0x7f5c141a19e0 con 0x7f5bfc06e940 2024-04-20T21:36:10.718 INFO:teuthology.orchestra.run.smithi062.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:a8c84f5997950fd7d008204839253291d52e6eda 2024-04-20T21:36:10.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:10.721+0000 7f5c1c400700 1 -- 172.21.15.62:0/1854005915 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f5bfc06e940 msgr2=0x7f5bfc070df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:36:10.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:10.721+0000 7f5c1c400700 1 --2- 172.21.15.62:0/1854005915 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f5bfc06e940 0x7f5bfc070df0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5c08005950 tx=0x7f5c080058e0 comp rx=0 tx=0).stop 2024-04-20T21:36:10.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:10.721+0000 7f5c1c400700 1 -- 172.21.15.62:0/1854005915 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c14075b90 msgr2=0x7f5c1419ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:36:10.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:10.721+0000 7f5c1c400700 1 --2- 172.21.15.62:0/1854005915 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c14075b90 0x7f5c1419ca70 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f5c00000c00 tx=0x7f5c000056a0 comp rx=0 tx=0).stop 2024-04-20T21:36:10.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:10.722+0000 7f5c1c400700 1 -- 172.21.15.62:0/1854005915 shutdown_connections 2024-04-20T21:36:10.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:10.722+0000 7f5c1c400700 1 --2- 172.21.15.62:0/1854005915 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f5bfc06e940 0x7f5bfc070df0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:10.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:10.722+0000 7f5c1c400700 1 --2- 172.21.15.62:0/1854005915 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c14075280 0x7f5c1419c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:10.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:10.722+0000 7f5c1c400700 1 --2- 172.21.15.62:0/1854005915 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c14075b90 0x7f5c1419ca70 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:10.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:10.722+0000 7f5c1c400700 1 -- 172.21.15.62:0/1854005915 >> 172.21.15.62:0/1854005915 conn(0x7f5c140fdf40 msgr2=0x7f5c140feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:36:10.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:10.722+0000 7f5c1c400700 1 -- 172.21.15.62:0/1854005915 shutdown_connections 2024-04-20T21:36:10.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:10.722+0000 7f5c1c400700 1 -- 172.21.15.62:0/1854005915 wait complete. 2024-04-20T21:36:11.500 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-04-20T21:36:11.501 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi062.front.sepia.ceph.com 2024-04-20T21:36:11.501 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -e sha1=a8c84f5997950fd7d008204839253291d52e6eda -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done' 2024-04-20T21:36:11.942 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T21:36:11.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:11 smithi189 ceph-mon[21562]: pgmap v140: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:36:11.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:11 smithi189 ceph-mon[21562]: from='client.14872 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:a8c84f5997950fd7d008204839253291d52e6eda", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:36:11.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:11 smithi189 ceph-mon[21562]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:a8c84f5997950fd7d008204839253291d52e6eda 2024-04-20T21:36:11.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:11 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:36:11.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:11 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:36:12.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:11 smithi062 ceph-mon[16221]: pgmap v140: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:36:12.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:11 smithi062 ceph-mon[16221]: from='client.14872 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:a8c84f5997950fd7d008204839253291d52e6eda", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:36:12.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:11 smithi062 ceph-mon[16221]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:a8c84f5997950fd7d008204839253291d52e6eda 2024-04-20T21:36:12.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:11 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:36:12.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:11 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:36:12.980 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:12 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:36:12.981 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:12 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T21:36:12.981 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:12 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:36:12.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:12 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:36:12.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:12 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T21:36:12.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:12 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:36:13.971 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:13 smithi062 ceph-mon[16221]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:a8c84f5997950fd7d008204839253291d52e6eda 2024-04-20T21:36:13.971 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:13 smithi062 ceph-mon[16221]: pgmap v141: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:36:13.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:13 smithi189 ceph-mon[21562]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:a8c84f5997950fd7d008204839253291d52e6eda 2024-04-20T21:36:13.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:13 smithi189 ceph-mon[21562]: pgmap v141: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:36:14.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:14.344+0000 7f700a400700 1 -- 172.21.15.62:0/3606230160 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7004100e60 msgr2=0x7f7004104e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:36:14.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:14.344+0000 7f700a400700 1 --2- 172.21.15.62:0/3606230160 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7004100e60 0x7f7004104e30 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f6ff0009910 tx=0x7f6ff0009c20 comp rx=0 tx=0).stop 2024-04-20T21:36:14.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:14.344+0000 7f700a400700 1 -- 172.21.15.62:0/3606230160 shutdown_connections 2024-04-20T21:36:14.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:14.344+0000 7f700a400700 1 --2- 172.21.15.62:0/3606230160 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7004100e60 0x7f7004104e30 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:14.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:14.344+0000 7f700a400700 1 --2- 172.21.15.62:0/3606230160 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70041004c0 0x7f7004100890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:14.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:14.344+0000 7f700a400700 1 -- 172.21.15.62:0/3606230160 >> 172.21.15.62:0/3606230160 conn(0x7f70040fbd80 msgr2=0x7f70040fe190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:36:14.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:14.345+0000 7f700a400700 1 -- 172.21.15.62:0/3606230160 shutdown_connections 2024-04-20T21:36:14.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:14.345+0000 7f700a400700 1 -- 172.21.15.62:0/3606230160 wait complete. 2024-04-20T21:36:14.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:14.346+0000 7f700a400700 1 Processor -- start 2024-04-20T21:36:14.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:14.346+0000 7f700a400700 1 -- start start 2024-04-20T21:36:14.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:14.346+0000 7f700a400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70041004c0 0x7f70041980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:36:14.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:14.346+0000 7f700a400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7004100e60 0x7f70041985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:36:14.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:14.346+0000 7f700a400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7004198cd0 con 0x7f7004100e60 2024-04-20T21:36:14.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:14.346+0000 7f700a400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f700419ca60 con 0x7f70041004c0 2024-04-20T21:36:14.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:14.347+0000 7f7002a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7004100e60 0x7f70041985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:36:14.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:14.347+0000 7f7002a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7004100e60 0x7f70041985f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37236/0 (socket says 172.21.15.62:37236) 2024-04-20T21:36:14.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:14.347+0000 7f7002a00700 1 -- 172.21.15.62:0/3617599522 learned_addr learned my addr 172.21.15.62:0/3617599522 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:36:14.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:14.347+0000 7f7003400700 1 --2- 172.21.15.62:0/3617599522 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70041004c0 0x7f70041980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:36:14.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:14.348+0000 7f7002a00700 1 -- 172.21.15.62:0/3617599522 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70041004c0 msgr2=0x7f70041980b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:36:14.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:14.348+0000 7f7002a00700 1 --2- 172.21.15.62:0/3617599522 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70041004c0 0x7f70041980b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:14.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:14.348+0000 7f7002a00700 1 -- 172.21.15.62:0/3617599522 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ff00095f0 con 0x7f7004100e60 2024-04-20T21:36:14.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:14.349+0000 7f7002a00700 1 --2- 172.21.15.62:0/3617599522 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7004100e60 0x7f70041985f0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f6ff000bda0 tx=0x7f6ff000bdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:36:14.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:14.349+0000 7f6ffbe00700 1 -- 172.21.15.62:0/3617599522 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6ff001d070 con 0x7f7004100e60 2024-04-20T21:36:14.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:14.349+0000 7f6ffbe00700 1 -- 172.21.15.62:0/3617599522 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6ff0005dc0 con 0x7f7004100e60 2024-04-20T21:36:14.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:14.349+0000 7f700a400700 1 -- 172.21.15.62:0/3617599522 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f700419cce0 con 0x7f7004100e60 2024-04-20T21:36:14.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:14.349+0000 7f6ffbe00700 1 -- 172.21.15.62:0/3617599522 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6ff0017400 con 0x7f7004100e60 2024-04-20T21:36:14.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:14.350+0000 7f700a400700 1 -- 172.21.15.62:0/3617599522 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f700419d1d0 con 0x7f7004100e60 2024-04-20T21:36:14.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:14.351+0000 7f700a400700 1 -- 172.21.15.62:0/3617599522 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7004104cc0 con 0x7f7004100e60 2024-04-20T21:36:14.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:14.352+0000 7f6ffbe00700 1 -- 172.21.15.62:0/3617599522 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 31) v1 ==== 92598+0+0 (secure 0 0 0) 0x7f6ff000f460 con 0x7f7004100e60 2024-04-20T21:36:14.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:14.353+0000 7f6ffbe00700 1 --2- 172.21.15.62:0/3617599522 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f6fe806e940 0x7f6fe8070df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:36:14.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:14.353+0000 7f6ffbe00700 1 -- 172.21.15.62:0/3617599522 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f6ff008fd20 con 0x7f7004100e60 2024-04-20T21:36:14.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:14.354+0000 7f7003400700 1 --2- 172.21.15.62:0/3617599522 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f6fe806e940 0x7f6fe8070df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:36:14.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:14.354+0000 7f7003400700 1 --2- 172.21.15.62:0/3617599522 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f6fe806e940 0x7f6fe8070df0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6ff4004630 tx=0x7f6ff400b3c0 comp rx=0 tx=0).ready entity=mgr.14658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:36:14.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:14.359+0000 7f6ffbe00700 1 -- 172.21.15.62:0/3617599522 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187527 (secure 0 0 0) 0x7f6ff0058f60 con 0x7f7004100e60 2024-04-20T21:36:14.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:14.638+0000 7f700a400700 1 -- 172.21.15.62:0/3617599522 --> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f700402cca0 con 0x7f6fe806e940 2024-04-20T21:36:14.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:14.640+0000 7f6ffbe00700 1 -- 172.21.15.62:0/3617599522 <== mgr.14658 v2:172.21.15.62:6800/1585760929 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+358 (secure 0 0 0) 0x7f700402cca0 con 0x7f6fe806e940 2024-04-20T21:36:14.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:14.643+0000 7f700a400700 1 -- 172.21.15.62:0/3617599522 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f6fe806e940 msgr2=0x7f6fe8070df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:36:14.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:14.643+0000 7f700a400700 1 --2- 172.21.15.62:0/3617599522 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f6fe806e940 0x7f6fe8070df0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6ff4004630 tx=0x7f6ff400b3c0 comp rx=0 tx=0).stop 2024-04-20T21:36:14.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:14.643+0000 7f700a400700 1 -- 172.21.15.62:0/3617599522 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7004100e60 msgr2=0x7f70041985f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:36:14.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:14.643+0000 7f700a400700 1 --2- 172.21.15.62:0/3617599522 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7004100e60 0x7f70041985f0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f6ff000bda0 tx=0x7f6ff000bdd0 comp rx=0 tx=0).stop 2024-04-20T21:36:14.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:14.643+0000 7f700a400700 1 -- 172.21.15.62:0/3617599522 shutdown_connections 2024-04-20T21:36:14.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:14.643+0000 7f700a400700 1 --2- 172.21.15.62:0/3617599522 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f6fe806e940 0x7f6fe8070df0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:14.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:14.643+0000 7f700a400700 1 --2- 172.21.15.62:0/3617599522 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70041004c0 0x7f70041980b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:14.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:14.643+0000 7f700a400700 1 --2- 172.21.15.62:0/3617599522 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7004100e60 0x7f70041985f0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:14.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:14.643+0000 7f700a400700 1 -- 172.21.15.62:0/3617599522 >> 172.21.15.62:0/3617599522 conn(0x7f70040fbd80 msgr2=0x7f7004101e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:36:14.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:14.644+0000 7f700a400700 1 -- 172.21.15.62:0/3617599522 shutdown_connections 2024-04-20T21:36:14.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:14.644+0000 7f700a400700 1 -- 172.21.15.62:0/3617599522 wait complete. 2024-04-20T21:36:14.659 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T21:36:14.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:14.813+0000 7f923d600700 1 -- 172.21.15.62:0/3480009761 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f922c0054b0 con 0x7f9240100e80 2024-04-20T21:36:14.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:14.813+0000 7f9245a00700 1 -- 172.21.15.62:0/3480009761 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9240100e80 msgr2=0x7f9240104e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:36:14.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:14.813+0000 7f9245a00700 1 --2- 172.21.15.62:0/3480009761 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9240100e80 0x7f9240104e50 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f922c009910 tx=0x7f922c009c20 comp rx=0 tx=0).stop 2024-04-20T21:36:14.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:14.814+0000 7f9245a00700 1 -- 172.21.15.62:0/3480009761 shutdown_connections 2024-04-20T21:36:14.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:14.814+0000 7f9245a00700 1 --2- 172.21.15.62:0/3480009761 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9240100e80 0x7f9240104e50 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:14.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:14.814+0000 7f9245a00700 1 --2- 172.21.15.62:0/3480009761 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f92401004e0 0x7f92401008b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:14.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:14.814+0000 7f9245a00700 1 -- 172.21.15.62:0/3480009761 >> 172.21.15.62:0/3480009761 conn(0x7f92400fbda0 msgr2=0x7f92400fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:36:14.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:14.814+0000 7f9245a00700 1 -- 172.21.15.62:0/3480009761 shutdown_connections 2024-04-20T21:36:14.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:14.814+0000 7f9245a00700 1 -- 172.21.15.62:0/3480009761 wait complete. 2024-04-20T21:36:14.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:14.815+0000 7f9245a00700 1 Processor -- start 2024-04-20T21:36:14.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:14.815+0000 7f9245a00700 1 -- start start 2024-04-20T21:36:14.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:14.816+0000 7f9245a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f92401004e0 0x7f9240198110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:36:14.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:14.816+0000 7f9245a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9240100e80 0x7f9240198650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:36:14.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:14.816+0000 7f9245a00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9240198d30 con 0x7f92401004e0 2024-04-20T21:36:14.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:14.816+0000 7f9245a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f924019cac0 con 0x7f9240100e80 2024-04-20T21:36:14.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:14.816+0000 7f923ea00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f92401004e0 0x7f9240198110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:36:14.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:14.816+0000 7f923ea00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f92401004e0 0x7f9240198110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37256/0 (socket says 172.21.15.62:37256) 2024-04-20T21:36:14.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:14.816+0000 7f923ea00700 1 -- 172.21.15.62:0/305257115 learned_addr learned my addr 172.21.15.62:0/305257115 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:36:14.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:14.816+0000 7f923e000700 1 --2- 172.21.15.62:0/305257115 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9240100e80 0x7f9240198650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:36:14.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:14.817+0000 7f923ea00700 1 -- 172.21.15.62:0/305257115 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9240100e80 msgr2=0x7f9240198650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:36:14.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:14.817+0000 7f923ea00700 1 --2- 172.21.15.62:0/305257115 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9240100e80 0x7f9240198650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:14.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:14.817+0000 7f923ea00700 1 -- 172.21.15.62:0/305257115 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f922c0095f0 con 0x7f92401004e0 2024-04-20T21:36:14.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:14.817+0000 7f923ea00700 1 --2- 172.21.15.62:0/305257115 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f92401004e0 0x7f9240198110 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f923000d8d0 tx=0x7f923000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:36:14.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:14.817+0000 7f9237400700 1 -- 172.21.15.62:0/305257115 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9230009940 con 0x7f92401004e0 2024-04-20T21:36:14.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:14.817+0000 7f9245a00700 1 -- 172.21.15.62:0/305257115 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f924019cda0 con 0x7f92401004e0 2024-04-20T21:36:14.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:14.817+0000 7f9237400700 1 -- 172.21.15.62:0/305257115 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9230010460 con 0x7f92401004e0 2024-04-20T21:36:14.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:14.817+0000 7f9237400700 1 -- 172.21.15.62:0/305257115 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f923001b620 con 0x7f92401004e0 2024-04-20T21:36:14.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:14.817+0000 7f9245a00700 1 -- 172.21.15.62:0/305257115 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f924019d2f0 con 0x7f92401004e0 2024-04-20T21:36:14.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:14.819+0000 7f9245a00700 1 -- 172.21.15.62:0/305257115 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92401045d0 con 0x7f92401004e0 2024-04-20T21:36:14.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:14.820+0000 7f9237400700 1 -- 172.21.15.62:0/305257115 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 31) v1 ==== 92598+0+0 (secure 0 0 0) 0x7f92300105d0 con 0x7f92401004e0 2024-04-20T21:36:14.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:14.821+0000 7f9237400700 1 --2- 172.21.15.62:0/305257115 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f922406e940 0x7f9224070df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:36:14.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:14.821+0000 7f9237400700 1 -- 172.21.15.62:0/305257115 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f923008ebd0 con 0x7f92401004e0 2024-04-20T21:36:14.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:14.821+0000 7f923e000700 1 --2- 172.21.15.62:0/305257115 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f922406e940 0x7f9224070df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:36:14.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:14.822+0000 7f923e000700 1 --2- 172.21.15.62:0/305257115 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f922406e940 0x7f9224070df0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f9240199730 tx=0x7f922c01a040 comp rx=0 tx=0).ready entity=mgr.14658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:36:14.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:14.826+0000 7f9237400700 1 -- 172.21.15.62:0/305257115 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187527 (secure 0 0 0) 0x7f9230057eb0 con 0x7f92401004e0 2024-04-20T21:36:15.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:15.087+0000 7f9245a00700 1 -- 172.21.15.62:0/305257115 --> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f924019d560 con 0x7f922406e940 2024-04-20T21:36:15.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:15.088+0000 7f9237400700 1 -- 172.21.15.62:0/305257115 <== mgr.14658 v2:172.21.15.62:6800/1585760929 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+358 (secure 0 0 0) 0x7f924019d560 con 0x7f922406e940 2024-04-20T21:36:15.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:15.091+0000 7f9245a00700 1 -- 172.21.15.62:0/305257115 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f922406e940 msgr2=0x7f9224070df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:36:15.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:15.091+0000 7f9245a00700 1 --2- 172.21.15.62:0/305257115 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f922406e940 0x7f9224070df0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f9240199730 tx=0x7f922c01a040 comp rx=0 tx=0).stop 2024-04-20T21:36:15.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:15.091+0000 7f9245a00700 1 -- 172.21.15.62:0/305257115 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f92401004e0 msgr2=0x7f9240198110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:36:15.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:15.092+0000 7f9245a00700 1 --2- 172.21.15.62:0/305257115 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f92401004e0 0x7f9240198110 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f923000d8d0 tx=0x7f923000dc90 comp rx=0 tx=0).stop 2024-04-20T21:36:15.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:15.092+0000 7f9245a00700 1 -- 172.21.15.62:0/305257115 shutdown_connections 2024-04-20T21:36:15.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:15.092+0000 7f9245a00700 1 --2- 172.21.15.62:0/305257115 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f922406e940 0x7f9224070df0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:15.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:15.092+0000 7f9245a00700 1 --2- 172.21.15.62:0/305257115 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f92401004e0 0x7f9240198110 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:15.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:15.092+0000 7f9245a00700 1 --2- 172.21.15.62:0/305257115 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9240100e80 0x7f9240198650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:15.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:15.092+0000 7f9245a00700 1 -- 172.21.15.62:0/305257115 >> 172.21.15.62:0/305257115 conn(0x7f92400fbda0 msgr2=0x7f92400fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:36:15.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:15.092+0000 7f9245a00700 1 -- 172.21.15.62:0/305257115 shutdown_connections 2024-04-20T21:36:15.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:15.092+0000 7f9245a00700 1 -- 172.21.15.62:0/305257115 wait complete. 2024-04-20T21:36:15.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:15.252+0000 7f4756c00700 1 -- 172.21.15.62:0/2958283853 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4750100e80 msgr2=0x7f4750104e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:36:15.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:15.252+0000 7f4756c00700 1 --2- 172.21.15.62:0/2958283853 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4750100e80 0x7f4750104e50 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f473c009910 tx=0x7f473c009c20 comp rx=0 tx=0).stop 2024-04-20T21:36:15.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:15.252+0000 7f4756c00700 1 -- 172.21.15.62:0/2958283853 shutdown_connections 2024-04-20T21:36:15.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:15.252+0000 7f4756c00700 1 --2- 172.21.15.62:0/2958283853 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4750100e80 0x7f4750104e50 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:15.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:15.252+0000 7f4756c00700 1 --2- 172.21.15.62:0/2958283853 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f47501004e0 0x7f47501008b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:15.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:15.252+0000 7f4756c00700 1 -- 172.21.15.62:0/2958283853 >> 172.21.15.62:0/2958283853 conn(0x7f47500fbda0 msgr2=0x7f47500fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:36:15.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:15.252+0000 7f4756c00700 1 -- 172.21.15.62:0/2958283853 shutdown_connections 2024-04-20T21:36:15.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:15.252+0000 7f4756c00700 1 -- 172.21.15.62:0/2958283853 wait complete. 2024-04-20T21:36:15.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:15.253+0000 7f4756c00700 1 Processor -- start 2024-04-20T21:36:15.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:15.253+0000 7f4756c00700 1 -- start start 2024-04-20T21:36:15.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:15.254+0000 7f4756c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f47501004e0 0x7f4750195ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:36:15.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:15.254+0000 7f4756c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4750100e80 0x7f47501963e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:36:15.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:15.254+0000 7f4756c00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4750196ac0 con 0x7f4750100e80 2024-04-20T21:36:15.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:15.254+0000 7f4756c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f475019a850 con 0x7f47501004e0 2024-04-20T21:36:15.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:15.254+0000 7f474f400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4750100e80 0x7f47501963e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:36:15.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:15.254+0000 7f474f400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4750100e80 0x7f47501963e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37268/0 (socket says 172.21.15.62:37268) 2024-04-20T21:36:15.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:15.254+0000 7f474f400700 1 -- 172.21.15.62:0/718067189 learned_addr learned my addr 172.21.15.62:0/718067189 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:36:15.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:15.254+0000 7f474fe00700 1 --2- 172.21.15.62:0/718067189 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f47501004e0 0x7f4750195ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:36:15.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:15.255+0000 7f474f400700 1 -- 172.21.15.62:0/718067189 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f47501004e0 msgr2=0x7f4750195ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:36:15.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:15.255+0000 7f474f400700 1 --2- 172.21.15.62:0/718067189 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f47501004e0 0x7f4750195ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:15.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:15.255+0000 7f474f400700 1 -- 172.21.15.62:0/718067189 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f473c0095f0 con 0x7f4750100e80 2024-04-20T21:36:15.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:15.255+0000 7f474f400700 1 --2- 172.21.15.62:0/718067189 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4750100e80 0x7f47501963e0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f473c00bda0 tx=0x7f473c00be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:36:15.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:15.255+0000 7f474cc00700 1 -- 172.21.15.62:0/718067189 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f473c01d070 con 0x7f4750100e80 2024-04-20T21:36:15.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:15.255+0000 7f4756c00700 1 -- 172.21.15.62:0/718067189 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f475019aad0 con 0x7f4750100e80 2024-04-20T21:36:15.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:15.255+0000 7f474cc00700 1 -- 172.21.15.62:0/718067189 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f473c004fb0 con 0x7f4750100e80 2024-04-20T21:36:15.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:15.255+0000 7f474cc00700 1 -- 172.21.15.62:0/718067189 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f473c00f460 con 0x7f4750100e80 2024-04-20T21:36:15.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:15.256+0000 7f4756c00700 1 -- 172.21.15.62:0/718067189 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f475019afc0 con 0x7f4750100e80 2024-04-20T21:36:15.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:15.257+0000 7f4756c00700 1 -- 172.21.15.62:0/718067189 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4750108760 con 0x7f4750100e80 2024-04-20T21:36:15.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:15.259+0000 7f474cc00700 1 -- 172.21.15.62:0/718067189 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 31) v1 ==== 92598+0+0 (secure 0 0 0) 0x7f473c022800 con 0x7f4750100e80 2024-04-20T21:36:15.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:15.260+0000 7f474cc00700 1 --2- 172.21.15.62:0/718067189 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f473406e940 0x7f4734070df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:36:15.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:15.260+0000 7f474cc00700 1 -- 172.21.15.62:0/718067189 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f473c08fe80 con 0x7f4750100e80 2024-04-20T21:36:15.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:15.261+0000 7f474fe00700 1 --2- 172.21.15.62:0/718067189 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f473406e940 0x7f4734070df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:36:15.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:15.261+0000 7f474fe00700 1 --2- 172.21.15.62:0/718067189 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f473406e940 0x7f4734070df0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f4740005950 tx=0x7f4740005eb0 comp rx=0 tx=0).ready entity=mgr.14658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:36:15.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:15.266+0000 7f474cc00700 1 -- 172.21.15.62:0/718067189 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187527 (secure 0 0 0) 0x7f473c059290 con 0x7f4750100e80 2024-04-20T21:36:15.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:15.519+0000 7f4756c00700 1 -- 172.21.15.62:0/718067189 --> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f4750197310 con 0x7f473406e940 2024-04-20T21:36:15.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:15.535+0000 7f474cc00700 1 -- 172.21.15.62:0/718067189 <== mgr.14658 v2:172.21.15.62:6800/1585760929 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f4750197310 con 0x7f473406e940 2024-04-20T21:36:15.535 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T21:36:15.535 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (2m) 77s ago 15m 13.8M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T21:36:15.535 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (16m) 77s ago 16m 8204k - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T21:36:15.535 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (14m) 3m ago 14m 7876k - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T21:36:15.535 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (16m) 77s ago 16m 6656k - 18.2.0 dc2bc1663786 36e10ef5be39 2024-04-20T21:36:15.535 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (14m) 3m ago 14m 6665k - 18.2.0 dc2bc1663786 4101f9953e3d 2024-04-20T21:36:15.535 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (88s) 77s ago 15m 49.1M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T21:36:15.535 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (8m) 77s ago 8m 65.7M - 18.2.0 dc2bc1663786 5e6460aa6159 2024-04-20T21:36:15.535 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (8m) 77s ago 8m 132M - 18.2.0 dc2bc1663786 e65f2267544e 2024-04-20T21:36:15.535 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (8m) 3m ago 8m 151M - 18.2.0 dc2bc1663786 fb2a8a223db8 2024-04-20T21:36:15.536 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (8m) 3m ago 8m 114M - 18.2.0 dc2bc1663786 f2eec2ee8951 2024-04-20T21:36:15.536 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (4m) 77s ago 17m 510M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T21:36:15.536 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (3m) 3m ago 14m 448M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T21:36:15.536 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (17m) 77s ago 17m 55.0M 2048M 18.2.0 dc2bc1663786 bd6efbd5c380 2024-04-20T21:36:15.536 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (14m) 3m ago 14m 45.7M 2048M 18.2.0 dc2bc1663786 3735c8e5db9a 2024-04-20T21:36:15.536 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (3m) 77s ago 16m 7386k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T21:36:15.536 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (3m) 3m ago 14m 3003k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T21:36:15.536 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (13m) 77s ago 13m 216M 1216M 18.2.0 dc2bc1663786 3f315c3afc1f 2024-04-20T21:36:15.536 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (13m) 77s ago 12m 210M 1216M 18.2.0 dc2bc1663786 44e4d07416ad 2024-04-20T21:36:15.536 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (12m) 77s ago 12m 171M 1216M 18.2.0 dc2bc1663786 31c5d9586e98 2024-04-20T21:36:15.536 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (11m) 3m ago 11m 211M 2240M 18.2.0 dc2bc1663786 611c1e242693 2024-04-20T21:36:15.536 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (11m) 3m ago 11m 190M 2240M 18.2.0 dc2bc1663786 6c8fce91d45d 2024-04-20T21:36:15.536 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (10m) 3m ago 10m 189M 2240M 18.2.0 dc2bc1663786 8e4574bae8db 2024-04-20T21:36:15.536 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (2m) 77s ago 15m 61.7M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T21:36:15.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:15.538+0000 7f4756c00700 1 -- 172.21.15.62:0/718067189 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f473406e940 msgr2=0x7f4734070df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:36:15.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:15.538+0000 7f4756c00700 1 --2- 172.21.15.62:0/718067189 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f473406e940 0x7f4734070df0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f4740005950 tx=0x7f4740005eb0 comp rx=0 tx=0).stop 2024-04-20T21:36:15.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:15.538+0000 7f4756c00700 1 -- 172.21.15.62:0/718067189 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4750100e80 msgr2=0x7f47501963e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:36:15.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:15.538+0000 7f4756c00700 1 --2- 172.21.15.62:0/718067189 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4750100e80 0x7f47501963e0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f473c00bda0 tx=0x7f473c00be80 comp rx=0 tx=0).stop 2024-04-20T21:36:15.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:15.538+0000 7f4756c00700 1 -- 172.21.15.62:0/718067189 shutdown_connections 2024-04-20T21:36:15.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:15.538+0000 7f4756c00700 1 --2- 172.21.15.62:0/718067189 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f473406e940 0x7f4734070df0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:15.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:15.539+0000 7f4756c00700 1 --2- 172.21.15.62:0/718067189 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f47501004e0 0x7f4750195ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:15.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:15.539+0000 7f4756c00700 1 --2- 172.21.15.62:0/718067189 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4750100e80 0x7f47501963e0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:15.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:15.539+0000 7f4756c00700 1 -- 172.21.15.62:0/718067189 >> 172.21.15.62:0/718067189 conn(0x7f47500fbda0 msgr2=0x7f47501046c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:36:15.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:15.539+0000 7f4756c00700 1 -- 172.21.15.62:0/718067189 shutdown_connections 2024-04-20T21:36:15.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:15.539+0000 7f4756c00700 1 -- 172.21.15.62:0/718067189 wait complete. 2024-04-20T21:36:15.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:15.707+0000 7f3ee4600700 1 -- 172.21.15.62:0/160385187 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3edc108470 msgr2=0x7f3edc108840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:36:15.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:15.707+0000 7f3ee4600700 1 --2- 172.21.15.62:0/160385187 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3edc108470 0x7f3edc108840 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f3ed0009910 tx=0x7f3ed0009c20 comp rx=0 tx=0).stop 2024-04-20T21:36:15.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:15.707+0000 7f3ee4600700 1 -- 172.21.15.62:0/160385187 shutdown_connections 2024-04-20T21:36:15.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:15.707+0000 7f3ee4600700 1 --2- 172.21.15.62:0/160385187 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3edc102470 0x7f3edc1028e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:15.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:15.707+0000 7f3ee4600700 1 --2- 172.21.15.62:0/160385187 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3edc108470 0x7f3edc108840 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:15.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:15.707+0000 7f3ee4600700 1 -- 172.21.15.62:0/160385187 >> 172.21.15.62:0/160385187 conn(0x7f3edc0fdf90 msgr2=0x7f3edc1003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:36:15.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:15.707+0000 7f3ee4600700 1 -- 172.21.15.62:0/160385187 shutdown_connections 2024-04-20T21:36:15.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:15.707+0000 7f3ee4600700 1 -- 172.21.15.62:0/160385187 wait complete. 2024-04-20T21:36:15.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:15.708+0000 7f3ee4600700 1 Processor -- start 2024-04-20T21:36:15.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:15.708+0000 7f3ee4600700 1 -- start start 2024-04-20T21:36:15.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:15.709+0000 7f3ee4600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3edc102470 0x7f3edc1981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:36:15.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:15.709+0000 7f3ee4600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3edc108470 0x7f3edc198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:36:15.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:15.709+0000 7f3ee4600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3edc198d80 con 0x7f3edc102470 2024-04-20T21:36:15.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:15.709+0000 7f3ee4600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3edc198ec0 con 0x7f3edc108470 2024-04-20T21:36:15.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:15.709+0000 7f3ee1c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3edc102470 0x7f3edc1981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:36:15.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:15.709+0000 7f3ee1c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3edc102470 0x7f3edc1981f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37276/0 (socket says 172.21.15.62:37276) 2024-04-20T21:36:15.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:15.709+0000 7f3ee1c00700 1 -- 172.21.15.62:0/2454230936 learned_addr learned my addr 172.21.15.62:0/2454230936 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:36:15.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:15.710+0000 7f3ee1200700 1 --2- 172.21.15.62:0/2454230936 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3edc108470 0x7f3edc198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:36:15.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:15.710+0000 7f3ee1c00700 1 -- 172.21.15.62:0/2454230936 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3edc108470 msgr2=0x7f3edc198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:36:15.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:15.710+0000 7f3ee1c00700 1 --2- 172.21.15.62:0/2454230936 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3edc108470 0x7f3edc198730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:15.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:15.710+0000 7f3ee1c00700 1 -- 172.21.15.62:0/2454230936 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ed00095f0 con 0x7f3edc102470 2024-04-20T21:36:15.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:15.710+0000 7f3ee1c00700 1 --2- 172.21.15.62:0/2454230936 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3edc102470 0x7f3edc1981f0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f3ed0006010 tx=0x7f3ed000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:36:15.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:15.710+0000 7f3eda000700 1 -- 172.21.15.62:0/2454230936 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3ed001d070 con 0x7f3edc102470 2024-04-20T21:36:15.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:15.710+0000 7f3eda000700 1 -- 172.21.15.62:0/2454230936 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3ed0004fb0 con 0x7f3edc102470 2024-04-20T21:36:15.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:15.710+0000 7f3eda000700 1 -- 172.21.15.62:0/2454230936 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3ed000f690 con 0x7f3edc102470 2024-04-20T21:36:15.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:15.710+0000 7f3ee4600700 1 -- 172.21.15.62:0/2454230936 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3edc19ccb0 con 0x7f3edc102470 2024-04-20T21:36:15.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:15.711+0000 7f3ee4600700 1 -- 172.21.15.62:0/2454230936 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3edc19d1a0 con 0x7f3edc102470 2024-04-20T21:36:15.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:15.712+0000 7f3ee4600700 1 -- 172.21.15.62:0/2454230936 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3edc10a9e0 con 0x7f3edc102470 2024-04-20T21:36:15.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:15.713+0000 7f3eda000700 1 -- 172.21.15.62:0/2454230936 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 31) v1 ==== 92598+0+0 (secure 0 0 0) 0x7f3ed00228a0 con 0x7f3edc102470 2024-04-20T21:36:15.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:15.714+0000 7f3eda000700 1 --2- 172.21.15.62:0/2454230936 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f3ec406e940 0x7f3ec4070df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:36:15.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:15.714+0000 7f3eda000700 1 -- 172.21.15.62:0/2454230936 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f3ed0090140 con 0x7f3edc102470 2024-04-20T21:36:15.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:15.714+0000 7f3ee1200700 1 --2- 172.21.15.62:0/2454230936 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f3ec406e940 0x7f3ec4070df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:36:15.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:15.715+0000 7f3ee1200700 1 --2- 172.21.15.62:0/2454230936 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f3ec406e940 0x7f3ec4070df0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3ec8005ea0 tx=0x7f3ec8005e30 comp rx=0 tx=0).ready entity=mgr.14658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:36:15.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:15.719+0000 7f3eda000700 1 -- 172.21.15.62:0/2454230936 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187527 (secure 0 0 0) 0x7f3ed0059400 con 0x7f3edc102470 2024-04-20T21:36:15.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:15 smithi062 ceph-mon[16221]: from='client.14876 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:36:15.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:15 smithi062 ceph-mon[16221]: pgmap v142: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:36:15.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:15 smithi189 ceph-mon[21562]: from='client.14876 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:36:15.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:15 smithi189 ceph-mon[21562]: pgmap v142: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:36:16.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.058+0000 7f3ee4600700 1 -- 172.21.15.62:0/2454230936 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f3edc04f2a0 con 0x7f3edc102470 2024-04-20T21:36:16.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.059+0000 7f3eda000700 1 -- 172.21.15.62:0/2454230936 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+675 (secure 0 0 0) 0x7f3ed005cac0 con 0x7f3edc102470 2024-04-20T21:36:16.058 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:36:16.058 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T21:36:16.058 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 2 2024-04-20T21:36:16.059 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:36:16.059 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T21:36:16.059 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:36:16.059 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:36:16.059 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T21:36:16.059 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 6 2024-04-20T21:36:16.059 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:36:16.059 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T21:36:16.059 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 4 2024-04-20T21:36:16.059 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:36:16.059 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T21:36:16.059 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 12, 2024-04-20T21:36:16.059 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:36:16.059 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T21:36:16.059 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:36:16.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.062+0000 7f3ee4600700 1 -- 172.21.15.62:0/2454230936 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f3ec406e940 msgr2=0x7f3ec4070df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:36:16.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.062+0000 7f3ee4600700 1 --2- 172.21.15.62:0/2454230936 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f3ec406e940 0x7f3ec4070df0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3ec8005ea0 tx=0x7f3ec8005e30 comp rx=0 tx=0).stop 2024-04-20T21:36:16.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.062+0000 7f3ee4600700 1 -- 172.21.15.62:0/2454230936 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3edc102470 msgr2=0x7f3edc1981f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:36:16.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.062+0000 7f3ee4600700 1 --2- 172.21.15.62:0/2454230936 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3edc102470 0x7f3edc1981f0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f3ed0006010 tx=0x7f3ed000be20 comp rx=0 tx=0).stop 2024-04-20T21:36:16.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.062+0000 7f3ee4600700 1 -- 172.21.15.62:0/2454230936 shutdown_connections 2024-04-20T21:36:16.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.062+0000 7f3ee4600700 1 --2- 172.21.15.62:0/2454230936 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f3ec406e940 0x7f3ec4070df0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:16.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.062+0000 7f3ee4600700 1 --2- 172.21.15.62:0/2454230936 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3edc102470 0x7f3edc1981f0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:16.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.062+0000 7f3ee4600700 1 --2- 172.21.15.62:0/2454230936 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3edc108470 0x7f3edc198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:16.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.062+0000 7f3ee4600700 1 -- 172.21.15.62:0/2454230936 >> 172.21.15.62:0/2454230936 conn(0x7f3edc0fdf90 msgr2=0x7f3edc1002c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:36:16.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.062+0000 7f3ee4600700 1 -- 172.21.15.62:0/2454230936 shutdown_connections 2024-04-20T21:36:16.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.062+0000 7f3ee4600700 1 -- 172.21.15.62:0/2454230936 wait complete. 2024-04-20T21:36:16.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.233+0000 7f2e0be00700 1 -- 172.21.15.62:0/3564049667 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e04106b80 msgr2=0x7f2e04106ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:36:16.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.233+0000 7f2e0be00700 1 --2- 172.21.15.62:0/3564049667 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e04106b80 0x7f2e04106ff0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f2df0009910 tx=0x7f2df0009c20 comp rx=0 tx=0).stop 2024-04-20T21:36:16.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.233+0000 7f2e0be00700 1 -- 172.21.15.62:0/3564049667 shutdown_connections 2024-04-20T21:36:16.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.233+0000 7f2e0be00700 1 --2- 172.21.15.62:0/3564049667 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e04106b80 0x7f2e04106ff0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:16.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.233+0000 7f2e0be00700 1 --2- 172.21.15.62:0/3564049667 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e0410c8f0 0x7f2e0410ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:16.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.233+0000 7f2e0be00700 1 -- 172.21.15.62:0/3564049667 >> 172.21.15.62:0/3564049667 conn(0x7f2e04076020 msgr2=0x7f2e04078430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:36:16.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.233+0000 7f2e0be00700 1 -- 172.21.15.62:0/3564049667 shutdown_connections 2024-04-20T21:36:16.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.233+0000 7f2e0be00700 1 -- 172.21.15.62:0/3564049667 wait complete. 2024-04-20T21:36:16.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.235+0000 7f2e0be00700 1 Processor -- start 2024-04-20T21:36:16.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.235+0000 7f2e0be00700 1 -- start start 2024-04-20T21:36:16.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.235+0000 7f2e0be00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e04106b80 0x7f2e0419c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:36:16.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.235+0000 7f2e0be00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e0410c8f0 0x7f2e0419c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:36:16.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.235+0000 7f2e0be00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e0419d070 con 0x7f2e04106b80 2024-04-20T21:36:16.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.235+0000 7f2e0be00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e041a0e00 con 0x7f2e0410c8f0 2024-04-20T21:36:16.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.236+0000 7f2e09400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e04106b80 0x7f2e0419c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:36:16.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.236+0000 7f2e09400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e04106b80 0x7f2e0419c450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37306/0 (socket says 172.21.15.62:37306) 2024-04-20T21:36:16.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.236+0000 7f2e09400700 1 -- 172.21.15.62:0/3620270716 learned_addr learned my addr 172.21.15.62:0/3620270716 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:36:16.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.236+0000 7f2e08a00700 1 --2- 172.21.15.62:0/3620270716 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e0410c8f0 0x7f2e0419c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:36:16.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.236+0000 7f2e09400700 1 -- 172.21.15.62:0/3620270716 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e0410c8f0 msgr2=0x7f2e0419c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:36:16.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.236+0000 7f2e09400700 1 --2- 172.21.15.62:0/3620270716 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e0410c8f0 0x7f2e0419c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:16.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.236+0000 7f2e09400700 1 -- 172.21.15.62:0/3620270716 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2df00095f0 con 0x7f2e04106b80 2024-04-20T21:36:16.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.237+0000 7f2e09400700 1 --2- 172.21.15.62:0/3620270716 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e04106b80 0x7f2e0419c450 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f2df80060b0 tx=0x7f2df800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:36:16.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.237+0000 7f2e02000700 1 -- 172.21.15.62:0/3620270716 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2df800dda0 con 0x7f2e04106b80 2024-04-20T21:36:16.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.237+0000 7f2e02000700 1 -- 172.21.15.62:0/3620270716 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2df800f040 con 0x7f2e04106b80 2024-04-20T21:36:16.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.237+0000 7f2e0be00700 1 -- 172.21.15.62:0/3620270716 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e041a10e0 con 0x7f2e04106b80 2024-04-20T21:36:16.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.237+0000 7f2e02000700 1 -- 172.21.15.62:0/3620270716 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2df8014790 con 0x7f2e04106b80 2024-04-20T21:36:16.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.237+0000 7f2e0be00700 1 -- 172.21.15.62:0/3620270716 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e041a1630 con 0x7f2e04106b80 2024-04-20T21:36:16.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.239+0000 7f2e0be00700 1 -- 172.21.15.62:0/3620270716 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e0404f2a0 con 0x7f2e04106b80 2024-04-20T21:36:16.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.239+0000 7f2e02000700 1 -- 172.21.15.62:0/3620270716 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 31) v1 ==== 92598+0+0 (secure 0 0 0) 0x7f2df80148f0 con 0x7f2e04106b80 2024-04-20T21:36:16.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.240+0000 7f2e02000700 1 --2- 172.21.15.62:0/3620270716 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f2dec06e940 0x7f2dec070df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:36:16.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.241+0000 7f2e02000700 1 -- 172.21.15.62:0/3620270716 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f2df808e270 con 0x7f2e04106b80 2024-04-20T21:36:16.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.241+0000 7f2e08a00700 1 --2- 172.21.15.62:0/3620270716 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f2dec06e940 0x7f2dec070df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:36:16.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.241+0000 7f2e08a00700 1 --2- 172.21.15.62:0/3620270716 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f2dec06e940 0x7f2dec070df0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f2e0419da70 tx=0x7f2df001a040 comp rx=0 tx=0).ready entity=mgr.14658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:36:16.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.248+0000 7f2e02000700 1 -- 172.21.15.62:0/3620270716 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187527 (secure 0 0 0) 0x7f2df8057680 con 0x7f2e04106b80 2024-04-20T21:36:16.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.539+0000 7f2e0be00700 1 -- 172.21.15.62:0/3620270716 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f2e041a19a0 con 0x7f2e04106b80 2024-04-20T21:36:16.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.540+0000 7f2e02000700 1 -- 172.21.15.62:0/3620270716 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 16 v16) v1 ==== 76+0+1881 (secure 0 0 0) 0x7f2df805ad40 con 0x7f2e04106b80 2024-04-20T21:36:16.541 INFO:teuthology.orchestra.run.smithi062.stdout:e16 2024-04-20T21:36:16.541 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T21:36:16.541 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T21:36:16.541 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T21:36:16.541 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:36:16.541 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T21:36:16.542 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T21:36:16.542 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 16 2024-04-20T21:36:16.542 INFO:teuthology.orchestra.run.smithi062.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-20T21:36:16.542 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T21:36:16.542 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T21:31:14.906617+0000 2024-04-20T21:36:16.542 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T21:36:16.542 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T21:36:16.542 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T21:36:16.542 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T21:36:16.542 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T21:36:16.542 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T21:36:16.542 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T21:36:16.542 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 0 2024-04-20T21:36:16.542 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T21:36:16.542 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 2 2024-04-20T21:36:16.542 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1 2024-04-20T21:36:16.542 INFO:teuthology.orchestra.run.smithi062.stdout:up {0=14458,1=24279} 2024-04-20T21:36:16.543 INFO:teuthology.orchestra.run.smithi062.stdout:failed 2024-04-20T21:36:16.543 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T21:36:16.543 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T21:36:16.543 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T21:36:16.543 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T21:36:16.543 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data enabled 2024-04-20T21:36:16.543 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T21:36:16.543 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T21:36:16.543 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 1 2024-04-20T21:36:16.543 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ryqblb{0:14458} state up:active seq 6 export targets 1 join_fscid=1 addr [v2:172.21.15.62:6826/1147127123,v1:172.21.15.62:6827/1147127123] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:36:16.543 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ozizwi{0:14474} state up:standby-replay seq 3 join_fscid=1 addr [v2:172.21.15.62:6828/3494174293,v1:172.21.15.62:6829/3494174293] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:36:16.543 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.dvimer{1:24279} state up:active seq 5 export targets 0 join_fscid=1 addr [v2:172.21.15.189:6824/237262514,v1:172.21.15.189:6825/237262514] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:36:16.543 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.xbgspw{1:24291} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.189:6826/1495356258,v1:172.21.15.189:6827/1495356258] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:36:16.543 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:36:16.543 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:36:16.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.545+0000 7f2e0be00700 1 -- 172.21.15.62:0/3620270716 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f2dec06e940 msgr2=0x7f2dec070df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:36:16.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.545+0000 7f2e0be00700 1 --2- 172.21.15.62:0/3620270716 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f2dec06e940 0x7f2dec070df0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f2e0419da70 tx=0x7f2df001a040 comp rx=0 tx=0).stop 2024-04-20T21:36:16.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.545+0000 7f2e0be00700 1 -- 172.21.15.62:0/3620270716 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e04106b80 msgr2=0x7f2e0419c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:36:16.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.545+0000 7f2e0be00700 1 --2- 172.21.15.62:0/3620270716 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e04106b80 0x7f2e0419c450 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f2df80060b0 tx=0x7f2df800d3b0 comp rx=0 tx=0).stop 2024-04-20T21:36:16.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.546+0000 7f2e0be00700 1 -- 172.21.15.62:0/3620270716 shutdown_connections 2024-04-20T21:36:16.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.546+0000 7f2e0be00700 1 --2- 172.21.15.62:0/3620270716 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f2dec06e940 0x7f2dec070df0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:16.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.546+0000 7f2e0be00700 1 --2- 172.21.15.62:0/3620270716 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e04106b80 0x7f2e0419c450 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:16.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.546+0000 7f2e0be00700 1 --2- 172.21.15.62:0/3620270716 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e0410c8f0 0x7f2e0419c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:16.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.546+0000 7f2e0be00700 1 -- 172.21.15.62:0/3620270716 >> 172.21.15.62:0/3620270716 conn(0x7f2e04076020 msgr2=0x7f2e04077a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:36:16.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.546+0000 7f2e0be00700 1 -- 172.21.15.62:0/3620270716 shutdown_connections 2024-04-20T21:36:16.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.546+0000 7f2e0be00700 1 -- 172.21.15.62:0/3620270716 wait complete. 2024-04-20T21:36:16.547 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 16 2024-04-20T21:36:16.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.714+0000 7f7509600700 1 -- 172.21.15.62:0/3978842968 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7504073a00 msgr2=0x7f750410c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:36:16.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.714+0000 7f7509600700 1 --2- 172.21.15.62:0/3978842968 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7504073a00 0x7f750410c350 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f74f0009910 tx=0x7f74f0009c20 comp rx=0 tx=0).stop 2024-04-20T21:36:16.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.714+0000 7f7509600700 1 -- 172.21.15.62:0/3978842968 shutdown_connections 2024-04-20T21:36:16.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.714+0000 7f7509600700 1 --2- 172.21.15.62:0/3978842968 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7504073a00 0x7f750410c350 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:16.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.714+0000 7f7509600700 1 --2- 172.21.15.62:0/3978842968 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75040730f0 0x7f75040734c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:16.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.714+0000 7f7509600700 1 -- 172.21.15.62:0/3978842968 >> 172.21.15.62:0/3978842968 conn(0x7f75040fbb50 msgr2=0x7f75040fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:36:16.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.714+0000 7f7509600700 1 -- 172.21.15.62:0/3978842968 shutdown_connections 2024-04-20T21:36:16.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.714+0000 7f7509600700 1 -- 172.21.15.62:0/3978842968 wait complete. 2024-04-20T21:36:16.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.715+0000 7f7509600700 1 Processor -- start 2024-04-20T21:36:16.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.716+0000 7f7509600700 1 -- start start 2024-04-20T21:36:16.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.716+0000 7f7509600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75040730f0 0x7f7504197d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:36:16.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.716+0000 7f7509600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7504073a00 0x7f7504198390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:36:16.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.716+0000 7f7509600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f750419c6d0 con 0x7f7504073a00 2024-04-20T21:36:16.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.716+0000 7f7509600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f750419c840 con 0x7f75040730f0 2024-04-20T21:36:16.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.716+0000 7f7503400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7504073a00 0x7f7504198390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:36:16.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.717+0000 7f7503400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7504073a00 0x7f7504198390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37318/0 (socket says 172.21.15.62:37318) 2024-04-20T21:36:16.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.717+0000 7f7503400700 1 -- 172.21.15.62:0/1523418736 learned_addr learned my addr 172.21.15.62:0/1523418736 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:36:16.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.717+0000 7f7503e00700 1 --2- 172.21.15.62:0/1523418736 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75040730f0 0x7f7504197d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:36:16.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.717+0000 7f7503400700 1 -- 172.21.15.62:0/1523418736 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75040730f0 msgr2=0x7f7504197d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:36:16.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.717+0000 7f7503400700 1 --2- 172.21.15.62:0/1523418736 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75040730f0 0x7f7504197d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:16.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.717+0000 7f7503400700 1 -- 172.21.15.62:0/1523418736 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74f00095f0 con 0x7f7504073a00 2024-04-20T21:36:16.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.717+0000 7f7503400700 1 --2- 172.21.15.62:0/1523418736 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7504073a00 0x7f7504198390 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f74f000b5c0 tx=0x7f74f000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:36:16.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.718+0000 7f7500c00700 1 -- 172.21.15.62:0/1523418736 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f74f001d070 con 0x7f7504073a00 2024-04-20T21:36:16.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.718+0000 7f7509600700 1 -- 172.21.15.62:0/1523418736 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f750419cb20 con 0x7f7504073a00 2024-04-20T21:36:16.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.718+0000 7f7500c00700 1 -- 172.21.15.62:0/1523418736 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f74f0005e00 con 0x7f7504073a00 2024-04-20T21:36:16.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.718+0000 7f7509600700 1 -- 172.21.15.62:0/1523418736 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f750419d070 con 0x7f7504073a00 2024-04-20T21:36:16.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.718+0000 7f7500c00700 1 -- 172.21.15.62:0/1523418736 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f74f002b440 con 0x7f7504073a00 2024-04-20T21:36:16.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.720+0000 7f7509600700 1 -- 172.21.15.62:0/1523418736 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7504109a50 con 0x7f7504073a00 2024-04-20T21:36:16.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.721+0000 7f7500c00700 1 -- 172.21.15.62:0/1523418736 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 31) v1 ==== 92598+0+0 (secure 0 0 0) 0x7f74f000f460 con 0x7f7504073a00 2024-04-20T21:36:16.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.722+0000 7f7500c00700 1 --2- 172.21.15.62:0/1523418736 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f74e806e940 0x7f74e8070df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:36:16.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.722+0000 7f7500c00700 1 -- 172.21.15.62:0/1523418736 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f74f008fdb0 con 0x7f7504073a00 2024-04-20T21:36:16.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.722+0000 7f7503e00700 1 --2- 172.21.15.62:0/1523418736 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f74e806e940 0x7f74e8070df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:36:16.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.723+0000 7f7503e00700 1 --2- 172.21.15.62:0/1523418736 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f74e806e940 0x7f74e8070df0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f74f4005950 tx=0x7f74f40058e0 comp rx=0 tx=0).ready entity=mgr.14658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:36:16.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.727+0000 7f7500c00700 1 -- 172.21.15.62:0/1523418736 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187527 (secure 0 0 0) 0x7f74f0058ff0 con 0x7f7504073a00 2024-04-20T21:36:16.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.984+0000 7f7509600700 1 -- 172.21.15.62:0/1523418736 --> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f75041990d0 con 0x7f74e806e940 2024-04-20T21:36:16.984 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:16 smithi062 ceph-mon[16221]: from='client.14880 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:36:16.984 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:16 smithi062 ceph-mon[16221]: from='client.14884 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:36:16.984 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:16 smithi062 ceph-mon[16221]: from='client.? 172.21.15.62:0/2454230936' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:36:16.984 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:16 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:36:16.984 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:16 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:36:16.984 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:16 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:36:16.984 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:16 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:36:16.984 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:16 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-20T21:36:16.984 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:16 smithi062 ceph-mon[16221]: from='client.? 172.21.15.62:0/3620270716' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T21:36:16.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.986+0000 7f7500c00700 1 -- 172.21.15.62:0/1523418736 <== mgr.14658 v2:172.21.15.62:6800/1585760929 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+368 (secure 0 0 0) 0x7f75041990d0 con 0x7f74e806e940 2024-04-20T21:36:16.986 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:36:16.986 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T21:36:16.986 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T21:36:16.986 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-20T21:36:16.986 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T21:36:16.986 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr" 2024-04-20T21:36:16.986 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T21:36:16.986 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "2/23 daemons upgraded", 2024-04-20T21:36:16.986 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading mon daemons", 2024-04-20T21:36:16.986 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T21:36:16.986 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:36:16.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:16 smithi189 ceph-mon[21562]: from='client.14880 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:36:16.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:16 smithi189 ceph-mon[21562]: from='client.14884 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:36:16.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:16 smithi189 ceph-mon[21562]: from='client.? 172.21.15.62:0/2454230936' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:36:16.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:16 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:36:16.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:16 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:36:16.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:16 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:36:16.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:16 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:36:16.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:16 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-20T21:36:16.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:16 smithi189 ceph-mon[21562]: from='client.? 172.21.15.62:0/3620270716' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T21:36:16.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.989+0000 7f7509600700 1 -- 172.21.15.62:0/1523418736 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f74e806e940 msgr2=0x7f74e8070df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:36:16.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.989+0000 7f7509600700 1 --2- 172.21.15.62:0/1523418736 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f74e806e940 0x7f74e8070df0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f74f4005950 tx=0x7f74f40058e0 comp rx=0 tx=0).stop 2024-04-20T21:36:16.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.989+0000 7f7509600700 1 -- 172.21.15.62:0/1523418736 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7504073a00 msgr2=0x7f7504198390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:36:16.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.989+0000 7f7509600700 1 --2- 172.21.15.62:0/1523418736 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7504073a00 0x7f7504198390 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f74f000b5c0 tx=0x7f74f000be20 comp rx=0 tx=0).stop 2024-04-20T21:36:16.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.989+0000 7f7509600700 1 -- 172.21.15.62:0/1523418736 shutdown_connections 2024-04-20T21:36:16.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.989+0000 7f7509600700 1 --2- 172.21.15.62:0/1523418736 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7f74e806e940 0x7f74e8070df0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:16.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.989+0000 7f7509600700 1 --2- 172.21.15.62:0/1523418736 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75040730f0 0x7f7504197d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:16.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.989+0000 7f7509600700 1 --2- 172.21.15.62:0/1523418736 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7504073a00 0x7f7504198390 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:16.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.989+0000 7f7509600700 1 -- 172.21.15.62:0/1523418736 >> 172.21.15.62:0/1523418736 conn(0x7f75040fbb50 msgr2=0x7f7504106b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:36:16.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.990+0000 7f7509600700 1 -- 172.21.15.62:0/1523418736 shutdown_connections 2024-04-20T21:36:16.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:16.990+0000 7f7509600700 1 -- 172.21.15.62:0/1523418736 wait complete. 2024-04-20T21:36:17.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:17.156+0000 7fa4ace00700 1 -- 172.21.15.62:0/2613125529 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa4a8100c70 msgr2=0x7fa4a8104c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:36:17.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:17.156+0000 7fa4ace00700 1 --2- 172.21.15.62:0/2613125529 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa4a8100c70 0x7fa4a8104c40 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fa494009910 tx=0x7fa494009c20 comp rx=0 tx=0).stop 2024-04-20T21:36:17.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:17.157+0000 7fa4ace00700 1 -- 172.21.15.62:0/2613125529 shutdown_connections 2024-04-20T21:36:17.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:17.157+0000 7fa4ace00700 1 --2- 172.21.15.62:0/2613125529 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa4a8100c70 0x7fa4a8104c40 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:17.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:17.157+0000 7fa4ace00700 1 --2- 172.21.15.62:0/2613125529 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4a81002d0 0x7fa4a81006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:17.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:17.157+0000 7fa4ace00700 1 -- 172.21.15.62:0/2613125529 >> 172.21.15.62:0/2613125529 conn(0x7fa4a80fbbb0 msgr2=0x7fa4a80fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:36:17.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:17.157+0000 7fa4ace00700 1 -- 172.21.15.62:0/2613125529 shutdown_connections 2024-04-20T21:36:17.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:17.157+0000 7fa4ace00700 1 -- 172.21.15.62:0/2613125529 wait complete. 2024-04-20T21:36:17.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:17.158+0000 7fa4ace00700 1 Processor -- start 2024-04-20T21:36:17.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:17.158+0000 7fa4ace00700 1 -- start start 2024-04-20T21:36:17.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:17.158+0000 7fa4ace00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa4a81002d0 0x7fa4a8073050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:36:17.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:17.158+0000 7fa4ace00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4a8100c70 0x7fa4a8073590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:36:17.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:17.158+0000 7fa4ace00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a8073ad0 con 0x7fa4a81002d0 2024-04-20T21:36:17.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:17.158+0000 7fa4ace00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a8073c10 con 0x7fa4a8100c70 2024-04-20T21:36:17.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:17.159+0000 7fa4a7400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa4a81002d0 0x7fa4a8073050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:36:17.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:17.159+0000 7fa4a7400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa4a81002d0 0x7fa4a8073050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37334/0 (socket says 172.21.15.62:37334) 2024-04-20T21:36:17.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:17.159+0000 7fa4a7400700 1 -- 172.21.15.62:0/2090481733 learned_addr learned my addr 172.21.15.62:0/2090481733 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:36:17.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:17.159+0000 7fa4a7400700 1 -- 172.21.15.62:0/2090481733 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4a8100c70 msgr2=0x7fa4a8073590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:36:17.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:17.159+0000 7fa4a7400700 1 --2- 172.21.15.62:0/2090481733 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4a8100c70 0x7fa4a8073590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:17.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:17.159+0000 7fa4a7400700 1 -- 172.21.15.62:0/2090481733 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4940095f0 con 0x7fa4a81002d0 2024-04-20T21:36:17.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:17.159+0000 7fa4a7400700 1 --2- 172.21.15.62:0/2090481733 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa4a81002d0 0x7fa4a8073050 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fa49800d8d0 tx=0x7fa49800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:36:17.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:17.160+0000 7fa49fe00700 1 -- 172.21.15.62:0/2090481733 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa498009940 con 0x7fa4a81002d0 2024-04-20T21:36:17.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:17.160+0000 7fa49fe00700 1 -- 172.21.15.62:0/2090481733 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa498010460 con 0x7fa4a81002d0 2024-04-20T21:36:17.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:17.160+0000 7fa4ace00700 1 -- 172.21.15.62:0/2090481733 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4a81a1e30 con 0x7fa4a81002d0 2024-04-20T21:36:17.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:17.160+0000 7fa49fe00700 1 -- 172.21.15.62:0/2090481733 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa49800f5f0 con 0x7fa4a81002d0 2024-04-20T21:36:17.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:17.160+0000 7fa4ace00700 1 -- 172.21.15.62:0/2090481733 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4a81a2240 con 0x7fa4a81002d0 2024-04-20T21:36:17.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:17.161+0000 7fa4ace00700 1 -- 172.21.15.62:0/2090481733 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4a8062380 con 0x7fa4a81002d0 2024-04-20T21:36:17.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:17.162+0000 7fa49fe00700 1 -- 172.21.15.62:0/2090481733 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 31) v1 ==== 92598+0+0 (secure 0 0 0) 0x7fa4980105d0 con 0x7fa4a81002d0 2024-04-20T21:36:17.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:17.163+0000 7fa49fe00700 1 --2- 172.21.15.62:0/2090481733 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7fa48c06e940 0x7fa48c070df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:36:17.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:17.163+0000 7fa49fe00700 1 -- 172.21.15.62:0/2090481733 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6070+0+0 (secure 0 0 0) 0x7fa49808e850 con 0x7fa4a81002d0 2024-04-20T21:36:17.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:17.164+0000 7fa4a6a00700 1 --2- 172.21.15.62:0/2090481733 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7fa48c06e940 0x7fa48c070df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:36:17.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:17.164+0000 7fa4a6a00700 1 --2- 172.21.15.62:0/2090481733 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7fa48c06e940 0x7fa48c070df0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa49400b5c0 tx=0x7fa494005fd0 comp rx=0 tx=0).ready entity=mgr.14658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:36:17.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:17.169+0000 7fa49fe00700 1 -- 172.21.15.62:0/2090481733 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187527 (secure 0 0 0) 0x7fa498057b10 con 0x7fa4a81002d0 2024-04-20T21:36:17.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:17.500+0000 7fa4ace00700 1 -- 172.21.15.62:0/2090481733 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fa4a8074580 con 0x7fa4a81002d0 2024-04-20T21:36:17.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:17.501+0000 7fa49fe00700 1 -- 172.21.15.62:0/2090481733 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7fa49805b1d0 con 0x7fa4a81002d0 2024-04-20T21:36:17.500 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-04-20T21:36:17.500 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T21:36:17.500 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T21:36:17.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:17.503+0000 7fa4ace00700 1 -- 172.21.15.62:0/2090481733 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7fa48c06e940 msgr2=0x7fa48c070df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:36:17.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:17.503+0000 7fa4ace00700 1 --2- 172.21.15.62:0/2090481733 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7fa48c06e940 0x7fa48c070df0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa49400b5c0 tx=0x7fa494005fd0 comp rx=0 tx=0).stop 2024-04-20T21:36:17.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:17.503+0000 7fa4ace00700 1 -- 172.21.15.62:0/2090481733 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa4a81002d0 msgr2=0x7fa4a8073050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:36:17.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:17.503+0000 7fa4ace00700 1 --2- 172.21.15.62:0/2090481733 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa4a81002d0 0x7fa4a8073050 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fa49800d8d0 tx=0x7fa49800dc90 comp rx=0 tx=0).stop 2024-04-20T21:36:17.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:17.504+0000 7fa4ace00700 1 -- 172.21.15.62:0/2090481733 shutdown_connections 2024-04-20T21:36:17.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:17.504+0000 7fa4ace00700 1 --2- 172.21.15.62:0/2090481733 >> [v2:172.21.15.62:6800/1585760929,v1:172.21.15.62:6801/1585760929] conn(0x7fa48c06e940 0x7fa48c070df0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:17.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:17.504+0000 7fa4ace00700 1 --2- 172.21.15.62:0/2090481733 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa4a81002d0 0x7fa4a8073050 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:17.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:17.504+0000 7fa4ace00700 1 --2- 172.21.15.62:0/2090481733 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4a8100c70 0x7fa4a8073590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:17.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:17.504+0000 7fa4ace00700 1 -- 172.21.15.62:0/2090481733 >> 172.21.15.62:0/2090481733 conn(0x7fa4a80fbbb0 msgr2=0x7fa4a80fd130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:36:17.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:17.504+0000 7fa4ace00700 1 -- 172.21.15.62:0/2090481733 shutdown_connections 2024-04-20T21:36:17.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:17.504+0000 7fa4ace00700 1 -- 172.21.15.62:0/2090481733 wait complete. 2024-04-20T21:36:17.975 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:17 smithi062 ceph-mon[16221]: Upgrade: Target is version 19.0.0-3133-ga8c84f59 (squid) 2024-04-20T21:36:17.975 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:17 smithi062 ceph-mon[16221]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, digests ['quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1'] 2024-04-20T21:36:17.975 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:17 smithi062 ceph-mon[16221]: Upgrade: Setting container_image for all mgr 2024-04-20T21:36:17.975 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:17 smithi062 ceph-mon[16221]: pgmap v143: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:36:17.976 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:17 smithi062 ceph-mon[16221]: from='client.? 172.21.15.62:0/2090481733' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T21:36:17.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:17 smithi189 ceph-mon[21562]: Upgrade: Target is version 19.0.0-3133-ga8c84f59 (squid) 2024-04-20T21:36:17.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:17 smithi189 ceph-mon[21562]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, digests ['quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1'] 2024-04-20T21:36:17.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:17 smithi189 ceph-mon[21562]: Upgrade: Setting container_image for all mgr 2024-04-20T21:36:17.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:17 smithi189 ceph-mon[21562]: pgmap v143: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:36:17.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:17 smithi189 ceph-mon[21562]: from='client.? 172.21.15.62:0/2090481733' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T21:36:19.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:18 smithi062 ceph-mon[16221]: from='client.14896 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:36:19.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:18 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:36:19.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:18 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-20T21:36:19.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:18 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-20T21:36:19.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:18 smithi062 ceph-mon[16221]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:36:19.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:18 smithi189 ceph-mon[21562]: from='client.14896 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:36:19.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:18 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:36:19.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:18 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-20T21:36:19.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:18 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-20T21:36:19.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:18 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:36:20.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:19 smithi062 ceph-mon[16221]: Upgrade: Updating mon.smithi062 2024-04-20T21:36:20.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:19 smithi062 ceph-mon[16221]: Deploying daemon mon.smithi062 on smithi062 2024-04-20T21:36:20.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:19 smithi062 ceph-mon[16221]: pgmap v144: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:36:20.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:19 smithi189 ceph-mon[21562]: Upgrade: Updating mon.smithi062 2024-04-20T21:36:20.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:19 smithi189 ceph-mon[21562]: Deploying daemon mon.smithi062 on smithi062 2024-04-20T21:36:20.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:19 smithi189 ceph-mon[21562]: pgmap v144: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:36:21.537 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:21 smithi062 systemd[1]: Stopping Ceph mon.smithi062 for 61ae3628-ff5b-11ee-bc93-c7b262605968... 2024-04-20T21:36:21.927 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:21 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-mon-smithi062[16216]: 2024-04-20T21:36:21.577+0000 7fa3ce000700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi062 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-20T21:36:21.928 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:21 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-mon-smithi062[16216]: 2024-04-20T21:36:21.577+0000 7fa3ce000700 -1 mon.smithi062@0(leader) e2 *** Got Signal Terminated *** 2024-04-20T21:36:21.928 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:21 smithi062 podman[76153]: 2024-04-20 21:36:21.639789516 +0000 UTC m=+0.179752267 container died bd6efbd5c380cda9118a1a2c96ddddc0fe61d4a9ab8109dbe18acf00687d2e4f (image=quay.io/ceph/ceph:v18.2.0, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-mon-smithi062, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-18.2.0, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, RELEASE=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20231212, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, ceph=True) 2024-04-20T21:36:22.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:22 smithi062 podman[76153]: 2024-04-20 21:36:22.103569445 +0000 UTC m=+0.643532229 container cleanup bd6efbd5c380cda9118a1a2c96ddddc0fe61d4a9ab8109dbe18acf00687d2e4f (image=quay.io/ceph/ceph:v18.2.0, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-mon-smithi062, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20231212, CEPH_POINT_RELEASE=-18.2.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, RELEASE=HEAD) 2024-04-20T21:36:22.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:22 smithi062 bash[76153]: ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-mon-smithi062 2024-04-20T21:36:22.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:22 smithi062 podman[76177]: 2024-04-20 21:36:22.346883148 +0000 UTC m=+0.691995322 container remove bd6efbd5c380cda9118a1a2c96ddddc0fe61d4a9ab8109dbe18acf00687d2e4f (image=quay.io/ceph/ceph:v18.2.0, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-mon-smithi062, org.label-schema.vendor=CentOS, ceph=True, RELEASE=HEAD, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, CEPH_POINT_RELEASE=-18.2.0, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20231212, io.buildah.version=1.29.1) 2024-04-20T21:36:23.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:22 smithi062 systemd[1]: ceph-61ae3628-ff5b-11ee-bc93-c7b262605968@mon.smithi062.service: Deactivated successfully. 2024-04-20T21:36:23.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:22 smithi062 systemd[1]: Stopped Ceph mon.smithi062 for 61ae3628-ff5b-11ee-bc93-c7b262605968. 2024-04-20T21:36:23.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:22 smithi062 systemd[1]: ceph-61ae3628-ff5b-11ee-bc93-c7b262605968@mon.smithi062.service: Consumed 33.899s CPU time. 2024-04-20T21:36:23.687 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:23 smithi062 systemd[1]: Starting Ceph mon.smithi062 for 61ae3628-ff5b-11ee-bc93-c7b262605968... 2024-04-20T21:36:23.967 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:23 smithi062 podman[76304]: 2024-04-20 21:36:23.597341136 +0000 UTC m=+0.042951093 image pull 76bc6b91b726a03b8b736eeeb35fc1ffd1469e23c7d60f5b444a217528d8906d quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 2024-04-20T21:36:23.967 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:23 smithi062 podman[76304]: 2024-04-20 21:36:23.697750943 +0000 UTC m=+0.143360929 container create 019eb14794852cc40f5e7beeaa6d021dc8a8cf60ed30bdb363900c0cd8f6e1d7 (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-mon-smithi062, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-20T21:36:23.967 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:23 smithi062 podman[76304]: 2024-04-20 21:36:23.968037787 +0000 UTC m=+0.413647779 container init 019eb14794852cc40f5e7beeaa6d021dc8a8cf60ed30bdb363900c0cd8f6e1d7 (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-mon-smithi062, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-20T21:36:24.287 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:23 smithi062 podman[76304]: 2024-04-20 21:36:23.979522907 +0000 UTC m=+0.425132877 container start 019eb14794852cc40f5e7beeaa6d021dc8a8cf60ed30bdb363900c0cd8f6e1d7 (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-mon-smithi062, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2) 2024-04-20T21:36:24.288 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: set uid:gid to 167:167 (ceph:ceph) 2024-04-20T21:36:24.288 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev), process ceph-mon, pid 2 2024-04-20T21:36:24.288 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: pidfile_write: ignore empty --pid-file 2024-04-20T21:36:24.288 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: load: jerasure load: lrc 2024-04-20T21:36:24.288 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: RocksDB version: 7.9.2 2024-04-20T21:36:24.288 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Git sha 0 2024-04-20T21:36:24.288 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Compile date 2024-04-17 00:07:32 2024-04-20T21:36:24.288 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: DB SUMMARY 2024-04-20T21:36:24.288 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: DB Session ID: YYW1LLBSVT91MJFSFU4U 2024-04-20T21:36:24.288 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: CURRENT file: CURRENT 2024-04-20T21:36:24.288 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: IDENTITY file: IDENTITY 2024-04-20T21:36:24.288 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: MANIFEST file: MANIFEST-000015 size: 1732 Bytes 2024-04-20T21:36:24.288 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi062/store.db dir, Total Num: 1, files: 000035.sst 2024-04-20T21:36:24.289 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi062/store.db: 000033.log size: 985297 ; 2024-04-20T21:36:24.289 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.error_if_exists: 0 2024-04-20T21:36:24.289 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.create_if_missing: 0 2024-04-20T21:36:24.289 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.paranoid_checks: 1 2024-04-20T21:36:24.289 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-20T21:36:24.289 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-20T21:36:24.289 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-20T21:36:24.289 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.env: 0x55bc624c0cc0 2024-04-20T21:36:24.289 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.fs: PosixFileSystem 2024-04-20T21:36:24.289 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.info_log: 0x55bc6b9b5b00 2024-04-20T21:36:24.289 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.max_file_opening_threads: 16 2024-04-20T21:36:24.289 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.statistics: (nil) 2024-04-20T21:36:24.289 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.use_fsync: 0 2024-04-20T21:36:24.289 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.max_log_file_size: 0 2024-04-20T21:36:24.289 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-20T21:36:24.290 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-20T21:36:24.290 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.keep_log_file_num: 1000 2024-04-20T21:36:24.290 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.recycle_log_file_num: 0 2024-04-20T21:36:24.290 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.allow_fallocate: 1 2024-04-20T21:36:24.290 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.allow_mmap_reads: 0 2024-04-20T21:36:24.290 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.allow_mmap_writes: 0 2024-04-20T21:36:24.290 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.use_direct_reads: 0 2024-04-20T21:36:24.290 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-20T21:36:24.290 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.create_missing_column_families: 0 2024-04-20T21:36:24.291 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.db_log_dir: 2024-04-20T21:36:24.291 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.wal_dir: 2024-04-20T21:36:24.291 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-20T21:36:24.291 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-20T21:36:24.291 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-20T21:36:24.291 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-20T21:36:24.291 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-20T21:36:24.291 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-20T21:36:24.291 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.advise_random_on_open: 1 2024-04-20T21:36:24.291 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.db_write_buffer_size: 0 2024-04-20T21:36:24.291 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.write_buffer_manager: 0x55bc6b9841e0 2024-04-20T21:36:24.291 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-20T21:36:24.291 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-20T21:36:24.291 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-20T21:36:24.291 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.rate_limiter: (nil) 2024-04-20T21:36:24.292 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-20T21:36:24.292 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.wal_recovery_mode: 2 2024-04-20T21:36:24.292 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.enable_thread_tracking: 0 2024-04-20T21:36:24.292 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.enable_pipelined_write: 0 2024-04-20T21:36:24.292 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.unordered_write: 0 2024-04-20T21:36:24.292 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-20T21:36:24.292 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-20T21:36:24.292 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-20T21:36:24.292 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-20T21:36:24.292 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.row_cache: None 2024-04-20T21:36:24.292 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.wal_filter: None 2024-04-20T21:36:24.292 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-20T21:36:24.292 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.allow_ingest_behind: 0 2024-04-20T21:36:24.292 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.two_write_queues: 0 2024-04-20T21:36:24.292 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.manual_wal_flush: 0 2024-04-20T21:36:24.293 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.wal_compression: 0 2024-04-20T21:36:24.293 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.atomic_flush: 0 2024-04-20T21:36:24.293 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-20T21:36:24.293 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-20T21:36:24.293 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-20T21:36:24.293 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.log_readahead_size: 0 2024-04-20T21:36:24.293 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-20T21:36:24.293 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.best_efforts_recovery: 0 2024-04-20T21:36:24.293 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-20T21:36:24.294 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-20T21:36:24.294 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.allow_data_in_errors: 0 2024-04-20T21:36:24.294 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.db_host_id: __hostname__ 2024-04-20T21:36:24.294 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-20T21:36:24.294 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.max_background_jobs: 2 2024-04-20T21:36:24.294 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.max_background_compactions: -1 2024-04-20T21:36:24.294 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.max_subcompactions: 1 2024-04-20T21:36:24.294 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-20T21:36:24.294 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-20T21:36:24.294 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-20T21:36:24.294 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.max_total_wal_size: 0 2024-04-20T21:36:24.294 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-20T21:36:24.294 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-20T21:36:24.294 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-20T21:36:24.294 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-20T21:36:24.295 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.max_open_files: -1 2024-04-20T21:36:24.295 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.bytes_per_sync: 0 2024-04-20T21:36:24.295 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-20T21:36:24.295 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-20T21:36:24.295 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.compaction_readahead_size: 0 2024-04-20T21:36:24.295 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.max_background_flushes: -1 2024-04-20T21:36:24.295 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Compression algorithms supported: 2024-04-20T21:36:24.295 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-20T21:36:24.295 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: kZSTD supported: 0 2024-04-20T21:36:24.295 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: kXpressCompression supported: 0 2024-04-20T21:36:24.295 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: kLZ4HCCompression supported: 1 2024-04-20T21:36:24.295 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: kZlibCompression supported: 1 2024-04-20T21:36:24.295 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: kSnappyCompression supported: 1 2024-04-20T21:36:24.295 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: kLZ4Compression supported: 1 2024-04-20T21:36:24.295 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: kBZip2Compression supported: 0 2024-04-20T21:36:24.296 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-20T21:36:24.296 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-20T21:36:24.296 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi062/store.db/MANIFEST-000015 2024-04-20T21:36:24.296 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-20T21:36:24.296 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-20T21:36:24.296 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.merge_operator: 2024-04-20T21:36:24.296 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.compaction_filter: None 2024-04-20T21:36:24.296 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.compaction_filter_factory: None 2024-04-20T21:36:24.296 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.sst_partitioner_factory: None 2024-04-20T21:36:24.297 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-20T21:36:24.297 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-20T21:36:24.297 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55bc6b9b5ce0) 2024-04-20T21:36:24.297 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: cache_index_and_filter_blocks: 1 2024-04-20T21:36:24.297 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-20T21:36:24.297 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-20T21:36:24.297 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: pin_top_level_index_and_filter: 1 2024-04-20T21:36:24.297 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: index_type: 0 2024-04-20T21:36:24.297 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: data_block_index_type: 0 2024-04-20T21:36:24.297 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: index_shortening: 1 2024-04-20T21:36:24.297 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-20T21:36:24.297 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: checksum: 4 2024-04-20T21:36:24.297 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: no_block_cache: 0 2024-04-20T21:36:24.297 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: block_cache: 0x55bc6b9b7090 2024-04-20T21:36:24.297 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: block_cache_name: BinnedLRUCache 2024-04-20T21:36:24.298 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: block_cache_options: 2024-04-20T21:36:24.298 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: capacity : 536870912 2024-04-20T21:36:24.298 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: num_shard_bits : 4 2024-04-20T21:36:24.298 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: strict_capacity_limit : 0 2024-04-20T21:36:24.298 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: high_pri_pool_ratio: 0.000 2024-04-20T21:36:24.298 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: block_cache_compressed: (nil) 2024-04-20T21:36:24.298 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: persistent_cache: (nil) 2024-04-20T21:36:24.298 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: block_size: 4096 2024-04-20T21:36:24.298 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: block_size_deviation: 10 2024-04-20T21:36:24.298 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: block_restart_interval: 16 2024-04-20T21:36:24.298 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: index_block_restart_interval: 1 2024-04-20T21:36:24.298 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: metadata_block_size: 4096 2024-04-20T21:36:24.298 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: partition_filters: 0 2024-04-20T21:36:24.298 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: use_delta_encoding: 1 2024-04-20T21:36:24.299 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: filter_policy: bloomfilter 2024-04-20T21:36:24.299 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: whole_key_filtering: 1 2024-04-20T21:36:24.299 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: verify_compression: 0 2024-04-20T21:36:24.299 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: read_amp_bytes_per_bit: 0 2024-04-20T21:36:24.299 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: format_version: 5 2024-04-20T21:36:24.299 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: enable_index_compression: 1 2024-04-20T21:36:24.299 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: block_align: 0 2024-04-20T21:36:24.299 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: max_auto_readahead_size: 262144 2024-04-20T21:36:24.299 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: prepopulate_block_cache: 0 2024-04-20T21:36:24.299 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: initial_auto_readahead_size: 8192 2024-04-20T21:36:24.299 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: num_file_reads_for_auto_readahead: 2 2024-04-20T21:36:24.299 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.write_buffer_size: 33554432 2024-04-20T21:36:24.299 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.max_write_buffer_number: 2 2024-04-20T21:36:24.299 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.compression: NoCompression 2024-04-20T21:36:24.300 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.bottommost_compression: Disabled 2024-04-20T21:36:24.300 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.prefix_extractor: nullptr 2024-04-20T21:36:24.300 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-20T21:36:24.300 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.num_levels: 7 2024-04-20T21:36:24.300 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-20T21:36:24.301 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-20T21:36:24.301 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-20T21:36:24.301 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-20T21:36:24.301 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-20T21:36:24.301 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-20T21:36:24.301 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-20T21:36:24.301 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-20T21:36:24.301 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-20T21:36:24.301 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-20T21:36:24.301 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-20T21:36:24.302 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-20T21:36:24.302 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-20T21:36:24.302 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.compression_opts.level: 32767 2024-04-20T21:36:24.302 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.compression_opts.strategy: 0 2024-04-20T21:36:24.302 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-20T21:36:24.302 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-20T21:36:24.302 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-20T21:36:24.302 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-20T21:36:24.302 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.compression_opts.enabled: false 2024-04-20T21:36:24.302 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-20T21:36:24.302 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-20T21:36:24.302 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-20T21:36:24.302 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-20T21:36:24.303 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.target_file_size_base: 67108864 2024-04-20T21:36:24.303 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-20T21:36:24.303 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-20T21:36:24.303 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-20T21:36:24.303 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-20T21:36:24.303 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-20T21:36:24.303 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-20T21:36:24.303 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-20T21:36:24.303 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-20T21:36:24.303 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-20T21:36:24.303 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-20T21:36:24.303 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-20T21:36:24.303 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-20T21:36:24.303 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-20T21:36:24.303 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-20T21:36:24.304 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.arena_block_size: 1048576 2024-04-20T21:36:24.304 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-20T21:36:24.304 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-20T21:36:24.304 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.disable_auto_compactions: 0 2024-04-20T21:36:24.304 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-20T21:36:24.304 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-20T21:36:24.304 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-20T21:36:24.304 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-20T21:36:24.304 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-20T21:36:24.304 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-20T21:36:24.304 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-20T21:36:24.304 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-20T21:36:24.304 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-20T21:36:24.304 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-20T21:36:24.304 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-20T21:36:24.305 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.inplace_update_support: 0 2024-04-20T21:36:24.305 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-20T21:36:24.305 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-20T21:36:24.305 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-20T21:36:24.305 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-20T21:36:24.305 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.bloom_locality: 0 2024-04-20T21:36:24.305 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.max_successive_merges: 0 2024-04-20T21:36:24.305 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-20T21:36:24.305 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.paranoid_file_checks: 0 2024-04-20T21:36:24.305 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.force_consistency_checks: 1 2024-04-20T21:36:24.305 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.report_bg_io_stats: 0 2024-04-20T21:36:24.305 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.ttl: 2592000 2024-04-20T21:36:24.305 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-20T21:36:24.305 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-20T21:36:24.305 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-20T21:36:24.306 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.enable_blob_files: false 2024-04-20T21:36:24.306 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.min_blob_size: 0 2024-04-20T21:36:24.306 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.blob_file_size: 268435456 2024-04-20T21:36:24.306 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-20T21:36:24.306 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-20T21:36:24.306 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-20T21:36:24.307 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-20T21:36:24.307 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-20T21:36:24.307 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.blob_file_starting_level: 0 2024-04-20T21:36:24.307 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-20T21:36:24.307 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-20T21:36:24.307 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi062/store.db/MANIFEST-000015 succeeded,manifest_file_number is 15, next_file_number is 37, last_sequence is 16465, log_number is 33,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 33 2024-04-20T21:36:24.307 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 33 2024-04-20T21:36:24.307 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 6e1ad0b0-7dc2-473c-b9e3-099eb8c1edd2 2024-04-20T21:36:24.307 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713648984039455, "job": 1, "event": "recovery_started", "wal_files": [33]} 2024-04-20T21:36:24.307 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #33 mode 2 2024-04-20T21:36:24.307 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713648984046791, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 38, "file_size": 914322, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 16466, "largest_seqno": 17784, "table_properties": {"data_size": 908788, "index_size": 3149, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1349, "raw_key_size": 12371, "raw_average_key_size": 23, "raw_value_size": 897714, "raw_average_value_size": 1713, "num_data_blocks": 143, "num_entries": 524, "num_filter_entries": 524, "num_deletions": 6, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1713648984, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "6e1ad0b0-7dc2-473c-b9e3-099eb8c1edd2", "db_session_id": "YYW1LLBSVT91MJFSFU4U", "orig_file_number": 38, "seqno_to_time_mapping": "N/A"}} 2024-04-20T21:36:24.307 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713648984046967, "job": 1, "event": "recovery_finished"} 2024-04-20T21:36:24.308 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: [db/version_set.cc:5047] Creating manifest 40 2024-04-20T21:36:24.308 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-20T21:36:24.308 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi062/store.db/000033.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-20T21:36:24.308 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55bc6baa4000 2024-04-20T21:36:24.308 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: DB pointer 0x55bc6ba8e000 2024-04-20T21:36:24.308 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-20T21:36:24.308 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-20T21:36:24.308 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: ** DB Stats ** 2024-04-20T21:36:24.308 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-20T21:36:24.308 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-20T21:36:24.308 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-20T21:36:24.308 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-20T21:36:24.308 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-20T21:36:24.308 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-20T21:36:24.308 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-20T21:36:24.309 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: 2024-04-20T21:36:24.309 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: ** Compaction Stats [default] ** 2024-04-20T21:36:24.309 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-20T21:36:24.309 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-20T21:36:24.309 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: L0 1/0 892.89 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 214.3 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-04-20T21:36:24.309 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: L6 1/0 7.76 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-04-20T21:36:24.309 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: Sum 2/0 8.63 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 214.3 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-04-20T21:36:24.309 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 214.3 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-04-20T21:36:24.309 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: 2024-04-20T21:36:24.309 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: ** Compaction Stats [default] ** 2024-04-20T21:36:24.309 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-20T21:36:24.309 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-20T21:36:24.309 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 214.3 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-04-20T21:36:24.309 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: 2024-04-20T21:36:24.309 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-20T21:36:24.309 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: 2024-04-20T21:36:24.310 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-20T21:36:24.310 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: Flush(GB): cumulative 0.001, interval 0.001 2024-04-20T21:36:24.310 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-20T21:36:24.310 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-20T21:36:24.310 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-20T21:36:24.310 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-20T21:36:24.310 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: Cumulative compaction: 0.00 GB write, 69.89 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-20T21:36:24.310 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: Interval compaction: 0.00 GB write, 69.89 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-20T21:36:24.310 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-20T21:36:24.310 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: Block cache BinnedLRUCache@0x55bc6b9b7090#2 capacity: 512.00 MB usage: 4.73 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.3e-05 secs_since: 0 2024-04-20T21:36:24.310 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,1.41 KB,0.000268221%) IndexBlock(1,3.33 KB,0.000634789%) Misc(1,0.00 KB,0%) 2024-04-20T21:36:24.310 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: 2024-04-20T21:36:24.310 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-20T21:36:24.311 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: starting mon.smithi062 rank 0 at public addrs [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] at bind addrs [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi062 fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 2024-04-20T21:36:24.311 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: mon.smithi062@-1(???) e2 preinit fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 2024-04-20T21:36:24.311 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: mon.smithi062@-1(???).mds e16 new map 2024-04-20T21:36:24.311 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: mon.smithi062@-1(???).mds e16 print_map 2024-04-20T21:36:24.311 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: e16 2024-04-20T21:36:24.311 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T21:36:24.311 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T21:36:24.311 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: legacy client fscid: 1 2024-04-20T21:36:24.311 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: 2024-04-20T21:36:24.311 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: Filesystem 'cephfs' (1) 2024-04-20T21:36:24.311 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: fs_name cephfs 2024-04-20T21:36:24.311 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: epoch 16 2024-04-20T21:36:24.311 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-20T21:36:24.311 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: created 2024-04-20T21:27:21.032337+0000 2024-04-20T21:36:24.312 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: modified 2024-04-20T21:31:14.906617+0000 2024-04-20T21:36:24.312 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: tableserver 0 2024-04-20T21:36:24.312 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: root 0 2024-04-20T21:36:24.312 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: session_timeout 60 2024-04-20T21:36:24.312 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: session_autoclose 300 2024-04-20T21:36:24.312 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: max_file_size 1099511627776 2024-04-20T21:36:24.312 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: max_xattr_size 65536 2024-04-20T21:36:24.312 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: required_client_features {} 2024-04-20T21:36:24.312 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: last_failure 0 2024-04-20T21:36:24.312 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: last_failure_osd_epoch 0 2024-04-20T21:36:24.312 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: snap_rstat 1 2024-04-20T21:36:24.312 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T21:36:24.312 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: max_mds 2 2024-04-20T21:36:24.312 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: in 0,1 2024-04-20T21:36:24.312 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: up {0=14458,1=24279} 2024-04-20T21:36:24.312 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: failed 2024-04-20T21:36:24.313 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: damaged 2024-04-20T21:36:24.313 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: stopped 2024-04-20T21:36:24.313 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: data_pools [3] 2024-04-20T21:36:24.313 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: metadata_pool 2 2024-04-20T21:36:24.313 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: inline_data enabled 2024-04-20T21:36:24.313 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: balancer 2024-04-20T21:36:24.313 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: bal_rank_mask -1 2024-04-20T21:36:24.313 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: standby_count_wanted 1 2024-04-20T21:36:24.313 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: qdb_cluster leader: 0 members: 2024-04-20T21:36:24.313 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: [mds.cephfs.smithi062.ryqblb{0:14458} state up:active seq 6 export targets 1 join_fscid=1 addr [v2:172.21.15.62:6826/1147127123,v1:172.21.15.62:6827/1147127123] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:36:24.313 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: [mds.cephfs.smithi062.ozizwi{0:14474} state up:standby-replay seq 3 join_fscid=1 addr [v2:172.21.15.62:6828/3494174293,v1:172.21.15.62:6829/3494174293] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:36:24.313 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: [mds.cephfs.smithi189.dvimer{1:24279} state up:active seq 5 export targets 0 join_fscid=1 addr [v2:172.21.15.189:6824/237262514,v1:172.21.15.189:6825/237262514] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:36:24.314 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: [mds.cephfs.smithi189.xbgspw{1:24291} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.189:6826/1495356258,v1:172.21.15.189:6827/1495356258] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:36:24.314 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: 2024-04-20T21:36:24.314 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout: 2024-04-20T21:36:24.314 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: mon.smithi062@-1(???).osd e47 crush map has features 3314933000852226048, adjusting msgr requires 2024-04-20T21:36:24.314 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: mon.smithi062@-1(???).osd e47 crush map has features 288514051259236352, adjusting msgr requires 2024-04-20T21:36:24.314 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: mon.smithi062@-1(???).osd e47 crush map has features 288514051259236352, adjusting msgr requires 2024-04-20T21:36:24.314 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: mon.smithi062@-1(???).osd e47 crush map has features 288514051259236352, adjusting msgr requires 2024-04-20T21:36:24.314 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: mon.smithi062@-1(???).paxosservice(auth 1..24) refresh upgraded, format 0 -> 3 2024-04-20T21:36:24.314 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 bash[76304]: 019eb14794852cc40f5e7beeaa6d021dc8a8cf60ed30bdb363900c0cd8f6e1d7 2024-04-20T21:36:24.314 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 systemd[1]: Started Ceph mon.smithi062 for 61ae3628-ff5b-11ee-bc93-c7b262605968. 2024-04-20T21:36:24.765 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mon metadata", "id": "smithi062"}]: dispatch 2024-04-20T21:36:24.765 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mon metadata", "id": "smithi189"}]: dispatch 2024-04-20T21:36:24.765 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:36:24.765 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: pgmap v146: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:36:24.765 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: mon.smithi062 calling monitor election 2024-04-20T21:36:24.765 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: mon.smithi062 is new leader, mons smithi062,smithi189 in quorum (ranks 0,1) 2024-04-20T21:36:25.034 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:24 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mon metadata", "id": "smithi062"}]: dispatch 2024-04-20T21:36:25.034 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:24 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mon metadata", "id": "smithi189"}]: dispatch 2024-04-20T21:36:25.034 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:24 smithi189 ceph-mon[21562]: from='mgr.14658 172.21.15.62:0/1937955399' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:36:25.034 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:24 smithi189 ceph-mon[21562]: pgmap v146: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:36:25.034 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:24 smithi189 ceph-mon[21562]: mon.smithi062 calling monitor election 2024-04-20T21:36:25.034 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:24 smithi189 ceph-mon[21562]: mon.smithi062 is new leader, mons smithi062,smithi189 in quorum (ranks 0,1) 2024-04-20T21:36:25.034 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:24 smithi189 ceph-mon[21562]: monmap epoch 2 2024-04-20T21:36:25.034 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:24 smithi189 ceph-mon[21562]: fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 2024-04-20T21:36:25.035 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:24 smithi189 ceph-mon[21562]: last_changed 2024-04-20T21:21:43.183410+0000 2024-04-20T21:36:25.035 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:24 smithi189 ceph-mon[21562]: created 2024-04-20T21:18:14.797594+0000 2024-04-20T21:36:25.035 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:24 smithi189 ceph-mon[21562]: min_mon_release 18 (reef) 2024-04-20T21:36:25.035 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:24 smithi189 ceph-mon[21562]: election_strategy: 1 2024-04-20T21:36:25.035 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:24 smithi189 ceph-mon[21562]: 0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.smithi062 2024-04-20T21:36:25.035 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:24 smithi189 ceph-mon[21562]: 1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.smithi189 2024-04-20T21:36:25.035 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:24 smithi189 ceph-mon[21562]: fsmap cephfs:2 {0=cephfs.smithi062.ryqblb=up:active,1=cephfs.smithi189.dvimer=up:active} 2 up:standby-replay 2024-04-20T21:36:25.035 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:24 smithi189 ceph-mon[21562]: osdmap e47: 6 total, 6 up, 6 in 2024-04-20T21:36:25.035 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:24 smithi189 ceph-mon[21562]: mgrmap e31: smithi062.rqxuxi(active, since 4m), standbys: smithi189.tawavz 2024-04-20T21:36:25.035 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:24 smithi189 ceph-mon[21562]: Health detail: HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-04-20T21:36:25.035 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:24 smithi189 ceph-mon[21562]: [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T21:36:25.035 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:24 smithi189 ceph-mon[21562]: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T21:36:25.035 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:24 smithi189 ceph-mon[21562]: from='mgr.14658 ' entity='' 2024-04-20T21:36:25.035 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:24 smithi189 ceph-mon[21562]: mgrmap e32: smithi062.rqxuxi(active, since 4m), standbys: smithi189.tawavz 2024-04-20T21:36:25.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: monmap epoch 2 2024-04-20T21:36:25.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 2024-04-20T21:36:25.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: last_changed 2024-04-20T21:21:43.183410+0000 2024-04-20T21:36:25.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: created 2024-04-20T21:18:14.797594+0000 2024-04-20T21:36:25.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: min_mon_release 18 (reef) 2024-04-20T21:36:25.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: election_strategy: 1 2024-04-20T21:36:25.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: 0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.smithi062 2024-04-20T21:36:25.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: 1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.smithi189 2024-04-20T21:36:25.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: fsmap cephfs:2 {0=cephfs.smithi062.ryqblb=up:active,1=cephfs.smithi189.dvimer=up:active} 2 up:standby-replay 2024-04-20T21:36:25.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: osdmap e47: 6 total, 6 up, 6 in 2024-04-20T21:36:25.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: mgrmap e31: smithi062.rqxuxi(active, since 4m), standbys: smithi189.tawavz 2024-04-20T21:36:25.037 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: Health detail: HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-04-20T21:36:25.037 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T21:36:25.037 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T21:36:25.037 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: from='mgr.14658 ' entity='' 2024-04-20T21:36:25.037 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:24 smithi062 ceph-mon[76322]: mgrmap e32: smithi062.rqxuxi(active, since 4m), standbys: smithi189.tawavz 2024-04-20T21:36:33.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:33 smithi189 ceph-mon[21562]: Standby manager daemon smithi189.tawavz restarted 2024-04-20T21:36:33.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:33 smithi189 ceph-mon[21562]: Standby manager daemon smithi189.tawavz started 2024-04-20T21:36:33.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:33 smithi189 ceph-mon[21562]: from='mgr.? 172.21.15.189:0/975725007' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi189.tawavz/crt"}]: dispatch 2024-04-20T21:36:33.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:33 smithi189 ceph-mon[21562]: from='mgr.? 172.21.15.189:0/975725007' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-20T21:36:33.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:33 smithi189 ceph-mon[21562]: from='mgr.? 172.21.15.189:0/975725007' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi189.tawavz/key"}]: dispatch 2024-04-20T21:36:33.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:33 smithi189 ceph-mon[21562]: from='mgr.? 172.21.15.189:0/975725007' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-20T21:36:34.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:33 smithi062 ceph-mon[76322]: Standby manager daemon smithi189.tawavz restarted 2024-04-20T21:36:34.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:33 smithi062 ceph-mon[76322]: Standby manager daemon smithi189.tawavz started 2024-04-20T21:36:34.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:33 smithi062 ceph-mon[76322]: from='mgr.? 172.21.15.189:0/975725007' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi189.tawavz/crt"}]: dispatch 2024-04-20T21:36:34.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:33 smithi062 ceph-mon[76322]: from='mgr.? 172.21.15.189:0/975725007' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-20T21:36:34.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:33 smithi062 ceph-mon[76322]: from='mgr.? 172.21.15.189:0/975725007' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi189.tawavz/key"}]: dispatch 2024-04-20T21:36:34.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:33 smithi062 ceph-mon[76322]: from='mgr.? 172.21.15.189:0/975725007' entity='mgr.smithi189.tawavz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-20T21:36:34.778 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:34 smithi062 ceph-mon[76322]: mgrmap e33: smithi062.rqxuxi(active, since 4m), standbys: smithi189.tawavz 2024-04-20T21:36:34.778 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:34 smithi062 ceph-mon[76322]: Active manager daemon smithi062.rqxuxi restarted 2024-04-20T21:36:34.778 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:34 smithi062 ceph-mon[76322]: Activating manager daemon smithi062.rqxuxi 2024-04-20T21:36:34.778 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:34 smithi062 ceph-mon[76322]: osdmap e48: 6 total, 6 up, 6 in 2024-04-20T21:36:34.778 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:34 smithi062 ceph-mon[76322]: mgrmap e34: smithi062.rqxuxi(active, starting, since 0.0193513s), standbys: smithi189.tawavz 2024-04-20T21:36:34.778 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:34 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mon metadata", "id": "smithi062"}]: dispatch 2024-04-20T21:36:34.778 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:34 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mon metadata", "id": "smithi189"}]: dispatch 2024-04-20T21:36:34.779 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:34 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi062.ryqblb"}]: dispatch 2024-04-20T21:36:34.779 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:34 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi062.ozizwi"}]: dispatch 2024-04-20T21:36:34.779 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:34 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi189.dvimer"}]: dispatch 2024-04-20T21:36:34.779 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:34 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi189.xbgspw"}]: dispatch 2024-04-20T21:36:34.779 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:34 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mgr metadata", "who": "smithi062.rqxuxi", "id": "smithi062.rqxuxi"}]: dispatch 2024-04-20T21:36:34.779 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:34 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mgr metadata", "who": "smithi189.tawavz", "id": "smithi189.tawavz"}]: dispatch 2024-04-20T21:36:34.779 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:34 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-20T21:36:34.779 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:34 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-20T21:36:34.779 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:34 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-20T21:36:34.779 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:34 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-20T21:36:34.779 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:34 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-20T21:36:34.779 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:34 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-20T21:36:34.779 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:34 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-20T21:36:34.779 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:34 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-20T21:36:34.779 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:34 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-20T21:36:34.780 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:34 smithi062 ceph-mon[76322]: Manager daemon smithi062.rqxuxi is now available 2024-04-20T21:36:34.780 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:34 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:36:34.780 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:34 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:36:34.780 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:34 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi062.rqxuxi/mirror_snapshot_schedule"}]: dispatch 2024-04-20T21:36:34.780 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:34 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi062.rqxuxi/trash_purge_schedule"}]: dispatch 2024-04-20T21:36:34.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:34 smithi189 ceph-mon[21562]: mgrmap e33: smithi062.rqxuxi(active, since 4m), standbys: smithi189.tawavz 2024-04-20T21:36:34.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:34 smithi189 ceph-mon[21562]: Active manager daemon smithi062.rqxuxi restarted 2024-04-20T21:36:34.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:34 smithi189 ceph-mon[21562]: Activating manager daemon smithi062.rqxuxi 2024-04-20T21:36:34.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:34 smithi189 ceph-mon[21562]: osdmap e48: 6 total, 6 up, 6 in 2024-04-20T21:36:34.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:34 smithi189 ceph-mon[21562]: mgrmap e34: smithi062.rqxuxi(active, starting, since 0.0193513s), standbys: smithi189.tawavz 2024-04-20T21:36:34.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:34 smithi189 ceph-mon[21562]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mon metadata", "id": "smithi062"}]: dispatch 2024-04-20T21:36:34.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:34 smithi189 ceph-mon[21562]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mon metadata", "id": "smithi189"}]: dispatch 2024-04-20T21:36:34.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:34 smithi189 ceph-mon[21562]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi062.ryqblb"}]: dispatch 2024-04-20T21:36:34.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:34 smithi189 ceph-mon[21562]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi062.ozizwi"}]: dispatch 2024-04-20T21:36:34.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:34 smithi189 ceph-mon[21562]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi189.dvimer"}]: dispatch 2024-04-20T21:36:34.989 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:34 smithi189 ceph-mon[21562]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi189.xbgspw"}]: dispatch 2024-04-20T21:36:34.989 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:34 smithi189 ceph-mon[21562]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mgr metadata", "who": "smithi062.rqxuxi", "id": "smithi062.rqxuxi"}]: dispatch 2024-04-20T21:36:34.989 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:34 smithi189 ceph-mon[21562]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mgr metadata", "who": "smithi189.tawavz", "id": "smithi189.tawavz"}]: dispatch 2024-04-20T21:36:34.989 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:34 smithi189 ceph-mon[21562]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-20T21:36:34.989 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:34 smithi189 ceph-mon[21562]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-20T21:36:34.989 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:34 smithi189 ceph-mon[21562]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-20T21:36:34.989 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:34 smithi189 ceph-mon[21562]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-20T21:36:34.989 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:34 smithi189 ceph-mon[21562]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-20T21:36:34.989 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:34 smithi189 ceph-mon[21562]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-20T21:36:34.989 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:34 smithi189 ceph-mon[21562]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-20T21:36:34.989 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:34 smithi189 ceph-mon[21562]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-20T21:36:34.989 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:34 smithi189 ceph-mon[21562]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-20T21:36:34.989 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:34 smithi189 ceph-mon[21562]: Manager daemon smithi062.rqxuxi is now available 2024-04-20T21:36:34.989 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:34 smithi189 ceph-mon[21562]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:36:34.989 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:34 smithi189 ceph-mon[21562]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:36:34.990 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:34 smithi189 ceph-mon[21562]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi062.rqxuxi/mirror_snapshot_schedule"}]: dispatch 2024-04-20T21:36:34.990 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:34 smithi189 ceph-mon[21562]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi062.rqxuxi/trash_purge_schedule"}]: dispatch 2024-04-20T21:36:36.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:36 smithi062 ceph-mon[76322]: mgrmap e35: smithi062.rqxuxi(active, since 1.02422s), standbys: smithi189.tawavz 2024-04-20T21:36:36.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:36 smithi062 ceph-mon[76322]: [20/Apr/2024:21:36:35] ENGINE Bus STARTING 2024-04-20T21:36:36.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:36 smithi062 ceph-mon[76322]: [20/Apr/2024:21:36:35] ENGINE Serving on https://172.21.15.62:7150 2024-04-20T21:36:36.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:36 smithi189 ceph-mon[21562]: mgrmap e35: smithi062.rqxuxi(active, since 1.02422s), standbys: smithi189.tawavz 2024-04-20T21:36:36.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:36 smithi189 ceph-mon[21562]: [20/Apr/2024:21:36:35] ENGINE Bus STARTING 2024-04-20T21:36:36.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:36 smithi189 ceph-mon[21562]: [20/Apr/2024:21:36:35] ENGINE Serving on https://172.21.15.62:7150 2024-04-20T21:36:37.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:37 smithi062 ceph-mon[76322]: [20/Apr/2024:21:36:35] ENGINE Serving on http://172.21.15.62:8765 2024-04-20T21:36:37.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:37 smithi062 ceph-mon[76322]: [20/Apr/2024:21:36:35] ENGINE Bus STARTED 2024-04-20T21:36:37.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:37 smithi062 ceph-mon[76322]: pgmap v4: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:36:37.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:37 smithi062 ceph-mon[76322]: mgrmap e36: smithi062.rqxuxi(active, since 2s), standbys: smithi189.tawavz 2024-04-20T21:36:37.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:37 smithi189 ceph-mon[21562]: [20/Apr/2024:21:36:35] ENGINE Serving on http://172.21.15.62:8765 2024-04-20T21:36:37.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:37 smithi189 ceph-mon[21562]: [20/Apr/2024:21:36:35] ENGINE Bus STARTED 2024-04-20T21:36:37.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:37 smithi189 ceph-mon[21562]: pgmap v4: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:36:37.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:37 smithi189 ceph-mon[21562]: mgrmap e36: smithi062.rqxuxi(active, since 2s), standbys: smithi189.tawavz 2024-04-20T21:36:39.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:39 smithi062 ceph-mon[76322]: pgmap v5: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:36:39.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:39 smithi062 ceph-mon[76322]: mgrmap e37: smithi062.rqxuxi(active, since 4s), standbys: smithi189.tawavz 2024-04-20T21:36:39.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:39 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:36:39.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:39 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:36:39.736 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:39 smithi189 ceph-mon[21562]: pgmap v5: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:36:39.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:39 smithi189 ceph-mon[21562]: mgrmap e37: smithi062.rqxuxi(active, since 4s), standbys: smithi189.tawavz 2024-04-20T21:36:39.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:39 smithi189 ceph-mon[21562]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:36:39.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:39 smithi189 ceph-mon[21562]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:36:41.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:41 smithi062 ceph-mon[76322]: pgmap v6: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:36:41.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:41 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:36:41.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:41 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:36:41.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:41 smithi189 ceph-mon[21562]: pgmap v6: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:36:41.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:41 smithi189 ceph-mon[21562]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:36:41.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:41 smithi189 ceph-mon[21562]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:36:43.192 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:42 smithi189 ceph-mon[21562]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:36:43.193 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:42 smithi189 ceph-mon[21562]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:36:43.193 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:42 smithi189 ceph-mon[21562]: pgmap v7: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 31 KiB/s rd, 0 B/s wr, 19 op/s 2024-04-20T21:36:43.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:42 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:36:43.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:42 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:36:43.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:42 smithi062 ceph-mon[76322]: pgmap v7: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 31 KiB/s rd, 0 B/s wr, 19 op/s 2024-04-20T21:36:45.363 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:45 smithi189 ceph-mon[21562]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:36:45.363 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:45 smithi189 ceph-mon[21562]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:36:45.363 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:45 smithi189 ceph-mon[21562]: pgmap v8: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 24 KiB/s rd, 0 B/s wr, 15 op/s 2024-04-20T21:36:45.491 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:45 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:36:45.491 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:45 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:36:45.491 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:45 smithi062 ceph-mon[76322]: pgmap v8: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 24 KiB/s rd, 0 B/s wr, 15 op/s 2024-04-20T21:36:46.452 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:46 smithi189 ceph-mon[21562]: fsmap cephfs:2 {0=cephfs.smithi062.ryqblb=up:active,1=cephfs.smithi189.dvimer=up:active} 2 up:standby-replay 2024-04-20T21:36:46.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:46 smithi062 ceph-mon[76322]: fsmap cephfs:2 {0=cephfs.smithi062.ryqblb=up:active,1=cephfs.smithi189.dvimer=up:active} 2 up:standby-replay 2024-04-20T21:36:47.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:47.693+0000 7f950cc00700 1 -- 172.21.15.62:0/1026208411 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9508100530 msgr2=0x7f9508100900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:36:47.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:47.693+0000 7f950cc00700 1 --2- 172.21.15.62:0/1026208411 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9508100530 0x7f9508100900 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f94f800b0a0 tx=0x7f94f800b3b0 comp rx=0 tx=0).stop 2024-04-20T21:36:47.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:47.694+0000 7f950cc00700 1 -- 172.21.15.62:0/1026208411 shutdown_connections 2024-04-20T21:36:47.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:47.694+0000 7f950cc00700 1 --2- 172.21.15.62:0/1026208411 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9508100ed0 0x7f9508104ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:47.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:47.694+0000 7f950cc00700 1 --2- 172.21.15.62:0/1026208411 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9508100530 0x7f9508100900 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:47.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:47.694+0000 7f950cc00700 1 -- 172.21.15.62:0/1026208411 >> 172.21.15.62:0/1026208411 conn(0x7f95080fbe10 msgr2=0x7f95080fe220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:36:47.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:47.694+0000 7f950cc00700 1 -- 172.21.15.62:0/1026208411 shutdown_connections 2024-04-20T21:36:47.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:47.694+0000 7f950cc00700 1 -- 172.21.15.62:0/1026208411 wait complete. 2024-04-20T21:36:47.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:47.695+0000 7f950cc00700 1 Processor -- start 2024-04-20T21:36:47.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:47.695+0000 7f950cc00700 1 -- start start 2024-04-20T21:36:47.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:47.696+0000 7f950cc00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9508100530 0x7f9508198190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:36:47.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:47.696+0000 7f950cc00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9508100ed0 0x7f95081986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:36:47.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:47.696+0000 7f950cc00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9508198db0 con 0x7f9508100530 2024-04-20T21:36:47.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:47.696+0000 7f950cc00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f950819cb40 con 0x7f9508100ed0 2024-04-20T21:36:47.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:47.696+0000 7f9505e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9508100530 0x7f9508198190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:36:47.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:47.696+0000 7f9505e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9508100530 0x7f9508198190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45296/0 (socket says 172.21.15.62:45296) 2024-04-20T21:36:47.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:47.696+0000 7f9505e00700 1 -- 172.21.15.62:0/547353800 learned_addr learned my addr 172.21.15.62:0/547353800 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:36:47.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:47.697+0000 7f9505e00700 1 -- 172.21.15.62:0/547353800 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9508100ed0 msgr2=0x7f95081986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:36:47.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:47.697+0000 7f9505e00700 1 --2- 172.21.15.62:0/547353800 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9508100ed0 0x7f95081986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:47.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:47.697+0000 7f9505e00700 1 -- 172.21.15.62:0/547353800 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94f8009d00 con 0x7f9508100530 2024-04-20T21:36:47.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:47.697+0000 7f9505e00700 1 --2- 172.21.15.62:0/547353800 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9508100530 0x7f9508198190 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f94f800b070 tx=0x7f94f800cdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:36:47.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:47.697+0000 7f94ff400700 1 -- 172.21.15.62:0/547353800 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f94f800e040 con 0x7f9508100530 2024-04-20T21:36:47.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:47.697+0000 7f94ff400700 1 -- 172.21.15.62:0/547353800 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f94f8003e60 con 0x7f9508100530 2024-04-20T21:36:47.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:47.698+0000 7f950cc00700 1 -- 172.21.15.62:0/547353800 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f950819cdc0 con 0x7f9508100530 2024-04-20T21:36:47.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:47.698+0000 7f94ff400700 1 -- 172.21.15.62:0/547353800 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f94f801d910 con 0x7f9508100530 2024-04-20T21:36:47.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:47.698+0000 7f950cc00700 1 -- 172.21.15.62:0/547353800 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f950819d2b0 con 0x7f9508100530 2024-04-20T21:36:47.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:47.699+0000 7f950cc00700 1 -- 172.21.15.62:0/547353800 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9508104620 con 0x7f9508100530 2024-04-20T21:36:47.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:47.701+0000 7f94ff400700 1 -- 172.21.15.62:0/547353800 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 37) v1 ==== 92847+0+0 (secure 0 0 0) 0x7f94f8004820 con 0x7f9508100530 2024-04-20T21:36:47.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:47.701+0000 7f94ff400700 1 --2- 172.21.15.62:0/547353800 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f94ec06edf0 0x7f94ec0712a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:36:47.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:47.702+0000 7f94ff400700 1 -- 172.21.15.62:0/547353800 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f94f8090a30 con 0x7f9508100530 2024-04-20T21:36:47.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:47.702+0000 7f94fd400700 1 --2- 172.21.15.62:0/547353800 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f94ec06edf0 0x7f94ec0712a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:36:47.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:47.703+0000 7f94fd400700 1 --2- 172.21.15.62:0/547353800 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f94ec06edf0 0x7f94ec0712a0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f95081997b0 tx=0x7f94f0006d20 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:36:47.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:47.708+0000 7f94ff400700 1 -- 172.21.15.62:0/547353800 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f94f80595d0 con 0x7f9508100530 2024-04-20T21:36:47.874 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:47 smithi062 ceph-mon[76322]: pgmap v9: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 21 KiB/s rd, 0 B/s wr, 14 op/s 2024-04-20T21:36:47.874 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:47 smithi062 ceph-mon[76322]: Detected new or changed devices on smithi189 2024-04-20T21:36:47.874 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:47 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:36:47.874 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:47 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:36:47.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:47.972+0000 7f950cc00700 1 -- 172.21.15.62:0/547353800 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f950802cca0 con 0x7f94ec06edf0 2024-04-20T21:36:47.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:47.975+0000 7f94ff400700 1 -- 172.21.15.62:0/547353800 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+337 (secure 0 0 0) 0x7f950802cca0 con 0x7f94ec06edf0 2024-04-20T21:36:47.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:47.977+0000 7f950cc00700 1 -- 172.21.15.62:0/547353800 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f94ec06edf0 msgr2=0x7f94ec0712a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:36:47.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:47.978+0000 7f950cc00700 1 --2- 172.21.15.62:0/547353800 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f94ec06edf0 0x7f94ec0712a0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f95081997b0 tx=0x7f94f0006d20 comp rx=0 tx=0).stop 2024-04-20T21:36:47.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:47.978+0000 7f950cc00700 1 -- 172.21.15.62:0/547353800 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9508100530 msgr2=0x7f9508198190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:36:47.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:47.978+0000 7f950cc00700 1 --2- 172.21.15.62:0/547353800 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9508100530 0x7f9508198190 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f94f800b070 tx=0x7f94f800cdd0 comp rx=0 tx=0).stop 2024-04-20T21:36:47.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:47.978+0000 7f950cc00700 1 -- 172.21.15.62:0/547353800 shutdown_connections 2024-04-20T21:36:47.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:47.978+0000 7f950cc00700 1 --2- 172.21.15.62:0/547353800 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f94ec06edf0 0x7f94ec0712a0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:47.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:47.978+0000 7f950cc00700 1 --2- 172.21.15.62:0/547353800 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9508100530 0x7f9508198190 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:47.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:47.978+0000 7f950cc00700 1 --2- 172.21.15.62:0/547353800 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9508100ed0 0x7f95081986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:47.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:47.978+0000 7f950cc00700 1 -- 172.21.15.62:0/547353800 >> 172.21.15.62:0/547353800 conn(0x7f95080fbe10 msgr2=0x7f95080fd540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:36:47.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:47.978+0000 7f950cc00700 1 -- 172.21.15.62:0/547353800 shutdown_connections 2024-04-20T21:36:47.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:47.978+0000 7f950cc00700 1 -- 172.21.15.62:0/547353800 wait complete. 2024-04-20T21:36:47.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:47 smithi189 ceph-mon[21562]: pgmap v9: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 21 KiB/s rd, 0 B/s wr, 14 op/s 2024-04-20T21:36:47.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:47 smithi189 ceph-mon[21562]: Detected new or changed devices on smithi189 2024-04-20T21:36:47.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:47 smithi189 ceph-mon[21562]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:36:47.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:47 smithi189 ceph-mon[21562]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:36:47.993 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T21:36:48.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:48.133+0000 7f5d9f800700 1 -- 172.21.15.62:0/3501813909 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d981004e0 msgr2=0x7f5d981008b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:36:48.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:48.133+0000 7f5d9f800700 1 --2- 172.21.15.62:0/3501813909 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d981004e0 0x7f5d981008b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f5d8c009910 tx=0x7f5d8c009c20 comp rx=0 tx=0).stop 2024-04-20T21:36:48.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:48.134+0000 7f5d9f800700 1 -- 172.21.15.62:0/3501813909 shutdown_connections 2024-04-20T21:36:48.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:48.134+0000 7f5d9f800700 1 --2- 172.21.15.62:0/3501813909 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d98100e80 0x7f5d98104e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:48.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:48.134+0000 7f5d9f800700 1 --2- 172.21.15.62:0/3501813909 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d981004e0 0x7f5d981008b0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:48.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:48.134+0000 7f5d9f800700 1 -- 172.21.15.62:0/3501813909 >> 172.21.15.62:0/3501813909 conn(0x7f5d980fbda0 msgr2=0x7f5d980fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:36:48.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:48.134+0000 7f5d9f800700 1 -- 172.21.15.62:0/3501813909 shutdown_connections 2024-04-20T21:36:48.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:48.134+0000 7f5d9f800700 1 -- 172.21.15.62:0/3501813909 wait complete. 2024-04-20T21:36:48.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:48.136+0000 7f5d9f800700 1 Processor -- start 2024-04-20T21:36:48.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:48.136+0000 7f5d9f800700 1 -- start start 2024-04-20T21:36:48.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:48.136+0000 7f5d9f800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d981004e0 0x7f5d98198210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:36:48.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:48.136+0000 7f5d9f800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d98100e80 0x7f5d98198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:36:48.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:48.136+0000 7f5d9f800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d98198e30 con 0x7f5d981004e0 2024-04-20T21:36:48.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:48.136+0000 7f5d9f800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d9819cbc0 con 0x7f5d98100e80 2024-04-20T21:36:48.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:48.137+0000 7f5d9ce00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d981004e0 0x7f5d98198210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:36:48.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:48.137+0000 7f5d9ce00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d981004e0 0x7f5d98198210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45318/0 (socket says 172.21.15.62:45318) 2024-04-20T21:36:48.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:48.137+0000 7f5d9ce00700 1 -- 172.21.15.62:0/159219016 learned_addr learned my addr 172.21.15.62:0/159219016 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:36:48.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:48.137+0000 7f5d97e00700 1 --2- 172.21.15.62:0/159219016 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d98100e80 0x7f5d98198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:36:48.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:48.137+0000 7f5d9ce00700 1 -- 172.21.15.62:0/159219016 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d98100e80 msgr2=0x7f5d98198750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:36:48.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:48.137+0000 7f5d9ce00700 1 --2- 172.21.15.62:0/159219016 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d98100e80 0x7f5d98198750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:48.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:48.137+0000 7f5d9ce00700 1 -- 172.21.15.62:0/159219016 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d8c0095f0 con 0x7f5d981004e0 2024-04-20T21:36:48.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:48.138+0000 7f5d9ce00700 1 --2- 172.21.15.62:0/159219016 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d981004e0 0x7f5d98198210 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5d8c006010 tx=0x7f5d8c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:36:48.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:48.138+0000 7f5d95600700 1 -- 172.21.15.62:0/159219016 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5d8c01d070 con 0x7f5d981004e0 2024-04-20T21:36:48.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:48.138+0000 7f5d95600700 1 -- 172.21.15.62:0/159219016 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5d8c004fb0 con 0x7f5d981004e0 2024-04-20T21:36:48.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:48.138+0000 7f5d9f800700 1 -- 172.21.15.62:0/159219016 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d9819ce40 con 0x7f5d981004e0 2024-04-20T21:36:48.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:48.138+0000 7f5d9f800700 1 -- 172.21.15.62:0/159219016 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d9819d330 con 0x7f5d981004e0 2024-04-20T21:36:48.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:48.139+0000 7f5d95600700 1 -- 172.21.15.62:0/159219016 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5d8c022ea0 con 0x7f5d981004e0 2024-04-20T21:36:48.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:48.140+0000 7f5d9f800700 1 -- 172.21.15.62:0/159219016 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d981045d0 con 0x7f5d981004e0 2024-04-20T21:36:48.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:48.141+0000 7f5d95600700 1 -- 172.21.15.62:0/159219016 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 37) v1 ==== 92847+0+0 (secure 0 0 0) 0x7f5d8c022470 con 0x7f5d981004e0 2024-04-20T21:36:48.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:48.142+0000 7f5d95600700 1 --2- 172.21.15.62:0/159219016 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5d8006ee00 0x7f5d800712b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:36:48.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:48.142+0000 7f5d95600700 1 -- 172.21.15.62:0/159219016 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f5d8c090690 con 0x7f5d981004e0 2024-04-20T21:36:48.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:48.142+0000 7f5d97e00700 1 --2- 172.21.15.62:0/159219016 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5d8006ee00 0x7f5d800712b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:36:48.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:48.143+0000 7f5d97e00700 1 --2- 172.21.15.62:0/159219016 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5d8006ee00 0x7f5d800712b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f5d84006fd0 tx=0x7f5d84008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:36:48.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:48.151+0000 7f5d95600700 1 -- 172.21.15.62:0/159219016 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f5d8c0592e0 con 0x7f5d981004e0 2024-04-20T21:36:48.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:48.428+0000 7f5d9f800700 1 -- 172.21.15.62:0/159219016 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5d981995e0 con 0x7f5d8006ee00 2024-04-20T21:36:48.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:48.430+0000 7f5d95600700 1 -- 172.21.15.62:0/159219016 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+337 (secure 0 0 0) 0x7f5d981995e0 con 0x7f5d8006ee00 2024-04-20T21:36:48.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:48.432+0000 7f5d9f800700 1 -- 172.21.15.62:0/159219016 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5d8006ee00 msgr2=0x7f5d800712b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:36:48.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:48.432+0000 7f5d9f800700 1 --2- 172.21.15.62:0/159219016 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5d8006ee00 0x7f5d800712b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f5d84006fd0 tx=0x7f5d84008040 comp rx=0 tx=0).stop 2024-04-20T21:36:48.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:48.432+0000 7f5d9f800700 1 -- 172.21.15.62:0/159219016 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d981004e0 msgr2=0x7f5d98198210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:36:48.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:48.432+0000 7f5d9f800700 1 --2- 172.21.15.62:0/159219016 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d981004e0 0x7f5d98198210 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5d8c006010 tx=0x7f5d8c00be20 comp rx=0 tx=0).stop 2024-04-20T21:36:48.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:48.433+0000 7f5d9f800700 1 -- 172.21.15.62:0/159219016 shutdown_connections 2024-04-20T21:36:48.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:48.433+0000 7f5d9f800700 1 --2- 172.21.15.62:0/159219016 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5d8006ee00 0x7f5d800712b0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:48.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:48.433+0000 7f5d9f800700 1 --2- 172.21.15.62:0/159219016 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d981004e0 0x7f5d98198210 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:48.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:48.433+0000 7f5d9f800700 1 --2- 172.21.15.62:0/159219016 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d98100e80 0x7f5d98198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:48.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:48.433+0000 7f5d9f800700 1 -- 172.21.15.62:0/159219016 >> 172.21.15.62:0/159219016 conn(0x7f5d980fbda0 msgr2=0x7f5d980fd460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:36:48.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:48.433+0000 7f5d9f800700 1 -- 172.21.15.62:0/159219016 shutdown_connections 2024-04-20T21:36:48.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:48.433+0000 7f5d9f800700 1 -- 172.21.15.62:0/159219016 wait complete. 2024-04-20T21:36:48.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:48.582+0000 7ff677a00700 1 -- 172.21.15.62:0/196238611 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff670102470 msgr2=0x7ff6701028e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:36:48.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:48.582+0000 7ff677a00700 1 --2- 172.21.15.62:0/196238611 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff670102470 0x7ff6701028e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7ff65c009910 tx=0x7ff65c009c20 comp rx=0 tx=0).stop 2024-04-20T21:36:48.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:48.582+0000 7ff677a00700 1 -- 172.21.15.62:0/196238611 shutdown_connections 2024-04-20T21:36:48.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:48.582+0000 7ff677a00700 1 --2- 172.21.15.62:0/196238611 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff670102470 0x7ff6701028e0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:48.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:48.582+0000 7ff677a00700 1 --2- 172.21.15.62:0/196238611 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff670108470 0x7ff670108840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:48.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:48.582+0000 7ff677a00700 1 -- 172.21.15.62:0/196238611 >> 172.21.15.62:0/196238611 conn(0x7ff6700fdf90 msgr2=0x7ff6701003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:36:48.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:48.582+0000 7ff677a00700 1 -- 172.21.15.62:0/196238611 shutdown_connections 2024-04-20T21:36:48.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:48.582+0000 7ff677a00700 1 -- 172.21.15.62:0/196238611 wait complete. 2024-04-20T21:36:48.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:48.583+0000 7ff677a00700 1 Processor -- start 2024-04-20T21:36:48.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:48.583+0000 7ff677a00700 1 -- start start 2024-04-20T21:36:48.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:48.584+0000 7ff677a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff670102470 0x7ff670198290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:36:48.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:48.584+0000 7ff677a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff670108470 0x7ff6701987d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:36:48.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:48.584+0000 7ff677a00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff670198eb0 con 0x7ff670102470 2024-04-20T21:36:48.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:48.584+0000 7ff677a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff67019cc40 con 0x7ff670108470 2024-04-20T21:36:48.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:48.584+0000 7ff675000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff670102470 0x7ff670198290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:36:48.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:48.584+0000 7ff675000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff670102470 0x7ff670198290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45338/0 (socket says 172.21.15.62:45338) 2024-04-20T21:36:48.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:48.584+0000 7ff675000700 1 -- 172.21.15.62:0/2439875302 learned_addr learned my addr 172.21.15.62:0/2439875302 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:36:48.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:48.585+0000 7ff66fe00700 1 --2- 172.21.15.62:0/2439875302 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff670108470 0x7ff6701987d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:36:48.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:48.585+0000 7ff675000700 1 -- 172.21.15.62:0/2439875302 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff670108470 msgr2=0x7ff6701987d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:36:48.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:48.585+0000 7ff675000700 1 --2- 172.21.15.62:0/2439875302 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff670108470 0x7ff6701987d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:48.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:48.585+0000 7ff675000700 1 -- 172.21.15.62:0/2439875302 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff65c0095f0 con 0x7ff670102470 2024-04-20T21:36:48.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:48.585+0000 7ff675000700 1 --2- 172.21.15.62:0/2439875302 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff670102470 0x7ff670198290 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7ff66400c8f0 tx=0x7ff66400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:36:48.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:48.586+0000 7ff66d600700 1 -- 172.21.15.62:0/2439875302 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff664007a70 con 0x7ff670102470 2024-04-20T21:36:48.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:48.586+0000 7ff677a00700 1 -- 172.21.15.62:0/2439875302 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff67019cf20 con 0x7ff670102470 2024-04-20T21:36:48.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:48.586+0000 7ff66d600700 1 -- 172.21.15.62:0/2439875302 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff664004cf0 con 0x7ff670102470 2024-04-20T21:36:48.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:48.586+0000 7ff66d600700 1 -- 172.21.15.62:0/2439875302 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff664005610 con 0x7ff670102470 2024-04-20T21:36:48.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:48.586+0000 7ff677a00700 1 -- 172.21.15.62:0/2439875302 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff67019d470 con 0x7ff670102470 2024-04-20T21:36:48.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:48.587+0000 7ff677a00700 1 -- 172.21.15.62:0/2439875302 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff67010a970 con 0x7ff670102470 2024-04-20T21:36:48.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:48.588+0000 7ff66d600700 1 -- 172.21.15.62:0/2439875302 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 37) v1 ==== 92847+0+0 (secure 0 0 0) 0x7ff664004750 con 0x7ff670102470 2024-04-20T21:36:48.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:48.589+0000 7ff66d600700 1 --2- 172.21.15.62:0/2439875302 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff65806edf0 0x7ff6580712a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:36:48.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:48.589+0000 7ff66d600700 1 -- 172.21.15.62:0/2439875302 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6328+0+0 (secure 0 0 0) 0x7ff664013070 con 0x7ff670102470 2024-04-20T21:36:48.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:48.589+0000 7ff66fe00700 1 --2- 172.21.15.62:0/2439875302 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff65806edf0 0x7ff6580712a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:36:48.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:48.590+0000 7ff66fe00700 1 --2- 172.21.15.62:0/2439875302 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff65806edf0 0x7ff6580712a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ff6701998b0 tx=0x7ff65c01a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:36:48.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:48.595+0000 7ff66d600700 1 -- 172.21.15.62:0/2439875302 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7ff664057a40 con 0x7ff670102470 2024-04-20T21:36:48.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:48.840+0000 7ff677a00700 1 -- 172.21.15.62:0/2439875302 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7ff670199700 con 0x7ff65806edf0 2024-04-20T21:36:48.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:48.852+0000 7ff66d600700 1 -- 172.21.15.62:0/2439875302 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7ff670199700 con 0x7ff65806edf0 2024-04-20T21:36:48.851 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T21:36:48.851 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (2m) 6s ago 16m 14.4M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T21:36:48.851 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (16m) 6s ago 16m 8338k - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T21:36:48.851 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (15m) 9s ago 15m 8162k - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T21:36:48.851 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (16m) 6s ago 16m 6656k - 18.2.0 dc2bc1663786 36e10ef5be39 2024-04-20T21:36:48.851 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (15m) 9s ago 15m 6665k - 18.2.0 dc2bc1663786 4101f9953e3d 2024-04-20T21:36:48.851 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (2m) 6s ago 16m 74.3M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T21:36:48.851 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (9m) 6s ago 9m 65.4M - 18.2.0 dc2bc1663786 5e6460aa6159 2024-04-20T21:36:48.851 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (9m) 6s ago 9m 132M - 18.2.0 dc2bc1663786 e65f2267544e 2024-04-20T21:36:48.852 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (9m) 9s ago 9m 117M - 18.2.0 dc2bc1663786 fb2a8a223db8 2024-04-20T21:36:48.852 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (9m) 9s ago 9m 55.9M - 18.2.0 dc2bc1663786 f2eec2ee8951 2024-04-20T21:36:48.852 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (5m) 6s ago 18m 476M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T21:36:48.852 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (4m) 9s ago 15m 448M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T21:36:48.852 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (25s) 6s ago 18m 29.3M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 019eb1479485 2024-04-20T21:36:48.852 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (15m) 9s ago 15m 49.9M 2048M 18.2.0 dc2bc1663786 3735c8e5db9a 2024-04-20T21:36:48.852 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (3m) 6s ago 16m 7512k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T21:36:48.852 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (3m) 9s ago 15m 7289k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T21:36:48.852 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (14m) 6s ago 14m 218M 1216M 18.2.0 dc2bc1663786 3f315c3afc1f 2024-04-20T21:36:48.852 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (13m) 6s ago 13m 210M 1216M 18.2.0 dc2bc1663786 44e4d07416ad 2024-04-20T21:36:48.852 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (12m) 6s ago 12m 173M 1216M 18.2.0 dc2bc1663786 31c5d9586e98 2024-04-20T21:36:48.852 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (12m) 9s ago 12m 241M 2240M 18.2.0 dc2bc1663786 611c1e242693 2024-04-20T21:36:48.852 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (11m) 9s ago 11m 219M 2240M 18.2.0 dc2bc1663786 6c8fce91d45d 2024-04-20T21:36:48.852 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (11m) 9s ago 11m 205M 2240M 18.2.0 dc2bc1663786 8e4574bae8db 2024-04-20T21:36:48.852 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (3m) 6s ago 15m 71.4M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T21:36:48.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:48.855+0000 7ff677a00700 1 -- 172.21.15.62:0/2439875302 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff65806edf0 msgr2=0x7ff6580712a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:36:48.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:48.855+0000 7ff677a00700 1 --2- 172.21.15.62:0/2439875302 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff65806edf0 0x7ff6580712a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ff6701998b0 tx=0x7ff65c01a040 comp rx=0 tx=0).stop 2024-04-20T21:36:48.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:48.855+0000 7ff677a00700 1 -- 172.21.15.62:0/2439875302 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff670102470 msgr2=0x7ff670198290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:36:48.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:48.855+0000 7ff677a00700 1 --2- 172.21.15.62:0/2439875302 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff670102470 0x7ff670198290 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7ff66400c8f0 tx=0x7ff66400ccb0 comp rx=0 tx=0).stop 2024-04-20T21:36:48.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:48.856+0000 7ff677a00700 1 -- 172.21.15.62:0/2439875302 shutdown_connections 2024-04-20T21:36:48.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:48.856+0000 7ff677a00700 1 --2- 172.21.15.62:0/2439875302 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff65806edf0 0x7ff6580712a0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:48.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:48.856+0000 7ff677a00700 1 --2- 172.21.15.62:0/2439875302 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff670102470 0x7ff670198290 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:48.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:48.856+0000 7ff677a00700 1 --2- 172.21.15.62:0/2439875302 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff670108470 0x7ff6701987d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:48.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:48.856+0000 7ff677a00700 1 -- 172.21.15.62:0/2439875302 >> 172.21.15.62:0/2439875302 conn(0x7ff6700fdf90 msgr2=0x7ff6700ff3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:36:48.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:48.856+0000 7ff677a00700 1 -- 172.21.15.62:0/2439875302 shutdown_connections 2024-04-20T21:36:48.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:48.856+0000 7ff677a00700 1 -- 172.21.15.62:0/2439875302 wait complete. 2024-04-20T21:36:49.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.008+0000 7f6d19800700 1 -- 172.21.15.62:0/3746877482 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d14106b80 msgr2=0x7f6d14106ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:36:49.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.008+0000 7f6d19800700 1 --2- 172.21.15.62:0/3746877482 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d14106b80 0x7f6d14106ff0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f6d00009910 tx=0x7f6d00009c20 comp rx=0 tx=0).stop 2024-04-20T21:36:49.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.008+0000 7f6d19800700 1 -- 172.21.15.62:0/3746877482 shutdown_connections 2024-04-20T21:36:49.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.008+0000 7f6d19800700 1 --2- 172.21.15.62:0/3746877482 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d14106b80 0x7f6d14106ff0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:49.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.008+0000 7f6d19800700 1 --2- 172.21.15.62:0/3746877482 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6d1410c8f0 0x7f6d1410ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:49.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.008+0000 7f6d19800700 1 -- 172.21.15.62:0/3746877482 >> 172.21.15.62:0/3746877482 conn(0x7f6d14076060 msgr2=0x7f6d14078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:36:49.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.009+0000 7f6d19800700 1 -- 172.21.15.62:0/3746877482 shutdown_connections 2024-04-20T21:36:49.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.009+0000 7f6d19800700 1 -- 172.21.15.62:0/3746877482 wait complete. 2024-04-20T21:36:49.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.010+0000 7f6d19800700 1 Processor -- start 2024-04-20T21:36:49.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.010+0000 7f6d19800700 1 -- start start 2024-04-20T21:36:49.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.010+0000 7f6d19800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d14106b80 0x7f6d1419c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:36:49.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.010+0000 7f6d19800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6d1410c8f0 0x7f6d1419c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:36:49.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.010+0000 7f6d19800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d1419d060 con 0x7f6d14106b80 2024-04-20T21:36:49.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.010+0000 7f6d19800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d141a0df0 con 0x7f6d1410c8f0 2024-04-20T21:36:49.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.010+0000 7f6d12a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d14106b80 0x7f6d1419c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:36:49.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.011+0000 7f6d12a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d14106b80 0x7f6d1419c440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45358/0 (socket says 172.21.15.62:45358) 2024-04-20T21:36:49.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.011+0000 7f6d12a00700 1 -- 172.21.15.62:0/3651294019 learned_addr learned my addr 172.21.15.62:0/3651294019 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:36:49.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.011+0000 7f6d12a00700 1 -- 172.21.15.62:0/3651294019 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6d1410c8f0 msgr2=0x7f6d1419c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:36:49.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.011+0000 7f6d12a00700 1 --2- 172.21.15.62:0/3651294019 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6d1410c8f0 0x7f6d1419c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:49.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.011+0000 7f6d12a00700 1 -- 172.21.15.62:0/3651294019 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d000095f0 con 0x7f6d14106b80 2024-04-20T21:36:49.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.011+0000 7f6d12a00700 1 --2- 172.21.15.62:0/3651294019 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d14106b80 0x7f6d1419c440 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6d0400c8a0 tx=0x7f6d0400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:36:49.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.011+0000 7f6d0b400700 1 -- 172.21.15.62:0/3651294019 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6d0400cea0 con 0x7f6d14106b80 2024-04-20T21:36:49.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.012+0000 7f6d0b400700 1 -- 172.21.15.62:0/3651294019 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6d04004830 con 0x7f6d14106b80 2024-04-20T21:36:49.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.012+0000 7f6d19800700 1 -- 172.21.15.62:0/3651294019 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d141a10d0 con 0x7f6d14106b80 2024-04-20T21:36:49.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.012+0000 7f6d0b400700 1 -- 172.21.15.62:0/3651294019 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6d040055b0 con 0x7f6d14106b80 2024-04-20T21:36:49.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.012+0000 7f6d19800700 1 -- 172.21.15.62:0/3651294019 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d141a1620 con 0x7f6d14106b80 2024-04-20T21:36:49.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.013+0000 7f6d19800700 1 -- 172.21.15.62:0/3651294019 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d1410ecc0 con 0x7f6d14106b80 2024-04-20T21:36:49.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.014+0000 7f6d0b400700 1 -- 172.21.15.62:0/3651294019 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 37) v1 ==== 92847+0+0 (secure 0 0 0) 0x7f6d040049a0 con 0x7f6d14106b80 2024-04-20T21:36:49.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.015+0000 7f6d0b400700 1 --2- 172.21.15.62:0/3651294019 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6cf806edf0 0x7f6cf80712a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:36:49.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.015+0000 7f6d0b400700 1 -- 172.21.15.62:0/3651294019 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f6d0408ecc0 con 0x7f6d14106b80 2024-04-20T21:36:49.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.015+0000 7f6d12000700 1 --2- 172.21.15.62:0/3651294019 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6cf806edf0 0x7f6cf80712a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:36:49.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.016+0000 7f6d12000700 1 --2- 172.21.15.62:0/3651294019 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6cf806edf0 0x7f6cf80712a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f6d00006010 tx=0x7f6d0000b540 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:36:49.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.020+0000 7f6d0b400700 1 -- 172.21.15.62:0/3651294019 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f6d04094020 con 0x7f6d14106b80 2024-04-20T21:36:49.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.352+0000 7f6d19800700 1 -- 172.21.15.62:0/3651294019 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f6d1404f2a0 con 0x7f6d14106b80 2024-04-20T21:36:49.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.353+0000 7f6d0b400700 1 -- 172.21.15.62:0/3651294019 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+779 (secure 0 0 0) 0x7f6d0401d320 con 0x7f6d14106b80 2024-04-20T21:36:49.352 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:36:49.352 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T21:36:49.352 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 1, 2024-04-20T21:36:49.352 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 1 2024-04-20T21:36:49.352 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:36:49.352 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T21:36:49.353 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:36:49.353 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:36:49.353 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T21:36:49.353 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 6 2024-04-20T21:36:49.353 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:36:49.353 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T21:36:49.353 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 4 2024-04-20T21:36:49.353 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:36:49.353 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T21:36:49.353 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 11, 2024-04-20T21:36:49.353 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 3 2024-04-20T21:36:49.353 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T21:36:49.353 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:36:49.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.355+0000 7f6d19800700 1 -- 172.21.15.62:0/3651294019 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6cf806edf0 msgr2=0x7f6cf80712a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:36:49.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.355+0000 7f6d19800700 1 --2- 172.21.15.62:0/3651294019 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6cf806edf0 0x7f6cf80712a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f6d00006010 tx=0x7f6d0000b540 comp rx=0 tx=0).stop 2024-04-20T21:36:49.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.356+0000 7f6d19800700 1 -- 172.21.15.62:0/3651294019 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d14106b80 msgr2=0x7f6d1419c440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:36:49.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.356+0000 7f6d19800700 1 --2- 172.21.15.62:0/3651294019 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d14106b80 0x7f6d1419c440 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6d0400c8a0 tx=0x7f6d0400cc60 comp rx=0 tx=0).stop 2024-04-20T21:36:49.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.356+0000 7f6d19800700 1 -- 172.21.15.62:0/3651294019 shutdown_connections 2024-04-20T21:36:49.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.356+0000 7f6d19800700 1 --2- 172.21.15.62:0/3651294019 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6cf806edf0 0x7f6cf80712a0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:49.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.356+0000 7f6d19800700 1 --2- 172.21.15.62:0/3651294019 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d14106b80 0x7f6d1419c440 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:49.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.356+0000 7f6d19800700 1 --2- 172.21.15.62:0/3651294019 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6d1410c8f0 0x7f6d1419c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:49.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.356+0000 7f6d19800700 1 -- 172.21.15.62:0/3651294019 >> 172.21.15.62:0/3651294019 conn(0x7f6d14076060 msgr2=0x7f6d14077ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:36:49.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.356+0000 7f6d19800700 1 -- 172.21.15.62:0/3651294019 shutdown_connections 2024-04-20T21:36:49.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.357+0000 7f6d19800700 1 -- 172.21.15.62:0/3651294019 wait complete. 2024-04-20T21:36:49.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.516+0000 7fb7dea00700 1 -- 172.21.15.62:0/2858017888 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb7d8100e80 msgr2=0x7fb7d8104e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:36:49.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.516+0000 7fb7dea00700 1 --2- 172.21.15.62:0/2858017888 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb7d8100e80 0x7fb7d8104e50 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb7c4009910 tx=0x7fb7c4009c20 comp rx=0 tx=0).stop 2024-04-20T21:36:49.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.516+0000 7fb7dea00700 1 -- 172.21.15.62:0/2858017888 shutdown_connections 2024-04-20T21:36:49.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.516+0000 7fb7dea00700 1 --2- 172.21.15.62:0/2858017888 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb7d8100e80 0x7fb7d8104e50 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:49.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.516+0000 7fb7dea00700 1 --2- 172.21.15.62:0/2858017888 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7d81004e0 0x7fb7d81008b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:49.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.516+0000 7fb7dea00700 1 -- 172.21.15.62:0/2858017888 >> 172.21.15.62:0/2858017888 conn(0x7fb7d80fbda0 msgr2=0x7fb7d80fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:36:49.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.516+0000 7fb7dea00700 1 -- 172.21.15.62:0/2858017888 shutdown_connections 2024-04-20T21:36:49.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.516+0000 7fb7dea00700 1 -- 172.21.15.62:0/2858017888 wait complete. 2024-04-20T21:36:49.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.517+0000 7fb7dea00700 1 Processor -- start 2024-04-20T21:36:49.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.517+0000 7fb7dea00700 1 -- start start 2024-04-20T21:36:49.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.518+0000 7fb7dea00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7d81004e0 0x7fb7d8073050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:36:49.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.518+0000 7fb7dea00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb7d8100e80 0x7fb7d8073590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:36:49.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.518+0000 7fb7dea00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d8073ad0 con 0x7fb7d8100e80 2024-04-20T21:36:49.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.518+0000 7fb7dea00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d8073c10 con 0x7fb7d81004e0 2024-04-20T21:36:49.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.518+0000 7fb7d7400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb7d8100e80 0x7fb7d8073590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:36:49.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.518+0000 7fb7d7400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb7d8100e80 0x7fb7d8073590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45370/0 (socket says 172.21.15.62:45370) 2024-04-20T21:36:49.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.518+0000 7fb7d7400700 1 -- 172.21.15.62:0/3939757232 learned_addr learned my addr 172.21.15.62:0/3939757232 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:36:49.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.518+0000 7fb7d7e00700 1 --2- 172.21.15.62:0/3939757232 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7d81004e0 0x7fb7d8073050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:36:49.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.518+0000 7fb7d7400700 1 -- 172.21.15.62:0/3939757232 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7d81004e0 msgr2=0x7fb7d8073050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:36:49.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.518+0000 7fb7d7400700 1 --2- 172.21.15.62:0/3939757232 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7d81004e0 0x7fb7d8073050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:49.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.518+0000 7fb7d7400700 1 -- 172.21.15.62:0/3939757232 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7c40095f0 con 0x7fb7d8100e80 2024-04-20T21:36:49.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.518+0000 7fb7d7e00700 1 --2- 172.21.15.62:0/3939757232 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7d81004e0 0x7fb7d8073050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T21:36:49.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.519+0000 7fb7d7400700 1 --2- 172.21.15.62:0/3939757232 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb7d8100e80 0x7fb7d8073590 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb7c400bda0 tx=0x7fb7c400be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:36:49.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.519+0000 7fb7d4c00700 1 -- 172.21.15.62:0/3939757232 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7c401d070 con 0x7fb7d8100e80 2024-04-20T21:36:49.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.519+0000 7fb7d4c00700 1 -- 172.21.15.62:0/3939757232 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7c4004fb0 con 0x7fb7d8100e80 2024-04-20T21:36:49.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.519+0000 7fb7d4c00700 1 -- 172.21.15.62:0/3939757232 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7c400f460 con 0x7fb7d8100e80 2024-04-20T21:36:49.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.519+0000 7fb7dea00700 1 -- 172.21.15.62:0/3939757232 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7d8073e90 con 0x7fb7d8100e80 2024-04-20T21:36:49.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.519+0000 7fb7dea00700 1 -- 172.21.15.62:0/3939757232 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7d81a2410 con 0x7fb7d8100e80 2024-04-20T21:36:49.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.521+0000 7fb7dea00700 1 -- 172.21.15.62:0/3939757232 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7d8108850 con 0x7fb7d8100e80 2024-04-20T21:36:49.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.521+0000 7fb7d4c00700 1 -- 172.21.15.62:0/3939757232 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 37) v1 ==== 92847+0+0 (secure 0 0 0) 0x7fb7c4022800 con 0x7fb7d8100e80 2024-04-20T21:36:49.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.522+0000 7fb7d4c00700 1 --2- 172.21.15.62:0/3939757232 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb7bc06edf0 0x7fb7bc0712a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:36:49.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.522+0000 7fb7d4c00700 1 -- 172.21.15.62:0/3939757232 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fb7c40904a0 con 0x7fb7d8100e80 2024-04-20T21:36:49.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.523+0000 7fb7d7e00700 1 --2- 172.21.15.62:0/3939757232 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb7bc06edf0 0x7fb7bc0712a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:36:49.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.523+0000 7fb7d7e00700 1 --2- 172.21.15.62:0/3939757232 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb7bc06edf0 0x7fb7bc0712a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fb7c8005fd0 tx=0x7fb7c8005dc0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:36:49.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.527+0000 7fb7d4c00700 1 -- 172.21.15.62:0/3939757232 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fb7c4059330 con 0x7fb7d8100e80 2024-04-20T21:36:49.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:49 smithi062 ceph-mon[76322]: from='client.34130 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:36:49.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:49 smithi062 ceph-mon[76322]: pgmap v10: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 19 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-20T21:36:49.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:49 smithi062 ceph-mon[76322]: from='client.34134 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:36:49.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:49 smithi062 ceph-mon[76322]: from='client.34138 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:36:49.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:49 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:36:49.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:49 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/3651294019' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:36:49.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.821+0000 7fb7dea00700 1 -- 172.21.15.62:0/3939757232 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fb7d804f2a0 con 0x7fb7d8100e80 2024-04-20T21:36:49.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.821+0000 7fb7d4c00700 1 -- 172.21.15.62:0/3939757232 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 17 v17) v1 ==== 76+0+1945 (secure 0 0 0) 0x7fb7c405c9f0 con 0x7fb7d8100e80 2024-04-20T21:36:49.820 INFO:teuthology.orchestra.run.smithi062.stdout:e17 2024-04-20T21:36:49.820 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T21:36:49.820 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T21:36:49.820 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T21:36:49.820 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:36:49.820 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T21:36:49.820 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T21:36:49.820 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 17 2024-04-20T21:36:49.821 INFO:teuthology.orchestra.run.smithi062.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-20T21:36:49.821 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T21:36:49.821 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T21:36:45.237235+0000 2024-04-20T21:36:49.821 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T21:36:49.821 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T21:36:49.821 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T21:36:49.821 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T21:36:49.821 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T21:36:49.821 INFO:teuthology.orchestra.run.smithi062.stdout:max_xattr_size 65536 2024-04-20T21:36:49.821 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T21:36:49.821 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T21:36:49.821 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 0 2024-04-20T21:36:49.821 INFO:teuthology.orchestra.run.smithi062.stdout:snap_rstat 1 2024-04-20T21:36:49.821 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T21:36:49.822 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 2 2024-04-20T21:36:49.822 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1 2024-04-20T21:36:49.822 INFO:teuthology.orchestra.run.smithi062.stdout:up {0=14458,1=24279} 2024-04-20T21:36:49.822 INFO:teuthology.orchestra.run.smithi062.stdout:failed 2024-04-20T21:36:49.822 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T21:36:49.822 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T21:36:49.822 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T21:36:49.822 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T21:36:49.822 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data enabled 2024-04-20T21:36:49.822 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T21:36:49.822 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T21:36:49.822 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 1 2024-04-20T21:36:49.822 INFO:teuthology.orchestra.run.smithi062.stdout:qdb_cluster leader: 14458 members: 14458,24279 2024-04-20T21:36:49.822 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ryqblb{0:14458} state up:active seq 6 export targets 1 join_fscid=1 addr [v2:172.21.15.62:6826/1147127123,v1:172.21.15.62:6827/1147127123] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:36:49.822 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ozizwi{0:14474} state up:standby-replay seq 3 join_fscid=1 addr [v2:172.21.15.62:6828/3494174293,v1:172.21.15.62:6829/3494174293] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:36:49.823 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.dvimer{1:24279} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.189:6824/237262514,v1:172.21.15.189:6825/237262514] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:36:49.823 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.xbgspw{1:24291} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.189:6826/1495356258,v1:172.21.15.189:6827/1495356258] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:36:49.823 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:36:49.823 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:36:49.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.826+0000 7fb7dea00700 1 -- 172.21.15.62:0/3939757232 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb7bc06edf0 msgr2=0x7fb7bc0712a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:36:49.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.826+0000 7fb7dea00700 1 --2- 172.21.15.62:0/3939757232 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb7bc06edf0 0x7fb7bc0712a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fb7c8005fd0 tx=0x7fb7c8005dc0 comp rx=0 tx=0).stop 2024-04-20T21:36:49.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.826+0000 7fb7dea00700 1 -- 172.21.15.62:0/3939757232 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb7d8100e80 msgr2=0x7fb7d8073590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:36:49.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.826+0000 7fb7dea00700 1 --2- 172.21.15.62:0/3939757232 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb7d8100e80 0x7fb7d8073590 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb7c400bda0 tx=0x7fb7c400be80 comp rx=0 tx=0).stop 2024-04-20T21:36:49.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.826+0000 7fb7dea00700 1 -- 172.21.15.62:0/3939757232 shutdown_connections 2024-04-20T21:36:49.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.826+0000 7fb7dea00700 1 --2- 172.21.15.62:0/3939757232 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb7bc06edf0 0x7fb7bc0712a0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:49.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.826+0000 7fb7dea00700 1 --2- 172.21.15.62:0/3939757232 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7d81004e0 0x7fb7d8073050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:49.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.826+0000 7fb7dea00700 1 --2- 172.21.15.62:0/3939757232 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb7d8100e80 0x7fb7d8073590 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:49.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.826+0000 7fb7dea00700 1 -- 172.21.15.62:0/3939757232 >> 172.21.15.62:0/3939757232 conn(0x7fb7d80fbda0 msgr2=0x7fb7d80fe180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:36:49.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.826+0000 7fb7dea00700 1 -- 172.21.15.62:0/3939757232 shutdown_connections 2024-04-20T21:36:49.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.826+0000 7fb7dea00700 1 -- 172.21.15.62:0/3939757232 wait complete. 2024-04-20T21:36:49.825 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 17 2024-04-20T21:36:49.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.981+0000 7fe092c00700 1 -- 172.21.15.62:0/3641694972 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe08c0739a0 msgr2=0x7fe08c10c5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:36:49.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.981+0000 7fe092c00700 1 --2- 172.21.15.62:0/3641694972 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe08c0739a0 0x7fe08c10c5e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe078009940 tx=0x7fe078009c50 comp rx=0 tx=0).stop 2024-04-20T21:36:49.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.981+0000 7fe092c00700 1 -- 172.21.15.62:0/3641694972 shutdown_connections 2024-04-20T21:36:49.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.981+0000 7fe092c00700 1 --2- 172.21.15.62:0/3641694972 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe08c0739a0 0x7fe08c10c5e0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:49.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.981+0000 7fe092c00700 1 --2- 172.21.15.62:0/3641694972 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe08c073090 0x7fe08c073460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:49.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.981+0000 7fe092c00700 1 -- 172.21.15.62:0/3641694972 >> 172.21.15.62:0/3641694972 conn(0x7fe08c0fbda0 msgr2=0x7fe08c0fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:36:49.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.982+0000 7fe092c00700 1 -- 172.21.15.62:0/3641694972 shutdown_connections 2024-04-20T21:36:49.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.982+0000 7fe092c00700 1 -- 172.21.15.62:0/3641694972 wait complete. 2024-04-20T21:36:49.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.983+0000 7fe092c00700 1 Processor -- start 2024-04-20T21:36:49.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.983+0000 7fe092c00700 1 -- start start 2024-04-20T21:36:49.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.984+0000 7fe092c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe08c073090 0x7fe08c1981b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:36:49.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.984+0000 7fe092c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe08c0739a0 0x7fe08c1986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:36:49.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.984+0000 7fe092c00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe08c198dd0 con 0x7fe08c0739a0 2024-04-20T21:36:49.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.984+0000 7fe092c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe08c19cb60 con 0x7fe08c073090 2024-04-20T21:36:49.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.984+0000 7fe08b400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe08c0739a0 0x7fe08c1986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:36:49.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.984+0000 7fe08b400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe08c0739a0 0x7fe08c1986f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45378/0 (socket says 172.21.15.62:45378) 2024-04-20T21:36:49.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.984+0000 7fe08b400700 1 -- 172.21.15.62:0/2834705645 learned_addr learned my addr 172.21.15.62:0/2834705645 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:36:49.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.984+0000 7fe08be00700 1 --2- 172.21.15.62:0/2834705645 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe08c073090 0x7fe08c1981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:36:49.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.985+0000 7fe08b400700 1 -- 172.21.15.62:0/2834705645 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe08c073090 msgr2=0x7fe08c1981b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:36:49.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.985+0000 7fe08b400700 1 --2- 172.21.15.62:0/2834705645 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe08c073090 0x7fe08c1981b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:49.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.985+0000 7fe08b400700 1 -- 172.21.15.62:0/2834705645 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0780095f0 con 0x7fe08c0739a0 2024-04-20T21:36:49.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.985+0000 7fe08b400700 1 --2- 172.21.15.62:0/2834705645 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe08c0739a0 0x7fe08c1986f0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe0780055f0 tx=0x7fe078005620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:36:49.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.985+0000 7fe088c00700 1 -- 172.21.15.62:0/2834705645 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe07801d070 con 0x7fe08c0739a0 2024-04-20T21:36:49.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.986+0000 7fe088c00700 1 -- 172.21.15.62:0/2834705645 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe078005dc0 con 0x7fe08c0739a0 2024-04-20T21:36:49.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.986+0000 7fe092c00700 1 -- 172.21.15.62:0/2834705645 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe08c19cde0 con 0x7fe08c0739a0 2024-04-20T21:36:49.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.986+0000 7fe088c00700 1 -- 172.21.15.62:0/2834705645 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe078021410 con 0x7fe08c0739a0 2024-04-20T21:36:49.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.986+0000 7fe092c00700 1 -- 172.21.15.62:0/2834705645 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe08c19d2d0 con 0x7fe08c0739a0 2024-04-20T21:36:49.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.987+0000 7fe092c00700 1 -- 172.21.15.62:0/2834705645 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe08c109ce0 con 0x7fe08c0739a0 2024-04-20T21:36:49.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.988+0000 7fe088c00700 1 -- 172.21.15.62:0/2834705645 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 37) v1 ==== 92847+0+0 (secure 0 0 0) 0x7fe07800f810 con 0x7fe08c0739a0 2024-04-20T21:36:49.986 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:49 smithi189 ceph-mon[21562]: from='client.34130 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:36:49.986 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:49 smithi189 ceph-mon[21562]: pgmap v10: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 19 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-20T21:36:49.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:49 smithi189 ceph-mon[21562]: from='client.34134 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:36:49.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:49 smithi189 ceph-mon[21562]: from='client.34138 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:36:49.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:49 smithi189 ceph-mon[21562]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:36:49.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:49 smithi189 ceph-mon[21562]: from='client.? 172.21.15.62:0/3651294019' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:36:49.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.989+0000 7fe088c00700 1 --2- 172.21.15.62:0/2834705645 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe07006edf0 0x7fe0700712a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:36:49.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.989+0000 7fe088c00700 1 -- 172.21.15.62:0/2834705645 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fe078090250 con 0x7fe08c0739a0 2024-04-20T21:36:49.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.989+0000 7fe08be00700 1 --2- 172.21.15.62:0/2834705645 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe07006edf0 0x7fe0700712a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:36:49.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.990+0000 7fe08be00700 1 --2- 172.21.15.62:0/2834705645 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe07006edf0 0x7fe0700712a0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fe07c005950 tx=0x7fe07c00b410 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:36:49.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:49.994+0000 7fe088c00700 1 -- 172.21.15.62:0/2834705645 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fe078058e90 con 0x7fe08c0739a0 2024-04-20T21:36:50.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:50.246+0000 7fe092c00700 1 -- 172.21.15.62:0/2834705645 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe08c199510 con 0x7fe07006edf0 2024-04-20T21:36:50.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:50.248+0000 7fe088c00700 1 -- 172.21.15.62:0/2834705645 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+337 (secure 0 0 0) 0x7fe08c199510 con 0x7fe07006edf0 2024-04-20T21:36:50.247 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:36:50.247 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T21:36:50.247 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T21:36:50.247 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-20T21:36:50.247 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T21:36:50.247 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr" 2024-04-20T21:36:50.247 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T21:36:50.247 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "3/23 daemons upgraded", 2024-04-20T21:36:50.247 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "", 2024-04-20T21:36:50.247 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T21:36:50.247 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:36:50.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:50.252+0000 7fe092c00700 1 -- 172.21.15.62:0/2834705645 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe07006edf0 msgr2=0x7fe0700712a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:36:50.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:50.252+0000 7fe092c00700 1 --2- 172.21.15.62:0/2834705645 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe07006edf0 0x7fe0700712a0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fe07c005950 tx=0x7fe07c00b410 comp rx=0 tx=0).stop 2024-04-20T21:36:50.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:50.252+0000 7fe092c00700 1 -- 172.21.15.62:0/2834705645 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe08c0739a0 msgr2=0x7fe08c1986f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:36:50.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:50.252+0000 7fe092c00700 1 --2- 172.21.15.62:0/2834705645 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe08c0739a0 0x7fe08c1986f0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe0780055f0 tx=0x7fe078005620 comp rx=0 tx=0).stop 2024-04-20T21:36:50.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:50.253+0000 7fe092c00700 1 -- 172.21.15.62:0/2834705645 shutdown_connections 2024-04-20T21:36:50.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:50.253+0000 7fe092c00700 1 --2- 172.21.15.62:0/2834705645 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe07006edf0 0x7fe0700712a0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:50.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:50.253+0000 7fe092c00700 1 --2- 172.21.15.62:0/2834705645 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe08c073090 0x7fe08c1981b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:50.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:50.253+0000 7fe092c00700 1 --2- 172.21.15.62:0/2834705645 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe08c0739a0 0x7fe08c1986f0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:50.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:50.253+0000 7fe092c00700 1 -- 172.21.15.62:0/2834705645 >> 172.21.15.62:0/2834705645 conn(0x7fe08c0fbda0 msgr2=0x7fe08c106e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:36:50.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:50.253+0000 7fe092c00700 1 -- 172.21.15.62:0/2834705645 shutdown_connections 2024-04-20T21:36:50.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:50.253+0000 7fe092c00700 1 -- 172.21.15.62:0/2834705645 wait complete. 2024-04-20T21:36:50.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:50.400+0000 7f5940400700 1 -- 172.21.15.62:0/4066461145 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5938100e80 msgr2=0x7f5938104e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:36:50.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:50.400+0000 7f5940400700 1 --2- 172.21.15.62:0/4066461145 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5938100e80 0x7f5938104e50 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f5924009910 tx=0x7f5924009c20 comp rx=0 tx=0).stop 2024-04-20T21:36:50.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:50.401+0000 7f5940400700 1 -- 172.21.15.62:0/4066461145 shutdown_connections 2024-04-20T21:36:50.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:50.401+0000 7f5940400700 1 --2- 172.21.15.62:0/4066461145 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5938100e80 0x7f5938104e50 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:50.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:50.401+0000 7f5940400700 1 --2- 172.21.15.62:0/4066461145 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f59381004e0 0x7f59381008b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:50.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:50.401+0000 7f5940400700 1 -- 172.21.15.62:0/4066461145 >> 172.21.15.62:0/4066461145 conn(0x7f59380fbda0 msgr2=0x7f59380fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:36:50.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:50.401+0000 7f5940400700 1 -- 172.21.15.62:0/4066461145 shutdown_connections 2024-04-20T21:36:50.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:50.401+0000 7f5940400700 1 -- 172.21.15.62:0/4066461145 wait complete. 2024-04-20T21:36:50.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:50.402+0000 7f5940400700 1 Processor -- start 2024-04-20T21:36:50.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:50.402+0000 7f5940400700 1 -- start start 2024-04-20T21:36:50.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:50.403+0000 7f5940400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f59381004e0 0x7f5938195f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:36:50.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:50.403+0000 7f5940400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5938100e80 0x7f5938196470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:36:50.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:50.403+0000 7f5940400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5938196b50 con 0x7f5938100e80 2024-04-20T21:36:50.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:50.403+0000 7f5940400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f593819a8e0 con 0x7f59381004e0 2024-04-20T21:36:50.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:50.403+0000 7f593d000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5938100e80 0x7f5938196470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:36:50.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:50.403+0000 7f593d000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5938100e80 0x7f5938196470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45388/0 (socket says 172.21.15.62:45388) 2024-04-20T21:36:50.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:50.403+0000 7f593d000700 1 -- 172.21.15.62:0/1372000334 learned_addr learned my addr 172.21.15.62:0/1372000334 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:36:50.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:50.404+0000 7f593da00700 1 --2- 172.21.15.62:0/1372000334 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f59381004e0 0x7f5938195f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:36:50.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:50.404+0000 7f593d000700 1 -- 172.21.15.62:0/1372000334 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f59381004e0 msgr2=0x7f5938195f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:36:50.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:50.404+0000 7f593d000700 1 --2- 172.21.15.62:0/1372000334 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f59381004e0 0x7f5938195f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:50.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:50.404+0000 7f593d000700 1 -- 172.21.15.62:0/1372000334 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59240095f0 con 0x7f5938100e80 2024-04-20T21:36:50.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:50.404+0000 7f593d000700 1 --2- 172.21.15.62:0/1372000334 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5938100e80 0x7f5938196470 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f5924009fd0 tx=0x7f592400f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:36:50.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:50.405+0000 7f5936000700 1 -- 172.21.15.62:0/1372000334 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f592401d070 con 0x7f5938100e80 2024-04-20T21:36:50.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:50.405+0000 7f5940400700 1 -- 172.21.15.62:0/1372000334 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f593819ab60 con 0x7f5938100e80 2024-04-20T21:36:50.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:50.405+0000 7f5936000700 1 -- 172.21.15.62:0/1372000334 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5924022470 con 0x7f5938100e80 2024-04-20T21:36:50.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:50.405+0000 7f5936000700 1 -- 172.21.15.62:0/1372000334 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5924017650 con 0x7f5938100e80 2024-04-20T21:36:50.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:50.405+0000 7f5940400700 1 -- 172.21.15.62:0/1372000334 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f593819aff0 con 0x7f5938100e80 2024-04-20T21:36:50.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:50.406+0000 7f5940400700 1 -- 172.21.15.62:0/1372000334 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5938108850 con 0x7f5938100e80 2024-04-20T21:36:50.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:50.408+0000 7f5936000700 1 -- 172.21.15.62:0/1372000334 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 37) v1 ==== 92847+0+0 (secure 0 0 0) 0x7f59240050c0 con 0x7f5938100e80 2024-04-20T21:36:50.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:50.409+0000 7f5936000700 1 --2- 172.21.15.62:0/1372000334 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f592006edf0 0x7f59200712a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:36:50.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:50.409+0000 7f5936000700 1 -- 172.21.15.62:0/1372000334 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f5924090680 con 0x7f5938100e80 2024-04-20T21:36:50.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:50.410+0000 7f593da00700 1 --2- 172.21.15.62:0/1372000334 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f592006edf0 0x7f59200712a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:36:50.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:50.410+0000 7f593da00700 1 --2- 172.21.15.62:0/1372000334 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f592006edf0 0x7f59200712a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f592c009c60 tx=0x7f592c009260 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:36:50.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:50.413+0000 7f5936000700 1 -- 172.21.15.62:0/1372000334 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f59240592a0 con 0x7f5938100e80 2024-04-20T21:36:50.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:50.770+0000 7f5940400700 1 -- 172.21.15.62:0/1372000334 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f593804f2a0 con 0x7f5938100e80 2024-04-20T21:36:50.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:50.770+0000 7f5936000700 1 -- 172.21.15.62:0/1372000334 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7f592405c960 con 0x7f5938100e80 2024-04-20T21:36:50.770 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-04-20T21:36:50.770 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T21:36:50.770 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T21:36:50.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:50.774+0000 7f5940400700 1 -- 172.21.15.62:0/1372000334 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f592006edf0 msgr2=0x7f59200712a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:36:50.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:50.774+0000 7f5940400700 1 --2- 172.21.15.62:0/1372000334 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f592006edf0 0x7f59200712a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f592c009c60 tx=0x7f592c009260 comp rx=0 tx=0).stop 2024-04-20T21:36:50.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:50.774+0000 7f5940400700 1 -- 172.21.15.62:0/1372000334 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5938100e80 msgr2=0x7f5938196470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:36:50.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:50.774+0000 7f5940400700 1 --2- 172.21.15.62:0/1372000334 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5938100e80 0x7f5938196470 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f5924009fd0 tx=0x7f592400f690 comp rx=0 tx=0).stop 2024-04-20T21:36:50.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:50.774+0000 7f5940400700 1 -- 172.21.15.62:0/1372000334 shutdown_connections 2024-04-20T21:36:50.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:50.774+0000 7f5940400700 1 --2- 172.21.15.62:0/1372000334 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f592006edf0 0x7f59200712a0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:50.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:50.774+0000 7f5940400700 1 --2- 172.21.15.62:0/1372000334 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f59381004e0 0x7f5938195f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:50.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:50.774+0000 7f5940400700 1 --2- 172.21.15.62:0/1372000334 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5938100e80 0x7f5938196470 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:36:50.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:50.774+0000 7f5940400700 1 -- 172.21.15.62:0/1372000334 >> 172.21.15.62:0/1372000334 conn(0x7f59380fbda0 msgr2=0x7f59380fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:36:50.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:50.774+0000 7f5940400700 1 -- 172.21.15.62:0/1372000334 shutdown_connections 2024-04-20T21:36:50.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:36:50.774+0000 7f5940400700 1 -- 172.21.15.62:0/1372000334 wait complete. 2024-04-20T21:36:50.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:50 smithi189 ceph-mon[21562]: Detected new or changed devices on smithi062 2024-04-20T21:36:50.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:50 smithi189 ceph-mon[21562]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:36:50.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:50 smithi189 ceph-mon[21562]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:36:50.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:50 smithi189 ceph-mon[21562]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:36:50.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:50 smithi189 ceph-mon[21562]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T21:36:50.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:50 smithi189 ceph-mon[21562]: Updating smithi062:/etc/ceph/ceph.conf 2024-04-20T21:36:50.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:50 smithi189 ceph-mon[21562]: Updating smithi189:/etc/ceph/ceph.conf 2024-04-20T21:36:50.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:50 smithi189 ceph-mon[21562]: from='client.? 172.21.15.62:0/3939757232' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T21:36:51.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:50 smithi062 ceph-mon[76322]: Detected new or changed devices on smithi062 2024-04-20T21:36:51.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:50 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:36:51.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:50 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:36:51.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:50 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:36:51.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:50 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T21:36:51.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:50 smithi062 ceph-mon[76322]: Updating smithi062:/etc/ceph/ceph.conf 2024-04-20T21:36:51.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:50 smithi062 ceph-mon[76322]: Updating smithi189:/etc/ceph/ceph.conf 2024-04-20T21:36:51.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:50 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/3939757232' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T21:36:52.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:51 smithi062 ceph-mon[76322]: from='client.34150 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:36:52.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:51 smithi062 ceph-mon[76322]: pgmap v11: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 19 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-20T21:36:52.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:51 smithi062 ceph-mon[76322]: Updating smithi062:/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-20T21:36:52.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:51 smithi062 ceph-mon[76322]: Updating smithi189:/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-20T21:36:52.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:51 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/1372000334' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T21:36:52.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:51 smithi189 ceph-mon[21562]: from='client.34150 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:36:52.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:51 smithi189 ceph-mon[21562]: pgmap v11: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 19 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-20T21:36:52.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:51 smithi189 ceph-mon[21562]: Updating smithi062:/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-20T21:36:52.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:51 smithi189 ceph-mon[21562]: Updating smithi189:/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-20T21:36:52.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:51 smithi189 ceph-mon[21562]: from='client.? 172.21.15.62:0/1372000334' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T21:36:53.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:52 smithi062 ceph-mon[76322]: Updating smithi062:/etc/ceph/ceph.client.admin.keyring 2024-04-20T21:36:53.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:52 smithi062 ceph-mon[76322]: Updating smithi189:/etc/ceph/ceph.client.admin.keyring 2024-04-20T21:36:53.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:52 smithi062 ceph-mon[76322]: Updating smithi062:/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-20T21:36:53.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:52 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:36:53.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:52 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:36:53.137 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:52 smithi189 ceph-mon[21562]: Updating smithi062:/etc/ceph/ceph.client.admin.keyring 2024-04-20T21:36:53.138 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:52 smithi189 ceph-mon[21562]: Updating smithi189:/etc/ceph/ceph.client.admin.keyring 2024-04-20T21:36:53.138 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:52 smithi189 ceph-mon[21562]: Updating smithi062:/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-20T21:36:53.138 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:52 smithi189 ceph-mon[21562]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:36:53.138 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:52 smithi189 ceph-mon[21562]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:36:54.141 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:53 smithi189 ceph-mon[21562]: Updating smithi189:/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-20T21:36:54.141 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:53 smithi189 ceph-mon[21562]: pgmap v12: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 20 KiB/s rd, 0 B/s wr, 14 op/s 2024-04-20T21:36:54.142 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:53 smithi189 ceph-mon[21562]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:36:54.142 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:53 smithi189 ceph-mon[21562]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:36:54.142 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:53 smithi189 ceph-mon[21562]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:36:54.142 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:53 smithi189 ceph-mon[21562]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:36:54.142 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:53 smithi189 ceph-mon[21562]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:36:54.142 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:53 smithi189 ceph-mon[21562]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-20T21:36:54.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:53 smithi062 ceph-mon[76322]: Updating smithi189:/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-20T21:36:54.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:53 smithi062 ceph-mon[76322]: pgmap v12: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 20 KiB/s rd, 0 B/s wr, 14 op/s 2024-04-20T21:36:54.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:53 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:36:54.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:53 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:36:54.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:53 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:36:54.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:53 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:36:54.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:53 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:36:54.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:53 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-20T21:36:56.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:55 smithi189 ceph-mon[21562]: pgmap v13: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:36:56.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:55 smithi189 ceph-mon[21562]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:36:56.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:55 smithi189 ceph-mon[21562]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-20T21:36:56.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:55 smithi189 ceph-mon[21562]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-20T21:36:56.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:55 smithi189 ceph-mon[21562]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:36:56.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:55 smithi062 ceph-mon[76322]: pgmap v13: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:36:56.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:55 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:36:56.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:55 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-20T21:36:56.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:55 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-20T21:36:56.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:55 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:36:56.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:56 smithi189 ceph-mon[21562]: Upgrade: Updating mon.smithi189 2024-04-20T21:36:56.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:56 smithi189 ceph-mon[21562]: Deploying daemon mon.smithi189 on smithi189 2024-04-20T21:36:57.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:56 smithi062 ceph-mon[76322]: Upgrade: Updating mon.smithi189 2024-04-20T21:36:57.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:56 smithi062 ceph-mon[76322]: Deploying daemon mon.smithi189 on smithi189 2024-04-20T21:36:58.143 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:36:57 smithi062 ceph-mon[76322]: pgmap v14: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:36:58.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:57 smithi189 ceph-mon[21562]: pgmap v14: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:36:58.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:58 smithi189 systemd[1]: Stopping Ceph mon.smithi189 for 61ae3628-ff5b-11ee-bc93-c7b262605968... 2024-04-20T21:36:58.661 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:58 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-mon-smithi189[21552]: 2024-04-20T21:36:58.454+0000 7feda8e00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi189 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-20T21:36:58.661 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:58 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-mon-smithi189[21552]: 2024-04-20T21:36:58.454+0000 7feda8e00700 -1 mon.smithi189@1(peon) e2 *** Got Signal Terminated *** 2024-04-20T21:36:58.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:58 smithi189 podman[56859]: 2024-04-20 21:36:58.6619465 +0000 UTC m=+0.345050633 container died 3735c8e5db9a6484b1b5471b5a0c6f02fc9522a1425a18d987470ee7a1080729 (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-mon-smithi189, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-18.2.0, RELEASE=HEAD, ceph=True, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, org.label-schema.schema-version=1.0, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, org.label-schema.build-date=20231212, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-04-20T21:36:59.366 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:59 smithi189 podman[56859]: 2024-04-20 21:36:59.124447215 +0000 UTC m=+0.807551347 container cleanup 3735c8e5db9a6484b1b5471b5a0c6f02fc9522a1425a18d987470ee7a1080729 (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-mon-smithi189, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20231212, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, RELEASE=HEAD) 2024-04-20T21:36:59.367 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:59 smithi189 bash[56859]: ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-mon-smithi189 2024-04-20T21:36:59.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:59 smithi189 podman[56873]: 2024-04-20 21:36:59.367117898 +0000 UTC m=+0.688658931 container remove 3735c8e5db9a6484b1b5471b5a0c6f02fc9522a1425a18d987470ee7a1080729 (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-mon-smithi189, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20231212, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-18.2.0, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS) 2024-04-20T21:37:00.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:59 smithi189 systemd[1]: ceph-61ae3628-ff5b-11ee-bc93-c7b262605968@mon.smithi189.service: Deactivated successfully. 2024-04-20T21:37:00.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:59 smithi189 systemd[1]: Stopped Ceph mon.smithi189 for 61ae3628-ff5b-11ee-bc93-c7b262605968. 2024-04-20T21:37:00.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:36:59 smithi189 systemd[1]: ceph-61ae3628-ff5b-11ee-bc93-c7b262605968@mon.smithi189.service: Consumed 20.784s CPU time. 2024-04-20T21:37:00.621 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 systemd[1]: Starting Ceph mon.smithi189 for 61ae3628-ff5b-11ee-bc93-c7b262605968... 2024-04-20T21:37:00.924 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 podman[57004]: 2024-04-20 21:37:00.526968338 +0000 UTC m=+0.032781818 image pull 76bc6b91b726a03b8b736eeeb35fc1ffd1469e23c7d60f5b444a217528d8906d quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 2024-04-20T21:37:00.924 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 podman[57004]: 2024-04-20 21:37:00.643120474 +0000 UTC m=+0.148933991 container create 55aac560aab6cf464252775feab1579f97a33fad9d546da9a8dc6e47acd49d99 (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-mon-smithi189, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416) 2024-04-20T21:37:00.924 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 podman[57004]: 2024-04-20 21:37:00.924722967 +0000 UTC m=+0.430536478 container init 55aac560aab6cf464252775feab1579f97a33fad9d546da9a8dc6e47acd49d99 (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-mon-smithi189, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, ceph=True) 2024-04-20T21:37:01.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 podman[57004]: 2024-04-20 21:37:00.93829356 +0000 UTC m=+0.444107036 container start 55aac560aab6cf464252775feab1579f97a33fad9d546da9a8dc6e47acd49d99 (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-mon-smithi189, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5) 2024-04-20T21:37:01.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: set uid:gid to 167:167 (ceph:ceph) 2024-04-20T21:37:01.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev), process ceph-mon, pid 2 2024-04-20T21:37:01.239 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: pidfile_write: ignore empty --pid-file 2024-04-20T21:37:01.239 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: load: jerasure load: lrc 2024-04-20T21:37:01.239 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: RocksDB version: 7.9.2 2024-04-20T21:37:01.239 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Git sha 0 2024-04-20T21:37:01.239 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Compile date 2024-04-17 00:07:32 2024-04-20T21:37:01.239 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: DB SUMMARY 2024-04-20T21:37:01.239 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: DB Session ID: EZFZVJLBFK85RSRJT2XY 2024-04-20T21:37:01.239 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: CURRENT file: CURRENT 2024-04-20T21:37:01.239 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: IDENTITY file: IDENTITY 2024-04-20T21:37:01.239 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: MANIFEST file: MANIFEST-000010 size: 1626 Bytes 2024-04-20T21:37:01.239 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi189/store.db dir, Total Num: 1, files: 000030.sst 2024-04-20T21:37:01.239 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi189/store.db: 000028.log size: 4577771 ; 2024-04-20T21:37:01.239 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.error_if_exists: 0 2024-04-20T21:37:01.239 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.create_if_missing: 0 2024-04-20T21:37:01.239 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.paranoid_checks: 1 2024-04-20T21:37:01.239 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-20T21:37:01.240 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-20T21:37:01.240 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-20T21:37:01.240 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.env: 0x5602b7afacc0 2024-04-20T21:37:01.240 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.fs: PosixFileSystem 2024-04-20T21:37:01.240 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.info_log: 0x5602beb3db00 2024-04-20T21:37:01.240 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.max_file_opening_threads: 16 2024-04-20T21:37:01.240 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.statistics: (nil) 2024-04-20T21:37:01.240 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.use_fsync: 0 2024-04-20T21:37:01.240 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.max_log_file_size: 0 2024-04-20T21:37:01.240 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-20T21:37:01.240 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-20T21:37:01.240 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.keep_log_file_num: 1000 2024-04-20T21:37:01.240 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.recycle_log_file_num: 0 2024-04-20T21:37:01.240 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.allow_fallocate: 1 2024-04-20T21:37:01.241 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.allow_mmap_reads: 0 2024-04-20T21:37:01.241 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.allow_mmap_writes: 0 2024-04-20T21:37:01.241 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.use_direct_reads: 0 2024-04-20T21:37:01.241 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-20T21:37:01.241 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.create_missing_column_families: 0 2024-04-20T21:37:01.241 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.db_log_dir: 2024-04-20T21:37:01.241 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.wal_dir: 2024-04-20T21:37:01.241 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-20T21:37:01.242 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-20T21:37:01.242 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-20T21:37:01.242 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-20T21:37:01.242 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-20T21:37:01.242 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-20T21:37:01.242 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.advise_random_on_open: 1 2024-04-20T21:37:01.242 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.db_write_buffer_size: 0 2024-04-20T21:37:01.242 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.write_buffer_manager: 0x5602beb0c1e0 2024-04-20T21:37:01.242 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-20T21:37:01.242 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-20T21:37:01.242 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-20T21:37:01.242 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.rate_limiter: (nil) 2024-04-20T21:37:01.242 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-20T21:37:01.243 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.wal_recovery_mode: 2 2024-04-20T21:37:01.243 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.enable_thread_tracking: 0 2024-04-20T21:37:01.243 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.enable_pipelined_write: 0 2024-04-20T21:37:01.243 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.unordered_write: 0 2024-04-20T21:37:01.243 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-20T21:37:01.243 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-20T21:37:01.243 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-20T21:37:01.243 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-20T21:37:01.243 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.row_cache: None 2024-04-20T21:37:01.243 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.wal_filter: None 2024-04-20T21:37:01.243 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-20T21:37:01.243 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.allow_ingest_behind: 0 2024-04-20T21:37:01.243 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.two_write_queues: 0 2024-04-20T21:37:01.243 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.manual_wal_flush: 0 2024-04-20T21:37:01.243 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.wal_compression: 0 2024-04-20T21:37:01.244 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.atomic_flush: 0 2024-04-20T21:37:01.244 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-20T21:37:01.244 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-20T21:37:01.244 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-20T21:37:01.244 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.log_readahead_size: 0 2024-04-20T21:37:01.244 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-20T21:37:01.244 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.best_efforts_recovery: 0 2024-04-20T21:37:01.244 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-20T21:37:01.244 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-20T21:37:01.244 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.allow_data_in_errors: 0 2024-04-20T21:37:01.244 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.db_host_id: __hostname__ 2024-04-20T21:37:01.244 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-20T21:37:01.244 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.max_background_jobs: 2 2024-04-20T21:37:01.244 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.max_background_compactions: -1 2024-04-20T21:37:01.245 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.max_subcompactions: 1 2024-04-20T21:37:01.245 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-20T21:37:01.245 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-20T21:37:01.245 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-20T21:37:01.245 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.max_total_wal_size: 0 2024-04-20T21:37:01.245 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-20T21:37:01.245 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-20T21:37:01.245 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-20T21:37:01.245 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-20T21:37:01.245 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.max_open_files: -1 2024-04-20T21:37:01.245 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.bytes_per_sync: 0 2024-04-20T21:37:01.245 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-20T21:37:01.245 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-20T21:37:01.245 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.compaction_readahead_size: 0 2024-04-20T21:37:01.245 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.max_background_flushes: -1 2024-04-20T21:37:01.246 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Compression algorithms supported: 2024-04-20T21:37:01.246 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-20T21:37:01.246 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: kZSTD supported: 0 2024-04-20T21:37:01.246 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: kXpressCompression supported: 0 2024-04-20T21:37:01.246 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: kLZ4HCCompression supported: 1 2024-04-20T21:37:01.246 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: kZlibCompression supported: 1 2024-04-20T21:37:01.246 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: kSnappyCompression supported: 1 2024-04-20T21:37:01.246 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: kLZ4Compression supported: 1 2024-04-20T21:37:01.246 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: kBZip2Compression supported: 0 2024-04-20T21:37:01.246 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-20T21:37:01.246 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-20T21:37:01.246 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi189/store.db/MANIFEST-000010 2024-04-20T21:37:01.246 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-20T21:37:01.246 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-20T21:37:01.247 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.merge_operator: 2024-04-20T21:37:01.248 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.compaction_filter: None 2024-04-20T21:37:01.248 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.compaction_filter_factory: None 2024-04-20T21:37:01.248 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.sst_partitioner_factory: None 2024-04-20T21:37:01.248 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-20T21:37:01.248 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-20T21:37:01.248 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5602beb3dce0) 2024-04-20T21:37:01.248 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: cache_index_and_filter_blocks: 1 2024-04-20T21:37:01.248 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-20T21:37:01.248 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-20T21:37:01.248 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: pin_top_level_index_and_filter: 1 2024-04-20T21:37:01.248 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: index_type: 0 2024-04-20T21:37:01.248 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: data_block_index_type: 0 2024-04-20T21:37:01.248 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: index_shortening: 1 2024-04-20T21:37:01.248 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-20T21:37:01.249 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: checksum: 4 2024-04-20T21:37:01.249 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: no_block_cache: 0 2024-04-20T21:37:01.249 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: block_cache: 0x5602beb3f090 2024-04-20T21:37:01.249 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: block_cache_name: BinnedLRUCache 2024-04-20T21:37:01.249 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: block_cache_options: 2024-04-20T21:37:01.249 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: capacity : 536870912 2024-04-20T21:37:01.249 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: num_shard_bits : 4 2024-04-20T21:37:01.249 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: strict_capacity_limit : 0 2024-04-20T21:37:01.249 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: high_pri_pool_ratio: 0.000 2024-04-20T21:37:01.249 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: block_cache_compressed: (nil) 2024-04-20T21:37:01.249 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: persistent_cache: (nil) 2024-04-20T21:37:01.249 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: block_size: 4096 2024-04-20T21:37:01.249 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: block_size_deviation: 10 2024-04-20T21:37:01.249 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: block_restart_interval: 16 2024-04-20T21:37:01.250 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: index_block_restart_interval: 1 2024-04-20T21:37:01.250 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: metadata_block_size: 4096 2024-04-20T21:37:01.250 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: partition_filters: 0 2024-04-20T21:37:01.250 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: use_delta_encoding: 1 2024-04-20T21:37:01.250 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: filter_policy: bloomfilter 2024-04-20T21:37:01.250 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: whole_key_filtering: 1 2024-04-20T21:37:01.250 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: verify_compression: 0 2024-04-20T21:37:01.250 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: read_amp_bytes_per_bit: 0 2024-04-20T21:37:01.250 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: format_version: 5 2024-04-20T21:37:01.250 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: enable_index_compression: 1 2024-04-20T21:37:01.250 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: block_align: 0 2024-04-20T21:37:01.250 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: max_auto_readahead_size: 262144 2024-04-20T21:37:01.250 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: prepopulate_block_cache: 0 2024-04-20T21:37:01.251 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: initial_auto_readahead_size: 8192 2024-04-20T21:37:01.251 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: num_file_reads_for_auto_readahead: 2 2024-04-20T21:37:01.251 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.write_buffer_size: 33554432 2024-04-20T21:37:01.251 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.max_write_buffer_number: 2 2024-04-20T21:37:01.251 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.compression: NoCompression 2024-04-20T21:37:01.251 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.bottommost_compression: Disabled 2024-04-20T21:37:01.251 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.prefix_extractor: nullptr 2024-04-20T21:37:01.251 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-20T21:37:01.251 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.num_levels: 7 2024-04-20T21:37:01.251 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-20T21:37:01.251 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-20T21:37:01.251 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-20T21:37:01.251 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-20T21:37:01.251 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-20T21:37:01.252 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-20T21:37:01.252 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-20T21:37:01.252 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-20T21:37:01.252 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-20T21:37:01.252 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-20T21:37:01.252 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-20T21:37:01.252 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-20T21:37:01.252 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-20T21:37:01.252 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.compression_opts.level: 32767 2024-04-20T21:37:01.252 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.compression_opts.strategy: 0 2024-04-20T21:37:01.252 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-20T21:37:01.252 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-20T21:37:01.252 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-20T21:37:01.252 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-20T21:37:01.253 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.compression_opts.enabled: false 2024-04-20T21:37:01.253 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-20T21:37:01.253 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-20T21:37:01.253 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-20T21:37:01.253 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-20T21:37:01.253 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.target_file_size_base: 67108864 2024-04-20T21:37:01.253 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-20T21:37:01.253 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-20T21:37:01.253 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-20T21:37:01.253 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-20T21:37:01.253 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-20T21:37:01.253 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-20T21:37:01.253 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-20T21:37:01.253 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-20T21:37:01.253 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-20T21:37:01.254 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-20T21:37:01.254 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-20T21:37:01.254 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-20T21:37:01.254 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-20T21:37:01.254 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-20T21:37:01.254 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.arena_block_size: 1048576 2024-04-20T21:37:01.254 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-20T21:37:01.254 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-20T21:37:01.254 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.disable_auto_compactions: 0 2024-04-20T21:37:01.254 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-20T21:37:01.254 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-20T21:37:01.254 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-20T21:37:01.254 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-20T21:37:01.254 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-20T21:37:01.255 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-20T21:37:01.255 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-20T21:37:01.255 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-20T21:37:01.255 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-20T21:37:01.255 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-20T21:37:01.255 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-20T21:37:01.255 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.inplace_update_support: 0 2024-04-20T21:37:01.255 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-20T21:37:01.255 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-20T21:37:01.255 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-20T21:37:01.255 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-20T21:37:01.255 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.bloom_locality: 0 2024-04-20T21:37:01.255 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.max_successive_merges: 0 2024-04-20T21:37:01.255 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-20T21:37:01.256 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.paranoid_file_checks: 0 2024-04-20T21:37:01.256 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.force_consistency_checks: 1 2024-04-20T21:37:01.256 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.report_bg_io_stats: 0 2024-04-20T21:37:01.256 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.ttl: 2592000 2024-04-20T21:37:01.256 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-20T21:37:01.256 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-20T21:37:01.256 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-20T21:37:01.256 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.enable_blob_files: false 2024-04-20T21:37:01.256 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.min_blob_size: 0 2024-04-20T21:37:01.256 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.blob_file_size: 268435456 2024-04-20T21:37:01.256 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-20T21:37:01.256 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-20T21:37:01.256 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-20T21:37:01.256 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-20T21:37:01.256 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-20T21:37:01.257 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.blob_file_starting_level: 0 2024-04-20T21:37:01.257 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-20T21:37:01.257 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-20T21:37:01.257 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi189/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 32, last_sequence is 16825, log_number is 28,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 28 2024-04-20T21:37:01.257 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 28 2024-04-20T21:37:01.257 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: bab33035-c1ec-43e0-802c-6d3d73eaa2db 2024-04-20T21:37:01.257 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713649020992700, "job": 1, "event": "recovery_started", "wal_files": [28]} 2024-04-20T21:37:01.257 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:00 smithi189 ceph-mon[57022]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #28 mode 2 2024-04-20T21:37:01.257 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:01 smithi189 ceph-mon[57022]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713649021011646, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 33, "file_size": 2870397, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 16830, "largest_seqno": 18902, "table_properties": {"data_size": 2862408, "index_size": 4959, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1989, "raw_key_size": 18615, "raw_average_key_size": 23, "raw_value_size": 2845636, "raw_average_value_size": 3602, "num_data_blocks": 225, "num_entries": 790, "num_filter_entries": 790, "num_deletions": 6, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1713649021, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "bab33035-c1ec-43e0-802c-6d3d73eaa2db", "db_session_id": "EZFZVJLBFK85RSRJT2XY", "orig_file_number": 33, "seqno_to_time_mapping": "N/A"}} 2024-04-20T21:37:01.257 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:01 smithi189 ceph-mon[57022]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713649021011921, "job": 1, "event": "recovery_finished"} 2024-04-20T21:37:01.257 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:01 smithi189 ceph-mon[57022]: rocksdb: [db/version_set.cc:5047] Creating manifest 35 2024-04-20T21:37:01.257 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:01 smithi189 ceph-mon[57022]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-20T21:37:01.257 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:01 smithi189 ceph-mon[57022]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi189/store.db/000028.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-20T21:37:01.257 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:01 smithi189 ceph-mon[57022]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5602bec2c000 2024-04-20T21:37:01.257 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:01 smithi189 ceph-mon[57022]: rocksdb: DB pointer 0x5602bec16000 2024-04-20T21:37:01.258 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:01 smithi189 ceph-mon[57022]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-20T21:37:01.258 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:01 smithi189 ceph-mon[57022]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-20T21:37:01.258 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: ** DB Stats ** 2024-04-20T21:37:01.258 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-20T21:37:01.258 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-20T21:37:01.258 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-20T21:37:01.258 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-20T21:37:01.258 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-20T21:37:01.258 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-20T21:37:01.258 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-20T21:37:01.258 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: 2024-04-20T21:37:01.258 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: ** Compaction Stats [default] ** 2024-04-20T21:37:01.258 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-20T21:37:01.258 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-20T21:37:01.259 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: L0 1/0 2.74 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 468.6 0.01 0.00 1 0.006 0 0 0.0 0.0 2024-04-20T21:37:01.259 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: L6 1/0 7.76 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-04-20T21:37:01.259 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: Sum 2/0 10.49 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 468.6 0.01 0.00 1 0.006 0 0 0.0 0.0 2024-04-20T21:37:01.259 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 468.6 0.01 0.00 1 0.006 0 0 0.0 0.0 2024-04-20T21:37:01.259 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: 2024-04-20T21:37:01.259 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: ** Compaction Stats [default] ** 2024-04-20T21:37:01.259 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-20T21:37:01.259 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-20T21:37:01.259 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 468.6 0.01 0.00 1 0.006 0 0 0.0 0.0 2024-04-20T21:37:01.259 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: 2024-04-20T21:37:01.259 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-20T21:37:01.259 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: 2024-04-20T21:37:01.259 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-20T21:37:01.259 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: Flush(GB): cumulative 0.003, interval 0.003 2024-04-20T21:37:01.259 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-20T21:37:01.260 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-20T21:37:01.260 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-20T21:37:01.260 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-20T21:37:01.260 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: Cumulative compaction: 0.00 GB write, 111.54 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-20T21:37:01.260 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: Interval compaction: 0.00 GB write, 111.54 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-20T21:37:01.260 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-20T21:37:01.260 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: Block cache BinnedLRUCache@0x5602beb3f090#2 capacity: 512.00 MB usage: 7.11 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.5e-05 secs_since: 0 2024-04-20T21:37:01.260 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,2.03 KB,0.00038743%) IndexBlock(1,5.08 KB,0.000968575%) Misc(1,0.00 KB,0%) 2024-04-20T21:37:01.260 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: 2024-04-20T21:37:01.260 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-20T21:37:01.260 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:01 smithi189 ceph-mon[57022]: starting mon.smithi189 rank 1 at public addrs [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] at bind addrs [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi189 fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 2024-04-20T21:37:01.260 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:01 smithi189 ceph-mon[57022]: mon.smithi189@-1(???) e2 preinit fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 2024-04-20T21:37:01.260 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:01 smithi189 ceph-mon[57022]: mon.smithi189@-1(???).mds e17 new map 2024-04-20T21:37:01.260 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:01 smithi189 ceph-mon[57022]: mon.smithi189@-1(???).mds e17 print_map 2024-04-20T21:37:01.260 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: e17 2024-04-20T21:37:01.261 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T21:37:01.261 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T21:37:01.261 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: legacy client fscid: 1 2024-04-20T21:37:01.261 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: 2024-04-20T21:37:01.261 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: Filesystem 'cephfs' (1) 2024-04-20T21:37:01.261 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: fs_name cephfs 2024-04-20T21:37:01.261 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: epoch 17 2024-04-20T21:37:01.261 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-20T21:37:01.261 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: created 2024-04-20T21:27:21.032337+0000 2024-04-20T21:37:01.261 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: modified 2024-04-20T21:36:45.237235+0000 2024-04-20T21:37:01.261 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: tableserver 0 2024-04-20T21:37:01.261 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: root 0 2024-04-20T21:37:01.261 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: session_timeout 60 2024-04-20T21:37:01.261 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: session_autoclose 300 2024-04-20T21:37:01.261 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: max_file_size 1099511627776 2024-04-20T21:37:01.262 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: max_xattr_size 65536 2024-04-20T21:37:01.262 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: required_client_features {} 2024-04-20T21:37:01.262 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: last_failure 0 2024-04-20T21:37:01.262 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: last_failure_osd_epoch 0 2024-04-20T21:37:01.262 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: snap_rstat 1 2024-04-20T21:37:01.262 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T21:37:01.262 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: max_mds 2 2024-04-20T21:37:01.262 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: in 0,1 2024-04-20T21:37:01.262 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: up {0=14458,1=24279} 2024-04-20T21:37:01.262 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: failed 2024-04-20T21:37:01.262 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: damaged 2024-04-20T21:37:01.262 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: stopped 2024-04-20T21:37:01.262 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: data_pools [3] 2024-04-20T21:37:01.262 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: metadata_pool 2 2024-04-20T21:37:01.262 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: inline_data enabled 2024-04-20T21:37:01.263 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: balancer 2024-04-20T21:37:01.263 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: bal_rank_mask -1 2024-04-20T21:37:01.263 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: standby_count_wanted 1 2024-04-20T21:37:01.263 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: qdb_cluster leader: 14458 members: 14458,24279 2024-04-20T21:37:01.263 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: [mds.cephfs.smithi062.ryqblb{0:14458} state up:active seq 6 export targets 1 join_fscid=1 addr [v2:172.21.15.62:6826/1147127123,v1:172.21.15.62:6827/1147127123] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:37:01.263 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: [mds.cephfs.smithi062.ozizwi{0:14474} state up:standby-replay seq 3 join_fscid=1 addr [v2:172.21.15.62:6828/3494174293,v1:172.21.15.62:6829/3494174293] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:37:01.263 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: [mds.cephfs.smithi189.dvimer{1:24279} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.189:6824/237262514,v1:172.21.15.189:6825/237262514] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:37:01.263 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: [mds.cephfs.smithi189.xbgspw{1:24291} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.189:6826/1495356258,v1:172.21.15.189:6827/1495356258] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:37:01.263 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: 2024-04-20T21:37:01.263 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout: 2024-04-20T21:37:01.263 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:01 smithi189 ceph-mon[57022]: mon.smithi189@-1(???).osd e48 crush map has features 3314933000852226048, adjusting msgr requires 2024-04-20T21:37:01.263 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:01 smithi189 ceph-mon[57022]: mon.smithi189@-1(???).osd e48 crush map has features 288514051259236352, adjusting msgr requires 2024-04-20T21:37:01.263 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:01 smithi189 ceph-mon[57022]: mon.smithi189@-1(???).osd e48 crush map has features 288514051259236352, adjusting msgr requires 2024-04-20T21:37:01.263 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:01 smithi189 ceph-mon[57022]: mon.smithi189@-1(???).osd e48 crush map has features 288514051259236352, adjusting msgr requires 2024-04-20T21:37:01.263 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:01 smithi189 ceph-mon[57022]: mon.smithi189@-1(???).paxosservice(auth 1..25) refresh upgraded, format 0 -> 3 2024-04-20T21:37:01.264 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:01 smithi189 bash[57004]: 55aac560aab6cf464252775feab1579f97a33fad9d546da9a8dc6e47acd49d99 2024-04-20T21:37:01.264 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:01 smithi189 systemd[1]: Started Ceph mon.smithi189 for 61ae3628-ff5b-11ee-bc93-c7b262605968. 2024-04-20T21:37:02.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:02 smithi189 ceph-mon[57022]: [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T21:37:02.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:02 smithi189 ceph-mon[57022]: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T21:37:02.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:02 smithi189 ceph-mon[57022]: mon.smithi189 calling monitor election 2024-04-20T21:37:02.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:02 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mon metadata", "id": "smithi062"}]: dispatch 2024-04-20T21:37:02.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:02 smithi189 ceph-mon[57022]: mon.smithi062 calling monitor election 2024-04-20T21:37:02.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:02 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mon metadata", "id": "smithi189"}]: dispatch 2024-04-20T21:37:02.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:02 smithi189 ceph-mon[57022]: mon.smithi062 is new leader, mons smithi062,smithi189 in quorum (ranks 0,1) 2024-04-20T21:37:02.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:02 smithi189 ceph-mon[57022]: monmap epoch 3 2024-04-20T21:37:02.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:02 smithi189 ceph-mon[57022]: fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 2024-04-20T21:37:02.989 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:02 smithi189 ceph-mon[57022]: last_changed 2024-04-20T21:37:01.715360+0000 2024-04-20T21:37:02.989 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:02 smithi189 ceph-mon[57022]: created 2024-04-20T21:18:14.797594+0000 2024-04-20T21:37:02.989 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:02 smithi189 ceph-mon[57022]: min_mon_release 19 (squid) 2024-04-20T21:37:02.989 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:02 smithi189 ceph-mon[57022]: election_strategy: 1 2024-04-20T21:37:02.989 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:02 smithi189 ceph-mon[57022]: 0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.smithi062 2024-04-20T21:37:02.989 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:02 smithi189 ceph-mon[57022]: 1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.smithi189 2024-04-20T21:37:02.989 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:02 smithi189 ceph-mon[57022]: fsmap cephfs:2 {0=cephfs.smithi062.ryqblb=up:active,1=cephfs.smithi189.dvimer=up:active} 2 up:standby-replay 2024-04-20T21:37:02.989 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:02 smithi189 ceph-mon[57022]: osdmap e48: 6 total, 6 up, 6 in 2024-04-20T21:37:02.989 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:02 smithi189 ceph-mon[57022]: mgrmap e37: smithi062.rqxuxi(active, since 27s), standbys: smithi189.tawavz 2024-04-20T21:37:02.989 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:02 smithi189 ceph-mon[57022]: Health detail: HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-04-20T21:37:02.989 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:02 smithi189 ceph-mon[57022]: [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T21:37:02.989 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:02 smithi189 ceph-mon[57022]: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T21:37:02.989 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:02 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:37:02.989 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:02 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:37:02.989 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:02 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:37:03.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:02 smithi062 ceph-mon[76322]: [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T21:37:03.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:02 smithi062 ceph-mon[76322]: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T21:37:03.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:02 smithi062 ceph-mon[76322]: mon.smithi189 calling monitor election 2024-04-20T21:37:03.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:02 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mon metadata", "id": "smithi062"}]: dispatch 2024-04-20T21:37:03.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:02 smithi062 ceph-mon[76322]: mon.smithi062 calling monitor election 2024-04-20T21:37:03.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:02 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mon metadata", "id": "smithi189"}]: dispatch 2024-04-20T21:37:03.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:02 smithi062 ceph-mon[76322]: mon.smithi062 is new leader, mons smithi062,smithi189 in quorum (ranks 0,1) 2024-04-20T21:37:03.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:02 smithi062 ceph-mon[76322]: monmap epoch 3 2024-04-20T21:37:03.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:02 smithi062 ceph-mon[76322]: fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 2024-04-20T21:37:03.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:02 smithi062 ceph-mon[76322]: last_changed 2024-04-20T21:37:01.715360+0000 2024-04-20T21:37:03.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:02 smithi062 ceph-mon[76322]: created 2024-04-20T21:18:14.797594+0000 2024-04-20T21:37:03.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:02 smithi062 ceph-mon[76322]: min_mon_release 19 (squid) 2024-04-20T21:37:03.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:02 smithi062 ceph-mon[76322]: election_strategy: 1 2024-04-20T21:37:03.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:02 smithi062 ceph-mon[76322]: 0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.smithi062 2024-04-20T21:37:03.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:02 smithi062 ceph-mon[76322]: 1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.smithi189 2024-04-20T21:37:03.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:02 smithi062 ceph-mon[76322]: fsmap cephfs:2 {0=cephfs.smithi062.ryqblb=up:active,1=cephfs.smithi189.dvimer=up:active} 2 up:standby-replay 2024-04-20T21:37:03.037 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:02 smithi062 ceph-mon[76322]: osdmap e48: 6 total, 6 up, 6 in 2024-04-20T21:37:03.037 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:02 smithi062 ceph-mon[76322]: mgrmap e37: smithi062.rqxuxi(active, since 27s), standbys: smithi189.tawavz 2024-04-20T21:37:03.037 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:02 smithi062 ceph-mon[76322]: Health detail: HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-04-20T21:37:03.037 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:02 smithi062 ceph-mon[76322]: [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T21:37:03.037 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:02 smithi062 ceph-mon[76322]: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T21:37:03.037 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:02 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:37:03.037 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:02 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:37:03.037 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:02 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:37:04.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:03 smithi062 ceph-mon[76322]: pgmap v17: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:37:04.164 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:03 smithi189 ceph-mon[57022]: pgmap v17: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:37:05.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:04 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:37:05.144 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:04 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:37:06.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:05 smithi189 ceph-mon[57022]: pgmap v18: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:37:06.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:05 smithi062 ceph-mon[76322]: pgmap v18: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:37:07.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:07 smithi189 ceph-mon[57022]: pgmap v19: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:37:07.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:07 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:37:07.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:07 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:37:07.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:07 smithi189 ceph-mon[57022]: fsmap cephfs:2 {0=cephfs.smithi062.ryqblb=up:active,1=cephfs.smithi189.dvimer=up:active} 2 up:standby-replay 2024-04-20T21:37:07.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:07 smithi062 ceph-mon[76322]: pgmap v19: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:37:07.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:07 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:37:07.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:07 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:37:07.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:07 smithi062 ceph-mon[76322]: fsmap cephfs:2 {0=cephfs.smithi062.ryqblb=up:active,1=cephfs.smithi189.dvimer=up:active} 2 up:standby-replay 2024-04-20T21:37:09.728 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:09 smithi189 ceph-mon[57022]: pgmap v20: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:37:09.728 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:09 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:37:09.728 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:09 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:37:09.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:09 smithi062 ceph-mon[76322]: pgmap v20: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:37:09.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:09 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:37:09.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:09 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:37:11.435 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:11 smithi189 ceph-mon[57022]: pgmap v21: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:37:11.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:11 smithi062 ceph-mon[76322]: pgmap v21: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:37:13.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:13 smithi189 ceph-mon[57022]: pgmap v22: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:37:13.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:13 smithi062 ceph-mon[76322]: pgmap v22: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:37:15.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:14 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:37:15.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:14 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:37:15.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:14 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:37:15.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:14 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T21:37:15.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:14 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:37:15.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:14 smithi189 ceph-mon[57022]: pgmap v23: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:37:15.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:14 smithi189 ceph-mon[57022]: Reconfiguring mon.smithi062 (monmap changed)... 2024-04-20T21:37:15.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:14 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-20T21:37:15.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:14 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-20T21:37:15.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:14 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:37:15.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:14 smithi189 ceph-mon[57022]: Reconfiguring daemon mon.smithi062 on smithi062 2024-04-20T21:37:15.242 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:14 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:37:15.242 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:14 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:37:15.242 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:14 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:37:15.242 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:14 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T21:37:15.242 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:14 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:37:15.242 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:14 smithi062 ceph-mon[76322]: pgmap v23: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:37:15.242 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:14 smithi062 ceph-mon[76322]: Reconfiguring mon.smithi062 (monmap changed)... 2024-04-20T21:37:15.242 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:14 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-20T21:37:15.243 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:14 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-20T21:37:15.243 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:14 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:37:15.243 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:14 smithi062 ceph-mon[76322]: Reconfiguring daemon mon.smithi062 on smithi062 2024-04-20T21:37:17.986 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:17 smithi062 ceph-mon[76322]: pgmap v24: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:37:17.986 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:17 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:37:17.986 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:17 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:37:17.986 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:17 smithi062 ceph-mon[76322]: Reconfiguring mgr.smithi062.rqxuxi (monmap changed)... 2024-04-20T21:37:17.986 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:17 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi062.rqxuxi", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-20T21:37:17.986 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:17 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-20T21:37:17.986 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:17 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:37:17.987 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:17 smithi062 ceph-mon[76322]: Reconfiguring daemon mgr.smithi062.rqxuxi on smithi062 2024-04-20T21:37:17.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:17 smithi189 ceph-mon[57022]: pgmap v24: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:37:17.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:17 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:37:17.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:17 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:37:17.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:17 smithi189 ceph-mon[57022]: Reconfiguring mgr.smithi062.rqxuxi (monmap changed)... 2024-04-20T21:37:17.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:17 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi062.rqxuxi", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-20T21:37:17.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:17 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-20T21:37:17.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:17 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:37:17.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:17 smithi189 ceph-mon[57022]: Reconfiguring daemon mgr.smithi062.rqxuxi on smithi062 2024-04-20T21:37:19.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:19 smithi189 ceph-mon[57022]: pgmap v25: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:37:19.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:19 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:37:19.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:19 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:37:19.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:19 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi062", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-20T21:37:19.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:19 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth caps", "entity": "client.ceph-exporter.smithi062", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-20T21:37:19.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:19 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "auth caps", "entity": "client.ceph-exporter.smithi062", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]': finished 2024-04-20T21:37:19.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:19 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.ceph-exporter.smithi062"}]: dispatch 2024-04-20T21:37:19.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:19 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:37:19.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:19 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:37:20.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:19 smithi062 ceph-mon[76322]: pgmap v25: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:37:20.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:19 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:37:20.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:19 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:37:20.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:19 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi062", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-20T21:37:20.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:19 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth caps", "entity": "client.ceph-exporter.smithi062", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-20T21:37:20.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:19 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "auth caps", "entity": "client.ceph-exporter.smithi062", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]': finished 2024-04-20T21:37:20.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:19 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.ceph-exporter.smithi062"}]: dispatch 2024-04-20T21:37:20.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:19 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:37:20.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:19 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:37:20.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:20.954+0000 7fcb16a00700 1 -- 172.21.15.62:0/1222356054 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcb10073090 msgr2=0x7fcb10073460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:37:20.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:20.954+0000 7fcb16a00700 1 --2- 172.21.15.62:0/1222356054 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcb10073090 0x7fcb10073460 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fcb00009940 tx=0x7fcb00009c50 comp rx=0 tx=0).stop 2024-04-20T21:37:20.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:20.954+0000 7fcb16a00700 1 -- 172.21.15.62:0/1222356054 shutdown_connections 2024-04-20T21:37:20.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:20.954+0000 7fcb16a00700 1 --2- 172.21.15.62:0/1222356054 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcb100739a0 0x7fcb1010c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:37:20.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:20.954+0000 7fcb16a00700 1 --2- 172.21.15.62:0/1222356054 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcb10073090 0x7fcb10073460 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:37:20.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:20.954+0000 7fcb16a00700 1 -- 172.21.15.62:0/1222356054 >> 172.21.15.62:0/1222356054 conn(0x7fcb100fbda0 msgr2=0x7fcb100fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:37:20.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:20.955+0000 7fcb16a00700 1 -- 172.21.15.62:0/1222356054 shutdown_connections 2024-04-20T21:37:20.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:20.955+0000 7fcb16a00700 1 -- 172.21.15.62:0/1222356054 wait complete. 2024-04-20T21:37:20.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:20.956+0000 7fcb16a00700 1 Processor -- start 2024-04-20T21:37:20.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:20.956+0000 7fcb16a00700 1 -- start start 2024-04-20T21:37:20.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:20.956+0000 7fcb16a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcb10073090 0x7fcb10198180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:37:20.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:20.956+0000 7fcb16a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcb100739a0 0x7fcb101986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:37:20.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:20.956+0000 7fcb16a00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb10198da0 con 0x7fcb100739a0 2024-04-20T21:37:20.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:20.956+0000 7fcb16a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb1019cb30 con 0x7fcb10073090 2024-04-20T21:37:20.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:20.957+0000 7fcb0f400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcb100739a0 0x7fcb101986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:37:20.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:20.957+0000 7fcb0f400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcb100739a0 0x7fcb101986c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:38668/0 (socket says 172.21.15.62:38668) 2024-04-20T21:37:20.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:20.957+0000 7fcb0f400700 1 -- 172.21.15.62:0/995715613 learned_addr learned my addr 172.21.15.62:0/995715613 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:37:20.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:20.957+0000 7fcb0fe00700 1 --2- 172.21.15.62:0/995715613 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcb10073090 0x7fcb10198180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:37:20.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:20.957+0000 7fcb0f400700 1 -- 172.21.15.62:0/995715613 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcb10073090 msgr2=0x7fcb10198180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:37:20.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:20.958+0000 7fcb0f400700 1 --2- 172.21.15.62:0/995715613 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcb10073090 0x7fcb10198180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:37:20.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:20.958+0000 7fcb0f400700 1 -- 172.21.15.62:0/995715613 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb000095f0 con 0x7fcb100739a0 2024-04-20T21:37:20.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:20.958+0000 7fcb0f400700 1 --2- 172.21.15.62:0/995715613 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcb100739a0 0x7fcb101986c0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fcafc00c8a0 tx=0x7fcafc00cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:37:20.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:20.958+0000 7fcb0cc00700 1 -- 172.21.15.62:0/995715613 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcafc0078e0 con 0x7fcb100739a0 2024-04-20T21:37:20.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:20.959+0000 7fcb0cc00700 1 -- 172.21.15.62:0/995715613 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcafc00f450 con 0x7fcb100739a0 2024-04-20T21:37:20.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:20.959+0000 7fcb16a00700 1 -- 172.21.15.62:0/995715613 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb1019ce10 con 0x7fcb100739a0 2024-04-20T21:37:20.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:20.959+0000 7fcb16a00700 1 -- 172.21.15.62:0/995715613 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb1019d360 con 0x7fcb100739a0 2024-04-20T21:37:20.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:20.959+0000 7fcb0cc00700 1 -- 172.21.15.62:0/995715613 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcafc00e5e0 con 0x7fcb100739a0 2024-04-20T21:37:20.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:20.960+0000 7fcb16a00700 1 -- 172.21.15.62:0/995715613 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcaf0005320 con 0x7fcb100739a0 2024-04-20T21:37:20.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:20.961+0000 7fcb0cc00700 1 -- 172.21.15.62:0/995715613 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 37) v1 ==== 92847+0+0 (secure 0 0 0) 0x7fcafc007a40 con 0x7fcb100739a0 2024-04-20T21:37:20.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:20.963+0000 7fcb0cc00700 1 --2- 172.21.15.62:0/995715613 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fcaf406ee00 0x7fcaf40712b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:37:20.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:20.963+0000 7fcb0cc00700 1 -- 172.21.15.62:0/995715613 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fcafc08ee90 con 0x7fcb100739a0 2024-04-20T21:37:20.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:20.964+0000 7fcb0fe00700 1 --2- 172.21.15.62:0/995715613 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fcaf406ee00 0x7fcaf40712b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:37:20.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:20.964+0000 7fcb0fe00700 1 --2- 172.21.15.62:0/995715613 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fcaf406ee00 0x7fcaf40712b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fcb00006010 tx=0x7fcb0000b9d0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:37:20.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:20.968+0000 7fcb0cc00700 1 -- 172.21.15.62:0/995715613 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fcafc057ab0 con 0x7fcb100739a0 2024-04-20T21:37:20.968 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:20 smithi062 ceph-mon[76322]: Reconfiguring ceph-exporter.smithi062 (monmap changed)... 2024-04-20T21:37:20.968 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:20 smithi062 ceph-mon[76322]: Unable to update caps for client.ceph-exporter.smithi062 2024-04-20T21:37:20.968 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:20 smithi062 ceph-mon[76322]: Reconfiguring daemon ceph-exporter.smithi062 on smithi062 2024-04-20T21:37:20.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:20 smithi189 ceph-mon[57022]: Reconfiguring ceph-exporter.smithi062 (monmap changed)... 2024-04-20T21:37:20.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:20 smithi189 ceph-mon[57022]: Unable to update caps for client.ceph-exporter.smithi062 2024-04-20T21:37:20.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:20 smithi189 ceph-mon[57022]: Reconfiguring daemon ceph-exporter.smithi062 on smithi062 2024-04-20T21:37:21.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:21.235+0000 7fcb16a00700 1 -- 172.21.15.62:0/995715613 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fcaf0000bf0 con 0x7fcaf406ee00 2024-04-20T21:37:21.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:21.238+0000 7fcb0cc00700 1 -- 172.21.15.62:0/995715613 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+383 (secure 0 0 0) 0x7fcaf0000bf0 con 0x7fcaf406ee00 2024-04-20T21:37:21.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:21.241+0000 7fcb16a00700 1 -- 172.21.15.62:0/995715613 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fcaf406ee00 msgr2=0x7fcaf40712b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:37:21.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:21.241+0000 7fcb16a00700 1 --2- 172.21.15.62:0/995715613 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fcaf406ee00 0x7fcaf40712b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fcb00006010 tx=0x7fcb0000b9d0 comp rx=0 tx=0).stop 2024-04-20T21:37:21.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:21.241+0000 7fcb16a00700 1 -- 172.21.15.62:0/995715613 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcb100739a0 msgr2=0x7fcb101986c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:37:21.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:21.241+0000 7fcb16a00700 1 --2- 172.21.15.62:0/995715613 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcb100739a0 0x7fcb101986c0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fcafc00c8a0 tx=0x7fcafc00cbb0 comp rx=0 tx=0).stop 2024-04-20T21:37:21.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:21.241+0000 7fcb16a00700 1 -- 172.21.15.62:0/995715613 shutdown_connections 2024-04-20T21:37:21.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:21.241+0000 7fcb16a00700 1 --2- 172.21.15.62:0/995715613 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fcaf406ee00 0x7fcaf40712b0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:37:21.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:21.241+0000 7fcb16a00700 1 --2- 172.21.15.62:0/995715613 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcb10073090 0x7fcb10198180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:37:21.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:21.241+0000 7fcb16a00700 1 --2- 172.21.15.62:0/995715613 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcb100739a0 0x7fcb101986c0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:37:21.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:21.241+0000 7fcb16a00700 1 -- 172.21.15.62:0/995715613 >> 172.21.15.62:0/995715613 conn(0x7fcb100fbda0 msgr2=0x7fcb10106e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:37:21.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:21.242+0000 7fcb16a00700 1 -- 172.21.15.62:0/995715613 shutdown_connections 2024-04-20T21:37:21.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:21.242+0000 7fcb16a00700 1 -- 172.21.15.62:0/995715613 wait complete. 2024-04-20T21:37:21.257 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T21:37:21.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:21.416+0000 7fd3bbe00700 1 -- 172.21.15.62:0/2687380151 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3b4073090 msgr2=0x7fd3b4073460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:37:21.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:21.416+0000 7fd3bbe00700 1 --2- 172.21.15.62:0/2687380151 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3b4073090 0x7fd3b4073460 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd3a8009940 tx=0x7fd3a8009c50 comp rx=0 tx=0).stop 2024-04-20T21:37:21.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:21.416+0000 7fd3bbe00700 1 -- 172.21.15.62:0/2687380151 shutdown_connections 2024-04-20T21:37:21.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:21.416+0000 7fd3bbe00700 1 --2- 172.21.15.62:0/2687380151 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3b40739a0 0x7fd3b410c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:37:21.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:21.416+0000 7fd3bbe00700 1 --2- 172.21.15.62:0/2687380151 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3b4073090 0x7fd3b4073460 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:37:21.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:21.416+0000 7fd3bbe00700 1 -- 172.21.15.62:0/2687380151 >> 172.21.15.62:0/2687380151 conn(0x7fd3b40fbda0 msgr2=0x7fd3b40fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:37:21.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:21.416+0000 7fd3bbe00700 1 -- 172.21.15.62:0/2687380151 shutdown_connections 2024-04-20T21:37:21.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:21.416+0000 7fd3bbe00700 1 -- 172.21.15.62:0/2687380151 wait complete. 2024-04-20T21:37:21.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:21.417+0000 7fd3bbe00700 1 Processor -- start 2024-04-20T21:37:21.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:21.418+0000 7fd3bbe00700 1 -- start start 2024-04-20T21:37:21.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:21.418+0000 7fd3bbe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3b4073090 0x7fd3b4198180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:37:21.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:21.418+0000 7fd3bbe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3b40739a0 0x7fd3b41986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:37:21.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:21.418+0000 7fd3bbe00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3b4198da0 con 0x7fd3b4073090 2024-04-20T21:37:21.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:21.418+0000 7fd3bbe00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3b419cb30 con 0x7fd3b40739a0 2024-04-20T21:37:21.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:21.418+0000 7fd3b9400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3b4073090 0x7fd3b4198180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:37:21.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:21.418+0000 7fd3b9400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3b4073090 0x7fd3b4198180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35306/0 (socket says 172.21.15.62:35306) 2024-04-20T21:37:21.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:21.419+0000 7fd3b9400700 1 -- 172.21.15.62:0/3270214334 learned_addr learned my addr 172.21.15.62:0/3270214334 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:37:21.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:21.419+0000 7fd3b8a00700 1 --2- 172.21.15.62:0/3270214334 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3b40739a0 0x7fd3b41986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:37:21.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:21.419+0000 7fd3b9400700 1 -- 172.21.15.62:0/3270214334 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3b40739a0 msgr2=0x7fd3b41986c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:37:21.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:21.419+0000 7fd3b9400700 1 --2- 172.21.15.62:0/3270214334 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3b40739a0 0x7fd3b41986c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:37:21.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:21.419+0000 7fd3b9400700 1 -- 172.21.15.62:0/3270214334 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3a80095f0 con 0x7fd3b4073090 2024-04-20T21:37:21.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:21.419+0000 7fd3b8a00700 1 --2- 172.21.15.62:0/3270214334 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3b40739a0 0x7fd3b41986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T21:37:21.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:21.419+0000 7fd3b9400700 1 --2- 172.21.15.62:0/3270214334 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3b4073090 0x7fd3b4198180 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd3a8005620 tx=0x7fd3a8005700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:37:21.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:21.420+0000 7fd3b2000700 1 -- 172.21.15.62:0/3270214334 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd3a801d070 con 0x7fd3b4073090 2024-04-20T21:37:21.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:21.420+0000 7fd3b2000700 1 -- 172.21.15.62:0/3270214334 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd3a800f460 con 0x7fd3b4073090 2024-04-20T21:37:21.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:21.420+0000 7fd3bbe00700 1 -- 172.21.15.62:0/3270214334 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3b419cdb0 con 0x7fd3b4073090 2024-04-20T21:37:21.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:21.420+0000 7fd3b2000700 1 -- 172.21.15.62:0/3270214334 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd3a8021640 con 0x7fd3b4073090 2024-04-20T21:37:21.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:21.420+0000 7fd3bbe00700 1 -- 172.21.15.62:0/3270214334 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3b419d2a0 con 0x7fd3b4073090 2024-04-20T21:37:21.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:21.421+0000 7fd3bbe00700 1 -- 172.21.15.62:0/3270214334 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3b4109ce0 con 0x7fd3b4073090 2024-04-20T21:37:21.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:21.422+0000 7fd3b2000700 1 -- 172.21.15.62:0/3270214334 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 37) v1 ==== 92847+0+0 (secure 0 0 0) 0x7fd3a800fa40 con 0x7fd3b4073090 2024-04-20T21:37:21.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:21.423+0000 7fd3b2000700 1 --2- 172.21.15.62:0/3270214334 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd39c06edf0 0x7fd39c0712a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:37:21.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:21.423+0000 7fd3b2000700 1 -- 172.21.15.62:0/3270214334 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fd3a8090450 con 0x7fd3b4073090 2024-04-20T21:37:21.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:21.424+0000 7fd3b8a00700 1 --2- 172.21.15.62:0/3270214334 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd39c06edf0 0x7fd39c0712a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:37:21.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:21.424+0000 7fd3b8a00700 1 --2- 172.21.15.62:0/3270214334 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd39c06edf0 0x7fd39c0712a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd3a0005fd0 tx=0x7fd3a0005dc0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:37:21.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:21.428+0000 7fd3b2000700 1 -- 172.21.15.62:0/3270214334 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fd3a8058ff0 con 0x7fd3b4073090 2024-04-20T21:37:21.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:21.704+0000 7fd3bbe00700 1 -- 172.21.15.62:0/3270214334 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd3b41995f0 con 0x7fd39c06edf0 2024-04-20T21:37:21.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:21.706+0000 7fd3b2000700 1 -- 172.21.15.62:0/3270214334 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+383 (secure 0 0 0) 0x7fd3b41995f0 con 0x7fd39c06edf0 2024-04-20T21:37:21.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:21.710+0000 7fd3bbe00700 1 -- 172.21.15.62:0/3270214334 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd39c06edf0 msgr2=0x7fd39c0712a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:37:21.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:21.710+0000 7fd3bbe00700 1 --2- 172.21.15.62:0/3270214334 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd39c06edf0 0x7fd39c0712a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd3a0005fd0 tx=0x7fd3a0005dc0 comp rx=0 tx=0).stop 2024-04-20T21:37:21.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:21.710+0000 7fd3bbe00700 1 -- 172.21.15.62:0/3270214334 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3b4073090 msgr2=0x7fd3b4198180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:37:21.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:21.710+0000 7fd3bbe00700 1 --2- 172.21.15.62:0/3270214334 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3b4073090 0x7fd3b4198180 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd3a8005620 tx=0x7fd3a8005700 comp rx=0 tx=0).stop 2024-04-20T21:37:21.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:21.710+0000 7fd3bbe00700 1 -- 172.21.15.62:0/3270214334 shutdown_connections 2024-04-20T21:37:21.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:21.710+0000 7fd3bbe00700 1 --2- 172.21.15.62:0/3270214334 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd39c06edf0 0x7fd39c0712a0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:37:21.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:21.710+0000 7fd3bbe00700 1 --2- 172.21.15.62:0/3270214334 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3b4073090 0x7fd3b4198180 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:37:21.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:21.710+0000 7fd3bbe00700 1 --2- 172.21.15.62:0/3270214334 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3b40739a0 0x7fd3b41986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:37:21.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:21.710+0000 7fd3bbe00700 1 -- 172.21.15.62:0/3270214334 >> 172.21.15.62:0/3270214334 conn(0x7fd3b40fbda0 msgr2=0x7fd3b4106e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:37:21.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:21.710+0000 7fd3bbe00700 1 -- 172.21.15.62:0/3270214334 shutdown_connections 2024-04-20T21:37:21.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:21.710+0000 7fd3bbe00700 1 -- 172.21.15.62:0/3270214334 wait complete. 2024-04-20T21:37:21.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:21.872+0000 7f55ed400700 1 -- 172.21.15.62:0/2296013393 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55e80ffbd0 msgr2=0x7f55e8100040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:37:21.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:21.872+0000 7f55ed400700 1 --2- 172.21.15.62:0/2296013393 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55e80ffbd0 0x7f55e8100040 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f55d4009910 tx=0x7f55d4009c20 comp rx=0 tx=0).stop 2024-04-20T21:37:21.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:21.872+0000 7f55ed400700 1 -- 172.21.15.62:0/2296013393 shutdown_connections 2024-04-20T21:37:21.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:21.872+0000 7f55ed400700 1 --2- 172.21.15.62:0/2296013393 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55e80ffbd0 0x7f55e8100040 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:37:21.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:21.872+0000 7f55ed400700 1 --2- 172.21.15.62:0/2296013393 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f55e8104130 0x7f55e8104500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:37:21.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:21.872+0000 7f55ed400700 1 -- 172.21.15.62:0/2296013393 >> 172.21.15.62:0/2296013393 conn(0x7f55e8075130 msgr2=0x7f55e8075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:37:21.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:21.872+0000 7f55ed400700 1 -- 172.21.15.62:0/2296013393 shutdown_connections 2024-04-20T21:37:21.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:21.872+0000 7f55ed400700 1 -- 172.21.15.62:0/2296013393 wait complete. 2024-04-20T21:37:21.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:21.873+0000 7f55ed400700 1 Processor -- start 2024-04-20T21:37:21.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:21.873+0000 7f55ed400700 1 -- start start 2024-04-20T21:37:21.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:21.874+0000 7f55ed400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f55e80ffbd0 0x7f55e8195e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:37:21.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:21.874+0000 7f55ed400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55e8104130 0x7f55e81963b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:37:21.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:21.874+0000 7f55ed400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55e8196a00 con 0x7f55e8104130 2024-04-20T21:37:21.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:21.874+0000 7f55ed400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55e8196b40 con 0x7f55e80ffbd0 2024-04-20T21:37:21.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:21.874+0000 7f55e5e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55e8104130 0x7f55e81963b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:37:21.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:21.874+0000 7f55e5e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55e8104130 0x7f55e81963b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35324/0 (socket says 172.21.15.62:35324) 2024-04-20T21:37:21.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:21.874+0000 7f55e5e00700 1 -- 172.21.15.62:0/7305610 learned_addr learned my addr 172.21.15.62:0/7305610 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:37:21.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:21.874+0000 7f55e6800700 1 --2- 172.21.15.62:0/7305610 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f55e80ffbd0 0x7f55e8195e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:37:21.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:21.875+0000 7f55e5e00700 1 -- 172.21.15.62:0/7305610 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f55e80ffbd0 msgr2=0x7f55e8195e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:37:21.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:21.875+0000 7f55e5e00700 1 --2- 172.21.15.62:0/7305610 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f55e80ffbd0 0x7f55e8195e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:37:21.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:21.875+0000 7f55e5e00700 1 -- 172.21.15.62:0/7305610 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55d40095f0 con 0x7f55e8104130 2024-04-20T21:37:21.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:21.875+0000 7f55e5e00700 1 --2- 172.21.15.62:0/7305610 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55e8104130 0x7f55e81963b0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f55d4006010 tx=0x7f55d400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:37:21.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:21.875+0000 7f55df400700 1 -- 172.21.15.62:0/7305610 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f55d401d070 con 0x7f55e8104130 2024-04-20T21:37:21.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:21.875+0000 7f55df400700 1 -- 172.21.15.62:0/7305610 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f55d4004fb0 con 0x7f55e8104130 2024-04-20T21:37:21.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:21.875+0000 7f55df400700 1 -- 172.21.15.62:0/7305610 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f55d400f690 con 0x7f55e8104130 2024-04-20T21:37:21.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:21.876+0000 7f55ed400700 1 -- 172.21.15.62:0/7305610 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55e819a930 con 0x7f55e8104130 2024-04-20T21:37:21.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:21.876+0000 7f55ed400700 1 -- 172.21.15.62:0/7305610 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55e819ae20 con 0x7f55e8104130 2024-04-20T21:37:21.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:21.877+0000 7f55ed400700 1 -- 172.21.15.62:0/7305610 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55e81086a0 con 0x7f55e8104130 2024-04-20T21:37:21.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:21.878+0000 7f55df400700 1 -- 172.21.15.62:0/7305610 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 37) v1 ==== 92847+0+0 (secure 0 0 0) 0x7f55d4022470 con 0x7f55e8104130 2024-04-20T21:37:21.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:21.879+0000 7f55df400700 1 --2- 172.21.15.62:0/7305610 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f55cc06ee00 0x7f55cc0712b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:37:21.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:21.879+0000 7f55df400700 1 -- 172.21.15.62:0/7305610 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f55d40908b0 con 0x7f55e8104130 2024-04-20T21:37:21.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:21.880+0000 7f55e6800700 1 --2- 172.21.15.62:0/7305610 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f55cc06ee00 0x7f55cc0712b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:37:21.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:21.880+0000 7f55e6800700 1 --2- 172.21.15.62:0/7305610 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f55cc06ee00 0x7f55cc0712b0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f55d8005ea0 tx=0x7f55d8005e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:37:21.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:21.884+0000 7f55df400700 1 -- 172.21.15.62:0/7305610 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f55d40595f0 con 0x7f55e8104130 2024-04-20T21:37:21.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:21 smithi189 ceph-mon[57022]: pgmap v26: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-20T21:37:21.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:21 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:37:21.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:21 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:37:21.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:21 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi062", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-20T21:37:21.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:21 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:37:22.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:21 smithi062 ceph-mon[76322]: pgmap v26: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-20T21:37:22.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:21 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:37:22.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:21 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:37:22.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:21 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi062", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-20T21:37:22.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:21 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:37:22.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:22.133+0000 7f55ed400700 1 -- 172.21.15.62:0/7305610 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f55e819b100 con 0x7f55cc06ee00 2024-04-20T21:37:22.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:22.145+0000 7f55df400700 1 -- 172.21.15.62:0/7305610 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f55e819b100 con 0x7f55cc06ee00 2024-04-20T21:37:22.145 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T21:37:22.145 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (3m) 40s ago 17m 14.4M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T21:37:22.145 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (17m) 40s ago 17m 8338k - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T21:37:22.145 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (16m) 15s ago 16m 8196k - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T21:37:22.145 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (17m) 40s ago 17m 6656k - 18.2.0 dc2bc1663786 36e10ef5be39 2024-04-20T21:37:22.145 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (15m) 15s ago 15m 6665k - 18.2.0 dc2bc1663786 4101f9953e3d 2024-04-20T21:37:22.145 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (2m) 40s ago 16m 74.3M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T21:37:22.145 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (9m) 40s ago 9m 65.4M - 18.2.0 dc2bc1663786 5e6460aa6159 2024-04-20T21:37:22.145 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (9m) 40s ago 9m 132M - 18.2.0 dc2bc1663786 e65f2267544e 2024-04-20T21:37:22.146 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (9m) 15s ago 9m 117M - 18.2.0 dc2bc1663786 fb2a8a223db8 2024-04-20T21:37:22.146 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (9m) 15s ago 9m 55.6M - 18.2.0 dc2bc1663786 f2eec2ee8951 2024-04-20T21:37:22.146 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (6m) 40s ago 18m 476M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T21:37:22.146 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (4m) 15s ago 15m 449M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T21:37:22.146 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (58s) 40s ago 19m 29.3M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 019eb1479485 2024-04-20T21:37:22.146 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (21s) 15s ago 15m 20.2M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 55aac560aab6 2024-04-20T21:37:22.146 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (4m) 40s ago 17m 7512k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T21:37:22.146 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (4m) 15s ago 15m 7482k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T21:37:22.146 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (14m) 40s ago 14m 218M 1216M 18.2.0 dc2bc1663786 3f315c3afc1f 2024-04-20T21:37:22.146 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (14m) 40s ago 14m 210M 1216M 18.2.0 dc2bc1663786 44e4d07416ad 2024-04-20T21:37:22.146 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (13m) 40s ago 13m 173M 1216M 18.2.0 dc2bc1663786 31c5d9586e98 2024-04-20T21:37:22.146 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (12m) 15s ago 12m 241M 2240M 18.2.0 dc2bc1663786 611c1e242693 2024-04-20T21:37:22.146 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (12m) 15s ago 12m 220M 2240M 18.2.0 dc2bc1663786 6c8fce91d45d 2024-04-20T21:37:22.146 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (11m) 15s ago 11m 206M 2240M 18.2.0 dc2bc1663786 8e4574bae8db 2024-04-20T21:37:22.147 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (3m) 40s ago 16m 71.4M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T21:37:22.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:22.148+0000 7f55ed400700 1 -- 172.21.15.62:0/7305610 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f55cc06ee00 msgr2=0x7f55cc0712b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:37:22.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:22.148+0000 7f55ed400700 1 --2- 172.21.15.62:0/7305610 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f55cc06ee00 0x7f55cc0712b0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f55d8005ea0 tx=0x7f55d8005e30 comp rx=0 tx=0).stop 2024-04-20T21:37:22.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:22.148+0000 7f55ed400700 1 -- 172.21.15.62:0/7305610 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55e8104130 msgr2=0x7f55e81963b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:37:22.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:22.148+0000 7f55ed400700 1 --2- 172.21.15.62:0/7305610 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55e8104130 0x7f55e81963b0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f55d4006010 tx=0x7f55d400be20 comp rx=0 tx=0).stop 2024-04-20T21:37:22.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:22.148+0000 7f55ed400700 1 -- 172.21.15.62:0/7305610 shutdown_connections 2024-04-20T21:37:22.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:22.148+0000 7f55ed400700 1 --2- 172.21.15.62:0/7305610 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f55cc06ee00 0x7f55cc0712b0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:37:22.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:22.148+0000 7f55ed400700 1 --2- 172.21.15.62:0/7305610 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f55e80ffbd0 0x7f55e8195e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:37:22.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:22.148+0000 7f55ed400700 1 --2- 172.21.15.62:0/7305610 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55e8104130 0x7f55e81963b0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:37:22.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:22.148+0000 7f55ed400700 1 -- 172.21.15.62:0/7305610 >> 172.21.15.62:0/7305610 conn(0x7f55e8075130 msgr2=0x7f55e80febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:37:22.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:22.149+0000 7f55ed400700 1 -- 172.21.15.62:0/7305610 shutdown_connections 2024-04-20T21:37:22.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:22.149+0000 7f55ed400700 1 -- 172.21.15.62:0/7305610 wait complete. 2024-04-20T21:37:22.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:22.310+0000 7f13c3a00700 1 -- 172.21.15.62:0/321206926 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f13bc073090 msgr2=0x7f13bc073460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:37:22.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:22.310+0000 7f13c3a00700 1 --2- 172.21.15.62:0/321206926 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f13bc073090 0x7f13bc073460 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f13b0009940 tx=0x7f13b0009c50 comp rx=0 tx=0).stop 2024-04-20T21:37:22.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:22.310+0000 7f13c3a00700 1 -- 172.21.15.62:0/321206926 shutdown_connections 2024-04-20T21:37:22.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:22.310+0000 7f13c3a00700 1 --2- 172.21.15.62:0/321206926 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f13bc0739a0 0x7f13bc10c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:37:22.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:22.310+0000 7f13c3a00700 1 --2- 172.21.15.62:0/321206926 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f13bc073090 0x7f13bc073460 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:37:22.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:22.310+0000 7f13c3a00700 1 -- 172.21.15.62:0/321206926 >> 172.21.15.62:0/321206926 conn(0x7f13bc0fbd80 msgr2=0x7f13bc0fe190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:37:22.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:22.311+0000 7f13c3a00700 1 -- 172.21.15.62:0/321206926 shutdown_connections 2024-04-20T21:37:22.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:22.311+0000 7f13c3a00700 1 -- 172.21.15.62:0/321206926 wait complete. 2024-04-20T21:37:22.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:22.312+0000 7f13c3a00700 1 Processor -- start 2024-04-20T21:37:22.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:22.312+0000 7f13c3a00700 1 -- start start 2024-04-20T21:37:22.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:22.312+0000 7f13c3a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f13bc073090 0x7f13bc1980e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:37:22.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:22.312+0000 7f13c3a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f13bc0739a0 0x7f13bc198620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:37:22.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:22.312+0000 7f13c3a00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13bc198d00 con 0x7f13bc073090 2024-04-20T21:37:22.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:22.312+0000 7f13c3a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13bc19ca40 con 0x7f13bc0739a0 2024-04-20T21:37:22.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:22.313+0000 7f13c1000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f13bc073090 0x7f13bc1980e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:37:22.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:22.313+0000 7f13c1000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f13bc073090 0x7f13bc1980e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35334/0 (socket says 172.21.15.62:35334) 2024-04-20T21:37:22.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:22.313+0000 7f13c1000700 1 -- 172.21.15.62:0/3317197009 learned_addr learned my addr 172.21.15.62:0/3317197009 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:37:22.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:22.313+0000 7f13bbe00700 1 --2- 172.21.15.62:0/3317197009 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f13bc0739a0 0x7f13bc198620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:37:22.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:22.313+0000 7f13c1000700 1 -- 172.21.15.62:0/3317197009 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f13bc0739a0 msgr2=0x7f13bc198620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:37:22.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:22.313+0000 7f13c1000700 1 --2- 172.21.15.62:0/3317197009 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f13bc0739a0 0x7f13bc198620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:37:22.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:22.313+0000 7f13c1000700 1 -- 172.21.15.62:0/3317197009 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13b00095f0 con 0x7f13bc073090 2024-04-20T21:37:22.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:22.314+0000 7f13c1000700 1 --2- 172.21.15.62:0/3317197009 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f13bc073090 0x7f13bc1980e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f13b000b870 tx=0x7f13b00056a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:37:22.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:22.314+0000 7f13b9600700 1 -- 172.21.15.62:0/3317197009 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f13b001d070 con 0x7f13bc073090 2024-04-20T21:37:22.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:22.314+0000 7f13b9600700 1 -- 172.21.15.62:0/3317197009 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f13b000f460 con 0x7f13bc073090 2024-04-20T21:37:22.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:22.314+0000 7f13c3a00700 1 -- 172.21.15.62:0/3317197009 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13bc19ccc0 con 0x7f13bc073090 2024-04-20T21:37:22.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:22.314+0000 7f13b9600700 1 -- 172.21.15.62:0/3317197009 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f13b0021640 con 0x7f13bc073090 2024-04-20T21:37:22.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:22.314+0000 7f13c3a00700 1 -- 172.21.15.62:0/3317197009 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13bc19d1b0 con 0x7f13bc073090 2024-04-20T21:37:22.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:22.315+0000 7f13c3a00700 1 -- 172.21.15.62:0/3317197009 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13bc109cc0 con 0x7f13bc073090 2024-04-20T21:37:22.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:22.317+0000 7f13b9600700 1 -- 172.21.15.62:0/3317197009 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 37) v1 ==== 92847+0+0 (secure 0 0 0) 0x7f13b000fa60 con 0x7f13bc073090 2024-04-20T21:37:22.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:22.318+0000 7f13b9600700 1 --2- 172.21.15.62:0/3317197009 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f13a406edf0 0x7f13a40712a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:37:22.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:22.318+0000 7f13b9600700 1 -- 172.21.15.62:0/3317197009 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f13b00903a0 con 0x7f13bc073090 2024-04-20T21:37:22.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:22.318+0000 7f13bbe00700 1 --2- 172.21.15.62:0/3317197009 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f13a406edf0 0x7f13a40712a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:37:22.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:22.320+0000 7f13bbe00700 1 --2- 172.21.15.62:0/3317197009 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f13a406edf0 0x7f13a40712a0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f13a8005fd0 tx=0x7f13a800a560 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:37:22.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:22.324+0000 7f13b9600700 1 -- 172.21.15.62:0/3317197009 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f13b0058f40 con 0x7f13bc073090 2024-04-20T21:37:22.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:22.698+0000 7f13c3a00700 1 -- 172.21.15.62:0/3317197009 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f13bc04f2a0 con 0x7f13bc073090 2024-04-20T21:37:22.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:22.699+0000 7f13b9600700 1 -- 172.21.15.62:0/3317197009 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+688 (secure 0 0 0) 0x7f13b005c600 con 0x7f13bc073090 2024-04-20T21:37:22.698 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:37:22.698 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T21:37:22.698 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:37:22.699 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:37:22.699 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T21:37:22.699 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:37:22.699 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:37:22.699 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T21:37:22.699 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 6 2024-04-20T21:37:22.699 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:37:22.699 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T21:37:22.699 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 4 2024-04-20T21:37:22.699 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:37:22.699 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T21:37:22.699 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 10, 2024-04-20T21:37:22.699 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 4 2024-04-20T21:37:22.699 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T21:37:22.699 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:37:22.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:22.702+0000 7f13c3a00700 1 -- 172.21.15.62:0/3317197009 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f13a406edf0 msgr2=0x7f13a40712a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:37:22.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:22.702+0000 7f13c3a00700 1 --2- 172.21.15.62:0/3317197009 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f13a406edf0 0x7f13a40712a0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f13a8005fd0 tx=0x7f13a800a560 comp rx=0 tx=0).stop 2024-04-20T21:37:22.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:22.702+0000 7f13c3a00700 1 -- 172.21.15.62:0/3317197009 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f13bc073090 msgr2=0x7f13bc1980e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:37:22.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:22.702+0000 7f13c3a00700 1 --2- 172.21.15.62:0/3317197009 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f13bc073090 0x7f13bc1980e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f13b000b870 tx=0x7f13b00056a0 comp rx=0 tx=0).stop 2024-04-20T21:37:22.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:22.703+0000 7f13c3a00700 1 -- 172.21.15.62:0/3317197009 shutdown_connections 2024-04-20T21:37:22.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:22.703+0000 7f13c3a00700 1 --2- 172.21.15.62:0/3317197009 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f13a406edf0 0x7f13a40712a0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:37:22.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:22.703+0000 7f13c3a00700 1 --2- 172.21.15.62:0/3317197009 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f13bc073090 0x7f13bc1980e0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:37:22.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:22.703+0000 7f13c3a00700 1 --2- 172.21.15.62:0/3317197009 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f13bc0739a0 0x7f13bc198620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:37:22.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:22.703+0000 7f13c3a00700 1 -- 172.21.15.62:0/3317197009 >> 172.21.15.62:0/3317197009 conn(0x7f13bc0fbd80 msgr2=0x7f13bc106e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:37:22.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:22.703+0000 7f13c3a00700 1 -- 172.21.15.62:0/3317197009 shutdown_connections 2024-04-20T21:37:22.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:22.703+0000 7f13c3a00700 1 -- 172.21.15.62:0/3317197009 wait complete. 2024-04-20T21:37:22.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:22.856+0000 7ff803200700 1 -- 172.21.15.62:0/3105695662 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff804101c30 msgr2=0x7ff8041020a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:37:22.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:22.856+0000 7ff803200700 1 --2- 172.21.15.62:0/3105695662 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff804101c30 0x7ff8041020a0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff7f0009910 tx=0x7ff7f0009c20 comp rx=0 tx=0).stop 2024-04-20T21:37:22.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:22.856+0000 7ff803200700 1 -- 172.21.15.62:0/3105695662 shutdown_connections 2024-04-20T21:37:22.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:22.856+0000 7ff803200700 1 --2- 172.21.15.62:0/3105695662 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff804101c30 0x7ff8041020a0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:37:22.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:22.856+0000 7ff803200700 1 --2- 172.21.15.62:0/3105695662 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8041003d0 0x7ff8041007a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:37:22.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:22.856+0000 7ff803200700 1 -- 172.21.15.62:0/3105695662 >> 172.21.15.62:0/3105695662 conn(0x7ff8040fbbd0 msgr2=0x7ff8040fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:37:22.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:22.856+0000 7ff803200700 1 -- 172.21.15.62:0/3105695662 shutdown_connections 2024-04-20T21:37:22.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:22.856+0000 7ff803200700 1 -- 172.21.15.62:0/3105695662 wait complete. 2024-04-20T21:37:22.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:22.857+0000 7ff803200700 1 Processor -- start 2024-04-20T21:37:22.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:22.857+0000 7ff803200700 1 -- start start 2024-04-20T21:37:22.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:22.858+0000 7ff803200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8041003d0 0x7ff804195e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:37:22.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:22.858+0000 7ff803200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff804101c30 0x7ff804196350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:37:22.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:22.858+0000 7ff803200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff804196a30 con 0x7ff804101c30 2024-04-20T21:37:22.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:22.858+0000 7ff803200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff80419a7c0 con 0x7ff8041003d0 2024-04-20T21:37:22.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:22.858+0000 7ff801400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff804101c30 0x7ff804196350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:37:22.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:22.858+0000 7ff801400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff804101c30 0x7ff804196350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35352/0 (socket says 172.21.15.62:35352) 2024-04-20T21:37:22.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:22.858+0000 7ff801400700 1 -- 172.21.15.62:0/3799271789 learned_addr learned my addr 172.21.15.62:0/3799271789 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:37:22.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:22.858+0000 7ff801e00700 1 --2- 172.21.15.62:0/3799271789 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8041003d0 0x7ff804195e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:37:22.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:22.859+0000 7ff801400700 1 -- 172.21.15.62:0/3799271789 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8041003d0 msgr2=0x7ff804195e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:37:22.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:22.859+0000 7ff801400700 1 --2- 172.21.15.62:0/3799271789 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8041003d0 0x7ff804195e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:37:22.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:22.859+0000 7ff801400700 1 -- 172.21.15.62:0/3799271789 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7f00095f0 con 0x7ff804101c30 2024-04-20T21:37:22.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:22.859+0000 7ff801400700 1 --2- 172.21.15.62:0/3799271789 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff804101c30 0x7ff804196350 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff7f0006010 tx=0x7ff7f000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:37:22.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:22.859+0000 7ff7faa00700 1 -- 172.21.15.62:0/3799271789 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff7f001d070 con 0x7ff804101c30 2024-04-20T21:37:22.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:22.859+0000 7ff7faa00700 1 -- 172.21.15.62:0/3799271789 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff7f0004fb0 con 0x7ff804101c30 2024-04-20T21:37:22.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:22.859+0000 7ff7faa00700 1 -- 172.21.15.62:0/3799271789 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff7f0022ea0 con 0x7ff804101c30 2024-04-20T21:37:22.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:22.859+0000 7ff803200700 1 -- 172.21.15.62:0/3799271789 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff80419aa40 con 0x7ff804101c30 2024-04-20T21:37:22.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:22.860+0000 7ff803200700 1 -- 172.21.15.62:0/3799271789 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff80419af30 con 0x7ff804101c30 2024-04-20T21:37:22.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:22.861+0000 7ff803200700 1 -- 172.21.15.62:0/3799271789 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8041086b0 con 0x7ff804101c30 2024-04-20T21:37:22.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:22.861+0000 7ff7faa00700 1 -- 172.21.15.62:0/3799271789 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 37) v1 ==== 92847+0+0 (secure 0 0 0) 0x7ff7f0022470 con 0x7ff804101c30 2024-04-20T21:37:22.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:22.862+0000 7ff7faa00700 1 --2- 172.21.15.62:0/3799271789 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff7e806ee00 0x7ff7e80712b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:37:22.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:22.862+0000 7ff7faa00700 1 -- 172.21.15.62:0/3799271789 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6328+0+0 (secure 0 0 0) 0x7ff7f0090520 con 0x7ff804101c30 2024-04-20T21:37:22.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:22.863+0000 7ff801e00700 1 --2- 172.21.15.62:0/3799271789 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff7e806ee00 0x7ff7e80712b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:37:22.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:22.863+0000 7ff801e00700 1 --2- 172.21.15.62:0/3799271789 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff7e806ee00 0x7ff7e80712b0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7ff7f4006fd0 tx=0x7ff7f4008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:37:22.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:22.867+0000 7ff7faa00700 1 -- 172.21.15.62:0/3799271789 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7ff7f0059290 con 0x7ff804101c30 2024-04-20T21:37:22.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:22 smithi189 ceph-mon[57022]: from='client.34158 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:37:22.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:22 smithi189 ceph-mon[57022]: Reconfiguring crash.smithi062 (monmap changed)... 2024-04-20T21:37:22.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:22 smithi189 ceph-mon[57022]: Reconfiguring daemon crash.smithi062 on smithi062 2024-04-20T21:37:22.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:22 smithi189 ceph-mon[57022]: from='client.34162 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:37:22.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:22 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/3317197009' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:37:23.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:22 smithi062 ceph-mon[76322]: from='client.34158 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:37:23.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:22 smithi062 ceph-mon[76322]: Reconfiguring crash.smithi062 (monmap changed)... 2024-04-20T21:37:23.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:22 smithi062 ceph-mon[76322]: Reconfiguring daemon crash.smithi062 on smithi062 2024-04-20T21:37:23.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:22 smithi062 ceph-mon[76322]: from='client.34162 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:37:23.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:22 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/3317197009' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:37:23.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:23.155+0000 7ff803200700 1 -- 172.21.15.62:0/3799271789 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7ff804062380 con 0x7ff804101c30 2024-04-20T21:37:23.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:23.156+0000 7ff7faa00700 1 -- 172.21.15.62:0/3799271789 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 18 v18) v1 ==== 76+0+1928 (secure 0 0 0) 0x7ff7f00277a0 con 0x7ff804101c30 2024-04-20T21:37:23.156 INFO:teuthology.orchestra.run.smithi062.stdout:e18 2024-04-20T21:37:23.156 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T21:37:23.156 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T21:37:23.157 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T21:37:23.157 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:37:23.157 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T21:37:23.157 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T21:37:23.157 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 18 2024-04-20T21:37:23.157 INFO:teuthology.orchestra.run.smithi062.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-20T21:37:23.157 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T21:37:23.157 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T21:37:06.018300+0000 2024-04-20T21:37:23.157 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T21:37:23.157 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T21:37:23.157 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T21:37:23.157 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T21:37:23.157 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T21:37:23.157 INFO:teuthology.orchestra.run.smithi062.stdout:max_xattr_size 65536 2024-04-20T21:37:23.157 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T21:37:23.158 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T21:37:23.158 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 0 2024-04-20T21:37:23.158 INFO:teuthology.orchestra.run.smithi062.stdout:snap_rstat 1 2024-04-20T21:37:23.158 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T21:37:23.158 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 2 2024-04-20T21:37:23.158 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1 2024-04-20T21:37:23.158 INFO:teuthology.orchestra.run.smithi062.stdout:up {0=14458,1=24279} 2024-04-20T21:37:23.158 INFO:teuthology.orchestra.run.smithi062.stdout:failed 2024-04-20T21:37:23.158 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T21:37:23.158 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T21:37:23.158 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T21:37:23.158 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T21:37:23.158 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data enabled 2024-04-20T21:37:23.158 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T21:37:23.159 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T21:37:23.159 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 1 2024-04-20T21:37:23.159 INFO:teuthology.orchestra.run.smithi062.stdout:qdb_cluster leader: 14458 members: 24279,14458 2024-04-20T21:37:23.159 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ryqblb{0:14458} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.62:6826/1147127123,v1:172.21.15.62:6827/1147127123] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:37:23.159 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ozizwi{0:14474} state up:standby-replay seq 3 join_fscid=1 addr [v2:172.21.15.62:6828/3494174293,v1:172.21.15.62:6829/3494174293] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:37:23.159 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.dvimer{1:24279} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.189:6824/237262514,v1:172.21.15.189:6825/237262514] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:37:23.159 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.xbgspw{1:24291} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.189:6826/1495356258,v1:172.21.15.189:6827/1495356258] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:37:23.159 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:37:23.159 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:37:23.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:23.159+0000 7ff803200700 1 -- 172.21.15.62:0/3799271789 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff7e806ee00 msgr2=0x7ff7e80712b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:37:23.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:23.159+0000 7ff803200700 1 --2- 172.21.15.62:0/3799271789 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff7e806ee00 0x7ff7e80712b0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7ff7f4006fd0 tx=0x7ff7f4008040 comp rx=0 tx=0).stop 2024-04-20T21:37:23.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:23.159+0000 7ff803200700 1 -- 172.21.15.62:0/3799271789 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff804101c30 msgr2=0x7ff804196350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:37:23.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:23.159+0000 7ff803200700 1 --2- 172.21.15.62:0/3799271789 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff804101c30 0x7ff804196350 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff7f0006010 tx=0x7ff7f000be20 comp rx=0 tx=0).stop 2024-04-20T21:37:23.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:23.159+0000 7ff803200700 1 -- 172.21.15.62:0/3799271789 shutdown_connections 2024-04-20T21:37:23.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:23.159+0000 7ff803200700 1 --2- 172.21.15.62:0/3799271789 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff7e806ee00 0x7ff7e80712b0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:37:23.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:23.159+0000 7ff803200700 1 --2- 172.21.15.62:0/3799271789 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8041003d0 0x7ff804195e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:37:23.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:23.159+0000 7ff803200700 1 --2- 172.21.15.62:0/3799271789 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff804101c30 0x7ff804196350 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:37:23.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:23.159+0000 7ff803200700 1 -- 172.21.15.62:0/3799271789 >> 172.21.15.62:0/3799271789 conn(0x7ff8040fbbd0 msgr2=0x7ff8040fc600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:37:23.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:23.159+0000 7ff803200700 1 -- 172.21.15.62:0/3799271789 shutdown_connections 2024-04-20T21:37:23.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:23.160+0000 7ff803200700 1 -- 172.21.15.62:0/3799271789 wait complete. 2024-04-20T21:37:23.165 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 18 2024-04-20T21:37:23.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:23.320+0000 7fc53fe00700 1 -- 172.21.15.62:0/2315404387 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc538073a00 msgr2=0x7fc53810c580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:37:23.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:23.320+0000 7fc53fe00700 1 --2- 172.21.15.62:0/2315404387 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc538073a00 0x7fc53810c580 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc524009910 tx=0x7fc524009c20 comp rx=0 tx=0).stop 2024-04-20T21:37:23.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:23.320+0000 7fc53fe00700 1 -- 172.21.15.62:0/2315404387 shutdown_connections 2024-04-20T21:37:23.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:23.320+0000 7fc53fe00700 1 --2- 172.21.15.62:0/2315404387 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc538073a00 0x7fc53810c580 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:37:23.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:23.320+0000 7fc53fe00700 1 --2- 172.21.15.62:0/2315404387 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5380730f0 0x7fc5380734c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:37:23.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:23.320+0000 7fc53fe00700 1 -- 172.21.15.62:0/2315404387 >> 172.21.15.62:0/2315404387 conn(0x7fc5380fbda0 msgr2=0x7fc5380fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:37:23.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:23.320+0000 7fc53fe00700 1 -- 172.21.15.62:0/2315404387 shutdown_connections 2024-04-20T21:37:23.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:23.320+0000 7fc53fe00700 1 -- 172.21.15.62:0/2315404387 wait complete. 2024-04-20T21:37:23.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:23.321+0000 7fc53fe00700 1 Processor -- start 2024-04-20T21:37:23.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:23.321+0000 7fc53fe00700 1 -- start start 2024-04-20T21:37:23.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:23.321+0000 7fc53fe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc5380730f0 0x7fc538197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:37:23.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:23.321+0000 7fc53fe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc538073a00 0x7fc538198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:37:23.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:23.321+0000 7fc53fe00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc538198be0 con 0x7fc5380730f0 2024-04-20T21:37:23.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:23.321+0000 7fc53fe00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc53819c970 con 0x7fc538073a00 2024-04-20T21:37:23.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:23.322+0000 7fc53d400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc5380730f0 0x7fc538197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:37:23.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:23.322+0000 7fc53d400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc5380730f0 0x7fc538197fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35374/0 (socket says 172.21.15.62:35374) 2024-04-20T21:37:23.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:23.322+0000 7fc53d400700 1 -- 172.21.15.62:0/4066405314 learned_addr learned my addr 172.21.15.62:0/4066405314 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:37:23.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:23.322+0000 7fc53ca00700 1 --2- 172.21.15.62:0/4066405314 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc538073a00 0x7fc538198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:37:23.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:23.322+0000 7fc53d400700 1 -- 172.21.15.62:0/4066405314 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc538073a00 msgr2=0x7fc538198500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:37:23.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:23.322+0000 7fc53d400700 1 --2- 172.21.15.62:0/4066405314 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc538073a00 0x7fc538198500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:37:23.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:23.322+0000 7fc53d400700 1 -- 172.21.15.62:0/4066405314 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5240095f0 con 0x7fc5380730f0 2024-04-20T21:37:23.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:23.323+0000 7fc53d400700 1 --2- 172.21.15.62:0/4066405314 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc5380730f0 0x7fc538197fc0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc52c00c930 tx=0x7fc52c00ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:37:23.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:23.323+0000 7fc536000700 1 -- 172.21.15.62:0/4066405314 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc52c007ad0 con 0x7fc5380730f0 2024-04-20T21:37:23.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:23.323+0000 7fc536000700 1 -- 172.21.15.62:0/4066405314 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc52c007c30 con 0x7fc5380730f0 2024-04-20T21:37:23.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:23.323+0000 7fc536000700 1 -- 172.21.15.62:0/4066405314 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc52c018610 con 0x7fc5380730f0 2024-04-20T21:37:23.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:23.323+0000 7fc53fe00700 1 -- 172.21.15.62:0/4066405314 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc53819cc50 con 0x7fc5380730f0 2024-04-20T21:37:23.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:23.323+0000 7fc53fe00700 1 -- 172.21.15.62:0/4066405314 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc53819d1a0 con 0x7fc5380730f0 2024-04-20T21:37:23.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:23.325+0000 7fc53fe00700 1 -- 172.21.15.62:0/4066405314 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc538109c80 con 0x7fc5380730f0 2024-04-20T21:37:23.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:23.326+0000 7fc536000700 1 -- 172.21.15.62:0/4066405314 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 37) v1 ==== 92847+0+0 (secure 0 0 0) 0x7fc52c00f450 con 0x7fc5380730f0 2024-04-20T21:37:23.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:23.326+0000 7fc536000700 1 --2- 172.21.15.62:0/4066405314 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc52006edf0 0x7fc5200712a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:37:23.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:23.327+0000 7fc536000700 1 -- 172.21.15.62:0/4066405314 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fc52c08f220 con 0x7fc5380730f0 2024-04-20T21:37:23.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:23.327+0000 7fc53ca00700 1 --2- 172.21.15.62:0/4066405314 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc52006edf0 0x7fc5200712a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:37:23.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:23.328+0000 7fc53ca00700 1 --2- 172.21.15.62:0/4066405314 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc52006edf0 0x7fc5200712a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fc5381995e0 tx=0x7fc52401a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:37:23.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:23.331+0000 7fc536000700 1 -- 172.21.15.62:0/4066405314 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fc52c057e40 con 0x7fc5380730f0 2024-04-20T21:37:23.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:23.596+0000 7fc53fe00700 1 -- 172.21.15.62:0/4066405314 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc53819d420 con 0x7fc52006edf0 2024-04-20T21:37:23.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:23.598+0000 7fc536000700 1 -- 172.21.15.62:0/4066405314 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+383 (secure 0 0 0) 0x7fc53819d420 con 0x7fc52006edf0 2024-04-20T21:37:23.598 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:37:23.598 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T21:37:23.598 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T21:37:23.598 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-20T21:37:23.598 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T21:37:23.598 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr", 2024-04-20T21:37:23.598 INFO:teuthology.orchestra.run.smithi062.stdout: "mon" 2024-04-20T21:37:23.599 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T21:37:23.599 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "4/23 daemons upgraded", 2024-04-20T21:37:23.599 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading mon daemons", 2024-04-20T21:37:23.599 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T21:37:23.599 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:37:23.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:23.601+0000 7fc53fe00700 1 -- 172.21.15.62:0/4066405314 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc52006edf0 msgr2=0x7fc5200712a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:37:23.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:23.601+0000 7fc53fe00700 1 --2- 172.21.15.62:0/4066405314 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc52006edf0 0x7fc5200712a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fc5381995e0 tx=0x7fc52401a040 comp rx=0 tx=0).stop 2024-04-20T21:37:23.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:23.601+0000 7fc53fe00700 1 -- 172.21.15.62:0/4066405314 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc5380730f0 msgr2=0x7fc538197fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:37:23.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:23.601+0000 7fc53fe00700 1 --2- 172.21.15.62:0/4066405314 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc5380730f0 0x7fc538197fc0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc52c00c930 tx=0x7fc52c00ccf0 comp rx=0 tx=0).stop 2024-04-20T21:37:23.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:23.601+0000 7fc53fe00700 1 -- 172.21.15.62:0/4066405314 shutdown_connections 2024-04-20T21:37:23.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:23.601+0000 7fc53fe00700 1 --2- 172.21.15.62:0/4066405314 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc52006edf0 0x7fc5200712a0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:37:23.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:23.601+0000 7fc53fe00700 1 --2- 172.21.15.62:0/4066405314 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc5380730f0 0x7fc538197fc0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:37:23.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:23.601+0000 7fc53fe00700 1 --2- 172.21.15.62:0/4066405314 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc538073a00 0x7fc538198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:37:23.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:23.601+0000 7fc53fe00700 1 -- 172.21.15.62:0/4066405314 >> 172.21.15.62:0/4066405314 conn(0x7fc5380fbda0 msgr2=0x7fc538106dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:37:23.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:23.602+0000 7fc53fe00700 1 -- 172.21.15.62:0/4066405314 shutdown_connections 2024-04-20T21:37:23.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:23.602+0000 7fc53fe00700 1 -- 172.21.15.62:0/4066405314 wait complete. 2024-04-20T21:37:23.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:23.756+0000 7f62c5400700 1 -- 172.21.15.62:0/704228541 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f62c00ffc40 msgr2=0x7f62c0103c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:37:23.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:23.756+0000 7f62c5400700 1 --2- 172.21.15.62:0/704228541 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f62c00ffc40 0x7f62c0103c10 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f62ac009910 tx=0x7f62ac009c20 comp rx=0 tx=0).stop 2024-04-20T21:37:23.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:23.756+0000 7f62c5400700 1 -- 172.21.15.62:0/704228541 shutdown_connections 2024-04-20T21:37:23.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:23.756+0000 7f62c5400700 1 --2- 172.21.15.62:0/704228541 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f62c00ffc40 0x7f62c0103c10 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:37:23.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:23.756+0000 7f62c5400700 1 --2- 172.21.15.62:0/704228541 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62c00ff2a0 0x7f62c00ff670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:37:23.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:23.756+0000 7f62c5400700 1 -- 172.21.15.62:0/704228541 >> 172.21.15.62:0/704228541 conn(0x7f62c00fab60 msgr2=0x7f62c00fcf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:37:23.756 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:23 smithi062 ceph-mon[76322]: from='client.34166 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:37:23.757 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:23 smithi062 ceph-mon[76322]: pgmap v27: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:37:23.757 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:23 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/3799271789' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T21:37:23.757 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:23 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:37:23.757 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:23 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:37:23.757 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:23 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-20T21:37:23.757 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:23 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:37:23.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:23.756+0000 7f62c5400700 1 -- 172.21.15.62:0/704228541 shutdown_connections 2024-04-20T21:37:23.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:23.756+0000 7f62c5400700 1 -- 172.21.15.62:0/704228541 wait complete. 2024-04-20T21:37:23.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:23.758+0000 7f62c5400700 1 Processor -- start 2024-04-20T21:37:23.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:23.758+0000 7f62c5400700 1 -- start start 2024-04-20T21:37:23.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:23.758+0000 7f62c5400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f62c00ff2a0 0x7f62c0197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:37:23.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:23.758+0000 7f62c5400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62c00ffc40 0x7f62c0198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:37:23.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:23.758+0000 7f62c5400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62c0198b30 con 0x7f62c00ff2a0 2024-04-20T21:37:23.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:23.758+0000 7f62c5400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62c019c8c0 con 0x7f62c00ffc40 2024-04-20T21:37:23.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:23.759+0000 7f62be800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f62c00ff2a0 0x7f62c0197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:37:23.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:23.759+0000 7f62be800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f62c00ff2a0 0x7f62c0197f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35396/0 (socket says 172.21.15.62:35396) 2024-04-20T21:37:23.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:23.759+0000 7f62be800700 1 -- 172.21.15.62:0/139994350 learned_addr learned my addr 172.21.15.62:0/139994350 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:37:23.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:23.759+0000 7f62bde00700 1 --2- 172.21.15.62:0/139994350 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62c00ffc40 0x7f62c0198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:37:23.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:23.759+0000 7f62be800700 1 -- 172.21.15.62:0/139994350 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62c00ffc40 msgr2=0x7f62c0198450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:37:23.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:23.759+0000 7f62be800700 1 --2- 172.21.15.62:0/139994350 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62c00ffc40 0x7f62c0198450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:37:23.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:23.759+0000 7f62be800700 1 -- 172.21.15.62:0/139994350 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62ac0095f0 con 0x7f62c00ff2a0 2024-04-20T21:37:23.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:23.760+0000 7f62be800700 1 --2- 172.21.15.62:0/139994350 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f62c00ff2a0 0x7f62c0197f10 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f62b000d8d0 tx=0x7f62b000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:37:23.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:23.760+0000 7f62b7400700 1 -- 172.21.15.62:0/139994350 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f62b0009940 con 0x7f62c00ff2a0 2024-04-20T21:37:23.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:23.760+0000 7f62b7400700 1 -- 172.21.15.62:0/139994350 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f62b0010460 con 0x7f62c00ff2a0 2024-04-20T21:37:23.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:23.760+0000 7f62b7400700 1 -- 172.21.15.62:0/139994350 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f62b000f5f0 con 0x7f62c00ff2a0 2024-04-20T21:37:23.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:23.760+0000 7f62c5400700 1 -- 172.21.15.62:0/139994350 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62c019cba0 con 0x7f62c00ff2a0 2024-04-20T21:37:23.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:23.761+0000 7f62c5400700 1 -- 172.21.15.62:0/139994350 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62c019d0f0 con 0x7f62c00ff2a0 2024-04-20T21:37:23.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:23.762+0000 7f62c5400700 1 -- 172.21.15.62:0/139994350 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62c004f2a0 con 0x7f62c00ff2a0 2024-04-20T21:37:23.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:23.763+0000 7f62b7400700 1 -- 172.21.15.62:0/139994350 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 37) v1 ==== 92847+0+0 (secure 0 0 0) 0x7f62b00105d0 con 0x7f62c00ff2a0 2024-04-20T21:37:23.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:23.764+0000 7f62b7400700 1 --2- 172.21.15.62:0/139994350 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f62a406edf0 0x7f62a40712a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:37:23.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:23.764+0000 7f62b7400700 1 -- 172.21.15.62:0/139994350 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f62b008eff0 con 0x7f62c00ff2a0 2024-04-20T21:37:23.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:23.765+0000 7f62bde00700 1 --2- 172.21.15.62:0/139994350 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f62a406edf0 0x7f62a40712a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:37:23.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:23.766+0000 7f62bde00700 1 --2- 172.21.15.62:0/139994350 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f62a406edf0 0x7f62a40712a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f62c0199530 tx=0x7f62ac01a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:37:23.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:23.769+0000 7f62b7400700 1 -- 172.21.15.62:0/139994350 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f62b00c3ff0 con 0x7f62c00ff2a0 2024-04-20T21:37:23.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:23 smithi189 ceph-mon[57022]: from='client.34166 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:37:23.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:23 smithi189 ceph-mon[57022]: pgmap v27: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:37:23.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:23 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/3799271789' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T21:37:23.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:23 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:37:23.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:23 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:37:23.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:23 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-20T21:37:23.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:23 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:37:24.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:24.103+0000 7f62c5400700 1 -- 172.21.15.62:0/139994350 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f62c0199270 con 0x7f62c00ff2a0 2024-04-20T21:37:24.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:24.104+0000 7f62b7400700 1 -- 172.21.15.62:0/139994350 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7f62b00160e0 con 0x7f62c00ff2a0 2024-04-20T21:37:24.103 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-04-20T21:37:24.103 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T21:37:24.103 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T21:37:24.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:24.107+0000 7f62c5400700 1 -- 172.21.15.62:0/139994350 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f62a406edf0 msgr2=0x7f62a40712a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:37:24.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:24.107+0000 7f62c5400700 1 --2- 172.21.15.62:0/139994350 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f62a406edf0 0x7f62a40712a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f62c0199530 tx=0x7f62ac01a040 comp rx=0 tx=0).stop 2024-04-20T21:37:24.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:24.107+0000 7f62c5400700 1 -- 172.21.15.62:0/139994350 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f62c00ff2a0 msgr2=0x7f62c0197f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:37:24.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:24.107+0000 7f62c5400700 1 --2- 172.21.15.62:0/139994350 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f62c00ff2a0 0x7f62c0197f10 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f62b000d8d0 tx=0x7f62b000dc90 comp rx=0 tx=0).stop 2024-04-20T21:37:24.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:24.108+0000 7f62c5400700 1 -- 172.21.15.62:0/139994350 shutdown_connections 2024-04-20T21:37:24.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:24.108+0000 7f62c5400700 1 --2- 172.21.15.62:0/139994350 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f62a406edf0 0x7f62a40712a0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:37:24.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:24.108+0000 7f62c5400700 1 --2- 172.21.15.62:0/139994350 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f62c00ff2a0 0x7f62c0197f10 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:37:24.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:24.108+0000 7f62c5400700 1 --2- 172.21.15.62:0/139994350 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62c00ffc40 0x7f62c0198450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:37:24.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:24.108+0000 7f62c5400700 1 -- 172.21.15.62:0/139994350 >> 172.21.15.62:0/139994350 conn(0x7f62c00fab60 msgr2=0x7f62c0068dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:37:24.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:24.108+0000 7f62c5400700 1 -- 172.21.15.62:0/139994350 shutdown_connections 2024-04-20T21:37:24.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:24.108+0000 7f62c5400700 1 -- 172.21.15.62:0/139994350 wait complete. 2024-04-20T21:37:25.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:24 smithi062 ceph-mon[76322]: from='client.34178 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:37:25.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:24 smithi062 ceph-mon[76322]: Reconfiguring osd.0 (monmap changed)... 2024-04-20T21:37:25.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:24 smithi062 ceph-mon[76322]: Reconfiguring daemon osd.0 on smithi062 2024-04-20T21:37:25.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:24 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/139994350' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T21:37:25.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:24 smithi189 ceph-mon[57022]: from='client.34178 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:37:25.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:24 smithi189 ceph-mon[57022]: Reconfiguring osd.0 (monmap changed)... 2024-04-20T21:37:25.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:24 smithi189 ceph-mon[57022]: Reconfiguring daemon osd.0 on smithi062 2024-04-20T21:37:25.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:24 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/139994350' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T21:37:25.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:25 smithi062 ceph-mon[76322]: pgmap v28: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:37:26.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:25 smithi189 ceph-mon[57022]: pgmap v28: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:37:27.283 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:27 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:37:27.283 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:27 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:37:27.283 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:27 smithi062 ceph-mon[76322]: Reconfiguring osd.1 (monmap changed)... 2024-04-20T21:37:27.283 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:27 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-20T21:37:27.283 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:27 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:37:27.283 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:27 smithi062 ceph-mon[76322]: Reconfiguring daemon osd.1 on smithi062 2024-04-20T21:37:27.284 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:27 smithi062 ceph-mon[76322]: pgmap v29: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:37:27.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:27 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:37:27.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:27 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:37:27.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:27 smithi189 ceph-mon[57022]: Reconfiguring osd.1 (monmap changed)... 2024-04-20T21:37:27.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:27 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-20T21:37:27.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:27 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:37:27.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:27 smithi189 ceph-mon[57022]: Reconfiguring daemon osd.1 on smithi062 2024-04-20T21:37:27.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:27 smithi189 ceph-mon[57022]: pgmap v29: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:37:29.713 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:29 smithi062 ceph-mon[76322]: pgmap v30: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:37:29.713 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:29 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:37:29.713 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:29 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:37:29.713 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:29 smithi062 ceph-mon[76322]: Reconfiguring osd.2 (monmap changed)... 2024-04-20T21:37:29.714 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:29 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-20T21:37:29.714 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:29 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:37:29.714 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:29 smithi062 ceph-mon[76322]: Reconfiguring daemon osd.2 on smithi062 2024-04-20T21:37:29.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:29 smithi189 ceph-mon[57022]: pgmap v30: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:37:29.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:29 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:37:29.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:29 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:37:29.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:29 smithi189 ceph-mon[57022]: Reconfiguring osd.2 (monmap changed)... 2024-04-20T21:37:29.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:29 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-20T21:37:29.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:29 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:37:29.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:29 smithi189 ceph-mon[57022]: Reconfiguring daemon osd.2 on smithi062 2024-04-20T21:37:31.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:31 smithi062 ceph-mon[76322]: pgmap v31: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-20T21:37:31.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:31 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:37:31.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:31 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:37:31.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:31 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi062.ryqblb", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-20T21:37:31.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:31 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:37:31.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:31 smithi189 ceph-mon[57022]: pgmap v31: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-20T21:37:31.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:31 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:37:31.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:31 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:37:31.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:31 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi062.ryqblb", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-20T21:37:31.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:31 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:37:32.770 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:32 smithi062 ceph-mon[76322]: Reconfiguring mds.cephfs.smithi062.ryqblb (monmap changed)... 2024-04-20T21:37:32.770 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:32 smithi062 ceph-mon[76322]: Reconfiguring daemon mds.cephfs.smithi062.ryqblb on smithi062 2024-04-20T21:37:32.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:32 smithi189 ceph-mon[57022]: Reconfiguring mds.cephfs.smithi062.ryqblb (monmap changed)... 2024-04-20T21:37:32.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:32 smithi189 ceph-mon[57022]: Reconfiguring daemon mds.cephfs.smithi062.ryqblb on smithi062 2024-04-20T21:37:33.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:33 smithi062 ceph-mon[76322]: pgmap v32: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-20T21:37:33.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:33 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:37:33.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:33 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:37:33.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:33 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi062.ozizwi", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-20T21:37:33.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:33 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:37:33.965 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:33 smithi189 ceph-mon[57022]: pgmap v32: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-20T21:37:33.965 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:33 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:37:33.965 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:33 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:37:33.965 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:33 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi062.ozizwi", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-20T21:37:33.965 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:33 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:37:34.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:34 smithi062 ceph-mon[76322]: Reconfiguring mds.cephfs.smithi062.ozizwi (monmap changed)... 2024-04-20T21:37:34.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:34 smithi062 ceph-mon[76322]: Reconfiguring daemon mds.cephfs.smithi062.ozizwi on smithi062 2024-04-20T21:37:34.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:34 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:37:34.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:34 smithi189 ceph-mon[57022]: Reconfiguring mds.cephfs.smithi062.ozizwi (monmap changed)... 2024-04-20T21:37:34.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:34 smithi189 ceph-mon[57022]: Reconfiguring daemon mds.cephfs.smithi062.ozizwi on smithi062 2024-04-20T21:37:34.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:34 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:37:35.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:35 smithi062 ceph-mon[76322]: pgmap v33: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:37:35.989 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:35 smithi189 ceph-mon[57022]: pgmap v33: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:37:36.802 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:36 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:37:36.802 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:36 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:37:36.803 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:36 smithi189 ceph-mon[57022]: Reconfiguring ceph-exporter.smithi189 (monmap changed)... 2024-04-20T21:37:36.803 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:36 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi189", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-20T21:37:36.803 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:36 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth caps", "entity": "client.ceph-exporter.smithi189", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-20T21:37:36.803 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:36 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "auth caps", "entity": "client.ceph-exporter.smithi189", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]': finished 2024-04-20T21:37:36.803 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:36 smithi189 ceph-mon[57022]: Unable to update caps for client.ceph-exporter.smithi189 2024-04-20T21:37:36.803 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:36 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.ceph-exporter.smithi189"}]: dispatch 2024-04-20T21:37:36.803 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:36 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:37:36.803 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:36 smithi189 ceph-mon[57022]: Reconfiguring daemon ceph-exporter.smithi189 on smithi189 2024-04-20T21:37:37.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:36 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:37:37.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:36 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:37:37.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:36 smithi062 ceph-mon[76322]: Reconfiguring ceph-exporter.smithi189 (monmap changed)... 2024-04-20T21:37:37.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:36 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi189", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-20T21:37:37.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:36 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth caps", "entity": "client.ceph-exporter.smithi189", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-20T21:37:37.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:36 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "auth caps", "entity": "client.ceph-exporter.smithi189", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]': finished 2024-04-20T21:37:37.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:36 smithi062 ceph-mon[76322]: Unable to update caps for client.ceph-exporter.smithi189 2024-04-20T21:37:37.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:36 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.ceph-exporter.smithi189"}]: dispatch 2024-04-20T21:37:37.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:36 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:37:37.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:36 smithi062 ceph-mon[76322]: Reconfiguring daemon ceph-exporter.smithi189 on smithi189 2024-04-20T21:37:37.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:37 smithi062 ceph-mon[76322]: pgmap v34: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:37:37.986 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:37 smithi189 ceph-mon[57022]: pgmap v34: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:37:39.110 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:38 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:37:39.110 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:38 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:37:39.110 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:38 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi189", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-20T21:37:39.111 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:38 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:37:39.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:38 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:37:39.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:38 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:37:39.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:38 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi189", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-20T21:37:39.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:38 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:37:40.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:39 smithi189 ceph-mon[57022]: Reconfiguring crash.smithi189 (monmap changed)... 2024-04-20T21:37:40.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:39 smithi189 ceph-mon[57022]: Reconfiguring daemon crash.smithi189 on smithi189 2024-04-20T21:37:40.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:39 smithi189 ceph-mon[57022]: pgmap v35: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:37:40.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:39 smithi062 ceph-mon[76322]: Reconfiguring crash.smithi189 (monmap changed)... 2024-04-20T21:37:40.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:39 smithi062 ceph-mon[76322]: Reconfiguring daemon crash.smithi189 on smithi189 2024-04-20T21:37:40.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:39 smithi062 ceph-mon[76322]: pgmap v35: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:37:41.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:41 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:37:41.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:41 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:37:41.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:41 smithi189 ceph-mon[57022]: Reconfiguring mgr.smithi189.tawavz (monmap changed)... 2024-04-20T21:37:41.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:41 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi189.tawavz", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-20T21:37:41.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:41 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-20T21:37:41.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:41 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:37:41.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:41 smithi189 ceph-mon[57022]: Reconfiguring daemon mgr.smithi189.tawavz on smithi189 2024-04-20T21:37:41.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:41 smithi189 ceph-mon[57022]: pgmap v36: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:37:41.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:41 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:37:41.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:41 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:37:41.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:41 smithi062 ceph-mon[76322]: Reconfiguring mgr.smithi189.tawavz (monmap changed)... 2024-04-20T21:37:41.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:41 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi189.tawavz", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-20T21:37:41.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:41 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-20T21:37:41.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:41 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:37:41.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:41 smithi062 ceph-mon[76322]: Reconfiguring daemon mgr.smithi189.tawavz on smithi189 2024-04-20T21:37:41.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:41 smithi062 ceph-mon[76322]: pgmap v36: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:37:43.597 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:43 smithi189 ceph-mon[57022]: pgmap v37: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-20T21:37:43.598 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:43 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:37:43.598 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:43 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:37:43.598 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:43 smithi189 ceph-mon[57022]: Reconfiguring mon.smithi189 (monmap changed)... 2024-04-20T21:37:43.598 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:43 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-20T21:37:43.598 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:43 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-20T21:37:43.598 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:43 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:37:43.598 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:43 smithi189 ceph-mon[57022]: Reconfiguring daemon mon.smithi189 on smithi189 2024-04-20T21:37:43.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:43 smithi062 ceph-mon[76322]: pgmap v37: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-20T21:37:43.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:43 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:37:43.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:43 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:37:43.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:43 smithi062 ceph-mon[76322]: Reconfiguring mon.smithi189 (monmap changed)... 2024-04-20T21:37:43.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:43 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-20T21:37:43.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:43 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-20T21:37:43.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:43 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:37:43.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:43 smithi062 ceph-mon[76322]: Reconfiguring daemon mon.smithi189 on smithi189 2024-04-20T21:37:45.976 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:45 smithi189 ceph-mon[57022]: pgmap v38: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:37:45.976 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:45 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:37:45.976 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:45 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:37:45.976 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:45 smithi189 ceph-mon[57022]: Reconfiguring osd.3 (monmap changed)... 2024-04-20T21:37:45.976 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:45 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-20T21:37:45.976 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:45 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:37:45.976 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:45 smithi189 ceph-mon[57022]: Reconfiguring daemon osd.3 on smithi189 2024-04-20T21:37:46.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:45 smithi062 ceph-mon[76322]: pgmap v38: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:37:46.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:45 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:37:46.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:45 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:37:46.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:45 smithi062 ceph-mon[76322]: Reconfiguring osd.3 (monmap changed)... 2024-04-20T21:37:46.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:45 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-20T21:37:46.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:45 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:37:46.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:45 smithi062 ceph-mon[76322]: Reconfiguring daemon osd.3 on smithi189 2024-04-20T21:37:48.101 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:47 smithi189 ceph-mon[57022]: pgmap v39: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:37:48.101 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:47 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:37:48.101 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:47 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:37:48.101 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:47 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-20T21:37:48.101 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:47 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:37:48.143 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:47 smithi062 ceph-mon[76322]: pgmap v39: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:37:48.144 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:47 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:37:48.144 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:47 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:37:48.144 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:47 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-20T21:37:48.144 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:47 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:37:49.232 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:48 smithi189 ceph-mon[57022]: Reconfiguring osd.4 (monmap changed)... 2024-04-20T21:37:49.233 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:48 smithi189 ceph-mon[57022]: Reconfiguring daemon osd.4 on smithi189 2024-04-20T21:37:49.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:48 smithi062 ceph-mon[76322]: Reconfiguring osd.4 (monmap changed)... 2024-04-20T21:37:49.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:48 smithi062 ceph-mon[76322]: Reconfiguring daemon osd.4 on smithi189 2024-04-20T21:37:49.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:49 smithi189 ceph-mon[57022]: pgmap v40: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:37:49.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:49 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:37:49.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:49 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:37:49.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:49 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:37:49.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:49 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-20T21:37:49.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:49 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:37:50.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:49 smithi062 ceph-mon[76322]: pgmap v40: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:37:50.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:49 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:37:50.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:49 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:37:50.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:49 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:37:50.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:49 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-20T21:37:50.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:49 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:37:50.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:50 smithi189 ceph-mon[57022]: Reconfiguring osd.5 (monmap changed)... 2024-04-20T21:37:50.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:50 smithi189 ceph-mon[57022]: Reconfiguring daemon osd.5 on smithi189 2024-04-20T21:37:51.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:50 smithi062 ceph-mon[76322]: Reconfiguring osd.5 (monmap changed)... 2024-04-20T21:37:51.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:50 smithi062 ceph-mon[76322]: Reconfiguring daemon osd.5 on smithi189 2024-04-20T21:37:52.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:51 smithi062 ceph-mon[76322]: pgmap v41: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:37:52.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:51 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:37:52.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:51 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:37:52.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:51 smithi189 ceph-mon[57022]: pgmap v41: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:37:52.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:51 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:37:52.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:51 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:37:53.119 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:52 smithi189 ceph-mon[57022]: Reconfiguring mds.cephfs.smithi189.dvimer (monmap changed)... 2024-04-20T21:37:53.119 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:52 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi189.dvimer", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-20T21:37:53.120 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:52 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:37:53.120 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:52 smithi189 ceph-mon[57022]: Reconfiguring daemon mds.cephfs.smithi189.dvimer on smithi189 2024-04-20T21:37:53.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:52 smithi062 ceph-mon[76322]: Reconfiguring mds.cephfs.smithi189.dvimer (monmap changed)... 2024-04-20T21:37:53.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:52 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi189.dvimer", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-20T21:37:53.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:52 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:37:53.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:52 smithi062 ceph-mon[76322]: Reconfiguring daemon mds.cephfs.smithi189.dvimer on smithi189 2024-04-20T21:37:54.236 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:53 smithi189 ceph-mon[57022]: pgmap v42: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:37:54.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:53 smithi062 ceph-mon[76322]: pgmap v42: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:37:54.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:54.288+0000 7f5e87400700 1 -- 172.21.15.62:0/2023799958 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e800687a0 msgr2=0x7f5e80068b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:37:54.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:54.288+0000 7f5e87400700 1 --2- 172.21.15.62:0/2023799958 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e800687a0 0x7f5e80068b70 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5e74009910 tx=0x7f5e74009c20 comp rx=0 tx=0).stop 2024-04-20T21:37:54.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:54.288+0000 7f5e87400700 1 -- 172.21.15.62:0/2023799958 shutdown_connections 2024-04-20T21:37:54.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:54.288+0000 7f5e87400700 1 --2- 172.21.15.62:0/2023799958 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e80069140 0x7f5e80104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:37:54.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:54.288+0000 7f5e87400700 1 --2- 172.21.15.62:0/2023799958 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e800687a0 0x7f5e80068b70 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:37:54.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:54.288+0000 7f5e87400700 1 -- 172.21.15.62:0/2023799958 >> 172.21.15.62:0/2023799958 conn(0x7f5e80075130 msgr2=0x7f5e80075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:37:54.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:54.288+0000 7f5e87400700 1 -- 172.21.15.62:0/2023799958 shutdown_connections 2024-04-20T21:37:54.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:54.288+0000 7f5e87400700 1 -- 172.21.15.62:0/2023799958 wait complete. 2024-04-20T21:37:54.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:54.289+0000 7f5e87400700 1 Processor -- start 2024-04-20T21:37:54.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:54.289+0000 7f5e87400700 1 -- start start 2024-04-20T21:37:54.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:54.290+0000 7f5e87400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e800687a0 0x7f5e80195ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:37:54.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:54.290+0000 7f5e87400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e80069140 0x7f5e80196420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:37:54.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:54.290+0000 7f5e87400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e80196b00 con 0x7f5e800687a0 2024-04-20T21:37:54.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:54.290+0000 7f5e87400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e8019a890 con 0x7f5e80069140 2024-04-20T21:37:54.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:54.290+0000 7f5e84a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e800687a0 0x7f5e80195ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:37:54.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:54.290+0000 7f5e84a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e800687a0 0x7f5e80195ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44998/0 (socket says 172.21.15.62:44998) 2024-04-20T21:37:54.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:54.290+0000 7f5e84a00700 1 -- 172.21.15.62:0/1743056795 learned_addr learned my addr 172.21.15.62:0/1743056795 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:37:54.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:54.290+0000 7f5e7fe00700 1 --2- 172.21.15.62:0/1743056795 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e80069140 0x7f5e80196420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:37:54.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:54.291+0000 7f5e84a00700 1 -- 172.21.15.62:0/1743056795 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e80069140 msgr2=0x7f5e80196420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:37:54.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:54.291+0000 7f5e84a00700 1 --2- 172.21.15.62:0/1743056795 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e80069140 0x7f5e80196420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:37:54.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:54.291+0000 7f5e84a00700 1 -- 172.21.15.62:0/1743056795 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e740095f0 con 0x7f5e800687a0 2024-04-20T21:37:54.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:54.291+0000 7f5e84a00700 1 --2- 172.21.15.62:0/1743056795 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e800687a0 0x7f5e80195ee0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f5e7400b5c0 tx=0x7f5e7400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:37:54.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:54.291+0000 7f5e7d600700 1 -- 172.21.15.62:0/1743056795 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5e7401d070 con 0x7f5e800687a0 2024-04-20T21:37:54.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:54.291+0000 7f5e87400700 1 -- 172.21.15.62:0/1743056795 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e8019ab10 con 0x7f5e800687a0 2024-04-20T21:37:54.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:54.291+0000 7f5e7d600700 1 -- 172.21.15.62:0/1743056795 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5e74005e00 con 0x7f5e800687a0 2024-04-20T21:37:54.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:54.291+0000 7f5e7d600700 1 -- 172.21.15.62:0/1743056795 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5e74017400 con 0x7f5e800687a0 2024-04-20T21:37:54.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:54.292+0000 7f5e87400700 1 -- 172.21.15.62:0/1743056795 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e8019b000 con 0x7f5e800687a0 2024-04-20T21:37:54.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:54.293+0000 7f5e87400700 1 -- 172.21.15.62:0/1743056795 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e80108790 con 0x7f5e800687a0 2024-04-20T21:37:54.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:54.294+0000 7f5e7d600700 1 -- 172.21.15.62:0/1743056795 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 37) v1 ==== 92847+0+0 (secure 0 0 0) 0x7f5e7400f6f0 con 0x7f5e800687a0 2024-04-20T21:37:54.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:54.295+0000 7f5e7d600700 1 --2- 172.21.15.62:0/1743056795 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5e6806edf0 0x7f5e680712a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:37:54.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:54.295+0000 7f5e7d600700 1 -- 172.21.15.62:0/1743056795 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f5e740904a0 con 0x7f5e800687a0 2024-04-20T21:37:54.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:54.295+0000 7f5e7fe00700 1 --2- 172.21.15.62:0/1743056795 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5e6806edf0 0x7f5e680712a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:37:54.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:54.296+0000 7f5e7fe00700 1 --2- 172.21.15.62:0/1743056795 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5e6806edf0 0x7f5e680712a0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f5e80197500 tx=0x7f5e6c00b410 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:37:54.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:54.300+0000 7f5e7d600700 1 -- 172.21.15.62:0/1743056795 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f5e740590c0 con 0x7f5e800687a0 2024-04-20T21:37:54.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:54.555+0000 7f5e87400700 1 -- 172.21.15.62:0/1743056795 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5e8019b270 con 0x7f5e6806edf0 2024-04-20T21:37:54.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:54.558+0000 7f5e7d600700 1 -- 172.21.15.62:0/1743056795 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+383 (secure 0 0 0) 0x7f5e8019b270 con 0x7f5e6806edf0 2024-04-20T21:37:54.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:54.561+0000 7f5e87400700 1 -- 172.21.15.62:0/1743056795 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5e6806edf0 msgr2=0x7f5e680712a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:37:54.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:54.561+0000 7f5e87400700 1 --2- 172.21.15.62:0/1743056795 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5e6806edf0 0x7f5e680712a0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f5e80197500 tx=0x7f5e6c00b410 comp rx=0 tx=0).stop 2024-04-20T21:37:54.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:54.561+0000 7f5e87400700 1 -- 172.21.15.62:0/1743056795 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e800687a0 msgr2=0x7f5e80195ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:37:54.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:54.561+0000 7f5e87400700 1 --2- 172.21.15.62:0/1743056795 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e800687a0 0x7f5e80195ee0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f5e7400b5c0 tx=0x7f5e7400bd70 comp rx=0 tx=0).stop 2024-04-20T21:37:54.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:54.561+0000 7f5e87400700 1 -- 172.21.15.62:0/1743056795 shutdown_connections 2024-04-20T21:37:54.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:54.561+0000 7f5e87400700 1 --2- 172.21.15.62:0/1743056795 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5e6806edf0 0x7f5e680712a0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:37:54.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:54.561+0000 7f5e87400700 1 --2- 172.21.15.62:0/1743056795 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e800687a0 0x7f5e80195ee0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:37:54.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:54.561+0000 7f5e87400700 1 --2- 172.21.15.62:0/1743056795 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e80069140 0x7f5e80196420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:37:54.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:54.561+0000 7f5e87400700 1 -- 172.21.15.62:0/1743056795 >> 172.21.15.62:0/1743056795 conn(0x7f5e80075130 msgr2=0x7f5e800fe7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:37:54.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:54.561+0000 7f5e87400700 1 -- 172.21.15.62:0/1743056795 shutdown_connections 2024-04-20T21:37:54.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:54.561+0000 7f5e87400700 1 -- 172.21.15.62:0/1743056795 wait complete. 2024-04-20T21:37:54.575 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T21:37:54.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:54.723+0000 7f587be00700 1 -- 172.21.15.62:0/2960665509 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f587c1025f0 msgr2=0x7f587c1029c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:37:54.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:54.723+0000 7f587be00700 1 --2- 172.21.15.62:0/2960665509 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f587c1025f0 0x7f587c1029c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f586c009910 tx=0x7f586c009c20 comp rx=0 tx=0).stop 2024-04-20T21:37:54.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:54.723+0000 7f587be00700 1 -- 172.21.15.62:0/2960665509 shutdown_connections 2024-04-20T21:37:54.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:54.723+0000 7f587be00700 1 --2- 172.21.15.62:0/2960665509 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f587c103e50 0x7f587c1042c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:37:54.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:54.723+0000 7f587be00700 1 --2- 172.21.15.62:0/2960665509 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f587c1025f0 0x7f587c1029c0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:37:54.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:54.723+0000 7f587be00700 1 -- 172.21.15.62:0/2960665509 >> 172.21.15.62:0/2960665509 conn(0x7f587c0fddb0 msgr2=0x7f587c1001c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:37:54.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:54.723+0000 7f587be00700 1 -- 172.21.15.62:0/2960665509 shutdown_connections 2024-04-20T21:37:54.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:54.723+0000 7f587be00700 1 -- 172.21.15.62:0/2960665509 wait complete. 2024-04-20T21:37:54.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:54.724+0000 7f587be00700 1 Processor -- start 2024-04-20T21:37:54.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:54.725+0000 7f587be00700 1 -- start start 2024-04-20T21:37:54.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:54.725+0000 7f587be00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f587c1025f0 0x7f587c197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:37:54.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:54.725+0000 7f587be00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f587c103e50 0x7f587c198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:37:54.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:54.725+0000 7f587be00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f587c198b40 con 0x7f587c103e50 2024-04-20T21:37:54.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:54.725+0000 7f587be00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f587c19c8d0 con 0x7f587c1025f0 2024-04-20T21:37:54.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:54.725+0000 7f587a000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f587c103e50 0x7f587c198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:37:54.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:54.725+0000 7f587a000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f587c103e50 0x7f587c198460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45008/0 (socket says 172.21.15.62:45008) 2024-04-20T21:37:54.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:54.725+0000 7f587a000700 1 -- 172.21.15.62:0/1230188336 learned_addr learned my addr 172.21.15.62:0/1230188336 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:37:54.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:54.726+0000 7f587aa00700 1 --2- 172.21.15.62:0/1230188336 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f587c1025f0 0x7f587c197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:37:54.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:54.726+0000 7f587a000700 1 -- 172.21.15.62:0/1230188336 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f587c1025f0 msgr2=0x7f587c197f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:37:54.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:54.726+0000 7f587a000700 1 --2- 172.21.15.62:0/1230188336 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f587c1025f0 0x7f587c197f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:37:54.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:54.726+0000 7f587a000700 1 -- 172.21.15.62:0/1230188336 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f586c0095f0 con 0x7f587c103e50 2024-04-20T21:37:54.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:54.726+0000 7f587a000700 1 --2- 172.21.15.62:0/1230188336 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f587c103e50 0x7f587c198460 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f586800e920 tx=0x7f586800ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:37:54.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:54.726+0000 7f5873400700 1 -- 172.21.15.62:0/1230188336 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5868009790 con 0x7f587c103e50 2024-04-20T21:37:54.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:54.726+0000 7f5873400700 1 -- 172.21.15.62:0/1230188336 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f586800cba0 con 0x7f587c103e50 2024-04-20T21:37:54.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:54.727+0000 7f5873400700 1 -- 172.21.15.62:0/1230188336 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5868018650 con 0x7f587c103e50 2024-04-20T21:37:54.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:54.727+0000 7f587be00700 1 -- 172.21.15.62:0/1230188336 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f587c19cbb0 con 0x7f587c103e50 2024-04-20T21:37:54.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:54.727+0000 7f587be00700 1 -- 172.21.15.62:0/1230188336 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f587c19d180 con 0x7f587c103e50 2024-04-20T21:37:54.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:54.728+0000 7f587be00700 1 -- 172.21.15.62:0/1230188336 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f587c062380 con 0x7f587c103e50 2024-04-20T21:37:54.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:54.729+0000 7f5873400700 1 -- 172.21.15.62:0/1230188336 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 37) v1 ==== 92847+0+0 (secure 0 0 0) 0x7f586800cd10 con 0x7f587c103e50 2024-04-20T21:37:54.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:54.730+0000 7f5873400700 1 --2- 172.21.15.62:0/1230188336 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f586006edf0 0x7f58600712a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:37:54.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:54.730+0000 7f5873400700 1 -- 172.21.15.62:0/1230188336 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f5868014070 con 0x7f587c103e50 2024-04-20T21:37:54.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:54.730+0000 7f587aa00700 1 --2- 172.21.15.62:0/1230188336 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f586006edf0 0x7f58600712a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:37:54.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:54.731+0000 7f587aa00700 1 --2- 172.21.15.62:0/1230188336 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f586006edf0 0x7f58600712a0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f586c00b760 tx=0x7f586c005fd0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:37:54.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:54.734+0000 7f5873400700 1 -- 172.21.15.62:0/1230188336 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f5868053ff0 con 0x7f587c103e50 2024-04-20T21:37:54.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:54.989+0000 7f587be00700 1 -- 172.21.15.62:0/1230188336 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f587c19d490 con 0x7f586006edf0 2024-04-20T21:37:54.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:54.992+0000 7f5873400700 1 -- 172.21.15.62:0/1230188336 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+383 (secure 0 0 0) 0x7f587c19d490 con 0x7f586006edf0 2024-04-20T21:37:54.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:54.995+0000 7f587be00700 1 -- 172.21.15.62:0/1230188336 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f586006edf0 msgr2=0x7f58600712a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:37:54.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:54.995+0000 7f587be00700 1 --2- 172.21.15.62:0/1230188336 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f586006edf0 0x7f58600712a0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f586c00b760 tx=0x7f586c005fd0 comp rx=0 tx=0).stop 2024-04-20T21:37:54.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:54.995+0000 7f587be00700 1 -- 172.21.15.62:0/1230188336 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f587c103e50 msgr2=0x7f587c198460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:37:54.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:54.995+0000 7f587be00700 1 --2- 172.21.15.62:0/1230188336 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f587c103e50 0x7f587c198460 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f586800e920 tx=0x7f586800ece0 comp rx=0 tx=0).stop 2024-04-20T21:37:54.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:54.996+0000 7f587be00700 1 -- 172.21.15.62:0/1230188336 shutdown_connections 2024-04-20T21:37:54.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:54.996+0000 7f587be00700 1 --2- 172.21.15.62:0/1230188336 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f586006edf0 0x7f58600712a0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:37:54.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:54.996+0000 7f587be00700 1 --2- 172.21.15.62:0/1230188336 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f587c1025f0 0x7f587c197f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:37:54.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:54.996+0000 7f587be00700 1 --2- 172.21.15.62:0/1230188336 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f587c103e50 0x7f587c198460 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:37:54.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:54.996+0000 7f587be00700 1 -- 172.21.15.62:0/1230188336 >> 172.21.15.62:0/1230188336 conn(0x7f587c0fddb0 msgr2=0x7f587c0fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:37:54.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:54.996+0000 7f587be00700 1 -- 172.21.15.62:0/1230188336 shutdown_connections 2024-04-20T21:37:54.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:54.996+0000 7f587be00700 1 -- 172.21.15.62:0/1230188336 wait complete. 2024-04-20T21:37:55.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:55.155+0000 7f303ce00700 1 -- 172.21.15.62:0/4176810666 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f303810c8f0 msgr2=0x7f303810ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:37:55.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:55.155+0000 7f303ce00700 1 --2- 172.21.15.62:0/4176810666 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f303810c8f0 0x7f303810ccc0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3028009910 tx=0x7f3028009c20 comp rx=0 tx=0).stop 2024-04-20T21:37:55.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:55.155+0000 7f303ce00700 1 -- 172.21.15.62:0/4176810666 shutdown_connections 2024-04-20T21:37:55.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:55.155+0000 7f303ce00700 1 --2- 172.21.15.62:0/4176810666 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3038106b80 0x7f3038106ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:37:55.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:55.155+0000 7f303ce00700 1 --2- 172.21.15.62:0/4176810666 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f303810c8f0 0x7f303810ccc0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:37:55.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:55.155+0000 7f303ce00700 1 -- 172.21.15.62:0/4176810666 >> 172.21.15.62:0/4176810666 conn(0x7f3038076060 msgr2=0x7f3038078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:37:55.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:55.155+0000 7f303ce00700 1 -- 172.21.15.62:0/4176810666 shutdown_connections 2024-04-20T21:37:55.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:55.155+0000 7f303ce00700 1 -- 172.21.15.62:0/4176810666 wait complete. 2024-04-20T21:37:55.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:55.156+0000 7f303ce00700 1 Processor -- start 2024-04-20T21:37:55.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:55.157+0000 7f303ce00700 1 -- start start 2024-04-20T21:37:55.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:55.157+0000 7f303ce00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3038106b80 0x7f303819c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:37:55.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:55.157+0000 7f303ce00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f303810c8f0 0x7f303819ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:37:55.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:55.157+0000 7f303ce00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f303819d090 con 0x7f3038106b80 2024-04-20T21:37:55.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:55.157+0000 7f303ce00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f303819d1d0 con 0x7f303810c8f0 2024-04-20T21:37:55.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:55.157+0000 7f3035e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3038106b80 0x7f303819c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:37:55.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:55.157+0000 7f3035e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3038106b80 0x7f303819c500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45032/0 (socket says 172.21.15.62:45032) 2024-04-20T21:37:55.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:55.157+0000 7f3035e00700 1 -- 172.21.15.62:0/1495725550 learned_addr learned my addr 172.21.15.62:0/1495725550 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:37:55.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:55.158+0000 7f3035400700 1 --2- 172.21.15.62:0/1495725550 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f303810c8f0 0x7f303819ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:37:55.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:55.158+0000 7f3035e00700 1 -- 172.21.15.62:0/1495725550 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f303810c8f0 msgr2=0x7f303819ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:37:55.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:55.158+0000 7f3035e00700 1 --2- 172.21.15.62:0/1495725550 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f303810c8f0 0x7f303819ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:37:55.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:55.158+0000 7f3035e00700 1 -- 172.21.15.62:0/1495725550 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30280095f0 con 0x7f3038106b80 2024-04-20T21:37:55.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:55.158+0000 7f3035e00700 1 --2- 172.21.15.62:0/1495725550 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3038106b80 0x7f303819c500 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f3028006010 tx=0x7f302800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:37:55.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:55.158+0000 7f302ea00700 1 -- 172.21.15.62:0/1495725550 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f302801d070 con 0x7f3038106b80 2024-04-20T21:37:55.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:55.158+0000 7f302ea00700 1 -- 172.21.15.62:0/1495725550 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3028004fb0 con 0x7f3038106b80 2024-04-20T21:37:55.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:55.158+0000 7f303ce00700 1 -- 172.21.15.62:0/1495725550 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30381a0fc0 con 0x7f3038106b80 2024-04-20T21:37:55.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:55.158+0000 7f302ea00700 1 -- 172.21.15.62:0/1495725550 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3028022ea0 con 0x7f3038106b80 2024-04-20T21:37:55.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:55.159+0000 7f303ce00700 1 -- 172.21.15.62:0/1495725550 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30381a14b0 con 0x7f3038106b80 2024-04-20T21:37:55.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:55.160+0000 7f303ce00700 1 -- 172.21.15.62:0/1495725550 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f303804f2a0 con 0x7f3038106b80 2024-04-20T21:37:55.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:55.161+0000 7f302ea00700 1 -- 172.21.15.62:0/1495725550 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 37) v1 ==== 92847+0+0 (secure 0 0 0) 0x7f3028022470 con 0x7f3038106b80 2024-04-20T21:37:55.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:55.161+0000 7f302ea00700 1 --2- 172.21.15.62:0/1495725550 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f301c06ee00 0x7f301c0712b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:37:55.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:55.162+0000 7f302ea00700 1 -- 172.21.15.62:0/1495725550 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f3028090670 con 0x7f3038106b80 2024-04-20T21:37:55.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:55.162+0000 7f3035400700 1 --2- 172.21.15.62:0/1495725550 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f301c06ee00 0x7f301c0712b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:37:55.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:55.162+0000 7f3035400700 1 --2- 172.21.15.62:0/1495725550 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f301c06ee00 0x7f301c0712b0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f3024005ea0 tx=0x7f3024005e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:37:55.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:55.166+0000 7f302ea00700 1 -- 172.21.15.62:0/1495725550 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f30280592c0 con 0x7f3038106b80 2024-04-20T21:37:55.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:55 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:37:55.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:55 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:37:55.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:55 smithi189 ceph-mon[57022]: Reconfiguring mds.cephfs.smithi189.xbgspw (monmap changed)... 2024-04-20T21:37:55.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:55 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi189.xbgspw", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-20T21:37:55.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:55 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:37:55.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:55 smithi189 ceph-mon[57022]: Reconfiguring daemon mds.cephfs.smithi189.xbgspw on smithi189 2024-04-20T21:37:55.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:55 smithi189 ceph-mon[57022]: pgmap v43: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:37:55.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:55 smithi189 ceph-mon[57022]: from='client.34186 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:37:55.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:55 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:37:55.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:55 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:37:55.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:55 smithi062 ceph-mon[76322]: Reconfiguring mds.cephfs.smithi189.xbgspw (monmap changed)... 2024-04-20T21:37:55.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:55 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi189.xbgspw", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-20T21:37:55.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:55 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:37:55.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:55 smithi062 ceph-mon[76322]: Reconfiguring daemon mds.cephfs.smithi189.xbgspw on smithi189 2024-04-20T21:37:55.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:55 smithi062 ceph-mon[76322]: pgmap v43: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:37:55.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:55 smithi062 ceph-mon[76322]: from='client.34186 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:37:55.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:55.410+0000 7f303ce00700 1 -- 172.21.15.62:0/1495725550 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f3038109570 con 0x7f301c06ee00 2024-04-20T21:37:55.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:55.443+0000 7f302ea00700 1 -- 172.21.15.62:0/1495725550 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f3038109570 con 0x7f301c06ee00 2024-04-20T21:37:55.443 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T21:37:55.443 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (3m) 73s ago 17m 14.4M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T21:37:55.443 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (17m) 73s ago 17m 8338k - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T21:37:55.443 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (16m) 49s ago 16m 8196k - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T21:37:55.443 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (17m) 73s ago 17m 6656k - 18.2.0 dc2bc1663786 36e10ef5be39 2024-04-20T21:37:55.443 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (16m) 49s ago 16m 6665k - 18.2.0 dc2bc1663786 4101f9953e3d 2024-04-20T21:37:55.443 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (3m) 73s ago 17m 74.3M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T21:37:55.444 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (10m) 73s ago 10m 65.4M - 18.2.0 dc2bc1663786 5e6460aa6159 2024-04-20T21:37:55.444 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (10m) 73s ago 10m 132M - 18.2.0 dc2bc1663786 e65f2267544e 2024-04-20T21:37:55.444 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (10m) 49s ago 10m 117M - 18.2.0 dc2bc1663786 fb2a8a223db8 2024-04-20T21:37:55.444 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (10m) 49s ago 10m 55.6M - 18.2.0 dc2bc1663786 f2eec2ee8951 2024-04-20T21:37:55.444 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (6m) 73s ago 19m 476M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T21:37:55.444 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (5m) 49s ago 16m 449M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T21:37:55.444 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (91s) 73s ago 19m 29.3M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 019eb1479485 2024-04-20T21:37:55.444 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (54s) 49s ago 16m 20.2M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 55aac560aab6 2024-04-20T21:37:55.444 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (4m) 73s ago 17m 7512k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T21:37:55.444 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (4m) 49s ago 16m 7482k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T21:37:55.444 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (15m) 73s ago 15m 218M 1216M 18.2.0 dc2bc1663786 3f315c3afc1f 2024-04-20T21:37:55.444 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (14m) 73s ago 14m 210M 1216M 18.2.0 dc2bc1663786 44e4d07416ad 2024-04-20T21:37:55.444 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (14m) 73s ago 13m 173M 1216M 18.2.0 dc2bc1663786 31c5d9586e98 2024-04-20T21:37:55.444 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (13m) 49s ago 13m 241M 2240M 18.2.0 dc2bc1663786 611c1e242693 2024-04-20T21:37:55.444 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (12m) 49s ago 12m 220M 2240M 18.2.0 dc2bc1663786 6c8fce91d45d 2024-04-20T21:37:55.445 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (12m) 49s ago 12m 206M 2240M 18.2.0 dc2bc1663786 8e4574bae8db 2024-04-20T21:37:55.445 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (4m) 73s ago 17m 71.4M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T21:37:55.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:55.446+0000 7f303ce00700 1 -- 172.21.15.62:0/1495725550 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f301c06ee00 msgr2=0x7f301c0712b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:37:55.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:55.446+0000 7f303ce00700 1 --2- 172.21.15.62:0/1495725550 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f301c06ee00 0x7f301c0712b0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f3024005ea0 tx=0x7f3024005e30 comp rx=0 tx=0).stop 2024-04-20T21:37:55.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:55.446+0000 7f303ce00700 1 -- 172.21.15.62:0/1495725550 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3038106b80 msgr2=0x7f303819c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:37:55.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:55.446+0000 7f303ce00700 1 --2- 172.21.15.62:0/1495725550 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3038106b80 0x7f303819c500 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f3028006010 tx=0x7f302800be20 comp rx=0 tx=0).stop 2024-04-20T21:37:55.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:55.446+0000 7f303ce00700 1 -- 172.21.15.62:0/1495725550 shutdown_connections 2024-04-20T21:37:55.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:55.446+0000 7f303ce00700 1 --2- 172.21.15.62:0/1495725550 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f301c06ee00 0x7f301c0712b0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:37:55.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:55.446+0000 7f303ce00700 1 --2- 172.21.15.62:0/1495725550 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3038106b80 0x7f303819c500 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:37:55.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:55.446+0000 7f303ce00700 1 --2- 172.21.15.62:0/1495725550 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f303810c8f0 0x7f303819ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:37:55.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:55.446+0000 7f303ce00700 1 -- 172.21.15.62:0/1495725550 >> 172.21.15.62:0/1495725550 conn(0x7f3038076060 msgr2=0x7f3038077aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:37:55.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:55.447+0000 7f303ce00700 1 -- 172.21.15.62:0/1495725550 shutdown_connections 2024-04-20T21:37:55.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:55.447+0000 7f303ce00700 1 -- 172.21.15.62:0/1495725550 wait complete. 2024-04-20T21:37:55.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:55.606+0000 7f2866e00700 1 -- 172.21.15.62:0/2169347541 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2860100290 msgr2=0x7f2860100660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:37:55.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:55.606+0000 7f2866e00700 1 --2- 172.21.15.62:0/2169347541 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2860100290 0x7f2860100660 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f2854009910 tx=0x7f2854009c20 comp rx=0 tx=0).stop 2024-04-20T21:37:55.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:55.607+0000 7f2866e00700 1 -- 172.21.15.62:0/2169347541 shutdown_connections 2024-04-20T21:37:55.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:55.607+0000 7f2866e00700 1 --2- 172.21.15.62:0/2169347541 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2860100c30 0x7f2860104c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:37:55.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:55.607+0000 7f2866e00700 1 --2- 172.21.15.62:0/2169347541 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2860100290 0x7f2860100660 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:37:55.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:55.607+0000 7f2866e00700 1 -- 172.21.15.62:0/2169347541 >> 172.21.15.62:0/2169347541 conn(0x7f28600fbb10 msgr2=0x7f28600fdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:37:55.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:55.607+0000 7f2866e00700 1 -- 172.21.15.62:0/2169347541 shutdown_connections 2024-04-20T21:37:55.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:55.607+0000 7f2866e00700 1 -- 172.21.15.62:0/2169347541 wait complete. 2024-04-20T21:37:55.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:55.608+0000 7f2866e00700 1 Processor -- start 2024-04-20T21:37:55.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:55.608+0000 7f2866e00700 1 -- start start 2024-04-20T21:37:55.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:55.609+0000 7f2866e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2860100290 0x7f2860073050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:37:55.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:55.609+0000 7f2866e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2860100c30 0x7f2860073590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:37:55.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:55.609+0000 7f2866e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2860073ad0 con 0x7f2860100c30 2024-04-20T21:37:55.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:55.609+0000 7f2866e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2860073c10 con 0x7f2860100290 2024-04-20T21:37:55.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:55.609+0000 7f2865000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2860100c30 0x7f2860073590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:37:55.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:55.609+0000 7f2865000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2860100c30 0x7f2860073590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45054/0 (socket says 172.21.15.62:45054) 2024-04-20T21:37:55.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:55.609+0000 7f2865000700 1 -- 172.21.15.62:0/3013976310 learned_addr learned my addr 172.21.15.62:0/3013976310 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:37:55.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:55.610+0000 7f2865a00700 1 --2- 172.21.15.62:0/3013976310 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2860100290 0x7f2860073050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:37:55.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:55.610+0000 7f2865000700 1 -- 172.21.15.62:0/3013976310 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2860100290 msgr2=0x7f2860073050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:37:55.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:55.610+0000 7f2865000700 1 --2- 172.21.15.62:0/3013976310 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2860100290 0x7f2860073050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:37:55.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:55.610+0000 7f2865000700 1 -- 172.21.15.62:0/3013976310 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28540095f0 con 0x7f2860100c30 2024-04-20T21:37:55.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:55.610+0000 7f2865000700 1 --2- 172.21.15.62:0/3013976310 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2860100c30 0x7f2860073590 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f284c00e8f0 tx=0x7f284c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:37:55.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:55.610+0000 7f285e000700 1 -- 172.21.15.62:0/3013976310 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f284c009790 con 0x7f2860100c30 2024-04-20T21:37:55.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:55.610+0000 7f285e000700 1 -- 172.21.15.62:0/3013976310 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f284c004cf0 con 0x7f2860100c30 2024-04-20T21:37:55.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:55.610+0000 7f285e000700 1 -- 172.21.15.62:0/3013976310 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f284c005690 con 0x7f2860100c30 2024-04-20T21:37:55.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:55.610+0000 7f2866e00700 1 -- 172.21.15.62:0/3013976310 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28601a1e00 con 0x7f2860100c30 2024-04-20T21:37:55.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:55.611+0000 7f2866e00700 1 -- 172.21.15.62:0/3013976310 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28601a2210 con 0x7f2860100c30 2024-04-20T21:37:55.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:55.612+0000 7f2866e00700 1 -- 172.21.15.62:0/3013976310 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28601085a0 con 0x7f2860100c30 2024-04-20T21:37:55.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:55.613+0000 7f285e000700 1 -- 172.21.15.62:0/3013976310 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 37) v1 ==== 92847+0+0 (secure 0 0 0) 0x7f284c004750 con 0x7f2860100c30 2024-04-20T21:37:55.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:55.614+0000 7f285e000700 1 --2- 172.21.15.62:0/3013976310 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f284806ee00 0x7f28480712b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:37:55.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:55.614+0000 7f285e000700 1 -- 172.21.15.62:0/3013976310 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f284c014070 con 0x7f2860100c30 2024-04-20T21:37:55.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:55.614+0000 7f2865a00700 1 --2- 172.21.15.62:0/3013976310 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f284806ee00 0x7f28480712b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:37:55.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:55.615+0000 7f2865a00700 1 --2- 172.21.15.62:0/3013976310 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f284806ee00 0x7f28480712b0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f285400b760 tx=0x7f2854005a90 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:37:55.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:55.618+0000 7f285e000700 1 -- 172.21.15.62:0/3013976310 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f284c058060 con 0x7f2860100c30 2024-04-20T21:37:55.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:55.953+0000 7f2866e00700 1 -- 172.21.15.62:0/3013976310 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f2860062380 con 0x7f2860100c30 2024-04-20T21:37:55.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:55.954+0000 7f285e000700 1 -- 172.21.15.62:0/3013976310 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+688 (secure 0 0 0) 0x7f284c05b720 con 0x7f2860100c30 2024-04-20T21:37:55.953 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:37:55.953 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T21:37:55.953 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:37:55.954 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:37:55.954 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T21:37:55.954 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:37:55.954 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:37:55.954 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T21:37:55.954 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 6 2024-04-20T21:37:55.954 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:37:55.954 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T21:37:55.954 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 4 2024-04-20T21:37:55.954 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:37:55.954 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T21:37:55.954 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 10, 2024-04-20T21:37:55.954 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 4 2024-04-20T21:37:55.954 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T21:37:55.955 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:37:55.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:55.957+0000 7f2866e00700 1 -- 172.21.15.62:0/3013976310 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f284806ee00 msgr2=0x7f28480712b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:37:55.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:55.957+0000 7f2866e00700 1 --2- 172.21.15.62:0/3013976310 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f284806ee00 0x7f28480712b0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f285400b760 tx=0x7f2854005a90 comp rx=0 tx=0).stop 2024-04-20T21:37:55.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:55.957+0000 7f2866e00700 1 -- 172.21.15.62:0/3013976310 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2860100c30 msgr2=0x7f2860073590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:37:55.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:55.957+0000 7f2866e00700 1 --2- 172.21.15.62:0/3013976310 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2860100c30 0x7f2860073590 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f284c00e8f0 tx=0x7f284c00ecb0 comp rx=0 tx=0).stop 2024-04-20T21:37:55.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:55.957+0000 7f2866e00700 1 -- 172.21.15.62:0/3013976310 shutdown_connections 2024-04-20T21:37:55.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:55.957+0000 7f2866e00700 1 --2- 172.21.15.62:0/3013976310 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f284806ee00 0x7f28480712b0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:37:55.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:55.957+0000 7f2866e00700 1 --2- 172.21.15.62:0/3013976310 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2860100290 0x7f2860073050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:37:55.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:55.958+0000 7f2866e00700 1 --2- 172.21.15.62:0/3013976310 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2860100c30 0x7f2860073590 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:37:55.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:55.958+0000 7f2866e00700 1 -- 172.21.15.62:0/3013976310 >> 172.21.15.62:0/3013976310 conn(0x7f28600fbb10 msgr2=0x7f28600fd1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:37:55.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:55.958+0000 7f2866e00700 1 -- 172.21.15.62:0/3013976310 shutdown_connections 2024-04-20T21:37:55.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:55.958+0000 7f2866e00700 1 -- 172.21.15.62:0/3013976310 wait complete. 2024-04-20T21:37:56.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:56.116+0000 7f652a800700 1 -- 172.21.15.62:0/3879767018 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f652410c8f0 msgr2=0x7f652410ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:37:56.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:56.116+0000 7f652a800700 1 --2- 172.21.15.62:0/3879767018 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f652410c8f0 0x7f652410ccc0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f6514009910 tx=0x7f6514009c20 comp rx=0 tx=0).stop 2024-04-20T21:37:56.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:56.116+0000 7f652a800700 1 -- 172.21.15.62:0/3879767018 shutdown_connections 2024-04-20T21:37:56.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:56.116+0000 7f652a800700 1 --2- 172.21.15.62:0/3879767018 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6524106b80 0x7f6524106ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:37:56.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:56.116+0000 7f652a800700 1 --2- 172.21.15.62:0/3879767018 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f652410c8f0 0x7f652410ccc0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:37:56.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:56.116+0000 7f652a800700 1 -- 172.21.15.62:0/3879767018 >> 172.21.15.62:0/3879767018 conn(0x7f6524076060 msgr2=0x7f6524078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:37:56.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:56.116+0000 7f652a800700 1 -- 172.21.15.62:0/3879767018 shutdown_connections 2024-04-20T21:37:56.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:56.116+0000 7f652a800700 1 -- 172.21.15.62:0/3879767018 wait complete. 2024-04-20T21:37:56.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:56.117+0000 7f652a800700 1 Processor -- start 2024-04-20T21:37:56.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:56.117+0000 7f652a800700 1 -- start start 2024-04-20T21:37:56.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:56.118+0000 7f652a800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6524106b80 0x7f652419c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:37:56.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:56.118+0000 7f652a800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f652410c8f0 0x7f652419ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:37:56.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:56.118+0000 7f652a800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f652419d090 con 0x7f6524106b80 2024-04-20T21:37:56.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:56.118+0000 7f652a800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f652419d1d0 con 0x7f652410c8f0 2024-04-20T21:37:56.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:56.118+0000 7f6523400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6524106b80 0x7f652419c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:37:56.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:56.118+0000 7f6523400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6524106b80 0x7f652419c500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45078/0 (socket says 172.21.15.62:45078) 2024-04-20T21:37:56.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:56.118+0000 7f6523400700 1 -- 172.21.15.62:0/2972993762 learned_addr learned my addr 172.21.15.62:0/2972993762 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:37:56.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:56.118+0000 7f6523400700 1 -- 172.21.15.62:0/2972993762 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f652410c8f0 msgr2=0x7f652419ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:37:56.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:56.118+0000 7f6522a00700 1 --2- 172.21.15.62:0/2972993762 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f652410c8f0 0x7f652419ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:37:56.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:56.118+0000 7f6523400700 1 --2- 172.21.15.62:0/2972993762 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f652410c8f0 0x7f652419ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:37:56.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:56.118+0000 7f6523400700 1 -- 172.21.15.62:0/2972993762 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65140095f0 con 0x7f6524106b80 2024-04-20T21:37:56.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:56.119+0000 7f6522a00700 1 --2- 172.21.15.62:0/2972993762 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f652410c8f0 0x7f652419ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T21:37:56.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:56.119+0000 7f6523400700 1 --2- 172.21.15.62:0/2972993762 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6524106b80 0x7f652419c500 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f651400b5c0 tx=0x7f651400be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:37:56.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:56.119+0000 7f651be00700 1 -- 172.21.15.62:0/2972993762 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f651401d070 con 0x7f6524106b80 2024-04-20T21:37:56.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:56.119+0000 7f651be00700 1 -- 172.21.15.62:0/2972993762 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f651400f460 con 0x7f6524106b80 2024-04-20T21:37:56.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:56.119+0000 7f652a800700 1 -- 172.21.15.62:0/2972993762 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65241a0fc0 con 0x7f6524106b80 2024-04-20T21:37:56.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:56.119+0000 7f651be00700 1 -- 172.21.15.62:0/2972993762 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6514017610 con 0x7f6524106b80 2024-04-20T21:37:56.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:56.119+0000 7f652a800700 1 -- 172.21.15.62:0/2972993762 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65241a14b0 con 0x7f6524106b80 2024-04-20T21:37:56.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:56.120+0000 7f652a800700 1 -- 172.21.15.62:0/2972993762 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6524109b20 con 0x7f6524106b80 2024-04-20T21:37:56.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:56.121+0000 7f651be00700 1 -- 172.21.15.62:0/2972993762 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 37) v1 ==== 92847+0+0 (secure 0 0 0) 0x7f651400f5d0 con 0x7f6524106b80 2024-04-20T21:37:56.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:56.123+0000 7f651be00700 1 --2- 172.21.15.62:0/2972993762 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f650806edf0 0x7f65080712a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:37:56.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:56.123+0000 7f651be00700 1 -- 172.21.15.62:0/2972993762 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f6514090580 con 0x7f6524106b80 2024-04-20T21:37:56.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:56.123+0000 7f6522a00700 1 --2- 172.21.15.62:0/2972993762 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f650806edf0 0x7f65080712a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:37:56.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:56.124+0000 7f6522a00700 1 --2- 172.21.15.62:0/2972993762 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f650806edf0 0x7f65080712a0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f6510005fd0 tx=0x7f6510005e20 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:37:56.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:56.128+0000 7f651be00700 1 -- 172.21.15.62:0/2972993762 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f6514059240 con 0x7f6524106b80 2024-04-20T21:37:56.229 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:56 smithi189 ceph-mon[57022]: from='client.34190 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:37:56.229 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:56 smithi189 ceph-mon[57022]: from='client.34194 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:37:56.229 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:56 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/3013976310' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:37:56.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:56 smithi062 ceph-mon[76322]: from='client.34190 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:37:56.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:56 smithi062 ceph-mon[76322]: from='client.34194 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:37:56.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:56 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/3013976310' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:37:56.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:56.408+0000 7f652a800700 1 -- 172.21.15.62:0/2972993762 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f652419d960 con 0x7f6524106b80 2024-04-20T21:37:56.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:56.411+0000 7f651be00700 1 -- 172.21.15.62:0/2972993762 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 18 v18) v1 ==== 76+0+1928 (secure 0 0 0) 0x7f651405c900 con 0x7f6524106b80 2024-04-20T21:37:56.411 INFO:teuthology.orchestra.run.smithi062.stdout:e18 2024-04-20T21:37:56.411 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T21:37:56.411 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T21:37:56.411 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T21:37:56.411 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:37:56.411 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T21:37:56.411 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T21:37:56.411 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 18 2024-04-20T21:37:56.411 INFO:teuthology.orchestra.run.smithi062.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-20T21:37:56.411 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T21:37:56.411 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T21:37:06.018300+0000 2024-04-20T21:37:56.412 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T21:37:56.412 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T21:37:56.412 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T21:37:56.412 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T21:37:56.412 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T21:37:56.412 INFO:teuthology.orchestra.run.smithi062.stdout:max_xattr_size 65536 2024-04-20T21:37:56.412 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T21:37:56.412 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T21:37:56.412 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 0 2024-04-20T21:37:56.412 INFO:teuthology.orchestra.run.smithi062.stdout:snap_rstat 1 2024-04-20T21:37:56.412 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T21:37:56.412 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 2 2024-04-20T21:37:56.412 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1 2024-04-20T21:37:56.412 INFO:teuthology.orchestra.run.smithi062.stdout:up {0=14458,1=24279} 2024-04-20T21:37:56.412 INFO:teuthology.orchestra.run.smithi062.stdout:failed 2024-04-20T21:37:56.412 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T21:37:56.412 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T21:37:56.413 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T21:37:56.413 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T21:37:56.413 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data enabled 2024-04-20T21:37:56.413 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T21:37:56.413 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T21:37:56.413 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 1 2024-04-20T21:37:56.413 INFO:teuthology.orchestra.run.smithi062.stdout:qdb_cluster leader: 14458 members: 24279,14458 2024-04-20T21:37:56.413 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ryqblb{0:14458} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.62:6826/1147127123,v1:172.21.15.62:6827/1147127123] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:37:56.413 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ozizwi{0:14474} state up:standby-replay seq 3 join_fscid=1 addr [v2:172.21.15.62:6828/3494174293,v1:172.21.15.62:6829/3494174293] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:37:56.413 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.dvimer{1:24279} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.189:6824/237262514,v1:172.21.15.189:6825/237262514] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:37:56.413 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.xbgspw{1:24291} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.189:6826/1495356258,v1:172.21.15.189:6827/1495356258] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:37:56.413 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:37:56.413 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:37:56.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:56.414+0000 7f652a800700 1 -- 172.21.15.62:0/2972993762 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f650806edf0 msgr2=0x7f65080712a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:37:56.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:56.414+0000 7f652a800700 1 --2- 172.21.15.62:0/2972993762 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f650806edf0 0x7f65080712a0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f6510005fd0 tx=0x7f6510005e20 comp rx=0 tx=0).stop 2024-04-20T21:37:56.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:56.414+0000 7f652a800700 1 -- 172.21.15.62:0/2972993762 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6524106b80 msgr2=0x7f652419c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:37:56.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:56.414+0000 7f652a800700 1 --2- 172.21.15.62:0/2972993762 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6524106b80 0x7f652419c500 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f651400b5c0 tx=0x7f651400be00 comp rx=0 tx=0).stop 2024-04-20T21:37:56.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:56.415+0000 7f652a800700 1 -- 172.21.15.62:0/2972993762 shutdown_connections 2024-04-20T21:37:56.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:56.415+0000 7f652a800700 1 --2- 172.21.15.62:0/2972993762 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f650806edf0 0x7f65080712a0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:37:56.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:56.415+0000 7f652a800700 1 --2- 172.21.15.62:0/2972993762 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6524106b80 0x7f652419c500 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:37:56.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:56.415+0000 7f652a800700 1 --2- 172.21.15.62:0/2972993762 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f652410c8f0 0x7f652419ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:37:56.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:56.415+0000 7f652a800700 1 -- 172.21.15.62:0/2972993762 >> 172.21.15.62:0/2972993762 conn(0x7f6524076060 msgr2=0x7f6524077aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:37:56.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:56.415+0000 7f652a800700 1 -- 172.21.15.62:0/2972993762 shutdown_connections 2024-04-20T21:37:56.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:56.415+0000 7f652a800700 1 -- 172.21.15.62:0/2972993762 wait complete. 2024-04-20T21:37:56.415 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 18 2024-04-20T21:37:56.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:56.583+0000 7fecdea00700 1 -- 172.21.15.62:0/45889315 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fecd8100e80 msgr2=0x7fecd8104e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:37:56.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:56.583+0000 7fecdea00700 1 --2- 172.21.15.62:0/45889315 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fecd8100e80 0x7fecd8104e50 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fecc4009910 tx=0x7fecc4009c20 comp rx=0 tx=0).stop 2024-04-20T21:37:56.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:56.584+0000 7fecdea00700 1 -- 172.21.15.62:0/45889315 shutdown_connections 2024-04-20T21:37:56.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:56.584+0000 7fecdea00700 1 --2- 172.21.15.62:0/45889315 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fecd8100e80 0x7fecd8104e50 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:37:56.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:56.584+0000 7fecdea00700 1 --2- 172.21.15.62:0/45889315 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fecd81004e0 0x7fecd81008b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:37:56.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:56.584+0000 7fecdea00700 1 -- 172.21.15.62:0/45889315 >> 172.21.15.62:0/45889315 conn(0x7fecd80fbda0 msgr2=0x7fecd80fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:37:56.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:56.584+0000 7fecdea00700 1 -- 172.21.15.62:0/45889315 shutdown_connections 2024-04-20T21:37:56.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:56.584+0000 7fecdea00700 1 -- 172.21.15.62:0/45889315 wait complete. 2024-04-20T21:37:56.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:56.586+0000 7fecdea00700 1 Processor -- start 2024-04-20T21:37:56.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:56.586+0000 7fecdea00700 1 -- start start 2024-04-20T21:37:56.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:56.586+0000 7fecdea00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fecd81004e0 0x7fecd8195f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:37:56.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:56.586+0000 7fecdea00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fecd8100e80 0x7fecd8196470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:37:56.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:56.586+0000 7fecdea00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecd8196b50 con 0x7fecd8100e80 2024-04-20T21:37:56.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:56.586+0000 7fecdea00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecd819a8e0 con 0x7fecd81004e0 2024-04-20T21:37:56.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:56.587+0000 7fecd7400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fecd8100e80 0x7fecd8196470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:37:56.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:56.587+0000 7fecd7400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fecd8100e80 0x7fecd8196470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45090/0 (socket says 172.21.15.62:45090) 2024-04-20T21:37:56.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:56.587+0000 7fecd7400700 1 -- 172.21.15.62:0/1911859710 learned_addr learned my addr 172.21.15.62:0/1911859710 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:37:56.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:56.587+0000 7fecd7e00700 1 --2- 172.21.15.62:0/1911859710 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fecd81004e0 0x7fecd8195f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:37:56.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:56.588+0000 7fecd7400700 1 -- 172.21.15.62:0/1911859710 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fecd81004e0 msgr2=0x7fecd8195f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:37:56.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:56.588+0000 7fecd7400700 1 --2- 172.21.15.62:0/1911859710 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fecd81004e0 0x7fecd8195f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:37:56.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:56.588+0000 7fecd7400700 1 -- 172.21.15.62:0/1911859710 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecc40095f0 con 0x7fecd8100e80 2024-04-20T21:37:56.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:56.588+0000 7fecd7e00700 1 --2- 172.21.15.62:0/1911859710 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fecd81004e0 0x7fecd8195f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-20T21:37:56.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:56.588+0000 7fecd7400700 1 --2- 172.21.15.62:0/1911859710 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fecd8100e80 0x7fecd8196470 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fecc400bd70 tx=0x7fecc400be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:37:56.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:56.589+0000 7fecd4c00700 1 -- 172.21.15.62:0/1911859710 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fecc401d070 con 0x7fecd8100e80 2024-04-20T21:37:56.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:56.589+0000 7fecdea00700 1 -- 172.21.15.62:0/1911859710 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecd819ab60 con 0x7fecd8100e80 2024-04-20T21:37:56.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:56.589+0000 7fecd4c00700 1 -- 172.21.15.62:0/1911859710 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fecc4005dc0 con 0x7fecd8100e80 2024-04-20T21:37:56.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:56.589+0000 7fecd4c00700 1 -- 172.21.15.62:0/1911859710 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fecc4017400 con 0x7fecd8100e80 2024-04-20T21:37:56.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:56.589+0000 7fecdea00700 1 -- 172.21.15.62:0/1911859710 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecd819b050 con 0x7fecd8100e80 2024-04-20T21:37:56.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:56.590+0000 7fecdea00700 1 -- 172.21.15.62:0/1911859710 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecd8108850 con 0x7fecd8100e80 2024-04-20T21:37:56.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:56.592+0000 7fecd4c00700 1 -- 172.21.15.62:0/1911859710 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 37) v1 ==== 92847+0+0 (secure 0 0 0) 0x7fecc400f7f0 con 0x7fecd8100e80 2024-04-20T21:37:56.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:56.593+0000 7fecd4c00700 1 --2- 172.21.15.62:0/1911859710 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fecbc06edf0 0x7fecbc0712a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:37:56.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:56.593+0000 7fecd4c00700 1 -- 172.21.15.62:0/1911859710 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fecc40904f0 con 0x7fecd8100e80 2024-04-20T21:37:56.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:56.594+0000 7fecd7e00700 1 --2- 172.21.15.62:0/1911859710 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fecbc06edf0 0x7fecbc0712a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:37:56.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:56.594+0000 7fecd7e00700 1 --2- 172.21.15.62:0/1911859710 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fecbc06edf0 0x7fecbc0712a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fecc8005950 tx=0x7fecc80058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:37:56.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:56.598+0000 7fecd4c00700 1 -- 172.21.15.62:0/1911859710 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fecc4059110 con 0x7fecd8100e80 2024-04-20T21:37:56.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:56.856+0000 7fecdea00700 1 -- 172.21.15.62:0/1911859710 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fecd81973a0 con 0x7fecbc06edf0 2024-04-20T21:37:56.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:56.859+0000 7fecd4c00700 1 -- 172.21.15.62:0/1911859710 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+385 (secure 0 0 0) 0x7fecd81973a0 con 0x7fecbc06edf0 2024-04-20T21:37:56.858 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:37:56.858 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T21:37:56.858 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T21:37:56.858 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-20T21:37:56.858 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T21:37:56.858 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr", 2024-04-20T21:37:56.858 INFO:teuthology.orchestra.run.smithi062.stdout: "mon" 2024-04-20T21:37:56.858 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T21:37:56.858 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "4/23 daemons upgraded", 2024-04-20T21:37:56.859 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading crash daemons", 2024-04-20T21:37:56.859 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T21:37:56.859 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:37:56.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:56.861+0000 7fecdea00700 1 -- 172.21.15.62:0/1911859710 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fecbc06edf0 msgr2=0x7fecbc0712a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:37:56.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:56.861+0000 7fecdea00700 1 --2- 172.21.15.62:0/1911859710 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fecbc06edf0 0x7fecbc0712a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fecc8005950 tx=0x7fecc80058e0 comp rx=0 tx=0).stop 2024-04-20T21:37:56.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:56.861+0000 7fecdea00700 1 -- 172.21.15.62:0/1911859710 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fecd8100e80 msgr2=0x7fecd8196470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:37:56.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:56.861+0000 7fecdea00700 1 --2- 172.21.15.62:0/1911859710 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fecd8100e80 0x7fecd8196470 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fecc400bd70 tx=0x7fecc400be50 comp rx=0 tx=0).stop 2024-04-20T21:37:56.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:56.861+0000 7fecdea00700 1 -- 172.21.15.62:0/1911859710 shutdown_connections 2024-04-20T21:37:56.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:56.861+0000 7fecdea00700 1 --2- 172.21.15.62:0/1911859710 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fecbc06edf0 0x7fecbc0712a0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:37:56.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:56.861+0000 7fecdea00700 1 --2- 172.21.15.62:0/1911859710 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fecd81004e0 0x7fecd8195f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:37:56.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:56.861+0000 7fecdea00700 1 --2- 172.21.15.62:0/1911859710 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fecd8100e80 0x7fecd8196470 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:37:56.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:56.862+0000 7fecdea00700 1 -- 172.21.15.62:0/1911859710 >> 172.21.15.62:0/1911859710 conn(0x7fecd80fbda0 msgr2=0x7fecd80fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:37:56.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:56.862+0000 7fecdea00700 1 -- 172.21.15.62:0/1911859710 shutdown_connections 2024-04-20T21:37:56.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:56.862+0000 7fecdea00700 1 -- 172.21.15.62:0/1911859710 wait complete. 2024-04-20T21:37:57.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:57.018+0000 7f7bee200700 1 -- 172.21.15.62:0/4204647947 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7be8068540 msgr2=0x7f7be8068910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:37:57.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:57.018+0000 7f7bee200700 1 --2- 172.21.15.62:0/4204647947 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7be8068540 0x7f7be8068910 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f7bdc009910 tx=0x7f7bdc009c20 comp rx=0 tx=0).stop 2024-04-20T21:37:57.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:57.018+0000 7f7bee200700 1 -- 172.21.15.62:0/4204647947 shutdown_connections 2024-04-20T21:37:57.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:57.018+0000 7f7bee200700 1 --2- 172.21.15.62:0/4204647947 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7be8068e50 0x7f7be8110780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:37:57.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:57.018+0000 7f7bee200700 1 --2- 172.21.15.62:0/4204647947 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7be8068540 0x7f7be8068910 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:37:57.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:57.018+0000 7f7bee200700 1 -- 172.21.15.62:0/4204647947 >> 172.21.15.62:0/4204647947 conn(0x7f7be8074e10 msgr2=0x7f7be8075210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:37:57.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:57.018+0000 7f7bee200700 1 -- 172.21.15.62:0/4204647947 shutdown_connections 2024-04-20T21:37:57.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:57.018+0000 7f7bee200700 1 -- 172.21.15.62:0/4204647947 wait complete. 2024-04-20T21:37:57.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:57.020+0000 7f7bee200700 1 Processor -- start 2024-04-20T21:37:57.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:57.020+0000 7f7bee200700 1 -- start start 2024-04-20T21:37:57.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:57.020+0000 7f7bee200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7be8068540 0x7f7be819bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:37:57.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:57.020+0000 7f7bee200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7be8068e50 0x7f7be819c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:37:57.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:57.020+0000 7f7bee200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7be819cc00 con 0x7f7be8068e50 2024-04-20T21:37:57.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:57.020+0000 7f7bee200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7be81a0990 con 0x7f7be8068540 2024-04-20T21:37:57.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:57.021+0000 7f7be7e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7be8068e50 0x7f7be819c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:37:57.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:57.021+0000 7f7be7e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7be8068e50 0x7f7be819c520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45106/0 (socket says 172.21.15.62:45106) 2024-04-20T21:37:57.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:57.021+0000 7f7be7e00700 1 -- 172.21.15.62:0/3199027178 learned_addr learned my addr 172.21.15.62:0/3199027178 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:37:57.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:57.021+0000 7f7bece00700 1 --2- 172.21.15.62:0/3199027178 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7be8068540 0x7f7be819bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:37:57.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:57.022+0000 7f7be7e00700 1 -- 172.21.15.62:0/3199027178 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7be8068540 msgr2=0x7f7be819bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:37:57.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:57.022+0000 7f7be7e00700 1 --2- 172.21.15.62:0/3199027178 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7be8068540 0x7f7be819bfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:37:57.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:57.022+0000 7f7be7e00700 1 -- 172.21.15.62:0/3199027178 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bdc0095f0 con 0x7f7be8068e50 2024-04-20T21:37:57.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:57.022+0000 7f7be7e00700 1 --2- 172.21.15.62:0/3199027178 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7be8068e50 0x7f7be819c520 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f7bd400d8d0 tx=0x7f7bd400dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:37:57.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:57.022+0000 7f7be5600700 1 -- 172.21.15.62:0/3199027178 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7bd40098a0 con 0x7f7be8068e50 2024-04-20T21:37:57.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:57.023+0000 7f7be5600700 1 -- 172.21.15.62:0/3199027178 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7bd4010460 con 0x7f7be8068e50 2024-04-20T21:37:57.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:57.023+0000 7f7be5600700 1 -- 172.21.15.62:0/3199027178 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7bd400f5f0 con 0x7f7be8068e50 2024-04-20T21:37:57.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:57.023+0000 7f7bee200700 1 -- 172.21.15.62:0/3199027178 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7be81a0c70 con 0x7f7be8068e50 2024-04-20T21:37:57.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:57.024+0000 7f7bee200700 1 -- 172.21.15.62:0/3199027178 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7be81a1240 con 0x7f7be8068e50 2024-04-20T21:37:57.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:57.025+0000 7f7bee200700 1 -- 172.21.15.62:0/3199027178 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7be8062380 con 0x7f7be8068e50 2024-04-20T21:37:57.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:57.027+0000 7f7be5600700 1 -- 172.21.15.62:0/3199027178 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 37) v1 ==== 92847+0+0 (secure 0 0 0) 0x7f7bd4009a00 con 0x7f7be8068e50 2024-04-20T21:37:57.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:57.029+0000 7f7be5600700 1 --2- 172.21.15.62:0/3199027178 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7bd006edf0 0x7f7bd00712a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:37:57.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:57.029+0000 7f7be5600700 1 -- 172.21.15.62:0/3199027178 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f7bd408e940 con 0x7f7be8068e50 2024-04-20T21:37:57.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:57.029+0000 7f7bece00700 1 --2- 172.21.15.62:0/3199027178 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7bd006edf0 0x7f7bd00712a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:37:57.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:57.030+0000 7f7bece00700 1 --2- 172.21.15.62:0/3199027178 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7bd006edf0 0x7f7bd00712a0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f7bdc006010 tx=0x7f7bdc01a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:37:57.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:57.037+0000 7f7be5600700 1 -- 172.21.15.62:0/3199027178 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f7bd4057560 con 0x7f7be8068e50 2024-04-20T21:37:57.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:57.387+0000 7f7bee200700 1 -- 172.21.15.62:0/3199027178 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f7be819d340 con 0x7f7be8068e50 2024-04-20T21:37:57.388 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:57 smithi062 ceph-mon[76322]: pgmap v44: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:37:57.388 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:57 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:37:57.388 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:57 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:37:57.388 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:57 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:37:57.388 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:57 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:37:57.388 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:57 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:37:57.388 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:57 smithi062 ceph-mon[76322]: Upgrade: Setting container_image for all mon 2024-04-20T21:37:57.388 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:57 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:37:57.388 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:57 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/2972993762' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T21:37:57.388 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:57 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi062"}]: dispatch 2024-04-20T21:37:57.388 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:57 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi062"}]': finished 2024-04-20T21:37:57.388 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:57 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi189"}]: dispatch 2024-04-20T21:37:57.388 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:57 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi189"}]': finished 2024-04-20T21:37:57.389 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:57 smithi062 ceph-mon[76322]: from='client.34206 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:37:57.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:57.388+0000 7f7be5600700 1 -- 172.21.15.62:0/3199027178 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7f7bd40200e0 con 0x7f7be8068e50 2024-04-20T21:37:57.389 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-04-20T21:37:57.389 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T21:37:57.389 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T21:37:57.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:57.391+0000 7f7bee200700 1 -- 172.21.15.62:0/3199027178 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7bd006edf0 msgr2=0x7f7bd00712a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:37:57.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:57.391+0000 7f7bee200700 1 --2- 172.21.15.62:0/3199027178 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7bd006edf0 0x7f7bd00712a0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f7bdc006010 tx=0x7f7bdc01a040 comp rx=0 tx=0).stop 2024-04-20T21:37:57.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:57.391+0000 7f7bee200700 1 -- 172.21.15.62:0/3199027178 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7be8068e50 msgr2=0x7f7be819c520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:37:57.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:57.391+0000 7f7bee200700 1 --2- 172.21.15.62:0/3199027178 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7be8068e50 0x7f7be819c520 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f7bd400d8d0 tx=0x7f7bd400dbe0 comp rx=0 tx=0).stop 2024-04-20T21:37:57.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:57.392+0000 7f7bee200700 1 -- 172.21.15.62:0/3199027178 shutdown_connections 2024-04-20T21:37:57.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:57.392+0000 7f7bee200700 1 --2- 172.21.15.62:0/3199027178 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7bd006edf0 0x7f7bd00712a0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:37:57.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:57.392+0000 7f7bee200700 1 --2- 172.21.15.62:0/3199027178 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7be8068540 0x7f7be819bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:37:57.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:57.392+0000 7f7bee200700 1 --2- 172.21.15.62:0/3199027178 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7be8068e50 0x7f7be819c520 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:37:57.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:57.392+0000 7f7bee200700 1 -- 172.21.15.62:0/3199027178 >> 172.21.15.62:0/3199027178 conn(0x7f7be8074e10 msgr2=0x7f7be810e980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:37:57.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:57.392+0000 7f7bee200700 1 -- 172.21.15.62:0/3199027178 shutdown_connections 2024-04-20T21:37:57.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:37:57.392+0000 7f7bee200700 1 -- 172.21.15.62:0/3199027178 wait complete. 2024-04-20T21:37:57.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:57 smithi189 ceph-mon[57022]: pgmap v44: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:37:57.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:57 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:37:57.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:57 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:37:57.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:57 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:37:57.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:57 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:37:57.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:57 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:37:57.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:57 smithi189 ceph-mon[57022]: Upgrade: Setting container_image for all mon 2024-04-20T21:37:57.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:57 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:37:57.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:57 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/2972993762' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T21:37:57.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:57 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi062"}]: dispatch 2024-04-20T21:37:57.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:57 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi062"}]': finished 2024-04-20T21:37:57.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:57 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi189"}]: dispatch 2024-04-20T21:37:57.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:57 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi189"}]': finished 2024-04-20T21:37:57.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:57 smithi189 ceph-mon[57022]: from='client.34206 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:37:58.591 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:58 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/3199027178' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T21:37:58.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:58 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/3199027178' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T21:37:59.905 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:59 smithi062 ceph-mon[76322]: pgmap v45: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:37:59.905 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:59 smithi062 ceph-mon[76322]: Upgrade: Updating crash.smithi062 (1/2) 2024-04-20T21:37:59.905 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:59 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:37:59.906 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:59 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi062", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-20T21:37:59.906 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:59 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:37:59.906 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:37:59 smithi062 ceph-mon[76322]: Deploying daemon crash.smithi062 on smithi062 2024-04-20T21:37:59.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:59 smithi189 ceph-mon[57022]: pgmap v45: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:37:59.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:59 smithi189 ceph-mon[57022]: Upgrade: Updating crash.smithi062 (1/2) 2024-04-20T21:37:59.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:59 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:37:59.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:59 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi062", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-20T21:37:59.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:59 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:37:59.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:37:59 smithi189 ceph-mon[57022]: Deploying daemon crash.smithi062 on smithi062 2024-04-20T21:38:01.960 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:01 smithi062 ceph-mon[76322]: pgmap v46: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:38:01.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:01 smithi189 ceph-mon[57022]: pgmap v46: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:38:03.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:03 smithi189 ceph-mon[57022]: pgmap v47: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:38:04.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:03 smithi062 ceph-mon[76322]: pgmap v47: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:38:05.454 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:05 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:38:05.454 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:05 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:38:05.454 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:05 smithi189 ceph-mon[57022]: pgmap v48: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:38:05.454 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:05 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:38:05.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:05 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:38:05.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:05 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:38:05.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:05 smithi062 ceph-mon[76322]: pgmap v48: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:38:05.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:05 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:38:07.469 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:07 smithi062 ceph-mon[76322]: pgmap v49: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 511 B/s wr, 5 op/s 2024-04-20T21:38:07.469 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:07 smithi062 ceph-mon[76322]: mgrmap e38: smithi062.rqxuxi(active, since 92s), standbys: smithi189.tawavz 2024-04-20T21:38:07.469 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:07 smithi062 ceph-mon[76322]: Upgrade: Updating crash.smithi189 (2/2) 2024-04-20T21:38:07.469 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:07 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:38:07.469 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:07 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi189", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-20T21:38:07.469 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:07 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:38:07.470 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:07 smithi062 ceph-mon[76322]: Deploying daemon crash.smithi189 on smithi189 2024-04-20T21:38:07.621 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:07 smithi189 ceph-mon[57022]: pgmap v49: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 511 B/s wr, 5 op/s 2024-04-20T21:38:07.621 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:07 smithi189 ceph-mon[57022]: mgrmap e38: smithi062.rqxuxi(active, since 92s), standbys: smithi189.tawavz 2024-04-20T21:38:07.621 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:07 smithi189 ceph-mon[57022]: Upgrade: Updating crash.smithi189 (2/2) 2024-04-20T21:38:07.621 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:07 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:38:07.621 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:07 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi189", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-20T21:38:07.621 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:07 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:38:07.621 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:07 smithi189 ceph-mon[57022]: Deploying daemon crash.smithi189 on smithi189 2024-04-20T21:38:09.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:09 smithi189 ceph-mon[57022]: pgmap v50: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 511 B/s wr, 3 op/s 2024-04-20T21:38:09.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:09 smithi062 ceph-mon[76322]: pgmap v50: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 511 B/s wr, 3 op/s 2024-04-20T21:38:11.700 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:11 smithi189 ceph-mon[57022]: pgmap v51: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 511 B/s wr, 5 op/s 2024-04-20T21:38:11.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:11 smithi062 ceph-mon[76322]: pgmap v51: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 511 B/s wr, 5 op/s 2024-04-20T21:38:13.592 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:13 smithi189 ceph-mon[57022]: pgmap v52: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 511 B/s wr, 3 op/s 2024-04-20T21:38:13.593 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:13 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:38:13.593 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:13 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:38:13.593 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:13 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:38:13.634 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:13 smithi062 ceph-mon[76322]: pgmap v52: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 511 B/s wr, 3 op/s 2024-04-20T21:38:13.634 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:13 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:38:13.634 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:13 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:38:13.634 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:13 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:38:15.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:15 smithi189 ceph-mon[57022]: pgmap v53: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 511 B/s wr, 3 op/s 2024-04-20T21:38:15.608 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:15 smithi062 ceph-mon[76322]: pgmap v53: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 511 B/s wr, 3 op/s 2024-04-20T21:38:17.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:17 smithi189 ceph-mon[57022]: pgmap v54: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 511 B/s wr, 5 op/s 2024-04-20T21:38:17.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:17 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:38:17.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:17 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:38:18.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:17 smithi062 ceph-mon[76322]: pgmap v54: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 511 B/s wr, 5 op/s 2024-04-20T21:38:18.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:17 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:38:18.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:17 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:38:20.017 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:19 smithi189 ceph-mon[57022]: pgmap v55: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:38:20.017 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:19 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:38:20.017 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:19 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:38:20.017 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:19 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:38:20.018 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:19 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:38:20.018 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:19 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:38:20.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:19 smithi062 ceph-mon[76322]: pgmap v55: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:38:20.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:19 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:38:20.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:19 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:38:20.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:19 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:38:20.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:19 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:38:20.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:19 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:38:22.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:21 smithi062 ceph-mon[76322]: pgmap v56: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:38:22.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:21 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:38:22.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:21 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:38:22.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:21 smithi189 ceph-mon[57022]: pgmap v56: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:38:22.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:21 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:38:22.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:21 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:38:24.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:23 smithi062 ceph-mon[76322]: pgmap v57: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:38:24.088 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:23 smithi189 ceph-mon[57022]: pgmap v57: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:38:25.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:25 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:38:25.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:25 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:38:25.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:25 smithi189 ceph-mon[57022]: pgmap v58: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:38:25.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:25 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:38:25.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:25 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:38:25.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:25 smithi062 ceph-mon[76322]: pgmap v58: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:38:27.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:27.579+0000 7ff11a400700 1 -- 172.21.15.62:0/2211675891 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff114102460 msgr2=0x7ff1141028d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:38:27.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:27.579+0000 7ff11a400700 1 --2- 172.21.15.62:0/2211675891 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff114102460 0x7ff1141028d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff100009910 tx=0x7ff100009c20 comp rx=0 tx=0).stop 2024-04-20T21:38:27.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:27.580+0000 7ff11a400700 1 -- 172.21.15.62:0/2211675891 shutdown_connections 2024-04-20T21:38:27.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:27.580+0000 7ff11a400700 1 --2- 172.21.15.62:0/2211675891 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff114102460 0x7ff1141028d0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:38:27.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:27.580+0000 7ff11a400700 1 --2- 172.21.15.62:0/2211675891 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff114108460 0x7ff114108830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:38:27.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:27.580+0000 7ff11a400700 1 -- 172.21.15.62:0/2211675891 >> 172.21.15.62:0/2211675891 conn(0x7ff1140fdf40 msgr2=0x7ff114100350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:38:27.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:27.580+0000 7ff11a400700 1 -- 172.21.15.62:0/2211675891 shutdown_connections 2024-04-20T21:38:27.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:27.580+0000 7ff11a400700 1 -- 172.21.15.62:0/2211675891 wait complete. 2024-04-20T21:38:27.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:27.581+0000 7ff11a400700 1 Processor -- start 2024-04-20T21:38:27.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:27.582+0000 7ff11a400700 1 -- start start 2024-04-20T21:38:27.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:27.582+0000 7ff11a400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff114102460 0x7ff114198200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:38:27.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:27.582+0000 7ff11a400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff114108460 0x7ff114198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:38:27.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:27.582+0000 7ff11a400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff114198e20 con 0x7ff114108460 2024-04-20T21:38:27.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:27.582+0000 7ff11a400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff11419cbb0 con 0x7ff114102460 2024-04-20T21:38:27.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:27.582+0000 7ff112a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff114108460 0x7ff114198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:38:27.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:27.583+0000 7ff112a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff114108460 0x7ff114198740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44214/0 (socket says 172.21.15.62:44214) 2024-04-20T21:38:27.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:27.583+0000 7ff112a00700 1 -- 172.21.15.62:0/1929264192 learned_addr learned my addr 172.21.15.62:0/1929264192 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:38:27.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:27.583+0000 7ff113400700 1 --2- 172.21.15.62:0/1929264192 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff114102460 0x7ff114198200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:38:27.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:27.583+0000 7ff112a00700 1 -- 172.21.15.62:0/1929264192 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff114102460 msgr2=0x7ff114198200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:38:27.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:27.583+0000 7ff112a00700 1 --2- 172.21.15.62:0/1929264192 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff114102460 0x7ff114198200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:38:27.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:27.583+0000 7ff112a00700 1 -- 172.21.15.62:0/1929264192 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1000095f0 con 0x7ff114108460 2024-04-20T21:38:27.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:27.584+0000 7ff112a00700 1 --2- 172.21.15.62:0/1929264192 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff114108460 0x7ff114198740 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff1000092e0 tx=0x7ff10000b6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:38:27.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:27.584+0000 7ff10be00700 1 -- 172.21.15.62:0/1929264192 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff10001d070 con 0x7ff114108460 2024-04-20T21:38:27.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:27.584+0000 7ff10be00700 1 -- 172.21.15.62:0/1929264192 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff1000052a0 con 0x7ff114108460 2024-04-20T21:38:27.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:27.584+0000 7ff11a400700 1 -- 172.21.15.62:0/1929264192 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff11419ce30 con 0x7ff114108460 2024-04-20T21:38:27.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:27.584+0000 7ff10be00700 1 -- 172.21.15.62:0/1929264192 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff100017400 con 0x7ff114108460 2024-04-20T21:38:27.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:27.584+0000 7ff11a400700 1 -- 172.21.15.62:0/1929264192 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff11419d320 con 0x7ff114108460 2024-04-20T21:38:27.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:27.586+0000 7ff11a400700 1 -- 172.21.15.62:0/1929264192 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff11410a960 con 0x7ff114108460 2024-04-20T21:38:27.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:27.587+0000 7ff10be00700 1 -- 172.21.15.62:0/1929264192 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff10000f820 con 0x7ff114108460 2024-04-20T21:38:27.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:27.588+0000 7ff10be00700 1 --2- 172.21.15.62:0/1929264192 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff0f806ecc0 0x7ff0f8071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:38:27.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:27.588+0000 7ff10be00700 1 -- 172.21.15.62:0/1929264192 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6328+0+0 (secure 0 0 0) 0x7ff1000900c0 con 0x7ff114108460 2024-04-20T21:38:27.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:27.589+0000 7ff113400700 1 --2- 172.21.15.62:0/1929264192 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff0f806ecc0 0x7ff0f8071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:38:27.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:27.591+0000 7ff113400700 1 --2- 172.21.15.62:0/1929264192 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff0f806ecc0 0x7ff0f8071170 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff104009c60 tx=0x7ff104009260 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:38:27.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:27.595+0000 7ff10be00700 1 -- 172.21.15.62:0/1929264192 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7ff100058f20 con 0x7ff114108460 2024-04-20T21:38:27.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:27 smithi189 ceph-mon[57022]: pgmap v59: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:38:27.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:27 smithi062 ceph-mon[76322]: pgmap v59: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:38:27.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:27.876+0000 7ff11a400700 1 -- 172.21.15.62:0/1929264192 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff114199560 con 0x7ff0f806ecc0 2024-04-20T21:38:27.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:27.879+0000 7ff10be00700 1 -- 172.21.15.62:0/1929264192 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7ff114199560 con 0x7ff0f806ecc0 2024-04-20T21:38:27.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:27.882+0000 7ff11a400700 1 -- 172.21.15.62:0/1929264192 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff0f806ecc0 msgr2=0x7ff0f8071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:38:27.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:27.882+0000 7ff11a400700 1 --2- 172.21.15.62:0/1929264192 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff0f806ecc0 0x7ff0f8071170 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff104009c60 tx=0x7ff104009260 comp rx=0 tx=0).stop 2024-04-20T21:38:27.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:27.882+0000 7ff11a400700 1 -- 172.21.15.62:0/1929264192 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff114108460 msgr2=0x7ff114198740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:38:27.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:27.882+0000 7ff11a400700 1 --2- 172.21.15.62:0/1929264192 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff114108460 0x7ff114198740 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff1000092e0 tx=0x7ff10000b6c0 comp rx=0 tx=0).stop 2024-04-20T21:38:27.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:27.883+0000 7ff11a400700 1 -- 172.21.15.62:0/1929264192 shutdown_connections 2024-04-20T21:38:27.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:27.883+0000 7ff11a400700 1 --2- 172.21.15.62:0/1929264192 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff0f806ecc0 0x7ff0f8071170 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:38:27.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:27.883+0000 7ff11a400700 1 --2- 172.21.15.62:0/1929264192 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff114102460 0x7ff114198200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:38:27.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:27.883+0000 7ff11a400700 1 --2- 172.21.15.62:0/1929264192 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff114108460 0x7ff114198740 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:38:27.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:27.883+0000 7ff11a400700 1 -- 172.21.15.62:0/1929264192 >> 172.21.15.62:0/1929264192 conn(0x7ff1140fdf40 msgr2=0x7ff1140ff2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:38:27.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:27.883+0000 7ff11a400700 1 -- 172.21.15.62:0/1929264192 shutdown_connections 2024-04-20T21:38:27.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:27.883+0000 7ff11a400700 1 -- 172.21.15.62:0/1929264192 wait complete. 2024-04-20T21:38:27.900 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T21:38:28.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.049+0000 7fbd37400700 1 -- 172.21.15.62:0/2129774734 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd30100e80 msgr2=0x7fbd30104e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:38:28.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.049+0000 7fbd37400700 1 --2- 172.21.15.62:0/2129774734 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd30100e80 0x7fbd30104e50 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fbd1c009910 tx=0x7fbd1c009c20 comp rx=0 tx=0).stop 2024-04-20T21:38:28.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.049+0000 7fbd37400700 1 -- 172.21.15.62:0/2129774734 shutdown_connections 2024-04-20T21:38:28.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.049+0000 7fbd37400700 1 --2- 172.21.15.62:0/2129774734 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd30100e80 0x7fbd30104e50 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:38:28.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.049+0000 7fbd37400700 1 --2- 172.21.15.62:0/2129774734 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd301004e0 0x7fbd301008b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:38:28.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.049+0000 7fbd37400700 1 -- 172.21.15.62:0/2129774734 >> 172.21.15.62:0/2129774734 conn(0x7fbd300fbda0 msgr2=0x7fbd300fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:38:28.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.049+0000 7fbd37400700 1 -- 172.21.15.62:0/2129774734 shutdown_connections 2024-04-20T21:38:28.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.049+0000 7fbd37400700 1 -- 172.21.15.62:0/2129774734 wait complete. 2024-04-20T21:38:28.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.050+0000 7fbd37400700 1 Processor -- start 2024-04-20T21:38:28.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.050+0000 7fbd37400700 1 -- start start 2024-04-20T21:38:28.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.051+0000 7fbd37400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd301004e0 0x7fbd30073050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:38:28.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.051+0000 7fbd37400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd30100e80 0x7fbd30073590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:38:28.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.051+0000 7fbd37400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd30073ad0 con 0x7fbd30100e80 2024-04-20T21:38:28.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.051+0000 7fbd37400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd30073c10 con 0x7fbd301004e0 2024-04-20T21:38:28.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.051+0000 7fbd2fe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd30100e80 0x7fbd30073590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:38:28.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.051+0000 7fbd2fe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd30100e80 0x7fbd30073590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44226/0 (socket says 172.21.15.62:44226) 2024-04-20T21:38:28.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.051+0000 7fbd2fe00700 1 -- 172.21.15.62:0/1761141078 learned_addr learned my addr 172.21.15.62:0/1761141078 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:38:28.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.051+0000 7fbd34a00700 1 --2- 172.21.15.62:0/1761141078 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd301004e0 0x7fbd30073050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:38:28.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.052+0000 7fbd2fe00700 1 -- 172.21.15.62:0/1761141078 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd301004e0 msgr2=0x7fbd30073050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:38:28.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.052+0000 7fbd2fe00700 1 --2- 172.21.15.62:0/1761141078 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd301004e0 0x7fbd30073050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:38:28.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.052+0000 7fbd2fe00700 1 -- 172.21.15.62:0/1761141078 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd1c0095f0 con 0x7fbd30100e80 2024-04-20T21:38:28.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.052+0000 7fbd34a00700 1 --2- 172.21.15.62:0/1761141078 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd301004e0 0x7fbd30073050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-20T21:38:28.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.052+0000 7fbd2fe00700 1 --2- 172.21.15.62:0/1761141078 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd30100e80 0x7fbd30073590 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fbd1c009fd0 tx=0x7fbd1c00f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:38:28.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.052+0000 7fbd2d600700 1 -- 172.21.15.62:0/1761141078 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbd1c01d070 con 0x7fbd30100e80 2024-04-20T21:38:28.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.053+0000 7fbd2d600700 1 -- 172.21.15.62:0/1761141078 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbd1c022470 con 0x7fbd30100e80 2024-04-20T21:38:28.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.053+0000 7fbd2d600700 1 -- 172.21.15.62:0/1761141078 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbd1c017610 con 0x7fbd30100e80 2024-04-20T21:38:28.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.053+0000 7fbd37400700 1 -- 172.21.15.62:0/1761141078 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd30073e90 con 0x7fbd30100e80 2024-04-20T21:38:28.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.053+0000 7fbd37400700 1 -- 172.21.15.62:0/1761141078 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd301a23d0 con 0x7fbd30100e80 2024-04-20T21:38:28.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.054+0000 7fbd37400700 1 -- 172.21.15.62:0/1761141078 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd30108850 con 0x7fbd30100e80 2024-04-20T21:38:28.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.056+0000 7fbd2d600700 1 -- 172.21.15.62:0/1761141078 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fbd1c0225e0 con 0x7fbd30100e80 2024-04-20T21:38:28.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.057+0000 7fbd2d600700 1 --2- 172.21.15.62:0/1761141078 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fbd1806ecc0 0x7fbd18071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:38:28.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.057+0000 7fbd2d600700 1 -- 172.21.15.62:0/1761141078 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fbd1c0903b0 con 0x7fbd30100e80 2024-04-20T21:38:28.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.058+0000 7fbd34a00700 1 --2- 172.21.15.62:0/1761141078 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fbd1806ecc0 0x7fbd18071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:38:28.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.059+0000 7fbd34a00700 1 --2- 172.21.15.62:0/1761141078 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fbd1806ecc0 0x7fbd18071170 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fbd24009cb0 tx=0x7fbd24009260 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:38:28.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.062+0000 7fbd2d600700 1 -- 172.21.15.62:0/1761141078 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fbd1c0591f0 con 0x7fbd30100e80 2024-04-20T21:38:28.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.357+0000 7fbd37400700 1 -- 172.21.15.62:0/1761141078 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fbd30061190 con 0x7fbd1806ecc0 2024-04-20T21:38:28.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.359+0000 7fbd2d600700 1 -- 172.21.15.62:0/1761141078 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7fbd30061190 con 0x7fbd1806ecc0 2024-04-20T21:38:28.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.362+0000 7fbd37400700 1 -- 172.21.15.62:0/1761141078 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fbd1806ecc0 msgr2=0x7fbd18071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:38:28.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.362+0000 7fbd37400700 1 --2- 172.21.15.62:0/1761141078 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fbd1806ecc0 0x7fbd18071170 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fbd24009cb0 tx=0x7fbd24009260 comp rx=0 tx=0).stop 2024-04-20T21:38:28.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.362+0000 7fbd37400700 1 -- 172.21.15.62:0/1761141078 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd30100e80 msgr2=0x7fbd30073590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:38:28.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.362+0000 7fbd37400700 1 --2- 172.21.15.62:0/1761141078 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd30100e80 0x7fbd30073590 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fbd1c009fd0 tx=0x7fbd1c00f690 comp rx=0 tx=0).stop 2024-04-20T21:38:28.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.363+0000 7fbd37400700 1 -- 172.21.15.62:0/1761141078 shutdown_connections 2024-04-20T21:38:28.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.363+0000 7fbd37400700 1 --2- 172.21.15.62:0/1761141078 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fbd1806ecc0 0x7fbd18071170 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:38:28.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.363+0000 7fbd37400700 1 --2- 172.21.15.62:0/1761141078 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd301004e0 0x7fbd30073050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:38:28.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.363+0000 7fbd37400700 1 --2- 172.21.15.62:0/1761141078 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd30100e80 0x7fbd30073590 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:38:28.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.363+0000 7fbd37400700 1 -- 172.21.15.62:0/1761141078 >> 172.21.15.62:0/1761141078 conn(0x7fbd300fbda0 msgr2=0x7fbd300fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:38:28.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.363+0000 7fbd37400700 1 -- 172.21.15.62:0/1761141078 shutdown_connections 2024-04-20T21:38:28.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.363+0000 7fbd37400700 1 -- 172.21.15.62:0/1761141078 wait complete. 2024-04-20T21:38:28.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.523+0000 7fda1ca00700 1 -- 172.21.15.62:0/2784251647 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fda18068540 msgr2=0x7fda180689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:38:28.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.523+0000 7fda1ca00700 1 --2- 172.21.15.62:0/2784251647 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fda18068540 0x7fda180689b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fda04009910 tx=0x7fda04009c20 comp rx=0 tx=0).stop 2024-04-20T21:38:28.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.523+0000 7fda1ca00700 1 -- 172.21.15.62:0/2784251647 shutdown_connections 2024-04-20T21:38:28.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.523+0000 7fda1ca00700 1 --2- 172.21.15.62:0/2784251647 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fda18068540 0x7fda180689b0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:38:28.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.523+0000 7fda1ca00700 1 --2- 172.21.15.62:0/2784251647 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fda18105230 0x7fda18105600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:38:28.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.523+0000 7fda1ca00700 1 -- 172.21.15.62:0/2784251647 >> 172.21.15.62:0/2784251647 conn(0x7fda18075130 msgr2=0x7fda18075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:38:28.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.523+0000 7fda1ca00700 1 -- 172.21.15.62:0/2784251647 shutdown_connections 2024-04-20T21:38:28.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.523+0000 7fda1ca00700 1 -- 172.21.15.62:0/2784251647 wait complete. 2024-04-20T21:38:28.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.525+0000 7fda1ca00700 1 Processor -- start 2024-04-20T21:38:28.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.525+0000 7fda1ca00700 1 -- start start 2024-04-20T21:38:28.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.525+0000 7fda1ca00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fda18068540 0x7fda180ffca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:38:28.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.525+0000 7fda1ca00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fda18105230 0x7fda181001e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:38:28.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.525+0000 7fda1ca00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda18100720 con 0x7fda18068540 2024-04-20T21:38:28.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.525+0000 7fda1ca00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda18100860 con 0x7fda18105230 2024-04-20T21:38:28.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.526+0000 7fda15e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fda18068540 0x7fda180ffca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:38:28.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.526+0000 7fda15e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fda18068540 0x7fda180ffca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44236/0 (socket says 172.21.15.62:44236) 2024-04-20T21:38:28.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.526+0000 7fda15e00700 1 -- 172.21.15.62:0/2726992817 learned_addr learned my addr 172.21.15.62:0/2726992817 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:38:28.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.526+0000 7fda15400700 1 --2- 172.21.15.62:0/2726992817 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fda18105230 0x7fda181001e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:38:28.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.526+0000 7fda15e00700 1 -- 172.21.15.62:0/2726992817 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fda18105230 msgr2=0x7fda181001e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:38:28.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.526+0000 7fda15e00700 1 --2- 172.21.15.62:0/2726992817 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fda18105230 0x7fda181001e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:38:28.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.526+0000 7fda15e00700 1 -- 172.21.15.62:0/2726992817 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda040095f0 con 0x7fda18068540 2024-04-20T21:38:28.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.526+0000 7fda15400700 1 --2- 172.21.15.62:0/2726992817 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fda18105230 0x7fda181001e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T21:38:28.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.527+0000 7fda15e00700 1 --2- 172.21.15.62:0/2726992817 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fda18068540 0x7fda180ffca0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fda0800b4d0 tx=0x7fda0800b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:38:28.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.527+0000 7fda0ea00700 1 -- 172.21.15.62:0/2726992817 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fda08011800 con 0x7fda18068540 2024-04-20T21:38:28.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.527+0000 7fda0ea00700 1 -- 172.21.15.62:0/2726992817 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fda08011e20 con 0x7fda18068540 2024-04-20T21:38:28.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.527+0000 7fda1ca00700 1 -- 172.21.15.62:0/2726992817 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda181a20f0 con 0x7fda18068540 2024-04-20T21:38:28.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.527+0000 7fda0ea00700 1 -- 172.21.15.62:0/2726992817 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fda0800f2e0 con 0x7fda18068540 2024-04-20T21:38:28.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.527+0000 7fda1ca00700 1 -- 172.21.15.62:0/2726992817 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda181a2500 con 0x7fda18068540 2024-04-20T21:38:28.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.529+0000 7fda1ca00700 1 -- 172.21.15.62:0/2726992817 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda18108750 con 0x7fda18068540 2024-04-20T21:38:28.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.530+0000 7fda0ea00700 1 -- 172.21.15.62:0/2726992817 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fda08010380 con 0x7fda18068540 2024-04-20T21:38:28.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.531+0000 7fda0ea00700 1 --2- 172.21.15.62:0/2726992817 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd9fc06ecc0 0x7fd9fc071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:38:28.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.532+0000 7fda15400700 1 --2- 172.21.15.62:0/2726992817 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd9fc06ecc0 0x7fd9fc071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:38:28.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.532+0000 7fda0ea00700 1 -- 172.21.15.62:0/2726992817 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fda0808ea10 con 0x7fda18068540 2024-04-20T21:38:28.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.533+0000 7fda15400700 1 --2- 172.21.15.62:0/2726992817 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd9fc06ecc0 0x7fd9fc071170 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fda04006010 tx=0x7fda040058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:38:28.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.536+0000 7fda0ea00700 1 -- 172.21.15.62:0/2726992817 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fda080577d0 con 0x7fda18068540 2024-04-20T21:38:28.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:28 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:38:28.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:28 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:38:28.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:28 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:38:28.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:28 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T21:38:28.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:28 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:38:28.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:28 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:38:28.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:28 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:38:28.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:28 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:38:28.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:28 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:38:28.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:28 smithi062 ceph-mon[76322]: Upgrade: Setting container_image for all crash 2024-04-20T21:38:28.787 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:28 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:38:28.787 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:28 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi062"}]: dispatch 2024-04-20T21:38:28.787 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:28 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi062"}]': finished 2024-04-20T21:38:28.787 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:28 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi189"}]: dispatch 2024-04-20T21:38:28.787 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:28 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi189"}]': finished 2024-04-20T21:38:28.787 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:28 smithi062 ceph-mon[76322]: from='client.34214 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:38:28.787 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:28 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-20T21:38:28.787 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:28 smithi062 ceph-mon[76322]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-20T21:38:28.787 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:28 smithi062 ceph-mon[76322]: Upgrade: osd.0 is safe to restart 2024-04-20T21:38:28.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.792+0000 7fda1ca00700 1 -- 172.21.15.62:0/2726992817 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fda18061190 con 0x7fd9fc06ecc0 2024-04-20T21:38:28.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.814+0000 7fda0ea00700 1 -- 172.21.15.62:0/2726992817 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fda18061190 con 0x7fd9fc06ecc0 2024-04-20T21:38:28.814 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T21:38:28.814 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (4m) 9s ago 18m 14.0M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T21:38:28.814 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (18m) 9s ago 18m 8464k - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T21:38:28.814 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (17m) 12s ago 17m 8288k - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T21:38:28.815 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (25s) 9s ago 18m 6660k - 19.0.0-3133-ga8c84f59 76bc6b91b726 2825d5ba11c6 2024-04-20T21:38:28.815 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (17s) 12s ago 17m 6665k - 19.0.0-3133-ga8c84f59 76bc6b91b726 20f746fa1bba 2024-04-20T21:38:28.815 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (3m) 9s ago 17m 74.7M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T21:38:28.815 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (10m) 9s ago 10m 65.8M - 18.2.0 dc2bc1663786 5e6460aa6159 2024-04-20T21:38:28.815 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (11m) 9s ago 11m 131M - 18.2.0 dc2bc1663786 e65f2267544e 2024-04-20T21:38:28.815 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (10m) 12s ago 10m 117M - 18.2.0 dc2bc1663786 fb2a8a223db8 2024-04-20T21:38:28.815 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (10m) 12s ago 10m 55.6M - 18.2.0 dc2bc1663786 f2eec2ee8951 2024-04-20T21:38:28.815 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (7m) 9s ago 19m 498M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T21:38:28.815 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (5m) 12s ago 16m 450M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T21:38:28.815 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (2m) 9s ago 20m 45.5M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 019eb1479485 2024-04-20T21:38:28.815 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (88s) 12s ago 16m 36.6M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 55aac560aab6 2024-04-20T21:38:28.815 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (5m) 9s ago 18m 7470k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T21:38:28.815 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (5m) 12s ago 16m 7972k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T21:38:28.815 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (15m) 9s ago 15m 218M 1216M 18.2.0 dc2bc1663786 3f315c3afc1f 2024-04-20T21:38:28.815 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (15m) 9s ago 15m 211M 1216M 18.2.0 dc2bc1663786 44e4d07416ad 2024-04-20T21:38:28.816 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (14m) 9s ago 14m 173M 1216M 18.2.0 dc2bc1663786 31c5d9586e98 2024-04-20T21:38:28.816 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (14m) 12s ago 14m 242M 2240M 18.2.0 dc2bc1663786 611c1e242693 2024-04-20T21:38:28.816 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (13m) 12s ago 13m 221M 2240M 18.2.0 dc2bc1663786 6c8fce91d45d 2024-04-20T21:38:28.816 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (12m) 12s ago 12m 205M 2240M 18.2.0 dc2bc1663786 8e4574bae8db 2024-04-20T21:38:28.816 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (4m) 9s ago 17m 72.9M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T21:38:28.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.818+0000 7fda1ca00700 1 -- 172.21.15.62:0/2726992817 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd9fc06ecc0 msgr2=0x7fd9fc071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:38:28.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.818+0000 7fda1ca00700 1 --2- 172.21.15.62:0/2726992817 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd9fc06ecc0 0x7fd9fc071170 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fda04006010 tx=0x7fda040058e0 comp rx=0 tx=0).stop 2024-04-20T21:38:28.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.818+0000 7fda1ca00700 1 -- 172.21.15.62:0/2726992817 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fda18068540 msgr2=0x7fda180ffca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:38:28.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.818+0000 7fda1ca00700 1 --2- 172.21.15.62:0/2726992817 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fda18068540 0x7fda180ffca0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fda0800b4d0 tx=0x7fda0800b890 comp rx=0 tx=0).stop 2024-04-20T21:38:28.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.819+0000 7fda1ca00700 1 -- 172.21.15.62:0/2726992817 shutdown_connections 2024-04-20T21:38:28.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.819+0000 7fda1ca00700 1 --2- 172.21.15.62:0/2726992817 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd9fc06ecc0 0x7fd9fc071170 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:38:28.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.819+0000 7fda1ca00700 1 --2- 172.21.15.62:0/2726992817 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fda18068540 0x7fda180ffca0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:38:28.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.819+0000 7fda1ca00700 1 --2- 172.21.15.62:0/2726992817 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fda18105230 0x7fda181001e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:38:28.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.819+0000 7fda1ca00700 1 -- 172.21.15.62:0/2726992817 >> 172.21.15.62:0/2726992817 conn(0x7fda18075130 msgr2=0x7fda180fdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:38:28.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.820+0000 7fda1ca00700 1 -- 172.21.15.62:0/2726992817 shutdown_connections 2024-04-20T21:38:28.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.820+0000 7fda1ca00700 1 -- 172.21.15.62:0/2726992817 wait complete. 2024-04-20T21:38:28.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.979+0000 7ff8eea00700 1 -- 172.21.15.62:0/994136461 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff8e8100290 msgr2=0x7ff8e8100660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:38:28.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.979+0000 7ff8eea00700 1 --2- 172.21.15.62:0/994136461 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff8e8100290 0x7ff8e8100660 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ff8dc009910 tx=0x7ff8dc009c20 comp rx=0 tx=0).stop 2024-04-20T21:38:28.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.979+0000 7ff8eea00700 1 -- 172.21.15.62:0/994136461 shutdown_connections 2024-04-20T21:38:28.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.979+0000 7ff8eea00700 1 --2- 172.21.15.62:0/994136461 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8e8100c30 0x7ff8e8104c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:38:28.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.979+0000 7ff8eea00700 1 --2- 172.21.15.62:0/994136461 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff8e8100290 0x7ff8e8100660 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:38:28.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.979+0000 7ff8eea00700 1 -- 172.21.15.62:0/994136461 >> 172.21.15.62:0/994136461 conn(0x7ff8e80fbb10 msgr2=0x7ff8e80fdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:38:28.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.979+0000 7ff8eea00700 1 -- 172.21.15.62:0/994136461 shutdown_connections 2024-04-20T21:38:28.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.979+0000 7ff8eea00700 1 -- 172.21.15.62:0/994136461 wait complete. 2024-04-20T21:38:28.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.980+0000 7ff8eea00700 1 Processor -- start 2024-04-20T21:38:28.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.981+0000 7ff8eea00700 1 -- start start 2024-04-20T21:38:28.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.981+0000 7ff8eea00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff8e8100290 0x7ff8e8197e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:38:28.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.981+0000 7ff8eea00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8e8100c30 0x7ff8e8198340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:38:28.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.981+0000 7ff8eea00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8e8198a20 con 0x7ff8e8100290 2024-04-20T21:38:28.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.981+0000 7ff8eea00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8e819c7b0 con 0x7ff8e8100c30 2024-04-20T21:38:28.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.981+0000 7ff8ed600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff8e8100290 0x7ff8e8197e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:38:28.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.981+0000 7ff8ed600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff8e8100290 0x7ff8e8197e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44254/0 (socket says 172.21.15.62:44254) 2024-04-20T21:38:28.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.981+0000 7ff8ed600700 1 -- 172.21.15.62:0/732851503 learned_addr learned my addr 172.21.15.62:0/732851503 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:38:28.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.982+0000 7ff8ecc00700 1 --2- 172.21.15.62:0/732851503 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8e8100c30 0x7ff8e8198340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:38:28.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.982+0000 7ff8ed600700 1 -- 172.21.15.62:0/732851503 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8e8100c30 msgr2=0x7ff8e8198340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:38:28.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.982+0000 7ff8ed600700 1 --2- 172.21.15.62:0/732851503 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8e8100c30 0x7ff8e8198340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:38:28.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.982+0000 7ff8ed600700 1 -- 172.21.15.62:0/732851503 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8dc0095f0 con 0x7ff8e8100290 2024-04-20T21:38:28.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.982+0000 7ff8ed600700 1 --2- 172.21.15.62:0/732851503 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff8e8100290 0x7ff8e8197e00 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff8dc0092e0 tx=0x7ff8dc00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:38:28.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.983+0000 7ff8e6000700 1 -- 172.21.15.62:0/732851503 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff8dc01d070 con 0x7ff8e8100290 2024-04-20T21:38:28.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.983+0000 7ff8e6000700 1 -- 172.21.15.62:0/732851503 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff8dc005e00 con 0x7ff8e8100290 2024-04-20T21:38:28.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.983+0000 7ff8e6000700 1 -- 172.21.15.62:0/732851503 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff8dc017400 con 0x7ff8e8100290 2024-04-20T21:38:28.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.983+0000 7ff8eea00700 1 -- 172.21.15.62:0/732851503 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8e819ca30 con 0x7ff8e8100290 2024-04-20T21:38:28.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.984+0000 7ff8eea00700 1 -- 172.21.15.62:0/732851503 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8e819cf20 con 0x7ff8e8100290 2024-04-20T21:38:28.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.985+0000 7ff8eea00700 1 -- 172.21.15.62:0/732851503 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8e8104380 con 0x7ff8e8100290 2024-04-20T21:38:28.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.987+0000 7ff8e6000700 1 -- 172.21.15.62:0/732851503 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff8dc00f460 con 0x7ff8e8100290 2024-04-20T21:38:28.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.988+0000 7ff8e6000700 1 --2- 172.21.15.62:0/732851503 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff8d006ecc0 0x7ff8d0071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:38:28.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.988+0000 7ff8e6000700 1 -- 172.21.15.62:0/732851503 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6328+0+0 (secure 0 0 0) 0x7ff8dc090280 con 0x7ff8e8100290 2024-04-20T21:38:28.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.988+0000 7ff8ecc00700 1 --2- 172.21.15.62:0/732851503 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff8d006ecc0 0x7ff8d0071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:38:28.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:28 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:38:28.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:28 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:38:28.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:28 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:38:28.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:28 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T21:38:28.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:28 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:38:28.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:28 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:38:28.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:28 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:38:28.989 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:28 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:38:28.989 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:28 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:38:28.989 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:28 smithi189 ceph-mon[57022]: Upgrade: Setting container_image for all crash 2024-04-20T21:38:28.989 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:28 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:38:28.989 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:28 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi062"}]: dispatch 2024-04-20T21:38:28.989 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:28 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi062"}]': finished 2024-04-20T21:38:28.989 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:28 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi189"}]: dispatch 2024-04-20T21:38:28.989 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:28 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi189"}]': finished 2024-04-20T21:38:28.989 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:28 smithi189 ceph-mon[57022]: from='client.34214 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:38:28.989 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:28 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-20T21:38:28.989 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:28 smithi189 ceph-mon[57022]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-20T21:38:28.989 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:28 smithi189 ceph-mon[57022]: Upgrade: osd.0 is safe to restart 2024-04-20T21:38:28.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.989+0000 7ff8ecc00700 1 --2- 172.21.15.62:0/732851503 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff8d006ecc0 0x7ff8d0071170 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff8d4005950 tx=0x7ff8d40058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:38:28.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:28.993+0000 7ff8e6000700 1 -- 172.21.15.62:0/732851503 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7ff8dc0590c0 con 0x7ff8e8100290 2024-04-20T21:38:29.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:29.327+0000 7ff8eea00700 1 -- 172.21.15.62:0/732851503 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7ff8e819d200 con 0x7ff8e8100290 2024-04-20T21:38:29.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:29.328+0000 7ff8e6000700 1 -- 172.21.15.62:0/732851503 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+688 (secure 0 0 0) 0x7ff8dc05c780 con 0x7ff8e8100290 2024-04-20T21:38:29.327 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:38:29.327 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T21:38:29.327 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:38:29.327 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:38:29.327 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T21:38:29.327 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:38:29.327 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:38:29.328 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T21:38:29.328 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 6 2024-04-20T21:38:29.328 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:38:29.328 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T21:38:29.328 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 4 2024-04-20T21:38:29.328 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:38:29.328 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T21:38:29.328 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 10, 2024-04-20T21:38:29.328 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 4 2024-04-20T21:38:29.328 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T21:38:29.328 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:38:29.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:29.330+0000 7ff8eea00700 1 -- 172.21.15.62:0/732851503 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff8d006ecc0 msgr2=0x7ff8d0071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:38:29.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:29.330+0000 7ff8eea00700 1 --2- 172.21.15.62:0/732851503 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff8d006ecc0 0x7ff8d0071170 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff8d4005950 tx=0x7ff8d40058e0 comp rx=0 tx=0).stop 2024-04-20T21:38:29.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:29.330+0000 7ff8eea00700 1 -- 172.21.15.62:0/732851503 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff8e8100290 msgr2=0x7ff8e8197e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:38:29.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:29.330+0000 7ff8eea00700 1 --2- 172.21.15.62:0/732851503 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff8e8100290 0x7ff8e8197e00 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff8dc0092e0 tx=0x7ff8dc00be20 comp rx=0 tx=0).stop 2024-04-20T21:38:29.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:29.330+0000 7ff8eea00700 1 -- 172.21.15.62:0/732851503 shutdown_connections 2024-04-20T21:38:29.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:29.330+0000 7ff8eea00700 1 --2- 172.21.15.62:0/732851503 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff8d006ecc0 0x7ff8d0071170 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:38:29.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:29.330+0000 7ff8eea00700 1 --2- 172.21.15.62:0/732851503 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff8e8100290 0x7ff8e8197e00 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:38:29.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:29.330+0000 7ff8eea00700 1 --2- 172.21.15.62:0/732851503 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8e8100c30 0x7ff8e8198340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:38:29.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:29.330+0000 7ff8eea00700 1 -- 172.21.15.62:0/732851503 >> 172.21.15.62:0/732851503 conn(0x7ff8e80fbb10 msgr2=0x7ff8e80fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:38:29.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:29.331+0000 7ff8eea00700 1 -- 172.21.15.62:0/732851503 shutdown_connections 2024-04-20T21:38:29.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:29.331+0000 7ff8eea00700 1 -- 172.21.15.62:0/732851503 wait complete. 2024-04-20T21:38:29.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:29.482+0000 7f3d94800700 1 -- 172.21.15.62:0/3351232919 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3d8c104130 msgr2=0x7f3d8c104500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:38:29.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:29.482+0000 7f3d94800700 1 --2- 172.21.15.62:0/3351232919 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3d8c104130 0x7f3d8c104500 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f3d80009910 tx=0x7f3d80009c20 comp rx=0 tx=0).stop 2024-04-20T21:38:29.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:29.482+0000 7f3d94800700 1 -- 172.21.15.62:0/3351232919 shutdown_connections 2024-04-20T21:38:29.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:29.482+0000 7f3d94800700 1 --2- 172.21.15.62:0/3351232919 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3d8c0ffbd0 0x7f3d8c100040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:38:29.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:29.482+0000 7f3d94800700 1 --2- 172.21.15.62:0/3351232919 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3d8c104130 0x7f3d8c104500 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:38:29.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:29.482+0000 7f3d94800700 1 -- 172.21.15.62:0/3351232919 >> 172.21.15.62:0/3351232919 conn(0x7f3d8c075130 msgr2=0x7f3d8c075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:38:29.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:29.482+0000 7f3d94800700 1 -- 172.21.15.62:0/3351232919 shutdown_connections 2024-04-20T21:38:29.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:29.482+0000 7f3d94800700 1 -- 172.21.15.62:0/3351232919 wait complete. 2024-04-20T21:38:29.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:29.483+0000 7f3d94800700 1 Processor -- start 2024-04-20T21:38:29.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:29.484+0000 7f3d94800700 1 -- start start 2024-04-20T21:38:29.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:29.484+0000 7f3d94800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3d8c0ffbd0 0x7f3d8c198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:38:29.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:29.484+0000 7f3d94800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3d8c104130 0x7f3d8c198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:38:29.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:29.484+0000 7f3d94800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d8c198c60 con 0x7f3d8c0ffbd0 2024-04-20T21:38:29.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:29.484+0000 7f3d94800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d8c19c9f0 con 0x7f3d8c104130 2024-04-20T21:38:29.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:29.484+0000 7f3d91e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3d8c0ffbd0 0x7f3d8c198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:38:29.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:29.485+0000 7f3d91e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3d8c0ffbd0 0x7f3d8c198040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44274/0 (socket says 172.21.15.62:44274) 2024-04-20T21:38:29.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:29.485+0000 7f3d91e00700 1 -- 172.21.15.62:0/2673146249 learned_addr learned my addr 172.21.15.62:0/2673146249 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:38:29.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:29.485+0000 7f3d91400700 1 --2- 172.21.15.62:0/2673146249 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3d8c104130 0x7f3d8c198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:38:29.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:29.485+0000 7f3d91e00700 1 -- 172.21.15.62:0/2673146249 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3d8c104130 msgr2=0x7f3d8c198580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:38:29.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:29.485+0000 7f3d91e00700 1 --2- 172.21.15.62:0/2673146249 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3d8c104130 0x7f3d8c198580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:38:29.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:29.485+0000 7f3d91e00700 1 -- 172.21.15.62:0/2673146249 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d800095f0 con 0x7f3d8c0ffbd0 2024-04-20T21:38:29.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:29.485+0000 7f3d91e00700 1 --2- 172.21.15.62:0/2673146249 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3d8c0ffbd0 0x7f3d8c198040 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3d800092e0 tx=0x7f3d8000be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:38:29.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:29.486+0000 7f3d86a00700 1 -- 172.21.15.62:0/2673146249 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3d8001d070 con 0x7f3d8c0ffbd0 2024-04-20T21:38:29.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:29.486+0000 7f3d86a00700 1 -- 172.21.15.62:0/2673146249 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3d8000f460 con 0x7f3d8c0ffbd0 2024-04-20T21:38:29.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:29.486+0000 7f3d86a00700 1 -- 172.21.15.62:0/2673146249 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3d80017630 con 0x7f3d8c0ffbd0 2024-04-20T21:38:29.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:29.486+0000 7f3d94800700 1 -- 172.21.15.62:0/2673146249 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d8c19cc70 con 0x7f3d8c0ffbd0 2024-04-20T21:38:29.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:29.486+0000 7f3d94800700 1 -- 172.21.15.62:0/2673146249 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d8c19d160 con 0x7f3d8c0ffbd0 2024-04-20T21:38:29.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:29.487+0000 7f3d94800700 1 -- 172.21.15.62:0/2673146249 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d8c04f2a0 con 0x7f3d8c0ffbd0 2024-04-20T21:38:29.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:29.489+0000 7f3d86a00700 1 -- 172.21.15.62:0/2673146249 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f3d8000f5d0 con 0x7f3d8c0ffbd0 2024-04-20T21:38:29.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:29.490+0000 7f3d86a00700 1 --2- 172.21.15.62:0/2673146249 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3d7406ecc0 0x7f3d74071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:38:29.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:29.490+0000 7f3d86a00700 1 -- 172.21.15.62:0/2673146249 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f3d80090380 con 0x7f3d8c0ffbd0 2024-04-20T21:38:29.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:29.490+0000 7f3d91400700 1 --2- 172.21.15.62:0/2673146249 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3d7406ecc0 0x7f3d74071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:38:29.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:29.491+0000 7f3d91400700 1 --2- 172.21.15.62:0/2673146249 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3d7406ecc0 0x7f3d74071170 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f3d7c005fd0 tx=0x7f3d7c00a560 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:38:29.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:29.500+0000 7f3d86a00700 1 -- 172.21.15.62:0/2673146249 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f3d800591c0 con 0x7f3d8c0ffbd0 2024-04-20T21:38:29.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:29.782+0000 7f3d94800700 1 -- 172.21.15.62:0/2673146249 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f3d8c19d490 con 0x7f3d8c0ffbd0 2024-04-20T21:38:29.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:29.783+0000 7f3d86a00700 1 -- 172.21.15.62:0/2673146249 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 18 v18) v1 ==== 76+0+1928 (secure 0 0 0) 0x7f3d8005c880 con 0x7f3d8c0ffbd0 2024-04-20T21:38:29.782 INFO:teuthology.orchestra.run.smithi062.stdout:e18 2024-04-20T21:38:29.783 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T21:38:29.783 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T21:38:29.783 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T21:38:29.783 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:38:29.783 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T21:38:29.783 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T21:38:29.783 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 18 2024-04-20T21:38:29.783 INFO:teuthology.orchestra.run.smithi062.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-20T21:38:29.783 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T21:38:29.783 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T21:37:06.018300+0000 2024-04-20T21:38:29.783 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T21:38:29.783 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T21:38:29.783 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T21:38:29.783 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T21:38:29.784 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T21:38:29.784 INFO:teuthology.orchestra.run.smithi062.stdout:max_xattr_size 65536 2024-04-20T21:38:29.784 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T21:38:29.784 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T21:38:29.784 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 0 2024-04-20T21:38:29.784 INFO:teuthology.orchestra.run.smithi062.stdout:snap_rstat 1 2024-04-20T21:38:29.784 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T21:38:29.784 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 2 2024-04-20T21:38:29.784 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1 2024-04-20T21:38:29.784 INFO:teuthology.orchestra.run.smithi062.stdout:up {0=14458,1=24279} 2024-04-20T21:38:29.784 INFO:teuthology.orchestra.run.smithi062.stdout:failed 2024-04-20T21:38:29.784 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T21:38:29.784 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T21:38:29.784 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T21:38:29.784 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T21:38:29.785 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data enabled 2024-04-20T21:38:29.785 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T21:38:29.785 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T21:38:29.785 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 1 2024-04-20T21:38:29.785 INFO:teuthology.orchestra.run.smithi062.stdout:qdb_cluster leader: 14458 members: 24279,14458 2024-04-20T21:38:29.785 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ryqblb{0:14458} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.62:6826/1147127123,v1:172.21.15.62:6827/1147127123] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:38:29.785 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ozizwi{0:14474} state up:standby-replay seq 3 join_fscid=1 addr [v2:172.21.15.62:6828/3494174293,v1:172.21.15.62:6829/3494174293] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:38:29.785 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.dvimer{1:24279} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.189:6824/237262514,v1:172.21.15.189:6825/237262514] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:38:29.785 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.xbgspw{1:24291} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.189:6826/1495356258,v1:172.21.15.189:6827/1495356258] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:38:29.785 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:38:29.785 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:38:29.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:29.786+0000 7f3d94800700 1 -- 172.21.15.62:0/2673146249 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3d7406ecc0 msgr2=0x7f3d74071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:38:29.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:29.786+0000 7f3d94800700 1 --2- 172.21.15.62:0/2673146249 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3d7406ecc0 0x7f3d74071170 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f3d7c005fd0 tx=0x7f3d7c00a560 comp rx=0 tx=0).stop 2024-04-20T21:38:29.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:29.786+0000 7f3d94800700 1 -- 172.21.15.62:0/2673146249 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3d8c0ffbd0 msgr2=0x7f3d8c198040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:38:29.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:29.786+0000 7f3d94800700 1 --2- 172.21.15.62:0/2673146249 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3d8c0ffbd0 0x7f3d8c198040 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3d800092e0 tx=0x7f3d8000be00 comp rx=0 tx=0).stop 2024-04-20T21:38:29.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:29.786+0000 7f3d94800700 1 -- 172.21.15.62:0/2673146249 shutdown_connections 2024-04-20T21:38:29.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:29.786+0000 7f3d94800700 1 --2- 172.21.15.62:0/2673146249 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3d7406ecc0 0x7f3d74071170 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:38:29.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:29.786+0000 7f3d94800700 1 --2- 172.21.15.62:0/2673146249 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3d8c0ffbd0 0x7f3d8c198040 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:38:29.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:29.786+0000 7f3d94800700 1 --2- 172.21.15.62:0/2673146249 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3d8c104130 0x7f3d8c198580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:38:29.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:29.786+0000 7f3d94800700 1 -- 172.21.15.62:0/2673146249 >> 172.21.15.62:0/2673146249 conn(0x7f3d8c075130 msgr2=0x7f3d8c068dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:38:29.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:29.786+0000 7f3d94800700 1 -- 172.21.15.62:0/2673146249 shutdown_connections 2024-04-20T21:38:29.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:29.786+0000 7f3d94800700 1 -- 172.21.15.62:0/2673146249 wait complete. 2024-04-20T21:38:29.787 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 18 2024-04-20T21:38:29.949 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:29.950+0000 7f6f92400700 1 -- 172.21.15.62:0/1288350820 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6f8c103040 msgr2=0x7f6f8c107010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:38:29.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:29.950+0000 7f6f92400700 1 --2- 172.21.15.62:0/1288350820 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6f8c103040 0x7f6f8c107010 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f6f78009910 tx=0x7f6f78009c20 comp rx=0 tx=0).stop 2024-04-20T21:38:29.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:29.950+0000 7f6f92400700 1 -- 172.21.15.62:0/1288350820 shutdown_connections 2024-04-20T21:38:29.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:29.950+0000 7f6f92400700 1 --2- 172.21.15.62:0/1288350820 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6f8c103040 0x7f6f8c107010 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:38:29.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:29.950+0000 7f6f92400700 1 --2- 172.21.15.62:0/1288350820 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f8c1026a0 0x7f6f8c102a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:38:29.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:29.950+0000 7f6f92400700 1 -- 172.21.15.62:0/1288350820 >> 172.21.15.62:0/1288350820 conn(0x7f6f8c0fdf40 msgr2=0x7f6f8c100350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:38:29.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:29.950+0000 7f6f92400700 1 -- 172.21.15.62:0/1288350820 shutdown_connections 2024-04-20T21:38:29.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:29.950+0000 7f6f92400700 1 -- 172.21.15.62:0/1288350820 wait complete. 2024-04-20T21:38:29.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:29.951+0000 7f6f92400700 1 Processor -- start 2024-04-20T21:38:29.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:29.951+0000 7f6f92400700 1 -- start start 2024-04-20T21:38:29.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:29.952+0000 7f6f92400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f8c1026a0 0x7f6f8c198280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:38:29.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:29.952+0000 7f6f92400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6f8c103040 0x7f6f8c1987c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:38:29.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:29.952+0000 7f6f92400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f8c198ea0 con 0x7f6f8c103040 2024-04-20T21:38:29.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:29.952+0000 7f6f92400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f8c19cc30 con 0x7f6f8c1026a0 2024-04-20T21:38:29.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:29.952+0000 7f6f8aa00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6f8c103040 0x7f6f8c1987c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:38:29.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:29.952+0000 7f6f8aa00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6f8c103040 0x7f6f8c1987c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44286/0 (socket says 172.21.15.62:44286) 2024-04-20T21:38:29.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:29.952+0000 7f6f8aa00700 1 -- 172.21.15.62:0/337365393 learned_addr learned my addr 172.21.15.62:0/337365393 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:38:29.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:29.952+0000 7f6f8b400700 1 --2- 172.21.15.62:0/337365393 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f8c1026a0 0x7f6f8c198280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:38:29.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:29.952+0000 7f6f8aa00700 1 -- 172.21.15.62:0/337365393 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f8c1026a0 msgr2=0x7f6f8c198280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:38:29.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:29.952+0000 7f6f8aa00700 1 --2- 172.21.15.62:0/337365393 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f8c1026a0 0x7f6f8c198280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:38:29.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:29.952+0000 7f6f8aa00700 1 -- 172.21.15.62:0/337365393 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f780095f0 con 0x7f6f8c103040 2024-04-20T21:38:29.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:29.953+0000 7f6f8aa00700 1 --2- 172.21.15.62:0/337365393 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6f8c103040 0x7f6f8c1987c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6f7800bda0 tx=0x7f6f7800be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:38:29.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:29.953+0000 7f6f83e00700 1 -- 172.21.15.62:0/337365393 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6f7801d070 con 0x7f6f8c103040 2024-04-20T21:38:29.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:29.953+0000 7f6f83e00700 1 -- 172.21.15.62:0/337365393 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6f78004fb0 con 0x7f6f8c103040 2024-04-20T21:38:29.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:29.953+0000 7f6f92400700 1 -- 172.21.15.62:0/337365393 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f8c19ceb0 con 0x7f6f8c103040 2024-04-20T21:38:29.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:29.953+0000 7f6f83e00700 1 -- 172.21.15.62:0/337365393 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6f7800f460 con 0x7f6f8c103040 2024-04-20T21:38:29.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:29.953+0000 7f6f92400700 1 -- 172.21.15.62:0/337365393 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f8c19d3a0 con 0x7f6f8c103040 2024-04-20T21:38:29.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:29.954+0000 7f6f92400700 1 -- 172.21.15.62:0/337365393 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f8c10a960 con 0x7f6f8c103040 2024-04-20T21:38:29.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:29.955+0000 7f6f83e00700 1 -- 172.21.15.62:0/337365393 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f6f78022800 con 0x7f6f8c103040 2024-04-20T21:38:29.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:29.956+0000 7f6f83e00700 1 --2- 172.21.15.62:0/337365393 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6f7006ecc0 0x7f6f70071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:38:29.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:29.956+0000 7f6f83e00700 1 -- 172.21.15.62:0/337365393 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f6f780902e0 con 0x7f6f8c103040 2024-04-20T21:38:29.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:29.957+0000 7f6f8b400700 1 --2- 172.21.15.62:0/337365393 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6f7006ecc0 0x7f6f70071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:38:29.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:29.957+0000 7f6f8b400700 1 --2- 172.21.15.62:0/337365393 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6f7006ecc0 0x7f6f70071170 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6f7c006fd0 tx=0x7f6f7c008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:38:29.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:29.961+0000 7f6f83e00700 1 -- 172.21.15.62:0/337365393 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f6f780592f0 con 0x7f6f8c103040 2024-04-20T21:38:30.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:29 smithi062 ceph-mon[76322]: pgmap v60: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:38:30.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:29 smithi062 ceph-mon[76322]: from='client.34218 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:38:30.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:29 smithi062 ceph-mon[76322]: from='client.34222 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:38:30.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:29 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/732851503' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:38:30.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:29 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/2673146249' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T21:38:30.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:30.210+0000 7f6f92400700 1 -- 172.21.15.62:0/337365393 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6f8c19d6d0 con 0x7f6f7006ecc0 2024-04-20T21:38:30.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:30.212+0000 7f6f83e00700 1 -- 172.21.15.62:0/337365393 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f6f8c19d6d0 con 0x7f6f7006ecc0 2024-04-20T21:38:30.212 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:38:30.212 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T21:38:30.212 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T21:38:30.212 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-20T21:38:30.212 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T21:38:30.212 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr", 2024-04-20T21:38:30.212 INFO:teuthology.orchestra.run.smithi062.stdout: "crash", 2024-04-20T21:38:30.212 INFO:teuthology.orchestra.run.smithi062.stdout: "mon" 2024-04-20T21:38:30.212 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T21:38:30.212 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "6/23 daemons upgraded", 2024-04-20T21:38:30.212 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading osd daemons", 2024-04-20T21:38:30.212 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T21:38:30.212 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:38:30.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:30.215+0000 7f6f92400700 1 -- 172.21.15.62:0/337365393 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6f7006ecc0 msgr2=0x7f6f70071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:38:30.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:30.215+0000 7f6f92400700 1 --2- 172.21.15.62:0/337365393 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6f7006ecc0 0x7f6f70071170 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6f7c006fd0 tx=0x7f6f7c008040 comp rx=0 tx=0).stop 2024-04-20T21:38:30.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:30.216+0000 7f6f92400700 1 -- 172.21.15.62:0/337365393 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6f8c103040 msgr2=0x7f6f8c1987c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:38:30.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:30.216+0000 7f6f92400700 1 --2- 172.21.15.62:0/337365393 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6f8c103040 0x7f6f8c1987c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6f7800bda0 tx=0x7f6f7800be80 comp rx=0 tx=0).stop 2024-04-20T21:38:30.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:30.216+0000 7f6f92400700 1 -- 172.21.15.62:0/337365393 shutdown_connections 2024-04-20T21:38:30.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:30.216+0000 7f6f92400700 1 --2- 172.21.15.62:0/337365393 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6f7006ecc0 0x7f6f70071170 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:38:30.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:30.216+0000 7f6f92400700 1 --2- 172.21.15.62:0/337365393 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f8c1026a0 0x7f6f8c198280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:38:30.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:30.216+0000 7f6f92400700 1 --2- 172.21.15.62:0/337365393 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6f8c103040 0x7f6f8c1987c0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:38:30.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:30.216+0000 7f6f92400700 1 -- 172.21.15.62:0/337365393 >> 172.21.15.62:0/337365393 conn(0x7f6f8c0fdf40 msgr2=0x7f6f8c0ff840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:38:30.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:30.216+0000 7f6f92400700 1 -- 172.21.15.62:0/337365393 shutdown_connections 2024-04-20T21:38:30.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:30.216+0000 7f6f92400700 1 -- 172.21.15.62:0/337365393 wait complete. 2024-04-20T21:38:30.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:29 smithi189 ceph-mon[57022]: pgmap v60: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:38:30.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:29 smithi189 ceph-mon[57022]: from='client.34218 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:38:30.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:29 smithi189 ceph-mon[57022]: from='client.34222 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:38:30.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:29 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/732851503' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:38:30.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:29 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/2673146249' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T21:38:30.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:30.380+0000 7fceaec00700 1 -- 172.21.15.62:0/1347929214 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcea8106320 msgr2=0x7fcea81066f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:38:30.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:30.380+0000 7fceaec00700 1 --2- 172.21.15.62:0/1347929214 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcea8106320 0x7fcea81066f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fce98009910 tx=0x7fce98009c20 comp rx=0 tx=0).stop 2024-04-20T21:38:30.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:30.380+0000 7fceaec00700 1 -- 172.21.15.62:0/1347929214 shutdown_connections 2024-04-20T21:38:30.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:30.380+0000 7fceaec00700 1 --2- 172.21.15.62:0/1347929214 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcea8068540 0x7fcea80689b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:38:30.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:30.380+0000 7fceaec00700 1 --2- 172.21.15.62:0/1347929214 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcea8106320 0x7fcea81066f0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:38:30.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:30.380+0000 7fceaec00700 1 -- 172.21.15.62:0/1347929214 >> 172.21.15.62:0/1347929214 conn(0x7fcea8075130 msgr2=0x7fcea8075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:38:30.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:30.381+0000 7fceaec00700 1 -- 172.21.15.62:0/1347929214 shutdown_connections 2024-04-20T21:38:30.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:30.381+0000 7fceaec00700 1 -- 172.21.15.62:0/1347929214 wait complete. 2024-04-20T21:38:30.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:30.383+0000 7fceaec00700 1 Processor -- start 2024-04-20T21:38:30.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:30.383+0000 7fceaec00700 1 -- start start 2024-04-20T21:38:30.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:30.383+0000 7fceaec00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcea8068540 0x7fcea8195f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:38:30.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:30.383+0000 7fceaec00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcea8106320 0x7fcea81964b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:38:30.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:30.383+0000 7fceaec00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcea8196b90 con 0x7fcea8068540 2024-04-20T21:38:30.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:30.383+0000 7fceaec00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcea819a920 con 0x7fcea8106320 2024-04-20T21:38:30.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:30.384+0000 7fcea7e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcea8068540 0x7fcea8195f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:38:30.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:30.384+0000 7fcea7e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcea8068540 0x7fcea8195f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44300/0 (socket says 172.21.15.62:44300) 2024-04-20T21:38:30.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:30.384+0000 7fcea7e00700 1 -- 172.21.15.62:0/3671116949 learned_addr learned my addr 172.21.15.62:0/3671116949 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:38:30.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:30.384+0000 7fcea7400700 1 --2- 172.21.15.62:0/3671116949 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcea8106320 0x7fcea81964b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:38:30.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:30.384+0000 7fcea7e00700 1 -- 172.21.15.62:0/3671116949 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcea8106320 msgr2=0x7fcea81964b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:38:30.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:30.384+0000 7fcea7e00700 1 --2- 172.21.15.62:0/3671116949 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcea8106320 0x7fcea81964b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:38:30.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:30.384+0000 7fcea7e00700 1 -- 172.21.15.62:0/3671116949 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce980095f0 con 0x7fcea8068540 2024-04-20T21:38:30.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:30.385+0000 7fcea7e00700 1 --2- 172.21.15.62:0/3671116949 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcea8068540 0x7fcea8195f70 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fce98006010 tx=0x7fce9800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:38:30.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:30.385+0000 7fcea4c00700 1 -- 172.21.15.62:0/3671116949 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce9801d070 con 0x7fcea8068540 2024-04-20T21:38:30.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:30.385+0000 7fceaec00700 1 -- 172.21.15.62:0/3671116949 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcea819aba0 con 0x7fcea8068540 2024-04-20T21:38:30.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:30.385+0000 7fcea4c00700 1 -- 172.21.15.62:0/3671116949 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fce98004fb0 con 0x7fcea8068540 2024-04-20T21:38:30.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:30.385+0000 7fcea4c00700 1 -- 172.21.15.62:0/3671116949 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce98022ea0 con 0x7fcea8068540 2024-04-20T21:38:30.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:30.385+0000 7fceaec00700 1 -- 172.21.15.62:0/3671116949 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcea819b090 con 0x7fcea8068540 2024-04-20T21:38:30.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:30.387+0000 7fceaec00700 1 -- 172.21.15.62:0/3671116949 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcea8108790 con 0x7fcea8068540 2024-04-20T21:38:30.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:30.387+0000 7fcea4c00700 1 -- 172.21.15.62:0/3671116949 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fce98022470 con 0x7fcea8068540 2024-04-20T21:38:30.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:30.388+0000 7fcea4c00700 1 --2- 172.21.15.62:0/3671116949 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fce8c06ecc0 0x7fce8c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:38:30.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:30.388+0000 7fcea4c00700 1 -- 172.21.15.62:0/3671116949 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fce98090310 con 0x7fcea8068540 2024-04-20T21:38:30.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:30.389+0000 7fcea7400700 1 --2- 172.21.15.62:0/3671116949 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fce8c06ecc0 0x7fce8c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:38:30.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:30.389+0000 7fcea7400700 1 --2- 172.21.15.62:0/3671116949 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fce8c06ecc0 0x7fce8c071170 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fce94006fd0 tx=0x7fce94008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:38:30.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:30.393+0000 7fcea4c00700 1 -- 172.21.15.62:0/3671116949 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fce98059200 con 0x7fcea8068540 2024-04-20T21:38:30.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:30.744+0000 7fceaec00700 1 -- 172.21.15.62:0/3671116949 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fcea8197370 con 0x7fcea8068540 2024-04-20T21:38:30.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:30.745+0000 7fcea4c00700 1 -- 172.21.15.62:0/3671116949 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7fce9805c8c0 con 0x7fcea8068540 2024-04-20T21:38:30.745 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-04-20T21:38:30.745 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T21:38:30.745 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T21:38:30.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:30.748+0000 7fceaec00700 1 -- 172.21.15.62:0/3671116949 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fce8c06ecc0 msgr2=0x7fce8c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:38:30.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:30.748+0000 7fceaec00700 1 --2- 172.21.15.62:0/3671116949 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fce8c06ecc0 0x7fce8c071170 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fce94006fd0 tx=0x7fce94008040 comp rx=0 tx=0).stop 2024-04-20T21:38:30.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:30.748+0000 7fceaec00700 1 -- 172.21.15.62:0/3671116949 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcea8068540 msgr2=0x7fcea8195f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:38:30.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:30.748+0000 7fceaec00700 1 --2- 172.21.15.62:0/3671116949 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcea8068540 0x7fcea8195f70 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fce98006010 tx=0x7fce9800be20 comp rx=0 tx=0).stop 2024-04-20T21:38:30.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:30.748+0000 7fceaec00700 1 -- 172.21.15.62:0/3671116949 shutdown_connections 2024-04-20T21:38:30.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:30.749+0000 7fceaec00700 1 --2- 172.21.15.62:0/3671116949 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fce8c06ecc0 0x7fce8c071170 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:38:30.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:30.749+0000 7fceaec00700 1 --2- 172.21.15.62:0/3671116949 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcea8068540 0x7fcea8195f70 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:38:30.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:30.749+0000 7fceaec00700 1 --2- 172.21.15.62:0/3671116949 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcea8106320 0x7fcea81964b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:38:30.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:30.749+0000 7fceaec00700 1 -- 172.21.15.62:0/3671116949 >> 172.21.15.62:0/3671116949 conn(0x7fcea8075130 msgr2=0x7fcea80fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:38:30.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:30.749+0000 7fceaec00700 1 -- 172.21.15.62:0/3671116949 shutdown_connections 2024-04-20T21:38:30.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:38:30.749+0000 7fceaec00700 1 -- 172.21.15.62:0/3671116949 wait complete. 2024-04-20T21:38:31.403 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:31 smithi062 ceph-mon[76322]: from='client.34234 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:38:31.403 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:31 smithi062 ceph-mon[76322]: Upgrade: Updating osd.0 2024-04-20T21:38:31.403 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:31 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:38:31.403 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:31 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-20T21:38:31.403 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:31 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:38:31.403 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:31 smithi062 ceph-mon[76322]: Deploying daemon osd.0 on smithi062 2024-04-20T21:38:31.403 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:31 smithi062 ceph-mon[76322]: pgmap v61: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:38:31.403 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:31 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/3671116949' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T21:38:31.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:31 smithi189 ceph-mon[57022]: from='client.34234 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:38:31.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:31 smithi189 ceph-mon[57022]: Upgrade: Updating osd.0 2024-04-20T21:38:31.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:31 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:38:31.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:31 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-20T21:38:31.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:31 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:38:31.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:31 smithi189 ceph-mon[57022]: Deploying daemon osd.0 on smithi062 2024-04-20T21:38:31.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:31 smithi189 ceph-mon[57022]: pgmap v61: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:38:31.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:31 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/3671116949' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T21:38:33.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:33 smithi189 ceph-mon[57022]: pgmap v62: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:38:33.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:33 smithi062 ceph-mon[76322]: pgmap v62: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:38:34.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:34 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:38:34.772 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:34 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:38:35.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:35 smithi062 ceph-mon[76322]: pgmap v63: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:38:35.535 INFO:journalctl@ceph.osd.0.smithi062.stdout:Apr 20 21:38:35 smithi062 systemd[1]: Stopping Ceph osd.0 for 61ae3628-ff5b-11ee-bc93-c7b262605968... 2024-04-20T21:38:35.713 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:35 smithi189 ceph-mon[57022]: pgmap v63: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:38:36.035 INFO:journalctl@ceph.osd.0.smithi062.stdout:Apr 20 21:38:35 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-0[31088]: 2024-04-20T21:38:35.627+0000 7faf0ba00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-20T21:38:36.035 INFO:journalctl@ceph.osd.0.smithi062.stdout:Apr 20 21:38:35 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-0[31088]: 2024-04-20T21:38:35.627+0000 7faf0ba00700 -1 osd.0 48 *** Got signal Terminated *** 2024-04-20T21:38:36.035 INFO:journalctl@ceph.osd.0.smithi062.stdout:Apr 20 21:38:35 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-0[31088]: 2024-04-20T21:38:35.627+0000 7faf0ba00700 -1 osd.0 48 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-20T21:38:36.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:36 smithi189 ceph-mon[57022]: osd.0 marked itself down and dead 2024-04-20T21:38:36.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:36 smithi062 ceph-mon[76322]: osd.0 marked itself down and dead 2024-04-20T21:38:37.369 INFO:journalctl@ceph.osd.0.smithi062.stdout:Apr 20 21:38:37 smithi062 podman[87689]: 2024-04-20 21:38:37.057385933 +0000 UTC m=+1.546075956 container died 3f315c3afc1f37301b8d6059a2e5b2f964bb68322ee4a0452f065260501fbff8 (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20231212, CEPH_POINT_RELEASE=-18.2.0, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.29.1, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-04-20T21:38:37.712 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:37 smithi062 ceph-mon[76322]: pgmap v64: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:38:37.712 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:37 smithi062 ceph-mon[76322]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-20T21:38:37.712 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:37 smithi062 ceph-mon[76322]: osdmap e49: 6 total, 5 up, 6 in 2024-04-20T21:38:37.713 INFO:journalctl@ceph.osd.0.smithi062.stdout:Apr 20 21:38:37 smithi062 podman[87689]: 2024-04-20 21:38:37.512289431 +0000 UTC m=+2.000979492 container cleanup 3f315c3afc1f37301b8d6059a2e5b2f964bb68322ee4a0452f065260501fbff8 (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-0, RELEASE=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-18.2.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20231212, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True) 2024-04-20T21:38:37.713 INFO:journalctl@ceph.osd.0.smithi062.stdout:Apr 20 21:38:37 smithi062 bash[87689]: ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-0 2024-04-20T21:38:37.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:37 smithi189 ceph-mon[57022]: pgmap v64: 65 pgs: 65 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:38:37.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:37 smithi189 ceph-mon[57022]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-20T21:38:37.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:37 smithi189 ceph-mon[57022]: osdmap e49: 6 total, 5 up, 6 in 2024-04-20T21:38:38.035 INFO:journalctl@ceph.osd.0.smithi062.stdout:Apr 20 21:38:37 smithi062 podman[87709]: 2024-04-20 21:38:37.713323405 +0000 UTC m=+0.646319959 container remove 3f315c3afc1f37301b8d6059a2e5b2f964bb68322ee4a0452f065260501fbff8 (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-0, io.buildah.version=1.29.1, org.label-schema.build-date=20231212, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, CEPH_POINT_RELEASE=-18.2.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, org.label-schema.schema-version=1.0) 2024-04-20T21:38:38.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:38 smithi189 ceph-mon[57022]: osdmap e50: 6 total, 5 up, 6 in 2024-04-20T21:38:38.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:38 smithi062 ceph-mon[76322]: osdmap e50: 6 total, 5 up, 6 in 2024-04-20T21:38:38.785 INFO:journalctl@ceph.osd.0.smithi062.stdout:Apr 20 21:38:38 smithi062 podman[87804]: 2024-04-20 21:38:38.504333239 +0000 UTC m=+0.038957227 image pull 76bc6b91b726a03b8b736eeeb35fc1ffd1469e23c7d60f5b444a217528d8906d quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 2024-04-20T21:38:38.785 INFO:journalctl@ceph.osd.0.smithi062.stdout:Apr 20 21:38:38 smithi062 podman[87804]: 2024-04-20 21:38:38.647761664 +0000 UTC m=+0.182385675 container create 133aae1ef03efd8c57a0bf2e6d66277851df556a786bf3104604857473554499 (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-0-deactivate, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-20T21:38:39.405 INFO:journalctl@ceph.osd.0.smithi062.stdout:Apr 20 21:38:39 smithi062 podman[87804]: 2024-04-20 21:38:39.177814751 +0000 UTC m=+0.712438769 container init 133aae1ef03efd8c57a0bf2e6d66277851df556a786bf3104604857473554499 (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-0-deactivate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-04-20T21:38:39.405 INFO:journalctl@ceph.osd.0.smithi062.stdout:Apr 20 21:38:39 smithi062 podman[87804]: 2024-04-20 21:38:39.189964403 +0000 UTC m=+0.724588416 container start 133aae1ef03efd8c57a0bf2e6d66277851df556a786bf3104604857473554499 (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-0-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-04-20T21:38:39.405 INFO:journalctl@ceph.osd.0.smithi062.stdout:Apr 20 21:38:39 smithi062 podman[87804]: 2024-04-20 21:38:39.275689651 +0000 UTC m=+0.810313706 container attach 133aae1ef03efd8c57a0bf2e6d66277851df556a786bf3104604857473554499 (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-0-deactivate, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-20T21:38:39.663 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:39 smithi062 ceph-mon[76322]: pgmap v67: 65 pgs: 9 stale+active+clean, 56 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-20T21:38:39.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:39 smithi189 ceph-mon[57022]: pgmap v67: 65 pgs: 9 stale+active+clean, 56 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-20T21:38:40.017 INFO:journalctl@ceph.osd.0.smithi062.stdout:Apr 20 21:38:39 smithi062 podman[87804]: 2024-04-20 21:38:39.664716345 +0000 UTC m=+1.199340319 container died 133aae1ef03efd8c57a0bf2e6d66277851df556a786bf3104604857473554499 (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-0-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-04-20T21:38:40.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:40 smithi189 ceph-mon[57022]: Health check failed: Degraded data redundancy: 71/366 objects degraded (19.399%), 20 pgs degraded (PG_DEGRADED) 2024-04-20T21:38:40.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:40 smithi062 ceph-mon[76322]: Health check failed: Degraded data redundancy: 71/366 objects degraded (19.399%), 20 pgs degraded (PG_DEGRADED) 2024-04-20T21:38:40.785 INFO:journalctl@ceph.osd.0.smithi062.stdout:Apr 20 21:38:40 smithi062 podman[87830]: 2024-04-20 21:38:40.404972174 +0000 UTC m=+0.716464888 container remove 133aae1ef03efd8c57a0bf2e6d66277851df556a786bf3104604857473554499 (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-0-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, org.label-schema.schema-version=1.0) 2024-04-20T21:38:40.785 INFO:journalctl@ceph.osd.0.smithi062.stdout:Apr 20 21:38:40 smithi062 systemd[1]: ceph-61ae3628-ff5b-11ee-bc93-c7b262605968@osd.0.service: Deactivated successfully. 2024-04-20T21:38:40.785 INFO:journalctl@ceph.osd.0.smithi062.stdout:Apr 20 21:38:40 smithi062 systemd[1]: Stopped Ceph osd.0 for 61ae3628-ff5b-11ee-bc93-c7b262605968. 2024-04-20T21:38:40.785 INFO:journalctl@ceph.osd.0.smithi062.stdout:Apr 20 21:38:40 smithi062 systemd[1]: ceph-61ae3628-ff5b-11ee-bc93-c7b262605968@osd.0.service: Consumed 1min 39.100s CPU time. 2024-04-20T21:38:41.206 INFO:journalctl@ceph.osd.0.smithi062.stdout:Apr 20 21:38:40 smithi062 systemd[1]: Starting Ceph osd.0 for 61ae3628-ff5b-11ee-bc93-c7b262605968... 2024-04-20T21:38:41.461 INFO:journalctl@ceph.osd.0.smithi062.stdout:Apr 20 21:38:41 smithi062 podman[87930]: 2024-04-20 21:38:41.113571044 +0000 UTC m=+0.037090977 image pull 76bc6b91b726a03b8b736eeeb35fc1ffd1469e23c7d60f5b444a217528d8906d quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 2024-04-20T21:38:41.461 INFO:journalctl@ceph.osd.0.smithi062.stdout:Apr 20 21:38:41 smithi062 podman[87930]: 2024-04-20 21:38:41.279567185 +0000 UTC m=+0.203087149 container create 6bc907eafb1dc542daa5bcd69f6303415855a9b0472a3e9d94f8c5c0e6ac070d (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-0-activate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-04-20T21:38:41.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:41 smithi189 ceph-mon[57022]: pgmap v68: 65 pgs: 20 active+undersized+degraded, 14 active+undersized, 31 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 71/366 objects degraded (19.399%) 2024-04-20T21:38:41.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:41 smithi062 ceph-mon[76322]: pgmap v68: 65 pgs: 20 active+undersized+degraded, 14 active+undersized, 31 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 71/366 objects degraded (19.399%) 2024-04-20T21:38:41.785 INFO:journalctl@ceph.osd.0.smithi062.stdout:Apr 20 21:38:41 smithi062 podman[87930]: 2024-04-20 21:38:41.560992512 +0000 UTC m=+0.484512483 container init 6bc907eafb1dc542daa5bcd69f6303415855a9b0472a3e9d94f8c5c0e6ac070d (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-0-activate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-20T21:38:41.785 INFO:journalctl@ceph.osd.0.smithi062.stdout:Apr 20 21:38:41 smithi062 podman[87930]: 2024-04-20 21:38:41.57221482 +0000 UTC m=+0.495734791 container start 6bc907eafb1dc542daa5bcd69f6303415855a9b0472a3e9d94f8c5c0e6ac070d (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-0-activate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2) 2024-04-20T21:38:41.785 INFO:journalctl@ceph.osd.0.smithi062.stdout:Apr 20 21:38:41 smithi062 podman[87930]: 2024-04-20 21:38:41.6508925 +0000 UTC m=+0.574412494 container attach 6bc907eafb1dc542daa5bcd69f6303415855a9b0472a3e9d94f8c5c0e6ac070d (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-0-activate, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-04-20T21:38:42.786 INFO:journalctl@ceph.osd.0.smithi062.stdout:Apr 20 21:38:42 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-0-activate[87944]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-20T21:38:42.786 INFO:journalctl@ceph.osd.0.smithi062.stdout:Apr 20 21:38:42 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-0-activate[87944]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-20T21:38:42.786 INFO:journalctl@ceph.osd.0.smithi062.stdout:Apr 20 21:38:42 smithi062 bash[87930]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-20T21:38:42.786 INFO:journalctl@ceph.osd.0.smithi062.stdout:Apr 20 21:38:42 smithi062 bash[87930]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-20T21:38:42.787 INFO:journalctl@ceph.osd.0.smithi062.stdout:Apr 20 21:38:42 smithi062 bash[87930]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-20T21:38:42.787 INFO:journalctl@ceph.osd.0.smithi062.stdout:Apr 20 21:38:42 smithi062 bash[87930]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-04-20T21:38:42.787 INFO:journalctl@ceph.osd.0.smithi062.stdout:Apr 20 21:38:42 smithi062 bash[87930]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-04-20T21:38:42.787 INFO:journalctl@ceph.osd.0.smithi062.stdout:Apr 20 21:38:42 smithi062 bash[87930]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-04-20T21:38:42.787 INFO:journalctl@ceph.osd.0.smithi062.stdout:Apr 20 21:38:42 smithi062 bash[87930]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-04-20T21:38:42.787 INFO:journalctl@ceph.osd.0.smithi062.stdout:Apr 20 21:38:42 smithi062 bash[87930]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-20T21:38:42.787 INFO:journalctl@ceph.osd.0.smithi062.stdout:Apr 20 21:38:42 smithi062 bash[87930]: --> ceph-volume raw activate successful for osd ID: 0 2024-04-20T21:38:42.787 INFO:journalctl@ceph.osd.0.smithi062.stdout:Apr 20 21:38:42 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-0-activate[87944]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-20T21:38:42.787 INFO:journalctl@ceph.osd.0.smithi062.stdout:Apr 20 21:38:42 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-0-activate[87944]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-04-20T21:38:42.787 INFO:journalctl@ceph.osd.0.smithi062.stdout:Apr 20 21:38:42 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-0-activate[87944]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-04-20T21:38:42.787 INFO:journalctl@ceph.osd.0.smithi062.stdout:Apr 20 21:38:42 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-0-activate[87944]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-04-20T21:38:42.787 INFO:journalctl@ceph.osd.0.smithi062.stdout:Apr 20 21:38:42 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-0-activate[87944]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-04-20T21:38:42.787 INFO:journalctl@ceph.osd.0.smithi062.stdout:Apr 20 21:38:42 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-0-activate[87944]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-20T21:38:42.787 INFO:journalctl@ceph.osd.0.smithi062.stdout:Apr 20 21:38:42 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-0-activate[87944]: --> ceph-volume raw activate successful for osd ID: 0 2024-04-20T21:38:42.787 INFO:journalctl@ceph.osd.0.smithi062.stdout:Apr 20 21:38:42 smithi062 podman[87930]: 2024-04-20 21:38:42.591689395 +0000 UTC m=+1.515209422 container died 6bc907eafb1dc542daa5bcd69f6303415855a9b0472a3e9d94f8c5c0e6ac070d (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-0-activate, GIT_CLEAN=True, org.label-schema.build-date=20240416, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-20T21:38:43.715 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:43 smithi062 ceph-mon[76322]: pgmap v69: 65 pgs: 20 active+undersized+degraded, 14 active+undersized, 31 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 71/366 objects degraded (19.399%) 2024-04-20T21:38:43.715 INFO:journalctl@ceph.osd.0.smithi062.stdout:Apr 20 21:38:43 smithi062 podman[88074]: 2024-04-20 21:38:43.338914807 +0000 UTC m=+0.726275810 container remove 6bc907eafb1dc542daa5bcd69f6303415855a9b0472a3e9d94f8c5c0e6ac070d (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-0-activate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-04-20T21:38:43.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:43 smithi189 ceph-mon[57022]: pgmap v69: 65 pgs: 20 active+undersized+degraded, 14 active+undersized, 31 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 71/366 objects degraded (19.399%) 2024-04-20T21:38:43.984 INFO:journalctl@ceph.osd.0.smithi062.stdout:Apr 20 21:38:43 smithi062 podman[88132]: 2024-04-20 21:38:43.622579667 +0000 UTC m=+0.033296614 image pull 76bc6b91b726a03b8b736eeeb35fc1ffd1469e23c7d60f5b444a217528d8906d quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 2024-04-20T21:38:43.984 INFO:journalctl@ceph.osd.0.smithi062.stdout:Apr 20 21:38:43 smithi062 podman[88132]: 2024-04-20 21:38:43.731541573 +0000 UTC m=+0.142258544 container create ee2bb901f26e6a4de27453ff07542d1965f02e9ac6d2b6d88d4004da7d789b60 (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-20T21:38:44.535 INFO:journalctl@ceph.osd.0.smithi062.stdout:Apr 20 21:38:44 smithi062 podman[88132]: 2024-04-20 21:38:44.085530676 +0000 UTC m=+0.496247690 container init ee2bb901f26e6a4de27453ff07542d1965f02e9ac6d2b6d88d4004da7d789b60 (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-0, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-20T21:38:44.535 INFO:journalctl@ceph.osd.0.smithi062.stdout:Apr 20 21:38:44 smithi062 podman[88132]: 2024-04-20 21:38:44.097865074 +0000 UTC m=+0.508582032 container start ee2bb901f26e6a4de27453ff07542d1965f02e9ac6d2b6d88d4004da7d789b60 (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-04-20T21:38:44.535 INFO:journalctl@ceph.osd.0.smithi062.stdout:Apr 20 21:38:44 smithi062 ceph-osd[88150]: -- 172.21.15.62:0/1507882206 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x5650f30ddba0 con 0x5650f30af000 2024-04-20T21:38:44.535 INFO:journalctl@ceph.osd.0.smithi062.stdout:Apr 20 21:38:44 smithi062 ceph-osd[88150]: -- 172.21.15.62:0/1507882206 <== mon.0 v2:172.21.15.62:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5650f30e5680 con 0x5650f30af000 2024-04-20T21:38:44.535 INFO:journalctl@ceph.osd.0.smithi062.stdout:Apr 20 21:38:44 smithi062 bash[88132]: ee2bb901f26e6a4de27453ff07542d1965f02e9ac6d2b6d88d4004da7d789b60 2024-04-20T21:38:44.535 INFO:journalctl@ceph.osd.0.smithi062.stdout:Apr 20 21:38:44 smithi062 systemd[1]: Started Ceph osd.0 for 61ae3628-ff5b-11ee-bc93-c7b262605968. 2024-04-20T21:38:45.034 INFO:journalctl@ceph.osd.0.smithi062.stdout:Apr 20 21:38:44 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-0[88146]: 2024-04-20T21:38:44.690+0000 7fe7d84687c0 -1 Falling back to public interface 2024-04-20T21:38:45.670 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:45 smithi062 ceph-mon[76322]: pgmap v70: 65 pgs: 20 active+undersized+degraded, 14 active+undersized, 31 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 71/366 objects degraded (19.399%) 2024-04-20T21:38:45.670 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:45 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:38:45.670 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:45 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:38:45.670 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:45 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:38:45.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:45 smithi189 ceph-mon[57022]: pgmap v70: 65 pgs: 20 active+undersized+degraded, 14 active+undersized, 31 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 71/366 objects degraded (19.399%) 2024-04-20T21:38:45.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:45 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:38:45.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:45 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:38:45.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:45 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:38:47.675 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:47 smithi062 ceph-mon[76322]: pgmap v71: 65 pgs: 20 active+undersized+degraded, 14 active+undersized, 31 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 71/366 objects degraded (19.399%) 2024-04-20T21:38:47.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:47 smithi189 ceph-mon[57022]: pgmap v71: 65 pgs: 20 active+undersized+degraded, 14 active+undersized, 31 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 71/366 objects degraded (19.399%) 2024-04-20T21:38:49.705 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:49 smithi062 ceph-mon[76322]: pgmap v72: 65 pgs: 20 active+undersized+degraded, 14 active+undersized, 31 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 71/366 objects degraded (19.399%) 2024-04-20T21:38:49.706 INFO:journalctl@ceph.osd.0.smithi062.stdout:Apr 20 21:38:49 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-0[88146]: 2024-04-20T21:38:49.320+0000 7fe7d84687c0 -1 osd.0 0 read_superblock omap replica is missing. 2024-04-20T21:38:49.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:49 smithi189 ceph-mon[57022]: pgmap v72: 65 pgs: 20 active+undersized+degraded, 14 active+undersized, 31 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 71/366 objects degraded (19.399%) 2024-04-20T21:38:50.035 INFO:journalctl@ceph.osd.0.smithi062.stdout:Apr 20 21:38:49 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-0[88146]: 2024-04-20T21:38:49.706+0000 7fe7d84687c0 -1 osd.0 48 log_to_monitors true 2024-04-20T21:38:50.649 INFO:journalctl@ceph.osd.0.smithi062.stdout:Apr 20 21:38:50 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-0[88146]: 2024-04-20T21:38:50.402+0000 7fe7c7800700 -1 osd.0 48 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-20T21:38:50.650 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:50 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:38:50.650 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:50 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:38:50.650 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:50 smithi062 ceph-mon[76322]: from='osd.0 [v2:172.21.15.62:6802/1209367302,v1:172.21.15.62:6803/1209367302]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-20T21:38:50.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:50 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:38:50.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:50 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:38:50.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:50 smithi189 ceph-mon[57022]: from='osd.0 [v2:172.21.15.62:6802/1209367302,v1:172.21.15.62:6803/1209367302]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-20T21:38:51.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:51 smithi189 ceph-mon[57022]: pgmap v73: 65 pgs: 20 active+undersized+degraded, 14 active+undersized, 31 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 1 op/s; 71/366 objects degraded (19.399%) 2024-04-20T21:38:51.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:51 smithi189 ceph-mon[57022]: from='osd.0 [v2:172.21.15.62:6802/1209367302,v1:172.21.15.62:6803/1209367302]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-20T21:38:51.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:51 smithi189 ceph-mon[57022]: osdmap e51: 6 total, 5 up, 6 in 2024-04-20T21:38:51.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:51 smithi189 ceph-mon[57022]: from='osd.0 [v2:172.21.15.62:6802/1209367302,v1:172.21.15.62:6803/1209367302]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-04-20T21:38:51.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:51 smithi062 ceph-mon[76322]: pgmap v73: 65 pgs: 20 active+undersized+degraded, 14 active+undersized, 31 active+clean; 315 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 1 op/s; 71/366 objects degraded (19.399%) 2024-04-20T21:38:51.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:51 smithi062 ceph-mon[76322]: from='osd.0 [v2:172.21.15.62:6802/1209367302,v1:172.21.15.62:6803/1209367302]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-20T21:38:51.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:51 smithi062 ceph-mon[76322]: osdmap e51: 6 total, 5 up, 6 in 2024-04-20T21:38:51.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:51 smithi062 ceph-mon[76322]: from='osd.0 [v2:172.21.15.62:6802/1209367302,v1:172.21.15.62:6803/1209367302]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-04-20T21:38:52.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:52 smithi189 ceph-mon[57022]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-20T21:38:52.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:52 smithi189 ceph-mon[57022]: osd.0 [v2:172.21.15.62:6802/1209367302,v1:172.21.15.62:6803/1209367302] boot 2024-04-20T21:38:52.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:52 smithi189 ceph-mon[57022]: osdmap e52: 6 total, 6 up, 6 in 2024-04-20T21:38:52.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:52 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-20T21:38:52.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:52 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:38:52.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:52 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:38:52.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:52 smithi189 ceph-mon[57022]: osdmap e53: 6 total, 6 up, 6 in 2024-04-20T21:38:52.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:52 smithi062 ceph-mon[76322]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-20T21:38:52.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:52 smithi062 ceph-mon[76322]: osd.0 [v2:172.21.15.62:6802/1209367302,v1:172.21.15.62:6803/1209367302] boot 2024-04-20T21:38:52.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:52 smithi062 ceph-mon[76322]: osdmap e52: 6 total, 6 up, 6 in 2024-04-20T21:38:52.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:52 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-20T21:38:52.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:52 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:38:52.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:52 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:38:52.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:52 smithi062 ceph-mon[76322]: osdmap e53: 6 total, 6 up, 6 in 2024-04-20T21:38:53.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:53 smithi189 ceph-mon[57022]: pgmap v76: 65 pgs: 20 active+undersized+degraded, 14 active+undersized, 31 active+clean; 315 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s; 71/366 objects degraded (19.399%) 2024-04-20T21:38:54.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:53 smithi062 ceph-mon[76322]: pgmap v76: 65 pgs: 20 active+undersized+degraded, 14 active+undersized, 31 active+clean; 315 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s; 71/366 objects degraded (19.399%) 2024-04-20T21:38:55.001 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:54 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:38:55.001 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:54 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:38:55.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:54 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:38:55.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:54 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:38:56.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:55 smithi062 ceph-mon[76322]: pgmap v78: 65 pgs: 20 active+undersized+degraded, 14 active+undersized, 31 active+clean; 315 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s; 71/366 objects degraded (19.399%) 2024-04-20T21:38:56.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:55 smithi189 ceph-mon[57022]: pgmap v78: 65 pgs: 20 active+undersized+degraded, 14 active+undersized, 31 active+clean; 315 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s; 71/366 objects degraded (19.399%) 2024-04-20T21:38:57.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:56 smithi189 ceph-mon[57022]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 71/366 objects degraded (19.399%), 20 pgs degraded) 2024-04-20T21:38:57.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:56 smithi062 ceph-mon[76322]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 71/366 objects degraded (19.399%), 20 pgs degraded) 2024-04-20T21:38:58.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:57 smithi062 ceph-mon[76322]: pgmap v79: 65 pgs: 65 active+clean; 315 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s rd, 2 op/s 2024-04-20T21:38:58.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:57 smithi189 ceph-mon[57022]: pgmap v79: 65 pgs: 65 active+clean; 315 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s rd, 2 op/s 2024-04-20T21:39:00.142 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:59 smithi062 ceph-mon[76322]: pgmap v80: 65 pgs: 65 active+clean; 315 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-20T21:39:00.142 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:59 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:39:00.142 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:59 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:39:00.143 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:59 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:39:00.143 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:59 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T21:39:00.143 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:38:59 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:39:00.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:59 smithi189 ceph-mon[57022]: pgmap v80: 65 pgs: 65 active+clean; 315 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-20T21:39:00.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:59 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:39:00.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:59 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:39:00.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:59 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:39:00.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:59 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T21:39:00.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:38:59 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:39:00.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:00.929+0000 7f6f7f000700 1 -- 172.21.15.62:0/2140601854 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6f78100e80 msgr2=0x7f6f78104e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:39:00.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:00.929+0000 7f6f7f000700 1 --2- 172.21.15.62:0/2140601854 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6f78100e80 0x7f6f78104e50 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f6f64009910 tx=0x7f6f64009c20 comp rx=0 tx=0).stop 2024-04-20T21:39:00.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:00.929+0000 7f6f7f000700 1 -- 172.21.15.62:0/2140601854 shutdown_connections 2024-04-20T21:39:00.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:00.929+0000 7f6f7f000700 1 --2- 172.21.15.62:0/2140601854 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6f78100e80 0x7f6f78104e50 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:39:00.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:00.929+0000 7f6f7f000700 1 --2- 172.21.15.62:0/2140601854 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f781004e0 0x7f6f781008b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:39:00.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:00.929+0000 7f6f7f000700 1 -- 172.21.15.62:0/2140601854 >> 172.21.15.62:0/2140601854 conn(0x7f6f780fbda0 msgr2=0x7f6f780fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:39:00.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:00.929+0000 7f6f7f000700 1 -- 172.21.15.62:0/2140601854 shutdown_connections 2024-04-20T21:39:00.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:00.929+0000 7f6f7f000700 1 -- 172.21.15.62:0/2140601854 wait complete. 2024-04-20T21:39:00.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:00.930+0000 7f6f7f000700 1 Processor -- start 2024-04-20T21:39:00.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:00.930+0000 7f6f7f000700 1 -- start start 2024-04-20T21:39:00.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:00.931+0000 7f6f7f000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f781004e0 0x7f6f78198130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:39:00.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:00.931+0000 7f6f7f000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6f78100e80 0x7f6f78198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:39:00.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:00.931+0000 7f6f7f000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f78198d50 con 0x7f6f78100e80 2024-04-20T21:39:00.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:00.931+0000 7f6f7f000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f7819cae0 con 0x7f6f781004e0 2024-04-20T21:39:00.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:00.931+0000 7f6f77400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6f78100e80 0x7f6f78198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:39:00.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:00.931+0000 7f6f77400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6f78100e80 0x7f6f78198670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60254/0 (socket says 172.21.15.62:60254) 2024-04-20T21:39:00.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:00.931+0000 7f6f77400700 1 -- 172.21.15.62:0/913919516 learned_addr learned my addr 172.21.15.62:0/913919516 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:39:00.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:00.932+0000 7f6f77e00700 1 --2- 172.21.15.62:0/913919516 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f781004e0 0x7f6f78198130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:39:00.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:00.932+0000 7f6f77400700 1 -- 172.21.15.62:0/913919516 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f781004e0 msgr2=0x7f6f78198130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:39:00.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:00.932+0000 7f6f77400700 1 --2- 172.21.15.62:0/913919516 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f781004e0 0x7f6f78198130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:39:00.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:00.932+0000 7f6f77400700 1 -- 172.21.15.62:0/913919516 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f640095f0 con 0x7f6f78100e80 2024-04-20T21:39:00.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:00.932+0000 7f6f77400700 1 --2- 172.21.15.62:0/913919516 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6f78100e80 0x7f6f78198670 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f6f6400bda0 tx=0x7f6f6400be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:39:00.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:00.932+0000 7f6f74c00700 1 -- 172.21.15.62:0/913919516 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6f6401d070 con 0x7f6f78100e80 2024-04-20T21:39:00.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:00.933+0000 7f6f74c00700 1 -- 172.21.15.62:0/913919516 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6f6400f460 con 0x7f6f78100e80 2024-04-20T21:39:00.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:00.933+0000 7f6f7f000700 1 -- 172.21.15.62:0/913919516 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f7819cd60 con 0x7f6f78100e80 2024-04-20T21:39:00.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:00.933+0000 7f6f74c00700 1 -- 172.21.15.62:0/913919516 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6f64017610 con 0x7f6f78100e80 2024-04-20T21:39:00.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:00.933+0000 7f6f7f000700 1 -- 172.21.15.62:0/913919516 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f7819d250 con 0x7f6f78100e80 2024-04-20T21:39:00.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:00.934+0000 7f6f7f000700 1 -- 172.21.15.62:0/913919516 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f781045d0 con 0x7f6f78100e80 2024-04-20T21:39:00.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:00.935+0000 7f6f74c00700 1 -- 172.21.15.62:0/913919516 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f6f6400fa40 con 0x7f6f78100e80 2024-04-20T21:39:00.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:00.937+0000 7f6f74c00700 1 --2- 172.21.15.62:0/913919516 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6f5c06ed10 0x7f6f5c0711c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:39:00.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:00.937+0000 7f6f74c00700 1 -- 172.21.15.62:0/913919516 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f6f640902e0 con 0x7f6f78100e80 2024-04-20T21:39:00.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:00.937+0000 7f6f77e00700 1 --2- 172.21.15.62:0/913919516 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6f5c06ed10 0x7f6f5c0711c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:39:00.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:00.938+0000 7f6f77e00700 1 --2- 172.21.15.62:0/913919516 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6f5c06ed10 0x7f6f5c0711c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f6f68005950 tx=0x7f6f680058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:39:00.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:00.942+0000 7f6f74c00700 1 -- 172.21.15.62:0/913919516 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f6f640590a0 con 0x7f6f78100e80 2024-04-20T21:39:01.189 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:00 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:39:01.189 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:00 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:39:01.189 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:00 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:39:01.189 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:00 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:39:01.189 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:00 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-20T21:39:01.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:01.207+0000 7f6f7f000700 1 -- 172.21.15.62:0/913919516 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6f781995a0 con 0x7f6f5c06ed10 2024-04-20T21:39:01.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:01.209+0000 7f6f74c00700 1 -- 172.21.15.62:0/913919516 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f6f781995a0 con 0x7f6f5c06ed10 2024-04-20T21:39:01.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:01.212+0000 7f6f7f000700 1 -- 172.21.15.62:0/913919516 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6f5c06ed10 msgr2=0x7f6f5c0711c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:39:01.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:01.212+0000 7f6f7f000700 1 --2- 172.21.15.62:0/913919516 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6f5c06ed10 0x7f6f5c0711c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f6f68005950 tx=0x7f6f680058e0 comp rx=0 tx=0).stop 2024-04-20T21:39:01.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:01.212+0000 7f6f7f000700 1 -- 172.21.15.62:0/913919516 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6f78100e80 msgr2=0x7f6f78198670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:39:01.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:01.212+0000 7f6f7f000700 1 --2- 172.21.15.62:0/913919516 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6f78100e80 0x7f6f78198670 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f6f6400bda0 tx=0x7f6f6400be80 comp rx=0 tx=0).stop 2024-04-20T21:39:01.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:01.212+0000 7f6f7f000700 1 -- 172.21.15.62:0/913919516 shutdown_connections 2024-04-20T21:39:01.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:01.212+0000 7f6f7f000700 1 --2- 172.21.15.62:0/913919516 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6f5c06ed10 0x7f6f5c0711c0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:39:01.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:01.212+0000 7f6f7f000700 1 --2- 172.21.15.62:0/913919516 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f781004e0 0x7f6f78198130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:39:01.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:01.212+0000 7f6f7f000700 1 --2- 172.21.15.62:0/913919516 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6f78100e80 0x7f6f78198670 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:39:01.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:01.212+0000 7f6f7f000700 1 -- 172.21.15.62:0/913919516 >> 172.21.15.62:0/913919516 conn(0x7f6f780fbda0 msgr2=0x7f6f780fe160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:39:01.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:01.212+0000 7f6f7f000700 1 -- 172.21.15.62:0/913919516 shutdown_connections 2024-04-20T21:39:01.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:01.212+0000 7f6f7f000700 1 -- 172.21.15.62:0/913919516 wait complete. 2024-04-20T21:39:01.226 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T21:39:01.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:00 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:39:01.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:00 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:39:01.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:00 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:39:01.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:00 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:39:01.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:00 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-20T21:39:01.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:01.371+0000 7f446ba00700 1 -- 172.21.15.62:0/2003890122 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4464108470 msgr2=0x7f4464108840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:39:01.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:01.371+0000 7f446ba00700 1 --2- 172.21.15.62:0/2003890122 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4464108470 0x7f4464108840 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f4458009910 tx=0x7f4458009c20 comp rx=0 tx=0).stop 2024-04-20T21:39:01.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:01.372+0000 7f446ba00700 1 -- 172.21.15.62:0/2003890122 shutdown_connections 2024-04-20T21:39:01.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:01.372+0000 7f446ba00700 1 --2- 172.21.15.62:0/2003890122 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4464102470 0x7f44641028e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:39:01.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:01.372+0000 7f446ba00700 1 --2- 172.21.15.62:0/2003890122 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4464108470 0x7f4464108840 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:39:01.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:01.372+0000 7f446ba00700 1 -- 172.21.15.62:0/2003890122 >> 172.21.15.62:0/2003890122 conn(0x7f44640fdf90 msgr2=0x7f44641003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:39:01.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:01.372+0000 7f446ba00700 1 -- 172.21.15.62:0/2003890122 shutdown_connections 2024-04-20T21:39:01.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:01.372+0000 7f446ba00700 1 -- 172.21.15.62:0/2003890122 wait complete. 2024-04-20T21:39:01.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:01.373+0000 7f446ba00700 1 Processor -- start 2024-04-20T21:39:01.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:01.373+0000 7f446ba00700 1 -- start start 2024-04-20T21:39:01.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:01.373+0000 7f446ba00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4464102470 0x7f44641981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:39:01.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:01.373+0000 7f446ba00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4464108470 0x7f4464198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:39:01.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:01.373+0000 7f446ba00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4464198e10 con 0x7f4464102470 2024-04-20T21:39:01.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:01.373+0000 7f446ba00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f446419cba0 con 0x7f4464108470 2024-04-20T21:39:01.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:01.374+0000 7f4469000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4464102470 0x7f44641981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:39:01.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:01.374+0000 7f4469000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4464102470 0x7f44641981f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:58284/0 (socket says 172.21.15.62:58284) 2024-04-20T21:39:01.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:01.374+0000 7f4469000700 1 -- 172.21.15.62:0/1548767839 learned_addr learned my addr 172.21.15.62:0/1548767839 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:39:01.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:01.374+0000 7f4463e00700 1 --2- 172.21.15.62:0/1548767839 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4464108470 0x7f4464198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:39:01.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:01.374+0000 7f4469000700 1 -- 172.21.15.62:0/1548767839 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4464108470 msgr2=0x7f4464198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:39:01.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:01.374+0000 7f4469000700 1 --2- 172.21.15.62:0/1548767839 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4464108470 0x7f4464198730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:39:01.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:01.374+0000 7f4469000700 1 -- 172.21.15.62:0/1548767839 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44580095f0 con 0x7f4464102470 2024-04-20T21:39:01.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:01.375+0000 7f4469000700 1 --2- 172.21.15.62:0/1548767839 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4464102470 0x7f44641981f0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f445800b5c0 tx=0x7f445800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:39:01.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:01.375+0000 7f4461600700 1 -- 172.21.15.62:0/1548767839 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f445801d070 con 0x7f4464102470 2024-04-20T21:39:01.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:01.375+0000 7f4461600700 1 -- 172.21.15.62:0/1548767839 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4458005e00 con 0x7f4464102470 2024-04-20T21:39:01.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:01.375+0000 7f4461600700 1 -- 172.21.15.62:0/1548767839 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4458017400 con 0x7f4464102470 2024-04-20T21:39:01.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:01.375+0000 7f446ba00700 1 -- 172.21.15.62:0/1548767839 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f446419ce20 con 0x7f4464102470 2024-04-20T21:39:01.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:01.375+0000 7f446ba00700 1 -- 172.21.15.62:0/1548767839 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f446419d310 con 0x7f4464102470 2024-04-20T21:39:01.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:01.376+0000 7f446ba00700 1 -- 172.21.15.62:0/1548767839 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f446410a9e0 con 0x7f4464102470 2024-04-20T21:39:01.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:01.377+0000 7f4461600700 1 -- 172.21.15.62:0/1548767839 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f445800f460 con 0x7f4464102470 2024-04-20T21:39:01.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:01.379+0000 7f4461600700 1 --2- 172.21.15.62:0/1548767839 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f444c06ecc0 0x7f444c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:39:01.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:01.379+0000 7f4461600700 1 -- 172.21.15.62:0/1548767839 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f44580902c0 con 0x7f4464102470 2024-04-20T21:39:01.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:01.379+0000 7f4463e00700 1 --2- 172.21.15.62:0/1548767839 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f444c06ecc0 0x7f444c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:39:01.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:01.380+0000 7f4463e00700 1 --2- 172.21.15.62:0/1548767839 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f444c06ecc0 0x7f444c071170 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4450005950 tx=0x7f44500058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:39:01.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:01.384+0000 7f4461600700 1 -- 172.21.15.62:0/1548767839 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f4458059100 con 0x7f4464102470 2024-04-20T21:39:01.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:01.645+0000 7f446ba00700 1 -- 172.21.15.62:0/1548767839 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4464199660 con 0x7f444c06ecc0 2024-04-20T21:39:01.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:01.648+0000 7f4461600700 1 -- 172.21.15.62:0/1548767839 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f4464199660 con 0x7f444c06ecc0 2024-04-20T21:39:01.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:01.651+0000 7f446ba00700 1 -- 172.21.15.62:0/1548767839 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f444c06ecc0 msgr2=0x7f444c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:39:01.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:01.651+0000 7f446ba00700 1 --2- 172.21.15.62:0/1548767839 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f444c06ecc0 0x7f444c071170 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4450005950 tx=0x7f44500058e0 comp rx=0 tx=0).stop 2024-04-20T21:39:01.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:01.651+0000 7f446ba00700 1 -- 172.21.15.62:0/1548767839 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4464102470 msgr2=0x7f44641981f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:39:01.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:01.651+0000 7f446ba00700 1 --2- 172.21.15.62:0/1548767839 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4464102470 0x7f44641981f0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f445800b5c0 tx=0x7f445800be20 comp rx=0 tx=0).stop 2024-04-20T21:39:01.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:01.651+0000 7f446ba00700 1 -- 172.21.15.62:0/1548767839 shutdown_connections 2024-04-20T21:39:01.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:01.651+0000 7f446ba00700 1 --2- 172.21.15.62:0/1548767839 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f444c06ecc0 0x7f444c071170 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:39:01.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:01.651+0000 7f446ba00700 1 --2- 172.21.15.62:0/1548767839 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4464102470 0x7f44641981f0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:39:01.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:01.651+0000 7f446ba00700 1 --2- 172.21.15.62:0/1548767839 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4464108470 0x7f4464198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:39:01.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:01.651+0000 7f446ba00700 1 -- 172.21.15.62:0/1548767839 >> 172.21.15.62:0/1548767839 conn(0x7f44640fdf90 msgr2=0x7f44641002c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:39:01.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:01.651+0000 7f446ba00700 1 -- 172.21.15.62:0/1548767839 shutdown_connections 2024-04-20T21:39:01.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:01.651+0000 7f446ba00700 1 -- 172.21.15.62:0/1548767839 wait complete. 2024-04-20T21:39:01.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:01.810+0000 7f4560c00700 1 -- 172.21.15.62:0/3826787565 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f455c0687a0 msgr2=0x7f455c068b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:39:01.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:01.810+0000 7f4560c00700 1 --2- 172.21.15.62:0/3826787565 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f455c0687a0 0x7f455c068b70 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f454c009910 tx=0x7f454c009c20 comp rx=0 tx=0).stop 2024-04-20T21:39:01.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:01.810+0000 7f4560c00700 1 -- 172.21.15.62:0/3826787565 shutdown_connections 2024-04-20T21:39:01.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:01.810+0000 7f4560c00700 1 --2- 172.21.15.62:0/3826787565 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f455c069140 0x7f455c104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:39:01.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:01.810+0000 7f4560c00700 1 --2- 172.21.15.62:0/3826787565 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f455c0687a0 0x7f455c068b70 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:39:01.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:01.810+0000 7f4560c00700 1 -- 172.21.15.62:0/3826787565 >> 172.21.15.62:0/3826787565 conn(0x7f455c075130 msgr2=0x7f455c075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:39:01.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:01.810+0000 7f4560c00700 1 -- 172.21.15.62:0/3826787565 shutdown_connections 2024-04-20T21:39:01.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:01.810+0000 7f4560c00700 1 -- 172.21.15.62:0/3826787565 wait complete. 2024-04-20T21:39:01.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:01.811+0000 7f4560c00700 1 Processor -- start 2024-04-20T21:39:01.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:01.811+0000 7f4560c00700 1 -- start start 2024-04-20T21:39:01.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:01.811+0000 7f4560c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f455c0687a0 0x7f455c195ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:39:01.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:01.811+0000 7f4560c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f455c069140 0x7f455c196420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:39:01.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:01.811+0000 7f4560c00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f455c196b00 con 0x7f455c069140 2024-04-20T21:39:01.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:01.811+0000 7f4560c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f455c19a890 con 0x7f455c0687a0 2024-04-20T21:39:01.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:01.812+0000 7f4559400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f455c069140 0x7f455c196420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:39:01.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:01.812+0000 7f4559400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f455c069140 0x7f455c196420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:58294/0 (socket says 172.21.15.62:58294) 2024-04-20T21:39:01.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:01.812+0000 7f4559400700 1 -- 172.21.15.62:0/2150383559 learned_addr learned my addr 172.21.15.62:0/2150383559 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:39:01.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:01.812+0000 7f4559e00700 1 --2- 172.21.15.62:0/2150383559 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f455c0687a0 0x7f455c195ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:39:01.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:01.812+0000 7f4559400700 1 -- 172.21.15.62:0/2150383559 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f455c0687a0 msgr2=0x7f455c195ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:39:01.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:01.812+0000 7f4559400700 1 --2- 172.21.15.62:0/2150383559 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f455c0687a0 0x7f455c195ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:39:01.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:01.812+0000 7f4559400700 1 -- 172.21.15.62:0/2150383559 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f454c0095f0 con 0x7f455c069140 2024-04-20T21:39:01.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:01.813+0000 7f4559400700 1 --2- 172.21.15.62:0/2150383559 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f455c069140 0x7f455c196420 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f454800c8f0 tx=0x7f454800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:39:01.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:01.813+0000 7f4552a00700 1 -- 172.21.15.62:0/2150383559 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4548007a50 con 0x7f455c069140 2024-04-20T21:39:01.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:01.813+0000 7f4552a00700 1 -- 172.21.15.62:0/2150383559 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4548007bb0 con 0x7f455c069140 2024-04-20T21:39:01.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:01.813+0000 7f4552a00700 1 -- 172.21.15.62:0/2150383559 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4548018630 con 0x7f455c069140 2024-04-20T21:39:01.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:01.813+0000 7f4560c00700 1 -- 172.21.15.62:0/2150383559 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f455c19ab70 con 0x7f455c069140 2024-04-20T21:39:01.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:01.813+0000 7f4560c00700 1 -- 172.21.15.62:0/2150383559 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f455c19b0c0 con 0x7f455c069140 2024-04-20T21:39:01.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:01.815+0000 7f4560c00700 1 -- 172.21.15.62:0/2150383559 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f455c108770 con 0x7f455c069140 2024-04-20T21:39:01.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:01.815+0000 7f4552a00700 1 -- 172.21.15.62:0/2150383559 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f4548007d20 con 0x7f455c069140 2024-04-20T21:39:01.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:01.816+0000 7f4552a00700 1 --2- 172.21.15.62:0/2150383559 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f454006ecc0 0x7f4540071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:39:01.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:01.816+0000 7f4552a00700 1 -- 172.21.15.62:0/2150383559 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f454808ecb0 con 0x7f455c069140 2024-04-20T21:39:01.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:01.817+0000 7f4559e00700 1 --2- 172.21.15.62:0/2150383559 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f454006ecc0 0x7f4540071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:39:01.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:01.817+0000 7f4559e00700 1 --2- 172.21.15.62:0/2150383559 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f454006ecc0 0x7f4540071170 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f454c006010 tx=0x7f454c01a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:39:01.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:01.821+0000 7f4552a00700 1 -- 172.21.15.62:0/2150383559 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f4548057b90 con 0x7f455c069140 2024-04-20T21:39:02.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:02.066+0000 7f4560c00700 1 -- 172.21.15.62:0/2150383559 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f455c19b390 con 0x7f454006ecc0 2024-04-20T21:39:02.066 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:01 smithi062 ceph-mon[76322]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-20T21:39:02.066 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:01 smithi062 ceph-mon[76322]: Upgrade: osd.1 is safe to restart 2024-04-20T21:39:02.066 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:01 smithi062 ceph-mon[76322]: pgmap v81: 65 pgs: 65 active+clean; 315 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-20T21:39:02.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:02.079+0000 7f4552a00700 1 -- 172.21.15.62:0/2150383559 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f455c19b390 con 0x7f454006ecc0 2024-04-20T21:39:02.079 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T21:39:02.079 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (4m) 10s ago 18m 14.1M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T21:39:02.079 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (19m) 10s ago 19m 8514k - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T21:39:02.079 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (17m) 45s ago 17m 8288k - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T21:39:02.079 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (58s) 10s ago 19m 6660k - 19.0.0-3133-ga8c84f59 76bc6b91b726 2825d5ba11c6 2024-04-20T21:39:02.079 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (50s) 45s ago 17m 6665k - 19.0.0-3133-ga8c84f59 76bc6b91b726 20f746fa1bba 2024-04-20T21:39:02.079 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (4m) 10s ago 18m 74.7M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T21:39:02.079 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (11m) 10s ago 11m 65.7M - 18.2.0 dc2bc1663786 5e6460aa6159 2024-04-20T21:39:02.079 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (11m) 10s ago 11m 130M - 18.2.0 dc2bc1663786 e65f2267544e 2024-04-20T21:39:02.079 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (11m) 45s ago 11m 117M - 18.2.0 dc2bc1663786 fb2a8a223db8 2024-04-20T21:39:02.079 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (11m) 45s ago 11m 55.6M - 18.2.0 dc2bc1663786 f2eec2ee8951 2024-04-20T21:39:02.080 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (7m) 10s ago 20m 500M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T21:39:02.080 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (6m) 45s ago 17m 450M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T21:39:02.080 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (2m) 10s ago 20m 46.8M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 019eb1479485 2024-04-20T21:39:02.080 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (2m) 45s ago 17m 36.6M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 55aac560aab6 2024-04-20T21:39:02.080 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (6m) 10s ago 18m 7717k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T21:39:02.080 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (5m) 45s ago 17m 7972k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T21:39:02.080 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (18s) 10s ago 16m 11.2M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 ee2bb901f26e 2024-04-20T21:39:02.080 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (15m) 10s ago 15m 212M 1216M 18.2.0 dc2bc1663786 44e4d07416ad 2024-04-20T21:39:02.080 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (15m) 10s ago 15m 173M 1216M 18.2.0 dc2bc1663786 31c5d9586e98 2024-04-20T21:39:02.080 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (14m) 45s ago 14m 242M 2240M 18.2.0 dc2bc1663786 611c1e242693 2024-04-20T21:39:02.080 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (14m) 45s ago 14m 221M 2240M 18.2.0 dc2bc1663786 6c8fce91d45d 2024-04-20T21:39:02.080 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (13m) 45s ago 13m 205M 2240M 18.2.0 dc2bc1663786 8e4574bae8db 2024-04-20T21:39:02.080 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (5m) 10s ago 18m 73.2M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T21:39:02.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:02.082+0000 7f4560c00700 1 -- 172.21.15.62:0/2150383559 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f454006ecc0 msgr2=0x7f4540071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:39:02.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:02.082+0000 7f4560c00700 1 --2- 172.21.15.62:0/2150383559 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f454006ecc0 0x7f4540071170 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f454c006010 tx=0x7f454c01a040 comp rx=0 tx=0).stop 2024-04-20T21:39:02.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:02.082+0000 7f4560c00700 1 -- 172.21.15.62:0/2150383559 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f455c069140 msgr2=0x7f455c196420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:39:02.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:02.082+0000 7f4560c00700 1 --2- 172.21.15.62:0/2150383559 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f455c069140 0x7f455c196420 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f454800c8f0 tx=0x7f454800ccb0 comp rx=0 tx=0).stop 2024-04-20T21:39:02.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:02.083+0000 7f4560c00700 1 -- 172.21.15.62:0/2150383559 shutdown_connections 2024-04-20T21:39:02.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:02.083+0000 7f4560c00700 1 --2- 172.21.15.62:0/2150383559 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f454006ecc0 0x7f4540071170 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:39:02.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:02.083+0000 7f4560c00700 1 --2- 172.21.15.62:0/2150383559 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f455c0687a0 0x7f455c195ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:39:02.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:02.083+0000 7f4560c00700 1 --2- 172.21.15.62:0/2150383559 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f455c069140 0x7f455c196420 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:39:02.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:02.083+0000 7f4560c00700 1 -- 172.21.15.62:0/2150383559 >> 172.21.15.62:0/2150383559 conn(0x7f455c075130 msgr2=0x7f455c0fe7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:39:02.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:02.083+0000 7f4560c00700 1 -- 172.21.15.62:0/2150383559 shutdown_connections 2024-04-20T21:39:02.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:02.083+0000 7f4560c00700 1 -- 172.21.15.62:0/2150383559 wait complete. 2024-04-20T21:39:02.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:01 smithi189 ceph-mon[57022]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-20T21:39:02.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:01 smithi189 ceph-mon[57022]: Upgrade: osd.1 is safe to restart 2024-04-20T21:39:02.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:01 smithi189 ceph-mon[57022]: pgmap v81: 65 pgs: 65 active+clean; 315 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-20T21:39:02.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:02.239+0000 7f72ff000700 1 -- 172.21.15.62:0/829368571 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72f8108470 msgr2=0x7f72f8108840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:39:02.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:02.239+0000 7f72ff000700 1 --2- 172.21.15.62:0/829368571 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72f8108470 0x7f72f8108840 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f72e8009910 tx=0x7f72e8009c20 comp rx=0 tx=0).stop 2024-04-20T21:39:02.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:02.239+0000 7f72ff000700 1 -- 172.21.15.62:0/829368571 shutdown_connections 2024-04-20T21:39:02.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:02.239+0000 7f72ff000700 1 --2- 172.21.15.62:0/829368571 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72f8102470 0x7f72f81028e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:39:02.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:02.239+0000 7f72ff000700 1 --2- 172.21.15.62:0/829368571 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72f8108470 0x7f72f8108840 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:39:02.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:02.239+0000 7f72ff000700 1 -- 172.21.15.62:0/829368571 >> 172.21.15.62:0/829368571 conn(0x7f72f80fdf90 msgr2=0x7f72f81003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:39:02.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:02.239+0000 7f72ff000700 1 -- 172.21.15.62:0/829368571 shutdown_connections 2024-04-20T21:39:02.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:02.240+0000 7f72ff000700 1 -- 172.21.15.62:0/829368571 wait complete. 2024-04-20T21:39:02.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:02.241+0000 7f72ff000700 1 Processor -- start 2024-04-20T21:39:02.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:02.241+0000 7f72ff000700 1 -- start start 2024-04-20T21:39:02.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:02.241+0000 7f72ff000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72f8102470 0x7f72f81981d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:39:02.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:02.241+0000 7f72ff000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72f8108470 0x7f72f8198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:39:02.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:02.241+0000 7f72ff000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72f8198df0 con 0x7f72f8102470 2024-04-20T21:39:02.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:02.241+0000 7f72ff000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72f819cb80 con 0x7f72f8108470 2024-04-20T21:39:02.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:02.241+0000 7f72f7e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72f8102470 0x7f72f81981d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:39:02.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:02.241+0000 7f72f7e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72f8102470 0x7f72f81981d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:58320/0 (socket says 172.21.15.62:58320) 2024-04-20T21:39:02.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:02.241+0000 7f72f7e00700 1 -- 172.21.15.62:0/3378514456 learned_addr learned my addr 172.21.15.62:0/3378514456 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:39:02.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:02.242+0000 7f72f7e00700 1 -- 172.21.15.62:0/3378514456 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72f8108470 msgr2=0x7f72f8198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:39:02.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:02.242+0000 7f72f7e00700 1 --2- 172.21.15.62:0/3378514456 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72f8108470 0x7f72f8198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:39:02.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:02.242+0000 7f72f7e00700 1 -- 172.21.15.62:0/3378514456 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72e80095f0 con 0x7f72f8102470 2024-04-20T21:39:02.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:02.242+0000 7f72f7e00700 1 --2- 172.21.15.62:0/3378514456 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72f8102470 0x7f72f81981d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f72e8006010 tx=0x7f72e800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:39:02.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:02.242+0000 7f72f4c00700 1 -- 172.21.15.62:0/3378514456 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f72e801d070 con 0x7f72f8102470 2024-04-20T21:39:02.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:02.242+0000 7f72f4c00700 1 -- 172.21.15.62:0/3378514456 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f72e8004fb0 con 0x7f72f8102470 2024-04-20T21:39:02.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:02.242+0000 7f72ff000700 1 -- 172.21.15.62:0/3378514456 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72f819ce00 con 0x7f72f8102470 2024-04-20T21:39:02.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:02.242+0000 7f72f4c00700 1 -- 172.21.15.62:0/3378514456 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f72e8022ea0 con 0x7f72f8102470 2024-04-20T21:39:02.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:02.243+0000 7f72ff000700 1 -- 172.21.15.62:0/3378514456 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72f819d2f0 con 0x7f72f8102470 2024-04-20T21:39:02.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:02.244+0000 7f72ff000700 1 -- 172.21.15.62:0/3378514456 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72f810a9e0 con 0x7f72f8102470 2024-04-20T21:39:02.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:02.245+0000 7f72f4c00700 1 -- 172.21.15.62:0/3378514456 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f72e8022470 con 0x7f72f8102470 2024-04-20T21:39:02.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:02.245+0000 7f72f4c00700 1 --2- 172.21.15.62:0/3378514456 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f72dc06ecc0 0x7f72dc071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:39:02.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:02.246+0000 7f72f4c00700 1 -- 172.21.15.62:0/3378514456 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f72e8090340 con 0x7f72f8102470 2024-04-20T21:39:02.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:02.246+0000 7f72f7400700 1 --2- 172.21.15.62:0/3378514456 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f72dc06ecc0 0x7f72dc071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:39:02.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:02.246+0000 7f72f7400700 1 --2- 172.21.15.62:0/3378514456 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f72dc06ecc0 0x7f72dc071170 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f72f81997f0 tx=0x7f72e4006d20 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:39:02.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:02.250+0000 7f72f4c00700 1 -- 172.21.15.62:0/3378514456 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f72e8059250 con 0x7f72f8102470 2024-04-20T21:39:02.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:02.583+0000 7f72ff000700 1 -- 172.21.15.62:0/3378514456 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f72f819d5d0 con 0x7f72f8102470 2024-04-20T21:39:02.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:02.583+0000 7f72f4c00700 1 -- 172.21.15.62:0/3378514456 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+791 (secure 0 0 0) 0x7f72e8022720 con 0x7f72f8102470 2024-04-20T21:39:02.583 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:39:02.583 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T21:39:02.583 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:39:02.583 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:39:02.583 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T21:39:02.583 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:39:02.583 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:39:02.583 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T21:39:02.584 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 5, 2024-04-20T21:39:02.584 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 1 2024-04-20T21:39:02.584 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:39:02.584 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T21:39:02.584 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 4 2024-04-20T21:39:02.584 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:39:02.584 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T21:39:02.584 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 9, 2024-04-20T21:39:02.584 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 5 2024-04-20T21:39:02.584 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T21:39:02.584 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:39:02.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:02.586+0000 7f72ff000700 1 -- 172.21.15.62:0/3378514456 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f72dc06ecc0 msgr2=0x7f72dc071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:39:02.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:02.586+0000 7f72ff000700 1 --2- 172.21.15.62:0/3378514456 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f72dc06ecc0 0x7f72dc071170 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f72f81997f0 tx=0x7f72e4006d20 comp rx=0 tx=0).stop 2024-04-20T21:39:02.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:02.586+0000 7f72ff000700 1 -- 172.21.15.62:0/3378514456 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72f8102470 msgr2=0x7f72f81981d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:39:02.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:02.586+0000 7f72ff000700 1 --2- 172.21.15.62:0/3378514456 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72f8102470 0x7f72f81981d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f72e8006010 tx=0x7f72e800be20 comp rx=0 tx=0).stop 2024-04-20T21:39:02.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:02.586+0000 7f72ff000700 1 -- 172.21.15.62:0/3378514456 shutdown_connections 2024-04-20T21:39:02.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:02.586+0000 7f72ff000700 1 --2- 172.21.15.62:0/3378514456 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f72dc06ecc0 0x7f72dc071170 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:39:02.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:02.586+0000 7f72ff000700 1 --2- 172.21.15.62:0/3378514456 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72f8102470 0x7f72f81981d0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:39:02.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:02.586+0000 7f72ff000700 1 --2- 172.21.15.62:0/3378514456 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72f8108470 0x7f72f8198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:39:02.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:02.586+0000 7f72ff000700 1 -- 172.21.15.62:0/3378514456 >> 172.21.15.62:0/3378514456 conn(0x7f72f80fdf90 msgr2=0x7f72f80feb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:39:02.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:02.587+0000 7f72ff000700 1 -- 172.21.15.62:0/3378514456 shutdown_connections 2024-04-20T21:39:02.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:02.587+0000 7f72ff000700 1 -- 172.21.15.62:0/3378514456 wait complete. 2024-04-20T21:39:02.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:02.739+0000 7f0120a00700 1 -- 172.21.15.62:0/3688133050 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f011c101d30 msgr2=0x7f011c10a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:39:02.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:02.739+0000 7f0120a00700 1 --2- 172.21.15.62:0/3688133050 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f011c101d30 0x7f011c10a230 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f0108009910 tx=0x7f0108009c20 comp rx=0 tx=0).stop 2024-04-20T21:39:02.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:02.739+0000 7f0120a00700 1 -- 172.21.15.62:0/3688133050 shutdown_connections 2024-04-20T21:39:02.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:02.739+0000 7f0120a00700 1 --2- 172.21.15.62:0/3688133050 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f011c101d30 0x7f011c10a230 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:39:02.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:02.739+0000 7f0120a00700 1 --2- 172.21.15.62:0/3688133050 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f011c101420 0x7f011c1017f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:39:02.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:02.739+0000 7f0120a00700 1 -- 172.21.15.62:0/3688133050 >> 172.21.15.62:0/3688133050 conn(0x7f011c0fac80 msgr2=0x7f011c0fd090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:39:02.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:02.739+0000 7f0120a00700 1 -- 172.21.15.62:0/3688133050 shutdown_connections 2024-04-20T21:39:02.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:02.739+0000 7f0120a00700 1 -- 172.21.15.62:0/3688133050 wait complete. 2024-04-20T21:39:02.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:02.740+0000 7f0120a00700 1 Processor -- start 2024-04-20T21:39:02.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:02.740+0000 7f0120a00700 1 -- start start 2024-04-20T21:39:02.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:02.741+0000 7f0120a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f011c101420 0x7f011c1980c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:39:02.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:02.741+0000 7f0120a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f011c101d30 0x7f011c198600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:39:02.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:02.741+0000 7f0120a00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f011c198ce0 con 0x7f011c101420 2024-04-20T21:39:02.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:02.741+0000 7f0120a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f011c19ca70 con 0x7f011c101d30 2024-04-20T21:39:02.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:02.741+0000 7f0119e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f011c101420 0x7f011c1980c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:39:02.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:02.741+0000 7f0119e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f011c101420 0x7f011c1980c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:58352/0 (socket says 172.21.15.62:58352) 2024-04-20T21:39:02.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:02.741+0000 7f0119e00700 1 -- 172.21.15.62:0/2072162980 learned_addr learned my addr 172.21.15.62:0/2072162980 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:39:02.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:02.742+0000 7f0119400700 1 --2- 172.21.15.62:0/2072162980 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f011c101d30 0x7f011c198600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:39:02.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:02.742+0000 7f0119e00700 1 -- 172.21.15.62:0/2072162980 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f011c101d30 msgr2=0x7f011c198600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:39:02.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:02.742+0000 7f0119e00700 1 --2- 172.21.15.62:0/2072162980 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f011c101d30 0x7f011c198600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:39:02.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:02.742+0000 7f0119e00700 1 -- 172.21.15.62:0/2072162980 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01080095f0 con 0x7f011c101420 2024-04-20T21:39:02.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:02.742+0000 7f0119e00700 1 --2- 172.21.15.62:0/2072162980 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f011c101420 0x7f011c1980c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f010c00d8d0 tx=0x7f010c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:39:02.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:02.743+0000 7f0112a00700 1 -- 172.21.15.62:0/2072162980 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f010c009940 con 0x7f011c101420 2024-04-20T21:39:02.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:02.743+0000 7f0112a00700 1 -- 172.21.15.62:0/2072162980 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f010c010460 con 0x7f011c101420 2024-04-20T21:39:02.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:02.743+0000 7f0120a00700 1 -- 172.21.15.62:0/2072162980 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f011c19cd50 con 0x7f011c101420 2024-04-20T21:39:02.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:02.743+0000 7f0112a00700 1 -- 172.21.15.62:0/2072162980 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f010c00f5f0 con 0x7f011c101420 2024-04-20T21:39:02.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:02.743+0000 7f0120a00700 1 -- 172.21.15.62:0/2072162980 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f011c19d2a0 con 0x7f011c101420 2024-04-20T21:39:02.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:02.744+0000 7f0120a00700 1 -- 172.21.15.62:0/2072162980 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f011c109d00 con 0x7f011c101420 2024-04-20T21:39:02.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:02.746+0000 7f0112a00700 1 -- 172.21.15.62:0/2072162980 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f010c0105d0 con 0x7f011c101420 2024-04-20T21:39:02.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:02.747+0000 7f0112a00700 1 --2- 172.21.15.62:0/2072162980 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f010006ecc0 0x7f0100071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:39:02.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:02.747+0000 7f0112a00700 1 -- 172.21.15.62:0/2072162980 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f010c08ee70 con 0x7f011c101420 2024-04-20T21:39:02.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:02.747+0000 7f0119400700 1 --2- 172.21.15.62:0/2072162980 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f010006ecc0 0x7f0100071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:39:02.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:02.748+0000 7f0119400700 1 --2- 172.21.15.62:0/2072162980 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f010006ecc0 0x7f0100071170 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f011c1996e0 tx=0x7f010801a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:39:02.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:02.752+0000 7f0112a00700 1 -- 172.21.15.62:0/2072162980 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f010c057c30 con 0x7f011c101420 2024-04-20T21:39:03.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.024+0000 7f0120a00700 1 -- 172.21.15.62:0/2072162980 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f011c19d4c0 con 0x7f011c101420 2024-04-20T21:39:03.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.024+0000 7f0112a00700 1 -- 172.21.15.62:0/2072162980 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 18 v18) v1 ==== 76+0+1928 (secure 0 0 0) 0x7f010c05b2f0 con 0x7f011c101420 2024-04-20T21:39:03.024 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:02 smithi062 ceph-mon[76322]: from='client.34246 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:39:03.024 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:02 smithi062 ceph-mon[76322]: from='client.34250 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:39:03.024 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:02 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:39:03.024 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:02 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-20T21:39:03.024 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:02 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:39:03.024 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:02 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/3378514456' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:39:03.025 INFO:teuthology.orchestra.run.smithi062.stdout:e18 2024-04-20T21:39:03.025 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T21:39:03.025 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T21:39:03.025 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T21:39:03.025 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:39:03.025 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T21:39:03.025 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T21:39:03.025 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 18 2024-04-20T21:39:03.026 INFO:teuthology.orchestra.run.smithi062.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-20T21:39:03.026 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T21:39:03.026 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T21:37:06.018300+0000 2024-04-20T21:39:03.026 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T21:39:03.026 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T21:39:03.026 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T21:39:03.026 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T21:39:03.026 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T21:39:03.026 INFO:teuthology.orchestra.run.smithi062.stdout:max_xattr_size 65536 2024-04-20T21:39:03.026 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T21:39:03.026 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T21:39:03.026 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 0 2024-04-20T21:39:03.026 INFO:teuthology.orchestra.run.smithi062.stdout:snap_rstat 1 2024-04-20T21:39:03.026 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T21:39:03.027 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 2 2024-04-20T21:39:03.027 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1 2024-04-20T21:39:03.027 INFO:teuthology.orchestra.run.smithi062.stdout:up {0=14458,1=24279} 2024-04-20T21:39:03.027 INFO:teuthology.orchestra.run.smithi062.stdout:failed 2024-04-20T21:39:03.027 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T21:39:03.027 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T21:39:03.027 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T21:39:03.027 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T21:39:03.027 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data enabled 2024-04-20T21:39:03.027 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T21:39:03.027 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T21:39:03.027 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 1 2024-04-20T21:39:03.027 INFO:teuthology.orchestra.run.smithi062.stdout:qdb_cluster leader: 14458 members: 24279,14458 2024-04-20T21:39:03.027 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ryqblb{0:14458} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.62:6826/1147127123,v1:172.21.15.62:6827/1147127123] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:39:03.028 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ozizwi{0:14474} state up:standby-replay seq 3 join_fscid=1 addr [v2:172.21.15.62:6828/3494174293,v1:172.21.15.62:6829/3494174293] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:39:03.028 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.dvimer{1:24279} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.189:6824/237262514,v1:172.21.15.189:6825/237262514] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:39:03.028 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.xbgspw{1:24291} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.189:6826/1495356258,v1:172.21.15.189:6827/1495356258] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:39:03.028 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:39:03.028 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:39:03.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.028+0000 7f0120a00700 1 -- 172.21.15.62:0/2072162980 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f010006ecc0 msgr2=0x7f0100071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:39:03.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.028+0000 7f0120a00700 1 --2- 172.21.15.62:0/2072162980 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f010006ecc0 0x7f0100071170 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f011c1996e0 tx=0x7f010801a040 comp rx=0 tx=0).stop 2024-04-20T21:39:03.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.028+0000 7f0120a00700 1 -- 172.21.15.62:0/2072162980 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f011c101420 msgr2=0x7f011c1980c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:39:03.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.028+0000 7f0120a00700 1 --2- 172.21.15.62:0/2072162980 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f011c101420 0x7f011c1980c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f010c00d8d0 tx=0x7f010c00dc90 comp rx=0 tx=0).stop 2024-04-20T21:39:03.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.028+0000 7f0120a00700 1 -- 172.21.15.62:0/2072162980 shutdown_connections 2024-04-20T21:39:03.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.028+0000 7f0120a00700 1 --2- 172.21.15.62:0/2072162980 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f010006ecc0 0x7f0100071170 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:39:03.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.028+0000 7f0120a00700 1 --2- 172.21.15.62:0/2072162980 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f011c101420 0x7f011c1980c0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:39:03.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.028+0000 7f0120a00700 1 --2- 172.21.15.62:0/2072162980 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f011c101d30 0x7f011c198600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:39:03.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.028+0000 7f0120a00700 1 -- 172.21.15.62:0/2072162980 >> 172.21.15.62:0/2072162980 conn(0x7f011c0fac80 msgr2=0x7f011c0ff840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:39:03.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.028+0000 7f0120a00700 1 -- 172.21.15.62:0/2072162980 shutdown_connections 2024-04-20T21:39:03.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.028+0000 7f0120a00700 1 -- 172.21.15.62:0/2072162980 wait complete. 2024-04-20T21:39:03.029 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 18 2024-04-20T21:39:03.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.182+0000 7f2be1600700 1 -- 172.21.15.62:0/4139059648 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2bdc0730f0 msgr2=0x7f2bdc0734c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:39:03.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.182+0000 7f2be1600700 1 --2- 172.21.15.62:0/4139059648 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2bdc0730f0 0x7f2bdc0734c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f2bcc009910 tx=0x7f2bcc009c20 comp rx=0 tx=0).stop 2024-04-20T21:39:03.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.183+0000 7f2be1600700 1 -- 172.21.15.62:0/4139059648 shutdown_connections 2024-04-20T21:39:03.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.183+0000 7f2be1600700 1 --2- 172.21.15.62:0/4139059648 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2bdc073a00 0x7f2bdc10c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:39:03.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.183+0000 7f2be1600700 1 --2- 172.21.15.62:0/4139059648 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2bdc0730f0 0x7f2bdc0734c0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:39:03.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.183+0000 7f2be1600700 1 -- 172.21.15.62:0/4139059648 >> 172.21.15.62:0/4139059648 conn(0x7f2bdc0fbda0 msgr2=0x7f2bdc0fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:39:03.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.183+0000 7f2be1600700 1 -- 172.21.15.62:0/4139059648 shutdown_connections 2024-04-20T21:39:03.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.183+0000 7f2be1600700 1 -- 172.21.15.62:0/4139059648 wait complete. 2024-04-20T21:39:03.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.184+0000 7f2be1600700 1 Processor -- start 2024-04-20T21:39:03.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.184+0000 7f2be1600700 1 -- start start 2024-04-20T21:39:03.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.185+0000 7f2be1600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2bdc0730f0 0x7f2bdc197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:39:03.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.185+0000 7f2be1600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2bdc073a00 0x7f2bdc198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:39:03.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.185+0000 7f2be1600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bdc198be0 con 0x7f2bdc073a00 2024-04-20T21:39:03.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.185+0000 7f2be1600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bdc19c970 con 0x7f2bdc0730f0 2024-04-20T21:39:03.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.185+0000 7f2bda000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2bdc073a00 0x7f2bdc198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:39:03.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.185+0000 7f2bda000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2bdc073a00 0x7f2bdc198500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:58382/0 (socket says 172.21.15.62:58382) 2024-04-20T21:39:03.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.185+0000 7f2bda000700 1 -- 172.21.15.62:0/2486006539 learned_addr learned my addr 172.21.15.62:0/2486006539 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:39:03.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.185+0000 7f2bdaa00700 1 --2- 172.21.15.62:0/2486006539 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2bdc0730f0 0x7f2bdc197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:39:03.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.185+0000 7f2bda000700 1 -- 172.21.15.62:0/2486006539 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2bdc0730f0 msgr2=0x7f2bdc197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:39:03.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.185+0000 7f2bda000700 1 --2- 172.21.15.62:0/2486006539 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2bdc0730f0 0x7f2bdc197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:39:03.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.185+0000 7f2bda000700 1 -- 172.21.15.62:0/2486006539 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bcc0095f0 con 0x7f2bdc073a00 2024-04-20T21:39:03.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.186+0000 7f2bda000700 1 --2- 172.21.15.62:0/2486006539 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2bdc073a00 0x7f2bdc198500 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f2bc800c930 tx=0x7f2bc800ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:39:03.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.186+0000 7f2bd3400700 1 -- 172.21.15.62:0/2486006539 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2bc8007ad0 con 0x7f2bdc073a00 2024-04-20T21:39:03.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.186+0000 7f2be1600700 1 -- 172.21.15.62:0/2486006539 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bdc19cc50 con 0x7f2bdc073a00 2024-04-20T21:39:03.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.186+0000 7f2bd3400700 1 -- 172.21.15.62:0/2486006539 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2bc8007c30 con 0x7f2bdc073a00 2024-04-20T21:39:03.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.186+0000 7f2bd3400700 1 -- 172.21.15.62:0/2486006539 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2bc8018610 con 0x7f2bdc073a00 2024-04-20T21:39:03.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.186+0000 7f2be1600700 1 -- 172.21.15.62:0/2486006539 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bdc19d1a0 con 0x7f2bdc073a00 2024-04-20T21:39:03.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.188+0000 7f2be1600700 1 -- 172.21.15.62:0/2486006539 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bdc109c80 con 0x7f2bdc073a00 2024-04-20T21:39:03.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.189+0000 7f2bd3400700 1 -- 172.21.15.62:0/2486006539 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f2bc800f450 con 0x7f2bdc073a00 2024-04-20T21:39:03.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.190+0000 7f2bd3400700 1 --2- 172.21.15.62:0/2486006539 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2bc006ecc0 0x7f2bc0071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:39:03.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.190+0000 7f2bd3400700 1 -- 172.21.15.62:0/2486006539 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f2bc808efe0 con 0x7f2bdc073a00 2024-04-20T21:39:03.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.190+0000 7f2bdaa00700 1 --2- 172.21.15.62:0/2486006539 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2bc006ecc0 0x7f2bc0071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:39:03.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.191+0000 7f2bdaa00700 1 --2- 172.21.15.62:0/2486006539 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2bc006ecc0 0x7f2bc0071170 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f2bcc006010 tx=0x7f2bcc01a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:39:03.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.198+0000 7f2bd3400700 1 -- 172.21.15.62:0/2486006539 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f2bc8057dc0 con 0x7f2bdc073a00 2024-04-20T21:39:03.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:02 smithi189 ceph-mon[57022]: from='client.34246 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:39:03.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:02 smithi189 ceph-mon[57022]: from='client.34250 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:39:03.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:02 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:39:03.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:02 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-20T21:39:03.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:02 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:39:03.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:02 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/3378514456' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:39:03.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.458+0000 7f2be1600700 1 -- 172.21.15.62:0/2486006539 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2bdc19d490 con 0x7f2bc006ecc0 2024-04-20T21:39:03.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.461+0000 7f2bd3400700 1 -- 172.21.15.62:0/2486006539 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f2bdc19d490 con 0x7f2bc006ecc0 2024-04-20T21:39:03.461 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:39:03.461 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T21:39:03.461 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T21:39:03.461 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-20T21:39:03.461 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T21:39:03.461 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr", 2024-04-20T21:39:03.461 INFO:teuthology.orchestra.run.smithi062.stdout: "crash", 2024-04-20T21:39:03.461 INFO:teuthology.orchestra.run.smithi062.stdout: "mon" 2024-04-20T21:39:03.461 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T21:39:03.462 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "7/23 daemons upgraded", 2024-04-20T21:39:03.462 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading osd daemons", 2024-04-20T21:39:03.462 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T21:39:03.462 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:39:03.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.464+0000 7f2be1600700 1 -- 172.21.15.62:0/2486006539 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2bc006ecc0 msgr2=0x7f2bc0071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:39:03.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.464+0000 7f2be1600700 1 --2- 172.21.15.62:0/2486006539 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2bc006ecc0 0x7f2bc0071170 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f2bcc006010 tx=0x7f2bcc01a040 comp rx=0 tx=0).stop 2024-04-20T21:39:03.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.464+0000 7f2be1600700 1 -- 172.21.15.62:0/2486006539 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2bdc073a00 msgr2=0x7f2bdc198500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:39:03.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.464+0000 7f2be1600700 1 --2- 172.21.15.62:0/2486006539 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2bdc073a00 0x7f2bdc198500 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f2bc800c930 tx=0x7f2bc800ccf0 comp rx=0 tx=0).stop 2024-04-20T21:39:03.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.465+0000 7f2be1600700 1 -- 172.21.15.62:0/2486006539 shutdown_connections 2024-04-20T21:39:03.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.465+0000 7f2be1600700 1 --2- 172.21.15.62:0/2486006539 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2bc006ecc0 0x7f2bc0071170 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:39:03.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.465+0000 7f2be1600700 1 --2- 172.21.15.62:0/2486006539 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2bdc0730f0 0x7f2bdc197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:39:03.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.465+0000 7f2be1600700 1 --2- 172.21.15.62:0/2486006539 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2bdc073a00 0x7f2bdc198500 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:39:03.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.465+0000 7f2be1600700 1 -- 172.21.15.62:0/2486006539 >> 172.21.15.62:0/2486006539 conn(0x7f2bdc0fbda0 msgr2=0x7f2bdc106dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:39:03.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.465+0000 7f2be1600700 1 -- 172.21.15.62:0/2486006539 shutdown_connections 2024-04-20T21:39:03.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.465+0000 7f2be1600700 1 -- 172.21.15.62:0/2486006539 wait complete. 2024-04-20T21:39:03.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.626+0000 7f8bd9c00700 1 -- 172.21.15.62:0/2410608061 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8bd4106b80 msgr2=0x7f8bd4106ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:39:03.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.626+0000 7f8bd9c00700 1 --2- 172.21.15.62:0/2410608061 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8bd4106b80 0x7f8bd4106ff0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f8bbc009910 tx=0x7f8bbc009c20 comp rx=0 tx=0).stop 2024-04-20T21:39:03.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.626+0000 7f8bd9c00700 1 -- 172.21.15.62:0/2410608061 shutdown_connections 2024-04-20T21:39:03.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.626+0000 7f8bd9c00700 1 --2- 172.21.15.62:0/2410608061 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8bd4106b80 0x7f8bd4106ff0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:39:03.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.626+0000 7f8bd9c00700 1 --2- 172.21.15.62:0/2410608061 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8bd410c8f0 0x7f8bd410ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:39:03.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.626+0000 7f8bd9c00700 1 -- 172.21.15.62:0/2410608061 >> 172.21.15.62:0/2410608061 conn(0x7f8bd4076020 msgr2=0x7f8bd4078430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:39:03.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.626+0000 7f8bd9c00700 1 -- 172.21.15.62:0/2410608061 shutdown_connections 2024-04-20T21:39:03.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.626+0000 7f8bd9c00700 1 -- 172.21.15.62:0/2410608061 wait complete. 2024-04-20T21:39:03.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.627+0000 7f8bd9c00700 1 Processor -- start 2024-04-20T21:39:03.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.627+0000 7f8bd9c00700 1 -- start start 2024-04-20T21:39:03.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.628+0000 7f8bd9c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8bd4106b80 0x7f8bd419c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:39:03.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.628+0000 7f8bd9c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8bd410c8f0 0x7f8bd419c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:39:03.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.628+0000 7f8bd9c00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bd419d070 con 0x7f8bd410c8f0 2024-04-20T21:39:03.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.628+0000 7f8bd9c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bd41a0e00 con 0x7f8bd4106b80 2024-04-20T21:39:03.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.628+0000 7f8bca000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8bd410c8f0 0x7f8bd419c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:39:03.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.628+0000 7f8bca000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8bd410c8f0 0x7f8bd419c990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:58398/0 (socket says 172.21.15.62:58398) 2024-04-20T21:39:03.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.628+0000 7f8bca000700 1 -- 172.21.15.62:0/4178729330 learned_addr learned my addr 172.21.15.62:0/4178729330 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:39:03.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.629+0000 7f8bd2a00700 1 --2- 172.21.15.62:0/4178729330 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8bd4106b80 0x7f8bd419c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:39:03.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.629+0000 7f8bca000700 1 -- 172.21.15.62:0/4178729330 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8bd4106b80 msgr2=0x7f8bd419c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:39:03.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.629+0000 7f8bca000700 1 --2- 172.21.15.62:0/4178729330 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8bd4106b80 0x7f8bd419c450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:39:03.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.629+0000 7f8bca000700 1 -- 172.21.15.62:0/4178729330 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bbc0095f0 con 0x7f8bd410c8f0 2024-04-20T21:39:03.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.629+0000 7f8bca000700 1 --2- 172.21.15.62:0/4178729330 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8bd410c8f0 0x7f8bd419c990 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f8bbc00b5c0 tx=0x7f8bbc00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:39:03.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.629+0000 7f8bcbe00700 1 -- 172.21.15.62:0/4178729330 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8bbc01d070 con 0x7f8bd410c8f0 2024-04-20T21:39:03.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.629+0000 7f8bcbe00700 1 -- 172.21.15.62:0/4178729330 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8bbc005e00 con 0x7f8bd410c8f0 2024-04-20T21:39:03.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.629+0000 7f8bd9c00700 1 -- 172.21.15.62:0/4178729330 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bd41a1080 con 0x7f8bd410c8f0 2024-04-20T21:39:03.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.629+0000 7f8bcbe00700 1 -- 172.21.15.62:0/4178729330 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8bbc017400 con 0x7f8bd410c8f0 2024-04-20T21:39:03.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.629+0000 7f8bd9c00700 1 -- 172.21.15.62:0/4178729330 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bd41a1570 con 0x7f8bd410c8f0 2024-04-20T21:39:03.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.631+0000 7f8bd9c00700 1 -- 172.21.15.62:0/4178729330 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bd404f2a0 con 0x7f8bd410c8f0 2024-04-20T21:39:03.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.632+0000 7f8bcbe00700 1 -- 172.21.15.62:0/4178729330 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f8bbc00f460 con 0x7f8bd410c8f0 2024-04-20T21:39:03.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.633+0000 7f8bcbe00700 1 --2- 172.21.15.62:0/4178729330 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f8bb806ecc0 0x7f8bb8071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:39:03.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.633+0000 7f8bcbe00700 1 -- 172.21.15.62:0/4178729330 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f8bbc090310 con 0x7f8bd410c8f0 2024-04-20T21:39:03.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.633+0000 7f8bd2a00700 1 --2- 172.21.15.62:0/4178729330 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f8bb806ecc0 0x7f8bb8071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:39:03.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.634+0000 7f8bd2a00700 1 --2- 172.21.15.62:0/4178729330 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f8bb806ecc0 0x7f8bb8071170 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f8bc4005950 tx=0x7f8bc40058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:39:03.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.638+0000 7f8bcbe00700 1 -- 172.21.15.62:0/4178729330 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f8bbc059170 con 0x7f8bd410c8f0 2024-04-20T21:39:03.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.973+0000 7f8bd9c00700 1 -- 172.21.15.62:0/4178729330 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f8bd419d7b0 con 0x7f8bd410c8f0 2024-04-20T21:39:03.973 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:03 smithi062 ceph-mon[76322]: from='client.34254 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:39:03.973 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:03 smithi062 ceph-mon[76322]: pgmap v82: 65 pgs: 65 active+clean; 315 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-20T21:39:03.973 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:03 smithi062 ceph-mon[76322]: Upgrade: Updating osd.1 2024-04-20T21:39:03.973 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:03 smithi062 ceph-mon[76322]: Deploying daemon osd.1 on smithi062 2024-04-20T21:39:03.973 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:03 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/2072162980' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T21:39:03.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.974+0000 7f8bcbe00700 1 -- 172.21.15.62:0/4178729330 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7f8bbc05c830 con 0x7f8bd410c8f0 2024-04-20T21:39:03.974 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-04-20T21:39:03.974 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T21:39:03.974 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T21:39:03.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.977+0000 7f8bd9c00700 1 -- 172.21.15.62:0/4178729330 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f8bb806ecc0 msgr2=0x7f8bb8071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:39:03.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.977+0000 7f8bd9c00700 1 --2- 172.21.15.62:0/4178729330 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f8bb806ecc0 0x7f8bb8071170 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f8bc4005950 tx=0x7f8bc40058e0 comp rx=0 tx=0).stop 2024-04-20T21:39:03.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.977+0000 7f8bd9c00700 1 -- 172.21.15.62:0/4178729330 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8bd410c8f0 msgr2=0x7f8bd419c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:39:03.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.977+0000 7f8bd9c00700 1 --2- 172.21.15.62:0/4178729330 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8bd410c8f0 0x7f8bd419c990 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f8bbc00b5c0 tx=0x7f8bbc00be20 comp rx=0 tx=0).stop 2024-04-20T21:39:03.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.978+0000 7f8bd9c00700 1 -- 172.21.15.62:0/4178729330 shutdown_connections 2024-04-20T21:39:03.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.978+0000 7f8bd9c00700 1 --2- 172.21.15.62:0/4178729330 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f8bb806ecc0 0x7f8bb8071170 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:39:03.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.978+0000 7f8bd9c00700 1 --2- 172.21.15.62:0/4178729330 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8bd4106b80 0x7f8bd419c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:39:03.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.978+0000 7f8bd9c00700 1 --2- 172.21.15.62:0/4178729330 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8bd410c8f0 0x7f8bd419c990 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:39:03.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.978+0000 7f8bd9c00700 1 -- 172.21.15.62:0/4178729330 >> 172.21.15.62:0/4178729330 conn(0x7f8bd4076020 msgr2=0x7f8bd4077a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:39:03.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.978+0000 7f8bd9c00700 1 -- 172.21.15.62:0/4178729330 shutdown_connections 2024-04-20T21:39:03.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:03.978+0000 7f8bd9c00700 1 -- 172.21.15.62:0/4178729330 wait complete. 2024-04-20T21:39:04.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:03 smithi189 ceph-mon[57022]: from='client.34254 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:39:04.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:03 smithi189 ceph-mon[57022]: pgmap v82: 65 pgs: 65 active+clean; 315 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-20T21:39:04.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:03 smithi189 ceph-mon[57022]: Upgrade: Updating osd.1 2024-04-20T21:39:04.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:03 smithi189 ceph-mon[57022]: Deploying daemon osd.1 on smithi062 2024-04-20T21:39:04.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:03 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/2072162980' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T21:39:05.108 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:04 smithi062 ceph-mon[76322]: from='client.34266 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:39:05.108 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:04 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/4178729330' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T21:39:05.108 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:04 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:39:05.109 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:04 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:39:05.109 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:04 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:39:05.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:04 smithi189 ceph-mon[57022]: from='client.34266 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:39:05.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:04 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/4178729330' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T21:39:05.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:04 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:39:05.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:04 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:39:05.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:04 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:39:06.203 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:05 smithi062 ceph-mon[76322]: pgmap v83: 65 pgs: 65 active+clean; 315 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-20T21:39:06.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:05 smithi189 ceph-mon[57022]: pgmap v83: 65 pgs: 65 active+clean; 315 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-20T21:39:07.863 INFO:journalctl@ceph.osd.1.smithi062.stdout:Apr 20 21:39:07 smithi062 systemd[1]: Stopping Ceph osd.1 for 61ae3628-ff5b-11ee-bc93-c7b262605968... 2024-04-20T21:39:08.144 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:07 smithi062 ceph-mon[76322]: pgmap v84: 65 pgs: 65 active+clean; 315 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-20T21:39:08.145 INFO:journalctl@ceph.osd.1.smithi062.stdout:Apr 20 21:39:07 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-1[36649]: 2024-04-20T21:39:07.986+0000 7f32f3400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-20T21:39:08.145 INFO:journalctl@ceph.osd.1.smithi062.stdout:Apr 20 21:39:07 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-1[36649]: 2024-04-20T21:39:07.986+0000 7f32f3400700 -1 osd.1 53 *** Got signal Terminated *** 2024-04-20T21:39:08.145 INFO:journalctl@ceph.osd.1.smithi062.stdout:Apr 20 21:39:07 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-1[36649]: 2024-04-20T21:39:07.986+0000 7f32f3400700 -1 osd.1 53 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-20T21:39:08.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:07 smithi189 ceph-mon[57022]: pgmap v84: 65 pgs: 65 active+clean; 315 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-20T21:39:09.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:08 smithi189 ceph-mon[57022]: osd.1 marked itself down and dead 2024-04-20T21:39:09.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:08 smithi062 ceph-mon[76322]: osd.1 marked itself down and dead 2024-04-20T21:39:09.785 INFO:journalctl@ceph.osd.1.smithi062.stdout:Apr 20 21:39:09 smithi062 podman[93058]: 2024-04-20 21:39:09.528953261 +0000 UTC m=+1.679085044 container died 44e4d07416ad91fcc251e521da596d78f9753a3a95e68bcb4d079260c317d636 (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-1, org.label-schema.schema-version=1.0, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, org.label-schema.build-date=20231212, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.0, RELEASE=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-04-20T21:39:10.065 INFO:journalctl@ceph.osd.1.smithi062.stdout:Apr 20 21:39:10 smithi062 podman[93058]: 2024-04-20 21:39:10.03044643 +0000 UTC m=+2.180578207 container cleanup 44e4d07416ad91fcc251e521da596d78f9753a3a95e68bcb4d079260c317d636 (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-1, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20231212, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.0, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, ceph=True, GIT_BRANCH=HEAD, RELEASE=HEAD, maintainer=Guillaume Abrioux ) 2024-04-20T21:39:10.065 INFO:journalctl@ceph.osd.1.smithi062.stdout:Apr 20 21:39:10 smithi062 bash[93058]: ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-1 2024-04-20T21:39:10.066 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:09 smithi062 ceph-mon[76322]: pgmap v85: 65 pgs: 65 active+clean; 315 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:39:10.066 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:09 smithi062 ceph-mon[76322]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-20T21:39:10.066 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:09 smithi062 ceph-mon[76322]: osdmap e54: 6 total, 5 up, 6 in 2024-04-20T21:39:10.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:09 smithi189 ceph-mon[57022]: pgmap v85: 65 pgs: 65 active+clean; 315 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:39:10.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:09 smithi189 ceph-mon[57022]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-20T21:39:10.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:09 smithi189 ceph-mon[57022]: osdmap e54: 6 total, 5 up, 6 in 2024-04-20T21:39:10.534 INFO:journalctl@ceph.osd.1.smithi062.stdout:Apr 20 21:39:10 smithi062 podman[93074]: 2024-04-20 21:39:10.282476359 +0000 UTC m=+0.739077472 container remove 44e4d07416ad91fcc251e521da596d78f9753a3a95e68bcb4d079260c317d636 (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-1, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, RELEASE=HEAD, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-18.2.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, io.buildah.version=1.29.1, org.label-schema.build-date=20231212) 2024-04-20T21:39:11.158 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:10 smithi062 ceph-mon[76322]: osdmap e55: 6 total, 5 up, 6 in 2024-04-20T21:39:11.158 INFO:journalctl@ceph.osd.1.smithi062.stdout:Apr 20 21:39:11 smithi062 podman[93165]: 2024-04-20 21:39:10.996174193 +0000 UTC m=+0.033020000 image pull 76bc6b91b726a03b8b736eeeb35fc1ffd1469e23c7d60f5b444a217528d8906d quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 2024-04-20T21:39:11.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:10 smithi189 ceph-mon[57022]: osdmap e55: 6 total, 5 up, 6 in 2024-04-20T21:39:11.420 INFO:journalctl@ceph.osd.1.smithi062.stdout:Apr 20 21:39:11 smithi062 podman[93165]: 2024-04-20 21:39:11.158387795 +0000 UTC m=+0.195233639 container create 5ef1ae3d4fd71f61409a96d76efa7aa844e503398c70aeff49d8d533322757e8 (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-1-deactivate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-20T21:39:11.420 INFO:journalctl@ceph.osd.1.smithi062.stdout:Apr 20 21:39:11 smithi062 podman[93165]: 2024-04-20 21:39:11.420894115 +0000 UTC m=+0.457739966 container init 5ef1ae3d4fd71f61409a96d76efa7aa844e503398c70aeff49d8d533322757e8 (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-1-deactivate, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-20T21:39:11.785 INFO:journalctl@ceph.osd.1.smithi062.stdout:Apr 20 21:39:11 smithi062 podman[93165]: 2024-04-20 21:39:11.433170705 +0000 UTC m=+0.470016555 container start 5ef1ae3d4fd71f61409a96d76efa7aa844e503398c70aeff49d8d533322757e8 (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416) 2024-04-20T21:39:11.785 INFO:journalctl@ceph.osd.1.smithi062.stdout:Apr 20 21:39:11 smithi062 podman[93165]: 2024-04-20 21:39:11.561379764 +0000 UTC m=+0.598225661 container attach 5ef1ae3d4fd71f61409a96d76efa7aa844e503398c70aeff49d8d533322757e8 (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-1-deactivate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-20T21:39:12.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:11 smithi189 ceph-mon[57022]: pgmap v88: 65 pgs: 14 active+undersized+degraded, 3 stale+active+clean, 15 active+undersized, 33 active+clean; 315 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 2 op/s; 49/366 objects degraded (13.388%) 2024-04-20T21:39:12.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:11 smithi189 ceph-mon[57022]: Health check failed: Degraded data redundancy: 49/366 objects degraded (13.388%), 14 pgs degraded (PG_DEGRADED) 2024-04-20T21:39:12.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:11 smithi062 ceph-mon[76322]: pgmap v88: 65 pgs: 14 active+undersized+degraded, 3 stale+active+clean, 15 active+undersized, 33 active+clean; 315 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 2 op/s; 49/366 objects degraded (13.388%) 2024-04-20T21:39:12.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:11 smithi062 ceph-mon[76322]: Health check failed: Degraded data redundancy: 49/366 objects degraded (13.388%), 14 pgs degraded (PG_DEGRADED) 2024-04-20T21:39:12.285 INFO:journalctl@ceph.osd.1.smithi062.stdout:Apr 20 21:39:11 smithi062 podman[93165]: 2024-04-20 21:39:11.883491733 +0000 UTC m=+0.920337630 container died 5ef1ae3d4fd71f61409a96d76efa7aa844e503398c70aeff49d8d533322757e8 (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-1-deactivate, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-04-20T21:39:12.916 INFO:journalctl@ceph.osd.1.smithi062.stdout:Apr 20 21:39:12 smithi062 podman[93191]: 2024-04-20 21:39:12.623365118 +0000 UTC m=+0.721548249 container remove 5ef1ae3d4fd71f61409a96d76efa7aa844e503398c70aeff49d8d533322757e8 (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-1-deactivate, ceph=True, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-04-20T21:39:12.916 INFO:journalctl@ceph.osd.1.smithi062.stdout:Apr 20 21:39:12 smithi062 systemd[1]: ceph-61ae3628-ff5b-11ee-bc93-c7b262605968@osd.1.service: Deactivated successfully. 2024-04-20T21:39:12.916 INFO:journalctl@ceph.osd.1.smithi062.stdout:Apr 20 21:39:12 smithi062 systemd[1]: Stopped Ceph osd.1 for 61ae3628-ff5b-11ee-bc93-c7b262605968. 2024-04-20T21:39:12.916 INFO:journalctl@ceph.osd.1.smithi062.stdout:Apr 20 21:39:12 smithi062 systemd[1]: ceph-61ae3628-ff5b-11ee-bc93-c7b262605968@osd.1.service: Consumed 1min 45.596s CPU time. 2024-04-20T21:39:13.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:12 smithi189 ceph-mon[57022]: pgmap v89: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 315 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 56/366 objects degraded (15.301%) 2024-04-20T21:39:13.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:12 smithi062 ceph-mon[76322]: pgmap v89: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 315 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 56/366 objects degraded (15.301%) 2024-04-20T21:39:13.285 INFO:journalctl@ceph.osd.1.smithi062.stdout:Apr 20 21:39:13 smithi062 systemd[1]: Starting Ceph osd.1 for 61ae3628-ff5b-11ee-bc93-c7b262605968... 2024-04-20T21:39:13.755 INFO:journalctl@ceph.osd.1.smithi062.stdout:Apr 20 21:39:13 smithi062 podman[93291]: 2024-04-20 21:39:13.324727063 +0000 UTC m=+0.036348152 image pull 76bc6b91b726a03b8b736eeeb35fc1ffd1469e23c7d60f5b444a217528d8906d quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 2024-04-20T21:39:13.755 INFO:journalctl@ceph.osd.1.smithi062.stdout:Apr 20 21:39:13 smithi062 podman[93291]: 2024-04-20 21:39:13.449501362 +0000 UTC m=+0.161122496 container create fd2b2f428e401678505eb8e906c4b95d8708adf8f71a4e2cce36a1bf6ea2e40e (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-1-activate, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5) 2024-04-20T21:39:13.755 INFO:journalctl@ceph.osd.1.smithi062.stdout:Apr 20 21:39:13 smithi062 podman[93291]: 2024-04-20 21:39:13.755927885 +0000 UTC m=+0.467549019 container init fd2b2f428e401678505eb8e906c4b95d8708adf8f71a4e2cce36a1bf6ea2e40e (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-1-activate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, GIT_CLEAN=True) 2024-04-20T21:39:14.035 INFO:journalctl@ceph.osd.1.smithi062.stdout:Apr 20 21:39:13 smithi062 podman[93291]: 2024-04-20 21:39:13.768260914 +0000 UTC m=+0.479882014 container start fd2b2f428e401678505eb8e906c4b95d8708adf8f71a4e2cce36a1bf6ea2e40e (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-1-activate, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5) 2024-04-20T21:39:14.035 INFO:journalctl@ceph.osd.1.smithi062.stdout:Apr 20 21:39:13 smithi062 podman[93291]: 2024-04-20 21:39:13.871235873 +0000 UTC m=+0.582857063 container attach fd2b2f428e401678505eb8e906c4b95d8708adf8f71a4e2cce36a1bf6ea2e40e (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-1-activate, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-04-20T21:39:15.036 INFO:journalctl@ceph.osd.1.smithi062.stdout:Apr 20 21:39:14 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-1-activate[93306]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-20T21:39:15.037 INFO:journalctl@ceph.osd.1.smithi062.stdout:Apr 20 21:39:14 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-1-activate[93306]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-20T21:39:15.037 INFO:journalctl@ceph.osd.1.smithi062.stdout:Apr 20 21:39:14 smithi062 bash[93291]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-20T21:39:15.037 INFO:journalctl@ceph.osd.1.smithi062.stdout:Apr 20 21:39:14 smithi062 bash[93291]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-20T21:39:15.037 INFO:journalctl@ceph.osd.1.smithi062.stdout:Apr 20 21:39:14 smithi062 bash[93291]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-20T21:39:15.037 INFO:journalctl@ceph.osd.1.smithi062.stdout:Apr 20 21:39:14 smithi062 bash[93291]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-04-20T21:39:15.037 INFO:journalctl@ceph.osd.1.smithi062.stdout:Apr 20 21:39:14 smithi062 bash[93291]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-04-20T21:39:15.037 INFO:journalctl@ceph.osd.1.smithi062.stdout:Apr 20 21:39:14 smithi062 bash[93291]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-04-20T21:39:15.037 INFO:journalctl@ceph.osd.1.smithi062.stdout:Apr 20 21:39:14 smithi062 bash[93291]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-04-20T21:39:15.037 INFO:journalctl@ceph.osd.1.smithi062.stdout:Apr 20 21:39:14 smithi062 bash[93291]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-20T21:39:15.037 INFO:journalctl@ceph.osd.1.smithi062.stdout:Apr 20 21:39:14 smithi062 bash[93291]: --> ceph-volume raw activate successful for osd ID: 1 2024-04-20T21:39:15.037 INFO:journalctl@ceph.osd.1.smithi062.stdout:Apr 20 21:39:14 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-1-activate[93306]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-20T21:39:15.037 INFO:journalctl@ceph.osd.1.smithi062.stdout:Apr 20 21:39:14 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-1-activate[93306]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-04-20T21:39:15.037 INFO:journalctl@ceph.osd.1.smithi062.stdout:Apr 20 21:39:14 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-1-activate[93306]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-04-20T21:39:15.037 INFO:journalctl@ceph.osd.1.smithi062.stdout:Apr 20 21:39:14 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-1-activate[93306]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-04-20T21:39:15.038 INFO:journalctl@ceph.osd.1.smithi062.stdout:Apr 20 21:39:14 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-1-activate[93306]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-04-20T21:39:15.038 INFO:journalctl@ceph.osd.1.smithi062.stdout:Apr 20 21:39:14 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-1-activate[93306]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-20T21:39:15.038 INFO:journalctl@ceph.osd.1.smithi062.stdout:Apr 20 21:39:14 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-1-activate[93306]: --> ceph-volume raw activate successful for osd ID: 1 2024-04-20T21:39:15.038 INFO:journalctl@ceph.osd.1.smithi062.stdout:Apr 20 21:39:14 smithi062 podman[93291]: 2024-04-20 21:39:14.841460603 +0000 UTC m=+1.553081745 container died fd2b2f428e401678505eb8e906c4b95d8708adf8f71a4e2cce36a1bf6ea2e40e (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-1-activate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-04-20T21:39:15.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:15 smithi062 ceph-mon[76322]: pgmap v90: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 315 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 56/366 objects degraded (15.301%) 2024-04-20T21:39:15.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:15 smithi189 ceph-mon[57022]: pgmap v90: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 315 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 56/366 objects degraded (15.301%) 2024-04-20T21:39:15.952 INFO:journalctl@ceph.osd.1.smithi062.stdout:Apr 20 21:39:15 smithi062 podman[93451]: 2024-04-20 21:39:15.592290478 +0000 UTC m=+0.728828150 container remove fd2b2f428e401678505eb8e906c4b95d8708adf8f71a4e2cce36a1bf6ea2e40e (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-1-activate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-20T21:39:16.237 INFO:journalctl@ceph.osd.1.smithi062.stdout:Apr 20 21:39:15 smithi062 podman[93516]: 2024-04-20 21:39:15.857952992 +0000 UTC m=+0.033471863 image pull 76bc6b91b726a03b8b736eeeb35fc1ffd1469e23c7d60f5b444a217528d8906d quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 2024-04-20T21:39:16.237 INFO:journalctl@ceph.osd.1.smithi062.stdout:Apr 20 21:39:15 smithi062 podman[93516]: 2024-04-20 21:39:15.968096112 +0000 UTC m=+0.143615043 container create 9075152d41629e2a60658c8ccccf136d6a2a45811f0c7a9786e49a741d6d62ca (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-1, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416) 2024-04-20T21:39:16.535 INFO:journalctl@ceph.osd.1.smithi062.stdout:Apr 20 21:39:16 smithi062 podman[93516]: 2024-04-20 21:39:16.23864684 +0000 UTC m=+0.414165755 container init 9075152d41629e2a60658c8ccccf136d6a2a45811f0c7a9786e49a741d6d62ca (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-1, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, GIT_CLEAN=True, ceph=True) 2024-04-20T21:39:16.535 INFO:journalctl@ceph.osd.1.smithi062.stdout:Apr 20 21:39:16 smithi062 podman[93516]: 2024-04-20 21:39:16.250224983 +0000 UTC m=+0.425743898 container start 9075152d41629e2a60658c8ccccf136d6a2a45811f0c7a9786e49a741d6d62ca (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-1, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-20T21:39:16.535 INFO:journalctl@ceph.osd.1.smithi062.stdout:Apr 20 21:39:16 smithi062 ceph-osd[93534]: -- 172.21.15.62:0/571685221 <== mon.0 v2:172.21.15.62:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558bdd4fb680 con 0x558bdd4c4c00 2024-04-20T21:39:16.535 INFO:journalctl@ceph.osd.1.smithi062.stdout:Apr 20 21:39:16 smithi062 bash[93516]: 9075152d41629e2a60658c8ccccf136d6a2a45811f0c7a9786e49a741d6d62ca 2024-04-20T21:39:16.535 INFO:journalctl@ceph.osd.1.smithi062.stdout:Apr 20 21:39:16 smithi062 systemd[1]: Started Ceph osd.1 for 61ae3628-ff5b-11ee-bc93-c7b262605968. 2024-04-20T21:39:17.285 INFO:journalctl@ceph.osd.1.smithi062.stdout:Apr 20 21:39:16 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-1[93530]: 2024-04-20T21:39:16.833+0000 7f22292ad7c0 -1 Falling back to public interface 2024-04-20T21:39:17.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:17 smithi189 ceph-mon[57022]: pgmap v91: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 315 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 56/366 objects degraded (15.301%) 2024-04-20T21:39:17.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:17 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:39:17.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:17 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:39:17.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:17 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:39:17.784 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:17 smithi062 ceph-mon[76322]: pgmap v91: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 315 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 56/366 objects degraded (15.301%) 2024-04-20T21:39:17.784 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:17 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:39:17.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:17 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:39:17.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:17 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:39:19.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:19 smithi062 ceph-mon[76322]: pgmap v92: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 315 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 56/366 objects degraded (15.301%) 2024-04-20T21:39:19.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:19 smithi062 ceph-mon[76322]: Health check update: Degraded data redundancy: 56/366 objects degraded (15.301%), 16 pgs degraded (PG_DEGRADED) 2024-04-20T21:39:19.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:19 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:39:19.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:19 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:39:19.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:19 smithi189 ceph-mon[57022]: pgmap v92: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 315 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 56/366 objects degraded (15.301%) 2024-04-20T21:39:19.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:19 smithi189 ceph-mon[57022]: Health check update: Degraded data redundancy: 56/366 objects degraded (15.301%), 16 pgs degraded (PG_DEGRADED) 2024-04-20T21:39:19.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:19 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:39:19.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:19 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:39:21.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:21 smithi062 ceph-mon[76322]: pgmap v93: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 315 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 1 op/s; 56/366 objects degraded (15.301%) 2024-04-20T21:39:21.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:21 smithi189 ceph-mon[57022]: pgmap v93: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 315 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 1 op/s; 56/366 objects degraded (15.301%) 2024-04-20T21:39:22.285 INFO:journalctl@ceph.osd.1.smithi062.stdout:Apr 20 21:39:22 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-1[93530]: 2024-04-20T21:39:22.091+0000 7f22292ad7c0 -1 osd.1 0 read_superblock omap replica is missing. 2024-04-20T21:39:22.785 INFO:journalctl@ceph.osd.1.smithi062.stdout:Apr 20 21:39:22 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-1[93530]: 2024-04-20T21:39:22.519+0000 7f22292ad7c0 -1 osd.1 53 log_to_monitors true 2024-04-20T21:39:23.784 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:23 smithi062 ceph-mon[76322]: pgmap v94: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 315 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 56/366 objects degraded (15.301%) 2024-04-20T21:39:23.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:23 smithi062 ceph-mon[76322]: from='osd.1 [v2:172.21.15.62:6810/2879835300,v1:172.21.15.62:6811/2879835300]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-20T21:39:23.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:23 smithi189 ceph-mon[57022]: pgmap v94: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 315 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 56/366 objects degraded (15.301%) 2024-04-20T21:39:23.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:23 smithi189 ceph-mon[57022]: from='osd.1 [v2:172.21.15.62:6810/2879835300,v1:172.21.15.62:6811/2879835300]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-20T21:39:24.785 INFO:journalctl@ceph.osd.1.smithi062.stdout:Apr 20 21:39:24 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-1[93530]: 2024-04-20T21:39:24.605+0000 7f2213600700 -1 osd.1 53 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-20T21:39:24.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:24 smithi062 ceph-mon[76322]: from='osd.1 [v2:172.21.15.62:6810/2879835300,v1:172.21.15.62:6811/2879835300]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-20T21:39:24.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:24 smithi062 ceph-mon[76322]: osdmap e56: 6 total, 5 up, 6 in 2024-04-20T21:39:24.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:24 smithi062 ceph-mon[76322]: from='osd.1 [v2:172.21.15.62:6810/2879835300,v1:172.21.15.62:6811/2879835300]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-04-20T21:39:24.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:24 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:39:24.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:24 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:39:24.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:24 smithi189 ceph-mon[57022]: from='osd.1 [v2:172.21.15.62:6810/2879835300,v1:172.21.15.62:6811/2879835300]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-20T21:39:24.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:24 smithi189 ceph-mon[57022]: osdmap e56: 6 total, 5 up, 6 in 2024-04-20T21:39:24.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:24 smithi189 ceph-mon[57022]: from='osd.1 [v2:172.21.15.62:6810/2879835300,v1:172.21.15.62:6811/2879835300]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-04-20T21:39:24.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:24 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:39:24.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:24 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:39:25.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:25 smithi189 ceph-mon[57022]: pgmap v96: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 315 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s; 56/366 objects degraded (15.301%) 2024-04-20T21:39:25.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:25 smithi189 ceph-mon[57022]: from='osd.1 [v2:172.21.15.62:6810/2879835300,v1:172.21.15.62:6811/2879835300]' entity='osd.1' 2024-04-20T21:39:26.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:25 smithi062 ceph-mon[76322]: pgmap v96: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 315 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s; 56/366 objects degraded (15.301%) 2024-04-20T21:39:26.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:25 smithi062 ceph-mon[76322]: from='osd.1 [v2:172.21.15.62:6810/2879835300,v1:172.21.15.62:6811/2879835300]' entity='osd.1' 2024-04-20T21:39:26.935 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:26 smithi062 ceph-mon[76322]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-20T21:39:26.935 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:26 smithi062 ceph-mon[76322]: osd.1 [v2:172.21.15.62:6810/2879835300,v1:172.21.15.62:6811/2879835300] boot 2024-04-20T21:39:26.935 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:26 smithi062 ceph-mon[76322]: osdmap e57: 6 total, 6 up, 6 in 2024-04-20T21:39:26.936 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:26 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-20T21:39:26.936 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:26 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:39:26.936 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:26 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:39:26.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:26 smithi189 ceph-mon[57022]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-20T21:39:26.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:26 smithi189 ceph-mon[57022]: osd.1 [v2:172.21.15.62:6810/2879835300,v1:172.21.15.62:6811/2879835300] boot 2024-04-20T21:39:26.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:26 smithi189 ceph-mon[57022]: osdmap e57: 6 total, 6 up, 6 in 2024-04-20T21:39:26.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:26 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-20T21:39:26.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:26 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:39:26.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:26 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:39:27.765 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:27 smithi062 ceph-mon[76322]: pgmap v98: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 315 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s; 56/366 objects degraded (15.301%) 2024-04-20T21:39:27.765 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:27 smithi062 ceph-mon[76322]: osdmap e58: 6 total, 6 up, 6 in 2024-04-20T21:39:27.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:27 smithi189 ceph-mon[57022]: pgmap v98: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 315 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s; 56/366 objects degraded (15.301%) 2024-04-20T21:39:27.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:27 smithi189 ceph-mon[57022]: osdmap e58: 6 total, 6 up, 6 in 2024-04-20T21:39:29.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:29 smithi189 ceph-mon[57022]: pgmap v100: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 315 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s; 56/366 objects degraded (15.301%) 2024-04-20T21:39:30.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:29 smithi062 ceph-mon[76322]: pgmap v100: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 315 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s; 56/366 objects degraded (15.301%) 2024-04-20T21:39:30.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:30 smithi189 ceph-mon[57022]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 56/366 objects degraded (15.301%), 16 pgs degraded) 2024-04-20T21:39:31.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:30 smithi062 ceph-mon[76322]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 56/366 objects degraded (15.301%), 16 pgs degraded) 2024-04-20T21:39:31.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:31 smithi189 ceph-mon[57022]: pgmap v101: 65 pgs: 2 active+undersized, 63 active+clean; 315 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-20T21:39:32.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:31 smithi062 ceph-mon[76322]: pgmap v101: 65 pgs: 2 active+undersized, 63 active+clean; 315 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-20T21:39:33.154 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:32 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:39:33.154 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:32 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:39:33.154 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:32 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:39:33.154 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:32 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T21:39:33.154 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:32 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:39:33.154 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:32 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:39:33.154 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:32 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:39:33.154 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:32 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:39:33.154 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:32 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:39:33.155 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:32 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-20T21:39:33.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:32 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:39:33.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:32 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:39:33.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:32 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:39:33.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:32 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T21:39:33.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:32 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:39:33.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:32 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:39:33.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:32 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:39:33.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:32 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:39:33.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:32 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:39:33.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:32 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-20T21:39:34.014 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:33 smithi062 ceph-mon[76322]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-20T21:39:34.014 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:33 smithi062 ceph-mon[76322]: Upgrade: osd.2 is safe to restart 2024-04-20T21:39:34.014 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:33 smithi062 ceph-mon[76322]: pgmap v102: 65 pgs: 65 active+clean; 315 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-20T21:39:34.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:34.159+0000 7f503ac00700 1 -- 172.21.15.62:0/2972188552 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f50340687a0 msgr2=0x7f5034068b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:39:34.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:34.159+0000 7f503ac00700 1 --2- 172.21.15.62:0/2972188552 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f50340687a0 0x7f5034068b70 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f5024009910 tx=0x7f5024009c20 comp rx=0 tx=0).stop 2024-04-20T21:39:34.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:34.160+0000 7f503ac00700 1 -- 172.21.15.62:0/2972188552 shutdown_connections 2024-04-20T21:39:34.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:34.160+0000 7f503ac00700 1 --2- 172.21.15.62:0/2972188552 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5034069140 0x7f5034104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:39:34.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:34.160+0000 7f503ac00700 1 --2- 172.21.15.62:0/2972188552 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f50340687a0 0x7f5034068b70 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:39:34.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:34.160+0000 7f503ac00700 1 -- 172.21.15.62:0/2972188552 >> 172.21.15.62:0/2972188552 conn(0x7f5034075130 msgr2=0x7f5034075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:39:34.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:34.160+0000 7f503ac00700 1 -- 172.21.15.62:0/2972188552 shutdown_connections 2024-04-20T21:39:34.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:34.160+0000 7f503ac00700 1 -- 172.21.15.62:0/2972188552 wait complete. 2024-04-20T21:39:34.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:34.162+0000 7f503ac00700 1 Processor -- start 2024-04-20T21:39:34.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:34.163+0000 7f503ac00700 1 -- start start 2024-04-20T21:39:34.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:34.163+0000 7f503ac00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f50340687a0 0x7f5034195f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:39:34.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:34.163+0000 7f503ac00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5034069140 0x7f50341964a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:39:34.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:34.163+0000 7f503ac00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5034196b80 con 0x7f50340687a0 2024-04-20T21:39:34.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:34.163+0000 7f503ac00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f503419a910 con 0x7f5034069140 2024-04-20T21:39:34.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:34.164+0000 7f5033e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f50340687a0 0x7f5034195f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:39:34.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:34.164+0000 7f5033e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f50340687a0 0x7f5034195f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40552/0 (socket says 172.21.15.62:40552) 2024-04-20T21:39:34.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:34.164+0000 7f5033e00700 1 -- 172.21.15.62:0/887998872 learned_addr learned my addr 172.21.15.62:0/887998872 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:39:34.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:34.164+0000 7f5033400700 1 --2- 172.21.15.62:0/887998872 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5034069140 0x7f50341964a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:39:34.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:34.164+0000 7f5033e00700 1 -- 172.21.15.62:0/887998872 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5034069140 msgr2=0x7f50341964a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:39:34.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:34.164+0000 7f5033e00700 1 --2- 172.21.15.62:0/887998872 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5034069140 0x7f50341964a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:39:34.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:34.164+0000 7f5033e00700 1 -- 172.21.15.62:0/887998872 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50240095f0 con 0x7f50340687a0 2024-04-20T21:39:34.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:34.165+0000 7f5033e00700 1 --2- 172.21.15.62:0/887998872 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f50340687a0 0x7f5034195f60 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f502400b5c0 tx=0x7f502400be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:39:34.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:34.165+0000 7f5030c00700 1 -- 172.21.15.62:0/887998872 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f502401d070 con 0x7f50340687a0 2024-04-20T21:39:34.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:34.165+0000 7f5030c00700 1 -- 172.21.15.62:0/887998872 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f502400f460 con 0x7f50340687a0 2024-04-20T21:39:34.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:34.165+0000 7f503ac00700 1 -- 172.21.15.62:0/887998872 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f503419ab90 con 0x7f50340687a0 2024-04-20T21:39:34.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:34.165+0000 7f5030c00700 1 -- 172.21.15.62:0/887998872 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5024017610 con 0x7f50340687a0 2024-04-20T21:39:34.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:34.166+0000 7f503ac00700 1 -- 172.21.15.62:0/887998872 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f503419b080 con 0x7f50340687a0 2024-04-20T21:39:34.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:34.167+0000 7f503ac00700 1 -- 172.21.15.62:0/887998872 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5034108790 con 0x7f50340687a0 2024-04-20T21:39:34.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:34.168+0000 7f5030c00700 1 -- 172.21.15.62:0/887998872 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f5024003a60 con 0x7f50340687a0 2024-04-20T21:39:34.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:34.170+0000 7f5030c00700 1 --2- 172.21.15.62:0/887998872 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f501806ecc0 0x7f5018071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:39:34.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:34.170+0000 7f5030c00700 1 -- 172.21.15.62:0/887998872 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f50240903e0 con 0x7f50340687a0 2024-04-20T21:39:34.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:34.170+0000 7f5033400700 1 --2- 172.21.15.62:0/887998872 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f501806ecc0 0x7f5018071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:39:34.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:34.171+0000 7f5033400700 1 --2- 172.21.15.62:0/887998872 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f501806ecc0 0x7f5018071170 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f5020006fd0 tx=0x7f5020008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:39:34.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:34.176+0000 7f5030c00700 1 -- 172.21.15.62:0/887998872 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f5024059240 con 0x7f50340687a0 2024-04-20T21:39:34.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:33 smithi189 ceph-mon[57022]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-20T21:39:34.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:33 smithi189 ceph-mon[57022]: Upgrade: osd.2 is safe to restart 2024-04-20T21:39:34.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:33 smithi189 ceph-mon[57022]: pgmap v102: 65 pgs: 65 active+clean; 315 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-20T21:39:34.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:34.442+0000 7f503ac00700 1 -- 172.21.15.62:0/887998872 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f503419b360 con 0x7f501806ecc0 2024-04-20T21:39:34.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:34.444+0000 7f5030c00700 1 -- 172.21.15.62:0/887998872 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f503419b360 con 0x7f501806ecc0 2024-04-20T21:39:34.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:34.447+0000 7f503ac00700 1 -- 172.21.15.62:0/887998872 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f501806ecc0 msgr2=0x7f5018071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:39:34.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:34.447+0000 7f503ac00700 1 --2- 172.21.15.62:0/887998872 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f501806ecc0 0x7f5018071170 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f5020006fd0 tx=0x7f5020008040 comp rx=0 tx=0).stop 2024-04-20T21:39:34.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:34.447+0000 7f503ac00700 1 -- 172.21.15.62:0/887998872 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f50340687a0 msgr2=0x7f5034195f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:39:34.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:34.447+0000 7f503ac00700 1 --2- 172.21.15.62:0/887998872 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f50340687a0 0x7f5034195f60 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f502400b5c0 tx=0x7f502400be00 comp rx=0 tx=0).stop 2024-04-20T21:39:34.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:34.448+0000 7f503ac00700 1 -- 172.21.15.62:0/887998872 shutdown_connections 2024-04-20T21:39:34.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:34.448+0000 7f503ac00700 1 --2- 172.21.15.62:0/887998872 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f501806ecc0 0x7f5018071170 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:39:34.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:34.448+0000 7f503ac00700 1 --2- 172.21.15.62:0/887998872 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f50340687a0 0x7f5034195f60 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:39:34.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:34.448+0000 7f503ac00700 1 --2- 172.21.15.62:0/887998872 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5034069140 0x7f50341964a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:39:34.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:34.448+0000 7f503ac00700 1 -- 172.21.15.62:0/887998872 >> 172.21.15.62:0/887998872 conn(0x7f5034075130 msgr2=0x7f50340fe7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:39:34.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:34.448+0000 7f503ac00700 1 -- 172.21.15.62:0/887998872 shutdown_connections 2024-04-20T21:39:34.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:34.448+0000 7f503ac00700 1 -- 172.21.15.62:0/887998872 wait complete. 2024-04-20T21:39:34.464 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T21:39:34.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:34.611+0000 7f233bc00700 1 -- 172.21.15.62:0/2864690135 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2334100e80 msgr2=0x7f2334104e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:39:34.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:34.611+0000 7f233bc00700 1 --2- 172.21.15.62:0/2864690135 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2334100e80 0x7f2334104e50 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f2320009910 tx=0x7f2320009c20 comp rx=0 tx=0).stop 2024-04-20T21:39:34.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:34.611+0000 7f233bc00700 1 -- 172.21.15.62:0/2864690135 shutdown_connections 2024-04-20T21:39:34.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:34.611+0000 7f233bc00700 1 --2- 172.21.15.62:0/2864690135 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2334100e80 0x7f2334104e50 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:39:34.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:34.611+0000 7f233bc00700 1 --2- 172.21.15.62:0/2864690135 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23341004e0 0x7f23341008b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:39:34.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:34.611+0000 7f233bc00700 1 -- 172.21.15.62:0/2864690135 >> 172.21.15.62:0/2864690135 conn(0x7f23340fbda0 msgr2=0x7f23340fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:39:34.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:34.611+0000 7f233bc00700 1 -- 172.21.15.62:0/2864690135 shutdown_connections 2024-04-20T21:39:34.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:34.611+0000 7f233bc00700 1 -- 172.21.15.62:0/2864690135 wait complete. 2024-04-20T21:39:34.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:34.612+0000 7f233bc00700 1 Processor -- start 2024-04-20T21:39:34.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:34.613+0000 7f233bc00700 1 -- start start 2024-04-20T21:39:34.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:34.613+0000 7f233bc00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f23341004e0 0x7f2334073050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:39:34.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:34.613+0000 7f233bc00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2334100e80 0x7f2334073590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:39:34.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:34.613+0000 7f233bc00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2334073ad0 con 0x7f23341004e0 2024-04-20T21:39:34.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:34.613+0000 7f233bc00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2334073c10 con 0x7f2334100e80 2024-04-20T21:39:34.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:34.613+0000 7f2339200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f23341004e0 0x7f2334073050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:39:34.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:34.613+0000 7f2339200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f23341004e0 0x7f2334073050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40570/0 (socket says 172.21.15.62:40570) 2024-04-20T21:39:34.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:34.613+0000 7f2339200700 1 -- 172.21.15.62:0/335804116 learned_addr learned my addr 172.21.15.62:0/335804116 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:39:34.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:34.614+0000 7f2333e00700 1 --2- 172.21.15.62:0/335804116 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2334100e80 0x7f2334073590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:39:34.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:34.614+0000 7f2339200700 1 -- 172.21.15.62:0/335804116 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2334100e80 msgr2=0x7f2334073590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:39:34.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:34.614+0000 7f2339200700 1 --2- 172.21.15.62:0/335804116 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2334100e80 0x7f2334073590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:39:34.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:34.614+0000 7f2339200700 1 -- 172.21.15.62:0/335804116 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23200095f0 con 0x7f23341004e0 2024-04-20T21:39:34.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:34.614+0000 7f2339200700 1 --2- 172.21.15.62:0/335804116 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f23341004e0 0x7f2334073050 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f232800c8a0 tx=0x7f232800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:39:34.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:34.614+0000 7f2331600700 1 -- 172.21.15.62:0/335804116 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f232800cea0 con 0x7f23341004e0 2024-04-20T21:39:34.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:34.615+0000 7f2331600700 1 -- 172.21.15.62:0/335804116 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2328004830 con 0x7f23341004e0 2024-04-20T21:39:34.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:34.615+0000 7f233bc00700 1 -- 172.21.15.62:0/335804116 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23341a2100 con 0x7f23341004e0 2024-04-20T21:39:34.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:34.615+0000 7f2331600700 1 -- 172.21.15.62:0/335804116 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f23280055b0 con 0x7f23341004e0 2024-04-20T21:39:34.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:34.615+0000 7f233bc00700 1 -- 172.21.15.62:0/335804116 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23341a2510 con 0x7f23341004e0 2024-04-20T21:39:34.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:34.616+0000 7f233bc00700 1 -- 172.21.15.62:0/335804116 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2334108850 con 0x7f23341004e0 2024-04-20T21:39:34.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:34.617+0000 7f2331600700 1 -- 172.21.15.62:0/335804116 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f23280049a0 con 0x7f23341004e0 2024-04-20T21:39:34.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:34.618+0000 7f2331600700 1 --2- 172.21.15.62:0/335804116 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f231c06ecc0 0x7f231c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:39:34.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:34.619+0000 7f2331600700 1 -- 172.21.15.62:0/335804116 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f232808eb40 con 0x7f23341004e0 2024-04-20T21:39:34.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:34.619+0000 7f2333e00700 1 --2- 172.21.15.62:0/335804116 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f231c06ecc0 0x7f231c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:39:34.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:34.619+0000 7f2333e00700 1 --2- 172.21.15.62:0/335804116 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f231c06ecc0 0x7f231c071170 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2334074840 tx=0x7f232001a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:39:34.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:34.623+0000 7f2331600700 1 -- 172.21.15.62:0/335804116 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f2328057900 con 0x7f23341004e0 2024-04-20T21:39:34.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:34.888+0000 7f233bc00700 1 -- 172.21.15.62:0/335804116 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2334061190 con 0x7f231c06ecc0 2024-04-20T21:39:34.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:34.890+0000 7f2331600700 1 -- 172.21.15.62:0/335804116 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f2334061190 con 0x7f231c06ecc0 2024-04-20T21:39:34.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:34.893+0000 7f233bc00700 1 -- 172.21.15.62:0/335804116 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f231c06ecc0 msgr2=0x7f231c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:39:34.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:34.893+0000 7f233bc00700 1 --2- 172.21.15.62:0/335804116 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f231c06ecc0 0x7f231c071170 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2334074840 tx=0x7f232001a040 comp rx=0 tx=0).stop 2024-04-20T21:39:34.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:34.893+0000 7f233bc00700 1 -- 172.21.15.62:0/335804116 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f23341004e0 msgr2=0x7f2334073050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:39:34.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:34.893+0000 7f233bc00700 1 --2- 172.21.15.62:0/335804116 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f23341004e0 0x7f2334073050 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f232800c8a0 tx=0x7f232800cc60 comp rx=0 tx=0).stop 2024-04-20T21:39:34.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:34.894+0000 7f233bc00700 1 -- 172.21.15.62:0/335804116 shutdown_connections 2024-04-20T21:39:34.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:34.894+0000 7f233bc00700 1 --2- 172.21.15.62:0/335804116 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f231c06ecc0 0x7f231c071170 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:39:34.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:34.894+0000 7f233bc00700 1 --2- 172.21.15.62:0/335804116 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f23341004e0 0x7f2334073050 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:39:34.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:34.894+0000 7f233bc00700 1 --2- 172.21.15.62:0/335804116 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2334100e80 0x7f2334073590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:39:34.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:34.894+0000 7f233bc00700 1 -- 172.21.15.62:0/335804116 >> 172.21.15.62:0/335804116 conn(0x7f23340fbda0 msgr2=0x7f23340fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:39:34.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:34.894+0000 7f233bc00700 1 -- 172.21.15.62:0/335804116 shutdown_connections 2024-04-20T21:39:34.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:34.894+0000 7f233bc00700 1 -- 172.21.15.62:0/335804116 wait complete. 2024-04-20T21:39:35.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:35.048+0000 7f4a35600700 1 -- 172.21.15.62:0/2107901320 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4a30073c00 msgr2=0x7f4a3010c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:39:35.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:35.048+0000 7f4a35600700 1 --2- 172.21.15.62:0/2107901320 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4a30073c00 0x7f4a3010c430 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f4a1c009910 tx=0x7f4a1c009c20 comp rx=0 tx=0).stop 2024-04-20T21:39:35.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:35.048+0000 7f4a35600700 1 -- 172.21.15.62:0/2107901320 shutdown_connections 2024-04-20T21:39:35.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:35.048+0000 7f4a35600700 1 --2- 172.21.15.62:0/2107901320 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4a30073c00 0x7f4a3010c430 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:39:35.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:35.048+0000 7f4a35600700 1 --2- 172.21.15.62:0/2107901320 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4a300732f0 0x7f4a300736c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:39:35.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:35.048+0000 7f4a35600700 1 -- 172.21.15.62:0/2107901320 >> 172.21.15.62:0/2107901320 conn(0x7f4a300fbbf0 msgr2=0x7f4a300fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:39:35.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:35.048+0000 7f4a35600700 1 -- 172.21.15.62:0/2107901320 shutdown_connections 2024-04-20T21:39:35.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:35.048+0000 7f4a35600700 1 -- 172.21.15.62:0/2107901320 wait complete. 2024-04-20T21:39:35.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:35.050+0000 7f4a35600700 1 Processor -- start 2024-04-20T21:39:35.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:35.050+0000 7f4a35600700 1 -- start start 2024-04-20T21:39:35.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:35.051+0000 7f4a35600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4a300732f0 0x7f4a30197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:39:35.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:35.051+0000 7f4a35600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4a30073c00 0x7f4a30198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:39:35.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:35.051+0000 7f4a35600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a30198b30 con 0x7f4a300732f0 2024-04-20T21:39:35.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:35.051+0000 7f4a35600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a3019c8c0 con 0x7f4a30073c00 2024-04-20T21:39:35.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:35.051+0000 7f4a2fe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4a300732f0 0x7f4a30197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:39:35.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:35.051+0000 7f4a2fe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4a300732f0 0x7f4a30197f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40588/0 (socket says 172.21.15.62:40588) 2024-04-20T21:39:35.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:35.051+0000 7f4a2fe00700 1 -- 172.21.15.62:0/1712105953 learned_addr learned my addr 172.21.15.62:0/1712105953 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:39:35.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:35.051+0000 7f4a2f400700 1 --2- 172.21.15.62:0/1712105953 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4a30073c00 0x7f4a30198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:39:35.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:35.052+0000 7f4a2fe00700 1 -- 172.21.15.62:0/1712105953 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4a30073c00 msgr2=0x7f4a30198450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:39:35.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:35.052+0000 7f4a2fe00700 1 --2- 172.21.15.62:0/1712105953 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4a30073c00 0x7f4a30198450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:39:35.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:35.052+0000 7f4a2fe00700 1 -- 172.21.15.62:0/1712105953 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a1c0095f0 con 0x7f4a300732f0 2024-04-20T21:39:35.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:35.052+0000 7f4a2fe00700 1 --2- 172.21.15.62:0/1712105953 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4a300732f0 0x7f4a30197f10 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f4a2000c8a0 tx=0x7f4a2000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:39:35.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:35.052+0000 7f4a2cc00700 1 -- 172.21.15.62:0/1712105953 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4a2000cea0 con 0x7f4a300732f0 2024-04-20T21:39:35.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:35.052+0000 7f4a2cc00700 1 -- 172.21.15.62:0/1712105953 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4a20004830 con 0x7f4a300732f0 2024-04-20T21:39:35.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:35.052+0000 7f4a2cc00700 1 -- 172.21.15.62:0/1712105953 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4a200055b0 con 0x7f4a300732f0 2024-04-20T21:39:35.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:35.052+0000 7f4a35600700 1 -- 172.21.15.62:0/1712105953 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a3019cba0 con 0x7f4a300732f0 2024-04-20T21:39:35.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:35.052+0000 7f4a35600700 1 -- 172.21.15.62:0/1712105953 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a3019d0f0 con 0x7f4a300732f0 2024-04-20T21:39:35.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:35.053+0000 7f4a35600700 1 -- 172.21.15.62:0/1712105953 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a30109b30 con 0x7f4a300732f0 2024-04-20T21:39:35.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:35.055+0000 7f4a2cc00700 1 -- 172.21.15.62:0/1712105953 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f4a200049a0 con 0x7f4a300732f0 2024-04-20T21:39:35.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:35.056+0000 7f4a2cc00700 1 --2- 172.21.15.62:0/1712105953 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f4a1406ecc0 0x7f4a14071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:39:35.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:35.056+0000 7f4a2cc00700 1 -- 172.21.15.62:0/1712105953 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f4a2008e9a0 con 0x7f4a300732f0 2024-04-20T21:39:35.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:35.056+0000 7f4a2f400700 1 --2- 172.21.15.62:0/1712105953 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f4a1406ecc0 0x7f4a14071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:39:35.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:35.057+0000 7f4a2f400700 1 --2- 172.21.15.62:0/1712105953 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f4a1406ecc0 0x7f4a14071170 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f4a30199530 tx=0x7f4a1c005a90 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:39:35.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:35.061+0000 7f4a2cc00700 1 -- 172.21.15.62:0/1712105953 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f4a20057880 con 0x7f4a300732f0 2024-04-20T21:39:35.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:35.319+0000 7f4a35600700 1 -- 172.21.15.62:0/1712105953 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f4a3019d420 con 0x7f4a1406ecc0 2024-04-20T21:39:35.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:35.341+0000 7f4a2cc00700 1 -- 172.21.15.62:0/1712105953 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f4a3019d420 con 0x7f4a1406ecc0 2024-04-20T21:39:35.341 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T21:39:35.341 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (5m) 11s ago 19m 14.2M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T21:39:35.341 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (19m) 11s ago 19m 8724k - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T21:39:35.341 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (18m) 78s ago 18m 8288k - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T21:39:35.341 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (91s) 11s ago 19m 6660k - 19.0.0-3133-ga8c84f59 76bc6b91b726 2825d5ba11c6 2024-04-20T21:39:35.342 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (83s) 78s ago 18m 6665k - 19.0.0-3133-ga8c84f59 76bc6b91b726 20f746fa1bba 2024-04-20T21:39:35.342 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (4m) 11s ago 18m 75.0M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T21:39:35.342 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (12m) 11s ago 12m 65.7M - 18.2.0 dc2bc1663786 5e6460aa6159 2024-04-20T21:39:35.342 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (12m) 11s ago 12m 131M - 18.2.0 dc2bc1663786 e65f2267544e 2024-04-20T21:39:35.342 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (12m) 78s ago 12m 117M - 18.2.0 dc2bc1663786 fb2a8a223db8 2024-04-20T21:39:35.342 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (11m) 78s ago 11m 55.6M - 18.2.0 dc2bc1663786 f2eec2ee8951 2024-04-20T21:39:35.342 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (8m) 11s ago 21m 502M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T21:39:35.342 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (6m) 78s ago 18m 450M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T21:39:35.342 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (3m) 11s ago 21m 48.2M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 019eb1479485 2024-04-20T21:39:35.342 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (2m) 78s ago 17m 36.6M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 55aac560aab6 2024-04-20T21:39:35.342 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (6m) 11s ago 19m 8090k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T21:39:35.342 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (6m) 78s ago 18m 7972k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T21:39:35.342 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (51s) 11s ago 16m 82.6M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 ee2bb901f26e 2024-04-20T21:39:35.342 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (19s) 11s ago 16m 11.2M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 9075152d4162 2024-04-20T21:39:35.343 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (15m) 11s ago 15m 173M 1216M 18.2.0 dc2bc1663786 31c5d9586e98 2024-04-20T21:39:35.343 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (15m) 78s ago 15m 242M 2240M 18.2.0 dc2bc1663786 611c1e242693 2024-04-20T21:39:35.343 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (14m) 78s ago 14m 221M 2240M 18.2.0 dc2bc1663786 6c8fce91d45d 2024-04-20T21:39:35.343 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (13m) 78s ago 13m 205M 2240M 18.2.0 dc2bc1663786 8e4574bae8db 2024-04-20T21:39:35.343 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (5m) 11s ago 18m 73.8M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T21:39:35.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:35.344+0000 7f4a35600700 1 -- 172.21.15.62:0/1712105953 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f4a1406ecc0 msgr2=0x7f4a14071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:39:35.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:35.344+0000 7f4a35600700 1 --2- 172.21.15.62:0/1712105953 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f4a1406ecc0 0x7f4a14071170 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f4a30199530 tx=0x7f4a1c005a90 comp rx=0 tx=0).stop 2024-04-20T21:39:35.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:35.344+0000 7f4a35600700 1 -- 172.21.15.62:0/1712105953 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4a300732f0 msgr2=0x7f4a30197f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:39:35.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:35.344+0000 7f4a35600700 1 --2- 172.21.15.62:0/1712105953 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4a300732f0 0x7f4a30197f10 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f4a2000c8a0 tx=0x7f4a2000cc60 comp rx=0 tx=0).stop 2024-04-20T21:39:35.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:35.344+0000 7f4a35600700 1 -- 172.21.15.62:0/1712105953 shutdown_connections 2024-04-20T21:39:35.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:35.344+0000 7f4a35600700 1 --2- 172.21.15.62:0/1712105953 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f4a1406ecc0 0x7f4a14071170 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:39:35.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:35.344+0000 7f4a35600700 1 --2- 172.21.15.62:0/1712105953 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4a300732f0 0x7f4a30197f10 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:39:35.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:35.344+0000 7f4a35600700 1 --2- 172.21.15.62:0/1712105953 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4a30073c00 0x7f4a30198450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:39:35.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:35.344+0000 7f4a35600700 1 -- 172.21.15.62:0/1712105953 >> 172.21.15.62:0/1712105953 conn(0x7f4a300fbbf0 msgr2=0x7f4a30106c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:39:35.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:35.345+0000 7f4a35600700 1 -- 172.21.15.62:0/1712105953 shutdown_connections 2024-04-20T21:39:35.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:35.345+0000 7f4a35600700 1 -- 172.21.15.62:0/1712105953 wait complete. 2024-04-20T21:39:35.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:35.499+0000 7fb06f200700 1 -- 172.21.15.62:0/2048281736 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb0700ff880 msgr2=0x7fb0700ffcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:39:35.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:35.499+0000 7fb06f200700 1 --2- 172.21.15.62:0/2048281736 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb0700ff880 0x7fb0700ffcf0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb05c009910 tx=0x7fb05c009c20 comp rx=0 tx=0).stop 2024-04-20T21:39:35.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:35.499+0000 7fb06f200700 1 -- 172.21.15.62:0/2048281736 shutdown_connections 2024-04-20T21:39:35.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:35.499+0000 7fb06f200700 1 --2- 172.21.15.62:0/2048281736 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb0700ff880 0x7fb0700ffcf0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:39:35.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:35.499+0000 7fb06f200700 1 --2- 172.21.15.62:0/2048281736 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb070100ef0 0x7fb0701012c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:39:35.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:35.499+0000 7fb06f200700 1 -- 172.21.15.62:0/2048281736 >> 172.21.15.62:0/2048281736 conn(0x7fb070075740 msgr2=0x7fb0700fd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:39:35.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:35.499+0000 7fb06f200700 1 -- 172.21.15.62:0/2048281736 shutdown_connections 2024-04-20T21:39:35.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:35.499+0000 7fb06f200700 1 -- 172.21.15.62:0/2048281736 wait complete. 2024-04-20T21:39:35.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:35.501+0000 7fb06f200700 1 Processor -- start 2024-04-20T21:39:35.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:35.501+0000 7fb06f200700 1 -- start start 2024-04-20T21:39:35.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:35.501+0000 7fb06f200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb0700ff880 0x7fb070195e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:39:35.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:35.501+0000 7fb06f200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb070100ef0 0x7fb070196340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:39:35.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:35.501+0000 7fb06f200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb070196a20 con 0x7fb0700ff880 2024-04-20T21:39:35.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:35.501+0000 7fb06f200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb07019a7b0 con 0x7fb070100ef0 2024-04-20T21:39:35.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:35.501+0000 7fb06de00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb0700ff880 0x7fb070195e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:39:35.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:35.501+0000 7fb06de00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb0700ff880 0x7fb070195e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40612/0 (socket says 172.21.15.62:40612) 2024-04-20T21:39:35.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:35.501+0000 7fb06de00700 1 -- 172.21.15.62:0/3361394623 learned_addr learned my addr 172.21.15.62:0/3361394623 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:39:35.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:35.502+0000 7fb06de00700 1 -- 172.21.15.62:0/3361394623 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb070100ef0 msgr2=0x7fb070196340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:39:35.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:35.502+0000 7fb06de00700 1 --2- 172.21.15.62:0/3361394623 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb070100ef0 0x7fb070196340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:39:35.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:35.502+0000 7fb06de00700 1 -- 172.21.15.62:0/3361394623 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb05c0095f0 con 0x7fb0700ff880 2024-04-20T21:39:35.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:35.502+0000 7fb06de00700 1 --2- 172.21.15.62:0/3361394623 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb0700ff880 0x7fb070195e00 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb06000d8d0 tx=0x7fb06000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:39:35.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:35.502+0000 7fb066a00700 1 -- 172.21.15.62:0/3361394623 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb060009940 con 0x7fb0700ff880 2024-04-20T21:39:35.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:35.503+0000 7fb066a00700 1 -- 172.21.15.62:0/3361394623 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb060010460 con 0x7fb0700ff880 2024-04-20T21:39:35.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:35.503+0000 7fb06f200700 1 -- 172.21.15.62:0/3361394623 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb07019aa90 con 0x7fb0700ff880 2024-04-20T21:39:35.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:35.503+0000 7fb066a00700 1 -- 172.21.15.62:0/3361394623 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb06000f5f0 con 0x7fb0700ff880 2024-04-20T21:39:35.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:35.503+0000 7fb06f200700 1 -- 172.21.15.62:0/3361394623 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb07019afe0 con 0x7fb0700ff880 2024-04-20T21:39:35.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:35.504+0000 7fb06f200700 1 -- 172.21.15.62:0/3361394623 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb070108650 con 0x7fb0700ff880 2024-04-20T21:39:35.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:35.505+0000 7fb066a00700 1 -- 172.21.15.62:0/3361394623 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fb0600105d0 con 0x7fb0700ff880 2024-04-20T21:39:35.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:35.506+0000 7fb066a00700 1 --2- 172.21.15.62:0/3361394623 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb05406ecc0 0x7fb054071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:39:35.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:35.506+0000 7fb066a00700 1 -- 172.21.15.62:0/3361394623 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fb06008ed60 con 0x7fb0700ff880 2024-04-20T21:39:35.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:35.507+0000 7fb06d400700 1 --2- 172.21.15.62:0/3361394623 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb05406ecc0 0x7fb054071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:39:35.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:35.507+0000 7fb06d400700 1 --2- 172.21.15.62:0/3361394623 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb05406ecc0 0x7fb054071170 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb05c00b5c0 tx=0x7fb05c005fd0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:39:35.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:35.511+0000 7fb066a00700 1 -- 172.21.15.62:0/3361394623 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fb060093070 con 0x7fb0700ff880 2024-04-20T21:39:35.639 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:35 smithi062 ceph-mon[76322]: pgmap v103: 65 pgs: 65 active+clean; 315 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 823 B/s rd, 1 op/s 2024-04-20T21:39:35.641 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:35 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:39:35.641 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:35 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:39:35.641 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:35 smithi062 ceph-mon[76322]: from='client.34278 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:39:35.641 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:35 smithi062 ceph-mon[76322]: Upgrade: Updating osd.2 2024-04-20T21:39:35.641 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:35 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:39:35.641 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:35 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-20T21:39:35.641 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:35 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:39:35.642 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:35 smithi062 ceph-mon[76322]: Deploying daemon osd.2 on smithi062 2024-04-20T21:39:35.642 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:35 smithi062 ceph-mon[76322]: from='client.34282 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:39:35.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:35 smithi189 ceph-mon[57022]: pgmap v103: 65 pgs: 65 active+clean; 315 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 823 B/s rd, 1 op/s 2024-04-20T21:39:35.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:35 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:39:35.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:35 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:39:35.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:35 smithi189 ceph-mon[57022]: from='client.34278 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:39:35.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:35 smithi189 ceph-mon[57022]: Upgrade: Updating osd.2 2024-04-20T21:39:35.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:35 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:39:35.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:35 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-20T21:39:35.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:35 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:39:35.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:35 smithi189 ceph-mon[57022]: Deploying daemon osd.2 on smithi062 2024-04-20T21:39:35.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:35 smithi189 ceph-mon[57022]: from='client.34282 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:39:35.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:35.851+0000 7fb06f200700 1 -- 172.21.15.62:0/3361394623 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fb070197260 con 0x7fb0700ff880 2024-04-20T21:39:35.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:35.852+0000 7fb066a00700 1 -- 172.21.15.62:0/3361394623 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+791 (secure 0 0 0) 0x7fb070197260 con 0x7fb0700ff880 2024-04-20T21:39:35.851 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:39:35.852 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T21:39:35.852 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:39:35.852 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:39:35.852 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T21:39:35.852 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:39:35.852 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:39:35.852 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T21:39:35.852 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 4, 2024-04-20T21:39:35.852 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:39:35.852 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:39:35.852 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T21:39:35.852 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 4 2024-04-20T21:39:35.852 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:39:35.852 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T21:39:35.852 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 8, 2024-04-20T21:39:35.853 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 6 2024-04-20T21:39:35.853 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T21:39:35.853 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:39:35.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:35.855+0000 7fb06f200700 1 -- 172.21.15.62:0/3361394623 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb05406ecc0 msgr2=0x7fb054071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:39:35.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:35.855+0000 7fb06f200700 1 --2- 172.21.15.62:0/3361394623 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb05406ecc0 0x7fb054071170 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb05c00b5c0 tx=0x7fb05c005fd0 comp rx=0 tx=0).stop 2024-04-20T21:39:35.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:35.855+0000 7fb06f200700 1 -- 172.21.15.62:0/3361394623 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb0700ff880 msgr2=0x7fb070195e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:39:35.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:35.855+0000 7fb06f200700 1 --2- 172.21.15.62:0/3361394623 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb0700ff880 0x7fb070195e00 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb06000d8d0 tx=0x7fb06000dc90 comp rx=0 tx=0).stop 2024-04-20T21:39:35.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:35.856+0000 7fb06f200700 1 -- 172.21.15.62:0/3361394623 shutdown_connections 2024-04-20T21:39:35.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:35.856+0000 7fb06f200700 1 --2- 172.21.15.62:0/3361394623 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb05406ecc0 0x7fb054071170 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:39:35.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:35.856+0000 7fb06f200700 1 --2- 172.21.15.62:0/3361394623 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb0700ff880 0x7fb070195e00 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:39:35.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:35.856+0000 7fb06f200700 1 --2- 172.21.15.62:0/3361394623 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb070100ef0 0x7fb070196340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:39:35.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:35.856+0000 7fb06f200700 1 -- 172.21.15.62:0/3361394623 >> 172.21.15.62:0/3361394623 conn(0x7fb070075740 msgr2=0x7fb0701028e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:39:35.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:35.856+0000 7fb06f200700 1 -- 172.21.15.62:0/3361394623 shutdown_connections 2024-04-20T21:39:35.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:35.856+0000 7fb06f200700 1 -- 172.21.15.62:0/3361394623 wait complete. 2024-04-20T21:39:36.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.014+0000 7f6ff3e00700 1 -- 172.21.15.62:0/3317225230 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6ff4106b80 msgr2=0x7f6ff4106ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:39:36.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.014+0000 7f6ff3e00700 1 --2- 172.21.15.62:0/3317225230 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6ff4106b80 0x7f6ff4106ff0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f6fe0009910 tx=0x7f6fe0009c20 comp rx=0 tx=0).stop 2024-04-20T21:39:36.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.014+0000 7f6ff3e00700 1 -- 172.21.15.62:0/3317225230 shutdown_connections 2024-04-20T21:39:36.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.014+0000 7f6ff3e00700 1 --2- 172.21.15.62:0/3317225230 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6ff4106b80 0x7f6ff4106ff0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:39:36.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.014+0000 7f6ff3e00700 1 --2- 172.21.15.62:0/3317225230 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ff410c660 0x7f6ff410ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:39:36.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.014+0000 7f6ff3e00700 1 -- 172.21.15.62:0/3317225230 >> 172.21.15.62:0/3317225230 conn(0x7f6ff4075e90 msgr2=0x7f6ff40782a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:39:36.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.014+0000 7f6ff3e00700 1 -- 172.21.15.62:0/3317225230 shutdown_connections 2024-04-20T21:39:36.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.015+0000 7f6ff3e00700 1 -- 172.21.15.62:0/3317225230 wait complete. 2024-04-20T21:39:36.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.016+0000 7f6ff3e00700 1 Processor -- start 2024-04-20T21:39:36.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.016+0000 7f6ff3e00700 1 -- start start 2024-04-20T21:39:36.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.017+0000 7f6ff3e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ff4106b80 0x7f6ff419c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:39:36.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.017+0000 7f6ff3e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6ff410c660 0x7f6ff419c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:39:36.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.017+0000 7f6ff3e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff419cf60 con 0x7f6ff410c660 2024-04-20T21:39:36.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.017+0000 7f6ff3e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff41a0cf0 con 0x7f6ff4106b80 2024-04-20T21:39:36.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.017+0000 7f6ff2000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6ff410c660 0x7f6ff419c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:39:36.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.017+0000 7f6ff2000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6ff410c660 0x7f6ff419c880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40618/0 (socket says 172.21.15.62:40618) 2024-04-20T21:39:36.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.017+0000 7f6ff2000700 1 -- 172.21.15.62:0/277387662 learned_addr learned my addr 172.21.15.62:0/277387662 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:39:36.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.018+0000 7f6ff2a00700 1 --2- 172.21.15.62:0/277387662 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ff4106b80 0x7f6ff419c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:39:36.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.018+0000 7f6ff2000700 1 -- 172.21.15.62:0/277387662 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ff4106b80 msgr2=0x7f6ff419c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:39:36.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.018+0000 7f6ff2000700 1 --2- 172.21.15.62:0/277387662 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ff4106b80 0x7f6ff419c340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:39:36.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.018+0000 7f6ff2000700 1 -- 172.21.15.62:0/277387662 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fe00095f0 con 0x7f6ff410c660 2024-04-20T21:39:36.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.019+0000 7f6ff2000700 1 --2- 172.21.15.62:0/277387662 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6ff410c660 0x7f6ff419c880 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f6fe0006010 tx=0x7f6fe000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:39:36.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.019+0000 7f6feb400700 1 -- 172.21.15.62:0/277387662 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6fe001d070 con 0x7f6ff410c660 2024-04-20T21:39:36.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.019+0000 7f6ff3e00700 1 -- 172.21.15.62:0/277387662 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ff41a0f70 con 0x7f6ff410c660 2024-04-20T21:39:36.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.019+0000 7f6feb400700 1 -- 172.21.15.62:0/277387662 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6fe0004fb0 con 0x7f6ff410c660 2024-04-20T21:39:36.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.019+0000 7f6feb400700 1 -- 172.21.15.62:0/277387662 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6fe0022ea0 con 0x7f6ff410c660 2024-04-20T21:39:36.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.019+0000 7f6ff3e00700 1 -- 172.21.15.62:0/277387662 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ff41a1460 con 0x7f6ff410c660 2024-04-20T21:39:36.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.021+0000 7f6ff3e00700 1 -- 172.21.15.62:0/277387662 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ff410eb60 con 0x7f6ff410c660 2024-04-20T21:39:36.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.022+0000 7f6feb400700 1 -- 172.21.15.62:0/277387662 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f6fe0022470 con 0x7f6ff410c660 2024-04-20T21:39:36.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.023+0000 7f6feb400700 1 --2- 172.21.15.62:0/277387662 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6fd806ecc0 0x7f6fd8071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:39:36.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.023+0000 7f6feb400700 1 -- 172.21.15.62:0/277387662 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f6fe00909d0 con 0x7f6ff410c660 2024-04-20T21:39:36.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.023+0000 7f6ff2a00700 1 --2- 172.21.15.62:0/277387662 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6fd806ecc0 0x7f6fd8071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:39:36.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.024+0000 7f6ff2a00700 1 --2- 172.21.15.62:0/277387662 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6fd806ecc0 0x7f6fd8071170 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f6fe4006fd0 tx=0x7f6fe4008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:39:36.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.027+0000 7f6feb400700 1 -- 172.21.15.62:0/277387662 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f6fe0059b20 con 0x7f6ff410c660 2024-04-20T21:39:36.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.316+0000 7f6ff3e00700 1 -- 172.21.15.62:0/277387662 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f6ff4062380 con 0x7f6ff410c660 2024-04-20T21:39:36.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.317+0000 7f6feb400700 1 -- 172.21.15.62:0/277387662 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 18 v18) v1 ==== 76+0+1928 (secure 0 0 0) 0x7f6fe00277a0 con 0x7f6ff410c660 2024-04-20T21:39:36.316 INFO:teuthology.orchestra.run.smithi062.stdout:e18 2024-04-20T21:39:36.316 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T21:39:36.317 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T21:39:36.317 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T21:39:36.317 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:39:36.317 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T21:39:36.317 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T21:39:36.317 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 18 2024-04-20T21:39:36.317 INFO:teuthology.orchestra.run.smithi062.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-20T21:39:36.317 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T21:39:36.317 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T21:37:06.018300+0000 2024-04-20T21:39:36.317 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T21:39:36.317 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T21:39:36.317 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T21:39:36.317 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T21:39:36.317 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T21:39:36.317 INFO:teuthology.orchestra.run.smithi062.stdout:max_xattr_size 65536 2024-04-20T21:39:36.318 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T21:39:36.318 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T21:39:36.318 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 0 2024-04-20T21:39:36.318 INFO:teuthology.orchestra.run.smithi062.stdout:snap_rstat 1 2024-04-20T21:39:36.318 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T21:39:36.318 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 2 2024-04-20T21:39:36.318 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1 2024-04-20T21:39:36.318 INFO:teuthology.orchestra.run.smithi062.stdout:up {0=14458,1=24279} 2024-04-20T21:39:36.318 INFO:teuthology.orchestra.run.smithi062.stdout:failed 2024-04-20T21:39:36.318 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T21:39:36.318 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T21:39:36.318 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T21:39:36.318 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T21:39:36.318 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data enabled 2024-04-20T21:39:36.319 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T21:39:36.319 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T21:39:36.319 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 1 2024-04-20T21:39:36.319 INFO:teuthology.orchestra.run.smithi062.stdout:qdb_cluster leader: 14458 members: 24279,14458 2024-04-20T21:39:36.319 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ryqblb{0:14458} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.62:6826/1147127123,v1:172.21.15.62:6827/1147127123] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:39:36.319 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ozizwi{0:14474} state up:standby-replay seq 3 join_fscid=1 addr [v2:172.21.15.62:6828/3494174293,v1:172.21.15.62:6829/3494174293] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:39:36.319 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.dvimer{1:24279} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.189:6824/237262514,v1:172.21.15.189:6825/237262514] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:39:36.319 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.xbgspw{1:24291} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.189:6826/1495356258,v1:172.21.15.189:6827/1495356258] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:39:36.319 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:39:36.319 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:39:36.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.320+0000 7f6ff3e00700 1 -- 172.21.15.62:0/277387662 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6fd806ecc0 msgr2=0x7f6fd8071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:39:36.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.320+0000 7f6ff3e00700 1 --2- 172.21.15.62:0/277387662 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6fd806ecc0 0x7f6fd8071170 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f6fe4006fd0 tx=0x7f6fe4008040 comp rx=0 tx=0).stop 2024-04-20T21:39:36.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.320+0000 7f6ff3e00700 1 -- 172.21.15.62:0/277387662 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6ff410c660 msgr2=0x7f6ff419c880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:39:36.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.320+0000 7f6ff3e00700 1 --2- 172.21.15.62:0/277387662 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6ff410c660 0x7f6ff419c880 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f6fe0006010 tx=0x7f6fe000be20 comp rx=0 tx=0).stop 2024-04-20T21:39:36.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.320+0000 7f6ff3e00700 1 -- 172.21.15.62:0/277387662 shutdown_connections 2024-04-20T21:39:36.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.320+0000 7f6ff3e00700 1 --2- 172.21.15.62:0/277387662 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6fd806ecc0 0x7f6fd8071170 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:39:36.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.320+0000 7f6ff3e00700 1 --2- 172.21.15.62:0/277387662 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ff4106b80 0x7f6ff419c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:39:36.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.320+0000 7f6ff3e00700 1 --2- 172.21.15.62:0/277387662 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6ff410c660 0x7f6ff419c880 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:39:36.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.320+0000 7f6ff3e00700 1 -- 172.21.15.62:0/277387662 >> 172.21.15.62:0/277387662 conn(0x7f6ff4075e90 msgr2=0x7f6ff4077b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:39:36.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.320+0000 7f6ff3e00700 1 -- 172.21.15.62:0/277387662 shutdown_connections 2024-04-20T21:39:36.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.320+0000 7f6ff3e00700 1 -- 172.21.15.62:0/277387662 wait complete. 2024-04-20T21:39:36.321 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 18 2024-04-20T21:39:36.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.482+0000 7f8d2d200700 1 -- 172.21.15.62:0/343880682 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d280730f0 msgr2=0x7f8d280734c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:39:36.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.482+0000 7f8d2d200700 1 --2- 172.21.15.62:0/343880682 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d280730f0 0x7f8d280734c0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f8d18009910 tx=0x7f8d18009c20 comp rx=0 tx=0).stop 2024-04-20T21:39:36.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.482+0000 7f8d2d200700 1 -- 172.21.15.62:0/343880682 shutdown_connections 2024-04-20T21:39:36.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.482+0000 7f8d2d200700 1 --2- 172.21.15.62:0/343880682 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d28073a00 0x7f8d2810c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:39:36.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.482+0000 7f8d2d200700 1 --2- 172.21.15.62:0/343880682 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d280730f0 0x7f8d280734c0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:39:36.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.482+0000 7f8d2d200700 1 -- 172.21.15.62:0/343880682 >> 172.21.15.62:0/343880682 conn(0x7f8d280fbda0 msgr2=0x7f8d280fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:39:36.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.482+0000 7f8d2d200700 1 -- 172.21.15.62:0/343880682 shutdown_connections 2024-04-20T21:39:36.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.482+0000 7f8d2d200700 1 -- 172.21.15.62:0/343880682 wait complete. 2024-04-20T21:39:36.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.484+0000 7f8d2d200700 1 Processor -- start 2024-04-20T21:39:36.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.484+0000 7f8d2d200700 1 -- start start 2024-04-20T21:39:36.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.484+0000 7f8d2d200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d280730f0 0x7f8d28197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:39:36.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.484+0000 7f8d2d200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d28073a00 0x7f8d28198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:39:36.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.485+0000 7f8d2d200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d2819c970 con 0x7f8d280730f0 2024-04-20T21:39:36.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.485+0000 7f8d2d200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d2819cae0 con 0x7f8d28073a00 2024-04-20T21:39:36.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.485+0000 7f8d25e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d280730f0 0x7f8d28197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:39:36.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.485+0000 7f8d25e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d280730f0 0x7f8d28197fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40636/0 (socket says 172.21.15.62:40636) 2024-04-20T21:39:36.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.485+0000 7f8d25e00700 1 -- 172.21.15.62:0/4215949651 learned_addr learned my addr 172.21.15.62:0/4215949651 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:39:36.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.485+0000 7f8d1d400700 1 --2- 172.21.15.62:0/4215949651 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d28073a00 0x7f8d28198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:39:36.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.485+0000 7f8d25e00700 1 -- 172.21.15.62:0/4215949651 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d28073a00 msgr2=0x7f8d28198630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:39:36.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.485+0000 7f8d25e00700 1 --2- 172.21.15.62:0/4215949651 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d28073a00 0x7f8d28198630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:39:36.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.485+0000 7f8d25e00700 1 -- 172.21.15.62:0/4215949651 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d180095f0 con 0x7f8d280730f0 2024-04-20T21:39:36.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.486+0000 7f8d25e00700 1 --2- 172.21.15.62:0/4215949651 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d280730f0 0x7f8d28197fc0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f8d1800b5c0 tx=0x7f8d1800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:39:36.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.486+0000 7f8d1f400700 1 -- 172.21.15.62:0/4215949651 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8d1801d070 con 0x7f8d280730f0 2024-04-20T21:39:36.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.486+0000 7f8d1f400700 1 -- 172.21.15.62:0/4215949651 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8d18005e00 con 0x7f8d280730f0 2024-04-20T21:39:36.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.486+0000 7f8d1f400700 1 -- 172.21.15.62:0/4215949651 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8d18017400 con 0x7f8d280730f0 2024-04-20T21:39:36.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.486+0000 7f8d2d200700 1 -- 172.21.15.62:0/4215949651 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d2819cdc0 con 0x7f8d280730f0 2024-04-20T21:39:36.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.486+0000 7f8d2d200700 1 -- 172.21.15.62:0/4215949651 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d2819d310 con 0x7f8d280730f0 2024-04-20T21:39:36.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.487+0000 7f8d2d200700 1 -- 172.21.15.62:0/4215949651 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d28109c80 con 0x7f8d280730f0 2024-04-20T21:39:36.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.488+0000 7f8d1f400700 1 -- 172.21.15.62:0/4215949651 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f8d1800f460 con 0x7f8d280730f0 2024-04-20T21:39:36.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.489+0000 7f8d1f400700 1 --2- 172.21.15.62:0/4215949651 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f8d0c06ecc0 0x7f8d0c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:39:36.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.489+0000 7f8d1f400700 1 -- 172.21.15.62:0/4215949651 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f8d18090260 con 0x7f8d280730f0 2024-04-20T21:39:36.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.490+0000 7f8d1d400700 1 --2- 172.21.15.62:0/4215949651 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f8d0c06ecc0 0x7f8d0c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:39:36.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.490+0000 7f8d1d400700 1 --2- 172.21.15.62:0/4215949651 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f8d0c06ecc0 0x7f8d0c071170 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8d10005950 tx=0x7f8d100058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:39:36.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.494+0000 7f8d1f400700 1 -- 172.21.15.62:0/4215949651 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f8d180590a0 con 0x7f8d280730f0 2024-04-20T21:39:36.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:36 smithi189 ceph-mon[57022]: from='client.34286 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:39:36.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:36 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/3361394623' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:39:36.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:36 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/277387662' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T21:39:36.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.756+0000 7f8d2d200700 1 -- 172.21.15.62:0/4215949651 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8d281993e0 con 0x7f8d0c06ecc0 2024-04-20T21:39:36.755 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:36 smithi062 ceph-mon[76322]: from='client.34286 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:39:36.755 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:36 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/3361394623' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:39:36.755 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:36 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/277387662' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T21:39:36.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.759+0000 7f8d1f400700 1 -- 172.21.15.62:0/4215949651 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f8d281993e0 con 0x7f8d0c06ecc0 2024-04-20T21:39:36.758 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:39:36.758 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T21:39:36.758 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T21:39:36.758 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-20T21:39:36.758 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T21:39:36.759 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr", 2024-04-20T21:39:36.759 INFO:teuthology.orchestra.run.smithi062.stdout: "crash", 2024-04-20T21:39:36.759 INFO:teuthology.orchestra.run.smithi062.stdout: "mon" 2024-04-20T21:39:36.759 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T21:39:36.759 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "8/23 daemons upgraded", 2024-04-20T21:39:36.759 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading osd daemons", 2024-04-20T21:39:36.759 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T21:39:36.759 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:39:36.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.762+0000 7f8d2d200700 1 -- 172.21.15.62:0/4215949651 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f8d0c06ecc0 msgr2=0x7f8d0c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:39:36.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.762+0000 7f8d2d200700 1 --2- 172.21.15.62:0/4215949651 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f8d0c06ecc0 0x7f8d0c071170 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8d10005950 tx=0x7f8d100058e0 comp rx=0 tx=0).stop 2024-04-20T21:39:36.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.762+0000 7f8d2d200700 1 -- 172.21.15.62:0/4215949651 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d280730f0 msgr2=0x7f8d28197fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:39:36.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.762+0000 7f8d2d200700 1 --2- 172.21.15.62:0/4215949651 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d280730f0 0x7f8d28197fc0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f8d1800b5c0 tx=0x7f8d1800be20 comp rx=0 tx=0).stop 2024-04-20T21:39:36.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.762+0000 7f8d2d200700 1 -- 172.21.15.62:0/4215949651 shutdown_connections 2024-04-20T21:39:36.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.762+0000 7f8d2d200700 1 --2- 172.21.15.62:0/4215949651 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f8d0c06ecc0 0x7f8d0c071170 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:39:36.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.762+0000 7f8d2d200700 1 --2- 172.21.15.62:0/4215949651 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d280730f0 0x7f8d28197fc0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:39:36.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.762+0000 7f8d2d200700 1 --2- 172.21.15.62:0/4215949651 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d28073a00 0x7f8d28198630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:39:36.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.762+0000 7f8d2d200700 1 -- 172.21.15.62:0/4215949651 >> 172.21.15.62:0/4215949651 conn(0x7f8d280fbda0 msgr2=0x7f8d28106dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:39:36.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.763+0000 7f8d2d200700 1 -- 172.21.15.62:0/4215949651 shutdown_connections 2024-04-20T21:39:36.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.763+0000 7f8d2d200700 1 -- 172.21.15.62:0/4215949651 wait complete. 2024-04-20T21:39:36.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.917+0000 7f280e200700 1 -- 172.21.15.62:0/2868464054 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f28080730f0 msgr2=0x7f28080734c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:39:36.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.917+0000 7f280e200700 1 --2- 172.21.15.62:0/2868464054 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f28080730f0 0x7f28080734c0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f27f8009910 tx=0x7f27f8009c20 comp rx=0 tx=0).stop 2024-04-20T21:39:36.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.917+0000 7f280e200700 1 -- 172.21.15.62:0/2868464054 shutdown_connections 2024-04-20T21:39:36.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.917+0000 7f280e200700 1 --2- 172.21.15.62:0/2868464054 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2808073a00 0x7f280810c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:39:36.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.917+0000 7f280e200700 1 --2- 172.21.15.62:0/2868464054 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f28080730f0 0x7f28080734c0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:39:36.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.917+0000 7f280e200700 1 -- 172.21.15.62:0/2868464054 >> 172.21.15.62:0/2868464054 conn(0x7f28080fbda0 msgr2=0x7f28080fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:39:36.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.918+0000 7f280e200700 1 -- 172.21.15.62:0/2868464054 shutdown_connections 2024-04-20T21:39:36.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.918+0000 7f280e200700 1 -- 172.21.15.62:0/2868464054 wait complete. 2024-04-20T21:39:36.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.919+0000 7f280e200700 1 Processor -- start 2024-04-20T21:39:36.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.919+0000 7f280e200700 1 -- start start 2024-04-20T21:39:36.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.919+0000 7f280e200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f28080730f0 0x7f2808197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:39:36.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.919+0000 7f280e200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2808073a00 0x7f2808198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:39:36.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.919+0000 7f280e200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f280819c970 con 0x7f28080730f0 2024-04-20T21:39:36.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.919+0000 7f280e200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f280819cae0 con 0x7f2808073a00 2024-04-20T21:39:36.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.919+0000 7f2807400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f28080730f0 0x7f2808197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:39:36.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.919+0000 7f2807400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f28080730f0 0x7f2808197fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40662/0 (socket says 172.21.15.62:40662) 2024-04-20T21:39:36.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.919+0000 7f2807400700 1 -- 172.21.15.62:0/844951835 learned_addr learned my addr 172.21.15.62:0/844951835 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:39:36.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.920+0000 7f2806a00700 1 --2- 172.21.15.62:0/844951835 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2808073a00 0x7f2808198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:39:36.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.920+0000 7f2807400700 1 -- 172.21.15.62:0/844951835 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2808073a00 msgr2=0x7f2808198630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:39:36.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.920+0000 7f2807400700 1 --2- 172.21.15.62:0/844951835 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2808073a00 0x7f2808198630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:39:36.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.920+0000 7f2807400700 1 -- 172.21.15.62:0/844951835 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27f80095f0 con 0x7f28080730f0 2024-04-20T21:39:36.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.920+0000 7f2807400700 1 --2- 172.21.15.62:0/844951835 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f28080730f0 0x7f2808197fc0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f27f800b5c0 tx=0x7f27f800be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:39:36.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.920+0000 7f27ffe00700 1 -- 172.21.15.62:0/844951835 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f27f801d070 con 0x7f28080730f0 2024-04-20T21:39:36.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.921+0000 7f27ffe00700 1 -- 172.21.15.62:0/844951835 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f27f800f460 con 0x7f28080730f0 2024-04-20T21:39:36.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.921+0000 7f280e200700 1 -- 172.21.15.62:0/844951835 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f280819cdc0 con 0x7f28080730f0 2024-04-20T21:39:36.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.921+0000 7f27ffe00700 1 -- 172.21.15.62:0/844951835 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f27f8017630 con 0x7f28080730f0 2024-04-20T21:39:36.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.921+0000 7f280e200700 1 -- 172.21.15.62:0/844951835 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f280819d310 con 0x7f28080730f0 2024-04-20T21:39:36.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.922+0000 7f280e200700 1 -- 172.21.15.62:0/844951835 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2808109c80 con 0x7f28080730f0 2024-04-20T21:39:36.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.923+0000 7f27ffe00700 1 -- 172.21.15.62:0/844951835 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f27f800f5d0 con 0x7f28080730f0 2024-04-20T21:39:36.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.924+0000 7f27ffe00700 1 --2- 172.21.15.62:0/844951835 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f27ec06ecc0 0x7f27ec071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:39:36.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.924+0000 7f27ffe00700 1 -- 172.21.15.62:0/844951835 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f27f80903e0 con 0x7f28080730f0 2024-04-20T21:39:36.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.924+0000 7f2806a00700 1 --2- 172.21.15.62:0/844951835 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f27ec06ecc0 0x7f27ec071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:39:36.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.925+0000 7f2806a00700 1 --2- 172.21.15.62:0/844951835 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f27ec06ecc0 0x7f27ec071170 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f27f4005fd0 tx=0x7f27f400a560 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:39:36.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:36.928+0000 7f27ffe00700 1 -- 172.21.15.62:0/844951835 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f27f80591a0 con 0x7f28080730f0 2024-04-20T21:39:37.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:37.259+0000 7f280e200700 1 -- 172.21.15.62:0/844951835 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f2808199320 con 0x7f28080730f0 2024-04-20T21:39:37.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:37.260+0000 7f27ffe00700 1 -- 172.21.15.62:0/844951835 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7f27f805c860 con 0x7f28080730f0 2024-04-20T21:39:37.259 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-04-20T21:39:37.259 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T21:39:37.259 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T21:39:37.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:37.263+0000 7f280e200700 1 -- 172.21.15.62:0/844951835 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f27ec06ecc0 msgr2=0x7f27ec071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:39:37.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:37.263+0000 7f280e200700 1 --2- 172.21.15.62:0/844951835 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f27ec06ecc0 0x7f27ec071170 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f27f4005fd0 tx=0x7f27f400a560 comp rx=0 tx=0).stop 2024-04-20T21:39:37.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:37.263+0000 7f280e200700 1 -- 172.21.15.62:0/844951835 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f28080730f0 msgr2=0x7f2808197fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:39:37.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:37.263+0000 7f280e200700 1 --2- 172.21.15.62:0/844951835 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f28080730f0 0x7f2808197fc0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f27f800b5c0 tx=0x7f27f800be00 comp rx=0 tx=0).stop 2024-04-20T21:39:37.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:37.263+0000 7f280e200700 1 -- 172.21.15.62:0/844951835 shutdown_connections 2024-04-20T21:39:37.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:37.263+0000 7f280e200700 1 --2- 172.21.15.62:0/844951835 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f27ec06ecc0 0x7f27ec071170 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:39:37.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:37.264+0000 7f280e200700 1 --2- 172.21.15.62:0/844951835 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f28080730f0 0x7f2808197fc0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:39:37.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:37.264+0000 7f280e200700 1 --2- 172.21.15.62:0/844951835 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2808073a00 0x7f2808198630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:39:37.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:37.264+0000 7f280e200700 1 -- 172.21.15.62:0/844951835 >> 172.21.15.62:0/844951835 conn(0x7f28080fbda0 msgr2=0x7f2808106dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:39:37.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:37.264+0000 7f280e200700 1 -- 172.21.15.62:0/844951835 shutdown_connections 2024-04-20T21:39:37.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:39:37.264+0000 7f280e200700 1 -- 172.21.15.62:0/844951835 wait complete. 2024-04-20T21:39:37.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:37 smithi189 ceph-mon[57022]: pgmap v104: 65 pgs: 65 active+clean; 315 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:39:37.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:37 smithi189 ceph-mon[57022]: from='client.34298 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:39:37.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:37 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/844951835' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T21:39:37.753 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:37 smithi062 ceph-mon[76322]: pgmap v104: 65 pgs: 65 active+clean; 315 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:39:37.753 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:37 smithi062 ceph-mon[76322]: from='client.34298 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:39:37.753 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:37 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/844951835' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T21:39:39.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:39 smithi062 ceph-mon[76322]: pgmap v105: 65 pgs: 65 active+clean; 315 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-20T21:39:39.536 INFO:journalctl@ceph.osd.2.smithi062.stdout:Apr 20 21:39:39 smithi062 systemd[1]: Stopping Ceph osd.2 for 61ae3628-ff5b-11ee-bc93-c7b262605968... 2024-04-20T21:39:39.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:39 smithi189 ceph-mon[57022]: pgmap v105: 65 pgs: 65 active+clean; 315 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-20T21:39:40.036 INFO:journalctl@ceph.osd.2.smithi062.stdout:Apr 20 21:39:39 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-2[42153]: 2024-04-20T21:39:39.763+0000 7f9cabe00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-20T21:39:40.036 INFO:journalctl@ceph.osd.2.smithi062.stdout:Apr 20 21:39:39 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-2[42153]: 2024-04-20T21:39:39.763+0000 7f9cabe00700 -1 osd.2 58 *** Got signal Terminated *** 2024-04-20T21:39:40.036 INFO:journalctl@ceph.osd.2.smithi062.stdout:Apr 20 21:39:39 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-2[42153]: 2024-04-20T21:39:39.763+0000 7f9cabe00700 -1 osd.2 58 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-20T21:39:40.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:40 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:39:40.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:40 smithi189 ceph-mon[57022]: osd.2 marked itself down and dead 2024-04-20T21:39:40.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:40 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:39:40.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:40 smithi062 ceph-mon[76322]: osd.2 marked itself down and dead 2024-04-20T21:39:41.094 INFO:journalctl@ceph.osd.2.smithi062.stdout:Apr 20 21:39:40 smithi062 podman[98621]: 2024-04-20 21:39:40.832070378 +0000 UTC m=+1.192363719 container died 31c5d9586e98d11e98c6a0883c09238f2bc28119261f5b6cd1ee6202b0e1c2cb (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-2, org.label-schema.build-date=20231212, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-18.2.0, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, RELEASE=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True) 2024-04-20T21:39:41.423 INFO:journalctl@ceph.osd.2.smithi062.stdout:Apr 20 21:39:41 smithi062 podman[98621]: 2024-04-20 21:39:41.239584374 +0000 UTC m=+1.599877753 container cleanup 31c5d9586e98d11e98c6a0883c09238f2bc28119261f5b6cd1ee6202b0e1c2cb (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-18.2.0, io.buildah.version=1.29.1, org.label-schema.build-date=20231212, org.label-schema.license=GPLv2, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-04-20T21:39:41.423 INFO:journalctl@ceph.osd.2.smithi062.stdout:Apr 20 21:39:41 smithi062 bash[98621]: ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-2 2024-04-20T21:39:41.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:41 smithi189 ceph-mon[57022]: pgmap v106: 65 pgs: 65 active+clean; 315 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-20T21:39:41.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:41 smithi189 ceph-mon[57022]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-20T21:39:41.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:41 smithi189 ceph-mon[57022]: osdmap e59: 6 total, 5 up, 6 in 2024-04-20T21:39:41.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:41 smithi062 ceph-mon[76322]: pgmap v106: 65 pgs: 65 active+clean; 315 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-20T21:39:41.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:41 smithi062 ceph-mon[76322]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-20T21:39:41.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:41 smithi062 ceph-mon[76322]: osdmap e59: 6 total, 5 up, 6 in 2024-04-20T21:39:41.785 INFO:journalctl@ceph.osd.2.smithi062.stdout:Apr 20 21:39:41 smithi062 podman[98640]: 2024-04-20 21:39:41.44884791 +0000 UTC m=+0.600245524 container remove 31c5d9586e98d11e98c6a0883c09238f2bc28119261f5b6cd1ee6202b0e1c2cb (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-2, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20231212, GIT_CLEAN=True, RELEASE=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1) 2024-04-20T21:39:42.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:42 smithi062 ceph-mon[76322]: osdmap e60: 6 total, 5 up, 6 in 2024-04-20T21:39:42.535 INFO:journalctl@ceph.osd.2.smithi062.stdout:Apr 20 21:39:42 smithi062 podman[98727]: 2024-04-20 21:39:42.141504432 +0000 UTC m=+0.036180625 image pull 76bc6b91b726a03b8b736eeeb35fc1ffd1469e23c7d60f5b444a217528d8906d quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 2024-04-20T21:39:42.536 INFO:journalctl@ceph.osd.2.smithi062.stdout:Apr 20 21:39:42 smithi062 podman[98727]: 2024-04-20 21:39:42.291701287 +0000 UTC m=+0.186377541 container create 1e1fec9ea65830812d212a2fc60ced243d15a497aae83e667896feb6798c007e (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, maintainer=Guillaume Abrioux ) 2024-04-20T21:39:42.722 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:42 smithi189 ceph-mon[57022]: osdmap e60: 6 total, 5 up, 6 in 2024-04-20T21:39:43.035 INFO:journalctl@ceph.osd.2.smithi062.stdout:Apr 20 21:39:42 smithi062 podman[98727]: 2024-04-20 21:39:42.566558976 +0000 UTC m=+0.461235213 container init 1e1fec9ea65830812d212a2fc60ced243d15a497aae83e667896feb6798c007e (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-2-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-04-20T21:39:43.036 INFO:journalctl@ceph.osd.2.smithi062.stdout:Apr 20 21:39:42 smithi062 podman[98727]: 2024-04-20 21:39:42.578187251 +0000 UTC m=+0.472863501 container start 1e1fec9ea65830812d212a2fc60ced243d15a497aae83e667896feb6798c007e (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-04-20T21:39:43.036 INFO:journalctl@ceph.osd.2.smithi062.stdout:Apr 20 21:39:42 smithi062 podman[98727]: 2024-04-20 21:39:42.678149518 +0000 UTC m=+0.572825788 container attach 1e1fec9ea65830812d212a2fc60ced243d15a497aae83e667896feb6798c007e (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-2-deactivate, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-04-20T21:39:43.354 INFO:journalctl@ceph.osd.2.smithi062.stdout:Apr 20 21:39:43 smithi062 podman[98727]: 2024-04-20 21:39:43.046311179 +0000 UTC m=+0.940987419 container died 1e1fec9ea65830812d212a2fc60ced243d15a497aae83e667896feb6798c007e (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-2-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, GIT_CLEAN=True) 2024-04-20T21:39:43.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:43 smithi189 ceph-mon[57022]: pgmap v109: 65 pgs: 9 stale+active+clean, 56 active+clean; 315 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:39:43.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:43 smithi062 ceph-mon[76322]: pgmap v109: 65 pgs: 9 stale+active+clean, 56 active+clean; 315 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T21:39:44.251 INFO:journalctl@ceph.osd.2.smithi062.stdout:Apr 20 21:39:43 smithi062 podman[98752]: 2024-04-20 21:39:43.82419264 +0000 UTC m=+0.757198238 container remove 1e1fec9ea65830812d212a2fc60ced243d15a497aae83e667896feb6798c007e (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-2-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux ) 2024-04-20T21:39:44.252 INFO:journalctl@ceph.osd.2.smithi062.stdout:Apr 20 21:39:43 smithi062 systemd[1]: ceph-61ae3628-ff5b-11ee-bc93-c7b262605968@osd.2.service: Deactivated successfully. 2024-04-20T21:39:44.252 INFO:journalctl@ceph.osd.2.smithi062.stdout:Apr 20 21:39:43 smithi062 systemd[1]: Stopped Ceph osd.2 for 61ae3628-ff5b-11ee-bc93-c7b262605968. 2024-04-20T21:39:44.252 INFO:journalctl@ceph.osd.2.smithi062.stdout:Apr 20 21:39:43 smithi062 systemd[1]: ceph-61ae3628-ff5b-11ee-bc93-c7b262605968@osd.2.service: Consumed 1min 26.443s CPU time. 2024-04-20T21:39:44.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:44 smithi062 ceph-mon[76322]: Health check failed: Degraded data redundancy: 13/366 objects degraded (3.552%), 3 pgs degraded (PG_DEGRADED) 2024-04-20T21:39:44.535 INFO:journalctl@ceph.osd.2.smithi062.stdout:Apr 20 21:39:44 smithi062 systemd[1]: Starting Ceph osd.2 for 61ae3628-ff5b-11ee-bc93-c7b262605968... 2024-04-20T21:39:44.880 INFO:journalctl@ceph.osd.2.smithi062.stdout:Apr 20 21:39:44 smithi062 podman[98849]: 2024-04-20 21:39:44.519954391 +0000 UTC m=+0.033913899 image pull 76bc6b91b726a03b8b736eeeb35fc1ffd1469e23c7d60f5b444a217528d8906d quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 2024-04-20T21:39:44.880 INFO:journalctl@ceph.osd.2.smithi062.stdout:Apr 20 21:39:44 smithi062 podman[98849]: 2024-04-20 21:39:44.700475818 +0000 UTC m=+0.214435349 container create 98cde5267f272c06ff4bf0754f1aab11838be5f920c87222fd93b8bd1966613f (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-2-activate, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-04-20T21:39:44.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:44 smithi189 ceph-mon[57022]: Health check failed: Degraded data redundancy: 13/366 objects degraded (3.552%), 3 pgs degraded (PG_DEGRADED) 2024-04-20T21:39:45.285 INFO:journalctl@ceph.osd.2.smithi062.stdout:Apr 20 21:39:45 smithi062 podman[98849]: 2024-04-20 21:39:45.021694011 +0000 UTC m=+0.535653585 container init 98cde5267f272c06ff4bf0754f1aab11838be5f920c87222fd93b8bd1966613f (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-2-activate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_CLEAN=True) 2024-04-20T21:39:45.286 INFO:journalctl@ceph.osd.2.smithi062.stdout:Apr 20 21:39:45 smithi062 podman[98849]: 2024-04-20 21:39:45.034826556 +0000 UTC m=+0.548786064 container start 98cde5267f272c06ff4bf0754f1aab11838be5f920c87222fd93b8bd1966613f (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-2-activate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-04-20T21:39:45.286 INFO:journalctl@ceph.osd.2.smithi062.stdout:Apr 20 21:39:45 smithi062 podman[98849]: 2024-04-20 21:39:45.11985796 +0000 UTC m=+0.633817544 container attach 98cde5267f272c06ff4bf0754f1aab11838be5f920c87222fd93b8bd1966613f (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-2-activate, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416) 2024-04-20T21:39:45.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:45 smithi062 ceph-mon[76322]: pgmap v110: 65 pgs: 3 active+undersized+degraded, 5 stale+active+clean, 4 active+undersized, 53 active+clean; 315 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 13/366 objects degraded (3.552%) 2024-04-20T21:39:45.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:45 smithi189 ceph-mon[57022]: pgmap v110: 65 pgs: 3 active+undersized+degraded, 5 stale+active+clean, 4 active+undersized, 53 active+clean; 315 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 13/366 objects degraded (3.552%) 2024-04-20T21:39:46.288 INFO:journalctl@ceph.osd.2.smithi062.stdout:Apr 20 21:39:46 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-2-activate[98863]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-20T21:39:46.288 INFO:journalctl@ceph.osd.2.smithi062.stdout:Apr 20 21:39:46 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-2-activate[98863]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-20T21:39:46.288 INFO:journalctl@ceph.osd.2.smithi062.stdout:Apr 20 21:39:46 smithi062 bash[98849]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-20T21:39:46.288 INFO:journalctl@ceph.osd.2.smithi062.stdout:Apr 20 21:39:46 smithi062 bash[98849]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-20T21:39:46.288 INFO:journalctl@ceph.osd.2.smithi062.stdout:Apr 20 21:39:46 smithi062 bash[98849]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-20T21:39:46.288 INFO:journalctl@ceph.osd.2.smithi062.stdout:Apr 20 21:39:46 smithi062 bash[98849]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-04-20T21:39:46.288 INFO:journalctl@ceph.osd.2.smithi062.stdout:Apr 20 21:39:46 smithi062 bash[98849]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-04-20T21:39:46.288 INFO:journalctl@ceph.osd.2.smithi062.stdout:Apr 20 21:39:46 smithi062 bash[98849]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-04-20T21:39:46.288 INFO:journalctl@ceph.osd.2.smithi062.stdout:Apr 20 21:39:46 smithi062 bash[98849]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-04-20T21:39:46.288 INFO:journalctl@ceph.osd.2.smithi062.stdout:Apr 20 21:39:46 smithi062 bash[98849]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-20T21:39:46.288 INFO:journalctl@ceph.osd.2.smithi062.stdout:Apr 20 21:39:46 smithi062 bash[98849]: --> ceph-volume raw activate successful for osd ID: 2 2024-04-20T21:39:46.288 INFO:journalctl@ceph.osd.2.smithi062.stdout:Apr 20 21:39:46 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-2-activate[98863]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-20T21:39:46.289 INFO:journalctl@ceph.osd.2.smithi062.stdout:Apr 20 21:39:46 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-2-activate[98863]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-04-20T21:39:46.289 INFO:journalctl@ceph.osd.2.smithi062.stdout:Apr 20 21:39:46 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-2-activate[98863]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-04-20T21:39:46.289 INFO:journalctl@ceph.osd.2.smithi062.stdout:Apr 20 21:39:46 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-2-activate[98863]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-04-20T21:39:46.289 INFO:journalctl@ceph.osd.2.smithi062.stdout:Apr 20 21:39:46 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-2-activate[98863]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-04-20T21:39:46.289 INFO:journalctl@ceph.osd.2.smithi062.stdout:Apr 20 21:39:46 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-2-activate[98863]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-20T21:39:46.289 INFO:journalctl@ceph.osd.2.smithi062.stdout:Apr 20 21:39:46 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-2-activate[98863]: --> ceph-volume raw activate successful for osd ID: 2 2024-04-20T21:39:46.289 INFO:journalctl@ceph.osd.2.smithi062.stdout:Apr 20 21:39:46 smithi062 podman[98849]: 2024-04-20 21:39:46.096528666 +0000 UTC m=+1.610488190 container died 98cde5267f272c06ff4bf0754f1aab11838be5f920c87222fd93b8bd1966613f (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-2-activate, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, GIT_BRANCH=HEAD, ceph=True) 2024-04-20T21:39:47.280 INFO:journalctl@ceph.osd.2.smithi062.stdout:Apr 20 21:39:46 smithi062 podman[98997]: 2024-04-20 21:39:46.916263497 +0000 UTC m=+0.793500076 container remove 98cde5267f272c06ff4bf0754f1aab11838be5f920c87222fd93b8bd1966613f (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-2-activate, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-04-20T21:39:47.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:47 smithi062 ceph-mon[76322]: pgmap v111: 65 pgs: 13 active+undersized+degraded, 14 active+undersized, 38 active+clean; 315 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 44/366 objects degraded (12.022%) 2024-04-20T21:39:47.535 INFO:journalctl@ceph.osd.2.smithi062.stdout:Apr 20 21:39:47 smithi062 podman[99059]: 2024-04-20 21:39:47.187377268 +0000 UTC m=+0.033478067 image pull 76bc6b91b726a03b8b736eeeb35fc1ffd1469e23c7d60f5b444a217528d8906d quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 2024-04-20T21:39:47.535 INFO:journalctl@ceph.osd.2.smithi062.stdout:Apr 20 21:39:47 smithi062 podman[99059]: 2024-04-20 21:39:47.317168288 +0000 UTC m=+0.163269123 container create 2d4f3e38dc0e10db7d042a56a546f30d4579ae68724f5b36ef4fce943f2b08a3 (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, org.label-schema.build-date=20240416) 2024-04-20T21:39:47.877 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:47 smithi189 ceph-mon[57022]: pgmap v111: 65 pgs: 13 active+undersized+degraded, 14 active+undersized, 38 active+clean; 315 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 44/366 objects degraded (12.022%) 2024-04-20T21:39:48.035 INFO:journalctl@ceph.osd.2.smithi062.stdout:Apr 20 21:39:47 smithi062 podman[99059]: 2024-04-20 21:39:47.662818875 +0000 UTC m=+0.508919714 container init 2d4f3e38dc0e10db7d042a56a546f30d4579ae68724f5b36ef4fce943f2b08a3 (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-2, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-04-20T21:39:48.035 INFO:journalctl@ceph.osd.2.smithi062.stdout:Apr 20 21:39:47 smithi062 podman[99059]: 2024-04-20 21:39:47.675364947 +0000 UTC m=+0.521465782 container start 2d4f3e38dc0e10db7d042a56a546f30d4579ae68724f5b36ef4fce943f2b08a3 (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-2, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=) 2024-04-20T21:39:48.035 INFO:journalctl@ceph.osd.2.smithi062.stdout:Apr 20 21:39:47 smithi062 ceph-osd[99078]: -- 172.21.15.62:0/2788729082 <== mon.0 v2:172.21.15.62:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b16817f680 con 0x55b168148c00 2024-04-20T21:39:48.035 INFO:journalctl@ceph.osd.2.smithi062.stdout:Apr 20 21:39:47 smithi062 bash[99059]: 2d4f3e38dc0e10db7d042a56a546f30d4579ae68724f5b36ef4fce943f2b08a3 2024-04-20T21:39:48.035 INFO:journalctl@ceph.osd.2.smithi062.stdout:Apr 20 21:39:47 smithi062 systemd[1]: Started Ceph osd.2 for 61ae3628-ff5b-11ee-bc93-c7b262605968. 2024-04-20T21:39:48.535 INFO:journalctl@ceph.osd.2.smithi062.stdout:Apr 20 21:39:48 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-2[99074]: 2024-04-20T21:39:48.255+0000 7fdc17bb97c0 -1 Falling back to public interface 2024-04-20T21:39:49.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:48 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:39:49.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:48 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:39:49.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:48 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:39:49.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:48 smithi189 ceph-mon[57022]: pgmap v112: 65 pgs: 13 active+undersized+degraded, 14 active+undersized, 38 active+clean; 315 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 44/366 objects degraded (12.022%) 2024-04-20T21:39:49.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:48 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:39:49.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:48 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:39:49.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:48 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:39:49.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:48 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:39:49.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:48 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:39:49.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:48 smithi062 ceph-mon[76322]: pgmap v112: 65 pgs: 13 active+undersized+degraded, 14 active+undersized, 38 active+clean; 315 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 44/366 objects degraded (12.022%) 2024-04-20T21:39:49.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:48 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:39:49.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:48 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:39:50.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:50 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:39:50.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:50 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:39:50.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:50 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:39:50.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:50 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:39:51.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:51 smithi062 ceph-mon[76322]: pgmap v113: 65 pgs: 13 active+undersized+degraded, 14 active+undersized, 38 active+clean; 315 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 44/366 objects degraded (12.022%) 2024-04-20T21:39:51.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:51 smithi189 ceph-mon[57022]: pgmap v113: 65 pgs: 13 active+undersized+degraded, 14 active+undersized, 38 active+clean; 315 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 44/366 objects degraded (12.022%) 2024-04-20T21:39:53.193 INFO:journalctl@ceph.osd.2.smithi062.stdout:Apr 20 21:39:52 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-2[99074]: 2024-04-20T21:39:52.858+0000 7fdc17bb97c0 -1 osd.2 0 read_superblock omap replica is missing. 2024-04-20T21:39:53.193 INFO:journalctl@ceph.osd.2.smithi062.stdout:Apr 20 21:39:53 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-2[99074]: 2024-04-20T21:39:53.195+0000 7fdc17bb97c0 -1 osd.2 58 log_to_monitors true 2024-04-20T21:39:53.535 INFO:journalctl@ceph.osd.2.smithi062.stdout:Apr 20 21:39:53 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-2[99074]: 2024-04-20T21:39:53.416+0000 7fdc07000700 -1 osd.2 58 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-20T21:39:53.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:53 smithi062 ceph-mon[76322]: pgmap v114: 65 pgs: 13 active+undersized+degraded, 14 active+undersized, 38 active+clean; 315 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 44/366 objects degraded (12.022%) 2024-04-20T21:39:53.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:53 smithi062 ceph-mon[76322]: from='osd.2 [v2:172.21.15.62:6818/4264123649,v1:172.21.15.62:6819/4264123649]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-20T21:39:53.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:53 smithi189 ceph-mon[57022]: pgmap v114: 65 pgs: 13 active+undersized+degraded, 14 active+undersized, 38 active+clean; 315 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 44/366 objects degraded (12.022%) 2024-04-20T21:39:53.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:53 smithi189 ceph-mon[57022]: from='osd.2 [v2:172.21.15.62:6818/4264123649,v1:172.21.15.62:6819/4264123649]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-20T21:39:54.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:54 smithi189 ceph-mon[57022]: from='osd.2 [v2:172.21.15.62:6818/4264123649,v1:172.21.15.62:6819/4264123649]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-20T21:39:54.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:54 smithi189 ceph-mon[57022]: osdmap e61: 6 total, 5 up, 6 in 2024-04-20T21:39:54.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:54 smithi189 ceph-mon[57022]: from='osd.2 [v2:172.21.15.62:6818/4264123649,v1:172.21.15.62:6819/4264123649]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-04-20T21:39:54.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:54 smithi189 ceph-mon[57022]: Health check update: Degraded data redundancy: 44/366 objects degraded (12.022%), 13 pgs degraded (PG_DEGRADED) 2024-04-20T21:39:54.784 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:54 smithi062 ceph-mon[76322]: from='osd.2 [v2:172.21.15.62:6818/4264123649,v1:172.21.15.62:6819/4264123649]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-20T21:39:54.784 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:54 smithi062 ceph-mon[76322]: osdmap e61: 6 total, 5 up, 6 in 2024-04-20T21:39:54.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:54 smithi062 ceph-mon[76322]: from='osd.2 [v2:172.21.15.62:6818/4264123649,v1:172.21.15.62:6819/4264123649]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-04-20T21:39:54.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:54 smithi062 ceph-mon[76322]: Health check update: Degraded data redundancy: 44/366 objects degraded (12.022%), 13 pgs degraded (PG_DEGRADED) 2024-04-20T21:39:55.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:55 smithi189 ceph-mon[57022]: pgmap v116: 65 pgs: 13 active+undersized+degraded, 14 active+undersized, 38 active+clean; 315 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 44/366 objects degraded (12.022%) 2024-04-20T21:39:55.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:55 smithi189 ceph-mon[57022]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-20T21:39:55.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:55 smithi189 ceph-mon[57022]: osd.2 [v2:172.21.15.62:6818/4264123649,v1:172.21.15.62:6819/4264123649] boot 2024-04-20T21:39:55.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:55 smithi189 ceph-mon[57022]: osdmap e62: 6 total, 6 up, 6 in 2024-04-20T21:39:55.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:55 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-20T21:39:55.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:55 smithi062 ceph-mon[76322]: pgmap v116: 65 pgs: 13 active+undersized+degraded, 14 active+undersized, 38 active+clean; 315 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 44/366 objects degraded (12.022%) 2024-04-20T21:39:55.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:55 smithi062 ceph-mon[76322]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-20T21:39:55.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:55 smithi062 ceph-mon[76322]: osd.2 [v2:172.21.15.62:6818/4264123649,v1:172.21.15.62:6819/4264123649] boot 2024-04-20T21:39:55.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:55 smithi062 ceph-mon[76322]: osdmap e62: 6 total, 6 up, 6 in 2024-04-20T21:39:55.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:55 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-20T21:39:56.687 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:56 smithi062 ceph-mon[76322]: osdmap e63: 6 total, 6 up, 6 in 2024-04-20T21:39:56.687 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:56 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:39:56.687 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:56 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:39:56.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:56 smithi189 ceph-mon[57022]: osdmap e63: 6 total, 6 up, 6 in 2024-04-20T21:39:56.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:56 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:39:56.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:56 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:39:57.700 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:57 smithi062 ceph-mon[76322]: pgmap v119: 65 pgs: 15 peering, 6 active+undersized+degraded, 6 active+undersized, 38 active+clean; 315 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s rd, 2 op/s; 20/366 objects degraded (5.464%) 2024-04-20T21:39:57.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:57 smithi189 ceph-mon[57022]: pgmap v119: 65 pgs: 15 peering, 6 active+undersized+degraded, 6 active+undersized, 38 active+clean; 315 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s rd, 2 op/s; 20/366 objects degraded (5.464%) 2024-04-20T21:39:59.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:59 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:39:59.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:59 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:39:59.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:39:59 smithi062 ceph-mon[76322]: pgmap v120: 65 pgs: 15 peering, 6 active+undersized+degraded, 6 active+undersized, 38 active+clean; 315 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s; 20/366 objects degraded (5.464%) 2024-04-20T21:39:59.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:59 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:39:59.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:59 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:39:59.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:39:59 smithi189 ceph-mon[57022]: pgmap v120: 65 pgs: 15 peering, 6 active+undersized+degraded, 6 active+undersized, 38 active+clean; 315 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s; 20/366 objects degraded (5.464%) 2024-04-20T21:40:00.426 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:00 smithi062 ceph-mon[76322]: Health check update: Degraded data redundancy: 20/366 objects degraded (5.464%), 6 pgs degraded (PG_DEGRADED) 2024-04-20T21:40:00.426 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:00 smithi062 ceph-mon[76322]: Health detail: HEALTH_WARN 1 filesystem with deprecated feature inline_data; Degraded data redundancy: 20/366 objects degraded (5.464%), 6 pgs degraded 2024-04-20T21:40:00.426 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:00 smithi062 ceph-mon[76322]: [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T21:40:00.426 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:00 smithi062 ceph-mon[76322]: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T21:40:00.427 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:00 smithi062 ceph-mon[76322]: [WRN] PG_DEGRADED: Degraded data redundancy: 20/366 objects degraded (5.464%), 6 pgs degraded 2024-04-20T21:40:00.427 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:00 smithi062 ceph-mon[76322]: pg 2.1 is active+undersized+degraded, acting [1,0] 2024-04-20T21:40:00.427 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:00 smithi062 ceph-mon[76322]: pg 2.c is active+undersized+degraded, acting [3,0] 2024-04-20T21:40:00.427 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:00 smithi062 ceph-mon[76322]: pg 2.d is active+undersized+degraded, acting [1,3] 2024-04-20T21:40:00.427 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:00 smithi062 ceph-mon[76322]: pg 2.e is active+undersized+degraded, acting [0,3] 2024-04-20T21:40:00.427 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:00 smithi062 ceph-mon[76322]: pg 2.10 is active+undersized+degraded, acting [1,0] 2024-04-20T21:40:00.427 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:00 smithi062 ceph-mon[76322]: pg 2.1e is active+undersized+degraded, acting [0,5] 2024-04-20T21:40:00.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:00 smithi189 ceph-mon[57022]: Health check update: Degraded data redundancy: 20/366 objects degraded (5.464%), 6 pgs degraded (PG_DEGRADED) 2024-04-20T21:40:00.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:00 smithi189 ceph-mon[57022]: Health detail: HEALTH_WARN 1 filesystem with deprecated feature inline_data; Degraded data redundancy: 20/366 objects degraded (5.464%), 6 pgs degraded 2024-04-20T21:40:00.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:00 smithi189 ceph-mon[57022]: [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T21:40:00.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:00 smithi189 ceph-mon[57022]: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T21:40:00.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:00 smithi189 ceph-mon[57022]: [WRN] PG_DEGRADED: Degraded data redundancy: 20/366 objects degraded (5.464%), 6 pgs degraded 2024-04-20T21:40:00.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:00 smithi189 ceph-mon[57022]: pg 2.1 is active+undersized+degraded, acting [1,0] 2024-04-20T21:40:00.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:00 smithi189 ceph-mon[57022]: pg 2.c is active+undersized+degraded, acting [3,0] 2024-04-20T21:40:00.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:00 smithi189 ceph-mon[57022]: pg 2.d is active+undersized+degraded, acting [1,3] 2024-04-20T21:40:00.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:00 smithi189 ceph-mon[57022]: pg 2.e is active+undersized+degraded, acting [0,3] 2024-04-20T21:40:00.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:00 smithi189 ceph-mon[57022]: pg 2.10 is active+undersized+degraded, acting [1,0] 2024-04-20T21:40:00.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:00 smithi189 ceph-mon[57022]: pg 2.1e is active+undersized+degraded, acting [0,5] 2024-04-20T21:40:01.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:01 smithi189 ceph-mon[57022]: pgmap v121: 65 pgs: 5 peering, 60 active+clean; 315 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-20T21:40:01.534 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:01 smithi062 ceph-mon[76322]: pgmap v121: 65 pgs: 5 peering, 60 active+clean; 315 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-20T21:40:02.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:02 smithi189 ceph-mon[57022]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 20/366 objects degraded (5.464%), 6 pgs degraded) 2024-04-20T21:40:02.534 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:02 smithi062 ceph-mon[76322]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 20/366 objects degraded (5.464%), 6 pgs degraded) 2024-04-20T21:40:03.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:03 smithi062 ceph-mon[76322]: pgmap v122: 65 pgs: 65 active+clean; 315 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-20T21:40:03.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:03 smithi189 ceph-mon[57022]: pgmap v122: 65 pgs: 65 active+clean; 315 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-20T21:40:05.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:05 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:40:05.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:05 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:40:05.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:05 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:40:05.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:05 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T21:40:05.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:05 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:40:05.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:05 smithi062 ceph-mon[76322]: pgmap v123: 65 pgs: 65 active+clean; 315 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:40:05.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:05 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:40:05.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:05 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:40:05.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:05 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:40:05.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:05 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:40:05.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:05 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:40:05.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:05 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:40:05.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:05 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-20T21:40:05.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:05 smithi062 ceph-mon[76322]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-20T21:40:05.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:05 smithi062 ceph-mon[76322]: Upgrade: osd.3 is safe to restart 2024-04-20T21:40:05.368 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:05 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:40:05.368 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:05 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:40:05.368 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:05 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:40:05.369 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:05 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T21:40:05.369 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:05 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:40:05.369 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:05 smithi189 ceph-mon[57022]: pgmap v123: 65 pgs: 65 active+clean; 315 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:40:05.369 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:05 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:40:05.369 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:05 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:40:05.369 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:05 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:40:05.369 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:05 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:40:05.369 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:05 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:40:05.369 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:05 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:40:05.369 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:05 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-20T21:40:05.369 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:05 smithi189 ceph-mon[57022]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-20T21:40:05.369 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:05 smithi189 ceph-mon[57022]: Upgrade: osd.3 is safe to restart 2024-04-20T21:40:07.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:07.443+0000 7f4b9f000700 1 -- 172.21.15.62:0/3558141887 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4b98106b80 msgr2=0x7f4b98106ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:40:07.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:07.443+0000 7f4b9f000700 1 --2- 172.21.15.62:0/3558141887 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4b98106b80 0x7f4b98106ff0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f4b84009910 tx=0x7f4b84009c20 comp rx=0 tx=0).stop 2024-04-20T21:40:07.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:07.443+0000 7f4b9f000700 1 -- 172.21.15.62:0/3558141887 shutdown_connections 2024-04-20T21:40:07.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:07.443+0000 7f4b9f000700 1 --2- 172.21.15.62:0/3558141887 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4b98106b80 0x7f4b98106ff0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:40:07.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:07.443+0000 7f4b9f000700 1 --2- 172.21.15.62:0/3558141887 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b9810c8f0 0x7f4b9810ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:40:07.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:07.443+0000 7f4b9f000700 1 -- 172.21.15.62:0/3558141887 >> 172.21.15.62:0/3558141887 conn(0x7f4b98076060 msgr2=0x7f4b98078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:40:07.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:07.443+0000 7f4b9f000700 1 -- 172.21.15.62:0/3558141887 shutdown_connections 2024-04-20T21:40:07.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:07.443+0000 7f4b9f000700 1 -- 172.21.15.62:0/3558141887 wait complete. 2024-04-20T21:40:07.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:07.444+0000 7f4b9f000700 1 Processor -- start 2024-04-20T21:40:07.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:07.444+0000 7f4b9f000700 1 -- start start 2024-04-20T21:40:07.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:07.445+0000 7f4b9f000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b98106b80 0x7f4b9819c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:40:07.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:07.445+0000 7f4b9f000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4b9810c8f0 0x7f4b9819c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:40:07.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:07.445+0000 7f4b9f000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b9819d060 con 0x7f4b9810c8f0 2024-04-20T21:40:07.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:07.445+0000 7f4b9f000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b981a0df0 con 0x7f4b98106b80 2024-04-20T21:40:07.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:07.445+0000 7f4b97400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4b9810c8f0 0x7f4b9819c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:40:07.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:07.445+0000 7f4b97400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4b9810c8f0 0x7f4b9819c980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49538/0 (socket says 172.21.15.62:49538) 2024-04-20T21:40:07.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:07.445+0000 7f4b97400700 1 -- 172.21.15.62:0/4074712649 learned_addr learned my addr 172.21.15.62:0/4074712649 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:40:07.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:07.445+0000 7f4b97e00700 1 --2- 172.21.15.62:0/4074712649 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b98106b80 0x7f4b9819c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:40:07.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:07.446+0000 7f4b97400700 1 -- 172.21.15.62:0/4074712649 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b98106b80 msgr2=0x7f4b9819c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:40:07.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:07.446+0000 7f4b97400700 1 --2- 172.21.15.62:0/4074712649 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b98106b80 0x7f4b9819c440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:40:07.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:07.446+0000 7f4b97400700 1 -- 172.21.15.62:0/4074712649 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b840095f0 con 0x7f4b9810c8f0 2024-04-20T21:40:07.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:07.446+0000 7f4b97400700 1 --2- 172.21.15.62:0/4074712649 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4b9810c8f0 0x7f4b9819c980 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f4b8400b5c0 tx=0x7f4b8400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:40:07.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:07.446+0000 7f4b94c00700 1 -- 172.21.15.62:0/4074712649 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4b8401d070 con 0x7f4b9810c8f0 2024-04-20T21:40:07.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:07.446+0000 7f4b94c00700 1 -- 172.21.15.62:0/4074712649 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4b84005e00 con 0x7f4b9810c8f0 2024-04-20T21:40:07.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:07.446+0000 7f4b94c00700 1 -- 172.21.15.62:0/4074712649 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4b84017400 con 0x7f4b9810c8f0 2024-04-20T21:40:07.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:07.446+0000 7f4b9f000700 1 -- 172.21.15.62:0/4074712649 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b981a1070 con 0x7f4b9810c8f0 2024-04-20T21:40:07.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:07.447+0000 7f4b9f000700 1 -- 172.21.15.62:0/4074712649 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b981a1560 con 0x7f4b9810c8f0 2024-04-20T21:40:07.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:07.448+0000 7f4b9f000700 1 -- 172.21.15.62:0/4074712649 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b9810ecc0 con 0x7f4b9810c8f0 2024-04-20T21:40:07.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:07.449+0000 7f4b94c00700 1 -- 172.21.15.62:0/4074712649 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f4b8400f6f0 con 0x7f4b9810c8f0 2024-04-20T21:40:07.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:07.450+0000 7f4b94c00700 1 --2- 172.21.15.62:0/4074712649 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f4b7c06ecc0 0x7f4b7c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:40:07.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:07.450+0000 7f4b94c00700 1 -- 172.21.15.62:0/4074712649 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f4b84090240 con 0x7f4b9810c8f0 2024-04-20T21:40:07.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:07.450+0000 7f4b97e00700 1 --2- 172.21.15.62:0/4074712649 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f4b7c06ecc0 0x7f4b7c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:40:07.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:07.451+0000 7f4b97e00700 1 --2- 172.21.15.62:0/4074712649 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f4b7c06ecc0 0x7f4b7c071170 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4b88005950 tx=0x7f4b8800b410 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:40:07.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:07.455+0000 7f4b94c00700 1 -- 172.21.15.62:0/4074712649 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f4b84059080 con 0x7f4b9810c8f0 2024-04-20T21:40:07.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:07.718+0000 7f4b9f000700 1 -- 172.21.15.62:0/4074712649 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4b9802cca0 con 0x7f4b7c06ecc0 2024-04-20T21:40:07.717 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:07 smithi062 ceph-mon[76322]: pgmap v124: 65 pgs: 65 active+clean; 315 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:40:07.718 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:07 smithi062 ceph-mon[76322]: Upgrade: Updating osd.3 2024-04-20T21:40:07.718 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:07 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:40:07.718 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:07 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-20T21:40:07.718 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:07 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:40:07.718 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:07 smithi062 ceph-mon[76322]: Deploying daemon osd.3 on smithi189 2024-04-20T21:40:07.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:07.721+0000 7f4b94c00700 1 -- 172.21.15.62:0/4074712649 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f4b9802cca0 con 0x7f4b7c06ecc0 2024-04-20T21:40:07.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:07.723+0000 7f4b9f000700 1 -- 172.21.15.62:0/4074712649 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f4b7c06ecc0 msgr2=0x7f4b7c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:40:07.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:07.723+0000 7f4b9f000700 1 --2- 172.21.15.62:0/4074712649 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f4b7c06ecc0 0x7f4b7c071170 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4b88005950 tx=0x7f4b8800b410 comp rx=0 tx=0).stop 2024-04-20T21:40:07.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:07.723+0000 7f4b9f000700 1 -- 172.21.15.62:0/4074712649 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4b9810c8f0 msgr2=0x7f4b9819c980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:40:07.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:07.723+0000 7f4b9f000700 1 --2- 172.21.15.62:0/4074712649 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4b9810c8f0 0x7f4b9819c980 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f4b8400b5c0 tx=0x7f4b8400bd70 comp rx=0 tx=0).stop 2024-04-20T21:40:07.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:07.723+0000 7f4b9f000700 1 -- 172.21.15.62:0/4074712649 shutdown_connections 2024-04-20T21:40:07.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:07.723+0000 7f4b9f000700 1 --2- 172.21.15.62:0/4074712649 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f4b7c06ecc0 0x7f4b7c071170 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:40:07.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:07.723+0000 7f4b9f000700 1 --2- 172.21.15.62:0/4074712649 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b98106b80 0x7f4b9819c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:40:07.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:07.723+0000 7f4b9f000700 1 --2- 172.21.15.62:0/4074712649 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4b9810c8f0 0x7f4b9819c980 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:40:07.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:07.723+0000 7f4b9f000700 1 -- 172.21.15.62:0/4074712649 >> 172.21.15.62:0/4074712649 conn(0x7f4b98076060 msgr2=0x7f4b98077ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:40:07.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:07.724+0000 7f4b9f000700 1 -- 172.21.15.62:0/4074712649 shutdown_connections 2024-04-20T21:40:07.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:07.724+0000 7f4b9f000700 1 -- 172.21.15.62:0/4074712649 wait complete. 2024-04-20T21:40:07.739 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T21:40:07.804 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:07 smithi189 ceph-mon[57022]: pgmap v124: 65 pgs: 65 active+clean; 315 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:40:07.805 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:07 smithi189 ceph-mon[57022]: Upgrade: Updating osd.3 2024-04-20T21:40:07.805 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:07 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:40:07.805 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:07 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-20T21:40:07.805 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:07 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:40:07.805 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:07 smithi189 ceph-mon[57022]: Deploying daemon osd.3 on smithi189 2024-04-20T21:40:07.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:07.884+0000 7f7e9e000700 1 -- 172.21.15.62:0/2238310844 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7e98073a00 msgr2=0x7f7e9810c580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:40:07.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:07.884+0000 7f7e9e000700 1 --2- 172.21.15.62:0/2238310844 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7e98073a00 0x7f7e9810c580 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f7e80009910 tx=0x7f7e80009c20 comp rx=0 tx=0).stop 2024-04-20T21:40:07.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:07.885+0000 7f7e9e000700 1 -- 172.21.15.62:0/2238310844 shutdown_connections 2024-04-20T21:40:07.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:07.885+0000 7f7e9e000700 1 --2- 172.21.15.62:0/2238310844 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7e98073a00 0x7f7e9810c580 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:40:07.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:07.885+0000 7f7e9e000700 1 --2- 172.21.15.62:0/2238310844 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e980730f0 0x7f7e980734c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:40:07.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:07.885+0000 7f7e9e000700 1 -- 172.21.15.62:0/2238310844 >> 172.21.15.62:0/2238310844 conn(0x7f7e980fbda0 msgr2=0x7f7e980fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:40:07.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:07.885+0000 7f7e9e000700 1 -- 172.21.15.62:0/2238310844 shutdown_connections 2024-04-20T21:40:07.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:07.885+0000 7f7e9e000700 1 -- 172.21.15.62:0/2238310844 wait complete. 2024-04-20T21:40:07.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:07.886+0000 7f7e9e000700 1 Processor -- start 2024-04-20T21:40:07.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:07.886+0000 7f7e9e000700 1 -- start start 2024-04-20T21:40:07.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:07.886+0000 7f7e9e000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7e980730f0 0x7f7e98197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:40:07.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:07.886+0000 7f7e9e000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e98073a00 0x7f7e98198610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:40:07.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:07.886+0000 7f7e9e000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e9819c950 con 0x7f7e980730f0 2024-04-20T21:40:07.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:07.887+0000 7f7e9e000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e9819cac0 con 0x7f7e98073a00 2024-04-20T21:40:07.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:07.887+0000 7f7e97400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7e980730f0 0x7f7e98197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:40:07.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:07.887+0000 7f7e97400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7e980730f0 0x7f7e98197fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49554/0 (socket says 172.21.15.62:49554) 2024-04-20T21:40:07.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:07.887+0000 7f7e97400700 1 -- 172.21.15.62:0/352229207 learned_addr learned my addr 172.21.15.62:0/352229207 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:40:07.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:07.887+0000 7f7e8ea00700 1 --2- 172.21.15.62:0/352229207 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e98073a00 0x7f7e98198610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:40:07.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:07.887+0000 7f7e97400700 1 -- 172.21.15.62:0/352229207 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e98073a00 msgr2=0x7f7e98198610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:40:07.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:07.887+0000 7f7e97400700 1 --2- 172.21.15.62:0/352229207 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e98073a00 0x7f7e98198610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:40:07.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:07.887+0000 7f7e97400700 1 -- 172.21.15.62:0/352229207 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e800095f0 con 0x7f7e980730f0 2024-04-20T21:40:07.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:07.887+0000 7f7e8ea00700 1 --2- 172.21.15.62:0/352229207 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e98073a00 0x7f7e98198610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T21:40:07.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:07.888+0000 7f7e97400700 1 --2- 172.21.15.62:0/352229207 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7e980730f0 0x7f7e98197fa0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f7e8800d900 tx=0x7f7e8800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:40:07.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:07.888+0000 7f7e94c00700 1 -- 172.21.15.62:0/352229207 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7e88004a00 con 0x7f7e980730f0 2024-04-20T21:40:07.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:07.888+0000 7f7e94c00700 1 -- 172.21.15.62:0/352229207 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7e880054e0 con 0x7f7e980730f0 2024-04-20T21:40:07.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:07.888+0000 7f7e9e000700 1 -- 172.21.15.62:0/352229207 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e9819cda0 con 0x7f7e980730f0 2024-04-20T21:40:07.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:07.888+0000 7f7e94c00700 1 -- 172.21.15.62:0/352229207 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7e88009d60 con 0x7f7e980730f0 2024-04-20T21:40:07.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:07.888+0000 7f7e9e000700 1 -- 172.21.15.62:0/352229207 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e9819d2f0 con 0x7f7e980730f0 2024-04-20T21:40:07.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:07.889+0000 7f7e9e000700 1 -- 172.21.15.62:0/352229207 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e98109c80 con 0x7f7e980730f0 2024-04-20T21:40:07.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:07.890+0000 7f7e94c00700 1 -- 172.21.15.62:0/352229207 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f7e88005020 con 0x7f7e980730f0 2024-04-20T21:40:07.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:07.892+0000 7f7e94c00700 1 --2- 172.21.15.62:0/352229207 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7e7c06ecc0 0x7f7e7c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:40:07.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:07.892+0000 7f7e94c00700 1 -- 172.21.15.62:0/352229207 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f7e8808ef90 con 0x7f7e980730f0 2024-04-20T21:40:07.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:07.892+0000 7f7e8ea00700 1 --2- 172.21.15.62:0/352229207 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7e7c06ecc0 0x7f7e7c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:40:07.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:07.893+0000 7f7e8ea00700 1 --2- 172.21.15.62:0/352229207 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7e7c06ecc0 0x7f7e7c071170 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f7e80006010 tx=0x7f7e8000b540 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:40:07.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:07.897+0000 7f7e94c00700 1 -- 172.21.15.62:0/352229207 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f7e88057dd0 con 0x7f7e980730f0 2024-04-20T21:40:08.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:08.140+0000 7f7e9e000700 1 -- 172.21.15.62:0/352229207 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7e981993e0 con 0x7f7e7c06ecc0 2024-04-20T21:40:08.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:08.142+0000 7f7e94c00700 1 -- 172.21.15.62:0/352229207 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f7e981993e0 con 0x7f7e7c06ecc0 2024-04-20T21:40:08.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:08.145+0000 7f7e9e000700 1 -- 172.21.15.62:0/352229207 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7e7c06ecc0 msgr2=0x7f7e7c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:40:08.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:08.145+0000 7f7e9e000700 1 --2- 172.21.15.62:0/352229207 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7e7c06ecc0 0x7f7e7c071170 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f7e80006010 tx=0x7f7e8000b540 comp rx=0 tx=0).stop 2024-04-20T21:40:08.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:08.145+0000 7f7e9e000700 1 -- 172.21.15.62:0/352229207 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7e980730f0 msgr2=0x7f7e98197fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:40:08.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:08.145+0000 7f7e9e000700 1 --2- 172.21.15.62:0/352229207 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7e980730f0 0x7f7e98197fa0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f7e8800d900 tx=0x7f7e8800dcc0 comp rx=0 tx=0).stop 2024-04-20T21:40:08.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:08.145+0000 7f7e9e000700 1 -- 172.21.15.62:0/352229207 shutdown_connections 2024-04-20T21:40:08.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:08.145+0000 7f7e9e000700 1 --2- 172.21.15.62:0/352229207 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7e7c06ecc0 0x7f7e7c071170 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:40:08.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:08.145+0000 7f7e9e000700 1 --2- 172.21.15.62:0/352229207 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7e980730f0 0x7f7e98197fa0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:40:08.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:08.145+0000 7f7e9e000700 1 --2- 172.21.15.62:0/352229207 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e98073a00 0x7f7e98198610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:40:08.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:08.145+0000 7f7e9e000700 1 -- 172.21.15.62:0/352229207 >> 172.21.15.62:0/352229207 conn(0x7f7e980fbda0 msgr2=0x7f7e98106dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:40:08.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:08.145+0000 7f7e9e000700 1 -- 172.21.15.62:0/352229207 shutdown_connections 2024-04-20T21:40:08.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:08.145+0000 7f7e9e000700 1 -- 172.21.15.62:0/352229207 wait complete. 2024-04-20T21:40:08.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:08.318+0000 7f81e1800700 1 -- 172.21.15.62:0/3529147724 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f81dc073a00 msgr2=0x7f81dc10c580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:40:08.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:08.318+0000 7f81e1800700 1 --2- 172.21.15.62:0/3529147724 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f81dc073a00 0x7f81dc10c580 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f81c4009910 tx=0x7f81c4009c20 comp rx=0 tx=0).stop 2024-04-20T21:40:08.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:08.318+0000 7f81e1800700 1 -- 172.21.15.62:0/3529147724 shutdown_connections 2024-04-20T21:40:08.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:08.318+0000 7f81e1800700 1 --2- 172.21.15.62:0/3529147724 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f81dc073a00 0x7f81dc10c580 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:40:08.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:08.318+0000 7f81e1800700 1 --2- 172.21.15.62:0/3529147724 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f81dc0730f0 0x7f81dc0734c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:40:08.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:08.318+0000 7f81e1800700 1 -- 172.21.15.62:0/3529147724 >> 172.21.15.62:0/3529147724 conn(0x7f81dc0fbda0 msgr2=0x7f81dc0fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:40:08.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:08.318+0000 7f81e1800700 1 -- 172.21.15.62:0/3529147724 shutdown_connections 2024-04-20T21:40:08.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:08.318+0000 7f81e1800700 1 -- 172.21.15.62:0/3529147724 wait complete. 2024-04-20T21:40:08.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:08.319+0000 7f81e1800700 1 Processor -- start 2024-04-20T21:40:08.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:08.319+0000 7f81e1800700 1 -- start start 2024-04-20T21:40:08.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:08.319+0000 7f81e1800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f81dc0730f0 0x7f81dc197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:40:08.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:08.319+0000 7f81e1800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f81dc073a00 0x7f81dc198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:40:08.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:08.320+0000 7f81e1800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81dc19c970 con 0x7f81dc0730f0 2024-04-20T21:40:08.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:08.320+0000 7f81e1800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81dc19cae0 con 0x7f81dc073a00 2024-04-20T21:40:08.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:08.320+0000 7f81daa00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f81dc0730f0 0x7f81dc197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:40:08.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:08.320+0000 7f81daa00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f81dc0730f0 0x7f81dc197fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49572/0 (socket says 172.21.15.62:49572) 2024-04-20T21:40:08.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:08.320+0000 7f81daa00700 1 -- 172.21.15.62:0/1621843345 learned_addr learned my addr 172.21.15.62:0/1621843345 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:40:08.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:08.320+0000 7f81daa00700 1 -- 172.21.15.62:0/1621843345 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f81dc073a00 msgr2=0x7f81dc198630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:40:08.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:08.320+0000 7f81daa00700 1 --2- 172.21.15.62:0/1621843345 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f81dc073a00 0x7f81dc198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:40:08.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:08.320+0000 7f81daa00700 1 -- 172.21.15.62:0/1621843345 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81c40095f0 con 0x7f81dc0730f0 2024-04-20T21:40:08.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:08.320+0000 7f81daa00700 1 --2- 172.21.15.62:0/1621843345 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f81dc0730f0 0x7f81dc197fc0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f81cc00d8d0 tx=0x7f81cc00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:40:08.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:08.321+0000 7f81d3400700 1 -- 172.21.15.62:0/1621843345 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f81cc009940 con 0x7f81dc0730f0 2024-04-20T21:40:08.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:08.321+0000 7f81d3400700 1 -- 172.21.15.62:0/1621843345 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f81cc010460 con 0x7f81dc0730f0 2024-04-20T21:40:08.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:08.321+0000 7f81d3400700 1 -- 172.21.15.62:0/1621843345 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f81cc00f5f0 con 0x7f81dc0730f0 2024-04-20T21:40:08.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:08.321+0000 7f81e1800700 1 -- 172.21.15.62:0/1621843345 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81dc19cdc0 con 0x7f81dc0730f0 2024-04-20T21:40:08.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:08.321+0000 7f81e1800700 1 -- 172.21.15.62:0/1621843345 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81dc19d310 con 0x7f81dc0730f0 2024-04-20T21:40:08.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:08.322+0000 7f81e1800700 1 -- 172.21.15.62:0/1621843345 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81dc109c80 con 0x7f81dc0730f0 2024-04-20T21:40:08.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:08.323+0000 7f81d3400700 1 -- 172.21.15.62:0/1621843345 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f81cc0105d0 con 0x7f81dc0730f0 2024-04-20T21:40:08.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:08.325+0000 7f81d3400700 1 --2- 172.21.15.62:0/1621843345 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f81c006ecc0 0x7f81c0071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:40:08.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:08.325+0000 7f81d3400700 1 -- 172.21.15.62:0/1621843345 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f81cc08ed60 con 0x7f81dc0730f0 2024-04-20T21:40:08.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:08.325+0000 7f81d3e00700 1 --2- 172.21.15.62:0/1621843345 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f81c006ecc0 0x7f81c0071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:40:08.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:08.326+0000 7f81d3e00700 1 --2- 172.21.15.62:0/1621843345 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f81c006ecc0 0x7f81c0071170 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f81c400b5c0 tx=0x7f81c4005960 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:40:08.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:08.330+0000 7f81d3400700 1 -- 172.21.15.62:0/1621843345 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f81cc093070 con 0x7f81dc0730f0 2024-04-20T21:40:08.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:08.581+0000 7f81e1800700 1 -- 172.21.15.62:0/1621843345 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f81dc1993c0 con 0x7f81c006ecc0 2024-04-20T21:40:08.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:08.609+0000 7f81d3400700 1 -- 172.21.15.62:0/1621843345 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7f81dc1993c0 con 0x7f81c006ecc0 2024-04-20T21:40:08.608 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T21:40:08.609 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (5m) 12s ago 19m 14.2M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T21:40:08.609 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (20m) 12s ago 20m 8992k - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T21:40:08.609 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (18m) 111s ago 18m 8288k - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T21:40:08.609 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (2m) 12s ago 20m 6660k - 19.0.0-3133-ga8c84f59 76bc6b91b726 2825d5ba11c6 2024-04-20T21:40:08.609 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (116s) 111s ago 18m 6665k - 19.0.0-3133-ga8c84f59 76bc6b91b726 20f746fa1bba 2024-04-20T21:40:08.609 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (5m) 12s ago 19m 75.0M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T21:40:08.609 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (12m) 12s ago 12m 65.5M - 18.2.0 dc2bc1663786 5e6460aa6159 2024-04-20T21:40:08.609 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (12m) 12s ago 12m 131M - 18.2.0 dc2bc1663786 e65f2267544e 2024-04-20T21:40:08.609 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (12m) 111s ago 12m 117M - 18.2.0 dc2bc1663786 fb2a8a223db8 2024-04-20T21:40:08.609 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (12m) 111s ago 12m 55.6M - 18.2.0 dc2bc1663786 f2eec2ee8951 2024-04-20T21:40:08.609 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (8m) 12s ago 21m 504M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T21:40:08.609 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (7m) 111s ago 18m 450M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T21:40:08.609 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (3m) 12s ago 21m 49.3M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 019eb1479485 2024-04-20T21:40:08.609 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (3m) 111s ago 18m 36.6M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 55aac560aab6 2024-04-20T21:40:08.609 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (7m) 12s ago 20m 7906k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T21:40:08.610 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (7m) 111s ago 18m 7972k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T21:40:08.610 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (84s) 12s ago 17m 83.0M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 ee2bb901f26e 2024-04-20T21:40:08.610 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (52s) 12s ago 16m 87.9M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 9075152d4162 2024-04-20T21:40:08.610 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (21s) 12s ago 16m 12.7M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 2d4f3e38dc0e 2024-04-20T21:40:08.610 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (15m) 111s ago 15m 242M 2240M 18.2.0 dc2bc1663786 611c1e242693 2024-04-20T21:40:08.610 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (15m) 111s ago 15m 221M 2240M 18.2.0 dc2bc1663786 6c8fce91d45d 2024-04-20T21:40:08.610 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (14m) 111s ago 14m 205M 2240M 18.2.0 dc2bc1663786 8e4574bae8db 2024-04-20T21:40:08.610 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (6m) 12s ago 19m 73.9M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T21:40:08.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:08.612+0000 7f81e1800700 1 -- 172.21.15.62:0/1621843345 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f81c006ecc0 msgr2=0x7f81c0071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:40:08.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:08.612+0000 7f81e1800700 1 --2- 172.21.15.62:0/1621843345 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f81c006ecc0 0x7f81c0071170 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f81c400b5c0 tx=0x7f81c4005960 comp rx=0 tx=0).stop 2024-04-20T21:40:08.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:08.612+0000 7f81e1800700 1 -- 172.21.15.62:0/1621843345 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f81dc0730f0 msgr2=0x7f81dc197fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:40:08.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:08.612+0000 7f81e1800700 1 --2- 172.21.15.62:0/1621843345 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f81dc0730f0 0x7f81dc197fc0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f81cc00d8d0 tx=0x7f81cc00dc90 comp rx=0 tx=0).stop 2024-04-20T21:40:08.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:08.612+0000 7f81e1800700 1 -- 172.21.15.62:0/1621843345 shutdown_connections 2024-04-20T21:40:08.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:08.612+0000 7f81e1800700 1 --2- 172.21.15.62:0/1621843345 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f81c006ecc0 0x7f81c0071170 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:40:08.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:08.612+0000 7f81e1800700 1 --2- 172.21.15.62:0/1621843345 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f81dc0730f0 0x7f81dc197fc0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:40:08.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:08.612+0000 7f81e1800700 1 --2- 172.21.15.62:0/1621843345 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f81dc073a00 0x7f81dc198630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:40:08.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:08.612+0000 7f81e1800700 1 -- 172.21.15.62:0/1621843345 >> 172.21.15.62:0/1621843345 conn(0x7f81dc0fbda0 msgr2=0x7f81dc106dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:40:08.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:08.613+0000 7f81e1800700 1 -- 172.21.15.62:0/1621843345 shutdown_connections 2024-04-20T21:40:08.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:08.613+0000 7f81e1800700 1 -- 172.21.15.62:0/1621843345 wait complete. 2024-04-20T21:40:08.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:08.772+0000 7fa770800700 1 -- 172.21.15.62:0/563928845 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa768108470 msgr2=0x7fa768108840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:40:08.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:08.772+0000 7fa770800700 1 --2- 172.21.15.62:0/563928845 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa768108470 0x7fa768108840 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fa75c009910 tx=0x7fa75c009c20 comp rx=0 tx=0).stop 2024-04-20T21:40:08.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:08.772+0000 7fa770800700 1 -- 172.21.15.62:0/563928845 shutdown_connections 2024-04-20T21:40:08.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:08.772+0000 7fa770800700 1 --2- 172.21.15.62:0/563928845 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa768102470 0x7fa7681028e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:40:08.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:08.772+0000 7fa770800700 1 --2- 172.21.15.62:0/563928845 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa768108470 0x7fa768108840 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:40:08.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:08.772+0000 7fa770800700 1 -- 172.21.15.62:0/563928845 >> 172.21.15.62:0/563928845 conn(0x7fa7680fdf90 msgr2=0x7fa7681003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:40:08.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:08.772+0000 7fa770800700 1 -- 172.21.15.62:0/563928845 shutdown_connections 2024-04-20T21:40:08.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:08.772+0000 7fa770800700 1 -- 172.21.15.62:0/563928845 wait complete. 2024-04-20T21:40:08.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:08.773+0000 7fa770800700 1 Processor -- start 2024-04-20T21:40:08.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:08.774+0000 7fa770800700 1 -- start start 2024-04-20T21:40:08.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:08.774+0000 7fa770800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa768102470 0x7fa7681981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:40:08.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:08.774+0000 7fa770800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa768108470 0x7fa768198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:40:08.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:08.774+0000 7fa770800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa768198e10 con 0x7fa768102470 2024-04-20T21:40:08.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:08.774+0000 7fa770800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa76819cba0 con 0x7fa768108470 2024-04-20T21:40:08.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:08.774+0000 7fa76de00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa768102470 0x7fa7681981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:40:08.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:08.774+0000 7fa76de00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa768102470 0x7fa7681981f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49596/0 (socket says 172.21.15.62:49596) 2024-04-20T21:40:08.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:08.774+0000 7fa76de00700 1 -- 172.21.15.62:0/2383894347 learned_addr learned my addr 172.21.15.62:0/2383894347 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:40:08.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:08.774+0000 7fa76d400700 1 --2- 172.21.15.62:0/2383894347 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa768108470 0x7fa768198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:40:08.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:08.775+0000 7fa76de00700 1 -- 172.21.15.62:0/2383894347 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa768108470 msgr2=0x7fa768198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:40:08.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:08.775+0000 7fa76de00700 1 --2- 172.21.15.62:0/2383894347 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa768108470 0x7fa768198730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:40:08.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:08.775+0000 7fa76de00700 1 -- 172.21.15.62:0/2383894347 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa75c0095f0 con 0x7fa768102470 2024-04-20T21:40:08.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:08.775+0000 7fa76de00700 1 --2- 172.21.15.62:0/2383894347 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa768102470 0x7fa7681981f0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fa75c0092e0 tx=0x7fa75c00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:40:08.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:08.775+0000 7fa762a00700 1 -- 172.21.15.62:0/2383894347 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa75c01d070 con 0x7fa768102470 2024-04-20T21:40:08.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:08.776+0000 7fa762a00700 1 -- 172.21.15.62:0/2383894347 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa75c005e00 con 0x7fa768102470 2024-04-20T21:40:08.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:08.776+0000 7fa762a00700 1 -- 172.21.15.62:0/2383894347 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa75c017400 con 0x7fa768102470 2024-04-20T21:40:08.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:08.776+0000 7fa770800700 1 -- 172.21.15.62:0/2383894347 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa76819ce20 con 0x7fa768102470 2024-04-20T21:40:08.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:08.776+0000 7fa770800700 1 -- 172.21.15.62:0/2383894347 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa76819d310 con 0x7fa768102470 2024-04-20T21:40:08.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:08.777+0000 7fa770800700 1 -- 172.21.15.62:0/2383894347 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa76810a9e0 con 0x7fa768102470 2024-04-20T21:40:08.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:08.778+0000 7fa762a00700 1 -- 172.21.15.62:0/2383894347 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fa75c00f6f0 con 0x7fa768102470 2024-04-20T21:40:08.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:08.779+0000 7fa762a00700 1 --2- 172.21.15.62:0/2383894347 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa75006ecc0 0x7fa750071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:40:08.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:08.779+0000 7fa762a00700 1 -- 172.21.15.62:0/2383894347 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fa75c090220 con 0x7fa768102470 2024-04-20T21:40:08.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:08.779+0000 7fa76d400700 1 --2- 172.21.15.62:0/2383894347 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa75006ecc0 0x7fa750071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:40:08.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:08.780+0000 7fa76d400700 1 --2- 172.21.15.62:0/2383894347 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa75006ecc0 0x7fa750071170 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fa768199810 tx=0x7fa75800b410 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:40:08.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:08.784+0000 7fa762a00700 1 -- 172.21.15.62:0/2383894347 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fa75c059060 con 0x7fa768102470 2024-04-20T21:40:08.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:08 smithi189 ceph-mon[57022]: from='client.34310 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:40:09.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:08 smithi062 ceph-mon[76322]: from='client.34310 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:40:09.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:09.116+0000 7fa770800700 1 -- 172.21.15.62:0/2383894347 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fa76804f2a0 con 0x7fa768102470 2024-04-20T21:40:09.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:09.117+0000 7fa762a00700 1 -- 172.21.15.62:0/2383894347 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+791 (secure 0 0 0) 0x7fa75c05c720 con 0x7fa768102470 2024-04-20T21:40:09.116 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:40:09.116 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T21:40:09.116 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:40:09.116 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:40:09.116 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T21:40:09.117 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:40:09.117 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:40:09.117 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T21:40:09.117 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 3, 2024-04-20T21:40:09.117 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 3 2024-04-20T21:40:09.117 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:40:09.117 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T21:40:09.117 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 4 2024-04-20T21:40:09.117 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:40:09.117 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T21:40:09.117 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 7, 2024-04-20T21:40:09.117 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 7 2024-04-20T21:40:09.117 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T21:40:09.117 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:40:09.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:09.120+0000 7fa770800700 1 -- 172.21.15.62:0/2383894347 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa75006ecc0 msgr2=0x7fa750071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:40:09.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:09.120+0000 7fa770800700 1 --2- 172.21.15.62:0/2383894347 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa75006ecc0 0x7fa750071170 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fa768199810 tx=0x7fa75800b410 comp rx=0 tx=0).stop 2024-04-20T21:40:09.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:09.120+0000 7fa770800700 1 -- 172.21.15.62:0/2383894347 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa768102470 msgr2=0x7fa7681981f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:40:09.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:09.120+0000 7fa770800700 1 --2- 172.21.15.62:0/2383894347 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa768102470 0x7fa7681981f0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fa75c0092e0 tx=0x7fa75c00bd70 comp rx=0 tx=0).stop 2024-04-20T21:40:09.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:09.121+0000 7fa770800700 1 -- 172.21.15.62:0/2383894347 shutdown_connections 2024-04-20T21:40:09.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:09.121+0000 7fa770800700 1 --2- 172.21.15.62:0/2383894347 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa75006ecc0 0x7fa750071170 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:40:09.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:09.121+0000 7fa770800700 1 --2- 172.21.15.62:0/2383894347 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa768102470 0x7fa7681981f0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:40:09.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:09.121+0000 7fa770800700 1 --2- 172.21.15.62:0/2383894347 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa768108470 0x7fa768198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:40:09.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:09.121+0000 7fa770800700 1 -- 172.21.15.62:0/2383894347 >> 172.21.15.62:0/2383894347 conn(0x7fa7680fdf90 msgr2=0x7fa7681002c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:40:09.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:09.121+0000 7fa770800700 1 -- 172.21.15.62:0/2383894347 shutdown_connections 2024-04-20T21:40:09.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:09.121+0000 7fa770800700 1 -- 172.21.15.62:0/2383894347 wait complete. 2024-04-20T21:40:09.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:09.289+0000 7f9959c00700 1 -- 172.21.15.62:0/1201126902 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9954108470 msgr2=0x7f9954108840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:40:09.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:09.289+0000 7f9959c00700 1 --2- 172.21.15.62:0/1201126902 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9954108470 0x7f9954108840 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f9944009910 tx=0x7f9944009c20 comp rx=0 tx=0).stop 2024-04-20T21:40:09.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:09.289+0000 7f9959c00700 1 -- 172.21.15.62:0/1201126902 shutdown_connections 2024-04-20T21:40:09.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:09.289+0000 7f9959c00700 1 --2- 172.21.15.62:0/1201126902 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9954102470 0x7f99541028e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:40:09.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:09.289+0000 7f9959c00700 1 --2- 172.21.15.62:0/1201126902 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9954108470 0x7f9954108840 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:40:09.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:09.289+0000 7f9959c00700 1 -- 172.21.15.62:0/1201126902 >> 172.21.15.62:0/1201126902 conn(0x7f99540fdf90 msgr2=0x7f99541003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:40:09.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:09.289+0000 7f9959c00700 1 -- 172.21.15.62:0/1201126902 shutdown_connections 2024-04-20T21:40:09.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:09.289+0000 7f9959c00700 1 -- 172.21.15.62:0/1201126902 wait complete. 2024-04-20T21:40:09.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:09.291+0000 7f9959c00700 1 Processor -- start 2024-04-20T21:40:09.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:09.291+0000 7f9959c00700 1 -- start start 2024-04-20T21:40:09.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:09.291+0000 7f9959c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9954102470 0x7f99541981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:40:09.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:09.291+0000 7f9959c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9954108470 0x7f9954198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:40:09.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:09.291+0000 7f9959c00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9954198e10 con 0x7f9954108470 2024-04-20T21:40:09.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:09.291+0000 7f9959c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f995419cba0 con 0x7f9954102470 2024-04-20T21:40:09.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:09.292+0000 7f994a000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9954108470 0x7f9954198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:40:09.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:09.292+0000 7f994a000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9954108470 0x7f9954198730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49614/0 (socket says 172.21.15.62:49614) 2024-04-20T21:40:09.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:09.292+0000 7f994a000700 1 -- 172.21.15.62:0/2682636937 learned_addr learned my addr 172.21.15.62:0/2682636937 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:40:09.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:09.292+0000 7f9952a00700 1 --2- 172.21.15.62:0/2682636937 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9954102470 0x7f99541981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:40:09.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:09.292+0000 7f994a000700 1 -- 172.21.15.62:0/2682636937 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9954102470 msgr2=0x7f99541981f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:40:09.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:09.292+0000 7f994a000700 1 --2- 172.21.15.62:0/2682636937 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9954102470 0x7f99541981f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:40:09.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:09.292+0000 7f994a000700 1 -- 172.21.15.62:0/2682636937 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99440095f0 con 0x7f9954108470 2024-04-20T21:40:09.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:09.292+0000 7f994a000700 1 --2- 172.21.15.62:0/2682636937 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9954108470 0x7f9954198730 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f993c00e920 tx=0x7f993c00ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:40:09.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:09.293+0000 7f994be00700 1 -- 172.21.15.62:0/2682636937 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f993c009790 con 0x7f9954108470 2024-04-20T21:40:09.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:09.293+0000 7f994be00700 1 -- 172.21.15.62:0/2682636937 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f993c00cba0 con 0x7f9954108470 2024-04-20T21:40:09.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:09.293+0000 7f994be00700 1 -- 172.21.15.62:0/2682636937 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f993c018650 con 0x7f9954108470 2024-04-20T21:40:09.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:09.293+0000 7f9959c00700 1 -- 172.21.15.62:0/2682636937 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f995419ce80 con 0x7f9954108470 2024-04-20T21:40:09.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:09.293+0000 7f9959c00700 1 -- 172.21.15.62:0/2682636937 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f995419d3d0 con 0x7f9954108470 2024-04-20T21:40:09.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:09.294+0000 7f9959c00700 1 -- 172.21.15.62:0/2682636937 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f995410a9e0 con 0x7f9954108470 2024-04-20T21:40:09.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:09.295+0000 7f994be00700 1 -- 172.21.15.62:0/2682636937 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f993c00cd10 con 0x7f9954108470 2024-04-20T21:40:09.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:09.296+0000 7f994be00700 1 --2- 172.21.15.62:0/2682636937 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f993806ecc0 0x7f9938071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:40:09.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:09.297+0000 7f994be00700 1 -- 172.21.15.62:0/2682636937 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f993c014070 con 0x7f9954108470 2024-04-20T21:40:09.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:09.297+0000 7f9952a00700 1 --2- 172.21.15.62:0/2682636937 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f993806ecc0 0x7f9938071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:40:09.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:09.297+0000 7f9952a00700 1 --2- 172.21.15.62:0/2682636937 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f993806ecc0 0x7f9938071170 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f994400b760 tx=0x7f9944005fd0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:40:09.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:09.301+0000 7f994be00700 1 -- 172.21.15.62:0/2682636937 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f993c053ff0 con 0x7f9954108470 2024-04-20T21:40:09.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:09.583+0000 7f9959c00700 1 -- 172.21.15.62:0/2682636937 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f995419d5f0 con 0x7f9954108470 2024-04-20T21:40:09.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:09.584+0000 7f994be00700 1 -- 172.21.15.62:0/2682636937 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 18 v18) v1 ==== 76+0+1928 (secure 0 0 0) 0x7f993c05bb40 con 0x7f9954108470 2024-04-20T21:40:09.584 INFO:teuthology.orchestra.run.smithi062.stdout:e18 2024-04-20T21:40:09.584 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T21:40:09.584 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T21:40:09.584 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T21:40:09.584 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:40:09.584 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T21:40:09.584 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T21:40:09.584 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 18 2024-04-20T21:40:09.584 INFO:teuthology.orchestra.run.smithi062.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-20T21:40:09.584 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T21:40:09.584 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T21:37:06.018300+0000 2024-04-20T21:40:09.585 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T21:40:09.585 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T21:40:09.585 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T21:40:09.585 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T21:40:09.585 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T21:40:09.585 INFO:teuthology.orchestra.run.smithi062.stdout:max_xattr_size 65536 2024-04-20T21:40:09.585 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T21:40:09.585 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T21:40:09.585 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 0 2024-04-20T21:40:09.585 INFO:teuthology.orchestra.run.smithi062.stdout:snap_rstat 1 2024-04-20T21:40:09.585 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T21:40:09.585 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 2 2024-04-20T21:40:09.585 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1 2024-04-20T21:40:09.585 INFO:teuthology.orchestra.run.smithi062.stdout:up {0=14458,1=24279} 2024-04-20T21:40:09.586 INFO:teuthology.orchestra.run.smithi062.stdout:failed 2024-04-20T21:40:09.586 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T21:40:09.586 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T21:40:09.586 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T21:40:09.586 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T21:40:09.586 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data enabled 2024-04-20T21:40:09.586 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T21:40:09.586 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T21:40:09.586 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 1 2024-04-20T21:40:09.586 INFO:teuthology.orchestra.run.smithi062.stdout:qdb_cluster leader: 14458 members: 24279,14458 2024-04-20T21:40:09.586 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ryqblb{0:14458} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.62:6826/1147127123,v1:172.21.15.62:6827/1147127123] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:40:09.586 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ozizwi{0:14474} state up:standby-replay seq 3 join_fscid=1 addr [v2:172.21.15.62:6828/3494174293,v1:172.21.15.62:6829/3494174293] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:40:09.586 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.dvimer{1:24279} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.189:6824/237262514,v1:172.21.15.189:6825/237262514] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:40:09.587 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.xbgspw{1:24291} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.189:6826/1495356258,v1:172.21.15.189:6827/1495356258] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:40:09.587 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:40:09.587 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:40:09.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:09.587+0000 7f9959c00700 1 -- 172.21.15.62:0/2682636937 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f993806ecc0 msgr2=0x7f9938071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:40:09.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:09.587+0000 7f9959c00700 1 --2- 172.21.15.62:0/2682636937 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f993806ecc0 0x7f9938071170 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f994400b760 tx=0x7f9944005fd0 comp rx=0 tx=0).stop 2024-04-20T21:40:09.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:09.587+0000 7f9959c00700 1 -- 172.21.15.62:0/2682636937 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9954108470 msgr2=0x7f9954198730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:40:09.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:09.587+0000 7f9959c00700 1 --2- 172.21.15.62:0/2682636937 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9954108470 0x7f9954198730 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f993c00e920 tx=0x7f993c00ece0 comp rx=0 tx=0).stop 2024-04-20T21:40:09.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:09.587+0000 7f9959c00700 1 -- 172.21.15.62:0/2682636937 shutdown_connections 2024-04-20T21:40:09.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:09.587+0000 7f9959c00700 1 --2- 172.21.15.62:0/2682636937 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f993806ecc0 0x7f9938071170 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:40:09.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:09.587+0000 7f9959c00700 1 --2- 172.21.15.62:0/2682636937 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9954102470 0x7f99541981f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:40:09.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:09.587+0000 7f9959c00700 1 --2- 172.21.15.62:0/2682636937 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9954108470 0x7f9954198730 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:40:09.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:09.587+0000 7f9959c00700 1 -- 172.21.15.62:0/2682636937 >> 172.21.15.62:0/2682636937 conn(0x7f99540fdf90 msgr2=0x7f99541002c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:40:09.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:09.587+0000 7f9959c00700 1 -- 172.21.15.62:0/2682636937 shutdown_connections 2024-04-20T21:40:09.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:09.587+0000 7f9959c00700 1 -- 172.21.15.62:0/2682636937 wait complete. 2024-04-20T21:40:09.588 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 18 2024-04-20T21:40:09.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:09.755+0000 7f082c000700 1 -- 172.21.15.62:0/2991653927 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0824101d30 msgr2=0x7f082410a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:40:09.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:09.755+0000 7f082c000700 1 --2- 172.21.15.62:0/2991653927 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0824101d30 0x7f082410a230 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f0810009910 tx=0x7f0810009c20 comp rx=0 tx=0).stop 2024-04-20T21:40:09.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:09.756+0000 7f082c000700 1 -- 172.21.15.62:0/2991653927 shutdown_connections 2024-04-20T21:40:09.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:09.756+0000 7f082c000700 1 --2- 172.21.15.62:0/2991653927 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0824101d30 0x7f082410a230 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:40:09.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:09.756+0000 7f082c000700 1 --2- 172.21.15.62:0/2991653927 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0824101420 0x7f08241017f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:40:09.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:09.756+0000 7f082c000700 1 -- 172.21.15.62:0/2991653927 >> 172.21.15.62:0/2991653927 conn(0x7f08240fac80 msgr2=0x7f08240fd090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:40:09.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:09.756+0000 7f082c000700 1 -- 172.21.15.62:0/2991653927 shutdown_connections 2024-04-20T21:40:09.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:09.756+0000 7f082c000700 1 -- 172.21.15.62:0/2991653927 wait complete. 2024-04-20T21:40:09.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:09.757+0000 7f082c000700 1 Processor -- start 2024-04-20T21:40:09.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:09.757+0000 7f082c000700 1 -- start start 2024-04-20T21:40:09.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:09.757+0000 7f082c000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0824101420 0x7f08240ff080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:40:09.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:09.757+0000 7f082c000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0824101d30 0x7f08240ff5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:40:09.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:09.757+0000 7f082c000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08240ffb00 con 0x7f0824101d30 2024-04-20T21:40:09.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:09.757+0000 7f082c000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08240ffc40 con 0x7f0824101420 2024-04-20T21:40:09.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:09.758+0000 7f0828c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0824101d30 0x7f08240ff5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:40:09.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:09.758+0000 7f0828c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0824101d30 0x7f08240ff5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49628/0 (socket says 172.21.15.62:49628) 2024-04-20T21:40:09.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:09.758+0000 7f0828c00700 1 -- 172.21.15.62:0/2351951742 learned_addr learned my addr 172.21.15.62:0/2351951742 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:40:09.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:09.758+0000 7f0829600700 1 --2- 172.21.15.62:0/2351951742 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0824101420 0x7f08240ff080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:40:09.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:09.758+0000 7f0828c00700 1 -- 172.21.15.62:0/2351951742 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0824101420 msgr2=0x7f08240ff080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:40:09.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:09.758+0000 7f0828c00700 1 --2- 172.21.15.62:0/2351951742 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0824101420 0x7f08240ff080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:40:09.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:09.758+0000 7f0828c00700 1 -- 172.21.15.62:0/2351951742 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08100095f0 con 0x7f0824101d30 2024-04-20T21:40:09.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:09.759+0000 7f0828c00700 1 --2- 172.21.15.62:0/2351951742 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0824101d30 0x7f08240ff5c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f081000b5c0 tx=0x7f081000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:40:09.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:09.759+0000 7f0822000700 1 -- 172.21.15.62:0/2351951742 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f081001d070 con 0x7f0824101d30 2024-04-20T21:40:09.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:09.759+0000 7f0822000700 1 -- 172.21.15.62:0/2351951742 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0810005e00 con 0x7f0824101d30 2024-04-20T21:40:09.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:09.759+0000 7f082c000700 1 -- 172.21.15.62:0/2351951742 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08240ffec0 con 0x7f0824101d30 2024-04-20T21:40:09.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:09.759+0000 7f0822000700 1 -- 172.21.15.62:0/2351951742 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0810017400 con 0x7f0824101d30 2024-04-20T21:40:09.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:09.759+0000 7f082c000700 1 -- 172.21.15.62:0/2351951742 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08241a2380 con 0x7f0824101d30 2024-04-20T21:40:09.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:09.761+0000 7f082c000700 1 -- 172.21.15.62:0/2351951742 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08241078a0 con 0x7f0824101d30 2024-04-20T21:40:09.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:09.761+0000 7f0822000700 1 -- 172.21.15.62:0/2351951742 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f081000f460 con 0x7f0824101d30 2024-04-20T21:40:09.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:09.763+0000 7f0822000700 1 --2- 172.21.15.62:0/2351951742 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f080c06ecc0 0x7f080c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:40:09.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:09.763+0000 7f0822000700 1 -- 172.21.15.62:0/2351951742 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f0810090200 con 0x7f0824101d30 2024-04-20T21:40:09.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:09.763+0000 7f0829600700 1 --2- 172.21.15.62:0/2351951742 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f080c06ecc0 0x7f080c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:40:09.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:09.764+0000 7f0829600700 1 --2- 172.21.15.62:0/2351951742 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f080c06ecc0 0x7f080c071170 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f0818005950 tx=0x7f08180058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:40:09.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:09.768+0000 7f0822000700 1 -- 172.21.15.62:0/2351951742 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f08100590e0 con 0x7f0824101d30 2024-04-20T21:40:09.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:09 smithi189 ceph-mon[57022]: from='client.34314 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:40:09.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:09 smithi189 ceph-mon[57022]: pgmap v125: 65 pgs: 65 active+clean; 315 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-20T21:40:09.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:09 smithi189 ceph-mon[57022]: from='client.34318 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:40:09.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:09 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/2383894347' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:40:09.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:09 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:40:09.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:09 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/2682636937' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T21:40:10.014 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:09 smithi062 ceph-mon[76322]: from='client.34314 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:40:10.014 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:09 smithi062 ceph-mon[76322]: pgmap v125: 65 pgs: 65 active+clean; 315 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-20T21:40:10.014 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:09 smithi062 ceph-mon[76322]: from='client.34318 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:40:10.014 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:09 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/2383894347' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:40:10.014 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:09 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:40:10.014 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:09 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/2682636937' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T21:40:10.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:10.022+0000 7f082c000700 1 -- 172.21.15.62:0/2351951742 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0824061190 con 0x7f080c06ecc0 2024-04-20T21:40:10.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:10.026+0000 7f0822000700 1 -- 172.21.15.62:0/2351951742 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f0824061190 con 0x7f080c06ecc0 2024-04-20T21:40:10.026 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:40:10.026 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T21:40:10.026 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T21:40:10.026 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-20T21:40:10.026 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T21:40:10.027 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr", 2024-04-20T21:40:10.027 INFO:teuthology.orchestra.run.smithi062.stdout: "crash", 2024-04-20T21:40:10.027 INFO:teuthology.orchestra.run.smithi062.stdout: "mon" 2024-04-20T21:40:10.027 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T21:40:10.027 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "9/23 daemons upgraded", 2024-04-20T21:40:10.027 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading osd daemons", 2024-04-20T21:40:10.027 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T21:40:10.027 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:40:10.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:10.029+0000 7f082c000700 1 -- 172.21.15.62:0/2351951742 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f080c06ecc0 msgr2=0x7f080c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:40:10.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:10.029+0000 7f082c000700 1 --2- 172.21.15.62:0/2351951742 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f080c06ecc0 0x7f080c071170 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f0818005950 tx=0x7f08180058e0 comp rx=0 tx=0).stop 2024-04-20T21:40:10.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:10.029+0000 7f082c000700 1 -- 172.21.15.62:0/2351951742 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0824101d30 msgr2=0x7f08240ff5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:40:10.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:10.029+0000 7f082c000700 1 --2- 172.21.15.62:0/2351951742 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0824101d30 0x7f08240ff5c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f081000b5c0 tx=0x7f081000be20 comp rx=0 tx=0).stop 2024-04-20T21:40:10.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:10.029+0000 7f082c000700 1 -- 172.21.15.62:0/2351951742 shutdown_connections 2024-04-20T21:40:10.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:10.029+0000 7f082c000700 1 --2- 172.21.15.62:0/2351951742 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f080c06ecc0 0x7f080c071170 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:40:10.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:10.029+0000 7f082c000700 1 --2- 172.21.15.62:0/2351951742 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0824101420 0x7f08240ff080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:40:10.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:10.029+0000 7f082c000700 1 --2- 172.21.15.62:0/2351951742 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0824101d30 0x7f08240ff5c0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:40:10.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:10.029+0000 7f082c000700 1 -- 172.21.15.62:0/2351951742 >> 172.21.15.62:0/2351951742 conn(0x7f08240fac80 msgr2=0x7f08241049e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:40:10.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:10.029+0000 7f082c000700 1 -- 172.21.15.62:0/2351951742 shutdown_connections 2024-04-20T21:40:10.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:10.030+0000 7f082c000700 1 -- 172.21.15.62:0/2351951742 wait complete. 2024-04-20T21:40:10.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:10.194+0000 7f86c3e00700 1 -- 172.21.15.62:0/1259830249 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f86bc073a00 msgr2=0x7f86bc110d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:40:10.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:10.194+0000 7f86c3e00700 1 --2- 172.21.15.62:0/1259830249 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f86bc073a00 0x7f86bc110d50 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f86a8009910 tx=0x7f86a8009c20 comp rx=0 tx=0).stop 2024-04-20T21:40:10.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:10.195+0000 7f86c3e00700 1 -- 172.21.15.62:0/1259830249 shutdown_connections 2024-04-20T21:40:10.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:10.195+0000 7f86c3e00700 1 --2- 172.21.15.62:0/1259830249 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f86bc073a00 0x7f86bc110d50 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:40:10.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:10.195+0000 7f86c3e00700 1 --2- 172.21.15.62:0/1259830249 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f86bc0730f0 0x7f86bc0734c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:40:10.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:10.195+0000 7f86c3e00700 1 -- 172.21.15.62:0/1259830249 >> 172.21.15.62:0/1259830249 conn(0x7f86bc0fbda0 msgr2=0x7f86bc0fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:40:10.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:10.195+0000 7f86c3e00700 1 -- 172.21.15.62:0/1259830249 shutdown_connections 2024-04-20T21:40:10.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:10.195+0000 7f86c3e00700 1 -- 172.21.15.62:0/1259830249 wait complete. 2024-04-20T21:40:10.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:10.196+0000 7f86c3e00700 1 Processor -- start 2024-04-20T21:40:10.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:10.196+0000 7f86c3e00700 1 -- start start 2024-04-20T21:40:10.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:10.196+0000 7f86c3e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f86bc0730f0 0x7f86bc19c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:40:10.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:10.196+0000 7f86c3e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f86bc073a00 0x7f86bc19c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:40:10.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:10.196+0000 7f86c3e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86bc19d050 con 0x7f86bc0730f0 2024-04-20T21:40:10.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:10.196+0000 7f86c3e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86bc1a0de0 con 0x7f86bc073a00 2024-04-20T21:40:10.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:10.197+0000 7f86c1400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f86bc0730f0 0x7f86bc19c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:40:10.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:10.197+0000 7f86c1400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f86bc0730f0 0x7f86bc19c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49642/0 (socket says 172.21.15.62:49642) 2024-04-20T21:40:10.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:10.197+0000 7f86c1400700 1 -- 172.21.15.62:0/1952650922 learned_addr learned my addr 172.21.15.62:0/1952650922 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:40:10.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:10.197+0000 7f86c0a00700 1 --2- 172.21.15.62:0/1952650922 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f86bc073a00 0x7f86bc19c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:40:10.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:10.197+0000 7f86c1400700 1 -- 172.21.15.62:0/1952650922 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f86bc073a00 msgr2=0x7f86bc19c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:40:10.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:10.197+0000 7f86c1400700 1 --2- 172.21.15.62:0/1952650922 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f86bc073a00 0x7f86bc19c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:40:10.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:10.197+0000 7f86c1400700 1 -- 172.21.15.62:0/1952650922 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86a80095f0 con 0x7f86bc0730f0 2024-04-20T21:40:10.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:10.197+0000 7f86c0a00700 1 --2- 172.21.15.62:0/1952650922 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f86bc073a00 0x7f86bc19c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T21:40:10.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:10.198+0000 7f86c1400700 1 --2- 172.21.15.62:0/1952650922 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f86bc0730f0 0x7f86bc19c430 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f86b00060b0 tx=0x7f86b000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:40:10.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:10.198+0000 7f86ba000700 1 -- 172.21.15.62:0/1952650922 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f86b000dda0 con 0x7f86bc0730f0 2024-04-20T21:40:10.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:10.198+0000 7f86ba000700 1 -- 172.21.15.62:0/1952650922 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f86b000f040 con 0x7f86bc0730f0 2024-04-20T21:40:10.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:10.198+0000 7f86ba000700 1 -- 172.21.15.62:0/1952650922 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f86b0014790 con 0x7f86bc0730f0 2024-04-20T21:40:10.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:10.198+0000 7f86c3e00700 1 -- 172.21.15.62:0/1952650922 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86bc1a10c0 con 0x7f86bc0730f0 2024-04-20T21:40:10.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:10.198+0000 7f86c3e00700 1 -- 172.21.15.62:0/1952650922 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86bc1a15c0 con 0x7f86bc0730f0 2024-04-20T21:40:10.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:10.199+0000 7f86c3e00700 1 -- 172.21.15.62:0/1952650922 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86bc10e4d0 con 0x7f86bc0730f0 2024-04-20T21:40:10.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:10.200+0000 7f86ba000700 1 -- 172.21.15.62:0/1952650922 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f86b0014980 con 0x7f86bc0730f0 2024-04-20T21:40:10.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:10.201+0000 7f86ba000700 1 --2- 172.21.15.62:0/1952650922 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f86a406ecc0 0x7f86a4071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:40:10.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:10.202+0000 7f86ba000700 1 -- 172.21.15.62:0/1952650922 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f86b008e8f0 con 0x7f86bc0730f0 2024-04-20T21:40:10.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:10.202+0000 7f86c0a00700 1 --2- 172.21.15.62:0/1952650922 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f86a406ecc0 0x7f86a4071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:40:10.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:10.202+0000 7f86c0a00700 1 --2- 172.21.15.62:0/1952650922 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f86a406ecc0 0x7f86a4071170 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f86a8006010 tx=0x7f86a800b540 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:40:10.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:10.206+0000 7f86ba000700 1 -- 172.21.15.62:0/1952650922 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f86b0057760 con 0x7f86bc0730f0 2024-04-20T21:40:10.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:10.523+0000 7f86c3e00700 1 -- 172.21.15.62:0/1952650922 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f86bc04f2a0 con 0x7f86bc0730f0 2024-04-20T21:40:10.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:10.524+0000 7f86ba000700 1 -- 172.21.15.62:0/1952650922 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7f86b005ae20 con 0x7f86bc0730f0 2024-04-20T21:40:10.523 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-04-20T21:40:10.524 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T21:40:10.524 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T21:40:10.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:10.527+0000 7f86c3e00700 1 -- 172.21.15.62:0/1952650922 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f86a406ecc0 msgr2=0x7f86a4071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:40:10.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:10.527+0000 7f86c3e00700 1 --2- 172.21.15.62:0/1952650922 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f86a406ecc0 0x7f86a4071170 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f86a8006010 tx=0x7f86a800b540 comp rx=0 tx=0).stop 2024-04-20T21:40:10.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:10.527+0000 7f86c3e00700 1 -- 172.21.15.62:0/1952650922 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f86bc0730f0 msgr2=0x7f86bc19c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:40:10.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:10.527+0000 7f86c3e00700 1 --2- 172.21.15.62:0/1952650922 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f86bc0730f0 0x7f86bc19c430 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f86b00060b0 tx=0x7f86b000d3b0 comp rx=0 tx=0).stop 2024-04-20T21:40:10.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:10.527+0000 7f86c3e00700 1 -- 172.21.15.62:0/1952650922 shutdown_connections 2024-04-20T21:40:10.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:10.527+0000 7f86c3e00700 1 --2- 172.21.15.62:0/1952650922 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f86a406ecc0 0x7f86a4071170 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:40:10.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:10.527+0000 7f86c3e00700 1 --2- 172.21.15.62:0/1952650922 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f86bc0730f0 0x7f86bc19c430 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:40:10.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:10.527+0000 7f86c3e00700 1 --2- 172.21.15.62:0/1952650922 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f86bc073a00 0x7f86bc19c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:40:10.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:10.527+0000 7f86c3e00700 1 -- 172.21.15.62:0/1952650922 >> 172.21.15.62:0/1952650922 conn(0x7f86bc0fbda0 msgr2=0x7f86bc102860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:40:10.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:10.527+0000 7f86c3e00700 1 -- 172.21.15.62:0/1952650922 shutdown_connections 2024-04-20T21:40:10.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:10.527+0000 7f86c3e00700 1 -- 172.21.15.62:0/1952650922 wait complete. 2024-04-20T21:40:10.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:10 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/1952650922' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T21:40:10.971 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:10 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/1952650922' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T21:40:11.658 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 20 21:40:11 smithi189 systemd[1]: Stopping Ceph osd.3 for 61ae3628-ff5b-11ee-bc93-c7b262605968... 2024-04-20T21:40:11.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:11 smithi189 ceph-mon[57022]: from='client.34330 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:40:11.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:11 smithi189 ceph-mon[57022]: pgmap v126: 65 pgs: 65 active+clean; 315 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-20T21:40:11.987 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 20 21:40:11 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-3[27490]: 2024-04-20T21:40:11.756+0000 7f6f94200700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-20T21:40:11.987 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 20 21:40:11 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-3[27490]: 2024-04-20T21:40:11.756+0000 7f6f94200700 -1 osd.3 63 *** Got signal Terminated *** 2024-04-20T21:40:11.987 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 20 21:40:11 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-3[27490]: 2024-04-20T21:40:11.756+0000 7f6f94200700 -1 osd.3 63 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-20T21:40:12.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:11 smithi062 ceph-mon[76322]: from='client.34330 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:40:12.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:11 smithi062 ceph-mon[76322]: pgmap v126: 65 pgs: 65 active+clean; 315 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-20T21:40:12.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:12 smithi189 ceph-mon[57022]: osd.3 marked itself down and dead 2024-04-20T21:40:13.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:12 smithi062 ceph-mon[76322]: osd.3 marked itself down and dead 2024-04-20T21:40:13.622 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 20 21:40:13 smithi189 podman[66048]: 2024-04-20 21:40:13.330831973 +0000 UTC m=+1.693840773 container stop 611c1e2426937bd4447f245668c7aaf36324ad2fd303dcc01cd7922a945bbca4 (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-3, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, org.label-schema.license=GPLv2, org.label-schema.build-date=20231212, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.0, org.label-schema.schema-version=1.0, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-20T21:40:13.622 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 20 21:40:13 smithi189 podman[66048]: 2024-04-20 21:40:13.356836282 +0000 UTC m=+1.719845074 container died 611c1e2426937bd4447f245668c7aaf36324ad2fd303dcc01cd7922a945bbca4 (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-3, org.label-schema.schema-version=1.0, org.label-schema.build-date=20231212, io.buildah.version=1.29.1, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, ceph=True, RELEASE=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-04-20T21:40:13.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:13 smithi189 ceph-mon[57022]: pgmap v127: 65 pgs: 65 active+clean; 315 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:40:13.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:13 smithi189 ceph-mon[57022]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-20T21:40:13.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:13 smithi189 ceph-mon[57022]: osdmap e64: 6 total, 5 up, 6 in 2024-04-20T21:40:13.988 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 20 21:40:13 smithi189 podman[66048]: 2024-04-20 21:40:13.797510452 +0000 UTC m=+2.160519256 container cleanup 611c1e2426937bd4447f245668c7aaf36324ad2fd303dcc01cd7922a945bbca4 (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-3, CEPH_POINT_RELEASE=-18.2.0, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, io.buildah.version=1.29.1, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20231212) 2024-04-20T21:40:13.988 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 20 21:40:13 smithi189 bash[66048]: ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-3 2024-04-20T21:40:14.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:13 smithi062 ceph-mon[76322]: pgmap v127: 65 pgs: 65 active+clean; 315 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:40:14.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:13 smithi062 ceph-mon[76322]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-20T21:40:14.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:13 smithi062 ceph-mon[76322]: osdmap e64: 6 total, 5 up, 6 in 2024-04-20T21:40:14.487 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 20 21:40:14 smithi189 podman[66067]: 2024-04-20 21:40:14.023897535 +0000 UTC m=+0.677161882 container remove 611c1e2426937bd4447f245668c7aaf36324ad2fd303dcc01cd7922a945bbca4 (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-3, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-18.2.0, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, GIT_BRANCH=HEAD, org.label-schema.build-date=20231212, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=HEAD) 2024-04-20T21:40:14.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:14 smithi189 ceph-mon[57022]: osdmap e65: 6 total, 5 up, 6 in 2024-04-20T21:40:14.987 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 20 21:40:14 smithi189 podman[66155]: 2024-04-20 21:40:14.731861273 +0000 UTC m=+0.033494097 image pull 76bc6b91b726a03b8b736eeeb35fc1ffd1469e23c7d60f5b444a217528d8906d quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 2024-04-20T21:40:14.987 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 20 21:40:14 smithi189 podman[66155]: 2024-04-20 21:40:14.866616667 +0000 UTC m=+0.168249534 container create 3d1117d748f85f3b24a9cfdd1bee1228b3a93e2071a665c7ea2c9cc7c0623401 (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-3-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, GIT_BRANCH=HEAD) 2024-04-20T21:40:15.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:14 smithi062 ceph-mon[76322]: osdmap e65: 6 total, 5 up, 6 in 2024-04-20T21:40:15.488 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 20 21:40:15 smithi189 podman[66155]: 2024-04-20 21:40:15.162634139 +0000 UTC m=+0.464267022 container init 3d1117d748f85f3b24a9cfdd1bee1228b3a93e2071a665c7ea2c9cc7c0623401 (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-3-deactivate, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-20T21:40:15.488 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 20 21:40:15 smithi189 podman[66155]: 2024-04-20 21:40:15.174253736 +0000 UTC m=+0.475886589 container start 3d1117d748f85f3b24a9cfdd1bee1228b3a93e2071a665c7ea2c9cc7c0623401 (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-3-deactivate, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, io.buildah.version=1.33.5) 2024-04-20T21:40:15.488 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 20 21:40:15 smithi189 podman[66155]: 2024-04-20 21:40:15.260657236 +0000 UTC m=+0.562290089 container attach 3d1117d748f85f3b24a9cfdd1bee1228b3a93e2071a665c7ea2c9cc7c0623401 (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-3-deactivate, org.label-schema.build-date=20240416, ceph=True, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-20T21:40:15.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:15 smithi189 ceph-mon[57022]: pgmap v130: 65 pgs: 13 peering, 13 stale+active+clean, 39 active+clean; 315 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:40:15.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:15 smithi189 ceph-mon[57022]: Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2024-04-20T21:40:15.988 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 20 21:40:15 smithi189 podman[66155]: 2024-04-20 21:40:15.627001809 +0000 UTC m=+0.928634695 container died 3d1117d748f85f3b24a9cfdd1bee1228b3a93e2071a665c7ea2c9cc7c0623401 (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-3-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-20T21:40:16.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:15 smithi062 ceph-mon[76322]: pgmap v130: 65 pgs: 13 peering, 13 stale+active+clean, 39 active+clean; 315 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:40:16.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:15 smithi062 ceph-mon[76322]: Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2024-04-20T21:40:16.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:16 smithi189 ceph-mon[57022]: Health check failed: Degraded data redundancy: 52/366 objects degraded (14.208%), 12 pgs degraded (PG_DEGRADED) 2024-04-20T21:40:16.987 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 20 21:40:16 smithi189 podman[66181]: 2024-04-20 21:40:16.597421478 +0000 UTC m=+0.954482249 container remove 3d1117d748f85f3b24a9cfdd1bee1228b3a93e2071a665c7ea2c9cc7c0623401 (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-3-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5) 2024-04-20T21:40:16.987 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 20 21:40:16 smithi189 systemd[1]: ceph-61ae3628-ff5b-11ee-bc93-c7b262605968@osd.3.service: Deactivated successfully. 2024-04-20T21:40:16.988 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 20 21:40:16 smithi189 systemd[1]: Stopped Ceph osd.3 for 61ae3628-ff5b-11ee-bc93-c7b262605968. 2024-04-20T21:40:16.988 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 20 21:40:16 smithi189 systemd[1]: ceph-61ae3628-ff5b-11ee-bc93-c7b262605968@osd.3.service: Consumed 2min 4.319s CPU time. 2024-04-20T21:40:17.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:16 smithi062 ceph-mon[76322]: Health check failed: Degraded data redundancy: 52/366 objects degraded (14.208%), 12 pgs degraded (PG_DEGRADED) 2024-04-20T21:40:17.368 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 20 21:40:17 smithi189 systemd[1]: Starting Ceph osd.3 for 61ae3628-ff5b-11ee-bc93-c7b262605968... 2024-04-20T21:40:17.644 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 20 21:40:17 smithi189 podman[66285]: 2024-04-20 21:40:17.273790623 +0000 UTC m=+0.036257310 image pull 76bc6b91b726a03b8b736eeeb35fc1ffd1469e23c7d60f5b444a217528d8906d quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 2024-04-20T21:40:17.644 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 20 21:40:17 smithi189 podman[66285]: 2024-04-20 21:40:17.382294138 +0000 UTC m=+0.144760862 container create 63e43214891359baa70438b0514b8414ab7b10975eb7bc84c110a5217269a367 (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-3-activate, org.label-schema.build-date=20240416, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5) 2024-04-20T21:40:17.644 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 20 21:40:17 smithi189 podman[66285]: 2024-04-20 21:40:17.644615904 +0000 UTC m=+0.407082635 container init 63e43214891359baa70438b0514b8414ab7b10975eb7bc84c110a5217269a367 (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-3-activate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-04-20T21:40:17.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:17 smithi189 ceph-mon[57022]: pgmap v131: 65 pgs: 12 active+undersized+degraded, 13 peering, 13 active+undersized, 27 active+clean; 315 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 52/366 objects degraded (14.208%) 2024-04-20T21:40:17.987 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 20 21:40:17 smithi189 podman[66285]: 2024-04-20 21:40:17.656917853 +0000 UTC m=+0.419384546 container start 63e43214891359baa70438b0514b8414ab7b10975eb7bc84c110a5217269a367 (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-3-activate, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, ceph=True, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-04-20T21:40:17.987 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 20 21:40:17 smithi189 podman[66285]: 2024-04-20 21:40:17.757128672 +0000 UTC m=+0.519595403 container attach 63e43214891359baa70438b0514b8414ab7b10975eb7bc84c110a5217269a367 (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-3-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-04-20T21:40:18.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:17 smithi062 ceph-mon[76322]: pgmap v131: 65 pgs: 12 active+undersized+degraded, 13 peering, 13 active+undersized, 27 active+clean; 315 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 52/366 objects degraded (14.208%) 2024-04-20T21:40:18.988 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 20 21:40:18 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-3-activate[66300]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-20T21:40:18.988 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 20 21:40:18 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-3-activate[66300]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-20T21:40:18.988 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 20 21:40:18 smithi189 bash[66285]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-20T21:40:18.988 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 20 21:40:18 smithi189 bash[66285]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-20T21:40:18.988 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 20 21:40:18 smithi189 bash[66285]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-20T21:40:18.988 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 20 21:40:18 smithi189 bash[66285]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-04-20T21:40:18.989 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 20 21:40:18 smithi189 bash[66285]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-04-20T21:40:18.989 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 20 21:40:18 smithi189 bash[66285]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-04-20T21:40:18.989 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 20 21:40:18 smithi189 bash[66285]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3/block 2024-04-20T21:40:18.989 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 20 21:40:18 smithi189 bash[66285]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-20T21:40:18.989 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 20 21:40:18 smithi189 bash[66285]: --> ceph-volume raw activate successful for osd ID: 3 2024-04-20T21:40:18.989 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 20 21:40:18 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-3-activate[66300]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-20T21:40:18.989 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 20 21:40:18 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-3-activate[66300]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-04-20T21:40:18.989 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 20 21:40:18 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-3-activate[66300]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-04-20T21:40:18.989 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 20 21:40:18 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-3-activate[66300]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-04-20T21:40:18.989 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 20 21:40:18 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-3-activate[66300]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3/block 2024-04-20T21:40:18.989 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 20 21:40:18 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-3-activate[66300]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-20T21:40:18.989 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 20 21:40:18 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-3-activate[66300]: --> ceph-volume raw activate successful for osd ID: 3 2024-04-20T21:40:18.989 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 20 21:40:18 smithi189 podman[66285]: 2024-04-20 21:40:18.651921882 +0000 UTC m=+1.414388616 container died 63e43214891359baa70438b0514b8414ab7b10975eb7bc84c110a5217269a367 (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-3-activate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-04-20T21:40:19.709 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 20 21:40:19 smithi189 podman[66425]: 2024-04-20 21:40:19.354725357 +0000 UTC m=+0.684177785 container remove 63e43214891359baa70438b0514b8414ab7b10975eb7bc84c110a5217269a367 (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-3-activate, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-04-20T21:40:19.990 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:19 smithi189 ceph-mon[57022]: pgmap v132: 65 pgs: 12 active+undersized+degraded, 13 peering, 13 active+undersized, 27 active+clean; 315 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 52/366 objects degraded (14.208%) 2024-04-20T21:40:19.990 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:19 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:40:19.990 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:19 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:40:19.990 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 20 21:40:19 smithi189 podman[66490]: 2024-04-20 21:40:19.615735318 +0000 UTC m=+0.034146218 image pull 76bc6b91b726a03b8b736eeeb35fc1ffd1469e23c7d60f5b444a217528d8906d quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 2024-04-20T21:40:19.990 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 20 21:40:19 smithi189 podman[66490]: 2024-04-20 21:40:19.730731115 +0000 UTC m=+0.149142050 container create 8f97b17ae0ebb32dda871a6d08fd56aaba57016a65a45f736c36aecb0de36b6f (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-3, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, org.label-schema.license=GPLv2) 2024-04-20T21:40:20.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:19 smithi062 ceph-mon[76322]: pgmap v132: 65 pgs: 12 active+undersized+degraded, 13 peering, 13 active+undersized, 27 active+clean; 315 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 52/366 objects degraded (14.208%) 2024-04-20T21:40:20.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:19 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:40:20.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:19 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:40:20.487 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 20 21:40:20 smithi189 podman[66490]: 2024-04-20 21:40:20.027055705 +0000 UTC m=+0.445466684 container init 8f97b17ae0ebb32dda871a6d08fd56aaba57016a65a45f736c36aecb0de36b6f (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-3, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux ) 2024-04-20T21:40:20.487 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 20 21:40:20 smithi189 podman[66490]: 2024-04-20 21:40:20.037662722 +0000 UTC m=+0.456073628 container start 8f97b17ae0ebb32dda871a6d08fd56aaba57016a65a45f736c36aecb0de36b6f (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-3, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-04-20T21:40:20.487 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 20 21:40:20 smithi189 ceph-osd[66508]: -- 172.21.15.189:0/3842073502 <== mon.1 v2:172.21.15.189:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c2aa91b680 con 0x55c2aa8e4c00 2024-04-20T21:40:20.487 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 20 21:40:20 smithi189 bash[66490]: 8f97b17ae0ebb32dda871a6d08fd56aaba57016a65a45f736c36aecb0de36b6f 2024-04-20T21:40:20.487 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 20 21:40:20 smithi189 systemd[1]: Started Ceph osd.3 for 61ae3628-ff5b-11ee-bc93-c7b262605968. 2024-04-20T21:40:20.986 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 20 21:40:20 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-3[66504]: 2024-04-20T21:40:20.622+0000 7f05b7fff7c0 -1 Falling back to public interface 2024-04-20T21:40:21.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:21 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:40:21.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:21 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:40:21.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:21 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:40:21.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:21 smithi062 ceph-mon[76322]: pgmap v133: 65 pgs: 20 active+undersized+degraded, 18 active+undersized, 27 active+clean; 315 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 77/366 objects degraded (21.038%) 2024-04-20T21:40:21.638 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:21 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:40:21.638 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:21 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:40:21.638 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:21 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:40:21.638 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:21 smithi189 ceph-mon[57022]: pgmap v133: 65 pgs: 20 active+undersized+degraded, 18 active+undersized, 27 active+clean; 315 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 77/366 objects degraded (21.038%) 2024-04-20T21:40:22.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:22 smithi062 ceph-mon[76322]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-04-20T21:40:22.537 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:22 smithi189 ceph-mon[57022]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-04-20T21:40:23.530 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:23 smithi189 ceph-mon[57022]: pgmap v134: 65 pgs: 20 active+undersized+degraded, 18 active+undersized, 27 active+clean; 315 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 77/366 objects degraded (21.038%) 2024-04-20T21:40:23.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:23 smithi062 ceph-mon[76322]: pgmap v134: 65 pgs: 20 active+undersized+degraded, 18 active+undersized, 27 active+clean; 315 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 77/366 objects degraded (21.038%) 2024-04-20T21:40:24.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:24 smithi189 ceph-mon[57022]: Health check update: Degraded data redundancy: 77/366 objects degraded (21.038%), 20 pgs degraded (PG_DEGRADED) 2024-04-20T21:40:24.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:24 smithi062 ceph-mon[76322]: Health check update: Degraded data redundancy: 77/366 objects degraded (21.038%), 20 pgs degraded (PG_DEGRADED) 2024-04-20T21:40:26.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:25 smithi189 ceph-mon[57022]: pgmap v135: 65 pgs: 20 active+undersized+degraded, 18 active+undersized, 27 active+clean; 315 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 77/366 objects degraded (21.038%) 2024-04-20T21:40:26.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:25 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:40:26.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:25 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:40:26.237 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 20 21:40:26 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-3[66504]: 2024-04-20T21:40:26.084+0000 7f05b7fff7c0 -1 osd.3 0 read_superblock omap replica is missing. 2024-04-20T21:40:26.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:25 smithi062 ceph-mon[76322]: pgmap v135: 65 pgs: 20 active+undersized+degraded, 18 active+undersized, 27 active+clean; 315 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 77/366 objects degraded (21.038%) 2024-04-20T21:40:26.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:25 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:40:26.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:25 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:40:26.859 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 20 21:40:26 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-3[66504]: 2024-04-20T21:40:26.497+0000 7f05b7fff7c0 -1 osd.3 63 log_to_monitors true 2024-04-20T21:40:27.010 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:26 smithi062 ceph-mon[76322]: from='osd.3 [v2:172.21.15.189:6800/3977023570,v1:172.21.15.189:6801/3977023570]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-20T21:40:27.010 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:26 smithi062 ceph-mon[76322]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-20T21:40:27.010 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:26 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:40:27.010 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:26 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:40:27.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:26 smithi189 ceph-mon[57022]: from='osd.3 [v2:172.21.15.189:6800/3977023570,v1:172.21.15.189:6801/3977023570]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-20T21:40:27.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:26 smithi189 ceph-mon[57022]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-20T21:40:27.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:26 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:40:27.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:26 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:40:27.237 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 20 21:40:26 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-3[66504]: 2024-04-20T21:40:26.885+0000 7f05a7400700 -1 osd.3 63 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-20T21:40:27.947 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:27 smithi189 ceph-mon[57022]: pgmap v136: 65 pgs: 20 active+undersized+degraded, 18 active+undersized, 27 active+clean; 315 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 77/366 objects degraded (21.038%) 2024-04-20T21:40:27.948 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:27 smithi189 ceph-mon[57022]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-20T21:40:27.948 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:27 smithi189 ceph-mon[57022]: osdmap e66: 6 total, 5 up, 6 in 2024-04-20T21:40:27.948 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:27 smithi189 ceph-mon[57022]: from='osd.3 [v2:172.21.15.189:6800/3977023570,v1:172.21.15.189:6801/3977023570]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-04-20T21:40:27.948 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:27 smithi189 ceph-mon[57022]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-04-20T21:40:28.145 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:27 smithi062 ceph-mon[76322]: pgmap v136: 65 pgs: 20 active+undersized+degraded, 18 active+undersized, 27 active+clean; 315 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 77/366 objects degraded (21.038%) 2024-04-20T21:40:28.145 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:27 smithi062 ceph-mon[76322]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-20T21:40:28.145 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:27 smithi062 ceph-mon[76322]: osdmap e66: 6 total, 5 up, 6 in 2024-04-20T21:40:28.145 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:27 smithi062 ceph-mon[76322]: from='osd.3 [v2:172.21.15.189:6800/3977023570,v1:172.21.15.189:6801/3977023570]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-04-20T21:40:28.145 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:27 smithi062 ceph-mon[76322]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-04-20T21:40:29.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:28 smithi189 ceph-mon[57022]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-20T21:40:29.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:28 smithi189 ceph-mon[57022]: osd.3 [v2:172.21.15.189:6800/3977023570,v1:172.21.15.189:6801/3977023570] boot 2024-04-20T21:40:29.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:28 smithi189 ceph-mon[57022]: osdmap e67: 6 total, 6 up, 6 in 2024-04-20T21:40:29.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:28 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-20T21:40:29.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:28 smithi062 ceph-mon[76322]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-20T21:40:29.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:28 smithi062 ceph-mon[76322]: osd.3 [v2:172.21.15.189:6800/3977023570,v1:172.21.15.189:6801/3977023570] boot 2024-04-20T21:40:29.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:28 smithi062 ceph-mon[76322]: osdmap e67: 6 total, 6 up, 6 in 2024-04-20T21:40:29.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:28 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-20T21:40:30.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:29 smithi189 ceph-mon[57022]: pgmap v139: 65 pgs: 20 active+undersized+degraded, 18 active+undersized, 27 active+clean; 315 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s; 77/366 objects degraded (21.038%) 2024-04-20T21:40:30.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:29 smithi189 ceph-mon[57022]: osdmap e68: 6 total, 6 up, 6 in 2024-04-20T21:40:30.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:29 smithi062 ceph-mon[76322]: pgmap v139: 65 pgs: 20 active+undersized+degraded, 18 active+undersized, 27 active+clean; 315 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s; 77/366 objects degraded (21.038%) 2024-04-20T21:40:30.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:29 smithi062 ceph-mon[76322]: osdmap e68: 6 total, 6 up, 6 in 2024-04-20T21:40:31.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:30 smithi062 ceph-mon[76322]: pgmap v141: 65 pgs: 10 peering, 11 active+undersized+degraded, 9 active+undersized, 35 active+clean; 315 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s; 49/366 objects degraded (13.388%) 2024-04-20T21:40:31.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:30 smithi062 ceph-mon[76322]: Health check update: Degraded data redundancy: 49/366 objects degraded (13.388%), 11 pgs degraded (PG_DEGRADED) 2024-04-20T21:40:31.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:30 smithi189 ceph-mon[57022]: pgmap v141: 65 pgs: 10 peering, 11 active+undersized+degraded, 9 active+undersized, 35 active+clean; 315 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s; 49/366 objects degraded (13.388%) 2024-04-20T21:40:31.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:30 smithi189 ceph-mon[57022]: Health check update: Degraded data redundancy: 49/366 objects degraded (13.388%), 11 pgs degraded (PG_DEGRADED) 2024-04-20T21:40:33.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:33 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:40:33.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:33 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:40:33.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:33 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:40:33.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:33 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T21:40:33.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:33 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:40:33.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:33 smithi189 ceph-mon[57022]: pgmap v142: 65 pgs: 10 peering, 55 active+clean; 315 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:40:33.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:33 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:40:33.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:33 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:40:33.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:33 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:40:33.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:33 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:40:33.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:33 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-20T21:40:33.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:33 smithi189 ceph-mon[57022]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-20T21:40:33.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:33 smithi189 ceph-mon[57022]: Upgrade: unsafe to stop osd(s) at this time (3 PGs are or would become offline) 2024-04-20T21:40:33.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:33 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:40:33.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:33 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:40:33.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:33 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:40:33.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:33 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T21:40:33.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:33 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:40:33.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:33 smithi062 ceph-mon[76322]: pgmap v142: 65 pgs: 10 peering, 55 active+clean; 315 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:40:33.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:33 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:40:33.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:33 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:40:33.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:33 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:40:33.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:33 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:40:33.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:33 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-20T21:40:33.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:33 smithi062 ceph-mon[76322]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-20T21:40:33.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:33 smithi062 ceph-mon[76322]: Upgrade: unsafe to stop osd(s) at this time (3 PGs are or would become offline) 2024-04-20T21:40:34.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:34 smithi189 ceph-mon[57022]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 49/366 objects degraded (13.388%), 11 pgs degraded) 2024-04-20T21:40:34.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:34 smithi062 ceph-mon[76322]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 49/366 objects degraded (13.388%), 11 pgs degraded) 2024-04-20T21:40:35.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:35 smithi189 ceph-mon[57022]: pgmap v143: 65 pgs: 65 active+clean; 315 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s rd, 3 op/s 2024-04-20T21:40:35.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:35 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:40:35.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:35 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:40:35.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:35 smithi062 ceph-mon[76322]: pgmap v143: 65 pgs: 65 active+clean; 315 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s rd, 3 op/s 2024-04-20T21:40:35.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:35 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:40:35.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:35 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:40:37.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:37 smithi189 ceph-mon[57022]: pgmap v144: 65 pgs: 65 active+clean; 315 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-20T21:40:37.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:37 smithi062 ceph-mon[76322]: pgmap v144: 65 pgs: 65 active+clean; 315 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-20T21:40:39.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:39 smithi189 ceph-mon[57022]: pgmap v145: 65 pgs: 65 active+clean; 315 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-20T21:40:39.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:39 smithi062 ceph-mon[76322]: pgmap v145: 65 pgs: 65 active+clean; 315 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-20T21:40:40.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:40.706+0000 7fbddd400700 1 -- 172.21.15.62:0/1110521291 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbdd8100e80 msgr2=0x7fbdd8104e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:40:40.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:40.706+0000 7fbddd400700 1 --2- 172.21.15.62:0/1110521291 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbdd8100e80 0x7fbdd8104e50 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fbdc4009910 tx=0x7fbdc4009c20 comp rx=0 tx=0).stop 2024-04-20T21:40:40.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:40.706+0000 7fbddd400700 1 -- 172.21.15.62:0/1110521291 shutdown_connections 2024-04-20T21:40:40.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:40.706+0000 7fbddd400700 1 --2- 172.21.15.62:0/1110521291 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbdd8100e80 0x7fbdd8104e50 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:40:40.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:40.706+0000 7fbddd400700 1 --2- 172.21.15.62:0/1110521291 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbdd81004e0 0x7fbdd81008b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:40:40.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:40.706+0000 7fbddd400700 1 -- 172.21.15.62:0/1110521291 >> 172.21.15.62:0/1110521291 conn(0x7fbdd80fbda0 msgr2=0x7fbdd80fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:40:40.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:40.706+0000 7fbddd400700 1 -- 172.21.15.62:0/1110521291 shutdown_connections 2024-04-20T21:40:40.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:40.706+0000 7fbddd400700 1 -- 172.21.15.62:0/1110521291 wait complete. 2024-04-20T21:40:40.707 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:40 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:40:40.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:40.707+0000 7fbddd400700 1 Processor -- start 2024-04-20T21:40:40.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:40.708+0000 7fbddd400700 1 -- start start 2024-04-20T21:40:40.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:40.708+0000 7fbddd400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbdd81004e0 0x7fbdd8195f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:40:40.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:40.708+0000 7fbddd400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbdd8100e80 0x7fbdd8196450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:40:40.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:40.708+0000 7fbddd400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdd8196b30 con 0x7fbdd8100e80 2024-04-20T21:40:40.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:40.708+0000 7fbddd400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdd819a8c0 con 0x7fbdd81004e0 2024-04-20T21:40:40.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:40.708+0000 7fbdd5e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbdd8100e80 0x7fbdd8196450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:40:40.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:40.708+0000 7fbdd5e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbdd8100e80 0x7fbdd8196450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53876/0 (socket says 172.21.15.62:53876) 2024-04-20T21:40:40.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:40.708+0000 7fbdd5e00700 1 -- 172.21.15.62:0/718217392 learned_addr learned my addr 172.21.15.62:0/718217392 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:40:40.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:40.709+0000 7fbdd6800700 1 --2- 172.21.15.62:0/718217392 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbdd81004e0 0x7fbdd8195f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:40:40.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:40.709+0000 7fbdd5e00700 1 -- 172.21.15.62:0/718217392 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbdd81004e0 msgr2=0x7fbdd8195f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:40:40.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:40.709+0000 7fbdd5e00700 1 --2- 172.21.15.62:0/718217392 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbdd81004e0 0x7fbdd8195f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:40:40.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:40.709+0000 7fbdd5e00700 1 -- 172.21.15.62:0/718217392 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdc40095f0 con 0x7fbdd8100e80 2024-04-20T21:40:40.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:40.709+0000 7fbdd5e00700 1 --2- 172.21.15.62:0/718217392 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbdd8100e80 0x7fbdd8196450 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fbdc400bda0 tx=0x7fbdc400be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:40:40.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:40.709+0000 7fbdcf400700 1 -- 172.21.15.62:0/718217392 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbdc401d070 con 0x7fbdd8100e80 2024-04-20T21:40:40.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:40.710+0000 7fbdcf400700 1 -- 172.21.15.62:0/718217392 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbdc4004fb0 con 0x7fbdd8100e80 2024-04-20T21:40:40.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:40.710+0000 7fbddd400700 1 -- 172.21.15.62:0/718217392 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdd819ab40 con 0x7fbdd8100e80 2024-04-20T21:40:40.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:40.710+0000 7fbdcf400700 1 -- 172.21.15.62:0/718217392 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbdc400f460 con 0x7fbdd8100e80 2024-04-20T21:40:40.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:40.710+0000 7fbddd400700 1 -- 172.21.15.62:0/718217392 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdd819b030 con 0x7fbdd8100e80 2024-04-20T21:40:40.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:40.711+0000 7fbddd400700 1 -- 172.21.15.62:0/718217392 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdd8108850 con 0x7fbdd8100e80 2024-04-20T21:40:40.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:40.712+0000 7fbdcf400700 1 -- 172.21.15.62:0/718217392 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fbdc4022800 con 0x7fbdd8100e80 2024-04-20T21:40:40.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:40.713+0000 7fbdcf400700 1 --2- 172.21.15.62:0/718217392 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fbdbc06ecc0 0x7fbdbc071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:40:40.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:40.713+0000 7fbdcf400700 1 -- 172.21.15.62:0/718217392 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fbdc4090390 con 0x7fbdd8100e80 2024-04-20T21:40:40.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:40.713+0000 7fbdd6800700 1 --2- 172.21.15.62:0/718217392 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fbdbc06ecc0 0x7fbdbc071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:40:40.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:40.714+0000 7fbdd6800700 1 --2- 172.21.15.62:0/718217392 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fbdbc06ecc0 0x7fbdbc071170 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fbdc8005fd0 tx=0x7fbdc800a560 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:40:40.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:40.718+0000 7fbdcf400700 1 -- 172.21.15.62:0/718217392 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fbdc4059320 con 0x7fbdd8100e80 2024-04-20T21:40:40.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:40 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:40:40.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:40.978+0000 7fbddd400700 1 -- 172.21.15.62:0/718217392 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fbdd819b250 con 0x7fbdbc06ecc0 2024-04-20T21:40:40.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:40.979+0000 7fbdcf400700 1 -- 172.21.15.62:0/718217392 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7fbdd819b250 con 0x7fbdbc06ecc0 2024-04-20T21:40:40.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:40.982+0000 7fbddd400700 1 -- 172.21.15.62:0/718217392 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fbdbc06ecc0 msgr2=0x7fbdbc071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:40:40.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:40.982+0000 7fbddd400700 1 --2- 172.21.15.62:0/718217392 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fbdbc06ecc0 0x7fbdbc071170 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fbdc8005fd0 tx=0x7fbdc800a560 comp rx=0 tx=0).stop 2024-04-20T21:40:40.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:40.982+0000 7fbddd400700 1 -- 172.21.15.62:0/718217392 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbdd8100e80 msgr2=0x7fbdd8196450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:40:40.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:40.982+0000 7fbddd400700 1 --2- 172.21.15.62:0/718217392 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbdd8100e80 0x7fbdd8196450 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fbdc400bda0 tx=0x7fbdc400be80 comp rx=0 tx=0).stop 2024-04-20T21:40:40.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:40.983+0000 7fbddd400700 1 -- 172.21.15.62:0/718217392 shutdown_connections 2024-04-20T21:40:40.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:40.983+0000 7fbddd400700 1 --2- 172.21.15.62:0/718217392 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fbdbc06ecc0 0x7fbdbc071170 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:40:40.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:40.983+0000 7fbddd400700 1 --2- 172.21.15.62:0/718217392 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbdd81004e0 0x7fbdd8195f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:40:40.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:40.983+0000 7fbddd400700 1 --2- 172.21.15.62:0/718217392 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbdd8100e80 0x7fbdd8196450 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:40:40.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:40.983+0000 7fbddd400700 1 -- 172.21.15.62:0/718217392 >> 172.21.15.62:0/718217392 conn(0x7fbdd80fbda0 msgr2=0x7fbdd80fe180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:40:40.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:40.983+0000 7fbddd400700 1 -- 172.21.15.62:0/718217392 shutdown_connections 2024-04-20T21:40:40.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:40.983+0000 7fbddd400700 1 -- 172.21.15.62:0/718217392 wait complete. 2024-04-20T21:40:40.998 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T21:40:41.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:41.134+0000 7f2ade400700 1 -- 172.21.15.62:0/4191360363 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2ad8076e20 msgr2=0x7f2ad80771f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:40:41.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:41.134+0000 7f2ade400700 1 --2- 172.21.15.62:0/4191360363 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2ad8076e20 0x7f2ad80771f0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f2acc009940 tx=0x7f2acc009c50 comp rx=0 tx=0).stop 2024-04-20T21:40:41.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:41.134+0000 7f2ade400700 1 -- 172.21.15.62:0/4191360363 shutdown_connections 2024-04-20T21:40:41.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:41.134+0000 7f2ade400700 1 --2- 172.21.15.62:0/4191360363 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ad8077730 0x7f2ad81109f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:40:41.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:41.134+0000 7f2ade400700 1 --2- 172.21.15.62:0/4191360363 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2ad8076e20 0x7f2ad80771f0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:40:41.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:41.134+0000 7f2ade400700 1 -- 172.21.15.62:0/4191360363 >> 172.21.15.62:0/4191360363 conn(0x7f2ad80fdd90 msgr2=0x7f2ad81001a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:40:41.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:41.135+0000 7f2ade400700 1 -- 172.21.15.62:0/4191360363 shutdown_connections 2024-04-20T21:40:41.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:41.135+0000 7f2ade400700 1 -- 172.21.15.62:0/4191360363 wait complete. 2024-04-20T21:40:41.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:41.136+0000 7f2ade400700 1 Processor -- start 2024-04-20T21:40:41.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:41.136+0000 7f2ade400700 1 -- start start 2024-04-20T21:40:41.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:41.136+0000 7f2ade400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2ad8076e20 0x7f2ad819c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:40:41.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:41.136+0000 7f2ade400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ad8077730 0x7f2ad819c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:40:41.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:41.136+0000 7f2ade400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ad819d030 con 0x7f2ad8076e20 2024-04-20T21:40:41.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:41.136+0000 7f2ade400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ad81a0dc0 con 0x7f2ad8077730 2024-04-20T21:40:41.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:41.136+0000 7f2add000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2ad8076e20 0x7f2ad819c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:40:41.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:41.137+0000 7f2add000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2ad8076e20 0x7f2ad819c410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53882/0 (socket says 172.21.15.62:53882) 2024-04-20T21:40:41.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:41.137+0000 7f2add000700 1 -- 172.21.15.62:0/2546972712 learned_addr learned my addr 172.21.15.62:0/2546972712 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:40:41.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:41.137+0000 7f2add000700 1 -- 172.21.15.62:0/2546972712 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ad8077730 msgr2=0x7f2ad819c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:40:41.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:41.137+0000 7f2add000700 1 --2- 172.21.15.62:0/2546972712 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ad8077730 0x7f2ad819c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:40:41.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:41.137+0000 7f2add000700 1 -- 172.21.15.62:0/2546972712 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2acc0095f0 con 0x7f2ad8076e20 2024-04-20T21:40:41.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:41.137+0000 7f2add000700 1 --2- 172.21.15.62:0/2546972712 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2ad8076e20 0x7f2ad819c410 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f2acc005620 tx=0x7f2acc005700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:40:41.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:41.138+0000 7f2ad5600700 1 -- 172.21.15.62:0/2546972712 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2acc01d070 con 0x7f2ad8076e20 2024-04-20T21:40:41.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:41.138+0000 7f2ad5600700 1 -- 172.21.15.62:0/2546972712 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2acc00f460 con 0x7f2ad8076e20 2024-04-20T21:40:41.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:41.138+0000 7f2ad5600700 1 -- 172.21.15.62:0/2546972712 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2acc021640 con 0x7f2ad8076e20 2024-04-20T21:40:41.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:41.138+0000 7f2ade400700 1 -- 172.21.15.62:0/2546972712 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ad81a10a0 con 0x7f2ad8076e20 2024-04-20T21:40:41.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:41.138+0000 7f2ade400700 1 -- 172.21.15.62:0/2546972712 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ad81a15f0 con 0x7f2ad8076e20 2024-04-20T21:40:41.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:41.139+0000 7f2ade400700 1 -- 172.21.15.62:0/2546972712 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ad810e160 con 0x7f2ad8076e20 2024-04-20T21:40:41.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:41.140+0000 7f2ad5600700 1 -- 172.21.15.62:0/2546972712 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f2acc00fa40 con 0x7f2ad8076e20 2024-04-20T21:40:41.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:41.141+0000 7f2ad5600700 1 --2- 172.21.15.62:0/2546972712 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2ac006ed10 0x7f2ac00711c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:40:41.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:41.141+0000 7f2ad5600700 1 -- 172.21.15.62:0/2546972712 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f2acc08ffe0 con 0x7f2ad8076e20 2024-04-20T21:40:41.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:41.141+0000 7f2ad7e00700 1 --2- 172.21.15.62:0/2546972712 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2ac006ed10 0x7f2ac00711c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:40:41.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:41.142+0000 7f2ad7e00700 1 --2- 172.21.15.62:0/2546972712 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2ac006ed10 0x7f2ac00711c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2ad819da30 tx=0x7f2ac4006d20 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:40:41.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:41.148+0000 7f2ad5600700 1 -- 172.21.15.62:0/2546972712 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f2acc058ee0 con 0x7f2ad8076e20 2024-04-20T21:40:41.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:41.395+0000 7f2ade400700 1 -- 172.21.15.62:0/2546972712 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2ad819d850 con 0x7f2ac006ed10 2024-04-20T21:40:41.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:41.397+0000 7f2ad5600700 1 -- 172.21.15.62:0/2546972712 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7f2ad819d850 con 0x7f2ac006ed10 2024-04-20T21:40:41.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:41.400+0000 7f2ade400700 1 -- 172.21.15.62:0/2546972712 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2ac006ed10 msgr2=0x7f2ac00711c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:40:41.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:41.400+0000 7f2ade400700 1 --2- 172.21.15.62:0/2546972712 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2ac006ed10 0x7f2ac00711c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2ad819da30 tx=0x7f2ac4006d20 comp rx=0 tx=0).stop 2024-04-20T21:40:41.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:41.400+0000 7f2ade400700 1 -- 172.21.15.62:0/2546972712 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2ad8076e20 msgr2=0x7f2ad819c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:40:41.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:41.400+0000 7f2ade400700 1 --2- 172.21.15.62:0/2546972712 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2ad8076e20 0x7f2ad819c410 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f2acc005620 tx=0x7f2acc005700 comp rx=0 tx=0).stop 2024-04-20T21:40:41.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:41.400+0000 7f2ade400700 1 -- 172.21.15.62:0/2546972712 shutdown_connections 2024-04-20T21:40:41.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:41.400+0000 7f2ade400700 1 --2- 172.21.15.62:0/2546972712 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2ac006ed10 0x7f2ac00711c0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:40:41.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:41.400+0000 7f2ade400700 1 --2- 172.21.15.62:0/2546972712 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2ad8076e20 0x7f2ad819c410 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:40:41.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:41.400+0000 7f2ade400700 1 --2- 172.21.15.62:0/2546972712 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ad8077730 0x7f2ad819c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:40:41.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:41.400+0000 7f2ade400700 1 -- 172.21.15.62:0/2546972712 >> 172.21.15.62:0/2546972712 conn(0x7f2ad80fdd90 msgr2=0x7f2ad80fe9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:40:41.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:41.400+0000 7f2ade400700 1 -- 172.21.15.62:0/2546972712 shutdown_connections 2024-04-20T21:40:41.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:41.400+0000 7f2ade400700 1 -- 172.21.15.62:0/2546972712 wait complete. 2024-04-20T21:40:41.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:41.566+0000 7f6294e00700 1 -- 172.21.15.62:0/1593274730 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6290108460 msgr2=0x7f6290108830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:40:41.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:41.566+0000 7f6294e00700 1 --2- 172.21.15.62:0/1593274730 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6290108460 0x7f6290108830 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f6280009910 tx=0x7f6280009c20 comp rx=0 tx=0).stop 2024-04-20T21:40:41.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:41.566+0000 7f6294e00700 1 -- 172.21.15.62:0/1593274730 shutdown_connections 2024-04-20T21:40:41.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:41.566+0000 7f6294e00700 1 --2- 172.21.15.62:0/1593274730 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6290102460 0x7f62901028d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:40:41.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:41.566+0000 7f6294e00700 1 --2- 172.21.15.62:0/1593274730 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6290108460 0x7f6290108830 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:40:41.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:41.566+0000 7f6294e00700 1 -- 172.21.15.62:0/1593274730 >> 172.21.15.62:0/1593274730 conn(0x7f62900fdf60 msgr2=0x7f6290100370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:40:41.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:41.566+0000 7f6294e00700 1 -- 172.21.15.62:0/1593274730 shutdown_connections 2024-04-20T21:40:41.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:41.566+0000 7f6294e00700 1 -- 172.21.15.62:0/1593274730 wait complete. 2024-04-20T21:40:41.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:41.567+0000 7f6294e00700 1 Processor -- start 2024-04-20T21:40:41.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:41.567+0000 7f6294e00700 1 -- start start 2024-04-20T21:40:41.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:41.568+0000 7f6294e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6290102460 0x7f62901981e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:40:41.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:41.568+0000 7f6294e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6290108460 0x7f6290198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:40:41.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:41.568+0000 7f6294e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6290198d70 con 0x7f6290108460 2024-04-20T21:40:41.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:41.568+0000 7f6294e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6290198eb0 con 0x7f6290102460 2024-04-20T21:40:41.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:41.568+0000 7f628d400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6290108460 0x7f6290198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:40:41.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:41.568+0000 7f628d400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6290108460 0x7f6290198720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47908/0 (socket says 172.21.15.62:47908) 2024-04-20T21:40:41.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:41.568+0000 7f628d400700 1 -- 172.21.15.62:0/2519476442 learned_addr learned my addr 172.21.15.62:0/2519476442 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:40:41.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:41.568+0000 7f628de00700 1 --2- 172.21.15.62:0/2519476442 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6290102460 0x7f62901981e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:40:41.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:41.568+0000 7f628d400700 1 -- 172.21.15.62:0/2519476442 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6290102460 msgr2=0x7f62901981e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:40:41.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:41.568+0000 7f628d400700 1 --2- 172.21.15.62:0/2519476442 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6290102460 0x7f62901981e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:40:41.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:41.568+0000 7f628d400700 1 -- 172.21.15.62:0/2519476442 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62800095f0 con 0x7f6290108460 2024-04-20T21:40:41.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:41.569+0000 7f628d400700 1 --2- 172.21.15.62:0/2519476442 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6290108460 0x7f6290198720 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f627c00c930 tx=0x7f627c00ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:40:41.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:41.569+0000 7f6286a00700 1 -- 172.21.15.62:0/2519476442 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f627c007ad0 con 0x7f6290108460 2024-04-20T21:40:41.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:41.569+0000 7f6286a00700 1 -- 172.21.15.62:0/2519476442 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f627c007c30 con 0x7f6290108460 2024-04-20T21:40:41.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:41.569+0000 7f6294e00700 1 -- 172.21.15.62:0/2519476442 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f629019cd00 con 0x7f6290108460 2024-04-20T21:40:41.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:41.569+0000 7f6286a00700 1 -- 172.21.15.62:0/2519476442 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f627c018610 con 0x7f6290108460 2024-04-20T21:40:41.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:41.569+0000 7f6294e00700 1 -- 172.21.15.62:0/2519476442 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f629019d250 con 0x7f6290108460 2024-04-20T21:40:41.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:41.570+0000 7f6294e00700 1 -- 172.21.15.62:0/2519476442 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f629010a9d0 con 0x7f6290108460 2024-04-20T21:40:41.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:41.571+0000 7f6286a00700 1 -- 172.21.15.62:0/2519476442 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f627c00f450 con 0x7f6290108460 2024-04-20T21:40:41.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:41.572+0000 7f6286a00700 1 --2- 172.21.15.62:0/2519476442 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f627406ecc0 0x7f6274071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:40:41.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:41.573+0000 7f6286a00700 1 -- 172.21.15.62:0/2519476442 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f627c08efa0 con 0x7f6290108460 2024-04-20T21:40:41.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:41.573+0000 7f628de00700 1 --2- 172.21.15.62:0/2519476442 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f627406ecc0 0x7f6274071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:40:41.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:41.574+0000 7f628de00700 1 --2- 172.21.15.62:0/2519476442 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f627406ecc0 0x7f6274071170 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f6280006010 tx=0x7f628001a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:40:41.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:41.578+0000 7f6286a00700 1 -- 172.21.15.62:0/2519476442 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f627c057de0 con 0x7f6290108460 2024-04-20T21:40:41.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:41 smithi189 ceph-mon[57022]: pgmap v146: 65 pgs: 65 active+clean; 315 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 3 op/s 2024-04-20T21:40:41.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:41 smithi062 ceph-mon[76322]: pgmap v146: 65 pgs: 65 active+clean; 315 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 3 op/s 2024-04-20T21:40:41.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:41.829+0000 7f6294e00700 1 -- 172.21.15.62:0/2519476442 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f62901995b0 con 0x7f627406ecc0 2024-04-20T21:40:41.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:41.842+0000 7f6286a00700 1 -- 172.21.15.62:0/2519476442 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7f62901995b0 con 0x7f627406ecc0 2024-04-20T21:40:41.841 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T21:40:41.842 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (6m) 45s ago 20m 14.2M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T21:40:41.842 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (20m) 45s ago 20m 8992k - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T21:40:41.842 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (19m) 17s ago 19m 8456k - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T21:40:41.842 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (2m) 45s ago 20m 6660k - 19.0.0-3133-ga8c84f59 76bc6b91b726 2825d5ba11c6 2024-04-20T21:40:41.842 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (2m) 17s ago 19m 6665k - 19.0.0-3133-ga8c84f59 76bc6b91b726 20f746fa1bba 2024-04-20T21:40:41.842 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (5m) 45s ago 20m 75.0M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T21:40:41.842 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (13m) 45s ago 13m 65.5M - 18.2.0 dc2bc1663786 5e6460aa6159 2024-04-20T21:40:41.842 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (13m) 45s ago 13m 131M - 18.2.0 dc2bc1663786 e65f2267544e 2024-04-20T21:40:41.842 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (13m) 16s ago 13m 117M - 18.2.0 dc2bc1663786 fb2a8a223db8 2024-04-20T21:40:41.842 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (13m) 16s ago 13m 55.9M - 18.2.0 dc2bc1663786 f2eec2ee8951 2024-04-20T21:40:41.842 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (9m) 45s ago 22m 504M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T21:40:41.842 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (8m) 17s ago 19m 451M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T21:40:41.842 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (4m) 45s ago 22m 49.3M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 019eb1479485 2024-04-20T21:40:41.842 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (3m) 16s ago 18m 38.3M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 55aac560aab6 2024-04-20T21:40:41.843 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (7m) 45s ago 20m 7906k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T21:40:41.843 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (7m) 17s ago 19m 7545k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T21:40:41.843 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (118s) 45s ago 18m 83.0M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 ee2bb901f26e 2024-04-20T21:40:41.843 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (85s) 45s ago 17m 87.9M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 9075152d4162 2024-04-20T21:40:41.843 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (54s) 45s ago 16m 12.7M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 2d4f3e38dc0e 2024-04-20T21:40:41.843 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (22s) 16s ago 16m 11.1M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 8f97b17ae0eb 2024-04-20T21:40:41.843 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (15m) 16s ago 15m 222M 2240M 18.2.0 dc2bc1663786 6c8fce91d45d 2024-04-20T21:40:41.843 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (15m) 16s ago 15m 203M 2240M 18.2.0 dc2bc1663786 8e4574bae8db 2024-04-20T21:40:41.843 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (7m) 45s ago 19m 73.9M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T21:40:41.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:41.845+0000 7f6294e00700 1 -- 172.21.15.62:0/2519476442 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f627406ecc0 msgr2=0x7f6274071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:40:41.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:41.845+0000 7f6294e00700 1 --2- 172.21.15.62:0/2519476442 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f627406ecc0 0x7f6274071170 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f6280006010 tx=0x7f628001a040 comp rx=0 tx=0).stop 2024-04-20T21:40:41.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:41.845+0000 7f6294e00700 1 -- 172.21.15.62:0/2519476442 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6290108460 msgr2=0x7f6290198720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:40:41.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:41.845+0000 7f6294e00700 1 --2- 172.21.15.62:0/2519476442 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6290108460 0x7f6290198720 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f627c00c930 tx=0x7f627c00ccf0 comp rx=0 tx=0).stop 2024-04-20T21:40:41.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:41.845+0000 7f6294e00700 1 -- 172.21.15.62:0/2519476442 shutdown_connections 2024-04-20T21:40:41.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:41.845+0000 7f6294e00700 1 --2- 172.21.15.62:0/2519476442 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f627406ecc0 0x7f6274071170 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:40:41.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:41.845+0000 7f6294e00700 1 --2- 172.21.15.62:0/2519476442 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6290102460 0x7f62901981e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:40:41.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:41.845+0000 7f6294e00700 1 --2- 172.21.15.62:0/2519476442 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6290108460 0x7f6290198720 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:40:41.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:41.845+0000 7f6294e00700 1 -- 172.21.15.62:0/2519476442 >> 172.21.15.62:0/2519476442 conn(0x7f62900fdf60 msgr2=0x7f6290100290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:40:41.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:41.845+0000 7f6294e00700 1 -- 172.21.15.62:0/2519476442 shutdown_connections 2024-04-20T21:40:41.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:41.845+0000 7f6294e00700 1 -- 172.21.15.62:0/2519476442 wait complete. 2024-04-20T21:40:42.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.001+0000 7fc91c400700 1 -- 172.21.15.62:0/1335429325 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc914068540 msgr2=0x7fc9140689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:40:42.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.001+0000 7fc91c400700 1 --2- 172.21.15.62:0/1335429325 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc914068540 0x7fc9140689b0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fc900009910 tx=0x7fc900009c20 comp rx=0 tx=0).stop 2024-04-20T21:40:42.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.001+0000 7fc91c400700 1 -- 172.21.15.62:0/1335429325 shutdown_connections 2024-04-20T21:40:42.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.001+0000 7fc91c400700 1 --2- 172.21.15.62:0/1335429325 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc914068540 0x7fc9140689b0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:40:42.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.001+0000 7fc91c400700 1 --2- 172.21.15.62:0/1335429325 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc914106320 0x7fc9141066f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:40:42.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.001+0000 7fc91c400700 1 -- 172.21.15.62:0/1335429325 >> 172.21.15.62:0/1335429325 conn(0x7fc914075130 msgr2=0x7fc914075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:40:42.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.001+0000 7fc91c400700 1 -- 172.21.15.62:0/1335429325 shutdown_connections 2024-04-20T21:40:42.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.001+0000 7fc91c400700 1 -- 172.21.15.62:0/1335429325 wait complete. 2024-04-20T21:40:42.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.002+0000 7fc91c400700 1 Processor -- start 2024-04-20T21:40:42.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.002+0000 7fc91c400700 1 -- start start 2024-04-20T21:40:42.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.003+0000 7fc91c400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc914068540 0x7fc914195e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:40:42.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.003+0000 7fc91c400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc914106320 0x7fc9141963a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:40:42.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.003+0000 7fc91c400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc914196a80 con 0x7fc914106320 2024-04-20T21:40:42.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.003+0000 7fc91c400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc91419a810 con 0x7fc914068540 2024-04-20T21:40:42.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.003+0000 7fc919000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc914106320 0x7fc9141963a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:40:42.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.003+0000 7fc919000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc914106320 0x7fc9141963a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47936/0 (socket says 172.21.15.62:47936) 2024-04-20T21:40:42.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.003+0000 7fc919000700 1 -- 172.21.15.62:0/2257522665 learned_addr learned my addr 172.21.15.62:0/2257522665 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:40:42.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.003+0000 7fc919a00700 1 --2- 172.21.15.62:0/2257522665 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc914068540 0x7fc914195e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:40:42.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.004+0000 7fc919000700 1 -- 172.21.15.62:0/2257522665 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc914068540 msgr2=0x7fc914195e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:40:42.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.004+0000 7fc919000700 1 --2- 172.21.15.62:0/2257522665 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc914068540 0x7fc914195e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:40:42.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.004+0000 7fc919000700 1 -- 172.21.15.62:0/2257522665 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9000095f0 con 0x7fc914106320 2024-04-20T21:40:42.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.004+0000 7fc919a00700 1 --2- 172.21.15.62:0/2257522665 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc914068540 0x7fc914195e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T21:40:42.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.004+0000 7fc919000700 1 --2- 172.21.15.62:0/2257522665 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc914106320 0x7fc9141963a0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fc900006010 tx=0x7fc90000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:40:42.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.004+0000 7fc912000700 1 -- 172.21.15.62:0/2257522665 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc90001d070 con 0x7fc914106320 2024-04-20T21:40:42.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.004+0000 7fc91c400700 1 -- 172.21.15.62:0/2257522665 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc91419aa90 con 0x7fc914106320 2024-04-20T21:40:42.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.004+0000 7fc912000700 1 -- 172.21.15.62:0/2257522665 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc900004fb0 con 0x7fc914106320 2024-04-20T21:40:42.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.004+0000 7fc912000700 1 -- 172.21.15.62:0/2257522665 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc900022ea0 con 0x7fc914106320 2024-04-20T21:40:42.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.004+0000 7fc91c400700 1 -- 172.21.15.62:0/2257522665 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc91419af80 con 0x7fc914106320 2024-04-20T21:40:42.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.006+0000 7fc91c400700 1 -- 172.21.15.62:0/2257522665 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc914108770 con 0x7fc914106320 2024-04-20T21:40:42.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.007+0000 7fc912000700 1 -- 172.21.15.62:0/2257522665 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fc900022470 con 0x7fc914106320 2024-04-20T21:40:42.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.008+0000 7fc912000700 1 --2- 172.21.15.62:0/2257522665 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc8fc06ecc0 0x7fc8fc071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:40:42.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.008+0000 7fc912000700 1 -- 172.21.15.62:0/2257522665 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fc900090310 con 0x7fc914106320 2024-04-20T21:40:42.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.008+0000 7fc919a00700 1 --2- 172.21.15.62:0/2257522665 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc8fc06ecc0 0x7fc8fc071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:40:42.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.009+0000 7fc919a00700 1 --2- 172.21.15.62:0/2257522665 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc8fc06ecc0 0x7fc8fc071170 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fc908005fd0 tx=0x7fc908005dc0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:40:42.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.013+0000 7fc912000700 1 -- 172.21.15.62:0/2257522665 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fc900059200 con 0x7fc914106320 2024-04-20T21:40:42.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.331+0000 7fc91c400700 1 -- 172.21.15.62:0/2257522665 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fc91404f2a0 con 0x7fc914106320 2024-04-20T21:40:42.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.331+0000 7fc912000700 1 -- 172.21.15.62:0/2257522665 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+791 (secure 0 0 0) 0x7fc90005c8c0 con 0x7fc914106320 2024-04-20T21:40:42.331 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:40:42.331 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T21:40:42.331 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:40:42.331 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:40:42.331 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T21:40:42.331 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:40:42.331 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:40:42.332 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T21:40:42.332 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 2, 2024-04-20T21:40:42.332 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 4 2024-04-20T21:40:42.332 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:40:42.332 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T21:40:42.332 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 4 2024-04-20T21:40:42.332 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:40:42.332 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T21:40:42.332 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 6, 2024-04-20T21:40:42.332 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 8 2024-04-20T21:40:42.332 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T21:40:42.332 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:40:42.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.334+0000 7fc91c400700 1 -- 172.21.15.62:0/2257522665 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc8fc06ecc0 msgr2=0x7fc8fc071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:40:42.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.334+0000 7fc91c400700 1 --2- 172.21.15.62:0/2257522665 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc8fc06ecc0 0x7fc8fc071170 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fc908005fd0 tx=0x7fc908005dc0 comp rx=0 tx=0).stop 2024-04-20T21:40:42.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.334+0000 7fc91c400700 1 -- 172.21.15.62:0/2257522665 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc914106320 msgr2=0x7fc9141963a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:40:42.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.334+0000 7fc91c400700 1 --2- 172.21.15.62:0/2257522665 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc914106320 0x7fc9141963a0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fc900006010 tx=0x7fc90000be20 comp rx=0 tx=0).stop 2024-04-20T21:40:42.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.334+0000 7fc91c400700 1 -- 172.21.15.62:0/2257522665 shutdown_connections 2024-04-20T21:40:42.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.334+0000 7fc91c400700 1 --2- 172.21.15.62:0/2257522665 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc8fc06ecc0 0x7fc8fc071170 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:40:42.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.334+0000 7fc91c400700 1 --2- 172.21.15.62:0/2257522665 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc914068540 0x7fc914195e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:40:42.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.334+0000 7fc91c400700 1 --2- 172.21.15.62:0/2257522665 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc914106320 0x7fc9141963a0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:40:42.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.334+0000 7fc91c400700 1 -- 172.21.15.62:0/2257522665 >> 172.21.15.62:0/2257522665 conn(0x7fc914075130 msgr2=0x7fc9140febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:40:42.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.335+0000 7fc91c400700 1 -- 172.21.15.62:0/2257522665 shutdown_connections 2024-04-20T21:40:42.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.335+0000 7fc91c400700 1 -- 172.21.15.62:0/2257522665 wait complete. 2024-04-20T21:40:42.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.496+0000 7f1b30000700 1 -- 172.21.15.62:0/3049640007 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1b2810c8f0 msgr2=0x7f1b2810ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:40:42.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.496+0000 7f1b30000700 1 --2- 172.21.15.62:0/3049640007 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1b2810c8f0 0x7f1b2810ccc0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f1b1c009910 tx=0x7f1b1c009c20 comp rx=0 tx=0).stop 2024-04-20T21:40:42.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.496+0000 7f1b30000700 1 -- 172.21.15.62:0/3049640007 shutdown_connections 2024-04-20T21:40:42.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.496+0000 7f1b30000700 1 --2- 172.21.15.62:0/3049640007 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1b28106b80 0x7f1b28106ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:40:42.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.496+0000 7f1b30000700 1 --2- 172.21.15.62:0/3049640007 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1b2810c8f0 0x7f1b2810ccc0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:40:42.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.496+0000 7f1b30000700 1 -- 172.21.15.62:0/3049640007 >> 172.21.15.62:0/3049640007 conn(0x7f1b28076060 msgr2=0x7f1b28078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:40:42.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.496+0000 7f1b30000700 1 -- 172.21.15.62:0/3049640007 shutdown_connections 2024-04-20T21:40:42.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.497+0000 7f1b30000700 1 -- 172.21.15.62:0/3049640007 wait complete. 2024-04-20T21:40:42.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.497+0000 7f1b30000700 1 Processor -- start 2024-04-20T21:40:42.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.498+0000 7f1b30000700 1 -- start start 2024-04-20T21:40:42.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.498+0000 7f1b30000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1b28106b80 0x7f1b2819c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:40:42.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.498+0000 7f1b30000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1b2810c8f0 0x7f1b2819ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:40:42.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.498+0000 7f1b30000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b2819d090 con 0x7f1b2810c8f0 2024-04-20T21:40:42.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.498+0000 7f1b30000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b2819d1d0 con 0x7f1b28106b80 2024-04-20T21:40:42.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.498+0000 7f1b2cc00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1b2810c8f0 0x7f1b2819ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:40:42.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.498+0000 7f1b2cc00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1b2810c8f0 0x7f1b2819ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47954/0 (socket says 172.21.15.62:47954) 2024-04-20T21:40:42.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.499+0000 7f1b2cc00700 1 -- 172.21.15.62:0/1720740907 learned_addr learned my addr 172.21.15.62:0/1720740907 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:40:42.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.499+0000 7f1b2d600700 1 --2- 172.21.15.62:0/1720740907 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1b28106b80 0x7f1b2819c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:40:42.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.499+0000 7f1b2cc00700 1 -- 172.21.15.62:0/1720740907 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1b28106b80 msgr2=0x7f1b2819c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:40:42.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.499+0000 7f1b2cc00700 1 --2- 172.21.15.62:0/1720740907 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1b28106b80 0x7f1b2819c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:40:42.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.499+0000 7f1b2cc00700 1 -- 172.21.15.62:0/1720740907 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b1c0095f0 con 0x7f1b2810c8f0 2024-04-20T21:40:42.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.499+0000 7f1b2cc00700 1 --2- 172.21.15.62:0/1720740907 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1b2810c8f0 0x7f1b2819ca40 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f1b1400c930 tx=0x7f1b1400ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:40:42.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.500+0000 7f1b26000700 1 -- 172.21.15.62:0/1720740907 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1b14007ad0 con 0x7f1b2810c8f0 2024-04-20T21:40:42.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.500+0000 7f1b26000700 1 -- 172.21.15.62:0/1720740907 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1b14007c30 con 0x7f1b2810c8f0 2024-04-20T21:40:42.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.500+0000 7f1b30000700 1 -- 172.21.15.62:0/1720740907 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b281a1020 con 0x7f1b2810c8f0 2024-04-20T21:40:42.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.500+0000 7f1b26000700 1 -- 172.21.15.62:0/1720740907 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1b14018610 con 0x7f1b2810c8f0 2024-04-20T21:40:42.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.500+0000 7f1b30000700 1 -- 172.21.15.62:0/1720740907 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b281a1570 con 0x7f1b2810c8f0 2024-04-20T21:40:42.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.501+0000 7f1b30000700 1 -- 172.21.15.62:0/1720740907 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b28109b20 con 0x7f1b2810c8f0 2024-04-20T21:40:42.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.502+0000 7f1b26000700 1 -- 172.21.15.62:0/1720740907 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f1b1400f450 con 0x7f1b2810c8f0 2024-04-20T21:40:42.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.503+0000 7f1b26000700 1 --2- 172.21.15.62:0/1720740907 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f1b1006ecc0 0x7f1b10071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:40:42.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.503+0000 7f1b26000700 1 -- 172.21.15.62:0/1720740907 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f1b1408ef00 con 0x7f1b2810c8f0 2024-04-20T21:40:42.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.503+0000 7f1b2d600700 1 --2- 172.21.15.62:0/1720740907 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f1b1006ecc0 0x7f1b10071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:40:42.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.504+0000 7f1b2d600700 1 --2- 172.21.15.62:0/1720740907 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f1b1006ecc0 0x7f1b10071170 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f1b1c006010 tx=0x7f1b1c01a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:40:42.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.508+0000 7f1b26000700 1 -- 172.21.15.62:0/1720740907 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f1b14057de0 con 0x7f1b2810c8f0 2024-04-20T21:40:42.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:42 smithi189 ceph-mon[57022]: from='client.34340 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:40:42.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:42 smithi189 ceph-mon[57022]: from='client.34344 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:40:42.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:42 smithi189 ceph-mon[57022]: from='client.34348 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:40:42.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:42 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/2257522665' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:40:42.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:42 smithi062 ceph-mon[76322]: from='client.34340 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:40:42.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:42 smithi062 ceph-mon[76322]: from='client.34344 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:40:42.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:42 smithi062 ceph-mon[76322]: from='client.34348 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:40:42.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:42 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/2257522665' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:40:42.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.791+0000 7f1b30000700 1 -- 172.21.15.62:0/1720740907 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f1b2804f2a0 con 0x7f1b2810c8f0 2024-04-20T21:40:42.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.792+0000 7f1b26000700 1 -- 172.21.15.62:0/1720740907 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 18 v18) v1 ==== 76+0+1928 (secure 0 0 0) 0x7f1b1405b4a0 con 0x7f1b2810c8f0 2024-04-20T21:40:42.792 INFO:teuthology.orchestra.run.smithi062.stdout:e18 2024-04-20T21:40:42.793 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T21:40:42.793 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T21:40:42.793 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T21:40:42.793 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:40:42.793 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T21:40:42.793 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T21:40:42.793 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 18 2024-04-20T21:40:42.793 INFO:teuthology.orchestra.run.smithi062.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-20T21:40:42.793 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T21:40:42.794 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T21:37:06.018300+0000 2024-04-20T21:40:42.794 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T21:40:42.794 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T21:40:42.794 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T21:40:42.794 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T21:40:42.794 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T21:40:42.794 INFO:teuthology.orchestra.run.smithi062.stdout:max_xattr_size 65536 2024-04-20T21:40:42.795 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T21:40:42.795 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T21:40:42.795 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 0 2024-04-20T21:40:42.795 INFO:teuthology.orchestra.run.smithi062.stdout:snap_rstat 1 2024-04-20T21:40:42.795 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T21:40:42.795 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 2 2024-04-20T21:40:42.796 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1 2024-04-20T21:40:42.796 INFO:teuthology.orchestra.run.smithi062.stdout:up {0=14458,1=24279} 2024-04-20T21:40:42.796 INFO:teuthology.orchestra.run.smithi062.stdout:failed 2024-04-20T21:40:42.796 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T21:40:42.796 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T21:40:42.796 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T21:40:42.796 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T21:40:42.797 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data enabled 2024-04-20T21:40:42.797 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T21:40:42.797 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T21:40:42.797 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 1 2024-04-20T21:40:42.797 INFO:teuthology.orchestra.run.smithi062.stdout:qdb_cluster leader: 14458 members: 24279,14458 2024-04-20T21:40:42.797 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ryqblb{0:14458} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.62:6826/1147127123,v1:172.21.15.62:6827/1147127123] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:40:42.798 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ozizwi{0:14474} state up:standby-replay seq 3 join_fscid=1 addr [v2:172.21.15.62:6828/3494174293,v1:172.21.15.62:6829/3494174293] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:40:42.798 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.dvimer{1:24279} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.189:6824/237262514,v1:172.21.15.189:6825/237262514] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:40:42.798 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.xbgspw{1:24291} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.189:6826/1495356258,v1:172.21.15.189:6827/1495356258] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:40:42.798 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:40:42.798 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:40:42.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.795+0000 7f1b30000700 1 -- 172.21.15.62:0/1720740907 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f1b1006ecc0 msgr2=0x7f1b10071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:40:42.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.795+0000 7f1b30000700 1 --2- 172.21.15.62:0/1720740907 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f1b1006ecc0 0x7f1b10071170 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f1b1c006010 tx=0x7f1b1c01a040 comp rx=0 tx=0).stop 2024-04-20T21:40:42.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.795+0000 7f1b30000700 1 -- 172.21.15.62:0/1720740907 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1b2810c8f0 msgr2=0x7f1b2819ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:40:42.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.795+0000 7f1b30000700 1 --2- 172.21.15.62:0/1720740907 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1b2810c8f0 0x7f1b2819ca40 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f1b1400c930 tx=0x7f1b1400ccf0 comp rx=0 tx=0).stop 2024-04-20T21:40:42.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.795+0000 7f1b30000700 1 -- 172.21.15.62:0/1720740907 shutdown_connections 2024-04-20T21:40:42.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.795+0000 7f1b30000700 1 --2- 172.21.15.62:0/1720740907 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f1b1006ecc0 0x7f1b10071170 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:40:42.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.795+0000 7f1b30000700 1 --2- 172.21.15.62:0/1720740907 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1b28106b80 0x7f1b2819c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:40:42.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.795+0000 7f1b30000700 1 --2- 172.21.15.62:0/1720740907 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1b2810c8f0 0x7f1b2819ca40 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:40:42.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.795+0000 7f1b30000700 1 -- 172.21.15.62:0/1720740907 >> 172.21.15.62:0/1720740907 conn(0x7f1b28076060 msgr2=0x7f1b28077aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:40:42.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.796+0000 7f1b30000700 1 -- 172.21.15.62:0/1720740907 shutdown_connections 2024-04-20T21:40:42.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.796+0000 7f1b30000700 1 -- 172.21.15.62:0/1720740907 wait complete. 2024-04-20T21:40:42.801 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 18 2024-04-20T21:40:42.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.956+0000 7fd1c9e00700 1 -- 172.21.15.62:0/3612460588 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd1c410c8f0 msgr2=0x7fd1c410ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:40:42.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.956+0000 7fd1c9e00700 1 --2- 172.21.15.62:0/3612460588 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd1c410c8f0 0x7fd1c410ccc0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fd1b4009910 tx=0x7fd1b4009c20 comp rx=0 tx=0).stop 2024-04-20T21:40:42.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.956+0000 7fd1c9e00700 1 -- 172.21.15.62:0/3612460588 shutdown_connections 2024-04-20T21:40:42.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.956+0000 7fd1c9e00700 1 --2- 172.21.15.62:0/3612460588 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd1c4106b80 0x7fd1c4106ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:40:42.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.956+0000 7fd1c9e00700 1 --2- 172.21.15.62:0/3612460588 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd1c410c8f0 0x7fd1c410ccc0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:40:42.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.956+0000 7fd1c9e00700 1 -- 172.21.15.62:0/3612460588 >> 172.21.15.62:0/3612460588 conn(0x7fd1c4076060 msgr2=0x7fd1c4078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:40:42.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.956+0000 7fd1c9e00700 1 -- 172.21.15.62:0/3612460588 shutdown_connections 2024-04-20T21:40:42.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.957+0000 7fd1c9e00700 1 -- 172.21.15.62:0/3612460588 wait complete. 2024-04-20T21:40:42.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.957+0000 7fd1c9e00700 1 Processor -- start 2024-04-20T21:40:42.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.958+0000 7fd1c9e00700 1 -- start start 2024-04-20T21:40:42.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.958+0000 7fd1c9e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd1c4106b80 0x7fd1c419c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:40:42.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.958+0000 7fd1c9e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd1c410c8f0 0x7fd1c419c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:40:42.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.958+0000 7fd1c9e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1c419cfb0 con 0x7fd1c4106b80 2024-04-20T21:40:42.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.958+0000 7fd1c9e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1c419d0f0 con 0x7fd1c410c8f0 2024-04-20T21:40:42.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.958+0000 7fd1c2a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd1c4106b80 0x7fd1c419c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:40:42.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.958+0000 7fd1c2a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd1c4106b80 0x7fd1c419c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47974/0 (socket says 172.21.15.62:47974) 2024-04-20T21:40:42.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.958+0000 7fd1c2a00700 1 -- 172.21.15.62:0/1052649025 learned_addr learned my addr 172.21.15.62:0/1052649025 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:40:42.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.959+0000 7fd1c2a00700 1 -- 172.21.15.62:0/1052649025 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd1c410c8f0 msgr2=0x7fd1c419c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:40:42.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.959+0000 7fd1c2a00700 1 --2- 172.21.15.62:0/1052649025 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd1c410c8f0 0x7fd1c419c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:40:42.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.959+0000 7fd1c2a00700 1 -- 172.21.15.62:0/1052649025 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1b40095f0 con 0x7fd1c4106b80 2024-04-20T21:40:42.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.959+0000 7fd1c2a00700 1 --2- 172.21.15.62:0/1052649025 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd1c4106b80 0x7fd1c419c420 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fd1b400b5c0 tx=0x7fd1b400be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:40:42.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.959+0000 7fd1c8a00700 1 -- 172.21.15.62:0/1052649025 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd1b401d070 con 0x7fd1c4106b80 2024-04-20T21:40:42.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.959+0000 7fd1c9e00700 1 -- 172.21.15.62:0/1052649025 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1c41a0ee0 con 0x7fd1c4106b80 2024-04-20T21:40:42.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.959+0000 7fd1c8a00700 1 -- 172.21.15.62:0/1052649025 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd1b400f460 con 0x7fd1c4106b80 2024-04-20T21:40:42.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.959+0000 7fd1c8a00700 1 -- 172.21.15.62:0/1052649025 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd1b4017610 con 0x7fd1c4106b80 2024-04-20T21:40:42.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.959+0000 7fd1c9e00700 1 -- 172.21.15.62:0/1052649025 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1c41a13d0 con 0x7fd1c4106b80 2024-04-20T21:40:42.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.961+0000 7fd1c9e00700 1 -- 172.21.15.62:0/1052649025 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1c404f2a0 con 0x7fd1c4106b80 2024-04-20T21:40:42.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.962+0000 7fd1c8a00700 1 -- 172.21.15.62:0/1052649025 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fd1b400f5d0 con 0x7fd1c4106b80 2024-04-20T21:40:42.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.962+0000 7fd1c8a00700 1 --2- 172.21.15.62:0/1052649025 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd1a806ecc0 0x7fd1a8071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:40:42.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.963+0000 7fd1c8a00700 1 -- 172.21.15.62:0/1052649025 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fd1b40903d0 con 0x7fd1c4106b80 2024-04-20T21:40:42.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.963+0000 7fd1c2000700 1 --2- 172.21.15.62:0/1052649025 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd1a806ecc0 0x7fd1a8071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:40:42.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.963+0000 7fd1c2000700 1 --2- 172.21.15.62:0/1052649025 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd1a806ecc0 0x7fd1a8071170 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fd1b0005fd0 tx=0x7fd1b0005d00 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:40:42.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:42.967+0000 7fd1c8a00700 1 -- 172.21.15.62:0/1052649025 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fd1b4059230 con 0x7fd1c4106b80 2024-04-20T21:40:43.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:43.225+0000 7fd1c9e00700 1 -- 172.21.15.62:0/1052649025 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd1c41097e0 con 0x7fd1a806ecc0 2024-04-20T21:40:43.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:43.229+0000 7fd1c8a00700 1 -- 172.21.15.62:0/1052649025 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7fd1c41097e0 con 0x7fd1a806ecc0 2024-04-20T21:40:43.228 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:40:43.229 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T21:40:43.229 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T21:40:43.229 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-20T21:40:43.229 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T21:40:43.229 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr", 2024-04-20T21:40:43.229 INFO:teuthology.orchestra.run.smithi062.stdout: "crash", 2024-04-20T21:40:43.229 INFO:teuthology.orchestra.run.smithi062.stdout: "mon" 2024-04-20T21:40:43.229 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T21:40:43.229 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "10/23 daemons upgraded", 2024-04-20T21:40:43.229 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading osd daemons", 2024-04-20T21:40:43.229 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T21:40:43.229 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:40:43.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:43.232+0000 7fd1c9e00700 1 -- 172.21.15.62:0/1052649025 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd1a806ecc0 msgr2=0x7fd1a8071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:40:43.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:43.232+0000 7fd1c9e00700 1 --2- 172.21.15.62:0/1052649025 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd1a806ecc0 0x7fd1a8071170 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fd1b0005fd0 tx=0x7fd1b0005d00 comp rx=0 tx=0).stop 2024-04-20T21:40:43.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:43.232+0000 7fd1c9e00700 1 -- 172.21.15.62:0/1052649025 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd1c4106b80 msgr2=0x7fd1c419c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:40:43.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:43.232+0000 7fd1c9e00700 1 --2- 172.21.15.62:0/1052649025 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd1c4106b80 0x7fd1c419c420 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fd1b400b5c0 tx=0x7fd1b400be00 comp rx=0 tx=0).stop 2024-04-20T21:40:43.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:43.232+0000 7fd1c9e00700 1 -- 172.21.15.62:0/1052649025 shutdown_connections 2024-04-20T21:40:43.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:43.232+0000 7fd1c9e00700 1 --2- 172.21.15.62:0/1052649025 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd1a806ecc0 0x7fd1a8071170 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:40:43.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:43.232+0000 7fd1c9e00700 1 --2- 172.21.15.62:0/1052649025 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd1c4106b80 0x7fd1c419c420 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:40:43.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:43.232+0000 7fd1c9e00700 1 --2- 172.21.15.62:0/1052649025 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd1c410c8f0 0x7fd1c419c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:40:43.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:43.232+0000 7fd1c9e00700 1 -- 172.21.15.62:0/1052649025 >> 172.21.15.62:0/1052649025 conn(0x7fd1c4076060 msgr2=0x7fd1c4077a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:40:43.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:43.232+0000 7fd1c9e00700 1 -- 172.21.15.62:0/1052649025 shutdown_connections 2024-04-20T21:40:43.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:43.233+0000 7fd1c9e00700 1 -- 172.21.15.62:0/1052649025 wait complete. 2024-04-20T21:40:43.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:43.393+0000 7f799b200700 1 -- 172.21.15.62:0/1862562743 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7994102470 msgr2=0x7f79941028e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:40:43.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:43.393+0000 7f799b200700 1 --2- 172.21.15.62:0/1862562743 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7994102470 0x7f79941028e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f7980009910 tx=0x7f7980009c20 comp rx=0 tx=0).stop 2024-04-20T21:40:43.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:43.393+0000 7f799b200700 1 -- 172.21.15.62:0/1862562743 shutdown_connections 2024-04-20T21:40:43.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:43.393+0000 7f799b200700 1 --2- 172.21.15.62:0/1862562743 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7994102470 0x7f79941028e0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:40:43.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:43.393+0000 7f799b200700 1 --2- 172.21.15.62:0/1862562743 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7994108470 0x7f7994108840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:40:43.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:43.393+0000 7f799b200700 1 -- 172.21.15.62:0/1862562743 >> 172.21.15.62:0/1862562743 conn(0x7f79940fdf90 msgr2=0x7f79941003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:40:43.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:43.394+0000 7f799b200700 1 -- 172.21.15.62:0/1862562743 shutdown_connections 2024-04-20T21:40:43.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:43.394+0000 7f799b200700 1 -- 172.21.15.62:0/1862562743 wait complete. 2024-04-20T21:40:43.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:43.395+0000 7f799b200700 1 Processor -- start 2024-04-20T21:40:43.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:43.395+0000 7f799b200700 1 -- start start 2024-04-20T21:40:43.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:43.395+0000 7f799b200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7994102470 0x7f7994198270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:40:43.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:43.395+0000 7f799b200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7994108470 0x7f79941987b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:40:43.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:43.395+0000 7f799b200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7994198e90 con 0x7f7994108470 2024-04-20T21:40:43.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:43.395+0000 7f799b200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f799419cc20 con 0x7f7994102470 2024-04-20T21:40:43.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:43.395+0000 7f7993400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7994108470 0x7f79941987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:40:43.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:43.396+0000 7f7993400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7994108470 0x7f79941987b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48004/0 (socket says 172.21.15.62:48004) 2024-04-20T21:40:43.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:43.396+0000 7f7993400700 1 -- 172.21.15.62:0/427134584 learned_addr learned my addr 172.21.15.62:0/427134584 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:40:43.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:43.396+0000 7f7993e00700 1 --2- 172.21.15.62:0/427134584 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7994102470 0x7f7994198270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:40:43.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:43.396+0000 7f7993400700 1 -- 172.21.15.62:0/427134584 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7994102470 msgr2=0x7f7994198270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:40:43.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:43.396+0000 7f7993400700 1 --2- 172.21.15.62:0/427134584 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7994102470 0x7f7994198270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:40:43.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:43.396+0000 7f7993400700 1 -- 172.21.15.62:0/427134584 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79800095f0 con 0x7f7994108470 2024-04-20T21:40:43.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:43.396+0000 7f7993400700 1 --2- 172.21.15.62:0/427134584 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7994108470 0x7f79941987b0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f7980006010 tx=0x7f798000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:40:43.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:43.397+0000 7f7990c00700 1 -- 172.21.15.62:0/427134584 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f798001d070 con 0x7f7994108470 2024-04-20T21:40:43.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:43.397+0000 7f7990c00700 1 -- 172.21.15.62:0/427134584 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f798000f460 con 0x7f7994108470 2024-04-20T21:40:43.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:43.397+0000 7f7990c00700 1 -- 172.21.15.62:0/427134584 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7980017610 con 0x7f7994108470 2024-04-20T21:40:43.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:43.397+0000 7f799b200700 1 -- 172.21.15.62:0/427134584 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f799419cea0 con 0x7f7994108470 2024-04-20T21:40:43.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:43.397+0000 7f799b200700 1 -- 172.21.15.62:0/427134584 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f799419d340 con 0x7f7994108470 2024-04-20T21:40:43.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:43.398+0000 7f799b200700 1 -- 172.21.15.62:0/427134584 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f799410a970 con 0x7f7994108470 2024-04-20T21:40:43.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:43.399+0000 7f7990c00700 1 -- 172.21.15.62:0/427134584 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f798000f5d0 con 0x7f7994108470 2024-04-20T21:40:43.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:43.400+0000 7f7990c00700 1 --2- 172.21.15.62:0/427134584 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f797806ecc0 0x7f7978071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:40:43.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:43.400+0000 7f7990c00700 1 -- 172.21.15.62:0/427134584 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f7980090200 con 0x7f7994108470 2024-04-20T21:40:43.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:43.400+0000 7f7993e00700 1 --2- 172.21.15.62:0/427134584 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f797806ecc0 0x7f7978071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:40:43.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:43.401+0000 7f7993e00700 1 --2- 172.21.15.62:0/427134584 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f797806ecc0 0x7f7978071170 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f7984005950 tx=0x7f79840058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:40:43.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:43.405+0000 7f7990c00700 1 -- 172.21.15.62:0/427134584 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f79800591a0 con 0x7f7994108470 2024-04-20T21:40:43.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:43 smithi062 ceph-mon[76322]: pgmap v147: 65 pgs: 65 active+clean; 315 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-20T21:40:43.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:43 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/1720740907' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T21:40:43.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:43.731+0000 7f799b200700 1 -- 172.21.15.62:0/427134584 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f79941996a0 con 0x7f7994108470 2024-04-20T21:40:43.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:43.732+0000 7f7990c00700 1 -- 172.21.15.62:0/427134584 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7f798005c860 con 0x7f7994108470 2024-04-20T21:40:43.732 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-04-20T21:40:43.732 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T21:40:43.732 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T21:40:43.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:43.735+0000 7f799b200700 1 -- 172.21.15.62:0/427134584 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f797806ecc0 msgr2=0x7f7978071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:40:43.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:43.735+0000 7f799b200700 1 --2- 172.21.15.62:0/427134584 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f797806ecc0 0x7f7978071170 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f7984005950 tx=0x7f79840058e0 comp rx=0 tx=0).stop 2024-04-20T21:40:43.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:43.735+0000 7f799b200700 1 -- 172.21.15.62:0/427134584 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7994108470 msgr2=0x7f79941987b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:40:43.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:43.735+0000 7f799b200700 1 --2- 172.21.15.62:0/427134584 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7994108470 0x7f79941987b0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f7980006010 tx=0x7f798000be20 comp rx=0 tx=0).stop 2024-04-20T21:40:43.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:43.735+0000 7f799b200700 1 -- 172.21.15.62:0/427134584 shutdown_connections 2024-04-20T21:40:43.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:43.735+0000 7f799b200700 1 --2- 172.21.15.62:0/427134584 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f797806ecc0 0x7f7978071170 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:40:43.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:43.735+0000 7f799b200700 1 --2- 172.21.15.62:0/427134584 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7994102470 0x7f7994198270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:40:43.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:43.735+0000 7f799b200700 1 --2- 172.21.15.62:0/427134584 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7994108470 0x7f79941987b0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:40:43.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:43.735+0000 7f799b200700 1 -- 172.21.15.62:0/427134584 >> 172.21.15.62:0/427134584 conn(0x7f79940fdf90 msgr2=0x7f79940ff410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:40:43.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:43.735+0000 7f799b200700 1 -- 172.21.15.62:0/427134584 shutdown_connections 2024-04-20T21:40:43.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:40:43.736+0000 7f799b200700 1 -- 172.21.15.62:0/427134584 wait complete. 2024-04-20T21:40:43.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:43 smithi189 ceph-mon[57022]: pgmap v147: 65 pgs: 65 active+clean; 315 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-20T21:40:43.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:43 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/1720740907' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T21:40:44.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:44 smithi189 ceph-mon[57022]: from='client.34360 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:40:44.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:44 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/427134584' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T21:40:44.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:44 smithi062 ceph-mon[76322]: from='client.34360 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:40:44.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:44 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/427134584' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T21:40:45.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:45 smithi189 ceph-mon[57022]: pgmap v148: 65 pgs: 65 active+clean; 315 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:40:45.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:45 smithi062 ceph-mon[76322]: pgmap v148: 65 pgs: 65 active+clean; 315 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:40:47.736 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:47 smithi189 ceph-mon[57022]: pgmap v149: 65 pgs: 65 active+clean; 315 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-20T21:40:47.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:47 smithi062 ceph-mon[76322]: pgmap v149: 65 pgs: 65 active+clean; 315 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-20T21:40:48.750 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:48 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-20T21:40:48.750 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:48 smithi189 ceph-mon[57022]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-20T21:40:48.751 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:48 smithi189 ceph-mon[57022]: Upgrade: osd.4 is safe to restart 2024-04-20T21:40:48.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:48 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-20T21:40:48.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:48 smithi062 ceph-mon[76322]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-20T21:40:48.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:48 smithi062 ceph-mon[76322]: Upgrade: osd.4 is safe to restart 2024-04-20T21:40:49.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:49 smithi189 ceph-mon[57022]: pgmap v150: 65 pgs: 65 active+clean; 315 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-20T21:40:49.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:49 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:40:49.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:49 smithi062 ceph-mon[76322]: pgmap v150: 65 pgs: 65 active+clean; 315 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-20T21:40:49.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:49 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:40:51.094 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:50 smithi189 ceph-mon[57022]: Upgrade: Updating osd.4 2024-04-20T21:40:51.094 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:50 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:40:51.094 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:50 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-20T21:40:51.094 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:50 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:40:51.094 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:50 smithi189 ceph-mon[57022]: Deploying daemon osd.4 on smithi189 2024-04-20T21:40:51.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:50 smithi062 ceph-mon[76322]: Upgrade: Updating osd.4 2024-04-20T21:40:51.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:50 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:40:51.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:50 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-20T21:40:51.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:50 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:40:51.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:50 smithi062 ceph-mon[76322]: Deploying daemon osd.4 on smithi189 2024-04-20T21:40:52.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:51 smithi189 ceph-mon[57022]: pgmap v151: 65 pgs: 65 active+clean; 315 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-20T21:40:52.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:51 smithi062 ceph-mon[76322]: pgmap v151: 65 pgs: 65 active+clean; 315 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-20T21:40:54.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:53 smithi189 ceph-mon[57022]: pgmap v152: 65 pgs: 65 active+clean; 315 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-20T21:40:54.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:53 smithi062 ceph-mon[76322]: pgmap v152: 65 pgs: 65 active+clean; 315 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-20T21:40:54.987 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 20 21:40:54 smithi189 systemd[1]: Stopping Ceph osd.4 for 61ae3628-ff5b-11ee-bc93-c7b262605968... 2024-04-20T21:40:55.487 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 20 21:40:55 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-4[32130]: 2024-04-20T21:40:55.174+0000 7fd674000700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-20T21:40:55.488 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 20 21:40:55 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-4[32130]: 2024-04-20T21:40:55.174+0000 7fd674000700 -1 osd.4 68 *** Got signal Terminated *** 2024-04-20T21:40:55.488 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 20 21:40:55 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-4[32130]: 2024-04-20T21:40:55.174+0000 7fd674000700 -1 osd.4 68 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-20T21:40:56.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:55 smithi189 ceph-mon[57022]: pgmap v153: 65 pgs: 65 active+clean; 315 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:40:56.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:55 smithi189 ceph-mon[57022]: osd.4 marked itself down and dead 2024-04-20T21:40:56.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:55 smithi062 ceph-mon[76322]: pgmap v153: 65 pgs: 65 active+clean; 315 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:40:56.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:55 smithi062 ceph-mon[76322]: osd.4 marked itself down and dead 2024-04-20T21:40:56.737 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 20 21:40:56 smithi189 podman[70926]: 2024-04-20 21:40:56.471801622 +0000 UTC m=+1.458762974 container died 6c8fce91d45d9631242e55bb089d13fc83ec84f8326cec26066b835de687b296 (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-4, CEPH_POINT_RELEASE=-18.2.0, org.label-schema.vendor=CentOS, io.buildah.version=1.29.1, GIT_CLEAN=True, org.label-schema.build-date=20231212, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, RELEASE=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-04-20T21:40:57.133 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:56 smithi189 ceph-mon[57022]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-20T21:40:57.133 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:56 smithi189 ceph-mon[57022]: osdmap e69: 6 total, 5 up, 6 in 2024-04-20T21:40:57.134 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:56 smithi189 ceph-mon[57022]: osdmap e70: 6 total, 5 up, 6 in 2024-04-20T21:40:57.134 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 20 21:40:56 smithi189 podman[70926]: 2024-04-20 21:40:56.917211405 +0000 UTC m=+1.904172774 container cleanup 6c8fce91d45d9631242e55bb089d13fc83ec84f8326cec26066b835de687b296 (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-4, RELEASE=HEAD, CEPH_POINT_RELEASE=-18.2.0, org.label-schema.vendor=CentOS, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.29.1, org.label-schema.build-date=20231212, org.label-schema.schema-version=1.0) 2024-04-20T21:40:57.134 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 20 21:40:56 smithi189 bash[70926]: ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-4 2024-04-20T21:40:57.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:56 smithi062 ceph-mon[76322]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-20T21:40:57.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:56 smithi062 ceph-mon[76322]: osdmap e69: 6 total, 5 up, 6 in 2024-04-20T21:40:57.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:56 smithi062 ceph-mon[76322]: osdmap e70: 6 total, 5 up, 6 in 2024-04-20T21:40:57.487 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 20 21:40:57 smithi189 podman[70945]: 2024-04-20 21:40:57.134450729 +0000 UTC m=+0.651864404 container remove 6c8fce91d45d9631242e55bb089d13fc83ec84f8326cec26066b835de687b296 (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-4, GIT_CLEAN=True, org.label-schema.build-date=20231212, CEPH_POINT_RELEASE=-18.2.0, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux ) 2024-04-20T21:40:58.191 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:57 smithi189 ceph-mon[57022]: pgmap v155: 65 pgs: 6 stale+active+clean, 59 active+clean; 315 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-20T21:40:58.191 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 20 21:40:57 smithi189 podman[71037]: 2024-04-20 21:40:57.858875365 +0000 UTC m=+0.032638984 image pull 76bc6b91b726a03b8b736eeeb35fc1ffd1469e23c7d60f5b444a217528d8906d quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 2024-04-20T21:40:58.191 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 20 21:40:58 smithi189 podman[71037]: 2024-04-20 21:40:58.019080051 +0000 UTC m=+0.192843694 container create f5bc77fa576b08a19a17f9f6046661d16a7948d7a3a51444b4b1d429b4e29207 (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-4-deactivate, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, ceph=True, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-20T21:40:58.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:57 smithi062 ceph-mon[76322]: pgmap v155: 65 pgs: 6 stale+active+clean, 59 active+clean; 315 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-20T21:40:58.737 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 20 21:40:58 smithi189 podman[71037]: 2024-04-20 21:40:58.39906972 +0000 UTC m=+0.572833396 container init f5bc77fa576b08a19a17f9f6046661d16a7948d7a3a51444b4b1d429b4e29207 (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-4-deactivate, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-04-20T21:40:58.737 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 20 21:40:58 smithi189 podman[71037]: 2024-04-20 21:40:58.411813025 +0000 UTC m=+0.585576675 container start f5bc77fa576b08a19a17f9f6046661d16a7948d7a3a51444b4b1d429b4e29207 (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-4-deactivate, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_BRANCH=HEAD) 2024-04-20T21:40:58.737 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 20 21:40:58 smithi189 podman[71037]: 2024-04-20 21:40:58.51309023 +0000 UTC m=+0.686853923 container attach f5bc77fa576b08a19a17f9f6046661d16a7948d7a3a51444b4b1d429b4e29207 (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-4-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-04-20T21:40:59.194 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:58 smithi189 ceph-mon[57022]: pgmap v157: 65 pgs: 11 peering, 5 stale+active+clean, 49 active+clean; 315 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 1 op/s 2024-04-20T21:40:59.195 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 20 21:40:58 smithi189 podman[71037]: 2024-04-20 21:40:58.862435701 +0000 UTC m=+1.036199351 container died f5bc77fa576b08a19a17f9f6046661d16a7948d7a3a51444b4b1d429b4e29207 (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-4-deactivate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, ceph=True, CEPH_POINT_RELEASE=) 2024-04-20T21:40:59.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:58 smithi062 ceph-mon[76322]: pgmap v157: 65 pgs: 11 peering, 5 stale+active+clean, 49 active+clean; 315 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 1 op/s 2024-04-20T21:40:59.936 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 20 21:40:59 smithi189 podman[71063]: 2024-04-20 21:40:59.524742396 +0000 UTC m=+0.643211627 container remove f5bc77fa576b08a19a17f9f6046661d16a7948d7a3a51444b4b1d429b4e29207 (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-4-deactivate, org.label-schema.build-date=20240416, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-04-20T21:40:59.936 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 20 21:40:59 smithi189 systemd[1]: ceph-61ae3628-ff5b-11ee-bc93-c7b262605968@osd.4.service: Deactivated successfully. 2024-04-20T21:40:59.936 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 20 21:40:59 smithi189 systemd[1]: Stopped Ceph osd.4 for 61ae3628-ff5b-11ee-bc93-c7b262605968. 2024-04-20T21:40:59.936 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 20 21:40:59 smithi189 systemd[1]: ceph-61ae3628-ff5b-11ee-bc93-c7b262605968@osd.4.service: Consumed 1min 44.904s CPU time. 2024-04-20T21:41:00.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:40:59 smithi189 ceph-mon[57022]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-20T21:41:00.237 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 20 21:40:59 smithi189 systemd[1]: Starting Ceph osd.4 for 61ae3628-ff5b-11ee-bc93-c7b262605968... 2024-04-20T21:41:00.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:40:59 smithi062 ceph-mon[76322]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-20T21:41:00.604 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 20 21:41:00 smithi189 podman[71162]: 2024-04-20 21:41:00.205925328 +0000 UTC m=+0.036910366 image pull 76bc6b91b726a03b8b736eeeb35fc1ffd1469e23c7d60f5b444a217528d8906d quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 2024-04-20T21:41:00.605 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 20 21:41:00 smithi189 podman[71162]: 2024-04-20 21:41:00.342905024 +0000 UTC m=+0.173890060 container create f6011da32614e7a1d2dce6920bc3e0758d3e0a404b02af1d44ecdfee1dbef5ef (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-4-activate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, ceph=True) 2024-04-20T21:41:00.605 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 20 21:41:00 smithi189 podman[71162]: 2024-04-20 21:41:00.605449505 +0000 UTC m=+0.436434579 container init f6011da32614e7a1d2dce6920bc3e0758d3e0a404b02af1d44ecdfee1dbef5ef (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-4-activate, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-04-20T21:41:00.961 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 20 21:41:00 smithi189 podman[71162]: 2024-04-20 21:41:00.617790978 +0000 UTC m=+0.448776076 container start f6011da32614e7a1d2dce6920bc3e0758d3e0a404b02af1d44ecdfee1dbef5ef (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-4-activate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, ceph=True, org.label-schema.schema-version=1.0) 2024-04-20T21:41:00.961 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 20 21:41:00 smithi189 podman[71162]: 2024-04-20 21:41:00.720615156 +0000 UTC m=+0.551600195 container attach f6011da32614e7a1d2dce6920bc3e0758d3e0a404b02af1d44ecdfee1dbef5ef (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-4-activate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True) 2024-04-20T21:41:01.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:00 smithi189 ceph-mon[57022]: pgmap v158: 65 pgs: 6 active+undersized+degraded, 11 peering, 2 stale+active+clean, 9 active+undersized, 37 active+clean; 315 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 19/366 objects degraded (5.191%) 2024-04-20T21:41:01.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:00 smithi062 ceph-mon[76322]: pgmap v158: 65 pgs: 6 active+undersized+degraded, 11 peering, 2 stale+active+clean, 9 active+undersized, 37 active+clean; 315 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 19/366 objects degraded (5.191%) 2024-04-20T21:41:01.966 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 20 21:41:01 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-4-activate[71177]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-20T21:41:01.967 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 20 21:41:01 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-4-activate[71177]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-20T21:41:01.967 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 20 21:41:01 smithi189 bash[71162]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-20T21:41:01.967 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 20 21:41:01 smithi189 bash[71162]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-20T21:41:01.967 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 20 21:41:01 smithi189 bash[71162]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-20T21:41:01.967 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 20 21:41:01 smithi189 bash[71162]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-04-20T21:41:01.967 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 20 21:41:01 smithi189 bash[71162]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-04-20T21:41:01.967 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 20 21:41:01 smithi189 bash[71162]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-04-20T21:41:01.967 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 20 21:41:01 smithi189 bash[71162]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-4/block 2024-04-20T21:41:01.967 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 20 21:41:01 smithi189 bash[71162]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-20T21:41:01.967 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 20 21:41:01 smithi189 bash[71162]: --> ceph-volume raw activate successful for osd ID: 4 2024-04-20T21:41:01.967 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 20 21:41:01 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-4-activate[71177]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-20T21:41:01.967 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 20 21:41:01 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-4-activate[71177]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-04-20T21:41:01.967 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 20 21:41:01 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-4-activate[71177]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-04-20T21:41:01.967 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 20 21:41:01 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-4-activate[71177]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-04-20T21:41:01.968 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 20 21:41:01 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-4-activate[71177]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-4/block 2024-04-20T21:41:01.968 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 20 21:41:01 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-4-activate[71177]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-20T21:41:01.968 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 20 21:41:01 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-4-activate[71177]: --> ceph-volume raw activate successful for osd ID: 4 2024-04-20T21:41:01.968 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 20 21:41:01 smithi189 podman[71162]: 2024-04-20 21:41:01.622766358 +0000 UTC m=+1.453751422 container died f6011da32614e7a1d2dce6920bc3e0758d3e0a404b02af1d44ecdfee1dbef5ef (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-4-activate, org.label-schema.build-date=20240416, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-04-20T21:41:02.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:01 smithi062 ceph-mon[76322]: Health check failed: Degraded data redundancy: 19/366 objects degraded (5.191%), 6 pgs degraded (PG_DEGRADED) 2024-04-20T21:41:02.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:01 smithi189 ceph-mon[57022]: Health check failed: Degraded data redundancy: 19/366 objects degraded (5.191%), 6 pgs degraded (PG_DEGRADED) 2024-04-20T21:41:02.671 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 20 21:41:02 smithi189 podman[71318]: 2024-04-20 21:41:02.32402736 +0000 UTC m=+0.682845354 container remove f6011da32614e7a1d2dce6920bc3e0758d3e0a404b02af1d44ecdfee1dbef5ef (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-4-activate, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-04-20T21:41:02.945 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 20 21:41:02 smithi189 podman[71377]: 2024-04-20 21:41:02.577193444 +0000 UTC m=+0.032319772 image pull 76bc6b91b726a03b8b736eeeb35fc1ffd1469e23c7d60f5b444a217528d8906d quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 2024-04-20T21:41:02.945 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 20 21:41:02 smithi189 podman[71377]: 2024-04-20 21:41:02.674974007 +0000 UTC m=+0.130100365 container create 653d8079967d3f53e729f038004fbcabe76e431cdaed1f685e973f1464f8534a (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-4, ceph=True, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-04-20T21:41:02.945 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 20 21:41:02 smithi189 podman[71377]: 2024-04-20 21:41:02.945644749 +0000 UTC m=+0.400771114 container init 653d8079967d3f53e729f038004fbcabe76e431cdaed1f685e973f1464f8534a (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-4, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-04-20T21:41:03.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:02 smithi189 ceph-mon[57022]: pgmap v159: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 315 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 62/366 objects degraded (16.940%) 2024-04-20T21:41:03.237 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 20 21:41:02 smithi189 podman[71377]: 2024-04-20 21:41:02.958781698 +0000 UTC m=+0.413908045 container start 653d8079967d3f53e729f038004fbcabe76e431cdaed1f685e973f1464f8534a (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-4, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-20T21:41:03.238 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 20 21:41:03 smithi189 ceph-osd[71396]: -- 172.21.15.189:0/2390353425 <== mon.1 v2:172.21.15.189:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5595ef1a7680 con 0x5595ef171000 2024-04-20T21:41:03.238 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 20 21:41:03 smithi189 bash[71377]: 653d8079967d3f53e729f038004fbcabe76e431cdaed1f685e973f1464f8534a 2024-04-20T21:41:03.238 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 20 21:41:03 smithi189 systemd[1]: Started Ceph osd.4 for 61ae3628-ff5b-11ee-bc93-c7b262605968. 2024-04-20T21:41:03.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:02 smithi062 ceph-mon[76322]: pgmap v159: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 315 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 62/366 objects degraded (16.940%) 2024-04-20T21:41:03.976 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 20 21:41:03 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-4[71392]: 2024-04-20T21:41:03.546+0000 7f2f47ad77c0 -1 Falling back to public interface 2024-04-20T21:41:04.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:03 smithi189 ceph-mon[57022]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-20T21:41:04.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:03 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:41:04.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:03 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:41:04.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:03 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:41:04.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:03 smithi062 ceph-mon[76322]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-20T21:41:04.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:03 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:41:04.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:03 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:41:04.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:03 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:41:05.703 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:05 smithi189 ceph-mon[57022]: pgmap v160: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 315 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 62/366 objects degraded (16.940%) 2024-04-20T21:41:05.703 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:05 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:41:05.703 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:05 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:41:05.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:05 smithi062 ceph-mon[76322]: pgmap v160: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 315 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 62/366 objects degraded (16.940%) 2024-04-20T21:41:05.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:05 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:41:05.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:05 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:41:07.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:07 smithi189 ceph-mon[57022]: pgmap v161: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 315 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 62/366 objects degraded (16.940%) 2024-04-20T21:41:07.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:07 smithi062 ceph-mon[76322]: pgmap v161: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 315 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 62/366 objects degraded (16.940%) 2024-04-20T21:41:08.986 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 20 21:41:08 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-4[71392]: 2024-04-20T21:41:08.774+0000 7f2f47ad77c0 -1 osd.4 0 read_superblock omap replica is missing. 2024-04-20T21:41:08.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:08 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:41:08.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:08 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:41:09.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:08 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:41:09.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:08 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:41:09.487 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 20 21:41:09 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-4[71392]: 2024-04-20T21:41:09.189+0000 7f2f47ad77c0 -1 osd.4 68 log_to_monitors true 2024-04-20T21:41:09.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:09 smithi189 ceph-mon[57022]: pgmap v162: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 315 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 62/366 objects degraded (16.940%) 2024-04-20T21:41:09.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:09 smithi189 ceph-mon[57022]: Health check update: Degraded data redundancy: 62/366 objects degraded (16.940%), 16 pgs degraded (PG_DEGRADED) 2024-04-20T21:41:09.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:09 smithi189 ceph-mon[57022]: from='osd.4 [v2:172.21.15.189:6808/2794032930,v1:172.21.15.189:6809/2794032930]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-20T21:41:09.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:09 smithi189 ceph-mon[57022]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-20T21:41:10.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:09 smithi062 ceph-mon[76322]: pgmap v162: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 315 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 62/366 objects degraded (16.940%) 2024-04-20T21:41:10.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:09 smithi062 ceph-mon[76322]: Health check update: Degraded data redundancy: 62/366 objects degraded (16.940%), 16 pgs degraded (PG_DEGRADED) 2024-04-20T21:41:10.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:09 smithi062 ceph-mon[76322]: from='osd.4 [v2:172.21.15.189:6808/2794032930,v1:172.21.15.189:6809/2794032930]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-20T21:41:10.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:09 smithi062 ceph-mon[76322]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-20T21:41:10.987 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 20 21:41:10 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-4[71392]: 2024-04-20T21:41:10.700+0000 7f2f31e00700 -1 osd.4 68 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-20T21:41:10.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:10 smithi189 ceph-mon[57022]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-20T21:41:10.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:10 smithi189 ceph-mon[57022]: osdmap e71: 6 total, 5 up, 6 in 2024-04-20T21:41:10.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:10 smithi189 ceph-mon[57022]: from='osd.4 [v2:172.21.15.189:6808/2794032930,v1:172.21.15.189:6809/2794032930]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-04-20T21:41:10.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:10 smithi189 ceph-mon[57022]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-04-20T21:41:10.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:10 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:41:10.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:10 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:41:11.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:10 smithi062 ceph-mon[76322]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-20T21:41:11.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:10 smithi062 ceph-mon[76322]: osdmap e71: 6 total, 5 up, 6 in 2024-04-20T21:41:11.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:10 smithi062 ceph-mon[76322]: from='osd.4 [v2:172.21.15.189:6808/2794032930,v1:172.21.15.189:6809/2794032930]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-04-20T21:41:11.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:10 smithi062 ceph-mon[76322]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-04-20T21:41:11.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:10 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:41:11.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:10 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:41:11.919 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:11 smithi189 ceph-mon[57022]: pgmap v164: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 315 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 2 op/s; 62/366 objects degraded (16.940%) 2024-04-20T21:41:11.919 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:11 smithi189 ceph-mon[57022]: from='osd.4 ' entity='osd.4' 2024-04-20T21:41:12.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:11 smithi062 ceph-mon[76322]: pgmap v164: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 315 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 2 op/s; 62/366 objects degraded (16.940%) 2024-04-20T21:41:12.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:11 smithi062 ceph-mon[76322]: from='osd.4 ' entity='osd.4' 2024-04-20T21:41:12.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:12 smithi189 ceph-mon[57022]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-20T21:41:12.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:12 smithi189 ceph-mon[57022]: osd.4 [v2:172.21.15.189:6808/2794032930,v1:172.21.15.189:6809/2794032930] boot 2024-04-20T21:41:12.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:12 smithi189 ceph-mon[57022]: osdmap e72: 6 total, 6 up, 6 in 2024-04-20T21:41:12.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:12 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-20T21:41:12.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:12 smithi189 ceph-mon[57022]: osdmap e73: 6 total, 6 up, 6 in 2024-04-20T21:41:13.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:12 smithi062 ceph-mon[76322]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-20T21:41:13.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:12 smithi062 ceph-mon[76322]: osd.4 [v2:172.21.15.189:6808/2794032930,v1:172.21.15.189:6809/2794032930] boot 2024-04-20T21:41:13.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:12 smithi062 ceph-mon[76322]: osdmap e72: 6 total, 6 up, 6 in 2024-04-20T21:41:13.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:12 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-20T21:41:13.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:12 smithi062 ceph-mon[76322]: osdmap e73: 6 total, 6 up, 6 in 2024-04-20T21:41:13.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:13.911+0000 7f627ec00700 1 -- 172.21.15.62:0/1009287911 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6278100e80 msgr2=0x7f6278104e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:41:13.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:13.911+0000 7f627ec00700 1 --2- 172.21.15.62:0/1009287911 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6278100e80 0x7f6278104e50 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f6264009910 tx=0x7f6264009c20 comp rx=0 tx=0).stop 2024-04-20T21:41:13.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:13.911+0000 7f627ec00700 1 -- 172.21.15.62:0/1009287911 shutdown_connections 2024-04-20T21:41:13.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:13.911+0000 7f627ec00700 1 --2- 172.21.15.62:0/1009287911 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6278100e80 0x7f6278104e50 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:41:13.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:13.911+0000 7f627ec00700 1 --2- 172.21.15.62:0/1009287911 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62781004e0 0x7f62781008b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:41:13.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:13.911+0000 7f627ec00700 1 -- 172.21.15.62:0/1009287911 >> 172.21.15.62:0/1009287911 conn(0x7f62780fbda0 msgr2=0x7f62780fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:41:13.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:13.911+0000 7f627ec00700 1 -- 172.21.15.62:0/1009287911 shutdown_connections 2024-04-20T21:41:13.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:13.911+0000 7f627ec00700 1 -- 172.21.15.62:0/1009287911 wait complete. 2024-04-20T21:41:13.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:13.913+0000 7f627ec00700 1 Processor -- start 2024-04-20T21:41:13.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:13.913+0000 7f627ec00700 1 -- start start 2024-04-20T21:41:13.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:13.913+0000 7f627ec00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62781004e0 0x7f6278073050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:41:13.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:13.913+0000 7f627ec00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6278100e80 0x7f6278073590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:41:13.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:13.913+0000 7f627ec00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6278073ad0 con 0x7f6278100e80 2024-04-20T21:41:13.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:13.913+0000 7f627ec00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6278073c10 con 0x7f62781004e0 2024-04-20T21:41:13.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:13.914+0000 7f6277400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6278100e80 0x7f6278073590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:41:13.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:13.914+0000 7f6277400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6278100e80 0x7f6278073590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53904/0 (socket says 172.21.15.62:53904) 2024-04-20T21:41:13.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:13.914+0000 7f6277400700 1 -- 172.21.15.62:0/1288989514 learned_addr learned my addr 172.21.15.62:0/1288989514 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:41:13.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:13.914+0000 7f6277e00700 1 --2- 172.21.15.62:0/1288989514 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62781004e0 0x7f6278073050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:41:13.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:13.914+0000 7f6277400700 1 -- 172.21.15.62:0/1288989514 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62781004e0 msgr2=0x7f6278073050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:41:13.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:13.914+0000 7f6277400700 1 --2- 172.21.15.62:0/1288989514 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62781004e0 0x7f6278073050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:41:13.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:13.914+0000 7f6277400700 1 -- 172.21.15.62:0/1288989514 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62640095f0 con 0x7f6278100e80 2024-04-20T21:41:13.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:13.915+0000 7f6277400700 1 --2- 172.21.15.62:0/1288989514 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6278100e80 0x7f6278073590 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f626400bd70 tx=0x7f626400bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:41:13.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:13.915+0000 7f6274c00700 1 -- 172.21.15.62:0/1288989514 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f626401d070 con 0x7f6278100e80 2024-04-20T21:41:13.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:13.915+0000 7f6274c00700 1 -- 172.21.15.62:0/1288989514 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6264005dc0 con 0x7f6278100e80 2024-04-20T21:41:13.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:13.915+0000 7f627ec00700 1 -- 172.21.15.62:0/1288989514 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6278073e90 con 0x7f6278100e80 2024-04-20T21:41:13.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:13.915+0000 7f6274c00700 1 -- 172.21.15.62:0/1288989514 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6264017400 con 0x7f6278100e80 2024-04-20T21:41:13.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:13.915+0000 7f627ec00700 1 -- 172.21.15.62:0/1288989514 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62781a2410 con 0x7f6278100e80 2024-04-20T21:41:13.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:13.916+0000 7f627ec00700 1 -- 172.21.15.62:0/1288989514 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6278108850 con 0x7f6278100e80 2024-04-20T21:41:13.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:13.917+0000 7f6274c00700 1 -- 172.21.15.62:0/1288989514 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f626400f460 con 0x7f6278100e80 2024-04-20T21:41:13.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:13.919+0000 7f6274c00700 1 --2- 172.21.15.62:0/1288989514 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f625c06ecc0 0x7f625c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:41:13.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:13.919+0000 7f6274c00700 1 -- 172.21.15.62:0/1288989514 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f6264090180 con 0x7f6278100e80 2024-04-20T21:41:13.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:13.919+0000 7f6277e00700 1 --2- 172.21.15.62:0/1288989514 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f625c06ecc0 0x7f625c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:41:13.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:13.920+0000 7f6277e00700 1 --2- 172.21.15.62:0/1288989514 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f625c06ecc0 0x7f625c071170 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f6268005950 tx=0x7f626800b410 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:41:13.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:13.924+0000 7f6274c00700 1 -- 172.21.15.62:0/1288989514 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f6264058fc0 con 0x7f6278100e80 2024-04-20T21:41:14.037 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:13 smithi062 ceph-mon[76322]: pgmap v166: 65 pgs: 10 peering, 11 active+undersized+degraded, 13 active+undersized, 31 active+clean; 315 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 3 op/s; 36/366 objects degraded (9.836%) 2024-04-20T21:41:14.115 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:13 smithi189 ceph-mon[57022]: pgmap v166: 65 pgs: 10 peering, 11 active+undersized+degraded, 13 active+undersized, 31 active+clean; 315 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 3 op/s; 36/366 objects degraded (9.836%) 2024-04-20T21:41:14.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:14.178+0000 7f627ec00700 1 -- 172.21.15.62:0/1288989514 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f62780745f0 con 0x7f625c06ecc0 2024-04-20T21:41:14.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:14.180+0000 7f6274c00700 1 -- 172.21.15.62:0/1288989514 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7f62780745f0 con 0x7f625c06ecc0 2024-04-20T21:41:14.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:14.183+0000 7f627ec00700 1 -- 172.21.15.62:0/1288989514 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f625c06ecc0 msgr2=0x7f625c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:41:14.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:14.183+0000 7f627ec00700 1 --2- 172.21.15.62:0/1288989514 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f625c06ecc0 0x7f625c071170 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f6268005950 tx=0x7f626800b410 comp rx=0 tx=0).stop 2024-04-20T21:41:14.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:14.183+0000 7f627ec00700 1 -- 172.21.15.62:0/1288989514 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6278100e80 msgr2=0x7f6278073590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:41:14.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:14.183+0000 7f627ec00700 1 --2- 172.21.15.62:0/1288989514 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6278100e80 0x7f6278073590 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f626400bd70 tx=0x7f626400bda0 comp rx=0 tx=0).stop 2024-04-20T21:41:14.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:14.183+0000 7f627ec00700 1 -- 172.21.15.62:0/1288989514 shutdown_connections 2024-04-20T21:41:14.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:14.183+0000 7f627ec00700 1 --2- 172.21.15.62:0/1288989514 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f625c06ecc0 0x7f625c071170 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:41:14.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:14.183+0000 7f627ec00700 1 --2- 172.21.15.62:0/1288989514 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62781004e0 0x7f6278073050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:41:14.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:14.183+0000 7f627ec00700 1 --2- 172.21.15.62:0/1288989514 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6278100e80 0x7f6278073590 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:41:14.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:14.183+0000 7f627ec00700 1 -- 172.21.15.62:0/1288989514 >> 172.21.15.62:0/1288989514 conn(0x7f62780fbda0 msgr2=0x7f62780fe180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:41:14.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:14.184+0000 7f627ec00700 1 -- 172.21.15.62:0/1288989514 shutdown_connections 2024-04-20T21:41:14.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:14.184+0000 7f627ec00700 1 -- 172.21.15.62:0/1288989514 wait complete. 2024-04-20T21:41:14.199 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T21:41:14.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:14.354+0000 7fd60d000700 1 -- 172.21.15.62:0/1542382552 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6080730f0 msgr2=0x7fd6080734c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:41:14.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:14.354+0000 7fd60d000700 1 --2- 172.21.15.62:0/1542382552 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6080730f0 0x7fd6080734c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fd5f8009910 tx=0x7fd5f8009c20 comp rx=0 tx=0).stop 2024-04-20T21:41:14.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:14.355+0000 7fd60d000700 1 -- 172.21.15.62:0/1542382552 shutdown_connections 2024-04-20T21:41:14.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:14.355+0000 7fd60d000700 1 --2- 172.21.15.62:0/1542382552 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd608073a00 0x7fd608110d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:41:14.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:14.355+0000 7fd60d000700 1 --2- 172.21.15.62:0/1542382552 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6080730f0 0x7fd6080734c0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:41:14.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:14.355+0000 7fd60d000700 1 -- 172.21.15.62:0/1542382552 >> 172.21.15.62:0/1542382552 conn(0x7fd6080fbda0 msgr2=0x7fd6080fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:41:14.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:14.355+0000 7fd60d000700 1 -- 172.21.15.62:0/1542382552 shutdown_connections 2024-04-20T21:41:14.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:14.355+0000 7fd60d000700 1 -- 172.21.15.62:0/1542382552 wait complete. 2024-04-20T21:41:14.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:14.356+0000 7fd60d000700 1 Processor -- start 2024-04-20T21:41:14.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:14.356+0000 7fd60d000700 1 -- start start 2024-04-20T21:41:14.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:14.357+0000 7fd60d000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6080730f0 0x7fd60819c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:41:14.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:14.357+0000 7fd60d000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd608073a00 0x7fd60819c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:41:14.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:14.357+0000 7fd60d000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd60819cf70 con 0x7fd608073a00 2024-04-20T21:41:14.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:14.357+0000 7fd60d000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd60819d0b0 con 0x7fd6080730f0 2024-04-20T21:41:14.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:14.357+0000 7fd5fd400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd608073a00 0x7fd60819c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:41:14.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:14.357+0000 7fd5fd400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd608073a00 0x7fd60819c920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53914/0 (socket says 172.21.15.62:53914) 2024-04-20T21:41:14.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:14.357+0000 7fd5fd400700 1 -- 172.21.15.62:0/3952012115 learned_addr learned my addr 172.21.15.62:0/3952012115 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:41:14.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:14.357+0000 7fd605e00700 1 --2- 172.21.15.62:0/3952012115 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6080730f0 0x7fd60819c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:41:14.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:14.357+0000 7fd5fd400700 1 -- 172.21.15.62:0/3952012115 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6080730f0 msgr2=0x7fd60819c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:41:14.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:14.357+0000 7fd5fd400700 1 --2- 172.21.15.62:0/3952012115 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6080730f0 0x7fd60819c3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:41:14.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:14.357+0000 7fd5fd400700 1 -- 172.21.15.62:0/3952012115 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5f80095f0 con 0x7fd608073a00 2024-04-20T21:41:14.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:14.358+0000 7fd5fd400700 1 --2- 172.21.15.62:0/3952012115 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd608073a00 0x7fd60819c920 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fd5f000c930 tx=0x7fd5f000ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:41:14.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:14.358+0000 7fd5ff400700 1 -- 172.21.15.62:0/3952012115 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd5f0007ad0 con 0x7fd608073a00 2024-04-20T21:41:14.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:14.358+0000 7fd5ff400700 1 -- 172.21.15.62:0/3952012115 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd5f0007c30 con 0x7fd608073a00 2024-04-20T21:41:14.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:14.358+0000 7fd60d000700 1 -- 172.21.15.62:0/3952012115 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6081a0f00 con 0x7fd608073a00 2024-04-20T21:41:14.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:14.358+0000 7fd5ff400700 1 -- 172.21.15.62:0/3952012115 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd5f0018610 con 0x7fd608073a00 2024-04-20T21:41:14.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:14.358+0000 7fd60d000700 1 -- 172.21.15.62:0/3952012115 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6081a1450 con 0x7fd608073a00 2024-04-20T21:41:14.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:14.359+0000 7fd60d000700 1 -- 172.21.15.62:0/3952012115 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd60810e4d0 con 0x7fd608073a00 2024-04-20T21:41:14.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:14.361+0000 7fd5ff400700 1 -- 172.21.15.62:0/3952012115 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fd5f000f450 con 0x7fd608073a00 2024-04-20T21:41:14.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:14.361+0000 7fd5ff400700 1 --2- 172.21.15.62:0/3952012115 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd5e806ecc0 0x7fd5e8071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:41:14.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:14.362+0000 7fd5ff400700 1 -- 172.21.15.62:0/3952012115 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fd5f008efa0 con 0x7fd608073a00 2024-04-20T21:41:14.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:14.362+0000 7fd605e00700 1 --2- 172.21.15.62:0/3952012115 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd5e806ecc0 0x7fd5e8071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:41:14.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:14.363+0000 7fd605e00700 1 --2- 172.21.15.62:0/3952012115 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd5e806ecc0 0x7fd5e8071170 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fd5f8006010 tx=0x7fd5f801a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:41:14.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:14.367+0000 7fd5ff400700 1 -- 172.21.15.62:0/3952012115 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fd5f0057de0 con 0x7fd608073a00 2024-04-20T21:41:14.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:14.626+0000 7fd60d000700 1 -- 172.21.15.62:0/3952012115 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd6081a16d0 con 0x7fd5e806ecc0 2024-04-20T21:41:14.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:14.628+0000 7fd5ff400700 1 -- 172.21.15.62:0/3952012115 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7fd6081a16d0 con 0x7fd5e806ecc0 2024-04-20T21:41:14.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:14.631+0000 7fd60d000700 1 -- 172.21.15.62:0/3952012115 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd5e806ecc0 msgr2=0x7fd5e8071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:41:14.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:14.631+0000 7fd60d000700 1 --2- 172.21.15.62:0/3952012115 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd5e806ecc0 0x7fd5e8071170 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fd5f8006010 tx=0x7fd5f801a040 comp rx=0 tx=0).stop 2024-04-20T21:41:14.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:14.631+0000 7fd60d000700 1 -- 172.21.15.62:0/3952012115 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd608073a00 msgr2=0x7fd60819c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:41:14.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:14.631+0000 7fd60d000700 1 --2- 172.21.15.62:0/3952012115 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd608073a00 0x7fd60819c920 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fd5f000c930 tx=0x7fd5f000ccf0 comp rx=0 tx=0).stop 2024-04-20T21:41:14.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:14.632+0000 7fd60d000700 1 -- 172.21.15.62:0/3952012115 shutdown_connections 2024-04-20T21:41:14.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:14.632+0000 7fd60d000700 1 --2- 172.21.15.62:0/3952012115 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd5e806ecc0 0x7fd5e8071170 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:41:14.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:14.632+0000 7fd60d000700 1 --2- 172.21.15.62:0/3952012115 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6080730f0 0x7fd60819c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:41:14.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:14.632+0000 7fd60d000700 1 --2- 172.21.15.62:0/3952012115 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd608073a00 0x7fd60819c920 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:41:14.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:14.632+0000 7fd60d000700 1 -- 172.21.15.62:0/3952012115 >> 172.21.15.62:0/3952012115 conn(0x7fd6080fbda0 msgr2=0x7fd608102860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:41:14.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:14.632+0000 7fd60d000700 1 -- 172.21.15.62:0/3952012115 shutdown_connections 2024-04-20T21:41:14.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:14.632+0000 7fd60d000700 1 -- 172.21.15.62:0/3952012115 wait complete. 2024-04-20T21:41:14.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:14.797+0000 7f7405400700 1 -- 172.21.15.62:0/3297303631 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7400100c70 msgr2=0x7f7400104c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:41:14.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:14.797+0000 7f7405400700 1 --2- 172.21.15.62:0/3297303631 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7400100c70 0x7f7400104c40 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f73ec009910 tx=0x7f73ec009c20 comp rx=0 tx=0).stop 2024-04-20T21:41:14.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:14.797+0000 7f7405400700 1 -- 172.21.15.62:0/3297303631 shutdown_connections 2024-04-20T21:41:14.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:14.797+0000 7f7405400700 1 --2- 172.21.15.62:0/3297303631 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7400100c70 0x7f7400104c40 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:41:14.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:14.797+0000 7f7405400700 1 --2- 172.21.15.62:0/3297303631 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f74001002d0 0x7f74001006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:41:14.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:14.797+0000 7f7405400700 1 -- 172.21.15.62:0/3297303631 >> 172.21.15.62:0/3297303631 conn(0x7f74000fbbb0 msgr2=0x7f74000fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:41:14.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:14.797+0000 7f7405400700 1 -- 172.21.15.62:0/3297303631 shutdown_connections 2024-04-20T21:41:14.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:14.797+0000 7f7405400700 1 -- 172.21.15.62:0/3297303631 wait complete. 2024-04-20T21:41:14.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:14.798+0000 7f7405400700 1 Processor -- start 2024-04-20T21:41:14.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:14.798+0000 7f7405400700 1 -- start start 2024-04-20T21:41:14.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:14.799+0000 7f7405400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f74001002d0 0x7f7400073050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:41:14.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:14.799+0000 7f7405400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7400100c70 0x7f7400073590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:41:14.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:14.799+0000 7f7405400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7400073ad0 con 0x7f74001002d0 2024-04-20T21:41:14.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:14.799+0000 7f7405400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7400073c10 con 0x7f7400100c70 2024-04-20T21:41:14.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:14.799+0000 7f73ffe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f74001002d0 0x7f7400073050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:41:14.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:14.799+0000 7f73ffe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f74001002d0 0x7f7400073050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53930/0 (socket says 172.21.15.62:53930) 2024-04-20T21:41:14.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:14.799+0000 7f73ffe00700 1 -- 172.21.15.62:0/3018236856 learned_addr learned my addr 172.21.15.62:0/3018236856 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:41:14.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:14.799+0000 7f73ff400700 1 --2- 172.21.15.62:0/3018236856 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7400100c70 0x7f7400073590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:41:14.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:14.799+0000 7f73ffe00700 1 -- 172.21.15.62:0/3018236856 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7400100c70 msgr2=0x7f7400073590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:41:14.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:14.799+0000 7f73ffe00700 1 --2- 172.21.15.62:0/3018236856 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7400100c70 0x7f7400073590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:41:14.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:14.799+0000 7f73ffe00700 1 -- 172.21.15.62:0/3018236856 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73ec0095f0 con 0x7f74001002d0 2024-04-20T21:41:14.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:14.800+0000 7f73ffe00700 1 --2- 172.21.15.62:0/3018236856 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f74001002d0 0x7f7400073050 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f73f000d8d0 tx=0x7f73f000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:41:14.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:14.800+0000 7f73fcc00700 1 -- 172.21.15.62:0/3018236856 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f73f0009940 con 0x7f74001002d0 2024-04-20T21:41:14.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:14.800+0000 7f73fcc00700 1 -- 172.21.15.62:0/3018236856 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f73f0010460 con 0x7f74001002d0 2024-04-20T21:41:14.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:14.800+0000 7f73fcc00700 1 -- 172.21.15.62:0/3018236856 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f73f000f5f0 con 0x7f74001002d0 2024-04-20T21:41:14.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:14.800+0000 7f7405400700 1 -- 172.21.15.62:0/3018236856 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74001a1e30 con 0x7f74001002d0 2024-04-20T21:41:14.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:14.800+0000 7f7405400700 1 -- 172.21.15.62:0/3018236856 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74001a2240 con 0x7f74001002d0 2024-04-20T21:41:14.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:14.801+0000 7f7405400700 1 -- 172.21.15.62:0/3018236856 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7400062380 con 0x7f74001002d0 2024-04-20T21:41:14.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:14.802+0000 7f73fcc00700 1 -- 172.21.15.62:0/3018236856 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f73f00105d0 con 0x7f74001002d0 2024-04-20T21:41:14.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:14.803+0000 7f73fcc00700 1 --2- 172.21.15.62:0/3018236856 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f73e406ecc0 0x7f73e4071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:41:14.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:14.804+0000 7f73fcc00700 1 -- 172.21.15.62:0/3018236856 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f73f008ed60 con 0x7f74001002d0 2024-04-20T21:41:14.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:14.804+0000 7f73ff400700 1 --2- 172.21.15.62:0/3018236856 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f73e406ecc0 0x7f73e4071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:41:14.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:14.805+0000 7f73ff400700 1 --2- 172.21.15.62:0/3018236856 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f73e406ecc0 0x7f73e4071170 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f7400074840 tx=0x7f73ec01a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:41:14.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:14.809+0000 7f73fcc00700 1 -- 172.21.15.62:0/3018236856 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f73f0093070 con 0x7f74001002d0 2024-04-20T21:41:14.859 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:14 smithi189 ceph-mon[57022]: Health check update: Degraded data redundancy: 36/366 objects degraded (9.836%), 11 pgs degraded (PG_DEGRADED) 2024-04-20T21:41:15.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:14 smithi062 ceph-mon[76322]: Health check update: Degraded data redundancy: 36/366 objects degraded (9.836%), 11 pgs degraded (PG_DEGRADED) 2024-04-20T21:41:15.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:15.072+0000 7f7405400700 1 -- 172.21.15.62:0/3018236856 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f74000fdf00 con 0x7f73e406ecc0 2024-04-20T21:41:15.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:15.088+0000 7f73fcc00700 1 -- 172.21.15.62:0/3018236856 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7f74000fdf00 con 0x7f73e406ecc0 2024-04-20T21:41:15.087 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T21:41:15.087 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (7m) 79s ago 20m 14.2M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T21:41:15.087 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (21m) 79s ago 21m 8992k - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T21:41:15.088 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (19m) 7s ago 19m 8707k - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T21:41:15.088 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (3m) 79s ago 21m 6660k - 19.0.0-3133-ga8c84f59 76bc6b91b726 2825d5ba11c6 2024-04-20T21:41:15.088 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (3m) 7s ago 19m 6665k - 19.0.0-3133-ga8c84f59 76bc6b91b726 20f746fa1bba 2024-04-20T21:41:15.088 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (6m) 79s ago 20m 75.0M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T21:41:15.088 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (13m) 79s ago 13m 65.5M - 18.2.0 dc2bc1663786 5e6460aa6159 2024-04-20T21:41:15.088 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (13m) 79s ago 13m 131M - 18.2.0 dc2bc1663786 e65f2267544e 2024-04-20T21:41:15.088 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (13m) 7s ago 13m 117M - 18.2.0 dc2bc1663786 fb2a8a223db8 2024-04-20T21:41:15.088 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (13m) 7s ago 13m 55.6M - 18.2.0 dc2bc1663786 f2eec2ee8951 2024-04-20T21:41:15.088 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (9m) 79s ago 22m 504M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T21:41:15.088 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (8m) 7s ago 19m 451M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T21:41:15.088 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (4m) 79s ago 22m 49.3M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 019eb1479485 2024-04-20T21:41:15.088 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (4m) 7s ago 19m 38.7M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 55aac560aab6 2024-04-20T21:41:15.088 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (8m) 79s ago 21m 7906k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T21:41:15.088 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (8m) 7s ago 19m 8136k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T21:41:15.089 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (2m) 79s ago 18m 83.0M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 ee2bb901f26e 2024-04-20T21:41:15.089 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (119s) 79s ago 17m 87.9M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 9075152d4162 2024-04-20T21:41:15.089 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (87s) 79s ago 17m 12.7M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 2d4f3e38dc0e 2024-04-20T21:41:15.089 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (55s) 7s ago 16m 85.7M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 8f97b17ae0eb 2024-04-20T21:41:15.089 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (12s) 7s ago 16m 11.1M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 653d8079967d 2024-04-20T21:41:15.089 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (15m) 7s ago 15m 204M 2240M 18.2.0 dc2bc1663786 8e4574bae8db 2024-04-20T21:41:15.089 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (7m) 79s ago 20m 73.9M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T21:41:15.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:15.091+0000 7f7405400700 1 -- 172.21.15.62:0/3018236856 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f73e406ecc0 msgr2=0x7f73e4071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:41:15.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:15.091+0000 7f7405400700 1 --2- 172.21.15.62:0/3018236856 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f73e406ecc0 0x7f73e4071170 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f7400074840 tx=0x7f73ec01a040 comp rx=0 tx=0).stop 2024-04-20T21:41:15.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:15.091+0000 7f7405400700 1 -- 172.21.15.62:0/3018236856 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f74001002d0 msgr2=0x7f7400073050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:41:15.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:15.091+0000 7f7405400700 1 --2- 172.21.15.62:0/3018236856 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f74001002d0 0x7f7400073050 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f73f000d8d0 tx=0x7f73f000dc90 comp rx=0 tx=0).stop 2024-04-20T21:41:15.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:15.091+0000 7f7405400700 1 -- 172.21.15.62:0/3018236856 shutdown_connections 2024-04-20T21:41:15.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:15.091+0000 7f7405400700 1 --2- 172.21.15.62:0/3018236856 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f73e406ecc0 0x7f73e4071170 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:41:15.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:15.091+0000 7f7405400700 1 --2- 172.21.15.62:0/3018236856 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f74001002d0 0x7f7400073050 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:41:15.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:15.091+0000 7f7405400700 1 --2- 172.21.15.62:0/3018236856 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7400100c70 0x7f7400073590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:41:15.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:15.091+0000 7f7405400700 1 -- 172.21.15.62:0/3018236856 >> 172.21.15.62:0/3018236856 conn(0x7f74000fbbb0 msgr2=0x7f74000fd130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:41:15.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:15.091+0000 7f7405400700 1 -- 172.21.15.62:0/3018236856 shutdown_connections 2024-04-20T21:41:15.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:15.091+0000 7f7405400700 1 -- 172.21.15.62:0/3018236856 wait complete. 2024-04-20T21:41:15.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:15.250+0000 7f587c200700 1 -- 172.21.15.62:0/4115054914 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5874100e80 msgr2=0x7f5874104e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:41:15.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:15.250+0000 7f587c200700 1 --2- 172.21.15.62:0/4115054914 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5874100e80 0x7f5874104e50 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f5860009910 tx=0x7f5860009c20 comp rx=0 tx=0).stop 2024-04-20T21:41:15.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:15.250+0000 7f587c200700 1 -- 172.21.15.62:0/4115054914 shutdown_connections 2024-04-20T21:41:15.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:15.250+0000 7f587c200700 1 --2- 172.21.15.62:0/4115054914 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5874100e80 0x7f5874104e50 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:41:15.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:15.250+0000 7f587c200700 1 --2- 172.21.15.62:0/4115054914 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f58741004e0 0x7f58741008b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:41:15.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:15.250+0000 7f587c200700 1 -- 172.21.15.62:0/4115054914 >> 172.21.15.62:0/4115054914 conn(0x7f58740fbda0 msgr2=0x7f58740fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:41:15.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:15.250+0000 7f587c200700 1 -- 172.21.15.62:0/4115054914 shutdown_connections 2024-04-20T21:41:15.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:15.250+0000 7f587c200700 1 -- 172.21.15.62:0/4115054914 wait complete. 2024-04-20T21:41:15.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:15.251+0000 7f587c200700 1 Processor -- start 2024-04-20T21:41:15.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:15.251+0000 7f587c200700 1 -- start start 2024-04-20T21:41:15.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:15.252+0000 7f587c200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f58741004e0 0x7f5874073050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:41:15.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:15.252+0000 7f587c200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5874100e80 0x7f5874073590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:41:15.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:15.252+0000 7f587c200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5874073ad0 con 0x7f5874100e80 2024-04-20T21:41:15.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:15.252+0000 7f587c200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5874073c10 con 0x7f58741004e0 2024-04-20T21:41:15.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:15.252+0000 7f5878e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5874100e80 0x7f5874073590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:41:15.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:15.252+0000 7f5878e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5874100e80 0x7f5874073590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53952/0 (socket says 172.21.15.62:53952) 2024-04-20T21:41:15.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:15.252+0000 7f5878e00700 1 -- 172.21.15.62:0/1737625488 learned_addr learned my addr 172.21.15.62:0/1737625488 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:41:15.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:15.253+0000 7f5879800700 1 --2- 172.21.15.62:0/1737625488 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f58741004e0 0x7f5874073050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:41:15.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:15.253+0000 7f5878e00700 1 -- 172.21.15.62:0/1737625488 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f58741004e0 msgr2=0x7f5874073050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:41:15.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:15.253+0000 7f5878e00700 1 --2- 172.21.15.62:0/1737625488 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f58741004e0 0x7f5874073050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:41:15.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:15.253+0000 7f5878e00700 1 -- 172.21.15.62:0/1737625488 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58600095f0 con 0x7f5874100e80 2024-04-20T21:41:15.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:15.253+0000 7f5879800700 1 --2- 172.21.15.62:0/1737625488 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f58741004e0 0x7f5874073050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T21:41:15.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:15.253+0000 7f5878e00700 1 --2- 172.21.15.62:0/1737625488 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5874100e80 0x7f5874073590 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f586000bda0 tx=0x7f586000be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:41:15.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:15.253+0000 7f5872000700 1 -- 172.21.15.62:0/1737625488 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f586001d070 con 0x7f5874100e80 2024-04-20T21:41:15.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:15.253+0000 7f5872000700 1 -- 172.21.15.62:0/1737625488 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f586000f460 con 0x7f5874100e80 2024-04-20T21:41:15.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:15.253+0000 7f5872000700 1 -- 172.21.15.62:0/1737625488 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5860017630 con 0x7f5874100e80 2024-04-20T21:41:15.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:15.253+0000 7f587c200700 1 -- 172.21.15.62:0/1737625488 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5874073e90 con 0x7f5874100e80 2024-04-20T21:41:15.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:15.254+0000 7f587c200700 1 -- 172.21.15.62:0/1737625488 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58741a2430 con 0x7f5874100e80 2024-04-20T21:41:15.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:15.255+0000 7f587c200700 1 -- 172.21.15.62:0/1737625488 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5874108850 con 0x7f5874100e80 2024-04-20T21:41:15.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:15.256+0000 7f5872000700 1 -- 172.21.15.62:0/1737625488 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f586000fa40 con 0x7f5874100e80 2024-04-20T21:41:15.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:15.258+0000 7f5872000700 1 --2- 172.21.15.62:0/1737625488 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f585c06ecc0 0x7f585c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:41:15.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:15.258+0000 7f5872000700 1 -- 172.21.15.62:0/1737625488 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f5860090430 con 0x7f5874100e80 2024-04-20T21:41:15.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:15.258+0000 7f5879800700 1 --2- 172.21.15.62:0/1737625488 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f585c06ecc0 0x7f585c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:41:15.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:15.259+0000 7f5879800700 1 --2- 172.21.15.62:0/1737625488 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f585c06ecc0 0x7f585c071170 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f5868005fd0 tx=0x7f5868005ee0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:41:15.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:15.262+0000 7f5872000700 1 -- 172.21.15.62:0/1737625488 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f58600592f0 con 0x7f5874100e80 2024-04-20T21:41:15.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:15.604+0000 7f587c200700 1 -- 172.21.15.62:0/1737625488 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f587404f2a0 con 0x7f5874100e80 2024-04-20T21:41:15.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:15.604+0000 7f5872000700 1 -- 172.21.15.62:0/1737625488 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+791 (secure 0 0 0) 0x7f586005c9b0 con 0x7f5874100e80 2024-04-20T21:41:15.604 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:41:15.604 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T21:41:15.604 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:41:15.604 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:41:15.604 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T21:41:15.604 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:41:15.605 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:41:15.605 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T21:41:15.605 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 1, 2024-04-20T21:41:15.605 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 5 2024-04-20T21:41:15.605 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:41:15.605 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T21:41:15.605 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 4 2024-04-20T21:41:15.605 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:41:15.605 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T21:41:15.605 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 5, 2024-04-20T21:41:15.605 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 9 2024-04-20T21:41:15.605 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T21:41:15.605 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:41:15.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:15.607+0000 7f587c200700 1 -- 172.21.15.62:0/1737625488 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f585c06ecc0 msgr2=0x7f585c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:41:15.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:15.607+0000 7f587c200700 1 --2- 172.21.15.62:0/1737625488 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f585c06ecc0 0x7f585c071170 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f5868005fd0 tx=0x7f5868005ee0 comp rx=0 tx=0).stop 2024-04-20T21:41:15.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:15.607+0000 7f587c200700 1 -- 172.21.15.62:0/1737625488 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5874100e80 msgr2=0x7f5874073590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:41:15.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:15.607+0000 7f587c200700 1 --2- 172.21.15.62:0/1737625488 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5874100e80 0x7f5874073590 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f586000bda0 tx=0x7f586000be80 comp rx=0 tx=0).stop 2024-04-20T21:41:15.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:15.608+0000 7f587c200700 1 -- 172.21.15.62:0/1737625488 shutdown_connections 2024-04-20T21:41:15.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:15.608+0000 7f587c200700 1 --2- 172.21.15.62:0/1737625488 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f585c06ecc0 0x7f585c071170 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:41:15.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:15.608+0000 7f587c200700 1 --2- 172.21.15.62:0/1737625488 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f58741004e0 0x7f5874073050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:41:15.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:15.608+0000 7f587c200700 1 --2- 172.21.15.62:0/1737625488 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5874100e80 0x7f5874073590 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:41:15.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:15.608+0000 7f587c200700 1 -- 172.21.15.62:0/1737625488 >> 172.21.15.62:0/1737625488 conn(0x7f58740fbda0 msgr2=0x7f58740fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:41:15.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:15.608+0000 7f587c200700 1 -- 172.21.15.62:0/1737625488 shutdown_connections 2024-04-20T21:41:15.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:15.608+0000 7f587c200700 1 -- 172.21.15.62:0/1737625488 wait complete. 2024-04-20T21:41:15.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:15.758+0000 7f74d7e00700 1 -- 172.21.15.62:0/3979633877 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f74d01012f0 msgr2=0x7f74d01016c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:41:15.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:15.758+0000 7f74d7e00700 1 --2- 172.21.15.62:0/3979633877 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f74d01012f0 0x7f74d01016c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f74c4009910 tx=0x7f74c4009c20 comp rx=0 tx=0).stop 2024-04-20T21:41:15.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:15.758+0000 7f74d7e00700 1 -- 172.21.15.62:0/3979633877 shutdown_connections 2024-04-20T21:41:15.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:15.758+0000 7f74d7e00700 1 --2- 172.21.15.62:0/3979633877 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f74d0101c00 0x7f74d010a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:41:15.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:15.758+0000 7f74d7e00700 1 --2- 172.21.15.62:0/3979633877 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f74d01012f0 0x7f74d01016c0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:41:15.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:15.758+0000 7f74d7e00700 1 -- 172.21.15.62:0/3979633877 >> 172.21.15.62:0/3979633877 conn(0x7f74d00fac60 msgr2=0x7f74d00fd070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:41:15.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:15.758+0000 7f74d7e00700 1 -- 172.21.15.62:0/3979633877 shutdown_connections 2024-04-20T21:41:15.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:15.758+0000 7f74d7e00700 1 -- 172.21.15.62:0/3979633877 wait complete. 2024-04-20T21:41:15.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:15.760+0000 7f74d7e00700 1 Processor -- start 2024-04-20T21:41:15.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:15.760+0000 7f74d7e00700 1 -- start start 2024-04-20T21:41:15.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:15.761+0000 7f74d7e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f74d01012f0 0x7f74d0198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:41:15.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:15.761+0000 7f74d7e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f74d0101c00 0x7f74d01985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:41:15.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:15.761+0000 7f74d7e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74d0198c00 con 0x7f74d0101c00 2024-04-20T21:41:15.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:15.761+0000 7f74d7e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74d0198d40 con 0x7f74d01012f0 2024-04-20T21:41:15.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:15.761+0000 7f74d4a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f74d0101c00 0x7f74d01985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:41:15.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:15.761+0000 7f74d4a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f74d0101c00 0x7f74d01985b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53964/0 (socket says 172.21.15.62:53964) 2024-04-20T21:41:15.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:15.761+0000 7f74d4a00700 1 -- 172.21.15.62:0/1504808013 learned_addr learned my addr 172.21.15.62:0/1504808013 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:41:15.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:15.761+0000 7f74d5400700 1 --2- 172.21.15.62:0/1504808013 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f74d01012f0 0x7f74d0198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:41:15.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:15.761+0000 7f74d4a00700 1 -- 172.21.15.62:0/1504808013 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f74d01012f0 msgr2=0x7f74d0198070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:41:15.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:15.762+0000 7f74d4a00700 1 --2- 172.21.15.62:0/1504808013 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f74d01012f0 0x7f74d0198070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:41:15.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:15.762+0000 7f74d4a00700 1 -- 172.21.15.62:0/1504808013 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74c40095f0 con 0x7f74d0101c00 2024-04-20T21:41:15.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:15.762+0000 7f74d4a00700 1 --2- 172.21.15.62:0/1504808013 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f74d0101c00 0x7f74d01985b0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f74bc00c930 tx=0x7f74bc00ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:41:15.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:15.763+0000 7f74ce000700 1 -- 172.21.15.62:0/1504808013 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f74bc007ad0 con 0x7f74d0101c00 2024-04-20T21:41:15.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:15.763+0000 7f74ce000700 1 -- 172.21.15.62:0/1504808013 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f74bc007c30 con 0x7f74d0101c00 2024-04-20T21:41:15.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:15.763+0000 7f74d7e00700 1 -- 172.21.15.62:0/1504808013 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74d019cb90 con 0x7f74d0101c00 2024-04-20T21:41:15.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:15.763+0000 7f74ce000700 1 -- 172.21.15.62:0/1504808013 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f74bc018610 con 0x7f74d0101c00 2024-04-20T21:41:15.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:15.763+0000 7f74d7e00700 1 -- 172.21.15.62:0/1504808013 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74d019d0e0 con 0x7f74d0101c00 2024-04-20T21:41:15.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:15.765+0000 7f74d7e00700 1 -- 172.21.15.62:0/1504808013 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74d0109dd0 con 0x7f74d0101c00 2024-04-20T21:41:15.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:15.766+0000 7f74ce000700 1 -- 172.21.15.62:0/1504808013 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f74bc00f450 con 0x7f74d0101c00 2024-04-20T21:41:15.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:15.767+0000 7f74ce000700 1 --2- 172.21.15.62:0/1504808013 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f74b806ecc0 0x7f74b8071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:41:15.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:15.767+0000 7f74ce000700 1 -- 172.21.15.62:0/1504808013 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f74bc08ef00 con 0x7f74d0101c00 2024-04-20T21:41:15.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:15.767+0000 7f74d5400700 1 --2- 172.21.15.62:0/1504808013 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f74b806ecc0 0x7f74b8071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:41:15.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:15.768+0000 7f74d5400700 1 --2- 172.21.15.62:0/1504808013 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f74b806ecc0 0x7f74b8071170 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f74c4006010 tx=0x7f74c401a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:41:15.769 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:15 smithi062 ceph-mon[76322]: from='client.34370 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:41:15.769 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:15 smithi062 ceph-mon[76322]: pgmap v168: 65 pgs: 10 peering, 7 active+undersized+degraded, 9 active+undersized, 39 active+clean; 315 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s rd, 3 op/s; 23/366 objects degraded (6.284%) 2024-04-20T21:41:15.769 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:15 smithi062 ceph-mon[76322]: from='client.34374 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:41:15.769 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:15 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:41:15.769 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:15 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:41:15.769 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:15 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:41:15.769 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:15 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T21:41:15.769 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:15 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:41:15.769 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:15 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:41:15.769 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:15 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:41:15.769 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:15 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:41:15.769 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:15 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:41:15.769 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:15 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-20T21:41:15.770 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:15 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/1737625488' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:41:15.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:15.772+0000 7f74ce000700 1 -- 172.21.15.62:0/1504808013 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f74bc057de0 con 0x7f74d0101c00 2024-04-20T21:41:16.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.042+0000 7f74d7e00700 1 -- 172.21.15.62:0/1504808013 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f74d004f2a0 con 0x7f74d0101c00 2024-04-20T21:41:16.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.043+0000 7f74ce000700 1 -- 172.21.15.62:0/1504808013 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 18 v18) v1 ==== 76+0+1928 (secure 0 0 0) 0x7f74bc05b4a0 con 0x7f74d0101c00 2024-04-20T21:41:16.043 INFO:teuthology.orchestra.run.smithi062.stdout:e18 2024-04-20T21:41:16.043 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T21:41:16.043 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T21:41:16.043 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T21:41:16.043 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:41:16.043 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T21:41:16.043 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T21:41:16.044 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 18 2024-04-20T21:41:16.044 INFO:teuthology.orchestra.run.smithi062.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-20T21:41:16.044 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T21:41:16.044 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T21:37:06.018300+0000 2024-04-20T21:41:16.044 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T21:41:16.044 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T21:41:16.044 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T21:41:16.044 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T21:41:16.044 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T21:41:16.044 INFO:teuthology.orchestra.run.smithi062.stdout:max_xattr_size 65536 2024-04-20T21:41:16.044 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T21:41:16.044 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T21:41:16.044 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 0 2024-04-20T21:41:16.044 INFO:teuthology.orchestra.run.smithi062.stdout:snap_rstat 1 2024-04-20T21:41:16.044 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T21:41:16.045 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 2 2024-04-20T21:41:16.045 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1 2024-04-20T21:41:16.045 INFO:teuthology.orchestra.run.smithi062.stdout:up {0=14458,1=24279} 2024-04-20T21:41:16.045 INFO:teuthology.orchestra.run.smithi062.stdout:failed 2024-04-20T21:41:16.045 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T21:41:16.045 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T21:41:16.045 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T21:41:16.045 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T21:41:16.045 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data enabled 2024-04-20T21:41:16.045 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T21:41:16.045 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T21:41:16.045 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 1 2024-04-20T21:41:16.045 INFO:teuthology.orchestra.run.smithi062.stdout:qdb_cluster leader: 14458 members: 24279,14458 2024-04-20T21:41:16.045 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ryqblb{0:14458} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.62:6826/1147127123,v1:172.21.15.62:6827/1147127123] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:41:16.045 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ozizwi{0:14474} state up:standby-replay seq 3 join_fscid=1 addr [v2:172.21.15.62:6828/3494174293,v1:172.21.15.62:6829/3494174293] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:41:16.046 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.dvimer{1:24279} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.189:6824/237262514,v1:172.21.15.189:6825/237262514] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:41:16.046 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.xbgspw{1:24291} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.189:6826/1495356258,v1:172.21.15.189:6827/1495356258] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:41:16.046 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:41:16.046 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:41:16.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.046+0000 7f74d7e00700 1 -- 172.21.15.62:0/1504808013 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f74b806ecc0 msgr2=0x7f74b8071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:41:16.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.046+0000 7f74d7e00700 1 --2- 172.21.15.62:0/1504808013 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f74b806ecc0 0x7f74b8071170 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f74c4006010 tx=0x7f74c401a040 comp rx=0 tx=0).stop 2024-04-20T21:41:16.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.046+0000 7f74d7e00700 1 -- 172.21.15.62:0/1504808013 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f74d0101c00 msgr2=0x7f74d01985b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:41:16.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.046+0000 7f74d7e00700 1 --2- 172.21.15.62:0/1504808013 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f74d0101c00 0x7f74d01985b0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f74bc00c930 tx=0x7f74bc00ccf0 comp rx=0 tx=0).stop 2024-04-20T21:41:16.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.046+0000 7f74d7e00700 1 -- 172.21.15.62:0/1504808013 shutdown_connections 2024-04-20T21:41:16.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.046+0000 7f74d7e00700 1 --2- 172.21.15.62:0/1504808013 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f74b806ecc0 0x7f74b8071170 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:41:16.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.046+0000 7f74d7e00700 1 --2- 172.21.15.62:0/1504808013 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f74d01012f0 0x7f74d0198070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:41:16.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.046+0000 7f74d7e00700 1 --2- 172.21.15.62:0/1504808013 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f74d0101c00 0x7f74d01985b0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:41:16.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.046+0000 7f74d7e00700 1 -- 172.21.15.62:0/1504808013 >> 172.21.15.62:0/1504808013 conn(0x7f74d00fac60 msgr2=0x7f74d00ff8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:41:16.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.046+0000 7f74d7e00700 1 -- 172.21.15.62:0/1504808013 shutdown_connections 2024-04-20T21:41:16.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.046+0000 7f74d7e00700 1 -- 172.21.15.62:0/1504808013 wait complete. 2024-04-20T21:41:16.048 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 18 2024-04-20T21:41:16.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.208+0000 7fe24c200700 1 -- 172.21.15.62:0/1748351841 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2441025b0 msgr2=0x7fe244102980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:41:16.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.208+0000 7fe24c200700 1 --2- 172.21.15.62:0/1748351841 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2441025b0 0x7fe244102980 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fe238009910 tx=0x7fe238009c20 comp rx=0 tx=0).stop 2024-04-20T21:41:16.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.208+0000 7fe24c200700 1 -- 172.21.15.62:0/1748351841 shutdown_connections 2024-04-20T21:41:16.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.208+0000 7fe24c200700 1 --2- 172.21.15.62:0/1748351841 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe244102f50 0x7fe244106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:41:16.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.208+0000 7fe24c200700 1 --2- 172.21.15.62:0/1748351841 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2441025b0 0x7fe244102980 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:41:16.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.208+0000 7fe24c200700 1 -- 172.21.15.62:0/1748351841 >> 172.21.15.62:0/1748351841 conn(0x7fe2440fde90 msgr2=0x7fe2441002a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:41:16.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.208+0000 7fe24c200700 1 -- 172.21.15.62:0/1748351841 shutdown_connections 2024-04-20T21:41:16.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.209+0000 7fe24c200700 1 -- 172.21.15.62:0/1748351841 wait complete. 2024-04-20T21:41:16.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.210+0000 7fe24c200700 1 Processor -- start 2024-04-20T21:41:16.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.210+0000 7fe24c200700 1 -- start start 2024-04-20T21:41:16.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.210+0000 7fe24c200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2441025b0 0x7fe244197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:41:16.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.210+0000 7fe24c200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe244102f50 0x7fe244198440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:41:16.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.210+0000 7fe24c200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe244198a90 con 0x7fe2441025b0 2024-04-20T21:41:16.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.210+0000 7fe24c200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe244198bd0 con 0x7fe244102f50 2024-04-20T21:41:16.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.210+0000 7fe249800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2441025b0 0x7fe244197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:41:16.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.210+0000 7fe249800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2441025b0 0x7fe244197f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53972/0 (socket says 172.21.15.62:53972) 2024-04-20T21:41:16.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.210+0000 7fe249800700 1 -- 172.21.15.62:0/1182425970 learned_addr learned my addr 172.21.15.62:0/1182425970 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:41:16.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.211+0000 7fe248e00700 1 --2- 172.21.15.62:0/1182425970 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe244102f50 0x7fe244198440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:41:16.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.211+0000 7fe249800700 1 -- 172.21.15.62:0/1182425970 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe244102f50 msgr2=0x7fe244198440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:41:16.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.211+0000 7fe249800700 1 --2- 172.21.15.62:0/1182425970 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe244102f50 0x7fe244198440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:41:16.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.211+0000 7fe249800700 1 -- 172.21.15.62:0/1182425970 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2380095f0 con 0x7fe2441025b0 2024-04-20T21:41:16.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.211+0000 7fe248e00700 1 --2- 172.21.15.62:0/1182425970 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe244102f50 0x7fe244198440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T21:41:16.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.211+0000 7fe249800700 1 --2- 172.21.15.62:0/1182425970 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2441025b0 0x7fe244197f00 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fe238006010 tx=0x7fe23800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:41:16.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.212+0000 7fe242000700 1 -- 172.21.15.62:0/1182425970 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe23801d070 con 0x7fe2441025b0 2024-04-20T21:41:16.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.212+0000 7fe242000700 1 -- 172.21.15.62:0/1182425970 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe238004fb0 con 0x7fe2441025b0 2024-04-20T21:41:16.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.212+0000 7fe24c200700 1 -- 172.21.15.62:0/1182425970 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe24419c9c0 con 0x7fe2441025b0 2024-04-20T21:41:16.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.212+0000 7fe242000700 1 -- 172.21.15.62:0/1182425970 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe238022ea0 con 0x7fe2441025b0 2024-04-20T21:41:16.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.212+0000 7fe24c200700 1 -- 172.21.15.62:0/1182425970 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe24419ceb0 con 0x7fe2441025b0 2024-04-20T21:41:16.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.213+0000 7fe24c200700 1 -- 172.21.15.62:0/1182425970 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe24410a8c0 con 0x7fe2441025b0 2024-04-20T21:41:16.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.214+0000 7fe242000700 1 -- 172.21.15.62:0/1182425970 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe238022470 con 0x7fe2441025b0 2024-04-20T21:41:16.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.215+0000 7fe242000700 1 --2- 172.21.15.62:0/1182425970 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe22c06ecc0 0x7fe22c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:41:16.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.215+0000 7fe242000700 1 -- 172.21.15.62:0/1182425970 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fe2380903e0 con 0x7fe2441025b0 2024-04-20T21:41:16.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.215+0000 7fe248e00700 1 --2- 172.21.15.62:0/1182425970 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe22c06ecc0 0x7fe22c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:41:16.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.216+0000 7fe248e00700 1 --2- 172.21.15.62:0/1182425970 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe22c06ecc0 0x7fe22c071170 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fe230005fd0 tx=0x7fe230005e20 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:41:16.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.220+0000 7fe242000700 1 -- 172.21.15.62:0/1182425970 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fe238059250 con 0x7fe2441025b0 2024-04-20T21:41:16.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:15 smithi189 ceph-mon[57022]: from='client.34370 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:41:16.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:15 smithi189 ceph-mon[57022]: pgmap v168: 65 pgs: 10 peering, 7 active+undersized+degraded, 9 active+undersized, 39 active+clean; 315 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s rd, 3 op/s; 23/366 objects degraded (6.284%) 2024-04-20T21:41:16.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:15 smithi189 ceph-mon[57022]: from='client.34374 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:41:16.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:15 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:41:16.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:15 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:41:16.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:15 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:41:16.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:15 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T21:41:16.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:15 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:41:16.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:15 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:41:16.239 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:15 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:41:16.239 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:15 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:41:16.239 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:15 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:41:16.239 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:15 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-20T21:41:16.239 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:15 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/1737625488' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:41:16.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.477+0000 7fe24c200700 1 -- 172.21.15.62:0/1182425970 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe24402cca0 con 0x7fe22c06ecc0 2024-04-20T21:41:16.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.479+0000 7fe242000700 1 -- 172.21.15.62:0/1182425970 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7fe24402cca0 con 0x7fe22c06ecc0 2024-04-20T21:41:16.479 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:41:16.479 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T21:41:16.479 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T21:41:16.479 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-20T21:41:16.479 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T21:41:16.479 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr", 2024-04-20T21:41:16.479 INFO:teuthology.orchestra.run.smithi062.stdout: "crash", 2024-04-20T21:41:16.479 INFO:teuthology.orchestra.run.smithi062.stdout: "mon" 2024-04-20T21:41:16.479 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T21:41:16.479 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "11/23 daemons upgraded", 2024-04-20T21:41:16.479 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading osd daemons", 2024-04-20T21:41:16.479 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T21:41:16.479 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:41:16.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.483+0000 7fe24c200700 1 -- 172.21.15.62:0/1182425970 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe22c06ecc0 msgr2=0x7fe22c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:41:16.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.483+0000 7fe24c200700 1 --2- 172.21.15.62:0/1182425970 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe22c06ecc0 0x7fe22c071170 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fe230005fd0 tx=0x7fe230005e20 comp rx=0 tx=0).stop 2024-04-20T21:41:16.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.483+0000 7fe24c200700 1 -- 172.21.15.62:0/1182425970 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2441025b0 msgr2=0x7fe244197f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:41:16.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.483+0000 7fe24c200700 1 --2- 172.21.15.62:0/1182425970 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2441025b0 0x7fe244197f00 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fe238006010 tx=0x7fe23800be20 comp rx=0 tx=0).stop 2024-04-20T21:41:16.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.483+0000 7fe24c200700 1 -- 172.21.15.62:0/1182425970 shutdown_connections 2024-04-20T21:41:16.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.483+0000 7fe24c200700 1 --2- 172.21.15.62:0/1182425970 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe22c06ecc0 0x7fe22c071170 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:41:16.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.483+0000 7fe24c200700 1 --2- 172.21.15.62:0/1182425970 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2441025b0 0x7fe244197f00 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:41:16.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.483+0000 7fe24c200700 1 --2- 172.21.15.62:0/1182425970 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe244102f50 0x7fe244198440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:41:16.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.483+0000 7fe24c200700 1 -- 172.21.15.62:0/1182425970 >> 172.21.15.62:0/1182425970 conn(0x7fe2440fde90 msgr2=0x7fe244100230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:41:16.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.483+0000 7fe24c200700 1 -- 172.21.15.62:0/1182425970 shutdown_connections 2024-04-20T21:41:16.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.483+0000 7fe24c200700 1 -- 172.21.15.62:0/1182425970 wait complete. 2024-04-20T21:41:16.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.641+0000 7f181be00700 1 -- 172.21.15.62:0/3352263161 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f181410c8f0 msgr2=0x7f181410ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:41:16.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.641+0000 7f181be00700 1 --2- 172.21.15.62:0/3352263161 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f181410c8f0 0x7f181410ccc0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f1808009910 tx=0x7f1808009c20 comp rx=0 tx=0).stop 2024-04-20T21:41:16.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.642+0000 7f181be00700 1 -- 172.21.15.62:0/3352263161 shutdown_connections 2024-04-20T21:41:16.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.642+0000 7f181be00700 1 --2- 172.21.15.62:0/3352263161 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1814106b80 0x7f1814106ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:41:16.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.642+0000 7f181be00700 1 --2- 172.21.15.62:0/3352263161 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f181410c8f0 0x7f181410ccc0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:41:16.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.642+0000 7f181be00700 1 -- 172.21.15.62:0/3352263161 >> 172.21.15.62:0/3352263161 conn(0x7f1814076060 msgr2=0x7f1814078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:41:16.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.642+0000 7f181be00700 1 -- 172.21.15.62:0/3352263161 shutdown_connections 2024-04-20T21:41:16.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.642+0000 7f181be00700 1 -- 172.21.15.62:0/3352263161 wait complete. 2024-04-20T21:41:16.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.643+0000 7f181be00700 1 Processor -- start 2024-04-20T21:41:16.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.643+0000 7f181be00700 1 -- start start 2024-04-20T21:41:16.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.643+0000 7f181be00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1814106b80 0x7f181419c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:41:16.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.643+0000 7f181be00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f181410c8f0 0x7f181419ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:41:16.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.643+0000 7f181be00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f181419d090 con 0x7f181410c8f0 2024-04-20T21:41:16.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.643+0000 7f181be00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f181419d1d0 con 0x7f1814106b80 2024-04-20T21:41:16.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.644+0000 7f1818a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f181410c8f0 0x7f181419ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:41:16.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.644+0000 7f1818a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f181410c8f0 0x7f181419ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53988/0 (socket says 172.21.15.62:53988) 2024-04-20T21:41:16.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.644+0000 7f1818a00700 1 -- 172.21.15.62:0/1082581387 learned_addr learned my addr 172.21.15.62:0/1082581387 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:41:16.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.644+0000 7f1819400700 1 --2- 172.21.15.62:0/1082581387 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1814106b80 0x7f181419c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:41:16.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.644+0000 7f1818a00700 1 -- 172.21.15.62:0/1082581387 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1814106b80 msgr2=0x7f181419c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:41:16.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.644+0000 7f1818a00700 1 --2- 172.21.15.62:0/1082581387 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1814106b80 0x7f181419c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:41:16.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.644+0000 7f1818a00700 1 -- 172.21.15.62:0/1082581387 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18080095f0 con 0x7f181410c8f0 2024-04-20T21:41:16.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.645+0000 7f1818a00700 1 --2- 172.21.15.62:0/1082581387 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f181410c8f0 0x7f181419ca40 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f180000c930 tx=0x7f180000ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:41:16.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.645+0000 7f1812000700 1 -- 172.21.15.62:0/1082581387 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1800007ad0 con 0x7f181410c8f0 2024-04-20T21:41:16.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.645+0000 7f1812000700 1 -- 172.21.15.62:0/1082581387 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1800007c30 con 0x7f181410c8f0 2024-04-20T21:41:16.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.645+0000 7f181be00700 1 -- 172.21.15.62:0/1082581387 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18141a1020 con 0x7f181410c8f0 2024-04-20T21:41:16.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.645+0000 7f1812000700 1 -- 172.21.15.62:0/1082581387 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1800018610 con 0x7f181410c8f0 2024-04-20T21:41:16.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.645+0000 7f181be00700 1 -- 172.21.15.62:0/1082581387 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18141a1570 con 0x7f181410c8f0 2024-04-20T21:41:16.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.646+0000 7f181be00700 1 -- 172.21.15.62:0/1082581387 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1814109b20 con 0x7f181410c8f0 2024-04-20T21:41:16.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.647+0000 7f1812000700 1 -- 172.21.15.62:0/1082581387 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f180000f450 con 0x7f181410c8f0 2024-04-20T21:41:16.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.648+0000 7f1812000700 1 --2- 172.21.15.62:0/1082581387 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f17fc06ecc0 0x7f17fc071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:41:16.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.648+0000 7f1812000700 1 -- 172.21.15.62:0/1082581387 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f180008f070 con 0x7f181410c8f0 2024-04-20T21:41:16.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.648+0000 7f1819400700 1 --2- 172.21.15.62:0/1082581387 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f17fc06ecc0 0x7f17fc071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:41:16.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.649+0000 7f1819400700 1 --2- 172.21.15.62:0/1082581387 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f17fc06ecc0 0x7f17fc071170 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f1808006010 tx=0x7f180801a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:41:16.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.653+0000 7f1812000700 1 -- 172.21.15.62:0/1082581387 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f18000c3ff0 con 0x7f181410c8f0 2024-04-20T21:41:16.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.987+0000 7f181be00700 1 -- 172.21.15.62:0/1082581387 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f181404f2a0 con 0x7f181410c8f0 2024-04-20T21:41:16.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.987+0000 7f1812000700 1 -- 172.21.15.62:0/1082581387 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7f1800057e30 con 0x7f181410c8f0 2024-04-20T21:41:16.987 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-04-20T21:41:16.987 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T21:41:16.987 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T21:41:16.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.990+0000 7f181be00700 1 -- 172.21.15.62:0/1082581387 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f17fc06ecc0 msgr2=0x7f17fc071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:41:16.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.990+0000 7f181be00700 1 --2- 172.21.15.62:0/1082581387 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f17fc06ecc0 0x7f17fc071170 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f1808006010 tx=0x7f180801a040 comp rx=0 tx=0).stop 2024-04-20T21:41:16.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.991+0000 7f181be00700 1 -- 172.21.15.62:0/1082581387 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f181410c8f0 msgr2=0x7f181419ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:41:16.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.991+0000 7f181be00700 1 --2- 172.21.15.62:0/1082581387 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f181410c8f0 0x7f181419ca40 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f180000c930 tx=0x7f180000ccf0 comp rx=0 tx=0).stop 2024-04-20T21:41:16.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.991+0000 7f181be00700 1 -- 172.21.15.62:0/1082581387 shutdown_connections 2024-04-20T21:41:16.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.991+0000 7f181be00700 1 --2- 172.21.15.62:0/1082581387 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f17fc06ecc0 0x7f17fc071170 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:41:16.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.991+0000 7f181be00700 1 --2- 172.21.15.62:0/1082581387 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1814106b80 0x7f181419c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:41:16.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.991+0000 7f181be00700 1 --2- 172.21.15.62:0/1082581387 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f181410c8f0 0x7f181419ca40 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:41:16.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.991+0000 7f181be00700 1 -- 172.21.15.62:0/1082581387 >> 172.21.15.62:0/1082581387 conn(0x7f1814076060 msgr2=0x7f1814077aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:41:16.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.991+0000 7f181be00700 1 -- 172.21.15.62:0/1082581387 shutdown_connections 2024-04-20T21:41:16.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:16.991+0000 7f181be00700 1 -- 172.21.15.62:0/1082581387 wait complete. 2024-04-20T21:41:17.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:16 smithi062 ceph-mon[76322]: from='client.34378 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:41:17.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:16 smithi062 ceph-mon[76322]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-20T21:41:17.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:16 smithi062 ceph-mon[76322]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-04-20T21:41:17.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:16 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/1504808013' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T21:41:17.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:16 smithi189 ceph-mon[57022]: from='client.34378 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:41:17.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:16 smithi189 ceph-mon[57022]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-20T21:41:17.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:16 smithi189 ceph-mon[57022]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-04-20T21:41:17.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:16 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/1504808013' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T21:41:18.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:17 smithi062 ceph-mon[76322]: pgmap v169: 65 pgs: 10 peering, 55 active+clean; 315 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 2 op/s 2024-04-20T21:41:18.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:17 smithi062 ceph-mon[76322]: from='client.34390 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:41:18.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:17 smithi062 ceph-mon[76322]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 23/366 objects degraded (6.284%), 7 pgs degraded) 2024-04-20T21:41:18.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:17 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/1082581387' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T21:41:18.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:17 smithi189 ceph-mon[57022]: pgmap v169: 65 pgs: 10 peering, 55 active+clean; 315 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 2 op/s 2024-04-20T21:41:18.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:17 smithi189 ceph-mon[57022]: from='client.34390 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:41:18.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:17 smithi189 ceph-mon[57022]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 23/366 objects degraded (6.284%), 7 pgs degraded) 2024-04-20T21:41:18.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:17 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/1082581387' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T21:41:20.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:19 smithi062 ceph-mon[76322]: pgmap v170: 65 pgs: 65 active+clean; 315 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s rd, 2 op/s 2024-04-20T21:41:20.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:19 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:41:20.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:19 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:41:20.203 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:19 smithi189 ceph-mon[57022]: pgmap v170: 65 pgs: 65 active+clean; 315 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s rd, 2 op/s 2024-04-20T21:41:20.203 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:19 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:41:20.203 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:19 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:41:22.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:21 smithi189 ceph-mon[57022]: pgmap v171: 65 pgs: 65 active+clean; 315 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-20T21:41:22.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:21 smithi062 ceph-mon[76322]: pgmap v171: 65 pgs: 65 active+clean; 315 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-20T21:41:24.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:23 smithi189 ceph-mon[57022]: pgmap v172: 65 pgs: 65 active+clean; 315 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-20T21:41:24.284 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:23 smithi062 ceph-mon[76322]: pgmap v172: 65 pgs: 65 active+clean; 315 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-20T21:41:25.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:25 smithi189 ceph-mon[57022]: pgmap v173: 65 pgs: 65 active+clean; 315 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-20T21:41:25.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:25 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:41:25.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:25 smithi062 ceph-mon[76322]: pgmap v173: 65 pgs: 65 active+clean; 315 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-20T21:41:25.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:25 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:41:27.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:27 smithi189 ceph-mon[57022]: pgmap v174: 65 pgs: 65 active+clean; 315 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-20T21:41:27.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:27 smithi062 ceph-mon[76322]: pgmap v174: 65 pgs: 65 active+clean; 315 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-20T21:41:29.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:29 smithi189 ceph-mon[57022]: pgmap v175: 65 pgs: 65 active+clean; 315 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-20T21:41:29.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:29 smithi062 ceph-mon[76322]: pgmap v175: 65 pgs: 65 active+clean; 315 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-20T21:41:30.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:30 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-20T21:41:30.986 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:30 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-20T21:41:31.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:31 smithi189 ceph-mon[57022]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-20T21:41:31.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:31 smithi189 ceph-mon[57022]: Upgrade: osd.5 is safe to restart 2024-04-20T21:41:31.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:31 smithi189 ceph-mon[57022]: pgmap v176: 65 pgs: 65 active+clean; 315 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-20T21:41:31.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:31 smithi062 ceph-mon[76322]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-20T21:41:31.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:31 smithi062 ceph-mon[76322]: Upgrade: osd.5 is safe to restart 2024-04-20T21:41:31.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:31 smithi062 ceph-mon[76322]: pgmap v176: 65 pgs: 65 active+clean; 315 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-20T21:41:33.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:33 smithi189 ceph-mon[57022]: pgmap v177: 65 pgs: 65 active+clean; 315 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-20T21:41:33.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:33 smithi189 ceph-mon[57022]: Upgrade: Updating osd.5 2024-04-20T21:41:33.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:33 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:41:33.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:33 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-20T21:41:33.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:33 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:41:33.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:33 smithi189 ceph-mon[57022]: Deploying daemon osd.5 on smithi189 2024-04-20T21:41:33.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:33 smithi062 ceph-mon[76322]: pgmap v177: 65 pgs: 65 active+clean; 315 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-20T21:41:33.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:33 smithi062 ceph-mon[76322]: Upgrade: Updating osd.5 2024-04-20T21:41:33.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:33 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:41:33.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:33 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-20T21:41:33.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:33 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:41:33.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:33 smithi062 ceph-mon[76322]: Deploying daemon osd.5 on smithi189 2024-04-20T21:41:34.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:34 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:41:34.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:34 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:41:35.642 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:35 smithi189 ceph-mon[57022]: pgmap v178: 65 pgs: 65 active+clean; 315 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:41:35.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:35 smithi062 ceph-mon[76322]: pgmap v178: 65 pgs: 65 active+clean; 315 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T21:41:37.479 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 21:41:37 smithi189 systemd[1]: Stopping Ceph osd.5 for 61ae3628-ff5b-11ee-bc93-c7b262605968... 2024-04-20T21:41:37.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:37 smithi189 ceph-mon[57022]: pgmap v179: 65 pgs: 65 active+clean; 315 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-20T21:41:37.737 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 21:41:37 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-5[36899]: 2024-04-20T21:41:37.565+0000 7f461a000700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-20T21:41:37.737 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 21:41:37 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-5[36899]: 2024-04-20T21:41:37.565+0000 7f461a000700 -1 osd.5 73 *** Got signal Terminated *** 2024-04-20T21:41:37.737 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 21:41:37 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-5[36899]: 2024-04-20T21:41:37.565+0000 7f461a000700 -1 osd.5 73 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-20T21:41:37.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:37 smithi062 ceph-mon[76322]: pgmap v179: 65 pgs: 65 active+clean; 315 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-20T21:41:38.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:38 smithi189 ceph-mon[57022]: osd.5 marked itself down and dead 2024-04-20T21:41:38.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:38 smithi062 ceph-mon[76322]: osd.5 marked itself down and dead 2024-04-20T21:41:39.464 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 21:41:39 smithi189 podman[75806]: 2024-04-20 21:41:39.193566256 +0000 UTC m=+1.773536419 container died 8e4574bae8db45649fd621947c9080072aaad6d8fb1e120de06d26f417edab68 (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-18.2.0, org.label-schema.vendor=CentOS, RELEASE=HEAD, GIT_CLEAN=True, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20231212, org.label-schema.license=GPLv2) 2024-04-20T21:41:39.737 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 21:41:39 smithi189 podman[75806]: 2024-04-20 21:41:39.600929684 +0000 UTC m=+2.180899840 container cleanup 8e4574bae8db45649fd621947c9080072aaad6d8fb1e120de06d26f417edab68 (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-5, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.0, io.buildah.version=1.29.1, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, org.label-schema.build-date=20231212, org.label-schema.license=GPLv2) 2024-04-20T21:41:39.737 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 21:41:39 smithi189 bash[75806]: ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-5 2024-04-20T21:41:39.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:39 smithi189 ceph-mon[57022]: pgmap v180: 65 pgs: 65 active+clean; 315 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-20T21:41:39.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:39 smithi189 ceph-mon[57022]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-20T21:41:39.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:39 smithi189 ceph-mon[57022]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-04-20T21:41:39.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:39 smithi189 ceph-mon[57022]: osdmap e74: 6 total, 5 up, 6 in 2024-04-20T21:41:39.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:39 smithi062 ceph-mon[76322]: pgmap v180: 65 pgs: 65 active+clean; 315 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-20T21:41:39.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:39 smithi062 ceph-mon[76322]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-20T21:41:39.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:39 smithi062 ceph-mon[76322]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-04-20T21:41:39.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:39 smithi062 ceph-mon[76322]: osdmap e74: 6 total, 5 up, 6 in 2024-04-20T21:41:40.237 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 21:41:39 smithi189 podman[75825]: 2024-04-20 21:41:39.810480713 +0000 UTC m=+0.598206977 container remove 8e4574bae8db45649fd621947c9080072aaad6d8fb1e120de06d26f417edab68 (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-5, RELEASE=HEAD, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.0, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, GIT_CLEAN=True, io.buildah.version=1.29.1, org.label-schema.build-date=20231212, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-20T21:41:40.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:40 smithi062 ceph-mon[76322]: osdmap e75: 6 total, 5 up, 6 in 2024-04-20T21:41:40.788 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:40 smithi189 ceph-mon[57022]: osdmap e75: 6 total, 5 up, 6 in 2024-04-20T21:41:40.788 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 21:41:40 smithi189 podman[75914]: 2024-04-20 21:41:40.568450454 +0000 UTC m=+0.037953821 image pull 76bc6b91b726a03b8b736eeeb35fc1ffd1469e23c7d60f5b444a217528d8906d quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 2024-04-20T21:41:40.788 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 21:41:40 smithi189 podman[75914]: 2024-04-20 21:41:40.686509675 +0000 UTC m=+0.156013079 container create 66921bdd1df3034b159d6d4ed80059043d977eadb1aed8401e65bb0509cbca07 (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-5-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416) 2024-04-20T21:41:41.079 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 21:41:40 smithi189 podman[75914]: 2024-04-20 21:41:40.972965099 +0000 UTC m=+0.442468476 container init 66921bdd1df3034b159d6d4ed80059043d977eadb1aed8401e65bb0509cbca07 (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-5-deactivate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5) 2024-04-20T21:41:41.080 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 21:41:40 smithi189 podman[75914]: 2024-04-20 21:41:40.984450477 +0000 UTC m=+0.453953848 container start 66921bdd1df3034b159d6d4ed80059043d977eadb1aed8401e65bb0509cbca07 (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-5-deactivate, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux ) 2024-04-20T21:41:41.080 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 21:41:41 smithi189 podman[75914]: 2024-04-20 21:41:41.080627122 +0000 UTC m=+0.550130583 container attach 66921bdd1df3034b159d6d4ed80059043d977eadb1aed8401e65bb0509cbca07 (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-5-deactivate, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS) 2024-04-20T21:41:41.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:41 smithi189 ceph-mon[57022]: pgmap v183: 65 pgs: 3 active+undersized+degraded, 3 peering, 8 stale+active+clean, 4 active+undersized, 47 active+clean; 315 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 1 op/s; 8/366 objects degraded (2.186%) 2024-04-20T21:41:41.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:41 smithi189 ceph-mon[57022]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-20T21:41:41.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:41 smithi189 ceph-mon[57022]: Health check failed: Degraded data redundancy: 8/366 objects degraded (2.186%), 3 pgs degraded (PG_DEGRADED) 2024-04-20T21:41:41.737 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 21:41:41 smithi189 podman[75914]: 2024-04-20 21:41:41.439870208 +0000 UTC m=+0.909373639 container died 66921bdd1df3034b159d6d4ed80059043d977eadb1aed8401e65bb0509cbca07 (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-5-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-04-20T21:41:41.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:41 smithi062 ceph-mon[76322]: pgmap v183: 65 pgs: 3 active+undersized+degraded, 3 peering, 8 stale+active+clean, 4 active+undersized, 47 active+clean; 315 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 1 op/s; 8/366 objects degraded (2.186%) 2024-04-20T21:41:41.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:41 smithi062 ceph-mon[76322]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-20T21:41:41.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:41 smithi062 ceph-mon[76322]: Health check failed: Degraded data redundancy: 8/366 objects degraded (2.186%), 3 pgs degraded (PG_DEGRADED) 2024-04-20T21:41:42.488 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 21:41:42 smithi189 podman[75940]: 2024-04-20 21:41:42.100834357 +0000 UTC m=+0.640969798 container remove 66921bdd1df3034b159d6d4ed80059043d977eadb1aed8401e65bb0509cbca07 (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-20T21:41:42.488 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 21:41:42 smithi189 systemd[1]: ceph-61ae3628-ff5b-11ee-bc93-c7b262605968@osd.5.service: Deactivated successfully. 2024-04-20T21:41:42.488 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 21:41:42 smithi189 systemd[1]: Stopped Ceph osd.5 for 61ae3628-ff5b-11ee-bc93-c7b262605968. 2024-04-20T21:41:42.488 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 21:41:42 smithi189 systemd[1]: ceph-61ae3628-ff5b-11ee-bc93-c7b262605968@osd.5.service: Consumed 1min 48.575s CPU time. 2024-04-20T21:41:42.927 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 21:41:42 smithi189 systemd[1]: Starting Ceph osd.5 for 61ae3628-ff5b-11ee-bc93-c7b262605968... 2024-04-20T21:41:43.237 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 21:41:42 smithi189 podman[76045]: 2024-04-20 21:41:42.832865381 +0000 UTC m=+0.032972868 image pull 76bc6b91b726a03b8b736eeeb35fc1ffd1469e23c7d60f5b444a217528d8906d quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 2024-04-20T21:41:43.237 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 21:41:42 smithi189 podman[76045]: 2024-04-20 21:41:42.952428145 +0000 UTC m=+0.152535676 container create 35d228032ecfd532db59e1f237b46f07dd6e1b451e0caaa5696b676f643a3bb4 (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-5-activate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-04-20T21:41:43.522 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 21:41:43 smithi189 podman[76045]: 2024-04-20 21:41:43.267110359 +0000 UTC m=+0.467217890 container init 35d228032ecfd532db59e1f237b46f07dd6e1b451e0caaa5696b676f643a3bb4 (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-5-activate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True) 2024-04-20T21:41:43.522 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 21:41:43 smithi189 podman[76045]: 2024-04-20 21:41:43.279101385 +0000 UTC m=+0.479208932 container start 35d228032ecfd532db59e1f237b46f07dd6e1b451e0caaa5696b676f643a3bb4 (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-5-activate, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-04-20T21:41:43.522 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 21:41:43 smithi189 podman[76045]: 2024-04-20 21:41:43.363052385 +0000 UTC m=+0.563159891 container attach 35d228032ecfd532db59e1f237b46f07dd6e1b451e0caaa5696b676f643a3bb4 (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-5-activate, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5) 2024-04-20T21:41:43.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:43 smithi062 ceph-mon[76322]: pgmap v184: 65 pgs: 9 active+undersized+degraded, 3 peering, 3 stale+active+clean, 11 active+undersized, 39 active+clean; 315 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 40/366 objects degraded (10.929%) 2024-04-20T21:41:43.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:43 smithi189 ceph-mon[57022]: pgmap v184: 65 pgs: 9 active+undersized+degraded, 3 peering, 3 stale+active+clean, 11 active+undersized, 39 active+clean; 315 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 40/366 objects degraded (10.929%) 2024-04-20T21:41:44.488 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 21:41:44 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-5-activate[76060]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-20T21:41:44.488 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 21:41:44 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-5-activate[76060]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-20T21:41:44.489 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 21:41:44 smithi189 bash[76045]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-20T21:41:44.489 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 21:41:44 smithi189 bash[76045]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-20T21:41:44.489 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 21:41:44 smithi189 bash[76045]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-20T21:41:44.489 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 21:41:44 smithi189 bash[76045]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-04-20T21:41:44.489 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 21:41:44 smithi189 bash[76045]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-04-20T21:41:44.489 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 21:41:44 smithi189 bash[76045]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-04-20T21:41:44.489 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 21:41:44 smithi189 bash[76045]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5/block 2024-04-20T21:41:44.489 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 21:41:44 smithi189 bash[76045]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-20T21:41:44.489 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 21:41:44 smithi189 bash[76045]: --> ceph-volume raw activate successful for osd ID: 5 2024-04-20T21:41:44.489 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 21:41:44 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-5-activate[76060]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-20T21:41:44.489 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 21:41:44 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-5-activate[76060]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-04-20T21:41:44.489 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 21:41:44 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-5-activate[76060]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-04-20T21:41:44.489 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 21:41:44 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-5-activate[76060]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-04-20T21:41:44.489 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 21:41:44 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-5-activate[76060]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5/block 2024-04-20T21:41:44.490 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 21:41:44 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-5-activate[76060]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-20T21:41:44.490 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 21:41:44 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-5-activate[76060]: --> ceph-volume raw activate successful for osd ID: 5 2024-04-20T21:41:44.490 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 21:41:44 smithi189 podman[76045]: 2024-04-20 21:41:44.28443991 +0000 UTC m=+1.484547440 container died 35d228032ecfd532db59e1f237b46f07dd6e1b451e0caaa5696b676f643a3bb4 (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-5-activate, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0) 2024-04-20T21:41:44.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:44 smithi189 ceph-mon[57022]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-20T21:41:45.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:44 smithi062 ceph-mon[76322]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-20T21:41:45.487 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 21:41:45 smithi189 podman[76200]: 2024-04-20 21:41:45.142397187 +0000 UTC m=+0.837703477 container remove 35d228032ecfd532db59e1f237b46f07dd6e1b451e0caaa5696b676f643a3bb4 (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-5-activate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, ceph=True) 2024-04-20T21:41:45.820 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:45 smithi189 ceph-mon[57022]: pgmap v185: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 315 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 56/366 objects degraded (15.301%) 2024-04-20T21:41:45.821 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 21:41:45 smithi189 podman[76260]: 2024-04-20 21:41:45.42390698 +0000 UTC m=+0.035035133 image pull 76bc6b91b726a03b8b736eeeb35fc1ffd1469e23c7d60f5b444a217528d8906d quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 2024-04-20T21:41:45.821 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 21:41:45 smithi189 podman[76260]: 2024-04-20 21:41:45.526584059 +0000 UTC m=+0.137712248 container create 596dded14c124b91e080c03569bfae21c361718e40c11237aac1d50c29eb3166 (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5) 2024-04-20T21:41:45.821 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 21:41:45 smithi189 podman[76260]: 2024-04-20 21:41:45.822101123 +0000 UTC m=+0.433229319 container init 596dded14c124b91e080c03569bfae21c361718e40c11237aac1d50c29eb3166 (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, ceph=True, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5) 2024-04-20T21:41:46.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:45 smithi062 ceph-mon[76322]: pgmap v185: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 315 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 56/366 objects degraded (15.301%) 2024-04-20T21:41:46.237 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 21:41:45 smithi189 podman[76260]: 2024-04-20 21:41:45.833611467 +0000 UTC m=+0.444739638 container start 596dded14c124b91e080c03569bfae21c361718e40c11237aac1d50c29eb3166 (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True) 2024-04-20T21:41:46.237 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 21:41:45 smithi189 ceph-osd[76279]: -- 172.21.15.189:0/489920603 <== mon.1 v2:172.21.15.189:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a141a7f680 con 0x55a141a49000 2024-04-20T21:41:46.237 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 21:41:45 smithi189 bash[76260]: 596dded14c124b91e080c03569bfae21c361718e40c11237aac1d50c29eb3166 2024-04-20T21:41:46.237 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 21:41:45 smithi189 systemd[1]: Started Ceph osd.5 for 61ae3628-ff5b-11ee-bc93-c7b262605968. 2024-04-20T21:41:46.736 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 21:41:46 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-5[76275]: 2024-04-20T21:41:46.410+0000 7f31780fc7c0 -1 Falling back to public interface 2024-04-20T21:41:47.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:47.170+0000 7f34ffa00700 1 -- 172.21.15.62:0/3058973188 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f34f8106b80 msgr2=0x7f34f8106ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:41:47.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:47.170+0000 7f34ffa00700 1 --2- 172.21.15.62:0/3058973188 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f34f8106b80 0x7f34f8106ff0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f34e4009910 tx=0x7f34e4009c20 comp rx=0 tx=0).stop 2024-04-20T21:41:47.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:47.170+0000 7f34ffa00700 1 -- 172.21.15.62:0/3058973188 shutdown_connections 2024-04-20T21:41:47.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:47.170+0000 7f34ffa00700 1 --2- 172.21.15.62:0/3058973188 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f34f8106b80 0x7f34f8106ff0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:41:47.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:47.170+0000 7f34ffa00700 1 --2- 172.21.15.62:0/3058973188 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f34f810c8f0 0x7f34f810ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:41:47.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:47.170+0000 7f34ffa00700 1 -- 172.21.15.62:0/3058973188 >> 172.21.15.62:0/3058973188 conn(0x7f34f8076060 msgr2=0x7f34f8078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:41:47.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:47.170+0000 7f34ffa00700 1 -- 172.21.15.62:0/3058973188 shutdown_connections 2024-04-20T21:41:47.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:47.170+0000 7f34ffa00700 1 -- 172.21.15.62:0/3058973188 wait complete. 2024-04-20T21:41:47.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:47.171+0000 7f34ffa00700 1 Processor -- start 2024-04-20T21:41:47.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:47.172+0000 7f34ffa00700 1 -- start start 2024-04-20T21:41:47.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:47.172+0000 7f34ffa00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f34f8106b80 0x7f34f819c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:41:47.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:47.172+0000 7f34ffa00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f34f810c8f0 0x7f34f819c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:41:47.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:47.172+0000 7f34ffa00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34f819d060 con 0x7f34f810c8f0 2024-04-20T21:41:47.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:47.172+0000 7f34ffa00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34f81a0df0 con 0x7f34f8106b80 2024-04-20T21:41:47.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:47.172+0000 7f34f7e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f34f810c8f0 0x7f34f819c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:41:47.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:47.172+0000 7f34f7e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f34f810c8f0 0x7f34f819c980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59470/0 (socket says 172.21.15.62:59470) 2024-04-20T21:41:47.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:47.173+0000 7f34f7e00700 1 -- 172.21.15.62:0/461757947 learned_addr learned my addr 172.21.15.62:0/461757947 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:41:47.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:47.173+0000 7f34fd000700 1 --2- 172.21.15.62:0/461757947 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f34f8106b80 0x7f34f819c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:41:47.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:47.173+0000 7f34f7e00700 1 -- 172.21.15.62:0/461757947 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f34f8106b80 msgr2=0x7f34f819c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:41:47.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:47.173+0000 7f34f7e00700 1 --2- 172.21.15.62:0/461757947 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f34f8106b80 0x7f34f819c440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:41:47.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:47.173+0000 7f34f7e00700 1 -- 172.21.15.62:0/461757947 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34e40095f0 con 0x7f34f810c8f0 2024-04-20T21:41:47.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:47.173+0000 7f34f7e00700 1 --2- 172.21.15.62:0/461757947 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f34f810c8f0 0x7f34f819c980 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f34e400b5c0 tx=0x7f34e400bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:41:47.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:47.174+0000 7f34f5600700 1 -- 172.21.15.62:0/461757947 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f34e401d070 con 0x7f34f810c8f0 2024-04-20T21:41:47.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:47.174+0000 7f34f5600700 1 -- 172.21.15.62:0/461757947 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f34e400f460 con 0x7f34f810c8f0 2024-04-20T21:41:47.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:47.174+0000 7f34f5600700 1 -- 172.21.15.62:0/461757947 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f34e4017630 con 0x7f34f810c8f0 2024-04-20T21:41:47.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:47.174+0000 7f34ffa00700 1 -- 172.21.15.62:0/461757947 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34f81a1070 con 0x7f34f810c8f0 2024-04-20T21:41:47.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:47.174+0000 7f34ffa00700 1 -- 172.21.15.62:0/461757947 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34f81a1560 con 0x7f34f810c8f0 2024-04-20T21:41:47.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:47.175+0000 7f34ffa00700 1 -- 172.21.15.62:0/461757947 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34f810ecc0 con 0x7f34f810c8f0 2024-04-20T21:41:47.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:47.176+0000 7f34f5600700 1 -- 172.21.15.62:0/461757947 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f34e400f5d0 con 0x7f34f810c8f0 2024-04-20T21:41:47.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:47.178+0000 7f34f5600700 1 --2- 172.21.15.62:0/461757947 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f34e006ecc0 0x7f34e0071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:41:47.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:47.179+0000 7f34f5600700 1 -- 172.21.15.62:0/461757947 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f34e40902b0 con 0x7f34f810c8f0 2024-04-20T21:41:47.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:47.179+0000 7f34fd000700 1 --2- 172.21.15.62:0/461757947 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f34e006ecc0 0x7f34e0071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:41:47.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:47.180+0000 7f34fd000700 1 --2- 172.21.15.62:0/461757947 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f34e006ecc0 0x7f34e0071170 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f34ec005950 tx=0x7f34ec00a300 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:41:47.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:47.186+0000 7f34f5600700 1 -- 172.21.15.62:0/461757947 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f34e40590f0 con 0x7f34f810c8f0 2024-04-20T21:41:47.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:47.444+0000 7f34ffa00700 1 -- 172.21.15.62:0/461757947 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f34f802cca0 con 0x7f34e006ecc0 2024-04-20T21:41:47.444 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:47 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:41:47.444 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:47 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:41:47.444 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:47 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:41:47.444 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:47 smithi062 ceph-mon[76322]: pgmap v186: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 315 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 56/366 objects degraded (15.301%) 2024-04-20T21:41:47.444 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:47 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:41:47.444 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:47 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:41:47.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:47.448+0000 7f34f5600700 1 -- 172.21.15.62:0/461757947 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7f34f802cca0 con 0x7f34e006ecc0 2024-04-20T21:41:47.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:47.452+0000 7f34ffa00700 1 -- 172.21.15.62:0/461757947 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f34e006ecc0 msgr2=0x7f34e0071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:41:47.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:47.452+0000 7f34ffa00700 1 --2- 172.21.15.62:0/461757947 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f34e006ecc0 0x7f34e0071170 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f34ec005950 tx=0x7f34ec00a300 comp rx=0 tx=0).stop 2024-04-20T21:41:47.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:47.452+0000 7f34ffa00700 1 -- 172.21.15.62:0/461757947 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f34f810c8f0 msgr2=0x7f34f819c980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:41:47.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:47.452+0000 7f34ffa00700 1 --2- 172.21.15.62:0/461757947 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f34f810c8f0 0x7f34f819c980 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f34e400b5c0 tx=0x7f34e400bd50 comp rx=0 tx=0).stop 2024-04-20T21:41:47.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:47.452+0000 7f34ffa00700 1 -- 172.21.15.62:0/461757947 shutdown_connections 2024-04-20T21:41:47.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:47.452+0000 7f34ffa00700 1 --2- 172.21.15.62:0/461757947 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f34e006ecc0 0x7f34e0071170 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:41:47.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:47.452+0000 7f34ffa00700 1 --2- 172.21.15.62:0/461757947 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f34f8106b80 0x7f34f819c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:41:47.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:47.452+0000 7f34ffa00700 1 --2- 172.21.15.62:0/461757947 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f34f810c8f0 0x7f34f819c980 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:41:47.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:47.452+0000 7f34ffa00700 1 -- 172.21.15.62:0/461757947 >> 172.21.15.62:0/461757947 conn(0x7f34f8076060 msgr2=0x7f34f8077ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:41:47.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:47.452+0000 7f34ffa00700 1 -- 172.21.15.62:0/461757947 shutdown_connections 2024-04-20T21:41:47.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:47.452+0000 7f34ffa00700 1 -- 172.21.15.62:0/461757947 wait complete. 2024-04-20T21:41:47.469 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T21:41:47.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:47 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:41:47.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:47 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:41:47.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:47 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:41:47.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:47 smithi189 ceph-mon[57022]: pgmap v186: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 315 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 56/366 objects degraded (15.301%) 2024-04-20T21:41:47.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:47 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:41:47.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:47 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:41:47.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:47.623+0000 7f410ce00700 1 -- 172.21.15.62:0/3667106527 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4108102470 msgr2=0x7f41081028e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:41:47.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:47.623+0000 7f410ce00700 1 --2- 172.21.15.62:0/3667106527 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4108102470 0x7f41081028e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f40f4009910 tx=0x7f40f4009c20 comp rx=0 tx=0).stop 2024-04-20T21:41:47.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:47.623+0000 7f410ce00700 1 -- 172.21.15.62:0/3667106527 shutdown_connections 2024-04-20T21:41:47.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:47.623+0000 7f410ce00700 1 --2- 172.21.15.62:0/3667106527 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4108102470 0x7f41081028e0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:41:47.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:47.623+0000 7f410ce00700 1 --2- 172.21.15.62:0/3667106527 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4108108470 0x7f4108108840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:41:47.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:47.623+0000 7f410ce00700 1 -- 172.21.15.62:0/3667106527 >> 172.21.15.62:0/3667106527 conn(0x7f41080fdf90 msgr2=0x7f41081003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:41:47.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:47.623+0000 7f410ce00700 1 -- 172.21.15.62:0/3667106527 shutdown_connections 2024-04-20T21:41:47.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:47.623+0000 7f410ce00700 1 -- 172.21.15.62:0/3667106527 wait complete. 2024-04-20T21:41:47.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:47.624+0000 7f410ce00700 1 Processor -- start 2024-04-20T21:41:47.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:47.624+0000 7f410ce00700 1 -- start start 2024-04-20T21:41:47.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:47.625+0000 7f410ce00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4108102470 0x7f4108198230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:41:47.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:47.625+0000 7f410ce00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4108108470 0x7f4108198770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:41:47.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:47.625+0000 7f410ce00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4108198e50 con 0x7f4108102470 2024-04-20T21:41:47.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:47.625+0000 7f410ce00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f410819cbe0 con 0x7f4108108470 2024-04-20T21:41:47.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:47.625+0000 7f4105e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4108102470 0x7f4108198230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:41:47.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:47.625+0000 7f4105e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4108102470 0x7f4108198230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59490/0 (socket says 172.21.15.62:59490) 2024-04-20T21:41:47.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:47.625+0000 7f4105e00700 1 -- 172.21.15.62:0/1696684853 learned_addr learned my addr 172.21.15.62:0/1696684853 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:41:47.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:47.625+0000 7f4105400700 1 --2- 172.21.15.62:0/1696684853 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4108108470 0x7f4108198770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:41:47.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:47.626+0000 7f4105e00700 1 -- 172.21.15.62:0/1696684853 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4108108470 msgr2=0x7f4108198770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:41:47.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:47.626+0000 7f4105e00700 1 --2- 172.21.15.62:0/1696684853 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4108108470 0x7f4108198770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:41:47.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:47.626+0000 7f4105e00700 1 -- 172.21.15.62:0/1696684853 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40f40095f0 con 0x7f4108102470 2024-04-20T21:41:47.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:47.626+0000 7f4105e00700 1 --2- 172.21.15.62:0/1696684853 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4108102470 0x7f4108198230 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f40f800e920 tx=0x7f40f800ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:41:47.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:47.626+0000 7f40fea00700 1 -- 172.21.15.62:0/1696684853 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f40f800c9d0 con 0x7f4108102470 2024-04-20T21:41:47.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:47.626+0000 7f410ce00700 1 -- 172.21.15.62:0/1696684853 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f410819cec0 con 0x7f4108102470 2024-04-20T21:41:47.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:47.626+0000 7f40fea00700 1 -- 172.21.15.62:0/1696684853 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f40f8010430 con 0x7f4108102470 2024-04-20T21:41:47.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:47.626+0000 7f40fea00700 1 -- 172.21.15.62:0/1696684853 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f40f8018630 con 0x7f4108102470 2024-04-20T21:41:47.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:47.627+0000 7f410ce00700 1 -- 172.21.15.62:0/1696684853 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f410819d410 con 0x7f4108102470 2024-04-20T21:41:47.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:47.628+0000 7f410ce00700 1 -- 172.21.15.62:0/1696684853 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f410810a970 con 0x7f4108102470 2024-04-20T21:41:47.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:47.629+0000 7f40fea00700 1 -- 172.21.15.62:0/1696684853 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f40f80105a0 con 0x7f4108102470 2024-04-20T21:41:47.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:47.630+0000 7f40fea00700 1 --2- 172.21.15.62:0/1696684853 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f40ec06ecc0 0x7f40ec071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:41:47.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:47.630+0000 7f40fea00700 1 -- 172.21.15.62:0/1696684853 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f40f8014070 con 0x7f4108102470 2024-04-20T21:41:47.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:47.630+0000 7f4105400700 1 --2- 172.21.15.62:0/1696684853 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f40ec06ecc0 0x7f40ec071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:41:47.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:47.631+0000 7f4105400700 1 --2- 172.21.15.62:0/1696684853 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f40ec06ecc0 0x7f40ec071170 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f4108199850 tx=0x7f40f4005a90 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:41:47.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:47.635+0000 7f40fea00700 1 -- 172.21.15.62:0/1696684853 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f40f8053ff0 con 0x7f4108102470 2024-04-20T21:41:47.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:47.892+0000 7f410ce00700 1 -- 172.21.15.62:0/1696684853 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4108199590 con 0x7f40ec06ecc0 2024-04-20T21:41:47.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:47.895+0000 7f40fea00700 1 -- 172.21.15.62:0/1696684853 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7f4108199590 con 0x7f40ec06ecc0 2024-04-20T21:41:47.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:47.898+0000 7f410ce00700 1 -- 172.21.15.62:0/1696684853 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f40ec06ecc0 msgr2=0x7f40ec071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:41:47.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:47.898+0000 7f410ce00700 1 --2- 172.21.15.62:0/1696684853 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f40ec06ecc0 0x7f40ec071170 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f4108199850 tx=0x7f40f4005a90 comp rx=0 tx=0).stop 2024-04-20T21:41:47.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:47.898+0000 7f410ce00700 1 -- 172.21.15.62:0/1696684853 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4108102470 msgr2=0x7f4108198230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:41:47.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:47.898+0000 7f410ce00700 1 --2- 172.21.15.62:0/1696684853 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4108102470 0x7f4108198230 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f40f800e920 tx=0x7f40f800ec30 comp rx=0 tx=0).stop 2024-04-20T21:41:47.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:47.898+0000 7f410ce00700 1 -- 172.21.15.62:0/1696684853 shutdown_connections 2024-04-20T21:41:47.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:47.898+0000 7f410ce00700 1 --2- 172.21.15.62:0/1696684853 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f40ec06ecc0 0x7f40ec071170 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:41:47.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:47.898+0000 7f410ce00700 1 --2- 172.21.15.62:0/1696684853 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4108102470 0x7f4108198230 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:41:47.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:47.898+0000 7f410ce00700 1 --2- 172.21.15.62:0/1696684853 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4108108470 0x7f4108198770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:41:47.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:47.898+0000 7f410ce00700 1 -- 172.21.15.62:0/1696684853 >> 172.21.15.62:0/1696684853 conn(0x7f41080fdf90 msgr2=0x7f41080ff320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:41:47.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:47.898+0000 7f410ce00700 1 -- 172.21.15.62:0/1696684853 shutdown_connections 2024-04-20T21:41:47.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:47.898+0000 7f410ce00700 1 -- 172.21.15.62:0/1696684853 wait complete. 2024-04-20T21:41:48.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:48.070+0000 7f10cca00700 1 -- 172.21.15.62:0/1168377075 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f10c8068540 msgr2=0x7f10c80689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:41:48.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:48.070+0000 7f10cca00700 1 --2- 172.21.15.62:0/1168377075 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f10c8068540 0x7f10c80689b0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f10b4009910 tx=0x7f10b4009c20 comp rx=0 tx=0).stop 2024-04-20T21:41:48.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:48.070+0000 7f10cca00700 1 -- 172.21.15.62:0/1168377075 shutdown_connections 2024-04-20T21:41:48.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:48.070+0000 7f10cca00700 1 --2- 172.21.15.62:0/1168377075 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f10c8068540 0x7f10c80689b0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:41:48.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:48.070+0000 7f10cca00700 1 --2- 172.21.15.62:0/1168377075 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10c8101000 0x7f10c81013d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:41:48.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:48.070+0000 7f10cca00700 1 -- 172.21.15.62:0/1168377075 >> 172.21.15.62:0/1168377075 conn(0x7f10c8075130 msgr2=0x7f10c8075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:41:48.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:48.071+0000 7f10cca00700 1 -- 172.21.15.62:0/1168377075 shutdown_connections 2024-04-20T21:41:48.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:48.071+0000 7f10cca00700 1 -- 172.21.15.62:0/1168377075 wait complete. 2024-04-20T21:41:48.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:48.072+0000 7f10cca00700 1 Processor -- start 2024-04-20T21:41:48.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:48.073+0000 7f10cca00700 1 -- start start 2024-04-20T21:41:48.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:48.073+0000 7f10cca00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10c8068540 0x7f10c8198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:41:48.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:48.073+0000 7f10cca00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f10c8101000 0x7f10c81985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:41:48.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:48.073+0000 7f10cca00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10c8198c90 con 0x7f10c8101000 2024-04-20T21:41:48.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:48.073+0000 7f10cca00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10c819ca20 con 0x7f10c8068540 2024-04-20T21:41:48.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:48.074+0000 7f10c5400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f10c8101000 0x7f10c81985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:41:48.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:48.074+0000 7f10c5400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f10c8101000 0x7f10c81985b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59510/0 (socket says 172.21.15.62:59510) 2024-04-20T21:41:48.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:48.074+0000 7f10c5400700 1 -- 172.21.15.62:0/859460766 learned_addr learned my addr 172.21.15.62:0/859460766 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:41:48.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:48.074+0000 7f10c5e00700 1 --2- 172.21.15.62:0/859460766 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10c8068540 0x7f10c8198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:41:48.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:48.074+0000 7f10c5400700 1 -- 172.21.15.62:0/859460766 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10c8068540 msgr2=0x7f10c8198070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:41:48.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:48.074+0000 7f10c5400700 1 --2- 172.21.15.62:0/859460766 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10c8068540 0x7f10c8198070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:41:48.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:48.074+0000 7f10c5400700 1 -- 172.21.15.62:0/859460766 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10b40095f0 con 0x7f10c8101000 2024-04-20T21:41:48.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:48.075+0000 7f10c5400700 1 --2- 172.21.15.62:0/859460766 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f10c8101000 0x7f10c81985b0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f10b400b5c0 tx=0x7f10b400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:41:48.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:48.076+0000 7f10bea00700 1 -- 172.21.15.62:0/859460766 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f10b401d070 con 0x7f10c8101000 2024-04-20T21:41:48.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:48.076+0000 7f10bea00700 1 -- 172.21.15.62:0/859460766 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f10b4005e00 con 0x7f10c8101000 2024-04-20T21:41:48.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:48.076+0000 7f10cca00700 1 -- 172.21.15.62:0/859460766 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10c819cca0 con 0x7f10c8101000 2024-04-20T21:41:48.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:48.076+0000 7f10bea00700 1 -- 172.21.15.62:0/859460766 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f10b4017400 con 0x7f10c8101000 2024-04-20T21:41:48.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:48.076+0000 7f10cca00700 1 -- 172.21.15.62:0/859460766 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10c819d190 con 0x7f10c8101000 2024-04-20T21:41:48.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:48.078+0000 7f10cca00700 1 -- 172.21.15.62:0/859460766 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10c804f2a0 con 0x7f10c8101000 2024-04-20T21:41:48.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:48.079+0000 7f10bea00700 1 -- 172.21.15.62:0/859460766 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f10b4003a60 con 0x7f10c8101000 2024-04-20T21:41:48.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:48.080+0000 7f10bea00700 1 --2- 172.21.15.62:0/859460766 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f10ac06ecc0 0x7f10ac071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:41:48.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:48.080+0000 7f10bea00700 1 -- 172.21.15.62:0/859460766 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f10b4090210 con 0x7f10c8101000 2024-04-20T21:41:48.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:48.080+0000 7f10c5e00700 1 --2- 172.21.15.62:0/859460766 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f10ac06ecc0 0x7f10ac071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:41:48.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:48.081+0000 7f10c5e00700 1 --2- 172.21.15.62:0/859460766 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f10ac06ecc0 0x7f10ac071170 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f10b8007be0 tx=0x7f10b80073d0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:41:48.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:48.085+0000 7f10bea00700 1 -- 172.21.15.62:0/859460766 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f10b4059100 con 0x7f10c8101000 2024-04-20T21:41:48.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:48 smithi062 ceph-mon[76322]: from='client.34398 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:41:48.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:48 smithi062 ceph-mon[76322]: from='client.34402 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:41:48.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:48.337+0000 7f10cca00700 1 -- 172.21.15.62:0/859460766 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f10c819d4c0 con 0x7f10ac06ecc0 2024-04-20T21:41:48.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:48.370+0000 7f10bea00700 1 -- 172.21.15.62:0/859460766 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f10c819d4c0 con 0x7f10ac06ecc0 2024-04-20T21:41:48.370 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T21:41:48.370 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (7m) 112s ago 21m 14.2M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T21:41:48.370 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (21m) 112s ago 21m 8992k - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T21:41:48.370 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (20m) 40s ago 20m 8707k - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T21:41:48.370 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (3m) 112s ago 21m 6660k - 19.0.0-3133-ga8c84f59 76bc6b91b726 2825d5ba11c6 2024-04-20T21:41:48.370 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (3m) 40s ago 20m 6665k - 19.0.0-3133-ga8c84f59 76bc6b91b726 20f746fa1bba 2024-04-20T21:41:48.370 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (7m) 112s ago 21m 75.0M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T21:41:48.370 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (14m) 112s ago 14m 65.5M - 18.2.0 dc2bc1663786 5e6460aa6159 2024-04-20T21:41:48.370 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (14m) 112s ago 14m 131M - 18.2.0 dc2bc1663786 e65f2267544e 2024-04-20T21:41:48.371 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (14m) 40s ago 14m 117M - 18.2.0 dc2bc1663786 fb2a8a223db8 2024-04-20T21:41:48.371 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (14m) 40s ago 14m 55.6M - 18.2.0 dc2bc1663786 f2eec2ee8951 2024-04-20T21:41:48.371 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (10m) 112s ago 23m 504M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T21:41:48.371 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (9m) 40s ago 20m 451M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T21:41:48.371 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (5m) 112s ago 23m 49.3M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 019eb1479485 2024-04-20T21:41:48.371 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (4m) 40s ago 20m 38.7M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 55aac560aab6 2024-04-20T21:41:48.371 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (8m) 112s ago 21m 7906k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T21:41:48.371 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (8m) 40s ago 20m 8136k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T21:41:48.371 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (3m) 112s ago 19m 83.0M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 ee2bb901f26e 2024-04-20T21:41:48.371 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (2m) 112s ago 18m 87.9M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 9075152d4162 2024-04-20T21:41:48.371 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (2m) 112s ago 17m 12.7M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 2d4f3e38dc0e 2024-04-20T21:41:48.371 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (88s) 40s ago 17m 85.7M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 8f97b17ae0eb 2024-04-20T21:41:48.371 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (45s) 40s ago 16m 11.1M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 653d8079967d 2024-04-20T21:41:48.371 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 starting - - - 2240M 2024-04-20T21:41:48.371 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (8m) 112s ago 20m 73.9M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T21:41:48.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:48.373+0000 7f10cca00700 1 -- 172.21.15.62:0/859460766 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f10ac06ecc0 msgr2=0x7f10ac071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:41:48.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:48.373+0000 7f10cca00700 1 --2- 172.21.15.62:0/859460766 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f10ac06ecc0 0x7f10ac071170 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f10b8007be0 tx=0x7f10b80073d0 comp rx=0 tx=0).stop 2024-04-20T21:41:48.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:48.373+0000 7f10cca00700 1 -- 172.21.15.62:0/859460766 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f10c8101000 msgr2=0x7f10c81985b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:41:48.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:48.373+0000 7f10cca00700 1 --2- 172.21.15.62:0/859460766 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f10c8101000 0x7f10c81985b0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f10b400b5c0 tx=0x7f10b400bd70 comp rx=0 tx=0).stop 2024-04-20T21:41:48.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:48.374+0000 7f10cca00700 1 -- 172.21.15.62:0/859460766 shutdown_connections 2024-04-20T21:41:48.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:48.374+0000 7f10cca00700 1 --2- 172.21.15.62:0/859460766 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f10ac06ecc0 0x7f10ac071170 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:41:48.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:48.374+0000 7f10cca00700 1 --2- 172.21.15.62:0/859460766 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10c8068540 0x7f10c8198070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:41:48.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:48.374+0000 7f10cca00700 1 --2- 172.21.15.62:0/859460766 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f10c8101000 0x7f10c81985b0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:41:48.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:48.374+0000 7f10cca00700 1 -- 172.21.15.62:0/859460766 >> 172.21.15.62:0/859460766 conn(0x7f10c8075130 msgr2=0x7f10c80fdb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:41:48.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:48.374+0000 7f10cca00700 1 -- 172.21.15.62:0/859460766 shutdown_connections 2024-04-20T21:41:48.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:48.374+0000 7f10cca00700 1 -- 172.21.15.62:0/859460766 wait complete. 2024-04-20T21:41:48.443 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:48 smithi189 ceph-mon[57022]: from='client.34398 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:41:48.443 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:48 smithi189 ceph-mon[57022]: from='client.34402 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:41:48.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:48.544+0000 7f139c000700 1 -- 172.21.15.62:0/2314831508 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1394068540 msgr2=0x7f13940689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:41:48.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:48.544+0000 7f139c000700 1 --2- 172.21.15.62:0/2314831508 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1394068540 0x7f13940689b0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f1380009910 tx=0x7f1380009c20 comp rx=0 tx=0).stop 2024-04-20T21:41:48.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:48.544+0000 7f139c000700 1 -- 172.21.15.62:0/2314831508 shutdown_connections 2024-04-20T21:41:48.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:48.544+0000 7f139c000700 1 --2- 172.21.15.62:0/2314831508 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1394068540 0x7f13940689b0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:41:48.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:48.544+0000 7f139c000700 1 --2- 172.21.15.62:0/2314831508 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1394106320 0x7f13941066f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:41:48.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:48.544+0000 7f139c000700 1 -- 172.21.15.62:0/2314831508 >> 172.21.15.62:0/2314831508 conn(0x7f1394075130 msgr2=0x7f1394075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:41:48.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:48.545+0000 7f139c000700 1 -- 172.21.15.62:0/2314831508 shutdown_connections 2024-04-20T21:41:48.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:48.545+0000 7f139c000700 1 -- 172.21.15.62:0/2314831508 wait complete. 2024-04-20T21:41:48.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:48.546+0000 7f139c000700 1 Processor -- start 2024-04-20T21:41:48.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:48.546+0000 7f139c000700 1 -- start start 2024-04-20T21:41:48.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:48.547+0000 7f139c000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1394068540 0x7f1394198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:41:48.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:48.547+0000 7f139c000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1394106320 0x7f1394198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:41:48.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:48.547+0000 7f139c000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1394198c40 con 0x7f1394068540 2024-04-20T21:41:48.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:48.547+0000 7f139c000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f139419c9d0 con 0x7f1394106320 2024-04-20T21:41:48.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:48.547+0000 7f1399600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1394068540 0x7f1394198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:41:48.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:48.547+0000 7f1399600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1394068540 0x7f1394198020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59536/0 (socket says 172.21.15.62:59536) 2024-04-20T21:41:48.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:48.547+0000 7f1399600700 1 -- 172.21.15.62:0/4119612361 learned_addr learned my addr 172.21.15.62:0/4119612361 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:41:48.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:48.547+0000 7f1398c00700 1 --2- 172.21.15.62:0/4119612361 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1394106320 0x7f1394198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:41:48.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:48.547+0000 7f1399600700 1 -- 172.21.15.62:0/4119612361 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1394106320 msgr2=0x7f1394198560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:41:48.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:48.547+0000 7f1399600700 1 --2- 172.21.15.62:0/4119612361 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1394106320 0x7f1394198560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:41:48.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:48.547+0000 7f1399600700 1 -- 172.21.15.62:0/4119612361 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13800095f0 con 0x7f1394068540 2024-04-20T21:41:48.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:48.547+0000 7f1398c00700 1 --2- 172.21.15.62:0/4119612361 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1394106320 0x7f1394198560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-20T21:41:48.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:48.548+0000 7f1399600700 1 --2- 172.21.15.62:0/4119612361 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1394068540 0x7f1394198020 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f138800d900 tx=0x7f138800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:41:48.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:48.548+0000 7f1392000700 1 -- 172.21.15.62:0/4119612361 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f13880041f0 con 0x7f1394068540 2024-04-20T21:41:48.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:48.548+0000 7f1392000700 1 -- 172.21.15.62:0/4119612361 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1388004cf0 con 0x7f1394068540 2024-04-20T21:41:48.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:48.548+0000 7f1392000700 1 -- 172.21.15.62:0/4119612361 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f138800b750 con 0x7f1394068540 2024-04-20T21:41:48.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:48.548+0000 7f139c000700 1 -- 172.21.15.62:0/4119612361 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f139419ccb0 con 0x7f1394068540 2024-04-20T21:41:48.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:48.548+0000 7f139c000700 1 -- 172.21.15.62:0/4119612361 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f139419d200 con 0x7f1394068540 2024-04-20T21:41:48.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:48.550+0000 7f139c000700 1 -- 172.21.15.62:0/4119612361 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f139404f2a0 con 0x7f1394068540 2024-04-20T21:41:48.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:48.551+0000 7f1392000700 1 -- 172.21.15.62:0/4119612361 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f1388004350 con 0x7f1394068540 2024-04-20T21:41:48.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:48.552+0000 7f1392000700 1 --2- 172.21.15.62:0/4119612361 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f137c06ecc0 0x7f137c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:41:48.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:48.552+0000 7f1392000700 1 -- 172.21.15.62:0/4119612361 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f138808ed20 con 0x7f1394068540 2024-04-20T21:41:48.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:48.552+0000 7f1398c00700 1 --2- 172.21.15.62:0/4119612361 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f137c06ecc0 0x7f137c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:41:48.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:48.553+0000 7f1398c00700 1 --2- 172.21.15.62:0/4119612361 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f137c06ecc0 0x7f137c071170 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f1380005b40 tx=0x7f138001a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:41:48.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:48.557+0000 7f1392000700 1 -- 172.21.15.62:0/4119612361 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f1388057b60 con 0x7f1394068540 2024-04-20T21:41:48.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:48.901+0000 7f139c000700 1 -- 172.21.15.62:0/4119612361 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f139419d620 con 0x7f1394068540 2024-04-20T21:41:48.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:48.902+0000 7f1392000700 1 -- 172.21.15.62:0/4119612361 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+700 (secure 0 0 0) 0x7f13880162d0 con 0x7f1394068540 2024-04-20T21:41:48.902 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:41:48.902 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T21:41:48.902 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:41:48.902 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:41:48.902 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T21:41:48.902 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:41:48.902 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:41:48.902 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T21:41:48.902 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 5 2024-04-20T21:41:48.902 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:41:48.902 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T21:41:48.902 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 4 2024-04-20T21:41:48.902 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:41:48.902 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T21:41:48.903 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 4, 2024-04-20T21:41:48.903 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 9 2024-04-20T21:41:48.903 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T21:41:48.903 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:41:48.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:48.905+0000 7f139c000700 1 -- 172.21.15.62:0/4119612361 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f137c06ecc0 msgr2=0x7f137c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:41:48.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:48.905+0000 7f139c000700 1 --2- 172.21.15.62:0/4119612361 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f137c06ecc0 0x7f137c071170 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f1380005b40 tx=0x7f138001a040 comp rx=0 tx=0).stop 2024-04-20T21:41:48.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:48.905+0000 7f139c000700 1 -- 172.21.15.62:0/4119612361 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1394068540 msgr2=0x7f1394198020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:41:48.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:48.905+0000 7f139c000700 1 --2- 172.21.15.62:0/4119612361 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1394068540 0x7f1394198020 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f138800d900 tx=0x7f138800dcc0 comp rx=0 tx=0).stop 2024-04-20T21:41:48.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:48.905+0000 7f139c000700 1 -- 172.21.15.62:0/4119612361 shutdown_connections 2024-04-20T21:41:48.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:48.905+0000 7f139c000700 1 --2- 172.21.15.62:0/4119612361 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f137c06ecc0 0x7f137c071170 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:41:48.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:48.905+0000 7f139c000700 1 --2- 172.21.15.62:0/4119612361 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1394068540 0x7f1394198020 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:41:48.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:48.905+0000 7f139c000700 1 --2- 172.21.15.62:0/4119612361 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1394106320 0x7f1394198560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:41:48.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:48.905+0000 7f139c000700 1 -- 172.21.15.62:0/4119612361 >> 172.21.15.62:0/4119612361 conn(0x7f1394075130 msgr2=0x7f13940ffff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:41:48.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:48.906+0000 7f139c000700 1 -- 172.21.15.62:0/4119612361 shutdown_connections 2024-04-20T21:41:48.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:48.906+0000 7f139c000700 1 -- 172.21.15.62:0/4119612361 wait complete. 2024-04-20T21:41:49.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.068+0000 7fdb07c00700 1 -- 172.21.15.62:0/1857502124 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdb00106320 msgr2=0x7fdb001066f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:41:49.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.068+0000 7fdb07c00700 1 --2- 172.21.15.62:0/1857502124 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdb00106320 0x7fdb001066f0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fdaf4009910 tx=0x7fdaf4009c20 comp rx=0 tx=0).stop 2024-04-20T21:41:49.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.068+0000 7fdb07c00700 1 -- 172.21.15.62:0/1857502124 shutdown_connections 2024-04-20T21:41:49.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.068+0000 7fdb07c00700 1 --2- 172.21.15.62:0/1857502124 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb00068540 0x7fdb000689b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:41:49.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.068+0000 7fdb07c00700 1 --2- 172.21.15.62:0/1857502124 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdb00106320 0x7fdb001066f0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:41:49.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.068+0000 7fdb07c00700 1 -- 172.21.15.62:0/1857502124 >> 172.21.15.62:0/1857502124 conn(0x7fdb00075130 msgr2=0x7fdb00075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:41:49.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.068+0000 7fdb07c00700 1 -- 172.21.15.62:0/1857502124 shutdown_connections 2024-04-20T21:41:49.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.068+0000 7fdb07c00700 1 -- 172.21.15.62:0/1857502124 wait complete. 2024-04-20T21:41:49.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.069+0000 7fdb07c00700 1 Processor -- start 2024-04-20T21:41:49.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.069+0000 7fdb07c00700 1 -- start start 2024-04-20T21:41:49.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.070+0000 7fdb07c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb00068540 0x7fdb00198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:41:49.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.070+0000 7fdb07c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdb00106320 0x7fdb001985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:41:49.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.070+0000 7fdb07c00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb00198c80 con 0x7fdb00106320 2024-04-20T21:41:49.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.070+0000 7fdb07c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb0019ca10 con 0x7fdb00068540 2024-04-20T21:41:49.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.070+0000 7fdaffe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdb00106320 0x7fdb001985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:41:49.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.070+0000 7fdaffe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdb00106320 0x7fdb001985a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59560/0 (socket says 172.21.15.62:59560) 2024-04-20T21:41:49.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.070+0000 7fdaffe00700 1 -- 172.21.15.62:0/3107588347 learned_addr learned my addr 172.21.15.62:0/3107588347 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:41:49.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.070+0000 7fdb05200700 1 --2- 172.21.15.62:0/3107588347 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb00068540 0x7fdb00198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:41:49.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.070+0000 7fdaffe00700 1 -- 172.21.15.62:0/3107588347 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb00068540 msgr2=0x7fdb00198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:41:49.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.071+0000 7fdaffe00700 1 --2- 172.21.15.62:0/3107588347 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb00068540 0x7fdb00198060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:41:49.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.071+0000 7fdaffe00700 1 -- 172.21.15.62:0/3107588347 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaf40095f0 con 0x7fdb00106320 2024-04-20T21:41:49.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.071+0000 7fdb05200700 1 --2- 172.21.15.62:0/3107588347 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb00068540 0x7fdb00198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-20T21:41:49.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.071+0000 7fdaffe00700 1 --2- 172.21.15.62:0/3107588347 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdb00106320 0x7fdb001985a0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fdaec00c8a0 tx=0x7fdaec00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:41:49.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.071+0000 7fdafd600700 1 -- 172.21.15.62:0/3107588347 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdaec00cea0 con 0x7fdb00106320 2024-04-20T21:41:49.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.071+0000 7fdafd600700 1 -- 172.21.15.62:0/3107588347 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdaec004830 con 0x7fdb00106320 2024-04-20T21:41:49.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.071+0000 7fdafd600700 1 -- 172.21.15.62:0/3107588347 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdaec0055b0 con 0x7fdb00106320 2024-04-20T21:41:49.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.071+0000 7fdb07c00700 1 -- 172.21.15.62:0/3107588347 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb0019ccf0 con 0x7fdb00106320 2024-04-20T21:41:49.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.071+0000 7fdb07c00700 1 -- 172.21.15.62:0/3107588347 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb0019d240 con 0x7fdb00106320 2024-04-20T21:41:49.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.073+0000 7fdb07c00700 1 -- 172.21.15.62:0/3107588347 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb0004f2a0 con 0x7fdb00106320 2024-04-20T21:41:49.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.074+0000 7fdafd600700 1 -- 172.21.15.62:0/3107588347 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fdaec0049a0 con 0x7fdb00106320 2024-04-20T21:41:49.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.075+0000 7fdafd600700 1 --2- 172.21.15.62:0/3107588347 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fdae806ecc0 0x7fdae8071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:41:49.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.075+0000 7fdafd600700 1 -- 172.21.15.62:0/3107588347 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fdaec08ea40 con 0x7fdb00106320 2024-04-20T21:41:49.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.075+0000 7fdb05200700 1 --2- 172.21.15.62:0/3107588347 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fdae806ecc0 0x7fdae8071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:41:49.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.076+0000 7fdb05200700 1 --2- 172.21.15.62:0/3107588347 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fdae806ecc0 0x7fdae8071170 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fdaf400b5c0 tx=0x7fdaf401a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:41:49.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.081+0000 7fdafd600700 1 -- 172.21.15.62:0/3107588347 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fdaec057880 con 0x7fdb00106320 2024-04-20T21:41:49.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:49 smithi062 ceph-mon[76322]: from='client.34406 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:41:49.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:49 smithi062 ceph-mon[76322]: pgmap v187: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 315 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 56/366 objects degraded (15.301%) 2024-04-20T21:41:49.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:49 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/4119612361' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:41:49.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.356+0000 7fdb07c00700 1 -- 172.21.15.62:0/3107588347 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fdb001993c0 con 0x7fdb00106320 2024-04-20T21:41:49.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.360+0000 7fdafd600700 1 -- 172.21.15.62:0/3107588347 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 18 v18) v1 ==== 76+0+1928 (secure 0 0 0) 0x7fdaec01d070 con 0x7fdb00106320 2024-04-20T21:41:49.360 INFO:teuthology.orchestra.run.smithi062.stdout:e18 2024-04-20T21:41:49.360 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T21:41:49.360 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T21:41:49.360 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T21:41:49.360 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:41:49.360 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T21:41:49.360 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T21:41:49.360 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 18 2024-04-20T21:41:49.360 INFO:teuthology.orchestra.run.smithi062.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-20T21:41:49.360 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T21:41:49.360 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T21:37:06.018300+0000 2024-04-20T21:41:49.360 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T21:41:49.360 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T21:41:49.361 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T21:41:49.361 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T21:41:49.361 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T21:41:49.361 INFO:teuthology.orchestra.run.smithi062.stdout:max_xattr_size 65536 2024-04-20T21:41:49.361 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T21:41:49.361 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T21:41:49.361 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 0 2024-04-20T21:41:49.361 INFO:teuthology.orchestra.run.smithi062.stdout:snap_rstat 1 2024-04-20T21:41:49.361 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T21:41:49.361 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 2 2024-04-20T21:41:49.361 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1 2024-04-20T21:41:49.361 INFO:teuthology.orchestra.run.smithi062.stdout:up {0=14458,1=24279} 2024-04-20T21:41:49.361 INFO:teuthology.orchestra.run.smithi062.stdout:failed 2024-04-20T21:41:49.362 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T21:41:49.362 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T21:41:49.362 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T21:41:49.362 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T21:41:49.362 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data enabled 2024-04-20T21:41:49.362 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T21:41:49.362 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T21:41:49.362 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 1 2024-04-20T21:41:49.362 INFO:teuthology.orchestra.run.smithi062.stdout:qdb_cluster leader: 14458 members: 24279,14458 2024-04-20T21:41:49.362 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ryqblb{0:14458} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.62:6826/1147127123,v1:172.21.15.62:6827/1147127123] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:41:49.362 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ozizwi{0:14474} state up:standby-replay seq 3 join_fscid=1 addr [v2:172.21.15.62:6828/3494174293,v1:172.21.15.62:6829/3494174293] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:41:49.362 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.dvimer{1:24279} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.189:6824/237262514,v1:172.21.15.189:6825/237262514] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:41:49.362 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.xbgspw{1:24291} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.189:6826/1495356258,v1:172.21.15.189:6827/1495356258] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:41:49.362 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:41:49.363 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:41:49.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.363+0000 7fdb07c00700 1 -- 172.21.15.62:0/3107588347 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fdae806ecc0 msgr2=0x7fdae8071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:41:49.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.363+0000 7fdb07c00700 1 --2- 172.21.15.62:0/3107588347 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fdae806ecc0 0x7fdae8071170 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fdaf400b5c0 tx=0x7fdaf401a040 comp rx=0 tx=0).stop 2024-04-20T21:41:49.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.363+0000 7fdb07c00700 1 -- 172.21.15.62:0/3107588347 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdb00106320 msgr2=0x7fdb001985a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:41:49.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.363+0000 7fdb07c00700 1 --2- 172.21.15.62:0/3107588347 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdb00106320 0x7fdb001985a0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fdaec00c8a0 tx=0x7fdaec00cc60 comp rx=0 tx=0).stop 2024-04-20T21:41:49.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.363+0000 7fdb07c00700 1 -- 172.21.15.62:0/3107588347 shutdown_connections 2024-04-20T21:41:49.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.363+0000 7fdb07c00700 1 --2- 172.21.15.62:0/3107588347 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fdae806ecc0 0x7fdae8071170 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:41:49.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.363+0000 7fdb07c00700 1 --2- 172.21.15.62:0/3107588347 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb00068540 0x7fdb00198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:41:49.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.363+0000 7fdb07c00700 1 --2- 172.21.15.62:0/3107588347 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdb00106320 0x7fdb001985a0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:41:49.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.364+0000 7fdb07c00700 1 -- 172.21.15.62:0/3107588347 >> 172.21.15.62:0/3107588347 conn(0x7fdb00075130 msgr2=0x7fdb000fffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:41:49.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.364+0000 7fdb07c00700 1 -- 172.21.15.62:0/3107588347 shutdown_connections 2024-04-20T21:41:49.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.364+0000 7fdb07c00700 1 -- 172.21.15.62:0/3107588347 wait complete. 2024-04-20T21:41:49.364 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 18 2024-04-20T21:41:49.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:49 smithi189 ceph-mon[57022]: from='client.34406 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:41:49.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:49 smithi189 ceph-mon[57022]: pgmap v187: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 315 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 56/366 objects degraded (15.301%) 2024-04-20T21:41:49.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:49 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/4119612361' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:41:49.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.523+0000 7f9443200700 1 -- 172.21.15.62:0/3018159973 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f943c0739a0 msgr2=0x7f943c10c5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:41:49.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.523+0000 7f9443200700 1 --2- 172.21.15.62:0/3018159973 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f943c0739a0 0x7f943c10c5e0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f9428009940 tx=0x7f9428009c50 comp rx=0 tx=0).stop 2024-04-20T21:41:49.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.523+0000 7f9443200700 1 -- 172.21.15.62:0/3018159973 shutdown_connections 2024-04-20T21:41:49.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.523+0000 7f9443200700 1 --2- 172.21.15.62:0/3018159973 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f943c0739a0 0x7f943c10c5e0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:41:49.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.523+0000 7f9443200700 1 --2- 172.21.15.62:0/3018159973 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f943c073090 0x7f943c073460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:41:49.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.523+0000 7f9443200700 1 -- 172.21.15.62:0/3018159973 >> 172.21.15.62:0/3018159973 conn(0x7f943c0fbda0 msgr2=0x7f943c0fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:41:49.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.524+0000 7f9443200700 1 -- 172.21.15.62:0/3018159973 shutdown_connections 2024-04-20T21:41:49.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.524+0000 7f9443200700 1 -- 172.21.15.62:0/3018159973 wait complete. 2024-04-20T21:41:49.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.525+0000 7f9443200700 1 Processor -- start 2024-04-20T21:41:49.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.525+0000 7f9443200700 1 -- start start 2024-04-20T21:41:49.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.526+0000 7f9443200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f943c073090 0x7f943c1981b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:41:49.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.526+0000 7f9443200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f943c0739a0 0x7f943c1986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:41:49.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.526+0000 7f9443200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f943c198dd0 con 0x7f943c073090 2024-04-20T21:41:49.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.526+0000 7f9443200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f943c19cb60 con 0x7f943c0739a0 2024-04-20T21:41:49.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.526+0000 7f943be00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f943c073090 0x7f943c1981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:41:49.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.526+0000 7f943be00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f943c073090 0x7f943c1981b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59582/0 (socket says 172.21.15.62:59582) 2024-04-20T21:41:49.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.526+0000 7f943be00700 1 -- 172.21.15.62:0/646791703 learned_addr learned my addr 172.21.15.62:0/646791703 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:41:49.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.527+0000 7f943b400700 1 --2- 172.21.15.62:0/646791703 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f943c0739a0 0x7f943c1986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:41:49.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.527+0000 7f943be00700 1 -- 172.21.15.62:0/646791703 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f943c0739a0 msgr2=0x7f943c1986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:41:49.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.527+0000 7f943be00700 1 --2- 172.21.15.62:0/646791703 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f943c0739a0 0x7f943c1986f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:41:49.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.527+0000 7f943be00700 1 -- 172.21.15.62:0/646791703 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94280095f0 con 0x7f943c073090 2024-04-20T21:41:49.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.527+0000 7f943be00700 1 --2- 172.21.15.62:0/646791703 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f943c073090 0x7f943c1981b0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f942c00c8a0 tx=0x7f942c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:41:49.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.528+0000 7f9438c00700 1 -- 172.21.15.62:0/646791703 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f942c00cea0 con 0x7f943c073090 2024-04-20T21:41:49.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.528+0000 7f9438c00700 1 -- 172.21.15.62:0/646791703 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f942c004830 con 0x7f943c073090 2024-04-20T21:41:49.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.528+0000 7f9443200700 1 -- 172.21.15.62:0/646791703 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f943c19ce40 con 0x7f943c073090 2024-04-20T21:41:49.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.528+0000 7f9438c00700 1 -- 172.21.15.62:0/646791703 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f942c0055b0 con 0x7f943c073090 2024-04-20T21:41:49.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.528+0000 7f9443200700 1 -- 172.21.15.62:0/646791703 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f943c19d390 con 0x7f943c073090 2024-04-20T21:41:49.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.529+0000 7f9443200700 1 -- 172.21.15.62:0/646791703 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f943c109ce0 con 0x7f943c073090 2024-04-20T21:41:49.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.532+0000 7f9438c00700 1 -- 172.21.15.62:0/646791703 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f942c0049a0 con 0x7f943c073090 2024-04-20T21:41:49.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.533+0000 7f9438c00700 1 --2- 172.21.15.62:0/646791703 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f942006ecc0 0x7f9420071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:41:49.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.533+0000 7f9438c00700 1 -- 172.21.15.62:0/646791703 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f942c08eef0 con 0x7f943c073090 2024-04-20T21:41:49.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.534+0000 7f943b400700 1 --2- 172.21.15.62:0/646791703 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f942006ecc0 0x7f9420071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:41:49.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.534+0000 7f943b400700 1 --2- 172.21.15.62:0/646791703 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f942006ecc0 0x7f9420071170 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f943c1997d0 tx=0x7f9428005c20 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:41:49.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.540+0000 7f9438c00700 1 -- 172.21.15.62:0/646791703 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f942c0bdff0 con 0x7f943c073090 2024-04-20T21:41:49.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.798+0000 7f9443200700 1 -- 172.21.15.62:0/646791703 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f943c19d610 con 0x7f942006ecc0 2024-04-20T21:41:49.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.801+0000 7f9438c00700 1 -- 172.21.15.62:0/646791703 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7f943c19d610 con 0x7f942006ecc0 2024-04-20T21:41:49.800 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:41:49.800 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T21:41:49.800 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T21:41:49.800 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-20T21:41:49.801 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T21:41:49.801 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr", 2024-04-20T21:41:49.801 INFO:teuthology.orchestra.run.smithi062.stdout: "crash", 2024-04-20T21:41:49.801 INFO:teuthology.orchestra.run.smithi062.stdout: "mon" 2024-04-20T21:41:49.801 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T21:41:49.801 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "11/23 daemons upgraded", 2024-04-20T21:41:49.801 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading osd daemons", 2024-04-20T21:41:49.801 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T21:41:49.801 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:41:49.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.804+0000 7f9443200700 1 -- 172.21.15.62:0/646791703 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f942006ecc0 msgr2=0x7f9420071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:41:49.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.804+0000 7f9443200700 1 --2- 172.21.15.62:0/646791703 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f942006ecc0 0x7f9420071170 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f943c1997d0 tx=0x7f9428005c20 comp rx=0 tx=0).stop 2024-04-20T21:41:49.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.804+0000 7f9443200700 1 -- 172.21.15.62:0/646791703 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f943c073090 msgr2=0x7f943c1981b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:41:49.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.804+0000 7f9443200700 1 --2- 172.21.15.62:0/646791703 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f943c073090 0x7f943c1981b0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f942c00c8a0 tx=0x7f942c00cc60 comp rx=0 tx=0).stop 2024-04-20T21:41:49.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.804+0000 7f9443200700 1 -- 172.21.15.62:0/646791703 shutdown_connections 2024-04-20T21:41:49.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.804+0000 7f9443200700 1 --2- 172.21.15.62:0/646791703 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f942006ecc0 0x7f9420071170 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:41:49.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.804+0000 7f9443200700 1 --2- 172.21.15.62:0/646791703 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f943c073090 0x7f943c1981b0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:41:49.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.804+0000 7f9443200700 1 --2- 172.21.15.62:0/646791703 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f943c0739a0 0x7f943c1986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:41:49.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.804+0000 7f9443200700 1 -- 172.21.15.62:0/646791703 >> 172.21.15.62:0/646791703 conn(0x7f943c0fbda0 msgr2=0x7f943c106e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:41:49.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.804+0000 7f9443200700 1 -- 172.21.15.62:0/646791703 shutdown_connections 2024-04-20T21:41:49.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.804+0000 7f9443200700 1 -- 172.21.15.62:0/646791703 wait complete. 2024-04-20T21:41:49.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.968+0000 7f00b2000700 1 -- 172.21.15.62:0/3058453686 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f00ac10c8f0 msgr2=0x7f00ac10ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:41:49.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.968+0000 7f00b2000700 1 --2- 172.21.15.62:0/3058453686 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f00ac10c8f0 0x7f00ac10ccc0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f009c009910 tx=0x7f009c009c20 comp rx=0 tx=0).stop 2024-04-20T21:41:49.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.969+0000 7f00b2000700 1 -- 172.21.15.62:0/3058453686 shutdown_connections 2024-04-20T21:41:49.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.969+0000 7f00b2000700 1 --2- 172.21.15.62:0/3058453686 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f00ac106b80 0x7f00ac106ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:41:49.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.969+0000 7f00b2000700 1 --2- 172.21.15.62:0/3058453686 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f00ac10c8f0 0x7f00ac10ccc0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:41:49.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.969+0000 7f00b2000700 1 -- 172.21.15.62:0/3058453686 >> 172.21.15.62:0/3058453686 conn(0x7f00ac076060 msgr2=0x7f00ac078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:41:49.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.969+0000 7f00b2000700 1 -- 172.21.15.62:0/3058453686 shutdown_connections 2024-04-20T21:41:49.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.969+0000 7f00b2000700 1 -- 172.21.15.62:0/3058453686 wait complete. 2024-04-20T21:41:49.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.970+0000 7f00b2000700 1 Processor -- start 2024-04-20T21:41:49.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.970+0000 7f00b2000700 1 -- start start 2024-04-20T21:41:49.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.970+0000 7f00b2000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f00ac106b80 0x7f00ac19c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:41:49.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.970+0000 7f00b2000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f00ac10c8f0 0x7f00ac19ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:41:49.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.970+0000 7f00b2000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00ac19d120 con 0x7f00ac106b80 2024-04-20T21:41:49.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.970+0000 7f00b2000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00ac1a0eb0 con 0x7f00ac10c8f0 2024-04-20T21:41:49.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.971+0000 7f00ab400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f00ac106b80 0x7f00ac19c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:41:49.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.971+0000 7f00ab400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f00ac106b80 0x7f00ac19c500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59590/0 (socket says 172.21.15.62:59590) 2024-04-20T21:41:49.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.971+0000 7f00ab400700 1 -- 172.21.15.62:0/3734327671 learned_addr learned my addr 172.21.15.62:0/3734327671 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:41:49.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.971+0000 7f00aaa00700 1 --2- 172.21.15.62:0/3734327671 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f00ac10c8f0 0x7f00ac19ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:41:49.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.972+0000 7f00ab400700 1 -- 172.21.15.62:0/3734327671 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f00ac10c8f0 msgr2=0x7f00ac19ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:41:49.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.972+0000 7f00ab400700 1 --2- 172.21.15.62:0/3734327671 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f00ac10c8f0 0x7f00ac19ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:41:49.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.972+0000 7f00ab400700 1 -- 172.21.15.62:0/3734327671 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f009c0095f0 con 0x7f00ac106b80 2024-04-20T21:41:49.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.972+0000 7f00ab400700 1 --2- 172.21.15.62:0/3734327671 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f00ac106b80 0x7f00ac19c500 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f009c0092e0 tx=0x7f009c00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:41:49.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.972+0000 7f00a3e00700 1 -- 172.21.15.62:0/3734327671 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f009c01d070 con 0x7f00ac106b80 2024-04-20T21:41:49.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.972+0000 7f00a3e00700 1 -- 172.21.15.62:0/3734327671 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f009c005e00 con 0x7f00ac106b80 2024-04-20T21:41:49.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.972+0000 7f00b2000700 1 -- 172.21.15.62:0/3734327671 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00ac1a1130 con 0x7f00ac106b80 2024-04-20T21:41:49.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.972+0000 7f00a3e00700 1 -- 172.21.15.62:0/3734327671 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f009c017400 con 0x7f00ac106b80 2024-04-20T21:41:49.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.973+0000 7f00b2000700 1 -- 172.21.15.62:0/3734327671 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00ac1a1620 con 0x7f00ac106b80 2024-04-20T21:41:49.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.974+0000 7f00b2000700 1 -- 172.21.15.62:0/3734327671 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f008c005320 con 0x7f00ac106b80 2024-04-20T21:41:49.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.975+0000 7f00a3e00700 1 -- 172.21.15.62:0/3734327671 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f009c003a60 con 0x7f00ac106b80 2024-04-20T21:41:49.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.976+0000 7f00a3e00700 1 --2- 172.21.15.62:0/3734327671 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f009006ecc0 0x7f0090071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:41:49.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.976+0000 7f00a3e00700 1 -- 172.21.15.62:0/3734327671 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f009c090250 con 0x7f00ac106b80 2024-04-20T21:41:49.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.976+0000 7f00aaa00700 1 --2- 172.21.15.62:0/3734327671 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f009006ecc0 0x7f0090071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:41:49.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.977+0000 7f00aaa00700 1 --2- 172.21.15.62:0/3734327671 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f009006ecc0 0x7f0090071170 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f00ac19db20 tx=0x7f009800b410 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:41:49.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:49.981+0000 7f00a3e00700 1 -- 172.21.15.62:0/3734327671 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f009c059010 con 0x7f00ac106b80 2024-04-20T21:41:50.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:50.307+0000 7f00b2000700 1 -- 172.21.15.62:0/3734327671 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f008c005190 con 0x7f00ac106b80 2024-04-20T21:41:50.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:50.308+0000 7f00a3e00700 1 -- 172.21.15.62:0/3734327671 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+1480 (secure 0 0 0) 0x7f009c05c6d0 con 0x7f00ac106b80 2024-04-20T21:41:50.307 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data; 1 osds down; all OSDs are running squid or later but require_osd_release < squid; Degraded data redundancy: 56/366 objects degraded (15.301%), 14 pgs degraded 2024-04-20T21:41:50.307 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T21:41:50.307 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T21:41:50.307 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] OSD_DOWN: 1 osds down 2024-04-20T21:41:50.307 INFO:teuthology.orchestra.run.smithi062.stdout: osd.5 (root=default,host=smithi189) is down 2024-04-20T21:41:50.307 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] OSD_UPGRADE_FINISHED: all OSDs are running squid or later but require_osd_release < squid 2024-04-20T21:41:50.307 INFO:teuthology.orchestra.run.smithi062.stdout: all OSDs are running squid or later but require_osd_release < squid 2024-04-20T21:41:50.307 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 56/366 objects degraded (15.301%), 14 pgs degraded 2024-04-20T21:41:50.308 INFO:teuthology.orchestra.run.smithi062.stdout: pg 2.2 is active+undersized+degraded, acting [1,0] 2024-04-20T21:41:50.308 INFO:teuthology.orchestra.run.smithi062.stdout: pg 2.3 is active+undersized+degraded, acting [2,1] 2024-04-20T21:41:50.308 INFO:teuthology.orchestra.run.smithi062.stdout: pg 2.8 is active+undersized+degraded, acting [3,0] 2024-04-20T21:41:50.308 INFO:teuthology.orchestra.run.smithi062.stdout: pg 2.b is active+undersized+degraded, acting [3,4] 2024-04-20T21:41:50.308 INFO:teuthology.orchestra.run.smithi062.stdout: pg 2.f is active+undersized+degraded, acting [4,0] 2024-04-20T21:41:50.308 INFO:teuthology.orchestra.run.smithi062.stdout: pg 2.14 is active+undersized+degraded, acting [3,4] 2024-04-20T21:41:50.308 INFO:teuthology.orchestra.run.smithi062.stdout: pg 2.16 is active+undersized+degraded, acting [3,2] 2024-04-20T21:41:50.308 INFO:teuthology.orchestra.run.smithi062.stdout: pg 2.17 is active+undersized+degraded, acting [2,1] 2024-04-20T21:41:50.308 INFO:teuthology.orchestra.run.smithi062.stdout: pg 2.18 is active+undersized+degraded, acting [4,3] 2024-04-20T21:41:50.308 INFO:teuthology.orchestra.run.smithi062.stdout: pg 2.1a is active+undersized+degraded, acting [3,4] 2024-04-20T21:41:50.308 INFO:teuthology.orchestra.run.smithi062.stdout: pg 2.1b is active+undersized+degraded, acting [1,0] 2024-04-20T21:41:50.308 INFO:teuthology.orchestra.run.smithi062.stdout: pg 2.1c is active+undersized+degraded, acting [4,2] 2024-04-20T21:41:50.308 INFO:teuthology.orchestra.run.smithi062.stdout: pg 2.1d is active+undersized+degraded, acting [3,0] 2024-04-20T21:41:50.308 INFO:teuthology.orchestra.run.smithi062.stdout: pg 2.1e is active+undersized+degraded, acting [2,0] 2024-04-20T21:41:50.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:50.310+0000 7f00b2000700 1 -- 172.21.15.62:0/3734327671 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f009006ecc0 msgr2=0x7f0090071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:41:50.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:50.310+0000 7f00b2000700 1 --2- 172.21.15.62:0/3734327671 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f009006ecc0 0x7f0090071170 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f00ac19db20 tx=0x7f009800b410 comp rx=0 tx=0).stop 2024-04-20T21:41:50.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:50.310+0000 7f00b2000700 1 -- 172.21.15.62:0/3734327671 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f00ac106b80 msgr2=0x7f00ac19c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:41:50.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:50.310+0000 7f00b2000700 1 --2- 172.21.15.62:0/3734327671 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f00ac106b80 0x7f00ac19c500 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f009c0092e0 tx=0x7f009c00bd70 comp rx=0 tx=0).stop 2024-04-20T21:41:50.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:50.311+0000 7f00b2000700 1 -- 172.21.15.62:0/3734327671 shutdown_connections 2024-04-20T21:41:50.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:50.311+0000 7f00b2000700 1 --2- 172.21.15.62:0/3734327671 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f009006ecc0 0x7f0090071170 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:41:50.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:50.311+0000 7f00b2000700 1 --2- 172.21.15.62:0/3734327671 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f00ac106b80 0x7f00ac19c500 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:41:50.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:50.311+0000 7f00b2000700 1 --2- 172.21.15.62:0/3734327671 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f00ac10c8f0 0x7f00ac19ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:41:50.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:50.311+0000 7f00b2000700 1 -- 172.21.15.62:0/3734327671 >> 172.21.15.62:0/3734327671 conn(0x7f00ac076060 msgr2=0x7f00ac077aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:41:50.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:50.311+0000 7f00b2000700 1 -- 172.21.15.62:0/3734327671 shutdown_connections 2024-04-20T21:41:50.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:41:50.311+0000 7f00b2000700 1 -- 172.21.15.62:0/3734327671 wait complete. 2024-04-20T21:41:50.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:50 smithi189 ceph-mon[57022]: Health check update: Degraded data redundancy: 56/366 objects degraded (15.301%), 14 pgs degraded (PG_DEGRADED) 2024-04-20T21:41:50.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:50 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:41:50.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:50 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/3107588347' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T21:41:50.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:50 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:41:50.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:50 smithi189 ceph-mon[57022]: from='client.34418 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:41:50.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:50 smithi062 ceph-mon[76322]: Health check update: Degraded data redundancy: 56/366 objects degraded (15.301%), 14 pgs degraded (PG_DEGRADED) 2024-04-20T21:41:50.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:50 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:41:50.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:50 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/3107588347' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T21:41:50.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:50 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:41:50.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:50 smithi062 ceph-mon[76322]: from='client.34418 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:41:51.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:51 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/3734327671' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T21:41:51.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:51 smithi189 ceph-mon[57022]: pgmap v188: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 315 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 56/366 objects degraded (15.301%) 2024-04-20T21:41:51.237 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 21:41:51 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-5[76275]: 2024-04-20T21:41:51.014+0000 7f31780fc7c0 -1 osd.5 0 read_superblock omap replica is missing. 2024-04-20T21:41:51.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:51 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/3734327671' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T21:41:51.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:51 smithi062 ceph-mon[76322]: pgmap v188: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 315 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 56/366 objects degraded (15.301%) 2024-04-20T21:41:51.737 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 21:41:51 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-5[76275]: 2024-04-20T21:41:51.363+0000 7f31780fc7c0 -1 osd.5 73 log_to_monitors true 2024-04-20T21:41:52.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:52 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:41:52.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:52 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:41:52.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:52 smithi189 ceph-mon[57022]: from='osd.5 [v2:172.21.15.189:6816/2760890917,v1:172.21.15.189:6817/2760890917]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-20T21:41:52.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:52 smithi189 ceph-mon[57022]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-20T21:41:52.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:52 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:41:52.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:52 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:41:52.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:52 smithi062 ceph-mon[76322]: from='osd.5 [v2:172.21.15.189:6816/2760890917,v1:172.21.15.189:6817/2760890917]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-20T21:41:52.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:52 smithi062 ceph-mon[76322]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-20T21:41:53.737 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 21:41:53 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-5[76275]: 2024-04-20T21:41:53.405+0000 7f3162400700 -1 osd.5 73 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-20T21:41:53.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:53 smithi189 ceph-mon[57022]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-20T21:41:53.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:53 smithi189 ceph-mon[57022]: from='osd.5 [v2:172.21.15.189:6816/2760890917,v1:172.21.15.189:6817/2760890917]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-04-20T21:41:53.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:53 smithi189 ceph-mon[57022]: osdmap e76: 6 total, 5 up, 6 in 2024-04-20T21:41:53.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:53 smithi189 ceph-mon[57022]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-04-20T21:41:53.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:53 smithi189 ceph-mon[57022]: pgmap v190: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 56/366 objects degraded (15.301%) 2024-04-20T21:41:53.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:53 smithi062 ceph-mon[76322]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-20T21:41:53.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:53 smithi062 ceph-mon[76322]: from='osd.5 [v2:172.21.15.189:6816/2760890917,v1:172.21.15.189:6817/2760890917]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-04-20T21:41:53.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:53 smithi062 ceph-mon[76322]: osdmap e76: 6 total, 5 up, 6 in 2024-04-20T21:41:53.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:53 smithi062 ceph-mon[76322]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-04-20T21:41:53.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:53 smithi062 ceph-mon[76322]: pgmap v190: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 56/366 objects degraded (15.301%) 2024-04-20T21:41:54.634 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:54 smithi189 ceph-mon[57022]: from='osd.5 ' entity='osd.5' 2024-04-20T21:41:54.634 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:54 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:41:54.634 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:54 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:41:54.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:54 smithi062 ceph-mon[76322]: from='osd.5 ' entity='osd.5' 2024-04-20T21:41:54.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:54 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:41:54.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:54 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:41:55.639 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:55 smithi189 ceph-mon[57022]: pgmap v191: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s; 56/366 objects degraded (15.301%) 2024-04-20T21:41:55.639 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:55 smithi189 ceph-mon[57022]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-20T21:41:55.639 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:55 smithi189 ceph-mon[57022]: osd.5 [v2:172.21.15.189:6816/2760890917,v1:172.21.15.189:6817/2760890917] boot 2024-04-20T21:41:55.639 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:55 smithi189 ceph-mon[57022]: osdmap e77: 6 total, 6 up, 6 in 2024-04-20T21:41:55.639 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:55 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-20T21:41:55.639 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:55 smithi189 ceph-mon[57022]: osdmap e78: 6 total, 6 up, 6 in 2024-04-20T21:41:55.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:55 smithi062 ceph-mon[76322]: pgmap v191: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s; 56/366 objects degraded (15.301%) 2024-04-20T21:41:55.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:55 smithi062 ceph-mon[76322]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-20T21:41:55.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:55 smithi062 ceph-mon[76322]: osd.5 [v2:172.21.15.189:6816/2760890917,v1:172.21.15.189:6817/2760890917] boot 2024-04-20T21:41:55.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:55 smithi062 ceph-mon[76322]: osdmap e77: 6 total, 6 up, 6 in 2024-04-20T21:41:55.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:55 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-20T21:41:55.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:55 smithi062 ceph-mon[76322]: osdmap e78: 6 total, 6 up, 6 in 2024-04-20T21:41:56.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:56 smithi189 ceph-mon[57022]: Health check update: Degraded data redundancy: 52/366 objects degraded (14.208%), 12 pgs degraded (PG_DEGRADED) 2024-04-20T21:41:56.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:56 smithi062 ceph-mon[76322]: Health check update: Degraded data redundancy: 52/366 objects degraded (14.208%), 12 pgs degraded (PG_DEGRADED) 2024-04-20T21:41:57.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:57 smithi062 ceph-mon[76322]: pgmap v194: 65 pgs: 4 peering, 12 active+undersized+degraded, 12 active+undersized, 37 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s; 52/366 objects degraded (14.208%) 2024-04-20T21:41:57.888 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:57 smithi189 ceph-mon[57022]: pgmap v194: 65 pgs: 4 peering, 12 active+undersized+degraded, 12 active+undersized, 37 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s; 52/366 objects degraded (14.208%) 2024-04-20T21:42:00.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:59 smithi062 ceph-mon[76322]: pgmap v195: 65 pgs: 4 peering, 2 active+undersized+degraded, 2 active+undersized, 57 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 7/366 objects degraded (1.913%) 2024-04-20T21:42:00.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:59 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:42:00.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:59 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:42:00.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:59 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:42:00.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:59 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T21:42:00.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:59 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:42:00.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:59 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:42:00.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:59 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:42:00.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:59 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:42:00.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:59 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:42:00.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:59 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:42:00.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:59 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:42:00.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:59 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-04-20T21:42:00.037 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:59 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-04-20T21:42:00.037 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:59 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-04-20T21:42:00.037 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:59 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-04-20T21:42:00.037 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:59 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-04-20T21:42:00.037 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:59 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-04-20T21:42:00.037 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:59 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-04-20T21:42:00.037 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:59 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-04-20T21:42:00.037 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:59 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-04-20T21:42:00.037 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:59 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-04-20T21:42:00.037 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:59 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-04-20T21:42:00.037 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:59 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-04-20T21:42:00.037 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:41:59 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-04-20T21:42:00.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:59 smithi189 ceph-mon[57022]: pgmap v195: 65 pgs: 4 peering, 2 active+undersized+degraded, 2 active+undersized, 57 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 7/366 objects degraded (1.913%) 2024-04-20T21:42:00.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:59 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:42:00.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:59 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:42:00.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:59 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:42:00.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:59 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T21:42:00.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:59 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:42:00.239 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:59 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:42:00.239 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:59 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:42:00.239 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:59 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:42:00.239 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:59 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:42:00.239 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:59 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:42:00.239 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:59 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:42:00.239 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:59 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-04-20T21:42:00.239 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:59 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-04-20T21:42:00.239 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:59 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-04-20T21:42:00.239 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:59 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-04-20T21:42:00.239 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:59 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-04-20T21:42:00.239 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:59 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-04-20T21:42:00.239 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:59 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-04-20T21:42:00.240 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:59 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-04-20T21:42:00.240 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:59 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-04-20T21:42:00.240 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:59 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-04-20T21:42:00.240 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:59 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-04-20T21:42:00.240 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:59 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-04-20T21:42:00.240 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:41:59 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-04-20T21:42:01.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:00 smithi062 ceph-mon[76322]: Upgrade: Setting container_image for all osd 2024-04-20T21:42:01.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:00 smithi062 ceph-mon[76322]: Upgrade: Setting require_osd_release to 19 squid 2024-04-20T21:42:01.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:00 smithi062 ceph-mon[76322]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-04-20T21:42:01.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:00 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-04-20T21:42:01.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:00 smithi062 ceph-mon[76322]: osdmap e79: 6 total, 6 up, 6 in 2024-04-20T21:42:01.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:00 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:42:01.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:00 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "0"}]: dispatch 2024-04-20T21:42:01.188 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:00 smithi189 ceph-mon[57022]: Upgrade: Setting container_image for all osd 2024-04-20T21:42:01.188 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:00 smithi189 ceph-mon[57022]: Upgrade: Setting require_osd_release to 19 squid 2024-04-20T21:42:01.188 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:00 smithi189 ceph-mon[57022]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-04-20T21:42:01.188 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:00 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-04-20T21:42:01.188 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:00 smithi189 ceph-mon[57022]: osdmap e79: 6 total, 6 up, 6 in 2024-04-20T21:42:01.188 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:00 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:42:01.188 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:00 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "0"}]: dispatch 2024-04-20T21:42:01.705 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:01 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-mon-smithi062[76318]: 2024-04-20T21:42:01.558+0000 7f6154c00700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-20T21:42:01.989 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:01 smithi189 ceph-mon[57022]: Upgrade: Disabling standby-replay for filesystem cephfs 2024-04-20T21:42:01.989 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:01 smithi189 ceph-mon[57022]: pgmap v197: 65 pgs: 1 peering, 64 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:42:01.989 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:01 smithi189 ceph-mon[57022]: osdmap e80: 6 total, 6 up, 6 in 2024-04-20T21:42:01.989 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:01 smithi189 ceph-mon[57022]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 7/366 objects degraded (1.913%), 2 pgs degraded) 2024-04-20T21:42:01.989 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:01 smithi189 ceph-mon[57022]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-04-20T21:42:01.989 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:01 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "0"}]': finished 2024-04-20T21:42:01.989 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:01 smithi189 ceph-mon[57022]: fsmap cephfs:2 {0=cephfs.smithi062.ryqblb=up:active,1=cephfs.smithi189.dvimer=up:active} 2024-04-20T21:42:01.990 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:01 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:42:01.990 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:01 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:42:01.990 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:01 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T21:42:01.990 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:01 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:42:01.990 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:01 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:42:01.990 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:01 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:42:01.990 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:01 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:42:01.990 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:01 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:42:01.990 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:01 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:42:01.990 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:01 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-20T21:42:01.990 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:01 smithi189 ceph-mon[57022]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-20T21:42:01.990 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:01 smithi189 ceph-mon[57022]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-20T21:42:01.990 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:01 smithi189 ceph-mon[57022]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-04-20T21:42:01.990 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:01 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-20T21:42:01.990 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:01 smithi189 ceph-mon[57022]: osdmap e81: 6 total, 6 up, 6 in 2024-04-20T21:42:01.991 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:01 smithi189 ceph-mon[57022]: mds.? [v2:172.21.15.62:6828/4253521164,v1:172.21.15.62:6829/4253521164] up:boot 2024-04-20T21:42:01.991 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:01 smithi189 ceph-mon[57022]: mds.? [v2:172.21.15.189:6826/3996040526,v1:172.21.15.189:6827/3996040526] up:boot 2024-04-20T21:42:01.991 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:01 smithi189 ceph-mon[57022]: fsmap cephfs:0/2 2 up:standby, 2 failed 2024-04-20T21:42:01.991 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:01 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi062.ozizwi"}]: dispatch 2024-04-20T21:42:01.991 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:01 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi189.xbgspw"}]: dispatch 2024-04-20T21:42:02.037 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:01 smithi062 ceph-mon[76322]: Upgrade: Disabling standby-replay for filesystem cephfs 2024-04-20T21:42:02.037 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:01 smithi062 ceph-mon[76322]: pgmap v197: 65 pgs: 1 peering, 64 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:42:02.037 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:01 smithi062 ceph-mon[76322]: osdmap e80: 6 total, 6 up, 6 in 2024-04-20T21:42:02.037 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:01 smithi062 ceph-mon[76322]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 7/366 objects degraded (1.913%), 2 pgs degraded) 2024-04-20T21:42:02.037 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:01 smithi062 ceph-mon[76322]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-04-20T21:42:02.037 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:01 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "0"}]': finished 2024-04-20T21:42:02.037 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:01 smithi062 ceph-mon[76322]: fsmap cephfs:2 {0=cephfs.smithi062.ryqblb=up:active,1=cephfs.smithi189.dvimer=up:active} 2024-04-20T21:42:02.037 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:01 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:42:02.037 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:01 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:42:02.037 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:01 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T21:42:02.037 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:01 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:42:02.038 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:01 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:42:02.038 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:01 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:42:02.038 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:01 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:42:02.038 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:01 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:42:02.038 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:01 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:42:02.038 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:01 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-20T21:42:02.038 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:01 smithi062 ceph-mon[76322]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-20T21:42:02.038 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:01 smithi062 ceph-mon[76322]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-20T21:42:02.038 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:01 smithi062 ceph-mon[76322]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-04-20T21:42:02.038 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:01 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-20T21:42:02.038 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:01 smithi062 ceph-mon[76322]: osdmap e81: 6 total, 6 up, 6 in 2024-04-20T21:42:02.038 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:01 smithi062 ceph-mon[76322]: mds.? [v2:172.21.15.62:6828/4253521164,v1:172.21.15.62:6829/4253521164] up:boot 2024-04-20T21:42:02.038 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:01 smithi062 ceph-mon[76322]: mds.? [v2:172.21.15.189:6826/3996040526,v1:172.21.15.189:6827/3996040526] up:boot 2024-04-20T21:42:02.038 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:01 smithi062 ceph-mon[76322]: fsmap cephfs:0/2 2 up:standby, 2 failed 2024-04-20T21:42:02.038 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:01 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi062.ozizwi"}]: dispatch 2024-04-20T21:42:02.039 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:01 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi189.xbgspw"}]: dispatch 2024-04-20T21:42:03.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:02 smithi062 ceph-mon[76322]: Upgrade: failing fs cephfs for rapid multi-rank mds upgrade 2024-04-20T21:42:03.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:02 smithi062 ceph-mon[76322]: mds.? [v2:172.21.15.189:6824/1743111843,v1:172.21.15.189:6825/1743111843] up:boot 2024-04-20T21:42:03.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:02 smithi062 ceph-mon[76322]: mds.? [v2:172.21.15.62:6826/985180362,v1:172.21.15.62:6827/985180362] up:boot 2024-04-20T21:42:03.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:02 smithi062 ceph-mon[76322]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-20T21:42:03.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:02 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi062.ryqblb"}]: dispatch 2024-04-20T21:42:03.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:02 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi189.dvimer"}]: dispatch 2024-04-20T21:42:03.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:02 smithi189 ceph-mon[57022]: Upgrade: failing fs cephfs for rapid multi-rank mds upgrade 2024-04-20T21:42:03.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:02 smithi189 ceph-mon[57022]: mds.? [v2:172.21.15.189:6824/1743111843,v1:172.21.15.189:6825/1743111843] up:boot 2024-04-20T21:42:03.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:02 smithi189 ceph-mon[57022]: mds.? [v2:172.21.15.62:6826/985180362,v1:172.21.15.62:6827/985180362] up:boot 2024-04-20T21:42:03.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:02 smithi189 ceph-mon[57022]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-20T21:42:03.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:02 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi062.ryqblb"}]: dispatch 2024-04-20T21:42:03.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:02 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi189.dvimer"}]: dispatch 2024-04-20T21:42:04.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:03 smithi062 ceph-mon[76322]: pgmap v200: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 0 op/s 2024-04-20T21:42:04.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:03 smithi189 ceph-mon[57022]: pgmap v200: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 0 op/s 2024-04-20T21:42:05.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:04 smithi062 ceph-mon[76322]: Upgrade: Updating mds.cephfs.smithi062.ryqblb 2024-04-20T21:42:05.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:04 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:42:05.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:04 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi062.ryqblb", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-20T21:42:05.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:04 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:42:05.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:04 smithi062 ceph-mon[76322]: Deploying daemon mds.cephfs.smithi062.ryqblb on smithi062 2024-04-20T21:42:05.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:04 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:42:05.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:04 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:42:05.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:04 smithi062 ceph-mon[76322]: pgmap v201: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s rd, 2 op/s 2024-04-20T21:42:05.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:04 smithi189 ceph-mon[57022]: Upgrade: Updating mds.cephfs.smithi062.ryqblb 2024-04-20T21:42:05.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:04 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:42:05.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:04 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi062.ryqblb", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-20T21:42:05.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:04 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:42:05.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:04 smithi189 ceph-mon[57022]: Deploying daemon mds.cephfs.smithi062.ryqblb on smithi062 2024-04-20T21:42:05.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:04 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:42:05.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:04 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:42:05.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:04 smithi189 ceph-mon[57022]: pgmap v201: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s rd, 2 op/s 2024-04-20T21:42:07.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:07 smithi189 ceph-mon[57022]: pgmap v202: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s rd, 2 op/s 2024-04-20T21:42:07.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:07 smithi062 ceph-mon[76322]: pgmap v202: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s rd, 2 op/s 2024-04-20T21:42:08.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:08 smithi189 ceph-mon[57022]: osdmap e82: 6 total, 6 up, 6 in 2024-04-20T21:42:08.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:08 smithi189 ceph-mon[57022]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-04-20T21:42:08.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:08 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi062.ryqblb"}]: dispatch 2024-04-20T21:42:08.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:08 smithi062 ceph-mon[76322]: osdmap e82: 6 total, 6 up, 6 in 2024-04-20T21:42:08.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:08 smithi062 ceph-mon[76322]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-04-20T21:42:08.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:08 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi062.ryqblb"}]: dispatch 2024-04-20T21:42:09.728 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:09 smithi062 ceph-mon[76322]: pgmap v204: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-20T21:42:09.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:09 smithi189 ceph-mon[57022]: pgmap v204: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-20T21:42:10.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:10 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:42:10.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:10 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:42:11.729 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:11 smithi062 ceph-mon[76322]: pgmap v205: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 928 B/s rd, 1 op/s 2024-04-20T21:42:11.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:11 smithi189 ceph-mon[57022]: pgmap v205: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 928 B/s rd, 1 op/s 2024-04-20T21:42:13.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:13 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:42:13.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:13 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:42:13.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:13 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:42:13.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:13 smithi062 ceph-mon[76322]: pgmap v206: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 0 op/s 2024-04-20T21:42:13.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:13 smithi062 ceph-mon[76322]: mds.? [v2:172.21.15.62:6826/4113916205,v1:172.21.15.62:6827/4113916205] up:boot 2024-04-20T21:42:13.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:13 smithi062 ceph-mon[76322]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-20T21:42:13.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:13 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi062.ryqblb"}]: dispatch 2024-04-20T21:42:13.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:13 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:42:13.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:13 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:42:13.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:13 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:42:13.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:13 smithi189 ceph-mon[57022]: pgmap v206: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 0 op/s 2024-04-20T21:42:13.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:13 smithi189 ceph-mon[57022]: mds.? [v2:172.21.15.62:6826/4113916205,v1:172.21.15.62:6827/4113916205] up:boot 2024-04-20T21:42:13.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:13 smithi189 ceph-mon[57022]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-20T21:42:13.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:13 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi062.ryqblb"}]: dispatch 2024-04-20T21:42:15.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:15 smithi189 ceph-mon[57022]: pgmap v207: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:42:15.780 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:15 smithi062 ceph-mon[76322]: pgmap v207: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:42:17.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:17 smithi189 ceph-mon[57022]: pgmap v208: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:42:17.784 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:17 smithi062 ceph-mon[76322]: pgmap v208: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:42:19.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:19 smithi189 ceph-mon[57022]: pgmap v209: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:42:19.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:19 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:42:19.784 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:19 smithi062 ceph-mon[76322]: pgmap v209: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:42:19.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:19 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:42:20.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:20.482+0000 7f7bad600700 1 -- 172.21.15.62:0/145924305 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ba81004e0 msgr2=0x7f7ba81008b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:42:20.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:20.482+0000 7f7bad600700 1 --2- 172.21.15.62:0/145924305 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ba81004e0 0x7f7ba81008b0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f7b98009910 tx=0x7f7b98009c20 comp rx=0 tx=0).stop 2024-04-20T21:42:20.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:20.483+0000 7f7bad600700 1 -- 172.21.15.62:0/145924305 shutdown_connections 2024-04-20T21:42:20.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:20.483+0000 7f7bad600700 1 --2- 172.21.15.62:0/145924305 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ba8100e80 0x7f7ba8104e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:42:20.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:20.483+0000 7f7bad600700 1 --2- 172.21.15.62:0/145924305 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ba81004e0 0x7f7ba81008b0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:42:20.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:20.483+0000 7f7bad600700 1 -- 172.21.15.62:0/145924305 >> 172.21.15.62:0/145924305 conn(0x7f7ba80fbda0 msgr2=0x7f7ba80fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:42:20.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:20.483+0000 7f7bad600700 1 -- 172.21.15.62:0/145924305 shutdown_connections 2024-04-20T21:42:20.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:20.483+0000 7f7bad600700 1 -- 172.21.15.62:0/145924305 wait complete. 2024-04-20T21:42:20.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:20.484+0000 7f7bad600700 1 Processor -- start 2024-04-20T21:42:20.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:20.484+0000 7f7bad600700 1 -- start start 2024-04-20T21:42:20.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:20.485+0000 7f7bad600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ba81004e0 0x7f7ba81980f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:42:20.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:20.485+0000 7f7bad600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ba8100e80 0x7f7ba8198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:42:20.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:20.485+0000 7f7bad600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ba8198c80 con 0x7f7ba8100e80 2024-04-20T21:42:20.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:20.485+0000 7f7bad600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ba8198dc0 con 0x7f7ba81004e0 2024-04-20T21:42:20.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:20.485+0000 7f7b9fe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ba8100e80 0x7f7ba8198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:42:20.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:20.485+0000 7f7b9fe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ba8100e80 0x7f7ba8198630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33196/0 (socket says 172.21.15.62:33196) 2024-04-20T21:42:20.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:20.485+0000 7f7b9fe00700 1 -- 172.21.15.62:0/2638817882 learned_addr learned my addr 172.21.15.62:0/2638817882 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:42:20.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:20.485+0000 7f7ba6a00700 1 --2- 172.21.15.62:0/2638817882 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ba81004e0 0x7f7ba81980f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:42:20.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:20.486+0000 7f7b9fe00700 1 -- 172.21.15.62:0/2638817882 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ba81004e0 msgr2=0x7f7ba81980f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:42:20.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:20.486+0000 7f7b9fe00700 1 --2- 172.21.15.62:0/2638817882 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ba81004e0 0x7f7ba81980f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:42:20.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:20.486+0000 7f7b9fe00700 1 -- 172.21.15.62:0/2638817882 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b980095f0 con 0x7f7ba8100e80 2024-04-20T21:42:20.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:20.486+0000 7f7b9fe00700 1 --2- 172.21.15.62:0/2638817882 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ba8100e80 0x7f7ba8198630 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f7b9000c960 tx=0x7f7b9000cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:42:20.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:20.486+0000 7f7b9f400700 1 -- 172.21.15.62:0/2638817882 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b90007a70 con 0x7f7ba8100e80 2024-04-20T21:42:20.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:20.486+0000 7f7b9f400700 1 -- 172.21.15.62:0/2638817882 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7b90007bd0 con 0x7f7ba8100e80 2024-04-20T21:42:20.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:20.486+0000 7f7bad600700 1 -- 172.21.15.62:0/2638817882 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ba819cc10 con 0x7f7ba8100e80 2024-04-20T21:42:20.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:20.487+0000 7f7b9f400700 1 -- 172.21.15.62:0/2638817882 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b90018630 con 0x7f7ba8100e80 2024-04-20T21:42:20.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:20.487+0000 7f7bad600700 1 -- 172.21.15.62:0/2638817882 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ba819d160 con 0x7f7ba8100e80 2024-04-20T21:42:20.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:20.488+0000 7f7bad600700 1 -- 172.21.15.62:0/2638817882 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ba81045d0 con 0x7f7ba8100e80 2024-04-20T21:42:20.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:20.489+0000 7f7b9f400700 1 -- 172.21.15.62:0/2638817882 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f7b9000f450 con 0x7f7ba8100e80 2024-04-20T21:42:20.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:20.490+0000 7f7b9f400700 1 --2- 172.21.15.62:0/2638817882 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7b8c06ecc0 0x7f7b8c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:42:20.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:20.491+0000 7f7b9f400700 1 -- 172.21.15.62:0/2638817882 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f7b9008ee90 con 0x7f7ba8100e80 2024-04-20T21:42:20.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:20.491+0000 7f7ba6a00700 1 --2- 172.21.15.62:0/2638817882 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7b8c06ecc0 0x7f7b8c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:42:20.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:20.492+0000 7f7ba6a00700 1 --2- 172.21.15.62:0/2638817882 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7b8c06ecc0 0x7f7b8c071170 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7b98006010 tx=0x7f7b9801a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:42:20.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:20.495+0000 7f7b9f400700 1 -- 172.21.15.62:0/2638817882 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f7b90057ba0 con 0x7f7ba8100e80 2024-04-20T21:42:20.784 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:20 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:42:20.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:20 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:42:20.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:20.797+0000 7f7bad600700 1 -- 172.21.15.62:0/2638817882 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7ba819d3e0 con 0x7f7b8c06ecc0 2024-04-20T21:42:20.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:20.798+0000 7f7b9f400700 1 -- 172.21.15.62:0/2638817882 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f7ba819d3e0 con 0x7f7b8c06ecc0 2024-04-20T21:42:20.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:20.801+0000 7f7bad600700 1 -- 172.21.15.62:0/2638817882 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7b8c06ecc0 msgr2=0x7f7b8c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:42:20.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:20.801+0000 7f7bad600700 1 --2- 172.21.15.62:0/2638817882 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7b8c06ecc0 0x7f7b8c071170 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7b98006010 tx=0x7f7b9801a040 comp rx=0 tx=0).stop 2024-04-20T21:42:20.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:20.801+0000 7f7bad600700 1 -- 172.21.15.62:0/2638817882 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ba8100e80 msgr2=0x7f7ba8198630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:42:20.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:20.801+0000 7f7bad600700 1 --2- 172.21.15.62:0/2638817882 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ba8100e80 0x7f7ba8198630 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f7b9000c960 tx=0x7f7b9000cc70 comp rx=0 tx=0).stop 2024-04-20T21:42:20.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:20.801+0000 7f7bad600700 1 -- 172.21.15.62:0/2638817882 shutdown_connections 2024-04-20T21:42:20.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:20.801+0000 7f7bad600700 1 --2- 172.21.15.62:0/2638817882 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7b8c06ecc0 0x7f7b8c071170 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:42:20.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:20.801+0000 7f7bad600700 1 --2- 172.21.15.62:0/2638817882 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ba81004e0 0x7f7ba81980f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:42:20.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:20.801+0000 7f7bad600700 1 --2- 172.21.15.62:0/2638817882 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ba8100e80 0x7f7ba8198630 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:42:20.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:20.801+0000 7f7bad600700 1 -- 172.21.15.62:0/2638817882 >> 172.21.15.62:0/2638817882 conn(0x7f7ba80fbda0 msgr2=0x7f7ba80fe160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:42:20.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:20.802+0000 7f7bad600700 1 -- 172.21.15.62:0/2638817882 shutdown_connections 2024-04-20T21:42:20.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:20.802+0000 7f7bad600700 1 -- 172.21.15.62:0/2638817882 wait complete. 2024-04-20T21:42:20.819 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T21:42:20.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:20.967+0000 7f3931a00700 1 -- 172.21.15.62:0/3159316095 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f392c0687a0 msgr2=0x7f392c068b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:42:20.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:20.967+0000 7f3931a00700 1 --2- 172.21.15.62:0/3159316095 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f392c0687a0 0x7f392c068b70 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f391c009910 tx=0x7f391c009c20 comp rx=0 tx=0).stop 2024-04-20T21:42:20.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:20.967+0000 7f3931a00700 1 -- 172.21.15.62:0/3159316095 shutdown_connections 2024-04-20T21:42:20.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:20.967+0000 7f3931a00700 1 --2- 172.21.15.62:0/3159316095 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f392c069140 0x7f392c104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:42:20.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:20.967+0000 7f3931a00700 1 --2- 172.21.15.62:0/3159316095 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f392c0687a0 0x7f392c068b70 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:42:20.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:20.967+0000 7f3931a00700 1 -- 172.21.15.62:0/3159316095 >> 172.21.15.62:0/3159316095 conn(0x7f392c075130 msgr2=0x7f392c075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:42:20.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:20.967+0000 7f3931a00700 1 -- 172.21.15.62:0/3159316095 shutdown_connections 2024-04-20T21:42:20.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:20.967+0000 7f3931a00700 1 -- 172.21.15.62:0/3159316095 wait complete. 2024-04-20T21:42:20.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:20.968+0000 7f3931a00700 1 Processor -- start 2024-04-20T21:42:20.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:20.969+0000 7f3931a00700 1 -- start start 2024-04-20T21:42:20.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:20.969+0000 7f3931a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f392c0687a0 0x7f392c1980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:42:20.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:20.969+0000 7f3931a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f392c069140 0x7f392c1985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:42:20.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:20.969+0000 7f3931a00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f392c198cd0 con 0x7f392c0687a0 2024-04-20T21:42:20.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:20.969+0000 7f3931a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f392c19ca60 con 0x7f392c069140 2024-04-20T21:42:20.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:20.969+0000 7f392aa00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f392c0687a0 0x7f392c1980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:42:20.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:20.970+0000 7f392aa00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f392c0687a0 0x7f392c1980b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33206/0 (socket says 172.21.15.62:33206) 2024-04-20T21:42:20.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:20.970+0000 7f392aa00700 1 -- 172.21.15.62:0/2262253358 learned_addr learned my addr 172.21.15.62:0/2262253358 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:42:20.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:20.970+0000 7f392a000700 1 --2- 172.21.15.62:0/2262253358 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f392c069140 0x7f392c1985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:42:20.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:20.970+0000 7f392aa00700 1 -- 172.21.15.62:0/2262253358 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f392c069140 msgr2=0x7f392c1985f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:42:20.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:20.970+0000 7f392aa00700 1 --2- 172.21.15.62:0/2262253358 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f392c069140 0x7f392c1985f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:42:20.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:20.970+0000 7f392aa00700 1 -- 172.21.15.62:0/2262253358 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f391c0095f0 con 0x7f392c0687a0 2024-04-20T21:42:20.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:20.970+0000 7f392aa00700 1 --2- 172.21.15.62:0/2262253358 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f392c0687a0 0x7f392c1980b0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f391c0092e0 tx=0x7f391c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:42:20.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:20.971+0000 7f3923400700 1 -- 172.21.15.62:0/2262253358 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f391c01d070 con 0x7f392c0687a0 2024-04-20T21:42:20.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:20.971+0000 7f3923400700 1 -- 172.21.15.62:0/2262253358 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f391c005e00 con 0x7f392c0687a0 2024-04-20T21:42:20.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:20.971+0000 7f3931a00700 1 -- 172.21.15.62:0/2262253358 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f392c19cce0 con 0x7f392c0687a0 2024-04-20T21:42:20.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:20.971+0000 7f3923400700 1 -- 172.21.15.62:0/2262253358 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f391c017400 con 0x7f392c0687a0 2024-04-20T21:42:20.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:20.971+0000 7f3931a00700 1 -- 172.21.15.62:0/2262253358 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f392c19d1d0 con 0x7f392c0687a0 2024-04-20T21:42:20.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:20.972+0000 7f3931a00700 1 -- 172.21.15.62:0/2262253358 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f392c04f2a0 con 0x7f392c0687a0 2024-04-20T21:42:20.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:20.973+0000 7f3923400700 1 -- 172.21.15.62:0/2262253358 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f391c00f460 con 0x7f392c0687a0 2024-04-20T21:42:20.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:20.974+0000 7f3923400700 1 --2- 172.21.15.62:0/2262253358 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f391006ecc0 0x7f3910071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:42:20.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:20.974+0000 7f3923400700 1 -- 172.21.15.62:0/2262253358 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f391c090390 con 0x7f392c0687a0 2024-04-20T21:42:20.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:20.975+0000 7f392a000700 1 --2- 172.21.15.62:0/2262253358 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f391006ecc0 0x7f3910071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:42:20.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:20.975+0000 7f392a000700 1 --2- 172.21.15.62:0/2262253358 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f391006ecc0 0x7f3910071170 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f3918005950 tx=0x7f39180058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:42:20.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:20.980+0000 7f3923400700 1 -- 172.21.15.62:0/2262253358 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f391c059140 con 0x7f392c0687a0 2024-04-20T21:42:20.986 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:20 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:42:20.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:20 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:42:21.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:21.241+0000 7f3931a00700 1 -- 172.21.15.62:0/2262253358 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f392c19d500 con 0x7f391006ecc0 2024-04-20T21:42:21.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:21.244+0000 7f3923400700 1 -- 172.21.15.62:0/2262253358 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f392c19d500 con 0x7f391006ecc0 2024-04-20T21:42:21.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:21.248+0000 7f3931a00700 1 -- 172.21.15.62:0/2262253358 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f391006ecc0 msgr2=0x7f3910071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:42:21.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:21.248+0000 7f3931a00700 1 --2- 172.21.15.62:0/2262253358 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f391006ecc0 0x7f3910071170 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f3918005950 tx=0x7f39180058e0 comp rx=0 tx=0).stop 2024-04-20T21:42:21.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:21.248+0000 7f3931a00700 1 -- 172.21.15.62:0/2262253358 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f392c0687a0 msgr2=0x7f392c1980b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:42:21.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:21.248+0000 7f3931a00700 1 --2- 172.21.15.62:0/2262253358 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f392c0687a0 0x7f392c1980b0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f391c0092e0 tx=0x7f391c00be20 comp rx=0 tx=0).stop 2024-04-20T21:42:21.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:21.248+0000 7f3931a00700 1 -- 172.21.15.62:0/2262253358 shutdown_connections 2024-04-20T21:42:21.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:21.248+0000 7f3931a00700 1 --2- 172.21.15.62:0/2262253358 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f391006ecc0 0x7f3910071170 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:42:21.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:21.248+0000 7f3931a00700 1 --2- 172.21.15.62:0/2262253358 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f392c0687a0 0x7f392c1980b0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:42:21.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:21.248+0000 7f3931a00700 1 --2- 172.21.15.62:0/2262253358 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f392c069140 0x7f392c1985f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:42:21.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:21.248+0000 7f3931a00700 1 -- 172.21.15.62:0/2262253358 >> 172.21.15.62:0/2262253358 conn(0x7f392c075130 msgr2=0x7f392c101e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:42:21.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:21.248+0000 7f3931a00700 1 -- 172.21.15.62:0/2262253358 shutdown_connections 2024-04-20T21:42:21.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:21.248+0000 7f3931a00700 1 -- 172.21.15.62:0/2262253358 wait complete. 2024-04-20T21:42:21.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:21.415+0000 7fa56a600700 1 -- 172.21.15.62:0/2283845420 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5640730f0 msgr2=0x7fa5640734c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:42:21.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:21.415+0000 7fa56a600700 1 --2- 172.21.15.62:0/2283845420 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5640730f0 0x7fa5640734c0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fa558009910 tx=0x7fa558009c20 comp rx=0 tx=0).stop 2024-04-20T21:42:21.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:21.415+0000 7fa56a600700 1 -- 172.21.15.62:0/2283845420 shutdown_connections 2024-04-20T21:42:21.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:21.415+0000 7fa56a600700 1 --2- 172.21.15.62:0/2283845420 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa564073a00 0x7fa564110b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:42:21.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:21.415+0000 7fa56a600700 1 --2- 172.21.15.62:0/2283845420 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5640730f0 0x7fa5640734c0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:42:21.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:21.415+0000 7fa56a600700 1 -- 172.21.15.62:0/2283845420 >> 172.21.15.62:0/2283845420 conn(0x7fa5640fbb10 msgr2=0x7fa5640fdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:42:21.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:21.415+0000 7fa56a600700 1 -- 172.21.15.62:0/2283845420 shutdown_connections 2024-04-20T21:42:21.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:21.415+0000 7fa56a600700 1 -- 172.21.15.62:0/2283845420 wait complete. 2024-04-20T21:42:21.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:21.416+0000 7fa56a600700 1 Processor -- start 2024-04-20T21:42:21.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:21.417+0000 7fa56a600700 1 -- start start 2024-04-20T21:42:21.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:21.417+0000 7fa56a600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5640730f0 0x7fa56419c160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:42:21.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:21.417+0000 7fa56a600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa564073a00 0x7fa56419c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:42:21.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:21.417+0000 7fa56a600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa56419cd80 con 0x7fa5640730f0 2024-04-20T21:42:21.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:21.417+0000 7fa56a600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5641a0b10 con 0x7fa564073a00 2024-04-20T21:42:21.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:21.417+0000 7fa569200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5640730f0 0x7fa56419c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:42:21.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:21.417+0000 7fa569200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5640730f0 0x7fa56419c160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:39858/0 (socket says 172.21.15.62:39858) 2024-04-20T21:42:21.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:21.417+0000 7fa569200700 1 -- 172.21.15.62:0/811552126 learned_addr learned my addr 172.21.15.62:0/811552126 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:42:21.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:21.418+0000 7fa563e00700 1 --2- 172.21.15.62:0/811552126 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa564073a00 0x7fa56419c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:42:21.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:21.418+0000 7fa569200700 1 -- 172.21.15.62:0/811552126 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa564073a00 msgr2=0x7fa56419c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:42:21.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:21.418+0000 7fa569200700 1 --2- 172.21.15.62:0/811552126 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa564073a00 0x7fa56419c6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:42:21.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:21.418+0000 7fa569200700 1 -- 172.21.15.62:0/811552126 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5580095f0 con 0x7fa5640730f0 2024-04-20T21:42:21.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:21.418+0000 7fa569200700 1 --2- 172.21.15.62:0/811552126 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5640730f0 0x7fa56419c160 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fa558006010 tx=0x7fa55800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:42:21.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:21.419+0000 7fa561600700 1 -- 172.21.15.62:0/811552126 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa55801d070 con 0x7fa5640730f0 2024-04-20T21:42:21.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:21.419+0000 7fa561600700 1 -- 172.21.15.62:0/811552126 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa55800f460 con 0x7fa5640730f0 2024-04-20T21:42:21.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:21.419+0000 7fa561600700 1 -- 172.21.15.62:0/811552126 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa558017630 con 0x7fa5640730f0 2024-04-20T21:42:21.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:21.419+0000 7fa56a600700 1 -- 172.21.15.62:0/811552126 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5641a0df0 con 0x7fa5640730f0 2024-04-20T21:42:21.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:21.419+0000 7fa56a600700 1 -- 172.21.15.62:0/811552126 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5641a1340 con 0x7fa5640730f0 2024-04-20T21:42:21.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:21.420+0000 7fa56a600700 1 -- 172.21.15.62:0/811552126 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa56410e280 con 0x7fa5640730f0 2024-04-20T21:42:21.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:21.421+0000 7fa561600700 1 -- 172.21.15.62:0/811552126 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fa55800f5d0 con 0x7fa5640730f0 2024-04-20T21:42:21.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:21.422+0000 7fa561600700 1 --2- 172.21.15.62:0/811552126 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa54c06ecc0 0x7fa54c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:42:21.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:21.422+0000 7fa561600700 1 -- 172.21.15.62:0/811552126 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fa558090230 con 0x7fa5640730f0 2024-04-20T21:42:21.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:21.423+0000 7fa563e00700 1 --2- 172.21.15.62:0/811552126 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa54c06ecc0 0x7fa54c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:42:21.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:21.423+0000 7fa563e00700 1 --2- 172.21.15.62:0/811552126 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa54c06ecc0 0x7fa54c071170 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fa550005950 tx=0x7fa5500058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:42:21.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:21.428+0000 7fa561600700 1 -- 172.21.15.62:0/811552126 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fa558059060 con 0x7fa5640730f0 2024-04-20T21:42:21.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:21.682+0000 7fa56a600700 1 -- 172.21.15.62:0/811552126 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fa5641a15d0 con 0x7fa54c06ecc0 2024-04-20T21:42:21.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:21.696+0000 7fa561600700 1 -- 172.21.15.62:0/811552126 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fa5641a15d0 con 0x7fa54c06ecc0 2024-04-20T21:42:21.696 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T21:42:21.696 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (8m) 2s ago 22m 14.2M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T21:42:21.696 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (22m) 2s ago 22m 9479k - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T21:42:21.696 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (21m) 30s ago 21m 8942k - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T21:42:21.696 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (4m) 2s ago 22m 6660k - 19.0.0-3133-ga8c84f59 76bc6b91b726 2825d5ba11c6 2024-04-20T21:42:21.696 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (4m) 30s ago 20m 6665k - 19.0.0-3133-ga8c84f59 76bc6b91b726 20f746fa1bba 2024-04-20T21:42:21.696 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (7m) 2s ago 21m 73.2M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T21:42:21.696 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (14m) 2s ago 14m 11.5M - 18.2.0 dc2bc1663786 5e6460aa6159 2024-04-20T21:42:21.696 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (10s) 2s ago 14m 11.0M - 19.0.0-3133-ga8c84f59 76bc6b91b726 cd793b7ee98e 2024-04-20T21:42:21.696 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (14m) 30s ago 14m 117M - 18.2.0 dc2bc1663786 fb2a8a223db8 2024-04-20T21:42:21.696 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (14m) 30s ago 14m 55.6M - 18.2.0 dc2bc1663786 f2eec2ee8951 2024-04-20T21:42:21.696 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (11m) 2s ago 23m 510M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T21:42:21.696 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (9m) 30s ago 20m 452M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T21:42:21.696 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (5m) 2s ago 24m 55.0M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 019eb1479485 2024-04-20T21:42:21.696 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (5m) 30s ago 20m 39.8M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 55aac560aab6 2024-04-20T21:42:21.697 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (9m) 2s ago 22m 7713k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T21:42:21.697 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (9m) 30s ago 20m 8304k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T21:42:21.697 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (3m) 2s ago 19m 87.5M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 ee2bb901f26e 2024-04-20T21:42:21.697 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (3m) 2s ago 19m 90.7M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 9075152d4162 2024-04-20T21:42:21.697 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (2m) 2s ago 18m 74.2M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 2d4f3e38dc0e 2024-04-20T21:42:21.697 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (2m) 30s ago 17m 87.6M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 8f97b17ae0eb 2024-04-20T21:42:21.697 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (79s) 30s ago 17m 90.7M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 653d8079967d 2024-04-20T21:42:21.697 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (36s) 30s ago 16m 11.5M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 596dded14c12 2024-04-20T21:42:21.697 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (8m) 2s ago 21m 73.6M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T21:42:21.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:21.699+0000 7fa56a600700 1 -- 172.21.15.62:0/811552126 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa54c06ecc0 msgr2=0x7fa54c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:42:21.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:21.699+0000 7fa56a600700 1 --2- 172.21.15.62:0/811552126 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa54c06ecc0 0x7fa54c071170 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fa550005950 tx=0x7fa5500058e0 comp rx=0 tx=0).stop 2024-04-20T21:42:21.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:21.699+0000 7fa56a600700 1 -- 172.21.15.62:0/811552126 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5640730f0 msgr2=0x7fa56419c160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:42:21.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:21.699+0000 7fa56a600700 1 --2- 172.21.15.62:0/811552126 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5640730f0 0x7fa56419c160 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fa558006010 tx=0x7fa55800be20 comp rx=0 tx=0).stop 2024-04-20T21:42:21.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:21.699+0000 7fa56a600700 1 -- 172.21.15.62:0/811552126 shutdown_connections 2024-04-20T21:42:21.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:21.699+0000 7fa56a600700 1 --2- 172.21.15.62:0/811552126 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa54c06ecc0 0x7fa54c071170 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:42:21.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:21.699+0000 7fa56a600700 1 --2- 172.21.15.62:0/811552126 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5640730f0 0x7fa56419c160 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:42:21.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:21.699+0000 7fa56a600700 1 --2- 172.21.15.62:0/811552126 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa564073a00 0x7fa56419c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:42:21.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:21.699+0000 7fa56a600700 1 -- 172.21.15.62:0/811552126 >> 172.21.15.62:0/811552126 conn(0x7fa5640fbb10 msgr2=0x7fa564102610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:42:21.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:21.700+0000 7fa56a600700 1 -- 172.21.15.62:0/811552126 shutdown_connections 2024-04-20T21:42:21.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:21.700+0000 7fa56a600700 1 -- 172.21.15.62:0/811552126 wait complete. 2024-04-20T21:42:21.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:21 smithi062 ceph-mon[76322]: pgmap v210: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:42:21.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:21 smithi062 ceph-mon[76322]: from='client.34448 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:42:21.851 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:21 smithi189 ceph-mon[57022]: pgmap v210: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:42:21.851 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:21 smithi189 ceph-mon[57022]: from='client.34448 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:42:21.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:21.862+0000 7f26abc00700 1 -- 172.21.15.62:0/559767416 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f26a40730f0 msgr2=0x7f26a40734c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:42:21.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:21.862+0000 7f26abc00700 1 --2- 172.21.15.62:0/559767416 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f26a40730f0 0x7f26a40734c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f2698009910 tx=0x7f2698009c20 comp rx=0 tx=0).stop 2024-04-20T21:42:21.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:21.862+0000 7f26abc00700 1 -- 172.21.15.62:0/559767416 shutdown_connections 2024-04-20T21:42:21.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:21.862+0000 7f26abc00700 1 --2- 172.21.15.62:0/559767416 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26a4073a00 0x7f26a410c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:42:21.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:21.862+0000 7f26abc00700 1 --2- 172.21.15.62:0/559767416 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f26a40730f0 0x7f26a40734c0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:42:21.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:21.862+0000 7f26abc00700 1 -- 172.21.15.62:0/559767416 >> 172.21.15.62:0/559767416 conn(0x7f26a40fbda0 msgr2=0x7f26a40fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:42:21.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:21.862+0000 7f26abc00700 1 -- 172.21.15.62:0/559767416 shutdown_connections 2024-04-20T21:42:21.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:21.862+0000 7f26abc00700 1 -- 172.21.15.62:0/559767416 wait complete. 2024-04-20T21:42:21.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:21.863+0000 7f26abc00700 1 Processor -- start 2024-04-20T21:42:21.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:21.863+0000 7f26abc00700 1 -- start start 2024-04-20T21:42:21.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:21.864+0000 7f26abc00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f26a40730f0 0x7f26a4197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:42:21.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:21.864+0000 7f26abc00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26a4073a00 0x7f26a4198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:42:21.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:21.864+0000 7f26abc00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26a419c970 con 0x7f26a40730f0 2024-04-20T21:42:21.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:21.864+0000 7f26abc00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26a419cae0 con 0x7f26a4073a00 2024-04-20T21:42:21.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:21.864+0000 7f26a9200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f26a40730f0 0x7f26a4197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:42:21.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:21.864+0000 7f26a9200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f26a40730f0 0x7f26a4197fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:39872/0 (socket says 172.21.15.62:39872) 2024-04-20T21:42:21.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:21.864+0000 7f26a9200700 1 -- 172.21.15.62:0/354137186 learned_addr learned my addr 172.21.15.62:0/354137186 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:42:21.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:21.865+0000 7f26a3e00700 1 --2- 172.21.15.62:0/354137186 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26a4073a00 0x7f26a4198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:42:21.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:21.865+0000 7f26a9200700 1 -- 172.21.15.62:0/354137186 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26a4073a00 msgr2=0x7f26a4198630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:42:21.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:21.865+0000 7f26a9200700 1 --2- 172.21.15.62:0/354137186 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26a4073a00 0x7f26a4198630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:42:21.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:21.865+0000 7f26a9200700 1 -- 172.21.15.62:0/354137186 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26980095f0 con 0x7f26a40730f0 2024-04-20T21:42:21.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:21.865+0000 7f26a9200700 1 --2- 172.21.15.62:0/354137186 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f26a40730f0 0x7f26a4197fc0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f269800b5c0 tx=0x7f269800be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:42:21.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:21.866+0000 7f26a1600700 1 -- 172.21.15.62:0/354137186 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f269801d070 con 0x7f26a40730f0 2024-04-20T21:42:21.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:21.866+0000 7f26abc00700 1 -- 172.21.15.62:0/354137186 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26a419cdc0 con 0x7f26a40730f0 2024-04-20T21:42:21.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:21.866+0000 7f26a1600700 1 -- 172.21.15.62:0/354137186 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f269800f460 con 0x7f26a40730f0 2024-04-20T21:42:21.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:21.866+0000 7f26a1600700 1 -- 172.21.15.62:0/354137186 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2698017630 con 0x7f26a40730f0 2024-04-20T21:42:21.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:21.866+0000 7f26abc00700 1 -- 172.21.15.62:0/354137186 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26a419d310 con 0x7f26a40730f0 2024-04-20T21:42:21.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:21.867+0000 7f26abc00700 1 -- 172.21.15.62:0/354137186 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26a4109c80 con 0x7f26a40730f0 2024-04-20T21:42:21.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:21.868+0000 7f26a1600700 1 -- 172.21.15.62:0/354137186 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f269800f5d0 con 0x7f26a40730f0 2024-04-20T21:42:21.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:21.870+0000 7f26a1600700 1 --2- 172.21.15.62:0/354137186 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f268c06ecc0 0x7f268c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:42:21.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:21.870+0000 7f26a1600700 1 -- 172.21.15.62:0/354137186 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f2698090580 con 0x7f26a40730f0 2024-04-20T21:42:21.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:21.870+0000 7f26a3e00700 1 --2- 172.21.15.62:0/354137186 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f268c06ecc0 0x7f268c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:42:21.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:21.871+0000 7f26a3e00700 1 --2- 172.21.15.62:0/354137186 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f268c06ecc0 0x7f268c071170 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f2690005950 tx=0x7f2690005cf0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:42:21.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:21.874+0000 7f26a1600700 1 -- 172.21.15.62:0/354137186 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f2698059450 con 0x7f26a40730f0 2024-04-20T21:42:22.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:22.210+0000 7f26abc00700 1 -- 172.21.15.62:0/354137186 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f26a404f2a0 con 0x7f26a40730f0 2024-04-20T21:42:22.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:22.211+0000 7f26a1600700 1 -- 172.21.15.62:0/354137186 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7f269805cb10 con 0x7f26a40730f0 2024-04-20T21:42:22.210 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:42:22.210 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T21:42:22.210 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:42:22.210 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:42:22.210 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T21:42:22.210 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:42:22.210 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:42:22.211 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T21:42:22.211 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 6 2024-04-20T21:42:22.211 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:42:22.211 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T21:42:22.211 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 3, 2024-04-20T21:42:22.211 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 1 2024-04-20T21:42:22.211 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:42:22.211 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T21:42:22.211 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 3, 2024-04-20T21:42:22.211 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 11 2024-04-20T21:42:22.211 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T21:42:22.211 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:42:22.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:22.214+0000 7f26abc00700 1 -- 172.21.15.62:0/354137186 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f268c06ecc0 msgr2=0x7f268c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:42:22.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:22.214+0000 7f26abc00700 1 --2- 172.21.15.62:0/354137186 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f268c06ecc0 0x7f268c071170 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f2690005950 tx=0x7f2690005cf0 comp rx=0 tx=0).stop 2024-04-20T21:42:22.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:22.214+0000 7f26abc00700 1 -- 172.21.15.62:0/354137186 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f26a40730f0 msgr2=0x7f26a4197fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:42:22.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:22.214+0000 7f26abc00700 1 --2- 172.21.15.62:0/354137186 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f26a40730f0 0x7f26a4197fc0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f269800b5c0 tx=0x7f269800be00 comp rx=0 tx=0).stop 2024-04-20T21:42:22.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:22.214+0000 7f26abc00700 1 -- 172.21.15.62:0/354137186 shutdown_connections 2024-04-20T21:42:22.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:22.214+0000 7f26abc00700 1 --2- 172.21.15.62:0/354137186 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f268c06ecc0 0x7f268c071170 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:42:22.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:22.214+0000 7f26abc00700 1 --2- 172.21.15.62:0/354137186 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f26a40730f0 0x7f26a4197fc0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:42:22.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:22.214+0000 7f26abc00700 1 --2- 172.21.15.62:0/354137186 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26a4073a00 0x7f26a4198630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:42:22.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:22.214+0000 7f26abc00700 1 -- 172.21.15.62:0/354137186 >> 172.21.15.62:0/354137186 conn(0x7f26a40fbda0 msgr2=0x7f26a4106dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:42:22.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:22.214+0000 7f26abc00700 1 -- 172.21.15.62:0/354137186 shutdown_connections 2024-04-20T21:42:22.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:22.214+0000 7f26abc00700 1 -- 172.21.15.62:0/354137186 wait complete. 2024-04-20T21:42:22.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:22.376+0000 7fcab0400700 1 -- 172.21.15.62:0/3639111911 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcaa8069140 msgr2=0x7fcaa8104eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:42:22.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:22.376+0000 7fcab0400700 1 --2- 172.21.15.62:0/3639111911 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcaa8069140 0x7fcaa8104eb0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fca94009910 tx=0x7fca94009c20 comp rx=0 tx=0).stop 2024-04-20T21:42:22.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:22.376+0000 7fcab0400700 1 -- 172.21.15.62:0/3639111911 shutdown_connections 2024-04-20T21:42:22.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:22.376+0000 7fcab0400700 1 --2- 172.21.15.62:0/3639111911 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcaa8069140 0x7fcaa8104eb0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:42:22.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:22.376+0000 7fcab0400700 1 --2- 172.21.15.62:0/3639111911 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcaa80687a0 0x7fcaa8068b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:42:22.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:22.376+0000 7fcab0400700 1 -- 172.21.15.62:0/3639111911 >> 172.21.15.62:0/3639111911 conn(0x7fcaa8075130 msgr2=0x7fcaa8075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:42:22.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:22.376+0000 7fcab0400700 1 -- 172.21.15.62:0/3639111911 shutdown_connections 2024-04-20T21:42:22.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:22.376+0000 7fcab0400700 1 -- 172.21.15.62:0/3639111911 wait complete. 2024-04-20T21:42:22.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:22.377+0000 7fcab0400700 1 Processor -- start 2024-04-20T21:42:22.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:22.377+0000 7fcab0400700 1 -- start start 2024-04-20T21:42:22.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:22.378+0000 7fcab0400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcaa80687a0 0x7fcaa8198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:42:22.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:22.378+0000 7fcab0400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcaa8069140 0x7fcaa8198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:42:22.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:22.378+0000 7fcab0400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaa8198c70 con 0x7fcaa8069140 2024-04-20T21:42:22.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:22.378+0000 7fcab0400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaa819ca00 con 0x7fcaa80687a0 2024-04-20T21:42:22.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:22.378+0000 7fcaad000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcaa8069140 0x7fcaa8198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:42:22.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:22.378+0000 7fcaad000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcaa8069140 0x7fcaa8198590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:39894/0 (socket says 172.21.15.62:39894) 2024-04-20T21:42:22.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:22.378+0000 7fcaad000700 1 -- 172.21.15.62:0/4134449184 learned_addr learned my addr 172.21.15.62:0/4134449184 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:42:22.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:22.378+0000 7fcaada00700 1 --2- 172.21.15.62:0/4134449184 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcaa80687a0 0x7fcaa8198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:42:22.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:22.379+0000 7fcaad000700 1 -- 172.21.15.62:0/4134449184 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcaa80687a0 msgr2=0x7fcaa8198050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:42:22.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:22.379+0000 7fcaad000700 1 --2- 172.21.15.62:0/4134449184 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcaa80687a0 0x7fcaa8198050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:42:22.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:22.379+0000 7fcaad000700 1 -- 172.21.15.62:0/4134449184 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca940095f0 con 0x7fcaa8069140 2024-04-20T21:42:22.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:22.379+0000 7fcaad000700 1 --2- 172.21.15.62:0/4134449184 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcaa8069140 0x7fcaa8198590 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fca940092e0 tx=0x7fca9400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:42:22.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:22.379+0000 7fcaa6000700 1 -- 172.21.15.62:0/4134449184 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fca9401d070 con 0x7fcaa8069140 2024-04-20T21:42:22.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:22.379+0000 7fcaa6000700 1 -- 172.21.15.62:0/4134449184 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fca94005dc0 con 0x7fcaa8069140 2024-04-20T21:42:22.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:22.379+0000 7fcaa6000700 1 -- 172.21.15.62:0/4134449184 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fca94017400 con 0x7fcaa8069140 2024-04-20T21:42:22.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:22.379+0000 7fcab0400700 1 -- 172.21.15.62:0/4134449184 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcaa819cc80 con 0x7fcaa8069140 2024-04-20T21:42:22.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:22.380+0000 7fcab0400700 1 -- 172.21.15.62:0/4134449184 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcaa819d170 con 0x7fcaa8069140 2024-04-20T21:42:22.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:22.381+0000 7fcab0400700 1 -- 172.21.15.62:0/4134449184 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcaa804f2a0 con 0x7fcaa8069140 2024-04-20T21:42:22.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:22.383+0000 7fcaa6000700 1 -- 172.21.15.62:0/4134449184 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fca9400f460 con 0x7fcaa8069140 2024-04-20T21:42:22.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:22.383+0000 7fcaa6000700 1 --2- 172.21.15.62:0/4134449184 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fca9006ecc0 0x7fca90071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:42:22.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:22.384+0000 7fcaa6000700 1 -- 172.21.15.62:0/4134449184 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fca94090270 con 0x7fcaa8069140 2024-04-20T21:42:22.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:22.384+0000 7fcaada00700 1 --2- 172.21.15.62:0/4134449184 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fca9006ecc0 0x7fca90071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:42:22.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:22.385+0000 7fcaada00700 1 --2- 172.21.15.62:0/4134449184 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fca9006ecc0 0x7fca90071170 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fca9c005950 tx=0x7fca9c0058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:42:22.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:22.391+0000 7fcaa6000700 1 -- 172.21.15.62:0/4134449184 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fca940590a0 con 0x7fcaa8069140 2024-04-20T21:42:22.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:22.696+0000 7fcab0400700 1 -- 172.21.15.62:0/4134449184 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fcaa81993b0 con 0x7fcaa8069140 2024-04-20T21:42:22.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:22.697+0000 7fcaa6000700 1 -- 172.21.15.62:0/4134449184 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 23 v23) v1 ==== 76+0+1886 (secure 0 0 0) 0x7fca9405c760 con 0x7fcaa8069140 2024-04-20T21:42:22.697 INFO:teuthology.orchestra.run.smithi062.stdout:e23 2024-04-20T21:42:22.697 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T21:42:22.697 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T21:42:22.697 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T21:42:22.697 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:42:22.697 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T21:42:22.697 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T21:42:22.697 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 20 2024-04-20T21:42:22.697 INFO:teuthology.orchestra.run.smithi062.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-20T21:42:22.697 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T21:42:22.698 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T21:42:01.559061+0000 2024-04-20T21:42:22.698 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T21:42:22.698 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T21:42:22.698 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T21:42:22.698 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T21:42:22.698 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T21:42:22.698 INFO:teuthology.orchestra.run.smithi062.stdout:max_xattr_size 65536 2024-04-20T21:42:22.698 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T21:42:22.698 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T21:42:22.698 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 81 2024-04-20T21:42:22.699 INFO:teuthology.orchestra.run.smithi062.stdout:snap_rstat 1 2024-04-20T21:42:22.699 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T21:42:22.699 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 2 2024-04-20T21:42:22.699 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1 2024-04-20T21:42:22.699 INFO:teuthology.orchestra.run.smithi062.stdout:up {} 2024-04-20T21:42:22.699 INFO:teuthology.orchestra.run.smithi062.stdout:failed 0,1 2024-04-20T21:42:22.699 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T21:42:22.699 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T21:42:22.699 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T21:42:22.699 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T21:42:22.699 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data enabled 2024-04-20T21:42:22.699 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T21:42:22.699 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T21:42:22.699 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 1 2024-04-20T21:42:22.699 INFO:teuthology.orchestra.run.smithi062.stdout:qdb_cluster leader: 0 members: 2024-04-20T21:42:22.699 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:42:22.700 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:42:22.700 INFO:teuthology.orchestra.run.smithi062.stdout:Standby daemons: 2024-04-20T21:42:22.700 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:42:22.700 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ozizwi{-1:34430} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6828/4253521164,v1:172.21.15.62:6829/4253521164] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:42:22.700 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ryqblb{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6826/4113916205,v1:172.21.15.62:6827/4113916205] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T21:42:22.700 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.xbgspw{-1:44285} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6826/3996040526,v1:172.21.15.189:6827/3996040526] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:42:22.700 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.dvimer{-1:44289} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6824/1743111843,v1:172.21.15.189:6825/1743111843] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:42:22.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:22.700+0000 7fcab0400700 1 -- 172.21.15.62:0/4134449184 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fca9006ecc0 msgr2=0x7fca90071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:42:22.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:22.700+0000 7fcab0400700 1 --2- 172.21.15.62:0/4134449184 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fca9006ecc0 0x7fca90071170 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fca9c005950 tx=0x7fca9c0058e0 comp rx=0 tx=0).stop 2024-04-20T21:42:22.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:22.700+0000 7fcab0400700 1 -- 172.21.15.62:0/4134449184 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcaa8069140 msgr2=0x7fcaa8198590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:42:22.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:22.700+0000 7fcab0400700 1 --2- 172.21.15.62:0/4134449184 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcaa8069140 0x7fcaa8198590 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fca940092e0 tx=0x7fca9400be20 comp rx=0 tx=0).stop 2024-04-20T21:42:22.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:22.700+0000 7fcab0400700 1 -- 172.21.15.62:0/4134449184 shutdown_connections 2024-04-20T21:42:22.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:22.700+0000 7fcab0400700 1 --2- 172.21.15.62:0/4134449184 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fca9006ecc0 0x7fca90071170 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:42:22.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:22.700+0000 7fcab0400700 1 --2- 172.21.15.62:0/4134449184 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcaa80687a0 0x7fcaa8198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:42:22.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:22.700+0000 7fcab0400700 1 --2- 172.21.15.62:0/4134449184 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcaa8069140 0x7fcaa8198590 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:42:22.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:22.700+0000 7fcab0400700 1 -- 172.21.15.62:0/4134449184 >> 172.21.15.62:0/4134449184 conn(0x7fcaa8075130 msgr2=0x7fcaa8101560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:42:22.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:22.701+0000 7fcab0400700 1 -- 172.21.15.62:0/4134449184 shutdown_connections 2024-04-20T21:42:22.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:22.701+0000 7fcab0400700 1 -- 172.21.15.62:0/4134449184 wait complete. 2024-04-20T21:42:22.702 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 23 2024-04-20T21:42:22.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:22 smithi062 ceph-mon[76322]: from='client.34452 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:42:22.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:22 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:42:22.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:22 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:42:22.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:22 smithi062 ceph-mon[76322]: from='client.34456 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:42:22.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:22 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/354137186' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:42:22.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:22.851+0000 7fe54b800700 1 -- 172.21.15.62:0/3635890709 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe5441026a0 msgr2=0x7fe544102a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:42:22.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:22.851+0000 7fe54b800700 1 --2- 172.21.15.62:0/3635890709 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe5441026a0 0x7fe544102a70 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fe538009910 tx=0x7fe538009c20 comp rx=0 tx=0).stop 2024-04-20T21:42:22.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:22.852+0000 7fe54b800700 1 -- 172.21.15.62:0/3635890709 shutdown_connections 2024-04-20T21:42:22.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:22.852+0000 7fe54b800700 1 --2- 172.21.15.62:0/3635890709 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe544103040 0x7fe544107010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:42:22.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:22.852+0000 7fe54b800700 1 --2- 172.21.15.62:0/3635890709 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe5441026a0 0x7fe544102a70 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:42:22.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:22.852+0000 7fe54b800700 1 -- 172.21.15.62:0/3635890709 >> 172.21.15.62:0/3635890709 conn(0x7fe5440fdf40 msgr2=0x7fe544100350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:42:22.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:22.852+0000 7fe54b800700 1 -- 172.21.15.62:0/3635890709 shutdown_connections 2024-04-20T21:42:22.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:22.852+0000 7fe54b800700 1 -- 172.21.15.62:0/3635890709 wait complete. 2024-04-20T21:42:22.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:22.853+0000 7fe54b800700 1 Processor -- start 2024-04-20T21:42:22.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:22.853+0000 7fe54b800700 1 -- start start 2024-04-20T21:42:22.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:22.853+0000 7fe54b800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe5441026a0 0x7fe544198130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:42:22.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:22.853+0000 7fe54b800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe544103040 0x7fe544198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:42:22.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:22.853+0000 7fe54b800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe544198d50 con 0x7fe544103040 2024-04-20T21:42:22.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:22.853+0000 7fe54b800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe54419cae0 con 0x7fe5441026a0 2024-04-20T21:42:22.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:22.854+0000 7fe543e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe544103040 0x7fe544198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:42:22.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:22.854+0000 7fe543e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe544103040 0x7fe544198670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:39912/0 (socket says 172.21.15.62:39912) 2024-04-20T21:42:22.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:22.854+0000 7fe543e00700 1 -- 172.21.15.62:0/1446133840 learned_addr learned my addr 172.21.15.62:0/1446133840 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:42:22.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:22.854+0000 7fe548e00700 1 --2- 172.21.15.62:0/1446133840 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe5441026a0 0x7fe544198130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:42:22.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:22.854+0000 7fe543e00700 1 -- 172.21.15.62:0/1446133840 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe5441026a0 msgr2=0x7fe544198130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:42:22.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:22.854+0000 7fe543e00700 1 --2- 172.21.15.62:0/1446133840 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe5441026a0 0x7fe544198130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:42:22.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:22.854+0000 7fe543e00700 1 -- 172.21.15.62:0/1446133840 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5380095f0 con 0x7fe544103040 2024-04-20T21:42:22.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:22.854+0000 7fe548e00700 1 --2- 172.21.15.62:0/1446133840 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe5441026a0 0x7fe544198130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T21:42:22.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:22.854+0000 7fe543e00700 1 --2- 172.21.15.62:0/1446133840 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe544103040 0x7fe544198670 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fe53000d8d0 tx=0x7fe53000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:42:22.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:22.855+0000 7fe541600700 1 -- 172.21.15.62:0/1446133840 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe5300041f0 con 0x7fe544103040 2024-04-20T21:42:22.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:22.855+0000 7fe541600700 1 -- 172.21.15.62:0/1446133840 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe530007620 con 0x7fe544103040 2024-04-20T21:42:22.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:22.855+0000 7fe54b800700 1 -- 172.21.15.62:0/1446133840 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe54419cdc0 con 0x7fe544103040 2024-04-20T21:42:22.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:22.855+0000 7fe541600700 1 -- 172.21.15.62:0/1446133840 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe530003c50 con 0x7fe544103040 2024-04-20T21:42:22.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:22.855+0000 7fe54b800700 1 -- 172.21.15.62:0/1446133840 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe54419d310 con 0x7fe544103040 2024-04-20T21:42:22.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:22.856+0000 7fe54b800700 1 -- 172.21.15.62:0/1446133840 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe54410a9b0 con 0x7fe544103040 2024-04-20T21:42:22.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:22.857+0000 7fe541600700 1 -- 172.21.15.62:0/1446133840 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe530004350 con 0x7fe544103040 2024-04-20T21:42:22.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:22.858+0000 7fe541600700 1 --2- 172.21.15.62:0/1446133840 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe52c06ecc0 0x7fe52c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:42:22.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:22.858+0000 7fe541600700 1 -- 172.21.15.62:0/1446133840 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fe53008ea00 con 0x7fe544103040 2024-04-20T21:42:22.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:22.858+0000 7fe548e00700 1 --2- 172.21.15.62:0/1446133840 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe52c06ecc0 0x7fe52c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:42:22.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:22.859+0000 7fe548e00700 1 --2- 172.21.15.62:0/1446133840 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe52c06ecc0 0x7fe52c071170 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fe538006010 tx=0x7fe53800b540 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:42:22.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:22.863+0000 7fe541600700 1 -- 172.21.15.62:0/1446133840 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fe5300577b0 con 0x7fe544103040 2024-04-20T21:42:22.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:22 smithi189 ceph-mon[57022]: from='client.34452 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:42:22.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:22 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:42:22.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:22 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:42:22.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:22 smithi189 ceph-mon[57022]: from='client.34456 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:42:22.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:22 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/354137186' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:42:23.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:23.113+0000 7fe54b800700 1 -- 172.21.15.62:0/1446133840 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe54419d530 con 0x7fe52c06ecc0 2024-04-20T21:42:23.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:23.115+0000 7fe541600700 1 -- 172.21.15.62:0/1446133840 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fe54419d530 con 0x7fe52c06ecc0 2024-04-20T21:42:23.115 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:42:23.115 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T21:42:23.115 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T21:42:23.115 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-20T21:42:23.115 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T21:42:23.115 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr", 2024-04-20T21:42:23.115 INFO:teuthology.orchestra.run.smithi062.stdout: "crash", 2024-04-20T21:42:23.115 INFO:teuthology.orchestra.run.smithi062.stdout: "mon", 2024-04-20T21:42:23.116 INFO:teuthology.orchestra.run.smithi062.stdout: "osd" 2024-04-20T21:42:23.116 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T21:42:23.116 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "13/23 daemons upgraded", 2024-04-20T21:42:23.116 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading mds daemons", 2024-04-20T21:42:23.116 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T21:42:23.116 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:42:23.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:23.118+0000 7fe54b800700 1 -- 172.21.15.62:0/1446133840 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe52c06ecc0 msgr2=0x7fe52c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:42:23.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:23.118+0000 7fe54b800700 1 --2- 172.21.15.62:0/1446133840 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe52c06ecc0 0x7fe52c071170 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fe538006010 tx=0x7fe53800b540 comp rx=0 tx=0).stop 2024-04-20T21:42:23.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:23.118+0000 7fe54b800700 1 -- 172.21.15.62:0/1446133840 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe544103040 msgr2=0x7fe544198670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:42:23.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:23.118+0000 7fe54b800700 1 --2- 172.21.15.62:0/1446133840 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe544103040 0x7fe544198670 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fe53000d8d0 tx=0x7fe53000dc90 comp rx=0 tx=0).stop 2024-04-20T21:42:23.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:23.118+0000 7fe54b800700 1 -- 172.21.15.62:0/1446133840 shutdown_connections 2024-04-20T21:42:23.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:23.118+0000 7fe54b800700 1 --2- 172.21.15.62:0/1446133840 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe52c06ecc0 0x7fe52c071170 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:42:23.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:23.118+0000 7fe54b800700 1 --2- 172.21.15.62:0/1446133840 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe5441026a0 0x7fe544198130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:42:23.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:23.118+0000 7fe54b800700 1 --2- 172.21.15.62:0/1446133840 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe544103040 0x7fe544198670 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:42:23.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:23.118+0000 7fe54b800700 1 -- 172.21.15.62:0/1446133840 >> 172.21.15.62:0/1446133840 conn(0x7fe5440fdf40 msgr2=0x7fe5440ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:42:23.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:23.119+0000 7fe54b800700 1 -- 172.21.15.62:0/1446133840 shutdown_connections 2024-04-20T21:42:23.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:23.119+0000 7fe54b800700 1 -- 172.21.15.62:0/1446133840 wait complete. 2024-04-20T21:42:23.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:23.274+0000 7ff16fe00700 1 -- 172.21.15.62:0/2333812965 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1701061b0 msgr2=0x7ff170106580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:42:23.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:23.274+0000 7ff16fe00700 1 --2- 172.21.15.62:0/2333812965 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1701061b0 0x7ff170106580 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7ff160009910 tx=0x7ff160009c20 comp rx=0 tx=0).stop 2024-04-20T21:42:23.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:23.274+0000 7ff16fe00700 1 -- 172.21.15.62:0/2333812965 shutdown_connections 2024-04-20T21:42:23.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:23.274+0000 7ff16fe00700 1 --2- 172.21.15.62:0/2333812965 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff170068540 0x7ff1700689b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:42:23.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:23.274+0000 7ff16fe00700 1 --2- 172.21.15.62:0/2333812965 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1701061b0 0x7ff170106580 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:42:23.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:23.274+0000 7ff16fe00700 1 -- 172.21.15.62:0/2333812965 >> 172.21.15.62:0/2333812965 conn(0x7ff170074fa0 msgr2=0x7ff1700753a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:42:23.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:23.274+0000 7ff16fe00700 1 -- 172.21.15.62:0/2333812965 shutdown_connections 2024-04-20T21:42:23.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:23.274+0000 7ff16fe00700 1 -- 172.21.15.62:0/2333812965 wait complete. 2024-04-20T21:42:23.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:23.276+0000 7ff16fe00700 1 Processor -- start 2024-04-20T21:42:23.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:23.276+0000 7ff16fe00700 1 -- start start 2024-04-20T21:42:23.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:23.276+0000 7ff16fe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff170068540 0x7ff170197d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:42:23.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:23.276+0000 7ff16fe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1701061b0 0x7ff1701982a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:42:23.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:23.276+0000 7ff16fe00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1701988f0 con 0x7ff1701061b0 2024-04-20T21:42:23.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:23.276+0000 7ff16fe00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff170198a30 con 0x7ff170068540 2024-04-20T21:42:23.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:23.277+0000 7ff16e000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1701061b0 0x7ff1701982a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:42:23.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:23.277+0000 7ff16e000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1701061b0 0x7ff1701982a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:39924/0 (socket says 172.21.15.62:39924) 2024-04-20T21:42:23.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:23.277+0000 7ff16e000700 1 -- 172.21.15.62:0/2351397612 learned_addr learned my addr 172.21.15.62:0/2351397612 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:42:23.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:23.277+0000 7ff16ea00700 1 --2- 172.21.15.62:0/2351397612 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff170068540 0x7ff170197d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:42:23.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:23.277+0000 7ff16e000700 1 -- 172.21.15.62:0/2351397612 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff170068540 msgr2=0x7ff170197d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:42:23.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:23.277+0000 7ff16e000700 1 --2- 172.21.15.62:0/2351397612 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff170068540 0x7ff170197d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:42:23.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:23.277+0000 7ff16e000700 1 -- 172.21.15.62:0/2351397612 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1600095f0 con 0x7ff1701061b0 2024-04-20T21:42:23.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:23.277+0000 7ff16e000700 1 --2- 172.21.15.62:0/2351397612 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1701061b0 0x7ff1701982a0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7ff15c00c930 tx=0x7ff15c00ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:42:23.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:23.278+0000 7ff167400700 1 -- 172.21.15.62:0/2351397612 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff15c007ad0 con 0x7ff1701061b0 2024-04-20T21:42:23.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:23.278+0000 7ff167400700 1 -- 172.21.15.62:0/2351397612 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff15c007c30 con 0x7ff1701061b0 2024-04-20T21:42:23.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:23.278+0000 7ff167400700 1 -- 172.21.15.62:0/2351397612 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff15c018610 con 0x7ff1701061b0 2024-04-20T21:42:23.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:23.278+0000 7ff16fe00700 1 -- 172.21.15.62:0/2351397612 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff17019c880 con 0x7ff1701061b0 2024-04-20T21:42:23.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:23.278+0000 7ff16fe00700 1 -- 172.21.15.62:0/2351397612 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff17019ce00 con 0x7ff1701061b0 2024-04-20T21:42:23.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:23.279+0000 7ff16fe00700 1 -- 172.21.15.62:0/2351397612 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff170100eb0 con 0x7ff1701061b0 2024-04-20T21:42:23.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:23.280+0000 7ff167400700 1 -- 172.21.15.62:0/2351397612 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff15c00f450 con 0x7ff1701061b0 2024-04-20T21:42:23.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:23.281+0000 7ff167400700 1 --2- 172.21.15.62:0/2351397612 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff15406ecc0 0x7ff154071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:42:23.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:23.281+0000 7ff167400700 1 -- 172.21.15.62:0/2351397612 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6500+0+0 (secure 0 0 0) 0x7ff15c08ef60 con 0x7ff1701061b0 2024-04-20T21:42:23.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:23.282+0000 7ff16ea00700 1 --2- 172.21.15.62:0/2351397612 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff15406ecc0 0x7ff154071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:42:23.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:23.282+0000 7ff16ea00700 1 --2- 172.21.15.62:0/2351397612 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff15406ecc0 0x7ff154071170 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ff160006010 tx=0x7ff16001a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:42:23.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:23.286+0000 7ff167400700 1 -- 172.21.15.62:0/2351397612 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7ff15c0c3ff0 con 0x7ff1701061b0 2024-04-20T21:42:23.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:23.605+0000 7ff16fe00700 1 -- 172.21.15.62:0/2351397612 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7ff170062380 con 0x7ff1701061b0 2024-04-20T21:42:23.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:23.605+0000 7ff167400700 1 -- 172.21.15.62:0/2351397612 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7ff15c057d90 con 0x7ff1701061b0 2024-04-20T21:42:23.605 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-20T21:42:23.605 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-20T21:42:23.605 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is degraded 2024-04-20T21:42:23.605 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T21:42:23.605 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T21:42:23.605 INFO:teuthology.orchestra.run.smithi062.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-20T21:42:23.605 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-20T21:42:23.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:23.608+0000 7ff16fe00700 1 -- 172.21.15.62:0/2351397612 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff15406ecc0 msgr2=0x7ff154071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:42:23.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:23.608+0000 7ff16fe00700 1 --2- 172.21.15.62:0/2351397612 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff15406ecc0 0x7ff154071170 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ff160006010 tx=0x7ff16001a040 comp rx=0 tx=0).stop 2024-04-20T21:42:23.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:23.608+0000 7ff16fe00700 1 -- 172.21.15.62:0/2351397612 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1701061b0 msgr2=0x7ff1701982a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:42:23.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:23.608+0000 7ff16fe00700 1 --2- 172.21.15.62:0/2351397612 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1701061b0 0x7ff1701982a0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7ff15c00c930 tx=0x7ff15c00ccf0 comp rx=0 tx=0).stop 2024-04-20T21:42:23.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:23.608+0000 7ff16fe00700 1 -- 172.21.15.62:0/2351397612 shutdown_connections 2024-04-20T21:42:23.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:23.608+0000 7ff16fe00700 1 --2- 172.21.15.62:0/2351397612 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff15406ecc0 0x7ff154071170 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:42:23.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:23.608+0000 7ff16fe00700 1 --2- 172.21.15.62:0/2351397612 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff170068540 0x7ff170197d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:42:23.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:23.608+0000 7ff16fe00700 1 --2- 172.21.15.62:0/2351397612 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1701061b0 0x7ff1701982a0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:42:23.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:23.608+0000 7ff16fe00700 1 -- 172.21.15.62:0/2351397612 >> 172.21.15.62:0/2351397612 conn(0x7ff170074fa0 msgr2=0x7ff170107100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:42:23.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:23.608+0000 7ff16fe00700 1 -- 172.21.15.62:0/2351397612 shutdown_connections 2024-04-20T21:42:23.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:23.608+0000 7ff16fe00700 1 -- 172.21.15.62:0/2351397612 wait complete. 2024-04-20T21:42:23.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:23 smithi189 ceph-mon[57022]: pgmap v211: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:42:23.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:23 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/4134449184' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T21:42:24.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:23 smithi062 ceph-mon[76322]: pgmap v211: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:42:24.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:23 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/4134449184' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T21:42:24.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:24 smithi062 ceph-mon[76322]: from='client.34468 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:42:24.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:24 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/2351397612' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T21:42:24.890 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:24 smithi189 ceph-mon[57022]: from='client.34468 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:42:24.890 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:24 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/2351397612' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T21:42:25.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:25 smithi189 ceph-mon[57022]: pgmap v212: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:42:26.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:25 smithi062 ceph-mon[76322]: pgmap v212: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:42:27.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:27 smithi062 ceph-mon[76322]: pgmap v213: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:42:27.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:27 smithi189 ceph-mon[57022]: pgmap v213: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:42:28.734 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:28 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:42:28.734 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:28 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:42:28.735 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:28 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:42:28.735 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:28 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T21:42:28.735 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:28 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:42:28.735 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:28 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:42:28.735 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:28 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:42:28.735 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:28 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:42:28.735 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:28 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:42:28.735 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:28 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:42:28.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:28 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:42:28.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:28 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:42:28.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:28 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:42:28.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:28 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T21:42:28.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:28 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:42:28.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:28 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:42:28.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:28 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:42:28.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:28 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:42:28.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:28 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:42:28.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:28 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:42:29.693 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:29 smithi062 ceph-mon[76322]: pgmap v214: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:42:29.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:29 smithi189 ceph-mon[57022]: pgmap v214: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:42:31.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:31 smithi062 ceph-mon[76322]: Upgrade: Updating mds.cephfs.smithi062.ozizwi 2024-04-20T21:42:31.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:31 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:42:31.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:31 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi062.ozizwi", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-20T21:42:31.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:31 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:42:31.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:31 smithi062 ceph-mon[76322]: Deploying daemon mds.cephfs.smithi062.ozizwi on smithi062 2024-04-20T21:42:31.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:31 smithi062 ceph-mon[76322]: pgmap v215: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:42:31.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:31 smithi189 ceph-mon[57022]: Upgrade: Updating mds.cephfs.smithi062.ozizwi 2024-04-20T21:42:31.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:31 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:42:31.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:31 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi062.ozizwi", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-20T21:42:31.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:31 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:42:31.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:31 smithi189 ceph-mon[57022]: Deploying daemon mds.cephfs.smithi062.ozizwi on smithi062 2024-04-20T21:42:31.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:31 smithi189 ceph-mon[57022]: pgmap v215: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:42:33.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:33 smithi189 ceph-mon[57022]: pgmap v216: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:42:33.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:33 smithi062 ceph-mon[76322]: pgmap v216: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:42:34.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:34 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:42:34.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:34 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:42:35.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:35 smithi189 ceph-mon[57022]: pgmap v217: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:42:35.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:35 smithi189 ceph-mon[57022]: osdmap e83: 6 total, 6 up, 6 in 2024-04-20T21:42:35.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:35 smithi189 ceph-mon[57022]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-04-20T21:42:35.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:35 smithi062 ceph-mon[76322]: pgmap v217: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:42:35.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:35 smithi062 ceph-mon[76322]: osdmap e83: 6 total, 6 up, 6 in 2024-04-20T21:42:35.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:35 smithi062 ceph-mon[76322]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-04-20T21:42:37.695 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:37 smithi062 ceph-mon[76322]: pgmap v219: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:42:37.695 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:37 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi062.ozizwi"}]: dispatch 2024-04-20T21:42:37.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:37 smithi189 ceph-mon[57022]: pgmap v219: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:42:37.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:37 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi062.ozizwi"}]: dispatch 2024-04-20T21:42:39.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:39 smithi189 ceph-mon[57022]: pgmap v220: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:42:39.740 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:39 smithi062 ceph-mon[76322]: pgmap v220: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:42:41.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:40 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:42:41.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:40 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:42:41.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:40 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:42:41.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:40 smithi189 ceph-mon[57022]: mds.? [v2:172.21.15.62:6828/3801975165,v1:172.21.15.62:6829/3801975165] up:boot 2024-04-20T21:42:41.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:40 smithi189 ceph-mon[57022]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-20T21:42:41.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:40 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi062.ozizwi"}]: dispatch 2024-04-20T21:42:41.259 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:40 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:42:41.260 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:40 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:42:41.260 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:40 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:42:41.260 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:40 smithi062 ceph-mon[76322]: mds.? [v2:172.21.15.62:6828/3801975165,v1:172.21.15.62:6829/3801975165] up:boot 2024-04-20T21:42:41.260 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:40 smithi062 ceph-mon[76322]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-20T21:42:41.260 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:40 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi062.ozizwi"}]: dispatch 2024-04-20T21:42:42.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:41 smithi189 ceph-mon[57022]: pgmap v221: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:42:42.266 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:41 smithi062 ceph-mon[76322]: pgmap v221: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:42:43.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:42 smithi189 ceph-mon[57022]: pgmap v222: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:42:43.245 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:42 smithi062 ceph-mon[76322]: pgmap v222: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:42:45.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:45 smithi189 ceph-mon[57022]: pgmap v223: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:42:45.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:45 smithi062 ceph-mon[76322]: pgmap v223: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:42:47.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:47 smithi189 ceph-mon[57022]: pgmap v224: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:42:47.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:47 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:42:47.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:47 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:42:47.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:47 smithi062 ceph-mon[76322]: pgmap v224: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:42:47.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:47 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:42:47.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:47 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:42:49.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:49 smithi189 ceph-mon[57022]: pgmap v225: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:42:49.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:49 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:42:49.784 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:49 smithi062 ceph-mon[76322]: pgmap v225: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:42:49.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:49 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:42:50.948 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:50 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:42:50.948 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:50 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:42:50.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:50 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:42:50.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:50 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:42:51.945 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:51 smithi062 ceph-mon[76322]: pgmap v226: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:42:51.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:51 smithi189 ceph-mon[57022]: pgmap v226: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:42:53.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:53.788+0000 7ff1b4000700 1 -- 172.21.15.62:0/4228758966 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1ac101d30 msgr2=0x7ff1ac10a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:42:53.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:53.788+0000 7ff1b4000700 1 --2- 172.21.15.62:0/4228758966 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1ac101d30 0x7ff1ac10a230 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7ff198009910 tx=0x7ff198009c20 comp rx=0 tx=0).stop 2024-04-20T21:42:53.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:53.788+0000 7ff1b4000700 1 -- 172.21.15.62:0/4228758966 shutdown_connections 2024-04-20T21:42:53.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:53.788+0000 7ff1b4000700 1 --2- 172.21.15.62:0/4228758966 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1ac101d30 0x7ff1ac10a230 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:42:53.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:53.788+0000 7ff1b4000700 1 --2- 172.21.15.62:0/4228758966 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1ac101420 0x7ff1ac1017f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:42:53.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:53.788+0000 7ff1b4000700 1 -- 172.21.15.62:0/4228758966 >> 172.21.15.62:0/4228758966 conn(0x7ff1ac0fac80 msgr2=0x7ff1ac0fd090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:42:53.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:53.788+0000 7ff1b4000700 1 -- 172.21.15.62:0/4228758966 shutdown_connections 2024-04-20T21:42:53.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:53.788+0000 7ff1b4000700 1 -- 172.21.15.62:0/4228758966 wait complete. 2024-04-20T21:42:53.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:53.789+0000 7ff1b4000700 1 Processor -- start 2024-04-20T21:42:53.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:53.789+0000 7ff1b4000700 1 -- start start 2024-04-20T21:42:53.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:53.790+0000 7ff1b4000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1ac101420 0x7ff1ac198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:42:53.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:53.790+0000 7ff1b4000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1ac101d30 0x7ff1ac1985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:42:53.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:53.790+0000 7ff1b4000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1ac198ca0 con 0x7ff1ac101420 2024-04-20T21:42:53.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:53.790+0000 7ff1b4000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1ac19ca30 con 0x7ff1ac101d30 2024-04-20T21:42:53.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:53.790+0000 7ff1b1600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1ac101420 0x7ff1ac198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:42:53.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:53.790+0000 7ff1b1600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1ac101420 0x7ff1ac198080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43790/0 (socket says 172.21.15.62:43790) 2024-04-20T21:42:53.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:53.790+0000 7ff1b1600700 1 -- 172.21.15.62:0/1980627808 learned_addr learned my addr 172.21.15.62:0/1980627808 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:42:53.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:53.790+0000 7ff1b0c00700 1 --2- 172.21.15.62:0/1980627808 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1ac101d30 0x7ff1ac1985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:42:53.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:53.791+0000 7ff1b1600700 1 -- 172.21.15.62:0/1980627808 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1ac101d30 msgr2=0x7ff1ac1985c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:42:53.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:53.791+0000 7ff1b1600700 1 --2- 172.21.15.62:0/1980627808 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1ac101d30 0x7ff1ac1985c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:42:53.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:53.791+0000 7ff1b1600700 1 -- 172.21.15.62:0/1980627808 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1980095f0 con 0x7ff1ac101420 2024-04-20T21:42:53.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:53.791+0000 7ff1b0c00700 1 --2- 172.21.15.62:0/1980627808 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1ac101d30 0x7ff1ac1985c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-20T21:42:53.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:53.791+0000 7ff1b1600700 1 --2- 172.21.15.62:0/1980627808 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1ac101420 0x7ff1ac198080 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7ff1a000d8d0 tx=0x7ff1a000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:42:53.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:53.792+0000 7ff1aa000700 1 -- 172.21.15.62:0/1980627808 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff1a0009940 con 0x7ff1ac101420 2024-04-20T21:42:53.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:53.792+0000 7ff1aa000700 1 -- 172.21.15.62:0/1980627808 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff1a0010460 con 0x7ff1ac101420 2024-04-20T21:42:53.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:53.792+0000 7ff1aa000700 1 -- 172.21.15.62:0/1980627808 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff1a000f5f0 con 0x7ff1ac101420 2024-04-20T21:42:53.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:53.792+0000 7ff1b4000700 1 -- 172.21.15.62:0/1980627808 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1ac19cd10 con 0x7ff1ac101420 2024-04-20T21:42:53.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:53.792+0000 7ff1b4000700 1 -- 172.21.15.62:0/1980627808 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1ac19d260 con 0x7ff1ac101420 2024-04-20T21:42:53.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:53.793+0000 7ff1b4000700 1 -- 172.21.15.62:0/1980627808 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1ac109ca0 con 0x7ff1ac101420 2024-04-20T21:42:53.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:53.794+0000 7ff1aa000700 1 -- 172.21.15.62:0/1980627808 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff1a00105d0 con 0x7ff1ac101420 2024-04-20T21:42:53.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:53.795+0000 7ff1aa000700 1 --2- 172.21.15.62:0/1980627808 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff19406ecc0 0x7ff194071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:42:53.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:53.795+0000 7ff1aa000700 1 -- 172.21.15.62:0/1980627808 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7ff1a008ef20 con 0x7ff1ac101420 2024-04-20T21:42:53.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:53.796+0000 7ff1b0c00700 1 --2- 172.21.15.62:0/1980627808 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff19406ecc0 0x7ff194071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:42:53.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:53.797+0000 7ff1b0c00700 1 --2- 172.21.15.62:0/1980627808 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff19406ecc0 0x7ff194071170 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ff19800b5c0 tx=0x7ff19801a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:42:53.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:53.801+0000 7ff1aa000700 1 -- 172.21.15.62:0/1980627808 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7ff1a0057c30 con 0x7ff1ac101420 2024-04-20T21:42:53.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:53 smithi189 ceph-mon[57022]: pgmap v227: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:42:54.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:53 smithi062 ceph-mon[76322]: pgmap v227: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:42:54.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:54.075+0000 7ff1b4000700 1 -- 172.21.15.62:0/1980627808 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff1ac19d480 con 0x7ff19406ecc0 2024-04-20T21:42:54.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:54.078+0000 7ff1aa000700 1 -- 172.21.15.62:0/1980627808 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7ff1ac19d480 con 0x7ff19406ecc0 2024-04-20T21:42:54.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:54.082+0000 7ff1b4000700 1 -- 172.21.15.62:0/1980627808 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff19406ecc0 msgr2=0x7ff194071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:42:54.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:54.082+0000 7ff1b4000700 1 --2- 172.21.15.62:0/1980627808 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff19406ecc0 0x7ff194071170 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ff19800b5c0 tx=0x7ff19801a040 comp rx=0 tx=0).stop 2024-04-20T21:42:54.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:54.082+0000 7ff1b4000700 1 -- 172.21.15.62:0/1980627808 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1ac101420 msgr2=0x7ff1ac198080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:42:54.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:54.082+0000 7ff1b4000700 1 --2- 172.21.15.62:0/1980627808 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1ac101420 0x7ff1ac198080 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7ff1a000d8d0 tx=0x7ff1a000dc90 comp rx=0 tx=0).stop 2024-04-20T21:42:54.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:54.082+0000 7ff1b4000700 1 -- 172.21.15.62:0/1980627808 shutdown_connections 2024-04-20T21:42:54.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:54.082+0000 7ff1b4000700 1 --2- 172.21.15.62:0/1980627808 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff19406ecc0 0x7ff194071170 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:42:54.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:54.082+0000 7ff1b4000700 1 --2- 172.21.15.62:0/1980627808 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1ac101420 0x7ff1ac198080 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:42:54.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:54.082+0000 7ff1b4000700 1 --2- 172.21.15.62:0/1980627808 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1ac101d30 0x7ff1ac1985c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:42:54.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:54.082+0000 7ff1b4000700 1 -- 172.21.15.62:0/1980627808 >> 172.21.15.62:0/1980627808 conn(0x7ff1ac0fac80 msgr2=0x7ff1ac106de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:42:54.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:54.083+0000 7ff1b4000700 1 -- 172.21.15.62:0/1980627808 shutdown_connections 2024-04-20T21:42:54.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:54.083+0000 7ff1b4000700 1 -- 172.21.15.62:0/1980627808 wait complete. 2024-04-20T21:42:54.098 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T21:42:54.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:54.283+0000 7efd55600700 1 -- 172.21.15.62:0/3122646144 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efd500687a0 msgr2=0x7efd50068b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:42:54.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:54.283+0000 7efd55600700 1 --2- 172.21.15.62:0/3122646144 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efd500687a0 0x7efd50068b70 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7efd40009910 tx=0x7efd40009c20 comp rx=0 tx=0).stop 2024-04-20T21:42:54.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:54.283+0000 7efd55600700 1 -- 172.21.15.62:0/3122646144 shutdown_connections 2024-04-20T21:42:54.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:54.283+0000 7efd55600700 1 --2- 172.21.15.62:0/3122646144 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd50069140 0x7efd50104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:42:54.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:54.283+0000 7efd55600700 1 --2- 172.21.15.62:0/3122646144 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efd500687a0 0x7efd50068b70 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:42:54.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:54.283+0000 7efd55600700 1 -- 172.21.15.62:0/3122646144 >> 172.21.15.62:0/3122646144 conn(0x7efd50075130 msgr2=0x7efd50075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:42:54.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:54.284+0000 7efd55600700 1 -- 172.21.15.62:0/3122646144 shutdown_connections 2024-04-20T21:42:54.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:54.284+0000 7efd55600700 1 -- 172.21.15.62:0/3122646144 wait complete. 2024-04-20T21:42:54.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:54.285+0000 7efd55600700 1 Processor -- start 2024-04-20T21:42:54.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:54.285+0000 7efd55600700 1 -- start start 2024-04-20T21:42:54.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:54.285+0000 7efd55600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd500687a0 0x7efd50198160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:42:54.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:54.285+0000 7efd55600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efd50069140 0x7efd501986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:42:54.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:54.285+0000 7efd55600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd50198d80 con 0x7efd50069140 2024-04-20T21:42:54.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:54.285+0000 7efd55600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd5019cb10 con 0x7efd500687a0 2024-04-20T21:42:54.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:54.286+0000 7efd4e000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efd50069140 0x7efd501986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:42:54.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:54.286+0000 7efd4e000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efd50069140 0x7efd501986a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43816/0 (socket says 172.21.15.62:43816) 2024-04-20T21:42:54.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:54.286+0000 7efd4e000700 1 -- 172.21.15.62:0/3063470428 learned_addr learned my addr 172.21.15.62:0/3063470428 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:42:54.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:54.286+0000 7efd4ea00700 1 --2- 172.21.15.62:0/3063470428 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd500687a0 0x7efd50198160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:42:54.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:54.286+0000 7efd4e000700 1 -- 172.21.15.62:0/3063470428 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd500687a0 msgr2=0x7efd50198160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:42:54.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:54.286+0000 7efd4e000700 1 --2- 172.21.15.62:0/3063470428 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd500687a0 0x7efd50198160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:42:54.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:54.286+0000 7efd4e000700 1 -- 172.21.15.62:0/3063470428 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd400095f0 con 0x7efd50069140 2024-04-20T21:42:54.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:54.287+0000 7efd4e000700 1 --2- 172.21.15.62:0/3063470428 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efd50069140 0x7efd501986a0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7efd3c00b480 tx=0x7efd3c00b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:42:54.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:54.287+0000 7efd47400700 1 -- 172.21.15.62:0/3063470428 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efd3c011800 con 0x7efd50069140 2024-04-20T21:42:54.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:54.287+0000 7efd47400700 1 -- 172.21.15.62:0/3063470428 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efd3c011e20 con 0x7efd50069140 2024-04-20T21:42:54.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:54.287+0000 7efd55600700 1 -- 172.21.15.62:0/3063470428 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd5019cdf0 con 0x7efd50069140 2024-04-20T21:42:54.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:54.287+0000 7efd47400700 1 -- 172.21.15.62:0/3063470428 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efd3c010ea0 con 0x7efd50069140 2024-04-20T21:42:54.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:54.288+0000 7efd55600700 1 -- 172.21.15.62:0/3063470428 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd5019d340 con 0x7efd50069140 2024-04-20T21:42:54.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:54.289+0000 7efd55600700 1 -- 172.21.15.62:0/3063470428 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd5004f2a0 con 0x7efd50069140 2024-04-20T21:42:54.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:54.290+0000 7efd47400700 1 -- 172.21.15.62:0/3063470428 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7efd3c010380 con 0x7efd50069140 2024-04-20T21:42:54.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:54.291+0000 7efd47400700 1 --2- 172.21.15.62:0/3063470428 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7efd3406ecc0 0x7efd34071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:42:54.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:54.291+0000 7efd47400700 1 -- 172.21.15.62:0/3063470428 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7efd3c08e930 con 0x7efd50069140 2024-04-20T21:42:54.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:54.291+0000 7efd4ea00700 1 --2- 172.21.15.62:0/3063470428 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7efd3406ecc0 0x7efd34071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:42:54.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:54.292+0000 7efd4ea00700 1 --2- 172.21.15.62:0/3063470428 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7efd3406ecc0 0x7efd34071170 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7efd40006010 tx=0x7efd4001a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:42:54.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:54.296+0000 7efd47400700 1 -- 172.21.15.62:0/3063470428 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7efd3c0576c0 con 0x7efd50069140 2024-04-20T21:42:54.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:54.586+0000 7efd55600700 1 -- 172.21.15.62:0/3063470428 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7efd5019d730 con 0x7efd3406ecc0 2024-04-20T21:42:54.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:54.589+0000 7efd47400700 1 -- 172.21.15.62:0/3063470428 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7efd5019d730 con 0x7efd3406ecc0 2024-04-20T21:42:54.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:54.592+0000 7efd55600700 1 -- 172.21.15.62:0/3063470428 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7efd3406ecc0 msgr2=0x7efd34071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:42:54.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:54.592+0000 7efd55600700 1 --2- 172.21.15.62:0/3063470428 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7efd3406ecc0 0x7efd34071170 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7efd40006010 tx=0x7efd4001a040 comp rx=0 tx=0).stop 2024-04-20T21:42:54.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:54.592+0000 7efd55600700 1 -- 172.21.15.62:0/3063470428 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efd50069140 msgr2=0x7efd501986a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:42:54.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:54.592+0000 7efd55600700 1 --2- 172.21.15.62:0/3063470428 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efd50069140 0x7efd501986a0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7efd3c00b480 tx=0x7efd3c00b790 comp rx=0 tx=0).stop 2024-04-20T21:42:54.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:54.592+0000 7efd55600700 1 -- 172.21.15.62:0/3063470428 shutdown_connections 2024-04-20T21:42:54.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:54.592+0000 7efd55600700 1 --2- 172.21.15.62:0/3063470428 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7efd3406ecc0 0x7efd34071170 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:42:54.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:54.592+0000 7efd55600700 1 --2- 172.21.15.62:0/3063470428 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd500687a0 0x7efd50198160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:42:54.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:54.592+0000 7efd55600700 1 --2- 172.21.15.62:0/3063470428 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efd50069140 0x7efd501986a0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:42:54.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:54.592+0000 7efd55600700 1 -- 172.21.15.62:0/3063470428 >> 172.21.15.62:0/3063470428 conn(0x7efd50075130 msgr2=0x7efd50101e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:42:54.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:54.592+0000 7efd55600700 1 -- 172.21.15.62:0/3063470428 shutdown_connections 2024-04-20T21:42:54.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:54.593+0000 7efd55600700 1 -- 172.21.15.62:0/3063470428 wait complete. 2024-04-20T21:42:54.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:54.763+0000 7f2a11000700 1 -- 172.21.15.62:0/521121824 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2a0c1024d0 msgr2=0x7f2a0c1028a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:42:54.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:54.763+0000 7f2a11000700 1 --2- 172.21.15.62:0/521121824 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2a0c1024d0 0x7f2a0c1028a0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f29fc00b0a0 tx=0x7f29fc00b3b0 comp rx=0 tx=0).stop 2024-04-20T21:42:54.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:54.763+0000 7f2a11000700 1 -- 172.21.15.62:0/521121824 shutdown_connections 2024-04-20T21:42:54.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:54.763+0000 7f2a11000700 1 --2- 172.21.15.62:0/521121824 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a0c102e70 0x7f2a0c106e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:42:54.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:54.763+0000 7f2a11000700 1 --2- 172.21.15.62:0/521121824 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2a0c1024d0 0x7f2a0c1028a0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:42:54.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:54.763+0000 7f2a11000700 1 -- 172.21.15.62:0/521121824 >> 172.21.15.62:0/521121824 conn(0x7f2a0c0fdd90 msgr2=0x7f2a0c1001a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:42:54.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:54.763+0000 7f2a11000700 1 -- 172.21.15.62:0/521121824 shutdown_connections 2024-04-20T21:42:54.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:54.763+0000 7f2a11000700 1 -- 172.21.15.62:0/521121824 wait complete. 2024-04-20T21:42:54.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:54.764+0000 7f2a11000700 1 Processor -- start 2024-04-20T21:42:54.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:54.765+0000 7f2a11000700 1 -- start start 2024-04-20T21:42:54.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:54.765+0000 7f2a11000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2a0c1024d0 0x7f2a0c197e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:42:54.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:54.765+0000 7f2a11000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a0c102e70 0x7f2a0c1983a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:42:54.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:54.765+0000 7f2a11000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a0c198a80 con 0x7f2a0c1024d0 2024-04-20T21:42:54.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:54.765+0000 7f2a11000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a0c19c810 con 0x7f2a0c102e70 2024-04-20T21:42:54.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:54.765+0000 7f2a0b400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2a0c1024d0 0x7f2a0c197e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:42:54.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:54.765+0000 7f2a0b400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2a0c1024d0 0x7f2a0c197e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43842/0 (socket says 172.21.15.62:43842) 2024-04-20T21:42:54.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:54.765+0000 7f2a0b400700 1 -- 172.21.15.62:0/2465115577 learned_addr learned my addr 172.21.15.62:0/2465115577 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:42:54.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:54.766+0000 7f2a02a00700 1 --2- 172.21.15.62:0/2465115577 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a0c102e70 0x7f2a0c1983a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:42:54.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:54.766+0000 7f2a0b400700 1 -- 172.21.15.62:0/2465115577 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a0c102e70 msgr2=0x7f2a0c1983a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:42:54.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:54.766+0000 7f2a0b400700 1 --2- 172.21.15.62:0/2465115577 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a0c102e70 0x7f2a0c1983a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:42:54.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:54.766+0000 7f2a0b400700 1 -- 172.21.15.62:0/2465115577 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29fc009d00 con 0x7f2a0c1024d0 2024-04-20T21:42:54.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:54.766+0000 7f2a0b400700 1 --2- 172.21.15.62:0/2465115577 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2a0c1024d0 0x7f2a0c197e60 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f29fc00b070 tx=0x7f29fc00cdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:42:54.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:54.767+0000 7f2a08c00700 1 -- 172.21.15.62:0/2465115577 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f29fc00e040 con 0x7f2a0c1024d0 2024-04-20T21:42:54.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:54.767+0000 7f2a08c00700 1 -- 172.21.15.62:0/2465115577 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f29fc003de0 con 0x7f2a0c1024d0 2024-04-20T21:42:54.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:54.767+0000 7f2a11000700 1 -- 172.21.15.62:0/2465115577 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a0c19ca90 con 0x7f2a0c1024d0 2024-04-20T21:42:54.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:54.767+0000 7f2a08c00700 1 -- 172.21.15.62:0/2465115577 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f29fc01d8e0 con 0x7f2a0c1024d0 2024-04-20T21:42:54.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:54.767+0000 7f2a11000700 1 -- 172.21.15.62:0/2465115577 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a0c19cf80 con 0x7f2a0c1024d0 2024-04-20T21:42:54.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:54.768+0000 7f2a11000700 1 -- 172.21.15.62:0/2465115577 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a0c10a7e0 con 0x7f2a0c1024d0 2024-04-20T21:42:54.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:54.769+0000 7f2a08c00700 1 -- 172.21.15.62:0/2465115577 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f29fc004040 con 0x7f2a0c1024d0 2024-04-20T21:42:54.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:54.770+0000 7f2a08c00700 1 --2- 172.21.15.62:0/2465115577 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f29f006ecc0 0x7f29f0071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:42:54.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:54.770+0000 7f2a08c00700 1 -- 172.21.15.62:0/2465115577 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f29fc090740 con 0x7f2a0c1024d0 2024-04-20T21:42:54.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:54.770+0000 7f2a02a00700 1 --2- 172.21.15.62:0/2465115577 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f29f006ecc0 0x7f29f0071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:42:54.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:54.771+0000 7f2a02a00700 1 --2- 172.21.15.62:0/2465115577 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f29f006ecc0 0x7f29f0071170 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f29f4005950 tx=0x7f29f40058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:42:54.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:54.775+0000 7f2a08c00700 1 -- 172.21.15.62:0/2465115577 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f29fc059450 con 0x7f2a0c1024d0 2024-04-20T21:42:55.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:55.026+0000 7f2a11000700 1 -- 172.21.15.62:0/2465115577 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f2a0c19d260 con 0x7f29f006ecc0 2024-04-20T21:42:55.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:55.039+0000 7f2a08c00700 1 -- 172.21.15.62:0/2465115577 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7f2a0c19d260 con 0x7f29f006ecc0 2024-04-20T21:42:55.039 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T21:42:55.039 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (8m) 7s ago 22m 14.2M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T21:42:55.039 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (22m) 7s ago 22m 9512k - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T21:42:55.039 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (21m) 63s ago 21m 8942k - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T21:42:55.039 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (4m) 7s ago 22m 6660k - 19.0.0-3133-ga8c84f59 76bc6b91b726 2825d5ba11c6 2024-04-20T21:42:55.039 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (4m) 63s ago 21m 6665k - 19.0.0-3133-ga8c84f59 76bc6b91b726 20f746fa1bba 2024-04-20T21:42:55.039 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (8m) 7s ago 22m 74.2M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T21:42:55.039 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (15s) 7s ago 15m 11.2M - 19.0.0-3133-ga8c84f59 76bc6b91b726 de80bc90a683 2024-04-20T21:42:55.039 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (43s) 7s ago 15m 11.9M - 19.0.0-3133-ga8c84f59 76bc6b91b726 cd793b7ee98e 2024-04-20T21:42:55.039 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (15m) 63s ago 15m 117M - 18.2.0 dc2bc1663786 fb2a8a223db8 2024-04-20T21:42:55.039 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (15m) 63s ago 15m 55.6M - 18.2.0 dc2bc1663786 f2eec2ee8951 2024-04-20T21:42:55.039 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (11m) 7s ago 24m 511M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T21:42:55.040 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (10m) 63s ago 21m 452M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T21:42:55.040 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (6m) 7s ago 24m 55.9M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 019eb1479485 2024-04-20T21:42:55.040 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (5m) 63s ago 21m 39.8M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 55aac560aab6 2024-04-20T21:42:55.040 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (9m) 7s ago 22m 7952k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T21:42:55.040 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (9m) 63s ago 21m 8304k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T21:42:55.040 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (4m) 7s ago 20m 88.0M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 ee2bb901f26e 2024-04-20T21:42:55.040 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (3m) 7s ago 19m 92.3M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 9075152d4162 2024-04-20T21:42:55.040 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (3m) 7s ago 18m 74.4M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 2d4f3e38dc0e 2024-04-20T21:42:55.040 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (2m) 63s ago 18m 87.6M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 8f97b17ae0eb 2024-04-20T21:42:55.040 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (112s) 63s ago 17m 90.7M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 653d8079967d 2024-04-20T21:42:55.040 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (69s) 63s ago 17m 11.5M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 596dded14c12 2024-04-20T21:42:55.040 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (9m) 7s ago 22m 74.0M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T21:42:55.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:55.042+0000 7f2a11000700 1 -- 172.21.15.62:0/2465115577 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f29f006ecc0 msgr2=0x7f29f0071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:42:55.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:55.042+0000 7f2a11000700 1 --2- 172.21.15.62:0/2465115577 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f29f006ecc0 0x7f29f0071170 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f29f4005950 tx=0x7f29f40058e0 comp rx=0 tx=0).stop 2024-04-20T21:42:55.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:55.042+0000 7f2a11000700 1 -- 172.21.15.62:0/2465115577 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2a0c1024d0 msgr2=0x7f2a0c197e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:42:55.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:55.042+0000 7f2a11000700 1 --2- 172.21.15.62:0/2465115577 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2a0c1024d0 0x7f2a0c197e60 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f29fc00b070 tx=0x7f29fc00cdd0 comp rx=0 tx=0).stop 2024-04-20T21:42:55.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:55.043+0000 7f2a11000700 1 -- 172.21.15.62:0/2465115577 shutdown_connections 2024-04-20T21:42:55.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:55.043+0000 7f2a11000700 1 --2- 172.21.15.62:0/2465115577 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f29f006ecc0 0x7f29f0071170 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:42:55.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:55.043+0000 7f2a11000700 1 --2- 172.21.15.62:0/2465115577 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2a0c1024d0 0x7f2a0c197e60 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:42:55.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:55.043+0000 7f2a11000700 1 --2- 172.21.15.62:0/2465115577 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a0c102e70 0x7f2a0c1983a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:42:55.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:55.043+0000 7f2a11000700 1 -- 172.21.15.62:0/2465115577 >> 172.21.15.62:0/2465115577 conn(0x7f2a0c0fdd90 msgr2=0x7f2a0c0ff4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:42:55.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:55.043+0000 7f2a11000700 1 -- 172.21.15.62:0/2465115577 shutdown_connections 2024-04-20T21:42:55.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:55.043+0000 7f2a11000700 1 -- 172.21.15.62:0/2465115577 wait complete. 2024-04-20T21:42:55.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:55.208+0000 7f0e32c00700 1 -- 172.21.15.62:0/2779841760 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e2c106320 msgr2=0x7f0e2c1066f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:42:55.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:55.208+0000 7f0e32c00700 1 --2- 172.21.15.62:0/2779841760 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e2c106320 0x7f0e2c1066f0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f0e1c009910 tx=0x7f0e1c009c20 comp rx=0 tx=0).stop 2024-04-20T21:42:55.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:55.208+0000 7f0e32c00700 1 -- 172.21.15.62:0/2779841760 shutdown_connections 2024-04-20T21:42:55.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:55.208+0000 7f0e32c00700 1 --2- 172.21.15.62:0/2779841760 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0e2c068540 0x7f0e2c0689b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:42:55.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:55.208+0000 7f0e32c00700 1 --2- 172.21.15.62:0/2779841760 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e2c106320 0x7f0e2c1066f0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:42:55.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:55.208+0000 7f0e32c00700 1 -- 172.21.15.62:0/2779841760 >> 172.21.15.62:0/2779841760 conn(0x7f0e2c075130 msgr2=0x7f0e2c075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:42:55.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:55.209+0000 7f0e32c00700 1 -- 172.21.15.62:0/2779841760 shutdown_connections 2024-04-20T21:42:55.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:55.209+0000 7f0e32c00700 1 -- 172.21.15.62:0/2779841760 wait complete. 2024-04-20T21:42:55.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:55.210+0000 7f0e32c00700 1 Processor -- start 2024-04-20T21:42:55.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:55.210+0000 7f0e32c00700 1 -- start start 2024-04-20T21:42:55.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:55.210+0000 7f0e32c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e2c068540 0x7f0e2c198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:42:55.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:55.210+0000 7f0e32c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0e2c106320 0x7f0e2c1985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:42:55.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:55.210+0000 7f0e32c00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e2c198c80 con 0x7f0e2c068540 2024-04-20T21:42:55.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:55.210+0000 7f0e32c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e2c19ca10 con 0x7f0e2c106320 2024-04-20T21:42:55.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:55.211+0000 7f0e2be00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e2c068540 0x7f0e2c198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:42:55.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:55.211+0000 7f0e2be00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e2c068540 0x7f0e2c198060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43848/0 (socket says 172.21.15.62:43848) 2024-04-20T21:42:55.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:55.211+0000 7f0e2be00700 1 -- 172.21.15.62:0/2552953627 learned_addr learned my addr 172.21.15.62:0/2552953627 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:42:55.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:55.211+0000 7f0e2b400700 1 --2- 172.21.15.62:0/2552953627 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0e2c106320 0x7f0e2c1985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:42:55.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:55.211+0000 7f0e2be00700 1 -- 172.21.15.62:0/2552953627 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0e2c106320 msgr2=0x7f0e2c1985a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:42:55.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:55.211+0000 7f0e2be00700 1 --2- 172.21.15.62:0/2552953627 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0e2c106320 0x7f0e2c1985a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:42:55.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:55.211+0000 7f0e2be00700 1 -- 172.21.15.62:0/2552953627 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e1c0095f0 con 0x7f0e2c068540 2024-04-20T21:42:55.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:55.212+0000 7f0e2be00700 1 --2- 172.21.15.62:0/2552953627 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e2c068540 0x7f0e2c198060 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f0e1c00b5c0 tx=0x7f0e1c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:42:55.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:55.212+0000 7f0e28c00700 1 -- 172.21.15.62:0/2552953627 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0e1c01d070 con 0x7f0e2c068540 2024-04-20T21:42:55.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:55.212+0000 7f0e28c00700 1 -- 172.21.15.62:0/2552953627 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0e1c005e00 con 0x7f0e2c068540 2024-04-20T21:42:55.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:55.212+0000 7f0e28c00700 1 -- 172.21.15.62:0/2552953627 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0e1c017400 con 0x7f0e2c068540 2024-04-20T21:42:55.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:55.212+0000 7f0e32c00700 1 -- 172.21.15.62:0/2552953627 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e2c19cc90 con 0x7f0e2c068540 2024-04-20T21:42:55.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:55.212+0000 7f0e32c00700 1 -- 172.21.15.62:0/2552953627 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e2c19d130 con 0x7f0e2c068540 2024-04-20T21:42:55.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:55.214+0000 7f0e32c00700 1 -- 172.21.15.62:0/2552953627 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e2c04f2a0 con 0x7f0e2c068540 2024-04-20T21:42:55.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:55.215+0000 7f0e28c00700 1 -- 172.21.15.62:0/2552953627 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f0e1c00f460 con 0x7f0e2c068540 2024-04-20T21:42:55.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:55.216+0000 7f0e28c00700 1 --2- 172.21.15.62:0/2552953627 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f0e1006ecc0 0x7f0e10071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:42:55.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:55.216+0000 7f0e28c00700 1 -- 172.21.15.62:0/2552953627 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f0e1c090380 con 0x7f0e2c068540 2024-04-20T21:42:55.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:55.216+0000 7f0e2b400700 1 --2- 172.21.15.62:0/2552953627 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f0e1006ecc0 0x7f0e10071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:42:55.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:55.217+0000 7f0e2b400700 1 --2- 172.21.15.62:0/2552953627 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f0e1006ecc0 0x7f0e10071170 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f0e18005950 tx=0x7f0e180058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:42:55.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:55.221+0000 7f0e28c00700 1 -- 172.21.15.62:0/2552953627 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f0e1c059130 con 0x7f0e2c068540 2024-04-20T21:42:55.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:55.556+0000 7f0e32c00700 1 -- 172.21.15.62:0/2552953627 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f0e2c19d4b0 con 0x7f0e2c068540 2024-04-20T21:42:55.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:55.557+0000 7f0e28c00700 1 -- 172.21.15.62:0/2552953627 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7f0e1c05c7f0 con 0x7f0e2c068540 2024-04-20T21:42:55.556 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:42:55.556 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T21:42:55.556 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:42:55.556 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:42:55.556 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T21:42:55.556 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:42:55.556 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:42:55.557 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T21:42:55.557 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 6 2024-04-20T21:42:55.557 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:42:55.557 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T21:42:55.557 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 2, 2024-04-20T21:42:55.557 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:42:55.557 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:42:55.557 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T21:42:55.557 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 2, 2024-04-20T21:42:55.557 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 12 2024-04-20T21:42:55.557 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T21:42:55.557 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:42:55.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:55.560+0000 7f0e32c00700 1 -- 172.21.15.62:0/2552953627 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f0e1006ecc0 msgr2=0x7f0e10071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:42:55.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:55.560+0000 7f0e32c00700 1 --2- 172.21.15.62:0/2552953627 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f0e1006ecc0 0x7f0e10071170 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f0e18005950 tx=0x7f0e180058e0 comp rx=0 tx=0).stop 2024-04-20T21:42:55.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:55.560+0000 7f0e32c00700 1 -- 172.21.15.62:0/2552953627 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e2c068540 msgr2=0x7f0e2c198060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:42:55.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:55.560+0000 7f0e32c00700 1 --2- 172.21.15.62:0/2552953627 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e2c068540 0x7f0e2c198060 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f0e1c00b5c0 tx=0x7f0e1c00be20 comp rx=0 tx=0).stop 2024-04-20T21:42:55.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:55.560+0000 7f0e32c00700 1 -- 172.21.15.62:0/2552953627 shutdown_connections 2024-04-20T21:42:55.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:55.560+0000 7f0e32c00700 1 --2- 172.21.15.62:0/2552953627 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f0e1006ecc0 0x7f0e10071170 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:42:55.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:55.560+0000 7f0e32c00700 1 --2- 172.21.15.62:0/2552953627 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e2c068540 0x7f0e2c198060 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:42:55.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:55.560+0000 7f0e32c00700 1 --2- 172.21.15.62:0/2552953627 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0e2c106320 0x7f0e2c1985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:42:55.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:55.561+0000 7f0e32c00700 1 -- 172.21.15.62:0/2552953627 >> 172.21.15.62:0/2552953627 conn(0x7f0e2c075130 msgr2=0x7f0e2c0fffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:42:55.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:55.561+0000 7f0e32c00700 1 -- 172.21.15.62:0/2552953627 shutdown_connections 2024-04-20T21:42:55.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:55.561+0000 7f0e32c00700 1 -- 172.21.15.62:0/2552953627 wait complete. 2024-04-20T21:42:55.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:55.732+0000 7f5fbf800700 1 -- 172.21.15.62:0/1997774506 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5fb80730b0 msgr2=0x7f5fb8073480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:42:55.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:55.732+0000 7f5fbf800700 1 --2- 172.21.15.62:0/1997774506 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5fb80730b0 0x7f5fb8073480 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f5fac009940 tx=0x7f5fac009c50 comp rx=0 tx=0).stop 2024-04-20T21:42:55.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:55.732+0000 7f5fbf800700 1 -- 172.21.15.62:0/1997774506 shutdown_connections 2024-04-20T21:42:55.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:55.732+0000 7f5fbf800700 1 --2- 172.21.15.62:0/1997774506 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5fb80739c0 0x7f5fb810c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:42:55.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:55.732+0000 7f5fbf800700 1 --2- 172.21.15.62:0/1997774506 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5fb80730b0 0x7f5fb8073480 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:42:55.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:55.732+0000 7f5fbf800700 1 -- 172.21.15.62:0/1997774506 >> 172.21.15.62:0/1997774506 conn(0x7f5fb80fbda0 msgr2=0x7f5fb80fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:42:55.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:55.732+0000 7f5fbf800700 1 -- 172.21.15.62:0/1997774506 shutdown_connections 2024-04-20T21:42:55.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:55.732+0000 7f5fbf800700 1 -- 172.21.15.62:0/1997774506 wait complete. 2024-04-20T21:42:55.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:55.733+0000 7f5fbf800700 1 Processor -- start 2024-04-20T21:42:55.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:55.734+0000 7f5fbf800700 1 -- start start 2024-04-20T21:42:55.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:55.734+0000 7f5fbf800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5fb80730b0 0x7f5fb8198250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:42:55.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:55.734+0000 7f5fbf800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5fb80739c0 0x7f5fb8198790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:42:55.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:55.734+0000 7f5fbf800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb8198de0 con 0x7f5fb80730b0 2024-04-20T21:42:55.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:55.734+0000 7f5fbf800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb8198f20 con 0x7f5fb80739c0 2024-04-20T21:42:55.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:55.734+0000 7f5fbce00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5fb80730b0 0x7f5fb8198250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:42:55.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:55.735+0000 7f5fbce00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5fb80730b0 0x7f5fb8198250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43864/0 (socket says 172.21.15.62:43864) 2024-04-20T21:42:55.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:55.735+0000 7f5fbce00700 1 -- 172.21.15.62:0/1728102477 learned_addr learned my addr 172.21.15.62:0/1728102477 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:42:55.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:55.735+0000 7f5fb7e00700 1 --2- 172.21.15.62:0/1728102477 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5fb80739c0 0x7f5fb8198790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:42:55.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:55.735+0000 7f5fbce00700 1 -- 172.21.15.62:0/1728102477 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5fb80739c0 msgr2=0x7f5fb8198790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:42:55.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:55.735+0000 7f5fbce00700 1 --2- 172.21.15.62:0/1728102477 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5fb80739c0 0x7f5fb8198790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:42:55.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:55.735+0000 7f5fbce00700 1 -- 172.21.15.62:0/1728102477 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fac0095f0 con 0x7f5fb80730b0 2024-04-20T21:42:55.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:55.735+0000 7f5fb7e00700 1 --2- 172.21.15.62:0/1728102477 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5fb80739c0 0x7f5fb8198790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T21:42:55.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:55.735+0000 7f5fbce00700 1 --2- 172.21.15.62:0/1728102477 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5fb80730b0 0x7f5fb8198250 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f5fac00b870 tx=0x7f5fac0056a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:42:55.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:55.736+0000 7f5fb5600700 1 -- 172.21.15.62:0/1728102477 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5fac01d070 con 0x7f5fb80730b0 2024-04-20T21:42:55.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:55.736+0000 7f5fb5600700 1 -- 172.21.15.62:0/1728102477 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5fac00f460 con 0x7f5fb80730b0 2024-04-20T21:42:55.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:55.736+0000 7f5fb5600700 1 -- 172.21.15.62:0/1728102477 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5fac021640 con 0x7f5fb80730b0 2024-04-20T21:42:55.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:55.736+0000 7f5fbf800700 1 -- 172.21.15.62:0/1728102477 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fb819cd10 con 0x7f5fb80730b0 2024-04-20T21:42:55.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:55.736+0000 7f5fbf800700 1 -- 172.21.15.62:0/1728102477 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fb819d200 con 0x7f5fb80730b0 2024-04-20T21:42:55.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:55.737+0000 7f5fbf800700 1 -- 172.21.15.62:0/1728102477 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fb8109cc0 con 0x7f5fb80730b0 2024-04-20T21:42:55.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:55.738+0000 7f5fb5600700 1 -- 172.21.15.62:0/1728102477 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f5fac00fa60 con 0x7f5fb80730b0 2024-04-20T21:42:55.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:55.739+0000 7f5fb5600700 1 --2- 172.21.15.62:0/1728102477 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5fa006ecc0 0x7f5fa0071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:42:55.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:55.740+0000 7f5fb5600700 1 -- 172.21.15.62:0/1728102477 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f5fac0901d0 con 0x7f5fb80730b0 2024-04-20T21:42:55.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:55.740+0000 7f5fb7e00700 1 --2- 172.21.15.62:0/1728102477 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5fa006ecc0 0x7f5fa0071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:42:55.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:55.740+0000 7f5fb7e00700 1 --2- 172.21.15.62:0/1728102477 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5fa006ecc0 0x7f5fa0071170 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f5fa4005fd0 tx=0x7f5fa4005e20 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:42:55.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:55.744+0000 7f5fb5600700 1 -- 172.21.15.62:0/1728102477 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f5fac058ee0 con 0x7f5fb80730b0 2024-04-20T21:42:55.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:55 smithi189 ceph-mon[57022]: from='client.34480 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:42:55.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:55 smithi189 ceph-mon[57022]: pgmap v228: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:42:55.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:55 smithi189 ceph-mon[57022]: from='client.34484 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:42:55.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:55 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/2552953627' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:42:56.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.029+0000 7f5fbf800700 1 -- 172.21.15.62:0/1728102477 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f5fb81995b0 con 0x7f5fb80730b0 2024-04-20T21:42:56.028 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:55 smithi062 ceph-mon[76322]: from='client.34480 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:42:56.028 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:55 smithi062 ceph-mon[76322]: pgmap v228: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:42:56.028 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:55 smithi062 ceph-mon[76322]: from='client.34484 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:42:56.028 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:55 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/2552953627' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:42:56.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.030+0000 7f5fb5600700 1 -- 172.21.15.62:0/1728102477 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 25 v25) v1 ==== 76+0+1887 (secure 0 0 0) 0x7f5fac05c5a0 con 0x7f5fb80730b0 2024-04-20T21:42:56.030 INFO:teuthology.orchestra.run.smithi062.stdout:e25 2024-04-20T21:42:56.030 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T21:42:56.030 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T21:42:56.030 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T21:42:56.030 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:42:56.030 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T21:42:56.030 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T21:42:56.030 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 20 2024-04-20T21:42:56.030 INFO:teuthology.orchestra.run.smithi062.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-20T21:42:56.030 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T21:42:56.030 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T21:42:01.559061+0000 2024-04-20T21:42:56.030 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T21:42:56.030 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T21:42:56.031 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T21:42:56.031 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T21:42:56.031 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T21:42:56.031 INFO:teuthology.orchestra.run.smithi062.stdout:max_xattr_size 65536 2024-04-20T21:42:56.031 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T21:42:56.031 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T21:42:56.031 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 81 2024-04-20T21:42:56.031 INFO:teuthology.orchestra.run.smithi062.stdout:snap_rstat 1 2024-04-20T21:42:56.031 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T21:42:56.031 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 2 2024-04-20T21:42:56.031 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1 2024-04-20T21:42:56.031 INFO:teuthology.orchestra.run.smithi062.stdout:up {} 2024-04-20T21:42:56.031 INFO:teuthology.orchestra.run.smithi062.stdout:failed 0,1 2024-04-20T21:42:56.031 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T21:42:56.031 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T21:42:56.031 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T21:42:56.031 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T21:42:56.032 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data enabled 2024-04-20T21:42:56.032 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T21:42:56.032 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T21:42:56.032 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 1 2024-04-20T21:42:56.032 INFO:teuthology.orchestra.run.smithi062.stdout:qdb_cluster leader: 0 members: 2024-04-20T21:42:56.032 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:42:56.032 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:42:56.032 INFO:teuthology.orchestra.run.smithi062.stdout:Standby daemons: 2024-04-20T21:42:56.032 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:42:56.032 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ryqblb{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6826/4113916205,v1:172.21.15.62:6827/4113916205] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T21:42:56.032 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ozizwi{-1:34476} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6828/3801975165,v1:172.21.15.62:6829/3801975165] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T21:42:56.032 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.xbgspw{-1:44285} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6826/3996040526,v1:172.21.15.189:6827/3996040526] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:42:56.032 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.dvimer{-1:44289} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6824/1743111843,v1:172.21.15.189:6825/1743111843] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:42:56.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.033+0000 7f5fbf800700 1 -- 172.21.15.62:0/1728102477 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5fa006ecc0 msgr2=0x7f5fa0071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:42:56.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.033+0000 7f5fbf800700 1 --2- 172.21.15.62:0/1728102477 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5fa006ecc0 0x7f5fa0071170 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f5fa4005fd0 tx=0x7f5fa4005e20 comp rx=0 tx=0).stop 2024-04-20T21:42:56.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.034+0000 7f5fbf800700 1 -- 172.21.15.62:0/1728102477 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5fb80730b0 msgr2=0x7f5fb8198250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:42:56.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.034+0000 7f5fbf800700 1 --2- 172.21.15.62:0/1728102477 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5fb80730b0 0x7f5fb8198250 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f5fac00b870 tx=0x7f5fac0056a0 comp rx=0 tx=0).stop 2024-04-20T21:42:56.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.034+0000 7f5fbf800700 1 -- 172.21.15.62:0/1728102477 shutdown_connections 2024-04-20T21:42:56.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.034+0000 7f5fbf800700 1 --2- 172.21.15.62:0/1728102477 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5fa006ecc0 0x7f5fa0071170 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:42:56.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.034+0000 7f5fbf800700 1 --2- 172.21.15.62:0/1728102477 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5fb80730b0 0x7f5fb8198250 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:42:56.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.034+0000 7f5fbf800700 1 --2- 172.21.15.62:0/1728102477 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5fb80739c0 0x7f5fb8198790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:42:56.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.034+0000 7f5fbf800700 1 -- 172.21.15.62:0/1728102477 >> 172.21.15.62:0/1728102477 conn(0x7f5fb80fbda0 msgr2=0x7f5fb8106e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:42:56.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.034+0000 7f5fbf800700 1 -- 172.21.15.62:0/1728102477 shutdown_connections 2024-04-20T21:42:56.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.034+0000 7f5fbf800700 1 -- 172.21.15.62:0/1728102477 wait complete. 2024-04-20T21:42:56.034 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 25 2024-04-20T21:42:56.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.202+0000 7f6f84e00700 1 -- 172.21.15.62:0/3713754491 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6f80073090 msgr2=0x7f6f80073460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:42:56.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.202+0000 7f6f84e00700 1 --2- 172.21.15.62:0/3713754491 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6f80073090 0x7f6f80073460 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f6f70009940 tx=0x7f6f70009c50 comp rx=0 tx=0).stop 2024-04-20T21:42:56.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.202+0000 7f6f84e00700 1 -- 172.21.15.62:0/3713754491 shutdown_connections 2024-04-20T21:42:56.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.202+0000 7f6f84e00700 1 --2- 172.21.15.62:0/3713754491 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f800739a0 0x7f6f8010c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:42:56.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.202+0000 7f6f84e00700 1 --2- 172.21.15.62:0/3713754491 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6f80073090 0x7f6f80073460 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:42:56.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.202+0000 7f6f84e00700 1 -- 172.21.15.62:0/3713754491 >> 172.21.15.62:0/3713754491 conn(0x7f6f800fbda0 msgr2=0x7f6f800fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:42:56.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.202+0000 7f6f84e00700 1 -- 172.21.15.62:0/3713754491 shutdown_connections 2024-04-20T21:42:56.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.202+0000 7f6f84e00700 1 -- 172.21.15.62:0/3713754491 wait complete. 2024-04-20T21:42:56.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.203+0000 7f6f84e00700 1 Processor -- start 2024-04-20T21:42:56.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.203+0000 7f6f84e00700 1 -- start start 2024-04-20T21:42:56.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.204+0000 7f6f84e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f80073090 0x7f6f80198180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:42:56.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.204+0000 7f6f84e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6f800739a0 0x7f6f801986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:42:56.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.204+0000 7f6f84e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f80198da0 con 0x7f6f800739a0 2024-04-20T21:42:56.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.204+0000 7f6f84e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f8019cb30 con 0x7f6f80073090 2024-04-20T21:42:56.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.204+0000 7f6f7d400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6f800739a0 0x7f6f801986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:42:56.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.204+0000 7f6f7d400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6f800739a0 0x7f6f801986c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43894/0 (socket says 172.21.15.62:43894) 2024-04-20T21:42:56.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.204+0000 7f6f7d400700 1 -- 172.21.15.62:0/2462803226 learned_addr learned my addr 172.21.15.62:0/2462803226 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:42:56.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.205+0000 7f6f7de00700 1 --2- 172.21.15.62:0/2462803226 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f80073090 0x7f6f80198180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:42:56.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.205+0000 7f6f7d400700 1 -- 172.21.15.62:0/2462803226 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f80073090 msgr2=0x7f6f80198180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:42:56.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.205+0000 7f6f7d400700 1 --2- 172.21.15.62:0/2462803226 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f80073090 0x7f6f80198180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:42:56.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.205+0000 7f6f7d400700 1 -- 172.21.15.62:0/2462803226 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f700095f0 con 0x7f6f800739a0 2024-04-20T21:42:56.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.205+0000 7f6f7de00700 1 --2- 172.21.15.62:0/2462803226 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f80073090 0x7f6f80198180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T21:42:56.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.205+0000 7f6f7d400700 1 --2- 172.21.15.62:0/2462803226 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6f800739a0 0x7f6f801986c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f6f6c00d8d0 tx=0x7f6f6c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:42:56.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.205+0000 7f6f76a00700 1 -- 172.21.15.62:0/2462803226 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6f6c009940 con 0x7f6f800739a0 2024-04-20T21:42:56.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.205+0000 7f6f76a00700 1 -- 172.21.15.62:0/2462803226 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6f6c010460 con 0x7f6f800739a0 2024-04-20T21:42:56.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.205+0000 7f6f84e00700 1 -- 172.21.15.62:0/2462803226 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f8019ce10 con 0x7f6f800739a0 2024-04-20T21:42:56.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.206+0000 7f6f76a00700 1 -- 172.21.15.62:0/2462803226 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6f6c00f5f0 con 0x7f6f800739a0 2024-04-20T21:42:56.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.206+0000 7f6f84e00700 1 -- 172.21.15.62:0/2462803226 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f8019d360 con 0x7f6f800739a0 2024-04-20T21:42:56.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.207+0000 7f6f84e00700 1 -- 172.21.15.62:0/2462803226 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f80109ce0 con 0x7f6f800739a0 2024-04-20T21:42:56.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.208+0000 7f6f76a00700 1 -- 172.21.15.62:0/2462803226 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f6f6c0105d0 con 0x7f6f800739a0 2024-04-20T21:42:56.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.209+0000 7f6f76a00700 1 --2- 172.21.15.62:0/2462803226 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6f6406ecc0 0x7f6f64071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:42:56.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.209+0000 7f6f76a00700 1 -- 172.21.15.62:0/2462803226 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f6f6c08ee10 con 0x7f6f800739a0 2024-04-20T21:42:56.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.209+0000 7f6f7de00700 1 --2- 172.21.15.62:0/2462803226 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6f6406ecc0 0x7f6f64071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:42:56.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.210+0000 7f6f7de00700 1 --2- 172.21.15.62:0/2462803226 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6f6406ecc0 0x7f6f64071170 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f6f70000c00 tx=0x7f6f70005fb0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:42:56.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.214+0000 7f6f76a00700 1 -- 172.21.15.62:0/2462803226 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f6f6c057ba0 con 0x7f6f800739a0 2024-04-20T21:42:56.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.477+0000 7f6f84e00700 1 -- 172.21.15.62:0/2462803226 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6f8019d5d0 con 0x7f6f6406ecc0 2024-04-20T21:42:56.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.480+0000 7f6f76a00700 1 -- 172.21.15.62:0/2462803226 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f6f8019d5d0 con 0x7f6f6406ecc0 2024-04-20T21:42:56.480 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:42:56.480 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T21:42:56.480 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T21:42:56.480 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-20T21:42:56.480 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T21:42:56.480 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr", 2024-04-20T21:42:56.480 INFO:teuthology.orchestra.run.smithi062.stdout: "crash", 2024-04-20T21:42:56.480 INFO:teuthology.orchestra.run.smithi062.stdout: "mon", 2024-04-20T21:42:56.480 INFO:teuthology.orchestra.run.smithi062.stdout: "osd" 2024-04-20T21:42:56.480 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T21:42:56.480 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "14/23 daemons upgraded", 2024-04-20T21:42:56.480 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading mds daemons", 2024-04-20T21:42:56.481 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T21:42:56.481 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:42:56.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.483+0000 7f6f84e00700 1 -- 172.21.15.62:0/2462803226 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6f6406ecc0 msgr2=0x7f6f64071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:42:56.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.483+0000 7f6f84e00700 1 --2- 172.21.15.62:0/2462803226 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6f6406ecc0 0x7f6f64071170 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f6f70000c00 tx=0x7f6f70005fb0 comp rx=0 tx=0).stop 2024-04-20T21:42:56.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.483+0000 7f6f84e00700 1 -- 172.21.15.62:0/2462803226 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6f800739a0 msgr2=0x7f6f801986c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:42:56.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.483+0000 7f6f84e00700 1 --2- 172.21.15.62:0/2462803226 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6f800739a0 0x7f6f801986c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f6f6c00d8d0 tx=0x7f6f6c00dc90 comp rx=0 tx=0).stop 2024-04-20T21:42:56.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.483+0000 7f6f84e00700 1 -- 172.21.15.62:0/2462803226 shutdown_connections 2024-04-20T21:42:56.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.483+0000 7f6f84e00700 1 --2- 172.21.15.62:0/2462803226 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6f6406ecc0 0x7f6f64071170 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:42:56.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.483+0000 7f6f84e00700 1 --2- 172.21.15.62:0/2462803226 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f80073090 0x7f6f80198180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:42:56.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.483+0000 7f6f84e00700 1 --2- 172.21.15.62:0/2462803226 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6f800739a0 0x7f6f801986c0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:42:56.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.483+0000 7f6f84e00700 1 -- 172.21.15.62:0/2462803226 >> 172.21.15.62:0/2462803226 conn(0x7f6f800fbda0 msgr2=0x7f6f80106e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:42:56.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.484+0000 7f6f84e00700 1 -- 172.21.15.62:0/2462803226 shutdown_connections 2024-04-20T21:42:56.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.484+0000 7f6f84e00700 1 -- 172.21.15.62:0/2462803226 wait complete. 2024-04-20T21:42:56.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.644+0000 7feb1ca00700 1 -- 172.21.15.62:0/1593889052 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb180ffd80 msgr2=0x7feb18100150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:42:56.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.644+0000 7feb1ca00700 1 --2- 172.21.15.62:0/1593889052 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb180ffd80 0x7feb18100150 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7feb08009910 tx=0x7feb08009c20 comp rx=0 tx=0).stop 2024-04-20T21:42:56.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.644+0000 7feb1ca00700 1 -- 172.21.15.62:0/1593889052 shutdown_connections 2024-04-20T21:42:56.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.644+0000 7feb1ca00700 1 --2- 172.21.15.62:0/1593889052 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb18100720 0x7feb18103c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:42:56.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.644+0000 7feb1ca00700 1 --2- 172.21.15.62:0/1593889052 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb180ffd80 0x7feb18100150 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:42:56.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.644+0000 7feb1ca00700 1 -- 172.21.15.62:0/1593889052 >> 172.21.15.62:0/1593889052 conn(0x7feb18075010 msgr2=0x7feb18075410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:42:56.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.645+0000 7feb1ca00700 1 -- 172.21.15.62:0/1593889052 shutdown_connections 2024-04-20T21:42:56.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.645+0000 7feb1ca00700 1 -- 172.21.15.62:0/1593889052 wait complete. 2024-04-20T21:42:56.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.646+0000 7feb1ca00700 1 Processor -- start 2024-04-20T21:42:56.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.646+0000 7feb1ca00700 1 -- start start 2024-04-20T21:42:56.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.646+0000 7feb1ca00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb180ffd80 0x7feb1819ba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:42:56.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.646+0000 7feb1ca00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb18100720 0x7feb1819bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:42:56.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.646+0000 7feb1ca00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb1819c600 con 0x7feb18100720 2024-04-20T21:42:56.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.646+0000 7feb1ca00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb18195ab0 con 0x7feb180ffd80 2024-04-20T21:42:56.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.647+0000 7feb15400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb18100720 0x7feb1819bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:42:56.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.647+0000 7feb15400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb18100720 0x7feb1819bf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43912/0 (socket says 172.21.15.62:43912) 2024-04-20T21:42:56.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.647+0000 7feb15400700 1 -- 172.21.15.62:0/2364640552 learned_addr learned my addr 172.21.15.62:0/2364640552 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:42:56.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.647+0000 7feb15e00700 1 --2- 172.21.15.62:0/2364640552 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb180ffd80 0x7feb1819ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:42:56.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.647+0000 7feb15400700 1 -- 172.21.15.62:0/2364640552 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb180ffd80 msgr2=0x7feb1819ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:42:56.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.647+0000 7feb15400700 1 --2- 172.21.15.62:0/2364640552 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb180ffd80 0x7feb1819ba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:42:56.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.647+0000 7feb15400700 1 -- 172.21.15.62:0/2364640552 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb080095f0 con 0x7feb18100720 2024-04-20T21:42:56.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.647+0000 7feb15400700 1 --2- 172.21.15.62:0/2364640552 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb18100720 0x7feb1819bf70 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7feb0400c8f0 tx=0x7feb0400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:42:56.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.648+0000 7feb0ea00700 1 -- 172.21.15.62:0/2364640552 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feb04007a50 con 0x7feb18100720 2024-04-20T21:42:56.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.648+0000 7feb0ea00700 1 -- 172.21.15.62:0/2364640552 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7feb04007bb0 con 0x7feb18100720 2024-04-20T21:42:56.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.648+0000 7feb1ca00700 1 -- 172.21.15.62:0/2364640552 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb18195d90 con 0x7feb18100720 2024-04-20T21:42:56.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.648+0000 7feb0ea00700 1 -- 172.21.15.62:0/2364640552 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feb04018630 con 0x7feb18100720 2024-04-20T21:42:56.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.648+0000 7feb1ca00700 1 -- 172.21.15.62:0/2364640552 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb18196360 con 0x7feb18100720 2024-04-20T21:42:56.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.649+0000 7feb1ca00700 1 -- 172.21.15.62:0/2364640552 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb1804f2a0 con 0x7feb18100720 2024-04-20T21:42:56.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.650+0000 7feb0ea00700 1 -- 172.21.15.62:0/2364640552 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7feb04007d20 con 0x7feb18100720 2024-04-20T21:42:56.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.651+0000 7feb0ea00700 1 --2- 172.21.15.62:0/2364640552 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7feafc06ecc0 0x7feafc071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:42:56.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.651+0000 7feb0ea00700 1 -- 172.21.15.62:0/2364640552 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7feb0408eef0 con 0x7feb18100720 2024-04-20T21:42:56.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.651+0000 7feb15e00700 1 --2- 172.21.15.62:0/2364640552 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7feafc06ecc0 0x7feafc071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:42:56.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.652+0000 7feb15e00700 1 --2- 172.21.15.62:0/2364640552 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7feafc06ecc0 0x7feafc071170 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7feb08000c00 tx=0x7feb08005fd0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:42:56.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.656+0000 7feb0ea00700 1 -- 172.21.15.62:0/2364640552 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7feb0405b730 con 0x7feb18100720 2024-04-20T21:42:56.784 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:56 smithi062 ceph-mon[76322]: from='client.34488 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:42:56.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:56 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/1728102477' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T21:42:56.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:56 smithi189 ceph-mon[57022]: from='client.34488 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:42:56.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:56 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/1728102477' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T21:42:56.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.995+0000 7feb1ca00700 1 -- 172.21.15.62:0/2364640552 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7feb181967e0 con 0x7feb18100720 2024-04-20T21:42:56.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.996+0000 7feb0ea00700 1 -- 172.21.15.62:0/2364640552 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7feb0405b2c0 con 0x7feb18100720 2024-04-20T21:42:56.995 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-20T21:42:56.996 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-20T21:42:56.996 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is degraded 2024-04-20T21:42:56.996 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T21:42:56.996 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T21:42:56.996 INFO:teuthology.orchestra.run.smithi062.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-20T21:42:56.996 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-20T21:42:56.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.999+0000 7feb1ca00700 1 -- 172.21.15.62:0/2364640552 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7feafc06ecc0 msgr2=0x7feafc071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:42:56.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.999+0000 7feb1ca00700 1 --2- 172.21.15.62:0/2364640552 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7feafc06ecc0 0x7feafc071170 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7feb08000c00 tx=0x7feb08005fd0 comp rx=0 tx=0).stop 2024-04-20T21:42:56.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.999+0000 7feb1ca00700 1 -- 172.21.15.62:0/2364640552 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb18100720 msgr2=0x7feb1819bf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:42:56.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.999+0000 7feb1ca00700 1 --2- 172.21.15.62:0/2364640552 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb18100720 0x7feb1819bf70 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7feb0400c8f0 tx=0x7feb0400ccb0 comp rx=0 tx=0).stop 2024-04-20T21:42:56.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.999+0000 7feb1ca00700 1 -- 172.21.15.62:0/2364640552 shutdown_connections 2024-04-20T21:42:56.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.999+0000 7feb1ca00700 1 --2- 172.21.15.62:0/2364640552 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7feafc06ecc0 0x7feafc071170 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:42:56.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.999+0000 7feb1ca00700 1 --2- 172.21.15.62:0/2364640552 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb180ffd80 0x7feb1819ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:42:56.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.999+0000 7feb1ca00700 1 --2- 172.21.15.62:0/2364640552 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb18100720 0x7feb1819bf70 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:42:56.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:56.999+0000 7feb1ca00700 1 -- 172.21.15.62:0/2364640552 >> 172.21.15.62:0/2364640552 conn(0x7feb18075010 msgr2=0x7feb181033c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:42:56.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:57.000+0000 7feb1ca00700 1 -- 172.21.15.62:0/2364640552 shutdown_connections 2024-04-20T21:42:57.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:42:57.000+0000 7feb1ca00700 1 -- 172.21.15.62:0/2364640552 wait complete. 2024-04-20T21:42:57.959 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:57 smithi189 ceph-mon[57022]: pgmap v229: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:42:57.960 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:57 smithi189 ceph-mon[57022]: from='client.34500 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:42:57.960 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:57 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/2364640552' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T21:42:57.990 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:57 smithi062 ceph-mon[76322]: pgmap v229: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:42:57.990 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:57 smithi062 ceph-mon[76322]: from='client.34500 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:42:57.990 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:57 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/2364640552' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T21:42:59.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:42:59 smithi189 ceph-mon[57022]: pgmap v230: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:43:00.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:42:59 smithi062 ceph-mon[76322]: pgmap v230: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:43:01.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:43:01 smithi189 ceph-mon[57022]: pgmap v231: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:43:02.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:43:01 smithi062 ceph-mon[76322]: pgmap v231: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:43:03.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:43:03 smithi189 ceph-mon[57022]: pgmap v232: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:43:04.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:43:03 smithi062 ceph-mon[76322]: pgmap v232: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:43:04.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:43:04 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:43:05.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:43:04 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:43:06.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:43:05 smithi062 ceph-mon[76322]: pgmap v233: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:43:06.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:43:05 smithi189 ceph-mon[57022]: pgmap v233: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:43:08.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:43:07 smithi062 ceph-mon[76322]: pgmap v234: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:43:08.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:43:07 smithi189 ceph-mon[57022]: pgmap v234: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:43:10.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:43:09 smithi062 ceph-mon[76322]: pgmap v235: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:43:10.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:43:09 smithi189 ceph-mon[57022]: pgmap v235: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:43:12.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:43:11 smithi062 ceph-mon[76322]: pgmap v236: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:43:12.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:43:11 smithi189 ceph-mon[57022]: pgmap v236: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:43:14.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:43:13 smithi062 ceph-mon[76322]: pgmap v237: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:43:14.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:43:13 smithi189 ceph-mon[57022]: pgmap v237: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:43:16.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:43:15 smithi062 ceph-mon[76322]: pgmap v238: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:43:16.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:43:15 smithi189 ceph-mon[57022]: pgmap v238: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:43:18.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:43:17 smithi062 ceph-mon[76322]: pgmap v239: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:43:18.236 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:43:17 smithi189 ceph-mon[57022]: pgmap v239: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:43:20.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:43:19 smithi062 ceph-mon[76322]: pgmap v240: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:43:20.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:43:19 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:43:20.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:43:19 smithi189 ceph-mon[57022]: pgmap v240: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:43:20.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:43:19 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:43:22.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:43:21 smithi189 ceph-mon[57022]: pgmap v241: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:43:22.284 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:43:21 smithi062 ceph-mon[76322]: pgmap v241: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:43:24.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:43:23 smithi189 ceph-mon[57022]: pgmap v242: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:43:24.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:43:23 smithi062 ceph-mon[76322]: pgmap v242: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:43:25.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:43:25 smithi189 ceph-mon[57022]: pgmap v243: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:43:25.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:43:25 smithi062 ceph-mon[76322]: pgmap v243: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:43:27.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:27.180+0000 7f89eaa00700 1 -- 172.21.15.62:0/878702946 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f89e4068540 msgr2=0x7f89e40689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:43:27.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:27.180+0000 7f89eaa00700 1 --2- 172.21.15.62:0/878702946 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f89e4068540 0x7f89e40689b0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f89cc009910 tx=0x7f89cc009c20 comp rx=0 tx=0).stop 2024-04-20T21:43:27.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:27.180+0000 7f89eaa00700 1 -- 172.21.15.62:0/878702946 shutdown_connections 2024-04-20T21:43:27.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:27.180+0000 7f89eaa00700 1 --2- 172.21.15.62:0/878702946 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f89e4068540 0x7f89e40689b0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:43:27.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:27.180+0000 7f89eaa00700 1 --2- 172.21.15.62:0/878702946 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89e41085b0 0x7f89e4108980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:43:27.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:27.180+0000 7f89eaa00700 1 -- 172.21.15.62:0/878702946 >> 172.21.15.62:0/878702946 conn(0x7f89e4075130 msgr2=0x7f89e4075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:43:27.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:27.180+0000 7f89eaa00700 1 -- 172.21.15.62:0/878702946 shutdown_connections 2024-04-20T21:43:27.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:27.180+0000 7f89eaa00700 1 -- 172.21.15.62:0/878702946 wait complete. 2024-04-20T21:43:27.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:27.181+0000 7f89eaa00700 1 Processor -- start 2024-04-20T21:43:27.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:27.182+0000 7f89eaa00700 1 -- start start 2024-04-20T21:43:27.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:27.182+0000 7f89eaa00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89e4068540 0x7f89e41980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:43:27.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:27.182+0000 7f89eaa00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f89e41085b0 0x7f89e41985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:43:27.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:27.182+0000 7f89eaa00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89e4198c40 con 0x7f89e41085b0 2024-04-20T21:43:27.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:27.182+0000 7f89eaa00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89e4198d80 con 0x7f89e4068540 2024-04-20T21:43:27.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:27.182+0000 7f89db400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f89e41085b0 0x7f89e41985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:43:27.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:27.182+0000 7f89db400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f89e41085b0 0x7f89e41985f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48560/0 (socket says 172.21.15.62:48560) 2024-04-20T21:43:27.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:27.182+0000 7f89db400700 1 -- 172.21.15.62:0/3409373371 learned_addr learned my addr 172.21.15.62:0/3409373371 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:43:27.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:27.183+0000 7f89db400700 1 -- 172.21.15.62:0/3409373371 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89e4068540 msgr2=0x7f89e41980b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:43:27.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:27.183+0000 7f89db400700 1 --2- 172.21.15.62:0/3409373371 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89e4068540 0x7f89e41980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:43:27.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:27.183+0000 7f89db400700 1 -- 172.21.15.62:0/3409373371 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89cc0095f0 con 0x7f89e41085b0 2024-04-20T21:43:27.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:27.183+0000 7f89db400700 1 --2- 172.21.15.62:0/3409373371 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f89e41085b0 0x7f89e41985f0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f89cc00b5c0 tx=0x7f89cc00be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:43:27.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:27.183+0000 7f89e1600700 1 -- 172.21.15.62:0/3409373371 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f89cc01d070 con 0x7f89e41085b0 2024-04-20T21:43:27.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:27.183+0000 7f89e1600700 1 -- 172.21.15.62:0/3409373371 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f89cc00f460 con 0x7f89e41085b0 2024-04-20T21:43:27.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:27.184+0000 7f89eaa00700 1 -- 172.21.15.62:0/3409373371 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89e419cb70 con 0x7f89e41085b0 2024-04-20T21:43:27.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:27.184+0000 7f89e1600700 1 -- 172.21.15.62:0/3409373371 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f89cc017610 con 0x7f89e41085b0 2024-04-20T21:43:27.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:27.184+0000 7f89eaa00700 1 -- 172.21.15.62:0/3409373371 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89e419d060 con 0x7f89e41085b0 2024-04-20T21:43:27.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:27.185+0000 7f89eaa00700 1 -- 172.21.15.62:0/3409373371 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89e404f2a0 con 0x7f89e41085b0 2024-04-20T21:43:27.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:27.186+0000 7f89e1600700 1 -- 172.21.15.62:0/3409373371 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f89cc00f5d0 con 0x7f89e41085b0 2024-04-20T21:43:27.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:27.187+0000 7f89e1600700 1 --2- 172.21.15.62:0/3409373371 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f89c406ecc0 0x7f89c4071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:43:27.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:27.187+0000 7f89e1600700 1 -- 172.21.15.62:0/3409373371 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f89cc090510 con 0x7f89e41085b0 2024-04-20T21:43:27.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:27.187+0000 7f89e3e00700 1 --2- 172.21.15.62:0/3409373371 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f89c406ecc0 0x7f89c4071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:43:27.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:27.188+0000 7f89e3e00700 1 --2- 172.21.15.62:0/3409373371 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f89c406ecc0 0x7f89c4071170 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f89e4101db0 tx=0x7f89d4005d20 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:43:27.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:27.192+0000 7f89e1600700 1 -- 172.21.15.62:0/3409373371 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f89cc059220 con 0x7f89e41085b0 2024-04-20T21:43:27.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:27.453+0000 7f89eaa00700 1 -- 172.21.15.62:0/3409373371 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f89e410a480 con 0x7f89c406ecc0 2024-04-20T21:43:27.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:27.454+0000 7f89e1600700 1 -- 172.21.15.62:0/3409373371 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f89e410a480 con 0x7f89c406ecc0 2024-04-20T21:43:27.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:27.457+0000 7f89eaa00700 1 -- 172.21.15.62:0/3409373371 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f89c406ecc0 msgr2=0x7f89c4071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:43:27.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:27.457+0000 7f89eaa00700 1 --2- 172.21.15.62:0/3409373371 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f89c406ecc0 0x7f89c4071170 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f89e4101db0 tx=0x7f89d4005d20 comp rx=0 tx=0).stop 2024-04-20T21:43:27.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:27.457+0000 7f89eaa00700 1 -- 172.21.15.62:0/3409373371 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f89e41085b0 msgr2=0x7f89e41985f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:43:27.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:27.457+0000 7f89eaa00700 1 --2- 172.21.15.62:0/3409373371 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f89e41085b0 0x7f89e41985f0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f89cc00b5c0 tx=0x7f89cc00be00 comp rx=0 tx=0).stop 2024-04-20T21:43:27.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:27.457+0000 7f89eaa00700 1 -- 172.21.15.62:0/3409373371 shutdown_connections 2024-04-20T21:43:27.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:27.457+0000 7f89eaa00700 1 --2- 172.21.15.62:0/3409373371 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f89c406ecc0 0x7f89c4071170 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:43:27.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:27.457+0000 7f89eaa00700 1 --2- 172.21.15.62:0/3409373371 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89e4068540 0x7f89e41980b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:43:27.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:27.457+0000 7f89eaa00700 1 --2- 172.21.15.62:0/3409373371 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f89e41085b0 0x7f89e41985f0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:43:27.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:27.457+0000 7f89eaa00700 1 -- 172.21.15.62:0/3409373371 >> 172.21.15.62:0/3409373371 conn(0x7f89e4075130 msgr2=0x7f89e4106390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:43:27.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:27.458+0000 7f89eaa00700 1 -- 172.21.15.62:0/3409373371 shutdown_connections 2024-04-20T21:43:27.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:27.458+0000 7f89eaa00700 1 -- 172.21.15.62:0/3409373371 wait complete. 2024-04-20T21:43:27.474 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T21:43:27.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:27.624+0000 7f1769200700 1 -- 172.21.15.62:0/1914684414 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f17640739a0 msgr2=0x7f176410c5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:43:27.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:27.624+0000 7f1769200700 1 --2- 172.21.15.62:0/1914684414 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f17640739a0 0x7f176410c5e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f1750009940 tx=0x7f1750009c50 comp rx=0 tx=0).stop 2024-04-20T21:43:27.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:27.624+0000 7f1769200700 1 -- 172.21.15.62:0/1914684414 shutdown_connections 2024-04-20T21:43:27.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:27.624+0000 7f1769200700 1 --2- 172.21.15.62:0/1914684414 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f17640739a0 0x7f176410c5e0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:43:27.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:27.624+0000 7f1769200700 1 --2- 172.21.15.62:0/1914684414 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1764073090 0x7f1764073460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:43:27.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:27.624+0000 7f1769200700 1 -- 172.21.15.62:0/1914684414 >> 172.21.15.62:0/1914684414 conn(0x7f17640fbda0 msgr2=0x7f17640fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:43:27.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:27.624+0000 7f1769200700 1 -- 172.21.15.62:0/1914684414 shutdown_connections 2024-04-20T21:43:27.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:27.624+0000 7f1769200700 1 -- 172.21.15.62:0/1914684414 wait complete. 2024-04-20T21:43:27.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:27.626+0000 7f1769200700 1 Processor -- start 2024-04-20T21:43:27.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:27.626+0000 7f1769200700 1 -- start start 2024-04-20T21:43:27.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:27.626+0000 7f1769200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1764073090 0x7f17641981b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:43:27.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:27.626+0000 7f1769200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f17640739a0 0x7f17641986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:43:27.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:27.626+0000 7f1769200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1764198dd0 con 0x7f17640739a0 2024-04-20T21:43:27.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:27.626+0000 7f1769200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f176419cb60 con 0x7f1764073090 2024-04-20T21:43:27.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:27.627+0000 7f1761400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f17640739a0 0x7f17641986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:43:27.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:27.627+0000 7f1761400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f17640739a0 0x7f17641986f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48572/0 (socket says 172.21.15.62:48572) 2024-04-20T21:43:27.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:27.627+0000 7f1761400700 1 -- 172.21.15.62:0/2739034703 learned_addr learned my addr 172.21.15.62:0/2739034703 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:43:27.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:27.627+0000 7f1761e00700 1 --2- 172.21.15.62:0/2739034703 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1764073090 0x7f17641981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:43:27.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:27.627+0000 7f1761400700 1 -- 172.21.15.62:0/2739034703 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1764073090 msgr2=0x7f17641981b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:43:27.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:27.627+0000 7f1761400700 1 --2- 172.21.15.62:0/2739034703 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1764073090 0x7f17641981b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:43:27.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:27.627+0000 7f1761400700 1 -- 172.21.15.62:0/2739034703 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17500095f0 con 0x7f17640739a0 2024-04-20T21:43:27.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:27.628+0000 7f1761400700 1 --2- 172.21.15.62:0/2739034703 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f17640739a0 0x7f17641986f0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f17500055f0 tx=0x7f17500056d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:43:27.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:27.628+0000 7f175aa00700 1 -- 172.21.15.62:0/2739034703 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f175001d070 con 0x7f17640739a0 2024-04-20T21:43:27.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:27.628+0000 7f175aa00700 1 -- 172.21.15.62:0/2739034703 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f175000f460 con 0x7f17640739a0 2024-04-20T21:43:27.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:27.628+0000 7f175aa00700 1 -- 172.21.15.62:0/2739034703 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1750021640 con 0x7f17640739a0 2024-04-20T21:43:27.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:27.628+0000 7f1769200700 1 -- 172.21.15.62:0/2739034703 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f176419cde0 con 0x7f17640739a0 2024-04-20T21:43:27.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:27.628+0000 7f1769200700 1 -- 172.21.15.62:0/2739034703 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f176419d2d0 con 0x7f17640739a0 2024-04-20T21:43:27.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:27.629+0000 7f1769200700 1 -- 172.21.15.62:0/2739034703 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1764109ce0 con 0x7f17640739a0 2024-04-20T21:43:27.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:27.630+0000 7f175aa00700 1 -- 172.21.15.62:0/2739034703 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f175000fa60 con 0x7f17640739a0 2024-04-20T21:43:27.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:27.632+0000 7f175aa00700 1 --2- 172.21.15.62:0/2739034703 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f174806ecc0 0x7f1748071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:43:27.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:27.632+0000 7f175aa00700 1 -- 172.21.15.62:0/2739034703 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f175008ffe0 con 0x7f17640739a0 2024-04-20T21:43:27.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:27.632+0000 7f1761e00700 1 --2- 172.21.15.62:0/2739034703 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f174806ecc0 0x7f1748071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:43:27.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:27.633+0000 7f1761e00700 1 --2- 172.21.15.62:0/2739034703 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f174806ecc0 0x7f1748071170 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f1754005950 tx=0x7f17540058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:43:27.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:27.637+0000 7f175aa00700 1 -- 172.21.15.62:0/2739034703 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f1750058dd0 con 0x7f17640739a0 2024-04-20T21:43:27.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:43:27 smithi189 ceph-mon[57022]: pgmap v244: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:43:27.784 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:43:27 smithi062 ceph-mon[76322]: pgmap v244: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:43:27.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:27.895+0000 7f1769200700 1 -- 172.21.15.62:0/2739034703 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f176419d540 con 0x7f174806ecc0 2024-04-20T21:43:27.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:27.898+0000 7f175aa00700 1 -- 172.21.15.62:0/2739034703 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f176419d540 con 0x7f174806ecc0 2024-04-20T21:43:27.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:27.901+0000 7f1769200700 1 -- 172.21.15.62:0/2739034703 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f174806ecc0 msgr2=0x7f1748071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:43:27.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:27.901+0000 7f1769200700 1 --2- 172.21.15.62:0/2739034703 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f174806ecc0 0x7f1748071170 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f1754005950 tx=0x7f17540058e0 comp rx=0 tx=0).stop 2024-04-20T21:43:27.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:27.901+0000 7f1769200700 1 -- 172.21.15.62:0/2739034703 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f17640739a0 msgr2=0x7f17641986f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:43:27.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:27.901+0000 7f1769200700 1 --2- 172.21.15.62:0/2739034703 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f17640739a0 0x7f17641986f0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f17500055f0 tx=0x7f17500056d0 comp rx=0 tx=0).stop 2024-04-20T21:43:27.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:27.901+0000 7f1769200700 1 -- 172.21.15.62:0/2739034703 shutdown_connections 2024-04-20T21:43:27.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:27.901+0000 7f1769200700 1 --2- 172.21.15.62:0/2739034703 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f174806ecc0 0x7f1748071170 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:43:27.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:27.901+0000 7f1769200700 1 --2- 172.21.15.62:0/2739034703 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1764073090 0x7f17641981b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:43:27.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:27.901+0000 7f1769200700 1 --2- 172.21.15.62:0/2739034703 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f17640739a0 0x7f17641986f0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:43:27.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:27.901+0000 7f1769200700 1 -- 172.21.15.62:0/2739034703 >> 172.21.15.62:0/2739034703 conn(0x7f17640fbda0 msgr2=0x7f1764106e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:43:27.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:27.901+0000 7f1769200700 1 -- 172.21.15.62:0/2739034703 shutdown_connections 2024-04-20T21:43:27.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:27.901+0000 7f1769200700 1 -- 172.21.15.62:0/2739034703 wait complete. 2024-04-20T21:43:28.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:28.066+0000 7fd6fea00700 1 -- 172.21.15.62:0/583588431 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6f81012f0 msgr2=0x7fd6f81016c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:43:28.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:28.066+0000 7fd6fea00700 1 --2- 172.21.15.62:0/583588431 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6f81012f0 0x7fd6f81016c0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fd6e8009910 tx=0x7fd6e8009c20 comp rx=0 tx=0).stop 2024-04-20T21:43:28.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:28.067+0000 7fd6fea00700 1 -- 172.21.15.62:0/583588431 shutdown_connections 2024-04-20T21:43:28.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:28.067+0000 7fd6fea00700 1 --2- 172.21.15.62:0/583588431 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6f8101c90 0x7fd6f810a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:43:28.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:28.067+0000 7fd6fea00700 1 --2- 172.21.15.62:0/583588431 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6f81012f0 0x7fd6f81016c0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:43:28.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:28.067+0000 7fd6fea00700 1 -- 172.21.15.62:0/583588431 >> 172.21.15.62:0/583588431 conn(0x7fd6f80fac80 msgr2=0x7fd6f80fd090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:43:28.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:28.067+0000 7fd6fea00700 1 -- 172.21.15.62:0/583588431 shutdown_connections 2024-04-20T21:43:28.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:28.067+0000 7fd6fea00700 1 -- 172.21.15.62:0/583588431 wait complete. 2024-04-20T21:43:28.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:28.068+0000 7fd6fea00700 1 Processor -- start 2024-04-20T21:43:28.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:28.068+0000 7fd6fea00700 1 -- start start 2024-04-20T21:43:28.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:28.069+0000 7fd6fea00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6f81012f0 0x7fd6f80ff120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:43:28.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:28.069+0000 7fd6fea00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6f8101c90 0x7fd6f80ff660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:43:28.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:28.069+0000 7fd6fea00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6f80ffba0 con 0x7fd6f8101c90 2024-04-20T21:43:28.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:28.069+0000 7fd6fea00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6f80ffce0 con 0x7fd6f81012f0 2024-04-20T21:43:28.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:28.069+0000 7fd6f7400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6f8101c90 0x7fd6f80ff660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:43:28.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:28.069+0000 7fd6f7400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6f8101c90 0x7fd6f80ff660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48588/0 (socket says 172.21.15.62:48588) 2024-04-20T21:43:28.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:28.069+0000 7fd6f7400700 1 -- 172.21.15.62:0/3558428312 learned_addr learned my addr 172.21.15.62:0/3558428312 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:43:28.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:28.069+0000 7fd6f7e00700 1 --2- 172.21.15.62:0/3558428312 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6f81012f0 0x7fd6f80ff120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:43:28.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:28.069+0000 7fd6f7400700 1 -- 172.21.15.62:0/3558428312 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6f81012f0 msgr2=0x7fd6f80ff120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:43:28.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:28.069+0000 7fd6f7400700 1 --2- 172.21.15.62:0/3558428312 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6f81012f0 0x7fd6f80ff120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:43:28.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:28.069+0000 7fd6f7400700 1 -- 172.21.15.62:0/3558428312 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6e80095f0 con 0x7fd6f8101c90 2024-04-20T21:43:28.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:28.070+0000 7fd6f7400700 1 --2- 172.21.15.62:0/3558428312 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6f8101c90 0x7fd6f80ff660 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fd6e400c930 tx=0x7fd6e400ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:43:28.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:28.070+0000 7fd6f4c00700 1 -- 172.21.15.62:0/3558428312 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd6e4007ad0 con 0x7fd6f8101c90 2024-04-20T21:43:28.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:28.070+0000 7fd6f4c00700 1 -- 172.21.15.62:0/3558428312 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd6e4007c30 con 0x7fd6f8101c90 2024-04-20T21:43:28.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:28.070+0000 7fd6fea00700 1 -- 172.21.15.62:0/3558428312 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6f81a2070 con 0x7fd6f8101c90 2024-04-20T21:43:28.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:28.070+0000 7fd6f4c00700 1 -- 172.21.15.62:0/3558428312 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd6e4018610 con 0x7fd6f8101c90 2024-04-20T21:43:28.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:28.070+0000 7fd6fea00700 1 -- 172.21.15.62:0/3558428312 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6f81a23d0 con 0x7fd6f8101c90 2024-04-20T21:43:28.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:28.071+0000 7fd6fea00700 1 -- 172.21.15.62:0/3558428312 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6f81079d0 con 0x7fd6f8101c90 2024-04-20T21:43:28.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:28.074+0000 7fd6f4c00700 1 -- 172.21.15.62:0/3558428312 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fd6e400f450 con 0x7fd6f8101c90 2024-04-20T21:43:28.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:28.074+0000 7fd6f4c00700 1 --2- 172.21.15.62:0/3558428312 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd6dc06ecc0 0x7fd6dc071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:43:28.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:28.075+0000 7fd6f4c00700 1 -- 172.21.15.62:0/3558428312 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fd6e408f200 con 0x7fd6f8101c90 2024-04-20T21:43:28.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:28.075+0000 7fd6f7e00700 1 --2- 172.21.15.62:0/3558428312 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd6dc06ecc0 0x7fd6dc071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:43:28.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:28.076+0000 7fd6f7e00700 1 --2- 172.21.15.62:0/3558428312 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd6dc06ecc0 0x7fd6dc071170 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fd6e800b5c0 tx=0x7fd6e8005fd0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:43:28.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:28.080+0000 7fd6f4c00700 1 -- 172.21.15.62:0/3558428312 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fd6e4058010 con 0x7fd6f8101c90 2024-04-20T21:43:28.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:28.319+0000 7fd6fea00700 1 -- 172.21.15.62:0/3558428312 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fd6f8061190 con 0x7fd6dc06ecc0 2024-04-20T21:43:28.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:28.345+0000 7fd6f4c00700 1 -- 172.21.15.62:0/3558428312 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7fd6f8061190 con 0x7fd6dc06ecc0 2024-04-20T21:43:28.345 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T21:43:28.345 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (9m) 40s ago 23m 14.2M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T21:43:28.345 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (23m) 40s ago 23m 9512k - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T21:43:28.345 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (22m) 97s ago 22m 8942k - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T21:43:28.345 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (5m) 40s ago 23m 6660k - 19.0.0-3133-ga8c84f59 76bc6b91b726 2825d5ba11c6 2024-04-20T21:43:28.345 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (5m) 97s ago 22m 6665k - 19.0.0-3133-ga8c84f59 76bc6b91b726 20f746fa1bba 2024-04-20T21:43:28.345 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (8m) 40s ago 22m 74.2M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T21:43:28.345 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (48s) 40s ago 15m 11.2M - 19.0.0-3133-ga8c84f59 76bc6b91b726 de80bc90a683 2024-04-20T21:43:28.345 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (76s) 40s ago 16m 11.9M - 19.0.0-3133-ga8c84f59 76bc6b91b726 cd793b7ee98e 2024-04-20T21:43:28.345 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (15m) 97s ago 15m 117M - 18.2.0 dc2bc1663786 fb2a8a223db8 2024-04-20T21:43:28.345 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (15m) 97s ago 15m 55.6M - 18.2.0 dc2bc1663786 f2eec2ee8951 2024-04-20T21:43:28.345 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (12m) 40s ago 24m 511M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T21:43:28.345 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (10m) 97s ago 21m 452M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T21:43:28.346 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (7m) 40s ago 25m 55.9M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 019eb1479485 2024-04-20T21:43:28.346 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (6m) 97s ago 21m 39.8M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 55aac560aab6 2024-04-20T21:43:28.346 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (10m) 40s ago 23m 7952k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T21:43:28.346 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (10m) 97s ago 21m 8304k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T21:43:28.346 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (4m) 40s ago 20m 88.0M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 ee2bb901f26e 2024-04-20T21:43:28.346 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (4m) 40s ago 20m 92.3M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 9075152d4162 2024-04-20T21:43:28.346 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (3m) 40s ago 19m 74.4M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 2d4f3e38dc0e 2024-04-20T21:43:28.346 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (3m) 97s ago 19m 87.6M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 8f97b17ae0eb 2024-04-20T21:43:28.346 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (2m) 97s ago 18m 90.7M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 653d8079967d 2024-04-20T21:43:28.346 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (102s) 97s ago 17m 11.5M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 596dded14c12 2024-04-20T21:43:28.346 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (9m) 40s ago 22m 74.0M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T21:43:28.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:28.348+0000 7fd6fea00700 1 -- 172.21.15.62:0/3558428312 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd6dc06ecc0 msgr2=0x7fd6dc071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:43:28.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:28.348+0000 7fd6fea00700 1 --2- 172.21.15.62:0/3558428312 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd6dc06ecc0 0x7fd6dc071170 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fd6e800b5c0 tx=0x7fd6e8005fd0 comp rx=0 tx=0).stop 2024-04-20T21:43:28.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:28.348+0000 7fd6fea00700 1 -- 172.21.15.62:0/3558428312 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6f8101c90 msgr2=0x7fd6f80ff660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:43:28.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:28.348+0000 7fd6fea00700 1 --2- 172.21.15.62:0/3558428312 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6f8101c90 0x7fd6f80ff660 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fd6e400c930 tx=0x7fd6e400ccf0 comp rx=0 tx=0).stop 2024-04-20T21:43:28.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:28.348+0000 7fd6fea00700 1 -- 172.21.15.62:0/3558428312 shutdown_connections 2024-04-20T21:43:28.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:28.348+0000 7fd6fea00700 1 --2- 172.21.15.62:0/3558428312 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd6dc06ecc0 0x7fd6dc071170 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:43:28.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:28.348+0000 7fd6fea00700 1 --2- 172.21.15.62:0/3558428312 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6f81012f0 0x7fd6f80ff120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:43:28.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:28.348+0000 7fd6fea00700 1 --2- 172.21.15.62:0/3558428312 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6f8101c90 0x7fd6f80ff660 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:43:28.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:28.348+0000 7fd6fea00700 1 -- 172.21.15.62:0/3558428312 >> 172.21.15.62:0/3558428312 conn(0x7fd6f80fac80 msgr2=0x7fd6f8104b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:43:28.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:28.349+0000 7fd6fea00700 1 -- 172.21.15.62:0/3558428312 shutdown_connections 2024-04-20T21:43:28.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:28.349+0000 7fd6fea00700 1 -- 172.21.15.62:0/3558428312 wait complete. 2024-04-20T21:43:28.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:28.507+0000 7fd613200700 1 -- 172.21.15.62:0/986911329 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd614106b80 msgr2=0x7fd614106ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:43:28.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:28.507+0000 7fd613200700 1 --2- 172.21.15.62:0/986911329 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd614106b80 0x7fd614106ff0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fd600009910 tx=0x7fd600009c20 comp rx=0 tx=0).stop 2024-04-20T21:43:28.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:28.507+0000 7fd613200700 1 -- 172.21.15.62:0/986911329 shutdown_connections 2024-04-20T21:43:28.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:28.507+0000 7fd613200700 1 --2- 172.21.15.62:0/986911329 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd614106b80 0x7fd614106ff0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:43:28.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:28.507+0000 7fd613200700 1 --2- 172.21.15.62:0/986911329 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd61410c660 0x7fd61410ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:43:28.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:28.508+0000 7fd613200700 1 -- 172.21.15.62:0/986911329 >> 172.21.15.62:0/986911329 conn(0x7fd614075e90 msgr2=0x7fd6140782a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:43:28.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:28.508+0000 7fd613200700 1 -- 172.21.15.62:0/986911329 shutdown_connections 2024-04-20T21:43:28.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:28.508+0000 7fd613200700 1 -- 172.21.15.62:0/986911329 wait complete. 2024-04-20T21:43:28.508 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:28.509+0000 7fd613200700 1 Processor -- start 2024-04-20T21:43:28.508 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:28.509+0000 7fd613200700 1 -- start start 2024-04-20T21:43:28.508 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:28.509+0000 7fd613200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd614106b80 0x7fd61419c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:43:28.508 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:28.509+0000 7fd613200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd61410c660 0x7fd61419c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:43:28.508 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:28.509+0000 7fd613200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd61419ce00 con 0x7fd614106b80 2024-04-20T21:43:28.508 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:28.509+0000 7fd613200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6141a0b90 con 0x7fd61410c660 2024-04-20T21:43:28.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:28.510+0000 7fd611e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd614106b80 0x7fd61419c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:43:28.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:28.510+0000 7fd611e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd614106b80 0x7fd61419c1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48606/0 (socket says 172.21.15.62:48606) 2024-04-20T21:43:28.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:28.510+0000 7fd611e00700 1 -- 172.21.15.62:0/1695084454 learned_addr learned my addr 172.21.15.62:0/1695084454 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:43:28.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:28.510+0000 7fd611400700 1 --2- 172.21.15.62:0/1695084454 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd61410c660 0x7fd61419c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:43:28.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:28.510+0000 7fd611e00700 1 -- 172.21.15.62:0/1695084454 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd61410c660 msgr2=0x7fd61419c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:43:28.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:28.510+0000 7fd611e00700 1 --2- 172.21.15.62:0/1695084454 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd61410c660 0x7fd61419c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:43:28.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:28.510+0000 7fd611e00700 1 -- 172.21.15.62:0/1695084454 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6000095f0 con 0x7fd614106b80 2024-04-20T21:43:28.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:28.510+0000 7fd611400700 1 --2- 172.21.15.62:0/1695084454 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd61410c660 0x7fd61419c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T21:43:28.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:28.510+0000 7fd611e00700 1 --2- 172.21.15.62:0/1695084454 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd614106b80 0x7fd61419c1e0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fd60400c8a0 tx=0x7fd60400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:43:28.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:28.511+0000 7fd60aa00700 1 -- 172.21.15.62:0/1695084454 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd60400cea0 con 0x7fd614106b80 2024-04-20T21:43:28.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:28.511+0000 7fd60aa00700 1 -- 172.21.15.62:0/1695084454 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd604004830 con 0x7fd614106b80 2024-04-20T21:43:28.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:28.511+0000 7fd613200700 1 -- 172.21.15.62:0/1695084454 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6141a0e70 con 0x7fd614106b80 2024-04-20T21:43:28.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:28.511+0000 7fd60aa00700 1 -- 172.21.15.62:0/1695084454 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd6040055b0 con 0x7fd614106b80 2024-04-20T21:43:28.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:28.511+0000 7fd613200700 1 -- 172.21.15.62:0/1695084454 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6141a13c0 con 0x7fd614106b80 2024-04-20T21:43:28.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:28.512+0000 7fd613200700 1 -- 172.21.15.62:0/1695084454 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd61410eb40 con 0x7fd614106b80 2024-04-20T21:43:28.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:28.513+0000 7fd60aa00700 1 -- 172.21.15.62:0/1695084454 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fd6040049a0 con 0x7fd614106b80 2024-04-20T21:43:28.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:28.514+0000 7fd60aa00700 1 --2- 172.21.15.62:0/1695084454 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd5f806ecc0 0x7fd5f8071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:43:28.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:28.514+0000 7fd60aa00700 1 -- 172.21.15.62:0/1695084454 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fd60408eaf0 con 0x7fd614106b80 2024-04-20T21:43:28.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:28.515+0000 7fd611400700 1 --2- 172.21.15.62:0/1695084454 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd5f806ecc0 0x7fd5f8071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:43:28.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:28.515+0000 7fd611400700 1 --2- 172.21.15.62:0/1695084454 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd5f806ecc0 0x7fd5f8071170 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fd600006010 tx=0x7fd60000b540 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:43:28.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:28.519+0000 7fd60aa00700 1 -- 172.21.15.62:0/1695084454 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fd604057880 con 0x7fd614106b80 2024-04-20T21:43:28.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:43:28 smithi189 ceph-mon[57022]: from='client.34508 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:43:28.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:43:28 smithi189 ceph-mon[57022]: from='client.34512 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:43:28.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:43:28 smithi062 ceph-mon[76322]: from='client.34508 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:43:28.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:43:28 smithi062 ceph-mon[76322]: from='client.34512 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:43:28.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:28.865+0000 7fd613200700 1 -- 172.21.15.62:0/1695084454 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fd6141a1690 con 0x7fd614106b80 2024-04-20T21:43:28.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:28.866+0000 7fd60aa00700 1 -- 172.21.15.62:0/1695084454 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7fd60401d070 con 0x7fd614106b80 2024-04-20T21:43:28.865 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:43:28.865 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T21:43:28.865 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:43:28.866 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:43:28.866 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T21:43:28.866 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:43:28.866 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:43:28.866 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T21:43:28.866 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 6 2024-04-20T21:43:28.866 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:43:28.866 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T21:43:28.866 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 2, 2024-04-20T21:43:28.866 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:43:28.866 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:43:28.866 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T21:43:28.866 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 2, 2024-04-20T21:43:28.866 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 12 2024-04-20T21:43:28.867 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T21:43:28.867 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:43:28.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:28.869+0000 7fd613200700 1 -- 172.21.15.62:0/1695084454 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd5f806ecc0 msgr2=0x7fd5f8071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:43:28.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:28.869+0000 7fd613200700 1 --2- 172.21.15.62:0/1695084454 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd5f806ecc0 0x7fd5f8071170 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fd600006010 tx=0x7fd60000b540 comp rx=0 tx=0).stop 2024-04-20T21:43:28.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:28.869+0000 7fd613200700 1 -- 172.21.15.62:0/1695084454 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd614106b80 msgr2=0x7fd61419c1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:43:28.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:28.869+0000 7fd613200700 1 --2- 172.21.15.62:0/1695084454 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd614106b80 0x7fd61419c1e0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fd60400c8a0 tx=0x7fd60400cc60 comp rx=0 tx=0).stop 2024-04-20T21:43:28.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:28.869+0000 7fd613200700 1 -- 172.21.15.62:0/1695084454 shutdown_connections 2024-04-20T21:43:28.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:28.869+0000 7fd613200700 1 --2- 172.21.15.62:0/1695084454 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd5f806ecc0 0x7fd5f8071170 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:43:28.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:28.869+0000 7fd613200700 1 --2- 172.21.15.62:0/1695084454 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd614106b80 0x7fd61419c1e0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:43:28.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:28.869+0000 7fd613200700 1 --2- 172.21.15.62:0/1695084454 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd61410c660 0x7fd61419c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:43:28.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:28.869+0000 7fd613200700 1 -- 172.21.15.62:0/1695084454 >> 172.21.15.62:0/1695084454 conn(0x7fd614075e90 msgr2=0x7fd6140778d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:43:28.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:28.869+0000 7fd613200700 1 -- 172.21.15.62:0/1695084454 shutdown_connections 2024-04-20T21:43:28.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:28.869+0000 7fd613200700 1 -- 172.21.15.62:0/1695084454 wait complete. 2024-04-20T21:43:29.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.028+0000 7f2f71200700 1 -- 172.21.15.62:0/3904599768 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f6c0687a0 msgr2=0x7f2f6c068b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:43:29.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.028+0000 7f2f71200700 1 --2- 172.21.15.62:0/3904599768 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f6c0687a0 0x7f2f6c068b70 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f2f5c00b0a0 tx=0x7f2f5c00b3b0 comp rx=0 tx=0).stop 2024-04-20T21:43:29.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.029+0000 7f2f71200700 1 -- 172.21.15.62:0/3904599768 shutdown_connections 2024-04-20T21:43:29.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.029+0000 7f2f71200700 1 --2- 172.21.15.62:0/3904599768 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f6c069140 0x7f2f6c104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:43:29.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.029+0000 7f2f71200700 1 --2- 172.21.15.62:0/3904599768 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f6c0687a0 0x7f2f6c068b70 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:43:29.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.029+0000 7f2f71200700 1 -- 172.21.15.62:0/3904599768 >> 172.21.15.62:0/3904599768 conn(0x7f2f6c075130 msgr2=0x7f2f6c075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:43:29.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.029+0000 7f2f71200700 1 -- 172.21.15.62:0/3904599768 shutdown_connections 2024-04-20T21:43:29.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.029+0000 7f2f71200700 1 -- 172.21.15.62:0/3904599768 wait complete. 2024-04-20T21:43:29.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.030+0000 7f2f71200700 1 Processor -- start 2024-04-20T21:43:29.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.030+0000 7f2f71200700 1 -- start start 2024-04-20T21:43:29.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.030+0000 7f2f71200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f6c0687a0 0x7f2f6c198150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:43:29.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.030+0000 7f2f71200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f6c069140 0x7f2f6c198690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:43:29.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.030+0000 7f2f71200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f6c198d70 con 0x7f2f6c069140 2024-04-20T21:43:29.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.030+0000 7f2f71200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f6c19cb00 con 0x7f2f6c0687a0 2024-04-20T21:43:29.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.031+0000 7f2f61400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f6c069140 0x7f2f6c198690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:43:29.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.031+0000 7f2f61400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f6c069140 0x7f2f6c198690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48630/0 (socket says 172.21.15.62:48630) 2024-04-20T21:43:29.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.031+0000 7f2f61400700 1 -- 172.21.15.62:0/438838319 learned_addr learned my addr 172.21.15.62:0/438838319 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:43:29.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.031+0000 7f2f69e00700 1 --2- 172.21.15.62:0/438838319 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f6c0687a0 0x7f2f6c198150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:43:29.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.031+0000 7f2f61400700 1 -- 172.21.15.62:0/438838319 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f6c0687a0 msgr2=0x7f2f6c198150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:43:29.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.031+0000 7f2f61400700 1 --2- 172.21.15.62:0/438838319 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f6c0687a0 0x7f2f6c198150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:43:29.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.031+0000 7f2f61400700 1 -- 172.21.15.62:0/438838319 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f5c009d00 con 0x7f2f6c069140 2024-04-20T21:43:29.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.031+0000 7f2f61400700 1 --2- 172.21.15.62:0/438838319 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f6c069140 0x7f2f6c198690 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f2f5400b4b0 tx=0x7f2f5400b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:43:29.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.032+0000 7f2f63400700 1 -- 172.21.15.62:0/438838319 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2f54011800 con 0x7f2f6c069140 2024-04-20T21:43:29.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.032+0000 7f2f63400700 1 -- 172.21.15.62:0/438838319 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2f54011e20 con 0x7f2f6c069140 2024-04-20T21:43:29.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.032+0000 7f2f63400700 1 -- 172.21.15.62:0/438838319 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2f5400f2e0 con 0x7f2f6c069140 2024-04-20T21:43:29.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.032+0000 7f2f71200700 1 -- 172.21.15.62:0/438838319 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f6c19cde0 con 0x7f2f6c069140 2024-04-20T21:43:29.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.032+0000 7f2f71200700 1 -- 172.21.15.62:0/438838319 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f6c19d330 con 0x7f2f6c069140 2024-04-20T21:43:29.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.033+0000 7f2f71200700 1 -- 172.21.15.62:0/438838319 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f6c04f2a0 con 0x7f2f6c069140 2024-04-20T21:43:29.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.034+0000 7f2f63400700 1 -- 172.21.15.62:0/438838319 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f2f54010380 con 0x7f2f6c069140 2024-04-20T21:43:29.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.035+0000 7f2f63400700 1 --2- 172.21.15.62:0/438838319 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2f5006ecc0 0x7f2f50071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:43:29.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.035+0000 7f2f63400700 1 -- 172.21.15.62:0/438838319 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f2f5408e990 con 0x7f2f6c069140 2024-04-20T21:43:29.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.035+0000 7f2f69e00700 1 --2- 172.21.15.62:0/438838319 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2f5006ecc0 0x7f2f50071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:43:29.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.036+0000 7f2f69e00700 1 --2- 172.21.15.62:0/438838319 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2f5006ecc0 0x7f2f50071170 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f2f5c000f80 tx=0x7f2f5c00b7f0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:43:29.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.040+0000 7f2f63400700 1 -- 172.21.15.62:0/438838319 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f2f54057870 con 0x7f2f6c069140 2024-04-20T21:43:29.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.309+0000 7f2f71200700 1 -- 172.21.15.62:0/438838319 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f2f6c1994b0 con 0x7f2f6c069140 2024-04-20T21:43:29.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.310+0000 7f2f63400700 1 -- 172.21.15.62:0/438838319 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 25 v25) v1 ==== 76+0+1887 (secure 0 0 0) 0x7f2f5405af30 con 0x7f2f6c069140 2024-04-20T21:43:29.309 INFO:teuthology.orchestra.run.smithi062.stdout:e25 2024-04-20T21:43:29.310 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T21:43:29.310 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T21:43:29.310 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T21:43:29.310 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:43:29.310 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T21:43:29.310 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T21:43:29.310 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 20 2024-04-20T21:43:29.310 INFO:teuthology.orchestra.run.smithi062.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-20T21:43:29.310 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T21:43:29.310 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T21:42:01.559061+0000 2024-04-20T21:43:29.310 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T21:43:29.310 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T21:43:29.311 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T21:43:29.311 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T21:43:29.311 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T21:43:29.311 INFO:teuthology.orchestra.run.smithi062.stdout:max_xattr_size 65536 2024-04-20T21:43:29.311 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T21:43:29.311 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T21:43:29.311 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 81 2024-04-20T21:43:29.311 INFO:teuthology.orchestra.run.smithi062.stdout:snap_rstat 1 2024-04-20T21:43:29.311 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T21:43:29.311 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 2 2024-04-20T21:43:29.311 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1 2024-04-20T21:43:29.311 INFO:teuthology.orchestra.run.smithi062.stdout:up {} 2024-04-20T21:43:29.311 INFO:teuthology.orchestra.run.smithi062.stdout:failed 0,1 2024-04-20T21:43:29.311 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T21:43:29.311 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T21:43:29.312 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T21:43:29.312 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T21:43:29.312 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data enabled 2024-04-20T21:43:29.312 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T21:43:29.312 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T21:43:29.312 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 1 2024-04-20T21:43:29.312 INFO:teuthology.orchestra.run.smithi062.stdout:qdb_cluster leader: 0 members: 2024-04-20T21:43:29.312 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:43:29.312 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:43:29.312 INFO:teuthology.orchestra.run.smithi062.stdout:Standby daemons: 2024-04-20T21:43:29.312 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:43:29.312 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ryqblb{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6826/4113916205,v1:172.21.15.62:6827/4113916205] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T21:43:29.312 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ozizwi{-1:34476} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6828/3801975165,v1:172.21.15.62:6829/3801975165] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T21:43:29.312 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.xbgspw{-1:44285} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6826/3996040526,v1:172.21.15.189:6827/3996040526] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:43:29.312 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.dvimer{-1:44289} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6824/1743111843,v1:172.21.15.189:6825/1743111843] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:43:29.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.315+0000 7f2f71200700 1 -- 172.21.15.62:0/438838319 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2f5006ecc0 msgr2=0x7f2f50071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:43:29.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.315+0000 7f2f71200700 1 --2- 172.21.15.62:0/438838319 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2f5006ecc0 0x7f2f50071170 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f2f5c000f80 tx=0x7f2f5c00b7f0 comp rx=0 tx=0).stop 2024-04-20T21:43:29.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.315+0000 7f2f71200700 1 -- 172.21.15.62:0/438838319 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f6c069140 msgr2=0x7f2f6c198690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:43:29.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.315+0000 7f2f71200700 1 --2- 172.21.15.62:0/438838319 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f6c069140 0x7f2f6c198690 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f2f5400b4b0 tx=0x7f2f5400b870 comp rx=0 tx=0).stop 2024-04-20T21:43:29.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.315+0000 7f2f71200700 1 -- 172.21.15.62:0/438838319 shutdown_connections 2024-04-20T21:43:29.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.315+0000 7f2f71200700 1 --2- 172.21.15.62:0/438838319 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2f5006ecc0 0x7f2f50071170 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:43:29.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.315+0000 7f2f71200700 1 --2- 172.21.15.62:0/438838319 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f6c0687a0 0x7f2f6c198150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:43:29.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.315+0000 7f2f71200700 1 --2- 172.21.15.62:0/438838319 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f6c069140 0x7f2f6c198690 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:43:29.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.315+0000 7f2f71200700 1 -- 172.21.15.62:0/438838319 >> 172.21.15.62:0/438838319 conn(0x7f2f6c075130 msgr2=0x7f2f6c101e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:43:29.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.315+0000 7f2f71200700 1 -- 172.21.15.62:0/438838319 shutdown_connections 2024-04-20T21:43:29.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.315+0000 7f2f71200700 1 -- 172.21.15.62:0/438838319 wait complete. 2024-04-20T21:43:29.315 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 25 2024-04-20T21:43:29.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.465+0000 7fa194800700 1 -- 172.21.15.62:0/3331103546 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa18c106b80 msgr2=0x7fa18c106ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:43:29.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.465+0000 7fa194800700 1 --2- 172.21.15.62:0/3331103546 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa18c106b80 0x7fa18c106ff0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fa17c009910 tx=0x7fa17c009c20 comp rx=0 tx=0).stop 2024-04-20T21:43:29.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.465+0000 7fa194800700 1 -- 172.21.15.62:0/3331103546 shutdown_connections 2024-04-20T21:43:29.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.465+0000 7fa194800700 1 --2- 172.21.15.62:0/3331103546 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa18c106b80 0x7fa18c106ff0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:43:29.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.465+0000 7fa194800700 1 --2- 172.21.15.62:0/3331103546 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa18c10c8f0 0x7fa18c10ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:43:29.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.465+0000 7fa194800700 1 -- 172.21.15.62:0/3331103546 >> 172.21.15.62:0/3331103546 conn(0x7fa18c076060 msgr2=0x7fa18c078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:43:29.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.466+0000 7fa194800700 1 -- 172.21.15.62:0/3331103546 shutdown_connections 2024-04-20T21:43:29.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.466+0000 7fa194800700 1 -- 172.21.15.62:0/3331103546 wait complete. 2024-04-20T21:43:29.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.466+0000 7fa194800700 1 Processor -- start 2024-04-20T21:43:29.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.467+0000 7fa194800700 1 -- start start 2024-04-20T21:43:29.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.467+0000 7fa194800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa18c106b80 0x7fa18c19c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:43:29.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.467+0000 7fa194800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa18c10c8f0 0x7fa18c19c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:43:29.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.467+0000 7fa194800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa18c19d000 con 0x7fa18c10c8f0 2024-04-20T21:43:29.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.467+0000 7fa194800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa18c19d140 con 0x7fa18c106b80 2024-04-20T21:43:29.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.467+0000 7fa191400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa18c10c8f0 0x7fa18c19c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:43:29.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.467+0000 7fa191400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa18c10c8f0 0x7fa18c19c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48644/0 (socket says 172.21.15.62:48644) 2024-04-20T21:43:29.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.467+0000 7fa191400700 1 -- 172.21.15.62:0/3975229387 learned_addr learned my addr 172.21.15.62:0/3975229387 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:43:29.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.468+0000 7fa191e00700 1 --2- 172.21.15.62:0/3975229387 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa18c106b80 0x7fa18c19c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:43:29.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.468+0000 7fa191400700 1 -- 172.21.15.62:0/3975229387 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa18c106b80 msgr2=0x7fa18c19c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:43:29.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.468+0000 7fa191400700 1 --2- 172.21.15.62:0/3975229387 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa18c106b80 0x7fa18c19c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:43:29.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.468+0000 7fa191400700 1 -- 172.21.15.62:0/3975229387 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa17c0095f0 con 0x7fa18c10c8f0 2024-04-20T21:43:29.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.468+0000 7fa191e00700 1 --2- 172.21.15.62:0/3975229387 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa18c106b80 0x7fa18c19c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T21:43:29.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.468+0000 7fa191400700 1 --2- 172.21.15.62:0/3975229387 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa18c10c8f0 0x7fa18c19c9b0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fa17c00b5c0 tx=0x7fa17c00be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:43:29.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.468+0000 7fa186a00700 1 -- 172.21.15.62:0/3975229387 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa17c01d070 con 0x7fa18c10c8f0 2024-04-20T21:43:29.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.468+0000 7fa186a00700 1 -- 172.21.15.62:0/3975229387 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa17c00f460 con 0x7fa18c10c8f0 2024-04-20T21:43:29.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.468+0000 7fa186a00700 1 -- 172.21.15.62:0/3975229387 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa17c017610 con 0x7fa18c10c8f0 2024-04-20T21:43:29.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.468+0000 7fa194800700 1 -- 172.21.15.62:0/3975229387 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa18c1a0f30 con 0x7fa18c10c8f0 2024-04-20T21:43:29.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.469+0000 7fa194800700 1 -- 172.21.15.62:0/3975229387 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa18c1a1420 con 0x7fa18c10c8f0 2024-04-20T21:43:29.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.470+0000 7fa194800700 1 -- 172.21.15.62:0/3975229387 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa18c10ecc0 con 0x7fa18c10c8f0 2024-04-20T21:43:29.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.471+0000 7fa186a00700 1 -- 172.21.15.62:0/3975229387 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fa17c00f5d0 con 0x7fa18c10c8f0 2024-04-20T21:43:29.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.472+0000 7fa186a00700 1 --2- 172.21.15.62:0/3975229387 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa17406ecc0 0x7fa174071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:43:29.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.472+0000 7fa186a00700 1 -- 172.21.15.62:0/3975229387 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fa17c090610 con 0x7fa18c10c8f0 2024-04-20T21:43:29.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.472+0000 7fa191e00700 1 --2- 172.21.15.62:0/3975229387 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa17406ecc0 0x7fa174071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:43:29.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.473+0000 7fa191e00700 1 --2- 172.21.15.62:0/3975229387 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa17406ecc0 0x7fa174071170 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fa180005fd0 tx=0x7fa180005e20 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:43:29.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.477+0000 7fa186a00700 1 -- 172.21.15.62:0/3975229387 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fa17c059420 con 0x7fa18c10c8f0 2024-04-20T21:43:29.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.722+0000 7fa194800700 1 -- 172.21.15.62:0/3975229387 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa18c02cca0 con 0x7fa17406ecc0 2024-04-20T21:43:29.722 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:43:29 smithi062 ceph-mon[76322]: from='client.34516 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:43:29.722 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:43:29 smithi062 ceph-mon[76322]: pgmap v245: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:43:29.722 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:43:29 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/1695084454' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:43:29.722 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:43:29 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/438838319' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T21:43:29.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.725+0000 7fa186a00700 1 -- 172.21.15.62:0/3975229387 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fa18c02cca0 con 0x7fa17406ecc0 2024-04-20T21:43:29.724 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:43:29.724 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T21:43:29.725 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T21:43:29.725 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-20T21:43:29.725 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T21:43:29.725 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr", 2024-04-20T21:43:29.725 INFO:teuthology.orchestra.run.smithi062.stdout: "crash", 2024-04-20T21:43:29.725 INFO:teuthology.orchestra.run.smithi062.stdout: "mon", 2024-04-20T21:43:29.725 INFO:teuthology.orchestra.run.smithi062.stdout: "osd" 2024-04-20T21:43:29.725 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T21:43:29.725 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "14/23 daemons upgraded", 2024-04-20T21:43:29.725 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading mds daemons", 2024-04-20T21:43:29.725 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T21:43:29.725 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:43:29.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.728+0000 7fa194800700 1 -- 172.21.15.62:0/3975229387 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa17406ecc0 msgr2=0x7fa174071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:43:29.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.729+0000 7fa194800700 1 --2- 172.21.15.62:0/3975229387 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa17406ecc0 0x7fa174071170 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fa180005fd0 tx=0x7fa180005e20 comp rx=0 tx=0).stop 2024-04-20T21:43:29.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.729+0000 7fa194800700 1 -- 172.21.15.62:0/3975229387 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa18c10c8f0 msgr2=0x7fa18c19c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:43:29.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.729+0000 7fa194800700 1 --2- 172.21.15.62:0/3975229387 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa18c10c8f0 0x7fa18c19c9b0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fa17c00b5c0 tx=0x7fa17c00be00 comp rx=0 tx=0).stop 2024-04-20T21:43:29.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.729+0000 7fa194800700 1 -- 172.21.15.62:0/3975229387 shutdown_connections 2024-04-20T21:43:29.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.729+0000 7fa194800700 1 --2- 172.21.15.62:0/3975229387 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa17406ecc0 0x7fa174071170 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:43:29.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.729+0000 7fa194800700 1 --2- 172.21.15.62:0/3975229387 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa18c106b80 0x7fa18c19c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:43:29.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.729+0000 7fa194800700 1 --2- 172.21.15.62:0/3975229387 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa18c10c8f0 0x7fa18c19c9b0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:43:29.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.729+0000 7fa194800700 1 -- 172.21.15.62:0/3975229387 >> 172.21.15.62:0/3975229387 conn(0x7fa18c076060 msgr2=0x7fa18c077c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:43:29.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.729+0000 7fa194800700 1 -- 172.21.15.62:0/3975229387 shutdown_connections 2024-04-20T21:43:29.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.729+0000 7fa194800700 1 -- 172.21.15.62:0/3975229387 wait complete. 2024-04-20T21:43:29.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:43:29 smithi189 ceph-mon[57022]: from='client.34516 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:43:29.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:43:29 smithi189 ceph-mon[57022]: pgmap v245: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:43:29.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:43:29 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/1695084454' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:43:29.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:43:29 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/438838319' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T21:43:29.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.887+0000 7f1eb1c00700 1 -- 172.21.15.62:0/443866460 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1eac100e80 msgr2=0x7f1eac104e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:43:29.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.887+0000 7f1eb1c00700 1 --2- 172.21.15.62:0/443866460 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1eac100e80 0x7f1eac104e50 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f1e98009910 tx=0x7f1e98009c20 comp rx=0 tx=0).stop 2024-04-20T21:43:29.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.887+0000 7f1eb1c00700 1 -- 172.21.15.62:0/443866460 shutdown_connections 2024-04-20T21:43:29.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.887+0000 7f1eb1c00700 1 --2- 172.21.15.62:0/443866460 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1eac100e80 0x7f1eac104e50 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:43:29.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.887+0000 7f1eb1c00700 1 --2- 172.21.15.62:0/443866460 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1eac1004e0 0x7f1eac1008b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:43:29.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.887+0000 7f1eb1c00700 1 -- 172.21.15.62:0/443866460 >> 172.21.15.62:0/443866460 conn(0x7f1eac0fbda0 msgr2=0x7f1eac0fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:43:29.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.887+0000 7f1eb1c00700 1 -- 172.21.15.62:0/443866460 shutdown_connections 2024-04-20T21:43:29.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.887+0000 7f1eb1c00700 1 -- 172.21.15.62:0/443866460 wait complete. 2024-04-20T21:43:29.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.889+0000 7f1eb1c00700 1 Processor -- start 2024-04-20T21:43:29.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.889+0000 7f1eb1c00700 1 -- start start 2024-04-20T21:43:29.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.889+0000 7f1eb1c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1eac1004e0 0x7f1eac198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:43:29.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.889+0000 7f1eb1c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1eac100e80 0x7f1eac1985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:43:29.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.889+0000 7f1eb1c00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eac198ca0 con 0x7f1eac1004e0 2024-04-20T21:43:29.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.889+0000 7f1eb1c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eac19ca30 con 0x7f1eac100e80 2024-04-20T21:43:29.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.889+0000 7f1eaaa00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1eac1004e0 0x7f1eac198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:43:29.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.889+0000 7f1eaaa00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1eac1004e0 0x7f1eac198080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48660/0 (socket says 172.21.15.62:48660) 2024-04-20T21:43:29.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.889+0000 7f1eaaa00700 1 -- 172.21.15.62:0/4217261903 learned_addr learned my addr 172.21.15.62:0/4217261903 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:43:29.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.890+0000 7f1eaaa00700 1 -- 172.21.15.62:0/4217261903 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1eac100e80 msgr2=0x7f1eac1985c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:43:29.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.890+0000 7f1eaaa00700 1 --2- 172.21.15.62:0/4217261903 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1eac100e80 0x7f1eac1985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:43:29.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.890+0000 7f1eaaa00700 1 -- 172.21.15.62:0/4217261903 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e980095f0 con 0x7f1eac1004e0 2024-04-20T21:43:29.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.890+0000 7f1eaaa00700 1 --2- 172.21.15.62:0/4217261903 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1eac1004e0 0x7f1eac198080 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f1e9c00d8d0 tx=0x7f1e9c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:43:29.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.890+0000 7f1ea3400700 1 -- 172.21.15.62:0/4217261903 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1e9c009940 con 0x7f1eac1004e0 2024-04-20T21:43:29.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.890+0000 7f1eb1c00700 1 -- 172.21.15.62:0/4217261903 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1eac19cd10 con 0x7f1eac1004e0 2024-04-20T21:43:29.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.891+0000 7f1ea3400700 1 -- 172.21.15.62:0/4217261903 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1e9c010460 con 0x7f1eac1004e0 2024-04-20T21:43:29.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.891+0000 7f1ea3400700 1 -- 172.21.15.62:0/4217261903 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1e9c00f5f0 con 0x7f1eac1004e0 2024-04-20T21:43:29.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.891+0000 7f1eb1c00700 1 -- 172.21.15.62:0/4217261903 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1eac19d260 con 0x7f1eac1004e0 2024-04-20T21:43:29.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.892+0000 7f1eb1c00700 1 -- 172.21.15.62:0/4217261903 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1eac1045d0 con 0x7f1eac1004e0 2024-04-20T21:43:29.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.893+0000 7f1ea3400700 1 -- 172.21.15.62:0/4217261903 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f1e9c0105d0 con 0x7f1eac1004e0 2024-04-20T21:43:29.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.894+0000 7f1ea3400700 1 --2- 172.21.15.62:0/4217261903 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f1e9006ecc0 0x7f1e90071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:43:29.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.894+0000 7f1ea3400700 1 -- 172.21.15.62:0/4217261903 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f1e9c08ee80 con 0x7f1eac1004e0 2024-04-20T21:43:29.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.895+0000 7f1eaa000700 1 --2- 172.21.15.62:0/4217261903 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f1e9006ecc0 0x7f1e90071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:43:29.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.895+0000 7f1eaa000700 1 --2- 172.21.15.62:0/4217261903 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f1e9006ecc0 0x7f1e90071170 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f1e98006010 tx=0x7f1e9800b540 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:43:29.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:29.900+0000 7f1ea3400700 1 -- 172.21.15.62:0/4217261903 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f1e9c057c30 con 0x7f1eac1004e0 2024-04-20T21:43:30.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:30.233+0000 7f1eb1c00700 1 -- 172.21.15.62:0/4217261903 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f1eac04f2a0 con 0x7f1eac1004e0 2024-04-20T21:43:30.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:30.234+0000 7f1ea3400700 1 -- 172.21.15.62:0/4217261903 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7f1e9c05b2f0 con 0x7f1eac1004e0 2024-04-20T21:43:30.233 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-20T21:43:30.233 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-20T21:43:30.233 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is degraded 2024-04-20T21:43:30.233 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T21:43:30.234 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T21:43:30.234 INFO:teuthology.orchestra.run.smithi062.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-20T21:43:30.234 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-20T21:43:30.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:30.237+0000 7f1eb1c00700 1 -- 172.21.15.62:0/4217261903 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f1e9006ecc0 msgr2=0x7f1e90071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:43:30.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:30.237+0000 7f1eb1c00700 1 --2- 172.21.15.62:0/4217261903 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f1e9006ecc0 0x7f1e90071170 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f1e98006010 tx=0x7f1e9800b540 comp rx=0 tx=0).stop 2024-04-20T21:43:30.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:30.237+0000 7f1eb1c00700 1 -- 172.21.15.62:0/4217261903 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1eac1004e0 msgr2=0x7f1eac198080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:43:30.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:30.237+0000 7f1eb1c00700 1 --2- 172.21.15.62:0/4217261903 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1eac1004e0 0x7f1eac198080 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f1e9c00d8d0 tx=0x7f1e9c00dc90 comp rx=0 tx=0).stop 2024-04-20T21:43:30.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:30.237+0000 7f1eb1c00700 1 -- 172.21.15.62:0/4217261903 shutdown_connections 2024-04-20T21:43:30.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:30.237+0000 7f1eb1c00700 1 --2- 172.21.15.62:0/4217261903 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f1e9006ecc0 0x7f1e90071170 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:43:30.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:30.237+0000 7f1eb1c00700 1 --2- 172.21.15.62:0/4217261903 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1eac1004e0 0x7f1eac198080 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:43:30.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:30.237+0000 7f1eb1c00700 1 --2- 172.21.15.62:0/4217261903 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1eac100e80 0x7f1eac1985c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:43:30.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:30.237+0000 7f1eb1c00700 1 -- 172.21.15.62:0/4217261903 >> 172.21.15.62:0/4217261903 conn(0x7f1eac0fbda0 msgr2=0x7f1eac0fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:43:30.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:30.238+0000 7f1eb1c00700 1 -- 172.21.15.62:0/4217261903 shutdown_connections 2024-04-20T21:43:30.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:43:30.238+0000 7f1eb1c00700 1 -- 172.21.15.62:0/4217261903 wait complete. 2024-04-20T21:43:30.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:43:30 smithi062 ceph-mon[76322]: from='client.34528 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:43:30.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:43:30 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/4217261903' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T21:43:30.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:43:30 smithi189 ceph-mon[57022]: from='client.34528 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:43:30.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:43:30 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/4217261903' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T21:43:31.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:43:31 smithi062 ceph-mon[76322]: pgmap v246: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:43:31.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:43:31 smithi189 ceph-mon[57022]: pgmap v246: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:43:33.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:43:33 smithi062 ceph-mon[76322]: pgmap v247: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:43:33.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:43:33 smithi189 ceph-mon[57022]: pgmap v247: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:43:34.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:43:34 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:43:34.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:43:34 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:43:35.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:43:35 smithi062 ceph-mon[76322]: pgmap v248: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:43:35.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:43:35 smithi189 ceph-mon[57022]: pgmap v248: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:43:37.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:43:37 smithi062 ceph-mon[76322]: pgmap v249: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:43:37.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:43:37 smithi189 ceph-mon[57022]: pgmap v249: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:43:39.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:43:39 smithi062 ceph-mon[76322]: pgmap v250: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:43:39.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:43:39 smithi189 ceph-mon[57022]: pgmap v250: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:43:41.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:43:41 smithi189 ceph-mon[57022]: pgmap v251: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:43:42.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:43:41 smithi062 ceph-mon[76322]: pgmap v251: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:43:43.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:43:43 smithi189 ceph-mon[57022]: pgmap v252: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:43:44.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:43:43 smithi062 ceph-mon[76322]: pgmap v252: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:43:45.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:43:45 smithi189 ceph-mon[57022]: pgmap v253: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:43:46.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:43:45 smithi062 ceph-mon[76322]: pgmap v253: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:43:47.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:43:47 smithi189 ceph-mon[57022]: pgmap v254: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:43:48.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:43:47 smithi062 ceph-mon[76322]: pgmap v254: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:43:49.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:43:49 smithi189 ceph-mon[57022]: pgmap v255: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:43:49.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:43:49 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:43:50.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:43:49 smithi062 ceph-mon[76322]: pgmap v255: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:43:50.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:43:49 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:43:51.887 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:43:51 smithi062 ceph-mon[76322]: pgmap v256: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:43:51.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:43:51 smithi189 ceph-mon[57022]: pgmap v256: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:43:53.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:43:53 smithi189 ceph-mon[57022]: pgmap v257: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:43:54.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:43:53 smithi062 ceph-mon[76322]: pgmap v257: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:43:55.889 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:43:55 smithi189 ceph-mon[57022]: pgmap v258: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:43:56.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:43:55 smithi062 ceph-mon[76322]: pgmap v258: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:43:57.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:43:57 smithi189 ceph-mon[57022]: pgmap v259: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:43:58.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:43:57 smithi062 ceph-mon[76322]: pgmap v259: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:43:59.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:43:59 smithi189 ceph-mon[57022]: pgmap v260: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:44:00.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:43:59 smithi062 ceph-mon[76322]: pgmap v260: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:44:00.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:00.416+0000 7efecee00700 1 -- 172.21.15.62:0/1760609558 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efec810c8f0 msgr2=0x7efec810ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:44:00.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:00.416+0000 7efecee00700 1 --2- 172.21.15.62:0/1760609558 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efec810c8f0 0x7efec810ccc0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7efeb8009910 tx=0x7efeb8009c20 comp rx=0 tx=0).stop 2024-04-20T21:44:00.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:00.416+0000 7efecee00700 1 -- 172.21.15.62:0/1760609558 shutdown_connections 2024-04-20T21:44:00.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:00.416+0000 7efecee00700 1 --2- 172.21.15.62:0/1760609558 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efec8106b80 0x7efec8106ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:44:00.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:00.416+0000 7efecee00700 1 --2- 172.21.15.62:0/1760609558 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efec810c8f0 0x7efec810ccc0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:44:00.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:00.416+0000 7efecee00700 1 -- 172.21.15.62:0/1760609558 >> 172.21.15.62:0/1760609558 conn(0x7efec8076060 msgr2=0x7efec8078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:44:00.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:00.417+0000 7efecee00700 1 -- 172.21.15.62:0/1760609558 shutdown_connections 2024-04-20T21:44:00.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:00.417+0000 7efecee00700 1 -- 172.21.15.62:0/1760609558 wait complete. 2024-04-20T21:44:00.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:00.418+0000 7efecee00700 1 Processor -- start 2024-04-20T21:44:00.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:00.418+0000 7efecee00700 1 -- start start 2024-04-20T21:44:00.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:00.418+0000 7efecee00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efec8106b80 0x7efec819c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:44:00.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:00.418+0000 7efecee00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efec810c8f0 0x7efec819c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:44:00.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:00.418+0000 7efecee00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efec819d040 con 0x7efec810c8f0 2024-04-20T21:44:00.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:00.418+0000 7efecee00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efec81a0dd0 con 0x7efec8106b80 2024-04-20T21:44:00.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:00.419+0000 7efec7400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efec810c8f0 0x7efec819c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:44:00.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:00.419+0000 7efec7400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efec810c8f0 0x7efec819c960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52988/0 (socket says 172.21.15.62:52988) 2024-04-20T21:44:00.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:00.419+0000 7efec7400700 1 -- 172.21.15.62:0/3766193542 learned_addr learned my addr 172.21.15.62:0/3766193542 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:44:00.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:00.419+0000 7efec7e00700 1 --2- 172.21.15.62:0/3766193542 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efec8106b80 0x7efec819c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:44:00.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:00.419+0000 7efec7400700 1 -- 172.21.15.62:0/3766193542 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efec8106b80 msgr2=0x7efec819c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:44:00.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:00.419+0000 7efec7400700 1 --2- 172.21.15.62:0/3766193542 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efec8106b80 0x7efec819c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:44:00.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:00.419+0000 7efec7400700 1 -- 172.21.15.62:0/3766193542 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efeb80095f0 con 0x7efec810c8f0 2024-04-20T21:44:00.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:00.419+0000 7efec7e00700 1 --2- 172.21.15.62:0/3766193542 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efec8106b80 0x7efec819c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T21:44:00.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:00.420+0000 7efec7400700 1 --2- 172.21.15.62:0/3766193542 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efec810c8f0 0x7efec819c960 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7efeb400d8d0 tx=0x7efeb400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:44:00.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:00.420+0000 7efec4c00700 1 -- 172.21.15.62:0/3766193542 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efeb4009940 con 0x7efec810c8f0 2024-04-20T21:44:00.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:00.420+0000 7efec4c00700 1 -- 172.21.15.62:0/3766193542 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efeb4010460 con 0x7efec810c8f0 2024-04-20T21:44:00.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:00.420+0000 7efec4c00700 1 -- 172.21.15.62:0/3766193542 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efeb400f5f0 con 0x7efec810c8f0 2024-04-20T21:44:00.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:00.420+0000 7efecee00700 1 -- 172.21.15.62:0/3766193542 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efec81a10b0 con 0x7efec810c8f0 2024-04-20T21:44:00.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:00.420+0000 7efecee00700 1 -- 172.21.15.62:0/3766193542 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efec81a1600 con 0x7efec810c8f0 2024-04-20T21:44:00.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:00.421+0000 7efecee00700 1 -- 172.21.15.62:0/3766193542 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efec8109b20 con 0x7efec810c8f0 2024-04-20T21:44:00.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:00.422+0000 7efec4c00700 1 -- 172.21.15.62:0/3766193542 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7efeb40105d0 con 0x7efec810c8f0 2024-04-20T21:44:00.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:00.423+0000 7efec4c00700 1 --2- 172.21.15.62:0/3766193542 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7efeac06ecc0 0x7efeac071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:44:00.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:00.424+0000 7efec4c00700 1 -- 172.21.15.62:0/3766193542 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7efeb408ee10 con 0x7efec810c8f0 2024-04-20T21:44:00.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:00.424+0000 7efec7e00700 1 --2- 172.21.15.62:0/3766193542 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7efeac06ecc0 0x7efeac071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:44:00.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:00.425+0000 7efec7e00700 1 --2- 172.21.15.62:0/3766193542 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7efeac06ecc0 0x7efeac071170 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7efeb800b5c0 tx=0x7efeb80053a0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:44:00.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:00.430+0000 7efec4c00700 1 -- 172.21.15.62:0/3766193542 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7efeb4057ba0 con 0x7efec810c8f0 2024-04-20T21:44:00.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:00.686+0000 7efecee00700 1 -- 172.21.15.62:0/3766193542 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7efec81a18e0 con 0x7efeac06ecc0 2024-04-20T21:44:00.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:00.688+0000 7efec4c00700 1 -- 172.21.15.62:0/3766193542 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7efec81a18e0 con 0x7efeac06ecc0 2024-04-20T21:44:00.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:00.691+0000 7efecee00700 1 -- 172.21.15.62:0/3766193542 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7efeac06ecc0 msgr2=0x7efeac071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:44:00.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:00.691+0000 7efecee00700 1 --2- 172.21.15.62:0/3766193542 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7efeac06ecc0 0x7efeac071170 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7efeb800b5c0 tx=0x7efeb80053a0 comp rx=0 tx=0).stop 2024-04-20T21:44:00.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:00.691+0000 7efecee00700 1 -- 172.21.15.62:0/3766193542 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efec810c8f0 msgr2=0x7efec819c960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:44:00.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:00.691+0000 7efecee00700 1 --2- 172.21.15.62:0/3766193542 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efec810c8f0 0x7efec819c960 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7efeb400d8d0 tx=0x7efeb400dc90 comp rx=0 tx=0).stop 2024-04-20T21:44:00.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:00.691+0000 7efecee00700 1 -- 172.21.15.62:0/3766193542 shutdown_connections 2024-04-20T21:44:00.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:00.691+0000 7efecee00700 1 --2- 172.21.15.62:0/3766193542 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7efeac06ecc0 0x7efeac071170 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:44:00.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:00.691+0000 7efecee00700 1 --2- 172.21.15.62:0/3766193542 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efec8106b80 0x7efec819c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:44:00.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:00.691+0000 7efecee00700 1 --2- 172.21.15.62:0/3766193542 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efec810c8f0 0x7efec819c960 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:44:00.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:00.691+0000 7efecee00700 1 -- 172.21.15.62:0/3766193542 >> 172.21.15.62:0/3766193542 conn(0x7efec8076060 msgr2=0x7efec8077a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:44:00.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:00.692+0000 7efecee00700 1 -- 172.21.15.62:0/3766193542 shutdown_connections 2024-04-20T21:44:00.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:00.692+0000 7efecee00700 1 -- 172.21.15.62:0/3766193542 wait complete. 2024-04-20T21:44:00.708 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T21:44:00.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:00.851+0000 7f5091200700 1 -- 172.21.15.62:0/2519340939 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f508c1025f0 msgr2=0x7f508c1029c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:44:00.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:00.851+0000 7f5091200700 1 --2- 172.21.15.62:0/2519340939 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f508c1025f0 0x7f508c1029c0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f507c009910 tx=0x7f507c009c20 comp rx=0 tx=0).stop 2024-04-20T21:44:00.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:00.851+0000 7f5091200700 1 -- 172.21.15.62:0/2519340939 shutdown_connections 2024-04-20T21:44:00.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:00.851+0000 7f5091200700 1 --2- 172.21.15.62:0/2519340939 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f508c103e50 0x7f508c1042c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:44:00.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:00.851+0000 7f5091200700 1 --2- 172.21.15.62:0/2519340939 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f508c1025f0 0x7f508c1029c0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:44:00.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:00.851+0000 7f5091200700 1 -- 172.21.15.62:0/2519340939 >> 172.21.15.62:0/2519340939 conn(0x7f508c0fddb0 msgr2=0x7f508c1001c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:44:00.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:00.851+0000 7f5091200700 1 -- 172.21.15.62:0/2519340939 shutdown_connections 2024-04-20T21:44:00.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:00.851+0000 7f5091200700 1 -- 172.21.15.62:0/2519340939 wait complete. 2024-04-20T21:44:00.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:00.852+0000 7f5091200700 1 Processor -- start 2024-04-20T21:44:00.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:00.852+0000 7f5091200700 1 -- start start 2024-04-20T21:44:00.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:00.853+0000 7f5091200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f508c1025f0 0x7f508c197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:44:00.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:00.853+0000 7f5091200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f508c103e50 0x7f508c198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:44:00.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:00.853+0000 7f5091200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f508c198be0 con 0x7f508c1025f0 2024-04-20T21:44:00.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:00.853+0000 7f5091200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f508c19c970 con 0x7f508c103e50 2024-04-20T21:44:00.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:00.853+0000 7f508b400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f508c1025f0 0x7f508c197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:44:00.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:00.853+0000 7f508b400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f508c1025f0 0x7f508c197fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53002/0 (socket says 172.21.15.62:53002) 2024-04-20T21:44:00.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:00.853+0000 7f508b400700 1 -- 172.21.15.62:0/3003471964 learned_addr learned my addr 172.21.15.62:0/3003471964 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:44:00.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:00.853+0000 7f508b400700 1 -- 172.21.15.62:0/3003471964 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f508c103e50 msgr2=0x7f508c198500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:44:00.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:00.853+0000 7f5083e00700 1 --2- 172.21.15.62:0/3003471964 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f508c103e50 0x7f508c198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:44:00.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:00.853+0000 7f508b400700 1 --2- 172.21.15.62:0/3003471964 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f508c103e50 0x7f508c198500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:44:00.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:00.853+0000 7f508b400700 1 -- 172.21.15.62:0/3003471964 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f507c0095f0 con 0x7f508c1025f0 2024-04-20T21:44:00.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:00.853+0000 7f5083e00700 1 --2- 172.21.15.62:0/3003471964 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f508c103e50 0x7f508c198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T21:44:00.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:00.854+0000 7f508b400700 1 --2- 172.21.15.62:0/3003471964 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f508c1025f0 0x7f508c197fc0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f507c00b5c0 tx=0x7f507c00be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:44:00.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:00.854+0000 7f5088c00700 1 -- 172.21.15.62:0/3003471964 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f507c01d070 con 0x7f508c1025f0 2024-04-20T21:44:00.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:00.854+0000 7f5088c00700 1 -- 172.21.15.62:0/3003471964 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f507c00f460 con 0x7f508c1025f0 2024-04-20T21:44:00.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:00.854+0000 7f5091200700 1 -- 172.21.15.62:0/3003471964 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f508c19cbf0 con 0x7f508c1025f0 2024-04-20T21:44:00.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:00.854+0000 7f5088c00700 1 -- 172.21.15.62:0/3003471964 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f507c017610 con 0x7f508c1025f0 2024-04-20T21:44:00.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:00.854+0000 7f5091200700 1 -- 172.21.15.62:0/3003471964 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f508c19d160 con 0x7f508c1025f0 2024-04-20T21:44:00.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:00.855+0000 7f5091200700 1 -- 172.21.15.62:0/3003471964 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f508c10a8d0 con 0x7f508c1025f0 2024-04-20T21:44:00.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:00.856+0000 7f5088c00700 1 -- 172.21.15.62:0/3003471964 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f507c00f5d0 con 0x7f508c1025f0 2024-04-20T21:44:00.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:00.857+0000 7f5088c00700 1 --2- 172.21.15.62:0/3003471964 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f507006ecc0 0x7f5070071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:44:00.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:00.857+0000 7f5088c00700 1 -- 172.21.15.62:0/3003471964 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f507c090510 con 0x7f508c1025f0 2024-04-20T21:44:00.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:00.858+0000 7f5083e00700 1 --2- 172.21.15.62:0/3003471964 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f507006ecc0 0x7f5070071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:44:00.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:00.858+0000 7f5083e00700 1 --2- 172.21.15.62:0/3003471964 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f507006ecc0 0x7f5070071170 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f508c1995e0 tx=0x7f5074008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:44:00.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:00.862+0000 7f5088c00700 1 -- 172.21.15.62:0/3003471964 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f507c059220 con 0x7f508c1025f0 2024-04-20T21:44:01.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:01.117+0000 7f5091200700 1 -- 172.21.15.62:0/3003471964 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f508c19d380 con 0x7f507006ecc0 2024-04-20T21:44:01.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:01.120+0000 7f5088c00700 1 -- 172.21.15.62:0/3003471964 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f508c19d380 con 0x7f507006ecc0 2024-04-20T21:44:01.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:01.123+0000 7f5091200700 1 -- 172.21.15.62:0/3003471964 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f507006ecc0 msgr2=0x7f5070071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:44:01.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:01.123+0000 7f5091200700 1 --2- 172.21.15.62:0/3003471964 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f507006ecc0 0x7f5070071170 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f508c1995e0 tx=0x7f5074008040 comp rx=0 tx=0).stop 2024-04-20T21:44:01.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:01.123+0000 7f5091200700 1 -- 172.21.15.62:0/3003471964 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f508c1025f0 msgr2=0x7f508c197fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:44:01.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:01.123+0000 7f5091200700 1 --2- 172.21.15.62:0/3003471964 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f508c1025f0 0x7f508c197fc0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f507c00b5c0 tx=0x7f507c00be00 comp rx=0 tx=0).stop 2024-04-20T21:44:01.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:01.123+0000 7f5091200700 1 -- 172.21.15.62:0/3003471964 shutdown_connections 2024-04-20T21:44:01.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:01.123+0000 7f5091200700 1 --2- 172.21.15.62:0/3003471964 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f507006ecc0 0x7f5070071170 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:44:01.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:01.123+0000 7f5091200700 1 --2- 172.21.15.62:0/3003471964 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f508c1025f0 0x7f508c197fc0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:44:01.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:01.123+0000 7f5091200700 1 --2- 172.21.15.62:0/3003471964 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f508c103e50 0x7f508c198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:44:01.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:01.123+0000 7f5091200700 1 -- 172.21.15.62:0/3003471964 >> 172.21.15.62:0/3003471964 conn(0x7f508c0fddb0 msgr2=0x7f508c0fe7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:44:01.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:01.123+0000 7f5091200700 1 -- 172.21.15.62:0/3003471964 shutdown_connections 2024-04-20T21:44:01.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:01.124+0000 7f5091200700 1 -- 172.21.15.62:0/3003471964 wait complete. 2024-04-20T21:44:01.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:01.281+0000 7f3e3b200700 1 -- 172.21.15.62:0/189517983 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e3c10c660 msgr2=0x7f3e3c10ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:44:01.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:01.281+0000 7f3e3b200700 1 --2- 172.21.15.62:0/189517983 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e3c10c660 0x7f3e3c10ca30 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f3e2c009910 tx=0x7f3e2c009c20 comp rx=0 tx=0).stop 2024-04-20T21:44:01.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:01.281+0000 7f3e3b200700 1 -- 172.21.15.62:0/189517983 shutdown_connections 2024-04-20T21:44:01.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:01.281+0000 7f3e3b200700 1 --2- 172.21.15.62:0/189517983 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e3c106b80 0x7f3e3c106ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:44:01.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:01.281+0000 7f3e3b200700 1 --2- 172.21.15.62:0/189517983 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e3c10c660 0x7f3e3c10ca30 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:44:01.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:01.281+0000 7f3e3b200700 1 -- 172.21.15.62:0/189517983 >> 172.21.15.62:0/189517983 conn(0x7f3e3c075e90 msgr2=0x7f3e3c0782a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:44:01.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:01.281+0000 7f3e3b200700 1 -- 172.21.15.62:0/189517983 shutdown_connections 2024-04-20T21:44:01.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:01.281+0000 7f3e3b200700 1 -- 172.21.15.62:0/189517983 wait complete. 2024-04-20T21:44:01.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:01.282+0000 7f3e3b200700 1 Processor -- start 2024-04-20T21:44:01.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:01.282+0000 7f3e3b200700 1 -- start start 2024-04-20T21:44:01.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:01.282+0000 7f3e3b200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e3c106b80 0x7f3e3c19c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:44:01.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:01.282+0000 7f3e3b200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e3c10c660 0x7f3e3c19c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:44:01.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:01.282+0000 7f3e3b200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e3c19cea0 con 0x7f3e3c10c660 2024-04-20T21:44:01.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:01.282+0000 7f3e3b200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e3c1a0c30 con 0x7f3e3c106b80 2024-04-20T21:44:01.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:01.283+0000 7f3e39400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e3c10c660 0x7f3e3c19c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:44:01.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:01.283+0000 7f3e39400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e3c10c660 0x7f3e3c19c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53008/0 (socket says 172.21.15.62:53008) 2024-04-20T21:44:01.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:01.283+0000 7f3e39400700 1 -- 172.21.15.62:0/2352046902 learned_addr learned my addr 172.21.15.62:0/2352046902 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:44:01.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:01.283+0000 7f3e39e00700 1 --2- 172.21.15.62:0/2352046902 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e3c106b80 0x7f3e3c19c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:44:01.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:01.283+0000 7f3e39400700 1 -- 172.21.15.62:0/2352046902 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e3c106b80 msgr2=0x7f3e3c19c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:44:01.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:01.283+0000 7f3e39400700 1 --2- 172.21.15.62:0/2352046902 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e3c106b80 0x7f3e3c19c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:44:01.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:01.283+0000 7f3e39400700 1 -- 172.21.15.62:0/2352046902 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e2c0095f0 con 0x7f3e3c10c660 2024-04-20T21:44:01.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:01.284+0000 7f3e39400700 1 --2- 172.21.15.62:0/2352046902 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e3c10c660 0x7f3e3c19c7c0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f3e2800c8a0 tx=0x7f3e2800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:44:01.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:01.284+0000 7f3e32a00700 1 -- 172.21.15.62:0/2352046902 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3e2800cea0 con 0x7f3e3c10c660 2024-04-20T21:44:01.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:01.284+0000 7f3e32a00700 1 -- 172.21.15.62:0/2352046902 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3e28004830 con 0x7f3e3c10c660 2024-04-20T21:44:01.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:01.284+0000 7f3e3b200700 1 -- 172.21.15.62:0/2352046902 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e3c1a0f10 con 0x7f3e3c10c660 2024-04-20T21:44:01.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:01.284+0000 7f3e32a00700 1 -- 172.21.15.62:0/2352046902 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3e280055b0 con 0x7f3e3c10c660 2024-04-20T21:44:01.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:01.284+0000 7f3e3b200700 1 -- 172.21.15.62:0/2352046902 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e3c1a1460 con 0x7f3e3c10c660 2024-04-20T21:44:01.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:01.285+0000 7f3e3b200700 1 -- 172.21.15.62:0/2352046902 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e3c10eb40 con 0x7f3e3c10c660 2024-04-20T21:44:01.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:01.286+0000 7f3e32a00700 1 -- 172.21.15.62:0/2352046902 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f3e280049a0 con 0x7f3e3c10c660 2024-04-20T21:44:01.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:01.287+0000 7f3e32a00700 1 --2- 172.21.15.62:0/2352046902 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3e2006ecc0 0x7f3e20071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:44:01.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:01.287+0000 7f3e32a00700 1 -- 172.21.15.62:0/2352046902 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f3e2808eaf0 con 0x7f3e3c10c660 2024-04-20T21:44:01.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:01.287+0000 7f3e39e00700 1 --2- 172.21.15.62:0/2352046902 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3e2006ecc0 0x7f3e20071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:44:01.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:01.288+0000 7f3e39e00700 1 --2- 172.21.15.62:0/2352046902 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3e2006ecc0 0x7f3e20071170 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f3e2c00b5c0 tx=0x7f3e2c01a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:44:01.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:01.292+0000 7f3e32a00700 1 -- 172.21.15.62:0/2352046902 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f3e28057880 con 0x7f3e3c10c660 2024-04-20T21:44:01.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:01.540+0000 7f3e3b200700 1 -- 172.21.15.62:0/2352046902 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f3e3c1a1780 con 0x7f3e2006ecc0 2024-04-20T21:44:01.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:01.595+0000 7f3e32a00700 1 -- 172.21.15.62:0/2352046902 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7f3e3c1a1780 con 0x7f3e2006ecc0 2024-04-20T21:44:01.594 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T21:44:01.595 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (9m) 74s ago 23m 14.2M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T21:44:01.595 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (24m) 74s ago 24m 9512k - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T21:44:01.595 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (22m) 2m ago 22m 8942k - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T21:44:01.595 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (5m) 74s ago 24m 6660k - 19.0.0-3133-ga8c84f59 76bc6b91b726 2825d5ba11c6 2024-04-20T21:44:01.595 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (5m) 2m ago 22m 6665k - 19.0.0-3133-ga8c84f59 76bc6b91b726 20f746fa1bba 2024-04-20T21:44:01.595 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (9m) 74s ago 23m 74.2M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T21:44:01.595 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (82s) 74s ago 16m 11.2M - 19.0.0-3133-ga8c84f59 76bc6b91b726 de80bc90a683 2024-04-20T21:44:01.595 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (109s) 74s ago 16m 11.9M - 19.0.0-3133-ga8c84f59 76bc6b91b726 cd793b7ee98e 2024-04-20T21:44:01.595 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (16m) 2m ago 16m 117M - 18.2.0 dc2bc1663786 fb2a8a223db8 2024-04-20T21:44:01.595 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (16m) 2m ago 16m 55.6M - 18.2.0 dc2bc1663786 f2eec2ee8951 2024-04-20T21:44:01.595 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (12m) 74s ago 25m 511M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T21:44:01.595 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (11m) 2m ago 22m 452M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T21:44:01.595 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (7m) 74s ago 25m 55.9M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 019eb1479485 2024-04-20T21:44:01.595 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (7m) 2m ago 22m 39.8M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 55aac560aab6 2024-04-20T21:44:01.596 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (11m) 74s ago 23m 7952k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T21:44:01.596 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (10m) 2m ago 22m 8304k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T21:44:01.596 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (5m) 74s ago 21m 88.0M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 ee2bb901f26e 2024-04-20T21:44:01.596 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (4m) 74s ago 20m 92.3M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 9075152d4162 2024-04-20T21:44:01.596 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (4m) 74s ago 20m 74.4M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 2d4f3e38dc0e 2024-04-20T21:44:01.596 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (3m) 2m ago 19m 87.6M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 8f97b17ae0eb 2024-04-20T21:44:01.596 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (2m) 2m ago 19m 90.7M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 653d8079967d 2024-04-20T21:44:01.596 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (2m) 2m ago 18m 11.5M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 596dded14c12 2024-04-20T21:44:01.596 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (10m) 74s ago 23m 74.0M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T21:44:01.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:01.599+0000 7f3e3b200700 1 -- 172.21.15.62:0/2352046902 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3e2006ecc0 msgr2=0x7f3e20071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:44:01.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:01.599+0000 7f3e3b200700 1 --2- 172.21.15.62:0/2352046902 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3e2006ecc0 0x7f3e20071170 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f3e2c00b5c0 tx=0x7f3e2c01a040 comp rx=0 tx=0).stop 2024-04-20T21:44:01.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:01.599+0000 7f3e3b200700 1 -- 172.21.15.62:0/2352046902 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e3c10c660 msgr2=0x7f3e3c19c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:44:01.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:01.599+0000 7f3e3b200700 1 --2- 172.21.15.62:0/2352046902 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e3c10c660 0x7f3e3c19c7c0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f3e2800c8a0 tx=0x7f3e2800cc60 comp rx=0 tx=0).stop 2024-04-20T21:44:01.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:01.599+0000 7f3e3b200700 1 -- 172.21.15.62:0/2352046902 shutdown_connections 2024-04-20T21:44:01.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:01.599+0000 7f3e3b200700 1 --2- 172.21.15.62:0/2352046902 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3e2006ecc0 0x7f3e20071170 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:44:01.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:01.599+0000 7f3e3b200700 1 --2- 172.21.15.62:0/2352046902 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e3c106b80 0x7f3e3c19c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:44:01.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:01.599+0000 7f3e3b200700 1 --2- 172.21.15.62:0/2352046902 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e3c10c660 0x7f3e3c19c7c0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:44:01.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:01.599+0000 7f3e3b200700 1 -- 172.21.15.62:0/2352046902 >> 172.21.15.62:0/2352046902 conn(0x7f3e3c075e90 msgr2=0x7f3e3c077980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:44:01.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:01.600+0000 7f3e3b200700 1 -- 172.21.15.62:0/2352046902 shutdown_connections 2024-04-20T21:44:01.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:01.600+0000 7f3e3b200700 1 -- 172.21.15.62:0/2352046902 wait complete. 2024-04-20T21:44:01.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:01.755+0000 7f6d46400700 1 -- 172.21.15.62:0/2816224757 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d40106b80 msgr2=0x7f6d40106ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:44:01.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:01.755+0000 7f6d46400700 1 --2- 172.21.15.62:0/2816224757 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d40106b80 0x7f6d40106ff0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f6d2c009910 tx=0x7f6d2c009c20 comp rx=0 tx=0).stop 2024-04-20T21:44:01.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:01.755+0000 7f6d46400700 1 -- 172.21.15.62:0/2816224757 shutdown_connections 2024-04-20T21:44:01.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:01.755+0000 7f6d46400700 1 --2- 172.21.15.62:0/2816224757 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d40106b80 0x7f6d40106ff0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:44:01.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:01.755+0000 7f6d46400700 1 --2- 172.21.15.62:0/2816224757 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6d4010c660 0x7f6d4010ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:44:01.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:01.755+0000 7f6d46400700 1 -- 172.21.15.62:0/2816224757 >> 172.21.15.62:0/2816224757 conn(0x7f6d40075e90 msgr2=0x7f6d400782a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:44:01.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:01.755+0000 7f6d46400700 1 -- 172.21.15.62:0/2816224757 shutdown_connections 2024-04-20T21:44:01.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:01.755+0000 7f6d46400700 1 -- 172.21.15.62:0/2816224757 wait complete. 2024-04-20T21:44:01.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:01.756+0000 7f6d46400700 1 Processor -- start 2024-04-20T21:44:01.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:01.756+0000 7f6d46400700 1 -- start start 2024-04-20T21:44:01.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:01.757+0000 7f6d46400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6d40106b80 0x7f6d4019c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:44:01.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:01.757+0000 7f6d46400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d4010c660 0x7f6d4019c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:44:01.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:01.757+0000 7f6d46400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d4019cd70 con 0x7f6d4010c660 2024-04-20T21:44:01.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:01.757+0000 7f6d46400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d4019ceb0 con 0x7f6d40106b80 2024-04-20T21:44:01.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:01.757+0000 7f6d3fe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d4010c660 0x7f6d4019c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:44:01.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:01.757+0000 7f6d3fe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d4010c660 0x7f6d4019c720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49034/0 (socket says 172.21.15.62:49034) 2024-04-20T21:44:01.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:01.757+0000 7f6d3fe00700 1 -- 172.21.15.62:0/1331006492 learned_addr learned my addr 172.21.15.62:0/1331006492 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:44:01.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:01.757+0000 7f6d45000700 1 --2- 172.21.15.62:0/1331006492 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6d40106b80 0x7f6d4019c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:44:01.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:01.757+0000 7f6d3fe00700 1 -- 172.21.15.62:0/1331006492 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6d40106b80 msgr2=0x7f6d4019c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:44:01.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:01.757+0000 7f6d3fe00700 1 --2- 172.21.15.62:0/1331006492 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6d40106b80 0x7f6d4019c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:44:01.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:01.757+0000 7f6d3fe00700 1 -- 172.21.15.62:0/1331006492 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d2c0095f0 con 0x7f6d4010c660 2024-04-20T21:44:01.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:01.758+0000 7f6d3fe00700 1 --2- 172.21.15.62:0/1331006492 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d4010c660 0x7f6d4019c720 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f6d2c00b5c0 tx=0x7f6d2c00be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:44:01.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:01.758+0000 7f6d3d600700 1 -- 172.21.15.62:0/1331006492 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6d2c01d070 con 0x7f6d4010c660 2024-04-20T21:44:01.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:01.758+0000 7f6d3d600700 1 -- 172.21.15.62:0/1331006492 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6d2c00f460 con 0x7f6d4010c660 2024-04-20T21:44:01.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:01.758+0000 7f6d46400700 1 -- 172.21.15.62:0/1331006492 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d401a0ca0 con 0x7f6d4010c660 2024-04-20T21:44:01.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:01.758+0000 7f6d3d600700 1 -- 172.21.15.62:0/1331006492 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6d2c017610 con 0x7f6d4010c660 2024-04-20T21:44:01.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:01.758+0000 7f6d46400700 1 -- 172.21.15.62:0/1331006492 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d401a1190 con 0x7f6d4010c660 2024-04-20T21:44:01.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:01.759+0000 7f6d46400700 1 -- 172.21.15.62:0/1331006492 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d40062380 con 0x7f6d4010c660 2024-04-20T21:44:01.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:01.760+0000 7f6d3d600700 1 -- 172.21.15.62:0/1331006492 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f6d2c00f5d0 con 0x7f6d4010c660 2024-04-20T21:44:01.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:01.761+0000 7f6d3d600700 1 --2- 172.21.15.62:0/1331006492 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6d2806ecc0 0x7f6d28071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:44:01.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:01.761+0000 7f6d3d600700 1 -- 172.21.15.62:0/1331006492 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f6d2c090510 con 0x7f6d4010c660 2024-04-20T21:44:01.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:01.762+0000 7f6d45000700 1 --2- 172.21.15.62:0/1331006492 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6d2806ecc0 0x7f6d28071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:44:01.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:01.762+0000 7f6d45000700 1 --2- 172.21.15.62:0/1331006492 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6d2806ecc0 0x7f6d28071170 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f6d34005ea0 tx=0x7f6d34005e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:44:01.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:01.766+0000 7f6d3d600700 1 -- 172.21.15.62:0/1331006492 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f6d2c059220 con 0x7f6d4010c660 2024-04-20T21:44:01.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:44:01 smithi189 ceph-mon[57022]: pgmap v261: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:44:01.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:44:01 smithi189 ceph-mon[57022]: from='client.34536 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:44:02.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:44:01 smithi062 ceph-mon[76322]: pgmap v261: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:44:02.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:44:01 smithi062 ceph-mon[76322]: from='client.34536 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:44:02.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:02.079+0000 7f6d46400700 1 -- 172.21.15.62:0/1331006492 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f6d4019d540 con 0x7f6d4010c660 2024-04-20T21:44:02.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:02.080+0000 7f6d3d600700 1 -- 172.21.15.62:0/1331006492 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7f6d2c05c8e0 con 0x7f6d4010c660 2024-04-20T21:44:02.080 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:44:02.126 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T21:44:02.126 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:44:02.126 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:44:02.126 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T21:44:02.126 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:44:02.126 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:44:02.126 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T21:44:02.127 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 6 2024-04-20T21:44:02.127 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:44:02.127 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T21:44:02.127 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 2, 2024-04-20T21:44:02.127 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:44:02.127 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:44:02.127 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T21:44:02.127 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 2, 2024-04-20T21:44:02.127 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 12 2024-04-20T21:44:02.127 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T21:44:02.127 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:44:02.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:02.083+0000 7f6d46400700 1 -- 172.21.15.62:0/1331006492 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6d2806ecc0 msgr2=0x7f6d28071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:44:02.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:02.083+0000 7f6d46400700 1 --2- 172.21.15.62:0/1331006492 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6d2806ecc0 0x7f6d28071170 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f6d34005ea0 tx=0x7f6d34005e30 comp rx=0 tx=0).stop 2024-04-20T21:44:02.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:02.083+0000 7f6d46400700 1 -- 172.21.15.62:0/1331006492 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d4010c660 msgr2=0x7f6d4019c720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:44:02.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:02.083+0000 7f6d46400700 1 --2- 172.21.15.62:0/1331006492 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d4010c660 0x7f6d4019c720 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f6d2c00b5c0 tx=0x7f6d2c00be00 comp rx=0 tx=0).stop 2024-04-20T21:44:02.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:02.083+0000 7f6d46400700 1 -- 172.21.15.62:0/1331006492 shutdown_connections 2024-04-20T21:44:02.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:02.083+0000 7f6d46400700 1 --2- 172.21.15.62:0/1331006492 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6d2806ecc0 0x7f6d28071170 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:44:02.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:02.083+0000 7f6d46400700 1 --2- 172.21.15.62:0/1331006492 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6d40106b80 0x7f6d4019c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:44:02.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:02.083+0000 7f6d46400700 1 --2- 172.21.15.62:0/1331006492 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d4010c660 0x7f6d4019c720 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:44:02.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:02.083+0000 7f6d46400700 1 -- 172.21.15.62:0/1331006492 >> 172.21.15.62:0/1331006492 conn(0x7f6d40075e90 msgr2=0x7f6d400778d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:44:02.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:02.084+0000 7f6d46400700 1 -- 172.21.15.62:0/1331006492 shutdown_connections 2024-04-20T21:44:02.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:02.084+0000 7f6d46400700 1 -- 172.21.15.62:0/1331006492 wait complete. 2024-04-20T21:44:02.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:02.249+0000 7f8188200700 1 -- 172.21.15.62:0/3123564220 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8180106b80 msgr2=0x7f8180106ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:44:02.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:02.249+0000 7f8188200700 1 --2- 172.21.15.62:0/3123564220 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8180106b80 0x7f8180106ff0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f816c009910 tx=0x7f816c009c20 comp rx=0 tx=0).stop 2024-04-20T21:44:02.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:02.250+0000 7f8188200700 1 -- 172.21.15.62:0/3123564220 shutdown_connections 2024-04-20T21:44:02.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:02.250+0000 7f8188200700 1 --2- 172.21.15.62:0/3123564220 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8180106b80 0x7f8180106ff0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:44:02.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:02.250+0000 7f8188200700 1 --2- 172.21.15.62:0/3123564220 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f818010c8f0 0x7f818010ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:44:02.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:02.250+0000 7f8188200700 1 -- 172.21.15.62:0/3123564220 >> 172.21.15.62:0/3123564220 conn(0x7f8180076060 msgr2=0x7f8180078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:44:02.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:02.250+0000 7f8188200700 1 -- 172.21.15.62:0/3123564220 shutdown_connections 2024-04-20T21:44:02.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:02.250+0000 7f8188200700 1 -- 172.21.15.62:0/3123564220 wait complete. 2024-04-20T21:44:02.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:02.251+0000 7f8188200700 1 Processor -- start 2024-04-20T21:44:02.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:02.251+0000 7f8188200700 1 -- start start 2024-04-20T21:44:02.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:02.251+0000 7f8188200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8180106b80 0x7f818019c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:44:02.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:02.251+0000 7f8188200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f818010c8f0 0x7f818019c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:44:02.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:02.251+0000 7f8188200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f818019cfd0 con 0x7f8180106b80 2024-04-20T21:44:02.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:02.252+0000 7f8188200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f818019d110 con 0x7f818010c8f0 2024-04-20T21:44:02.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:02.252+0000 7f8185800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8180106b80 0x7f818019c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:44:02.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:02.252+0000 7f8185800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8180106b80 0x7f818019c440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49046/0 (socket says 172.21.15.62:49046) 2024-04-20T21:44:02.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:02.252+0000 7f8185800700 1 -- 172.21.15.62:0/1373372733 learned_addr learned my addr 172.21.15.62:0/1373372733 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:44:02.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:02.252+0000 7f8184e00700 1 --2- 172.21.15.62:0/1373372733 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f818010c8f0 0x7f818019c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:44:02.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:02.252+0000 7f8185800700 1 -- 172.21.15.62:0/1373372733 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f818010c8f0 msgr2=0x7f818019c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:44:02.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:02.252+0000 7f8185800700 1 --2- 172.21.15.62:0/1373372733 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f818010c8f0 0x7f818019c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:44:02.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:02.252+0000 7f8185800700 1 -- 172.21.15.62:0/1373372733 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f816c0095f0 con 0x7f8180106b80 2024-04-20T21:44:02.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:02.253+0000 7f8185800700 1 --2- 172.21.15.62:0/1373372733 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8180106b80 0x7f818019c440 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f817400c930 tx=0x7f817400ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:44:02.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:02.253+0000 7f817e000700 1 -- 172.21.15.62:0/1373372733 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8174007ad0 con 0x7f8180106b80 2024-04-20T21:44:02.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:02.253+0000 7f817e000700 1 -- 172.21.15.62:0/1373372733 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8174007c30 con 0x7f8180106b80 2024-04-20T21:44:02.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:02.253+0000 7f817e000700 1 -- 172.21.15.62:0/1373372733 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8174018610 con 0x7f8180106b80 2024-04-20T21:44:02.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:02.253+0000 7f8188200700 1 -- 172.21.15.62:0/1373372733 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81801a0f60 con 0x7f8180106b80 2024-04-20T21:44:02.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:02.253+0000 7f8188200700 1 -- 172.21.15.62:0/1373372733 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81801a1460 con 0x7f8180106b80 2024-04-20T21:44:02.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:02.254+0000 7f8188200700 1 -- 172.21.15.62:0/1373372733 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f818010ecc0 con 0x7f8180106b80 2024-04-20T21:44:02.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:02.256+0000 7f817e000700 1 -- 172.21.15.62:0/1373372733 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f817400f450 con 0x7f8180106b80 2024-04-20T21:44:02.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:02.257+0000 7f817e000700 1 --2- 172.21.15.62:0/1373372733 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f816806ecc0 0x7f8168071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:44:02.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:02.257+0000 7f817e000700 1 -- 172.21.15.62:0/1373372733 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f817408ef60 con 0x7f8180106b80 2024-04-20T21:44:02.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:02.257+0000 7f8184e00700 1 --2- 172.21.15.62:0/1373372733 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f816806ecc0 0x7f8168071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:44:02.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:02.258+0000 7f8184e00700 1 --2- 172.21.15.62:0/1373372733 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f816806ecc0 0x7f8168071170 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f818019da60 tx=0x7f816c01a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:44:02.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:02.266+0000 7f817e000700 1 -- 172.21.15.62:0/1373372733 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f8174057d90 con 0x7f8180106b80 2024-04-20T21:44:02.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:02.556+0000 7f8188200700 1 -- 172.21.15.62:0/1373372733 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f818010e6a0 con 0x7f8180106b80 2024-04-20T21:44:02.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:02.557+0000 7f817e000700 1 -- 172.21.15.62:0/1373372733 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 25 v25) v1 ==== 76+0+1887 (secure 0 0 0) 0x7f817405b450 con 0x7f8180106b80 2024-04-20T21:44:02.557 INFO:teuthology.orchestra.run.smithi062.stdout:e25 2024-04-20T21:44:02.557 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T21:44:02.557 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T21:44:02.558 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T21:44:02.558 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:44:02.558 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T21:44:02.558 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T21:44:02.558 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 20 2024-04-20T21:44:02.558 INFO:teuthology.orchestra.run.smithi062.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-20T21:44:02.558 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T21:44:02.558 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T21:42:01.559061+0000 2024-04-20T21:44:02.558 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T21:44:02.558 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T21:44:02.558 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T21:44:02.558 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T21:44:02.558 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T21:44:02.558 INFO:teuthology.orchestra.run.smithi062.stdout:max_xattr_size 65536 2024-04-20T21:44:02.559 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T21:44:02.559 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T21:44:02.559 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 81 2024-04-20T21:44:02.559 INFO:teuthology.orchestra.run.smithi062.stdout:snap_rstat 1 2024-04-20T21:44:02.559 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T21:44:02.559 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 2 2024-04-20T21:44:02.559 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1 2024-04-20T21:44:02.559 INFO:teuthology.orchestra.run.smithi062.stdout:up {} 2024-04-20T21:44:02.559 INFO:teuthology.orchestra.run.smithi062.stdout:failed 0,1 2024-04-20T21:44:02.559 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T21:44:02.559 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T21:44:02.559 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T21:44:02.559 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T21:44:02.559 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data enabled 2024-04-20T21:44:02.559 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T21:44:02.560 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T21:44:02.560 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 1 2024-04-20T21:44:02.560 INFO:teuthology.orchestra.run.smithi062.stdout:qdb_cluster leader: 0 members: 2024-04-20T21:44:02.560 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:44:02.560 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:44:02.560 INFO:teuthology.orchestra.run.smithi062.stdout:Standby daemons: 2024-04-20T21:44:02.560 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:44:02.560 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ryqblb{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6826/4113916205,v1:172.21.15.62:6827/4113916205] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T21:44:02.560 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ozizwi{-1:34476} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6828/3801975165,v1:172.21.15.62:6829/3801975165] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T21:44:02.560 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.xbgspw{-1:44285} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6826/3996040526,v1:172.21.15.189:6827/3996040526] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:44:02.560 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.dvimer{-1:44289} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6824/1743111843,v1:172.21.15.189:6825/1743111843] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:44:02.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:02.560+0000 7f8188200700 1 -- 172.21.15.62:0/1373372733 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f816806ecc0 msgr2=0x7f8168071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:44:02.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:02.560+0000 7f8188200700 1 --2- 172.21.15.62:0/1373372733 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f816806ecc0 0x7f8168071170 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f818019da60 tx=0x7f816c01a040 comp rx=0 tx=0).stop 2024-04-20T21:44:02.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:02.560+0000 7f8188200700 1 -- 172.21.15.62:0/1373372733 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8180106b80 msgr2=0x7f818019c440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:44:02.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:02.560+0000 7f8188200700 1 --2- 172.21.15.62:0/1373372733 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8180106b80 0x7f818019c440 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f817400c930 tx=0x7f817400ccf0 comp rx=0 tx=0).stop 2024-04-20T21:44:02.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:02.560+0000 7f8188200700 1 -- 172.21.15.62:0/1373372733 shutdown_connections 2024-04-20T21:44:02.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:02.560+0000 7f8188200700 1 --2- 172.21.15.62:0/1373372733 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f816806ecc0 0x7f8168071170 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:44:02.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:02.560+0000 7f8188200700 1 --2- 172.21.15.62:0/1373372733 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8180106b80 0x7f818019c440 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:44:02.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:02.560+0000 7f8188200700 1 --2- 172.21.15.62:0/1373372733 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f818010c8f0 0x7f818019c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:44:02.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:02.560+0000 7f8188200700 1 -- 172.21.15.62:0/1373372733 >> 172.21.15.62:0/1373372733 conn(0x7f8180076060 msgr2=0x7f8180077ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:44:02.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:02.561+0000 7f8188200700 1 -- 172.21.15.62:0/1373372733 shutdown_connections 2024-04-20T21:44:02.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:02.561+0000 7f8188200700 1 -- 172.21.15.62:0/1373372733 wait complete. 2024-04-20T21:44:02.562 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 25 2024-04-20T21:44:02.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:02.726+0000 7f5d91600700 1 -- 172.21.15.62:0/66556657 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d8c102470 msgr2=0x7f5d8c1028e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:44:02.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:02.726+0000 7f5d91600700 1 --2- 172.21.15.62:0/66556657 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d8c102470 0x7f5d8c1028e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f5d78009910 tx=0x7f5d78009c20 comp rx=0 tx=0).stop 2024-04-20T21:44:02.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:02.726+0000 7f5d91600700 1 -- 172.21.15.62:0/66556657 shutdown_connections 2024-04-20T21:44:02.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:02.726+0000 7f5d91600700 1 --2- 172.21.15.62:0/66556657 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d8c102470 0x7f5d8c1028e0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:44:02.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:02.726+0000 7f5d91600700 1 --2- 172.21.15.62:0/66556657 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d8c108470 0x7f5d8c108840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:44:02.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:02.726+0000 7f5d91600700 1 -- 172.21.15.62:0/66556657 >> 172.21.15.62:0/66556657 conn(0x7f5d8c0fdf90 msgr2=0x7f5d8c1003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:44:02.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:02.726+0000 7f5d91600700 1 -- 172.21.15.62:0/66556657 shutdown_connections 2024-04-20T21:44:02.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:02.726+0000 7f5d91600700 1 -- 172.21.15.62:0/66556657 wait complete. 2024-04-20T21:44:02.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:02.727+0000 7f5d91600700 1 Processor -- start 2024-04-20T21:44:02.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:02.728+0000 7f5d91600700 1 -- start start 2024-04-20T21:44:02.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:02.728+0000 7f5d91600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d8c102470 0x7f5d8c1981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:44:02.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:02.728+0000 7f5d91600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d8c108470 0x7f5d8c198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:44:02.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:02.728+0000 7f5d91600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d8c198e10 con 0x7f5d8c108470 2024-04-20T21:44:02.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:02.728+0000 7f5d91600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d8c19cba0 con 0x7f5d8c102470 2024-04-20T21:44:02.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:02.728+0000 7f5d8a000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d8c108470 0x7f5d8c198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:44:02.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:02.729+0000 7f5d8a000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d8c108470 0x7f5d8c198730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49076/0 (socket says 172.21.15.62:49076) 2024-04-20T21:44:02.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:02.729+0000 7f5d8a000700 1 -- 172.21.15.62:0/3127297583 learned_addr learned my addr 172.21.15.62:0/3127297583 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:44:02.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:02.729+0000 7f5d8aa00700 1 --2- 172.21.15.62:0/3127297583 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d8c102470 0x7f5d8c1981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:44:02.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:02.729+0000 7f5d8a000700 1 -- 172.21.15.62:0/3127297583 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d8c102470 msgr2=0x7f5d8c1981f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:44:02.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:02.729+0000 7f5d8a000700 1 --2- 172.21.15.62:0/3127297583 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d8c102470 0x7f5d8c1981f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:44:02.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:02.729+0000 7f5d8a000700 1 -- 172.21.15.62:0/3127297583 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d780095f0 con 0x7f5d8c108470 2024-04-20T21:44:02.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:02.729+0000 7f5d8a000700 1 --2- 172.21.15.62:0/3127297583 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d8c108470 0x7f5d8c198730 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f5d7800b5c0 tx=0x7f5d7800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:44:02.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:02.730+0000 7f5d83400700 1 -- 172.21.15.62:0/3127297583 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5d7801d070 con 0x7f5d8c108470 2024-04-20T21:44:02.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:02.730+0000 7f5d83400700 1 -- 172.21.15.62:0/3127297583 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5d78005e00 con 0x7f5d8c108470 2024-04-20T21:44:02.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:02.730+0000 7f5d91600700 1 -- 172.21.15.62:0/3127297583 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d8c19ce20 con 0x7f5d8c108470 2024-04-20T21:44:02.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:02.730+0000 7f5d83400700 1 -- 172.21.15.62:0/3127297583 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5d78017400 con 0x7f5d8c108470 2024-04-20T21:44:02.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:02.730+0000 7f5d91600700 1 -- 172.21.15.62:0/3127297583 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d8c19d310 con 0x7f5d8c108470 2024-04-20T21:44:02.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:02.731+0000 7f5d91600700 1 -- 172.21.15.62:0/3127297583 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d8c10a970 con 0x7f5d8c108470 2024-04-20T21:44:02.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:02.732+0000 7f5d83400700 1 -- 172.21.15.62:0/3127297583 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f5d7800f460 con 0x7f5d8c108470 2024-04-20T21:44:02.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:02.733+0000 7f5d83400700 1 --2- 172.21.15.62:0/3127297583 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5d7006ecc0 0x7f5d70071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:44:02.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:02.733+0000 7f5d83400700 1 -- 172.21.15.62:0/3127297583 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f5d78090370 con 0x7f5d8c108470 2024-04-20T21:44:02.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:02.733+0000 7f5d8aa00700 1 --2- 172.21.15.62:0/3127297583 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5d7006ecc0 0x7f5d70071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:44:02.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:02.734+0000 7f5d8aa00700 1 --2- 172.21.15.62:0/3127297583 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5d7006ecc0 0x7f5d70071170 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f5d7c005950 tx=0x7f5d7c0058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:44:02.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:02.738+0000 7f5d83400700 1 -- 172.21.15.62:0/3127297583 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f5d78059100 con 0x7f5d8c108470 2024-04-20T21:44:02.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:44:02 smithi189 ceph-mon[57022]: from='client.34540 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:44:02.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:44:02 smithi189 ceph-mon[57022]: from='client.34544 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:44:02.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:44:02 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/1331006492' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:44:02.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:44:02 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/1373372733' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T21:44:02.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:02.994+0000 7f5d91600700 1 -- 172.21.15.62:0/3127297583 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5d8c19d580 con 0x7f5d7006ecc0 2024-04-20T21:44:02.994 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:44:02 smithi062 ceph-mon[76322]: from='client.34540 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:44:02.994 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:44:02 smithi062 ceph-mon[76322]: from='client.34544 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:44:02.994 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:44:02 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/1331006492' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:44:02.994 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:44:02 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/1373372733' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T21:44:02.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:02.998+0000 7f5d83400700 1 -- 172.21.15.62:0/3127297583 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f5d8c19d580 con 0x7f5d7006ecc0 2024-04-20T21:44:02.997 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:44:02.997 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T21:44:02.997 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T21:44:02.997 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-20T21:44:02.997 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T21:44:02.997 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr", 2024-04-20T21:44:02.998 INFO:teuthology.orchestra.run.smithi062.stdout: "crash", 2024-04-20T21:44:02.998 INFO:teuthology.orchestra.run.smithi062.stdout: "mon", 2024-04-20T21:44:02.998 INFO:teuthology.orchestra.run.smithi062.stdout: "osd" 2024-04-20T21:44:02.998 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T21:44:02.998 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "14/23 daemons upgraded", 2024-04-20T21:44:02.998 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading mds daemons", 2024-04-20T21:44:02.998 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T21:44:02.998 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:44:03.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:03.001+0000 7f5d91600700 1 -- 172.21.15.62:0/3127297583 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5d7006ecc0 msgr2=0x7f5d70071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:44:03.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:03.001+0000 7f5d91600700 1 --2- 172.21.15.62:0/3127297583 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5d7006ecc0 0x7f5d70071170 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f5d7c005950 tx=0x7f5d7c0058e0 comp rx=0 tx=0).stop 2024-04-20T21:44:03.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:03.001+0000 7f5d91600700 1 -- 172.21.15.62:0/3127297583 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d8c108470 msgr2=0x7f5d8c198730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:44:03.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:03.001+0000 7f5d91600700 1 --2- 172.21.15.62:0/3127297583 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d8c108470 0x7f5d8c198730 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f5d7800b5c0 tx=0x7f5d7800be20 comp rx=0 tx=0).stop 2024-04-20T21:44:03.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:03.001+0000 7f5d91600700 1 -- 172.21.15.62:0/3127297583 shutdown_connections 2024-04-20T21:44:03.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:03.001+0000 7f5d91600700 1 --2- 172.21.15.62:0/3127297583 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5d7006ecc0 0x7f5d70071170 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:44:03.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:03.001+0000 7f5d91600700 1 --2- 172.21.15.62:0/3127297583 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d8c102470 0x7f5d8c1981f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:44:03.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:03.001+0000 7f5d91600700 1 --2- 172.21.15.62:0/3127297583 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d8c108470 0x7f5d8c198730 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:44:03.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:03.001+0000 7f5d91600700 1 -- 172.21.15.62:0/3127297583 >> 172.21.15.62:0/3127297583 conn(0x7f5d8c0fdf90 msgr2=0x7f5d8c0ff3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:44:03.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:03.001+0000 7f5d91600700 1 -- 172.21.15.62:0/3127297583 shutdown_connections 2024-04-20T21:44:03.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:03.001+0000 7f5d91600700 1 -- 172.21.15.62:0/3127297583 wait complete. 2024-04-20T21:44:03.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:03.159+0000 7f6073e00700 1 -- 172.21.15.62:0/3692983632 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6074103ec0 msgr2=0x7f6074104290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:44:03.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:03.159+0000 7f6073e00700 1 --2- 172.21.15.62:0/3692983632 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6074103ec0 0x7f6074104290 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f6064009910 tx=0x7f6064009c20 comp rx=0 tx=0).stop 2024-04-20T21:44:03.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:03.159+0000 7f6073e00700 1 -- 172.21.15.62:0/3692983632 shutdown_connections 2024-04-20T21:44:03.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:03.159+0000 7f6073e00700 1 --2- 172.21.15.62:0/3692983632 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60740ff9d0 0x7f60740ffe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:44:03.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:03.159+0000 7f6073e00700 1 --2- 172.21.15.62:0/3692983632 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6074103ec0 0x7f6074104290 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:44:03.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:03.159+0000 7f6073e00700 1 -- 172.21.15.62:0/3692983632 >> 172.21.15.62:0/3692983632 conn(0x7f60740f9d40 msgr2=0x7f60740fc150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:44:03.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:03.159+0000 7f6073e00700 1 -- 172.21.15.62:0/3692983632 shutdown_connections 2024-04-20T21:44:03.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:03.159+0000 7f6073e00700 1 -- 172.21.15.62:0/3692983632 wait complete. 2024-04-20T21:44:03.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:03.160+0000 7f6073e00700 1 Processor -- start 2024-04-20T21:44:03.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:03.161+0000 7f6073e00700 1 -- start start 2024-04-20T21:44:03.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:03.161+0000 7f6073e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60740ff9d0 0x7f6074197da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:44:03.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:03.161+0000 7f6073e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6074103ec0 0x7f60741982e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:44:03.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:03.161+0000 7f6073e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60741989c0 con 0x7f6074103ec0 2024-04-20T21:44:03.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:03.161+0000 7f6073e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f607419c750 con 0x7f60740ff9d0 2024-04-20T21:44:03.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:03.161+0000 7f6072000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6074103ec0 0x7f60741982e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:44:03.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:03.161+0000 7f6072000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6074103ec0 0x7f60741982e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49094/0 (socket says 172.21.15.62:49094) 2024-04-20T21:44:03.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:03.161+0000 7f6072000700 1 -- 172.21.15.62:0/1684194526 learned_addr learned my addr 172.21.15.62:0/1684194526 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:44:03.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:03.162+0000 7f6072a00700 1 --2- 172.21.15.62:0/1684194526 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60740ff9d0 0x7f6074197da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:44:03.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:03.162+0000 7f6072000700 1 -- 172.21.15.62:0/1684194526 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60740ff9d0 msgr2=0x7f6074197da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:44:03.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:03.162+0000 7f6072000700 1 --2- 172.21.15.62:0/1684194526 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60740ff9d0 0x7f6074197da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:44:03.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:03.162+0000 7f6072000700 1 -- 172.21.15.62:0/1684194526 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60640095f0 con 0x7f6074103ec0 2024-04-20T21:44:03.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:03.162+0000 7f6072000700 1 --2- 172.21.15.62:0/1684194526 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6074103ec0 0x7f60741982e0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f606000d8d0 tx=0x7f606000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:44:03.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:03.162+0000 7f6078a00700 1 -- 172.21.15.62:0/1684194526 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6060009940 con 0x7f6074103ec0 2024-04-20T21:44:03.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:03.163+0000 7f6078a00700 1 -- 172.21.15.62:0/1684194526 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6060010460 con 0x7f6074103ec0 2024-04-20T21:44:03.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:03.163+0000 7f6073e00700 1 -- 172.21.15.62:0/1684194526 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f607419ca30 con 0x7f6074103ec0 2024-04-20T21:44:03.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:03.163+0000 7f6078a00700 1 -- 172.21.15.62:0/1684194526 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f606000f5f0 con 0x7f6074103ec0 2024-04-20T21:44:03.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:03.163+0000 7f6073e00700 1 -- 172.21.15.62:0/1684194526 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f607419d000 con 0x7f6074103ec0 2024-04-20T21:44:03.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:03.164+0000 7f6073e00700 1 -- 172.21.15.62:0/1684194526 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6074062380 con 0x7f6074103ec0 2024-04-20T21:44:03.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:03.165+0000 7f6078a00700 1 -- 172.21.15.62:0/1684194526 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f60600105d0 con 0x7f6074103ec0 2024-04-20T21:44:03.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:03.166+0000 7f6078a00700 1 --2- 172.21.15.62:0/1684194526 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f605806ecc0 0x7f6058071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:44:03.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:03.167+0000 7f6078a00700 1 -- 172.21.15.62:0/1684194526 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f606008ef20 con 0x7f6074103ec0 2024-04-20T21:44:03.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:03.167+0000 7f6072a00700 1 --2- 172.21.15.62:0/1684194526 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f605806ecc0 0x7f6058071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:44:03.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:03.167+0000 7f6072a00700 1 --2- 172.21.15.62:0/1684194526 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f605806ecc0 0x7f6058071170 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f6064006010 tx=0x7f606401a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:44:03.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:03.172+0000 7f6078a00700 1 -- 172.21.15.62:0/1684194526 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f6060057c30 con 0x7f6074103ec0 2024-04-20T21:44:03.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:03.503+0000 7f6073e00700 1 -- 172.21.15.62:0/1684194526 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f6074199100 con 0x7f6074103ec0 2024-04-20T21:44:03.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:03.504+0000 7f6078a00700 1 -- 172.21.15.62:0/1684194526 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7f606005b2f0 con 0x7f6074103ec0 2024-04-20T21:44:03.503 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-20T21:44:03.503 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-20T21:44:03.503 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is degraded 2024-04-20T21:44:03.504 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T21:44:03.504 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T21:44:03.504 INFO:teuthology.orchestra.run.smithi062.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-20T21:44:03.504 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-20T21:44:03.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:03.507+0000 7f6073e00700 1 -- 172.21.15.62:0/1684194526 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f605806ecc0 msgr2=0x7f6058071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:44:03.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:03.507+0000 7f6073e00700 1 --2- 172.21.15.62:0/1684194526 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f605806ecc0 0x7f6058071170 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f6064006010 tx=0x7f606401a040 comp rx=0 tx=0).stop 2024-04-20T21:44:03.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:03.507+0000 7f6073e00700 1 -- 172.21.15.62:0/1684194526 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6074103ec0 msgr2=0x7f60741982e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:44:03.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:03.507+0000 7f6073e00700 1 --2- 172.21.15.62:0/1684194526 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6074103ec0 0x7f60741982e0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f606000d8d0 tx=0x7f606000dc90 comp rx=0 tx=0).stop 2024-04-20T21:44:03.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:03.508+0000 7f6073e00700 1 -- 172.21.15.62:0/1684194526 shutdown_connections 2024-04-20T21:44:03.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:03.508+0000 7f6073e00700 1 --2- 172.21.15.62:0/1684194526 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f605806ecc0 0x7f6058071170 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:44:03.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:03.508+0000 7f6073e00700 1 --2- 172.21.15.62:0/1684194526 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60740ff9d0 0x7f6074197da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:44:03.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:03.508+0000 7f6073e00700 1 --2- 172.21.15.62:0/1684194526 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6074103ec0 0x7f60741982e0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:44:03.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:03.508+0000 7f6073e00700 1 -- 172.21.15.62:0/1684194526 >> 172.21.15.62:0/1684194526 conn(0x7f60740f9d40 msgr2=0x7f60740fab40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:44:03.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:03.508+0000 7f6073e00700 1 -- 172.21.15.62:0/1684194526 shutdown_connections 2024-04-20T21:44:03.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:03.508+0000 7f6073e00700 1 -- 172.21.15.62:0/1684194526 wait complete. 2024-04-20T21:44:03.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:44:03 smithi062 ceph-mon[76322]: pgmap v262: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:44:03.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:44:03 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/1684194526' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T21:44:03.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:44:03 smithi189 ceph-mon[57022]: pgmap v262: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:44:03.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:44:03 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/1684194526' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T21:44:04.959 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:44:04 smithi189 ceph-mon[57022]: from='client.34556 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:44:04.959 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:44:04 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:44:05.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:44:04 smithi062 ceph-mon[76322]: from='client.34556 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:44:05.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:44:04 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:44:05.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:44:05 smithi189 ceph-mon[57022]: pgmap v263: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:44:06.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:44:05 smithi062 ceph-mon[76322]: pgmap v263: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:44:07.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:44:07 smithi189 ceph-mon[57022]: pgmap v264: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:44:08.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:44:07 smithi062 ceph-mon[76322]: pgmap v264: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:44:09.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:44:09 smithi189 ceph-mon[57022]: pgmap v265: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:44:10.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:44:09 smithi062 ceph-mon[76322]: pgmap v265: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:44:11.986 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:44:11 smithi189 ceph-mon[57022]: pgmap v266: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:44:12.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:44:11 smithi062 ceph-mon[76322]: pgmap v266: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:44:13.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:44:13 smithi189 ceph-mon[57022]: pgmap v267: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:44:14.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:44:13 smithi062 ceph-mon[76322]: pgmap v267: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:44:15.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:44:15 smithi189 ceph-mon[57022]: pgmap v268: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:44:16.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:44:15 smithi062 ceph-mon[76322]: pgmap v268: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:44:17.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:44:17 smithi189 ceph-mon[57022]: pgmap v269: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:44:18.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:44:17 smithi062 ceph-mon[76322]: pgmap v269: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:44:19.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:44:19 smithi189 ceph-mon[57022]: pgmap v270: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:44:19.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:44:19 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:44:20.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:44:19 smithi062 ceph-mon[76322]: pgmap v270: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:44:20.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:44:19 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:44:21.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:44:21 smithi189 ceph-mon[57022]: pgmap v271: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:44:22.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:44:21 smithi062 ceph-mon[76322]: pgmap v271: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:44:23.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:44:23 smithi189 ceph-mon[57022]: pgmap v272: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:44:24.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:44:23 smithi062 ceph-mon[76322]: pgmap v272: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:44:25.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:44:25 smithi189 ceph-mon[57022]: pgmap v273: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:44:26.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:44:25 smithi062 ceph-mon[76322]: pgmap v273: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:44:27.973 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:44:27 smithi062 ceph-mon[76322]: pgmap v274: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:44:27.986 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:44:27 smithi189 ceph-mon[57022]: pgmap v274: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:44:29.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:44:29 smithi189 ceph-mon[57022]: pgmap v275: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:44:30.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:44:29 smithi062 ceph-mon[76322]: pgmap v275: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:44:31.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:44:31 smithi189 ceph-mon[57022]: pgmap v276: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:44:32.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:44:31 smithi062 ceph-mon[76322]: pgmap v276: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:44:33.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:33.686+0000 7f556b600700 1 -- 172.21.15.62:0/222149062 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f556410c8f0 msgr2=0x7f556410ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:44:33.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:33.686+0000 7f556b600700 1 --2- 172.21.15.62:0/222149062 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f556410c8f0 0x7f556410ccc0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f5558009910 tx=0x7f5558009c20 comp rx=0 tx=0).stop 2024-04-20T21:44:33.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:33.687+0000 7f556b600700 1 -- 172.21.15.62:0/222149062 shutdown_connections 2024-04-20T21:44:33.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:33.687+0000 7f556b600700 1 --2- 172.21.15.62:0/222149062 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5564106b80 0x7f5564106ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:44:33.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:33.687+0000 7f556b600700 1 --2- 172.21.15.62:0/222149062 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f556410c8f0 0x7f556410ccc0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:44:33.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:33.687+0000 7f556b600700 1 -- 172.21.15.62:0/222149062 >> 172.21.15.62:0/222149062 conn(0x7f5564076060 msgr2=0x7f5564078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:44:33.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:33.687+0000 7f556b600700 1 -- 172.21.15.62:0/222149062 shutdown_connections 2024-04-20T21:44:33.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:33.687+0000 7f556b600700 1 -- 172.21.15.62:0/222149062 wait complete. 2024-04-20T21:44:33.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:33.688+0000 7f556b600700 1 Processor -- start 2024-04-20T21:44:33.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:33.688+0000 7f556b600700 1 -- start start 2024-04-20T21:44:33.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:33.688+0000 7f556b600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5564106b80 0x7f556419c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:44:33.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:33.688+0000 7f556b600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f556410c8f0 0x7f556419ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:44:33.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:33.688+0000 7f556b600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f556419d120 con 0x7f556410c8f0 2024-04-20T21:44:33.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:33.688+0000 7f556b600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55641a0eb0 con 0x7f5564106b80 2024-04-20T21:44:33.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:33.689+0000 7f5563e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f556410c8f0 0x7f556419ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:44:33.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:33.689+0000 7f5563e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f556410c8f0 0x7f556419ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54588/0 (socket says 172.21.15.62:54588) 2024-04-20T21:44:33.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:33.689+0000 7f5563e00700 1 -- 172.21.15.62:0/3196338570 learned_addr learned my addr 172.21.15.62:0/3196338570 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:44:33.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:33.689+0000 7f5568c00700 1 --2- 172.21.15.62:0/3196338570 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5564106b80 0x7f556419c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:44:33.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:33.689+0000 7f5563e00700 1 -- 172.21.15.62:0/3196338570 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5564106b80 msgr2=0x7f556419c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:44:33.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:33.689+0000 7f5563e00700 1 --2- 172.21.15.62:0/3196338570 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5564106b80 0x7f556419c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:44:33.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:33.689+0000 7f5563e00700 1 -- 172.21.15.62:0/3196338570 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55580095f0 con 0x7f556410c8f0 2024-04-20T21:44:33.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:33.690+0000 7f5563e00700 1 --2- 172.21.15.62:0/3196338570 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f556410c8f0 0x7f556419ca40 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f555000c8a0 tx=0x7f555000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:44:33.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:33.690+0000 7f5561600700 1 -- 172.21.15.62:0/3196338570 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f555000cea0 con 0x7f556410c8f0 2024-04-20T21:44:33.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:33.690+0000 7f5561600700 1 -- 172.21.15.62:0/3196338570 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5550004830 con 0x7f556410c8f0 2024-04-20T21:44:33.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:33.690+0000 7f556b600700 1 -- 172.21.15.62:0/3196338570 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55641a1190 con 0x7f556410c8f0 2024-04-20T21:44:33.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:33.690+0000 7f5561600700 1 -- 172.21.15.62:0/3196338570 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f55500055b0 con 0x7f556410c8f0 2024-04-20T21:44:33.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:33.690+0000 7f556b600700 1 -- 172.21.15.62:0/3196338570 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55641a16e0 con 0x7f556410c8f0 2024-04-20T21:44:33.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:33.691+0000 7f556b600700 1 -- 172.21.15.62:0/3196338570 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5564109b20 con 0x7f556410c8f0 2024-04-20T21:44:33.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:33.692+0000 7f5561600700 1 -- 172.21.15.62:0/3196338570 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f55500049a0 con 0x7f556410c8f0 2024-04-20T21:44:33.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:33.693+0000 7f5561600700 1 --2- 172.21.15.62:0/3196338570 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f554c06ecc0 0x7f554c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:44:33.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:33.693+0000 7f5561600700 1 -- 172.21.15.62:0/3196338570 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f555008ebf0 con 0x7f556410c8f0 2024-04-20T21:44:33.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:33.694+0000 7f5568c00700 1 --2- 172.21.15.62:0/3196338570 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f554c06ecc0 0x7f554c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:44:33.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:33.694+0000 7f5568c00700 1 --2- 172.21.15.62:0/3196338570 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f554c06ecc0 0x7f554c071170 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f5558006010 tx=0x7f555801a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:44:33.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:33.699+0000 7f5561600700 1 -- 172.21.15.62:0/3196338570 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f5550057900 con 0x7f556410c8f0 2024-04-20T21:44:33.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:33.960+0000 7f556b600700 1 -- 172.21.15.62:0/3196338570 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f55641a19d0 con 0x7f554c06ecc0 2024-04-20T21:44:33.960 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:44:33 smithi062 ceph-mon[76322]: pgmap v277: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:44:33.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:33.963+0000 7f5561600700 1 -- 172.21.15.62:0/3196338570 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f55641a19d0 con 0x7f554c06ecc0 2024-04-20T21:44:33.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:33.967+0000 7f556b600700 1 -- 172.21.15.62:0/3196338570 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f554c06ecc0 msgr2=0x7f554c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:44:33.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:33.967+0000 7f556b600700 1 --2- 172.21.15.62:0/3196338570 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f554c06ecc0 0x7f554c071170 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f5558006010 tx=0x7f555801a040 comp rx=0 tx=0).stop 2024-04-20T21:44:33.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:33.967+0000 7f556b600700 1 -- 172.21.15.62:0/3196338570 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f556410c8f0 msgr2=0x7f556419ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:44:33.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:33.967+0000 7f556b600700 1 --2- 172.21.15.62:0/3196338570 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f556410c8f0 0x7f556419ca40 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f555000c8a0 tx=0x7f555000cc60 comp rx=0 tx=0).stop 2024-04-20T21:44:33.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:33.967+0000 7f556b600700 1 -- 172.21.15.62:0/3196338570 shutdown_connections 2024-04-20T21:44:33.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:33.967+0000 7f556b600700 1 --2- 172.21.15.62:0/3196338570 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f554c06ecc0 0x7f554c071170 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:44:33.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:33.967+0000 7f556b600700 1 --2- 172.21.15.62:0/3196338570 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5564106b80 0x7f556419c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:44:33.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:33.967+0000 7f556b600700 1 --2- 172.21.15.62:0/3196338570 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f556410c8f0 0x7f556419ca40 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:44:33.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:33.967+0000 7f556b600700 1 -- 172.21.15.62:0/3196338570 >> 172.21.15.62:0/3196338570 conn(0x7f5564076060 msgr2=0x7f5564077aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:44:33.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:33.967+0000 7f556b600700 1 -- 172.21.15.62:0/3196338570 shutdown_connections 2024-04-20T21:44:33.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:33.967+0000 7f556b600700 1 -- 172.21.15.62:0/3196338570 wait complete. 2024-04-20T21:44:33.983 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T21:44:33.986 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:44:33 smithi189 ceph-mon[57022]: pgmap v277: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:44:34.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:34.136+0000 7f6a39000700 1 -- 172.21.15.62:0/73976316 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6a34108470 msgr2=0x7f6a34108840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:44:34.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:34.136+0000 7f6a39000700 1 --2- 172.21.15.62:0/73976316 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6a34108470 0x7f6a34108840 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f6a24009910 tx=0x7f6a24009c20 comp rx=0 tx=0).stop 2024-04-20T21:44:34.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:34.136+0000 7f6a39000700 1 -- 172.21.15.62:0/73976316 shutdown_connections 2024-04-20T21:44:34.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:34.136+0000 7f6a39000700 1 --2- 172.21.15.62:0/73976316 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a34102470 0x7f6a341028e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:44:34.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:34.136+0000 7f6a39000700 1 --2- 172.21.15.62:0/73976316 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6a34108470 0x7f6a34108840 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:44:34.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:34.136+0000 7f6a39000700 1 -- 172.21.15.62:0/73976316 >> 172.21.15.62:0/73976316 conn(0x7f6a340fdf90 msgr2=0x7f6a341003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:44:34.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:34.136+0000 7f6a39000700 1 -- 172.21.15.62:0/73976316 shutdown_connections 2024-04-20T21:44:34.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:34.136+0000 7f6a39000700 1 -- 172.21.15.62:0/73976316 wait complete. 2024-04-20T21:44:34.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:34.137+0000 7f6a39000700 1 Processor -- start 2024-04-20T21:44:34.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:34.138+0000 7f6a39000700 1 -- start start 2024-04-20T21:44:34.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:34.138+0000 7f6a39000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6a34102470 0x7f6a341981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:44:34.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:34.138+0000 7f6a39000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a34108470 0x7f6a34198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:44:34.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:34.138+0000 7f6a39000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a34198e10 con 0x7f6a34102470 2024-04-20T21:44:34.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:34.138+0000 7f6a39000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a3419cba0 con 0x7f6a34108470 2024-04-20T21:44:34.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:34.138+0000 7f6a31e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6a34102470 0x7f6a341981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:44:34.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:34.138+0000 7f6a31e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6a34102470 0x7f6a341981f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54608/0 (socket says 172.21.15.62:54608) 2024-04-20T21:44:34.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:34.138+0000 7f6a31e00700 1 -- 172.21.15.62:0/385154289 learned_addr learned my addr 172.21.15.62:0/385154289 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:44:34.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:34.139+0000 7f6a31e00700 1 -- 172.21.15.62:0/385154289 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a34108470 msgr2=0x7f6a34198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:44:34.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:34.139+0000 7f6a31e00700 1 --2- 172.21.15.62:0/385154289 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a34108470 0x7f6a34198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:44:34.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:34.139+0000 7f6a31e00700 1 -- 172.21.15.62:0/385154289 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a240095f0 con 0x7f6a34102470 2024-04-20T21:44:34.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:34.139+0000 7f6a31e00700 1 --2- 172.21.15.62:0/385154289 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6a34102470 0x7f6a341981f0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f6a240092e0 tx=0x7f6a2400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:44:34.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:34.139+0000 7f6a2aa00700 1 -- 172.21.15.62:0/385154289 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6a2401d070 con 0x7f6a34102470 2024-04-20T21:44:34.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:34.139+0000 7f6a2aa00700 1 -- 172.21.15.62:0/385154289 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6a24005e00 con 0x7f6a34102470 2024-04-20T21:44:34.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:34.139+0000 7f6a2aa00700 1 -- 172.21.15.62:0/385154289 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6a24017400 con 0x7f6a34102470 2024-04-20T21:44:34.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:34.139+0000 7f6a39000700 1 -- 172.21.15.62:0/385154289 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a3419ce20 con 0x7f6a34102470 2024-04-20T21:44:34.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:34.140+0000 7f6a39000700 1 -- 172.21.15.62:0/385154289 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a3419d310 con 0x7f6a34102470 2024-04-20T21:44:34.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:34.141+0000 7f6a39000700 1 -- 172.21.15.62:0/385154289 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a3410a9e0 con 0x7f6a34102470 2024-04-20T21:44:34.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:34.142+0000 7f6a2aa00700 1 -- 172.21.15.62:0/385154289 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f6a2400f460 con 0x7f6a34102470 2024-04-20T21:44:34.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:34.143+0000 7f6a2aa00700 1 --2- 172.21.15.62:0/385154289 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6a1806ecc0 0x7f6a18071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:44:34.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:34.143+0000 7f6a2aa00700 1 -- 172.21.15.62:0/385154289 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f6a24090290 con 0x7f6a34102470 2024-04-20T21:44:34.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:34.143+0000 7f6a31400700 1 --2- 172.21.15.62:0/385154289 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6a1806ecc0 0x7f6a18071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:44:34.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:34.144+0000 7f6a31400700 1 --2- 172.21.15.62:0/385154289 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6a1806ecc0 0x7f6a18071170 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f6a1c005950 tx=0x7f6a1c0058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:44:34.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:34.148+0000 7f6a2aa00700 1 -- 172.21.15.62:0/385154289 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f6a240590c0 con 0x7f6a34102470 2024-04-20T21:44:34.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:34.406+0000 7f6a39000700 1 -- 172.21.15.62:0/385154289 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6a341995f0 con 0x7f6a1806ecc0 2024-04-20T21:44:34.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:34.418+0000 7f6a2aa00700 1 -- 172.21.15.62:0/385154289 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f6a341995f0 con 0x7f6a1806ecc0 2024-04-20T21:44:34.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:34.421+0000 7f6a39000700 1 -- 172.21.15.62:0/385154289 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6a1806ecc0 msgr2=0x7f6a18071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:44:34.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:34.421+0000 7f6a39000700 1 --2- 172.21.15.62:0/385154289 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6a1806ecc0 0x7f6a18071170 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f6a1c005950 tx=0x7f6a1c0058e0 comp rx=0 tx=0).stop 2024-04-20T21:44:34.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:34.421+0000 7f6a39000700 1 -- 172.21.15.62:0/385154289 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6a34102470 msgr2=0x7f6a341981f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:44:34.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:34.422+0000 7f6a39000700 1 --2- 172.21.15.62:0/385154289 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6a34102470 0x7f6a341981f0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f6a240092e0 tx=0x7f6a2400be20 comp rx=0 tx=0).stop 2024-04-20T21:44:34.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:34.422+0000 7f6a39000700 1 -- 172.21.15.62:0/385154289 shutdown_connections 2024-04-20T21:44:34.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:34.422+0000 7f6a39000700 1 --2- 172.21.15.62:0/385154289 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6a1806ecc0 0x7f6a18071170 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:44:34.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:34.422+0000 7f6a39000700 1 --2- 172.21.15.62:0/385154289 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6a34102470 0x7f6a341981f0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:44:34.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:34.422+0000 7f6a39000700 1 --2- 172.21.15.62:0/385154289 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a34108470 0x7f6a34198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:44:34.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:34.422+0000 7f6a39000700 1 -- 172.21.15.62:0/385154289 >> 172.21.15.62:0/385154289 conn(0x7f6a340fdf90 msgr2=0x7f6a341002c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:44:34.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:34.422+0000 7f6a39000700 1 -- 172.21.15.62:0/385154289 shutdown_connections 2024-04-20T21:44:34.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:34.422+0000 7f6a39000700 1 -- 172.21.15.62:0/385154289 wait complete. 2024-04-20T21:44:34.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:34.585+0000 7f6cd4e00700 1 -- 172.21.15.62:0/3359395168 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6cc8095d60 msgr2=0x7f6cc8099d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:44:34.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:34.585+0000 7f6cd4e00700 1 --2- 172.21.15.62:0/3359395168 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6cc8095d60 0x7f6cc8099d30 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f6cbc009910 tx=0x7f6cbc009c20 comp rx=0 tx=0).stop 2024-04-20T21:44:34.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:34.585+0000 7f6cd4e00700 1 -- 172.21.15.62:0/3359395168 shutdown_connections 2024-04-20T21:44:34.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:34.585+0000 7f6cd4e00700 1 --2- 172.21.15.62:0/3359395168 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6cc8095d60 0x7f6cc8099d30 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:44:34.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:34.585+0000 7f6cd4e00700 1 --2- 172.21.15.62:0/3359395168 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6cc80953c0 0x7f6cc8095790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:44:34.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:34.585+0000 7f6cd4e00700 1 -- 172.21.15.62:0/3359395168 >> 172.21.15.62:0/3359395168 conn(0x7f6cc800be20 msgr2=0x7f6cc800c220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:44:34.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:34.585+0000 7f6cd4e00700 1 -- 172.21.15.62:0/3359395168 shutdown_connections 2024-04-20T21:44:34.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:34.585+0000 7f6cd4e00700 1 -- 172.21.15.62:0/3359395168 wait complete. 2024-04-20T21:44:34.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:34.586+0000 7f6cd4e00700 1 Processor -- start 2024-04-20T21:44:34.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:34.587+0000 7f6cd4e00700 1 -- start start 2024-04-20T21:44:34.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:34.587+0000 7f6cd4e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6cc80953c0 0x7f6cc812cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:44:34.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:34.587+0000 7f6cd4e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6cc8095d60 0x7f6cc812d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:44:34.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:34.587+0000 7f6cd4e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cc812dbf0 con 0x7f6cc80953c0 2024-04-20T21:44:34.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:34.587+0000 7f6cd4e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cc8131980 con 0x7f6cc8095d60 2024-04-20T21:44:34.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:34.587+0000 7f6ccf400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6cc80953c0 0x7f6cc812cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:44:34.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:34.587+0000 7f6ccf400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6cc80953c0 0x7f6cc812cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54630/0 (socket says 172.21.15.62:54630) 2024-04-20T21:44:34.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:34.587+0000 7f6ccf400700 1 -- 172.21.15.62:0/2737303967 learned_addr learned my addr 172.21.15.62:0/2737303967 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:44:34.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:34.588+0000 7f6ccea00700 1 --2- 172.21.15.62:0/2737303967 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6cc8095d60 0x7f6cc812d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:44:34.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:34.588+0000 7f6ccf400700 1 -- 172.21.15.62:0/2737303967 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6cc8095d60 msgr2=0x7f6cc812d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:44:34.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:34.588+0000 7f6ccf400700 1 --2- 172.21.15.62:0/2737303967 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6cc8095d60 0x7f6cc812d510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:44:34.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:34.588+0000 7f6ccf400700 1 -- 172.21.15.62:0/2737303967 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cbc0095f0 con 0x7f6cc80953c0 2024-04-20T21:44:34.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:34.588+0000 7f6ccf400700 1 --2- 172.21.15.62:0/2737303967 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6cc80953c0 0x7f6cc812cfd0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f6cc400d8d0 tx=0x7f6cc400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:44:34.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:34.589+0000 7f6cc3e00700 1 -- 172.21.15.62:0/2737303967 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6cc4009940 con 0x7f6cc80953c0 2024-04-20T21:44:34.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:34.589+0000 7f6cc3e00700 1 -- 172.21.15.62:0/2737303967 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6cc4010460 con 0x7f6cc80953c0 2024-04-20T21:44:34.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:34.589+0000 7f6cd4e00700 1 -- 172.21.15.62:0/2737303967 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cc8131c60 con 0x7f6cc80953c0 2024-04-20T21:44:34.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:34.589+0000 7f6cc3e00700 1 -- 172.21.15.62:0/2737303967 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6cc400f5f0 con 0x7f6cc80953c0 2024-04-20T21:44:34.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:34.589+0000 7f6cd4e00700 1 -- 172.21.15.62:0/2737303967 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cc81321b0 con 0x7f6cc80953c0 2024-04-20T21:44:34.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:34.590+0000 7f6cd4e00700 1 -- 172.21.15.62:0/2737303967 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cc8005e20 con 0x7f6cc80953c0 2024-04-20T21:44:34.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:34.591+0000 7f6cc3e00700 1 -- 172.21.15.62:0/2737303967 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f6cc40105d0 con 0x7f6cc80953c0 2024-04-20T21:44:34.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:34.593+0000 7f6cc3e00700 1 --2- 172.21.15.62:0/2737303967 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6cb806ecc0 0x7f6cb8071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:44:34.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:34.593+0000 7f6cc3e00700 1 -- 172.21.15.62:0/2737303967 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f6cc408ee10 con 0x7f6cc80953c0 2024-04-20T21:44:34.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:34.593+0000 7f6ccea00700 1 --2- 172.21.15.62:0/2737303967 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6cb806ecc0 0x7f6cb8071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:44:34.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:34.594+0000 7f6ccea00700 1 --2- 172.21.15.62:0/2737303967 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6cb806ecc0 0x7f6cb8071170 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f6cbc000c00 tx=0x7f6cbc005fd0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:44:34.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:34.598+0000 7f6cc3e00700 1 -- 172.21.15.62:0/2737303967 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f6cc4057ba0 con 0x7f6cc80953c0 2024-04-20T21:44:34.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:34.847+0000 7f6cd4e00700 1 -- 172.21.15.62:0/2737303967 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f6cc8132610 con 0x7f6cb806ecc0 2024-04-20T21:44:34.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:34.859+0000 7f6cc3e00700 1 -- 172.21.15.62:0/2737303967 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7f6cc8132610 con 0x7f6cb806ecc0 2024-04-20T21:44:34.859 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T21:44:34.859 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (10m) 107s ago 24m 14.2M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T21:44:34.859 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (24m) 107s ago 24m 9512k - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T21:44:34.859 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (23m) 2m ago 23m 8942k - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T21:44:34.859 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (6m) 107s ago 24m 6660k - 19.0.0-3133-ga8c84f59 76bc6b91b726 2825d5ba11c6 2024-04-20T21:44:34.859 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (6m) 2m ago 23m 6665k - 19.0.0-3133-ga8c84f59 76bc6b91b726 20f746fa1bba 2024-04-20T21:44:34.859 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (9m) 107s ago 23m 74.2M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T21:44:34.859 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (115s) 107s ago 16m 11.2M - 19.0.0-3133-ga8c84f59 76bc6b91b726 de80bc90a683 2024-04-20T21:44:34.859 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (2m) 107s ago 17m 11.9M - 19.0.0-3133-ga8c84f59 76bc6b91b726 cd793b7ee98e 2024-04-20T21:44:34.859 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (17m) 2m ago 17m 117M - 18.2.0 dc2bc1663786 fb2a8a223db8 2024-04-20T21:44:34.859 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (16m) 2m ago 16m 55.6M - 18.2.0 dc2bc1663786 f2eec2ee8951 2024-04-20T21:44:34.860 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (13m) 107s ago 26m 511M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T21:44:34.860 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (11m) 2m ago 22m 452M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T21:44:34.860 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (8m) 107s ago 26m 55.9M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 019eb1479485 2024-04-20T21:44:34.860 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (7m) 2m ago 22m 39.8M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 55aac560aab6 2024-04-20T21:44:34.860 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (11m) 107s ago 24m 7952k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T21:44:34.860 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (11m) 2m ago 23m 8304k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T21:44:34.860 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (5m) 107s ago 21m 88.0M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 ee2bb901f26e 2024-04-20T21:44:34.860 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (5m) 107s ago 21m 92.3M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 9075152d4162 2024-04-20T21:44:34.860 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (4m) 107s ago 20m 74.4M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 2d4f3e38dc0e 2024-04-20T21:44:34.860 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (4m) 2m ago 20m 87.6M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 8f97b17ae0eb 2024-04-20T21:44:34.860 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (3m) 2m ago 19m 90.7M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 653d8079967d 2024-04-20T21:44:34.860 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (2m) 2m ago 18m 11.5M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 596dded14c12 2024-04-20T21:44:34.860 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (10m) 107s ago 23m 74.0M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T21:44:34.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:34.862+0000 7f6cd4e00700 1 -- 172.21.15.62:0/2737303967 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6cb806ecc0 msgr2=0x7f6cb8071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:44:34.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:34.862+0000 7f6cd4e00700 1 --2- 172.21.15.62:0/2737303967 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6cb806ecc0 0x7f6cb8071170 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f6cbc000c00 tx=0x7f6cbc005fd0 comp rx=0 tx=0).stop 2024-04-20T21:44:34.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:34.862+0000 7f6cd4e00700 1 -- 172.21.15.62:0/2737303967 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6cc80953c0 msgr2=0x7f6cc812cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:44:34.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:34.862+0000 7f6cd4e00700 1 --2- 172.21.15.62:0/2737303967 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6cc80953c0 0x7f6cc812cfd0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f6cc400d8d0 tx=0x7f6cc400dc90 comp rx=0 tx=0).stop 2024-04-20T21:44:34.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:34.863+0000 7f6cd4e00700 1 -- 172.21.15.62:0/2737303967 shutdown_connections 2024-04-20T21:44:34.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:34.863+0000 7f6cd4e00700 1 --2- 172.21.15.62:0/2737303967 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6cb806ecc0 0x7f6cb8071170 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:44:34.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:34.863+0000 7f6cd4e00700 1 --2- 172.21.15.62:0/2737303967 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6cc80953c0 0x7f6cc812cfd0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:44:34.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:34.863+0000 7f6cd4e00700 1 --2- 172.21.15.62:0/2737303967 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6cc8095d60 0x7f6cc812d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:44:34.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:34.863+0000 7f6cd4e00700 1 -- 172.21.15.62:0/2737303967 >> 172.21.15.62:0/2737303967 conn(0x7f6cc800be20 msgr2=0x7f6cc8093610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:44:34.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:34.863+0000 7f6cd4e00700 1 -- 172.21.15.62:0/2737303967 shutdown_connections 2024-04-20T21:44:34.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:34.863+0000 7f6cd4e00700 1 -- 172.21.15.62:0/2737303967 wait complete. 2024-04-20T21:44:34.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:44:34 smithi189 ceph-mon[57022]: from='client.34564 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:44:34.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:44:34 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:44:35.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:35.026+0000 7fe19b200700 1 -- 172.21.15.62:0/3577424896 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe194106b80 msgr2=0x7fe194106ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:44:35.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:35.026+0000 7fe19b200700 1 --2- 172.21.15.62:0/3577424896 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe194106b80 0x7fe194106ff0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fe184009910 tx=0x7fe184009c20 comp rx=0 tx=0).stop 2024-04-20T21:44:35.025 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:44:34 smithi062 ceph-mon[76322]: from='client.34564 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:44:35.025 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:44:34 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:44:35.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:35.026+0000 7fe19b200700 1 -- 172.21.15.62:0/3577424896 shutdown_connections 2024-04-20T21:44:35.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:35.026+0000 7fe19b200700 1 --2- 172.21.15.62:0/3577424896 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe194106b80 0x7fe194106ff0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:44:35.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:35.026+0000 7fe19b200700 1 --2- 172.21.15.62:0/3577424896 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe19410c660 0x7fe19410ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:44:35.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:35.026+0000 7fe19b200700 1 -- 172.21.15.62:0/3577424896 >> 172.21.15.62:0/3577424896 conn(0x7fe194075e90 msgr2=0x7fe1940782a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:44:35.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:35.026+0000 7fe19b200700 1 -- 172.21.15.62:0/3577424896 shutdown_connections 2024-04-20T21:44:35.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:35.026+0000 7fe19b200700 1 -- 172.21.15.62:0/3577424896 wait complete. 2024-04-20T21:44:35.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:35.027+0000 7fe19b200700 1 Processor -- start 2024-04-20T21:44:35.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:35.027+0000 7fe19b200700 1 -- start start 2024-04-20T21:44:35.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:35.028+0000 7fe19b200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe194106b80 0x7fe19419c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:44:35.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:35.028+0000 7fe19b200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe19410c660 0x7fe19419c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:44:35.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:35.028+0000 7fe19b200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe19419ce00 con 0x7fe19410c660 2024-04-20T21:44:35.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:35.028+0000 7fe19b200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1941a0b90 con 0x7fe194106b80 2024-04-20T21:44:35.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:35.028+0000 7fe199400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe19410c660 0x7fe19419c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:44:35.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:35.028+0000 7fe199400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe19410c660 0x7fe19419c720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54656/0 (socket says 172.21.15.62:54656) 2024-04-20T21:44:35.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:35.028+0000 7fe199400700 1 -- 172.21.15.62:0/4088373507 learned_addr learned my addr 172.21.15.62:0/4088373507 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:44:35.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:35.029+0000 7fe199e00700 1 --2- 172.21.15.62:0/4088373507 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe194106b80 0x7fe19419c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:44:35.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:35.029+0000 7fe199400700 1 -- 172.21.15.62:0/4088373507 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe194106b80 msgr2=0x7fe19419c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:44:35.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:35.029+0000 7fe199400700 1 --2- 172.21.15.62:0/4088373507 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe194106b80 0x7fe19419c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:44:35.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:35.029+0000 7fe199400700 1 -- 172.21.15.62:0/4088373507 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1840095f0 con 0x7fe19410c660 2024-04-20T21:44:35.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:35.029+0000 7fe199400700 1 --2- 172.21.15.62:0/4088373507 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe19410c660 0x7fe19419c720 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fe184006010 tx=0x7fe18400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:44:35.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:35.030+0000 7fe18ea00700 1 -- 172.21.15.62:0/4088373507 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe18401d070 con 0x7fe19410c660 2024-04-20T21:44:35.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:35.030+0000 7fe18ea00700 1 -- 172.21.15.62:0/4088373507 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe18400f460 con 0x7fe19410c660 2024-04-20T21:44:35.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:35.030+0000 7fe19b200700 1 -- 172.21.15.62:0/4088373507 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1941a0e10 con 0x7fe19410c660 2024-04-20T21:44:35.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:35.030+0000 7fe18ea00700 1 -- 172.21.15.62:0/4088373507 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe184017630 con 0x7fe19410c660 2024-04-20T21:44:35.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:35.030+0000 7fe19b200700 1 -- 172.21.15.62:0/4088373507 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1941a1300 con 0x7fe19410c660 2024-04-20T21:44:35.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:35.031+0000 7fe19b200700 1 -- 172.21.15.62:0/4088373507 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe19410eb40 con 0x7fe19410c660 2024-04-20T21:44:35.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:35.032+0000 7fe18ea00700 1 -- 172.21.15.62:0/4088373507 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe18400f5d0 con 0x7fe19410c660 2024-04-20T21:44:35.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:35.033+0000 7fe18ea00700 1 --2- 172.21.15.62:0/4088373507 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe17c06ecc0 0x7fe17c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:44:35.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:35.033+0000 7fe18ea00700 1 -- 172.21.15.62:0/4088373507 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fe184090390 con 0x7fe19410c660 2024-04-20T21:44:35.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:35.034+0000 7fe199e00700 1 --2- 172.21.15.62:0/4088373507 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe17c06ecc0 0x7fe17c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:44:35.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:35.035+0000 7fe199e00700 1 --2- 172.21.15.62:0/4088373507 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe17c06ecc0 0x7fe17c071170 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fe188005950 tx=0x7fe1880058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:44:35.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:35.039+0000 7fe18ea00700 1 -- 172.21.15.62:0/4088373507 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fe1840590a0 con 0x7fe19410c660 2024-04-20T21:44:35.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:35.373+0000 7fe19b200700 1 -- 172.21.15.62:0/4088373507 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fe194062380 con 0x7fe19410c660 2024-04-20T21:44:35.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:35.374+0000 7fe18ea00700 1 -- 172.21.15.62:0/4088373507 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7fe18405c760 con 0x7fe19410c660 2024-04-20T21:44:35.374 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:44:35.374 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T21:44:35.374 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:44:35.374 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:44:35.374 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T21:44:35.374 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:44:35.374 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:44:35.374 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T21:44:35.375 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 6 2024-04-20T21:44:35.375 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:44:35.375 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T21:44:35.375 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 2, 2024-04-20T21:44:35.375 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:44:35.375 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:44:35.375 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T21:44:35.375 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 2, 2024-04-20T21:44:35.375 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 12 2024-04-20T21:44:35.375 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T21:44:35.375 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:44:35.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:35.378+0000 7fe19b200700 1 -- 172.21.15.62:0/4088373507 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe17c06ecc0 msgr2=0x7fe17c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:44:35.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:35.378+0000 7fe19b200700 1 --2- 172.21.15.62:0/4088373507 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe17c06ecc0 0x7fe17c071170 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fe188005950 tx=0x7fe1880058e0 comp rx=0 tx=0).stop 2024-04-20T21:44:35.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:35.378+0000 7fe19b200700 1 -- 172.21.15.62:0/4088373507 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe19410c660 msgr2=0x7fe19419c720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:44:35.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:35.378+0000 7fe19b200700 1 --2- 172.21.15.62:0/4088373507 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe19410c660 0x7fe19419c720 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fe184006010 tx=0x7fe18400be20 comp rx=0 tx=0).stop 2024-04-20T21:44:35.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:35.378+0000 7fe19b200700 1 -- 172.21.15.62:0/4088373507 shutdown_connections 2024-04-20T21:44:35.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:35.378+0000 7fe19b200700 1 --2- 172.21.15.62:0/4088373507 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe17c06ecc0 0x7fe17c071170 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:44:35.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:35.378+0000 7fe19b200700 1 --2- 172.21.15.62:0/4088373507 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe194106b80 0x7fe19419c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:44:35.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:35.378+0000 7fe19b200700 1 --2- 172.21.15.62:0/4088373507 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe19410c660 0x7fe19419c720 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:44:35.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:35.378+0000 7fe19b200700 1 -- 172.21.15.62:0/4088373507 >> 172.21.15.62:0/4088373507 conn(0x7fe194075e90 msgr2=0x7fe1940778d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:44:35.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:35.379+0000 7fe19b200700 1 -- 172.21.15.62:0/4088373507 shutdown_connections 2024-04-20T21:44:35.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:35.379+0000 7fe19b200700 1 -- 172.21.15.62:0/4088373507 wait complete. 2024-04-20T21:44:35.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:35.550+0000 7f146aa00700 1 -- 172.21.15.62:0/3742493048 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f14641004e0 msgr2=0x7f14641008b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:44:35.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:35.550+0000 7f146aa00700 1 --2- 172.21.15.62:0/3742493048 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f14641004e0 0x7f14641008b0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f1454009910 tx=0x7f1454009c20 comp rx=0 tx=0).stop 2024-04-20T21:44:35.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:35.550+0000 7f146aa00700 1 -- 172.21.15.62:0/3742493048 shutdown_connections 2024-04-20T21:44:35.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:35.550+0000 7f146aa00700 1 --2- 172.21.15.62:0/3742493048 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1464100e80 0x7f1464104e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:44:35.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:35.550+0000 7f146aa00700 1 --2- 172.21.15.62:0/3742493048 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f14641004e0 0x7f14641008b0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:44:35.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:35.550+0000 7f146aa00700 1 -- 172.21.15.62:0/3742493048 >> 172.21.15.62:0/3742493048 conn(0x7f14640fbda0 msgr2=0x7f14640fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:44:35.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:35.551+0000 7f146aa00700 1 -- 172.21.15.62:0/3742493048 shutdown_connections 2024-04-20T21:44:35.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:35.551+0000 7f146aa00700 1 -- 172.21.15.62:0/3742493048 wait complete. 2024-04-20T21:44:35.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:35.552+0000 7f146aa00700 1 Processor -- start 2024-04-20T21:44:35.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:35.552+0000 7f146aa00700 1 -- start start 2024-04-20T21:44:35.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:35.552+0000 7f146aa00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f14641004e0 0x7f1464073050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:44:35.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:35.552+0000 7f146aa00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1464100e80 0x7f1464073590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:44:35.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:35.552+0000 7f146aa00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1464073ad0 con 0x7f14641004e0 2024-04-20T21:44:35.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:35.552+0000 7f146aa00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1464073c10 con 0x7f1464100e80 2024-04-20T21:44:35.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:35.553+0000 7f1463e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f14641004e0 0x7f1464073050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:44:35.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:35.553+0000 7f1463e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f14641004e0 0x7f1464073050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54670/0 (socket says 172.21.15.62:54670) 2024-04-20T21:44:35.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:35.553+0000 7f1463e00700 1 -- 172.21.15.62:0/4085903817 learned_addr learned my addr 172.21.15.62:0/4085903817 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:44:35.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:35.553+0000 7f145b400700 1 --2- 172.21.15.62:0/4085903817 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1464100e80 0x7f1464073590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:44:35.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:35.553+0000 7f1463e00700 1 -- 172.21.15.62:0/4085903817 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1464100e80 msgr2=0x7f1464073590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:44:35.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:35.553+0000 7f1463e00700 1 --2- 172.21.15.62:0/4085903817 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1464100e80 0x7f1464073590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:44:35.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:35.553+0000 7f1463e00700 1 -- 172.21.15.62:0/4085903817 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14540095f0 con 0x7f14641004e0 2024-04-20T21:44:35.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:35.554+0000 7f1463e00700 1 --2- 172.21.15.62:0/4085903817 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f14641004e0 0x7f1464073050 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f1454006010 tx=0x7f145400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:44:35.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:35.554+0000 7f1461600700 1 -- 172.21.15.62:0/4085903817 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f145401d070 con 0x7f14641004e0 2024-04-20T21:44:35.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:35.554+0000 7f1461600700 1 -- 172.21.15.62:0/4085903817 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f145400f460 con 0x7f14641004e0 2024-04-20T21:44:35.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:35.554+0000 7f1461600700 1 -- 172.21.15.62:0/4085903817 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1454017630 con 0x7f14641004e0 2024-04-20T21:44:35.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:35.554+0000 7f146aa00700 1 -- 172.21.15.62:0/4085903817 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1464073e90 con 0x7f14641004e0 2024-04-20T21:44:35.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:35.554+0000 7f146aa00700 1 -- 172.21.15.62:0/4085903817 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14641a23a0 con 0x7f14641004e0 2024-04-20T21:44:35.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:35.555+0000 7f146aa00700 1 -- 172.21.15.62:0/4085903817 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14641087f0 con 0x7f14641004e0 2024-04-20T21:44:35.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:35.556+0000 7f1461600700 1 -- 172.21.15.62:0/4085903817 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f145400f5d0 con 0x7f14641004e0 2024-04-20T21:44:35.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:35.558+0000 7f1461600700 1 --2- 172.21.15.62:0/4085903817 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f144806ecc0 0x7f1448071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:44:35.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:35.558+0000 7f1461600700 1 -- 172.21.15.62:0/4085903817 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f1454090320 con 0x7f14641004e0 2024-04-20T21:44:35.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:35.558+0000 7f145b400700 1 --2- 172.21.15.62:0/4085903817 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f144806ecc0 0x7f1448071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:44:35.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:35.559+0000 7f145b400700 1 --2- 172.21.15.62:0/4085903817 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f144806ecc0 0x7f1448071170 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f144c005950 tx=0x7f144c0058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:44:35.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:35.563+0000 7f1461600700 1 -- 172.21.15.62:0/4085903817 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f14540590b0 con 0x7f14641004e0 2024-04-20T21:44:35.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:35.849+0000 7f146aa00700 1 -- 172.21.15.62:0/4085903817 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f146404f2a0 con 0x7f14641004e0 2024-04-20T21:44:35.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:35.849+0000 7f1461600700 1 -- 172.21.15.62:0/4085903817 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 25 v25) v1 ==== 76+0+1887 (secure 0 0 0) 0x7f145405c770 con 0x7f14641004e0 2024-04-20T21:44:35.849 INFO:teuthology.orchestra.run.smithi062.stdout:e25 2024-04-20T21:44:35.849 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T21:44:35.849 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T21:44:35.849 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T21:44:35.849 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:44:35.849 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T21:44:35.849 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T21:44:35.850 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 20 2024-04-20T21:44:35.850 INFO:teuthology.orchestra.run.smithi062.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-20T21:44:35.850 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T21:44:35.850 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T21:42:01.559061+0000 2024-04-20T21:44:35.850 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T21:44:35.850 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T21:44:35.850 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T21:44:35.850 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T21:44:35.850 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T21:44:35.850 INFO:teuthology.orchestra.run.smithi062.stdout:max_xattr_size 65536 2024-04-20T21:44:35.850 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T21:44:35.850 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T21:44:35.850 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 81 2024-04-20T21:44:35.850 INFO:teuthology.orchestra.run.smithi062.stdout:snap_rstat 1 2024-04-20T21:44:35.851 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T21:44:35.851 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 2 2024-04-20T21:44:35.851 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1 2024-04-20T21:44:35.851 INFO:teuthology.orchestra.run.smithi062.stdout:up {} 2024-04-20T21:44:35.851 INFO:teuthology.orchestra.run.smithi062.stdout:failed 0,1 2024-04-20T21:44:35.851 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T21:44:35.851 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T21:44:35.851 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T21:44:35.851 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T21:44:35.851 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data enabled 2024-04-20T21:44:35.851 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T21:44:35.851 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T21:44:35.851 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 1 2024-04-20T21:44:35.851 INFO:teuthology.orchestra.run.smithi062.stdout:qdb_cluster leader: 0 members: 2024-04-20T21:44:35.851 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:44:35.851 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:44:35.852 INFO:teuthology.orchestra.run.smithi062.stdout:Standby daemons: 2024-04-20T21:44:35.852 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:44:35.852 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ryqblb{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6826/4113916205,v1:172.21.15.62:6827/4113916205] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T21:44:35.852 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ozizwi{-1:34476} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6828/3801975165,v1:172.21.15.62:6829/3801975165] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T21:44:35.852 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.xbgspw{-1:44285} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6826/3996040526,v1:172.21.15.189:6827/3996040526] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:44:35.852 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.dvimer{-1:44289} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6824/1743111843,v1:172.21.15.189:6825/1743111843] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:44:35.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:35.852+0000 7f146aa00700 1 -- 172.21.15.62:0/4085903817 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f144806ecc0 msgr2=0x7f1448071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:44:35.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:35.852+0000 7f146aa00700 1 --2- 172.21.15.62:0/4085903817 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f144806ecc0 0x7f1448071170 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f144c005950 tx=0x7f144c0058e0 comp rx=0 tx=0).stop 2024-04-20T21:44:35.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:35.852+0000 7f146aa00700 1 -- 172.21.15.62:0/4085903817 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f14641004e0 msgr2=0x7f1464073050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:44:35.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:35.852+0000 7f146aa00700 1 --2- 172.21.15.62:0/4085903817 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f14641004e0 0x7f1464073050 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f1454006010 tx=0x7f145400be20 comp rx=0 tx=0).stop 2024-04-20T21:44:35.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:35.853+0000 7f146aa00700 1 -- 172.21.15.62:0/4085903817 shutdown_connections 2024-04-20T21:44:35.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:35.853+0000 7f146aa00700 1 --2- 172.21.15.62:0/4085903817 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f144806ecc0 0x7f1448071170 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:44:35.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:35.853+0000 7f146aa00700 1 --2- 172.21.15.62:0/4085903817 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f14641004e0 0x7f1464073050 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:44:35.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:35.853+0000 7f146aa00700 1 --2- 172.21.15.62:0/4085903817 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1464100e80 0x7f1464073590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:44:35.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:35.853+0000 7f146aa00700 1 -- 172.21.15.62:0/4085903817 >> 172.21.15.62:0/4085903817 conn(0x7f14640fbda0 msgr2=0x7f14640fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:44:35.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:35.853+0000 7f146aa00700 1 -- 172.21.15.62:0/4085903817 shutdown_connections 2024-04-20T21:44:35.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:35.853+0000 7f146aa00700 1 -- 172.21.15.62:0/4085903817 wait complete. 2024-04-20T21:44:35.854 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 25 2024-04-20T21:44:35.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:44:35 smithi189 ceph-mon[57022]: from='client.34568 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:44:35.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:44:35 smithi189 ceph-mon[57022]: pgmap v278: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:44:35.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:44:35 smithi189 ceph-mon[57022]: from='client.34572 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:44:35.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:44:35 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/4088373507' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:44:36.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:36.016+0000 7fab80600700 1 -- 172.21.15.62:0/2103931497 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fab78106b80 msgr2=0x7fab78106ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:44:36.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:36.016+0000 7fab80600700 1 --2- 172.21.15.62:0/2103931497 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fab78106b80 0x7fab78106ff0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fab64009910 tx=0x7fab64009c20 comp rx=0 tx=0).stop 2024-04-20T21:44:36.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:36.016+0000 7fab80600700 1 -- 172.21.15.62:0/2103931497 shutdown_connections 2024-04-20T21:44:36.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:36.016+0000 7fab80600700 1 --2- 172.21.15.62:0/2103931497 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fab78106b80 0x7fab78106ff0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:44:36.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:36.016+0000 7fab80600700 1 --2- 172.21.15.62:0/2103931497 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fab7810c8f0 0x7fab7810ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:44:36.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:36.016+0000 7fab80600700 1 -- 172.21.15.62:0/2103931497 >> 172.21.15.62:0/2103931497 conn(0x7fab78076020 msgr2=0x7fab78078430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:44:36.016 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:44:35 smithi062 ceph-mon[76322]: from='client.34568 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:44:36.016 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:44:35 smithi062 ceph-mon[76322]: pgmap v278: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:44:36.016 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:44:35 smithi062 ceph-mon[76322]: from='client.34572 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:44:36.016 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:44:35 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/4088373507' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:44:36.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:36.017+0000 7fab80600700 1 -- 172.21.15.62:0/2103931497 shutdown_connections 2024-04-20T21:44:36.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:36.017+0000 7fab80600700 1 -- 172.21.15.62:0/2103931497 wait complete. 2024-04-20T21:44:36.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:36.018+0000 7fab80600700 1 Processor -- start 2024-04-20T21:44:36.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:36.018+0000 7fab80600700 1 -- start start 2024-04-20T21:44:36.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:36.018+0000 7fab80600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fab78106b80 0x7fab7819c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:44:36.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:36.018+0000 7fab80600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fab7810c8f0 0x7fab7819c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:44:36.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:36.018+0000 7fab80600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab7819cfe0 con 0x7fab7810c8f0 2024-04-20T21:44:36.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:36.018+0000 7fab80600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab7819d120 con 0x7fab78106b80 2024-04-20T21:44:36.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:36.019+0000 7fab7d200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fab7810c8f0 0x7fab7819c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:44:36.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:36.019+0000 7fab7d200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fab7810c8f0 0x7fab7819c990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54682/0 (socket says 172.21.15.62:54682) 2024-04-20T21:44:36.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:36.019+0000 7fab7d200700 1 -- 172.21.15.62:0/286327552 learned_addr learned my addr 172.21.15.62:0/286327552 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:44:36.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:36.019+0000 7fab7dc00700 1 --2- 172.21.15.62:0/286327552 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fab78106b80 0x7fab7819c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:44:36.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:36.019+0000 7fab7d200700 1 -- 172.21.15.62:0/286327552 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fab78106b80 msgr2=0x7fab7819c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:44:36.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:36.019+0000 7fab7d200700 1 --2- 172.21.15.62:0/286327552 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fab78106b80 0x7fab7819c450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:44:36.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:36.019+0000 7fab7d200700 1 -- 172.21.15.62:0/286327552 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab640095f0 con 0x7fab7810c8f0 2024-04-20T21:44:36.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:36.019+0000 7fab7dc00700 1 --2- 172.21.15.62:0/286327552 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fab78106b80 0x7fab7819c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-20T21:44:36.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:36.020+0000 7fab7d200700 1 --2- 172.21.15.62:0/286327552 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fab7810c8f0 0x7fab7819c990 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fab6400b5c0 tx=0x7fab6400be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:44:36.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:36.020+0000 7fab76000700 1 -- 172.21.15.62:0/286327552 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fab6401d070 con 0x7fab7810c8f0 2024-04-20T21:44:36.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:36.020+0000 7fab76000700 1 -- 172.21.15.62:0/286327552 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fab6400f460 con 0x7fab7810c8f0 2024-04-20T21:44:36.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:36.020+0000 7fab80600700 1 -- 172.21.15.62:0/286327552 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab781a0f10 con 0x7fab7810c8f0 2024-04-20T21:44:36.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:36.020+0000 7fab76000700 1 -- 172.21.15.62:0/286327552 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fab64017610 con 0x7fab7810c8f0 2024-04-20T21:44:36.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:36.020+0000 7fab80600700 1 -- 172.21.15.62:0/286327552 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab781a1400 con 0x7fab7810c8f0 2024-04-20T21:44:36.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:36.021+0000 7fab80600700 1 -- 172.21.15.62:0/286327552 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab7804f2a0 con 0x7fab7810c8f0 2024-04-20T21:44:36.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:36.023+0000 7fab76000700 1 -- 172.21.15.62:0/286327552 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fab6400f5d0 con 0x7fab7810c8f0 2024-04-20T21:44:36.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:36.023+0000 7fab76000700 1 --2- 172.21.15.62:0/286327552 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fab6006ecc0 0x7fab60071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:44:36.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:36.024+0000 7fab76000700 1 -- 172.21.15.62:0/286327552 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fab64090470 con 0x7fab7810c8f0 2024-04-20T21:44:36.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:36.024+0000 7fab7dc00700 1 --2- 172.21.15.62:0/286327552 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fab6006ecc0 0x7fab60071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:44:36.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:36.025+0000 7fab7dc00700 1 --2- 172.21.15.62:0/286327552 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fab6006ecc0 0x7fab60071170 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fab6c005fd0 tx=0x7fab6c005e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:44:36.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:36.029+0000 7fab76000700 1 -- 172.21.15.62:0/286327552 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fab64059220 con 0x7fab7810c8f0 2024-04-20T21:44:36.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:36.284+0000 7fab80600700 1 -- 172.21.15.62:0/286327552 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fab781a17c0 con 0x7fab6006ecc0 2024-04-20T21:44:36.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:36.287+0000 7fab76000700 1 -- 172.21.15.62:0/286327552 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fab781a17c0 con 0x7fab6006ecc0 2024-04-20T21:44:36.286 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:44:36.286 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T21:44:36.286 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T21:44:36.286 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-20T21:44:36.286 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T21:44:36.286 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr", 2024-04-20T21:44:36.287 INFO:teuthology.orchestra.run.smithi062.stdout: "crash", 2024-04-20T21:44:36.287 INFO:teuthology.orchestra.run.smithi062.stdout: "mon", 2024-04-20T21:44:36.287 INFO:teuthology.orchestra.run.smithi062.stdout: "osd" 2024-04-20T21:44:36.287 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T21:44:36.287 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "14/23 daemons upgraded", 2024-04-20T21:44:36.287 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading mds daemons", 2024-04-20T21:44:36.287 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T21:44:36.287 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:44:36.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:36.289+0000 7fab80600700 1 -- 172.21.15.62:0/286327552 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fab6006ecc0 msgr2=0x7fab60071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:44:36.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:36.289+0000 7fab80600700 1 --2- 172.21.15.62:0/286327552 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fab6006ecc0 0x7fab60071170 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fab6c005fd0 tx=0x7fab6c005e30 comp rx=0 tx=0).stop 2024-04-20T21:44:36.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:36.289+0000 7fab80600700 1 -- 172.21.15.62:0/286327552 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fab7810c8f0 msgr2=0x7fab7819c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:44:36.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:36.289+0000 7fab80600700 1 --2- 172.21.15.62:0/286327552 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fab7810c8f0 0x7fab7819c990 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fab6400b5c0 tx=0x7fab6400be00 comp rx=0 tx=0).stop 2024-04-20T21:44:36.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:36.289+0000 7fab80600700 1 -- 172.21.15.62:0/286327552 shutdown_connections 2024-04-20T21:44:36.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:36.289+0000 7fab80600700 1 --2- 172.21.15.62:0/286327552 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fab6006ecc0 0x7fab60071170 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:44:36.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:36.289+0000 7fab80600700 1 --2- 172.21.15.62:0/286327552 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fab78106b80 0x7fab7819c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:44:36.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:36.289+0000 7fab80600700 1 --2- 172.21.15.62:0/286327552 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fab7810c8f0 0x7fab7819c990 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:44:36.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:36.289+0000 7fab80600700 1 -- 172.21.15.62:0/286327552 >> 172.21.15.62:0/286327552 conn(0x7fab78076020 msgr2=0x7fab78077a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:44:36.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:36.290+0000 7fab80600700 1 -- 172.21.15.62:0/286327552 shutdown_connections 2024-04-20T21:44:36.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:36.290+0000 7fab80600700 1 -- 172.21.15.62:0/286327552 wait complete. 2024-04-20T21:44:36.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:36.447+0000 7fb905000700 1 -- 172.21.15.62:0/2344084763 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb900073a00 msgr2=0x7fb90010c580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:44:36.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:36.447+0000 7fb905000700 1 --2- 172.21.15.62:0/2344084763 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb900073a00 0x7fb90010c580 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fb8e8009910 tx=0x7fb8e8009c20 comp rx=0 tx=0).stop 2024-04-20T21:44:36.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:36.447+0000 7fb905000700 1 -- 172.21.15.62:0/2344084763 shutdown_connections 2024-04-20T21:44:36.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:36.447+0000 7fb905000700 1 --2- 172.21.15.62:0/2344084763 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb900073a00 0x7fb90010c580 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:44:36.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:36.447+0000 7fb905000700 1 --2- 172.21.15.62:0/2344084763 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb9000730f0 0x7fb9000734c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:44:36.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:36.447+0000 7fb905000700 1 -- 172.21.15.62:0/2344084763 >> 172.21.15.62:0/2344084763 conn(0x7fb9000fbda0 msgr2=0x7fb9000fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:44:36.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:36.447+0000 7fb905000700 1 -- 172.21.15.62:0/2344084763 shutdown_connections 2024-04-20T21:44:36.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:36.447+0000 7fb905000700 1 -- 172.21.15.62:0/2344084763 wait complete. 2024-04-20T21:44:36.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:36.448+0000 7fb905000700 1 Processor -- start 2024-04-20T21:44:36.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:36.448+0000 7fb905000700 1 -- start start 2024-04-20T21:44:36.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:36.449+0000 7fb905000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb9000730f0 0x7fb900198210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:44:36.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:36.449+0000 7fb905000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb900073a00 0x7fb900198880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:44:36.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:36.449+0000 7fb905000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb90019cbc0 con 0x7fb900073a00 2024-04-20T21:44:36.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:36.449+0000 7fb905000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb90019cd30 con 0x7fb9000730f0 2024-04-20T21:44:36.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:36.449+0000 7fb8f7e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb900073a00 0x7fb900198880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:44:36.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:36.449+0000 7fb8f7e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb900073a00 0x7fb900198880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54692/0 (socket says 172.21.15.62:54692) 2024-04-20T21:44:36.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:36.449+0000 7fb8f7e00700 1 -- 172.21.15.62:0/2716573838 learned_addr learned my addr 172.21.15.62:0/2716573838 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:44:36.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:36.449+0000 7fb8fde00700 1 --2- 172.21.15.62:0/2716573838 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb9000730f0 0x7fb900198210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:44:36.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:36.449+0000 7fb8f7e00700 1 -- 172.21.15.62:0/2716573838 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb9000730f0 msgr2=0x7fb900198210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:44:36.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:36.449+0000 7fb8f7e00700 1 --2- 172.21.15.62:0/2716573838 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb9000730f0 0x7fb900198210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:44:36.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:36.450+0000 7fb8f7e00700 1 -- 172.21.15.62:0/2716573838 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8e80095f0 con 0x7fb900073a00 2024-04-20T21:44:36.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:36.450+0000 7fb8fde00700 1 --2- 172.21.15.62:0/2716573838 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb9000730f0 0x7fb900198210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T21:44:36.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:36.450+0000 7fb8f7e00700 1 --2- 172.21.15.62:0/2716573838 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb900073a00 0x7fb900198880 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fb8e800b5c0 tx=0x7fb8e800be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:44:36.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:36.450+0000 7fb8f6a00700 1 -- 172.21.15.62:0/2716573838 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb8e801d070 con 0x7fb900073a00 2024-04-20T21:44:36.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:36.450+0000 7fb8f6a00700 1 -- 172.21.15.62:0/2716573838 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb8e800f460 con 0x7fb900073a00 2024-04-20T21:44:36.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:36.450+0000 7fb905000700 1 -- 172.21.15.62:0/2716573838 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb90019d010 con 0x7fb900073a00 2024-04-20T21:44:36.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:36.450+0000 7fb8f6a00700 1 -- 172.21.15.62:0/2716573838 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb8e8017630 con 0x7fb900073a00 2024-04-20T21:44:36.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:36.450+0000 7fb905000700 1 -- 172.21.15.62:0/2716573838 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb90019d560 con 0x7fb900073a00 2024-04-20T21:44:36.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:36.451+0000 7fb905000700 1 -- 172.21.15.62:0/2716573838 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb900109c80 con 0x7fb900073a00 2024-04-20T21:44:36.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:36.452+0000 7fb8f6a00700 1 -- 172.21.15.62:0/2716573838 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fb8e800f5d0 con 0x7fb900073a00 2024-04-20T21:44:36.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:36.454+0000 7fb8f6a00700 1 --2- 172.21.15.62:0/2716573838 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb8e406ecc0 0x7fb8e4071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:44:36.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:36.454+0000 7fb8f6a00700 1 -- 172.21.15.62:0/2716573838 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fb8e80903c0 con 0x7fb900073a00 2024-04-20T21:44:36.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:36.454+0000 7fb8fde00700 1 --2- 172.21.15.62:0/2716573838 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb8e406ecc0 0x7fb8e4071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:44:36.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:36.455+0000 7fb8fde00700 1 --2- 172.21.15.62:0/2716573838 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb8e406ecc0 0x7fb8e4071170 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fb8f0007eb0 tx=0x7fb8f00073c0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:44:36.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:36.459+0000 7fb8f6a00700 1 -- 172.21.15.62:0/2716573838 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fb8e8059150 con 0x7fb900073a00 2024-04-20T21:44:36.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:36.774+0000 7fb905000700 1 -- 172.21.15.62:0/2716573838 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fb900199570 con 0x7fb900073a00 2024-04-20T21:44:36.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:36.775+0000 7fb8f6a00700 1 -- 172.21.15.62:0/2716573838 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7fb8e805c810 con 0x7fb900073a00 2024-04-20T21:44:36.774 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-20T21:44:36.774 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-20T21:44:36.774 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is degraded 2024-04-20T21:44:36.775 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T21:44:36.775 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T21:44:36.775 INFO:teuthology.orchestra.run.smithi062.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-20T21:44:36.775 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-20T21:44:36.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:36.777+0000 7fb905000700 1 -- 172.21.15.62:0/2716573838 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb8e406ecc0 msgr2=0x7fb8e4071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:44:36.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:36.777+0000 7fb905000700 1 --2- 172.21.15.62:0/2716573838 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb8e406ecc0 0x7fb8e4071170 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fb8f0007eb0 tx=0x7fb8f00073c0 comp rx=0 tx=0).stop 2024-04-20T21:44:36.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:36.778+0000 7fb905000700 1 -- 172.21.15.62:0/2716573838 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb900073a00 msgr2=0x7fb900198880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:44:36.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:36.778+0000 7fb905000700 1 --2- 172.21.15.62:0/2716573838 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb900073a00 0x7fb900198880 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fb8e800b5c0 tx=0x7fb8e800be00 comp rx=0 tx=0).stop 2024-04-20T21:44:36.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:36.778+0000 7fb905000700 1 -- 172.21.15.62:0/2716573838 shutdown_connections 2024-04-20T21:44:36.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:36.778+0000 7fb905000700 1 --2- 172.21.15.62:0/2716573838 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb8e406ecc0 0x7fb8e4071170 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:44:36.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:36.778+0000 7fb905000700 1 --2- 172.21.15.62:0/2716573838 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb9000730f0 0x7fb900198210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:44:36.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:36.778+0000 7fb905000700 1 --2- 172.21.15.62:0/2716573838 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb900073a00 0x7fb900198880 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:44:36.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:36.778+0000 7fb905000700 1 -- 172.21.15.62:0/2716573838 >> 172.21.15.62:0/2716573838 conn(0x7fb9000fbda0 msgr2=0x7fb900106dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:44:36.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:36.778+0000 7fb905000700 1 -- 172.21.15.62:0/2716573838 shutdown_connections 2024-04-20T21:44:36.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:44:36.778+0000 7fb905000700 1 -- 172.21.15.62:0/2716573838 wait complete. 2024-04-20T21:44:36.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:44:36 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/4085903817' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T21:44:37.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:44:36 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/4085903817' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T21:44:38.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:44:37 smithi062 ceph-mon[76322]: from='client.34584 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:44:38.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:44:37 smithi062 ceph-mon[76322]: pgmap v279: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:44:38.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:44:37 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/2716573838' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T21:44:38.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:44:37 smithi189 ceph-mon[57022]: from='client.34584 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:44:38.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:44:37 smithi189 ceph-mon[57022]: pgmap v279: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:44:38.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:44:37 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/2716573838' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T21:44:40.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:44:39 smithi062 ceph-mon[76322]: pgmap v280: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:44:40.236 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:44:39 smithi189 ceph-mon[57022]: pgmap v280: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:44:42.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:44:41 smithi062 ceph-mon[76322]: pgmap v281: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:44:42.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:44:41 smithi189 ceph-mon[57022]: pgmap v281: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:44:44.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:44:43 smithi062 ceph-mon[76322]: pgmap v282: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:44:44.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:44:43 smithi189 ceph-mon[57022]: pgmap v282: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:44:46.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:44:45 smithi062 ceph-mon[76322]: pgmap v283: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:44:46.214 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:44:45 smithi189 ceph-mon[57022]: pgmap v283: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:44:48.143 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:44:47 smithi062 ceph-mon[76322]: pgmap v284: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:44:48.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:44:47 smithi189 ceph-mon[57022]: pgmap v284: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:44:50.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:44:49 smithi189 ceph-mon[57022]: pgmap v285: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:44:50.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:44:49 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:44:50.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:44:49 smithi062 ceph-mon[76322]: pgmap v285: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:44:50.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:44:49 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:44:52.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:44:51 smithi189 ceph-mon[57022]: pgmap v286: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:44:52.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:44:51 smithi062 ceph-mon[76322]: pgmap v286: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:44:54.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:44:53 smithi189 ceph-mon[57022]: pgmap v287: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:44:54.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:44:53 smithi062 ceph-mon[76322]: pgmap v287: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:44:56.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:44:55 smithi189 ceph-mon[57022]: pgmap v288: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:44:56.284 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:44:55 smithi062 ceph-mon[76322]: pgmap v288: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:44:58.087 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:44:57 smithi062 ceph-mon[76322]: pgmap v289: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:44:58.236 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:44:57 smithi189 ceph-mon[57022]: pgmap v289: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:45:00.236 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:44:59 smithi189 ceph-mon[57022]: pgmap v290: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:45:00.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:44:59 smithi062 ceph-mon[76322]: pgmap v290: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:45:01.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:45:01 smithi189 ceph-mon[57022]: pgmap v291: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:45:02.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:45:01 smithi062 ceph-mon[76322]: pgmap v291: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:45:04.138 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:45:03 smithi062 ceph-mon[76322]: pgmap v292: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:45:04.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:45:03 smithi189 ceph-mon[57022]: pgmap v292: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:45:05.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:45:04 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:45:05.284 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:45:04 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:45:06.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:45:05 smithi189 ceph-mon[57022]: pgmap v293: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:45:06.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:45:05 smithi062 ceph-mon[76322]: pgmap v293: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:45:06.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:06.958+0000 7f408e200700 1 -- 172.21.15.62:0/2067208820 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4088106320 msgr2=0x7f40881066f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:45:06.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:06.958+0000 7f408e200700 1 --2- 172.21.15.62:0/2067208820 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4088106320 0x7f40881066f0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f407800b0a0 tx=0x7f407800b3b0 comp rx=0 tx=0).stop 2024-04-20T21:45:06.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:06.959+0000 7f408e200700 1 -- 172.21.15.62:0/2067208820 shutdown_connections 2024-04-20T21:45:06.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:06.959+0000 7f408e200700 1 --2- 172.21.15.62:0/2067208820 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4088068540 0x7f40880689b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:45:06.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:06.959+0000 7f408e200700 1 --2- 172.21.15.62:0/2067208820 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4088106320 0x7f40881066f0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:45:06.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:06.959+0000 7f408e200700 1 -- 172.21.15.62:0/2067208820 >> 172.21.15.62:0/2067208820 conn(0x7f4088075130 msgr2=0x7f4088075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:45:06.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:06.959+0000 7f408e200700 1 -- 172.21.15.62:0/2067208820 shutdown_connections 2024-04-20T21:45:06.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:06.959+0000 7f408e200700 1 -- 172.21.15.62:0/2067208820 wait complete. 2024-04-20T21:45:06.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:06.960+0000 7f408e200700 1 Processor -- start 2024-04-20T21:45:06.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:06.960+0000 7f408e200700 1 -- start start 2024-04-20T21:45:06.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:06.960+0000 7f408e200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4088068540 0x7f4088198160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:45:06.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:06.960+0000 7f408e200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4088106320 0x7f40881986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:45:06.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:06.960+0000 7f408e200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4088198d80 con 0x7f4088068540 2024-04-20T21:45:06.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:06.960+0000 7f408e200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f408819cb10 con 0x7f4088106320 2024-04-20T21:45:06.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:06.961+0000 7f4087400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4088068540 0x7f4088198160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:45:06.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:06.961+0000 7f4087400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4088068540 0x7f4088198160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52602/0 (socket says 172.21.15.62:52602) 2024-04-20T21:45:06.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:06.961+0000 7f4087400700 1 -- 172.21.15.62:0/4132749018 learned_addr learned my addr 172.21.15.62:0/4132749018 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:45:06.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:06.961+0000 7f407ea00700 1 --2- 172.21.15.62:0/4132749018 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4088106320 0x7f40881986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:45:06.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:06.961+0000 7f4087400700 1 -- 172.21.15.62:0/4132749018 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4088106320 msgr2=0x7f40881986a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:45:06.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:06.961+0000 7f4087400700 1 --2- 172.21.15.62:0/4132749018 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4088106320 0x7f40881986a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:45:06.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:06.961+0000 7f4087400700 1 -- 172.21.15.62:0/4132749018 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4078009d00 con 0x7f4088068540 2024-04-20T21:45:06.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:06.961+0000 7f407ea00700 1 --2- 172.21.15.62:0/4132749018 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4088106320 0x7f40881986a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T21:45:06.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:06.962+0000 7f4087400700 1 --2- 172.21.15.62:0/4132749018 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4088068540 0x7f4088198160 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f407800b070 tx=0x7f4078003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:45:06.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:06.962+0000 7f4084c00700 1 -- 172.21.15.62:0/4132749018 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f407800e040 con 0x7f4088068540 2024-04-20T21:45:06.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:06.962+0000 7f4084c00700 1 -- 172.21.15.62:0/4132749018 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4078007c00 con 0x7f4088068540 2024-04-20T21:45:06.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:06.962+0000 7f4084c00700 1 -- 172.21.15.62:0/4132749018 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f407801d740 con 0x7f4088068540 2024-04-20T21:45:06.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:06.962+0000 7f408e200700 1 -- 172.21.15.62:0/4132749018 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f408819cd90 con 0x7f4088068540 2024-04-20T21:45:06.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:06.962+0000 7f408e200700 1 -- 172.21.15.62:0/4132749018 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f408819d280 con 0x7f4088068540 2024-04-20T21:45:06.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:06.963+0000 7f408e200700 1 -- 172.21.15.62:0/4132749018 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f408804f2a0 con 0x7f4088068540 2024-04-20T21:45:06.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:06.964+0000 7f4084c00700 1 -- 172.21.15.62:0/4132749018 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f4078004040 con 0x7f4088068540 2024-04-20T21:45:06.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:06.965+0000 7f4084c00700 1 --2- 172.21.15.62:0/4132749018 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f406c06ecc0 0x7f406c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:45:06.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:06.965+0000 7f4084c00700 1 -- 172.21.15.62:0/4132749018 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f4078090790 con 0x7f4088068540 2024-04-20T21:45:06.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:06.966+0000 7f407ea00700 1 --2- 172.21.15.62:0/4132749018 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f406c06ecc0 0x7f406c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:45:06.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:06.967+0000 7f407ea00700 1 --2- 172.21.15.62:0/4132749018 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f406c06ecc0 0x7f406c071170 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f4070005fd0 tx=0x7f4070005dc0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:45:06.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:06.971+0000 7f4084c00700 1 -- 172.21.15.62:0/4132749018 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f40780594a0 con 0x7f4088068540 2024-04-20T21:45:07.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:07.227+0000 7f408e200700 1 -- 172.21.15.62:0/4132749018 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f408819d610 con 0x7f406c06ecc0 2024-04-20T21:45:07.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:07.231+0000 7f4084c00700 1 -- 172.21.15.62:0/4132749018 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f408819d610 con 0x7f406c06ecc0 2024-04-20T21:45:07.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:07.234+0000 7f408e200700 1 -- 172.21.15.62:0/4132749018 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f406c06ecc0 msgr2=0x7f406c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:45:07.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:07.234+0000 7f408e200700 1 --2- 172.21.15.62:0/4132749018 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f406c06ecc0 0x7f406c071170 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f4070005fd0 tx=0x7f4070005dc0 comp rx=0 tx=0).stop 2024-04-20T21:45:07.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:07.234+0000 7f408e200700 1 -- 172.21.15.62:0/4132749018 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4088068540 msgr2=0x7f4088198160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:45:07.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:07.234+0000 7f408e200700 1 --2- 172.21.15.62:0/4132749018 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4088068540 0x7f4088198160 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f407800b070 tx=0x7f4078003ce0 comp rx=0 tx=0).stop 2024-04-20T21:45:07.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:07.234+0000 7f408e200700 1 -- 172.21.15.62:0/4132749018 shutdown_connections 2024-04-20T21:45:07.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:07.234+0000 7f408e200700 1 --2- 172.21.15.62:0/4132749018 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f406c06ecc0 0x7f406c071170 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:45:07.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:07.234+0000 7f408e200700 1 --2- 172.21.15.62:0/4132749018 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4088068540 0x7f4088198160 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:45:07.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:07.234+0000 7f408e200700 1 --2- 172.21.15.62:0/4132749018 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4088106320 0x7f40881986a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:45:07.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:07.234+0000 7f408e200700 1 -- 172.21.15.62:0/4132749018 >> 172.21.15.62:0/4132749018 conn(0x7f4088075130 msgr2=0x7f40880fea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:45:07.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:07.234+0000 7f408e200700 1 -- 172.21.15.62:0/4132749018 shutdown_connections 2024-04-20T21:45:07.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:07.234+0000 7f408e200700 1 -- 172.21.15.62:0/4132749018 wait complete. 2024-04-20T21:45:07.250 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T21:45:07.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:07.403+0000 7f4e2fe00700 1 -- 172.21.15.62:0/2891403786 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4e301002d0 msgr2=0x7f4e301006a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:45:07.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:07.403+0000 7f4e2fe00700 1 --2- 172.21.15.62:0/2891403786 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4e301002d0 0x7f4e301006a0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f4e20009910 tx=0x7f4e20009c20 comp rx=0 tx=0).stop 2024-04-20T21:45:07.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:07.403+0000 7f4e2fe00700 1 -- 172.21.15.62:0/2891403786 shutdown_connections 2024-04-20T21:45:07.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:07.403+0000 7f4e2fe00700 1 --2- 172.21.15.62:0/2891403786 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e30100c70 0x7f4e30104c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:45:07.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:07.403+0000 7f4e2fe00700 1 --2- 172.21.15.62:0/2891403786 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4e301002d0 0x7f4e301006a0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:45:07.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:07.403+0000 7f4e2fe00700 1 -- 172.21.15.62:0/2891403786 >> 172.21.15.62:0/2891403786 conn(0x7f4e300fbbb0 msgr2=0x7f4e300fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:45:07.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:07.403+0000 7f4e2fe00700 1 -- 172.21.15.62:0/2891403786 shutdown_connections 2024-04-20T21:45:07.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:07.403+0000 7f4e2fe00700 1 -- 172.21.15.62:0/2891403786 wait complete. 2024-04-20T21:45:07.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:07.404+0000 7f4e2fe00700 1 Processor -- start 2024-04-20T21:45:07.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:07.404+0000 7f4e2fe00700 1 -- start start 2024-04-20T21:45:07.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:07.405+0000 7f4e2fe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4e301002d0 0x7f4e30197ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:45:07.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:07.405+0000 7f4e2fe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e30100c70 0x7f4e30198400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:45:07.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:07.405+0000 7f4e2fe00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e30198ae0 con 0x7f4e301002d0 2024-04-20T21:45:07.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:07.405+0000 7f4e2fe00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e3019c870 con 0x7f4e30100c70 2024-04-20T21:45:07.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:07.405+0000 7f4e2ea00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4e301002d0 0x7f4e30197ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:45:07.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:07.405+0000 7f4e2ea00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4e301002d0 0x7f4e30197ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52622/0 (socket says 172.21.15.62:52622) 2024-04-20T21:45:07.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:07.405+0000 7f4e2ea00700 1 -- 172.21.15.62:0/3835472655 learned_addr learned my addr 172.21.15.62:0/3835472655 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:45:07.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:07.405+0000 7f4e27e00700 1 --2- 172.21.15.62:0/3835472655 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e30100c70 0x7f4e30198400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:45:07.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:07.405+0000 7f4e2ea00700 1 -- 172.21.15.62:0/3835472655 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e30100c70 msgr2=0x7f4e30198400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:45:07.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:07.405+0000 7f4e2ea00700 1 --2- 172.21.15.62:0/3835472655 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e30100c70 0x7f4e30198400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:45:07.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:07.406+0000 7f4e2ea00700 1 -- 172.21.15.62:0/3835472655 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e200095f0 con 0x7f4e301002d0 2024-04-20T21:45:07.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:07.406+0000 7f4e2ea00700 1 --2- 172.21.15.62:0/3835472655 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4e301002d0 0x7f4e30197ec0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f4e2000b5c0 tx=0x7f4e2000be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:45:07.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:07.406+0000 7f4e34a00700 1 -- 172.21.15.62:0/3835472655 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4e2001d070 con 0x7f4e301002d0 2024-04-20T21:45:07.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:07.406+0000 7f4e34a00700 1 -- 172.21.15.62:0/3835472655 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4e2000f460 con 0x7f4e301002d0 2024-04-20T21:45:07.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:07.406+0000 7f4e34a00700 1 -- 172.21.15.62:0/3835472655 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4e20017610 con 0x7f4e301002d0 2024-04-20T21:45:07.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:07.406+0000 7f4e2fe00700 1 -- 172.21.15.62:0/3835472655 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e3019caf0 con 0x7f4e301002d0 2024-04-20T21:45:07.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:07.406+0000 7f4e2fe00700 1 -- 172.21.15.62:0/3835472655 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e3019cfe0 con 0x7f4e301002d0 2024-04-20T21:45:07.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:07.408+0000 7f4e2fe00700 1 -- 172.21.15.62:0/3835472655 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e301043c0 con 0x7f4e301002d0 2024-04-20T21:45:07.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:07.409+0000 7f4e34a00700 1 -- 172.21.15.62:0/3835472655 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f4e2000f5d0 con 0x7f4e301002d0 2024-04-20T21:45:07.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:07.410+0000 7f4e34a00700 1 --2- 172.21.15.62:0/3835472655 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f4e1406ecc0 0x7f4e14071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:45:07.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:07.410+0000 7f4e34a00700 1 -- 172.21.15.62:0/3835472655 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f4e20090520 con 0x7f4e301002d0 2024-04-20T21:45:07.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:07.411+0000 7f4e27e00700 1 --2- 172.21.15.62:0/3835472655 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f4e1406ecc0 0x7f4e14071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:45:07.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:07.411+0000 7f4e27e00700 1 --2- 172.21.15.62:0/3835472655 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f4e1406ecc0 0x7f4e14071170 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f4e18005fd0 tx=0x7f4e1800a560 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:45:07.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:07.416+0000 7f4e34a00700 1 -- 172.21.15.62:0/3835472655 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f4e20059230 con 0x7f4e301002d0 2024-04-20T21:45:07.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:07.674+0000 7f4e2fe00700 1 -- 172.21.15.62:0/3835472655 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4e3019d2c0 con 0x7f4e1406ecc0 2024-04-20T21:45:07.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:07.676+0000 7f4e34a00700 1 -- 172.21.15.62:0/3835472655 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f4e3019d2c0 con 0x7f4e1406ecc0 2024-04-20T21:45:07.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:07.679+0000 7f4e2fe00700 1 -- 172.21.15.62:0/3835472655 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f4e1406ecc0 msgr2=0x7f4e14071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:45:07.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:07.679+0000 7f4e2fe00700 1 --2- 172.21.15.62:0/3835472655 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f4e1406ecc0 0x7f4e14071170 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f4e18005fd0 tx=0x7f4e1800a560 comp rx=0 tx=0).stop 2024-04-20T21:45:07.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:07.679+0000 7f4e2fe00700 1 -- 172.21.15.62:0/3835472655 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4e301002d0 msgr2=0x7f4e30197ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:45:07.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:07.679+0000 7f4e2fe00700 1 --2- 172.21.15.62:0/3835472655 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4e301002d0 0x7f4e30197ec0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f4e2000b5c0 tx=0x7f4e2000be00 comp rx=0 tx=0).stop 2024-04-20T21:45:07.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:07.679+0000 7f4e2fe00700 1 -- 172.21.15.62:0/3835472655 shutdown_connections 2024-04-20T21:45:07.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:07.679+0000 7f4e2fe00700 1 --2- 172.21.15.62:0/3835472655 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f4e1406ecc0 0x7f4e14071170 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:45:07.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:07.679+0000 7f4e2fe00700 1 --2- 172.21.15.62:0/3835472655 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4e301002d0 0x7f4e30197ec0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:45:07.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:07.679+0000 7f4e2fe00700 1 --2- 172.21.15.62:0/3835472655 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e30100c70 0x7f4e30198400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:45:07.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:07.679+0000 7f4e2fe00700 1 -- 172.21.15.62:0/3835472655 >> 172.21.15.62:0/3835472655 conn(0x7f4e300fbbb0 msgr2=0x7f4e300fdf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:45:07.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:07.679+0000 7f4e2fe00700 1 -- 172.21.15.62:0/3835472655 shutdown_connections 2024-04-20T21:45:07.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:07.679+0000 7f4e2fe00700 1 -- 172.21.15.62:0/3835472655 wait complete. 2024-04-20T21:45:07.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:07.841+0000 7f55d7e00700 1 -- 172.21.15.62:0/3199950966 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55d81024d0 msgr2=0x7f55d81028a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:45:07.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:07.841+0000 7f55d7e00700 1 --2- 172.21.15.62:0/3199950966 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55d81024d0 0x7f55d81028a0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f55c8009910 tx=0x7f55c8009c20 comp rx=0 tx=0).stop 2024-04-20T21:45:07.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:07.841+0000 7f55d7e00700 1 -- 172.21.15.62:0/3199950966 shutdown_connections 2024-04-20T21:45:07.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:07.841+0000 7f55d7e00700 1 --2- 172.21.15.62:0/3199950966 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f55d8102e70 0x7f55d8106e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:45:07.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:07.841+0000 7f55d7e00700 1 --2- 172.21.15.62:0/3199950966 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55d81024d0 0x7f55d81028a0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:45:07.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:07.842+0000 7f55d7e00700 1 -- 172.21.15.62:0/3199950966 >> 172.21.15.62:0/3199950966 conn(0x7f55d80fdd90 msgr2=0x7f55d81001a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:45:07.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:07.842+0000 7f55d7e00700 1 -- 172.21.15.62:0/3199950966 shutdown_connections 2024-04-20T21:45:07.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:07.842+0000 7f55d7e00700 1 -- 172.21.15.62:0/3199950966 wait complete. 2024-04-20T21:45:07.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:07.843+0000 7f55d7e00700 1 Processor -- start 2024-04-20T21:45:07.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:07.843+0000 7f55d7e00700 1 -- start start 2024-04-20T21:45:07.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:07.843+0000 7f55d7e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55d81024d0 0x7f55d8197e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:45:07.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:07.843+0000 7f55d7e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f55d8102e70 0x7f55d81983a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:45:07.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:07.843+0000 7f55d7e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55d8198a80 con 0x7f55d81024d0 2024-04-20T21:45:07.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:07.843+0000 7f55d7e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55d819c810 con 0x7f55d8102e70 2024-04-20T21:45:07.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:07.844+0000 7f55d6a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55d81024d0 0x7f55d8197e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:45:07.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:07.844+0000 7f55d6a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55d81024d0 0x7f55d8197e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52640/0 (socket says 172.21.15.62:52640) 2024-04-20T21:45:07.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:07.844+0000 7f55d6a00700 1 -- 172.21.15.62:0/2187670967 learned_addr learned my addr 172.21.15.62:0/2187670967 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:45:07.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:07.844+0000 7f55ce000700 1 --2- 172.21.15.62:0/2187670967 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f55d8102e70 0x7f55d81983a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:45:07.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:07.844+0000 7f55d6a00700 1 -- 172.21.15.62:0/2187670967 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f55d8102e70 msgr2=0x7f55d81983a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:45:07.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:07.844+0000 7f55d6a00700 1 --2- 172.21.15.62:0/2187670967 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f55d8102e70 0x7f55d81983a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:45:07.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:07.844+0000 7f55d6a00700 1 -- 172.21.15.62:0/2187670967 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55c80095f0 con 0x7f55d81024d0 2024-04-20T21:45:07.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:07.844+0000 7f55ce000700 1 --2- 172.21.15.62:0/2187670967 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f55d8102e70 0x7f55d81983a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T21:45:07.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:07.844+0000 7f55d6a00700 1 --2- 172.21.15.62:0/2187670967 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55d81024d0 0x7f55d8197e60 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f55c800b5c0 tx=0x7f55c800be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:45:07.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:07.845+0000 7f55cfe00700 1 -- 172.21.15.62:0/2187670967 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f55c801d070 con 0x7f55d81024d0 2024-04-20T21:45:07.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:07.845+0000 7f55cfe00700 1 -- 172.21.15.62:0/2187670967 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f55c800f460 con 0x7f55d81024d0 2024-04-20T21:45:07.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:07.845+0000 7f55cfe00700 1 -- 172.21.15.62:0/2187670967 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f55c8017610 con 0x7f55d81024d0 2024-04-20T21:45:07.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:07.845+0000 7f55d7e00700 1 -- 172.21.15.62:0/2187670967 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55d819ca90 con 0x7f55d81024d0 2024-04-20T21:45:07.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:07.845+0000 7f55d7e00700 1 -- 172.21.15.62:0/2187670967 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55d819cf80 con 0x7f55d81024d0 2024-04-20T21:45:07.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:07.846+0000 7f55d7e00700 1 -- 172.21.15.62:0/2187670967 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55d810a7e0 con 0x7f55d81024d0 2024-04-20T21:45:07.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:07.848+0000 7f55cfe00700 1 -- 172.21.15.62:0/2187670967 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f55c800f5d0 con 0x7f55d81024d0 2024-04-20T21:45:07.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:07.848+0000 7f55cfe00700 1 --2- 172.21.15.62:0/2187670967 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f55bc06ecc0 0x7f55bc071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:45:07.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:07.849+0000 7f55cfe00700 1 -- 172.21.15.62:0/2187670967 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f55c8090440 con 0x7f55d81024d0 2024-04-20T21:45:07.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:07.849+0000 7f55ce000700 1 --2- 172.21.15.62:0/2187670967 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f55bc06ecc0 0x7f55bc071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:45:07.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:07.849+0000 7f55ce000700 1 --2- 172.21.15.62:0/2187670967 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f55bc06ecc0 0x7f55bc071170 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f55c0005fd0 tx=0x7f55c0005dc0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:45:07.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:07.854+0000 7f55cfe00700 1 -- 172.21.15.62:0/2187670967 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f55c80591d0 con 0x7f55d81024d0 2024-04-20T21:45:08.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:45:07 smithi062 ceph-mon[76322]: pgmap v294: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:45:08.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:08.104+0000 7f55d7e00700 1 -- 172.21.15.62:0/2187670967 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f55d819d320 con 0x7f55bc06ecc0 2024-04-20T21:45:08.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:08.136+0000 7f55cfe00700 1 -- 172.21.15.62:0/2187670967 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f55d819d320 con 0x7f55bc06ecc0 2024-04-20T21:45:08.136 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T21:45:08.136 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (10m) 2m ago 24m 14.2M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T21:45:08.136 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (25m) 2m ago 25m 9512k - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T21:45:08.136 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (23m) 3m ago 23m 8942k - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T21:45:08.136 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (7m) 2m ago 25m 6660k - 19.0.0-3133-ga8c84f59 76bc6b91b726 2825d5ba11c6 2024-04-20T21:45:08.136 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (6m) 3m ago 23m 6665k - 19.0.0-3133-ga8c84f59 76bc6b91b726 20f746fa1bba 2024-04-20T21:45:08.136 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (10m) 2m ago 24m 74.2M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T21:45:08.136 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (2m) 2m ago 17m 11.2M - 19.0.0-3133-ga8c84f59 76bc6b91b726 de80bc90a683 2024-04-20T21:45:08.136 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (2m) 2m ago 17m 11.9M - 19.0.0-3133-ga8c84f59 76bc6b91b726 cd793b7ee98e 2024-04-20T21:45:08.136 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (17m) 3m ago 17m 117M - 18.2.0 dc2bc1663786 fb2a8a223db8 2024-04-20T21:45:08.136 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (17m) 3m ago 17m 55.6M - 18.2.0 dc2bc1663786 f2eec2ee8951 2024-04-20T21:45:08.136 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (13m) 2m ago 26m 511M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T21:45:08.137 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (12m) 3m ago 23m 452M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T21:45:08.137 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (8m) 2m ago 26m 55.9M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 019eb1479485 2024-04-20T21:45:08.137 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (8m) 3m ago 23m 39.8M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 55aac560aab6 2024-04-20T21:45:08.137 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (12m) 2m ago 25m 7952k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T21:45:08.137 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (12m) 3m ago 23m 8304k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T21:45:08.137 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (6m) 2m ago 22m 88.0M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 ee2bb901f26e 2024-04-20T21:45:08.137 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (5m) 2m ago 21m 92.3M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 9075152d4162 2024-04-20T21:45:08.137 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (5m) 2m ago 21m 74.4M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 2d4f3e38dc0e 2024-04-20T21:45:08.137 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (4m) 3m ago 20m 87.6M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 8f97b17ae0eb 2024-04-20T21:45:08.137 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (4m) 3m ago 20m 90.7M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 653d8079967d 2024-04-20T21:45:08.137 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (3m) 3m ago 19m 11.5M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 596dded14c12 2024-04-20T21:45:08.137 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (11m) 2m ago 24m 74.0M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T21:45:08.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:08.139+0000 7f55d7e00700 1 -- 172.21.15.62:0/2187670967 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f55bc06ecc0 msgr2=0x7f55bc071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:45:08.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:08.139+0000 7f55d7e00700 1 --2- 172.21.15.62:0/2187670967 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f55bc06ecc0 0x7f55bc071170 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f55c0005fd0 tx=0x7f55c0005dc0 comp rx=0 tx=0).stop 2024-04-20T21:45:08.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:08.139+0000 7f55d7e00700 1 -- 172.21.15.62:0/2187670967 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55d81024d0 msgr2=0x7f55d8197e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:45:08.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:08.139+0000 7f55d7e00700 1 --2- 172.21.15.62:0/2187670967 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55d81024d0 0x7f55d8197e60 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f55c800b5c0 tx=0x7f55c800be00 comp rx=0 tx=0).stop 2024-04-20T21:45:08.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:08.140+0000 7f55d7e00700 1 -- 172.21.15.62:0/2187670967 shutdown_connections 2024-04-20T21:45:08.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:08.140+0000 7f55d7e00700 1 --2- 172.21.15.62:0/2187670967 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f55bc06ecc0 0x7f55bc071170 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:45:08.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:08.140+0000 7f55d7e00700 1 --2- 172.21.15.62:0/2187670967 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55d81024d0 0x7f55d8197e60 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:45:08.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:08.140+0000 7f55d7e00700 1 --2- 172.21.15.62:0/2187670967 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f55d8102e70 0x7f55d81983a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:45:08.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:08.140+0000 7f55d7e00700 1 -- 172.21.15.62:0/2187670967 >> 172.21.15.62:0/2187670967 conn(0x7f55d80fdd90 msgr2=0x7f55d8100180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:45:08.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:08.140+0000 7f55d7e00700 1 -- 172.21.15.62:0/2187670967 shutdown_connections 2024-04-20T21:45:08.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:08.140+0000 7f55d7e00700 1 -- 172.21.15.62:0/2187670967 wait complete. 2024-04-20T21:45:08.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:45:07 smithi189 ceph-mon[57022]: pgmap v294: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:45:08.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:08.302+0000 7f757b200700 1 -- 172.21.15.62:0/1007630969 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f757c103e50 msgr2=0x7f757c1042c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:45:08.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:08.302+0000 7f757b200700 1 --2- 172.21.15.62:0/1007630969 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f757c103e50 0x7f757c1042c0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f7564009910 tx=0x7f7564009c20 comp rx=0 tx=0).stop 2024-04-20T21:45:08.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:08.302+0000 7f757b200700 1 -- 172.21.15.62:0/1007630969 shutdown_connections 2024-04-20T21:45:08.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:08.302+0000 7f757b200700 1 --2- 172.21.15.62:0/1007630969 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f757c103e50 0x7f757c1042c0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:45:08.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:08.302+0000 7f757b200700 1 --2- 172.21.15.62:0/1007630969 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f757c1025f0 0x7f757c1029c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:45:08.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:08.302+0000 7f757b200700 1 -- 172.21.15.62:0/1007630969 >> 172.21.15.62:0/1007630969 conn(0x7f757c0fddb0 msgr2=0x7f757c1001c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:45:08.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:08.302+0000 7f757b200700 1 -- 172.21.15.62:0/1007630969 shutdown_connections 2024-04-20T21:45:08.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:08.303+0000 7f757b200700 1 -- 172.21.15.62:0/1007630969 wait complete. 2024-04-20T21:45:08.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:08.304+0000 7f757b200700 1 Processor -- start 2024-04-20T21:45:08.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:08.304+0000 7f757b200700 1 -- start start 2024-04-20T21:45:08.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:08.304+0000 7f757b200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f757c1025f0 0x7f757c075260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:45:08.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:08.304+0000 7f757b200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f757c103e50 0x7f757c0757a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:45:08.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:08.304+0000 7f757b200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f757c0793f0 con 0x7f757c103e50 2024-04-20T21:45:08.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:08.304+0000 7f757b200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f757c075ce0 con 0x7f757c1025f0 2024-04-20T21:45:08.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:08.304+0000 7f7573e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f757c103e50 0x7f757c0757a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:45:08.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:08.304+0000 7f7573e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f757c103e50 0x7f757c0757a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52658/0 (socket says 172.21.15.62:52658) 2024-04-20T21:45:08.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:08.304+0000 7f7573e00700 1 -- 172.21.15.62:0/492489341 learned_addr learned my addr 172.21.15.62:0/492489341 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:45:08.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:08.305+0000 7f7573e00700 1 -- 172.21.15.62:0/492489341 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f757c1025f0 msgr2=0x7f757c075260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:45:08.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:08.305+0000 7f7579e00700 1 --2- 172.21.15.62:0/492489341 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f757c1025f0 0x7f757c075260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:45:08.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:08.305+0000 7f7573e00700 1 --2- 172.21.15.62:0/492489341 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f757c1025f0 0x7f757c075260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:45:08.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:08.305+0000 7f7573e00700 1 -- 172.21.15.62:0/492489341 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75640095f0 con 0x7f757c103e50 2024-04-20T21:45:08.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:08.305+0000 7f7573e00700 1 --2- 172.21.15.62:0/492489341 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f757c103e50 0x7f757c0757a0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f7564006010 tx=0x7f756400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:45:08.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:08.306+0000 7f7572a00700 1 -- 172.21.15.62:0/492489341 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f756401d070 con 0x7f757c103e50 2024-04-20T21:45:08.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:08.306+0000 7f7572a00700 1 -- 172.21.15.62:0/492489341 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7564004fb0 con 0x7f757c103e50 2024-04-20T21:45:08.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:08.306+0000 7f7572a00700 1 -- 172.21.15.62:0/492489341 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f756400f690 con 0x7f757c103e50 2024-04-20T21:45:08.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:08.306+0000 7f757b200700 1 -- 172.21.15.62:0/492489341 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f757c075f60 con 0x7f757c103e50 2024-04-20T21:45:08.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:08.306+0000 7f757b200700 1 -- 172.21.15.62:0/492489341 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f757c1a67e0 con 0x7f757c103e50 2024-04-20T21:45:08.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:08.307+0000 7f757b200700 1 -- 172.21.15.62:0/492489341 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f757c10a8d0 con 0x7f757c103e50 2024-04-20T21:45:08.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:08.308+0000 7f7572a00700 1 -- 172.21.15.62:0/492489341 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f75640228a0 con 0x7f757c103e50 2024-04-20T21:45:08.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:08.309+0000 7f7572a00700 1 --2- 172.21.15.62:0/492489341 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f756006ecc0 0x7f7560071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:45:08.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:08.309+0000 7f7572a00700 1 -- 172.21.15.62:0/492489341 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f75640906d0 con 0x7f757c103e50 2024-04-20T21:45:08.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:08.309+0000 7f7579e00700 1 --2- 172.21.15.62:0/492489341 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f756006ecc0 0x7f7560071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:45:08.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:08.310+0000 7f7579e00700 1 --2- 172.21.15.62:0/492489341 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f756006ecc0 0x7f7560071170 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f756c006fd0 tx=0x7f756c008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:45:08.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:08.314+0000 7f7572a00700 1 -- 172.21.15.62:0/492489341 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f7564059460 con 0x7f757c103e50 2024-04-20T21:45:08.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:08.655+0000 7f757b200700 1 -- 172.21.15.62:0/492489341 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f757c062380 con 0x7f757c103e50 2024-04-20T21:45:08.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:08.656+0000 7f7572a00700 1 -- 172.21.15.62:0/492489341 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7f756405cb20 con 0x7f757c103e50 2024-04-20T21:45:08.656 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:45:08.656 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T21:45:08.656 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:45:08.656 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:45:08.656 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T21:45:08.656 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:45:08.656 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:45:08.656 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T21:45:08.657 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 6 2024-04-20T21:45:08.657 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:45:08.657 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T21:45:08.657 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 2, 2024-04-20T21:45:08.657 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:45:08.657 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:45:08.657 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T21:45:08.657 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 2, 2024-04-20T21:45:08.657 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 12 2024-04-20T21:45:08.657 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T21:45:08.657 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:45:08.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:08.660+0000 7f757b200700 1 -- 172.21.15.62:0/492489341 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f756006ecc0 msgr2=0x7f7560071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:45:08.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:08.660+0000 7f757b200700 1 --2- 172.21.15.62:0/492489341 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f756006ecc0 0x7f7560071170 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f756c006fd0 tx=0x7f756c008040 comp rx=0 tx=0).stop 2024-04-20T21:45:08.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:08.660+0000 7f757b200700 1 -- 172.21.15.62:0/492489341 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f757c103e50 msgr2=0x7f757c0757a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:45:08.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:08.660+0000 7f757b200700 1 --2- 172.21.15.62:0/492489341 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f757c103e50 0x7f757c0757a0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f7564006010 tx=0x7f756400be20 comp rx=0 tx=0).stop 2024-04-20T21:45:08.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:08.660+0000 7f757b200700 1 -- 172.21.15.62:0/492489341 shutdown_connections 2024-04-20T21:45:08.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:08.660+0000 7f757b200700 1 --2- 172.21.15.62:0/492489341 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f756006ecc0 0x7f7560071170 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:45:08.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:08.660+0000 7f757b200700 1 --2- 172.21.15.62:0/492489341 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f757c1025f0 0x7f757c075260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:45:08.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:08.660+0000 7f757b200700 1 --2- 172.21.15.62:0/492489341 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f757c103e50 0x7f757c0757a0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:45:08.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:08.660+0000 7f757b200700 1 -- 172.21.15.62:0/492489341 >> 172.21.15.62:0/492489341 conn(0x7f757c0fddb0 msgr2=0x7f757c0fe810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:45:08.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:08.660+0000 7f757b200700 1 -- 172.21.15.62:0/492489341 shutdown_connections 2024-04-20T21:45:08.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:08.660+0000 7f757b200700 1 -- 172.21.15.62:0/492489341 wait complete. 2024-04-20T21:45:08.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:08.820+0000 7f69cc600700 1 -- 172.21.15.62:0/108011952 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f69c40ffd60 msgr2=0x7f69c4103d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:45:08.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:08.820+0000 7f69cc600700 1 --2- 172.21.15.62:0/108011952 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f69c40ffd60 0x7f69c4103d30 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f69b0009910 tx=0x7f69b0009c20 comp rx=0 tx=0).stop 2024-04-20T21:45:08.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:08.821+0000 7f69cc600700 1 -- 172.21.15.62:0/108011952 shutdown_connections 2024-04-20T21:45:08.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:08.821+0000 7f69cc600700 1 --2- 172.21.15.62:0/108011952 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f69c40ffd60 0x7f69c4103d30 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:45:08.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:08.821+0000 7f69cc600700 1 --2- 172.21.15.62:0/108011952 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f69c40ff3c0 0x7f69c40ff790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:45:08.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:08.821+0000 7f69cc600700 1 -- 172.21.15.62:0/108011952 >> 172.21.15.62:0/108011952 conn(0x7f69c40fac80 msgr2=0x7f69c40fd090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:45:08.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:08.821+0000 7f69cc600700 1 -- 172.21.15.62:0/108011952 shutdown_connections 2024-04-20T21:45:08.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:08.821+0000 7f69cc600700 1 -- 172.21.15.62:0/108011952 wait complete. 2024-04-20T21:45:08.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:08.822+0000 7f69cc600700 1 Processor -- start 2024-04-20T21:45:08.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:08.822+0000 7f69cc600700 1 -- start start 2024-04-20T21:45:08.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:08.822+0000 7f69cc600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f69c40ff3c0 0x7f69c4195e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:45:08.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:08.822+0000 7f69cc600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f69c40ffd60 0x7f69c41963d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:45:08.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:08.822+0000 7f69cc600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69c4196ab0 con 0x7f69c40ffd60 2024-04-20T21:45:08.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:08.822+0000 7f69cc600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69c419a840 con 0x7f69c40ff3c0 2024-04-20T21:45:08.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:08.823+0000 7f69c9200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f69c40ffd60 0x7f69c41963d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:45:08.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:08.823+0000 7f69c9200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f69c40ffd60 0x7f69c41963d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52684/0 (socket says 172.21.15.62:52684) 2024-04-20T21:45:08.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:08.823+0000 7f69c9200700 1 -- 172.21.15.62:0/4228406636 learned_addr learned my addr 172.21.15.62:0/4228406636 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:45:08.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:08.823+0000 7f69c9c00700 1 --2- 172.21.15.62:0/4228406636 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f69c40ff3c0 0x7f69c4195e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:45:08.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:08.823+0000 7f69c9200700 1 -- 172.21.15.62:0/4228406636 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f69c40ff3c0 msgr2=0x7f69c4195e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:45:08.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:08.823+0000 7f69c9200700 1 --2- 172.21.15.62:0/4228406636 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f69c40ff3c0 0x7f69c4195e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:45:08.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:08.823+0000 7f69c9200700 1 -- 172.21.15.62:0/4228406636 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69b00095f0 con 0x7f69c40ffd60 2024-04-20T21:45:08.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:08.824+0000 7f69c9200700 1 --2- 172.21.15.62:0/4228406636 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f69c40ffd60 0x7f69c41963d0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f69b000bda0 tx=0x7f69b000be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:45:08.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:08.824+0000 7f69c2000700 1 -- 172.21.15.62:0/4228406636 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f69b001d070 con 0x7f69c40ffd60 2024-04-20T21:45:08.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:08.824+0000 7f69c2000700 1 -- 172.21.15.62:0/4228406636 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f69b0004fb0 con 0x7f69c40ffd60 2024-04-20T21:45:08.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:08.824+0000 7f69c2000700 1 -- 172.21.15.62:0/4228406636 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f69b000f480 con 0x7f69c40ffd60 2024-04-20T21:45:08.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:08.824+0000 7f69cc600700 1 -- 172.21.15.62:0/4228406636 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69c419aac0 con 0x7f69c40ffd60 2024-04-20T21:45:08.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:08.824+0000 7f69cc600700 1 -- 172.21.15.62:0/4228406636 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69c419afb0 con 0x7f69c40ffd60 2024-04-20T21:45:08.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:08.825+0000 7f69cc600700 1 -- 172.21.15.62:0/4228406636 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69c4108740 con 0x7f69c40ffd60 2024-04-20T21:45:08.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:08.826+0000 7f69c2000700 1 -- 172.21.15.62:0/4228406636 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f69b0022870 con 0x7f69c40ffd60 2024-04-20T21:45:08.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:08.827+0000 7f69c2000700 1 --2- 172.21.15.62:0/4228406636 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f69ac06ecc0 0x7f69ac071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:45:08.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:08.827+0000 7f69c2000700 1 -- 172.21.15.62:0/4228406636 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f69b0090370 con 0x7f69c40ffd60 2024-04-20T21:45:08.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:08.827+0000 7f69c9c00700 1 --2- 172.21.15.62:0/4228406636 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f69ac06ecc0 0x7f69ac071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:45:08.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:08.828+0000 7f69c9c00700 1 --2- 172.21.15.62:0/4228406636 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f69ac06ecc0 0x7f69ac071170 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f69b8005fd0 tx=0x7f69b800a560 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:45:08.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:08.832+0000 7f69c2000700 1 -- 172.21.15.62:0/4228406636 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f69b00591b0 con 0x7f69c40ffd60 2024-04-20T21:45:09.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:45:08 smithi062 ceph-mon[76322]: from='client.34592 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:45:09.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:45:08 smithi062 ceph-mon[76322]: from='client.34596 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:45:09.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:45:08 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/492489341' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:45:09.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:09.121+0000 7f69cc600700 1 -- 172.21.15.62:0/4228406636 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f69c404f2a0 con 0x7f69c40ffd60 2024-04-20T21:45:09.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:09.122+0000 7f69c2000700 1 -- 172.21.15.62:0/4228406636 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 25 v25) v1 ==== 76+0+1887 (secure 0 0 0) 0x7f69b005c870 con 0x7f69c40ffd60 2024-04-20T21:45:09.122 INFO:teuthology.orchestra.run.smithi062.stdout:e25 2024-04-20T21:45:09.122 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T21:45:09.122 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T21:45:09.122 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T21:45:09.122 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:45:09.122 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T21:45:09.122 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T21:45:09.122 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 20 2024-04-20T21:45:09.123 INFO:teuthology.orchestra.run.smithi062.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-20T21:45:09.123 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T21:45:09.123 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T21:42:01.559061+0000 2024-04-20T21:45:09.123 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T21:45:09.123 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T21:45:09.123 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T21:45:09.123 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T21:45:09.123 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T21:45:09.123 INFO:teuthology.orchestra.run.smithi062.stdout:max_xattr_size 65536 2024-04-20T21:45:09.123 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T21:45:09.123 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T21:45:09.123 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 81 2024-04-20T21:45:09.123 INFO:teuthology.orchestra.run.smithi062.stdout:snap_rstat 1 2024-04-20T21:45:09.123 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T21:45:09.124 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 2 2024-04-20T21:45:09.124 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1 2024-04-20T21:45:09.124 INFO:teuthology.orchestra.run.smithi062.stdout:up {} 2024-04-20T21:45:09.124 INFO:teuthology.orchestra.run.smithi062.stdout:failed 0,1 2024-04-20T21:45:09.124 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T21:45:09.124 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T21:45:09.124 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T21:45:09.124 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T21:45:09.124 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data enabled 2024-04-20T21:45:09.124 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T21:45:09.124 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T21:45:09.124 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 1 2024-04-20T21:45:09.124 INFO:teuthology.orchestra.run.smithi062.stdout:qdb_cluster leader: 0 members: 2024-04-20T21:45:09.124 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:45:09.125 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:45:09.125 INFO:teuthology.orchestra.run.smithi062.stdout:Standby daemons: 2024-04-20T21:45:09.125 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:45:09.125 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ryqblb{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6826/4113916205,v1:172.21.15.62:6827/4113916205] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T21:45:09.125 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ozizwi{-1:34476} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6828/3801975165,v1:172.21.15.62:6829/3801975165] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T21:45:09.125 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.xbgspw{-1:44285} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6826/3996040526,v1:172.21.15.189:6827/3996040526] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:45:09.125 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.dvimer{-1:44289} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6824/1743111843,v1:172.21.15.189:6825/1743111843] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:45:09.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:09.125+0000 7f69cc600700 1 -- 172.21.15.62:0/4228406636 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f69ac06ecc0 msgr2=0x7f69ac071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:45:09.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:09.125+0000 7f69cc600700 1 --2- 172.21.15.62:0/4228406636 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f69ac06ecc0 0x7f69ac071170 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f69b8005fd0 tx=0x7f69b800a560 comp rx=0 tx=0).stop 2024-04-20T21:45:09.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:09.125+0000 7f69cc600700 1 -- 172.21.15.62:0/4228406636 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f69c40ffd60 msgr2=0x7f69c41963d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:45:09.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:09.125+0000 7f69cc600700 1 --2- 172.21.15.62:0/4228406636 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f69c40ffd60 0x7f69c41963d0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f69b000bda0 tx=0x7f69b000be80 comp rx=0 tx=0).stop 2024-04-20T21:45:09.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:09.125+0000 7f69cc600700 1 -- 172.21.15.62:0/4228406636 shutdown_connections 2024-04-20T21:45:09.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:09.125+0000 7f69cc600700 1 --2- 172.21.15.62:0/4228406636 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f69ac06ecc0 0x7f69ac071170 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:45:09.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:09.125+0000 7f69cc600700 1 --2- 172.21.15.62:0/4228406636 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f69c40ff3c0 0x7f69c4195e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:45:09.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:09.125+0000 7f69cc600700 1 --2- 172.21.15.62:0/4228406636 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f69c40ffd60 0x7f69c41963d0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:45:09.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:09.125+0000 7f69cc600700 1 -- 172.21.15.62:0/4228406636 >> 172.21.15.62:0/4228406636 conn(0x7f69c40fac80 msgr2=0x7f69c40690f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:45:09.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:09.125+0000 7f69cc600700 1 -- 172.21.15.62:0/4228406636 shutdown_connections 2024-04-20T21:45:09.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:09.126+0000 7f69cc600700 1 -- 172.21.15.62:0/4228406636 wait complete. 2024-04-20T21:45:09.127 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 25 2024-04-20T21:45:09.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:45:08 smithi189 ceph-mon[57022]: from='client.34592 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:45:09.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:45:08 smithi189 ceph-mon[57022]: from='client.34596 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:45:09.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:45:08 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/492489341' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:45:09.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:09.294+0000 7efdf1200700 1 -- 172.21.15.62:0/2331642292 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efdec073a00 msgr2=0x7efdec10c360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:45:09.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:09.294+0000 7efdf1200700 1 --2- 172.21.15.62:0/2331642292 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efdec073a00 0x7efdec10c360 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7efdd4009910 tx=0x7efdd4009c20 comp rx=0 tx=0).stop 2024-04-20T21:45:09.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:09.294+0000 7efdf1200700 1 -- 172.21.15.62:0/2331642292 shutdown_connections 2024-04-20T21:45:09.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:09.294+0000 7efdf1200700 1 --2- 172.21.15.62:0/2331642292 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efdec073a00 0x7efdec10c360 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:45:09.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:09.294+0000 7efdf1200700 1 --2- 172.21.15.62:0/2331642292 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efdec0730f0 0x7efdec0734c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:45:09.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:09.294+0000 7efdf1200700 1 -- 172.21.15.62:0/2331642292 >> 172.21.15.62:0/2331642292 conn(0x7efdec0fbb80 msgr2=0x7efdec0fdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:45:09.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:09.295+0000 7efdf1200700 1 -- 172.21.15.62:0/2331642292 shutdown_connections 2024-04-20T21:45:09.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:09.295+0000 7efdf1200700 1 -- 172.21.15.62:0/2331642292 wait complete. 2024-04-20T21:45:09.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:09.296+0000 7efdf1200700 1 Processor -- start 2024-04-20T21:45:09.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:09.296+0000 7efdf1200700 1 -- start start 2024-04-20T21:45:09.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:09.296+0000 7efdf1200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efdec0730f0 0x7efdec197cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:45:09.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:09.296+0000 7efdf1200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efdec073a00 0x7efdec198210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:45:09.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:09.296+0000 7efdf1200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdec1988a0 con 0x7efdec073a00 2024-04-20T21:45:09.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:09.296+0000 7efdf1200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdec19b5d0 con 0x7efdec0730f0 2024-04-20T21:45:09.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:09.297+0000 7efde3e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efdec073a00 0x7efdec198210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:45:09.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:09.297+0000 7efde3e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efdec073a00 0x7efdec198210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52698/0 (socket says 172.21.15.62:52698) 2024-04-20T21:45:09.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:09.297+0000 7efde3e00700 1 -- 172.21.15.62:0/3675957798 learned_addr learned my addr 172.21.15.62:0/3675957798 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:45:09.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:09.297+0000 7efdeb400700 1 --2- 172.21.15.62:0/3675957798 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efdec0730f0 0x7efdec197cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:45:09.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:09.297+0000 7efde3e00700 1 -- 172.21.15.62:0/3675957798 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efdec0730f0 msgr2=0x7efdec197cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:45:09.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:09.297+0000 7efde3e00700 1 --2- 172.21.15.62:0/3675957798 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efdec0730f0 0x7efdec197cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:45:09.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:09.297+0000 7efde3e00700 1 -- 172.21.15.62:0/3675957798 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdd40095f0 con 0x7efdec073a00 2024-04-20T21:45:09.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:09.298+0000 7efde3e00700 1 --2- 172.21.15.62:0/3675957798 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efdec073a00 0x7efdec198210 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7efdd400b5c0 tx=0x7efdd400be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:45:09.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:09.298+0000 7efde8c00700 1 -- 172.21.15.62:0/3675957798 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efdd401d070 con 0x7efdec073a00 2024-04-20T21:45:09.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:09.298+0000 7efde8c00700 1 -- 172.21.15.62:0/3675957798 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efdd400f460 con 0x7efdec073a00 2024-04-20T21:45:09.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:09.298+0000 7efdf1200700 1 -- 172.21.15.62:0/3675957798 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdec19b8b0 con 0x7efdec073a00 2024-04-20T21:45:09.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:09.298+0000 7efde8c00700 1 -- 172.21.15.62:0/3675957798 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efdd4017630 con 0x7efdec073a00 2024-04-20T21:45:09.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:09.298+0000 7efdf1200700 1 -- 172.21.15.62:0/3675957798 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdec19be00 con 0x7efdec073a00 2024-04-20T21:45:09.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:09.299+0000 7efdf1200700 1 -- 172.21.15.62:0/3675957798 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdec109a60 con 0x7efdec073a00 2024-04-20T21:45:09.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:09.301+0000 7efde8c00700 1 -- 172.21.15.62:0/3675957798 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7efdd400f5d0 con 0x7efdec073a00 2024-04-20T21:45:09.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:09.302+0000 7efde8c00700 1 --2- 172.21.15.62:0/3675957798 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7efdd006ecc0 0x7efdd0071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:45:09.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:09.302+0000 7efde8c00700 1 -- 172.21.15.62:0/3675957798 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7efdd4090490 con 0x7efdec073a00 2024-04-20T21:45:09.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:09.302+0000 7efdeb400700 1 --2- 172.21.15.62:0/3675957798 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7efdd006ecc0 0x7efdd0071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:45:09.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:09.303+0000 7efdeb400700 1 --2- 172.21.15.62:0/3675957798 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7efdd006ecc0 0x7efdd0071170 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7efddc005ea0 tx=0x7efddc005e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:45:09.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:09.310+0000 7efde8c00700 1 -- 172.21.15.62:0/3675957798 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7efdd40591a0 con 0x7efdec073a00 2024-04-20T21:45:09.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:09.580+0000 7efdf1200700 1 -- 172.21.15.62:0/3675957798 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7efdec19c020 con 0x7efdd006ecc0 2024-04-20T21:45:09.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:09.583+0000 7efde8c00700 1 -- 172.21.15.62:0/3675957798 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7efdec19c020 con 0x7efdd006ecc0 2024-04-20T21:45:09.583 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:45:09.583 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T21:45:09.583 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T21:45:09.583 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-20T21:45:09.583 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T21:45:09.583 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr", 2024-04-20T21:45:09.583 INFO:teuthology.orchestra.run.smithi062.stdout: "crash", 2024-04-20T21:45:09.583 INFO:teuthology.orchestra.run.smithi062.stdout: "mon", 2024-04-20T21:45:09.583 INFO:teuthology.orchestra.run.smithi062.stdout: "osd" 2024-04-20T21:45:09.583 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T21:45:09.583 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "14/23 daemons upgraded", 2024-04-20T21:45:09.583 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading mds daemons", 2024-04-20T21:45:09.583 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T21:45:09.583 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:45:09.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:09.586+0000 7efdf1200700 1 -- 172.21.15.62:0/3675957798 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7efdd006ecc0 msgr2=0x7efdd0071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:45:09.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:09.586+0000 7efdf1200700 1 --2- 172.21.15.62:0/3675957798 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7efdd006ecc0 0x7efdd0071170 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7efddc005ea0 tx=0x7efddc005e30 comp rx=0 tx=0).stop 2024-04-20T21:45:09.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:09.586+0000 7efdf1200700 1 -- 172.21.15.62:0/3675957798 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efdec073a00 msgr2=0x7efdec198210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:45:09.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:09.586+0000 7efdf1200700 1 --2- 172.21.15.62:0/3675957798 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efdec073a00 0x7efdec198210 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7efdd400b5c0 tx=0x7efdd400be00 comp rx=0 tx=0).stop 2024-04-20T21:45:09.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:09.586+0000 7efdf1200700 1 -- 172.21.15.62:0/3675957798 shutdown_connections 2024-04-20T21:45:09.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:09.586+0000 7efdf1200700 1 --2- 172.21.15.62:0/3675957798 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7efdd006ecc0 0x7efdd0071170 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:45:09.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:09.586+0000 7efdf1200700 1 --2- 172.21.15.62:0/3675957798 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efdec0730f0 0x7efdec197cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:45:09.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:09.586+0000 7efdf1200700 1 --2- 172.21.15.62:0/3675957798 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efdec073a00 0x7efdec198210 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:45:09.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:09.586+0000 7efdf1200700 1 -- 172.21.15.62:0/3675957798 >> 172.21.15.62:0/3675957798 conn(0x7efdec0fbb80 msgr2=0x7efdec106ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:45:09.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:09.587+0000 7efdf1200700 1 -- 172.21.15.62:0/3675957798 shutdown_connections 2024-04-20T21:45:09.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:09.587+0000 7efdf1200700 1 -- 172.21.15.62:0/3675957798 wait complete. 2024-04-20T21:45:09.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:09.744+0000 7f241c000700 1 -- 172.21.15.62:0/3811729408 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f241410c8f0 msgr2=0x7f241410ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:45:09.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:09.744+0000 7f241c000700 1 --2- 172.21.15.62:0/3811729408 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f241410c8f0 0x7f241410ccc0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f2408009910 tx=0x7f2408009c20 comp rx=0 tx=0).stop 2024-04-20T21:45:09.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:09.745+0000 7f241c000700 1 -- 172.21.15.62:0/3811729408 shutdown_connections 2024-04-20T21:45:09.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:09.745+0000 7f241c000700 1 --2- 172.21.15.62:0/3811729408 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2414106b80 0x7f2414106ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:45:09.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:09.745+0000 7f241c000700 1 --2- 172.21.15.62:0/3811729408 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f241410c8f0 0x7f241410ccc0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:45:09.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:09.745+0000 7f241c000700 1 -- 172.21.15.62:0/3811729408 >> 172.21.15.62:0/3811729408 conn(0x7f2414076060 msgr2=0x7f2414078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:45:09.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:09.745+0000 7f241c000700 1 -- 172.21.15.62:0/3811729408 shutdown_connections 2024-04-20T21:45:09.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:09.745+0000 7f241c000700 1 -- 172.21.15.62:0/3811729408 wait complete. 2024-04-20T21:45:09.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:09.746+0000 7f241c000700 1 Processor -- start 2024-04-20T21:45:09.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:09.746+0000 7f241c000700 1 -- start start 2024-04-20T21:45:09.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:09.746+0000 7f241c000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2414106b80 0x7f241419c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:45:09.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:09.747+0000 7f241c000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f241410c8f0 0x7f241419ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:45:09.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:09.747+0000 7f241c000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f241419d090 con 0x7f241410c8f0 2024-04-20T21:45:09.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:09.747+0000 7f241c000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f241419d1d0 con 0x7f2414106b80 2024-04-20T21:45:09.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:09.747+0000 7f2418c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f241410c8f0 0x7f241419ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:45:09.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:09.747+0000 7f2418c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f241410c8f0 0x7f241419ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52730/0 (socket says 172.21.15.62:52730) 2024-04-20T21:45:09.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:09.747+0000 7f2418c00700 1 -- 172.21.15.62:0/2497278693 learned_addr learned my addr 172.21.15.62:0/2497278693 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:45:09.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:09.747+0000 7f2419600700 1 --2- 172.21.15.62:0/2497278693 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2414106b80 0x7f241419c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:45:09.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:09.747+0000 7f2418c00700 1 -- 172.21.15.62:0/2497278693 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2414106b80 msgr2=0x7f241419c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:45:09.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:09.747+0000 7f2418c00700 1 --2- 172.21.15.62:0/2497278693 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2414106b80 0x7f241419c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:45:09.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:09.747+0000 7f2418c00700 1 -- 172.21.15.62:0/2497278693 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24080095f0 con 0x7f241410c8f0 2024-04-20T21:45:09.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:09.748+0000 7f2418c00700 1 --2- 172.21.15.62:0/2497278693 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f241410c8f0 0x7f241419ca40 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f240000c930 tx=0x7f240000ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:45:09.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:09.748+0000 7f2412000700 1 -- 172.21.15.62:0/2497278693 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2400007ad0 con 0x7f241410c8f0 2024-04-20T21:45:09.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:09.748+0000 7f241c000700 1 -- 172.21.15.62:0/2497278693 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24141a1020 con 0x7f241410c8f0 2024-04-20T21:45:09.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:09.748+0000 7f2412000700 1 -- 172.21.15.62:0/2497278693 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2400007c30 con 0x7f241410c8f0 2024-04-20T21:45:09.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:09.748+0000 7f2412000700 1 -- 172.21.15.62:0/2497278693 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2400018610 con 0x7f241410c8f0 2024-04-20T21:45:09.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:09.748+0000 7f241c000700 1 -- 172.21.15.62:0/2497278693 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24141a1570 con 0x7f241410c8f0 2024-04-20T21:45:09.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:09.749+0000 7f241c000700 1 -- 172.21.15.62:0/2497278693 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2414109b20 con 0x7f241410c8f0 2024-04-20T21:45:09.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:09.750+0000 7f2412000700 1 -- 172.21.15.62:0/2497278693 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f240000f450 con 0x7f241410c8f0 2024-04-20T21:45:09.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:09.752+0000 7f2412000700 1 --2- 172.21.15.62:0/2497278693 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f23fc06ecc0 0x7f23fc071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:45:09.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:09.752+0000 7f2412000700 1 -- 172.21.15.62:0/2497278693 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f240008f140 con 0x7f241410c8f0 2024-04-20T21:45:09.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:09.752+0000 7f2419600700 1 --2- 172.21.15.62:0/2497278693 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f23fc06ecc0 0x7f23fc071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:45:09.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:09.753+0000 7f2419600700 1 --2- 172.21.15.62:0/2497278693 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f23fc06ecc0 0x7f23fc071170 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f240800b5c0 tx=0x7f240801a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:45:09.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:09.756+0000 7f2412000700 1 -- 172.21.15.62:0/2497278693 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f2400057e50 con 0x7f241410c8f0 2024-04-20T21:45:10.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:10.092+0000 7f241c000700 1 -- 172.21.15.62:0/2497278693 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f241404f2a0 con 0x7f241410c8f0 2024-04-20T21:45:10.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:10.093+0000 7f2412000700 1 -- 172.21.15.62:0/2497278693 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7f240005b510 con 0x7f241410c8f0 2024-04-20T21:45:10.092 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-20T21:45:10.093 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-20T21:45:10.093 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is degraded 2024-04-20T21:45:10.093 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T21:45:10.093 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T21:45:10.093 INFO:teuthology.orchestra.run.smithi062.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-20T21:45:10.093 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-20T21:45:10.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:10.096+0000 7f241c000700 1 -- 172.21.15.62:0/2497278693 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f23fc06ecc0 msgr2=0x7f23fc071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:45:10.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:10.096+0000 7f241c000700 1 --2- 172.21.15.62:0/2497278693 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f23fc06ecc0 0x7f23fc071170 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f240800b5c0 tx=0x7f240801a040 comp rx=0 tx=0).stop 2024-04-20T21:45:10.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:10.096+0000 7f241c000700 1 -- 172.21.15.62:0/2497278693 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f241410c8f0 msgr2=0x7f241419ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:45:10.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:10.096+0000 7f241c000700 1 --2- 172.21.15.62:0/2497278693 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f241410c8f0 0x7f241419ca40 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f240000c930 tx=0x7f240000ccf0 comp rx=0 tx=0).stop 2024-04-20T21:45:10.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:10.096+0000 7f241c000700 1 -- 172.21.15.62:0/2497278693 shutdown_connections 2024-04-20T21:45:10.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:10.096+0000 7f241c000700 1 --2- 172.21.15.62:0/2497278693 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f23fc06ecc0 0x7f23fc071170 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:45:10.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:10.096+0000 7f241c000700 1 --2- 172.21.15.62:0/2497278693 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2414106b80 0x7f241419c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:45:10.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:10.096+0000 7f241c000700 1 --2- 172.21.15.62:0/2497278693 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f241410c8f0 0x7f241419ca40 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:45:10.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:10.096+0000 7f241c000700 1 -- 172.21.15.62:0/2497278693 >> 172.21.15.62:0/2497278693 conn(0x7f2414076060 msgr2=0x7f2414077aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:45:10.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:10.097+0000 7f241c000700 1 -- 172.21.15.62:0/2497278693 shutdown_connections 2024-04-20T21:45:10.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:10.097+0000 7f241c000700 1 -- 172.21.15.62:0/2497278693 wait complete. 2024-04-20T21:45:10.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:45:09 smithi189 ceph-mon[57022]: from='client.34600 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:45:10.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:45:09 smithi189 ceph-mon[57022]: pgmap v295: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:45:10.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:45:09 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/4228406636' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T21:45:10.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:45:09 smithi062 ceph-mon[76322]: from='client.34600 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:45:10.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:45:09 smithi062 ceph-mon[76322]: pgmap v295: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:45:10.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:45:09 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/4228406636' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T21:45:11.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:45:10 smithi189 ceph-mon[57022]: from='client.34612 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:45:11.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:45:10 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/2497278693' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T21:45:11.284 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:45:10 smithi062 ceph-mon[76322]: from='client.34612 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:45:11.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:45:10 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/2497278693' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T21:45:12.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:45:11 smithi189 ceph-mon[57022]: pgmap v296: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:45:12.284 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:45:11 smithi062 ceph-mon[76322]: pgmap v296: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:45:14.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:45:13 smithi189 ceph-mon[57022]: pgmap v297: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:45:14.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:45:13 smithi062 ceph-mon[76322]: pgmap v297: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:45:16.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:45:15 smithi189 ceph-mon[57022]: pgmap v298: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:45:16.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:45:15 smithi062 ceph-mon[76322]: pgmap v298: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:45:18.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:45:17 smithi189 ceph-mon[57022]: pgmap v299: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:45:18.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:45:17 smithi062 ceph-mon[76322]: pgmap v299: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:45:20.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:45:19 smithi189 ceph-mon[57022]: pgmap v300: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:45:20.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:45:19 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:45:20.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:45:19 smithi062 ceph-mon[76322]: pgmap v300: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:45:20.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:45:19 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:45:22.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:45:21 smithi189 ceph-mon[57022]: pgmap v301: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:45:22.284 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:45:21 smithi062 ceph-mon[76322]: pgmap v301: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:45:24.236 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:45:23 smithi189 ceph-mon[57022]: pgmap v302: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:45:24.284 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:45:23 smithi062 ceph-mon[76322]: pgmap v302: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:45:26.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:45:25 smithi189 ceph-mon[57022]: pgmap v303: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:45:26.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:45:25 smithi062 ceph-mon[76322]: pgmap v303: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:45:28.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:45:27 smithi189 ceph-mon[57022]: pgmap v304: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:45:28.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:45:27 smithi062 ceph-mon[76322]: pgmap v304: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:45:30.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:45:29 smithi189 ceph-mon[57022]: pgmap v305: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:45:30.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:45:29 smithi062 ceph-mon[76322]: pgmap v305: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:45:32.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:45:31 smithi189 ceph-mon[57022]: pgmap v306: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:45:32.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:45:31 smithi062 ceph-mon[76322]: pgmap v306: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:45:34.236 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:45:33 smithi189 ceph-mon[57022]: pgmap v307: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:45:34.284 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:45:33 smithi062 ceph-mon[76322]: pgmap v307: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:45:35.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:45:34 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:45:35.284 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:45:34 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:45:36.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:45:35 smithi189 ceph-mon[57022]: pgmap v308: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:45:36.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:45:35 smithi062 ceph-mon[76322]: pgmap v308: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:45:38.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:45:37 smithi189 ceph-mon[57022]: pgmap v309: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:45:38.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:45:37 smithi062 ceph-mon[76322]: pgmap v309: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:45:40.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:40.281+0000 7fb635400700 1 -- 172.21.15.62:0/2355969507 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb6300687a0 msgr2=0x7fb630068b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:45:40.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:40.281+0000 7fb635400700 1 --2- 172.21.15.62:0/2355969507 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb6300687a0 0x7fb630068b70 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fb620009910 tx=0x7fb620009c20 comp rx=0 tx=0).stop 2024-04-20T21:45:40.281 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:45:39 smithi062 ceph-mon[76322]: pgmap v310: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:45:40.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:40.281+0000 7fb635400700 1 -- 172.21.15.62:0/2355969507 shutdown_connections 2024-04-20T21:45:40.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:40.281+0000 7fb635400700 1 --2- 172.21.15.62:0/2355969507 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb630069140 0x7fb630108090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:45:40.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:40.281+0000 7fb635400700 1 --2- 172.21.15.62:0/2355969507 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb6300687a0 0x7fb630068b70 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:45:40.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:40.281+0000 7fb635400700 1 -- 172.21.15.62:0/2355969507 >> 172.21.15.62:0/2355969507 conn(0x7fb630074fa0 msgr2=0x7fb6300753a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:45:40.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:40.282+0000 7fb635400700 1 -- 172.21.15.62:0/2355969507 shutdown_connections 2024-04-20T21:45:40.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:40.282+0000 7fb635400700 1 -- 172.21.15.62:0/2355969507 wait complete. 2024-04-20T21:45:40.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:40.283+0000 7fb635400700 1 Processor -- start 2024-04-20T21:45:40.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:40.283+0000 7fb635400700 1 -- start start 2024-04-20T21:45:40.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:40.283+0000 7fb635400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb6300687a0 0x7fb630197df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:45:40.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:40.284+0000 7fb635400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb630069140 0x7fb630198330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:45:40.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:40.284+0000 7fb635400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb630198a10 con 0x7fb630069140 2024-04-20T21:45:40.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:40.284+0000 7fb635400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb63019c7a0 con 0x7fb6300687a0 2024-04-20T21:45:40.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:40.284+0000 7fb627400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb630069140 0x7fb630198330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:45:40.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:40.284+0000 7fb627400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb630069140 0x7fb630198330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53516/0 (socket says 172.21.15.62:53516) 2024-04-20T21:45:40.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:40.284+0000 7fb627400700 1 -- 172.21.15.62:0/229243980 learned_addr learned my addr 172.21.15.62:0/229243980 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:45:40.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:40.284+0000 7fb62fe00700 1 --2- 172.21.15.62:0/229243980 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb6300687a0 0x7fb630197df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:45:40.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:40.284+0000 7fb627400700 1 -- 172.21.15.62:0/229243980 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb6300687a0 msgr2=0x7fb630197df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:45:40.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:40.284+0000 7fb627400700 1 --2- 172.21.15.62:0/229243980 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb6300687a0 0x7fb630197df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:45:40.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:40.284+0000 7fb627400700 1 -- 172.21.15.62:0/229243980 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6200095f0 con 0x7fb630069140 2024-04-20T21:45:40.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:40.285+0000 7fb627400700 1 --2- 172.21.15.62:0/229243980 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb630069140 0x7fb630198330 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fb61800d8d0 tx=0x7fb61800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:45:40.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:40.285+0000 7fb62d600700 1 -- 172.21.15.62:0/229243980 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb618009900 con 0x7fb630069140 2024-04-20T21:45:40.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:40.285+0000 7fb62d600700 1 -- 172.21.15.62:0/229243980 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb618010460 con 0x7fb630069140 2024-04-20T21:45:40.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:40.285+0000 7fb635400700 1 -- 172.21.15.62:0/229243980 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb63019ca80 con 0x7fb630069140 2024-04-20T21:45:40.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:40.285+0000 7fb62d600700 1 -- 172.21.15.62:0/229243980 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb61800f5f0 con 0x7fb630069140 2024-04-20T21:45:40.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:40.285+0000 7fb635400700 1 -- 172.21.15.62:0/229243980 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb63019d050 con 0x7fb630069140 2024-04-20T21:45:40.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:40.287+0000 7fb635400700 1 -- 172.21.15.62:0/229243980 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb630062380 con 0x7fb630069140 2024-04-20T21:45:40.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:40.288+0000 7fb62d600700 1 -- 172.21.15.62:0/229243980 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fb6180105d0 con 0x7fb630069140 2024-04-20T21:45:40.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:40.289+0000 7fb62d600700 1 --2- 172.21.15.62:0/229243980 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb61406ecc0 0x7fb614071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:45:40.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:40.289+0000 7fb62d600700 1 -- 172.21.15.62:0/229243980 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fb61808eb20 con 0x7fb630069140 2024-04-20T21:45:40.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:40.289+0000 7fb62fe00700 1 --2- 172.21.15.62:0/229243980 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb61406ecc0 0x7fb614071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:45:40.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:40.290+0000 7fb62fe00700 1 --2- 172.21.15.62:0/229243980 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb61406ecc0 0x7fb614071170 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fb620006010 tx=0x7fb62001a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:45:40.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:40.294+0000 7fb62d600700 1 -- 172.21.15.62:0/229243980 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fb6180578d0 con 0x7fb630069140 2024-04-20T21:45:40.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:45:39 smithi189 ceph-mon[57022]: pgmap v310: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:45:40.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:40.553+0000 7fb635400700 1 -- 172.21.15.62:0/229243980 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb63019d3e0 con 0x7fb61406ecc0 2024-04-20T21:45:40.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:40.555+0000 7fb62d600700 1 -- 172.21.15.62:0/229243980 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fb63019d3e0 con 0x7fb61406ecc0 2024-04-20T21:45:40.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:40.558+0000 7fb635400700 1 -- 172.21.15.62:0/229243980 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb61406ecc0 msgr2=0x7fb614071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:45:40.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:40.559+0000 7fb635400700 1 --2- 172.21.15.62:0/229243980 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb61406ecc0 0x7fb614071170 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fb620006010 tx=0x7fb62001a040 comp rx=0 tx=0).stop 2024-04-20T21:45:40.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:40.559+0000 7fb635400700 1 -- 172.21.15.62:0/229243980 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb630069140 msgr2=0x7fb630198330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:45:40.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:40.559+0000 7fb635400700 1 --2- 172.21.15.62:0/229243980 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb630069140 0x7fb630198330 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fb61800d8d0 tx=0x7fb61800dc90 comp rx=0 tx=0).stop 2024-04-20T21:45:40.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:40.559+0000 7fb635400700 1 -- 172.21.15.62:0/229243980 shutdown_connections 2024-04-20T21:45:40.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:40.559+0000 7fb635400700 1 --2- 172.21.15.62:0/229243980 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb61406ecc0 0x7fb614071170 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:45:40.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:40.559+0000 7fb635400700 1 --2- 172.21.15.62:0/229243980 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb6300687a0 0x7fb630197df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:45:40.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:40.559+0000 7fb635400700 1 --2- 172.21.15.62:0/229243980 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb630069140 0x7fb630198330 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:45:40.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:40.559+0000 7fb635400700 1 -- 172.21.15.62:0/229243980 >> 172.21.15.62:0/229243980 conn(0x7fb630074fa0 msgr2=0x7fb6301063f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:45:40.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:40.559+0000 7fb635400700 1 -- 172.21.15.62:0/229243980 shutdown_connections 2024-04-20T21:45:40.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:40.559+0000 7fb635400700 1 -- 172.21.15.62:0/229243980 wait complete. 2024-04-20T21:45:40.576 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T21:45:40.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:40.729+0000 7faac6000700 1 -- 172.21.15.62:0/2563634688 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faac0106b80 msgr2=0x7faac0106ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:45:40.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:40.729+0000 7faac6000700 1 --2- 172.21.15.62:0/2563634688 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faac0106b80 0x7faac0106ff0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7faaac009910 tx=0x7faaac009c20 comp rx=0 tx=0).stop 2024-04-20T21:45:40.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:40.729+0000 7faac6000700 1 -- 172.21.15.62:0/2563634688 shutdown_connections 2024-04-20T21:45:40.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:40.729+0000 7faac6000700 1 --2- 172.21.15.62:0/2563634688 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faac0106b80 0x7faac0106ff0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:45:40.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:40.729+0000 7faac6000700 1 --2- 172.21.15.62:0/2563634688 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faac010c8f0 0x7faac010ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:45:40.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:40.729+0000 7faac6000700 1 -- 172.21.15.62:0/2563634688 >> 172.21.15.62:0/2563634688 conn(0x7faac0076060 msgr2=0x7faac0078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:45:40.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:40.729+0000 7faac6000700 1 -- 172.21.15.62:0/2563634688 shutdown_connections 2024-04-20T21:45:40.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:40.729+0000 7faac6000700 1 -- 172.21.15.62:0/2563634688 wait complete. 2024-04-20T21:45:40.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:40.730+0000 7faac6000700 1 Processor -- start 2024-04-20T21:45:40.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:40.731+0000 7faac6000700 1 -- start start 2024-04-20T21:45:40.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:40.731+0000 7faac6000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faac0106b80 0x7faac019c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:45:40.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:40.731+0000 7faac6000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faac010c8f0 0x7faac019c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:45:40.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:40.731+0000 7faac6000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faac019d060 con 0x7faac010c8f0 2024-04-20T21:45:40.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:40.731+0000 7faac6000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faac01a0df0 con 0x7faac0106b80 2024-04-20T21:45:40.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:40.731+0000 7faabea00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faac010c8f0 0x7faac019c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:45:40.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:40.731+0000 7faabea00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faac010c8f0 0x7faac019c980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53532/0 (socket says 172.21.15.62:53532) 2024-04-20T21:45:40.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:40.731+0000 7faabea00700 1 -- 172.21.15.62:0/1799868262 learned_addr learned my addr 172.21.15.62:0/1799868262 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:45:40.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:40.732+0000 7faabf400700 1 --2- 172.21.15.62:0/1799868262 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faac0106b80 0x7faac019c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:45:40.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:40.732+0000 7faabea00700 1 -- 172.21.15.62:0/1799868262 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faac0106b80 msgr2=0x7faac019c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:45:40.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:40.732+0000 7faabea00700 1 --2- 172.21.15.62:0/1799868262 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faac0106b80 0x7faac019c440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:45:40.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:40.732+0000 7faabea00700 1 -- 172.21.15.62:0/1799868262 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaac0095f0 con 0x7faac010c8f0 2024-04-20T21:45:40.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:40.732+0000 7faabf400700 1 --2- 172.21.15.62:0/1799868262 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faac0106b80 0x7faac019c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T21:45:40.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:40.732+0000 7faabea00700 1 --2- 172.21.15.62:0/1799868262 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faac010c8f0 0x7faac019c980 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7faaac006010 tx=0x7faaac00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:45:40.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:40.733+0000 7faab7e00700 1 -- 172.21.15.62:0/1799868262 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faaac01d070 con 0x7faac010c8f0 2024-04-20T21:45:40.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:40.733+0000 7faab7e00700 1 -- 172.21.15.62:0/1799868262 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7faaac004fb0 con 0x7faac010c8f0 2024-04-20T21:45:40.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:40.733+0000 7faab7e00700 1 -- 172.21.15.62:0/1799868262 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faaac022ea0 con 0x7faac010c8f0 2024-04-20T21:45:40.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:40.733+0000 7faac6000700 1 -- 172.21.15.62:0/1799868262 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faac01a1070 con 0x7faac010c8f0 2024-04-20T21:45:40.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:40.733+0000 7faac6000700 1 -- 172.21.15.62:0/1799868262 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faac01a1560 con 0x7faac010c8f0 2024-04-20T21:45:40.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:40.734+0000 7faac6000700 1 -- 172.21.15.62:0/1799868262 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faac010ecc0 con 0x7faac010c8f0 2024-04-20T21:45:40.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:40.735+0000 7faab7e00700 1 -- 172.21.15.62:0/1799868262 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7faaac022470 con 0x7faac010c8f0 2024-04-20T21:45:40.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:40.736+0000 7faab7e00700 1 --2- 172.21.15.62:0/1799868262 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7faaa406ecc0 0x7faaa4071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:45:40.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:40.737+0000 7faab7e00700 1 -- 172.21.15.62:0/1799868262 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7faaac0903f0 con 0x7faac010c8f0 2024-04-20T21:45:40.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:40.737+0000 7faabf400700 1 --2- 172.21.15.62:0/1799868262 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7faaa406ecc0 0x7faaa4071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:45:40.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:40.737+0000 7faabf400700 1 --2- 172.21.15.62:0/1799868262 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7faaa406ecc0 0x7faaa4071170 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7faab0005fd0 tx=0x7faab0005dc0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:45:40.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:40.741+0000 7faab7e00700 1 -- 172.21.15.62:0/1799868262 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7faaac059230 con 0x7faac010c8f0 2024-04-20T21:45:41.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.005+0000 7faac6000700 1 -- 172.21.15.62:0/1799868262 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7faac01a1890 con 0x7faaa406ecc0 2024-04-20T21:45:41.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.007+0000 7faab7e00700 1 -- 172.21.15.62:0/1799868262 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7faac01a1890 con 0x7faaa406ecc0 2024-04-20T21:45:41.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.010+0000 7faac6000700 1 -- 172.21.15.62:0/1799868262 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7faaa406ecc0 msgr2=0x7faaa4071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:45:41.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.010+0000 7faac6000700 1 --2- 172.21.15.62:0/1799868262 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7faaa406ecc0 0x7faaa4071170 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7faab0005fd0 tx=0x7faab0005dc0 comp rx=0 tx=0).stop 2024-04-20T21:45:41.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.010+0000 7faac6000700 1 -- 172.21.15.62:0/1799868262 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faac010c8f0 msgr2=0x7faac019c980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:45:41.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.010+0000 7faac6000700 1 --2- 172.21.15.62:0/1799868262 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faac010c8f0 0x7faac019c980 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7faaac006010 tx=0x7faaac00be20 comp rx=0 tx=0).stop 2024-04-20T21:45:41.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.010+0000 7faac6000700 1 -- 172.21.15.62:0/1799868262 shutdown_connections 2024-04-20T21:45:41.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.010+0000 7faac6000700 1 --2- 172.21.15.62:0/1799868262 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7faaa406ecc0 0x7faaa4071170 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:45:41.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.010+0000 7faac6000700 1 --2- 172.21.15.62:0/1799868262 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faac0106b80 0x7faac019c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:45:41.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.010+0000 7faac6000700 1 --2- 172.21.15.62:0/1799868262 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faac010c8f0 0x7faac019c980 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:45:41.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.010+0000 7faac6000700 1 -- 172.21.15.62:0/1799868262 >> 172.21.15.62:0/1799868262 conn(0x7faac0076060 msgr2=0x7faac0077ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:45:41.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.010+0000 7faac6000700 1 -- 172.21.15.62:0/1799868262 shutdown_connections 2024-04-20T21:45:41.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.010+0000 7faac6000700 1 -- 172.21.15.62:0/1799868262 wait complete. 2024-04-20T21:45:41.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.168+0000 7f3fbfc00700 1 -- 172.21.15.62:0/957764254 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3fb81004e0 msgr2=0x7f3fb81008b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:45:41.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.168+0000 7f3fbfc00700 1 --2- 172.21.15.62:0/957764254 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3fb81004e0 0x7f3fb81008b0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f3fac009910 tx=0x7f3fac009c20 comp rx=0 tx=0).stop 2024-04-20T21:45:41.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.168+0000 7f3fbfc00700 1 -- 172.21.15.62:0/957764254 shutdown_connections 2024-04-20T21:45:41.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.168+0000 7f3fbfc00700 1 --2- 172.21.15.62:0/957764254 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3fb8100e80 0x7f3fb8104e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:45:41.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.169+0000 7f3fbfc00700 1 --2- 172.21.15.62:0/957764254 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3fb81004e0 0x7f3fb81008b0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:45:41.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.169+0000 7f3fbfc00700 1 -- 172.21.15.62:0/957764254 >> 172.21.15.62:0/957764254 conn(0x7f3fb80fbda0 msgr2=0x7f3fb80fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:45:41.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.169+0000 7f3fbfc00700 1 -- 172.21.15.62:0/957764254 shutdown_connections 2024-04-20T21:45:41.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.169+0000 7f3fbfc00700 1 -- 172.21.15.62:0/957764254 wait complete. 2024-04-20T21:45:41.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.170+0000 7f3fbfc00700 1 Processor -- start 2024-04-20T21:45:41.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.170+0000 7f3fbfc00700 1 -- start start 2024-04-20T21:45:41.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.170+0000 7f3fbfc00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3fb81004e0 0x7f3fb8195f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:45:41.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.170+0000 7f3fbfc00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3fb8100e80 0x7f3fb81964d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:45:41.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.170+0000 7f3fbfc00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fb8196bb0 con 0x7f3fb8100e80 2024-04-20T21:45:41.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.170+0000 7f3fbfc00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fb819a940 con 0x7f3fb81004e0 2024-04-20T21:45:41.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.171+0000 7f3fb7e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3fb8100e80 0x7f3fb81964d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:45:41.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.171+0000 7f3fb7e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3fb8100e80 0x7f3fb81964d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53550/0 (socket says 172.21.15.62:53550) 2024-04-20T21:45:41.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.171+0000 7f3fb7e00700 1 -- 172.21.15.62:0/471039804 learned_addr learned my addr 172.21.15.62:0/471039804 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:45:41.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.171+0000 7f3fbd200700 1 --2- 172.21.15.62:0/471039804 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3fb81004e0 0x7f3fb8195f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:45:41.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.171+0000 7f3fb7e00700 1 -- 172.21.15.62:0/471039804 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3fb81004e0 msgr2=0x7f3fb8195f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:45:41.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.171+0000 7f3fb7e00700 1 --2- 172.21.15.62:0/471039804 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3fb81004e0 0x7f3fb8195f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:45:41.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.171+0000 7f3fb7e00700 1 -- 172.21.15.62:0/471039804 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fac0095f0 con 0x7f3fb8100e80 2024-04-20T21:45:41.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.172+0000 7f3fb7e00700 1 --2- 172.21.15.62:0/471039804 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3fb8100e80 0x7f3fb81964d0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f3fa400b480 tx=0x7f3fa400b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:45:41.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.172+0000 7f3fb5600700 1 -- 172.21.15.62:0/471039804 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3fa40118c0 con 0x7f3fb8100e80 2024-04-20T21:45:41.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.172+0000 7f3fb5600700 1 -- 172.21.15.62:0/471039804 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3fa4010380 con 0x7f3fb8100e80 2024-04-20T21:45:41.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.172+0000 7f3fb5600700 1 -- 172.21.15.62:0/471039804 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3fa400f4f0 con 0x7f3fb8100e80 2024-04-20T21:45:41.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.172+0000 7f3fbfc00700 1 -- 172.21.15.62:0/471039804 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fb819ac20 con 0x7f3fb8100e80 2024-04-20T21:45:41.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.172+0000 7f3fbfc00700 1 -- 172.21.15.62:0/471039804 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fb819b170 con 0x7f3fb8100e80 2024-04-20T21:45:41.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.173+0000 7f3fbfc00700 1 -- 172.21.15.62:0/471039804 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fb81087f0 con 0x7f3fb8100e80 2024-04-20T21:45:41.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.174+0000 7f3fb5600700 1 -- 172.21.15.62:0/471039804 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f3fa40104f0 con 0x7f3fb8100e80 2024-04-20T21:45:41.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.175+0000 7f3fb5600700 1 --2- 172.21.15.62:0/471039804 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3fa006ecc0 0x7f3fa0071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:45:41.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.175+0000 7f3fb5600700 1 -- 172.21.15.62:0/471039804 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f3fa408ea70 con 0x7f3fb8100e80 2024-04-20T21:45:41.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.176+0000 7f3fbd200700 1 --2- 172.21.15.62:0/471039804 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3fa006ecc0 0x7f3fa0071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:45:41.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.176+0000 7f3fbd200700 1 --2- 172.21.15.62:0/471039804 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3fa006ecc0 0x7f3fa0071170 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f3fac006010 tx=0x7f3fac01a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:45:41.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.180+0000 7f3fb5600700 1 -- 172.21.15.62:0/471039804 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f3fa4057800 con 0x7f3fb8100e80 2024-04-20T21:45:41.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:45:40 smithi062 ceph-mon[76322]: pgmap v311: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:45:41.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:45:40 smithi062 ceph-mon[76322]: from='client.34620 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:45:41.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.416+0000 7f3fbfc00700 1 -- 172.21.15.62:0/471039804 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f3fb819b4b0 con 0x7f3fa006ecc0 2024-04-20T21:45:41.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.445+0000 7f3fb5600700 1 -- 172.21.15.62:0/471039804 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f3fb819b4b0 con 0x7f3fa006ecc0 2024-04-20T21:45:41.444 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T21:45:41.445 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (11m) 2m ago 25m 14.2M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T21:45:41.445 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (25m) 2m ago 25m 9512k - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T21:45:41.445 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (24m) 3m ago 24m 8942k - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T21:45:41.445 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (7m) 2m ago 25m 6660k - 19.0.0-3133-ga8c84f59 76bc6b91b726 2825d5ba11c6 2024-04-20T21:45:41.445 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (7m) 3m ago 24m 6665k - 19.0.0-3133-ga8c84f59 76bc6b91b726 20f746fa1bba 2024-04-20T21:45:41.445 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (10m) 2m ago 25m 74.2M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T21:45:41.445 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (3m) 2m ago 18m 11.2M - 19.0.0-3133-ga8c84f59 76bc6b91b726 de80bc90a683 2024-04-20T21:45:41.445 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (3m) 2m ago 18m 11.9M - 19.0.0-3133-ga8c84f59 76bc6b91b726 cd793b7ee98e 2024-04-20T21:45:41.445 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (18m) 3m ago 18m 117M - 18.2.0 dc2bc1663786 fb2a8a223db8 2024-04-20T21:45:41.445 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (18m) 3m ago 18m 55.6M - 18.2.0 dc2bc1663786 f2eec2ee8951 2024-04-20T21:45:41.445 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (14m) 2m ago 27m 511M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T21:45:41.445 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (13m) 3m ago 24m 452M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T21:45:41.445 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (9m) 2m ago 27m 55.9M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 019eb1479485 2024-04-20T21:45:41.445 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (8m) 3m ago 23m 39.8M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 55aac560aab6 2024-04-20T21:45:41.446 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (12m) 2m ago 25m 7952k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T21:45:41.446 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (12m) 3m ago 24m 8304k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T21:45:41.446 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (6m) 2m ago 23m 88.0M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 ee2bb901f26e 2024-04-20T21:45:41.446 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (6m) 2m ago 22m 92.3M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 9075152d4162 2024-04-20T21:45:41.446 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (5m) 2m ago 21m 74.4M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 2d4f3e38dc0e 2024-04-20T21:45:41.446 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (5m) 3m ago 21m 87.6M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 8f97b17ae0eb 2024-04-20T21:45:41.446 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (4m) 3m ago 20m 90.7M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 653d8079967d 2024-04-20T21:45:41.446 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (3m) 3m ago 20m 11.5M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 596dded14c12 2024-04-20T21:45:41.446 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (12m) 2m ago 24m 74.0M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T21:45:41.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.448+0000 7f3fbfc00700 1 -- 172.21.15.62:0/471039804 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3fa006ecc0 msgr2=0x7f3fa0071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:45:41.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.448+0000 7f3fbfc00700 1 --2- 172.21.15.62:0/471039804 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3fa006ecc0 0x7f3fa0071170 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f3fac006010 tx=0x7f3fac01a040 comp rx=0 tx=0).stop 2024-04-20T21:45:41.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.448+0000 7f3fbfc00700 1 -- 172.21.15.62:0/471039804 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3fb8100e80 msgr2=0x7f3fb81964d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:45:41.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.448+0000 7f3fbfc00700 1 --2- 172.21.15.62:0/471039804 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3fb8100e80 0x7f3fb81964d0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f3fa400b480 tx=0x7f3fa400b840 comp rx=0 tx=0).stop 2024-04-20T21:45:41.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.448+0000 7f3fbfc00700 1 -- 172.21.15.62:0/471039804 shutdown_connections 2024-04-20T21:45:41.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.448+0000 7f3fbfc00700 1 --2- 172.21.15.62:0/471039804 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3fa006ecc0 0x7f3fa0071170 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:45:41.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.448+0000 7f3fbfc00700 1 --2- 172.21.15.62:0/471039804 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3fb81004e0 0x7f3fb8195f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:45:41.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.448+0000 7f3fbfc00700 1 --2- 172.21.15.62:0/471039804 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3fb8100e80 0x7f3fb81964d0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:45:41.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.448+0000 7f3fbfc00700 1 -- 172.21.15.62:0/471039804 >> 172.21.15.62:0/471039804 conn(0x7f3fb80fbda0 msgr2=0x7f3fb80fd470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:45:41.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.448+0000 7f3fbfc00700 1 -- 172.21.15.62:0/471039804 shutdown_connections 2024-04-20T21:45:41.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.448+0000 7f3fbfc00700 1 -- 172.21.15.62:0/471039804 wait complete. 2024-04-20T21:45:41.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:45:41 smithi189 ceph-mon[57022]: pgmap v311: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:45:41.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:45:41 smithi189 ceph-mon[57022]: from='client.34620 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:45:41.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.611+0000 7fd0bd800700 1 -- 172.21.15.62:0/1152265065 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd0b8073980 msgr2=0x7fd0b810c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:45:41.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.611+0000 7fd0bd800700 1 --2- 172.21.15.62:0/1152265065 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd0b8073980 0x7fd0b810c600 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fd0a4009940 tx=0x7fd0a4009c50 comp rx=0 tx=0).stop 2024-04-20T21:45:41.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.611+0000 7fd0bd800700 1 -- 172.21.15.62:0/1152265065 shutdown_connections 2024-04-20T21:45:41.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.611+0000 7fd0bd800700 1 --2- 172.21.15.62:0/1152265065 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd0b8073980 0x7fd0b810c600 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:45:41.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.612+0000 7fd0bd800700 1 --2- 172.21.15.62:0/1152265065 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd0b8073070 0x7fd0b8073440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:45:41.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.612+0000 7fd0bd800700 1 -- 172.21.15.62:0/1152265065 >> 172.21.15.62:0/1152265065 conn(0x7fd0b80fbda0 msgr2=0x7fd0b80fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:45:41.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.612+0000 7fd0bd800700 1 -- 172.21.15.62:0/1152265065 shutdown_connections 2024-04-20T21:45:41.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.612+0000 7fd0bd800700 1 -- 172.21.15.62:0/1152265065 wait complete. 2024-04-20T21:45:41.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.613+0000 7fd0bd800700 1 Processor -- start 2024-04-20T21:45:41.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.613+0000 7fd0bd800700 1 -- start start 2024-04-20T21:45:41.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.613+0000 7fd0bd800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd0b8073070 0x7fd0b81981b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:45:41.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.613+0000 7fd0bd800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd0b8073980 0x7fd0b81986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:45:41.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.613+0000 7fd0bd800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0b8198dd0 con 0x7fd0b8073070 2024-04-20T21:45:41.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.613+0000 7fd0bd800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0b819cb60 con 0x7fd0b8073980 2024-04-20T21:45:41.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.614+0000 7fd0b6a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd0b8073070 0x7fd0b81981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:45:41.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.614+0000 7fd0b6a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd0b8073070 0x7fd0b81981b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53560/0 (socket says 172.21.15.62:53560) 2024-04-20T21:45:41.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.614+0000 7fd0b6a00700 1 -- 172.21.15.62:0/2121418084 learned_addr learned my addr 172.21.15.62:0/2121418084 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:45:41.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.614+0000 7fd0b6000700 1 --2- 172.21.15.62:0/2121418084 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd0b8073980 0x7fd0b81986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:45:41.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.614+0000 7fd0b6a00700 1 -- 172.21.15.62:0/2121418084 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd0b8073980 msgr2=0x7fd0b81986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:45:41.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.614+0000 7fd0b6a00700 1 --2- 172.21.15.62:0/2121418084 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd0b8073980 0x7fd0b81986f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:45:41.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.614+0000 7fd0b6a00700 1 -- 172.21.15.62:0/2121418084 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0a40095f0 con 0x7fd0b8073070 2024-04-20T21:45:41.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.615+0000 7fd0b6a00700 1 --2- 172.21.15.62:0/2121418084 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd0b8073070 0x7fd0b81981b0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fd0a800b4b0 tx=0x7fd0a800b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:45:41.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.615+0000 7fd0af400700 1 -- 172.21.15.62:0/2121418084 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd0a8011800 con 0x7fd0b8073070 2024-04-20T21:45:41.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.615+0000 7fd0af400700 1 -- 172.21.15.62:0/2121418084 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd0a8011e20 con 0x7fd0b8073070 2024-04-20T21:45:41.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.615+0000 7fd0af400700 1 -- 172.21.15.62:0/2121418084 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd0a800f2e0 con 0x7fd0b8073070 2024-04-20T21:45:41.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.615+0000 7fd0bd800700 1 -- 172.21.15.62:0/2121418084 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0b819ce40 con 0x7fd0b8073070 2024-04-20T21:45:41.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.615+0000 7fd0bd800700 1 -- 172.21.15.62:0/2121418084 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0b819d360 con 0x7fd0b8073070 2024-04-20T21:45:41.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.616+0000 7fd0bd800700 1 -- 172.21.15.62:0/2121418084 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0b8109d00 con 0x7fd0b8073070 2024-04-20T21:45:41.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.617+0000 7fd0af400700 1 -- 172.21.15.62:0/2121418084 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fd0a8010380 con 0x7fd0b8073070 2024-04-20T21:45:41.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.619+0000 7fd0af400700 1 --2- 172.21.15.62:0/2121418084 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd09c06ecc0 0x7fd09c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:45:41.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.619+0000 7fd0af400700 1 -- 172.21.15.62:0/2121418084 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fd0a808ea30 con 0x7fd0b8073070 2024-04-20T21:45:41.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.619+0000 7fd0b6000700 1 --2- 172.21.15.62:0/2121418084 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd09c06ecc0 0x7fd09c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:45:41.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.620+0000 7fd0b6000700 1 --2- 172.21.15.62:0/2121418084 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd09c06ecc0 0x7fd09c071170 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fd0a4006010 tx=0x7fd0a4005a90 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:45:41.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.624+0000 7fd0af400700 1 -- 172.21.15.62:0/2121418084 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fd0a8057870 con 0x7fd0b8073070 2024-04-20T21:45:41.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.954+0000 7fd0bd800700 1 -- 172.21.15.62:0/2121418084 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fd0b8199660 con 0x7fd0b8073070 2024-04-20T21:45:41.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.955+0000 7fd0af400700 1 -- 172.21.15.62:0/2121418084 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7fd0b8199660 con 0x7fd0b8073070 2024-04-20T21:45:41.955 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:45:41.955 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T21:45:41.955 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:45:41.955 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:45:41.955 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T21:45:41.955 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:45:41.955 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:45:41.955 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T21:45:41.955 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 6 2024-04-20T21:45:41.955 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:45:41.955 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T21:45:41.955 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 2, 2024-04-20T21:45:41.956 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:45:41.956 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:45:41.956 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T21:45:41.956 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 2, 2024-04-20T21:45:41.956 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 12 2024-04-20T21:45:41.956 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T21:45:41.956 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:45:41.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.959+0000 7fd0bd800700 1 -- 172.21.15.62:0/2121418084 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd09c06ecc0 msgr2=0x7fd09c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:45:41.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.959+0000 7fd0bd800700 1 --2- 172.21.15.62:0/2121418084 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd09c06ecc0 0x7fd09c071170 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fd0a4006010 tx=0x7fd0a4005a90 comp rx=0 tx=0).stop 2024-04-20T21:45:41.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.959+0000 7fd0bd800700 1 -- 172.21.15.62:0/2121418084 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd0b8073070 msgr2=0x7fd0b81981b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:45:41.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.959+0000 7fd0bd800700 1 --2- 172.21.15.62:0/2121418084 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd0b8073070 0x7fd0b81981b0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fd0a800b4b0 tx=0x7fd0a800b870 comp rx=0 tx=0).stop 2024-04-20T21:45:41.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.959+0000 7fd0bd800700 1 -- 172.21.15.62:0/2121418084 shutdown_connections 2024-04-20T21:45:41.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.959+0000 7fd0bd800700 1 --2- 172.21.15.62:0/2121418084 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd09c06ecc0 0x7fd09c071170 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:45:41.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.959+0000 7fd0bd800700 1 --2- 172.21.15.62:0/2121418084 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd0b8073070 0x7fd0b81981b0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:45:41.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.959+0000 7fd0bd800700 1 --2- 172.21.15.62:0/2121418084 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd0b8073980 0x7fd0b81986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:45:41.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.959+0000 7fd0bd800700 1 -- 172.21.15.62:0/2121418084 >> 172.21.15.62:0/2121418084 conn(0x7fd0b80fbda0 msgr2=0x7fd0b8106e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:45:41.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.959+0000 7fd0bd800700 1 -- 172.21.15.62:0/2121418084 shutdown_connections 2024-04-20T21:45:41.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:41.959+0000 7fd0bd800700 1 -- 172.21.15.62:0/2121418084 wait complete. 2024-04-20T21:45:42.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:42.128+0000 7f0a2ce00700 1 -- 172.21.15.62:0/323471949 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a28068540 msgr2=0x7f0a280689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:45:42.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:42.128+0000 7f0a2ce00700 1 --2- 172.21.15.62:0/323471949 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a28068540 0x7f0a280689b0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f0a14009910 tx=0x7f0a14009c20 comp rx=0 tx=0).stop 2024-04-20T21:45:42.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:42.129+0000 7f0a2ce00700 1 -- 172.21.15.62:0/323471949 shutdown_connections 2024-04-20T21:45:42.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:42.129+0000 7f0a2ce00700 1 --2- 172.21.15.62:0/323471949 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a28068540 0x7f0a280689b0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:45:42.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:42.129+0000 7f0a2ce00700 1 --2- 172.21.15.62:0/323471949 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a28106320 0x7f0a281066f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:45:42.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:42.129+0000 7f0a2ce00700 1 -- 172.21.15.62:0/323471949 >> 172.21.15.62:0/323471949 conn(0x7f0a28075130 msgr2=0x7f0a28075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:45:42.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:42.129+0000 7f0a2ce00700 1 -- 172.21.15.62:0/323471949 shutdown_connections 2024-04-20T21:45:42.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:42.129+0000 7f0a2ce00700 1 -- 172.21.15.62:0/323471949 wait complete. 2024-04-20T21:45:42.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:42.130+0000 7f0a2ce00700 1 Processor -- start 2024-04-20T21:45:42.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:42.130+0000 7f0a2ce00700 1 -- start start 2024-04-20T21:45:42.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:42.130+0000 7f0a2ce00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a28068540 0x7f0a281980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:45:42.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:42.130+0000 7f0a2ce00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a28106320 0x7f0a281985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:45:42.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:42.131+0000 7f0a2ce00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a28198cc0 con 0x7f0a28106320 2024-04-20T21:45:42.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:42.131+0000 7f0a2ce00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a2819ca50 con 0x7f0a28068540 2024-04-20T21:45:42.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:42.131+0000 7f0a25400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a28106320 0x7f0a281985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:45:42.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:42.131+0000 7f0a25400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a28106320 0x7f0a281985e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53590/0 (socket says 172.21.15.62:53590) 2024-04-20T21:45:42.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:42.131+0000 7f0a25400700 1 -- 172.21.15.62:0/3150731640 learned_addr learned my addr 172.21.15.62:0/3150731640 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:45:42.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:42.131+0000 7f0a25e00700 1 --2- 172.21.15.62:0/3150731640 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a28068540 0x7f0a281980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:45:42.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:42.131+0000 7f0a25400700 1 -- 172.21.15.62:0/3150731640 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a28068540 msgr2=0x7f0a281980a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:45:42.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:42.131+0000 7f0a25400700 1 --2- 172.21.15.62:0/3150731640 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a28068540 0x7f0a281980a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:45:42.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:42.131+0000 7f0a25400700 1 -- 172.21.15.62:0/3150731640 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a140095f0 con 0x7f0a28106320 2024-04-20T21:45:42.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:42.132+0000 7f0a25400700 1 --2- 172.21.15.62:0/3150731640 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a28106320 0x7f0a281985e0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f0a1400b5c0 tx=0x7f0a1400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:45:42.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:42.132+0000 7f0a1ea00700 1 -- 172.21.15.62:0/3150731640 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0a1401d070 con 0x7f0a28106320 2024-04-20T21:45:42.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:42.132+0000 7f0a2ce00700 1 -- 172.21.15.62:0/3150731640 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a2819ccd0 con 0x7f0a28106320 2024-04-20T21:45:42.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:42.132+0000 7f0a1ea00700 1 -- 172.21.15.62:0/3150731640 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0a14005e00 con 0x7f0a28106320 2024-04-20T21:45:42.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:42.132+0000 7f0a1ea00700 1 -- 172.21.15.62:0/3150731640 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0a14017400 con 0x7f0a28106320 2024-04-20T21:45:42.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:42.132+0000 7f0a2ce00700 1 -- 172.21.15.62:0/3150731640 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a2819d1c0 con 0x7f0a28106320 2024-04-20T21:45:42.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:42.133+0000 7f0a2ce00700 1 -- 172.21.15.62:0/3150731640 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a2804f2a0 con 0x7f0a28106320 2024-04-20T21:45:42.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:42.134+0000 7f0a1ea00700 1 -- 172.21.15.62:0/3150731640 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f0a1400f460 con 0x7f0a28106320 2024-04-20T21:45:42.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:42.135+0000 7f0a1ea00700 1 --2- 172.21.15.62:0/3150731640 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f0a0c06ecc0 0x7f0a0c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:45:42.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:42.136+0000 7f0a1ea00700 1 -- 172.21.15.62:0/3150731640 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f0a14090480 con 0x7f0a28106320 2024-04-20T21:45:42.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:42.136+0000 7f0a25e00700 1 --2- 172.21.15.62:0/3150731640 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f0a0c06ecc0 0x7f0a0c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:45:42.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:42.137+0000 7f0a25e00700 1 --2- 172.21.15.62:0/3150731640 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f0a0c06ecc0 0x7f0a0c071170 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f0a18005950 tx=0x7f0a180058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:45:42.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:42.141+0000 7f0a1ea00700 1 -- 172.21.15.62:0/3150731640 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f0a14059190 con 0x7f0a28106320 2024-04-20T21:45:42.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:45:42 smithi062 ceph-mon[76322]: from='client.34624 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:45:42.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:45:42 smithi062 ceph-mon[76322]: from='client.34628 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:45:42.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:45:42 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/2121418084' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:45:42.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:42.434+0000 7f0a2ce00700 1 -- 172.21.15.62:0/3150731640 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f0a2819d4a0 con 0x7f0a28106320 2024-04-20T21:45:42.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:42.435+0000 7f0a1ea00700 1 -- 172.21.15.62:0/3150731640 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 25 v25) v1 ==== 76+0+1887 (secure 0 0 0) 0x7f0a1405c850 con 0x7f0a28106320 2024-04-20T21:45:42.435 INFO:teuthology.orchestra.run.smithi062.stdout:e25 2024-04-20T21:45:42.435 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T21:45:42.435 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T21:45:42.435 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T21:45:42.435 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:45:42.435 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T21:45:42.435 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T21:45:42.435 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 20 2024-04-20T21:45:42.435 INFO:teuthology.orchestra.run.smithi062.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-20T21:45:42.435 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T21:45:42.435 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T21:42:01.559061+0000 2024-04-20T21:45:42.435 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T21:45:42.436 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T21:45:42.436 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T21:45:42.436 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T21:45:42.436 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T21:45:42.436 INFO:teuthology.orchestra.run.smithi062.stdout:max_xattr_size 65536 2024-04-20T21:45:42.436 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T21:45:42.436 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T21:45:42.436 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 81 2024-04-20T21:45:42.436 INFO:teuthology.orchestra.run.smithi062.stdout:snap_rstat 1 2024-04-20T21:45:42.436 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T21:45:42.436 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 2 2024-04-20T21:45:42.436 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1 2024-04-20T21:45:42.436 INFO:teuthology.orchestra.run.smithi062.stdout:up {} 2024-04-20T21:45:42.436 INFO:teuthology.orchestra.run.smithi062.stdout:failed 0,1 2024-04-20T21:45:42.436 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T21:45:42.437 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T21:45:42.437 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T21:45:42.437 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T21:45:42.437 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data enabled 2024-04-20T21:45:42.437 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T21:45:42.437 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T21:45:42.437 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 1 2024-04-20T21:45:42.437 INFO:teuthology.orchestra.run.smithi062.stdout:qdb_cluster leader: 0 members: 2024-04-20T21:45:42.437 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:45:42.437 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:45:42.437 INFO:teuthology.orchestra.run.smithi062.stdout:Standby daemons: 2024-04-20T21:45:42.437 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:45:42.437 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ryqblb{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6826/4113916205,v1:172.21.15.62:6827/4113916205] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T21:45:42.437 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ozizwi{-1:34476} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6828/3801975165,v1:172.21.15.62:6829/3801975165] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T21:45:42.437 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.xbgspw{-1:44285} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6826/3996040526,v1:172.21.15.189:6827/3996040526] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:45:42.438 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.dvimer{-1:44289} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6824/1743111843,v1:172.21.15.189:6825/1743111843] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:45:42.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:42.438+0000 7f0a2ce00700 1 -- 172.21.15.62:0/3150731640 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f0a0c06ecc0 msgr2=0x7f0a0c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:45:42.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:42.438+0000 7f0a2ce00700 1 --2- 172.21.15.62:0/3150731640 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f0a0c06ecc0 0x7f0a0c071170 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f0a18005950 tx=0x7f0a180058e0 comp rx=0 tx=0).stop 2024-04-20T21:45:42.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:42.438+0000 7f0a2ce00700 1 -- 172.21.15.62:0/3150731640 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a28106320 msgr2=0x7f0a281985e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:45:42.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:42.438+0000 7f0a2ce00700 1 --2- 172.21.15.62:0/3150731640 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a28106320 0x7f0a281985e0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f0a1400b5c0 tx=0x7f0a1400be20 comp rx=0 tx=0).stop 2024-04-20T21:45:42.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:42.438+0000 7f0a2ce00700 1 -- 172.21.15.62:0/3150731640 shutdown_connections 2024-04-20T21:45:42.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:42.438+0000 7f0a2ce00700 1 --2- 172.21.15.62:0/3150731640 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f0a0c06ecc0 0x7f0a0c071170 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:45:42.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:42.438+0000 7f0a2ce00700 1 --2- 172.21.15.62:0/3150731640 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a28068540 0x7f0a281980a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:45:42.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:42.438+0000 7f0a2ce00700 1 --2- 172.21.15.62:0/3150731640 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a28106320 0x7f0a281985e0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:45:42.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:42.438+0000 7f0a2ce00700 1 -- 172.21.15.62:0/3150731640 >> 172.21.15.62:0/3150731640 conn(0x7f0a28075130 msgr2=0x7f0a280fe9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:45:42.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:42.438+0000 7f0a2ce00700 1 -- 172.21.15.62:0/3150731640 shutdown_connections 2024-04-20T21:45:42.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:42.439+0000 7f0a2ce00700 1 -- 172.21.15.62:0/3150731640 wait complete. 2024-04-20T21:45:42.439 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 25 2024-04-20T21:45:42.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:45:42 smithi189 ceph-mon[57022]: from='client.34624 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:45:42.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:45:42 smithi189 ceph-mon[57022]: from='client.34628 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:45:42.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:45:42 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/2121418084' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:45:42.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:42.604+0000 7f145a000700 1 -- 172.21.15.62:0/2800109745 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1454102470 msgr2=0x7f14541028e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:45:42.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:42.604+0000 7f145a000700 1 --2- 172.21.15.62:0/2800109745 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1454102470 0x7f14541028e0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f1440009910 tx=0x7f1440009c20 comp rx=0 tx=0).stop 2024-04-20T21:45:42.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:42.604+0000 7f145a000700 1 -- 172.21.15.62:0/2800109745 shutdown_connections 2024-04-20T21:45:42.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:42.604+0000 7f145a000700 1 --2- 172.21.15.62:0/2800109745 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1454102470 0x7f14541028e0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:45:42.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:42.604+0000 7f145a000700 1 --2- 172.21.15.62:0/2800109745 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1454108470 0x7f1454108840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:45:42.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:42.604+0000 7f145a000700 1 -- 172.21.15.62:0/2800109745 >> 172.21.15.62:0/2800109745 conn(0x7f14540fdf90 msgr2=0x7f14541003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:45:42.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:42.604+0000 7f145a000700 1 -- 172.21.15.62:0/2800109745 shutdown_connections 2024-04-20T21:45:42.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:42.604+0000 7f145a000700 1 -- 172.21.15.62:0/2800109745 wait complete. 2024-04-20T21:45:42.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:42.605+0000 7f145a000700 1 Processor -- start 2024-04-20T21:45:42.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:42.605+0000 7f145a000700 1 -- start start 2024-04-20T21:45:42.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:42.605+0000 7f145a000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1454102470 0x7f14541981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:45:42.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:42.606+0000 7f145a000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1454108470 0x7f1454198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:45:42.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:42.606+0000 7f145a000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1454198d80 con 0x7f1454102470 2024-04-20T21:45:42.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:42.606+0000 7f145a000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1454198ec0 con 0x7f1454108470 2024-04-20T21:45:42.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:42.606+0000 7f1453400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1454102470 0x7f14541981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:45:42.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:42.606+0000 7f1453400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1454102470 0x7f14541981f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53614/0 (socket says 172.21.15.62:53614) 2024-04-20T21:45:42.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:42.606+0000 7f1453400700 1 -- 172.21.15.62:0/1065418418 learned_addr learned my addr 172.21.15.62:0/1065418418 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:45:42.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:42.606+0000 7f1452a00700 1 --2- 172.21.15.62:0/1065418418 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1454108470 0x7f1454198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:45:42.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:42.606+0000 7f1453400700 1 -- 172.21.15.62:0/1065418418 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1454108470 msgr2=0x7f1454198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:45:42.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:42.606+0000 7f1453400700 1 --2- 172.21.15.62:0/1065418418 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1454108470 0x7f1454198730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:45:42.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:42.606+0000 7f1453400700 1 -- 172.21.15.62:0/1065418418 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14400095f0 con 0x7f1454102470 2024-04-20T21:45:42.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:42.607+0000 7f1453400700 1 --2- 172.21.15.62:0/1065418418 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1454102470 0x7f14541981f0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f144400c930 tx=0x7f144400ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:45:42.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:42.607+0000 7f144be00700 1 -- 172.21.15.62:0/1065418418 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1444007ad0 con 0x7f1454102470 2024-04-20T21:45:42.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:42.607+0000 7f144be00700 1 -- 172.21.15.62:0/1065418418 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1444007c30 con 0x7f1454102470 2024-04-20T21:45:42.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:42.607+0000 7f145a000700 1 -- 172.21.15.62:0/1065418418 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f145419cd10 con 0x7f1454102470 2024-04-20T21:45:42.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:42.607+0000 7f144be00700 1 -- 172.21.15.62:0/1065418418 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1444018610 con 0x7f1454102470 2024-04-20T21:45:42.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:42.607+0000 7f145a000700 1 -- 172.21.15.62:0/1065418418 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f145419d260 con 0x7f1454102470 2024-04-20T21:45:42.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:42.609+0000 7f145a000700 1 -- 172.21.15.62:0/1065418418 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f145410a970 con 0x7f1454102470 2024-04-20T21:45:42.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:42.609+0000 7f144be00700 1 -- 172.21.15.62:0/1065418418 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f144400f450 con 0x7f1454102470 2024-04-20T21:45:42.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:42.610+0000 7f144be00700 1 --2- 172.21.15.62:0/1065418418 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f143806ecc0 0x7f1438071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:45:42.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:42.610+0000 7f144be00700 1 -- 172.21.15.62:0/1065418418 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f144408f140 con 0x7f1454102470 2024-04-20T21:45:42.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:42.611+0000 7f1452a00700 1 --2- 172.21.15.62:0/1065418418 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f143806ecc0 0x7f1438071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:45:42.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:42.611+0000 7f1452a00700 1 --2- 172.21.15.62:0/1065418418 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f143806ecc0 0x7f1438071170 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f1440000c00 tx=0x7f1440005fd0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:45:42.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:42.615+0000 7f144be00700 1 -- 172.21.15.62:0/1065418418 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f1444057e50 con 0x7f1454102470 2024-04-20T21:45:42.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:42.879+0000 7f145a000700 1 -- 172.21.15.62:0/1065418418 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f145419d4e0 con 0x7f143806ecc0 2024-04-20T21:45:42.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:42.881+0000 7f144be00700 1 -- 172.21.15.62:0/1065418418 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f145419d4e0 con 0x7f143806ecc0 2024-04-20T21:45:42.881 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:45:42.881 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T21:45:42.881 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T21:45:42.881 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-20T21:45:42.881 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T21:45:42.881 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr", 2024-04-20T21:45:42.881 INFO:teuthology.orchestra.run.smithi062.stdout: "crash", 2024-04-20T21:45:42.881 INFO:teuthology.orchestra.run.smithi062.stdout: "mon", 2024-04-20T21:45:42.881 INFO:teuthology.orchestra.run.smithi062.stdout: "osd" 2024-04-20T21:45:42.881 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T21:45:42.881 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "14/23 daemons upgraded", 2024-04-20T21:45:42.881 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading mds daemons", 2024-04-20T21:45:42.881 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T21:45:42.881 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:45:42.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:42.884+0000 7f145a000700 1 -- 172.21.15.62:0/1065418418 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f143806ecc0 msgr2=0x7f1438071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:45:42.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:42.884+0000 7f145a000700 1 --2- 172.21.15.62:0/1065418418 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f143806ecc0 0x7f1438071170 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f1440000c00 tx=0x7f1440005fd0 comp rx=0 tx=0).stop 2024-04-20T21:45:42.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:42.884+0000 7f145a000700 1 -- 172.21.15.62:0/1065418418 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1454102470 msgr2=0x7f14541981f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:45:42.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:42.884+0000 7f145a000700 1 --2- 172.21.15.62:0/1065418418 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1454102470 0x7f14541981f0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f144400c930 tx=0x7f144400ccf0 comp rx=0 tx=0).stop 2024-04-20T21:45:42.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:42.885+0000 7f145a000700 1 -- 172.21.15.62:0/1065418418 shutdown_connections 2024-04-20T21:45:42.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:42.885+0000 7f145a000700 1 --2- 172.21.15.62:0/1065418418 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f143806ecc0 0x7f1438071170 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:45:42.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:42.885+0000 7f145a000700 1 --2- 172.21.15.62:0/1065418418 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1454102470 0x7f14541981f0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:45:42.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:42.885+0000 7f145a000700 1 --2- 172.21.15.62:0/1065418418 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1454108470 0x7f1454198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:45:42.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:42.885+0000 7f145a000700 1 -- 172.21.15.62:0/1065418418 >> 172.21.15.62:0/1065418418 conn(0x7f14540fdf90 msgr2=0x7f14540ff3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:45:42.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:42.885+0000 7f145a000700 1 -- 172.21.15.62:0/1065418418 shutdown_connections 2024-04-20T21:45:42.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:42.885+0000 7f145a000700 1 -- 172.21.15.62:0/1065418418 wait complete. 2024-04-20T21:45:43.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:43.049+0000 7f2887000700 1 -- 172.21.15.62:0/1690542620 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f288010c660 msgr2=0x7f288010ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:45:43.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:43.049+0000 7f2887000700 1 --2- 172.21.15.62:0/1690542620 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f288010c660 0x7f288010ca30 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f2874009910 tx=0x7f2874009c20 comp rx=0 tx=0).stop 2024-04-20T21:45:43.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:43.050+0000 7f2887000700 1 -- 172.21.15.62:0/1690542620 shutdown_connections 2024-04-20T21:45:43.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:43.050+0000 7f2887000700 1 --2- 172.21.15.62:0/1690542620 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2880106b80 0x7f2880106ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:45:43.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:43.050+0000 7f2887000700 1 --2- 172.21.15.62:0/1690542620 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f288010c660 0x7f288010ca30 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:45:43.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:43.050+0000 7f2887000700 1 -- 172.21.15.62:0/1690542620 >> 172.21.15.62:0/1690542620 conn(0x7f2880075e90 msgr2=0x7f28800782a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:45:43.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:43.050+0000 7f2887000700 1 -- 172.21.15.62:0/1690542620 shutdown_connections 2024-04-20T21:45:43.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:43.050+0000 7f2887000700 1 -- 172.21.15.62:0/1690542620 wait complete. 2024-04-20T21:45:43.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:43.051+0000 7f2887000700 1 Processor -- start 2024-04-20T21:45:43.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:43.051+0000 7f2887000700 1 -- start start 2024-04-20T21:45:43.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:43.051+0000 7f2887000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2880106b80 0x7f288019c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:45:43.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:43.051+0000 7f2887000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f288010c660 0x7f288019c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:45:43.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:43.051+0000 7f2887000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f288019cea0 con 0x7f288010c660 2024-04-20T21:45:43.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:43.051+0000 7f2887000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28801a0bc0 con 0x7f2880106b80 2024-04-20T21:45:43.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:43.052+0000 7f2885200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f288010c660 0x7f288019c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:45:43.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:43.052+0000 7f2885200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f288010c660 0x7f288019c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53628/0 (socket says 172.21.15.62:53628) 2024-04-20T21:45:43.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:43.052+0000 7f2885200700 1 -- 172.21.15.62:0/3202029358 learned_addr learned my addr 172.21.15.62:0/3202029358 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:45:43.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:43.052+0000 7f2885c00700 1 --2- 172.21.15.62:0/3202029358 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2880106b80 0x7f288019c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:45:43.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:43.052+0000 7f2885200700 1 -- 172.21.15.62:0/3202029358 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2880106b80 msgr2=0x7f288019c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:45:43.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:43.052+0000 7f2885200700 1 --2- 172.21.15.62:0/3202029358 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2880106b80 0x7f288019c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:45:43.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:43.052+0000 7f2885200700 1 -- 172.21.15.62:0/3202029358 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28740095f0 con 0x7f288010c660 2024-04-20T21:45:43.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:43.053+0000 7f2885200700 1 --2- 172.21.15.62:0/3202029358 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f288010c660 0x7f288019c7c0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f286c00d8d0 tx=0x7f286c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:45:43.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:43.053+0000 7f287e000700 1 -- 172.21.15.62:0/3202029358 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f286c009940 con 0x7f288010c660 2024-04-20T21:45:43.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:43.053+0000 7f287e000700 1 -- 172.21.15.62:0/3202029358 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f286c010460 con 0x7f288010c660 2024-04-20T21:45:43.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:43.053+0000 7f287e000700 1 -- 172.21.15.62:0/3202029358 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f286c00f5f0 con 0x7f288010c660 2024-04-20T21:45:43.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:43.053+0000 7f2887000700 1 -- 172.21.15.62:0/3202029358 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28801a0ea0 con 0x7f288010c660 2024-04-20T21:45:43.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:43.053+0000 7f2887000700 1 -- 172.21.15.62:0/3202029358 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28801a13f0 con 0x7f288010c660 2024-04-20T21:45:43.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:43.054+0000 7f2887000700 1 -- 172.21.15.62:0/3202029358 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f288010eb40 con 0x7f288010c660 2024-04-20T21:45:43.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:43.055+0000 7f287e000700 1 -- 172.21.15.62:0/3202029358 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f286c0105d0 con 0x7f288010c660 2024-04-20T21:45:43.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:43.056+0000 7f287e000700 1 --2- 172.21.15.62:0/3202029358 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f286806ecc0 0x7f2868071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:45:43.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:43.056+0000 7f287e000700 1 -- 172.21.15.62:0/3202029358 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f286c08ee80 con 0x7f288010c660 2024-04-20T21:45:43.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:43.057+0000 7f2885c00700 1 --2- 172.21.15.62:0/3202029358 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f286806ecc0 0x7f2868071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:45:43.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:43.057+0000 7f2885c00700 1 --2- 172.21.15.62:0/3202029358 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f286806ecc0 0x7f2868071170 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f2874006010 tx=0x7f287401a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:45:43.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:43.061+0000 7f287e000700 1 -- 172.21.15.62:0/3202029358 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f286c057c30 con 0x7f288010c660 2024-04-20T21:45:43.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:45:43 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/3150731640' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T21:45:43.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:45:43 smithi189 ceph-mon[57022]: pgmap v312: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:45:43.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:45:43 smithi189 ceph-mon[57022]: from='client.34640 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:45:43.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:45:43 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/3150731640' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T21:45:43.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:45:43 smithi062 ceph-mon[76322]: pgmap v312: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:45:43.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:45:43 smithi062 ceph-mon[76322]: from='client.34640 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:45:43.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:43.372+0000 7f2887000700 1 -- 172.21.15.62:0/3202029358 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f28801a1610 con 0x7f288010c660 2024-04-20T21:45:43.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:43.372+0000 7f287e000700 1 -- 172.21.15.62:0/3202029358 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7f286c05b2f0 con 0x7f288010c660 2024-04-20T21:45:43.372 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-20T21:45:43.372 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-20T21:45:43.372 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is degraded 2024-04-20T21:45:43.372 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T21:45:43.372 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T21:45:43.372 INFO:teuthology.orchestra.run.smithi062.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-20T21:45:43.372 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-20T21:45:43.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:43.375+0000 7f2887000700 1 -- 172.21.15.62:0/3202029358 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f286806ecc0 msgr2=0x7f2868071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:45:43.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:43.375+0000 7f2887000700 1 --2- 172.21.15.62:0/3202029358 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f286806ecc0 0x7f2868071170 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f2874006010 tx=0x7f287401a040 comp rx=0 tx=0).stop 2024-04-20T21:45:43.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:43.375+0000 7f2887000700 1 -- 172.21.15.62:0/3202029358 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f288010c660 msgr2=0x7f288019c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:45:43.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:43.375+0000 7f2887000700 1 --2- 172.21.15.62:0/3202029358 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f288010c660 0x7f288019c7c0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f286c00d8d0 tx=0x7f286c00dc90 comp rx=0 tx=0).stop 2024-04-20T21:45:43.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:43.375+0000 7f2887000700 1 -- 172.21.15.62:0/3202029358 shutdown_connections 2024-04-20T21:45:43.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:43.375+0000 7f2887000700 1 --2- 172.21.15.62:0/3202029358 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f286806ecc0 0x7f2868071170 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:45:43.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:43.375+0000 7f2887000700 1 --2- 172.21.15.62:0/3202029358 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2880106b80 0x7f288019c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:45:43.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:43.375+0000 7f2887000700 1 --2- 172.21.15.62:0/3202029358 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f288010c660 0x7f288019c7c0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:45:43.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:43.375+0000 7f2887000700 1 -- 172.21.15.62:0/3202029358 >> 172.21.15.62:0/3202029358 conn(0x7f2880075e90 msgr2=0x7f2880077980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:45:43.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:43.376+0000 7f2887000700 1 -- 172.21.15.62:0/3202029358 shutdown_connections 2024-04-20T21:45:43.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:45:43.376+0000 7f2887000700 1 -- 172.21.15.62:0/3202029358 wait complete. 2024-04-20T21:45:44.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:45:44 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/3202029358' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T21:45:44.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:45:44 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/3202029358' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T21:45:45.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:45:45 smithi062 ceph-mon[76322]: pgmap v313: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:45:45.486 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:45:45 smithi189 ceph-mon[57022]: pgmap v313: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:45:47.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:45:47 smithi062 ceph-mon[76322]: pgmap v314: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:45:47.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:45:47 smithi189 ceph-mon[57022]: pgmap v314: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:45:49.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:45:49 smithi062 ceph-mon[76322]: pgmap v315: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:45:49.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:45:49 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:45:49.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:45:49 smithi189 ceph-mon[57022]: pgmap v315: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:45:49.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:45:49 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:45:51.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:45:51 smithi189 ceph-mon[57022]: pgmap v316: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:45:52.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:45:51 smithi062 ceph-mon[76322]: pgmap v316: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:45:53.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:45:53 smithi189 ceph-mon[57022]: pgmap v317: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:45:54.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:45:53 smithi062 ceph-mon[76322]: pgmap v317: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:45:55.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:45:55 smithi189 ceph-mon[57022]: pgmap v318: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:45:56.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:45:55 smithi062 ceph-mon[76322]: pgmap v318: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:45:57.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:45:57 smithi189 ceph-mon[57022]: pgmap v319: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:45:58.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:45:57 smithi062 ceph-mon[76322]: pgmap v319: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:45:59.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:45:59 smithi189 ceph-mon[57022]: pgmap v320: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:46:00.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:45:59 smithi062 ceph-mon[76322]: pgmap v320: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:46:01.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:46:01 smithi189 ceph-mon[57022]: pgmap v321: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:46:02.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:46:01 smithi062 ceph-mon[76322]: pgmap v321: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:46:03.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:46:03 smithi189 ceph-mon[57022]: pgmap v322: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:46:04.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:46:03 smithi062 ceph-mon[76322]: pgmap v322: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:46:04.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:46:04 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:46:05.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:46:04 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:46:05.967 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:46:05 smithi062 ceph-mon[76322]: pgmap v323: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:46:05.986 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:46:05 smithi189 ceph-mon[57022]: pgmap v323: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:46:07.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:46:07 smithi189 ceph-mon[57022]: pgmap v324: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:46:08.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:46:07 smithi062 ceph-mon[76322]: pgmap v324: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:46:09.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:46:09 smithi189 ceph-mon[57022]: pgmap v325: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:46:10.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:46:09 smithi062 ceph-mon[76322]: pgmap v325: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:46:11.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:46:11 smithi189 ceph-mon[57022]: pgmap v326: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:46:12.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:46:11 smithi062 ceph-mon[76322]: pgmap v326: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:46:13.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:13.552+0000 7f223e600700 1 -- 172.21.15.62:0/3722786051 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f22380739a0 msgr2=0x7f223810c5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:46:13.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:13.552+0000 7f223e600700 1 --2- 172.21.15.62:0/3722786051 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f22380739a0 0x7f223810c5e0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f2224009940 tx=0x7f2224009c50 comp rx=0 tx=0).stop 2024-04-20T21:46:13.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:13.552+0000 7f223e600700 1 -- 172.21.15.62:0/3722786051 shutdown_connections 2024-04-20T21:46:13.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:13.552+0000 7f223e600700 1 --2- 172.21.15.62:0/3722786051 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f22380739a0 0x7f223810c5e0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:46:13.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:13.552+0000 7f223e600700 1 --2- 172.21.15.62:0/3722786051 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2238073090 0x7f2238073460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:46:13.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:13.552+0000 7f223e600700 1 -- 172.21.15.62:0/3722786051 >> 172.21.15.62:0/3722786051 conn(0x7f22380fbda0 msgr2=0x7f22380fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:46:13.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:13.553+0000 7f223e600700 1 -- 172.21.15.62:0/3722786051 shutdown_connections 2024-04-20T21:46:13.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:13.553+0000 7f223e600700 1 -- 172.21.15.62:0/3722786051 wait complete. 2024-04-20T21:46:13.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:13.554+0000 7f223e600700 1 Processor -- start 2024-04-20T21:46:13.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:13.554+0000 7f223e600700 1 -- start start 2024-04-20T21:46:13.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:13.555+0000 7f223e600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2238073090 0x7f2238198220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:46:13.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:13.555+0000 7f223e600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f22380739a0 0x7f2238198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:46:13.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:13.555+0000 7f223e600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2238198e40 con 0x7f2238073090 2024-04-20T21:46:13.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:13.555+0000 7f223e600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f223819cbd0 con 0x7f22380739a0 2024-04-20T21:46:13.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:13.555+0000 7f2237400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2238073090 0x7f2238198220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:46:13.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:13.555+0000 7f2237400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2238073090 0x7f2238198220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49976/0 (socket says 172.21.15.62:49976) 2024-04-20T21:46:13.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:13.555+0000 7f2237400700 1 -- 172.21.15.62:0/2041303236 learned_addr learned my addr 172.21.15.62:0/2041303236 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:46:13.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:13.555+0000 7f2236a00700 1 --2- 172.21.15.62:0/2041303236 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f22380739a0 0x7f2238198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:46:13.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:13.555+0000 7f2237400700 1 -- 172.21.15.62:0/2041303236 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f22380739a0 msgr2=0x7f2238198760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:46:13.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:13.555+0000 7f2237400700 1 --2- 172.21.15.62:0/2041303236 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f22380739a0 0x7f2238198760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:46:13.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:13.555+0000 7f2237400700 1 -- 172.21.15.62:0/2041303236 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22240095f0 con 0x7f2238073090 2024-04-20T21:46:13.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:13.556+0000 7f2237400700 1 --2- 172.21.15.62:0/2041303236 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2238073090 0x7f2238198220 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f222800c8a0 tx=0x7f222800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:46:13.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:13.556+0000 7f222fe00700 1 -- 172.21.15.62:0/2041303236 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f222800cea0 con 0x7f2238073090 2024-04-20T21:46:13.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:13.556+0000 7f222fe00700 1 -- 172.21.15.62:0/2041303236 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2228004830 con 0x7f2238073090 2024-04-20T21:46:13.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:13.556+0000 7f222fe00700 1 -- 172.21.15.62:0/2041303236 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f22280055b0 con 0x7f2238073090 2024-04-20T21:46:13.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:13.556+0000 7f223e600700 1 -- 172.21.15.62:0/2041303236 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f223819ceb0 con 0x7f2238073090 2024-04-20T21:46:13.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:13.556+0000 7f223e600700 1 -- 172.21.15.62:0/2041303236 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f223819d400 con 0x7f2238073090 2024-04-20T21:46:13.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:13.557+0000 7f223e600700 1 -- 172.21.15.62:0/2041303236 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2238109ce0 con 0x7f2238073090 2024-04-20T21:46:13.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:13.559+0000 7f222fe00700 1 -- 172.21.15.62:0/2041303236 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f22280049a0 con 0x7f2238073090 2024-04-20T21:46:13.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:13.560+0000 7f222fe00700 1 --2- 172.21.15.62:0/2041303236 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f221c06ecc0 0x7f221c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:46:13.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:13.560+0000 7f222fe00700 1 -- 172.21.15.62:0/2041303236 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f222808eaf0 con 0x7f2238073090 2024-04-20T21:46:13.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:13.560+0000 7f2236a00700 1 --2- 172.21.15.62:0/2041303236 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f221c06ecc0 0x7f221c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:46:13.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:13.561+0000 7f2236a00700 1 --2- 172.21.15.62:0/2041303236 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f221c06ecc0 0x7f221c071170 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f2238199840 tx=0x7f2224005c20 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:46:13.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:13.567+0000 7f222fe00700 1 -- 172.21.15.62:0/2041303236 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f2228057880 con 0x7f2238073090 2024-04-20T21:46:13.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:13.824+0000 7f223e600700 1 -- 172.21.15.62:0/2041303236 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f223819d6d0 con 0x7f221c06ecc0 2024-04-20T21:46:13.823 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:46:13 smithi062 ceph-mon[76322]: pgmap v327: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:46:13.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:13.826+0000 7f222fe00700 1 -- 172.21.15.62:0/2041303236 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f223819d6d0 con 0x7f221c06ecc0 2024-04-20T21:46:13.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:13.829+0000 7f223e600700 1 -- 172.21.15.62:0/2041303236 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f221c06ecc0 msgr2=0x7f221c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:46:13.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:13.829+0000 7f223e600700 1 --2- 172.21.15.62:0/2041303236 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f221c06ecc0 0x7f221c071170 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f2238199840 tx=0x7f2224005c20 comp rx=0 tx=0).stop 2024-04-20T21:46:13.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:13.829+0000 7f223e600700 1 -- 172.21.15.62:0/2041303236 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2238073090 msgr2=0x7f2238198220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:46:13.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:13.829+0000 7f223e600700 1 --2- 172.21.15.62:0/2041303236 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2238073090 0x7f2238198220 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f222800c8a0 tx=0x7f222800cc60 comp rx=0 tx=0).stop 2024-04-20T21:46:13.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:13.829+0000 7f223e600700 1 -- 172.21.15.62:0/2041303236 shutdown_connections 2024-04-20T21:46:13.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:13.829+0000 7f223e600700 1 --2- 172.21.15.62:0/2041303236 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f221c06ecc0 0x7f221c071170 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:46:13.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:13.829+0000 7f223e600700 1 --2- 172.21.15.62:0/2041303236 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2238073090 0x7f2238198220 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:46:13.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:13.829+0000 7f223e600700 1 --2- 172.21.15.62:0/2041303236 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f22380739a0 0x7f2238198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:46:13.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:13.829+0000 7f223e600700 1 -- 172.21.15.62:0/2041303236 >> 172.21.15.62:0/2041303236 conn(0x7f22380fbda0 msgr2=0x7f2238106e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:46:13.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:13.829+0000 7f223e600700 1 -- 172.21.15.62:0/2041303236 shutdown_connections 2024-04-20T21:46:13.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:13.829+0000 7f223e600700 1 -- 172.21.15.62:0/2041303236 wait complete. 2024-04-20T21:46:13.844 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T21:46:13.874 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:46:13 smithi189 ceph-mon[57022]: pgmap v327: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:46:13.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:13.994+0000 7f8383000700 1 -- 172.21.15.62:0/2241112739 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f837c10c8f0 msgr2=0x7f837c10ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:46:13.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:13.994+0000 7f8383000700 1 --2- 172.21.15.62:0/2241112739 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f837c10c8f0 0x7f837c10ccc0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f836c009910 tx=0x7f836c009c20 comp rx=0 tx=0).stop 2024-04-20T21:46:13.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:13.994+0000 7f8383000700 1 -- 172.21.15.62:0/2241112739 shutdown_connections 2024-04-20T21:46:13.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:13.994+0000 7f8383000700 1 --2- 172.21.15.62:0/2241112739 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f837c106b80 0x7f837c106ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:46:13.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:13.994+0000 7f8383000700 1 --2- 172.21.15.62:0/2241112739 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f837c10c8f0 0x7f837c10ccc0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:46:13.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:13.994+0000 7f8383000700 1 -- 172.21.15.62:0/2241112739 >> 172.21.15.62:0/2241112739 conn(0x7f837c076060 msgr2=0x7f837c078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:46:13.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:13.994+0000 7f8383000700 1 -- 172.21.15.62:0/2241112739 shutdown_connections 2024-04-20T21:46:13.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:13.994+0000 7f8383000700 1 -- 172.21.15.62:0/2241112739 wait complete. 2024-04-20T21:46:13.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:13.995+0000 7f8383000700 1 Processor -- start 2024-04-20T21:46:13.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:13.995+0000 7f8383000700 1 -- start start 2024-04-20T21:46:13.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:13.996+0000 7f8383000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f837c106b80 0x7f837c19c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:46:13.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:13.996+0000 7f8383000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f837c10c8f0 0x7f837c19ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:46:13.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:13.996+0000 7f8383000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f837c19d120 con 0x7f837c10c8f0 2024-04-20T21:46:13.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:13.996+0000 7f8383000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f837c1a0eb0 con 0x7f837c106b80 2024-04-20T21:46:13.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:13.996+0000 7f837b400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f837c10c8f0 0x7f837c19ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:46:13.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:13.996+0000 7f837b400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f837c10c8f0 0x7f837c19ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49994/0 (socket says 172.21.15.62:49994) 2024-04-20T21:46:13.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:13.996+0000 7f837b400700 1 -- 172.21.15.62:0/4162362666 learned_addr learned my addr 172.21.15.62:0/4162362666 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:46:13.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:13.997+0000 7f837be00700 1 --2- 172.21.15.62:0/4162362666 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f837c106b80 0x7f837c19c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:46:13.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:13.997+0000 7f837b400700 1 -- 172.21.15.62:0/4162362666 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f837c106b80 msgr2=0x7f837c19c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:46:13.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:13.997+0000 7f837b400700 1 --2- 172.21.15.62:0/4162362666 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f837c106b80 0x7f837c19c500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:46:13.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:13.997+0000 7f837b400700 1 -- 172.21.15.62:0/4162362666 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f836c0095f0 con 0x7f837c10c8f0 2024-04-20T21:46:13.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:13.997+0000 7f837b400700 1 --2- 172.21.15.62:0/4162362666 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f837c10c8f0 0x7f837c19ca40 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f836800d8d0 tx=0x7f836800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:46:13.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:13.997+0000 7f8378c00700 1 -- 172.21.15.62:0/4162362666 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8368009940 con 0x7f837c10c8f0 2024-04-20T21:46:13.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:13.997+0000 7f8378c00700 1 -- 172.21.15.62:0/4162362666 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8368010460 con 0x7f837c10c8f0 2024-04-20T21:46:13.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:13.997+0000 7f8378c00700 1 -- 172.21.15.62:0/4162362666 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f836800f5f0 con 0x7f837c10c8f0 2024-04-20T21:46:13.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:13.997+0000 7f8383000700 1 -- 172.21.15.62:0/4162362666 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f837c1a1190 con 0x7f837c10c8f0 2024-04-20T21:46:13.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:13.997+0000 7f8383000700 1 -- 172.21.15.62:0/4162362666 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f837c1a16e0 con 0x7f837c10c8f0 2024-04-20T21:46:13.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:13.999+0000 7f8383000700 1 -- 172.21.15.62:0/4162362666 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f837c109b20 con 0x7f837c10c8f0 2024-04-20T21:46:13.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:13.999+0000 7f8378c00700 1 -- 172.21.15.62:0/4162362666 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f83680105d0 con 0x7f837c10c8f0 2024-04-20T21:46:13.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.000+0000 7f8378c00700 1 --2- 172.21.15.62:0/4162362666 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f836006ecc0 0x7f8360071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:46:13.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.001+0000 7f8378c00700 1 -- 172.21.15.62:0/4162362666 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f836808ee10 con 0x7f837c10c8f0 2024-04-20T21:46:14.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.001+0000 7f837be00700 1 --2- 172.21.15.62:0/4162362666 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f836006ecc0 0x7f8360071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:46:14.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.001+0000 7f837be00700 1 --2- 172.21.15.62:0/4162362666 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f836006ecc0 0x7f8360071170 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f836c000c00 tx=0x7f836c005fd0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:46:14.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.005+0000 7f8378c00700 1 -- 172.21.15.62:0/4162362666 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f8368057ba0 con 0x7f837c10c8f0 2024-04-20T21:46:14.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.269+0000 7f8383000700 1 -- 172.21.15.62:0/4162362666 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f837c1a1950 con 0x7f836006ecc0 2024-04-20T21:46:14.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.273+0000 7f8378c00700 1 -- 172.21.15.62:0/4162362666 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f837c1a1950 con 0x7f836006ecc0 2024-04-20T21:46:14.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.276+0000 7f8383000700 1 -- 172.21.15.62:0/4162362666 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f836006ecc0 msgr2=0x7f8360071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:46:14.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.276+0000 7f8383000700 1 --2- 172.21.15.62:0/4162362666 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f836006ecc0 0x7f8360071170 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f836c000c00 tx=0x7f836c005fd0 comp rx=0 tx=0).stop 2024-04-20T21:46:14.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.276+0000 7f8383000700 1 -- 172.21.15.62:0/4162362666 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f837c10c8f0 msgr2=0x7f837c19ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:46:14.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.276+0000 7f8383000700 1 --2- 172.21.15.62:0/4162362666 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f837c10c8f0 0x7f837c19ca40 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f836800d8d0 tx=0x7f836800dc90 comp rx=0 tx=0).stop 2024-04-20T21:46:14.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.276+0000 7f8383000700 1 -- 172.21.15.62:0/4162362666 shutdown_connections 2024-04-20T21:46:14.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.276+0000 7f8383000700 1 --2- 172.21.15.62:0/4162362666 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f836006ecc0 0x7f8360071170 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:46:14.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.276+0000 7f8383000700 1 --2- 172.21.15.62:0/4162362666 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f837c106b80 0x7f837c19c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:46:14.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.276+0000 7f8383000700 1 --2- 172.21.15.62:0/4162362666 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f837c10c8f0 0x7f837c19ca40 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:46:14.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.276+0000 7f8383000700 1 -- 172.21.15.62:0/4162362666 >> 172.21.15.62:0/4162362666 conn(0x7f837c076060 msgr2=0x7f837c077aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:46:14.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.276+0000 7f8383000700 1 -- 172.21.15.62:0/4162362666 shutdown_connections 2024-04-20T21:46:14.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.277+0000 7f8383000700 1 -- 172.21.15.62:0/4162362666 wait complete. 2024-04-20T21:46:14.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.436+0000 7f73d4800700 1 -- 172.21.15.62:0/28955933 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f73cc1004e0 msgr2=0x7f73cc1008b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:46:14.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.436+0000 7f73d4800700 1 --2- 172.21.15.62:0/28955933 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f73cc1004e0 0x7f73cc1008b0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f73c0009910 tx=0x7f73c0009c20 comp rx=0 tx=0).stop 2024-04-20T21:46:14.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.436+0000 7f73d4800700 1 -- 172.21.15.62:0/28955933 shutdown_connections 2024-04-20T21:46:14.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.436+0000 7f73d4800700 1 --2- 172.21.15.62:0/28955933 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f73cc100e80 0x7f73cc104e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:46:14.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.436+0000 7f73d4800700 1 --2- 172.21.15.62:0/28955933 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f73cc1004e0 0x7f73cc1008b0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:46:14.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.436+0000 7f73d4800700 1 -- 172.21.15.62:0/28955933 >> 172.21.15.62:0/28955933 conn(0x7f73cc0fbda0 msgr2=0x7f73cc0fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:46:14.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.436+0000 7f73d4800700 1 -- 172.21.15.62:0/28955933 shutdown_connections 2024-04-20T21:46:14.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.436+0000 7f73d4800700 1 -- 172.21.15.62:0/28955933 wait complete. 2024-04-20T21:46:14.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.437+0000 7f73d4800700 1 Processor -- start 2024-04-20T21:46:14.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.437+0000 7f73d4800700 1 -- start start 2024-04-20T21:46:14.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.438+0000 7f73d4800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f73cc1004e0 0x7f73cc195ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:46:14.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.438+0000 7f73d4800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f73cc100e80 0x7f73cc196430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:46:14.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.438+0000 7f73d4800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73cc196b10 con 0x7f73cc1004e0 2024-04-20T21:46:14.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.438+0000 7f73d4800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73cc19a8a0 con 0x7f73cc100e80 2024-04-20T21:46:14.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.438+0000 7f73d1e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f73cc1004e0 0x7f73cc195ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:46:14.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.438+0000 7f73d1e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f73cc1004e0 0x7f73cc195ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50004/0 (socket says 172.21.15.62:50004) 2024-04-20T21:46:14.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.438+0000 7f73d1e00700 1 -- 172.21.15.62:0/656907058 learned_addr learned my addr 172.21.15.62:0/656907058 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:46:14.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.438+0000 7f73d1400700 1 --2- 172.21.15.62:0/656907058 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f73cc100e80 0x7f73cc196430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:46:14.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.439+0000 7f73d1e00700 1 -- 172.21.15.62:0/656907058 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f73cc100e80 msgr2=0x7f73cc196430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:46:14.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.439+0000 7f73d1e00700 1 --2- 172.21.15.62:0/656907058 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f73cc100e80 0x7f73cc196430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:46:14.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.439+0000 7f73d1e00700 1 -- 172.21.15.62:0/656907058 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73c00095f0 con 0x7f73cc1004e0 2024-04-20T21:46:14.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.439+0000 7f73d1400700 1 --2- 172.21.15.62:0/656907058 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f73cc100e80 0x7f73cc196430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T21:46:14.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.439+0000 7f73d1e00700 1 --2- 172.21.15.62:0/656907058 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f73cc1004e0 0x7f73cc195ef0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f73c000b5c0 tx=0x7f73c000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:46:14.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.439+0000 7f73c6a00700 1 -- 172.21.15.62:0/656907058 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f73c001d070 con 0x7f73cc1004e0 2024-04-20T21:46:14.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.439+0000 7f73c6a00700 1 -- 172.21.15.62:0/656907058 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f73c0005e00 con 0x7f73cc1004e0 2024-04-20T21:46:14.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.439+0000 7f73d4800700 1 -- 172.21.15.62:0/656907058 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73cc19ab20 con 0x7f73cc1004e0 2024-04-20T21:46:14.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.439+0000 7f73c6a00700 1 -- 172.21.15.62:0/656907058 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f73c0017400 con 0x7f73cc1004e0 2024-04-20T21:46:14.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.440+0000 7f73d4800700 1 -- 172.21.15.62:0/656907058 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73cc19b010 con 0x7f73cc1004e0 2024-04-20T21:46:14.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.440+0000 7f73d4800700 1 -- 172.21.15.62:0/656907058 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73cc1087f0 con 0x7f73cc1004e0 2024-04-20T21:46:14.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.442+0000 7f73c6a00700 1 -- 172.21.15.62:0/656907058 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f73c000f460 con 0x7f73cc1004e0 2024-04-20T21:46:14.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.443+0000 7f73c6a00700 1 --2- 172.21.15.62:0/656907058 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f73b406ecc0 0x7f73b4071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:46:14.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.443+0000 7f73c6a00700 1 -- 172.21.15.62:0/656907058 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f73c0090440 con 0x7f73cc1004e0 2024-04-20T21:46:14.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.443+0000 7f73d1400700 1 --2- 172.21.15.62:0/656907058 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f73b406ecc0 0x7f73b4071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:46:14.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.444+0000 7f73d1400700 1 --2- 172.21.15.62:0/656907058 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f73b406ecc0 0x7f73b4071170 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f73bc005d90 tx=0x7f73bc005d00 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:46:14.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.448+0000 7f73c6a00700 1 -- 172.21.15.62:0/656907058 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f73c0059150 con 0x7f73cc1004e0 2024-04-20T21:46:14.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.686+0000 7f73d4800700 1 -- 172.21.15.62:0/656907058 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f73cc19b2f0 con 0x7f73b406ecc0 2024-04-20T21:46:14.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.720+0000 7f73c6a00700 1 -- 172.21.15.62:0/656907058 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f73cc19b2f0 con 0x7f73b406ecc0 2024-04-20T21:46:14.719 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T21:46:14.720 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (12m) 3m ago 25m 14.2M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T21:46:14.720 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (26m) 3m ago 26m 9512k - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T21:46:14.720 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (24m) 4m ago 24m 8942k - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T21:46:14.720 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (8m) 3m ago 26m 6660k - 19.0.0-3133-ga8c84f59 76bc6b91b726 2825d5ba11c6 2024-04-20T21:46:14.720 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (8m) 4m ago 24m 6665k - 19.0.0-3133-ga8c84f59 76bc6b91b726 20f746fa1bba 2024-04-20T21:46:14.720 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (11m) 3m ago 25m 74.2M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T21:46:14.720 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (3m) 3m ago 18m 11.2M - 19.0.0-3133-ga8c84f59 76bc6b91b726 de80bc90a683 2024-04-20T21:46:14.720 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (4m) 3m ago 18m 11.9M - 19.0.0-3133-ga8c84f59 76bc6b91b726 cd793b7ee98e 2024-04-20T21:46:14.720 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (18m) 4m ago 18m 117M - 18.2.0 dc2bc1663786 fb2a8a223db8 2024-04-20T21:46:14.720 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (18m) 4m ago 18m 55.6M - 18.2.0 dc2bc1663786 f2eec2ee8951 2024-04-20T21:46:14.720 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (14m) 3m ago 27m 511M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T21:46:14.720 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (13m) 4m ago 24m 452M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T21:46:14.721 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (9m) 3m ago 27m 55.9M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 019eb1479485 2024-04-20T21:46:14.721 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (9m) 4m ago 24m 39.8M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 55aac560aab6 2024-04-20T21:46:14.721 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (13m) 3m ago 26m 7952k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T21:46:14.721 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (13m) 4m ago 24m 8304k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T21:46:14.721 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (7m) 3m ago 23m 88.0M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 ee2bb901f26e 2024-04-20T21:46:14.721 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (6m) 3m ago 22m 92.3M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 9075152d4162 2024-04-20T21:46:14.721 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (6m) 3m ago 22m 74.4M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 2d4f3e38dc0e 2024-04-20T21:46:14.721 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (5m) 4m ago 21m 87.6M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 8f97b17ae0eb 2024-04-20T21:46:14.721 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (5m) 4m ago 21m 90.7M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 653d8079967d 2024-04-20T21:46:14.721 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (4m) 4m ago 20m 11.5M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 596dded14c12 2024-04-20T21:46:14.721 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (12m) 3m ago 25m 74.0M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T21:46:14.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.722+0000 7f73d4800700 1 -- 172.21.15.62:0/656907058 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f73b406ecc0 msgr2=0x7f73b4071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:46:14.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.722+0000 7f73d4800700 1 --2- 172.21.15.62:0/656907058 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f73b406ecc0 0x7f73b4071170 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f73bc005d90 tx=0x7f73bc005d00 comp rx=0 tx=0).stop 2024-04-20T21:46:14.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.722+0000 7f73d4800700 1 -- 172.21.15.62:0/656907058 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f73cc1004e0 msgr2=0x7f73cc195ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:46:14.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.722+0000 7f73d4800700 1 --2- 172.21.15.62:0/656907058 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f73cc1004e0 0x7f73cc195ef0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f73c000b5c0 tx=0x7f73c000be20 comp rx=0 tx=0).stop 2024-04-20T21:46:14.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.723+0000 7f73d4800700 1 -- 172.21.15.62:0/656907058 shutdown_connections 2024-04-20T21:46:14.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.723+0000 7f73d4800700 1 --2- 172.21.15.62:0/656907058 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f73b406ecc0 0x7f73b4071170 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:46:14.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.723+0000 7f73d4800700 1 --2- 172.21.15.62:0/656907058 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f73cc1004e0 0x7f73cc195ef0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:46:14.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.723+0000 7f73d4800700 1 --2- 172.21.15.62:0/656907058 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f73cc100e80 0x7f73cc196430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:46:14.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.723+0000 7f73d4800700 1 -- 172.21.15.62:0/656907058 >> 172.21.15.62:0/656907058 conn(0x7f73cc0fbda0 msgr2=0x7f73cc0fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:46:14.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.723+0000 7f73d4800700 1 -- 172.21.15.62:0/656907058 shutdown_connections 2024-04-20T21:46:14.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.723+0000 7f73d4800700 1 -- 172.21.15.62:0/656907058 wait complete. 2024-04-20T21:46:14.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.881+0000 7ff26f000700 1 -- 172.21.15.62:0/2084825286 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff268108470 msgr2=0x7ff268108840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:46:14.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.881+0000 7ff26f000700 1 --2- 172.21.15.62:0/2084825286 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff268108470 0x7ff268108840 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7ff258009910 tx=0x7ff258009c20 comp rx=0 tx=0).stop 2024-04-20T21:46:14.881 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:46:14 smithi062 ceph-mon[76322]: from='client.34648 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:46:14.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.881+0000 7ff26f000700 1 -- 172.21.15.62:0/2084825286 shutdown_connections 2024-04-20T21:46:14.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.881+0000 7ff26f000700 1 --2- 172.21.15.62:0/2084825286 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff268102470 0x7ff2681028e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:46:14.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.881+0000 7ff26f000700 1 --2- 172.21.15.62:0/2084825286 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff268108470 0x7ff268108840 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:46:14.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.881+0000 7ff26f000700 1 -- 172.21.15.62:0/2084825286 >> 172.21.15.62:0/2084825286 conn(0x7ff2680fdf90 msgr2=0x7ff2681003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:46:14.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.882+0000 7ff26f000700 1 -- 172.21.15.62:0/2084825286 shutdown_connections 2024-04-20T21:46:14.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.882+0000 7ff26f000700 1 -- 172.21.15.62:0/2084825286 wait complete. 2024-04-20T21:46:14.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.883+0000 7ff26f000700 1 Processor -- start 2024-04-20T21:46:14.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.883+0000 7ff26f000700 1 -- start start 2024-04-20T21:46:14.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.883+0000 7ff26f000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff268102470 0x7ff2681981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:46:14.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.883+0000 7ff26f000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff268108470 0x7ff268198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:46:14.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.883+0000 7ff26f000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff268198e10 con 0x7ff268108470 2024-04-20T21:46:14.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.883+0000 7ff26f000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff26819cba0 con 0x7ff268102470 2024-04-20T21:46:14.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.884+0000 7ff267400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff268108470 0x7ff268198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:46:14.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.884+0000 7ff267400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff268108470 0x7ff268198730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50016/0 (socket says 172.21.15.62:50016) 2024-04-20T21:46:14.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.884+0000 7ff267400700 1 -- 172.21.15.62:0/1157918532 learned_addr learned my addr 172.21.15.62:0/1157918532 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:46:14.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.884+0000 7ff267e00700 1 --2- 172.21.15.62:0/1157918532 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff268102470 0x7ff2681981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:46:14.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.884+0000 7ff267400700 1 -- 172.21.15.62:0/1157918532 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff268102470 msgr2=0x7ff2681981f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:46:14.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.884+0000 7ff267400700 1 --2- 172.21.15.62:0/1157918532 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff268102470 0x7ff2681981f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:46:14.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.884+0000 7ff267400700 1 -- 172.21.15.62:0/1157918532 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2580095f0 con 0x7ff268108470 2024-04-20T21:46:14.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.885+0000 7ff267400700 1 --2- 172.21.15.62:0/1157918532 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff268108470 0x7ff268198730 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7ff25400d8d0 tx=0x7ff25400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:46:14.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.885+0000 7ff264c00700 1 -- 172.21.15.62:0/1157918532 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff254009940 con 0x7ff268108470 2024-04-20T21:46:14.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.885+0000 7ff264c00700 1 -- 172.21.15.62:0/1157918532 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff254010460 con 0x7ff268108470 2024-04-20T21:46:14.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.885+0000 7ff26f000700 1 -- 172.21.15.62:0/1157918532 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff26819ce80 con 0x7ff268108470 2024-04-20T21:46:14.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.885+0000 7ff264c00700 1 -- 172.21.15.62:0/1157918532 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff25400f5f0 con 0x7ff268108470 2024-04-20T21:46:14.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.885+0000 7ff26f000700 1 -- 172.21.15.62:0/1157918532 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff26819d3d0 con 0x7ff268108470 2024-04-20T21:46:14.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.887+0000 7ff26f000700 1 -- 172.21.15.62:0/1157918532 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff26810a9e0 con 0x7ff268108470 2024-04-20T21:46:14.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.888+0000 7ff264c00700 1 -- 172.21.15.62:0/1157918532 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff2540105d0 con 0x7ff268108470 2024-04-20T21:46:14.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.889+0000 7ff264c00700 1 --2- 172.21.15.62:0/1157918532 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff24c06ecc0 0x7ff24c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:46:14.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.889+0000 7ff264c00700 1 -- 172.21.15.62:0/1157918532 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7ff25408ef20 con 0x7ff268108470 2024-04-20T21:46:14.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.889+0000 7ff267e00700 1 --2- 172.21.15.62:0/1157918532 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff24c06ecc0 0x7ff24c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:46:14.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.890+0000 7ff267e00700 1 --2- 172.21.15.62:0/1157918532 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff24c06ecc0 0x7ff24c071170 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7ff258006010 tx=0x7ff25801a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:46:14.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:14.894+0000 7ff264c00700 1 -- 172.21.15.62:0/1157918532 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7ff254057c30 con 0x7ff268108470 2024-04-20T21:46:14.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:46:14 smithi189 ceph-mon[57022]: from='client.34648 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:46:15.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:15.226+0000 7ff26f000700 1 -- 172.21.15.62:0/1157918532 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7ff26819d5f0 con 0x7ff268108470 2024-04-20T21:46:15.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:15.228+0000 7ff264c00700 1 -- 172.21.15.62:0/1157918532 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7ff25405b2f0 con 0x7ff268108470 2024-04-20T21:46:15.228 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:46:15.228 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T21:46:15.228 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:46:15.228 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:46:15.228 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T21:46:15.228 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:46:15.228 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:46:15.228 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T21:46:15.228 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 6 2024-04-20T21:46:15.228 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:46:15.228 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T21:46:15.228 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 2, 2024-04-20T21:46:15.229 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:46:15.229 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:46:15.229 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T21:46:15.229 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 2, 2024-04-20T21:46:15.229 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 12 2024-04-20T21:46:15.229 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T21:46:15.229 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:46:15.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:15.231+0000 7ff26f000700 1 -- 172.21.15.62:0/1157918532 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff24c06ecc0 msgr2=0x7ff24c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:46:15.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:15.231+0000 7ff26f000700 1 --2- 172.21.15.62:0/1157918532 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff24c06ecc0 0x7ff24c071170 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7ff258006010 tx=0x7ff25801a040 comp rx=0 tx=0).stop 2024-04-20T21:46:15.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:15.231+0000 7ff26f000700 1 -- 172.21.15.62:0/1157918532 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff268108470 msgr2=0x7ff268198730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:46:15.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:15.231+0000 7ff26f000700 1 --2- 172.21.15.62:0/1157918532 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff268108470 0x7ff268198730 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7ff25400d8d0 tx=0x7ff25400dc90 comp rx=0 tx=0).stop 2024-04-20T21:46:15.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:15.231+0000 7ff26f000700 1 -- 172.21.15.62:0/1157918532 shutdown_connections 2024-04-20T21:46:15.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:15.231+0000 7ff26f000700 1 --2- 172.21.15.62:0/1157918532 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff24c06ecc0 0x7ff24c071170 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:46:15.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:15.231+0000 7ff26f000700 1 --2- 172.21.15.62:0/1157918532 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff268102470 0x7ff2681981f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:46:15.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:15.231+0000 7ff26f000700 1 --2- 172.21.15.62:0/1157918532 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff268108470 0x7ff268198730 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:46:15.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:15.231+0000 7ff26f000700 1 -- 172.21.15.62:0/1157918532 >> 172.21.15.62:0/1157918532 conn(0x7ff2680fdf90 msgr2=0x7ff2681002c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:46:15.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:15.231+0000 7ff26f000700 1 -- 172.21.15.62:0/1157918532 shutdown_connections 2024-04-20T21:46:15.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:15.231+0000 7ff26f000700 1 -- 172.21.15.62:0/1157918532 wait complete. 2024-04-20T21:46:15.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:15.395+0000 7f7ec7e00700 1 -- 172.21.15.62:0/1722772499 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ec80ff170 msgr2=0x7f7ec80ff540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:46:15.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:15.395+0000 7f7ec7e00700 1 --2- 172.21.15.62:0/1722772499 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ec80ff170 0x7f7ec80ff540 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f7eb8009910 tx=0x7f7eb8009c20 comp rx=0 tx=0).stop 2024-04-20T21:46:15.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:15.395+0000 7f7ec7e00700 1 -- 172.21.15.62:0/1722772499 shutdown_connections 2024-04-20T21:46:15.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:15.395+0000 7f7ec7e00700 1 --2- 172.21.15.62:0/1722772499 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ec80ffa80 0x7f7ec810a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:46:15.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:15.395+0000 7f7ec7e00700 1 --2- 172.21.15.62:0/1722772499 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ec80ff170 0x7f7ec80ff540 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:46:15.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:15.395+0000 7f7ec7e00700 1 -- 172.21.15.62:0/1722772499 >> 172.21.15.62:0/1722772499 conn(0x7f7ec8074770 msgr2=0x7f7ec8074b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:46:15.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:15.396+0000 7f7ec7e00700 1 -- 172.21.15.62:0/1722772499 shutdown_connections 2024-04-20T21:46:15.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:15.396+0000 7f7ec7e00700 1 -- 172.21.15.62:0/1722772499 wait complete. 2024-04-20T21:46:15.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:15.397+0000 7f7ec7e00700 1 Processor -- start 2024-04-20T21:46:15.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:15.397+0000 7f7ec7e00700 1 -- start start 2024-04-20T21:46:15.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:15.397+0000 7f7ec7e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ec80ff170 0x7f7ec819b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:46:15.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:15.397+0000 7f7ec7e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ec80ffa80 0x7f7ec819be50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:46:15.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:15.397+0000 7f7ec7e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ec819c4e0 con 0x7f7ec80ffa80 2024-04-20T21:46:15.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:15.397+0000 7f7ec7e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ec8195990 con 0x7f7ec80ff170 2024-04-20T21:46:15.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:15.398+0000 7f7ebfe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ec80ffa80 0x7f7ec819be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:46:15.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:15.398+0000 7f7ebfe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ec80ffa80 0x7f7ec819be50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50044/0 (socket says 172.21.15.62:50044) 2024-04-20T21:46:15.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:15.398+0000 7f7ebfe00700 1 -- 172.21.15.62:0/805713670 learned_addr learned my addr 172.21.15.62:0/805713670 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:46:15.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:15.398+0000 7f7ec6a00700 1 --2- 172.21.15.62:0/805713670 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ec80ff170 0x7f7ec819b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:46:15.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:15.398+0000 7f7ebfe00700 1 -- 172.21.15.62:0/805713670 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ec80ff170 msgr2=0x7f7ec819b910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:46:15.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:15.398+0000 7f7ebfe00700 1 --2- 172.21.15.62:0/805713670 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ec80ff170 0x7f7ec819b910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:46:15.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:15.398+0000 7f7ebfe00700 1 -- 172.21.15.62:0/805713670 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7eb80095f0 con 0x7f7ec80ffa80 2024-04-20T21:46:15.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:15.398+0000 7f7ec6a00700 1 --2- 172.21.15.62:0/805713670 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ec80ff170 0x7f7ec819b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T21:46:15.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:15.398+0000 7f7ebfe00700 1 --2- 172.21.15.62:0/805713670 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ec80ffa80 0x7f7ec819be50 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f7eb000d8d0 tx=0x7f7eb000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:46:15.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:15.399+0000 7f7ecca00700 1 -- 172.21.15.62:0/805713670 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7eb0009940 con 0x7f7ec80ffa80 2024-04-20T21:46:15.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:15.399+0000 7f7ecca00700 1 -- 172.21.15.62:0/805713670 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7eb0010460 con 0x7f7ec80ffa80 2024-04-20T21:46:15.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:15.399+0000 7f7ecca00700 1 -- 172.21.15.62:0/805713670 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7eb000f5f0 con 0x7f7ec80ffa80 2024-04-20T21:46:15.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:15.399+0000 7f7ec7e00700 1 -- 172.21.15.62:0/805713670 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ec8195c70 con 0x7f7ec80ffa80 2024-04-20T21:46:15.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:15.399+0000 7f7ec7e00700 1 -- 172.21.15.62:0/805713670 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ec81961c0 con 0x7f7ec80ffa80 2024-04-20T21:46:15.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:15.400+0000 7f7ec7e00700 1 -- 172.21.15.62:0/805713670 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ec8107bf0 con 0x7f7ec80ffa80 2024-04-20T21:46:15.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:15.401+0000 7f7ecca00700 1 -- 172.21.15.62:0/805713670 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f7eb00105d0 con 0x7f7ec80ffa80 2024-04-20T21:46:15.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:15.402+0000 7f7ecca00700 1 --2- 172.21.15.62:0/805713670 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7eac06ecc0 0x7f7eac071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:46:15.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:15.402+0000 7f7ecca00700 1 -- 172.21.15.62:0/805713670 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f7eb008ee10 con 0x7f7ec80ffa80 2024-04-20T21:46:15.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:15.402+0000 7f7ec6a00700 1 --2- 172.21.15.62:0/805713670 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7eac06ecc0 0x7f7eac071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:46:15.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:15.403+0000 7f7ec6a00700 1 --2- 172.21.15.62:0/805713670 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7eac06ecc0 0x7f7eac071170 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f7eb8006010 tx=0x7f7eb800b540 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:46:15.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:15.407+0000 7f7ecca00700 1 -- 172.21.15.62:0/805713670 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f7eb005b6d0 con 0x7f7ec80ffa80 2024-04-20T21:46:15.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:15.694+0000 7f7ec7e00700 1 -- 172.21.15.62:0/805713670 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f7ec8062380 con 0x7f7ec80ffa80 2024-04-20T21:46:15.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:15.695+0000 7f7ecca00700 1 -- 172.21.15.62:0/805713670 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 25 v25) v1 ==== 76+0+1887 (secure 0 0 0) 0x7f7eb0016770 con 0x7f7ec80ffa80 2024-04-20T21:46:15.696 INFO:teuthology.orchestra.run.smithi062.stdout:e25 2024-04-20T21:46:15.696 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T21:46:15.696 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T21:46:15.696 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T21:46:15.696 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:46:15.696 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T21:46:15.696 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T21:46:15.696 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 20 2024-04-20T21:46:15.696 INFO:teuthology.orchestra.run.smithi062.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-20T21:46:15.697 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T21:46:15.697 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T21:42:01.559061+0000 2024-04-20T21:46:15.697 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T21:46:15.697 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T21:46:15.697 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T21:46:15.697 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T21:46:15.697 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T21:46:15.697 INFO:teuthology.orchestra.run.smithi062.stdout:max_xattr_size 65536 2024-04-20T21:46:15.697 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T21:46:15.697 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T21:46:15.697 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 81 2024-04-20T21:46:15.697 INFO:teuthology.orchestra.run.smithi062.stdout:snap_rstat 1 2024-04-20T21:46:15.697 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T21:46:15.697 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 2 2024-04-20T21:46:15.697 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1 2024-04-20T21:46:15.697 INFO:teuthology.orchestra.run.smithi062.stdout:up {} 2024-04-20T21:46:15.697 INFO:teuthology.orchestra.run.smithi062.stdout:failed 0,1 2024-04-20T21:46:15.698 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T21:46:15.698 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T21:46:15.698 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T21:46:15.698 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T21:46:15.698 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data enabled 2024-04-20T21:46:15.698 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T21:46:15.698 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T21:46:15.698 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 1 2024-04-20T21:46:15.698 INFO:teuthology.orchestra.run.smithi062.stdout:qdb_cluster leader: 0 members: 2024-04-20T21:46:15.698 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:46:15.698 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:46:15.698 INFO:teuthology.orchestra.run.smithi062.stdout:Standby daemons: 2024-04-20T21:46:15.698 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:46:15.698 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ryqblb{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6826/4113916205,v1:172.21.15.62:6827/4113916205] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T21:46:15.698 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ozizwi{-1:34476} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6828/3801975165,v1:172.21.15.62:6829/3801975165] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T21:46:15.699 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.xbgspw{-1:44285} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6826/3996040526,v1:172.21.15.189:6827/3996040526] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:46:15.699 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.dvimer{-1:44289} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6824/1743111843,v1:172.21.15.189:6825/1743111843] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:46:15.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:15.698+0000 7f7ec7e00700 1 -- 172.21.15.62:0/805713670 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7eac06ecc0 msgr2=0x7f7eac071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:46:15.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:15.698+0000 7f7ec7e00700 1 --2- 172.21.15.62:0/805713670 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7eac06ecc0 0x7f7eac071170 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f7eb8006010 tx=0x7f7eb800b540 comp rx=0 tx=0).stop 2024-04-20T21:46:15.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:15.698+0000 7f7ec7e00700 1 -- 172.21.15.62:0/805713670 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ec80ffa80 msgr2=0x7f7ec819be50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:46:15.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:15.698+0000 7f7ec7e00700 1 --2- 172.21.15.62:0/805713670 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ec80ffa80 0x7f7ec819be50 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f7eb000d8d0 tx=0x7f7eb000dc90 comp rx=0 tx=0).stop 2024-04-20T21:46:15.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:15.698+0000 7f7ec7e00700 1 -- 172.21.15.62:0/805713670 shutdown_connections 2024-04-20T21:46:15.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:15.698+0000 7f7ec7e00700 1 --2- 172.21.15.62:0/805713670 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7eac06ecc0 0x7f7eac071170 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:46:15.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:15.698+0000 7f7ec7e00700 1 --2- 172.21.15.62:0/805713670 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ec80ff170 0x7f7ec819b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:46:15.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:15.698+0000 7f7ec7e00700 1 --2- 172.21.15.62:0/805713670 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ec80ffa80 0x7f7ec819be50 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:46:15.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:15.698+0000 7f7ec7e00700 1 -- 172.21.15.62:0/805713670 >> 172.21.15.62:0/805713670 conn(0x7f7ec8074770 msgr2=0x7f7ec8104d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:46:15.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:15.699+0000 7f7ec7e00700 1 -- 172.21.15.62:0/805713670 shutdown_connections 2024-04-20T21:46:15.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:15.699+0000 7f7ec7e00700 1 -- 172.21.15.62:0/805713670 wait complete. 2024-04-20T21:46:15.700 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 25 2024-04-20T21:46:15.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:15.860+0000 7f9bcbc00700 1 -- 172.21.15.62:0/3646551796 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9bc4101420 msgr2=0x7f9bc41017f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:46:15.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:15.860+0000 7f9bcbc00700 1 --2- 172.21.15.62:0/3646551796 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9bc4101420 0x7f9bc41017f0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f9bb8009910 tx=0x7f9bb8009c20 comp rx=0 tx=0).stop 2024-04-20T21:46:15.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:15.860+0000 7f9bcbc00700 1 -- 172.21.15.62:0/3646551796 shutdown_connections 2024-04-20T21:46:15.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:15.860+0000 7f9bcbc00700 1 --2- 172.21.15.62:0/3646551796 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9bc4101d30 0x7f9bc410a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:46:15.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:15.860+0000 7f9bcbc00700 1 --2- 172.21.15.62:0/3646551796 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9bc4101420 0x7f9bc41017f0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:46:15.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:15.860+0000 7f9bcbc00700 1 -- 172.21.15.62:0/3646551796 >> 172.21.15.62:0/3646551796 conn(0x7f9bc40fac80 msgr2=0x7f9bc40fd090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:46:15.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:15.860+0000 7f9bcbc00700 1 -- 172.21.15.62:0/3646551796 shutdown_connections 2024-04-20T21:46:15.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:15.860+0000 7f9bcbc00700 1 -- 172.21.15.62:0/3646551796 wait complete. 2024-04-20T21:46:15.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:15.861+0000 7f9bcbc00700 1 Processor -- start 2024-04-20T21:46:15.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:15.861+0000 7f9bcbc00700 1 -- start start 2024-04-20T21:46:15.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:15.862+0000 7f9bcbc00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9bc4101420 0x7f9bc4195e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:46:15.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:15.862+0000 7f9bcbc00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9bc4101d30 0x7f9bc41963b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:46:15.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:15.862+0000 7f9bcbc00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bc4196a90 con 0x7f9bc4101420 2024-04-20T21:46:15.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:15.862+0000 7f9bcbc00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bc419a820 con 0x7f9bc4101d30 2024-04-20T21:46:15.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:15.862+0000 7f9bc9200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9bc4101420 0x7f9bc4195e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:46:15.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:15.862+0000 7f9bc9200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9bc4101420 0x7f9bc4195e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50068/0 (socket says 172.21.15.62:50068) 2024-04-20T21:46:15.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:15.862+0000 7f9bc9200700 1 -- 172.21.15.62:0/919595070 learned_addr learned my addr 172.21.15.62:0/919595070 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:46:15.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:15.863+0000 7f9bc9200700 1 -- 172.21.15.62:0/919595070 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9bc4101d30 msgr2=0x7f9bc41963b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:46:15.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:15.863+0000 7f9bc3e00700 1 --2- 172.21.15.62:0/919595070 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9bc4101d30 0x7f9bc41963b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:46:15.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:15.863+0000 7f9bc9200700 1 --2- 172.21.15.62:0/919595070 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9bc4101d30 0x7f9bc41963b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:46:15.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:15.863+0000 7f9bc9200700 1 -- 172.21.15.62:0/919595070 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bb80095f0 con 0x7f9bc4101420 2024-04-20T21:46:15.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:15.863+0000 7f9bc9200700 1 --2- 172.21.15.62:0/919595070 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9bc4101420 0x7f9bc4195e70 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f9bb8006010 tx=0x7f9bb800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:46:15.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:15.863+0000 7f9bc1600700 1 -- 172.21.15.62:0/919595070 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9bb801d070 con 0x7f9bc4101420 2024-04-20T21:46:15.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:15.863+0000 7f9bcbc00700 1 -- 172.21.15.62:0/919595070 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bc419aaa0 con 0x7f9bc4101420 2024-04-20T21:46:15.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:15.863+0000 7f9bc1600700 1 -- 172.21.15.62:0/919595070 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9bb8004fb0 con 0x7f9bc4101420 2024-04-20T21:46:15.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:15.863+0000 7f9bc1600700 1 -- 172.21.15.62:0/919595070 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9bb800f690 con 0x7f9bc4101420 2024-04-20T21:46:15.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:15.863+0000 7f9bcbc00700 1 -- 172.21.15.62:0/919595070 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bc419af90 con 0x7f9bc4101420 2024-04-20T21:46:15.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:15.864+0000 7f9bcbc00700 1 -- 172.21.15.62:0/919595070 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bc4107900 con 0x7f9bc4101420 2024-04-20T21:46:15.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:15.866+0000 7f9bc1600700 1 -- 172.21.15.62:0/919595070 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f9bb80228a0 con 0x7f9bc4101420 2024-04-20T21:46:15.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:15.866+0000 7f9bc1600700 1 --2- 172.21.15.62:0/919595070 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9bac06ecc0 0x7f9bac071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:46:15.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:15.867+0000 7f9bc1600700 1 -- 172.21.15.62:0/919595070 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f9bb8090660 con 0x7f9bc4101420 2024-04-20T21:46:15.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:15.867+0000 7f9bc3e00700 1 --2- 172.21.15.62:0/919595070 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9bac06ecc0 0x7f9bac071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:46:15.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:15.868+0000 7f9bc3e00700 1 --2- 172.21.15.62:0/919595070 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9bac06ecc0 0x7f9bac071170 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f9bc4197490 tx=0x7f9bb0005cf0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:46:15.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:15.871+0000 7f9bc1600700 1 -- 172.21.15.62:0/919595070 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f9bb8059490 con 0x7f9bc4101420 2024-04-20T21:46:15.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:46:15 smithi189 ceph-mon[57022]: from='client.34652 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:46:15.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:46:15 smithi189 ceph-mon[57022]: pgmap v328: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:46:15.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:46:15 smithi189 ceph-mon[57022]: from='client.34656 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:46:15.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:46:15 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/1157918532' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:46:16.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:46:15 smithi062 ceph-mon[76322]: from='client.34652 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:46:16.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:46:15 smithi062 ceph-mon[76322]: pgmap v328: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:46:16.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:46:15 smithi062 ceph-mon[76322]: from='client.34656 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:46:16.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:46:15 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/1157918532' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:46:16.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:16.123+0000 7f9bcbc00700 1 -- 172.21.15.62:0/919595070 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9bc419b2c0 con 0x7f9bac06ecc0 2024-04-20T21:46:16.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:16.126+0000 7f9bc1600700 1 -- 172.21.15.62:0/919595070 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f9bc419b2c0 con 0x7f9bac06ecc0 2024-04-20T21:46:16.125 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:46:16.126 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T21:46:16.126 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T21:46:16.126 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-20T21:46:16.126 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T21:46:16.126 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr", 2024-04-20T21:46:16.126 INFO:teuthology.orchestra.run.smithi062.stdout: "crash", 2024-04-20T21:46:16.126 INFO:teuthology.orchestra.run.smithi062.stdout: "mon", 2024-04-20T21:46:16.126 INFO:teuthology.orchestra.run.smithi062.stdout: "osd" 2024-04-20T21:46:16.126 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T21:46:16.126 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "14/23 daemons upgraded", 2024-04-20T21:46:16.126 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading mds daemons", 2024-04-20T21:46:16.126 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T21:46:16.126 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:46:16.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:16.128+0000 7f9bcbc00700 1 -- 172.21.15.62:0/919595070 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9bac06ecc0 msgr2=0x7f9bac071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:46:16.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:16.128+0000 7f9bcbc00700 1 --2- 172.21.15.62:0/919595070 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9bac06ecc0 0x7f9bac071170 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f9bc4197490 tx=0x7f9bb0005cf0 comp rx=0 tx=0).stop 2024-04-20T21:46:16.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:16.128+0000 7f9bcbc00700 1 -- 172.21.15.62:0/919595070 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9bc4101420 msgr2=0x7f9bc4195e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:46:16.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:16.128+0000 7f9bcbc00700 1 --2- 172.21.15.62:0/919595070 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9bc4101420 0x7f9bc4195e70 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f9bb8006010 tx=0x7f9bb800be20 comp rx=0 tx=0).stop 2024-04-20T21:46:16.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:16.129+0000 7f9bcbc00700 1 -- 172.21.15.62:0/919595070 shutdown_connections 2024-04-20T21:46:16.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:16.129+0000 7f9bcbc00700 1 --2- 172.21.15.62:0/919595070 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9bac06ecc0 0x7f9bac071170 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:46:16.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:16.129+0000 7f9bcbc00700 1 --2- 172.21.15.62:0/919595070 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9bc4101420 0x7f9bc4195e70 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:46:16.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:16.129+0000 7f9bcbc00700 1 --2- 172.21.15.62:0/919595070 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9bc4101d30 0x7f9bc41963b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:46:16.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:16.129+0000 7f9bcbc00700 1 -- 172.21.15.62:0/919595070 >> 172.21.15.62:0/919595070 conn(0x7f9bc40fac80 msgr2=0x7f9bc4104a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:46:16.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:16.129+0000 7f9bcbc00700 1 -- 172.21.15.62:0/919595070 shutdown_connections 2024-04-20T21:46:16.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:16.129+0000 7f9bcbc00700 1 -- 172.21.15.62:0/919595070 wait complete. 2024-04-20T21:46:16.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:16.294+0000 7f7f92600700 1 -- 172.21.15.62:0/729537210 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f8c0687a0 msgr2=0x7f7f8c068b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:46:16.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:16.294+0000 7f7f92600700 1 --2- 172.21.15.62:0/729537210 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f8c0687a0 0x7f7f8c068b70 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f7f7c009910 tx=0x7f7f7c009c20 comp rx=0 tx=0).stop 2024-04-20T21:46:16.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:16.294+0000 7f7f92600700 1 -- 172.21.15.62:0/729537210 shutdown_connections 2024-04-20T21:46:16.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:16.294+0000 7f7f92600700 1 --2- 172.21.15.62:0/729537210 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f8c069140 0x7f7f8c104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:46:16.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:16.294+0000 7f7f92600700 1 --2- 172.21.15.62:0/729537210 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f8c0687a0 0x7f7f8c068b70 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:46:16.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:16.294+0000 7f7f92600700 1 -- 172.21.15.62:0/729537210 >> 172.21.15.62:0/729537210 conn(0x7f7f8c075130 msgr2=0x7f7f8c075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:46:16.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:16.295+0000 7f7f92600700 1 -- 172.21.15.62:0/729537210 shutdown_connections 2024-04-20T21:46:16.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:16.295+0000 7f7f92600700 1 -- 172.21.15.62:0/729537210 wait complete. 2024-04-20T21:46:16.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:16.296+0000 7f7f92600700 1 Processor -- start 2024-04-20T21:46:16.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:16.296+0000 7f7f92600700 1 -- start start 2024-04-20T21:46:16.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:16.297+0000 7f7f92600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f8c0687a0 0x7f7f8c198160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:46:16.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:16.297+0000 7f7f92600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f8c069140 0x7f7f8c1986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:46:16.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:16.297+0000 7f7f92600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f8c198d80 con 0x7f7f8c0687a0 2024-04-20T21:46:16.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:16.297+0000 7f7f92600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f8c19cb10 con 0x7f7f8c069140 2024-04-20T21:46:16.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:16.297+0000 7f7f8b400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f8c0687a0 0x7f7f8c198160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:46:16.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:16.297+0000 7f7f8b400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f8c0687a0 0x7f7f8c198160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50086/0 (socket says 172.21.15.62:50086) 2024-04-20T21:46:16.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:16.297+0000 7f7f8b400700 1 -- 172.21.15.62:0/2226599326 learned_addr learned my addr 172.21.15.62:0/2226599326 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:46:16.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:16.298+0000 7f7f8aa00700 1 --2- 172.21.15.62:0/2226599326 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f8c069140 0x7f7f8c1986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:46:16.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:16.298+0000 7f7f8b400700 1 -- 172.21.15.62:0/2226599326 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f8c069140 msgr2=0x7f7f8c1986a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:46:16.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:16.298+0000 7f7f8b400700 1 --2- 172.21.15.62:0/2226599326 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f8c069140 0x7f7f8c1986a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:46:16.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:16.298+0000 7f7f8b400700 1 -- 172.21.15.62:0/2226599326 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f7c0095f0 con 0x7f7f8c0687a0 2024-04-20T21:46:16.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:16.298+0000 7f7f8b400700 1 --2- 172.21.15.62:0/2226599326 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f8c0687a0 0x7f7f8c198160 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f7f7c00b5c0 tx=0x7f7f7c00be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:46:16.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:16.299+0000 7f7f83e00700 1 -- 172.21.15.62:0/2226599326 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7f7c01d070 con 0x7f7f8c0687a0 2024-04-20T21:46:16.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:16.299+0000 7f7f83e00700 1 -- 172.21.15.62:0/2226599326 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7f7c00f460 con 0x7f7f8c0687a0 2024-04-20T21:46:16.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:16.299+0000 7f7f83e00700 1 -- 172.21.15.62:0/2226599326 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7f7c017610 con 0x7f7f8c0687a0 2024-04-20T21:46:16.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:16.299+0000 7f7f92600700 1 -- 172.21.15.62:0/2226599326 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f8c19cd90 con 0x7f7f8c0687a0 2024-04-20T21:46:16.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:16.299+0000 7f7f92600700 1 -- 172.21.15.62:0/2226599326 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f8c19d280 con 0x7f7f8c0687a0 2024-04-20T21:46:16.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:16.300+0000 7f7f92600700 1 -- 172.21.15.62:0/2226599326 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f8c04f2a0 con 0x7f7f8c0687a0 2024-04-20T21:46:16.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:16.301+0000 7f7f83e00700 1 -- 172.21.15.62:0/2226599326 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f7f7c00f5d0 con 0x7f7f8c0687a0 2024-04-20T21:46:16.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:16.302+0000 7f7f83e00700 1 --2- 172.21.15.62:0/2226599326 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7f7006ecc0 0x7f7f70071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:46:16.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:16.302+0000 7f7f83e00700 1 -- 172.21.15.62:0/2226599326 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f7f7c0903b0 con 0x7f7f8c0687a0 2024-04-20T21:46:16.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:16.303+0000 7f7f8aa00700 1 --2- 172.21.15.62:0/2226599326 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7f7006ecc0 0x7f7f70071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:46:16.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:16.303+0000 7f7f8aa00700 1 --2- 172.21.15.62:0/2226599326 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7f7006ecc0 0x7f7f70071170 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f7f78006fd0 tx=0x7f7f78008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:46:16.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:16.307+0000 7f7f83e00700 1 -- 172.21.15.62:0/2226599326 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f7f7c0591e0 con 0x7f7f8c0687a0 2024-04-20T21:46:16.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:16.635+0000 7f7f92600700 1 -- 172.21.15.62:0/2226599326 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f7f8c1994c0 con 0x7f7f8c0687a0 2024-04-20T21:46:16.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:16.636+0000 7f7f83e00700 1 -- 172.21.15.62:0/2226599326 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7f7f7c05c8a0 con 0x7f7f8c0687a0 2024-04-20T21:46:16.635 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-20T21:46:16.635 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-20T21:46:16.635 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is degraded 2024-04-20T21:46:16.635 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T21:46:16.635 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T21:46:16.636 INFO:teuthology.orchestra.run.smithi062.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-20T21:46:16.636 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-20T21:46:16.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:16.639+0000 7f7f92600700 1 -- 172.21.15.62:0/2226599326 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7f7006ecc0 msgr2=0x7f7f70071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:46:16.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:16.639+0000 7f7f92600700 1 --2- 172.21.15.62:0/2226599326 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7f7006ecc0 0x7f7f70071170 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f7f78006fd0 tx=0x7f7f78008040 comp rx=0 tx=0).stop 2024-04-20T21:46:16.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:16.639+0000 7f7f92600700 1 -- 172.21.15.62:0/2226599326 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f8c0687a0 msgr2=0x7f7f8c198160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:46:16.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:16.639+0000 7f7f92600700 1 --2- 172.21.15.62:0/2226599326 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f8c0687a0 0x7f7f8c198160 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f7f7c00b5c0 tx=0x7f7f7c00be00 comp rx=0 tx=0).stop 2024-04-20T21:46:16.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:16.639+0000 7f7f92600700 1 -- 172.21.15.62:0/2226599326 shutdown_connections 2024-04-20T21:46:16.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:16.639+0000 7f7f92600700 1 --2- 172.21.15.62:0/2226599326 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7f7006ecc0 0x7f7f70071170 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:46:16.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:16.639+0000 7f7f92600700 1 --2- 172.21.15.62:0/2226599326 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f8c0687a0 0x7f7f8c198160 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:46:16.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:16.639+0000 7f7f92600700 1 --2- 172.21.15.62:0/2226599326 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f8c069140 0x7f7f8c1986a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:46:16.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:16.639+0000 7f7f92600700 1 -- 172.21.15.62:0/2226599326 >> 172.21.15.62:0/2226599326 conn(0x7f7f8c075130 msgr2=0x7f7f8c101e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:46:16.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:16.640+0000 7f7f92600700 1 -- 172.21.15.62:0/2226599326 shutdown_connections 2024-04-20T21:46:16.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:16.640+0000 7f7f92600700 1 -- 172.21.15.62:0/2226599326 wait complete. 2024-04-20T21:46:16.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:46:16 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/805713670' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T21:46:17.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:46:16 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/805713670' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T21:46:17.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:46:17 smithi189 ceph-mon[57022]: from='client.34668 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:46:17.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:46:17 smithi189 ceph-mon[57022]: pgmap v329: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:46:17.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:46:17 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/2226599326' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T21:46:18.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:46:17 smithi062 ceph-mon[76322]: from='client.34668 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:46:18.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:46:17 smithi062 ceph-mon[76322]: pgmap v329: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:46:18.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:46:17 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/2226599326' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T21:46:19.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:46:19 smithi189 ceph-mon[57022]: pgmap v330: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:46:19.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:46:19 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:46:20.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:46:19 smithi062 ceph-mon[76322]: pgmap v330: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:46:20.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:46:19 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:46:21.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:46:21 smithi189 ceph-mon[57022]: pgmap v331: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:46:22.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:46:21 smithi062 ceph-mon[76322]: pgmap v331: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:46:23.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:46:23 smithi189 ceph-mon[57022]: pgmap v332: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:46:24.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:46:23 smithi062 ceph-mon[76322]: pgmap v332: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:46:25.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:46:25 smithi062 ceph-mon[76322]: pgmap v333: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:46:25.736 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:46:25 smithi189 ceph-mon[57022]: pgmap v333: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:46:27.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:46:27 smithi189 ceph-mon[57022]: pgmap v334: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:46:28.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:46:27 smithi062 ceph-mon[76322]: pgmap v334: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:46:29.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:46:29 smithi189 ceph-mon[57022]: pgmap v335: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:46:30.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:46:29 smithi062 ceph-mon[76322]: pgmap v335: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:46:31.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:46:31 smithi189 ceph-mon[57022]: pgmap v336: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:46:32.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:46:31 smithi062 ceph-mon[76322]: pgmap v336: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:46:33.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:46:33 smithi189 ceph-mon[57022]: pgmap v337: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:46:34.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:46:33 smithi062 ceph-mon[76322]: pgmap v337: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:46:34.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:46:34 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:46:35.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:46:34 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:46:35.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:46:35 smithi189 ceph-mon[57022]: pgmap v338: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:46:36.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:46:35 smithi062 ceph-mon[76322]: pgmap v338: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:46:37.986 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:46:37 smithi189 ceph-mon[57022]: pgmap v339: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:46:38.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:46:37 smithi062 ceph-mon[76322]: pgmap v339: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:46:39.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:46:39 smithi189 ceph-mon[57022]: pgmap v340: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:46:40.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:46:39 smithi062 ceph-mon[76322]: pgmap v340: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:46:41.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:46:41 smithi189 ceph-mon[57022]: pgmap v341: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:46:42.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:46:41 smithi062 ceph-mon[76322]: pgmap v341: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:46:43.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:46:43 smithi189 ceph-mon[57022]: pgmap v342: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:46:44.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:46:43 smithi062 ceph-mon[76322]: pgmap v342: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:46:45.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:46:45 smithi189 ceph-mon[57022]: pgmap v343: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:46:46.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:46:45 smithi062 ceph-mon[76322]: pgmap v343: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:46:46.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:46.816+0000 7f8d38000700 1 -- 172.21.15.62:0/2162073350 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d301004e0 msgr2=0x7f8d301008b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:46:46.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:46.816+0000 7f8d38000700 1 --2- 172.21.15.62:0/2162073350 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d301004e0 0x7f8d301008b0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f8d24009910 tx=0x7f8d24009c20 comp rx=0 tx=0).stop 2024-04-20T21:46:46.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:46.817+0000 7f8d38000700 1 -- 172.21.15.62:0/2162073350 shutdown_connections 2024-04-20T21:46:46.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:46.817+0000 7f8d38000700 1 --2- 172.21.15.62:0/2162073350 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d30100e80 0x7f8d30104e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:46:46.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:46.817+0000 7f8d38000700 1 --2- 172.21.15.62:0/2162073350 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d301004e0 0x7f8d301008b0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:46:46.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:46.817+0000 7f8d38000700 1 -- 172.21.15.62:0/2162073350 >> 172.21.15.62:0/2162073350 conn(0x7f8d300fbda0 msgr2=0x7f8d300fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:46:46.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:46.817+0000 7f8d38000700 1 -- 172.21.15.62:0/2162073350 shutdown_connections 2024-04-20T21:46:46.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:46.817+0000 7f8d38000700 1 -- 172.21.15.62:0/2162073350 wait complete. 2024-04-20T21:46:46.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:46.818+0000 7f8d38000700 1 Processor -- start 2024-04-20T21:46:46.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:46.818+0000 7f8d38000700 1 -- start start 2024-04-20T21:46:46.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:46.818+0000 7f8d38000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d301004e0 0x7f8d30195ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:46:46.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:46.818+0000 7f8d38000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d30100e80 0x7f8d30196430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:46:46.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:46.818+0000 7f8d38000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d30196b10 con 0x7f8d301004e0 2024-04-20T21:46:46.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:46.818+0000 7f8d38000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d3019a8a0 con 0x7f8d30100e80 2024-04-20T21:46:46.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:46.819+0000 7f8d35600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d301004e0 0x7f8d30195ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:46:46.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:46.819+0000 7f8d35600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d301004e0 0x7f8d30195ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56150/0 (socket says 172.21.15.62:56150) 2024-04-20T21:46:46.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:46.819+0000 7f8d35600700 1 -- 172.21.15.62:0/1985104463 learned_addr learned my addr 172.21.15.62:0/1985104463 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:46:46.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:46.819+0000 7f8d34c00700 1 --2- 172.21.15.62:0/1985104463 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d30100e80 0x7f8d30196430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:46:46.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:46.819+0000 7f8d35600700 1 -- 172.21.15.62:0/1985104463 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d30100e80 msgr2=0x7f8d30196430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:46:46.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:46.819+0000 7f8d35600700 1 --2- 172.21.15.62:0/1985104463 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d30100e80 0x7f8d30196430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:46:46.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:46.819+0000 7f8d35600700 1 -- 172.21.15.62:0/1985104463 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d240095f0 con 0x7f8d301004e0 2024-04-20T21:46:46.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:46.819+0000 7f8d34c00700 1 --2- 172.21.15.62:0/1985104463 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d30100e80 0x7f8d30196430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T21:46:46.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:46.820+0000 7f8d35600700 1 --2- 172.21.15.62:0/1985104463 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d301004e0 0x7f8d30195ef0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f8d2400b5c0 tx=0x7f8d2400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:46:46.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:46.820+0000 7f8d2e000700 1 -- 172.21.15.62:0/1985104463 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8d2401d070 con 0x7f8d301004e0 2024-04-20T21:46:46.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:46.820+0000 7f8d2e000700 1 -- 172.21.15.62:0/1985104463 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8d24005e00 con 0x7f8d301004e0 2024-04-20T21:46:46.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:46.820+0000 7f8d2e000700 1 -- 172.21.15.62:0/1985104463 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8d24017400 con 0x7f8d301004e0 2024-04-20T21:46:46.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:46.820+0000 7f8d38000700 1 -- 172.21.15.62:0/1985104463 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d3019ab20 con 0x7f8d301004e0 2024-04-20T21:46:46.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:46.820+0000 7f8d38000700 1 -- 172.21.15.62:0/1985104463 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d3019b010 con 0x7f8d301004e0 2024-04-20T21:46:46.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:46.821+0000 7f8d38000700 1 -- 172.21.15.62:0/1985104463 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d301087f0 con 0x7f8d301004e0 2024-04-20T21:46:46.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:46.822+0000 7f8d2e000700 1 -- 172.21.15.62:0/1985104463 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f8d2400f460 con 0x7f8d301004e0 2024-04-20T21:46:46.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:46.823+0000 7f8d2e000700 1 --2- 172.21.15.62:0/1985104463 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f8d1806ecc0 0x7f8d18071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:46:46.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:46.823+0000 7f8d2e000700 1 -- 172.21.15.62:0/1985104463 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f8d24090370 con 0x7f8d301004e0 2024-04-20T21:46:46.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:46.824+0000 7f8d34c00700 1 --2- 172.21.15.62:0/1985104463 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f8d1806ecc0 0x7f8d18071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:46:46.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:46.824+0000 7f8d34c00700 1 --2- 172.21.15.62:0/1985104463 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f8d1806ecc0 0x7f8d18071170 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f8d1c005fd0 tx=0x7f8d1c005ee0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:46:46.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:46.829+0000 7f8d2e000700 1 -- 172.21.15.62:0/1985104463 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f8d24059100 con 0x7f8d301004e0 2024-04-20T21:46:47.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:47.087+0000 7f8d38000700 1 -- 172.21.15.62:0/1985104463 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8d3019b280 con 0x7f8d1806ecc0 2024-04-20T21:46:47.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:47.090+0000 7f8d2e000700 1 -- 172.21.15.62:0/1985104463 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f8d3019b280 con 0x7f8d1806ecc0 2024-04-20T21:46:47.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:47.093+0000 7f8d38000700 1 -- 172.21.15.62:0/1985104463 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f8d1806ecc0 msgr2=0x7f8d18071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:46:47.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:47.093+0000 7f8d38000700 1 --2- 172.21.15.62:0/1985104463 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f8d1806ecc0 0x7f8d18071170 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f8d1c005fd0 tx=0x7f8d1c005ee0 comp rx=0 tx=0).stop 2024-04-20T21:46:47.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:47.093+0000 7f8d38000700 1 -- 172.21.15.62:0/1985104463 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d301004e0 msgr2=0x7f8d30195ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:46:47.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:47.093+0000 7f8d38000700 1 --2- 172.21.15.62:0/1985104463 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d301004e0 0x7f8d30195ef0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f8d2400b5c0 tx=0x7f8d2400be20 comp rx=0 tx=0).stop 2024-04-20T21:46:47.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:47.093+0000 7f8d38000700 1 -- 172.21.15.62:0/1985104463 shutdown_connections 2024-04-20T21:46:47.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:47.093+0000 7f8d38000700 1 --2- 172.21.15.62:0/1985104463 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f8d1806ecc0 0x7f8d18071170 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:46:47.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:47.093+0000 7f8d38000700 1 --2- 172.21.15.62:0/1985104463 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d301004e0 0x7f8d30195ef0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:46:47.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:47.093+0000 7f8d38000700 1 --2- 172.21.15.62:0/1985104463 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d30100e80 0x7f8d30196430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:46:47.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:47.093+0000 7f8d38000700 1 -- 172.21.15.62:0/1985104463 >> 172.21.15.62:0/1985104463 conn(0x7f8d300fbda0 msgr2=0x7f8d300fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:46:47.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:47.093+0000 7f8d38000700 1 -- 172.21.15.62:0/1985104463 shutdown_connections 2024-04-20T21:46:47.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:47.093+0000 7f8d38000700 1 -- 172.21.15.62:0/1985104463 wait complete. 2024-04-20T21:46:47.109 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T21:46:47.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:47.268+0000 7fc2ba800700 1 -- 172.21.15.62:0/1893461413 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc2b4100e80 msgr2=0x7fc2b4104e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:46:47.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:47.268+0000 7fc2ba800700 1 --2- 172.21.15.62:0/1893461413 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc2b4100e80 0x7fc2b4104e50 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fc2a0009910 tx=0x7fc2a0009c20 comp rx=0 tx=0).stop 2024-04-20T21:46:47.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:47.268+0000 7fc2ba800700 1 -- 172.21.15.62:0/1893461413 shutdown_connections 2024-04-20T21:46:47.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:47.268+0000 7fc2ba800700 1 --2- 172.21.15.62:0/1893461413 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc2b4100e80 0x7fc2b4104e50 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:46:47.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:47.268+0000 7fc2ba800700 1 --2- 172.21.15.62:0/1893461413 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc2b41004e0 0x7fc2b41008b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:46:47.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:47.268+0000 7fc2ba800700 1 -- 172.21.15.62:0/1893461413 >> 172.21.15.62:0/1893461413 conn(0x7fc2b40fbda0 msgr2=0x7fc2b40fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:46:47.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:47.268+0000 7fc2ba800700 1 -- 172.21.15.62:0/1893461413 shutdown_connections 2024-04-20T21:46:47.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:47.268+0000 7fc2ba800700 1 -- 172.21.15.62:0/1893461413 wait complete. 2024-04-20T21:46:47.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:47.269+0000 7fc2ba800700 1 Processor -- start 2024-04-20T21:46:47.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:47.270+0000 7fc2ba800700 1 -- start start 2024-04-20T21:46:47.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:47.270+0000 7fc2ba800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc2b41004e0 0x7fc2b41980e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:46:47.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:47.270+0000 7fc2ba800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc2b4100e80 0x7fc2b4198620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:46:47.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:47.270+0000 7fc2ba800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2b4198d00 con 0x7fc2b41004e0 2024-04-20T21:46:47.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:47.270+0000 7fc2ba800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2b419ca90 con 0x7fc2b4100e80 2024-04-20T21:46:47.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:47.270+0000 7fc2b3400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc2b41004e0 0x7fc2b41980e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:46:47.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:47.270+0000 7fc2b3400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc2b41004e0 0x7fc2b41980e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56166/0 (socket says 172.21.15.62:56166) 2024-04-20T21:46:47.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:47.271+0000 7fc2b3400700 1 -- 172.21.15.62:0/3221431517 learned_addr learned my addr 172.21.15.62:0/3221431517 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:46:47.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:47.271+0000 7fc2b2a00700 1 --2- 172.21.15.62:0/3221431517 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc2b4100e80 0x7fc2b4198620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:46:47.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:47.271+0000 7fc2b3400700 1 -- 172.21.15.62:0/3221431517 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc2b4100e80 msgr2=0x7fc2b4198620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:46:47.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:47.271+0000 7fc2b3400700 1 --2- 172.21.15.62:0/3221431517 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc2b4100e80 0x7fc2b4198620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:46:47.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:47.271+0000 7fc2b3400700 1 -- 172.21.15.62:0/3221431517 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2a00095f0 con 0x7fc2b41004e0 2024-04-20T21:46:47.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:47.271+0000 7fc2b2a00700 1 --2- 172.21.15.62:0/3221431517 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc2b4100e80 0x7fc2b4198620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T21:46:47.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:47.271+0000 7fc2b3400700 1 --2- 172.21.15.62:0/3221431517 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc2b41004e0 0x7fc2b41980e0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fc2a400d8d0 tx=0x7fc2a400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:46:47.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:47.272+0000 7fc2abe00700 1 -- 172.21.15.62:0/3221431517 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc2a4009940 con 0x7fc2b41004e0 2024-04-20T21:46:47.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:47.272+0000 7fc2abe00700 1 -- 172.21.15.62:0/3221431517 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc2a4010460 con 0x7fc2b41004e0 2024-04-20T21:46:47.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:47.272+0000 7fc2abe00700 1 -- 172.21.15.62:0/3221431517 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc2a400f5f0 con 0x7fc2b41004e0 2024-04-20T21:46:47.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:47.272+0000 7fc2ba800700 1 -- 172.21.15.62:0/3221431517 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2b419cd70 con 0x7fc2b41004e0 2024-04-20T21:46:47.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:47.272+0000 7fc2ba800700 1 -- 172.21.15.62:0/3221431517 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2b419d2c0 con 0x7fc2b41004e0 2024-04-20T21:46:47.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:47.273+0000 7fc2ba800700 1 -- 172.21.15.62:0/3221431517 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2b41045d0 con 0x7fc2b41004e0 2024-04-20T21:46:47.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:47.274+0000 7fc2abe00700 1 -- 172.21.15.62:0/3221431517 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fc2a40105d0 con 0x7fc2b41004e0 2024-04-20T21:46:47.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:47.275+0000 7fc2abe00700 1 --2- 172.21.15.62:0/3221431517 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc29806ecc0 0x7fc298071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:46:47.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:47.275+0000 7fc2abe00700 1 -- 172.21.15.62:0/3221431517 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fc2a408ee10 con 0x7fc2b41004e0 2024-04-20T21:46:47.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:47.276+0000 7fc2b2a00700 1 --2- 172.21.15.62:0/3221431517 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc29806ecc0 0x7fc298071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:46:47.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:47.276+0000 7fc2b2a00700 1 --2- 172.21.15.62:0/3221431517 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc29806ecc0 0x7fc298071170 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fc2a0006010 tx=0x7fc2a000b540 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:46:47.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:47.280+0000 7fc2abe00700 1 -- 172.21.15.62:0/3221431517 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fc2a4057ba0 con 0x7fc2b41004e0 2024-04-20T21:46:47.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:47.538+0000 7fc2ba800700 1 -- 172.21.15.62:0/3221431517 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc2b419d530 con 0x7fc29806ecc0 2024-04-20T21:46:47.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:47.541+0000 7fc2abe00700 1 -- 172.21.15.62:0/3221431517 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fc2b419d530 con 0x7fc29806ecc0 2024-04-20T21:46:47.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:47.544+0000 7fc2ba800700 1 -- 172.21.15.62:0/3221431517 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc29806ecc0 msgr2=0x7fc298071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:46:47.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:47.544+0000 7fc2ba800700 1 --2- 172.21.15.62:0/3221431517 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc29806ecc0 0x7fc298071170 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fc2a0006010 tx=0x7fc2a000b540 comp rx=0 tx=0).stop 2024-04-20T21:46:47.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:47.544+0000 7fc2ba800700 1 -- 172.21.15.62:0/3221431517 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc2b41004e0 msgr2=0x7fc2b41980e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:46:47.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:47.544+0000 7fc2ba800700 1 --2- 172.21.15.62:0/3221431517 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc2b41004e0 0x7fc2b41980e0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fc2a400d8d0 tx=0x7fc2a400dc90 comp rx=0 tx=0).stop 2024-04-20T21:46:47.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:47.544+0000 7fc2ba800700 1 -- 172.21.15.62:0/3221431517 shutdown_connections 2024-04-20T21:46:47.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:47.544+0000 7fc2ba800700 1 --2- 172.21.15.62:0/3221431517 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc29806ecc0 0x7fc298071170 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:46:47.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:47.544+0000 7fc2ba800700 1 --2- 172.21.15.62:0/3221431517 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc2b41004e0 0x7fc2b41980e0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:46:47.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:47.544+0000 7fc2ba800700 1 --2- 172.21.15.62:0/3221431517 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc2b4100e80 0x7fc2b4198620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:46:47.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:47.544+0000 7fc2ba800700 1 -- 172.21.15.62:0/3221431517 >> 172.21.15.62:0/3221431517 conn(0x7fc2b40fbda0 msgr2=0x7fc2b40fd3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:46:47.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:47.545+0000 7fc2ba800700 1 -- 172.21.15.62:0/3221431517 shutdown_connections 2024-04-20T21:46:47.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:47.545+0000 7fc2ba800700 1 -- 172.21.15.62:0/3221431517 wait complete. 2024-04-20T21:46:47.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:47.711+0000 7f9827e00700 1 -- 172.21.15.62:0/267544981 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9820073a00 msgr2=0x7f9820110d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:46:47.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:47.711+0000 7f9827e00700 1 --2- 172.21.15.62:0/267544981 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9820073a00 0x7f9820110d50 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f980c009910 tx=0x7f980c009c20 comp rx=0 tx=0).stop 2024-04-20T21:46:47.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:47.711+0000 7f9827e00700 1 -- 172.21.15.62:0/267544981 shutdown_connections 2024-04-20T21:46:47.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:47.711+0000 7f9827e00700 1 --2- 172.21.15.62:0/267544981 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9820073a00 0x7f9820110d50 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:46:47.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:47.711+0000 7f9827e00700 1 --2- 172.21.15.62:0/267544981 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98200730f0 0x7f98200734c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:46:47.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:47.711+0000 7f9827e00700 1 -- 172.21.15.62:0/267544981 >> 172.21.15.62:0/267544981 conn(0x7f98200fbda0 msgr2=0x7f98200fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:46:47.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:47.712+0000 7f9827e00700 1 -- 172.21.15.62:0/267544981 shutdown_connections 2024-04-20T21:46:47.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:47.712+0000 7f9827e00700 1 -- 172.21.15.62:0/267544981 wait complete. 2024-04-20T21:46:47.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:47.713+0000 7f9827e00700 1 Processor -- start 2024-04-20T21:46:47.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:47.713+0000 7f9827e00700 1 -- start start 2024-04-20T21:46:47.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:47.713+0000 7f9827e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98200730f0 0x7f982019c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:46:47.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:47.713+0000 7f9827e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9820073a00 0x7f982019c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:46:47.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:47.713+0000 7f9827e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f982019d050 con 0x7f9820073a00 2024-04-20T21:46:47.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:47.713+0000 7f9827e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98201a0de0 con 0x7f98200730f0 2024-04-20T21:46:47.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:47.713+0000 7f9824a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9820073a00 0x7f982019c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:46:47.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:47.714+0000 7f9824a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9820073a00 0x7f982019c970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56178/0 (socket says 172.21.15.62:56178) 2024-04-20T21:46:47.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:47.714+0000 7f9824a00700 1 -- 172.21.15.62:0/3800447227 learned_addr learned my addr 172.21.15.62:0/3800447227 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:46:47.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:47.714+0000 7f9825400700 1 --2- 172.21.15.62:0/3800447227 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98200730f0 0x7f982019c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:46:47.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:47.714+0000 7f9824a00700 1 -- 172.21.15.62:0/3800447227 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98200730f0 msgr2=0x7f982019c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:46:47.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:47.714+0000 7f9824a00700 1 --2- 172.21.15.62:0/3800447227 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98200730f0 0x7f982019c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:46:47.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:47.714+0000 7f9824a00700 1 -- 172.21.15.62:0/3800447227 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f980c0095f0 con 0x7f9820073a00 2024-04-20T21:46:47.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:47.714+0000 7f9824a00700 1 --2- 172.21.15.62:0/3800447227 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9820073a00 0x7f982019c970 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f980c006010 tx=0x7f980c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:46:47.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:47.715+0000 7f981e000700 1 -- 172.21.15.62:0/3800447227 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f980c01d070 con 0x7f9820073a00 2024-04-20T21:46:47.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:47.715+0000 7f981e000700 1 -- 172.21.15.62:0/3800447227 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f980c00f460 con 0x7f9820073a00 2024-04-20T21:46:47.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:47.715+0000 7f981e000700 1 -- 172.21.15.62:0/3800447227 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f980c017630 con 0x7f9820073a00 2024-04-20T21:46:47.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:47.715+0000 7f9827e00700 1 -- 172.21.15.62:0/3800447227 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98201a10c0 con 0x7f9820073a00 2024-04-20T21:46:47.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:47.715+0000 7f9827e00700 1 -- 172.21.15.62:0/3800447227 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98201a1610 con 0x7f9820073a00 2024-04-20T21:46:47.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:47.716+0000 7f9827e00700 1 -- 172.21.15.62:0/3800447227 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f982010e4d0 con 0x7f9820073a00 2024-04-20T21:46:47.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:47.717+0000 7f981e000700 1 -- 172.21.15.62:0/3800447227 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f980c00f5d0 con 0x7f9820073a00 2024-04-20T21:46:47.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:47.718+0000 7f981e000700 1 --2- 172.21.15.62:0/3800447227 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f980806ecc0 0x7f9808071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:46:47.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:47.718+0000 7f981e000700 1 -- 172.21.15.62:0/3800447227 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f980c090350 con 0x7f9820073a00 2024-04-20T21:46:47.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:47.718+0000 7f9825400700 1 --2- 172.21.15.62:0/3800447227 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f980806ecc0 0x7f9808071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:46:47.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:47.719+0000 7f9825400700 1 --2- 172.21.15.62:0/3800447227 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f980806ecc0 0x7f9808071170 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f9814005950 tx=0x7f98140058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:46:47.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:47.723+0000 7f981e000700 1 -- 172.21.15.62:0/3800447227 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f980c059060 con 0x7f9820073a00 2024-04-20T21:46:47.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:47.967+0000 7f9827e00700 1 -- 172.21.15.62:0/3800447227 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f98201a18a0 con 0x7f980806ecc0 2024-04-20T21:46:47.967 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:46:47 smithi062 ceph-mon[76322]: pgmap v344: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:46:47.986 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:46:47 smithi189 ceph-mon[57022]: pgmap v344: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:46:47.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.000+0000 7f981e000700 1 -- 172.21.15.62:0/3800447227 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f98201a18a0 con 0x7f980806ecc0 2024-04-20T21:46:48.000 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T21:46:48.000 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (12m) 4m ago 26m 14.2M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T21:46:48.000 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (26m) 4m ago 26m 9512k - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T21:46:48.000 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (25m) 4m ago 25m 8942k - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T21:46:48.000 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (8m) 4m ago 26m 6660k - 19.0.0-3133-ga8c84f59 76bc6b91b726 2825d5ba11c6 2024-04-20T21:46:48.000 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (8m) 4m ago 25m 6665k - 19.0.0-3133-ga8c84f59 76bc6b91b726 20f746fa1bba 2024-04-20T21:46:48.000 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (12m) 4m ago 26m 74.2M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T21:46:48.000 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (4m) 4m ago 19m 11.2M - 19.0.0-3133-ga8c84f59 76bc6b91b726 de80bc90a683 2024-04-20T21:46:48.000 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (4m) 4m ago 19m 11.9M - 19.0.0-3133-ga8c84f59 76bc6b91b726 cd793b7ee98e 2024-04-20T21:46:48.000 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (19m) 4m ago 19m 117M - 18.2.0 dc2bc1663786 fb2a8a223db8 2024-04-20T21:46:48.000 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (19m) 4m ago 19m 55.6M - 18.2.0 dc2bc1663786 f2eec2ee8951 2024-04-20T21:46:48.000 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (15m) 4m ago 28m 511M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T21:46:48.001 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (14m) 4m ago 25m 452M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T21:46:48.001 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (10m) 4m ago 28m 55.9M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 019eb1479485 2024-04-20T21:46:48.001 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (9m) 4m ago 25m 39.8M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 55aac560aab6 2024-04-20T21:46:48.001 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (13m) 4m ago 26m 7952k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T21:46:48.001 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (13m) 4m ago 25m 8304k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T21:46:48.001 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (8m) 4m ago 24m 88.0M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 ee2bb901f26e 2024-04-20T21:46:48.001 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (7m) 4m ago 23m 92.3M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 9075152d4162 2024-04-20T21:46:48.001 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (7m) 4m ago 22m 74.4M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 2d4f3e38dc0e 2024-04-20T21:46:48.001 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (6m) 4m ago 22m 87.6M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 8f97b17ae0eb 2024-04-20T21:46:48.001 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (5m) 4m ago 21m 90.7M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 653d8079967d 2024-04-20T21:46:48.001 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (5m) 4m ago 21m 11.5M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 596dded14c12 2024-04-20T21:46:48.001 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (13m) 4m ago 25m 74.0M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T21:46:48.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.004+0000 7f9827e00700 1 -- 172.21.15.62:0/3800447227 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f980806ecc0 msgr2=0x7f9808071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:46:48.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.004+0000 7f9827e00700 1 --2- 172.21.15.62:0/3800447227 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f980806ecc0 0x7f9808071170 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f9814005950 tx=0x7f98140058e0 comp rx=0 tx=0).stop 2024-04-20T21:46:48.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.004+0000 7f9827e00700 1 -- 172.21.15.62:0/3800447227 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9820073a00 msgr2=0x7f982019c970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:46:48.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.004+0000 7f9827e00700 1 --2- 172.21.15.62:0/3800447227 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9820073a00 0x7f982019c970 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f980c006010 tx=0x7f980c00be20 comp rx=0 tx=0).stop 2024-04-20T21:46:48.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.005+0000 7f9827e00700 1 -- 172.21.15.62:0/3800447227 shutdown_connections 2024-04-20T21:46:48.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.005+0000 7f9827e00700 1 --2- 172.21.15.62:0/3800447227 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f980806ecc0 0x7f9808071170 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:46:48.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.005+0000 7f9827e00700 1 --2- 172.21.15.62:0/3800447227 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98200730f0 0x7f982019c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:46:48.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.005+0000 7f9827e00700 1 --2- 172.21.15.62:0/3800447227 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9820073a00 0x7f982019c970 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:46:48.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.005+0000 7f9827e00700 1 -- 172.21.15.62:0/3800447227 >> 172.21.15.62:0/3800447227 conn(0x7f98200fbda0 msgr2=0x7f9820102860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:46:48.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.005+0000 7f9827e00700 1 -- 172.21.15.62:0/3800447227 shutdown_connections 2024-04-20T21:46:48.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.005+0000 7f9827e00700 1 -- 172.21.15.62:0/3800447227 wait complete. 2024-04-20T21:46:48.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.173+0000 7fe6d2200700 1 -- 172.21.15.62:0/2731627208 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6cc106b80 msgr2=0x7fe6cc106ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:46:48.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.173+0000 7fe6d2200700 1 --2- 172.21.15.62:0/2731627208 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6cc106b80 0x7fe6cc106ff0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fe6b8009910 tx=0x7fe6b8009c20 comp rx=0 tx=0).stop 2024-04-20T21:46:48.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.173+0000 7fe6d2200700 1 -- 172.21.15.62:0/2731627208 shutdown_connections 2024-04-20T21:46:48.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.173+0000 7fe6d2200700 1 --2- 172.21.15.62:0/2731627208 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6cc106b80 0x7fe6cc106ff0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:46:48.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.173+0000 7fe6d2200700 1 --2- 172.21.15.62:0/2731627208 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe6cc10c8f0 0x7fe6cc10ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:46:48.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.173+0000 7fe6d2200700 1 -- 172.21.15.62:0/2731627208 >> 172.21.15.62:0/2731627208 conn(0x7fe6cc076060 msgr2=0x7fe6cc078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:46:48.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.173+0000 7fe6d2200700 1 -- 172.21.15.62:0/2731627208 shutdown_connections 2024-04-20T21:46:48.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.173+0000 7fe6d2200700 1 -- 172.21.15.62:0/2731627208 wait complete. 2024-04-20T21:46:48.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.174+0000 7fe6d2200700 1 Processor -- start 2024-04-20T21:46:48.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.175+0000 7fe6d2200700 1 -- start start 2024-04-20T21:46:48.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.175+0000 7fe6d2200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe6cc106b80 0x7fe6cc19c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:46:48.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.175+0000 7fe6d2200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6cc10c8f0 0x7fe6cc19c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:46:48.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.175+0000 7fe6d2200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6cc19d060 con 0x7fe6cc10c8f0 2024-04-20T21:46:48.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.175+0000 7fe6d2200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6cc1a0df0 con 0x7fe6cc106b80 2024-04-20T21:46:48.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.175+0000 7fe6caa00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6cc10c8f0 0x7fe6cc19c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:46:48.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.175+0000 7fe6caa00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6cc10c8f0 0x7fe6cc19c980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56184/0 (socket says 172.21.15.62:56184) 2024-04-20T21:46:48.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.175+0000 7fe6caa00700 1 -- 172.21.15.62:0/656916327 learned_addr learned my addr 172.21.15.62:0/656916327 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:46:48.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.176+0000 7fe6cb400700 1 --2- 172.21.15.62:0/656916327 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe6cc106b80 0x7fe6cc19c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:46:48.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.176+0000 7fe6caa00700 1 -- 172.21.15.62:0/656916327 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe6cc106b80 msgr2=0x7fe6cc19c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:46:48.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.176+0000 7fe6caa00700 1 --2- 172.21.15.62:0/656916327 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe6cc106b80 0x7fe6cc19c440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:46:48.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.176+0000 7fe6caa00700 1 -- 172.21.15.62:0/656916327 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6b80095f0 con 0x7fe6cc10c8f0 2024-04-20T21:46:48.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.176+0000 7fe6cb400700 1 --2- 172.21.15.62:0/656916327 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe6cc106b80 0x7fe6cc19c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T21:46:48.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.176+0000 7fe6caa00700 1 --2- 172.21.15.62:0/656916327 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6cc10c8f0 0x7fe6cc19c980 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fe6b80092e0 tx=0x7fe6b800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:46:48.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.176+0000 7fe6c3e00700 1 -- 172.21.15.62:0/656916327 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe6b801d070 con 0x7fe6cc10c8f0 2024-04-20T21:46:48.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.176+0000 7fe6c3e00700 1 -- 172.21.15.62:0/656916327 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe6b8005e00 con 0x7fe6cc10c8f0 2024-04-20T21:46:48.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.176+0000 7fe6d2200700 1 -- 172.21.15.62:0/656916327 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6cc1a1070 con 0x7fe6cc10c8f0 2024-04-20T21:46:48.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.176+0000 7fe6c3e00700 1 -- 172.21.15.62:0/656916327 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe6b8017400 con 0x7fe6cc10c8f0 2024-04-20T21:46:48.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.177+0000 7fe6d2200700 1 -- 172.21.15.62:0/656916327 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6cc1a1560 con 0x7fe6cc10c8f0 2024-04-20T21:46:48.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.178+0000 7fe6d2200700 1 -- 172.21.15.62:0/656916327 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6cc10ecc0 con 0x7fe6cc10c8f0 2024-04-20T21:46:48.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.179+0000 7fe6c3e00700 1 -- 172.21.15.62:0/656916327 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe6b800f460 con 0x7fe6cc10c8f0 2024-04-20T21:46:48.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.180+0000 7fe6c3e00700 1 --2- 172.21.15.62:0/656916327 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe6b006ecc0 0x7fe6b0071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:46:48.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.180+0000 7fe6c3e00700 1 -- 172.21.15.62:0/656916327 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fe6b8090430 con 0x7fe6cc10c8f0 2024-04-20T21:46:48.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.180+0000 7fe6cb400700 1 --2- 172.21.15.62:0/656916327 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe6b006ecc0 0x7fe6b0071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:46:48.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.181+0000 7fe6cb400700 1 --2- 172.21.15.62:0/656916327 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe6b006ecc0 0x7fe6b0071170 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fe6bc005fd0 tx=0x7fe6bc005ee0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:46:48.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.185+0000 7fe6c3e00700 1 -- 172.21.15.62:0/656916327 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fe6b8059140 con 0x7fe6cc10c8f0 2024-04-20T21:46:48.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.512+0000 7fe6d2200700 1 -- 172.21.15.62:0/656916327 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fe6cc10e6a0 con 0x7fe6cc10c8f0 2024-04-20T21:46:48.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.514+0000 7fe6c3e00700 1 -- 172.21.15.62:0/656916327 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7fe6b805c800 con 0x7fe6cc10c8f0 2024-04-20T21:46:48.513 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:46:48.513 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T21:46:48.514 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:46:48.514 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:46:48.514 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T21:46:48.514 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:46:48.514 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:46:48.514 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T21:46:48.514 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 6 2024-04-20T21:46:48.514 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:46:48.514 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T21:46:48.514 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 2, 2024-04-20T21:46:48.514 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:46:48.514 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:46:48.514 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T21:46:48.514 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 2, 2024-04-20T21:46:48.514 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 12 2024-04-20T21:46:48.515 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T21:46:48.515 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:46:48.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.518+0000 7fe6d2200700 1 -- 172.21.15.62:0/656916327 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe6b006ecc0 msgr2=0x7fe6b0071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:46:48.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.518+0000 7fe6d2200700 1 --2- 172.21.15.62:0/656916327 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe6b006ecc0 0x7fe6b0071170 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fe6bc005fd0 tx=0x7fe6bc005ee0 comp rx=0 tx=0).stop 2024-04-20T21:46:48.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.518+0000 7fe6d2200700 1 -- 172.21.15.62:0/656916327 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6cc10c8f0 msgr2=0x7fe6cc19c980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:46:48.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.518+0000 7fe6d2200700 1 --2- 172.21.15.62:0/656916327 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6cc10c8f0 0x7fe6cc19c980 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fe6b80092e0 tx=0x7fe6b800be20 comp rx=0 tx=0).stop 2024-04-20T21:46:48.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.518+0000 7fe6d2200700 1 -- 172.21.15.62:0/656916327 shutdown_connections 2024-04-20T21:46:48.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.518+0000 7fe6d2200700 1 --2- 172.21.15.62:0/656916327 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe6b006ecc0 0x7fe6b0071170 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:46:48.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.518+0000 7fe6d2200700 1 --2- 172.21.15.62:0/656916327 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe6cc106b80 0x7fe6cc19c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:46:48.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.518+0000 7fe6d2200700 1 --2- 172.21.15.62:0/656916327 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6cc10c8f0 0x7fe6cc19c980 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:46:48.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.518+0000 7fe6d2200700 1 -- 172.21.15.62:0/656916327 >> 172.21.15.62:0/656916327 conn(0x7fe6cc076060 msgr2=0x7fe6cc077ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:46:48.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.518+0000 7fe6d2200700 1 -- 172.21.15.62:0/656916327 shutdown_connections 2024-04-20T21:46:48.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.518+0000 7fe6d2200700 1 -- 172.21.15.62:0/656916327 wait complete. 2024-04-20T21:46:48.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.681+0000 7f8f12200700 1 -- 172.21.15.62:0/1655988651 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8f0c106b80 msgr2=0x7f8f0c106ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:46:48.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.681+0000 7f8f12200700 1 --2- 172.21.15.62:0/1655988651 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8f0c106b80 0x7f8f0c106ff0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f8ef8009910 tx=0x7f8ef8009c20 comp rx=0 tx=0).stop 2024-04-20T21:46:48.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.681+0000 7f8f12200700 1 -- 172.21.15.62:0/1655988651 shutdown_connections 2024-04-20T21:46:48.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.681+0000 7f8f12200700 1 --2- 172.21.15.62:0/1655988651 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8f0c106b80 0x7f8f0c106ff0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:46:48.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.681+0000 7f8f12200700 1 --2- 172.21.15.62:0/1655988651 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8f0c10c660 0x7f8f0c10ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:46:48.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.681+0000 7f8f12200700 1 -- 172.21.15.62:0/1655988651 >> 172.21.15.62:0/1655988651 conn(0x7f8f0c075e90 msgr2=0x7f8f0c0782a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:46:48.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.681+0000 7f8f12200700 1 -- 172.21.15.62:0/1655988651 shutdown_connections 2024-04-20T21:46:48.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.681+0000 7f8f12200700 1 -- 172.21.15.62:0/1655988651 wait complete. 2024-04-20T21:46:48.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.682+0000 7f8f12200700 1 Processor -- start 2024-04-20T21:46:48.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.682+0000 7f8f12200700 1 -- start start 2024-04-20T21:46:48.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.682+0000 7f8f12200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8f0c106b80 0x7f8f0c19c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:46:48.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.682+0000 7f8f12200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8f0c10c660 0x7f8f0c19c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:46:48.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.683+0000 7f8f12200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f0c19cf60 con 0x7f8f0c10c660 2024-04-20T21:46:48.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.683+0000 7f8f12200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f0c1a0cf0 con 0x7f8f0c106b80 2024-04-20T21:46:48.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.683+0000 7f8f0be00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8f0c10c660 0x7f8f0c19c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:46:48.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.683+0000 7f8f0be00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8f0c10c660 0x7f8f0c19c880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56212/0 (socket says 172.21.15.62:56212) 2024-04-20T21:46:48.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.683+0000 7f8f0be00700 1 -- 172.21.15.62:0/4130378721 learned_addr learned my addr 172.21.15.62:0/4130378721 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:46:48.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.683+0000 7f8f10e00700 1 --2- 172.21.15.62:0/4130378721 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8f0c106b80 0x7f8f0c19c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:46:48.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.683+0000 7f8f0be00700 1 -- 172.21.15.62:0/4130378721 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8f0c106b80 msgr2=0x7f8f0c19c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:46:48.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.683+0000 7f8f0be00700 1 --2- 172.21.15.62:0/4130378721 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8f0c106b80 0x7f8f0c19c340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:46:48.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.683+0000 7f8f0be00700 1 -- 172.21.15.62:0/4130378721 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ef80095f0 con 0x7f8f0c10c660 2024-04-20T21:46:48.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.684+0000 7f8f0be00700 1 --2- 172.21.15.62:0/4130378721 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8f0c10c660 0x7f8f0c19c880 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f8ef8006010 tx=0x7f8ef800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:46:48.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.684+0000 7f8f09600700 1 -- 172.21.15.62:0/4130378721 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8ef801d070 con 0x7f8f0c10c660 2024-04-20T21:46:48.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.684+0000 7f8f12200700 1 -- 172.21.15.62:0/4130378721 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f0c1a0f70 con 0x7f8f0c10c660 2024-04-20T21:46:48.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.684+0000 7f8f09600700 1 -- 172.21.15.62:0/4130378721 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8ef8004fb0 con 0x7f8f0c10c660 2024-04-20T21:46:48.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.684+0000 7f8f12200700 1 -- 172.21.15.62:0/4130378721 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f0c1a1460 con 0x7f8f0c10c660 2024-04-20T21:46:48.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.684+0000 7f8f09600700 1 -- 172.21.15.62:0/4130378721 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8ef800f630 con 0x7f8f0c10c660 2024-04-20T21:46:48.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.685+0000 7f8f12200700 1 -- 172.21.15.62:0/4130378721 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f0c10eb60 con 0x7f8f0c10c660 2024-04-20T21:46:48.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.686+0000 7f8f09600700 1 -- 172.21.15.62:0/4130378721 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f8ef80228a0 con 0x7f8f0c10c660 2024-04-20T21:46:48.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.687+0000 7f8f09600700 1 --2- 172.21.15.62:0/4130378721 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f8ef406ecc0 0x7f8ef4071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:46:48.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.687+0000 7f8f09600700 1 -- 172.21.15.62:0/4130378721 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f8ef8090510 con 0x7f8f0c10c660 2024-04-20T21:46:48.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.687+0000 7f8f10e00700 1 --2- 172.21.15.62:0/4130378721 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f8ef406ecc0 0x7f8ef4071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:46:48.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.688+0000 7f8f10e00700 1 --2- 172.21.15.62:0/4130378721 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f8ef406ecc0 0x7f8ef4071170 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f8f00006fd0 tx=0x7f8f00008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:46:48.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.692+0000 7f8f09600700 1 -- 172.21.15.62:0/4130378721 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f8ef80592d0 con 0x7f8f0c10c660 2024-04-20T21:46:48.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:46:48 smithi062 ceph-mon[76322]: from='client.34676 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:46:48.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:46:48 smithi062 ceph-mon[76322]: from='client.34680 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:46:48.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:46:48 smithi062 ceph-mon[76322]: from='client.34684 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:46:48.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:46:48 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/656916327' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:46:48.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.982+0000 7f8f12200700 1 -- 172.21.15.62:0/4130378721 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f8f0c062380 con 0x7f8f0c10c660 2024-04-20T21:46:48.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.983+0000 7f8f09600700 1 -- 172.21.15.62:0/4130378721 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 25 v25) v1 ==== 76+0+1887 (secure 0 0 0) 0x7f8ef805c990 con 0x7f8f0c10c660 2024-04-20T21:46:48.983 INFO:teuthology.orchestra.run.smithi062.stdout:e25 2024-04-20T21:46:48.983 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T21:46:48.983 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T21:46:48.983 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T21:46:48.983 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:46:48.983 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T21:46:48.983 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T21:46:48.983 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 20 2024-04-20T21:46:48.984 INFO:teuthology.orchestra.run.smithi062.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-20T21:46:48.984 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T21:46:48.984 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T21:42:01.559061+0000 2024-04-20T21:46:48.984 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T21:46:48.984 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T21:46:48.984 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T21:46:48.984 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T21:46:48.984 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T21:46:48.984 INFO:teuthology.orchestra.run.smithi062.stdout:max_xattr_size 65536 2024-04-20T21:46:48.984 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T21:46:48.984 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T21:46:48.984 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 81 2024-04-20T21:46:48.984 INFO:teuthology.orchestra.run.smithi062.stdout:snap_rstat 1 2024-04-20T21:46:48.984 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T21:46:48.984 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 2 2024-04-20T21:46:48.984 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1 2024-04-20T21:46:48.984 INFO:teuthology.orchestra.run.smithi062.stdout:up {} 2024-04-20T21:46:48.984 INFO:teuthology.orchestra.run.smithi062.stdout:failed 0,1 2024-04-20T21:46:48.985 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T21:46:48.985 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T21:46:48.985 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T21:46:48.985 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T21:46:48.985 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data enabled 2024-04-20T21:46:48.985 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T21:46:48.985 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T21:46:48.985 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 1 2024-04-20T21:46:48.985 INFO:teuthology.orchestra.run.smithi062.stdout:qdb_cluster leader: 0 members: 2024-04-20T21:46:48.985 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:46:48.985 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:46:48.985 INFO:teuthology.orchestra.run.smithi062.stdout:Standby daemons: 2024-04-20T21:46:48.985 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:46:48.985 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ryqblb{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6826/4113916205,v1:172.21.15.62:6827/4113916205] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T21:46:48.985 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ozizwi{-1:34476} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6828/3801975165,v1:172.21.15.62:6829/3801975165] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T21:46:48.986 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.xbgspw{-1:44285} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6826/3996040526,v1:172.21.15.189:6827/3996040526] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:46:48.986 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.dvimer{-1:44289} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6824/1743111843,v1:172.21.15.189:6825/1743111843] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:46:48.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.986+0000 7f8f12200700 1 -- 172.21.15.62:0/4130378721 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f8ef406ecc0 msgr2=0x7f8ef4071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:46:48.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.986+0000 7f8f12200700 1 --2- 172.21.15.62:0/4130378721 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f8ef406ecc0 0x7f8ef4071170 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f8f00006fd0 tx=0x7f8f00008040 comp rx=0 tx=0).stop 2024-04-20T21:46:48.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.986+0000 7f8f12200700 1 -- 172.21.15.62:0/4130378721 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8f0c10c660 msgr2=0x7f8f0c19c880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:46:48.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.986+0000 7f8f12200700 1 --2- 172.21.15.62:0/4130378721 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8f0c10c660 0x7f8f0c19c880 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f8ef8006010 tx=0x7f8ef800be20 comp rx=0 tx=0).stop 2024-04-20T21:46:48.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.986+0000 7f8f12200700 1 -- 172.21.15.62:0/4130378721 shutdown_connections 2024-04-20T21:46:48.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.986+0000 7f8f12200700 1 --2- 172.21.15.62:0/4130378721 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f8ef406ecc0 0x7f8ef4071170 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:46:48.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.986+0000 7f8f12200700 1 --2- 172.21.15.62:0/4130378721 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8f0c106b80 0x7f8f0c19c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:46:48.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.986+0000 7f8f12200700 1 --2- 172.21.15.62:0/4130378721 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8f0c10c660 0x7f8f0c19c880 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:46:48.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.986+0000 7f8f12200700 1 -- 172.21.15.62:0/4130378721 >> 172.21.15.62:0/4130378721 conn(0x7f8f0c075e90 msgr2=0x7f8f0c077b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:46:48.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.986+0000 7f8f12200700 1 -- 172.21.15.62:0/4130378721 shutdown_connections 2024-04-20T21:46:48.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:48.986+0000 7f8f12200700 1 -- 172.21.15.62:0/4130378721 wait complete. 2024-04-20T21:46:48.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:46:48 smithi189 ceph-mon[57022]: from='client.34676 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:46:48.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:46:48 smithi189 ceph-mon[57022]: from='client.34680 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:46:48.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:46:48 smithi189 ceph-mon[57022]: from='client.34684 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:46:48.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:46:48 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/656916327' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:46:48.988 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 25 2024-04-20T21:46:49.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:49.140+0000 7fa449a00700 1 -- 172.21.15.62:0/887939845 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa444103040 msgr2=0x7fa444107010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:46:49.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:49.140+0000 7fa449a00700 1 --2- 172.21.15.62:0/887939845 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa444103040 0x7fa444107010 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fa430009910 tx=0x7fa430009c20 comp rx=0 tx=0).stop 2024-04-20T21:46:49.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:49.141+0000 7fa449a00700 1 -- 172.21.15.62:0/887939845 shutdown_connections 2024-04-20T21:46:49.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:49.141+0000 7fa449a00700 1 --2- 172.21.15.62:0/887939845 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa444103040 0x7fa444107010 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:46:49.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:49.141+0000 7fa449a00700 1 --2- 172.21.15.62:0/887939845 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4441026a0 0x7fa444102a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:46:49.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:49.141+0000 7fa449a00700 1 -- 172.21.15.62:0/887939845 >> 172.21.15.62:0/887939845 conn(0x7fa4440fdf40 msgr2=0x7fa444100350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:46:49.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:49.141+0000 7fa449a00700 1 -- 172.21.15.62:0/887939845 shutdown_connections 2024-04-20T21:46:49.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:49.141+0000 7fa449a00700 1 -- 172.21.15.62:0/887939845 wait complete. 2024-04-20T21:46:49.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:49.142+0000 7fa449a00700 1 Processor -- start 2024-04-20T21:46:49.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:49.142+0000 7fa449a00700 1 -- start start 2024-04-20T21:46:49.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:49.142+0000 7fa449a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4441026a0 0x7fa444198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:46:49.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:49.142+0000 7fa449a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa444103040 0x7fa4441985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:46:49.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:49.142+0000 7fa449a00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa444198c20 con 0x7fa444103040 2024-04-20T21:46:49.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:49.142+0000 7fa449a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa444198d60 con 0x7fa4441026a0 2024-04-20T21:46:49.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:49.143+0000 7fa442000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa444103040 0x7fa4441985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:46:49.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:49.143+0000 7fa442000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa444103040 0x7fa4441985d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56232/0 (socket says 172.21.15.62:56232) 2024-04-20T21:46:49.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:49.143+0000 7fa442000700 1 -- 172.21.15.62:0/198767554 learned_addr learned my addr 172.21.15.62:0/198767554 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:46:49.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:49.143+0000 7fa442a00700 1 --2- 172.21.15.62:0/198767554 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4441026a0 0x7fa444198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:46:49.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:49.143+0000 7fa442000700 1 -- 172.21.15.62:0/198767554 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4441026a0 msgr2=0x7fa444198090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:46:49.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:49.143+0000 7fa442000700 1 --2- 172.21.15.62:0/198767554 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4441026a0 0x7fa444198090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:46:49.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:49.143+0000 7fa442000700 1 -- 172.21.15.62:0/198767554 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4300095f0 con 0x7fa444103040 2024-04-20T21:46:49.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:49.144+0000 7fa442000700 1 --2- 172.21.15.62:0/198767554 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa444103040 0x7fa4441985d0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fa43000bda0 tx=0x7fa43000be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:46:49.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:49.144+0000 7fa43b400700 1 -- 172.21.15.62:0/198767554 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa43001d070 con 0x7fa444103040 2024-04-20T21:46:49.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:49.144+0000 7fa43b400700 1 -- 172.21.15.62:0/198767554 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa430004fb0 con 0x7fa444103040 2024-04-20T21:46:49.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:49.144+0000 7fa449a00700 1 -- 172.21.15.62:0/198767554 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa44419cb50 con 0x7fa444103040 2024-04-20T21:46:49.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:49.144+0000 7fa43b400700 1 -- 172.21.15.62:0/198767554 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa43000f480 con 0x7fa444103040 2024-04-20T21:46:49.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:49.144+0000 7fa449a00700 1 -- 172.21.15.62:0/198767554 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa44419d040 con 0x7fa444103040 2024-04-20T21:46:49.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:49.145+0000 7fa449a00700 1 -- 172.21.15.62:0/198767554 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa44410aa10 con 0x7fa444103040 2024-04-20T21:46:49.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:49.147+0000 7fa43b400700 1 -- 172.21.15.62:0/198767554 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fa430022870 con 0x7fa444103040 2024-04-20T21:46:49.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:49.148+0000 7fa43b400700 1 --2- 172.21.15.62:0/198767554 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa42806ecc0 0x7fa428071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:46:49.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:49.148+0000 7fa43b400700 1 -- 172.21.15.62:0/198767554 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fa430090350 con 0x7fa444103040 2024-04-20T21:46:49.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:49.148+0000 7fa442a00700 1 --2- 172.21.15.62:0/198767554 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa42806ecc0 0x7fa428071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:46:49.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:49.149+0000 7fa442a00700 1 --2- 172.21.15.62:0/198767554 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa42806ecc0 0x7fa428071170 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fa434005fd0 tx=0x7fa434005e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:46:49.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:49.153+0000 7fa43b400700 1 -- 172.21.15.62:0/198767554 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fa430059190 con 0x7fa444103040 2024-04-20T21:46:49.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:49.396+0000 7fa449a00700 1 -- 172.21.15.62:0/198767554 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa44402cca0 con 0x7fa42806ecc0 2024-04-20T21:46:49.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:49.402+0000 7fa43b400700 1 -- 172.21.15.62:0/198767554 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fa44402cca0 con 0x7fa42806ecc0 2024-04-20T21:46:49.401 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:46:49.401 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T21:46:49.402 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T21:46:49.402 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-20T21:46:49.402 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T21:46:49.402 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr", 2024-04-20T21:46:49.402 INFO:teuthology.orchestra.run.smithi062.stdout: "crash", 2024-04-20T21:46:49.402 INFO:teuthology.orchestra.run.smithi062.stdout: "mon", 2024-04-20T21:46:49.403 INFO:teuthology.orchestra.run.smithi062.stdout: "osd" 2024-04-20T21:46:49.403 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T21:46:49.403 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "14/23 daemons upgraded", 2024-04-20T21:46:49.403 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading mds daemons", 2024-04-20T21:46:49.403 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T21:46:49.403 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:46:49.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:49.404+0000 7fa449a00700 1 -- 172.21.15.62:0/198767554 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa42806ecc0 msgr2=0x7fa428071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:46:49.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:49.405+0000 7fa449a00700 1 --2- 172.21.15.62:0/198767554 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa42806ecc0 0x7fa428071170 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fa434005fd0 tx=0x7fa434005e30 comp rx=0 tx=0).stop 2024-04-20T21:46:49.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:49.405+0000 7fa449a00700 1 -- 172.21.15.62:0/198767554 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa444103040 msgr2=0x7fa4441985d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:46:49.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:49.405+0000 7fa449a00700 1 --2- 172.21.15.62:0/198767554 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa444103040 0x7fa4441985d0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fa43000bda0 tx=0x7fa43000be80 comp rx=0 tx=0).stop 2024-04-20T21:46:49.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:49.405+0000 7fa449a00700 1 -- 172.21.15.62:0/198767554 shutdown_connections 2024-04-20T21:46:49.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:49.405+0000 7fa449a00700 1 --2- 172.21.15.62:0/198767554 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa42806ecc0 0x7fa428071170 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:46:49.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:49.405+0000 7fa449a00700 1 --2- 172.21.15.62:0/198767554 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4441026a0 0x7fa444198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:46:49.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:49.405+0000 7fa449a00700 1 --2- 172.21.15.62:0/198767554 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa444103040 0x7fa4441985d0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:46:49.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:49.405+0000 7fa449a00700 1 -- 172.21.15.62:0/198767554 >> 172.21.15.62:0/198767554 conn(0x7fa4440fdf40 msgr2=0x7fa444100340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:46:49.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:49.405+0000 7fa449a00700 1 -- 172.21.15.62:0/198767554 shutdown_connections 2024-04-20T21:46:49.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:49.405+0000 7fa449a00700 1 -- 172.21.15.62:0/198767554 wait complete. 2024-04-20T21:46:49.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:49.569+0000 7fd782200700 1 -- 172.21.15.62:0/3466448994 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd77c100e30 msgr2=0x7fd77c101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:46:49.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:49.569+0000 7fd782200700 1 --2- 172.21.15.62:0/3466448994 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd77c100e30 0x7fd77c101200 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fd770009910 tx=0x7fd770009c20 comp rx=0 tx=0).stop 2024-04-20T21:46:49.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:49.569+0000 7fd782200700 1 -- 172.21.15.62:0/3466448994 shutdown_connections 2024-04-20T21:46:49.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:49.569+0000 7fd782200700 1 --2- 172.21.15.62:0/3466448994 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd77c068540 0x7fd77c0689b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:46:49.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:49.569+0000 7fd782200700 1 --2- 172.21.15.62:0/3466448994 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd77c100e30 0x7fd77c101200 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:46:49.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:49.569+0000 7fd782200700 1 -- 172.21.15.62:0/3466448994 >> 172.21.15.62:0/3466448994 conn(0x7fd77c074fa0 msgr2=0x7fd77c0753a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:46:49.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:49.570+0000 7fd782200700 1 -- 172.21.15.62:0/3466448994 shutdown_connections 2024-04-20T21:46:49.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:49.570+0000 7fd782200700 1 -- 172.21.15.62:0/3466448994 wait complete. 2024-04-20T21:46:49.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:49.571+0000 7fd782200700 1 Processor -- start 2024-04-20T21:46:49.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:49.571+0000 7fd782200700 1 -- start start 2024-04-20T21:46:49.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:49.571+0000 7fd782200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd77c068540 0x7fd77c197da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:46:49.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:49.571+0000 7fd782200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd77c100e30 0x7fd77c1982e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:46:49.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:49.571+0000 7fd782200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd77c1989c0 con 0x7fd77c100e30 2024-04-20T21:46:49.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:49.571+0000 7fd782200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd77c19c750 con 0x7fd77c068540 2024-04-20T21:46:49.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:49.572+0000 7fd77be00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd77c100e30 0x7fd77c1982e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:46:49.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:49.572+0000 7fd77be00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd77c100e30 0x7fd77c1982e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56252/0 (socket says 172.21.15.62:56252) 2024-04-20T21:46:49.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:49.572+0000 7fd77be00700 1 -- 172.21.15.62:0/3639515673 learned_addr learned my addr 172.21.15.62:0/3639515673 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:46:49.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:49.572+0000 7fd780e00700 1 --2- 172.21.15.62:0/3639515673 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd77c068540 0x7fd77c197da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:46:49.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:49.572+0000 7fd77be00700 1 -- 172.21.15.62:0/3639515673 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd77c068540 msgr2=0x7fd77c197da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:46:49.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:49.572+0000 7fd77be00700 1 --2- 172.21.15.62:0/3639515673 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd77c068540 0x7fd77c197da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:46:49.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:49.572+0000 7fd77be00700 1 -- 172.21.15.62:0/3639515673 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7700095f0 con 0x7fd77c100e30 2024-04-20T21:46:49.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:49.572+0000 7fd780e00700 1 --2- 172.21.15.62:0/3639515673 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd77c068540 0x7fd77c197da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-20T21:46:49.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:49.573+0000 7fd77be00700 1 --2- 172.21.15.62:0/3639515673 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd77c100e30 0x7fd77c1982e0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fd76800d8d0 tx=0x7fd76800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:46:49.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:49.573+0000 7fd779600700 1 -- 172.21.15.62:0/3639515673 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd768009940 con 0x7fd77c100e30 2024-04-20T21:46:49.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:49.573+0000 7fd779600700 1 -- 172.21.15.62:0/3639515673 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd768010460 con 0x7fd77c100e30 2024-04-20T21:46:49.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:49.573+0000 7fd782200700 1 -- 172.21.15.62:0/3639515673 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd77c19ca30 con 0x7fd77c100e30 2024-04-20T21:46:49.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:49.573+0000 7fd779600700 1 -- 172.21.15.62:0/3639515673 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd76800f5f0 con 0x7fd77c100e30 2024-04-20T21:46:49.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:49.573+0000 7fd782200700 1 -- 172.21.15.62:0/3639515673 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd77c19d000 con 0x7fd77c100e30 2024-04-20T21:46:49.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:49.575+0000 7fd782200700 1 -- 172.21.15.62:0/3639515673 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd77c062380 con 0x7fd77c100e30 2024-04-20T21:46:49.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:49.578+0000 7fd779600700 1 -- 172.21.15.62:0/3639515673 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fd7680105d0 con 0x7fd77c100e30 2024-04-20T21:46:49.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:49.579+0000 7fd779600700 1 --2- 172.21.15.62:0/3639515673 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd76406ecc0 0x7fd764071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:46:49.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:49.579+0000 7fd779600700 1 -- 172.21.15.62:0/3639515673 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fd76808ee00 con 0x7fd77c100e30 2024-04-20T21:46:49.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:49.579+0000 7fd780e00700 1 --2- 172.21.15.62:0/3639515673 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd76406ecc0 0x7fd764071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:46:49.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:49.580+0000 7fd780e00700 1 --2- 172.21.15.62:0/3639515673 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd76406ecc0 0x7fd764071170 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fd77000b5c0 tx=0x7fd77001a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:46:49.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:49.582+0000 7fd779600700 1 -- 172.21.15.62:0/3639515673 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fd768057cc0 con 0x7fd77c100e30 2024-04-20T21:46:49.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:46:49 smithi062 ceph-mon[76322]: pgmap v345: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:46:49.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:46:49 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/4130378721' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T21:46:49.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:46:49 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:46:49.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:49.908+0000 7fd782200700 1 -- 172.21.15.62:0/3639515673 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fd77c199100 con 0x7fd77c100e30 2024-04-20T21:46:49.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:49.909+0000 7fd779600700 1 -- 172.21.15.62:0/3639515673 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7fd7680201c0 con 0x7fd77c100e30 2024-04-20T21:46:49.908 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-20T21:46:49.908 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-20T21:46:49.908 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is degraded 2024-04-20T21:46:49.909 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T21:46:49.909 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T21:46:49.909 INFO:teuthology.orchestra.run.smithi062.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-20T21:46:49.909 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-20T21:46:49.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:49.912+0000 7fd782200700 1 -- 172.21.15.62:0/3639515673 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd76406ecc0 msgr2=0x7fd764071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:46:49.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:49.912+0000 7fd782200700 1 --2- 172.21.15.62:0/3639515673 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd76406ecc0 0x7fd764071170 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fd77000b5c0 tx=0x7fd77001a040 comp rx=0 tx=0).stop 2024-04-20T21:46:49.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:49.912+0000 7fd782200700 1 -- 172.21.15.62:0/3639515673 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd77c100e30 msgr2=0x7fd77c1982e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:46:49.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:49.912+0000 7fd782200700 1 --2- 172.21.15.62:0/3639515673 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd77c100e30 0x7fd77c1982e0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fd76800d8d0 tx=0x7fd76800dc90 comp rx=0 tx=0).stop 2024-04-20T21:46:49.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:49.912+0000 7fd782200700 1 -- 172.21.15.62:0/3639515673 shutdown_connections 2024-04-20T21:46:49.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:49.912+0000 7fd782200700 1 --2- 172.21.15.62:0/3639515673 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd76406ecc0 0x7fd764071170 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:46:49.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:49.912+0000 7fd782200700 1 --2- 172.21.15.62:0/3639515673 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd77c068540 0x7fd77c197da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:46:49.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:49.912+0000 7fd782200700 1 --2- 172.21.15.62:0/3639515673 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd77c100e30 0x7fd77c1982e0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:46:49.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:49.912+0000 7fd782200700 1 -- 172.21.15.62:0/3639515673 >> 172.21.15.62:0/3639515673 conn(0x7fd77c074fa0 msgr2=0x7fd77c0fd8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:46:49.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:49.912+0000 7fd782200700 1 -- 172.21.15.62:0/3639515673 shutdown_connections 2024-04-20T21:46:49.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:46:49.913+0000 7fd782200700 1 -- 172.21.15.62:0/3639515673 wait complete. 2024-04-20T21:46:49.986 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:46:49 smithi189 ceph-mon[57022]: pgmap v345: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:46:49.986 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:46:49 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/4130378721' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T21:46:49.986 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:46:49 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:46:50.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:46:50 smithi189 ceph-mon[57022]: from='client.34696 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:46:50.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:46:50 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/3639515673' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T21:46:51.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:46:50 smithi062 ceph-mon[76322]: from='client.34696 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:46:51.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:46:50 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/3639515673' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T21:46:51.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:46:51 smithi189 ceph-mon[57022]: pgmap v346: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:46:52.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:46:51 smithi062 ceph-mon[76322]: pgmap v346: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:46:53.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:46:53 smithi189 ceph-mon[57022]: pgmap v347: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:46:54.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:46:53 smithi062 ceph-mon[76322]: pgmap v347: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:46:55.986 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:46:55 smithi189 ceph-mon[57022]: pgmap v348: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:46:56.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:46:55 smithi062 ceph-mon[76322]: pgmap v348: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:46:57.784 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:46:57 smithi062 ceph-mon[76322]: pgmap v349: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:46:57.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:46:57 smithi189 ceph-mon[57022]: pgmap v349: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:46:59.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:46:59 smithi189 ceph-mon[57022]: pgmap v350: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:47:00.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:46:59 smithi062 ceph-mon[76322]: pgmap v350: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:47:01.723 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:47:01 smithi062 ceph-mon[76322]: pgmap v351: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:47:01.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:47:01 smithi189 ceph-mon[57022]: pgmap v351: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:47:03.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:47:03 smithi189 ceph-mon[57022]: pgmap v352: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:47:04.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:47:03 smithi062 ceph-mon[76322]: pgmap v352: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:47:04.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:47:04 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:47:05.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:47:04 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:47:05.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:47:05 smithi189 ceph-mon[57022]: pgmap v353: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:47:06.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:47:05 smithi062 ceph-mon[76322]: pgmap v353: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:47:07.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:47:07 smithi189 ceph-mon[57022]: pgmap v354: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:47:08.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:47:07 smithi062 ceph-mon[76322]: pgmap v354: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:47:09.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:47:09 smithi189 ceph-mon[57022]: pgmap v355: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:47:10.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:47:09 smithi062 ceph-mon[76322]: pgmap v355: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:47:11.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:47:11 smithi189 ceph-mon[57022]: pgmap v356: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:47:12.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:47:11 smithi062 ceph-mon[76322]: pgmap v356: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:47:13.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:47:13 smithi189 ceph-mon[57022]: pgmap v357: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:47:14.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:47:13 smithi062 ceph-mon[76322]: pgmap v357: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:47:15.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:47:15 smithi189 ceph-mon[57022]: pgmap v358: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:47:16.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:47:15 smithi062 ceph-mon[76322]: pgmap v358: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:47:17.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:47:17 smithi189 ceph-mon[57022]: pgmap v359: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:47:18.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:47:17 smithi062 ceph-mon[76322]: pgmap v359: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:47:20.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:47:19 smithi062 ceph-mon[76322]: pgmap v360: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:47:20.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:47:19 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:47:20.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.092+0000 7fc550e00700 1 -- 172.21.15.62:0/4106308873 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc54c102f50 msgr2=0x7fc54c106f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:47:20.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.092+0000 7fc550e00700 1 --2- 172.21.15.62:0/4106308873 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc54c102f50 0x7fc54c106f20 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fc534009910 tx=0x7fc534009c20 comp rx=0 tx=0).stop 2024-04-20T21:47:20.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.092+0000 7fc550e00700 1 -- 172.21.15.62:0/4106308873 shutdown_connections 2024-04-20T21:47:20.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.092+0000 7fc550e00700 1 --2- 172.21.15.62:0/4106308873 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc54c102f50 0x7fc54c106f20 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:47:20.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.092+0000 7fc550e00700 1 --2- 172.21.15.62:0/4106308873 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc54c1025b0 0x7fc54c102980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:47:20.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.092+0000 7fc550e00700 1 -- 172.21.15.62:0/4106308873 >> 172.21.15.62:0/4106308873 conn(0x7fc54c0fde50 msgr2=0x7fc54c100260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:47:20.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.092+0000 7fc550e00700 1 -- 172.21.15.62:0/4106308873 shutdown_connections 2024-04-20T21:47:20.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.092+0000 7fc550e00700 1 -- 172.21.15.62:0/4106308873 wait complete. 2024-04-20T21:47:20.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.093+0000 7fc550e00700 1 Processor -- start 2024-04-20T21:47:20.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.093+0000 7fc550e00700 1 -- start start 2024-04-20T21:47:20.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.094+0000 7fc550e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc54c1025b0 0x7fc54c197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:47:20.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.094+0000 7fc550e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc54c102f50 0x7fc54c1984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:47:20.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.094+0000 7fc550e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc54c198b80 con 0x7fc54c102f50 2024-04-20T21:47:20.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.094+0000 7fc550e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc54c19c910 con 0x7fc54c1025b0 2024-04-20T21:47:20.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.094+0000 7fc541400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc54c102f50 0x7fc54c1984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:47:20.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.094+0000 7fc541400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc54c102f50 0x7fc54c1984a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34034/0 (socket says 172.21.15.62:34034) 2024-04-20T21:47:20.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.094+0000 7fc541400700 1 -- 172.21.15.62:0/1463314580 learned_addr learned my addr 172.21.15.62:0/1463314580 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:47:20.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.094+0000 7fc549e00700 1 --2- 172.21.15.62:0/1463314580 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc54c1025b0 0x7fc54c197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:47:20.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.095+0000 7fc541400700 1 -- 172.21.15.62:0/1463314580 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc54c1025b0 msgr2=0x7fc54c197f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:47:20.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.095+0000 7fc541400700 1 --2- 172.21.15.62:0/1463314580 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc54c1025b0 0x7fc54c197f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:47:20.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.095+0000 7fc541400700 1 -- 172.21.15.62:0/1463314580 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5340095f0 con 0x7fc54c102f50 2024-04-20T21:47:20.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.095+0000 7fc541400700 1 --2- 172.21.15.62:0/1463314580 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc54c102f50 0x7fc54c1984a0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fc53400bda0 tx=0x7fc53400be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:47:20.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.095+0000 7fc543400700 1 -- 172.21.15.62:0/1463314580 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc53401d070 con 0x7fc54c102f50 2024-04-20T21:47:20.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.095+0000 7fc543400700 1 -- 172.21.15.62:0/1463314580 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc53400f460 con 0x7fc54c102f50 2024-04-20T21:47:20.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.095+0000 7fc543400700 1 -- 172.21.15.62:0/1463314580 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc534017610 con 0x7fc54c102f50 2024-04-20T21:47:20.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.095+0000 7fc550e00700 1 -- 172.21.15.62:0/1463314580 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc54c19cb90 con 0x7fc54c102f50 2024-04-20T21:47:20.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.096+0000 7fc550e00700 1 -- 172.21.15.62:0/1463314580 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc54c19d080 con 0x7fc54c102f50 2024-04-20T21:47:20.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.097+0000 7fc550e00700 1 -- 172.21.15.62:0/1463314580 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc54c04f2a0 con 0x7fc54c102f50 2024-04-20T21:47:20.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.098+0000 7fc543400700 1 -- 172.21.15.62:0/1463314580 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fc53400fa40 con 0x7fc54c102f50 2024-04-20T21:47:20.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.099+0000 7fc543400700 1 --2- 172.21.15.62:0/1463314580 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc53006ecc0 0x7fc530071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:47:20.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.100+0000 7fc543400700 1 -- 172.21.15.62:0/1463314580 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fc5340902c0 con 0x7fc54c102f50 2024-04-20T21:47:20.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.100+0000 7fc549e00700 1 --2- 172.21.15.62:0/1463314580 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc53006ecc0 0x7fc530071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:47:20.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.100+0000 7fc549e00700 1 --2- 172.21.15.62:0/1463314580 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc53006ecc0 0x7fc530071170 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fc53c009d10 tx=0x7fc53c009040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:47:20.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.105+0000 7fc543400700 1 -- 172.21.15.62:0/1463314580 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fc534059050 con 0x7fc54c102f50 2024-04-20T21:47:20.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:47:19 smithi189 ceph-mon[57022]: pgmap v360: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:47:20.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:47:19 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:47:20.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.364+0000 7fc550e00700 1 -- 172.21.15.62:0/1463314580 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc54c19d5b0 con 0x7fc53006ecc0 2024-04-20T21:47:20.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.367+0000 7fc543400700 1 -- 172.21.15.62:0/1463314580 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fc54c19d5b0 con 0x7fc53006ecc0 2024-04-20T21:47:20.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.370+0000 7fc550e00700 1 -- 172.21.15.62:0/1463314580 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc53006ecc0 msgr2=0x7fc530071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:47:20.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.370+0000 7fc550e00700 1 --2- 172.21.15.62:0/1463314580 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc53006ecc0 0x7fc530071170 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fc53c009d10 tx=0x7fc53c009040 comp rx=0 tx=0).stop 2024-04-20T21:47:20.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.370+0000 7fc550e00700 1 -- 172.21.15.62:0/1463314580 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc54c102f50 msgr2=0x7fc54c1984a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:47:20.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.370+0000 7fc550e00700 1 --2- 172.21.15.62:0/1463314580 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc54c102f50 0x7fc54c1984a0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fc53400bda0 tx=0x7fc53400be80 comp rx=0 tx=0).stop 2024-04-20T21:47:20.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.371+0000 7fc550e00700 1 -- 172.21.15.62:0/1463314580 shutdown_connections 2024-04-20T21:47:20.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.371+0000 7fc550e00700 1 --2- 172.21.15.62:0/1463314580 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc53006ecc0 0x7fc530071170 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:47:20.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.371+0000 7fc550e00700 1 --2- 172.21.15.62:0/1463314580 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc54c1025b0 0x7fc54c197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:47:20.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.371+0000 7fc550e00700 1 --2- 172.21.15.62:0/1463314580 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc54c102f50 0x7fc54c1984a0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:47:20.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.371+0000 7fc550e00700 1 -- 172.21.15.62:0/1463314580 >> 172.21.15.62:0/1463314580 conn(0x7fc54c0fde50 msgr2=0x7fc54c0ff410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:47:20.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.371+0000 7fc550e00700 1 -- 172.21.15.62:0/1463314580 shutdown_connections 2024-04-20T21:47:20.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.371+0000 7fc550e00700 1 -- 172.21.15.62:0/1463314580 wait complete. 2024-04-20T21:47:20.388 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T21:47:20.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.539+0000 7f3aa4a00700 1 -- 172.21.15.62:0/2121807710 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3aa0102470 msgr2=0x7f3aa01028e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:47:20.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.539+0000 7f3aa4a00700 1 --2- 172.21.15.62:0/2121807710 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3aa0102470 0x7f3aa01028e0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f3a8c009910 tx=0x7f3a8c009c20 comp rx=0 tx=0).stop 2024-04-20T21:47:20.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.540+0000 7f3aa4a00700 1 -- 172.21.15.62:0/2121807710 shutdown_connections 2024-04-20T21:47:20.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.540+0000 7f3aa4a00700 1 --2- 172.21.15.62:0/2121807710 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3aa0102470 0x7f3aa01028e0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:47:20.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.540+0000 7f3aa4a00700 1 --2- 172.21.15.62:0/2121807710 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3aa0108470 0x7f3aa0108840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:47:20.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.540+0000 7f3aa4a00700 1 -- 172.21.15.62:0/2121807710 >> 172.21.15.62:0/2121807710 conn(0x7f3aa00fdf90 msgr2=0x7f3aa01003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:47:20.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.540+0000 7f3aa4a00700 1 -- 172.21.15.62:0/2121807710 shutdown_connections 2024-04-20T21:47:20.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.540+0000 7f3aa4a00700 1 -- 172.21.15.62:0/2121807710 wait complete. 2024-04-20T21:47:20.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.541+0000 7f3aa4a00700 1 Processor -- start 2024-04-20T21:47:20.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.541+0000 7f3aa4a00700 1 -- start start 2024-04-20T21:47:20.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.542+0000 7f3aa4a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3aa0102470 0x7f3aa01981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:47:20.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.542+0000 7f3aa4a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3aa0108470 0x7f3aa0198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:47:20.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.542+0000 7f3aa4a00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3aa0198e10 con 0x7f3aa0108470 2024-04-20T21:47:20.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.542+0000 7f3aa4a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3aa019cba0 con 0x7f3aa0102470 2024-04-20T21:47:20.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.542+0000 7f3a9d400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3aa0108470 0x7f3aa0198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:47:20.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.542+0000 7f3a9d400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3aa0108470 0x7f3aa0198730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34044/0 (socket says 172.21.15.62:34044) 2024-04-20T21:47:20.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.542+0000 7f3a9d400700 1 -- 172.21.15.62:0/1754115526 learned_addr learned my addr 172.21.15.62:0/1754115526 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:47:20.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.542+0000 7f3a9de00700 1 --2- 172.21.15.62:0/1754115526 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3aa0102470 0x7f3aa01981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:47:20.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.543+0000 7f3a9d400700 1 -- 172.21.15.62:0/1754115526 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3aa0102470 msgr2=0x7f3aa01981f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:47:20.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.543+0000 7f3a9d400700 1 --2- 172.21.15.62:0/1754115526 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3aa0102470 0x7f3aa01981f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:47:20.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.543+0000 7f3a9d400700 1 -- 172.21.15.62:0/1754115526 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a8c0095f0 con 0x7f3aa0108470 2024-04-20T21:47:20.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.543+0000 7f3a9de00700 1 --2- 172.21.15.62:0/1754115526 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3aa0102470 0x7f3aa01981f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-20T21:47:20.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.543+0000 7f3a9d400700 1 --2- 172.21.15.62:0/1754115526 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3aa0108470 0x7f3aa0198730 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f3a8c0092e0 tx=0x7f3a8c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:47:20.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.543+0000 7f3a96a00700 1 -- 172.21.15.62:0/1754115526 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3a8c01d070 con 0x7f3aa0108470 2024-04-20T21:47:20.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.543+0000 7f3a96a00700 1 -- 172.21.15.62:0/1754115526 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3a8c005e00 con 0x7f3aa0108470 2024-04-20T21:47:20.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.543+0000 7f3aa4a00700 1 -- 172.21.15.62:0/1754115526 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3aa019ce20 con 0x7f3aa0108470 2024-04-20T21:47:20.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.543+0000 7f3a96a00700 1 -- 172.21.15.62:0/1754115526 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3a8c017400 con 0x7f3aa0108470 2024-04-20T21:47:20.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.544+0000 7f3aa4a00700 1 -- 172.21.15.62:0/1754115526 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3aa019d310 con 0x7f3aa0108470 2024-04-20T21:47:20.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.545+0000 7f3aa4a00700 1 -- 172.21.15.62:0/1754115526 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3aa010a970 con 0x7f3aa0108470 2024-04-20T21:47:20.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.546+0000 7f3a96a00700 1 -- 172.21.15.62:0/1754115526 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f3a8c00f460 con 0x7f3aa0108470 2024-04-20T21:47:20.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.547+0000 7f3a96a00700 1 --2- 172.21.15.62:0/1754115526 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3a8406ecc0 0x7f3a84071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:47:20.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.547+0000 7f3a96a00700 1 -- 172.21.15.62:0/1754115526 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f3a8c090450 con 0x7f3aa0108470 2024-04-20T21:47:20.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.547+0000 7f3a9de00700 1 --2- 172.21.15.62:0/1754115526 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3a8406ecc0 0x7f3a84071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:47:20.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.548+0000 7f3a9de00700 1 --2- 172.21.15.62:0/1754115526 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3a8406ecc0 0x7f3a84071170 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f3a90005950 tx=0x7f3a900058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:47:20.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.552+0000 7f3a96a00700 1 -- 172.21.15.62:0/1754115526 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f3a8c059160 con 0x7f3aa0108470 2024-04-20T21:47:20.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.814+0000 7f3aa4a00700 1 -- 172.21.15.62:0/1754115526 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3aa019d530 con 0x7f3a8406ecc0 2024-04-20T21:47:20.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.817+0000 7f3a96a00700 1 -- 172.21.15.62:0/1754115526 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f3aa019d530 con 0x7f3a8406ecc0 2024-04-20T21:47:20.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.820+0000 7f3aa4a00700 1 -- 172.21.15.62:0/1754115526 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3a8406ecc0 msgr2=0x7f3a84071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:47:20.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.820+0000 7f3aa4a00700 1 --2- 172.21.15.62:0/1754115526 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3a8406ecc0 0x7f3a84071170 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f3a90005950 tx=0x7f3a900058e0 comp rx=0 tx=0).stop 2024-04-20T21:47:20.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.820+0000 7f3aa4a00700 1 -- 172.21.15.62:0/1754115526 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3aa0108470 msgr2=0x7f3aa0198730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:47:20.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.820+0000 7f3aa4a00700 1 --2- 172.21.15.62:0/1754115526 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3aa0108470 0x7f3aa0198730 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f3a8c0092e0 tx=0x7f3a8c00be20 comp rx=0 tx=0).stop 2024-04-20T21:47:20.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.820+0000 7f3aa4a00700 1 -- 172.21.15.62:0/1754115526 shutdown_connections 2024-04-20T21:47:20.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.820+0000 7f3aa4a00700 1 --2- 172.21.15.62:0/1754115526 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3a8406ecc0 0x7f3a84071170 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:47:20.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.820+0000 7f3aa4a00700 1 --2- 172.21.15.62:0/1754115526 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3aa0102470 0x7f3aa01981f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:47:20.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.820+0000 7f3aa4a00700 1 --2- 172.21.15.62:0/1754115526 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3aa0108470 0x7f3aa0198730 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:47:20.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.820+0000 7f3aa4a00700 1 -- 172.21.15.62:0/1754115526 >> 172.21.15.62:0/1754115526 conn(0x7f3aa00fdf90 msgr2=0x7f3aa00ff3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:47:20.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.820+0000 7f3aa4a00700 1 -- 172.21.15.62:0/1754115526 shutdown_connections 2024-04-20T21:47:20.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.820+0000 7f3aa4a00700 1 -- 172.21.15.62:0/1754115526 wait complete. 2024-04-20T21:47:20.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.983+0000 7ffb64a00700 1 -- 172.21.15.62:0/3012355583 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffb60102e70 msgr2=0x7ffb60106e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:47:20.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.983+0000 7ffb64a00700 1 --2- 172.21.15.62:0/3012355583 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffb60102e70 0x7ffb60106e40 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7ffb4c009910 tx=0x7ffb4c009c20 comp rx=0 tx=0).stop 2024-04-20T21:47:20.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.983+0000 7ffb64a00700 1 -- 172.21.15.62:0/3012355583 shutdown_connections 2024-04-20T21:47:20.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.983+0000 7ffb64a00700 1 --2- 172.21.15.62:0/3012355583 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffb60102e70 0x7ffb60106e40 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:47:20.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.983+0000 7ffb64a00700 1 --2- 172.21.15.62:0/3012355583 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb601024d0 0x7ffb601028a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:47:20.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.983+0000 7ffb64a00700 1 -- 172.21.15.62:0/3012355583 >> 172.21.15.62:0/3012355583 conn(0x7ffb600fdd90 msgr2=0x7ffb601001a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:47:20.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.983+0000 7ffb64a00700 1 -- 172.21.15.62:0/3012355583 shutdown_connections 2024-04-20T21:47:20.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.984+0000 7ffb64a00700 1 -- 172.21.15.62:0/3012355583 wait complete. 2024-04-20T21:47:20.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.985+0000 7ffb64a00700 1 Processor -- start 2024-04-20T21:47:20.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.985+0000 7ffb64a00700 1 -- start start 2024-04-20T21:47:20.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.985+0000 7ffb64a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffb601024d0 0x7ffb60197e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:47:20.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.985+0000 7ffb64a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb60102e70 0x7ffb60198390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:47:20.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.985+0000 7ffb64a00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb60198a70 con 0x7ffb601024d0 2024-04-20T21:47:20.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.985+0000 7ffb64a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb6019c800 con 0x7ffb60102e70 2024-04-20T21:47:20.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.985+0000 7ffb5e800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffb601024d0 0x7ffb60197e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:47:20.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.986+0000 7ffb5e800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffb601024d0 0x7ffb60197e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34052/0 (socket says 172.21.15.62:34052) 2024-04-20T21:47:20.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.986+0000 7ffb5e800700 1 -- 172.21.15.62:0/3847127391 learned_addr learned my addr 172.21.15.62:0/3847127391 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:47:20.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.986+0000 7ffb5de00700 1 --2- 172.21.15.62:0/3847127391 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb60102e70 0x7ffb60198390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:47:20.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.986+0000 7ffb5e800700 1 -- 172.21.15.62:0/3847127391 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb60102e70 msgr2=0x7ffb60198390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:47:20.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.986+0000 7ffb5e800700 1 --2- 172.21.15.62:0/3847127391 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb60102e70 0x7ffb60198390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:47:20.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.986+0000 7ffb5e800700 1 -- 172.21.15.62:0/3847127391 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb4c0095f0 con 0x7ffb601024d0 2024-04-20T21:47:20.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.986+0000 7ffb5e800700 1 --2- 172.21.15.62:0/3847127391 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffb601024d0 0x7ffb60197e50 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7ffb5000c8f0 tx=0x7ffb5000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:47:20.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.987+0000 7ffb57400700 1 -- 172.21.15.62:0/3847127391 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffb50007a70 con 0x7ffb601024d0 2024-04-20T21:47:20.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.987+0000 7ffb57400700 1 -- 172.21.15.62:0/3847127391 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ffb50004cf0 con 0x7ffb601024d0 2024-04-20T21:47:20.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.987+0000 7ffb64a00700 1 -- 172.21.15.62:0/3847127391 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb6019cae0 con 0x7ffb601024d0 2024-04-20T21:47:20.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.987+0000 7ffb57400700 1 -- 172.21.15.62:0/3847127391 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffb50005610 con 0x7ffb601024d0 2024-04-20T21:47:20.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.987+0000 7ffb64a00700 1 -- 172.21.15.62:0/3847127391 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb6019d030 con 0x7ffb601024d0 2024-04-20T21:47:20.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.988+0000 7ffb64a00700 1 -- 172.21.15.62:0/3847127391 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb6010a750 con 0x7ffb601024d0 2024-04-20T21:47:20.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.989+0000 7ffb57400700 1 -- 172.21.15.62:0/3847127391 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ffb50004750 con 0x7ffb601024d0 2024-04-20T21:47:20.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.990+0000 7ffb57400700 1 --2- 172.21.15.62:0/3847127391 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ffb4406ecc0 0x7ffb44071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:47:20.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.990+0000 7ffb57400700 1 -- 172.21.15.62:0/3847127391 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7ffb50013070 con 0x7ffb601024d0 2024-04-20T21:47:20.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.990+0000 7ffb5de00700 1 --2- 172.21.15.62:0/3847127391 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ffb4406ecc0 0x7ffb44071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:47:20.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.991+0000 7ffb5de00700 1 --2- 172.21.15.62:0/3847127391 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ffb4406ecc0 0x7ffb44071170 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7ffb60199470 tx=0x7ffb4c01a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:47:20.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:20.995+0000 7ffb57400700 1 -- 172.21.15.62:0/3847127391 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7ffb500579e0 con 0x7ffb601024d0 2024-04-20T21:47:21.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:21.243+0000 7ffb64a00700 1 -- 172.21.15.62:0/3847127391 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7ffb6019d370 con 0x7ffb4406ecc0 2024-04-20T21:47:21.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:21.283+0000 7ffb57400700 1 -- 172.21.15.62:0/3847127391 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7ffb6019d370 con 0x7ffb4406ecc0 2024-04-20T21:47:21.283 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T21:47:21.283 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (13m) 4m ago 27m 14.2M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T21:47:21.283 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (27m) 4m ago 27m 9512k - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T21:47:21.283 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (26m) 5m ago 26m 8942k - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T21:47:21.283 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (9m) 4m ago 27m 6660k - 19.0.0-3133-ga8c84f59 76bc6b91b726 2825d5ba11c6 2024-04-20T21:47:21.283 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (9m) 5m ago 25m 6665k - 19.0.0-3133-ga8c84f59 76bc6b91b726 20f746fa1bba 2024-04-20T21:47:21.283 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (12m) 4m ago 26m 74.2M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T21:47:21.283 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (4m) 4m ago 19m 11.2M - 19.0.0-3133-ga8c84f59 76bc6b91b726 de80bc90a683 2024-04-20T21:47:21.283 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (5m) 4m ago 19m 11.9M - 19.0.0-3133-ga8c84f59 76bc6b91b726 cd793b7ee98e 2024-04-20T21:47:21.283 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (19m) 5m ago 19m 117M - 18.2.0 dc2bc1663786 fb2a8a223db8 2024-04-20T21:47:21.283 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (19m) 5m ago 19m 55.6M - 18.2.0 dc2bc1663786 f2eec2ee8951 2024-04-20T21:47:21.283 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (16m) 4m ago 28m 511M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T21:47:21.284 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (14m) 5m ago 25m 452M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T21:47:21.284 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (10m) 4m ago 29m 55.9M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 019eb1479485 2024-04-20T21:47:21.284 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (10m) 5m ago 25m 39.8M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 55aac560aab6 2024-04-20T21:47:21.284 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (14m) 4m ago 27m 7952k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T21:47:21.284 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (14m) 5m ago 25m 8304k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T21:47:21.284 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (8m) 4m ago 24m 88.0M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 ee2bb901f26e 2024-04-20T21:47:21.284 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (8m) 4m ago 24m 92.3M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 9075152d4162 2024-04-20T21:47:21.284 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (7m) 4m ago 23m 74.4M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 2d4f3e38dc0e 2024-04-20T21:47:21.284 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (7m) 5m ago 22m 87.6M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 8f97b17ae0eb 2024-04-20T21:47:21.284 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (6m) 5m ago 22m 90.7M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 653d8079967d 2024-04-20T21:47:21.284 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (5m) 5m ago 21m 11.5M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 596dded14c12 2024-04-20T21:47:21.284 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (13m) 4m ago 26m 74.0M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T21:47:21.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:21.287+0000 7ffb64a00700 1 -- 172.21.15.62:0/3847127391 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ffb4406ecc0 msgr2=0x7ffb44071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:47:21.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:21.287+0000 7ffb64a00700 1 --2- 172.21.15.62:0/3847127391 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ffb4406ecc0 0x7ffb44071170 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7ffb60199470 tx=0x7ffb4c01a040 comp rx=0 tx=0).stop 2024-04-20T21:47:21.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:21.287+0000 7ffb64a00700 1 -- 172.21.15.62:0/3847127391 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffb601024d0 msgr2=0x7ffb60197e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:47:21.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:21.287+0000 7ffb64a00700 1 --2- 172.21.15.62:0/3847127391 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffb601024d0 0x7ffb60197e50 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7ffb5000c8f0 tx=0x7ffb5000ccb0 comp rx=0 tx=0).stop 2024-04-20T21:47:21.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:21.287+0000 7ffb64a00700 1 -- 172.21.15.62:0/3847127391 shutdown_connections 2024-04-20T21:47:21.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:21.287+0000 7ffb64a00700 1 --2- 172.21.15.62:0/3847127391 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ffb4406ecc0 0x7ffb44071170 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:47:21.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:21.287+0000 7ffb64a00700 1 --2- 172.21.15.62:0/3847127391 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffb601024d0 0x7ffb60197e50 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:47:21.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:21.287+0000 7ffb64a00700 1 --2- 172.21.15.62:0/3847127391 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb60102e70 0x7ffb60198390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:47:21.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:21.287+0000 7ffb64a00700 1 -- 172.21.15.62:0/3847127391 >> 172.21.15.62:0/3847127391 conn(0x7ffb600fdd90 msgr2=0x7ffb601066b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:47:21.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:21.288+0000 7ffb64a00700 1 -- 172.21.15.62:0/3847127391 shutdown_connections 2024-04-20T21:47:21.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:21.288+0000 7ffb64a00700 1 -- 172.21.15.62:0/3847127391 wait complete. 2024-04-20T21:47:21.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:21.453+0000 7ff226a00700 1 -- 172.21.15.62:0/2433953389 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff2200739a0 msgr2=0x7ff22010c5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:47:21.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:21.453+0000 7ff226a00700 1 --2- 172.21.15.62:0/2433953389 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff2200739a0 0x7ff22010c5e0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7ff208009940 tx=0x7ff208009c50 comp rx=0 tx=0).stop 2024-04-20T21:47:21.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:21.453+0000 7ff226a00700 1 -- 172.21.15.62:0/2433953389 shutdown_connections 2024-04-20T21:47:21.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:21.453+0000 7ff226a00700 1 --2- 172.21.15.62:0/2433953389 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff2200739a0 0x7ff22010c5e0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:47:21.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:21.453+0000 7ff226a00700 1 --2- 172.21.15.62:0/2433953389 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff220073090 0x7ff220073460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:47:21.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:21.453+0000 7ff226a00700 1 -- 172.21.15.62:0/2433953389 >> 172.21.15.62:0/2433953389 conn(0x7ff2200fbda0 msgr2=0x7ff2200fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:47:21.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:21.453+0000 7ff226a00700 1 -- 172.21.15.62:0/2433953389 shutdown_connections 2024-04-20T21:47:21.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:21.453+0000 7ff226a00700 1 -- 172.21.15.62:0/2433953389 wait complete. 2024-04-20T21:47:21.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:21.454+0000 7ff226a00700 1 Processor -- start 2024-04-20T21:47:21.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:21.454+0000 7ff226a00700 1 -- start start 2024-04-20T21:47:21.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:21.455+0000 7ff226a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff220073090 0x7ff2201981b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:47:21.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:21.455+0000 7ff226a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2200739a0 0x7ff2201986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:47:21.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:21.455+0000 7ff226a00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff220198dd0 con 0x7ff220073090 2024-04-20T21:47:21.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:21.455+0000 7ff226a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff22019cb60 con 0x7ff2200739a0 2024-04-20T21:47:21.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:21.455+0000 7ff21fe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff220073090 0x7ff2201981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:47:21.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:21.455+0000 7ff21fe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff220073090 0x7ff2201981b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51644/0 (socket says 172.21.15.62:51644) 2024-04-20T21:47:21.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:21.455+0000 7ff21fe00700 1 -- 172.21.15.62:0/3411869503 learned_addr learned my addr 172.21.15.62:0/3411869503 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:47:21.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:21.456+0000 7ff217e00700 1 --2- 172.21.15.62:0/3411869503 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2200739a0 0x7ff2201986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:47:21.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:21.456+0000 7ff21fe00700 1 -- 172.21.15.62:0/3411869503 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2200739a0 msgr2=0x7ff2201986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:47:21.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:21.456+0000 7ff21fe00700 1 --2- 172.21.15.62:0/3411869503 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2200739a0 0x7ff2201986f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:47:21.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:21.456+0000 7ff21fe00700 1 -- 172.21.15.62:0/3411869503 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2080095f0 con 0x7ff220073090 2024-04-20T21:47:21.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:21.456+0000 7ff217e00700 1 --2- 172.21.15.62:0/3411869503 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2200739a0 0x7ff2201986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T21:47:21.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:21.456+0000 7ff21fe00700 1 --2- 172.21.15.62:0/3411869503 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff220073090 0x7ff2201981b0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7ff21000d8d0 tx=0x7ff21000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:47:21.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:21.456+0000 7ff21d600700 1 -- 172.21.15.62:0/3411869503 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff210009940 con 0x7ff220073090 2024-04-20T21:47:21.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:21.456+0000 7ff21d600700 1 -- 172.21.15.62:0/3411869503 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff210010460 con 0x7ff220073090 2024-04-20T21:47:21.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:21.456+0000 7ff21d600700 1 -- 172.21.15.62:0/3411869503 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff21000f5f0 con 0x7ff220073090 2024-04-20T21:47:21.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:21.456+0000 7ff226a00700 1 -- 172.21.15.62:0/3411869503 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff22019ce40 con 0x7ff220073090 2024-04-20T21:47:21.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:21.457+0000 7ff226a00700 1 -- 172.21.15.62:0/3411869503 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff22019d390 con 0x7ff220073090 2024-04-20T21:47:21.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:21.458+0000 7ff226a00700 1 -- 172.21.15.62:0/3411869503 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff220109ce0 con 0x7ff220073090 2024-04-20T21:47:21.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:21.459+0000 7ff21d600700 1 -- 172.21.15.62:0/3411869503 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff2100105d0 con 0x7ff220073090 2024-04-20T21:47:21.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:21.460+0000 7ff21d600700 1 --2- 172.21.15.62:0/3411869503 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff20406ecc0 0x7ff204071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:47:21.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:21.460+0000 7ff21d600700 1 -- 172.21.15.62:0/3411869503 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7ff21008ee10 con 0x7ff220073090 2024-04-20T21:47:21.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:21.460+0000 7ff217e00700 1 --2- 172.21.15.62:0/3411869503 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff20406ecc0 0x7ff204071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:47:21.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:21.461+0000 7ff217e00700 1 --2- 172.21.15.62:0/3411869503 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff20406ecc0 0x7ff204071170 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7ff208000c00 tx=0x7ff208005fb0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:47:21.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:21.465+0000 7ff21d600700 1 -- 172.21.15.62:0/3411869503 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7ff210057ba0 con 0x7ff220073090 2024-04-20T21:47:21.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:21.800+0000 7ff226a00700 1 -- 172.21.15.62:0/3411869503 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7ff2201995e0 con 0x7ff220073090 2024-04-20T21:47:21.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:21.801+0000 7ff21d600700 1 -- 172.21.15.62:0/3411869503 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7ff21005b260 con 0x7ff220073090 2024-04-20T21:47:21.800 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:47:21.800 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T21:47:21.801 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:47:21.801 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:47:21.801 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T21:47:21.801 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:47:21.801 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:47:21.801 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T21:47:21.801 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 6 2024-04-20T21:47:21.801 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:47:21.801 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T21:47:21.801 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 2, 2024-04-20T21:47:21.801 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:47:21.801 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:47:21.801 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T21:47:21.801 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 2, 2024-04-20T21:47:21.802 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 12 2024-04-20T21:47:21.802 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T21:47:21.802 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:47:21.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:21.804+0000 7ff226a00700 1 -- 172.21.15.62:0/3411869503 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff20406ecc0 msgr2=0x7ff204071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:47:21.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:21.804+0000 7ff226a00700 1 --2- 172.21.15.62:0/3411869503 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff20406ecc0 0x7ff204071170 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7ff208000c00 tx=0x7ff208005fb0 comp rx=0 tx=0).stop 2024-04-20T21:47:21.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:21.804+0000 7ff226a00700 1 -- 172.21.15.62:0/3411869503 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff220073090 msgr2=0x7ff2201981b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:47:21.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:21.804+0000 7ff226a00700 1 --2- 172.21.15.62:0/3411869503 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff220073090 0x7ff2201981b0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7ff21000d8d0 tx=0x7ff21000dc90 comp rx=0 tx=0).stop 2024-04-20T21:47:21.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:21.804+0000 7ff226a00700 1 -- 172.21.15.62:0/3411869503 shutdown_connections 2024-04-20T21:47:21.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:21.804+0000 7ff226a00700 1 --2- 172.21.15.62:0/3411869503 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff20406ecc0 0x7ff204071170 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:47:21.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:21.804+0000 7ff226a00700 1 --2- 172.21.15.62:0/3411869503 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff220073090 0x7ff2201981b0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:47:21.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:21.804+0000 7ff226a00700 1 --2- 172.21.15.62:0/3411869503 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2200739a0 0x7ff2201986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:47:21.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:21.804+0000 7ff226a00700 1 -- 172.21.15.62:0/3411869503 >> 172.21.15.62:0/3411869503 conn(0x7ff2200fbda0 msgr2=0x7ff220106e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:47:21.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:21.805+0000 7ff226a00700 1 -- 172.21.15.62:0/3411869503 shutdown_connections 2024-04-20T21:47:21.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:21.805+0000 7ff226a00700 1 -- 172.21.15.62:0/3411869503 wait complete. 2024-04-20T21:47:21.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:21.964+0000 7f760fc00700 1 -- 172.21.15.62:0/2823562989 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7608100e80 msgr2=0x7f7608104e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:47:21.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:21.964+0000 7f760fc00700 1 --2- 172.21.15.62:0/2823562989 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7608100e80 0x7f7608104e50 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f75f4009910 tx=0x7f75f4009c20 comp rx=0 tx=0).stop 2024-04-20T21:47:21.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:21.965+0000 7f760fc00700 1 -- 172.21.15.62:0/2823562989 shutdown_connections 2024-04-20T21:47:21.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:21.965+0000 7f760fc00700 1 --2- 172.21.15.62:0/2823562989 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7608100e80 0x7f7608104e50 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:47:21.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:21.965+0000 7f760fc00700 1 --2- 172.21.15.62:0/2823562989 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f76081004e0 0x7f76081008b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:47:21.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:21.965+0000 7f760fc00700 1 -- 172.21.15.62:0/2823562989 >> 172.21.15.62:0/2823562989 conn(0x7f76080fbda0 msgr2=0x7f76080fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:47:21.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:21.965+0000 7f760fc00700 1 -- 172.21.15.62:0/2823562989 shutdown_connections 2024-04-20T21:47:21.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:21.965+0000 7f760fc00700 1 -- 172.21.15.62:0/2823562989 wait complete. 2024-04-20T21:47:21.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:21.966+0000 7f760fc00700 1 Processor -- start 2024-04-20T21:47:21.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:21.966+0000 7f760fc00700 1 -- start start 2024-04-20T21:47:21.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:21.966+0000 7f760fc00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76081004e0 0x7f7608198110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:47:21.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:21.966+0000 7f760fc00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7608100e80 0x7f7608198650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:47:21.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:21.966+0000 7f760fc00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7608198d30 con 0x7f76081004e0 2024-04-20T21:47:21.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:21.966+0000 7f760fc00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f760819cac0 con 0x7f7608100e80 2024-04-20T21:47:21.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:21.967+0000 7f760d200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76081004e0 0x7f7608198110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:47:21.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:21.967+0000 7f760d200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76081004e0 0x7f7608198110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51668/0 (socket says 172.21.15.62:51668) 2024-04-20T21:47:21.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:21.967+0000 7f760d200700 1 -- 172.21.15.62:0/3285876361 learned_addr learned my addr 172.21.15.62:0/3285876361 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:47:21.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:21.967+0000 7f7607e00700 1 --2- 172.21.15.62:0/3285876361 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7608100e80 0x7f7608198650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:47:21.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:21.967+0000 7f760d200700 1 -- 172.21.15.62:0/3285876361 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7608100e80 msgr2=0x7f7608198650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:47:21.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:21.967+0000 7f760d200700 1 --2- 172.21.15.62:0/3285876361 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7608100e80 0x7f7608198650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:47:21.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:21.967+0000 7f760d200700 1 -- 172.21.15.62:0/3285876361 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75f40095f0 con 0x7f76081004e0 2024-04-20T21:47:21.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:21.967+0000 7f7607e00700 1 --2- 172.21.15.62:0/3285876361 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7608100e80 0x7f7608198650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T21:47:21.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:21.968+0000 7f760d200700 1 --2- 172.21.15.62:0/3285876361 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76081004e0 0x7f7608198110 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f75fc00d8d0 tx=0x7f75fc00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:47:21.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:21.968+0000 7f7605600700 1 -- 172.21.15.62:0/3285876361 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f75fc009940 con 0x7f76081004e0 2024-04-20T21:47:21.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:21.968+0000 7f760fc00700 1 -- 172.21.15.62:0/3285876361 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f760819cda0 con 0x7f76081004e0 2024-04-20T21:47:21.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:21.968+0000 7f7605600700 1 -- 172.21.15.62:0/3285876361 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f75fc010460 con 0x7f76081004e0 2024-04-20T21:47:21.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:21.968+0000 7f760fc00700 1 -- 172.21.15.62:0/3285876361 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f760819d2f0 con 0x7f76081004e0 2024-04-20T21:47:21.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:21.968+0000 7f7605600700 1 -- 172.21.15.62:0/3285876361 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f75fc00f5f0 con 0x7f76081004e0 2024-04-20T21:47:21.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:21.969+0000 7f760fc00700 1 -- 172.21.15.62:0/3285876361 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76081045d0 con 0x7f76081004e0 2024-04-20T21:47:21.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:21.970+0000 7f7605600700 1 -- 172.21.15.62:0/3285876361 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f75fc0105d0 con 0x7f76081004e0 2024-04-20T21:47:21.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:21.971+0000 7f7605600700 1 --2- 172.21.15.62:0/3285876361 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f75f006ecc0 0x7f75f0071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:47:21.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:21.972+0000 7f7605600700 1 -- 172.21.15.62:0/3285876361 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f75fc08ef10 con 0x7f76081004e0 2024-04-20T21:47:21.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:21.972+0000 7f7607e00700 1 --2- 172.21.15.62:0/3285876361 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f75f006ecc0 0x7f75f0071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:47:21.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:21.972+0000 7f7607e00700 1 --2- 172.21.15.62:0/3285876361 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f75f006ecc0 0x7f75f0071170 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f75f4006010 tx=0x7f75f400b540 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:47:21.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:21.976+0000 7f7605600700 1 -- 172.21.15.62:0/3285876361 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f75fc057c20 con 0x7f76081004e0 2024-04-20T21:47:22.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:47:21 smithi062 ceph-mon[76322]: from='client.34704 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:47:22.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:47:21 smithi062 ceph-mon[76322]: pgmap v361: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:47:22.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:47:21 smithi062 ceph-mon[76322]: from='client.34708 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:47:22.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:47:21 smithi189 ceph-mon[57022]: from='client.34704 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:47:22.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:47:21 smithi189 ceph-mon[57022]: pgmap v361: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:47:22.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:47:21 smithi189 ceph-mon[57022]: from='client.34708 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:47:22.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:22.259+0000 7f760fc00700 1 -- 172.21.15.62:0/3285876361 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f760819d560 con 0x7f76081004e0 2024-04-20T21:47:22.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:22.260+0000 7f7605600700 1 -- 172.21.15.62:0/3285876361 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 25 v25) v1 ==== 76+0+1887 (secure 0 0 0) 0x7f75fc05b2e0 con 0x7f76081004e0 2024-04-20T21:47:22.259 INFO:teuthology.orchestra.run.smithi062.stdout:e25 2024-04-20T21:47:22.259 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T21:47:22.260 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T21:47:22.260 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T21:47:22.260 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:47:22.260 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T21:47:22.260 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T21:47:22.260 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 20 2024-04-20T21:47:22.260 INFO:teuthology.orchestra.run.smithi062.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-20T21:47:22.260 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T21:47:22.260 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T21:42:01.559061+0000 2024-04-20T21:47:22.260 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T21:47:22.260 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T21:47:22.260 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T21:47:22.260 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T21:47:22.260 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T21:47:22.261 INFO:teuthology.orchestra.run.smithi062.stdout:max_xattr_size 65536 2024-04-20T21:47:22.261 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T21:47:22.261 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T21:47:22.261 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 81 2024-04-20T21:47:22.261 INFO:teuthology.orchestra.run.smithi062.stdout:snap_rstat 1 2024-04-20T21:47:22.261 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T21:47:22.261 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 2 2024-04-20T21:47:22.261 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1 2024-04-20T21:47:22.261 INFO:teuthology.orchestra.run.smithi062.stdout:up {} 2024-04-20T21:47:22.261 INFO:teuthology.orchestra.run.smithi062.stdout:failed 0,1 2024-04-20T21:47:22.261 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T21:47:22.261 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T21:47:22.261 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T21:47:22.261 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T21:47:22.261 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data enabled 2024-04-20T21:47:22.261 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T21:47:22.262 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T21:47:22.262 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 1 2024-04-20T21:47:22.262 INFO:teuthology.orchestra.run.smithi062.stdout:qdb_cluster leader: 0 members: 2024-04-20T21:47:22.262 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:47:22.262 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:47:22.262 INFO:teuthology.orchestra.run.smithi062.stdout:Standby daemons: 2024-04-20T21:47:22.262 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:47:22.262 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ryqblb{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6826/4113916205,v1:172.21.15.62:6827/4113916205] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T21:47:22.262 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ozizwi{-1:34476} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6828/3801975165,v1:172.21.15.62:6829/3801975165] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T21:47:22.262 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.xbgspw{-1:44285} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6826/3996040526,v1:172.21.15.189:6827/3996040526] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:47:22.262 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.dvimer{-1:44289} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6824/1743111843,v1:172.21.15.189:6825/1743111843] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:47:22.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:22.263+0000 7f760fc00700 1 -- 172.21.15.62:0/3285876361 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f75f006ecc0 msgr2=0x7f75f0071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:47:22.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:22.263+0000 7f760fc00700 1 --2- 172.21.15.62:0/3285876361 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f75f006ecc0 0x7f75f0071170 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f75f4006010 tx=0x7f75f400b540 comp rx=0 tx=0).stop 2024-04-20T21:47:22.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:22.263+0000 7f760fc00700 1 -- 172.21.15.62:0/3285876361 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76081004e0 msgr2=0x7f7608198110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:47:22.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:22.263+0000 7f760fc00700 1 --2- 172.21.15.62:0/3285876361 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76081004e0 0x7f7608198110 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f75fc00d8d0 tx=0x7f75fc00dc90 comp rx=0 tx=0).stop 2024-04-20T21:47:22.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:22.263+0000 7f760fc00700 1 -- 172.21.15.62:0/3285876361 shutdown_connections 2024-04-20T21:47:22.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:22.263+0000 7f760fc00700 1 --2- 172.21.15.62:0/3285876361 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f75f006ecc0 0x7f75f0071170 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:47:22.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:22.263+0000 7f760fc00700 1 --2- 172.21.15.62:0/3285876361 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76081004e0 0x7f7608198110 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:47:22.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:22.263+0000 7f760fc00700 1 --2- 172.21.15.62:0/3285876361 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7608100e80 0x7f7608198650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:47:22.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:22.263+0000 7f760fc00700 1 -- 172.21.15.62:0/3285876361 >> 172.21.15.62:0/3285876361 conn(0x7f76080fbda0 msgr2=0x7f76080fe150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:47:22.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:22.263+0000 7f760fc00700 1 -- 172.21.15.62:0/3285876361 shutdown_connections 2024-04-20T21:47:22.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:22.263+0000 7f760fc00700 1 -- 172.21.15.62:0/3285876361 wait complete. 2024-04-20T21:47:22.264 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 25 2024-04-20T21:47:22.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:22.419+0000 7f85be800700 1 -- 172.21.15.62:0/3194339243 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f85b80730f0 msgr2=0x7f85b80734c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:47:22.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:22.419+0000 7f85be800700 1 --2- 172.21.15.62:0/3194339243 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f85b80730f0 0x7f85b80734c0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f85a8009910 tx=0x7f85a8009c20 comp rx=0 tx=0).stop 2024-04-20T21:47:22.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:22.419+0000 7f85be800700 1 -- 172.21.15.62:0/3194339243 shutdown_connections 2024-04-20T21:47:22.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:22.419+0000 7f85be800700 1 --2- 172.21.15.62:0/3194339243 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f85b8073a00 0x7f85b8110d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:47:22.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:22.419+0000 7f85be800700 1 --2- 172.21.15.62:0/3194339243 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f85b80730f0 0x7f85b80734c0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:47:22.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:22.419+0000 7f85be800700 1 -- 172.21.15.62:0/3194339243 >> 172.21.15.62:0/3194339243 conn(0x7f85b80fbda0 msgr2=0x7f85b80fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:47:22.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:22.420+0000 7f85be800700 1 -- 172.21.15.62:0/3194339243 shutdown_connections 2024-04-20T21:47:22.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:22.420+0000 7f85be800700 1 -- 172.21.15.62:0/3194339243 wait complete. 2024-04-20T21:47:22.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:22.421+0000 7f85be800700 1 Processor -- start 2024-04-20T21:47:22.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:22.421+0000 7f85be800700 1 -- start start 2024-04-20T21:47:22.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:22.421+0000 7f85be800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f85b80730f0 0x7f85b819c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:47:22.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:22.421+0000 7f85be800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f85b8073a00 0x7f85b819c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:47:22.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:22.421+0000 7f85be800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85b819d000 con 0x7f85b8073a00 2024-04-20T21:47:22.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:22.421+0000 7f85be800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85b81a0d90 con 0x7f85b80730f0 2024-04-20T21:47:22.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:22.422+0000 7f85b6a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f85b8073a00 0x7f85b819c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:47:22.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:22.422+0000 7f85b6a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f85b8073a00 0x7f85b819c920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51700/0 (socket says 172.21.15.62:51700) 2024-04-20T21:47:22.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:22.422+0000 7f85b6a00700 1 -- 172.21.15.62:0/3474663518 learned_addr learned my addr 172.21.15.62:0/3474663518 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:47:22.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:22.422+0000 7f85b7400700 1 --2- 172.21.15.62:0/3474663518 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f85b80730f0 0x7f85b819c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:47:22.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:22.422+0000 7f85b6a00700 1 -- 172.21.15.62:0/3474663518 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f85b80730f0 msgr2=0x7f85b819c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:47:22.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:22.422+0000 7f85b6a00700 1 --2- 172.21.15.62:0/3474663518 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f85b80730f0 0x7f85b819c3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:47:22.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:22.422+0000 7f85b6a00700 1 -- 172.21.15.62:0/3474663518 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85a80095f0 con 0x7f85b8073a00 2024-04-20T21:47:22.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:22.422+0000 7f85b7400700 1 --2- 172.21.15.62:0/3474663518 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f85b80730f0 0x7f85b819c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T21:47:22.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:22.422+0000 7f85b6a00700 1 --2- 172.21.15.62:0/3474663518 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f85b8073a00 0x7f85b819c920 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f85a400d8d0 tx=0x7f85a400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:47:22.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:22.423+0000 7f85afe00700 1 -- 172.21.15.62:0/3474663518 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f85a4009940 con 0x7f85b8073a00 2024-04-20T21:47:22.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:22.423+0000 7f85afe00700 1 -- 172.21.15.62:0/3474663518 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f85a4010460 con 0x7f85b8073a00 2024-04-20T21:47:22.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:22.423+0000 7f85be800700 1 -- 172.21.15.62:0/3474663518 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85b81a1070 con 0x7f85b8073a00 2024-04-20T21:47:22.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:22.423+0000 7f85afe00700 1 -- 172.21.15.62:0/3474663518 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f85a400f5f0 con 0x7f85b8073a00 2024-04-20T21:47:22.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:22.423+0000 7f85be800700 1 -- 172.21.15.62:0/3474663518 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85b81a15c0 con 0x7f85b8073a00 2024-04-20T21:47:22.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:22.424+0000 7f85be800700 1 -- 172.21.15.62:0/3474663518 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85b810e4d0 con 0x7f85b8073a00 2024-04-20T21:47:22.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:22.425+0000 7f85afe00700 1 -- 172.21.15.62:0/3474663518 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f85a40105d0 con 0x7f85b8073a00 2024-04-20T21:47:22.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:22.426+0000 7f85afe00700 1 --2- 172.21.15.62:0/3474663518 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f859c06ecc0 0x7f859c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:47:22.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:22.426+0000 7f85afe00700 1 -- 172.21.15.62:0/3474663518 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f85a408ee10 con 0x7f85b8073a00 2024-04-20T21:47:22.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:22.427+0000 7f85b7400700 1 --2- 172.21.15.62:0/3474663518 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f859c06ecc0 0x7f859c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:47:22.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:22.427+0000 7f85b7400700 1 --2- 172.21.15.62:0/3474663518 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f859c06ecc0 0x7f859c071170 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f85a8006010 tx=0x7f85a800b540 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:47:22.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:22.431+0000 7f85afe00700 1 -- 172.21.15.62:0/3474663518 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f85a4057ba0 con 0x7f85b8073a00 2024-04-20T21:47:22.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:22.685+0000 7f85be800700 1 -- 172.21.15.62:0/3474663518 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f85b81a18a0 con 0x7f859c06ecc0 2024-04-20T21:47:22.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:22.688+0000 7f85afe00700 1 -- 172.21.15.62:0/3474663518 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f85b81a18a0 con 0x7f859c06ecc0 2024-04-20T21:47:22.687 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:47:22.687 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T21:47:22.688 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T21:47:22.688 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-20T21:47:22.688 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T21:47:22.688 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr", 2024-04-20T21:47:22.688 INFO:teuthology.orchestra.run.smithi062.stdout: "crash", 2024-04-20T21:47:22.688 INFO:teuthology.orchestra.run.smithi062.stdout: "mon", 2024-04-20T21:47:22.688 INFO:teuthology.orchestra.run.smithi062.stdout: "osd" 2024-04-20T21:47:22.688 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T21:47:22.688 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "14/23 daemons upgraded", 2024-04-20T21:47:22.689 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading mds daemons", 2024-04-20T21:47:22.689 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T21:47:22.689 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:47:22.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:22.691+0000 7f85be800700 1 -- 172.21.15.62:0/3474663518 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f859c06ecc0 msgr2=0x7f859c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:47:22.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:22.691+0000 7f85be800700 1 --2- 172.21.15.62:0/3474663518 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f859c06ecc0 0x7f859c071170 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f85a8006010 tx=0x7f85a800b540 comp rx=0 tx=0).stop 2024-04-20T21:47:22.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:22.691+0000 7f85be800700 1 -- 172.21.15.62:0/3474663518 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f85b8073a00 msgr2=0x7f85b819c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:47:22.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:22.691+0000 7f85be800700 1 --2- 172.21.15.62:0/3474663518 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f85b8073a00 0x7f85b819c920 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f85a400d8d0 tx=0x7f85a400dc90 comp rx=0 tx=0).stop 2024-04-20T21:47:22.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:22.691+0000 7f85be800700 1 -- 172.21.15.62:0/3474663518 shutdown_connections 2024-04-20T21:47:22.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:22.691+0000 7f85be800700 1 --2- 172.21.15.62:0/3474663518 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f859c06ecc0 0x7f859c071170 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:47:22.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:22.691+0000 7f85be800700 1 --2- 172.21.15.62:0/3474663518 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f85b80730f0 0x7f85b819c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:47:22.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:22.691+0000 7f85be800700 1 --2- 172.21.15.62:0/3474663518 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f85b8073a00 0x7f85b819c920 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:47:22.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:22.691+0000 7f85be800700 1 -- 172.21.15.62:0/3474663518 >> 172.21.15.62:0/3474663518 conn(0x7f85b80fbda0 msgr2=0x7f85b8102860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:47:22.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:22.692+0000 7f85be800700 1 -- 172.21.15.62:0/3474663518 shutdown_connections 2024-04-20T21:47:22.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:22.692+0000 7f85be800700 1 -- 172.21.15.62:0/3474663518 wait complete. 2024-04-20T21:47:22.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:22.856+0000 7f2a80a00700 1 -- 172.21.15.62:0/3128545735 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2a7c106b80 msgr2=0x7f2a7c106ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:47:22.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:22.856+0000 7f2a80a00700 1 --2- 172.21.15.62:0/3128545735 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2a7c106b80 0x7f2a7c106ff0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f2a64009910 tx=0x7f2a64009c20 comp rx=0 tx=0).stop 2024-04-20T21:47:22.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:22.856+0000 7f2a80a00700 1 -- 172.21.15.62:0/3128545735 shutdown_connections 2024-04-20T21:47:22.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:22.856+0000 7f2a80a00700 1 --2- 172.21.15.62:0/3128545735 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2a7c106b80 0x7f2a7c106ff0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:47:22.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:22.856+0000 7f2a80a00700 1 --2- 172.21.15.62:0/3128545735 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a7c10c8f0 0x7f2a7c10ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:47:22.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:22.856+0000 7f2a80a00700 1 -- 172.21.15.62:0/3128545735 >> 172.21.15.62:0/3128545735 conn(0x7f2a7c076060 msgr2=0x7f2a7c078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:47:22.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:22.856+0000 7f2a80a00700 1 -- 172.21.15.62:0/3128545735 shutdown_connections 2024-04-20T21:47:22.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:22.856+0000 7f2a80a00700 1 -- 172.21.15.62:0/3128545735 wait complete. 2024-04-20T21:47:22.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:22.857+0000 7f2a80a00700 1 Processor -- start 2024-04-20T21:47:22.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:22.857+0000 7f2a80a00700 1 -- start start 2024-04-20T21:47:22.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:22.858+0000 7f2a80a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2a7c106b80 0x7f2a7c19c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:47:22.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:22.858+0000 7f2a80a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a7c10c8f0 0x7f2a7c19c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:47:22.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:22.858+0000 7f2a80a00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a7c19d060 con 0x7f2a7c106b80 2024-04-20T21:47:22.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:22.858+0000 7f2a80a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a7c1a0df0 con 0x7f2a7c10c8f0 2024-04-20T21:47:22.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:22.858+0000 7f2a79e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2a7c106b80 0x7f2a7c19c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:47:22.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:22.858+0000 7f2a79e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2a7c106b80 0x7f2a7c19c440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51720/0 (socket says 172.21.15.62:51720) 2024-04-20T21:47:22.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:22.858+0000 7f2a79e00700 1 -- 172.21.15.62:0/14789476 learned_addr learned my addr 172.21.15.62:0/14789476 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:47:22.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:22.858+0000 7f2a73e00700 1 --2- 172.21.15.62:0/14789476 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a7c10c8f0 0x7f2a7c19c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:47:22.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:22.858+0000 7f2a79e00700 1 -- 172.21.15.62:0/14789476 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a7c10c8f0 msgr2=0x7f2a7c19c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:47:22.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:22.858+0000 7f2a79e00700 1 --2- 172.21.15.62:0/14789476 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a7c10c8f0 0x7f2a7c19c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:47:22.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:22.858+0000 7f2a79e00700 1 -- 172.21.15.62:0/14789476 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a640095f0 con 0x7f2a7c106b80 2024-04-20T21:47:22.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:22.859+0000 7f2a79e00700 1 --2- 172.21.15.62:0/14789476 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2a7c106b80 0x7f2a7c19c440 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f2a6c00d8d0 tx=0x7f2a6c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:47:22.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:22.859+0000 7f2a72a00700 1 -- 172.21.15.62:0/14789476 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2a6c009940 con 0x7f2a7c106b80 2024-04-20T21:47:22.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:22.859+0000 7f2a72a00700 1 -- 172.21.15.62:0/14789476 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2a6c010460 con 0x7f2a7c106b80 2024-04-20T21:47:22.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:22.859+0000 7f2a80a00700 1 -- 172.21.15.62:0/14789476 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a7c1a10d0 con 0x7f2a7c106b80 2024-04-20T21:47:22.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:22.859+0000 7f2a72a00700 1 -- 172.21.15.62:0/14789476 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2a6c00f5f0 con 0x7f2a7c106b80 2024-04-20T21:47:22.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:22.859+0000 7f2a80a00700 1 -- 172.21.15.62:0/14789476 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a7c1a1620 con 0x7f2a7c106b80 2024-04-20T21:47:22.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:22.860+0000 7f2a80a00700 1 -- 172.21.15.62:0/14789476 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a7c10ecc0 con 0x7f2a7c106b80 2024-04-20T21:47:22.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:22.862+0000 7f2a72a00700 1 -- 172.21.15.62:0/14789476 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f2a6c0105d0 con 0x7f2a7c106b80 2024-04-20T21:47:22.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:22.863+0000 7f2a72a00700 1 --2- 172.21.15.62:0/14789476 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2a5c06ecc0 0x7f2a5c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:47:22.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:22.863+0000 7f2a72a00700 1 -- 172.21.15.62:0/14789476 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f2a6c08ef20 con 0x7f2a7c106b80 2024-04-20T21:47:22.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:22.863+0000 7f2a73e00700 1 --2- 172.21.15.62:0/14789476 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2a5c06ecc0 0x7f2a5c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:47:22.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:22.864+0000 7f2a73e00700 1 --2- 172.21.15.62:0/14789476 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2a5c06ecc0 0x7f2a5c071170 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f2a7c19da60 tx=0x7f2a6401a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:47:22.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:22.868+0000 7f2a72a00700 1 -- 172.21.15.62:0/14789476 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f2a6c057c30 con 0x7f2a7c106b80 2024-04-20T21:47:23.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:47:22 smithi062 ceph-mon[76322]: from='client.34712 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:47:23.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:47:22 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/3411869503' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:47:23.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:47:22 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/3285876361' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T21:47:23.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:23.186+0000 7f2a80a00700 1 -- 172.21.15.62:0/14789476 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f2a7c04f2a0 con 0x7f2a7c106b80 2024-04-20T21:47:23.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:23.187+0000 7f2a72a00700 1 -- 172.21.15.62:0/14789476 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7f2a6c05b2f0 con 0x7f2a7c106b80 2024-04-20T21:47:23.186 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-20T21:47:23.187 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-20T21:47:23.187 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is degraded 2024-04-20T21:47:23.187 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T21:47:23.187 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T21:47:23.187 INFO:teuthology.orchestra.run.smithi062.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-20T21:47:23.187 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-20T21:47:23.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:23.190+0000 7f2a80a00700 1 -- 172.21.15.62:0/14789476 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2a5c06ecc0 msgr2=0x7f2a5c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:47:23.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:23.190+0000 7f2a80a00700 1 --2- 172.21.15.62:0/14789476 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2a5c06ecc0 0x7f2a5c071170 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f2a7c19da60 tx=0x7f2a6401a040 comp rx=0 tx=0).stop 2024-04-20T21:47:23.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:23.190+0000 7f2a80a00700 1 -- 172.21.15.62:0/14789476 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2a7c106b80 msgr2=0x7f2a7c19c440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:47:23.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:23.190+0000 7f2a80a00700 1 --2- 172.21.15.62:0/14789476 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2a7c106b80 0x7f2a7c19c440 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f2a6c00d8d0 tx=0x7f2a6c00dc90 comp rx=0 tx=0).stop 2024-04-20T21:47:23.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:23.190+0000 7f2a80a00700 1 -- 172.21.15.62:0/14789476 shutdown_connections 2024-04-20T21:47:23.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:23.190+0000 7f2a80a00700 1 --2- 172.21.15.62:0/14789476 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2a5c06ecc0 0x7f2a5c071170 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:47:23.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:23.190+0000 7f2a80a00700 1 --2- 172.21.15.62:0/14789476 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2a7c106b80 0x7f2a7c19c440 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:47:23.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:23.190+0000 7f2a80a00700 1 --2- 172.21.15.62:0/14789476 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a7c10c8f0 0x7f2a7c19c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:47:23.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:23.190+0000 7f2a80a00700 1 -- 172.21.15.62:0/14789476 >> 172.21.15.62:0/14789476 conn(0x7f2a7c076060 msgr2=0x7f2a7c077ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:47:23.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:23.191+0000 7f2a80a00700 1 -- 172.21.15.62:0/14789476 shutdown_connections 2024-04-20T21:47:23.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:23.191+0000 7f2a80a00700 1 -- 172.21.15.62:0/14789476 wait complete. 2024-04-20T21:47:23.236 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:47:22 smithi189 ceph-mon[57022]: from='client.34712 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:47:23.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:47:22 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/3411869503' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:47:23.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:47:22 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/3285876361' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T21:47:24.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:47:23 smithi062 ceph-mon[76322]: pgmap v362: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:47:24.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:47:23 smithi062 ceph-mon[76322]: from='client.34724 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:47:24.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:47:23 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/14789476' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T21:47:24.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:47:23 smithi189 ceph-mon[57022]: pgmap v362: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:47:24.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:47:23 smithi189 ceph-mon[57022]: from='client.34724 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:47:24.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:47:23 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/14789476' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T21:47:26.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:47:25 smithi062 ceph-mon[76322]: pgmap v363: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:47:26.076 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:47:25 smithi189 ceph-mon[57022]: pgmap v363: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:47:28.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:47:27 smithi062 ceph-mon[76322]: pgmap v364: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:47:28.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:47:27 smithi189 ceph-mon[57022]: pgmap v364: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:47:30.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:47:29 smithi062 ceph-mon[76322]: pgmap v365: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:47:30.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:47:29 smithi189 ceph-mon[57022]: pgmap v365: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:47:32.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:47:31 smithi189 ceph-mon[57022]: pgmap v366: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:47:32.284 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:47:31 smithi062 ceph-mon[76322]: pgmap v366: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:47:34.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:47:33 smithi062 ceph-mon[76322]: pgmap v367: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:47:34.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:47:33 smithi189 ceph-mon[57022]: pgmap v367: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:47:35.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:47:34 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:47:35.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:47:34 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:47:36.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:47:35 smithi189 ceph-mon[57022]: pgmap v368: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:47:36.284 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:47:35 smithi062 ceph-mon[76322]: pgmap v368: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:47:38.143 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:47:37 smithi062 ceph-mon[76322]: pgmap v369: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:47:38.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:47:37 smithi189 ceph-mon[57022]: pgmap v369: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:47:40.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:47:39 smithi189 ceph-mon[57022]: pgmap v370: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:47:40.284 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:47:39 smithi062 ceph-mon[76322]: pgmap v370: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:47:42.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:47:41 smithi189 ceph-mon[57022]: pgmap v371: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:47:42.284 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:47:41 smithi062 ceph-mon[76322]: pgmap v371: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:47:44.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:47:43 smithi189 ceph-mon[57022]: pgmap v372: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:47:44.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:47:43 smithi062 ceph-mon[76322]: pgmap v372: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:47:46.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:47:45 smithi189 ceph-mon[57022]: pgmap v373: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:47:46.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:47:45 smithi062 ceph-mon[76322]: pgmap v373: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:47:48.142 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:47:47 smithi062 ceph-mon[76322]: pgmap v374: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:47:48.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:47:47 smithi189 ceph-mon[57022]: pgmap v374: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:47:50.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:47:49 smithi189 ceph-mon[57022]: pgmap v375: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:47:50.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:47:49 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:47:50.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:47:49 smithi062 ceph-mon[76322]: pgmap v375: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:47:50.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:47:49 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:47:52.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:47:51 smithi189 ceph-mon[57022]: pgmap v376: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:47:52.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:47:51 smithi062 ceph-mon[76322]: pgmap v376: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:47:53.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:53.368+0000 7f3307000700 1 -- 172.21.15.62:0/1540337355 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3300102470 msgr2=0x7f33001028e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:47:53.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:53.368+0000 7f3307000700 1 --2- 172.21.15.62:0/1540337355 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3300102470 0x7f33001028e0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f32ec009910 tx=0x7f32ec009c20 comp rx=0 tx=0).stop 2024-04-20T21:47:53.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:53.368+0000 7f3307000700 1 -- 172.21.15.62:0/1540337355 shutdown_connections 2024-04-20T21:47:53.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:53.368+0000 7f3307000700 1 --2- 172.21.15.62:0/1540337355 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3300102470 0x7f33001028e0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:47:53.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:53.368+0000 7f3307000700 1 --2- 172.21.15.62:0/1540337355 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3300108470 0x7f3300108840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:47:53.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:53.368+0000 7f3307000700 1 -- 172.21.15.62:0/1540337355 >> 172.21.15.62:0/1540337355 conn(0x7f33000fdf90 msgr2=0x7f33001003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:47:53.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:53.368+0000 7f3307000700 1 -- 172.21.15.62:0/1540337355 shutdown_connections 2024-04-20T21:47:53.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:53.368+0000 7f3307000700 1 -- 172.21.15.62:0/1540337355 wait complete. 2024-04-20T21:47:53.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:53.369+0000 7f3307000700 1 Processor -- start 2024-04-20T21:47:53.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:53.369+0000 7f3307000700 1 -- start start 2024-04-20T21:47:53.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:53.370+0000 7f3307000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3300102470 0x7f33001981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:47:53.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:53.370+0000 7f3307000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3300108470 0x7f3300198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:47:53.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:53.370+0000 7f3307000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3300198e10 con 0x7f3300102470 2024-04-20T21:47:53.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:53.370+0000 7f3307000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f330019cba0 con 0x7f3300108470 2024-04-20T21:47:53.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:53.370+0000 7f32ffe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3300102470 0x7f33001981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:47:53.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:53.370+0000 7f32ffe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3300102470 0x7f33001981f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37088/0 (socket says 172.21.15.62:37088) 2024-04-20T21:47:53.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:53.370+0000 7f32ffe00700 1 -- 172.21.15.62:0/3967210745 learned_addr learned my addr 172.21.15.62:0/3967210745 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:47:53.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:53.371+0000 7f32ff400700 1 --2- 172.21.15.62:0/3967210745 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3300108470 0x7f3300198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:47:53.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:53.371+0000 7f32ffe00700 1 -- 172.21.15.62:0/3967210745 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3300108470 msgr2=0x7f3300198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:47:53.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:53.371+0000 7f32ffe00700 1 --2- 172.21.15.62:0/3967210745 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3300108470 0x7f3300198730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:47:53.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:53.371+0000 7f32ffe00700 1 -- 172.21.15.62:0/3967210745 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32ec0095f0 con 0x7f3300102470 2024-04-20T21:47:53.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:53.371+0000 7f32ffe00700 1 --2- 172.21.15.62:0/3967210745 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3300102470 0x7f33001981f0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f32f000d8d0 tx=0x7f32f000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:47:53.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:53.371+0000 7f32fcc00700 1 -- 172.21.15.62:0/3967210745 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f32f0009940 con 0x7f3300102470 2024-04-20T21:47:53.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:53.372+0000 7f32fcc00700 1 -- 172.21.15.62:0/3967210745 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f32f0010460 con 0x7f3300102470 2024-04-20T21:47:53.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:53.372+0000 7f32fcc00700 1 -- 172.21.15.62:0/3967210745 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f32f000f5f0 con 0x7f3300102470 2024-04-20T21:47:53.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:53.372+0000 7f3307000700 1 -- 172.21.15.62:0/3967210745 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f330019ce80 con 0x7f3300102470 2024-04-20T21:47:53.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:53.372+0000 7f3307000700 1 -- 172.21.15.62:0/3967210745 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f330019d3d0 con 0x7f3300102470 2024-04-20T21:47:53.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:53.373+0000 7f3307000700 1 -- 172.21.15.62:0/3967210745 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f330010a970 con 0x7f3300102470 2024-04-20T21:47:53.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:53.374+0000 7f32fcc00700 1 -- 172.21.15.62:0/3967210745 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f32f00105d0 con 0x7f3300102470 2024-04-20T21:47:53.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:53.375+0000 7f32fcc00700 1 --2- 172.21.15.62:0/3967210745 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f32e406ecc0 0x7f32e4071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:47:53.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:53.375+0000 7f32fcc00700 1 -- 172.21.15.62:0/3967210745 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f32f008ee10 con 0x7f3300102470 2024-04-20T21:47:53.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:53.375+0000 7f32ff400700 1 --2- 172.21.15.62:0/3967210745 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f32e406ecc0 0x7f32e4071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:47:53.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:53.376+0000 7f32ff400700 1 --2- 172.21.15.62:0/3967210745 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f32e406ecc0 0x7f32e4071170 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f32ec000c00 tx=0x7f32ec005fd0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:47:53.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:53.381+0000 7f32fcc00700 1 -- 172.21.15.62:0/3967210745 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f32f0057ba0 con 0x7f3300102470 2024-04-20T21:47:53.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:53.638+0000 7f3307000700 1 -- 172.21.15.62:0/3967210745 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f330019d640 con 0x7f32e406ecc0 2024-04-20T21:47:53.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:53.640+0000 7f32fcc00700 1 -- 172.21.15.62:0/3967210745 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f330019d640 con 0x7f32e406ecc0 2024-04-20T21:47:53.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:53.643+0000 7f3307000700 1 -- 172.21.15.62:0/3967210745 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f32e406ecc0 msgr2=0x7f32e4071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:47:53.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:53.643+0000 7f3307000700 1 --2- 172.21.15.62:0/3967210745 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f32e406ecc0 0x7f32e4071170 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f32ec000c00 tx=0x7f32ec005fd0 comp rx=0 tx=0).stop 2024-04-20T21:47:53.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:53.643+0000 7f3307000700 1 -- 172.21.15.62:0/3967210745 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3300102470 msgr2=0x7f33001981f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:47:53.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:53.643+0000 7f3307000700 1 --2- 172.21.15.62:0/3967210745 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3300102470 0x7f33001981f0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f32f000d8d0 tx=0x7f32f000dc90 comp rx=0 tx=0).stop 2024-04-20T21:47:53.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:53.643+0000 7f3307000700 1 -- 172.21.15.62:0/3967210745 shutdown_connections 2024-04-20T21:47:53.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:53.643+0000 7f3307000700 1 --2- 172.21.15.62:0/3967210745 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f32e406ecc0 0x7f32e4071170 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:47:53.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:53.643+0000 7f3307000700 1 --2- 172.21.15.62:0/3967210745 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3300102470 0x7f33001981f0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:47:53.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:53.643+0000 7f3307000700 1 --2- 172.21.15.62:0/3967210745 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3300108470 0x7f3300198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:47:53.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:53.643+0000 7f3307000700 1 -- 172.21.15.62:0/3967210745 >> 172.21.15.62:0/3967210745 conn(0x7f33000fdf90 msgr2=0x7f33000ff3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:47:53.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:53.644+0000 7f3307000700 1 -- 172.21.15.62:0/3967210745 shutdown_connections 2024-04-20T21:47:53.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:53.644+0000 7f3307000700 1 -- 172.21.15.62:0/3967210745 wait complete. 2024-04-20T21:47:53.660 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T21:47:53.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:53.807+0000 7f8f0f200700 1 -- 172.21.15.62:0/1252224630 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8f101024d0 msgr2=0x7f8f101028a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:47:53.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:53.807+0000 7f8f0f200700 1 --2- 172.21.15.62:0/1252224630 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8f101024d0 0x7f8f101028a0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f8f00009910 tx=0x7f8f00009c20 comp rx=0 tx=0).stop 2024-04-20T21:47:53.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:53.807+0000 7f8f0f200700 1 -- 172.21.15.62:0/1252224630 shutdown_connections 2024-04-20T21:47:53.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:53.807+0000 7f8f0f200700 1 --2- 172.21.15.62:0/1252224630 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8f10102e70 0x7f8f10106e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:47:53.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:53.807+0000 7f8f0f200700 1 --2- 172.21.15.62:0/1252224630 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8f101024d0 0x7f8f101028a0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:47:53.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:53.807+0000 7f8f0f200700 1 -- 172.21.15.62:0/1252224630 >> 172.21.15.62:0/1252224630 conn(0x7f8f100fdd90 msgr2=0x7f8f101001a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:47:53.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:53.807+0000 7f8f0f200700 1 -- 172.21.15.62:0/1252224630 shutdown_connections 2024-04-20T21:47:53.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:53.807+0000 7f8f0f200700 1 -- 172.21.15.62:0/1252224630 wait complete. 2024-04-20T21:47:53.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:53.808+0000 7f8f0f200700 1 Processor -- start 2024-04-20T21:47:53.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:53.808+0000 7f8f0f200700 1 -- start start 2024-04-20T21:47:53.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:53.809+0000 7f8f0f200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8f101024d0 0x7f8f10197e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:47:53.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:53.809+0000 7f8f0f200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8f10102e70 0x7f8f101983a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:47:53.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:53.809+0000 7f8f0f200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f10198a80 con 0x7f8f101024d0 2024-04-20T21:47:53.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:53.809+0000 7f8f0f200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f1019c810 con 0x7f8f10102e70 2024-04-20T21:47:53.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:53.809+0000 7f8f0de00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8f101024d0 0x7f8f10197e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:47:53.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:53.809+0000 7f8f0de00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8f101024d0 0x7f8f10197e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37118/0 (socket says 172.21.15.62:37118) 2024-04-20T21:47:53.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:53.809+0000 7f8f0de00700 1 -- 172.21.15.62:0/978507482 learned_addr learned my addr 172.21.15.62:0/978507482 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:47:53.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:53.810+0000 7f8f0d400700 1 --2- 172.21.15.62:0/978507482 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8f10102e70 0x7f8f101983a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:47:53.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:53.810+0000 7f8f0de00700 1 -- 172.21.15.62:0/978507482 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8f10102e70 msgr2=0x7f8f101983a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:47:53.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:53.810+0000 7f8f0de00700 1 --2- 172.21.15.62:0/978507482 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8f10102e70 0x7f8f101983a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:47:53.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:53.810+0000 7f8f0de00700 1 -- 172.21.15.62:0/978507482 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f000095f0 con 0x7f8f101024d0 2024-04-20T21:47:53.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:53.810+0000 7f8f0de00700 1 --2- 172.21.15.62:0/978507482 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8f101024d0 0x7f8f10197e60 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f8f00006010 tx=0x7f8f0000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:47:53.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:53.810+0000 7f8f06a00700 1 -- 172.21.15.62:0/978507482 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8f0001d070 con 0x7f8f101024d0 2024-04-20T21:47:53.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:53.811+0000 7f8f06a00700 1 -- 172.21.15.62:0/978507482 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8f0000f460 con 0x7f8f101024d0 2024-04-20T21:47:53.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:53.811+0000 7f8f0f200700 1 -- 172.21.15.62:0/978507482 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f1019ca90 con 0x7f8f101024d0 2024-04-20T21:47:53.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:53.811+0000 7f8f06a00700 1 -- 172.21.15.62:0/978507482 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8f00017630 con 0x7f8f101024d0 2024-04-20T21:47:53.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:53.811+0000 7f8f0f200700 1 -- 172.21.15.62:0/978507482 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f1019cf80 con 0x7f8f101024d0 2024-04-20T21:47:53.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:53.812+0000 7f8f0f200700 1 -- 172.21.15.62:0/978507482 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f1010a7e0 con 0x7f8f101024d0 2024-04-20T21:47:53.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:53.813+0000 7f8f06a00700 1 -- 172.21.15.62:0/978507482 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f8f0000f5d0 con 0x7f8f101024d0 2024-04-20T21:47:53.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:53.814+0000 7f8f06a00700 1 --2- 172.21.15.62:0/978507482 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f8ef406ecc0 0x7f8ef4071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:47:53.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:53.814+0000 7f8f06a00700 1 -- 172.21.15.62:0/978507482 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f8f000903b0 con 0x7f8f101024d0 2024-04-20T21:47:53.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:53.814+0000 7f8f0d400700 1 --2- 172.21.15.62:0/978507482 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f8ef406ecc0 0x7f8ef4071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:47:53.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:53.815+0000 7f8f0d400700 1 --2- 172.21.15.62:0/978507482 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f8ef406ecc0 0x7f8ef4071170 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f8efc005950 tx=0x7f8efc0058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:47:53.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:53.819+0000 7f8f06a00700 1 -- 172.21.15.62:0/978507482 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f8f000590c0 con 0x7f8f101024d0 2024-04-20T21:47:54.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:47:53 smithi062 ceph-mon[76322]: pgmap v377: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:47:54.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:54.077+0000 7f8f0f200700 1 -- 172.21.15.62:0/978507482 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8f1019d260 con 0x7f8ef406ecc0 2024-04-20T21:47:54.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:54.079+0000 7f8f06a00700 1 -- 172.21.15.62:0/978507482 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f8f1019d260 con 0x7f8ef406ecc0 2024-04-20T21:47:54.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:54.082+0000 7f8f0f200700 1 -- 172.21.15.62:0/978507482 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f8ef406ecc0 msgr2=0x7f8ef4071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:47:54.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:54.082+0000 7f8f0f200700 1 --2- 172.21.15.62:0/978507482 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f8ef406ecc0 0x7f8ef4071170 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f8efc005950 tx=0x7f8efc0058e0 comp rx=0 tx=0).stop 2024-04-20T21:47:54.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:54.082+0000 7f8f0f200700 1 -- 172.21.15.62:0/978507482 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8f101024d0 msgr2=0x7f8f10197e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:47:54.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:54.082+0000 7f8f0f200700 1 --2- 172.21.15.62:0/978507482 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8f101024d0 0x7f8f10197e60 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f8f00006010 tx=0x7f8f0000be20 comp rx=0 tx=0).stop 2024-04-20T21:47:54.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:54.083+0000 7f8f0f200700 1 -- 172.21.15.62:0/978507482 shutdown_connections 2024-04-20T21:47:54.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:54.083+0000 7f8f0f200700 1 --2- 172.21.15.62:0/978507482 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f8ef406ecc0 0x7f8ef4071170 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:47:54.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:54.083+0000 7f8f0f200700 1 --2- 172.21.15.62:0/978507482 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8f101024d0 0x7f8f10197e60 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:47:54.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:54.083+0000 7f8f0f200700 1 --2- 172.21.15.62:0/978507482 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8f10102e70 0x7f8f101983a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:47:54.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:54.083+0000 7f8f0f200700 1 -- 172.21.15.62:0/978507482 >> 172.21.15.62:0/978507482 conn(0x7f8f100fdd90 msgr2=0x7f8f10100180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:47:54.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:54.083+0000 7f8f0f200700 1 -- 172.21.15.62:0/978507482 shutdown_connections 2024-04-20T21:47:54.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:54.083+0000 7f8f0f200700 1 -- 172.21.15.62:0/978507482 wait complete. 2024-04-20T21:47:54.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:47:53 smithi189 ceph-mon[57022]: pgmap v377: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:47:54.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:54.244+0000 7f5492a00700 1 -- 172.21.15.62:0/260083286 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f548c102e70 msgr2=0x7f548c106e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:47:54.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:54.244+0000 7f5492a00700 1 --2- 172.21.15.62:0/260083286 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f548c102e70 0x7f548c106e40 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f5478009910 tx=0x7f5478009c20 comp rx=0 tx=0).stop 2024-04-20T21:47:54.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:54.244+0000 7f5492a00700 1 -- 172.21.15.62:0/260083286 shutdown_connections 2024-04-20T21:47:54.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:54.244+0000 7f5492a00700 1 --2- 172.21.15.62:0/260083286 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f548c102e70 0x7f548c106e40 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:47:54.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:54.244+0000 7f5492a00700 1 --2- 172.21.15.62:0/260083286 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f548c1024d0 0x7f548c1028a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:47:54.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:54.244+0000 7f5492a00700 1 -- 172.21.15.62:0/260083286 >> 172.21.15.62:0/260083286 conn(0x7f548c0fdd90 msgr2=0x7f548c1001a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:47:54.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:54.244+0000 7f5492a00700 1 -- 172.21.15.62:0/260083286 shutdown_connections 2024-04-20T21:47:54.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:54.244+0000 7f5492a00700 1 -- 172.21.15.62:0/260083286 wait complete. 2024-04-20T21:47:54.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:54.245+0000 7f5492a00700 1 Processor -- start 2024-04-20T21:47:54.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:54.246+0000 7f5492a00700 1 -- start start 2024-04-20T21:47:54.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:54.246+0000 7f5492a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f548c1024d0 0x7f548c197e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:47:54.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:54.246+0000 7f5492a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f548c102e70 0x7f548c198350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:47:54.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:54.246+0000 7f5492a00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f548c198a30 con 0x7f548c102e70 2024-04-20T21:47:54.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:54.246+0000 7f5492a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f548c19c7c0 con 0x7f548c1024d0 2024-04-20T21:47:54.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:54.246+0000 7f5490c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f548c102e70 0x7f548c198350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:47:54.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:54.246+0000 7f5490c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f548c102e70 0x7f548c198350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37140/0 (socket says 172.21.15.62:37140) 2024-04-20T21:47:54.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:54.246+0000 7f5490c00700 1 -- 172.21.15.62:0/724875298 learned_addr learned my addr 172.21.15.62:0/724875298 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:47:54.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:54.247+0000 7f5491600700 1 --2- 172.21.15.62:0/724875298 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f548c1024d0 0x7f548c197e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:47:54.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:54.247+0000 7f5490c00700 1 -- 172.21.15.62:0/724875298 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f548c1024d0 msgr2=0x7f548c197e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:47:54.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:54.247+0000 7f5490c00700 1 --2- 172.21.15.62:0/724875298 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f548c1024d0 0x7f548c197e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:47:54.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:54.247+0000 7f5490c00700 1 -- 172.21.15.62:0/724875298 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54780095f0 con 0x7f548c102e70 2024-04-20T21:47:54.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:54.247+0000 7f5491600700 1 --2- 172.21.15.62:0/724875298 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f548c1024d0 0x7f548c197e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T21:47:54.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:54.247+0000 7f5490c00700 1 --2- 172.21.15.62:0/724875298 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f548c102e70 0x7f548c198350 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f547800bda0 tx=0x7f547800be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:47:54.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:54.248+0000 7f548a000700 1 -- 172.21.15.62:0/724875298 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f547801d070 con 0x7f548c102e70 2024-04-20T21:47:54.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:54.248+0000 7f548a000700 1 -- 172.21.15.62:0/724875298 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5478004fb0 con 0x7f548c102e70 2024-04-20T21:47:54.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:54.248+0000 7f548a000700 1 -- 172.21.15.62:0/724875298 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f547800f480 con 0x7f548c102e70 2024-04-20T21:47:54.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:54.248+0000 7f5492a00700 1 -- 172.21.15.62:0/724875298 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f548c19ca40 con 0x7f548c102e70 2024-04-20T21:47:54.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:54.248+0000 7f5492a00700 1 -- 172.21.15.62:0/724875298 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f548c19cf30 con 0x7f548c102e70 2024-04-20T21:47:54.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:54.249+0000 7f5492a00700 1 -- 172.21.15.62:0/724875298 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f548c10a840 con 0x7f548c102e70 2024-04-20T21:47:54.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:54.250+0000 7f548a000700 1 -- 172.21.15.62:0/724875298 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f5478022870 con 0x7f548c102e70 2024-04-20T21:47:54.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:54.251+0000 7f548a000700 1 --2- 172.21.15.62:0/724875298 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f547406ecc0 0x7f5474071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:47:54.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:54.251+0000 7f548a000700 1 -- 172.21.15.62:0/724875298 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f5478090350 con 0x7f548c102e70 2024-04-20T21:47:54.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:54.252+0000 7f5491600700 1 --2- 172.21.15.62:0/724875298 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f547406ecc0 0x7f5474071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:47:54.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:54.252+0000 7f5491600700 1 --2- 172.21.15.62:0/724875298 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f547406ecc0 0x7f5474071170 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f5480005fd0 tx=0x7f5480005dc0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:47:54.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:54.256+0000 7f548a000700 1 -- 172.21.15.62:0/724875298 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f5478059190 con 0x7f548c102e70 2024-04-20T21:47:54.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:54.507+0000 7f5492a00700 1 -- 172.21.15.62:0/724875298 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f548c02ccf0 con 0x7f547406ecc0 2024-04-20T21:47:54.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:54.540+0000 7f548a000700 1 -- 172.21.15.62:0/724875298 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f548c02ccf0 con 0x7f547406ecc0 2024-04-20T21:47:54.540 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T21:47:54.540 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (13m) 5m ago 27m 14.2M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T21:47:54.540 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (27m) 5m ago 27m 9512k - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T21:47:54.540 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (26m) 6m ago 26m 8942k - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T21:47:54.540 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (9m) 5m ago 27m 6660k - 19.0.0-3133-ga8c84f59 76bc6b91b726 2825d5ba11c6 2024-04-20T21:47:54.540 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (9m) 6m ago 26m 6665k - 19.0.0-3133-ga8c84f59 76bc6b91b726 20f746fa1bba 2024-04-20T21:47:54.540 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (13m) 5m ago 27m 74.2M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T21:47:54.540 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (5m) 5m ago 20m 11.2M - 19.0.0-3133-ga8c84f59 76bc6b91b726 de80bc90a683 2024-04-20T21:47:54.540 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (5m) 5m ago 20m 11.9M - 19.0.0-3133-ga8c84f59 76bc6b91b726 cd793b7ee98e 2024-04-20T21:47:54.540 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (20m) 6m ago 20m 117M - 18.2.0 dc2bc1663786 fb2a8a223db8 2024-04-20T21:47:54.540 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (20m) 6m ago 20m 55.6M - 18.2.0 dc2bc1663786 f2eec2ee8951 2024-04-20T21:47:54.541 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (16m) 5m ago 29m 511M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T21:47:54.541 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (15m) 6m ago 26m 452M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T21:47:54.541 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (11m) 5m ago 29m 55.9M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 019eb1479485 2024-04-20T21:47:54.541 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (10m) 6m ago 26m 39.8M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 55aac560aab6 2024-04-20T21:47:54.541 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (14m) 5m ago 27m 7952k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T21:47:54.541 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (14m) 6m ago 26m 8304k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T21:47:54.541 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (9m) 5m ago 25m 88.0M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 ee2bb901f26e 2024-04-20T21:47:54.541 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (8m) 5m ago 24m 92.3M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 9075152d4162 2024-04-20T21:47:54.541 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (8m) 5m ago 23m 74.4M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 2d4f3e38dc0e 2024-04-20T21:47:54.541 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (7m) 6m ago 23m 87.6M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 8f97b17ae0eb 2024-04-20T21:47:54.541 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (6m) 6m ago 22m 90.7M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 653d8079967d 2024-04-20T21:47:54.541 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (6m) 6m ago 22m 11.5M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 596dded14c12 2024-04-20T21:47:54.541 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (14m) 5m ago 27m 74.0M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T21:47:54.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:54.544+0000 7f5492a00700 1 -- 172.21.15.62:0/724875298 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f547406ecc0 msgr2=0x7f5474071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:47:54.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:54.544+0000 7f5492a00700 1 --2- 172.21.15.62:0/724875298 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f547406ecc0 0x7f5474071170 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f5480005fd0 tx=0x7f5480005dc0 comp rx=0 tx=0).stop 2024-04-20T21:47:54.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:54.544+0000 7f5492a00700 1 -- 172.21.15.62:0/724875298 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f548c102e70 msgr2=0x7f548c198350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:47:54.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:54.544+0000 7f5492a00700 1 --2- 172.21.15.62:0/724875298 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f548c102e70 0x7f548c198350 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f547800bda0 tx=0x7f547800be80 comp rx=0 tx=0).stop 2024-04-20T21:47:54.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:54.544+0000 7f5492a00700 1 -- 172.21.15.62:0/724875298 shutdown_connections 2024-04-20T21:47:54.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:54.544+0000 7f5492a00700 1 --2- 172.21.15.62:0/724875298 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f547406ecc0 0x7f5474071170 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:47:54.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:54.544+0000 7f5492a00700 1 --2- 172.21.15.62:0/724875298 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f548c1024d0 0x7f548c197e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:47:54.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:54.544+0000 7f5492a00700 1 --2- 172.21.15.62:0/724875298 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f548c102e70 0x7f548c198350 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:47:54.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:54.544+0000 7f5492a00700 1 -- 172.21.15.62:0/724875298 >> 172.21.15.62:0/724875298 conn(0x7f548c0fdd90 msgr2=0x7f548c100180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:47:54.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:54.545+0000 7f5492a00700 1 -- 172.21.15.62:0/724875298 shutdown_connections 2024-04-20T21:47:54.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:54.545+0000 7f5492a00700 1 -- 172.21.15.62:0/724875298 wait complete. 2024-04-20T21:47:54.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:54.710+0000 7f21d1400700 1 -- 172.21.15.62:0/1993650263 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f21cc1024c0 msgr2=0x7f21cc102890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:47:54.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:54.710+0000 7f21d1400700 1 --2- 172.21.15.62:0/1993650263 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f21cc1024c0 0x7f21cc102890 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f21bc009910 tx=0x7f21bc009c20 comp rx=0 tx=0).stop 2024-04-20T21:47:54.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:54.710+0000 7f21d1400700 1 -- 172.21.15.62:0/1993650263 shutdown_connections 2024-04-20T21:47:54.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:54.710+0000 7f21d1400700 1 --2- 172.21.15.62:0/1993650263 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21cc102e60 0x7f21cc106e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:47:54.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:54.710+0000 7f21d1400700 1 --2- 172.21.15.62:0/1993650263 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f21cc1024c0 0x7f21cc102890 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:47:54.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:54.710+0000 7f21d1400700 1 -- 172.21.15.62:0/1993650263 >> 172.21.15.62:0/1993650263 conn(0x7f21cc0fdd60 msgr2=0x7f21cc100170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:47:54.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:54.711+0000 7f21d1400700 1 -- 172.21.15.62:0/1993650263 shutdown_connections 2024-04-20T21:47:54.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:54.711+0000 7f21d1400700 1 -- 172.21.15.62:0/1993650263 wait complete. 2024-04-20T21:47:54.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:54.712+0000 7f21d1400700 1 Processor -- start 2024-04-20T21:47:54.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:54.712+0000 7f21d1400700 1 -- start start 2024-04-20T21:47:54.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:54.712+0000 7f21d1400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21cc1024c0 0x7f21cc197e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:47:54.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:54.712+0000 7f21d1400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f21cc102e60 0x7f21cc198370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:47:54.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:54.712+0000 7f21d1400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc198a50 con 0x7f21cc102e60 2024-04-20T21:47:54.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:54.712+0000 7f21d1400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc19c7e0 con 0x7f21cc1024c0 2024-04-20T21:47:54.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:54.713+0000 7f21cb400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f21cc102e60 0x7f21cc198370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:47:54.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:54.713+0000 7f21cb400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f21cc102e60 0x7f21cc198370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37154/0 (socket says 172.21.15.62:37154) 2024-04-20T21:47:54.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:54.713+0000 7f21cb400700 1 -- 172.21.15.62:0/3784455701 learned_addr learned my addr 172.21.15.62:0/3784455701 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:47:54.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:54.713+0000 7f21cbe00700 1 --2- 172.21.15.62:0/3784455701 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21cc1024c0 0x7f21cc197e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:47:54.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:54.713+0000 7f21cb400700 1 -- 172.21.15.62:0/3784455701 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21cc1024c0 msgr2=0x7f21cc197e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:47:54.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:54.713+0000 7f21cb400700 1 --2- 172.21.15.62:0/3784455701 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21cc1024c0 0x7f21cc197e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:47:54.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:54.713+0000 7f21cb400700 1 -- 172.21.15.62:0/3784455701 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21bc0095f0 con 0x7f21cc102e60 2024-04-20T21:47:54.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:54.713+0000 7f21cbe00700 1 --2- 172.21.15.62:0/3784455701 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21cc1024c0 0x7f21cc197e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T21:47:54.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:54.713+0000 7f21cb400700 1 --2- 172.21.15.62:0/3784455701 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f21cc102e60 0x7f21cc198370 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f21b800d900 tx=0x7f21b800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:47:54.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:54.714+0000 7f21c8c00700 1 -- 172.21.15.62:0/3784455701 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f21b8009900 con 0x7f21cc102e60 2024-04-20T21:47:54.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:54.714+0000 7f21c8c00700 1 -- 172.21.15.62:0/3784455701 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f21b8010460 con 0x7f21cc102e60 2024-04-20T21:47:54.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:54.714+0000 7f21c8c00700 1 -- 172.21.15.62:0/3784455701 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f21b800f5f0 con 0x7f21cc102e60 2024-04-20T21:47:54.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:54.714+0000 7f21d1400700 1 -- 172.21.15.62:0/3784455701 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21cc19cac0 con 0x7f21cc102e60 2024-04-20T21:47:54.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:54.714+0000 7f21d1400700 1 -- 172.21.15.62:0/3784455701 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21cc19d010 con 0x7f21cc102e60 2024-04-20T21:47:54.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:54.715+0000 7f21d1400700 1 -- 172.21.15.62:0/3784455701 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21cc10a7d0 con 0x7f21cc102e60 2024-04-20T21:47:54.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:54.716+0000 7f21c8c00700 1 -- 172.21.15.62:0/3784455701 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f21b80105d0 con 0x7f21cc102e60 2024-04-20T21:47:54.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:54.717+0000 7f21c8c00700 1 --2- 172.21.15.62:0/3784455701 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f21b006ecc0 0x7f21b0071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:47:54.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:54.717+0000 7f21c8c00700 1 -- 172.21.15.62:0/3784455701 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f21b808eca0 con 0x7f21cc102e60 2024-04-20T21:47:54.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:54.717+0000 7f21cbe00700 1 --2- 172.21.15.62:0/3784455701 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f21b006ecc0 0x7f21b0071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:47:54.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:54.718+0000 7f21cbe00700 1 --2- 172.21.15.62:0/3784455701 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f21b006ecc0 0x7f21b0071170 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f21bc00b5c0 tx=0x7f21bc0053a0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:47:54.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:54.722+0000 7f21c8c00700 1 -- 172.21.15.62:0/3784455701 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f21b8057a30 con 0x7f21cc102e60 2024-04-20T21:47:55.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.059+0000 7f21d1400700 1 -- 172.21.15.62:0/3784455701 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f21cc062380 con 0x7f21cc102e60 2024-04-20T21:47:55.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.059+0000 7f21c8c00700 1 -- 172.21.15.62:0/3784455701 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7f21b805b0f0 con 0x7f21cc102e60 2024-04-20T21:47:55.059 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:47:55.059 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T21:47:55.059 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:47:55.059 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:47:55.059 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T21:47:55.059 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:47:55.059 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:47:55.059 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T21:47:55.059 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 6 2024-04-20T21:47:55.059 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:47:55.060 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T21:47:55.060 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 2, 2024-04-20T21:47:55.060 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:47:55.060 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:47:55.060 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T21:47:55.060 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 2, 2024-04-20T21:47:55.060 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 12 2024-04-20T21:47:55.060 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T21:47:55.060 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:47:55.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.062+0000 7f21d1400700 1 -- 172.21.15.62:0/3784455701 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f21b006ecc0 msgr2=0x7f21b0071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:47:55.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.062+0000 7f21d1400700 1 --2- 172.21.15.62:0/3784455701 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f21b006ecc0 0x7f21b0071170 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f21bc00b5c0 tx=0x7f21bc0053a0 comp rx=0 tx=0).stop 2024-04-20T21:47:55.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.062+0000 7f21d1400700 1 -- 172.21.15.62:0/3784455701 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f21cc102e60 msgr2=0x7f21cc198370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:47:55.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.062+0000 7f21d1400700 1 --2- 172.21.15.62:0/3784455701 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f21cc102e60 0x7f21cc198370 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f21b800d900 tx=0x7f21b800dcc0 comp rx=0 tx=0).stop 2024-04-20T21:47:55.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.063+0000 7f21d1400700 1 -- 172.21.15.62:0/3784455701 shutdown_connections 2024-04-20T21:47:55.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.063+0000 7f21d1400700 1 --2- 172.21.15.62:0/3784455701 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f21b006ecc0 0x7f21b0071170 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:47:55.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.063+0000 7f21d1400700 1 --2- 172.21.15.62:0/3784455701 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21cc1024c0 0x7f21cc197e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:47:55.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.063+0000 7f21d1400700 1 --2- 172.21.15.62:0/3784455701 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f21cc102e60 0x7f21cc198370 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:47:55.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.063+0000 7f21d1400700 1 -- 172.21.15.62:0/3784455701 >> 172.21.15.62:0/3784455701 conn(0x7f21cc0fdd60 msgr2=0x7f21cc100120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:47:55.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.063+0000 7f21d1400700 1 -- 172.21.15.62:0/3784455701 shutdown_connections 2024-04-20T21:47:55.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.063+0000 7f21d1400700 1 -- 172.21.15.62:0/3784455701 wait complete. 2024-04-20T21:47:55.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.210+0000 7f5198200700 1 -- 172.21.15.62:0/3841415367 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5190100e80 msgr2=0x7f5190104e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:47:55.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.210+0000 7f5198200700 1 --2- 172.21.15.62:0/3841415367 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5190100e80 0x7f5190104e50 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f517c009910 tx=0x7f517c009c20 comp rx=0 tx=0).stop 2024-04-20T21:47:55.210 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:47:54 smithi062 ceph-mon[76322]: from='client.34732 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:47:55.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.210+0000 7f5198200700 1 -- 172.21.15.62:0/3841415367 shutdown_connections 2024-04-20T21:47:55.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.210+0000 7f5198200700 1 --2- 172.21.15.62:0/3841415367 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5190100e80 0x7f5190104e50 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:47:55.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.210+0000 7f5198200700 1 --2- 172.21.15.62:0/3841415367 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f51901004e0 0x7f51901008b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:47:55.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.210+0000 7f5198200700 1 -- 172.21.15.62:0/3841415367 >> 172.21.15.62:0/3841415367 conn(0x7f51900fbda0 msgr2=0x7f51900fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:47:55.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.210+0000 7f5198200700 1 -- 172.21.15.62:0/3841415367 shutdown_connections 2024-04-20T21:47:55.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.210+0000 7f5198200700 1 -- 172.21.15.62:0/3841415367 wait complete. 2024-04-20T21:47:55.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.212+0000 7f5198200700 1 Processor -- start 2024-04-20T21:47:55.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.212+0000 7f5198200700 1 -- start start 2024-04-20T21:47:55.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.213+0000 7f5198200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51901004e0 0x7f5190195f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:47:55.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.213+0000 7f5198200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5190100e80 0x7f5190196450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:47:55.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.213+0000 7f5198200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5190196b30 con 0x7f51901004e0 2024-04-20T21:47:55.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.213+0000 7f5198200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f519019a8c0 con 0x7f5190100e80 2024-04-20T21:47:55.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.213+0000 7f5195800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51901004e0 0x7f5190195f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:47:55.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.213+0000 7f5195800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51901004e0 0x7f5190195f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37166/0 (socket says 172.21.15.62:37166) 2024-04-20T21:47:55.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.213+0000 7f5195800700 1 -- 172.21.15.62:0/47554386 learned_addr learned my addr 172.21.15.62:0/47554386 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:47:55.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.213+0000 7f5194e00700 1 --2- 172.21.15.62:0/47554386 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5190100e80 0x7f5190196450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:47:55.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.213+0000 7f5195800700 1 -- 172.21.15.62:0/47554386 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5190100e80 msgr2=0x7f5190196450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:47:55.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.213+0000 7f5195800700 1 --2- 172.21.15.62:0/47554386 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5190100e80 0x7f5190196450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:47:55.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.213+0000 7f5195800700 1 -- 172.21.15.62:0/47554386 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f517c0095f0 con 0x7f51901004e0 2024-04-20T21:47:55.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.213+0000 7f5194e00700 1 --2- 172.21.15.62:0/47554386 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5190100e80 0x7f5190196450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T21:47:55.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.214+0000 7f5195800700 1 --2- 172.21.15.62:0/47554386 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51901004e0 0x7f5190195f10 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f518400d8d0 tx=0x7f518400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:47:55.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.214+0000 7f518e000700 1 -- 172.21.15.62:0/47554386 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5184009940 con 0x7f51901004e0 2024-04-20T21:47:55.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.214+0000 7f518e000700 1 -- 172.21.15.62:0/47554386 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5184010460 con 0x7f51901004e0 2024-04-20T21:47:55.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.214+0000 7f5198200700 1 -- 172.21.15.62:0/47554386 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f519019aba0 con 0x7f51901004e0 2024-04-20T21:47:55.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.214+0000 7f518e000700 1 -- 172.21.15.62:0/47554386 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f518400f5f0 con 0x7f51901004e0 2024-04-20T21:47:55.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.214+0000 7f5198200700 1 -- 172.21.15.62:0/47554386 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f519019b0f0 con 0x7f51901004e0 2024-04-20T21:47:55.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.216+0000 7f5198200700 1 -- 172.21.15.62:0/47554386 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5190108850 con 0x7f51901004e0 2024-04-20T21:47:55.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.217+0000 7f518e000700 1 -- 172.21.15.62:0/47554386 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f51840105d0 con 0x7f51901004e0 2024-04-20T21:47:55.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.218+0000 7f518e000700 1 --2- 172.21.15.62:0/47554386 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f517806ecc0 0x7f5178071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:47:55.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.218+0000 7f518e000700 1 -- 172.21.15.62:0/47554386 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f518408ef20 con 0x7f51901004e0 2024-04-20T21:47:55.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.218+0000 7f5194e00700 1 --2- 172.21.15.62:0/47554386 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f517806ecc0 0x7f5178071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:47:55.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.219+0000 7f5194e00700 1 --2- 172.21.15.62:0/47554386 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f517806ecc0 0x7f5178071170 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f517c006010 tx=0x7f517c005960 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:47:55.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.223+0000 7f518e000700 1 -- 172.21.15.62:0/47554386 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f5184057c30 con 0x7f51901004e0 2024-04-20T21:47:55.236 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:47:54 smithi189 ceph-mon[57022]: from='client.34732 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:47:55.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.511+0000 7f5198200700 1 -- 172.21.15.62:0/47554386 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f519004f2a0 con 0x7f51901004e0 2024-04-20T21:47:55.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.512+0000 7f518e000700 1 -- 172.21.15.62:0/47554386 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 25 v25) v1 ==== 76+0+1887 (secure 0 0 0) 0x7f518405b2f0 con 0x7f51901004e0 2024-04-20T21:47:55.512 INFO:teuthology.orchestra.run.smithi062.stdout:e25 2024-04-20T21:47:55.512 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T21:47:55.512 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T21:47:55.512 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T21:47:55.512 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:47:55.512 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T21:47:55.512 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T21:47:55.512 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 20 2024-04-20T21:47:55.512 INFO:teuthology.orchestra.run.smithi062.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-20T21:47:55.512 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T21:47:55.512 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T21:42:01.559061+0000 2024-04-20T21:47:55.512 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T21:47:55.513 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T21:47:55.513 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T21:47:55.513 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T21:47:55.513 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T21:47:55.513 INFO:teuthology.orchestra.run.smithi062.stdout:max_xattr_size 65536 2024-04-20T21:47:55.513 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T21:47:55.513 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T21:47:55.513 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 81 2024-04-20T21:47:55.513 INFO:teuthology.orchestra.run.smithi062.stdout:snap_rstat 1 2024-04-20T21:47:55.513 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T21:47:55.513 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 2 2024-04-20T21:47:55.513 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1 2024-04-20T21:47:55.513 INFO:teuthology.orchestra.run.smithi062.stdout:up {} 2024-04-20T21:47:55.513 INFO:teuthology.orchestra.run.smithi062.stdout:failed 0,1 2024-04-20T21:47:55.513 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T21:47:55.514 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T21:47:55.514 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T21:47:55.514 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T21:47:55.514 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data enabled 2024-04-20T21:47:55.514 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T21:47:55.514 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T21:47:55.514 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 1 2024-04-20T21:47:55.514 INFO:teuthology.orchestra.run.smithi062.stdout:qdb_cluster leader: 0 members: 2024-04-20T21:47:55.514 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:47:55.514 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:47:55.514 INFO:teuthology.orchestra.run.smithi062.stdout:Standby daemons: 2024-04-20T21:47:55.514 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:47:55.514 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ryqblb{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6826/4113916205,v1:172.21.15.62:6827/4113916205] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T21:47:55.514 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ozizwi{-1:34476} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6828/3801975165,v1:172.21.15.62:6829/3801975165] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T21:47:55.515 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.xbgspw{-1:44285} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6826/3996040526,v1:172.21.15.189:6827/3996040526] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:47:55.515 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.dvimer{-1:44289} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6824/1743111843,v1:172.21.15.189:6825/1743111843] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:47:55.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.515+0000 7f5198200700 1 -- 172.21.15.62:0/47554386 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f517806ecc0 msgr2=0x7f5178071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:47:55.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.515+0000 7f5198200700 1 --2- 172.21.15.62:0/47554386 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f517806ecc0 0x7f5178071170 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f517c006010 tx=0x7f517c005960 comp rx=0 tx=0).stop 2024-04-20T21:47:55.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.515+0000 7f5198200700 1 -- 172.21.15.62:0/47554386 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51901004e0 msgr2=0x7f5190195f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:47:55.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.515+0000 7f5198200700 1 --2- 172.21.15.62:0/47554386 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51901004e0 0x7f5190195f10 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f518400d8d0 tx=0x7f518400dc90 comp rx=0 tx=0).stop 2024-04-20T21:47:55.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.515+0000 7f5198200700 1 -- 172.21.15.62:0/47554386 shutdown_connections 2024-04-20T21:47:55.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.515+0000 7f5198200700 1 --2- 172.21.15.62:0/47554386 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f517806ecc0 0x7f5178071170 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:47:55.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.515+0000 7f5198200700 1 --2- 172.21.15.62:0/47554386 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51901004e0 0x7f5190195f10 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:47:55.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.515+0000 7f5198200700 1 --2- 172.21.15.62:0/47554386 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5190100e80 0x7f5190196450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:47:55.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.515+0000 7f5198200700 1 -- 172.21.15.62:0/47554386 >> 172.21.15.62:0/47554386 conn(0x7f51900fbda0 msgr2=0x7f51900fe180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:47:55.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.515+0000 7f5198200700 1 -- 172.21.15.62:0/47554386 shutdown_connections 2024-04-20T21:47:55.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.515+0000 7f5198200700 1 -- 172.21.15.62:0/47554386 wait complete. 2024-04-20T21:47:55.516 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 25 2024-04-20T21:47:55.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.667+0000 7f9af6600700 1 -- 172.21.15.62:0/38860595 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9af0073a00 msgr2=0x7f9af0110b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:47:55.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.667+0000 7f9af6600700 1 --2- 172.21.15.62:0/38860595 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9af0073a00 0x7f9af0110b00 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f9adc009910 tx=0x7f9adc009c20 comp rx=0 tx=0).stop 2024-04-20T21:47:55.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.667+0000 7f9af6600700 1 -- 172.21.15.62:0/38860595 shutdown_connections 2024-04-20T21:47:55.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.667+0000 7f9af6600700 1 --2- 172.21.15.62:0/38860595 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9af0073a00 0x7f9af0110b00 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:47:55.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.667+0000 7f9af6600700 1 --2- 172.21.15.62:0/38860595 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9af00730f0 0x7f9af00734c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:47:55.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.668+0000 7f9af6600700 1 -- 172.21.15.62:0/38860595 >> 172.21.15.62:0/38860595 conn(0x7f9af00fbb10 msgr2=0x7f9af00fdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:47:55.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.668+0000 7f9af6600700 1 -- 172.21.15.62:0/38860595 shutdown_connections 2024-04-20T21:47:55.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.668+0000 7f9af6600700 1 -- 172.21.15.62:0/38860595 wait complete. 2024-04-20T21:47:55.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.669+0000 7f9af6600700 1 Processor -- start 2024-04-20T21:47:55.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.669+0000 7f9af6600700 1 -- start start 2024-04-20T21:47:55.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.669+0000 7f9af6600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9af00730f0 0x7f9af019c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:47:55.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.669+0000 7f9af6600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9af0073a00 0x7f9af019c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:47:55.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.669+0000 7f9af6600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9af019cdc0 con 0x7f9af00730f0 2024-04-20T21:47:55.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.669+0000 7f9af6600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9af01a0b50 con 0x7f9af0073a00 2024-04-20T21:47:55.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.670+0000 7f9af5200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9af00730f0 0x7f9af019c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:47:55.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.670+0000 7f9af5200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9af00730f0 0x7f9af019c1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37180/0 (socket says 172.21.15.62:37180) 2024-04-20T21:47:55.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.670+0000 7f9af5200700 1 -- 172.21.15.62:0/2403070607 learned_addr learned my addr 172.21.15.62:0/2403070607 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:47:55.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.670+0000 7f9ae7e00700 1 --2- 172.21.15.62:0/2403070607 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9af0073a00 0x7f9af019c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:47:55.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.670+0000 7f9af5200700 1 -- 172.21.15.62:0/2403070607 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9af0073a00 msgr2=0x7f9af019c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:47:55.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.670+0000 7f9af5200700 1 --2- 172.21.15.62:0/2403070607 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9af0073a00 0x7f9af019c6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:47:55.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.670+0000 7f9af5200700 1 -- 172.21.15.62:0/2403070607 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9adc0095f0 con 0x7f9af00730f0 2024-04-20T21:47:55.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.670+0000 7f9ae7e00700 1 --2- 172.21.15.62:0/2403070607 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9af0073a00 0x7f9af019c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T21:47:55.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.670+0000 7f9af5200700 1 --2- 172.21.15.62:0/2403070607 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9af00730f0 0x7f9af019c1a0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f9ae000c8a0 tx=0x7f9ae000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:47:55.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.671+0000 7f9aee000700 1 -- 172.21.15.62:0/2403070607 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9ae000cea0 con 0x7f9af00730f0 2024-04-20T21:47:55.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.671+0000 7f9aee000700 1 -- 172.21.15.62:0/2403070607 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9ae0004830 con 0x7f9af00730f0 2024-04-20T21:47:55.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.671+0000 7f9aee000700 1 -- 172.21.15.62:0/2403070607 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9ae00055b0 con 0x7f9af00730f0 2024-04-20T21:47:55.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.671+0000 7f9af6600700 1 -- 172.21.15.62:0/2403070607 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9af01a0e30 con 0x7f9af00730f0 2024-04-20T21:47:55.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.671+0000 7f9af6600700 1 -- 172.21.15.62:0/2403070607 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9af01a1330 con 0x7f9af00730f0 2024-04-20T21:47:55.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.672+0000 7f9af6600700 1 -- 172.21.15.62:0/2403070607 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9af010e280 con 0x7f9af00730f0 2024-04-20T21:47:55.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.673+0000 7f9aee000700 1 -- 172.21.15.62:0/2403070607 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f9ae00049a0 con 0x7f9af00730f0 2024-04-20T21:47:55.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.674+0000 7f9aee000700 1 --2- 172.21.15.62:0/2403070607 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9ad406ecc0 0x7f9ad4071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:47:55.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.675+0000 7f9aee000700 1 -- 172.21.15.62:0/2403070607 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f9ae008ebf0 con 0x7f9af00730f0 2024-04-20T21:47:55.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.675+0000 7f9ae7e00700 1 --2- 172.21.15.62:0/2403070607 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9ad406ecc0 0x7f9ad4071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:47:55.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.675+0000 7f9ae7e00700 1 --2- 172.21.15.62:0/2403070607 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9ad406ecc0 0x7f9ad4071170 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f9adc006010 tx=0x7f9adc00b540 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:47:55.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.679+0000 7f9aee000700 1 -- 172.21.15.62:0/2403070607 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f9ae0057900 con 0x7f9af00730f0 2024-04-20T21:47:55.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.931+0000 7f9af6600700 1 -- 172.21.15.62:0/2403070607 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9af01a1600 con 0x7f9ad406ecc0 2024-04-20T21:47:55.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.934+0000 7f9aee000700 1 -- 172.21.15.62:0/2403070607 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f9af01a1600 con 0x7f9ad406ecc0 2024-04-20T21:47:55.933 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:47:55.934 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T21:47:55.934 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T21:47:55.934 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-20T21:47:55.934 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T21:47:55.934 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr", 2024-04-20T21:47:55.934 INFO:teuthology.orchestra.run.smithi062.stdout: "crash", 2024-04-20T21:47:55.934 INFO:teuthology.orchestra.run.smithi062.stdout: "mon", 2024-04-20T21:47:55.934 INFO:teuthology.orchestra.run.smithi062.stdout: "osd" 2024-04-20T21:47:55.934 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T21:47:55.934 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "14/23 daemons upgraded", 2024-04-20T21:47:55.934 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading mds daemons", 2024-04-20T21:47:55.934 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T21:47:55.934 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:47:55.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.937+0000 7f9af6600700 1 -- 172.21.15.62:0/2403070607 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9ad406ecc0 msgr2=0x7f9ad4071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:47:55.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.937+0000 7f9af6600700 1 --2- 172.21.15.62:0/2403070607 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9ad406ecc0 0x7f9ad4071170 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f9adc006010 tx=0x7f9adc00b540 comp rx=0 tx=0).stop 2024-04-20T21:47:55.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.937+0000 7f9af6600700 1 -- 172.21.15.62:0/2403070607 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9af00730f0 msgr2=0x7f9af019c1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:47:55.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.937+0000 7f9af6600700 1 --2- 172.21.15.62:0/2403070607 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9af00730f0 0x7f9af019c1a0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f9ae000c8a0 tx=0x7f9ae000cc60 comp rx=0 tx=0).stop 2024-04-20T21:47:55.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.938+0000 7f9af6600700 1 -- 172.21.15.62:0/2403070607 shutdown_connections 2024-04-20T21:47:55.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.938+0000 7f9af6600700 1 --2- 172.21.15.62:0/2403070607 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9ad406ecc0 0x7f9ad4071170 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:47:55.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.938+0000 7f9af6600700 1 --2- 172.21.15.62:0/2403070607 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9af00730f0 0x7f9af019c1a0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:47:55.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.938+0000 7f9af6600700 1 --2- 172.21.15.62:0/2403070607 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9af0073a00 0x7f9af019c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:47:55.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.938+0000 7f9af6600700 1 -- 172.21.15.62:0/2403070607 >> 172.21.15.62:0/2403070607 conn(0x7f9af00fbb10 msgr2=0x7f9af0102610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:47:55.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.938+0000 7f9af6600700 1 -- 172.21.15.62:0/2403070607 shutdown_connections 2024-04-20T21:47:55.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:55.938+0000 7f9af6600700 1 -- 172.21.15.62:0/2403070607 wait complete. 2024-04-20T21:47:56.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:56.098+0000 7fb7aec00700 1 -- 172.21.15.62:0/4189959424 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb7a8106b80 msgr2=0x7fb7a8106ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:47:56.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:56.098+0000 7fb7aec00700 1 --2- 172.21.15.62:0/4189959424 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb7a8106b80 0x7fb7a8106ff0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fb794009910 tx=0x7fb794009c20 comp rx=0 tx=0).stop 2024-04-20T21:47:56.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:56.098+0000 7fb7aec00700 1 -- 172.21.15.62:0/4189959424 shutdown_connections 2024-04-20T21:47:56.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:56.098+0000 7fb7aec00700 1 --2- 172.21.15.62:0/4189959424 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb7a8106b80 0x7fb7a8106ff0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:47:56.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:56.098+0000 7fb7aec00700 1 --2- 172.21.15.62:0/4189959424 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7a810c8f0 0x7fb7a810ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:47:56.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:56.098+0000 7fb7aec00700 1 -- 172.21.15.62:0/4189959424 >> 172.21.15.62:0/4189959424 conn(0x7fb7a8076060 msgr2=0x7fb7a8078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:47:56.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:56.098+0000 7fb7aec00700 1 -- 172.21.15.62:0/4189959424 shutdown_connections 2024-04-20T21:47:56.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:56.098+0000 7fb7aec00700 1 -- 172.21.15.62:0/4189959424 wait complete. 2024-04-20T21:47:56.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:56.099+0000 7fb7aec00700 1 Processor -- start 2024-04-20T21:47:56.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:56.099+0000 7fb7aec00700 1 -- start start 2024-04-20T21:47:56.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:56.100+0000 7fb7aec00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb7a8106b80 0x7fb7a819c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:47:56.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:56.100+0000 7fb7aec00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7a810c8f0 0x7fb7a819c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:47:56.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:56.100+0000 7fb7aec00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7a819cfd0 con 0x7fb7a8106b80 2024-04-20T21:47:56.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:56.100+0000 7fb7aec00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7a819d110 con 0x7fb7a810c8f0 2024-04-20T21:47:56.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:56.100+0000 7fb7a7e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb7a8106b80 0x7fb7a819c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:47:56.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:56.100+0000 7fb7a7e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb7a8106b80 0x7fb7a819c440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37188/0 (socket says 172.21.15.62:37188) 2024-04-20T21:47:56.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:56.100+0000 7fb7a7e00700 1 -- 172.21.15.62:0/2671420906 learned_addr learned my addr 172.21.15.62:0/2671420906 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:47:56.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:56.100+0000 7fb7a7400700 1 --2- 172.21.15.62:0/2671420906 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7a810c8f0 0x7fb7a819c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:47:56.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:56.101+0000 7fb7a7e00700 1 -- 172.21.15.62:0/2671420906 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7a810c8f0 msgr2=0x7fb7a819c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:47:56.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:56.101+0000 7fb7a7e00700 1 --2- 172.21.15.62:0/2671420906 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7a810c8f0 0x7fb7a819c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:47:56.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:56.101+0000 7fb7a7e00700 1 -- 172.21.15.62:0/2671420906 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7940095f0 con 0x7fb7a8106b80 2024-04-20T21:47:56.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:56.101+0000 7fb7a7400700 1 --2- 172.21.15.62:0/2671420906 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7a810c8f0 0x7fb7a819c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T21:47:56.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:56.101+0000 7fb7a7e00700 1 --2- 172.21.15.62:0/2671420906 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb7a8106b80 0x7fb7a819c440 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fb79800c930 tx=0x7fb79800ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:47:56.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:56.101+0000 7fb7a4c00700 1 -- 172.21.15.62:0/2671420906 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb798007ad0 con 0x7fb7a8106b80 2024-04-20T21:47:56.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:56.101+0000 7fb7aec00700 1 -- 172.21.15.62:0/2671420906 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7a81a0f60 con 0x7fb7a8106b80 2024-04-20T21:47:56.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:56.101+0000 7fb7a4c00700 1 -- 172.21.15.62:0/2671420906 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb798007c30 con 0x7fb7a8106b80 2024-04-20T21:47:56.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:56.101+0000 7fb7a4c00700 1 -- 172.21.15.62:0/2671420906 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb798018610 con 0x7fb7a8106b80 2024-04-20T21:47:56.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:56.101+0000 7fb7aec00700 1 -- 172.21.15.62:0/2671420906 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7a81a14b0 con 0x7fb7a8106b80 2024-04-20T21:47:56.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:56.102+0000 7fb7aec00700 1 -- 172.21.15.62:0/2671420906 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7a810ecc0 con 0x7fb7a8106b80 2024-04-20T21:47:56.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:56.104+0000 7fb7a4c00700 1 -- 172.21.15.62:0/2671420906 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fb79800f450 con 0x7fb7a8106b80 2024-04-20T21:47:56.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:56.104+0000 7fb7a4c00700 1 --2- 172.21.15.62:0/2671420906 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb78c06ecc0 0x7fb78c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:47:56.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:56.105+0000 7fb7a4c00700 1 -- 172.21.15.62:0/2671420906 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fb79808eff0 con 0x7fb7a8106b80 2024-04-20T21:47:56.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:56.105+0000 7fb7a7400700 1 --2- 172.21.15.62:0/2671420906 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb78c06ecc0 0x7fb78c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:47:56.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:56.105+0000 7fb7a7400700 1 --2- 172.21.15.62:0/2671420906 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb78c06ecc0 0x7fb78c071170 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fb79400b5c0 tx=0x7fb794005960 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:47:56.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:56.109+0000 7fb7a4c00700 1 -- 172.21.15.62:0/2671420906 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fb798057de0 con 0x7fb7a8106b80 2024-04-20T21:47:56.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:47:55 smithi189 ceph-mon[57022]: from='client.34736 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:47:56.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:47:55 smithi189 ceph-mon[57022]: from='client.34740 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:47:56.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:47:55 smithi189 ceph-mon[57022]: pgmap v378: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:47:56.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:47:55 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/3784455701' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:47:56.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:47:55 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/47554386' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T21:47:56.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:47:55 smithi062 ceph-mon[76322]: from='client.34736 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:47:56.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:47:55 smithi062 ceph-mon[76322]: from='client.34740 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:47:56.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:47:55 smithi062 ceph-mon[76322]: pgmap v378: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:47:56.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:47:55 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/3784455701' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:47:56.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:47:55 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/47554386' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T21:47:56.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:56.434+0000 7fb7aec00700 1 -- 172.21.15.62:0/2671420906 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fb7a804f2a0 con 0x7fb7a8106b80 2024-04-20T21:47:56.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:56.435+0000 7fb7a4c00700 1 -- 172.21.15.62:0/2671420906 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7fb79805b4a0 con 0x7fb7a8106b80 2024-04-20T21:47:56.434 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-20T21:47:56.434 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-20T21:47:56.434 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is degraded 2024-04-20T21:47:56.434 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T21:47:56.434 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T21:47:56.434 INFO:teuthology.orchestra.run.smithi062.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-20T21:47:56.434 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-20T21:47:56.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:56.438+0000 7fb7aec00700 1 -- 172.21.15.62:0/2671420906 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb78c06ecc0 msgr2=0x7fb78c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:47:56.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:56.438+0000 7fb7aec00700 1 --2- 172.21.15.62:0/2671420906 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb78c06ecc0 0x7fb78c071170 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fb79400b5c0 tx=0x7fb794005960 comp rx=0 tx=0).stop 2024-04-20T21:47:56.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:56.438+0000 7fb7aec00700 1 -- 172.21.15.62:0/2671420906 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb7a8106b80 msgr2=0x7fb7a819c440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:47:56.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:56.438+0000 7fb7aec00700 1 --2- 172.21.15.62:0/2671420906 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb7a8106b80 0x7fb7a819c440 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fb79800c930 tx=0x7fb79800ccf0 comp rx=0 tx=0).stop 2024-04-20T21:47:56.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:56.438+0000 7fb7aec00700 1 -- 172.21.15.62:0/2671420906 shutdown_connections 2024-04-20T21:47:56.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:56.438+0000 7fb7aec00700 1 --2- 172.21.15.62:0/2671420906 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb78c06ecc0 0x7fb78c071170 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:47:56.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:56.438+0000 7fb7aec00700 1 --2- 172.21.15.62:0/2671420906 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb7a8106b80 0x7fb7a819c440 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:47:56.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:56.438+0000 7fb7aec00700 1 --2- 172.21.15.62:0/2671420906 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7a810c8f0 0x7fb7a819c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:47:56.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:56.438+0000 7fb7aec00700 1 -- 172.21.15.62:0/2671420906 >> 172.21.15.62:0/2671420906 conn(0x7fb7a8076060 msgr2=0x7fb7a8077ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:47:56.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:56.438+0000 7fb7aec00700 1 -- 172.21.15.62:0/2671420906 shutdown_connections 2024-04-20T21:47:56.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:47:56.438+0000 7fb7aec00700 1 -- 172.21.15.62:0/2671420906 wait complete. 2024-04-20T21:47:57.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:47:56 smithi189 ceph-mon[57022]: from='client.34752 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:47:57.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:47:56 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/2671420906' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T21:47:57.284 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:47:56 smithi062 ceph-mon[76322]: from='client.34752 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:47:57.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:47:56 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/2671420906' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T21:47:58.140 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:47:57 smithi189 ceph-mon[57022]: pgmap v379: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:47:58.142 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:47:57 smithi062 ceph-mon[76322]: pgmap v379: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:48:00.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:47:59 smithi189 ceph-mon[57022]: pgmap v380: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:48:00.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:47:59 smithi062 ceph-mon[76322]: pgmap v380: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:48:02.184 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:48:01 smithi189 ceph-mon[57022]: pgmap v381: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:48:02.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:48:01 smithi062 ceph-mon[76322]: pgmap v381: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:48:04.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:48:03 smithi189 ceph-mon[57022]: pgmap v382: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:48:04.284 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:48:03 smithi062 ceph-mon[76322]: pgmap v382: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:48:05.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:48:04 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:48:05.287 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:48:04 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:48:06.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:48:05 smithi189 ceph-mon[57022]: pgmap v383: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:48:06.284 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:48:05 smithi062 ceph-mon[76322]: pgmap v383: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:48:08.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:48:07 smithi189 ceph-mon[57022]: pgmap v384: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:48:08.284 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:48:07 smithi062 ceph-mon[76322]: pgmap v384: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:48:10.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:48:09 smithi062 ceph-mon[76322]: pgmap v385: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:48:10.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:48:09 smithi189 ceph-mon[57022]: pgmap v385: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:48:12.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:48:11 smithi189 ceph-mon[57022]: pgmap v386: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:48:12.284 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:48:11 smithi062 ceph-mon[76322]: pgmap v386: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:48:14.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:48:13 smithi189 ceph-mon[57022]: pgmap v387: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:48:14.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:48:13 smithi062 ceph-mon[76322]: pgmap v387: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:48:16.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:48:15 smithi189 ceph-mon[57022]: pgmap v388: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:48:16.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:48:15 smithi062 ceph-mon[76322]: pgmap v388: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:48:18.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:48:17 smithi189 ceph-mon[57022]: pgmap v389: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:48:18.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:48:17 smithi062 ceph-mon[76322]: pgmap v389: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:48:20.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:48:19 smithi189 ceph-mon[57022]: pgmap v390: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:48:20.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:48:19 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:48:20.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:48:19 smithi062 ceph-mon[76322]: pgmap v390: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:48:20.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:48:19 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:48:22.236 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:48:21 smithi189 ceph-mon[57022]: pgmap v391: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:48:22.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:48:21 smithi062 ceph-mon[76322]: pgmap v391: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:48:24.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:48:23 smithi189 ceph-mon[57022]: pgmap v392: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:48:24.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:48:23 smithi062 ceph-mon[76322]: pgmap v392: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:48:26.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:48:25 smithi189 ceph-mon[57022]: pgmap v393: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:48:26.284 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:48:25 smithi062 ceph-mon[76322]: pgmap v393: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:48:26.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:26.616+0000 7f4f94200700 1 -- 172.21.15.62:0/3432328038 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f8c106b80 msgr2=0x7f4f8c106ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:48:26.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:26.616+0000 7f4f94200700 1 --2- 172.21.15.62:0/3432328038 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f8c106b80 0x7f4f8c106ff0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f4f78009910 tx=0x7f4f78009c20 comp rx=0 tx=0).stop 2024-04-20T21:48:26.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:26.616+0000 7f4f94200700 1 -- 172.21.15.62:0/3432328038 shutdown_connections 2024-04-20T21:48:26.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:26.616+0000 7f4f94200700 1 --2- 172.21.15.62:0/3432328038 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f8c106b80 0x7f4f8c106ff0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:48:26.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:26.616+0000 7f4f94200700 1 --2- 172.21.15.62:0/3432328038 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f8c10c8f0 0x7f4f8c10ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:48:26.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:26.616+0000 7f4f94200700 1 -- 172.21.15.62:0/3432328038 >> 172.21.15.62:0/3432328038 conn(0x7f4f8c076060 msgr2=0x7f4f8c078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:48:26.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:26.617+0000 7f4f94200700 1 -- 172.21.15.62:0/3432328038 shutdown_connections 2024-04-20T21:48:26.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:26.617+0000 7f4f94200700 1 -- 172.21.15.62:0/3432328038 wait complete. 2024-04-20T21:48:26.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:26.618+0000 7f4f94200700 1 Processor -- start 2024-04-20T21:48:26.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:26.618+0000 7f4f94200700 1 -- start start 2024-04-20T21:48:26.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:26.618+0000 7f4f94200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f8c10c8f0 0x7f4f8c19c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:48:26.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:26.618+0000 7f4f94200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f8c19cb90 0x7f4f8c1a1000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:48:26.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:26.618+0000 7f4f94200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f8c19d1a0 con 0x7f4f8c19cb90 2024-04-20T21:48:26.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:26.618+0000 7f4f94200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f8c19d310 con 0x7f4f8c10c8f0 2024-04-20T21:48:26.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:26.619+0000 7f4f90e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f8c19cb90 0x7f4f8c1a1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:48:26.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:26.619+0000 7f4f90e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f8c19cb90 0x7f4f8c1a1000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35180/0 (socket says 172.21.15.62:35180) 2024-04-20T21:48:26.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:26.619+0000 7f4f90e00700 1 -- 172.21.15.62:0/3217798770 learned_addr learned my addr 172.21.15.62:0/3217798770 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:48:26.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:26.619+0000 7f4f91800700 1 --2- 172.21.15.62:0/3217798770 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f8c10c8f0 0x7f4f8c19c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:48:26.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:26.619+0000 7f4f90e00700 1 -- 172.21.15.62:0/3217798770 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f8c10c8f0 msgr2=0x7f4f8c19c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:48:26.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:26.619+0000 7f4f90e00700 1 --2- 172.21.15.62:0/3217798770 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f8c10c8f0 0x7f4f8c19c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:48:26.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:26.619+0000 7f4f90e00700 1 -- 172.21.15.62:0/3217798770 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f780095f0 con 0x7f4f8c19cb90 2024-04-20T21:48:26.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:26.619+0000 7f4f91800700 1 --2- 172.21.15.62:0/3217798770 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f8c10c8f0 0x7f4f8c19c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-20T21:48:26.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:26.620+0000 7f4f90e00700 1 --2- 172.21.15.62:0/3217798770 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f8c19cb90 0x7f4f8c1a1000 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f4f7800bd70 tx=0x7f4f7800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:48:26.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:26.620+0000 7f4f8a000700 1 -- 172.21.15.62:0/3217798770 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4f7801d070 con 0x7f4f8c19cb90 2024-04-20T21:48:26.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:26.620+0000 7f4f8a000700 1 -- 172.21.15.62:0/3217798770 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4f78005dc0 con 0x7f4f8c19cb90 2024-04-20T21:48:26.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:26.620+0000 7f4f94200700 1 -- 172.21.15.62:0/3217798770 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f8c1a15a0 con 0x7f4f8c19cb90 2024-04-20T21:48:26.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:26.620+0000 7f4f8a000700 1 -- 172.21.15.62:0/3217798770 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4f78017400 con 0x7f4f8c19cb90 2024-04-20T21:48:26.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:26.620+0000 7f4f94200700 1 -- 172.21.15.62:0/3217798770 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f8c1a1af0 con 0x7f4f8c19cb90 2024-04-20T21:48:26.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:26.621+0000 7f4f94200700 1 -- 172.21.15.62:0/3217798770 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f8c109b20 con 0x7f4f8c19cb90 2024-04-20T21:48:26.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:26.623+0000 7f4f8a000700 1 -- 172.21.15.62:0/3217798770 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f4f7800f6d0 con 0x7f4f8c19cb90 2024-04-20T21:48:26.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:26.624+0000 7f4f8a000700 1 --2- 172.21.15.62:0/3217798770 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f4f7406ecc0 0x7f4f74071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:48:26.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:26.624+0000 7f4f8a000700 1 -- 172.21.15.62:0/3217798770 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f4f780902f0 con 0x7f4f8c19cb90 2024-04-20T21:48:26.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:26.624+0000 7f4f91800700 1 --2- 172.21.15.62:0/3217798770 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f4f7406ecc0 0x7f4f74071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:48:26.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:26.625+0000 7f4f91800700 1 --2- 172.21.15.62:0/3217798770 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f4f7406ecc0 0x7f4f74071170 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f4f80005950 tx=0x7f4f800058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:48:26.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:26.629+0000 7f4f8a000700 1 -- 172.21.15.62:0/3217798770 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f4f78059080 con 0x7f4f8c19cb90 2024-04-20T21:48:26.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:26.886+0000 7f4f94200700 1 -- 172.21.15.62:0/3217798770 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4f8c1a1dd0 con 0x7f4f7406ecc0 2024-04-20T21:48:26.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:26.889+0000 7f4f8a000700 1 -- 172.21.15.62:0/3217798770 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f4f8c1a1dd0 con 0x7f4f7406ecc0 2024-04-20T21:48:26.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:26.892+0000 7f4f94200700 1 -- 172.21.15.62:0/3217798770 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f4f7406ecc0 msgr2=0x7f4f74071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:48:26.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:26.892+0000 7f4f94200700 1 --2- 172.21.15.62:0/3217798770 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f4f7406ecc0 0x7f4f74071170 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f4f80005950 tx=0x7f4f800058e0 comp rx=0 tx=0).stop 2024-04-20T21:48:26.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:26.892+0000 7f4f94200700 1 -- 172.21.15.62:0/3217798770 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f8c19cb90 msgr2=0x7f4f8c1a1000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:48:26.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:26.892+0000 7f4f94200700 1 --2- 172.21.15.62:0/3217798770 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f8c19cb90 0x7f4f8c1a1000 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f4f7800bd70 tx=0x7f4f7800be50 comp rx=0 tx=0).stop 2024-04-20T21:48:26.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:26.892+0000 7f4f94200700 1 -- 172.21.15.62:0/3217798770 shutdown_connections 2024-04-20T21:48:26.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:26.892+0000 7f4f94200700 1 --2- 172.21.15.62:0/3217798770 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f4f7406ecc0 0x7f4f74071170 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:48:26.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:26.892+0000 7f4f94200700 1 --2- 172.21.15.62:0/3217798770 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f8c10c8f0 0x7f4f8c19c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:48:26.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:26.892+0000 7f4f94200700 1 --2- 172.21.15.62:0/3217798770 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f8c19cb90 0x7f4f8c1a1000 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:48:26.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:26.892+0000 7f4f94200700 1 -- 172.21.15.62:0/3217798770 >> 172.21.15.62:0/3217798770 conn(0x7f4f8c076060 msgr2=0x7f4f8c077c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:48:26.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:26.893+0000 7f4f94200700 1 -- 172.21.15.62:0/3217798770 shutdown_connections 2024-04-20T21:48:26.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:26.893+0000 7f4f94200700 1 -- 172.21.15.62:0/3217798770 wait complete. 2024-04-20T21:48:26.908 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T21:48:27.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.063+0000 7fd0f2800700 1 -- 172.21.15.62:0/1861491664 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd0ec108470 msgr2=0x7fd0ec108840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:48:27.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.063+0000 7fd0f2800700 1 --2- 172.21.15.62:0/1861491664 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd0ec108470 0x7fd0ec108840 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fd0dc00b0a0 tx=0x7fd0dc00b3b0 comp rx=0 tx=0).stop 2024-04-20T21:48:27.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.063+0000 7fd0f2800700 1 -- 172.21.15.62:0/1861491664 shutdown_connections 2024-04-20T21:48:27.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.063+0000 7fd0f2800700 1 --2- 172.21.15.62:0/1861491664 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd0ec102470 0x7fd0ec1028e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:48:27.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.063+0000 7fd0f2800700 1 --2- 172.21.15.62:0/1861491664 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd0ec108470 0x7fd0ec108840 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:48:27.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.063+0000 7fd0f2800700 1 -- 172.21.15.62:0/1861491664 >> 172.21.15.62:0/1861491664 conn(0x7fd0ec0fdf90 msgr2=0x7fd0ec1003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:48:27.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.063+0000 7fd0f2800700 1 -- 172.21.15.62:0/1861491664 shutdown_connections 2024-04-20T21:48:27.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.063+0000 7fd0f2800700 1 -- 172.21.15.62:0/1861491664 wait complete. 2024-04-20T21:48:27.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.064+0000 7fd0f2800700 1 Processor -- start 2024-04-20T21:48:27.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.064+0000 7fd0f2800700 1 -- start start 2024-04-20T21:48:27.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.065+0000 7fd0f2800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd0ec102470 0x7fd0ec198210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:48:27.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.065+0000 7fd0f2800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd0ec108470 0x7fd0ec198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:48:27.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.065+0000 7fd0f2800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0ec198e30 con 0x7fd0ec108470 2024-04-20T21:48:27.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.065+0000 7fd0f2800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0ec19cbc0 con 0x7fd0ec102470 2024-04-20T21:48:27.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.065+0000 7fd0eaa00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd0ec108470 0x7fd0ec198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:48:27.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.065+0000 7fd0eaa00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd0ec108470 0x7fd0ec198750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35194/0 (socket says 172.21.15.62:35194) 2024-04-20T21:48:27.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.065+0000 7fd0eaa00700 1 -- 172.21.15.62:0/2161349099 learned_addr learned my addr 172.21.15.62:0/2161349099 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:48:27.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.066+0000 7fd0eb400700 1 --2- 172.21.15.62:0/2161349099 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd0ec102470 0x7fd0ec198210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:48:27.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.066+0000 7fd0eaa00700 1 -- 172.21.15.62:0/2161349099 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd0ec102470 msgr2=0x7fd0ec198210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:48:27.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.066+0000 7fd0eaa00700 1 --2- 172.21.15.62:0/2161349099 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd0ec102470 0x7fd0ec198210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:48:27.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.066+0000 7fd0eaa00700 1 -- 172.21.15.62:0/2161349099 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0dc009d00 con 0x7fd0ec108470 2024-04-20T21:48:27.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.066+0000 7fd0eaa00700 1 --2- 172.21.15.62:0/2161349099 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd0ec108470 0x7fd0ec198750 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fd0d800b4b0 tx=0x7fd0d800b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:48:27.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.066+0000 7fd0e3e00700 1 -- 172.21.15.62:0/2161349099 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd0d8011800 con 0x7fd0ec108470 2024-04-20T21:48:27.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.066+0000 7fd0e3e00700 1 -- 172.21.15.62:0/2161349099 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd0d8011e20 con 0x7fd0ec108470 2024-04-20T21:48:27.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.066+0000 7fd0f2800700 1 -- 172.21.15.62:0/2161349099 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0ec19cea0 con 0x7fd0ec108470 2024-04-20T21:48:27.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.067+0000 7fd0e3e00700 1 -- 172.21.15.62:0/2161349099 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd0d800f2e0 con 0x7fd0ec108470 2024-04-20T21:48:27.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.067+0000 7fd0f2800700 1 -- 172.21.15.62:0/2161349099 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0ec19d3f0 con 0x7fd0ec108470 2024-04-20T21:48:27.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.068+0000 7fd0f2800700 1 -- 172.21.15.62:0/2161349099 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0ec10a9e0 con 0x7fd0ec108470 2024-04-20T21:48:27.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.069+0000 7fd0e3e00700 1 -- 172.21.15.62:0/2161349099 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fd0d8010380 con 0x7fd0ec108470 2024-04-20T21:48:27.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.070+0000 7fd0e3e00700 1 --2- 172.21.15.62:0/2161349099 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd0d006ecc0 0x7fd0d0071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:48:27.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.070+0000 7fd0e3e00700 1 -- 172.21.15.62:0/2161349099 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fd0d808ea30 con 0x7fd0ec108470 2024-04-20T21:48:27.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.070+0000 7fd0eb400700 1 --2- 172.21.15.62:0/2161349099 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd0d006ecc0 0x7fd0d0071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:48:27.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.071+0000 7fd0eb400700 1 --2- 172.21.15.62:0/2161349099 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd0d006ecc0 0x7fd0d0071170 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fd0dc009480 tx=0x7fd0dc00b7f0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:48:27.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.075+0000 7fd0e3e00700 1 -- 172.21.15.62:0/2161349099 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fd0d8057870 con 0x7fd0ec108470 2024-04-20T21:48:27.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.333+0000 7fd0f2800700 1 -- 172.21.15.62:0/2161349099 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd0ec19d6c0 con 0x7fd0d006ecc0 2024-04-20T21:48:27.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.336+0000 7fd0e3e00700 1 -- 172.21.15.62:0/2161349099 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fd0ec19d6c0 con 0x7fd0d006ecc0 2024-04-20T21:48:27.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.338+0000 7fd0f2800700 1 -- 172.21.15.62:0/2161349099 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd0d006ecc0 msgr2=0x7fd0d0071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:48:27.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.338+0000 7fd0f2800700 1 --2- 172.21.15.62:0/2161349099 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd0d006ecc0 0x7fd0d0071170 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fd0dc009480 tx=0x7fd0dc00b7f0 comp rx=0 tx=0).stop 2024-04-20T21:48:27.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.338+0000 7fd0f2800700 1 -- 172.21.15.62:0/2161349099 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd0ec108470 msgr2=0x7fd0ec198750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:48:27.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.338+0000 7fd0f2800700 1 --2- 172.21.15.62:0/2161349099 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd0ec108470 0x7fd0ec198750 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fd0d800b4b0 tx=0x7fd0d800b870 comp rx=0 tx=0).stop 2024-04-20T21:48:27.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.338+0000 7fd0f2800700 1 -- 172.21.15.62:0/2161349099 shutdown_connections 2024-04-20T21:48:27.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.338+0000 7fd0f2800700 1 --2- 172.21.15.62:0/2161349099 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd0d006ecc0 0x7fd0d0071170 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:48:27.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.338+0000 7fd0f2800700 1 --2- 172.21.15.62:0/2161349099 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd0ec102470 0x7fd0ec198210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:48:27.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.338+0000 7fd0f2800700 1 --2- 172.21.15.62:0/2161349099 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd0ec108470 0x7fd0ec198750 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:48:27.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.338+0000 7fd0f2800700 1 -- 172.21.15.62:0/2161349099 >> 172.21.15.62:0/2161349099 conn(0x7fd0ec0fdf90 msgr2=0x7fd0ec0feb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:48:27.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.339+0000 7fd0f2800700 1 -- 172.21.15.62:0/2161349099 shutdown_connections 2024-04-20T21:48:27.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.339+0000 7fd0f2800700 1 -- 172.21.15.62:0/2161349099 wait complete. 2024-04-20T21:48:27.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.491+0000 7ff349600700 1 -- 172.21.15.62:0/4265812050 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff3440730f0 msgr2=0x7ff3440734c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:48:27.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.491+0000 7ff349600700 1 --2- 172.21.15.62:0/4265812050 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff3440730f0 0x7ff3440734c0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7ff334009910 tx=0x7ff334009c20 comp rx=0 tx=0).stop 2024-04-20T21:48:27.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.492+0000 7ff349600700 1 -- 172.21.15.62:0/4265812050 shutdown_connections 2024-04-20T21:48:27.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.492+0000 7ff349600700 1 --2- 172.21.15.62:0/4265812050 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff344073a00 0x7ff34410c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:48:27.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.492+0000 7ff349600700 1 --2- 172.21.15.62:0/4265812050 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff3440730f0 0x7ff3440734c0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:48:27.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.492+0000 7ff349600700 1 -- 172.21.15.62:0/4265812050 >> 172.21.15.62:0/4265812050 conn(0x7ff3440fbb10 msgr2=0x7ff3440fdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:48:27.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.492+0000 7ff349600700 1 -- 172.21.15.62:0/4265812050 shutdown_connections 2024-04-20T21:48:27.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.492+0000 7ff349600700 1 -- 172.21.15.62:0/4265812050 wait complete. 2024-04-20T21:48:27.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.493+0000 7ff349600700 1 Processor -- start 2024-04-20T21:48:27.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.493+0000 7ff349600700 1 -- start start 2024-04-20T21:48:27.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.493+0000 7ff349600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3440730f0 0x7ff344197e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:48:27.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.493+0000 7ff349600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff344073a00 0x7ff344198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:48:27.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.493+0000 7ff349600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff34419c780 con 0x7ff344073a00 2024-04-20T21:48:27.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.493+0000 7ff349600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff34419c8f0 con 0x7ff3440730f0 2024-04-20T21:48:27.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.494+0000 7ff33b400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff344073a00 0x7ff344198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:48:27.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.494+0000 7ff33b400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff344073a00 0x7ff344198490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35204/0 (socket says 172.21.15.62:35204) 2024-04-20T21:48:27.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.494+0000 7ff33b400700 1 -- 172.21.15.62:0/278368338 learned_addr learned my addr 172.21.15.62:0/278368338 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:48:27.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.494+0000 7ff343e00700 1 --2- 172.21.15.62:0/278368338 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3440730f0 0x7ff344197e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:48:27.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.494+0000 7ff33b400700 1 -- 172.21.15.62:0/278368338 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3440730f0 msgr2=0x7ff344197e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:48:27.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.494+0000 7ff33b400700 1 --2- 172.21.15.62:0/278368338 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3440730f0 0x7ff344197e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:48:27.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.494+0000 7ff33b400700 1 -- 172.21.15.62:0/278368338 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3340095f0 con 0x7ff344073a00 2024-04-20T21:48:27.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.494+0000 7ff343e00700 1 --2- 172.21.15.62:0/278368338 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3440730f0 0x7ff344197e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-20T21:48:27.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.495+0000 7ff33b400700 1 --2- 172.21.15.62:0/278368338 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff344073a00 0x7ff344198490 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7ff32c00b4b0 tx=0x7ff32c00b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:48:27.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.495+0000 7ff341600700 1 -- 172.21.15.62:0/278368338 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff32c011800 con 0x7ff344073a00 2024-04-20T21:48:27.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.495+0000 7ff341600700 1 -- 172.21.15.62:0/278368338 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff32c011e20 con 0x7ff344073a00 2024-04-20T21:48:27.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.495+0000 7ff349600700 1 -- 172.21.15.62:0/278368338 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff34419cbd0 con 0x7ff344073a00 2024-04-20T21:48:27.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.495+0000 7ff341600700 1 -- 172.21.15.62:0/278368338 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff32c00f2e0 con 0x7ff344073a00 2024-04-20T21:48:27.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.495+0000 7ff349600700 1 -- 172.21.15.62:0/278368338 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff34419d120 con 0x7ff344073a00 2024-04-20T21:48:27.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.496+0000 7ff349600700 1 -- 172.21.15.62:0/278368338 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff344109a30 con 0x7ff344073a00 2024-04-20T21:48:27.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.497+0000 7ff341600700 1 -- 172.21.15.62:0/278368338 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff32c010380 con 0x7ff344073a00 2024-04-20T21:48:27.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.498+0000 7ff341600700 1 --2- 172.21.15.62:0/278368338 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff32806ecc0 0x7ff328071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:48:27.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.498+0000 7ff341600700 1 -- 172.21.15.62:0/278368338 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7ff32c08e620 con 0x7ff344073a00 2024-04-20T21:48:27.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.498+0000 7ff343e00700 1 --2- 172.21.15.62:0/278368338 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff32806ecc0 0x7ff328071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:48:27.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.499+0000 7ff343e00700 1 --2- 172.21.15.62:0/278368338 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff32806ecc0 0x7ff328071170 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7ff33400b5c0 tx=0x7ff33401a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:48:27.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.503+0000 7ff341600700 1 -- 172.21.15.62:0/278368338 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7ff32c057450 con 0x7ff344073a00 2024-04-20T21:48:27.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.737+0000 7ff349600700 1 -- 172.21.15.62:0/278368338 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7ff3441991f0 con 0x7ff32806ecc0 2024-04-20T21:48:27.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.758+0000 7ff341600700 1 -- 172.21.15.62:0/278368338 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7ff3441991f0 con 0x7ff32806ecc0 2024-04-20T21:48:27.758 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T21:48:27.758 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (14m) 5m ago 28m 14.2M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T21:48:27.758 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (28m) 5m ago 28m 9512k - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T21:48:27.759 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (27m) 6m ago 27m 8942k - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T21:48:27.759 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (10m) 5m ago 28m 6660k - 19.0.0-3133-ga8c84f59 76bc6b91b726 2825d5ba11c6 2024-04-20T21:48:27.759 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (10m) 6m ago 27m 6665k - 19.0.0-3133-ga8c84f59 76bc6b91b726 20f746fa1bba 2024-04-20T21:48:27.759 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (13m) 5m ago 27m 74.2M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T21:48:27.759 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (5m) 5m ago 20m 11.2M - 19.0.0-3133-ga8c84f59 76bc6b91b726 de80bc90a683 2024-04-20T21:48:27.759 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (6m) 5m ago 21m 11.9M - 19.0.0-3133-ga8c84f59 76bc6b91b726 cd793b7ee98e 2024-04-20T21:48:27.759 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (20m) 6m ago 20m 117M - 18.2.0 dc2bc1663786 fb2a8a223db8 2024-04-20T21:48:27.759 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (20m) 6m ago 20m 55.6M - 18.2.0 dc2bc1663786 f2eec2ee8951 2024-04-20T21:48:27.759 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (17m) 5m ago 29m 511M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T21:48:27.759 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (15m) 6m ago 26m 452M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T21:48:27.759 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (12m) 5m ago 30m 55.9M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 019eb1479485 2024-04-20T21:48:27.759 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (11m) 6m ago 26m 39.8M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 55aac560aab6 2024-04-20T21:48:27.759 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (15m) 5m ago 28m 7952k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T21:48:27.759 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (15m) 6m ago 26m 8304k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T21:48:27.760 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (9m) 5m ago 25m 88.0M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 ee2bb901f26e 2024-04-20T21:48:27.760 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (9m) 5m ago 25m 92.3M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 9075152d4162 2024-04-20T21:48:27.760 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (8m) 5m ago 24m 74.4M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 2d4f3e38dc0e 2024-04-20T21:48:27.760 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (8m) 6m ago 23m 87.6M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 8f97b17ae0eb 2024-04-20T21:48:27.760 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (7m) 6m ago 23m 90.7M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 653d8079967d 2024-04-20T21:48:27.760 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (6m) 6m ago 22m 11.5M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 596dded14c12 2024-04-20T21:48:27.760 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (14m) 5m ago 27m 74.0M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T21:48:27.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.761+0000 7ff349600700 1 -- 172.21.15.62:0/278368338 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff32806ecc0 msgr2=0x7ff328071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:48:27.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.761+0000 7ff349600700 1 --2- 172.21.15.62:0/278368338 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff32806ecc0 0x7ff328071170 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7ff33400b5c0 tx=0x7ff33401a040 comp rx=0 tx=0).stop 2024-04-20T21:48:27.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.761+0000 7ff349600700 1 -- 172.21.15.62:0/278368338 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff344073a00 msgr2=0x7ff344198490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:48:27.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.761+0000 7ff349600700 1 --2- 172.21.15.62:0/278368338 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff344073a00 0x7ff344198490 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7ff32c00b4b0 tx=0x7ff32c00b870 comp rx=0 tx=0).stop 2024-04-20T21:48:27.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.761+0000 7ff349600700 1 -- 172.21.15.62:0/278368338 shutdown_connections 2024-04-20T21:48:27.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.761+0000 7ff349600700 1 --2- 172.21.15.62:0/278368338 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff32806ecc0 0x7ff328071170 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:48:27.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.761+0000 7ff349600700 1 --2- 172.21.15.62:0/278368338 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3440730f0 0x7ff344197e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:48:27.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.761+0000 7ff349600700 1 --2- 172.21.15.62:0/278368338 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff344073a00 0x7ff344198490 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:48:27.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.761+0000 7ff349600700 1 -- 172.21.15.62:0/278368338 >> 172.21.15.62:0/278368338 conn(0x7ff3440fbb10 msgr2=0x7ff344106b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:48:27.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.762+0000 7ff349600700 1 -- 172.21.15.62:0/278368338 shutdown_connections 2024-04-20T21:48:27.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.762+0000 7ff349600700 1 -- 172.21.15.62:0/278368338 wait complete. 2024-04-20T21:48:27.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.910+0000 7f2f6d200700 1 -- 172.21.15.62:0/2163245168 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f68100e80 msgr2=0x7f2f68104e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:48:27.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.910+0000 7f2f6d200700 1 --2- 172.21.15.62:0/2163245168 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f68100e80 0x7f2f68104e50 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f2f54009910 tx=0x7f2f54009c20 comp rx=0 tx=0).stop 2024-04-20T21:48:27.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.910+0000 7f2f6d200700 1 -- 172.21.15.62:0/2163245168 shutdown_connections 2024-04-20T21:48:27.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.910+0000 7f2f6d200700 1 --2- 172.21.15.62:0/2163245168 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f68100e80 0x7f2f68104e50 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:48:27.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.910+0000 7f2f6d200700 1 --2- 172.21.15.62:0/2163245168 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f681004e0 0x7f2f681008b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:48:27.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.910+0000 7f2f6d200700 1 -- 172.21.15.62:0/2163245168 >> 172.21.15.62:0/2163245168 conn(0x7f2f680fbda0 msgr2=0x7f2f680fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:48:27.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.911+0000 7f2f6d200700 1 -- 172.21.15.62:0/2163245168 shutdown_connections 2024-04-20T21:48:27.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.911+0000 7f2f6d200700 1 -- 172.21.15.62:0/2163245168 wait complete. 2024-04-20T21:48:27.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.912+0000 7f2f6d200700 1 Processor -- start 2024-04-20T21:48:27.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.912+0000 7f2f6d200700 1 -- start start 2024-04-20T21:48:27.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.912+0000 7f2f6d200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f681004e0 0x7f2f68195f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:48:27.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.912+0000 7f2f6d200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f68100e80 0x7f2f68196450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:48:27.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.912+0000 7f2f6d200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f68196b30 con 0x7f2f681004e0 2024-04-20T21:48:27.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.912+0000 7f2f6d200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f6819a8c0 con 0x7f2f68100e80 2024-04-20T21:48:27.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.912+0000 7f2f65e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f681004e0 0x7f2f68195f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:48:27.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.913+0000 7f2f65e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f681004e0 0x7f2f68195f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35222/0 (socket says 172.21.15.62:35222) 2024-04-20T21:48:27.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.913+0000 7f2f65e00700 1 -- 172.21.15.62:0/1807816672 learned_addr learned my addr 172.21.15.62:0/1807816672 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:48:27.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.913+0000 7f2f65400700 1 --2- 172.21.15.62:0/1807816672 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f68100e80 0x7f2f68196450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:48:27.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.913+0000 7f2f65e00700 1 -- 172.21.15.62:0/1807816672 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f68100e80 msgr2=0x7f2f68196450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:48:27.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.913+0000 7f2f65e00700 1 --2- 172.21.15.62:0/1807816672 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f68100e80 0x7f2f68196450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:48:27.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.913+0000 7f2f65e00700 1 -- 172.21.15.62:0/1807816672 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f540095f0 con 0x7f2f681004e0 2024-04-20T21:48:27.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.913+0000 7f2f65e00700 1 --2- 172.21.15.62:0/1807816672 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f681004e0 0x7f2f68195f10 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f2f5800d8d0 tx=0x7f2f5800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:48:27.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.914+0000 7f2f5ea00700 1 -- 172.21.15.62:0/1807816672 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2f58009940 con 0x7f2f681004e0 2024-04-20T21:48:27.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.914+0000 7f2f5ea00700 1 -- 172.21.15.62:0/1807816672 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2f58010460 con 0x7f2f681004e0 2024-04-20T21:48:27.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.914+0000 7f2f5ea00700 1 -- 172.21.15.62:0/1807816672 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2f5800f5f0 con 0x7f2f681004e0 2024-04-20T21:48:27.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.914+0000 7f2f6d200700 1 -- 172.21.15.62:0/1807816672 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f6819aba0 con 0x7f2f681004e0 2024-04-20T21:48:27.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.914+0000 7f2f6d200700 1 -- 172.21.15.62:0/1807816672 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f6819b0f0 con 0x7f2f681004e0 2024-04-20T21:48:27.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.915+0000 7f2f6d200700 1 -- 172.21.15.62:0/1807816672 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f68108850 con 0x7f2f681004e0 2024-04-20T21:48:27.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.916+0000 7f2f5ea00700 1 -- 172.21.15.62:0/1807816672 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f2f580105d0 con 0x7f2f681004e0 2024-04-20T21:48:27.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.917+0000 7f2f5ea00700 1 --2- 172.21.15.62:0/1807816672 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2f4c06ecc0 0x7f2f4c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:48:27.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.917+0000 7f2f5ea00700 1 -- 172.21.15.62:0/1807816672 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f2f5808ef20 con 0x7f2f681004e0 2024-04-20T21:48:27.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.917+0000 7f2f65400700 1 --2- 172.21.15.62:0/1807816672 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2f4c06ecc0 0x7f2f4c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:48:27.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.918+0000 7f2f65400700 1 --2- 172.21.15.62:0/1807816672 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2f4c06ecc0 0x7f2f4c071170 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f2f68197530 tx=0x7f2f5401a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:48:27.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:27.922+0000 7f2f5ea00700 1 -- 172.21.15.62:0/1807816672 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f2f58057c30 con 0x7f2f681004e0 2024-04-20T21:48:28.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:48:27 smithi189 ceph-mon[57022]: pgmap v394: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:48:28.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:48:27 smithi189 ceph-mon[57022]: from='client.34760 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:48:28.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:28.261+0000 7f2f6d200700 1 -- 172.21.15.62:0/1807816672 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f2f6804f2a0 con 0x7f2f681004e0 2024-04-20T21:48:28.261 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:48:27 smithi062 ceph-mon[76322]: pgmap v394: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:48:28.261 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:48:27 smithi062 ceph-mon[76322]: from='client.34760 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:48:28.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:28.263+0000 7f2f5ea00700 1 -- 172.21.15.62:0/1807816672 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7f2f5805b2f0 con 0x7f2f681004e0 2024-04-20T21:48:28.262 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:48:28.262 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T21:48:28.262 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:48:28.262 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:48:28.262 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T21:48:28.262 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:48:28.262 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:48:28.262 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T21:48:28.263 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 6 2024-04-20T21:48:28.263 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:48:28.263 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T21:48:28.263 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 2, 2024-04-20T21:48:28.263 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:48:28.263 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:48:28.263 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T21:48:28.263 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 2, 2024-04-20T21:48:28.263 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 12 2024-04-20T21:48:28.263 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T21:48:28.263 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:48:28.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:28.265+0000 7f2f6d200700 1 -- 172.21.15.62:0/1807816672 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2f4c06ecc0 msgr2=0x7f2f4c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:48:28.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:28.265+0000 7f2f6d200700 1 --2- 172.21.15.62:0/1807816672 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2f4c06ecc0 0x7f2f4c071170 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f2f68197530 tx=0x7f2f5401a040 comp rx=0 tx=0).stop 2024-04-20T21:48:28.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:28.265+0000 7f2f6d200700 1 -- 172.21.15.62:0/1807816672 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f681004e0 msgr2=0x7f2f68195f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:48:28.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:28.266+0000 7f2f6d200700 1 --2- 172.21.15.62:0/1807816672 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f681004e0 0x7f2f68195f10 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f2f5800d8d0 tx=0x7f2f5800dc90 comp rx=0 tx=0).stop 2024-04-20T21:48:28.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:28.266+0000 7f2f6d200700 1 -- 172.21.15.62:0/1807816672 shutdown_connections 2024-04-20T21:48:28.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:28.266+0000 7f2f6d200700 1 --2- 172.21.15.62:0/1807816672 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2f4c06ecc0 0x7f2f4c071170 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:48:28.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:28.266+0000 7f2f6d200700 1 --2- 172.21.15.62:0/1807816672 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f681004e0 0x7f2f68195f10 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:48:28.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:28.266+0000 7f2f6d200700 1 --2- 172.21.15.62:0/1807816672 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f68100e80 0x7f2f68196450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:48:28.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:28.266+0000 7f2f6d200700 1 -- 172.21.15.62:0/1807816672 >> 172.21.15.62:0/1807816672 conn(0x7f2f680fbda0 msgr2=0x7f2f680fe180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:48:28.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:28.266+0000 7f2f6d200700 1 -- 172.21.15.62:0/1807816672 shutdown_connections 2024-04-20T21:48:28.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:28.266+0000 7f2f6d200700 1 -- 172.21.15.62:0/1807816672 wait complete. 2024-04-20T21:48:28.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:28.426+0000 7f5ac8800700 1 -- 172.21.15.62:0/2753644945 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5ac0073a00 msgr2=0x7f5ac010c580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:48:28.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:28.426+0000 7f5ac8800700 1 --2- 172.21.15.62:0/2753644945 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5ac0073a00 0x7f5ac010c580 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f5ab0009910 tx=0x7f5ab0009c20 comp rx=0 tx=0).stop 2024-04-20T21:48:28.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:28.426+0000 7f5ac8800700 1 -- 172.21.15.62:0/2753644945 shutdown_connections 2024-04-20T21:48:28.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:28.426+0000 7f5ac8800700 1 --2- 172.21.15.62:0/2753644945 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5ac0073a00 0x7f5ac010c580 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:48:28.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:28.426+0000 7f5ac8800700 1 --2- 172.21.15.62:0/2753644945 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ac00730f0 0x7f5ac00734c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:48:28.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:28.426+0000 7f5ac8800700 1 -- 172.21.15.62:0/2753644945 >> 172.21.15.62:0/2753644945 conn(0x7f5ac00fbda0 msgr2=0x7f5ac00fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:48:28.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:28.426+0000 7f5ac8800700 1 -- 172.21.15.62:0/2753644945 shutdown_connections 2024-04-20T21:48:28.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:28.426+0000 7f5ac8800700 1 -- 172.21.15.62:0/2753644945 wait complete. 2024-04-20T21:48:28.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:28.427+0000 7f5ac8800700 1 Processor -- start 2024-04-20T21:48:28.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:28.427+0000 7f5ac8800700 1 -- start start 2024-04-20T21:48:28.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:28.428+0000 7f5ac8800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ac00730f0 0x7f5ac0197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:48:28.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:28.428+0000 7f5ac8800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5ac0073a00 0x7f5ac0198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:48:28.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:28.428+0000 7f5ac8800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ac019c970 con 0x7f5ac0073a00 2024-04-20T21:48:28.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:28.428+0000 7f5ac8800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ac019cae0 con 0x7f5ac00730f0 2024-04-20T21:48:28.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:28.428+0000 7f5ac5400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5ac0073a00 0x7f5ac0198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:48:28.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:28.428+0000 7f5ac5400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5ac0073a00 0x7f5ac0198630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35238/0 (socket says 172.21.15.62:35238) 2024-04-20T21:48:28.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:28.428+0000 7f5ac5400700 1 -- 172.21.15.62:0/690193373 learned_addr learned my addr 172.21.15.62:0/690193373 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:48:28.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:28.428+0000 7f5ac5e00700 1 --2- 172.21.15.62:0/690193373 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ac00730f0 0x7f5ac0197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:48:28.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:28.428+0000 7f5ac5400700 1 -- 172.21.15.62:0/690193373 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ac00730f0 msgr2=0x7f5ac0197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:48:28.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:28.428+0000 7f5ac5400700 1 --2- 172.21.15.62:0/690193373 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ac00730f0 0x7f5ac0197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:48:28.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:28.428+0000 7f5ac5400700 1 -- 172.21.15.62:0/690193373 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ab00095f0 con 0x7f5ac0073a00 2024-04-20T21:48:28.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:28.429+0000 7f5ac5400700 1 --2- 172.21.15.62:0/690193373 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5ac0073a00 0x7f5ac0198630 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f5ab000b5c0 tx=0x7f5ab000be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:48:28.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:28.429+0000 7f5abaa00700 1 -- 172.21.15.62:0/690193373 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5ab001d070 con 0x7f5ac0073a00 2024-04-20T21:48:28.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:28.429+0000 7f5abaa00700 1 -- 172.21.15.62:0/690193373 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5ab000f460 con 0x7f5ac0073a00 2024-04-20T21:48:28.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:28.429+0000 7f5abaa00700 1 -- 172.21.15.62:0/690193373 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5ab0017630 con 0x7f5ac0073a00 2024-04-20T21:48:28.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:28.429+0000 7f5ac8800700 1 -- 172.21.15.62:0/690193373 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ac019cdc0 con 0x7f5ac0073a00 2024-04-20T21:48:28.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:28.429+0000 7f5ac8800700 1 -- 172.21.15.62:0/690193373 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ac019d2c0 con 0x7f5ac0073a00 2024-04-20T21:48:28.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:28.430+0000 7f5ac8800700 1 -- 172.21.15.62:0/690193373 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ac0109c80 con 0x7f5ac0073a00 2024-04-20T21:48:28.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:28.432+0000 7f5abaa00700 1 -- 172.21.15.62:0/690193373 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f5ab000f5d0 con 0x7f5ac0073a00 2024-04-20T21:48:28.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:28.433+0000 7f5abaa00700 1 --2- 172.21.15.62:0/690193373 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5aa806ecc0 0x7f5aa8071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:48:28.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:28.433+0000 7f5abaa00700 1 -- 172.21.15.62:0/690193373 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f5ab00904b0 con 0x7f5ac0073a00 2024-04-20T21:48:28.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:28.433+0000 7f5ac5e00700 1 --2- 172.21.15.62:0/690193373 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5aa806ecc0 0x7f5aa8071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:48:28.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:28.434+0000 7f5ac5e00700 1 --2- 172.21.15.62:0/690193373 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5aa806ecc0 0x7f5aa8071170 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f5ab4005fd0 tx=0x7f5ab400a560 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:48:28.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:28.438+0000 7f5abaa00700 1 -- 172.21.15.62:0/690193373 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f5ab00591c0 con 0x7f5ac0073a00 2024-04-20T21:48:28.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:28.739+0000 7f5ac8800700 1 -- 172.21.15.62:0/690193373 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f5ac004f2a0 con 0x7f5ac0073a00 2024-04-20T21:48:28.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:28.740+0000 7f5abaa00700 1 -- 172.21.15.62:0/690193373 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 25 v25) v1 ==== 76+0+1887 (secure 0 0 0) 0x7f5ab005c880 con 0x7f5ac0073a00 2024-04-20T21:48:28.740 INFO:teuthology.orchestra.run.smithi062.stdout:e25 2024-04-20T21:48:28.740 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T21:48:28.740 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T21:48:28.741 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T21:48:28.741 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:48:28.741 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T21:48:28.741 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T21:48:28.741 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 20 2024-04-20T21:48:28.741 INFO:teuthology.orchestra.run.smithi062.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-20T21:48:28.741 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T21:48:28.741 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T21:42:01.559061+0000 2024-04-20T21:48:28.741 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T21:48:28.741 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T21:48:28.742 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T21:48:28.742 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T21:48:28.742 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T21:48:28.742 INFO:teuthology.orchestra.run.smithi062.stdout:max_xattr_size 65536 2024-04-20T21:48:28.742 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T21:48:28.742 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T21:48:28.742 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 81 2024-04-20T21:48:28.742 INFO:teuthology.orchestra.run.smithi062.stdout:snap_rstat 1 2024-04-20T21:48:28.742 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T21:48:28.743 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 2 2024-04-20T21:48:28.743 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1 2024-04-20T21:48:28.743 INFO:teuthology.orchestra.run.smithi062.stdout:up {} 2024-04-20T21:48:28.743 INFO:teuthology.orchestra.run.smithi062.stdout:failed 0,1 2024-04-20T21:48:28.743 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T21:48:28.743 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T21:48:28.743 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T21:48:28.743 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T21:48:28.743 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data enabled 2024-04-20T21:48:28.743 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T21:48:28.744 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T21:48:28.744 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 1 2024-04-20T21:48:28.744 INFO:teuthology.orchestra.run.smithi062.stdout:qdb_cluster leader: 0 members: 2024-04-20T21:48:28.744 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:48:28.744 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:48:28.744 INFO:teuthology.orchestra.run.smithi062.stdout:Standby daemons: 2024-04-20T21:48:28.744 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:48:28.744 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ryqblb{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6826/4113916205,v1:172.21.15.62:6827/4113916205] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T21:48:28.744 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ozizwi{-1:34476} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6828/3801975165,v1:172.21.15.62:6829/3801975165] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T21:48:28.744 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.xbgspw{-1:44285} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6826/3996040526,v1:172.21.15.189:6827/3996040526] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:48:28.745 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.dvimer{-1:44289} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6824/1743111843,v1:172.21.15.189:6825/1743111843] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:48:28.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:28.745+0000 7f5ac8800700 1 -- 172.21.15.62:0/690193373 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5aa806ecc0 msgr2=0x7f5aa8071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:48:28.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:28.745+0000 7f5ac8800700 1 --2- 172.21.15.62:0/690193373 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5aa806ecc0 0x7f5aa8071170 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f5ab4005fd0 tx=0x7f5ab400a560 comp rx=0 tx=0).stop 2024-04-20T21:48:28.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:28.745+0000 7f5ac8800700 1 -- 172.21.15.62:0/690193373 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5ac0073a00 msgr2=0x7f5ac0198630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:48:28.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:28.745+0000 7f5ac8800700 1 --2- 172.21.15.62:0/690193373 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5ac0073a00 0x7f5ac0198630 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f5ab000b5c0 tx=0x7f5ab000be00 comp rx=0 tx=0).stop 2024-04-20T21:48:28.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:28.745+0000 7f5ac8800700 1 -- 172.21.15.62:0/690193373 shutdown_connections 2024-04-20T21:48:28.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:28.745+0000 7f5ac8800700 1 --2- 172.21.15.62:0/690193373 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5aa806ecc0 0x7f5aa8071170 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:48:28.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:28.745+0000 7f5ac8800700 1 --2- 172.21.15.62:0/690193373 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ac00730f0 0x7f5ac0197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:48:28.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:28.745+0000 7f5ac8800700 1 --2- 172.21.15.62:0/690193373 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5ac0073a00 0x7f5ac0198630 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:48:28.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:28.745+0000 7f5ac8800700 1 -- 172.21.15.62:0/690193373 >> 172.21.15.62:0/690193373 conn(0x7f5ac00fbda0 msgr2=0x7f5ac0106dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:48:28.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:28.745+0000 7f5ac8800700 1 -- 172.21.15.62:0/690193373 shutdown_connections 2024-04-20T21:48:28.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:28.745+0000 7f5ac8800700 1 -- 172.21.15.62:0/690193373 wait complete. 2024-04-20T21:48:28.747 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 25 2024-04-20T21:48:28.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:28.908+0000 7face3200700 1 -- 172.21.15.62:0/3619209963 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7face4106b80 msgr2=0x7face4106ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:48:28.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:28.908+0000 7face3200700 1 --2- 172.21.15.62:0/3619209963 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7face4106b80 0x7face4106ff0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7facd0009910 tx=0x7facd0009c20 comp rx=0 tx=0).stop 2024-04-20T21:48:28.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:28.908+0000 7face3200700 1 -- 172.21.15.62:0/3619209963 shutdown_connections 2024-04-20T21:48:28.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:28.908+0000 7face3200700 1 --2- 172.21.15.62:0/3619209963 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7face4106b80 0x7face4106ff0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:48:28.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:28.908+0000 7face3200700 1 --2- 172.21.15.62:0/3619209963 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7face410c660 0x7face410ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:48:28.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:28.908+0000 7face3200700 1 -- 172.21.15.62:0/3619209963 >> 172.21.15.62:0/3619209963 conn(0x7face4075e90 msgr2=0x7face40782a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:48:28.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:28.909+0000 7face3200700 1 -- 172.21.15.62:0/3619209963 shutdown_connections 2024-04-20T21:48:28.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:28.909+0000 7face3200700 1 -- 172.21.15.62:0/3619209963 wait complete. 2024-04-20T21:48:28.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:28.909+0000 7face3200700 1 Processor -- start 2024-04-20T21:48:28.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:28.910+0000 7face3200700 1 -- start start 2024-04-20T21:48:28.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:28.910+0000 7face3200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7face4106b80 0x7face419c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:48:28.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:28.910+0000 7face3200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7face410c660 0x7face419c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:48:28.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:28.910+0000 7face3200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7face419ce00 con 0x7face4106b80 2024-04-20T21:48:28.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:28.910+0000 7face3200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7face41a0b90 con 0x7face410c660 2024-04-20T21:48:28.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:28.910+0000 7face1e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7face4106b80 0x7face419c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:48:28.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:28.910+0000 7face1e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7face4106b80 0x7face419c1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35250/0 (socket says 172.21.15.62:35250) 2024-04-20T21:48:28.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:28.910+0000 7face1e00700 1 -- 172.21.15.62:0/3544330189 learned_addr learned my addr 172.21.15.62:0/3544330189 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:48:28.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:28.911+0000 7face1e00700 1 -- 172.21.15.62:0/3544330189 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7face410c660 msgr2=0x7face419c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:48:28.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:28.911+0000 7face1e00700 1 --2- 172.21.15.62:0/3544330189 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7face410c660 0x7face419c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:48:28.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:28.911+0000 7face1e00700 1 -- 172.21.15.62:0/3544330189 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facd00095f0 con 0x7face4106b80 2024-04-20T21:48:28.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:28.911+0000 7face1e00700 1 --2- 172.21.15.62:0/3544330189 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7face4106b80 0x7face419c1e0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7facd400d8d0 tx=0x7facd400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:48:28.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:28.911+0000 7facdaa00700 1 -- 172.21.15.62:0/3544330189 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7facd4009940 con 0x7face4106b80 2024-04-20T21:48:28.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:28.911+0000 7facdaa00700 1 -- 172.21.15.62:0/3544330189 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7facd4010460 con 0x7face4106b80 2024-04-20T21:48:28.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:28.911+0000 7facdaa00700 1 -- 172.21.15.62:0/3544330189 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7facd400f5f0 con 0x7face4106b80 2024-04-20T21:48:28.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:28.911+0000 7face3200700 1 -- 172.21.15.62:0/3544330189 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7face41a0e70 con 0x7face4106b80 2024-04-20T21:48:28.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:28.911+0000 7face3200700 1 -- 172.21.15.62:0/3544330189 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7face41a13c0 con 0x7face4106b80 2024-04-20T21:48:28.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:28.912+0000 7face3200700 1 -- 172.21.15.62:0/3544330189 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7face410eb40 con 0x7face4106b80 2024-04-20T21:48:28.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:28.913+0000 7facdaa00700 1 -- 172.21.15.62:0/3544330189 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7facd40105d0 con 0x7face4106b80 2024-04-20T21:48:28.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:28.914+0000 7facdaa00700 1 --2- 172.21.15.62:0/3544330189 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7facc806ecc0 0x7facc8071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:48:28.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:28.914+0000 7facdaa00700 1 -- 172.21.15.62:0/3544330189 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7facd408ee10 con 0x7face4106b80 2024-04-20T21:48:28.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:28.915+0000 7face1400700 1 --2- 172.21.15.62:0/3544330189 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7facc806ecc0 0x7facc8071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:48:28.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:28.915+0000 7face1400700 1 --2- 172.21.15.62:0/3544330189 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7facc806ecc0 0x7facc8071170 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7facd000b5c0 tx=0x7facd0005fd0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:48:28.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:28.919+0000 7facdaa00700 1 -- 172.21.15.62:0/3544330189 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7facd4057ba0 con 0x7face4106b80 2024-04-20T21:48:29.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:29.171+0000 7face3200700 1 -- 172.21.15.62:0/3544330189 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7face41a16a0 con 0x7facc806ecc0 2024-04-20T21:48:29.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:29.173+0000 7facdaa00700 1 -- 172.21.15.62:0/3544330189 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7face41a16a0 con 0x7facc806ecc0 2024-04-20T21:48:29.172 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:48:29.172 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T21:48:29.172 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T21:48:29.172 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-20T21:48:29.172 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T21:48:29.172 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr", 2024-04-20T21:48:29.172 INFO:teuthology.orchestra.run.smithi062.stdout: "crash", 2024-04-20T21:48:29.172 INFO:teuthology.orchestra.run.smithi062.stdout: "mon", 2024-04-20T21:48:29.173 INFO:teuthology.orchestra.run.smithi062.stdout: "osd" 2024-04-20T21:48:29.173 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T21:48:29.173 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "14/23 daemons upgraded", 2024-04-20T21:48:29.173 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading mds daemons", 2024-04-20T21:48:29.173 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T21:48:29.173 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:48:29.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:29.175+0000 7face3200700 1 -- 172.21.15.62:0/3544330189 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7facc806ecc0 msgr2=0x7facc8071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:48:29.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:29.175+0000 7face3200700 1 --2- 172.21.15.62:0/3544330189 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7facc806ecc0 0x7facc8071170 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7facd000b5c0 tx=0x7facd0005fd0 comp rx=0 tx=0).stop 2024-04-20T21:48:29.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:29.175+0000 7face3200700 1 -- 172.21.15.62:0/3544330189 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7face4106b80 msgr2=0x7face419c1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:48:29.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:29.175+0000 7face3200700 1 --2- 172.21.15.62:0/3544330189 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7face4106b80 0x7face419c1e0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7facd400d8d0 tx=0x7facd400dc90 comp rx=0 tx=0).stop 2024-04-20T21:48:29.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:29.175+0000 7face3200700 1 -- 172.21.15.62:0/3544330189 shutdown_connections 2024-04-20T21:48:29.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:29.175+0000 7face3200700 1 --2- 172.21.15.62:0/3544330189 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7facc806ecc0 0x7facc8071170 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:48:29.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:29.176+0000 7face3200700 1 --2- 172.21.15.62:0/3544330189 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7face4106b80 0x7face419c1e0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:48:29.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:29.176+0000 7face3200700 1 --2- 172.21.15.62:0/3544330189 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7face410c660 0x7face419c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:48:29.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:29.176+0000 7face3200700 1 -- 172.21.15.62:0/3544330189 >> 172.21.15.62:0/3544330189 conn(0x7face4075e90 msgr2=0x7face40778d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:48:29.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:29.176+0000 7face3200700 1 -- 172.21.15.62:0/3544330189 shutdown_connections 2024-04-20T21:48:29.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:29.176+0000 7face3200700 1 -- 172.21.15.62:0/3544330189 wait complete. 2024-04-20T21:48:29.284 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:48:29 smithi062 ceph-mon[76322]: from='client.34764 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:48:29.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:48:29 smithi062 ceph-mon[76322]: from='client.34768 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:48:29.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:48:29 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/1807816672' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:48:29.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:48:29 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/690193373' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T21:48:29.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:29.330+0000 7fec3ac00700 1 -- 172.21.15.62:0/4218054891 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fec341004e0 msgr2=0x7fec341008b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:48:29.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:29.330+0000 7fec3ac00700 1 --2- 172.21.15.62:0/4218054891 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fec341004e0 0x7fec341008b0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fec24009910 tx=0x7fec24009c20 comp rx=0 tx=0).stop 2024-04-20T21:48:29.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:29.331+0000 7fec3ac00700 1 -- 172.21.15.62:0/4218054891 shutdown_connections 2024-04-20T21:48:29.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:29.331+0000 7fec3ac00700 1 --2- 172.21.15.62:0/4218054891 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec34100e80 0x7fec34104e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:48:29.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:29.331+0000 7fec3ac00700 1 --2- 172.21.15.62:0/4218054891 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fec341004e0 0x7fec341008b0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:48:29.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:29.331+0000 7fec3ac00700 1 -- 172.21.15.62:0/4218054891 >> 172.21.15.62:0/4218054891 conn(0x7fec340fbda0 msgr2=0x7fec340fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:48:29.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:29.331+0000 7fec3ac00700 1 -- 172.21.15.62:0/4218054891 shutdown_connections 2024-04-20T21:48:29.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:29.331+0000 7fec3ac00700 1 -- 172.21.15.62:0/4218054891 wait complete. 2024-04-20T21:48:29.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:29.332+0000 7fec3ac00700 1 Processor -- start 2024-04-20T21:48:29.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:29.332+0000 7fec3ac00700 1 -- start start 2024-04-20T21:48:29.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:29.332+0000 7fec3ac00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec341004e0 0x7fec34073050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:48:29.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:29.332+0000 7fec3ac00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fec34100e80 0x7fec34073590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:48:29.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:29.332+0000 7fec3ac00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec34073ad0 con 0x7fec34100e80 2024-04-20T21:48:29.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:29.332+0000 7fec3ac00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec34073c10 con 0x7fec341004e0 2024-04-20T21:48:29.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:29.333+0000 7fec33400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fec34100e80 0x7fec34073590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:48:29.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:29.333+0000 7fec33400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fec34100e80 0x7fec34073590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35270/0 (socket says 172.21.15.62:35270) 2024-04-20T21:48:29.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:29.333+0000 7fec33400700 1 -- 172.21.15.62:0/1992799275 learned_addr learned my addr 172.21.15.62:0/1992799275 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:48:29.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:29.333+0000 7fec33e00700 1 --2- 172.21.15.62:0/1992799275 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec341004e0 0x7fec34073050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:48:29.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:29.333+0000 7fec33400700 1 -- 172.21.15.62:0/1992799275 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec341004e0 msgr2=0x7fec34073050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:48:29.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:29.333+0000 7fec33400700 1 --2- 172.21.15.62:0/1992799275 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec341004e0 0x7fec34073050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:48:29.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:29.333+0000 7fec33400700 1 -- 172.21.15.62:0/1992799275 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec240095f0 con 0x7fec34100e80 2024-04-20T21:48:29.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:29.334+0000 7fec33400700 1 --2- 172.21.15.62:0/1992799275 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fec34100e80 0x7fec34073590 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fec2000b480 tx=0x7fec2000b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:48:29.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:29.334+0000 7fec30c00700 1 -- 172.21.15.62:0/1992799275 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fec200118c0 con 0x7fec34100e80 2024-04-20T21:48:29.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:29.334+0000 7fec30c00700 1 -- 172.21.15.62:0/1992799275 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fec20010380 con 0x7fec34100e80 2024-04-20T21:48:29.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:29.334+0000 7fec3ac00700 1 -- 172.21.15.62:0/1992799275 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec341a2160 con 0x7fec34100e80 2024-04-20T21:48:29.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:29.334+0000 7fec30c00700 1 -- 172.21.15.62:0/1992799275 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fec2000f4f0 con 0x7fec34100e80 2024-04-20T21:48:29.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:29.334+0000 7fec3ac00700 1 -- 172.21.15.62:0/1992799275 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec341a2570 con 0x7fec34100e80 2024-04-20T21:48:29.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:29.335+0000 7fec3ac00700 1 -- 172.21.15.62:0/1992799275 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec341087f0 con 0x7fec34100e80 2024-04-20T21:48:29.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:29.336+0000 7fec30c00700 1 -- 172.21.15.62:0/1992799275 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fec200104f0 con 0x7fec34100e80 2024-04-20T21:48:29.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:29.337+0000 7fec30c00700 1 --2- 172.21.15.62:0/1992799275 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fec1806ecc0 0x7fec18071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:48:29.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:29.337+0000 7fec30c00700 1 -- 172.21.15.62:0/1992799275 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fec2008eb30 con 0x7fec34100e80 2024-04-20T21:48:29.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:29.337+0000 7fec33e00700 1 --2- 172.21.15.62:0/1992799275 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fec1806ecc0 0x7fec18071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:48:29.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:29.338+0000 7fec33e00700 1 --2- 172.21.15.62:0/1992799275 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fec1806ecc0 0x7fec18071170 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fec24006010 tx=0x7fec2401a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:48:29.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:29.343+0000 7fec30c00700 1 -- 172.21.15.62:0/1992799275 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fec20057840 con 0x7fec34100e80 2024-04-20T21:48:29.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:48:29 smithi189 ceph-mon[57022]: from='client.34764 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:48:29.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:48:29 smithi189 ceph-mon[57022]: from='client.34768 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:48:29.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:48:29 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/1807816672' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:48:29.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:48:29 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/690193373' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T21:48:29.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:29.668+0000 7fec3ac00700 1 -- 172.21.15.62:0/1992799275 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fec341081d0 con 0x7fec34100e80 2024-04-20T21:48:29.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:29.669+0000 7fec30c00700 1 -- 172.21.15.62:0/1992799275 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7fec2005af00 con 0x7fec34100e80 2024-04-20T21:48:29.667 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-20T21:48:29.667 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-20T21:48:29.667 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is degraded 2024-04-20T21:48:29.667 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T21:48:29.667 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T21:48:29.667 INFO:teuthology.orchestra.run.smithi062.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-20T21:48:29.667 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-20T21:48:29.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:29.672+0000 7fec3ac00700 1 -- 172.21.15.62:0/1992799275 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fec1806ecc0 msgr2=0x7fec18071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:48:29.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:29.672+0000 7fec3ac00700 1 --2- 172.21.15.62:0/1992799275 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fec1806ecc0 0x7fec18071170 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fec24006010 tx=0x7fec2401a040 comp rx=0 tx=0).stop 2024-04-20T21:48:29.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:29.672+0000 7fec3ac00700 1 -- 172.21.15.62:0/1992799275 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fec34100e80 msgr2=0x7fec34073590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:48:29.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:29.672+0000 7fec3ac00700 1 --2- 172.21.15.62:0/1992799275 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fec34100e80 0x7fec34073590 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fec2000b480 tx=0x7fec2000b840 comp rx=0 tx=0).stop 2024-04-20T21:48:29.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:29.672+0000 7fec3ac00700 1 -- 172.21.15.62:0/1992799275 shutdown_connections 2024-04-20T21:48:29.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:29.672+0000 7fec3ac00700 1 --2- 172.21.15.62:0/1992799275 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fec1806ecc0 0x7fec18071170 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:48:29.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:29.672+0000 7fec3ac00700 1 --2- 172.21.15.62:0/1992799275 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec341004e0 0x7fec34073050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:48:29.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:29.672+0000 7fec3ac00700 1 --2- 172.21.15.62:0/1992799275 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fec34100e80 0x7fec34073590 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:48:29.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:29.672+0000 7fec3ac00700 1 -- 172.21.15.62:0/1992799275 >> 172.21.15.62:0/1992799275 conn(0x7fec340fbda0 msgr2=0x7fec340fd470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:48:29.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:29.672+0000 7fec3ac00700 1 -- 172.21.15.62:0/1992799275 shutdown_connections 2024-04-20T21:48:29.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:29.672+0000 7fec3ac00700 1 -- 172.21.15.62:0/1992799275 wait complete. 2024-04-20T21:48:30.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:48:30 smithi062 ceph-mon[76322]: pgmap v395: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:48:30.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:48:30 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/1992799275' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T21:48:30.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:48:30 smithi189 ceph-mon[57022]: pgmap v395: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:48:30.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:48:30 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/1992799275' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T21:48:31.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:48:31 smithi062 ceph-mon[76322]: from='client.34780 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:48:31.486 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:48:31 smithi189 ceph-mon[57022]: from='client.34780 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:48:32.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:48:32 smithi062 ceph-mon[76322]: pgmap v396: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:48:32.486 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:48:32 smithi189 ceph-mon[57022]: pgmap v396: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:48:33.486 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:48:33 smithi189 ceph-mon[57022]: pgmap v397: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:48:33.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:48:33 smithi062 ceph-mon[76322]: pgmap v397: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:48:34.736 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:48:34 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:48:34.784 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:48:34 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:48:35.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:48:35 smithi189 ceph-mon[57022]: pgmap v398: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:48:35.784 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:48:35 smithi062 ceph-mon[76322]: pgmap v398: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:48:37.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:48:37 smithi189 ceph-mon[57022]: pgmap v399: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:48:38.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:48:37 smithi062 ceph-mon[76322]: pgmap v399: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:48:39.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:48:39 smithi189 ceph-mon[57022]: pgmap v400: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:48:40.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:48:39 smithi062 ceph-mon[76322]: pgmap v400: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:48:41.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:48:41 smithi189 ceph-mon[57022]: pgmap v401: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:48:42.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:48:41 smithi062 ceph-mon[76322]: pgmap v401: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:48:43.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:48:43 smithi189 ceph-mon[57022]: pgmap v402: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:48:44.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:48:43 smithi062 ceph-mon[76322]: pgmap v402: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:48:45.878 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:48:45 smithi062 ceph-mon[76322]: pgmap v403: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:48:45.986 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:48:45 smithi189 ceph-mon[57022]: pgmap v403: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:48:47.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:48:47 smithi189 ceph-mon[57022]: pgmap v404: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:48:48.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:48:47 smithi062 ceph-mon[76322]: pgmap v404: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:48:49.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:48:49 smithi189 ceph-mon[57022]: pgmap v405: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:48:49.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:48:49 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:48:50.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:48:49 smithi062 ceph-mon[76322]: pgmap v405: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:48:50.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:48:49 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:48:51.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:48:51 smithi189 ceph-mon[57022]: pgmap v406: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:48:52.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:48:51 smithi062 ceph-mon[76322]: pgmap v406: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:48:53.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:48:53 smithi189 ceph-mon[57022]: pgmap v407: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:48:54.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:48:53 smithi062 ceph-mon[76322]: pgmap v407: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:48:55.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:48:55 smithi189 ceph-mon[57022]: pgmap v408: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:48:56.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:48:55 smithi062 ceph-mon[76322]: pgmap v408: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:48:57.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:48:57 smithi189 ceph-mon[57022]: pgmap v409: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:48:58.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:48:57 smithi062 ceph-mon[76322]: pgmap v409: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:48:59.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:59.849+0000 7fe5e0000700 1 -- 172.21.15.62:0/250578003 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe5d8106b80 msgr2=0x7fe5d8106ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:48:59.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:59.849+0000 7fe5e0000700 1 --2- 172.21.15.62:0/250578003 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe5d8106b80 0x7fe5d8106ff0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fe5c4009910 tx=0x7fe5c4009c20 comp rx=0 tx=0).stop 2024-04-20T21:48:59.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:59.849+0000 7fe5e0000700 1 -- 172.21.15.62:0/250578003 shutdown_connections 2024-04-20T21:48:59.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:59.849+0000 7fe5e0000700 1 --2- 172.21.15.62:0/250578003 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe5d8106b80 0x7fe5d8106ff0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:48:59.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:59.849+0000 7fe5e0000700 1 --2- 172.21.15.62:0/250578003 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe5d810c8f0 0x7fe5d810ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:48:59.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:59.849+0000 7fe5e0000700 1 -- 172.21.15.62:0/250578003 >> 172.21.15.62:0/250578003 conn(0x7fe5d8076060 msgr2=0x7fe5d8078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:48:59.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:59.849+0000 7fe5e0000700 1 -- 172.21.15.62:0/250578003 shutdown_connections 2024-04-20T21:48:59.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:59.849+0000 7fe5e0000700 1 -- 172.21.15.62:0/250578003 wait complete. 2024-04-20T21:48:59.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:59.850+0000 7fe5e0000700 1 Processor -- start 2024-04-20T21:48:59.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:59.851+0000 7fe5e0000700 1 -- start start 2024-04-20T21:48:59.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:59.851+0000 7fe5e0000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe5d8106b80 0x7fe5d819c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:48:59.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:59.851+0000 7fe5e0000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe5d810c8f0 0x7fe5d819c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:48:59.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:59.851+0000 7fe5e0000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5d819d060 con 0x7fe5d8106b80 2024-04-20T21:48:59.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:59.851+0000 7fe5e0000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5d81a0df0 con 0x7fe5d810c8f0 2024-04-20T21:48:59.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:59.851+0000 7fe5dd600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe5d8106b80 0x7fe5d819c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:48:59.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:59.851+0000 7fe5dd600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe5d8106b80 0x7fe5d819c440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47892/0 (socket says 172.21.15.62:47892) 2024-04-20T21:48:59.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:59.851+0000 7fe5dd600700 1 -- 172.21.15.62:0/2390854737 learned_addr learned my addr 172.21.15.62:0/2390854737 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:48:59.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:59.852+0000 7fe5dcc00700 1 --2- 172.21.15.62:0/2390854737 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe5d810c8f0 0x7fe5d819c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:48:59.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:59.852+0000 7fe5dd600700 1 -- 172.21.15.62:0/2390854737 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe5d810c8f0 msgr2=0x7fe5d819c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:48:59.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:59.852+0000 7fe5dd600700 1 --2- 172.21.15.62:0/2390854737 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe5d810c8f0 0x7fe5d819c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:48:59.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:59.852+0000 7fe5dd600700 1 -- 172.21.15.62:0/2390854737 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5c40095f0 con 0x7fe5d8106b80 2024-04-20T21:48:59.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:59.852+0000 7fe5dcc00700 1 --2- 172.21.15.62:0/2390854737 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe5d810c8f0 0x7fe5d819c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T21:48:59.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:59.852+0000 7fe5dd600700 1 --2- 172.21.15.62:0/2390854737 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe5d8106b80 0x7fe5d819c440 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fe5cc00d8d0 tx=0x7fe5cc00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:48:59.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:59.852+0000 7fe5d6000700 1 -- 172.21.15.62:0/2390854737 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe5cc009940 con 0x7fe5d8106b80 2024-04-20T21:48:59.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:59.853+0000 7fe5d6000700 1 -- 172.21.15.62:0/2390854737 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe5cc010460 con 0x7fe5d8106b80 2024-04-20T21:48:59.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:59.853+0000 7fe5d6000700 1 -- 172.21.15.62:0/2390854737 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe5cc00f5f0 con 0x7fe5d8106b80 2024-04-20T21:48:59.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:59.853+0000 7fe5e0000700 1 -- 172.21.15.62:0/2390854737 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5d81a10d0 con 0x7fe5d8106b80 2024-04-20T21:48:59.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:59.853+0000 7fe5e0000700 1 -- 172.21.15.62:0/2390854737 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5d81a1620 con 0x7fe5d8106b80 2024-04-20T21:48:59.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:59.854+0000 7fe5e0000700 1 -- 172.21.15.62:0/2390854737 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5d810ecc0 con 0x7fe5d8106b80 2024-04-20T21:48:59.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:59.855+0000 7fe5d6000700 1 -- 172.21.15.62:0/2390854737 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe5cc0105d0 con 0x7fe5d8106b80 2024-04-20T21:48:59.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:59.856+0000 7fe5d6000700 1 --2- 172.21.15.62:0/2390854737 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe5c006ecc0 0x7fe5c0071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:48:59.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:59.856+0000 7fe5d6000700 1 -- 172.21.15.62:0/2390854737 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fe5cc08ef20 con 0x7fe5d8106b80 2024-04-20T21:48:59.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:59.856+0000 7fe5dcc00700 1 --2- 172.21.15.62:0/2390854737 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe5c006ecc0 0x7fe5c0071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:48:59.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:59.857+0000 7fe5dcc00700 1 --2- 172.21.15.62:0/2390854737 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe5c006ecc0 0x7fe5c0071170 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fe5c400b5c0 tx=0x7fe5c4005960 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:48:59.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:48:59.861+0000 7fe5d6000700 1 -- 172.21.15.62:0/2390854737 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fe5cc057c30 con 0x7fe5d8106b80 2024-04-20T21:48:59.986 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:48:59 smithi189 ceph-mon[57022]: pgmap v410: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:49:00.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:48:59 smithi062 ceph-mon[76322]: pgmap v410: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:49:00.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:00.122+0000 7fe5e0000700 1 -- 172.21.15.62:0/2390854737 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe5d81a1840 con 0x7fe5c006ecc0 2024-04-20T21:49:00.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:00.124+0000 7fe5d6000700 1 -- 172.21.15.62:0/2390854737 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fe5d81a1840 con 0x7fe5c006ecc0 2024-04-20T21:49:00.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:00.128+0000 7fe5e0000700 1 -- 172.21.15.62:0/2390854737 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe5c006ecc0 msgr2=0x7fe5c0071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:49:00.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:00.128+0000 7fe5e0000700 1 --2- 172.21.15.62:0/2390854737 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe5c006ecc0 0x7fe5c0071170 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fe5c400b5c0 tx=0x7fe5c4005960 comp rx=0 tx=0).stop 2024-04-20T21:49:00.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:00.128+0000 7fe5e0000700 1 -- 172.21.15.62:0/2390854737 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe5d8106b80 msgr2=0x7fe5d819c440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:49:00.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:00.128+0000 7fe5e0000700 1 --2- 172.21.15.62:0/2390854737 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe5d8106b80 0x7fe5d819c440 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fe5cc00d8d0 tx=0x7fe5cc00dc90 comp rx=0 tx=0).stop 2024-04-20T21:49:00.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:00.128+0000 7fe5e0000700 1 -- 172.21.15.62:0/2390854737 shutdown_connections 2024-04-20T21:49:00.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:00.128+0000 7fe5e0000700 1 --2- 172.21.15.62:0/2390854737 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe5c006ecc0 0x7fe5c0071170 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:49:00.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:00.128+0000 7fe5e0000700 1 --2- 172.21.15.62:0/2390854737 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe5d8106b80 0x7fe5d819c440 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:49:00.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:00.128+0000 7fe5e0000700 1 --2- 172.21.15.62:0/2390854737 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe5d810c8f0 0x7fe5d819c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:49:00.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:00.128+0000 7fe5e0000700 1 -- 172.21.15.62:0/2390854737 >> 172.21.15.62:0/2390854737 conn(0x7fe5d8076060 msgr2=0x7fe5d8077ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:49:00.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:00.128+0000 7fe5e0000700 1 -- 172.21.15.62:0/2390854737 shutdown_connections 2024-04-20T21:49:00.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:00.128+0000 7fe5e0000700 1 -- 172.21.15.62:0/2390854737 wait complete. 2024-04-20T21:49:00.144 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T21:49:00.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:00.290+0000 7fc732000700 1 -- 172.21.15.62:0/1869463849 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc72c106b80 msgr2=0x7fc72c106ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:49:00.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:00.290+0000 7fc732000700 1 --2- 172.21.15.62:0/1869463849 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc72c106b80 0x7fc72c106ff0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fc718009910 tx=0x7fc718009c20 comp rx=0 tx=0).stop 2024-04-20T21:49:00.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:00.290+0000 7fc732000700 1 -- 172.21.15.62:0/1869463849 shutdown_connections 2024-04-20T21:49:00.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:00.290+0000 7fc732000700 1 --2- 172.21.15.62:0/1869463849 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc72c106b80 0x7fc72c106ff0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:49:00.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:00.290+0000 7fc732000700 1 --2- 172.21.15.62:0/1869463849 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc72c10c8f0 0x7fc72c10ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:49:00.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:00.290+0000 7fc732000700 1 -- 172.21.15.62:0/1869463849 >> 172.21.15.62:0/1869463849 conn(0x7fc72c076060 msgr2=0x7fc72c078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:49:00.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:00.291+0000 7fc732000700 1 -- 172.21.15.62:0/1869463849 shutdown_connections 2024-04-20T21:49:00.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:00.291+0000 7fc732000700 1 -- 172.21.15.62:0/1869463849 wait complete. 2024-04-20T21:49:00.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:00.292+0000 7fc732000700 1 Processor -- start 2024-04-20T21:49:00.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:00.292+0000 7fc732000700 1 -- start start 2024-04-20T21:49:00.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:00.292+0000 7fc732000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc72c106b80 0x7fc72c19c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:49:00.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:00.292+0000 7fc732000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc72c10c8f0 0x7fc72c19c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:49:00.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:00.292+0000 7fc732000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc72c19d060 con 0x7fc72c106b80 2024-04-20T21:49:00.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:00.292+0000 7fc732000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc72c1a0df0 con 0x7fc72c10c8f0 2024-04-20T21:49:00.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:00.293+0000 7fc72b400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc72c106b80 0x7fc72c19c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:49:00.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:00.293+0000 7fc72b400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc72c106b80 0x7fc72c19c440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47902/0 (socket says 172.21.15.62:47902) 2024-04-20T21:49:00.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:00.293+0000 7fc72b400700 1 -- 172.21.15.62:0/3897488157 learned_addr learned my addr 172.21.15.62:0/3897488157 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:49:00.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:00.293+0000 7fc72aa00700 1 --2- 172.21.15.62:0/3897488157 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc72c10c8f0 0x7fc72c19c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:49:00.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:00.293+0000 7fc72b400700 1 -- 172.21.15.62:0/3897488157 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc72c10c8f0 msgr2=0x7fc72c19c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:49:00.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:00.293+0000 7fc72b400700 1 --2- 172.21.15.62:0/3897488157 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc72c10c8f0 0x7fc72c19c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:49:00.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:00.293+0000 7fc72b400700 1 -- 172.21.15.62:0/3897488157 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7180095f0 con 0x7fc72c106b80 2024-04-20T21:49:00.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:00.294+0000 7fc72b400700 1 --2- 172.21.15.62:0/3897488157 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc72c106b80 0x7fc72c19c440 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fc71c00d8d0 tx=0x7fc71c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:49:00.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:00.294+0000 7fc723e00700 1 -- 172.21.15.62:0/3897488157 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc71c009940 con 0x7fc72c106b80 2024-04-20T21:49:00.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:00.294+0000 7fc723e00700 1 -- 172.21.15.62:0/3897488157 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc71c010460 con 0x7fc72c106b80 2024-04-20T21:49:00.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:00.294+0000 7fc723e00700 1 -- 172.21.15.62:0/3897488157 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc71c00f5f0 con 0x7fc72c106b80 2024-04-20T21:49:00.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:00.294+0000 7fc732000700 1 -- 172.21.15.62:0/3897488157 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc72c1a10d0 con 0x7fc72c106b80 2024-04-20T21:49:00.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:00.294+0000 7fc732000700 1 -- 172.21.15.62:0/3897488157 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc72c1a1620 con 0x7fc72c106b80 2024-04-20T21:49:00.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:00.296+0000 7fc732000700 1 -- 172.21.15.62:0/3897488157 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc72c10ecc0 con 0x7fc72c106b80 2024-04-20T21:49:00.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:00.297+0000 7fc723e00700 1 -- 172.21.15.62:0/3897488157 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fc71c0105d0 con 0x7fc72c106b80 2024-04-20T21:49:00.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:00.297+0000 7fc723e00700 1 --2- 172.21.15.62:0/3897488157 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc71006ecc0 0x7fc710071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:49:00.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:00.298+0000 7fc723e00700 1 -- 172.21.15.62:0/3897488157 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fc71c08ed70 con 0x7fc72c106b80 2024-04-20T21:49:00.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:00.298+0000 7fc72aa00700 1 --2- 172.21.15.62:0/3897488157 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc71006ecc0 0x7fc710071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:49:00.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:00.299+0000 7fc72aa00700 1 --2- 172.21.15.62:0/3897488157 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc71006ecc0 0x7fc710071170 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fc72c19da60 tx=0x7fc71801a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:49:00.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:00.302+0000 7fc723e00700 1 -- 172.21.15.62:0/3897488157 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fc71c057ba0 con 0x7fc72c106b80 2024-04-20T21:49:00.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:00.556+0000 7fc732000700 1 -- 172.21.15.62:0/3897488157 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc72c1a1890 con 0x7fc71006ecc0 2024-04-20T21:49:00.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:00.558+0000 7fc723e00700 1 -- 172.21.15.62:0/3897488157 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fc72c1a1890 con 0x7fc71006ecc0 2024-04-20T21:49:00.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:00.561+0000 7fc732000700 1 -- 172.21.15.62:0/3897488157 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc71006ecc0 msgr2=0x7fc710071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:49:00.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:00.561+0000 7fc732000700 1 --2- 172.21.15.62:0/3897488157 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc71006ecc0 0x7fc710071170 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fc72c19da60 tx=0x7fc71801a040 comp rx=0 tx=0).stop 2024-04-20T21:49:00.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:00.561+0000 7fc732000700 1 -- 172.21.15.62:0/3897488157 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc72c106b80 msgr2=0x7fc72c19c440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:49:00.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:00.561+0000 7fc732000700 1 --2- 172.21.15.62:0/3897488157 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc72c106b80 0x7fc72c19c440 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fc71c00d8d0 tx=0x7fc71c00dc90 comp rx=0 tx=0).stop 2024-04-20T21:49:00.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:00.561+0000 7fc732000700 1 -- 172.21.15.62:0/3897488157 shutdown_connections 2024-04-20T21:49:00.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:00.561+0000 7fc732000700 1 --2- 172.21.15.62:0/3897488157 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc71006ecc0 0x7fc710071170 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:49:00.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:00.561+0000 7fc732000700 1 --2- 172.21.15.62:0/3897488157 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc72c106b80 0x7fc72c19c440 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:49:00.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:00.561+0000 7fc732000700 1 --2- 172.21.15.62:0/3897488157 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc72c10c8f0 0x7fc72c19c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:49:00.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:00.561+0000 7fc732000700 1 -- 172.21.15.62:0/3897488157 >> 172.21.15.62:0/3897488157 conn(0x7fc72c076060 msgr2=0x7fc72c077ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:49:00.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:00.561+0000 7fc732000700 1 -- 172.21.15.62:0/3897488157 shutdown_connections 2024-04-20T21:49:00.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:00.562+0000 7fc732000700 1 -- 172.21.15.62:0/3897488157 wait complete. 2024-04-20T21:49:00.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:00.724+0000 7f436dc00700 1 -- 172.21.15.62:0/964448940 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4368100e80 msgr2=0x7f4368104e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:49:00.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:00.724+0000 7f436dc00700 1 --2- 172.21.15.62:0/964448940 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4368100e80 0x7f4368104e50 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f4350009910 tx=0x7f4350009c20 comp rx=0 tx=0).stop 2024-04-20T21:49:00.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:00.725+0000 7f436dc00700 1 -- 172.21.15.62:0/964448940 shutdown_connections 2024-04-20T21:49:00.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:00.725+0000 7f436dc00700 1 --2- 172.21.15.62:0/964448940 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4368100e80 0x7f4368104e50 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:49:00.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:00.725+0000 7f436dc00700 1 --2- 172.21.15.62:0/964448940 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f43681004e0 0x7f43681008b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:49:00.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:00.725+0000 7f436dc00700 1 -- 172.21.15.62:0/964448940 >> 172.21.15.62:0/964448940 conn(0x7f43680fbda0 msgr2=0x7f43680fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:49:00.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:00.725+0000 7f436dc00700 1 -- 172.21.15.62:0/964448940 shutdown_connections 2024-04-20T21:49:00.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:00.725+0000 7f436dc00700 1 -- 172.21.15.62:0/964448940 wait complete. 2024-04-20T21:49:00.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:00.726+0000 7f436dc00700 1 Processor -- start 2024-04-20T21:49:00.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:00.726+0000 7f436dc00700 1 -- start start 2024-04-20T21:49:00.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:00.726+0000 7f436dc00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f43681004e0 0x7f4368198110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:49:00.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:00.726+0000 7f436dc00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4368100e80 0x7f4368198650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:49:00.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:00.727+0000 7f436dc00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4368198d30 con 0x7f4368100e80 2024-04-20T21:49:00.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:00.727+0000 7f436dc00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f436819cac0 con 0x7f43681004e0 2024-04-20T21:49:00.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:00.727+0000 7f4366000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4368100e80 0x7f4368198650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:49:00.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:00.727+0000 7f4366000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4368100e80 0x7f4368198650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47926/0 (socket says 172.21.15.62:47926) 2024-04-20T21:49:00.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:00.727+0000 7f4366000700 1 -- 172.21.15.62:0/3254997536 learned_addr learned my addr 172.21.15.62:0/3254997536 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:49:00.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:00.727+0000 7f4366a00700 1 --2- 172.21.15.62:0/3254997536 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f43681004e0 0x7f4368198110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:49:00.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:00.728+0000 7f4366000700 1 -- 172.21.15.62:0/3254997536 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f43681004e0 msgr2=0x7f4368198110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:49:00.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:00.728+0000 7f4366000700 1 --2- 172.21.15.62:0/3254997536 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f43681004e0 0x7f4368198110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:49:00.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:00.728+0000 7f4366000700 1 -- 172.21.15.62:0/3254997536 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43500095f0 con 0x7f4368100e80 2024-04-20T21:49:00.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:00.728+0000 7f4366000700 1 --2- 172.21.15.62:0/3254997536 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4368100e80 0x7f4368198650 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f435000bd70 tx=0x7f435000bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:49:00.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:00.728+0000 7f435f400700 1 -- 172.21.15.62:0/3254997536 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f435001d070 con 0x7f4368100e80 2024-04-20T21:49:00.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:00.728+0000 7f435f400700 1 -- 172.21.15.62:0/3254997536 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4350005dc0 con 0x7f4368100e80 2024-04-20T21:49:00.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:00.728+0000 7f436dc00700 1 -- 172.21.15.62:0/3254997536 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f436819cd40 con 0x7f4368100e80 2024-04-20T21:49:00.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:00.728+0000 7f435f400700 1 -- 172.21.15.62:0/3254997536 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4350017400 con 0x7f4368100e80 2024-04-20T21:49:00.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:00.728+0000 7f436dc00700 1 -- 172.21.15.62:0/3254997536 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f436819d230 con 0x7f4368100e80 2024-04-20T21:49:00.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:00.730+0000 7f436dc00700 1 -- 172.21.15.62:0/3254997536 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43681045d0 con 0x7f4368100e80 2024-04-20T21:49:00.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:00.730+0000 7f435f400700 1 -- 172.21.15.62:0/3254997536 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f435000f460 con 0x7f4368100e80 2024-04-20T21:49:00.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:00.731+0000 7f435f400700 1 --2- 172.21.15.62:0/3254997536 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f434c06ecc0 0x7f434c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:49:00.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:00.732+0000 7f435f400700 1 -- 172.21.15.62:0/3254997536 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f4350090350 con 0x7f4368100e80 2024-04-20T21:49:00.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:00.732+0000 7f4366a00700 1 --2- 172.21.15.62:0/3254997536 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f434c06ecc0 0x7f434c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:49:00.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:00.733+0000 7f4366a00700 1 --2- 172.21.15.62:0/3254997536 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f434c06ecc0 0x7f434c071170 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f4358005950 tx=0x7f435800b410 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:49:00.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:00.736+0000 7f435f400700 1 -- 172.21.15.62:0/3254997536 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f4350059060 con 0x7f4368100e80 2024-04-20T21:49:00.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:00.989+0000 7f436dc00700 1 -- 172.21.15.62:0/3254997536 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f436819d450 con 0x7f434c06ecc0 2024-04-20T21:49:01.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.021+0000 7f435f400700 1 -- 172.21.15.62:0/3254997536 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f436819d450 con 0x7f434c06ecc0 2024-04-20T21:49:01.021 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T21:49:01.021 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (14m) 6m ago 28m 14.2M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T21:49:01.021 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (29m) 6m ago 29m 9512k - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T21:49:01.021 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (27m) 7m ago 27m 8942k - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T21:49:01.021 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (10m) 6m ago 29m 6660k - 19.0.0-3133-ga8c84f59 76bc6b91b726 2825d5ba11c6 2024-04-20T21:49:01.021 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (10m) 7m ago 27m 6665k - 19.0.0-3133-ga8c84f59 76bc6b91b726 20f746fa1bba 2024-04-20T21:49:01.021 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (14m) 6m ago 28m 74.2M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T21:49:01.021 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (6m) 6m ago 21m 11.2M - 19.0.0-3133-ga8c84f59 76bc6b91b726 de80bc90a683 2024-04-20T21:49:01.021 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (6m) 6m ago 21m 11.9M - 19.0.0-3133-ga8c84f59 76bc6b91b726 cd793b7ee98e 2024-04-20T21:49:01.021 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (21m) 7m ago 21m 117M - 18.2.0 dc2bc1663786 fb2a8a223db8 2024-04-20T21:49:01.021 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (21m) 7m ago 21m 55.6M - 18.2.0 dc2bc1663786 f2eec2ee8951 2024-04-20T21:49:01.021 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (17m) 6m ago 30m 511M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T21:49:01.022 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (16m) 7m ago 27m 452M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T21:49:01.022 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (12m) 6m ago 30m 55.9M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 019eb1479485 2024-04-20T21:49:01.022 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (12m) 7m ago 27m 39.8M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 55aac560aab6 2024-04-20T21:49:01.022 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (16m) 6m ago 28m 7952k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T21:49:01.022 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (15m) 7m ago 27m 8304k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T21:49:01.022 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (10m) 6m ago 26m 88.0M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 ee2bb901f26e 2024-04-20T21:49:01.022 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (9m) 6m ago 25m 92.3M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 9075152d4162 2024-04-20T21:49:01.022 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (9m) 6m ago 25m 74.4M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 2d4f3e38dc0e 2024-04-20T21:49:01.022 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (8m) 7m ago 24m 87.6M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 8f97b17ae0eb 2024-04-20T21:49:01.022 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (7m) 7m ago 23m 90.7M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 653d8079967d 2024-04-20T21:49:01.022 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (7m) 7m ago 23m 11.5M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 596dded14c12 2024-04-20T21:49:01.023 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (15m) 6m ago 28m 74.0M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T21:49:01.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.024+0000 7f436dc00700 1 -- 172.21.15.62:0/3254997536 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f434c06ecc0 msgr2=0x7f434c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:49:01.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.024+0000 7f436dc00700 1 --2- 172.21.15.62:0/3254997536 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f434c06ecc0 0x7f434c071170 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f4358005950 tx=0x7f435800b410 comp rx=0 tx=0).stop 2024-04-20T21:49:01.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.024+0000 7f436dc00700 1 -- 172.21.15.62:0/3254997536 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4368100e80 msgr2=0x7f4368198650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:49:01.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.024+0000 7f436dc00700 1 --2- 172.21.15.62:0/3254997536 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4368100e80 0x7f4368198650 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f435000bd70 tx=0x7f435000bda0 comp rx=0 tx=0).stop 2024-04-20T21:49:01.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.025+0000 7f436dc00700 1 -- 172.21.15.62:0/3254997536 shutdown_connections 2024-04-20T21:49:01.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.025+0000 7f436dc00700 1 --2- 172.21.15.62:0/3254997536 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f434c06ecc0 0x7f434c071170 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:49:01.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.025+0000 7f436dc00700 1 --2- 172.21.15.62:0/3254997536 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f43681004e0 0x7f4368198110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:49:01.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.025+0000 7f436dc00700 1 --2- 172.21.15.62:0/3254997536 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4368100e80 0x7f4368198650 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:49:01.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.025+0000 7f436dc00700 1 -- 172.21.15.62:0/3254997536 >> 172.21.15.62:0/3254997536 conn(0x7f43680fbda0 msgr2=0x7f43680fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:49:01.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.025+0000 7f436dc00700 1 -- 172.21.15.62:0/3254997536 shutdown_connections 2024-04-20T21:49:01.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.025+0000 7f436dc00700 1 -- 172.21.15.62:0/3254997536 wait complete. 2024-04-20T21:49:01.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.182+0000 7f504fc00700 1 -- 172.21.15.62:0/1638593243 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5048068540 msgr2=0x7f50480689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:49:01.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.182+0000 7f504fc00700 1 --2- 172.21.15.62:0/1638593243 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5048068540 0x7f50480689b0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f5030009910 tx=0x7f5030009c20 comp rx=0 tx=0).stop 2024-04-20T21:49:01.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.183+0000 7f504fc00700 1 -- 172.21.15.62:0/1638593243 shutdown_connections 2024-04-20T21:49:01.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.183+0000 7f504fc00700 1 --2- 172.21.15.62:0/1638593243 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5048068540 0x7f50480689b0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:49:01.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.183+0000 7f504fc00700 1 --2- 172.21.15.62:0/1638593243 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5048101070 0x7f5048101440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:49:01.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.183+0000 7f504fc00700 1 -- 172.21.15.62:0/1638593243 >> 172.21.15.62:0/1638593243 conn(0x7f5048075130 msgr2=0x7f5048075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:49:01.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.183+0000 7f504fc00700 1 -- 172.21.15.62:0/1638593243 shutdown_connections 2024-04-20T21:49:01.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.183+0000 7f504fc00700 1 -- 172.21.15.62:0/1638593243 wait complete. 2024-04-20T21:49:01.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.184+0000 7f504fc00700 1 Processor -- start 2024-04-20T21:49:01.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.184+0000 7f504fc00700 1 -- start start 2024-04-20T21:49:01.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.185+0000 7f504fc00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5048068540 0x7f5048198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:49:01.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.185+0000 7f504fc00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5048101070 0x7f50481985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:49:01.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.185+0000 7f504fc00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5048198c00 con 0x7f5048068540 2024-04-20T21:49:01.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.185+0000 7f504fc00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5048198d40 con 0x7f5048101070 2024-04-20T21:49:01.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.185+0000 7f504d200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5048068540 0x7f5048198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:49:01.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.185+0000 7f504d200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5048068540 0x7f5048198070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47934/0 (socket says 172.21.15.62:47934) 2024-04-20T21:49:01.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.185+0000 7f504d200700 1 -- 172.21.15.62:0/3747670068 learned_addr learned my addr 172.21.15.62:0/3747670068 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:49:01.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.186+0000 7f503fe00700 1 --2- 172.21.15.62:0/3747670068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5048101070 0x7f50481985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:49:01.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.186+0000 7f504d200700 1 -- 172.21.15.62:0/3747670068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5048101070 msgr2=0x7f50481985b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:49:01.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.186+0000 7f504d200700 1 --2- 172.21.15.62:0/3747670068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5048101070 0x7f50481985b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:49:01.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.186+0000 7f504d200700 1 -- 172.21.15.62:0/3747670068 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50300095f0 con 0x7f5048068540 2024-04-20T21:49:01.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.186+0000 7f503fe00700 1 --2- 172.21.15.62:0/3747670068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5048101070 0x7f50481985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T21:49:01.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.186+0000 7f504d200700 1 --2- 172.21.15.62:0/3747670068 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5048068540 0x7f5048198070 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f503800c930 tx=0x7f503800ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:49:01.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.186+0000 7f5046000700 1 -- 172.21.15.62:0/3747670068 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5038007ad0 con 0x7f5048068540 2024-04-20T21:49:01.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.186+0000 7f5046000700 1 -- 172.21.15.62:0/3747670068 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5038007c30 con 0x7f5048068540 2024-04-20T21:49:01.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.186+0000 7f504fc00700 1 -- 172.21.15.62:0/3747670068 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f504819cb90 con 0x7f5048068540 2024-04-20T21:49:01.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.186+0000 7f5046000700 1 -- 172.21.15.62:0/3747670068 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5038018610 con 0x7f5048068540 2024-04-20T21:49:01.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.187+0000 7f504fc00700 1 -- 172.21.15.62:0/3747670068 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f504819d0e0 con 0x7f5048068540 2024-04-20T21:49:01.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.188+0000 7f504fc00700 1 -- 172.21.15.62:0/3747670068 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f504804f2a0 con 0x7f5048068540 2024-04-20T21:49:01.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.189+0000 7f5046000700 1 -- 172.21.15.62:0/3747670068 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f503800f450 con 0x7f5048068540 2024-04-20T21:49:01.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.190+0000 7f5046000700 1 --2- 172.21.15.62:0/3747670068 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f502806ecc0 0x7f5028071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:49:01.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.190+0000 7f5046000700 1 -- 172.21.15.62:0/3747670068 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f503808f120 con 0x7f5048068540 2024-04-20T21:49:01.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.190+0000 7f503fe00700 1 --2- 172.21.15.62:0/3747670068 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f502806ecc0 0x7f5028071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:49:01.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.191+0000 7f503fe00700 1 --2- 172.21.15.62:0/3747670068 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f502806ecc0 0x7f5028071170 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f5030006010 tx=0x7f503000b540 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:49:01.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.195+0000 7f5046000700 1 -- 172.21.15.62:0/3747670068 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f5038057e30 con 0x7f5048068540 2024-04-20T21:49:01.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.520+0000 7f504fc00700 1 -- 172.21.15.62:0/3747670068 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f50481993d0 con 0x7f5048068540 2024-04-20T21:49:01.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.521+0000 7f5046000700 1 -- 172.21.15.62:0/3747670068 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7f503805b4f0 con 0x7f5048068540 2024-04-20T21:49:01.520 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:49:01.520 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T21:49:01.520 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:49:01.520 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:49:01.520 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T21:49:01.520 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:49:01.520 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:49:01.521 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T21:49:01.521 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 6 2024-04-20T21:49:01.521 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:49:01.521 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T21:49:01.521 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 2, 2024-04-20T21:49:01.521 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:49:01.521 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:49:01.521 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T21:49:01.521 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 2, 2024-04-20T21:49:01.521 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 12 2024-04-20T21:49:01.521 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T21:49:01.521 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:49:01.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.524+0000 7f504fc00700 1 -- 172.21.15.62:0/3747670068 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f502806ecc0 msgr2=0x7f5028071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:49:01.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.524+0000 7f504fc00700 1 --2- 172.21.15.62:0/3747670068 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f502806ecc0 0x7f5028071170 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f5030006010 tx=0x7f503000b540 comp rx=0 tx=0).stop 2024-04-20T21:49:01.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.524+0000 7f504fc00700 1 -- 172.21.15.62:0/3747670068 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5048068540 msgr2=0x7f5048198070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:49:01.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.524+0000 7f504fc00700 1 --2- 172.21.15.62:0/3747670068 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5048068540 0x7f5048198070 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f503800c930 tx=0x7f503800ccf0 comp rx=0 tx=0).stop 2024-04-20T21:49:01.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.524+0000 7f504fc00700 1 -- 172.21.15.62:0/3747670068 shutdown_connections 2024-04-20T21:49:01.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.524+0000 7f504fc00700 1 --2- 172.21.15.62:0/3747670068 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f502806ecc0 0x7f5028071170 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:49:01.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.524+0000 7f504fc00700 1 --2- 172.21.15.62:0/3747670068 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5048068540 0x7f5048198070 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:49:01.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.524+0000 7f504fc00700 1 --2- 172.21.15.62:0/3747670068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5048101070 0x7f50481985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:49:01.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.524+0000 7f504fc00700 1 -- 172.21.15.62:0/3747670068 >> 172.21.15.62:0/3747670068 conn(0x7f5048075130 msgr2=0x7f50480fdb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:49:01.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.524+0000 7f504fc00700 1 -- 172.21.15.62:0/3747670068 shutdown_connections 2024-04-20T21:49:01.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.524+0000 7f504fc00700 1 -- 172.21.15.62:0/3747670068 wait complete. 2024-04-20T21:49:01.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.690+0000 7ff1c1c00700 1 -- 172.21.15.62:0/540503719 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1bc068540 msgr2=0x7ff1bc0689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:49:01.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.690+0000 7ff1c1c00700 1 --2- 172.21.15.62:0/540503719 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1bc068540 0x7ff1bc0689b0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7ff1a4009910 tx=0x7ff1a4009c20 comp rx=0 tx=0).stop 2024-04-20T21:49:01.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.690+0000 7ff1c1c00700 1 -- 172.21.15.62:0/540503719 shutdown_connections 2024-04-20T21:49:01.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.690+0000 7ff1c1c00700 1 --2- 172.21.15.62:0/540503719 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1bc068540 0x7ff1bc0689b0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:49:01.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.690+0000 7ff1c1c00700 1 --2- 172.21.15.62:0/540503719 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1bc106320 0x7ff1bc1066f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:49:01.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.690+0000 7ff1c1c00700 1 -- 172.21.15.62:0/540503719 >> 172.21.15.62:0/540503719 conn(0x7ff1bc075130 msgr2=0x7ff1bc075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:49:01.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.690+0000 7ff1c1c00700 1 -- 172.21.15.62:0/540503719 shutdown_connections 2024-04-20T21:49:01.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.691+0000 7ff1c1c00700 1 -- 172.21.15.62:0/540503719 wait complete. 2024-04-20T21:49:01.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.692+0000 7ff1c1c00700 1 Processor -- start 2024-04-20T21:49:01.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.692+0000 7ff1c1c00700 1 -- start start 2024-04-20T21:49:01.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.692+0000 7ff1c1c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1bc068540 0x7ff1bc1980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:49:01.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.692+0000 7ff1c1c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1bc106320 0x7ff1bc1985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:49:01.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.692+0000 7ff1c1c00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1bc198cc0 con 0x7ff1bc106320 2024-04-20T21:49:01.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.692+0000 7ff1c1c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1bc19ca50 con 0x7ff1bc068540 2024-04-20T21:49:01.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.692+0000 7ff1b3e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1bc106320 0x7ff1bc1985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:49:01.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.692+0000 7ff1b3e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1bc106320 0x7ff1bc1985e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55780/0 (socket says 172.21.15.62:55780) 2024-04-20T21:49:01.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.693+0000 7ff1b3e00700 1 -- 172.21.15.62:0/4267057281 learned_addr learned my addr 172.21.15.62:0/4267057281 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:49:01.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.693+0000 7ff1baa00700 1 --2- 172.21.15.62:0/4267057281 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1bc068540 0x7ff1bc1980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:49:01.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.693+0000 7ff1b3e00700 1 -- 172.21.15.62:0/4267057281 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1bc068540 msgr2=0x7ff1bc1980a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:49:01.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.693+0000 7ff1b3e00700 1 --2- 172.21.15.62:0/4267057281 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1bc068540 0x7ff1bc1980a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:49:01.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.693+0000 7ff1b3e00700 1 -- 172.21.15.62:0/4267057281 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1a40095f0 con 0x7ff1bc106320 2024-04-20T21:49:01.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.693+0000 7ff1baa00700 1 --2- 172.21.15.62:0/4267057281 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1bc068540 0x7ff1bc1980a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T21:49:01.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.693+0000 7ff1b3e00700 1 --2- 172.21.15.62:0/4267057281 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1bc106320 0x7ff1bc1985e0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7ff1a400b5c0 tx=0x7ff1a400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:49:01.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.694+0000 7ff1b3400700 1 -- 172.21.15.62:0/4267057281 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff1a401d070 con 0x7ff1bc106320 2024-04-20T21:49:01.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.694+0000 7ff1b3400700 1 -- 172.21.15.62:0/4267057281 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff1a4005e00 con 0x7ff1bc106320 2024-04-20T21:49:01.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.694+0000 7ff1b3400700 1 -- 172.21.15.62:0/4267057281 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff1a4017400 con 0x7ff1bc106320 2024-04-20T21:49:01.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.694+0000 7ff1c1c00700 1 -- 172.21.15.62:0/4267057281 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1bc19ccd0 con 0x7ff1bc106320 2024-04-20T21:49:01.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.694+0000 7ff1c1c00700 1 -- 172.21.15.62:0/4267057281 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1bc19d1c0 con 0x7ff1bc106320 2024-04-20T21:49:01.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.695+0000 7ff1c1c00700 1 -- 172.21.15.62:0/4267057281 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1bc04f2a0 con 0x7ff1bc106320 2024-04-20T21:49:01.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.696+0000 7ff1b3400700 1 -- 172.21.15.62:0/4267057281 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff1a400f460 con 0x7ff1bc106320 2024-04-20T21:49:01.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.697+0000 7ff1b3400700 1 --2- 172.21.15.62:0/4267057281 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff1a006ecc0 0x7ff1a0071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:49:01.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.697+0000 7ff1b3400700 1 -- 172.21.15.62:0/4267057281 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7ff1a4090370 con 0x7ff1bc106320 2024-04-20T21:49:01.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.697+0000 7ff1baa00700 1 --2- 172.21.15.62:0/4267057281 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff1a006ecc0 0x7ff1a0071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:49:01.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.698+0000 7ff1baa00700 1 --2- 172.21.15.62:0/4267057281 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff1a006ecc0 0x7ff1a0071170 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff1ac005fd0 tx=0x7ff1ac005ee0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:49:01.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.702+0000 7ff1b3400700 1 -- 172.21.15.62:0/4267057281 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7ff1a4059100 con 0x7ff1bc106320 2024-04-20T21:49:01.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.973+0000 7ff1c1c00700 1 -- 172.21.15.62:0/4267057281 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7ff1bc199400 con 0x7ff1bc106320 2024-04-20T21:49:01.973 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:49:01 smithi062 ceph-mon[76322]: from='client.34788 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:49:01.973 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:49:01 smithi062 ceph-mon[76322]: pgmap v411: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:49:01.973 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:49:01 smithi062 ceph-mon[76322]: from='client.34792 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:49:01.973 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:49:01 smithi062 ceph-mon[76322]: from='client.34796 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:49:01.973 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:49:01 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/3747670068' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:49:01.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.975+0000 7ff1b3400700 1 -- 172.21.15.62:0/4267057281 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 25 v25) v1 ==== 76+0+1887 (secure 0 0 0) 0x7ff1a405c7c0 con 0x7ff1bc106320 2024-04-20T21:49:01.975 INFO:teuthology.orchestra.run.smithi062.stdout:e25 2024-04-20T21:49:01.975 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T21:49:01.975 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T21:49:01.975 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T21:49:01.975 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:49:01.975 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T21:49:01.975 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T21:49:01.975 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 20 2024-04-20T21:49:01.975 INFO:teuthology.orchestra.run.smithi062.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-20T21:49:01.975 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T21:49:01.975 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T21:42:01.559061+0000 2024-04-20T21:49:01.975 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T21:49:01.975 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T21:49:01.976 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T21:49:01.976 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T21:49:01.976 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T21:49:01.976 INFO:teuthology.orchestra.run.smithi062.stdout:max_xattr_size 65536 2024-04-20T21:49:01.976 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T21:49:01.976 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T21:49:01.976 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 81 2024-04-20T21:49:01.976 INFO:teuthology.orchestra.run.smithi062.stdout:snap_rstat 1 2024-04-20T21:49:01.976 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T21:49:01.976 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 2 2024-04-20T21:49:01.976 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1 2024-04-20T21:49:01.976 INFO:teuthology.orchestra.run.smithi062.stdout:up {} 2024-04-20T21:49:01.976 INFO:teuthology.orchestra.run.smithi062.stdout:failed 0,1 2024-04-20T21:49:01.976 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T21:49:01.976 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T21:49:01.977 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T21:49:01.977 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T21:49:01.977 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data enabled 2024-04-20T21:49:01.977 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T21:49:01.977 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T21:49:01.977 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 1 2024-04-20T21:49:01.977 INFO:teuthology.orchestra.run.smithi062.stdout:qdb_cluster leader: 0 members: 2024-04-20T21:49:01.977 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:49:01.977 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:49:01.977 INFO:teuthology.orchestra.run.smithi062.stdout:Standby daemons: 2024-04-20T21:49:01.977 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:49:01.977 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ryqblb{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6826/4113916205,v1:172.21.15.62:6827/4113916205] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T21:49:01.977 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ozizwi{-1:34476} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6828/3801975165,v1:172.21.15.62:6829/3801975165] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T21:49:01.977 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.xbgspw{-1:44285} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6826/3996040526,v1:172.21.15.189:6827/3996040526] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:49:01.977 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.dvimer{-1:44289} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6824/1743111843,v1:172.21.15.189:6825/1743111843] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:49:01.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.977+0000 7ff1c1c00700 1 -- 172.21.15.62:0/4267057281 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff1a006ecc0 msgr2=0x7ff1a0071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:49:01.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.977+0000 7ff1c1c00700 1 --2- 172.21.15.62:0/4267057281 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff1a006ecc0 0x7ff1a0071170 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff1ac005fd0 tx=0x7ff1ac005ee0 comp rx=0 tx=0).stop 2024-04-20T21:49:01.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.977+0000 7ff1c1c00700 1 -- 172.21.15.62:0/4267057281 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1bc106320 msgr2=0x7ff1bc1985e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:49:01.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.977+0000 7ff1c1c00700 1 --2- 172.21.15.62:0/4267057281 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1bc106320 0x7ff1bc1985e0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7ff1a400b5c0 tx=0x7ff1a400be20 comp rx=0 tx=0).stop 2024-04-20T21:49:01.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.978+0000 7ff1c1c00700 1 -- 172.21.15.62:0/4267057281 shutdown_connections 2024-04-20T21:49:01.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.978+0000 7ff1c1c00700 1 --2- 172.21.15.62:0/4267057281 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff1a006ecc0 0x7ff1a0071170 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:49:01.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.978+0000 7ff1c1c00700 1 --2- 172.21.15.62:0/4267057281 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1bc068540 0x7ff1bc1980a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:49:01.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.978+0000 7ff1c1c00700 1 --2- 172.21.15.62:0/4267057281 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1bc106320 0x7ff1bc1985e0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:49:01.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.978+0000 7ff1c1c00700 1 -- 172.21.15.62:0/4267057281 >> 172.21.15.62:0/4267057281 conn(0x7ff1bc075130 msgr2=0x7ff1bc0fe9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:49:01.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.978+0000 7ff1c1c00700 1 -- 172.21.15.62:0/4267057281 shutdown_connections 2024-04-20T21:49:01.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:01.978+0000 7ff1c1c00700 1 -- 172.21.15.62:0/4267057281 wait complete. 2024-04-20T21:49:01.979 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 25 2024-04-20T21:49:01.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:49:01 smithi189 ceph-mon[57022]: from='client.34788 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:49:01.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:49:01 smithi189 ceph-mon[57022]: pgmap v411: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:49:01.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:49:01 smithi189 ceph-mon[57022]: from='client.34792 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:49:01.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:49:01 smithi189 ceph-mon[57022]: from='client.34796 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:49:01.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:49:01 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/3747670068' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:49:02.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:02.130+0000 7fce25200700 1 -- 172.21.15.62:0/531559042 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce20101070 msgr2=0x7fce20101440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:49:02.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:02.130+0000 7fce25200700 1 --2- 172.21.15.62:0/531559042 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce20101070 0x7fce20101440 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fce10009910 tx=0x7fce10009c20 comp rx=0 tx=0).stop 2024-04-20T21:49:02.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:02.130+0000 7fce25200700 1 -- 172.21.15.62:0/531559042 shutdown_connections 2024-04-20T21:49:02.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:02.130+0000 7fce25200700 1 --2- 172.21.15.62:0/531559042 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce20068540 0x7fce200689b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:49:02.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:02.130+0000 7fce25200700 1 --2- 172.21.15.62:0/531559042 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce20101070 0x7fce20101440 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:49:02.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:02.130+0000 7fce25200700 1 -- 172.21.15.62:0/531559042 >> 172.21.15.62:0/531559042 conn(0x7fce20075130 msgr2=0x7fce20075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:49:02.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:02.130+0000 7fce25200700 1 -- 172.21.15.62:0/531559042 shutdown_connections 2024-04-20T21:49:02.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:02.130+0000 7fce25200700 1 -- 172.21.15.62:0/531559042 wait complete. 2024-04-20T21:49:02.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:02.131+0000 7fce25200700 1 Processor -- start 2024-04-20T21:49:02.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:02.131+0000 7fce25200700 1 -- start start 2024-04-20T21:49:02.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:02.132+0000 7fce25200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce20068540 0x7fce20198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:49:02.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:02.132+0000 7fce25200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce20101070 0x7fce20198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:49:02.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:02.132+0000 7fce25200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce20198c70 con 0x7fce20068540 2024-04-20T21:49:02.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:02.132+0000 7fce25200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce2019ca00 con 0x7fce20101070 2024-04-20T21:49:02.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:02.132+0000 7fce1de00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce20068540 0x7fce20198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:49:02.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:02.132+0000 7fce1de00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce20068540 0x7fce20198050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55800/0 (socket says 172.21.15.62:55800) 2024-04-20T21:49:02.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:02.132+0000 7fce1de00700 1 -- 172.21.15.62:0/3570108395 learned_addr learned my addr 172.21.15.62:0/3570108395 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:49:02.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:02.133+0000 7fce1de00700 1 -- 172.21.15.62:0/3570108395 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce20101070 msgr2=0x7fce20198590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:49:02.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:02.133+0000 7fce1de00700 1 --2- 172.21.15.62:0/3570108395 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce20101070 0x7fce20198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:49:02.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:02.133+0000 7fce1de00700 1 -- 172.21.15.62:0/3570108395 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce100095f0 con 0x7fce20068540 2024-04-20T21:49:02.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:02.133+0000 7fce1de00700 1 --2- 172.21.15.62:0/3570108395 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce20068540 0x7fce20198050 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fce1000b5c0 tx=0x7fce1000be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:49:02.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:02.133+0000 7fce16a00700 1 -- 172.21.15.62:0/3570108395 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce1001d070 con 0x7fce20068540 2024-04-20T21:49:02.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:02.133+0000 7fce16a00700 1 -- 172.21.15.62:0/3570108395 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fce1000f460 con 0x7fce20068540 2024-04-20T21:49:02.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:02.133+0000 7fce25200700 1 -- 172.21.15.62:0/3570108395 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce2019cc80 con 0x7fce20068540 2024-04-20T21:49:02.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:02.133+0000 7fce16a00700 1 -- 172.21.15.62:0/3570108395 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce10017610 con 0x7fce20068540 2024-04-20T21:49:02.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:02.134+0000 7fce25200700 1 -- 172.21.15.62:0/3570108395 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce2019d170 con 0x7fce20068540 2024-04-20T21:49:02.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:02.135+0000 7fce25200700 1 -- 172.21.15.62:0/3570108395 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce2004f2a0 con 0x7fce20068540 2024-04-20T21:49:02.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:02.136+0000 7fce16a00700 1 -- 172.21.15.62:0/3570108395 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fce1000f5d0 con 0x7fce20068540 2024-04-20T21:49:02.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:02.137+0000 7fce16a00700 1 --2- 172.21.15.62:0/3570108395 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fce0406ecc0 0x7fce04071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:49:02.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:02.137+0000 7fce16a00700 1 -- 172.21.15.62:0/3570108395 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fce10090510 con 0x7fce20068540 2024-04-20T21:49:02.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:02.137+0000 7fce1d400700 1 --2- 172.21.15.62:0/3570108395 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fce0406ecc0 0x7fce04071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:49:02.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:02.138+0000 7fce1d400700 1 --2- 172.21.15.62:0/3570108395 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fce0406ecc0 0x7fce04071170 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fce20199670 tx=0x7fce0c005c80 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:49:02.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:02.142+0000 7fce16a00700 1 -- 172.21.15.62:0/3570108395 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fce10059220 con 0x7fce20068540 2024-04-20T21:49:02.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:02.397+0000 7fce25200700 1 -- 172.21.15.62:0/3570108395 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fce2019d600 con 0x7fce0406ecc0 2024-04-20T21:49:02.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:02.400+0000 7fce16a00700 1 -- 172.21.15.62:0/3570108395 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fce2019d600 con 0x7fce0406ecc0 2024-04-20T21:49:02.399 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:49:02.399 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T21:49:02.399 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T21:49:02.399 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-20T21:49:02.399 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T21:49:02.399 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr", 2024-04-20T21:49:02.399 INFO:teuthology.orchestra.run.smithi062.stdout: "crash", 2024-04-20T21:49:02.399 INFO:teuthology.orchestra.run.smithi062.stdout: "mon", 2024-04-20T21:49:02.399 INFO:teuthology.orchestra.run.smithi062.stdout: "osd" 2024-04-20T21:49:02.400 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T21:49:02.400 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "14/23 daemons upgraded", 2024-04-20T21:49:02.400 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading mds daemons", 2024-04-20T21:49:02.400 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T21:49:02.400 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:49:02.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:02.403+0000 7fce25200700 1 -- 172.21.15.62:0/3570108395 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fce0406ecc0 msgr2=0x7fce04071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:49:02.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:02.403+0000 7fce25200700 1 --2- 172.21.15.62:0/3570108395 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fce0406ecc0 0x7fce04071170 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fce20199670 tx=0x7fce0c005c80 comp rx=0 tx=0).stop 2024-04-20T21:49:02.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:02.403+0000 7fce25200700 1 -- 172.21.15.62:0/3570108395 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce20068540 msgr2=0x7fce20198050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:49:02.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:02.403+0000 7fce25200700 1 --2- 172.21.15.62:0/3570108395 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce20068540 0x7fce20198050 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fce1000b5c0 tx=0x7fce1000be00 comp rx=0 tx=0).stop 2024-04-20T21:49:02.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:02.403+0000 7fce25200700 1 -- 172.21.15.62:0/3570108395 shutdown_connections 2024-04-20T21:49:02.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:02.403+0000 7fce25200700 1 --2- 172.21.15.62:0/3570108395 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fce0406ecc0 0x7fce04071170 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:49:02.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:02.403+0000 7fce25200700 1 --2- 172.21.15.62:0/3570108395 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce20068540 0x7fce20198050 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:49:02.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:02.403+0000 7fce25200700 1 --2- 172.21.15.62:0/3570108395 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce20101070 0x7fce20198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:49:02.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:02.403+0000 7fce25200700 1 -- 172.21.15.62:0/3570108395 >> 172.21.15.62:0/3570108395 conn(0x7fce20075130 msgr2=0x7fce200fdb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:49:02.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:02.403+0000 7fce25200700 1 -- 172.21.15.62:0/3570108395 shutdown_connections 2024-04-20T21:49:02.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:02.404+0000 7fce25200700 1 -- 172.21.15.62:0/3570108395 wait complete. 2024-04-20T21:49:02.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:02.558+0000 7f935ce00700 1 -- 172.21.15.62:0/1223177854 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9358102470 msgr2=0x7f93581028e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:49:02.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:02.558+0000 7f935ce00700 1 --2- 172.21.15.62:0/1223177854 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9358102470 0x7f93581028e0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f9340009910 tx=0x7f9340009c20 comp rx=0 tx=0).stop 2024-04-20T21:49:02.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:02.558+0000 7f935ce00700 1 -- 172.21.15.62:0/1223177854 shutdown_connections 2024-04-20T21:49:02.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:02.558+0000 7f935ce00700 1 --2- 172.21.15.62:0/1223177854 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9358102470 0x7f93581028e0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:49:02.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:02.558+0000 7f935ce00700 1 --2- 172.21.15.62:0/1223177854 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9358108470 0x7f9358108840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:49:02.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:02.558+0000 7f935ce00700 1 -- 172.21.15.62:0/1223177854 >> 172.21.15.62:0/1223177854 conn(0x7f93580fdf90 msgr2=0x7f93581003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:49:02.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:02.558+0000 7f935ce00700 1 -- 172.21.15.62:0/1223177854 shutdown_connections 2024-04-20T21:49:02.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:02.558+0000 7f935ce00700 1 -- 172.21.15.62:0/1223177854 wait complete. 2024-04-20T21:49:02.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:02.559+0000 7f935ce00700 1 Processor -- start 2024-04-20T21:49:02.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:02.559+0000 7f935ce00700 1 -- start start 2024-04-20T21:49:02.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:02.559+0000 7f935ce00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9358102470 0x7f93581981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:49:02.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:02.559+0000 7f935ce00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9358108470 0x7f9358198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:49:02.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:02.560+0000 7f935ce00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9358198e10 con 0x7f9358108470 2024-04-20T21:49:02.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:02.560+0000 7f935ce00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f935819cba0 con 0x7f9358102470 2024-04-20T21:49:02.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:02.560+0000 7f934fe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9358108470 0x7f9358198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:49:02.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:02.560+0000 7f934fe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9358108470 0x7f9358198730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55804/0 (socket says 172.21.15.62:55804) 2024-04-20T21:49:02.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:02.560+0000 7f934fe00700 1 -- 172.21.15.62:0/299401061 learned_addr learned my addr 172.21.15.62:0/299401061 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:49:02.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:02.560+0000 7f9355e00700 1 --2- 172.21.15.62:0/299401061 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9358102470 0x7f93581981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:49:02.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:02.560+0000 7f934fe00700 1 -- 172.21.15.62:0/299401061 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9358102470 msgr2=0x7f93581981f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:49:02.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:02.560+0000 7f934fe00700 1 --2- 172.21.15.62:0/299401061 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9358102470 0x7f93581981f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:49:02.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:02.560+0000 7f934fe00700 1 -- 172.21.15.62:0/299401061 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93400095f0 con 0x7f9358108470 2024-04-20T21:49:02.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:02.561+0000 7f934fe00700 1 --2- 172.21.15.62:0/299401061 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9358108470 0x7f9358198730 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f9340006010 tx=0x7f934000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:49:02.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:02.561+0000 7f934ea00700 1 -- 172.21.15.62:0/299401061 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f934001d070 con 0x7f9358108470 2024-04-20T21:49:02.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:02.561+0000 7f934ea00700 1 -- 172.21.15.62:0/299401061 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f934000f460 con 0x7f9358108470 2024-04-20T21:49:02.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:02.561+0000 7f934ea00700 1 -- 172.21.15.62:0/299401061 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9340017630 con 0x7f9358108470 2024-04-20T21:49:02.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:02.561+0000 7f935ce00700 1 -- 172.21.15.62:0/299401061 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f935819ce20 con 0x7f9358108470 2024-04-20T21:49:02.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:02.561+0000 7f935ce00700 1 -- 172.21.15.62:0/299401061 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f935819d2c0 con 0x7f9358108470 2024-04-20T21:49:02.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:02.562+0000 7f935ce00700 1 -- 172.21.15.62:0/299401061 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f935810a970 con 0x7f9358108470 2024-04-20T21:49:02.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:02.564+0000 7f934ea00700 1 -- 172.21.15.62:0/299401061 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f934000f5d0 con 0x7f9358108470 2024-04-20T21:49:02.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:02.565+0000 7f934ea00700 1 --2- 172.21.15.62:0/299401061 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f933c06ecc0 0x7f933c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:49:02.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:02.565+0000 7f934ea00700 1 -- 172.21.15.62:0/299401061 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f9340090330 con 0x7f9358108470 2024-04-20T21:49:02.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:02.565+0000 7f9355e00700 1 --2- 172.21.15.62:0/299401061 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f933c06ecc0 0x7f933c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:49:02.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:02.565+0000 7f9355e00700 1 --2- 172.21.15.62:0/299401061 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f933c06ecc0 0x7f933c071170 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f9348005950 tx=0x7f93480058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:49:02.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:02.570+0000 7f934ea00700 1 -- 172.21.15.62:0/299401061 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f93400590c0 con 0x7f9358108470 2024-04-20T21:49:02.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:02.889+0000 7f935ce00700 1 -- 172.21.15.62:0/299401061 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f935810a350 con 0x7f9358108470 2024-04-20T21:49:02.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:02.890+0000 7f934ea00700 1 -- 172.21.15.62:0/299401061 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7f934005c780 con 0x7f9358108470 2024-04-20T21:49:02.889 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-20T21:49:02.889 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-20T21:49:02.889 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is degraded 2024-04-20T21:49:02.889 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T21:49:02.889 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T21:49:02.889 INFO:teuthology.orchestra.run.smithi062.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-20T21:49:02.889 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-20T21:49:02.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:02.892+0000 7f935ce00700 1 -- 172.21.15.62:0/299401061 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f933c06ecc0 msgr2=0x7f933c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:49:02.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:02.892+0000 7f935ce00700 1 --2- 172.21.15.62:0/299401061 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f933c06ecc0 0x7f933c071170 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f9348005950 tx=0x7f93480058e0 comp rx=0 tx=0).stop 2024-04-20T21:49:02.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:02.892+0000 7f935ce00700 1 -- 172.21.15.62:0/299401061 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9358108470 msgr2=0x7f9358198730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:49:02.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:02.892+0000 7f935ce00700 1 --2- 172.21.15.62:0/299401061 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9358108470 0x7f9358198730 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f9340006010 tx=0x7f934000be20 comp rx=0 tx=0).stop 2024-04-20T21:49:02.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:02.893+0000 7f935ce00700 1 -- 172.21.15.62:0/299401061 shutdown_connections 2024-04-20T21:49:02.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:02.893+0000 7f935ce00700 1 --2- 172.21.15.62:0/299401061 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f933c06ecc0 0x7f933c071170 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:49:02.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:02.893+0000 7f935ce00700 1 --2- 172.21.15.62:0/299401061 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9358102470 0x7f93581981f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:49:02.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:02.893+0000 7f935ce00700 1 --2- 172.21.15.62:0/299401061 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9358108470 0x7f9358198730 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:49:02.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:02.893+0000 7f935ce00700 1 -- 172.21.15.62:0/299401061 >> 172.21.15.62:0/299401061 conn(0x7f93580fdf90 msgr2=0x7f93580ff3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:49:02.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:02.893+0000 7f935ce00700 1 -- 172.21.15.62:0/299401061 shutdown_connections 2024-04-20T21:49:02.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:02.893+0000 7f935ce00700 1 -- 172.21.15.62:0/299401061 wait complete. 2024-04-20T21:49:02.986 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:49:02 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/4267057281' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T21:49:03.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:49:02 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/4267057281' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T21:49:03.986 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:49:03 smithi189 ceph-mon[57022]: from='client.34808 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:49:03.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:49:03 smithi189 ceph-mon[57022]: pgmap v412: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:49:03.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:49:03 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/299401061' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T21:49:04.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:49:03 smithi062 ceph-mon[76322]: from='client.34808 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:49:04.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:49:03 smithi062 ceph-mon[76322]: pgmap v412: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:49:04.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:49:03 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/299401061' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T21:49:04.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:49:04 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:49:05.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:49:04 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:49:05.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:49:05 smithi189 ceph-mon[57022]: pgmap v413: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:49:06.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:49:05 smithi062 ceph-mon[76322]: pgmap v413: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:49:07.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:49:07 smithi189 ceph-mon[57022]: pgmap v414: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:49:08.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:49:07 smithi062 ceph-mon[76322]: pgmap v414: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:49:09.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:49:09 smithi189 ceph-mon[57022]: pgmap v415: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:49:10.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:49:09 smithi062 ceph-mon[76322]: pgmap v415: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:49:11.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:49:11 smithi189 ceph-mon[57022]: pgmap v416: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:49:12.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:49:11 smithi062 ceph-mon[76322]: pgmap v416: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:49:13.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:49:13 smithi189 ceph-mon[57022]: pgmap v417: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:49:14.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:49:13 smithi062 ceph-mon[76322]: pgmap v417: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:49:15.986 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:49:15 smithi189 ceph-mon[57022]: pgmap v418: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:49:16.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:49:15 smithi062 ceph-mon[76322]: pgmap v418: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:49:18.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:49:17 smithi062 ceph-mon[76322]: pgmap v419: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:49:18.236 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:49:17 smithi189 ceph-mon[57022]: pgmap v419: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:49:19.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:49:19 smithi189 ceph-mon[57022]: pgmap v420: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:49:19.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:49:19 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:49:20.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:49:19 smithi062 ceph-mon[76322]: pgmap v420: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:49:20.721 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:49:19 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:49:22.014 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:49:21 smithi062 ceph-mon[76322]: pgmap v421: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:49:22.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:49:21 smithi189 ceph-mon[57022]: pgmap v421: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:49:24.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:49:23 smithi062 ceph-mon[76322]: pgmap v422: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:49:24.236 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:49:23 smithi189 ceph-mon[57022]: pgmap v422: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:49:26.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:49:25 smithi062 ceph-mon[76322]: pgmap v423: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:49:26.236 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:49:25 smithi189 ceph-mon[57022]: pgmap v423: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:49:28.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:49:27 smithi062 ceph-mon[76322]: pgmap v424: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:49:28.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:49:27 smithi189 ceph-mon[57022]: pgmap v424: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:49:30.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:49:29 smithi062 ceph-mon[76322]: pgmap v425: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:49:30.039 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:49:29 smithi189 ceph-mon[57022]: pgmap v425: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:49:32.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:49:31 smithi189 ceph-mon[57022]: pgmap v426: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:49:32.284 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:49:31 smithi062 ceph-mon[76322]: pgmap v426: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:49:33.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.072+0000 7f172ca00700 1 -- 172.21.15.62:0/3271839971 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1728102e70 msgr2=0x7f1728106e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:49:33.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.072+0000 7f172ca00700 1 --2- 172.21.15.62:0/3271839971 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1728102e70 0x7f1728106e40 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f1714009910 tx=0x7f1714009c20 comp rx=0 tx=0).stop 2024-04-20T21:49:33.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.072+0000 7f172ca00700 1 -- 172.21.15.62:0/3271839971 shutdown_connections 2024-04-20T21:49:33.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.072+0000 7f172ca00700 1 --2- 172.21.15.62:0/3271839971 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1728102e70 0x7f1728106e40 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:49:33.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.072+0000 7f172ca00700 1 --2- 172.21.15.62:0/3271839971 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f17281024d0 0x7f17281028a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:49:33.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.072+0000 7f172ca00700 1 -- 172.21.15.62:0/3271839971 >> 172.21.15.62:0/3271839971 conn(0x7f17280fdd90 msgr2=0x7f17281001a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:49:33.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.073+0000 7f172ca00700 1 -- 172.21.15.62:0/3271839971 shutdown_connections 2024-04-20T21:49:33.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.073+0000 7f172ca00700 1 -- 172.21.15.62:0/3271839971 wait complete. 2024-04-20T21:49:33.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.074+0000 7f172ca00700 1 Processor -- start 2024-04-20T21:49:33.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.074+0000 7f172ca00700 1 -- start start 2024-04-20T21:49:33.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.074+0000 7f172ca00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f17281024d0 0x7f1728197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:49:33.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.074+0000 7f172ca00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1728102e70 0x7f17281984c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:49:33.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.074+0000 7f172ca00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1728198ba0 con 0x7f1728102e70 2024-04-20T21:49:33.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.074+0000 7f172ca00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f172819c930 con 0x7f17281024d0 2024-04-20T21:49:33.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.075+0000 7f1726a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1728102e70 0x7f17281984c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:49:33.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.075+0000 7f1726a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1728102e70 0x7f17281984c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40906/0 (socket says 172.21.15.62:40906) 2024-04-20T21:49:33.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.075+0000 7f1726a00700 1 -- 172.21.15.62:0/362507948 learned_addr learned my addr 172.21.15.62:0/362507948 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:49:33.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.075+0000 7f1727400700 1 --2- 172.21.15.62:0/362507948 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f17281024d0 0x7f1728197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:49:33.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.075+0000 7f1726a00700 1 -- 172.21.15.62:0/362507948 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f17281024d0 msgr2=0x7f1728197f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:49:33.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.075+0000 7f1726a00700 1 --2- 172.21.15.62:0/362507948 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f17281024d0 0x7f1728197f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:49:33.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.075+0000 7f1726a00700 1 -- 172.21.15.62:0/362507948 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17140095f0 con 0x7f1728102e70 2024-04-20T21:49:33.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.075+0000 7f1727400700 1 --2- 172.21.15.62:0/362507948 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f17281024d0 0x7f1728197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T21:49:33.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.076+0000 7f1726a00700 1 --2- 172.21.15.62:0/362507948 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1728102e70 0x7f17281984c0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f171400bda0 tx=0x7f171400be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:49:33.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.076+0000 7f171fe00700 1 -- 172.21.15.62:0/362507948 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f171401d070 con 0x7f1728102e70 2024-04-20T21:49:33.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.076+0000 7f171fe00700 1 -- 172.21.15.62:0/362507948 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1714004fb0 con 0x7f1728102e70 2024-04-20T21:49:33.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.076+0000 7f171fe00700 1 -- 172.21.15.62:0/362507948 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f171400f460 con 0x7f1728102e70 2024-04-20T21:49:33.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.076+0000 7f172ca00700 1 -- 172.21.15.62:0/362507948 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f172819cbb0 con 0x7f1728102e70 2024-04-20T21:49:33.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.076+0000 7f172ca00700 1 -- 172.21.15.62:0/362507948 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f172819d0a0 con 0x7f1728102e70 2024-04-20T21:49:33.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.077+0000 7f172ca00700 1 -- 172.21.15.62:0/362507948 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f172810a790 con 0x7f1728102e70 2024-04-20T21:49:33.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.079+0000 7f171fe00700 1 -- 172.21.15.62:0/362507948 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f1714022800 con 0x7f1728102e70 2024-04-20T21:49:33.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.080+0000 7f171fe00700 1 --2- 172.21.15.62:0/362507948 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f170c06ecc0 0x7f170c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:49:33.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.080+0000 7f171fe00700 1 -- 172.21.15.62:0/362507948 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f1714090390 con 0x7f1728102e70 2024-04-20T21:49:33.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.081+0000 7f1727400700 1 --2- 172.21.15.62:0/362507948 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f170c06ecc0 0x7f170c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:49:33.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.081+0000 7f1727400700 1 --2- 172.21.15.62:0/362507948 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f170c06ecc0 0x7f170c071170 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f1718007900 tx=0x7f1718008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:49:33.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.086+0000 7f171fe00700 1 -- 172.21.15.62:0/362507948 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f17140592f0 con 0x7f1728102e70 2024-04-20T21:49:33.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.341+0000 7f172ca00700 1 -- 172.21.15.62:0/362507948 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f172802cd10 con 0x7f170c06ecc0 2024-04-20T21:49:33.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.344+0000 7f171fe00700 1 -- 172.21.15.62:0/362507948 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f172802cd10 con 0x7f170c06ecc0 2024-04-20T21:49:33.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.347+0000 7f172ca00700 1 -- 172.21.15.62:0/362507948 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f170c06ecc0 msgr2=0x7f170c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:49:33.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.347+0000 7f172ca00700 1 --2- 172.21.15.62:0/362507948 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f170c06ecc0 0x7f170c071170 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f1718007900 tx=0x7f1718008040 comp rx=0 tx=0).stop 2024-04-20T21:49:33.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.347+0000 7f172ca00700 1 -- 172.21.15.62:0/362507948 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1728102e70 msgr2=0x7f17281984c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:49:33.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.347+0000 7f172ca00700 1 --2- 172.21.15.62:0/362507948 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1728102e70 0x7f17281984c0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f171400bda0 tx=0x7f171400be80 comp rx=0 tx=0).stop 2024-04-20T21:49:33.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.347+0000 7f172ca00700 1 -- 172.21.15.62:0/362507948 shutdown_connections 2024-04-20T21:49:33.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.347+0000 7f172ca00700 1 --2- 172.21.15.62:0/362507948 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f170c06ecc0 0x7f170c071170 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:49:33.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.347+0000 7f172ca00700 1 --2- 172.21.15.62:0/362507948 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f17281024d0 0x7f1728197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:49:33.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.347+0000 7f172ca00700 1 --2- 172.21.15.62:0/362507948 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1728102e70 0x7f17281984c0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:49:33.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.347+0000 7f172ca00700 1 -- 172.21.15.62:0/362507948 >> 172.21.15.62:0/362507948 conn(0x7f17280fdd90 msgr2=0x7f17280ff4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:49:33.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.347+0000 7f172ca00700 1 -- 172.21.15.62:0/362507948 shutdown_connections 2024-04-20T21:49:33.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.347+0000 7f172ca00700 1 -- 172.21.15.62:0/362507948 wait complete. 2024-04-20T21:49:33.362 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T21:49:33.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.515+0000 7f396a600700 1 -- 172.21.15.62:0/1387462757 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39640fee90 msgr2=0x7f39640ff260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:49:33.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.515+0000 7f396a600700 1 --2- 172.21.15.62:0/1387462757 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39640fee90 0x7f39640ff260 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f3958009910 tx=0x7f3958009c20 comp rx=0 tx=0).stop 2024-04-20T21:49:33.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.515+0000 7f396a600700 1 -- 172.21.15.62:0/1387462757 shutdown_connections 2024-04-20T21:49:33.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.515+0000 7f396a600700 1 --2- 172.21.15.62:0/1387462757 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39640ff830 0x7f396410c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:49:33.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.515+0000 7f396a600700 1 --2- 172.21.15.62:0/1387462757 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39640fee90 0x7f39640ff260 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:49:33.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.515+0000 7f396a600700 1 -- 172.21.15.62:0/1387462757 >> 172.21.15.62:0/1387462757 conn(0x7f39640faad0 msgr2=0x7f39640fcee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:49:33.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.516+0000 7f396a600700 1 -- 172.21.15.62:0/1387462757 shutdown_connections 2024-04-20T21:49:33.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.516+0000 7f396a600700 1 -- 172.21.15.62:0/1387462757 wait complete. 2024-04-20T21:49:33.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.517+0000 7f396a600700 1 Processor -- start 2024-04-20T21:49:33.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.517+0000 7f396a600700 1 -- start start 2024-04-20T21:49:33.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.517+0000 7f396a600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39640fee90 0x7f3964197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:49:33.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.517+0000 7f396a600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39640ff830 0x7f3964198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:49:33.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.517+0000 7f396a600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3964198af0 con 0x7f39640ff830 2024-04-20T21:49:33.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.517+0000 7f396a600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f396419c880 con 0x7f39640fee90 2024-04-20T21:49:33.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.518+0000 7f3963e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39640ff830 0x7f3964198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:49:33.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.518+0000 7f3963e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39640ff830 0x7f3964198410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40912/0 (socket says 172.21.15.62:40912) 2024-04-20T21:49:33.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.518+0000 7f3963e00700 1 -- 172.21.15.62:0/2292767989 learned_addr learned my addr 172.21.15.62:0/2292767989 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:49:33.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.518+0000 7f3969200700 1 --2- 172.21.15.62:0/2292767989 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39640fee90 0x7f3964197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:49:33.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.518+0000 7f3963e00700 1 -- 172.21.15.62:0/2292767989 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39640fee90 msgr2=0x7f3964197ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:49:33.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.518+0000 7f3963e00700 1 --2- 172.21.15.62:0/2292767989 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39640fee90 0x7f3964197ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:49:33.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.518+0000 7f3963e00700 1 -- 172.21.15.62:0/2292767989 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39580095f0 con 0x7f39640ff830 2024-04-20T21:49:33.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.519+0000 7f3963e00700 1 --2- 172.21.15.62:0/2292767989 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39640ff830 0x7f3964198410 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f395000c8a0 tx=0x7f395000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:49:33.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.519+0000 7f3961600700 1 -- 172.21.15.62:0/2292767989 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f395000cea0 con 0x7f39640ff830 2024-04-20T21:49:33.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.519+0000 7f3961600700 1 -- 172.21.15.62:0/2292767989 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3950004830 con 0x7f39640ff830 2024-04-20T21:49:33.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.519+0000 7f3961600700 1 -- 172.21.15.62:0/2292767989 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f39500055b0 con 0x7f39640ff830 2024-04-20T21:49:33.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.519+0000 7f396a600700 1 -- 172.21.15.62:0/2292767989 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f396419cb60 con 0x7f39640ff830 2024-04-20T21:49:33.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.519+0000 7f396a600700 1 -- 172.21.15.62:0/2292767989 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f396419d0b0 con 0x7f39640ff830 2024-04-20T21:49:33.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.520+0000 7f396a600700 1 -- 172.21.15.62:0/2292767989 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3964109be0 con 0x7f39640ff830 2024-04-20T21:49:33.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.521+0000 7f3961600700 1 -- 172.21.15.62:0/2292767989 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f39500049a0 con 0x7f39640ff830 2024-04-20T21:49:33.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.522+0000 7f3961600700 1 --2- 172.21.15.62:0/2292767989 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f394c06ecc0 0x7f394c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:49:33.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.522+0000 7f3961600700 1 -- 172.21.15.62:0/2292767989 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f395008ebf0 con 0x7f39640ff830 2024-04-20T21:49:33.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.523+0000 7f3969200700 1 --2- 172.21.15.62:0/2292767989 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f394c06ecc0 0x7f394c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:49:33.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.523+0000 7f3969200700 1 --2- 172.21.15.62:0/2292767989 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f394c06ecc0 0x7f394c071170 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f3958006010 tx=0x7f395801a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:49:33.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.527+0000 7f3961600700 1 -- 172.21.15.62:0/2292767989 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f3950057900 con 0x7f39640ff830 2024-04-20T21:49:33.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.783+0000 7f396a600700 1 -- 172.21.15.62:0/2292767989 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3964199340 con 0x7f394c06ecc0 2024-04-20T21:49:33.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.787+0000 7f3961600700 1 -- 172.21.15.62:0/2292767989 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f3964199340 con 0x7f394c06ecc0 2024-04-20T21:49:33.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.790+0000 7f396a600700 1 -- 172.21.15.62:0/2292767989 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f394c06ecc0 msgr2=0x7f394c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:49:33.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.790+0000 7f396a600700 1 --2- 172.21.15.62:0/2292767989 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f394c06ecc0 0x7f394c071170 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f3958006010 tx=0x7f395801a040 comp rx=0 tx=0).stop 2024-04-20T21:49:33.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.790+0000 7f396a600700 1 -- 172.21.15.62:0/2292767989 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39640ff830 msgr2=0x7f3964198410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:49:33.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.790+0000 7f396a600700 1 --2- 172.21.15.62:0/2292767989 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39640ff830 0x7f3964198410 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f395000c8a0 tx=0x7f395000cc60 comp rx=0 tx=0).stop 2024-04-20T21:49:33.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.790+0000 7f396a600700 1 -- 172.21.15.62:0/2292767989 shutdown_connections 2024-04-20T21:49:33.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.790+0000 7f396a600700 1 --2- 172.21.15.62:0/2292767989 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f394c06ecc0 0x7f394c071170 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:49:33.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.790+0000 7f396a600700 1 --2- 172.21.15.62:0/2292767989 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39640fee90 0x7f3964197ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:49:33.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.790+0000 7f396a600700 1 --2- 172.21.15.62:0/2292767989 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39640ff830 0x7f3964198410 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:49:33.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.790+0000 7f396a600700 1 -- 172.21.15.62:0/2292767989 >> 172.21.15.62:0/2292767989 conn(0x7f39640faad0 msgr2=0x7f39640fbfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:49:33.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.790+0000 7f396a600700 1 -- 172.21.15.62:0/2292767989 shutdown_connections 2024-04-20T21:49:33.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.790+0000 7f396a600700 1 -- 172.21.15.62:0/2292767989 wait complete. 2024-04-20T21:49:33.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.953+0000 7f5b3f000700 1 -- 172.21.15.62:0/3181292285 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5b38100c60 msgr2=0x7f5b38104c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:49:33.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.953+0000 7f5b3f000700 1 --2- 172.21.15.62:0/3181292285 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5b38100c60 0x7f5b38104c30 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f5b24009910 tx=0x7f5b24009c20 comp rx=0 tx=0).stop 2024-04-20T21:49:33.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.954+0000 7f5b3f000700 1 -- 172.21.15.62:0/3181292285 shutdown_connections 2024-04-20T21:49:33.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.954+0000 7f5b3f000700 1 --2- 172.21.15.62:0/3181292285 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5b38100c60 0x7f5b38104c30 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:49:33.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.954+0000 7f5b3f000700 1 --2- 172.21.15.62:0/3181292285 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b381002c0 0x7f5b38100690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:49:33.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.954+0000 7f5b3f000700 1 -- 172.21.15.62:0/3181292285 >> 172.21.15.62:0/3181292285 conn(0x7f5b380fbb60 msgr2=0x7f5b380fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:49:33.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.954+0000 7f5b3f000700 1 -- 172.21.15.62:0/3181292285 shutdown_connections 2024-04-20T21:49:33.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.954+0000 7f5b3f000700 1 -- 172.21.15.62:0/3181292285 wait complete. 2024-04-20T21:49:33.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.955+0000 7f5b3f000700 1 Processor -- start 2024-04-20T21:49:33.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.955+0000 7f5b3f000700 1 -- start start 2024-04-20T21:49:33.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.955+0000 7f5b3f000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b381002c0 0x7f5b3819b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:49:33.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.955+0000 7f5b3f000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5b38100c60 0x7f5b3819bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:49:33.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.955+0000 7f5b3f000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b3819c5b0 con 0x7f5b38100c60 2024-04-20T21:49:33.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.955+0000 7f5b3f000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b38195a60 con 0x7f5b381002c0 2024-04-20T21:49:33.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.956+0000 7f5b3d200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5b38100c60 0x7f5b3819bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:49:33.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.956+0000 7f5b3d200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5b38100c60 0x7f5b3819bf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40934/0 (socket says 172.21.15.62:40934) 2024-04-20T21:49:33.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.956+0000 7f5b3d200700 1 -- 172.21.15.62:0/4159105900 learned_addr learned my addr 172.21.15.62:0/4159105900 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:49:33.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.956+0000 7f5b3dc00700 1 --2- 172.21.15.62:0/4159105900 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b381002c0 0x7f5b3819b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:49:33.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.956+0000 7f5b3d200700 1 -- 172.21.15.62:0/4159105900 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b381002c0 msgr2=0x7f5b3819b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:49:33.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.956+0000 7f5b3d200700 1 --2- 172.21.15.62:0/4159105900 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b381002c0 0x7f5b3819b9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:49:33.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.956+0000 7f5b3d200700 1 -- 172.21.15.62:0/4159105900 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b240095f0 con 0x7f5b38100c60 2024-04-20T21:49:33.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.956+0000 7f5b3d200700 1 --2- 172.21.15.62:0/4159105900 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5b38100c60 0x7f5b3819bf20 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f5b2400bda0 tx=0x7f5b2400be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:49:33.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.957+0000 7f5b36000700 1 -- 172.21.15.62:0/4159105900 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5b2401d070 con 0x7f5b38100c60 2024-04-20T21:49:33.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.957+0000 7f5b36000700 1 -- 172.21.15.62:0/4159105900 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5b24004fb0 con 0x7f5b38100c60 2024-04-20T21:49:33.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.957+0000 7f5b3f000700 1 -- 172.21.15.62:0/4159105900 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b38195ce0 con 0x7f5b38100c60 2024-04-20T21:49:33.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.957+0000 7f5b36000700 1 -- 172.21.15.62:0/4159105900 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5b2400f460 con 0x7f5b38100c60 2024-04-20T21:49:33.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.957+0000 7f5b3f000700 1 -- 172.21.15.62:0/4159105900 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b381961d0 con 0x7f5b38100c60 2024-04-20T21:49:33.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.958+0000 7f5b3f000700 1 -- 172.21.15.62:0/4159105900 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b38108630 con 0x7f5b38100c60 2024-04-20T21:49:33.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.959+0000 7f5b36000700 1 -- 172.21.15.62:0/4159105900 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f5b24022800 con 0x7f5b38100c60 2024-04-20T21:49:33.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.960+0000 7f5b36000700 1 --2- 172.21.15.62:0/4159105900 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5b2006ecc0 0x7f5b20071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:49:33.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.960+0000 7f5b36000700 1 -- 172.21.15.62:0/4159105900 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f5b24090390 con 0x7f5b38100c60 2024-04-20T21:49:33.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.960+0000 7f5b3dc00700 1 --2- 172.21.15.62:0/4159105900 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5b2006ecc0 0x7f5b20071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:49:33.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.961+0000 7f5b3dc00700 1 --2- 172.21.15.62:0/4159105900 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5b2006ecc0 0x7f5b20071170 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f5b2c005950 tx=0x7f5b2c005cf0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:49:33.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:33.966+0000 7f5b36000700 1 -- 172.21.15.62:0/4159105900 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f5b2405ce20 con 0x7f5b38100c60 2024-04-20T21:49:34.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:49:33 smithi062 ceph-mon[76322]: pgmap v427: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:49:34.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:34.215+0000 7f5b3f000700 1 -- 172.21.15.62:0/4159105900 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f5b3802cca0 con 0x7f5b2006ecc0 2024-04-20T21:49:34.236 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:49:33 smithi189 ceph-mon[57022]: pgmap v427: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:49:34.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:34.248+0000 7f5b36000700 1 -- 172.21.15.62:0/4159105900 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f5b3802cca0 con 0x7f5b2006ecc0 2024-04-20T21:49:34.246 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T21:49:34.246 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (15m) 6m ago 29m 14.2M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T21:49:34.246 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (29m) 6m ago 29m 9512k - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T21:49:34.247 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (28m) 7m ago 28m 8942k - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T21:49:34.247 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (11m) 6m ago 29m 6660k - 19.0.0-3133-ga8c84f59 76bc6b91b726 2825d5ba11c6 2024-04-20T21:49:34.247 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (11m) 7m ago 28m 6665k - 19.0.0-3133-ga8c84f59 76bc6b91b726 20f746fa1bba 2024-04-20T21:49:34.247 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (14m) 6m ago 28m 74.2M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T21:49:34.247 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (6m) 6m ago 21m 11.2M - 19.0.0-3133-ga8c84f59 76bc6b91b726 de80bc90a683 2024-04-20T21:49:34.247 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (7m) 6m ago 22m 11.9M - 19.0.0-3133-ga8c84f59 76bc6b91b726 cd793b7ee98e 2024-04-20T21:49:34.247 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (22m) 7m ago 22m 117M - 18.2.0 dc2bc1663786 fb2a8a223db8 2024-04-20T21:49:34.247 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (21m) 7m ago 21m 55.6M - 18.2.0 dc2bc1663786 f2eec2ee8951 2024-04-20T21:49:34.247 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (18m) 6m ago 31m 511M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T21:49:34.247 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (16m) 7m ago 27m 452M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T21:49:34.247 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (13m) 6m ago 31m 55.9M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 019eb1479485 2024-04-20T21:49:34.247 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (12m) 7m ago 27m 39.8M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 55aac560aab6 2024-04-20T21:49:34.247 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (16m) 6m ago 29m 7952k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T21:49:34.247 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (16m) 7m ago 28m 8304k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T21:49:34.248 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (10m) 6m ago 26m 88.0M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 ee2bb901f26e 2024-04-20T21:49:34.248 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (10m) 6m ago 26m 92.3M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 9075152d4162 2024-04-20T21:49:34.248 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (9m) 6m ago 25m 74.4M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 2d4f3e38dc0e 2024-04-20T21:49:34.248 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (9m) 7m ago 25m 87.6M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 8f97b17ae0eb 2024-04-20T21:49:34.248 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (8m) 7m ago 24m 90.7M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 653d8079967d 2024-04-20T21:49:34.248 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (7m) 7m ago 23m 11.5M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 596dded14c12 2024-04-20T21:49:34.248 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (15m) 6m ago 28m 74.0M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T21:49:34.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:34.252+0000 7f5b3f000700 1 -- 172.21.15.62:0/4159105900 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5b2006ecc0 msgr2=0x7f5b20071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:49:34.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:34.252+0000 7f5b3f000700 1 --2- 172.21.15.62:0/4159105900 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5b2006ecc0 0x7f5b20071170 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f5b2c005950 tx=0x7f5b2c005cf0 comp rx=0 tx=0).stop 2024-04-20T21:49:34.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:34.252+0000 7f5b3f000700 1 -- 172.21.15.62:0/4159105900 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5b38100c60 msgr2=0x7f5b3819bf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:49:34.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:34.252+0000 7f5b3f000700 1 --2- 172.21.15.62:0/4159105900 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5b38100c60 0x7f5b3819bf20 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f5b2400bda0 tx=0x7f5b2400be80 comp rx=0 tx=0).stop 2024-04-20T21:49:34.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:34.252+0000 7f5b3f000700 1 -- 172.21.15.62:0/4159105900 shutdown_connections 2024-04-20T21:49:34.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:34.252+0000 7f5b3f000700 1 --2- 172.21.15.62:0/4159105900 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5b2006ecc0 0x7f5b20071170 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:49:34.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:34.252+0000 7f5b3f000700 1 --2- 172.21.15.62:0/4159105900 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b381002c0 0x7f5b3819b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:49:34.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:34.252+0000 7f5b3f000700 1 --2- 172.21.15.62:0/4159105900 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5b38100c60 0x7f5b3819bf20 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:49:34.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:34.252+0000 7f5b3f000700 1 -- 172.21.15.62:0/4159105900 >> 172.21.15.62:0/4159105900 conn(0x7f5b380fbb60 msgr2=0x7f5b380fdf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:49:34.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:34.252+0000 7f5b3f000700 1 -- 172.21.15.62:0/4159105900 shutdown_connections 2024-04-20T21:49:34.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:34.252+0000 7f5b3f000700 1 -- 172.21.15.62:0/4159105900 wait complete. 2024-04-20T21:49:34.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:34.419+0000 7fbf60800700 1 -- 172.21.15.62:0/258958604 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbf580739a0 msgr2=0x7fbf5810c5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:49:34.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:34.419+0000 7fbf60800700 1 --2- 172.21.15.62:0/258958604 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbf580739a0 0x7fbf5810c5e0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fbf48009940 tx=0x7fbf48009c50 comp rx=0 tx=0).stop 2024-04-20T21:49:34.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:34.419+0000 7fbf60800700 1 -- 172.21.15.62:0/258958604 shutdown_connections 2024-04-20T21:49:34.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:34.419+0000 7fbf60800700 1 --2- 172.21.15.62:0/258958604 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbf580739a0 0x7fbf5810c5e0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:49:34.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:34.419+0000 7fbf60800700 1 --2- 172.21.15.62:0/258958604 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf58073090 0x7fbf58073460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:49:34.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:34.419+0000 7fbf60800700 1 -- 172.21.15.62:0/258958604 >> 172.21.15.62:0/258958604 conn(0x7fbf580fbda0 msgr2=0x7fbf580fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:49:34.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:34.419+0000 7fbf60800700 1 -- 172.21.15.62:0/258958604 shutdown_connections 2024-04-20T21:49:34.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:34.419+0000 7fbf60800700 1 -- 172.21.15.62:0/258958604 wait complete. 2024-04-20T21:49:34.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:34.420+0000 7fbf60800700 1 Processor -- start 2024-04-20T21:49:34.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:34.421+0000 7fbf60800700 1 -- start start 2024-04-20T21:49:34.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:34.421+0000 7fbf60800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf58073090 0x7fbf581981b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:49:34.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:34.421+0000 7fbf60800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbf580739a0 0x7fbf581986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:49:34.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:34.421+0000 7fbf60800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf58198dd0 con 0x7fbf580739a0 2024-04-20T21:49:34.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:34.421+0000 7fbf60800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf5819cb60 con 0x7fbf58073090 2024-04-20T21:49:34.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:34.421+0000 7fbf5d400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbf580739a0 0x7fbf581986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:49:34.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:34.421+0000 7fbf5d400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbf580739a0 0x7fbf581986f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40952/0 (socket says 172.21.15.62:40952) 2024-04-20T21:49:34.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:34.421+0000 7fbf5d400700 1 -- 172.21.15.62:0/1883266046 learned_addr learned my addr 172.21.15.62:0/1883266046 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:49:34.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:34.422+0000 7fbf5de00700 1 --2- 172.21.15.62:0/1883266046 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf58073090 0x7fbf581981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:49:34.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:34.422+0000 7fbf5d400700 1 -- 172.21.15.62:0/1883266046 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf58073090 msgr2=0x7fbf581981b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:49:34.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:34.422+0000 7fbf5d400700 1 --2- 172.21.15.62:0/1883266046 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf58073090 0x7fbf581981b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:49:34.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:34.422+0000 7fbf5d400700 1 -- 172.21.15.62:0/1883266046 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf480095f0 con 0x7fbf580739a0 2024-04-20T21:49:34.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:34.422+0000 7fbf5d400700 1 --2- 172.21.15.62:0/1883266046 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbf580739a0 0x7fbf581986f0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fbf480055f0 tx=0x7fbf480056d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:49:34.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:34.422+0000 7fbf52a00700 1 -- 172.21.15.62:0/1883266046 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbf4801d070 con 0x7fbf580739a0 2024-04-20T21:49:34.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:34.423+0000 7fbf52a00700 1 -- 172.21.15.62:0/1883266046 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbf4800f460 con 0x7fbf580739a0 2024-04-20T21:49:34.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:34.423+0000 7fbf60800700 1 -- 172.21.15.62:0/1883266046 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf5819cde0 con 0x7fbf580739a0 2024-04-20T21:49:34.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:34.423+0000 7fbf52a00700 1 -- 172.21.15.62:0/1883266046 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbf48021640 con 0x7fbf580739a0 2024-04-20T21:49:34.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:34.423+0000 7fbf60800700 1 -- 172.21.15.62:0/1883266046 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf5819d280 con 0x7fbf580739a0 2024-04-20T21:49:34.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:34.424+0000 7fbf60800700 1 -- 172.21.15.62:0/1883266046 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf58109ce0 con 0x7fbf580739a0 2024-04-20T21:49:34.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:34.425+0000 7fbf52a00700 1 -- 172.21.15.62:0/1883266046 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fbf4800fa20 con 0x7fbf580739a0 2024-04-20T21:49:34.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:34.426+0000 7fbf52a00700 1 --2- 172.21.15.62:0/1883266046 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fbf4006ecc0 0x7fbf40071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:49:34.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:34.426+0000 7fbf52a00700 1 -- 172.21.15.62:0/1883266046 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fbf48090160 con 0x7fbf580739a0 2024-04-20T21:49:34.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:34.426+0000 7fbf5de00700 1 --2- 172.21.15.62:0/1883266046 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fbf4006ecc0 0x7fbf40071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:49:34.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:34.427+0000 7fbf5de00700 1 --2- 172.21.15.62:0/1883266046 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fbf4006ecc0 0x7fbf40071170 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fbf4c005950 tx=0x7fbf4c0058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:49:34.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:34.432+0000 7fbf52a00700 1 -- 172.21.15.62:0/1883266046 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fbf48058e70 con 0x7fbf580739a0 2024-04-20T21:49:34.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:34.777+0000 7fbf60800700 1 -- 172.21.15.62:0/1883266046 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fbf581995b0 con 0x7fbf580739a0 2024-04-20T21:49:34.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:34.778+0000 7fbf52a00700 1 -- 172.21.15.62:0/1883266046 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7fbf4805c530 con 0x7fbf580739a0 2024-04-20T21:49:34.777 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:49:34.777 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T21:49:34.777 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:49:34.777 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:49:34.777 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T21:49:34.777 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:49:34.777 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:49:34.777 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T21:49:34.777 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 6 2024-04-20T21:49:34.777 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:49:34.777 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T21:49:34.777 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 2, 2024-04-20T21:49:34.777 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:49:34.778 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:49:34.778 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T21:49:34.778 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 2, 2024-04-20T21:49:34.778 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 12 2024-04-20T21:49:34.778 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T21:49:34.778 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:49:34.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:34.781+0000 7fbf60800700 1 -- 172.21.15.62:0/1883266046 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fbf4006ecc0 msgr2=0x7fbf40071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:49:34.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:34.781+0000 7fbf60800700 1 --2- 172.21.15.62:0/1883266046 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fbf4006ecc0 0x7fbf40071170 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fbf4c005950 tx=0x7fbf4c0058e0 comp rx=0 tx=0).stop 2024-04-20T21:49:34.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:34.781+0000 7fbf60800700 1 -- 172.21.15.62:0/1883266046 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbf580739a0 msgr2=0x7fbf581986f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:49:34.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:34.781+0000 7fbf60800700 1 --2- 172.21.15.62:0/1883266046 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbf580739a0 0x7fbf581986f0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fbf480055f0 tx=0x7fbf480056d0 comp rx=0 tx=0).stop 2024-04-20T21:49:34.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:34.781+0000 7fbf60800700 1 -- 172.21.15.62:0/1883266046 shutdown_connections 2024-04-20T21:49:34.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:34.781+0000 7fbf60800700 1 --2- 172.21.15.62:0/1883266046 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fbf4006ecc0 0x7fbf40071170 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:49:34.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:34.781+0000 7fbf60800700 1 --2- 172.21.15.62:0/1883266046 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf58073090 0x7fbf581981b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:49:34.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:34.781+0000 7fbf60800700 1 --2- 172.21.15.62:0/1883266046 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbf580739a0 0x7fbf581986f0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:49:34.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:34.781+0000 7fbf60800700 1 -- 172.21.15.62:0/1883266046 >> 172.21.15.62:0/1883266046 conn(0x7fbf580fbda0 msgr2=0x7fbf58106e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:49:34.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:34.782+0000 7fbf60800700 1 -- 172.21.15.62:0/1883266046 shutdown_connections 2024-04-20T21:49:34.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:34.782+0000 7fbf60800700 1 -- 172.21.15.62:0/1883266046 wait complete. 2024-04-20T21:49:34.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:34.950+0000 7faafbe00700 1 -- 172.21.15.62:0/4137770140 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faafc106b80 msgr2=0x7faafc106ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:49:34.949 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:34.950+0000 7faafbe00700 1 --2- 172.21.15.62:0/4137770140 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faafc106b80 0x7faafc106ff0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7faae4009910 tx=0x7faae4009c20 comp rx=0 tx=0).stop 2024-04-20T21:49:34.949 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:34.950+0000 7faafbe00700 1 -- 172.21.15.62:0/4137770140 shutdown_connections 2024-04-20T21:49:34.949 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:34.950+0000 7faafbe00700 1 --2- 172.21.15.62:0/4137770140 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faafc106b80 0x7faafc106ff0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:49:34.949 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:34.950+0000 7faafbe00700 1 --2- 172.21.15.62:0/4137770140 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faafc10c660 0x7faafc10ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:49:34.949 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:34.950+0000 7faafbe00700 1 -- 172.21.15.62:0/4137770140 >> 172.21.15.62:0/4137770140 conn(0x7faafc075e90 msgr2=0x7faafc0782a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:49:34.949 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:34.950+0000 7faafbe00700 1 -- 172.21.15.62:0/4137770140 shutdown_connections 2024-04-20T21:49:34.949 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:34.950+0000 7faafbe00700 1 -- 172.21.15.62:0/4137770140 wait complete. 2024-04-20T21:49:34.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:34.951+0000 7faafbe00700 1 Processor -- start 2024-04-20T21:49:34.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:34.952+0000 7faafbe00700 1 -- start start 2024-04-20T21:49:34.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:34.952+0000 7faafbe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faafc106b80 0x7faafc19c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:49:34.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:34.952+0000 7faafbe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faafc10c660 0x7faafc19c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:49:34.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:34.952+0000 7faafbe00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faafc19ce00 con 0x7faafc106b80 2024-04-20T21:49:34.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:34.952+0000 7faafbe00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faafc1a0b90 con 0x7faafc10c660 2024-04-20T21:49:34.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:34.952+0000 7faafaa00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faafc106b80 0x7faafc19c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:49:34.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:34.952+0000 7faafaa00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faafc106b80 0x7faafc19c1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40966/0 (socket says 172.21.15.62:40966) 2024-04-20T21:49:34.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:34.952+0000 7faafaa00700 1 -- 172.21.15.62:0/1533308889 learned_addr learned my addr 172.21.15.62:0/1533308889 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:49:34.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:34.953+0000 7faaf3e00700 1 --2- 172.21.15.62:0/1533308889 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faafc10c660 0x7faafc19c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:49:34.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:34.953+0000 7faafaa00700 1 -- 172.21.15.62:0/1533308889 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faafc10c660 msgr2=0x7faafc19c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:49:34.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:34.953+0000 7faafaa00700 1 --2- 172.21.15.62:0/1533308889 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faafc10c660 0x7faafc19c720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:49:34.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:34.953+0000 7faafaa00700 1 -- 172.21.15.62:0/1533308889 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faae40095f0 con 0x7faafc106b80 2024-04-20T21:49:34.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:34.953+0000 7faafaa00700 1 --2- 172.21.15.62:0/1533308889 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faafc106b80 0x7faafc19c1e0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7faaec00c8a0 tx=0x7faaec00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:49:34.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:34.953+0000 7faaf3400700 1 -- 172.21.15.62:0/1533308889 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faaec00cea0 con 0x7faafc106b80 2024-04-20T21:49:34.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:34.953+0000 7faaf3400700 1 -- 172.21.15.62:0/1533308889 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7faaec004830 con 0x7faafc106b80 2024-04-20T21:49:34.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:34.953+0000 7faafbe00700 1 -- 172.21.15.62:0/1533308889 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faafc1a0e70 con 0x7faafc106b80 2024-04-20T21:49:34.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:34.954+0000 7faaf3400700 1 -- 172.21.15.62:0/1533308889 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faaec0055b0 con 0x7faafc106b80 2024-04-20T21:49:34.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:34.954+0000 7faafbe00700 1 -- 172.21.15.62:0/1533308889 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faafc1a13c0 con 0x7faafc106b80 2024-04-20T21:49:34.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:34.955+0000 7faafbe00700 1 -- 172.21.15.62:0/1533308889 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faafc10eb40 con 0x7faafc106b80 2024-04-20T21:49:34.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:34.956+0000 7faaf3400700 1 -- 172.21.15.62:0/1533308889 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7faaec0049a0 con 0x7faafc106b80 2024-04-20T21:49:34.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:34.957+0000 7faaf3400700 1 --2- 172.21.15.62:0/1533308889 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7faae006ecc0 0x7faae0071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:49:34.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:34.957+0000 7faaf3400700 1 -- 172.21.15.62:0/1533308889 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7faaec08ebf0 con 0x7faafc106b80 2024-04-20T21:49:34.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:34.957+0000 7faaf3e00700 1 --2- 172.21.15.62:0/1533308889 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7faae006ecc0 0x7faae0071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:49:34.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:34.958+0000 7faaf3e00700 1 --2- 172.21.15.62:0/1533308889 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7faae006ecc0 0x7faae0071170 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7faae4000c00 tx=0x7faae4005fd0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:49:34.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:34.962+0000 7faaf3400700 1 -- 172.21.15.62:0/1533308889 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7faaec057900 con 0x7faafc106b80 2024-04-20T21:49:35.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:49:34 smithi062 ceph-mon[76322]: from='client.34816 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:49:35.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:49:34 smithi062 ceph-mon[76322]: from='client.34820 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:49:35.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:49:34 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:49:35.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:49:34 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/1883266046' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:49:35.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:49:34 smithi189 ceph-mon[57022]: from='client.34816 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:49:35.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:49:34 smithi189 ceph-mon[57022]: from='client.34820 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:49:35.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:49:34 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:49:35.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:49:34 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/1883266046' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:49:35.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:35.256+0000 7faafbe00700 1 -- 172.21.15.62:0/1533308889 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7faafc1a1640 con 0x7faafc106b80 2024-04-20T21:49:35.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:35.257+0000 7faaf3400700 1 -- 172.21.15.62:0/1533308889 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 25 v25) v1 ==== 76+0+1887 (secure 0 0 0) 0x7faaec01d770 con 0x7faafc106b80 2024-04-20T21:49:35.255 INFO:teuthology.orchestra.run.smithi062.stdout:e25 2024-04-20T21:49:35.255 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T21:49:35.255 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T21:49:35.256 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T21:49:35.256 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:49:35.256 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T21:49:35.256 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T21:49:35.256 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 20 2024-04-20T21:49:35.256 INFO:teuthology.orchestra.run.smithi062.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-20T21:49:35.256 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T21:49:35.256 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T21:42:01.559061+0000 2024-04-20T21:49:35.256 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T21:49:35.256 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T21:49:35.256 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T21:49:35.256 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T21:49:35.256 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T21:49:35.256 INFO:teuthology.orchestra.run.smithi062.stdout:max_xattr_size 65536 2024-04-20T21:49:35.256 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T21:49:35.257 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T21:49:35.257 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 81 2024-04-20T21:49:35.257 INFO:teuthology.orchestra.run.smithi062.stdout:snap_rstat 1 2024-04-20T21:49:35.257 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T21:49:35.257 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 2 2024-04-20T21:49:35.257 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1 2024-04-20T21:49:35.257 INFO:teuthology.orchestra.run.smithi062.stdout:up {} 2024-04-20T21:49:35.257 INFO:teuthology.orchestra.run.smithi062.stdout:failed 0,1 2024-04-20T21:49:35.257 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T21:49:35.257 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T21:49:35.257 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T21:49:35.257 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T21:49:35.257 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data enabled 2024-04-20T21:49:35.257 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T21:49:35.258 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T21:49:35.258 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 1 2024-04-20T21:49:35.258 INFO:teuthology.orchestra.run.smithi062.stdout:qdb_cluster leader: 0 members: 2024-04-20T21:49:35.258 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:49:35.258 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:49:35.258 INFO:teuthology.orchestra.run.smithi062.stdout:Standby daemons: 2024-04-20T21:49:35.258 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:49:35.258 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ryqblb{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6826/4113916205,v1:172.21.15.62:6827/4113916205] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T21:49:35.258 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ozizwi{-1:34476} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6828/3801975165,v1:172.21.15.62:6829/3801975165] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T21:49:35.258 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.xbgspw{-1:44285} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6826/3996040526,v1:172.21.15.189:6827/3996040526] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:49:35.258 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.dvimer{-1:44289} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6824/1743111843,v1:172.21.15.189:6825/1743111843] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:49:35.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:35.260+0000 7faafbe00700 1 -- 172.21.15.62:0/1533308889 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7faae006ecc0 msgr2=0x7faae0071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:49:35.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:35.260+0000 7faafbe00700 1 --2- 172.21.15.62:0/1533308889 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7faae006ecc0 0x7faae0071170 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7faae4000c00 tx=0x7faae4005fd0 comp rx=0 tx=0).stop 2024-04-20T21:49:35.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:35.260+0000 7faafbe00700 1 -- 172.21.15.62:0/1533308889 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faafc106b80 msgr2=0x7faafc19c1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:49:35.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:35.260+0000 7faafbe00700 1 --2- 172.21.15.62:0/1533308889 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faafc106b80 0x7faafc19c1e0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7faaec00c8a0 tx=0x7faaec00cc60 comp rx=0 tx=0).stop 2024-04-20T21:49:35.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:35.260+0000 7faafbe00700 1 -- 172.21.15.62:0/1533308889 shutdown_connections 2024-04-20T21:49:35.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:35.260+0000 7faafbe00700 1 --2- 172.21.15.62:0/1533308889 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7faae006ecc0 0x7faae0071170 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:49:35.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:35.260+0000 7faafbe00700 1 --2- 172.21.15.62:0/1533308889 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faafc106b80 0x7faafc19c1e0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:49:35.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:35.260+0000 7faafbe00700 1 --2- 172.21.15.62:0/1533308889 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faafc10c660 0x7faafc19c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:49:35.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:35.260+0000 7faafbe00700 1 -- 172.21.15.62:0/1533308889 >> 172.21.15.62:0/1533308889 conn(0x7faafc075e90 msgr2=0x7faafc0778d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:49:35.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:35.260+0000 7faafbe00700 1 -- 172.21.15.62:0/1533308889 shutdown_connections 2024-04-20T21:49:35.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:35.260+0000 7faafbe00700 1 -- 172.21.15.62:0/1533308889 wait complete. 2024-04-20T21:49:35.260 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 25 2024-04-20T21:49:35.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:35.424+0000 7f8ae6200700 1 -- 172.21.15.62:0/1438867117 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8ae0068540 msgr2=0x7f8ae0068910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:49:35.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:35.424+0000 7f8ae6200700 1 --2- 172.21.15.62:0/1438867117 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8ae0068540 0x7f8ae0068910 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f8ad0009940 tx=0x7f8ad0009c50 comp rx=0 tx=0).stop 2024-04-20T21:49:35.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:35.424+0000 7f8ae6200700 1 -- 172.21.15.62:0/1438867117 shutdown_connections 2024-04-20T21:49:35.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:35.424+0000 7f8ae6200700 1 --2- 172.21.15.62:0/1438867117 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ae0068ee0 0x7f8ae0110a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:49:35.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:35.424+0000 7f8ae6200700 1 --2- 172.21.15.62:0/1438867117 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8ae0068540 0x7f8ae0068910 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:49:35.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:35.424+0000 7f8ae6200700 1 -- 172.21.15.62:0/1438867117 >> 172.21.15.62:0/1438867117 conn(0x7f8ae0075130 msgr2=0x7f8ae0075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:49:35.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:35.424+0000 7f8ae6200700 1 -- 172.21.15.62:0/1438867117 shutdown_connections 2024-04-20T21:49:35.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:35.424+0000 7f8ae6200700 1 -- 172.21.15.62:0/1438867117 wait complete. 2024-04-20T21:49:35.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:35.426+0000 7f8ae6200700 1 Processor -- start 2024-04-20T21:49:35.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:35.426+0000 7f8ae6200700 1 -- start start 2024-04-20T21:49:35.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:35.427+0000 7f8ae6200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8ae0068540 0x7f8ae019c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:49:35.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:35.427+0000 7f8ae6200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ae0068ee0 0x7f8ae019c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:49:35.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:35.427+0000 7f8ae6200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ae019d070 con 0x7f8ae0068540 2024-04-20T21:49:35.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:35.427+0000 7f8ae6200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ae01a0e00 con 0x7f8ae0068ee0 2024-04-20T21:49:35.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:35.427+0000 7f8adf400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8ae0068540 0x7f8ae019c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:49:35.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:35.427+0000 7f8adf400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8ae0068540 0x7f8ae019c450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40970/0 (socket says 172.21.15.62:40970) 2024-04-20T21:49:35.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:35.427+0000 7f8adf400700 1 -- 172.21.15.62:0/1569164260 learned_addr learned my addr 172.21.15.62:0/1569164260 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:49:35.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:35.427+0000 7f8ad7e00700 1 --2- 172.21.15.62:0/1569164260 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ae0068ee0 0x7f8ae019c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:49:35.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:35.427+0000 7f8adf400700 1 -- 172.21.15.62:0/1569164260 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ae0068ee0 msgr2=0x7f8ae019c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:49:35.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:35.427+0000 7f8adf400700 1 --2- 172.21.15.62:0/1569164260 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ae0068ee0 0x7f8ae019c990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:49:35.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:35.427+0000 7f8adf400700 1 -- 172.21.15.62:0/1569164260 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ad00095f0 con 0x7f8ae0068540 2024-04-20T21:49:35.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:35.428+0000 7f8adf400700 1 --2- 172.21.15.62:0/1569164260 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8ae0068540 0x7f8ae019c450 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f8ad00055f0 tx=0x7f8ad00056d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:49:35.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:35.428+0000 7f8adcc00700 1 -- 172.21.15.62:0/1569164260 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8ad001d070 con 0x7f8ae0068540 2024-04-20T21:49:35.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:35.428+0000 7f8adcc00700 1 -- 172.21.15.62:0/1569164260 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8ad000f460 con 0x7f8ae0068540 2024-04-20T21:49:35.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:35.428+0000 7f8adcc00700 1 -- 172.21.15.62:0/1569164260 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8ad0021640 con 0x7f8ae0068540 2024-04-20T21:49:35.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:35.428+0000 7f8ae6200700 1 -- 172.21.15.62:0/1569164260 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ae01a1080 con 0x7f8ae0068540 2024-04-20T21:49:35.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:35.428+0000 7f8ae6200700 1 -- 172.21.15.62:0/1569164260 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ae01a1580 con 0x7f8ae0068540 2024-04-20T21:49:35.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:35.430+0000 7f8ae6200700 1 -- 172.21.15.62:0/1569164260 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ae0101410 con 0x7f8ae0068540 2024-04-20T21:49:35.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:35.431+0000 7f8adcc00700 1 -- 172.21.15.62:0/1569164260 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f8ad000fab0 con 0x7f8ae0068540 2024-04-20T21:49:35.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:35.432+0000 7f8adcc00700 1 --2- 172.21.15.62:0/1569164260 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f8ac406ecc0 0x7f8ac4071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:49:35.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:35.432+0000 7f8adcc00700 1 -- 172.21.15.62:0/1569164260 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f8ad0090190 con 0x7f8ae0068540 2024-04-20T21:49:35.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:35.432+0000 7f8ad7e00700 1 --2- 172.21.15.62:0/1569164260 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f8ac406ecc0 0x7f8ac4071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:49:35.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:35.433+0000 7f8ad7e00700 1 --2- 172.21.15.62:0/1569164260 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f8ac406ecc0 0x7f8ac4071170 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f8ac8005950 tx=0x7f8ac80058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:49:35.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:35.437+0000 7f8adcc00700 1 -- 172.21.15.62:0/1569164260 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f8ad0058ea0 con 0x7f8ae0068540 2024-04-20T21:49:35.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:35.693+0000 7f8ae6200700 1 -- 172.21.15.62:0/1569164260 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8ae01a17a0 con 0x7f8ac406ecc0 2024-04-20T21:49:35.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:35.695+0000 7f8adcc00700 1 -- 172.21.15.62:0/1569164260 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f8ae01a17a0 con 0x7f8ac406ecc0 2024-04-20T21:49:35.693 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:49:35.694 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T21:49:35.694 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T21:49:35.694 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-20T21:49:35.694 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T21:49:35.694 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr", 2024-04-20T21:49:35.694 INFO:teuthology.orchestra.run.smithi062.stdout: "crash", 2024-04-20T21:49:35.694 INFO:teuthology.orchestra.run.smithi062.stdout: "mon", 2024-04-20T21:49:35.694 INFO:teuthology.orchestra.run.smithi062.stdout: "osd" 2024-04-20T21:49:35.694 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T21:49:35.694 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "14/23 daemons upgraded", 2024-04-20T21:49:35.694 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading mds daemons", 2024-04-20T21:49:35.694 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T21:49:35.694 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:49:35.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:35.698+0000 7f8ae6200700 1 -- 172.21.15.62:0/1569164260 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f8ac406ecc0 msgr2=0x7f8ac4071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:49:35.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:35.698+0000 7f8ae6200700 1 --2- 172.21.15.62:0/1569164260 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f8ac406ecc0 0x7f8ac4071170 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f8ac8005950 tx=0x7f8ac80058e0 comp rx=0 tx=0).stop 2024-04-20T21:49:35.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:35.698+0000 7f8ae6200700 1 -- 172.21.15.62:0/1569164260 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8ae0068540 msgr2=0x7f8ae019c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:49:35.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:35.698+0000 7f8ae6200700 1 --2- 172.21.15.62:0/1569164260 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8ae0068540 0x7f8ae019c450 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f8ad00055f0 tx=0x7f8ad00056d0 comp rx=0 tx=0).stop 2024-04-20T21:49:35.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:35.699+0000 7f8ae6200700 1 -- 172.21.15.62:0/1569164260 shutdown_connections 2024-04-20T21:49:35.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:35.699+0000 7f8ae6200700 1 --2- 172.21.15.62:0/1569164260 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f8ac406ecc0 0x7f8ac4071170 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:49:35.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:35.699+0000 7f8ae6200700 1 --2- 172.21.15.62:0/1569164260 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8ae0068540 0x7f8ae019c450 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:49:35.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:35.699+0000 7f8ae6200700 1 --2- 172.21.15.62:0/1569164260 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ae0068ee0 0x7f8ae019c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:49:35.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:35.699+0000 7f8ae6200700 1 -- 172.21.15.62:0/1569164260 >> 172.21.15.62:0/1569164260 conn(0x7f8ae0075130 msgr2=0x7f8ae010eea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:49:35.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:35.699+0000 7f8ae6200700 1 -- 172.21.15.62:0/1569164260 shutdown_connections 2024-04-20T21:49:35.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:35.699+0000 7f8ae6200700 1 -- 172.21.15.62:0/1569164260 wait complete. 2024-04-20T21:49:35.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:35.866+0000 7f73a9600700 1 -- 172.21.15.62:0/2440201419 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f73a4102e30 msgr2=0x7f73a4103200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:49:35.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:35.866+0000 7f73a9600700 1 --2- 172.21.15.62:0/2440201419 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f73a4102e30 0x7f73a4103200 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f7394009910 tx=0x7f7394009c20 comp rx=0 tx=0).stop 2024-04-20T21:49:35.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:35.866+0000 7f73a9600700 1 -- 172.21.15.62:0/2440201419 shutdown_connections 2024-04-20T21:49:35.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:35.866+0000 7f73a9600700 1 --2- 172.21.15.62:0/2440201419 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f73a40ff9e0 0x7f73a40ffe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:49:35.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:35.866+0000 7f73a9600700 1 --2- 172.21.15.62:0/2440201419 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f73a4102e30 0x7f73a4103200 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:49:35.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:35.866+0000 7f73a9600700 1 -- 172.21.15.62:0/2440201419 >> 172.21.15.62:0/2440201419 conn(0x7f73a4074fa0 msgr2=0x7f73a40753a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:49:35.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:35.866+0000 7f73a9600700 1 -- 172.21.15.62:0/2440201419 shutdown_connections 2024-04-20T21:49:35.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:35.866+0000 7f73a9600700 1 -- 172.21.15.62:0/2440201419 wait complete. 2024-04-20T21:49:35.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:35.868+0000 7f73a9600700 1 Processor -- start 2024-04-20T21:49:35.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:35.868+0000 7f73a9600700 1 -- start start 2024-04-20T21:49:35.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:35.868+0000 7f73a9600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f73a40ff9e0 0x7f73a4197e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:49:35.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:35.869+0000 7f73a9600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f73a4102e30 0x7f73a4198340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:49:35.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:35.869+0000 7f73a9600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73a4198a20 con 0x7f73a4102e30 2024-04-20T21:49:35.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:35.869+0000 7f73a9600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73a419c7b0 con 0x7f73a40ff9e0 2024-04-20T21:49:35.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:35.869+0000 7f73a3400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f73a4102e30 0x7f73a4198340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:49:35.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:35.869+0000 7f73a3400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f73a4102e30 0x7f73a4198340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40994/0 (socket says 172.21.15.62:40994) 2024-04-20T21:49:35.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:35.869+0000 7f73a3400700 1 -- 172.21.15.62:0/2389319796 learned_addr learned my addr 172.21.15.62:0/2389319796 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:49:35.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:35.869+0000 7f73a3e00700 1 --2- 172.21.15.62:0/2389319796 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f73a40ff9e0 0x7f73a4197e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:49:35.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:35.869+0000 7f73a3400700 1 -- 172.21.15.62:0/2389319796 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f73a40ff9e0 msgr2=0x7f73a4197e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:49:35.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:35.869+0000 7f73a3400700 1 --2- 172.21.15.62:0/2389319796 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f73a40ff9e0 0x7f73a4197e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:49:35.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:35.870+0000 7f73a3400700 1 -- 172.21.15.62:0/2389319796 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73940095f0 con 0x7f73a4102e30 2024-04-20T21:49:35.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:35.870+0000 7f73a3e00700 1 --2- 172.21.15.62:0/2389319796 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f73a40ff9e0 0x7f73a4197e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-20T21:49:35.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:35.870+0000 7f73a3400700 1 --2- 172.21.15.62:0/2389319796 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f73a4102e30 0x7f73a4198340 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f739000d8d0 tx=0x7f739000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:49:35.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:35.870+0000 7f73a0c00700 1 -- 172.21.15.62:0/2389319796 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7390009940 con 0x7f73a4102e30 2024-04-20T21:49:35.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:35.870+0000 7f73a0c00700 1 -- 172.21.15.62:0/2389319796 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7390010460 con 0x7f73a4102e30 2024-04-20T21:49:35.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:35.870+0000 7f73a9600700 1 -- 172.21.15.62:0/2389319796 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73a419ca90 con 0x7f73a4102e30 2024-04-20T21:49:35.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:35.870+0000 7f73a0c00700 1 -- 172.21.15.62:0/2389319796 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f739000f5f0 con 0x7f73a4102e30 2024-04-20T21:49:35.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:35.870+0000 7f73a9600700 1 -- 172.21.15.62:0/2389319796 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73a419d060 con 0x7f73a4102e30 2024-04-20T21:49:35.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:35.871+0000 7f73a9600700 1 -- 172.21.15.62:0/2389319796 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73a4062380 con 0x7f73a4102e30 2024-04-20T21:49:35.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:35.873+0000 7f73a0c00700 1 -- 172.21.15.62:0/2389319796 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f73900105d0 con 0x7f73a4102e30 2024-04-20T21:49:35.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:35.875+0000 7f73a0c00700 1 --2- 172.21.15.62:0/2389319796 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f738806ecc0 0x7f7388071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:49:35.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:35.875+0000 7f73a0c00700 1 -- 172.21.15.62:0/2389319796 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f739008ee00 con 0x7f73a4102e30 2024-04-20T21:49:35.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:35.875+0000 7f73a3e00700 1 --2- 172.21.15.62:0/2389319796 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f738806ecc0 0x7f7388071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:49:35.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:35.876+0000 7f73a3e00700 1 --2- 172.21.15.62:0/2389319796 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f738806ecc0 0x7f7388071170 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f7394005b40 tx=0x7f739401a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:49:35.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:35.879+0000 7f73a0c00700 1 -- 172.21.15.62:0/2389319796 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f7390057cc0 con 0x7f73a4102e30 2024-04-20T21:49:36.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:49:35 smithi062 ceph-mon[76322]: from='client.34824 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:49:36.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:49:35 smithi062 ceph-mon[76322]: pgmap v428: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:49:36.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:49:35 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/1533308889' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T21:49:36.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:36.209+0000 7f73a9600700 1 -- 172.21.15.62:0/2389319796 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f73a4199160 con 0x7f73a4102e30 2024-04-20T21:49:36.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:36.210+0000 7f73a0c00700 1 -- 172.21.15.62:0/2389319796 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7f73900201c0 con 0x7f73a4102e30 2024-04-20T21:49:36.208 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-20T21:49:36.209 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-20T21:49:36.209 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is degraded 2024-04-20T21:49:36.209 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T21:49:36.209 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T21:49:36.209 INFO:teuthology.orchestra.run.smithi062.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-20T21:49:36.209 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-20T21:49:36.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:36.213+0000 7f73a9600700 1 -- 172.21.15.62:0/2389319796 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f738806ecc0 msgr2=0x7f7388071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:49:36.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:36.213+0000 7f73a9600700 1 --2- 172.21.15.62:0/2389319796 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f738806ecc0 0x7f7388071170 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f7394005b40 tx=0x7f739401a040 comp rx=0 tx=0).stop 2024-04-20T21:49:36.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:36.213+0000 7f73a9600700 1 -- 172.21.15.62:0/2389319796 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f73a4102e30 msgr2=0x7f73a4198340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:49:36.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:36.213+0000 7f73a9600700 1 --2- 172.21.15.62:0/2389319796 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f73a4102e30 0x7f73a4198340 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f739000d8d0 tx=0x7f739000dc90 comp rx=0 tx=0).stop 2024-04-20T21:49:36.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:36.213+0000 7f73a9600700 1 -- 172.21.15.62:0/2389319796 shutdown_connections 2024-04-20T21:49:36.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:36.213+0000 7f73a9600700 1 --2- 172.21.15.62:0/2389319796 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f738806ecc0 0x7f7388071170 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:49:36.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:36.213+0000 7f73a9600700 1 --2- 172.21.15.62:0/2389319796 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f73a40ff9e0 0x7f73a4197e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:49:36.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:36.213+0000 7f73a9600700 1 --2- 172.21.15.62:0/2389319796 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f73a4102e30 0x7f73a4198340 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:49:36.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:36.213+0000 7f73a9600700 1 -- 172.21.15.62:0/2389319796 >> 172.21.15.62:0/2389319796 conn(0x7f73a4074fa0 msgr2=0x7f73a40fe9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:49:36.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:36.213+0000 7f73a9600700 1 -- 172.21.15.62:0/2389319796 shutdown_connections 2024-04-20T21:49:36.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:49:36.213+0000 7f73a9600700 1 -- 172.21.15.62:0/2389319796 wait complete. 2024-04-20T21:49:36.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:49:35 smithi189 ceph-mon[57022]: from='client.34824 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:49:36.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:49:35 smithi189 ceph-mon[57022]: pgmap v428: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:49:36.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:49:35 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/1533308889' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T21:49:37.236 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:49:36 smithi189 ceph-mon[57022]: from='client.34836 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:49:37.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:49:36 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/2389319796' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T21:49:37.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:49:36 smithi062 ceph-mon[76322]: from='client.34836 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:49:37.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:49:36 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/2389319796' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T21:49:38.142 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:49:37 smithi062 ceph-mon[76322]: pgmap v429: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:49:38.236 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:49:37 smithi189 ceph-mon[57022]: pgmap v429: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:49:40.195 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:49:39 smithi189 ceph-mon[57022]: pgmap v430: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:49:40.284 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:49:39 smithi062 ceph-mon[76322]: pgmap v430: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:49:42.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:49:41 smithi189 ceph-mon[57022]: pgmap v431: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:49:42.284 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:49:41 smithi062 ceph-mon[76322]: pgmap v431: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:49:44.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:49:43 smithi189 ceph-mon[57022]: pgmap v432: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:49:44.284 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:49:43 smithi062 ceph-mon[76322]: pgmap v432: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:49:46.236 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:49:45 smithi189 ceph-mon[57022]: pgmap v433: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:49:46.284 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:49:45 smithi062 ceph-mon[76322]: pgmap v433: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:49:48.143 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:49:47 smithi062 ceph-mon[76322]: pgmap v434: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:49:48.236 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:49:47 smithi189 ceph-mon[57022]: pgmap v434: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:49:50.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:49:49 smithi189 ceph-mon[57022]: pgmap v435: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:49:50.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:49:49 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:49:50.284 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:49:49 smithi062 ceph-mon[76322]: pgmap v435: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:49:50.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:49:49 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:49:52.236 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:49:51 smithi189 ceph-mon[57022]: pgmap v436: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:49:52.284 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:49:51 smithi062 ceph-mon[76322]: pgmap v436: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:49:54.236 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:49:53 smithi189 ceph-mon[57022]: pgmap v437: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:49:54.284 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:49:53 smithi062 ceph-mon[76322]: pgmap v437: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:49:55.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:49:55 smithi189 ceph-mon[57022]: pgmap v438: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:49:56.284 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:49:55 smithi062 ceph-mon[76322]: pgmap v438: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:49:58.142 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:49:57 smithi062 ceph-mon[76322]: pgmap v439: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:49:58.236 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:49:57 smithi189 ceph-mon[57022]: pgmap v439: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:50:00.236 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:49:59 smithi189 ceph-mon[57022]: pgmap v440: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:50:00.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:49:59 smithi062 ceph-mon[76322]: pgmap v440: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:50:00.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:50:00 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-mon-smithi062[76318]: 2024-04-20T21:49:59.999+0000 7f6157e00700 -1 log_channel(cluster) log [ERR] : Health detail: HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-20T21:50:00.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:50:00 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-mon-smithi062[76318]: 2024-04-20T21:49:59.999+0000 7f6157e00700 -1 log_channel(cluster) log [ERR] : [WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-20T21:50:00.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:50:00 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-mon-smithi062[76318]: 2024-04-20T21:49:59.999+0000 7f6157e00700 -1 log_channel(cluster) log [ERR] : fs cephfs is degraded 2024-04-20T21:50:00.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:50:00 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-mon-smithi062[76318]: 2024-04-20T21:49:59.999+0000 7f6157e00700 -1 log_channel(cluster) log [ERR] : [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T21:50:00.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:50:00 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-mon-smithi062[76318]: 2024-04-20T21:49:59.999+0000 7f6157e00700 -1 log_channel(cluster) log [ERR] : fs cephfs has deprecated feature inline_data enabled. 2024-04-20T21:50:00.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:50:00 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-mon-smithi062[76318]: 2024-04-20T21:49:59.999+0000 7f6157e00700 -1 log_channel(cluster) log [ERR] : [ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-20T21:50:00.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:50:00 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-mon-smithi062[76318]: 2024-04-20T21:49:59.999+0000 7f6157e00700 -1 log_channel(cluster) log [ERR] : fs cephfs is offline because no MDS is active for it. 2024-04-20T21:50:01.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:50:00 smithi189 ceph-mon[57022]: Health detail: HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-20T21:50:01.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:50:00 smithi189 ceph-mon[57022]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-20T21:50:01.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:50:00 smithi189 ceph-mon[57022]: fs cephfs is degraded 2024-04-20T21:50:01.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:50:00 smithi189 ceph-mon[57022]: [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T21:50:01.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:50:00 smithi189 ceph-mon[57022]: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T21:50:01.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:50:00 smithi189 ceph-mon[57022]: [ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-20T21:50:01.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:50:00 smithi189 ceph-mon[57022]: fs cephfs is offline because no MDS is active for it. 2024-04-20T21:50:01.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:50:00 smithi062 ceph-mon[76322]: Health detail: HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-20T21:50:01.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:50:00 smithi062 ceph-mon[76322]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-20T21:50:01.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:50:00 smithi062 ceph-mon[76322]: fs cephfs is degraded 2024-04-20T21:50:01.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:50:00 smithi062 ceph-mon[76322]: [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T21:50:01.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:50:00 smithi062 ceph-mon[76322]: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T21:50:01.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:50:00 smithi062 ceph-mon[76322]: [ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-20T21:50:01.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:50:00 smithi062 ceph-mon[76322]: fs cephfs is offline because no MDS is active for it. 2024-04-20T21:50:02.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:50:01 smithi189 ceph-mon[57022]: pgmap v441: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:50:02.284 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:50:01 smithi062 ceph-mon[76322]: pgmap v441: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:50:04.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:50:03 smithi189 ceph-mon[57022]: pgmap v442: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:50:04.284 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:50:03 smithi062 ceph-mon[76322]: pgmap v442: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:50:05.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:50:04 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:50:05.284 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:50:04 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:50:06.236 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:50:05 smithi189 ceph-mon[57022]: pgmap v443: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:50:06.284 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:50:05 smithi062 ceph-mon[76322]: pgmap v443: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:50:06.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:06.389+0000 7f9769c00700 1 -- 172.21.15.62:0/598390982 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9764103040 msgr2=0x7f9764107010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:50:06.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:06.389+0000 7f9769c00700 1 --2- 172.21.15.62:0/598390982 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9764103040 0x7f9764107010 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f974c009910 tx=0x7f974c009c20 comp rx=0 tx=0).stop 2024-04-20T21:50:06.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:06.389+0000 7f9769c00700 1 -- 172.21.15.62:0/598390982 shutdown_connections 2024-04-20T21:50:06.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:06.389+0000 7f9769c00700 1 --2- 172.21.15.62:0/598390982 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9764103040 0x7f9764107010 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:50:06.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:06.389+0000 7f9769c00700 1 --2- 172.21.15.62:0/598390982 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97641026a0 0x7f9764102a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:50:06.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:06.389+0000 7f9769c00700 1 -- 172.21.15.62:0/598390982 >> 172.21.15.62:0/598390982 conn(0x7f97640fdf40 msgr2=0x7f9764100350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:50:06.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:06.389+0000 7f9769c00700 1 -- 172.21.15.62:0/598390982 shutdown_connections 2024-04-20T21:50:06.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:06.389+0000 7f9769c00700 1 -- 172.21.15.62:0/598390982 wait complete. 2024-04-20T21:50:06.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:06.390+0000 7f9769c00700 1 Processor -- start 2024-04-20T21:50:06.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:06.391+0000 7f9769c00700 1 -- start start 2024-04-20T21:50:06.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:06.391+0000 7f9769c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f97641026a0 0x7f9764198280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:50:06.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:06.391+0000 7f9769c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9764103040 0x7f97641987c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:50:06.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:06.391+0000 7f9769c00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9764198ea0 con 0x7f97641026a0 2024-04-20T21:50:06.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:06.391+0000 7f9769c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f976419cc30 con 0x7f9764103040 2024-04-20T21:50:06.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:06.391+0000 7f9762a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f97641026a0 0x7f9764198280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:50:06.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:06.391+0000 7f9762a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f97641026a0 0x7f9764198280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43852/0 (socket says 172.21.15.62:43852) 2024-04-20T21:50:06.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:06.391+0000 7f9762a00700 1 -- 172.21.15.62:0/2617955229 learned_addr learned my addr 172.21.15.62:0/2617955229 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:50:06.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:06.392+0000 7f975be00700 1 --2- 172.21.15.62:0/2617955229 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9764103040 0x7f97641987c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:50:06.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:06.392+0000 7f9762a00700 1 -- 172.21.15.62:0/2617955229 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9764103040 msgr2=0x7f97641987c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:50:06.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:06.392+0000 7f9762a00700 1 --2- 172.21.15.62:0/2617955229 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9764103040 0x7f97641987c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:50:06.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:06.392+0000 7f9762a00700 1 -- 172.21.15.62:0/2617955229 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f974c0095f0 con 0x7f97641026a0 2024-04-20T21:50:06.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:06.392+0000 7f9762a00700 1 --2- 172.21.15.62:0/2617955229 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f97641026a0 0x7f9764198280 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f975400b4b0 tx=0x7f975400b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:50:06.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:06.392+0000 7f975b400700 1 -- 172.21.15.62:0/2617955229 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9754011800 con 0x7f97641026a0 2024-04-20T21:50:06.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:06.392+0000 7f975b400700 1 -- 172.21.15.62:0/2617955229 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9754011e20 con 0x7f97641026a0 2024-04-20T21:50:06.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:06.393+0000 7f9769c00700 1 -- 172.21.15.62:0/2617955229 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f976419cf10 con 0x7f97641026a0 2024-04-20T21:50:06.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:06.393+0000 7f975b400700 1 -- 172.21.15.62:0/2617955229 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f975400f2e0 con 0x7f97641026a0 2024-04-20T21:50:06.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:06.393+0000 7f9769c00700 1 -- 172.21.15.62:0/2617955229 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f976419d460 con 0x7f97641026a0 2024-04-20T21:50:06.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:06.394+0000 7f9769c00700 1 -- 172.21.15.62:0/2617955229 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f976410a940 con 0x7f97641026a0 2024-04-20T21:50:06.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:06.395+0000 7f975b400700 1 -- 172.21.15.62:0/2617955229 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f9754010380 con 0x7f97641026a0 2024-04-20T21:50:06.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:06.396+0000 7f975b400700 1 --2- 172.21.15.62:0/2617955229 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f974806ecc0 0x7f9748071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:50:06.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:06.396+0000 7f975b400700 1 -- 172.21.15.62:0/2617955229 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f975408eaa0 con 0x7f97641026a0 2024-04-20T21:50:06.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:06.396+0000 7f975be00700 1 --2- 172.21.15.62:0/2617955229 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f974806ecc0 0x7f9748071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:50:06.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:06.397+0000 7f975be00700 1 --2- 172.21.15.62:0/2617955229 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f974806ecc0 0x7f9748071170 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f974c000c00 tx=0x7f974c005fd0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:50:06.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:06.401+0000 7f975b400700 1 -- 172.21.15.62:0/2617955229 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f97540577b0 con 0x7f97641026a0 2024-04-20T21:50:06.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:06.658+0000 7f9769c00700 1 -- 172.21.15.62:0/2617955229 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f976419d6e0 con 0x7f974806ecc0 2024-04-20T21:50:06.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:06.661+0000 7f975b400700 1 -- 172.21.15.62:0/2617955229 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f976419d6e0 con 0x7f974806ecc0 2024-04-20T21:50:06.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:06.664+0000 7f9769c00700 1 -- 172.21.15.62:0/2617955229 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f974806ecc0 msgr2=0x7f9748071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:50:06.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:06.664+0000 7f9769c00700 1 --2- 172.21.15.62:0/2617955229 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f974806ecc0 0x7f9748071170 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f974c000c00 tx=0x7f974c005fd0 comp rx=0 tx=0).stop 2024-04-20T21:50:06.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:06.664+0000 7f9769c00700 1 -- 172.21.15.62:0/2617955229 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f97641026a0 msgr2=0x7f9764198280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:50:06.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:06.664+0000 7f9769c00700 1 --2- 172.21.15.62:0/2617955229 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f97641026a0 0x7f9764198280 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f975400b4b0 tx=0x7f975400b870 comp rx=0 tx=0).stop 2024-04-20T21:50:06.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:06.665+0000 7f9769c00700 1 -- 172.21.15.62:0/2617955229 shutdown_connections 2024-04-20T21:50:06.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:06.665+0000 7f9769c00700 1 --2- 172.21.15.62:0/2617955229 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f974806ecc0 0x7f9748071170 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:50:06.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:06.665+0000 7f9769c00700 1 --2- 172.21.15.62:0/2617955229 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f97641026a0 0x7f9764198280 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:50:06.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:06.665+0000 7f9769c00700 1 --2- 172.21.15.62:0/2617955229 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9764103040 0x7f97641987c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:50:06.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:06.665+0000 7f9769c00700 1 -- 172.21.15.62:0/2617955229 >> 172.21.15.62:0/2617955229 conn(0x7f97640fdf40 msgr2=0x7f97640ff870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:50:06.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:06.665+0000 7f9769c00700 1 -- 172.21.15.62:0/2617955229 shutdown_connections 2024-04-20T21:50:06.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:06.665+0000 7f9769c00700 1 -- 172.21.15.62:0/2617955229 wait complete. 2024-04-20T21:50:06.681 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T21:50:06.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:06.828+0000 7f26da000700 1 -- 172.21.15.62:0/494094157 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f26d4106b80 msgr2=0x7f26d4106ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:50:06.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:06.828+0000 7f26da000700 1 --2- 172.21.15.62:0/494094157 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f26d4106b80 0x7f26d4106ff0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f26c0009910 tx=0x7f26c0009c20 comp rx=0 tx=0).stop 2024-04-20T21:50:06.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:06.828+0000 7f26da000700 1 -- 172.21.15.62:0/494094157 shutdown_connections 2024-04-20T21:50:06.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:06.828+0000 7f26da000700 1 --2- 172.21.15.62:0/494094157 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f26d4106b80 0x7f26d4106ff0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:50:06.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:06.828+0000 7f26da000700 1 --2- 172.21.15.62:0/494094157 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26d410c660 0x7f26d410ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:50:06.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:06.828+0000 7f26da000700 1 -- 172.21.15.62:0/494094157 >> 172.21.15.62:0/494094157 conn(0x7f26d4075e90 msgr2=0x7f26d40782a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:50:06.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:06.828+0000 7f26da000700 1 -- 172.21.15.62:0/494094157 shutdown_connections 2024-04-20T21:50:06.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:06.828+0000 7f26da000700 1 -- 172.21.15.62:0/494094157 wait complete. 2024-04-20T21:50:06.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:06.829+0000 7f26da000700 1 Processor -- start 2024-04-20T21:50:06.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:06.830+0000 7f26da000700 1 -- start start 2024-04-20T21:50:06.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:06.830+0000 7f26da000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f26d4106b80 0x7f26d419c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:50:06.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:06.830+0000 7f26da000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26d410c660 0x7f26d419c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:50:06.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:06.830+0000 7f26da000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26d419ce00 con 0x7f26d4106b80 2024-04-20T21:50:06.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:06.830+0000 7f26da000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26d41a0b90 con 0x7f26d410c660 2024-04-20T21:50:06.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:06.830+0000 7f26d8c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f26d4106b80 0x7f26d419c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:50:06.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:06.830+0000 7f26d8c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f26d4106b80 0x7f26d419c1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43856/0 (socket says 172.21.15.62:43856) 2024-04-20T21:50:06.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:06.830+0000 7f26d8c00700 1 -- 172.21.15.62:0/2493306687 learned_addr learned my addr 172.21.15.62:0/2493306687 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:50:06.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:06.831+0000 7f26d3e00700 1 --2- 172.21.15.62:0/2493306687 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26d410c660 0x7f26d419c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:50:06.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:06.831+0000 7f26d8c00700 1 -- 172.21.15.62:0/2493306687 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26d410c660 msgr2=0x7f26d419c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:50:06.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:06.831+0000 7f26d8c00700 1 --2- 172.21.15.62:0/2493306687 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26d410c660 0x7f26d419c720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:50:06.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:06.831+0000 7f26d8c00700 1 -- 172.21.15.62:0/2493306687 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26c00095f0 con 0x7f26d4106b80 2024-04-20T21:50:06.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:06.831+0000 7f26d8c00700 1 --2- 172.21.15.62:0/2493306687 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f26d4106b80 0x7f26d419c1e0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f26c800d8d0 tx=0x7f26c800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:50:06.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:06.831+0000 7f26d1600700 1 -- 172.21.15.62:0/2493306687 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f26c8009940 con 0x7f26d4106b80 2024-04-20T21:50:06.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:06.831+0000 7f26d1600700 1 -- 172.21.15.62:0/2493306687 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f26c8010460 con 0x7f26d4106b80 2024-04-20T21:50:06.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:06.831+0000 7f26d1600700 1 -- 172.21.15.62:0/2493306687 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f26c800f5f0 con 0x7f26d4106b80 2024-04-20T21:50:06.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:06.831+0000 7f26da000700 1 -- 172.21.15.62:0/2493306687 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26d41a0e70 con 0x7f26d4106b80 2024-04-20T21:50:06.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:06.832+0000 7f26da000700 1 -- 172.21.15.62:0/2493306687 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26d41a13c0 con 0x7f26d4106b80 2024-04-20T21:50:06.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:06.833+0000 7f26da000700 1 -- 172.21.15.62:0/2493306687 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26d410eb40 con 0x7f26d4106b80 2024-04-20T21:50:06.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:06.834+0000 7f26d1600700 1 -- 172.21.15.62:0/2493306687 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f26c80105d0 con 0x7f26d4106b80 2024-04-20T21:50:06.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:06.835+0000 7f26d1600700 1 --2- 172.21.15.62:0/2493306687 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f26bc06ecc0 0x7f26bc071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:50:06.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:06.835+0000 7f26d1600700 1 -- 172.21.15.62:0/2493306687 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f26c808ee10 con 0x7f26d4106b80 2024-04-20T21:50:06.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:06.835+0000 7f26d3e00700 1 --2- 172.21.15.62:0/2493306687 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f26bc06ecc0 0x7f26bc071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:50:06.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:06.836+0000 7f26d3e00700 1 --2- 172.21.15.62:0/2493306687 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f26bc06ecc0 0x7f26bc071170 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f26c0006010 tx=0x7f26c000b540 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:50:06.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:06.840+0000 7f26d1600700 1 -- 172.21.15.62:0/2493306687 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f26c8057ba0 con 0x7f26d4106b80 2024-04-20T21:50:07.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:07.096+0000 7f26da000700 1 -- 172.21.15.62:0/2493306687 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f26d41a1630 con 0x7f26bc06ecc0 2024-04-20T21:50:07.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:07.098+0000 7f26d1600700 1 -- 172.21.15.62:0/2493306687 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f26d41a1630 con 0x7f26bc06ecc0 2024-04-20T21:50:07.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:07.101+0000 7f26da000700 1 -- 172.21.15.62:0/2493306687 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f26bc06ecc0 msgr2=0x7f26bc071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:50:07.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:07.101+0000 7f26da000700 1 --2- 172.21.15.62:0/2493306687 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f26bc06ecc0 0x7f26bc071170 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f26c0006010 tx=0x7f26c000b540 comp rx=0 tx=0).stop 2024-04-20T21:50:07.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:07.101+0000 7f26da000700 1 -- 172.21.15.62:0/2493306687 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f26d4106b80 msgr2=0x7f26d419c1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:50:07.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:07.101+0000 7f26da000700 1 --2- 172.21.15.62:0/2493306687 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f26d4106b80 0x7f26d419c1e0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f26c800d8d0 tx=0x7f26c800dc90 comp rx=0 tx=0).stop 2024-04-20T21:50:07.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:07.101+0000 7f26da000700 1 -- 172.21.15.62:0/2493306687 shutdown_connections 2024-04-20T21:50:07.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:07.101+0000 7f26da000700 1 --2- 172.21.15.62:0/2493306687 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f26bc06ecc0 0x7f26bc071170 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:50:07.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:07.101+0000 7f26da000700 1 --2- 172.21.15.62:0/2493306687 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f26d4106b80 0x7f26d419c1e0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:50:07.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:07.101+0000 7f26da000700 1 --2- 172.21.15.62:0/2493306687 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26d410c660 0x7f26d419c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:50:07.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:07.101+0000 7f26da000700 1 -- 172.21.15.62:0/2493306687 >> 172.21.15.62:0/2493306687 conn(0x7f26d4075e90 msgr2=0x7f26d40778d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:50:07.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:07.102+0000 7f26da000700 1 -- 172.21.15.62:0/2493306687 shutdown_connections 2024-04-20T21:50:07.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:07.102+0000 7f26da000700 1 -- 172.21.15.62:0/2493306687 wait complete. 2024-04-20T21:50:07.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:07.262+0000 7fa3e9400700 1 -- 172.21.15.62:0/2073986994 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa3e4068540 msgr2=0x7fa3e40689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:50:07.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:07.262+0000 7fa3e9400700 1 --2- 172.21.15.62:0/2073986994 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa3e4068540 0x7fa3e40689b0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fa3d0009910 tx=0x7fa3d0009c20 comp rx=0 tx=0).stop 2024-04-20T21:50:07.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:07.263+0000 7fa3e9400700 1 -- 172.21.15.62:0/2073986994 shutdown_connections 2024-04-20T21:50:07.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:07.263+0000 7fa3e9400700 1 --2- 172.21.15.62:0/2073986994 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa3e4068540 0x7fa3e40689b0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:50:07.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:07.263+0000 7fa3e9400700 1 --2- 172.21.15.62:0/2073986994 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa3e41085b0 0x7fa3e4108980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:50:07.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:07.263+0000 7fa3e9400700 1 -- 172.21.15.62:0/2073986994 >> 172.21.15.62:0/2073986994 conn(0x7fa3e4075130 msgr2=0x7fa3e4075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:50:07.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:07.263+0000 7fa3e9400700 1 -- 172.21.15.62:0/2073986994 shutdown_connections 2024-04-20T21:50:07.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:07.263+0000 7fa3e9400700 1 -- 172.21.15.62:0/2073986994 wait complete. 2024-04-20T21:50:07.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:07.264+0000 7fa3e9400700 1 Processor -- start 2024-04-20T21:50:07.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:07.264+0000 7fa3e9400700 1 -- start start 2024-04-20T21:50:07.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:07.264+0000 7fa3e9400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa3e4068540 0x7fa3e41980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:50:07.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:07.264+0000 7fa3e9400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa3e41085b0 0x7fa3e41985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:50:07.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:07.264+0000 7fa3e9400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3e4198cd0 con 0x7fa3e4068540 2024-04-20T21:50:07.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:07.264+0000 7fa3e9400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3e419ca60 con 0x7fa3e41085b0 2024-04-20T21:50:07.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:07.265+0000 7fa3e2800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa3e4068540 0x7fa3e41980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:50:07.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:07.265+0000 7fa3e2800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa3e4068540 0x7fa3e41980b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43884/0 (socket says 172.21.15.62:43884) 2024-04-20T21:50:07.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:07.265+0000 7fa3e2800700 1 -- 172.21.15.62:0/635755056 learned_addr learned my addr 172.21.15.62:0/635755056 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:50:07.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:07.265+0000 7fa3e1e00700 1 --2- 172.21.15.62:0/635755056 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa3e41085b0 0x7fa3e41985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:50:07.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:07.265+0000 7fa3e2800700 1 -- 172.21.15.62:0/635755056 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa3e41085b0 msgr2=0x7fa3e41985f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:50:07.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:07.265+0000 7fa3e2800700 1 --2- 172.21.15.62:0/635755056 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa3e41085b0 0x7fa3e41985f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:50:07.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:07.265+0000 7fa3e2800700 1 -- 172.21.15.62:0/635755056 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3d00095f0 con 0x7fa3e4068540 2024-04-20T21:50:07.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:07.266+0000 7fa3e2800700 1 --2- 172.21.15.62:0/635755056 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa3e4068540 0x7fa3e41980b0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fa3d400d900 tx=0x7fa3d400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:50:07.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:07.266+0000 7fa3db400700 1 -- 172.21.15.62:0/635755056 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa3d40041f0 con 0x7fa3e4068540 2024-04-20T21:50:07.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:07.266+0000 7fa3db400700 1 -- 172.21.15.62:0/635755056 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa3d4004350 con 0x7fa3e4068540 2024-04-20T21:50:07.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:07.266+0000 7fa3db400700 1 -- 172.21.15.62:0/635755056 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa3d4003dc0 con 0x7fa3e4068540 2024-04-20T21:50:07.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:07.266+0000 7fa3e9400700 1 -- 172.21.15.62:0/635755056 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3e419cd40 con 0x7fa3e4068540 2024-04-20T21:50:07.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:07.266+0000 7fa3e9400700 1 -- 172.21.15.62:0/635755056 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3e419d290 con 0x7fa3e4068540 2024-04-20T21:50:07.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:07.267+0000 7fa3e9400700 1 -- 172.21.15.62:0/635755056 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3e410aa00 con 0x7fa3e4068540 2024-04-20T21:50:07.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:07.268+0000 7fa3db400700 1 -- 172.21.15.62:0/635755056 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fa3d4009730 con 0x7fa3e4068540 2024-04-20T21:50:07.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:07.269+0000 7fa3db400700 1 --2- 172.21.15.62:0/635755056 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa3c806ecc0 0x7fa3c8071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:50:07.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:07.269+0000 7fa3db400700 1 -- 172.21.15.62:0/635755056 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fa3d408f0b0 con 0x7fa3e4068540 2024-04-20T21:50:07.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:07.270+0000 7fa3e1e00700 1 --2- 172.21.15.62:0/635755056 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa3c806ecc0 0x7fa3c8071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:50:07.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:07.270+0000 7fa3e1e00700 1 --2- 172.21.15.62:0/635755056 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa3c806ecc0 0x7fa3c8071170 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fa3e41996d0 tx=0x7fa3d001a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:50:07.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:07.274+0000 7fa3db400700 1 -- 172.21.15.62:0/635755056 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fa3d4057dc0 con 0x7fa3e4068540 2024-04-20T21:50:07.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:07.520+0000 7fa3e9400700 1 -- 172.21.15.62:0/635755056 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fa3e419d510 con 0x7fa3c806ecc0 2024-04-20T21:50:07.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:07.554+0000 7fa3db400700 1 -- 172.21.15.62:0/635755056 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fa3e419d510 con 0x7fa3c806ecc0 2024-04-20T21:50:07.554 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T21:50:07.554 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (15m) 7m ago 29m 14.2M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T21:50:07.554 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (30m) 7m ago 30m 9512k - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T21:50:07.554 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (28m) 8m ago 28m 8942k - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T21:50:07.554 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (12m) 7m ago 30m 6660k - 19.0.0-3133-ga8c84f59 76bc6b91b726 2825d5ba11c6 2024-04-20T21:50:07.554 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (11m) 8m ago 28m 6665k - 19.0.0-3133-ga8c84f59 76bc6b91b726 20f746fa1bba 2024-04-20T21:50:07.555 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (15m) 7m ago 29m 74.2M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T21:50:07.555 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (7m) 7m ago 22m 11.2M - 19.0.0-3133-ga8c84f59 76bc6b91b726 de80bc90a683 2024-04-20T21:50:07.555 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (7m) 7m ago 22m 11.9M - 19.0.0-3133-ga8c84f59 76bc6b91b726 cd793b7ee98e 2024-04-20T21:50:07.555 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (22m) 8m ago 22m 117M - 18.2.0 dc2bc1663786 fb2a8a223db8 2024-04-20T21:50:07.555 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (22m) 8m ago 22m 55.6M - 18.2.0 dc2bc1663786 f2eec2ee8951 2024-04-20T21:50:07.555 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (18m) 7m ago 31m 511M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T21:50:07.555 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (17m) 8m ago 28m 452M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T21:50:07.555 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (13m) 7m ago 31m 55.9M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 019eb1479485 2024-04-20T21:50:07.555 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (13m) 8m ago 28m 39.8M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 55aac560aab6 2024-04-20T21:50:07.555 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (17m) 7m ago 30m 7952k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T21:50:07.555 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (17m) 8m ago 28m 8304k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T21:50:07.555 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (11m) 7m ago 27m 88.0M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 ee2bb901f26e 2024-04-20T21:50:07.555 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (10m) 7m ago 26m 92.3M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 9075152d4162 2024-04-20T21:50:07.555 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (10m) 7m ago 26m 74.4M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 2d4f3e38dc0e 2024-04-20T21:50:07.556 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (9m) 8m ago 25m 87.6M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 8f97b17ae0eb 2024-04-20T21:50:07.556 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (9m) 8m ago 25m 90.7M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 653d8079967d 2024-04-20T21:50:07.556 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (8m) 8m ago 24m 11.5M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 596dded14c12 2024-04-20T21:50:07.556 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (16m) 7m ago 29m 74.0M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T21:50:07.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:07.558+0000 7fa3e9400700 1 -- 172.21.15.62:0/635755056 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa3c806ecc0 msgr2=0x7fa3c8071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:50:07.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:07.558+0000 7fa3e9400700 1 --2- 172.21.15.62:0/635755056 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa3c806ecc0 0x7fa3c8071170 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fa3e41996d0 tx=0x7fa3d001a040 comp rx=0 tx=0).stop 2024-04-20T21:50:07.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:07.558+0000 7fa3e9400700 1 -- 172.21.15.62:0/635755056 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa3e4068540 msgr2=0x7fa3e41980b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:50:07.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:07.558+0000 7fa3e9400700 1 --2- 172.21.15.62:0/635755056 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa3e4068540 0x7fa3e41980b0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fa3d400d900 tx=0x7fa3d400dcc0 comp rx=0 tx=0).stop 2024-04-20T21:50:07.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:07.558+0000 7fa3e9400700 1 -- 172.21.15.62:0/635755056 shutdown_connections 2024-04-20T21:50:07.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:07.558+0000 7fa3e9400700 1 --2- 172.21.15.62:0/635755056 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa3c806ecc0 0x7fa3c8071170 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:50:07.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:07.558+0000 7fa3e9400700 1 --2- 172.21.15.62:0/635755056 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa3e4068540 0x7fa3e41980b0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:50:07.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:07.558+0000 7fa3e9400700 1 --2- 172.21.15.62:0/635755056 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa3e41085b0 0x7fa3e41985f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:50:07.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:07.558+0000 7fa3e9400700 1 -- 172.21.15.62:0/635755056 >> 172.21.15.62:0/635755056 conn(0x7fa3e4075130 msgr2=0x7fa3e4106390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:50:07.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:07.559+0000 7fa3e9400700 1 -- 172.21.15.62:0/635755056 shutdown_connections 2024-04-20T21:50:07.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:07.559+0000 7fa3e9400700 1 -- 172.21.15.62:0/635755056 wait complete. 2024-04-20T21:50:07.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:07.711+0000 7fccbae00700 1 -- 172.21.15.62:0/777718596 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fccb4073090 msgr2=0x7fccb4073460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:50:07.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:07.711+0000 7fccbae00700 1 --2- 172.21.15.62:0/777718596 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fccb4073090 0x7fccb4073460 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fcca4009940 tx=0x7fcca4009c50 comp rx=0 tx=0).stop 2024-04-20T21:50:07.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:07.711+0000 7fccbae00700 1 -- 172.21.15.62:0/777718596 shutdown_connections 2024-04-20T21:50:07.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:07.711+0000 7fccbae00700 1 --2- 172.21.15.62:0/777718596 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fccb40739a0 0x7fccb410c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:50:07.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:07.711+0000 7fccbae00700 1 --2- 172.21.15.62:0/777718596 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fccb4073090 0x7fccb4073460 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:50:07.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:07.711+0000 7fccbae00700 1 -- 172.21.15.62:0/777718596 >> 172.21.15.62:0/777718596 conn(0x7fccb40fbda0 msgr2=0x7fccb40fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:50:07.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:07.711+0000 7fccbae00700 1 -- 172.21.15.62:0/777718596 shutdown_connections 2024-04-20T21:50:07.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:07.711+0000 7fccbae00700 1 -- 172.21.15.62:0/777718596 wait complete. 2024-04-20T21:50:07.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:07.712+0000 7fccbae00700 1 Processor -- start 2024-04-20T21:50:07.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:07.712+0000 7fccbae00700 1 -- start start 2024-04-20T21:50:07.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:07.713+0000 7fccbae00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fccb4073090 0x7fccb41981c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:50:07.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:07.713+0000 7fccbae00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fccb40739a0 0x7fccb4198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:50:07.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:07.713+0000 7fccbae00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccb4198de0 con 0x7fccb4073090 2024-04-20T21:50:07.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:07.713+0000 7fccbae00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccb419cb70 con 0x7fccb40739a0 2024-04-20T21:50:07.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:07.713+0000 7fccb3e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fccb4073090 0x7fccb41981c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:50:07.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:07.713+0000 7fccb3e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fccb4073090 0x7fccb41981c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43912/0 (socket says 172.21.15.62:43912) 2024-04-20T21:50:07.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:07.713+0000 7fccb3e00700 1 -- 172.21.15.62:0/1568855213 learned_addr learned my addr 172.21.15.62:0/1568855213 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:50:07.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:07.714+0000 7fccb3e00700 1 -- 172.21.15.62:0/1568855213 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fccb40739a0 msgr2=0x7fccb4198700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:50:07.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:07.714+0000 7fccabe00700 1 --2- 172.21.15.62:0/1568855213 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fccb40739a0 0x7fccb4198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:50:07.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:07.714+0000 7fccb3e00700 1 --2- 172.21.15.62:0/1568855213 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fccb40739a0 0x7fccb4198700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:50:07.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:07.714+0000 7fccb3e00700 1 -- 172.21.15.62:0/1568855213 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcca40095f0 con 0x7fccb4073090 2024-04-20T21:50:07.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:07.714+0000 7fccb3e00700 1 --2- 172.21.15.62:0/1568855213 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fccb4073090 0x7fccb41981c0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fcca400b870 tx=0x7fcca40056a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:50:07.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:07.714+0000 7fccb1600700 1 -- 172.21.15.62:0/1568855213 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcca401d070 con 0x7fccb4073090 2024-04-20T21:50:07.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:07.714+0000 7fccb1600700 1 -- 172.21.15.62:0/1568855213 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcca400f460 con 0x7fccb4073090 2024-04-20T21:50:07.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:07.714+0000 7fccbae00700 1 -- 172.21.15.62:0/1568855213 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccb419cdf0 con 0x7fccb4073090 2024-04-20T21:50:07.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:07.714+0000 7fccb1600700 1 -- 172.21.15.62:0/1568855213 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcca4021640 con 0x7fccb4073090 2024-04-20T21:50:07.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:07.714+0000 7fccbae00700 1 -- 172.21.15.62:0/1568855213 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccb419d2e0 con 0x7fccb4073090 2024-04-20T21:50:07.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:07.715+0000 7fccbae00700 1 -- 172.21.15.62:0/1568855213 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccb4109ce0 con 0x7fccb4073090 2024-04-20T21:50:07.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:07.717+0000 7fccb1600700 1 -- 172.21.15.62:0/1568855213 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fcca400fa60 con 0x7fccb4073090 2024-04-20T21:50:07.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:07.718+0000 7fccb1600700 1 --2- 172.21.15.62:0/1568855213 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fcc9806ecc0 0x7fcc98071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:50:07.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:07.718+0000 7fccb1600700 1 -- 172.21.15.62:0/1568855213 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fcca40901d0 con 0x7fccb4073090 2024-04-20T21:50:07.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:07.718+0000 7fccabe00700 1 --2- 172.21.15.62:0/1568855213 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fcc9806ecc0 0x7fcc98071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:50:07.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:07.719+0000 7fccabe00700 1 --2- 172.21.15.62:0/1568855213 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fcc9806ecc0 0x7fcc98071170 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fcc9c006fd0 tx=0x7fcc9c008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:50:07.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:07.723+0000 7fccb1600700 1 -- 172.21.15.62:0/1568855213 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fcca4058ee0 con 0x7fccb4073090 2024-04-20T21:50:08.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.034+0000 7fccbae00700 1 -- 172.21.15.62:0/1568855213 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fccb419d610 con 0x7fccb4073090 2024-04-20T21:50:08.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.035+0000 7fccb1600700 1 -- 172.21.15.62:0/1568855213 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7fcca405c5a0 con 0x7fccb4073090 2024-04-20T21:50:08.034 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:50:08.034 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T21:50:08.035 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:50:08.035 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:50:08.035 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T21:50:08.035 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:50:08.035 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:50:08.035 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T21:50:08.035 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 6 2024-04-20T21:50:08.035 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:50:08.035 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T21:50:08.035 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 2, 2024-04-20T21:50:08.035 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:50:08.035 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:50:08.035 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T21:50:08.035 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 2, 2024-04-20T21:50:08.035 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 12 2024-04-20T21:50:08.036 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T21:50:08.036 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:50:08.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.038+0000 7fccbae00700 1 -- 172.21.15.62:0/1568855213 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fcc9806ecc0 msgr2=0x7fcc98071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:50:08.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.038+0000 7fccbae00700 1 --2- 172.21.15.62:0/1568855213 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fcc9806ecc0 0x7fcc98071170 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fcc9c006fd0 tx=0x7fcc9c008040 comp rx=0 tx=0).stop 2024-04-20T21:50:08.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.038+0000 7fccbae00700 1 -- 172.21.15.62:0/1568855213 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fccb4073090 msgr2=0x7fccb41981c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:50:08.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.038+0000 7fccbae00700 1 --2- 172.21.15.62:0/1568855213 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fccb4073090 0x7fccb41981c0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fcca400b870 tx=0x7fcca40056a0 comp rx=0 tx=0).stop 2024-04-20T21:50:08.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.038+0000 7fccbae00700 1 -- 172.21.15.62:0/1568855213 shutdown_connections 2024-04-20T21:50:08.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.038+0000 7fccbae00700 1 --2- 172.21.15.62:0/1568855213 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fcc9806ecc0 0x7fcc98071170 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:50:08.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.038+0000 7fccbae00700 1 --2- 172.21.15.62:0/1568855213 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fccb4073090 0x7fccb41981c0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:50:08.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.038+0000 7fccbae00700 1 --2- 172.21.15.62:0/1568855213 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fccb40739a0 0x7fccb4198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:50:08.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.038+0000 7fccbae00700 1 -- 172.21.15.62:0/1568855213 >> 172.21.15.62:0/1568855213 conn(0x7fccb40fbda0 msgr2=0x7fccb4106e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:50:08.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.039+0000 7fccbae00700 1 -- 172.21.15.62:0/1568855213 shutdown_connections 2024-04-20T21:50:08.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.039+0000 7fccbae00700 1 -- 172.21.15.62:0/1568855213 wait complete. 2024-04-20T21:50:08.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.203+0000 7fe329a00700 1 -- 172.21.15.62:0/3683936355 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe324100e80 msgr2=0x7fe324104e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:50:08.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.203+0000 7fe329a00700 1 --2- 172.21.15.62:0/3683936355 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe324100e80 0x7fe324104e50 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fe310009910 tx=0x7fe310009c20 comp rx=0 tx=0).stop 2024-04-20T21:50:08.202 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:50:07 smithi062 ceph-mon[76322]: pgmap v444: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:50:08.203 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:50:07 smithi062 ceph-mon[76322]: from='client.34844 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:50:08.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.203+0000 7fe329a00700 1 -- 172.21.15.62:0/3683936355 shutdown_connections 2024-04-20T21:50:08.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.203+0000 7fe329a00700 1 --2- 172.21.15.62:0/3683936355 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe324100e80 0x7fe324104e50 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:50:08.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.203+0000 7fe329a00700 1 --2- 172.21.15.62:0/3683936355 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3241004e0 0x7fe3241008b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:50:08.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.203+0000 7fe329a00700 1 -- 172.21.15.62:0/3683936355 >> 172.21.15.62:0/3683936355 conn(0x7fe3240fbda0 msgr2=0x7fe3240fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:50:08.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.203+0000 7fe329a00700 1 -- 172.21.15.62:0/3683936355 shutdown_connections 2024-04-20T21:50:08.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.203+0000 7fe329a00700 1 -- 172.21.15.62:0/3683936355 wait complete. 2024-04-20T21:50:08.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.204+0000 7fe329a00700 1 Processor -- start 2024-04-20T21:50:08.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.204+0000 7fe329a00700 1 -- start start 2024-04-20T21:50:08.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.205+0000 7fe329a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe3241004e0 0x7fe324198130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:50:08.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.205+0000 7fe329a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe324100e80 0x7fe324198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:50:08.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.205+0000 7fe329a00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe324198d50 con 0x7fe3241004e0 2024-04-20T21:50:08.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.205+0000 7fe329a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe32419cae0 con 0x7fe324100e80 2024-04-20T21:50:08.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.205+0000 7fe322a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe3241004e0 0x7fe324198130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:50:08.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.205+0000 7fe322a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe3241004e0 0x7fe324198130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43936/0 (socket says 172.21.15.62:43936) 2024-04-20T21:50:08.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.205+0000 7fe322a00700 1 -- 172.21.15.62:0/3099570570 learned_addr learned my addr 172.21.15.62:0/3099570570 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:50:08.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.206+0000 7fe322a00700 1 -- 172.21.15.62:0/3099570570 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe324100e80 msgr2=0x7fe324198670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:50:08.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.206+0000 7fe322000700 1 --2- 172.21.15.62:0/3099570570 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe324100e80 0x7fe324198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:50:08.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.206+0000 7fe322a00700 1 --2- 172.21.15.62:0/3099570570 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe324100e80 0x7fe324198670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:50:08.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.206+0000 7fe322a00700 1 -- 172.21.15.62:0/3099570570 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3100095f0 con 0x7fe3241004e0 2024-04-20T21:50:08.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.206+0000 7fe322a00700 1 --2- 172.21.15.62:0/3099570570 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe3241004e0 0x7fe324198130 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fe3140060b0 tx=0x7fe31400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:50:08.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.206+0000 7fe31b400700 1 -- 172.21.15.62:0/3099570570 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe31400dda0 con 0x7fe3241004e0 2024-04-20T21:50:08.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.206+0000 7fe31b400700 1 -- 172.21.15.62:0/3099570570 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe31400f040 con 0x7fe3241004e0 2024-04-20T21:50:08.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.206+0000 7fe329a00700 1 -- 172.21.15.62:0/3099570570 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe32419cdc0 con 0x7fe3241004e0 2024-04-20T21:50:08.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.206+0000 7fe31b400700 1 -- 172.21.15.62:0/3099570570 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe314014790 con 0x7fe3241004e0 2024-04-20T21:50:08.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.207+0000 7fe329a00700 1 -- 172.21.15.62:0/3099570570 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe32419d310 con 0x7fe3241004e0 2024-04-20T21:50:08.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.208+0000 7fe329a00700 1 -- 172.21.15.62:0/3099570570 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3241045d0 con 0x7fe3241004e0 2024-04-20T21:50:08.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.209+0000 7fe31b400700 1 -- 172.21.15.62:0/3099570570 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe314014980 con 0x7fe3241004e0 2024-04-20T21:50:08.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.210+0000 7fe31b400700 1 --2- 172.21.15.62:0/3099570570 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe30806ecc0 0x7fe308071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:50:08.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.211+0000 7fe31b400700 1 -- 172.21.15.62:0/3099570570 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fe31408e900 con 0x7fe3241004e0 2024-04-20T21:50:08.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.211+0000 7fe322000700 1 --2- 172.21.15.62:0/3099570570 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe30806ecc0 0x7fe308071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:50:08.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.212+0000 7fe322000700 1 --2- 172.21.15.62:0/3099570570 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe30806ecc0 0x7fe308071170 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fe310006010 tx=0x7fe31000b540 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:50:08.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.215+0000 7fe31b400700 1 -- 172.21.15.62:0/3099570570 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fe314057760 con 0x7fe3241004e0 2024-04-20T21:50:08.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:50:07 smithi189 ceph-mon[57022]: pgmap v444: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:50:08.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:50:07 smithi189 ceph-mon[57022]: from='client.34844 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:50:08.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.508+0000 7fe329a00700 1 -- 172.21.15.62:0/3099570570 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fe32419d590 con 0x7fe3241004e0 2024-04-20T21:50:08.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.509+0000 7fe31b400700 1 -- 172.21.15.62:0/3099570570 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 25 v25) v1 ==== 76+0+1887 (secure 0 0 0) 0x7fe31405ae20 con 0x7fe3241004e0 2024-04-20T21:50:08.509 INFO:teuthology.orchestra.run.smithi062.stdout:e25 2024-04-20T21:50:08.509 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T21:50:08.509 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T21:50:08.510 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T21:50:08.510 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:50:08.510 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T21:50:08.510 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T21:50:08.510 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 20 2024-04-20T21:50:08.510 INFO:teuthology.orchestra.run.smithi062.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-20T21:50:08.510 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T21:50:08.510 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T21:42:01.559061+0000 2024-04-20T21:50:08.510 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T21:50:08.510 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T21:50:08.510 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T21:50:08.510 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T21:50:08.510 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T21:50:08.510 INFO:teuthology.orchestra.run.smithi062.stdout:max_xattr_size 65536 2024-04-20T21:50:08.511 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T21:50:08.511 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T21:50:08.511 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 81 2024-04-20T21:50:08.511 INFO:teuthology.orchestra.run.smithi062.stdout:snap_rstat 1 2024-04-20T21:50:08.511 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T21:50:08.511 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 2 2024-04-20T21:50:08.511 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1 2024-04-20T21:50:08.511 INFO:teuthology.orchestra.run.smithi062.stdout:up {} 2024-04-20T21:50:08.511 INFO:teuthology.orchestra.run.smithi062.stdout:failed 0,1 2024-04-20T21:50:08.511 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T21:50:08.511 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T21:50:08.511 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T21:50:08.511 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T21:50:08.511 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data enabled 2024-04-20T21:50:08.512 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T21:50:08.512 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T21:50:08.512 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 1 2024-04-20T21:50:08.512 INFO:teuthology.orchestra.run.smithi062.stdout:qdb_cluster leader: 0 members: 2024-04-20T21:50:08.512 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:50:08.512 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:50:08.512 INFO:teuthology.orchestra.run.smithi062.stdout:Standby daemons: 2024-04-20T21:50:08.512 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:50:08.512 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ryqblb{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6826/4113916205,v1:172.21.15.62:6827/4113916205] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T21:50:08.512 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ozizwi{-1:34476} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6828/3801975165,v1:172.21.15.62:6829/3801975165] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T21:50:08.512 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.xbgspw{-1:44285} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6826/3996040526,v1:172.21.15.189:6827/3996040526] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:50:08.512 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.dvimer{-1:44289} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6824/1743111843,v1:172.21.15.189:6825/1743111843] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:50:08.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.512+0000 7fe329a00700 1 -- 172.21.15.62:0/3099570570 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe30806ecc0 msgr2=0x7fe308071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:50:08.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.512+0000 7fe329a00700 1 --2- 172.21.15.62:0/3099570570 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe30806ecc0 0x7fe308071170 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fe310006010 tx=0x7fe31000b540 comp rx=0 tx=0).stop 2024-04-20T21:50:08.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.512+0000 7fe329a00700 1 -- 172.21.15.62:0/3099570570 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe3241004e0 msgr2=0x7fe324198130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:50:08.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.512+0000 7fe329a00700 1 --2- 172.21.15.62:0/3099570570 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe3241004e0 0x7fe324198130 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fe3140060b0 tx=0x7fe31400d3b0 comp rx=0 tx=0).stop 2024-04-20T21:50:08.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.512+0000 7fe329a00700 1 -- 172.21.15.62:0/3099570570 shutdown_connections 2024-04-20T21:50:08.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.512+0000 7fe329a00700 1 --2- 172.21.15.62:0/3099570570 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe30806ecc0 0x7fe308071170 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:50:08.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.512+0000 7fe329a00700 1 --2- 172.21.15.62:0/3099570570 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe3241004e0 0x7fe324198130 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:50:08.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.512+0000 7fe329a00700 1 --2- 172.21.15.62:0/3099570570 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe324100e80 0x7fe324198670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:50:08.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.512+0000 7fe329a00700 1 -- 172.21.15.62:0/3099570570 >> 172.21.15.62:0/3099570570 conn(0x7fe3240fbda0 msgr2=0x7fe3240fe160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:50:08.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.512+0000 7fe329a00700 1 -- 172.21.15.62:0/3099570570 shutdown_connections 2024-04-20T21:50:08.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.512+0000 7fe329a00700 1 -- 172.21.15.62:0/3099570570 wait complete. 2024-04-20T21:50:08.514 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 25 2024-04-20T21:50:08.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.676+0000 7f9044800700 1 -- 172.21.15.62:0/1829630404 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f903c1004e0 msgr2=0x7f903c1008b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:50:08.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.676+0000 7f9044800700 1 --2- 172.21.15.62:0/1829630404 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f903c1004e0 0x7f903c1008b0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f9030009910 tx=0x7f9030009c20 comp rx=0 tx=0).stop 2024-04-20T21:50:08.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.676+0000 7f9044800700 1 -- 172.21.15.62:0/1829630404 shutdown_connections 2024-04-20T21:50:08.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.676+0000 7f9044800700 1 --2- 172.21.15.62:0/1829630404 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f903c100e80 0x7f903c104e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:50:08.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.676+0000 7f9044800700 1 --2- 172.21.15.62:0/1829630404 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f903c1004e0 0x7f903c1008b0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:50:08.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.676+0000 7f9044800700 1 -- 172.21.15.62:0/1829630404 >> 172.21.15.62:0/1829630404 conn(0x7f903c0fbda0 msgr2=0x7f903c0fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:50:08.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.676+0000 7f9044800700 1 -- 172.21.15.62:0/1829630404 shutdown_connections 2024-04-20T21:50:08.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.676+0000 7f9044800700 1 -- 172.21.15.62:0/1829630404 wait complete. 2024-04-20T21:50:08.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.678+0000 7f9044800700 1 Processor -- start 2024-04-20T21:50:08.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.678+0000 7f9044800700 1 -- start start 2024-04-20T21:50:08.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.678+0000 7f9044800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f903c1004e0 0x7f903c1981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:50:08.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.678+0000 7f9044800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f903c100e80 0x7f903c198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:50:08.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.678+0000 7f9044800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f903c198e10 con 0x7f903c1004e0 2024-04-20T21:50:08.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.678+0000 7f9044800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f903c19cba0 con 0x7f903c100e80 2024-04-20T21:50:08.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.679+0000 7f9041e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f903c1004e0 0x7f903c1981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:50:08.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.679+0000 7f9041e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f903c1004e0 0x7f903c1981f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43956/0 (socket says 172.21.15.62:43956) 2024-04-20T21:50:08.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.679+0000 7f9041e00700 1 -- 172.21.15.62:0/813073565 learned_addr learned my addr 172.21.15.62:0/813073565 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:50:08.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.679+0000 7f9041400700 1 --2- 172.21.15.62:0/813073565 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f903c100e80 0x7f903c198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:50:08.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.679+0000 7f9041e00700 1 -- 172.21.15.62:0/813073565 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f903c100e80 msgr2=0x7f903c198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:50:08.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.679+0000 7f9041e00700 1 --2- 172.21.15.62:0/813073565 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f903c100e80 0x7f903c198730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:50:08.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.679+0000 7f9041e00700 1 -- 172.21.15.62:0/813073565 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90300095f0 con 0x7f903c1004e0 2024-04-20T21:50:08.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.679+0000 7f9041400700 1 --2- 172.21.15.62:0/813073565 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f903c100e80 0x7f903c198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T21:50:08.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.679+0000 7f9041e00700 1 --2- 172.21.15.62:0/813073565 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f903c1004e0 0x7f903c1981f0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f9030006010 tx=0x7f903000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:50:08.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.680+0000 7f9036a00700 1 -- 172.21.15.62:0/813073565 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f903001d070 con 0x7f903c1004e0 2024-04-20T21:50:08.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.680+0000 7f9036a00700 1 -- 172.21.15.62:0/813073565 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9030004fb0 con 0x7f903c1004e0 2024-04-20T21:50:08.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.680+0000 7f9044800700 1 -- 172.21.15.62:0/813073565 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f903c19ce20 con 0x7f903c1004e0 2024-04-20T21:50:08.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.680+0000 7f9036a00700 1 -- 172.21.15.62:0/813073565 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9030022ea0 con 0x7f903c1004e0 2024-04-20T21:50:08.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.680+0000 7f9044800700 1 -- 172.21.15.62:0/813073565 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f903c19d310 con 0x7f903c1004e0 2024-04-20T21:50:08.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.681+0000 7f9044800700 1 -- 172.21.15.62:0/813073565 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f903c1045d0 con 0x7f903c1004e0 2024-04-20T21:50:08.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.682+0000 7f9036a00700 1 -- 172.21.15.62:0/813073565 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f9030022470 con 0x7f903c1004e0 2024-04-20T21:50:08.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.683+0000 7f9036a00700 1 --2- 172.21.15.62:0/813073565 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f902406ecc0 0x7f9024071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:50:08.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.684+0000 7f9036a00700 1 -- 172.21.15.62:0/813073565 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f90300904a0 con 0x7f903c1004e0 2024-04-20T21:50:08.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.684+0000 7f9041400700 1 --2- 172.21.15.62:0/813073565 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f902406ecc0 0x7f9024071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:50:08.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.684+0000 7f9041400700 1 --2- 172.21.15.62:0/813073565 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f902406ecc0 0x7f9024071170 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f903c199810 tx=0x7f902c008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:50:08.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.688+0000 7f9036a00700 1 -- 172.21.15.62:0/813073565 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f9030059260 con 0x7f903c1004e0 2024-04-20T21:50:08.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.952+0000 7f9044800700 1 -- 172.21.15.62:0/813073565 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f903c19d5f0 con 0x7f902406ecc0 2024-04-20T21:50:08.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.954+0000 7f9036a00700 1 -- 172.21.15.62:0/813073565 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f903c19d5f0 con 0x7f902406ecc0 2024-04-20T21:50:08.953 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:50:08.954 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T21:50:08.954 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T21:50:08.954 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-20T21:50:08.954 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T21:50:08.954 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr", 2024-04-20T21:50:08.954 INFO:teuthology.orchestra.run.smithi062.stdout: "crash", 2024-04-20T21:50:08.954 INFO:teuthology.orchestra.run.smithi062.stdout: "mon", 2024-04-20T21:50:08.954 INFO:teuthology.orchestra.run.smithi062.stdout: "osd" 2024-04-20T21:50:08.954 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T21:50:08.954 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "14/23 daemons upgraded", 2024-04-20T21:50:08.954 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading mds daemons", 2024-04-20T21:50:08.954 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T21:50:08.955 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:50:08.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.957+0000 7f9044800700 1 -- 172.21.15.62:0/813073565 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f902406ecc0 msgr2=0x7f9024071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:50:08.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.957+0000 7f9044800700 1 --2- 172.21.15.62:0/813073565 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f902406ecc0 0x7f9024071170 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f903c199810 tx=0x7f902c008040 comp rx=0 tx=0).stop 2024-04-20T21:50:08.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.957+0000 7f9044800700 1 -- 172.21.15.62:0/813073565 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f903c1004e0 msgr2=0x7f903c1981f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:50:08.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.957+0000 7f9044800700 1 --2- 172.21.15.62:0/813073565 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f903c1004e0 0x7f903c1981f0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f9030006010 tx=0x7f903000be20 comp rx=0 tx=0).stop 2024-04-20T21:50:08.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.958+0000 7f9044800700 1 -- 172.21.15.62:0/813073565 shutdown_connections 2024-04-20T21:50:08.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.958+0000 7f9044800700 1 --2- 172.21.15.62:0/813073565 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f902406ecc0 0x7f9024071170 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:50:08.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.958+0000 7f9044800700 1 --2- 172.21.15.62:0/813073565 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f903c1004e0 0x7f903c1981f0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:50:08.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.958+0000 7f9044800700 1 --2- 172.21.15.62:0/813073565 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f903c100e80 0x7f903c198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:50:08.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.958+0000 7f9044800700 1 -- 172.21.15.62:0/813073565 >> 172.21.15.62:0/813073565 conn(0x7f903c0fbda0 msgr2=0x7f903c0fd450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:50:08.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.958+0000 7f9044800700 1 -- 172.21.15.62:0/813073565 shutdown_connections 2024-04-20T21:50:08.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:08.958+0000 7f9044800700 1 -- 172.21.15.62:0/813073565 wait complete. 2024-04-20T21:50:09.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:09.125+0000 7fcd35600700 1 -- 172.21.15.62:0/1187820892 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcd301004e0 msgr2=0x7fcd301008b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:50:09.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:09.125+0000 7fcd35600700 1 --2- 172.21.15.62:0/1187820892 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcd301004e0 0x7fcd301008b0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fcd20009910 tx=0x7fcd20009c20 comp rx=0 tx=0).stop 2024-04-20T21:50:09.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:09.125+0000 7fcd35600700 1 -- 172.21.15.62:0/1187820892 shutdown_connections 2024-04-20T21:50:09.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:09.125+0000 7fcd35600700 1 --2- 172.21.15.62:0/1187820892 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcd30100e80 0x7fcd30104e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:50:09.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:09.125+0000 7fcd35600700 1 --2- 172.21.15.62:0/1187820892 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcd301004e0 0x7fcd301008b0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:50:09.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:09.125+0000 7fcd35600700 1 -- 172.21.15.62:0/1187820892 >> 172.21.15.62:0/1187820892 conn(0x7fcd300fbda0 msgr2=0x7fcd300fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:50:09.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:09.126+0000 7fcd35600700 1 -- 172.21.15.62:0/1187820892 shutdown_connections 2024-04-20T21:50:09.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:09.126+0000 7fcd35600700 1 -- 172.21.15.62:0/1187820892 wait complete. 2024-04-20T21:50:09.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:09.127+0000 7fcd35600700 1 Processor -- start 2024-04-20T21:50:09.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:09.127+0000 7fcd35600700 1 -- start start 2024-04-20T21:50:09.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:09.127+0000 7fcd35600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcd301004e0 0x7fcd301980f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:50:09.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:09.127+0000 7fcd35600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcd30100e80 0x7fcd30198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:50:09.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:09.127+0000 7fcd35600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd30198d10 con 0x7fcd301004e0 2024-04-20T21:50:09.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:09.127+0000 7fcd35600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd3019caa0 con 0x7fcd30100e80 2024-04-20T21:50:09.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:09.128+0000 7fcd2ea00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcd301004e0 0x7fcd301980f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:50:09.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:09.128+0000 7fcd2ea00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcd301004e0 0x7fcd301980f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43978/0 (socket says 172.21.15.62:43978) 2024-04-20T21:50:09.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:09.128+0000 7fcd2ea00700 1 -- 172.21.15.62:0/1165731012 learned_addr learned my addr 172.21.15.62:0/1165731012 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:50:09.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:09.128+0000 7fcd2e000700 1 --2- 172.21.15.62:0/1165731012 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcd30100e80 0x7fcd30198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:50:09.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:09.128+0000 7fcd2ea00700 1 -- 172.21.15.62:0/1165731012 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcd30100e80 msgr2=0x7fcd30198630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:50:09.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:09.128+0000 7fcd2ea00700 1 --2- 172.21.15.62:0/1165731012 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcd30100e80 0x7fcd30198630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:50:09.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:09.128+0000 7fcd2ea00700 1 -- 172.21.15.62:0/1165731012 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd200095f0 con 0x7fcd301004e0 2024-04-20T21:50:09.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:09.128+0000 7fcd2e000700 1 --2- 172.21.15.62:0/1165731012 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcd30100e80 0x7fcd30198630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T21:50:09.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:09.129+0000 7fcd2ea00700 1 --2- 172.21.15.62:0/1165731012 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcd301004e0 0x7fcd301980f0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fcd2000b5c0 tx=0x7fcd2000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:50:09.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:09.129+0000 7fcd27400700 1 -- 172.21.15.62:0/1165731012 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcd2001d070 con 0x7fcd301004e0 2024-04-20T21:50:09.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:09.129+0000 7fcd27400700 1 -- 172.21.15.62:0/1165731012 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcd20005e00 con 0x7fcd301004e0 2024-04-20T21:50:09.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:09.129+0000 7fcd27400700 1 -- 172.21.15.62:0/1165731012 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcd20017400 con 0x7fcd301004e0 2024-04-20T21:50:09.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:09.129+0000 7fcd35600700 1 -- 172.21.15.62:0/1165731012 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd3019cd20 con 0x7fcd301004e0 2024-04-20T21:50:09.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:09.129+0000 7fcd35600700 1 -- 172.21.15.62:0/1165731012 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd3019d210 con 0x7fcd301004e0 2024-04-20T21:50:09.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:09.130+0000 7fcd35600700 1 -- 172.21.15.62:0/1165731012 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd301045d0 con 0x7fcd301004e0 2024-04-20T21:50:09.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:09.131+0000 7fcd27400700 1 -- 172.21.15.62:0/1165731012 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fcd2000f460 con 0x7fcd301004e0 2024-04-20T21:50:09.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:09.132+0000 7fcd27400700 1 --2- 172.21.15.62:0/1165731012 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fcd1406ecc0 0x7fcd14071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:50:09.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:09.132+0000 7fcd27400700 1 -- 172.21.15.62:0/1165731012 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fcd20090480 con 0x7fcd301004e0 2024-04-20T21:50:09.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:09.133+0000 7fcd2e000700 1 --2- 172.21.15.62:0/1165731012 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fcd1406ecc0 0x7fcd14071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:50:09.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:09.133+0000 7fcd2e000700 1 --2- 172.21.15.62:0/1165731012 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fcd1406ecc0 0x7fcd14071170 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fcd1c005fd0 tx=0x7fcd1c005ee0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:50:09.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:09.137+0000 7fcd27400700 1 -- 172.21.15.62:0/1165731012 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fcd20059190 con 0x7fcd301004e0 2024-04-20T21:50:09.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:50:08 smithi189 ceph-mon[57022]: from='client.34848 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:50:09.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:50:08 smithi189 ceph-mon[57022]: from='client.34852 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:50:09.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:50:08 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/1568855213' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:50:09.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:50:08 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/3099570570' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T21:50:09.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:50:08 smithi062 ceph-mon[76322]: from='client.34848 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:50:09.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:50:08 smithi062 ceph-mon[76322]: from='client.34852 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:50:09.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:50:08 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/1568855213' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:50:09.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:50:08 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/3099570570' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T21:50:09.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:09.471+0000 7fcd35600700 1 -- 172.21.15.62:0/1165731012 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fcd30103fb0 con 0x7fcd301004e0 2024-04-20T21:50:09.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:09.471+0000 7fcd27400700 1 -- 172.21.15.62:0/1165731012 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7fcd2005c850 con 0x7fcd301004e0 2024-04-20T21:50:09.471 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-20T21:50:09.471 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-20T21:50:09.471 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is degraded 2024-04-20T21:50:09.471 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T21:50:09.471 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T21:50:09.471 INFO:teuthology.orchestra.run.smithi062.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-20T21:50:09.471 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-20T21:50:09.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:09.474+0000 7fcd35600700 1 -- 172.21.15.62:0/1165731012 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fcd1406ecc0 msgr2=0x7fcd14071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:50:09.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:09.474+0000 7fcd35600700 1 --2- 172.21.15.62:0/1165731012 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fcd1406ecc0 0x7fcd14071170 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fcd1c005fd0 tx=0x7fcd1c005ee0 comp rx=0 tx=0).stop 2024-04-20T21:50:09.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:09.474+0000 7fcd35600700 1 -- 172.21.15.62:0/1165731012 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcd301004e0 msgr2=0x7fcd301980f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:50:09.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:09.474+0000 7fcd35600700 1 --2- 172.21.15.62:0/1165731012 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcd301004e0 0x7fcd301980f0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fcd2000b5c0 tx=0x7fcd2000be20 comp rx=0 tx=0).stop 2024-04-20T21:50:09.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:09.474+0000 7fcd35600700 1 -- 172.21.15.62:0/1165731012 shutdown_connections 2024-04-20T21:50:09.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:09.474+0000 7fcd35600700 1 --2- 172.21.15.62:0/1165731012 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fcd1406ecc0 0x7fcd14071170 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:50:09.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:09.474+0000 7fcd35600700 1 --2- 172.21.15.62:0/1165731012 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcd301004e0 0x7fcd301980f0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:50:09.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:09.474+0000 7fcd35600700 1 --2- 172.21.15.62:0/1165731012 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcd30100e80 0x7fcd30198630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:50:09.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:09.474+0000 7fcd35600700 1 -- 172.21.15.62:0/1165731012 >> 172.21.15.62:0/1165731012 conn(0x7fcd300fbda0 msgr2=0x7fcd300fe160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:50:09.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:09.475+0000 7fcd35600700 1 -- 172.21.15.62:0/1165731012 shutdown_connections 2024-04-20T21:50:09.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:09.475+0000 7fcd35600700 1 -- 172.21.15.62:0/1165731012 wait complete. 2024-04-20T21:50:10.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:50:09 smithi189 ceph-mon[57022]: pgmap v445: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:50:10.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:50:09 smithi189 ceph-mon[57022]: from='client.34864 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:50:10.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:50:09 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/1165731012' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T21:50:10.284 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:50:09 smithi062 ceph-mon[76322]: pgmap v445: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:50:10.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:50:09 smithi062 ceph-mon[76322]: from='client.34864 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:50:10.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:50:09 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/1165731012' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T21:50:12.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:50:11 smithi189 ceph-mon[57022]: pgmap v446: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:50:12.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:50:11 smithi062 ceph-mon[76322]: pgmap v446: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:50:14.236 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:50:13 smithi189 ceph-mon[57022]: pgmap v447: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:50:14.284 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:50:13 smithi062 ceph-mon[76322]: pgmap v447: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:50:16.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:50:15 smithi189 ceph-mon[57022]: pgmap v448: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:50:16.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:50:15 smithi062 ceph-mon[76322]: pgmap v448: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:50:18.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:50:17 smithi189 ceph-mon[57022]: pgmap v449: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:50:18.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:50:17 smithi062 ceph-mon[76322]: pgmap v449: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:50:20.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:50:19 smithi189 ceph-mon[57022]: pgmap v450: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:50:20.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:50:19 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:50:20.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:50:19 smithi062 ceph-mon[76322]: pgmap v450: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:50:20.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:50:19 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:50:22.284 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:50:21 smithi062 ceph-mon[76322]: pgmap v451: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:50:22.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:50:21 smithi189 ceph-mon[57022]: pgmap v451: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:50:24.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:50:23 smithi062 ceph-mon[76322]: pgmap v452: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:50:24.486 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:50:23 smithi189 ceph-mon[57022]: pgmap v452: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:50:25.736 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:50:25 smithi189 ceph-mon[57022]: pgmap v453: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:50:25.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:50:25 smithi062 ceph-mon[76322]: pgmap v453: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:50:27.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:50:27 smithi189 ceph-mon[57022]: pgmap v454: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:50:28.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:50:27 smithi062 ceph-mon[76322]: pgmap v454: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:50:29.986 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:50:29 smithi189 ceph-mon[57022]: pgmap v455: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:50:30.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:50:29 smithi062 ceph-mon[76322]: pgmap v455: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:50:31.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:50:31 smithi189 ceph-mon[57022]: pgmap v456: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:50:32.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:50:31 smithi062 ceph-mon[76322]: pgmap v456: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:50:33.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:50:33 smithi189 ceph-mon[57022]: pgmap v457: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:50:34.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:50:33 smithi062 ceph-mon[76322]: pgmap v457: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:50:35.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:50:35 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:50:35.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:50:35 smithi189 ceph-mon[57022]: pgmap v458: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:50:35.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:50:35 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:50:35.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:50:35 smithi062 ceph-mon[76322]: pgmap v458: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:50:37.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:50:37 smithi189 ceph-mon[57022]: pgmap v459: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:50:38.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:50:37 smithi062 ceph-mon[76322]: pgmap v459: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:50:39.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:39.655+0000 7f1ad6800700 1 -- 172.21.15.62:0/320280996 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1ad00730f0 msgr2=0x7f1ad00734c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:50:39.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:39.655+0000 7f1ad6800700 1 --2- 172.21.15.62:0/320280996 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1ad00730f0 0x7f1ad00734c0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f1ac0009910 tx=0x7f1ac0009c20 comp rx=0 tx=0).stop 2024-04-20T21:50:39.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:39.655+0000 7f1ad6800700 1 -- 172.21.15.62:0/320280996 shutdown_connections 2024-04-20T21:50:39.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:39.655+0000 7f1ad6800700 1 --2- 172.21.15.62:0/320280996 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ad0073a00 0x7f1ad010c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:50:39.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:39.655+0000 7f1ad6800700 1 --2- 172.21.15.62:0/320280996 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1ad00730f0 0x7f1ad00734c0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:50:39.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:39.655+0000 7f1ad6800700 1 -- 172.21.15.62:0/320280996 >> 172.21.15.62:0/320280996 conn(0x7f1ad00fbda0 msgr2=0x7f1ad00fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:50:39.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:39.655+0000 7f1ad6800700 1 -- 172.21.15.62:0/320280996 shutdown_connections 2024-04-20T21:50:39.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:39.655+0000 7f1ad6800700 1 -- 172.21.15.62:0/320280996 wait complete. 2024-04-20T21:50:39.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:39.656+0000 7f1ad6800700 1 Processor -- start 2024-04-20T21:50:39.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:39.656+0000 7f1ad6800700 1 -- start start 2024-04-20T21:50:39.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:39.657+0000 7f1ad6800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ad00730f0 0x7f1ad0197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:50:39.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:39.657+0000 7f1ad6800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1ad0073a00 0x7f1ad0198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:50:39.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:39.657+0000 7f1ad6800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ad019c970 con 0x7f1ad0073a00 2024-04-20T21:50:39.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:39.657+0000 7f1ad6800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ad019cae0 con 0x7f1ad00730f0 2024-04-20T21:50:39.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:39.657+0000 7f1acea00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1ad0073a00 0x7f1ad0198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:50:39.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:39.657+0000 7f1acea00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1ad0073a00 0x7f1ad0198630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34010/0 (socket says 172.21.15.62:34010) 2024-04-20T21:50:39.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:39.657+0000 7f1acea00700 1 -- 172.21.15.62:0/3477791637 learned_addr learned my addr 172.21.15.62:0/3477791637 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:50:39.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:39.657+0000 7f1acf400700 1 --2- 172.21.15.62:0/3477791637 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ad00730f0 0x7f1ad0197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:50:39.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:39.658+0000 7f1acea00700 1 -- 172.21.15.62:0/3477791637 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ad00730f0 msgr2=0x7f1ad0197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:50:39.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:39.658+0000 7f1acea00700 1 --2- 172.21.15.62:0/3477791637 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ad00730f0 0x7f1ad0197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:50:39.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:39.658+0000 7f1acea00700 1 -- 172.21.15.62:0/3477791637 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ac00095f0 con 0x7f1ad0073a00 2024-04-20T21:50:39.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:39.658+0000 7f1acea00700 1 --2- 172.21.15.62:0/3477791637 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1ad0073a00 0x7f1ad0198630 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f1abc00c8a0 tx=0x7f1abc00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:50:39.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:39.659+0000 7f1ac7e00700 1 -- 172.21.15.62:0/3477791637 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1abc00cea0 con 0x7f1ad0073a00 2024-04-20T21:50:39.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:39.659+0000 7f1ac7e00700 1 -- 172.21.15.62:0/3477791637 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1abc004830 con 0x7f1ad0073a00 2024-04-20T21:50:39.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:39.659+0000 7f1ac7e00700 1 -- 172.21.15.62:0/3477791637 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1abc0055b0 con 0x7f1ad0073a00 2024-04-20T21:50:39.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:39.659+0000 7f1ad6800700 1 -- 172.21.15.62:0/3477791637 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ad019cdc0 con 0x7f1ad0073a00 2024-04-20T21:50:39.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:39.659+0000 7f1ad6800700 1 -- 172.21.15.62:0/3477791637 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ad019d310 con 0x7f1ad0073a00 2024-04-20T21:50:39.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:39.661+0000 7f1ad6800700 1 -- 172.21.15.62:0/3477791637 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ad0109c80 con 0x7f1ad0073a00 2024-04-20T21:50:39.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:39.662+0000 7f1ac7e00700 1 -- 172.21.15.62:0/3477791637 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f1abc0049a0 con 0x7f1ad0073a00 2024-04-20T21:50:39.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:39.662+0000 7f1ac7e00700 1 --2- 172.21.15.62:0/3477791637 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f1ab406ecc0 0x7f1ab4071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:50:39.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:39.663+0000 7f1ac7e00700 1 -- 172.21.15.62:0/3477791637 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f1abc08eaf0 con 0x7f1ad0073a00 2024-04-20T21:50:39.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:39.663+0000 7f1acf400700 1 --2- 172.21.15.62:0/3477791637 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f1ab406ecc0 0x7f1ab4071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:50:39.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:39.664+0000 7f1acf400700 1 --2- 172.21.15.62:0/3477791637 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f1ab406ecc0 0x7f1ab4071170 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f1ac0006010 tx=0x7f1ac001a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:50:39.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:39.670+0000 7f1ac7e00700 1 -- 172.21.15.62:0/3477791637 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f1abc057880 con 0x7f1ad0073a00 2024-04-20T21:50:39.784 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:50:39 smithi062 ceph-mon[76322]: pgmap v460: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:50:39.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:39.927+0000 7f1ad6800700 1 -- 172.21.15.62:0/3477791637 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1ad0199390 con 0x7f1ab406ecc0 2024-04-20T21:50:39.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:39.930+0000 7f1ac7e00700 1 -- 172.21.15.62:0/3477791637 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f1ad0199390 con 0x7f1ab406ecc0 2024-04-20T21:50:39.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:39.933+0000 7f1ad6800700 1 -- 172.21.15.62:0/3477791637 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f1ab406ecc0 msgr2=0x7f1ab4071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:50:39.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:39.933+0000 7f1ad6800700 1 --2- 172.21.15.62:0/3477791637 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f1ab406ecc0 0x7f1ab4071170 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f1ac0006010 tx=0x7f1ac001a040 comp rx=0 tx=0).stop 2024-04-20T21:50:39.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:39.933+0000 7f1ad6800700 1 -- 172.21.15.62:0/3477791637 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1ad0073a00 msgr2=0x7f1ad0198630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:50:39.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:39.933+0000 7f1ad6800700 1 --2- 172.21.15.62:0/3477791637 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1ad0073a00 0x7f1ad0198630 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f1abc00c8a0 tx=0x7f1abc00cc60 comp rx=0 tx=0).stop 2024-04-20T21:50:39.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:39.933+0000 7f1ad6800700 1 -- 172.21.15.62:0/3477791637 shutdown_connections 2024-04-20T21:50:39.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:39.933+0000 7f1ad6800700 1 --2- 172.21.15.62:0/3477791637 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f1ab406ecc0 0x7f1ab4071170 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:50:39.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:39.933+0000 7f1ad6800700 1 --2- 172.21.15.62:0/3477791637 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ad00730f0 0x7f1ad0197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:50:39.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:39.933+0000 7f1ad6800700 1 --2- 172.21.15.62:0/3477791637 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1ad0073a00 0x7f1ad0198630 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:50:39.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:39.933+0000 7f1ad6800700 1 -- 172.21.15.62:0/3477791637 >> 172.21.15.62:0/3477791637 conn(0x7f1ad00fbda0 msgr2=0x7f1ad0106dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:50:39.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:39.934+0000 7f1ad6800700 1 -- 172.21.15.62:0/3477791637 shutdown_connections 2024-04-20T21:50:39.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:39.934+0000 7f1ad6800700 1 -- 172.21.15.62:0/3477791637 wait complete. 2024-04-20T21:50:39.949 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T21:50:39.986 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:50:39 smithi189 ceph-mon[57022]: pgmap v460: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:50:40.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.104+0000 7f48afe00700 1 -- 172.21.15.62:0/3012125004 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f48b00730f0 msgr2=0x7f48b00734c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:50:40.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.104+0000 7f48afe00700 1 --2- 172.21.15.62:0/3012125004 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f48b00730f0 0x7f48b00734c0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f48a0009910 tx=0x7f48a0009c20 comp rx=0 tx=0).stop 2024-04-20T21:50:40.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.104+0000 7f48afe00700 1 -- 172.21.15.62:0/3012125004 shutdown_connections 2024-04-20T21:50:40.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.104+0000 7f48afe00700 1 --2- 172.21.15.62:0/3012125004 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f48b0073a00 0x7f48b010c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:50:40.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.104+0000 7f48afe00700 1 --2- 172.21.15.62:0/3012125004 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f48b00730f0 0x7f48b00734c0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:50:40.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.104+0000 7f48afe00700 1 -- 172.21.15.62:0/3012125004 >> 172.21.15.62:0/3012125004 conn(0x7f48b00fbb60 msgr2=0x7f48b00fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:50:40.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.104+0000 7f48afe00700 1 -- 172.21.15.62:0/3012125004 shutdown_connections 2024-04-20T21:50:40.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.104+0000 7f48afe00700 1 -- 172.21.15.62:0/3012125004 wait complete. 2024-04-20T21:50:40.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.105+0000 7f48afe00700 1 Processor -- start 2024-04-20T21:50:40.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.105+0000 7f48afe00700 1 -- start start 2024-04-20T21:50:40.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.106+0000 7f48afe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f48b00730f0 0x7f48b0197d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:50:40.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.106+0000 7f48afe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f48b0073a00 0x7f48b01983b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:50:40.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.106+0000 7f48afe00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48b019c6f0 con 0x7f48b00730f0 2024-04-20T21:50:40.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.106+0000 7f48afe00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48b019c860 con 0x7f48b0073a00 2024-04-20T21:50:40.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.106+0000 7f48aea00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f48b00730f0 0x7f48b0197d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:50:40.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.106+0000 7f48aea00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f48b00730f0 0x7f48b0197d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34030/0 (socket says 172.21.15.62:34030) 2024-04-20T21:50:40.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.106+0000 7f48aea00700 1 -- 172.21.15.62:0/4020776129 learned_addr learned my addr 172.21.15.62:0/4020776129 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:50:40.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.106+0000 7f48ae000700 1 --2- 172.21.15.62:0/4020776129 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f48b0073a00 0x7f48b01983b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:50:40.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.106+0000 7f48aea00700 1 -- 172.21.15.62:0/4020776129 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f48b0073a00 msgr2=0x7f48b01983b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:50:40.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.106+0000 7f48aea00700 1 --2- 172.21.15.62:0/4020776129 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f48b0073a00 0x7f48b01983b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:50:40.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.106+0000 7f48aea00700 1 -- 172.21.15.62:0/4020776129 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48a00095f0 con 0x7f48b00730f0 2024-04-20T21:50:40.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.106+0000 7f48ae000700 1 --2- 172.21.15.62:0/4020776129 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f48b0073a00 0x7f48b01983b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T21:50:40.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.107+0000 7f48aea00700 1 --2- 172.21.15.62:0/4020776129 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f48b00730f0 0x7f48b0197d40 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f48a000b5c0 tx=0x7f48a000be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:50:40.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.107+0000 7f48a7400700 1 -- 172.21.15.62:0/4020776129 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f48a001d070 con 0x7f48b00730f0 2024-04-20T21:50:40.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.107+0000 7f48a7400700 1 -- 172.21.15.62:0/4020776129 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f48a000f460 con 0x7f48b00730f0 2024-04-20T21:50:40.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.107+0000 7f48afe00700 1 -- 172.21.15.62:0/4020776129 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48b019cb40 con 0x7f48b00730f0 2024-04-20T21:50:40.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.107+0000 7f48a7400700 1 -- 172.21.15.62:0/4020776129 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f48a0017630 con 0x7f48b00730f0 2024-04-20T21:50:40.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.107+0000 7f48afe00700 1 -- 172.21.15.62:0/4020776129 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48b019d090 con 0x7f48b00730f0 2024-04-20T21:50:40.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.109+0000 7f48afe00700 1 -- 172.21.15.62:0/4020776129 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48b0109a60 con 0x7f48b00730f0 2024-04-20T21:50:40.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.110+0000 7f48a7400700 1 -- 172.21.15.62:0/4020776129 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f48a000f5d0 con 0x7f48b00730f0 2024-04-20T21:50:40.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.111+0000 7f48a7400700 1 --2- 172.21.15.62:0/4020776129 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f489406ecc0 0x7f4894071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:50:40.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.111+0000 7f48a7400700 1 -- 172.21.15.62:0/4020776129 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f48a0090490 con 0x7f48b00730f0 2024-04-20T21:50:40.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.111+0000 7f48ae000700 1 --2- 172.21.15.62:0/4020776129 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f489406ecc0 0x7f4894071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:50:40.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.112+0000 7f48ae000700 1 --2- 172.21.15.62:0/4020776129 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f489406ecc0 0x7f4894071170 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f489c005fd0 tx=0x7f489c005dc0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:50:40.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.116+0000 7f48a7400700 1 -- 172.21.15.62:0/4020776129 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f48a00591a0 con 0x7f48b00730f0 2024-04-20T21:50:40.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.372+0000 7f48afe00700 1 -- 172.21.15.62:0/4020776129 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f48b01991b0 con 0x7f489406ecc0 2024-04-20T21:50:40.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.374+0000 7f48a7400700 1 -- 172.21.15.62:0/4020776129 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f48b01991b0 con 0x7f489406ecc0 2024-04-20T21:50:40.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.377+0000 7f48afe00700 1 -- 172.21.15.62:0/4020776129 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f489406ecc0 msgr2=0x7f4894071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:50:40.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.377+0000 7f48afe00700 1 --2- 172.21.15.62:0/4020776129 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f489406ecc0 0x7f4894071170 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f489c005fd0 tx=0x7f489c005dc0 comp rx=0 tx=0).stop 2024-04-20T21:50:40.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.377+0000 7f48afe00700 1 -- 172.21.15.62:0/4020776129 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f48b00730f0 msgr2=0x7f48b0197d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:50:40.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.377+0000 7f48afe00700 1 --2- 172.21.15.62:0/4020776129 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f48b00730f0 0x7f48b0197d40 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f48a000b5c0 tx=0x7f48a000be00 comp rx=0 tx=0).stop 2024-04-20T21:50:40.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.377+0000 7f48afe00700 1 -- 172.21.15.62:0/4020776129 shutdown_connections 2024-04-20T21:50:40.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.377+0000 7f48afe00700 1 --2- 172.21.15.62:0/4020776129 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f489406ecc0 0x7f4894071170 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:50:40.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.377+0000 7f48afe00700 1 --2- 172.21.15.62:0/4020776129 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f48b00730f0 0x7f48b0197d40 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:50:40.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.377+0000 7f48afe00700 1 --2- 172.21.15.62:0/4020776129 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f48b0073a00 0x7f48b01983b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:50:40.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.377+0000 7f48afe00700 1 -- 172.21.15.62:0/4020776129 >> 172.21.15.62:0/4020776129 conn(0x7f48b00fbb60 msgr2=0x7f48b0106ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:50:40.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.378+0000 7f48afe00700 1 -- 172.21.15.62:0/4020776129 shutdown_connections 2024-04-20T21:50:40.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.378+0000 7f48afe00700 1 -- 172.21.15.62:0/4020776129 wait complete. 2024-04-20T21:50:40.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.545+0000 7f9c8d000700 1 -- 172.21.15.62:0/592716846 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9c8810c8f0 msgr2=0x7f9c8810ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:50:40.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.545+0000 7f9c8d000700 1 --2- 172.21.15.62:0/592716846 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9c8810c8f0 0x7f9c8810ccc0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f9c7800b0a0 tx=0x7f9c7800b3b0 comp rx=0 tx=0).stop 2024-04-20T21:50:40.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.545+0000 7f9c8d000700 1 -- 172.21.15.62:0/592716846 shutdown_connections 2024-04-20T21:50:40.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.546+0000 7f9c8d000700 1 --2- 172.21.15.62:0/592716846 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9c88106b80 0x7f9c88106ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:50:40.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.546+0000 7f9c8d000700 1 --2- 172.21.15.62:0/592716846 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9c8810c8f0 0x7f9c8810ccc0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:50:40.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.546+0000 7f9c8d000700 1 -- 172.21.15.62:0/592716846 >> 172.21.15.62:0/592716846 conn(0x7f9c88076060 msgr2=0x7f9c88078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:50:40.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.546+0000 7f9c8d000700 1 -- 172.21.15.62:0/592716846 shutdown_connections 2024-04-20T21:50:40.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.546+0000 7f9c8d000700 1 -- 172.21.15.62:0/592716846 wait complete. 2024-04-20T21:50:40.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.547+0000 7f9c8d000700 1 Processor -- start 2024-04-20T21:50:40.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.547+0000 7f9c8d000700 1 -- start start 2024-04-20T21:50:40.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.547+0000 7f9c8d000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9c88106b80 0x7f9c8819c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:50:40.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.547+0000 7f9c8d000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9c8810c8f0 0x7f9c8819c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:50:40.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.547+0000 7f9c8d000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c8819cfd0 con 0x7f9c88106b80 2024-04-20T21:50:40.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.547+0000 7f9c8d000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c8819d110 con 0x7f9c8810c8f0 2024-04-20T21:50:40.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.548+0000 7f9c85e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9c88106b80 0x7f9c8819c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:50:40.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.548+0000 7f9c85e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9c88106b80 0x7f9c8819c440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34052/0 (socket says 172.21.15.62:34052) 2024-04-20T21:50:40.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.548+0000 7f9c85e00700 1 -- 172.21.15.62:0/1270893700 learned_addr learned my addr 172.21.15.62:0/1270893700 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:50:40.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.548+0000 7f9c7d400700 1 --2- 172.21.15.62:0/1270893700 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9c8810c8f0 0x7f9c8819c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:50:40.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.548+0000 7f9c85e00700 1 -- 172.21.15.62:0/1270893700 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9c8810c8f0 msgr2=0x7f9c8819c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:50:40.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.548+0000 7f9c85e00700 1 --2- 172.21.15.62:0/1270893700 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9c8810c8f0 0x7f9c8819c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:50:40.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.548+0000 7f9c85e00700 1 -- 172.21.15.62:0/1270893700 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c78009d00 con 0x7f9c88106b80 2024-04-20T21:50:40.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.548+0000 7f9c85e00700 1 --2- 172.21.15.62:0/1270893700 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9c88106b80 0x7f9c8819c440 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f9c7800b070 tx=0x7f9c7800cdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:50:40.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.549+0000 7f9c7f400700 1 -- 172.21.15.62:0/1270893700 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9c7800e040 con 0x7f9c88106b80 2024-04-20T21:50:40.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.549+0000 7f9c7f400700 1 -- 172.21.15.62:0/1270893700 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9c78003e60 con 0x7f9c88106b80 2024-04-20T21:50:40.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.549+0000 7f9c7f400700 1 -- 172.21.15.62:0/1270893700 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9c7801d910 con 0x7f9c88106b80 2024-04-20T21:50:40.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.549+0000 7f9c8d000700 1 -- 172.21.15.62:0/1270893700 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c881a0f00 con 0x7f9c88106b80 2024-04-20T21:50:40.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.549+0000 7f9c8d000700 1 -- 172.21.15.62:0/1270893700 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c881a13f0 con 0x7f9c88106b80 2024-04-20T21:50:40.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.550+0000 7f9c8d000700 1 -- 172.21.15.62:0/1270893700 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c88109b20 con 0x7f9c88106b80 2024-04-20T21:50:40.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.551+0000 7f9c7f400700 1 -- 172.21.15.62:0/1270893700 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f9c78004820 con 0x7f9c88106b80 2024-04-20T21:50:40.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.552+0000 7f9c7f400700 1 --2- 172.21.15.62:0/1270893700 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9c6c06ecc0 0x7f9c6c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:50:40.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.552+0000 7f9c7f400700 1 -- 172.21.15.62:0/1270893700 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f9c780907a0 con 0x7f9c88106b80 2024-04-20T21:50:40.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.553+0000 7f9c7d400700 1 --2- 172.21.15.62:0/1270893700 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9c6c06ecc0 0x7f9c6c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:50:40.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.553+0000 7f9c7d400700 1 --2- 172.21.15.62:0/1270893700 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9c6c06ecc0 0x7f9c6c071170 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f9c70005ea0 tx=0x7f9c70005e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:50:40.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.557+0000 7f9c7f400700 1 -- 172.21.15.62:0/1270893700 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f9c78059530 con 0x7f9c88106b80 2024-04-20T21:50:40.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.808+0000 7f9c8d000700 1 -- 172.21.15.62:0/1270893700 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f9c881a1720 con 0x7f9c6c06ecc0 2024-04-20T21:50:40.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.825+0000 7f9c7f400700 1 -- 172.21.15.62:0/1270893700 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f9c881a1720 con 0x7f9c6c06ecc0 2024-04-20T21:50:40.825 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T21:50:40.825 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (16m) 7m ago 30m 14.2M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T21:50:40.825 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (30m) 7m ago 30m 9512k - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T21:50:40.825 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (29m) 8m ago 29m 8942k - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T21:50:40.825 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (12m) 7m ago 30m 6660k - 19.0.0-3133-ga8c84f59 76bc6b91b726 2825d5ba11c6 2024-04-20T21:50:40.825 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (12m) 8m ago 29m 6665k - 19.0.0-3133-ga8c84f59 76bc6b91b726 20f746fa1bba 2024-04-20T21:50:40.825 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (15m) 7m ago 30m 74.2M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T21:50:40.825 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (8m) 7m ago 23m 11.2M - 19.0.0-3133-ga8c84f59 76bc6b91b726 de80bc90a683 2024-04-20T21:50:40.825 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (8m) 7m ago 23m 11.9M - 19.0.0-3133-ga8c84f59 76bc6b91b726 cd793b7ee98e 2024-04-20T21:50:40.826 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (23m) 8m ago 23m 117M - 18.2.0 dc2bc1663786 fb2a8a223db8 2024-04-20T21:50:40.826 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (23m) 8m ago 23m 55.6M - 18.2.0 dc2bc1663786 f2eec2ee8951 2024-04-20T21:50:40.826 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (19m) 7m ago 32m 511M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T21:50:40.826 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (18m) 8m ago 29m 452M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T21:50:40.826 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (14m) 7m ago 32m 55.9M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 019eb1479485 2024-04-20T21:50:40.826 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (13m) 8m ago 28m 39.8M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 55aac560aab6 2024-04-20T21:50:40.826 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (17m) 7m ago 30m 7952k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T21:50:40.826 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (17m) 8m ago 29m 8304k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T21:50:40.826 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (11m) 7m ago 28m 88.0M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 ee2bb901f26e 2024-04-20T21:50:40.826 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (11m) 7m ago 27m 92.3M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 9075152d4162 2024-04-20T21:50:40.826 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (10m) 7m ago 26m 74.4M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 2d4f3e38dc0e 2024-04-20T21:50:40.826 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (10m) 8m ago 26m 87.6M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 8f97b17ae0eb 2024-04-20T21:50:40.826 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (9m) 8m ago 25m 90.7M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 653d8079967d 2024-04-20T21:50:40.826 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (8m) 8m ago 25m 11.5M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 596dded14c12 2024-04-20T21:50:40.827 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (17m) 7m ago 29m 74.0M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T21:50:40.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.828+0000 7f9c8d000700 1 -- 172.21.15.62:0/1270893700 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9c6c06ecc0 msgr2=0x7f9c6c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:50:40.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.828+0000 7f9c8d000700 1 --2- 172.21.15.62:0/1270893700 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9c6c06ecc0 0x7f9c6c071170 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f9c70005ea0 tx=0x7f9c70005e30 comp rx=0 tx=0).stop 2024-04-20T21:50:40.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.828+0000 7f9c8d000700 1 -- 172.21.15.62:0/1270893700 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9c88106b80 msgr2=0x7f9c8819c440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:50:40.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.828+0000 7f9c8d000700 1 --2- 172.21.15.62:0/1270893700 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9c88106b80 0x7f9c8819c440 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f9c7800b070 tx=0x7f9c7800cdd0 comp rx=0 tx=0).stop 2024-04-20T21:50:40.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.828+0000 7f9c8d000700 1 -- 172.21.15.62:0/1270893700 shutdown_connections 2024-04-20T21:50:40.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.828+0000 7f9c8d000700 1 --2- 172.21.15.62:0/1270893700 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9c6c06ecc0 0x7f9c6c071170 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:50:40.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.828+0000 7f9c8d000700 1 --2- 172.21.15.62:0/1270893700 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9c88106b80 0x7f9c8819c440 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:50:40.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.828+0000 7f9c8d000700 1 --2- 172.21.15.62:0/1270893700 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9c8810c8f0 0x7f9c8819c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:50:40.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.828+0000 7f9c8d000700 1 -- 172.21.15.62:0/1270893700 >> 172.21.15.62:0/1270893700 conn(0x7f9c88076060 msgr2=0x7f9c880779c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:50:40.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.829+0000 7f9c8d000700 1 -- 172.21.15.62:0/1270893700 shutdown_connections 2024-04-20T21:50:40.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.829+0000 7f9c8d000700 1 -- 172.21.15.62:0/1270893700 wait complete. 2024-04-20T21:50:40.986 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:50:40 smithi189 ceph-mon[57022]: from='client.34872 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:50:40.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.993+0000 7f18ddc00700 1 -- 172.21.15.62:0/3480902666 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f18d80730f0 msgr2=0x7f18d80734c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:50:40.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.993+0000 7f18ddc00700 1 --2- 172.21.15.62:0/3480902666 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f18d80730f0 0x7f18d80734c0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f18c8009910 tx=0x7f18c8009c20 comp rx=0 tx=0).stop 2024-04-20T21:50:40.992 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:50:40 smithi062 ceph-mon[76322]: from='client.34872 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:50:40.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.993+0000 7f18ddc00700 1 -- 172.21.15.62:0/3480902666 shutdown_connections 2024-04-20T21:50:40.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.993+0000 7f18ddc00700 1 --2- 172.21.15.62:0/3480902666 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18d8073a00 0x7f18d8110d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:50:40.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.993+0000 7f18ddc00700 1 --2- 172.21.15.62:0/3480902666 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f18d80730f0 0x7f18d80734c0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:50:40.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.993+0000 7f18ddc00700 1 -- 172.21.15.62:0/3480902666 >> 172.21.15.62:0/3480902666 conn(0x7f18d80fbda0 msgr2=0x7f18d80fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:50:40.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.993+0000 7f18ddc00700 1 -- 172.21.15.62:0/3480902666 shutdown_connections 2024-04-20T21:50:40.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.993+0000 7f18ddc00700 1 -- 172.21.15.62:0/3480902666 wait complete. 2024-04-20T21:50:40.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.995+0000 7f18ddc00700 1 Processor -- start 2024-04-20T21:50:40.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.995+0000 7f18ddc00700 1 -- start start 2024-04-20T21:50:40.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.995+0000 7f18ddc00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f18d80730f0 0x7f18d819c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:50:40.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.995+0000 7f18ddc00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18d8073a00 0x7f18d819c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:50:40.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.995+0000 7f18ddc00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18d819d020 con 0x7f18d80730f0 2024-04-20T21:50:40.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.995+0000 7f18ddc00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18d81a0db0 con 0x7f18d8073a00 2024-04-20T21:50:40.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.996+0000 7f18d6a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f18d80730f0 0x7f18d819c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:50:40.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.996+0000 7f18d6a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f18d80730f0 0x7f18d819c400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34060/0 (socket says 172.21.15.62:34060) 2024-04-20T21:50:40.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.996+0000 7f18d6a00700 1 -- 172.21.15.62:0/3113977658 learned_addr learned my addr 172.21.15.62:0/3113977658 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:50:40.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.996+0000 7f18d6000700 1 --2- 172.21.15.62:0/3113977658 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18d8073a00 0x7f18d819c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:50:40.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.996+0000 7f18d6a00700 1 -- 172.21.15.62:0/3113977658 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18d8073a00 msgr2=0x7f18d819c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:50:40.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.996+0000 7f18d6a00700 1 --2- 172.21.15.62:0/3113977658 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18d8073a00 0x7f18d819c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:50:40.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.996+0000 7f18d6a00700 1 -- 172.21.15.62:0/3113977658 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18c80095f0 con 0x7f18d80730f0 2024-04-20T21:50:40.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.997+0000 7f18d6a00700 1 --2- 172.21.15.62:0/3113977658 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f18d80730f0 0x7f18d819c400 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f18c8006010 tx=0x7f18c800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:50:40.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.997+0000 7f18cf400700 1 -- 172.21.15.62:0/3113977658 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f18c801d070 con 0x7f18d80730f0 2024-04-20T21:50:40.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.997+0000 7f18cf400700 1 -- 172.21.15.62:0/3113977658 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f18c800f460 con 0x7f18d80730f0 2024-04-20T21:50:40.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.997+0000 7f18ddc00700 1 -- 172.21.15.62:0/3113977658 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18d81a1090 con 0x7f18d80730f0 2024-04-20T21:50:40.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.997+0000 7f18cf400700 1 -- 172.21.15.62:0/3113977658 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f18c8017630 con 0x7f18d80730f0 2024-04-20T21:50:40.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.997+0000 7f18ddc00700 1 -- 172.21.15.62:0/3113977658 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18d81a15e0 con 0x7f18d80730f0 2024-04-20T21:50:40.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:40.998+0000 7f18ddc00700 1 -- 172.21.15.62:0/3113977658 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18d810e4d0 con 0x7f18d80730f0 2024-04-20T21:50:40.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.000+0000 7f18cf400700 1 -- 172.21.15.62:0/3113977658 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f18c800f5d0 con 0x7f18d80730f0 2024-04-20T21:50:41.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.001+0000 7f18cf400700 1 --2- 172.21.15.62:0/3113977658 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f18bc06ecc0 0x7f18bc071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:50:41.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.001+0000 7f18cf400700 1 -- 172.21.15.62:0/3113977658 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f18c8090340 con 0x7f18d80730f0 2024-04-20T21:50:41.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.001+0000 7f18d6000700 1 --2- 172.21.15.62:0/3113977658 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f18bc06ecc0 0x7f18bc071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:50:41.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.002+0000 7f18d6000700 1 --2- 172.21.15.62:0/3113977658 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f18bc06ecc0 0x7f18bc071170 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f18c4005950 tx=0x7f18c40058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:50:41.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.006+0000 7f18cf400700 1 -- 172.21.15.62:0/3113977658 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f18c8059050 con 0x7f18d80730f0 2024-04-20T21:50:41.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.343+0000 7f18ddc00700 1 -- 172.21.15.62:0/3113977658 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f18d804f2a0 con 0x7f18d80730f0 2024-04-20T21:50:41.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.344+0000 7f18cf400700 1 -- 172.21.15.62:0/3113977658 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7f18c805c710 con 0x7f18d80730f0 2024-04-20T21:50:41.343 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:50:41.343 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T21:50:41.343 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:50:41.343 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:50:41.343 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T21:50:41.343 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:50:41.343 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:50:41.343 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T21:50:41.343 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 6 2024-04-20T21:50:41.343 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:50:41.343 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T21:50:41.344 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 2, 2024-04-20T21:50:41.344 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:50:41.344 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:50:41.344 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T21:50:41.344 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 2, 2024-04-20T21:50:41.344 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 12 2024-04-20T21:50:41.344 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T21:50:41.344 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:50:41.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.347+0000 7f18ddc00700 1 -- 172.21.15.62:0/3113977658 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f18bc06ecc0 msgr2=0x7f18bc071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:50:41.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.347+0000 7f18ddc00700 1 --2- 172.21.15.62:0/3113977658 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f18bc06ecc0 0x7f18bc071170 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f18c4005950 tx=0x7f18c40058e0 comp rx=0 tx=0).stop 2024-04-20T21:50:41.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.347+0000 7f18ddc00700 1 -- 172.21.15.62:0/3113977658 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f18d80730f0 msgr2=0x7f18d819c400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:50:41.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.347+0000 7f18ddc00700 1 --2- 172.21.15.62:0/3113977658 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f18d80730f0 0x7f18d819c400 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f18c8006010 tx=0x7f18c800be20 comp rx=0 tx=0).stop 2024-04-20T21:50:41.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.347+0000 7f18ddc00700 1 -- 172.21.15.62:0/3113977658 shutdown_connections 2024-04-20T21:50:41.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.347+0000 7f18ddc00700 1 --2- 172.21.15.62:0/3113977658 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f18bc06ecc0 0x7f18bc071170 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:50:41.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.347+0000 7f18ddc00700 1 --2- 172.21.15.62:0/3113977658 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f18d80730f0 0x7f18d819c400 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:50:41.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.347+0000 7f18ddc00700 1 --2- 172.21.15.62:0/3113977658 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18d8073a00 0x7f18d819c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:50:41.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.347+0000 7f18ddc00700 1 -- 172.21.15.62:0/3113977658 >> 172.21.15.62:0/3113977658 conn(0x7f18d80fbda0 msgr2=0x7f18d8102860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:50:41.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.348+0000 7f18ddc00700 1 -- 172.21.15.62:0/3113977658 shutdown_connections 2024-04-20T21:50:41.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.348+0000 7f18ddc00700 1 -- 172.21.15.62:0/3113977658 wait complete. 2024-04-20T21:50:41.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.510+0000 7f83e7800700 1 -- 172.21.15.62:0/1665171368 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f83e0100e80 msgr2=0x7f83e0104e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:50:41.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.510+0000 7f83e7800700 1 --2- 172.21.15.62:0/1665171368 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f83e0100e80 0x7f83e0104e50 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f83d0009910 tx=0x7f83d0009c20 comp rx=0 tx=0).stop 2024-04-20T21:50:41.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.511+0000 7f83e7800700 1 -- 172.21.15.62:0/1665171368 shutdown_connections 2024-04-20T21:50:41.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.511+0000 7f83e7800700 1 --2- 172.21.15.62:0/1665171368 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f83e0100e80 0x7f83e0104e50 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:50:41.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.511+0000 7f83e7800700 1 --2- 172.21.15.62:0/1665171368 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f83e01004e0 0x7f83e01008b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:50:41.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.511+0000 7f83e7800700 1 -- 172.21.15.62:0/1665171368 >> 172.21.15.62:0/1665171368 conn(0x7f83e00fbda0 msgr2=0x7f83e00fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:50:41.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.511+0000 7f83e7800700 1 -- 172.21.15.62:0/1665171368 shutdown_connections 2024-04-20T21:50:41.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.511+0000 7f83e7800700 1 -- 172.21.15.62:0/1665171368 wait complete. 2024-04-20T21:50:41.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.512+0000 7f83e7800700 1 Processor -- start 2024-04-20T21:50:41.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.512+0000 7f83e7800700 1 -- start start 2024-04-20T21:50:41.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.513+0000 7f83e7800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f83e01004e0 0x7f83e0198110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:50:41.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.513+0000 7f83e7800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f83e0100e80 0x7f83e0198650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:50:41.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.513+0000 7f83e7800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83e0198d30 con 0x7f83e0100e80 2024-04-20T21:50:41.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.513+0000 7f83e7800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83e019cac0 con 0x7f83e01004e0 2024-04-20T21:50:41.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.513+0000 7f83d7e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f83e0100e80 0x7f83e0198650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:50:41.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.513+0000 7f83d7e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f83e0100e80 0x7f83e0198650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52326/0 (socket says 172.21.15.62:52326) 2024-04-20T21:50:41.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.513+0000 7f83d7e00700 1 -- 172.21.15.62:0/3256387514 learned_addr learned my addr 172.21.15.62:0/3256387514 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:50:41.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.513+0000 7f83e4e00700 1 --2- 172.21.15.62:0/3256387514 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f83e01004e0 0x7f83e0198110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:50:41.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.513+0000 7f83d7e00700 1 -- 172.21.15.62:0/3256387514 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f83e01004e0 msgr2=0x7f83e0198110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:50:41.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.513+0000 7f83d7e00700 1 --2- 172.21.15.62:0/3256387514 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f83e01004e0 0x7f83e0198110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:50:41.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.513+0000 7f83d7e00700 1 -- 172.21.15.62:0/3256387514 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83d00095f0 con 0x7f83e0100e80 2024-04-20T21:50:41.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.514+0000 7f83d7e00700 1 --2- 172.21.15.62:0/3256387514 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f83e0100e80 0x7f83e0198650 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f83d0005b40 tx=0x7f83d000f740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:50:41.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.514+0000 7f83de000700 1 -- 172.21.15.62:0/3256387514 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f83d001d070 con 0x7f83e0100e80 2024-04-20T21:50:41.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.514+0000 7f83de000700 1 -- 172.21.15.62:0/3256387514 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f83d000bed0 con 0x7f83e0100e80 2024-04-20T21:50:41.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.514+0000 7f83de000700 1 -- 172.21.15.62:0/3256387514 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f83d0017400 con 0x7f83e0100e80 2024-04-20T21:50:41.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.514+0000 7f83e7800700 1 -- 172.21.15.62:0/3256387514 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83e019cd40 con 0x7f83e0100e80 2024-04-20T21:50:41.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.514+0000 7f83e7800700 1 -- 172.21.15.62:0/3256387514 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83e019d230 con 0x7f83e0100e80 2024-04-20T21:50:41.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.516+0000 7f83e7800700 1 -- 172.21.15.62:0/3256387514 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83e01045d0 con 0x7f83e0100e80 2024-04-20T21:50:41.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.516+0000 7f83de000700 1 -- 172.21.15.62:0/3256387514 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f83d00050d0 con 0x7f83e0100e80 2024-04-20T21:50:41.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.517+0000 7f83de000700 1 --2- 172.21.15.62:0/3256387514 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f83c006ecc0 0x7f83c0071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:50:41.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.518+0000 7f83de000700 1 -- 172.21.15.62:0/3256387514 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f83d00901c0 con 0x7f83e0100e80 2024-04-20T21:50:41.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.518+0000 7f83e4e00700 1 --2- 172.21.15.62:0/3256387514 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f83c006ecc0 0x7f83c0071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:50:41.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.518+0000 7f83e4e00700 1 --2- 172.21.15.62:0/3256387514 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f83c006ecc0 0x7f83c0071170 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f83c8009bf0 tx=0x7f83c8009260 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:50:41.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.522+0000 7f83de000700 1 -- 172.21.15.62:0/3256387514 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f83d0058f50 con 0x7f83e0100e80 2024-04-20T21:50:41.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.806+0000 7f83e7800700 1 -- 172.21.15.62:0/3256387514 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f83e0103fb0 con 0x7f83e0100e80 2024-04-20T21:50:41.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.807+0000 7f83de000700 1 -- 172.21.15.62:0/3256387514 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 25 v25) v1 ==== 76+0+1887 (secure 0 0 0) 0x7f83d005c610 con 0x7f83e0100e80 2024-04-20T21:50:41.807 INFO:teuthology.orchestra.run.smithi062.stdout:e25 2024-04-20T21:50:41.807 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T21:50:41.807 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T21:50:41.807 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T21:50:41.807 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:50:41.807 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T21:50:41.807 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T21:50:41.807 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 20 2024-04-20T21:50:41.807 INFO:teuthology.orchestra.run.smithi062.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-20T21:50:41.807 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T21:50:41.807 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T21:42:01.559061+0000 2024-04-20T21:50:41.807 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T21:50:41.807 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T21:50:41.807 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T21:50:41.808 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T21:50:41.808 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T21:50:41.808 INFO:teuthology.orchestra.run.smithi062.stdout:max_xattr_size 65536 2024-04-20T21:50:41.808 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T21:50:41.808 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T21:50:41.808 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 81 2024-04-20T21:50:41.808 INFO:teuthology.orchestra.run.smithi062.stdout:snap_rstat 1 2024-04-20T21:50:41.808 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T21:50:41.808 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 2 2024-04-20T21:50:41.808 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1 2024-04-20T21:50:41.808 INFO:teuthology.orchestra.run.smithi062.stdout:up {} 2024-04-20T21:50:41.808 INFO:teuthology.orchestra.run.smithi062.stdout:failed 0,1 2024-04-20T21:50:41.808 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T21:50:41.808 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T21:50:41.808 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T21:50:41.808 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T21:50:41.809 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data enabled 2024-04-20T21:50:41.809 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T21:50:41.809 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T21:50:41.809 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 1 2024-04-20T21:50:41.809 INFO:teuthology.orchestra.run.smithi062.stdout:qdb_cluster leader: 0 members: 2024-04-20T21:50:41.809 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:50:41.809 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:50:41.809 INFO:teuthology.orchestra.run.smithi062.stdout:Standby daemons: 2024-04-20T21:50:41.809 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:50:41.809 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ryqblb{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6826/4113916205,v1:172.21.15.62:6827/4113916205] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T21:50:41.809 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ozizwi{-1:34476} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6828/3801975165,v1:172.21.15.62:6829/3801975165] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T21:50:41.809 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.xbgspw{-1:44285} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6826/3996040526,v1:172.21.15.189:6827/3996040526] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:50:41.809 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.dvimer{-1:44289} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6824/1743111843,v1:172.21.15.189:6825/1743111843] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:50:41.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.810+0000 7f83e7800700 1 -- 172.21.15.62:0/3256387514 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f83c006ecc0 msgr2=0x7f83c0071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:50:41.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.810+0000 7f83e7800700 1 --2- 172.21.15.62:0/3256387514 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f83c006ecc0 0x7f83c0071170 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f83c8009bf0 tx=0x7f83c8009260 comp rx=0 tx=0).stop 2024-04-20T21:50:41.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.810+0000 7f83e7800700 1 -- 172.21.15.62:0/3256387514 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f83e0100e80 msgr2=0x7f83e0198650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:50:41.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.810+0000 7f83e7800700 1 --2- 172.21.15.62:0/3256387514 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f83e0100e80 0x7f83e0198650 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f83d0005b40 tx=0x7f83d000f740 comp rx=0 tx=0).stop 2024-04-20T21:50:41.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.810+0000 7f83e7800700 1 -- 172.21.15.62:0/3256387514 shutdown_connections 2024-04-20T21:50:41.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.810+0000 7f83e7800700 1 --2- 172.21.15.62:0/3256387514 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f83c006ecc0 0x7f83c0071170 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:50:41.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.810+0000 7f83e7800700 1 --2- 172.21.15.62:0/3256387514 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f83e01004e0 0x7f83e0198110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:50:41.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.810+0000 7f83e7800700 1 --2- 172.21.15.62:0/3256387514 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f83e0100e80 0x7f83e0198650 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:50:41.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.810+0000 7f83e7800700 1 -- 172.21.15.62:0/3256387514 >> 172.21.15.62:0/3256387514 conn(0x7f83e00fbda0 msgr2=0x7f83e00fe150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:50:41.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.811+0000 7f83e7800700 1 -- 172.21.15.62:0/3256387514 shutdown_connections 2024-04-20T21:50:41.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.811+0000 7f83e7800700 1 -- 172.21.15.62:0/3256387514 wait complete. 2024-04-20T21:50:41.811 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 25 2024-04-20T21:50:41.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.961+0000 7fc82f400700 1 -- 172.21.15.62:0/1482029787 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc828100e80 msgr2=0x7fc828104e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:50:41.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.961+0000 7fc82f400700 1 --2- 172.21.15.62:0/1482029787 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc828100e80 0x7fc828104e50 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fc814009910 tx=0x7fc814009c20 comp rx=0 tx=0).stop 2024-04-20T21:50:41.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.961+0000 7fc82f400700 1 -- 172.21.15.62:0/1482029787 shutdown_connections 2024-04-20T21:50:41.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.961+0000 7fc82f400700 1 --2- 172.21.15.62:0/1482029787 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc828100e80 0x7fc828104e50 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:50:41.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.961+0000 7fc82f400700 1 --2- 172.21.15.62:0/1482029787 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc8281004e0 0x7fc8281008b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:50:41.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.961+0000 7fc82f400700 1 -- 172.21.15.62:0/1482029787 >> 172.21.15.62:0/1482029787 conn(0x7fc8280fbda0 msgr2=0x7fc8280fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:50:41.961 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:50:41 smithi062 ceph-mon[76322]: from='client.34876 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:50:41.962 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:50:41 smithi062 ceph-mon[76322]: pgmap v461: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:50:41.962 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:50:41 smithi062 ceph-mon[76322]: from='client.34880 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:50:41.962 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:50:41 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/3113977658' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:50:41.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.962+0000 7fc82f400700 1 -- 172.21.15.62:0/1482029787 shutdown_connections 2024-04-20T21:50:41.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.962+0000 7fc82f400700 1 -- 172.21.15.62:0/1482029787 wait complete. 2024-04-20T21:50:41.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.963+0000 7fc82f400700 1 Processor -- start 2024-04-20T21:50:41.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.963+0000 7fc82f400700 1 -- start start 2024-04-20T21:50:41.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.964+0000 7fc82f400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8281004e0 0x7fc828073050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:50:41.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.964+0000 7fc82f400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc828100e80 0x7fc828073590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:50:41.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.964+0000 7fc82f400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc828073ad0 con 0x7fc8281004e0 2024-04-20T21:50:41.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.964+0000 7fc82f400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc828073c10 con 0x7fc828100e80 2024-04-20T21:50:41.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.964+0000 7fc82ca00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8281004e0 0x7fc828073050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:50:41.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.964+0000 7fc82ca00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8281004e0 0x7fc828073050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52336/0 (socket says 172.21.15.62:52336) 2024-04-20T21:50:41.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.964+0000 7fc82ca00700 1 -- 172.21.15.62:0/376109292 learned_addr learned my addr 172.21.15.62:0/376109292 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:50:41.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.964+0000 7fc827e00700 1 --2- 172.21.15.62:0/376109292 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc828100e80 0x7fc828073590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:50:41.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.964+0000 7fc82ca00700 1 -- 172.21.15.62:0/376109292 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc828100e80 msgr2=0x7fc828073590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:50:41.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.964+0000 7fc82ca00700 1 --2- 172.21.15.62:0/376109292 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc828100e80 0x7fc828073590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:50:41.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.964+0000 7fc82ca00700 1 -- 172.21.15.62:0/376109292 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8140095f0 con 0x7fc8281004e0 2024-04-20T21:50:41.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.965+0000 7fc82ca00700 1 --2- 172.21.15.62:0/376109292 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8281004e0 0x7fc828073050 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fc81c00d900 tx=0x7fc81c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:50:41.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.965+0000 7fc825600700 1 -- 172.21.15.62:0/376109292 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc81c0041f0 con 0x7fc8281004e0 2024-04-20T21:50:41.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.965+0000 7fc825600700 1 -- 172.21.15.62:0/376109292 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc81c004cf0 con 0x7fc8281004e0 2024-04-20T21:50:41.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.965+0000 7fc82f400700 1 -- 172.21.15.62:0/376109292 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8281a2100 con 0x7fc8281004e0 2024-04-20T21:50:41.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.965+0000 7fc825600700 1 -- 172.21.15.62:0/376109292 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc81c00b750 con 0x7fc8281004e0 2024-04-20T21:50:41.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.965+0000 7fc82f400700 1 -- 172.21.15.62:0/376109292 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8281a2510 con 0x7fc8281004e0 2024-04-20T21:50:41.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.967+0000 7fc82f400700 1 -- 172.21.15.62:0/376109292 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc828108850 con 0x7fc8281004e0 2024-04-20T21:50:41.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.968+0000 7fc825600700 1 -- 172.21.15.62:0/376109292 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fc81c004350 con 0x7fc8281004e0 2024-04-20T21:50:41.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.969+0000 7fc825600700 1 --2- 172.21.15.62:0/376109292 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc81006ecc0 0x7fc810071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:50:41.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.969+0000 7fc825600700 1 -- 172.21.15.62:0/376109292 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fc81c08efb0 con 0x7fc8281004e0 2024-04-20T21:50:41.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.969+0000 7fc827e00700 1 --2- 172.21.15.62:0/376109292 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc81006ecc0 0x7fc810071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:50:41.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.970+0000 7fc827e00700 1 --2- 172.21.15.62:0/376109292 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc81006ecc0 0x7fc810071170 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fc828074840 tx=0x7fc81401a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:50:41.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:41.974+0000 7fc825600700 1 -- 172.21.15.62:0/376109292 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fc81c0c3ff0 con 0x7fc8281004e0 2024-04-20T21:50:41.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:50:41 smithi189 ceph-mon[57022]: from='client.34876 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:50:41.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:50:41 smithi189 ceph-mon[57022]: pgmap v461: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:50:41.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:50:41 smithi189 ceph-mon[57022]: from='client.34880 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:50:41.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:50:41 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/3113977658' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:50:42.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:42.219+0000 7fc82f400700 1 -- 172.21.15.62:0/376109292 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc828061190 con 0x7fc81006ecc0 2024-04-20T21:50:42.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:42.221+0000 7fc825600700 1 -- 172.21.15.62:0/376109292 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fc828061190 con 0x7fc81006ecc0 2024-04-20T21:50:42.220 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:50:42.220 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T21:50:42.220 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T21:50:42.220 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-20T21:50:42.220 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T21:50:42.221 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr", 2024-04-20T21:50:42.221 INFO:teuthology.orchestra.run.smithi062.stdout: "crash", 2024-04-20T21:50:42.221 INFO:teuthology.orchestra.run.smithi062.stdout: "mon", 2024-04-20T21:50:42.221 INFO:teuthology.orchestra.run.smithi062.stdout: "osd" 2024-04-20T21:50:42.221 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T21:50:42.221 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "14/23 daemons upgraded", 2024-04-20T21:50:42.221 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading mds daemons", 2024-04-20T21:50:42.221 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T21:50:42.221 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:50:42.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:42.223+0000 7fc82f400700 1 -- 172.21.15.62:0/376109292 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc81006ecc0 msgr2=0x7fc810071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:50:42.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:42.223+0000 7fc82f400700 1 --2- 172.21.15.62:0/376109292 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc81006ecc0 0x7fc810071170 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fc828074840 tx=0x7fc81401a040 comp rx=0 tx=0).stop 2024-04-20T21:50:42.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:42.223+0000 7fc82f400700 1 -- 172.21.15.62:0/376109292 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8281004e0 msgr2=0x7fc828073050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:50:42.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:42.223+0000 7fc82f400700 1 --2- 172.21.15.62:0/376109292 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8281004e0 0x7fc828073050 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fc81c00d900 tx=0x7fc81c00dcc0 comp rx=0 tx=0).stop 2024-04-20T21:50:42.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:42.224+0000 7fc82f400700 1 -- 172.21.15.62:0/376109292 shutdown_connections 2024-04-20T21:50:42.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:42.224+0000 7fc82f400700 1 --2- 172.21.15.62:0/376109292 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc81006ecc0 0x7fc810071170 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:50:42.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:42.224+0000 7fc82f400700 1 --2- 172.21.15.62:0/376109292 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8281004e0 0x7fc828073050 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:50:42.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:42.224+0000 7fc82f400700 1 --2- 172.21.15.62:0/376109292 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc828100e80 0x7fc828073590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:50:42.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:42.224+0000 7fc82f400700 1 -- 172.21.15.62:0/376109292 >> 172.21.15.62:0/376109292 conn(0x7fc8280fbda0 msgr2=0x7fc8280fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:50:42.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:42.224+0000 7fc82f400700 1 -- 172.21.15.62:0/376109292 shutdown_connections 2024-04-20T21:50:42.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:42.224+0000 7fc82f400700 1 -- 172.21.15.62:0/376109292 wait complete. 2024-04-20T21:50:42.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:42.387+0000 7f7815e00700 1 -- 172.21.15.62:0/1078273324 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f78101004e0 msgr2=0x7f78101008b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:50:42.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:42.387+0000 7f7815e00700 1 --2- 172.21.15.62:0/1078273324 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f78101004e0 0x7f78101008b0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f7800009910 tx=0x7f7800009c20 comp rx=0 tx=0).stop 2024-04-20T21:50:42.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:42.388+0000 7f7815e00700 1 -- 172.21.15.62:0/1078273324 shutdown_connections 2024-04-20T21:50:42.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:42.388+0000 7f7815e00700 1 --2- 172.21.15.62:0/1078273324 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7810100e80 0x7f7810104e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:50:42.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:42.388+0000 7f7815e00700 1 --2- 172.21.15.62:0/1078273324 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f78101004e0 0x7f78101008b0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:50:42.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:42.388+0000 7f7815e00700 1 -- 172.21.15.62:0/1078273324 >> 172.21.15.62:0/1078273324 conn(0x7f78100fbda0 msgr2=0x7f78100fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:50:42.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:42.388+0000 7f7815e00700 1 -- 172.21.15.62:0/1078273324 shutdown_connections 2024-04-20T21:50:42.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:42.388+0000 7f7815e00700 1 -- 172.21.15.62:0/1078273324 wait complete. 2024-04-20T21:50:42.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:42.389+0000 7f7815e00700 1 Processor -- start 2024-04-20T21:50:42.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:42.389+0000 7f7815e00700 1 -- start start 2024-04-20T21:50:42.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:42.389+0000 7f7815e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f78101004e0 0x7f78101980f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:50:42.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:42.389+0000 7f7815e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7810100e80 0x7f7810198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:50:42.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:42.389+0000 7f7815e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7810198d10 con 0x7f7810100e80 2024-04-20T21:50:42.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:42.389+0000 7f7815e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f781019caa0 con 0x7f78101004e0 2024-04-20T21:50:42.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:42.390+0000 7f7806000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7810100e80 0x7f7810198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:50:42.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:42.390+0000 7f7806000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7810100e80 0x7f7810198630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52358/0 (socket says 172.21.15.62:52358) 2024-04-20T21:50:42.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:42.390+0000 7f7806000700 1 -- 172.21.15.62:0/3521280766 learned_addr learned my addr 172.21.15.62:0/3521280766 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:50:42.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:42.390+0000 7f780ea00700 1 --2- 172.21.15.62:0/3521280766 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f78101004e0 0x7f78101980f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:50:42.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:42.390+0000 7f7806000700 1 -- 172.21.15.62:0/3521280766 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f78101004e0 msgr2=0x7f78101980f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:50:42.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:42.390+0000 7f7806000700 1 --2- 172.21.15.62:0/3521280766 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f78101004e0 0x7f78101980f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:50:42.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:42.390+0000 7f7806000700 1 -- 172.21.15.62:0/3521280766 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78000095f0 con 0x7f7810100e80 2024-04-20T21:50:42.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:42.391+0000 7f7806000700 1 --2- 172.21.15.62:0/3521280766 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7810100e80 0x7f7810198630 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f77f800c8f0 tx=0x7f77f800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:50:42.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:42.391+0000 7f7814a00700 1 -- 172.21.15.62:0/3521280766 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f77f8007a50 con 0x7f7810100e80 2024-04-20T21:50:42.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:42.391+0000 7f7814a00700 1 -- 172.21.15.62:0/3521280766 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f77f8007bb0 con 0x7f7810100e80 2024-04-20T21:50:42.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:42.391+0000 7f7815e00700 1 -- 172.21.15.62:0/3521280766 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f781019cd80 con 0x7f7810100e80 2024-04-20T21:50:42.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:42.391+0000 7f7814a00700 1 -- 172.21.15.62:0/3521280766 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f77f8018630 con 0x7f7810100e80 2024-04-20T21:50:42.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:42.391+0000 7f7815e00700 1 -- 172.21.15.62:0/3521280766 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f781019d2d0 con 0x7f7810100e80 2024-04-20T21:50:42.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:42.392+0000 7f7815e00700 1 -- 172.21.15.62:0/3521280766 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78101045d0 con 0x7f7810100e80 2024-04-20T21:50:42.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:42.393+0000 7f7814a00700 1 -- 172.21.15.62:0/3521280766 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f77f8007d20 con 0x7f7810100e80 2024-04-20T21:50:42.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:42.394+0000 7f7814a00700 1 --2- 172.21.15.62:0/3521280766 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f77f406ecc0 0x7f77f4071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:50:42.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:42.394+0000 7f7814a00700 1 -- 172.21.15.62:0/3521280766 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f77f808eef0 con 0x7f7810100e80 2024-04-20T21:50:42.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:42.395+0000 7f780ea00700 1 --2- 172.21.15.62:0/3521280766 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f77f406ecc0 0x7f77f4071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:50:42.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:42.395+0000 7f780ea00700 1 --2- 172.21.15.62:0/3521280766 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f77f406ecc0 0x7f77f4071170 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f780000b5c0 tx=0x7f780001a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:50:42.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:42.400+0000 7f7814a00700 1 -- 172.21.15.62:0/3521280766 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f77f8057c00 con 0x7f7810100e80 2024-04-20T21:50:42.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:42.724+0000 7f7815e00700 1 -- 172.21.15.62:0/3521280766 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f781019d550 con 0x7f7810100e80 2024-04-20T21:50:42.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:42.725+0000 7f7814a00700 1 -- 172.21.15.62:0/3521280766 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7f77f805b2c0 con 0x7f7810100e80 2024-04-20T21:50:42.724 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-20T21:50:42.724 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-20T21:50:42.724 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is degraded 2024-04-20T21:50:42.724 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T21:50:42.725 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T21:50:42.725 INFO:teuthology.orchestra.run.smithi062.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-20T21:50:42.725 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-20T21:50:42.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:42.728+0000 7f7815e00700 1 -- 172.21.15.62:0/3521280766 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f77f406ecc0 msgr2=0x7f77f4071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:50:42.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:42.728+0000 7f7815e00700 1 --2- 172.21.15.62:0/3521280766 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f77f406ecc0 0x7f77f4071170 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f780000b5c0 tx=0x7f780001a040 comp rx=0 tx=0).stop 2024-04-20T21:50:42.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:42.728+0000 7f7815e00700 1 -- 172.21.15.62:0/3521280766 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7810100e80 msgr2=0x7f7810198630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:50:42.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:42.728+0000 7f7815e00700 1 --2- 172.21.15.62:0/3521280766 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7810100e80 0x7f7810198630 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f77f800c8f0 tx=0x7f77f800ccb0 comp rx=0 tx=0).stop 2024-04-20T21:50:42.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:42.728+0000 7f7815e00700 1 -- 172.21.15.62:0/3521280766 shutdown_connections 2024-04-20T21:50:42.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:42.728+0000 7f7815e00700 1 --2- 172.21.15.62:0/3521280766 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f77f406ecc0 0x7f77f4071170 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:50:42.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:42.728+0000 7f7815e00700 1 --2- 172.21.15.62:0/3521280766 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f78101004e0 0x7f78101980f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:50:42.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:42.728+0000 7f7815e00700 1 --2- 172.21.15.62:0/3521280766 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7810100e80 0x7f7810198630 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:50:42.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:42.728+0000 7f7815e00700 1 -- 172.21.15.62:0/3521280766 >> 172.21.15.62:0/3521280766 conn(0x7f78100fbda0 msgr2=0x7f78100fe160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:50:42.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:42.728+0000 7f7815e00700 1 -- 172.21.15.62:0/3521280766 shutdown_connections 2024-04-20T21:50:42.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:50:42.729+0000 7f7815e00700 1 -- 172.21.15.62:0/3521280766 wait complete. 2024-04-20T21:50:42.986 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:50:42 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/3256387514' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T21:50:43.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:50:42 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/3256387514' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T21:50:43.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:50:43 smithi189 ceph-mon[57022]: from='client.34892 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:50:43.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:50:43 smithi189 ceph-mon[57022]: pgmap v462: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:50:43.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:50:43 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/3521280766' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T21:50:44.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:50:43 smithi062 ceph-mon[76322]: from='client.34892 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:50:44.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:50:43 smithi062 ceph-mon[76322]: pgmap v462: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:50:44.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:50:43 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/3521280766' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T21:50:45.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:50:45 smithi189 ceph-mon[57022]: pgmap v463: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:50:46.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:50:45 smithi062 ceph-mon[76322]: pgmap v463: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:50:47.986 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:50:47 smithi189 ceph-mon[57022]: pgmap v464: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:50:48.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:50:47 smithi062 ceph-mon[76322]: pgmap v464: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:50:49.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:50:49 smithi189 ceph-mon[57022]: pgmap v465: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:50:49.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:50:49 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:50:50.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:50:49 smithi062 ceph-mon[76322]: pgmap v465: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:50:50.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:50:49 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:50:51.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:50:51 smithi189 ceph-mon[57022]: pgmap v466: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:50:52.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:50:51 smithi062 ceph-mon[76322]: pgmap v466: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:50:53.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:50:53 smithi189 ceph-mon[57022]: pgmap v467: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:50:54.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:50:53 smithi062 ceph-mon[76322]: pgmap v467: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:50:55.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:50:55 smithi189 ceph-mon[57022]: pgmap v468: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:50:56.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:50:55 smithi062 ceph-mon[76322]: pgmap v468: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:50:57.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:50:57 smithi189 ceph-mon[57022]: pgmap v469: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:50:58.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:50:57 smithi062 ceph-mon[76322]: pgmap v469: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:50:59.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:50:59 smithi189 ceph-mon[57022]: pgmap v470: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:51:00.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:50:59 smithi062 ceph-mon[76322]: pgmap v470: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:51:02.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:51:01 smithi062 ceph-mon[76322]: pgmap v471: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:51:02.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:51:01 smithi189 ceph-mon[57022]: pgmap v471: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:51:04.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:51:03 smithi062 ceph-mon[76322]: pgmap v472: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:51:04.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:51:03 smithi189 ceph-mon[57022]: pgmap v472: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:51:05.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:51:04 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:51:05.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:51:04 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:51:06.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:51:05 smithi062 ceph-mon[76322]: pgmap v473: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:51:06.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:51:05 smithi189 ceph-mon[57022]: pgmap v473: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:51:08.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:51:07 smithi062 ceph-mon[76322]: pgmap v474: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:51:08.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:51:07 smithi189 ceph-mon[57022]: pgmap v474: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:51:10.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:51:09 smithi062 ceph-mon[76322]: pgmap v475: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:51:10.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:51:09 smithi189 ceph-mon[57022]: pgmap v475: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:51:12.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:51:11 smithi062 ceph-mon[76322]: pgmap v476: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:51:12.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:51:11 smithi189 ceph-mon[57022]: pgmap v476: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:51:12.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:12.912+0000 7fee32000700 1 -- 172.21.15.62:0/1457698579 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee2c10c8f0 msgr2=0x7fee2c10ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:51:12.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:12.912+0000 7fee32000700 1 --2- 172.21.15.62:0/1457698579 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee2c10c8f0 0x7fee2c10ccc0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fee1c009910 tx=0x7fee1c009c20 comp rx=0 tx=0).stop 2024-04-20T21:51:12.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:12.912+0000 7fee32000700 1 -- 172.21.15.62:0/1457698579 shutdown_connections 2024-04-20T21:51:12.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:12.912+0000 7fee32000700 1 --2- 172.21.15.62:0/1457698579 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee2c106b80 0x7fee2c106ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:51:12.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:12.912+0000 7fee32000700 1 --2- 172.21.15.62:0/1457698579 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee2c10c8f0 0x7fee2c10ccc0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:51:12.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:12.912+0000 7fee32000700 1 -- 172.21.15.62:0/1457698579 >> 172.21.15.62:0/1457698579 conn(0x7fee2c076060 msgr2=0x7fee2c078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:51:12.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:12.912+0000 7fee32000700 1 -- 172.21.15.62:0/1457698579 shutdown_connections 2024-04-20T21:51:12.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:12.912+0000 7fee32000700 1 -- 172.21.15.62:0/1457698579 wait complete. 2024-04-20T21:51:12.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:12.913+0000 7fee32000700 1 Processor -- start 2024-04-20T21:51:12.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:12.914+0000 7fee32000700 1 -- start start 2024-04-20T21:51:12.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:12.914+0000 7fee32000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee2c106b80 0x7fee2c19c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:51:12.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:12.914+0000 7fee32000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee2c10c8f0 0x7fee2c19ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:51:12.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:12.914+0000 7fee32000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee2c19d120 con 0x7fee2c10c8f0 2024-04-20T21:51:12.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:12.914+0000 7fee32000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee2c1a0eb0 con 0x7fee2c106b80 2024-04-20T21:51:12.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:12.915+0000 7fee2aa00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee2c10c8f0 0x7fee2c19ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:51:12.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:12.915+0000 7fee2aa00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee2c10c8f0 0x7fee2c19ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37822/0 (socket says 172.21.15.62:37822) 2024-04-20T21:51:12.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:12.915+0000 7fee2aa00700 1 -- 172.21.15.62:0/2261375665 learned_addr learned my addr 172.21.15.62:0/2261375665 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:51:12.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:12.915+0000 7fee2b400700 1 --2- 172.21.15.62:0/2261375665 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee2c106b80 0x7fee2c19c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:51:12.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:12.915+0000 7fee2aa00700 1 -- 172.21.15.62:0/2261375665 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee2c106b80 msgr2=0x7fee2c19c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:51:12.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:12.915+0000 7fee2aa00700 1 --2- 172.21.15.62:0/2261375665 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee2c106b80 0x7fee2c19c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:51:12.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:12.915+0000 7fee2aa00700 1 -- 172.21.15.62:0/2261375665 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee1c0095f0 con 0x7fee2c10c8f0 2024-04-20T21:51:12.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:12.916+0000 7fee2aa00700 1 --2- 172.21.15.62:0/2261375665 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee2c10c8f0 0x7fee2c19ca40 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fee1400d8d0 tx=0x7fee1400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:51:12.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:12.916+0000 7fee23e00700 1 -- 172.21.15.62:0/2261375665 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fee14009940 con 0x7fee2c10c8f0 2024-04-20T21:51:12.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:12.916+0000 7fee23e00700 1 -- 172.21.15.62:0/2261375665 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fee14010460 con 0x7fee2c10c8f0 2024-04-20T21:51:12.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:12.916+0000 7fee32000700 1 -- 172.21.15.62:0/2261375665 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee2c1a1190 con 0x7fee2c10c8f0 2024-04-20T21:51:12.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:12.916+0000 7fee23e00700 1 -- 172.21.15.62:0/2261375665 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fee1400f5f0 con 0x7fee2c10c8f0 2024-04-20T21:51:12.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:12.916+0000 7fee32000700 1 -- 172.21.15.62:0/2261375665 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee2c1a16e0 con 0x7fee2c10c8f0 2024-04-20T21:51:12.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:12.917+0000 7fee32000700 1 -- 172.21.15.62:0/2261375665 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee2c109b20 con 0x7fee2c10c8f0 2024-04-20T21:51:12.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:12.919+0000 7fee23e00700 1 -- 172.21.15.62:0/2261375665 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fee140105d0 con 0x7fee2c10c8f0 2024-04-20T21:51:12.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:12.920+0000 7fee23e00700 1 --2- 172.21.15.62:0/2261375665 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fee1006ecc0 0x7fee10071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:51:12.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:12.920+0000 7fee23e00700 1 -- 172.21.15.62:0/2261375665 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fee1408ee80 con 0x7fee2c10c8f0 2024-04-20T21:51:12.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:12.920+0000 7fee2b400700 1 --2- 172.21.15.62:0/2261375665 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fee1006ecc0 0x7fee10071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:51:12.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:12.921+0000 7fee2b400700 1 --2- 172.21.15.62:0/2261375665 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fee1006ecc0 0x7fee10071170 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fee1c006010 tx=0x7fee1c01a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:51:12.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:12.925+0000 7fee23e00700 1 -- 172.21.15.62:0/2261375665 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fee14057c30 con 0x7fee2c10c8f0 2024-04-20T21:51:13.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:13.182+0000 7fee32000700 1 -- 172.21.15.62:0/2261375665 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fee2c1a1970 con 0x7fee1006ecc0 2024-04-20T21:51:13.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:13.185+0000 7fee23e00700 1 -- 172.21.15.62:0/2261375665 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fee2c1a1970 con 0x7fee1006ecc0 2024-04-20T21:51:13.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:13.188+0000 7fee32000700 1 -- 172.21.15.62:0/2261375665 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fee1006ecc0 msgr2=0x7fee10071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:51:13.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:13.188+0000 7fee32000700 1 --2- 172.21.15.62:0/2261375665 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fee1006ecc0 0x7fee10071170 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fee1c006010 tx=0x7fee1c01a040 comp rx=0 tx=0).stop 2024-04-20T21:51:13.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:13.188+0000 7fee32000700 1 -- 172.21.15.62:0/2261375665 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee2c10c8f0 msgr2=0x7fee2c19ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:51:13.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:13.188+0000 7fee32000700 1 --2- 172.21.15.62:0/2261375665 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee2c10c8f0 0x7fee2c19ca40 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fee1400d8d0 tx=0x7fee1400dc90 comp rx=0 tx=0).stop 2024-04-20T21:51:13.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:13.188+0000 7fee32000700 1 -- 172.21.15.62:0/2261375665 shutdown_connections 2024-04-20T21:51:13.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:13.188+0000 7fee32000700 1 --2- 172.21.15.62:0/2261375665 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fee1006ecc0 0x7fee10071170 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:51:13.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:13.188+0000 7fee32000700 1 --2- 172.21.15.62:0/2261375665 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee2c106b80 0x7fee2c19c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:51:13.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:13.188+0000 7fee32000700 1 --2- 172.21.15.62:0/2261375665 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee2c10c8f0 0x7fee2c19ca40 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:51:13.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:13.188+0000 7fee32000700 1 -- 172.21.15.62:0/2261375665 >> 172.21.15.62:0/2261375665 conn(0x7fee2c076060 msgr2=0x7fee2c077aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:51:13.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:13.188+0000 7fee32000700 1 -- 172.21.15.62:0/2261375665 shutdown_connections 2024-04-20T21:51:13.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:13.188+0000 7fee32000700 1 -- 172.21.15.62:0/2261375665 wait complete. 2024-04-20T21:51:13.204 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T21:51:13.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:13.356+0000 7fc691200700 1 -- 172.21.15.62:0/3491922277 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc68c1004e0 msgr2=0x7fc68c1008b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:51:13.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:13.356+0000 7fc691200700 1 --2- 172.21.15.62:0/3491922277 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc68c1004e0 0x7fc68c1008b0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fc67c009910 tx=0x7fc67c009c20 comp rx=0 tx=0).stop 2024-04-20T21:51:13.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:13.356+0000 7fc691200700 1 -- 172.21.15.62:0/3491922277 shutdown_connections 2024-04-20T21:51:13.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:13.356+0000 7fc691200700 1 --2- 172.21.15.62:0/3491922277 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc68c100e80 0x7fc68c104e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:51:13.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:13.356+0000 7fc691200700 1 --2- 172.21.15.62:0/3491922277 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc68c1004e0 0x7fc68c1008b0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:51:13.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:13.356+0000 7fc691200700 1 -- 172.21.15.62:0/3491922277 >> 172.21.15.62:0/3491922277 conn(0x7fc68c0fbda0 msgr2=0x7fc68c0fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:51:13.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:13.356+0000 7fc691200700 1 -- 172.21.15.62:0/3491922277 shutdown_connections 2024-04-20T21:51:13.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:13.356+0000 7fc691200700 1 -- 172.21.15.62:0/3491922277 wait complete. 2024-04-20T21:51:13.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:13.357+0000 7fc691200700 1 Processor -- start 2024-04-20T21:51:13.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:13.358+0000 7fc691200700 1 -- start start 2024-04-20T21:51:13.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:13.358+0000 7fc691200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc68c1004e0 0x7fc68c198110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:51:13.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:13.358+0000 7fc691200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc68c100e80 0x7fc68c198650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:51:13.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:13.358+0000 7fc691200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc68c198d30 con 0x7fc68c1004e0 2024-04-20T21:51:13.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:13.358+0000 7fc691200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc68c19cac0 con 0x7fc68c100e80 2024-04-20T21:51:13.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:13.358+0000 7fc689e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc68c1004e0 0x7fc68c198110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:51:13.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:13.358+0000 7fc689e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc68c1004e0 0x7fc68c198110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37838/0 (socket says 172.21.15.62:37838) 2024-04-20T21:51:13.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:13.358+0000 7fc689e00700 1 -- 172.21.15.62:0/3830666246 learned_addr learned my addr 172.21.15.62:0/3830666246 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:51:13.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:13.359+0000 7fc689e00700 1 -- 172.21.15.62:0/3830666246 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc68c100e80 msgr2=0x7fc68c198650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:51:13.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:13.359+0000 7fc689e00700 1 --2- 172.21.15.62:0/3830666246 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc68c100e80 0x7fc68c198650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:51:13.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:13.359+0000 7fc689e00700 1 -- 172.21.15.62:0/3830666246 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc67c0095f0 con 0x7fc68c1004e0 2024-04-20T21:51:13.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:13.359+0000 7fc689e00700 1 --2- 172.21.15.62:0/3830666246 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc68c1004e0 0x7fc68c198110 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fc67c00b5c0 tx=0x7fc67c00be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:51:13.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:13.359+0000 7fc682a00700 1 -- 172.21.15.62:0/3830666246 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc67c01d070 con 0x7fc68c1004e0 2024-04-20T21:51:13.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:13.359+0000 7fc682a00700 1 -- 172.21.15.62:0/3830666246 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc67c00f460 con 0x7fc68c1004e0 2024-04-20T21:51:13.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:13.359+0000 7fc682a00700 1 -- 172.21.15.62:0/3830666246 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc67c017610 con 0x7fc68c1004e0 2024-04-20T21:51:13.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:13.359+0000 7fc691200700 1 -- 172.21.15.62:0/3830666246 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc68c19cd40 con 0x7fc68c1004e0 2024-04-20T21:51:13.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:13.360+0000 7fc691200700 1 -- 172.21.15.62:0/3830666246 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc68c19d230 con 0x7fc68c1004e0 2024-04-20T21:51:13.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:13.361+0000 7fc691200700 1 -- 172.21.15.62:0/3830666246 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc68c1045d0 con 0x7fc68c1004e0 2024-04-20T21:51:13.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:13.362+0000 7fc682a00700 1 -- 172.21.15.62:0/3830666246 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fc67c00f5d0 con 0x7fc68c1004e0 2024-04-20T21:51:13.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:13.363+0000 7fc682a00700 1 --2- 172.21.15.62:0/3830666246 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc67006ecc0 0x7fc670071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:51:13.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:13.363+0000 7fc682a00700 1 -- 172.21.15.62:0/3830666246 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fc67c090520 con 0x7fc68c1004e0 2024-04-20T21:51:13.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:13.363+0000 7fc689400700 1 --2- 172.21.15.62:0/3830666246 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc67006ecc0 0x7fc670071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:51:13.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:13.364+0000 7fc689400700 1 --2- 172.21.15.62:0/3830666246 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc67006ecc0 0x7fc670071170 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fc68c199730 tx=0x7fc678005c80 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:51:13.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:13.368+0000 7fc682a00700 1 -- 172.21.15.62:0/3830666246 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fc67c059230 con 0x7fc68c1004e0 2024-04-20T21:51:13.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:13.618+0000 7fc691200700 1 -- 172.21.15.62:0/3830666246 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc68c19d510 con 0x7fc67006ecc0 2024-04-20T21:51:13.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:13.620+0000 7fc682a00700 1 -- 172.21.15.62:0/3830666246 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fc68c19d510 con 0x7fc67006ecc0 2024-04-20T21:51:13.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:13.623+0000 7fc691200700 1 -- 172.21.15.62:0/3830666246 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc67006ecc0 msgr2=0x7fc670071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:51:13.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:13.624+0000 7fc691200700 1 --2- 172.21.15.62:0/3830666246 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc67006ecc0 0x7fc670071170 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fc68c199730 tx=0x7fc678005c80 comp rx=0 tx=0).stop 2024-04-20T21:51:13.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:13.624+0000 7fc691200700 1 -- 172.21.15.62:0/3830666246 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc68c1004e0 msgr2=0x7fc68c198110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:51:13.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:13.624+0000 7fc691200700 1 --2- 172.21.15.62:0/3830666246 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc68c1004e0 0x7fc68c198110 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fc67c00b5c0 tx=0x7fc67c00be00 comp rx=0 tx=0).stop 2024-04-20T21:51:13.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:13.624+0000 7fc691200700 1 -- 172.21.15.62:0/3830666246 shutdown_connections 2024-04-20T21:51:13.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:13.624+0000 7fc691200700 1 --2- 172.21.15.62:0/3830666246 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc67006ecc0 0x7fc670071170 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:51:13.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:13.624+0000 7fc691200700 1 --2- 172.21.15.62:0/3830666246 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc68c1004e0 0x7fc68c198110 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:51:13.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:13.624+0000 7fc691200700 1 --2- 172.21.15.62:0/3830666246 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc68c100e80 0x7fc68c198650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:51:13.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:13.624+0000 7fc691200700 1 -- 172.21.15.62:0/3830666246 >> 172.21.15.62:0/3830666246 conn(0x7fc68c0fbda0 msgr2=0x7fc68c0fe160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:51:13.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:13.624+0000 7fc691200700 1 -- 172.21.15.62:0/3830666246 shutdown_connections 2024-04-20T21:51:13.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:13.625+0000 7fc691200700 1 -- 172.21.15.62:0/3830666246 wait complete. 2024-04-20T21:51:13.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:13.788+0000 7f9232c00700 1 -- 172.21.15.62:0/1175933720 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f922c106b80 msgr2=0x7f922c106ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:51:13.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:13.788+0000 7f9232c00700 1 --2- 172.21.15.62:0/1175933720 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f922c106b80 0x7f922c106ff0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f9218009910 tx=0x7f9218009c20 comp rx=0 tx=0).stop 2024-04-20T21:51:13.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:13.789+0000 7f9232c00700 1 -- 172.21.15.62:0/1175933720 shutdown_connections 2024-04-20T21:51:13.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:13.789+0000 7f9232c00700 1 --2- 172.21.15.62:0/1175933720 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f922c106b80 0x7f922c106ff0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:51:13.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:13.789+0000 7f9232c00700 1 --2- 172.21.15.62:0/1175933720 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f922c10c8f0 0x7f922c10ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:51:13.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:13.789+0000 7f9232c00700 1 -- 172.21.15.62:0/1175933720 >> 172.21.15.62:0/1175933720 conn(0x7f922c076060 msgr2=0x7f922c078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:51:13.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:13.789+0000 7f9232c00700 1 -- 172.21.15.62:0/1175933720 shutdown_connections 2024-04-20T21:51:13.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:13.789+0000 7f9232c00700 1 -- 172.21.15.62:0/1175933720 wait complete. 2024-04-20T21:51:13.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:13.790+0000 7f9232c00700 1 Processor -- start 2024-04-20T21:51:13.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:13.790+0000 7f9232c00700 1 -- start start 2024-04-20T21:51:13.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:13.791+0000 7f9232c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f922c106b80 0x7f922c19c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:51:13.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:13.791+0000 7f9232c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f922c10c8f0 0x7f922c19c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:51:13.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:13.791+0000 7f9232c00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f922c19d060 con 0x7f922c10c8f0 2024-04-20T21:51:13.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:13.791+0000 7f9232c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f922c1a0df0 con 0x7f922c106b80 2024-04-20T21:51:13.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:13.791+0000 7f922b400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f922c10c8f0 0x7f922c19c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:51:13.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:13.791+0000 7f922b400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f922c10c8f0 0x7f922c19c980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37856/0 (socket says 172.21.15.62:37856) 2024-04-20T21:51:13.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:13.791+0000 7f922b400700 1 -- 172.21.15.62:0/148059220 learned_addr learned my addr 172.21.15.62:0/148059220 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:51:13.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:13.791+0000 7f922be00700 1 --2- 172.21.15.62:0/148059220 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f922c106b80 0x7f922c19c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:51:13.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:13.792+0000 7f922b400700 1 -- 172.21.15.62:0/148059220 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f922c106b80 msgr2=0x7f922c19c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:51:13.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:13.792+0000 7f922b400700 1 --2- 172.21.15.62:0/148059220 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f922c106b80 0x7f922c19c440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:51:13.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:13.792+0000 7f922b400700 1 -- 172.21.15.62:0/148059220 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92180095f0 con 0x7f922c10c8f0 2024-04-20T21:51:13.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:13.792+0000 7f922b400700 1 --2- 172.21.15.62:0/148059220 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f922c10c8f0 0x7f922c19c980 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f92180092e0 tx=0x7f921800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:51:13.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:13.792+0000 7f9228c00700 1 -- 172.21.15.62:0/148059220 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f921801d070 con 0x7f922c10c8f0 2024-04-20T21:51:13.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:13.792+0000 7f9228c00700 1 -- 172.21.15.62:0/148059220 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9218005e00 con 0x7f922c10c8f0 2024-04-20T21:51:13.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:13.792+0000 7f9232c00700 1 -- 172.21.15.62:0/148059220 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f922c1a1070 con 0x7f922c10c8f0 2024-04-20T21:51:13.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:13.792+0000 7f9228c00700 1 -- 172.21.15.62:0/148059220 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9218017400 con 0x7f922c10c8f0 2024-04-20T21:51:13.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:13.792+0000 7f9232c00700 1 -- 172.21.15.62:0/148059220 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f922c1a1560 con 0x7f922c10c8f0 2024-04-20T21:51:13.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:13.794+0000 7f9232c00700 1 -- 172.21.15.62:0/148059220 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f922c10ecc0 con 0x7f922c10c8f0 2024-04-20T21:51:13.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:13.795+0000 7f9228c00700 1 -- 172.21.15.62:0/148059220 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f921800f460 con 0x7f922c10c8f0 2024-04-20T21:51:13.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:13.796+0000 7f9228c00700 1 --2- 172.21.15.62:0/148059220 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f921006ecc0 0x7f9210071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:51:13.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:13.796+0000 7f9228c00700 1 -- 172.21.15.62:0/148059220 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f9218090450 con 0x7f922c10c8f0 2024-04-20T21:51:13.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:13.796+0000 7f922be00700 1 --2- 172.21.15.62:0/148059220 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f921006ecc0 0x7f9210071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:51:13.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:13.797+0000 7f922be00700 1 --2- 172.21.15.62:0/148059220 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f921006ecc0 0x7f9210071170 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f921c005950 tx=0x7f921c0058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:51:13.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:13.801+0000 7f9228c00700 1 -- 172.21.15.62:0/148059220 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f9218059160 con 0x7f922c10c8f0 2024-04-20T21:51:14.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:51:13 smithi062 ceph-mon[76322]: pgmap v477: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:51:14.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:14.045+0000 7f9232c00700 1 -- 172.21.15.62:0/148059220 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f922c1a1840 con 0x7f921006ecc0 2024-04-20T21:51:14.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:14.059+0000 7f9228c00700 1 -- 172.21.15.62:0/148059220 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f922c1a1840 con 0x7f921006ecc0 2024-04-20T21:51:14.059 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T21:51:14.059 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (17m) 8m ago 30m 14.2M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T21:51:14.059 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (31m) 8m ago 31m 9512k - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T21:51:14.059 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (29m) 9m ago 29m 8942k - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T21:51:14.059 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (13m) 8m ago 31m 6660k - 19.0.0-3133-ga8c84f59 76bc6b91b726 2825d5ba11c6 2024-04-20T21:51:14.059 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (13m) 9m ago 29m 6665k - 19.0.0-3133-ga8c84f59 76bc6b91b726 20f746fa1bba 2024-04-20T21:51:14.059 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (16m) 8m ago 30m 74.2M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T21:51:14.059 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (8m) 8m ago 23m 11.2M - 19.0.0-3133-ga8c84f59 76bc6b91b726 de80bc90a683 2024-04-20T21:51:14.059 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (9m) 8m ago 23m 11.9M - 19.0.0-3133-ga8c84f59 76bc6b91b726 cd793b7ee98e 2024-04-20T21:51:14.060 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (23m) 9m ago 23m 117M - 18.2.0 dc2bc1663786 fb2a8a223db8 2024-04-20T21:51:14.060 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (23m) 9m ago 23m 55.6M - 18.2.0 dc2bc1663786 f2eec2ee8951 2024-04-20T21:51:14.060 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (19m) 8m ago 32m 511M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T21:51:14.060 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (18m) 9m ago 29m 452M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T21:51:14.060 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (14m) 8m ago 32m 55.9M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 019eb1479485 2024-04-20T21:51:14.060 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (14m) 9m ago 29m 39.8M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 55aac560aab6 2024-04-20T21:51:14.060 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (18m) 8m ago 31m 7952k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T21:51:14.060 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (18m) 9m ago 29m 8304k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T21:51:14.061 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (12m) 8m ago 28m 88.0M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 ee2bb901f26e 2024-04-20T21:51:14.061 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (11m) 8m ago 27m 92.3M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 9075152d4162 2024-04-20T21:51:14.061 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (11m) 8m ago 27m 74.4M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 2d4f3e38dc0e 2024-04-20T21:51:14.061 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (10m) 9m ago 26m 87.6M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 8f97b17ae0eb 2024-04-20T21:51:14.061 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (10m) 9m ago 26m 90.7M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 653d8079967d 2024-04-20T21:51:14.061 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (9m) 9m ago 25m 11.5M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 596dded14c12 2024-04-20T21:51:14.062 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (17m) 8m ago 30m 74.0M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T21:51:14.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:14.062+0000 7f9232c00700 1 -- 172.21.15.62:0/148059220 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f921006ecc0 msgr2=0x7f9210071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:51:14.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:14.062+0000 7f9232c00700 1 --2- 172.21.15.62:0/148059220 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f921006ecc0 0x7f9210071170 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f921c005950 tx=0x7f921c0058e0 comp rx=0 tx=0).stop 2024-04-20T21:51:14.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:14.062+0000 7f9232c00700 1 -- 172.21.15.62:0/148059220 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f922c10c8f0 msgr2=0x7f922c19c980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:51:14.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:14.062+0000 7f9232c00700 1 --2- 172.21.15.62:0/148059220 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f922c10c8f0 0x7f922c19c980 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f92180092e0 tx=0x7f921800be20 comp rx=0 tx=0).stop 2024-04-20T21:51:14.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:14.063+0000 7f9232c00700 1 -- 172.21.15.62:0/148059220 shutdown_connections 2024-04-20T21:51:14.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:14.063+0000 7f9232c00700 1 --2- 172.21.15.62:0/148059220 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f921006ecc0 0x7f9210071170 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:51:14.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:14.063+0000 7f9232c00700 1 --2- 172.21.15.62:0/148059220 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f922c106b80 0x7f922c19c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:51:14.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:14.063+0000 7f9232c00700 1 --2- 172.21.15.62:0/148059220 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f922c10c8f0 0x7f922c19c980 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:51:14.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:14.063+0000 7f9232c00700 1 -- 172.21.15.62:0/148059220 >> 172.21.15.62:0/148059220 conn(0x7f922c076060 msgr2=0x7f922c077ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:51:14.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:14.063+0000 7f9232c00700 1 -- 172.21.15.62:0/148059220 shutdown_connections 2024-04-20T21:51:14.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:14.063+0000 7f9232c00700 1 -- 172.21.15.62:0/148059220 wait complete. 2024-04-20T21:51:14.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:14.220+0000 7fbbf3200700 1 -- 172.21.15.62:0/3686613897 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbbf4106b80 msgr2=0x7fbbf4106ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:51:14.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:14.220+0000 7fbbf3200700 1 --2- 172.21.15.62:0/3686613897 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbbf4106b80 0x7fbbf4106ff0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fbbe0009910 tx=0x7fbbe0009c20 comp rx=0 tx=0).stop 2024-04-20T21:51:14.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:14.220+0000 7fbbf3200700 1 -- 172.21.15.62:0/3686613897 shutdown_connections 2024-04-20T21:51:14.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:14.220+0000 7fbbf3200700 1 --2- 172.21.15.62:0/3686613897 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbbf4106b80 0x7fbbf4106ff0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:51:14.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:14.220+0000 7fbbf3200700 1 --2- 172.21.15.62:0/3686613897 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbbf410b640 0x7fbbf410ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:51:14.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:14.220+0000 7fbbf3200700 1 -- 172.21.15.62:0/3686613897 >> 172.21.15.62:0/3686613897 conn(0x7fbbf4075e90 msgr2=0x7fbbf40782a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:51:14.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:14.220+0000 7fbbf3200700 1 -- 172.21.15.62:0/3686613897 shutdown_connections 2024-04-20T21:51:14.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:14.221+0000 7fbbf3200700 1 -- 172.21.15.62:0/3686613897 wait complete. 2024-04-20T21:51:14.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:14.222+0000 7fbbf3200700 1 Processor -- start 2024-04-20T21:51:14.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:14.222+0000 7fbbf3200700 1 -- start start 2024-04-20T21:51:14.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:14.222+0000 7fbbf3200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbbf4106b80 0x7fbbf419c180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:51:14.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:14.222+0000 7fbbf3200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbbf410b640 0x7fbbf419c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:51:14.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:14.222+0000 7fbbf3200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbf419cda0 con 0x7fbbf410b640 2024-04-20T21:51:14.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:14.222+0000 7fbbf3200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbf41a0b30 con 0x7fbbf4106b80 2024-04-20T21:51:14.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:14.222+0000 7fbbf1400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbbf410b640 0x7fbbf419c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:51:14.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:14.222+0000 7fbbf1400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbbf410b640 0x7fbbf419c6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37870/0 (socket says 172.21.15.62:37870) 2024-04-20T21:51:14.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:14.223+0000 7fbbf1400700 1 -- 172.21.15.62:0/776716675 learned_addr learned my addr 172.21.15.62:0/776716675 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:51:14.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:14.223+0000 7fbbf1e00700 1 --2- 172.21.15.62:0/776716675 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbbf4106b80 0x7fbbf419c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:51:14.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:14.223+0000 7fbbf1400700 1 -- 172.21.15.62:0/776716675 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbbf4106b80 msgr2=0x7fbbf419c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:51:14.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:14.223+0000 7fbbf1400700 1 --2- 172.21.15.62:0/776716675 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbbf4106b80 0x7fbbf419c180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:51:14.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:14.223+0000 7fbbf1400700 1 -- 172.21.15.62:0/776716675 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbe00095f0 con 0x7fbbf410b640 2024-04-20T21:51:14.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:14.223+0000 7fbbf1400700 1 --2- 172.21.15.62:0/776716675 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbbf410b640 0x7fbbf419c6c0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fbbe000b5c0 tx=0x7fbbe000be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:51:14.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:14.224+0000 7fbbeaa00700 1 -- 172.21.15.62:0/776716675 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbbe001d070 con 0x7fbbf410b640 2024-04-20T21:51:14.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:14.224+0000 7fbbeaa00700 1 -- 172.21.15.62:0/776716675 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbbe000f460 con 0x7fbbf410b640 2024-04-20T21:51:14.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:14.224+0000 7fbbf3200700 1 -- 172.21.15.62:0/776716675 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbf41a0db0 con 0x7fbbf410b640 2024-04-20T21:51:14.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:14.224+0000 7fbbeaa00700 1 -- 172.21.15.62:0/776716675 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbbe0017610 con 0x7fbbf410b640 2024-04-20T21:51:14.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:14.224+0000 7fbbf3200700 1 -- 172.21.15.62:0/776716675 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbf41a1320 con 0x7fbbf410b640 2024-04-20T21:51:14.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:14.225+0000 7fbbf3200700 1 -- 172.21.15.62:0/776716675 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbf410eb10 con 0x7fbbf410b640 2024-04-20T21:51:14.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:14.226+0000 7fbbeaa00700 1 -- 172.21.15.62:0/776716675 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fbbe000f5d0 con 0x7fbbf410b640 2024-04-20T21:51:14.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:14.227+0000 7fbbeaa00700 1 --2- 172.21.15.62:0/776716675 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fbbd806ecc0 0x7fbbd8071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:51:14.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:14.227+0000 7fbbeaa00700 1 -- 172.21.15.62:0/776716675 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fbbe0090520 con 0x7fbbf410b640 2024-04-20T21:51:14.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:14.227+0000 7fbbf1e00700 1 --2- 172.21.15.62:0/776716675 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fbbd806ecc0 0x7fbbd8071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:51:14.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:14.228+0000 7fbbf1e00700 1 --2- 172.21.15.62:0/776716675 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fbbd806ecc0 0x7fbbd8071170 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fbbe4005fd0 tx=0x7fbbe400a560 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:51:14.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:14.232+0000 7fbbeaa00700 1 -- 172.21.15.62:0/776716675 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fbbe0059230 con 0x7fbbf410b640 2024-04-20T21:51:14.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:51:13 smithi189 ceph-mon[57022]: pgmap v477: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:51:14.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:14.560+0000 7fbbf3200700 1 -- 172.21.15.62:0/776716675 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fbbf4062380 con 0x7fbbf410b640 2024-04-20T21:51:14.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:14.561+0000 7fbbeaa00700 1 -- 172.21.15.62:0/776716675 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7fbbe005c8f0 con 0x7fbbf410b640 2024-04-20T21:51:14.560 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:51:14.560 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T21:51:14.560 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:51:14.560 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:51:14.560 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T21:51:14.560 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:51:14.560 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:51:14.560 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T21:51:14.560 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 6 2024-04-20T21:51:14.560 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:51:14.561 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T21:51:14.561 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 2, 2024-04-20T21:51:14.561 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:51:14.561 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:51:14.561 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T21:51:14.561 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 2, 2024-04-20T21:51:14.561 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 12 2024-04-20T21:51:14.561 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T21:51:14.561 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:51:14.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:14.564+0000 7fbbf3200700 1 -- 172.21.15.62:0/776716675 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fbbd806ecc0 msgr2=0x7fbbd8071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:51:14.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:14.564+0000 7fbbf3200700 1 --2- 172.21.15.62:0/776716675 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fbbd806ecc0 0x7fbbd8071170 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fbbe4005fd0 tx=0x7fbbe400a560 comp rx=0 tx=0).stop 2024-04-20T21:51:14.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:14.564+0000 7fbbf3200700 1 -- 172.21.15.62:0/776716675 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbbf410b640 msgr2=0x7fbbf419c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:51:14.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:14.564+0000 7fbbf3200700 1 --2- 172.21.15.62:0/776716675 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbbf410b640 0x7fbbf419c6c0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fbbe000b5c0 tx=0x7fbbe000be00 comp rx=0 tx=0).stop 2024-04-20T21:51:14.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:14.564+0000 7fbbf3200700 1 -- 172.21.15.62:0/776716675 shutdown_connections 2024-04-20T21:51:14.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:14.564+0000 7fbbf3200700 1 --2- 172.21.15.62:0/776716675 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fbbd806ecc0 0x7fbbd8071170 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:51:14.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:14.564+0000 7fbbf3200700 1 --2- 172.21.15.62:0/776716675 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbbf4106b80 0x7fbbf419c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:51:14.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:14.564+0000 7fbbf3200700 1 --2- 172.21.15.62:0/776716675 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbbf410b640 0x7fbbf419c6c0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:51:14.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:14.564+0000 7fbbf3200700 1 -- 172.21.15.62:0/776716675 >> 172.21.15.62:0/776716675 conn(0x7fbbf4075e90 msgr2=0x7fbbf40778d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:51:14.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:14.564+0000 7fbbf3200700 1 -- 172.21.15.62:0/776716675 shutdown_connections 2024-04-20T21:51:14.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:14.564+0000 7fbbf3200700 1 -- 172.21.15.62:0/776716675 wait complete. 2024-04-20T21:51:14.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:14.713+0000 7fd2cb800700 1 -- 172.21.15.62:0/1228124673 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2c4106b80 msgr2=0x7fd2c4106ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:51:14.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:14.713+0000 7fd2cb800700 1 --2- 172.21.15.62:0/1228124673 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2c4106b80 0x7fd2c4106ff0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fd2b0009910 tx=0x7fd2b0009c20 comp rx=0 tx=0).stop 2024-04-20T21:51:14.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:14.713+0000 7fd2cb800700 1 -- 172.21.15.62:0/1228124673 shutdown_connections 2024-04-20T21:51:14.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:14.713+0000 7fd2cb800700 1 --2- 172.21.15.62:0/1228124673 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2c4106b80 0x7fd2c4106ff0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:51:14.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:14.713+0000 7fd2cb800700 1 --2- 172.21.15.62:0/1228124673 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2c410c8f0 0x7fd2c410ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:51:14.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:14.713+0000 7fd2cb800700 1 -- 172.21.15.62:0/1228124673 >> 172.21.15.62:0/1228124673 conn(0x7fd2c4076020 msgr2=0x7fd2c4078430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:51:14.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:14.713+0000 7fd2cb800700 1 -- 172.21.15.62:0/1228124673 shutdown_connections 2024-04-20T21:51:14.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:14.713+0000 7fd2cb800700 1 -- 172.21.15.62:0/1228124673 wait complete. 2024-04-20T21:51:14.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:14.714+0000 7fd2cb800700 1 Processor -- start 2024-04-20T21:51:14.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:14.714+0000 7fd2cb800700 1 -- start start 2024-04-20T21:51:14.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:14.715+0000 7fd2cb800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2c4106b80 0x7fd2c419c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:51:14.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:14.715+0000 7fd2cb800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2c410c8f0 0x7fd2c419c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:51:14.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:14.715+0000 7fd2cb800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2c419d070 con 0x7fd2c4106b80 2024-04-20T21:51:14.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:14.715+0000 7fd2cb800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2c41a0e00 con 0x7fd2c410c8f0 2024-04-20T21:51:14.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:14.715+0000 7fd2c8e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2c4106b80 0x7fd2c419c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:51:14.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:14.715+0000 7fd2c8e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2c4106b80 0x7fd2c419c450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37888/0 (socket says 172.21.15.62:37888) 2024-04-20T21:51:14.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:14.715+0000 7fd2c8e00700 1 -- 172.21.15.62:0/2504823256 learned_addr learned my addr 172.21.15.62:0/2504823256 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:51:14.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:14.715+0000 7fd2c8e00700 1 -- 172.21.15.62:0/2504823256 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2c410c8f0 msgr2=0x7fd2c419c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:51:14.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:14.715+0000 7fd2c8e00700 1 --2- 172.21.15.62:0/2504823256 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2c410c8f0 0x7fd2c419c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:51:14.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:14.715+0000 7fd2c8e00700 1 -- 172.21.15.62:0/2504823256 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2b00095f0 con 0x7fd2c4106b80 2024-04-20T21:51:14.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:14.716+0000 7fd2c8e00700 1 --2- 172.21.15.62:0/2504823256 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2c4106b80 0x7fd2c419c450 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fd2b800d8d0 tx=0x7fd2b800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:51:14.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:14.716+0000 7fd2c1600700 1 -- 172.21.15.62:0/2504823256 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd2b8009940 con 0x7fd2c4106b80 2024-04-20T21:51:14.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:14.716+0000 7fd2c1600700 1 -- 172.21.15.62:0/2504823256 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd2b8010460 con 0x7fd2c4106b80 2024-04-20T21:51:14.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:14.716+0000 7fd2cb800700 1 -- 172.21.15.62:0/2504823256 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2c41a10e0 con 0x7fd2c4106b80 2024-04-20T21:51:14.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:14.716+0000 7fd2c1600700 1 -- 172.21.15.62:0/2504823256 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd2b800f5f0 con 0x7fd2c4106b80 2024-04-20T21:51:14.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:14.716+0000 7fd2cb800700 1 -- 172.21.15.62:0/2504823256 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2c41a1630 con 0x7fd2c4106b80 2024-04-20T21:51:14.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:14.717+0000 7fd2cb800700 1 -- 172.21.15.62:0/2504823256 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2c404f2a0 con 0x7fd2c4106b80 2024-04-20T21:51:14.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:14.718+0000 7fd2c1600700 1 -- 172.21.15.62:0/2504823256 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fd2b80105d0 con 0x7fd2c4106b80 2024-04-20T21:51:14.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:14.719+0000 7fd2c1600700 1 --2- 172.21.15.62:0/2504823256 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd2ac06ecc0 0x7fd2ac071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:51:14.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:14.719+0000 7fd2c1600700 1 -- 172.21.15.62:0/2504823256 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fd2b808ed70 con 0x7fd2c4106b80 2024-04-20T21:51:14.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:14.720+0000 7fd2c3e00700 1 --2- 172.21.15.62:0/2504823256 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd2ac06ecc0 0x7fd2ac071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:51:14.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:14.720+0000 7fd2c3e00700 1 --2- 172.21.15.62:0/2504823256 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd2ac06ecc0 0x7fd2ac071170 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fd2b0006010 tx=0x7fd2b000b540 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:51:14.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:14.724+0000 7fd2c1600700 1 -- 172.21.15.62:0/2504823256 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fd2b8057ba0 con 0x7fd2c4106b80 2024-04-20T21:51:14.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:14.999+0000 7fd2cb800700 1 -- 172.21.15.62:0/2504823256 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fd2c41a19a0 con 0x7fd2c4106b80 2024-04-20T21:51:14.999 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:51:14 smithi062 ceph-mon[76322]: from='client.34900 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:51:14.999 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:51:14 smithi062 ceph-mon[76322]: from='client.34904 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:51:14.999 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:51:14 smithi062 ceph-mon[76322]: from='client.34908 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:51:14.999 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:51:14 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/776716675' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:51:15.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.000+0000 7fd2c1600700 1 -- 172.21.15.62:0/2504823256 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 25 v25) v1 ==== 76+0+1887 (secure 0 0 0) 0x7fd2b8016770 con 0x7fd2c4106b80 2024-04-20T21:51:15.000 INFO:teuthology.orchestra.run.smithi062.stdout:e25 2024-04-20T21:51:15.000 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T21:51:15.000 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T21:51:15.000 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T21:51:15.001 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:51:15.001 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T21:51:15.001 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T21:51:15.001 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 20 2024-04-20T21:51:15.001 INFO:teuthology.orchestra.run.smithi062.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-20T21:51:15.001 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T21:51:15.001 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T21:42:01.559061+0000 2024-04-20T21:51:15.001 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T21:51:15.001 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T21:51:15.001 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T21:51:15.001 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T21:51:15.001 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T21:51:15.001 INFO:teuthology.orchestra.run.smithi062.stdout:max_xattr_size 65536 2024-04-20T21:51:15.001 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T21:51:15.002 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T21:51:15.002 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 81 2024-04-20T21:51:15.002 INFO:teuthology.orchestra.run.smithi062.stdout:snap_rstat 1 2024-04-20T21:51:15.002 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T21:51:15.002 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 2 2024-04-20T21:51:15.002 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1 2024-04-20T21:51:15.002 INFO:teuthology.orchestra.run.smithi062.stdout:up {} 2024-04-20T21:51:15.002 INFO:teuthology.orchestra.run.smithi062.stdout:failed 0,1 2024-04-20T21:51:15.002 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T21:51:15.002 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T21:51:15.002 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T21:51:15.002 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T21:51:15.002 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data enabled 2024-04-20T21:51:15.002 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T21:51:15.003 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T21:51:15.003 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 1 2024-04-20T21:51:15.003 INFO:teuthology.orchestra.run.smithi062.stdout:qdb_cluster leader: 0 members: 2024-04-20T21:51:15.003 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:51:15.003 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:51:15.003 INFO:teuthology.orchestra.run.smithi062.stdout:Standby daemons: 2024-04-20T21:51:15.003 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:51:15.003 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ryqblb{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6826/4113916205,v1:172.21.15.62:6827/4113916205] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T21:51:15.003 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ozizwi{-1:34476} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6828/3801975165,v1:172.21.15.62:6829/3801975165] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T21:51:15.003 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.xbgspw{-1:44285} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6826/3996040526,v1:172.21.15.189:6827/3996040526] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:51:15.003 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.dvimer{-1:44289} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6824/1743111843,v1:172.21.15.189:6825/1743111843] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:51:15.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.004+0000 7fd2cb800700 1 -- 172.21.15.62:0/2504823256 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd2ac06ecc0 msgr2=0x7fd2ac071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:51:15.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.004+0000 7fd2cb800700 1 --2- 172.21.15.62:0/2504823256 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd2ac06ecc0 0x7fd2ac071170 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fd2b0006010 tx=0x7fd2b000b540 comp rx=0 tx=0).stop 2024-04-20T21:51:15.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.004+0000 7fd2cb800700 1 -- 172.21.15.62:0/2504823256 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2c4106b80 msgr2=0x7fd2c419c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:51:15.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.004+0000 7fd2cb800700 1 --2- 172.21.15.62:0/2504823256 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2c4106b80 0x7fd2c419c450 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fd2b800d8d0 tx=0x7fd2b800dc90 comp rx=0 tx=0).stop 2024-04-20T21:51:15.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.004+0000 7fd2cb800700 1 -- 172.21.15.62:0/2504823256 shutdown_connections 2024-04-20T21:51:15.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.004+0000 7fd2cb800700 1 --2- 172.21.15.62:0/2504823256 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd2ac06ecc0 0x7fd2ac071170 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:51:15.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.004+0000 7fd2cb800700 1 --2- 172.21.15.62:0/2504823256 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2c4106b80 0x7fd2c419c450 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:51:15.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.004+0000 7fd2cb800700 1 --2- 172.21.15.62:0/2504823256 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2c410c8f0 0x7fd2c419c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:51:15.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.004+0000 7fd2cb800700 1 -- 172.21.15.62:0/2504823256 >> 172.21.15.62:0/2504823256 conn(0x7fd2c4076020 msgr2=0x7fd2c4077a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:51:15.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.005+0000 7fd2cb800700 1 -- 172.21.15.62:0/2504823256 shutdown_connections 2024-04-20T21:51:15.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.005+0000 7fd2cb800700 1 -- 172.21.15.62:0/2504823256 wait complete. 2024-04-20T21:51:15.005 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 25 2024-04-20T21:51:15.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.166+0000 7f20b1400700 1 -- 172.21.15.62:0/3265770388 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f20ac100e80 msgr2=0x7f20ac104e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:51:15.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.166+0000 7f20b1400700 1 --2- 172.21.15.62:0/3265770388 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f20ac100e80 0x7f20ac104e50 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f2098009910 tx=0x7f2098009c20 comp rx=0 tx=0).stop 2024-04-20T21:51:15.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.166+0000 7f20b1400700 1 -- 172.21.15.62:0/3265770388 shutdown_connections 2024-04-20T21:51:15.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.166+0000 7f20b1400700 1 --2- 172.21.15.62:0/3265770388 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f20ac100e80 0x7f20ac104e50 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:51:15.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.166+0000 7f20b1400700 1 --2- 172.21.15.62:0/3265770388 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f20ac1004e0 0x7f20ac1008b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:51:15.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.166+0000 7f20b1400700 1 -- 172.21.15.62:0/3265770388 >> 172.21.15.62:0/3265770388 conn(0x7f20ac0fbda0 msgr2=0x7f20ac0fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:51:15.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.166+0000 7f20b1400700 1 -- 172.21.15.62:0/3265770388 shutdown_connections 2024-04-20T21:51:15.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.166+0000 7f20b1400700 1 -- 172.21.15.62:0/3265770388 wait complete. 2024-04-20T21:51:15.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.167+0000 7f20b1400700 1 Processor -- start 2024-04-20T21:51:15.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.167+0000 7f20b1400700 1 -- start start 2024-04-20T21:51:15.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.168+0000 7f20b1400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f20ac1004e0 0x7f20ac1980f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:51:15.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.168+0000 7f20b1400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f20ac100e80 0x7f20ac198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:51:15.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.168+0000 7f20b1400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20ac198c80 con 0x7f20ac1004e0 2024-04-20T21:51:15.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.168+0000 7f20b1400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20ac198dc0 con 0x7f20ac100e80 2024-04-20T21:51:15.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.168+0000 7f20aa800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f20ac1004e0 0x7f20ac1980f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:51:15.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.168+0000 7f20aa800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f20ac1004e0 0x7f20ac1980f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37894/0 (socket says 172.21.15.62:37894) 2024-04-20T21:51:15.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.168+0000 7f20aa800700 1 -- 172.21.15.62:0/1944887566 learned_addr learned my addr 172.21.15.62:0/1944887566 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:51:15.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.168+0000 7f20a9e00700 1 --2- 172.21.15.62:0/1944887566 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f20ac100e80 0x7f20ac198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:51:15.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.168+0000 7f20aa800700 1 -- 172.21.15.62:0/1944887566 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f20ac100e80 msgr2=0x7f20ac198630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:51:15.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.168+0000 7f20aa800700 1 --2- 172.21.15.62:0/1944887566 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f20ac100e80 0x7f20ac198630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:51:15.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.169+0000 7f20aa800700 1 -- 172.21.15.62:0/1944887566 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20980095f0 con 0x7f20ac1004e0 2024-04-20T21:51:15.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.169+0000 7f20a9e00700 1 --2- 172.21.15.62:0/1944887566 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f20ac100e80 0x7f20ac198630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T21:51:15.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.169+0000 7f20aa800700 1 --2- 172.21.15.62:0/1944887566 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f20ac1004e0 0x7f20ac1980f0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f209c00c930 tx=0x7f209c00ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:51:15.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.169+0000 7f20a3400700 1 -- 172.21.15.62:0/1944887566 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f209c007ad0 con 0x7f20ac1004e0 2024-04-20T21:51:15.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.169+0000 7f20a3400700 1 -- 172.21.15.62:0/1944887566 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f209c007c30 con 0x7f20ac1004e0 2024-04-20T21:51:15.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.169+0000 7f20b1400700 1 -- 172.21.15.62:0/1944887566 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20ac19cc10 con 0x7f20ac1004e0 2024-04-20T21:51:15.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.169+0000 7f20a3400700 1 -- 172.21.15.62:0/1944887566 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f209c018610 con 0x7f20ac1004e0 2024-04-20T21:51:15.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.169+0000 7f20b1400700 1 -- 172.21.15.62:0/1944887566 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20ac19d160 con 0x7f20ac1004e0 2024-04-20T21:51:15.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.170+0000 7f20b1400700 1 -- 172.21.15.62:0/1944887566 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20ac1045d0 con 0x7f20ac1004e0 2024-04-20T21:51:15.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.171+0000 7f20a3400700 1 -- 172.21.15.62:0/1944887566 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f209c00f450 con 0x7f20ac1004e0 2024-04-20T21:51:15.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.172+0000 7f20a3400700 1 --2- 172.21.15.62:0/1944887566 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f209006ecc0 0x7f2090071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:51:15.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.172+0000 7f20a3400700 1 -- 172.21.15.62:0/1944887566 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f209c08eff0 con 0x7f20ac1004e0 2024-04-20T21:51:15.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.173+0000 7f20a9e00700 1 --2- 172.21.15.62:0/1944887566 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f209006ecc0 0x7f2090071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:51:15.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.173+0000 7f20a9e00700 1 --2- 172.21.15.62:0/1944887566 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f209006ecc0 0x7f2090071170 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f2098006010 tx=0x7f209800b540 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:51:15.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.177+0000 7f20a3400700 1 -- 172.21.15.62:0/1944887566 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f209c057de0 con 0x7f20ac1004e0 2024-04-20T21:51:15.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:51:14 smithi189 ceph-mon[57022]: from='client.34900 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:51:15.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:51:14 smithi189 ceph-mon[57022]: from='client.34904 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:51:15.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:51:14 smithi189 ceph-mon[57022]: from='client.34908 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:51:15.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:51:14 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/776716675' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:51:15.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.426+0000 7f20b1400700 1 -- 172.21.15.62:0/1944887566 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f20ac19d3e0 con 0x7f209006ecc0 2024-04-20T21:51:15.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.428+0000 7f20a3400700 1 -- 172.21.15.62:0/1944887566 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f20ac19d3e0 con 0x7f209006ecc0 2024-04-20T21:51:15.427 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:51:15.428 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T21:51:15.428 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T21:51:15.428 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-20T21:51:15.428 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T21:51:15.428 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr", 2024-04-20T21:51:15.428 INFO:teuthology.orchestra.run.smithi062.stdout: "crash", 2024-04-20T21:51:15.428 INFO:teuthology.orchestra.run.smithi062.stdout: "mon", 2024-04-20T21:51:15.428 INFO:teuthology.orchestra.run.smithi062.stdout: "osd" 2024-04-20T21:51:15.428 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T21:51:15.428 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "14/23 daemons upgraded", 2024-04-20T21:51:15.428 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading mds daemons", 2024-04-20T21:51:15.428 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T21:51:15.428 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:51:15.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.431+0000 7f20b1400700 1 -- 172.21.15.62:0/1944887566 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f209006ecc0 msgr2=0x7f2090071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:51:15.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.431+0000 7f20b1400700 1 --2- 172.21.15.62:0/1944887566 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f209006ecc0 0x7f2090071170 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f2098006010 tx=0x7f209800b540 comp rx=0 tx=0).stop 2024-04-20T21:51:15.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.431+0000 7f20b1400700 1 -- 172.21.15.62:0/1944887566 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f20ac1004e0 msgr2=0x7f20ac1980f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:51:15.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.431+0000 7f20b1400700 1 --2- 172.21.15.62:0/1944887566 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f20ac1004e0 0x7f20ac1980f0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f209c00c930 tx=0x7f209c00ccf0 comp rx=0 tx=0).stop 2024-04-20T21:51:15.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.431+0000 7f20b1400700 1 -- 172.21.15.62:0/1944887566 shutdown_connections 2024-04-20T21:51:15.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.431+0000 7f20b1400700 1 --2- 172.21.15.62:0/1944887566 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f209006ecc0 0x7f2090071170 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:51:15.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.431+0000 7f20b1400700 1 --2- 172.21.15.62:0/1944887566 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f20ac1004e0 0x7f20ac1980f0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:51:15.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.431+0000 7f20b1400700 1 --2- 172.21.15.62:0/1944887566 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f20ac100e80 0x7f20ac198630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:51:15.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.431+0000 7f20b1400700 1 -- 172.21.15.62:0/1944887566 >> 172.21.15.62:0/1944887566 conn(0x7f20ac0fbda0 msgr2=0x7f20ac0fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:51:15.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.432+0000 7f20b1400700 1 -- 172.21.15.62:0/1944887566 shutdown_connections 2024-04-20T21:51:15.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.432+0000 7f20b1400700 1 -- 172.21.15.62:0/1944887566 wait complete. 2024-04-20T21:51:15.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.603+0000 7ff42ac00700 1 -- 172.21.15.62:0/1074292944 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff424106b80 msgr2=0x7ff424106ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:51:15.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.603+0000 7ff42ac00700 1 --2- 172.21.15.62:0/1074292944 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff424106b80 0x7ff424106ff0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7ff410009910 tx=0x7ff410009c20 comp rx=0 tx=0).stop 2024-04-20T21:51:15.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.603+0000 7ff42ac00700 1 -- 172.21.15.62:0/1074292944 shutdown_connections 2024-04-20T21:51:15.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.603+0000 7ff42ac00700 1 --2- 172.21.15.62:0/1074292944 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff424106b80 0x7ff424106ff0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:51:15.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.603+0000 7ff42ac00700 1 --2- 172.21.15.62:0/1074292944 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff42410c8f0 0x7ff42410ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:51:15.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.603+0000 7ff42ac00700 1 -- 172.21.15.62:0/1074292944 >> 172.21.15.62:0/1074292944 conn(0x7ff424076060 msgr2=0x7ff424078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:51:15.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.603+0000 7ff42ac00700 1 -- 172.21.15.62:0/1074292944 shutdown_connections 2024-04-20T21:51:15.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.603+0000 7ff42ac00700 1 -- 172.21.15.62:0/1074292944 wait complete. 2024-04-20T21:51:15.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.604+0000 7ff42ac00700 1 Processor -- start 2024-04-20T21:51:15.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.604+0000 7ff42ac00700 1 -- start start 2024-04-20T21:51:15.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.605+0000 7ff42ac00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff424106b80 0x7ff42419c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:51:15.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.605+0000 7ff42ac00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff42410c8f0 0x7ff42419c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:51:15.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.605+0000 7ff42ac00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff42419d060 con 0x7ff42410c8f0 2024-04-20T21:51:15.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.605+0000 7ff42ac00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4241a0df0 con 0x7ff424106b80 2024-04-20T21:51:15.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.605+0000 7ff423400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff42410c8f0 0x7ff42419c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:51:15.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.605+0000 7ff423400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff42410c8f0 0x7ff42419c980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37922/0 (socket says 172.21.15.62:37922) 2024-04-20T21:51:15.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.605+0000 7ff423400700 1 -- 172.21.15.62:0/3375329211 learned_addr learned my addr 172.21.15.62:0/3375329211 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:51:15.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.605+0000 7ff423e00700 1 --2- 172.21.15.62:0/3375329211 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff424106b80 0x7ff42419c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:51:15.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.606+0000 7ff423400700 1 -- 172.21.15.62:0/3375329211 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff424106b80 msgr2=0x7ff42419c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:51:15.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.606+0000 7ff423400700 1 --2- 172.21.15.62:0/3375329211 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff424106b80 0x7ff42419c440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:51:15.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.606+0000 7ff423400700 1 -- 172.21.15.62:0/3375329211 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4100095f0 con 0x7ff42410c8f0 2024-04-20T21:51:15.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.606+0000 7ff423400700 1 --2- 172.21.15.62:0/3375329211 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff42410c8f0 0x7ff42419c980 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7ff41000b5c0 tx=0x7ff41000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:51:15.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.606+0000 7ff420c00700 1 -- 172.21.15.62:0/3375329211 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff41001d070 con 0x7ff42410c8f0 2024-04-20T21:51:15.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.606+0000 7ff420c00700 1 -- 172.21.15.62:0/3375329211 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff410005e00 con 0x7ff42410c8f0 2024-04-20T21:51:15.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.606+0000 7ff42ac00700 1 -- 172.21.15.62:0/3375329211 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4241a1070 con 0x7ff42410c8f0 2024-04-20T21:51:15.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.606+0000 7ff420c00700 1 -- 172.21.15.62:0/3375329211 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff410017400 con 0x7ff42410c8f0 2024-04-20T21:51:15.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.606+0000 7ff42ac00700 1 -- 172.21.15.62:0/3375329211 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4241a1560 con 0x7ff42410c8f0 2024-04-20T21:51:15.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.608+0000 7ff42ac00700 1 -- 172.21.15.62:0/3375329211 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff42410ecc0 con 0x7ff42410c8f0 2024-04-20T21:51:15.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.609+0000 7ff420c00700 1 -- 172.21.15.62:0/3375329211 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff41000f460 con 0x7ff42410c8f0 2024-04-20T21:51:15.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.610+0000 7ff420c00700 1 --2- 172.21.15.62:0/3375329211 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff40806ecc0 0x7ff408071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:51:15.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.610+0000 7ff420c00700 1 -- 172.21.15.62:0/3375329211 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7ff410090540 con 0x7ff42410c8f0 2024-04-20T21:51:15.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.611+0000 7ff423e00700 1 --2- 172.21.15.62:0/3375329211 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff40806ecc0 0x7ff408071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:51:15.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.611+0000 7ff423e00700 1 --2- 172.21.15.62:0/3375329211 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff40806ecc0 0x7ff408071170 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7ff414005950 tx=0x7ff4140058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:51:15.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.618+0000 7ff420c00700 1 -- 172.21.15.62:0/3375329211 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7ff410059350 con 0x7ff42410c8f0 2024-04-20T21:51:15.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.948+0000 7ff42ac00700 1 -- 172.21.15.62:0/3375329211 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7ff42404f2a0 con 0x7ff42410c8f0 2024-04-20T21:51:15.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.948+0000 7ff420c00700 1 -- 172.21.15.62:0/3375329211 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7ff4100163b0 con 0x7ff42410c8f0 2024-04-20T21:51:15.948 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-20T21:51:15.948 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-20T21:51:15.948 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is degraded 2024-04-20T21:51:15.948 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T21:51:15.948 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T21:51:15.948 INFO:teuthology.orchestra.run.smithi062.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-20T21:51:15.948 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-20T21:51:15.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.951+0000 7ff42ac00700 1 -- 172.21.15.62:0/3375329211 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff40806ecc0 msgr2=0x7ff408071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:51:15.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.951+0000 7ff42ac00700 1 --2- 172.21.15.62:0/3375329211 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff40806ecc0 0x7ff408071170 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7ff414005950 tx=0x7ff4140058e0 comp rx=0 tx=0).stop 2024-04-20T21:51:15.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.951+0000 7ff42ac00700 1 -- 172.21.15.62:0/3375329211 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff42410c8f0 msgr2=0x7ff42419c980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:51:15.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.951+0000 7ff42ac00700 1 --2- 172.21.15.62:0/3375329211 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff42410c8f0 0x7ff42419c980 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7ff41000b5c0 tx=0x7ff41000be20 comp rx=0 tx=0).stop 2024-04-20T21:51:15.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.952+0000 7ff42ac00700 1 -- 172.21.15.62:0/3375329211 shutdown_connections 2024-04-20T21:51:15.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.952+0000 7ff42ac00700 1 --2- 172.21.15.62:0/3375329211 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff40806ecc0 0x7ff408071170 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:51:15.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.952+0000 7ff42ac00700 1 --2- 172.21.15.62:0/3375329211 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff424106b80 0x7ff42419c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:51:15.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.952+0000 7ff42ac00700 1 --2- 172.21.15.62:0/3375329211 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff42410c8f0 0x7ff42419c980 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:51:15.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.952+0000 7ff42ac00700 1 -- 172.21.15.62:0/3375329211 >> 172.21.15.62:0/3375329211 conn(0x7ff424076060 msgr2=0x7ff424077ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:51:15.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.952+0000 7ff42ac00700 1 -- 172.21.15.62:0/3375329211 shutdown_connections 2024-04-20T21:51:15.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:15.952+0000 7ff42ac00700 1 -- 172.21.15.62:0/3375329211 wait complete. 2024-04-20T21:51:16.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:51:15 smithi189 ceph-mon[57022]: pgmap v478: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:51:16.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:51:15 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/2504823256' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T21:51:16.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:51:15 smithi062 ceph-mon[76322]: pgmap v478: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:51:16.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:51:15 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/2504823256' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T21:51:17.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:51:16 smithi189 ceph-mon[57022]: from='client.34920 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:51:17.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:51:16 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/3375329211' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T21:51:17.284 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:51:16 smithi062 ceph-mon[76322]: from='client.34920 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:51:17.284 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:51:16 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/3375329211' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T21:51:18.143 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:51:17 smithi062 ceph-mon[76322]: pgmap v479: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:51:18.195 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:51:17 smithi189 ceph-mon[57022]: pgmap v479: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:51:20.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:51:19 smithi189 ceph-mon[57022]: pgmap v480: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:51:20.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:51:19 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:51:20.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:51:19 smithi062 ceph-mon[76322]: pgmap v480: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:51:20.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:51:19 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:51:22.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:51:21 smithi189 ceph-mon[57022]: pgmap v481: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:51:22.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:51:21 smithi062 ceph-mon[76322]: pgmap v481: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:51:24.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:51:23 smithi189 ceph-mon[57022]: pgmap v482: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:51:24.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:51:23 smithi062 ceph-mon[76322]: pgmap v482: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:51:25.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:51:25 smithi189 ceph-mon[57022]: pgmap v483: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:51:25.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:51:25 smithi062 ceph-mon[76322]: pgmap v483: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:51:27.990 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:51:27 smithi189 ceph-mon[57022]: pgmap v484: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:51:28.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:51:27 smithi062 ceph-mon[76322]: pgmap v484: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:51:29.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:51:29 smithi189 ceph-mon[57022]: pgmap v485: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:51:30.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:51:29 smithi062 ceph-mon[76322]: pgmap v485: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:51:31.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:51:31 smithi189 ceph-mon[57022]: pgmap v486: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:51:32.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:51:31 smithi062 ceph-mon[76322]: pgmap v486: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:51:33.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:51:33 smithi189 ceph-mon[57022]: pgmap v487: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:51:34.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:51:33 smithi062 ceph-mon[76322]: pgmap v487: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:51:34.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:51:34 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:51:35.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:51:34 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:51:35.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:51:35 smithi189 ceph-mon[57022]: pgmap v488: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:51:36.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:51:35 smithi062 ceph-mon[76322]: pgmap v488: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:51:37.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:51:37 smithi189 ceph-mon[57022]: pgmap v489: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:51:38.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:51:37 smithi062 ceph-mon[76322]: pgmap v489: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:51:39.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:51:39 smithi189 ceph-mon[57022]: pgmap v490: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:51:40.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:51:39 smithi062 ceph-mon[76322]: pgmap v490: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:51:41.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:51:41 smithi189 ceph-mon[57022]: pgmap v491: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:51:42.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:51:41 smithi062 ceph-mon[76322]: pgmap v491: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:51:43.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:51:43 smithi189 ceph-mon[57022]: pgmap v492: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:51:44.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:51:43 smithi062 ceph-mon[76322]: pgmap v492: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:51:45.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:51:45 smithi189 ceph-mon[57022]: pgmap v493: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:51:46.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:51:45 smithi062 ceph-mon[76322]: pgmap v493: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:51:46.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:46.132+0000 7f1923600700 1 -- 172.21.15.62:0/3313651106 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f191c10c8f0 msgr2=0x7f191c10ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:51:46.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:46.132+0000 7f1923600700 1 --2- 172.21.15.62:0/3313651106 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f191c10c8f0 0x7f191c10ccc0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f1910009910 tx=0x7f1910009c20 comp rx=0 tx=0).stop 2024-04-20T21:51:46.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:46.132+0000 7f1923600700 1 -- 172.21.15.62:0/3313651106 shutdown_connections 2024-04-20T21:51:46.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:46.132+0000 7f1923600700 1 --2- 172.21.15.62:0/3313651106 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f191c106b80 0x7f191c106ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:51:46.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:46.132+0000 7f1923600700 1 --2- 172.21.15.62:0/3313651106 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f191c10c8f0 0x7f191c10ccc0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:51:46.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:46.132+0000 7f1923600700 1 -- 172.21.15.62:0/3313651106 >> 172.21.15.62:0/3313651106 conn(0x7f191c076060 msgr2=0x7f191c078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:51:46.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:46.132+0000 7f1923600700 1 -- 172.21.15.62:0/3313651106 shutdown_connections 2024-04-20T21:51:46.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:46.132+0000 7f1923600700 1 -- 172.21.15.62:0/3313651106 wait complete. 2024-04-20T21:51:46.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:46.133+0000 7f1923600700 1 Processor -- start 2024-04-20T21:51:46.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:46.134+0000 7f1923600700 1 -- start start 2024-04-20T21:51:46.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:46.134+0000 7f1923600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f191c106b80 0x7f191c19c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:51:46.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:46.134+0000 7f1923600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f191c10c8f0 0x7f191c19ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:51:46.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:46.134+0000 7f1923600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f191c19d090 con 0x7f191c106b80 2024-04-20T21:51:46.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:46.134+0000 7f1923600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f191c19d1d0 con 0x7f191c10c8f0 2024-04-20T21:51:46.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:46.135+0000 7f1920c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f191c106b80 0x7f191c19c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:51:46.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:46.135+0000 7f1920c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f191c106b80 0x7f191c19c500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50966/0 (socket says 172.21.15.62:50966) 2024-04-20T21:51:46.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:46.135+0000 7f1920c00700 1 -- 172.21.15.62:0/2843788555 learned_addr learned my addr 172.21.15.62:0/2843788555 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:51:46.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:46.135+0000 7f191be00700 1 --2- 172.21.15.62:0/2843788555 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f191c10c8f0 0x7f191c19ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:51:46.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:46.135+0000 7f1920c00700 1 -- 172.21.15.62:0/2843788555 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f191c10c8f0 msgr2=0x7f191c19ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:51:46.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:46.135+0000 7f1920c00700 1 --2- 172.21.15.62:0/2843788555 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f191c10c8f0 0x7f191c19ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:51:46.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:46.135+0000 7f1920c00700 1 -- 172.21.15.62:0/2843788555 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19100095f0 con 0x7f191c106b80 2024-04-20T21:51:46.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:46.135+0000 7f191be00700 1 --2- 172.21.15.62:0/2843788555 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f191c10c8f0 0x7f191c19ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T21:51:46.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:46.135+0000 7f1920c00700 1 --2- 172.21.15.62:0/2843788555 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f191c106b80 0x7f191c19c500 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f191000b5c0 tx=0x7f191000be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:51:46.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:46.136+0000 7f1919600700 1 -- 172.21.15.62:0/2843788555 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f191001d070 con 0x7f191c106b80 2024-04-20T21:51:46.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:46.136+0000 7f1919600700 1 -- 172.21.15.62:0/2843788555 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f191000f460 con 0x7f191c106b80 2024-04-20T21:51:46.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:46.136+0000 7f1919600700 1 -- 172.21.15.62:0/2843788555 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1910017610 con 0x7f191c106b80 2024-04-20T21:51:46.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:46.136+0000 7f1923600700 1 -- 172.21.15.62:0/2843788555 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f191c1a0fc0 con 0x7f191c106b80 2024-04-20T21:51:46.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:46.136+0000 7f1923600700 1 -- 172.21.15.62:0/2843788555 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f191c1a14b0 con 0x7f191c106b80 2024-04-20T21:51:46.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:46.137+0000 7f1923600700 1 -- 172.21.15.62:0/2843788555 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f191c109b20 con 0x7f191c106b80 2024-04-20T21:51:46.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:46.138+0000 7f1919600700 1 -- 172.21.15.62:0/2843788555 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f191000f5d0 con 0x7f191c106b80 2024-04-20T21:51:46.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:46.139+0000 7f1919600700 1 --2- 172.21.15.62:0/2843788555 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f190406ecc0 0x7f1904071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:51:46.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:46.140+0000 7f1919600700 1 -- 172.21.15.62:0/2843788555 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f1910090440 con 0x7f191c106b80 2024-04-20T21:51:46.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:46.140+0000 7f191be00700 1 --2- 172.21.15.62:0/2843788555 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f190406ecc0 0x7f1904071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:51:46.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:46.140+0000 7f191be00700 1 --2- 172.21.15.62:0/2843788555 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f190406ecc0 0x7f1904071170 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f1908005fd0 tx=0x7f1908005e20 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:51:46.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:46.145+0000 7f1919600700 1 -- 172.21.15.62:0/2843788555 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f19100591d0 con 0x7f191c106b80 2024-04-20T21:51:46.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:46.402+0000 7f1923600700 1 -- 172.21.15.62:0/2843788555 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f191c19d970 con 0x7f190406ecc0 2024-04-20T21:51:46.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:46.404+0000 7f1919600700 1 -- 172.21.15.62:0/2843788555 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f191c19d970 con 0x7f190406ecc0 2024-04-20T21:51:46.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:46.407+0000 7f1923600700 1 -- 172.21.15.62:0/2843788555 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f190406ecc0 msgr2=0x7f1904071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:51:46.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:46.407+0000 7f1923600700 1 --2- 172.21.15.62:0/2843788555 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f190406ecc0 0x7f1904071170 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f1908005fd0 tx=0x7f1908005e20 comp rx=0 tx=0).stop 2024-04-20T21:51:46.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:46.407+0000 7f1923600700 1 -- 172.21.15.62:0/2843788555 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f191c106b80 msgr2=0x7f191c19c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:51:46.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:46.407+0000 7f1923600700 1 --2- 172.21.15.62:0/2843788555 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f191c106b80 0x7f191c19c500 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f191000b5c0 tx=0x7f191000be00 comp rx=0 tx=0).stop 2024-04-20T21:51:46.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:46.407+0000 7f1923600700 1 -- 172.21.15.62:0/2843788555 shutdown_connections 2024-04-20T21:51:46.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:46.407+0000 7f1923600700 1 --2- 172.21.15.62:0/2843788555 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f190406ecc0 0x7f1904071170 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:51:46.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:46.407+0000 7f1923600700 1 --2- 172.21.15.62:0/2843788555 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f191c106b80 0x7f191c19c500 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:51:46.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:46.407+0000 7f1923600700 1 --2- 172.21.15.62:0/2843788555 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f191c10c8f0 0x7f191c19ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:51:46.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:46.407+0000 7f1923600700 1 -- 172.21.15.62:0/2843788555 >> 172.21.15.62:0/2843788555 conn(0x7f191c076060 msgr2=0x7f191c077aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:51:46.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:46.408+0000 7f1923600700 1 -- 172.21.15.62:0/2843788555 shutdown_connections 2024-04-20T21:51:46.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:46.408+0000 7f1923600700 1 -- 172.21.15.62:0/2843788555 wait complete. 2024-04-20T21:51:46.424 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T21:51:46.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:46.578+0000 7fe78dc00700 1 -- 172.21.15.62:0/1732529496 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe7881002b0 msgr2=0x7fe788100680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:51:46.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:46.578+0000 7fe78dc00700 1 --2- 172.21.15.62:0/1732529496 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe7881002b0 0x7fe788100680 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fe778009910 tx=0x7fe778009c20 comp rx=0 tx=0).stop 2024-04-20T21:51:46.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:46.578+0000 7fe78dc00700 1 -- 172.21.15.62:0/1732529496 shutdown_connections 2024-04-20T21:51:46.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:46.578+0000 7fe78dc00700 1 --2- 172.21.15.62:0/1732529496 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe788100c50 0x7fe788104c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:51:46.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:46.578+0000 7fe78dc00700 1 --2- 172.21.15.62:0/1732529496 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe7881002b0 0x7fe788100680 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:51:46.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:46.578+0000 7fe78dc00700 1 -- 172.21.15.62:0/1732529496 >> 172.21.15.62:0/1732529496 conn(0x7fe7880fbb50 msgr2=0x7fe7880fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:51:46.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:46.579+0000 7fe78dc00700 1 -- 172.21.15.62:0/1732529496 shutdown_connections 2024-04-20T21:51:46.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:46.579+0000 7fe78dc00700 1 -- 172.21.15.62:0/1732529496 wait complete. 2024-04-20T21:51:46.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:46.580+0000 7fe78dc00700 1 Processor -- start 2024-04-20T21:51:46.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:46.580+0000 7fe78dc00700 1 -- start start 2024-04-20T21:51:46.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:46.581+0000 7fe78dc00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe7881002b0 0x7fe788073050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:51:46.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:46.581+0000 7fe78dc00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe788100c50 0x7fe788073590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:51:46.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:46.581+0000 7fe78dc00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe788073ad0 con 0x7fe7881002b0 2024-04-20T21:51:46.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:46.581+0000 7fe78dc00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe788073c10 con 0x7fe788100c50 2024-04-20T21:51:46.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:46.581+0000 7fe787e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe7881002b0 0x7fe788073050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:51:46.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:46.581+0000 7fe787e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe7881002b0 0x7fe788073050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50984/0 (socket says 172.21.15.62:50984) 2024-04-20T21:51:46.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:46.581+0000 7fe787e00700 1 -- 172.21.15.62:0/1103651619 learned_addr learned my addr 172.21.15.62:0/1103651619 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:51:46.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:46.581+0000 7fe787400700 1 --2- 172.21.15.62:0/1103651619 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe788100c50 0x7fe788073590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:51:46.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:46.582+0000 7fe787e00700 1 -- 172.21.15.62:0/1103651619 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe788100c50 msgr2=0x7fe788073590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:51:46.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:46.582+0000 7fe787e00700 1 --2- 172.21.15.62:0/1103651619 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe788100c50 0x7fe788073590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:51:46.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:46.582+0000 7fe787e00700 1 -- 172.21.15.62:0/1103651619 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7780095f0 con 0x7fe7881002b0 2024-04-20T21:51:46.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:46.582+0000 7fe787e00700 1 --2- 172.21.15.62:0/1103651619 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe7881002b0 0x7fe788073050 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fe77800f690 tx=0x7fe77800f770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:51:46.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:46.582+0000 7fe784c00700 1 -- 172.21.15.62:0/1103651619 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe77801d070 con 0x7fe7881002b0 2024-04-20T21:51:46.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:46.582+0000 7fe784c00700 1 -- 172.21.15.62:0/1103651619 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe7780050d0 con 0x7fe7881002b0 2024-04-20T21:51:46.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:46.582+0000 7fe78dc00700 1 -- 172.21.15.62:0/1103651619 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe788073e90 con 0x7fe7881002b0 2024-04-20T21:51:46.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:46.582+0000 7fe784c00700 1 -- 172.21.15.62:0/1103651619 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe778022470 con 0x7fe7881002b0 2024-04-20T21:51:46.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:46.583+0000 7fe78dc00700 1 -- 172.21.15.62:0/1103651619 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7881a2150 con 0x7fe7881002b0 2024-04-20T21:51:46.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:46.584+0000 7fe78dc00700 1 -- 172.21.15.62:0/1103651619 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7881085c0 con 0x7fe7881002b0 2024-04-20T21:51:46.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:46.586+0000 7fe784c00700 1 -- 172.21.15.62:0/1103651619 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe778003680 con 0x7fe7881002b0 2024-04-20T21:51:46.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:46.587+0000 7fe784c00700 1 --2- 172.21.15.62:0/1103651619 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe76c06ecc0 0x7fe76c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:51:46.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:46.588+0000 7fe784c00700 1 -- 172.21.15.62:0/1103651619 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fe778090380 con 0x7fe7881002b0 2024-04-20T21:51:46.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:46.588+0000 7fe787400700 1 --2- 172.21.15.62:0/1103651619 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe76c06ecc0 0x7fe76c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:51:46.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:46.589+0000 7fe787400700 1 --2- 172.21.15.62:0/1103651619 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe76c06ecc0 0x7fe76c071170 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fe788074840 tx=0x7fe774009260 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:51:46.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:46.594+0000 7fe784c00700 1 -- 172.21.15.62:0/1103651619 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fe778059090 con 0x7fe7881002b0 2024-04-20T21:51:46.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:46.852+0000 7fe78dc00700 1 -- 172.21.15.62:0/1103651619 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe788061190 con 0x7fe76c06ecc0 2024-04-20T21:51:46.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:46.854+0000 7fe784c00700 1 -- 172.21.15.62:0/1103651619 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fe788061190 con 0x7fe76c06ecc0 2024-04-20T21:51:46.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:46.857+0000 7fe78dc00700 1 -- 172.21.15.62:0/1103651619 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe76c06ecc0 msgr2=0x7fe76c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:51:46.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:46.857+0000 7fe78dc00700 1 --2- 172.21.15.62:0/1103651619 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe76c06ecc0 0x7fe76c071170 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fe788074840 tx=0x7fe774009260 comp rx=0 tx=0).stop 2024-04-20T21:51:46.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:46.857+0000 7fe78dc00700 1 -- 172.21.15.62:0/1103651619 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe7881002b0 msgr2=0x7fe788073050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:51:46.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:46.857+0000 7fe78dc00700 1 --2- 172.21.15.62:0/1103651619 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe7881002b0 0x7fe788073050 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fe77800f690 tx=0x7fe77800f770 comp rx=0 tx=0).stop 2024-04-20T21:51:46.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:46.858+0000 7fe78dc00700 1 -- 172.21.15.62:0/1103651619 shutdown_connections 2024-04-20T21:51:46.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:46.858+0000 7fe78dc00700 1 --2- 172.21.15.62:0/1103651619 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe76c06ecc0 0x7fe76c071170 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:51:46.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:46.858+0000 7fe78dc00700 1 --2- 172.21.15.62:0/1103651619 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe7881002b0 0x7fe788073050 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:51:46.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:46.858+0000 7fe78dc00700 1 --2- 172.21.15.62:0/1103651619 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe788100c50 0x7fe788073590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:51:46.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:46.858+0000 7fe78dc00700 1 -- 172.21.15.62:0/1103651619 >> 172.21.15.62:0/1103651619 conn(0x7fe7880fbb50 msgr2=0x7fe7880fdf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:51:46.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:46.858+0000 7fe78dc00700 1 -- 172.21.15.62:0/1103651619 shutdown_connections 2024-04-20T21:51:46.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:46.858+0000 7fe78dc00700 1 -- 172.21.15.62:0/1103651619 wait complete. 2024-04-20T21:51:47.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.017+0000 7f5d8b000700 1 -- 172.21.15.62:0/1534755541 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d840739a0 msgr2=0x7f5d8410c5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:51:47.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.017+0000 7f5d8b000700 1 --2- 172.21.15.62:0/1534755541 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d840739a0 0x7f5d8410c5c0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f5d70009940 tx=0x7f5d70009c50 comp rx=0 tx=0).stop 2024-04-20T21:51:47.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.017+0000 7f5d8b000700 1 -- 172.21.15.62:0/1534755541 shutdown_connections 2024-04-20T21:51:47.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.017+0000 7f5d8b000700 1 --2- 172.21.15.62:0/1534755541 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d840739a0 0x7f5d8410c5c0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:51:47.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.017+0000 7f5d8b000700 1 --2- 172.21.15.62:0/1534755541 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d84073090 0x7f5d84073460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:51:47.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.017+0000 7f5d8b000700 1 -- 172.21.15.62:0/1534755541 >> 172.21.15.62:0/1534755541 conn(0x7f5d840fbd80 msgr2=0x7f5d840fe190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:51:47.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.018+0000 7f5d8b000700 1 -- 172.21.15.62:0/1534755541 shutdown_connections 2024-04-20T21:51:47.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.018+0000 7f5d8b000700 1 -- 172.21.15.62:0/1534755541 wait complete. 2024-04-20T21:51:47.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.019+0000 7f5d8b000700 1 Processor -- start 2024-04-20T21:51:47.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.019+0000 7f5d8b000700 1 -- start start 2024-04-20T21:51:47.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.019+0000 7f5d8b000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d84073090 0x7f5d84198220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:51:47.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.019+0000 7f5d8b000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d840739a0 0x7f5d84198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:51:47.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.019+0000 7f5d8b000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d84198e40 con 0x7f5d840739a0 2024-04-20T21:51:47.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.019+0000 7f5d8b000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d8419cbd0 con 0x7f5d84073090 2024-04-20T21:51:47.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.020+0000 7f5d83400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d840739a0 0x7f5d84198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:51:47.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.020+0000 7f5d83400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d840739a0 0x7f5d84198760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51004/0 (socket says 172.21.15.62:51004) 2024-04-20T21:51:47.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.020+0000 7f5d83400700 1 -- 172.21.15.62:0/3300052915 learned_addr learned my addr 172.21.15.62:0/3300052915 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:51:47.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.020+0000 7f5d83e00700 1 --2- 172.21.15.62:0/3300052915 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d84073090 0x7f5d84198220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:51:47.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.020+0000 7f5d83400700 1 -- 172.21.15.62:0/3300052915 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d84073090 msgr2=0x7f5d84198220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:51:47.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.020+0000 7f5d83400700 1 --2- 172.21.15.62:0/3300052915 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d84073090 0x7f5d84198220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:51:47.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.020+0000 7f5d83400700 1 -- 172.21.15.62:0/3300052915 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d700095f0 con 0x7f5d840739a0 2024-04-20T21:51:47.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.021+0000 7f5d83400700 1 --2- 172.21.15.62:0/3300052915 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d840739a0 0x7f5d84198760 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f5d700055f0 tx=0x7f5d700056d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:51:47.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.021+0000 7f5d80c00700 1 -- 172.21.15.62:0/3300052915 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5d7001d070 con 0x7f5d840739a0 2024-04-20T21:51:47.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.021+0000 7f5d8b000700 1 -- 172.21.15.62:0/3300052915 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d8419ce50 con 0x7f5d840739a0 2024-04-20T21:51:47.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.021+0000 7f5d80c00700 1 -- 172.21.15.62:0/3300052915 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5d7000f460 con 0x7f5d840739a0 2024-04-20T21:51:47.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.021+0000 7f5d8b000700 1 -- 172.21.15.62:0/3300052915 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d8419d340 con 0x7f5d840739a0 2024-04-20T21:51:47.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.021+0000 7f5d80c00700 1 -- 172.21.15.62:0/3300052915 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5d700215e0 con 0x7f5d840739a0 2024-04-20T21:51:47.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.023+0000 7f5d8b000700 1 -- 172.21.15.62:0/3300052915 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d84109cc0 con 0x7f5d840739a0 2024-04-20T21:51:47.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.024+0000 7f5d80c00700 1 -- 172.21.15.62:0/3300052915 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f5d7000fa60 con 0x7f5d840739a0 2024-04-20T21:51:47.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.025+0000 7f5d80c00700 1 --2- 172.21.15.62:0/3300052915 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5d6806ecc0 0x7f5d68071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:51:47.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.025+0000 7f5d80c00700 1 -- 172.21.15.62:0/3300052915 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f5d700900e0 con 0x7f5d840739a0 2024-04-20T21:51:47.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.025+0000 7f5d83e00700 1 --2- 172.21.15.62:0/3300052915 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5d6806ecc0 0x7f5d68071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:51:47.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.026+0000 7f5d83e00700 1 --2- 172.21.15.62:0/3300052915 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5d6806ecc0 0x7f5d68071170 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f5d74005fd0 tx=0x7f5d7400a560 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:51:47.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.030+0000 7f5d80c00700 1 -- 172.21.15.62:0/3300052915 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f5d70058ea0 con 0x7f5d840739a0 2024-04-20T21:51:47.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.269+0000 7f5d8b000700 1 -- 172.21.15.62:0/3300052915 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f5d84199690 con 0x7f5d6806ecc0 2024-04-20T21:51:47.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.303+0000 7f5d80c00700 1 -- 172.21.15.62:0/3300052915 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f5d84199690 con 0x7f5d6806ecc0 2024-04-20T21:51:47.303 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T21:51:47.303 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (17m) 8m ago 31m 14.2M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T21:51:47.303 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (31m) 8m ago 31m 9512k - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T21:51:47.303 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (30m) 9m ago 30m 8942k - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T21:51:47.303 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (13m) 8m ago 31m 6660k - 19.0.0-3133-ga8c84f59 76bc6b91b726 2825d5ba11c6 2024-04-20T21:51:47.303 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (13m) 9m ago 30m 6665k - 19.0.0-3133-ga8c84f59 76bc6b91b726 20f746fa1bba 2024-04-20T21:51:47.303 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (16m) 8m ago 31m 74.2M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T21:51:47.303 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (9m) 8m ago 24m 11.2M - 19.0.0-3133-ga8c84f59 76bc6b91b726 de80bc90a683 2024-04-20T21:51:47.303 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (9m) 8m ago 24m 11.9M - 19.0.0-3133-ga8c84f59 76bc6b91b726 cd793b7ee98e 2024-04-20T21:51:47.303 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (24m) 9m ago 24m 117M - 18.2.0 dc2bc1663786 fb2a8a223db8 2024-04-20T21:51:47.303 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (24m) 9m ago 24m 55.6M - 18.2.0 dc2bc1663786 f2eec2ee8951 2024-04-20T21:51:47.303 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (20m) 8m ago 33m 511M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T21:51:47.303 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (19m) 9m ago 30m 452M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T21:51:47.304 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (15m) 8m ago 33m 55.9M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 019eb1479485 2024-04-20T21:51:47.304 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (14m) 9m ago 30m 39.8M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 55aac560aab6 2024-04-20T21:51:47.304 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (18m) 8m ago 31m 7952k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T21:51:47.304 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (18m) 9m ago 30m 8304k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T21:51:47.304 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (13m) 8m ago 29m 88.0M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 ee2bb901f26e 2024-04-20T21:51:47.304 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (12m) 8m ago 28m 92.3M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 9075152d4162 2024-04-20T21:51:47.304 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (12m) 8m ago 27m 74.4M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 2d4f3e38dc0e 2024-04-20T21:51:47.304 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (11m) 9m ago 27m 87.6M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 8f97b17ae0eb 2024-04-20T21:51:47.304 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (10m) 9m ago 26m 90.7M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 653d8079967d 2024-04-20T21:51:47.304 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (10m) 9m ago 26m 11.5M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 596dded14c12 2024-04-20T21:51:47.304 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (18m) 8m ago 30m 74.0M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T21:51:47.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.306+0000 7f5d8b000700 1 -- 172.21.15.62:0/3300052915 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5d6806ecc0 msgr2=0x7f5d68071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:51:47.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.306+0000 7f5d8b000700 1 --2- 172.21.15.62:0/3300052915 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5d6806ecc0 0x7f5d68071170 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f5d74005fd0 tx=0x7f5d7400a560 comp rx=0 tx=0).stop 2024-04-20T21:51:47.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.306+0000 7f5d8b000700 1 -- 172.21.15.62:0/3300052915 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d840739a0 msgr2=0x7f5d84198760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:51:47.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.306+0000 7f5d8b000700 1 --2- 172.21.15.62:0/3300052915 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d840739a0 0x7f5d84198760 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f5d700055f0 tx=0x7f5d700056d0 comp rx=0 tx=0).stop 2024-04-20T21:51:47.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.306+0000 7f5d8b000700 1 -- 172.21.15.62:0/3300052915 shutdown_connections 2024-04-20T21:51:47.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.306+0000 7f5d8b000700 1 --2- 172.21.15.62:0/3300052915 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5d6806ecc0 0x7f5d68071170 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:51:47.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.306+0000 7f5d8b000700 1 --2- 172.21.15.62:0/3300052915 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d84073090 0x7f5d84198220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:51:47.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.306+0000 7f5d8b000700 1 --2- 172.21.15.62:0/3300052915 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d840739a0 0x7f5d84198760 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:51:47.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.306+0000 7f5d8b000700 1 -- 172.21.15.62:0/3300052915 >> 172.21.15.62:0/3300052915 conn(0x7f5d840fbd80 msgr2=0x7f5d84106e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:51:47.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.307+0000 7f5d8b000700 1 -- 172.21.15.62:0/3300052915 shutdown_connections 2024-04-20T21:51:47.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.307+0000 7f5d8b000700 1 -- 172.21.15.62:0/3300052915 wait complete. 2024-04-20T21:51:47.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.459+0000 7fa2fe800700 1 -- 172.21.15.62:0/3068646746 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa2f8102e70 msgr2=0x7fa2f8106e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:51:47.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.459+0000 7fa2fe800700 1 --2- 172.21.15.62:0/3068646746 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa2f8102e70 0x7fa2f8106e40 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fa2e4009910 tx=0x7fa2e4009c20 comp rx=0 tx=0).stop 2024-04-20T21:51:47.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.459+0000 7fa2fe800700 1 -- 172.21.15.62:0/3068646746 shutdown_connections 2024-04-20T21:51:47.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.459+0000 7fa2fe800700 1 --2- 172.21.15.62:0/3068646746 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa2f8102e70 0x7fa2f8106e40 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:51:47.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.459+0000 7fa2fe800700 1 --2- 172.21.15.62:0/3068646746 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2f81024d0 0x7fa2f81028a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:51:47.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.459+0000 7fa2fe800700 1 -- 172.21.15.62:0/3068646746 >> 172.21.15.62:0/3068646746 conn(0x7fa2f80fdd90 msgr2=0x7fa2f81001a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:51:47.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.459+0000 7fa2fe800700 1 -- 172.21.15.62:0/3068646746 shutdown_connections 2024-04-20T21:51:47.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.460+0000 7fa2fe800700 1 -- 172.21.15.62:0/3068646746 wait complete. 2024-04-20T21:51:47.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.460+0000 7fa2fe800700 1 Processor -- start 2024-04-20T21:51:47.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.461+0000 7fa2fe800700 1 -- start start 2024-04-20T21:51:47.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.461+0000 7fa2fe800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2f81024d0 0x7fa2f8197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:51:47.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.461+0000 7fa2fe800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa2f8102e70 0x7fa2f81984c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:51:47.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.461+0000 7fa2fe800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2f8198ba0 con 0x7fa2f8102e70 2024-04-20T21:51:47.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.461+0000 7fa2fe800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2f819c930 con 0x7fa2f81024d0 2024-04-20T21:51:47.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.461+0000 7fa2fca00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa2f8102e70 0x7fa2f81984c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:51:47.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.462+0000 7fa2fca00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa2f8102e70 0x7fa2f81984c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51032/0 (socket says 172.21.15.62:51032) 2024-04-20T21:51:47.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.462+0000 7fa2fca00700 1 -- 172.21.15.62:0/315264403 learned_addr learned my addr 172.21.15.62:0/315264403 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:51:47.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.462+0000 7fa2fd400700 1 --2- 172.21.15.62:0/315264403 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2f81024d0 0x7fa2f8197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:51:47.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.462+0000 7fa2fca00700 1 -- 172.21.15.62:0/315264403 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2f81024d0 msgr2=0x7fa2f8197f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:51:47.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.462+0000 7fa2fca00700 1 --2- 172.21.15.62:0/315264403 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2f81024d0 0x7fa2f8197f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:51:47.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.462+0000 7fa2fca00700 1 -- 172.21.15.62:0/315264403 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2e40095f0 con 0x7fa2f8102e70 2024-04-20T21:51:47.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.462+0000 7fa2fd400700 1 --2- 172.21.15.62:0/315264403 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2f81024d0 0x7fa2f8197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T21:51:47.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.462+0000 7fa2fca00700 1 --2- 172.21.15.62:0/315264403 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa2f8102e70 0x7fa2f81984c0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fa2e400bda0 tx=0x7fa2e400be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:51:47.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.463+0000 7fa2f6000700 1 -- 172.21.15.62:0/315264403 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa2e401d070 con 0x7fa2f8102e70 2024-04-20T21:51:47.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.463+0000 7fa2f6000700 1 -- 172.21.15.62:0/315264403 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa2e4004fb0 con 0x7fa2f8102e70 2024-04-20T21:51:47.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.463+0000 7fa2fe800700 1 -- 172.21.15.62:0/315264403 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2f819cbb0 con 0x7fa2f8102e70 2024-04-20T21:51:47.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.463+0000 7fa2f6000700 1 -- 172.21.15.62:0/315264403 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa2e400f480 con 0x7fa2f8102e70 2024-04-20T21:51:47.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.463+0000 7fa2fe800700 1 -- 172.21.15.62:0/315264403 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2f819d0a0 con 0x7fa2f8102e70 2024-04-20T21:51:47.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.464+0000 7fa2fe800700 1 -- 172.21.15.62:0/315264403 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2f810a790 con 0x7fa2f8102e70 2024-04-20T21:51:47.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.465+0000 7fa2f6000700 1 -- 172.21.15.62:0/315264403 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fa2e4022870 con 0x7fa2f8102e70 2024-04-20T21:51:47.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.466+0000 7fa2f6000700 1 --2- 172.21.15.62:0/315264403 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa2e006ecc0 0x7fa2e0071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:51:47.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.466+0000 7fa2f6000700 1 -- 172.21.15.62:0/315264403 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fa2e4090490 con 0x7fa2f8102e70 2024-04-20T21:51:47.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.466+0000 7fa2fd400700 1 --2- 172.21.15.62:0/315264403 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa2e006ecc0 0x7fa2e0071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:51:47.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.467+0000 7fa2fd400700 1 --2- 172.21.15.62:0/315264403 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa2e006ecc0 0x7fa2e0071170 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fa2ec007900 tx=0x7fa2ec008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:51:47.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.471+0000 7fa2f6000700 1 -- 172.21.15.62:0/315264403 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fa2e4059250 con 0x7fa2f8102e70 2024-04-20T21:51:47.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.791+0000 7fa2fe800700 1 -- 172.21.15.62:0/315264403 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fa2f8062380 con 0x7fa2f8102e70 2024-04-20T21:51:47.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.792+0000 7fa2f6000700 1 -- 172.21.15.62:0/315264403 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7fa2e405c910 con 0x7fa2f8102e70 2024-04-20T21:51:47.791 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:51:47.791 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T21:51:47.791 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:51:47.791 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:51:47.791 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T21:51:47.791 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:51:47.791 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:51:47.792 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T21:51:47.792 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 6 2024-04-20T21:51:47.792 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:51:47.792 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T21:51:47.792 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 2, 2024-04-20T21:51:47.792 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:51:47.792 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:51:47.792 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T21:51:47.792 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 2, 2024-04-20T21:51:47.792 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 12 2024-04-20T21:51:47.792 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T21:51:47.792 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:51:47.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.795+0000 7fa2fe800700 1 -- 172.21.15.62:0/315264403 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa2e006ecc0 msgr2=0x7fa2e0071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:51:47.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.795+0000 7fa2fe800700 1 --2- 172.21.15.62:0/315264403 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa2e006ecc0 0x7fa2e0071170 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fa2ec007900 tx=0x7fa2ec008040 comp rx=0 tx=0).stop 2024-04-20T21:51:47.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.795+0000 7fa2fe800700 1 -- 172.21.15.62:0/315264403 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa2f8102e70 msgr2=0x7fa2f81984c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:51:47.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.795+0000 7fa2fe800700 1 --2- 172.21.15.62:0/315264403 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa2f8102e70 0x7fa2f81984c0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fa2e400bda0 tx=0x7fa2e400be80 comp rx=0 tx=0).stop 2024-04-20T21:51:47.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.795+0000 7fa2fe800700 1 -- 172.21.15.62:0/315264403 shutdown_connections 2024-04-20T21:51:47.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.795+0000 7fa2fe800700 1 --2- 172.21.15.62:0/315264403 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa2e006ecc0 0x7fa2e0071170 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:51:47.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.795+0000 7fa2fe800700 1 --2- 172.21.15.62:0/315264403 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2f81024d0 0x7fa2f8197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:51:47.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.795+0000 7fa2fe800700 1 --2- 172.21.15.62:0/315264403 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa2f8102e70 0x7fa2f81984c0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:51:47.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.795+0000 7fa2fe800700 1 -- 172.21.15.62:0/315264403 >> 172.21.15.62:0/315264403 conn(0x7fa2f80fdd90 msgr2=0x7fa2f80ff4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:51:47.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.795+0000 7fa2fe800700 1 -- 172.21.15.62:0/315264403 shutdown_connections 2024-04-20T21:51:47.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.795+0000 7fa2fe800700 1 -- 172.21.15.62:0/315264403 wait complete. 2024-04-20T21:51:47.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.953+0000 7ff89de00700 1 -- 172.21.15.62:0/1317434258 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff89810b640 msgr2=0x7ff89810ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:51:47.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.953+0000 7ff89de00700 1 --2- 172.21.15.62:0/1317434258 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff89810b640 0x7ff89810ba10 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7ff888009910 tx=0x7ff888009c20 comp rx=0 tx=0).stop 2024-04-20T21:51:47.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.953+0000 7ff89de00700 1 -- 172.21.15.62:0/1317434258 shutdown_connections 2024-04-20T21:51:47.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.953+0000 7ff89de00700 1 --2- 172.21.15.62:0/1317434258 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff898106b80 0x7ff898106ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:51:47.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.953+0000 7ff89de00700 1 --2- 172.21.15.62:0/1317434258 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff89810b640 0x7ff89810ba10 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:51:47.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.953+0000 7ff89de00700 1 -- 172.21.15.62:0/1317434258 >> 172.21.15.62:0/1317434258 conn(0x7ff898075e90 msgr2=0x7ff8980782a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:51:47.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.953+0000 7ff89de00700 1 -- 172.21.15.62:0/1317434258 shutdown_connections 2024-04-20T21:51:47.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.954+0000 7ff89de00700 1 -- 172.21.15.62:0/1317434258 wait complete. 2024-04-20T21:51:47.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.954+0000 7ff89de00700 1 Processor -- start 2024-04-20T21:51:47.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.955+0000 7ff89de00700 1 -- start start 2024-04-20T21:51:47.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.955+0000 7ff89de00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff898106b80 0x7ff89819c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:51:47.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.955+0000 7ff89de00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff89810b640 0x7ff89819c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:51:47.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.955+0000 7ff89de00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff89819cec0 con 0x7ff898106b80 2024-04-20T21:51:47.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.955+0000 7ff89de00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8981a0c50 con 0x7ff89810b640 2024-04-20T21:51:47.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.955+0000 7ff89ca00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff898106b80 0x7ff89819c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:51:47.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.955+0000 7ff89ca00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff898106b80 0x7ff89819c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51054/0 (socket says 172.21.15.62:51054) 2024-04-20T21:51:47.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.955+0000 7ff89ca00700 1 -- 172.21.15.62:0/1546292524 learned_addr learned my addr 172.21.15.62:0/1546292524 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:51:47.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.956+0000 7ff89ca00700 1 -- 172.21.15.62:0/1546292524 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff89810b640 msgr2=0x7ff89819c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:51:47.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.956+0000 7ff89ca00700 1 --2- 172.21.15.62:0/1546292524 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff89810b640 0x7ff89819c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:51:47.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.956+0000 7ff89ca00700 1 -- 172.21.15.62:0/1546292524 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8880095f0 con 0x7ff898106b80 2024-04-20T21:51:47.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.956+0000 7ff89ca00700 1 --2- 172.21.15.62:0/1546292524 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff898106b80 0x7ff89819c2a0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7ff88800b5c0 tx=0x7ff88800be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:51:47.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.956+0000 7ff896000700 1 -- 172.21.15.62:0/1546292524 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff88801d070 con 0x7ff898106b80 2024-04-20T21:51:47.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.956+0000 7ff896000700 1 -- 172.21.15.62:0/1546292524 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff88800f460 con 0x7ff898106b80 2024-04-20T21:51:47.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.956+0000 7ff89de00700 1 -- 172.21.15.62:0/1546292524 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8981a0ed0 con 0x7ff898106b80 2024-04-20T21:51:47.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.957+0000 7ff896000700 1 -- 172.21.15.62:0/1546292524 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff888017610 con 0x7ff898106b80 2024-04-20T21:51:47.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.957+0000 7ff89de00700 1 -- 172.21.15.62:0/1546292524 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8981a1440 con 0x7ff898106b80 2024-04-20T21:51:47.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.958+0000 7ff89de00700 1 -- 172.21.15.62:0/1546292524 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff898062380 con 0x7ff898106b80 2024-04-20T21:51:47.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.959+0000 7ff896000700 1 -- 172.21.15.62:0/1546292524 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff88800f5d0 con 0x7ff898106b80 2024-04-20T21:51:47.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.960+0000 7ff896000700 1 --2- 172.21.15.62:0/1546292524 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff87c06ecc0 0x7ff87c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:51:47.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.960+0000 7ff896000700 1 -- 172.21.15.62:0/1546292524 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7ff888090510 con 0x7ff898106b80 2024-04-20T21:51:47.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.960+0000 7ff88fe00700 1 --2- 172.21.15.62:0/1546292524 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff87c06ecc0 0x7ff87c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:51:47.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.961+0000 7ff88fe00700 1 --2- 172.21.15.62:0/1546292524 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff87c06ecc0 0x7ff87c071170 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7ff89819d8c0 tx=0x7ff880006d20 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:51:47.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:47.964+0000 7ff896000700 1 -- 172.21.15.62:0/1546292524 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7ff888059220 con 0x7ff898106b80 2024-04-20T21:51:48.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:51:47 smithi062 ceph-mon[76322]: from='client.34928 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:51:48.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:51:47 smithi062 ceph-mon[76322]: pgmap v494: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:51:48.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:51:47 smithi062 ceph-mon[76322]: from='client.34932 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:51:48.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:51:47 smithi189 ceph-mon[57022]: from='client.34928 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:51:48.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:51:47 smithi189 ceph-mon[57022]: pgmap v494: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:51:48.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:51:47 smithi189 ceph-mon[57022]: from='client.34932 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:51:48.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:48.260+0000 7ff89de00700 1 -- 172.21.15.62:0/1546292524 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7ff89819d600 con 0x7ff898106b80 2024-04-20T21:51:48.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:48.261+0000 7ff896000700 1 -- 172.21.15.62:0/1546292524 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 25 v25) v1 ==== 76+0+1887 (secure 0 0 0) 0x7ff88805c8e0 con 0x7ff898106b80 2024-04-20T21:51:48.261 INFO:teuthology.orchestra.run.smithi062.stdout:e25 2024-04-20T21:51:48.261 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T21:51:48.261 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T21:51:48.261 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T21:51:48.261 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:51:48.261 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T21:51:48.261 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T21:51:48.261 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 20 2024-04-20T21:51:48.262 INFO:teuthology.orchestra.run.smithi062.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-20T21:51:48.262 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T21:51:48.262 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T21:42:01.559061+0000 2024-04-20T21:51:48.262 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T21:51:48.262 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T21:51:48.262 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T21:51:48.263 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T21:51:48.263 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T21:51:48.263 INFO:teuthology.orchestra.run.smithi062.stdout:max_xattr_size 65536 2024-04-20T21:51:48.263 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T21:51:48.263 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T21:51:48.263 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 81 2024-04-20T21:51:48.264 INFO:teuthology.orchestra.run.smithi062.stdout:snap_rstat 1 2024-04-20T21:51:48.264 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T21:51:48.264 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 2 2024-04-20T21:51:48.264 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1 2024-04-20T21:51:48.264 INFO:teuthology.orchestra.run.smithi062.stdout:up {} 2024-04-20T21:51:48.264 INFO:teuthology.orchestra.run.smithi062.stdout:failed 0,1 2024-04-20T21:51:48.264 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T21:51:48.264 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T21:51:48.264 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T21:51:48.265 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T21:51:48.265 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data enabled 2024-04-20T21:51:48.265 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T21:51:48.265 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T21:51:48.265 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 1 2024-04-20T21:51:48.265 INFO:teuthology.orchestra.run.smithi062.stdout:qdb_cluster leader: 0 members: 2024-04-20T21:51:48.265 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:51:48.266 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:51:48.266 INFO:teuthology.orchestra.run.smithi062.stdout:Standby daemons: 2024-04-20T21:51:48.266 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:51:48.266 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ryqblb{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6826/4113916205,v1:172.21.15.62:6827/4113916205] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T21:51:48.266 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ozizwi{-1:34476} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6828/3801975165,v1:172.21.15.62:6829/3801975165] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T21:51:48.266 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.xbgspw{-1:44285} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6826/3996040526,v1:172.21.15.189:6827/3996040526] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:51:48.266 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.dvimer{-1:44289} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6824/1743111843,v1:172.21.15.189:6825/1743111843] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:51:48.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:48.263+0000 7ff89de00700 1 -- 172.21.15.62:0/1546292524 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff87c06ecc0 msgr2=0x7ff87c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:51:48.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:48.264+0000 7ff89de00700 1 --2- 172.21.15.62:0/1546292524 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff87c06ecc0 0x7ff87c071170 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7ff89819d8c0 tx=0x7ff880006d20 comp rx=0 tx=0).stop 2024-04-20T21:51:48.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:48.264+0000 7ff89de00700 1 -- 172.21.15.62:0/1546292524 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff898106b80 msgr2=0x7ff89819c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:51:48.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:48.264+0000 7ff89de00700 1 --2- 172.21.15.62:0/1546292524 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff898106b80 0x7ff89819c2a0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7ff88800b5c0 tx=0x7ff88800be00 comp rx=0 tx=0).stop 2024-04-20T21:51:48.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:48.264+0000 7ff89de00700 1 -- 172.21.15.62:0/1546292524 shutdown_connections 2024-04-20T21:51:48.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:48.264+0000 7ff89de00700 1 --2- 172.21.15.62:0/1546292524 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff87c06ecc0 0x7ff87c071170 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:51:48.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:48.264+0000 7ff89de00700 1 --2- 172.21.15.62:0/1546292524 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff898106b80 0x7ff89819c2a0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:51:48.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:48.264+0000 7ff89de00700 1 --2- 172.21.15.62:0/1546292524 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff89810b640 0x7ff89819c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:51:48.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:48.264+0000 7ff89de00700 1 -- 172.21.15.62:0/1546292524 >> 172.21.15.62:0/1546292524 conn(0x7ff898075e90 msgr2=0x7ff8980767f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:51:48.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:48.264+0000 7ff89de00700 1 -- 172.21.15.62:0/1546292524 shutdown_connections 2024-04-20T21:51:48.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:48.264+0000 7ff89de00700 1 -- 172.21.15.62:0/1546292524 wait complete. 2024-04-20T21:51:48.270 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 25 2024-04-20T21:51:48.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:48.428+0000 7fed48a00700 1 -- 172.21.15.62:0/4009260873 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fed4410c660 msgr2=0x7fed4410ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:51:48.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:48.428+0000 7fed48a00700 1 --2- 172.21.15.62:0/4009260873 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fed4410c660 0x7fed4410ca30 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fed34009910 tx=0x7fed34009c20 comp rx=0 tx=0).stop 2024-04-20T21:51:48.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:48.428+0000 7fed48a00700 1 -- 172.21.15.62:0/4009260873 shutdown_connections 2024-04-20T21:51:48.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:48.428+0000 7fed48a00700 1 --2- 172.21.15.62:0/4009260873 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fed44106b80 0x7fed44106ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:51:48.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:48.428+0000 7fed48a00700 1 --2- 172.21.15.62:0/4009260873 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fed4410c660 0x7fed4410ca30 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:51:48.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:48.428+0000 7fed48a00700 1 -- 172.21.15.62:0/4009260873 >> 172.21.15.62:0/4009260873 conn(0x7fed44075e90 msgr2=0x7fed440782a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:51:48.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:48.428+0000 7fed48a00700 1 -- 172.21.15.62:0/4009260873 shutdown_connections 2024-04-20T21:51:48.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:48.429+0000 7fed48a00700 1 -- 172.21.15.62:0/4009260873 wait complete. 2024-04-20T21:51:48.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:48.430+0000 7fed48a00700 1 Processor -- start 2024-04-20T21:51:48.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:48.430+0000 7fed48a00700 1 -- start start 2024-04-20T21:51:48.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:48.430+0000 7fed48a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fed44106b80 0x7fed4419c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:51:48.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:48.430+0000 7fed48a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fed4410c660 0x7fed4419c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:51:48.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:48.430+0000 7fed48a00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed4419cea0 con 0x7fed44106b80 2024-04-20T21:51:48.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:48.430+0000 7fed48a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed441a0bc0 con 0x7fed4410c660 2024-04-20T21:51:48.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:48.431+0000 7fed42800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fed44106b80 0x7fed4419c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:51:48.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:48.431+0000 7fed42800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fed44106b80 0x7fed4419c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51072/0 (socket says 172.21.15.62:51072) 2024-04-20T21:51:48.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:48.431+0000 7fed42800700 1 -- 172.21.15.62:0/1325899436 learned_addr learned my addr 172.21.15.62:0/1325899436 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:51:48.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:48.431+0000 7fed41e00700 1 --2- 172.21.15.62:0/1325899436 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fed4410c660 0x7fed4419c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:51:48.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:48.431+0000 7fed42800700 1 -- 172.21.15.62:0/1325899436 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fed4410c660 msgr2=0x7fed4419c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:51:48.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:48.431+0000 7fed42800700 1 --2- 172.21.15.62:0/1325899436 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fed4410c660 0x7fed4419c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:51:48.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:48.431+0000 7fed42800700 1 -- 172.21.15.62:0/1325899436 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed340095f0 con 0x7fed44106b80 2024-04-20T21:51:48.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:48.432+0000 7fed42800700 1 --2- 172.21.15.62:0/1325899436 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fed44106b80 0x7fed4419c280 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fed34006010 tx=0x7fed3400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:51:48.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:48.432+0000 7fed3b400700 1 -- 172.21.15.62:0/1325899436 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fed3401d070 con 0x7fed44106b80 2024-04-20T21:51:48.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:48.432+0000 7fed3b400700 1 -- 172.21.15.62:0/1325899436 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fed34004fb0 con 0x7fed44106b80 2024-04-20T21:51:48.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:48.432+0000 7fed3b400700 1 -- 172.21.15.62:0/1325899436 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fed34022ea0 con 0x7fed44106b80 2024-04-20T21:51:48.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:48.432+0000 7fed48a00700 1 -- 172.21.15.62:0/1325899436 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed441a0e40 con 0x7fed44106b80 2024-04-20T21:51:48.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:48.432+0000 7fed48a00700 1 -- 172.21.15.62:0/1325899436 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed441a1330 con 0x7fed44106b80 2024-04-20T21:51:48.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:48.434+0000 7fed48a00700 1 -- 172.21.15.62:0/1325899436 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed4410eb60 con 0x7fed44106b80 2024-04-20T21:51:48.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:48.435+0000 7fed3b400700 1 -- 172.21.15.62:0/1325899436 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fed34022470 con 0x7fed44106b80 2024-04-20T21:51:48.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:48.436+0000 7fed3b400700 1 --2- 172.21.15.62:0/1325899436 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fed2806ecc0 0x7fed28071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:51:48.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:48.436+0000 7fed3b400700 1 -- 172.21.15.62:0/1325899436 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fed340903f0 con 0x7fed44106b80 2024-04-20T21:51:48.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:48.437+0000 7fed41e00700 1 --2- 172.21.15.62:0/1325899436 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fed2806ecc0 0x7fed28071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:51:48.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:48.437+0000 7fed41e00700 1 --2- 172.21.15.62:0/1325899436 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fed2806ecc0 0x7fed28071170 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fed30005fd0 tx=0x7fed3000a560 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:51:48.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:48.441+0000 7fed3b400700 1 -- 172.21.15.62:0/1325899436 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fed34059230 con 0x7fed44106b80 2024-04-20T21:51:48.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:48.703+0000 7fed48a00700 1 -- 172.21.15.62:0/1325899436 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fed441a1660 con 0x7fed2806ecc0 2024-04-20T21:51:48.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:48.705+0000 7fed3b400700 1 -- 172.21.15.62:0/1325899436 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fed441a1660 con 0x7fed2806ecc0 2024-04-20T21:51:48.705 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:51:48.705 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T21:51:48.705 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T21:51:48.705 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-20T21:51:48.705 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T21:51:48.705 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr", 2024-04-20T21:51:48.705 INFO:teuthology.orchestra.run.smithi062.stdout: "crash", 2024-04-20T21:51:48.705 INFO:teuthology.orchestra.run.smithi062.stdout: "mon", 2024-04-20T21:51:48.705 INFO:teuthology.orchestra.run.smithi062.stdout: "osd" 2024-04-20T21:51:48.705 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T21:51:48.705 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "14/23 daemons upgraded", 2024-04-20T21:51:48.705 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading mds daemons", 2024-04-20T21:51:48.705 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T21:51:48.705 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:51:48.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:48.708+0000 7fed48a00700 1 -- 172.21.15.62:0/1325899436 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fed2806ecc0 msgr2=0x7fed28071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:51:48.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:48.708+0000 7fed48a00700 1 --2- 172.21.15.62:0/1325899436 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fed2806ecc0 0x7fed28071170 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fed30005fd0 tx=0x7fed3000a560 comp rx=0 tx=0).stop 2024-04-20T21:51:48.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:48.708+0000 7fed48a00700 1 -- 172.21.15.62:0/1325899436 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fed44106b80 msgr2=0x7fed4419c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:51:48.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:48.708+0000 7fed48a00700 1 --2- 172.21.15.62:0/1325899436 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fed44106b80 0x7fed4419c280 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fed34006010 tx=0x7fed3400be20 comp rx=0 tx=0).stop 2024-04-20T21:51:48.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:48.708+0000 7fed48a00700 1 -- 172.21.15.62:0/1325899436 shutdown_connections 2024-04-20T21:51:48.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:48.708+0000 7fed48a00700 1 --2- 172.21.15.62:0/1325899436 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fed2806ecc0 0x7fed28071170 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:51:48.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:48.708+0000 7fed48a00700 1 --2- 172.21.15.62:0/1325899436 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fed44106b80 0x7fed4419c280 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:51:48.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:48.708+0000 7fed48a00700 1 --2- 172.21.15.62:0/1325899436 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fed4410c660 0x7fed4419c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:51:48.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:48.708+0000 7fed48a00700 1 -- 172.21.15.62:0/1325899436 >> 172.21.15.62:0/1325899436 conn(0x7fed44075e90 msgr2=0x7fed44077980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:51:48.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:48.708+0000 7fed48a00700 1 -- 172.21.15.62:0/1325899436 shutdown_connections 2024-04-20T21:51:48.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:48.709+0000 7fed48a00700 1 -- 172.21.15.62:0/1325899436 wait complete. 2024-04-20T21:51:48.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:48.873+0000 7f714ca00700 1 -- 172.21.15.62:0/2363381530 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7148101c50 msgr2=0x7f71481020c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:51:48.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:48.873+0000 7f714ca00700 1 --2- 172.21.15.62:0/2363381530 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7148101c50 0x7f71481020c0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f7134009910 tx=0x7f7134009c20 comp rx=0 tx=0).stop 2024-04-20T21:51:48.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:48.874+0000 7f714ca00700 1 -- 172.21.15.62:0/2363381530 shutdown_connections 2024-04-20T21:51:48.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:48.874+0000 7f714ca00700 1 --2- 172.21.15.62:0/2363381530 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7148101c50 0x7f71481020c0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:51:48.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:48.874+0000 7f714ca00700 1 --2- 172.21.15.62:0/2363381530 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f71481003f0 0x7f71481007c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:51:48.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:48.874+0000 7f714ca00700 1 -- 172.21.15.62:0/2363381530 >> 172.21.15.62:0/2363381530 conn(0x7f71480fbbf0 msgr2=0x7f71480fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:51:48.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:48.874+0000 7f714ca00700 1 -- 172.21.15.62:0/2363381530 shutdown_connections 2024-04-20T21:51:48.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:48.874+0000 7f714ca00700 1 -- 172.21.15.62:0/2363381530 wait complete. 2024-04-20T21:51:48.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:48.875+0000 7f714ca00700 1 Processor -- start 2024-04-20T21:51:48.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:48.875+0000 7f714ca00700 1 -- start start 2024-04-20T21:51:48.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:48.875+0000 7f714ca00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f71481003f0 0x7f7148197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:51:48.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:48.875+0000 7f714ca00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7148101c50 0x7f7148198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:51:48.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:48.875+0000 7f714ca00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7148198a60 con 0x7f7148101c50 2024-04-20T21:51:48.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:48.875+0000 7f714ca00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7148198ba0 con 0x7f71481003f0 2024-04-20T21:51:48.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:48.876+0000 7f7145e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7148101c50 0x7f7148198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:51:48.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:48.876+0000 7f7145e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7148101c50 0x7f7148198410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51082/0 (socket says 172.21.15.62:51082) 2024-04-20T21:51:48.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:48.876+0000 7f7145e00700 1 -- 172.21.15.62:0/2482927754 learned_addr learned my addr 172.21.15.62:0/2482927754 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:51:48.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:48.876+0000 7f7146800700 1 --2- 172.21.15.62:0/2482927754 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f71481003f0 0x7f7148197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:51:48.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:48.876+0000 7f7145e00700 1 -- 172.21.15.62:0/2482927754 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f71481003f0 msgr2=0x7f7148197ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:51:48.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:48.876+0000 7f7145e00700 1 --2- 172.21.15.62:0/2482927754 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f71481003f0 0x7f7148197ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:51:48.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:48.876+0000 7f7145e00700 1 -- 172.21.15.62:0/2482927754 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71340095f0 con 0x7f7148101c50 2024-04-20T21:51:48.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:48.877+0000 7f7145e00700 1 --2- 172.21.15.62:0/2482927754 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7148101c50 0x7f7148198410 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f713400b5c0 tx=0x7f713400be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:51:48.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:48.877+0000 7f713f400700 1 -- 172.21.15.62:0/2482927754 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f713401d070 con 0x7f7148101c50 2024-04-20T21:51:48.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:48.877+0000 7f713f400700 1 -- 172.21.15.62:0/2482927754 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f713400f460 con 0x7f7148101c50 2024-04-20T21:51:48.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:48.877+0000 7f713f400700 1 -- 172.21.15.62:0/2482927754 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7134017610 con 0x7f7148101c50 2024-04-20T21:51:48.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:48.877+0000 7f714ca00700 1 -- 172.21.15.62:0/2482927754 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f714819c990 con 0x7f7148101c50 2024-04-20T21:51:48.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:48.877+0000 7f714ca00700 1 -- 172.21.15.62:0/2482927754 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f714819cf00 con 0x7f7148101c50 2024-04-20T21:51:48.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:48.878+0000 7f714ca00700 1 -- 172.21.15.62:0/2482927754 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7148062380 con 0x7f7148101c50 2024-04-20T21:51:48.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:48.880+0000 7f713f400700 1 -- 172.21.15.62:0/2482927754 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f713400f5d0 con 0x7f7148101c50 2024-04-20T21:51:48.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:48.880+0000 7f713f400700 1 --2- 172.21.15.62:0/2482927754 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f712c06ecc0 0x7f712c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:51:48.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:48.881+0000 7f713f400700 1 -- 172.21.15.62:0/2482927754 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f7134090520 con 0x7f7148101c50 2024-04-20T21:51:48.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:48.881+0000 7f7146800700 1 --2- 172.21.15.62:0/2482927754 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f712c06ecc0 0x7f712c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:51:48.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:48.882+0000 7f7146800700 1 --2- 172.21.15.62:0/2482927754 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f712c06ecc0 0x7f712c071170 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f713800ba60 tx=0x7f7138005d50 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:51:48.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:48.886+0000 7f713f400700 1 -- 172.21.15.62:0/2482927754 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f7134059230 con 0x7f7148101c50 2024-04-20T21:51:49.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:51:48 smithi062 ceph-mon[76322]: from='client.34936 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:51:49.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:51:48 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/315264403' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:51:49.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:51:48 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/1546292524' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T21:51:49.151 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:51:48 smithi189 ceph-mon[57022]: from='client.34936 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:51:49.151 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:51:48 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/315264403' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:51:49.151 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:51:48 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/1546292524' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T21:51:49.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:49.217+0000 7f714ca00700 1 -- 172.21.15.62:0/2482927754 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f7148199230 con 0x7f7148101c50 2024-04-20T21:51:49.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:49.217+0000 7f713f400700 1 -- 172.21.15.62:0/2482927754 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7f713405c8f0 con 0x7f7148101c50 2024-04-20T21:51:49.217 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-20T21:51:49.217 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-20T21:51:49.217 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is degraded 2024-04-20T21:51:49.217 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T21:51:49.217 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T21:51:49.217 INFO:teuthology.orchestra.run.smithi062.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-20T21:51:49.217 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-20T21:51:49.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:49.221+0000 7f714ca00700 1 -- 172.21.15.62:0/2482927754 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f712c06ecc0 msgr2=0x7f712c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:51:49.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:49.221+0000 7f714ca00700 1 --2- 172.21.15.62:0/2482927754 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f712c06ecc0 0x7f712c071170 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f713800ba60 tx=0x7f7138005d50 comp rx=0 tx=0).stop 2024-04-20T21:51:49.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:49.221+0000 7f714ca00700 1 -- 172.21.15.62:0/2482927754 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7148101c50 msgr2=0x7f7148198410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:51:49.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:49.221+0000 7f714ca00700 1 --2- 172.21.15.62:0/2482927754 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7148101c50 0x7f7148198410 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f713400b5c0 tx=0x7f713400be00 comp rx=0 tx=0).stop 2024-04-20T21:51:49.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:49.221+0000 7f714ca00700 1 -- 172.21.15.62:0/2482927754 shutdown_connections 2024-04-20T21:51:49.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:49.221+0000 7f714ca00700 1 --2- 172.21.15.62:0/2482927754 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f712c06ecc0 0x7f712c071170 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:51:49.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:49.221+0000 7f714ca00700 1 --2- 172.21.15.62:0/2482927754 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f71481003f0 0x7f7148197ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:51:49.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:49.221+0000 7f714ca00700 1 --2- 172.21.15.62:0/2482927754 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7148101c50 0x7f7148198410 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:51:49.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:49.221+0000 7f714ca00700 1 -- 172.21.15.62:0/2482927754 >> 172.21.15.62:0/2482927754 conn(0x7f71480fbbf0 msgr2=0x7f71480fc370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:51:49.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:49.221+0000 7f714ca00700 1 -- 172.21.15.62:0/2482927754 shutdown_connections 2024-04-20T21:51:49.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:51:49.221+0000 7f714ca00700 1 -- 172.21.15.62:0/2482927754 wait complete. 2024-04-20T21:51:50.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:51:49 smithi062 ceph-mon[76322]: pgmap v495: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:51:50.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:51:49 smithi062 ceph-mon[76322]: from='client.34948 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:51:50.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:51:49 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/2482927754' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T21:51:50.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:51:49 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:51:50.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:51:49 smithi189 ceph-mon[57022]: pgmap v495: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:51:50.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:51:49 smithi189 ceph-mon[57022]: from='client.34948 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:51:50.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:51:49 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/2482927754' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T21:51:50.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:51:49 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:51:52.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:51:51 smithi062 ceph-mon[76322]: pgmap v496: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:51:52.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:51:51 smithi189 ceph-mon[57022]: pgmap v496: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:51:54.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:51:53 smithi062 ceph-mon[76322]: pgmap v497: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:51:54.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:51:53 smithi189 ceph-mon[57022]: pgmap v497: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:51:56.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:51:55 smithi062 ceph-mon[76322]: pgmap v498: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:51:56.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:51:55 smithi189 ceph-mon[57022]: pgmap v498: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:51:58.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:51:57 smithi062 ceph-mon[76322]: pgmap v499: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:51:58.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:51:57 smithi189 ceph-mon[57022]: pgmap v499: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:52:00.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:51:59 smithi062 ceph-mon[76322]: pgmap v500: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:52:00.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:51:59 smithi189 ceph-mon[57022]: pgmap v500: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:52:02.041 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:52:01 smithi189 ceph-mon[57022]: pgmap v501: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:52:02.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:52:01 smithi062 ceph-mon[76322]: pgmap v501: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:52:04.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:52:03 smithi189 ceph-mon[57022]: pgmap v502: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:52:04.284 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:52:03 smithi062 ceph-mon[76322]: pgmap v502: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:52:04.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:52:04 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:52:05.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:52:04 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:52:06.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:52:05 smithi189 ceph-mon[57022]: pgmap v503: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:52:06.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:52:05 smithi062 ceph-mon[76322]: pgmap v503: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:52:08.143 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:52:07 smithi062 ceph-mon[76322]: pgmap v504: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:52:08.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:52:07 smithi189 ceph-mon[57022]: pgmap v504: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:52:10.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:52:09 smithi189 ceph-mon[57022]: pgmap v505: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:52:10.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:52:09 smithi062 ceph-mon[76322]: pgmap v505: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:52:12.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:52:11 smithi189 ceph-mon[57022]: pgmap v506: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:52:12.249 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:52:11 smithi062 ceph-mon[76322]: pgmap v506: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:52:14.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:52:13 smithi189 ceph-mon[57022]: pgmap v507: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:52:14.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:52:13 smithi062 ceph-mon[76322]: pgmap v507: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:52:16.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:52:15 smithi189 ceph-mon[57022]: pgmap v508: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:52:16.284 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:52:15 smithi062 ceph-mon[76322]: pgmap v508: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:52:18.143 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:52:17 smithi062 ceph-mon[76322]: pgmap v509: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:52:18.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:52:17 smithi189 ceph-mon[57022]: pgmap v509: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:52:19.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:19.399+0000 7fee1e400700 1 -- 172.21.15.62:0/1713747709 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee18073090 msgr2=0x7fee18073460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:52:19.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:19.399+0000 7fee1e400700 1 --2- 172.21.15.62:0/1713747709 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee18073090 0x7fee18073460 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fee08009870 tx=0x7fee08009b80 comp rx=0 tx=0).stop 2024-04-20T21:52:19.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:19.399+0000 7fee1e400700 1 -- 172.21.15.62:0/1713747709 shutdown_connections 2024-04-20T21:52:19.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:19.399+0000 7fee1e400700 1 --2- 172.21.15.62:0/1713747709 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee180739a0 0x7fee1810c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:52:19.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:19.399+0000 7fee1e400700 1 --2- 172.21.15.62:0/1713747709 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee18073090 0x7fee18073460 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:52:19.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:19.399+0000 7fee1e400700 1 -- 172.21.15.62:0/1713747709 >> 172.21.15.62:0/1713747709 conn(0x7fee180fbda0 msgr2=0x7fee180fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:52:19.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:19.399+0000 7fee1e400700 1 -- 172.21.15.62:0/1713747709 shutdown_connections 2024-04-20T21:52:19.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:19.399+0000 7fee1e400700 1 -- 172.21.15.62:0/1713747709 wait complete. 2024-04-20T21:52:19.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:19.400+0000 7fee1e400700 1 Processor -- start 2024-04-20T21:52:19.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:19.401+0000 7fee1e400700 1 -- start start 2024-04-20T21:52:19.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:19.401+0000 7fee1e400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee18073090 0x7fee18198190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:52:19.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:19.401+0000 7fee1e400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee180739a0 0x7fee181986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:52:19.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:19.401+0000 7fee1e400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee18198db0 con 0x7fee18073090 2024-04-20T21:52:19.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:19.401+0000 7fee1e400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee1819cb40 con 0x7fee180739a0 2024-04-20T21:52:19.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:19.401+0000 7fee17400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee18073090 0x7fee18198190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:52:19.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:19.402+0000 7fee17400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee18073090 0x7fee18198190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59180/0 (socket says 172.21.15.62:59180) 2024-04-20T21:52:19.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:19.402+0000 7fee17400700 1 -- 172.21.15.62:0/2714187034 learned_addr learned my addr 172.21.15.62:0/2714187034 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:52:19.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:19.402+0000 7fee16a00700 1 --2- 172.21.15.62:0/2714187034 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee180739a0 0x7fee181986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:52:19.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:19.402+0000 7fee17400700 1 -- 172.21.15.62:0/2714187034 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee180739a0 msgr2=0x7fee181986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:52:19.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:19.402+0000 7fee17400700 1 --2- 172.21.15.62:0/2714187034 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee180739a0 0x7fee181986d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:52:19.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:19.402+0000 7fee17400700 1 -- 172.21.15.62:0/2714187034 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee08009520 con 0x7fee18073090 2024-04-20T21:52:19.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:19.402+0000 7fee17400700 1 --2- 172.21.15.62:0/2714187034 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee18073090 0x7fee18198190 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fee08009230 tx=0x7fee0800bfc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:52:19.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:19.403+0000 7fee0fe00700 1 -- 172.21.15.62:0/2714187034 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fee0801c070 con 0x7fee18073090 2024-04-20T21:52:19.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:19.403+0000 7fee1e400700 1 -- 172.21.15.62:0/2714187034 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee1819cdc0 con 0x7fee18073090 2024-04-20T21:52:19.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:19.403+0000 7fee0fe00700 1 -- 172.21.15.62:0/2714187034 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fee080046c0 con 0x7fee18073090 2024-04-20T21:52:19.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:19.403+0000 7fee0fe00700 1 -- 172.21.15.62:0/2714187034 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fee0800fdb0 con 0x7fee18073090 2024-04-20T21:52:19.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:19.403+0000 7fee1e400700 1 -- 172.21.15.62:0/2714187034 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee1819d2b0 con 0x7fee18073090 2024-04-20T21:52:19.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:19.404+0000 7fee1e400700 1 -- 172.21.15.62:0/2714187034 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee18109ce0 con 0x7fee18073090 2024-04-20T21:52:19.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:19.406+0000 7fee0fe00700 1 -- 172.21.15.62:0/2714187034 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fee08020070 con 0x7fee18073090 2024-04-20T21:52:19.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:19.406+0000 7fee0fe00700 1 --2- 172.21.15.62:0/2714187034 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fedfc06ecc0 0x7fedfc071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:52:19.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:19.407+0000 7fee0fe00700 1 -- 172.21.15.62:0/2714187034 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fee0808ffe0 con 0x7fee18073090 2024-04-20T21:52:19.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:19.407+0000 7fee16a00700 1 --2- 172.21.15.62:0/2714187034 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fedfc06ecc0 0x7fedfc071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:52:19.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:19.408+0000 7fee16a00700 1 --2- 172.21.15.62:0/2714187034 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fedfc06ecc0 0x7fedfc071170 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fee181997b0 tx=0x7fee04009310 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:52:19.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:19.412+0000 7fee0fe00700 1 -- 172.21.15.62:0/2714187034 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fee080c4ff0 con 0x7fee18073090 2024-04-20T21:52:19.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:19.674+0000 7fee1e400700 1 -- 172.21.15.62:0/2714187034 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fee1819d520 con 0x7fedfc06ecc0 2024-04-20T21:52:19.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:19.676+0000 7fee0fe00700 1 -- 172.21.15.62:0/2714187034 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fee1819d520 con 0x7fedfc06ecc0 2024-04-20T21:52:19.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:19.679+0000 7fee1e400700 1 -- 172.21.15.62:0/2714187034 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fedfc06ecc0 msgr2=0x7fedfc071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:52:19.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:19.679+0000 7fee1e400700 1 --2- 172.21.15.62:0/2714187034 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fedfc06ecc0 0x7fedfc071170 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fee181997b0 tx=0x7fee04009310 comp rx=0 tx=0).stop 2024-04-20T21:52:19.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:19.679+0000 7fee1e400700 1 -- 172.21.15.62:0/2714187034 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee18073090 msgr2=0x7fee18198190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:52:19.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:19.679+0000 7fee1e400700 1 --2- 172.21.15.62:0/2714187034 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee18073090 0x7fee18198190 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fee08009230 tx=0x7fee0800bfc0 comp rx=0 tx=0).stop 2024-04-20T21:52:19.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:19.679+0000 7fee1e400700 1 -- 172.21.15.62:0/2714187034 shutdown_connections 2024-04-20T21:52:19.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:19.679+0000 7fee1e400700 1 --2- 172.21.15.62:0/2714187034 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fedfc06ecc0 0x7fedfc071170 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:52:19.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:19.679+0000 7fee1e400700 1 --2- 172.21.15.62:0/2714187034 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee18073090 0x7fee18198190 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:52:19.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:19.679+0000 7fee1e400700 1 --2- 172.21.15.62:0/2714187034 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee180739a0 0x7fee181986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:52:19.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:19.679+0000 7fee1e400700 1 -- 172.21.15.62:0/2714187034 >> 172.21.15.62:0/2714187034 conn(0x7fee180fbda0 msgr2=0x7fee18106e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:52:19.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:19.679+0000 7fee1e400700 1 -- 172.21.15.62:0/2714187034 shutdown_connections 2024-04-20T21:52:19.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:19.679+0000 7fee1e400700 1 -- 172.21.15.62:0/2714187034 wait complete. 2024-04-20T21:52:19.695 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T21:52:19.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:19.840+0000 7efdfb600700 1 -- 172.21.15.62:0/743377573 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efdf41004e0 msgr2=0x7efdf41008b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:52:19.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:19.840+0000 7efdfb600700 1 --2- 172.21.15.62:0/743377573 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efdf41004e0 0x7efdf41008b0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7efde8009910 tx=0x7efde8009c20 comp rx=0 tx=0).stop 2024-04-20T21:52:19.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:19.841+0000 7efdfb600700 1 -- 172.21.15.62:0/743377573 shutdown_connections 2024-04-20T21:52:19.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:19.841+0000 7efdfb600700 1 --2- 172.21.15.62:0/743377573 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efdf4100e80 0x7efdf4104e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:52:19.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:19.841+0000 7efdfb600700 1 --2- 172.21.15.62:0/743377573 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efdf41004e0 0x7efdf41008b0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:52:19.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:19.841+0000 7efdfb600700 1 -- 172.21.15.62:0/743377573 >> 172.21.15.62:0/743377573 conn(0x7efdf40fbda0 msgr2=0x7efdf40fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:52:19.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:19.841+0000 7efdfb600700 1 -- 172.21.15.62:0/743377573 shutdown_connections 2024-04-20T21:52:19.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:19.841+0000 7efdfb600700 1 -- 172.21.15.62:0/743377573 wait complete. 2024-04-20T21:52:19.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:19.842+0000 7efdfb600700 1 Processor -- start 2024-04-20T21:52:19.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:19.842+0000 7efdfb600700 1 -- start start 2024-04-20T21:52:19.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:19.843+0000 7efdfb600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efdf41004e0 0x7efdf4073050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:52:19.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:19.843+0000 7efdfb600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efdf4100e80 0x7efdf4073590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:52:19.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:19.843+0000 7efdfb600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdf4073ad0 con 0x7efdf41004e0 2024-04-20T21:52:19.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:19.843+0000 7efdfb600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdf4073c10 con 0x7efdf4100e80 2024-04-20T21:52:19.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:19.843+0000 7efdf8c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efdf41004e0 0x7efdf4073050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:52:19.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:19.843+0000 7efdf8c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efdf41004e0 0x7efdf4073050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59198/0 (socket says 172.21.15.62:59198) 2024-04-20T21:52:19.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:19.843+0000 7efdf8c00700 1 -- 172.21.15.62:0/2202448278 learned_addr learned my addr 172.21.15.62:0/2202448278 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:52:19.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:19.843+0000 7efdf3e00700 1 --2- 172.21.15.62:0/2202448278 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efdf4100e80 0x7efdf4073590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:52:19.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:19.843+0000 7efdf8c00700 1 -- 172.21.15.62:0/2202448278 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efdf4100e80 msgr2=0x7efdf4073590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:52:19.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:19.843+0000 7efdf8c00700 1 --2- 172.21.15.62:0/2202448278 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efdf4100e80 0x7efdf4073590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:52:19.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:19.843+0000 7efdf8c00700 1 -- 172.21.15.62:0/2202448278 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efde80095f0 con 0x7efdf41004e0 2024-04-20T21:52:19.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:19.844+0000 7efdf8c00700 1 --2- 172.21.15.62:0/2202448278 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efdf41004e0 0x7efdf4073050 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7efde800b5c0 tx=0x7efde800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:52:19.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:19.844+0000 7efdf1600700 1 -- 172.21.15.62:0/2202448278 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efde801d070 con 0x7efdf41004e0 2024-04-20T21:52:19.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:19.844+0000 7efdfb600700 1 -- 172.21.15.62:0/2202448278 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdf4073e90 con 0x7efdf41004e0 2024-04-20T21:52:19.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:19.844+0000 7efdf1600700 1 -- 172.21.15.62:0/2202448278 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efde8005e00 con 0x7efdf41004e0 2024-04-20T21:52:19.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:19.844+0000 7efdf1600700 1 -- 172.21.15.62:0/2202448278 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efde8017400 con 0x7efdf41004e0 2024-04-20T21:52:19.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:19.845+0000 7efdfb600700 1 -- 172.21.15.62:0/2202448278 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdf41a23f0 con 0x7efdf41004e0 2024-04-20T21:52:19.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:19.846+0000 7efdfb600700 1 -- 172.21.15.62:0/2202448278 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdf41087f0 con 0x7efdf41004e0 2024-04-20T21:52:19.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:19.847+0000 7efdf1600700 1 -- 172.21.15.62:0/2202448278 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7efde800f460 con 0x7efdf41004e0 2024-04-20T21:52:19.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:19.848+0000 7efdf1600700 1 --2- 172.21.15.62:0/2202448278 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7efddc06ecc0 0x7efddc071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:52:19.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:19.848+0000 7efdf1600700 1 -- 172.21.15.62:0/2202448278 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7efde8090490 con 0x7efdf41004e0 2024-04-20T21:52:19.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:19.848+0000 7efdf3e00700 1 --2- 172.21.15.62:0/2202448278 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7efddc06ecc0 0x7efddc071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:52:19.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:19.849+0000 7efdf3e00700 1 --2- 172.21.15.62:0/2202448278 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7efddc06ecc0 0x7efddc071170 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7efde0005950 tx=0x7efde00058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:52:19.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:19.853+0000 7efdf1600700 1 -- 172.21.15.62:0/2202448278 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7efde80591a0 con 0x7efdf41004e0 2024-04-20T21:52:20.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:52:19 smithi062 ceph-mon[76322]: pgmap v510: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:52:20.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:52:19 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:52:20.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:20.106+0000 7efdfb600700 1 -- 172.21.15.62:0/2202448278 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7efdf4061190 con 0x7efddc06ecc0 2024-04-20T21:52:20.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:20.108+0000 7efdf1600700 1 -- 172.21.15.62:0/2202448278 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7efdf4061190 con 0x7efddc06ecc0 2024-04-20T21:52:20.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:20.111+0000 7efdfb600700 1 -- 172.21.15.62:0/2202448278 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7efddc06ecc0 msgr2=0x7efddc071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:52:20.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:20.111+0000 7efdfb600700 1 --2- 172.21.15.62:0/2202448278 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7efddc06ecc0 0x7efddc071170 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7efde0005950 tx=0x7efde00058e0 comp rx=0 tx=0).stop 2024-04-20T21:52:20.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:20.111+0000 7efdfb600700 1 -- 172.21.15.62:0/2202448278 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efdf41004e0 msgr2=0x7efdf4073050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:52:20.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:20.111+0000 7efdfb600700 1 --2- 172.21.15.62:0/2202448278 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efdf41004e0 0x7efdf4073050 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7efde800b5c0 tx=0x7efde800be20 comp rx=0 tx=0).stop 2024-04-20T21:52:20.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:20.111+0000 7efdfb600700 1 -- 172.21.15.62:0/2202448278 shutdown_connections 2024-04-20T21:52:20.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:20.111+0000 7efdfb600700 1 --2- 172.21.15.62:0/2202448278 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7efddc06ecc0 0x7efddc071170 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:52:20.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:20.111+0000 7efdfb600700 1 --2- 172.21.15.62:0/2202448278 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efdf41004e0 0x7efdf4073050 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:52:20.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:20.111+0000 7efdfb600700 1 --2- 172.21.15.62:0/2202448278 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efdf4100e80 0x7efdf4073590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:52:20.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:20.111+0000 7efdfb600700 1 -- 172.21.15.62:0/2202448278 >> 172.21.15.62:0/2202448278 conn(0x7efdf40fbda0 msgr2=0x7efdf40fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:52:20.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:20.111+0000 7efdfb600700 1 -- 172.21.15.62:0/2202448278 shutdown_connections 2024-04-20T21:52:20.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:20.111+0000 7efdfb600700 1 -- 172.21.15.62:0/2202448278 wait complete. 2024-04-20T21:52:20.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:52:19 smithi189 ceph-mon[57022]: pgmap v510: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:52:20.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:52:19 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:52:20.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:20.275+0000 7ffa32e00700 1 -- 172.21.15.62:0/3955465079 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa2c069140 msgr2=0x7ffa2c104eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:52:20.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:20.275+0000 7ffa32e00700 1 --2- 172.21.15.62:0/3955465079 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa2c069140 0x7ffa2c104eb0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7ffa18009910 tx=0x7ffa18009c20 comp rx=0 tx=0).stop 2024-04-20T21:52:20.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:20.275+0000 7ffa32e00700 1 -- 172.21.15.62:0/3955465079 shutdown_connections 2024-04-20T21:52:20.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:20.275+0000 7ffa32e00700 1 --2- 172.21.15.62:0/3955465079 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa2c069140 0x7ffa2c104eb0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:52:20.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:20.275+0000 7ffa32e00700 1 --2- 172.21.15.62:0/3955465079 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffa2c0687a0 0x7ffa2c068b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:52:20.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:20.275+0000 7ffa32e00700 1 -- 172.21.15.62:0/3955465079 >> 172.21.15.62:0/3955465079 conn(0x7ffa2c075130 msgr2=0x7ffa2c075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:52:20.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:20.276+0000 7ffa32e00700 1 -- 172.21.15.62:0/3955465079 shutdown_connections 2024-04-20T21:52:20.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:20.276+0000 7ffa32e00700 1 -- 172.21.15.62:0/3955465079 wait complete. 2024-04-20T21:52:20.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:20.277+0000 7ffa32e00700 1 Processor -- start 2024-04-20T21:52:20.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:20.277+0000 7ffa32e00700 1 -- start start 2024-04-20T21:52:20.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:20.277+0000 7ffa32e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffa2c0687a0 0x7ffa2c198140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:52:20.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:20.277+0000 7ffa32e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa2c069140 0x7ffa2c198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:52:20.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:20.277+0000 7ffa32e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa2c198d60 con 0x7ffa2c069140 2024-04-20T21:52:20.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:20.277+0000 7ffa32e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa2c19caf0 con 0x7ffa2c0687a0 2024-04-20T21:52:20.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:20.278+0000 7ffa2b400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa2c069140 0x7ffa2c198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:52:20.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:20.278+0000 7ffa2b400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa2c069140 0x7ffa2c198680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59224/0 (socket says 172.21.15.62:59224) 2024-04-20T21:52:20.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:20.278+0000 7ffa2b400700 1 -- 172.21.15.62:0/246783944 learned_addr learned my addr 172.21.15.62:0/246783944 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:52:20.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:20.278+0000 7ffa2be00700 1 --2- 172.21.15.62:0/246783944 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffa2c0687a0 0x7ffa2c198140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:52:20.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:20.278+0000 7ffa2b400700 1 -- 172.21.15.62:0/246783944 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffa2c0687a0 msgr2=0x7ffa2c198140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:52:20.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:20.278+0000 7ffa2b400700 1 --2- 172.21.15.62:0/246783944 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffa2c0687a0 0x7ffa2c198140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:52:20.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:20.278+0000 7ffa2b400700 1 -- 172.21.15.62:0/246783944 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa180095f0 con 0x7ffa2c069140 2024-04-20T21:52:20.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:20.279+0000 7ffa2b400700 1 --2- 172.21.15.62:0/246783944 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa2c069140 0x7ffa2c198680 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7ffa18005f50 tx=0x7ffa1800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:52:20.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:20.279+0000 7ffa28c00700 1 -- 172.21.15.62:0/246783944 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffa1801d070 con 0x7ffa2c069140 2024-04-20T21:52:20.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:20.279+0000 7ffa28c00700 1 -- 172.21.15.62:0/246783944 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ffa1800f460 con 0x7ffa2c069140 2024-04-20T21:52:20.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:20.279+0000 7ffa32e00700 1 -- 172.21.15.62:0/246783944 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa2c19cd70 con 0x7ffa2c069140 2024-04-20T21:52:20.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:20.279+0000 7ffa28c00700 1 -- 172.21.15.62:0/246783944 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffa18017630 con 0x7ffa2c069140 2024-04-20T21:52:20.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:20.279+0000 7ffa32e00700 1 -- 172.21.15.62:0/246783944 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa2c19d260 con 0x7ffa2c069140 2024-04-20T21:52:20.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:20.280+0000 7ffa32e00700 1 -- 172.21.15.62:0/246783944 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa2c04f2a0 con 0x7ffa2c069140 2024-04-20T21:52:20.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:20.281+0000 7ffa28c00700 1 -- 172.21.15.62:0/246783944 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ffa1800f5d0 con 0x7ffa2c069140 2024-04-20T21:52:20.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:20.282+0000 7ffa28c00700 1 --2- 172.21.15.62:0/246783944 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ffa1006ecc0 0x7ffa10071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:52:20.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:20.282+0000 7ffa28c00700 1 -- 172.21.15.62:0/246783944 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7ffa18090420 con 0x7ffa2c069140 2024-04-20T21:52:20.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:20.283+0000 7ffa2be00700 1 --2- 172.21.15.62:0/246783944 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ffa1006ecc0 0x7ffa10071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:52:20.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:20.284+0000 7ffa2be00700 1 --2- 172.21.15.62:0/246783944 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ffa1006ecc0 0x7ffa10071170 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c006fd0 tx=0x7ffa1c008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:52:20.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:20.287+0000 7ffa28c00700 1 -- 172.21.15.62:0/246783944 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7ffa180591b0 con 0x7ffa2c069140 2024-04-20T21:52:20.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:20.533+0000 7ffa32e00700 1 -- 172.21.15.62:0/246783944 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7ffa2c19d670 con 0x7ffa1006ecc0 2024-04-20T21:52:20.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:20.567+0000 7ffa28c00700 1 -- 172.21.15.62:0/246783944 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7ffa2c19d670 con 0x7ffa1006ecc0 2024-04-20T21:52:20.567 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T21:52:20.567 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (18m) 9m ago 32m 14.2M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T21:52:20.567 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (32m) 9m ago 32m 9512k - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T21:52:20.567 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (31m) 10m ago 31m 8942k - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T21:52:20.567 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (14m) 9m ago 32m 6660k - 19.0.0-3133-ga8c84f59 76bc6b91b726 2825d5ba11c6 2024-04-20T21:52:20.567 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (14m) 10m ago 30m 6665k - 19.0.0-3133-ga8c84f59 76bc6b91b726 20f746fa1bba 2024-04-20T21:52:20.567 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (17m) 9m ago 31m 74.2M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T21:52:20.567 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (9m) 9m ago 24m 11.2M - 19.0.0-3133-ga8c84f59 76bc6b91b726 de80bc90a683 2024-04-20T21:52:20.567 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (10m) 9m ago 24m 11.9M - 19.0.0-3133-ga8c84f59 76bc6b91b726 cd793b7ee98e 2024-04-20T21:52:20.567 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (24m) 10m ago 24m 117M - 18.2.0 dc2bc1663786 fb2a8a223db8 2024-04-20T21:52:20.568 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (24m) 10m ago 24m 55.6M - 18.2.0 dc2bc1663786 f2eec2ee8951 2024-04-20T21:52:20.568 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (21m) 9m ago 33m 511M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T21:52:20.568 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (19m) 10m ago 30m 452M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T21:52:20.568 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (15m) 9m ago 34m 55.9M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 019eb1479485 2024-04-20T21:52:20.568 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (15m) 10m ago 30m 39.8M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 55aac560aab6 2024-04-20T21:52:20.568 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (19m) 9m ago 32m 7952k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T21:52:20.568 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (19m) 10m ago 30m 8304k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T21:52:20.568 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (13m) 9m ago 29m 88.0M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 ee2bb901f26e 2024-04-20T21:52:20.568 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (13m) 9m ago 29m 92.3M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 9075152d4162 2024-04-20T21:52:20.568 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (12m) 9m ago 28m 74.4M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 2d4f3e38dc0e 2024-04-20T21:52:20.568 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (12m) 10m ago 27m 87.6M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 8f97b17ae0eb 2024-04-20T21:52:20.568 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (11m) 10m ago 27m 90.7M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 653d8079967d 2024-04-20T21:52:20.568 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (10m) 10m ago 26m 11.5M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 596dded14c12 2024-04-20T21:52:20.568 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (18m) 9m ago 31m 74.0M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T21:52:20.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:20.571+0000 7ffa32e00700 1 -- 172.21.15.62:0/246783944 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ffa1006ecc0 msgr2=0x7ffa10071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:52:20.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:20.571+0000 7ffa32e00700 1 --2- 172.21.15.62:0/246783944 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ffa1006ecc0 0x7ffa10071170 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c006fd0 tx=0x7ffa1c008040 comp rx=0 tx=0).stop 2024-04-20T21:52:20.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:20.571+0000 7ffa32e00700 1 -- 172.21.15.62:0/246783944 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa2c069140 msgr2=0x7ffa2c198680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:52:20.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:20.571+0000 7ffa32e00700 1 --2- 172.21.15.62:0/246783944 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa2c069140 0x7ffa2c198680 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7ffa18005f50 tx=0x7ffa1800be20 comp rx=0 tx=0).stop 2024-04-20T21:52:20.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:20.571+0000 7ffa32e00700 1 -- 172.21.15.62:0/246783944 shutdown_connections 2024-04-20T21:52:20.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:20.571+0000 7ffa32e00700 1 --2- 172.21.15.62:0/246783944 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ffa1006ecc0 0x7ffa10071170 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:52:20.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:20.571+0000 7ffa32e00700 1 --2- 172.21.15.62:0/246783944 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffa2c0687a0 0x7ffa2c198140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:52:20.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:20.571+0000 7ffa32e00700 1 --2- 172.21.15.62:0/246783944 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa2c069140 0x7ffa2c198680 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:52:20.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:20.571+0000 7ffa32e00700 1 -- 172.21.15.62:0/246783944 >> 172.21.15.62:0/246783944 conn(0x7ffa2c075130 msgr2=0x7ffa2c101e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:52:20.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:20.571+0000 7ffa32e00700 1 -- 172.21.15.62:0/246783944 shutdown_connections 2024-04-20T21:52:20.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:20.571+0000 7ffa32e00700 1 -- 172.21.15.62:0/246783944 wait complete. 2024-04-20T21:52:20.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:20.731+0000 7f3bea600700 1 -- 172.21.15.62:0/665106967 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3be4100ed0 msgr2=0x7f3be4104ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:52:20.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:20.731+0000 7f3bea600700 1 --2- 172.21.15.62:0/665106967 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3be4100ed0 0x7f3be4104ea0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f3bcc009910 tx=0x7f3bcc009c20 comp rx=0 tx=0).stop 2024-04-20T21:52:20.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:20.732+0000 7f3bea600700 1 -- 172.21.15.62:0/665106967 shutdown_connections 2024-04-20T21:52:20.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:20.732+0000 7f3bea600700 1 --2- 172.21.15.62:0/665106967 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3be4100ed0 0x7f3be4104ea0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:52:20.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:20.732+0000 7f3bea600700 1 --2- 172.21.15.62:0/665106967 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3be4100530 0x7f3be4100900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:52:20.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:20.732+0000 7f3bea600700 1 -- 172.21.15.62:0/665106967 >> 172.21.15.62:0/665106967 conn(0x7f3be40fbe10 msgr2=0x7f3be40fe220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:52:20.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:20.732+0000 7f3bea600700 1 -- 172.21.15.62:0/665106967 shutdown_connections 2024-04-20T21:52:20.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:20.732+0000 7f3bea600700 1 -- 172.21.15.62:0/665106967 wait complete. 2024-04-20T21:52:20.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:20.733+0000 7f3bea600700 1 Processor -- start 2024-04-20T21:52:20.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:20.733+0000 7f3bea600700 1 -- start start 2024-04-20T21:52:20.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:20.734+0000 7f3bea600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3be4100530 0x7f3be4073050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:52:20.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:20.734+0000 7f3bea600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3be4100ed0 0x7f3be4073590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:52:20.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:20.734+0000 7f3bea600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3be4073ad0 con 0x7f3be4100530 2024-04-20T21:52:20.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:20.734+0000 7f3bea600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3be4073c10 con 0x7f3be4100ed0 2024-04-20T21:52:20.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:20.734+0000 7f3be3400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3be4100530 0x7f3be4073050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:52:20.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:20.734+0000 7f3be3400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3be4100530 0x7f3be4073050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59254/0 (socket says 172.21.15.62:59254) 2024-04-20T21:52:20.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:20.734+0000 7f3be3400700 1 -- 172.21.15.62:0/2567474278 learned_addr learned my addr 172.21.15.62:0/2567474278 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:52:20.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:20.734+0000 7f3bdbe00700 1 --2- 172.21.15.62:0/2567474278 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3be4100ed0 0x7f3be4073590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:52:20.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:20.734+0000 7f3be3400700 1 -- 172.21.15.62:0/2567474278 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3be4100ed0 msgr2=0x7f3be4073590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:52:20.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:20.734+0000 7f3be3400700 1 --2- 172.21.15.62:0/2567474278 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3be4100ed0 0x7f3be4073590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:52:20.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:20.734+0000 7f3be3400700 1 -- 172.21.15.62:0/2567474278 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bcc0095f0 con 0x7f3be4100530 2024-04-20T21:52:20.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:20.734+0000 7f3bdbe00700 1 --2- 172.21.15.62:0/2567474278 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3be4100ed0 0x7f3be4073590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T21:52:20.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:20.735+0000 7f3be3400700 1 --2- 172.21.15.62:0/2567474278 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3be4100530 0x7f3be4073050 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f3bd400d8d0 tx=0x7f3bd400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:52:20.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:20.735+0000 7f3be0c00700 1 -- 172.21.15.62:0/2567474278 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3bd4009940 con 0x7f3be4100530 2024-04-20T21:52:20.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:20.735+0000 7f3be0c00700 1 -- 172.21.15.62:0/2567474278 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3bd4010460 con 0x7f3be4100530 2024-04-20T21:52:20.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:20.735+0000 7f3bea600700 1 -- 172.21.15.62:0/2567474278 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3be41a20c0 con 0x7f3be4100530 2024-04-20T21:52:20.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:20.735+0000 7f3be0c00700 1 -- 172.21.15.62:0/2567474278 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3bd400f5f0 con 0x7f3be4100530 2024-04-20T21:52:20.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:20.735+0000 7f3bea600700 1 -- 172.21.15.62:0/2567474278 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3be41a24d0 con 0x7f3be4100530 2024-04-20T21:52:20.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:20.736+0000 7f3bea600700 1 -- 172.21.15.62:0/2567474278 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3be41088a0 con 0x7f3be4100530 2024-04-20T21:52:20.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:20.737+0000 7f3be0c00700 1 -- 172.21.15.62:0/2567474278 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f3bd40105d0 con 0x7f3be4100530 2024-04-20T21:52:20.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:20.738+0000 7f3be0c00700 1 --2- 172.21.15.62:0/2567474278 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3bc806ecc0 0x7f3bc8071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:52:20.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:20.738+0000 7f3be0c00700 1 -- 172.21.15.62:0/2567474278 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f3bd408ef20 con 0x7f3be4100530 2024-04-20T21:52:20.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:20.739+0000 7f3bdbe00700 1 --2- 172.21.15.62:0/2567474278 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3bc806ecc0 0x7f3bc8071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:52:20.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:20.739+0000 7f3bdbe00700 1 --2- 172.21.15.62:0/2567474278 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3bc806ecc0 0x7f3bc8071170 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f3bcc006010 tx=0x7f3bcc00b540 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:52:20.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:20.743+0000 7f3be0c00700 1 -- 172.21.15.62:0/2567474278 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f3bd4057c30 con 0x7f3be4100530 2024-04-20T21:52:21.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.066+0000 7f3bea600700 1 -- 172.21.15.62:0/2567474278 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f3be4108280 con 0x7f3be4100530 2024-04-20T21:52:21.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.068+0000 7f3be0c00700 1 -- 172.21.15.62:0/2567474278 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7f3bd405b2f0 con 0x7f3be4100530 2024-04-20T21:52:21.067 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:52:21.067 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T21:52:21.067 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:52:21.067 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:52:21.067 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T21:52:21.068 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:52:21.068 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:52:21.068 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T21:52:21.068 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 6 2024-04-20T21:52:21.068 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:52:21.068 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T21:52:21.068 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 2, 2024-04-20T21:52:21.068 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:52:21.068 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:52:21.068 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T21:52:21.068 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 2, 2024-04-20T21:52:21.068 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 12 2024-04-20T21:52:21.068 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T21:52:21.068 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:52:21.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.070+0000 7f3bea600700 1 -- 172.21.15.62:0/2567474278 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3bc806ecc0 msgr2=0x7f3bc8071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:52:21.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.070+0000 7f3bea600700 1 --2- 172.21.15.62:0/2567474278 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3bc806ecc0 0x7f3bc8071170 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f3bcc006010 tx=0x7f3bcc00b540 comp rx=0 tx=0).stop 2024-04-20T21:52:21.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.070+0000 7f3bea600700 1 -- 172.21.15.62:0/2567474278 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3be4100530 msgr2=0x7f3be4073050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:52:21.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.070+0000 7f3bea600700 1 --2- 172.21.15.62:0/2567474278 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3be4100530 0x7f3be4073050 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f3bd400d8d0 tx=0x7f3bd400dc90 comp rx=0 tx=0).stop 2024-04-20T21:52:21.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.070+0000 7f3bea600700 1 -- 172.21.15.62:0/2567474278 shutdown_connections 2024-04-20T21:52:21.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.070+0000 7f3bea600700 1 --2- 172.21.15.62:0/2567474278 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3bc806ecc0 0x7f3bc8071170 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:52:21.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.070+0000 7f3bea600700 1 --2- 172.21.15.62:0/2567474278 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3be4100530 0x7f3be4073050 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:52:21.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.070+0000 7f3bea600700 1 --2- 172.21.15.62:0/2567474278 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3be4100ed0 0x7f3be4073590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:52:21.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.070+0000 7f3bea600700 1 -- 172.21.15.62:0/2567474278 >> 172.21.15.62:0/2567474278 conn(0x7f3be40fbe10 msgr2=0x7f3be40fd3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:52:21.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.071+0000 7f3bea600700 1 -- 172.21.15.62:0/2567474278 shutdown_connections 2024-04-20T21:52:21.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.071+0000 7f3bea600700 1 -- 172.21.15.62:0/2567474278 wait complete. 2024-04-20T21:52:21.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.233+0000 7f1191c00700 1 -- 172.21.15.62:0/2845781981 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f118c073090 msgr2=0x7f118c073460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:52:21.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.233+0000 7f1191c00700 1 --2- 172.21.15.62:0/2845781981 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f118c073090 0x7f118c073460 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f117c009940 tx=0x7f117c009c50 comp rx=0 tx=0).stop 2024-04-20T21:52:21.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.234+0000 7f1191c00700 1 -- 172.21.15.62:0/2845781981 shutdown_connections 2024-04-20T21:52:21.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.234+0000 7f1191c00700 1 --2- 172.21.15.62:0/2845781981 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f118c0739a0 0x7f118c10c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:52:21.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.234+0000 7f1191c00700 1 --2- 172.21.15.62:0/2845781981 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f118c073090 0x7f118c073460 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:52:21.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.234+0000 7f1191c00700 1 -- 172.21.15.62:0/2845781981 >> 172.21.15.62:0/2845781981 conn(0x7f118c0fbda0 msgr2=0x7f118c0fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:52:21.234 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:52:20 smithi062 ceph-mon[76322]: from='client.34956 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:52:21.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.234+0000 7f1191c00700 1 -- 172.21.15.62:0/2845781981 shutdown_connections 2024-04-20T21:52:21.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.234+0000 7f1191c00700 1 -- 172.21.15.62:0/2845781981 wait complete. 2024-04-20T21:52:21.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.235+0000 7f1191c00700 1 Processor -- start 2024-04-20T21:52:21.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.235+0000 7f1191c00700 1 -- start start 2024-04-20T21:52:21.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.235+0000 7f1191c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f118c073090 0x7f118c198180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:52:21.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.235+0000 7f1191c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f118c0739a0 0x7f118c1986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:52:21.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.235+0000 7f1191c00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f118c198d10 con 0x7f118c0739a0 2024-04-20T21:52:21.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.235+0000 7f1191c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f118c198e50 con 0x7f118c073090 2024-04-20T21:52:21.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.236+0000 7f118a000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f118c0739a0 0x7f118c1986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:52:21.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.236+0000 7f118a000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f118c0739a0 0x7f118c1986c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59268/0 (socket says 172.21.15.62:59268) 2024-04-20T21:52:21.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.236+0000 7f118a000700 1 -- 172.21.15.62:0/2826614818 learned_addr learned my addr 172.21.15.62:0/2826614818 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:52:21.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.236+0000 7f118aa00700 1 --2- 172.21.15.62:0/2826614818 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f118c073090 0x7f118c198180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:52:21.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.236+0000 7f118a000700 1 -- 172.21.15.62:0/2826614818 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f118c073090 msgr2=0x7f118c198180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:52:21.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.236+0000 7f118a000700 1 --2- 172.21.15.62:0/2826614818 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f118c073090 0x7f118c198180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:52:21.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.236+0000 7f118a000700 1 -- 172.21.15.62:0/2826614818 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f117c0095f0 con 0x7f118c0739a0 2024-04-20T21:52:21.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.236+0000 7f118aa00700 1 --2- 172.21.15.62:0/2826614818 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f118c073090 0x7f118c198180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-20T21:52:21.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.237+0000 7f118a000700 1 --2- 172.21.15.62:0/2826614818 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f118c0739a0 0x7f118c1986c0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f117800c930 tx=0x7f117800ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:52:21.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.237+0000 7f1183400700 1 -- 172.21.15.62:0/2826614818 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1178007ad0 con 0x7f118c0739a0 2024-04-20T21:52:21.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.237+0000 7f1183400700 1 -- 172.21.15.62:0/2826614818 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1178007c30 con 0x7f118c0739a0 2024-04-20T21:52:21.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.237+0000 7f1191c00700 1 -- 172.21.15.62:0/2826614818 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f118c19cca0 con 0x7f118c0739a0 2024-04-20T21:52:21.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.237+0000 7f1183400700 1 -- 172.21.15.62:0/2826614818 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1178018610 con 0x7f118c0739a0 2024-04-20T21:52:21.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.237+0000 7f1191c00700 1 -- 172.21.15.62:0/2826614818 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f118c19d1f0 con 0x7f118c0739a0 2024-04-20T21:52:21.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:52:20 smithi189 ceph-mon[57022]: from='client.34956 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:52:21.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.239+0000 7f1191c00700 1 -- 172.21.15.62:0/2826614818 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f118c109ce0 con 0x7f118c0739a0 2024-04-20T21:52:21.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.240+0000 7f1183400700 1 -- 172.21.15.62:0/2826614818 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f117800f450 con 0x7f118c0739a0 2024-04-20T21:52:21.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.241+0000 7f1183400700 1 --2- 172.21.15.62:0/2826614818 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f117006ecc0 0x7f1170071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:52:21.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.241+0000 7f1183400700 1 -- 172.21.15.62:0/2826614818 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f117808f140 con 0x7f118c0739a0 2024-04-20T21:52:21.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.241+0000 7f118aa00700 1 --2- 172.21.15.62:0/2826614818 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f117006ecc0 0x7f1170071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:52:21.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.242+0000 7f118aa00700 1 --2- 172.21.15.62:0/2826614818 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f117006ecc0 0x7f1170071170 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f117c005b40 tx=0x7f117c005ab0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:52:21.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.245+0000 7f1183400700 1 -- 172.21.15.62:0/2826614818 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f1178057e50 con 0x7f118c0739a0 2024-04-20T21:52:21.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.533+0000 7f1191c00700 1 -- 172.21.15.62:0/2826614818 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f118c1994e0 con 0x7f118c0739a0 2024-04-20T21:52:21.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.534+0000 7f1183400700 1 -- 172.21.15.62:0/2826614818 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 25 v25) v1 ==== 76+0+1887 (secure 0 0 0) 0x7f117805b510 con 0x7f118c0739a0 2024-04-20T21:52:21.533 INFO:teuthology.orchestra.run.smithi062.stdout:e25 2024-04-20T21:52:21.533 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T21:52:21.534 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T21:52:21.534 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T21:52:21.534 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:52:21.534 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T21:52:21.534 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T21:52:21.534 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 20 2024-04-20T21:52:21.534 INFO:teuthology.orchestra.run.smithi062.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-20T21:52:21.534 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T21:52:21.534 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T21:42:01.559061+0000 2024-04-20T21:52:21.534 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T21:52:21.534 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T21:52:21.534 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T21:52:21.534 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T21:52:21.535 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T21:52:21.535 INFO:teuthology.orchestra.run.smithi062.stdout:max_xattr_size 65536 2024-04-20T21:52:21.535 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T21:52:21.535 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T21:52:21.535 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 81 2024-04-20T21:52:21.535 INFO:teuthology.orchestra.run.smithi062.stdout:snap_rstat 1 2024-04-20T21:52:21.535 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T21:52:21.535 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 2 2024-04-20T21:52:21.535 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1 2024-04-20T21:52:21.535 INFO:teuthology.orchestra.run.smithi062.stdout:up {} 2024-04-20T21:52:21.535 INFO:teuthology.orchestra.run.smithi062.stdout:failed 0,1 2024-04-20T21:52:21.535 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T21:52:21.536 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T21:52:21.536 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T21:52:21.536 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T21:52:21.536 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data enabled 2024-04-20T21:52:21.536 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T21:52:21.536 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T21:52:21.536 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 1 2024-04-20T21:52:21.536 INFO:teuthology.orchestra.run.smithi062.stdout:qdb_cluster leader: 0 members: 2024-04-20T21:52:21.536 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:52:21.536 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:52:21.536 INFO:teuthology.orchestra.run.smithi062.stdout:Standby daemons: 2024-04-20T21:52:21.537 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:52:21.537 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ryqblb{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6826/4113916205,v1:172.21.15.62:6827/4113916205] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T21:52:21.537 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ozizwi{-1:34476} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6828/3801975165,v1:172.21.15.62:6829/3801975165] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T21:52:21.537 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.xbgspw{-1:44285} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6826/3996040526,v1:172.21.15.189:6827/3996040526] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:52:21.537 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.dvimer{-1:44289} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6824/1743111843,v1:172.21.15.189:6825/1743111843] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:52:21.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.537+0000 7f1191c00700 1 -- 172.21.15.62:0/2826614818 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f117006ecc0 msgr2=0x7f1170071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:52:21.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.537+0000 7f1191c00700 1 --2- 172.21.15.62:0/2826614818 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f117006ecc0 0x7f1170071170 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f117c005b40 tx=0x7f117c005ab0 comp rx=0 tx=0).stop 2024-04-20T21:52:21.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.537+0000 7f1191c00700 1 -- 172.21.15.62:0/2826614818 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f118c0739a0 msgr2=0x7f118c1986c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:52:21.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.537+0000 7f1191c00700 1 --2- 172.21.15.62:0/2826614818 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f118c0739a0 0x7f118c1986c0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f117800c930 tx=0x7f117800ccf0 comp rx=0 tx=0).stop 2024-04-20T21:52:21.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.537+0000 7f1191c00700 1 -- 172.21.15.62:0/2826614818 shutdown_connections 2024-04-20T21:52:21.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.537+0000 7f1191c00700 1 --2- 172.21.15.62:0/2826614818 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f117006ecc0 0x7f1170071170 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:52:21.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.537+0000 7f1191c00700 1 --2- 172.21.15.62:0/2826614818 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f118c073090 0x7f118c198180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:52:21.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.537+0000 7f1191c00700 1 --2- 172.21.15.62:0/2826614818 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f118c0739a0 0x7f118c1986c0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:52:21.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.537+0000 7f1191c00700 1 -- 172.21.15.62:0/2826614818 >> 172.21.15.62:0/2826614818 conn(0x7f118c0fbda0 msgr2=0x7f118c106e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:52:21.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.538+0000 7f1191c00700 1 -- 172.21.15.62:0/2826614818 shutdown_connections 2024-04-20T21:52:21.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.538+0000 7f1191c00700 1 -- 172.21.15.62:0/2826614818 wait complete. 2024-04-20T21:52:21.539 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 25 2024-04-20T21:52:21.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.689+0000 7f77e9c00700 1 -- 172.21.15.62:0/1001510192 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f77e4108470 msgr2=0x7f77e4108840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:52:21.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.689+0000 7f77e9c00700 1 --2- 172.21.15.62:0/1001510192 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f77e4108470 0x7f77e4108840 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f77d4009910 tx=0x7f77d4009c20 comp rx=0 tx=0).stop 2024-04-20T21:52:21.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.689+0000 7f77e9c00700 1 -- 172.21.15.62:0/1001510192 shutdown_connections 2024-04-20T21:52:21.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.689+0000 7f77e9c00700 1 --2- 172.21.15.62:0/1001510192 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f77e4102470 0x7f77e41028e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:52:21.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.689+0000 7f77e9c00700 1 --2- 172.21.15.62:0/1001510192 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f77e4108470 0x7f77e4108840 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:52:21.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.689+0000 7f77e9c00700 1 -- 172.21.15.62:0/1001510192 >> 172.21.15.62:0/1001510192 conn(0x7f77e40fdf90 msgr2=0x7f77e41003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:52:21.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.689+0000 7f77e9c00700 1 -- 172.21.15.62:0/1001510192 shutdown_connections 2024-04-20T21:52:21.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.689+0000 7f77e9c00700 1 -- 172.21.15.62:0/1001510192 wait complete. 2024-04-20T21:52:21.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.690+0000 7f77e9c00700 1 Processor -- start 2024-04-20T21:52:21.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.691+0000 7f77e9c00700 1 -- start start 2024-04-20T21:52:21.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.691+0000 7f77e9c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f77e4102470 0x7f77e4198230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:52:21.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.691+0000 7f77e9c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f77e4108470 0x7f77e4198770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:52:21.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.691+0000 7f77e9c00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77e4198e50 con 0x7f77e4102470 2024-04-20T21:52:21.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.691+0000 7f77e9c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77e419cbe0 con 0x7f77e4108470 2024-04-20T21:52:21.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.691+0000 7f77e2a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f77e4102470 0x7f77e4198230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:52:21.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.691+0000 7f77e2a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f77e4102470 0x7f77e4198230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42996/0 (socket says 172.21.15.62:42996) 2024-04-20T21:52:21.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.691+0000 7f77e2a00700 1 -- 172.21.15.62:0/1089465041 learned_addr learned my addr 172.21.15.62:0/1089465041 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:52:21.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.692+0000 7f77e2a00700 1 -- 172.21.15.62:0/1089465041 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f77e4108470 msgr2=0x7f77e4198770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:52:21.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.692+0000 7f77e2000700 1 --2- 172.21.15.62:0/1089465041 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f77e4108470 0x7f77e4198770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:52:21.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.692+0000 7f77e2a00700 1 --2- 172.21.15.62:0/1089465041 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f77e4108470 0x7f77e4198770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:52:21.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.692+0000 7f77e2a00700 1 -- 172.21.15.62:0/1089465041 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77d40095f0 con 0x7f77e4102470 2024-04-20T21:52:21.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.692+0000 7f77e2a00700 1 --2- 172.21.15.62:0/1089465041 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f77e4102470 0x7f77e4198230 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f77d40092e0 tx=0x7f77d400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:52:21.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.692+0000 7f77db400700 1 -- 172.21.15.62:0/1089465041 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f77d401d070 con 0x7f77e4102470 2024-04-20T21:52:21.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.692+0000 7f77db400700 1 -- 172.21.15.62:0/1089465041 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f77d4005e00 con 0x7f77e4102470 2024-04-20T21:52:21.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.692+0000 7f77db400700 1 -- 172.21.15.62:0/1089465041 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f77d4017400 con 0x7f77e4102470 2024-04-20T21:52:21.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.692+0000 7f77e9c00700 1 -- 172.21.15.62:0/1089465041 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77e419ce60 con 0x7f77e4102470 2024-04-20T21:52:21.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.693+0000 7f77e9c00700 1 -- 172.21.15.62:0/1089465041 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77e419d350 con 0x7f77e4102470 2024-04-20T21:52:21.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.694+0000 7f77e9c00700 1 -- 172.21.15.62:0/1089465041 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77e410a9e0 con 0x7f77e4102470 2024-04-20T21:52:21.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.695+0000 7f77db400700 1 -- 172.21.15.62:0/1089465041 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f77d400f460 con 0x7f77e4102470 2024-04-20T21:52:21.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.696+0000 7f77db400700 1 --2- 172.21.15.62:0/1089465041 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f77c806ecc0 0x7f77c8071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:52:21.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.696+0000 7f77db400700 1 -- 172.21.15.62:0/1089465041 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f77d4090330 con 0x7f77e4102470 2024-04-20T21:52:21.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.696+0000 7f77e2000700 1 --2- 172.21.15.62:0/1089465041 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f77c806ecc0 0x7f77c8071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:52:21.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.697+0000 7f77e2000700 1 --2- 172.21.15.62:0/1089465041 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f77c806ecc0 0x7f77c8071170 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f77d0005950 tx=0x7f77d00058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:52:21.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.701+0000 7f77db400700 1 -- 172.21.15.62:0/1089465041 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f77d40590c0 con 0x7f77e4102470 2024-04-20T21:52:21.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.948+0000 7f77e9c00700 1 -- 172.21.15.62:0/1089465041 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f77e41996a0 con 0x7f77c806ecc0 2024-04-20T21:52:21.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.951+0000 7f77db400700 1 -- 172.21.15.62:0/1089465041 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f77e41996a0 con 0x7f77c806ecc0 2024-04-20T21:52:21.950 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:52:21.950 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T21:52:21.950 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T21:52:21.951 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-20T21:52:21.951 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T21:52:21.951 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr", 2024-04-20T21:52:21.951 INFO:teuthology.orchestra.run.smithi062.stdout: "crash", 2024-04-20T21:52:21.951 INFO:teuthology.orchestra.run.smithi062.stdout: "mon", 2024-04-20T21:52:21.951 INFO:teuthology.orchestra.run.smithi062.stdout: "osd" 2024-04-20T21:52:21.951 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T21:52:21.951 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "14/23 daemons upgraded", 2024-04-20T21:52:21.951 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading mds daemons", 2024-04-20T21:52:21.951 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T21:52:21.951 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:52:21.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.954+0000 7f77e9c00700 1 -- 172.21.15.62:0/1089465041 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f77c806ecc0 msgr2=0x7f77c8071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:52:21.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.954+0000 7f77e9c00700 1 --2- 172.21.15.62:0/1089465041 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f77c806ecc0 0x7f77c8071170 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f77d0005950 tx=0x7f77d00058e0 comp rx=0 tx=0).stop 2024-04-20T21:52:21.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.954+0000 7f77e9c00700 1 -- 172.21.15.62:0/1089465041 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f77e4102470 msgr2=0x7f77e4198230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:52:21.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.954+0000 7f77e9c00700 1 --2- 172.21.15.62:0/1089465041 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f77e4102470 0x7f77e4198230 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f77d40092e0 tx=0x7f77d400be20 comp rx=0 tx=0).stop 2024-04-20T21:52:21.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.954+0000 7f77e9c00700 1 -- 172.21.15.62:0/1089465041 shutdown_connections 2024-04-20T21:52:21.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.954+0000 7f77e9c00700 1 --2- 172.21.15.62:0/1089465041 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f77c806ecc0 0x7f77c8071170 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:52:21.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.954+0000 7f77e9c00700 1 --2- 172.21.15.62:0/1089465041 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f77e4102470 0x7f77e4198230 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:52:21.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.954+0000 7f77e9c00700 1 --2- 172.21.15.62:0/1089465041 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f77e4108470 0x7f77e4198770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:52:21.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.954+0000 7f77e9c00700 1 -- 172.21.15.62:0/1089465041 >> 172.21.15.62:0/1089465041 conn(0x7f77e40fdf90 msgr2=0x7f77e40ff300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:52:21.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.954+0000 7f77e9c00700 1 -- 172.21.15.62:0/1089465041 shutdown_connections 2024-04-20T21:52:21.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:21.955+0000 7f77e9c00700 1 -- 172.21.15.62:0/1089465041 wait complete. 2024-04-20T21:52:22.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:22.115+0000 7fe82ea00700 1 -- 172.21.15.62:0/619083012 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe8281026a0 msgr2=0x7fe828102a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:52:22.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:22.115+0000 7fe82ea00700 1 --2- 172.21.15.62:0/619083012 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe8281026a0 0x7fe828102a70 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fe818009910 tx=0x7fe818009c20 comp rx=0 tx=0).stop 2024-04-20T21:52:22.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:22.115+0000 7fe82ea00700 1 -- 172.21.15.62:0/619083012 shutdown_connections 2024-04-20T21:52:22.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:22.115+0000 7fe82ea00700 1 --2- 172.21.15.62:0/619083012 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe828103040 0x7fe828107010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:52:22.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:22.115+0000 7fe82ea00700 1 --2- 172.21.15.62:0/619083012 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe8281026a0 0x7fe828102a70 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:52:22.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:22.115+0000 7fe82ea00700 1 -- 172.21.15.62:0/619083012 >> 172.21.15.62:0/619083012 conn(0x7fe8280fdf40 msgr2=0x7fe828100350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:52:22.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:22.115+0000 7fe82ea00700 1 -- 172.21.15.62:0/619083012 shutdown_connections 2024-04-20T21:52:22.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:22.115+0000 7fe82ea00700 1 -- 172.21.15.62:0/619083012 wait complete. 2024-04-20T21:52:22.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:22.117+0000 7fe82ea00700 1 Processor -- start 2024-04-20T21:52:22.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:22.117+0000 7fe82ea00700 1 -- start start 2024-04-20T21:52:22.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:22.117+0000 7fe82ea00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe8281026a0 0x7fe828198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:52:22.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:22.117+0000 7fe82ea00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe828103040 0x7fe828198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:52:22.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:22.117+0000 7fe82ea00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe828198c70 con 0x7fe8281026a0 2024-04-20T21:52:22.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:22.117+0000 7fe82ea00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe82819ca00 con 0x7fe828103040 2024-04-20T21:52:22.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:22.118+0000 7fe827e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe8281026a0 0x7fe828198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:52:22.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:22.118+0000 7fe827e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe8281026a0 0x7fe828198050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43006/0 (socket says 172.21.15.62:43006) 2024-04-20T21:52:22.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:22.118+0000 7fe827e00700 1 -- 172.21.15.62:0/2559109883 learned_addr learned my addr 172.21.15.62:0/2559109883 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:52:22.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:22.118+0000 7fe827400700 1 --2- 172.21.15.62:0/2559109883 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe828103040 0x7fe828198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:52:22.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:22.118+0000 7fe827e00700 1 -- 172.21.15.62:0/2559109883 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe828103040 msgr2=0x7fe828198590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:52:22.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:22.118+0000 7fe827e00700 1 --2- 172.21.15.62:0/2559109883 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe828103040 0x7fe828198590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:52:22.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:22.118+0000 7fe827e00700 1 -- 172.21.15.62:0/2559109883 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8180095f0 con 0x7fe8281026a0 2024-04-20T21:52:22.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:22.118+0000 7fe827400700 1 --2- 172.21.15.62:0/2559109883 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe828103040 0x7fe828198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T21:52:22.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:22.118+0000 7fe827e00700 1 --2- 172.21.15.62:0/2559109883 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe8281026a0 0x7fe828198050 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fe8180092e0 tx=0x7fe81800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:52:22.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:22.119+0000 7fe824c00700 1 -- 172.21.15.62:0/2559109883 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe81801d070 con 0x7fe8281026a0 2024-04-20T21:52:22.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:22.119+0000 7fe824c00700 1 -- 172.21.15.62:0/2559109883 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe818005e00 con 0x7fe8281026a0 2024-04-20T21:52:22.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:22.119+0000 7fe824c00700 1 -- 172.21.15.62:0/2559109883 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe818017400 con 0x7fe8281026a0 2024-04-20T21:52:22.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:22.119+0000 7fe82ea00700 1 -- 172.21.15.62:0/2559109883 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe82819cc80 con 0x7fe8281026a0 2024-04-20T21:52:22.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:22.119+0000 7fe82ea00700 1 -- 172.21.15.62:0/2559109883 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe82819d170 con 0x7fe8281026a0 2024-04-20T21:52:22.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:22.120+0000 7fe82ea00700 1 -- 172.21.15.62:0/2559109883 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe82810a9b0 con 0x7fe8281026a0 2024-04-20T21:52:22.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:22.121+0000 7fe824c00700 1 -- 172.21.15.62:0/2559109883 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe81800f460 con 0x7fe8281026a0 2024-04-20T21:52:22.121 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:52:21 smithi062 ceph-mon[76322]: from='client.34960 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:52:22.121 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:52:21 smithi062 ceph-mon[76322]: from='client.34964 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:52:22.121 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:52:21 smithi062 ceph-mon[76322]: pgmap v511: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:52:22.121 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:52:21 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/2567474278' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:52:22.121 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:52:21 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/2826614818' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T21:52:22.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:22.123+0000 7fe824c00700 1 --2- 172.21.15.62:0/2559109883 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe80c06ecc0 0x7fe80c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:52:22.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:22.123+0000 7fe824c00700 1 -- 172.21.15.62:0/2559109883 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fe818090330 con 0x7fe8281026a0 2024-04-20T21:52:22.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:22.123+0000 7fe827400700 1 --2- 172.21.15.62:0/2559109883 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe80c06ecc0 0x7fe80c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:52:22.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:22.124+0000 7fe827400700 1 --2- 172.21.15.62:0/2559109883 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe80c06ecc0 0x7fe80c071170 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fe814005fd0 tx=0x7fe814005ee0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:52:22.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:22.128+0000 7fe824c00700 1 -- 172.21.15.62:0/2559109883 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fe8180590c0 con 0x7fe8281026a0 2024-04-20T21:52:22.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:52:21 smithi189 ceph-mon[57022]: from='client.34960 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:52:22.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:52:21 smithi189 ceph-mon[57022]: from='client.34964 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:52:22.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:52:21 smithi189 ceph-mon[57022]: pgmap v511: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:52:22.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:52:21 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/2567474278' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:52:22.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:52:21 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/2826614818' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T21:52:22.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:22.455+0000 7fe82ea00700 1 -- 172.21.15.62:0/2559109883 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fe82804f2a0 con 0x7fe8281026a0 2024-04-20T21:52:22.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:22.456+0000 7fe824c00700 1 -- 172.21.15.62:0/2559109883 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7fe81805c780 con 0x7fe8281026a0 2024-04-20T21:52:22.456 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-20T21:52:22.456 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-20T21:52:22.456 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is degraded 2024-04-20T21:52:22.456 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T21:52:22.456 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T21:52:22.456 INFO:teuthology.orchestra.run.smithi062.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-20T21:52:22.456 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-20T21:52:22.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:22.459+0000 7fe82ea00700 1 -- 172.21.15.62:0/2559109883 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe80c06ecc0 msgr2=0x7fe80c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:52:22.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:22.459+0000 7fe82ea00700 1 --2- 172.21.15.62:0/2559109883 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe80c06ecc0 0x7fe80c071170 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fe814005fd0 tx=0x7fe814005ee0 comp rx=0 tx=0).stop 2024-04-20T21:52:22.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:22.459+0000 7fe82ea00700 1 -- 172.21.15.62:0/2559109883 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe8281026a0 msgr2=0x7fe828198050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:52:22.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:22.459+0000 7fe82ea00700 1 --2- 172.21.15.62:0/2559109883 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe8281026a0 0x7fe828198050 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fe8180092e0 tx=0x7fe81800be20 comp rx=0 tx=0).stop 2024-04-20T21:52:22.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:22.459+0000 7fe82ea00700 1 -- 172.21.15.62:0/2559109883 shutdown_connections 2024-04-20T21:52:22.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:22.459+0000 7fe82ea00700 1 --2- 172.21.15.62:0/2559109883 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe80c06ecc0 0x7fe80c071170 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:52:22.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:22.459+0000 7fe82ea00700 1 --2- 172.21.15.62:0/2559109883 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe8281026a0 0x7fe828198050 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:52:22.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:22.459+0000 7fe82ea00700 1 --2- 172.21.15.62:0/2559109883 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe828103040 0x7fe828198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:52:22.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:22.459+0000 7fe82ea00700 1 -- 172.21.15.62:0/2559109883 >> 172.21.15.62:0/2559109883 conn(0x7fe8280fdf40 msgr2=0x7fe8280ff5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:52:22.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:22.460+0000 7fe82ea00700 1 -- 172.21.15.62:0/2559109883 shutdown_connections 2024-04-20T21:52:22.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:22.460+0000 7fe82ea00700 1 -- 172.21.15.62:0/2559109883 wait complete. 2024-04-20T21:52:23.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:52:22 smithi189 ceph-mon[57022]: from='client.34976 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:52:23.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:52:22 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/2559109883' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T21:52:23.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:52:22 smithi062 ceph-mon[76322]: from='client.34976 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:52:23.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:52:22 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/2559109883' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T21:52:24.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:52:23 smithi189 ceph-mon[57022]: pgmap v512: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:52:24.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:52:23 smithi062 ceph-mon[76322]: pgmap v512: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:52:26.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:52:25 smithi189 ceph-mon[57022]: pgmap v513: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:52:26.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:52:25 smithi062 ceph-mon[76322]: pgmap v513: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:52:28.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:52:27 smithi062 ceph-mon[76322]: pgmap v514: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:52:28.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:52:27 smithi189 ceph-mon[57022]: pgmap v514: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:52:30.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:52:29 smithi189 ceph-mon[57022]: pgmap v515: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:52:30.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:52:29 smithi062 ceph-mon[76322]: pgmap v515: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:52:32.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:52:31 smithi189 ceph-mon[57022]: pgmap v516: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:52:32.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:52:31 smithi062 ceph-mon[76322]: pgmap v516: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:52:34.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:52:33 smithi189 ceph-mon[57022]: pgmap v517: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:52:34.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:52:33 smithi062 ceph-mon[76322]: pgmap v517: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:52:35.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:52:34 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:52:35.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:52:34 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:52:36.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:52:35 smithi189 ceph-mon[57022]: pgmap v518: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:52:36.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:52:35 smithi062 ceph-mon[76322]: pgmap v518: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:52:38.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:52:37 smithi189 ceph-mon[57022]: pgmap v519: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:52:38.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:52:37 smithi062 ceph-mon[76322]: pgmap v519: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:52:40.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:52:39 smithi189 ceph-mon[57022]: pgmap v520: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:52:40.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:52:39 smithi062 ceph-mon[76322]: pgmap v520: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:52:42.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:52:41 smithi189 ceph-mon[57022]: pgmap v521: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:52:42.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:52:41 smithi062 ceph-mon[76322]: pgmap v521: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:52:44.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:52:43 smithi189 ceph-mon[57022]: pgmap v522: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:52:44.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:52:43 smithi062 ceph-mon[76322]: pgmap v522: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:52:46.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:52:45 smithi189 ceph-mon[57022]: pgmap v523: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:52:46.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:52:45 smithi062 ceph-mon[76322]: pgmap v523: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:52:48.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:52:47 smithi189 ceph-mon[57022]: pgmap v524: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:52:48.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:52:47 smithi062 ceph-mon[76322]: pgmap v524: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:52:50.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:52:49 smithi189 ceph-mon[57022]: pgmap v525: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:52:50.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:52:49 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:52:50.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:52:49 smithi062 ceph-mon[76322]: pgmap v525: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:52:50.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:52:49 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:52:52.288 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:52:51 smithi062 ceph-mon[76322]: pgmap v526: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:52:52.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:52:51 smithi189 ceph-mon[57022]: pgmap v526: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:52:52.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:52.638+0000 7ff4bc000700 1 -- 172.21.15.62:0/3625546597 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff4b41026a0 msgr2=0x7ff4b4102a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:52:52.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:52.638+0000 7ff4bc000700 1 --2- 172.21.15.62:0/3625546597 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff4b41026a0 0x7ff4b4102a70 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7ff4a8009910 tx=0x7ff4a8009c20 comp rx=0 tx=0).stop 2024-04-20T21:52:52.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:52.638+0000 7ff4bc000700 1 -- 172.21.15.62:0/3625546597 shutdown_connections 2024-04-20T21:52:52.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:52.638+0000 7ff4bc000700 1 --2- 172.21.15.62:0/3625546597 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4b4103040 0x7ff4b4107010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:52:52.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:52.638+0000 7ff4bc000700 1 --2- 172.21.15.62:0/3625546597 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff4b41026a0 0x7ff4b4102a70 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:52:52.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:52.638+0000 7ff4bc000700 1 -- 172.21.15.62:0/3625546597 >> 172.21.15.62:0/3625546597 conn(0x7ff4b40fdf40 msgr2=0x7ff4b4100350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:52:52.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:52.638+0000 7ff4bc000700 1 -- 172.21.15.62:0/3625546597 shutdown_connections 2024-04-20T21:52:52.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:52.638+0000 7ff4bc000700 1 -- 172.21.15.62:0/3625546597 wait complete. 2024-04-20T21:52:52.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:52.639+0000 7ff4bc000700 1 Processor -- start 2024-04-20T21:52:52.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:52.640+0000 7ff4bc000700 1 -- start start 2024-04-20T21:52:52.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:52.640+0000 7ff4bc000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff4b41026a0 0x7ff4b41981d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:52:52.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:52.640+0000 7ff4bc000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4b4103040 0x7ff4b4198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:52:52.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:52.640+0000 7ff4bc000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4b4198df0 con 0x7ff4b41026a0 2024-04-20T21:52:52.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:52.640+0000 7ff4bc000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4b419cb80 con 0x7ff4b4103040 2024-04-20T21:52:52.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:52.640+0000 7ff4b9600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff4b41026a0 0x7ff4b41981d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:52:52.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:52.640+0000 7ff4b9600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff4b41026a0 0x7ff4b41981d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60240/0 (socket says 172.21.15.62:60240) 2024-04-20T21:52:52.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:52.640+0000 7ff4b9600700 1 -- 172.21.15.62:0/991519530 learned_addr learned my addr 172.21.15.62:0/991519530 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:52:52.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:52.641+0000 7ff4b9600700 1 -- 172.21.15.62:0/991519530 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4b4103040 msgr2=0x7ff4b4198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:52:52.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:52.641+0000 7ff4b9600700 1 --2- 172.21.15.62:0/991519530 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4b4103040 0x7ff4b4198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:52:52.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:52.641+0000 7ff4b9600700 1 -- 172.21.15.62:0/991519530 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4a80095f0 con 0x7ff4b41026a0 2024-04-20T21:52:52.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:52.641+0000 7ff4b9600700 1 --2- 172.21.15.62:0/991519530 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff4b41026a0 0x7ff4b41981d0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7ff4a800b5c0 tx=0x7ff4a800be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:52:52.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:52.641+0000 7ff4b2000700 1 -- 172.21.15.62:0/991519530 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff4a801d070 con 0x7ff4b41026a0 2024-04-20T21:52:52.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:52.641+0000 7ff4b2000700 1 -- 172.21.15.62:0/991519530 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff4a800f460 con 0x7ff4b41026a0 2024-04-20T21:52:52.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:52.641+0000 7ff4bc000700 1 -- 172.21.15.62:0/991519530 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4b419ce00 con 0x7ff4b41026a0 2024-04-20T21:52:52.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:52.641+0000 7ff4b2000700 1 -- 172.21.15.62:0/991519530 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff4a8017610 con 0x7ff4b41026a0 2024-04-20T21:52:52.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:52.642+0000 7ff4bc000700 1 -- 172.21.15.62:0/991519530 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4b419d2f0 con 0x7ff4b41026a0 2024-04-20T21:52:52.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:52.643+0000 7ff4bc000700 1 -- 172.21.15.62:0/991519530 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4b410a9b0 con 0x7ff4b41026a0 2024-04-20T21:52:52.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:52.644+0000 7ff4b2000700 1 -- 172.21.15.62:0/991519530 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff4a800f5d0 con 0x7ff4b41026a0 2024-04-20T21:52:52.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:52.645+0000 7ff4b2000700 1 --2- 172.21.15.62:0/991519530 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff49c06ecc0 0x7ff49c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:52:52.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:52.645+0000 7ff4b2000700 1 -- 172.21.15.62:0/991519530 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7ff4a8090440 con 0x7ff4b41026a0 2024-04-20T21:52:52.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:52.645+0000 7ff4b8c00700 1 --2- 172.21.15.62:0/991519530 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff49c06ecc0 0x7ff49c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:52:52.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:52.646+0000 7ff4b8c00700 1 --2- 172.21.15.62:0/991519530 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff49c06ecc0 0x7ff49c071170 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7ff4b41997f0 tx=0x7ff4a0006d20 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:52:52.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:52.650+0000 7ff4b2000700 1 -- 172.21.15.62:0/991519530 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7ff4a80591d0 con 0x7ff4b41026a0 2024-04-20T21:52:52.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:52.903+0000 7ff4bc000700 1 -- 172.21.15.62:0/991519530 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff4b419d5c0 con 0x7ff49c06ecc0 2024-04-20T21:52:52.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:52.905+0000 7ff4b2000700 1 -- 172.21.15.62:0/991519530 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7ff4b419d5c0 con 0x7ff49c06ecc0 2024-04-20T21:52:52.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:52.909+0000 7ff4bc000700 1 -- 172.21.15.62:0/991519530 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff49c06ecc0 msgr2=0x7ff49c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:52:52.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:52.909+0000 7ff4bc000700 1 --2- 172.21.15.62:0/991519530 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff49c06ecc0 0x7ff49c071170 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7ff4b41997f0 tx=0x7ff4a0006d20 comp rx=0 tx=0).stop 2024-04-20T21:52:52.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:52.909+0000 7ff4bc000700 1 -- 172.21.15.62:0/991519530 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff4b41026a0 msgr2=0x7ff4b41981d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:52:52.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:52.909+0000 7ff4bc000700 1 --2- 172.21.15.62:0/991519530 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff4b41026a0 0x7ff4b41981d0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7ff4a800b5c0 tx=0x7ff4a800be00 comp rx=0 tx=0).stop 2024-04-20T21:52:52.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:52.909+0000 7ff4bc000700 1 -- 172.21.15.62:0/991519530 shutdown_connections 2024-04-20T21:52:52.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:52.909+0000 7ff4bc000700 1 --2- 172.21.15.62:0/991519530 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff49c06ecc0 0x7ff49c071170 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:52:52.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:52.909+0000 7ff4bc000700 1 --2- 172.21.15.62:0/991519530 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff4b41026a0 0x7ff4b41981d0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:52:52.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:52.909+0000 7ff4bc000700 1 --2- 172.21.15.62:0/991519530 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4b4103040 0x7ff4b4198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:52:52.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:52.909+0000 7ff4bc000700 1 -- 172.21.15.62:0/991519530 >> 172.21.15.62:0/991519530 conn(0x7ff4b40fdf40 msgr2=0x7ff4b40ff810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:52:52.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:52.909+0000 7ff4bc000700 1 -- 172.21.15.62:0/991519530 shutdown_connections 2024-04-20T21:52:52.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:52.909+0000 7ff4bc000700 1 -- 172.21.15.62:0/991519530 wait complete. 2024-04-20T21:52:52.924 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T21:52:53.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.074+0000 7fc5a7600700 1 -- 172.21.15.62:0/2219321475 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc5a010c8f0 msgr2=0x7fc5a010ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:52:53.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.074+0000 7fc5a7600700 1 --2- 172.21.15.62:0/2219321475 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc5a010c8f0 0x7fc5a010ccc0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fc594009910 tx=0x7fc594009c20 comp rx=0 tx=0).stop 2024-04-20T21:52:53.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.075+0000 7fc5a7600700 1 -- 172.21.15.62:0/2219321475 shutdown_connections 2024-04-20T21:52:53.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.075+0000 7fc5a7600700 1 --2- 172.21.15.62:0/2219321475 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5a0106b80 0x7fc5a0106ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:52:53.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.075+0000 7fc5a7600700 1 --2- 172.21.15.62:0/2219321475 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc5a010c8f0 0x7fc5a010ccc0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:52:53.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.075+0000 7fc5a7600700 1 -- 172.21.15.62:0/2219321475 >> 172.21.15.62:0/2219321475 conn(0x7fc5a0076060 msgr2=0x7fc5a0078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:52:53.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.075+0000 7fc5a7600700 1 -- 172.21.15.62:0/2219321475 shutdown_connections 2024-04-20T21:52:53.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.075+0000 7fc5a7600700 1 -- 172.21.15.62:0/2219321475 wait complete. 2024-04-20T21:52:53.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.076+0000 7fc5a7600700 1 Processor -- start 2024-04-20T21:52:53.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.076+0000 7fc5a7600700 1 -- start start 2024-04-20T21:52:53.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.076+0000 7fc5a7600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc5a0106b80 0x7fc5a019c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:52:53.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.076+0000 7fc5a7600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5a010c8f0 0x7fc5a019ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:52:53.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.076+0000 7fc5a7600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5a019d100 con 0x7fc5a0106b80 2024-04-20T21:52:53.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.076+0000 7fc5a7600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5a01a0e90 con 0x7fc5a010c8f0 2024-04-20T21:52:53.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.077+0000 7fc5a4c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc5a0106b80 0x7fc5a019c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:52:53.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.077+0000 7fc5a4c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc5a0106b80 0x7fc5a019c4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60254/0 (socket says 172.21.15.62:60254) 2024-04-20T21:52:53.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.077+0000 7fc5a4c00700 1 -- 172.21.15.62:0/4090774714 learned_addr learned my addr 172.21.15.62:0/4090774714 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:52:53.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.077+0000 7fc59fe00700 1 --2- 172.21.15.62:0/4090774714 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5a010c8f0 0x7fc5a019ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:52:53.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.077+0000 7fc5a4c00700 1 -- 172.21.15.62:0/4090774714 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5a010c8f0 msgr2=0x7fc5a019ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:52:53.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.077+0000 7fc5a4c00700 1 --2- 172.21.15.62:0/4090774714 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5a010c8f0 0x7fc5a019ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:52:53.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.077+0000 7fc5a4c00700 1 -- 172.21.15.62:0/4090774714 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5940095f0 con 0x7fc5a0106b80 2024-04-20T21:52:53.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.078+0000 7fc5a4c00700 1 --2- 172.21.15.62:0/4090774714 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc5a0106b80 0x7fc5a019c4e0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fc594006010 tx=0x7fc59400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:52:53.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.078+0000 7fc59d600700 1 -- 172.21.15.62:0/4090774714 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc59401d070 con 0x7fc5a0106b80 2024-04-20T21:52:53.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.078+0000 7fc59d600700 1 -- 172.21.15.62:0/4090774714 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc594004fb0 con 0x7fc5a0106b80 2024-04-20T21:52:53.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.078+0000 7fc59d600700 1 -- 172.21.15.62:0/4090774714 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc59400f690 con 0x7fc5a0106b80 2024-04-20T21:52:53.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.078+0000 7fc5a7600700 1 -- 172.21.15.62:0/4090774714 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5a01a1110 con 0x7fc5a0106b80 2024-04-20T21:52:53.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.078+0000 7fc5a7600700 1 -- 172.21.15.62:0/4090774714 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5a01a1600 con 0x7fc5a0106b80 2024-04-20T21:52:53.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.080+0000 7fc5a7600700 1 -- 172.21.15.62:0/4090774714 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5a0109b20 con 0x7fc5a0106b80 2024-04-20T21:52:53.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.080+0000 7fc59d600700 1 -- 172.21.15.62:0/4090774714 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fc5940228a0 con 0x7fc5a0106b80 2024-04-20T21:52:53.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.081+0000 7fc59d600700 1 --2- 172.21.15.62:0/4090774714 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc58806ecc0 0x7fc588071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:52:53.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.082+0000 7fc59d600700 1 -- 172.21.15.62:0/4090774714 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fc594090660 con 0x7fc5a0106b80 2024-04-20T21:52:53.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.082+0000 7fc59fe00700 1 --2- 172.21.15.62:0/4090774714 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc58806ecc0 0x7fc588071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:52:53.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.083+0000 7fc59fe00700 1 --2- 172.21.15.62:0/4090774714 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc58806ecc0 0x7fc588071170 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fc58c006fd0 tx=0x7fc58c008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:52:53.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.086+0000 7fc59d600700 1 -- 172.21.15.62:0/4090774714 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fc594059490 con 0x7fc5a0106b80 2024-04-20T21:52:53.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.345+0000 7fc5a7600700 1 -- 172.21.15.62:0/4090774714 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc5a01a1870 con 0x7fc58806ecc0 2024-04-20T21:52:53.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.347+0000 7fc59d600700 1 -- 172.21.15.62:0/4090774714 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fc5a01a1870 con 0x7fc58806ecc0 2024-04-20T21:52:53.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.351+0000 7fc5a7600700 1 -- 172.21.15.62:0/4090774714 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc58806ecc0 msgr2=0x7fc588071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:52:53.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.351+0000 7fc5a7600700 1 --2- 172.21.15.62:0/4090774714 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc58806ecc0 0x7fc588071170 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fc58c006fd0 tx=0x7fc58c008040 comp rx=0 tx=0).stop 2024-04-20T21:52:53.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.351+0000 7fc5a7600700 1 -- 172.21.15.62:0/4090774714 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc5a0106b80 msgr2=0x7fc5a019c4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:52:53.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.351+0000 7fc5a7600700 1 --2- 172.21.15.62:0/4090774714 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc5a0106b80 0x7fc5a019c4e0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fc594006010 tx=0x7fc59400be20 comp rx=0 tx=0).stop 2024-04-20T21:52:53.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.351+0000 7fc5a7600700 1 -- 172.21.15.62:0/4090774714 shutdown_connections 2024-04-20T21:52:53.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.351+0000 7fc5a7600700 1 --2- 172.21.15.62:0/4090774714 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc58806ecc0 0x7fc588071170 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:52:53.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.351+0000 7fc5a7600700 1 --2- 172.21.15.62:0/4090774714 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc5a0106b80 0x7fc5a019c4e0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:52:53.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.351+0000 7fc5a7600700 1 --2- 172.21.15.62:0/4090774714 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5a010c8f0 0x7fc5a019ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:52:53.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.351+0000 7fc5a7600700 1 -- 172.21.15.62:0/4090774714 >> 172.21.15.62:0/4090774714 conn(0x7fc5a0076060 msgr2=0x7fc5a0077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:52:53.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.351+0000 7fc5a7600700 1 -- 172.21.15.62:0/4090774714 shutdown_connections 2024-04-20T21:52:53.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.352+0000 7fc5a7600700 1 -- 172.21.15.62:0/4090774714 wait complete. 2024-04-20T21:52:53.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.512+0000 7fcc3f600700 1 -- 172.21.15.62:0/3851417282 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcc381004e0 msgr2=0x7fcc381008b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:52:53.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.512+0000 7fcc3f600700 1 --2- 172.21.15.62:0/3851417282 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcc381004e0 0x7fcc381008b0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fcc2c009910 tx=0x7fcc2c009c20 comp rx=0 tx=0).stop 2024-04-20T21:52:53.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.512+0000 7fcc3f600700 1 -- 172.21.15.62:0/3851417282 shutdown_connections 2024-04-20T21:52:53.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.512+0000 7fcc3f600700 1 --2- 172.21.15.62:0/3851417282 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc38100e80 0x7fcc38104e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:52:53.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.512+0000 7fcc3f600700 1 --2- 172.21.15.62:0/3851417282 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcc381004e0 0x7fcc381008b0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:52:53.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.512+0000 7fcc3f600700 1 -- 172.21.15.62:0/3851417282 >> 172.21.15.62:0/3851417282 conn(0x7fcc380fbda0 msgr2=0x7fcc380fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:52:53.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.512+0000 7fcc3f600700 1 -- 172.21.15.62:0/3851417282 shutdown_connections 2024-04-20T21:52:53.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.512+0000 7fcc3f600700 1 -- 172.21.15.62:0/3851417282 wait complete. 2024-04-20T21:52:53.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.514+0000 7fcc3f600700 1 Processor -- start 2024-04-20T21:52:53.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.515+0000 7fcc3f600700 1 -- start start 2024-04-20T21:52:53.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.515+0000 7fcc3f600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcc381004e0 0x7fcc38195ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:52:53.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.515+0000 7fcc3f600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc38100e80 0x7fcc38196430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:52:53.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.515+0000 7fcc3f600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc38196b10 con 0x7fcc381004e0 2024-04-20T21:52:53.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.515+0000 7fcc3f600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc3819a8a0 con 0x7fcc38100e80 2024-04-20T21:52:53.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.515+0000 7fcc3cc00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcc381004e0 0x7fcc38195ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:52:53.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.516+0000 7fcc3cc00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcc381004e0 0x7fcc38195ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60268/0 (socket says 172.21.15.62:60268) 2024-04-20T21:52:53.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.516+0000 7fcc3cc00700 1 -- 172.21.15.62:0/2589575982 learned_addr learned my addr 172.21.15.62:0/2589575982 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:52:53.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.516+0000 7fcc37e00700 1 --2- 172.21.15.62:0/2589575982 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc38100e80 0x7fcc38196430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:52:53.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.516+0000 7fcc3cc00700 1 -- 172.21.15.62:0/2589575982 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc38100e80 msgr2=0x7fcc38196430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:52:53.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.516+0000 7fcc3cc00700 1 --2- 172.21.15.62:0/2589575982 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc38100e80 0x7fcc38196430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:52:53.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.516+0000 7fcc3cc00700 1 -- 172.21.15.62:0/2589575982 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc2c0095f0 con 0x7fcc381004e0 2024-04-20T21:52:53.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.516+0000 7fcc3cc00700 1 --2- 172.21.15.62:0/2589575982 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcc381004e0 0x7fcc38195ef0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fcc2c0092e0 tx=0x7fcc2c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:52:53.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.517+0000 7fcc35600700 1 -- 172.21.15.62:0/2589575982 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcc2c01d070 con 0x7fcc381004e0 2024-04-20T21:52:53.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.517+0000 7fcc35600700 1 -- 172.21.15.62:0/2589575982 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcc2c005e00 con 0x7fcc381004e0 2024-04-20T21:52:53.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.517+0000 7fcc3f600700 1 -- 172.21.15.62:0/2589575982 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc3819ab20 con 0x7fcc381004e0 2024-04-20T21:52:53.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.517+0000 7fcc35600700 1 -- 172.21.15.62:0/2589575982 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcc2c017400 con 0x7fcc381004e0 2024-04-20T21:52:53.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.517+0000 7fcc3f600700 1 -- 172.21.15.62:0/2589575982 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc3819b010 con 0x7fcc381004e0 2024-04-20T21:52:53.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.518+0000 7fcc3f600700 1 -- 172.21.15.62:0/2589575982 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc381087f0 con 0x7fcc381004e0 2024-04-20T21:52:53.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.519+0000 7fcc35600700 1 -- 172.21.15.62:0/2589575982 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fcc2c00f460 con 0x7fcc381004e0 2024-04-20T21:52:53.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.520+0000 7fcc35600700 1 --2- 172.21.15.62:0/2589575982 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fcc2006ecc0 0x7fcc20071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:52:53.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.520+0000 7fcc35600700 1 -- 172.21.15.62:0/2589575982 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fcc2c090330 con 0x7fcc381004e0 2024-04-20T21:52:53.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.520+0000 7fcc37e00700 1 --2- 172.21.15.62:0/2589575982 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fcc2006ecc0 0x7fcc20071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:52:53.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.521+0000 7fcc37e00700 1 --2- 172.21.15.62:0/2589575982 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fcc2006ecc0 0x7fcc20071170 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fcc24005950 tx=0x7fcc240058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:52:53.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.526+0000 7fcc35600700 1 -- 172.21.15.62:0/2589575982 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fcc2c0590c0 con 0x7fcc381004e0 2024-04-20T21:52:53.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.768+0000 7fcc3f600700 1 -- 172.21.15.62:0/2589575982 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fcc38197360 con 0x7fcc2006ecc0 2024-04-20T21:52:53.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.787+0000 7fcc35600700 1 -- 172.21.15.62:0/2589575982 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fcc38197360 con 0x7fcc2006ecc0 2024-04-20T21:52:53.786 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T21:52:53.786 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (18m) 10m ago 32m 14.2M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T21:52:53.786 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (32m) 10m ago 32m 9512k - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T21:52:53.786 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (31m) 11m ago 31m 8942k - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T21:52:53.787 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (14m) 10m ago 32m 6660k - 19.0.0-3133-ga8c84f59 76bc6b91b726 2825d5ba11c6 2024-04-20T21:52:53.787 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (14m) 11m ago 31m 6665k - 19.0.0-3133-ga8c84f59 76bc6b91b726 20f746fa1bba 2024-04-20T21:52:53.787 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (18m) 10m ago 32m 74.2M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T21:52:53.787 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (10m) 10m ago 25m 11.2M - 19.0.0-3133-ga8c84f59 76bc6b91b726 de80bc90a683 2024-04-20T21:52:53.787 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (10m) 10m ago 25m 11.9M - 19.0.0-3133-ga8c84f59 76bc6b91b726 cd793b7ee98e 2024-04-20T21:52:53.787 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (25m) 11m ago 25m 117M - 18.2.0 dc2bc1663786 fb2a8a223db8 2024-04-20T21:52:53.787 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (25m) 11m ago 25m 55.6M - 18.2.0 dc2bc1663786 f2eec2ee8951 2024-04-20T21:52:53.787 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (21m) 10m ago 34m 511M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T21:52:53.787 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (20m) 11m ago 31m 452M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T21:52:53.787 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (16m) 10m ago 34m 55.9M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 019eb1479485 2024-04-20T21:52:53.787 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (15m) 11m ago 31m 39.8M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 55aac560aab6 2024-04-20T21:52:53.787 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (19m) 10m ago 32m 7952k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T21:52:53.787 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (19m) 11m ago 31m 8304k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T21:52:53.787 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (14m) 10m ago 30m 88.0M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 ee2bb901f26e 2024-04-20T21:52:53.788 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (13m) 10m ago 29m 92.3M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 9075152d4162 2024-04-20T21:52:53.788 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (13m) 10m ago 28m 74.4M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 2d4f3e38dc0e 2024-04-20T21:52:53.788 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (12m) 11m ago 28m 87.6M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 8f97b17ae0eb 2024-04-20T21:52:53.788 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (11m) 11m ago 27m 90.7M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 653d8079967d 2024-04-20T21:52:53.788 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (11m) 11m ago 27m 11.5M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 596dded14c12 2024-04-20T21:52:53.788 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (19m) 10m ago 32m 74.0M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T21:52:53.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.789+0000 7fcc3f600700 1 -- 172.21.15.62:0/2589575982 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fcc2006ecc0 msgr2=0x7fcc20071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:52:53.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.789+0000 7fcc3f600700 1 --2- 172.21.15.62:0/2589575982 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fcc2006ecc0 0x7fcc20071170 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fcc24005950 tx=0x7fcc240058e0 comp rx=0 tx=0).stop 2024-04-20T21:52:53.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.790+0000 7fcc3f600700 1 -- 172.21.15.62:0/2589575982 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcc381004e0 msgr2=0x7fcc38195ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:52:53.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.790+0000 7fcc3f600700 1 --2- 172.21.15.62:0/2589575982 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcc381004e0 0x7fcc38195ef0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fcc2c0092e0 tx=0x7fcc2c00be20 comp rx=0 tx=0).stop 2024-04-20T21:52:53.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.790+0000 7fcc3f600700 1 -- 172.21.15.62:0/2589575982 shutdown_connections 2024-04-20T21:52:53.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.790+0000 7fcc3f600700 1 --2- 172.21.15.62:0/2589575982 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fcc2006ecc0 0x7fcc20071170 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:52:53.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.790+0000 7fcc3f600700 1 --2- 172.21.15.62:0/2589575982 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcc381004e0 0x7fcc38195ef0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:52:53.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.790+0000 7fcc3f600700 1 --2- 172.21.15.62:0/2589575982 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc38100e80 0x7fcc38196430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:52:53.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.790+0000 7fcc3f600700 1 -- 172.21.15.62:0/2589575982 >> 172.21.15.62:0/2589575982 conn(0x7fcc380fbda0 msgr2=0x7fcc380fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:52:53.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.790+0000 7fcc3f600700 1 -- 172.21.15.62:0/2589575982 shutdown_connections 2024-04-20T21:52:53.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.790+0000 7fcc3f600700 1 -- 172.21.15.62:0/2589575982 wait complete. 2024-04-20T21:52:53.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.941+0000 7fdc13e00700 1 -- 172.21.15.62:0/1367207442 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc141025f0 msgr2=0x7fdc141029c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:52:53.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.941+0000 7fdc13e00700 1 --2- 172.21.15.62:0/1367207442 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc141025f0 0x7fdc141029c0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fdc04009910 tx=0x7fdc04009c20 comp rx=0 tx=0).stop 2024-04-20T21:52:53.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.941+0000 7fdc13e00700 1 -- 172.21.15.62:0/1367207442 shutdown_connections 2024-04-20T21:52:53.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.941+0000 7fdc13e00700 1 --2- 172.21.15.62:0/1367207442 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc14103e50 0x7fdc141042c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:52:53.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.941+0000 7fdc13e00700 1 --2- 172.21.15.62:0/1367207442 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc141025f0 0x7fdc141029c0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:52:53.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.941+0000 7fdc13e00700 1 -- 172.21.15.62:0/1367207442 >> 172.21.15.62:0/1367207442 conn(0x7fdc140fddb0 msgr2=0x7fdc141001c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:52:53.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.941+0000 7fdc13e00700 1 -- 172.21.15.62:0/1367207442 shutdown_connections 2024-04-20T21:52:53.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.941+0000 7fdc13e00700 1 -- 172.21.15.62:0/1367207442 wait complete. 2024-04-20T21:52:53.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.942+0000 7fdc13e00700 1 Processor -- start 2024-04-20T21:52:53.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.942+0000 7fdc13e00700 1 -- start start 2024-04-20T21:52:53.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.943+0000 7fdc13e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc141025f0 0x7fdc14197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:52:53.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.943+0000 7fdc13e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc14103e50 0x7fdc14198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:52:53.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.943+0000 7fdc13e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc14198b00 con 0x7fdc141025f0 2024-04-20T21:52:53.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.943+0000 7fdc13e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc1419c890 con 0x7fdc14103e50 2024-04-20T21:52:53.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.943+0000 7fdc12a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc141025f0 0x7fdc14197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:52:53.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.943+0000 7fdc12a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc141025f0 0x7fdc14197ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60280/0 (socket says 172.21.15.62:60280) 2024-04-20T21:52:53.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.943+0000 7fdc12a00700 1 -- 172.21.15.62:0/351766382 learned_addr learned my addr 172.21.15.62:0/351766382 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:52:53.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.943+0000 7fdc12a00700 1 -- 172.21.15.62:0/351766382 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc14103e50 msgr2=0x7fdc14198420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:52:53.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.943+0000 7fdc12000700 1 --2- 172.21.15.62:0/351766382 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc14103e50 0x7fdc14198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:52:53.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.943+0000 7fdc12a00700 1 --2- 172.21.15.62:0/351766382 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc14103e50 0x7fdc14198420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:52:53.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.943+0000 7fdc12a00700 1 -- 172.21.15.62:0/351766382 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc040095f0 con 0x7fdc141025f0 2024-04-20T21:52:53.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.944+0000 7fdc12a00700 1 --2- 172.21.15.62:0/351766382 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc141025f0 0x7fdc14197ee0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fdc04006010 tx=0x7fdc0400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:52:53.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.944+0000 7fdc0b400700 1 -- 172.21.15.62:0/351766382 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdc0401d070 con 0x7fdc141025f0 2024-04-20T21:52:53.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.944+0000 7fdc0b400700 1 -- 172.21.15.62:0/351766382 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdc0400f460 con 0x7fdc141025f0 2024-04-20T21:52:53.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.944+0000 7fdc13e00700 1 -- 172.21.15.62:0/351766382 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc1419cb10 con 0x7fdc141025f0 2024-04-20T21:52:53.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.944+0000 7fdc0b400700 1 -- 172.21.15.62:0/351766382 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdc04017630 con 0x7fdc141025f0 2024-04-20T21:52:53.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.944+0000 7fdc13e00700 1 -- 172.21.15.62:0/351766382 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc1419d080 con 0x7fdc141025f0 2024-04-20T21:52:53.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.945+0000 7fdc13e00700 1 -- 172.21.15.62:0/351766382 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc14062380 con 0x7fdc141025f0 2024-04-20T21:52:53.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.946+0000 7fdc0b400700 1 -- 172.21.15.62:0/351766382 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fdc0400f5d0 con 0x7fdc141025f0 2024-04-20T21:52:53.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.947+0000 7fdc0b400700 1 --2- 172.21.15.62:0/351766382 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fdbf806ecc0 0x7fdbf8071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:52:53.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.947+0000 7fdc0b400700 1 -- 172.21.15.62:0/351766382 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fdc04090330 con 0x7fdc141025f0 2024-04-20T21:52:53.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.948+0000 7fdc12000700 1 --2- 172.21.15.62:0/351766382 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fdbf806ecc0 0x7fdbf8071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:52:53.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.948+0000 7fdc12000700 1 --2- 172.21.15.62:0/351766382 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fdbf806ecc0 0x7fdbf8071170 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fdc00005950 tx=0x7fdc000058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:52:53.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:53.952+0000 7fdc0b400700 1 -- 172.21.15.62:0/351766382 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fdc040590c0 con 0x7fdc141025f0 2024-04-20T21:52:54.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:52:53 smithi062 ceph-mon[76322]: pgmap v527: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:52:54.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:52:53 smithi062 ceph-mon[76322]: from='client.34984 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:52:54.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:54.281+0000 7fdc13e00700 1 -- 172.21.15.62:0/351766382 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fdc1419d340 con 0x7fdc141025f0 2024-04-20T21:52:54.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:54.283+0000 7fdc0b400700 1 -- 172.21.15.62:0/351766382 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7fdc0405c780 con 0x7fdc141025f0 2024-04-20T21:52:54.282 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:52:54.282 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T21:52:54.282 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:52:54.282 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:52:54.282 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T21:52:54.283 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:52:54.283 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:52:54.283 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T21:52:54.283 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 6 2024-04-20T21:52:54.283 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:52:54.283 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T21:52:54.283 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 2, 2024-04-20T21:52:54.283 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:52:54.283 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:52:54.283 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T21:52:54.283 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 2, 2024-04-20T21:52:54.283 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 12 2024-04-20T21:52:54.283 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T21:52:54.283 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:52:54.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:54.286+0000 7fdc13e00700 1 -- 172.21.15.62:0/351766382 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fdbf806ecc0 msgr2=0x7fdbf8071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:52:54.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:54.286+0000 7fdc13e00700 1 --2- 172.21.15.62:0/351766382 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fdbf806ecc0 0x7fdbf8071170 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fdc00005950 tx=0x7fdc000058e0 comp rx=0 tx=0).stop 2024-04-20T21:52:54.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:54.286+0000 7fdc13e00700 1 -- 172.21.15.62:0/351766382 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc141025f0 msgr2=0x7fdc14197ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:52:54.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:54.286+0000 7fdc13e00700 1 --2- 172.21.15.62:0/351766382 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc141025f0 0x7fdc14197ee0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fdc04006010 tx=0x7fdc0400be20 comp rx=0 tx=0).stop 2024-04-20T21:52:54.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:54.286+0000 7fdc13e00700 1 -- 172.21.15.62:0/351766382 shutdown_connections 2024-04-20T21:52:54.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:54.286+0000 7fdc13e00700 1 --2- 172.21.15.62:0/351766382 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fdbf806ecc0 0x7fdbf8071170 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:52:54.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:54.286+0000 7fdc13e00700 1 --2- 172.21.15.62:0/351766382 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc141025f0 0x7fdc14197ee0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:52:54.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:54.286+0000 7fdc13e00700 1 --2- 172.21.15.62:0/351766382 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc14103e50 0x7fdc14198420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:52:54.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:54.286+0000 7fdc13e00700 1 -- 172.21.15.62:0/351766382 >> 172.21.15.62:0/351766382 conn(0x7fdc140fddb0 msgr2=0x7fdc140fe6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:52:54.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:54.286+0000 7fdc13e00700 1 -- 172.21.15.62:0/351766382 shutdown_connections 2024-04-20T21:52:54.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:54.286+0000 7fdc13e00700 1 -- 172.21.15.62:0/351766382 wait complete. 2024-04-20T21:52:54.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:54.438+0000 7f2de7000700 1 -- 172.21.15.62:0/502349175 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2de0108470 msgr2=0x7f2de0108840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:52:54.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:54.438+0000 7f2de7000700 1 --2- 172.21.15.62:0/502349175 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2de0108470 0x7f2de0108840 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f2dd0009910 tx=0x7f2dd0009c20 comp rx=0 tx=0).stop 2024-04-20T21:52:54.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:54.438+0000 7f2de7000700 1 -- 172.21.15.62:0/502349175 shutdown_connections 2024-04-20T21:52:54.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:54.438+0000 7f2de7000700 1 --2- 172.21.15.62:0/502349175 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2de0102470 0x7f2de01028e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:52:54.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:54.438+0000 7f2de7000700 1 --2- 172.21.15.62:0/502349175 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2de0108470 0x7f2de0108840 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:52:54.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:54.438+0000 7f2de7000700 1 -- 172.21.15.62:0/502349175 >> 172.21.15.62:0/502349175 conn(0x7f2de00fdf90 msgr2=0x7f2de01003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:52:54.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:54.438+0000 7f2de7000700 1 -- 172.21.15.62:0/502349175 shutdown_connections 2024-04-20T21:52:54.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:54.438+0000 7f2de7000700 1 -- 172.21.15.62:0/502349175 wait complete. 2024-04-20T21:52:54.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:54.439+0000 7f2de7000700 1 Processor -- start 2024-04-20T21:52:54.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:54.440+0000 7f2de7000700 1 -- start start 2024-04-20T21:52:54.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:54.440+0000 7f2de7000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2de0102470 0x7f2de01981d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:52:54.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:54.440+0000 7f2de7000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2de0108470 0x7f2de0198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:52:54.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:54.440+0000 7f2de7000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2de0198df0 con 0x7f2de0108470 2024-04-20T21:52:54.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:54.440+0000 7f2de7000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2de019cb80 con 0x7f2de0102470 2024-04-20T21:52:54.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:54.440+0000 7f2ddf400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2de0108470 0x7f2de0198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:52:54.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:54.440+0000 7f2ddf400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2de0108470 0x7f2de0198710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60286/0 (socket says 172.21.15.62:60286) 2024-04-20T21:52:54.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:54.440+0000 7f2ddf400700 1 -- 172.21.15.62:0/3643532865 learned_addr learned my addr 172.21.15.62:0/3643532865 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:52:54.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:54.441+0000 7f2ddfe00700 1 --2- 172.21.15.62:0/3643532865 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2de0102470 0x7f2de01981d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:52:54.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:54.441+0000 7f2ddf400700 1 -- 172.21.15.62:0/3643532865 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2de0102470 msgr2=0x7f2de01981d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:52:54.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:54.441+0000 7f2ddf400700 1 --2- 172.21.15.62:0/3643532865 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2de0102470 0x7f2de01981d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:52:54.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:54.441+0000 7f2ddf400700 1 -- 172.21.15.62:0/3643532865 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dd00095f0 con 0x7f2de0108470 2024-04-20T21:52:54.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:54.441+0000 7f2ddf400700 1 --2- 172.21.15.62:0/3643532865 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2de0108470 0x7f2de0198710 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f2dcc00b4b0 tx=0x7f2dcc00b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:52:54.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:54.441+0000 7f2ddcc00700 1 -- 172.21.15.62:0/3643532865 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2dcc011800 con 0x7f2de0108470 2024-04-20T21:52:54.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:54.441+0000 7f2ddcc00700 1 -- 172.21.15.62:0/3643532865 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2dcc011e20 con 0x7f2de0108470 2024-04-20T21:52:54.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:54.441+0000 7f2de7000700 1 -- 172.21.15.62:0/3643532865 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2de019ce60 con 0x7f2de0108470 2024-04-20T21:52:54.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:54.441+0000 7f2ddcc00700 1 -- 172.21.15.62:0/3643532865 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2dcc00f2e0 con 0x7f2de0108470 2024-04-20T21:52:54.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:54.442+0000 7f2de7000700 1 -- 172.21.15.62:0/3643532865 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2de019d3b0 con 0x7f2de0108470 2024-04-20T21:52:54.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:54.442+0000 7f2de7000700 1 -- 172.21.15.62:0/3643532865 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2de010a9e0 con 0x7f2de0108470 2024-04-20T21:52:54.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:54.444+0000 7f2ddcc00700 1 -- 172.21.15.62:0/3643532865 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f2dcc010380 con 0x7f2de0108470 2024-04-20T21:52:54.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:54.445+0000 7f2ddcc00700 1 --2- 172.21.15.62:0/3643532865 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2dc406ecc0 0x7f2dc4071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:52:54.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:54.445+0000 7f2ddcc00700 1 -- 172.21.15.62:0/3643532865 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f2dcc08ea30 con 0x7f2de0108470 2024-04-20T21:52:54.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:54.445+0000 7f2ddfe00700 1 --2- 172.21.15.62:0/3643532865 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2dc406ecc0 0x7f2dc4071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:52:54.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:54.446+0000 7f2ddfe00700 1 --2- 172.21.15.62:0/3643532865 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2dc406ecc0 0x7f2dc4071170 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f2dd000b5c0 tx=0x7f2dd001a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:52:54.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:54.450+0000 7f2ddcc00700 1 -- 172.21.15.62:0/3643532865 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f2dcc057870 con 0x7f2de0108470 2024-04-20T21:52:54.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:52:53 smithi189 ceph-mon[57022]: pgmap v527: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:52:54.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:52:53 smithi189 ceph-mon[57022]: from='client.34984 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:52:54.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:54.736+0000 7f2de7000700 1 -- 172.21.15.62:0/3643532865 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f2de004f2a0 con 0x7f2de0108470 2024-04-20T21:52:54.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:54.737+0000 7f2ddcc00700 1 -- 172.21.15.62:0/3643532865 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 25 v25) v1 ==== 76+0+1887 (secure 0 0 0) 0x7f2dcc05af30 con 0x7f2de0108470 2024-04-20T21:52:54.738 INFO:teuthology.orchestra.run.smithi062.stdout:e25 2024-04-20T21:52:54.738 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T21:52:54.738 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T21:52:54.739 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T21:52:54.739 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:52:54.739 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T21:52:54.739 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T21:52:54.739 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 20 2024-04-20T21:52:54.739 INFO:teuthology.orchestra.run.smithi062.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-20T21:52:54.739 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T21:52:54.739 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T21:42:01.559061+0000 2024-04-20T21:52:54.739 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T21:52:54.739 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T21:52:54.739 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T21:52:54.739 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T21:52:54.739 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T21:52:54.739 INFO:teuthology.orchestra.run.smithi062.stdout:max_xattr_size 65536 2024-04-20T21:52:54.739 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T21:52:54.740 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T21:52:54.740 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 81 2024-04-20T21:52:54.740 INFO:teuthology.orchestra.run.smithi062.stdout:snap_rstat 1 2024-04-20T21:52:54.740 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T21:52:54.740 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 2 2024-04-20T21:52:54.740 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1 2024-04-20T21:52:54.740 INFO:teuthology.orchestra.run.smithi062.stdout:up {} 2024-04-20T21:52:54.740 INFO:teuthology.orchestra.run.smithi062.stdout:failed 0,1 2024-04-20T21:52:54.740 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T21:52:54.740 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T21:52:54.740 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T21:52:54.740 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T21:52:54.740 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data enabled 2024-04-20T21:52:54.740 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T21:52:54.740 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T21:52:54.741 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 1 2024-04-20T21:52:54.741 INFO:teuthology.orchestra.run.smithi062.stdout:qdb_cluster leader: 0 members: 2024-04-20T21:52:54.741 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:52:54.741 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:52:54.741 INFO:teuthology.orchestra.run.smithi062.stdout:Standby daemons: 2024-04-20T21:52:54.741 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:52:54.741 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ryqblb{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6826/4113916205,v1:172.21.15.62:6827/4113916205] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T21:52:54.741 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ozizwi{-1:34476} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6828/3801975165,v1:172.21.15.62:6829/3801975165] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T21:52:54.741 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.xbgspw{-1:44285} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6826/3996040526,v1:172.21.15.189:6827/3996040526] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:52:54.741 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.dvimer{-1:44289} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6824/1743111843,v1:172.21.15.189:6825/1743111843] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:52:54.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:54.740+0000 7f2de7000700 1 -- 172.21.15.62:0/3643532865 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2dc406ecc0 msgr2=0x7f2dc4071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:52:54.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:54.740+0000 7f2de7000700 1 --2- 172.21.15.62:0/3643532865 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2dc406ecc0 0x7f2dc4071170 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f2dd000b5c0 tx=0x7f2dd001a040 comp rx=0 tx=0).stop 2024-04-20T21:52:54.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:54.740+0000 7f2de7000700 1 -- 172.21.15.62:0/3643532865 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2de0108470 msgr2=0x7f2de0198710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:52:54.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:54.740+0000 7f2de7000700 1 --2- 172.21.15.62:0/3643532865 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2de0108470 0x7f2de0198710 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f2dcc00b4b0 tx=0x7f2dcc00b870 comp rx=0 tx=0).stop 2024-04-20T21:52:54.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:54.740+0000 7f2de7000700 1 -- 172.21.15.62:0/3643532865 shutdown_connections 2024-04-20T21:52:54.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:54.740+0000 7f2de7000700 1 --2- 172.21.15.62:0/3643532865 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2dc406ecc0 0x7f2dc4071170 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:52:54.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:54.740+0000 7f2de7000700 1 --2- 172.21.15.62:0/3643532865 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2de0102470 0x7f2de01981d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:52:54.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:54.740+0000 7f2de7000700 1 --2- 172.21.15.62:0/3643532865 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2de0108470 0x7f2de0198710 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:52:54.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:54.740+0000 7f2de7000700 1 -- 172.21.15.62:0/3643532865 >> 172.21.15.62:0/3643532865 conn(0x7f2de00fdf90 msgr2=0x7f2de00feb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:52:54.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:54.741+0000 7f2de7000700 1 -- 172.21.15.62:0/3643532865 shutdown_connections 2024-04-20T21:52:54.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:54.741+0000 7f2de7000700 1 -- 172.21.15.62:0/3643532865 wait complete. 2024-04-20T21:52:54.743 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 25 2024-04-20T21:52:54.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:54.893+0000 7efd20c00700 1 -- 172.21.15.62:0/3122753805 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efd1c0ffbd0 msgr2=0x7efd1c100040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:52:54.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:54.893+0000 7efd20c00700 1 --2- 172.21.15.62:0/3122753805 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efd1c0ffbd0 0x7efd1c100040 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7efd08009910 tx=0x7efd08009c20 comp rx=0 tx=0).stop 2024-04-20T21:52:54.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:54.893+0000 7efd20c00700 1 -- 172.21.15.62:0/3122753805 shutdown_connections 2024-04-20T21:52:54.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:54.893+0000 7efd20c00700 1 --2- 172.21.15.62:0/3122753805 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efd1c0ffbd0 0x7efd1c100040 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:52:54.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:54.893+0000 7efd20c00700 1 --2- 172.21.15.62:0/3122753805 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd1c104130 0x7efd1c104500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:52:54.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:54.893+0000 7efd20c00700 1 -- 172.21.15.62:0/3122753805 >> 172.21.15.62:0/3122753805 conn(0x7efd1c075130 msgr2=0x7efd1c075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:52:54.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:54.893+0000 7efd20c00700 1 -- 172.21.15.62:0/3122753805 shutdown_connections 2024-04-20T21:52:54.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:54.893+0000 7efd20c00700 1 -- 172.21.15.62:0/3122753805 wait complete. 2024-04-20T21:52:54.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:54.894+0000 7efd20c00700 1 Processor -- start 2024-04-20T21:52:54.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:54.894+0000 7efd20c00700 1 -- start start 2024-04-20T21:52:54.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:54.895+0000 7efd20c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efd1c0ffbd0 0x7efd1c198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:52:54.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:54.895+0000 7efd20c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd1c104130 0x7efd1c198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:52:54.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:54.895+0000 7efd20c00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd1c198c50 con 0x7efd1c0ffbd0 2024-04-20T21:52:54.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:54.895+0000 7efd20c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd1c19c9e0 con 0x7efd1c104130 2024-04-20T21:52:54.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:54.895+0000 7efd19e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efd1c0ffbd0 0x7efd1c198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:52:54.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:54.895+0000 7efd19e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efd1c0ffbd0 0x7efd1c198030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60308/0 (socket says 172.21.15.62:60308) 2024-04-20T21:52:54.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:54.895+0000 7efd19e00700 1 -- 172.21.15.62:0/873462652 learned_addr learned my addr 172.21.15.62:0/873462652 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:52:54.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:54.895+0000 7efd19400700 1 --2- 172.21.15.62:0/873462652 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd1c104130 0x7efd1c198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:52:54.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:54.896+0000 7efd19e00700 1 -- 172.21.15.62:0/873462652 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd1c104130 msgr2=0x7efd1c198570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:52:54.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:54.896+0000 7efd19e00700 1 --2- 172.21.15.62:0/873462652 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd1c104130 0x7efd1c198570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:52:54.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:54.896+0000 7efd19e00700 1 -- 172.21.15.62:0/873462652 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd080095f0 con 0x7efd1c0ffbd0 2024-04-20T21:52:54.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:54.896+0000 7efd19e00700 1 --2- 172.21.15.62:0/873462652 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efd1c0ffbd0 0x7efd1c198030 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7efd0c00d900 tx=0x7efd0c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:52:54.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:54.896+0000 7efd12a00700 1 -- 172.21.15.62:0/873462652 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efd0c0041f0 con 0x7efd1c0ffbd0 2024-04-20T21:52:54.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:54.896+0000 7efd20c00700 1 -- 172.21.15.62:0/873462652 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd1c19ccc0 con 0x7efd1c0ffbd0 2024-04-20T21:52:54.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:54.896+0000 7efd12a00700 1 -- 172.21.15.62:0/873462652 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efd0c004cf0 con 0x7efd1c0ffbd0 2024-04-20T21:52:54.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:54.896+0000 7efd12a00700 1 -- 172.21.15.62:0/873462652 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efd0c00b750 con 0x7efd1c0ffbd0 2024-04-20T21:52:54.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:54.896+0000 7efd20c00700 1 -- 172.21.15.62:0/873462652 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd1c19d210 con 0x7efd1c0ffbd0 2024-04-20T21:52:54.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:54.898+0000 7efd20c00700 1 -- 172.21.15.62:0/873462652 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd1c04f2a0 con 0x7efd1c0ffbd0 2024-04-20T21:52:54.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:54.899+0000 7efd12a00700 1 -- 172.21.15.62:0/873462652 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7efd0c004350 con 0x7efd1c0ffbd0 2024-04-20T21:52:54.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:54.900+0000 7efd12a00700 1 --2- 172.21.15.62:0/873462652 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7efd0006ecc0 0x7efd00071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:52:54.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:54.900+0000 7efd12a00700 1 -- 172.21.15.62:0/873462652 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7efd0c08efb0 con 0x7efd1c0ffbd0 2024-04-20T21:52:54.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:54.900+0000 7efd19400700 1 --2- 172.21.15.62:0/873462652 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7efd0006ecc0 0x7efd00071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:52:54.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:54.901+0000 7efd19400700 1 --2- 172.21.15.62:0/873462652 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7efd0006ecc0 0x7efd00071170 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7efd08005b40 tx=0x7efd0801a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:52:54.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:54.904+0000 7efd12a00700 1 -- 172.21.15.62:0/873462652 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7efd0c0c3ff0 con 0x7efd1c0ffbd0 2024-04-20T21:52:55.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:55.158+0000 7efd20c00700 1 -- 172.21.15.62:0/873462652 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7efd1c19d590 con 0x7efd0006ecc0 2024-04-20T21:52:55.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:55.160+0000 7efd12a00700 1 -- 172.21.15.62:0/873462652 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7efd1c19d590 con 0x7efd0006ecc0 2024-04-20T21:52:55.160 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:52:55.160 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T21:52:55.160 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T21:52:55.160 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-20T21:52:55.160 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T21:52:55.160 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr", 2024-04-20T21:52:55.160 INFO:teuthology.orchestra.run.smithi062.stdout: "crash", 2024-04-20T21:52:55.160 INFO:teuthology.orchestra.run.smithi062.stdout: "mon", 2024-04-20T21:52:55.160 INFO:teuthology.orchestra.run.smithi062.stdout: "osd" 2024-04-20T21:52:55.160 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T21:52:55.160 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "14/23 daemons upgraded", 2024-04-20T21:52:55.160 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading mds daemons", 2024-04-20T21:52:55.160 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T21:52:55.161 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:52:55.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:55.163+0000 7efd20c00700 1 -- 172.21.15.62:0/873462652 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7efd0006ecc0 msgr2=0x7efd00071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:52:55.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:55.163+0000 7efd20c00700 1 --2- 172.21.15.62:0/873462652 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7efd0006ecc0 0x7efd00071170 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7efd08005b40 tx=0x7efd0801a040 comp rx=0 tx=0).stop 2024-04-20T21:52:55.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:55.163+0000 7efd20c00700 1 -- 172.21.15.62:0/873462652 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efd1c0ffbd0 msgr2=0x7efd1c198030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:52:55.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:55.163+0000 7efd20c00700 1 --2- 172.21.15.62:0/873462652 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efd1c0ffbd0 0x7efd1c198030 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7efd0c00d900 tx=0x7efd0c00dcc0 comp rx=0 tx=0).stop 2024-04-20T21:52:55.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:55.163+0000 7efd20c00700 1 -- 172.21.15.62:0/873462652 shutdown_connections 2024-04-20T21:52:55.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:55.163+0000 7efd20c00700 1 --2- 172.21.15.62:0/873462652 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7efd0006ecc0 0x7efd00071170 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:52:55.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:55.163+0000 7efd20c00700 1 --2- 172.21.15.62:0/873462652 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efd1c0ffbd0 0x7efd1c198030 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:52:55.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:55.163+0000 7efd20c00700 1 --2- 172.21.15.62:0/873462652 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd1c104130 0x7efd1c198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:52:55.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:55.163+0000 7efd20c00700 1 -- 172.21.15.62:0/873462652 >> 172.21.15.62:0/873462652 conn(0x7efd1c075130 msgr2=0x7efd1c068dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:52:55.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:55.163+0000 7efd20c00700 1 -- 172.21.15.62:0/873462652 shutdown_connections 2024-04-20T21:52:55.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:55.163+0000 7efd20c00700 1 -- 172.21.15.62:0/873462652 wait complete. 2024-04-20T21:52:55.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:52:55 smithi062 ceph-mon[76322]: from='client.34988 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:52:55.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:52:55 smithi062 ceph-mon[76322]: from='client.34992 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:52:55.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:52:55 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/351766382' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:52:55.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:52:55 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/3643532865' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T21:52:55.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:55.325+0000 7f3d06c00700 1 -- 172.21.15.62:0/3951285645 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3d00073090 msgr2=0x7f3d00073460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:52:55.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:55.325+0000 7f3d06c00700 1 --2- 172.21.15.62:0/3951285645 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3d00073090 0x7f3d00073460 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f3cf000b0d0 tx=0x7f3cf000b3e0 comp rx=0 tx=0).stop 2024-04-20T21:52:55.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:55.325+0000 7f3d06c00700 1 -- 172.21.15.62:0/3951285645 shutdown_connections 2024-04-20T21:52:55.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:55.325+0000 7f3d06c00700 1 --2- 172.21.15.62:0/3951285645 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3d000739a0 0x7f3d0010c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:52:55.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:55.325+0000 7f3d06c00700 1 --2- 172.21.15.62:0/3951285645 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3d00073090 0x7f3d00073460 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:52:55.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:55.325+0000 7f3d06c00700 1 -- 172.21.15.62:0/3951285645 >> 172.21.15.62:0/3951285645 conn(0x7f3d000fbda0 msgr2=0x7f3d000fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:52:55.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:55.325+0000 7f3d06c00700 1 -- 172.21.15.62:0/3951285645 shutdown_connections 2024-04-20T21:52:55.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:55.325+0000 7f3d06c00700 1 -- 172.21.15.62:0/3951285645 wait complete. 2024-04-20T21:52:55.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:55.326+0000 7f3d06c00700 1 Processor -- start 2024-04-20T21:52:55.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:55.327+0000 7f3d06c00700 1 -- start start 2024-04-20T21:52:55.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:55.327+0000 7f3d06c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3d00073090 0x7f3d00198220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:52:55.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:55.327+0000 7f3d06c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3d000739a0 0x7f3d00198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:52:55.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:55.327+0000 7f3d06c00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d00198e40 con 0x7f3d000739a0 2024-04-20T21:52:55.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:55.327+0000 7f3d06c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d0019cbd0 con 0x7f3d00073090 2024-04-20T21:52:55.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:55.327+0000 7f3cf7400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3d000739a0 0x7f3d00198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:52:55.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:55.327+0000 7f3cf7400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3d000739a0 0x7f3d00198760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60322/0 (socket says 172.21.15.62:60322) 2024-04-20T21:52:55.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:55.327+0000 7f3cf7400700 1 -- 172.21.15.62:0/3982014226 learned_addr learned my addr 172.21.15.62:0/3982014226 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:52:55.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:55.328+0000 7f3cffe00700 1 --2- 172.21.15.62:0/3982014226 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3d00073090 0x7f3d00198220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:52:55.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:55.328+0000 7f3cf7400700 1 -- 172.21.15.62:0/3982014226 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3d00073090 msgr2=0x7f3d00198220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:52:55.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:55.328+0000 7f3cf7400700 1 --2- 172.21.15.62:0/3982014226 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3d00073090 0x7f3d00198220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:52:55.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:55.328+0000 7f3cf7400700 1 -- 172.21.15.62:0/3982014226 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cf0009d00 con 0x7f3d000739a0 2024-04-20T21:52:55.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:55.329+0000 7f3cf7400700 1 --2- 172.21.15.62:0/3982014226 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3d000739a0 0x7f3d00198760 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f3ce800d8d0 tx=0x7f3ce800dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:52:55.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:55.329+0000 7f3cfd600700 1 -- 172.21.15.62:0/3982014226 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3ce80098a0 con 0x7f3d000739a0 2024-04-20T21:52:55.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:55.329+0000 7f3cfd600700 1 -- 172.21.15.62:0/3982014226 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3ce8010460 con 0x7f3d000739a0 2024-04-20T21:52:55.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:55.329+0000 7f3d06c00700 1 -- 172.21.15.62:0/3982014226 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d0019ceb0 con 0x7f3d000739a0 2024-04-20T21:52:55.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:55.329+0000 7f3cfd600700 1 -- 172.21.15.62:0/3982014226 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3ce800f5f0 con 0x7f3d000739a0 2024-04-20T21:52:55.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:55.329+0000 7f3d06c00700 1 -- 172.21.15.62:0/3982014226 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d0019d400 con 0x7f3d000739a0 2024-04-20T21:52:55.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:55.330+0000 7f3d06c00700 1 -- 172.21.15.62:0/3982014226 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d00109ce0 con 0x7f3d000739a0 2024-04-20T21:52:55.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:55.332+0000 7f3cfd600700 1 -- 172.21.15.62:0/3982014226 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f3ce8009a00 con 0x7f3d000739a0 2024-04-20T21:52:55.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:55.333+0000 7f3cfd600700 1 --2- 172.21.15.62:0/3982014226 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3ce406ecc0 0x7f3ce4071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:52:55.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:55.333+0000 7f3cfd600700 1 -- 172.21.15.62:0/3982014226 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f3ce808edd0 con 0x7f3d000739a0 2024-04-20T21:52:55.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:55.333+0000 7f3cffe00700 1 --2- 172.21.15.62:0/3982014226 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3ce406ecc0 0x7f3ce4071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:52:55.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:55.334+0000 7f3cffe00700 1 --2- 172.21.15.62:0/3982014226 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3ce406ecc0 0x7f3ce4071170 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f3cf0000f80 tx=0x7f3cf000b040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:52:55.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:55.338+0000 7f3cfd600700 1 -- 172.21.15.62:0/3982014226 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f3ce8057ae0 con 0x7f3d000739a0 2024-04-20T21:52:55.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:52:55 smithi189 ceph-mon[57022]: from='client.34988 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:52:55.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:52:55 smithi189 ceph-mon[57022]: from='client.34992 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:52:55.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:52:55 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/351766382' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:52:55.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:52:55 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/3643532865' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T21:52:55.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:55.673+0000 7f3d06c00700 1 -- 172.21.15.62:0/3982014226 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f3d00199650 con 0x7f3d000739a0 2024-04-20T21:52:55.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:55.673+0000 7f3cfd600700 1 -- 172.21.15.62:0/3982014226 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7f3ce805b1a0 con 0x7f3d000739a0 2024-04-20T21:52:55.673 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-20T21:52:55.673 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-20T21:52:55.673 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is degraded 2024-04-20T21:52:55.673 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T21:52:55.673 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T21:52:55.673 INFO:teuthology.orchestra.run.smithi062.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-20T21:52:55.673 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-20T21:52:55.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:55.677+0000 7f3d06c00700 1 -- 172.21.15.62:0/3982014226 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3ce406ecc0 msgr2=0x7f3ce4071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:52:55.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:55.677+0000 7f3d06c00700 1 --2- 172.21.15.62:0/3982014226 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3ce406ecc0 0x7f3ce4071170 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f3cf0000f80 tx=0x7f3cf000b040 comp rx=0 tx=0).stop 2024-04-20T21:52:55.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:55.677+0000 7f3d06c00700 1 -- 172.21.15.62:0/3982014226 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3d000739a0 msgr2=0x7f3d00198760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:52:55.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:55.677+0000 7f3d06c00700 1 --2- 172.21.15.62:0/3982014226 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3d000739a0 0x7f3d00198760 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f3ce800d8d0 tx=0x7f3ce800dbe0 comp rx=0 tx=0).stop 2024-04-20T21:52:55.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:55.677+0000 7f3d06c00700 1 -- 172.21.15.62:0/3982014226 shutdown_connections 2024-04-20T21:52:55.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:55.677+0000 7f3d06c00700 1 --2- 172.21.15.62:0/3982014226 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3ce406ecc0 0x7f3ce4071170 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:52:55.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:55.677+0000 7f3d06c00700 1 --2- 172.21.15.62:0/3982014226 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3d00073090 0x7f3d00198220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:52:55.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:55.677+0000 7f3d06c00700 1 --2- 172.21.15.62:0/3982014226 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3d000739a0 0x7f3d00198760 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:52:55.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:55.677+0000 7f3d06c00700 1 -- 172.21.15.62:0/3982014226 >> 172.21.15.62:0/3982014226 conn(0x7f3d000fbda0 msgr2=0x7f3d00106e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:52:55.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:55.677+0000 7f3d06c00700 1 -- 172.21.15.62:0/3982014226 shutdown_connections 2024-04-20T21:52:55.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:52:55.677+0000 7f3d06c00700 1 -- 172.21.15.62:0/3982014226 wait complete. 2024-04-20T21:52:56.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:52:56 smithi062 ceph-mon[76322]: pgmap v528: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:52:56.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:52:56 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/3982014226' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T21:52:56.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:52:56 smithi189 ceph-mon[57022]: pgmap v528: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:52:56.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:52:56 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/3982014226' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T21:52:57.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:52:57 smithi062 ceph-mon[76322]: from='client.35004 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:52:57.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:52:57 smithi189 ceph-mon[57022]: from='client.35004 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:52:58.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:52:58 smithi062 ceph-mon[76322]: pgmap v529: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:52:58.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:52:58 smithi189 ceph-mon[57022]: pgmap v529: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:53:00.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:53:00 smithi062 ceph-mon[76322]: pgmap v530: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:53:00.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:53:00 smithi189 ceph-mon[57022]: pgmap v530: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:53:02.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:53:02 smithi062 ceph-mon[76322]: pgmap v531: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:53:02.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:53:02 smithi189 ceph-mon[57022]: pgmap v531: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:53:04.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:53:04 smithi062 ceph-mon[76322]: pgmap v532: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:53:04.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:53:04 smithi189 ceph-mon[57022]: pgmap v532: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:53:05.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:53:05 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:53:05.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:53:05 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:53:06.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:53:06 smithi189 ceph-mon[57022]: pgmap v533: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:53:06.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:53:06 smithi062 ceph-mon[76322]: pgmap v533: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:53:08.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:53:08 smithi189 ceph-mon[57022]: pgmap v534: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:53:08.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:53:08 smithi062 ceph-mon[76322]: pgmap v534: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:53:10.323 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:53:10 smithi062 ceph-mon[76322]: pgmap v535: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:53:10.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:53:10 smithi189 ceph-mon[57022]: pgmap v535: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:53:12.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:53:12 smithi189 ceph-mon[57022]: pgmap v536: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:53:12.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:53:12 smithi062 ceph-mon[76322]: pgmap v536: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:53:14.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:53:14 smithi189 ceph-mon[57022]: pgmap v537: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:53:14.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:53:14 smithi062 ceph-mon[76322]: pgmap v537: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:53:16.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:53:16 smithi189 ceph-mon[57022]: pgmap v538: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:53:16.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:53:16 smithi062 ceph-mon[76322]: pgmap v538: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:53:17.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:53:17 smithi189 ceph-mon[57022]: pgmap v539: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:53:17.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:53:17 smithi062 ceph-mon[76322]: pgmap v539: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:53:19.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:53:19 smithi189 ceph-mon[57022]: pgmap v540: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:53:19.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:53:19 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:53:20.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:53:19 smithi062 ceph-mon[76322]: pgmap v540: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:53:20.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:53:19 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:53:21.976 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:53:21 smithi189 ceph-mon[57022]: pgmap v541: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:53:22.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:53:21 smithi062 ceph-mon[76322]: pgmap v541: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:53:23.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:53:23 smithi189 ceph-mon[57022]: pgmap v542: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:53:24.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:53:23 smithi062 ceph-mon[76322]: pgmap v542: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:53:25.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:25.857+0000 7fb517e00700 1 -- 172.21.15.62:0/400905360 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb5181002d0 msgr2=0x7fb5181006a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:53:25.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:25.857+0000 7fb517e00700 1 --2- 172.21.15.62:0/400905360 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb5181002d0 0x7fb5181006a0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fb508009910 tx=0x7fb508009c20 comp rx=0 tx=0).stop 2024-04-20T21:53:25.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:25.857+0000 7fb517e00700 1 -- 172.21.15.62:0/400905360 shutdown_connections 2024-04-20T21:53:25.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:25.857+0000 7fb517e00700 1 --2- 172.21.15.62:0/400905360 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb518100c70 0x7fb518104c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:53:25.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:25.857+0000 7fb517e00700 1 --2- 172.21.15.62:0/400905360 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb5181002d0 0x7fb5181006a0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:53:25.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:25.857+0000 7fb517e00700 1 -- 172.21.15.62:0/400905360 >> 172.21.15.62:0/400905360 conn(0x7fb5180fbbb0 msgr2=0x7fb5180fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:53:25.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:25.857+0000 7fb517e00700 1 -- 172.21.15.62:0/400905360 shutdown_connections 2024-04-20T21:53:25.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:25.857+0000 7fb517e00700 1 -- 172.21.15.62:0/400905360 wait complete. 2024-04-20T21:53:25.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:25.858+0000 7fb517e00700 1 Processor -- start 2024-04-20T21:53:25.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:25.859+0000 7fb517e00700 1 -- start start 2024-04-20T21:53:25.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:25.859+0000 7fb517e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb5181002d0 0x7fb518195d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:53:25.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:25.859+0000 7fb517e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb518100c70 0x7fb5181962a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:53:25.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:25.859+0000 7fb517e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb518196980 con 0x7fb518100c70 2024-04-20T21:53:25.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:25.859+0000 7fb517e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb51819a710 con 0x7fb5181002d0 2024-04-20T21:53:25.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:25.859+0000 7fb50fe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb518100c70 0x7fb5181962a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:53:25.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:25.860+0000 7fb50fe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb518100c70 0x7fb5181962a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34660/0 (socket says 172.21.15.62:34660) 2024-04-20T21:53:25.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:25.860+0000 7fb50fe00700 1 -- 172.21.15.62:0/2054949580 learned_addr learned my addr 172.21.15.62:0/2054949580 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:53:25.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:25.860+0000 7fb516a00700 1 --2- 172.21.15.62:0/2054949580 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb5181002d0 0x7fb518195d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:53:25.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:25.860+0000 7fb50fe00700 1 -- 172.21.15.62:0/2054949580 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb5181002d0 msgr2=0x7fb518195d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:53:25.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:25.860+0000 7fb50fe00700 1 --2- 172.21.15.62:0/2054949580 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb5181002d0 0x7fb518195d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:53:25.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:25.860+0000 7fb50fe00700 1 -- 172.21.15.62:0/2054949580 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5080095f0 con 0x7fb518100c70 2024-04-20T21:53:25.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:25.860+0000 7fb50fe00700 1 --2- 172.21.15.62:0/2054949580 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb518100c70 0x7fb5181962a0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fb50000b480 tx=0x7fb50000b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:53:25.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:25.861+0000 7fb50f400700 1 -- 172.21.15.62:0/2054949580 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb5000118c0 con 0x7fb518100c70 2024-04-20T21:53:25.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:25.861+0000 7fb50f400700 1 -- 172.21.15.62:0/2054949580 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb500010380 con 0x7fb518100c70 2024-04-20T21:53:25.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:25.861+0000 7fb50f400700 1 -- 172.21.15.62:0/2054949580 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb50000f4f0 con 0x7fb518100c70 2024-04-20T21:53:25.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:25.861+0000 7fb517e00700 1 -- 172.21.15.62:0/2054949580 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb51819a9f0 con 0x7fb518100c70 2024-04-20T21:53:25.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:25.861+0000 7fb517e00700 1 -- 172.21.15.62:0/2054949580 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb51819af40 con 0x7fb518100c70 2024-04-20T21:53:25.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:25.862+0000 7fb517e00700 1 -- 172.21.15.62:0/2054949580 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5181085e0 con 0x7fb518100c70 2024-04-20T21:53:25.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:25.863+0000 7fb50f400700 1 -- 172.21.15.62:0/2054949580 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fb5000104f0 con 0x7fb518100c70 2024-04-20T21:53:25.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:25.864+0000 7fb50f400700 1 --2- 172.21.15.62:0/2054949580 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb4fc06ecc0 0x7fb4fc071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:53:25.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:25.864+0000 7fb50f400700 1 -- 172.21.15.62:0/2054949580 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fb50008ea70 con 0x7fb518100c70 2024-04-20T21:53:25.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:25.865+0000 7fb516a00700 1 --2- 172.21.15.62:0/2054949580 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb4fc06ecc0 0x7fb4fc071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:53:25.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:25.866+0000 7fb516a00700 1 --2- 172.21.15.62:0/2054949580 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb4fc06ecc0 0x7fb4fc071170 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fb508006010 tx=0x7fb50801a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:53:25.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:25.871+0000 7fb50f400700 1 -- 172.21.15.62:0/2054949580 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fb500057800 con 0x7fb518100c70 2024-04-20T21:53:25.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:53:25 smithi189 ceph-mon[57022]: pgmap v543: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:53:26.034 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:53:25 smithi062 ceph-mon[76322]: pgmap v543: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:53:26.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:26.130+0000 7fb517e00700 1 -- 172.21.15.62:0/2054949580 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb51819b280 con 0x7fb4fc06ecc0 2024-04-20T21:53:26.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:26.133+0000 7fb50f400700 1 -- 172.21.15.62:0/2054949580 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fb51819b280 con 0x7fb4fc06ecc0 2024-04-20T21:53:26.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:26.136+0000 7fb517e00700 1 -- 172.21.15.62:0/2054949580 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb4fc06ecc0 msgr2=0x7fb4fc071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:53:26.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:26.136+0000 7fb517e00700 1 --2- 172.21.15.62:0/2054949580 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb4fc06ecc0 0x7fb4fc071170 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fb508006010 tx=0x7fb50801a040 comp rx=0 tx=0).stop 2024-04-20T21:53:26.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:26.136+0000 7fb517e00700 1 -- 172.21.15.62:0/2054949580 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb518100c70 msgr2=0x7fb5181962a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:53:26.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:26.136+0000 7fb517e00700 1 --2- 172.21.15.62:0/2054949580 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb518100c70 0x7fb5181962a0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fb50000b480 tx=0x7fb50000b840 comp rx=0 tx=0).stop 2024-04-20T21:53:26.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:26.136+0000 7fb517e00700 1 -- 172.21.15.62:0/2054949580 shutdown_connections 2024-04-20T21:53:26.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:26.136+0000 7fb517e00700 1 --2- 172.21.15.62:0/2054949580 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb4fc06ecc0 0x7fb4fc071170 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:53:26.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:26.136+0000 7fb517e00700 1 --2- 172.21.15.62:0/2054949580 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb5181002d0 0x7fb518195d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:53:26.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:26.136+0000 7fb517e00700 1 --2- 172.21.15.62:0/2054949580 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb518100c70 0x7fb5181962a0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:53:26.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:26.136+0000 7fb517e00700 1 -- 172.21.15.62:0/2054949580 >> 172.21.15.62:0/2054949580 conn(0x7fb5180fbbb0 msgr2=0x7fb5180fd290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:53:26.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:26.136+0000 7fb517e00700 1 -- 172.21.15.62:0/2054949580 shutdown_connections 2024-04-20T21:53:26.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:26.136+0000 7fb517e00700 1 -- 172.21.15.62:0/2054949580 wait complete. 2024-04-20T21:53:26.152 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T21:53:26.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:26.304+0000 7f6c78000700 1 -- 172.21.15.62:0/3336184599 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c70106320 msgr2=0x7f6c701066f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:53:26.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:26.304+0000 7f6c78000700 1 --2- 172.21.15.62:0/3336184599 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c70106320 0x7f6c701066f0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f6c64009910 tx=0x7f6c64009c20 comp rx=0 tx=0).stop 2024-04-20T21:53:26.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:26.304+0000 7f6c78000700 1 -- 172.21.15.62:0/3336184599 shutdown_connections 2024-04-20T21:53:26.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:26.304+0000 7f6c78000700 1 --2- 172.21.15.62:0/3336184599 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c70068540 0x7f6c700689b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:53:26.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:26.304+0000 7f6c78000700 1 --2- 172.21.15.62:0/3336184599 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c70106320 0x7f6c701066f0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:53:26.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:26.304+0000 7f6c78000700 1 -- 172.21.15.62:0/3336184599 >> 172.21.15.62:0/3336184599 conn(0x7f6c70075130 msgr2=0x7f6c70075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:53:26.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:26.304+0000 7f6c78000700 1 -- 172.21.15.62:0/3336184599 shutdown_connections 2024-04-20T21:53:26.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:26.304+0000 7f6c78000700 1 -- 172.21.15.62:0/3336184599 wait complete. 2024-04-20T21:53:26.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:26.305+0000 7f6c78000700 1 Processor -- start 2024-04-20T21:53:26.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:26.306+0000 7f6c78000700 1 -- start start 2024-04-20T21:53:26.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:26.306+0000 7f6c78000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c70068540 0x7f6c70195e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:53:26.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:26.306+0000 7f6c78000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c70106320 0x7f6c701963b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:53:26.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:26.306+0000 7f6c78000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c70196a90 con 0x7f6c70068540 2024-04-20T21:53:26.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:26.306+0000 7f6c78000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c7019a820 con 0x7f6c70106320 2024-04-20T21:53:26.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:26.306+0000 7f6c75600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c70068540 0x7f6c70195e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:53:26.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:26.306+0000 7f6c75600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c70068540 0x7f6c70195e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34672/0 (socket says 172.21.15.62:34672) 2024-04-20T21:53:26.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:26.306+0000 7f6c75600700 1 -- 172.21.15.62:0/3259545567 learned_addr learned my addr 172.21.15.62:0/3259545567 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:53:26.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:26.307+0000 7f6c74c00700 1 --2- 172.21.15.62:0/3259545567 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c70106320 0x7f6c701963b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:53:26.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:26.307+0000 7f6c75600700 1 -- 172.21.15.62:0/3259545567 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c70106320 msgr2=0x7f6c701963b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:53:26.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:26.307+0000 7f6c75600700 1 --2- 172.21.15.62:0/3259545567 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c70106320 0x7f6c701963b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:53:26.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:26.307+0000 7f6c75600700 1 -- 172.21.15.62:0/3259545567 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c640095f0 con 0x7f6c70068540 2024-04-20T21:53:26.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:26.307+0000 7f6c75600700 1 --2- 172.21.15.62:0/3259545567 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c70068540 0x7f6c70195e70 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f6c640092e0 tx=0x7f6c6400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:53:26.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:26.308+0000 7f6c6e000700 1 -- 172.21.15.62:0/3259545567 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6c6401d070 con 0x7f6c70068540 2024-04-20T21:53:26.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:26.308+0000 7f6c6e000700 1 -- 172.21.15.62:0/3259545567 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6c64005e00 con 0x7f6c70068540 2024-04-20T21:53:26.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:26.308+0000 7f6c6e000700 1 -- 172.21.15.62:0/3259545567 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6c64017400 con 0x7f6c70068540 2024-04-20T21:53:26.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:26.308+0000 7f6c78000700 1 -- 172.21.15.62:0/3259545567 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c7019aaa0 con 0x7f6c70068540 2024-04-20T21:53:26.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:26.308+0000 7f6c78000700 1 -- 172.21.15.62:0/3259545567 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c7019af90 con 0x7f6c70068540 2024-04-20T21:53:26.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:26.309+0000 7f6c78000700 1 -- 172.21.15.62:0/3259545567 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c70108790 con 0x7f6c70068540 2024-04-20T21:53:26.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:26.310+0000 7f6c6e000700 1 -- 172.21.15.62:0/3259545567 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f6c6400f460 con 0x7f6c70068540 2024-04-20T21:53:26.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:26.311+0000 7f6c6e000700 1 --2- 172.21.15.62:0/3259545567 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6c5806ecc0 0x7f6c58071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:53:26.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:26.311+0000 7f6c6e000700 1 -- 172.21.15.62:0/3259545567 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f6c64090330 con 0x7f6c70068540 2024-04-20T21:53:26.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:26.311+0000 7f6c74c00700 1 --2- 172.21.15.62:0/3259545567 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6c5806ecc0 0x7f6c58071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:53:26.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:26.312+0000 7f6c74c00700 1 --2- 172.21.15.62:0/3259545567 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6c5806ecc0 0x7f6c58071170 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f6c5c005950 tx=0x7f6c5c0058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:53:26.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:26.316+0000 7f6c6e000700 1 -- 172.21.15.62:0/3259545567 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f6c640590c0 con 0x7f6c70068540 2024-04-20T21:53:26.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:26.579+0000 7f6c78000700 1 -- 172.21.15.62:0/3259545567 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6c7019b200 con 0x7f6c5806ecc0 2024-04-20T21:53:26.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:26.582+0000 7f6c6e000700 1 -- 172.21.15.62:0/3259545567 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f6c7019b200 con 0x7f6c5806ecc0 2024-04-20T21:53:26.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:26.585+0000 7f6c78000700 1 -- 172.21.15.62:0/3259545567 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6c5806ecc0 msgr2=0x7f6c58071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:53:26.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:26.585+0000 7f6c78000700 1 --2- 172.21.15.62:0/3259545567 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6c5806ecc0 0x7f6c58071170 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f6c5c005950 tx=0x7f6c5c0058e0 comp rx=0 tx=0).stop 2024-04-20T21:53:26.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:26.585+0000 7f6c78000700 1 -- 172.21.15.62:0/3259545567 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c70068540 msgr2=0x7f6c70195e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:53:26.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:26.585+0000 7f6c78000700 1 --2- 172.21.15.62:0/3259545567 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c70068540 0x7f6c70195e70 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f6c640092e0 tx=0x7f6c6400be20 comp rx=0 tx=0).stop 2024-04-20T21:53:26.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:26.585+0000 7f6c78000700 1 -- 172.21.15.62:0/3259545567 shutdown_connections 2024-04-20T21:53:26.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:26.585+0000 7f6c78000700 1 --2- 172.21.15.62:0/3259545567 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6c5806ecc0 0x7f6c58071170 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:53:26.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:26.585+0000 7f6c78000700 1 --2- 172.21.15.62:0/3259545567 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c70068540 0x7f6c70195e70 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:53:26.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:26.585+0000 7f6c78000700 1 --2- 172.21.15.62:0/3259545567 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c70106320 0x7f6c701963b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:53:26.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:26.585+0000 7f6c78000700 1 -- 172.21.15.62:0/3259545567 >> 172.21.15.62:0/3259545567 conn(0x7f6c70075130 msgr2=0x7f6c700fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:53:26.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:26.585+0000 7f6c78000700 1 -- 172.21.15.62:0/3259545567 shutdown_connections 2024-04-20T21:53:26.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:26.585+0000 7f6c78000700 1 -- 172.21.15.62:0/3259545567 wait complete. 2024-04-20T21:53:26.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:26.753+0000 7fe3fd000700 1 -- 172.21.15.62:0/236505267 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe3f81004e0 msgr2=0x7fe3f81008b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:53:26.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:26.753+0000 7fe3fd000700 1 --2- 172.21.15.62:0/236505267 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe3f81004e0 0x7fe3f81008b0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fe3e8009910 tx=0x7fe3e8009c20 comp rx=0 tx=0).stop 2024-04-20T21:53:26.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:26.753+0000 7fe3fd000700 1 -- 172.21.15.62:0/236505267 shutdown_connections 2024-04-20T21:53:26.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:26.753+0000 7fe3fd000700 1 --2- 172.21.15.62:0/236505267 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3f8100e80 0x7fe3f8104e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:53:26.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:26.753+0000 7fe3fd000700 1 --2- 172.21.15.62:0/236505267 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe3f81004e0 0x7fe3f81008b0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:53:26.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:26.753+0000 7fe3fd000700 1 -- 172.21.15.62:0/236505267 >> 172.21.15.62:0/236505267 conn(0x7fe3f80fbda0 msgr2=0x7fe3f80fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:53:26.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:26.753+0000 7fe3fd000700 1 -- 172.21.15.62:0/236505267 shutdown_connections 2024-04-20T21:53:26.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:26.753+0000 7fe3fd000700 1 -- 172.21.15.62:0/236505267 wait complete. 2024-04-20T21:53:26.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:26.755+0000 7fe3fd000700 1 Processor -- start 2024-04-20T21:53:26.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:26.755+0000 7fe3fd000700 1 -- start start 2024-04-20T21:53:26.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:26.755+0000 7fe3fd000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3f81004e0 0x7fe3f8195ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:53:26.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:26.755+0000 7fe3fd000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe3f8100e80 0x7fe3f8196430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:53:26.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:26.755+0000 7fe3fd000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3f8196b10 con 0x7fe3f8100e80 2024-04-20T21:53:26.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:26.755+0000 7fe3fd000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3f819a8a0 con 0x7fe3f81004e0 2024-04-20T21:53:26.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:26.756+0000 7fe3f5400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe3f8100e80 0x7fe3f8196430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:53:26.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:26.756+0000 7fe3f5400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe3f8100e80 0x7fe3f8196430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34692/0 (socket says 172.21.15.62:34692) 2024-04-20T21:53:26.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:26.756+0000 7fe3f5400700 1 -- 172.21.15.62:0/1394765232 learned_addr learned my addr 172.21.15.62:0/1394765232 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:53:26.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:26.756+0000 7fe3f5e00700 1 --2- 172.21.15.62:0/1394765232 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3f81004e0 0x7fe3f8195ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:53:26.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:26.756+0000 7fe3f5400700 1 -- 172.21.15.62:0/1394765232 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3f81004e0 msgr2=0x7fe3f8195ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:53:26.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:26.756+0000 7fe3f5400700 1 --2- 172.21.15.62:0/1394765232 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3f81004e0 0x7fe3f8195ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:53:26.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:26.756+0000 7fe3f5400700 1 -- 172.21.15.62:0/1394765232 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3e80095f0 con 0x7fe3f8100e80 2024-04-20T21:53:26.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:26.757+0000 7fe3f5400700 1 --2- 172.21.15.62:0/1394765232 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe3f8100e80 0x7fe3f8196430 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fe3e400c8f0 tx=0x7fe3e400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:53:26.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:26.757+0000 7fe3eea00700 1 -- 172.21.15.62:0/1394765232 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe3e4007a50 con 0x7fe3f8100e80 2024-04-20T21:53:26.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:26.757+0000 7fe3eea00700 1 -- 172.21.15.62:0/1394765232 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe3e4007bb0 con 0x7fe3f8100e80 2024-04-20T21:53:26.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:26.757+0000 7fe3fd000700 1 -- 172.21.15.62:0/1394765232 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3f819ab80 con 0x7fe3f8100e80 2024-04-20T21:53:26.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:26.757+0000 7fe3eea00700 1 -- 172.21.15.62:0/1394765232 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe3e4018630 con 0x7fe3f8100e80 2024-04-20T21:53:26.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:26.757+0000 7fe3fd000700 1 -- 172.21.15.62:0/1394765232 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3f819b0d0 con 0x7fe3f8100e80 2024-04-20T21:53:26.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:26.758+0000 7fe3fd000700 1 -- 172.21.15.62:0/1394765232 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3f81087f0 con 0x7fe3f8100e80 2024-04-20T21:53:26.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:26.759+0000 7fe3eea00700 1 -- 172.21.15.62:0/1394765232 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe3e4007d20 con 0x7fe3f8100e80 2024-04-20T21:53:26.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:26.760+0000 7fe3eea00700 1 --2- 172.21.15.62:0/1394765232 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe3dc06ecc0 0x7fe3dc071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:53:26.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:26.760+0000 7fe3eea00700 1 -- 172.21.15.62:0/1394765232 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fe3e408ee00 con 0x7fe3f8100e80 2024-04-20T21:53:26.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:26.761+0000 7fe3f5e00700 1 --2- 172.21.15.62:0/1394765232 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe3dc06ecc0 0x7fe3dc071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:53:26.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:26.761+0000 7fe3f5e00700 1 --2- 172.21.15.62:0/1394765232 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe3dc06ecc0 0x7fe3dc071170 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fe3e8006010 tx=0x7fe3e801a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:53:26.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:26.765+0000 7fe3eea00700 1 -- 172.21.15.62:0/1394765232 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fe3e4057b90 con 0x7fe3f8100e80 2024-04-20T21:53:27.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.013+0000 7fe3fd000700 1 -- 172.21.15.62:0/1394765232 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fe3f819b3a0 con 0x7fe3dc06ecc0 2024-04-20T21:53:27.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.033+0000 7fe3eea00700 1 -- 172.21.15.62:0/1394765232 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fe3f819b3a0 con 0x7fe3dc06ecc0 2024-04-20T21:53:27.033 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T21:53:27.033 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (19m) 10m ago 33m 14.2M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T21:53:27.033 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (33m) 10m ago 33m 9512k - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T21:53:27.033 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (32m) 11m ago 32m 8942k - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T21:53:27.033 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (15m) 10m ago 33m 6660k - 19.0.0-3133-ga8c84f59 76bc6b91b726 2825d5ba11c6 2024-04-20T21:53:27.033 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (15m) 11m ago 32m 6665k - 19.0.0-3133-ga8c84f59 76bc6b91b726 20f746fa1bba 2024-04-20T21:53:27.033 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (18m) 10m ago 32m 74.2M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T21:53:27.033 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (10m) 10m ago 25m 11.2M - 19.0.0-3133-ga8c84f59 76bc6b91b726 de80bc90a683 2024-04-20T21:53:27.033 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (11m) 10m ago 26m 11.9M - 19.0.0-3133-ga8c84f59 76bc6b91b726 cd793b7ee98e 2024-04-20T21:53:27.033 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (25m) 11m ago 25m 117M - 18.2.0 dc2bc1663786 fb2a8a223db8 2024-04-20T21:53:27.033 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (25m) 11m ago 25m 55.6M - 18.2.0 dc2bc1663786 f2eec2ee8951 2024-04-20T21:53:27.034 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (22m) 10m ago 34m 511M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T21:53:27.034 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (20m) 11m ago 31m 452M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T21:53:27.034 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (17m) 10m ago 35m 55.9M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 019eb1479485 2024-04-20T21:53:27.034 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (16m) 11m ago 31m 39.8M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 55aac560aab6 2024-04-20T21:53:27.034 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (20m) 10m ago 33m 7952k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T21:53:27.034 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (20m) 11m ago 31m 8304k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T21:53:27.034 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (14m) 10m ago 30m 88.0M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 ee2bb901f26e 2024-04-20T21:53:27.034 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (14m) 10m ago 30m 92.3M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 9075152d4162 2024-04-20T21:53:27.034 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (13m) 10m ago 29m 74.4M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 2d4f3e38dc0e 2024-04-20T21:53:27.034 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (13m) 11m ago 28m 87.6M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 8f97b17ae0eb 2024-04-20T21:53:27.034 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (12m) 11m ago 28m 90.7M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 653d8079967d 2024-04-20T21:53:27.034 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (11m) 11m ago 27m 11.5M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 596dded14c12 2024-04-20T21:53:27.034 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (19m) 10m ago 32m 74.0M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T21:53:27.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.036+0000 7fe3fd000700 1 -- 172.21.15.62:0/1394765232 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe3dc06ecc0 msgr2=0x7fe3dc071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:53:27.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.036+0000 7fe3fd000700 1 --2- 172.21.15.62:0/1394765232 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe3dc06ecc0 0x7fe3dc071170 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fe3e8006010 tx=0x7fe3e801a040 comp rx=0 tx=0).stop 2024-04-20T21:53:27.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.036+0000 7fe3fd000700 1 -- 172.21.15.62:0/1394765232 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe3f8100e80 msgr2=0x7fe3f8196430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:53:27.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.036+0000 7fe3fd000700 1 --2- 172.21.15.62:0/1394765232 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe3f8100e80 0x7fe3f8196430 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fe3e400c8f0 tx=0x7fe3e400ccb0 comp rx=0 tx=0).stop 2024-04-20T21:53:27.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.036+0000 7fe3fd000700 1 -- 172.21.15.62:0/1394765232 shutdown_connections 2024-04-20T21:53:27.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.036+0000 7fe3fd000700 1 --2- 172.21.15.62:0/1394765232 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe3dc06ecc0 0x7fe3dc071170 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:53:27.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.036+0000 7fe3fd000700 1 --2- 172.21.15.62:0/1394765232 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3f81004e0 0x7fe3f8195ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:53:27.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.036+0000 7fe3fd000700 1 --2- 172.21.15.62:0/1394765232 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe3f8100e80 0x7fe3f8196430 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:53:27.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.036+0000 7fe3fd000700 1 -- 172.21.15.62:0/1394765232 >> 172.21.15.62:0/1394765232 conn(0x7fe3f80fbda0 msgr2=0x7fe3f80fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:53:27.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.036+0000 7fe3fd000700 1 -- 172.21.15.62:0/1394765232 shutdown_connections 2024-04-20T21:53:27.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.036+0000 7fe3fd000700 1 -- 172.21.15.62:0/1394765232 wait complete. 2024-04-20T21:53:27.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.193+0000 7f6bda800700 1 -- 172.21.15.62:0/1119418999 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6bd4106b80 msgr2=0x7f6bd4106ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:53:27.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.193+0000 7f6bda800700 1 --2- 172.21.15.62:0/1119418999 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6bd4106b80 0x7f6bd4106ff0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f6bc0009910 tx=0x7f6bc0009c20 comp rx=0 tx=0).stop 2024-04-20T21:53:27.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.193+0000 7f6bda800700 1 -- 172.21.15.62:0/1119418999 shutdown_connections 2024-04-20T21:53:27.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.193+0000 7f6bda800700 1 --2- 172.21.15.62:0/1119418999 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6bd4106b80 0x7f6bd4106ff0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:53:27.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.193+0000 7f6bda800700 1 --2- 172.21.15.62:0/1119418999 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6bd410c660 0x7f6bd410ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:53:27.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.193+0000 7f6bda800700 1 -- 172.21.15.62:0/1119418999 >> 172.21.15.62:0/1119418999 conn(0x7f6bd4075e90 msgr2=0x7f6bd40782a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:53:27.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.194+0000 7f6bda800700 1 -- 172.21.15.62:0/1119418999 shutdown_connections 2024-04-20T21:53:27.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.194+0000 7f6bda800700 1 -- 172.21.15.62:0/1119418999 wait complete. 2024-04-20T21:53:27.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.195+0000 7f6bda800700 1 Processor -- start 2024-04-20T21:53:27.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.195+0000 7f6bda800700 1 -- start start 2024-04-20T21:53:27.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.195+0000 7f6bda800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6bd4106b80 0x7f6bd419c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:53:27.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.195+0000 7f6bda800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6bd410c660 0x7f6bd419c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:53:27.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.195+0000 7f6bda800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bd419ce00 con 0x7f6bd410c660 2024-04-20T21:53:27.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.195+0000 7f6bda800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bd41a0b90 con 0x7f6bd4106b80 2024-04-20T21:53:27.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.195+0000 7f6bd8a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6bd410c660 0x7f6bd419c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:53:27.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.196+0000 7f6bd8a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6bd410c660 0x7f6bd419c720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34712/0 (socket says 172.21.15.62:34712) 2024-04-20T21:53:27.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.196+0000 7f6bd8a00700 1 -- 172.21.15.62:0/1835203796 learned_addr learned my addr 172.21.15.62:0/1835203796 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:53:27.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.196+0000 7f6bd9400700 1 --2- 172.21.15.62:0/1835203796 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6bd4106b80 0x7f6bd419c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:53:27.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.196+0000 7f6bd8a00700 1 -- 172.21.15.62:0/1835203796 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6bd4106b80 msgr2=0x7f6bd419c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:53:27.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.196+0000 7f6bd8a00700 1 --2- 172.21.15.62:0/1835203796 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6bd4106b80 0x7f6bd419c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:53:27.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.196+0000 7f6bd8a00700 1 -- 172.21.15.62:0/1835203796 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bc00095f0 con 0x7f6bd410c660 2024-04-20T21:53:27.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.196+0000 7f6bd8a00700 1 --2- 172.21.15.62:0/1835203796 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6bd410c660 0x7f6bd419c720 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f6bc0006010 tx=0x7f6bc000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:53:27.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.197+0000 7f6bd2000700 1 -- 172.21.15.62:0/1835203796 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6bc001d070 con 0x7f6bd410c660 2024-04-20T21:53:27.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.197+0000 7f6bd2000700 1 -- 172.21.15.62:0/1835203796 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6bc000f460 con 0x7f6bd410c660 2024-04-20T21:53:27.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.197+0000 7f6bd2000700 1 -- 172.21.15.62:0/1835203796 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6bc0017610 con 0x7f6bd410c660 2024-04-20T21:53:27.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.197+0000 7f6bda800700 1 -- 172.21.15.62:0/1835203796 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bd41a0e10 con 0x7f6bd410c660 2024-04-20T21:53:27.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.197+0000 7f6bda800700 1 -- 172.21.15.62:0/1835203796 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bd41a1300 con 0x7f6bd410c660 2024-04-20T21:53:27.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.198+0000 7f6bda800700 1 -- 172.21.15.62:0/1835203796 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bd410eb40 con 0x7f6bd410c660 2024-04-20T21:53:27.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.199+0000 7f6bd2000700 1 -- 172.21.15.62:0/1835203796 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f6bc000f5d0 con 0x7f6bd410c660 2024-04-20T21:53:27.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.200+0000 7f6bd2000700 1 --2- 172.21.15.62:0/1835203796 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6bbc06ecc0 0x7f6bbc071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:53:27.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.201+0000 7f6bd2000700 1 -- 172.21.15.62:0/1835203796 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f6bc0090330 con 0x7f6bd410c660 2024-04-20T21:53:27.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.201+0000 7f6bd9400700 1 --2- 172.21.15.62:0/1835203796 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6bbc06ecc0 0x7f6bbc071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:53:27.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.201+0000 7f6bd9400700 1 --2- 172.21.15.62:0/1835203796 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6bbc06ecc0 0x7f6bbc071170 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f6bc8005950 tx=0x7f6bc80058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:53:27.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.205+0000 7f6bd2000700 1 -- 172.21.15.62:0/1835203796 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f6bc0059180 con 0x7f6bd410c660 2024-04-20T21:53:27.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.535+0000 7f6bda800700 1 -- 172.21.15.62:0/1835203796 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f6bd41a15e0 con 0x7f6bd410c660 2024-04-20T21:53:27.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.536+0000 7f6bd2000700 1 -- 172.21.15.62:0/1835203796 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7f6bc005c840 con 0x7f6bd410c660 2024-04-20T21:53:27.536 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:53:27.536 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T21:53:27.536 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:53:27.536 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:53:27.536 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T21:53:27.536 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:53:27.536 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:53:27.536 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T21:53:27.537 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 6 2024-04-20T21:53:27.537 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:53:27.537 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T21:53:27.537 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 2, 2024-04-20T21:53:27.537 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:53:27.537 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:53:27.537 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T21:53:27.537 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 2, 2024-04-20T21:53:27.537 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 12 2024-04-20T21:53:27.537 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T21:53:27.537 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:53:27.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.539+0000 7f6bda800700 1 -- 172.21.15.62:0/1835203796 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6bbc06ecc0 msgr2=0x7f6bbc071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:53:27.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.539+0000 7f6bda800700 1 --2- 172.21.15.62:0/1835203796 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6bbc06ecc0 0x7f6bbc071170 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f6bc8005950 tx=0x7f6bc80058e0 comp rx=0 tx=0).stop 2024-04-20T21:53:27.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.539+0000 7f6bda800700 1 -- 172.21.15.62:0/1835203796 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6bd410c660 msgr2=0x7f6bd419c720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:53:27.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.539+0000 7f6bda800700 1 --2- 172.21.15.62:0/1835203796 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6bd410c660 0x7f6bd419c720 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f6bc0006010 tx=0x7f6bc000be20 comp rx=0 tx=0).stop 2024-04-20T21:53:27.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.539+0000 7f6bda800700 1 -- 172.21.15.62:0/1835203796 shutdown_connections 2024-04-20T21:53:27.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.539+0000 7f6bda800700 1 --2- 172.21.15.62:0/1835203796 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6bbc06ecc0 0x7f6bbc071170 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:53:27.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.539+0000 7f6bda800700 1 --2- 172.21.15.62:0/1835203796 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6bd4106b80 0x7f6bd419c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:53:27.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.539+0000 7f6bda800700 1 --2- 172.21.15.62:0/1835203796 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6bd410c660 0x7f6bd419c720 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:53:27.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.539+0000 7f6bda800700 1 -- 172.21.15.62:0/1835203796 >> 172.21.15.62:0/1835203796 conn(0x7f6bd4075e90 msgr2=0x7f6bd40778d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:53:27.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.540+0000 7f6bda800700 1 -- 172.21.15.62:0/1835203796 shutdown_connections 2024-04-20T21:53:27.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.540+0000 7f6bda800700 1 -- 172.21.15.62:0/1835203796 wait complete. 2024-04-20T21:53:27.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.699+0000 7f0616200700 1 -- 172.21.15.62:0/1668955106 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f06101061b0 msgr2=0x7f0610106580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:53:27.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.699+0000 7f0616200700 1 --2- 172.21.15.62:0/1668955106 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f06101061b0 0x7f0610106580 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f0604009910 tx=0x7f0604009c20 comp rx=0 tx=0).stop 2024-04-20T21:53:27.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.699+0000 7f0616200700 1 -- 172.21.15.62:0/1668955106 shutdown_connections 2024-04-20T21:53:27.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.699+0000 7f0616200700 1 --2- 172.21.15.62:0/1668955106 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0610068540 0x7f06100689b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:53:27.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.699+0000 7f0616200700 1 --2- 172.21.15.62:0/1668955106 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f06101061b0 0x7f0610106580 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:53:27.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.699+0000 7f0616200700 1 -- 172.21.15.62:0/1668955106 >> 172.21.15.62:0/1668955106 conn(0x7f0610074fa0 msgr2=0x7f06100753a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:53:27.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.699+0000 7f0616200700 1 -- 172.21.15.62:0/1668955106 shutdown_connections 2024-04-20T21:53:27.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.699+0000 7f0616200700 1 -- 172.21.15.62:0/1668955106 wait complete. 2024-04-20T21:53:27.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.700+0000 7f0616200700 1 Processor -- start 2024-04-20T21:53:27.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.700+0000 7f0616200700 1 -- start start 2024-04-20T21:53:27.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.701+0000 7f0616200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0610068540 0x7f0610197db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:53:27.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.701+0000 7f0616200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f06101061b0 0x7f06101982f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:53:27.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.701+0000 7f0616200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06101989d0 con 0x7f06101061b0 2024-04-20T21:53:27.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.701+0000 7f0616200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f061019c760 con 0x7f0610068540 2024-04-20T21:53:27.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.701+0000 7f060fe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f06101061b0 0x7f06101982f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:53:27.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.701+0000 7f060fe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f06101061b0 0x7f06101982f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34726/0 (socket says 172.21.15.62:34726) 2024-04-20T21:53:27.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.701+0000 7f060fe00700 1 -- 172.21.15.62:0/2189791919 learned_addr learned my addr 172.21.15.62:0/2189791919 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:53:27.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.701+0000 7f0614e00700 1 --2- 172.21.15.62:0/2189791919 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0610068540 0x7f0610197db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:53:27.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.702+0000 7f060fe00700 1 -- 172.21.15.62:0/2189791919 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0610068540 msgr2=0x7f0610197db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:53:27.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.702+0000 7f060fe00700 1 --2- 172.21.15.62:0/2189791919 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0610068540 0x7f0610197db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:53:27.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.702+0000 7f060fe00700 1 -- 172.21.15.62:0/2189791919 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06040095f0 con 0x7f06101061b0 2024-04-20T21:53:27.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.702+0000 7f060fe00700 1 --2- 172.21.15.62:0/2189791919 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f06101061b0 0x7f06101982f0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f05fc00d900 tx=0x7f05fc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:53:27.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.702+0000 7f060d600700 1 -- 172.21.15.62:0/2189791919 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f05fc004a00 con 0x7f06101061b0 2024-04-20T21:53:27.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.702+0000 7f060d600700 1 -- 172.21.15.62:0/2189791919 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f05fc0054e0 con 0x7f06101061b0 2024-04-20T21:53:27.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.702+0000 7f0616200700 1 -- 172.21.15.62:0/2189791919 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f061019ca40 con 0x7f06101061b0 2024-04-20T21:53:27.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.702+0000 7f060d600700 1 -- 172.21.15.62:0/2189791919 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f05fc009d60 con 0x7f06101061b0 2024-04-20T21:53:27.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.702+0000 7f0616200700 1 -- 172.21.15.62:0/2189791919 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f061019d010 con 0x7f06101061b0 2024-04-20T21:53:27.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.703+0000 7f0616200700 1 -- 172.21.15.62:0/2189791919 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0610062380 con 0x7f06101061b0 2024-04-20T21:53:27.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.704+0000 7f060d600700 1 -- 172.21.15.62:0/2189791919 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f05fc005020 con 0x7f06101061b0 2024-04-20T21:53:27.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.705+0000 7f060d600700 1 --2- 172.21.15.62:0/2189791919 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f05f806ecc0 0x7f05f8071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:53:27.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.706+0000 7f060d600700 1 -- 172.21.15.62:0/2189791919 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f05fc08efe0 con 0x7f06101061b0 2024-04-20T21:53:27.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.706+0000 7f0614e00700 1 --2- 172.21.15.62:0/2189791919 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f05f806ecc0 0x7f05f8071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:53:27.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.707+0000 7f0614e00700 1 --2- 172.21.15.62:0/2189791919 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f05f806ecc0 0x7f05f8071170 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f0604006010 tx=0x7f060401a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:53:27.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.710+0000 7f060d600700 1 -- 172.21.15.62:0/2189791919 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f05fc057dd0 con 0x7f06101061b0 2024-04-20T21:53:27.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.989+0000 7f0616200700 1 -- 172.21.15.62:0/2189791919 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f061019d380 con 0x7f06101061b0 2024-04-20T21:53:27.989 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:53:27 smithi062 ceph-mon[76322]: from='client.35012 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:53:27.989 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:53:27 smithi062 ceph-mon[76322]: from='client.35016 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:53:27.989 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:53:27 smithi062 ceph-mon[76322]: pgmap v544: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:53:27.990 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:53:27 smithi062 ceph-mon[76322]: from='client.35020 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:53:27.990 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:53:27 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/1835203796' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:53:27.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.990+0000 7f060d600700 1 -- 172.21.15.62:0/2189791919 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 25 v25) v1 ==== 76+0+1887 (secure 0 0 0) 0x7f05fc017070 con 0x7f06101061b0 2024-04-20T21:53:27.990 INFO:teuthology.orchestra.run.smithi062.stdout:e25 2024-04-20T21:53:27.990 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T21:53:27.990 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T21:53:27.990 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T21:53:27.991 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:53:27.991 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T21:53:27.991 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T21:53:27.991 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 20 2024-04-20T21:53:27.991 INFO:teuthology.orchestra.run.smithi062.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-20T21:53:27.991 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T21:53:27.991 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T21:42:01.559061+0000 2024-04-20T21:53:27.991 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T21:53:27.991 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T21:53:27.991 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T21:53:27.991 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T21:53:27.991 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T21:53:27.991 INFO:teuthology.orchestra.run.smithi062.stdout:max_xattr_size 65536 2024-04-20T21:53:27.991 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T21:53:27.991 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T21:53:27.992 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 81 2024-04-20T21:53:27.992 INFO:teuthology.orchestra.run.smithi062.stdout:snap_rstat 1 2024-04-20T21:53:27.992 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T21:53:27.992 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 2 2024-04-20T21:53:27.992 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1 2024-04-20T21:53:27.992 INFO:teuthology.orchestra.run.smithi062.stdout:up {} 2024-04-20T21:53:27.992 INFO:teuthology.orchestra.run.smithi062.stdout:failed 0,1 2024-04-20T21:53:27.992 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T21:53:27.992 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T21:53:27.992 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T21:53:27.992 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T21:53:27.992 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data enabled 2024-04-20T21:53:27.992 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T21:53:27.992 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T21:53:27.992 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 1 2024-04-20T21:53:27.993 INFO:teuthology.orchestra.run.smithi062.stdout:qdb_cluster leader: 0 members: 2024-04-20T21:53:27.993 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:53:27.993 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:53:27.993 INFO:teuthology.orchestra.run.smithi062.stdout:Standby daemons: 2024-04-20T21:53:27.993 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:53:27.993 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ryqblb{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6826/4113916205,v1:172.21.15.62:6827/4113916205] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T21:53:27.993 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ozizwi{-1:34476} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6828/3801975165,v1:172.21.15.62:6829/3801975165] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T21:53:27.993 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.xbgspw{-1:44285} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6826/3996040526,v1:172.21.15.189:6827/3996040526] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:53:27.993 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.dvimer{-1:44289} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6824/1743111843,v1:172.21.15.189:6825/1743111843] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:53:27.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.993+0000 7f0616200700 1 -- 172.21.15.62:0/2189791919 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f05f806ecc0 msgr2=0x7f05f8071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:53:27.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.993+0000 7f0616200700 1 --2- 172.21.15.62:0/2189791919 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f05f806ecc0 0x7f05f8071170 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f0604006010 tx=0x7f060401a040 comp rx=0 tx=0).stop 2024-04-20T21:53:27.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.993+0000 7f0616200700 1 -- 172.21.15.62:0/2189791919 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f06101061b0 msgr2=0x7f06101982f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:53:27.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.993+0000 7f0616200700 1 --2- 172.21.15.62:0/2189791919 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f06101061b0 0x7f06101982f0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f05fc00d900 tx=0x7f05fc00dcc0 comp rx=0 tx=0).stop 2024-04-20T21:53:27.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.993+0000 7f0616200700 1 -- 172.21.15.62:0/2189791919 shutdown_connections 2024-04-20T21:53:27.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.993+0000 7f0616200700 1 --2- 172.21.15.62:0/2189791919 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f05f806ecc0 0x7f05f8071170 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:53:27.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.993+0000 7f0616200700 1 --2- 172.21.15.62:0/2189791919 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0610068540 0x7f0610197db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:53:27.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.993+0000 7f0616200700 1 --2- 172.21.15.62:0/2189791919 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f06101061b0 0x7f06101982f0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:53:27.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.993+0000 7f0616200700 1 -- 172.21.15.62:0/2189791919 >> 172.21.15.62:0/2189791919 conn(0x7f0610074fa0 msgr2=0x7f06100fe710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:53:27.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.994+0000 7f0616200700 1 -- 172.21.15.62:0/2189791919 shutdown_connections 2024-04-20T21:53:27.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:27.994+0000 7f0616200700 1 -- 172.21.15.62:0/2189791919 wait complete. 2024-04-20T21:53:27.995 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 25 2024-04-20T21:53:28.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:28.151+0000 7fc6c8a00700 1 -- 172.21.15.62:0/3173781465 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc6c4102470 msgr2=0x7fc6c41028e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:53:28.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:28.151+0000 7fc6c8a00700 1 --2- 172.21.15.62:0/3173781465 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc6c4102470 0x7fc6c41028e0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fc6ac009910 tx=0x7fc6ac009c20 comp rx=0 tx=0).stop 2024-04-20T21:53:28.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:28.152+0000 7fc6c8a00700 1 -- 172.21.15.62:0/3173781465 shutdown_connections 2024-04-20T21:53:28.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:28.152+0000 7fc6c8a00700 1 --2- 172.21.15.62:0/3173781465 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc6c4102470 0x7fc6c41028e0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:53:28.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:28.152+0000 7fc6c8a00700 1 --2- 172.21.15.62:0/3173781465 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6c4108470 0x7fc6c4108840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:53:28.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:28.152+0000 7fc6c8a00700 1 -- 172.21.15.62:0/3173781465 >> 172.21.15.62:0/3173781465 conn(0x7fc6c40fdf90 msgr2=0x7fc6c41003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:53:28.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:28.152+0000 7fc6c8a00700 1 -- 172.21.15.62:0/3173781465 shutdown_connections 2024-04-20T21:53:28.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:28.152+0000 7fc6c8a00700 1 -- 172.21.15.62:0/3173781465 wait complete. 2024-04-20T21:53:28.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:28.153+0000 7fc6c8a00700 1 Processor -- start 2024-04-20T21:53:28.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:28.153+0000 7fc6c8a00700 1 -- start start 2024-04-20T21:53:28.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:28.154+0000 7fc6c8a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc6c4102470 0x7fc6c41981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:53:28.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:28.154+0000 7fc6c8a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6c4108470 0x7fc6c4198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:53:28.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:28.154+0000 7fc6c8a00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6c4198e10 con 0x7fc6c4102470 2024-04-20T21:53:28.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:28.154+0000 7fc6c8a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6c419cba0 con 0x7fc6c4108470 2024-04-20T21:53:28.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:28.154+0000 7fc6c1e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc6c4102470 0x7fc6c41981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:53:28.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:28.154+0000 7fc6c1e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc6c4102470 0x7fc6c41981f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34750/0 (socket says 172.21.15.62:34750) 2024-04-20T21:53:28.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:28.154+0000 7fc6c1e00700 1 -- 172.21.15.62:0/223058571 learned_addr learned my addr 172.21.15.62:0/223058571 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:53:28.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:28.154+0000 7fc6b9400700 1 --2- 172.21.15.62:0/223058571 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6c4108470 0x7fc6c4198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:53:28.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:28.154+0000 7fc6c1e00700 1 -- 172.21.15.62:0/223058571 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6c4108470 msgr2=0x7fc6c4198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:53:28.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:28.154+0000 7fc6c1e00700 1 --2- 172.21.15.62:0/223058571 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6c4108470 0x7fc6c4198730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:53:28.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:28.154+0000 7fc6c1e00700 1 -- 172.21.15.62:0/223058571 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6ac0095f0 con 0x7fc6c4102470 2024-04-20T21:53:28.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:28.155+0000 7fc6c1e00700 1 --2- 172.21.15.62:0/223058571 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc6c4102470 0x7fc6c41981f0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fc6b400d8d0 tx=0x7fc6b400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:53:28.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:28.155+0000 7fc6bb400700 1 -- 172.21.15.62:0/223058571 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc6b4009940 con 0x7fc6c4102470 2024-04-20T21:53:28.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:28.155+0000 7fc6bb400700 1 -- 172.21.15.62:0/223058571 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc6b4010460 con 0x7fc6c4102470 2024-04-20T21:53:28.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:28.155+0000 7fc6bb400700 1 -- 172.21.15.62:0/223058571 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc6b400f5f0 con 0x7fc6c4102470 2024-04-20T21:53:28.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:28.155+0000 7fc6c8a00700 1 -- 172.21.15.62:0/223058571 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6c419ce80 con 0x7fc6c4102470 2024-04-20T21:53:28.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:28.155+0000 7fc6c8a00700 1 -- 172.21.15.62:0/223058571 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6c419d3d0 con 0x7fc6c4102470 2024-04-20T21:53:28.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:28.156+0000 7fc6c8a00700 1 -- 172.21.15.62:0/223058571 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6c410a970 con 0x7fc6c4102470 2024-04-20T21:53:28.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:28.157+0000 7fc6bb400700 1 -- 172.21.15.62:0/223058571 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fc6b40105d0 con 0x7fc6c4102470 2024-04-20T21:53:28.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:28.158+0000 7fc6bb400700 1 --2- 172.21.15.62:0/223058571 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc6a406ecc0 0x7fc6a4071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:53:28.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:28.158+0000 7fc6bb400700 1 -- 172.21.15.62:0/223058571 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fc6b408ef20 con 0x7fc6c4102470 2024-04-20T21:53:28.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:28.159+0000 7fc6b9400700 1 --2- 172.21.15.62:0/223058571 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc6a406ecc0 0x7fc6a4071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:53:28.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:28.159+0000 7fc6b9400700 1 --2- 172.21.15.62:0/223058571 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc6a406ecc0 0x7fc6a4071170 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fc6ac005b40 tx=0x7fc6ac01a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:53:28.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:28.163+0000 7fc6bb400700 1 -- 172.21.15.62:0/223058571 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fc6b4057c30 con 0x7fc6c4102470 2024-04-20T21:53:28.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:53:27 smithi189 ceph-mon[57022]: from='client.35012 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:53:28.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:53:27 smithi189 ceph-mon[57022]: from='client.35016 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:53:28.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:53:27 smithi189 ceph-mon[57022]: pgmap v544: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:53:28.239 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:53:27 smithi189 ceph-mon[57022]: from='client.35020 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:53:28.239 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:53:27 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/1835203796' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:53:28.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:28.413+0000 7fc6c8a00700 1 -- 172.21.15.62:0/223058571 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc6c4199690 con 0x7fc6a406ecc0 2024-04-20T21:53:28.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:28.416+0000 7fc6bb400700 1 -- 172.21.15.62:0/223058571 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fc6ac005620 con 0x7fc6a406ecc0 2024-04-20T21:53:28.415 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:53:28.415 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T21:53:28.416 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T21:53:28.416 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-20T21:53:28.416 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T21:53:28.416 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr", 2024-04-20T21:53:28.416 INFO:teuthology.orchestra.run.smithi062.stdout: "crash", 2024-04-20T21:53:28.416 INFO:teuthology.orchestra.run.smithi062.stdout: "mon", 2024-04-20T21:53:28.416 INFO:teuthology.orchestra.run.smithi062.stdout: "osd" 2024-04-20T21:53:28.416 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T21:53:28.416 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "14/23 daemons upgraded", 2024-04-20T21:53:28.416 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading mds daemons", 2024-04-20T21:53:28.416 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T21:53:28.416 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:53:28.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:28.418+0000 7fc6c8a00700 1 -- 172.21.15.62:0/223058571 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc6a406ecc0 msgr2=0x7fc6a4071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:53:28.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:28.418+0000 7fc6c8a00700 1 --2- 172.21.15.62:0/223058571 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc6a406ecc0 0x7fc6a4071170 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fc6ac005b40 tx=0x7fc6ac01a040 comp rx=0 tx=0).stop 2024-04-20T21:53:28.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:28.418+0000 7fc6c8a00700 1 -- 172.21.15.62:0/223058571 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc6c4102470 msgr2=0x7fc6c41981f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:53:28.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:28.418+0000 7fc6c8a00700 1 --2- 172.21.15.62:0/223058571 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc6c4102470 0x7fc6c41981f0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fc6b400d8d0 tx=0x7fc6b400dc90 comp rx=0 tx=0).stop 2024-04-20T21:53:28.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:28.418+0000 7fc6c8a00700 1 -- 172.21.15.62:0/223058571 shutdown_connections 2024-04-20T21:53:28.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:28.418+0000 7fc6c8a00700 1 --2- 172.21.15.62:0/223058571 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc6a406ecc0 0x7fc6a4071170 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:53:28.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:28.418+0000 7fc6c8a00700 1 --2- 172.21.15.62:0/223058571 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc6c4102470 0x7fc6c41981f0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:53:28.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:28.418+0000 7fc6c8a00700 1 --2- 172.21.15.62:0/223058571 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6c4108470 0x7fc6c4198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:53:28.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:28.418+0000 7fc6c8a00700 1 -- 172.21.15.62:0/223058571 >> 172.21.15.62:0/223058571 conn(0x7fc6c40fdf90 msgr2=0x7fc6c40ff3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:53:28.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:28.419+0000 7fc6c8a00700 1 -- 172.21.15.62:0/223058571 shutdown_connections 2024-04-20T21:53:28.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:28.419+0000 7fc6c8a00700 1 -- 172.21.15.62:0/223058571 wait complete. 2024-04-20T21:53:28.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:28.574+0000 7fab7f200700 1 -- 172.21.15.62:0/1451992794 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fab800ff2f0 msgr2=0x7fab800ff6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:53:28.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:28.574+0000 7fab7f200700 1 --2- 172.21.15.62:0/1451992794 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fab800ff2f0 0x7fab800ff6c0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fab70009910 tx=0x7fab70009c20 comp rx=0 tx=0).stop 2024-04-20T21:53:28.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:28.574+0000 7fab7f200700 1 -- 172.21.15.62:0/1451992794 shutdown_connections 2024-04-20T21:53:28.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:28.574+0000 7fab7f200700 1 --2- 172.21.15.62:0/1451992794 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fab80100b50 0x7fab80100fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:53:28.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:28.574+0000 7fab7f200700 1 --2- 172.21.15.62:0/1451992794 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fab800ff2f0 0x7fab800ff6c0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:53:28.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:28.574+0000 7fab7f200700 1 -- 172.21.15.62:0/1451992794 >> 172.21.15.62:0/1451992794 conn(0x7fab800faad0 msgr2=0x7fab800fcee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:53:28.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:28.574+0000 7fab7f200700 1 -- 172.21.15.62:0/1451992794 shutdown_connections 2024-04-20T21:53:28.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:28.574+0000 7fab7f200700 1 -- 172.21.15.62:0/1451992794 wait complete. 2024-04-20T21:53:28.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:28.575+0000 7fab7f200700 1 Processor -- start 2024-04-20T21:53:28.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:28.575+0000 7fab7f200700 1 -- start start 2024-04-20T21:53:28.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:28.576+0000 7fab7f200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fab800ff2f0 0x7fab80197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:53:28.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:28.576+0000 7fab7f200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fab80100b50 0x7fab80198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:53:28.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:28.576+0000 7fab7f200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab80198b70 con 0x7fab80100b50 2024-04-20T21:53:28.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:28.576+0000 7fab7f200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab80198cb0 con 0x7fab800ff2f0 2024-04-20T21:53:28.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:28.576+0000 7fab7d400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fab80100b50 0x7fab80198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:53:28.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:28.576+0000 7fab7d400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fab80100b50 0x7fab80198520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34772/0 (socket says 172.21.15.62:34772) 2024-04-20T21:53:28.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:28.576+0000 7fab7d400700 1 -- 172.21.15.62:0/4161653079 learned_addr learned my addr 172.21.15.62:0/4161653079 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:53:28.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:28.576+0000 7fab7de00700 1 --2- 172.21.15.62:0/4161653079 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fab800ff2f0 0x7fab80197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:53:28.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:28.577+0000 7fab7d400700 1 -- 172.21.15.62:0/4161653079 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fab800ff2f0 msgr2=0x7fab80197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:53:28.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:28.577+0000 7fab7d400700 1 --2- 172.21.15.62:0/4161653079 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fab800ff2f0 0x7fab80197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:53:28.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:28.577+0000 7fab7d400700 1 -- 172.21.15.62:0/4161653079 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab700095f0 con 0x7fab80100b50 2024-04-20T21:53:28.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:28.577+0000 7fab7d400700 1 --2- 172.21.15.62:0/4161653079 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fab80100b50 0x7fab80198520 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fab6c00c930 tx=0x7fab6c00ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:53:28.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:28.577+0000 7fab76a00700 1 -- 172.21.15.62:0/4161653079 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fab6c007ad0 con 0x7fab80100b50 2024-04-20T21:53:28.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:28.577+0000 7fab76a00700 1 -- 172.21.15.62:0/4161653079 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fab6c007c30 con 0x7fab80100b50 2024-04-20T21:53:28.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:28.577+0000 7fab76a00700 1 -- 172.21.15.62:0/4161653079 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fab6c018610 con 0x7fab80100b50 2024-04-20T21:53:28.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:28.577+0000 7fab7f200700 1 -- 172.21.15.62:0/4161653079 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab8019cb00 con 0x7fab80100b50 2024-04-20T21:53:28.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:28.578+0000 7fab7f200700 1 -- 172.21.15.62:0/4161653079 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab8019d050 con 0x7fab80100b50 2024-04-20T21:53:28.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:28.579+0000 7fab7f200700 1 -- 172.21.15.62:0/4161653079 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab80062380 con 0x7fab80100b50 2024-04-20T21:53:28.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:28.580+0000 7fab76a00700 1 -- 172.21.15.62:0/4161653079 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fab6c00f450 con 0x7fab80100b50 2024-04-20T21:53:28.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:28.581+0000 7fab76a00700 1 --2- 172.21.15.62:0/4161653079 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fab6406ecc0 0x7fab64071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:53:28.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:28.581+0000 7fab76a00700 1 -- 172.21.15.62:0/4161653079 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fab6c08eff0 con 0x7fab80100b50 2024-04-20T21:53:28.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:28.585+0000 7fab7de00700 1 --2- 172.21.15.62:0/4161653079 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fab6406ecc0 0x7fab64071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:53:28.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:28.587+0000 7fab7de00700 1 --2- 172.21.15.62:0/4161653079 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fab6406ecc0 0x7fab64071170 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fab70006010 tx=0x7fab7001a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:53:28.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:28.590+0000 7fab76a00700 1 -- 172.21.15.62:0/4161653079 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fab6c057de0 con 0x7fab80100b50 2024-04-20T21:53:28.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:28.920+0000 7fab7f200700 1 -- 172.21.15.62:0/4161653079 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fab80199340 con 0x7fab80100b50 2024-04-20T21:53:28.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:28.920+0000 7fab76a00700 1 -- 172.21.15.62:0/4161653079 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7fab6c05b4a0 con 0x7fab80100b50 2024-04-20T21:53:28.920 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-20T21:53:28.920 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-20T21:53:28.920 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is degraded 2024-04-20T21:53:28.920 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T21:53:28.920 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T21:53:28.920 INFO:teuthology.orchestra.run.smithi062.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-20T21:53:28.920 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-20T21:53:28.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:28.923+0000 7fab7f200700 1 -- 172.21.15.62:0/4161653079 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fab6406ecc0 msgr2=0x7fab64071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:53:28.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:28.923+0000 7fab7f200700 1 --2- 172.21.15.62:0/4161653079 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fab6406ecc0 0x7fab64071170 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fab70006010 tx=0x7fab7001a040 comp rx=0 tx=0).stop 2024-04-20T21:53:28.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:28.923+0000 7fab7f200700 1 -- 172.21.15.62:0/4161653079 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fab80100b50 msgr2=0x7fab80198520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:53:28.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:28.923+0000 7fab7f200700 1 --2- 172.21.15.62:0/4161653079 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fab80100b50 0x7fab80198520 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fab6c00c930 tx=0x7fab6c00ccf0 comp rx=0 tx=0).stop 2024-04-20T21:53:28.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:28.923+0000 7fab7f200700 1 -- 172.21.15.62:0/4161653079 shutdown_connections 2024-04-20T21:53:28.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:28.923+0000 7fab7f200700 1 --2- 172.21.15.62:0/4161653079 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fab6406ecc0 0x7fab64071170 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:53:28.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:28.923+0000 7fab7f200700 1 --2- 172.21.15.62:0/4161653079 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fab800ff2f0 0x7fab80197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:53:28.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:28.923+0000 7fab7f200700 1 --2- 172.21.15.62:0/4161653079 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fab80100b50 0x7fab80198520 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:53:28.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:28.923+0000 7fab7f200700 1 -- 172.21.15.62:0/4161653079 >> 172.21.15.62:0/4161653079 conn(0x7fab800faad0 msgr2=0x7fab80068e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:53:28.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:28.924+0000 7fab7f200700 1 -- 172.21.15.62:0/4161653079 shutdown_connections 2024-04-20T21:53:28.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:28.924+0000 7fab7f200700 1 -- 172.21.15.62:0/4161653079 wait complete. 2024-04-20T21:53:29.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:53:28 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/2189791919' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T21:53:29.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:53:28 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/2189791919' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T21:53:30.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:53:29 smithi062 ceph-mon[76322]: from='client.35032 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:53:30.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:53:29 smithi062 ceph-mon[76322]: pgmap v545: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:53:30.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:53:29 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/4161653079' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T21:53:30.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:53:29 smithi189 ceph-mon[57022]: from='client.35032 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:53:30.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:53:29 smithi189 ceph-mon[57022]: pgmap v545: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:53:30.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:53:29 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/4161653079' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T21:53:32.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:53:31 smithi062 ceph-mon[76322]: pgmap v546: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:53:32.039 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:53:31 smithi189 ceph-mon[57022]: pgmap v546: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:53:34.038 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:53:33 smithi062 ceph-mon[76322]: pgmap v547: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:53:34.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:53:33 smithi189 ceph-mon[57022]: pgmap v547: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:53:35.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:53:34 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:53:35.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:53:34 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:53:36.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:53:35 smithi062 ceph-mon[76322]: pgmap v548: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:53:36.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:53:35 smithi189 ceph-mon[57022]: pgmap v548: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:53:38.142 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:53:37 smithi062 ceph-mon[76322]: pgmap v549: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:53:38.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:53:37 smithi189 ceph-mon[57022]: pgmap v549: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:53:40.049 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:53:39 smithi062 ceph-mon[76322]: pgmap v550: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:53:40.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:53:39 smithi189 ceph-mon[57022]: pgmap v550: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:53:42.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:53:41 smithi189 ceph-mon[57022]: pgmap v551: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:53:42.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:53:41 smithi062 ceph-mon[76322]: pgmap v551: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:53:44.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:53:43 smithi189 ceph-mon[57022]: pgmap v552: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:53:44.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:53:43 smithi062 ceph-mon[76322]: pgmap v552: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:53:46.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:53:45 smithi189 ceph-mon[57022]: pgmap v553: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:53:46.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:53:45 smithi062 ceph-mon[76322]: pgmap v553: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:53:48.142 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:53:47 smithi062 ceph-mon[76322]: pgmap v554: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:53:48.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:53:47 smithi189 ceph-mon[57022]: pgmap v554: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:53:50.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:53:49 smithi189 ceph-mon[57022]: pgmap v555: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:53:50.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:53:49 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:53:50.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:53:49 smithi062 ceph-mon[76322]: pgmap v555: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:53:50.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:53:49 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:53:52.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:53:51 smithi189 ceph-mon[57022]: pgmap v556: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:53:52.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:53:51 smithi062 ceph-mon[76322]: pgmap v556: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:53:54.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:53:53 smithi189 ceph-mon[57022]: pgmap v557: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:53:54.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:53:53 smithi062 ceph-mon[76322]: pgmap v557: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:53:56.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:53:55 smithi189 ceph-mon[57022]: pgmap v558: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:53:56.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:53:55 smithi062 ceph-mon[76322]: pgmap v558: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:53:58.143 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:53:57 smithi062 ceph-mon[76322]: pgmap v559: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:53:58.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:53:57 smithi189 ceph-mon[57022]: pgmap v559: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:53:59.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.102+0000 7fa514a00700 1 -- 172.21.15.62:0/846665646 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa510100c70 msgr2=0x7fa510104c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:53:59.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.102+0000 7fa514a00700 1 --2- 172.21.15.62:0/846665646 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa510100c70 0x7fa510104c40 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fa4f8009910 tx=0x7fa4f8009c20 comp rx=0 tx=0).stop 2024-04-20T21:53:59.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.102+0000 7fa514a00700 1 -- 172.21.15.62:0/846665646 shutdown_connections 2024-04-20T21:53:59.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.102+0000 7fa514a00700 1 --2- 172.21.15.62:0/846665646 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa510100c70 0x7fa510104c40 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:53:59.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.102+0000 7fa514a00700 1 --2- 172.21.15.62:0/846665646 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5101002d0 0x7fa5101006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:53:59.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.102+0000 7fa514a00700 1 -- 172.21.15.62:0/846665646 >> 172.21.15.62:0/846665646 conn(0x7fa5100fbbb0 msgr2=0x7fa5100fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:53:59.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.102+0000 7fa514a00700 1 -- 172.21.15.62:0/846665646 shutdown_connections 2024-04-20T21:53:59.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.102+0000 7fa514a00700 1 -- 172.21.15.62:0/846665646 wait complete. 2024-04-20T21:53:59.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.103+0000 7fa514a00700 1 Processor -- start 2024-04-20T21:53:59.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.103+0000 7fa514a00700 1 -- start start 2024-04-20T21:53:59.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.104+0000 7fa514a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5101002d0 0x7fa510073050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:53:59.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.104+0000 7fa514a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa510100c70 0x7fa510073590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:53:59.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.104+0000 7fa514a00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa510073ad0 con 0x7fa510100c70 2024-04-20T21:53:59.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.104+0000 7fa514a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa510073c10 con 0x7fa5101002d0 2024-04-20T21:53:59.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.104+0000 7fa506a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa510100c70 0x7fa510073590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:53:59.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.104+0000 7fa506a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa510100c70 0x7fa510073590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59850/0 (socket says 172.21.15.62:59850) 2024-04-20T21:53:59.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.104+0000 7fa506a00700 1 -- 172.21.15.62:0/889974696 learned_addr learned my addr 172.21.15.62:0/889974696 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:53:59.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.104+0000 7fa50f400700 1 --2- 172.21.15.62:0/889974696 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5101002d0 0x7fa510073050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:53:59.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.104+0000 7fa506a00700 1 -- 172.21.15.62:0/889974696 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5101002d0 msgr2=0x7fa510073050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:53:59.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.104+0000 7fa506a00700 1 --2- 172.21.15.62:0/889974696 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5101002d0 0x7fa510073050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:53:59.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.104+0000 7fa506a00700 1 -- 172.21.15.62:0/889974696 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4f80095f0 con 0x7fa510100c70 2024-04-20T21:53:59.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.105+0000 7fa506a00700 1 --2- 172.21.15.62:0/889974696 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa510100c70 0x7fa510073590 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fa4f800bda0 tx=0x7fa4f800be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:53:59.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.105+0000 7fa50cc00700 1 -- 172.21.15.62:0/889974696 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa4f801d070 con 0x7fa510100c70 2024-04-20T21:53:59.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.105+0000 7fa50cc00700 1 -- 172.21.15.62:0/889974696 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa4f8004fb0 con 0x7fa510100c70 2024-04-20T21:53:59.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.105+0000 7fa50cc00700 1 -- 172.21.15.62:0/889974696 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa4f800f480 con 0x7fa510100c70 2024-04-20T21:53:59.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.105+0000 7fa514a00700 1 -- 172.21.15.62:0/889974696 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa510073e90 con 0x7fa510100c70 2024-04-20T21:53:59.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.105+0000 7fa514a00700 1 -- 172.21.15.62:0/889974696 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5101a2160 con 0x7fa510100c70 2024-04-20T21:53:59.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.107+0000 7fa514a00700 1 -- 172.21.15.62:0/889974696 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa510062380 con 0x7fa510100c70 2024-04-20T21:53:59.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.108+0000 7fa50cc00700 1 -- 172.21.15.62:0/889974696 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fa4f8022870 con 0x7fa510100c70 2024-04-20T21:53:59.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.109+0000 7fa50cc00700 1 --2- 172.21.15.62:0/889974696 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa4f406ecc0 0x7fa4f4071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:53:59.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.109+0000 7fa50cc00700 1 -- 172.21.15.62:0/889974696 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fa4f80902b0 con 0x7fa510100c70 2024-04-20T21:53:59.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.109+0000 7fa50f400700 1 --2- 172.21.15.62:0/889974696 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa4f406ecc0 0x7fa4f4071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:53:59.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.110+0000 7fa50f400700 1 --2- 172.21.15.62:0/889974696 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa4f406ecc0 0x7fa4f4071170 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fa500005950 tx=0x7fa500005cf0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:53:59.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.114+0000 7fa50cc00700 1 -- 172.21.15.62:0/889974696 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fa4f8059190 con 0x7fa510100c70 2024-04-20T21:53:59.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.370+0000 7fa514a00700 1 -- 172.21.15.62:0/889974696 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa5100fdf00 con 0x7fa4f406ecc0 2024-04-20T21:53:59.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.372+0000 7fa50cc00700 1 -- 172.21.15.62:0/889974696 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fa5100fdf00 con 0x7fa4f406ecc0 2024-04-20T21:53:59.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.376+0000 7fa514a00700 1 -- 172.21.15.62:0/889974696 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa4f406ecc0 msgr2=0x7fa4f4071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:53:59.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.376+0000 7fa514a00700 1 --2- 172.21.15.62:0/889974696 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa4f406ecc0 0x7fa4f4071170 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fa500005950 tx=0x7fa500005cf0 comp rx=0 tx=0).stop 2024-04-20T21:53:59.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.376+0000 7fa514a00700 1 -- 172.21.15.62:0/889974696 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa510100c70 msgr2=0x7fa510073590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:53:59.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.376+0000 7fa514a00700 1 --2- 172.21.15.62:0/889974696 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa510100c70 0x7fa510073590 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fa4f800bda0 tx=0x7fa4f800be80 comp rx=0 tx=0).stop 2024-04-20T21:53:59.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.376+0000 7fa514a00700 1 -- 172.21.15.62:0/889974696 shutdown_connections 2024-04-20T21:53:59.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.376+0000 7fa514a00700 1 --2- 172.21.15.62:0/889974696 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa4f406ecc0 0x7fa4f4071170 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:53:59.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.376+0000 7fa514a00700 1 --2- 172.21.15.62:0/889974696 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5101002d0 0x7fa510073050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:53:59.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.376+0000 7fa514a00700 1 --2- 172.21.15.62:0/889974696 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa510100c70 0x7fa510073590 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:53:59.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.376+0000 7fa514a00700 1 -- 172.21.15.62:0/889974696 >> 172.21.15.62:0/889974696 conn(0x7fa5100fbbb0 msgr2=0x7fa5100fd130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:53:59.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.377+0000 7fa514a00700 1 -- 172.21.15.62:0/889974696 shutdown_connections 2024-04-20T21:53:59.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.377+0000 7fa514a00700 1 -- 172.21.15.62:0/889974696 wait complete. 2024-04-20T21:53:59.393 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T21:53:59.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.547+0000 7fb945c00700 1 -- 172.21.15.62:0/265784778 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb9401004e0 msgr2=0x7fb9401008b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:53:59.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.547+0000 7fb945c00700 1 --2- 172.21.15.62:0/265784778 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb9401004e0 0x7fb9401008b0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fb93000b0a0 tx=0x7fb93000b3b0 comp rx=0 tx=0).stop 2024-04-20T21:53:59.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.547+0000 7fb945c00700 1 -- 172.21.15.62:0/265784778 shutdown_connections 2024-04-20T21:53:59.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.547+0000 7fb945c00700 1 --2- 172.21.15.62:0/265784778 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb940100e80 0x7fb940104e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:53:59.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.547+0000 7fb945c00700 1 --2- 172.21.15.62:0/265784778 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb9401004e0 0x7fb9401008b0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:53:59.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.547+0000 7fb945c00700 1 -- 172.21.15.62:0/265784778 >> 172.21.15.62:0/265784778 conn(0x7fb9400fbda0 msgr2=0x7fb9400fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:53:59.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.547+0000 7fb945c00700 1 -- 172.21.15.62:0/265784778 shutdown_connections 2024-04-20T21:53:59.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.548+0000 7fb945c00700 1 -- 172.21.15.62:0/265784778 wait complete. 2024-04-20T21:53:59.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.549+0000 7fb945c00700 1 Processor -- start 2024-04-20T21:53:59.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.549+0000 7fb945c00700 1 -- start start 2024-04-20T21:53:59.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.549+0000 7fb945c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb9401004e0 0x7fb940198260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:53:59.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.549+0000 7fb945c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb940100e80 0x7fb9401987a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:53:59.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.549+0000 7fb945c00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb940198e80 con 0x7fb940100e80 2024-04-20T21:53:59.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.549+0000 7fb945c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb94019cc10 con 0x7fb9401004e0 2024-04-20T21:53:59.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.550+0000 7fb93e000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb940100e80 0x7fb9401987a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:53:59.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.550+0000 7fb93e000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb940100e80 0x7fb9401987a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59866/0 (socket says 172.21.15.62:59866) 2024-04-20T21:53:59.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.550+0000 7fb93e000700 1 -- 172.21.15.62:0/604717331 learned_addr learned my addr 172.21.15.62:0/604717331 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:53:59.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.550+0000 7fb93ea00700 1 --2- 172.21.15.62:0/604717331 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb9401004e0 0x7fb940198260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:53:59.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.550+0000 7fb93e000700 1 -- 172.21.15.62:0/604717331 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb9401004e0 msgr2=0x7fb940198260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:53:59.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.550+0000 7fb93e000700 1 --2- 172.21.15.62:0/604717331 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb9401004e0 0x7fb940198260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:53:59.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.550+0000 7fb93e000700 1 -- 172.21.15.62:0/604717331 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb930009d00 con 0x7fb940100e80 2024-04-20T21:53:59.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.550+0000 7fb93e000700 1 --2- 172.21.15.62:0/604717331 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb940100e80 0x7fb9401987a0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fb92c00b480 tx=0x7fb92c00b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:53:59.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.551+0000 7fb937400700 1 -- 172.21.15.62:0/604717331 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb92c0118c0 con 0x7fb940100e80 2024-04-20T21:53:59.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.551+0000 7fb937400700 1 -- 172.21.15.62:0/604717331 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb92c010380 con 0x7fb940100e80 2024-04-20T21:53:59.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.551+0000 7fb937400700 1 -- 172.21.15.62:0/604717331 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb92c00f4f0 con 0x7fb940100e80 2024-04-20T21:53:59.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.551+0000 7fb945c00700 1 -- 172.21.15.62:0/604717331 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb94019cef0 con 0x7fb940100e80 2024-04-20T21:53:59.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.551+0000 7fb945c00700 1 -- 172.21.15.62:0/604717331 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb94019d440 con 0x7fb940100e80 2024-04-20T21:53:59.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.552+0000 7fb945c00700 1 -- 172.21.15.62:0/604717331 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9401045d0 con 0x7fb940100e80 2024-04-20T21:53:59.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.553+0000 7fb937400700 1 -- 172.21.15.62:0/604717331 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fb92c0104f0 con 0x7fb940100e80 2024-04-20T21:53:59.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.554+0000 7fb937400700 1 --2- 172.21.15.62:0/604717331 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb92406ecc0 0x7fb924071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:53:59.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.554+0000 7fb937400700 1 -- 172.21.15.62:0/604717331 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fb92c08ea70 con 0x7fb940100e80 2024-04-20T21:53:59.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.554+0000 7fb93ea00700 1 --2- 172.21.15.62:0/604717331 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb92406ecc0 0x7fb924071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:53:59.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.555+0000 7fb93ea00700 1 --2- 172.21.15.62:0/604717331 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb92406ecc0 0x7fb924071170 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fb930015040 tx=0x7fb930000f40 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:53:59.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.559+0000 7fb937400700 1 -- 172.21.15.62:0/604717331 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fb92c057800 con 0x7fb940100e80 2024-04-20T21:53:59.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.818+0000 7fb945c00700 1 -- 172.21.15.62:0/604717331 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb94019d710 con 0x7fb92406ecc0 2024-04-20T21:53:59.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.821+0000 7fb937400700 1 -- 172.21.15.62:0/604717331 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fb94019d710 con 0x7fb92406ecc0 2024-04-20T21:53:59.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.824+0000 7fb945c00700 1 -- 172.21.15.62:0/604717331 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb92406ecc0 msgr2=0x7fb924071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:53:59.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.824+0000 7fb945c00700 1 --2- 172.21.15.62:0/604717331 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb92406ecc0 0x7fb924071170 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fb930015040 tx=0x7fb930000f40 comp rx=0 tx=0).stop 2024-04-20T21:53:59.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.824+0000 7fb945c00700 1 -- 172.21.15.62:0/604717331 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb940100e80 msgr2=0x7fb9401987a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:53:59.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.824+0000 7fb945c00700 1 --2- 172.21.15.62:0/604717331 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb940100e80 0x7fb9401987a0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fb92c00b480 tx=0x7fb92c00b840 comp rx=0 tx=0).stop 2024-04-20T21:53:59.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.824+0000 7fb945c00700 1 -- 172.21.15.62:0/604717331 shutdown_connections 2024-04-20T21:53:59.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.824+0000 7fb945c00700 1 --2- 172.21.15.62:0/604717331 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb92406ecc0 0x7fb924071170 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:53:59.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.824+0000 7fb945c00700 1 --2- 172.21.15.62:0/604717331 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb9401004e0 0x7fb940198260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:53:59.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.825+0000 7fb945c00700 1 --2- 172.21.15.62:0/604717331 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb940100e80 0x7fb9401987a0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:53:59.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.825+0000 7fb945c00700 1 -- 172.21.15.62:0/604717331 >> 172.21.15.62:0/604717331 conn(0x7fb9400fbda0 msgr2=0x7fb9400fd480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:53:59.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.825+0000 7fb945c00700 1 -- 172.21.15.62:0/604717331 shutdown_connections 2024-04-20T21:53:59.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.825+0000 7fb945c00700 1 -- 172.21.15.62:0/604717331 wait complete. 2024-04-20T21:53:59.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.989+0000 7f435de00700 1 -- 172.21.15.62:0/3611346731 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f43581002c0 msgr2=0x7f4358100690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:53:59.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.989+0000 7f435de00700 1 --2- 172.21.15.62:0/3611346731 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f43581002c0 0x7f4358100690 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f4348009910 tx=0x7f4348009c20 comp rx=0 tx=0).stop 2024-04-20T21:53:59.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.989+0000 7f435de00700 1 -- 172.21.15.62:0/3611346731 shutdown_connections 2024-04-20T21:53:59.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.989+0000 7f435de00700 1 --2- 172.21.15.62:0/3611346731 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4358100c60 0x7f4358104c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:53:59.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.989+0000 7f435de00700 1 --2- 172.21.15.62:0/3611346731 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f43581002c0 0x7f4358100690 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:53:59.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.989+0000 7f435de00700 1 -- 172.21.15.62:0/3611346731 >> 172.21.15.62:0/3611346731 conn(0x7f43580fbb60 msgr2=0x7f43580fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:53:59.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.989+0000 7f435de00700 1 -- 172.21.15.62:0/3611346731 shutdown_connections 2024-04-20T21:53:59.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.989+0000 7f435de00700 1 -- 172.21.15.62:0/3611346731 wait complete. 2024-04-20T21:53:59.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.990+0000 7f435de00700 1 Processor -- start 2024-04-20T21:53:59.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.991+0000 7f435de00700 1 -- start start 2024-04-20T21:53:59.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.991+0000 7f435de00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f43581002c0 0x7f4358073050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:53:59.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.991+0000 7f435de00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4358100c60 0x7f4358073590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:53:59.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.991+0000 7f435de00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4358073ad0 con 0x7f43581002c0 2024-04-20T21:53:59.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.991+0000 7f435de00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4358073c10 con 0x7f4358100c60 2024-04-20T21:53:59.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.991+0000 7f435ca00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f43581002c0 0x7f4358073050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:53:59.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.992+0000 7f435ca00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f43581002c0 0x7f4358073050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59878/0 (socket says 172.21.15.62:59878) 2024-04-20T21:53:59.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.992+0000 7f435ca00700 1 -- 172.21.15.62:0/2031514679 learned_addr learned my addr 172.21.15.62:0/2031514679 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:53:59.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.992+0000 7f435ca00700 1 -- 172.21.15.62:0/2031514679 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4358100c60 msgr2=0x7f4358073590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:53:59.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.992+0000 7f435ca00700 1 --2- 172.21.15.62:0/2031514679 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4358100c60 0x7f4358073590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:53:59.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.992+0000 7f435ca00700 1 -- 172.21.15.62:0/2031514679 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43480095f0 con 0x7f43581002c0 2024-04-20T21:53:59.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.992+0000 7f435ca00700 1 --2- 172.21.15.62:0/2031514679 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f43581002c0 0x7f4358073050 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f43480092e0 tx=0x7f434800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:53:59.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.992+0000 7f4356000700 1 -- 172.21.15.62:0/2031514679 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f434801d070 con 0x7f43581002c0 2024-04-20T21:53:59.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.993+0000 7f4356000700 1 -- 172.21.15.62:0/2031514679 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4348005e00 con 0x7f43581002c0 2024-04-20T21:53:59.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.993+0000 7f4356000700 1 -- 172.21.15.62:0/2031514679 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4348017400 con 0x7f43581002c0 2024-04-20T21:53:59.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.993+0000 7f435de00700 1 -- 172.21.15.62:0/2031514679 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4358073e90 con 0x7f43581002c0 2024-04-20T21:53:59.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.993+0000 7f435de00700 1 -- 172.21.15.62:0/2031514679 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43581a20f0 con 0x7f43581002c0 2024-04-20T21:53:59.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.994+0000 7f435de00700 1 -- 172.21.15.62:0/2031514679 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43581085d0 con 0x7f43581002c0 2024-04-20T21:53:59.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.995+0000 7f4356000700 1 -- 172.21.15.62:0/2031514679 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f434800f460 con 0x7f43581002c0 2024-04-20T21:53:59.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.996+0000 7f4356000700 1 --2- 172.21.15.62:0/2031514679 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f433c06ecc0 0x7f433c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:53:59.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.997+0000 7f4356000700 1 -- 172.21.15.62:0/2031514679 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f4348090290 con 0x7f43581002c0 2024-04-20T21:53:59.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.997+0000 7f434fe00700 1 --2- 172.21.15.62:0/2031514679 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f433c06ecc0 0x7f433c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:53:59.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:53:59.998+0000 7f434fe00700 1 --2- 172.21.15.62:0/2031514679 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f433c06ecc0 0x7f433c071170 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f4340005950 tx=0x7f43400058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:54:00.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:00.001+0000 7f4356000700 1 -- 172.21.15.62:0/2031514679 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f43480590c0 con 0x7f43581002c0 2024-04-20T21:54:00.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:53:59 smithi189 ceph-mon[57022]: pgmap v560: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:54:00.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:00.253+0000 7f435de00700 1 -- 172.21.15.62:0/2031514679 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f435802ccf0 con 0x7f433c06ecc0 2024-04-20T21:54:00.252 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:53:59 smithi062 ceph-mon[76322]: pgmap v560: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:54:00.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:00.285+0000 7f4356000700 1 -- 172.21.15.62:0/2031514679 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f435802ccf0 con 0x7f433c06ecc0 2024-04-20T21:54:00.285 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T21:54:00.285 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (19m) 11m ago 33m 14.2M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T21:54:00.285 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (34m) 11m ago 34m 9512k - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T21:54:00.285 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (32m) 12m ago 32m 8942k - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T21:54:00.285 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (15m) 11m ago 33m 6660k - 19.0.0-3133-ga8c84f59 76bc6b91b726 2825d5ba11c6 2024-04-20T21:54:00.285 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (15m) 12m ago 32m 6665k - 19.0.0-3133-ga8c84f59 76bc6b91b726 20f746fa1bba 2024-04-20T21:54:00.285 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (19m) 11m ago 33m 74.2M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T21:54:00.285 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (11m) 11m ago 26m 11.2M - 19.0.0-3133-ga8c84f59 76bc6b91b726 de80bc90a683 2024-04-20T21:54:00.285 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (11m) 11m ago 26m 11.9M - 19.0.0-3133-ga8c84f59 76bc6b91b726 cd793b7ee98e 2024-04-20T21:54:00.285 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (26m) 12m ago 26m 117M - 18.2.0 dc2bc1663786 fb2a8a223db8 2024-04-20T21:54:00.285 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (26m) 12m ago 26m 55.6M - 18.2.0 dc2bc1663786 f2eec2ee8951 2024-04-20T21:54:00.285 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (22m) 11m ago 35m 511M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T21:54:00.285 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (21m) 12m ago 32m 452M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T21:54:00.286 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (17m) 11m ago 35m 55.9M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 019eb1479485 2024-04-20T21:54:00.286 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (16m) 12m ago 32m 39.8M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 55aac560aab6 2024-04-20T21:54:00.286 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (21m) 11m ago 33m 7952k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T21:54:00.286 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (20m) 12m ago 32m 8304k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T21:54:00.286 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (15m) 11m ago 31m 88.0M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 ee2bb901f26e 2024-04-20T21:54:00.286 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (14m) 11m ago 30m 92.3M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 9075152d4162 2024-04-20T21:54:00.286 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (14m) 11m ago 30m 74.4M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 2d4f3e38dc0e 2024-04-20T21:54:00.286 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (13m) 12m ago 29m 87.6M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 8f97b17ae0eb 2024-04-20T21:54:00.286 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (12m) 12m ago 28m 90.7M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 653d8079967d 2024-04-20T21:54:00.286 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (12m) 12m ago 28m 11.5M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 596dded14c12 2024-04-20T21:54:00.286 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (20m) 11m ago 33m 74.0M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T21:54:00.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:00.289+0000 7f435de00700 1 -- 172.21.15.62:0/2031514679 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f433c06ecc0 msgr2=0x7f433c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:54:00.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:00.289+0000 7f435de00700 1 --2- 172.21.15.62:0/2031514679 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f433c06ecc0 0x7f433c071170 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f4340005950 tx=0x7f43400058e0 comp rx=0 tx=0).stop 2024-04-20T21:54:00.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:00.289+0000 7f435de00700 1 -- 172.21.15.62:0/2031514679 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f43581002c0 msgr2=0x7f4358073050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:54:00.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:00.289+0000 7f435de00700 1 --2- 172.21.15.62:0/2031514679 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f43581002c0 0x7f4358073050 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f43480092e0 tx=0x7f434800be20 comp rx=0 tx=0).stop 2024-04-20T21:54:00.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:00.289+0000 7f435de00700 1 -- 172.21.15.62:0/2031514679 shutdown_connections 2024-04-20T21:54:00.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:00.289+0000 7f435de00700 1 --2- 172.21.15.62:0/2031514679 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f433c06ecc0 0x7f433c071170 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:54:00.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:00.289+0000 7f435de00700 1 --2- 172.21.15.62:0/2031514679 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f43581002c0 0x7f4358073050 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:54:00.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:00.289+0000 7f435de00700 1 --2- 172.21.15.62:0/2031514679 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4358100c60 0x7f4358073590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:54:00.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:00.289+0000 7f435de00700 1 -- 172.21.15.62:0/2031514679 >> 172.21.15.62:0/2031514679 conn(0x7f43580fbb60 msgr2=0x7f43580fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:54:00.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:00.289+0000 7f435de00700 1 -- 172.21.15.62:0/2031514679 shutdown_connections 2024-04-20T21:54:00.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:00.289+0000 7f435de00700 1 -- 172.21.15.62:0/2031514679 wait complete. 2024-04-20T21:54:00.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:00.455+0000 7fec6dc00700 1 -- 172.21.15.62:0/882452135 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fec68073a00 msgr2=0x7fec6810c5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:54:00.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:00.455+0000 7fec6dc00700 1 --2- 172.21.15.62:0/882452135 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fec68073a00 0x7fec6810c5d0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fec50009910 tx=0x7fec50009c20 comp rx=0 tx=0).stop 2024-04-20T21:54:00.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:00.455+0000 7fec6dc00700 1 -- 172.21.15.62:0/882452135 shutdown_connections 2024-04-20T21:54:00.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:00.455+0000 7fec6dc00700 1 --2- 172.21.15.62:0/882452135 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fec68073a00 0x7fec6810c5d0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:54:00.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:00.455+0000 7fec6dc00700 1 --2- 172.21.15.62:0/882452135 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec680730f0 0x7fec680734c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:54:00.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:00.455+0000 7fec6dc00700 1 -- 172.21.15.62:0/882452135 >> 172.21.15.62:0/882452135 conn(0x7fec680fbe10 msgr2=0x7fec680fe220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:54:00.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:00.455+0000 7fec6dc00700 1 -- 172.21.15.62:0/882452135 shutdown_connections 2024-04-20T21:54:00.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:00.455+0000 7fec6dc00700 1 -- 172.21.15.62:0/882452135 wait complete. 2024-04-20T21:54:00.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:00.456+0000 7fec6dc00700 1 Processor -- start 2024-04-20T21:54:00.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:00.456+0000 7fec6dc00700 1 -- start start 2024-04-20T21:54:00.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:00.457+0000 7fec6dc00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fec680730f0 0x7fec68197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:54:00.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:00.457+0000 7fec6dc00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec68073a00 0x7fec68198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:54:00.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:00.457+0000 7fec6dc00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec6819c920 con 0x7fec680730f0 2024-04-20T21:54:00.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:00.457+0000 7fec6dc00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec6819ca90 con 0x7fec68073a00 2024-04-20T21:54:00.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:00.457+0000 7fec66a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fec680730f0 0x7fec68197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:54:00.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:00.457+0000 7fec66a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fec680730f0 0x7fec68197fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59906/0 (socket says 172.21.15.62:59906) 2024-04-20T21:54:00.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:00.457+0000 7fec66a00700 1 -- 172.21.15.62:0/2598230697 learned_addr learned my addr 172.21.15.62:0/2598230697 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:54:00.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:00.457+0000 7fec5e000700 1 --2- 172.21.15.62:0/2598230697 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec68073a00 0x7fec68198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:54:00.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:00.458+0000 7fec66a00700 1 -- 172.21.15.62:0/2598230697 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec68073a00 msgr2=0x7fec68198630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:54:00.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:00.458+0000 7fec66a00700 1 --2- 172.21.15.62:0/2598230697 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec68073a00 0x7fec68198630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:54:00.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:00.458+0000 7fec66a00700 1 -- 172.21.15.62:0/2598230697 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec500095f0 con 0x7fec680730f0 2024-04-20T21:54:00.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:00.458+0000 7fec66a00700 1 --2- 172.21.15.62:0/2598230697 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fec680730f0 0x7fec68197fc0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fec58007e90 tx=0x7fec5800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:54:00.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:00.458+0000 7fec5fe00700 1 -- 172.21.15.62:0/2598230697 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fec58017070 con 0x7fec680730f0 2024-04-20T21:54:00.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:00.458+0000 7fec6dc00700 1 -- 172.21.15.62:0/2598230697 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec6819cd70 con 0x7fec680730f0 2024-04-20T21:54:00.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:00.458+0000 7fec5fe00700 1 -- 172.21.15.62:0/2598230697 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fec5800f040 con 0x7fec680730f0 2024-04-20T21:54:00.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:00.458+0000 7fec5fe00700 1 -- 172.21.15.62:0/2598230697 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fec58005970 con 0x7fec680730f0 2024-04-20T21:54:00.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:00.459+0000 7fec6dc00700 1 -- 172.21.15.62:0/2598230697 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec6819d2c0 con 0x7fec680730f0 2024-04-20T21:54:00.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:00.460+0000 7fec6dc00700 1 -- 172.21.15.62:0/2598230697 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec68109cd0 con 0x7fec680730f0 2024-04-20T21:54:00.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:00.461+0000 7fec5fe00700 1 -- 172.21.15.62:0/2598230697 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fec58005020 con 0x7fec680730f0 2024-04-20T21:54:00.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:00.462+0000 7fec5fe00700 1 --2- 172.21.15.62:0/2598230697 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fec4c06ecc0 0x7fec4c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:54:00.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:00.462+0000 7fec5fe00700 1 -- 172.21.15.62:0/2598230697 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fec5808ea90 con 0x7fec680730f0 2024-04-20T21:54:00.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:00.462+0000 7fec5e000700 1 --2- 172.21.15.62:0/2598230697 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fec4c06ecc0 0x7fec4c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:54:00.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:00.463+0000 7fec5e000700 1 --2- 172.21.15.62:0/2598230697 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fec4c06ecc0 0x7fec4c071170 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fec50000c00 tx=0x7fec50005fd0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:54:00.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:00.467+0000 7fec5fe00700 1 -- 172.21.15.62:0/2598230697 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fec580577a0 con 0x7fec680730f0 2024-04-20T21:54:00.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:00.807+0000 7fec6dc00700 1 -- 172.21.15.62:0/2598230697 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fec681992d0 con 0x7fec680730f0 2024-04-20T21:54:00.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:00.808+0000 7fec5fe00700 1 -- 172.21.15.62:0/2598230697 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7fec5805ae60 con 0x7fec680730f0 2024-04-20T21:54:00.807 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:54:00.807 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T21:54:00.807 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:54:00.807 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:54:00.807 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T21:54:00.808 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:54:00.808 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:54:00.808 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T21:54:00.808 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 6 2024-04-20T21:54:00.808 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:54:00.808 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T21:54:00.808 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 2, 2024-04-20T21:54:00.808 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:54:00.808 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:54:00.808 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T21:54:00.808 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 2, 2024-04-20T21:54:00.808 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 12 2024-04-20T21:54:00.808 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T21:54:00.808 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:54:00.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:00.810+0000 7fec6dc00700 1 -- 172.21.15.62:0/2598230697 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fec4c06ecc0 msgr2=0x7fec4c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:54:00.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:00.810+0000 7fec6dc00700 1 --2- 172.21.15.62:0/2598230697 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fec4c06ecc0 0x7fec4c071170 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fec50000c00 tx=0x7fec50005fd0 comp rx=0 tx=0).stop 2024-04-20T21:54:00.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:00.810+0000 7fec6dc00700 1 -- 172.21.15.62:0/2598230697 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fec680730f0 msgr2=0x7fec68197fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:54:00.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:00.810+0000 7fec6dc00700 1 --2- 172.21.15.62:0/2598230697 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fec680730f0 0x7fec68197fc0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fec58007e90 tx=0x7fec5800d3b0 comp rx=0 tx=0).stop 2024-04-20T21:54:00.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:00.810+0000 7fec6dc00700 1 -- 172.21.15.62:0/2598230697 shutdown_connections 2024-04-20T21:54:00.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:00.810+0000 7fec6dc00700 1 --2- 172.21.15.62:0/2598230697 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fec4c06ecc0 0x7fec4c071170 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:54:00.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:00.810+0000 7fec6dc00700 1 --2- 172.21.15.62:0/2598230697 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fec680730f0 0x7fec68197fc0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:54:00.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:00.810+0000 7fec6dc00700 1 --2- 172.21.15.62:0/2598230697 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec68073a00 0x7fec68198630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:54:00.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:00.811+0000 7fec6dc00700 1 -- 172.21.15.62:0/2598230697 >> 172.21.15.62:0/2598230697 conn(0x7fec680fbe10 msgr2=0x7fec68106e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:54:00.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:00.811+0000 7fec6dc00700 1 -- 172.21.15.62:0/2598230697 shutdown_connections 2024-04-20T21:54:00.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:00.811+0000 7fec6dc00700 1 -- 172.21.15.62:0/2598230697 wait complete. 2024-04-20T21:54:00.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:00.963+0000 7fa8d5000700 1 -- 172.21.15.62:0/93263119 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa8d0106b80 msgr2=0x7fa8d0106ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:54:00.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:00.963+0000 7fa8d5000700 1 --2- 172.21.15.62:0/93263119 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa8d0106b80 0x7fa8d0106ff0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fa8bc009910 tx=0x7fa8bc009c20 comp rx=0 tx=0).stop 2024-04-20T21:54:00.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:00.963+0000 7fa8d5000700 1 -- 172.21.15.62:0/93263119 shutdown_connections 2024-04-20T21:54:00.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:00.963+0000 7fa8d5000700 1 --2- 172.21.15.62:0/93263119 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa8d0106b80 0x7fa8d0106ff0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:54:00.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:00.963+0000 7fa8d5000700 1 --2- 172.21.15.62:0/93263119 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8d010c8f0 0x7fa8d010ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:54:00.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:00.963+0000 7fa8d5000700 1 -- 172.21.15.62:0/93263119 >> 172.21.15.62:0/93263119 conn(0x7fa8d0076020 msgr2=0x7fa8d0078430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:54:00.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:00.963+0000 7fa8d5000700 1 -- 172.21.15.62:0/93263119 shutdown_connections 2024-04-20T21:54:00.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:00.963+0000 7fa8d5000700 1 -- 172.21.15.62:0/93263119 wait complete. 2024-04-20T21:54:00.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:00.964+0000 7fa8d5000700 1 Processor -- start 2024-04-20T21:54:00.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:00.964+0000 7fa8d5000700 1 -- start start 2024-04-20T21:54:00.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:00.964+0000 7fa8d5000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8d0106b80 0x7fa8d019c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:54:00.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:00.964+0000 7fa8d5000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa8d010c8f0 0x7fa8d019c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:54:00.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:00.965+0000 7fa8d5000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8d019d070 con 0x7fa8d010c8f0 2024-04-20T21:54:00.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:00.965+0000 7fa8d5000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8d01a0e00 con 0x7fa8d0106b80 2024-04-20T21:54:00.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:00.965+0000 7fa8cd400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa8d010c8f0 0x7fa8d019c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:54:00.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:00.965+0000 7fa8cd400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa8d010c8f0 0x7fa8d019c990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59916/0 (socket says 172.21.15.62:59916) 2024-04-20T21:54:00.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:00.965+0000 7fa8cd400700 1 -- 172.21.15.62:0/1127118616 learned_addr learned my addr 172.21.15.62:0/1127118616 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:54:00.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:00.965+0000 7fa8cd400700 1 -- 172.21.15.62:0/1127118616 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8d0106b80 msgr2=0x7fa8d019c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:54:00.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:00.965+0000 7fa8cde00700 1 --2- 172.21.15.62:0/1127118616 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8d0106b80 0x7fa8d019c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:54:00.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:00.965+0000 7fa8cd400700 1 --2- 172.21.15.62:0/1127118616 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8d0106b80 0x7fa8d019c450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:54:00.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:00.965+0000 7fa8cd400700 1 -- 172.21.15.62:0/1127118616 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8bc0095f0 con 0x7fa8d010c8f0 2024-04-20T21:54:00.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:00.966+0000 7fa8cd400700 1 --2- 172.21.15.62:0/1127118616 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa8d010c8f0 0x7fa8d019c990 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fa8bc00b5c0 tx=0x7fa8bc00be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:54:00.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:00.966+0000 7fa8c6a00700 1 -- 172.21.15.62:0/1127118616 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa8bc01d070 con 0x7fa8d010c8f0 2024-04-20T21:54:00.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:00.966+0000 7fa8c6a00700 1 -- 172.21.15.62:0/1127118616 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa8bc00f460 con 0x7fa8d010c8f0 2024-04-20T21:54:00.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:00.966+0000 7fa8c6a00700 1 -- 172.21.15.62:0/1127118616 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa8bc017610 con 0x7fa8d010c8f0 2024-04-20T21:54:00.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:00.966+0000 7fa8d5000700 1 -- 172.21.15.62:0/1127118616 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8d01a1080 con 0x7fa8d010c8f0 2024-04-20T21:54:00.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:00.966+0000 7fa8d5000700 1 -- 172.21.15.62:0/1127118616 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8d01a1570 con 0x7fa8d010c8f0 2024-04-20T21:54:00.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:00.967+0000 7fa8d5000700 1 -- 172.21.15.62:0/1127118616 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8d004f2a0 con 0x7fa8d010c8f0 2024-04-20T21:54:00.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:00.968+0000 7fa8c6a00700 1 -- 172.21.15.62:0/1127118616 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fa8bc00f5d0 con 0x7fa8d010c8f0 2024-04-20T21:54:00.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:00.969+0000 7fa8c6a00700 1 --2- 172.21.15.62:0/1127118616 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa8b406ecc0 0x7fa8b4071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:54:00.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:00.969+0000 7fa8c6a00700 1 -- 172.21.15.62:0/1127118616 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fa8bc090450 con 0x7fa8d010c8f0 2024-04-20T21:54:00.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:00.969+0000 7fa8cde00700 1 --2- 172.21.15.62:0/1127118616 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa8b406ecc0 0x7fa8b4071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:54:00.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:00.970+0000 7fa8cde00700 1 --2- 172.21.15.62:0/1127118616 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa8b406ecc0 0x7fa8b4071170 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fa8c0005950 tx=0x7fa8c0005cf0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:54:00.973 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:54:00 smithi062 ceph-mon[76322]: from='client.35040 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:54:00.973 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:54:00 smithi062 ceph-mon[76322]: from='client.35044 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:54:00.973 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:54:00 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/2598230697' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:54:00.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:00.974+0000 7fa8c6a00700 1 -- 172.21.15.62:0/1127118616 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fa8bc0591e0 con 0x7fa8d010c8f0 2024-04-20T21:54:01.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:54:00 smithi189 ceph-mon[57022]: from='client.35040 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:54:01.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:54:00 smithi189 ceph-mon[57022]: from='client.35044 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:54:01.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:54:00 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/2598230697' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:54:01.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:01.249+0000 7fa8d5000700 1 -- 172.21.15.62:0/1127118616 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fa8d019d7b0 con 0x7fa8d010c8f0 2024-04-20T21:54:01.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:01.250+0000 7fa8c6a00700 1 -- 172.21.15.62:0/1127118616 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 25 v25) v1 ==== 76+0+1887 (secure 0 0 0) 0x7fa8bc05c8a0 con 0x7fa8d010c8f0 2024-04-20T21:54:01.251 INFO:teuthology.orchestra.run.smithi062.stdout:e25 2024-04-20T21:54:01.251 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T21:54:01.251 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T21:54:01.251 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T21:54:01.251 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:54:01.251 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T21:54:01.251 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T21:54:01.251 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 20 2024-04-20T21:54:01.252 INFO:teuthology.orchestra.run.smithi062.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-20T21:54:01.252 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T21:54:01.252 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T21:42:01.559061+0000 2024-04-20T21:54:01.252 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T21:54:01.252 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T21:54:01.252 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T21:54:01.252 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T21:54:01.252 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T21:54:01.253 INFO:teuthology.orchestra.run.smithi062.stdout:max_xattr_size 65536 2024-04-20T21:54:01.253 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T21:54:01.253 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T21:54:01.253 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 81 2024-04-20T21:54:01.253 INFO:teuthology.orchestra.run.smithi062.stdout:snap_rstat 1 2024-04-20T21:54:01.253 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T21:54:01.253 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 2 2024-04-20T21:54:01.253 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1 2024-04-20T21:54:01.253 INFO:teuthology.orchestra.run.smithi062.stdout:up {} 2024-04-20T21:54:01.254 INFO:teuthology.orchestra.run.smithi062.stdout:failed 0,1 2024-04-20T21:54:01.254 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T21:54:01.254 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T21:54:01.254 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T21:54:01.254 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T21:54:01.254 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data enabled 2024-04-20T21:54:01.254 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T21:54:01.254 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T21:54:01.254 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 1 2024-04-20T21:54:01.254 INFO:teuthology.orchestra.run.smithi062.stdout:qdb_cluster leader: 0 members: 2024-04-20T21:54:01.255 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:54:01.255 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:54:01.255 INFO:teuthology.orchestra.run.smithi062.stdout:Standby daemons: 2024-04-20T21:54:01.255 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:54:01.255 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ryqblb{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6826/4113916205,v1:172.21.15.62:6827/4113916205] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T21:54:01.255 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ozizwi{-1:34476} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6828/3801975165,v1:172.21.15.62:6829/3801975165] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T21:54:01.255 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.xbgspw{-1:44285} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6826/3996040526,v1:172.21.15.189:6827/3996040526] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:54:01.255 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.dvimer{-1:44289} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6824/1743111843,v1:172.21.15.189:6825/1743111843] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:54:01.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:01.253+0000 7fa8d5000700 1 -- 172.21.15.62:0/1127118616 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa8b406ecc0 msgr2=0x7fa8b4071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:54:01.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:01.253+0000 7fa8d5000700 1 --2- 172.21.15.62:0/1127118616 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa8b406ecc0 0x7fa8b4071170 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fa8c0005950 tx=0x7fa8c0005cf0 comp rx=0 tx=0).stop 2024-04-20T21:54:01.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:01.253+0000 7fa8d5000700 1 -- 172.21.15.62:0/1127118616 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa8d010c8f0 msgr2=0x7fa8d019c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:54:01.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:01.253+0000 7fa8d5000700 1 --2- 172.21.15.62:0/1127118616 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa8d010c8f0 0x7fa8d019c990 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fa8bc00b5c0 tx=0x7fa8bc00be00 comp rx=0 tx=0).stop 2024-04-20T21:54:01.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:01.253+0000 7fa8d5000700 1 -- 172.21.15.62:0/1127118616 shutdown_connections 2024-04-20T21:54:01.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:01.253+0000 7fa8d5000700 1 --2- 172.21.15.62:0/1127118616 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa8b406ecc0 0x7fa8b4071170 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:54:01.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:01.253+0000 7fa8d5000700 1 --2- 172.21.15.62:0/1127118616 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8d0106b80 0x7fa8d019c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:54:01.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:01.253+0000 7fa8d5000700 1 --2- 172.21.15.62:0/1127118616 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa8d010c8f0 0x7fa8d019c990 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:54:01.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:01.253+0000 7fa8d5000700 1 -- 172.21.15.62:0/1127118616 >> 172.21.15.62:0/1127118616 conn(0x7fa8d0076020 msgr2=0x7fa8d0077a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:54:01.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:01.254+0000 7fa8d5000700 1 -- 172.21.15.62:0/1127118616 shutdown_connections 2024-04-20T21:54:01.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:01.254+0000 7fa8d5000700 1 -- 172.21.15.62:0/1127118616 wait complete. 2024-04-20T21:54:01.257 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 25 2024-04-20T21:54:01.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:01.412+0000 7f1b99e00700 1 -- 172.21.15.62:0/975020791 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1b94103ec0 msgr2=0x7f1b94104290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:54:01.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:01.412+0000 7f1b99e00700 1 --2- 172.21.15.62:0/975020791 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1b94103ec0 0x7f1b94104290 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f1b88009910 tx=0x7f1b88009c20 comp rx=0 tx=0).stop 2024-04-20T21:54:01.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:01.412+0000 7f1b99e00700 1 -- 172.21.15.62:0/975020791 shutdown_connections 2024-04-20T21:54:01.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:01.412+0000 7f1b99e00700 1 --2- 172.21.15.62:0/975020791 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1b940ff9d0 0x7f1b940ffe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:54:01.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:01.412+0000 7f1b99e00700 1 --2- 172.21.15.62:0/975020791 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1b94103ec0 0x7f1b94104290 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:54:01.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:01.412+0000 7f1b99e00700 1 -- 172.21.15.62:0/975020791 >> 172.21.15.62:0/975020791 conn(0x7f1b940f9d40 msgr2=0x7f1b940fc150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:54:01.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:01.412+0000 7f1b99e00700 1 -- 172.21.15.62:0/975020791 shutdown_connections 2024-04-20T21:54:01.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:01.412+0000 7f1b99e00700 1 -- 172.21.15.62:0/975020791 wait complete. 2024-04-20T21:54:01.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:01.413+0000 7f1b99e00700 1 Processor -- start 2024-04-20T21:54:01.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:01.414+0000 7f1b99e00700 1 -- start start 2024-04-20T21:54:01.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:01.414+0000 7f1b99e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1b940ff9d0 0x7f1b94197da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:54:01.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:01.414+0000 7f1b99e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1b94103ec0 0x7f1b941982e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:54:01.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:01.414+0000 7f1b99e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b941989c0 con 0x7f1b94103ec0 2024-04-20T21:54:01.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:01.414+0000 7f1b99e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b9419c750 con 0x7f1b940ff9d0 2024-04-20T21:54:01.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:01.414+0000 7f1b93e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1b94103ec0 0x7f1b941982e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:54:01.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:01.414+0000 7f1b93e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1b94103ec0 0x7f1b941982e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54296/0 (socket says 172.21.15.62:54296) 2024-04-20T21:54:01.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:01.414+0000 7f1b93e00700 1 -- 172.21.15.62:0/852076902 learned_addr learned my addr 172.21.15.62:0/852076902 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:54:01.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:01.415+0000 7f1b98a00700 1 --2- 172.21.15.62:0/852076902 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1b940ff9d0 0x7f1b94197da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:54:01.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:01.415+0000 7f1b93e00700 1 -- 172.21.15.62:0/852076902 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1b940ff9d0 msgr2=0x7f1b94197da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:54:01.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:01.415+0000 7f1b93e00700 1 --2- 172.21.15.62:0/852076902 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1b940ff9d0 0x7f1b94197da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:54:01.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:01.415+0000 7f1b93e00700 1 -- 172.21.15.62:0/852076902 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b880095f0 con 0x7f1b94103ec0 2024-04-20T21:54:01.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:01.415+0000 7f1b93e00700 1 --2- 172.21.15.62:0/852076902 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1b94103ec0 0x7f1b941982e0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f1b8000d8d0 tx=0x7f1b8000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:54:01.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:01.415+0000 7f1b91600700 1 -- 172.21.15.62:0/852076902 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1b80009940 con 0x7f1b94103ec0 2024-04-20T21:54:01.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:01.415+0000 7f1b91600700 1 -- 172.21.15.62:0/852076902 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1b80010460 con 0x7f1b94103ec0 2024-04-20T21:54:01.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:01.415+0000 7f1b91600700 1 -- 172.21.15.62:0/852076902 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1b8000f5f0 con 0x7f1b94103ec0 2024-04-20T21:54:01.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:01.415+0000 7f1b99e00700 1 -- 172.21.15.62:0/852076902 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b9419ca30 con 0x7f1b94103ec0 2024-04-20T21:54:01.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:01.416+0000 7f1b99e00700 1 -- 172.21.15.62:0/852076902 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b9419d000 con 0x7f1b94103ec0 2024-04-20T21:54:01.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:01.417+0000 7f1b99e00700 1 -- 172.21.15.62:0/852076902 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b94062380 con 0x7f1b94103ec0 2024-04-20T21:54:01.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:01.418+0000 7f1b91600700 1 -- 172.21.15.62:0/852076902 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f1b800105d0 con 0x7f1b94103ec0 2024-04-20T21:54:01.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:01.419+0000 7f1b91600700 1 --2- 172.21.15.62:0/852076902 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f1b7c06ecc0 0x7f1b7c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:54:01.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:01.419+0000 7f1b91600700 1 -- 172.21.15.62:0/852076902 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f1b8008ee10 con 0x7f1b94103ec0 2024-04-20T21:54:01.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:01.419+0000 7f1b98a00700 1 --2- 172.21.15.62:0/852076902 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f1b7c06ecc0 0x7f1b7c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:54:01.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:01.420+0000 7f1b98a00700 1 --2- 172.21.15.62:0/852076902 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f1b7c06ecc0 0x7f1b7c071170 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f1b88006010 tx=0x7f1b8800b540 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:54:01.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:01.423+0000 7f1b91600700 1 -- 172.21.15.62:0/852076902 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f1b80057ba0 con 0x7f1b94103ec0 2024-04-20T21:54:01.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:01.680+0000 7f1b99e00700 1 -- 172.21.15.62:0/852076902 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1b9419d530 con 0x7f1b7c06ecc0 2024-04-20T21:54:01.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:01.682+0000 7f1b91600700 1 -- 172.21.15.62:0/852076902 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f1b9419d530 con 0x7f1b7c06ecc0 2024-04-20T21:54:01.681 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:54:01.681 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T21:54:01.682 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T21:54:01.682 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-20T21:54:01.682 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T21:54:01.682 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr", 2024-04-20T21:54:01.682 INFO:teuthology.orchestra.run.smithi062.stdout: "crash", 2024-04-20T21:54:01.682 INFO:teuthology.orchestra.run.smithi062.stdout: "mon", 2024-04-20T21:54:01.682 INFO:teuthology.orchestra.run.smithi062.stdout: "osd" 2024-04-20T21:54:01.682 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T21:54:01.682 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "14/23 daemons upgraded", 2024-04-20T21:54:01.682 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading mds daemons", 2024-04-20T21:54:01.682 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T21:54:01.682 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:54:01.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:01.685+0000 7f1b99e00700 1 -- 172.21.15.62:0/852076902 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f1b7c06ecc0 msgr2=0x7f1b7c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:54:01.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:01.685+0000 7f1b99e00700 1 --2- 172.21.15.62:0/852076902 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f1b7c06ecc0 0x7f1b7c071170 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f1b88006010 tx=0x7f1b8800b540 comp rx=0 tx=0).stop 2024-04-20T21:54:01.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:01.685+0000 7f1b99e00700 1 -- 172.21.15.62:0/852076902 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1b94103ec0 msgr2=0x7f1b941982e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:54:01.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:01.685+0000 7f1b99e00700 1 --2- 172.21.15.62:0/852076902 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1b94103ec0 0x7f1b941982e0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f1b8000d8d0 tx=0x7f1b8000dc90 comp rx=0 tx=0).stop 2024-04-20T21:54:01.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:01.685+0000 7f1b99e00700 1 -- 172.21.15.62:0/852076902 shutdown_connections 2024-04-20T21:54:01.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:01.685+0000 7f1b99e00700 1 --2- 172.21.15.62:0/852076902 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f1b7c06ecc0 0x7f1b7c071170 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:54:01.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:01.685+0000 7f1b99e00700 1 --2- 172.21.15.62:0/852076902 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1b940ff9d0 0x7f1b94197da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:54:01.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:01.685+0000 7f1b99e00700 1 --2- 172.21.15.62:0/852076902 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1b94103ec0 0x7f1b941982e0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:54:01.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:01.685+0000 7f1b99e00700 1 -- 172.21.15.62:0/852076902 >> 172.21.15.62:0/852076902 conn(0x7f1b940f9d40 msgr2=0x7f1b940fab40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:54:01.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:01.686+0000 7f1b99e00700 1 -- 172.21.15.62:0/852076902 shutdown_connections 2024-04-20T21:54:01.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:01.686+0000 7f1b99e00700 1 -- 172.21.15.62:0/852076902 wait complete. 2024-04-20T21:54:01.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:01.835+0000 7ff2c0a00700 1 -- 172.21.15.62:0/3190482600 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff2bc102470 msgr2=0x7ff2bc1028e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:54:01.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:01.835+0000 7ff2c0a00700 1 --2- 172.21.15.62:0/3190482600 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff2bc102470 0x7ff2bc1028e0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7ff2a8009910 tx=0x7ff2a8009c20 comp rx=0 tx=0).stop 2024-04-20T21:54:01.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:01.835+0000 7ff2c0a00700 1 -- 172.21.15.62:0/3190482600 shutdown_connections 2024-04-20T21:54:01.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:01.835+0000 7ff2c0a00700 1 --2- 172.21.15.62:0/3190482600 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff2bc102470 0x7ff2bc1028e0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:54:01.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:01.835+0000 7ff2c0a00700 1 --2- 172.21.15.62:0/3190482600 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2bc108470 0x7ff2bc108840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:54:01.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:01.835+0000 7ff2c0a00700 1 -- 172.21.15.62:0/3190482600 >> 172.21.15.62:0/3190482600 conn(0x7ff2bc0fdf90 msgr2=0x7ff2bc1003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:54:01.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:01.835+0000 7ff2c0a00700 1 -- 172.21.15.62:0/3190482600 shutdown_connections 2024-04-20T21:54:01.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:01.835+0000 7ff2c0a00700 1 -- 172.21.15.62:0/3190482600 wait complete. 2024-04-20T21:54:01.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:01.836+0000 7ff2c0a00700 1 Processor -- start 2024-04-20T21:54:01.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:01.836+0000 7ff2c0a00700 1 -- start start 2024-04-20T21:54:01.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:01.837+0000 7ff2c0a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff2bc102470 0x7ff2bc198270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:54:01.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:01.837+0000 7ff2c0a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2bc108470 0x7ff2bc1987b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:54:01.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:01.837+0000 7ff2c0a00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2bc198e90 con 0x7ff2bc102470 2024-04-20T21:54:01.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:01.837+0000 7ff2c0a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2bc19cbd0 con 0x7ff2bc108470 2024-04-20T21:54:01.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:01.837+0000 7ff2b9e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff2bc102470 0x7ff2bc198270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:54:01.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:01.837+0000 7ff2b9e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff2bc102470 0x7ff2bc198270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54316/0 (socket says 172.21.15.62:54316) 2024-04-20T21:54:01.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:01.837+0000 7ff2b9e00700 1 -- 172.21.15.62:0/272050831 learned_addr learned my addr 172.21.15.62:0/272050831 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:54:01.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:01.837+0000 7ff2b9400700 1 --2- 172.21.15.62:0/272050831 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2bc108470 0x7ff2bc1987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:54:01.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:01.837+0000 7ff2b9e00700 1 -- 172.21.15.62:0/272050831 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2bc108470 msgr2=0x7ff2bc1987b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:54:01.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:01.837+0000 7ff2b9e00700 1 --2- 172.21.15.62:0/272050831 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2bc108470 0x7ff2bc1987b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:54:01.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:01.837+0000 7ff2b9e00700 1 -- 172.21.15.62:0/272050831 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2a80095f0 con 0x7ff2bc102470 2024-04-20T21:54:01.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:01.838+0000 7ff2b9e00700 1 --2- 172.21.15.62:0/272050831 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff2bc102470 0x7ff2bc198270 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7ff2ac00c8a0 tx=0x7ff2ac00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:54:01.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:01.838+0000 7ff2b2a00700 1 -- 172.21.15.62:0/272050831 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff2ac00cea0 con 0x7ff2bc102470 2024-04-20T21:54:01.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:01.838+0000 7ff2b2a00700 1 -- 172.21.15.62:0/272050831 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff2ac004830 con 0x7ff2bc102470 2024-04-20T21:54:01.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:01.838+0000 7ff2b2a00700 1 -- 172.21.15.62:0/272050831 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff2ac0055b0 con 0x7ff2bc102470 2024-04-20T21:54:01.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:01.838+0000 7ff2c0a00700 1 -- 172.21.15.62:0/272050831 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2bc19ceb0 con 0x7ff2bc102470 2024-04-20T21:54:01.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:01.838+0000 7ff2c0a00700 1 -- 172.21.15.62:0/272050831 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2bc19d400 con 0x7ff2bc102470 2024-04-20T21:54:01.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:01.839+0000 7ff2c0a00700 1 -- 172.21.15.62:0/272050831 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2bc10a970 con 0x7ff2bc102470 2024-04-20T21:54:01.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:01.841+0000 7ff2b2a00700 1 -- 172.21.15.62:0/272050831 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff2ac0049a0 con 0x7ff2bc102470 2024-04-20T21:54:01.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:01.841+0000 7ff2b2a00700 1 --2- 172.21.15.62:0/272050831 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff2a006ecc0 0x7ff2a0071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:54:01.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:01.842+0000 7ff2b2a00700 1 -- 172.21.15.62:0/272050831 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7ff2ac08eaf0 con 0x7ff2bc102470 2024-04-20T21:54:01.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:01.842+0000 7ff2b9400700 1 --2- 172.21.15.62:0/272050831 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff2a006ecc0 0x7ff2a0071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:54:01.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:01.842+0000 7ff2b9400700 1 --2- 172.21.15.62:0/272050831 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff2a006ecc0 0x7ff2a0071170 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7ff2a8005f50 tx=0x7ff2a801a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:54:01.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:01.846+0000 7ff2b2a00700 1 -- 172.21.15.62:0/272050831 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7ff2ac057880 con 0x7ff2bc102470 2024-04-20T21:54:02.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:54:01 smithi062 ceph-mon[76322]: from='client.35048 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:54:02.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:54:01 smithi062 ceph-mon[76322]: pgmap v561: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:54:02.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:54:01 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/1127118616' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T21:54:02.139 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:54:01 smithi189 ceph-mon[57022]: from='client.35048 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:54:02.139 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:54:01 smithi189 ceph-mon[57022]: pgmap v561: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:54:02.139 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:54:01 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/1127118616' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T21:54:02.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:02.164+0000 7ff2c0a00700 1 -- 172.21.15.62:0/272050831 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7ff2bc04f2a0 con 0x7ff2bc102470 2024-04-20T21:54:02.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:02.164+0000 7ff2b2a00700 1 -- 172.21.15.62:0/272050831 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7ff2ac01d090 con 0x7ff2bc102470 2024-04-20T21:54:02.164 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-20T21:54:02.164 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-20T21:54:02.164 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is degraded 2024-04-20T21:54:02.164 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T21:54:02.164 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T21:54:02.164 INFO:teuthology.orchestra.run.smithi062.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-20T21:54:02.164 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-20T21:54:02.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:02.167+0000 7ff2c0a00700 1 -- 172.21.15.62:0/272050831 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff2a006ecc0 msgr2=0x7ff2a0071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:54:02.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:02.167+0000 7ff2c0a00700 1 --2- 172.21.15.62:0/272050831 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff2a006ecc0 0x7ff2a0071170 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7ff2a8005f50 tx=0x7ff2a801a040 comp rx=0 tx=0).stop 2024-04-20T21:54:02.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:02.167+0000 7ff2c0a00700 1 -- 172.21.15.62:0/272050831 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff2bc102470 msgr2=0x7ff2bc198270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:54:02.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:02.167+0000 7ff2c0a00700 1 --2- 172.21.15.62:0/272050831 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff2bc102470 0x7ff2bc198270 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7ff2ac00c8a0 tx=0x7ff2ac00cc60 comp rx=0 tx=0).stop 2024-04-20T21:54:02.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:02.167+0000 7ff2c0a00700 1 -- 172.21.15.62:0/272050831 shutdown_connections 2024-04-20T21:54:02.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:02.167+0000 7ff2c0a00700 1 --2- 172.21.15.62:0/272050831 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff2a006ecc0 0x7ff2a0071170 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:54:02.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:02.167+0000 7ff2c0a00700 1 --2- 172.21.15.62:0/272050831 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff2bc102470 0x7ff2bc198270 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:54:02.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:02.167+0000 7ff2c0a00700 1 --2- 172.21.15.62:0/272050831 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2bc108470 0x7ff2bc1987b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:54:02.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:02.167+0000 7ff2c0a00700 1 -- 172.21.15.62:0/272050831 >> 172.21.15.62:0/272050831 conn(0x7ff2bc0fdf90 msgr2=0x7ff2bc0ff470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:54:02.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:02.167+0000 7ff2c0a00700 1 -- 172.21.15.62:0/272050831 shutdown_connections 2024-04-20T21:54:02.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:02.168+0000 7ff2c0a00700 1 -- 172.21.15.62:0/272050831 wait complete. 2024-04-20T21:54:03.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:54:02 smithi189 ceph-mon[57022]: from='client.35060 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:54:03.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:54:02 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/272050831' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T21:54:03.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:54:02 smithi062 ceph-mon[76322]: from='client.35060 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:54:03.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:54:02 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/272050831' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T21:54:04.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:54:03 smithi189 ceph-mon[57022]: pgmap v562: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:54:04.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:54:03 smithi062 ceph-mon[76322]: pgmap v562: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:54:05.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:54:04 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:54:05.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:54:04 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:54:06.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:54:05 smithi189 ceph-mon[57022]: pgmap v563: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:54:06.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:54:05 smithi062 ceph-mon[76322]: pgmap v563: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:54:08.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:54:07 smithi189 ceph-mon[57022]: pgmap v564: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:54:08.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:54:07 smithi062 ceph-mon[76322]: pgmap v564: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:54:10.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:54:09 smithi189 ceph-mon[57022]: pgmap v565: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:54:10.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:54:09 smithi062 ceph-mon[76322]: pgmap v565: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:54:12.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:54:11 smithi189 ceph-mon[57022]: pgmap v566: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:54:12.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:54:11 smithi062 ceph-mon[76322]: pgmap v566: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:54:14.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:54:13 smithi189 ceph-mon[57022]: pgmap v567: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:54:14.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:54:13 smithi062 ceph-mon[76322]: pgmap v567: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:54:16.224 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:54:15 smithi062 ceph-mon[76322]: pgmap v568: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:54:16.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:54:15 smithi189 ceph-mon[57022]: pgmap v568: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:54:18.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:54:17 smithi189 ceph-mon[57022]: pgmap v569: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:54:18.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:54:17 smithi062 ceph-mon[76322]: pgmap v569: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:54:20.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:54:19 smithi189 ceph-mon[57022]: pgmap v570: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:54:20.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:54:19 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:54:20.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:54:19 smithi062 ceph-mon[76322]: pgmap v570: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:54:20.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:54:19 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:54:22.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:54:21 smithi189 ceph-mon[57022]: pgmap v571: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:54:22.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:54:21 smithi062 ceph-mon[76322]: pgmap v571: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:54:24.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:54:23 smithi189 ceph-mon[57022]: pgmap v572: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:54:24.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:54:23 smithi062 ceph-mon[76322]: pgmap v572: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:54:25.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:54:25 smithi189 ceph-mon[57022]: pgmap v573: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:54:25.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:54:25 smithi062 ceph-mon[76322]: pgmap v573: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:54:28.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:54:27 smithi062 ceph-mon[76322]: pgmap v574: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:54:28.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:54:27 smithi189 ceph-mon[57022]: pgmap v574: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:54:30.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:54:29 smithi062 ceph-mon[76322]: pgmap v575: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:54:30.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:54:29 smithi189 ceph-mon[57022]: pgmap v575: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:54:32.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:54:31 smithi062 ceph-mon[76322]: pgmap v576: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:54:32.040 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:54:31 smithi189 ceph-mon[57022]: pgmap v576: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:54:32.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:32.344+0000 7f8835c00700 1 -- 172.21.15.62:0/784474213 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f88301004e0 msgr2=0x7f88301008b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:54:32.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:32.344+0000 7f8835c00700 1 --2- 172.21.15.62:0/784474213 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f88301004e0 0x7f88301008b0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f8820009910 tx=0x7f8820009c20 comp rx=0 tx=0).stop 2024-04-20T21:54:32.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:32.344+0000 7f8835c00700 1 -- 172.21.15.62:0/784474213 shutdown_connections 2024-04-20T21:54:32.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:32.344+0000 7f8835c00700 1 --2- 172.21.15.62:0/784474213 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8830100e80 0x7f8830104e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:54:32.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:32.344+0000 7f8835c00700 1 --2- 172.21.15.62:0/784474213 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f88301004e0 0x7f88301008b0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:54:32.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:32.344+0000 7f8835c00700 1 -- 172.21.15.62:0/784474213 >> 172.21.15.62:0/784474213 conn(0x7f88300fbda0 msgr2=0x7f88300fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:54:32.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:32.344+0000 7f8835c00700 1 -- 172.21.15.62:0/784474213 shutdown_connections 2024-04-20T21:54:32.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:32.344+0000 7f8835c00700 1 -- 172.21.15.62:0/784474213 wait complete. 2024-04-20T21:54:32.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:32.345+0000 7f8835c00700 1 Processor -- start 2024-04-20T21:54:32.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:32.345+0000 7f8835c00700 1 -- start start 2024-04-20T21:54:32.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:32.346+0000 7f8835c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88301004e0 0x7f8830195ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:54:32.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:32.346+0000 7f8835c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8830100e80 0x7f8830196430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:54:32.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:32.346+0000 7f8835c00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8830196b10 con 0x7f8830100e80 2024-04-20T21:54:32.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:32.346+0000 7f8835c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f883019a8a0 con 0x7f88301004e0 2024-04-20T21:54:32.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:32.346+0000 7f882e000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8830100e80 0x7f8830196430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:54:32.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:32.346+0000 7f882e000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8830100e80 0x7f8830196430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45640/0 (socket says 172.21.15.62:45640) 2024-04-20T21:54:32.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:32.346+0000 7f882e000700 1 -- 172.21.15.62:0/4201228529 learned_addr learned my addr 172.21.15.62:0/4201228529 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:54:32.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:32.346+0000 7f882ea00700 1 --2- 172.21.15.62:0/4201228529 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88301004e0 0x7f8830195ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:54:32.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:32.347+0000 7f882e000700 1 -- 172.21.15.62:0/4201228529 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88301004e0 msgr2=0x7f8830195ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:54:32.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:32.347+0000 7f882e000700 1 --2- 172.21.15.62:0/4201228529 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88301004e0 0x7f8830195ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:54:32.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:32.347+0000 7f882e000700 1 -- 172.21.15.62:0/4201228529 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88200095f0 con 0x7f8830100e80 2024-04-20T21:54:32.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:32.347+0000 7f882ea00700 1 --2- 172.21.15.62:0/4201228529 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88301004e0 0x7f8830195ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-20T21:54:32.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:32.347+0000 7f882e000700 1 --2- 172.21.15.62:0/4201228529 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8830100e80 0x7f8830196430 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f881c00c8f0 tx=0x7f881c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:54:32.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:32.347+0000 7f8827400700 1 -- 172.21.15.62:0/4201228529 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f881c007a50 con 0x7f8830100e80 2024-04-20T21:54:32.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:32.347+0000 7f8827400700 1 -- 172.21.15.62:0/4201228529 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f881c007bb0 con 0x7f8830100e80 2024-04-20T21:54:32.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:32.347+0000 7f8827400700 1 -- 172.21.15.62:0/4201228529 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f881c018630 con 0x7f8830100e80 2024-04-20T21:54:32.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:32.347+0000 7f8835c00700 1 -- 172.21.15.62:0/4201228529 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f883019ab80 con 0x7f8830100e80 2024-04-20T21:54:32.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:32.348+0000 7f8835c00700 1 -- 172.21.15.62:0/4201228529 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f883019b0d0 con 0x7f8830100e80 2024-04-20T21:54:32.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:32.349+0000 7f8835c00700 1 -- 172.21.15.62:0/4201228529 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88301087f0 con 0x7f8830100e80 2024-04-20T21:54:32.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:32.350+0000 7f8827400700 1 -- 172.21.15.62:0/4201228529 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f881c007d20 con 0x7f8830100e80 2024-04-20T21:54:32.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:32.351+0000 7f8827400700 1 --2- 172.21.15.62:0/4201228529 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f881406ecc0 0x7f8814071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:54:32.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:32.351+0000 7f8827400700 1 -- 172.21.15.62:0/4201228529 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f881c08ee00 con 0x7f8830100e80 2024-04-20T21:54:32.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:32.351+0000 7f882ea00700 1 --2- 172.21.15.62:0/4201228529 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f881406ecc0 0x7f8814071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:54:32.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:32.352+0000 7f882ea00700 1 --2- 172.21.15.62:0/4201228529 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f881406ecc0 0x7f8814071170 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f882000b5c0 tx=0x7f882001a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:54:32.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:32.356+0000 7f8827400700 1 -- 172.21.15.62:0/4201228529 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f881c057b90 con 0x7f8830100e80 2024-04-20T21:54:32.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:32.610+0000 7f8835c00700 1 -- 172.21.15.62:0/4201228529 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f883019b410 con 0x7f881406ecc0 2024-04-20T21:54:32.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:32.613+0000 7f8827400700 1 -- 172.21.15.62:0/4201228529 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f883019b410 con 0x7f881406ecc0 2024-04-20T21:54:32.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:32.615+0000 7f8835c00700 1 -- 172.21.15.62:0/4201228529 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f881406ecc0 msgr2=0x7f8814071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:54:32.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:32.615+0000 7f8835c00700 1 --2- 172.21.15.62:0/4201228529 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f881406ecc0 0x7f8814071170 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f882000b5c0 tx=0x7f882001a040 comp rx=0 tx=0).stop 2024-04-20T21:54:32.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:32.615+0000 7f8835c00700 1 -- 172.21.15.62:0/4201228529 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8830100e80 msgr2=0x7f8830196430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:54:32.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:32.615+0000 7f8835c00700 1 --2- 172.21.15.62:0/4201228529 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8830100e80 0x7f8830196430 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f881c00c8f0 tx=0x7f881c00ccb0 comp rx=0 tx=0).stop 2024-04-20T21:54:32.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:32.616+0000 7f8835c00700 1 -- 172.21.15.62:0/4201228529 shutdown_connections 2024-04-20T21:54:32.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:32.616+0000 7f8835c00700 1 --2- 172.21.15.62:0/4201228529 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f881406ecc0 0x7f8814071170 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:54:32.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:32.616+0000 7f8835c00700 1 --2- 172.21.15.62:0/4201228529 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88301004e0 0x7f8830195ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:54:32.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:32.616+0000 7f8835c00700 1 --2- 172.21.15.62:0/4201228529 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8830100e80 0x7f8830196430 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:54:32.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:32.616+0000 7f8835c00700 1 -- 172.21.15.62:0/4201228529 >> 172.21.15.62:0/4201228529 conn(0x7f88300fbda0 msgr2=0x7f88300fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:54:32.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:32.616+0000 7f8835c00700 1 -- 172.21.15.62:0/4201228529 shutdown_connections 2024-04-20T21:54:32.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:32.616+0000 7f8835c00700 1 -- 172.21.15.62:0/4201228529 wait complete. 2024-04-20T21:54:32.632 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T21:54:32.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:32.782+0000 7f32a2c00700 1 -- 172.21.15.62:0/2977082124 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f329c108470 msgr2=0x7f329c108840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:54:32.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:32.782+0000 7f32a2c00700 1 --2- 172.21.15.62:0/2977082124 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f329c108470 0x7f329c108840 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f328c009910 tx=0x7f328c009c20 comp rx=0 tx=0).stop 2024-04-20T21:54:32.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:32.783+0000 7f32a2c00700 1 -- 172.21.15.62:0/2977082124 shutdown_connections 2024-04-20T21:54:32.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:32.783+0000 7f32a2c00700 1 --2- 172.21.15.62:0/2977082124 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f329c102470 0x7f329c1028e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:54:32.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:32.783+0000 7f32a2c00700 1 --2- 172.21.15.62:0/2977082124 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f329c108470 0x7f329c108840 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:54:32.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:32.783+0000 7f32a2c00700 1 -- 172.21.15.62:0/2977082124 >> 172.21.15.62:0/2977082124 conn(0x7f329c0fdf90 msgr2=0x7f329c1003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:54:32.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:32.783+0000 7f32a2c00700 1 -- 172.21.15.62:0/2977082124 shutdown_connections 2024-04-20T21:54:32.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:32.783+0000 7f32a2c00700 1 -- 172.21.15.62:0/2977082124 wait complete. 2024-04-20T21:54:32.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:32.784+0000 7f32a2c00700 1 Processor -- start 2024-04-20T21:54:32.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:32.785+0000 7f32a2c00700 1 -- start start 2024-04-20T21:54:32.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:32.785+0000 7f32a2c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f329c102470 0x7f329c198210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:54:32.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:32.785+0000 7f32a2c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f329c108470 0x7f329c198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:54:32.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:32.785+0000 7f32a2c00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f329c198e30 con 0x7f329c102470 2024-04-20T21:54:32.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:32.785+0000 7f32a2c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f329c19cbc0 con 0x7f329c108470 2024-04-20T21:54:32.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:32.785+0000 7f329be00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f329c102470 0x7f329c198210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:54:32.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:32.785+0000 7f329be00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f329c102470 0x7f329c198210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45666/0 (socket says 172.21.15.62:45666) 2024-04-20T21:54:32.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:32.786+0000 7f329be00700 1 -- 172.21.15.62:0/3186162605 learned_addr learned my addr 172.21.15.62:0/3186162605 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:54:32.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:32.786+0000 7f329b400700 1 --2- 172.21.15.62:0/3186162605 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f329c108470 0x7f329c198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:54:32.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:32.786+0000 7f329be00700 1 -- 172.21.15.62:0/3186162605 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f329c108470 msgr2=0x7f329c198750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:54:32.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:32.786+0000 7f329be00700 1 --2- 172.21.15.62:0/3186162605 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f329c108470 0x7f329c198750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:54:32.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:32.786+0000 7f329be00700 1 -- 172.21.15.62:0/3186162605 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f328c0095f0 con 0x7f329c102470 2024-04-20T21:54:32.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:32.786+0000 7f329b400700 1 --2- 172.21.15.62:0/3186162605 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f329c108470 0x7f329c198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T21:54:32.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:32.786+0000 7f329be00700 1 --2- 172.21.15.62:0/3186162605 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f329c102470 0x7f329c198210 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f328c00b5c0 tx=0x7f328c00be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:54:32.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:32.787+0000 7f3298c00700 1 -- 172.21.15.62:0/3186162605 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f328c01d070 con 0x7f329c102470 2024-04-20T21:54:32.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:32.787+0000 7f3298c00700 1 -- 172.21.15.62:0/3186162605 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f328c00f460 con 0x7f329c102470 2024-04-20T21:54:32.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:32.787+0000 7f3298c00700 1 -- 172.21.15.62:0/3186162605 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f328c017610 con 0x7f329c102470 2024-04-20T21:54:32.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:32.787+0000 7f32a2c00700 1 -- 172.21.15.62:0/3186162605 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f329c19ce40 con 0x7f329c102470 2024-04-20T21:54:32.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:32.787+0000 7f32a2c00700 1 -- 172.21.15.62:0/3186162605 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f329c19d330 con 0x7f329c102470 2024-04-20T21:54:32.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:32.788+0000 7f32a2c00700 1 -- 172.21.15.62:0/3186162605 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f329c10a9e0 con 0x7f329c102470 2024-04-20T21:54:32.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:32.790+0000 7f3298c00700 1 -- 172.21.15.62:0/3186162605 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f328c00f5d0 con 0x7f329c102470 2024-04-20T21:54:32.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:32.791+0000 7f3298c00700 1 --2- 172.21.15.62:0/3186162605 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f328006ecc0 0x7f3280071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:54:32.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:32.791+0000 7f3298c00700 1 -- 172.21.15.62:0/3186162605 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f328c090660 con 0x7f329c102470 2024-04-20T21:54:32.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:32.792+0000 7f329b400700 1 --2- 172.21.15.62:0/3186162605 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f328006ecc0 0x7f3280071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:54:32.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:32.793+0000 7f329b400700 1 --2- 172.21.15.62:0/3186162605 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f328006ecc0 0x7f3280071170 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f329c199830 tx=0x7f3288008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:54:32.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:32.795+0000 7f3298c00700 1 -- 172.21.15.62:0/3186162605 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f328c059470 con 0x7f329c102470 2024-04-20T21:54:33.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.054+0000 7f32a2c00700 1 -- 172.21.15.62:0/3186162605 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f329c19d610 con 0x7f328006ecc0 2024-04-20T21:54:33.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.056+0000 7f3298c00700 1 -- 172.21.15.62:0/3186162605 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f329c19d610 con 0x7f328006ecc0 2024-04-20T21:54:33.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.059+0000 7f32a2c00700 1 -- 172.21.15.62:0/3186162605 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f328006ecc0 msgr2=0x7f3280071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:54:33.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.059+0000 7f32a2c00700 1 --2- 172.21.15.62:0/3186162605 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f328006ecc0 0x7f3280071170 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f329c199830 tx=0x7f3288008040 comp rx=0 tx=0).stop 2024-04-20T21:54:33.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.059+0000 7f32a2c00700 1 -- 172.21.15.62:0/3186162605 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f329c102470 msgr2=0x7f329c198210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:54:33.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.059+0000 7f32a2c00700 1 --2- 172.21.15.62:0/3186162605 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f329c102470 0x7f329c198210 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f328c00b5c0 tx=0x7f328c00be00 comp rx=0 tx=0).stop 2024-04-20T21:54:33.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.060+0000 7f32a2c00700 1 -- 172.21.15.62:0/3186162605 shutdown_connections 2024-04-20T21:54:33.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.060+0000 7f32a2c00700 1 --2- 172.21.15.62:0/3186162605 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f328006ecc0 0x7f3280071170 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:54:33.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.060+0000 7f32a2c00700 1 --2- 172.21.15.62:0/3186162605 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f329c102470 0x7f329c198210 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:54:33.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.060+0000 7f32a2c00700 1 --2- 172.21.15.62:0/3186162605 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f329c108470 0x7f329c198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:54:33.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.060+0000 7f32a2c00700 1 -- 172.21.15.62:0/3186162605 >> 172.21.15.62:0/3186162605 conn(0x7f329c0fdf90 msgr2=0x7f329c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:54:33.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.060+0000 7f32a2c00700 1 -- 172.21.15.62:0/3186162605 shutdown_connections 2024-04-20T21:54:33.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.060+0000 7f32a2c00700 1 -- 172.21.15.62:0/3186162605 wait complete. 2024-04-20T21:54:33.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.228+0000 7f0f92c00700 1 -- 172.21.15.62:0/129854987 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0f8c106b80 msgr2=0x7f0f8c106ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:54:33.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.228+0000 7f0f92c00700 1 --2- 172.21.15.62:0/129854987 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0f8c106b80 0x7f0f8c106ff0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f0f78009910 tx=0x7f0f78009c20 comp rx=0 tx=0).stop 2024-04-20T21:54:33.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.228+0000 7f0f92c00700 1 -- 172.21.15.62:0/129854987 shutdown_connections 2024-04-20T21:54:33.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.228+0000 7f0f92c00700 1 --2- 172.21.15.62:0/129854987 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0f8c106b80 0x7f0f8c106ff0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:54:33.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.228+0000 7f0f92c00700 1 --2- 172.21.15.62:0/129854987 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f8c10c8f0 0x7f0f8c10ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:54:33.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.228+0000 7f0f92c00700 1 -- 172.21.15.62:0/129854987 >> 172.21.15.62:0/129854987 conn(0x7f0f8c076060 msgr2=0x7f0f8c078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:54:33.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.228+0000 7f0f92c00700 1 -- 172.21.15.62:0/129854987 shutdown_connections 2024-04-20T21:54:33.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.228+0000 7f0f92c00700 1 -- 172.21.15.62:0/129854987 wait complete. 2024-04-20T21:54:33.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.229+0000 7f0f92c00700 1 Processor -- start 2024-04-20T21:54:33.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.230+0000 7f0f92c00700 1 -- start start 2024-04-20T21:54:33.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.230+0000 7f0f92c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f8c106b80 0x7f0f8c19c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:54:33.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.230+0000 7f0f92c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0f8c10c8f0 0x7f0f8c19c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:54:33.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.230+0000 7f0f92c00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f8c19d060 con 0x7f0f8c10c8f0 2024-04-20T21:54:33.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.230+0000 7f0f92c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f8c1a0df0 con 0x7f0f8c106b80 2024-04-20T21:54:33.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.230+0000 7f0f8b400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0f8c10c8f0 0x7f0f8c19c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:54:33.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.230+0000 7f0f8b400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0f8c10c8f0 0x7f0f8c19c980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45686/0 (socket says 172.21.15.62:45686) 2024-04-20T21:54:33.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.230+0000 7f0f8b400700 1 -- 172.21.15.62:0/1529636154 learned_addr learned my addr 172.21.15.62:0/1529636154 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:54:33.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.231+0000 7f0f8be00700 1 --2- 172.21.15.62:0/1529636154 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f8c106b80 0x7f0f8c19c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:54:33.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.231+0000 7f0f8b400700 1 -- 172.21.15.62:0/1529636154 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f8c106b80 msgr2=0x7f0f8c19c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:54:33.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.231+0000 7f0f8b400700 1 --2- 172.21.15.62:0/1529636154 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f8c106b80 0x7f0f8c19c440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:54:33.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.231+0000 7f0f8b400700 1 -- 172.21.15.62:0/1529636154 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f780095f0 con 0x7f0f8c10c8f0 2024-04-20T21:54:33.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.231+0000 7f0f8b400700 1 --2- 172.21.15.62:0/1529636154 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0f8c10c8f0 0x7f0f8c19c980 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f0f780092e0 tx=0x7f0f7800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:54:33.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.231+0000 7f0f88c00700 1 -- 172.21.15.62:0/1529636154 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0f7801d070 con 0x7f0f8c10c8f0 2024-04-20T21:54:33.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.232+0000 7f0f88c00700 1 -- 172.21.15.62:0/1529636154 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0f78005e00 con 0x7f0f8c10c8f0 2024-04-20T21:54:33.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.232+0000 7f0f88c00700 1 -- 172.21.15.62:0/1529636154 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0f78017400 con 0x7f0f8c10c8f0 2024-04-20T21:54:33.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.232+0000 7f0f92c00700 1 -- 172.21.15.62:0/1529636154 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f8c1a1070 con 0x7f0f8c10c8f0 2024-04-20T21:54:33.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.232+0000 7f0f92c00700 1 -- 172.21.15.62:0/1529636154 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f8c1a1560 con 0x7f0f8c10c8f0 2024-04-20T21:54:33.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.233+0000 7f0f92c00700 1 -- 172.21.15.62:0/1529636154 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f8c10ecc0 con 0x7f0f8c10c8f0 2024-04-20T21:54:33.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.234+0000 7f0f88c00700 1 -- 172.21.15.62:0/1529636154 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f0f7800f460 con 0x7f0f8c10c8f0 2024-04-20T21:54:33.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.235+0000 7f0f88c00700 1 --2- 172.21.15.62:0/1529636154 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f0f7006ecc0 0x7f0f70071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:54:33.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.235+0000 7f0f88c00700 1 -- 172.21.15.62:0/1529636154 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f0f78090420 con 0x7f0f8c10c8f0 2024-04-20T21:54:33.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.235+0000 7f0f8be00700 1 --2- 172.21.15.62:0/1529636154 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f0f7006ecc0 0x7f0f70071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:54:33.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.236+0000 7f0f8be00700 1 --2- 172.21.15.62:0/1529636154 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f0f7006ecc0 0x7f0f70071170 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f0f7c005950 tx=0x7f0f7c0058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:54:33.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.240+0000 7f0f88c00700 1 -- 172.21.15.62:0/1529636154 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f0f78059130 con 0x7f0f8c10c8f0 2024-04-20T21:54:33.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.487+0000 7f0f92c00700 1 -- 172.21.15.62:0/1529636154 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f0f8c1a1840 con 0x7f0f7006ecc0 2024-04-20T21:54:33.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.502+0000 7f0f88c00700 1 -- 172.21.15.62:0/1529636154 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f0f8c1a1840 con 0x7f0f7006ecc0 2024-04-20T21:54:33.502 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T21:54:33.502 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (20m) 11m ago 34m 14.2M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T21:54:33.502 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (34m) 11m ago 34m 9512k - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T21:54:33.502 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (33m) 12m ago 33m 8942k - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T21:54:33.502 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (16m) 11m ago 34m 6660k - 19.0.0-3133-ga8c84f59 76bc6b91b726 2825d5ba11c6 2024-04-20T21:54:33.502 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (16m) 12m ago 33m 6665k - 19.0.0-3133-ga8c84f59 76bc6b91b726 20f746fa1bba 2024-04-20T21:54:33.503 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (19m) 11m ago 33m 74.2M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T21:54:33.503 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (11m) 11m ago 26m 11.2M - 19.0.0-3133-ga8c84f59 76bc6b91b726 de80bc90a683 2024-04-20T21:54:33.503 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (12m) 11m ago 27m 11.9M - 19.0.0-3133-ga8c84f59 76bc6b91b726 cd793b7ee98e 2024-04-20T21:54:33.503 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (27m) 12m ago 27m 117M - 18.2.0 dc2bc1663786 fb2a8a223db8 2024-04-20T21:54:33.503 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (26m) 12m ago 26m 55.6M - 18.2.0 dc2bc1663786 f2eec2ee8951 2024-04-20T21:54:33.503 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (23m) 11m ago 36m 511M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T21:54:33.503 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (21m) 12m ago 32m 452M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T21:54:33.504 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (18m) 11m ago 36m 55.9M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 019eb1479485 2024-04-20T21:54:33.504 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (17m) 12m ago 32m 39.8M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 55aac560aab6 2024-04-20T21:54:33.504 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (21m) 11m ago 34m 7952k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T21:54:33.504 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (21m) 12m ago 33m 8304k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T21:54:33.504 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (15m) 11m ago 31m 88.0M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 ee2bb901f26e 2024-04-20T21:54:33.504 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (15m) 11m ago 31m 92.3M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 9075152d4162 2024-04-20T21:54:33.504 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (14m) 11m ago 30m 74.4M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 2d4f3e38dc0e 2024-04-20T21:54:33.504 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (14m) 12m ago 30m 87.6M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 8f97b17ae0eb 2024-04-20T21:54:33.504 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (13m) 12m ago 29m 90.7M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 653d8079967d 2024-04-20T21:54:33.505 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (12m) 12m ago 28m 11.5M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 596dded14c12 2024-04-20T21:54:33.505 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (20m) 11m ago 33m 74.0M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T21:54:33.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.505+0000 7f0f92c00700 1 -- 172.21.15.62:0/1529636154 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f0f7006ecc0 msgr2=0x7f0f70071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:54:33.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.505+0000 7f0f92c00700 1 --2- 172.21.15.62:0/1529636154 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f0f7006ecc0 0x7f0f70071170 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f0f7c005950 tx=0x7f0f7c0058e0 comp rx=0 tx=0).stop 2024-04-20T21:54:33.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.505+0000 7f0f92c00700 1 -- 172.21.15.62:0/1529636154 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0f8c10c8f0 msgr2=0x7f0f8c19c980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:54:33.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.505+0000 7f0f92c00700 1 --2- 172.21.15.62:0/1529636154 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0f8c10c8f0 0x7f0f8c19c980 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f0f780092e0 tx=0x7f0f7800be20 comp rx=0 tx=0).stop 2024-04-20T21:54:33.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.506+0000 7f0f92c00700 1 -- 172.21.15.62:0/1529636154 shutdown_connections 2024-04-20T21:54:33.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.506+0000 7f0f92c00700 1 --2- 172.21.15.62:0/1529636154 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f0f7006ecc0 0x7f0f70071170 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:54:33.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.506+0000 7f0f92c00700 1 --2- 172.21.15.62:0/1529636154 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f8c106b80 0x7f0f8c19c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:54:33.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.506+0000 7f0f92c00700 1 --2- 172.21.15.62:0/1529636154 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0f8c10c8f0 0x7f0f8c19c980 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:54:33.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.506+0000 7f0f92c00700 1 -- 172.21.15.62:0/1529636154 >> 172.21.15.62:0/1529636154 conn(0x7f0f8c076060 msgr2=0x7f0f8c077ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:54:33.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.506+0000 7f0f92c00700 1 -- 172.21.15.62:0/1529636154 shutdown_connections 2024-04-20T21:54:33.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.506+0000 7f0f92c00700 1 -- 172.21.15.62:0/1529636154 wait complete. 2024-04-20T21:54:33.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.661+0000 7f3f0ac00700 1 -- 172.21.15.62:0/607058408 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3f04073a00 msgr2=0x7f3f0410c580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:54:33.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.661+0000 7f3f0ac00700 1 --2- 172.21.15.62:0/607058408 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3f04073a00 0x7f3f0410c580 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f3eec009910 tx=0x7f3eec009c20 comp rx=0 tx=0).stop 2024-04-20T21:54:33.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.662+0000 7f3f0ac00700 1 -- 172.21.15.62:0/607058408 shutdown_connections 2024-04-20T21:54:33.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.662+0000 7f3f0ac00700 1 --2- 172.21.15.62:0/607058408 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3f04073a00 0x7f3f0410c580 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:54:33.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.662+0000 7f3f0ac00700 1 --2- 172.21.15.62:0/607058408 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f040730f0 0x7f3f040734c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:54:33.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.662+0000 7f3f0ac00700 1 -- 172.21.15.62:0/607058408 >> 172.21.15.62:0/607058408 conn(0x7f3f040fbda0 msgr2=0x7f3f040fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:54:33.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.662+0000 7f3f0ac00700 1 -- 172.21.15.62:0/607058408 shutdown_connections 2024-04-20T21:54:33.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.662+0000 7f3f0ac00700 1 -- 172.21.15.62:0/607058408 wait complete. 2024-04-20T21:54:33.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.663+0000 7f3f0ac00700 1 Processor -- start 2024-04-20T21:54:33.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.663+0000 7f3f0ac00700 1 -- start start 2024-04-20T21:54:33.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.663+0000 7f3f0ac00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f040730f0 0x7f3f04197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:54:33.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.663+0000 7f3f0ac00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3f04073a00 0x7f3f04198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:54:33.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.663+0000 7f3f0ac00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f0419c970 con 0x7f3f04073a00 2024-04-20T21:54:33.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.663+0000 7f3f0ac00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f0419cae0 con 0x7f3f040730f0 2024-04-20T21:54:33.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.664+0000 7f3efb400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3f04073a00 0x7f3f04198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:54:33.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.664+0000 7f3efb400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3f04073a00 0x7f3f04198630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45696/0 (socket says 172.21.15.62:45696) 2024-04-20T21:54:33.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.664+0000 7f3efb400700 1 -- 172.21.15.62:0/2507393410 learned_addr learned my addr 172.21.15.62:0/2507393410 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:54:33.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.664+0000 7f3f03e00700 1 --2- 172.21.15.62:0/2507393410 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f040730f0 0x7f3f04197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:54:33.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.664+0000 7f3efb400700 1 -- 172.21.15.62:0/2507393410 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f040730f0 msgr2=0x7f3f04197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:54:33.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.664+0000 7f3efb400700 1 --2- 172.21.15.62:0/2507393410 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f040730f0 0x7f3f04197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:54:33.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.664+0000 7f3efb400700 1 -- 172.21.15.62:0/2507393410 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3eec0095f0 con 0x7f3f04073a00 2024-04-20T21:54:33.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.664+0000 7f3efb400700 1 --2- 172.21.15.62:0/2507393410 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3f04073a00 0x7f3f04198630 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f3eec00b5c0 tx=0x7f3eec00be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:54:33.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.665+0000 7f3f01600700 1 -- 172.21.15.62:0/2507393410 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3eec01d070 con 0x7f3f04073a00 2024-04-20T21:54:33.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.665+0000 7f3f01600700 1 -- 172.21.15.62:0/2507393410 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3eec00f460 con 0x7f3f04073a00 2024-04-20T21:54:33.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.665+0000 7f3f01600700 1 -- 172.21.15.62:0/2507393410 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3eec017630 con 0x7f3f04073a00 2024-04-20T21:54:33.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.665+0000 7f3f0ac00700 1 -- 172.21.15.62:0/2507393410 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f0419cdc0 con 0x7f3f04073a00 2024-04-20T21:54:33.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.665+0000 7f3f0ac00700 1 -- 172.21.15.62:0/2507393410 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f0419d310 con 0x7f3f04073a00 2024-04-20T21:54:33.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.666+0000 7f3f0ac00700 1 -- 172.21.15.62:0/2507393410 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f04109c80 con 0x7f3f04073a00 2024-04-20T21:54:33.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.667+0000 7f3f01600700 1 -- 172.21.15.62:0/2507393410 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f3eec00f5d0 con 0x7f3f04073a00 2024-04-20T21:54:33.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.668+0000 7f3f01600700 1 --2- 172.21.15.62:0/2507393410 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3ee806ecc0 0x7f3ee8071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:54:33.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.668+0000 7f3f01600700 1 -- 172.21.15.62:0/2507393410 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f3eec0903c0 con 0x7f3f04073a00 2024-04-20T21:54:33.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.668+0000 7f3f03e00700 1 --2- 172.21.15.62:0/2507393410 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3ee806ecc0 0x7f3ee8071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:54:33.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.669+0000 7f3f03e00700 1 --2- 172.21.15.62:0/2507393410 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3ee806ecc0 0x7f3ee8071170 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f3ef40060b0 tx=0x7f3ef4007b80 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:54:33.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.673+0000 7f3f01600700 1 -- 172.21.15.62:0/2507393410 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f3eec059150 con 0x7f3f04073a00 2024-04-20T21:54:33.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.991+0000 7f3f0ac00700 1 -- 172.21.15.62:0/2507393410 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f3f0404f2a0 con 0x7f3f04073a00 2024-04-20T21:54:33.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.992+0000 7f3f01600700 1 -- 172.21.15.62:0/2507393410 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7f3eec05c810 con 0x7f3f04073a00 2024-04-20T21:54:33.991 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:54:33.991 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T21:54:33.991 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:54:33.991 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:54:33.991 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T21:54:33.991 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:54:33.991 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:54:33.992 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T21:54:33.992 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 6 2024-04-20T21:54:33.992 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:54:33.992 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T21:54:33.992 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 2, 2024-04-20T21:54:33.992 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:54:33.992 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:54:33.992 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T21:54:33.992 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 2, 2024-04-20T21:54:33.992 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 12 2024-04-20T21:54:33.992 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T21:54:33.992 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:54:33.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.994+0000 7f3f0ac00700 1 -- 172.21.15.62:0/2507393410 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3ee806ecc0 msgr2=0x7f3ee8071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:54:33.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.994+0000 7f3f0ac00700 1 --2- 172.21.15.62:0/2507393410 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3ee806ecc0 0x7f3ee8071170 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f3ef40060b0 tx=0x7f3ef4007b80 comp rx=0 tx=0).stop 2024-04-20T21:54:33.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.994+0000 7f3f0ac00700 1 -- 172.21.15.62:0/2507393410 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3f04073a00 msgr2=0x7f3f04198630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:54:33.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.994+0000 7f3f0ac00700 1 --2- 172.21.15.62:0/2507393410 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3f04073a00 0x7f3f04198630 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f3eec00b5c0 tx=0x7f3eec00be00 comp rx=0 tx=0).stop 2024-04-20T21:54:33.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.995+0000 7f3f0ac00700 1 -- 172.21.15.62:0/2507393410 shutdown_connections 2024-04-20T21:54:33.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.995+0000 7f3f0ac00700 1 --2- 172.21.15.62:0/2507393410 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3ee806ecc0 0x7f3ee8071170 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:54:33.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.995+0000 7f3f0ac00700 1 --2- 172.21.15.62:0/2507393410 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f040730f0 0x7f3f04197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:54:33.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.995+0000 7f3f0ac00700 1 --2- 172.21.15.62:0/2507393410 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3f04073a00 0x7f3f04198630 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:54:33.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.995+0000 7f3f0ac00700 1 -- 172.21.15.62:0/2507393410 >> 172.21.15.62:0/2507393410 conn(0x7f3f040fbda0 msgr2=0x7f3f04106dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:54:33.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.995+0000 7f3f0ac00700 1 -- 172.21.15.62:0/2507393410 shutdown_connections 2024-04-20T21:54:33.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:33.995+0000 7f3f0ac00700 1 -- 172.21.15.62:0/2507393410 wait complete. 2024-04-20T21:54:34.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:54:33 smithi062 ceph-mon[76322]: from='client.35068 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:54:34.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:54:33 smithi062 ceph-mon[76322]: pgmap v577: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:54:34.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:54:33 smithi062 ceph-mon[76322]: from='client.35072 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:54:34.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:34.159+0000 7f42e6400700 1 -- 172.21.15.62:0/2867863654 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f42e0101c40 msgr2=0x7f42e01020b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:54:34.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:34.159+0000 7f42e6400700 1 --2- 172.21.15.62:0/2867863654 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f42e0101c40 0x7f42e01020b0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f42d0009910 tx=0x7f42d0009c20 comp rx=0 tx=0).stop 2024-04-20T21:54:34.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:34.159+0000 7f42e6400700 1 -- 172.21.15.62:0/2867863654 shutdown_connections 2024-04-20T21:54:34.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:34.159+0000 7f42e6400700 1 --2- 172.21.15.62:0/2867863654 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f42e0101c40 0x7f42e01020b0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:54:34.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:34.159+0000 7f42e6400700 1 --2- 172.21.15.62:0/2867863654 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f42e01003e0 0x7f42e01007b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:54:34.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:34.159+0000 7f42e6400700 1 -- 172.21.15.62:0/2867863654 >> 172.21.15.62:0/2867863654 conn(0x7f42e00fbbc0 msgr2=0x7f42e00fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:54:34.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:34.159+0000 7f42e6400700 1 -- 172.21.15.62:0/2867863654 shutdown_connections 2024-04-20T21:54:34.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:34.159+0000 7f42e6400700 1 -- 172.21.15.62:0/2867863654 wait complete. 2024-04-20T21:54:34.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:34.160+0000 7f42e6400700 1 Processor -- start 2024-04-20T21:54:34.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:34.161+0000 7f42e6400700 1 -- start start 2024-04-20T21:54:34.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:34.161+0000 7f42e6400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f42e01003e0 0x7f42e019bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:54:34.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:34.161+0000 7f42e6400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f42e0101c40 0x7f42e019c010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:54:34.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:34.161+0000 7f42e6400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42e019c6a0 con 0x7f42e01003e0 2024-04-20T21:54:34.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:34.161+0000 7f42e6400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42e0195ba0 con 0x7f42e0101c40 2024-04-20T21:54:34.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:34.161+0000 7f42e5000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f42e01003e0 0x7f42e019bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:54:34.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:34.162+0000 7f42e5000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f42e01003e0 0x7f42e019bad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45710/0 (socket says 172.21.15.62:45710) 2024-04-20T21:54:34.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:34.162+0000 7f42e5000700 1 -- 172.21.15.62:0/6137596 learned_addr learned my addr 172.21.15.62:0/6137596 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:54:34.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:34.162+0000 7f42e5000700 1 -- 172.21.15.62:0/6137596 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f42e0101c40 msgr2=0x7f42e019c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:54:34.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:34.162+0000 7f42e5000700 1 --2- 172.21.15.62:0/6137596 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f42e0101c40 0x7f42e019c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:54:34.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:34.162+0000 7f42e5000700 1 -- 172.21.15.62:0/6137596 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42d00095f0 con 0x7f42e01003e0 2024-04-20T21:54:34.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:34.162+0000 7f42e5000700 1 --2- 172.21.15.62:0/6137596 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f42e01003e0 0x7f42e019bad0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f42c800d900 tx=0x7f42c800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:54:34.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:34.162+0000 7f42de000700 1 -- 172.21.15.62:0/6137596 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f42c8004a00 con 0x7f42e01003e0 2024-04-20T21:54:34.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:34.163+0000 7f42de000700 1 -- 172.21.15.62:0/6137596 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f42c80054e0 con 0x7f42e01003e0 2024-04-20T21:54:34.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:34.163+0000 7f42de000700 1 -- 172.21.15.62:0/6137596 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f42c8009d60 con 0x7f42e01003e0 2024-04-20T21:54:34.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:34.163+0000 7f42e6400700 1 -- 172.21.15.62:0/6137596 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42e0195e80 con 0x7f42e01003e0 2024-04-20T21:54:34.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:34.163+0000 7f42e6400700 1 -- 172.21.15.62:0/6137596 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42e0196380 con 0x7f42e01003e0 2024-04-20T21:54:34.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:34.164+0000 7f42e6400700 1 -- 172.21.15.62:0/6137596 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42e01086c0 con 0x7f42e01003e0 2024-04-20T21:54:34.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:34.165+0000 7f42de000700 1 -- 172.21.15.62:0/6137596 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f42c8005020 con 0x7f42e01003e0 2024-04-20T21:54:34.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:34.166+0000 7f42de000700 1 --2- 172.21.15.62:0/6137596 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f42c006ecc0 0x7f42c0071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:54:34.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:34.166+0000 7f42de000700 1 -- 172.21.15.62:0/6137596 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f42c808f0d0 con 0x7f42e01003e0 2024-04-20T21:54:34.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:34.167+0000 7f42d7e00700 1 --2- 172.21.15.62:0/6137596 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f42c006ecc0 0x7f42c0071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:54:34.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:34.167+0000 7f42d7e00700 1 --2- 172.21.15.62:0/6137596 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f42c006ecc0 0x7f42c0071170 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f42d000b5c0 tx=0x7f42d0005fd0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:54:34.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:34.171+0000 7f42de000700 1 -- 172.21.15.62:0/6137596 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f42c805b910 con 0x7f42e01003e0 2024-04-20T21:54:34.195 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:54:33 smithi189 ceph-mon[57022]: from='client.35068 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:54:34.195 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:54:33 smithi189 ceph-mon[57022]: pgmap v577: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:54:34.195 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:54:33 smithi189 ceph-mon[57022]: from='client.35072 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:54:34.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:34.447+0000 7f42e6400700 1 -- 172.21.15.62:0/6137596 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f42e01970f0 con 0x7f42e01003e0 2024-04-20T21:54:34.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:34.448+0000 7f42de000700 1 -- 172.21.15.62:0/6137596 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 25 v25) v1 ==== 76+0+1887 (secure 0 0 0) 0x7f42e01970f0 con 0x7f42e01003e0 2024-04-20T21:54:34.448 INFO:teuthology.orchestra.run.smithi062.stdout:e25 2024-04-20T21:54:34.448 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T21:54:34.448 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T21:54:34.448 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T21:54:34.448 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:54:34.448 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T21:54:34.448 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T21:54:34.448 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 20 2024-04-20T21:54:34.448 INFO:teuthology.orchestra.run.smithi062.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-20T21:54:34.448 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T21:54:34.448 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T21:42:01.559061+0000 2024-04-20T21:54:34.448 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T21:54:34.448 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T21:54:34.449 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T21:54:34.449 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T21:54:34.449 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T21:54:34.449 INFO:teuthology.orchestra.run.smithi062.stdout:max_xattr_size 65536 2024-04-20T21:54:34.449 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T21:54:34.449 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T21:54:34.449 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 81 2024-04-20T21:54:34.449 INFO:teuthology.orchestra.run.smithi062.stdout:snap_rstat 1 2024-04-20T21:54:34.449 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T21:54:34.449 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 2 2024-04-20T21:54:34.449 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1 2024-04-20T21:54:34.449 INFO:teuthology.orchestra.run.smithi062.stdout:up {} 2024-04-20T21:54:34.449 INFO:teuthology.orchestra.run.smithi062.stdout:failed 0,1 2024-04-20T21:54:34.449 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T21:54:34.450 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T21:54:34.450 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T21:54:34.450 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T21:54:34.450 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data enabled 2024-04-20T21:54:34.450 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T21:54:34.450 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T21:54:34.450 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 1 2024-04-20T21:54:34.450 INFO:teuthology.orchestra.run.smithi062.stdout:qdb_cluster leader: 0 members: 2024-04-20T21:54:34.450 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:54:34.450 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:54:34.450 INFO:teuthology.orchestra.run.smithi062.stdout:Standby daemons: 2024-04-20T21:54:34.450 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:54:34.450 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ryqblb{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6826/4113916205,v1:172.21.15.62:6827/4113916205] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T21:54:34.450 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ozizwi{-1:34476} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6828/3801975165,v1:172.21.15.62:6829/3801975165] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T21:54:34.451 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.xbgspw{-1:44285} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6826/3996040526,v1:172.21.15.189:6827/3996040526] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:54:34.451 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.dvimer{-1:44289} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6824/1743111843,v1:172.21.15.189:6825/1743111843] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:54:34.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:34.451+0000 7f42e6400700 1 -- 172.21.15.62:0/6137596 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f42c006ecc0 msgr2=0x7f42c0071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:54:34.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:34.451+0000 7f42e6400700 1 --2- 172.21.15.62:0/6137596 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f42c006ecc0 0x7f42c0071170 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f42d000b5c0 tx=0x7f42d0005fd0 comp rx=0 tx=0).stop 2024-04-20T21:54:34.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:34.451+0000 7f42e6400700 1 -- 172.21.15.62:0/6137596 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f42e01003e0 msgr2=0x7f42e019bad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:54:34.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:34.451+0000 7f42e6400700 1 --2- 172.21.15.62:0/6137596 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f42e01003e0 0x7f42e019bad0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f42c800d900 tx=0x7f42c800dcc0 comp rx=0 tx=0).stop 2024-04-20T21:54:34.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:34.452+0000 7f42e6400700 1 -- 172.21.15.62:0/6137596 shutdown_connections 2024-04-20T21:54:34.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:34.452+0000 7f42e6400700 1 --2- 172.21.15.62:0/6137596 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f42c006ecc0 0x7f42c0071170 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:54:34.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:34.452+0000 7f42e6400700 1 --2- 172.21.15.62:0/6137596 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f42e01003e0 0x7f42e019bad0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:54:34.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:34.452+0000 7f42e6400700 1 --2- 172.21.15.62:0/6137596 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f42e0101c40 0x7f42e019c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:54:34.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:34.452+0000 7f42e6400700 1 -- 172.21.15.62:0/6137596 >> 172.21.15.62:0/6137596 conn(0x7f42e00fbbc0 msgr2=0x7f42e00fc510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:54:34.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:34.452+0000 7f42e6400700 1 -- 172.21.15.62:0/6137596 shutdown_connections 2024-04-20T21:54:34.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:34.452+0000 7f42e6400700 1 -- 172.21.15.62:0/6137596 wait complete. 2024-04-20T21:54:34.453 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 25 2024-04-20T21:54:34.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:34.617+0000 7f3c4aa00700 1 -- 172.21.15.62:0/1189962642 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3c440730b0 msgr2=0x7f3c44073480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:54:34.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:34.617+0000 7f3c4aa00700 1 --2- 172.21.15.62:0/1189962642 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3c440730b0 0x7f3c44073480 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f3c34009940 tx=0x7f3c34009c50 comp rx=0 tx=0).stop 2024-04-20T21:54:34.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:34.617+0000 7f3c4aa00700 1 -- 172.21.15.62:0/1189962642 shutdown_connections 2024-04-20T21:54:34.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:34.617+0000 7f3c4aa00700 1 --2- 172.21.15.62:0/1189962642 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3c440739c0 0x7f3c4410c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:54:34.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:34.617+0000 7f3c4aa00700 1 --2- 172.21.15.62:0/1189962642 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3c440730b0 0x7f3c44073480 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:54:34.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:34.617+0000 7f3c4aa00700 1 -- 172.21.15.62:0/1189962642 >> 172.21.15.62:0/1189962642 conn(0x7f3c440fbda0 msgr2=0x7f3c440fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:54:34.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:34.617+0000 7f3c4aa00700 1 -- 172.21.15.62:0/1189962642 shutdown_connections 2024-04-20T21:54:34.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:34.617+0000 7f3c4aa00700 1 -- 172.21.15.62:0/1189962642 wait complete. 2024-04-20T21:54:34.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:34.618+0000 7f3c4aa00700 1 Processor -- start 2024-04-20T21:54:34.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:34.618+0000 7f3c4aa00700 1 -- start start 2024-04-20T21:54:34.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:34.618+0000 7f3c4aa00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3c440730b0 0x7f3c44198210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:54:34.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:34.619+0000 7f3c4aa00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3c440739c0 0x7f3c44198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:54:34.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:34.619+0000 7f3c4aa00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c44198e30 con 0x7f3c440730b0 2024-04-20T21:54:34.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:34.619+0000 7f3c4aa00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c4419cbc0 con 0x7f3c440739c0 2024-04-20T21:54:34.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:34.619+0000 7f3c43e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3c440730b0 0x7f3c44198210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:54:34.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:34.619+0000 7f3c43e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3c440730b0 0x7f3c44198210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45730/0 (socket says 172.21.15.62:45730) 2024-04-20T21:54:34.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:34.619+0000 7f3c43e00700 1 -- 172.21.15.62:0/3310531852 learned_addr learned my addr 172.21.15.62:0/3310531852 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:54:34.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:34.619+0000 7f3c43400700 1 --2- 172.21.15.62:0/3310531852 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3c440739c0 0x7f3c44198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:54:34.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:34.619+0000 7f3c43e00700 1 -- 172.21.15.62:0/3310531852 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3c440739c0 msgr2=0x7f3c44198750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:54:34.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:34.619+0000 7f3c43e00700 1 --2- 172.21.15.62:0/3310531852 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3c440739c0 0x7f3c44198750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:54:34.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:34.619+0000 7f3c43e00700 1 -- 172.21.15.62:0/3310531852 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c340095f0 con 0x7f3c440730b0 2024-04-20T21:54:34.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:34.619+0000 7f3c43400700 1 --2- 172.21.15.62:0/3310531852 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3c440739c0 0x7f3c44198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T21:54:34.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:34.620+0000 7f3c43e00700 1 --2- 172.21.15.62:0/3310531852 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3c440730b0 0x7f3c44198210 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f3c3400b870 tx=0x7f3c340056a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:54:34.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:34.620+0000 7f3c40c00700 1 -- 172.21.15.62:0/3310531852 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3c3401d070 con 0x7f3c440730b0 2024-04-20T21:54:34.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:34.620+0000 7f3c40c00700 1 -- 172.21.15.62:0/3310531852 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3c3400f460 con 0x7f3c440730b0 2024-04-20T21:54:34.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:34.620+0000 7f3c40c00700 1 -- 172.21.15.62:0/3310531852 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3c34021640 con 0x7f3c440730b0 2024-04-20T21:54:34.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:34.620+0000 7f3c4aa00700 1 -- 172.21.15.62:0/3310531852 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c4419ce40 con 0x7f3c440730b0 2024-04-20T21:54:34.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:34.620+0000 7f3c4aa00700 1 -- 172.21.15.62:0/3310531852 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c4419d330 con 0x7f3c440730b0 2024-04-20T21:54:34.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:34.621+0000 7f3c4aa00700 1 -- 172.21.15.62:0/3310531852 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c44109cc0 con 0x7f3c440730b0 2024-04-20T21:54:34.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:34.622+0000 7f3c40c00700 1 -- 172.21.15.62:0/3310531852 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f3c3400fa60 con 0x7f3c440730b0 2024-04-20T21:54:34.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:34.623+0000 7f3c40c00700 1 --2- 172.21.15.62:0/3310531852 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3c2806ecc0 0x7f3c28071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:54:34.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:34.623+0000 7f3c40c00700 1 -- 172.21.15.62:0/3310531852 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f3c340902d0 con 0x7f3c440730b0 2024-04-20T21:54:34.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:34.624+0000 7f3c43400700 1 --2- 172.21.15.62:0/3310531852 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3c2806ecc0 0x7f3c28071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:54:34.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:34.624+0000 7f3c43400700 1 --2- 172.21.15.62:0/3310531852 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3c2806ecc0 0x7f3c28071170 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f3c30005fd0 tx=0x7f3c30005dc0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:54:34.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:34.629+0000 7f3c40c00700 1 -- 172.21.15.62:0/3310531852 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f3c34058fe0 con 0x7f3c440730b0 2024-04-20T21:54:34.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:34.882+0000 7f3c4aa00700 1 -- 172.21.15.62:0/3310531852 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3c441996b0 con 0x7f3c2806ecc0 2024-04-20T21:54:34.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:34.885+0000 7f3c40c00700 1 -- 172.21.15.62:0/3310531852 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f3c300086c0 con 0x7f3c2806ecc0 2024-04-20T21:54:34.884 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:54:34.884 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T21:54:34.885 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T21:54:34.885 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-20T21:54:34.885 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T21:54:34.885 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr", 2024-04-20T21:54:34.885 INFO:teuthology.orchestra.run.smithi062.stdout: "crash", 2024-04-20T21:54:34.885 INFO:teuthology.orchestra.run.smithi062.stdout: "mon", 2024-04-20T21:54:34.885 INFO:teuthology.orchestra.run.smithi062.stdout: "osd" 2024-04-20T21:54:34.885 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T21:54:34.885 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "14/23 daemons upgraded", 2024-04-20T21:54:34.885 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading mds daemons", 2024-04-20T21:54:34.885 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T21:54:34.885 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:54:34.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:34.888+0000 7f3c4aa00700 1 -- 172.21.15.62:0/3310531852 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3c2806ecc0 msgr2=0x7f3c28071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:54:34.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:34.888+0000 7f3c4aa00700 1 --2- 172.21.15.62:0/3310531852 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3c2806ecc0 0x7f3c28071170 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f3c30005fd0 tx=0x7f3c30005dc0 comp rx=0 tx=0).stop 2024-04-20T21:54:34.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:34.888+0000 7f3c4aa00700 1 -- 172.21.15.62:0/3310531852 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3c440730b0 msgr2=0x7f3c44198210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:54:34.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:34.888+0000 7f3c4aa00700 1 --2- 172.21.15.62:0/3310531852 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3c440730b0 0x7f3c44198210 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f3c3400b870 tx=0x7f3c340056a0 comp rx=0 tx=0).stop 2024-04-20T21:54:34.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:34.888+0000 7f3c4aa00700 1 -- 172.21.15.62:0/3310531852 shutdown_connections 2024-04-20T21:54:34.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:34.888+0000 7f3c4aa00700 1 --2- 172.21.15.62:0/3310531852 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3c2806ecc0 0x7f3c28071170 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:54:34.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:34.888+0000 7f3c4aa00700 1 --2- 172.21.15.62:0/3310531852 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3c440730b0 0x7f3c44198210 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:54:34.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:34.888+0000 7f3c4aa00700 1 --2- 172.21.15.62:0/3310531852 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3c440739c0 0x7f3c44198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:54:34.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:34.888+0000 7f3c4aa00700 1 -- 172.21.15.62:0/3310531852 >> 172.21.15.62:0/3310531852 conn(0x7f3c440fbda0 msgr2=0x7f3c44106e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:54:34.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:34.888+0000 7f3c4aa00700 1 -- 172.21.15.62:0/3310531852 shutdown_connections 2024-04-20T21:54:34.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:34.888+0000 7f3c4aa00700 1 -- 172.21.15.62:0/3310531852 wait complete. 2024-04-20T21:54:35.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:54:34 smithi062 ceph-mon[76322]: from='client.35076 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:54:35.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:54:34 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/2507393410' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:54:35.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:54:34 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:54:35.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:54:34 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/6137596' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T21:54:35.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:35.043+0000 7fd8e8200700 1 -- 172.21.15.62:0/4079160514 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd8e00687a0 msgr2=0x7fd8e0068b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:54:35.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:35.043+0000 7fd8e8200700 1 --2- 172.21.15.62:0/4079160514 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd8e00687a0 0x7fd8e0068b70 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fd8d4009910 tx=0x7fd8d4009c20 comp rx=0 tx=0).stop 2024-04-20T21:54:35.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:35.043+0000 7fd8e8200700 1 -- 172.21.15.62:0/4079160514 shutdown_connections 2024-04-20T21:54:35.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:35.043+0000 7fd8e8200700 1 --2- 172.21.15.62:0/4079160514 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8e0069140 0x7fd8e0104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:54:35.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:35.043+0000 7fd8e8200700 1 --2- 172.21.15.62:0/4079160514 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd8e00687a0 0x7fd8e0068b70 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:54:35.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:35.043+0000 7fd8e8200700 1 -- 172.21.15.62:0/4079160514 >> 172.21.15.62:0/4079160514 conn(0x7fd8e0075130 msgr2=0x7fd8e0075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:54:35.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:35.043+0000 7fd8e8200700 1 -- 172.21.15.62:0/4079160514 shutdown_connections 2024-04-20T21:54:35.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:35.043+0000 7fd8e8200700 1 -- 172.21.15.62:0/4079160514 wait complete. 2024-04-20T21:54:35.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:35.044+0000 7fd8e8200700 1 Processor -- start 2024-04-20T21:54:35.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:35.045+0000 7fd8e8200700 1 -- start start 2024-04-20T21:54:35.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:35.045+0000 7fd8e8200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8e00687a0 0x7fd8e01980e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:54:35.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:35.045+0000 7fd8e8200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd8e0069140 0x7fd8e0198620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:54:35.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:35.045+0000 7fd8e8200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8e0198c70 con 0x7fd8e0069140 2024-04-20T21:54:35.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:35.045+0000 7fd8e8200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8e0198db0 con 0x7fd8e00687a0 2024-04-20T21:54:35.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:35.045+0000 7fd8e4e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd8e0069140 0x7fd8e0198620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:54:35.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:35.045+0000 7fd8e4e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd8e0069140 0x7fd8e0198620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45742/0 (socket says 172.21.15.62:45742) 2024-04-20T21:54:35.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:35.045+0000 7fd8e4e00700 1 -- 172.21.15.62:0/3004251213 learned_addr learned my addr 172.21.15.62:0/3004251213 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:54:35.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:35.046+0000 7fd8e4e00700 1 -- 172.21.15.62:0/3004251213 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8e00687a0 msgr2=0x7fd8e01980e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:54:35.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:35.046+0000 7fd8e4e00700 1 --2- 172.21.15.62:0/3004251213 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8e00687a0 0x7fd8e01980e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:54:35.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:35.046+0000 7fd8e4e00700 1 -- 172.21.15.62:0/3004251213 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8d40095f0 con 0x7fd8e0069140 2024-04-20T21:54:35.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:35.046+0000 7fd8e4e00700 1 --2- 172.21.15.62:0/3004251213 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd8e0069140 0x7fd8e0198620 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fd8cc00c960 tx=0x7fd8cc00cd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:54:35.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:35.046+0000 7fd8de000700 1 -- 172.21.15.62:0/3004251213 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd8cc007a70 con 0x7fd8e0069140 2024-04-20T21:54:35.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:35.046+0000 7fd8de000700 1 -- 172.21.15.62:0/3004251213 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd8cc007bd0 con 0x7fd8e0069140 2024-04-20T21:54:35.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:35.046+0000 7fd8de000700 1 -- 172.21.15.62:0/3004251213 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd8cc0186c0 con 0x7fd8e0069140 2024-04-20T21:54:35.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:35.046+0000 7fd8e8200700 1 -- 172.21.15.62:0/3004251213 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8e019cc00 con 0x7fd8e0069140 2024-04-20T21:54:35.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:35.046+0000 7fd8e8200700 1 -- 172.21.15.62:0/3004251213 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8e019d150 con 0x7fd8e0069140 2024-04-20T21:54:35.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:35.047+0000 7fd8e8200700 1 -- 172.21.15.62:0/3004251213 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8e004f2a0 con 0x7fd8e0069140 2024-04-20T21:54:35.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:35.048+0000 7fd8de000700 1 -- 172.21.15.62:0/3004251213 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fd8cc00faa0 con 0x7fd8e0069140 2024-04-20T21:54:35.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:35.049+0000 7fd8de000700 1 --2- 172.21.15.62:0/3004251213 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd8c806ecc0 0x7fd8c8071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:54:35.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:35.049+0000 7fd8de000700 1 -- 172.21.15.62:0/3004251213 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fd8cc08eea0 con 0x7fd8e0069140 2024-04-20T21:54:35.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:35.050+0000 7fd8e5800700 1 --2- 172.21.15.62:0/3004251213 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd8c806ecc0 0x7fd8c8071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:54:35.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:35.050+0000 7fd8e5800700 1 --2- 172.21.15.62:0/3004251213 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd8c806ecc0 0x7fd8c8071170 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fd8d400b5c0 tx=0x7fd8d4005fd0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:54:35.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:35.054+0000 7fd8de000700 1 -- 172.21.15.62:0/3004251213 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fd8cc05b760 con 0x7fd8e0069140 2024-04-20T21:54:35.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:54:34 smithi189 ceph-mon[57022]: from='client.35076 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:54:35.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:54:34 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/2507393410' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:54:35.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:54:34 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:54:35.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:54:34 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/6137596' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T21:54:35.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:35.386+0000 7fd8e8200700 1 -- 172.21.15.62:0/3004251213 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fd8e0199440 con 0x7fd8e0069140 2024-04-20T21:54:35.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:35.387+0000 7fd8de000700 1 -- 172.21.15.62:0/3004251213 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7fd8cc05b2f0 con 0x7fd8e0069140 2024-04-20T21:54:35.386 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-20T21:54:35.387 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-20T21:54:35.387 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is degraded 2024-04-20T21:54:35.387 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T21:54:35.387 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T21:54:35.387 INFO:teuthology.orchestra.run.smithi062.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-20T21:54:35.387 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-20T21:54:35.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:35.390+0000 7fd8e8200700 1 -- 172.21.15.62:0/3004251213 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd8c806ecc0 msgr2=0x7fd8c8071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:54:35.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:35.390+0000 7fd8e8200700 1 --2- 172.21.15.62:0/3004251213 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd8c806ecc0 0x7fd8c8071170 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fd8d400b5c0 tx=0x7fd8d4005fd0 comp rx=0 tx=0).stop 2024-04-20T21:54:35.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:35.390+0000 7fd8e8200700 1 -- 172.21.15.62:0/3004251213 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd8e0069140 msgr2=0x7fd8e0198620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:54:35.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:35.390+0000 7fd8e8200700 1 --2- 172.21.15.62:0/3004251213 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd8e0069140 0x7fd8e0198620 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fd8cc00c960 tx=0x7fd8cc00cd20 comp rx=0 tx=0).stop 2024-04-20T21:54:35.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:35.390+0000 7fd8e8200700 1 -- 172.21.15.62:0/3004251213 shutdown_connections 2024-04-20T21:54:35.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:35.390+0000 7fd8e8200700 1 --2- 172.21.15.62:0/3004251213 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd8c806ecc0 0x7fd8c8071170 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:54:35.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:35.390+0000 7fd8e8200700 1 --2- 172.21.15.62:0/3004251213 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8e00687a0 0x7fd8e01980e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:54:35.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:35.390+0000 7fd8e8200700 1 --2- 172.21.15.62:0/3004251213 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd8e0069140 0x7fd8e0198620 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:54:35.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:35.390+0000 7fd8e8200700 1 -- 172.21.15.62:0/3004251213 >> 172.21.15.62:0/3004251213 conn(0x7fd8e0075130 msgr2=0x7fd8e0103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:54:35.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:35.390+0000 7fd8e8200700 1 -- 172.21.15.62:0/3004251213 shutdown_connections 2024-04-20T21:54:35.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:54:35.390+0000 7fd8e8200700 1 -- 172.21.15.62:0/3004251213 wait complete. 2024-04-20T21:54:36.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:54:35 smithi062 ceph-mon[76322]: pgmap v578: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:54:36.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:54:35 smithi062 ceph-mon[76322]: from='client.35088 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:54:36.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:54:35 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/3004251213' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T21:54:36.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:54:35 smithi189 ceph-mon[57022]: pgmap v578: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:54:36.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:54:35 smithi189 ceph-mon[57022]: from='client.35088 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:54:36.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:54:35 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/3004251213' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T21:54:38.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:54:37 smithi062 ceph-mon[76322]: pgmap v579: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:54:38.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:54:37 smithi189 ceph-mon[57022]: pgmap v579: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:54:40.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:54:39 smithi062 ceph-mon[76322]: pgmap v580: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:54:40.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:54:39 smithi189 ceph-mon[57022]: pgmap v580: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:54:42.046 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:54:41 smithi062 ceph-mon[76322]: pgmap v581: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:54:42.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:54:41 smithi189 ceph-mon[57022]: pgmap v581: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:54:44.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:54:43 smithi189 ceph-mon[57022]: pgmap v582: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:54:44.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:54:43 smithi062 ceph-mon[76322]: pgmap v582: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:54:46.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:54:45 smithi189 ceph-mon[57022]: pgmap v583: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:54:46.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:54:45 smithi062 ceph-mon[76322]: pgmap v583: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:54:48.143 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:54:47 smithi062 ceph-mon[76322]: pgmap v584: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:54:48.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:54:47 smithi189 ceph-mon[57022]: pgmap v584: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:54:49.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:54:49 smithi189 ceph-mon[57022]: pgmap v585: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:54:49.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:54:49 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:54:50.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:54:49 smithi062 ceph-mon[76322]: pgmap v585: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:54:50.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:54:49 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:54:52.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:54:51 smithi189 ceph-mon[57022]: pgmap v586: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:54:52.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:54:51 smithi062 ceph-mon[76322]: pgmap v586: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:54:54.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:54:53 smithi189 ceph-mon[57022]: pgmap v587: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:54:54.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:54:53 smithi062 ceph-mon[76322]: pgmap v587: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:54:56.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:54:55 smithi189 ceph-mon[57022]: pgmap v588: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:54:56.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:54:55 smithi062 ceph-mon[76322]: pgmap v588: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:54:58.144 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:54:57 smithi062 ceph-mon[76322]: pgmap v589: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:54:58.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:54:57 smithi189 ceph-mon[57022]: pgmap v589: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:55:00.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:54:59 smithi189 ceph-mon[57022]: pgmap v590: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:55:00.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:54:59 smithi062 ceph-mon[76322]: pgmap v590: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:55:02.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:55:01 smithi189 ceph-mon[57022]: pgmap v591: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:55:02.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:55:01 smithi062 ceph-mon[76322]: pgmap v591: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:55:04.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:55:03 smithi189 ceph-mon[57022]: pgmap v592: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:55:04.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:55:03 smithi062 ceph-mon[76322]: pgmap v592: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:55:05.141 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:55:04 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:55:05.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:55:04 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:55:05.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:05.566+0000 7ff556600700 1 -- 172.21.15.62:0/1210986355 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff550102470 msgr2=0x7ff5501028e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:55:05.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:05.566+0000 7ff556600700 1 --2- 172.21.15.62:0/1210986355 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff550102470 0x7ff5501028e0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7ff538009910 tx=0x7ff538009c20 comp rx=0 tx=0).stop 2024-04-20T21:55:05.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:05.567+0000 7ff556600700 1 -- 172.21.15.62:0/1210986355 shutdown_connections 2024-04-20T21:55:05.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:05.567+0000 7ff556600700 1 --2- 172.21.15.62:0/1210986355 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff550102470 0x7ff5501028e0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:55:05.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:05.567+0000 7ff556600700 1 --2- 172.21.15.62:0/1210986355 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff550108470 0x7ff550108840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:55:05.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:05.567+0000 7ff556600700 1 -- 172.21.15.62:0/1210986355 >> 172.21.15.62:0/1210986355 conn(0x7ff5500fdf90 msgr2=0x7ff5501003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:55:05.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:05.567+0000 7ff556600700 1 -- 172.21.15.62:0/1210986355 shutdown_connections 2024-04-20T21:55:05.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:05.567+0000 7ff556600700 1 -- 172.21.15.62:0/1210986355 wait complete. 2024-04-20T21:55:05.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:05.568+0000 7ff556600700 1 Processor -- start 2024-04-20T21:55:05.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:05.568+0000 7ff556600700 1 -- start start 2024-04-20T21:55:05.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:05.569+0000 7ff556600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff550102470 0x7ff550198230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:55:05.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:05.569+0000 7ff556600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff550108470 0x7ff550198770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:55:05.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:05.569+0000 7ff556600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff550198e50 con 0x7ff550102470 2024-04-20T21:55:05.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:05.569+0000 7ff556600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff55019cbe0 con 0x7ff550108470 2024-04-20T21:55:05.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:05.569+0000 7ff54f400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff550102470 0x7ff550198230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:55:05.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:05.569+0000 7ff54f400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff550102470 0x7ff550198230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59030/0 (socket says 172.21.15.62:59030) 2024-04-20T21:55:05.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:05.569+0000 7ff54f400700 1 -- 172.21.15.62:0/832172818 learned_addr learned my addr 172.21.15.62:0/832172818 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:55:05.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:05.570+0000 7ff547e00700 1 --2- 172.21.15.62:0/832172818 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff550108470 0x7ff550198770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:55:05.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:05.570+0000 7ff54f400700 1 -- 172.21.15.62:0/832172818 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff550108470 msgr2=0x7ff550198770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:55:05.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:05.570+0000 7ff54f400700 1 --2- 172.21.15.62:0/832172818 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff550108470 0x7ff550198770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:55:05.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:05.570+0000 7ff54f400700 1 -- 172.21.15.62:0/832172818 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5380095f0 con 0x7ff550102470 2024-04-20T21:55:05.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:05.570+0000 7ff54f400700 1 --2- 172.21.15.62:0/832172818 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff550102470 0x7ff550198230 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7ff54000d8d0 tx=0x7ff54000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:55:05.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:05.571+0000 7ff54cc00700 1 -- 172.21.15.62:0/832172818 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff540009940 con 0x7ff550102470 2024-04-20T21:55:05.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:05.571+0000 7ff54cc00700 1 -- 172.21.15.62:0/832172818 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff540010460 con 0x7ff550102470 2024-04-20T21:55:05.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:05.571+0000 7ff556600700 1 -- 172.21.15.62:0/832172818 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff55019cec0 con 0x7ff550102470 2024-04-20T21:55:05.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:05.571+0000 7ff54cc00700 1 -- 172.21.15.62:0/832172818 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff54000f5f0 con 0x7ff550102470 2024-04-20T21:55:05.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:05.571+0000 7ff556600700 1 -- 172.21.15.62:0/832172818 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff55019d410 con 0x7ff550102470 2024-04-20T21:55:05.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:05.572+0000 7ff556600700 1 -- 172.21.15.62:0/832172818 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff55010a970 con 0x7ff550102470 2024-04-20T21:55:05.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:05.573+0000 7ff54cc00700 1 -- 172.21.15.62:0/832172818 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff5400105d0 con 0x7ff550102470 2024-04-20T21:55:05.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:05.575+0000 7ff54cc00700 1 --2- 172.21.15.62:0/832172818 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff53406ed10 0x7ff5340711c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:55:05.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:05.575+0000 7ff54cc00700 1 -- 172.21.15.62:0/832172818 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7ff54008ef20 con 0x7ff550102470 2024-04-20T21:55:05.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:05.575+0000 7ff547e00700 1 --2- 172.21.15.62:0/832172818 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff53406ed10 0x7ff5340711c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:55:05.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:05.576+0000 7ff547e00700 1 --2- 172.21.15.62:0/832172818 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff53406ed10 0x7ff5340711c0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7ff53800b5c0 tx=0x7ff53801a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:55:05.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:05.581+0000 7ff54cc00700 1 -- 172.21.15.62:0/832172818 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7ff540057c30 con 0x7ff550102470 2024-04-20T21:55:05.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:05.836+0000 7ff556600700 1 -- 172.21.15.62:0/832172818 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff55019d630 con 0x7ff53406ed10 2024-04-20T21:55:05.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:05.839+0000 7ff54cc00700 1 -- 172.21.15.62:0/832172818 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7ff55019d630 con 0x7ff53406ed10 2024-04-20T21:55:05.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:05.841+0000 7ff556600700 1 -- 172.21.15.62:0/832172818 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff53406ed10 msgr2=0x7ff5340711c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:55:05.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:05.841+0000 7ff556600700 1 --2- 172.21.15.62:0/832172818 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff53406ed10 0x7ff5340711c0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7ff53800b5c0 tx=0x7ff53801a040 comp rx=0 tx=0).stop 2024-04-20T21:55:05.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:05.841+0000 7ff556600700 1 -- 172.21.15.62:0/832172818 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff550102470 msgr2=0x7ff550198230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:55:05.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:05.841+0000 7ff556600700 1 --2- 172.21.15.62:0/832172818 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff550102470 0x7ff550198230 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7ff54000d8d0 tx=0x7ff54000dc90 comp rx=0 tx=0).stop 2024-04-20T21:55:05.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:05.841+0000 7ff556600700 1 -- 172.21.15.62:0/832172818 shutdown_connections 2024-04-20T21:55:05.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:05.841+0000 7ff556600700 1 --2- 172.21.15.62:0/832172818 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff53406ed10 0x7ff5340711c0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:55:05.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:05.841+0000 7ff556600700 1 --2- 172.21.15.62:0/832172818 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff550102470 0x7ff550198230 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:55:05.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:05.841+0000 7ff556600700 1 --2- 172.21.15.62:0/832172818 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff550108470 0x7ff550198770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:55:05.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:05.842+0000 7ff556600700 1 -- 172.21.15.62:0/832172818 >> 172.21.15.62:0/832172818 conn(0x7ff5500fdf90 msgr2=0x7ff5500ff320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:55:05.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:05.842+0000 7ff556600700 1 -- 172.21.15.62:0/832172818 shutdown_connections 2024-04-20T21:55:05.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:05.842+0000 7ff556600700 1 -- 172.21.15.62:0/832172818 wait complete. 2024-04-20T21:55:05.856 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T21:55:06.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.006+0000 7f2203e00700 1 -- 172.21.15.62:0/3594591862 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f22041061b0 msgr2=0x7f2204106580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:55:06.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.006+0000 7f2203e00700 1 --2- 172.21.15.62:0/3594591862 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f22041061b0 0x7f2204106580 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f21f4009910 tx=0x7f21f4009c20 comp rx=0 tx=0).stop 2024-04-20T21:55:06.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.006+0000 7f2203e00700 1 -- 172.21.15.62:0/3594591862 shutdown_connections 2024-04-20T21:55:06.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.006+0000 7f2203e00700 1 --2- 172.21.15.62:0/3594591862 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2204068540 0x7f22040689b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:55:06.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.006+0000 7f2203e00700 1 --2- 172.21.15.62:0/3594591862 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f22041061b0 0x7f2204106580 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:55:06.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.006+0000 7f2203e00700 1 -- 172.21.15.62:0/3594591862 >> 172.21.15.62:0/3594591862 conn(0x7f2204074fa0 msgr2=0x7f22040753a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:55:06.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.007+0000 7f2203e00700 1 -- 172.21.15.62:0/3594591862 shutdown_connections 2024-04-20T21:55:06.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.007+0000 7f2203e00700 1 -- 172.21.15.62:0/3594591862 wait complete. 2024-04-20T21:55:06.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.008+0000 7f2203e00700 1 Processor -- start 2024-04-20T21:55:06.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.008+0000 7f2203e00700 1 -- start start 2024-04-20T21:55:06.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.009+0000 7f2203e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2204068540 0x7f2204197d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:55:06.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.009+0000 7f2203e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f22041061b0 0x7f22041982a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:55:06.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.009+0000 7f2203e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2204198980 con 0x7f2204068540 2024-04-20T21:55:06.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.009+0000 7f2203e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f220419c710 con 0x7f22041061b0 2024-04-20T21:55:06.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.009+0000 7f2202a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2204068540 0x7f2204197d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:55:06.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.009+0000 7f2202a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2204068540 0x7f2204197d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59044/0 (socket says 172.21.15.62:59044) 2024-04-20T21:55:06.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.009+0000 7f2202a00700 1 -- 172.21.15.62:0/3888956468 learned_addr learned my addr 172.21.15.62:0/3888956468 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:55:06.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.009+0000 7f2202000700 1 --2- 172.21.15.62:0/3888956468 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f22041061b0 0x7f22041982a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:55:06.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.010+0000 7f2202a00700 1 -- 172.21.15.62:0/3888956468 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f22041061b0 msgr2=0x7f22041982a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:55:06.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.010+0000 7f2202a00700 1 --2- 172.21.15.62:0/3888956468 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f22041061b0 0x7f22041982a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:55:06.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.010+0000 7f2202a00700 1 -- 172.21.15.62:0/3888956468 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21f40095f0 con 0x7f2204068540 2024-04-20T21:55:06.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.010+0000 7f2202a00700 1 --2- 172.21.15.62:0/3888956468 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2204068540 0x7f2204197d60 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f21f400b5c0 tx=0x7f21f400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:55:06.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.010+0000 7f21fb400700 1 -- 172.21.15.62:0/3888956468 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f21f401d070 con 0x7f2204068540 2024-04-20T21:55:06.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.010+0000 7f21fb400700 1 -- 172.21.15.62:0/3888956468 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f21f4005e00 con 0x7f2204068540 2024-04-20T21:55:06.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.010+0000 7f2203e00700 1 -- 172.21.15.62:0/3888956468 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f220419c990 con 0x7f2204068540 2024-04-20T21:55:06.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.010+0000 7f21fb400700 1 -- 172.21.15.62:0/3888956468 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f21f4017400 con 0x7f2204068540 2024-04-20T21:55:06.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.011+0000 7f2203e00700 1 -- 172.21.15.62:0/3888956468 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f220419ce80 con 0x7f2204068540 2024-04-20T21:55:06.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.012+0000 7f2203e00700 1 -- 172.21.15.62:0/3888956468 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2204100eb0 con 0x7f2204068540 2024-04-20T21:55:06.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.013+0000 7f21fb400700 1 -- 172.21.15.62:0/3888956468 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f21f400f460 con 0x7f2204068540 2024-04-20T21:55:06.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.014+0000 7f21fb400700 1 --2- 172.21.15.62:0/3888956468 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f21e806ecc0 0x7f21e8071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:55:06.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.014+0000 7f21fb400700 1 -- 172.21.15.62:0/3888956468 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f21f4090370 con 0x7f2204068540 2024-04-20T21:55:06.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.014+0000 7f2202000700 1 --2- 172.21.15.62:0/3888956468 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f21e806ecc0 0x7f21e8071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:55:06.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.015+0000 7f2202000700 1 --2- 172.21.15.62:0/3888956468 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f21e806ecc0 0x7f21e8071170 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f2204199380 tx=0x7f21f000b5b0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:55:06.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.020+0000 7f21fb400700 1 -- 172.21.15.62:0/3888956468 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f21f4059100 con 0x7f2204068540 2024-04-20T21:55:06.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:55:05 smithi189 ceph-mon[57022]: pgmap v593: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:55:06.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.278+0000 7f2203e00700 1 -- 172.21.15.62:0/3888956468 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f220419d0f0 con 0x7f21e806ecc0 2024-04-20T21:55:06.277 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:55:05 smithi062 ceph-mon[76322]: pgmap v593: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:55:06.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.280+0000 7f21fb400700 1 -- 172.21.15.62:0/3888956468 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f220419d0f0 con 0x7f21e806ecc0 2024-04-20T21:55:06.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.283+0000 7f2203e00700 1 -- 172.21.15.62:0/3888956468 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f21e806ecc0 msgr2=0x7f21e8071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:55:06.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.283+0000 7f2203e00700 1 --2- 172.21.15.62:0/3888956468 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f21e806ecc0 0x7f21e8071170 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f2204199380 tx=0x7f21f000b5b0 comp rx=0 tx=0).stop 2024-04-20T21:55:06.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.283+0000 7f2203e00700 1 -- 172.21.15.62:0/3888956468 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2204068540 msgr2=0x7f2204197d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:55:06.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.283+0000 7f2203e00700 1 --2- 172.21.15.62:0/3888956468 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2204068540 0x7f2204197d60 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f21f400b5c0 tx=0x7f21f400be20 comp rx=0 tx=0).stop 2024-04-20T21:55:06.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.283+0000 7f2203e00700 1 -- 172.21.15.62:0/3888956468 shutdown_connections 2024-04-20T21:55:06.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.283+0000 7f2203e00700 1 --2- 172.21.15.62:0/3888956468 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f21e806ecc0 0x7f21e8071170 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:55:06.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.283+0000 7f2203e00700 1 --2- 172.21.15.62:0/3888956468 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2204068540 0x7f2204197d60 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:55:06.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.283+0000 7f2203e00700 1 --2- 172.21.15.62:0/3888956468 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f22041061b0 0x7f22041982a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:55:06.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.283+0000 7f2203e00700 1 -- 172.21.15.62:0/3888956468 >> 172.21.15.62:0/3888956468 conn(0x7f2204074fa0 msgr2=0x7f2204107100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:55:06.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.284+0000 7f2203e00700 1 -- 172.21.15.62:0/3888956468 shutdown_connections 2024-04-20T21:55:06.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.284+0000 7f2203e00700 1 -- 172.21.15.62:0/3888956468 wait complete. 2024-04-20T21:55:06.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.439+0000 7f51d5000700 1 -- 172.21.15.62:0/2591520052 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51d01024d0 msgr2=0x7f51d01028a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:55:06.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.439+0000 7f51d5000700 1 --2- 172.21.15.62:0/2591520052 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51d01024d0 0x7f51d01028a0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f51c0009910 tx=0x7f51c0009c20 comp rx=0 tx=0).stop 2024-04-20T21:55:06.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.439+0000 7f51d5000700 1 -- 172.21.15.62:0/2591520052 shutdown_connections 2024-04-20T21:55:06.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.439+0000 7f51d5000700 1 --2- 172.21.15.62:0/2591520052 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f51d0102e70 0x7f51d0106e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:55:06.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.439+0000 7f51d5000700 1 --2- 172.21.15.62:0/2591520052 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51d01024d0 0x7f51d01028a0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:55:06.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.439+0000 7f51d5000700 1 -- 172.21.15.62:0/2591520052 >> 172.21.15.62:0/2591520052 conn(0x7f51d00fdd90 msgr2=0x7f51d01001a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:55:06.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.439+0000 7f51d5000700 1 -- 172.21.15.62:0/2591520052 shutdown_connections 2024-04-20T21:55:06.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.439+0000 7f51d5000700 1 -- 172.21.15.62:0/2591520052 wait complete. 2024-04-20T21:55:06.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.440+0000 7f51d5000700 1 Processor -- start 2024-04-20T21:55:06.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.441+0000 7f51d5000700 1 -- start start 2024-04-20T21:55:06.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.441+0000 7f51d5000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f51d01024d0 0x7f51d0197e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:55:06.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.441+0000 7f51d5000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51d0102e70 0x7f51d01983a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:55:06.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.441+0000 7f51d5000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51d0198a80 con 0x7f51d0102e70 2024-04-20T21:55:06.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.441+0000 7f51d5000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51d019c7a0 con 0x7f51d01024d0 2024-04-20T21:55:06.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.441+0000 7f51cea00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51d0102e70 0x7f51d01983a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:55:06.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.441+0000 7f51cea00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51d0102e70 0x7f51d01983a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59060/0 (socket says 172.21.15.62:59060) 2024-04-20T21:55:06.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.441+0000 7f51cea00700 1 -- 172.21.15.62:0/714505040 learned_addr learned my addr 172.21.15.62:0/714505040 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:55:06.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.442+0000 7f51cea00700 1 -- 172.21.15.62:0/714505040 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f51d01024d0 msgr2=0x7f51d0197e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:55:06.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.442+0000 7f51cea00700 1 --2- 172.21.15.62:0/714505040 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f51d01024d0 0x7f51d0197e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:55:06.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.442+0000 7f51cea00700 1 -- 172.21.15.62:0/714505040 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51c00095f0 con 0x7f51d0102e70 2024-04-20T21:55:06.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.442+0000 7f51cea00700 1 --2- 172.21.15.62:0/714505040 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51d0102e70 0x7f51d01983a0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f51bc00e8f0 tx=0x7f51bc00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:55:06.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.442+0000 7f51c7e00700 1 -- 172.21.15.62:0/714505040 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f51bc009790 con 0x7f51d0102e70 2024-04-20T21:55:06.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.442+0000 7f51c7e00700 1 -- 172.21.15.62:0/714505040 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f51bc004cf0 con 0x7f51d0102e70 2024-04-20T21:55:06.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.442+0000 7f51c7e00700 1 -- 172.21.15.62:0/714505040 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f51bc005690 con 0x7f51d0102e70 2024-04-20T21:55:06.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.442+0000 7f51d5000700 1 -- 172.21.15.62:0/714505040 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51d019ca80 con 0x7f51d0102e70 2024-04-20T21:55:06.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.442+0000 7f51d5000700 1 -- 172.21.15.62:0/714505040 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51d019cfd0 con 0x7f51d0102e70 2024-04-20T21:55:06.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.444+0000 7f51d5000700 1 -- 172.21.15.62:0/714505040 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51d010a7e0 con 0x7f51d0102e70 2024-04-20T21:55:06.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.445+0000 7f51c7e00700 1 -- 172.21.15.62:0/714505040 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f51bc004750 con 0x7f51d0102e70 2024-04-20T21:55:06.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.445+0000 7f51c7e00700 1 --2- 172.21.15.62:0/714505040 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f51b406ecc0 0x7f51b4071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:55:06.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.446+0000 7f51c7e00700 1 -- 172.21.15.62:0/714505040 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f51bc014070 con 0x7f51d0102e70 2024-04-20T21:55:06.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.446+0000 7f51cf400700 1 --2- 172.21.15.62:0/714505040 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f51b406ecc0 0x7f51b4071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:55:06.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.446+0000 7f51cf400700 1 --2- 172.21.15.62:0/714505040 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f51b406ecc0 0x7f51b4071170 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f51c000b760 tx=0x7f51c0000bc0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:55:06.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.450+0000 7f51c7e00700 1 -- 172.21.15.62:0/714505040 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f51bc058000 con 0x7f51d0102e70 2024-04-20T21:55:06.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.684+0000 7f51d5000700 1 -- 172.21.15.62:0/714505040 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f51d019d2c0 con 0x7f51b406ecc0 2024-04-20T21:55:06.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.698+0000 7f51c7e00700 1 -- 172.21.15.62:0/714505040 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f51d019d2c0 con 0x7f51b406ecc0 2024-04-20T21:55:06.698 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T21:55:06.698 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (20m) 12m ago 34m 14.2M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T21:55:06.698 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (35m) 12m ago 35m 9512k - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T21:55:06.698 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (33m) 13m ago 33m 8942k - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T21:55:06.698 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (17m) 12m ago 35m 6660k - 19.0.0-3133-ga8c84f59 76bc6b91b726 2825d5ba11c6 2024-04-20T21:55:06.698 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (16m) 13m ago 33m 6665k - 19.0.0-3133-ga8c84f59 76bc6b91b726 20f746fa1bba 2024-04-20T21:55:06.698 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (20m) 12m ago 34m 74.2M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T21:55:06.698 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (12m) 12m ago 27m 11.2M - 19.0.0-3133-ga8c84f59 76bc6b91b726 de80bc90a683 2024-04-20T21:55:06.698 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (12m) 12m ago 27m 11.9M - 19.0.0-3133-ga8c84f59 76bc6b91b726 cd793b7ee98e 2024-04-20T21:55:06.698 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (27m) 13m ago 27m 117M - 18.2.0 dc2bc1663786 fb2a8a223db8 2024-04-20T21:55:06.698 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (27m) 13m ago 27m 55.6M - 18.2.0 dc2bc1663786 f2eec2ee8951 2024-04-20T21:55:06.698 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (23m) 12m ago 36m 511M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T21:55:06.699 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (22m) 13m ago 33m 452M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T21:55:06.699 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (18m) 12m ago 36m 55.9M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 019eb1479485 2024-04-20T21:55:06.699 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (18m) 13m ago 33m 39.8M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 55aac560aab6 2024-04-20T21:55:06.699 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (22m) 12m ago 35m 7952k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T21:55:06.699 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (22m) 13m ago 33m 8304k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T21:55:06.699 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (16m) 12m ago 32m 88.0M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 ee2bb901f26e 2024-04-20T21:55:06.699 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (15m) 12m ago 31m 92.3M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 9075152d4162 2024-04-20T21:55:06.699 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (15m) 12m ago 31m 74.4M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 2d4f3e38dc0e 2024-04-20T21:55:06.699 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (14m) 13m ago 30m 87.6M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 8f97b17ae0eb 2024-04-20T21:55:06.699 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (14m) 13m ago 30m 90.7M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 653d8079967d 2024-04-20T21:55:06.699 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (13m) 13m ago 29m 11.5M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 596dded14c12 2024-04-20T21:55:06.699 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (21m) 12m ago 34m 74.0M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T21:55:06.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.701+0000 7f51d5000700 1 -- 172.21.15.62:0/714505040 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f51b406ecc0 msgr2=0x7f51b4071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:55:06.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.701+0000 7f51d5000700 1 --2- 172.21.15.62:0/714505040 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f51b406ecc0 0x7f51b4071170 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f51c000b760 tx=0x7f51c0000bc0 comp rx=0 tx=0).stop 2024-04-20T21:55:06.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.701+0000 7f51d5000700 1 -- 172.21.15.62:0/714505040 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51d0102e70 msgr2=0x7f51d01983a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:55:06.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.701+0000 7f51d5000700 1 --2- 172.21.15.62:0/714505040 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51d0102e70 0x7f51d01983a0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f51bc00e8f0 tx=0x7f51bc00ecb0 comp rx=0 tx=0).stop 2024-04-20T21:55:06.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.701+0000 7f51d5000700 1 -- 172.21.15.62:0/714505040 shutdown_connections 2024-04-20T21:55:06.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.701+0000 7f51d5000700 1 --2- 172.21.15.62:0/714505040 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f51b406ecc0 0x7f51b4071170 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:55:06.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.701+0000 7f51d5000700 1 --2- 172.21.15.62:0/714505040 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f51d01024d0 0x7f51d0197e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:55:06.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.701+0000 7f51d5000700 1 --2- 172.21.15.62:0/714505040 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51d0102e70 0x7f51d01983a0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:55:06.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.701+0000 7f51d5000700 1 -- 172.21.15.62:0/714505040 >> 172.21.15.62:0/714505040 conn(0x7f51d00fdd90 msgr2=0x7f51d0100180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:55:06.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.701+0000 7f51d5000700 1 -- 172.21.15.62:0/714505040 shutdown_connections 2024-04-20T21:55:06.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.701+0000 7f51d5000700 1 -- 172.21.15.62:0/714505040 wait complete. 2024-04-20T21:55:06.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.861+0000 7f939b600700 1 -- 172.21.15.62:0/3083614819 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f939410c8f0 msgr2=0x7f939410ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:55:06.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.861+0000 7f939b600700 1 --2- 172.21.15.62:0/3083614819 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f939410c8f0 0x7f939410ccc0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f9388009910 tx=0x7f9388009c20 comp rx=0 tx=0).stop 2024-04-20T21:55:06.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.861+0000 7f939b600700 1 -- 172.21.15.62:0/3083614819 shutdown_connections 2024-04-20T21:55:06.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.861+0000 7f939b600700 1 --2- 172.21.15.62:0/3083614819 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9394106b80 0x7f9394106ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:55:06.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.862+0000 7f939b600700 1 --2- 172.21.15.62:0/3083614819 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f939410c8f0 0x7f939410ccc0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:55:06.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.862+0000 7f939b600700 1 -- 172.21.15.62:0/3083614819 >> 172.21.15.62:0/3083614819 conn(0x7f9394076060 msgr2=0x7f9394078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:55:06.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.862+0000 7f939b600700 1 -- 172.21.15.62:0/3083614819 shutdown_connections 2024-04-20T21:55:06.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.862+0000 7f939b600700 1 -- 172.21.15.62:0/3083614819 wait complete. 2024-04-20T21:55:06.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.863+0000 7f939b600700 1 Processor -- start 2024-04-20T21:55:06.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.863+0000 7f939b600700 1 -- start start 2024-04-20T21:55:06.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.863+0000 7f939b600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9394106b80 0x7f939419c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:55:06.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.864+0000 7f939b600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f939410c8f0 0x7f939419ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:55:06.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.864+0000 7f939b600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f939419d100 con 0x7f939410c8f0 2024-04-20T21:55:06.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.864+0000 7f939b600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93941a0e90 con 0x7f9394106b80 2024-04-20T21:55:06.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.864+0000 7f9393e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f939410c8f0 0x7f939419ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:55:06.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.864+0000 7f9393e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f939410c8f0 0x7f939419ca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59076/0 (socket says 172.21.15.62:59076) 2024-04-20T21:55:06.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.864+0000 7f9393e00700 1 -- 172.21.15.62:0/1650577840 learned_addr learned my addr 172.21.15.62:0/1650577840 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:55:06.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.864+0000 7f9398c00700 1 --2- 172.21.15.62:0/1650577840 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9394106b80 0x7f939419c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:55:06.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.864+0000 7f9393e00700 1 -- 172.21.15.62:0/1650577840 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9394106b80 msgr2=0x7f939419c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:55:06.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.864+0000 7f9393e00700 1 --2- 172.21.15.62:0/1650577840 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9394106b80 0x7f939419c4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:55:06.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.864+0000 7f9393e00700 1 -- 172.21.15.62:0/1650577840 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93880095f0 con 0x7f939410c8f0 2024-04-20T21:55:06.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.864+0000 7f9398c00700 1 --2- 172.21.15.62:0/1650577840 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9394106b80 0x7f939419c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T21:55:06.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.865+0000 7f9393e00700 1 --2- 172.21.15.62:0/1650577840 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f939410c8f0 0x7f939419ca20 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f938000b4b0 tx=0x7f938000b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:55:06.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.865+0000 7f9391600700 1 -- 172.21.15.62:0/1650577840 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9380011800 con 0x7f939410c8f0 2024-04-20T21:55:06.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.865+0000 7f9391600700 1 -- 172.21.15.62:0/1650577840 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9380011e20 con 0x7f939410c8f0 2024-04-20T21:55:06.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.865+0000 7f9391600700 1 -- 172.21.15.62:0/1650577840 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f938000f2e0 con 0x7f939410c8f0 2024-04-20T21:55:06.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.865+0000 7f939b600700 1 -- 172.21.15.62:0/1650577840 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93941a1170 con 0x7f939410c8f0 2024-04-20T21:55:06.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.865+0000 7f939b600700 1 -- 172.21.15.62:0/1650577840 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93941a16c0 con 0x7f939410c8f0 2024-04-20T21:55:06.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.867+0000 7f939b600700 1 -- 172.21.15.62:0/1650577840 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9394109b20 con 0x7f939410c8f0 2024-04-20T21:55:06.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.868+0000 7f9391600700 1 -- 172.21.15.62:0/1650577840 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f9380010380 con 0x7f939410c8f0 2024-04-20T21:55:06.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.869+0000 7f9391600700 1 --2- 172.21.15.62:0/1650577840 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f937c06ecc0 0x7f937c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:55:06.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.869+0000 7f9391600700 1 -- 172.21.15.62:0/1650577840 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f938008e990 con 0x7f939410c8f0 2024-04-20T21:55:06.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.869+0000 7f9398c00700 1 --2- 172.21.15.62:0/1650577840 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f937c06ecc0 0x7f937c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:55:06.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.870+0000 7f9398c00700 1 --2- 172.21.15.62:0/1650577840 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f937c06ecc0 0x7f937c071170 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f9388006010 tx=0x7f938800b540 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:55:06.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:06.874+0000 7f9391600700 1 -- 172.21.15.62:0/1650577840 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f9380057870 con 0x7f939410c8f0 2024-04-20T21:55:07.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:55:06 smithi062 ceph-mon[76322]: from='client.35096 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:55:07.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:07.205+0000 7f939b600700 1 -- 172.21.15.62:0/1650577840 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f939404f2a0 con 0x7f939410c8f0 2024-04-20T21:55:07.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:07.205+0000 7f9391600700 1 -- 172.21.15.62:0/1650577840 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7f938005af30 con 0x7f939410c8f0 2024-04-20T21:55:07.205 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:55:07.205 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T21:55:07.205 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:55:07.205 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:55:07.205 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T21:55:07.206 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:55:07.206 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:55:07.206 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T21:55:07.206 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 6 2024-04-20T21:55:07.206 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:55:07.206 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T21:55:07.206 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 2, 2024-04-20T21:55:07.206 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:55:07.206 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:55:07.206 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T21:55:07.206 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 2, 2024-04-20T21:55:07.206 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 12 2024-04-20T21:55:07.206 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T21:55:07.206 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:55:07.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:07.208+0000 7f939b600700 1 -- 172.21.15.62:0/1650577840 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f937c06ecc0 msgr2=0x7f937c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:55:07.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:07.208+0000 7f939b600700 1 --2- 172.21.15.62:0/1650577840 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f937c06ecc0 0x7f937c071170 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f9388006010 tx=0x7f938800b540 comp rx=0 tx=0).stop 2024-04-20T21:55:07.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:07.208+0000 7f939b600700 1 -- 172.21.15.62:0/1650577840 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f939410c8f0 msgr2=0x7f939419ca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:55:07.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:07.208+0000 7f939b600700 1 --2- 172.21.15.62:0/1650577840 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f939410c8f0 0x7f939419ca20 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f938000b4b0 tx=0x7f938000b870 comp rx=0 tx=0).stop 2024-04-20T21:55:07.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:07.209+0000 7f939b600700 1 -- 172.21.15.62:0/1650577840 shutdown_connections 2024-04-20T21:55:07.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:07.209+0000 7f939b600700 1 --2- 172.21.15.62:0/1650577840 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f937c06ecc0 0x7f937c071170 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:55:07.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:07.209+0000 7f939b600700 1 --2- 172.21.15.62:0/1650577840 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9394106b80 0x7f939419c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:55:07.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:07.209+0000 7f939b600700 1 --2- 172.21.15.62:0/1650577840 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f939410c8f0 0x7f939419ca20 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:55:07.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:07.209+0000 7f939b600700 1 -- 172.21.15.62:0/1650577840 >> 172.21.15.62:0/1650577840 conn(0x7f9394076060 msgr2=0x7f9394077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:55:07.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:07.209+0000 7f939b600700 1 -- 172.21.15.62:0/1650577840 shutdown_connections 2024-04-20T21:55:07.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:07.209+0000 7f939b600700 1 -- 172.21.15.62:0/1650577840 wait complete. 2024-04-20T21:55:07.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:55:06 smithi189 ceph-mon[57022]: from='client.35096 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:55:07.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:07.365+0000 7f6740a00700 1 -- 172.21.15.62:0/735278814 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f673c0ffd60 msgr2=0x7f673c103d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:55:07.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:07.365+0000 7f6740a00700 1 --2- 172.21.15.62:0/735278814 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f673c0ffd60 0x7f673c103d30 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f6728009910 tx=0x7f6728009c20 comp rx=0 tx=0).stop 2024-04-20T21:55:07.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:07.366+0000 7f6740a00700 1 -- 172.21.15.62:0/735278814 shutdown_connections 2024-04-20T21:55:07.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:07.366+0000 7f6740a00700 1 --2- 172.21.15.62:0/735278814 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f673c0ffd60 0x7f673c103d30 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:55:07.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:07.366+0000 7f6740a00700 1 --2- 172.21.15.62:0/735278814 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f673c0ff3c0 0x7f673c0ff790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:55:07.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:07.366+0000 7f6740a00700 1 -- 172.21.15.62:0/735278814 >> 172.21.15.62:0/735278814 conn(0x7f673c0fac80 msgr2=0x7f673c0fd090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:55:07.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:07.366+0000 7f6740a00700 1 -- 172.21.15.62:0/735278814 shutdown_connections 2024-04-20T21:55:07.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:07.366+0000 7f6740a00700 1 -- 172.21.15.62:0/735278814 wait complete. 2024-04-20T21:55:07.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:07.367+0000 7f6740a00700 1 Processor -- start 2024-04-20T21:55:07.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:07.367+0000 7f6740a00700 1 -- start start 2024-04-20T21:55:07.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:07.367+0000 7f6740a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f673c0ff3c0 0x7f673c198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:55:07.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:07.367+0000 7f6740a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f673c0ffd60 0x7f673c1985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:55:07.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:07.367+0000 7f6740a00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f673c198cb0 con 0x7f673c0ff3c0 2024-04-20T21:55:07.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:07.367+0000 7f6740a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f673c19ca40 con 0x7f673c0ffd60 2024-04-20T21:55:07.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:07.368+0000 7f6739e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f673c0ff3c0 0x7f673c198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:55:07.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:07.368+0000 7f6739e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f673c0ff3c0 0x7f673c198090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59100/0 (socket says 172.21.15.62:59100) 2024-04-20T21:55:07.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:07.368+0000 7f6739e00700 1 -- 172.21.15.62:0/1996534837 learned_addr learned my addr 172.21.15.62:0/1996534837 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:55:07.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:07.368+0000 7f6739400700 1 --2- 172.21.15.62:0/1996534837 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f673c0ffd60 0x7f673c1985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:55:07.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:07.368+0000 7f6739e00700 1 -- 172.21.15.62:0/1996534837 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f673c0ffd60 msgr2=0x7f673c1985d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:55:07.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:07.368+0000 7f6739e00700 1 --2- 172.21.15.62:0/1996534837 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f673c0ffd60 0x7f673c1985d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:55:07.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:07.368+0000 7f6739e00700 1 -- 172.21.15.62:0/1996534837 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67280095f0 con 0x7f673c0ff3c0 2024-04-20T21:55:07.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:07.368+0000 7f6739e00700 1 --2- 172.21.15.62:0/1996534837 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f673c0ff3c0 0x7f673c198090 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f672c00d8d0 tx=0x7f672c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:55:07.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:07.369+0000 7f6732a00700 1 -- 172.21.15.62:0/1996534837 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f672c009940 con 0x7f673c0ff3c0 2024-04-20T21:55:07.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:07.369+0000 7f6732a00700 1 -- 172.21.15.62:0/1996534837 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f672c010460 con 0x7f673c0ff3c0 2024-04-20T21:55:07.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:07.369+0000 7f6740a00700 1 -- 172.21.15.62:0/1996534837 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f673c19cd20 con 0x7f673c0ff3c0 2024-04-20T21:55:07.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:07.369+0000 7f6732a00700 1 -- 172.21.15.62:0/1996534837 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f672c00f5f0 con 0x7f673c0ff3c0 2024-04-20T21:55:07.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:07.369+0000 7f6740a00700 1 -- 172.21.15.62:0/1996534837 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f673c19d270 con 0x7f673c0ff3c0 2024-04-20T21:55:07.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:07.370+0000 7f6740a00700 1 -- 172.21.15.62:0/1996534837 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f673c04f2a0 con 0x7f673c0ff3c0 2024-04-20T21:55:07.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:07.371+0000 7f6732a00700 1 -- 172.21.15.62:0/1996534837 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f672c0105d0 con 0x7f673c0ff3c0 2024-04-20T21:55:07.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:07.372+0000 7f6732a00700 1 --2- 172.21.15.62:0/1996534837 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f672006ecc0 0x7f6720071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:55:07.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:07.372+0000 7f6732a00700 1 -- 172.21.15.62:0/1996534837 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f672c08ee10 con 0x7f673c0ff3c0 2024-04-20T21:55:07.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:07.372+0000 7f6739400700 1 --2- 172.21.15.62:0/1996534837 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f672006ecc0 0x7f6720071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:55:07.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:07.373+0000 7f6739400700 1 --2- 172.21.15.62:0/1996534837 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f672006ecc0 0x7f6720071170 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f6728000c00 tx=0x7f6728005fd0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:55:07.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:07.376+0000 7f6732a00700 1 -- 172.21.15.62:0/1996534837 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f672c057ba0 con 0x7f673c0ff3c0 2024-04-20T21:55:07.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:07.658+0000 7f6740a00700 1 -- 172.21.15.62:0/1996534837 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f673c1993f0 con 0x7f673c0ff3c0 2024-04-20T21:55:07.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:07.658+0000 7f6732a00700 1 -- 172.21.15.62:0/1996534837 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 25 v25) v1 ==== 76+0+1887 (secure 0 0 0) 0x7f672c016770 con 0x7f673c0ff3c0 2024-04-20T21:55:07.659 INFO:teuthology.orchestra.run.smithi062.stdout:e25 2024-04-20T21:55:07.659 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T21:55:07.659 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T21:55:07.659 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T21:55:07.659 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:55:07.659 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T21:55:07.659 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T21:55:07.659 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 20 2024-04-20T21:55:07.660 INFO:teuthology.orchestra.run.smithi062.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-20T21:55:07.660 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T21:55:07.660 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T21:42:01.559061+0000 2024-04-20T21:55:07.660 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T21:55:07.660 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T21:55:07.660 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T21:55:07.660 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T21:55:07.660 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T21:55:07.660 INFO:teuthology.orchestra.run.smithi062.stdout:max_xattr_size 65536 2024-04-20T21:55:07.660 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T21:55:07.660 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T21:55:07.660 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 81 2024-04-20T21:55:07.660 INFO:teuthology.orchestra.run.smithi062.stdout:snap_rstat 1 2024-04-20T21:55:07.660 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T21:55:07.661 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 2 2024-04-20T21:55:07.661 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1 2024-04-20T21:55:07.661 INFO:teuthology.orchestra.run.smithi062.stdout:up {} 2024-04-20T21:55:07.661 INFO:teuthology.orchestra.run.smithi062.stdout:failed 0,1 2024-04-20T21:55:07.661 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T21:55:07.661 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T21:55:07.661 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T21:55:07.661 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T21:55:07.661 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data enabled 2024-04-20T21:55:07.661 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T21:55:07.661 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T21:55:07.661 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 1 2024-04-20T21:55:07.661 INFO:teuthology.orchestra.run.smithi062.stdout:qdb_cluster leader: 0 members: 2024-04-20T21:55:07.661 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:55:07.661 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:55:07.662 INFO:teuthology.orchestra.run.smithi062.stdout:Standby daemons: 2024-04-20T21:55:07.662 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:55:07.662 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ryqblb{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6826/4113916205,v1:172.21.15.62:6827/4113916205] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T21:55:07.662 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ozizwi{-1:34476} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6828/3801975165,v1:172.21.15.62:6829/3801975165] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T21:55:07.662 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.xbgspw{-1:44285} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6826/3996040526,v1:172.21.15.189:6827/3996040526] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:55:07.662 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.dvimer{-1:44289} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6824/1743111843,v1:172.21.15.189:6825/1743111843] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:55:07.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:07.661+0000 7f6740a00700 1 -- 172.21.15.62:0/1996534837 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f672006ecc0 msgr2=0x7f6720071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:55:07.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:07.661+0000 7f6740a00700 1 --2- 172.21.15.62:0/1996534837 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f672006ecc0 0x7f6720071170 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f6728000c00 tx=0x7f6728005fd0 comp rx=0 tx=0).stop 2024-04-20T21:55:07.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:07.661+0000 7f6740a00700 1 -- 172.21.15.62:0/1996534837 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f673c0ff3c0 msgr2=0x7f673c198090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:55:07.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:07.661+0000 7f6740a00700 1 --2- 172.21.15.62:0/1996534837 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f673c0ff3c0 0x7f673c198090 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f672c00d8d0 tx=0x7f672c00dc90 comp rx=0 tx=0).stop 2024-04-20T21:55:07.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:07.661+0000 7f6740a00700 1 -- 172.21.15.62:0/1996534837 shutdown_connections 2024-04-20T21:55:07.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:07.661+0000 7f6740a00700 1 --2- 172.21.15.62:0/1996534837 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f672006ecc0 0x7f6720071170 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:55:07.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:07.661+0000 7f6740a00700 1 --2- 172.21.15.62:0/1996534837 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f673c0ff3c0 0x7f673c198090 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:55:07.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:07.662+0000 7f6740a00700 1 --2- 172.21.15.62:0/1996534837 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f673c0ffd60 0x7f673c1985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:55:07.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:07.662+0000 7f6740a00700 1 -- 172.21.15.62:0/1996534837 >> 172.21.15.62:0/1996534837 conn(0x7f673c0fac80 msgr2=0x7f673c102d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:55:07.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:07.662+0000 7f6740a00700 1 -- 172.21.15.62:0/1996534837 shutdown_connections 2024-04-20T21:55:07.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:07.662+0000 7f6740a00700 1 -- 172.21.15.62:0/1996534837 wait complete. 2024-04-20T21:55:07.663 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 25 2024-04-20T21:55:07.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:07.823+0000 7fa1ebe00700 1 -- 172.21.15.62:0/4039884606 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1ec1012d0 msgr2=0x7fa1ec1016a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:55:07.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:07.823+0000 7fa1ebe00700 1 --2- 172.21.15.62:0/4039884606 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1ec1012d0 0x7fa1ec1016a0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc009910 tx=0x7fa1dc009c20 comp rx=0 tx=0).stop 2024-04-20T21:55:07.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:07.824+0000 7fa1ebe00700 1 -- 172.21.15.62:0/4039884606 shutdown_connections 2024-04-20T21:55:07.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:07.824+0000 7fa1ebe00700 1 --2- 172.21.15.62:0/4039884606 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1ec101c70 0x7fa1ec10a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:55:07.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:07.824+0000 7fa1ebe00700 1 --2- 172.21.15.62:0/4039884606 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1ec1012d0 0x7fa1ec1016a0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:55:07.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:07.824+0000 7fa1ebe00700 1 -- 172.21.15.62:0/4039884606 >> 172.21.15.62:0/4039884606 conn(0x7fa1ec0faad0 msgr2=0x7fa1ec0fcee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:55:07.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:07.824+0000 7fa1ebe00700 1 -- 172.21.15.62:0/4039884606 shutdown_connections 2024-04-20T21:55:07.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:07.824+0000 7fa1ebe00700 1 -- 172.21.15.62:0/4039884606 wait complete. 2024-04-20T21:55:07.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:07.825+0000 7fa1ebe00700 1 Processor -- start 2024-04-20T21:55:07.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:07.825+0000 7fa1ebe00700 1 -- start start 2024-04-20T21:55:07.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:07.825+0000 7fa1ebe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1ec1012d0 0x7fa1ec197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:55:07.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:07.825+0000 7fa1ebe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1ec101c70 0x7fa1ec198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:55:07.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:07.825+0000 7fa1ebe00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1ec198b50 con 0x7fa1ec101c70 2024-04-20T21:55:07.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:07.826+0000 7fa1ebe00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1ec19c8e0 con 0x7fa1ec1012d0 2024-04-20T21:55:07.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:07.826+0000 7fa1ea000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1ec101c70 0x7fa1ec198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:55:07.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:07.826+0000 7fa1ea000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1ec101c70 0x7fa1ec198470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59124/0 (socket says 172.21.15.62:59124) 2024-04-20T21:55:07.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:07.826+0000 7fa1ea000700 1 -- 172.21.15.62:0/2797188275 learned_addr learned my addr 172.21.15.62:0/2797188275 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:55:07.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:07.826+0000 7fa1eaa00700 1 --2- 172.21.15.62:0/2797188275 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1ec1012d0 0x7fa1ec197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:55:07.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:07.826+0000 7fa1ea000700 1 -- 172.21.15.62:0/2797188275 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1ec1012d0 msgr2=0x7fa1ec197f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:55:07.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:07.826+0000 7fa1ea000700 1 --2- 172.21.15.62:0/2797188275 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1ec1012d0 0x7fa1ec197f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:55:07.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:07.826+0000 7fa1ea000700 1 -- 172.21.15.62:0/2797188275 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1dc0095f0 con 0x7fa1ec101c70 2024-04-20T21:55:07.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:07.827+0000 7fa1ea000700 1 --2- 172.21.15.62:0/2797188275 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1ec101c70 0x7fa1ec198470 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fa1d800d900 tx=0x7fa1d800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:55:07.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:07.827+0000 7fa1e3400700 1 -- 172.21.15.62:0/2797188275 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa1d8009900 con 0x7fa1ec101c70 2024-04-20T21:55:07.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:07.827+0000 7fa1e3400700 1 -- 172.21.15.62:0/2797188275 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa1d8010460 con 0x7fa1ec101c70 2024-04-20T21:55:07.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:07.827+0000 7fa1e3400700 1 -- 172.21.15.62:0/2797188275 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa1d800f5f0 con 0x7fa1ec101c70 2024-04-20T21:55:07.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:07.827+0000 7fa1ebe00700 1 -- 172.21.15.62:0/2797188275 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1ec19cbc0 con 0x7fa1ec101c70 2024-04-20T21:55:07.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:07.827+0000 7fa1ebe00700 1 -- 172.21.15.62:0/2797188275 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1ec19d110 con 0x7fa1ec101c70 2024-04-20T21:55:07.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:07.828+0000 7fa1ebe00700 1 -- 172.21.15.62:0/2797188275 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1ec109be0 con 0x7fa1ec101c70 2024-04-20T21:55:07.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:07.830+0000 7fa1e3400700 1 -- 172.21.15.62:0/2797188275 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fa1d80105d0 con 0x7fa1ec101c70 2024-04-20T21:55:07.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:07.830+0000 7fa1e3400700 1 --2- 172.21.15.62:0/2797188275 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa1d006ecc0 0x7fa1d0071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:55:07.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:07.831+0000 7fa1e3400700 1 -- 172.21.15.62:0/2797188275 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fa1d808eca0 con 0x7fa1ec101c70 2024-04-20T21:55:07.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:07.831+0000 7fa1eaa00700 1 --2- 172.21.15.62:0/2797188275 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa1d006ecc0 0x7fa1d0071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:55:07.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:07.832+0000 7fa1eaa00700 1 --2- 172.21.15.62:0/2797188275 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa1d006ecc0 0x7fa1d0071170 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc00b5c0 tx=0x7fa1dc01a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:55:07.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:07.835+0000 7fa1e3400700 1 -- 172.21.15.62:0/2797188275 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fa1d8057a30 con 0x7fa1ec101c70 2024-04-20T21:55:08.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:55:07 smithi062 ceph-mon[76322]: from='client.35100 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:55:08.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:55:07 smithi062 ceph-mon[76322]: from='client.35104 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:55:08.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:55:07 smithi062 ceph-mon[76322]: pgmap v594: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:55:08.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:55:07 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/1650577840' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:55:08.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:55:07 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/1996534837' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T21:55:08.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:08.101+0000 7fa1ebe00700 1 -- 172.21.15.62:0/2797188275 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa1ec19d380 con 0x7fa1d006ecc0 2024-04-20T21:55:08.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:08.104+0000 7fa1e3400700 1 -- 172.21.15.62:0/2797188275 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fa1ec19d380 con 0x7fa1d006ecc0 2024-04-20T21:55:08.104 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:55:08.104 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T21:55:08.104 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T21:55:08.104 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-20T21:55:08.104 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T21:55:08.104 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr", 2024-04-20T21:55:08.104 INFO:teuthology.orchestra.run.smithi062.stdout: "crash", 2024-04-20T21:55:08.104 INFO:teuthology.orchestra.run.smithi062.stdout: "mon", 2024-04-20T21:55:08.105 INFO:teuthology.orchestra.run.smithi062.stdout: "osd" 2024-04-20T21:55:08.105 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T21:55:08.105 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "14/23 daemons upgraded", 2024-04-20T21:55:08.105 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading mds daemons", 2024-04-20T21:55:08.105 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T21:55:08.105 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:55:08.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:08.107+0000 7fa1ebe00700 1 -- 172.21.15.62:0/2797188275 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa1d006ecc0 msgr2=0x7fa1d0071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:55:08.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:08.107+0000 7fa1ebe00700 1 --2- 172.21.15.62:0/2797188275 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa1d006ecc0 0x7fa1d0071170 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc00b5c0 tx=0x7fa1dc01a040 comp rx=0 tx=0).stop 2024-04-20T21:55:08.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:08.107+0000 7fa1ebe00700 1 -- 172.21.15.62:0/2797188275 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1ec101c70 msgr2=0x7fa1ec198470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:55:08.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:08.107+0000 7fa1ebe00700 1 --2- 172.21.15.62:0/2797188275 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1ec101c70 0x7fa1ec198470 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fa1d800d900 tx=0x7fa1d800dcc0 comp rx=0 tx=0).stop 2024-04-20T21:55:08.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:08.107+0000 7fa1ebe00700 1 -- 172.21.15.62:0/2797188275 shutdown_connections 2024-04-20T21:55:08.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:08.107+0000 7fa1ebe00700 1 --2- 172.21.15.62:0/2797188275 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa1d006ecc0 0x7fa1d0071170 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:55:08.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:08.107+0000 7fa1ebe00700 1 --2- 172.21.15.62:0/2797188275 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1ec1012d0 0x7fa1ec197f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:55:08.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:08.107+0000 7fa1ebe00700 1 --2- 172.21.15.62:0/2797188275 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1ec101c70 0x7fa1ec198470 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:55:08.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:08.107+0000 7fa1ebe00700 1 -- 172.21.15.62:0/2797188275 >> 172.21.15.62:0/2797188275 conn(0x7fa1ec0faad0 msgr2=0x7fa1ec106d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:55:08.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:08.107+0000 7fa1ebe00700 1 -- 172.21.15.62:0/2797188275 shutdown_connections 2024-04-20T21:55:08.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:08.107+0000 7fa1ebe00700 1 -- 172.21.15.62:0/2797188275 wait complete. 2024-04-20T21:55:08.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:55:07 smithi189 ceph-mon[57022]: from='client.35100 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:55:08.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:55:07 smithi189 ceph-mon[57022]: from='client.35104 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:55:08.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:55:07 smithi189 ceph-mon[57022]: pgmap v594: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:55:08.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:55:07 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/1650577840' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:55:08.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:55:07 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/1996534837' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T21:55:08.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:08.268+0000 7fe205200700 1 -- 172.21.15.62:0/3059984946 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2001012f0 msgr2=0x7fe2001016c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:55:08.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:08.268+0000 7fe205200700 1 --2- 172.21.15.62:0/3059984946 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2001012f0 0x7fe2001016c0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fe1f0009910 tx=0x7fe1f0009c20 comp rx=0 tx=0).stop 2024-04-20T21:55:08.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:08.268+0000 7fe205200700 1 -- 172.21.15.62:0/3059984946 shutdown_connections 2024-04-20T21:55:08.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:08.268+0000 7fe205200700 1 --2- 172.21.15.62:0/3059984946 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe200101c00 0x7fe20010a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:55:08.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:08.268+0000 7fe205200700 1 --2- 172.21.15.62:0/3059984946 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2001012f0 0x7fe2001016c0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:55:08.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:08.268+0000 7fe205200700 1 -- 172.21.15.62:0/3059984946 >> 172.21.15.62:0/3059984946 conn(0x7fe2000fac60 msgr2=0x7fe2000fd070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:55:08.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:08.269+0000 7fe205200700 1 -- 172.21.15.62:0/3059984946 shutdown_connections 2024-04-20T21:55:08.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:08.269+0000 7fe205200700 1 -- 172.21.15.62:0/3059984946 wait complete. 2024-04-20T21:55:08.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:08.270+0000 7fe205200700 1 Processor -- start 2024-04-20T21:55:08.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:08.270+0000 7fe205200700 1 -- start start 2024-04-20T21:55:08.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:08.270+0000 7fe205200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2001012f0 0x7fe2000ff120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:55:08.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:08.270+0000 7fe205200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe200101c00 0x7fe2000ff660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:55:08.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:08.270+0000 7fe205200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2000ffba0 con 0x7fe2001012f0 2024-04-20T21:55:08.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:08.270+0000 7fe205200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2000ffce0 con 0x7fe200101c00 2024-04-20T21:55:08.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:08.271+0000 7fe1fde00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2001012f0 0x7fe2000ff120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:55:08.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:08.271+0000 7fe1fde00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2001012f0 0x7fe2000ff120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59140/0 (socket says 172.21.15.62:59140) 2024-04-20T21:55:08.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:08.271+0000 7fe1fde00700 1 -- 172.21.15.62:0/1599970015 learned_addr learned my addr 172.21.15.62:0/1599970015 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:55:08.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:08.271+0000 7fe1fd400700 1 --2- 172.21.15.62:0/1599970015 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe200101c00 0x7fe2000ff660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:55:08.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:08.271+0000 7fe1fde00700 1 -- 172.21.15.62:0/1599970015 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe200101c00 msgr2=0x7fe2000ff660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:55:08.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:08.271+0000 7fe1fde00700 1 --2- 172.21.15.62:0/1599970015 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe200101c00 0x7fe2000ff660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:55:08.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:08.271+0000 7fe1fde00700 1 -- 172.21.15.62:0/1599970015 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1f00095f0 con 0x7fe2001012f0 2024-04-20T21:55:08.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:08.272+0000 7fe1fde00700 1 --2- 172.21.15.62:0/1599970015 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2001012f0 0x7fe2000ff120 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fe1f000b5c0 tx=0x7fe1f000be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:55:08.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:08.272+0000 7fe1f6a00700 1 -- 172.21.15.62:0/1599970015 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe1f001d070 con 0x7fe2001012f0 2024-04-20T21:55:08.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:08.272+0000 7fe205200700 1 -- 172.21.15.62:0/1599970015 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2000fff60 con 0x7fe2001012f0 2024-04-20T21:55:08.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:08.272+0000 7fe1f6a00700 1 -- 172.21.15.62:0/1599970015 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe1f000f460 con 0x7fe2001012f0 2024-04-20T21:55:08.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:08.272+0000 7fe1f6a00700 1 -- 172.21.15.62:0/1599970015 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe1f0017610 con 0x7fe2001012f0 2024-04-20T21:55:08.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:08.272+0000 7fe205200700 1 -- 172.21.15.62:0/1599970015 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2001a2490 con 0x7fe2001012f0 2024-04-20T21:55:08.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:08.273+0000 7fe205200700 1 -- 172.21.15.62:0/1599970015 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe200107970 con 0x7fe2001012f0 2024-04-20T21:55:08.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:08.275+0000 7fe1f6a00700 1 -- 172.21.15.62:0/1599970015 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe1f000f5d0 con 0x7fe2001012f0 2024-04-20T21:55:08.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:08.276+0000 7fe1f6a00700 1 --2- 172.21.15.62:0/1599970015 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe1e406ecc0 0x7fe1e4071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:55:08.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:08.276+0000 7fe1f6a00700 1 -- 172.21.15.62:0/1599970015 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fe1f0090660 con 0x7fe2001012f0 2024-04-20T21:55:08.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:08.276+0000 7fe1fd400700 1 --2- 172.21.15.62:0/1599970015 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe1e406ecc0 0x7fe1e4071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:55:08.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:08.277+0000 7fe1fd400700 1 --2- 172.21.15.62:0/1599970015 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe1e406ecc0 0x7fe1e4071170 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fe1ec006fd0 tx=0x7fe1ec008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:55:08.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:08.279+0000 7fe1f6a00700 1 -- 172.21.15.62:0/1599970015 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fe1f0059470 con 0x7fe2001012f0 2024-04-20T21:55:08.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:08.593+0000 7fe205200700 1 -- 172.21.15.62:0/1599970015 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fe200100710 con 0x7fe2001012f0 2024-04-20T21:55:08.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:08.594+0000 7fe1f6a00700 1 -- 172.21.15.62:0/1599970015 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7fe1f005cb30 con 0x7fe2001012f0 2024-04-20T21:55:08.593 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-20T21:55:08.593 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-20T21:55:08.593 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is degraded 2024-04-20T21:55:08.593 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T21:55:08.593 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T21:55:08.593 INFO:teuthology.orchestra.run.smithi062.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-20T21:55:08.593 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-20T21:55:08.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:08.596+0000 7fe205200700 1 -- 172.21.15.62:0/1599970015 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe1e406ecc0 msgr2=0x7fe1e4071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:55:08.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:08.596+0000 7fe205200700 1 --2- 172.21.15.62:0/1599970015 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe1e406ecc0 0x7fe1e4071170 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fe1ec006fd0 tx=0x7fe1ec008040 comp rx=0 tx=0).stop 2024-04-20T21:55:08.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:08.596+0000 7fe205200700 1 -- 172.21.15.62:0/1599970015 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2001012f0 msgr2=0x7fe2000ff120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:55:08.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:08.596+0000 7fe205200700 1 --2- 172.21.15.62:0/1599970015 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2001012f0 0x7fe2000ff120 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fe1f000b5c0 tx=0x7fe1f000be00 comp rx=0 tx=0).stop 2024-04-20T21:55:08.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:08.596+0000 7fe205200700 1 -- 172.21.15.62:0/1599970015 shutdown_connections 2024-04-20T21:55:08.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:08.596+0000 7fe205200700 1 --2- 172.21.15.62:0/1599970015 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe1e406ecc0 0x7fe1e4071170 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:55:08.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:08.597+0000 7fe205200700 1 --2- 172.21.15.62:0/1599970015 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2001012f0 0x7fe2000ff120 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:55:08.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:08.597+0000 7fe205200700 1 --2- 172.21.15.62:0/1599970015 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe200101c00 0x7fe2000ff660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:55:08.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:08.597+0000 7fe205200700 1 -- 172.21.15.62:0/1599970015 >> 172.21.15.62:0/1599970015 conn(0x7fe2000fac60 msgr2=0x7fe200104ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:55:08.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:08.597+0000 7fe205200700 1 -- 172.21.15.62:0/1599970015 shutdown_connections 2024-04-20T21:55:08.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:08.597+0000 7fe205200700 1 -- 172.21.15.62:0/1599970015 wait complete. 2024-04-20T21:55:09.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:55:08 smithi189 ceph-mon[57022]: from='client.35116 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:55:09.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:55:08 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/1599970015' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T21:55:09.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:55:08 smithi062 ceph-mon[76322]: from='client.35116 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:55:09.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:55:08 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/1599970015' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T21:55:10.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:55:09 smithi189 ceph-mon[57022]: pgmap v595: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:55:10.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:55:09 smithi062 ceph-mon[76322]: pgmap v595: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:55:12.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:55:11 smithi189 ceph-mon[57022]: pgmap v596: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:55:12.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:55:11 smithi062 ceph-mon[76322]: pgmap v596: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:55:14.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:55:13 smithi189 ceph-mon[57022]: pgmap v597: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:55:14.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:55:13 smithi062 ceph-mon[76322]: pgmap v597: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:55:16.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:55:15 smithi189 ceph-mon[57022]: pgmap v598: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:55:16.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:55:15 smithi062 ceph-mon[76322]: pgmap v598: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:55:18.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:55:17 smithi189 ceph-mon[57022]: pgmap v599: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:55:18.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:55:17 smithi062 ceph-mon[76322]: pgmap v599: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:55:20.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:55:19 smithi189 ceph-mon[57022]: pgmap v600: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:55:20.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:55:19 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:55:20.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:55:19 smithi062 ceph-mon[76322]: pgmap v600: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:55:20.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:55:19 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:55:22.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:55:21 smithi189 ceph-mon[57022]: pgmap v601: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:55:22.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:55:21 smithi062 ceph-mon[76322]: pgmap v601: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:55:24.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:55:23 smithi189 ceph-mon[57022]: pgmap v602: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:55:24.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:55:23 smithi062 ceph-mon[76322]: pgmap v602: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:55:26.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:55:25 smithi189 ceph-mon[57022]: pgmap v603: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:55:26.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:55:25 smithi062 ceph-mon[76322]: pgmap v603: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:55:28.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:55:27 smithi189 ceph-mon[57022]: pgmap v604: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:55:28.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:55:27 smithi062 ceph-mon[76322]: pgmap v604: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:55:30.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:55:29 smithi189 ceph-mon[57022]: pgmap v605: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:55:30.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:55:29 smithi062 ceph-mon[76322]: pgmap v605: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:55:32.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:55:31 smithi189 ceph-mon[57022]: pgmap v606: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:55:32.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:55:31 smithi062 ceph-mon[76322]: pgmap v606: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:55:34.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:55:33 smithi189 ceph-mon[57022]: pgmap v607: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:55:34.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:55:33 smithi062 ceph-mon[76322]: pgmap v607: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:55:35.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:55:34 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:55:35.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:55:34 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:55:36.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:55:35 smithi062 ceph-mon[76322]: pgmap v608: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:55:36.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:55:35 smithi189 ceph-mon[57022]: pgmap v608: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:55:38.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:55:38 smithi062 ceph-mon[76322]: pgmap v609: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:55:38.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:55:38 smithi189 ceph-mon[57022]: pgmap v609: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:55:38.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:38.775+0000 7f07abc00700 1 -- 172.21.15.62:0/2867605616 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f07a4108470 msgr2=0x7f07a4108840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:55:38.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:38.775+0000 7f07abc00700 1 --2- 172.21.15.62:0/2867605616 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f07a4108470 0x7f07a4108840 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f0798009910 tx=0x7f0798009c20 comp rx=0 tx=0).stop 2024-04-20T21:55:38.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:38.775+0000 7f07abc00700 1 -- 172.21.15.62:0/2867605616 shutdown_connections 2024-04-20T21:55:38.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:38.775+0000 7f07abc00700 1 --2- 172.21.15.62:0/2867605616 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07a4102470 0x7f07a41028e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:55:38.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:38.775+0000 7f07abc00700 1 --2- 172.21.15.62:0/2867605616 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f07a4108470 0x7f07a4108840 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:55:38.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:38.775+0000 7f07abc00700 1 -- 172.21.15.62:0/2867605616 >> 172.21.15.62:0/2867605616 conn(0x7f07a40fdf90 msgr2=0x7f07a41003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:55:38.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:38.775+0000 7f07abc00700 1 -- 172.21.15.62:0/2867605616 shutdown_connections 2024-04-20T21:55:38.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:38.775+0000 7f07abc00700 1 -- 172.21.15.62:0/2867605616 wait complete. 2024-04-20T21:55:38.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:38.776+0000 7f07abc00700 1 Processor -- start 2024-04-20T21:55:38.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:38.777+0000 7f07abc00700 1 -- start start 2024-04-20T21:55:38.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:38.777+0000 7f07abc00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f07a4102470 0x7f07a41981d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:55:38.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:38.777+0000 7f07abc00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07a4108470 0x7f07a4198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:55:38.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:38.777+0000 7f07abc00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07a4198df0 con 0x7f07a4102470 2024-04-20T21:55:38.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:38.777+0000 7f07abc00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07a419cb80 con 0x7f07a4108470 2024-04-20T21:55:38.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:38.777+0000 7f07a9200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f07a4102470 0x7f07a41981d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:55:38.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:38.777+0000 7f07a9200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f07a4102470 0x7f07a41981d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:57426/0 (socket says 172.21.15.62:57426) 2024-04-20T21:55:38.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:38.777+0000 7f07a9200700 1 -- 172.21.15.62:0/298823993 learned_addr learned my addr 172.21.15.62:0/298823993 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:55:38.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:38.778+0000 7f07a3e00700 1 --2- 172.21.15.62:0/298823993 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07a4108470 0x7f07a4198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:55:38.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:38.778+0000 7f07a9200700 1 -- 172.21.15.62:0/298823993 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07a4108470 msgr2=0x7f07a4198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:55:38.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:38.778+0000 7f07a9200700 1 --2- 172.21.15.62:0/298823993 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07a4108470 0x7f07a4198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:55:38.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:38.778+0000 7f07a9200700 1 -- 172.21.15.62:0/298823993 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07980095f0 con 0x7f07a4102470 2024-04-20T21:55:38.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:38.778+0000 7f07a3e00700 1 --2- 172.21.15.62:0/298823993 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07a4108470 0x7f07a4198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T21:55:38.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:38.778+0000 7f07a9200700 1 --2- 172.21.15.62:0/298823993 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f07a4102470 0x7f07a41981d0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f079800b5c0 tx=0x7f079800be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:55:38.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:38.778+0000 7f07a1600700 1 -- 172.21.15.62:0/298823993 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f079801d070 con 0x7f07a4102470 2024-04-20T21:55:38.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:38.778+0000 7f07a1600700 1 -- 172.21.15.62:0/298823993 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f079800f460 con 0x7f07a4102470 2024-04-20T21:55:38.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:38.779+0000 7f07abc00700 1 -- 172.21.15.62:0/298823993 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07a419ce00 con 0x7f07a4102470 2024-04-20T21:55:38.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:38.779+0000 7f07a1600700 1 -- 172.21.15.62:0/298823993 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0798017610 con 0x7f07a4102470 2024-04-20T21:55:38.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:38.779+0000 7f07abc00700 1 -- 172.21.15.62:0/298823993 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07a419d2f0 con 0x7f07a4102470 2024-04-20T21:55:38.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:38.780+0000 7f07abc00700 1 -- 172.21.15.62:0/298823993 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07a410a9e0 con 0x7f07a4102470 2024-04-20T21:55:38.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:38.781+0000 7f07a1600700 1 -- 172.21.15.62:0/298823993 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f079800f5d0 con 0x7f07a4102470 2024-04-20T21:55:38.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:38.782+0000 7f07a1600700 1 --2- 172.21.15.62:0/298823993 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f078c06ecc0 0x7f078c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:55:38.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:38.782+0000 7f07a1600700 1 -- 172.21.15.62:0/298823993 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f0798090520 con 0x7f07a4102470 2024-04-20T21:55:38.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:38.782+0000 7f07a3e00700 1 --2- 172.21.15.62:0/298823993 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f078c06ecc0 0x7f078c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:55:38.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:38.783+0000 7f07a3e00700 1 --2- 172.21.15.62:0/298823993 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f078c06ecc0 0x7f078c071170 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f07a41997f0 tx=0x7f0790008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:55:38.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:38.787+0000 7f07a1600700 1 -- 172.21.15.62:0/298823993 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f0798059230 con 0x7f07a4102470 2024-04-20T21:55:39.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.049+0000 7f07abc00700 1 -- 172.21.15.62:0/298823993 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f07a402cca0 con 0x7f078c06ecc0 2024-04-20T21:55:39.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.052+0000 7f07a1600700 1 -- 172.21.15.62:0/298823993 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f07a402cca0 con 0x7f078c06ecc0 2024-04-20T21:55:39.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.055+0000 7f07abc00700 1 -- 172.21.15.62:0/298823993 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f078c06ecc0 msgr2=0x7f078c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:55:39.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.055+0000 7f07abc00700 1 --2- 172.21.15.62:0/298823993 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f078c06ecc0 0x7f078c071170 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f07a41997f0 tx=0x7f0790008040 comp rx=0 tx=0).stop 2024-04-20T21:55:39.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.055+0000 7f07abc00700 1 -- 172.21.15.62:0/298823993 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f07a4102470 msgr2=0x7f07a41981d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:55:39.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.055+0000 7f07abc00700 1 --2- 172.21.15.62:0/298823993 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f07a4102470 0x7f07a41981d0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f079800b5c0 tx=0x7f079800be00 comp rx=0 tx=0).stop 2024-04-20T21:55:39.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.055+0000 7f07abc00700 1 -- 172.21.15.62:0/298823993 shutdown_connections 2024-04-20T21:55:39.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.055+0000 7f07abc00700 1 --2- 172.21.15.62:0/298823993 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f078c06ecc0 0x7f078c071170 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:55:39.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.055+0000 7f07abc00700 1 --2- 172.21.15.62:0/298823993 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f07a4102470 0x7f07a41981d0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:55:39.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.055+0000 7f07abc00700 1 --2- 172.21.15.62:0/298823993 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07a4108470 0x7f07a4198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:55:39.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.055+0000 7f07abc00700 1 -- 172.21.15.62:0/298823993 >> 172.21.15.62:0/298823993 conn(0x7f07a40fdf90 msgr2=0x7f07a40feb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:55:39.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.055+0000 7f07abc00700 1 -- 172.21.15.62:0/298823993 shutdown_connections 2024-04-20T21:55:39.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.055+0000 7f07abc00700 1 -- 172.21.15.62:0/298823993 wait complete. 2024-04-20T21:55:39.070 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T21:55:39.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.221+0000 7f5e97c00700 1 -- 172.21.15.62:0/1201738917 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e9010c8f0 msgr2=0x7f5e9010ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:55:39.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.221+0000 7f5e97c00700 1 --2- 172.21.15.62:0/1201738917 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e9010c8f0 0x7f5e9010ccc0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f5e84009910 tx=0x7f5e84009c20 comp rx=0 tx=0).stop 2024-04-20T21:55:39.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.221+0000 7f5e97c00700 1 -- 172.21.15.62:0/1201738917 shutdown_connections 2024-04-20T21:55:39.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.221+0000 7f5e97c00700 1 --2- 172.21.15.62:0/1201738917 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e90106b80 0x7f5e90106ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:55:39.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.221+0000 7f5e97c00700 1 --2- 172.21.15.62:0/1201738917 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e9010c8f0 0x7f5e9010ccc0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:55:39.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.221+0000 7f5e97c00700 1 -- 172.21.15.62:0/1201738917 >> 172.21.15.62:0/1201738917 conn(0x7f5e90076060 msgr2=0x7f5e90078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:55:39.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.222+0000 7f5e97c00700 1 -- 172.21.15.62:0/1201738917 shutdown_connections 2024-04-20T21:55:39.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.222+0000 7f5e97c00700 1 -- 172.21.15.62:0/1201738917 wait complete. 2024-04-20T21:55:39.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.223+0000 7f5e97c00700 1 Processor -- start 2024-04-20T21:55:39.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.223+0000 7f5e97c00700 1 -- start start 2024-04-20T21:55:39.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.223+0000 7f5e97c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e90106b80 0x7f5e9019c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:55:39.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.223+0000 7f5e97c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e9010c8f0 0x7f5e9019ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:55:39.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.223+0000 7f5e97c00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e9019d120 con 0x7f5e9010c8f0 2024-04-20T21:55:39.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.223+0000 7f5e97c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e901a0eb0 con 0x7f5e90106b80 2024-04-20T21:55:39.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.224+0000 7f5e8fe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e9010c8f0 0x7f5e9019ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:55:39.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.224+0000 7f5e8fe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e9010c8f0 0x7f5e9019ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:57434/0 (socket says 172.21.15.62:57434) 2024-04-20T21:55:39.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.224+0000 7f5e8fe00700 1 -- 172.21.15.62:0/2662912930 learned_addr learned my addr 172.21.15.62:0/2662912930 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:55:39.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.224+0000 7f5e95200700 1 --2- 172.21.15.62:0/2662912930 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e90106b80 0x7f5e9019c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:55:39.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.224+0000 7f5e8fe00700 1 -- 172.21.15.62:0/2662912930 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e90106b80 msgr2=0x7f5e9019c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:55:39.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.224+0000 7f5e8fe00700 1 --2- 172.21.15.62:0/2662912930 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e90106b80 0x7f5e9019c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:55:39.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.224+0000 7f5e8fe00700 1 -- 172.21.15.62:0/2662912930 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e840095f0 con 0x7f5e9010c8f0 2024-04-20T21:55:39.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.224+0000 7f5e95200700 1 --2- 172.21.15.62:0/2662912930 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e90106b80 0x7f5e9019c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T21:55:39.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.224+0000 7f5e8fe00700 1 --2- 172.21.15.62:0/2662912930 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e9010c8f0 0x7f5e9019ca40 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f5e7c00d900 tx=0x7f5e7c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:55:39.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.225+0000 7f5e8d600700 1 -- 172.21.15.62:0/2662912930 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5e7c0041f0 con 0x7f5e9010c8f0 2024-04-20T21:55:39.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.225+0000 7f5e8d600700 1 -- 172.21.15.62:0/2662912930 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5e7c004cf0 con 0x7f5e9010c8f0 2024-04-20T21:55:39.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.225+0000 7f5e8d600700 1 -- 172.21.15.62:0/2662912930 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5e7c00b750 con 0x7f5e9010c8f0 2024-04-20T21:55:39.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.225+0000 7f5e97c00700 1 -- 172.21.15.62:0/2662912930 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e901a1190 con 0x7f5e9010c8f0 2024-04-20T21:55:39.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.225+0000 7f5e97c00700 1 -- 172.21.15.62:0/2662912930 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e901a16e0 con 0x7f5e9010c8f0 2024-04-20T21:55:39.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.226+0000 7f5e97c00700 1 -- 172.21.15.62:0/2662912930 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e90109b20 con 0x7f5e9010c8f0 2024-04-20T21:55:39.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.227+0000 7f5e8d600700 1 -- 172.21.15.62:0/2662912930 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f5e7c004350 con 0x7f5e9010c8f0 2024-04-20T21:55:39.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.228+0000 7f5e8d600700 1 --2- 172.21.15.62:0/2662912930 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5e7806ecc0 0x7f5e78071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:55:39.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.228+0000 7f5e8d600700 1 -- 172.21.15.62:0/2662912930 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f5e7c08efb0 con 0x7f5e9010c8f0 2024-04-20T21:55:39.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.229+0000 7f5e95200700 1 --2- 172.21.15.62:0/2662912930 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5e7806ecc0 0x7f5e78071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:55:39.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.229+0000 7f5e95200700 1 --2- 172.21.15.62:0/2662912930 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5e7806ecc0 0x7f5e78071170 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f5e8400b5c0 tx=0x7f5e84005960 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:55:39.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.233+0000 7f5e8d600700 1 -- 172.21.15.62:0/2662912930 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f5e7c0c3ff0 con 0x7f5e9010c8f0 2024-04-20T21:55:39.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.495+0000 7f5e97c00700 1 -- 172.21.15.62:0/2662912930 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5e901a1950 con 0x7f5e7806ecc0 2024-04-20T21:55:39.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.498+0000 7f5e8d600700 1 -- 172.21.15.62:0/2662912930 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f5e901a1950 con 0x7f5e7806ecc0 2024-04-20T21:55:39.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.502+0000 7f5e97c00700 1 -- 172.21.15.62:0/2662912930 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5e7806ecc0 msgr2=0x7f5e78071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:55:39.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.502+0000 7f5e97c00700 1 --2- 172.21.15.62:0/2662912930 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5e7806ecc0 0x7f5e78071170 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f5e8400b5c0 tx=0x7f5e84005960 comp rx=0 tx=0).stop 2024-04-20T21:55:39.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.502+0000 7f5e97c00700 1 -- 172.21.15.62:0/2662912930 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e9010c8f0 msgr2=0x7f5e9019ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:55:39.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.502+0000 7f5e97c00700 1 --2- 172.21.15.62:0/2662912930 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e9010c8f0 0x7f5e9019ca40 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f5e7c00d900 tx=0x7f5e7c00dcc0 comp rx=0 tx=0).stop 2024-04-20T21:55:39.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.502+0000 7f5e97c00700 1 -- 172.21.15.62:0/2662912930 shutdown_connections 2024-04-20T21:55:39.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.502+0000 7f5e97c00700 1 --2- 172.21.15.62:0/2662912930 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5e7806ecc0 0x7f5e78071170 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:55:39.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.502+0000 7f5e97c00700 1 --2- 172.21.15.62:0/2662912930 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e90106b80 0x7f5e9019c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:55:39.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.502+0000 7f5e97c00700 1 --2- 172.21.15.62:0/2662912930 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e9010c8f0 0x7f5e9019ca40 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:55:39.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.502+0000 7f5e97c00700 1 -- 172.21.15.62:0/2662912930 >> 172.21.15.62:0/2662912930 conn(0x7f5e90076060 msgr2=0x7f5e90077aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:55:39.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.503+0000 7f5e97c00700 1 -- 172.21.15.62:0/2662912930 shutdown_connections 2024-04-20T21:55:39.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.503+0000 7f5e97c00700 1 -- 172.21.15.62:0/2662912930 wait complete. 2024-04-20T21:55:39.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.663+0000 7f3e58000700 1 -- 172.21.15.62:0/302826057 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e50100e80 msgr2=0x7f3e50104e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:55:39.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.663+0000 7f3e58000700 1 --2- 172.21.15.62:0/302826057 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e50100e80 0x7f3e50104e50 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f3e3c009910 tx=0x7f3e3c009c20 comp rx=0 tx=0).stop 2024-04-20T21:55:39.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.663+0000 7f3e58000700 1 -- 172.21.15.62:0/302826057 shutdown_connections 2024-04-20T21:55:39.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.663+0000 7f3e58000700 1 --2- 172.21.15.62:0/302826057 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e50100e80 0x7f3e50104e50 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:55:39.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.663+0000 7f3e58000700 1 --2- 172.21.15.62:0/302826057 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e501004e0 0x7f3e501008b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:55:39.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.663+0000 7f3e58000700 1 -- 172.21.15.62:0/302826057 >> 172.21.15.62:0/302826057 conn(0x7f3e500fbda0 msgr2=0x7f3e500fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:55:39.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.663+0000 7f3e58000700 1 -- 172.21.15.62:0/302826057 shutdown_connections 2024-04-20T21:55:39.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.663+0000 7f3e58000700 1 -- 172.21.15.62:0/302826057 wait complete. 2024-04-20T21:55:39.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.664+0000 7f3e58000700 1 Processor -- start 2024-04-20T21:55:39.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.664+0000 7f3e58000700 1 -- start start 2024-04-20T21:55:39.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.665+0000 7f3e58000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e501004e0 0x7f3e50198110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:55:39.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.665+0000 7f3e58000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e50100e80 0x7f3e50198650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:55:39.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.665+0000 7f3e58000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e50198d30 con 0x7f3e50100e80 2024-04-20T21:55:39.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.665+0000 7f3e58000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e5019cac0 con 0x7f3e501004e0 2024-04-20T21:55:39.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.665+0000 7f3e54c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e50100e80 0x7f3e50198650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:55:39.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.665+0000 7f3e54c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e50100e80 0x7f3e50198650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:57450/0 (socket says 172.21.15.62:57450) 2024-04-20T21:55:39.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.665+0000 7f3e54c00700 1 -- 172.21.15.62:0/1268110338 learned_addr learned my addr 172.21.15.62:0/1268110338 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:55:39.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.665+0000 7f3e55600700 1 --2- 172.21.15.62:0/1268110338 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e501004e0 0x7f3e50198110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:55:39.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.666+0000 7f3e54c00700 1 -- 172.21.15.62:0/1268110338 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e501004e0 msgr2=0x7f3e50198110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:55:39.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.666+0000 7f3e54c00700 1 --2- 172.21.15.62:0/1268110338 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e501004e0 0x7f3e50198110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:55:39.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.666+0000 7f3e54c00700 1 -- 172.21.15.62:0/1268110338 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e3c0095f0 con 0x7f3e50100e80 2024-04-20T21:55:39.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.666+0000 7f3e54c00700 1 --2- 172.21.15.62:0/1268110338 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e50100e80 0x7f3e50198650 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f3e3c00bda0 tx=0x7f3e3c00be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:55:39.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.666+0000 7f3e4e000700 1 -- 172.21.15.62:0/1268110338 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3e3c01d070 con 0x7f3e50100e80 2024-04-20T21:55:39.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.666+0000 7f3e4e000700 1 -- 172.21.15.62:0/1268110338 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3e3c00f460 con 0x7f3e50100e80 2024-04-20T21:55:39.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.666+0000 7f3e58000700 1 -- 172.21.15.62:0/1268110338 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e5019cd40 con 0x7f3e50100e80 2024-04-20T21:55:39.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.666+0000 7f3e4e000700 1 -- 172.21.15.62:0/1268110338 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3e3c017630 con 0x7f3e50100e80 2024-04-20T21:55:39.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.666+0000 7f3e58000700 1 -- 172.21.15.62:0/1268110338 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e5019d230 con 0x7f3e50100e80 2024-04-20T21:55:39.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.667+0000 7f3e58000700 1 -- 172.21.15.62:0/1268110338 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e501045d0 con 0x7f3e50100e80 2024-04-20T21:55:39.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.668+0000 7f3e4e000700 1 -- 172.21.15.62:0/1268110338 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f3e3c00fa40 con 0x7f3e50100e80 2024-04-20T21:55:39.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.670+0000 7f3e4e000700 1 --2- 172.21.15.62:0/1268110338 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3e3806ecc0 0x7f3e38071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:55:39.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.670+0000 7f3e4e000700 1 -- 172.21.15.62:0/1268110338 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f3e3c090270 con 0x7f3e50100e80 2024-04-20T21:55:39.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.670+0000 7f3e55600700 1 --2- 172.21.15.62:0/1268110338 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3e3806ecc0 0x7f3e38071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:55:39.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.671+0000 7f3e55600700 1 --2- 172.21.15.62:0/1268110338 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3e3806ecc0 0x7f3e38071170 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f3e44005950 tx=0x7f3e440058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:55:39.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.675+0000 7f3e4e000700 1 -- 172.21.15.62:0/1268110338 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f3e3c059000 con 0x7f3e50100e80 2024-04-20T21:55:39.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.920+0000 7f3e58000700 1 -- 172.21.15.62:0/1268110338 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f3e5002cca0 con 0x7f3e3806ecc0 2024-04-20T21:55:39.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.952+0000 7f3e4e000700 1 -- 172.21.15.62:0/1268110338 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f3e5002cca0 con 0x7f3e3806ecc0 2024-04-20T21:55:39.952 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T21:55:39.952 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (21m) 12m ago 35m 14.2M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T21:55:39.952 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (35m) 12m ago 35m 9512k - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T21:55:39.952 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (34m) 13m ago 34m 8942k - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T21:55:39.952 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (17m) 12m ago 35m 6660k - 19.0.0-3133-ga8c84f59 76bc6b91b726 2825d5ba11c6 2024-04-20T21:55:39.952 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (17m) 13m ago 34m 6665k - 19.0.0-3133-ga8c84f59 76bc6b91b726 20f746fa1bba 2024-04-20T21:55:39.952 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (20m) 12m ago 34m 74.2M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T21:55:39.953 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (13m) 12m ago 28m 11.2M - 19.0.0-3133-ga8c84f59 76bc6b91b726 de80bc90a683 2024-04-20T21:55:39.953 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (13m) 12m ago 28m 11.9M - 19.0.0-3133-ga8c84f59 76bc6b91b726 cd793b7ee98e 2024-04-20T21:55:39.953 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (28m) 13m ago 28m 117M - 18.2.0 dc2bc1663786 fb2a8a223db8 2024-04-20T21:55:39.953 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (28m) 13m ago 28m 55.6M - 18.2.0 dc2bc1663786 f2eec2ee8951 2024-04-20T21:55:39.953 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (24m) 12m ago 37m 511M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T21:55:39.953 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (23m) 13m ago 34m 452M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T21:55:39.953 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (19m) 12m ago 37m 55.9M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 019eb1479485 2024-04-20T21:55:39.953 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (18m) 13m ago 33m 39.8M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 55aac560aab6 2024-04-20T21:55:39.953 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (22m) 12m ago 35m 7952k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T21:55:39.953 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (22m) 13m ago 34m 8304k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T21:55:39.953 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (16m) 12m ago 32m 88.0M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 ee2bb901f26e 2024-04-20T21:55:39.954 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (16m) 12m ago 32m 92.3M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 9075152d4162 2024-04-20T21:55:39.954 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (15m) 12m ago 31m 74.4M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 2d4f3e38dc0e 2024-04-20T21:55:39.954 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (15m) 13m ago 31m 87.6M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 8f97b17ae0eb 2024-04-20T21:55:39.954 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (14m) 13m ago 30m 90.7M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 653d8079967d 2024-04-20T21:55:39.954 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (13m) 13m ago 30m 11.5M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 596dded14c12 2024-04-20T21:55:39.954 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (22m) 12m ago 34m 74.0M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T21:55:39.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.955+0000 7f3e58000700 1 -- 172.21.15.62:0/1268110338 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3e3806ecc0 msgr2=0x7f3e38071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:55:39.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.955+0000 7f3e58000700 1 --2- 172.21.15.62:0/1268110338 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3e3806ecc0 0x7f3e38071170 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f3e44005950 tx=0x7f3e440058e0 comp rx=0 tx=0).stop 2024-04-20T21:55:39.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.955+0000 7f3e58000700 1 -- 172.21.15.62:0/1268110338 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e50100e80 msgr2=0x7f3e50198650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:55:39.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.955+0000 7f3e58000700 1 --2- 172.21.15.62:0/1268110338 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e50100e80 0x7f3e50198650 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f3e3c00bda0 tx=0x7f3e3c00be80 comp rx=0 tx=0).stop 2024-04-20T21:55:39.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.955+0000 7f3e58000700 1 -- 172.21.15.62:0/1268110338 shutdown_connections 2024-04-20T21:55:39.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.955+0000 7f3e58000700 1 --2- 172.21.15.62:0/1268110338 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3e3806ecc0 0x7f3e38071170 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:55:39.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.955+0000 7f3e58000700 1 --2- 172.21.15.62:0/1268110338 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e501004e0 0x7f3e50198110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:55:39.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.955+0000 7f3e58000700 1 --2- 172.21.15.62:0/1268110338 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e50100e80 0x7f3e50198650 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:55:39.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.955+0000 7f3e58000700 1 -- 172.21.15.62:0/1268110338 >> 172.21.15.62:0/1268110338 conn(0x7f3e500fbda0 msgr2=0x7f3e500fe150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:55:39.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.956+0000 7f3e58000700 1 -- 172.21.15.62:0/1268110338 shutdown_connections 2024-04-20T21:55:39.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:39.956+0000 7f3e58000700 1 -- 172.21.15.62:0/1268110338 wait complete. 2024-04-20T21:55:40.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:40.116+0000 7f03cfa00700 1 -- 172.21.15.62:0/76029513 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f03c8100e80 msgr2=0x7f03c8104e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:55:40.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:40.116+0000 7f03cfa00700 1 --2- 172.21.15.62:0/76029513 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f03c8100e80 0x7f03c8104e50 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f03b4009910 tx=0x7f03b4009c20 comp rx=0 tx=0).stop 2024-04-20T21:55:40.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:40.116+0000 7f03cfa00700 1 -- 172.21.15.62:0/76029513 shutdown_connections 2024-04-20T21:55:40.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:40.116+0000 7f03cfa00700 1 --2- 172.21.15.62:0/76029513 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f03c8100e80 0x7f03c8104e50 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:55:40.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:40.116+0000 7f03cfa00700 1 --2- 172.21.15.62:0/76029513 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03c81004e0 0x7f03c81008b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:55:40.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:40.116+0000 7f03cfa00700 1 -- 172.21.15.62:0/76029513 >> 172.21.15.62:0/76029513 conn(0x7f03c80fbda0 msgr2=0x7f03c80fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:55:40.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:40.116+0000 7f03cfa00700 1 -- 172.21.15.62:0/76029513 shutdown_connections 2024-04-20T21:55:40.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:40.116+0000 7f03cfa00700 1 -- 172.21.15.62:0/76029513 wait complete. 2024-04-20T21:55:40.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:40.117+0000 7f03cfa00700 1 Processor -- start 2024-04-20T21:55:40.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:40.118+0000 7f03cfa00700 1 -- start start 2024-04-20T21:55:40.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:40.118+0000 7f03cfa00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03c81004e0 0x7f03c8198130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:55:40.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:40.118+0000 7f03cfa00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f03c8100e80 0x7f03c8198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:55:40.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:40.118+0000 7f03cfa00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03c8198d50 con 0x7f03c8100e80 2024-04-20T21:55:40.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:40.118+0000 7f03cfa00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03c819cae0 con 0x7f03c81004e0 2024-04-20T21:55:40.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:40.118+0000 7f03c7e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f03c8100e80 0x7f03c8198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:55:40.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:40.118+0000 7f03c7e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f03c8100e80 0x7f03c8198670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:57466/0 (socket says 172.21.15.62:57466) 2024-04-20T21:55:40.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:40.118+0000 7f03c7e00700 1 -- 172.21.15.62:0/333903263 learned_addr learned my addr 172.21.15.62:0/333903263 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:55:40.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:40.119+0000 7f03c7e00700 1 -- 172.21.15.62:0/333903263 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03c81004e0 msgr2=0x7f03c8198130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:55:40.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:40.119+0000 7f03c7e00700 1 --2- 172.21.15.62:0/333903263 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03c81004e0 0x7f03c8198130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:55:40.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:40.119+0000 7f03c7e00700 1 -- 172.21.15.62:0/333903263 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03b40095f0 con 0x7f03c8100e80 2024-04-20T21:55:40.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:40.119+0000 7f03c7e00700 1 --2- 172.21.15.62:0/333903263 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f03c8100e80 0x7f03c8198670 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f03b400bda0 tx=0x7f03b400be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:55:40.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:40.119+0000 7f03c5600700 1 -- 172.21.15.62:0/333903263 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f03b401d070 con 0x7f03c8100e80 2024-04-20T21:55:40.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:40.119+0000 7f03c5600700 1 -- 172.21.15.62:0/333903263 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f03b4004fb0 con 0x7f03c8100e80 2024-04-20T21:55:40.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:40.119+0000 7f03c5600700 1 -- 172.21.15.62:0/333903263 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f03b400f460 con 0x7f03c8100e80 2024-04-20T21:55:40.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:40.119+0000 7f03cfa00700 1 -- 172.21.15.62:0/333903263 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03c819cd60 con 0x7f03c8100e80 2024-04-20T21:55:40.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:40.120+0000 7f03cfa00700 1 -- 172.21.15.62:0/333903263 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03c819d250 con 0x7f03c8100e80 2024-04-20T21:55:40.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:40.121+0000 7f03cfa00700 1 -- 172.21.15.62:0/333903263 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03c81045d0 con 0x7f03c8100e80 2024-04-20T21:55:40.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:40.122+0000 7f03c5600700 1 -- 172.21.15.62:0/333903263 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f03b4022800 con 0x7f03c8100e80 2024-04-20T21:55:40.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:40.123+0000 7f03c5600700 1 --2- 172.21.15.62:0/333903263 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f03b006ecc0 0x7f03b0071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:55:40.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:40.123+0000 7f03c5600700 1 -- 172.21.15.62:0/333903263 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f03b4090390 con 0x7f03c8100e80 2024-04-20T21:55:40.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:40.123+0000 7f03cd000700 1 --2- 172.21.15.62:0/333903263 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f03b006ecc0 0x7f03b0071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:55:40.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:40.124+0000 7f03cd000700 1 --2- 172.21.15.62:0/333903263 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f03b006ecc0 0x7f03b0071170 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f03bc00a990 tx=0x7f03bc005c80 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:55:40.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:40.128+0000 7f03c5600700 1 -- 172.21.15.62:0/333903263 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f03b40592f0 con 0x7f03c8100e80 2024-04-20T21:55:40.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:55:40 smithi062 ceph-mon[76322]: pgmap v610: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:55:40.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:55:40 smithi062 ceph-mon[76322]: from='client.35124 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:55:40.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:40.457+0000 7f03cfa00700 1 -- 172.21.15.62:0/333903263 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f03c804f2a0 con 0x7f03c8100e80 2024-04-20T21:55:40.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:40.458+0000 7f03c5600700 1 -- 172.21.15.62:0/333903263 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7f03b405c9b0 con 0x7f03c8100e80 2024-04-20T21:55:40.458 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:55:40.458 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T21:55:40.458 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:55:40.458 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:55:40.458 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T21:55:40.458 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:55:40.458 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:55:40.458 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T21:55:40.458 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 6 2024-04-20T21:55:40.458 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:55:40.458 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T21:55:40.459 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 2, 2024-04-20T21:55:40.459 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:55:40.459 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:55:40.459 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T21:55:40.459 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 2, 2024-04-20T21:55:40.459 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 12 2024-04-20T21:55:40.459 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T21:55:40.459 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:55:40.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:40.461+0000 7f03cfa00700 1 -- 172.21.15.62:0/333903263 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f03b006ecc0 msgr2=0x7f03b0071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:55:40.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:40.461+0000 7f03cfa00700 1 --2- 172.21.15.62:0/333903263 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f03b006ecc0 0x7f03b0071170 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f03bc00a990 tx=0x7f03bc005c80 comp rx=0 tx=0).stop 2024-04-20T21:55:40.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:40.461+0000 7f03cfa00700 1 -- 172.21.15.62:0/333903263 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f03c8100e80 msgr2=0x7f03c8198670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:55:40.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:40.461+0000 7f03cfa00700 1 --2- 172.21.15.62:0/333903263 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f03c8100e80 0x7f03c8198670 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f03b400bda0 tx=0x7f03b400be80 comp rx=0 tx=0).stop 2024-04-20T21:55:40.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:40.461+0000 7f03cfa00700 1 -- 172.21.15.62:0/333903263 shutdown_connections 2024-04-20T21:55:40.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:40.461+0000 7f03cfa00700 1 --2- 172.21.15.62:0/333903263 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f03b006ecc0 0x7f03b0071170 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:55:40.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:40.461+0000 7f03cfa00700 1 --2- 172.21.15.62:0/333903263 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03c81004e0 0x7f03c8198130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:55:40.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:40.461+0000 7f03cfa00700 1 --2- 172.21.15.62:0/333903263 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f03c8100e80 0x7f03c8198670 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:55:40.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:40.461+0000 7f03cfa00700 1 -- 172.21.15.62:0/333903263 >> 172.21.15.62:0/333903263 conn(0x7f03c80fbda0 msgr2=0x7f03c80fe160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:55:40.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:40.461+0000 7f03cfa00700 1 -- 172.21.15.62:0/333903263 shutdown_connections 2024-04-20T21:55:40.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:40.461+0000 7f03cfa00700 1 -- 172.21.15.62:0/333903263 wait complete. 2024-04-20T21:55:40.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:55:40 smithi189 ceph-mon[57022]: pgmap v610: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:55:40.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:55:40 smithi189 ceph-mon[57022]: from='client.35124 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:55:40.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:40.628+0000 7feee9800700 1 -- 172.21.15.62:0/2633587511 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feee410c8f0 msgr2=0x7feee410ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:55:40.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:40.628+0000 7feee9800700 1 --2- 172.21.15.62:0/2633587511 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feee410c8f0 0x7feee410ccc0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7feed4009910 tx=0x7feed4009c20 comp rx=0 tx=0).stop 2024-04-20T21:55:40.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:40.629+0000 7feee9800700 1 -- 172.21.15.62:0/2633587511 shutdown_connections 2024-04-20T21:55:40.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:40.629+0000 7feee9800700 1 --2- 172.21.15.62:0/2633587511 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feee4106b80 0x7feee4106ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:55:40.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:40.629+0000 7feee9800700 1 --2- 172.21.15.62:0/2633587511 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feee410c8f0 0x7feee410ccc0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:55:40.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:40.629+0000 7feee9800700 1 -- 172.21.15.62:0/2633587511 >> 172.21.15.62:0/2633587511 conn(0x7feee4076060 msgr2=0x7feee4078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:55:40.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:40.629+0000 7feee9800700 1 -- 172.21.15.62:0/2633587511 shutdown_connections 2024-04-20T21:55:40.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:40.629+0000 7feee9800700 1 -- 172.21.15.62:0/2633587511 wait complete. 2024-04-20T21:55:40.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:40.630+0000 7feee9800700 1 Processor -- start 2024-04-20T21:55:40.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:40.630+0000 7feee9800700 1 -- start start 2024-04-20T21:55:40.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:40.630+0000 7feee9800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feee4106b80 0x7feee419c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:55:40.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:40.630+0000 7feee9800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feee410c8f0 0x7feee419ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:55:40.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:40.630+0000 7feee9800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feee419d090 con 0x7feee410c8f0 2024-04-20T21:55:40.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:40.630+0000 7feee9800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feee419d1d0 con 0x7feee4106b80 2024-04-20T21:55:40.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:40.631+0000 7feee2000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feee410c8f0 0x7feee419ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:55:40.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:40.631+0000 7feee2000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feee410c8f0 0x7feee419ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:57492/0 (socket says 172.21.15.62:57492) 2024-04-20T21:55:40.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:40.631+0000 7feee2000700 1 -- 172.21.15.62:0/3721864127 learned_addr learned my addr 172.21.15.62:0/3721864127 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:55:40.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:40.631+0000 7feee2a00700 1 --2- 172.21.15.62:0/3721864127 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feee4106b80 0x7feee419c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:55:40.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:40.631+0000 7feee2000700 1 -- 172.21.15.62:0/3721864127 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feee4106b80 msgr2=0x7feee419c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:55:40.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:40.631+0000 7feee2000700 1 --2- 172.21.15.62:0/3721864127 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feee4106b80 0x7feee419c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:55:40.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:40.631+0000 7feee2000700 1 -- 172.21.15.62:0/3721864127 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feed40095f0 con 0x7feee410c8f0 2024-04-20T21:55:40.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:40.631+0000 7feee2000700 1 --2- 172.21.15.62:0/3721864127 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feee410c8f0 0x7feee419ca40 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7feed000c930 tx=0x7feed000ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:55:40.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:40.632+0000 7feedb400700 1 -- 172.21.15.62:0/3721864127 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feed0007ad0 con 0x7feee410c8f0 2024-04-20T21:55:40.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:40.632+0000 7feedb400700 1 -- 172.21.15.62:0/3721864127 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7feed0007c30 con 0x7feee410c8f0 2024-04-20T21:55:40.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:40.632+0000 7feee9800700 1 -- 172.21.15.62:0/3721864127 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feee41a1020 con 0x7feee410c8f0 2024-04-20T21:55:40.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:40.632+0000 7feedb400700 1 -- 172.21.15.62:0/3721864127 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feed0018610 con 0x7feee410c8f0 2024-04-20T21:55:40.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:40.632+0000 7feee9800700 1 -- 172.21.15.62:0/3721864127 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feee41a1570 con 0x7feee410c8f0 2024-04-20T21:55:40.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:40.633+0000 7feee9800700 1 -- 172.21.15.62:0/3721864127 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feee4109b20 con 0x7feee410c8f0 2024-04-20T21:55:40.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:40.634+0000 7feedb400700 1 -- 172.21.15.62:0/3721864127 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7feed000f450 con 0x7feee410c8f0 2024-04-20T21:55:40.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:40.635+0000 7feedb400700 1 --2- 172.21.15.62:0/3721864127 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7feec806ecc0 0x7feec8071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:55:40.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:40.635+0000 7feedb400700 1 -- 172.21.15.62:0/3721864127 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7feed008eff0 con 0x7feee410c8f0 2024-04-20T21:55:40.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:40.635+0000 7feee2a00700 1 --2- 172.21.15.62:0/3721864127 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7feec806ecc0 0x7feec8071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:55:40.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:40.636+0000 7feee2a00700 1 --2- 172.21.15.62:0/3721864127 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7feec806ecc0 0x7feec8071170 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7feed4006010 tx=0x7feed401a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:55:40.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:40.640+0000 7feedb400700 1 -- 172.21.15.62:0/3721864127 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7feed0057de0 con 0x7feee410c8f0 2024-04-20T21:55:40.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:40.925+0000 7feee9800700 1 -- 172.21.15.62:0/3721864127 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7feee402cc30 con 0x7feee410c8f0 2024-04-20T21:55:40.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:40.926+0000 7feedb400700 1 -- 172.21.15.62:0/3721864127 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 25 v25) v1 ==== 76+0+1887 (secure 0 0 0) 0x7feed005b4a0 con 0x7feee410c8f0 2024-04-20T21:55:40.926 INFO:teuthology.orchestra.run.smithi062.stdout:e25 2024-04-20T21:55:40.926 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T21:55:40.926 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T21:55:40.926 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T21:55:40.927 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:55:40.927 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T21:55:40.927 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T21:55:40.927 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 20 2024-04-20T21:55:40.927 INFO:teuthology.orchestra.run.smithi062.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-20T21:55:40.927 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T21:55:40.927 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T21:42:01.559061+0000 2024-04-20T21:55:40.927 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T21:55:40.927 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T21:55:40.927 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T21:55:40.927 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T21:55:40.927 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T21:55:40.927 INFO:teuthology.orchestra.run.smithi062.stdout:max_xattr_size 65536 2024-04-20T21:55:40.927 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T21:55:40.928 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T21:55:40.928 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 81 2024-04-20T21:55:40.928 INFO:teuthology.orchestra.run.smithi062.stdout:snap_rstat 1 2024-04-20T21:55:40.928 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T21:55:40.928 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 2 2024-04-20T21:55:40.928 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1 2024-04-20T21:55:40.928 INFO:teuthology.orchestra.run.smithi062.stdout:up {} 2024-04-20T21:55:40.928 INFO:teuthology.orchestra.run.smithi062.stdout:failed 0,1 2024-04-20T21:55:40.928 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T21:55:40.928 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T21:55:40.928 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T21:55:40.928 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T21:55:40.928 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data enabled 2024-04-20T21:55:40.928 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T21:55:40.928 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T21:55:40.929 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 1 2024-04-20T21:55:40.929 INFO:teuthology.orchestra.run.smithi062.stdout:qdb_cluster leader: 0 members: 2024-04-20T21:55:40.929 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:55:40.929 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:55:40.929 INFO:teuthology.orchestra.run.smithi062.stdout:Standby daemons: 2024-04-20T21:55:40.929 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:55:40.929 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ryqblb{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6826/4113916205,v1:172.21.15.62:6827/4113916205] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T21:55:40.929 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ozizwi{-1:34476} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6828/3801975165,v1:172.21.15.62:6829/3801975165] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T21:55:40.929 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.xbgspw{-1:44285} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6826/3996040526,v1:172.21.15.189:6827/3996040526] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:55:40.929 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.dvimer{-1:44289} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6824/1743111843,v1:172.21.15.189:6825/1743111843] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:55:40.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:40.929+0000 7feee9800700 1 -- 172.21.15.62:0/3721864127 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7feec806ecc0 msgr2=0x7feec8071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:55:40.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:40.929+0000 7feee9800700 1 --2- 172.21.15.62:0/3721864127 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7feec806ecc0 0x7feec8071170 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7feed4006010 tx=0x7feed401a040 comp rx=0 tx=0).stop 2024-04-20T21:55:40.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:40.929+0000 7feee9800700 1 -- 172.21.15.62:0/3721864127 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feee410c8f0 msgr2=0x7feee419ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:55:40.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:40.929+0000 7feee9800700 1 --2- 172.21.15.62:0/3721864127 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feee410c8f0 0x7feee419ca40 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7feed000c930 tx=0x7feed000ccf0 comp rx=0 tx=0).stop 2024-04-20T21:55:40.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:40.929+0000 7feee9800700 1 -- 172.21.15.62:0/3721864127 shutdown_connections 2024-04-20T21:55:40.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:40.929+0000 7feee9800700 1 --2- 172.21.15.62:0/3721864127 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7feec806ecc0 0x7feec8071170 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:55:40.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:40.929+0000 7feee9800700 1 --2- 172.21.15.62:0/3721864127 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feee4106b80 0x7feee419c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:55:40.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:40.929+0000 7feee9800700 1 --2- 172.21.15.62:0/3721864127 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feee410c8f0 0x7feee419ca40 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:55:40.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:40.929+0000 7feee9800700 1 -- 172.21.15.62:0/3721864127 >> 172.21.15.62:0/3721864127 conn(0x7feee4076060 msgr2=0x7feee4077aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:55:40.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:40.930+0000 7feee9800700 1 -- 172.21.15.62:0/3721864127 shutdown_connections 2024-04-20T21:55:40.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:40.930+0000 7feee9800700 1 -- 172.21.15.62:0/3721864127 wait complete. 2024-04-20T21:55:40.931 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 25 2024-04-20T21:55:41.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:41.095+0000 7f4326600700 1 -- 172.21.15.62:0/937244261 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4320106b80 msgr2=0x7f4320106ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:55:41.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:41.095+0000 7f4326600700 1 --2- 172.21.15.62:0/937244261 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4320106b80 0x7f4320106ff0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f4308009910 tx=0x7f4308009c20 comp rx=0 tx=0).stop 2024-04-20T21:55:41.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:41.095+0000 7f4326600700 1 -- 172.21.15.62:0/937244261 shutdown_connections 2024-04-20T21:55:41.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:41.095+0000 7f4326600700 1 --2- 172.21.15.62:0/937244261 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4320106b80 0x7f4320106ff0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:55:41.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:41.095+0000 7f4326600700 1 --2- 172.21.15.62:0/937244261 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f432010c860 0x7f432010cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:55:41.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:41.095+0000 7f4326600700 1 -- 172.21.15.62:0/937244261 >> 172.21.15.62:0/937244261 conn(0x7f4320075f50 msgr2=0x7f4320078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:55:41.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:41.095+0000 7f4326600700 1 -- 172.21.15.62:0/937244261 shutdown_connections 2024-04-20T21:55:41.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:41.095+0000 7f4326600700 1 -- 172.21.15.62:0/937244261 wait complete. 2024-04-20T21:55:41.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:41.096+0000 7f4326600700 1 Processor -- start 2024-04-20T21:55:41.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:41.096+0000 7f4326600700 1 -- start start 2024-04-20T21:55:41.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:41.096+0000 7f4326600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4320106b80 0x7f432019c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:55:41.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:41.096+0000 7f4326600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f432010c860 0x7f432019c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:55:41.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:41.096+0000 7f4326600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f432019cea0 con 0x7f432010c860 2024-04-20T21:55:41.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:41.096+0000 7f4326600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43201a0c30 con 0x7f4320106b80 2024-04-20T21:55:41.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:41.097+0000 7f4317e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f432010c860 0x7f432019c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:55:41.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:41.097+0000 7f4317e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f432010c860 0x7f432019c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:57508/0 (socket says 172.21.15.62:57508) 2024-04-20T21:55:41.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:41.097+0000 7f4317e00700 1 -- 172.21.15.62:0/1690941533 learned_addr learned my addr 172.21.15.62:0/1690941533 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:55:41.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:41.097+0000 7f431f400700 1 --2- 172.21.15.62:0/1690941533 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4320106b80 0x7f432019c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:55:41.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:41.097+0000 7f4317e00700 1 -- 172.21.15.62:0/1690941533 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4320106b80 msgr2=0x7f432019c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:55:41.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:41.097+0000 7f4317e00700 1 --2- 172.21.15.62:0/1690941533 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4320106b80 0x7f432019c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:55:41.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:41.097+0000 7f4317e00700 1 -- 172.21.15.62:0/1690941533 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43080095f0 con 0x7f432010c860 2024-04-20T21:55:41.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:41.098+0000 7f4317e00700 1 --2- 172.21.15.62:0/1690941533 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f432010c860 0x7f432019c7c0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f430800b5c0 tx=0x7f430800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:55:41.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:41.098+0000 7f431cc00700 1 -- 172.21.15.62:0/1690941533 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f430801d070 con 0x7f432010c860 2024-04-20T21:55:41.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:41.098+0000 7f431cc00700 1 -- 172.21.15.62:0/1690941533 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4308005e00 con 0x7f432010c860 2024-04-20T21:55:41.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:41.098+0000 7f4326600700 1 -- 172.21.15.62:0/1690941533 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43201a0eb0 con 0x7f432010c860 2024-04-20T21:55:41.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:41.098+0000 7f431cc00700 1 -- 172.21.15.62:0/1690941533 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4308017400 con 0x7f432010c860 2024-04-20T21:55:41.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:41.098+0000 7f4326600700 1 -- 172.21.15.62:0/1690941533 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43201a1420 con 0x7f432010c860 2024-04-20T21:55:41.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:41.099+0000 7f4326600700 1 -- 172.21.15.62:0/1690941533 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f432010ec80 con 0x7f432010c860 2024-04-20T21:55:41.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:41.100+0000 7f431cc00700 1 -- 172.21.15.62:0/1690941533 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f430800f460 con 0x7f432010c860 2024-04-20T21:55:41.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:41.101+0000 7f431cc00700 1 --2- 172.21.15.62:0/1690941533 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f430406ecc0 0x7f4304071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:55:41.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:41.101+0000 7f431cc00700 1 -- 172.21.15.62:0/1690941533 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f4308090370 con 0x7f432010c860 2024-04-20T21:55:41.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:41.101+0000 7f431f400700 1 --2- 172.21.15.62:0/1690941533 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f430406ecc0 0x7f4304071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:55:41.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:41.102+0000 7f431f400700 1 --2- 172.21.15.62:0/1690941533 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f430406ecc0 0x7f4304071170 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f4310005950 tx=0x7f43100058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:55:41.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:41.107+0000 7f431cc00700 1 -- 172.21.15.62:0/1690941533 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f4308059100 con 0x7f432010c860 2024-04-20T21:55:41.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:55:41 smithi062 ceph-mon[76322]: from='client.35128 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:55:41.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:55:41 smithi062 ceph-mon[76322]: from='client.35132 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:55:41.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:55:41 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/333903263' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:55:41.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:55:41 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/3721864127' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T21:55:41.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:41.361+0000 7f4326600700 1 -- 172.21.15.62:0/1690941533 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f43201a17c0 con 0x7f430406ecc0 2024-04-20T21:55:41.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:41.363+0000 7f431cc00700 1 -- 172.21.15.62:0/1690941533 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f43201a17c0 con 0x7f430406ecc0 2024-04-20T21:55:41.363 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:55:41.363 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T21:55:41.363 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T21:55:41.363 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-20T21:55:41.363 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T21:55:41.363 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr", 2024-04-20T21:55:41.363 INFO:teuthology.orchestra.run.smithi062.stdout: "crash", 2024-04-20T21:55:41.363 INFO:teuthology.orchestra.run.smithi062.stdout: "mon", 2024-04-20T21:55:41.363 INFO:teuthology.orchestra.run.smithi062.stdout: "osd" 2024-04-20T21:55:41.363 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T21:55:41.363 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "14/23 daemons upgraded", 2024-04-20T21:55:41.363 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading mds daemons", 2024-04-20T21:55:41.364 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T21:55:41.364 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:55:41.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:41.366+0000 7f4326600700 1 -- 172.21.15.62:0/1690941533 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f430406ecc0 msgr2=0x7f4304071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:55:41.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:41.366+0000 7f4326600700 1 --2- 172.21.15.62:0/1690941533 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f430406ecc0 0x7f4304071170 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f4310005950 tx=0x7f43100058e0 comp rx=0 tx=0).stop 2024-04-20T21:55:41.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:41.366+0000 7f4326600700 1 -- 172.21.15.62:0/1690941533 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f432010c860 msgr2=0x7f432019c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:55:41.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:41.366+0000 7f4326600700 1 --2- 172.21.15.62:0/1690941533 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f432010c860 0x7f432019c7c0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f430800b5c0 tx=0x7f430800be20 comp rx=0 tx=0).stop 2024-04-20T21:55:41.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:41.366+0000 7f4326600700 1 -- 172.21.15.62:0/1690941533 shutdown_connections 2024-04-20T21:55:41.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:41.366+0000 7f4326600700 1 --2- 172.21.15.62:0/1690941533 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f430406ecc0 0x7f4304071170 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:55:41.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:41.367+0000 7f4326600700 1 --2- 172.21.15.62:0/1690941533 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4320106b80 0x7f432019c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:55:41.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:41.367+0000 7f4326600700 1 --2- 172.21.15.62:0/1690941533 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f432010c860 0x7f432019c7c0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:55:41.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:41.367+0000 7f4326600700 1 -- 172.21.15.62:0/1690941533 >> 172.21.15.62:0/1690941533 conn(0x7f4320075f50 msgr2=0x7f43200768d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:55:41.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:41.367+0000 7f4326600700 1 -- 172.21.15.62:0/1690941533 shutdown_connections 2024-04-20T21:55:41.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:41.367+0000 7f4326600700 1 -- 172.21.15.62:0/1690941533 wait complete. 2024-04-20T21:55:41.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:55:41 smithi189 ceph-mon[57022]: from='client.35128 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:55:41.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:55:41 smithi189 ceph-mon[57022]: from='client.35132 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:55:41.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:55:41 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/333903263' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:55:41.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:55:41 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/3721864127' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T21:55:41.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:41.534+0000 7f5890c00700 1 -- 172.21.15.62:0/3586680667 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f588c106320 msgr2=0x7f588c1066f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:55:41.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:41.534+0000 7f5890c00700 1 --2- 172.21.15.62:0/3586680667 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f588c106320 0x7f588c1066f0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f587c009910 tx=0x7f587c009c20 comp rx=0 tx=0).stop 2024-04-20T21:55:41.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:41.534+0000 7f5890c00700 1 -- 172.21.15.62:0/3586680667 shutdown_connections 2024-04-20T21:55:41.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:41.534+0000 7f5890c00700 1 --2- 172.21.15.62:0/3586680667 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f588c068540 0x7f588c0689b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:55:41.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:41.534+0000 7f5890c00700 1 --2- 172.21.15.62:0/3586680667 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f588c106320 0x7f588c1066f0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:55:41.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:41.534+0000 7f5890c00700 1 -- 172.21.15.62:0/3586680667 >> 172.21.15.62:0/3586680667 conn(0x7f588c075130 msgr2=0x7f588c075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:55:41.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:41.534+0000 7f5890c00700 1 -- 172.21.15.62:0/3586680667 shutdown_connections 2024-04-20T21:55:41.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:41.534+0000 7f5890c00700 1 -- 172.21.15.62:0/3586680667 wait complete. 2024-04-20T21:55:41.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:41.535+0000 7f5890c00700 1 Processor -- start 2024-04-20T21:55:41.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:41.536+0000 7f5890c00700 1 -- start start 2024-04-20T21:55:41.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:41.536+0000 7f5890c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f588c068540 0x7f588c198170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:55:41.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:41.536+0000 7f5890c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f588c106320 0x7f588c1986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:55:41.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:41.536+0000 7f5890c00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f588c198d90 con 0x7f588c068540 2024-04-20T21:55:41.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:41.536+0000 7f5890c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f588c19cb20 con 0x7f588c106320 2024-04-20T21:55:41.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:41.536+0000 7f5889e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f588c068540 0x7f588c198170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:55:41.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:41.536+0000 7f5889e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f588c068540 0x7f588c198170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:57100/0 (socket says 172.21.15.62:57100) 2024-04-20T21:55:41.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:41.536+0000 7f5889e00700 1 -- 172.21.15.62:0/1476552066 learned_addr learned my addr 172.21.15.62:0/1476552066 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:55:41.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:41.537+0000 7f5889400700 1 --2- 172.21.15.62:0/1476552066 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f588c106320 0x7f588c1986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:55:41.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:41.537+0000 7f5889e00700 1 -- 172.21.15.62:0/1476552066 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f588c106320 msgr2=0x7f588c1986b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:55:41.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:41.537+0000 7f5889e00700 1 --2- 172.21.15.62:0/1476552066 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f588c106320 0x7f588c1986b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:55:41.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:41.537+0000 7f5889e00700 1 -- 172.21.15.62:0/1476552066 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f587c0095f0 con 0x7f588c068540 2024-04-20T21:55:41.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:41.537+0000 7f5889400700 1 --2- 172.21.15.62:0/1476552066 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f588c106320 0x7f588c1986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T21:55:41.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:41.537+0000 7f5889e00700 1 --2- 172.21.15.62:0/1476552066 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f588c068540 0x7f588c198170 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f587c006010 tx=0x7f587c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:55:41.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:41.537+0000 7f5882a00700 1 -- 172.21.15.62:0/1476552066 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f587c01d070 con 0x7f588c068540 2024-04-20T21:55:41.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:41.538+0000 7f5882a00700 1 -- 172.21.15.62:0/1476552066 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f587c004fb0 con 0x7f588c068540 2024-04-20T21:55:41.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:41.538+0000 7f5890c00700 1 -- 172.21.15.62:0/1476552066 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f588c19cda0 con 0x7f588c068540 2024-04-20T21:55:41.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:41.538+0000 7f5882a00700 1 -- 172.21.15.62:0/1476552066 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f587c00f690 con 0x7f588c068540 2024-04-20T21:55:41.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:41.538+0000 7f5890c00700 1 -- 172.21.15.62:0/1476552066 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f588c19d290 con 0x7f588c068540 2024-04-20T21:55:41.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:41.539+0000 7f5890c00700 1 -- 172.21.15.62:0/1476552066 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f588c04f2a0 con 0x7f588c068540 2024-04-20T21:55:41.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:41.540+0000 7f5882a00700 1 -- 172.21.15.62:0/1476552066 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f587c0228a0 con 0x7f588c068540 2024-04-20T21:55:41.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:41.541+0000 7f5882a00700 1 --2- 172.21.15.62:0/1476552066 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f587006ecc0 0x7f5870071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:55:41.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:41.541+0000 7f5882a00700 1 -- 172.21.15.62:0/1476552066 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f587c090700 con 0x7f588c068540 2024-04-20T21:55:41.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:41.541+0000 7f5889400700 1 --2- 172.21.15.62:0/1476552066 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f587006ecc0 0x7f5870071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:55:41.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:41.543+0000 7f5889400700 1 --2- 172.21.15.62:0/1476552066 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f587006ecc0 0x7f5870071170 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f588c199790 tx=0x7f5878008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:55:41.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:41.546+0000 7f5882a00700 1 -- 172.21.15.62:0/1476552066 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f587c059490 con 0x7f588c068540 2024-04-20T21:55:41.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:41.878+0000 7f5890c00700 1 -- 172.21.15.62:0/1476552066 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f588c1994d0 con 0x7f588c068540 2024-04-20T21:55:41.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:41.879+0000 7f5882a00700 1 -- 172.21.15.62:0/1476552066 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7f587c05cb50 con 0x7f588c068540 2024-04-20T21:55:41.877 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-20T21:55:41.877 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-20T21:55:41.877 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is degraded 2024-04-20T21:55:41.877 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T21:55:41.877 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T21:55:41.877 INFO:teuthology.orchestra.run.smithi062.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-20T21:55:41.878 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-20T21:55:41.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:41.882+0000 7f5890c00700 1 -- 172.21.15.62:0/1476552066 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f587006ecc0 msgr2=0x7f5870071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:55:41.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:41.882+0000 7f5890c00700 1 --2- 172.21.15.62:0/1476552066 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f587006ecc0 0x7f5870071170 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f588c199790 tx=0x7f5878008040 comp rx=0 tx=0).stop 2024-04-20T21:55:41.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:41.882+0000 7f5890c00700 1 -- 172.21.15.62:0/1476552066 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f588c068540 msgr2=0x7f588c198170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:55:41.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:41.882+0000 7f5890c00700 1 --2- 172.21.15.62:0/1476552066 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f588c068540 0x7f588c198170 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f587c006010 tx=0x7f587c00be20 comp rx=0 tx=0).stop 2024-04-20T21:55:41.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:41.882+0000 7f5890c00700 1 -- 172.21.15.62:0/1476552066 shutdown_connections 2024-04-20T21:55:41.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:41.882+0000 7f5890c00700 1 --2- 172.21.15.62:0/1476552066 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f587006ecc0 0x7f5870071170 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:55:41.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:41.882+0000 7f5890c00700 1 --2- 172.21.15.62:0/1476552066 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f588c068540 0x7f588c198170 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:55:41.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:41.882+0000 7f5890c00700 1 --2- 172.21.15.62:0/1476552066 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f588c106320 0x7f588c1986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:55:41.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:41.882+0000 7f5890c00700 1 -- 172.21.15.62:0/1476552066 >> 172.21.15.62:0/1476552066 conn(0x7f588c075130 msgr2=0x7f588c0fe9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:55:41.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:41.882+0000 7f5890c00700 1 -- 172.21.15.62:0/1476552066 shutdown_connections 2024-04-20T21:55:41.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:55:41.883+0000 7f5890c00700 1 -- 172.21.15.62:0/1476552066 wait complete. 2024-04-20T21:55:42.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:55:42 smithi062 ceph-mon[76322]: pgmap v611: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:55:42.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:55:42 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/1476552066' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T21:55:42.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:55:42 smithi189 ceph-mon[57022]: pgmap v611: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:55:42.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:55:42 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/1476552066' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T21:55:43.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:55:43 smithi062 ceph-mon[76322]: from='client.35144 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:55:43.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:55:43 smithi189 ceph-mon[57022]: from='client.35144 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:55:44.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:55:44 smithi062 ceph-mon[76322]: pgmap v612: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:55:44.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:55:44 smithi189 ceph-mon[57022]: pgmap v612: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:55:46.411 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:55:46 smithi189 ceph-mon[57022]: pgmap v613: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:55:46.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:55:46 smithi062 ceph-mon[76322]: pgmap v613: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:55:48.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:55:48 smithi189 ceph-mon[57022]: pgmap v614: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:55:48.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:55:48 smithi062 ceph-mon[76322]: pgmap v614: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:55:50.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:55:50 smithi189 ceph-mon[57022]: pgmap v615: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:55:50.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:55:50 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:55:50.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:55:50 smithi062 ceph-mon[76322]: pgmap v615: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:55:50.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:55:50 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:55:52.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:55:52 smithi189 ceph-mon[57022]: pgmap v616: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:55:52.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:55:52 smithi062 ceph-mon[76322]: pgmap v616: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:55:54.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:55:54 smithi189 ceph-mon[57022]: pgmap v617: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:55:54.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:55:54 smithi062 ceph-mon[76322]: pgmap v617: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:55:56.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:55:56 smithi189 ceph-mon[57022]: pgmap v618: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:55:56.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:55:56 smithi062 ceph-mon[76322]: pgmap v618: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:55:58.338 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:55:58 smithi062 ceph-mon[76322]: pgmap v619: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:55:58.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:55:58 smithi189 ceph-mon[57022]: pgmap v619: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:56:00.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:56:00 smithi189 ceph-mon[57022]: pgmap v620: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:56:00.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:56:00 smithi062 ceph-mon[76322]: pgmap v620: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:56:02.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:56:02 smithi189 ceph-mon[57022]: pgmap v621: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:56:02.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:56:02 smithi062 ceph-mon[76322]: pgmap v621: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:56:04.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:56:04 smithi189 ceph-mon[57022]: pgmap v622: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:56:04.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:56:04 smithi062 ceph-mon[76322]: pgmap v622: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:56:05.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:56:05 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:56:05.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:56:05 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:56:06.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:56:06 smithi189 ceph-mon[57022]: pgmap v623: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:56:06.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:56:06 smithi062 ceph-mon[76322]: pgmap v623: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:56:07.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:56:07 smithi189 ceph-mon[57022]: pgmap v624: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:56:07.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:56:07 smithi062 ceph-mon[76322]: pgmap v624: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:56:10.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:56:09 smithi062 ceph-mon[76322]: pgmap v625: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:56:10.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:56:09 smithi189 ceph-mon[57022]: pgmap v625: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:56:12.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.061+0000 7f46ffe00700 1 -- 172.21.15.62:0/3148382544 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f47000730f0 msgr2=0x7f47000734c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:56:12.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.061+0000 7f46ffe00700 1 --2- 172.21.15.62:0/3148382544 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f47000730f0 0x7f47000734c0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f46f0009910 tx=0x7f46f0009c20 comp rx=0 tx=0).stop 2024-04-20T21:56:12.061 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:56:11 smithi062 ceph-mon[76322]: pgmap v626: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:56:12.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.061+0000 7f46ffe00700 1 -- 172.21.15.62:0/3148382544 shutdown_connections 2024-04-20T21:56:12.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.061+0000 7f46ffe00700 1 --2- 172.21.15.62:0/3148382544 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4700073a00 0x7f470010c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:56:12.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.061+0000 7f46ffe00700 1 --2- 172.21.15.62:0/3148382544 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f47000730f0 0x7f47000734c0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:56:12.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.061+0000 7f46ffe00700 1 -- 172.21.15.62:0/3148382544 >> 172.21.15.62:0/3148382544 conn(0x7f47000fbb60 msgr2=0x7f47000fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:56:12.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.061+0000 7f46ffe00700 1 -- 172.21.15.62:0/3148382544 shutdown_connections 2024-04-20T21:56:12.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.061+0000 7f46ffe00700 1 -- 172.21.15.62:0/3148382544 wait complete. 2024-04-20T21:56:12.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.063+0000 7f46ffe00700 1 Processor -- start 2024-04-20T21:56:12.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.063+0000 7f46ffe00700 1 -- start start 2024-04-20T21:56:12.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.063+0000 7f46ffe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f47000730f0 0x7f4700197e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:56:12.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.063+0000 7f46ffe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4700073a00 0x7f47001984c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:56:12.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.063+0000 7f46ffe00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f470019c800 con 0x7f47000730f0 2024-04-20T21:56:12.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.063+0000 7f46ffe00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f470019c970 con 0x7f4700073a00 2024-04-20T21:56:12.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.063+0000 7f46fea00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f47000730f0 0x7f4700197e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:56:12.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.064+0000 7f46fea00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f47000730f0 0x7f4700197e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54082/0 (socket says 172.21.15.62:54082) 2024-04-20T21:56:12.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.064+0000 7f46fea00700 1 -- 172.21.15.62:0/565005360 learned_addr learned my addr 172.21.15.62:0/565005360 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:56:12.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.064+0000 7f46fe000700 1 --2- 172.21.15.62:0/565005360 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4700073a00 0x7f47001984c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:56:12.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.064+0000 7f46fea00700 1 -- 172.21.15.62:0/565005360 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4700073a00 msgr2=0x7f47001984c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:56:12.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.064+0000 7f46fea00700 1 --2- 172.21.15.62:0/565005360 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4700073a00 0x7f47001984c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:56:12.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.064+0000 7f46fea00700 1 -- 172.21.15.62:0/565005360 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46f00095f0 con 0x7f47000730f0 2024-04-20T21:56:12.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.064+0000 7f46fea00700 1 --2- 172.21.15.62:0/565005360 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f47000730f0 0x7f4700197e50 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f46f000b5c0 tx=0x7f46f000be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:56:12.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.065+0000 7f46f7400700 1 -- 172.21.15.62:0/565005360 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f46f001d070 con 0x7f47000730f0 2024-04-20T21:56:12.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.065+0000 7f46f7400700 1 -- 172.21.15.62:0/565005360 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f46f000f460 con 0x7f47000730f0 2024-04-20T21:56:12.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.065+0000 7f46ffe00700 1 -- 172.21.15.62:0/565005360 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f470019cc50 con 0x7f47000730f0 2024-04-20T21:56:12.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.065+0000 7f46f7400700 1 -- 172.21.15.62:0/565005360 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f46f0017630 con 0x7f47000730f0 2024-04-20T21:56:12.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.065+0000 7f46ffe00700 1 -- 172.21.15.62:0/565005360 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f470019d1a0 con 0x7f47000730f0 2024-04-20T21:56:12.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.066+0000 7f46ffe00700 1 -- 172.21.15.62:0/565005360 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4700109a60 con 0x7f47000730f0 2024-04-20T21:56:12.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.067+0000 7f46f7400700 1 -- 172.21.15.62:0/565005360 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f46f000f5d0 con 0x7f47000730f0 2024-04-20T21:56:12.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.068+0000 7f46f7400700 1 --2- 172.21.15.62:0/565005360 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f46e406ecc0 0x7f46e4071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:56:12.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.068+0000 7f46f7400700 1 -- 172.21.15.62:0/565005360 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f46f0090490 con 0x7f47000730f0 2024-04-20T21:56:12.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.069+0000 7f46fe000700 1 --2- 172.21.15.62:0/565005360 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f46e406ecc0 0x7f46e4071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:56:12.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.069+0000 7f46fe000700 1 --2- 172.21.15.62:0/565005360 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f46e406ecc0 0x7f46e4071170 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f46ec006fd0 tx=0x7f46ec008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:56:12.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.074+0000 7f46f7400700 1 -- 172.21.15.62:0/565005360 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f46f00591a0 con 0x7f47000730f0 2024-04-20T21:56:12.204 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:56:11 smithi189 ceph-mon[57022]: pgmap v626: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:56:12.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.330+0000 7f46ffe00700 1 -- 172.21.15.62:0/565005360 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f47001991b0 con 0x7f46e406ecc0 2024-04-20T21:56:12.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.332+0000 7f46f7400700 1 -- 172.21.15.62:0/565005360 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f47001991b0 con 0x7f46e406ecc0 2024-04-20T21:56:12.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.335+0000 7f46ffe00700 1 -- 172.21.15.62:0/565005360 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f46e406ecc0 msgr2=0x7f46e4071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:56:12.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.335+0000 7f46ffe00700 1 --2- 172.21.15.62:0/565005360 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f46e406ecc0 0x7f46e4071170 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f46ec006fd0 tx=0x7f46ec008040 comp rx=0 tx=0).stop 2024-04-20T21:56:12.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.335+0000 7f46ffe00700 1 -- 172.21.15.62:0/565005360 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f47000730f0 msgr2=0x7f4700197e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:56:12.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.335+0000 7f46ffe00700 1 --2- 172.21.15.62:0/565005360 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f47000730f0 0x7f4700197e50 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f46f000b5c0 tx=0x7f46f000be00 comp rx=0 tx=0).stop 2024-04-20T21:56:12.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.335+0000 7f46ffe00700 1 -- 172.21.15.62:0/565005360 shutdown_connections 2024-04-20T21:56:12.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.335+0000 7f46ffe00700 1 --2- 172.21.15.62:0/565005360 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f46e406ecc0 0x7f46e4071170 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:56:12.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.335+0000 7f46ffe00700 1 --2- 172.21.15.62:0/565005360 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f47000730f0 0x7f4700197e50 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:56:12.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.335+0000 7f46ffe00700 1 --2- 172.21.15.62:0/565005360 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4700073a00 0x7f47001984c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:56:12.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.335+0000 7f46ffe00700 1 -- 172.21.15.62:0/565005360 >> 172.21.15.62:0/565005360 conn(0x7f47000fbb60 msgr2=0x7f4700106ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:56:12.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.336+0000 7f46ffe00700 1 -- 172.21.15.62:0/565005360 shutdown_connections 2024-04-20T21:56:12.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.336+0000 7f46ffe00700 1 -- 172.21.15.62:0/565005360 wait complete. 2024-04-20T21:56:12.351 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T21:56:12.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.495+0000 7fb41d000700 1 -- 172.21.15.62:0/1541262901 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb41810c8f0 msgr2=0x7fb41810ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:56:12.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.495+0000 7fb41d000700 1 --2- 172.21.15.62:0/1541262901 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb41810c8f0 0x7fb41810ccc0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fb408009910 tx=0x7fb408009c20 comp rx=0 tx=0).stop 2024-04-20T21:56:12.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.495+0000 7fb41d000700 1 -- 172.21.15.62:0/1541262901 shutdown_connections 2024-04-20T21:56:12.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.495+0000 7fb41d000700 1 --2- 172.21.15.62:0/1541262901 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb418106b80 0x7fb418106ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:56:12.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.495+0000 7fb41d000700 1 --2- 172.21.15.62:0/1541262901 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb41810c8f0 0x7fb41810ccc0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:56:12.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.495+0000 7fb41d000700 1 -- 172.21.15.62:0/1541262901 >> 172.21.15.62:0/1541262901 conn(0x7fb418076060 msgr2=0x7fb418078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:56:12.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.496+0000 7fb41d000700 1 -- 172.21.15.62:0/1541262901 shutdown_connections 2024-04-20T21:56:12.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.496+0000 7fb41d000700 1 -- 172.21.15.62:0/1541262901 wait complete. 2024-04-20T21:56:12.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.497+0000 7fb41d000700 1 Processor -- start 2024-04-20T21:56:12.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.497+0000 7fb41d000700 1 -- start start 2024-04-20T21:56:12.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.497+0000 7fb41d000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb418106b80 0x7fb41819c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:56:12.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.497+0000 7fb41d000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb41810c8f0 0x7fb41819ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:56:12.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.497+0000 7fb41d000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb41819d120 con 0x7fb41810c8f0 2024-04-20T21:56:12.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.497+0000 7fb41d000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4181a0eb0 con 0x7fb418106b80 2024-04-20T21:56:12.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.497+0000 7fb415400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb41810c8f0 0x7fb41819ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:56:12.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.497+0000 7fb415400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb41810c8f0 0x7fb41819ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54104/0 (socket says 172.21.15.62:54104) 2024-04-20T21:56:12.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.498+0000 7fb415400700 1 -- 172.21.15.62:0/2214476665 learned_addr learned my addr 172.21.15.62:0/2214476665 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:56:12.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.498+0000 7fb415e00700 1 --2- 172.21.15.62:0/2214476665 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb418106b80 0x7fb41819c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:56:12.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.498+0000 7fb415400700 1 -- 172.21.15.62:0/2214476665 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb418106b80 msgr2=0x7fb41819c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:56:12.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.498+0000 7fb415400700 1 --2- 172.21.15.62:0/2214476665 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb418106b80 0x7fb41819c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:56:12.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.498+0000 7fb415400700 1 -- 172.21.15.62:0/2214476665 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4080095f0 con 0x7fb41810c8f0 2024-04-20T21:56:12.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.498+0000 7fb415400700 1 --2- 172.21.15.62:0/2214476665 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb41810c8f0 0x7fb41819ca40 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fb40400c8a0 tx=0x7fb40400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:56:12.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.499+0000 7fb40ea00700 1 -- 172.21.15.62:0/2214476665 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb40400cea0 con 0x7fb41810c8f0 2024-04-20T21:56:12.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.499+0000 7fb40ea00700 1 -- 172.21.15.62:0/2214476665 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb404004830 con 0x7fb41810c8f0 2024-04-20T21:56:12.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.499+0000 7fb40ea00700 1 -- 172.21.15.62:0/2214476665 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb4040055b0 con 0x7fb41810c8f0 2024-04-20T21:56:12.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.499+0000 7fb41d000700 1 -- 172.21.15.62:0/2214476665 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4181a1190 con 0x7fb41810c8f0 2024-04-20T21:56:12.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.499+0000 7fb41d000700 1 -- 172.21.15.62:0/2214476665 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4181a16e0 con 0x7fb41810c8f0 2024-04-20T21:56:12.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.500+0000 7fb41d000700 1 -- 172.21.15.62:0/2214476665 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb418109b20 con 0x7fb41810c8f0 2024-04-20T21:56:12.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.501+0000 7fb40ea00700 1 -- 172.21.15.62:0/2214476665 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fb4040049a0 con 0x7fb41810c8f0 2024-04-20T21:56:12.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.502+0000 7fb40ea00700 1 --2- 172.21.15.62:0/2214476665 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb3fc06ecc0 0x7fb3fc071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:56:12.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.503+0000 7fb40ea00700 1 -- 172.21.15.62:0/2214476665 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fb40408ef40 con 0x7fb41810c8f0 2024-04-20T21:56:12.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.503+0000 7fb415e00700 1 --2- 172.21.15.62:0/2214476665 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb3fc06ecc0 0x7fb3fc071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:56:12.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.503+0000 7fb415e00700 1 --2- 172.21.15.62:0/2214476665 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb3fc06ecc0 0x7fb3fc071170 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fb408006010 tx=0x7fb40801a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:56:12.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.506+0000 7fb40ea00700 1 -- 172.21.15.62:0/2214476665 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fb404057d50 con 0x7fb41810c8f0 2024-04-20T21:56:12.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.762+0000 7fb41d000700 1 -- 172.21.15.62:0/2214476665 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb4181a1a20 con 0x7fb3fc06ecc0 2024-04-20T21:56:12.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.764+0000 7fb40ea00700 1 -- 172.21.15.62:0/2214476665 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fb4181a1a20 con 0x7fb3fc06ecc0 2024-04-20T21:56:12.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.766+0000 7fb41d000700 1 -- 172.21.15.62:0/2214476665 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb3fc06ecc0 msgr2=0x7fb3fc071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:56:12.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.766+0000 7fb41d000700 1 --2- 172.21.15.62:0/2214476665 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb3fc06ecc0 0x7fb3fc071170 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fb408006010 tx=0x7fb40801a040 comp rx=0 tx=0).stop 2024-04-20T21:56:12.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.766+0000 7fb41d000700 1 -- 172.21.15.62:0/2214476665 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb41810c8f0 msgr2=0x7fb41819ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:56:12.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.766+0000 7fb41d000700 1 --2- 172.21.15.62:0/2214476665 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb41810c8f0 0x7fb41819ca40 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fb40400c8a0 tx=0x7fb40400cc60 comp rx=0 tx=0).stop 2024-04-20T21:56:12.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.766+0000 7fb41d000700 1 -- 172.21.15.62:0/2214476665 shutdown_connections 2024-04-20T21:56:12.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.766+0000 7fb41d000700 1 --2- 172.21.15.62:0/2214476665 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb3fc06ecc0 0x7fb3fc071170 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:56:12.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.766+0000 7fb41d000700 1 --2- 172.21.15.62:0/2214476665 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb418106b80 0x7fb41819c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:56:12.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.766+0000 7fb41d000700 1 --2- 172.21.15.62:0/2214476665 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb41810c8f0 0x7fb41819ca40 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:56:12.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.766+0000 7fb41d000700 1 -- 172.21.15.62:0/2214476665 >> 172.21.15.62:0/2214476665 conn(0x7fb418076060 msgr2=0x7fb418077aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:56:12.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.767+0000 7fb41d000700 1 -- 172.21.15.62:0/2214476665 shutdown_connections 2024-04-20T21:56:12.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.767+0000 7fb41d000700 1 -- 172.21.15.62:0/2214476665 wait complete. 2024-04-20T21:56:12.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.918+0000 7f5698e00700 1 -- 172.21.15.62:0/860277485 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5694103040 msgr2=0x7f5694107010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:56:12.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.918+0000 7f5698e00700 1 --2- 172.21.15.62:0/860277485 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5694103040 0x7f5694107010 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f5680009910 tx=0x7f5680009c20 comp rx=0 tx=0).stop 2024-04-20T21:56:12.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.918+0000 7f5698e00700 1 -- 172.21.15.62:0/860277485 shutdown_connections 2024-04-20T21:56:12.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.918+0000 7f5698e00700 1 --2- 172.21.15.62:0/860277485 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5694103040 0x7f5694107010 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:56:12.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.918+0000 7f5698e00700 1 --2- 172.21.15.62:0/860277485 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56941026a0 0x7f5694102a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:56:12.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.918+0000 7f5698e00700 1 -- 172.21.15.62:0/860277485 >> 172.21.15.62:0/860277485 conn(0x7f56940fdf40 msgr2=0x7f5694100350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:56:12.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.918+0000 7f5698e00700 1 -- 172.21.15.62:0/860277485 shutdown_connections 2024-04-20T21:56:12.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.918+0000 7f5698e00700 1 -- 172.21.15.62:0/860277485 wait complete. 2024-04-20T21:56:12.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.919+0000 7f5698e00700 1 Processor -- start 2024-04-20T21:56:12.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.919+0000 7f5698e00700 1 -- start start 2024-04-20T21:56:12.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.920+0000 7f5698e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f56941026a0 0x7f5694198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:56:12.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.920+0000 7f5698e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5694103040 0x7f5694198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:56:12.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.920+0000 7f5698e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5694198be0 con 0x7f56941026a0 2024-04-20T21:56:12.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.920+0000 7f5698e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5694198d20 con 0x7f5694103040 2024-04-20T21:56:12.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.920+0000 7f5691e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f56941026a0 0x7f5694198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:56:12.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.920+0000 7f5691e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f56941026a0 0x7f5694198050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54120/0 (socket says 172.21.15.62:54120) 2024-04-20T21:56:12.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.920+0000 7f5691e00700 1 -- 172.21.15.62:0/971803480 learned_addr learned my addr 172.21.15.62:0/971803480 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:56:12.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.921+0000 7f5691e00700 1 -- 172.21.15.62:0/971803480 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5694103040 msgr2=0x7f5694198590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:56:12.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.921+0000 7f5691e00700 1 --2- 172.21.15.62:0/971803480 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5694103040 0x7f5694198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:56:12.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.921+0000 7f5691e00700 1 -- 172.21.15.62:0/971803480 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56800095f0 con 0x7f56941026a0 2024-04-20T21:56:12.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.921+0000 7f5691e00700 1 --2- 172.21.15.62:0/971803480 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f56941026a0 0x7f5694198050 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f568400c930 tx=0x7f568400ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:56:12.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.921+0000 7f568aa00700 1 -- 172.21.15.62:0/971803480 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5684007ad0 con 0x7f56941026a0 2024-04-20T21:56:12.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.921+0000 7f568aa00700 1 -- 172.21.15.62:0/971803480 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5684007c30 con 0x7f56941026a0 2024-04-20T21:56:12.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.921+0000 7f5698e00700 1 -- 172.21.15.62:0/971803480 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f569419cb70 con 0x7f56941026a0 2024-04-20T21:56:12.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.921+0000 7f568aa00700 1 -- 172.21.15.62:0/971803480 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5684018610 con 0x7f56941026a0 2024-04-20T21:56:12.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.921+0000 7f5698e00700 1 -- 172.21.15.62:0/971803480 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f569419d0c0 con 0x7f56941026a0 2024-04-20T21:56:12.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.923+0000 7f5698e00700 1 -- 172.21.15.62:0/971803480 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f569410aa10 con 0x7f56941026a0 2024-04-20T21:56:12.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.924+0000 7f568aa00700 1 -- 172.21.15.62:0/971803480 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f568400f450 con 0x7f56941026a0 2024-04-20T21:56:12.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.925+0000 7f568aa00700 1 --2- 172.21.15.62:0/971803480 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f567806ecc0 0x7f5678071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:56:12.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.925+0000 7f568aa00700 1 -- 172.21.15.62:0/971803480 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f568408eff0 con 0x7f56941026a0 2024-04-20T21:56:12.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.925+0000 7f5691400700 1 --2- 172.21.15.62:0/971803480 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f567806ecc0 0x7f5678071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:56:12.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.926+0000 7f5691400700 1 --2- 172.21.15.62:0/971803480 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f567806ecc0 0x7f5678071170 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f568000b5c0 tx=0x7f5680005fd0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:56:12.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:12.929+0000 7f568aa00700 1 -- 172.21.15.62:0/971803480 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f5684057de0 con 0x7f56941026a0 2024-04-20T21:56:13.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:13.169+0000 7f5698e00700 1 -- 172.21.15.62:0/971803480 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f569419d340 con 0x7f567806ecc0 2024-04-20T21:56:13.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:13.181+0000 7f568aa00700 1 -- 172.21.15.62:0/971803480 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f569419d340 con 0x7f567806ecc0 2024-04-20T21:56:13.181 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T21:56:13.181 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (22m) 13m ago 35m 14.2M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T21:56:13.181 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (36m) 13m ago 36m 9512k - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T21:56:13.181 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (34m) 14m ago 34m 8942k - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T21:56:13.181 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (18m) 13m ago 36m 6660k - 19.0.0-3133-ga8c84f59 76bc6b91b726 2825d5ba11c6 2024-04-20T21:56:13.181 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (18m) 14m ago 34m 6665k - 19.0.0-3133-ga8c84f59 76bc6b91b726 20f746fa1bba 2024-04-20T21:56:13.181 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (21m) 13m ago 35m 74.2M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T21:56:13.181 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (13m) 13m ago 28m 11.2M - 19.0.0-3133-ga8c84f59 76bc6b91b726 de80bc90a683 2024-04-20T21:56:13.181 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (14m) 13m ago 28m 11.9M - 19.0.0-3133-ga8c84f59 76bc6b91b726 cd793b7ee98e 2024-04-20T21:56:13.181 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (28m) 14m ago 28m 117M - 18.2.0 dc2bc1663786 fb2a8a223db8 2024-04-20T21:56:13.181 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (28m) 14m ago 28m 55.6M - 18.2.0 dc2bc1663786 f2eec2ee8951 2024-04-20T21:56:13.181 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (24m) 13m ago 37m 511M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T21:56:13.182 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (23m) 14m ago 34m 452M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T21:56:13.182 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (19m) 13m ago 37m 55.9M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 019eb1479485 2024-04-20T21:56:13.182 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (19m) 14m ago 34m 39.8M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 55aac560aab6 2024-04-20T21:56:13.182 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (23m) 13m ago 36m 7952k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T21:56:13.182 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (23m) 14m ago 34m 8304k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T21:56:13.182 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (17m) 13m ago 33m 88.0M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 ee2bb901f26e 2024-04-20T21:56:13.182 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (16m) 13m ago 32m 92.3M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 9075152d4162 2024-04-20T21:56:13.182 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (16m) 13m ago 32m 74.4M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 2d4f3e38dc0e 2024-04-20T21:56:13.182 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (15m) 14m ago 31m 87.6M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 8f97b17ae0eb 2024-04-20T21:56:13.182 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (15m) 14m ago 31m 90.7M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 653d8079967d 2024-04-20T21:56:13.182 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (14m) 14m ago 30m 11.5M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 596dded14c12 2024-04-20T21:56:13.182 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (22m) 13m ago 35m 74.0M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T21:56:13.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:13.184+0000 7f5698e00700 1 -- 172.21.15.62:0/971803480 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f567806ecc0 msgr2=0x7f5678071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:56:13.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:13.184+0000 7f5698e00700 1 --2- 172.21.15.62:0/971803480 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f567806ecc0 0x7f5678071170 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f568000b5c0 tx=0x7f5680005fd0 comp rx=0 tx=0).stop 2024-04-20T21:56:13.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:13.184+0000 7f5698e00700 1 -- 172.21.15.62:0/971803480 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f56941026a0 msgr2=0x7f5694198050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:56:13.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:13.184+0000 7f5698e00700 1 --2- 172.21.15.62:0/971803480 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f56941026a0 0x7f5694198050 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f568400c930 tx=0x7f568400ccf0 comp rx=0 tx=0).stop 2024-04-20T21:56:13.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:13.184+0000 7f5698e00700 1 -- 172.21.15.62:0/971803480 shutdown_connections 2024-04-20T21:56:13.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:13.184+0000 7f5698e00700 1 --2- 172.21.15.62:0/971803480 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f567806ecc0 0x7f5678071170 secure :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f568000b5c0 tx=0x7f5680005fd0 comp rx=0 tx=0).stop 2024-04-20T21:56:13.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:13.184+0000 7f5698e00700 1 --2- 172.21.15.62:0/971803480 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f56941026a0 0x7f5694198050 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:56:13.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:13.184+0000 7f5698e00700 1 --2- 172.21.15.62:0/971803480 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5694103040 0x7f5694198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:56:13.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:13.184+0000 7f5698e00700 1 -- 172.21.15.62:0/971803480 >> 172.21.15.62:0/971803480 conn(0x7f56940fdf40 msgr2=0x7f5694100320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:56:13.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:13.184+0000 7f5698e00700 1 -- 172.21.15.62:0/971803480 shutdown_connections 2024-04-20T21:56:13.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:13.185+0000 7f5698e00700 1 -- 172.21.15.62:0/971803480 wait complete. 2024-04-20T21:56:13.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:13.342+0000 7f2d50a00700 1 -- 172.21.15.62:0/2875604354 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2d4c1004e0 msgr2=0x7f2d4c1008b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:56:13.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:13.342+0000 7f2d50a00700 1 --2- 172.21.15.62:0/2875604354 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2d4c1004e0 0x7f2d4c1008b0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f2d3c009910 tx=0x7f2d3c009c20 comp rx=0 tx=0).stop 2024-04-20T21:56:13.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:13.343+0000 7f2d50a00700 1 -- 172.21.15.62:0/2875604354 shutdown_connections 2024-04-20T21:56:13.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:13.343+0000 7f2d50a00700 1 --2- 172.21.15.62:0/2875604354 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2d4c100e80 0x7f2d4c104e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:56:13.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:13.343+0000 7f2d50a00700 1 --2- 172.21.15.62:0/2875604354 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2d4c1004e0 0x7f2d4c1008b0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:56:13.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:13.343+0000 7f2d50a00700 1 -- 172.21.15.62:0/2875604354 >> 172.21.15.62:0/2875604354 conn(0x7f2d4c0fbda0 msgr2=0x7f2d4c0fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:56:13.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:13.343+0000 7f2d50a00700 1 -- 172.21.15.62:0/2875604354 shutdown_connections 2024-04-20T21:56:13.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:13.343+0000 7f2d50a00700 1 -- 172.21.15.62:0/2875604354 wait complete. 2024-04-20T21:56:13.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:13.344+0000 7f2d50a00700 1 Processor -- start 2024-04-20T21:56:13.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:13.344+0000 7f2d50a00700 1 -- start start 2024-04-20T21:56:13.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:13.344+0000 7f2d50a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2d4c1004e0 0x7f2d4c1980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:56:13.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:13.344+0000 7f2d50a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2d4c100e80 0x7f2d4c1985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:56:13.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:13.344+0000 7f2d50a00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d4c198c30 con 0x7f2d4c100e80 2024-04-20T21:56:13.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:13.344+0000 7f2d50a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d4c198d70 con 0x7f2d4c1004e0 2024-04-20T21:56:13.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:13.345+0000 7f2d49400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2d4c100e80 0x7f2d4c1985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:56:13.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:13.345+0000 7f2d49400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2d4c100e80 0x7f2d4c1985e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54136/0 (socket says 172.21.15.62:54136) 2024-04-20T21:56:13.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:13.345+0000 7f2d49400700 1 -- 172.21.15.62:0/258499330 learned_addr learned my addr 172.21.15.62:0/258499330 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:56:13.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:13.345+0000 7f2d49e00700 1 --2- 172.21.15.62:0/258499330 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2d4c1004e0 0x7f2d4c1980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:56:13.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:13.345+0000 7f2d49400700 1 -- 172.21.15.62:0/258499330 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2d4c1004e0 msgr2=0x7f2d4c1980a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:56:13.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:13.345+0000 7f2d49400700 1 --2- 172.21.15.62:0/258499330 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2d4c1004e0 0x7f2d4c1980a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:56:13.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:13.345+0000 7f2d49400700 1 -- 172.21.15.62:0/258499330 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d3c0095f0 con 0x7f2d4c100e80 2024-04-20T21:56:13.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:13.345+0000 7f2d49e00700 1 --2- 172.21.15.62:0/258499330 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2d4c1004e0 0x7f2d4c1980a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T21:56:13.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:13.345+0000 7f2d49400700 1 --2- 172.21.15.62:0/258499330 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2d4c100e80 0x7f2d4c1985e0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f2d3800c960 tx=0x7f2d3800cd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:56:13.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:13.346+0000 7f2d42a00700 1 -- 172.21.15.62:0/258499330 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2d38007a70 con 0x7f2d4c100e80 2024-04-20T21:56:13.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:13.346+0000 7f2d42a00700 1 -- 172.21.15.62:0/258499330 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2d38007bd0 con 0x7f2d4c100e80 2024-04-20T21:56:13.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:13.346+0000 7f2d50a00700 1 -- 172.21.15.62:0/258499330 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d4c19cbc0 con 0x7f2d4c100e80 2024-04-20T21:56:13.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:13.346+0000 7f2d42a00700 1 -- 172.21.15.62:0/258499330 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2d380186c0 con 0x7f2d4c100e80 2024-04-20T21:56:13.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:13.346+0000 7f2d50a00700 1 -- 172.21.15.62:0/258499330 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d4c19d110 con 0x7f2d4c100e80 2024-04-20T21:56:13.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:13.347+0000 7f2d50a00700 1 -- 172.21.15.62:0/258499330 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d4c1045d0 con 0x7f2d4c100e80 2024-04-20T21:56:13.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:13.348+0000 7f2d42a00700 1 -- 172.21.15.62:0/258499330 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f2d3800faa0 con 0x7f2d4c100e80 2024-04-20T21:56:13.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:13.349+0000 7f2d42a00700 1 --2- 172.21.15.62:0/258499330 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2d3006ecc0 0x7f2d30071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:56:13.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:13.349+0000 7f2d42a00700 1 -- 172.21.15.62:0/258499330 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f2d3808eea0 con 0x7f2d4c100e80 2024-04-20T21:56:13.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:13.350+0000 7f2d49e00700 1 --2- 172.21.15.62:0/258499330 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2d3006ecc0 0x7f2d30071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:56:13.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:13.350+0000 7f2d49e00700 1 --2- 172.21.15.62:0/258499330 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2d3006ecc0 0x7f2d30071170 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f2d3c006010 tx=0x7f2d3c00b540 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:56:13.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:13.354+0000 7f2d42a00700 1 -- 172.21.15.62:0/258499330 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f2d3805b760 con 0x7f2d4c100e80 2024-04-20T21:56:13.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:13.690+0000 7f2d50a00700 1 -- 172.21.15.62:0/258499330 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f2d4c103fb0 con 0x7f2d4c100e80 2024-04-20T21:56:13.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:13.691+0000 7f2d42a00700 1 -- 172.21.15.62:0/258499330 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7f2d3805b2f0 con 0x7f2d4c100e80 2024-04-20T21:56:13.691 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:56:13.691 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T21:56:13.691 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:56:13.691 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:56:13.691 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T21:56:13.691 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:56:13.691 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:56:13.691 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T21:56:13.691 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 6 2024-04-20T21:56:13.692 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:56:13.692 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T21:56:13.692 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 2, 2024-04-20T21:56:13.692 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:56:13.692 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:56:13.692 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T21:56:13.692 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 2, 2024-04-20T21:56:13.692 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 12 2024-04-20T21:56:13.692 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T21:56:13.692 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:56:13.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:13.695+0000 7f2d50a00700 1 -- 172.21.15.62:0/258499330 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2d3006ecc0 msgr2=0x7f2d30071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:56:13.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:13.695+0000 7f2d50a00700 1 --2- 172.21.15.62:0/258499330 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2d3006ecc0 0x7f2d30071170 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f2d3c006010 tx=0x7f2d3c00b540 comp rx=0 tx=0).stop 2024-04-20T21:56:13.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:13.695+0000 7f2d50a00700 1 -- 172.21.15.62:0/258499330 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2d4c100e80 msgr2=0x7f2d4c1985e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:56:13.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:13.695+0000 7f2d50a00700 1 --2- 172.21.15.62:0/258499330 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2d4c100e80 0x7f2d4c1985e0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f2d3800c960 tx=0x7f2d3800cd20 comp rx=0 tx=0).stop 2024-04-20T21:56:13.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:13.695+0000 7f2d50a00700 1 -- 172.21.15.62:0/258499330 shutdown_connections 2024-04-20T21:56:13.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:13.695+0000 7f2d50a00700 1 --2- 172.21.15.62:0/258499330 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2d3006ecc0 0x7f2d30071170 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:56:13.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:13.695+0000 7f2d50a00700 1 --2- 172.21.15.62:0/258499330 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2d4c1004e0 0x7f2d4c1980a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:56:13.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:13.695+0000 7f2d50a00700 1 --2- 172.21.15.62:0/258499330 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2d4c100e80 0x7f2d4c1985e0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:56:13.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:13.695+0000 7f2d50a00700 1 -- 172.21.15.62:0/258499330 >> 172.21.15.62:0/258499330 conn(0x7f2d4c0fbda0 msgr2=0x7f2d4c0fd3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:56:13.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:13.695+0000 7f2d50a00700 1 -- 172.21.15.62:0/258499330 shutdown_connections 2024-04-20T21:56:13.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:13.696+0000 7f2d50a00700 1 -- 172.21.15.62:0/258499330 wait complete. 2024-04-20T21:56:13.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:13.861+0000 7f7b75600700 1 -- 172.21.15.62:0/3859283053 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b680054b0 con 0x7f7b781004e0 2024-04-20T21:56:13.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:13.861+0000 7f7b7dc00700 1 -- 172.21.15.62:0/3859283053 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7b781004e0 msgr2=0x7f7b781008b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:56:13.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:13.861+0000 7f7b7dc00700 1 --2- 172.21.15.62:0/3859283053 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7b781004e0 0x7f7b781008b0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f7b68009910 tx=0x7f7b68009c20 comp rx=0 tx=0).stop 2024-04-20T21:56:13.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:13.861+0000 7f7b7dc00700 1 -- 172.21.15.62:0/3859283053 shutdown_connections 2024-04-20T21:56:13.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:13.861+0000 7f7b7dc00700 1 --2- 172.21.15.62:0/3859283053 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7b78100e80 0x7f7b78104e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:56:13.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:13.861+0000 7f7b7dc00700 1 --2- 172.21.15.62:0/3859283053 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7b781004e0 0x7f7b781008b0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:56:13.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:13.861+0000 7f7b7dc00700 1 -- 172.21.15.62:0/3859283053 >> 172.21.15.62:0/3859283053 conn(0x7f7b780fbda0 msgr2=0x7f7b780fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:56:13.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:13.862+0000 7f7b7dc00700 1 -- 172.21.15.62:0/3859283053 shutdown_connections 2024-04-20T21:56:13.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:13.862+0000 7f7b7dc00700 1 -- 172.21.15.62:0/3859283053 wait complete. 2024-04-20T21:56:13.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:13.863+0000 7f7b7dc00700 1 Processor -- start 2024-04-20T21:56:13.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:13.863+0000 7f7b7dc00700 1 -- start start 2024-04-20T21:56:13.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:13.863+0000 7f7b7dc00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7b781004e0 0x7f7b78195ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:56:13.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:13.863+0000 7f7b7dc00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7b78100e80 0x7f7b78196430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:56:13.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:13.863+0000 7f7b7dc00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b78196a80 con 0x7f7b781004e0 2024-04-20T21:56:13.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:13.863+0000 7f7b7dc00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b78196bc0 con 0x7f7b78100e80 2024-04-20T21:56:13.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:13.864+0000 7f7b76a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7b781004e0 0x7f7b78195ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:56:13.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:13.864+0000 7f7b76a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7b781004e0 0x7f7b78195ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54148/0 (socket says 172.21.15.62:54148) 2024-04-20T21:56:13.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:13.864+0000 7f7b76a00700 1 -- 172.21.15.62:0/2416825308 learned_addr learned my addr 172.21.15.62:0/2416825308 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:56:13.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:13.864+0000 7f7b76000700 1 --2- 172.21.15.62:0/2416825308 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7b78100e80 0x7f7b78196430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:56:13.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:13.864+0000 7f7b76a00700 1 -- 172.21.15.62:0/2416825308 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7b78100e80 msgr2=0x7f7b78196430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:56:13.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:13.864+0000 7f7b76a00700 1 --2- 172.21.15.62:0/2416825308 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7b78100e80 0x7f7b78196430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:56:13.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:13.864+0000 7f7b76a00700 1 -- 172.21.15.62:0/2416825308 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b680095f0 con 0x7f7b781004e0 2024-04-20T21:56:13.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:13.864+0000 7f7b76a00700 1 --2- 172.21.15.62:0/2416825308 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7b781004e0 0x7f7b78195ef0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f7b68006010 tx=0x7f7b6800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:56:13.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:13.865+0000 7f7b6f400700 1 -- 172.21.15.62:0/2416825308 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b6801d070 con 0x7f7b781004e0 2024-04-20T21:56:13.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:13.865+0000 7f7b6f400700 1 -- 172.21.15.62:0/2416825308 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7b68004fb0 con 0x7f7b781004e0 2024-04-20T21:56:13.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:13.865+0000 7f7b7dc00700 1 -- 172.21.15.62:0/2416825308 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b7819a9b0 con 0x7f7b781004e0 2024-04-20T21:56:13.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:13.865+0000 7f7b6f400700 1 -- 172.21.15.62:0/2416825308 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b68022ea0 con 0x7f7b781004e0 2024-04-20T21:56:13.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:13.865+0000 7f7b7dc00700 1 -- 172.21.15.62:0/2416825308 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b7819aea0 con 0x7f7b781004e0 2024-04-20T21:56:13.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:13.866+0000 7f7b7dc00700 1 -- 172.21.15.62:0/2416825308 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b781087f0 con 0x7f7b781004e0 2024-04-20T21:56:13.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:13.867+0000 7f7b6f400700 1 -- 172.21.15.62:0/2416825308 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f7b68022470 con 0x7f7b781004e0 2024-04-20T21:56:13.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:13.868+0000 7f7b6f400700 1 --2- 172.21.15.62:0/2416825308 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7b5c06ecc0 0x7f7b5c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:56:13.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:13.868+0000 7f7b6f400700 1 -- 172.21.15.62:0/2416825308 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f7b680903f0 con 0x7f7b781004e0 2024-04-20T21:56:13.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:13.868+0000 7f7b76000700 1 --2- 172.21.15.62:0/2416825308 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7b5c06ecc0 0x7f7b5c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:56:13.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:13.869+0000 7f7b76000700 1 --2- 172.21.15.62:0/2416825308 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7b5c06ecc0 0x7f7b5c071170 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f7b78197510 tx=0x7f7b64005d50 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:56:13.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:13.873+0000 7f7b6f400700 1 -- 172.21.15.62:0/2416825308 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f7b68059230 con 0x7f7b781004e0 2024-04-20T21:56:14.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:56:13 smithi062 ceph-mon[76322]: from='client.35152 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:56:14.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:56:13 smithi062 ceph-mon[76322]: pgmap v627: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:56:14.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:56:13 smithi062 ceph-mon[76322]: from='client.35156 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:56:14.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:56:13 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/258499330' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:56:14.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:14.164+0000 7f7b7dc00700 1 -- 172.21.15.62:0/2416825308 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f7b781081d0 con 0x7f7b781004e0 2024-04-20T21:56:14.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:14.165+0000 7f7b6f400700 1 -- 172.21.15.62:0/2416825308 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 25 v25) v1 ==== 76+0+1887 (secure 0 0 0) 0x7f7b68022720 con 0x7f7b781004e0 2024-04-20T21:56:14.165 INFO:teuthology.orchestra.run.smithi062.stdout:e25 2024-04-20T21:56:14.165 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T21:56:14.165 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T21:56:14.165 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T21:56:14.165 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:56:14.165 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T21:56:14.166 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T21:56:14.166 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 20 2024-04-20T21:56:14.166 INFO:teuthology.orchestra.run.smithi062.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-20T21:56:14.166 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T21:56:14.166 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T21:42:01.559061+0000 2024-04-20T21:56:14.166 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T21:56:14.166 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T21:56:14.166 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T21:56:14.166 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T21:56:14.166 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T21:56:14.166 INFO:teuthology.orchestra.run.smithi062.stdout:max_xattr_size 65536 2024-04-20T21:56:14.166 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T21:56:14.166 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T21:56:14.166 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 81 2024-04-20T21:56:14.166 INFO:teuthology.orchestra.run.smithi062.stdout:snap_rstat 1 2024-04-20T21:56:14.167 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T21:56:14.167 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 2 2024-04-20T21:56:14.167 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1 2024-04-20T21:56:14.167 INFO:teuthology.orchestra.run.smithi062.stdout:up {} 2024-04-20T21:56:14.167 INFO:teuthology.orchestra.run.smithi062.stdout:failed 0,1 2024-04-20T21:56:14.167 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T21:56:14.167 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T21:56:14.167 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T21:56:14.167 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T21:56:14.167 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data enabled 2024-04-20T21:56:14.167 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T21:56:14.167 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T21:56:14.167 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 1 2024-04-20T21:56:14.167 INFO:teuthology.orchestra.run.smithi062.stdout:qdb_cluster leader: 0 members: 2024-04-20T21:56:14.167 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:56:14.168 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:56:14.168 INFO:teuthology.orchestra.run.smithi062.stdout:Standby daemons: 2024-04-20T21:56:14.168 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:56:14.168 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ryqblb{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6826/4113916205,v1:172.21.15.62:6827/4113916205] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T21:56:14.168 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ozizwi{-1:34476} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6828/3801975165,v1:172.21.15.62:6829/3801975165] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T21:56:14.168 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.xbgspw{-1:44285} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6826/3996040526,v1:172.21.15.189:6827/3996040526] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:56:14.168 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.dvimer{-1:44289} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6824/1743111843,v1:172.21.15.189:6825/1743111843] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:56:14.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:14.169+0000 7f7b7dc00700 1 -- 172.21.15.62:0/2416825308 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7b5c06ecc0 msgr2=0x7f7b5c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:56:14.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:14.169+0000 7f7b7dc00700 1 --2- 172.21.15.62:0/2416825308 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7b5c06ecc0 0x7f7b5c071170 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f7b78197510 tx=0x7f7b64005d50 comp rx=0 tx=0).stop 2024-04-20T21:56:14.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:14.169+0000 7f7b7dc00700 1 -- 172.21.15.62:0/2416825308 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7b781004e0 msgr2=0x7f7b78195ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:56:14.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:14.169+0000 7f7b7dc00700 1 --2- 172.21.15.62:0/2416825308 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7b781004e0 0x7f7b78195ef0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f7b68006010 tx=0x7f7b6800be20 comp rx=0 tx=0).stop 2024-04-20T21:56:14.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:14.170+0000 7f7b7dc00700 1 -- 172.21.15.62:0/2416825308 shutdown_connections 2024-04-20T21:56:14.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:14.170+0000 7f7b7dc00700 1 --2- 172.21.15.62:0/2416825308 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7b5c06ecc0 0x7f7b5c071170 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:56:14.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:14.170+0000 7f7b7dc00700 1 --2- 172.21.15.62:0/2416825308 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7b781004e0 0x7f7b78195ef0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:56:14.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:14.170+0000 7f7b7dc00700 1 --2- 172.21.15.62:0/2416825308 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7b78100e80 0x7f7b78196430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:56:14.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:14.170+0000 7f7b7dc00700 1 -- 172.21.15.62:0/2416825308 >> 172.21.15.62:0/2416825308 conn(0x7f7b780fbda0 msgr2=0x7f7b780fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:56:14.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:14.170+0000 7f7b7dc00700 1 -- 172.21.15.62:0/2416825308 shutdown_connections 2024-04-20T21:56:14.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:14.170+0000 7f7b7dc00700 1 -- 172.21.15.62:0/2416825308 wait complete. 2024-04-20T21:56:14.171 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 25 2024-04-20T21:56:14.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:56:13 smithi189 ceph-mon[57022]: from='client.35152 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:56:14.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:56:13 smithi189 ceph-mon[57022]: pgmap v627: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:56:14.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:56:13 smithi189 ceph-mon[57022]: from='client.35156 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:56:14.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:56:13 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/258499330' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:56:14.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:14.341+0000 7f0e49200700 1 -- 172.21.15.62:0/3939115680 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e4410c8f0 msgr2=0x7f0e4410ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:56:14.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:14.341+0000 7f0e49200700 1 --2- 172.21.15.62:0/3939115680 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e4410c8f0 0x7f0e4410ccc0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f0e34009910 tx=0x7f0e34009c20 comp rx=0 tx=0).stop 2024-04-20T21:56:14.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:14.341+0000 7f0e49200700 1 -- 172.21.15.62:0/3939115680 shutdown_connections 2024-04-20T21:56:14.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:14.341+0000 7f0e49200700 1 --2- 172.21.15.62:0/3939115680 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0e44106b80 0x7f0e44106ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:56:14.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:14.341+0000 7f0e49200700 1 --2- 172.21.15.62:0/3939115680 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e4410c8f0 0x7f0e4410ccc0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:56:14.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:14.341+0000 7f0e49200700 1 -- 172.21.15.62:0/3939115680 >> 172.21.15.62:0/3939115680 conn(0x7f0e44076060 msgr2=0x7f0e44078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:56:14.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:14.341+0000 7f0e49200700 1 -- 172.21.15.62:0/3939115680 shutdown_connections 2024-04-20T21:56:14.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:14.341+0000 7f0e49200700 1 -- 172.21.15.62:0/3939115680 wait complete. 2024-04-20T21:56:14.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:14.342+0000 7f0e49200700 1 Processor -- start 2024-04-20T21:56:14.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:14.343+0000 7f0e49200700 1 -- start start 2024-04-20T21:56:14.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:14.343+0000 7f0e49200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0e44106b80 0x7f0e4419c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:56:14.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:14.343+0000 7f0e49200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e4410c8f0 0x7f0e4419c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:56:14.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:14.343+0000 7f0e49200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e4419d050 con 0x7f0e4410c8f0 2024-04-20T21:56:14.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:14.343+0000 7f0e49200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e441a0de0 con 0x7f0e44106b80 2024-04-20T21:56:14.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:14.343+0000 7f0e41400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e4410c8f0 0x7f0e4419c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:56:14.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:14.343+0000 7f0e41400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e4410c8f0 0x7f0e4419c970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54162/0 (socket says 172.21.15.62:54162) 2024-04-20T21:56:14.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:14.343+0000 7f0e41400700 1 -- 172.21.15.62:0/2659119186 learned_addr learned my addr 172.21.15.62:0/2659119186 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:56:14.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:14.344+0000 7f0e41e00700 1 --2- 172.21.15.62:0/2659119186 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0e44106b80 0x7f0e4419c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:56:14.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:14.344+0000 7f0e41400700 1 -- 172.21.15.62:0/2659119186 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0e44106b80 msgr2=0x7f0e4419c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:56:14.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:14.344+0000 7f0e41400700 1 --2- 172.21.15.62:0/2659119186 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0e44106b80 0x7f0e4419c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:56:14.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:14.344+0000 7f0e41400700 1 -- 172.21.15.62:0/2659119186 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e340095f0 con 0x7f0e4410c8f0 2024-04-20T21:56:14.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:14.344+0000 7f0e41400700 1 --2- 172.21.15.62:0/2659119186 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e4410c8f0 0x7f0e4419c970 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f0e3000d900 tx=0x7f0e3000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:56:14.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:14.344+0000 7f0e3aa00700 1 -- 172.21.15.62:0/2659119186 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0e30004a00 con 0x7f0e4410c8f0 2024-04-20T21:56:14.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:14.345+0000 7f0e3aa00700 1 -- 172.21.15.62:0/2659119186 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0e300054e0 con 0x7f0e4410c8f0 2024-04-20T21:56:14.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:14.345+0000 7f0e49200700 1 -- 172.21.15.62:0/2659119186 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e441a10c0 con 0x7f0e4410c8f0 2024-04-20T21:56:14.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:14.345+0000 7f0e3aa00700 1 -- 172.21.15.62:0/2659119186 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0e30009d60 con 0x7f0e4410c8f0 2024-04-20T21:56:14.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:14.345+0000 7f0e49200700 1 -- 172.21.15.62:0/2659119186 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e441a1610 con 0x7f0e4410c8f0 2024-04-20T21:56:14.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:14.346+0000 7f0e49200700 1 -- 172.21.15.62:0/2659119186 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e44109b20 con 0x7f0e4410c8f0 2024-04-20T21:56:14.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:14.347+0000 7f0e3aa00700 1 -- 172.21.15.62:0/2659119186 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f0e30005020 con 0x7f0e4410c8f0 2024-04-20T21:56:14.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:14.348+0000 7f0e3aa00700 1 --2- 172.21.15.62:0/2659119186 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f0e2806ecc0 0x7f0e28071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:56:14.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:14.348+0000 7f0e3aa00700 1 -- 172.21.15.62:0/2659119186 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f0e3008f100 con 0x7f0e4410c8f0 2024-04-20T21:56:14.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:14.349+0000 7f0e41e00700 1 --2- 172.21.15.62:0/2659119186 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f0e2806ecc0 0x7f0e28071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:56:14.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:14.349+0000 7f0e41e00700 1 --2- 172.21.15.62:0/2659119186 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f0e2806ecc0 0x7f0e28071170 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f0e34006010 tx=0x7f0e3401a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:56:14.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:14.353+0000 7f0e3aa00700 1 -- 172.21.15.62:0/2659119186 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f0e30057e10 con 0x7f0e4410c8f0 2024-04-20T21:56:14.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:14.612+0000 7f0e49200700 1 -- 172.21.15.62:0/2659119186 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0e441a1910 con 0x7f0e2806ecc0 2024-04-20T21:56:14.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:14.615+0000 7f0e3aa00700 1 -- 172.21.15.62:0/2659119186 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f0e441a1910 con 0x7f0e2806ecc0 2024-04-20T21:56:14.614 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:56:14.614 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T21:56:14.614 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T21:56:14.615 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-20T21:56:14.615 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T21:56:14.615 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr", 2024-04-20T21:56:14.615 INFO:teuthology.orchestra.run.smithi062.stdout: "crash", 2024-04-20T21:56:14.615 INFO:teuthology.orchestra.run.smithi062.stdout: "mon", 2024-04-20T21:56:14.615 INFO:teuthology.orchestra.run.smithi062.stdout: "osd" 2024-04-20T21:56:14.615 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T21:56:14.615 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "14/23 daemons upgraded", 2024-04-20T21:56:14.615 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading mds daemons", 2024-04-20T21:56:14.615 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T21:56:14.615 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:56:14.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:14.618+0000 7f0e49200700 1 -- 172.21.15.62:0/2659119186 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f0e2806ecc0 msgr2=0x7f0e28071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:56:14.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:14.618+0000 7f0e49200700 1 --2- 172.21.15.62:0/2659119186 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f0e2806ecc0 0x7f0e28071170 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f0e34006010 tx=0x7f0e3401a040 comp rx=0 tx=0).stop 2024-04-20T21:56:14.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:14.618+0000 7f0e49200700 1 -- 172.21.15.62:0/2659119186 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e4410c8f0 msgr2=0x7f0e4419c970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:56:14.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:14.618+0000 7f0e49200700 1 --2- 172.21.15.62:0/2659119186 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e4410c8f0 0x7f0e4419c970 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f0e3000d900 tx=0x7f0e3000dcc0 comp rx=0 tx=0).stop 2024-04-20T21:56:14.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:14.618+0000 7f0e49200700 1 -- 172.21.15.62:0/2659119186 shutdown_connections 2024-04-20T21:56:14.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:14.618+0000 7f0e49200700 1 --2- 172.21.15.62:0/2659119186 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f0e2806ecc0 0x7f0e28071170 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:56:14.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:14.618+0000 7f0e49200700 1 --2- 172.21.15.62:0/2659119186 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0e44106b80 0x7f0e4419c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:56:14.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:14.618+0000 7f0e49200700 1 --2- 172.21.15.62:0/2659119186 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e4410c8f0 0x7f0e4419c970 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:56:14.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:14.618+0000 7f0e49200700 1 -- 172.21.15.62:0/2659119186 >> 172.21.15.62:0/2659119186 conn(0x7f0e44076060 msgr2=0x7f0e440779a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:56:14.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:14.618+0000 7f0e49200700 1 -- 172.21.15.62:0/2659119186 shutdown_connections 2024-04-20T21:56:14.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:14.618+0000 7f0e49200700 1 -- 172.21.15.62:0/2659119186 wait complete. 2024-04-20T21:56:14.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:14.767+0000 7fa1f9800700 1 -- 172.21.15.62:0/991282373 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1f41061b0 msgr2=0x7fa1f4106580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:56:14.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:14.767+0000 7fa1f9800700 1 --2- 172.21.15.62:0/991282373 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1f41061b0 0x7fa1f4106580 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fa1e4009910 tx=0x7fa1e4009c20 comp rx=0 tx=0).stop 2024-04-20T21:56:14.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:14.767+0000 7fa1f9800700 1 -- 172.21.15.62:0/991282373 shutdown_connections 2024-04-20T21:56:14.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:14.767+0000 7fa1f9800700 1 --2- 172.21.15.62:0/991282373 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1f4068540 0x7fa1f40689b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:56:14.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:14.767+0000 7fa1f9800700 1 --2- 172.21.15.62:0/991282373 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1f41061b0 0x7fa1f4106580 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:56:14.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:14.767+0000 7fa1f9800700 1 -- 172.21.15.62:0/991282373 >> 172.21.15.62:0/991282373 conn(0x7fa1f4074fa0 msgr2=0x7fa1f40753a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:56:14.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:14.767+0000 7fa1f9800700 1 -- 172.21.15.62:0/991282373 shutdown_connections 2024-04-20T21:56:14.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:14.767+0000 7fa1f9800700 1 -- 172.21.15.62:0/991282373 wait complete. 2024-04-20T21:56:14.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:14.768+0000 7fa1f9800700 1 Processor -- start 2024-04-20T21:56:14.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:14.768+0000 7fa1f9800700 1 -- start start 2024-04-20T21:56:14.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:14.769+0000 7fa1f9800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1f4068540 0x7fa1f4197d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:56:14.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:14.769+0000 7fa1f9800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1f41061b0 0x7fa1f41982a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:56:14.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:14.769+0000 7fa1f9800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1f4198980 con 0x7fa1f41061b0 2024-04-20T21:56:14.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:14.769+0000 7fa1f9800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1f419c710 con 0x7fa1f4068540 2024-04-20T21:56:14.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:14.769+0000 7fa1ebe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1f41061b0 0x7fa1f41982a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:56:14.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:14.769+0000 7fa1ebe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1f41061b0 0x7fa1f41982a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54172/0 (socket says 172.21.15.62:54172) 2024-04-20T21:56:14.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:14.769+0000 7fa1ebe00700 1 -- 172.21.15.62:0/2798794341 learned_addr learned my addr 172.21.15.62:0/2798794341 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:56:14.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:14.769+0000 7fa1ebe00700 1 -- 172.21.15.62:0/2798794341 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1f4068540 msgr2=0x7fa1f4197d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:56:14.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:14.769+0000 7fa1f3e00700 1 --2- 172.21.15.62:0/2798794341 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1f4068540 0x7fa1f4197d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:56:14.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:14.770+0000 7fa1ebe00700 1 --2- 172.21.15.62:0/2798794341 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1f4068540 0x7fa1f4197d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:56:14.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:14.770+0000 7fa1ebe00700 1 -- 172.21.15.62:0/2798794341 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1e40095f0 con 0x7fa1f41061b0 2024-04-20T21:56:14.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:14.770+0000 7fa1ebe00700 1 --2- 172.21.15.62:0/2798794341 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1f41061b0 0x7fa1f41982a0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc00d8d0 tx=0x7fa1dc00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:56:14.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:14.770+0000 7fa1f1600700 1 -- 172.21.15.62:0/2798794341 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa1dc009940 con 0x7fa1f41061b0 2024-04-20T21:56:14.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:14.770+0000 7fa1f1600700 1 -- 172.21.15.62:0/2798794341 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa1dc010460 con 0x7fa1f41061b0 2024-04-20T21:56:14.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:14.770+0000 7fa1f9800700 1 -- 172.21.15.62:0/2798794341 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1f419c9f0 con 0x7fa1f41061b0 2024-04-20T21:56:14.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:14.770+0000 7fa1f1600700 1 -- 172.21.15.62:0/2798794341 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa1dc00f5f0 con 0x7fa1f41061b0 2024-04-20T21:56:14.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:14.770+0000 7fa1f9800700 1 -- 172.21.15.62:0/2798794341 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1f419cfc0 con 0x7fa1f41061b0 2024-04-20T21:56:14.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:14.771+0000 7fa1f9800700 1 -- 172.21.15.62:0/2798794341 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1f4100eb0 con 0x7fa1f41061b0 2024-04-20T21:56:14.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:14.773+0000 7fa1f1600700 1 -- 172.21.15.62:0/2798794341 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fa1dc0105d0 con 0x7fa1f41061b0 2024-04-20T21:56:14.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:14.774+0000 7fa1f1600700 1 --2- 172.21.15.62:0/2798794341 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa1d806ecc0 0x7fa1d8071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:56:14.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:14.774+0000 7fa1f1600700 1 -- 172.21.15.62:0/2798794341 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fa1dc08ee10 con 0x7fa1f41061b0 2024-04-20T21:56:14.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:14.774+0000 7fa1f3e00700 1 --2- 172.21.15.62:0/2798794341 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa1d806ecc0 0x7fa1d8071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:56:14.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:14.775+0000 7fa1f3e00700 1 --2- 172.21.15.62:0/2798794341 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa1d806ecc0 0x7fa1d8071170 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fa1e4006010 tx=0x7fa1e400b540 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:56:14.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:14.779+0000 7fa1f1600700 1 -- 172.21.15.62:0/2798794341 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fa1dc057ba0 con 0x7fa1f41061b0 2024-04-20T21:56:14.929 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:56:14 smithi062 ceph-mon[76322]: from='client.35160 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:56:14.929 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:56:14 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/2416825308' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T21:56:15.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:15.115+0000 7fa1f9800700 1 -- 172.21.15.62:0/2798794341 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fa1f4062380 con 0x7fa1f41061b0 2024-04-20T21:56:15.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:15.116+0000 7fa1f1600700 1 -- 172.21.15.62:0/2798794341 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7fa1dc0161c0 con 0x7fa1f41061b0 2024-04-20T21:56:15.115 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-20T21:56:15.116 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-20T21:56:15.116 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is degraded 2024-04-20T21:56:15.116 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T21:56:15.116 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T21:56:15.116 INFO:teuthology.orchestra.run.smithi062.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-20T21:56:15.116 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-20T21:56:15.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:15.119+0000 7fa1f9800700 1 -- 172.21.15.62:0/2798794341 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa1d806ecc0 msgr2=0x7fa1d8071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:56:15.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:15.119+0000 7fa1f9800700 1 --2- 172.21.15.62:0/2798794341 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa1d806ecc0 0x7fa1d8071170 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fa1e4006010 tx=0x7fa1e400b540 comp rx=0 tx=0).stop 2024-04-20T21:56:15.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:15.119+0000 7fa1f9800700 1 -- 172.21.15.62:0/2798794341 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1f41061b0 msgr2=0x7fa1f41982a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:56:15.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:15.119+0000 7fa1f9800700 1 --2- 172.21.15.62:0/2798794341 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1f41061b0 0x7fa1f41982a0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc00d8d0 tx=0x7fa1dc00dc90 comp rx=0 tx=0).stop 2024-04-20T21:56:15.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:15.120+0000 7fa1f9800700 1 -- 172.21.15.62:0/2798794341 shutdown_connections 2024-04-20T21:56:15.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:15.120+0000 7fa1f9800700 1 --2- 172.21.15.62:0/2798794341 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa1d806ecc0 0x7fa1d8071170 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:56:15.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:15.120+0000 7fa1f9800700 1 --2- 172.21.15.62:0/2798794341 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1f4068540 0x7fa1f4197d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:56:15.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:15.120+0000 7fa1f9800700 1 --2- 172.21.15.62:0/2798794341 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1f41061b0 0x7fa1f41982a0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:56:15.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:15.120+0000 7fa1f9800700 1 -- 172.21.15.62:0/2798794341 >> 172.21.15.62:0/2798794341 conn(0x7fa1f4074fa0 msgr2=0x7fa1f4107100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:56:15.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:15.120+0000 7fa1f9800700 1 -- 172.21.15.62:0/2798794341 shutdown_connections 2024-04-20T21:56:15.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:15.120+0000 7fa1f9800700 1 -- 172.21.15.62:0/2798794341 wait complete. 2024-04-20T21:56:15.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:56:14 smithi189 ceph-mon[57022]: from='client.35160 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:56:15.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:56:14 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/2416825308' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T21:56:16.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:56:15 smithi189 ceph-mon[57022]: from='client.35172 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:56:16.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:56:15 smithi189 ceph-mon[57022]: pgmap v628: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:56:16.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:56:15 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/2798794341' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T21:56:16.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:56:15 smithi062 ceph-mon[76322]: from='client.35172 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:56:16.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:56:15 smithi062 ceph-mon[76322]: pgmap v628: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:56:16.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:56:15 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/2798794341' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T21:56:18.143 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:56:17 smithi062 ceph-mon[76322]: pgmap v629: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:56:18.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:56:17 smithi189 ceph-mon[57022]: pgmap v629: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:56:20.082 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:56:19 smithi062 ceph-mon[76322]: pgmap v630: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:56:20.082 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:56:19 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:56:20.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:56:19 smithi189 ceph-mon[57022]: pgmap v630: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:56:20.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:56:19 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:56:22.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:56:21 smithi189 ceph-mon[57022]: pgmap v631: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:56:22.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:56:21 smithi062 ceph-mon[76322]: pgmap v631: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:56:24.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:56:23 smithi189 ceph-mon[57022]: pgmap v632: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:56:24.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:56:23 smithi062 ceph-mon[76322]: pgmap v632: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:56:26.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:56:25 smithi189 ceph-mon[57022]: pgmap v633: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:56:26.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:56:25 smithi062 ceph-mon[76322]: pgmap v633: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:56:27.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:56:27 smithi189 ceph-mon[57022]: pgmap v634: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:56:28.143 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:56:27 smithi062 ceph-mon[76322]: pgmap v634: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:56:30.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:56:29 smithi189 ceph-mon[57022]: pgmap v635: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:56:30.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:56:29 smithi062 ceph-mon[76322]: pgmap v635: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:56:32.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:56:31 smithi189 ceph-mon[57022]: pgmap v636: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:56:32.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:56:31 smithi062 ceph-mon[76322]: pgmap v636: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:56:34.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:56:33 smithi189 ceph-mon[57022]: pgmap v637: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:56:34.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:56:33 smithi062 ceph-mon[76322]: pgmap v637: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:56:35.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:56:34 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:56:35.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:56:34 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:56:36.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:56:35 smithi189 ceph-mon[57022]: pgmap v638: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:56:36.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:56:35 smithi062 ceph-mon[76322]: pgmap v638: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:56:38.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:56:37 smithi189 ceph-mon[57022]: pgmap v639: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:56:38.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:56:37 smithi062 ceph-mon[76322]: pgmap v639: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:56:40.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:56:39 smithi189 ceph-mon[57022]: pgmap v640: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:56:40.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:56:39 smithi062 ceph-mon[76322]: pgmap v640: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:56:42.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:56:41 smithi189 ceph-mon[57022]: pgmap v641: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:56:42.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:56:41 smithi062 ceph-mon[76322]: pgmap v641: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:56:44.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:56:43 smithi189 ceph-mon[57022]: pgmap v642: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:56:44.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:56:43 smithi062 ceph-mon[76322]: pgmap v642: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:56:45.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:45.299+0000 7fa543e00700 1 -- 172.21.15.62:0/2767279356 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5440ff170 msgr2=0x7fa5440ff540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:56:45.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:45.299+0000 7fa543e00700 1 --2- 172.21.15.62:0/2767279356 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5440ff170 0x7fa5440ff540 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fa534009910 tx=0x7fa534009c20 comp rx=0 tx=0).stop 2024-04-20T21:56:45.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:45.299+0000 7fa543e00700 1 -- 172.21.15.62:0/2767279356 shutdown_connections 2024-04-20T21:56:45.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:45.299+0000 7fa543e00700 1 --2- 172.21.15.62:0/2767279356 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5440ffa80 0x7fa54410a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:56:45.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:45.299+0000 7fa543e00700 1 --2- 172.21.15.62:0/2767279356 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5440ff170 0x7fa5440ff540 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:56:45.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:45.299+0000 7fa543e00700 1 -- 172.21.15.62:0/2767279356 >> 172.21.15.62:0/2767279356 conn(0x7fa544074770 msgr2=0x7fa544074b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:56:45.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:45.299+0000 7fa543e00700 1 -- 172.21.15.62:0/2767279356 shutdown_connections 2024-04-20T21:56:45.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:45.299+0000 7fa543e00700 1 -- 172.21.15.62:0/2767279356 wait complete. 2024-04-20T21:56:45.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:45.300+0000 7fa543e00700 1 Processor -- start 2024-04-20T21:56:45.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:45.300+0000 7fa543e00700 1 -- start start 2024-04-20T21:56:45.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:45.301+0000 7fa543e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5440ff170 0x7fa54419b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:56:45.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:45.301+0000 7fa543e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5440ffa80 0x7fa54419be50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:56:45.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:45.301+0000 7fa543e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa54419c4e0 con 0x7fa5440ffa80 2024-04-20T21:56:45.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:45.301+0000 7fa543e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa544195990 con 0x7fa5440ff170 2024-04-20T21:56:45.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:45.301+0000 7fa542000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5440ffa80 0x7fa54419be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:56:45.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:45.301+0000 7fa542000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5440ffa80 0x7fa54419be50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45318/0 (socket says 172.21.15.62:45318) 2024-04-20T21:56:45.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:45.301+0000 7fa542000700 1 -- 172.21.15.62:0/3995204616 learned_addr learned my addr 172.21.15.62:0/3995204616 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:56:45.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:45.301+0000 7fa542a00700 1 --2- 172.21.15.62:0/3995204616 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5440ff170 0x7fa54419b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:56:45.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:45.302+0000 7fa542000700 1 -- 172.21.15.62:0/3995204616 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5440ff170 msgr2=0x7fa54419b910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:56:45.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:45.302+0000 7fa542000700 1 --2- 172.21.15.62:0/3995204616 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5440ff170 0x7fa54419b910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:56:45.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:45.302+0000 7fa542000700 1 -- 172.21.15.62:0/3995204616 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5340095f0 con 0x7fa5440ffa80 2024-04-20T21:56:45.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:45.302+0000 7fa542a00700 1 --2- 172.21.15.62:0/3995204616 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5440ff170 0x7fa54419b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-20T21:56:45.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:45.302+0000 7fa542000700 1 --2- 172.21.15.62:0/3995204616 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5440ffa80 0x7fa54419be50 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fa53000d8d0 tx=0x7fa53000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:56:45.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:45.302+0000 7fa53b400700 1 -- 172.21.15.62:0/3995204616 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa530009940 con 0x7fa5440ffa80 2024-04-20T21:56:45.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:45.302+0000 7fa543e00700 1 -- 172.21.15.62:0/3995204616 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa544195c70 con 0x7fa5440ffa80 2024-04-20T21:56:45.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:45.302+0000 7fa53b400700 1 -- 172.21.15.62:0/3995204616 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa530010460 con 0x7fa5440ffa80 2024-04-20T21:56:45.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:45.302+0000 7fa53b400700 1 -- 172.21.15.62:0/3995204616 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa53000f5f0 con 0x7fa5440ffa80 2024-04-20T21:56:45.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:45.302+0000 7fa543e00700 1 -- 172.21.15.62:0/3995204616 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5441961c0 con 0x7fa5440ffa80 2024-04-20T21:56:45.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:45.304+0000 7fa543e00700 1 -- 172.21.15.62:0/3995204616 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa544107bf0 con 0x7fa5440ffa80 2024-04-20T21:56:45.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:45.305+0000 7fa53b400700 1 -- 172.21.15.62:0/3995204616 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fa5300105d0 con 0x7fa5440ffa80 2024-04-20T21:56:45.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:45.306+0000 7fa53b400700 1 --2- 172.21.15.62:0/3995204616 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa52806ecc0 0x7fa528071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:56:45.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:45.306+0000 7fa53b400700 1 -- 172.21.15.62:0/3995204616 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fa53008ec20 con 0x7fa5440ffa80 2024-04-20T21:56:45.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:45.306+0000 7fa542a00700 1 --2- 172.21.15.62:0/3995204616 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa52806ecc0 0x7fa528071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:56:45.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:45.307+0000 7fa542a00700 1 --2- 172.21.15.62:0/3995204616 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa52806ecc0 0x7fa528071170 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fa53400b5c0 tx=0x7fa53401a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:56:45.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:45.311+0000 7fa53b400700 1 -- 172.21.15.62:0/3995204616 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fa53005b500 con 0x7fa5440ffa80 2024-04-20T21:56:45.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:45.570+0000 7fa543e00700 1 -- 172.21.15.62:0/3995204616 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa5441963e0 con 0x7fa52806ecc0 2024-04-20T21:56:45.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:45.573+0000 7fa53b400700 1 -- 172.21.15.62:0/3995204616 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fa5441963e0 con 0x7fa52806ecc0 2024-04-20T21:56:45.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:45.577+0000 7fa543e00700 1 -- 172.21.15.62:0/3995204616 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa52806ecc0 msgr2=0x7fa528071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:56:45.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:45.577+0000 7fa543e00700 1 --2- 172.21.15.62:0/3995204616 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa52806ecc0 0x7fa528071170 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fa53400b5c0 tx=0x7fa53401a040 comp rx=0 tx=0).stop 2024-04-20T21:56:45.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:45.577+0000 7fa543e00700 1 -- 172.21.15.62:0/3995204616 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5440ffa80 msgr2=0x7fa54419be50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:56:45.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:45.577+0000 7fa543e00700 1 --2- 172.21.15.62:0/3995204616 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5440ffa80 0x7fa54419be50 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fa53000d8d0 tx=0x7fa53000dc90 comp rx=0 tx=0).stop 2024-04-20T21:56:45.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:45.577+0000 7fa543e00700 1 -- 172.21.15.62:0/3995204616 shutdown_connections 2024-04-20T21:56:45.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:45.577+0000 7fa543e00700 1 --2- 172.21.15.62:0/3995204616 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa52806ecc0 0x7fa528071170 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:56:45.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:45.577+0000 7fa543e00700 1 --2- 172.21.15.62:0/3995204616 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5440ff170 0x7fa54419b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:56:45.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:45.577+0000 7fa543e00700 1 --2- 172.21.15.62:0/3995204616 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5440ffa80 0x7fa54419be50 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:56:45.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:45.577+0000 7fa543e00700 1 -- 172.21.15.62:0/3995204616 >> 172.21.15.62:0/3995204616 conn(0x7fa544074770 msgr2=0x7fa544104d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:56:45.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:45.577+0000 7fa543e00700 1 -- 172.21.15.62:0/3995204616 shutdown_connections 2024-04-20T21:56:45.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:45.577+0000 7fa543e00700 1 -- 172.21.15.62:0/3995204616 wait complete. 2024-04-20T21:56:45.593 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T21:56:45.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:45.747+0000 7fa1f2400700 1 -- 172.21.15.62:0/2602328027 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1ec102470 msgr2=0x7fa1ec1028e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:56:45.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:45.747+0000 7fa1f2400700 1 --2- 172.21.15.62:0/2602328027 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1ec102470 0x7fa1ec1028e0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fa1d8009910 tx=0x7fa1d8009c20 comp rx=0 tx=0).stop 2024-04-20T21:56:45.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:45.747+0000 7fa1f2400700 1 -- 172.21.15.62:0/2602328027 shutdown_connections 2024-04-20T21:56:45.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:45.747+0000 7fa1f2400700 1 --2- 172.21.15.62:0/2602328027 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1ec102470 0x7fa1ec1028e0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:56:45.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:45.747+0000 7fa1f2400700 1 --2- 172.21.15.62:0/2602328027 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1ec108470 0x7fa1ec108840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:56:45.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:45.747+0000 7fa1f2400700 1 -- 172.21.15.62:0/2602328027 >> 172.21.15.62:0/2602328027 conn(0x7fa1ec0fdf90 msgr2=0x7fa1ec1003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:56:45.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:45.747+0000 7fa1f2400700 1 -- 172.21.15.62:0/2602328027 shutdown_connections 2024-04-20T21:56:45.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:45.747+0000 7fa1f2400700 1 -- 172.21.15.62:0/2602328027 wait complete. 2024-04-20T21:56:45.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:45.748+0000 7fa1f2400700 1 Processor -- start 2024-04-20T21:56:45.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:45.748+0000 7fa1f2400700 1 -- start start 2024-04-20T21:56:45.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:45.749+0000 7fa1f2400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1ec102470 0x7fa1ec1981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:56:45.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:45.749+0000 7fa1f2400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1ec108470 0x7fa1ec198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:56:45.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:45.749+0000 7fa1f2400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1ec198e10 con 0x7fa1ec102470 2024-04-20T21:56:45.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:45.749+0000 7fa1f2400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1ec19cb30 con 0x7fa1ec108470 2024-04-20T21:56:45.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:45.749+0000 7fa1eb400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1ec102470 0x7fa1ec1981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:56:45.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:45.749+0000 7fa1eb400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1ec102470 0x7fa1ec1981f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45332/0 (socket says 172.21.15.62:45332) 2024-04-20T21:56:45.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:45.749+0000 7fa1eb400700 1 -- 172.21.15.62:0/2071076822 learned_addr learned my addr 172.21.15.62:0/2071076822 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:56:45.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:45.749+0000 7fa1eaa00700 1 --2- 172.21.15.62:0/2071076822 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1ec108470 0x7fa1ec198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:56:45.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:45.749+0000 7fa1eb400700 1 -- 172.21.15.62:0/2071076822 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1ec108470 msgr2=0x7fa1ec198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:56:45.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:45.749+0000 7fa1eb400700 1 --2- 172.21.15.62:0/2071076822 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1ec108470 0x7fa1ec198730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:56:45.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:45.749+0000 7fa1eb400700 1 -- 172.21.15.62:0/2071076822 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1d80095f0 con 0x7fa1ec102470 2024-04-20T21:56:45.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:45.750+0000 7fa1eb400700 1 --2- 172.21.15.62:0/2071076822 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1ec102470 0x7fa1ec1981f0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc00e920 tx=0x7fa1dc00ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:56:45.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:45.750+0000 7fa1e3e00700 1 -- 172.21.15.62:0/2071076822 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa1dc009790 con 0x7fa1ec102470 2024-04-20T21:56:45.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:45.750+0000 7fa1e3e00700 1 -- 172.21.15.62:0/2071076822 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa1dc00cba0 con 0x7fa1ec102470 2024-04-20T21:56:45.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:45.750+0000 7fa1e3e00700 1 -- 172.21.15.62:0/2071076822 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa1dc018650 con 0x7fa1ec102470 2024-04-20T21:56:45.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:45.750+0000 7fa1f2400700 1 -- 172.21.15.62:0/2071076822 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1ec19ce10 con 0x7fa1ec102470 2024-04-20T21:56:45.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:45.750+0000 7fa1f2400700 1 -- 172.21.15.62:0/2071076822 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1ec19d360 con 0x7fa1ec102470 2024-04-20T21:56:45.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:45.751+0000 7fa1f2400700 1 -- 172.21.15.62:0/2071076822 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1ec10a970 con 0x7fa1ec102470 2024-04-20T21:56:45.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:45.752+0000 7fa1e3e00700 1 -- 172.21.15.62:0/2071076822 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fa1dc00cd10 con 0x7fa1ec102470 2024-04-20T21:56:45.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:45.753+0000 7fa1e3e00700 1 --2- 172.21.15.62:0/2071076822 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa1d006ecc0 0x7fa1d0071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:56:45.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:45.753+0000 7fa1e3e00700 1 -- 172.21.15.62:0/2071076822 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fa1dc014070 con 0x7fa1ec102470 2024-04-20T21:56:45.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:45.754+0000 7fa1eaa00700 1 --2- 172.21.15.62:0/2071076822 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa1d006ecc0 0x7fa1d0071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:56:45.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:45.754+0000 7fa1eaa00700 1 --2- 172.21.15.62:0/2071076822 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa1d006ecc0 0x7fa1d0071170 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fa1ec1997a0 tx=0x7fa1d8009f70 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:56:45.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:45.758+0000 7fa1e3e00700 1 -- 172.21.15.62:0/2071076822 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fa1dc053ff0 con 0x7fa1ec102470 2024-04-20T21:56:46.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.024+0000 7fa1f2400700 1 -- 172.21.15.62:0/2071076822 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa1ec199550 con 0x7fa1d006ecc0 2024-04-20T21:56:46.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.027+0000 7fa1e3e00700 1 -- 172.21.15.62:0/2071076822 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fa1ec199550 con 0x7fa1d006ecc0 2024-04-20T21:56:46.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.031+0000 7fa1f2400700 1 -- 172.21.15.62:0/2071076822 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa1d006ecc0 msgr2=0x7fa1d0071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:56:46.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.031+0000 7fa1f2400700 1 --2- 172.21.15.62:0/2071076822 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa1d006ecc0 0x7fa1d0071170 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fa1ec1997a0 tx=0x7fa1d8009f70 comp rx=0 tx=0).stop 2024-04-20T21:56:46.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.031+0000 7fa1f2400700 1 -- 172.21.15.62:0/2071076822 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1ec102470 msgr2=0x7fa1ec1981f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:56:46.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.031+0000 7fa1f2400700 1 --2- 172.21.15.62:0/2071076822 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1ec102470 0x7fa1ec1981f0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc00e920 tx=0x7fa1dc00ece0 comp rx=0 tx=0).stop 2024-04-20T21:56:46.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.032+0000 7fa1f2400700 1 -- 172.21.15.62:0/2071076822 shutdown_connections 2024-04-20T21:56:46.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.032+0000 7fa1f2400700 1 --2- 172.21.15.62:0/2071076822 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa1d006ecc0 0x7fa1d0071170 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:56:46.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.032+0000 7fa1f2400700 1 --2- 172.21.15.62:0/2071076822 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1ec102470 0x7fa1ec1981f0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:56:46.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.032+0000 7fa1f2400700 1 --2- 172.21.15.62:0/2071076822 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1ec108470 0x7fa1ec198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:56:46.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.032+0000 7fa1f2400700 1 -- 172.21.15.62:0/2071076822 >> 172.21.15.62:0/2071076822 conn(0x7fa1ec0fdf90 msgr2=0x7fa1ec0ff3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:56:46.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.032+0000 7fa1f2400700 1 -- 172.21.15.62:0/2071076822 shutdown_connections 2024-04-20T21:56:46.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.032+0000 7fa1f2400700 1 -- 172.21.15.62:0/2071076822 wait complete. 2024-04-20T21:56:46.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.199+0000 7fb053800700 1 -- 172.21.15.62:0/1451114320 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb04c108470 msgr2=0x7fb04c108840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:56:46.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.199+0000 7fb053800700 1 --2- 172.21.15.62:0/1451114320 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb04c108470 0x7fb04c108840 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fb040009910 tx=0x7fb040009c20 comp rx=0 tx=0).stop 2024-04-20T21:56:46.198 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:56:45 smithi062 ceph-mon[76322]: pgmap v643: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:56:46.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.199+0000 7fb053800700 1 -- 172.21.15.62:0/1451114320 shutdown_connections 2024-04-20T21:56:46.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.199+0000 7fb053800700 1 --2- 172.21.15.62:0/1451114320 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb04c102470 0x7fb04c1028e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:56:46.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.199+0000 7fb053800700 1 --2- 172.21.15.62:0/1451114320 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb04c108470 0x7fb04c108840 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:56:46.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.199+0000 7fb053800700 1 -- 172.21.15.62:0/1451114320 >> 172.21.15.62:0/1451114320 conn(0x7fb04c0fdf90 msgr2=0x7fb04c1003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:56:46.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.199+0000 7fb053800700 1 -- 172.21.15.62:0/1451114320 shutdown_connections 2024-04-20T21:56:46.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.199+0000 7fb053800700 1 -- 172.21.15.62:0/1451114320 wait complete. 2024-04-20T21:56:46.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.200+0000 7fb053800700 1 Processor -- start 2024-04-20T21:56:46.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.201+0000 7fb053800700 1 -- start start 2024-04-20T21:56:46.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.201+0000 7fb053800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb04c102470 0x7fb04c1981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:56:46.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.201+0000 7fb053800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb04c108470 0x7fb04c198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:56:46.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.201+0000 7fb053800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb04c198e10 con 0x7fb04c108470 2024-04-20T21:56:46.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.201+0000 7fb053800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb04c19cba0 con 0x7fb04c102470 2024-04-20T21:56:46.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.201+0000 7fb04be00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb04c108470 0x7fb04c198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:56:46.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.201+0000 7fb04be00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb04c108470 0x7fb04c198730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45358/0 (socket says 172.21.15.62:45358) 2024-04-20T21:56:46.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.201+0000 7fb04be00700 1 -- 172.21.15.62:0/82915599 learned_addr learned my addr 172.21.15.62:0/82915599 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:56:46.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.202+0000 7fb050e00700 1 --2- 172.21.15.62:0/82915599 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb04c102470 0x7fb04c1981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:56:46.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.202+0000 7fb04be00700 1 -- 172.21.15.62:0/82915599 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb04c102470 msgr2=0x7fb04c1981f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:56:46.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.202+0000 7fb04be00700 1 --2- 172.21.15.62:0/82915599 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb04c102470 0x7fb04c1981f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:56:46.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.202+0000 7fb04be00700 1 -- 172.21.15.62:0/82915599 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0400095f0 con 0x7fb04c108470 2024-04-20T21:56:46.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.202+0000 7fb04be00700 1 --2- 172.21.15.62:0/82915599 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb04c108470 0x7fb04c198730 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fb03800d8d0 tx=0x7fb03800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:56:46.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.203+0000 7fb049600700 1 -- 172.21.15.62:0/82915599 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb038009940 con 0x7fb04c108470 2024-04-20T21:56:46.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.203+0000 7fb049600700 1 -- 172.21.15.62:0/82915599 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb038010460 con 0x7fb04c108470 2024-04-20T21:56:46.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.203+0000 7fb053800700 1 -- 172.21.15.62:0/82915599 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb04c19ce80 con 0x7fb04c108470 2024-04-20T21:56:46.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.203+0000 7fb049600700 1 -- 172.21.15.62:0/82915599 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb03800f5f0 con 0x7fb04c108470 2024-04-20T21:56:46.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.203+0000 7fb053800700 1 -- 172.21.15.62:0/82915599 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb04c19d3d0 con 0x7fb04c108470 2024-04-20T21:56:46.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.204+0000 7fb053800700 1 -- 172.21.15.62:0/82915599 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb04c10a9e0 con 0x7fb04c108470 2024-04-20T21:56:46.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.205+0000 7fb049600700 1 -- 172.21.15.62:0/82915599 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fb0380105d0 con 0x7fb04c108470 2024-04-20T21:56:46.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.206+0000 7fb049600700 1 --2- 172.21.15.62:0/82915599 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb03406ecc0 0x7fb034071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:56:46.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.207+0000 7fb049600700 1 -- 172.21.15.62:0/82915599 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fb03808ef20 con 0x7fb04c108470 2024-04-20T21:56:46.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.207+0000 7fb050e00700 1 --2- 172.21.15.62:0/82915599 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb03406ecc0 0x7fb034071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:56:46.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.207+0000 7fb050e00700 1 --2- 172.21.15.62:0/82915599 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb03406ecc0 0x7fb034071170 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fb040006010 tx=0x7fb04001a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:56:46.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.212+0000 7fb049600700 1 -- 172.21.15.62:0/82915599 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fb038057c30 con 0x7fb04c108470 2024-04-20T21:56:46.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:56:45 smithi189 ceph-mon[57022]: pgmap v643: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:56:46.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.459+0000 7fb053800700 1 -- 172.21.15.62:0/82915599 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fb04c1995c0 con 0x7fb03406ecc0 2024-04-20T21:56:46.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.493+0000 7fb049600700 1 -- 172.21.15.62:0/82915599 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fb04c1995c0 con 0x7fb03406ecc0 2024-04-20T21:56:46.492 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T21:56:46.492 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (22m) 13m ago 36m 14.2M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T21:56:46.492 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (36m) 13m ago 36m 9512k - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T21:56:46.492 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (35m) 14m ago 35m 8942k - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T21:56:46.492 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (18m) 13m ago 36m 6660k - 19.0.0-3133-ga8c84f59 76bc6b91b726 2825d5ba11c6 2024-04-20T21:56:46.492 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (18m) 14m ago 35m 6665k - 19.0.0-3133-ga8c84f59 76bc6b91b726 20f746fa1bba 2024-04-20T21:56:46.492 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (21m) 13m ago 36m 74.2M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T21:56:46.493 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (14m) 13m ago 29m 11.2M - 19.0.0-3133-ga8c84f59 76bc6b91b726 de80bc90a683 2024-04-20T21:56:46.493 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (14m) 13m ago 29m 11.9M - 19.0.0-3133-ga8c84f59 76bc6b91b726 cd793b7ee98e 2024-04-20T21:56:46.493 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (29m) 14m ago 29m 117M - 18.2.0 dc2bc1663786 fb2a8a223db8 2024-04-20T21:56:46.493 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (29m) 14m ago 29m 55.6M - 18.2.0 dc2bc1663786 f2eec2ee8951 2024-04-20T21:56:46.493 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (25m) 13m ago 38m 511M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T21:56:46.493 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (24m) 14m ago 35m 452M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T21:56:46.493 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (20m) 13m ago 38m 55.9M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 019eb1479485 2024-04-20T21:56:46.493 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (19m) 14m ago 35m 39.8M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 55aac560aab6 2024-04-20T21:56:46.493 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (23m) 13m ago 36m 7952k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T21:56:46.493 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (23m) 14m ago 35m 8304k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T21:56:46.493 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (18m) 13m ago 34m 88.0M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 ee2bb901f26e 2024-04-20T21:56:46.493 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (17m) 13m ago 33m 92.3M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 9075152d4162 2024-04-20T21:56:46.493 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (16m) 13m ago 32m 74.4M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 2d4f3e38dc0e 2024-04-20T21:56:46.493 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (16m) 14m ago 32m 87.6M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 8f97b17ae0eb 2024-04-20T21:56:46.493 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (15m) 14m ago 31m 90.7M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 653d8079967d 2024-04-20T21:56:46.494 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (15m) 14m ago 31m 11.5M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 596dded14c12 2024-04-20T21:56:46.494 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (23m) 13m ago 35m 74.0M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T21:56:46.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.496+0000 7fb053800700 1 -- 172.21.15.62:0/82915599 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb03406ecc0 msgr2=0x7fb034071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:56:46.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.496+0000 7fb053800700 1 --2- 172.21.15.62:0/82915599 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb03406ecc0 0x7fb034071170 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fb040006010 tx=0x7fb04001a040 comp rx=0 tx=0).stop 2024-04-20T21:56:46.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.496+0000 7fb053800700 1 -- 172.21.15.62:0/82915599 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb04c108470 msgr2=0x7fb04c198730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:56:46.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.496+0000 7fb053800700 1 --2- 172.21.15.62:0/82915599 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb04c108470 0x7fb04c198730 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fb03800d8d0 tx=0x7fb03800dc90 comp rx=0 tx=0).stop 2024-04-20T21:56:46.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.496+0000 7fb053800700 1 -- 172.21.15.62:0/82915599 shutdown_connections 2024-04-20T21:56:46.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.496+0000 7fb053800700 1 --2- 172.21.15.62:0/82915599 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb03406ecc0 0x7fb034071170 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:56:46.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.496+0000 7fb053800700 1 --2- 172.21.15.62:0/82915599 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb04c102470 0x7fb04c1981f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:56:46.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.496+0000 7fb053800700 1 --2- 172.21.15.62:0/82915599 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb04c108470 0x7fb04c198730 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:56:46.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.496+0000 7fb053800700 1 -- 172.21.15.62:0/82915599 >> 172.21.15.62:0/82915599 conn(0x7fb04c0fdf90 msgr2=0x7fb04c1002c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:56:46.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.496+0000 7fb053800700 1 -- 172.21.15.62:0/82915599 shutdown_connections 2024-04-20T21:56:46.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.496+0000 7fb053800700 1 -- 172.21.15.62:0/82915599 wait complete. 2024-04-20T21:56:46.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.655+0000 7f0dc1e00700 1 -- 172.21.15.62:0/2241090969 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0dbc10c660 msgr2=0x7f0dbc10ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:56:46.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.655+0000 7f0dc1e00700 1 --2- 172.21.15.62:0/2241090969 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0dbc10c660 0x7f0dbc10ca30 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f0db0009910 tx=0x7f0db0009c20 comp rx=0 tx=0).stop 2024-04-20T21:56:46.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.656+0000 7f0dc1e00700 1 -- 172.21.15.62:0/2241090969 shutdown_connections 2024-04-20T21:56:46.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.656+0000 7f0dc1e00700 1 --2- 172.21.15.62:0/2241090969 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0dbc106b80 0x7f0dbc106ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:56:46.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.656+0000 7f0dc1e00700 1 --2- 172.21.15.62:0/2241090969 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0dbc10c660 0x7f0dbc10ca30 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:56:46.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.656+0000 7f0dc1e00700 1 -- 172.21.15.62:0/2241090969 >> 172.21.15.62:0/2241090969 conn(0x7f0dbc075e90 msgr2=0x7f0dbc0782a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:56:46.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.656+0000 7f0dc1e00700 1 -- 172.21.15.62:0/2241090969 shutdown_connections 2024-04-20T21:56:46.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.656+0000 7f0dc1e00700 1 -- 172.21.15.62:0/2241090969 wait complete. 2024-04-20T21:56:46.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.657+0000 7f0dc1e00700 1 Processor -- start 2024-04-20T21:56:46.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.657+0000 7f0dc1e00700 1 -- start start 2024-04-20T21:56:46.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.657+0000 7f0dc1e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0dbc106b80 0x7f0dbc19c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:56:46.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.657+0000 7f0dc1e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0dbc10c660 0x7f0dbc19c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:56:46.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.657+0000 7f0dc1e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dbc19cea0 con 0x7f0dbc10c660 2024-04-20T21:56:46.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.657+0000 7f0dc1e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dbc1a0c30 con 0x7f0dbc106b80 2024-04-20T21:56:46.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.658+0000 7f0dbbe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0dbc10c660 0x7f0dbc19c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:56:46.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.658+0000 7f0dbbe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0dbc10c660 0x7f0dbc19c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45380/0 (socket says 172.21.15.62:45380) 2024-04-20T21:56:46.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.658+0000 7f0dbbe00700 1 -- 172.21.15.62:0/3028782962 learned_addr learned my addr 172.21.15.62:0/3028782962 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:56:46.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.658+0000 7f0dc0a00700 1 --2- 172.21.15.62:0/3028782962 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0dbc106b80 0x7f0dbc19c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:56:46.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.658+0000 7f0dbbe00700 1 -- 172.21.15.62:0/3028782962 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0dbc106b80 msgr2=0x7f0dbc19c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:56:46.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.658+0000 7f0dbbe00700 1 --2- 172.21.15.62:0/3028782962 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0dbc106b80 0x7f0dbc19c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:56:46.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.658+0000 7f0dbbe00700 1 -- 172.21.15.62:0/3028782962 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0db00095f0 con 0x7f0dbc10c660 2024-04-20T21:56:46.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.659+0000 7f0dbbe00700 1 --2- 172.21.15.62:0/3028782962 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0dbc10c660 0x7f0dbc19c7c0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f0da800c8a0 tx=0x7f0da800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:56:46.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.659+0000 7f0db9600700 1 -- 172.21.15.62:0/3028782962 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0da800cea0 con 0x7f0dbc10c660 2024-04-20T21:56:46.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.659+0000 7f0db9600700 1 -- 172.21.15.62:0/3028782962 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0da8004830 con 0x7f0dbc10c660 2024-04-20T21:56:46.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.659+0000 7f0dc1e00700 1 -- 172.21.15.62:0/3028782962 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0dbc1a0f10 con 0x7f0dbc10c660 2024-04-20T21:56:46.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.659+0000 7f0db9600700 1 -- 172.21.15.62:0/3028782962 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0da80055b0 con 0x7f0dbc10c660 2024-04-20T21:56:46.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.659+0000 7f0dc1e00700 1 -- 172.21.15.62:0/3028782962 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0dbc1a1460 con 0x7f0dbc10c660 2024-04-20T21:56:46.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.660+0000 7f0dc1e00700 1 -- 172.21.15.62:0/3028782962 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0dbc10eb40 con 0x7f0dbc10c660 2024-04-20T21:56:46.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.661+0000 7f0db9600700 1 -- 172.21.15.62:0/3028782962 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f0da80049a0 con 0x7f0dbc10c660 2024-04-20T21:56:46.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.662+0000 7f0db9600700 1 --2- 172.21.15.62:0/3028782962 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f0da406ecc0 0x7f0da4071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:56:46.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.662+0000 7f0db9600700 1 -- 172.21.15.62:0/3028782962 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f0da808eaf0 con 0x7f0dbc10c660 2024-04-20T21:56:46.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.663+0000 7f0dc0a00700 1 --2- 172.21.15.62:0/3028782962 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f0da406ecc0 0x7f0da4071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:56:46.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.663+0000 7f0dc0a00700 1 --2- 172.21.15.62:0/3028782962 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f0da406ecc0 0x7f0da4071170 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f0db0000c00 tx=0x7f0db0005fd0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:56:46.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.667+0000 7f0db9600700 1 -- 172.21.15.62:0/3028782962 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f0da8057880 con 0x7f0dbc10c660 2024-04-20T21:56:46.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.986+0000 7f0dc1e00700 1 -- 172.21.15.62:0/3028782962 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f0dbc062380 con 0x7f0dbc10c660 2024-04-20T21:56:46.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.987+0000 7f0db9600700 1 -- 172.21.15.62:0/3028782962 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7f0da801d070 con 0x7f0dbc10c660 2024-04-20T21:56:46.986 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:56:46.986 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T21:56:46.986 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:56:46.986 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:56:46.986 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T21:56:46.986 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:56:46.986 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:56:46.986 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T21:56:46.986 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 6 2024-04-20T21:56:46.986 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:56:46.986 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T21:56:46.986 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 2, 2024-04-20T21:56:46.986 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:56:46.987 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:56:46.987 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T21:56:46.987 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 2, 2024-04-20T21:56:46.987 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 12 2024-04-20T21:56:46.987 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T21:56:46.987 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:56:46.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.990+0000 7f0dc1e00700 1 -- 172.21.15.62:0/3028782962 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f0da406ecc0 msgr2=0x7f0da4071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:56:46.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.990+0000 7f0dc1e00700 1 --2- 172.21.15.62:0/3028782962 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f0da406ecc0 0x7f0da4071170 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f0db0000c00 tx=0x7f0db0005fd0 comp rx=0 tx=0).stop 2024-04-20T21:56:46.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.990+0000 7f0dc1e00700 1 -- 172.21.15.62:0/3028782962 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0dbc10c660 msgr2=0x7f0dbc19c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:56:46.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.990+0000 7f0dc1e00700 1 --2- 172.21.15.62:0/3028782962 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0dbc10c660 0x7f0dbc19c7c0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f0da800c8a0 tx=0x7f0da800cc60 comp rx=0 tx=0).stop 2024-04-20T21:56:46.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.990+0000 7f0dc1e00700 1 -- 172.21.15.62:0/3028782962 shutdown_connections 2024-04-20T21:56:46.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.990+0000 7f0dc1e00700 1 --2- 172.21.15.62:0/3028782962 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f0da406ecc0 0x7f0da4071170 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:56:46.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.990+0000 7f0dc1e00700 1 --2- 172.21.15.62:0/3028782962 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0dbc106b80 0x7f0dbc19c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:56:46.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.990+0000 7f0dc1e00700 1 --2- 172.21.15.62:0/3028782962 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0dbc10c660 0x7f0dbc19c7c0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:56:46.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.990+0000 7f0dc1e00700 1 -- 172.21.15.62:0/3028782962 >> 172.21.15.62:0/3028782962 conn(0x7f0dbc075e90 msgr2=0x7f0dbc077980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:56:46.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.990+0000 7f0dc1e00700 1 -- 172.21.15.62:0/3028782962 shutdown_connections 2024-04-20T21:56:46.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:46.991+0000 7f0dc1e00700 1 -- 172.21.15.62:0/3028782962 wait complete. 2024-04-20T21:56:47.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:47.152+0000 7efeaf600700 1 -- 172.21.15.62:0/2294838697 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efea8101000 msgr2=0x7efea81013d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:56:47.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:47.152+0000 7efeaf600700 1 --2- 172.21.15.62:0/2294838697 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efea8101000 0x7efea81013d0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7efe98009910 tx=0x7efe98009c20 comp rx=0 tx=0).stop 2024-04-20T21:56:47.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:47.152+0000 7efeaf600700 1 -- 172.21.15.62:0/2294838697 shutdown_connections 2024-04-20T21:56:47.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:47.152+0000 7efeaf600700 1 --2- 172.21.15.62:0/2294838697 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efea8068540 0x7efea80689b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:56:47.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:47.152+0000 7efeaf600700 1 --2- 172.21.15.62:0/2294838697 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efea8101000 0x7efea81013d0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:56:47.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:47.152+0000 7efeaf600700 1 -- 172.21.15.62:0/2294838697 >> 172.21.15.62:0/2294838697 conn(0x7efea8075130 msgr2=0x7efea8075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:56:47.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:47.152+0000 7efeaf600700 1 -- 172.21.15.62:0/2294838697 shutdown_connections 2024-04-20T21:56:47.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:47.152+0000 7efeaf600700 1 -- 172.21.15.62:0/2294838697 wait complete. 2024-04-20T21:56:47.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:47.153+0000 7efeaf600700 1 Processor -- start 2024-04-20T21:56:47.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:47.154+0000 7efeaf600700 1 -- start start 2024-04-20T21:56:47.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:47.154+0000 7efeaf600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efea8068540 0x7efea8101eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:56:47.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:47.154+0000 7efeaf600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efea8101000 0x7efea81023f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:56:47.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:47.154+0000 7efeaf600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efea8106040 con 0x7efea8101000 2024-04-20T21:56:47.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:47.154+0000 7efeaf600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efea8102930 con 0x7efea8068540 2024-04-20T21:56:47.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:47.155+0000 7efe9fe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efea8101000 0x7efea81023f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:56:47.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:47.155+0000 7efe9fe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efea8101000 0x7efea81023f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45396/0 (socket says 172.21.15.62:45396) 2024-04-20T21:56:47.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:47.155+0000 7efe9fe00700 1 -- 172.21.15.62:0/1547598451 learned_addr learned my addr 172.21.15.62:0/1547598451 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:56:47.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:47.155+0000 7efeacc00700 1 --2- 172.21.15.62:0/1547598451 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efea8068540 0x7efea8101eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:56:47.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:47.155+0000 7efe9fe00700 1 -- 172.21.15.62:0/1547598451 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efea8068540 msgr2=0x7efea8101eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:56:47.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:47.155+0000 7efe9fe00700 1 --2- 172.21.15.62:0/1547598451 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efea8068540 0x7efea8101eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:56:47.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:47.155+0000 7efe9fe00700 1 -- 172.21.15.62:0/1547598451 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe980095f0 con 0x7efea8101000 2024-04-20T21:56:47.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:47.156+0000 7efe9fe00700 1 --2- 172.21.15.62:0/1547598451 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efea8101000 0x7efea81023f0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7efe9000d8d0 tx=0x7efe9000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:56:47.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:47.156+0000 7efea6000700 1 -- 172.21.15.62:0/1547598451 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efe90009940 con 0x7efea8101000 2024-04-20T21:56:47.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:47.156+0000 7efea6000700 1 -- 172.21.15.62:0/1547598451 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efe90010460 con 0x7efea8101000 2024-04-20T21:56:47.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:47.156+0000 7efeaf600700 1 -- 172.21.15.62:0/1547598451 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efea8102c10 con 0x7efea8101000 2024-04-20T21:56:47.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:47.156+0000 7efea6000700 1 -- 172.21.15.62:0/1547598451 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efe9000f5f0 con 0x7efea8101000 2024-04-20T21:56:47.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:47.156+0000 7efeaf600700 1 -- 172.21.15.62:0/1547598451 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efea81a68a0 con 0x7efea8101000 2024-04-20T21:56:47.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:47.157+0000 7efeaf600700 1 -- 172.21.15.62:0/1547598451 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efea804f2a0 con 0x7efea8101000 2024-04-20T21:56:47.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:47.158+0000 7efea6000700 1 -- 172.21.15.62:0/1547598451 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7efe900105d0 con 0x7efea8101000 2024-04-20T21:56:47.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:47.159+0000 7efea6000700 1 --2- 172.21.15.62:0/1547598451 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7efe8c06ecc0 0x7efe8c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:56:47.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:47.159+0000 7efea6000700 1 -- 172.21.15.62:0/1547598451 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7efe9008ef10 con 0x7efea8101000 2024-04-20T21:56:47.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:47.159+0000 7efeacc00700 1 --2- 172.21.15.62:0/1547598451 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7efe8c06ecc0 0x7efe8c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:56:47.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:47.160+0000 7efeacc00700 1 --2- 172.21.15.62:0/1547598451 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7efe8c06ecc0 0x7efe8c071170 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7efe98006010 tx=0x7efe9801a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:56:47.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:47.164+0000 7efea6000700 1 -- 172.21.15.62:0/1547598451 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7efe90057c20 con 0x7efea8101000 2024-04-20T21:56:47.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:56:46 smithi189 ceph-mon[57022]: from='client.35180 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:56:47.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:56:46 smithi189 ceph-mon[57022]: from='client.35184 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:56:47.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:56:46 smithi062 ceph-mon[76322]: from='client.35180 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:56:47.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:56:46 smithi062 ceph-mon[76322]: from='client.35184 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:56:47.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:47.446+0000 7efeaf600700 1 -- 172.21.15.62:0/1547598451 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7efea81a6a90 con 0x7efea8101000 2024-04-20T21:56:47.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:47.447+0000 7efea6000700 1 -- 172.21.15.62:0/1547598451 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 25 v25) v1 ==== 76+0+1887 (secure 0 0 0) 0x7efe9005b2e0 con 0x7efea8101000 2024-04-20T21:56:47.447 INFO:teuthology.orchestra.run.smithi062.stdout:e25 2024-04-20T21:56:47.447 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T21:56:47.447 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T21:56:47.447 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T21:56:47.447 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:56:47.447 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T21:56:47.447 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T21:56:47.447 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 20 2024-04-20T21:56:47.447 INFO:teuthology.orchestra.run.smithi062.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-20T21:56:47.447 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T21:56:47.447 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T21:42:01.559061+0000 2024-04-20T21:56:47.448 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T21:56:47.448 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T21:56:47.448 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T21:56:47.448 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T21:56:47.448 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T21:56:47.448 INFO:teuthology.orchestra.run.smithi062.stdout:max_xattr_size 65536 2024-04-20T21:56:47.448 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T21:56:47.448 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T21:56:47.448 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 81 2024-04-20T21:56:47.448 INFO:teuthology.orchestra.run.smithi062.stdout:snap_rstat 1 2024-04-20T21:56:47.448 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T21:56:47.448 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 2 2024-04-20T21:56:47.448 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1 2024-04-20T21:56:47.448 INFO:teuthology.orchestra.run.smithi062.stdout:up {} 2024-04-20T21:56:47.449 INFO:teuthology.orchestra.run.smithi062.stdout:failed 0,1 2024-04-20T21:56:47.449 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T21:56:47.449 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T21:56:47.449 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T21:56:47.449 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T21:56:47.449 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data enabled 2024-04-20T21:56:47.449 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T21:56:47.449 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T21:56:47.449 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 1 2024-04-20T21:56:47.449 INFO:teuthology.orchestra.run.smithi062.stdout:qdb_cluster leader: 0 members: 2024-04-20T21:56:47.449 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:56:47.449 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:56:47.449 INFO:teuthology.orchestra.run.smithi062.stdout:Standby daemons: 2024-04-20T21:56:47.449 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:56:47.449 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ryqblb{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6826/4113916205,v1:172.21.15.62:6827/4113916205] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T21:56:47.450 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ozizwi{-1:34476} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6828/3801975165,v1:172.21.15.62:6829/3801975165] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T21:56:47.450 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.xbgspw{-1:44285} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6826/3996040526,v1:172.21.15.189:6827/3996040526] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:56:47.450 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.dvimer{-1:44289} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6824/1743111843,v1:172.21.15.189:6825/1743111843] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:56:47.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:47.450+0000 7efeaf600700 1 -- 172.21.15.62:0/1547598451 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7efe8c06ecc0 msgr2=0x7efe8c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:56:47.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:47.450+0000 7efeaf600700 1 --2- 172.21.15.62:0/1547598451 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7efe8c06ecc0 0x7efe8c071170 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7efe98006010 tx=0x7efe9801a040 comp rx=0 tx=0).stop 2024-04-20T21:56:47.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:47.450+0000 7efeaf600700 1 -- 172.21.15.62:0/1547598451 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efea8101000 msgr2=0x7efea81023f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:56:47.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:47.450+0000 7efeaf600700 1 --2- 172.21.15.62:0/1547598451 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efea8101000 0x7efea81023f0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7efe9000d8d0 tx=0x7efe9000dc90 comp rx=0 tx=0).stop 2024-04-20T21:56:47.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:47.450+0000 7efeaf600700 1 -- 172.21.15.62:0/1547598451 shutdown_connections 2024-04-20T21:56:47.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:47.450+0000 7efeaf600700 1 --2- 172.21.15.62:0/1547598451 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7efe8c06ecc0 0x7efe8c071170 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:56:47.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:47.450+0000 7efeaf600700 1 --2- 172.21.15.62:0/1547598451 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efea8068540 0x7efea8101eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:56:47.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:47.450+0000 7efeaf600700 1 --2- 172.21.15.62:0/1547598451 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efea8101000 0x7efea81023f0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:56:47.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:47.450+0000 7efeaf600700 1 -- 172.21.15.62:0/1547598451 >> 172.21.15.62:0/1547598451 conn(0x7efea8075130 msgr2=0x7efea80fdae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:56:47.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:47.451+0000 7efeaf600700 1 -- 172.21.15.62:0/1547598451 shutdown_connections 2024-04-20T21:56:47.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:47.451+0000 7efeaf600700 1 -- 172.21.15.62:0/1547598451 wait complete. 2024-04-20T21:56:47.451 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 25 2024-04-20T21:56:47.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:47.612+0000 7f0fe4400700 1 -- 172.21.15.62:0/4054817982 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0fdc100e80 msgr2=0x7f0fdc104e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:56:47.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:47.612+0000 7f0fe4400700 1 --2- 172.21.15.62:0/4054817982 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0fdc100e80 0x7f0fdc104e50 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f0fc8009910 tx=0x7f0fc8009c20 comp rx=0 tx=0).stop 2024-04-20T21:56:47.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:47.612+0000 7f0fe4400700 1 -- 172.21.15.62:0/4054817982 shutdown_connections 2024-04-20T21:56:47.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:47.612+0000 7f0fe4400700 1 --2- 172.21.15.62:0/4054817982 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0fdc100e80 0x7f0fdc104e50 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:56:47.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:47.612+0000 7f0fe4400700 1 --2- 172.21.15.62:0/4054817982 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0fdc1004e0 0x7f0fdc1008b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:56:47.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:47.612+0000 7f0fe4400700 1 -- 172.21.15.62:0/4054817982 >> 172.21.15.62:0/4054817982 conn(0x7f0fdc0fbda0 msgr2=0x7f0fdc0fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:56:47.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:47.613+0000 7f0fe4400700 1 -- 172.21.15.62:0/4054817982 shutdown_connections 2024-04-20T21:56:47.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:47.613+0000 7f0fe4400700 1 -- 172.21.15.62:0/4054817982 wait complete. 2024-04-20T21:56:47.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:47.614+0000 7f0fe4400700 1 Processor -- start 2024-04-20T21:56:47.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:47.614+0000 7f0fe4400700 1 -- start start 2024-04-20T21:56:47.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:47.615+0000 7f0fe4400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0fdc1004e0 0x7f0fdc073050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:56:47.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:47.615+0000 7f0fe4400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0fdc100e80 0x7f0fdc073590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:56:47.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:47.615+0000 7f0fe4400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fdc073ad0 con 0x7f0fdc1004e0 2024-04-20T21:56:47.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:47.615+0000 7f0fe4400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fdc073c10 con 0x7f0fdc100e80 2024-04-20T21:56:47.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:47.615+0000 7f0fe1a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0fdc1004e0 0x7f0fdc073050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:56:47.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:47.615+0000 7f0fe1a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0fdc1004e0 0x7f0fdc073050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45428/0 (socket says 172.21.15.62:45428) 2024-04-20T21:56:47.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:47.615+0000 7f0fe1a00700 1 -- 172.21.15.62:0/2523875729 learned_addr learned my addr 172.21.15.62:0/2523875729 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:56:47.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:47.615+0000 7f0fe1000700 1 --2- 172.21.15.62:0/2523875729 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0fdc100e80 0x7f0fdc073590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:56:47.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:47.615+0000 7f0fe1a00700 1 -- 172.21.15.62:0/2523875729 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0fdc100e80 msgr2=0x7f0fdc073590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:56:47.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:47.615+0000 7f0fe1a00700 1 --2- 172.21.15.62:0/2523875729 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0fdc100e80 0x7f0fdc073590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:56:47.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:47.615+0000 7f0fe1a00700 1 -- 172.21.15.62:0/2523875729 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fc80095f0 con 0x7f0fdc1004e0 2024-04-20T21:56:47.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:47.616+0000 7f0fe1a00700 1 --2- 172.21.15.62:0/2523875729 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0fdc1004e0 0x7f0fdc073050 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f0fd000d8d0 tx=0x7f0fd000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:56:47.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:47.616+0000 7f0fda000700 1 -- 172.21.15.62:0/2523875729 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0fd0009940 con 0x7f0fdc1004e0 2024-04-20T21:56:47.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:47.616+0000 7f0fda000700 1 -- 172.21.15.62:0/2523875729 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0fd0010460 con 0x7f0fdc1004e0 2024-04-20T21:56:47.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:47.616+0000 7f0fda000700 1 -- 172.21.15.62:0/2523875729 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0fd000f5f0 con 0x7f0fdc1004e0 2024-04-20T21:56:47.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:47.616+0000 7f0fe4400700 1 -- 172.21.15.62:0/2523875729 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fdc1a2100 con 0x7f0fdc1004e0 2024-04-20T21:56:47.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:47.616+0000 7f0fe4400700 1 -- 172.21.15.62:0/2523875729 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fdc1a2510 con 0x7f0fdc1004e0 2024-04-20T21:56:47.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:47.617+0000 7f0fe4400700 1 -- 172.21.15.62:0/2523875729 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fdc108850 con 0x7f0fdc1004e0 2024-04-20T21:56:47.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:47.619+0000 7f0fda000700 1 -- 172.21.15.62:0/2523875729 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f0fd00105d0 con 0x7f0fdc1004e0 2024-04-20T21:56:47.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:47.620+0000 7f0fda000700 1 --2- 172.21.15.62:0/2523875729 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f0fc406ecc0 0x7f0fc4071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:56:47.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:47.620+0000 7f0fda000700 1 -- 172.21.15.62:0/2523875729 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f0fd008ee10 con 0x7f0fdc1004e0 2024-04-20T21:56:47.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:47.620+0000 7f0fe1000700 1 --2- 172.21.15.62:0/2523875729 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f0fc406ecc0 0x7f0fc4071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:56:47.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:47.621+0000 7f0fe1000700 1 --2- 172.21.15.62:0/2523875729 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f0fc406ecc0 0x7f0fc4071170 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f0fdc074840 tx=0x7f0fc801a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:56:47.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:47.624+0000 7f0fda000700 1 -- 172.21.15.62:0/2523875729 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f0fd0057ba0 con 0x7f0fdc1004e0 2024-04-20T21:56:47.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:47.876+0000 7f0fe4400700 1 -- 172.21.15.62:0/2523875729 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0fdc061190 con 0x7f0fc406ecc0 2024-04-20T21:56:47.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:47.878+0000 7f0fda000700 1 -- 172.21.15.62:0/2523875729 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f0fdc061190 con 0x7f0fc406ecc0 2024-04-20T21:56:47.877 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:56:47.877 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T21:56:47.877 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T21:56:47.877 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-20T21:56:47.877 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T21:56:47.877 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr", 2024-04-20T21:56:47.877 INFO:teuthology.orchestra.run.smithi062.stdout: "crash", 2024-04-20T21:56:47.877 INFO:teuthology.orchestra.run.smithi062.stdout: "mon", 2024-04-20T21:56:47.877 INFO:teuthology.orchestra.run.smithi062.stdout: "osd" 2024-04-20T21:56:47.877 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T21:56:47.878 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "14/23 daemons upgraded", 2024-04-20T21:56:47.878 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading mds daemons", 2024-04-20T21:56:47.878 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T21:56:47.878 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:56:47.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:47.881+0000 7f0fe4400700 1 -- 172.21.15.62:0/2523875729 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f0fc406ecc0 msgr2=0x7f0fc4071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:56:47.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:47.881+0000 7f0fe4400700 1 --2- 172.21.15.62:0/2523875729 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f0fc406ecc0 0x7f0fc4071170 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f0fdc074840 tx=0x7f0fc801a040 comp rx=0 tx=0).stop 2024-04-20T21:56:47.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:47.881+0000 7f0fe4400700 1 -- 172.21.15.62:0/2523875729 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0fdc1004e0 msgr2=0x7f0fdc073050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:56:47.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:47.881+0000 7f0fe4400700 1 --2- 172.21.15.62:0/2523875729 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0fdc1004e0 0x7f0fdc073050 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f0fd000d8d0 tx=0x7f0fd000dc90 comp rx=0 tx=0).stop 2024-04-20T21:56:47.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:47.881+0000 7f0fe4400700 1 -- 172.21.15.62:0/2523875729 shutdown_connections 2024-04-20T21:56:47.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:47.881+0000 7f0fe4400700 1 --2- 172.21.15.62:0/2523875729 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f0fc406ecc0 0x7f0fc4071170 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:56:47.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:47.881+0000 7f0fe4400700 1 --2- 172.21.15.62:0/2523875729 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0fdc1004e0 0x7f0fdc073050 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:56:47.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:47.881+0000 7f0fe4400700 1 --2- 172.21.15.62:0/2523875729 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0fdc100e80 0x7f0fdc073590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:56:47.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:47.881+0000 7f0fe4400700 1 -- 172.21.15.62:0/2523875729 >> 172.21.15.62:0/2523875729 conn(0x7f0fdc0fbda0 msgr2=0x7f0fdc0fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:56:47.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:47.881+0000 7f0fe4400700 1 -- 172.21.15.62:0/2523875729 shutdown_connections 2024-04-20T21:56:47.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:47.881+0000 7f0fe4400700 1 -- 172.21.15.62:0/2523875729 wait complete. 2024-04-20T21:56:48.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:48.041+0000 7f4d01600700 1 -- 172.21.15.62:0/3899049055 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4cfc102470 msgr2=0x7f4cfc1028e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:56:48.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:48.041+0000 7f4d01600700 1 --2- 172.21.15.62:0/3899049055 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4cfc102470 0x7f4cfc1028e0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f4ce8009910 tx=0x7f4ce8009c20 comp rx=0 tx=0).stop 2024-04-20T21:56:48.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:48.041+0000 7f4d01600700 1 -- 172.21.15.62:0/3899049055 shutdown_connections 2024-04-20T21:56:48.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:48.041+0000 7f4d01600700 1 --2- 172.21.15.62:0/3899049055 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4cfc102470 0x7f4cfc1028e0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:56:48.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:48.041+0000 7f4d01600700 1 --2- 172.21.15.62:0/3899049055 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4cfc108470 0x7f4cfc108840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:56:48.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:48.041+0000 7f4d01600700 1 -- 172.21.15.62:0/3899049055 >> 172.21.15.62:0/3899049055 conn(0x7f4cfc0fdf90 msgr2=0x7f4cfc1003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:56:48.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:48.041+0000 7f4d01600700 1 -- 172.21.15.62:0/3899049055 shutdown_connections 2024-04-20T21:56:48.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:48.041+0000 7f4d01600700 1 -- 172.21.15.62:0/3899049055 wait complete. 2024-04-20T21:56:48.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:48.043+0000 7f4d01600700 1 Processor -- start 2024-04-20T21:56:48.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:48.043+0000 7f4d01600700 1 -- start start 2024-04-20T21:56:48.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:48.043+0000 7f4d01600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4cfc102470 0x7f4cfc1981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:56:48.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:48.043+0000 7f4d01600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4cfc108470 0x7f4cfc198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:56:48.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:48.043+0000 7f4d01600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cfc198e10 con 0x7f4cfc108470 2024-04-20T21:56:48.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:48.043+0000 7f4d01600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cfc19cba0 con 0x7f4cfc102470 2024-04-20T21:56:48.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:48.043+0000 7f4cfa000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4cfc108470 0x7f4cfc198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:56:48.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:48.044+0000 7f4cfa000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4cfc108470 0x7f4cfc198730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45444/0 (socket says 172.21.15.62:45444) 2024-04-20T21:56:48.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:48.044+0000 7f4cfa000700 1 -- 172.21.15.62:0/4131507067 learned_addr learned my addr 172.21.15.62:0/4131507067 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:56:48.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:48.044+0000 7f4cfaa00700 1 --2- 172.21.15.62:0/4131507067 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4cfc102470 0x7f4cfc1981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:56:48.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:48.044+0000 7f4cfa000700 1 -- 172.21.15.62:0/4131507067 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4cfc102470 msgr2=0x7f4cfc1981f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:56:48.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:48.044+0000 7f4cfa000700 1 --2- 172.21.15.62:0/4131507067 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4cfc102470 0x7f4cfc1981f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:56:48.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:48.044+0000 7f4cfa000700 1 -- 172.21.15.62:0/4131507067 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ce80095f0 con 0x7f4cfc108470 2024-04-20T21:56:48.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:48.044+0000 7f4cfa000700 1 --2- 172.21.15.62:0/4131507067 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4cfc108470 0x7f4cfc198730 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f4ce800b5c0 tx=0x7f4ce800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:56:48.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:48.045+0000 7f4cf3400700 1 -- 172.21.15.62:0/4131507067 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4ce801d070 con 0x7f4cfc108470 2024-04-20T21:56:48.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:48.045+0000 7f4cf3400700 1 -- 172.21.15.62:0/4131507067 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4ce8005e00 con 0x7f4cfc108470 2024-04-20T21:56:48.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:48.045+0000 7f4cf3400700 1 -- 172.21.15.62:0/4131507067 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4ce8017400 con 0x7f4cfc108470 2024-04-20T21:56:48.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:48.045+0000 7f4d01600700 1 -- 172.21.15.62:0/4131507067 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cfc19ce20 con 0x7f4cfc108470 2024-04-20T21:56:48.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:48.045+0000 7f4d01600700 1 -- 172.21.15.62:0/4131507067 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cfc19d310 con 0x7f4cfc108470 2024-04-20T21:56:48.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:48.046+0000 7f4d01600700 1 -- 172.21.15.62:0/4131507067 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cfc10a970 con 0x7f4cfc108470 2024-04-20T21:56:48.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:48.047+0000 7f4cf3400700 1 -- 172.21.15.62:0/4131507067 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f4ce800f460 con 0x7f4cfc108470 2024-04-20T21:56:48.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:48.048+0000 7f4cf3400700 1 --2- 172.21.15.62:0/4131507067 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f4ce006ecc0 0x7f4ce0071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:56:48.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:48.048+0000 7f4cf3400700 1 -- 172.21.15.62:0/4131507067 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f4ce8090340 con 0x7f4cfc108470 2024-04-20T21:56:48.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:48.048+0000 7f4cfaa00700 1 --2- 172.21.15.62:0/4131507067 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f4ce006ecc0 0x7f4ce0071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:56:48.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:48.049+0000 7f4cfaa00700 1 --2- 172.21.15.62:0/4131507067 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f4ce006ecc0 0x7f4ce0071170 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f4cec005950 tx=0x7f4cec0058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:56:48.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:48.053+0000 7f4cf3400700 1 -- 172.21.15.62:0/4131507067 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f4ce80590d0 con 0x7f4cfc108470 2024-04-20T21:56:48.143 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:56:47 smithi062 ceph-mon[76322]: from='client.35188 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:56:48.144 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:56:47 smithi062 ceph-mon[76322]: pgmap v644: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:56:48.144 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:56:47 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/3028782962' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:56:48.144 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:56:47 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/1547598451' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T21:56:48.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:56:47 smithi189 ceph-mon[57022]: from='client.35188 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:56:48.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:56:47 smithi189 ceph-mon[57022]: pgmap v644: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:56:48.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:56:47 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/3028782962' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:56:48.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:56:47 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/1547598451' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T21:56:48.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:48.382+0000 7f4d01600700 1 -- 172.21.15.62:0/4131507067 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f4cfc04f2a0 con 0x7f4cfc108470 2024-04-20T21:56:48.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:48.383+0000 7f4cf3400700 1 -- 172.21.15.62:0/4131507067 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7f4ce805c790 con 0x7f4cfc108470 2024-04-20T21:56:48.382 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-20T21:56:48.382 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-20T21:56:48.382 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is degraded 2024-04-20T21:56:48.382 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T21:56:48.382 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T21:56:48.382 INFO:teuthology.orchestra.run.smithi062.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-20T21:56:48.382 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-20T21:56:48.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:48.386+0000 7f4d01600700 1 -- 172.21.15.62:0/4131507067 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f4ce006ecc0 msgr2=0x7f4ce0071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:56:48.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:48.386+0000 7f4d01600700 1 --2- 172.21.15.62:0/4131507067 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f4ce006ecc0 0x7f4ce0071170 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f4cec005950 tx=0x7f4cec0058e0 comp rx=0 tx=0).stop 2024-04-20T21:56:48.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:48.386+0000 7f4d01600700 1 -- 172.21.15.62:0/4131507067 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4cfc108470 msgr2=0x7f4cfc198730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:56:48.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:48.386+0000 7f4d01600700 1 --2- 172.21.15.62:0/4131507067 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4cfc108470 0x7f4cfc198730 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f4ce800b5c0 tx=0x7f4ce800be20 comp rx=0 tx=0).stop 2024-04-20T21:56:48.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:48.386+0000 7f4d01600700 1 -- 172.21.15.62:0/4131507067 shutdown_connections 2024-04-20T21:56:48.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:48.386+0000 7f4d01600700 1 --2- 172.21.15.62:0/4131507067 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f4ce006ecc0 0x7f4ce0071170 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:56:48.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:48.386+0000 7f4d01600700 1 --2- 172.21.15.62:0/4131507067 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4cfc102470 0x7f4cfc1981f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:56:48.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:48.386+0000 7f4d01600700 1 --2- 172.21.15.62:0/4131507067 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4cfc108470 0x7f4cfc198730 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:56:48.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:48.386+0000 7f4d01600700 1 -- 172.21.15.62:0/4131507067 >> 172.21.15.62:0/4131507067 conn(0x7f4cfc0fdf90 msgr2=0x7f4cfc0ff3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:56:48.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:48.386+0000 7f4d01600700 1 -- 172.21.15.62:0/4131507067 shutdown_connections 2024-04-20T21:56:48.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:56:48.387+0000 7f4d01600700 1 -- 172.21.15.62:0/4131507067 wait complete. 2024-04-20T21:56:49.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:56:48 smithi189 ceph-mon[57022]: from='client.35200 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:56:49.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:56:48 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/4131507067' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T21:56:49.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:56:48 smithi062 ceph-mon[76322]: from='client.35200 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:56:49.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:56:48 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/4131507067' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T21:56:50.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:56:49 smithi189 ceph-mon[57022]: pgmap v645: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:56:50.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:56:49 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:56:50.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:56:49 smithi062 ceph-mon[76322]: pgmap v645: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:56:50.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:56:49 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:56:52.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:56:51 smithi189 ceph-mon[57022]: pgmap v646: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:56:52.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:56:51 smithi062 ceph-mon[76322]: pgmap v646: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:56:54.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:56:53 smithi189 ceph-mon[57022]: pgmap v647: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:56:54.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:56:53 smithi062 ceph-mon[76322]: pgmap v647: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:56:56.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:56:55 smithi189 ceph-mon[57022]: pgmap v648: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:56:56.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:56:55 smithi062 ceph-mon[76322]: pgmap v648: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:56:58.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:56:57 smithi189 ceph-mon[57022]: pgmap v649: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:56:58.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:56:57 smithi062 ceph-mon[76322]: pgmap v649: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:57:00.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:56:59 smithi189 ceph-mon[57022]: pgmap v650: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:57:00.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:56:59 smithi062 ceph-mon[76322]: pgmap v650: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:57:02.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:57:01 smithi062 ceph-mon[76322]: pgmap v651: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:57:02.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:57:01 smithi189 ceph-mon[57022]: pgmap v651: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:57:04.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:57:03 smithi189 ceph-mon[57022]: pgmap v652: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:57:04.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:57:03 smithi062 ceph-mon[76322]: pgmap v652: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:57:05.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:57:05 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:57:05.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:57:05 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:57:06.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:57:06 smithi062 ceph-mon[76322]: pgmap v653: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:57:06.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:57:06 smithi189 ceph-mon[57022]: pgmap v653: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:57:08.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:57:08 smithi062 ceph-mon[76322]: pgmap v654: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:57:08.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:57:08 smithi189 ceph-mon[57022]: pgmap v654: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:57:10.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:57:10 smithi062 ceph-mon[76322]: pgmap v655: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:57:10.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:57:10 smithi189 ceph-mon[57022]: pgmap v655: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:57:12.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:57:12 smithi062 ceph-mon[76322]: pgmap v656: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:57:12.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:57:12 smithi189 ceph-mon[57022]: pgmap v656: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:57:14.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:57:14 smithi189 ceph-mon[57022]: pgmap v657: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:57:14.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:57:14 smithi062 ceph-mon[76322]: pgmap v657: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:57:16.123 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:57:16 smithi062 ceph-mon[76322]: pgmap v658: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:57:16.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:57:16 smithi189 ceph-mon[57022]: pgmap v658: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:57:18.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:57:18 smithi189 ceph-mon[57022]: pgmap v659: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:57:18.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:57:18 smithi062 ceph-mon[76322]: pgmap v659: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:57:18.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:18.562+0000 7fb13f400700 1 -- 172.21.15.62:0/2857287146 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb138108470 msgr2=0x7fb138108840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:57:18.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:18.562+0000 7fb13f400700 1 --2- 172.21.15.62:0/2857287146 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb138108470 0x7fb138108840 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fb12c009910 tx=0x7fb12c009c20 comp rx=0 tx=0).stop 2024-04-20T21:57:18.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:18.562+0000 7fb13f400700 1 -- 172.21.15.62:0/2857287146 shutdown_connections 2024-04-20T21:57:18.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:18.562+0000 7fb13f400700 1 --2- 172.21.15.62:0/2857287146 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb138102470 0x7fb1381028e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:57:18.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:18.562+0000 7fb13f400700 1 --2- 172.21.15.62:0/2857287146 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb138108470 0x7fb138108840 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:57:18.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:18.562+0000 7fb13f400700 1 -- 172.21.15.62:0/2857287146 >> 172.21.15.62:0/2857287146 conn(0x7fb1380fdf90 msgr2=0x7fb1381003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:57:18.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:18.562+0000 7fb13f400700 1 -- 172.21.15.62:0/2857287146 shutdown_connections 2024-04-20T21:57:18.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:18.562+0000 7fb13f400700 1 -- 172.21.15.62:0/2857287146 wait complete. 2024-04-20T21:57:18.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:18.563+0000 7fb13f400700 1 Processor -- start 2024-04-20T21:57:18.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:18.563+0000 7fb13f400700 1 -- start start 2024-04-20T21:57:18.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:18.564+0000 7fb13f400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb138102470 0x7fb1381981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:57:18.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:18.564+0000 7fb13f400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb138108470 0x7fb138198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:57:18.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:18.564+0000 7fb13f400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb138198e10 con 0x7fb138108470 2024-04-20T21:57:18.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:18.564+0000 7fb13f400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb13819cba0 con 0x7fb138102470 2024-04-20T21:57:18.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:18.564+0000 7fb137e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb138108470 0x7fb138198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:57:18.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:18.564+0000 7fb137e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb138108470 0x7fb138198730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50040/0 (socket says 172.21.15.62:50040) 2024-04-20T21:57:18.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:18.564+0000 7fb137e00700 1 -- 172.21.15.62:0/2770771417 learned_addr learned my addr 172.21.15.62:0/2770771417 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:57:18.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:18.564+0000 7fb13ca00700 1 --2- 172.21.15.62:0/2770771417 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb138102470 0x7fb1381981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:57:18.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:18.565+0000 7fb137e00700 1 -- 172.21.15.62:0/2770771417 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb138102470 msgr2=0x7fb1381981f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:57:18.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:18.565+0000 7fb137e00700 1 --2- 172.21.15.62:0/2770771417 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb138102470 0x7fb1381981f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:57:18.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:18.565+0000 7fb137e00700 1 -- 172.21.15.62:0/2770771417 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb12c0095f0 con 0x7fb138108470 2024-04-20T21:57:18.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:18.565+0000 7fb137e00700 1 --2- 172.21.15.62:0/2770771417 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb138108470 0x7fb138198730 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fb12400d8d0 tx=0x7fb12400dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:57:18.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:18.565+0000 7fb135600700 1 -- 172.21.15.62:0/2770771417 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb1240098a0 con 0x7fb138108470 2024-04-20T21:57:18.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:18.565+0000 7fb135600700 1 -- 172.21.15.62:0/2770771417 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb124010460 con 0x7fb138108470 2024-04-20T21:57:18.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:18.565+0000 7fb135600700 1 -- 172.21.15.62:0/2770771417 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb12400f5f0 con 0x7fb138108470 2024-04-20T21:57:18.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:18.565+0000 7fb13f400700 1 -- 172.21.15.62:0/2770771417 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb13819ce80 con 0x7fb138108470 2024-04-20T21:57:18.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:18.566+0000 7fb13f400700 1 -- 172.21.15.62:0/2770771417 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb13819d3d0 con 0x7fb138108470 2024-04-20T21:57:18.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:18.567+0000 7fb13f400700 1 -- 172.21.15.62:0/2770771417 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb13810a9e0 con 0x7fb138108470 2024-04-20T21:57:18.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:18.568+0000 7fb135600700 1 -- 172.21.15.62:0/2770771417 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fb124009a00 con 0x7fb138108470 2024-04-20T21:57:18.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:18.569+0000 7fb135600700 1 --2- 172.21.15.62:0/2770771417 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb12006ecc0 0x7fb120071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:57:18.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:18.569+0000 7fb135600700 1 -- 172.21.15.62:0/2770771417 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fb12408ecd0 con 0x7fb138108470 2024-04-20T21:57:18.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:18.569+0000 7fb13ca00700 1 --2- 172.21.15.62:0/2770771417 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb12006ecc0 0x7fb120071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:57:18.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:18.570+0000 7fb13ca00700 1 --2- 172.21.15.62:0/2770771417 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb12006ecc0 0x7fb120071170 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fb12c006010 tx=0x7fb12c01a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:57:18.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:18.574+0000 7fb135600700 1 -- 172.21.15.62:0/2770771417 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fb124057a60 con 0x7fb138108470 2024-04-20T21:57:18.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:18.831+0000 7fb13f400700 1 -- 172.21.15.62:0/2770771417 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb13819d640 con 0x7fb12006ecc0 2024-04-20T21:57:18.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:18.835+0000 7fb135600700 1 -- 172.21.15.62:0/2770771417 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fb13819d640 con 0x7fb12006ecc0 2024-04-20T21:57:18.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:18.838+0000 7fb13f400700 1 -- 172.21.15.62:0/2770771417 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb12006ecc0 msgr2=0x7fb120071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:57:18.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:18.838+0000 7fb13f400700 1 --2- 172.21.15.62:0/2770771417 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb12006ecc0 0x7fb120071170 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fb12c006010 tx=0x7fb12c01a040 comp rx=0 tx=0).stop 2024-04-20T21:57:18.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:18.839+0000 7fb13f400700 1 -- 172.21.15.62:0/2770771417 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb138108470 msgr2=0x7fb138198730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:57:18.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:18.839+0000 7fb13f400700 1 --2- 172.21.15.62:0/2770771417 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb138108470 0x7fb138198730 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fb12400d8d0 tx=0x7fb12400dbe0 comp rx=0 tx=0).stop 2024-04-20T21:57:18.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:18.839+0000 7fb13f400700 1 -- 172.21.15.62:0/2770771417 shutdown_connections 2024-04-20T21:57:18.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:18.839+0000 7fb13f400700 1 --2- 172.21.15.62:0/2770771417 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb12006ecc0 0x7fb120071170 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:57:18.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:18.839+0000 7fb13f400700 1 --2- 172.21.15.62:0/2770771417 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb138102470 0x7fb1381981f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:57:18.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:18.839+0000 7fb13f400700 1 --2- 172.21.15.62:0/2770771417 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb138108470 0x7fb138198730 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:57:18.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:18.839+0000 7fb13f400700 1 -- 172.21.15.62:0/2770771417 >> 172.21.15.62:0/2770771417 conn(0x7fb1380fdf90 msgr2=0x7fb1381002c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:57:18.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:18.839+0000 7fb13f400700 1 -- 172.21.15.62:0/2770771417 shutdown_connections 2024-04-20T21:57:18.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:18.839+0000 7fb13f400700 1 -- 172.21.15.62:0/2770771417 wait complete. 2024-04-20T21:57:18.855 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T21:57:19.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.006+0000 7efd63200700 1 -- 172.21.15.62:0/3138410011 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efd5c100e80 msgr2=0x7efd5c104e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:57:19.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.006+0000 7efd63200700 1 --2- 172.21.15.62:0/3138410011 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efd5c100e80 0x7efd5c104e50 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7efd48009910 tx=0x7efd48009c20 comp rx=0 tx=0).stop 2024-04-20T21:57:19.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.006+0000 7efd63200700 1 -- 172.21.15.62:0/3138410011 shutdown_connections 2024-04-20T21:57:19.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.006+0000 7efd63200700 1 --2- 172.21.15.62:0/3138410011 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efd5c100e80 0x7efd5c104e50 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:57:19.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.006+0000 7efd63200700 1 --2- 172.21.15.62:0/3138410011 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd5c1004e0 0x7efd5c1008b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:57:19.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.006+0000 7efd63200700 1 -- 172.21.15.62:0/3138410011 >> 172.21.15.62:0/3138410011 conn(0x7efd5c0fbda0 msgr2=0x7efd5c0fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:57:19.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.006+0000 7efd63200700 1 -- 172.21.15.62:0/3138410011 shutdown_connections 2024-04-20T21:57:19.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.007+0000 7efd63200700 1 -- 172.21.15.62:0/3138410011 wait complete. 2024-04-20T21:57:19.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.008+0000 7efd63200700 1 Processor -- start 2024-04-20T21:57:19.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.008+0000 7efd63200700 1 -- start start 2024-04-20T21:57:19.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.009+0000 7efd63200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd5c1004e0 0x7efd5c198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:57:19.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.009+0000 7efd63200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efd5c100e80 0x7efd5c1985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:57:19.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.009+0000 7efd63200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd5c198ca0 con 0x7efd5c100e80 2024-04-20T21:57:19.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.009+0000 7efd63200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd5c19ca30 con 0x7efd5c1004e0 2024-04-20T21:57:19.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.009+0000 7efd5b400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efd5c100e80 0x7efd5c1985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:57:19.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.009+0000 7efd5b400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efd5c100e80 0x7efd5c1985c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50064/0 (socket says 172.21.15.62:50064) 2024-04-20T21:57:19.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.009+0000 7efd5b400700 1 -- 172.21.15.62:0/2116816631 learned_addr learned my addr 172.21.15.62:0/2116816631 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:57:19.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.010+0000 7efd5be00700 1 --2- 172.21.15.62:0/2116816631 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd5c1004e0 0x7efd5c198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:57:19.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.010+0000 7efd5b400700 1 -- 172.21.15.62:0/2116816631 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd5c1004e0 msgr2=0x7efd5c198080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:57:19.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.010+0000 7efd5b400700 1 --2- 172.21.15.62:0/2116816631 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd5c1004e0 0x7efd5c198080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:57:19.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.010+0000 7efd5b400700 1 -- 172.21.15.62:0/2116816631 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd480095f0 con 0x7efd5c100e80 2024-04-20T21:57:19.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.010+0000 7efd5b400700 1 --2- 172.21.15.62:0/2116816631 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efd5c100e80 0x7efd5c1985c0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7efd4800bda0 tx=0x7efd4800be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:57:19.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.010+0000 7efd58c00700 1 -- 172.21.15.62:0/2116816631 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efd4801d070 con 0x7efd5c100e80 2024-04-20T21:57:19.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.011+0000 7efd58c00700 1 -- 172.21.15.62:0/2116816631 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efd4800f460 con 0x7efd5c100e80 2024-04-20T21:57:19.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.011+0000 7efd58c00700 1 -- 172.21.15.62:0/2116816631 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efd48017610 con 0x7efd5c100e80 2024-04-20T21:57:19.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.011+0000 7efd63200700 1 -- 172.21.15.62:0/2116816631 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd5c19ccb0 con 0x7efd5c100e80 2024-04-20T21:57:19.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.011+0000 7efd63200700 1 -- 172.21.15.62:0/2116816631 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd5c19d1a0 con 0x7efd5c100e80 2024-04-20T21:57:19.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.012+0000 7efd63200700 1 -- 172.21.15.62:0/2116816631 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd5c1045d0 con 0x7efd5c100e80 2024-04-20T21:57:19.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.013+0000 7efd58c00700 1 -- 172.21.15.62:0/2116816631 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7efd4800fa40 con 0x7efd5c100e80 2024-04-20T21:57:19.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.014+0000 7efd58c00700 1 --2- 172.21.15.62:0/2116816631 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7efd4006ecc0 0x7efd40071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:57:19.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.014+0000 7efd58c00700 1 -- 172.21.15.62:0/2116816631 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7efd48090390 con 0x7efd5c100e80 2024-04-20T21:57:19.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.015+0000 7efd5be00700 1 --2- 172.21.15.62:0/2116816631 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7efd4006ecc0 0x7efd40071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:57:19.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.016+0000 7efd5be00700 1 --2- 172.21.15.62:0/2116816631 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7efd4006ecc0 0x7efd40071170 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7efd4c005950 tx=0x7efd4c0058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:57:19.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.019+0000 7efd58c00700 1 -- 172.21.15.62:0/2116816631 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7efd480590a0 con 0x7efd5c100e80 2024-04-20T21:57:19.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.276+0000 7efd63200700 1 -- 172.21.15.62:0/2116816631 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7efd5c19d480 con 0x7efd4006ecc0 2024-04-20T21:57:19.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.279+0000 7efd58c00700 1 -- 172.21.15.62:0/2116816631 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7efd5c19d480 con 0x7efd4006ecc0 2024-04-20T21:57:19.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.282+0000 7efd63200700 1 -- 172.21.15.62:0/2116816631 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7efd4006ecc0 msgr2=0x7efd40071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:57:19.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.282+0000 7efd63200700 1 --2- 172.21.15.62:0/2116816631 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7efd4006ecc0 0x7efd40071170 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7efd4c005950 tx=0x7efd4c0058e0 comp rx=0 tx=0).stop 2024-04-20T21:57:19.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.282+0000 7efd63200700 1 -- 172.21.15.62:0/2116816631 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efd5c100e80 msgr2=0x7efd5c1985c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:57:19.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.282+0000 7efd63200700 1 --2- 172.21.15.62:0/2116816631 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efd5c100e80 0x7efd5c1985c0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7efd4800bda0 tx=0x7efd4800be80 comp rx=0 tx=0).stop 2024-04-20T21:57:19.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.282+0000 7efd63200700 1 -- 172.21.15.62:0/2116816631 shutdown_connections 2024-04-20T21:57:19.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.282+0000 7efd63200700 1 --2- 172.21.15.62:0/2116816631 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7efd4006ecc0 0x7efd40071170 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:57:19.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.282+0000 7efd63200700 1 --2- 172.21.15.62:0/2116816631 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd5c1004e0 0x7efd5c198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:57:19.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.282+0000 7efd63200700 1 --2- 172.21.15.62:0/2116816631 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efd5c100e80 0x7efd5c1985c0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:57:19.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.282+0000 7efd63200700 1 -- 172.21.15.62:0/2116816631 >> 172.21.15.62:0/2116816631 conn(0x7efd5c0fbda0 msgr2=0x7efd5c0fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:57:19.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.282+0000 7efd63200700 1 -- 172.21.15.62:0/2116816631 shutdown_connections 2024-04-20T21:57:19.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.282+0000 7efd63200700 1 -- 172.21.15.62:0/2116816631 wait complete. 2024-04-20T21:57:19.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.454+0000 7fb759200700 1 -- 172.21.15.62:0/2533504796 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb754101070 msgr2=0x7fb754101440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:57:19.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.454+0000 7fb759200700 1 --2- 172.21.15.62:0/2533504796 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb754101070 0x7fb754101440 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fb744009910 tx=0x7fb744009c20 comp rx=0 tx=0).stop 2024-04-20T21:57:19.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.454+0000 7fb759200700 1 -- 172.21.15.62:0/2533504796 shutdown_connections 2024-04-20T21:57:19.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.454+0000 7fb759200700 1 --2- 172.21.15.62:0/2533504796 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb754068540 0x7fb7540689b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:57:19.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.454+0000 7fb759200700 1 --2- 172.21.15.62:0/2533504796 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb754101070 0x7fb754101440 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:57:19.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.454+0000 7fb759200700 1 -- 172.21.15.62:0/2533504796 >> 172.21.15.62:0/2533504796 conn(0x7fb754075130 msgr2=0x7fb754075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:57:19.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.455+0000 7fb759200700 1 -- 172.21.15.62:0/2533504796 shutdown_connections 2024-04-20T21:57:19.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.455+0000 7fb759200700 1 -- 172.21.15.62:0/2533504796 wait complete. 2024-04-20T21:57:19.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.456+0000 7fb759200700 1 Processor -- start 2024-04-20T21:57:19.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.456+0000 7fb759200700 1 -- start start 2024-04-20T21:57:19.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.456+0000 7fb759200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb754068540 0x7fb754198230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:57:19.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.456+0000 7fb759200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb754101070 0x7fb754198770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:57:19.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.456+0000 7fb759200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb754198e50 con 0x7fb754101070 2024-04-20T21:57:19.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.456+0000 7fb759200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb75419cbe0 con 0x7fb754068540 2024-04-20T21:57:19.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.457+0000 7fb751400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb754101070 0x7fb754198770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:57:19.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.457+0000 7fb751400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb754101070 0x7fb754198770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50078/0 (socket says 172.21.15.62:50078) 2024-04-20T21:57:19.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.457+0000 7fb751400700 1 -- 172.21.15.62:0/2952800057 learned_addr learned my addr 172.21.15.62:0/2952800057 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:57:19.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.457+0000 7fb751e00700 1 --2- 172.21.15.62:0/2952800057 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb754068540 0x7fb754198230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:57:19.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.457+0000 7fb751400700 1 -- 172.21.15.62:0/2952800057 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb754068540 msgr2=0x7fb754198230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:57:19.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.457+0000 7fb751400700 1 --2- 172.21.15.62:0/2952800057 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb754068540 0x7fb754198230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:57:19.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.457+0000 7fb751400700 1 -- 172.21.15.62:0/2952800057 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7440095f0 con 0x7fb754101070 2024-04-20T21:57:19.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.457+0000 7fb751400700 1 --2- 172.21.15.62:0/2952800057 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb754101070 0x7fb754198770 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fb74000b4b0 tx=0x7fb74000b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:57:19.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.458+0000 7fb74aa00700 1 -- 172.21.15.62:0/2952800057 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb740011800 con 0x7fb754101070 2024-04-20T21:57:19.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.458+0000 7fb74aa00700 1 -- 172.21.15.62:0/2952800057 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb740011e20 con 0x7fb754101070 2024-04-20T21:57:19.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.458+0000 7fb74aa00700 1 -- 172.21.15.62:0/2952800057 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb74000f2e0 con 0x7fb754101070 2024-04-20T21:57:19.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.458+0000 7fb759200700 1 -- 172.21.15.62:0/2952800057 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb75419cec0 con 0x7fb754101070 2024-04-20T21:57:19.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.458+0000 7fb759200700 1 -- 172.21.15.62:0/2952800057 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb75419d410 con 0x7fb754101070 2024-04-20T21:57:19.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.459+0000 7fb759200700 1 -- 172.21.15.62:0/2952800057 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb75410a9b0 con 0x7fb754101070 2024-04-20T21:57:19.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.460+0000 7fb74aa00700 1 -- 172.21.15.62:0/2952800057 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fb740010380 con 0x7fb754101070 2024-04-20T21:57:19.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.461+0000 7fb74aa00700 1 --2- 172.21.15.62:0/2952800057 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb73806ecc0 0x7fb738071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:57:19.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.461+0000 7fb74aa00700 1 -- 172.21.15.62:0/2952800057 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fb74008ea30 con 0x7fb754101070 2024-04-20T21:57:19.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.461+0000 7fb751e00700 1 --2- 172.21.15.62:0/2952800057 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb73806ecc0 0x7fb738071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:57:19.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.462+0000 7fb751e00700 1 --2- 172.21.15.62:0/2952800057 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb73806ecc0 0x7fb738071170 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fb744006010 tx=0x7fb74401a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:57:19.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.466+0000 7fb74aa00700 1 -- 172.21.15.62:0/2952800057 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fb740057870 con 0x7fb754101070 2024-04-20T21:57:19.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.713+0000 7fb759200700 1 -- 172.21.15.62:0/2952800057 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fb75419d6e0 con 0x7fb73806ecc0 2024-04-20T21:57:19.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.731+0000 7fb74aa00700 1 -- 172.21.15.62:0/2952800057 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fb75419d6e0 con 0x7fb73806ecc0 2024-04-20T21:57:19.731 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T21:57:19.731 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (23m) 14m ago 37m 14.2M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T21:57:19.731 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (37m) 14m ago 37m 9512k - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T21:57:19.731 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (36m) 15m ago 35m 8942k - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T21:57:19.731 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (19m) 14m ago 37m 6660k - 19.0.0-3133-ga8c84f59 76bc6b91b726 2825d5ba11c6 2024-04-20T21:57:19.731 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (19m) 15m ago 35m 6665k - 19.0.0-3133-ga8c84f59 76bc6b91b726 20f746fa1bba 2024-04-20T21:57:19.731 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (22m) 14m ago 36m 74.2M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T21:57:19.731 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (14m) 14m ago 29m 11.2M - 19.0.0-3133-ga8c84f59 76bc6b91b726 de80bc90a683 2024-04-20T21:57:19.731 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (15m) 14m ago 29m 11.9M - 19.0.0-3133-ga8c84f59 76bc6b91b726 cd793b7ee98e 2024-04-20T21:57:19.731 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (29m) 15m ago 29m 117M - 18.2.0 dc2bc1663786 fb2a8a223db8 2024-04-20T21:57:19.731 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (29m) 15m ago 29m 55.6M - 18.2.0 dc2bc1663786 f2eec2ee8951 2024-04-20T21:57:19.732 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (26m) 14m ago 38m 511M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T21:57:19.732 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (24m) 15m ago 35m 452M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T21:57:19.732 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (20m) 14m ago 38m 55.9M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 019eb1479485 2024-04-20T21:57:19.732 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (20m) 15m ago 35m 39.8M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 55aac560aab6 2024-04-20T21:57:19.732 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (24m) 14m ago 37m 7952k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T21:57:19.732 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (24m) 15m ago 35m 8304k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T21:57:19.732 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (18m) 14m ago 34m 88.0M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 ee2bb901f26e 2024-04-20T21:57:19.732 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (18m) 14m ago 34m 92.3M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 9075152d4162 2024-04-20T21:57:19.732 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (17m) 14m ago 33m 74.4M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 2d4f3e38dc0e 2024-04-20T21:57:19.732 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (17m) 15m ago 32m 87.6M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 8f97b17ae0eb 2024-04-20T21:57:19.732 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (16m) 15m ago 32m 90.7M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 653d8079967d 2024-04-20T21:57:19.732 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (15m) 15m ago 31m 11.5M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 596dded14c12 2024-04-20T21:57:19.732 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (23m) 14m ago 36m 74.0M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T21:57:19.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.734+0000 7fb759200700 1 -- 172.21.15.62:0/2952800057 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb73806ecc0 msgr2=0x7fb738071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:57:19.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.734+0000 7fb759200700 1 --2- 172.21.15.62:0/2952800057 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb73806ecc0 0x7fb738071170 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fb744006010 tx=0x7fb74401a040 comp rx=0 tx=0).stop 2024-04-20T21:57:19.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.734+0000 7fb759200700 1 -- 172.21.15.62:0/2952800057 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb754101070 msgr2=0x7fb754198770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:57:19.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.734+0000 7fb759200700 1 --2- 172.21.15.62:0/2952800057 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb754101070 0x7fb754198770 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fb74000b4b0 tx=0x7fb74000b870 comp rx=0 tx=0).stop 2024-04-20T21:57:19.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.734+0000 7fb759200700 1 -- 172.21.15.62:0/2952800057 shutdown_connections 2024-04-20T21:57:19.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.734+0000 7fb759200700 1 --2- 172.21.15.62:0/2952800057 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb73806ecc0 0x7fb738071170 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:57:19.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.734+0000 7fb759200700 1 --2- 172.21.15.62:0/2952800057 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb754068540 0x7fb754198230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:57:19.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.734+0000 7fb759200700 1 --2- 172.21.15.62:0/2952800057 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb754101070 0x7fb754198770 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:57:19.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.734+0000 7fb759200700 1 -- 172.21.15.62:0/2952800057 >> 172.21.15.62:0/2952800057 conn(0x7fb754075130 msgr2=0x7fb7540ff3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:57:19.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.734+0000 7fb759200700 1 -- 172.21.15.62:0/2952800057 shutdown_connections 2024-04-20T21:57:19.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.734+0000 7fb759200700 1 -- 172.21.15.62:0/2952800057 wait complete. 2024-04-20T21:57:19.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.897+0000 7f6e32e00700 1 -- 172.21.15.62:0/3914981326 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6e2c1025f0 msgr2=0x7f6e2c1029c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:57:19.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.897+0000 7f6e32e00700 1 --2- 172.21.15.62:0/3914981326 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6e2c1025f0 0x7f6e2c1029c0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f6e2000b0a0 tx=0x7f6e2000b3b0 comp rx=0 tx=0).stop 2024-04-20T21:57:19.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.898+0000 7f6e32e00700 1 -- 172.21.15.62:0/3914981326 shutdown_connections 2024-04-20T21:57:19.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.898+0000 7f6e32e00700 1 --2- 172.21.15.62:0/3914981326 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e2c103e50 0x7f6e2c1042c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:57:19.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.898+0000 7f6e32e00700 1 --2- 172.21.15.62:0/3914981326 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6e2c1025f0 0x7f6e2c1029c0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:57:19.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.898+0000 7f6e32e00700 1 -- 172.21.15.62:0/3914981326 >> 172.21.15.62:0/3914981326 conn(0x7f6e2c0fddb0 msgr2=0x7f6e2c1001c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:57:19.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.898+0000 7f6e32e00700 1 -- 172.21.15.62:0/3914981326 shutdown_connections 2024-04-20T21:57:19.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.898+0000 7f6e32e00700 1 -- 172.21.15.62:0/3914981326 wait complete. 2024-04-20T21:57:19.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.899+0000 7f6e32e00700 1 Processor -- start 2024-04-20T21:57:19.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.899+0000 7f6e32e00700 1 -- start start 2024-04-20T21:57:19.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.900+0000 7f6e32e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e2c1025f0 0x7f6e2c197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:57:19.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.900+0000 7f6e32e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6e2c103e50 0x7f6e2c198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:57:19.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.900+0000 7f6e32e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e2c198aa0 con 0x7f6e2c103e50 2024-04-20T21:57:19.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.900+0000 7f6e32e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e2c198be0 con 0x7f6e2c1025f0 2024-04-20T21:57:19.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.900+0000 7f6e31000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6e2c103e50 0x7f6e2c198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:57:19.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.900+0000 7f6e31000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6e2c103e50 0x7f6e2c198450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50106/0 (socket says 172.21.15.62:50106) 2024-04-20T21:57:19.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.900+0000 7f6e31000700 1 -- 172.21.15.62:0/2393391444 learned_addr learned my addr 172.21.15.62:0/2393391444 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:57:19.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.901+0000 7f6e31a00700 1 --2- 172.21.15.62:0/2393391444 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e2c1025f0 0x7f6e2c197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:57:19.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.901+0000 7f6e31000700 1 -- 172.21.15.62:0/2393391444 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e2c1025f0 msgr2=0x7f6e2c197f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:57:19.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.901+0000 7f6e31000700 1 --2- 172.21.15.62:0/2393391444 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e2c1025f0 0x7f6e2c197f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:57:19.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.901+0000 7f6e31000700 1 -- 172.21.15.62:0/2393391444 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e20009d00 con 0x7f6e2c103e50 2024-04-20T21:57:19.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.901+0000 7f6e31000700 1 --2- 172.21.15.62:0/2393391444 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6e2c103e50 0x7f6e2c198450 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f6e1800c930 tx=0x7f6e1800ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:57:19.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.901+0000 7f6e2a000700 1 -- 172.21.15.62:0/2393391444 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6e18007ad0 con 0x7f6e2c103e50 2024-04-20T21:57:19.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.901+0000 7f6e2a000700 1 -- 172.21.15.62:0/2393391444 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6e18007c30 con 0x7f6e2c103e50 2024-04-20T21:57:19.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.901+0000 7f6e2a000700 1 -- 172.21.15.62:0/2393391444 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6e18018610 con 0x7f6e2c103e50 2024-04-20T21:57:19.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.901+0000 7f6e32e00700 1 -- 172.21.15.62:0/2393391444 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e2c19ca30 con 0x7f6e2c103e50 2024-04-20T21:57:19.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.902+0000 7f6e32e00700 1 -- 172.21.15.62:0/2393391444 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e2c19d000 con 0x7f6e2c103e50 2024-04-20T21:57:19.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.903+0000 7f6e32e00700 1 -- 172.21.15.62:0/2393391444 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e2c062380 con 0x7f6e2c103e50 2024-04-20T21:57:19.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.904+0000 7f6e2a000700 1 -- 172.21.15.62:0/2393391444 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f6e1800f450 con 0x7f6e2c103e50 2024-04-20T21:57:19.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.905+0000 7f6e2a000700 1 --2- 172.21.15.62:0/2393391444 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6e1406ecc0 0x7f6e14071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:57:19.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.905+0000 7f6e2a000700 1 -- 172.21.15.62:0/2393391444 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f6e1808eff0 con 0x7f6e2c103e50 2024-04-20T21:57:19.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.905+0000 7f6e31a00700 1 --2- 172.21.15.62:0/2393391444 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6e1406ecc0 0x7f6e14071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:57:19.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.906+0000 7f6e31a00700 1 --2- 172.21.15.62:0/2393391444 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6e1406ecc0 0x7f6e14071170 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f6e20015040 tx=0x7f6e20000f40 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:57:19.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:19.910+0000 7f6e2a000700 1 -- 172.21.15.62:0/2393391444 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f6e18057de0 con 0x7f6e2c103e50 2024-04-20T21:57:20.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:20.241+0000 7f6e32e00700 1 -- 172.21.15.62:0/2393391444 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f6e2c199270 con 0x7f6e2c103e50 2024-04-20T21:57:20.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:20.241+0000 7f6e2a000700 1 -- 172.21.15.62:0/2393391444 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7f6e1805b4a0 con 0x7f6e2c103e50 2024-04-20T21:57:20.241 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:57:20.241 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T21:57:20.241 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:57:20.241 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:57:20.241 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T21:57:20.242 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:57:20.242 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:57:20.242 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T21:57:20.242 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 6 2024-04-20T21:57:20.242 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:57:20.242 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T21:57:20.242 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 2, 2024-04-20T21:57:20.242 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:57:20.242 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:57:20.242 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T21:57:20.242 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 2, 2024-04-20T21:57:20.242 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 12 2024-04-20T21:57:20.242 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T21:57:20.242 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:57:20.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:20.244+0000 7f6e32e00700 1 -- 172.21.15.62:0/2393391444 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6e1406ecc0 msgr2=0x7f6e14071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:57:20.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:20.244+0000 7f6e32e00700 1 --2- 172.21.15.62:0/2393391444 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6e1406ecc0 0x7f6e14071170 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f6e20015040 tx=0x7f6e20000f40 comp rx=0 tx=0).stop 2024-04-20T21:57:20.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:20.244+0000 7f6e32e00700 1 -- 172.21.15.62:0/2393391444 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6e2c103e50 msgr2=0x7f6e2c198450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:57:20.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:20.244+0000 7f6e32e00700 1 --2- 172.21.15.62:0/2393391444 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6e2c103e50 0x7f6e2c198450 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f6e1800c930 tx=0x7f6e1800ccf0 comp rx=0 tx=0).stop 2024-04-20T21:57:20.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:20.244+0000 7f6e32e00700 1 -- 172.21.15.62:0/2393391444 shutdown_connections 2024-04-20T21:57:20.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:20.244+0000 7f6e32e00700 1 --2- 172.21.15.62:0/2393391444 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6e1406ecc0 0x7f6e14071170 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:57:20.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:20.244+0000 7f6e32e00700 1 --2- 172.21.15.62:0/2393391444 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e2c1025f0 0x7f6e2c197f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:57:20.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:20.244+0000 7f6e32e00700 1 --2- 172.21.15.62:0/2393391444 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6e2c103e50 0x7f6e2c198450 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:57:20.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:20.244+0000 7f6e32e00700 1 -- 172.21.15.62:0/2393391444 >> 172.21.15.62:0/2393391444 conn(0x7f6e2c0fddb0 msgr2=0x7f6e2c0fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:57:20.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:20.245+0000 7f6e32e00700 1 -- 172.21.15.62:0/2393391444 shutdown_connections 2024-04-20T21:57:20.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:20.245+0000 7f6e32e00700 1 -- 172.21.15.62:0/2393391444 wait complete. 2024-04-20T21:57:20.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:20.395+0000 7fa96f200700 1 -- 172.21.15.62:0/2682040404 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa968100ed0 msgr2=0x7fa968104ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:57:20.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:20.395+0000 7fa96f200700 1 --2- 172.21.15.62:0/2682040404 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa968100ed0 0x7fa968104ea0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fa95000b0a0 tx=0x7fa95000b3b0 comp rx=0 tx=0).stop 2024-04-20T21:57:20.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:20.395+0000 7fa96f200700 1 -- 172.21.15.62:0/2682040404 shutdown_connections 2024-04-20T21:57:20.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:20.395+0000 7fa96f200700 1 --2- 172.21.15.62:0/2682040404 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa968100ed0 0x7fa968104ea0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:57:20.395 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:57:20 smithi062 ceph-mon[76322]: pgmap v660: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:57:20.395 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:57:20 smithi062 ceph-mon[76322]: from='client.35208 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:57:20.395 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:57:20 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:57:20.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:20.395+0000 7fa96f200700 1 --2- 172.21.15.62:0/2682040404 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa968100530 0x7fa968100900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:57:20.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:20.395+0000 7fa96f200700 1 -- 172.21.15.62:0/2682040404 >> 172.21.15.62:0/2682040404 conn(0x7fa9680fbe10 msgr2=0x7fa9680fe220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:57:20.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:20.395+0000 7fa96f200700 1 -- 172.21.15.62:0/2682040404 shutdown_connections 2024-04-20T21:57:20.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:20.395+0000 7fa96f200700 1 -- 172.21.15.62:0/2682040404 wait complete. 2024-04-20T21:57:20.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:20.397+0000 7fa96f200700 1 Processor -- start 2024-04-20T21:57:20.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:20.397+0000 7fa96f200700 1 -- start start 2024-04-20T21:57:20.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:20.397+0000 7fa96f200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa968100530 0x7fa968195f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:57:20.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:20.397+0000 7fa96f200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa968100ed0 0x7fa968196490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:57:20.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:20.397+0000 7fa96f200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa968196b70 con 0x7fa968100530 2024-04-20T21:57:20.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:20.397+0000 7fa96f200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa96819a900 con 0x7fa968100ed0 2024-04-20T21:57:20.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:20.398+0000 7fa967e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa968100530 0x7fa968195f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:57:20.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:20.398+0000 7fa967e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa968100530 0x7fa968195f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50116/0 (socket says 172.21.15.62:50116) 2024-04-20T21:57:20.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:20.398+0000 7fa967e00700 1 -- 172.21.15.62:0/2776392583 learned_addr learned my addr 172.21.15.62:0/2776392583 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:57:20.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:20.398+0000 7fa95f400700 1 --2- 172.21.15.62:0/2776392583 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa968100ed0 0x7fa968196490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:57:20.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:20.398+0000 7fa967e00700 1 -- 172.21.15.62:0/2776392583 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa968100ed0 msgr2=0x7fa968196490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:57:20.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:20.398+0000 7fa967e00700 1 --2- 172.21.15.62:0/2776392583 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa968100ed0 0x7fa968196490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:57:20.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:20.398+0000 7fa967e00700 1 -- 172.21.15.62:0/2776392583 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa950009d00 con 0x7fa968100530 2024-04-20T21:57:20.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:20.399+0000 7fa967e00700 1 --2- 172.21.15.62:0/2776392583 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa968100530 0x7fa968195f50 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fa95800e920 tx=0x7fa95800ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:57:20.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:20.399+0000 7fa965600700 1 -- 172.21.15.62:0/2776392583 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa958009790 con 0x7fa968100530 2024-04-20T21:57:20.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:20.399+0000 7fa965600700 1 -- 172.21.15.62:0/2776392583 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa95800cba0 con 0x7fa968100530 2024-04-20T21:57:20.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:20.399+0000 7fa96f200700 1 -- 172.21.15.62:0/2776392583 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa96819abe0 con 0x7fa968100530 2024-04-20T21:57:20.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:20.399+0000 7fa965600700 1 -- 172.21.15.62:0/2776392583 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa958018650 con 0x7fa968100530 2024-04-20T21:57:20.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:20.399+0000 7fa96f200700 1 -- 172.21.15.62:0/2776392583 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa96819b100 con 0x7fa968100530 2024-04-20T21:57:20.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:20.400+0000 7fa96f200700 1 -- 172.21.15.62:0/2776392583 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9681088a0 con 0x7fa968100530 2024-04-20T21:57:20.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:20.402+0000 7fa965600700 1 -- 172.21.15.62:0/2776392583 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fa95800cd10 con 0x7fa968100530 2024-04-20T21:57:20.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:20.402+0000 7fa965600700 1 --2- 172.21.15.62:0/2776392583 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa94c06ecc0 0x7fa94c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:57:20.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:20.403+0000 7fa965600700 1 -- 172.21.15.62:0/2776392583 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fa958014070 con 0x7fa968100530 2024-04-20T21:57:20.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:20.403+0000 7fa95f400700 1 --2- 172.21.15.62:0/2776392583 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa94c06ecc0 0x7fa94c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:57:20.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:20.404+0000 7fa95f400700 1 --2- 172.21.15.62:0/2776392583 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa94c06ecc0 0x7fa94c071170 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fa950000f80 tx=0x7fa95000b740 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:57:20.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:20.407+0000 7fa965600700 1 -- 172.21.15.62:0/2776392583 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fa95805bc80 con 0x7fa968100530 2024-04-20T21:57:20.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:57:20 smithi189 ceph-mon[57022]: pgmap v660: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:57:20.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:57:20 smithi189 ceph-mon[57022]: from='client.35208 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:57:20.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:57:20 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:57:20.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:20.696+0000 7fa96f200700 1 -- 172.21.15.62:0/2776392583 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fa96819b370 con 0x7fa968100530 2024-04-20T21:57:20.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:20.697+0000 7fa965600700 1 -- 172.21.15.62:0/2776392583 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 25 v25) v1 ==== 76+0+1887 (secure 0 0 0) 0x7fa95805b810 con 0x7fa968100530 2024-04-20T21:57:20.696 INFO:teuthology.orchestra.run.smithi062.stdout:e25 2024-04-20T21:57:20.697 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T21:57:20.697 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T21:57:20.697 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T21:57:20.697 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:57:20.697 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T21:57:20.697 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T21:57:20.697 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 20 2024-04-20T21:57:20.697 INFO:teuthology.orchestra.run.smithi062.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-20T21:57:20.697 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T21:57:20.697 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T21:42:01.559061+0000 2024-04-20T21:57:20.697 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T21:57:20.697 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T21:57:20.697 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T21:57:20.697 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T21:57:20.697 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T21:57:20.698 INFO:teuthology.orchestra.run.smithi062.stdout:max_xattr_size 65536 2024-04-20T21:57:20.698 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T21:57:20.698 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T21:57:20.698 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 81 2024-04-20T21:57:20.698 INFO:teuthology.orchestra.run.smithi062.stdout:snap_rstat 1 2024-04-20T21:57:20.698 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T21:57:20.698 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 2 2024-04-20T21:57:20.698 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1 2024-04-20T21:57:20.698 INFO:teuthology.orchestra.run.smithi062.stdout:up {} 2024-04-20T21:57:20.698 INFO:teuthology.orchestra.run.smithi062.stdout:failed 0,1 2024-04-20T21:57:20.698 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T21:57:20.698 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T21:57:20.698 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T21:57:20.698 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T21:57:20.698 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data enabled 2024-04-20T21:57:20.699 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T21:57:20.699 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T21:57:20.699 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 1 2024-04-20T21:57:20.699 INFO:teuthology.orchestra.run.smithi062.stdout:qdb_cluster leader: 0 members: 2024-04-20T21:57:20.699 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:57:20.699 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:57:20.699 INFO:teuthology.orchestra.run.smithi062.stdout:Standby daemons: 2024-04-20T21:57:20.699 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:57:20.699 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ryqblb{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6826/4113916205,v1:172.21.15.62:6827/4113916205] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T21:57:20.699 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ozizwi{-1:34476} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6828/3801975165,v1:172.21.15.62:6829/3801975165] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T21:57:20.699 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.xbgspw{-1:44285} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6826/3996040526,v1:172.21.15.189:6827/3996040526] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:57:20.699 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.dvimer{-1:44289} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6824/1743111843,v1:172.21.15.189:6825/1743111843] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:57:20.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:20.700+0000 7fa96f200700 1 -- 172.21.15.62:0/2776392583 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa94c06ecc0 msgr2=0x7fa94c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:57:20.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:20.700+0000 7fa96f200700 1 --2- 172.21.15.62:0/2776392583 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa94c06ecc0 0x7fa94c071170 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fa950000f80 tx=0x7fa95000b740 comp rx=0 tx=0).stop 2024-04-20T21:57:20.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:20.700+0000 7fa96f200700 1 -- 172.21.15.62:0/2776392583 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa968100530 msgr2=0x7fa968195f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:57:20.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:20.700+0000 7fa96f200700 1 --2- 172.21.15.62:0/2776392583 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa968100530 0x7fa968195f50 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fa95800e920 tx=0x7fa95800ece0 comp rx=0 tx=0).stop 2024-04-20T21:57:20.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:20.700+0000 7fa96f200700 1 -- 172.21.15.62:0/2776392583 shutdown_connections 2024-04-20T21:57:20.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:20.700+0000 7fa96f200700 1 --2- 172.21.15.62:0/2776392583 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa94c06ecc0 0x7fa94c071170 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:57:20.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:20.700+0000 7fa96f200700 1 --2- 172.21.15.62:0/2776392583 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa968100530 0x7fa968195f50 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:57:20.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:20.700+0000 7fa96f200700 1 --2- 172.21.15.62:0/2776392583 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa968100ed0 0x7fa968196490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:57:20.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:20.700+0000 7fa96f200700 1 -- 172.21.15.62:0/2776392583 >> 172.21.15.62:0/2776392583 conn(0x7fa9680fbe10 msgr2=0x7fa9680fd450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:57:20.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:20.700+0000 7fa96f200700 1 -- 172.21.15.62:0/2776392583 shutdown_connections 2024-04-20T21:57:20.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:20.700+0000 7fa96f200700 1 -- 172.21.15.62:0/2776392583 wait complete. 2024-04-20T21:57:20.701 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 25 2024-04-20T21:57:20.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:20.868+0000 7f7dee400700 1 -- 172.21.15.62:0/3321602280 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7de80fefb0 msgr2=0x7f7de80ff380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:57:20.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:20.868+0000 7f7dee400700 1 --2- 172.21.15.62:0/3321602280 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7de80fefb0 0x7f7de80ff380 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f7dd8009910 tx=0x7f7dd8009c20 comp rx=0 tx=0).stop 2024-04-20T21:57:20.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:20.869+0000 7f7dee400700 1 -- 172.21.15.62:0/3321602280 shutdown_connections 2024-04-20T21:57:20.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:20.869+0000 7f7dee400700 1 --2- 172.21.15.62:0/3321602280 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7de80ff8c0 0x7f7de810c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:57:20.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:20.869+0000 7f7dee400700 1 --2- 172.21.15.62:0/3321602280 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7de80fefb0 0x7f7de80ff380 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:57:20.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:20.869+0000 7f7dee400700 1 -- 172.21.15.62:0/3321602280 >> 172.21.15.62:0/3321602280 conn(0x7f7de80fac60 msgr2=0x7f7de80fd070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:57:20.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:20.869+0000 7f7dee400700 1 -- 172.21.15.62:0/3321602280 shutdown_connections 2024-04-20T21:57:20.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:20.869+0000 7f7dee400700 1 -- 172.21.15.62:0/3321602280 wait complete. 2024-04-20T21:57:20.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:20.870+0000 7f7dee400700 1 Processor -- start 2024-04-20T21:57:20.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:20.871+0000 7f7dee400700 1 -- start start 2024-04-20T21:57:20.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:20.871+0000 7f7dee400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7de80fefb0 0x7f7de8198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:57:20.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:20.871+0000 7f7dee400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7de80ff8c0 0x7f7de8198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:57:20.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:20.871+0000 7f7dee400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7de8198c30 con 0x7f7de80ff8c0 2024-04-20T21:57:20.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:20.871+0000 7f7dee400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7de819c950 con 0x7f7de80fefb0 2024-04-20T21:57:20.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:20.871+0000 7f7ddfe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7de80ff8c0 0x7f7de8198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:57:20.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:20.871+0000 7f7ddfe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7de80ff8c0 0x7f7de8198550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50136/0 (socket says 172.21.15.62:50136) 2024-04-20T21:57:20.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:20.871+0000 7f7ddfe00700 1 -- 172.21.15.62:0/1442031898 learned_addr learned my addr 172.21.15.62:0/1442031898 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:57:20.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:20.872+0000 7f7de7400700 1 --2- 172.21.15.62:0/1442031898 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7de80fefb0 0x7f7de8198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:57:20.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:20.872+0000 7f7ddfe00700 1 -- 172.21.15.62:0/1442031898 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7de80fefb0 msgr2=0x7f7de8198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:57:20.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:20.872+0000 7f7ddfe00700 1 --2- 172.21.15.62:0/1442031898 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7de80fefb0 0x7f7de8198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:57:20.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:20.872+0000 7f7ddfe00700 1 -- 172.21.15.62:0/1442031898 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dd80095f0 con 0x7f7de80ff8c0 2024-04-20T21:57:20.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:20.872+0000 7f7ddfe00700 1 --2- 172.21.15.62:0/1442031898 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7de80ff8c0 0x7f7de8198550 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f7dd000e920 tx=0x7f7dd000ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:57:20.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:20.873+0000 7f7de4c00700 1 -- 172.21.15.62:0/1442031898 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7dd0009790 con 0x7f7de80ff8c0 2024-04-20T21:57:20.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:20.873+0000 7f7de4c00700 1 -- 172.21.15.62:0/1442031898 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7dd000cba0 con 0x7f7de80ff8c0 2024-04-20T21:57:20.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:20.873+0000 7f7dee400700 1 -- 172.21.15.62:0/1442031898 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7de819cc30 con 0x7f7de80ff8c0 2024-04-20T21:57:20.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:20.873+0000 7f7de4c00700 1 -- 172.21.15.62:0/1442031898 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7dd0018650 con 0x7f7de80ff8c0 2024-04-20T21:57:20.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:20.873+0000 7f7dee400700 1 -- 172.21.15.62:0/1442031898 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7de819d180 con 0x7f7de80ff8c0 2024-04-20T21:57:20.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:20.874+0000 7f7dee400700 1 -- 172.21.15.62:0/1442031898 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7de8109d90 con 0x7f7de80ff8c0 2024-04-20T21:57:20.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:20.875+0000 7f7de4c00700 1 -- 172.21.15.62:0/1442031898 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f7dd000cd10 con 0x7f7de80ff8c0 2024-04-20T21:57:20.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:20.876+0000 7f7de4c00700 1 --2- 172.21.15.62:0/1442031898 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7dcc06ecc0 0x7f7dcc071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:57:20.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:20.876+0000 7f7de4c00700 1 -- 172.21.15.62:0/1442031898 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f7dd0014070 con 0x7f7de80ff8c0 2024-04-20T21:57:20.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:20.877+0000 7f7de7400700 1 --2- 172.21.15.62:0/1442031898 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7dcc06ecc0 0x7f7dcc071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:57:20.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:20.877+0000 7f7de7400700 1 --2- 172.21.15.62:0/1442031898 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7dcc06ecc0 0x7f7dcc071170 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f7dd800b760 tx=0x7f7dd8009f90 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:57:20.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:20.881+0000 7f7de4c00700 1 -- 172.21.15.62:0/1442031898 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f7dd0053ff0 con 0x7f7de80ff8c0 2024-04-20T21:57:21.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:21.141+0000 7f7dee400700 1 -- 172.21.15.62:0/1442031898 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7de819d3f0 con 0x7f7dcc06ecc0 2024-04-20T21:57:21.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:21.143+0000 7f7de4c00700 1 -- 172.21.15.62:0/1442031898 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f7de819d3f0 con 0x7f7dcc06ecc0 2024-04-20T21:57:21.143 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:57:21.143 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T21:57:21.143 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T21:57:21.144 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-20T21:57:21.144 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T21:57:21.144 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr", 2024-04-20T21:57:21.144 INFO:teuthology.orchestra.run.smithi062.stdout: "crash", 2024-04-20T21:57:21.144 INFO:teuthology.orchestra.run.smithi062.stdout: "mon", 2024-04-20T21:57:21.144 INFO:teuthology.orchestra.run.smithi062.stdout: "osd" 2024-04-20T21:57:21.144 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T21:57:21.144 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "14/23 daemons upgraded", 2024-04-20T21:57:21.144 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading mds daemons", 2024-04-20T21:57:21.144 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T21:57:21.144 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:57:21.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:21.146+0000 7f7dee400700 1 -- 172.21.15.62:0/1442031898 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7dcc06ecc0 msgr2=0x7f7dcc071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:57:21.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:21.146+0000 7f7dee400700 1 --2- 172.21.15.62:0/1442031898 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7dcc06ecc0 0x7f7dcc071170 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f7dd800b760 tx=0x7f7dd8009f90 comp rx=0 tx=0).stop 2024-04-20T21:57:21.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:21.146+0000 7f7dee400700 1 -- 172.21.15.62:0/1442031898 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7de80ff8c0 msgr2=0x7f7de8198550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:57:21.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:21.146+0000 7f7dee400700 1 --2- 172.21.15.62:0/1442031898 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7de80ff8c0 0x7f7de8198550 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f7dd000e920 tx=0x7f7dd000ece0 comp rx=0 tx=0).stop 2024-04-20T21:57:21.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:21.146+0000 7f7dee400700 1 -- 172.21.15.62:0/1442031898 shutdown_connections 2024-04-20T21:57:21.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:21.146+0000 7f7dee400700 1 --2- 172.21.15.62:0/1442031898 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7dcc06ecc0 0x7f7dcc071170 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:57:21.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:21.146+0000 7f7dee400700 1 --2- 172.21.15.62:0/1442031898 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7de80fefb0 0x7f7de8198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:57:21.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:21.146+0000 7f7dee400700 1 --2- 172.21.15.62:0/1442031898 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7de80ff8c0 0x7f7de8198550 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:57:21.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:21.146+0000 7f7dee400700 1 -- 172.21.15.62:0/1442031898 >> 172.21.15.62:0/1442031898 conn(0x7f7de80fac60 msgr2=0x7f7de80fc290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:57:21.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:21.146+0000 7f7dee400700 1 -- 172.21.15.62:0/1442031898 shutdown_connections 2024-04-20T21:57:21.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:21.146+0000 7f7dee400700 1 -- 172.21.15.62:0/1442031898 wait complete. 2024-04-20T21:57:21.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:21.302+0000 7fca0e200700 1 -- 172.21.15.62:0/1002272794 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fca08100e80 msgr2=0x7fca08104e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:57:21.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:21.302+0000 7fca0e200700 1 --2- 172.21.15.62:0/1002272794 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fca08100e80 0x7fca08104e50 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fc9f4009910 tx=0x7fc9f4009c20 comp rx=0 tx=0).stop 2024-04-20T21:57:21.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:21.302+0000 7fca0e200700 1 -- 172.21.15.62:0/1002272794 shutdown_connections 2024-04-20T21:57:21.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:21.302+0000 7fca0e200700 1 --2- 172.21.15.62:0/1002272794 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fca08100e80 0x7fca08104e50 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:57:21.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:21.302+0000 7fca0e200700 1 --2- 172.21.15.62:0/1002272794 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca081004e0 0x7fca081008b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:57:21.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:21.302+0000 7fca0e200700 1 -- 172.21.15.62:0/1002272794 >> 172.21.15.62:0/1002272794 conn(0x7fca080fbda0 msgr2=0x7fca080fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:57:21.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:21.302+0000 7fca0e200700 1 -- 172.21.15.62:0/1002272794 shutdown_connections 2024-04-20T21:57:21.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:21.302+0000 7fca0e200700 1 -- 172.21.15.62:0/1002272794 wait complete. 2024-04-20T21:57:21.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:21.303+0000 7fca0e200700 1 Processor -- start 2024-04-20T21:57:21.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:21.303+0000 7fca0e200700 1 -- start start 2024-04-20T21:57:21.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:21.303+0000 7fca0e200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca081004e0 0x7fca08198190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:57:21.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:21.303+0000 7fca0e200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fca08100e80 0x7fca081986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:57:21.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:21.303+0000 7fca0e200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca08198db0 con 0x7fca08100e80 2024-04-20T21:57:21.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:21.303+0000 7fca0e200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca0819cb40 con 0x7fca081004e0 2024-04-20T21:57:21.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:21.304+0000 7fca06a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fca08100e80 0x7fca081986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:57:21.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:21.304+0000 7fca06a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fca08100e80 0x7fca081986d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50168/0 (socket says 172.21.15.62:50168) 2024-04-20T21:57:21.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:21.304+0000 7fca06a00700 1 -- 172.21.15.62:0/3992626632 learned_addr learned my addr 172.21.15.62:0/3992626632 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:57:21.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:21.304+0000 7fca06a00700 1 -- 172.21.15.62:0/3992626632 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca081004e0 msgr2=0x7fca08198190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:57:21.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:21.304+0000 7fca06a00700 1 --2- 172.21.15.62:0/3992626632 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca081004e0 0x7fca08198190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:57:21.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:21.304+0000 7fca06a00700 1 -- 172.21.15.62:0/3992626632 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9f40095f0 con 0x7fca08100e80 2024-04-20T21:57:21.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:21.304+0000 7fca06a00700 1 --2- 172.21.15.62:0/3992626632 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fca08100e80 0x7fca081986d0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fc9f400bda0 tx=0x7fc9f400be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:57:21.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:21.305+0000 7fc9ffe00700 1 -- 172.21.15.62:0/3992626632 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc9f401d070 con 0x7fca08100e80 2024-04-20T21:57:21.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:21.305+0000 7fc9ffe00700 1 -- 172.21.15.62:0/3992626632 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc9f4004fb0 con 0x7fca08100e80 2024-04-20T21:57:21.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:21.305+0000 7fca0e200700 1 -- 172.21.15.62:0/3992626632 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca0819cdc0 con 0x7fca08100e80 2024-04-20T21:57:21.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:21.305+0000 7fc9ffe00700 1 -- 172.21.15.62:0/3992626632 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc9f400f460 con 0x7fca08100e80 2024-04-20T21:57:21.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:21.305+0000 7fca0e200700 1 -- 172.21.15.62:0/3992626632 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca0819d2b0 con 0x7fca08100e80 2024-04-20T21:57:21.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:21.306+0000 7fca0e200700 1 -- 172.21.15.62:0/3992626632 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca081045d0 con 0x7fca08100e80 2024-04-20T21:57:21.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:21.307+0000 7fc9ffe00700 1 -- 172.21.15.62:0/3992626632 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fc9f4022800 con 0x7fca08100e80 2024-04-20T21:57:21.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:21.308+0000 7fc9ffe00700 1 --2- 172.21.15.62:0/3992626632 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc9ec06ecc0 0x7fc9ec071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:57:21.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:21.308+0000 7fc9ffe00700 1 -- 172.21.15.62:0/3992626632 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fc9f4090390 con 0x7fca08100e80 2024-04-20T21:57:21.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:21.308+0000 7fca07400700 1 --2- 172.21.15.62:0/3992626632 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc9ec06ecc0 0x7fc9ec071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:57:21.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:21.309+0000 7fca07400700 1 --2- 172.21.15.62:0/3992626632 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc9ec06ecc0 0x7fc9ec071170 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fc9f80097b0 tx=0x7fc9f8006d20 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:57:21.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:21.313+0000 7fc9ffe00700 1 -- 172.21.15.62:0/3992626632 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fc9f40592f0 con 0x7fca08100e80 2024-04-20T21:57:21.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:57:21 smithi189 ceph-mon[57022]: from='client.35212 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:57:21.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:57:21 smithi189 ceph-mon[57022]: from='client.35216 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:57:21.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:57:21 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/2393391444' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:57:21.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:57:21 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/2776392583' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T21:57:21.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:57:21 smithi062 ceph-mon[76322]: from='client.35212 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:57:21.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:57:21 smithi062 ceph-mon[76322]: from='client.35216 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:57:21.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:57:21 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/2393391444' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:57:21.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:57:21 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/2776392583' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T21:57:21.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:21.640+0000 7fca0e200700 1 -- 172.21.15.62:0/3992626632 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fca08103fb0 con 0x7fca08100e80 2024-04-20T21:57:21.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:21.641+0000 7fc9ffe00700 1 -- 172.21.15.62:0/3992626632 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7fc9f40271c0 con 0x7fca08100e80 2024-04-20T21:57:21.641 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-20T21:57:21.641 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-20T21:57:21.641 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is degraded 2024-04-20T21:57:21.642 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T21:57:21.642 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T21:57:21.642 INFO:teuthology.orchestra.run.smithi062.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-20T21:57:21.642 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-20T21:57:21.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:21.644+0000 7fca0e200700 1 -- 172.21.15.62:0/3992626632 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc9ec06ecc0 msgr2=0x7fc9ec071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:57:21.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:21.644+0000 7fca0e200700 1 --2- 172.21.15.62:0/3992626632 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc9ec06ecc0 0x7fc9ec071170 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fc9f80097b0 tx=0x7fc9f8006d20 comp rx=0 tx=0).stop 2024-04-20T21:57:21.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:21.644+0000 7fca0e200700 1 -- 172.21.15.62:0/3992626632 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fca08100e80 msgr2=0x7fca081986d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:57:21.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:21.644+0000 7fca0e200700 1 --2- 172.21.15.62:0/3992626632 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fca08100e80 0x7fca081986d0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fc9f400bda0 tx=0x7fc9f400be80 comp rx=0 tx=0).stop 2024-04-20T21:57:21.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:21.644+0000 7fca0e200700 1 -- 172.21.15.62:0/3992626632 shutdown_connections 2024-04-20T21:57:21.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:21.644+0000 7fca0e200700 1 --2- 172.21.15.62:0/3992626632 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc9ec06ecc0 0x7fc9ec071170 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:57:21.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:21.644+0000 7fca0e200700 1 --2- 172.21.15.62:0/3992626632 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca081004e0 0x7fca08198190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:57:21.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:21.644+0000 7fca0e200700 1 --2- 172.21.15.62:0/3992626632 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fca08100e80 0x7fca081986d0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:57:21.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:21.644+0000 7fca0e200700 1 -- 172.21.15.62:0/3992626632 >> 172.21.15.62:0/3992626632 conn(0x7fca080fbda0 msgr2=0x7fca080fd520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:57:21.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:21.644+0000 7fca0e200700 1 -- 172.21.15.62:0/3992626632 shutdown_connections 2024-04-20T21:57:21.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:21.644+0000 7fca0e200700 1 -- 172.21.15.62:0/3992626632 wait complete. 2024-04-20T21:57:22.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:57:22 smithi062 ceph-mon[76322]: pgmap v661: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:57:22.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:57:22 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/3992626632' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T21:57:22.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:57:22 smithi189 ceph-mon[57022]: pgmap v661: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:57:22.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:57:22 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/3992626632' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T21:57:23.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:57:23 smithi189 ceph-mon[57022]: from='client.35228 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:57:23.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:57:23 smithi062 ceph-mon[76322]: from='client.35228 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:57:24.400 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:57:24 smithi189 ceph-mon[57022]: pgmap v662: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:57:24.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:57:24 smithi062 ceph-mon[76322]: pgmap v662: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:57:26.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:57:26 smithi189 ceph-mon[57022]: pgmap v663: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:57:26.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:57:26 smithi062 ceph-mon[76322]: pgmap v663: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:57:28.437 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:57:28 smithi062 ceph-mon[76322]: pgmap v664: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:57:28.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:57:28 smithi189 ceph-mon[57022]: pgmap v664: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:57:30.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:57:30 smithi189 ceph-mon[57022]: pgmap v665: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:57:30.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:57:30 smithi062 ceph-mon[76322]: pgmap v665: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:57:32.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:57:32 smithi189 ceph-mon[57022]: pgmap v666: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:57:32.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:57:32 smithi062 ceph-mon[76322]: pgmap v666: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:57:34.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:57:34 smithi189 ceph-mon[57022]: pgmap v667: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:57:34.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:57:34 smithi062 ceph-mon[76322]: pgmap v667: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:57:35.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:57:35 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:57:35.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:57:35 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:57:36.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:57:36 smithi189 ceph-mon[57022]: pgmap v668: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:57:36.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:57:36 smithi062 ceph-mon[76322]: pgmap v668: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:57:38.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:57:38 smithi189 ceph-mon[57022]: pgmap v669: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:57:38.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:57:38 smithi062 ceph-mon[76322]: pgmap v669: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:57:39.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:57:39 smithi189 ceph-mon[57022]: pgmap v670: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:57:39.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:57:39 smithi062 ceph-mon[76322]: pgmap v670: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:57:42.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:57:41 smithi189 ceph-mon[57022]: pgmap v671: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:57:42.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:57:41 smithi062 ceph-mon[76322]: pgmap v671: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:57:44.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:57:43 smithi189 ceph-mon[57022]: pgmap v672: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:57:44.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:57:43 smithi062 ceph-mon[76322]: pgmap v672: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:57:45.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:57:45 smithi062 ceph-mon[76322]: pgmap v673: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:57:45.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:57:45 smithi189 ceph-mon[57022]: pgmap v673: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:57:48.144 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:57:47 smithi062 ceph-mon[76322]: pgmap v674: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:57:48.144 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:57:47 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:57:48.144 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:57:47 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T21:57:48.144 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:57:47 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:57:48.144 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:57:47 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:57:48.144 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:57:47 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:57:48.144 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:57:47 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:57:48.144 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:57:47 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:57:48.144 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:57:47 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:57:48.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:57:47 smithi189 ceph-mon[57022]: pgmap v674: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:57:48.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:57:47 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:57:48.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:57:47 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T21:57:48.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:57:47 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:57:48.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:57:47 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:57:48.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:57:47 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:57:48.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:57:47 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:57:48.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:57:47 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:57:48.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:57:47 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:57:48.937 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:57:48 smithi189 ceph-mon[57022]: pgmap v675: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:57:48.937 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:57:48 smithi189 ceph-mon[57022]: Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED) 2024-04-20T21:57:49.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:57:48 smithi062 ceph-mon[76322]: pgmap v675: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:57:49.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:57:48 smithi062 ceph-mon[76322]: Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED) 2024-04-20T21:57:50.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:57:50 smithi189 ceph-mon[57022]: pgmap v676: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:57:50.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:57:50 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:57:50.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:57:50 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:57:50.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:57:50 smithi189 ceph-mon[57022]: Upgrade: Updating mds.cephfs.smithi189.dvimer 2024-04-20T21:57:50.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:57:50 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:57:50.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:57:50 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi189.dvimer", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-20T21:57:50.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:57:50 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:57:50.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:57:50 smithi189 ceph-mon[57022]: Deploying daemon mds.cephfs.smithi189.dvimer on smithi189 2024-04-20T21:57:50.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:57:50 smithi062 ceph-mon[76322]: pgmap v676: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:57:50.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:57:50 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:57:50.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:57:50 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:57:50.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:57:50 smithi062 ceph-mon[76322]: Upgrade: Updating mds.cephfs.smithi189.dvimer 2024-04-20T21:57:50.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:57:50 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:57:50.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:57:50 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi189.dvimer", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-20T21:57:50.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:57:50 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T21:57:50.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:57:50 smithi062 ceph-mon[76322]: Deploying daemon mds.cephfs.smithi189.dvimer on smithi189 2024-04-20T21:57:51.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:51.818+0000 7f75a7400700 1 -- 172.21.15.62:0/683507906 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f75a0100e40 msgr2=0x7f75a0104e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:57:51.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:51.818+0000 7f75a7400700 1 --2- 172.21.15.62:0/683507906 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f75a0100e40 0x7f75a0104e10 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f758c009910 tx=0x7f758c009c20 comp rx=0 tx=0).stop 2024-04-20T21:57:51.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:51.818+0000 7f75a7400700 1 -- 172.21.15.62:0/683507906 shutdown_connections 2024-04-20T21:57:51.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:51.818+0000 7f75a7400700 1 --2- 172.21.15.62:0/683507906 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f75a0100e40 0x7f75a0104e10 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:57:51.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:51.818+0000 7f75a7400700 1 --2- 172.21.15.62:0/683507906 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75a01004a0 0x7f75a0100870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:57:51.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:51.818+0000 7f75a7400700 1 -- 172.21.15.62:0/683507906 >> 172.21.15.62:0/683507906 conn(0x7f75a00fbd40 msgr2=0x7f75a00fe150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:57:51.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:51.819+0000 7f75a7400700 1 -- 172.21.15.62:0/683507906 shutdown_connections 2024-04-20T21:57:51.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:51.819+0000 7f75a7400700 1 -- 172.21.15.62:0/683507906 wait complete. 2024-04-20T21:57:51.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:51.820+0000 7f75a7400700 1 Processor -- start 2024-04-20T21:57:51.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:51.820+0000 7f75a7400700 1 -- start start 2024-04-20T21:57:51.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:51.821+0000 7f75a7400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f75a01004a0 0x7f75a0073050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:57:51.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:51.821+0000 7f75a7400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75a0100e40 0x7f75a0073590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:57:51.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:51.821+0000 7f75a7400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75a0073ad0 con 0x7f75a01004a0 2024-04-20T21:57:51.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:51.821+0000 7f75a7400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75a0073c10 con 0x7f75a0100e40 2024-04-20T21:57:51.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:51.821+0000 7f75a4a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f75a01004a0 0x7f75a0073050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:57:51.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:51.821+0000 7f75a4a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f75a01004a0 0x7f75a0073050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56856/0 (socket says 172.21.15.62:56856) 2024-04-20T21:57:51.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:51.821+0000 7f75a4a00700 1 -- 172.21.15.62:0/4251865721 learned_addr learned my addr 172.21.15.62:0/4251865721 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:57:51.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:51.821+0000 7f759fe00700 1 --2- 172.21.15.62:0/4251865721 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75a0100e40 0x7f75a0073590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:57:51.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:51.822+0000 7f75a4a00700 1 -- 172.21.15.62:0/4251865721 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75a0100e40 msgr2=0x7f75a0073590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:57:51.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:51.822+0000 7f75a4a00700 1 --2- 172.21.15.62:0/4251865721 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75a0100e40 0x7f75a0073590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:57:51.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:51.822+0000 7f75a4a00700 1 -- 172.21.15.62:0/4251865721 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f758c0095f0 con 0x7f75a01004a0 2024-04-20T21:57:51.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:51.822+0000 7f75a4a00700 1 --2- 172.21.15.62:0/4251865721 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f75a01004a0 0x7f75a0073050 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f759400b4b0 tx=0x7f759400b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:57:51.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:51.823+0000 7f759d600700 1 -- 172.21.15.62:0/4251865721 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7594011800 con 0x7f75a01004a0 2024-04-20T21:57:51.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:51.823+0000 7f759d600700 1 -- 172.21.15.62:0/4251865721 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7594011e20 con 0x7f75a01004a0 2024-04-20T21:57:51.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:51.823+0000 7f75a7400700 1 -- 172.21.15.62:0/4251865721 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75a01a2110 con 0x7f75a01004a0 2024-04-20T21:57:51.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:51.823+0000 7f759d600700 1 -- 172.21.15.62:0/4251865721 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f759400f2e0 con 0x7f75a01004a0 2024-04-20T21:57:51.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:51.823+0000 7f75a7400700 1 -- 172.21.15.62:0/4251865721 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75a01a2520 con 0x7f75a01004a0 2024-04-20T21:57:51.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:51.824+0000 7f75a7400700 1 -- 172.21.15.62:0/4251865721 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75a0108740 con 0x7f75a01004a0 2024-04-20T21:57:51.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:51.825+0000 7f759d600700 1 -- 172.21.15.62:0/4251865721 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f7594010380 con 0x7f75a01004a0 2024-04-20T21:57:51.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:51.826+0000 7f759d600700 1 --2- 172.21.15.62:0/4251865721 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f758806ecc0 0x7f7588071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:57:51.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:51.826+0000 7f759d600700 1 -- 172.21.15.62:0/4251865721 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f759408e930 con 0x7f75a01004a0 2024-04-20T21:57:51.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:51.826+0000 7f759fe00700 1 --2- 172.21.15.62:0/4251865721 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f758806ecc0 0x7f7588071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:57:51.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:51.827+0000 7f759fe00700 1 --2- 172.21.15.62:0/4251865721 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f758806ecc0 0x7f7588071170 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f75a0074840 tx=0x7f758c01a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:57:51.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:51.831+0000 7f759d600700 1 -- 172.21.15.62:0/4251865721 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f75940576c0 con 0x7f75a01004a0 2024-04-20T21:57:52.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.083+0000 7f75a7400700 1 -- 172.21.15.62:0/4251865721 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f75a0061190 con 0x7f758806ecc0 2024-04-20T21:57:52.085 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.086+0000 7f759d600700 1 -- 172.21.15.62:0/4251865721 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f75a0061190 con 0x7f758806ecc0 2024-04-20T21:57:52.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.089+0000 7f75a7400700 1 -- 172.21.15.62:0/4251865721 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f758806ecc0 msgr2=0x7f7588071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:57:52.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.089+0000 7f75a7400700 1 --2- 172.21.15.62:0/4251865721 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f758806ecc0 0x7f7588071170 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f75a0074840 tx=0x7f758c01a040 comp rx=0 tx=0).stop 2024-04-20T21:57:52.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.089+0000 7f75a7400700 1 -- 172.21.15.62:0/4251865721 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f75a01004a0 msgr2=0x7f75a0073050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:57:52.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.089+0000 7f75a7400700 1 --2- 172.21.15.62:0/4251865721 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f75a01004a0 0x7f75a0073050 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f759400b4b0 tx=0x7f759400b7c0 comp rx=0 tx=0).stop 2024-04-20T21:57:52.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.089+0000 7f75a7400700 1 -- 172.21.15.62:0/4251865721 shutdown_connections 2024-04-20T21:57:52.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.089+0000 7f75a7400700 1 --2- 172.21.15.62:0/4251865721 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f758806ecc0 0x7f7588071170 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:57:52.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.089+0000 7f75a7400700 1 --2- 172.21.15.62:0/4251865721 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f75a01004a0 0x7f75a0073050 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:57:52.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.089+0000 7f75a7400700 1 --2- 172.21.15.62:0/4251865721 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75a0100e40 0x7f75a0073590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:57:52.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.089+0000 7f75a7400700 1 -- 172.21.15.62:0/4251865721 >> 172.21.15.62:0/4251865721 conn(0x7f75a00fbd40 msgr2=0x7f75a00fd510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:57:52.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.089+0000 7f75a7400700 1 -- 172.21.15.62:0/4251865721 shutdown_connections 2024-04-20T21:57:52.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.090+0000 7f75a7400700 1 -- 172.21.15.62:0/4251865721 wait complete. 2024-04-20T21:57:52.105 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T21:57:52.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.251+0000 7f1bc4400700 1 -- 172.21.15.62:0/3502742076 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1bbc106b80 msgr2=0x7f1bbc106ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:57:52.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.251+0000 7f1bc4400700 1 --2- 172.21.15.62:0/3502742076 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1bbc106b80 0x7f1bbc106ff0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f1ba8009910 tx=0x7f1ba8009c20 comp rx=0 tx=0).stop 2024-04-20T21:57:52.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.251+0000 7f1bc4400700 1 -- 172.21.15.62:0/3502742076 shutdown_connections 2024-04-20T21:57:52.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.251+0000 7f1bc4400700 1 --2- 172.21.15.62:0/3502742076 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1bbc106b80 0x7f1bbc106ff0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:57:52.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.251+0000 7f1bc4400700 1 --2- 172.21.15.62:0/3502742076 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1bbc10c8f0 0x7f1bbc10ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:57:52.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.251+0000 7f1bc4400700 1 -- 172.21.15.62:0/3502742076 >> 172.21.15.62:0/3502742076 conn(0x7f1bbc076060 msgr2=0x7f1bbc078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:57:52.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.251+0000 7f1bc4400700 1 -- 172.21.15.62:0/3502742076 shutdown_connections 2024-04-20T21:57:52.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.251+0000 7f1bc4400700 1 -- 172.21.15.62:0/3502742076 wait complete. 2024-04-20T21:57:52.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.252+0000 7f1bc4400700 1 Processor -- start 2024-04-20T21:57:52.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.253+0000 7f1bc4400700 1 -- start start 2024-04-20T21:57:52.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.253+0000 7f1bc4400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1bbc106b80 0x7f1bbc19c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:57:52.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.253+0000 7f1bc4400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1bbc10c8f0 0x7f1bbc19c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:57:52.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.253+0000 7f1bc4400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bbc19d060 con 0x7f1bbc10c8f0 2024-04-20T21:57:52.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.253+0000 7f1bc4400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bbc1a0df0 con 0x7f1bbc106b80 2024-04-20T21:57:52.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.253+0000 7f1bc1000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1bbc10c8f0 0x7f1bbc19c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:57:52.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.253+0000 7f1bc1000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1bbc10c8f0 0x7f1bbc19c980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56870/0 (socket says 172.21.15.62:56870) 2024-04-20T21:57:52.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.253+0000 7f1bc1000700 1 -- 172.21.15.62:0/2890569568 learned_addr learned my addr 172.21.15.62:0/2890569568 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:57:52.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.254+0000 7f1bc1a00700 1 --2- 172.21.15.62:0/2890569568 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1bbc106b80 0x7f1bbc19c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:57:52.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.254+0000 7f1bc1000700 1 -- 172.21.15.62:0/2890569568 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1bbc106b80 msgr2=0x7f1bbc19c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:57:52.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.254+0000 7f1bc1000700 1 --2- 172.21.15.62:0/2890569568 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1bbc106b80 0x7f1bbc19c440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:57:52.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.254+0000 7f1bc1000700 1 -- 172.21.15.62:0/2890569568 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ba80095f0 con 0x7f1bbc10c8f0 2024-04-20T21:57:52.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.254+0000 7f1bc1000700 1 --2- 172.21.15.62:0/2890569568 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1bbc10c8f0 0x7f1bbc19c980 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f1ba800b5c0 tx=0x7f1ba800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:57:52.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.255+0000 7f1bba000700 1 -- 172.21.15.62:0/2890569568 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ba801d070 con 0x7f1bbc10c8f0 2024-04-20T21:57:52.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.255+0000 7f1bba000700 1 -- 172.21.15.62:0/2890569568 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1ba8005e00 con 0x7f1bbc10c8f0 2024-04-20T21:57:52.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.255+0000 7f1bba000700 1 -- 172.21.15.62:0/2890569568 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ba8017400 con 0x7f1bbc10c8f0 2024-04-20T21:57:52.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.255+0000 7f1bc4400700 1 -- 172.21.15.62:0/2890569568 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bbc1a1070 con 0x7f1bbc10c8f0 2024-04-20T21:57:52.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.255+0000 7f1bc4400700 1 -- 172.21.15.62:0/2890569568 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bbc1a1560 con 0x7f1bbc10c8f0 2024-04-20T21:57:52.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.256+0000 7f1bc4400700 1 -- 172.21.15.62:0/2890569568 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bbc10ecc0 con 0x7f1bbc10c8f0 2024-04-20T21:57:52.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.257+0000 7f1bba000700 1 -- 172.21.15.62:0/2890569568 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f1ba800f460 con 0x7f1bbc10c8f0 2024-04-20T21:57:52.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.258+0000 7f1bba000700 1 --2- 172.21.15.62:0/2890569568 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f1ba406ecc0 0x7f1ba4071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:57:52.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.258+0000 7f1bba000700 1 -- 172.21.15.62:0/2890569568 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f1ba8090370 con 0x7f1bbc10c8f0 2024-04-20T21:57:52.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.258+0000 7f1bc1a00700 1 --2- 172.21.15.62:0/2890569568 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f1ba406ecc0 0x7f1ba4071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:57:52.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.259+0000 7f1bc1a00700 1 --2- 172.21.15.62:0/2890569568 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f1ba406ecc0 0x7f1ba4071170 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f1bb0005950 tx=0x7f1bb00058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:57:52.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.263+0000 7f1bba000700 1 -- 172.21.15.62:0/2890569568 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f1ba8059100 con 0x7f1bbc10c8f0 2024-04-20T21:57:52.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.514+0000 7f1bc4400700 1 -- 172.21.15.62:0/2890569568 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1bbc1a1890 con 0x7f1ba406ecc0 2024-04-20T21:57:52.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.516+0000 7f1bba000700 1 -- 172.21.15.62:0/2890569568 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f1bbc1a1890 con 0x7f1ba406ecc0 2024-04-20T21:57:52.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.520+0000 7f1bc4400700 1 -- 172.21.15.62:0/2890569568 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f1ba406ecc0 msgr2=0x7f1ba4071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:57:52.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.520+0000 7f1bc4400700 1 --2- 172.21.15.62:0/2890569568 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f1ba406ecc0 0x7f1ba4071170 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f1bb0005950 tx=0x7f1bb00058e0 comp rx=0 tx=0).stop 2024-04-20T21:57:52.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.520+0000 7f1bc4400700 1 -- 172.21.15.62:0/2890569568 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1bbc10c8f0 msgr2=0x7f1bbc19c980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:57:52.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.520+0000 7f1bc4400700 1 --2- 172.21.15.62:0/2890569568 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1bbc10c8f0 0x7f1bbc19c980 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f1ba800b5c0 tx=0x7f1ba800be20 comp rx=0 tx=0).stop 2024-04-20T21:57:52.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.520+0000 7f1bc4400700 1 -- 172.21.15.62:0/2890569568 shutdown_connections 2024-04-20T21:57:52.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.520+0000 7f1bc4400700 1 --2- 172.21.15.62:0/2890569568 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f1ba406ecc0 0x7f1ba4071170 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:57:52.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.520+0000 7f1bc4400700 1 --2- 172.21.15.62:0/2890569568 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1bbc106b80 0x7f1bbc19c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:57:52.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.520+0000 7f1bc4400700 1 --2- 172.21.15.62:0/2890569568 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1bbc10c8f0 0x7f1bbc19c980 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:57:52.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.520+0000 7f1bc4400700 1 -- 172.21.15.62:0/2890569568 >> 172.21.15.62:0/2890569568 conn(0x7f1bbc076060 msgr2=0x7f1bbc077ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:57:52.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.520+0000 7f1bc4400700 1 -- 172.21.15.62:0/2890569568 shutdown_connections 2024-04-20T21:57:52.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.521+0000 7f1bc4400700 1 -- 172.21.15.62:0/2890569568 wait complete. 2024-04-20T21:57:52.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.685+0000 7f8359600700 1 -- 172.21.15.62:0/2571144958 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f83540687a0 msgr2=0x7f8354068b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:57:52.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.685+0000 7f8359600700 1 --2- 172.21.15.62:0/2571144958 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f83540687a0 0x7f8354068b70 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f8344009910 tx=0x7f8344009c20 comp rx=0 tx=0).stop 2024-04-20T21:57:52.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.685+0000 7f8359600700 1 -- 172.21.15.62:0/2571144958 shutdown_connections 2024-04-20T21:57:52.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.685+0000 7f8359600700 1 --2- 172.21.15.62:0/2571144958 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8354069140 0x7f8354104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:57:52.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.685+0000 7f8359600700 1 --2- 172.21.15.62:0/2571144958 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f83540687a0 0x7f8354068b70 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:57:52.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.685+0000 7f8359600700 1 -- 172.21.15.62:0/2571144958 >> 172.21.15.62:0/2571144958 conn(0x7f8354075130 msgr2=0x7f8354075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:57:52.685 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:57:52 smithi062 ceph-mon[76322]: pgmap v677: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:57:52.685 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:57:52 smithi062 ceph-mon[76322]: from='client.35236 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:57:52.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.685+0000 7f8359600700 1 -- 172.21.15.62:0/2571144958 shutdown_connections 2024-04-20T21:57:52.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.685+0000 7f8359600700 1 -- 172.21.15.62:0/2571144958 wait complete. 2024-04-20T21:57:52.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.686+0000 7f8359600700 1 Processor -- start 2024-04-20T21:57:52.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.687+0000 7f8359600700 1 -- start start 2024-04-20T21:57:52.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.687+0000 7f8359600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f83540687a0 0x7f83541980e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:57:52.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.687+0000 7f8359600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8354069140 0x7f8354198620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:57:52.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.687+0000 7f8359600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8354198d00 con 0x7f83540687a0 2024-04-20T21:57:52.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.687+0000 7f8359600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f835419ca90 con 0x7f8354069140 2024-04-20T21:57:52.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.687+0000 7f8352a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f83540687a0 0x7f83541980e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:57:52.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.687+0000 7f8352a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f83540687a0 0x7f83541980e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56890/0 (socket says 172.21.15.62:56890) 2024-04-20T21:57:52.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.687+0000 7f8352a00700 1 -- 172.21.15.62:0/2432904001 learned_addr learned my addr 172.21.15.62:0/2432904001 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:57:52.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.688+0000 7f8352000700 1 --2- 172.21.15.62:0/2432904001 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8354069140 0x7f8354198620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:57:52.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.688+0000 7f8352a00700 1 -- 172.21.15.62:0/2432904001 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8354069140 msgr2=0x7f8354198620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:57:52.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.688+0000 7f8352a00700 1 --2- 172.21.15.62:0/2432904001 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8354069140 0x7f8354198620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:57:52.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.688+0000 7f8352a00700 1 -- 172.21.15.62:0/2432904001 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83440095f0 con 0x7f83540687a0 2024-04-20T21:57:52.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.688+0000 7f8352a00700 1 --2- 172.21.15.62:0/2432904001 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f83540687a0 0x7f83541980e0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f834400b5c0 tx=0x7f834400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:57:52.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.689+0000 7f834b400700 1 -- 172.21.15.62:0/2432904001 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f834401d070 con 0x7f83540687a0 2024-04-20T21:57:52.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.689+0000 7f834b400700 1 -- 172.21.15.62:0/2432904001 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8344005e00 con 0x7f83540687a0 2024-04-20T21:57:52.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.689+0000 7f834b400700 1 -- 172.21.15.62:0/2432904001 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8344017400 con 0x7f83540687a0 2024-04-20T21:57:52.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.689+0000 7f8359600700 1 -- 172.21.15.62:0/2432904001 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f835419cd10 con 0x7f83540687a0 2024-04-20T21:57:52.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.689+0000 7f8359600700 1 -- 172.21.15.62:0/2432904001 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f835419d200 con 0x7f83540687a0 2024-04-20T21:57:52.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.690+0000 7f8359600700 1 -- 172.21.15.62:0/2432904001 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f835404f2a0 con 0x7f83540687a0 2024-04-20T21:57:52.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.691+0000 7f834b400700 1 -- 172.21.15.62:0/2432904001 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f834400f460 con 0x7f83540687a0 2024-04-20T21:57:52.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.692+0000 7f834b400700 1 --2- 172.21.15.62:0/2432904001 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f833806ecc0 0x7f8338071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:57:52.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.692+0000 7f834b400700 1 -- 172.21.15.62:0/2432904001 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f8344090480 con 0x7f83540687a0 2024-04-20T21:57:52.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.693+0000 7f8352000700 1 --2- 172.21.15.62:0/2432904001 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f833806ecc0 0x7f8338071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:57:52.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.693+0000 7f8352000700 1 --2- 172.21.15.62:0/2432904001 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f833806ecc0 0x7f8338071170 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f8340005fd0 tx=0x7f8340005f60 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:57:52.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.698+0000 7f834b400700 1 -- 172.21.15.62:0/2432904001 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f8344059190 con 0x7f83540687a0 2024-04-20T21:57:52.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:57:52 smithi189 ceph-mon[57022]: pgmap v677: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:57:52.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:57:52 smithi189 ceph-mon[57022]: from='client.35236 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:57:52.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.948+0000 7f8359600700 1 -- 172.21.15.62:0/2432904001 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f835419d550 con 0x7f833806ecc0 2024-04-20T21:57:52.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.966+0000 7f834b400700 1 -- 172.21.15.62:0/2432904001 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f835419d550 con 0x7f833806ecc0 2024-04-20T21:57:52.966 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T21:57:52.967 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (23m) 15m ago 37m 14.2M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T21:57:52.967 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (37m) 15m ago 37m 9512k - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T21:57:52.967 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (36m) 16m ago 36m 8942k - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T21:57:52.967 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (19m) 15m ago 37m 6660k - 19.0.0-3133-ga8c84f59 76bc6b91b726 2825d5ba11c6 2024-04-20T21:57:52.967 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (19m) 16m ago 36m 6665k - 19.0.0-3133-ga8c84f59 76bc6b91b726 20f746fa1bba 2024-04-20T21:57:52.967 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (23m) 15m ago 37m 74.2M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T21:57:52.967 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (15m) 15m ago 30m 11.2M - 19.0.0-3133-ga8c84f59 76bc6b91b726 de80bc90a683 2024-04-20T21:57:52.967 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (15m) 15m ago 30m 11.9M - 19.0.0-3133-ga8c84f59 76bc6b91b726 cd793b7ee98e 2024-04-20T21:57:52.967 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (30m) 16m ago 30m 117M - 18.2.0 dc2bc1663786 fb2a8a223db8 2024-04-20T21:57:52.967 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (30m) 16m ago 30m 55.6M - 18.2.0 dc2bc1663786 f2eec2ee8951 2024-04-20T21:57:52.967 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (26m) 15m ago 39m 511M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T21:57:52.967 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (25m) 16m ago 36m 452M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T21:57:52.967 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (21m) 15m ago 39m 55.9M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 019eb1479485 2024-04-20T21:57:52.967 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (20m) 16m ago 36m 39.8M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 55aac560aab6 2024-04-20T21:57:52.967 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (24m) 15m ago 37m 7952k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T21:57:52.968 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (24m) 16m ago 36m 8304k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T21:57:52.968 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (19m) 15m ago 35m 88.0M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 ee2bb901f26e 2024-04-20T21:57:52.968 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (18m) 15m ago 34m 92.3M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 9075152d4162 2024-04-20T21:57:52.968 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (18m) 15m ago 33m 74.4M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 2d4f3e38dc0e 2024-04-20T21:57:52.968 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (17m) 16m ago 33m 87.6M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 8f97b17ae0eb 2024-04-20T21:57:52.968 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (16m) 16m ago 32m 90.7M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 653d8079967d 2024-04-20T21:57:52.968 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (16m) 16m ago 32m 11.5M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 596dded14c12 2024-04-20T21:57:52.968 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (24m) 15m ago 36m 74.0M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T21:57:52.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.971+0000 7f8359600700 1 -- 172.21.15.62:0/2432904001 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f833806ecc0 msgr2=0x7f8338071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:57:52.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.971+0000 7f8359600700 1 --2- 172.21.15.62:0/2432904001 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f833806ecc0 0x7f8338071170 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f8340005fd0 tx=0x7f8340005f60 comp rx=0 tx=0).stop 2024-04-20T21:57:52.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.971+0000 7f8359600700 1 -- 172.21.15.62:0/2432904001 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f83540687a0 msgr2=0x7f83541980e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:57:52.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.971+0000 7f8359600700 1 --2- 172.21.15.62:0/2432904001 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f83540687a0 0x7f83541980e0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f834400b5c0 tx=0x7f834400be20 comp rx=0 tx=0).stop 2024-04-20T21:57:52.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.971+0000 7f8359600700 1 -- 172.21.15.62:0/2432904001 shutdown_connections 2024-04-20T21:57:52.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.971+0000 7f8359600700 1 --2- 172.21.15.62:0/2432904001 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f833806ecc0 0x7f8338071170 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:57:52.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.971+0000 7f8359600700 1 --2- 172.21.15.62:0/2432904001 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f83540687a0 0x7f83541980e0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:57:52.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.971+0000 7f8359600700 1 --2- 172.21.15.62:0/2432904001 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8354069140 0x7f8354198620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:57:52.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.971+0000 7f8359600700 1 -- 172.21.15.62:0/2432904001 >> 172.21.15.62:0/2432904001 conn(0x7f8354075130 msgr2=0x7f83540fe7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:57:52.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.971+0000 7f8359600700 1 -- 172.21.15.62:0/2432904001 shutdown_connections 2024-04-20T21:57:52.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:52.971+0000 7f8359600700 1 -- 172.21.15.62:0/2432904001 wait complete. 2024-04-20T21:57:53.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:53.142+0000 7f40a7200700 1 -- 172.21.15.62:0/2497846744 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40a8106120 msgr2=0x7f40a81064f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:57:53.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:53.142+0000 7f40a7200700 1 --2- 172.21.15.62:0/2497846744 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40a8106120 0x7f40a81064f0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f4098009910 tx=0x7f4098009c20 comp rx=0 tx=0).stop 2024-04-20T21:57:53.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:53.142+0000 7f40a7200700 1 -- 172.21.15.62:0/2497846744 shutdown_connections 2024-04-20T21:57:53.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:53.142+0000 7f40a7200700 1 --2- 172.21.15.62:0/2497846744 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f40a8068540 0x7f40a80689b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:57:53.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:53.142+0000 7f40a7200700 1 --2- 172.21.15.62:0/2497846744 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40a8106120 0x7f40a81064f0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:57:53.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:53.142+0000 7f40a7200700 1 -- 172.21.15.62:0/2497846744 >> 172.21.15.62:0/2497846744 conn(0x7f40a8074fa0 msgr2=0x7f40a80753a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:57:53.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:53.142+0000 7f40a7200700 1 -- 172.21.15.62:0/2497846744 shutdown_connections 2024-04-20T21:57:53.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:53.142+0000 7f40a7200700 1 -- 172.21.15.62:0/2497846744 wait complete. 2024-04-20T21:57:53.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:53.143+0000 7f40a7200700 1 Processor -- start 2024-04-20T21:57:53.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:53.144+0000 7f40a7200700 1 -- start start 2024-04-20T21:57:53.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:53.144+0000 7f40a7200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f40a8068540 0x7f40a8197d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:57:53.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:53.144+0000 7f40a7200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40a8106120 0x7f40a81982b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:57:53.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:53.144+0000 7f40a7200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40a8198900 con 0x7f40a8106120 2024-04-20T21:57:53.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:53.144+0000 7f40a7200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40a8198a40 con 0x7f40a8068540 2024-04-20T21:57:53.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:53.145+0000 7f40a5400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40a8106120 0x7f40a81982b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:57:53.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:53.145+0000 7f40a5400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40a8106120 0x7f40a81982b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56912/0 (socket says 172.21.15.62:56912) 2024-04-20T21:57:53.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:53.145+0000 7f40a5400700 1 -- 172.21.15.62:0/3183163771 learned_addr learned my addr 172.21.15.62:0/3183163771 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:57:53.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:53.145+0000 7f40a5e00700 1 --2- 172.21.15.62:0/3183163771 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f40a8068540 0x7f40a8197d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:57:53.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:53.145+0000 7f40a5400700 1 -- 172.21.15.62:0/3183163771 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f40a8068540 msgr2=0x7f40a8197d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:57:53.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:53.145+0000 7f40a5400700 1 --2- 172.21.15.62:0/3183163771 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f40a8068540 0x7f40a8197d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:57:53.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:53.145+0000 7f40a5400700 1 -- 172.21.15.62:0/3183163771 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40980095f0 con 0x7f40a8106120 2024-04-20T21:57:53.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:53.145+0000 7f40a5e00700 1 --2- 172.21.15.62:0/3183163771 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f40a8068540 0x7f40a8197d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-20T21:57:53.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:53.146+0000 7f40a5400700 1 --2- 172.21.15.62:0/3183163771 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40a8106120 0x7f40a81982b0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f409400c930 tx=0x7f409400ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:57:53.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:53.146+0000 7f409ea00700 1 -- 172.21.15.62:0/3183163771 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4094007ad0 con 0x7f40a8106120 2024-04-20T21:57:53.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:53.146+0000 7f409ea00700 1 -- 172.21.15.62:0/3183163771 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4094007c30 con 0x7f40a8106120 2024-04-20T21:57:53.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:53.146+0000 7f40a7200700 1 -- 172.21.15.62:0/3183163771 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40a819c890 con 0x7f40a8106120 2024-04-20T21:57:53.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:53.146+0000 7f409ea00700 1 -- 172.21.15.62:0/3183163771 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4094018610 con 0x7f40a8106120 2024-04-20T21:57:53.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:53.146+0000 7f40a7200700 1 -- 172.21.15.62:0/3183163771 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40a819ce60 con 0x7f40a8106120 2024-04-20T21:57:53.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:53.147+0000 7f40a7200700 1 -- 172.21.15.62:0/3183163771 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4088005320 con 0x7f40a8106120 2024-04-20T21:57:53.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:53.148+0000 7f409ea00700 1 -- 172.21.15.62:0/3183163771 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f409400f450 con 0x7f40a8106120 2024-04-20T21:57:53.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:53.149+0000 7f409ea00700 1 --2- 172.21.15.62:0/3183163771 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f408c06ecc0 0x7f408c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:57:53.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:53.150+0000 7f409ea00700 1 -- 172.21.15.62:0/3183163771 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f409408ef80 con 0x7f40a8106120 2024-04-20T21:57:53.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:53.150+0000 7f40a5e00700 1 --2- 172.21.15.62:0/3183163771 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f408c06ecc0 0x7f408c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:57:53.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:53.151+0000 7f40a5e00700 1 --2- 172.21.15.62:0/3183163771 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f408c06ecc0 0x7f408c071170 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f4098005b40 tx=0x7f409801a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:57:53.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:53.155+0000 7f409ea00700 1 -- 172.21.15.62:0/3183163771 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f4094057d10 con 0x7f40a8106120 2024-04-20T21:57:53.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:53.485+0000 7f40a7200700 1 -- 172.21.15.62:0/3183163771 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f4088006200 con 0x7f40a8106120 2024-04-20T21:57:53.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:53.486+0000 7f409ea00700 1 -- 172.21.15.62:0/3183163771 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7f409405b3d0 con 0x7f40a8106120 2024-04-20T21:57:53.486 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:57:53.486 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T21:57:53.486 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:57:53.486 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:57:53.486 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T21:57:53.487 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:57:53.487 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:57:53.487 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T21:57:53.487 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 6 2024-04-20T21:57:53.487 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:57:53.487 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T21:57:53.487 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 1, 2024-04-20T21:57:53.487 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:57:53.487 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:57:53.487 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T21:57:53.487 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 1, 2024-04-20T21:57:53.487 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 12 2024-04-20T21:57:53.487 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T21:57:53.487 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:57:53.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:53.489+0000 7f40a7200700 1 -- 172.21.15.62:0/3183163771 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f408c06ecc0 msgr2=0x7f408c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:57:53.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:53.489+0000 7f40a7200700 1 --2- 172.21.15.62:0/3183163771 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f408c06ecc0 0x7f408c071170 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f4098005b40 tx=0x7f409801a040 comp rx=0 tx=0).stop 2024-04-20T21:57:53.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:53.489+0000 7f40a7200700 1 -- 172.21.15.62:0/3183163771 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40a8106120 msgr2=0x7f40a81982b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:57:53.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:53.489+0000 7f40a7200700 1 --2- 172.21.15.62:0/3183163771 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40a8106120 0x7f40a81982b0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f409400c930 tx=0x7f409400ccf0 comp rx=0 tx=0).stop 2024-04-20T21:57:53.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:53.490+0000 7f40a7200700 1 -- 172.21.15.62:0/3183163771 shutdown_connections 2024-04-20T21:57:53.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:53.490+0000 7f40a7200700 1 --2- 172.21.15.62:0/3183163771 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f408c06ecc0 0x7f408c071170 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:57:53.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:53.490+0000 7f40a7200700 1 --2- 172.21.15.62:0/3183163771 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f40a8068540 0x7f40a8197d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:57:53.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:53.490+0000 7f40a7200700 1 --2- 172.21.15.62:0/3183163771 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40a8106120 0x7f40a81982b0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:57:53.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:53.490+0000 7f40a7200700 1 -- 172.21.15.62:0/3183163771 >> 172.21.15.62:0/3183163771 conn(0x7f40a8074fa0 msgr2=0x7f40a8107030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:57:53.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:53.490+0000 7f40a7200700 1 -- 172.21.15.62:0/3183163771 shutdown_connections 2024-04-20T21:57:53.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:53.490+0000 7f40a7200700 1 -- 172.21.15.62:0/3183163771 wait complete. 2024-04-20T21:57:53.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:53.651+0000 7f2b6c400700 1 -- 172.21.15.62:0/76985988 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2b64073090 msgr2=0x7f2b64073460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:57:53.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:53.651+0000 7f2b6c400700 1 --2- 172.21.15.62:0/76985988 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2b64073090 0x7f2b64073460 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f2b58009940 tx=0x7f2b58009c50 comp rx=0 tx=0).stop 2024-04-20T21:57:53.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:53.651+0000 7f2b6c400700 1 -- 172.21.15.62:0/76985988 shutdown_connections 2024-04-20T21:57:53.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:53.651+0000 7f2b6c400700 1 --2- 172.21.15.62:0/76985988 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2b640739a0 0x7f2b6410c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:57:53.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:53.651+0000 7f2b6c400700 1 --2- 172.21.15.62:0/76985988 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2b64073090 0x7f2b64073460 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:57:53.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:53.651+0000 7f2b6c400700 1 -- 172.21.15.62:0/76985988 >> 172.21.15.62:0/76985988 conn(0x7f2b640fbda0 msgr2=0x7f2b640fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:57:53.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:53.652+0000 7f2b6c400700 1 -- 172.21.15.62:0/76985988 shutdown_connections 2024-04-20T21:57:53.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:53.652+0000 7f2b6c400700 1 -- 172.21.15.62:0/76985988 wait complete. 2024-04-20T21:57:53.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:53.653+0000 7f2b6c400700 1 Processor -- start 2024-04-20T21:57:53.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:53.653+0000 7f2b6c400700 1 -- start start 2024-04-20T21:57:53.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:53.653+0000 7f2b6c400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2b64073090 0x7f2b64198180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:57:53.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:53.653+0000 7f2b6c400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2b640739a0 0x7f2b641986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:57:53.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:53.653+0000 7f2b6c400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b64198da0 con 0x7f2b64073090 2024-04-20T21:57:53.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:53.653+0000 7f2b6c400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b6419cb30 con 0x7f2b640739a0 2024-04-20T21:57:53.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:53.653+0000 7f2b69a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2b64073090 0x7f2b64198180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:57:53.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:53.653+0000 7f2b69a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2b64073090 0x7f2b64198180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56924/0 (socket says 172.21.15.62:56924) 2024-04-20T21:57:53.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:53.653+0000 7f2b69a00700 1 -- 172.21.15.62:0/2148400622 learned_addr learned my addr 172.21.15.62:0/2148400622 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:57:53.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:53.654+0000 7f2b69000700 1 --2- 172.21.15.62:0/2148400622 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2b640739a0 0x7f2b641986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:57:53.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:53.654+0000 7f2b69a00700 1 -- 172.21.15.62:0/2148400622 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2b640739a0 msgr2=0x7f2b641986c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:57:53.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:53.654+0000 7f2b69a00700 1 --2- 172.21.15.62:0/2148400622 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2b640739a0 0x7f2b641986c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:57:53.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:53.654+0000 7f2b69a00700 1 -- 172.21.15.62:0/2148400622 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b580095f0 con 0x7f2b64073090 2024-04-20T21:57:53.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:53.654+0000 7f2b69a00700 1 --2- 172.21.15.62:0/2148400622 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2b64073090 0x7f2b64198180 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f2b580055f0 tx=0x7f2b580056d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:57:53.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:53.654+0000 7f2b62000700 1 -- 172.21.15.62:0/2148400622 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2b5801d070 con 0x7f2b64073090 2024-04-20T21:57:53.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:53.655+0000 7f2b6c400700 1 -- 172.21.15.62:0/2148400622 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b6419cdb0 con 0x7f2b64073090 2024-04-20T21:57:53.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:53.655+0000 7f2b62000700 1 -- 172.21.15.62:0/2148400622 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2b5800f460 con 0x7f2b64073090 2024-04-20T21:57:53.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:53.655+0000 7f2b62000700 1 -- 172.21.15.62:0/2148400622 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2b58021640 con 0x7f2b64073090 2024-04-20T21:57:53.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:53.655+0000 7f2b6c400700 1 -- 172.21.15.62:0/2148400622 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b6419d2a0 con 0x7f2b64073090 2024-04-20T21:57:53.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:53.656+0000 7f2b6c400700 1 -- 172.21.15.62:0/2148400622 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b64109ce0 con 0x7f2b64073090 2024-04-20T21:57:53.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:53.657+0000 7f2b62000700 1 -- 172.21.15.62:0/2148400622 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f2b5800fa20 con 0x7f2b64073090 2024-04-20T21:57:53.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:53.658+0000 7f2b62000700 1 --2- 172.21.15.62:0/2148400622 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2b4c06ecc0 0x7f2b4c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:57:53.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:53.658+0000 7f2b62000700 1 -- 172.21.15.62:0/2148400622 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f2b58090160 con 0x7f2b64073090 2024-04-20T21:57:53.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:53.658+0000 7f2b69000700 1 --2- 172.21.15.62:0/2148400622 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2b4c06ecc0 0x7f2b4c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:57:53.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:53.659+0000 7f2b69000700 1 --2- 172.21.15.62:0/2148400622 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2b4c06ecc0 0x7f2b4c071170 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f2b50005950 tx=0x7f2b500058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:57:53.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:53.663+0000 7f2b62000700 1 -- 172.21.15.62:0/2148400622 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f2b58058e70 con 0x7f2b64073090 2024-04-20T21:57:53.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:57:53 smithi189 ceph-mon[57022]: from='client.35240 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:57:53.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:57:53 smithi189 ceph-mon[57022]: from='client.35244 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:57:53.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:57:53 smithi062 ceph-mon[76322]: from='client.35240 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:57:53.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:57:53 smithi062 ceph-mon[76322]: from='client.35244 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:57:53.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:53.951+0000 7f2b6c400700 1 -- 172.21.15.62:0/2148400622 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f2b641995d0 con 0x7f2b64073090 2024-04-20T21:57:53.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:53.952+0000 7f2b62000700 1 -- 172.21.15.62:0/2148400622 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 26 v26) v1 ==== 76+0+1709 (secure 0 0 0) 0x7f2b5805c530 con 0x7f2b64073090 2024-04-20T21:57:53.952 INFO:teuthology.orchestra.run.smithi062.stdout:e26 2024-04-20T21:57:53.952 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T21:57:53.952 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T21:57:53.952 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T21:57:53.953 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:57:53.953 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T21:57:53.953 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T21:57:53.953 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 20 2024-04-20T21:57:53.953 INFO:teuthology.orchestra.run.smithi062.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-20T21:57:53.953 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T21:57:53.953 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T21:42:01.559061+0000 2024-04-20T21:57:53.953 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T21:57:53.953 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T21:57:53.953 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T21:57:53.953 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T21:57:53.953 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T21:57:53.953 INFO:teuthology.orchestra.run.smithi062.stdout:max_xattr_size 65536 2024-04-20T21:57:53.953 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T21:57:53.953 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T21:57:53.954 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 81 2024-04-20T21:57:53.954 INFO:teuthology.orchestra.run.smithi062.stdout:snap_rstat 1 2024-04-20T21:57:53.954 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T21:57:53.954 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 2 2024-04-20T21:57:53.954 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1 2024-04-20T21:57:53.954 INFO:teuthology.orchestra.run.smithi062.stdout:up {} 2024-04-20T21:57:53.954 INFO:teuthology.orchestra.run.smithi062.stdout:failed 0,1 2024-04-20T21:57:53.954 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T21:57:53.954 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T21:57:53.954 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T21:57:53.954 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T21:57:53.954 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data enabled 2024-04-20T21:57:53.954 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T21:57:53.954 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T21:57:53.954 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 1 2024-04-20T21:57:53.955 INFO:teuthology.orchestra.run.smithi062.stdout:qdb_cluster leader: 0 members: 2024-04-20T21:57:53.955 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:57:53.955 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:57:53.955 INFO:teuthology.orchestra.run.smithi062.stdout:Standby daemons: 2024-04-20T21:57:53.955 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:57:53.955 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ryqblb{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6826/4113916205,v1:172.21.15.62:6827/4113916205] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T21:57:53.955 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ozizwi{-1:34476} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6828/3801975165,v1:172.21.15.62:6829/3801975165] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T21:57:53.955 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.xbgspw{-1:44285} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6826/3996040526,v1:172.21.15.189:6827/3996040526] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:57:53.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:53.955+0000 7f2b6c400700 1 -- 172.21.15.62:0/2148400622 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2b4c06ecc0 msgr2=0x7f2b4c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:57:53.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:53.955+0000 7f2b6c400700 1 --2- 172.21.15.62:0/2148400622 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2b4c06ecc0 0x7f2b4c071170 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f2b50005950 tx=0x7f2b500058e0 comp rx=0 tx=0).stop 2024-04-20T21:57:53.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:53.955+0000 7f2b6c400700 1 -- 172.21.15.62:0/2148400622 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2b64073090 msgr2=0x7f2b64198180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:57:53.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:53.955+0000 7f2b6c400700 1 --2- 172.21.15.62:0/2148400622 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2b64073090 0x7f2b64198180 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f2b580055f0 tx=0x7f2b580056d0 comp rx=0 tx=0).stop 2024-04-20T21:57:53.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:53.955+0000 7f2b6c400700 1 -- 172.21.15.62:0/2148400622 shutdown_connections 2024-04-20T21:57:53.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:53.955+0000 7f2b6c400700 1 --2- 172.21.15.62:0/2148400622 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2b4c06ecc0 0x7f2b4c071170 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:57:53.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:53.955+0000 7f2b6c400700 1 --2- 172.21.15.62:0/2148400622 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2b64073090 0x7f2b64198180 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:57:53.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:53.955+0000 7f2b6c400700 1 --2- 172.21.15.62:0/2148400622 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2b640739a0 0x7f2b641986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:57:53.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:53.955+0000 7f2b6c400700 1 -- 172.21.15.62:0/2148400622 >> 172.21.15.62:0/2148400622 conn(0x7f2b640fbda0 msgr2=0x7f2b64106e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:57:53.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:53.956+0000 7f2b6c400700 1 -- 172.21.15.62:0/2148400622 shutdown_connections 2024-04-20T21:57:53.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:53.956+0000 7f2b6c400700 1 -- 172.21.15.62:0/2148400622 wait complete. 2024-04-20T21:57:53.957 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 26 2024-04-20T21:57:54.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:54.119+0000 7f636fe00700 1 -- 172.21.15.62:0/1850883529 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6368106b80 msgr2=0x7f6368106ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:57:54.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:54.119+0000 7f636fe00700 1 --2- 172.21.15.62:0/1850883529 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6368106b80 0x7f6368106ff0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f6354009910 tx=0x7f6354009c20 comp rx=0 tx=0).stop 2024-04-20T21:57:54.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:54.119+0000 7f636fe00700 1 -- 172.21.15.62:0/1850883529 shutdown_connections 2024-04-20T21:57:54.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:54.119+0000 7f636fe00700 1 --2- 172.21.15.62:0/1850883529 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6368106b80 0x7f6368106ff0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:57:54.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:54.119+0000 7f636fe00700 1 --2- 172.21.15.62:0/1850883529 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f636810c8f0 0x7f636810ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:57:54.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:54.119+0000 7f636fe00700 1 -- 172.21.15.62:0/1850883529 >> 172.21.15.62:0/1850883529 conn(0x7f6368076060 msgr2=0x7f6368078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:57:54.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:54.119+0000 7f636fe00700 1 -- 172.21.15.62:0/1850883529 shutdown_connections 2024-04-20T21:57:54.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:54.119+0000 7f636fe00700 1 -- 172.21.15.62:0/1850883529 wait complete. 2024-04-20T21:57:54.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:54.120+0000 7f636fe00700 1 Processor -- start 2024-04-20T21:57:54.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:54.120+0000 7f636fe00700 1 -- start start 2024-04-20T21:57:54.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:54.121+0000 7f636fe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6368106b80 0x7f636819c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:57:54.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:54.121+0000 7f636fe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f636810c8f0 0x7f636819c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:57:54.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:54.121+0000 7f636fe00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f636819d060 con 0x7f636810c8f0 2024-04-20T21:57:54.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:54.121+0000 7f636fe00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63681a0df0 con 0x7f6368106b80 2024-04-20T21:57:54.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:54.121+0000 7f636ca00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f636810c8f0 0x7f636819c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:57:54.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:54.121+0000 7f636ca00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f636810c8f0 0x7f636819c980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56936/0 (socket says 172.21.15.62:56936) 2024-04-20T21:57:54.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:54.121+0000 7f636ca00700 1 -- 172.21.15.62:0/3692537798 learned_addr learned my addr 172.21.15.62:0/3692537798 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:57:54.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:54.121+0000 7f636d400700 1 --2- 172.21.15.62:0/3692537798 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6368106b80 0x7f636819c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:57:54.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:54.122+0000 7f636ca00700 1 -- 172.21.15.62:0/3692537798 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6368106b80 msgr2=0x7f636819c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:57:54.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:54.122+0000 7f636ca00700 1 --2- 172.21.15.62:0/3692537798 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6368106b80 0x7f636819c440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:57:54.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:54.122+0000 7f636ca00700 1 -- 172.21.15.62:0/3692537798 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63540095f0 con 0x7f636810c8f0 2024-04-20T21:57:54.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:54.122+0000 7f636ca00700 1 --2- 172.21.15.62:0/3692537798 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f636810c8f0 0x7f636819c980 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f635400b5c0 tx=0x7f635400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:57:54.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:54.122+0000 7f6366000700 1 -- 172.21.15.62:0/3692537798 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f635401d070 con 0x7f636810c8f0 2024-04-20T21:57:54.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:54.122+0000 7f6366000700 1 -- 172.21.15.62:0/3692537798 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6354005e00 con 0x7f636810c8f0 2024-04-20T21:57:54.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:54.122+0000 7f6366000700 1 -- 172.21.15.62:0/3692537798 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6354017400 con 0x7f636810c8f0 2024-04-20T21:57:54.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:54.122+0000 7f636fe00700 1 -- 172.21.15.62:0/3692537798 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63681a1070 con 0x7f636810c8f0 2024-04-20T21:57:54.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:54.123+0000 7f636fe00700 1 -- 172.21.15.62:0/3692537798 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63681a1560 con 0x7f636810c8f0 2024-04-20T21:57:54.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:54.124+0000 7f636fe00700 1 -- 172.21.15.62:0/3692537798 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f636810ecc0 con 0x7f636810c8f0 2024-04-20T21:57:54.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:54.125+0000 7f6366000700 1 -- 172.21.15.62:0/3692537798 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f635400f460 con 0x7f636810c8f0 2024-04-20T21:57:54.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:54.126+0000 7f6366000700 1 --2- 172.21.15.62:0/3692537798 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f635006ecc0 0x7f6350071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:57:54.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:54.126+0000 7f6366000700 1 -- 172.21.15.62:0/3692537798 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f6354090370 con 0x7f636810c8f0 2024-04-20T21:57:54.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:54.126+0000 7f636d400700 1 --2- 172.21.15.62:0/3692537798 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f635006ecc0 0x7f6350071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:57:54.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:54.127+0000 7f636d400700 1 --2- 172.21.15.62:0/3692537798 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f635006ecc0 0x7f6350071170 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f635c005950 tx=0x7f635c0058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:57:54.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:54.131+0000 7f6366000700 1 -- 172.21.15.62:0/3692537798 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f6354059100 con 0x7f636810c8f0 2024-04-20T21:57:54.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:54.390+0000 7f636fe00700 1 -- 172.21.15.62:0/3692537798 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f63681a1890 con 0x7f635006ecc0 2024-04-20T21:57:54.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:54.393+0000 7f6366000700 1 -- 172.21.15.62:0/3692537798 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f63681a1890 con 0x7f635006ecc0 2024-04-20T21:57:54.393 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:57:54.393 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T21:57:54.393 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T21:57:54.393 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-20T21:57:54.393 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T21:57:54.393 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr", 2024-04-20T21:57:54.393 INFO:teuthology.orchestra.run.smithi062.stdout: "crash", 2024-04-20T21:57:54.393 INFO:teuthology.orchestra.run.smithi062.stdout: "mon", 2024-04-20T21:57:54.394 INFO:teuthology.orchestra.run.smithi062.stdout: "osd" 2024-04-20T21:57:54.394 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T21:57:54.394 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "14/23 daemons upgraded", 2024-04-20T21:57:54.394 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading mds daemons", 2024-04-20T21:57:54.394 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T21:57:54.394 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:57:54.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:54.396+0000 7f636fe00700 1 -- 172.21.15.62:0/3692537798 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f635006ecc0 msgr2=0x7f6350071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:57:54.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:54.396+0000 7f636fe00700 1 --2- 172.21.15.62:0/3692537798 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f635006ecc0 0x7f6350071170 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f635c005950 tx=0x7f635c0058e0 comp rx=0 tx=0).stop 2024-04-20T21:57:54.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:54.396+0000 7f636fe00700 1 -- 172.21.15.62:0/3692537798 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f636810c8f0 msgr2=0x7f636819c980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:57:54.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:54.396+0000 7f636fe00700 1 --2- 172.21.15.62:0/3692537798 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f636810c8f0 0x7f636819c980 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f635400b5c0 tx=0x7f635400be20 comp rx=0 tx=0).stop 2024-04-20T21:57:54.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:54.396+0000 7f636fe00700 1 -- 172.21.15.62:0/3692537798 shutdown_connections 2024-04-20T21:57:54.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:54.396+0000 7f636fe00700 1 --2- 172.21.15.62:0/3692537798 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f635006ecc0 0x7f6350071170 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:57:54.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:54.396+0000 7f636fe00700 1 --2- 172.21.15.62:0/3692537798 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6368106b80 0x7f636819c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:57:54.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:54.396+0000 7f636fe00700 1 --2- 172.21.15.62:0/3692537798 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f636810c8f0 0x7f636819c980 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:57:54.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:54.396+0000 7f636fe00700 1 -- 172.21.15.62:0/3692537798 >> 172.21.15.62:0/3692537798 conn(0x7f6368076060 msgr2=0x7f6368077ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:57:54.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:54.397+0000 7f636fe00700 1 -- 172.21.15.62:0/3692537798 shutdown_connections 2024-04-20T21:57:54.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:54.397+0000 7f636fe00700 1 -- 172.21.15.62:0/3692537798 wait complete. 2024-04-20T21:57:54.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:54.565+0000 7fdd4d000700 1 -- 172.21.15.62:0/1286953815 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd48108470 msgr2=0x7fdd48108840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:57:54.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:54.565+0000 7fdd4d000700 1 --2- 172.21.15.62:0/1286953815 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd48108470 0x7fdd48108840 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fdd38009910 tx=0x7fdd38009c20 comp rx=0 tx=0).stop 2024-04-20T21:57:54.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:54.565+0000 7fdd4d000700 1 -- 172.21.15.62:0/1286953815 shutdown_connections 2024-04-20T21:57:54.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:54.565+0000 7fdd4d000700 1 --2- 172.21.15.62:0/1286953815 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd48102470 0x7fdd481028e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:57:54.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:54.565+0000 7fdd4d000700 1 --2- 172.21.15.62:0/1286953815 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd48108470 0x7fdd48108840 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:57:54.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:54.565+0000 7fdd4d000700 1 -- 172.21.15.62:0/1286953815 >> 172.21.15.62:0/1286953815 conn(0x7fdd480fdf90 msgr2=0x7fdd481003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:57:54.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:54.565+0000 7fdd4d000700 1 -- 172.21.15.62:0/1286953815 shutdown_connections 2024-04-20T21:57:54.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:54.565+0000 7fdd4d000700 1 -- 172.21.15.62:0/1286953815 wait complete. 2024-04-20T21:57:54.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:54.566+0000 7fdd4d000700 1 Processor -- start 2024-04-20T21:57:54.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:54.567+0000 7fdd4d000700 1 -- start start 2024-04-20T21:57:54.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:54.567+0000 7fdd4d000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd48102470 0x7fdd48198210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:57:54.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:54.567+0000 7fdd4d000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd48108470 0x7fdd48198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:57:54.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:54.567+0000 7fdd4d000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd48198e30 con 0x7fdd48108470 2024-04-20T21:57:54.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:54.567+0000 7fdd4d000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd4819cbc0 con 0x7fdd48102470 2024-04-20T21:57:54.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:54.567+0000 7fdd45400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd48108470 0x7fdd48198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:57:54.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:54.567+0000 7fdd45400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd48108470 0x7fdd48198750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56960/0 (socket says 172.21.15.62:56960) 2024-04-20T21:57:54.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:54.567+0000 7fdd45400700 1 -- 172.21.15.62:0/1501353822 learned_addr learned my addr 172.21.15.62:0/1501353822 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:57:54.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:54.568+0000 7fdd45e00700 1 --2- 172.21.15.62:0/1501353822 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd48102470 0x7fdd48198210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:57:54.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:54.568+0000 7fdd45400700 1 -- 172.21.15.62:0/1501353822 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd48102470 msgr2=0x7fdd48198210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:57:54.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:54.568+0000 7fdd45400700 1 --2- 172.21.15.62:0/1501353822 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd48102470 0x7fdd48198210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:57:54.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:54.568+0000 7fdd45400700 1 -- 172.21.15.62:0/1501353822 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd380095f0 con 0x7fdd48108470 2024-04-20T21:57:54.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:54.568+0000 7fdd45400700 1 --2- 172.21.15.62:0/1501353822 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd48108470 0x7fdd48198750 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fdd3400b4b0 tx=0x7fdd3400b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:57:54.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:54.568+0000 7fdd3ea00700 1 -- 172.21.15.62:0/1501353822 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdd34011800 con 0x7fdd48108470 2024-04-20T21:57:54.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:54.568+0000 7fdd3ea00700 1 -- 172.21.15.62:0/1501353822 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdd34011e20 con 0x7fdd48108470 2024-04-20T21:57:54.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:54.569+0000 7fdd4d000700 1 -- 172.21.15.62:0/1501353822 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd4819cea0 con 0x7fdd48108470 2024-04-20T21:57:54.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:54.569+0000 7fdd3ea00700 1 -- 172.21.15.62:0/1501353822 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdd3400f2e0 con 0x7fdd48108470 2024-04-20T21:57:54.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:54.569+0000 7fdd4d000700 1 -- 172.21.15.62:0/1501353822 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd4819d3f0 con 0x7fdd48108470 2024-04-20T21:57:54.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:54.570+0000 7fdd4d000700 1 -- 172.21.15.62:0/1501353822 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd4810a9e0 con 0x7fdd48108470 2024-04-20T21:57:54.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:54.571+0000 7fdd3ea00700 1 -- 172.21.15.62:0/1501353822 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fdd34010380 con 0x7fdd48108470 2024-04-20T21:57:54.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:54.572+0000 7fdd3ea00700 1 --2- 172.21.15.62:0/1501353822 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fdd2c06ecc0 0x7fdd2c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:57:54.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:54.572+0000 7fdd3ea00700 1 -- 172.21.15.62:0/1501353822 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fdd3408eaa0 con 0x7fdd48108470 2024-04-20T21:57:54.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:54.573+0000 7fdd45e00700 1 --2- 172.21.15.62:0/1501353822 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fdd2c06ecc0 0x7fdd2c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:57:54.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:54.573+0000 7fdd45e00700 1 --2- 172.21.15.62:0/1501353822 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fdd2c06ecc0 0x7fdd2c071170 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fdd38006010 tx=0x7fdd3801a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:57:54.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:54.577+0000 7fdd3ea00700 1 -- 172.21.15.62:0/1501353822 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fdd340577b0 con 0x7fdd48108470 2024-04-20T21:57:54.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:57:54 smithi189 ceph-mon[57022]: pgmap v678: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:57:54.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:57:54 smithi189 ceph-mon[57022]: osdmap e84: 6 total, 6 up, 6 in 2024-04-20T21:57:54.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:57:54 smithi189 ceph-mon[57022]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-04-20T21:57:54.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:57:54 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/3183163771' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:57:54.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:57:54 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/2148400622' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T21:57:54.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:57:54 smithi062 ceph-mon[76322]: pgmap v678: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:57:54.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:57:54 smithi062 ceph-mon[76322]: osdmap e84: 6 total, 6 up, 6 in 2024-04-20T21:57:54.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:57:54 smithi062 ceph-mon[76322]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-04-20T21:57:54.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:57:54 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/3183163771' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:57:54.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:57:54 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/2148400622' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T21:57:54.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:54.905+0000 7fdd4d000700 1 -- 172.21.15.62:0/1501353822 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fdd4804f2a0 con 0x7fdd48108470 2024-04-20T21:57:54.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:54.906+0000 7fdd3ea00700 1 -- 172.21.15.62:0/1501353822 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7fdd3405ae70 con 0x7fdd48108470 2024-04-20T21:57:54.906 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-20T21:57:54.906 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-20T21:57:54.906 INFO:teuthology.orchestra.run.smithi062.stdout: host smithi062 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T21:57:54.906 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-20T21:57:54.906 INFO:teuthology.orchestra.run.smithi062.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi062 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpq7y7oa_r:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T21:57:54.906 INFO:teuthology.orchestra.run.smithi062.stdout:Traceback (most recent call last): 2024-04-20T21:57:54.906 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-20T21:57:54.906 INFO:teuthology.orchestra.run.smithi062.stdout: return _run_code(code, main_globals, None, 2024-04-20T21:57:54.906 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-20T21:57:54.906 INFO:teuthology.orchestra.run.smithi062.stdout: exec(code, run_globals) 2024-04-20T21:57:54.906 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5618, in 2024-04-20T21:57:54.906 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5606, in main 2024-04-20T21:57:54.907 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 408, in _infer_config 2024-04-20T21:57:54.907 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 323, in _infer_fsid 2024-04-20T21:57:54.907 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 436, in _infer_image 2024-04-20T21:57:54.907 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 310, in _validate_fsid 2024-04-20T21:57:54.907 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 3359, in command_ceph_volume 2024-04-20T21:57:54.907 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-20T21:57:54.907 INFO:teuthology.orchestra.run.smithi062.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi062 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpq7y7oa_r:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T21:57:54.907 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-20T21:57:54.907 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is degraded 2024-04-20T21:57:54.907 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T21:57:54.907 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T21:57:54.907 INFO:teuthology.orchestra.run.smithi062.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-20T21:57:54.907 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-20T21:57:54.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:54.909+0000 7fdd4d000700 1 -- 172.21.15.62:0/1501353822 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fdd2c06ecc0 msgr2=0x7fdd2c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:57:54.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:54.909+0000 7fdd4d000700 1 --2- 172.21.15.62:0/1501353822 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fdd2c06ecc0 0x7fdd2c071170 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fdd38006010 tx=0x7fdd3801a040 comp rx=0 tx=0).stop 2024-04-20T21:57:54.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:54.909+0000 7fdd4d000700 1 -- 172.21.15.62:0/1501353822 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd48108470 msgr2=0x7fdd48198750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:57:54.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:54.909+0000 7fdd4d000700 1 --2- 172.21.15.62:0/1501353822 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd48108470 0x7fdd48198750 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fdd3400b4b0 tx=0x7fdd3400b870 comp rx=0 tx=0).stop 2024-04-20T21:57:54.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:54.909+0000 7fdd4d000700 1 -- 172.21.15.62:0/1501353822 shutdown_connections 2024-04-20T21:57:54.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:54.909+0000 7fdd4d000700 1 --2- 172.21.15.62:0/1501353822 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fdd2c06ecc0 0x7fdd2c071170 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:57:54.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:54.909+0000 7fdd4d000700 1 --2- 172.21.15.62:0/1501353822 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd48102470 0x7fdd48198210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:57:54.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:54.909+0000 7fdd4d000700 1 --2- 172.21.15.62:0/1501353822 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd48108470 0x7fdd48198750 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:57:54.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:54.909+0000 7fdd4d000700 1 -- 172.21.15.62:0/1501353822 >> 172.21.15.62:0/1501353822 conn(0x7fdd480fdf90 msgr2=0x7fdd480febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:57:54.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:54.909+0000 7fdd4d000700 1 -- 172.21.15.62:0/1501353822 shutdown_connections 2024-04-20T21:57:54.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:57:54.909+0000 7fdd4d000700 1 -- 172.21.15.62:0/1501353822 wait complete. 2024-04-20T21:57:55.697 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:57:55 smithi189 ceph-mon[57022]: from='client.35256 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:57:55.698 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:57:55 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/1501353822' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T21:57:55.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:57:55 smithi062 ceph-mon[76322]: from='client.35256 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:57:55.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:57:55 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/1501353822' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T21:57:56.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:57:56 smithi189 ceph-mon[57022]: pgmap v680: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:57:56.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:57:56 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:57:56.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:57:56 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:57:56.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:57:56 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:57:56.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:57:56 smithi062 ceph-mon[76322]: pgmap v680: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:57:56.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:57:56 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:57:56.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:57:56 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:57:56.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:57:56 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T21:57:57.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:57:57 smithi189 ceph-mon[57022]: mds.? [v2:172.21.15.189:6824/3576699232,v1:172.21.15.189:6825/3576699232] up:boot 2024-04-20T21:57:57.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:57:57 smithi189 ceph-mon[57022]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-20T21:57:57.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:57:57 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi189.dvimer"}]: dispatch 2024-04-20T21:57:57.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:57:57 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:57:57.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:57:57 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:57:57.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:57:57 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:57:57.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:57:57 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:57:57.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:57:57 smithi062 ceph-mon[76322]: mds.? [v2:172.21.15.189:6824/3576699232,v1:172.21.15.189:6825/3576699232] up:boot 2024-04-20T21:57:57.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:57:57 smithi062 ceph-mon[76322]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-20T21:57:57.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:57:57 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi189.dvimer"}]: dispatch 2024-04-20T21:57:57.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:57:57 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:57:57.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:57:57 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:57:57.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:57:57 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:57:57.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:57:57 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:57:58.723 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:57:58 smithi189 ceph-mon[57022]: pgmap v681: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:57:58.726 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:57:58 smithi062 ceph-mon[76322]: pgmap v681: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:58:00.559 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:58:00 smithi189 ceph-mon[57022]: pgmap v682: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:58:00.714 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:58:00 smithi062 ceph-mon[76322]: pgmap v682: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:58:02.886 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:58:02 smithi062 ceph-mon[76322]: pgmap v683: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:58:02.886 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:58:02 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:58:02.886 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:58:02 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:58:02.951 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:58:02 smithi189 ceph-mon[57022]: pgmap v683: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:58:02.951 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:58:02 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:58:02.951 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:58:02 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:58:04.853 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:58:04 smithi189 ceph-mon[57022]: pgmap v684: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:58:04.854 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:58:04 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:58:04.854 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:58:04 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:58:04.854 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:58:04 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:58:05.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:58:04 smithi062 ceph-mon[76322]: pgmap v684: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:58:05.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:58:04 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:58:05.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:58:04 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:58:05.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:58:04 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:58:05.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:58:05 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:58:05.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:58:05 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:58:06.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:58:05 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:58:06.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:58:05 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T21:58:07.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:58:06 smithi189 ceph-mon[57022]: pgmap v685: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:58:07.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:58:06 smithi062 ceph-mon[76322]: pgmap v685: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:58:09.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:58:08 smithi189 ceph-mon[57022]: pgmap v686: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:58:09.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:58:08 smithi062 ceph-mon[76322]: pgmap v686: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:58:10.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:58:10 smithi189 ceph-mon[57022]: pgmap v687: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:58:11.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:58:10 smithi062 ceph-mon[76322]: pgmap v687: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:58:13.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:58:12 smithi189 ceph-mon[57022]: pgmap v688: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:58:13.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:58:12 smithi062 ceph-mon[76322]: pgmap v688: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:58:14.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:58:14 smithi189 ceph-mon[57022]: pgmap v689: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:58:15.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:58:14 smithi062 ceph-mon[76322]: pgmap v689: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:58:17.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:58:16 smithi189 ceph-mon[57022]: pgmap v690: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:58:17.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:58:16 smithi062 ceph-mon[76322]: pgmap v690: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:58:19.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:58:18 smithi062 ceph-mon[76322]: pgmap v691: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:58:19.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:58:18 smithi189 ceph-mon[57022]: pgmap v691: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:58:20.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:58:19 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:58:20.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:58:19 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:58:21.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:58:20 smithi189 ceph-mon[57022]: pgmap v692: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:58:21.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:58:20 smithi062 ceph-mon[76322]: pgmap v692: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:58:23.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:58:22 smithi189 ceph-mon[57022]: pgmap v693: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:58:23.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:58:22 smithi062 ceph-mon[76322]: pgmap v693: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:58:25.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.090+0000 7f342a200700 1 -- 172.21.15.62:0/3381797665 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3424108470 msgr2=0x7f3424108840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:58:25.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.090+0000 7f342a200700 1 --2- 172.21.15.62:0/3381797665 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3424108470 0x7f3424108840 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f3414009910 tx=0x7f3414009c20 comp rx=0 tx=0).stop 2024-04-20T21:58:25.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.091+0000 7f342a200700 1 -- 172.21.15.62:0/3381797665 shutdown_connections 2024-04-20T21:58:25.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.091+0000 7f342a200700 1 --2- 172.21.15.62:0/3381797665 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3424102470 0x7f34241028e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:58:25.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.091+0000 7f342a200700 1 --2- 172.21.15.62:0/3381797665 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3424108470 0x7f3424108840 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:58:25.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.091+0000 7f342a200700 1 -- 172.21.15.62:0/3381797665 >> 172.21.15.62:0/3381797665 conn(0x7f34240fdf90 msgr2=0x7f34241003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:58:25.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.091+0000 7f342a200700 1 -- 172.21.15.62:0/3381797665 shutdown_connections 2024-04-20T21:58:25.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.091+0000 7f342a200700 1 -- 172.21.15.62:0/3381797665 wait complete. 2024-04-20T21:58:25.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.092+0000 7f342a200700 1 Processor -- start 2024-04-20T21:58:25.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.092+0000 7f342a200700 1 -- start start 2024-04-20T21:58:25.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.092+0000 7f342a200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3424102470 0x7f34241981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:58:25.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.093+0000 7f342a200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3424108470 0x7f3424198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:58:25.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.093+0000 7f342a200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3424198e10 con 0x7f3424108470 2024-04-20T21:58:25.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.093+0000 7f342a200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f342419cba0 con 0x7f3424102470 2024-04-20T21:58:25.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.093+0000 7f3422a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3424108470 0x7f3424198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:58:25.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.093+0000 7f3422a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3424108470 0x7f3424198730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43312/0 (socket says 172.21.15.62:43312) 2024-04-20T21:58:25.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.093+0000 7f3422a00700 1 -- 172.21.15.62:0/2611111346 learned_addr learned my addr 172.21.15.62:0/2611111346 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:58:25.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.093+0000 7f3423400700 1 --2- 172.21.15.62:0/2611111346 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3424102470 0x7f34241981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:58:25.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.093+0000 7f3422a00700 1 -- 172.21.15.62:0/2611111346 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3424102470 msgr2=0x7f34241981f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:58:25.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.093+0000 7f3422a00700 1 --2- 172.21.15.62:0/2611111346 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3424102470 0x7f34241981f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:58:25.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.093+0000 7f3422a00700 1 -- 172.21.15.62:0/2611111346 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34140095f0 con 0x7f3424108470 2024-04-20T21:58:25.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.094+0000 7f3422a00700 1 --2- 172.21.15.62:0/2611111346 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3424108470 0x7f3424198730 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f341000c8a0 tx=0x7f341000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:58:25.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.094+0000 7f341be00700 1 -- 172.21.15.62:0/2611111346 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f341000cea0 con 0x7f3424108470 2024-04-20T21:58:25.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.094+0000 7f341be00700 1 -- 172.21.15.62:0/2611111346 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3410004830 con 0x7f3424108470 2024-04-20T21:58:25.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.094+0000 7f341be00700 1 -- 172.21.15.62:0/2611111346 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f34100055b0 con 0x7f3424108470 2024-04-20T21:58:25.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.094+0000 7f342a200700 1 -- 172.21.15.62:0/2611111346 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f342419ce80 con 0x7f3424108470 2024-04-20T21:58:25.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.094+0000 7f342a200700 1 -- 172.21.15.62:0/2611111346 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f342419d3d0 con 0x7f3424108470 2024-04-20T21:58:25.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.095+0000 7f342a200700 1 -- 172.21.15.62:0/2611111346 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f342410a9e0 con 0x7f3424108470 2024-04-20T21:58:25.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.096+0000 7f341be00700 1 -- 172.21.15.62:0/2611111346 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f34100049a0 con 0x7f3424108470 2024-04-20T21:58:25.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.097+0000 7f341be00700 1 --2- 172.21.15.62:0/2611111346 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f340806ecc0 0x7f3408071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:58:25.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.098+0000 7f341be00700 1 -- 172.21.15.62:0/2611111346 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f341008eaf0 con 0x7f3424108470 2024-04-20T21:58:25.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.098+0000 7f3423400700 1 --2- 172.21.15.62:0/2611111346 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f340806ecc0 0x7f3408071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:58:25.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.099+0000 7f3423400700 1 --2- 172.21.15.62:0/2611111346 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f340806ecc0 0x7f3408071170 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f3414006010 tx=0x7f341401a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:58:25.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.103+0000 7f341be00700 1 -- 172.21.15.62:0/2611111346 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f3410057880 con 0x7f3424108470 2024-04-20T21:58:25.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:58:24 smithi189 ceph-mon[57022]: pgmap v694: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:58:25.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:58:24 smithi062 ceph-mon[76322]: pgmap v694: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:58:25.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.366+0000 7f342a200700 1 -- 172.21.15.62:0/2611111346 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3424199660 con 0x7f340806ecc0 2024-04-20T21:58:25.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.368+0000 7f341be00700 1 -- 172.21.15.62:0/2611111346 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f3424199660 con 0x7f340806ecc0 2024-04-20T21:58:25.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.371+0000 7f342a200700 1 -- 172.21.15.62:0/2611111346 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f340806ecc0 msgr2=0x7f3408071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:58:25.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.371+0000 7f342a200700 1 --2- 172.21.15.62:0/2611111346 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f340806ecc0 0x7f3408071170 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f3414006010 tx=0x7f341401a040 comp rx=0 tx=0).stop 2024-04-20T21:58:25.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.371+0000 7f342a200700 1 -- 172.21.15.62:0/2611111346 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3424108470 msgr2=0x7f3424198730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:58:25.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.371+0000 7f342a200700 1 --2- 172.21.15.62:0/2611111346 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3424108470 0x7f3424198730 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f341000c8a0 tx=0x7f341000cc60 comp rx=0 tx=0).stop 2024-04-20T21:58:25.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.371+0000 7f342a200700 1 -- 172.21.15.62:0/2611111346 shutdown_connections 2024-04-20T21:58:25.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.371+0000 7f342a200700 1 --2- 172.21.15.62:0/2611111346 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f340806ecc0 0x7f3408071170 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:58:25.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.371+0000 7f342a200700 1 --2- 172.21.15.62:0/2611111346 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3424102470 0x7f34241981f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:58:25.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.371+0000 7f342a200700 1 --2- 172.21.15.62:0/2611111346 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3424108470 0x7f3424198730 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:58:25.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.371+0000 7f342a200700 1 -- 172.21.15.62:0/2611111346 >> 172.21.15.62:0/2611111346 conn(0x7f34240fdf90 msgr2=0x7f34241002c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:58:25.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.371+0000 7f342a200700 1 -- 172.21.15.62:0/2611111346 shutdown_connections 2024-04-20T21:58:25.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.371+0000 7f342a200700 1 -- 172.21.15.62:0/2611111346 wait complete. 2024-04-20T21:58:25.388 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T21:58:25.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.537+0000 7f8b53600700 1 -- 172.21.15.62:0/3403431742 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8b4c10c8f0 msgr2=0x7f8b4c10ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:58:25.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.537+0000 7f8b53600700 1 --2- 172.21.15.62:0/3403431742 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8b4c10c8f0 0x7f8b4c10ccc0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f8b34009910 tx=0x7f8b34009c20 comp rx=0 tx=0).stop 2024-04-20T21:58:25.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.538+0000 7f8b53600700 1 -- 172.21.15.62:0/3403431742 shutdown_connections 2024-04-20T21:58:25.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.538+0000 7f8b53600700 1 --2- 172.21.15.62:0/3403431742 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8b4c106b80 0x7f8b4c106ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:58:25.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.538+0000 7f8b53600700 1 --2- 172.21.15.62:0/3403431742 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8b4c10c8f0 0x7f8b4c10ccc0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:58:25.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.538+0000 7f8b53600700 1 -- 172.21.15.62:0/3403431742 >> 172.21.15.62:0/3403431742 conn(0x7f8b4c076060 msgr2=0x7f8b4c078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:58:25.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.538+0000 7f8b53600700 1 -- 172.21.15.62:0/3403431742 shutdown_connections 2024-04-20T21:58:25.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.538+0000 7f8b53600700 1 -- 172.21.15.62:0/3403431742 wait complete. 2024-04-20T21:58:25.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.539+0000 7f8b53600700 1 Processor -- start 2024-04-20T21:58:25.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.539+0000 7f8b53600700 1 -- start start 2024-04-20T21:58:25.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.539+0000 7f8b53600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8b4c106b80 0x7f8b4c19c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:58:25.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.539+0000 7f8b53600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8b4c10c8f0 0x7f8b4c19ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:58:25.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.539+0000 7f8b53600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b4c19d120 con 0x7f8b4c106b80 2024-04-20T21:58:25.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.539+0000 7f8b53600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b4c1a0eb0 con 0x7f8b4c10c8f0 2024-04-20T21:58:25.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.540+0000 7f8b50c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8b4c106b80 0x7f8b4c19c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:58:25.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.540+0000 7f8b50c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8b4c106b80 0x7f8b4c19c500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43330/0 (socket says 172.21.15.62:43330) 2024-04-20T21:58:25.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.540+0000 7f8b50c00700 1 -- 172.21.15.62:0/3389919527 learned_addr learned my addr 172.21.15.62:0/3389919527 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:58:25.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.540+0000 7f8b43e00700 1 --2- 172.21.15.62:0/3389919527 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8b4c10c8f0 0x7f8b4c19ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:58:25.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.540+0000 7f8b50c00700 1 -- 172.21.15.62:0/3389919527 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8b4c10c8f0 msgr2=0x7f8b4c19ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:58:25.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.540+0000 7f8b50c00700 1 --2- 172.21.15.62:0/3389919527 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8b4c10c8f0 0x7f8b4c19ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:58:25.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.540+0000 7f8b50c00700 1 -- 172.21.15.62:0/3389919527 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b340095f0 con 0x7f8b4c106b80 2024-04-20T21:58:25.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.540+0000 7f8b43e00700 1 --2- 172.21.15.62:0/3389919527 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8b4c10c8f0 0x7f8b4c19ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T21:58:25.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.541+0000 7f8b50c00700 1 --2- 172.21.15.62:0/3389919527 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8b4c106b80 0x7f8b4c19c500 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f8b3400f690 tx=0x7f8b3400f770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:58:25.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.541+0000 7f8b4a000700 1 -- 172.21.15.62:0/3389919527 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8b3401d070 con 0x7f8b4c106b80 2024-04-20T21:58:25.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.541+0000 7f8b4a000700 1 -- 172.21.15.62:0/3389919527 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8b340050d0 con 0x7f8b4c106b80 2024-04-20T21:58:25.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.541+0000 7f8b4a000700 1 -- 172.21.15.62:0/3389919527 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8b34022470 con 0x7f8b4c106b80 2024-04-20T21:58:25.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.541+0000 7f8b53600700 1 -- 172.21.15.62:0/3389919527 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b4c1a1130 con 0x7f8b4c106b80 2024-04-20T21:58:25.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.541+0000 7f8b53600700 1 -- 172.21.15.62:0/3389919527 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b4c1a1620 con 0x7f8b4c106b80 2024-04-20T21:58:25.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.542+0000 7f8b53600700 1 -- 172.21.15.62:0/3389919527 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b4c109b20 con 0x7f8b4c106b80 2024-04-20T21:58:25.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.543+0000 7f8b4a000700 1 -- 172.21.15.62:0/3389919527 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f8b34003680 con 0x7f8b4c106b80 2024-04-20T21:58:25.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.544+0000 7f8b4a000700 1 --2- 172.21.15.62:0/3389919527 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f8b3006ecc0 0x7f8b30071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:58:25.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.544+0000 7f8b4a000700 1 -- 172.21.15.62:0/3389919527 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f8b340902e0 con 0x7f8b4c106b80 2024-04-20T21:58:25.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.545+0000 7f8b43e00700 1 --2- 172.21.15.62:0/3389919527 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f8b3006ecc0 0x7f8b30071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:58:25.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.545+0000 7f8b43e00700 1 --2- 172.21.15.62:0/3389919527 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f8b3006ecc0 0x7f8b30071170 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f8b4c19db20 tx=0x7f8b3c009260 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:58:25.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.549+0000 7f8b4a000700 1 -- 172.21.15.62:0/3389919527 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f8b34059070 con 0x7f8b4c106b80 2024-04-20T21:58:25.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.812+0000 7f8b53600700 1 -- 172.21.15.62:0/3389919527 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8b4c1a1890 con 0x7f8b3006ecc0 2024-04-20T21:58:25.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.813+0000 7f8b4a000700 1 -- 172.21.15.62:0/3389919527 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f8b4c1a1890 con 0x7f8b3006ecc0 2024-04-20T21:58:25.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.816+0000 7f8b53600700 1 -- 172.21.15.62:0/3389919527 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f8b3006ecc0 msgr2=0x7f8b30071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:58:25.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.816+0000 7f8b53600700 1 --2- 172.21.15.62:0/3389919527 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f8b3006ecc0 0x7f8b30071170 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f8b4c19db20 tx=0x7f8b3c009260 comp rx=0 tx=0).stop 2024-04-20T21:58:25.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.816+0000 7f8b53600700 1 -- 172.21.15.62:0/3389919527 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8b4c106b80 msgr2=0x7f8b4c19c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:58:25.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.816+0000 7f8b53600700 1 --2- 172.21.15.62:0/3389919527 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8b4c106b80 0x7f8b4c19c500 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f8b3400f690 tx=0x7f8b3400f770 comp rx=0 tx=0).stop 2024-04-20T21:58:25.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.816+0000 7f8b53600700 1 -- 172.21.15.62:0/3389919527 shutdown_connections 2024-04-20T21:58:25.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.816+0000 7f8b53600700 1 --2- 172.21.15.62:0/3389919527 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f8b3006ecc0 0x7f8b30071170 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:58:25.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.816+0000 7f8b53600700 1 --2- 172.21.15.62:0/3389919527 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8b4c106b80 0x7f8b4c19c500 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:58:25.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.816+0000 7f8b53600700 1 --2- 172.21.15.62:0/3389919527 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8b4c10c8f0 0x7f8b4c19ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:58:25.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.816+0000 7f8b53600700 1 -- 172.21.15.62:0/3389919527 >> 172.21.15.62:0/3389919527 conn(0x7f8b4c076060 msgr2=0x7f8b4c077aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:58:25.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.816+0000 7f8b53600700 1 -- 172.21.15.62:0/3389919527 shutdown_connections 2024-04-20T21:58:25.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.817+0000 7f8b53600700 1 -- 172.21.15.62:0/3389919527 wait complete. 2024-04-20T21:58:25.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.982+0000 7fc195200700 1 -- 172.21.15.62:0/1970052379 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc19010c660 msgr2=0x7fc19010ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:58:25.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.982+0000 7fc195200700 1 --2- 172.21.15.62:0/1970052379 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc19010c660 0x7fc19010ca30 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fc180009910 tx=0x7fc180009c20 comp rx=0 tx=0).stop 2024-04-20T21:58:25.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.982+0000 7fc195200700 1 -- 172.21.15.62:0/1970052379 shutdown_connections 2024-04-20T21:58:25.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.982+0000 7fc195200700 1 --2- 172.21.15.62:0/1970052379 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc190106b80 0x7fc190106ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:58:25.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.982+0000 7fc195200700 1 --2- 172.21.15.62:0/1970052379 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc19010c660 0x7fc19010ca30 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:58:25.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.982+0000 7fc195200700 1 -- 172.21.15.62:0/1970052379 >> 172.21.15.62:0/1970052379 conn(0x7fc190075e90 msgr2=0x7fc1900782a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:58:25.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.982+0000 7fc195200700 1 -- 172.21.15.62:0/1970052379 shutdown_connections 2024-04-20T21:58:25.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.982+0000 7fc195200700 1 -- 172.21.15.62:0/1970052379 wait complete. 2024-04-20T21:58:25.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.983+0000 7fc195200700 1 Processor -- start 2024-04-20T21:58:25.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.983+0000 7fc195200700 1 -- start start 2024-04-20T21:58:25.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.984+0000 7fc195200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc190106b80 0x7fc19019c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:58:25.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.984+0000 7fc195200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc19010c660 0x7fc19019c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:58:25.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.984+0000 7fc195200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc19019cea0 con 0x7fc190106b80 2024-04-20T21:58:25.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.984+0000 7fc195200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1901a0bc0 con 0x7fc19010c660 2024-04-20T21:58:25.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.984+0000 7fc18f400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc190106b80 0x7fc19019c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:58:25.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.984+0000 7fc18f400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc190106b80 0x7fc19019c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43354/0 (socket says 172.21.15.62:43354) 2024-04-20T21:58:25.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.984+0000 7fc18f400700 1 -- 172.21.15.62:0/2140852654 learned_addr learned my addr 172.21.15.62:0/2140852654 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:58:25.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.984+0000 7fc18ea00700 1 --2- 172.21.15.62:0/2140852654 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc19010c660 0x7fc19019c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:58:25.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.984+0000 7fc18f400700 1 -- 172.21.15.62:0/2140852654 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc19010c660 msgr2=0x7fc19019c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:58:25.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.984+0000 7fc18f400700 1 --2- 172.21.15.62:0/2140852654 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc19010c660 0x7fc19019c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:58:25.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.984+0000 7fc18f400700 1 -- 172.21.15.62:0/2140852654 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1800095f0 con 0x7fc190106b80 2024-04-20T21:58:25.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.984+0000 7fc18ea00700 1 --2- 172.21.15.62:0/2140852654 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc19010c660 0x7fc19019c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T21:58:25.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.985+0000 7fc18f400700 1 --2- 172.21.15.62:0/2140852654 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc190106b80 0x7fc19019c280 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fc18000b5c0 tx=0x7fc18000be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:58:25.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.985+0000 7fc187e00700 1 -- 172.21.15.62:0/2140852654 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc18001d070 con 0x7fc190106b80 2024-04-20T21:58:25.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.985+0000 7fc187e00700 1 -- 172.21.15.62:0/2140852654 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc18000f460 con 0x7fc190106b80 2024-04-20T21:58:25.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.985+0000 7fc195200700 1 -- 172.21.15.62:0/2140852654 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1901a0e40 con 0x7fc190106b80 2024-04-20T21:58:25.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.985+0000 7fc187e00700 1 -- 172.21.15.62:0/2140852654 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc180017610 con 0x7fc190106b80 2024-04-20T21:58:25.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.985+0000 7fc195200700 1 -- 172.21.15.62:0/2140852654 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1901a1330 con 0x7fc190106b80 2024-04-20T21:58:25.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.986+0000 7fc195200700 1 -- 172.21.15.62:0/2140852654 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc19010eb60 con 0x7fc190106b80 2024-04-20T21:58:25.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.988+0000 7fc187e00700 1 -- 172.21.15.62:0/2140852654 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fc18000f5d0 con 0x7fc190106b80 2024-04-20T21:58:25.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.989+0000 7fc187e00700 1 --2- 172.21.15.62:0/2140852654 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc17406ecc0 0x7fc174071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:58:25.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.989+0000 7fc187e00700 1 -- 172.21.15.62:0/2140852654 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fc180090450 con 0x7fc190106b80 2024-04-20T21:58:25.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.989+0000 7fc18ea00700 1 --2- 172.21.15.62:0/2140852654 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc17406ecc0 0x7fc174071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:58:25.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.990+0000 7fc18ea00700 1 --2- 172.21.15.62:0/2140852654 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc17406ecc0 0x7fc174071170 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fc17c005fd0 tx=0x7fc17c005dc0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:58:25.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:25.994+0000 7fc187e00700 1 -- 172.21.15.62:0/2140852654 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fc1800591e0 con 0x7fc190106b80 2024-04-20T21:58:26.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:26.238+0000 7fc195200700 1 -- 172.21.15.62:0/2140852654 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fc1901a15a0 con 0x7fc17406ecc0 2024-04-20T21:58:26.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:26.251+0000 7fc187e00700 1 -- 172.21.15.62:0/2140852654 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fc1901a15a0 con 0x7fc17406ecc0 2024-04-20T21:58:26.251 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T21:58:26.251 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (24m) 21s ago 38m 15.3M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T21:58:26.251 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (38m) 21s ago 38m 10.4M - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T21:58:26.251 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (37m) 24s ago 37m 10.1M - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T21:58:26.251 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (20m) 21s ago 38m 6660k - 19.0.0-3133-ga8c84f59 76bc6b91b726 2825d5ba11c6 2024-04-20T21:58:26.251 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (20m) 24s ago 37m 6665k - 19.0.0-3133-ga8c84f59 76bc6b91b726 20f746fa1bba 2024-04-20T21:58:26.251 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (23m) 21s ago 37m 73.5M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T21:58:26.251 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (15m) 21s ago 30m 18.5M - 19.0.0-3133-ga8c84f59 76bc6b91b726 de80bc90a683 2024-04-20T21:58:26.251 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (16m) 21s ago 31m 18.7M - 19.0.0-3133-ga8c84f59 76bc6b91b726 cd793b7ee98e 2024-04-20T21:58:26.251 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (30s) 24s ago 30m 11.0M - 19.0.0-3133-ga8c84f59 76bc6b91b726 0e7b0bc4d1ab 2024-04-20T21:58:26.252 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (30m) 24s ago 30m 18.6M - 18.2.0 dc2bc1663786 f2eec2ee8951 2024-04-20T21:58:26.252 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (27m) 21s ago 39m 513M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T21:58:26.252 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (25m) 24s ago 36m 459M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T21:58:26.252 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (22m) 21s ago 40m 71.7M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 019eb1479485 2024-04-20T21:58:26.252 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (21m) 24s ago 36m 51.0M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 55aac560aab6 2024-04-20T21:58:26.252 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (25m) 21s ago 38m 8472k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T21:58:26.252 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (25m) 24s ago 36m 8048k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T21:58:26.252 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (19m) 21s ago 35m 91.2M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 ee2bb901f26e 2024-04-20T21:58:26.252 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (19m) 21s ago 35m 94.6M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 9075152d4162 2024-04-20T21:58:26.252 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (18m) 21s ago 34m 78.1M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 2d4f3e38dc0e 2024-04-20T21:58:26.252 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (18m) 24s ago 33m 92.6M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 8f97b17ae0eb 2024-04-20T21:58:26.252 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (17m) 24s ago 33m 95.6M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 653d8079967d 2024-04-20T21:58:26.252 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (16m) 24s ago 32m 89.2M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 596dded14c12 2024-04-20T21:58:26.252 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (24m) 21s ago 37m 82.1M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T21:58:26.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:26.254+0000 7fc195200700 1 -- 172.21.15.62:0/2140852654 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc17406ecc0 msgr2=0x7fc174071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:58:26.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:26.254+0000 7fc195200700 1 --2- 172.21.15.62:0/2140852654 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc17406ecc0 0x7fc174071170 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fc17c005fd0 tx=0x7fc17c005dc0 comp rx=0 tx=0).stop 2024-04-20T21:58:26.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:26.254+0000 7fc195200700 1 -- 172.21.15.62:0/2140852654 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc190106b80 msgr2=0x7fc19019c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:58:26.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:26.254+0000 7fc195200700 1 --2- 172.21.15.62:0/2140852654 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc190106b80 0x7fc19019c280 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fc18000b5c0 tx=0x7fc18000be00 comp rx=0 tx=0).stop 2024-04-20T21:58:26.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:26.254+0000 7fc195200700 1 -- 172.21.15.62:0/2140852654 shutdown_connections 2024-04-20T21:58:26.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:26.254+0000 7fc195200700 1 --2- 172.21.15.62:0/2140852654 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc17406ecc0 0x7fc174071170 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:58:26.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:26.254+0000 7fc195200700 1 --2- 172.21.15.62:0/2140852654 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc190106b80 0x7fc19019c280 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:58:26.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:26.254+0000 7fc195200700 1 --2- 172.21.15.62:0/2140852654 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc19010c660 0x7fc19019c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:58:26.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:26.254+0000 7fc195200700 1 -- 172.21.15.62:0/2140852654 >> 172.21.15.62:0/2140852654 conn(0x7fc190075e90 msgr2=0x7fc190077980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:58:26.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:26.255+0000 7fc195200700 1 -- 172.21.15.62:0/2140852654 shutdown_connections 2024-04-20T21:58:26.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:26.255+0000 7fc195200700 1 -- 172.21.15.62:0/2140852654 wait complete. 2024-04-20T21:58:26.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:26.414+0000 7fb0c5c00700 1 -- 172.21.15.62:0/912686132 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb0c0102470 msgr2=0x7fb0c01028e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:58:26.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:26.414+0000 7fb0c5c00700 1 --2- 172.21.15.62:0/912686132 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb0c0102470 0x7fb0c01028e0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fb0a8009910 tx=0x7fb0a8009c20 comp rx=0 tx=0).stop 2024-04-20T21:58:26.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:26.414+0000 7fb0c5c00700 1 -- 172.21.15.62:0/912686132 shutdown_connections 2024-04-20T21:58:26.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:26.414+0000 7fb0c5c00700 1 --2- 172.21.15.62:0/912686132 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb0c0102470 0x7fb0c01028e0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:58:26.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:26.414+0000 7fb0c5c00700 1 --2- 172.21.15.62:0/912686132 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb0c0108470 0x7fb0c0108840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:58:26.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:26.414+0000 7fb0c5c00700 1 -- 172.21.15.62:0/912686132 >> 172.21.15.62:0/912686132 conn(0x7fb0c00fdf90 msgr2=0x7fb0c01003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:58:26.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:26.414+0000 7fb0c5c00700 1 -- 172.21.15.62:0/912686132 shutdown_connections 2024-04-20T21:58:26.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:26.414+0000 7fb0c5c00700 1 -- 172.21.15.62:0/912686132 wait complete. 2024-04-20T21:58:26.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:26.415+0000 7fb0c5c00700 1 Processor -- start 2024-04-20T21:58:26.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:26.415+0000 7fb0c5c00700 1 -- start start 2024-04-20T21:58:26.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:26.416+0000 7fb0c5c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb0c0102470 0x7fb0c0198190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:58:26.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:26.416+0000 7fb0c5c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb0c0108470 0x7fb0c01986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:58:26.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:26.416+0000 7fb0c5c00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c0198db0 con 0x7fb0c0108470 2024-04-20T21:58:26.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:26.416+0000 7fb0c5c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c019cb40 con 0x7fb0c0102470 2024-04-20T21:58:26.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:26.416+0000 7fb0b6000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb0c0108470 0x7fb0c01986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:58:26.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:26.416+0000 7fb0b6000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb0c0108470 0x7fb0c01986d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43380/0 (socket says 172.21.15.62:43380) 2024-04-20T21:58:26.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:26.416+0000 7fb0b6000700 1 -- 172.21.15.62:0/2693640535 learned_addr learned my addr 172.21.15.62:0/2693640535 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:58:26.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:26.416+0000 7fb0bea00700 1 --2- 172.21.15.62:0/2693640535 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb0c0102470 0x7fb0c0198190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:58:26.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:26.416+0000 7fb0b6000700 1 -- 172.21.15.62:0/2693640535 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb0c0102470 msgr2=0x7fb0c0198190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:58:26.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:26.416+0000 7fb0b6000700 1 --2- 172.21.15.62:0/2693640535 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb0c0102470 0x7fb0c0198190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:58:26.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:26.416+0000 7fb0b6000700 1 -- 172.21.15.62:0/2693640535 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0a80095f0 con 0x7fb0c0108470 2024-04-20T21:58:26.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:26.417+0000 7fb0b6000700 1 --2- 172.21.15.62:0/2693640535 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb0c0108470 0x7fb0c01986d0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fb0a8006010 tx=0x7fb0a800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:58:26.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:26.417+0000 7fb0b7e00700 1 -- 172.21.15.62:0/2693640535 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb0a801d070 con 0x7fb0c0108470 2024-04-20T21:58:26.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:26.417+0000 7fb0b7e00700 1 -- 172.21.15.62:0/2693640535 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb0a8004fb0 con 0x7fb0c0108470 2024-04-20T21:58:26.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:26.417+0000 7fb0b7e00700 1 -- 172.21.15.62:0/2693640535 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb0a800f690 con 0x7fb0c0108470 2024-04-20T21:58:26.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:26.417+0000 7fb0c5c00700 1 -- 172.21.15.62:0/2693640535 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0c019cdc0 con 0x7fb0c0108470 2024-04-20T21:58:26.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:26.417+0000 7fb0c5c00700 1 -- 172.21.15.62:0/2693640535 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0c019d2b0 con 0x7fb0c0108470 2024-04-20T21:58:26.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:26.418+0000 7fb0c5c00700 1 -- 172.21.15.62:0/2693640535 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0c010aa00 con 0x7fb0c0108470 2024-04-20T21:58:26.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:26.420+0000 7fb0b7e00700 1 -- 172.21.15.62:0/2693640535 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fb0a80228a0 con 0x7fb0c0108470 2024-04-20T21:58:26.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:26.420+0000 7fb0b7e00700 1 --2- 172.21.15.62:0/2693640535 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb0a406ecc0 0x7fb0a4071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:58:26.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:26.421+0000 7fb0b7e00700 1 -- 172.21.15.62:0/2693640535 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fb0a8090700 con 0x7fb0c0108470 2024-04-20T21:58:26.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:26.421+0000 7fb0bea00700 1 --2- 172.21.15.62:0/2693640535 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb0a406ecc0 0x7fb0a4071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:58:26.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:26.422+0000 7fb0bea00700 1 --2- 172.21.15.62:0/2693640535 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb0a406ecc0 0x7fb0a4071170 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fb0b0009f20 tx=0x7fb0b00094f0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:58:26.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:26.425+0000 7fb0b7e00700 1 -- 172.21.15.62:0/2693640535 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fb0a8059490 con 0x7fb0c0108470 2024-04-20T21:58:26.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:26.762+0000 7fb0c5c00700 1 -- 172.21.15.62:0/2693640535 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fb0c004f2a0 con 0x7fb0c0108470 2024-04-20T21:58:26.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:26.763+0000 7fb0b7e00700 1 -- 172.21.15.62:0/2693640535 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7fb0a805cb50 con 0x7fb0c0108470 2024-04-20T21:58:26.762 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:58:26.762 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T21:58:26.763 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:58:26.763 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:58:26.763 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T21:58:26.763 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:58:26.763 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:58:26.763 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T21:58:26.763 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 6 2024-04-20T21:58:26.763 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:58:26.763 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T21:58:26.763 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 1, 2024-04-20T21:58:26.763 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 3 2024-04-20T21:58:26.763 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:58:26.763 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T21:58:26.763 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 1, 2024-04-20T21:58:26.763 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 13 2024-04-20T21:58:26.764 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T21:58:26.764 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:58:26.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:26.765+0000 7fb0c5c00700 1 -- 172.21.15.62:0/2693640535 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb0a406ecc0 msgr2=0x7fb0a4071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:58:26.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:26.765+0000 7fb0c5c00700 1 --2- 172.21.15.62:0/2693640535 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb0a406ecc0 0x7fb0a4071170 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fb0b0009f20 tx=0x7fb0b00094f0 comp rx=0 tx=0).stop 2024-04-20T21:58:26.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:26.765+0000 7fb0c5c00700 1 -- 172.21.15.62:0/2693640535 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb0c0108470 msgr2=0x7fb0c01986d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:58:26.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:26.765+0000 7fb0c5c00700 1 --2- 172.21.15.62:0/2693640535 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb0c0108470 0x7fb0c01986d0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fb0a8006010 tx=0x7fb0a800be20 comp rx=0 tx=0).stop 2024-04-20T21:58:26.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:26.766+0000 7fb0c5c00700 1 -- 172.21.15.62:0/2693640535 shutdown_connections 2024-04-20T21:58:26.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:26.766+0000 7fb0c5c00700 1 --2- 172.21.15.62:0/2693640535 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb0a406ecc0 0x7fb0a4071170 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:58:26.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:26.766+0000 7fb0c5c00700 1 --2- 172.21.15.62:0/2693640535 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb0c0102470 0x7fb0c0198190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:58:26.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:26.766+0000 7fb0c5c00700 1 --2- 172.21.15.62:0/2693640535 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb0c0108470 0x7fb0c01986d0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:58:26.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:26.766+0000 7fb0c5c00700 1 -- 172.21.15.62:0/2693640535 >> 172.21.15.62:0/2693640535 conn(0x7fb0c00fdf90 msgr2=0x7fb0c00feb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:58:26.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:26.766+0000 7fb0c5c00700 1 -- 172.21.15.62:0/2693640535 shutdown_connections 2024-04-20T21:58:26.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:26.766+0000 7fb0c5c00700 1 -- 172.21.15.62:0/2693640535 wait complete. 2024-04-20T21:58:26.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:26.927+0000 7fc00e800700 1 -- 172.21.15.62:0/1244205798 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc008077730 msgr2=0x7fc008110a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:58:26.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:26.927+0000 7fc00e800700 1 --2- 172.21.15.62:0/1244205798 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc008077730 0x7fc008110a10 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fbff4009940 tx=0x7fbff4009c50 comp rx=0 tx=0).stop 2024-04-20T21:58:26.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:26.927+0000 7fc00e800700 1 -- 172.21.15.62:0/1244205798 shutdown_connections 2024-04-20T21:58:26.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:26.927+0000 7fc00e800700 1 --2- 172.21.15.62:0/1244205798 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc008077730 0x7fc008110a10 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:58:26.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:26.927+0000 7fc00e800700 1 --2- 172.21.15.62:0/1244205798 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc008076e20 0x7fc0080771f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:58:26.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:26.927+0000 7fc00e800700 1 -- 172.21.15.62:0/1244205798 >> 172.21.15.62:0/1244205798 conn(0x7fc0080fddb0 msgr2=0x7fc0081001c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:58:26.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:26.927+0000 7fc00e800700 1 -- 172.21.15.62:0/1244205798 shutdown_connections 2024-04-20T21:58:26.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:26.927+0000 7fc00e800700 1 -- 172.21.15.62:0/1244205798 wait complete. 2024-04-20T21:58:26.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:26.928+0000 7fc00e800700 1 Processor -- start 2024-04-20T21:58:26.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:26.929+0000 7fc00e800700 1 -- start start 2024-04-20T21:58:26.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:26.929+0000 7fc00e800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc008076e20 0x7fc00819c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:58:26.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:26.929+0000 7fc00e800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc008077730 0x7fc00819c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:58:26.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:26.929+0000 7fc00e800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc00819d000 con 0x7fc008076e20 2024-04-20T21:58:26.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:26.929+0000 7fc00e800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0081a0d90 con 0x7fc008077730 2024-04-20T21:58:26.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:26.930+0000 7fc00d400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc008076e20 0x7fc00819c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:58:26.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:26.930+0000 7fc00d400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc008076e20 0x7fc00819c3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43402/0 (socket says 172.21.15.62:43402) 2024-04-20T21:58:26.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:26.930+0000 7fc00d400700 1 -- 172.21.15.62:0/780627030 learned_addr learned my addr 172.21.15.62:0/780627030 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:58:26.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:26.930+0000 7fc00ca00700 1 --2- 172.21.15.62:0/780627030 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc008077730 0x7fc00819c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:58:26.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:26.930+0000 7fc00d400700 1 -- 172.21.15.62:0/780627030 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc008077730 msgr2=0x7fc00819c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:58:26.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:26.930+0000 7fc00d400700 1 --2- 172.21.15.62:0/780627030 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc008077730 0x7fc00819c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:58:26.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:26.930+0000 7fc00d400700 1 -- 172.21.15.62:0/780627030 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbff40095f0 con 0x7fc008076e20 2024-04-20T21:58:26.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:26.930+0000 7fc00d400700 1 --2- 172.21.15.62:0/780627030 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc008076e20 0x7fc00819c3e0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fbffc00b4b0 tx=0x7fbffc00b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:58:26.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:26.931+0000 7fc006000700 1 -- 172.21.15.62:0/780627030 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbffc011800 con 0x7fc008076e20 2024-04-20T21:58:26.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:26.931+0000 7fc006000700 1 -- 172.21.15.62:0/780627030 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbffc011e20 con 0x7fc008076e20 2024-04-20T21:58:26.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:26.931+0000 7fc00e800700 1 -- 172.21.15.62:0/780627030 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0081a1070 con 0x7fc008076e20 2024-04-20T21:58:26.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:26.931+0000 7fc006000700 1 -- 172.21.15.62:0/780627030 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbffc00f2e0 con 0x7fc008076e20 2024-04-20T21:58:26.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:26.931+0000 7fc00e800700 1 -- 172.21.15.62:0/780627030 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0081a15c0 con 0x7fc008076e20 2024-04-20T21:58:26.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:26.932+0000 7fc00e800700 1 -- 172.21.15.62:0/780627030 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc00810e180 con 0x7fc008076e20 2024-04-20T21:58:26.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:26.933+0000 7fc006000700 1 -- 172.21.15.62:0/780627030 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fbffc010380 con 0x7fc008076e20 2024-04-20T21:58:26.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:26.934+0000 7fc006000700 1 --2- 172.21.15.62:0/780627030 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fbff006ecc0 0x7fbff0071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:58:26.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:26.934+0000 7fc006000700 1 -- 172.21.15.62:0/780627030 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fbffc08e9c0 con 0x7fc008076e20 2024-04-20T21:58:26.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:26.934+0000 7fc00ca00700 1 --2- 172.21.15.62:0/780627030 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fbff006ecc0 0x7fbff0071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:58:26.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:26.935+0000 7fc00ca00700 1 --2- 172.21.15.62:0/780627030 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fbff006ecc0 0x7fbff0071170 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fc00819da00 tx=0x7fbff4005a90 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:58:26.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:26.939+0000 7fc006000700 1 -- 172.21.15.62:0/780627030 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fbffc057750 con 0x7fc008076e20 2024-04-20T21:58:27.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:58:26 smithi062 ceph-mon[76322]: pgmap v695: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:58:27.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:58:26 smithi062 ceph-mon[76322]: from='client.35264 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:58:27.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:58:26 smithi062 ceph-mon[76322]: from='client.35268 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:58:27.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:58:26 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/2693640535' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:58:27.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:27.214+0000 7fc00e800700 1 -- 172.21.15.62:0/780627030 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fc008062380 con 0x7fc008076e20 2024-04-20T21:58:27.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:27.215+0000 7fc006000700 1 -- 172.21.15.62:0/780627030 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 27 v27) v1 ==== 76+0+1888 (secure 0 0 0) 0x7fbffc05ae10 con 0x7fc008076e20 2024-04-20T21:58:27.215 INFO:teuthology.orchestra.run.smithi062.stdout:e27 2024-04-20T21:58:27.215 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T21:58:27.215 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T21:58:27.215 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T21:58:27.215 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:58:27.216 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T21:58:27.216 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T21:58:27.216 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 20 2024-04-20T21:58:27.216 INFO:teuthology.orchestra.run.smithi062.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-20T21:58:27.216 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T21:58:27.216 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T21:42:01.559061+0000 2024-04-20T21:58:27.216 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T21:58:27.216 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T21:58:27.216 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T21:58:27.216 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T21:58:27.216 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T21:58:27.216 INFO:teuthology.orchestra.run.smithi062.stdout:max_xattr_size 65536 2024-04-20T21:58:27.216 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T21:58:27.216 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T21:58:27.216 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 81 2024-04-20T21:58:27.217 INFO:teuthology.orchestra.run.smithi062.stdout:snap_rstat 1 2024-04-20T21:58:27.217 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T21:58:27.217 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 2 2024-04-20T21:58:27.217 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1 2024-04-20T21:58:27.217 INFO:teuthology.orchestra.run.smithi062.stdout:up {} 2024-04-20T21:58:27.217 INFO:teuthology.orchestra.run.smithi062.stdout:failed 0,1 2024-04-20T21:58:27.217 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T21:58:27.217 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T21:58:27.217 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T21:58:27.217 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T21:58:27.217 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data enabled 2024-04-20T21:58:27.217 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T21:58:27.217 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T21:58:27.217 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 1 2024-04-20T21:58:27.217 INFO:teuthology.orchestra.run.smithi062.stdout:qdb_cluster leader: 0 members: 2024-04-20T21:58:27.218 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:58:27.218 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:58:27.218 INFO:teuthology.orchestra.run.smithi062.stdout:Standby daemons: 2024-04-20T21:58:27.218 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:58:27.218 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ryqblb{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6826/4113916205,v1:172.21.15.62:6827/4113916205] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T21:58:27.218 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ozizwi{-1:34476} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6828/3801975165,v1:172.21.15.62:6829/3801975165] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T21:58:27.218 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.xbgspw{-1:44285} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6826/3996040526,v1:172.21.15.189:6827/3996040526] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:58:27.218 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.dvimer{-1:44727} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6824/3576699232,v1:172.21.15.189:6825/3576699232] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T21:58:27.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:27.218+0000 7fc00e800700 1 -- 172.21.15.62:0/780627030 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fbff006ecc0 msgr2=0x7fbff0071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:58:27.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:27.218+0000 7fc00e800700 1 --2- 172.21.15.62:0/780627030 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fbff006ecc0 0x7fbff0071170 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fc00819da00 tx=0x7fbff4005a90 comp rx=0 tx=0).stop 2024-04-20T21:58:27.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:27.218+0000 7fc00e800700 1 -- 172.21.15.62:0/780627030 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc008076e20 msgr2=0x7fc00819c3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:58:27.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:27.218+0000 7fc00e800700 1 --2- 172.21.15.62:0/780627030 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc008076e20 0x7fc00819c3e0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fbffc00b4b0 tx=0x7fbffc00b870 comp rx=0 tx=0).stop 2024-04-20T21:58:27.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:27.219+0000 7fc00e800700 1 -- 172.21.15.62:0/780627030 shutdown_connections 2024-04-20T21:58:27.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:27.219+0000 7fc00e800700 1 --2- 172.21.15.62:0/780627030 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fbff006ecc0 0x7fbff0071170 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:58:27.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:27.219+0000 7fc00e800700 1 --2- 172.21.15.62:0/780627030 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc008076e20 0x7fc00819c3e0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:58:27.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:27.219+0000 7fc00e800700 1 --2- 172.21.15.62:0/780627030 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc008077730 0x7fc00819c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:58:27.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:27.219+0000 7fc00e800700 1 -- 172.21.15.62:0/780627030 >> 172.21.15.62:0/780627030 conn(0x7fc0080fddb0 msgr2=0x7fc0080fea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:58:27.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:27.219+0000 7fc00e800700 1 -- 172.21.15.62:0/780627030 shutdown_connections 2024-04-20T21:58:27.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:27.219+0000 7fc00e800700 1 -- 172.21.15.62:0/780627030 wait complete. 2024-04-20T21:58:27.220 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 27 2024-04-20T21:58:27.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:58:26 smithi189 ceph-mon[57022]: pgmap v695: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:58:27.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:58:26 smithi189 ceph-mon[57022]: from='client.35264 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:58:27.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:58:26 smithi189 ceph-mon[57022]: from='client.35268 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:58:27.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:58:26 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/2693640535' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:58:27.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:27.378+0000 7f85d7000700 1 -- 172.21.15.62:0/1393223205 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f85d00730f0 msgr2=0x7f85d00734c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:58:27.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:27.378+0000 7f85d7000700 1 --2- 172.21.15.62:0/1393223205 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f85d00730f0 0x7f85d00734c0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f85c4009910 tx=0x7f85c4009c20 comp rx=0 tx=0).stop 2024-04-20T21:58:27.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:27.378+0000 7f85d7000700 1 -- 172.21.15.62:0/1393223205 shutdown_connections 2024-04-20T21:58:27.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:27.378+0000 7f85d7000700 1 --2- 172.21.15.62:0/1393223205 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f85d0073a00 0x7f85d0110b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:58:27.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:27.378+0000 7f85d7000700 1 --2- 172.21.15.62:0/1393223205 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f85d00730f0 0x7f85d00734c0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:58:27.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:27.378+0000 7f85d7000700 1 -- 172.21.15.62:0/1393223205 >> 172.21.15.62:0/1393223205 conn(0x7f85d00fbbb0 msgr2=0x7f85d00fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:58:27.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:27.378+0000 7f85d7000700 1 -- 172.21.15.62:0/1393223205 shutdown_connections 2024-04-20T21:58:27.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:27.378+0000 7f85d7000700 1 -- 172.21.15.62:0/1393223205 wait complete. 2024-04-20T21:58:27.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:27.379+0000 7f85d7000700 1 Processor -- start 2024-04-20T21:58:27.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:27.379+0000 7f85d7000700 1 -- start start 2024-04-20T21:58:27.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:27.380+0000 7f85d7000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f85d00730f0 0x7f85d019c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:58:27.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:27.380+0000 7f85d7000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f85d0073a00 0x7f85d019c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:58:27.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:27.380+0000 7f85d7000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85d019cdc0 con 0x7f85d00730f0 2024-04-20T21:58:27.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:27.380+0000 7f85d7000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85d01a0b50 con 0x7f85d0073a00 2024-04-20T21:58:27.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:27.380+0000 7f85d5c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f85d00730f0 0x7f85d019c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:58:27.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:27.380+0000 7f85d5c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f85d00730f0 0x7f85d019c1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43420/0 (socket says 172.21.15.62:43420) 2024-04-20T21:58:27.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:27.380+0000 7f85d5c00700 1 -- 172.21.15.62:0/1891401897 learned_addr learned my addr 172.21.15.62:0/1891401897 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:58:27.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:27.380+0000 7f85d5c00700 1 -- 172.21.15.62:0/1891401897 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f85d0073a00 msgr2=0x7f85d019c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:58:27.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:27.380+0000 7f85d5c00700 1 --2- 172.21.15.62:0/1891401897 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f85d0073a00 0x7f85d019c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:58:27.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:27.380+0000 7f85d5c00700 1 -- 172.21.15.62:0/1891401897 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85c40095f0 con 0x7f85d00730f0 2024-04-20T21:58:27.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:27.381+0000 7f85d5c00700 1 --2- 172.21.15.62:0/1891401897 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f85d00730f0 0x7f85d019c1a0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f85c400b5c0 tx=0x7f85c400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:58:27.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:27.381+0000 7f85ce000700 1 -- 172.21.15.62:0/1891401897 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f85c401d070 con 0x7f85d00730f0 2024-04-20T21:58:27.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:27.381+0000 7f85ce000700 1 -- 172.21.15.62:0/1891401897 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f85c4005e00 con 0x7f85d00730f0 2024-04-20T21:58:27.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:27.381+0000 7f85ce000700 1 -- 172.21.15.62:0/1891401897 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f85c4017400 con 0x7f85d00730f0 2024-04-20T21:58:27.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:27.381+0000 7f85d7000700 1 -- 172.21.15.62:0/1891401897 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85d01a0e30 con 0x7f85d00730f0 2024-04-20T21:58:27.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:27.381+0000 7f85d7000700 1 -- 172.21.15.62:0/1891401897 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85d01a1380 con 0x7f85d00730f0 2024-04-20T21:58:27.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:27.382+0000 7f85d7000700 1 -- 172.21.15.62:0/1891401897 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85d010e2c0 con 0x7f85d00730f0 2024-04-20T21:58:27.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:27.383+0000 7f85ce000700 1 -- 172.21.15.62:0/1891401897 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f85c400f460 con 0x7f85d00730f0 2024-04-20T21:58:27.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:27.384+0000 7f85ce000700 1 --2- 172.21.15.62:0/1891401897 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f85b806ecc0 0x7f85b8071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:58:27.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:27.384+0000 7f85ce000700 1 -- 172.21.15.62:0/1891401897 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f85c4090310 con 0x7f85d00730f0 2024-04-20T21:58:27.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:27.384+0000 7f85d5200700 1 --2- 172.21.15.62:0/1891401897 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f85b806ecc0 0x7f85b8071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:58:27.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:27.385+0000 7f85d5200700 1 --2- 172.21.15.62:0/1891401897 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f85b806ecc0 0x7f85b8071170 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f85bc005950 tx=0x7f85bc0058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:58:27.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:27.389+0000 7f85ce000700 1 -- 172.21.15.62:0/1891401897 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f85c40590a0 con 0x7f85d00730f0 2024-04-20T21:58:27.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:27.658+0000 7f85d7000700 1 -- 172.21.15.62:0/1891401897 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f85d01a1660 con 0x7f85b806ecc0 2024-04-20T21:58:27.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:27.660+0000 7f85ce000700 1 -- 172.21.15.62:0/1891401897 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f85d01a1660 con 0x7f85b806ecc0 2024-04-20T21:58:27.660 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:58:27.660 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T21:58:27.660 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T21:58:27.660 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-20T21:58:27.660 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T21:58:27.660 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr", 2024-04-20T21:58:27.661 INFO:teuthology.orchestra.run.smithi062.stdout: "crash", 2024-04-20T21:58:27.661 INFO:teuthology.orchestra.run.smithi062.stdout: "mon", 2024-04-20T21:58:27.661 INFO:teuthology.orchestra.run.smithi062.stdout: "osd" 2024-04-20T21:58:27.661 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T21:58:27.661 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "15/23 daemons upgraded", 2024-04-20T21:58:27.661 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading mds daemons", 2024-04-20T21:58:27.661 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T21:58:27.661 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:58:27.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:27.664+0000 7f85d7000700 1 -- 172.21.15.62:0/1891401897 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f85b806ecc0 msgr2=0x7f85b8071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:58:27.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:27.664+0000 7f85d7000700 1 --2- 172.21.15.62:0/1891401897 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f85b806ecc0 0x7f85b8071170 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f85bc005950 tx=0x7f85bc0058e0 comp rx=0 tx=0).stop 2024-04-20T21:58:27.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:27.664+0000 7f85d7000700 1 -- 172.21.15.62:0/1891401897 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f85d00730f0 msgr2=0x7f85d019c1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:58:27.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:27.664+0000 7f85d7000700 1 --2- 172.21.15.62:0/1891401897 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f85d00730f0 0x7f85d019c1a0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f85c400b5c0 tx=0x7f85c400be20 comp rx=0 tx=0).stop 2024-04-20T21:58:27.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:27.664+0000 7f85d7000700 1 -- 172.21.15.62:0/1891401897 shutdown_connections 2024-04-20T21:58:27.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:27.664+0000 7f85d7000700 1 --2- 172.21.15.62:0/1891401897 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f85b806ecc0 0x7f85b8071170 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:58:27.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:27.664+0000 7f85d7000700 1 --2- 172.21.15.62:0/1891401897 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f85d00730f0 0x7f85d019c1a0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:58:27.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:27.664+0000 7f85d7000700 1 --2- 172.21.15.62:0/1891401897 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f85d0073a00 0x7f85d019c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:58:27.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:27.664+0000 7f85d7000700 1 -- 172.21.15.62:0/1891401897 >> 172.21.15.62:0/1891401897 conn(0x7f85d00fbbb0 msgr2=0x7f85d0102670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:58:27.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:27.664+0000 7f85d7000700 1 -- 172.21.15.62:0/1891401897 shutdown_connections 2024-04-20T21:58:27.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:27.664+0000 7f85d7000700 1 -- 172.21.15.62:0/1891401897 wait complete. 2024-04-20T21:58:27.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:27.830+0000 7fe9f9a00700 1 -- 172.21.15.62:0/4046264267 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9f41004e0 msgr2=0x7fe9f41008b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:58:27.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:27.830+0000 7fe9f9a00700 1 --2- 172.21.15.62:0/4046264267 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9f41004e0 0x7fe9f41008b0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fe9e4009910 tx=0x7fe9e4009c20 comp rx=0 tx=0).stop 2024-04-20T21:58:27.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:27.830+0000 7fe9f9a00700 1 -- 172.21.15.62:0/4046264267 shutdown_connections 2024-04-20T21:58:27.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:27.830+0000 7fe9f9a00700 1 --2- 172.21.15.62:0/4046264267 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe9f4100e80 0x7fe9f4104e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:58:27.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:27.830+0000 7fe9f9a00700 1 --2- 172.21.15.62:0/4046264267 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9f41004e0 0x7fe9f41008b0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:58:27.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:27.830+0000 7fe9f9a00700 1 -- 172.21.15.62:0/4046264267 >> 172.21.15.62:0/4046264267 conn(0x7fe9f40fbda0 msgr2=0x7fe9f40fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:58:27.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:27.831+0000 7fe9f9a00700 1 -- 172.21.15.62:0/4046264267 shutdown_connections 2024-04-20T21:58:27.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:27.831+0000 7fe9f9a00700 1 -- 172.21.15.62:0/4046264267 wait complete. 2024-04-20T21:58:27.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:27.832+0000 7fe9f9a00700 1 Processor -- start 2024-04-20T21:58:27.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:27.832+0000 7fe9f9a00700 1 -- start start 2024-04-20T21:58:27.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:27.832+0000 7fe9f9a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9f41004e0 0x7fe9f4195ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:58:27.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:27.832+0000 7fe9f9a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe9f4100e80 0x7fe9f4196430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:58:27.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:27.832+0000 7fe9f9a00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9f4196b10 con 0x7fe9f41004e0 2024-04-20T21:58:27.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:27.832+0000 7fe9f9a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9f419a8a0 con 0x7fe9f4100e80 2024-04-20T21:58:27.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:27.832+0000 7fe9f2a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9f41004e0 0x7fe9f4195ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:58:27.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:27.832+0000 7fe9f2a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9f41004e0 0x7fe9f4195ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43426/0 (socket says 172.21.15.62:43426) 2024-04-20T21:58:27.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:27.832+0000 7fe9f2a00700 1 -- 172.21.15.62:0/2331138787 learned_addr learned my addr 172.21.15.62:0/2331138787 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:58:27.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:27.833+0000 7fe9ea000700 1 --2- 172.21.15.62:0/2331138787 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe9f4100e80 0x7fe9f4196430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:58:27.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:27.833+0000 7fe9f2a00700 1 -- 172.21.15.62:0/2331138787 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe9f4100e80 msgr2=0x7fe9f4196430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:58:27.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:27.833+0000 7fe9f2a00700 1 --2- 172.21.15.62:0/2331138787 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe9f4100e80 0x7fe9f4196430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:58:27.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:27.833+0000 7fe9f2a00700 1 -- 172.21.15.62:0/2331138787 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9e40095f0 con 0x7fe9f41004e0 2024-04-20T21:58:27.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:27.833+0000 7fe9ea000700 1 --2- 172.21.15.62:0/2331138787 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe9f4100e80 0x7fe9f4196430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T21:58:27.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:27.833+0000 7fe9f2a00700 1 --2- 172.21.15.62:0/2331138787 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9f41004e0 0x7fe9f4195ef0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fe9e40092e0 tx=0x7fe9e400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:58:27.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:27.833+0000 7fe9ebe00700 1 -- 172.21.15.62:0/2331138787 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe9e401d070 con 0x7fe9f41004e0 2024-04-20T21:58:27.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:27.834+0000 7fe9ebe00700 1 -- 172.21.15.62:0/2331138787 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe9e4005e00 con 0x7fe9f41004e0 2024-04-20T21:58:27.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:27.834+0000 7fe9ebe00700 1 -- 172.21.15.62:0/2331138787 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe9e4017400 con 0x7fe9f41004e0 2024-04-20T21:58:27.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:27.834+0000 7fe9f9a00700 1 -- 172.21.15.62:0/2331138787 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9f419ab20 con 0x7fe9f41004e0 2024-04-20T21:58:27.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:27.834+0000 7fe9f9a00700 1 -- 172.21.15.62:0/2331138787 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9f419b010 con 0x7fe9f41004e0 2024-04-20T21:58:27.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:27.835+0000 7fe9f9a00700 1 -- 172.21.15.62:0/2331138787 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9f41087f0 con 0x7fe9f41004e0 2024-04-20T21:58:27.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:27.836+0000 7fe9ebe00700 1 -- 172.21.15.62:0/2331138787 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe9e400f460 con 0x7fe9f41004e0 2024-04-20T21:58:27.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:27.837+0000 7fe9ebe00700 1 --2- 172.21.15.62:0/2331138787 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe9d806ecc0 0x7fe9d8071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:58:27.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:27.837+0000 7fe9ebe00700 1 -- 172.21.15.62:0/2331138787 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fe9e4090330 con 0x7fe9f41004e0 2024-04-20T21:58:27.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:27.838+0000 7fe9ea000700 1 --2- 172.21.15.62:0/2331138787 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe9d806ecc0 0x7fe9d8071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:58:27.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:27.838+0000 7fe9ea000700 1 --2- 172.21.15.62:0/2331138787 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe9d806ecc0 0x7fe9d8071170 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fe9dc005fd0 tx=0x7fe9dc005ee0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:58:27.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:27.842+0000 7fe9ebe00700 1 -- 172.21.15.62:0/2331138787 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fe9e40590c0 con 0x7fe9f41004e0 2024-04-20T21:58:28.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:58:27 smithi062 ceph-mon[76322]: from='client.35272 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:58:28.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:58:27 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/780627030' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T21:58:28.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:28.174+0000 7fe9f9a00700 1 -- 172.21.15.62:0/2331138787 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fe9f404f2a0 con 0x7fe9f41004e0 2024-04-20T21:58:28.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:28.175+0000 7fe9ebe00700 1 -- 172.21.15.62:0/2331138787 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7fe9e405c780 con 0x7fe9f41004e0 2024-04-20T21:58:28.175 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-20T21:58:28.175 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-20T21:58:28.175 INFO:teuthology.orchestra.run.smithi062.stdout: host smithi062 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T21:58:28.175 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-20T21:58:28.175 INFO:teuthology.orchestra.run.smithi062.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi062 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpq7y7oa_r:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T21:58:28.176 INFO:teuthology.orchestra.run.smithi062.stdout:Traceback (most recent call last): 2024-04-20T21:58:28.176 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-20T21:58:28.176 INFO:teuthology.orchestra.run.smithi062.stdout: return _run_code(code, main_globals, None, 2024-04-20T21:58:28.176 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-20T21:58:28.176 INFO:teuthology.orchestra.run.smithi062.stdout: exec(code, run_globals) 2024-04-20T21:58:28.176 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5618, in 2024-04-20T21:58:28.176 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5606, in main 2024-04-20T21:58:28.176 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 408, in _infer_config 2024-04-20T21:58:28.176 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 323, in _infer_fsid 2024-04-20T21:58:28.176 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 436, in _infer_image 2024-04-20T21:58:28.176 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 310, in _validate_fsid 2024-04-20T21:58:28.176 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 3359, in command_ceph_volume 2024-04-20T21:58:28.176 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-20T21:58:28.176 INFO:teuthology.orchestra.run.smithi062.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi062 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpq7y7oa_r:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T21:58:28.177 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-20T21:58:28.177 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is degraded 2024-04-20T21:58:28.177 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T21:58:28.177 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T21:58:28.177 INFO:teuthology.orchestra.run.smithi062.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-20T21:58:28.177 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-20T21:58:28.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:28.178+0000 7fe9f9a00700 1 -- 172.21.15.62:0/2331138787 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe9d806ecc0 msgr2=0x7fe9d8071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:58:28.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:28.178+0000 7fe9f9a00700 1 --2- 172.21.15.62:0/2331138787 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe9d806ecc0 0x7fe9d8071170 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fe9dc005fd0 tx=0x7fe9dc005ee0 comp rx=0 tx=0).stop 2024-04-20T21:58:28.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:28.178+0000 7fe9f9a00700 1 -- 172.21.15.62:0/2331138787 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9f41004e0 msgr2=0x7fe9f4195ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:58:28.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:28.178+0000 7fe9f9a00700 1 --2- 172.21.15.62:0/2331138787 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9f41004e0 0x7fe9f4195ef0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fe9e40092e0 tx=0x7fe9e400be20 comp rx=0 tx=0).stop 2024-04-20T21:58:28.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:28.178+0000 7fe9f9a00700 1 -- 172.21.15.62:0/2331138787 shutdown_connections 2024-04-20T21:58:28.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:28.178+0000 7fe9f9a00700 1 --2- 172.21.15.62:0/2331138787 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe9d806ecc0 0x7fe9d8071170 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:58:28.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:28.178+0000 7fe9f9a00700 1 --2- 172.21.15.62:0/2331138787 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9f41004e0 0x7fe9f4195ef0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:58:28.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:28.178+0000 7fe9f9a00700 1 --2- 172.21.15.62:0/2331138787 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe9f4100e80 0x7fe9f4196430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:58:28.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:28.178+0000 7fe9f9a00700 1 -- 172.21.15.62:0/2331138787 >> 172.21.15.62:0/2331138787 conn(0x7fe9f40fbda0 msgr2=0x7fe9f40fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:58:28.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:28.178+0000 7fe9f9a00700 1 -- 172.21.15.62:0/2331138787 shutdown_connections 2024-04-20T21:58:28.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:28.178+0000 7fe9f9a00700 1 -- 172.21.15.62:0/2331138787 wait complete. 2024-04-20T21:58:28.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:58:27 smithi189 ceph-mon[57022]: from='client.35272 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:58:28.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:58:27 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/780627030' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T21:58:29.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:58:28 smithi189 ceph-mon[57022]: pgmap v696: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:58:29.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:58:28 smithi189 ceph-mon[57022]: from='client.35284 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:58:29.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:58:28 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/2331138787' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T21:58:29.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:58:28 smithi062 ceph-mon[76322]: pgmap v696: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:58:29.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:58:28 smithi062 ceph-mon[76322]: from='client.35284 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:58:29.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:58:28 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/2331138787' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T21:58:31.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:58:30 smithi062 ceph-mon[76322]: pgmap v697: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:58:31.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:58:30 smithi189 ceph-mon[57022]: pgmap v697: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:58:33.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:58:32 smithi189 ceph-mon[57022]: pgmap v698: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:58:33.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:58:32 smithi062 ceph-mon[76322]: pgmap v698: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:58:35.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:58:34 smithi189 ceph-mon[57022]: pgmap v699: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:58:35.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:58:34 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:58:35.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:58:34 smithi062 ceph-mon[76322]: pgmap v699: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:58:35.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:58:34 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:58:37.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:58:36 smithi189 ceph-mon[57022]: pgmap v700: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:58:37.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:58:36 smithi062 ceph-mon[76322]: pgmap v700: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:58:39.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:58:38 smithi189 ceph-mon[57022]: pgmap v701: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:58:39.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:58:38 smithi062 ceph-mon[76322]: pgmap v701: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:58:41.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:58:40 smithi189 ceph-mon[57022]: pgmap v702: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:58:41.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:58:40 smithi062 ceph-mon[76322]: pgmap v702: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:58:43.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:58:42 smithi189 ceph-mon[57022]: pgmap v703: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:58:43.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:58:42 smithi062 ceph-mon[76322]: pgmap v703: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:58:45.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:58:44 smithi189 ceph-mon[57022]: pgmap v704: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:58:45.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:58:44 smithi062 ceph-mon[76322]: pgmap v704: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:58:47.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:58:46 smithi189 ceph-mon[57022]: pgmap v705: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:58:47.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:58:46 smithi062 ceph-mon[76322]: pgmap v705: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:58:49.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:58:48 smithi189 ceph-mon[57022]: pgmap v706: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:58:49.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:58:48 smithi062 ceph-mon[76322]: pgmap v706: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:58:50.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:58:49 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:58:50.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:58:49 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:58:51.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:58:50 smithi189 ceph-mon[57022]: pgmap v707: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:58:51.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:58:50 smithi062 ceph-mon[76322]: pgmap v707: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:58:53.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:58:52 smithi189 ceph-mon[57022]: pgmap v708: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:58:53.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:58:52 smithi062 ceph-mon[76322]: pgmap v708: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:58:55.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:58:54 smithi062 ceph-mon[76322]: pgmap v709: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:58:55.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:58:54 smithi189 ceph-mon[57022]: pgmap v709: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:58:57.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:58:57 smithi062 ceph-mon[76322]: pgmap v710: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:58:57.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:58:57 smithi189 ceph-mon[57022]: pgmap v710: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:58:58.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:58.356+0000 7f2453e00700 1 -- 172.21.15.62:0/1487095931 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f244c073090 msgr2=0x7f244c073460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:58:58.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:58.356+0000 7f2453e00700 1 --2- 172.21.15.62:0/1487095931 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f244c073090 0x7f244c073460 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f2440009940 tx=0x7f2440009c50 comp rx=0 tx=0).stop 2024-04-20T21:58:58.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:58.357+0000 7f2453e00700 1 -- 172.21.15.62:0/1487095931 shutdown_connections 2024-04-20T21:58:58.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:58.357+0000 7f2453e00700 1 --2- 172.21.15.62:0/1487095931 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f244c0739a0 0x7f244c10c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:58:58.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:58.357+0000 7f2453e00700 1 --2- 172.21.15.62:0/1487095931 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f244c073090 0x7f244c073460 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:58:58.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:58.357+0000 7f2453e00700 1 -- 172.21.15.62:0/1487095931 >> 172.21.15.62:0/1487095931 conn(0x7f244c0fbda0 msgr2=0x7f244c0fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:58:58.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:58.357+0000 7f2453e00700 1 -- 172.21.15.62:0/1487095931 shutdown_connections 2024-04-20T21:58:58.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:58.357+0000 7f2453e00700 1 -- 172.21.15.62:0/1487095931 wait complete. 2024-04-20T21:58:58.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:58.358+0000 7f2453e00700 1 Processor -- start 2024-04-20T21:58:58.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:58.358+0000 7f2453e00700 1 -- start start 2024-04-20T21:58:58.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:58.359+0000 7f2453e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f244c073090 0x7f244c198180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:58:58.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:58.359+0000 7f2453e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f244c0739a0 0x7f244c1986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:58:58.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:58.359+0000 7f2453e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f244c198da0 con 0x7f244c0739a0 2024-04-20T21:58:58.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:58.359+0000 7f2453e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f244c19cb30 con 0x7f244c073090 2024-04-20T21:58:58.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:58.359+0000 7f2450a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f244c0739a0 0x7f244c1986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:58:58.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:58.359+0000 7f2450a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f244c0739a0 0x7f244c1986c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:39788/0 (socket says 172.21.15.62:39788) 2024-04-20T21:58:58.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:58.359+0000 7f2450a00700 1 -- 172.21.15.62:0/3793571441 learned_addr learned my addr 172.21.15.62:0/3793571441 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:58:58.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:58.359+0000 7f2451400700 1 --2- 172.21.15.62:0/3793571441 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f244c073090 0x7f244c198180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:58:58.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:58.360+0000 7f2450a00700 1 -- 172.21.15.62:0/3793571441 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f244c073090 msgr2=0x7f244c198180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:58:58.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:58.360+0000 7f2450a00700 1 --2- 172.21.15.62:0/3793571441 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f244c073090 0x7f244c198180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:58:58.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:58.360+0000 7f2450a00700 1 -- 172.21.15.62:0/3793571441 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24400095f0 con 0x7f244c0739a0 2024-04-20T21:58:58.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:58.360+0000 7f2450a00700 1 --2- 172.21.15.62:0/3793571441 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f244c0739a0 0x7f244c1986c0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f243800d900 tx=0x7f243800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:58:58.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:58.360+0000 7f244a000700 1 -- 172.21.15.62:0/3793571441 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2438004a00 con 0x7f244c0739a0 2024-04-20T21:58:58.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:58.361+0000 7f2453e00700 1 -- 172.21.15.62:0/3793571441 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f244c19ce10 con 0x7f244c0739a0 2024-04-20T21:58:58.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:58.361+0000 7f244a000700 1 -- 172.21.15.62:0/3793571441 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f24380054e0 con 0x7f244c0739a0 2024-04-20T21:58:58.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:58.361+0000 7f244a000700 1 -- 172.21.15.62:0/3793571441 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2438009d60 con 0x7f244c0739a0 2024-04-20T21:58:58.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:58.361+0000 7f2453e00700 1 -- 172.21.15.62:0/3793571441 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f244c19d360 con 0x7f244c0739a0 2024-04-20T21:58:58.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:58.362+0000 7f2453e00700 1 -- 172.21.15.62:0/3793571441 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f244c109ce0 con 0x7f244c0739a0 2024-04-20T21:58:58.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:58.363+0000 7f244a000700 1 -- 172.21.15.62:0/3793571441 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f2438005020 con 0x7f244c0739a0 2024-04-20T21:58:58.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:58.364+0000 7f244a000700 1 --2- 172.21.15.62:0/3793571441 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f243406ecc0 0x7f2434071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:58:58.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:58.364+0000 7f244a000700 1 -- 172.21.15.62:0/3793571441 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f243808efe0 con 0x7f244c0739a0 2024-04-20T21:58:58.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:58.365+0000 7f2451400700 1 --2- 172.21.15.62:0/3793571441 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f243406ecc0 0x7f2434071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:58:58.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:58.365+0000 7f2451400700 1 --2- 172.21.15.62:0/3793571441 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f243406ecc0 0x7f2434071170 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f2440005b40 tx=0x7f2440005ab0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:58:58.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:58.373+0000 7f244a000700 1 -- 172.21.15.62:0/3793571441 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f24380c3ff0 con 0x7f244c0739a0 2024-04-20T21:58:58.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:58.636+0000 7f2453e00700 1 -- 172.21.15.62:0/3793571441 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f244c19d5d0 con 0x7f243406ecc0 2024-04-20T21:58:58.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:58.638+0000 7f244a000700 1 -- 172.21.15.62:0/3793571441 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f244c19d5d0 con 0x7f243406ecc0 2024-04-20T21:58:58.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:58.641+0000 7f2453e00700 1 -- 172.21.15.62:0/3793571441 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f243406ecc0 msgr2=0x7f2434071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:58:58.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:58.641+0000 7f2453e00700 1 --2- 172.21.15.62:0/3793571441 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f243406ecc0 0x7f2434071170 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f2440005b40 tx=0x7f2440005ab0 comp rx=0 tx=0).stop 2024-04-20T21:58:58.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:58.641+0000 7f2453e00700 1 -- 172.21.15.62:0/3793571441 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f244c0739a0 msgr2=0x7f244c1986c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:58:58.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:58.641+0000 7f2453e00700 1 --2- 172.21.15.62:0/3793571441 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f244c0739a0 0x7f244c1986c0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f243800d900 tx=0x7f243800dcc0 comp rx=0 tx=0).stop 2024-04-20T21:58:58.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:58.642+0000 7f2453e00700 1 -- 172.21.15.62:0/3793571441 shutdown_connections 2024-04-20T21:58:58.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:58.642+0000 7f2453e00700 1 --2- 172.21.15.62:0/3793571441 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f243406ecc0 0x7f2434071170 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:58:58.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:58.642+0000 7f2453e00700 1 --2- 172.21.15.62:0/3793571441 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f244c073090 0x7f244c198180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:58:58.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:58.642+0000 7f2453e00700 1 --2- 172.21.15.62:0/3793571441 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f244c0739a0 0x7f244c1986c0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:58:58.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:58.642+0000 7f2453e00700 1 -- 172.21.15.62:0/3793571441 >> 172.21.15.62:0/3793571441 conn(0x7f244c0fbda0 msgr2=0x7f244c106e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:58:58.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:58.642+0000 7f2453e00700 1 -- 172.21.15.62:0/3793571441 shutdown_connections 2024-04-20T21:58:58.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:58.642+0000 7f2453e00700 1 -- 172.21.15.62:0/3793571441 wait complete. 2024-04-20T21:58:58.658 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T21:58:58.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:58.807+0000 7fe4a4200700 1 -- 172.21.15.62:0/1842425055 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe49c106b80 msgr2=0x7fe49c106ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:58:58.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:58.807+0000 7fe4a4200700 1 --2- 172.21.15.62:0/1842425055 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe49c106b80 0x7fe49c106ff0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fe488009910 tx=0x7fe488009c20 comp rx=0 tx=0).stop 2024-04-20T21:58:58.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:58.807+0000 7fe4a4200700 1 -- 172.21.15.62:0/1842425055 shutdown_connections 2024-04-20T21:58:58.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:58.807+0000 7fe4a4200700 1 --2- 172.21.15.62:0/1842425055 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe49c106b80 0x7fe49c106ff0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:58:58.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:58.807+0000 7fe4a4200700 1 --2- 172.21.15.62:0/1842425055 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe49c10c8f0 0x7fe49c10ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:58:58.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:58.807+0000 7fe4a4200700 1 -- 172.21.15.62:0/1842425055 >> 172.21.15.62:0/1842425055 conn(0x7fe49c076060 msgr2=0x7fe49c078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:58:58.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:58.807+0000 7fe4a4200700 1 -- 172.21.15.62:0/1842425055 shutdown_connections 2024-04-20T21:58:58.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:58.807+0000 7fe4a4200700 1 -- 172.21.15.62:0/1842425055 wait complete. 2024-04-20T21:58:58.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:58.808+0000 7fe4a4200700 1 Processor -- start 2024-04-20T21:58:58.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:58.809+0000 7fe4a4200700 1 -- start start 2024-04-20T21:58:58.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:58.809+0000 7fe4a4200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe49c106b80 0x7fe49c19c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:58:58.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:58.809+0000 7fe4a4200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe49c10c8f0 0x7fe49c19c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:58:58.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:58.809+0000 7fe4a4200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe49c19d060 con 0x7fe49c10c8f0 2024-04-20T21:58:58.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:58.809+0000 7fe4a4200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe49c1a0df0 con 0x7fe49c106b80 2024-04-20T21:58:58.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:58.809+0000 7fe4a0e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe49c10c8f0 0x7fe49c19c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:58:58.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:58.809+0000 7fe4a0e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe49c10c8f0 0x7fe49c19c980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:39802/0 (socket says 172.21.15.62:39802) 2024-04-20T21:58:58.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:58.809+0000 7fe4a0e00700 1 -- 172.21.15.62:0/1032097028 learned_addr learned my addr 172.21.15.62:0/1032097028 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:58:58.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:58.810+0000 7fe4a1800700 1 --2- 172.21.15.62:0/1032097028 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe49c106b80 0x7fe49c19c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:58:58.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:58.810+0000 7fe4a0e00700 1 -- 172.21.15.62:0/1032097028 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe49c106b80 msgr2=0x7fe49c19c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:58:58.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:58.810+0000 7fe4a0e00700 1 --2- 172.21.15.62:0/1032097028 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe49c106b80 0x7fe49c19c440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:58:58.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:58.810+0000 7fe4a0e00700 1 -- 172.21.15.62:0/1032097028 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4880095f0 con 0x7fe49c10c8f0 2024-04-20T21:58:58.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:58.810+0000 7fe4a0e00700 1 --2- 172.21.15.62:0/1032097028 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe49c10c8f0 0x7fe49c19c980 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fe488006010 tx=0x7fe48800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:58:58.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:58.810+0000 7fe49a000700 1 -- 172.21.15.62:0/1032097028 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe48801d070 con 0x7fe49c10c8f0 2024-04-20T21:58:58.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:58.810+0000 7fe4a4200700 1 -- 172.21.15.62:0/1032097028 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe49c1a1070 con 0x7fe49c10c8f0 2024-04-20T21:58:58.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:58.810+0000 7fe49a000700 1 -- 172.21.15.62:0/1032097028 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe48800f460 con 0x7fe49c10c8f0 2024-04-20T21:58:58.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:58.811+0000 7fe49a000700 1 -- 172.21.15.62:0/1032097028 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe488017630 con 0x7fe49c10c8f0 2024-04-20T21:58:58.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:58.811+0000 7fe4a4200700 1 -- 172.21.15.62:0/1032097028 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe49c1a1560 con 0x7fe49c10c8f0 2024-04-20T21:58:58.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:58.812+0000 7fe4a4200700 1 -- 172.21.15.62:0/1032097028 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe49c10ecc0 con 0x7fe49c10c8f0 2024-04-20T21:58:58.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:58.813+0000 7fe49a000700 1 -- 172.21.15.62:0/1032097028 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe48800f5d0 con 0x7fe49c10c8f0 2024-04-20T21:58:58.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:58.814+0000 7fe49a000700 1 --2- 172.21.15.62:0/1032097028 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe48406ecc0 0x7fe484071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:58:58.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:58.814+0000 7fe49a000700 1 -- 172.21.15.62:0/1032097028 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fe4880903a0 con 0x7fe49c10c8f0 2024-04-20T21:58:58.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:58.814+0000 7fe4a1800700 1 --2- 172.21.15.62:0/1032097028 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe48406ecc0 0x7fe484071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:58:58.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:58.815+0000 7fe4a1800700 1 --2- 172.21.15.62:0/1032097028 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe48406ecc0 0x7fe484071170 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fe490005950 tx=0x7fe4900058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:58:58.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:58.819+0000 7fe49a000700 1 -- 172.21.15.62:0/1032097028 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fe4880590b0 con 0x7fe49c10c8f0 2024-04-20T21:58:59.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:59.075+0000 7fe4a4200700 1 -- 172.21.15.62:0/1032097028 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe49c02cca0 con 0x7fe48406ecc0 2024-04-20T21:58:59.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:59.078+0000 7fe49a000700 1 -- 172.21.15.62:0/1032097028 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fe49c02cca0 con 0x7fe48406ecc0 2024-04-20T21:58:59.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:59.080+0000 7fe4a4200700 1 -- 172.21.15.62:0/1032097028 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe48406ecc0 msgr2=0x7fe484071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:58:59.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:59.080+0000 7fe4a4200700 1 --2- 172.21.15.62:0/1032097028 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe48406ecc0 0x7fe484071170 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fe490005950 tx=0x7fe4900058e0 comp rx=0 tx=0).stop 2024-04-20T21:58:59.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:59.080+0000 7fe4a4200700 1 -- 172.21.15.62:0/1032097028 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe49c10c8f0 msgr2=0x7fe49c19c980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:58:59.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:59.080+0000 7fe4a4200700 1 --2- 172.21.15.62:0/1032097028 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe49c10c8f0 0x7fe49c19c980 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fe488006010 tx=0x7fe48800be20 comp rx=0 tx=0).stop 2024-04-20T21:58:59.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:59.081+0000 7fe4a4200700 1 -- 172.21.15.62:0/1032097028 shutdown_connections 2024-04-20T21:58:59.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:59.081+0000 7fe4a4200700 1 --2- 172.21.15.62:0/1032097028 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe48406ecc0 0x7fe484071170 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:58:59.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:59.081+0000 7fe4a4200700 1 --2- 172.21.15.62:0/1032097028 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe49c106b80 0x7fe49c19c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:58:59.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:59.081+0000 7fe4a4200700 1 --2- 172.21.15.62:0/1032097028 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe49c10c8f0 0x7fe49c19c980 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:58:59.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:59.081+0000 7fe4a4200700 1 -- 172.21.15.62:0/1032097028 >> 172.21.15.62:0/1032097028 conn(0x7fe49c076060 msgr2=0x7fe49c077ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:58:59.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:59.081+0000 7fe4a4200700 1 -- 172.21.15.62:0/1032097028 shutdown_connections 2024-04-20T21:58:59.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:59.081+0000 7fe4a4200700 1 -- 172.21.15.62:0/1032097028 wait complete. 2024-04-20T21:58:59.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:59.248+0000 7f415b000700 1 -- 172.21.15.62:0/3647844393 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4154100530 msgr2=0x7f4154100900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:58:59.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:59.248+0000 7f415b000700 1 --2- 172.21.15.62:0/3647844393 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4154100530 0x7f4154100900 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f4144009910 tx=0x7f4144009c20 comp rx=0 tx=0).stop 2024-04-20T21:58:59.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:59.248+0000 7f415b000700 1 -- 172.21.15.62:0/3647844393 shutdown_connections 2024-04-20T21:58:59.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:59.248+0000 7f415b000700 1 --2- 172.21.15.62:0/3647844393 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4154100ed0 0x7f4154104ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:58:59.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:59.248+0000 7f415b000700 1 --2- 172.21.15.62:0/3647844393 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4154100530 0x7f4154100900 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:58:59.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:59.248+0000 7f415b000700 1 -- 172.21.15.62:0/3647844393 >> 172.21.15.62:0/3647844393 conn(0x7f41540fbe10 msgr2=0x7f41540fe220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:58:59.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:59.248+0000 7f415b000700 1 -- 172.21.15.62:0/3647844393 shutdown_connections 2024-04-20T21:58:59.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:59.248+0000 7f415b000700 1 -- 172.21.15.62:0/3647844393 wait complete. 2024-04-20T21:58:59.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:59.250+0000 7f415b000700 1 Processor -- start 2024-04-20T21:58:59.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:59.250+0000 7f415b000700 1 -- start start 2024-04-20T21:58:59.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:59.250+0000 7f415b000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4154100530 0x7f4154198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:58:59.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:59.250+0000 7f415b000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4154100ed0 0x7f4154198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:58:59.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:59.250+0000 7f415b000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4154198c50 con 0x7f4154100ed0 2024-04-20T21:58:59.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:59.250+0000 7f415b000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f415419c9e0 con 0x7f4154100530 2024-04-20T21:58:59.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:59.251+0000 7f414be00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4154100ed0 0x7f4154198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:58:59.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:59.251+0000 7f414be00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4154100ed0 0x7f4154198570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:39814/0 (socket says 172.21.15.62:39814) 2024-04-20T21:58:59.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:59.251+0000 7f414be00700 1 -- 172.21.15.62:0/717382354 learned_addr learned my addr 172.21.15.62:0/717382354 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:58:59.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:59.251+0000 7f4153e00700 1 --2- 172.21.15.62:0/717382354 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4154100530 0x7f4154198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:58:59.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:59.251+0000 7f414be00700 1 -- 172.21.15.62:0/717382354 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4154100530 msgr2=0x7f4154198030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:58:59.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:59.251+0000 7f414be00700 1 --2- 172.21.15.62:0/717382354 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4154100530 0x7f4154198030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:58:59.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:59.251+0000 7f414be00700 1 -- 172.21.15.62:0/717382354 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41440095f0 con 0x7f4154100ed0 2024-04-20T21:58:59.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:59.251+0000 7f4153e00700 1 --2- 172.21.15.62:0/717382354 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4154100530 0x7f4154198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-20T21:58:59.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:59.252+0000 7f414be00700 1 --2- 172.21.15.62:0/717382354 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4154100ed0 0x7f4154198570 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f413c00d900 tx=0x7f413c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:58:59.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:59.252+0000 7f4151600700 1 -- 172.21.15.62:0/717382354 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f413c009900 con 0x7f4154100ed0 2024-04-20T21:58:59.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:59.252+0000 7f4151600700 1 -- 172.21.15.62:0/717382354 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f413c010460 con 0x7f4154100ed0 2024-04-20T21:58:59.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:59.252+0000 7f415b000700 1 -- 172.21.15.62:0/717382354 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f415419ccc0 con 0x7f4154100ed0 2024-04-20T21:58:59.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:59.252+0000 7f4151600700 1 -- 172.21.15.62:0/717382354 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f413c00f5f0 con 0x7f4154100ed0 2024-04-20T21:58:59.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:59.252+0000 7f415b000700 1 -- 172.21.15.62:0/717382354 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f415419d210 con 0x7f4154100ed0 2024-04-20T21:58:59.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:59.253+0000 7f415b000700 1 -- 172.21.15.62:0/717382354 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4154104620 con 0x7f4154100ed0 2024-04-20T21:58:59.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:59.254+0000 7f4151600700 1 -- 172.21.15.62:0/717382354 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f413c0105d0 con 0x7f4154100ed0 2024-04-20T21:58:59.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:59.256+0000 7f4151600700 1 --2- 172.21.15.62:0/717382354 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f413806ecc0 0x7f4138071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:58:59.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:59.256+0000 7f4151600700 1 -- 172.21.15.62:0/717382354 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f413c08ed50 con 0x7f4154100ed0 2024-04-20T21:58:59.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:59.256+0000 7f4153e00700 1 --2- 172.21.15.62:0/717382354 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f413806ecc0 0x7f4138071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:58:59.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:59.257+0000 7f4153e00700 1 --2- 172.21.15.62:0/717382354 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f413806ecc0 0x7f4138071170 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f414400b5c0 tx=0x7f414401a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:58:59.256 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:58:59 smithi062 ceph-mon[76322]: pgmap v711: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:58:59.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:59.264+0000 7f4151600700 1 -- 172.21.15.62:0/717382354 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f413c057a60 con 0x7f4154100ed0 2024-04-20T21:58:59.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:58:59 smithi189 ceph-mon[57022]: pgmap v711: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:58:59.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:59.510+0000 7f415b000700 1 -- 172.21.15.62:0/717382354 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f415419d430 con 0x7f413806ecc0 2024-04-20T21:58:59.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:59.544+0000 7f4151600700 1 -- 172.21.15.62:0/717382354 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f415419d430 con 0x7f413806ecc0 2024-04-20T21:58:59.544 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T21:58:59.544 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (24m) 54s ago 38m 15.3M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T21:58:59.544 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (39m) 54s ago 39m 10.4M - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T21:58:59.544 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (37m) 57s ago 37m 10.1M - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T21:58:59.544 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (20m) 54s ago 38m 6660k - 19.0.0-3133-ga8c84f59 76bc6b91b726 2825d5ba11c6 2024-04-20T21:58:59.544 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (20m) 57s ago 37m 6665k - 19.0.0-3133-ga8c84f59 76bc6b91b726 20f746fa1bba 2024-04-20T21:58:59.544 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (24m) 54s ago 38m 73.5M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T21:58:59.544 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (16m) 54s ago 31m 18.5M - 19.0.0-3133-ga8c84f59 76bc6b91b726 de80bc90a683 2024-04-20T21:58:59.544 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (16m) 54s ago 31m 18.7M - 19.0.0-3133-ga8c84f59 76bc6b91b726 cd793b7ee98e 2024-04-20T21:58:59.544 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (63s) 57s ago 31m 11.0M - 19.0.0-3133-ga8c84f59 76bc6b91b726 0e7b0bc4d1ab 2024-04-20T21:58:59.544 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (31m) 57s ago 31m 18.6M - 18.2.0 dc2bc1663786 f2eec2ee8951 2024-04-20T21:58:59.544 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (27m) 54s ago 40m 513M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T21:58:59.545 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (26m) 57s ago 37m 459M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T21:58:59.545 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (22m) 54s ago 40m 71.7M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 019eb1479485 2024-04-20T21:58:59.545 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (21m) 57s ago 37m 51.0M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 55aac560aab6 2024-04-20T21:58:59.545 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (26m) 54s ago 38m 8472k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T21:58:59.545 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (25m) 57s ago 37m 8048k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T21:58:59.545 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (20m) 54s ago 36m 91.2M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 ee2bb901f26e 2024-04-20T21:58:59.545 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (19m) 54s ago 35m 94.6M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 9075152d4162 2024-04-20T21:58:59.545 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (19m) 54s ago 35m 78.1M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 2d4f3e38dc0e 2024-04-20T21:58:59.545 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (18m) 57s ago 34m 92.6M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 8f97b17ae0eb 2024-04-20T21:58:59.545 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (17m) 57s ago 33m 95.6M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 653d8079967d 2024-04-20T21:58:59.545 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (17m) 57s ago 33m 89.2M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 596dded14c12 2024-04-20T21:58:59.545 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (25m) 54s ago 38m 82.1M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T21:58:59.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:59.546+0000 7f415b000700 1 -- 172.21.15.62:0/717382354 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f413806ecc0 msgr2=0x7f4138071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:58:59.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:59.546+0000 7f415b000700 1 --2- 172.21.15.62:0/717382354 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f413806ecc0 0x7f4138071170 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f414400b5c0 tx=0x7f414401a040 comp rx=0 tx=0).stop 2024-04-20T21:58:59.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:59.546+0000 7f415b000700 1 -- 172.21.15.62:0/717382354 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4154100ed0 msgr2=0x7f4154198570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:58:59.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:59.546+0000 7f415b000700 1 --2- 172.21.15.62:0/717382354 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4154100ed0 0x7f4154198570 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f413c00d900 tx=0x7f413c00dcc0 comp rx=0 tx=0).stop 2024-04-20T21:58:59.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:59.547+0000 7f415b000700 1 -- 172.21.15.62:0/717382354 shutdown_connections 2024-04-20T21:58:59.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:59.547+0000 7f415b000700 1 --2- 172.21.15.62:0/717382354 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f413806ecc0 0x7f4138071170 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:58:59.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:59.547+0000 7f415b000700 1 --2- 172.21.15.62:0/717382354 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4154100530 0x7f4154198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:58:59.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:59.547+0000 7f415b000700 1 --2- 172.21.15.62:0/717382354 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4154100ed0 0x7f4154198570 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:58:59.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:59.547+0000 7f415b000700 1 -- 172.21.15.62:0/717382354 >> 172.21.15.62:0/717382354 conn(0x7f41540fbe10 msgr2=0x7f41540fd1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:58:59.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:59.547+0000 7f415b000700 1 -- 172.21.15.62:0/717382354 shutdown_connections 2024-04-20T21:58:59.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:59.547+0000 7f415b000700 1 -- 172.21.15.62:0/717382354 wait complete. 2024-04-20T21:58:59.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:59.707+0000 7f7e6b200700 1 -- 172.21.15.62:0/31795128 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7e6c10c660 msgr2=0x7f7e6c10ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:58:59.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:59.707+0000 7f7e6b200700 1 --2- 172.21.15.62:0/31795128 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7e6c10c660 0x7f7e6c10ca30 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f7e5c009910 tx=0x7f7e5c009c20 comp rx=0 tx=0).stop 2024-04-20T21:58:59.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:59.707+0000 7f7e6b200700 1 -- 172.21.15.62:0/31795128 shutdown_connections 2024-04-20T21:58:59.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:59.707+0000 7f7e6b200700 1 --2- 172.21.15.62:0/31795128 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e6c106b80 0x7f7e6c106ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:58:59.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:59.707+0000 7f7e6b200700 1 --2- 172.21.15.62:0/31795128 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7e6c10c660 0x7f7e6c10ca30 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:58:59.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:59.707+0000 7f7e6b200700 1 -- 172.21.15.62:0/31795128 >> 172.21.15.62:0/31795128 conn(0x7f7e6c075e90 msgr2=0x7f7e6c0782a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:58:59.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:59.708+0000 7f7e6b200700 1 -- 172.21.15.62:0/31795128 shutdown_connections 2024-04-20T21:58:59.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:59.708+0000 7f7e6b200700 1 -- 172.21.15.62:0/31795128 wait complete. 2024-04-20T21:58:59.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:59.709+0000 7f7e6b200700 1 Processor -- start 2024-04-20T21:58:59.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:59.710+0000 7f7e6b200700 1 -- start start 2024-04-20T21:58:59.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:59.710+0000 7f7e6b200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7e6c106b80 0x7f7e6c19c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:58:59.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:59.710+0000 7f7e6b200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e6c10c660 0x7f7e6c19c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:58:59.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:59.710+0000 7f7e6b200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e6c19cea0 con 0x7f7e6c106b80 2024-04-20T21:58:59.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:59.710+0000 7f7e6b200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e6c1a0c30 con 0x7f7e6c10c660 2024-04-20T21:58:59.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:59.711+0000 7f7e69e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7e6c106b80 0x7f7e6c19c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:58:59.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:59.711+0000 7f7e69e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7e6c106b80 0x7f7e6c19c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:39840/0 (socket says 172.21.15.62:39840) 2024-04-20T21:58:59.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:59.711+0000 7f7e69e00700 1 -- 172.21.15.62:0/2815614965 learned_addr learned my addr 172.21.15.62:0/2815614965 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:58:59.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:59.711+0000 7f7e69400700 1 --2- 172.21.15.62:0/2815614965 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e6c10c660 0x7f7e6c19c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:58:59.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:59.711+0000 7f7e69e00700 1 -- 172.21.15.62:0/2815614965 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e6c10c660 msgr2=0x7f7e6c19c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:58:59.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:59.711+0000 7f7e69e00700 1 --2- 172.21.15.62:0/2815614965 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e6c10c660 0x7f7e6c19c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:58:59.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:59.711+0000 7f7e69e00700 1 -- 172.21.15.62:0/2815614965 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e5c0095f0 con 0x7f7e6c106b80 2024-04-20T21:58:59.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:59.711+0000 7f7e69e00700 1 --2- 172.21.15.62:0/2815614965 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7e6c106b80 0x7f7e6c19c280 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f7e5c005f50 tx=0x7f7e5c00f740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:58:59.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:59.712+0000 7f7e62a00700 1 -- 172.21.15.62:0/2815614965 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7e5c01d070 con 0x7f7e6c106b80 2024-04-20T21:58:59.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:59.712+0000 7f7e62a00700 1 -- 172.21.15.62:0/2815614965 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7e5c00b610 con 0x7f7e6c106b80 2024-04-20T21:58:59.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:59.712+0000 7f7e6b200700 1 -- 172.21.15.62:0/2815614965 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e6c1a0eb0 con 0x7f7e6c106b80 2024-04-20T21:58:59.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:59.712+0000 7f7e62a00700 1 -- 172.21.15.62:0/2815614965 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7e5c022470 con 0x7f7e6c106b80 2024-04-20T21:58:59.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:59.712+0000 7f7e6b200700 1 -- 172.21.15.62:0/2815614965 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e6c1a13a0 con 0x7f7e6c106b80 2024-04-20T21:58:59.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:59.713+0000 7f7e6b200700 1 -- 172.21.15.62:0/2815614965 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e6c10eb40 con 0x7f7e6c106b80 2024-04-20T21:58:59.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:59.714+0000 7f7e62a00700 1 -- 172.21.15.62:0/2815614965 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f7e5c00fca0 con 0x7f7e6c106b80 2024-04-20T21:58:59.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:59.715+0000 7f7e62a00700 1 --2- 172.21.15.62:0/2815614965 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7e5006ecc0 0x7f7e50071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:58:59.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:59.715+0000 7f7e62a00700 1 -- 172.21.15.62:0/2815614965 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f7e5c090370 con 0x7f7e6c106b80 2024-04-20T21:58:59.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:59.716+0000 7f7e69400700 1 --2- 172.21.15.62:0/2815614965 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7e5006ecc0 0x7f7e50071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:58:59.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:59.716+0000 7f7e69400700 1 --2- 172.21.15.62:0/2815614965 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7e5006ecc0 0x7f7e50071170 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c19d8a0 tx=0x7f7e58009260 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:58:59.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:58:59.720+0000 7f7e62a00700 1 -- 172.21.15.62:0/2815614965 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f7e5c059080 con 0x7f7e6c106b80 2024-04-20T21:59:00.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.052+0000 7f7e6b200700 1 -- 172.21.15.62:0/2815614965 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f7e6c062380 con 0x7f7e6c106b80 2024-04-20T21:59:00.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.053+0000 7f7e62a00700 1 -- 172.21.15.62:0/2815614965 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7f7e5c05c740 con 0x7f7e6c106b80 2024-04-20T21:59:00.053 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:59:00.053 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T21:59:00.053 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:59:00.053 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:59:00.053 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T21:59:00.053 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:59:00.053 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:59:00.053 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T21:59:00.054 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 6 2024-04-20T21:59:00.054 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:59:00.054 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T21:59:00.054 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 1, 2024-04-20T21:59:00.054 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 3 2024-04-20T21:59:00.054 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:59:00.054 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T21:59:00.054 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 1, 2024-04-20T21:59:00.054 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 13 2024-04-20T21:59:00.054 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T21:59:00.054 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:59:00.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.056+0000 7f7e6b200700 1 -- 172.21.15.62:0/2815614965 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7e5006ecc0 msgr2=0x7f7e50071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:59:00.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.056+0000 7f7e6b200700 1 --2- 172.21.15.62:0/2815614965 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7e5006ecc0 0x7f7e50071170 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c19d8a0 tx=0x7f7e58009260 comp rx=0 tx=0).stop 2024-04-20T21:59:00.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.056+0000 7f7e6b200700 1 -- 172.21.15.62:0/2815614965 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7e6c106b80 msgr2=0x7f7e6c19c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:59:00.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.056+0000 7f7e6b200700 1 --2- 172.21.15.62:0/2815614965 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7e6c106b80 0x7f7e6c19c280 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f7e5c005f50 tx=0x7f7e5c00f740 comp rx=0 tx=0).stop 2024-04-20T21:59:00.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.056+0000 7f7e6b200700 1 -- 172.21.15.62:0/2815614965 shutdown_connections 2024-04-20T21:59:00.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.056+0000 7f7e6b200700 1 --2- 172.21.15.62:0/2815614965 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7e5006ecc0 0x7f7e50071170 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:59:00.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.056+0000 7f7e6b200700 1 --2- 172.21.15.62:0/2815614965 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7e6c106b80 0x7f7e6c19c280 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:59:00.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.056+0000 7f7e6b200700 1 --2- 172.21.15.62:0/2815614965 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e6c10c660 0x7f7e6c19c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:59:00.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.056+0000 7f7e6b200700 1 -- 172.21.15.62:0/2815614965 >> 172.21.15.62:0/2815614965 conn(0x7f7e6c075e90 msgr2=0x7f7e6c077980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:59:00.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.056+0000 7f7e6b200700 1 -- 172.21.15.62:0/2815614965 shutdown_connections 2024-04-20T21:59:00.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.056+0000 7f7e6b200700 1 -- 172.21.15.62:0/2815614965 wait complete. 2024-04-20T21:59:00.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.216+0000 7fc2e7e00700 1 -- 172.21.15.62:0/672878596 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc2e8102e70 msgr2=0x7fc2e8106e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:59:00.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.216+0000 7fc2e7e00700 1 --2- 172.21.15.62:0/672878596 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc2e8102e70 0x7fc2e8106e40 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fc2d4009910 tx=0x7fc2d4009c20 comp rx=0 tx=0).stop 2024-04-20T21:59:00.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.216+0000 7fc2e7e00700 1 -- 172.21.15.62:0/672878596 shutdown_connections 2024-04-20T21:59:00.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.216+0000 7fc2e7e00700 1 --2- 172.21.15.62:0/672878596 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc2e8102e70 0x7fc2e8106e40 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:59:00.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.216+0000 7fc2e7e00700 1 --2- 172.21.15.62:0/672878596 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc2e81024d0 0x7fc2e81028a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:59:00.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.216+0000 7fc2e7e00700 1 -- 172.21.15.62:0/672878596 >> 172.21.15.62:0/672878596 conn(0x7fc2e80fdd90 msgr2=0x7fc2e81001a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:59:00.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.216+0000 7fc2e7e00700 1 -- 172.21.15.62:0/672878596 shutdown_connections 2024-04-20T21:59:00.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.216+0000 7fc2e7e00700 1 -- 172.21.15.62:0/672878596 wait complete. 2024-04-20T21:59:00.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.217+0000 7fc2e7e00700 1 Processor -- start 2024-04-20T21:59:00.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.217+0000 7fc2e7e00700 1 -- start start 2024-04-20T21:59:00.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.217+0000 7fc2e7e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc2e81024d0 0x7fc2e8197e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:59:00.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.217+0000 7fc2e7e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc2e8102e70 0x7fc2e8198350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:59:00.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.217+0000 7fc2e7e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e81989a0 con 0x7fc2e81024d0 2024-04-20T21:59:00.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.217+0000 7fc2e7e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e8198ae0 con 0x7fc2e8102e70 2024-04-20T21:59:00.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.218+0000 7fc2e6a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc2e81024d0 0x7fc2e8197e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:59:00.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.218+0000 7fc2e6a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc2e81024d0 0x7fc2e8197e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:39850/0 (socket says 172.21.15.62:39850) 2024-04-20T21:59:00.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.218+0000 7fc2e6a00700 1 -- 172.21.15.62:0/3903075049 learned_addr learned my addr 172.21.15.62:0/3903075049 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:59:00.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.218+0000 7fc2e6000700 1 --2- 172.21.15.62:0/3903075049 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc2e8102e70 0x7fc2e8198350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:59:00.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.218+0000 7fc2e6a00700 1 -- 172.21.15.62:0/3903075049 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc2e8102e70 msgr2=0x7fc2e8198350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:59:00.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.218+0000 7fc2e6a00700 1 --2- 172.21.15.62:0/3903075049 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc2e8102e70 0x7fc2e8198350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:59:00.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.218+0000 7fc2e6a00700 1 -- 172.21.15.62:0/3903075049 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2d40095f0 con 0x7fc2e81024d0 2024-04-20T21:59:00.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.218+0000 7fc2e6000700 1 --2- 172.21.15.62:0/3903075049 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc2e8102e70 0x7fc2e8198350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T21:59:00.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.218+0000 7fc2e6a00700 1 --2- 172.21.15.62:0/3903075049 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc2e81024d0 0x7fc2e8197e10 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fc2d800c930 tx=0x7fc2d800ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:59:00.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.219+0000 7fc2df400700 1 -- 172.21.15.62:0/3903075049 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc2d8007ad0 con 0x7fc2e81024d0 2024-04-20T21:59:00.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.219+0000 7fc2df400700 1 -- 172.21.15.62:0/3903075049 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc2d8007c30 con 0x7fc2e81024d0 2024-04-20T21:59:00.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.219+0000 7fc2df400700 1 -- 172.21.15.62:0/3903075049 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc2d8018610 con 0x7fc2e81024d0 2024-04-20T21:59:00.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.219+0000 7fc2e7e00700 1 -- 172.21.15.62:0/3903075049 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2e819c930 con 0x7fc2e81024d0 2024-04-20T21:59:00.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.219+0000 7fc2e7e00700 1 -- 172.21.15.62:0/3903075049 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2e819ce30 con 0x7fc2e81024d0 2024-04-20T21:59:00.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.220+0000 7fc2e7e00700 1 -- 172.21.15.62:0/3903075049 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2e810a840 con 0x7fc2e81024d0 2024-04-20T21:59:00.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.222+0000 7fc2df400700 1 -- 172.21.15.62:0/3903075049 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fc2d800f450 con 0x7fc2e81024d0 2024-04-20T21:59:00.221 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:59:00 smithi062 ceph-mon[76322]: from='client.35292 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:59:00.221 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:59:00 smithi062 ceph-mon[76322]: from='client.35296 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:59:00.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.223+0000 7fc2df400700 1 --2- 172.21.15.62:0/3903075049 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc2cc06ecc0 0x7fc2cc071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:59:00.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.223+0000 7fc2df400700 1 -- 172.21.15.62:0/3903075049 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fc2d808efe0 con 0x7fc2e81024d0 2024-04-20T21:59:00.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.224+0000 7fc2e6000700 1 --2- 172.21.15.62:0/3903075049 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc2cc06ecc0 0x7fc2cc071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:59:00.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.224+0000 7fc2e6000700 1 --2- 172.21.15.62:0/3903075049 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc2cc06ecc0 0x7fc2cc071170 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fc2d4006010 tx=0x7fc2d400b540 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:59:00.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.228+0000 7fc2df400700 1 -- 172.21.15.62:0/3903075049 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fc2d8057d90 con 0x7fc2e81024d0 2024-04-20T21:59:00.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:59:00 smithi189 ceph-mon[57022]: from='client.35292 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:59:00.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:59:00 smithi189 ceph-mon[57022]: from='client.35296 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:59:00.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.499+0000 7fc2e7e00700 1 -- 172.21.15.62:0/3903075049 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fc2e802cc30 con 0x7fc2e81024d0 2024-04-20T21:59:00.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.500+0000 7fc2df400700 1 -- 172.21.15.62:0/3903075049 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 27 v27) v1 ==== 76+0+1888 (secure 0 0 0) 0x7fc2d805b450 con 0x7fc2e81024d0 2024-04-20T21:59:00.500 INFO:teuthology.orchestra.run.smithi062.stdout:e27 2024-04-20T21:59:00.500 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T21:59:00.500 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T21:59:00.500 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T21:59:00.501 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:59:00.501 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T21:59:00.501 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T21:59:00.501 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 20 2024-04-20T21:59:00.501 INFO:teuthology.orchestra.run.smithi062.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-20T21:59:00.501 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T21:59:00.501 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T21:42:01.559061+0000 2024-04-20T21:59:00.501 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T21:59:00.501 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T21:59:00.501 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T21:59:00.501 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T21:59:00.501 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T21:59:00.501 INFO:teuthology.orchestra.run.smithi062.stdout:max_xattr_size 65536 2024-04-20T21:59:00.501 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T21:59:00.501 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T21:59:00.502 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 81 2024-04-20T21:59:00.502 INFO:teuthology.orchestra.run.smithi062.stdout:snap_rstat 1 2024-04-20T21:59:00.502 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T21:59:00.502 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 2 2024-04-20T21:59:00.502 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1 2024-04-20T21:59:00.502 INFO:teuthology.orchestra.run.smithi062.stdout:up {} 2024-04-20T21:59:00.502 INFO:teuthology.orchestra.run.smithi062.stdout:failed 0,1 2024-04-20T21:59:00.502 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T21:59:00.502 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T21:59:00.502 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T21:59:00.502 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T21:59:00.502 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data enabled 2024-04-20T21:59:00.502 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T21:59:00.502 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T21:59:00.503 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 1 2024-04-20T21:59:00.503 INFO:teuthology.orchestra.run.smithi062.stdout:qdb_cluster leader: 0 members: 2024-04-20T21:59:00.503 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:59:00.503 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:59:00.503 INFO:teuthology.orchestra.run.smithi062.stdout:Standby daemons: 2024-04-20T21:59:00.503 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:59:00.503 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ryqblb{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6826/4113916205,v1:172.21.15.62:6827/4113916205] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T21:59:00.503 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ozizwi{-1:34476} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6828/3801975165,v1:172.21.15.62:6829/3801975165] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T21:59:00.503 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.xbgspw{-1:44285} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6826/3996040526,v1:172.21.15.189:6827/3996040526] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:59:00.503 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.dvimer{-1:44727} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6824/3576699232,v1:172.21.15.189:6825/3576699232] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T21:59:00.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.503+0000 7fc2e7e00700 1 -- 172.21.15.62:0/3903075049 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc2cc06ecc0 msgr2=0x7fc2cc071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:59:00.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.503+0000 7fc2e7e00700 1 --2- 172.21.15.62:0/3903075049 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc2cc06ecc0 0x7fc2cc071170 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fc2d4006010 tx=0x7fc2d400b540 comp rx=0 tx=0).stop 2024-04-20T21:59:00.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.503+0000 7fc2e7e00700 1 -- 172.21.15.62:0/3903075049 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc2e81024d0 msgr2=0x7fc2e8197e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:59:00.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.503+0000 7fc2e7e00700 1 --2- 172.21.15.62:0/3903075049 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc2e81024d0 0x7fc2e8197e10 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fc2d800c930 tx=0x7fc2d800ccf0 comp rx=0 tx=0).stop 2024-04-20T21:59:00.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.504+0000 7fc2e7e00700 1 -- 172.21.15.62:0/3903075049 shutdown_connections 2024-04-20T21:59:00.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.504+0000 7fc2e7e00700 1 --2- 172.21.15.62:0/3903075049 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc2cc06ecc0 0x7fc2cc071170 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:59:00.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.504+0000 7fc2e7e00700 1 --2- 172.21.15.62:0/3903075049 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc2e81024d0 0x7fc2e8197e10 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:59:00.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.504+0000 7fc2e7e00700 1 --2- 172.21.15.62:0/3903075049 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc2e8102e70 0x7fc2e8198350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:59:00.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.504+0000 7fc2e7e00700 1 -- 172.21.15.62:0/3903075049 >> 172.21.15.62:0/3903075049 conn(0x7fc2e80fdd90 msgr2=0x7fc2e8100180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:59:00.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.504+0000 7fc2e7e00700 1 -- 172.21.15.62:0/3903075049 shutdown_connections 2024-04-20T21:59:00.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.504+0000 7fc2e7e00700 1 -- 172.21.15.62:0/3903075049 wait complete. 2024-04-20T21:59:00.505 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 27 2024-04-20T21:59:00.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.668+0000 7fd846a00700 1 -- 172.21.15.62:0/488706449 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd84010c8f0 msgr2=0x7fd84010ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:59:00.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.668+0000 7fd846a00700 1 --2- 172.21.15.62:0/488706449 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd84010c8f0 0x7fd84010ccc0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fd830009910 tx=0x7fd830009c20 comp rx=0 tx=0).stop 2024-04-20T21:59:00.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.668+0000 7fd846a00700 1 -- 172.21.15.62:0/488706449 shutdown_connections 2024-04-20T21:59:00.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.668+0000 7fd846a00700 1 --2- 172.21.15.62:0/488706449 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd840106b80 0x7fd840106ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:59:00.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.668+0000 7fd846a00700 1 --2- 172.21.15.62:0/488706449 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd84010c8f0 0x7fd84010ccc0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:59:00.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.668+0000 7fd846a00700 1 -- 172.21.15.62:0/488706449 >> 172.21.15.62:0/488706449 conn(0x7fd840076060 msgr2=0x7fd840078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:59:00.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.669+0000 7fd846a00700 1 -- 172.21.15.62:0/488706449 shutdown_connections 2024-04-20T21:59:00.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.669+0000 7fd846a00700 1 -- 172.21.15.62:0/488706449 wait complete. 2024-04-20T21:59:00.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.670+0000 7fd846a00700 1 Processor -- start 2024-04-20T21:59:00.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.670+0000 7fd846a00700 1 -- start start 2024-04-20T21:59:00.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.670+0000 7fd846a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd840106b80 0x7fd84019c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:59:00.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.670+0000 7fd846a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd84010c8f0 0x7fd84019ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:59:00.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.670+0000 7fd846a00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd84019d120 con 0x7fd840106b80 2024-04-20T21:59:00.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.670+0000 7fd846a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8401a0eb0 con 0x7fd84010c8f0 2024-04-20T21:59:00.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.671+0000 7fd83fe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd840106b80 0x7fd84019c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:59:00.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.671+0000 7fd83fe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd840106b80 0x7fd84019c500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:39870/0 (socket says 172.21.15.62:39870) 2024-04-20T21:59:00.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.671+0000 7fd83fe00700 1 -- 172.21.15.62:0/2361867490 learned_addr learned my addr 172.21.15.62:0/2361867490 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:59:00.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.671+0000 7fd83f400700 1 --2- 172.21.15.62:0/2361867490 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd84010c8f0 0x7fd84019ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:59:00.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.671+0000 7fd83fe00700 1 -- 172.21.15.62:0/2361867490 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd84010c8f0 msgr2=0x7fd84019ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:59:00.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.671+0000 7fd83fe00700 1 --2- 172.21.15.62:0/2361867490 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd84010c8f0 0x7fd84019ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:59:00.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.671+0000 7fd83fe00700 1 -- 172.21.15.62:0/2361867490 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8300095f0 con 0x7fd840106b80 2024-04-20T21:59:00.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.672+0000 7fd83fe00700 1 --2- 172.21.15.62:0/2361867490 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd840106b80 0x7fd84019c500 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fd830006010 tx=0x7fd83000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:59:00.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.672+0000 7fd83cc00700 1 -- 172.21.15.62:0/2361867490 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd83001d070 con 0x7fd840106b80 2024-04-20T21:59:00.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.672+0000 7fd83cc00700 1 -- 172.21.15.62:0/2361867490 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd83000f460 con 0x7fd840106b80 2024-04-20T21:59:00.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.672+0000 7fd83cc00700 1 -- 172.21.15.62:0/2361867490 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd830017630 con 0x7fd840106b80 2024-04-20T21:59:00.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.672+0000 7fd846a00700 1 -- 172.21.15.62:0/2361867490 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8401a1130 con 0x7fd840106b80 2024-04-20T21:59:00.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.672+0000 7fd846a00700 1 -- 172.21.15.62:0/2361867490 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8401a1620 con 0x7fd840106b80 2024-04-20T21:59:00.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.674+0000 7fd846a00700 1 -- 172.21.15.62:0/2361867490 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd840109b20 con 0x7fd840106b80 2024-04-20T21:59:00.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.674+0000 7fd83cc00700 1 -- 172.21.15.62:0/2361867490 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fd83000f5d0 con 0x7fd840106b80 2024-04-20T21:59:00.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.675+0000 7fd83cc00700 1 --2- 172.21.15.62:0/2361867490 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd82406ecc0 0x7fd824071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:59:00.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.676+0000 7fd83cc00700 1 -- 172.21.15.62:0/2361867490 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fd830090350 con 0x7fd840106b80 2024-04-20T21:59:00.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.676+0000 7fd83f400700 1 --2- 172.21.15.62:0/2361867490 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd82406ecc0 0x7fd824071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:59:00.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.677+0000 7fd83f400700 1 --2- 172.21.15.62:0/2361867490 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd82406ecc0 0x7fd824071170 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fd82c005950 tx=0x7fd82c0058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:59:00.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.680+0000 7fd83cc00700 1 -- 172.21.15.62:0/2361867490 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fd8300590e0 con 0x7fd840106b80 2024-04-20T21:59:00.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.941+0000 7fd846a00700 1 -- 172.21.15.62:0/2361867490 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd84019d970 con 0x7fd82406ecc0 2024-04-20T21:59:00.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.943+0000 7fd83cc00700 1 -- 172.21.15.62:0/2361867490 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fd84019d970 con 0x7fd82406ecc0 2024-04-20T21:59:00.942 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:59:00.943 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T21:59:00.943 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T21:59:00.943 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-20T21:59:00.943 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T21:59:00.943 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr", 2024-04-20T21:59:00.943 INFO:teuthology.orchestra.run.smithi062.stdout: "crash", 2024-04-20T21:59:00.943 INFO:teuthology.orchestra.run.smithi062.stdout: "mon", 2024-04-20T21:59:00.943 INFO:teuthology.orchestra.run.smithi062.stdout: "osd" 2024-04-20T21:59:00.943 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T21:59:00.943 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "15/23 daemons upgraded", 2024-04-20T21:59:00.943 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading mds daemons", 2024-04-20T21:59:00.943 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T21:59:00.943 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:59:00.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.946+0000 7fd846a00700 1 -- 172.21.15.62:0/2361867490 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd82406ecc0 msgr2=0x7fd824071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:59:00.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.946+0000 7fd846a00700 1 --2- 172.21.15.62:0/2361867490 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd82406ecc0 0x7fd824071170 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fd82c005950 tx=0x7fd82c0058e0 comp rx=0 tx=0).stop 2024-04-20T21:59:00.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.946+0000 7fd846a00700 1 -- 172.21.15.62:0/2361867490 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd840106b80 msgr2=0x7fd84019c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:59:00.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.946+0000 7fd846a00700 1 --2- 172.21.15.62:0/2361867490 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd840106b80 0x7fd84019c500 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fd830006010 tx=0x7fd83000be20 comp rx=0 tx=0).stop 2024-04-20T21:59:00.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.946+0000 7fd846a00700 1 -- 172.21.15.62:0/2361867490 shutdown_connections 2024-04-20T21:59:00.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.946+0000 7fd846a00700 1 --2- 172.21.15.62:0/2361867490 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd82406ecc0 0x7fd824071170 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:59:00.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.946+0000 7fd846a00700 1 --2- 172.21.15.62:0/2361867490 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd840106b80 0x7fd84019c500 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:59:00.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.946+0000 7fd846a00700 1 --2- 172.21.15.62:0/2361867490 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd84010c8f0 0x7fd84019ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:59:00.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.946+0000 7fd846a00700 1 -- 172.21.15.62:0/2361867490 >> 172.21.15.62:0/2361867490 conn(0x7fd840076060 msgr2=0x7fd840077aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:59:00.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.946+0000 7fd846a00700 1 -- 172.21.15.62:0/2361867490 shutdown_connections 2024-04-20T21:59:00.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:00.946+0000 7fd846a00700 1 -- 172.21.15.62:0/2361867490 wait complete. 2024-04-20T21:59:01.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:01.113+0000 7f5389a00700 1 -- 172.21.15.62:0/2079187263 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5384073090 msgr2=0x7f5384073460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:59:01.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:01.113+0000 7f5389a00700 1 --2- 172.21.15.62:0/2079187263 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5384073090 0x7f5384073460 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f5374009940 tx=0x7f5374009c50 comp rx=0 tx=0).stop 2024-04-20T21:59:01.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:01.113+0000 7f5389a00700 1 -- 172.21.15.62:0/2079187263 shutdown_connections 2024-04-20T21:59:01.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:01.113+0000 7f5389a00700 1 --2- 172.21.15.62:0/2079187263 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53840739a0 0x7f538410c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:59:01.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:01.113+0000 7f5389a00700 1 --2- 172.21.15.62:0/2079187263 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5384073090 0x7f5384073460 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:59:01.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:01.113+0000 7f5389a00700 1 -- 172.21.15.62:0/2079187263 >> 172.21.15.62:0/2079187263 conn(0x7f53840fbda0 msgr2=0x7f53840fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:59:01.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:01.113+0000 7f5389a00700 1 -- 172.21.15.62:0/2079187263 shutdown_connections 2024-04-20T21:59:01.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:01.113+0000 7f5389a00700 1 -- 172.21.15.62:0/2079187263 wait complete. 2024-04-20T21:59:01.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:01.114+0000 7f5389a00700 1 Processor -- start 2024-04-20T21:59:01.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:01.114+0000 7f5389a00700 1 -- start start 2024-04-20T21:59:01.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:01.115+0000 7f5389a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5384073090 0x7f5384198180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:59:01.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:01.115+0000 7f5389a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f53840739a0 0x7f53841986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:59:01.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:01.115+0000 7f5389a00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5384198d10 con 0x7f53840739a0 2024-04-20T21:59:01.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:01.115+0000 7f5389a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5384198e50 con 0x7f5384073090 2024-04-20T21:59:01.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:01.115+0000 7f537be00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f53840739a0 0x7f53841986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:59:01.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:01.115+0000 7f537be00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f53840739a0 0x7f53841986c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:39878/0 (socket says 172.21.15.62:39878) 2024-04-20T21:59:01.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:01.115+0000 7f537be00700 1 -- 172.21.15.62:0/3212868157 learned_addr learned my addr 172.21.15.62:0/3212868157 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:59:01.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:01.116+0000 7f5382a00700 1 --2- 172.21.15.62:0/3212868157 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5384073090 0x7f5384198180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:59:01.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:01.116+0000 7f537be00700 1 -- 172.21.15.62:0/3212868157 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5384073090 msgr2=0x7f5384198180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:59:01.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:01.116+0000 7f537be00700 1 --2- 172.21.15.62:0/3212868157 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5384073090 0x7f5384198180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:59:01.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:01.116+0000 7f537be00700 1 -- 172.21.15.62:0/3212868157 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53740095f0 con 0x7f53840739a0 2024-04-20T21:59:01.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:01.116+0000 7f5382a00700 1 --2- 172.21.15.62:0/3212868157 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5384073090 0x7f5384198180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-20T21:59:01.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:01.116+0000 7f537be00700 1 --2- 172.21.15.62:0/3212868157 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f53840739a0 0x7f53841986c0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f536c00c930 tx=0x7f536c00ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:59:01.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:01.117+0000 7f537b400700 1 -- 172.21.15.62:0/3212868157 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f536c007ad0 con 0x7f53840739a0 2024-04-20T21:59:01.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:01.117+0000 7f537b400700 1 -- 172.21.15.62:0/3212868157 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f536c007c30 con 0x7f53840739a0 2024-04-20T21:59:01.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:01.117+0000 7f537b400700 1 -- 172.21.15.62:0/3212868157 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f536c018610 con 0x7f53840739a0 2024-04-20T21:59:01.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:01.117+0000 7f5389a00700 1 -- 172.21.15.62:0/3212868157 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f538419cca0 con 0x7f53840739a0 2024-04-20T21:59:01.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:01.117+0000 7f5389a00700 1 -- 172.21.15.62:0/3212868157 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f538419d1a0 con 0x7f53840739a0 2024-04-20T21:59:01.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:01.118+0000 7f5389a00700 1 -- 172.21.15.62:0/3212868157 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5384109ce0 con 0x7f53840739a0 2024-04-20T21:59:01.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:01.119+0000 7f537b400700 1 -- 172.21.15.62:0/3212868157 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f536c00f450 con 0x7f53840739a0 2024-04-20T21:59:01.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:01.120+0000 7f537b400700 1 --2- 172.21.15.62:0/3212868157 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f536806ecc0 0x7f5368071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:59:01.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:01.120+0000 7f537b400700 1 -- 172.21.15.62:0/3212868157 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f536c08efe0 con 0x7f53840739a0 2024-04-20T21:59:01.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:01.121+0000 7f5382a00700 1 --2- 172.21.15.62:0/3212868157 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f536806ecc0 0x7f5368071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:59:01.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:01.121+0000 7f5382a00700 1 --2- 172.21.15.62:0/3212868157 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f536806ecc0 0x7f5368071170 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f5374005b40 tx=0x7f5374005ab0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:59:01.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:01.125+0000 7f537b400700 1 -- 172.21.15.62:0/3212868157 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f536c057d90 con 0x7f53840739a0 2024-04-20T21:59:01.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:59:01 smithi062 ceph-mon[76322]: pgmap v712: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:59:01.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:59:01 smithi062 ceph-mon[76322]: from='client.35300 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:59:01.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:59:01 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/2815614965' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:59:01.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:59:01 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/3903075049' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T21:59:01.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:01.458+0000 7f5389a00700 1 -- 172.21.15.62:0/3212868157 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f538404f2a0 con 0x7f53840739a0 2024-04-20T21:59:01.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:01.459+0000 7f537b400700 1 -- 172.21.15.62:0/3212868157 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7f536c05b450 con 0x7f53840739a0 2024-04-20T21:59:01.459 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-20T21:59:01.459 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-20T21:59:01.459 INFO:teuthology.orchestra.run.smithi062.stdout: host smithi062 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T21:59:01.459 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-20T21:59:01.459 INFO:teuthology.orchestra.run.smithi062.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi062 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpq7y7oa_r:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T21:59:01.459 INFO:teuthology.orchestra.run.smithi062.stdout:Traceback (most recent call last): 2024-04-20T21:59:01.459 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-20T21:59:01.459 INFO:teuthology.orchestra.run.smithi062.stdout: return _run_code(code, main_globals, None, 2024-04-20T21:59:01.460 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-20T21:59:01.460 INFO:teuthology.orchestra.run.smithi062.stdout: exec(code, run_globals) 2024-04-20T21:59:01.460 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5618, in 2024-04-20T21:59:01.460 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5606, in main 2024-04-20T21:59:01.460 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 408, in _infer_config 2024-04-20T21:59:01.460 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 323, in _infer_fsid 2024-04-20T21:59:01.460 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 436, in _infer_image 2024-04-20T21:59:01.460 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 310, in _validate_fsid 2024-04-20T21:59:01.460 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 3359, in command_ceph_volume 2024-04-20T21:59:01.460 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-20T21:59:01.460 INFO:teuthology.orchestra.run.smithi062.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi062 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpq7y7oa_r:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T21:59:01.460 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-20T21:59:01.460 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is degraded 2024-04-20T21:59:01.460 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T21:59:01.460 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T21:59:01.460 INFO:teuthology.orchestra.run.smithi062.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-20T21:59:01.461 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-20T21:59:01.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:01.462+0000 7f5389a00700 1 -- 172.21.15.62:0/3212868157 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f536806ecc0 msgr2=0x7f5368071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:59:01.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:01.462+0000 7f5389a00700 1 --2- 172.21.15.62:0/3212868157 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f536806ecc0 0x7f5368071170 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f5374005b40 tx=0x7f5374005ab0 comp rx=0 tx=0).stop 2024-04-20T21:59:01.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:01.462+0000 7f5389a00700 1 -- 172.21.15.62:0/3212868157 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f53840739a0 msgr2=0x7f53841986c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:59:01.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:01.462+0000 7f5389a00700 1 --2- 172.21.15.62:0/3212868157 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f53840739a0 0x7f53841986c0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f536c00c930 tx=0x7f536c00ccf0 comp rx=0 tx=0).stop 2024-04-20T21:59:01.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:01.462+0000 7f5389a00700 1 -- 172.21.15.62:0/3212868157 shutdown_connections 2024-04-20T21:59:01.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:01.462+0000 7f5389a00700 1 --2- 172.21.15.62:0/3212868157 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f536806ecc0 0x7f5368071170 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:59:01.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:01.462+0000 7f5389a00700 1 --2- 172.21.15.62:0/3212868157 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5384073090 0x7f5384198180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:59:01.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:01.462+0000 7f5389a00700 1 --2- 172.21.15.62:0/3212868157 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f53840739a0 0x7f53841986c0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:59:01.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:01.462+0000 7f5389a00700 1 -- 172.21.15.62:0/3212868157 >> 172.21.15.62:0/3212868157 conn(0x7f53840fbda0 msgr2=0x7f5384106e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:59:01.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:01.463+0000 7f5389a00700 1 -- 172.21.15.62:0/3212868157 shutdown_connections 2024-04-20T21:59:01.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:01.463+0000 7f5389a00700 1 -- 172.21.15.62:0/3212868157 wait complete. 2024-04-20T21:59:01.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:59:01 smithi189 ceph-mon[57022]: pgmap v712: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:59:01.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:59:01 smithi189 ceph-mon[57022]: from='client.35300 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:59:01.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:59:01 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/2815614965' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:59:01.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:59:01 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/3903075049' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T21:59:02.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:59:02 smithi062 ceph-mon[76322]: from='client.35312 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:59:02.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:59:02 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/3212868157' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T21:59:02.324 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:59:02 smithi189 ceph-mon[57022]: from='client.35312 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:59:02.324 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:59:02 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/3212868157' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T21:59:03.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:59:03 smithi062 ceph-mon[76322]: pgmap v713: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:59:03.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:59:03 smithi189 ceph-mon[57022]: pgmap v713: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:59:05.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:59:05 smithi189 ceph-mon[57022]: pgmap v714: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:59:05.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:59:05 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:59:05.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:59:05 smithi062 ceph-mon[76322]: pgmap v714: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:59:05.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:59:05 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:59:07.486 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:59:07 smithi189 ceph-mon[57022]: pgmap v715: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:59:07.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:59:07 smithi062 ceph-mon[76322]: pgmap v715: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:59:09.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:59:09 smithi189 ceph-mon[57022]: pgmap v716: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:59:09.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:59:09 smithi062 ceph-mon[76322]: pgmap v716: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:59:11.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:59:11 smithi189 ceph-mon[57022]: pgmap v717: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:59:11.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:59:11 smithi062 ceph-mon[76322]: pgmap v717: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:59:13.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:59:13 smithi189 ceph-mon[57022]: pgmap v718: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:59:13.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:59:13 smithi062 ceph-mon[76322]: pgmap v718: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:59:15.419 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:59:15 smithi062 ceph-mon[76322]: pgmap v719: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:59:15.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:59:15 smithi189 ceph-mon[57022]: pgmap v719: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:59:17.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:59:17 smithi189 ceph-mon[57022]: pgmap v720: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:59:17.537 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:59:17 smithi062 ceph-mon[76322]: pgmap v720: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:59:19.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:59:19 smithi062 ceph-mon[76322]: pgmap v721: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:59:19.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:59:19 smithi189 ceph-mon[57022]: pgmap v721: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:59:20.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:59:20 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:59:20.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:59:20 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:59:21.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:59:21 smithi189 ceph-mon[57022]: pgmap v722: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:59:21.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:59:21 smithi062 ceph-mon[76322]: pgmap v722: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:59:23.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:59:23 smithi189 ceph-mon[57022]: pgmap v723: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:59:23.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:59:23 smithi062 ceph-mon[76322]: pgmap v723: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:59:25.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:59:25 smithi189 ceph-mon[57022]: pgmap v724: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:59:25.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:59:25 smithi062 ceph-mon[76322]: pgmap v724: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:59:27.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:59:27 smithi189 ceph-mon[57022]: pgmap v725: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:59:27.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:59:27 smithi062 ceph-mon[76322]: pgmap v725: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:59:29.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:59:29 smithi189 ceph-mon[57022]: pgmap v726: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:59:29.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:59:29 smithi062 ceph-mon[76322]: pgmap v726: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:59:31.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:59:31 smithi062 ceph-mon[76322]: pgmap v727: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:59:31.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:59:31 smithi189 ceph-mon[57022]: pgmap v727: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:59:31.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:31.644+0000 7ffa75400700 1 -- 172.21.15.62:0/3328502526 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa70073980 msgr2=0x7ffa7010c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:59:31.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:31.644+0000 7ffa75400700 1 --2- 172.21.15.62:0/3328502526 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa70073980 0x7ffa7010c600 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7ffa58009940 tx=0x7ffa58009c50 comp rx=0 tx=0).stop 2024-04-20T21:59:31.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:31.644+0000 7ffa75400700 1 -- 172.21.15.62:0/3328502526 shutdown_connections 2024-04-20T21:59:31.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:31.644+0000 7ffa75400700 1 --2- 172.21.15.62:0/3328502526 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa70073980 0x7ffa7010c600 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:59:31.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:31.644+0000 7ffa75400700 1 --2- 172.21.15.62:0/3328502526 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffa70073070 0x7ffa70073440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:59:31.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:31.644+0000 7ffa75400700 1 -- 172.21.15.62:0/3328502526 >> 172.21.15.62:0/3328502526 conn(0x7ffa700fbda0 msgr2=0x7ffa700fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:59:31.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:31.644+0000 7ffa75400700 1 -- 172.21.15.62:0/3328502526 shutdown_connections 2024-04-20T21:59:31.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:31.644+0000 7ffa75400700 1 -- 172.21.15.62:0/3328502526 wait complete. 2024-04-20T21:59:31.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:31.646+0000 7ffa75400700 1 Processor -- start 2024-04-20T21:59:31.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:31.646+0000 7ffa75400700 1 -- start start 2024-04-20T21:59:31.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:31.646+0000 7ffa75400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffa70073070 0x7ffa701981c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:59:31.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:31.646+0000 7ffa75400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa70073980 0x7ffa70198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:59:31.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:31.647+0000 7ffa75400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa70198de0 con 0x7ffa70073980 2024-04-20T21:59:31.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:31.647+0000 7ffa75400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa7019cb70 con 0x7ffa70073070 2024-04-20T21:59:31.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:31.647+0000 7ffa65e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa70073980 0x7ffa70198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:59:31.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:31.647+0000 7ffa65e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa70073980 0x7ffa70198700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52224/0 (socket says 172.21.15.62:52224) 2024-04-20T21:59:31.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:31.647+0000 7ffa65e00700 1 -- 172.21.15.62:0/1208583772 learned_addr learned my addr 172.21.15.62:0/1208583772 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:59:31.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:31.647+0000 7ffa6e800700 1 --2- 172.21.15.62:0/1208583772 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffa70073070 0x7ffa701981c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:59:31.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:31.647+0000 7ffa65e00700 1 -- 172.21.15.62:0/1208583772 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffa70073070 msgr2=0x7ffa701981c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:59:31.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:31.647+0000 7ffa65e00700 1 --2- 172.21.15.62:0/1208583772 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffa70073070 0x7ffa701981c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:59:31.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:31.647+0000 7ffa65e00700 1 -- 172.21.15.62:0/1208583772 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa580095f0 con 0x7ffa70073980 2024-04-20T21:59:31.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:31.648+0000 7ffa65e00700 1 --2- 172.21.15.62:0/1208583772 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa70073980 0x7ffa70198700 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7ffa580055f0 tx=0x7ffa580056d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:59:31.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:31.648+0000 7ffa67e00700 1 -- 172.21.15.62:0/1208583772 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffa5801d070 con 0x7ffa70073980 2024-04-20T21:59:31.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:31.648+0000 7ffa67e00700 1 -- 172.21.15.62:0/1208583772 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ffa5800f460 con 0x7ffa70073980 2024-04-20T21:59:31.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:31.648+0000 7ffa67e00700 1 -- 172.21.15.62:0/1208583772 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffa58021640 con 0x7ffa70073980 2024-04-20T21:59:31.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:31.648+0000 7ffa75400700 1 -- 172.21.15.62:0/1208583772 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa7019cdf0 con 0x7ffa70073980 2024-04-20T21:59:31.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:31.648+0000 7ffa75400700 1 -- 172.21.15.62:0/1208583772 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa7019d2e0 con 0x7ffa70073980 2024-04-20T21:59:31.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:31.649+0000 7ffa75400700 1 -- 172.21.15.62:0/1208583772 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa70109d00 con 0x7ffa70073980 2024-04-20T21:59:31.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:31.650+0000 7ffa67e00700 1 -- 172.21.15.62:0/1208583772 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ffa5800fa60 con 0x7ffa70073980 2024-04-20T21:59:31.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:31.651+0000 7ffa67e00700 1 --2- 172.21.15.62:0/1208583772 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ffa5406ecc0 0x7ffa54071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:59:31.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:31.652+0000 7ffa67e00700 1 -- 172.21.15.62:0/1208583772 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7ffa58090110 con 0x7ffa70073980 2024-04-20T21:59:31.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:31.652+0000 7ffa6e800700 1 --2- 172.21.15.62:0/1208583772 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ffa5406ecc0 0x7ffa54071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:59:31.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:31.652+0000 7ffa6e800700 1 --2- 172.21.15.62:0/1208583772 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ffa5406ecc0 0x7ffa54071170 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7ffa60009c10 tx=0x7ffa60009040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:59:31.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:31.657+0000 7ffa67e00700 1 -- 172.21.15.62:0/1208583772 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7ffa58058e20 con 0x7ffa70073980 2024-04-20T21:59:31.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:31.911+0000 7ffa75400700 1 -- 172.21.15.62:0/1208583772 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ffa7019d500 con 0x7ffa5406ecc0 2024-04-20T21:59:31.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:31.914+0000 7ffa67e00700 1 -- 172.21.15.62:0/1208583772 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7ffa7019d500 con 0x7ffa5406ecc0 2024-04-20T21:59:31.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:31.917+0000 7ffa75400700 1 -- 172.21.15.62:0/1208583772 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ffa5406ecc0 msgr2=0x7ffa54071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:59:31.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:31.917+0000 7ffa75400700 1 --2- 172.21.15.62:0/1208583772 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ffa5406ecc0 0x7ffa54071170 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7ffa60009c10 tx=0x7ffa60009040 comp rx=0 tx=0).stop 2024-04-20T21:59:31.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:31.917+0000 7ffa75400700 1 -- 172.21.15.62:0/1208583772 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa70073980 msgr2=0x7ffa70198700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:59:31.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:31.917+0000 7ffa75400700 1 --2- 172.21.15.62:0/1208583772 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa70073980 0x7ffa70198700 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7ffa580055f0 tx=0x7ffa580056d0 comp rx=0 tx=0).stop 2024-04-20T21:59:31.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:31.917+0000 7ffa75400700 1 -- 172.21.15.62:0/1208583772 shutdown_connections 2024-04-20T21:59:31.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:31.918+0000 7ffa75400700 1 --2- 172.21.15.62:0/1208583772 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ffa5406ecc0 0x7ffa54071170 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:59:31.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:31.918+0000 7ffa75400700 1 --2- 172.21.15.62:0/1208583772 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffa70073070 0x7ffa701981c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:59:31.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:31.918+0000 7ffa75400700 1 --2- 172.21.15.62:0/1208583772 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa70073980 0x7ffa70198700 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:59:31.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:31.918+0000 7ffa75400700 1 -- 172.21.15.62:0/1208583772 >> 172.21.15.62:0/1208583772 conn(0x7ffa700fbda0 msgr2=0x7ffa70106e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:59:31.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:31.918+0000 7ffa75400700 1 -- 172.21.15.62:0/1208583772 shutdown_connections 2024-04-20T21:59:31.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:31.918+0000 7ffa75400700 1 -- 172.21.15.62:0/1208583772 wait complete. 2024-04-20T21:59:31.933 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T21:59:32.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.071+0000 7fc093200700 1 -- 172.21.15.62:0/1177574099 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc09410c660 msgr2=0x7fc09410ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:59:32.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.071+0000 7fc093200700 1 --2- 172.21.15.62:0/1177574099 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc09410c660 0x7fc09410ca30 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fc084009910 tx=0x7fc084009c20 comp rx=0 tx=0).stop 2024-04-20T21:59:32.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.071+0000 7fc093200700 1 -- 172.21.15.62:0/1177574099 shutdown_connections 2024-04-20T21:59:32.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.071+0000 7fc093200700 1 --2- 172.21.15.62:0/1177574099 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc094106b80 0x7fc094106ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:59:32.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.071+0000 7fc093200700 1 --2- 172.21.15.62:0/1177574099 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc09410c660 0x7fc09410ca30 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:59:32.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.071+0000 7fc093200700 1 -- 172.21.15.62:0/1177574099 >> 172.21.15.62:0/1177574099 conn(0x7fc094075e90 msgr2=0x7fc0940782a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:59:32.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.071+0000 7fc093200700 1 -- 172.21.15.62:0/1177574099 shutdown_connections 2024-04-20T21:59:32.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.071+0000 7fc093200700 1 -- 172.21.15.62:0/1177574099 wait complete. 2024-04-20T21:59:32.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.072+0000 7fc093200700 1 Processor -- start 2024-04-20T21:59:32.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.072+0000 7fc093200700 1 -- start start 2024-04-20T21:59:32.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.073+0000 7fc093200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc094106b80 0x7fc09419c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:59:32.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.073+0000 7fc093200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc09410c660 0x7fc09419c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:59:32.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.073+0000 7fc093200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc09419cea0 con 0x7fc094106b80 2024-04-20T21:59:32.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.073+0000 7fc093200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0941a0bc0 con 0x7fc09410c660 2024-04-20T21:59:32.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.073+0000 7fc091e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc094106b80 0x7fc09419c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:59:32.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.073+0000 7fc091e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc094106b80 0x7fc09419c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52230/0 (socket says 172.21.15.62:52230) 2024-04-20T21:59:32.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.073+0000 7fc091e00700 1 -- 172.21.15.62:0/2087805757 learned_addr learned my addr 172.21.15.62:0/2087805757 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:59:32.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.073+0000 7fc091e00700 1 -- 172.21.15.62:0/2087805757 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc09410c660 msgr2=0x7fc09419c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:59:32.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.073+0000 7fc091e00700 1 --2- 172.21.15.62:0/2087805757 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc09410c660 0x7fc09419c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:59:32.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.073+0000 7fc091e00700 1 -- 172.21.15.62:0/2087805757 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0840095f0 con 0x7fc094106b80 2024-04-20T21:59:32.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.074+0000 7fc091e00700 1 --2- 172.21.15.62:0/2087805757 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc094106b80 0x7fc09419c280 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fc084006010 tx=0x7fc08400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:59:32.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.074+0000 7fc08aa00700 1 -- 172.21.15.62:0/2087805757 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc08401d070 con 0x7fc094106b80 2024-04-20T21:59:32.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.074+0000 7fc08aa00700 1 -- 172.21.15.62:0/2087805757 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc084004fb0 con 0x7fc094106b80 2024-04-20T21:59:32.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.074+0000 7fc08aa00700 1 -- 172.21.15.62:0/2087805757 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc08400f690 con 0x7fc094106b80 2024-04-20T21:59:32.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.074+0000 7fc093200700 1 -- 172.21.15.62:0/2087805757 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0941a0e40 con 0x7fc094106b80 2024-04-20T21:59:32.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.074+0000 7fc093200700 1 -- 172.21.15.62:0/2087805757 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0941a1330 con 0x7fc094106b80 2024-04-20T21:59:32.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.075+0000 7fc093200700 1 -- 172.21.15.62:0/2087805757 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc09410eb60 con 0x7fc094106b80 2024-04-20T21:59:32.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.076+0000 7fc08aa00700 1 -- 172.21.15.62:0/2087805757 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fc0840228a0 con 0x7fc094106b80 2024-04-20T21:59:32.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.077+0000 7fc08aa00700 1 --2- 172.21.15.62:0/2087805757 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc07806ecc0 0x7fc078071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:59:32.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.078+0000 7fc08aa00700 1 -- 172.21.15.62:0/2087805757 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fc084090700 con 0x7fc094106b80 2024-04-20T21:59:32.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.078+0000 7fc091400700 1 --2- 172.21.15.62:0/2087805757 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc07806ecc0 0x7fc078071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:59:32.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.079+0000 7fc091400700 1 --2- 172.21.15.62:0/2087805757 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc07806ecc0 0x7fc078071170 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fc09419d830 tx=0x7fc080005c80 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:59:32.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.082+0000 7fc08aa00700 1 -- 172.21.15.62:0/2087805757 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fc084059490 con 0x7fc094106b80 2024-04-20T21:59:32.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.330+0000 7fc093200700 1 -- 172.21.15.62:0/2087805757 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc0941a1660 con 0x7fc07806ecc0 2024-04-20T21:59:32.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.333+0000 7fc08aa00700 1 -- 172.21.15.62:0/2087805757 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fc0941a1660 con 0x7fc07806ecc0 2024-04-20T21:59:32.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.336+0000 7fc093200700 1 -- 172.21.15.62:0/2087805757 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc07806ecc0 msgr2=0x7fc078071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:59:32.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.336+0000 7fc093200700 1 --2- 172.21.15.62:0/2087805757 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc07806ecc0 0x7fc078071170 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fc09419d830 tx=0x7fc080005c80 comp rx=0 tx=0).stop 2024-04-20T21:59:32.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.336+0000 7fc093200700 1 -- 172.21.15.62:0/2087805757 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc094106b80 msgr2=0x7fc09419c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:59:32.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.336+0000 7fc093200700 1 --2- 172.21.15.62:0/2087805757 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc094106b80 0x7fc09419c280 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fc084006010 tx=0x7fc08400be20 comp rx=0 tx=0).stop 2024-04-20T21:59:32.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.336+0000 7fc093200700 1 -- 172.21.15.62:0/2087805757 shutdown_connections 2024-04-20T21:59:32.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.336+0000 7fc093200700 1 --2- 172.21.15.62:0/2087805757 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc07806ecc0 0x7fc078071170 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:59:32.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.336+0000 7fc093200700 1 --2- 172.21.15.62:0/2087805757 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc094106b80 0x7fc09419c280 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:59:32.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.336+0000 7fc093200700 1 --2- 172.21.15.62:0/2087805757 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc09410c660 0x7fc09419c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:59:32.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.336+0000 7fc093200700 1 -- 172.21.15.62:0/2087805757 >> 172.21.15.62:0/2087805757 conn(0x7fc094075e90 msgr2=0x7fc094077980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:59:32.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.336+0000 7fc093200700 1 -- 172.21.15.62:0/2087805757 shutdown_connections 2024-04-20T21:59:32.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.336+0000 7fc093200700 1 -- 172.21.15.62:0/2087805757 wait complete. 2024-04-20T21:59:32.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.485+0000 7f0a1bc00700 1 -- 172.21.15.62:0/4214548018 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a140ff3c0 msgr2=0x7f0a140ff790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:59:32.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.485+0000 7f0a1bc00700 1 --2- 172.21.15.62:0/4214548018 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a140ff3c0 0x7f0a140ff790 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f09fc009910 tx=0x7f09fc009c20 comp rx=0 tx=0).stop 2024-04-20T21:59:32.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.486+0000 7f0a1bc00700 1 -- 172.21.15.62:0/4214548018 shutdown_connections 2024-04-20T21:59:32.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.486+0000 7f0a1bc00700 1 --2- 172.21.15.62:0/4214548018 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a140ffd60 0x7f0a14103d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:59:32.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.486+0000 7f0a1bc00700 1 --2- 172.21.15.62:0/4214548018 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a140ff3c0 0x7f0a140ff790 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:59:32.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.486+0000 7f0a1bc00700 1 -- 172.21.15.62:0/4214548018 >> 172.21.15.62:0/4214548018 conn(0x7f0a140fac60 msgr2=0x7f0a140fd070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:59:32.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.486+0000 7f0a1bc00700 1 -- 172.21.15.62:0/4214548018 shutdown_connections 2024-04-20T21:59:32.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.486+0000 7f0a1bc00700 1 -- 172.21.15.62:0/4214548018 wait complete. 2024-04-20T21:59:32.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.487+0000 7f0a1bc00700 1 Processor -- start 2024-04-20T21:59:32.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.487+0000 7f0a1bc00700 1 -- start start 2024-04-20T21:59:32.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.488+0000 7f0a1bc00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a140ff3c0 0x7f0a14195e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:59:32.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.488+0000 7f0a1bc00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a140ffd60 0x7f0a141963b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:59:32.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.488+0000 7f0a1bc00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a14196a90 con 0x7f0a140ff3c0 2024-04-20T21:59:32.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.488+0000 7f0a1bc00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a1419a7b0 con 0x7f0a140ffd60 2024-04-20T21:59:32.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.488+0000 7f0a19200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a140ff3c0 0x7f0a14195e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:59:32.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.488+0000 7f0a19200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a140ff3c0 0x7f0a14195e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52246/0 (socket says 172.21.15.62:52246) 2024-04-20T21:59:32.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.488+0000 7f0a19200700 1 -- 172.21.15.62:0/4252156848 learned_addr learned my addr 172.21.15.62:0/4252156848 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:59:32.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.489+0000 7f0a0be00700 1 --2- 172.21.15.62:0/4252156848 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a140ffd60 0x7f0a141963b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:59:32.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.489+0000 7f0a19200700 1 -- 172.21.15.62:0/4252156848 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a140ffd60 msgr2=0x7f0a141963b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:59:32.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.489+0000 7f0a19200700 1 --2- 172.21.15.62:0/4252156848 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a140ffd60 0x7f0a141963b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:59:32.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.489+0000 7f0a19200700 1 -- 172.21.15.62:0/4252156848 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09fc0095f0 con 0x7f0a140ff3c0 2024-04-20T21:59:32.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.489+0000 7f0a19200700 1 --2- 172.21.15.62:0/4252156848 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a140ff3c0 0x7f0a14195e70 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f09fc00b5c0 tx=0x7f09fc00be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:59:32.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.489+0000 7f0a12000700 1 -- 172.21.15.62:0/4252156848 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f09fc01d070 con 0x7f0a140ff3c0 2024-04-20T21:59:32.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.490+0000 7f0a1bc00700 1 -- 172.21.15.62:0/4252156848 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a1419aa30 con 0x7f0a140ff3c0 2024-04-20T21:59:32.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.490+0000 7f0a12000700 1 -- 172.21.15.62:0/4252156848 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f09fc00f460 con 0x7f0a140ff3c0 2024-04-20T21:59:32.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.490+0000 7f0a12000700 1 -- 172.21.15.62:0/4252156848 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f09fc017610 con 0x7f0a140ff3c0 2024-04-20T21:59:32.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.490+0000 7f0a1bc00700 1 -- 172.21.15.62:0/4252156848 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a1419af20 con 0x7f0a140ff3c0 2024-04-20T21:59:32.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.491+0000 7f0a1bc00700 1 -- 172.21.15.62:0/4252156848 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a1404f2a0 con 0x7f0a140ff3c0 2024-04-20T21:59:32.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.492+0000 7f0a12000700 1 -- 172.21.15.62:0/4252156848 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f09fc00f5d0 con 0x7f0a140ff3c0 2024-04-20T21:59:32.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.493+0000 7f0a12000700 1 --2- 172.21.15.62:0/4252156848 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f09f406ecc0 0x7f09f4071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:59:32.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.493+0000 7f0a12000700 1 -- 172.21.15.62:0/4252156848 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f09fc090510 con 0x7f0a140ff3c0 2024-04-20T21:59:32.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.493+0000 7f0a0be00700 1 --2- 172.21.15.62:0/4252156848 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f09f406ecc0 0x7f09f4071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:59:32.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.494+0000 7f0a0be00700 1 --2- 172.21.15.62:0/4252156848 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f09f406ecc0 0x7f09f4071170 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f0a14197420 tx=0x7f0a04005cf0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:59:32.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.498+0000 7f0a12000700 1 -- 172.21.15.62:0/4252156848 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f09fc059220 con 0x7f0a140ff3c0 2024-04-20T21:59:32.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.731+0000 7f0a1bc00700 1 -- 172.21.15.62:0/4252156848 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f0a1419b330 con 0x7f09f406ecc0 2024-04-20T21:59:32.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.749+0000 7f0a12000700 1 -- 172.21.15.62:0/4252156848 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f0a1419b330 con 0x7f09f406ecc0 2024-04-20T21:59:32.749 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T21:59:32.749 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (25m) 87s ago 39m 15.3M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T21:59:32.749 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (39m) 87s ago 39m 10.4M - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T21:59:32.749 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (38m) 91s ago 38m 10.1M - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T21:59:32.749 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (21m) 87s ago 39m 6660k - 19.0.0-3133-ga8c84f59 76bc6b91b726 2825d5ba11c6 2024-04-20T21:59:32.749 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (21m) 91s ago 38m 6665k - 19.0.0-3133-ga8c84f59 76bc6b91b726 20f746fa1bba 2024-04-20T21:59:32.749 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (24m) 87s ago 38m 73.5M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T21:59:32.749 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (16m) 87s ago 31m 18.5M - 19.0.0-3133-ga8c84f59 76bc6b91b726 de80bc90a683 2024-04-20T21:59:32.749 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (17m) 87s ago 32m 18.7M - 19.0.0-3133-ga8c84f59 76bc6b91b726 cd793b7ee98e 2024-04-20T21:59:32.750 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (97s) 91s ago 32m 11.0M - 19.0.0-3133-ga8c84f59 76bc6b91b726 0e7b0bc4d1ab 2024-04-20T21:59:32.750 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (31m) 91s ago 31m 18.6M - 18.2.0 dc2bc1663786 f2eec2ee8951 2024-04-20T21:59:32.750 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (28m) 87s ago 41m 513M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T21:59:32.750 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (26m) 91s ago 37m 459M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T21:59:32.750 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (23m) 87s ago 41m 71.7M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 019eb1479485 2024-04-20T21:59:32.750 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (22m) 91s ago 37m 51.0M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 55aac560aab6 2024-04-20T21:59:32.750 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (26m) 87s ago 39m 8472k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T21:59:32.750 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (26m) 91s ago 38m 8048k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T21:59:32.750 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (20m) 87s ago 36m 91.2M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 ee2bb901f26e 2024-04-20T21:59:32.750 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (20m) 87s ago 36m 94.6M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 9075152d4162 2024-04-20T21:59:32.750 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (19m) 87s ago 35m 78.1M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 2d4f3e38dc0e 2024-04-20T21:59:32.750 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (19m) 91s ago 35m 92.6M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 8f97b17ae0eb 2024-04-20T21:59:32.750 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (18m) 91s ago 34m 95.6M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 653d8079967d 2024-04-20T21:59:32.750 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (17m) 91s ago 33m 89.2M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 596dded14c12 2024-04-20T21:59:32.751 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (25m) 87s ago 38m 82.1M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T21:59:32.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.752+0000 7f0a1bc00700 1 -- 172.21.15.62:0/4252156848 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f09f406ecc0 msgr2=0x7f09f4071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:59:32.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.752+0000 7f0a1bc00700 1 --2- 172.21.15.62:0/4252156848 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f09f406ecc0 0x7f09f4071170 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f0a14197420 tx=0x7f0a04005cf0 comp rx=0 tx=0).stop 2024-04-20T21:59:32.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.752+0000 7f0a1bc00700 1 -- 172.21.15.62:0/4252156848 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a140ff3c0 msgr2=0x7f0a14195e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:59:32.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.752+0000 7f0a1bc00700 1 --2- 172.21.15.62:0/4252156848 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a140ff3c0 0x7f0a14195e70 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f09fc00b5c0 tx=0x7f09fc00be00 comp rx=0 tx=0).stop 2024-04-20T21:59:32.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.752+0000 7f0a1bc00700 1 -- 172.21.15.62:0/4252156848 shutdown_connections 2024-04-20T21:59:32.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.752+0000 7f0a1bc00700 1 --2- 172.21.15.62:0/4252156848 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f09f406ecc0 0x7f09f4071170 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:59:32.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.752+0000 7f0a1bc00700 1 --2- 172.21.15.62:0/4252156848 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a140ff3c0 0x7f0a14195e70 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:59:32.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.752+0000 7f0a1bc00700 1 --2- 172.21.15.62:0/4252156848 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a140ffd60 0x7f0a141963b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:59:32.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.752+0000 7f0a1bc00700 1 -- 172.21.15.62:0/4252156848 >> 172.21.15.62:0/4252156848 conn(0x7f0a140fac60 msgr2=0x7f0a14103480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:59:32.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.752+0000 7f0a1bc00700 1 -- 172.21.15.62:0/4252156848 shutdown_connections 2024-04-20T21:59:32.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.753+0000 7f0a1bc00700 1 -- 172.21.15.62:0/4252156848 wait complete. 2024-04-20T21:59:32.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.912+0000 7fcc36e00700 1 -- 172.21.15.62:0/274333606 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcc30103e30 msgr2=0x7fcc301042a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:59:32.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.912+0000 7fcc36e00700 1 --2- 172.21.15.62:0/274333606 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcc30103e30 0x7fcc301042a0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fcc1c009910 tx=0x7fcc1c009c20 comp rx=0 tx=0).stop 2024-04-20T21:59:32.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.912+0000 7fcc36e00700 1 -- 172.21.15.62:0/274333606 shutdown_connections 2024-04-20T21:59:32.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.912+0000 7fcc36e00700 1 --2- 172.21.15.62:0/274333606 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcc30103e30 0x7fcc301042a0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:59:32.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.912+0000 7fcc36e00700 1 --2- 172.21.15.62:0/274333606 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc301025d0 0x7fcc301029a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:59:32.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.912+0000 7fcc36e00700 1 -- 172.21.15.62:0/274333606 >> 172.21.15.62:0/274333606 conn(0x7fcc300fdd90 msgr2=0x7fcc301001a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:59:32.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.912+0000 7fcc36e00700 1 -- 172.21.15.62:0/274333606 shutdown_connections 2024-04-20T21:59:32.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.913+0000 7fcc36e00700 1 -- 172.21.15.62:0/274333606 wait complete. 2024-04-20T21:59:32.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.914+0000 7fcc36e00700 1 Processor -- start 2024-04-20T21:59:32.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.914+0000 7fcc36e00700 1 -- start start 2024-04-20T21:59:32.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.914+0000 7fcc36e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc301025d0 0x7fcc30197e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:59:32.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.914+0000 7fcc36e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcc30103e30 0x7fcc301983b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:59:32.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.914+0000 7fcc36e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc30198a00 con 0x7fcc30103e30 2024-04-20T21:59:32.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.914+0000 7fcc36e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc30198b40 con 0x7fcc301025d0 2024-04-20T21:59:32.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.914+0000 7fcc35000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcc30103e30 0x7fcc301983b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:59:32.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.915+0000 7fcc35000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcc30103e30 0x7fcc301983b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52274/0 (socket says 172.21.15.62:52274) 2024-04-20T21:59:32.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.915+0000 7fcc35000700 1 -- 172.21.15.62:0/1992190769 learned_addr learned my addr 172.21.15.62:0/1992190769 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:59:32.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.915+0000 7fcc35a00700 1 --2- 172.21.15.62:0/1992190769 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc301025d0 0x7fcc30197e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:59:32.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.915+0000 7fcc35000700 1 -- 172.21.15.62:0/1992190769 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc301025d0 msgr2=0x7fcc30197e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:59:32.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.915+0000 7fcc35000700 1 --2- 172.21.15.62:0/1992190769 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc301025d0 0x7fcc30197e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:59:32.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.915+0000 7fcc35000700 1 -- 172.21.15.62:0/1992190769 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc1c0095f0 con 0x7fcc30103e30 2024-04-20T21:59:32.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.915+0000 7fcc35000700 1 --2- 172.21.15.62:0/1992190769 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcc30103e30 0x7fcc301983b0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fcc1c00b5c0 tx=0x7fcc1c00be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:59:32.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.916+0000 7fcc2e000700 1 -- 172.21.15.62:0/1992190769 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcc1c01d070 con 0x7fcc30103e30 2024-04-20T21:59:32.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.916+0000 7fcc2e000700 1 -- 172.21.15.62:0/1992190769 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcc1c00f460 con 0x7fcc30103e30 2024-04-20T21:59:32.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.916+0000 7fcc36e00700 1 -- 172.21.15.62:0/1992190769 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc3019c930 con 0x7fcc30103e30 2024-04-20T21:59:32.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.916+0000 7fcc2e000700 1 -- 172.21.15.62:0/1992190769 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcc1c017610 con 0x7fcc30103e30 2024-04-20T21:59:32.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.916+0000 7fcc36e00700 1 -- 172.21.15.62:0/1992190769 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc3019cea0 con 0x7fcc30103e30 2024-04-20T21:59:32.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.917+0000 7fcc36e00700 1 -- 172.21.15.62:0/1992190769 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc30062380 con 0x7fcc30103e30 2024-04-20T21:59:32.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.918+0000 7fcc2e000700 1 -- 172.21.15.62:0/1992190769 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fcc1c00f5d0 con 0x7fcc30103e30 2024-04-20T21:59:32.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.919+0000 7fcc2e000700 1 --2- 172.21.15.62:0/1992190769 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fcc1806ecc0 0x7fcc18071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:59:32.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.919+0000 7fcc2e000700 1 -- 172.21.15.62:0/1992190769 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fcc1c090440 con 0x7fcc30103e30 2024-04-20T21:59:32.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.919+0000 7fcc35a00700 1 --2- 172.21.15.62:0/1992190769 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fcc1806ecc0 0x7fcc18071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:59:32.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.920+0000 7fcc35a00700 1 --2- 172.21.15.62:0/1992190769 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fcc1806ecc0 0x7fcc18071170 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fcc2400ba60 tx=0x7fcc24005d50 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:59:32.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:32.924+0000 7fcc2e000700 1 -- 172.21.15.62:0/1992190769 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fcc1c0591d0 con 0x7fcc30103e30 2024-04-20T21:59:33.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:33.249+0000 7fcc36e00700 1 -- 172.21.15.62:0/1992190769 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fcc3019d1d0 con 0x7fcc30103e30 2024-04-20T21:59:33.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:33.250+0000 7fcc2e000700 1 -- 172.21.15.62:0/1992190769 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7fcc1c05c890 con 0x7fcc30103e30 2024-04-20T21:59:33.249 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:59:33.249 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T21:59:33.250 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:59:33.250 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:59:33.250 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T21:59:33.250 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T21:59:33.250 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:59:33.250 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T21:59:33.250 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 6 2024-04-20T21:59:33.250 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:59:33.250 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T21:59:33.250 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 1, 2024-04-20T21:59:33.250 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 3 2024-04-20T21:59:33.250 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T21:59:33.250 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T21:59:33.250 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 1, 2024-04-20T21:59:33.250 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 13 2024-04-20T21:59:33.251 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T21:59:33.251 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:59:33.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:33.252+0000 7fcc36e00700 1 -- 172.21.15.62:0/1992190769 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fcc1806ecc0 msgr2=0x7fcc18071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:59:33.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:33.253+0000 7fcc36e00700 1 --2- 172.21.15.62:0/1992190769 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fcc1806ecc0 0x7fcc18071170 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fcc2400ba60 tx=0x7fcc24005d50 comp rx=0 tx=0).stop 2024-04-20T21:59:33.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:33.253+0000 7fcc36e00700 1 -- 172.21.15.62:0/1992190769 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcc30103e30 msgr2=0x7fcc301983b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:59:33.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:33.253+0000 7fcc36e00700 1 --2- 172.21.15.62:0/1992190769 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcc30103e30 0x7fcc301983b0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fcc1c00b5c0 tx=0x7fcc1c00be00 comp rx=0 tx=0).stop 2024-04-20T21:59:33.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:33.253+0000 7fcc36e00700 1 -- 172.21.15.62:0/1992190769 shutdown_connections 2024-04-20T21:59:33.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:33.253+0000 7fcc36e00700 1 --2- 172.21.15.62:0/1992190769 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fcc1806ecc0 0x7fcc18071170 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:59:33.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:33.253+0000 7fcc36e00700 1 --2- 172.21.15.62:0/1992190769 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc301025d0 0x7fcc30197e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:59:33.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:33.253+0000 7fcc36e00700 1 --2- 172.21.15.62:0/1992190769 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcc30103e30 0x7fcc301983b0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:59:33.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:33.253+0000 7fcc36e00700 1 -- 172.21.15.62:0/1992190769 >> 172.21.15.62:0/1992190769 conn(0x7fcc300fdd90 msgr2=0x7fcc300fe720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:59:33.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:33.253+0000 7fcc36e00700 1 -- 172.21.15.62:0/1992190769 shutdown_connections 2024-04-20T21:59:33.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:33.253+0000 7fcc36e00700 1 -- 172.21.15.62:0/1992190769 wait complete. 2024-04-20T21:59:33.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:33.410+0000 7f6ff6800700 1 -- 172.21.15.62:0/3084678470 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6ff0106b80 msgr2=0x7f6ff0106ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:59:33.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:33.410+0000 7f6ff6800700 1 --2- 172.21.15.62:0/3084678470 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6ff0106b80 0x7f6ff0106ff0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f6fdc009910 tx=0x7f6fdc009c20 comp rx=0 tx=0).stop 2024-04-20T21:59:33.410 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:59:33 smithi062 ceph-mon[76322]: pgmap v728: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:59:33.410 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:59:33 smithi062 ceph-mon[76322]: from='client.35320 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:59:33.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:33.410+0000 7f6ff6800700 1 -- 172.21.15.62:0/3084678470 shutdown_connections 2024-04-20T21:59:33.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:33.410+0000 7f6ff6800700 1 --2- 172.21.15.62:0/3084678470 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6ff0106b80 0x7f6ff0106ff0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:59:33.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:33.410+0000 7f6ff6800700 1 --2- 172.21.15.62:0/3084678470 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ff010c660 0x7f6ff010ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:59:33.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:33.410+0000 7f6ff6800700 1 -- 172.21.15.62:0/3084678470 >> 172.21.15.62:0/3084678470 conn(0x7f6ff0075e90 msgr2=0x7f6ff00782a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:59:33.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:33.411+0000 7f6ff6800700 1 -- 172.21.15.62:0/3084678470 shutdown_connections 2024-04-20T21:59:33.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:33.411+0000 7f6ff6800700 1 -- 172.21.15.62:0/3084678470 wait complete. 2024-04-20T21:59:33.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:33.412+0000 7f6ff6800700 1 Processor -- start 2024-04-20T21:59:33.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:33.412+0000 7f6ff6800700 1 -- start start 2024-04-20T21:59:33.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:33.412+0000 7f6ff6800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6ff0106b80 0x7f6ff019c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:59:33.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:33.412+0000 7f6ff6800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ff010c660 0x7f6ff019c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:59:33.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:33.412+0000 7f6ff6800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff019cf60 con 0x7f6ff0106b80 2024-04-20T21:59:33.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:33.412+0000 7f6ff6800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff01a0cf0 con 0x7f6ff010c660 2024-04-20T21:59:33.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:33.413+0000 7f6ff5400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6ff0106b80 0x7f6ff019c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:59:33.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:33.413+0000 7f6ff5400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6ff0106b80 0x7f6ff019c340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52286/0 (socket says 172.21.15.62:52286) 2024-04-20T21:59:33.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:33.413+0000 7f6ff5400700 1 -- 172.21.15.62:0/2416280464 learned_addr learned my addr 172.21.15.62:0/2416280464 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:59:33.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:33.413+0000 7f6ff4a00700 1 --2- 172.21.15.62:0/2416280464 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ff010c660 0x7f6ff019c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:59:33.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:33.413+0000 7f6ff5400700 1 -- 172.21.15.62:0/2416280464 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ff010c660 msgr2=0x7f6ff019c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:59:33.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:33.413+0000 7f6ff5400700 1 --2- 172.21.15.62:0/2416280464 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ff010c660 0x7f6ff019c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:59:33.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:33.413+0000 7f6ff5400700 1 -- 172.21.15.62:0/2416280464 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fdc0095f0 con 0x7f6ff0106b80 2024-04-20T21:59:33.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:33.414+0000 7f6ff5400700 1 --2- 172.21.15.62:0/2416280464 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6ff0106b80 0x7f6ff019c340 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f6fe400ba50 tx=0x7f6fe400be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:59:33.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:33.414+0000 7f6fee000700 1 -- 172.21.15.62:0/2416280464 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6fe400ca00 con 0x7f6ff0106b80 2024-04-20T21:59:33.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:33.414+0000 7f6fee000700 1 -- 172.21.15.62:0/2416280464 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6fe400cb60 con 0x7f6ff0106b80 2024-04-20T21:59:33.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:33.414+0000 7f6ff6800700 1 -- 172.21.15.62:0/2416280464 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ff01a0fd0 con 0x7f6ff0106b80 2024-04-20T21:59:33.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:33.414+0000 7f6fee000700 1 -- 172.21.15.62:0/2416280464 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6fe40075a0 con 0x7f6ff0106b80 2024-04-20T21:59:33.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:33.414+0000 7f6ff6800700 1 -- 172.21.15.62:0/2416280464 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ff01a1520 con 0x7f6ff0106b80 2024-04-20T21:59:33.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:33.416+0000 7f6ff6800700 1 -- 172.21.15.62:0/2416280464 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ff010eb60 con 0x7f6ff0106b80 2024-04-20T21:59:33.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:33.417+0000 7f6fee000700 1 -- 172.21.15.62:0/2416280464 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f6fe400ccd0 con 0x7f6ff0106b80 2024-04-20T21:59:33.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:33.418+0000 7f6fee000700 1 --2- 172.21.15.62:0/2416280464 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6fd806ecc0 0x7f6fd8071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:59:33.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:33.418+0000 7f6fee000700 1 -- 172.21.15.62:0/2416280464 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f6fe4006e40 con 0x7f6ff0106b80 2024-04-20T21:59:33.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:33.418+0000 7f6ff4a00700 1 --2- 172.21.15.62:0/2416280464 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6fd806ecc0 0x7f6fd8071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:59:33.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:33.419+0000 7f6ff4a00700 1 --2- 172.21.15.62:0/2416280464 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6fd806ecc0 0x7f6fd8071170 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f6ff019d960 tx=0x7f6fdc01a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:59:33.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:33.423+0000 7f6fee000700 1 -- 172.21.15.62:0/2416280464 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f6fe40573f0 con 0x7f6ff0106b80 2024-04-20T21:59:33.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:59:33 smithi189 ceph-mon[57022]: pgmap v728: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:59:33.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:59:33 smithi189 ceph-mon[57022]: from='client.35320 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:59:33.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:33.702+0000 7f6ff6800700 1 -- 172.21.15.62:0/2416280464 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f6ff01a17f0 con 0x7f6ff0106b80 2024-04-20T21:59:33.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:33.703+0000 7f6fee000700 1 -- 172.21.15.62:0/2416280464 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 27 v27) v1 ==== 76+0+1888 (secure 0 0 0) 0x7f6fe4019770 con 0x7f6ff0106b80 2024-04-20T21:59:33.703 INFO:teuthology.orchestra.run.smithi062.stdout:e27 2024-04-20T21:59:33.703 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T21:59:33.703 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T21:59:33.703 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T21:59:33.703 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:59:33.704 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T21:59:33.704 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T21:59:33.704 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 20 2024-04-20T21:59:33.704 INFO:teuthology.orchestra.run.smithi062.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-20T21:59:33.704 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T21:59:33.704 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T21:42:01.559061+0000 2024-04-20T21:59:33.704 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T21:59:33.704 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T21:59:33.704 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T21:59:33.704 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T21:59:33.704 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T21:59:33.704 INFO:teuthology.orchestra.run.smithi062.stdout:max_xattr_size 65536 2024-04-20T21:59:33.704 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T21:59:33.704 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T21:59:33.704 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 81 2024-04-20T21:59:33.704 INFO:teuthology.orchestra.run.smithi062.stdout:snap_rstat 1 2024-04-20T21:59:33.705 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T21:59:33.705 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 2 2024-04-20T21:59:33.705 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1 2024-04-20T21:59:33.705 INFO:teuthology.orchestra.run.smithi062.stdout:up {} 2024-04-20T21:59:33.705 INFO:teuthology.orchestra.run.smithi062.stdout:failed 0,1 2024-04-20T21:59:33.705 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T21:59:33.705 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T21:59:33.705 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T21:59:33.705 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T21:59:33.705 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data enabled 2024-04-20T21:59:33.705 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T21:59:33.705 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T21:59:33.705 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 1 2024-04-20T21:59:33.705 INFO:teuthology.orchestra.run.smithi062.stdout:qdb_cluster leader: 0 members: 2024-04-20T21:59:33.705 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:59:33.706 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:59:33.706 INFO:teuthology.orchestra.run.smithi062.stdout:Standby daemons: 2024-04-20T21:59:33.706 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T21:59:33.706 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ryqblb{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6826/4113916205,v1:172.21.15.62:6827/4113916205] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T21:59:33.706 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ozizwi{-1:34476} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6828/3801975165,v1:172.21.15.62:6829/3801975165] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T21:59:33.706 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.xbgspw{-1:44285} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6826/3996040526,v1:172.21.15.189:6827/3996040526] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T21:59:33.706 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.dvimer{-1:44727} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6824/3576699232,v1:172.21.15.189:6825/3576699232] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T21:59:33.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:33.706+0000 7f6ff6800700 1 -- 172.21.15.62:0/2416280464 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6fd806ecc0 msgr2=0x7f6fd8071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:59:33.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:33.706+0000 7f6ff6800700 1 --2- 172.21.15.62:0/2416280464 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6fd806ecc0 0x7f6fd8071170 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f6ff019d960 tx=0x7f6fdc01a040 comp rx=0 tx=0).stop 2024-04-20T21:59:33.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:33.706+0000 7f6ff6800700 1 -- 172.21.15.62:0/2416280464 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6ff0106b80 msgr2=0x7f6ff019c340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:59:33.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:33.706+0000 7f6ff6800700 1 --2- 172.21.15.62:0/2416280464 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6ff0106b80 0x7f6ff019c340 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f6fe400ba50 tx=0x7f6fe400be10 comp rx=0 tx=0).stop 2024-04-20T21:59:33.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:33.707+0000 7f6ff6800700 1 -- 172.21.15.62:0/2416280464 shutdown_connections 2024-04-20T21:59:33.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:33.707+0000 7f6ff6800700 1 --2- 172.21.15.62:0/2416280464 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6fd806ecc0 0x7f6fd8071170 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:59:33.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:33.707+0000 7f6ff6800700 1 --2- 172.21.15.62:0/2416280464 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6ff0106b80 0x7f6ff019c340 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:59:33.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:33.707+0000 7f6ff6800700 1 --2- 172.21.15.62:0/2416280464 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ff010c660 0x7f6ff019c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:59:33.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:33.707+0000 7f6ff6800700 1 -- 172.21.15.62:0/2416280464 >> 172.21.15.62:0/2416280464 conn(0x7f6ff0075e90 msgr2=0x7f6ff0077b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:59:33.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:33.707+0000 7f6ff6800700 1 -- 172.21.15.62:0/2416280464 shutdown_connections 2024-04-20T21:59:33.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:33.707+0000 7f6ff6800700 1 -- 172.21.15.62:0/2416280464 wait complete. 2024-04-20T21:59:33.708 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 27 2024-04-20T21:59:33.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:33.870+0000 7fb663a00700 1 -- 172.21.15.62:0/2243952016 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb65c1003b0 msgr2=0x7fb65c100780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:59:33.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:33.870+0000 7fb663a00700 1 --2- 172.21.15.62:0/2243952016 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb65c1003b0 0x7fb65c100780 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fb650009910 tx=0x7fb650009c20 comp rx=0 tx=0).stop 2024-04-20T21:59:33.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:33.870+0000 7fb663a00700 1 -- 172.21.15.62:0/2243952016 shutdown_connections 2024-04-20T21:59:33.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:33.870+0000 7fb663a00700 1 --2- 172.21.15.62:0/2243952016 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb65c100d50 0x7fb65c104d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:59:33.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:33.870+0000 7fb663a00700 1 --2- 172.21.15.62:0/2243952016 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb65c1003b0 0x7fb65c100780 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:59:33.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:33.870+0000 7fb663a00700 1 -- 172.21.15.62:0/2243952016 >> 172.21.15.62:0/2243952016 conn(0x7fb65c0fbc70 msgr2=0x7fb65c0fe080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:59:33.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:33.870+0000 7fb663a00700 1 -- 172.21.15.62:0/2243952016 shutdown_connections 2024-04-20T21:59:33.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:33.870+0000 7fb663a00700 1 -- 172.21.15.62:0/2243952016 wait complete. 2024-04-20T21:59:33.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:33.871+0000 7fb663a00700 1 Processor -- start 2024-04-20T21:59:33.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:33.872+0000 7fb663a00700 1 -- start start 2024-04-20T21:59:33.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:33.872+0000 7fb663a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb65c1003b0 0x7fb65c197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:59:33.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:33.872+0000 7fb663a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb65c100d50 0x7fb65c198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:59:33.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:33.872+0000 7fb663a00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb65c198b70 con 0x7fb65c100d50 2024-04-20T21:59:33.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:33.872+0000 7fb663a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb65c19c8b0 con 0x7fb65c1003b0 2024-04-20T21:59:33.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:33.872+0000 7fb65be00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb65c100d50 0x7fb65c198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:59:33.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:33.872+0000 7fb65be00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb65c100d50 0x7fb65c198490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52302/0 (socket says 172.21.15.62:52302) 2024-04-20T21:59:33.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:33.872+0000 7fb65be00700 1 -- 172.21.15.62:0/4275739646 learned_addr learned my addr 172.21.15.62:0/4275739646 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:59:33.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:33.873+0000 7fb65be00700 1 -- 172.21.15.62:0/4275739646 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb65c1003b0 msgr2=0x7fb65c197f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:59:33.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:33.873+0000 7fb661000700 1 --2- 172.21.15.62:0/4275739646 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb65c1003b0 0x7fb65c197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:59:33.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:33.873+0000 7fb65be00700 1 --2- 172.21.15.62:0/4275739646 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb65c1003b0 0x7fb65c197f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:59:33.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:33.873+0000 7fb65be00700 1 -- 172.21.15.62:0/4275739646 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6500095f0 con 0x7fb65c100d50 2024-04-20T21:59:33.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:33.873+0000 7fb65be00700 1 --2- 172.21.15.62:0/4275739646 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb65c100d50 0x7fb65c198490 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fb64800d8d0 tx=0x7fb64800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:59:33.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:33.873+0000 7fb659600700 1 -- 172.21.15.62:0/4275739646 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb648004a00 con 0x7fb65c100d50 2024-04-20T21:59:33.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:33.873+0000 7fb663a00700 1 -- 172.21.15.62:0/4275739646 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb65c19cb90 con 0x7fb65c100d50 2024-04-20T21:59:33.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:33.873+0000 7fb659600700 1 -- 172.21.15.62:0/4275739646 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb6480054e0 con 0x7fb65c100d50 2024-04-20T21:59:33.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:33.873+0000 7fb659600700 1 -- 172.21.15.62:0/4275739646 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb648009d40 con 0x7fb65c100d50 2024-04-20T21:59:33.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:33.874+0000 7fb663a00700 1 -- 172.21.15.62:0/4275739646 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb65c19d0e0 con 0x7fb65c100d50 2024-04-20T21:59:33.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:33.875+0000 7fb663a00700 1 -- 172.21.15.62:0/4275739646 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb65c1044a0 con 0x7fb65c100d50 2024-04-20T21:59:33.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:33.876+0000 7fb659600700 1 -- 172.21.15.62:0/4275739646 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fb648005020 con 0x7fb65c100d50 2024-04-20T21:59:33.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:33.877+0000 7fb659600700 1 --2- 172.21.15.62:0/4275739646 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb64406ecc0 0x7fb644071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:59:33.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:33.877+0000 7fb659600700 1 -- 172.21.15.62:0/4275739646 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fb64808e770 con 0x7fb65c100d50 2024-04-20T21:59:33.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:33.877+0000 7fb661000700 1 --2- 172.21.15.62:0/4275739646 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb64406ecc0 0x7fb644071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:59:33.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:33.878+0000 7fb661000700 1 --2- 172.21.15.62:0/4275739646 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb64406ecc0 0x7fb644071170 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fb650000c00 tx=0x7fb650005fd0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:59:33.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:33.884+0000 7fb659600700 1 -- 172.21.15.62:0/4275739646 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fb648057500 con 0x7fb65c100d50 2024-04-20T21:59:34.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:34.130+0000 7fb663a00700 1 -- 172.21.15.62:0/4275739646 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb65c19d460 con 0x7fb64406ecc0 2024-04-20T21:59:34.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:34.133+0000 7fb659600700 1 -- 172.21.15.62:0/4275739646 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fb65c19d460 con 0x7fb64406ecc0 2024-04-20T21:59:34.133 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T21:59:34.133 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T21:59:34.133 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T21:59:34.133 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-20T21:59:34.133 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T21:59:34.133 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr", 2024-04-20T21:59:34.133 INFO:teuthology.orchestra.run.smithi062.stdout: "crash", 2024-04-20T21:59:34.133 INFO:teuthology.orchestra.run.smithi062.stdout: "mon", 2024-04-20T21:59:34.133 INFO:teuthology.orchestra.run.smithi062.stdout: "osd" 2024-04-20T21:59:34.133 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T21:59:34.133 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "15/23 daemons upgraded", 2024-04-20T21:59:34.133 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading mds daemons", 2024-04-20T21:59:34.133 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T21:59:34.133 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T21:59:34.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:34.136+0000 7fb663a00700 1 -- 172.21.15.62:0/4275739646 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb64406ecc0 msgr2=0x7fb644071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:59:34.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:34.136+0000 7fb663a00700 1 --2- 172.21.15.62:0/4275739646 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb64406ecc0 0x7fb644071170 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fb650000c00 tx=0x7fb650005fd0 comp rx=0 tx=0).stop 2024-04-20T21:59:34.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:34.136+0000 7fb663a00700 1 -- 172.21.15.62:0/4275739646 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb65c100d50 msgr2=0x7fb65c198490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:59:34.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:34.136+0000 7fb663a00700 1 --2- 172.21.15.62:0/4275739646 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb65c100d50 0x7fb65c198490 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fb64800d8d0 tx=0x7fb64800dc90 comp rx=0 tx=0).stop 2024-04-20T21:59:34.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:34.136+0000 7fb663a00700 1 -- 172.21.15.62:0/4275739646 shutdown_connections 2024-04-20T21:59:34.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:34.136+0000 7fb663a00700 1 --2- 172.21.15.62:0/4275739646 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb64406ecc0 0x7fb644071170 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:59:34.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:34.136+0000 7fb663a00700 1 --2- 172.21.15.62:0/4275739646 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb65c1003b0 0x7fb65c197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:59:34.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:34.136+0000 7fb663a00700 1 --2- 172.21.15.62:0/4275739646 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb65c100d50 0x7fb65c198490 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:59:34.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:34.136+0000 7fb663a00700 1 -- 172.21.15.62:0/4275739646 >> 172.21.15.62:0/4275739646 conn(0x7fb65c0fbc70 msgr2=0x7fb65c0fd330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:59:34.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:34.136+0000 7fb663a00700 1 -- 172.21.15.62:0/4275739646 shutdown_connections 2024-04-20T21:59:34.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:34.136+0000 7fb663a00700 1 -- 172.21.15.62:0/4275739646 wait complete. 2024-04-20T21:59:34.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:34.296+0000 7f727b800700 1 -- 172.21.15.62:0/3244135933 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f727410c8f0 msgr2=0x7f727410ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:59:34.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:34.296+0000 7f727b800700 1 --2- 172.21.15.62:0/3244135933 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f727410c8f0 0x7f727410ccc0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f7268009910 tx=0x7f7268009c20 comp rx=0 tx=0).stop 2024-04-20T21:59:34.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:34.297+0000 7f727b800700 1 -- 172.21.15.62:0/3244135933 shutdown_connections 2024-04-20T21:59:34.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:34.297+0000 7f727b800700 1 --2- 172.21.15.62:0/3244135933 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7274106b80 0x7f7274106ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:59:34.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:34.297+0000 7f727b800700 1 --2- 172.21.15.62:0/3244135933 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f727410c8f0 0x7f727410ccc0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:59:34.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:34.297+0000 7f727b800700 1 -- 172.21.15.62:0/3244135933 >> 172.21.15.62:0/3244135933 conn(0x7f7274076060 msgr2=0x7f7274078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:59:34.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:34.297+0000 7f727b800700 1 -- 172.21.15.62:0/3244135933 shutdown_connections 2024-04-20T21:59:34.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:34.297+0000 7f727b800700 1 -- 172.21.15.62:0/3244135933 wait complete. 2024-04-20T21:59:34.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:34.298+0000 7f727b800700 1 Processor -- start 2024-04-20T21:59:34.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:34.298+0000 7f727b800700 1 -- start start 2024-04-20T21:59:34.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:34.298+0000 7f727b800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7274106b80 0x7f727419c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:59:34.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:34.298+0000 7f727b800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f727410c8f0 0x7f727419ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:59:34.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:34.298+0000 7f727b800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f727419d120 con 0x7f7274106b80 2024-04-20T21:59:34.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:34.298+0000 7f727b800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72741a0eb0 con 0x7f727410c8f0 2024-04-20T21:59:34.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:34.299+0000 7f7278e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7274106b80 0x7f727419c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:59:34.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:34.299+0000 7f7278e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7274106b80 0x7f727419c500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52322/0 (socket says 172.21.15.62:52322) 2024-04-20T21:59:34.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:34.299+0000 7f7278e00700 1 -- 172.21.15.62:0/3456831223 learned_addr learned my addr 172.21.15.62:0/3456831223 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T21:59:34.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:34.299+0000 7f7273e00700 1 --2- 172.21.15.62:0/3456831223 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f727410c8f0 0x7f727419ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:59:34.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:34.299+0000 7f7278e00700 1 -- 172.21.15.62:0/3456831223 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f727410c8f0 msgr2=0x7f727419ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:59:34.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:34.299+0000 7f7278e00700 1 --2- 172.21.15.62:0/3456831223 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f727410c8f0 0x7f727419ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:59:34.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:34.299+0000 7f7278e00700 1 -- 172.21.15.62:0/3456831223 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72680095f0 con 0x7f7274106b80 2024-04-20T21:59:34.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:34.299+0000 7f7273e00700 1 --2- 172.21.15.62:0/3456831223 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f727410c8f0 0x7f727419ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T21:59:34.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:34.300+0000 7f7278e00700 1 --2- 172.21.15.62:0/3456831223 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7274106b80 0x7f727419c500 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f72680092e0 tx=0x7f726800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:59:34.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:34.300+0000 7f7271600700 1 -- 172.21.15.62:0/3456831223 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f726801d070 con 0x7f7274106b80 2024-04-20T21:59:34.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:34.300+0000 7f7271600700 1 -- 172.21.15.62:0/3456831223 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7268005e00 con 0x7f7274106b80 2024-04-20T21:59:34.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:34.300+0000 7f7271600700 1 -- 172.21.15.62:0/3456831223 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7268017400 con 0x7f7274106b80 2024-04-20T21:59:34.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:34.300+0000 7f727b800700 1 -- 172.21.15.62:0/3456831223 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72741a1130 con 0x7f7274106b80 2024-04-20T21:59:34.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:34.300+0000 7f727b800700 1 -- 172.21.15.62:0/3456831223 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72741a1620 con 0x7f7274106b80 2024-04-20T21:59:34.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:34.301+0000 7f727b800700 1 -- 172.21.15.62:0/3456831223 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7274109b20 con 0x7f7274106b80 2024-04-20T21:59:34.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:34.303+0000 7f7271600700 1 -- 172.21.15.62:0/3456831223 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f726800f460 con 0x7f7274106b80 2024-04-20T21:59:34.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:34.303+0000 7f7271600700 1 --2- 172.21.15.62:0/3456831223 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f725c06ecc0 0x7f725c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T21:59:34.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:34.304+0000 7f7271600700 1 -- 172.21.15.62:0/3456831223 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f7268090330 con 0x7f7274106b80 2024-04-20T21:59:34.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:34.304+0000 7f7273e00700 1 --2- 172.21.15.62:0/3456831223 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f725c06ecc0 0x7f725c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T21:59:34.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:34.304+0000 7f7273e00700 1 --2- 172.21.15.62:0/3456831223 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f725c06ecc0 0x7f725c071170 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f7260005fd0 tx=0x7f7260005ee0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T21:59:34.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:34.309+0000 7f7271600700 1 -- 172.21.15.62:0/3456831223 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f72680590c0 con 0x7f7274106b80 2024-04-20T21:59:34.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:59:34 smithi189 ceph-mon[57022]: from='client.35324 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:59:34.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:59:34 smithi189 ceph-mon[57022]: from='client.35328 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:59:34.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:59:34 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/1992190769' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:59:34.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:59:34 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/2416280464' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T21:59:34.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:59:34 smithi062 ceph-mon[76322]: from='client.35324 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:59:34.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:59:34 smithi062 ceph-mon[76322]: from='client.35328 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:59:34.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:59:34 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/1992190769' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T21:59:34.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:59:34 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/2416280464' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T21:59:34.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:34.653+0000 7f727b800700 1 -- 172.21.15.62:0/3456831223 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f727404f2a0 con 0x7f7274106b80 2024-04-20T21:59:34.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:34.654+0000 7f7271600700 1 -- 172.21.15.62:0/3456831223 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7f726805c780 con 0x7f7274106b80 2024-04-20T21:59:34.654 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-20T21:59:34.654 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-20T21:59:34.655 INFO:teuthology.orchestra.run.smithi062.stdout: host smithi062 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T21:59:34.655 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-20T21:59:34.655 INFO:teuthology.orchestra.run.smithi062.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi062 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpq7y7oa_r:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T21:59:34.655 INFO:teuthology.orchestra.run.smithi062.stdout:Traceback (most recent call last): 2024-04-20T21:59:34.655 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-20T21:59:34.655 INFO:teuthology.orchestra.run.smithi062.stdout: return _run_code(code, main_globals, None, 2024-04-20T21:59:34.655 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-20T21:59:34.655 INFO:teuthology.orchestra.run.smithi062.stdout: exec(code, run_globals) 2024-04-20T21:59:34.655 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5618, in 2024-04-20T21:59:34.655 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5606, in main 2024-04-20T21:59:34.656 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 408, in _infer_config 2024-04-20T21:59:34.656 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 323, in _infer_fsid 2024-04-20T21:59:34.656 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 436, in _infer_image 2024-04-20T21:59:34.656 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 310, in _validate_fsid 2024-04-20T21:59:34.656 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 3359, in command_ceph_volume 2024-04-20T21:59:34.656 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-20T21:59:34.656 INFO:teuthology.orchestra.run.smithi062.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi062 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpq7y7oa_r:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T21:59:34.656 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-20T21:59:34.656 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is degraded 2024-04-20T21:59:34.656 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T21:59:34.656 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T21:59:34.657 INFO:teuthology.orchestra.run.smithi062.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-20T21:59:34.657 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-20T21:59:34.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:34.657+0000 7f727b800700 1 -- 172.21.15.62:0/3456831223 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f725c06ecc0 msgr2=0x7f725c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:59:34.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:34.657+0000 7f727b800700 1 --2- 172.21.15.62:0/3456831223 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f725c06ecc0 0x7f725c071170 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f7260005fd0 tx=0x7f7260005ee0 comp rx=0 tx=0).stop 2024-04-20T21:59:34.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:34.657+0000 7f727b800700 1 -- 172.21.15.62:0/3456831223 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7274106b80 msgr2=0x7f727419c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T21:59:34.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:34.657+0000 7f727b800700 1 --2- 172.21.15.62:0/3456831223 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7274106b80 0x7f727419c500 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f72680092e0 tx=0x7f726800be20 comp rx=0 tx=0).stop 2024-04-20T21:59:34.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:34.657+0000 7f727b800700 1 -- 172.21.15.62:0/3456831223 shutdown_connections 2024-04-20T21:59:34.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:34.657+0000 7f727b800700 1 --2- 172.21.15.62:0/3456831223 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f725c06ecc0 0x7f725c071170 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:59:34.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:34.657+0000 7f727b800700 1 --2- 172.21.15.62:0/3456831223 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7274106b80 0x7f727419c500 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:59:34.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:34.657+0000 7f727b800700 1 --2- 172.21.15.62:0/3456831223 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f727410c8f0 0x7f727419ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T21:59:34.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:34.657+0000 7f727b800700 1 -- 172.21.15.62:0/3456831223 >> 172.21.15.62:0/3456831223 conn(0x7f7274076060 msgr2=0x7f7274077aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T21:59:34.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:34.658+0000 7f727b800700 1 -- 172.21.15.62:0/3456831223 shutdown_connections 2024-04-20T21:59:34.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T21:59:34.658+0000 7f727b800700 1 -- 172.21.15.62:0/3456831223 wait complete. 2024-04-20T21:59:35.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:59:35 smithi189 ceph-mon[57022]: pgmap v729: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:59:35.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:59:35 smithi189 ceph-mon[57022]: from='client.35340 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:59:35.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:59:35 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:59:35.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:59:35 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/3456831223' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T21:59:35.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:59:35 smithi062 ceph-mon[76322]: pgmap v729: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:59:35.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:59:35 smithi062 ceph-mon[76322]: from='client.35340 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T21:59:35.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:59:35 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:59:35.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:59:35 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/3456831223' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T21:59:37.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:59:37 smithi189 ceph-mon[57022]: pgmap v730: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:59:37.538 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:59:37 smithi062 ceph-mon[76322]: pgmap v730: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:59:38.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:59:38 smithi189 ceph-mon[57022]: pgmap v731: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:59:38.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:59:38 smithi062 ceph-mon[76322]: pgmap v731: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:59:40.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:59:40 smithi189 ceph-mon[57022]: pgmap v732: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:59:40.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:59:40 smithi062 ceph-mon[76322]: pgmap v732: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:59:42.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:59:42 smithi189 ceph-mon[57022]: pgmap v733: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:59:42.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:59:42 smithi062 ceph-mon[76322]: pgmap v733: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:59:44.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:59:44 smithi189 ceph-mon[57022]: pgmap v734: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:59:44.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:59:44 smithi062 ceph-mon[76322]: pgmap v734: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:59:46.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:59:46 smithi189 ceph-mon[57022]: pgmap v735: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:59:46.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:59:46 smithi062 ceph-mon[76322]: pgmap v735: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:59:48.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:59:48 smithi189 ceph-mon[57022]: pgmap v736: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:59:48.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:59:48 smithi062 ceph-mon[76322]: pgmap v736: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:59:49.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:59:49 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:59:49.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:59:49 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T21:59:50.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:59:50 smithi189 ceph-mon[57022]: pgmap v737: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:59:50.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:59:50 smithi062 ceph-mon[76322]: pgmap v737: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:59:52.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:59:52 smithi189 ceph-mon[57022]: pgmap v738: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:59:52.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:59:52 smithi062 ceph-mon[76322]: pgmap v738: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:59:54.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:59:54 smithi062 ceph-mon[76322]: pgmap v739: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:59:54.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:59:54 smithi189 ceph-mon[57022]: pgmap v739: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:59:56.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:59:56 smithi062 ceph-mon[76322]: pgmap v740: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:59:56.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:59:56 smithi189 ceph-mon[57022]: pgmap v740: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:59:58.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 21:59:58 smithi062 ceph-mon[76322]: pgmap v741: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T21:59:58.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 21:59:58 smithi189 ceph-mon[57022]: pgmap v741: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:00:00.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:00:00 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-mon-smithi062[76318]: 2024-04-20T21:59:59.999+0000 7f6157e00700 -1 log_channel(cluster) log [ERR] : Health detail: HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-20T22:00:00.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:00:00 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-mon-smithi062[76318]: 2024-04-20T21:59:59.999+0000 7f6157e00700 -1 log_channel(cluster) log [ERR] : [WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-20T22:00:00.287 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:00:00 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-mon-smithi062[76318]: 2024-04-20T21:59:59.999+0000 7f6157e00700 -1 log_channel(cluster) log [ERR] : host smithi062 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T22:00:00.287 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:00:00 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-mon-smithi062[76318]: 2024-04-20T21:59:59.999+0000 7f6157e00700 -1 log_channel(cluster) log [ERR] : /usr/bin/podman: timeout after 895 seconds 2024-04-20T22:00:00.287 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:00:00 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-mon-smithi062[76318]: 2024-04-20T21:59:59.999+0000 7f6157e00700 -1 log_channel(cluster) log [ERR] : Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi062 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpq7y7oa_r:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:00:00.287 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:00:00 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-mon-smithi062[76318]: 2024-04-20T21:59:59.999+0000 7f6157e00700 -1 log_channel(cluster) log [ERR] : Traceback (most recent call last): 2024-04-20T22:00:00.287 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:00:00 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-mon-smithi062[76318]: 2024-04-20T21:59:59.999+0000 7f6157e00700 -1 log_channel(cluster) log [ERR] : File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-20T22:00:00.287 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:00:00 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-mon-smithi062[76318]: 2024-04-20T21:59:59.999+0000 7f6157e00700 -1 log_channel(cluster) log [ERR] : return _run_code(code, main_globals, None, 2024-04-20T22:00:00.287 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:00:00 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-mon-smithi062[76318]: 2024-04-20T21:59:59.999+0000 7f6157e00700 -1 log_channel(cluster) log [ERR] : File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-20T22:00:00.287 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:00:00 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-mon-smithi062[76318]: 2024-04-20T21:59:59.999+0000 7f6157e00700 -1 log_channel(cluster) log [ERR] : exec(code, run_globals) 2024-04-20T22:00:00.287 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:00:00 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-mon-smithi062[76318]: 2024-04-20T21:59:59.999+0000 7f6157e00700 -1 log_channel(cluster) log [ERR] : File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5618, in 2024-04-20T22:00:00.288 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:00:00 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-mon-smithi062[76318]: 2024-04-20T21:59:59.999+0000 7f6157e00700 -1 log_channel(cluster) log [ERR] : File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5606, in main 2024-04-20T22:00:00.288 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:00:00 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-mon-smithi062[76318]: 2024-04-20T21:59:59.999+0000 7f6157e00700 -1 log_channel(cluster) log [ERR] : File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 408, in _infer_config 2024-04-20T22:00:00.288 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:00:00 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-mon-smithi062[76318]: 2024-04-20T21:59:59.999+0000 7f6157e00700 -1 log_channel(cluster) log [ERR] : File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 323, in _infer_fsid 2024-04-20T22:00:00.288 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:00:00 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-mon-smithi062[76318]: 2024-04-20T21:59:59.999+0000 7f6157e00700 -1 log_channel(cluster) log [ERR] : File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 436, in _infer_image 2024-04-20T22:00:00.288 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:00:00 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-mon-smithi062[76318]: 2024-04-20T21:59:59.999+0000 7f6157e00700 -1 log_channel(cluster) log [ERR] : File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 310, in _validate_fsid 2024-04-20T22:00:00.288 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:00:00 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-mon-smithi062[76318]: 2024-04-20T21:59:59.999+0000 7f6157e00700 -1 log_channel(cluster) log [ERR] : File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 3359, in command_ceph_volume 2024-04-20T22:00:00.288 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:00:00 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-mon-smithi062[76318]: 2024-04-20T21:59:59.999+0000 7f6157e00700 -1 log_channel(cluster) log [ERR] : File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-20T22:00:00.288 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:00:00 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-mon-smithi062[76318]: 2024-04-20T21:59:59.999+0000 7f6157e00700 -1 log_channel(cluster) log [ERR] : RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi062 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpq7y7oa_r:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:00:00.288 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:00:00 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-mon-smithi062[76318]: 2024-04-20T21:59:59.999+0000 7f6157e00700 -1 log_channel(cluster) log [ERR] : [WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-20T22:00:00.288 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:00:00 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-mon-smithi062[76318]: 2024-04-20T21:59:59.999+0000 7f6157e00700 -1 log_channel(cluster) log [ERR] : fs cephfs is degraded 2024-04-20T22:00:00.288 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:00:00 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-mon-smithi062[76318]: 2024-04-20T21:59:59.999+0000 7f6157e00700 -1 log_channel(cluster) log [ERR] : [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T22:00:00.288 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:00:00 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-mon-smithi062[76318]: 2024-04-20T21:59:59.999+0000 7f6157e00700 -1 log_channel(cluster) log [ERR] : fs cephfs has deprecated feature inline_data enabled. 2024-04-20T22:00:00.289 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:00:00 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-mon-smithi062[76318]: 2024-04-20T21:59:59.999+0000 7f6157e00700 -1 log_channel(cluster) log [ERR] : [ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-20T22:00:00.289 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:00:00 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-mon-smithi062[76318]: 2024-04-20T22:00:00.000+0000 7f6157e00700 -1 log_channel(cluster) log [ERR] : fs cephfs is offline because no MDS is active for it. 2024-04-20T22:00:00.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:00:00 smithi062 ceph-mon[76322]: pgmap v742: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:00:00.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:00:00 smithi062 ceph-mon[76322]: Health detail: HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-20T22:00:00.787 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:00:00 smithi062 ceph-mon[76322]: [WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-20T22:00:00.787 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:00:00 smithi062 ceph-mon[76322]: host smithi062 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T22:00:00.787 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:00:00 smithi062 ceph-mon[76322]: /usr/bin/podman: timeout after 895 seconds 2024-04-20T22:00:00.787 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:00:00 smithi062 ceph-mon[76322]: Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi062 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpq7y7oa_r:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:00:00.787 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:00:00 smithi062 ceph-mon[76322]: Traceback (most recent call last): 2024-04-20T22:00:00.787 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:00:00 smithi062 ceph-mon[76322]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-20T22:00:00.787 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:00:00 smithi062 ceph-mon[76322]: return _run_code(code, main_globals, None, 2024-04-20T22:00:00.787 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:00:00 smithi062 ceph-mon[76322]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-20T22:00:00.787 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:00:00 smithi062 ceph-mon[76322]: exec(code, run_globals) 2024-04-20T22:00:00.787 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:00:00 smithi062 ceph-mon[76322]: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5618, in 2024-04-20T22:00:00.787 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:00:00 smithi062 ceph-mon[76322]: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5606, in main 2024-04-20T22:00:00.788 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:00:00 smithi062 ceph-mon[76322]: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 408, in _infer_config 2024-04-20T22:00:00.788 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:00:00 smithi062 ceph-mon[76322]: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 323, in _infer_fsid 2024-04-20T22:00:00.788 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:00:00 smithi062 ceph-mon[76322]: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 436, in _infer_image 2024-04-20T22:00:00.788 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:00:00 smithi062 ceph-mon[76322]: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 310, in _validate_fsid 2024-04-20T22:00:00.788 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:00:00 smithi062 ceph-mon[76322]: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 3359, in command_ceph_volume 2024-04-20T22:00:00.788 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:00:00 smithi062 ceph-mon[76322]: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-20T22:00:00.788 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:00:00 smithi062 ceph-mon[76322]: RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi062 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpq7y7oa_r:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:00:00.788 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:00:00 smithi062 ceph-mon[76322]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-20T22:00:00.788 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:00:00 smithi062 ceph-mon[76322]: fs cephfs is degraded 2024-04-20T22:00:00.788 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:00:00 smithi062 ceph-mon[76322]: [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T22:00:00.788 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:00:00 smithi062 ceph-mon[76322]: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T22:00:00.788 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:00:00 smithi062 ceph-mon[76322]: [ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-20T22:00:00.788 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:00:00 smithi062 ceph-mon[76322]: fs cephfs is offline because no MDS is active for it. 2024-04-20T22:00:00.989 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:00:00 smithi189 ceph-mon[57022]: pgmap v742: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:00:00.989 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:00:00 smithi189 ceph-mon[57022]: Health detail: HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-20T22:00:00.989 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:00:00 smithi189 ceph-mon[57022]: [WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-20T22:00:00.989 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:00:00 smithi189 ceph-mon[57022]: host smithi062 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T22:00:00.989 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:00:00 smithi189 ceph-mon[57022]: /usr/bin/podman: timeout after 895 seconds 2024-04-20T22:00:00.989 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:00:00 smithi189 ceph-mon[57022]: Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi062 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpq7y7oa_r:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:00:00.990 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:00:00 smithi189 ceph-mon[57022]: Traceback (most recent call last): 2024-04-20T22:00:00.990 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:00:00 smithi189 ceph-mon[57022]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-20T22:00:00.990 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:00:00 smithi189 ceph-mon[57022]: return _run_code(code, main_globals, None, 2024-04-20T22:00:00.990 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:00:00 smithi189 ceph-mon[57022]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-20T22:00:00.990 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:00:00 smithi189 ceph-mon[57022]: exec(code, run_globals) 2024-04-20T22:00:00.990 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:00:00 smithi189 ceph-mon[57022]: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5618, in 2024-04-20T22:00:00.990 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:00:00 smithi189 ceph-mon[57022]: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5606, in main 2024-04-20T22:00:00.990 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:00:00 smithi189 ceph-mon[57022]: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 408, in _infer_config 2024-04-20T22:00:00.990 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:00:00 smithi189 ceph-mon[57022]: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 323, in _infer_fsid 2024-04-20T22:00:00.990 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:00:00 smithi189 ceph-mon[57022]: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 436, in _infer_image 2024-04-20T22:00:00.990 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:00:00 smithi189 ceph-mon[57022]: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 310, in _validate_fsid 2024-04-20T22:00:00.990 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:00:00 smithi189 ceph-mon[57022]: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 3359, in command_ceph_volume 2024-04-20T22:00:00.990 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:00:00 smithi189 ceph-mon[57022]: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-20T22:00:00.991 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:00:00 smithi189 ceph-mon[57022]: RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi062 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpq7y7oa_r:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:00:00.991 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:00:00 smithi189 ceph-mon[57022]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-20T22:00:00.991 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:00:00 smithi189 ceph-mon[57022]: fs cephfs is degraded 2024-04-20T22:00:00.991 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:00:00 smithi189 ceph-mon[57022]: [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T22:00:00.991 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:00:00 smithi189 ceph-mon[57022]: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T22:00:00.991 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:00:00 smithi189 ceph-mon[57022]: [ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-20T22:00:00.991 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:00:00 smithi189 ceph-mon[57022]: fs cephfs is offline because no MDS is active for it. 2024-04-20T22:00:02.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:00:02 smithi062 ceph-mon[76322]: pgmap v743: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:00:02.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:00:02 smithi189 ceph-mon[57022]: pgmap v743: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:00:04.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:00:04 smithi062 ceph-mon[76322]: pgmap v744: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:00:04.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:00:04 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:00:04.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:04.834+0000 7fd23d400700 1 -- 172.21.15.62:0/4289759921 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2381024d0 msgr2=0x7fd2381028a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:00:04.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:04.834+0000 7fd23d400700 1 --2- 172.21.15.62:0/4289759921 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2381024d0 0x7fd2381028a0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fd228009910 tx=0x7fd228009c20 comp rx=0 tx=0).stop 2024-04-20T22:00:04.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:04.835+0000 7fd23d400700 1 -- 172.21.15.62:0/4289759921 shutdown_connections 2024-04-20T22:00:04.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:04.835+0000 7fd23d400700 1 --2- 172.21.15.62:0/4289759921 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd238102e70 0x7fd238106e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:00:04.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:04.835+0000 7fd23d400700 1 --2- 172.21.15.62:0/4289759921 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2381024d0 0x7fd2381028a0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:00:04.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:04.835+0000 7fd23d400700 1 -- 172.21.15.62:0/4289759921 >> 172.21.15.62:0/4289759921 conn(0x7fd2380fdd90 msgr2=0x7fd2381001a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:00:04.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:04.835+0000 7fd23d400700 1 -- 172.21.15.62:0/4289759921 shutdown_connections 2024-04-20T22:00:04.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:04.835+0000 7fd23d400700 1 -- 172.21.15.62:0/4289759921 wait complete. 2024-04-20T22:00:04.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:04.836+0000 7fd23d400700 1 Processor -- start 2024-04-20T22:00:04.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:04.836+0000 7fd23d400700 1 -- start start 2024-04-20T22:00:04.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:04.836+0000 7fd23d400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2381024d0 0x7fd238197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:00:04.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:04.837+0000 7fd23d400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd238102e70 0x7fd238198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:00:04.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:04.837+0000 7fd23d400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd238198b50 con 0x7fd238102e70 2024-04-20T22:00:04.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:04.837+0000 7fd23d400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd23819c8e0 con 0x7fd2381024d0 2024-04-20T22:00:04.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:04.837+0000 7fd22f400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd238102e70 0x7fd238198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:00:04.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:04.837+0000 7fd22f400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd238102e70 0x7fd238198470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55554/0 (socket says 172.21.15.62:55554) 2024-04-20T22:00:04.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:04.837+0000 7fd22f400700 1 -- 172.21.15.62:0/433153265 learned_addr learned my addr 172.21.15.62:0/433153265 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:00:04.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:04.837+0000 7fd237e00700 1 --2- 172.21.15.62:0/433153265 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2381024d0 0x7fd238197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:00:04.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:04.837+0000 7fd22f400700 1 -- 172.21.15.62:0/433153265 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2381024d0 msgr2=0x7fd238197f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:00:04.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:04.837+0000 7fd22f400700 1 --2- 172.21.15.62:0/433153265 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2381024d0 0x7fd238197f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:00:04.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:04.837+0000 7fd22f400700 1 -- 172.21.15.62:0/433153265 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2280095f0 con 0x7fd238102e70 2024-04-20T22:00:04.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:04.837+0000 7fd237e00700 1 --2- 172.21.15.62:0/433153265 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2381024d0 0x7fd238197f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-20T22:00:04.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:04.838+0000 7fd22f400700 1 --2- 172.21.15.62:0/433153265 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd238102e70 0x7fd238198470 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fd22000b480 tx=0x7fd22000b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:00:04.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:04.838+0000 7fd235600700 1 -- 172.21.15.62:0/433153265 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd2200118c0 con 0x7fd238102e70 2024-04-20T22:00:04.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:04.838+0000 7fd235600700 1 -- 172.21.15.62:0/433153265 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd220010380 con 0x7fd238102e70 2024-04-20T22:00:04.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:04.838+0000 7fd23d400700 1 -- 172.21.15.62:0/433153265 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd23819cbc0 con 0x7fd238102e70 2024-04-20T22:00:04.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:04.838+0000 7fd235600700 1 -- 172.21.15.62:0/433153265 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd22000f4f0 con 0x7fd238102e70 2024-04-20T22:00:04.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:04.838+0000 7fd23d400700 1 -- 172.21.15.62:0/433153265 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd23819d110 con 0x7fd238102e70 2024-04-20T22:00:04.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:04.840+0000 7fd23d400700 1 -- 172.21.15.62:0/433153265 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd23810a7e0 con 0x7fd238102e70 2024-04-20T22:00:04.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:04.841+0000 7fd235600700 1 -- 172.21.15.62:0/433153265 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fd2200104f0 con 0x7fd238102e70 2024-04-20T22:00:04.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:04.842+0000 7fd235600700 1 --2- 172.21.15.62:0/433153265 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd21c06ecc0 0x7fd21c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:00:04.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:04.842+0000 7fd235600700 1 -- 172.21.15.62:0/433153265 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fd22008ea90 con 0x7fd238102e70 2024-04-20T22:00:04.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:04.842+0000 7fd237e00700 1 --2- 172.21.15.62:0/433153265 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd21c06ecc0 0x7fd21c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:00:04.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:04.843+0000 7fd237e00700 1 --2- 172.21.15.62:0/433153265 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd21c06ecc0 0x7fd21c071170 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fd22800b5c0 tx=0x7fd22801a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:00:04.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:04.847+0000 7fd235600700 1 -- 172.21.15.62:0/433153265 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fd220057840 con 0x7fd238102e70 2024-04-20T22:00:04.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:00:04 smithi189 ceph-mon[57022]: pgmap v744: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:00:04.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:00:04 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:00:05.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:05.107+0000 7fd23d400700 1 -- 172.21.15.62:0/433153265 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd23819d400 con 0x7fd21c06ecc0 2024-04-20T22:00:05.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:05.110+0000 7fd235600700 1 -- 172.21.15.62:0/433153265 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fd23819d400 con 0x7fd21c06ecc0 2024-04-20T22:00:05.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:05.113+0000 7fd23d400700 1 -- 172.21.15.62:0/433153265 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd21c06ecc0 msgr2=0x7fd21c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:00:05.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:05.113+0000 7fd23d400700 1 --2- 172.21.15.62:0/433153265 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd21c06ecc0 0x7fd21c071170 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fd22800b5c0 tx=0x7fd22801a040 comp rx=0 tx=0).stop 2024-04-20T22:00:05.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:05.113+0000 7fd23d400700 1 -- 172.21.15.62:0/433153265 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd238102e70 msgr2=0x7fd238198470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:00:05.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:05.113+0000 7fd23d400700 1 --2- 172.21.15.62:0/433153265 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd238102e70 0x7fd238198470 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fd22000b480 tx=0x7fd22000b840 comp rx=0 tx=0).stop 2024-04-20T22:00:05.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:05.113+0000 7fd23d400700 1 -- 172.21.15.62:0/433153265 shutdown_connections 2024-04-20T22:00:05.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:05.113+0000 7fd23d400700 1 --2- 172.21.15.62:0/433153265 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd21c06ecc0 0x7fd21c071170 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:00:05.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:05.113+0000 7fd23d400700 1 --2- 172.21.15.62:0/433153265 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2381024d0 0x7fd238197f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:00:05.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:05.113+0000 7fd23d400700 1 --2- 172.21.15.62:0/433153265 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd238102e70 0x7fd238198470 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:00:05.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:05.113+0000 7fd23d400700 1 -- 172.21.15.62:0/433153265 >> 172.21.15.62:0/433153265 conn(0x7fd2380fdd90 msgr2=0x7fd2380ff580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:00:05.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:05.113+0000 7fd23d400700 1 -- 172.21.15.62:0/433153265 shutdown_connections 2024-04-20T22:00:05.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:05.113+0000 7fd23d400700 1 -- 172.21.15.62:0/433153265 wait complete. 2024-04-20T22:00:05.129 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T22:00:05.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:05.281+0000 7f36b9800700 1 -- 172.21.15.62:0/4058236718 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f36b410c660 msgr2=0x7f36b410ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:00:05.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:05.281+0000 7f36b9800700 1 --2- 172.21.15.62:0/4058236718 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f36b410c660 0x7f36b410ca30 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f36a4009910 tx=0x7f36a4009c20 comp rx=0 tx=0).stop 2024-04-20T22:00:05.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:05.282+0000 7f36b9800700 1 -- 172.21.15.62:0/4058236718 shutdown_connections 2024-04-20T22:00:05.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:05.282+0000 7f36b9800700 1 --2- 172.21.15.62:0/4058236718 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f36b4106b80 0x7f36b4106ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:00:05.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:05.282+0000 7f36b9800700 1 --2- 172.21.15.62:0/4058236718 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f36b410c660 0x7f36b410ca30 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:00:05.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:05.282+0000 7f36b9800700 1 -- 172.21.15.62:0/4058236718 >> 172.21.15.62:0/4058236718 conn(0x7f36b4075e90 msgr2=0x7f36b40782a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:00:05.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:05.282+0000 7f36b9800700 1 -- 172.21.15.62:0/4058236718 shutdown_connections 2024-04-20T22:00:05.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:05.282+0000 7f36b9800700 1 -- 172.21.15.62:0/4058236718 wait complete. 2024-04-20T22:00:05.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:05.283+0000 7f36b9800700 1 Processor -- start 2024-04-20T22:00:05.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:05.283+0000 7f36b9800700 1 -- start start 2024-04-20T22:00:05.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:05.283+0000 7f36b9800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f36b4106b80 0x7f36b419c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:00:05.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:05.283+0000 7f36b9800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f36b410c660 0x7f36b419c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:00:05.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:05.283+0000 7f36b9800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36b419cf70 con 0x7f36b4106b80 2024-04-20T22:00:05.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:05.283+0000 7f36b9800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36b41a0d00 con 0x7f36b410c660 2024-04-20T22:00:05.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:05.284+0000 7f36b3e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f36b4106b80 0x7f36b419c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:00:05.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:05.284+0000 7f36b3e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f36b4106b80 0x7f36b419c350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55576/0 (socket says 172.21.15.62:55576) 2024-04-20T22:00:05.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:05.284+0000 7f36b3e00700 1 -- 172.21.15.62:0/3760996638 learned_addr learned my addr 172.21.15.62:0/3760996638 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:00:05.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:05.284+0000 7f36b3400700 1 --2- 172.21.15.62:0/3760996638 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f36b410c660 0x7f36b419c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:00:05.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:05.284+0000 7f36b3e00700 1 -- 172.21.15.62:0/3760996638 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f36b410c660 msgr2=0x7f36b419c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:00:05.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:05.284+0000 7f36b3e00700 1 --2- 172.21.15.62:0/3760996638 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f36b410c660 0x7f36b419c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:00:05.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:05.284+0000 7f36b3e00700 1 -- 172.21.15.62:0/3760996638 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36a40095f0 con 0x7f36b4106b80 2024-04-20T22:00:05.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:05.285+0000 7f36b3e00700 1 --2- 172.21.15.62:0/3760996638 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f36b4106b80 0x7f36b419c350 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f36a4006010 tx=0x7f36a400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:00:05.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:05.285+0000 7f36b0c00700 1 -- 172.21.15.62:0/3760996638 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f36a401d070 con 0x7f36b4106b80 2024-04-20T22:00:05.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:05.285+0000 7f36b0c00700 1 -- 172.21.15.62:0/3760996638 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f36a4004fb0 con 0x7f36b4106b80 2024-04-20T22:00:05.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:05.285+0000 7f36b0c00700 1 -- 172.21.15.62:0/3760996638 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f36a400f690 con 0x7f36b4106b80 2024-04-20T22:00:05.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:05.285+0000 7f36b9800700 1 -- 172.21.15.62:0/3760996638 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36b41a0f80 con 0x7f36b4106b80 2024-04-20T22:00:05.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:05.285+0000 7f36b9800700 1 -- 172.21.15.62:0/3760996638 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36b41a1470 con 0x7f36b4106b80 2024-04-20T22:00:05.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:05.286+0000 7f36b9800700 1 -- 172.21.15.62:0/3760996638 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36b410eb60 con 0x7f36b4106b80 2024-04-20T22:00:05.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:05.287+0000 7f36b0c00700 1 -- 172.21.15.62:0/3760996638 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f36a40228a0 con 0x7f36b4106b80 2024-04-20T22:00:05.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:05.288+0000 7f36b0c00700 1 --2- 172.21.15.62:0/3760996638 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f369806ecc0 0x7f3698071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:00:05.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:05.288+0000 7f36b0c00700 1 -- 172.21.15.62:0/3760996638 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f36a4090700 con 0x7f36b4106b80 2024-04-20T22:00:05.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:05.289+0000 7f36b3400700 1 --2- 172.21.15.62:0/3760996638 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f369806ecc0 0x7f3698071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:00:05.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:05.289+0000 7f36b3400700 1 --2- 172.21.15.62:0/3760996638 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f369806ecc0 0x7f3698071170 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f36a0006fd0 tx=0x7f36a0008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:00:05.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:05.293+0000 7f36b0c00700 1 -- 172.21.15.62:0/3760996638 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f36a4059490 con 0x7f36b4106b80 2024-04-20T22:00:05.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:05.554+0000 7f36b9800700 1 -- 172.21.15.62:0/3760996638 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f36b41a16e0 con 0x7f369806ecc0 2024-04-20T22:00:05.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:05.557+0000 7f36b0c00700 1 -- 172.21.15.62:0/3760996638 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f36b41a16e0 con 0x7f369806ecc0 2024-04-20T22:00:05.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:05.560+0000 7f36b9800700 1 -- 172.21.15.62:0/3760996638 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f369806ecc0 msgr2=0x7f3698071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:00:05.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:05.560+0000 7f36b9800700 1 --2- 172.21.15.62:0/3760996638 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f369806ecc0 0x7f3698071170 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f36a0006fd0 tx=0x7f36a0008040 comp rx=0 tx=0).stop 2024-04-20T22:00:05.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:05.560+0000 7f36b9800700 1 -- 172.21.15.62:0/3760996638 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f36b4106b80 msgr2=0x7f36b419c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:00:05.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:05.560+0000 7f36b9800700 1 --2- 172.21.15.62:0/3760996638 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f36b4106b80 0x7f36b419c350 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f36a4006010 tx=0x7f36a400be20 comp rx=0 tx=0).stop 2024-04-20T22:00:05.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:05.560+0000 7f36b9800700 1 -- 172.21.15.62:0/3760996638 shutdown_connections 2024-04-20T22:00:05.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:05.560+0000 7f36b9800700 1 --2- 172.21.15.62:0/3760996638 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f369806ecc0 0x7f3698071170 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:00:05.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:05.560+0000 7f36b9800700 1 --2- 172.21.15.62:0/3760996638 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f36b4106b80 0x7f36b419c350 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:00:05.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:05.560+0000 7f36b9800700 1 --2- 172.21.15.62:0/3760996638 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f36b410c660 0x7f36b419c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:00:05.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:05.560+0000 7f36b9800700 1 -- 172.21.15.62:0/3760996638 >> 172.21.15.62:0/3760996638 conn(0x7f36b4075e90 msgr2=0x7f36b4077a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:00:05.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:05.560+0000 7f36b9800700 1 -- 172.21.15.62:0/3760996638 shutdown_connections 2024-04-20T22:00:05.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:05.560+0000 7f36b9800700 1 -- 172.21.15.62:0/3760996638 wait complete. 2024-04-20T22:00:05.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:05.725+0000 7f9ad9a00700 1 -- 172.21.15.62:0/613093547 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ad410c8f0 msgr2=0x7f9ad410ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:00:05.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:05.725+0000 7f9ad9a00700 1 --2- 172.21.15.62:0/613093547 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ad410c8f0 0x7f9ad410ccc0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f9ac4009910 tx=0x7f9ac4009c20 comp rx=0 tx=0).stop 2024-04-20T22:00:05.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:05.726+0000 7f9ad9a00700 1 -- 172.21.15.62:0/613093547 shutdown_connections 2024-04-20T22:00:05.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:05.726+0000 7f9ad9a00700 1 --2- 172.21.15.62:0/613093547 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ad4106b80 0x7f9ad4106ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:00:05.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:05.726+0000 7f9ad9a00700 1 --2- 172.21.15.62:0/613093547 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ad410c8f0 0x7f9ad410ccc0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:00:05.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:05.726+0000 7f9ad9a00700 1 -- 172.21.15.62:0/613093547 >> 172.21.15.62:0/613093547 conn(0x7f9ad4076060 msgr2=0x7f9ad4078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:00:05.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:05.726+0000 7f9ad9a00700 1 -- 172.21.15.62:0/613093547 shutdown_connections 2024-04-20T22:00:05.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:05.726+0000 7f9ad9a00700 1 -- 172.21.15.62:0/613093547 wait complete. 2024-04-20T22:00:05.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:05.727+0000 7f9ad9a00700 1 Processor -- start 2024-04-20T22:00:05.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:05.727+0000 7f9ad9a00700 1 -- start start 2024-04-20T22:00:05.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:05.727+0000 7f9ad9a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ad4106b80 0x7f9ad419c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:00:05.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:05.727+0000 7f9ad9a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ad410c8f0 0x7f9ad419ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:00:05.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:05.727+0000 7f9ad9a00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ad419d120 con 0x7f9ad410c8f0 2024-04-20T22:00:05.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:05.727+0000 7f9ad9a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ad41a0eb0 con 0x7f9ad4106b80 2024-04-20T22:00:05.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:05.728+0000 7f9aca000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ad410c8f0 0x7f9ad419ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:00:05.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:05.728+0000 7f9aca000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ad410c8f0 0x7f9ad419ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55594/0 (socket says 172.21.15.62:55594) 2024-04-20T22:00:05.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:05.728+0000 7f9aca000700 1 -- 172.21.15.62:0/775296210 learned_addr learned my addr 172.21.15.62:0/775296210 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:00:05.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:05.728+0000 7f9ad2a00700 1 --2- 172.21.15.62:0/775296210 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ad4106b80 0x7f9ad419c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:00:05.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:05.728+0000 7f9aca000700 1 -- 172.21.15.62:0/775296210 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ad4106b80 msgr2=0x7f9ad419c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:00:05.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:05.728+0000 7f9aca000700 1 --2- 172.21.15.62:0/775296210 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ad4106b80 0x7f9ad419c500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:00:05.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:05.728+0000 7f9aca000700 1 -- 172.21.15.62:0/775296210 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ac40095f0 con 0x7f9ad410c8f0 2024-04-20T22:00:05.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:05.728+0000 7f9aca000700 1 --2- 172.21.15.62:0/775296210 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ad410c8f0 0x7f9ad419ca40 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f9abc00d8d0 tx=0x7f9abc00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:00:05.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:05.729+0000 7f9acbe00700 1 -- 172.21.15.62:0/775296210 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9abc009940 con 0x7f9ad410c8f0 2024-04-20T22:00:05.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:05.729+0000 7f9acbe00700 1 -- 172.21.15.62:0/775296210 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9abc010460 con 0x7f9ad410c8f0 2024-04-20T22:00:05.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:05.729+0000 7f9acbe00700 1 -- 172.21.15.62:0/775296210 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9abc00f5f0 con 0x7f9ad410c8f0 2024-04-20T22:00:05.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:05.729+0000 7f9ad9a00700 1 -- 172.21.15.62:0/775296210 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ad41a1190 con 0x7f9ad410c8f0 2024-04-20T22:00:05.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:05.729+0000 7f9ad9a00700 1 -- 172.21.15.62:0/775296210 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ad41a16e0 con 0x7f9ad410c8f0 2024-04-20T22:00:05.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:05.730+0000 7f9ad9a00700 1 -- 172.21.15.62:0/775296210 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ad4109b20 con 0x7f9ad410c8f0 2024-04-20T22:00:05.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:05.731+0000 7f9acbe00700 1 -- 172.21.15.62:0/775296210 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f9abc0105d0 con 0x7f9ad410c8f0 2024-04-20T22:00:05.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:05.732+0000 7f9acbe00700 1 --2- 172.21.15.62:0/775296210 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9ab406ecc0 0x7f9ab4071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:00:05.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:05.732+0000 7f9acbe00700 1 -- 172.21.15.62:0/775296210 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f9abc08ee10 con 0x7f9ad410c8f0 2024-04-20T22:00:05.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:05.732+0000 7f9ad2a00700 1 --2- 172.21.15.62:0/775296210 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9ab406ecc0 0x7f9ab4071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:00:05.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:05.733+0000 7f9ad2a00700 1 --2- 172.21.15.62:0/775296210 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9ab406ecc0 0x7f9ab4071170 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f9ac4000c00 tx=0x7f9ac4005fd0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:00:05.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:05.737+0000 7f9acbe00700 1 -- 172.21.15.62:0/775296210 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f9abc057ba0 con 0x7f9ad410c8f0 2024-04-20T22:00:05.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:05.983+0000 7f9ad9a00700 1 -- 172.21.15.62:0/775296210 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f9ad41a19c0 con 0x7f9ab406ecc0 2024-04-20T22:00:05.983 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:00:05 smithi062 ceph-mon[76322]: from='client.35348 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:00:05.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:00:05 smithi189 ceph-mon[57022]: from='client.35348 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:00:06.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.001+0000 7f9acbe00700 1 -- 172.21.15.62:0/775296210 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f9ad41a19c0 con 0x7f9ab406ecc0 2024-04-20T22:00:06.001 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T22:00:06.001 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (25m) 2m ago 39m 15.3M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T22:00:06.001 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (40m) 2m ago 40m 10.4M - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T22:00:06.001 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (38m) 2m ago 38m 10.1M - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T22:00:06.001 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (22m) 2m ago 40m 6660k - 19.0.0-3133-ga8c84f59 76bc6b91b726 2825d5ba11c6 2024-04-20T22:00:06.001 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (21m) 2m ago 38m 6665k - 19.0.0-3133-ga8c84f59 76bc6b91b726 20f746fa1bba 2024-04-20T22:00:06.001 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (25m) 2m ago 39m 73.5M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T22:00:06.001 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (17m) 2m ago 32m 18.5M - 19.0.0-3133-ga8c84f59 76bc6b91b726 de80bc90a683 2024-04-20T22:00:06.001 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (17m) 2m ago 32m 18.7M - 19.0.0-3133-ga8c84f59 76bc6b91b726 cd793b7ee98e 2024-04-20T22:00:06.001 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (2m) 2m ago 32m 11.0M - 19.0.0-3133-ga8c84f59 76bc6b91b726 0e7b0bc4d1ab 2024-04-20T22:00:06.001 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (32m) 2m ago 32m 18.6M - 18.2.0 dc2bc1663786 f2eec2ee8951 2024-04-20T22:00:06.001 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (28m) 2m ago 41m 513M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T22:00:06.001 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (27m) 2m ago 38m 459M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T22:00:06.002 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (23m) 2m ago 41m 71.7M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 019eb1479485 2024-04-20T22:00:06.002 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (23m) 2m ago 38m 51.0M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 55aac560aab6 2024-04-20T22:00:06.002 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (27m) 2m ago 39m 8472k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T22:00:06.002 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (27m) 2m ago 38m 8048k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T22:00:06.002 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (21m) 2m ago 37m 91.2M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 ee2bb901f26e 2024-04-20T22:00:06.002 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (20m) 2m ago 36m 94.6M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 9075152d4162 2024-04-20T22:00:06.002 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (20m) 2m ago 36m 78.1M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 2d4f3e38dc0e 2024-04-20T22:00:06.002 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (19m) 2m ago 35m 92.6M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 8f97b17ae0eb 2024-04-20T22:00:06.002 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (19m) 2m ago 35m 95.6M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 653d8079967d 2024-04-20T22:00:06.002 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (18m) 2m ago 34m 89.2M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 596dded14c12 2024-04-20T22:00:06.002 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (26m) 2m ago 39m 82.1M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T22:00:06.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.003+0000 7f9ad9a00700 1 -- 172.21.15.62:0/775296210 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9ab406ecc0 msgr2=0x7f9ab4071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:00:06.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.003+0000 7f9ad9a00700 1 --2- 172.21.15.62:0/775296210 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9ab406ecc0 0x7f9ab4071170 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f9ac4000c00 tx=0x7f9ac4005fd0 comp rx=0 tx=0).stop 2024-04-20T22:00:06.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.003+0000 7f9ad9a00700 1 -- 172.21.15.62:0/775296210 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ad410c8f0 msgr2=0x7f9ad419ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:00:06.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.003+0000 7f9ad9a00700 1 --2- 172.21.15.62:0/775296210 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ad410c8f0 0x7f9ad419ca40 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f9abc00d8d0 tx=0x7f9abc00dc90 comp rx=0 tx=0).stop 2024-04-20T22:00:06.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.004+0000 7f9ad9a00700 1 -- 172.21.15.62:0/775296210 shutdown_connections 2024-04-20T22:00:06.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.004+0000 7f9ad9a00700 1 --2- 172.21.15.62:0/775296210 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9ab406ecc0 0x7f9ab4071170 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:00:06.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.004+0000 7f9ad9a00700 1 --2- 172.21.15.62:0/775296210 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ad4106b80 0x7f9ad419c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:00:06.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.004+0000 7f9ad9a00700 1 --2- 172.21.15.62:0/775296210 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ad410c8f0 0x7f9ad419ca40 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:00:06.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.004+0000 7f9ad9a00700 1 -- 172.21.15.62:0/775296210 >> 172.21.15.62:0/775296210 conn(0x7f9ad4076060 msgr2=0x7f9ad4077aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:00:06.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.004+0000 7f9ad9a00700 1 -- 172.21.15.62:0/775296210 shutdown_connections 2024-04-20T22:00:06.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.004+0000 7f9ad9a00700 1 -- 172.21.15.62:0/775296210 wait complete. 2024-04-20T22:00:06.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.168+0000 7fa5baa00700 1 -- 172.21.15.62:0/2292002932 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5b4102e30 msgr2=0x7fa5b4103200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:00:06.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.168+0000 7fa5baa00700 1 --2- 172.21.15.62:0/2292002932 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5b4102e30 0x7fa5b4103200 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fa5a8009910 tx=0x7fa5a8009c20 comp rx=0 tx=0).stop 2024-04-20T22:00:06.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.168+0000 7fa5baa00700 1 -- 172.21.15.62:0/2292002932 shutdown_connections 2024-04-20T22:00:06.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.168+0000 7fa5baa00700 1 --2- 172.21.15.62:0/2292002932 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5b40ff9e0 0x7fa5b40ffe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:00:06.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.168+0000 7fa5baa00700 1 --2- 172.21.15.62:0/2292002932 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5b4102e30 0x7fa5b4103200 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:00:06.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.168+0000 7fa5baa00700 1 -- 172.21.15.62:0/2292002932 >> 172.21.15.62:0/2292002932 conn(0x7fa5b4074fa0 msgr2=0x7fa5b40753a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:00:06.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.168+0000 7fa5baa00700 1 -- 172.21.15.62:0/2292002932 shutdown_connections 2024-04-20T22:00:06.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.168+0000 7fa5baa00700 1 -- 172.21.15.62:0/2292002932 wait complete. 2024-04-20T22:00:06.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.169+0000 7fa5baa00700 1 Processor -- start 2024-04-20T22:00:06.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.169+0000 7fa5baa00700 1 -- start start 2024-04-20T22:00:06.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.169+0000 7fa5baa00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5b40ff9e0 0x7fa5b4197e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:00:06.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.170+0000 7fa5baa00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5b4102e30 0x7fa5b4198340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:00:06.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.170+0000 7fa5baa00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5b4198990 con 0x7fa5b40ff9e0 2024-04-20T22:00:06.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.170+0000 7fa5baa00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5b4198ad0 con 0x7fa5b4102e30 2024-04-20T22:00:06.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.170+0000 7fa5b9600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5b40ff9e0 0x7fa5b4197e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:00:06.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.170+0000 7fa5b9600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5b40ff9e0 0x7fa5b4197e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55614/0 (socket says 172.21.15.62:55614) 2024-04-20T22:00:06.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.170+0000 7fa5b9600700 1 -- 172.21.15.62:0/3423437457 learned_addr learned my addr 172.21.15.62:0/3423437457 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:00:06.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.170+0000 7fa5b8c00700 1 --2- 172.21.15.62:0/3423437457 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5b4102e30 0x7fa5b4198340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:00:06.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.170+0000 7fa5b9600700 1 -- 172.21.15.62:0/3423437457 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5b4102e30 msgr2=0x7fa5b4198340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:00:06.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.170+0000 7fa5b9600700 1 --2- 172.21.15.62:0/3423437457 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5b4102e30 0x7fa5b4198340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:00:06.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.170+0000 7fa5b9600700 1 -- 172.21.15.62:0/3423437457 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5a80095f0 con 0x7fa5b40ff9e0 2024-04-20T22:00:06.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.171+0000 7fa5b9600700 1 --2- 172.21.15.62:0/3423437457 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5b40ff9e0 0x7fa5b4197e00 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fa5a800b5c0 tx=0x7fa5a800be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:00:06.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.171+0000 7fa5b2000700 1 -- 172.21.15.62:0/3423437457 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa5a801d070 con 0x7fa5b40ff9e0 2024-04-20T22:00:06.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.171+0000 7fa5b2000700 1 -- 172.21.15.62:0/3423437457 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa5a800f460 con 0x7fa5b40ff9e0 2024-04-20T22:00:06.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.171+0000 7fa5baa00700 1 -- 172.21.15.62:0/3423437457 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5b419c8c0 con 0x7fa5b40ff9e0 2024-04-20T22:00:06.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.171+0000 7fa5b2000700 1 -- 172.21.15.62:0/3423437457 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa5a8017610 con 0x7fa5b40ff9e0 2024-04-20T22:00:06.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.171+0000 7fa5baa00700 1 -- 172.21.15.62:0/3423437457 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5b419ce30 con 0x7fa5b40ff9e0 2024-04-20T22:00:06.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.172+0000 7fa5baa00700 1 -- 172.21.15.62:0/3423437457 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5b4062380 con 0x7fa5b40ff9e0 2024-04-20T22:00:06.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.174+0000 7fa5b2000700 1 -- 172.21.15.62:0/3423437457 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fa5a800f5d0 con 0x7fa5b40ff9e0 2024-04-20T22:00:06.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.175+0000 7fa5b2000700 1 --2- 172.21.15.62:0/3423437457 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa59c06ecc0 0x7fa59c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:00:06.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.175+0000 7fa5b2000700 1 -- 172.21.15.62:0/3423437457 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fa5a8090440 con 0x7fa5b40ff9e0 2024-04-20T22:00:06.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.175+0000 7fa5b8c00700 1 --2- 172.21.15.62:0/3423437457 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa59c06ecc0 0x7fa59c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:00:06.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.176+0000 7fa5b8c00700 1 --2- 172.21.15.62:0/3423437457 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa59c06ecc0 0x7fa59c071170 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fa5a0005ea0 tx=0x7fa5a0005e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:00:06.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.179+0000 7fa5b2000700 1 -- 172.21.15.62:0/3423437457 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fa5a80591d0 con 0x7fa5b40ff9e0 2024-04-20T22:00:06.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.501+0000 7fa5baa00700 1 -- 172.21.15.62:0/3423437457 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fa5b419d270 con 0x7fa5b40ff9e0 2024-04-20T22:00:06.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.502+0000 7fa5b2000700 1 -- 172.21.15.62:0/3423437457 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7fa5a805c890 con 0x7fa5b40ff9e0 2024-04-20T22:00:06.502 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:00:06.502 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T22:00:06.502 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:00:06.502 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:00:06.502 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T22:00:06.502 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:00:06.502 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:00:06.502 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T22:00:06.502 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 6 2024-04-20T22:00:06.502 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:00:06.502 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T22:00:06.503 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 1, 2024-04-20T22:00:06.503 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 3 2024-04-20T22:00:06.503 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:00:06.503 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T22:00:06.503 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 1, 2024-04-20T22:00:06.503 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 13 2024-04-20T22:00:06.503 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T22:00:06.503 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:00:06.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.505+0000 7fa5baa00700 1 -- 172.21.15.62:0/3423437457 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa59c06ecc0 msgr2=0x7fa59c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:00:06.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.505+0000 7fa5baa00700 1 --2- 172.21.15.62:0/3423437457 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa59c06ecc0 0x7fa59c071170 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fa5a0005ea0 tx=0x7fa5a0005e30 comp rx=0 tx=0).stop 2024-04-20T22:00:06.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.505+0000 7fa5baa00700 1 -- 172.21.15.62:0/3423437457 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5b40ff9e0 msgr2=0x7fa5b4197e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:00:06.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.505+0000 7fa5baa00700 1 --2- 172.21.15.62:0/3423437457 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5b40ff9e0 0x7fa5b4197e00 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fa5a800b5c0 tx=0x7fa5a800be00 comp rx=0 tx=0).stop 2024-04-20T22:00:06.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.505+0000 7fa5baa00700 1 -- 172.21.15.62:0/3423437457 shutdown_connections 2024-04-20T22:00:06.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.505+0000 7fa5baa00700 1 --2- 172.21.15.62:0/3423437457 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa59c06ecc0 0x7fa59c071170 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:00:06.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.505+0000 7fa5baa00700 1 --2- 172.21.15.62:0/3423437457 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5b40ff9e0 0x7fa5b4197e00 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:00:06.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.505+0000 7fa5baa00700 1 --2- 172.21.15.62:0/3423437457 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5b4102e30 0x7fa5b4198340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:00:06.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.505+0000 7fa5baa00700 1 -- 172.21.15.62:0/3423437457 >> 172.21.15.62:0/3423437457 conn(0x7fa5b4074fa0 msgr2=0x7fa5b40fe9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:00:06.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.505+0000 7fa5baa00700 1 -- 172.21.15.62:0/3423437457 shutdown_connections 2024-04-20T22:00:06.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.506+0000 7fa5baa00700 1 -- 172.21.15.62:0/3423437457 wait complete. 2024-04-20T22:00:06.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.669+0000 7fc4a4800700 1 -- 172.21.15.62:0/1116110487 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc49c0ffbd0 msgr2=0x7fc49c100040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:00:06.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.669+0000 7fc4a4800700 1 --2- 172.21.15.62:0/1116110487 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc49c0ffbd0 0x7fc49c100040 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fc48c009910 tx=0x7fc48c009c20 comp rx=0 tx=0).stop 2024-04-20T22:00:06.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.670+0000 7fc4a4800700 1 -- 172.21.15.62:0/1116110487 shutdown_connections 2024-04-20T22:00:06.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.670+0000 7fc4a4800700 1 --2- 172.21.15.62:0/1116110487 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc49c0ffbd0 0x7fc49c100040 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:00:06.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.670+0000 7fc4a4800700 1 --2- 172.21.15.62:0/1116110487 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc49c104130 0x7fc49c104500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:00:06.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.670+0000 7fc4a4800700 1 -- 172.21.15.62:0/1116110487 >> 172.21.15.62:0/1116110487 conn(0x7fc49c075130 msgr2=0x7fc49c075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:00:06.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.670+0000 7fc4a4800700 1 -- 172.21.15.62:0/1116110487 shutdown_connections 2024-04-20T22:00:06.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.670+0000 7fc4a4800700 1 -- 172.21.15.62:0/1116110487 wait complete. 2024-04-20T22:00:06.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.671+0000 7fc4a4800700 1 Processor -- start 2024-04-20T22:00:06.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.671+0000 7fc4a4800700 1 -- start start 2024-04-20T22:00:06.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.672+0000 7fc4a4800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc49c0ffbd0 0x7fc49c100e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:00:06.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.672+0000 7fc4a4800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc49c104130 0x7fc49c101370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:00:06.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.672+0000 7fc4a4800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc49c1018b0 con 0x7fc49c0ffbd0 2024-04-20T22:00:06.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.672+0000 7fc4a4800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc49c1019f0 con 0x7fc49c104130 2024-04-20T22:00:06.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.672+0000 7fc4a1e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc49c0ffbd0 0x7fc49c100e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:00:06.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.672+0000 7fc4a1e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc49c0ffbd0 0x7fc49c100e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55634/0 (socket says 172.21.15.62:55634) 2024-04-20T22:00:06.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.672+0000 7fc4a1e00700 1 -- 172.21.15.62:0/3438676474 learned_addr learned my addr 172.21.15.62:0/3438676474 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:00:06.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.672+0000 7fc4a1400700 1 --2- 172.21.15.62:0/3438676474 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc49c104130 0x7fc49c101370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:00:06.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.673+0000 7fc4a1e00700 1 -- 172.21.15.62:0/3438676474 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc49c104130 msgr2=0x7fc49c101370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:00:06.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.673+0000 7fc4a1e00700 1 --2- 172.21.15.62:0/3438676474 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc49c104130 0x7fc49c101370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:00:06.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.673+0000 7fc4a1e00700 1 -- 172.21.15.62:0/3438676474 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc48c0095f0 con 0x7fc49c0ffbd0 2024-04-20T22:00:06.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.673+0000 7fc4a1e00700 1 --2- 172.21.15.62:0/3438676474 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc49c0ffbd0 0x7fc49c100e30 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fc49000e920 tx=0x7fc49000ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:00:06.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.673+0000 7fc496a00700 1 -- 172.21.15.62:0/3438676474 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc490009790 con 0x7fc49c0ffbd0 2024-04-20T22:00:06.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.673+0000 7fc496a00700 1 -- 172.21.15.62:0/3438676474 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc49000cba0 con 0x7fc49c0ffbd0 2024-04-20T22:00:06.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.673+0000 7fc496a00700 1 -- 172.21.15.62:0/3438676474 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc490018650 con 0x7fc49c0ffbd0 2024-04-20T22:00:06.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.673+0000 7fc4a4800700 1 -- 172.21.15.62:0/3438676474 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc49c1a2040 con 0x7fc49c0ffbd0 2024-04-20T22:00:06.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.673+0000 7fc4a4800700 1 -- 172.21.15.62:0/3438676474 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc49c1a2450 con 0x7fc49c0ffbd0 2024-04-20T22:00:06.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.675+0000 7fc4a4800700 1 -- 172.21.15.62:0/3438676474 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc49c1086a0 con 0x7fc49c0ffbd0 2024-04-20T22:00:06.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.676+0000 7fc496a00700 1 -- 172.21.15.62:0/3438676474 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fc49000cd10 con 0x7fc49c0ffbd0 2024-04-20T22:00:06.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.677+0000 7fc496a00700 1 --2- 172.21.15.62:0/3438676474 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc48406ecc0 0x7fc484071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:00:06.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.677+0000 7fc496a00700 1 -- 172.21.15.62:0/3438676474 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fc490014070 con 0x7fc49c0ffbd0 2024-04-20T22:00:06.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.677+0000 7fc4a1400700 1 --2- 172.21.15.62:0/3438676474 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc48406ecc0 0x7fc484071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:00:06.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.678+0000 7fc4a1400700 1 --2- 172.21.15.62:0/3438676474 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc48406ecc0 0x7fc484071170 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fc49c102620 tx=0x7fc48c005fb0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:00:06.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.682+0000 7fc496a00700 1 -- 172.21.15.62:0/3438676474 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fc490053ff0 con 0x7fc49c0ffbd0 2024-04-20T22:00:06.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:00:06 smithi062 ceph-mon[76322]: pgmap v745: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:00:06.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:00:06 smithi062 ceph-mon[76322]: from='client.35352 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:00:06.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:00:06 smithi062 ceph-mon[76322]: from='client.35356 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:00:06.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:00:06 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/3423437457' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:00:06.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.962+0000 7fc4a4800700 1 -- 172.21.15.62:0/3438676474 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fc49c108080 con 0x7fc49c0ffbd0 2024-04-20T22:00:06.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.963+0000 7fc496a00700 1 -- 172.21.15.62:0/3438676474 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 27 v27) v1 ==== 76+0+1888 (secure 0 0 0) 0x7fc49005bad0 con 0x7fc49c0ffbd0 2024-04-20T22:00:06.963 INFO:teuthology.orchestra.run.smithi062.stdout:e27 2024-04-20T22:00:06.963 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T22:00:06.964 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:00:06.964 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T22:00:06.964 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:00:06.964 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T22:00:06.964 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T22:00:06.964 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 20 2024-04-20T22:00:06.964 INFO:teuthology.orchestra.run.smithi062.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-20T22:00:06.964 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T22:00:06.964 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T21:42:01.559061+0000 2024-04-20T22:00:06.964 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T22:00:06.964 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T22:00:06.964 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T22:00:06.964 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T22:00:06.964 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T22:00:06.964 INFO:teuthology.orchestra.run.smithi062.stdout:max_xattr_size 65536 2024-04-20T22:00:06.964 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T22:00:06.965 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T22:00:06.965 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 81 2024-04-20T22:00:06.965 INFO:teuthology.orchestra.run.smithi062.stdout:snap_rstat 1 2024-04-20T22:00:06.965 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:00:06.965 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 2 2024-04-20T22:00:06.965 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1 2024-04-20T22:00:06.965 INFO:teuthology.orchestra.run.smithi062.stdout:up {} 2024-04-20T22:00:06.965 INFO:teuthology.orchestra.run.smithi062.stdout:failed 0,1 2024-04-20T22:00:06.965 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T22:00:06.965 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T22:00:06.965 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T22:00:06.965 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T22:00:06.965 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data enabled 2024-04-20T22:00:06.965 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T22:00:06.965 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T22:00:06.966 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 1 2024-04-20T22:00:06.966 INFO:teuthology.orchestra.run.smithi062.stdout:qdb_cluster leader: 0 members: 2024-04-20T22:00:06.966 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:00:06.966 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:00:06.966 INFO:teuthology.orchestra.run.smithi062.stdout:Standby daemons: 2024-04-20T22:00:06.966 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:00:06.966 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ryqblb{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6826/4113916205,v1:172.21.15.62:6827/4113916205] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:00:06.966 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ozizwi{-1:34476} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6828/3801975165,v1:172.21.15.62:6829/3801975165] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:00:06.966 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.xbgspw{-1:44285} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6826/3996040526,v1:172.21.15.189:6827/3996040526] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T22:00:06.966 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.dvimer{-1:44727} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6824/3576699232,v1:172.21.15.189:6825/3576699232] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:00:06.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.966+0000 7fc4a4800700 1 -- 172.21.15.62:0/3438676474 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc48406ecc0 msgr2=0x7fc484071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:00:06.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.966+0000 7fc4a4800700 1 --2- 172.21.15.62:0/3438676474 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc48406ecc0 0x7fc484071170 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fc49c102620 tx=0x7fc48c005fb0 comp rx=0 tx=0).stop 2024-04-20T22:00:06.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.966+0000 7fc4a4800700 1 -- 172.21.15.62:0/3438676474 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc49c0ffbd0 msgr2=0x7fc49c100e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:00:06.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.966+0000 7fc4a4800700 1 --2- 172.21.15.62:0/3438676474 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc49c0ffbd0 0x7fc49c100e30 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fc49000e920 tx=0x7fc49000ece0 comp rx=0 tx=0).stop 2024-04-20T22:00:06.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.966+0000 7fc4a4800700 1 -- 172.21.15.62:0/3438676474 shutdown_connections 2024-04-20T22:00:06.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.966+0000 7fc4a4800700 1 --2- 172.21.15.62:0/3438676474 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc48406ecc0 0x7fc484071170 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:00:06.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.966+0000 7fc4a4800700 1 --2- 172.21.15.62:0/3438676474 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc49c0ffbd0 0x7fc49c100e30 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:00:06.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.966+0000 7fc4a4800700 1 --2- 172.21.15.62:0/3438676474 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc49c104130 0x7fc49c101370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:00:06.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.966+0000 7fc4a4800700 1 -- 172.21.15.62:0/3438676474 >> 172.21.15.62:0/3438676474 conn(0x7fc49c075130 msgr2=0x7fc49c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:00:06.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.966+0000 7fc4a4800700 1 -- 172.21.15.62:0/3438676474 shutdown_connections 2024-04-20T22:00:06.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:06.966+0000 7fc4a4800700 1 -- 172.21.15.62:0/3438676474 wait complete. 2024-04-20T22:00:06.968 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 27 2024-04-20T22:00:06.989 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:00:06 smithi189 ceph-mon[57022]: pgmap v745: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:00:06.989 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:00:06 smithi189 ceph-mon[57022]: from='client.35352 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:00:06.989 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:00:06 smithi189 ceph-mon[57022]: from='client.35356 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:00:06.989 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:00:06 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/3423437457' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:00:07.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:07.130+0000 7f72b6800700 1 -- 172.21.15.62:0/1199781200 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72b01004e0 msgr2=0x7f72b01008b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:00:07.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:07.130+0000 7f72b6800700 1 --2- 172.21.15.62:0/1199781200 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72b01004e0 0x7f72b01008b0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f72a0009910 tx=0x7f72a0009c20 comp rx=0 tx=0).stop 2024-04-20T22:00:07.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:07.130+0000 7f72b6800700 1 -- 172.21.15.62:0/1199781200 shutdown_connections 2024-04-20T22:00:07.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:07.130+0000 7f72b6800700 1 --2- 172.21.15.62:0/1199781200 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72b0100e80 0x7f72b0104e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:00:07.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:07.130+0000 7f72b6800700 1 --2- 172.21.15.62:0/1199781200 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72b01004e0 0x7f72b01008b0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:00:07.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:07.130+0000 7f72b6800700 1 -- 172.21.15.62:0/1199781200 >> 172.21.15.62:0/1199781200 conn(0x7f72b00fbda0 msgr2=0x7f72b00fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:00:07.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:07.130+0000 7f72b6800700 1 -- 172.21.15.62:0/1199781200 shutdown_connections 2024-04-20T22:00:07.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:07.130+0000 7f72b6800700 1 -- 172.21.15.62:0/1199781200 wait complete. 2024-04-20T22:00:07.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:07.132+0000 7f72b6800700 1 Processor -- start 2024-04-20T22:00:07.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:07.132+0000 7f72b6800700 1 -- start start 2024-04-20T22:00:07.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:07.132+0000 7f72b6800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72b01004e0 0x7f72b0198110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:00:07.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:07.132+0000 7f72b6800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72b0100e80 0x7f72b0198650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:00:07.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:07.132+0000 7f72b6800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72b0198d30 con 0x7f72b0100e80 2024-04-20T22:00:07.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:07.132+0000 7f72b6800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72b019cac0 con 0x7f72b01004e0 2024-04-20T22:00:07.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:07.132+0000 7f72a6a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72b0100e80 0x7f72b0198650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:00:07.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:07.133+0000 7f72a6a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72b0100e80 0x7f72b0198650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55650/0 (socket says 172.21.15.62:55650) 2024-04-20T22:00:07.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:07.133+0000 7f72a6a00700 1 -- 172.21.15.62:0/3556043228 learned_addr learned my addr 172.21.15.62:0/3556043228 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:00:07.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:07.133+0000 7f72af400700 1 --2- 172.21.15.62:0/3556043228 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72b01004e0 0x7f72b0198110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:00:07.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:07.133+0000 7f72a6a00700 1 -- 172.21.15.62:0/3556043228 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72b01004e0 msgr2=0x7f72b0198110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:00:07.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:07.133+0000 7f72a6a00700 1 --2- 172.21.15.62:0/3556043228 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72b01004e0 0x7f72b0198110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:00:07.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:07.133+0000 7f72a6a00700 1 -- 172.21.15.62:0/3556043228 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72a00095f0 con 0x7f72b0100e80 2024-04-20T22:00:07.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:07.133+0000 7f72a6a00700 1 --2- 172.21.15.62:0/3556043228 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72b0100e80 0x7f72b0198650 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f729800d900 tx=0x7f729800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:00:07.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:07.134+0000 7f72acc00700 1 -- 172.21.15.62:0/3556043228 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7298009900 con 0x7f72b0100e80 2024-04-20T22:00:07.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:07.134+0000 7f72acc00700 1 -- 172.21.15.62:0/3556043228 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7298010460 con 0x7f72b0100e80 2024-04-20T22:00:07.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:07.134+0000 7f72b6800700 1 -- 172.21.15.62:0/3556043228 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72b019cda0 con 0x7f72b0100e80 2024-04-20T22:00:07.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:07.134+0000 7f72acc00700 1 -- 172.21.15.62:0/3556043228 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f729800f5f0 con 0x7f72b0100e80 2024-04-20T22:00:07.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:07.134+0000 7f72b6800700 1 -- 172.21.15.62:0/3556043228 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72b019d2f0 con 0x7f72b0100e80 2024-04-20T22:00:07.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:07.135+0000 7f72b6800700 1 -- 172.21.15.62:0/3556043228 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72b004f2a0 con 0x7f72b0100e80 2024-04-20T22:00:07.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:07.136+0000 7f72acc00700 1 -- 172.21.15.62:0/3556043228 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f72980105d0 con 0x7f72b0100e80 2024-04-20T22:00:07.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:07.137+0000 7f72acc00700 1 --2- 172.21.15.62:0/3556043228 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f729406ecc0 0x7f7294071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:00:07.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:07.137+0000 7f72acc00700 1 -- 172.21.15.62:0/3556043228 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f729808ec00 con 0x7f72b0100e80 2024-04-20T22:00:07.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:07.138+0000 7f72af400700 1 --2- 172.21.15.62:0/3556043228 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f729406ecc0 0x7f7294071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:00:07.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:07.138+0000 7f72af400700 1 --2- 172.21.15.62:0/3556043228 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f729406ecc0 0x7f7294071170 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f72a0006010 tx=0x7f72a001a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:00:07.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:07.143+0000 7f72acc00700 1 -- 172.21.15.62:0/3556043228 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f7298094020 con 0x7f72b0100e80 2024-04-20T22:00:07.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:07.395+0000 7f72b6800700 1 -- 172.21.15.62:0/3556043228 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f72b019d670 con 0x7f729406ecc0 2024-04-20T22:00:07.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:07.398+0000 7f72acc00700 1 -- 172.21.15.62:0/3556043228 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f72b019d670 con 0x7f729406ecc0 2024-04-20T22:00:07.398 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:00:07.398 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T22:00:07.398 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T22:00:07.398 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-20T22:00:07.398 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T22:00:07.398 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr", 2024-04-20T22:00:07.398 INFO:teuthology.orchestra.run.smithi062.stdout: "crash", 2024-04-20T22:00:07.399 INFO:teuthology.orchestra.run.smithi062.stdout: "mon", 2024-04-20T22:00:07.399 INFO:teuthology.orchestra.run.smithi062.stdout: "osd" 2024-04-20T22:00:07.399 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T22:00:07.399 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "15/23 daemons upgraded", 2024-04-20T22:00:07.399 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading mds daemons", 2024-04-20T22:00:07.399 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T22:00:07.399 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:00:07.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:07.401+0000 7f72b6800700 1 -- 172.21.15.62:0/3556043228 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f729406ecc0 msgr2=0x7f7294071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:00:07.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:07.401+0000 7f72b6800700 1 --2- 172.21.15.62:0/3556043228 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f729406ecc0 0x7f7294071170 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f72a0006010 tx=0x7f72a001a040 comp rx=0 tx=0).stop 2024-04-20T22:00:07.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:07.401+0000 7f72b6800700 1 -- 172.21.15.62:0/3556043228 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72b0100e80 msgr2=0x7f72b0198650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:00:07.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:07.401+0000 7f72b6800700 1 --2- 172.21.15.62:0/3556043228 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72b0100e80 0x7f72b0198650 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f729800d900 tx=0x7f729800dcc0 comp rx=0 tx=0).stop 2024-04-20T22:00:07.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:07.401+0000 7f72b6800700 1 -- 172.21.15.62:0/3556043228 shutdown_connections 2024-04-20T22:00:07.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:07.401+0000 7f72b6800700 1 --2- 172.21.15.62:0/3556043228 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f729406ecc0 0x7f7294071170 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:00:07.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:07.401+0000 7f72b6800700 1 --2- 172.21.15.62:0/3556043228 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72b01004e0 0x7f72b0198110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:00:07.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:07.401+0000 7f72b6800700 1 --2- 172.21.15.62:0/3556043228 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72b0100e80 0x7f72b0198650 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:00:07.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:07.401+0000 7f72b6800700 1 -- 172.21.15.62:0/3556043228 >> 172.21.15.62:0/3556043228 conn(0x7f72b00fbda0 msgr2=0x7f72b0103e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:00:07.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:07.401+0000 7f72b6800700 1 -- 172.21.15.62:0/3556043228 shutdown_connections 2024-04-20T22:00:07.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:07.401+0000 7f72b6800700 1 -- 172.21.15.62:0/3556043228 wait complete. 2024-04-20T22:00:07.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:07.560+0000 7fe95c200700 1 -- 172.21.15.62:0/336843435 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe954105230 msgr2=0x7fe954105600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:00:07.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:07.561+0000 7fe95c200700 1 --2- 172.21.15.62:0/336843435 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe954105230 0x7fe954105600 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fe948009910 tx=0x7fe948009c20 comp rx=0 tx=0).stop 2024-04-20T22:00:07.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:07.561+0000 7fe95c200700 1 -- 172.21.15.62:0/336843435 shutdown_connections 2024-04-20T22:00:07.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:07.561+0000 7fe95c200700 1 --2- 172.21.15.62:0/336843435 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe954068540 0x7fe9540689b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:00:07.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:07.561+0000 7fe95c200700 1 --2- 172.21.15.62:0/336843435 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe954105230 0x7fe954105600 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:00:07.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:07.561+0000 7fe95c200700 1 -- 172.21.15.62:0/336843435 >> 172.21.15.62:0/336843435 conn(0x7fe954075130 msgr2=0x7fe954075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:00:07.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:07.561+0000 7fe95c200700 1 -- 172.21.15.62:0/336843435 shutdown_connections 2024-04-20T22:00:07.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:07.561+0000 7fe95c200700 1 -- 172.21.15.62:0/336843435 wait complete. 2024-04-20T22:00:07.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:07.562+0000 7fe95c200700 1 Processor -- start 2024-04-20T22:00:07.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:07.562+0000 7fe95c200700 1 -- start start 2024-04-20T22:00:07.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:07.562+0000 7fe95c200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe954068540 0x7fe954197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:00:07.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:07.563+0000 7fe95c200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe954105230 0x7fe954198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:00:07.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:07.563+0000 7fe95c200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe954198c00 con 0x7fe954105230 2024-04-20T22:00:07.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:07.563+0000 7fe95c200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe95419c990 con 0x7fe954068540 2024-04-20T22:00:07.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:07.563+0000 7fe958e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe954105230 0x7fe954198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:00:07.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:07.563+0000 7fe958e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe954105230 0x7fe954198520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55682/0 (socket says 172.21.15.62:55682) 2024-04-20T22:00:07.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:07.563+0000 7fe958e00700 1 -- 172.21.15.62:0/3648198479 learned_addr learned my addr 172.21.15.62:0/3648198479 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:00:07.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:07.563+0000 7fe959800700 1 --2- 172.21.15.62:0/3648198479 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe954068540 0x7fe954197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:00:07.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:07.563+0000 7fe958e00700 1 -- 172.21.15.62:0/3648198479 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe954068540 msgr2=0x7fe954197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:00:07.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:07.563+0000 7fe958e00700 1 --2- 172.21.15.62:0/3648198479 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe954068540 0x7fe954197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:00:07.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:07.563+0000 7fe958e00700 1 -- 172.21.15.62:0/3648198479 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9480095f0 con 0x7fe954105230 2024-04-20T22:00:07.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:07.564+0000 7fe958e00700 1 --2- 172.21.15.62:0/3648198479 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe954105230 0x7fe954198520 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fe94000d8d0 tx=0x7fe94000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:00:07.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:07.564+0000 7fe952000700 1 -- 172.21.15.62:0/3648198479 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe940009940 con 0x7fe954105230 2024-04-20T22:00:07.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:07.564+0000 7fe952000700 1 -- 172.21.15.62:0/3648198479 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe940010460 con 0x7fe954105230 2024-04-20T22:00:07.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:07.564+0000 7fe95c200700 1 -- 172.21.15.62:0/3648198479 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe95419cc70 con 0x7fe954105230 2024-04-20T22:00:07.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:07.564+0000 7fe952000700 1 -- 172.21.15.62:0/3648198479 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe94000f5f0 con 0x7fe954105230 2024-04-20T22:00:07.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:07.564+0000 7fe95c200700 1 -- 172.21.15.62:0/3648198479 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe95419d1c0 con 0x7fe954105230 2024-04-20T22:00:07.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:07.565+0000 7fe95c200700 1 -- 172.21.15.62:0/3648198479 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe95404f2a0 con 0x7fe954105230 2024-04-20T22:00:07.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:07.567+0000 7fe952000700 1 -- 172.21.15.62:0/3648198479 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe9400105d0 con 0x7fe954105230 2024-04-20T22:00:07.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:07.568+0000 7fe952000700 1 --2- 172.21.15.62:0/3648198479 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe93c06ecc0 0x7fe93c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:00:07.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:07.568+0000 7fe952000700 1 -- 172.21.15.62:0/3648198479 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fe94008ee10 con 0x7fe954105230 2024-04-20T22:00:07.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:07.568+0000 7fe959800700 1 --2- 172.21.15.62:0/3648198479 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe93c06ecc0 0x7fe93c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:00:07.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:07.569+0000 7fe959800700 1 --2- 172.21.15.62:0/3648198479 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe93c06ecc0 0x7fe93c071170 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fe948006010 tx=0x7fe94801a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:00:07.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:07.573+0000 7fe952000700 1 -- 172.21.15.62:0/3648198479 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fe940057ba0 con 0x7fe954105230 2024-04-20T22:00:07.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:00:07 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/3438676474' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:00:07.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:07.889+0000 7fe95c200700 1 -- 172.21.15.62:0/3648198479 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fe954199340 con 0x7fe954105230 2024-04-20T22:00:07.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:07.890+0000 7fe952000700 1 -- 172.21.15.62:0/3648198479 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7fe94005b260 con 0x7fe954105230 2024-04-20T22:00:07.890 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-20T22:00:07.890 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-20T22:00:07.890 INFO:teuthology.orchestra.run.smithi062.stdout: host smithi062 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T22:00:07.890 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-20T22:00:07.891 INFO:teuthology.orchestra.run.smithi062.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi062 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpq7y7oa_r:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:00:07.891 INFO:teuthology.orchestra.run.smithi062.stdout:Traceback (most recent call last): 2024-04-20T22:00:07.891 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-20T22:00:07.891 INFO:teuthology.orchestra.run.smithi062.stdout: return _run_code(code, main_globals, None, 2024-04-20T22:00:07.891 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-20T22:00:07.891 INFO:teuthology.orchestra.run.smithi062.stdout: exec(code, run_globals) 2024-04-20T22:00:07.891 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5618, in 2024-04-20T22:00:07.891 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5606, in main 2024-04-20T22:00:07.891 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 408, in _infer_config 2024-04-20T22:00:07.891 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 323, in _infer_fsid 2024-04-20T22:00:07.891 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 436, in _infer_image 2024-04-20T22:00:07.891 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 310, in _validate_fsid 2024-04-20T22:00:07.891 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 3359, in command_ceph_volume 2024-04-20T22:00:07.891 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-20T22:00:07.892 INFO:teuthology.orchestra.run.smithi062.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi062 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpq7y7oa_r:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:00:07.892 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-20T22:00:07.892 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is degraded 2024-04-20T22:00:07.892 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T22:00:07.892 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T22:00:07.892 INFO:teuthology.orchestra.run.smithi062.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-20T22:00:07.892 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-20T22:00:07.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:07.894+0000 7fe95c200700 1 -- 172.21.15.62:0/3648198479 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe93c06ecc0 msgr2=0x7fe93c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:00:07.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:07.894+0000 7fe95c200700 1 --2- 172.21.15.62:0/3648198479 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe93c06ecc0 0x7fe93c071170 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fe948006010 tx=0x7fe94801a040 comp rx=0 tx=0).stop 2024-04-20T22:00:07.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:07.894+0000 7fe95c200700 1 -- 172.21.15.62:0/3648198479 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe954105230 msgr2=0x7fe954198520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:00:07.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:07.894+0000 7fe95c200700 1 --2- 172.21.15.62:0/3648198479 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe954105230 0x7fe954198520 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fe94000d8d0 tx=0x7fe94000dc90 comp rx=0 tx=0).stop 2024-04-20T22:00:07.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:07.894+0000 7fe95c200700 1 -- 172.21.15.62:0/3648198479 shutdown_connections 2024-04-20T22:00:07.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:07.894+0000 7fe95c200700 1 --2- 172.21.15.62:0/3648198479 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe93c06ecc0 0x7fe93c071170 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:00:07.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:07.894+0000 7fe95c200700 1 --2- 172.21.15.62:0/3648198479 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe954068540 0x7fe954197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:00:07.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:07.894+0000 7fe95c200700 1 --2- 172.21.15.62:0/3648198479 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe954105230 0x7fe954198520 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:00:07.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:07.894+0000 7fe95c200700 1 -- 172.21.15.62:0/3648198479 >> 172.21.15.62:0/3648198479 conn(0x7fe954075130 msgr2=0x7fe9540fda40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:00:07.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:07.895+0000 7fe95c200700 1 -- 172.21.15.62:0/3648198479 shutdown_connections 2024-04-20T22:00:07.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:07.895+0000 7fe95c200700 1 -- 172.21.15.62:0/3648198479 wait complete. 2024-04-20T22:00:07.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:00:07 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/3438676474' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:00:08.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:00:08 smithi062 ceph-mon[76322]: pgmap v746: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:00:08.909 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:00:08 smithi062 ceph-mon[76322]: from='client.35368 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:00:08.909 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:00:08 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/3648198479' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:00:08.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:00:08 smithi189 ceph-mon[57022]: pgmap v746: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:00:08.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:00:08 smithi189 ceph-mon[57022]: from='client.35368 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:00:08.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:00:08 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/3648198479' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:00:10.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:00:10 smithi189 ceph-mon[57022]: pgmap v747: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:00:11.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:00:10 smithi062 ceph-mon[76322]: pgmap v747: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:00:12.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:00:12 smithi189 ceph-mon[57022]: pgmap v748: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:00:13.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:00:12 smithi062 ceph-mon[76322]: pgmap v748: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:00:14.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:00:14 smithi189 ceph-mon[57022]: pgmap v749: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:00:15.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:00:14 smithi062 ceph-mon[76322]: pgmap v749: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:00:16.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:00:16 smithi189 ceph-mon[57022]: pgmap v750: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:00:17.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:00:16 smithi062 ceph-mon[76322]: pgmap v750: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:00:18.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:00:18 smithi189 ceph-mon[57022]: pgmap v751: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:00:19.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:00:18 smithi062 ceph-mon[76322]: pgmap v751: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:00:19.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:00:19 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:00:20.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:00:19 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:00:20.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:00:20 smithi189 ceph-mon[57022]: pgmap v752: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:00:21.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:00:20 smithi062 ceph-mon[76322]: pgmap v752: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:00:22.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:00:22 smithi062 ceph-mon[76322]: pgmap v753: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:00:22.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:00:22 smithi189 ceph-mon[57022]: pgmap v753: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:00:24.881 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:00:24 smithi189 ceph-mon[57022]: pgmap v754: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:00:25.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:00:24 smithi062 ceph-mon[76322]: pgmap v754: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:00:26.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:00:26 smithi189 ceph-mon[57022]: pgmap v755: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:00:27.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:00:26 smithi062 ceph-mon[76322]: pgmap v755: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:00:28.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:00:28 smithi189 ceph-mon[57022]: pgmap v756: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:00:29.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:00:28 smithi062 ceph-mon[76322]: pgmap v756: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:00:30.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:00:30 smithi189 ceph-mon[57022]: pgmap v757: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:00:31.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:00:30 smithi062 ceph-mon[76322]: pgmap v757: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:00:32.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:00:32 smithi189 ceph-mon[57022]: pgmap v758: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:00:33.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:00:32 smithi062 ceph-mon[76322]: pgmap v758: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:00:34.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:00:34 smithi189 ceph-mon[57022]: pgmap v759: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:00:34.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:00:34 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:00:35.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:00:34 smithi062 ceph-mon[76322]: pgmap v759: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:00:35.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:00:34 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:00:36.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:00:36 smithi189 ceph-mon[57022]: pgmap v760: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:00:37.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:00:36 smithi062 ceph-mon[76322]: pgmap v760: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:00:38.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.074+0000 7f0cb9e00700 1 -- 172.21.15.62:0/3729638950 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0cb4102470 msgr2=0x7f0cb41028e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:00:38.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.074+0000 7f0cb9e00700 1 --2- 172.21.15.62:0/3729638950 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0cb4102470 0x7f0cb41028e0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f0ca0009910 tx=0x7f0ca0009c20 comp rx=0 tx=0).stop 2024-04-20T22:00:38.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.074+0000 7f0cb9e00700 1 -- 172.21.15.62:0/3729638950 shutdown_connections 2024-04-20T22:00:38.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.074+0000 7f0cb9e00700 1 --2- 172.21.15.62:0/3729638950 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0cb4102470 0x7f0cb41028e0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:00:38.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.074+0000 7f0cb9e00700 1 --2- 172.21.15.62:0/3729638950 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0cb4108470 0x7f0cb4108840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:00:38.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.074+0000 7f0cb9e00700 1 -- 172.21.15.62:0/3729638950 >> 172.21.15.62:0/3729638950 conn(0x7f0cb40fdf90 msgr2=0x7f0cb41003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:00:38.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.074+0000 7f0cb9e00700 1 -- 172.21.15.62:0/3729638950 shutdown_connections 2024-04-20T22:00:38.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.074+0000 7f0cb9e00700 1 -- 172.21.15.62:0/3729638950 wait complete. 2024-04-20T22:00:38.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.076+0000 7f0cb9e00700 1 Processor -- start 2024-04-20T22:00:38.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.076+0000 7f0cb9e00700 1 -- start start 2024-04-20T22:00:38.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.076+0000 7f0cb9e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0cb4102470 0x7f0cb4198270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:00:38.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.077+0000 7f0cb9e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0cb4108470 0x7f0cb41987b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:00:38.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.077+0000 7f0cb9e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb4198e90 con 0x7f0cb4108470 2024-04-20T22:00:38.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.077+0000 7f0cb9e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb419cbd0 con 0x7f0cb4102470 2024-04-20T22:00:38.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.077+0000 7f0cb2000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0cb4108470 0x7f0cb41987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:00:38.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.077+0000 7f0cb2000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0cb4108470 0x7f0cb41987b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:38442/0 (socket says 172.21.15.62:38442) 2024-04-20T22:00:38.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.077+0000 7f0cb2000700 1 -- 172.21.15.62:0/422320200 learned_addr learned my addr 172.21.15.62:0/422320200 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:00:38.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.077+0000 7f0cb2a00700 1 --2- 172.21.15.62:0/422320200 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0cb4102470 0x7f0cb4198270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:00:38.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.077+0000 7f0cb2000700 1 -- 172.21.15.62:0/422320200 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0cb4102470 msgr2=0x7f0cb4198270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:00:38.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.077+0000 7f0cb2000700 1 --2- 172.21.15.62:0/422320200 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0cb4102470 0x7f0cb4198270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:00:38.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.077+0000 7f0cb2000700 1 -- 172.21.15.62:0/422320200 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ca00095f0 con 0x7f0cb4108470 2024-04-20T22:00:38.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.078+0000 7f0cb2000700 1 --2- 172.21.15.62:0/422320200 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0cb4108470 0x7f0cb41987b0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f0ca00092e0 tx=0x7f0ca000f460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:00:38.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.078+0000 7f0cb8a00700 1 -- 172.21.15.62:0/422320200 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0ca001d070 con 0x7f0cb4108470 2024-04-20T22:00:38.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.078+0000 7f0cb8a00700 1 -- 172.21.15.62:0/422320200 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0ca00052a0 con 0x7f0cb4108470 2024-04-20T22:00:38.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.078+0000 7f0cb9e00700 1 -- 172.21.15.62:0/422320200 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cb419ce50 con 0x7f0cb4108470 2024-04-20T22:00:38.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.078+0000 7f0cb8a00700 1 -- 172.21.15.62:0/422320200 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0ca0017630 con 0x7f0cb4108470 2024-04-20T22:00:38.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.078+0000 7f0cb9e00700 1 -- 172.21.15.62:0/422320200 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cb419d340 con 0x7f0cb4108470 2024-04-20T22:00:38.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.080+0000 7f0cb9e00700 1 -- 172.21.15.62:0/422320200 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cb410a970 con 0x7f0cb4108470 2024-04-20T22:00:38.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.081+0000 7f0cb8a00700 1 -- 172.21.15.62:0/422320200 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f0ca00178a0 con 0x7f0cb4108470 2024-04-20T22:00:38.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.082+0000 7f0cb8a00700 1 --2- 172.21.15.62:0/422320200 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f0c9806ecc0 0x7f0c98071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:00:38.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.083+0000 7f0cb8a00700 1 -- 172.21.15.62:0/422320200 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f0ca00901e0 con 0x7f0cb4108470 2024-04-20T22:00:38.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.083+0000 7f0cb2a00700 1 --2- 172.21.15.62:0/422320200 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f0c9806ecc0 0x7f0c98071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:00:38.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.084+0000 7f0cb2a00700 1 --2- 172.21.15.62:0/422320200 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f0c9806ecc0 0x7f0c98071170 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f0ca4009bf0 tx=0x7f0ca4009260 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:00:38.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.087+0000 7f0cb8a00700 1 -- 172.21.15.62:0/422320200 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f0ca0058f70 con 0x7f0cb4108470 2024-04-20T22:00:38.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.345+0000 7f0cb9e00700 1 -- 172.21.15.62:0/422320200 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0cb419d5b0 con 0x7f0c9806ecc0 2024-04-20T22:00:38.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.347+0000 7f0cb8a00700 1 -- 172.21.15.62:0/422320200 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f0cb419d5b0 con 0x7f0c9806ecc0 2024-04-20T22:00:38.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.350+0000 7f0cb9e00700 1 -- 172.21.15.62:0/422320200 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f0c9806ecc0 msgr2=0x7f0c98071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:00:38.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.350+0000 7f0cb9e00700 1 --2- 172.21.15.62:0/422320200 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f0c9806ecc0 0x7f0c98071170 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f0ca4009bf0 tx=0x7f0ca4009260 comp rx=0 tx=0).stop 2024-04-20T22:00:38.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.350+0000 7f0cb9e00700 1 -- 172.21.15.62:0/422320200 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0cb4108470 msgr2=0x7f0cb41987b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:00:38.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.350+0000 7f0cb9e00700 1 --2- 172.21.15.62:0/422320200 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0cb4108470 0x7f0cb41987b0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f0ca00092e0 tx=0x7f0ca000f460 comp rx=0 tx=0).stop 2024-04-20T22:00:38.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.350+0000 7f0cb9e00700 1 -- 172.21.15.62:0/422320200 shutdown_connections 2024-04-20T22:00:38.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.350+0000 7f0cb9e00700 1 --2- 172.21.15.62:0/422320200 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f0c9806ecc0 0x7f0c98071170 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:00:38.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.350+0000 7f0cb9e00700 1 --2- 172.21.15.62:0/422320200 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0cb4102470 0x7f0cb4198270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:00:38.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.350+0000 7f0cb9e00700 1 --2- 172.21.15.62:0/422320200 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0cb4108470 0x7f0cb41987b0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:00:38.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.350+0000 7f0cb9e00700 1 -- 172.21.15.62:0/422320200 >> 172.21.15.62:0/422320200 conn(0x7f0cb40fdf90 msgr2=0x7f0cb40ff470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:00:38.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.351+0000 7f0cb9e00700 1 -- 172.21.15.62:0/422320200 shutdown_connections 2024-04-20T22:00:38.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.351+0000 7f0cb9e00700 1 -- 172.21.15.62:0/422320200 wait complete. 2024-04-20T22:00:38.368 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T22:00:38.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.512+0000 7f1821c00700 1 -- 172.21.15.62:0/2365236932 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f181c10c8f0 msgr2=0x7f181c10ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:00:38.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.512+0000 7f1821c00700 1 --2- 172.21.15.62:0/2365236932 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f181c10c8f0 0x7f181c10ccc0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f180c009910 tx=0x7f180c009c20 comp rx=0 tx=0).stop 2024-04-20T22:00:38.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.512+0000 7f1821c00700 1 -- 172.21.15.62:0/2365236932 shutdown_connections 2024-04-20T22:00:38.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.512+0000 7f1821c00700 1 --2- 172.21.15.62:0/2365236932 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f181c106b80 0x7f181c106ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:00:38.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.512+0000 7f1821c00700 1 --2- 172.21.15.62:0/2365236932 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f181c10c8f0 0x7f181c10ccc0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:00:38.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.512+0000 7f1821c00700 1 -- 172.21.15.62:0/2365236932 >> 172.21.15.62:0/2365236932 conn(0x7f181c076060 msgr2=0x7f181c078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:00:38.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.512+0000 7f1821c00700 1 -- 172.21.15.62:0/2365236932 shutdown_connections 2024-04-20T22:00:38.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.513+0000 7f1821c00700 1 -- 172.21.15.62:0/2365236932 wait complete. 2024-04-20T22:00:38.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.513+0000 7f1821c00700 1 Processor -- start 2024-04-20T22:00:38.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.514+0000 7f1821c00700 1 -- start start 2024-04-20T22:00:38.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.514+0000 7f1821c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f181c106b80 0x7f181c19c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:00:38.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.514+0000 7f1821c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f181c10c8f0 0x7f181c19ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:00:38.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.514+0000 7f1821c00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f181c19d120 con 0x7f181c106b80 2024-04-20T22:00:38.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.514+0000 7f1821c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f181c1a0eb0 con 0x7f181c10c8f0 2024-04-20T22:00:38.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.514+0000 7f181aa00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f181c106b80 0x7f181c19c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:00:38.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.514+0000 7f181aa00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f181c106b80 0x7f181c19c500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:38452/0 (socket says 172.21.15.62:38452) 2024-04-20T22:00:38.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.514+0000 7f181aa00700 1 -- 172.21.15.62:0/1349004965 learned_addr learned my addr 172.21.15.62:0/1349004965 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:00:38.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.515+0000 7f181aa00700 1 -- 172.21.15.62:0/1349004965 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f181c10c8f0 msgr2=0x7f181c19ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:00:38.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.515+0000 7f181aa00700 1 --2- 172.21.15.62:0/1349004965 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f181c10c8f0 0x7f181c19ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:00:38.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.515+0000 7f181aa00700 1 -- 172.21.15.62:0/1349004965 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f180c0095f0 con 0x7f181c106b80 2024-04-20T22:00:38.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.515+0000 7f181aa00700 1 --2- 172.21.15.62:0/1349004965 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f181c106b80 0x7f181c19c500 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f180c00b5c0 tx=0x7f180c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:00:38.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.515+0000 7f1813e00700 1 -- 172.21.15.62:0/1349004965 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f180c01d070 con 0x7f181c106b80 2024-04-20T22:00:38.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.515+0000 7f1813e00700 1 -- 172.21.15.62:0/1349004965 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f180c005e00 con 0x7f181c106b80 2024-04-20T22:00:38.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.515+0000 7f1821c00700 1 -- 172.21.15.62:0/1349004965 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f181c1a1130 con 0x7f181c106b80 2024-04-20T22:00:38.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.516+0000 7f1813e00700 1 -- 172.21.15.62:0/1349004965 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f180c017400 con 0x7f181c106b80 2024-04-20T22:00:38.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.516+0000 7f1821c00700 1 -- 172.21.15.62:0/1349004965 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f181c1a1620 con 0x7f181c106b80 2024-04-20T22:00:38.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.517+0000 7f1821c00700 1 -- 172.21.15.62:0/1349004965 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f181c109b20 con 0x7f181c106b80 2024-04-20T22:00:38.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.518+0000 7f1813e00700 1 -- 172.21.15.62:0/1349004965 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f180c00f460 con 0x7f181c106b80 2024-04-20T22:00:38.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.519+0000 7f1813e00700 1 --2- 172.21.15.62:0/1349004965 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f180006ecc0 0x7f1800071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:00:38.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.519+0000 7f1813e00700 1 -- 172.21.15.62:0/1349004965 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f180c090460 con 0x7f181c106b80 2024-04-20T22:00:38.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.519+0000 7f1812000700 1 --2- 172.21.15.62:0/1349004965 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f180006ecc0 0x7f1800071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:00:38.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.520+0000 7f1812000700 1 --2- 172.21.15.62:0/1349004965 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f180006ecc0 0x7f1800071170 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f1804005950 tx=0x7f18040058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:00:38.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.524+0000 7f1813e00700 1 -- 172.21.15.62:0/1349004965 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f180c059170 con 0x7f181c106b80 2024-04-20T22:00:38.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.789+0000 7f1821c00700 1 -- 172.21.15.62:0/1349004965 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f181c1a1840 con 0x7f180006ecc0 2024-04-20T22:00:38.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.791+0000 7f1813e00700 1 -- 172.21.15.62:0/1349004965 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f181c1a1840 con 0x7f180006ecc0 2024-04-20T22:00:38.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.794+0000 7f1821c00700 1 -- 172.21.15.62:0/1349004965 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f180006ecc0 msgr2=0x7f1800071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:00:38.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.794+0000 7f1821c00700 1 --2- 172.21.15.62:0/1349004965 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f180006ecc0 0x7f1800071170 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f1804005950 tx=0x7f18040058e0 comp rx=0 tx=0).stop 2024-04-20T22:00:38.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.794+0000 7f1821c00700 1 -- 172.21.15.62:0/1349004965 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f181c106b80 msgr2=0x7f181c19c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:00:38.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.794+0000 7f1821c00700 1 --2- 172.21.15.62:0/1349004965 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f181c106b80 0x7f181c19c500 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f180c00b5c0 tx=0x7f180c00be20 comp rx=0 tx=0).stop 2024-04-20T22:00:38.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.795+0000 7f1821c00700 1 -- 172.21.15.62:0/1349004965 shutdown_connections 2024-04-20T22:00:38.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.795+0000 7f1821c00700 1 --2- 172.21.15.62:0/1349004965 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f180006ecc0 0x7f1800071170 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:00:38.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.795+0000 7f1821c00700 1 --2- 172.21.15.62:0/1349004965 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f181c106b80 0x7f181c19c500 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:00:38.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.795+0000 7f1821c00700 1 --2- 172.21.15.62:0/1349004965 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f181c10c8f0 0x7f181c19ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:00:38.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.795+0000 7f1821c00700 1 -- 172.21.15.62:0/1349004965 >> 172.21.15.62:0/1349004965 conn(0x7f181c076060 msgr2=0x7f181c077aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:00:38.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.795+0000 7f1821c00700 1 -- 172.21.15.62:0/1349004965 shutdown_connections 2024-04-20T22:00:38.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.795+0000 7f1821c00700 1 -- 172.21.15.62:0/1349004965 wait complete. 2024-04-20T22:00:38.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.962+0000 7ff1fa600700 1 -- 172.21.15.62:0/3541922444 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1f4108470 msgr2=0x7ff1f4108840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:00:38.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.962+0000 7ff1fa600700 1 --2- 172.21.15.62:0/3541922444 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1f4108470 0x7ff1f4108840 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7ff1e4009910 tx=0x7ff1e4009c20 comp rx=0 tx=0).stop 2024-04-20T22:00:38.962 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:00:38 smithi062 ceph-mon[76322]: pgmap v761: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:00:38.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.962+0000 7ff1fa600700 1 -- 172.21.15.62:0/3541922444 shutdown_connections 2024-04-20T22:00:38.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.962+0000 7ff1fa600700 1 --2- 172.21.15.62:0/3541922444 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1f4102470 0x7ff1f41028e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:00:38.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.962+0000 7ff1fa600700 1 --2- 172.21.15.62:0/3541922444 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1f4108470 0x7ff1f4108840 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:00:38.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.962+0000 7ff1fa600700 1 -- 172.21.15.62:0/3541922444 >> 172.21.15.62:0/3541922444 conn(0x7ff1f40fdf90 msgr2=0x7ff1f41003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:00:38.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.962+0000 7ff1fa600700 1 -- 172.21.15.62:0/3541922444 shutdown_connections 2024-04-20T22:00:38.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.963+0000 7ff1fa600700 1 -- 172.21.15.62:0/3541922444 wait complete. 2024-04-20T22:00:38.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.964+0000 7ff1fa600700 1 Processor -- start 2024-04-20T22:00:38.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.965+0000 7ff1fa600700 1 -- start start 2024-04-20T22:00:38.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.965+0000 7ff1fa600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1f4102470 0x7ff1f41981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:00:38.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.965+0000 7ff1fa600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1f4108470 0x7ff1f4198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:00:38.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.965+0000 7ff1fa600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1f4198d80 con 0x7ff1f4108470 2024-04-20T22:00:38.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.965+0000 7ff1fa600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1f4198ec0 con 0x7ff1f4102470 2024-04-20T22:00:38.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.965+0000 7ff1f2a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1f4108470 0x7ff1f4198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:00:38.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.965+0000 7ff1f2a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1f4108470 0x7ff1f4198730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:38476/0 (socket says 172.21.15.62:38476) 2024-04-20T22:00:38.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.966+0000 7ff1f2a00700 1 -- 172.21.15.62:0/2527117945 learned_addr learned my addr 172.21.15.62:0/2527117945 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:00:38.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.966+0000 7ff1f3400700 1 --2- 172.21.15.62:0/2527117945 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1f4102470 0x7ff1f41981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:00:38.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.966+0000 7ff1f2a00700 1 -- 172.21.15.62:0/2527117945 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1f4102470 msgr2=0x7ff1f41981f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:00:38.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.966+0000 7ff1f2a00700 1 --2- 172.21.15.62:0/2527117945 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1f4102470 0x7ff1f41981f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:00:38.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.966+0000 7ff1f2a00700 1 -- 172.21.15.62:0/2527117945 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1e40095f0 con 0x7ff1f4108470 2024-04-20T22:00:38.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.966+0000 7ff1f2a00700 1 --2- 172.21.15.62:0/2527117945 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1f4108470 0x7ff1f4198730 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7ff1e000c930 tx=0x7ff1e000ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:00:38.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.967+0000 7ff1ebe00700 1 -- 172.21.15.62:0/2527117945 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff1e0007ad0 con 0x7ff1f4108470 2024-04-20T22:00:38.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.967+0000 7ff1ebe00700 1 -- 172.21.15.62:0/2527117945 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff1e0007c30 con 0x7ff1f4108470 2024-04-20T22:00:38.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.967+0000 7ff1ebe00700 1 -- 172.21.15.62:0/2527117945 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff1e0018610 con 0x7ff1f4108470 2024-04-20T22:00:38.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.967+0000 7ff1fa600700 1 -- 172.21.15.62:0/2527117945 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1f419cd10 con 0x7ff1f4108470 2024-04-20T22:00:38.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.967+0000 7ff1fa600700 1 -- 172.21.15.62:0/2527117945 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1f419d260 con 0x7ff1f4108470 2024-04-20T22:00:38.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.968+0000 7ff1fa600700 1 -- 172.21.15.62:0/2527117945 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1f410a9e0 con 0x7ff1f4108470 2024-04-20T22:00:38.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.969+0000 7ff1ebe00700 1 -- 172.21.15.62:0/2527117945 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff1e000f450 con 0x7ff1f4108470 2024-04-20T22:00:38.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.970+0000 7ff1ebe00700 1 --2- 172.21.15.62:0/2527117945 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff1d806ecc0 0x7ff1d8071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:00:38.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.970+0000 7ff1ebe00700 1 -- 172.21.15.62:0/2527117945 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7ff1e008eff0 con 0x7ff1f4108470 2024-04-20T22:00:38.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.971+0000 7ff1f3400700 1 --2- 172.21.15.62:0/2527117945 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff1d806ecc0 0x7ff1d8071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:00:38.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.971+0000 7ff1f3400700 1 --2- 172.21.15.62:0/2527117945 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff1d806ecc0 0x7ff1d8071170 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7ff1e4006010 tx=0x7ff1e401a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:00:38.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:38.975+0000 7ff1ebe00700 1 -- 172.21.15.62:0/2527117945 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7ff1e0057de0 con 0x7ff1f4108470 2024-04-20T22:00:38.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:00:38 smithi189 ceph-mon[57022]: pgmap v761: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:00:39.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:39.223+0000 7ff1fa600700 1 -- 172.21.15.62:0/2527117945 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7ff1f41995c0 con 0x7ff1d806ecc0 2024-04-20T22:00:39.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:39.242+0000 7ff1ebe00700 1 -- 172.21.15.62:0/2527117945 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7ff1f41995c0 con 0x7ff1d806ecc0 2024-04-20T22:00:39.242 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T22:00:39.242 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (26m) 2m ago 40m 15.3M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T22:00:39.242 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (40m) 2m ago 40m 10.4M - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T22:00:39.242 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (39m) 2m ago 39m 10.1M - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T22:00:39.242 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (22m) 2m ago 40m 6660k - 19.0.0-3133-ga8c84f59 76bc6b91b726 2825d5ba11c6 2024-04-20T22:00:39.242 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (22m) 2m ago 39m 6665k - 19.0.0-3133-ga8c84f59 76bc6b91b726 20f746fa1bba 2024-04-20T22:00:39.243 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (25m) 2m ago 39m 73.5M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T22:00:39.243 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (17m) 2m ago 33m 18.5M - 19.0.0-3133-ga8c84f59 76bc6b91b726 de80bc90a683 2024-04-20T22:00:39.243 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (18m) 2m ago 33m 18.7M - 19.0.0-3133-ga8c84f59 76bc6b91b726 cd793b7ee98e 2024-04-20T22:00:39.243 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (2m) 2m ago 33m 11.0M - 19.0.0-3133-ga8c84f59 76bc6b91b726 0e7b0bc4d1ab 2024-04-20T22:00:39.243 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (33m) 2m ago 33m 18.6M - 18.2.0 dc2bc1663786 f2eec2ee8951 2024-04-20T22:00:39.243 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (29m) 2m ago 42m 513M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T22:00:39.243 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (28m) 2m ago 39m 459M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T22:00:39.243 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (24m) 2m ago 42m 71.7M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 019eb1479485 2024-04-20T22:00:39.243 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (23m) 2m ago 38m 51.0M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 55aac560aab6 2024-04-20T22:00:39.243 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (27m) 2m ago 40m 8472k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T22:00:39.243 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (27m) 2m ago 39m 8048k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T22:00:39.243 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (21m) 2m ago 37m 91.2M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 ee2bb901f26e 2024-04-20T22:00:39.243 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (21m) 2m ago 37m 94.6M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 9075152d4162 2024-04-20T22:00:39.243 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (20m) 2m ago 36m 78.1M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 2d4f3e38dc0e 2024-04-20T22:00:39.244 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (20m) 2m ago 36m 92.6M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 8f97b17ae0eb 2024-04-20T22:00:39.244 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (19m) 2m ago 35m 95.6M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 653d8079967d 2024-04-20T22:00:39.244 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (18m) 2m ago 35m 89.2M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 596dded14c12 2024-04-20T22:00:39.244 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (27m) 2m ago 39m 82.1M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T22:00:39.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:39.245+0000 7ff1fa600700 1 -- 172.21.15.62:0/2527117945 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff1d806ecc0 msgr2=0x7ff1d8071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:00:39.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:39.245+0000 7ff1fa600700 1 --2- 172.21.15.62:0/2527117945 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff1d806ecc0 0x7ff1d8071170 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7ff1e4006010 tx=0x7ff1e401a040 comp rx=0 tx=0).stop 2024-04-20T22:00:39.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:39.245+0000 7ff1fa600700 1 -- 172.21.15.62:0/2527117945 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1f4108470 msgr2=0x7ff1f4198730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:00:39.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:39.245+0000 7ff1fa600700 1 --2- 172.21.15.62:0/2527117945 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1f4108470 0x7ff1f4198730 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7ff1e000c930 tx=0x7ff1e000ccf0 comp rx=0 tx=0).stop 2024-04-20T22:00:39.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:39.245+0000 7ff1fa600700 1 -- 172.21.15.62:0/2527117945 shutdown_connections 2024-04-20T22:00:39.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:39.245+0000 7ff1fa600700 1 --2- 172.21.15.62:0/2527117945 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff1d806ecc0 0x7ff1d8071170 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:00:39.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:39.245+0000 7ff1fa600700 1 --2- 172.21.15.62:0/2527117945 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1f4102470 0x7ff1f41981f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:00:39.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:39.245+0000 7ff1fa600700 1 --2- 172.21.15.62:0/2527117945 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1f4108470 0x7ff1f4198730 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:00:39.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:39.245+0000 7ff1fa600700 1 -- 172.21.15.62:0/2527117945 >> 172.21.15.62:0/2527117945 conn(0x7ff1f40fdf90 msgr2=0x7ff1f41002c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:00:39.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:39.246+0000 7ff1fa600700 1 -- 172.21.15.62:0/2527117945 shutdown_connections 2024-04-20T22:00:39.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:39.246+0000 7ff1fa600700 1 -- 172.21.15.62:0/2527117945 wait complete. 2024-04-20T22:00:39.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:39.403+0000 7f361f600700 1 -- 172.21.15.62:0/1313266498 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f361810c8f0 msgr2=0x7f361810ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:00:39.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:39.403+0000 7f361f600700 1 --2- 172.21.15.62:0/1313266498 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f361810c8f0 0x7f361810ccc0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f360c009910 tx=0x7f360c009c20 comp rx=0 tx=0).stop 2024-04-20T22:00:39.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:39.404+0000 7f361f600700 1 -- 172.21.15.62:0/1313266498 shutdown_connections 2024-04-20T22:00:39.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:39.404+0000 7f361f600700 1 --2- 172.21.15.62:0/1313266498 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3618106b80 0x7f3618106ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:00:39.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:39.404+0000 7f361f600700 1 --2- 172.21.15.62:0/1313266498 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f361810c8f0 0x7f361810ccc0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:00:39.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:39.404+0000 7f361f600700 1 -- 172.21.15.62:0/1313266498 >> 172.21.15.62:0/1313266498 conn(0x7f3618076060 msgr2=0x7f3618078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:00:39.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:39.404+0000 7f361f600700 1 -- 172.21.15.62:0/1313266498 shutdown_connections 2024-04-20T22:00:39.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:39.404+0000 7f361f600700 1 -- 172.21.15.62:0/1313266498 wait complete. 2024-04-20T22:00:39.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:39.405+0000 7f361f600700 1 Processor -- start 2024-04-20T22:00:39.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:39.405+0000 7f361f600700 1 -- start start 2024-04-20T22:00:39.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:39.405+0000 7f361f600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3618106b80 0x7f361819c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:00:39.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:39.405+0000 7f361f600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f361810c8f0 0x7f361819ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:00:39.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:39.406+0000 7f361f600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f361819d120 con 0x7f3618106b80 2024-04-20T22:00:39.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:39.406+0000 7f361f600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36181a0eb0 con 0x7f361810c8f0 2024-04-20T22:00:39.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:39.406+0000 7f361cc00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3618106b80 0x7f361819c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:00:39.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:39.406+0000 7f361cc00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3618106b80 0x7f361819c500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:38498/0 (socket says 172.21.15.62:38498) 2024-04-20T22:00:39.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:39.406+0000 7f361cc00700 1 -- 172.21.15.62:0/3850824178 learned_addr learned my addr 172.21.15.62:0/3850824178 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:00:39.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:39.406+0000 7f3617e00700 1 --2- 172.21.15.62:0/3850824178 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f361810c8f0 0x7f361819ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:00:39.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:39.406+0000 7f361cc00700 1 -- 172.21.15.62:0/3850824178 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f361810c8f0 msgr2=0x7f361819ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:00:39.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:39.406+0000 7f361cc00700 1 --2- 172.21.15.62:0/3850824178 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f361810c8f0 0x7f361819ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:00:39.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:39.406+0000 7f361cc00700 1 -- 172.21.15.62:0/3850824178 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f360c0095f0 con 0x7f3618106b80 2024-04-20T22:00:39.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:39.406+0000 7f3617e00700 1 --2- 172.21.15.62:0/3850824178 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f361810c8f0 0x7f361819ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T22:00:39.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:39.407+0000 7f361cc00700 1 --2- 172.21.15.62:0/3850824178 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3618106b80 0x7f361819c500 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f360c00f690 tx=0x7f360c00f770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:00:39.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:39.407+0000 7f3615600700 1 -- 172.21.15.62:0/3850824178 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f360c01d070 con 0x7f3618106b80 2024-04-20T22:00:39.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:39.407+0000 7f3615600700 1 -- 172.21.15.62:0/3850824178 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f360c0050d0 con 0x7f3618106b80 2024-04-20T22:00:39.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:39.407+0000 7f3615600700 1 -- 172.21.15.62:0/3850824178 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f360c022470 con 0x7f3618106b80 2024-04-20T22:00:39.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:39.407+0000 7f361f600700 1 -- 172.21.15.62:0/3850824178 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36181a1130 con 0x7f3618106b80 2024-04-20T22:00:39.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:39.407+0000 7f361f600700 1 -- 172.21.15.62:0/3850824178 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36181a1620 con 0x7f3618106b80 2024-04-20T22:00:39.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:39.408+0000 7f361f600700 1 -- 172.21.15.62:0/3850824178 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3618109b20 con 0x7f3618106b80 2024-04-20T22:00:39.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:39.410+0000 7f3615600700 1 -- 172.21.15.62:0/3850824178 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f360c003680 con 0x7f3618106b80 2024-04-20T22:00:39.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:39.410+0000 7f3615600700 1 --2- 172.21.15.62:0/3850824178 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f360006ecc0 0x7f3600071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:00:39.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:39.411+0000 7f3615600700 1 -- 172.21.15.62:0/3850824178 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f360c0902e0 con 0x7f3618106b80 2024-04-20T22:00:39.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:39.411+0000 7f3617e00700 1 --2- 172.21.15.62:0/3850824178 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f360006ecc0 0x7f3600071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:00:39.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:39.412+0000 7f3617e00700 1 --2- 172.21.15.62:0/3850824178 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f360006ecc0 0x7f3600071170 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f361819db20 tx=0x7f3604009260 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:00:39.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:39.416+0000 7f3615600700 1 -- 172.21.15.62:0/3850824178 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f360c059070 con 0x7f3618106b80 2024-04-20T22:00:39.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:39.735+0000 7f361f600700 1 -- 172.21.15.62:0/3850824178 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f361819d960 con 0x7f3618106b80 2024-04-20T22:00:39.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:39.736+0000 7f3615600700 1 -- 172.21.15.62:0/3850824178 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7f360c05c730 con 0x7f3618106b80 2024-04-20T22:00:39.736 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:00:39.736 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T22:00:39.736 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:00:39.737 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:00:39.737 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T22:00:39.737 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:00:39.737 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:00:39.737 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T22:00:39.737 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 6 2024-04-20T22:00:39.737 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:00:39.737 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T22:00:39.737 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 1, 2024-04-20T22:00:39.737 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 3 2024-04-20T22:00:39.737 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:00:39.737 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T22:00:39.737 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 1, 2024-04-20T22:00:39.737 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 13 2024-04-20T22:00:39.738 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T22:00:39.738 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:00:39.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:39.739+0000 7f361f600700 1 -- 172.21.15.62:0/3850824178 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f360006ecc0 msgr2=0x7f3600071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:00:39.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:39.739+0000 7f361f600700 1 --2- 172.21.15.62:0/3850824178 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f360006ecc0 0x7f3600071170 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f361819db20 tx=0x7f3604009260 comp rx=0 tx=0).stop 2024-04-20T22:00:39.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:39.739+0000 7f361f600700 1 -- 172.21.15.62:0/3850824178 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3618106b80 msgr2=0x7f361819c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:00:39.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:39.739+0000 7f361f600700 1 --2- 172.21.15.62:0/3850824178 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3618106b80 0x7f361819c500 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f360c00f690 tx=0x7f360c00f770 comp rx=0 tx=0).stop 2024-04-20T22:00:39.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:39.739+0000 7f361f600700 1 -- 172.21.15.62:0/3850824178 shutdown_connections 2024-04-20T22:00:39.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:39.740+0000 7f361f600700 1 --2- 172.21.15.62:0/3850824178 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f360006ecc0 0x7f3600071170 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:00:39.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:39.740+0000 7f361f600700 1 --2- 172.21.15.62:0/3850824178 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3618106b80 0x7f361819c500 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:00:39.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:39.740+0000 7f361f600700 1 --2- 172.21.15.62:0/3850824178 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f361810c8f0 0x7f361819ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:00:39.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:39.740+0000 7f361f600700 1 -- 172.21.15.62:0/3850824178 >> 172.21.15.62:0/3850824178 conn(0x7f3618076060 msgr2=0x7f3618077aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:00:39.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:39.740+0000 7f361f600700 1 -- 172.21.15.62:0/3850824178 shutdown_connections 2024-04-20T22:00:39.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:39.740+0000 7f361f600700 1 -- 172.21.15.62:0/3850824178 wait complete. 2024-04-20T22:00:39.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:39.887+0000 7f1e88e00700 1 -- 172.21.15.62:0/82583895 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1e8410b640 msgr2=0x7f1e8410ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:00:39.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:39.887+0000 7f1e88e00700 1 --2- 172.21.15.62:0/82583895 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1e8410b640 0x7f1e8410ba10 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f1e74009910 tx=0x7f1e74009c20 comp rx=0 tx=0).stop 2024-04-20T22:00:39.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:39.888+0000 7f1e88e00700 1 -- 172.21.15.62:0/82583895 shutdown_connections 2024-04-20T22:00:39.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:39.888+0000 7f1e88e00700 1 --2- 172.21.15.62:0/82583895 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1e84106b80 0x7f1e84106ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:00:39.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:39.888+0000 7f1e88e00700 1 --2- 172.21.15.62:0/82583895 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1e8410b640 0x7f1e8410ba10 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:00:39.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:39.888+0000 7f1e88e00700 1 -- 172.21.15.62:0/82583895 >> 172.21.15.62:0/82583895 conn(0x7f1e84075e90 msgr2=0x7f1e840782a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:00:39.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:39.888+0000 7f1e88e00700 1 -- 172.21.15.62:0/82583895 shutdown_connections 2024-04-20T22:00:39.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:39.888+0000 7f1e88e00700 1 -- 172.21.15.62:0/82583895 wait complete. 2024-04-20T22:00:39.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:39.889+0000 7f1e88e00700 1 Processor -- start 2024-04-20T22:00:39.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:39.889+0000 7f1e88e00700 1 -- start start 2024-04-20T22:00:39.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:39.889+0000 7f1e88e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1e84106b80 0x7f1e8419c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:00:39.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:39.889+0000 7f1e88e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1e8410b640 0x7f1e8419c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:00:39.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:39.889+0000 7f1e88e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e8419cec0 con 0x7f1e84106b80 2024-04-20T22:00:39.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:39.889+0000 7f1e88e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e841a0c50 con 0x7f1e8410b640 2024-04-20T22:00:39.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:39.890+0000 7f1e83400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1e84106b80 0x7f1e8419c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:00:39.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:39.890+0000 7f1e83400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1e84106b80 0x7f1e8419c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:38518/0 (socket says 172.21.15.62:38518) 2024-04-20T22:00:39.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:39.890+0000 7f1e83400700 1 -- 172.21.15.62:0/1742831550 learned_addr learned my addr 172.21.15.62:0/1742831550 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:00:39.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:39.890+0000 7f1e83400700 1 -- 172.21.15.62:0/1742831550 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1e8410b640 msgr2=0x7f1e8419c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:00:39.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:39.890+0000 7f1e83400700 1 --2- 172.21.15.62:0/1742831550 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1e8410b640 0x7f1e8419c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:00:39.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:39.890+0000 7f1e83400700 1 -- 172.21.15.62:0/1742831550 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e740095f0 con 0x7f1e84106b80 2024-04-20T22:00:39.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:39.890+0000 7f1e83400700 1 --2- 172.21.15.62:0/1742831550 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1e84106b80 0x7f1e8419c2a0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f1e7400b5c0 tx=0x7f1e7400be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:00:39.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:39.891+0000 7f1e80c00700 1 -- 172.21.15.62:0/1742831550 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1e7401d070 con 0x7f1e84106b80 2024-04-20T22:00:39.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:39.891+0000 7f1e80c00700 1 -- 172.21.15.62:0/1742831550 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1e7400f460 con 0x7f1e84106b80 2024-04-20T22:00:39.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:39.891+0000 7f1e88e00700 1 -- 172.21.15.62:0/1742831550 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e841a0ed0 con 0x7f1e84106b80 2024-04-20T22:00:39.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:39.891+0000 7f1e80c00700 1 -- 172.21.15.62:0/1742831550 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1e74017610 con 0x7f1e84106b80 2024-04-20T22:00:39.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:39.891+0000 7f1e88e00700 1 -- 172.21.15.62:0/1742831550 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e841a1440 con 0x7f1e84106b80 2024-04-20T22:00:39.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:39.892+0000 7f1e88e00700 1 -- 172.21.15.62:0/1742831550 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e84062380 con 0x7f1e84106b80 2024-04-20T22:00:39.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:39.893+0000 7f1e80c00700 1 -- 172.21.15.62:0/1742831550 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f1e7400f5d0 con 0x7f1e84106b80 2024-04-20T22:00:39.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:39.894+0000 7f1e80c00700 1 --2- 172.21.15.62:0/1742831550 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f1e6806ecc0 0x7f1e68071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:00:39.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:39.894+0000 7f1e80c00700 1 -- 172.21.15.62:0/1742831550 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f1e74090520 con 0x7f1e84106b80 2024-04-20T22:00:39.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:39.894+0000 7f1e7be00700 1 --2- 172.21.15.62:0/1742831550 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f1e6806ecc0 0x7f1e68071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:00:39.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:39.895+0000 7f1e7be00700 1 --2- 172.21.15.62:0/1742831550 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f1e6806ecc0 0x7f1e68071170 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f1e8419d8c0 tx=0x7f1e6c006d20 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:00:39.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:39.899+0000 7f1e80c00700 1 -- 172.21.15.62:0/1742831550 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f1e74059230 con 0x7f1e84106b80 2024-04-20T22:00:39.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:00:39 smithi189 ceph-mon[57022]: from='client.35376 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:00:39.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:00:39 smithi189 ceph-mon[57022]: from='client.35380 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:00:40.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:00:39 smithi062 ceph-mon[76322]: from='client.35376 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:00:40.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:00:39 smithi062 ceph-mon[76322]: from='client.35380 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:00:40.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:40.169+0000 7f1e88e00700 1 -- 172.21.15.62:0/1742831550 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f1e841a1850 con 0x7f1e84106b80 2024-04-20T22:00:40.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:40.169+0000 7f1e80c00700 1 -- 172.21.15.62:0/1742831550 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 27 v27) v1 ==== 76+0+1888 (secure 0 0 0) 0x7f1e7405c8f0 con 0x7f1e84106b80 2024-04-20T22:00:40.170 INFO:teuthology.orchestra.run.smithi062.stdout:e27 2024-04-20T22:00:40.170 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T22:00:40.170 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:00:40.170 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T22:00:40.170 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:00:40.171 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T22:00:40.171 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T22:00:40.171 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 20 2024-04-20T22:00:40.171 INFO:teuthology.orchestra.run.smithi062.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-20T22:00:40.171 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T22:00:40.171 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T21:42:01.559061+0000 2024-04-20T22:00:40.171 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T22:00:40.171 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T22:00:40.171 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T22:00:40.171 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T22:00:40.171 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T22:00:40.171 INFO:teuthology.orchestra.run.smithi062.stdout:max_xattr_size 65536 2024-04-20T22:00:40.171 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T22:00:40.171 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T22:00:40.172 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 81 2024-04-20T22:00:40.172 INFO:teuthology.orchestra.run.smithi062.stdout:snap_rstat 1 2024-04-20T22:00:40.172 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:00:40.172 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 2 2024-04-20T22:00:40.172 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1 2024-04-20T22:00:40.172 INFO:teuthology.orchestra.run.smithi062.stdout:up {} 2024-04-20T22:00:40.172 INFO:teuthology.orchestra.run.smithi062.stdout:failed 0,1 2024-04-20T22:00:40.172 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T22:00:40.172 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T22:00:40.172 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T22:00:40.172 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T22:00:40.172 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data enabled 2024-04-20T22:00:40.172 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T22:00:40.172 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T22:00:40.172 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 1 2024-04-20T22:00:40.173 INFO:teuthology.orchestra.run.smithi062.stdout:qdb_cluster leader: 0 members: 2024-04-20T22:00:40.173 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:00:40.173 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:00:40.173 INFO:teuthology.orchestra.run.smithi062.stdout:Standby daemons: 2024-04-20T22:00:40.173 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:00:40.173 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ryqblb{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6826/4113916205,v1:172.21.15.62:6827/4113916205] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:00:40.173 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ozizwi{-1:34476} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6828/3801975165,v1:172.21.15.62:6829/3801975165] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:00:40.173 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.xbgspw{-1:44285} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6826/3996040526,v1:172.21.15.189:6827/3996040526] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T22:00:40.173 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.dvimer{-1:44727} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6824/3576699232,v1:172.21.15.189:6825/3576699232] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:00:40.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:40.174+0000 7f1e88e00700 1 -- 172.21.15.62:0/1742831550 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f1e6806ecc0 msgr2=0x7f1e68071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:00:40.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:40.174+0000 7f1e88e00700 1 --2- 172.21.15.62:0/1742831550 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f1e6806ecc0 0x7f1e68071170 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f1e8419d8c0 tx=0x7f1e6c006d20 comp rx=0 tx=0).stop 2024-04-20T22:00:40.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:40.174+0000 7f1e88e00700 1 -- 172.21.15.62:0/1742831550 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1e84106b80 msgr2=0x7f1e8419c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:00:40.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:40.174+0000 7f1e88e00700 1 --2- 172.21.15.62:0/1742831550 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1e84106b80 0x7f1e8419c2a0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f1e7400b5c0 tx=0x7f1e7400be00 comp rx=0 tx=0).stop 2024-04-20T22:00:40.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:40.174+0000 7f1e88e00700 1 -- 172.21.15.62:0/1742831550 shutdown_connections 2024-04-20T22:00:40.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:40.174+0000 7f1e88e00700 1 --2- 172.21.15.62:0/1742831550 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f1e6806ecc0 0x7f1e68071170 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:00:40.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:40.174+0000 7f1e88e00700 1 --2- 172.21.15.62:0/1742831550 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1e84106b80 0x7f1e8419c2a0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:00:40.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:40.174+0000 7f1e88e00700 1 --2- 172.21.15.62:0/1742831550 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1e8410b640 0x7f1e8419c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:00:40.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:40.174+0000 7f1e88e00700 1 -- 172.21.15.62:0/1742831550 >> 172.21.15.62:0/1742831550 conn(0x7f1e84075e90 msgr2=0x7f1e840767f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:00:40.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:40.174+0000 7f1e88e00700 1 -- 172.21.15.62:0/1742831550 shutdown_connections 2024-04-20T22:00:40.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:40.174+0000 7f1e88e00700 1 -- 172.21.15.62:0/1742831550 wait complete. 2024-04-20T22:00:40.175 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 27 2024-04-20T22:00:40.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:40.331+0000 7fe3e6e00700 1 -- 172.21.15.62:0/918059158 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe3e0101070 msgr2=0x7fe3e0101440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:00:40.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:40.331+0000 7fe3e6e00700 1 --2- 172.21.15.62:0/918059158 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe3e0101070 0x7fe3e0101440 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fe3d0009910 tx=0x7fe3d0009c20 comp rx=0 tx=0).stop 2024-04-20T22:00:40.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:40.331+0000 7fe3e6e00700 1 -- 172.21.15.62:0/918059158 shutdown_connections 2024-04-20T22:00:40.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:40.331+0000 7fe3e6e00700 1 --2- 172.21.15.62:0/918059158 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3e0068540 0x7fe3e00689b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:00:40.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:40.331+0000 7fe3e6e00700 1 --2- 172.21.15.62:0/918059158 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe3e0101070 0x7fe3e0101440 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:00:40.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:40.331+0000 7fe3e6e00700 1 -- 172.21.15.62:0/918059158 >> 172.21.15.62:0/918059158 conn(0x7fe3e0075130 msgr2=0x7fe3e0075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:00:40.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:40.331+0000 7fe3e6e00700 1 -- 172.21.15.62:0/918059158 shutdown_connections 2024-04-20T22:00:40.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:40.331+0000 7fe3e6e00700 1 -- 172.21.15.62:0/918059158 wait complete. 2024-04-20T22:00:40.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:40.332+0000 7fe3e6e00700 1 Processor -- start 2024-04-20T22:00:40.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:40.333+0000 7fe3e6e00700 1 -- start start 2024-04-20T22:00:40.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:40.333+0000 7fe3e6e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3e0068540 0x7fe3e0198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:00:40.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:40.333+0000 7fe3e6e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe3e0101070 0x7fe3e0198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:00:40.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:40.333+0000 7fe3e6e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3e0198c70 con 0x7fe3e0101070 2024-04-20T22:00:40.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:40.333+0000 7fe3e6e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3e019ca00 con 0x7fe3e0068540 2024-04-20T22:00:40.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:40.333+0000 7fe3df400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe3e0101070 0x7fe3e0198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:00:40.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:40.333+0000 7fe3df400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe3e0101070 0x7fe3e0198590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:38528/0 (socket says 172.21.15.62:38528) 2024-04-20T22:00:40.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:40.333+0000 7fe3df400700 1 -- 172.21.15.62:0/2783603537 learned_addr learned my addr 172.21.15.62:0/2783603537 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:00:40.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:40.334+0000 7fe3dfe00700 1 --2- 172.21.15.62:0/2783603537 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3e0068540 0x7fe3e0198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:00:40.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:40.334+0000 7fe3df400700 1 -- 172.21.15.62:0/2783603537 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3e0068540 msgr2=0x7fe3e0198050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:00:40.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:40.334+0000 7fe3df400700 1 --2- 172.21.15.62:0/2783603537 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3e0068540 0x7fe3e0198050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:00:40.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:40.334+0000 7fe3df400700 1 -- 172.21.15.62:0/2783603537 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3d00095f0 con 0x7fe3e0101070 2024-04-20T22:00:40.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:40.334+0000 7fe3df400700 1 --2- 172.21.15.62:0/2783603537 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe3e0101070 0x7fe3e0198590 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fe3cc00c8a0 tx=0x7fe3cc00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:00:40.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:40.334+0000 7fe3dcc00700 1 -- 172.21.15.62:0/2783603537 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe3cc00cea0 con 0x7fe3e0101070 2024-04-20T22:00:40.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:40.334+0000 7fe3dcc00700 1 -- 172.21.15.62:0/2783603537 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe3cc004830 con 0x7fe3e0101070 2024-04-20T22:00:40.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:40.334+0000 7fe3dcc00700 1 -- 172.21.15.62:0/2783603537 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe3cc0055b0 con 0x7fe3e0101070 2024-04-20T22:00:40.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:40.334+0000 7fe3e6e00700 1 -- 172.21.15.62:0/2783603537 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3e019cce0 con 0x7fe3e0101070 2024-04-20T22:00:40.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:40.334+0000 7fe3e6e00700 1 -- 172.21.15.62:0/2783603537 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3e019d230 con 0x7fe3e0101070 2024-04-20T22:00:40.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:40.336+0000 7fe3e6e00700 1 -- 172.21.15.62:0/2783603537 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3e004f2a0 con 0x7fe3e0101070 2024-04-20T22:00:40.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:40.337+0000 7fe3dcc00700 1 -- 172.21.15.62:0/2783603537 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe3cc0049a0 con 0x7fe3e0101070 2024-04-20T22:00:40.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:40.338+0000 7fe3dcc00700 1 --2- 172.21.15.62:0/2783603537 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe3c406ecc0 0x7fe3c4071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:00:40.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:40.338+0000 7fe3dcc00700 1 -- 172.21.15.62:0/2783603537 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fe3cc08ebf0 con 0x7fe3e0101070 2024-04-20T22:00:40.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:40.338+0000 7fe3dfe00700 1 --2- 172.21.15.62:0/2783603537 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe3c406ecc0 0x7fe3c4071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:00:40.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:40.339+0000 7fe3dfe00700 1 --2- 172.21.15.62:0/2783603537 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe3c406ecc0 0x7fe3c4071170 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fe3d0000c00 tx=0x7fe3d0005fd0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:00:40.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:40.343+0000 7fe3dcc00700 1 -- 172.21.15.62:0/2783603537 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fe3cc057900 con 0x7fe3e0101070 2024-04-20T22:00:40.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:40.597+0000 7fe3e6e00700 1 -- 172.21.15.62:0/2783603537 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe3e019d5c0 con 0x7fe3c406ecc0 2024-04-20T22:00:40.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:40.601+0000 7fe3dcc00700 1 -- 172.21.15.62:0/2783603537 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fe3e019d5c0 con 0x7fe3c406ecc0 2024-04-20T22:00:40.600 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:00:40.600 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T22:00:40.600 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T22:00:40.600 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-20T22:00:40.601 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T22:00:40.601 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr", 2024-04-20T22:00:40.601 INFO:teuthology.orchestra.run.smithi062.stdout: "crash", 2024-04-20T22:00:40.601 INFO:teuthology.orchestra.run.smithi062.stdout: "mon", 2024-04-20T22:00:40.601 INFO:teuthology.orchestra.run.smithi062.stdout: "osd" 2024-04-20T22:00:40.601 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T22:00:40.601 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "15/23 daemons upgraded", 2024-04-20T22:00:40.601 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading mds daemons", 2024-04-20T22:00:40.601 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T22:00:40.601 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:00:40.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:40.603+0000 7fe3e6e00700 1 -- 172.21.15.62:0/2783603537 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe3c406ecc0 msgr2=0x7fe3c4071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:00:40.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:40.603+0000 7fe3e6e00700 1 --2- 172.21.15.62:0/2783603537 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe3c406ecc0 0x7fe3c4071170 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fe3d0000c00 tx=0x7fe3d0005fd0 comp rx=0 tx=0).stop 2024-04-20T22:00:40.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:40.603+0000 7fe3e6e00700 1 -- 172.21.15.62:0/2783603537 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe3e0101070 msgr2=0x7fe3e0198590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:00:40.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:40.603+0000 7fe3e6e00700 1 --2- 172.21.15.62:0/2783603537 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe3e0101070 0x7fe3e0198590 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fe3cc00c8a0 tx=0x7fe3cc00cc60 comp rx=0 tx=0).stop 2024-04-20T22:00:40.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:40.604+0000 7fe3e6e00700 1 -- 172.21.15.62:0/2783603537 shutdown_connections 2024-04-20T22:00:40.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:40.604+0000 7fe3e6e00700 1 --2- 172.21.15.62:0/2783603537 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe3c406ecc0 0x7fe3c4071170 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:00:40.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:40.604+0000 7fe3e6e00700 1 --2- 172.21.15.62:0/2783603537 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3e0068540 0x7fe3e0198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:00:40.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:40.604+0000 7fe3e6e00700 1 --2- 172.21.15.62:0/2783603537 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe3e0101070 0x7fe3e0198590 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:00:40.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:40.604+0000 7fe3e6e00700 1 -- 172.21.15.62:0/2783603537 >> 172.21.15.62:0/2783603537 conn(0x7fe3e0075130 msgr2=0x7fe3e00fdb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:00:40.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:40.604+0000 7fe3e6e00700 1 -- 172.21.15.62:0/2783603537 shutdown_connections 2024-04-20T22:00:40.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:40.604+0000 7fe3e6e00700 1 -- 172.21.15.62:0/2783603537 wait complete. 2024-04-20T22:00:40.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:40.755+0000 7f06a1000700 1 -- 172.21.15.62:0/641878543 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f069c10c8f0 msgr2=0x7f069c10ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:00:40.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:40.755+0000 7f06a1000700 1 --2- 172.21.15.62:0/641878543 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f069c10c8f0 0x7f069c10ccc0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f068c009910 tx=0x7f068c009c20 comp rx=0 tx=0).stop 2024-04-20T22:00:40.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:40.756+0000 7f06a1000700 1 -- 172.21.15.62:0/641878543 shutdown_connections 2024-04-20T22:00:40.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:40.756+0000 7f06a1000700 1 --2- 172.21.15.62:0/641878543 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f069c106b80 0x7f069c106ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:00:40.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:40.756+0000 7f06a1000700 1 --2- 172.21.15.62:0/641878543 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f069c10c8f0 0x7f069c10ccc0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:00:40.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:40.756+0000 7f06a1000700 1 -- 172.21.15.62:0/641878543 >> 172.21.15.62:0/641878543 conn(0x7f069c076060 msgr2=0x7f069c078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:00:40.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:40.756+0000 7f06a1000700 1 -- 172.21.15.62:0/641878543 shutdown_connections 2024-04-20T22:00:40.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:40.756+0000 7f06a1000700 1 -- 172.21.15.62:0/641878543 wait complete. 2024-04-20T22:00:40.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:40.757+0000 7f06a1000700 1 Processor -- start 2024-04-20T22:00:40.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:40.757+0000 7f06a1000700 1 -- start start 2024-04-20T22:00:40.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:40.757+0000 7f06a1000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f069c106b80 0x7f069c19c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:00:40.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:40.757+0000 7f06a1000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f069c10c8f0 0x7f069c19ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:00:40.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:40.757+0000 7f06a1000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f069c19d120 con 0x7f069c10c8f0 2024-04-20T22:00:40.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:40.757+0000 7f06a1000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f069c1a0eb0 con 0x7f069c106b80 2024-04-20T22:00:40.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:40.758+0000 7f0691400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f069c10c8f0 0x7f069c19ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:00:40.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:40.758+0000 7f0691400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f069c10c8f0 0x7f069c19ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:38558/0 (socket says 172.21.15.62:38558) 2024-04-20T22:00:40.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:40.758+0000 7f0691400700 1 -- 172.21.15.62:0/672920734 learned_addr learned my addr 172.21.15.62:0/672920734 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:00:40.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:40.758+0000 7f0699e00700 1 --2- 172.21.15.62:0/672920734 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f069c106b80 0x7f069c19c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:00:40.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:40.758+0000 7f0691400700 1 -- 172.21.15.62:0/672920734 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f069c106b80 msgr2=0x7f069c19c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:00:40.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:40.758+0000 7f0691400700 1 --2- 172.21.15.62:0/672920734 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f069c106b80 0x7f069c19c500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:00:40.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:40.758+0000 7f0691400700 1 -- 172.21.15.62:0/672920734 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f068c0095f0 con 0x7f069c10c8f0 2024-04-20T22:00:40.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:40.758+0000 7f0691400700 1 --2- 172.21.15.62:0/672920734 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f069c10c8f0 0x7f069c19ca40 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f068400d8d0 tx=0x7f068400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:00:40.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:40.759+0000 7f0693400700 1 -- 172.21.15.62:0/672920734 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0684009940 con 0x7f069c10c8f0 2024-04-20T22:00:40.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:40.759+0000 7f0693400700 1 -- 172.21.15.62:0/672920734 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0684010460 con 0x7f069c10c8f0 2024-04-20T22:00:40.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:40.759+0000 7f0693400700 1 -- 172.21.15.62:0/672920734 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f068400f5f0 con 0x7f069c10c8f0 2024-04-20T22:00:40.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:40.759+0000 7f06a1000700 1 -- 172.21.15.62:0/672920734 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f069c1a1190 con 0x7f069c10c8f0 2024-04-20T22:00:40.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:40.759+0000 7f06a1000700 1 -- 172.21.15.62:0/672920734 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f069c1a16e0 con 0x7f069c10c8f0 2024-04-20T22:00:40.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:40.760+0000 7f06a1000700 1 -- 172.21.15.62:0/672920734 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f069c109b20 con 0x7f069c10c8f0 2024-04-20T22:00:40.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:40.761+0000 7f0693400700 1 -- 172.21.15.62:0/672920734 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f06840105d0 con 0x7f069c10c8f0 2024-04-20T22:00:40.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:40.762+0000 7f0693400700 1 --2- 172.21.15.62:0/672920734 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f068006ecc0 0x7f0680071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:00:40.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:40.762+0000 7f0693400700 1 -- 172.21.15.62:0/672920734 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f068408ee10 con 0x7f069c10c8f0 2024-04-20T22:00:40.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:40.762+0000 7f0699e00700 1 --2- 172.21.15.62:0/672920734 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f068006ecc0 0x7f0680071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:00:40.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:40.763+0000 7f0699e00700 1 --2- 172.21.15.62:0/672920734 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f068006ecc0 0x7f0680071170 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f068c00b5c0 tx=0x7f068c005960 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:00:40.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:40.767+0000 7f0693400700 1 -- 172.21.15.62:0/672920734 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f0684057ba0 con 0x7f069c10c8f0 2024-04-20T22:00:40.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:00:40 smithi189 ceph-mon[57022]: from='client.35384 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:00:40.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:00:40 smithi189 ceph-mon[57022]: pgmap v762: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:00:40.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:00:40 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/3850824178' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:00:40.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:00:40 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/1742831550' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:00:41.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:00:40 smithi062 ceph-mon[76322]: from='client.35384 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:00:41.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:00:40 smithi062 ceph-mon[76322]: pgmap v762: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:00:41.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:00:40 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/3850824178' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:00:41.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:00:40 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/1742831550' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:00:41.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:41.084+0000 7f06a1000700 1 -- 172.21.15.62:0/672920734 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f069c04f2a0 con 0x7f069c10c8f0 2024-04-20T22:00:41.085 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:41.085+0000 7f0693400700 1 -- 172.21.15.62:0/672920734 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7f068405b260 con 0x7f069c10c8f0 2024-04-20T22:00:41.085 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-20T22:00:41.085 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-20T22:00:41.085 INFO:teuthology.orchestra.run.smithi062.stdout: host smithi062 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T22:00:41.085 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-20T22:00:41.086 INFO:teuthology.orchestra.run.smithi062.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi062 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpq7y7oa_r:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:00:41.086 INFO:teuthology.orchestra.run.smithi062.stdout:Traceback (most recent call last): 2024-04-20T22:00:41.086 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-20T22:00:41.086 INFO:teuthology.orchestra.run.smithi062.stdout: return _run_code(code, main_globals, None, 2024-04-20T22:00:41.086 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-20T22:00:41.086 INFO:teuthology.orchestra.run.smithi062.stdout: exec(code, run_globals) 2024-04-20T22:00:41.086 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5618, in 2024-04-20T22:00:41.086 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5606, in main 2024-04-20T22:00:41.086 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 408, in _infer_config 2024-04-20T22:00:41.086 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 323, in _infer_fsid 2024-04-20T22:00:41.086 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 436, in _infer_image 2024-04-20T22:00:41.086 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 310, in _validate_fsid 2024-04-20T22:00:41.086 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 3359, in command_ceph_volume 2024-04-20T22:00:41.087 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-20T22:00:41.087 INFO:teuthology.orchestra.run.smithi062.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi062 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpq7y7oa_r:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:00:41.087 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-20T22:00:41.087 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is degraded 2024-04-20T22:00:41.087 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T22:00:41.087 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T22:00:41.087 INFO:teuthology.orchestra.run.smithi062.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-20T22:00:41.087 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-20T22:00:41.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:41.088+0000 7f06a1000700 1 -- 172.21.15.62:0/672920734 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f068006ecc0 msgr2=0x7f0680071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:00:41.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:41.088+0000 7f06a1000700 1 --2- 172.21.15.62:0/672920734 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f068006ecc0 0x7f0680071170 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f068c00b5c0 tx=0x7f068c005960 comp rx=0 tx=0).stop 2024-04-20T22:00:41.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:41.088+0000 7f06a1000700 1 -- 172.21.15.62:0/672920734 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f069c10c8f0 msgr2=0x7f069c19ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:00:41.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:41.088+0000 7f06a1000700 1 --2- 172.21.15.62:0/672920734 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f069c10c8f0 0x7f069c19ca40 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f068400d8d0 tx=0x7f068400dc90 comp rx=0 tx=0).stop 2024-04-20T22:00:41.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:41.088+0000 7f06a1000700 1 -- 172.21.15.62:0/672920734 shutdown_connections 2024-04-20T22:00:41.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:41.088+0000 7f06a1000700 1 --2- 172.21.15.62:0/672920734 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f068006ecc0 0x7f0680071170 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:00:41.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:41.088+0000 7f06a1000700 1 --2- 172.21.15.62:0/672920734 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f069c106b80 0x7f069c19c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:00:41.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:41.088+0000 7f06a1000700 1 --2- 172.21.15.62:0/672920734 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f069c10c8f0 0x7f069c19ca40 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:00:41.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:41.088+0000 7f06a1000700 1 -- 172.21.15.62:0/672920734 >> 172.21.15.62:0/672920734 conn(0x7f069c076060 msgr2=0x7f069c077aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:00:41.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:41.089+0000 7f06a1000700 1 -- 172.21.15.62:0/672920734 shutdown_connections 2024-04-20T22:00:41.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:00:41.089+0000 7f06a1000700 1 -- 172.21.15.62:0/672920734 wait complete. 2024-04-20T22:00:41.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:00:41 smithi189 ceph-mon[57022]: from='client.35396 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:00:41.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:00:41 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/672920734' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:00:42.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:00:41 smithi062 ceph-mon[76322]: from='client.35396 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:00:42.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:00:41 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/672920734' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:00:42.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:00:42 smithi189 ceph-mon[57022]: pgmap v763: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:00:43.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:00:42 smithi062 ceph-mon[76322]: pgmap v763: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:00:44.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:00:44 smithi189 ceph-mon[57022]: pgmap v764: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:00:45.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:00:44 smithi062 ceph-mon[76322]: pgmap v764: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:00:46.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:00:46 smithi189 ceph-mon[57022]: pgmap v765: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:00:47.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:00:46 smithi062 ceph-mon[76322]: pgmap v765: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:00:48.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:00:48 smithi189 ceph-mon[57022]: pgmap v766: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:00:49.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:00:48 smithi062 ceph-mon[76322]: pgmap v766: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:00:49.990 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:00:49 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:00:50.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:00:49 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:00:50.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:00:50 smithi189 ceph-mon[57022]: pgmap v767: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:00:51.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:00:50 smithi062 ceph-mon[76322]: pgmap v767: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:00:52.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:00:52 smithi189 ceph-mon[57022]: pgmap v768: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:00:53.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:00:52 smithi062 ceph-mon[76322]: pgmap v768: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:00:55.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:00:54 smithi062 ceph-mon[76322]: pgmap v769: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:00:55.226 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:00:54 smithi189 ceph-mon[57022]: pgmap v769: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:00:57.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:00:56 smithi062 ceph-mon[76322]: pgmap v770: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:00:57.155 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:00:56 smithi189 ceph-mon[57022]: pgmap v770: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:00:59.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:00:58 smithi062 ceph-mon[76322]: pgmap v771: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:00:59.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:00:58 smithi189 ceph-mon[57022]: pgmap v771: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:01:01.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:01:00 smithi062 ceph-mon[76322]: pgmap v772: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:01:01.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:01:00 smithi189 ceph-mon[57022]: pgmap v772: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:01:03.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:01:02 smithi062 ceph-mon[76322]: pgmap v773: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:01:03.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:01:02 smithi189 ceph-mon[57022]: pgmap v773: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:01:05.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:01:04 smithi062 ceph-mon[76322]: pgmap v774: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:01:05.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:01:04 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:01:05.040 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:01:04 smithi189 ceph-mon[57022]: pgmap v774: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:01:05.040 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:01:04 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:01:06.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:01:06 smithi189 ceph-mon[57022]: pgmap v775: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:01:07.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:01:06 smithi062 ceph-mon[76322]: pgmap v775: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:01:09.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:01:08 smithi189 ceph-mon[57022]: pgmap v776: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:01:09.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:01:08 smithi062 ceph-mon[76322]: pgmap v776: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:01:11.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:01:10 smithi189 ceph-mon[57022]: pgmap v777: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:01:11.265 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:01:10 smithi062 ceph-mon[76322]: pgmap v777: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:01:11.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:11.265+0000 7fb40ea00700 1 -- 172.21.15.62:0/348779563 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb408106b80 msgr2=0x7fb408106ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:01:11.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:11.265+0000 7fb40ea00700 1 --2- 172.21.15.62:0/348779563 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb408106b80 0x7fb408106ff0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fb3f0009910 tx=0x7fb3f0009c20 comp rx=0 tx=0).stop 2024-04-20T22:01:11.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:11.265+0000 7fb40ea00700 1 -- 172.21.15.62:0/348779563 shutdown_connections 2024-04-20T22:01:11.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:11.265+0000 7fb40ea00700 1 --2- 172.21.15.62:0/348779563 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb408106b80 0x7fb408106ff0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:01:11.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:11.265+0000 7fb40ea00700 1 --2- 172.21.15.62:0/348779563 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb40810c8f0 0x7fb40810ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:01:11.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:11.265+0000 7fb40ea00700 1 -- 172.21.15.62:0/348779563 >> 172.21.15.62:0/348779563 conn(0x7fb408076060 msgr2=0x7fb408078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:01:11.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:11.265+0000 7fb40ea00700 1 -- 172.21.15.62:0/348779563 shutdown_connections 2024-04-20T22:01:11.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:11.265+0000 7fb40ea00700 1 -- 172.21.15.62:0/348779563 wait complete. 2024-04-20T22:01:11.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:11.266+0000 7fb40ea00700 1 Processor -- start 2024-04-20T22:01:11.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:11.267+0000 7fb40ea00700 1 -- start start 2024-04-20T22:01:11.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:11.267+0000 7fb40ea00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb408106b80 0x7fb40819c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:01:11.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:11.267+0000 7fb40ea00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb40810c8f0 0x7fb40819c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:01:11.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:11.267+0000 7fb40ea00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb40819d060 con 0x7fb408106b80 2024-04-20T22:01:11.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:11.267+0000 7fb40ea00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4081a0df0 con 0x7fb40810c8f0 2024-04-20T22:01:11.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:11.267+0000 7fb407e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb408106b80 0x7fb40819c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:01:11.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:11.267+0000 7fb407e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb408106b80 0x7fb40819c440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33922/0 (socket says 172.21.15.62:33922) 2024-04-20T22:01:11.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:11.267+0000 7fb407e00700 1 -- 172.21.15.62:0/1115842530 learned_addr learned my addr 172.21.15.62:0/1115842530 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:01:11.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:11.268+0000 7fb3ffe00700 1 --2- 172.21.15.62:0/1115842530 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb40810c8f0 0x7fb40819c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:01:11.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:11.268+0000 7fb407e00700 1 -- 172.21.15.62:0/1115842530 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb40810c8f0 msgr2=0x7fb40819c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:01:11.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:11.268+0000 7fb407e00700 1 --2- 172.21.15.62:0/1115842530 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb40810c8f0 0x7fb40819c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:01:11.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:11.268+0000 7fb407e00700 1 -- 172.21.15.62:0/1115842530 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3f00095f0 con 0x7fb408106b80 2024-04-20T22:01:11.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:11.268+0000 7fb407e00700 1 --2- 172.21.15.62:0/1115842530 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb408106b80 0x7fb40819c440 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fb3f800d8d0 tx=0x7fb3f800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:01:11.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:11.269+0000 7fb405600700 1 -- 172.21.15.62:0/1115842530 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb3f8009940 con 0x7fb408106b80 2024-04-20T22:01:11.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:11.269+0000 7fb405600700 1 -- 172.21.15.62:0/1115842530 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb3f8010460 con 0x7fb408106b80 2024-04-20T22:01:11.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:11.269+0000 7fb405600700 1 -- 172.21.15.62:0/1115842530 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb3f800f5f0 con 0x7fb408106b80 2024-04-20T22:01:11.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:11.269+0000 7fb40ea00700 1 -- 172.21.15.62:0/1115842530 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4081a10d0 con 0x7fb408106b80 2024-04-20T22:01:11.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:11.269+0000 7fb40ea00700 1 -- 172.21.15.62:0/1115842530 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4081a1620 con 0x7fb408106b80 2024-04-20T22:01:11.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:11.270+0000 7fb40ea00700 1 -- 172.21.15.62:0/1115842530 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb40810ecc0 con 0x7fb408106b80 2024-04-20T22:01:11.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:11.271+0000 7fb405600700 1 -- 172.21.15.62:0/1115842530 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fb3f80105d0 con 0x7fb408106b80 2024-04-20T22:01:11.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:11.272+0000 7fb405600700 1 --2- 172.21.15.62:0/1115842530 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb3ec06ecc0 0x7fb3ec071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:01:11.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:11.272+0000 7fb405600700 1 -- 172.21.15.62:0/1115842530 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fb3f808ed70 con 0x7fb408106b80 2024-04-20T22:01:11.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:11.273+0000 7fb3ffe00700 1 --2- 172.21.15.62:0/1115842530 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb3ec06ecc0 0x7fb3ec071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:01:11.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:11.273+0000 7fb3ffe00700 1 --2- 172.21.15.62:0/1115842530 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb3ec06ecc0 0x7fb3ec071170 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fb40819da60 tx=0x7fb3f001a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:01:11.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:11.278+0000 7fb405600700 1 -- 172.21.15.62:0/1115842530 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fb3f8057ba0 con 0x7fb408106b80 2024-04-20T22:01:11.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:11.535+0000 7fb40ea00700 1 -- 172.21.15.62:0/1115842530 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb4081a1890 con 0x7fb3ec06ecc0 2024-04-20T22:01:11.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:11.538+0000 7fb405600700 1 -- 172.21.15.62:0/1115842530 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fb4081a1890 con 0x7fb3ec06ecc0 2024-04-20T22:01:11.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:11.541+0000 7fb40ea00700 1 -- 172.21.15.62:0/1115842530 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb3ec06ecc0 msgr2=0x7fb3ec071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:01:11.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:11.541+0000 7fb40ea00700 1 --2- 172.21.15.62:0/1115842530 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb3ec06ecc0 0x7fb3ec071170 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fb40819da60 tx=0x7fb3f001a040 comp rx=0 tx=0).stop 2024-04-20T22:01:11.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:11.541+0000 7fb40ea00700 1 -- 172.21.15.62:0/1115842530 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb408106b80 msgr2=0x7fb40819c440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:01:11.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:11.541+0000 7fb40ea00700 1 --2- 172.21.15.62:0/1115842530 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb408106b80 0x7fb40819c440 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fb3f800d8d0 tx=0x7fb3f800dc90 comp rx=0 tx=0).stop 2024-04-20T22:01:11.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:11.541+0000 7fb40ea00700 1 -- 172.21.15.62:0/1115842530 shutdown_connections 2024-04-20T22:01:11.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:11.541+0000 7fb40ea00700 1 --2- 172.21.15.62:0/1115842530 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb3ec06ecc0 0x7fb3ec071170 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:01:11.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:11.541+0000 7fb40ea00700 1 --2- 172.21.15.62:0/1115842530 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb408106b80 0x7fb40819c440 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:01:11.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:11.541+0000 7fb40ea00700 1 --2- 172.21.15.62:0/1115842530 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb40810c8f0 0x7fb40819c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:01:11.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:11.541+0000 7fb40ea00700 1 -- 172.21.15.62:0/1115842530 >> 172.21.15.62:0/1115842530 conn(0x7fb408076060 msgr2=0x7fb408077ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:01:11.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:11.542+0000 7fb40ea00700 1 -- 172.21.15.62:0/1115842530 shutdown_connections 2024-04-20T22:01:11.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:11.542+0000 7fb40ea00700 1 -- 172.21.15.62:0/1115842530 wait complete. 2024-04-20T22:01:11.557 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T22:01:11.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:11.702+0000 7fa111800700 1 -- 172.21.15.62:0/47035200 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa10c1004e0 msgr2=0x7fa10c1008b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:01:11.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:11.702+0000 7fa111800700 1 --2- 172.21.15.62:0/47035200 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa10c1004e0 0x7fa10c1008b0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fa0fc009910 tx=0x7fa0fc009c20 comp rx=0 tx=0).stop 2024-04-20T22:01:11.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:11.703+0000 7fa111800700 1 -- 172.21.15.62:0/47035200 shutdown_connections 2024-04-20T22:01:11.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:11.703+0000 7fa111800700 1 --2- 172.21.15.62:0/47035200 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa10c100e80 0x7fa10c104e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:01:11.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:11.703+0000 7fa111800700 1 --2- 172.21.15.62:0/47035200 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa10c1004e0 0x7fa10c1008b0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:01:11.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:11.703+0000 7fa111800700 1 -- 172.21.15.62:0/47035200 >> 172.21.15.62:0/47035200 conn(0x7fa10c0fbda0 msgr2=0x7fa10c0fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:01:11.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:11.703+0000 7fa111800700 1 -- 172.21.15.62:0/47035200 shutdown_connections 2024-04-20T22:01:11.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:11.703+0000 7fa111800700 1 -- 172.21.15.62:0/47035200 wait complete. 2024-04-20T22:01:11.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:11.704+0000 7fa111800700 1 Processor -- start 2024-04-20T22:01:11.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:11.704+0000 7fa111800700 1 -- start start 2024-04-20T22:01:11.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:11.704+0000 7fa111800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa10c1004e0 0x7fa10c1980f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:01:11.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:11.704+0000 7fa111800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa10c100e80 0x7fa10c198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:01:11.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:11.704+0000 7fa111800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa10c198d10 con 0x7fa10c100e80 2024-04-20T22:01:11.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:11.704+0000 7fa111800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa10c19caa0 con 0x7fa10c1004e0 2024-04-20T22:01:11.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:11.705+0000 7fa103e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa10c100e80 0x7fa10c198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:01:11.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:11.705+0000 7fa103e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa10c100e80 0x7fa10c198630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35506/0 (socket says 172.21.15.62:35506) 2024-04-20T22:01:11.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:11.705+0000 7fa103e00700 1 -- 172.21.15.62:0/2017125341 learned_addr learned my addr 172.21.15.62:0/2017125341 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:01:11.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:11.705+0000 7fa10aa00700 1 --2- 172.21.15.62:0/2017125341 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa10c1004e0 0x7fa10c1980f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:01:11.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:11.705+0000 7fa103e00700 1 -- 172.21.15.62:0/2017125341 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa10c1004e0 msgr2=0x7fa10c1980f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:01:11.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:11.705+0000 7fa103e00700 1 --2- 172.21.15.62:0/2017125341 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa10c1004e0 0x7fa10c1980f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:01:11.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:11.705+0000 7fa103e00700 1 -- 172.21.15.62:0/2017125341 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0fc0095f0 con 0x7fa10c100e80 2024-04-20T22:01:11.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:11.706+0000 7fa103e00700 1 --2- 172.21.15.62:0/2017125341 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa10c100e80 0x7fa10c198630 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fa0f400d900 tx=0x7fa0f400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:01:11.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:11.706+0000 7fa103400700 1 -- 172.21.15.62:0/2017125341 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa0f4009900 con 0x7fa10c100e80 2024-04-20T22:01:11.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:11.706+0000 7fa103400700 1 -- 172.21.15.62:0/2017125341 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa0f4010460 con 0x7fa10c100e80 2024-04-20T22:01:11.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:11.706+0000 7fa103400700 1 -- 172.21.15.62:0/2017125341 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa0f400f5f0 con 0x7fa10c100e80 2024-04-20T22:01:11.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:11.706+0000 7fa111800700 1 -- 172.21.15.62:0/2017125341 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa10c19cd80 con 0x7fa10c100e80 2024-04-20T22:01:11.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:11.706+0000 7fa111800700 1 -- 172.21.15.62:0/2017125341 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa10c19d2d0 con 0x7fa10c100e80 2024-04-20T22:01:11.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:11.707+0000 7fa111800700 1 -- 172.21.15.62:0/2017125341 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa10c1045d0 con 0x7fa10c100e80 2024-04-20T22:01:11.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:11.708+0000 7fa103400700 1 -- 172.21.15.62:0/2017125341 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fa0f40105d0 con 0x7fa10c100e80 2024-04-20T22:01:11.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:11.709+0000 7fa103400700 1 --2- 172.21.15.62:0/2017125341 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa0f006ecc0 0x7fa0f0071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:01:11.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:11.709+0000 7fa103400700 1 -- 172.21.15.62:0/2017125341 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fa0f408ea70 con 0x7fa10c100e80 2024-04-20T22:01:11.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:11.710+0000 7fa10aa00700 1 --2- 172.21.15.62:0/2017125341 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa0f006ecc0 0x7fa0f0071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:01:11.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:11.710+0000 7fa10aa00700 1 --2- 172.21.15.62:0/2017125341 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa0f006ecc0 0x7fa0f0071170 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fa0fc006010 tx=0x7fa0fc01a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:01:11.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:11.714+0000 7fa103400700 1 -- 172.21.15.62:0/2017125341 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fa0f4057800 con 0x7fa10c100e80 2024-04-20T22:01:11.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:11.968+0000 7fa111800700 1 -- 172.21.15.62:0/2017125341 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa10c19d600 con 0x7fa0f006ecc0 2024-04-20T22:01:11.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:11.971+0000 7fa103400700 1 -- 172.21.15.62:0/2017125341 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fa10c19d600 con 0x7fa0f006ecc0 2024-04-20T22:01:11.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:11.974+0000 7fa111800700 1 -- 172.21.15.62:0/2017125341 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa0f006ecc0 msgr2=0x7fa0f0071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:01:11.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:11.974+0000 7fa111800700 1 --2- 172.21.15.62:0/2017125341 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa0f006ecc0 0x7fa0f0071170 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fa0fc006010 tx=0x7fa0fc01a040 comp rx=0 tx=0).stop 2024-04-20T22:01:11.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:11.974+0000 7fa111800700 1 -- 172.21.15.62:0/2017125341 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa10c100e80 msgr2=0x7fa10c198630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:01:11.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:11.974+0000 7fa111800700 1 --2- 172.21.15.62:0/2017125341 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa10c100e80 0x7fa10c198630 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fa0f400d900 tx=0x7fa0f400dcc0 comp rx=0 tx=0).stop 2024-04-20T22:01:11.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:11.974+0000 7fa111800700 1 -- 172.21.15.62:0/2017125341 shutdown_connections 2024-04-20T22:01:11.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:11.974+0000 7fa111800700 1 --2- 172.21.15.62:0/2017125341 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa0f006ecc0 0x7fa0f0071170 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:01:11.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:11.974+0000 7fa111800700 1 --2- 172.21.15.62:0/2017125341 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa10c1004e0 0x7fa10c1980f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:01:11.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:11.974+0000 7fa111800700 1 --2- 172.21.15.62:0/2017125341 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa10c100e80 0x7fa10c198630 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:01:11.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:11.974+0000 7fa111800700 1 -- 172.21.15.62:0/2017125341 >> 172.21.15.62:0/2017125341 conn(0x7fa10c0fbda0 msgr2=0x7fa10c0fe160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:01:11.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:11.975+0000 7fa111800700 1 -- 172.21.15.62:0/2017125341 shutdown_connections 2024-04-20T22:01:11.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:11.975+0000 7fa111800700 1 -- 172.21.15.62:0/2017125341 wait complete. 2024-04-20T22:01:12.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:12.125+0000 7f5631600700 1 -- 172.21.15.62:0/2907203851 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f562c108470 msgr2=0x7f562c108840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:01:12.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:12.125+0000 7f5631600700 1 --2- 172.21.15.62:0/2907203851 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f562c108470 0x7f562c108840 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f561c009910 tx=0x7f561c009c20 comp rx=0 tx=0).stop 2024-04-20T22:01:12.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:12.125+0000 7f5631600700 1 -- 172.21.15.62:0/2907203851 shutdown_connections 2024-04-20T22:01:12.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:12.125+0000 7f5631600700 1 --2- 172.21.15.62:0/2907203851 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f562c102470 0x7f562c1028e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:01:12.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:12.125+0000 7f5631600700 1 --2- 172.21.15.62:0/2907203851 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f562c108470 0x7f562c108840 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:01:12.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:12.125+0000 7f5631600700 1 -- 172.21.15.62:0/2907203851 >> 172.21.15.62:0/2907203851 conn(0x7f562c0fdf90 msgr2=0x7f562c1003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:01:12.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:12.126+0000 7f5631600700 1 -- 172.21.15.62:0/2907203851 shutdown_connections 2024-04-20T22:01:12.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:12.126+0000 7f5631600700 1 -- 172.21.15.62:0/2907203851 wait complete. 2024-04-20T22:01:12.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:12.127+0000 7f5631600700 1 Processor -- start 2024-04-20T22:01:12.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:12.127+0000 7f5631600700 1 -- start start 2024-04-20T22:01:12.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:12.127+0000 7f5631600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f562c102470 0x7f562c198210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:01:12.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:12.127+0000 7f5631600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f562c108470 0x7f562c198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:01:12.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:12.127+0000 7f5631600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f562c198e30 con 0x7f562c102470 2024-04-20T22:01:12.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:12.127+0000 7f5631600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f562c19cbc0 con 0x7f562c108470 2024-04-20T22:01:12.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:12.127+0000 7f562aa00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f562c102470 0x7f562c198210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:01:12.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:12.128+0000 7f562aa00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f562c102470 0x7f562c198210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35534/0 (socket says 172.21.15.62:35534) 2024-04-20T22:01:12.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:12.128+0000 7f562aa00700 1 -- 172.21.15.62:0/2598106956 learned_addr learned my addr 172.21.15.62:0/2598106956 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:01:12.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:12.128+0000 7f562a000700 1 --2- 172.21.15.62:0/2598106956 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f562c108470 0x7f562c198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:01:12.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:12.128+0000 7f562aa00700 1 -- 172.21.15.62:0/2598106956 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f562c108470 msgr2=0x7f562c198750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:01:12.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:12.128+0000 7f562aa00700 1 --2- 172.21.15.62:0/2598106956 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f562c108470 0x7f562c198750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:01:12.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:12.128+0000 7f562aa00700 1 -- 172.21.15.62:0/2598106956 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f561c0095f0 con 0x7f562c102470 2024-04-20T22:01:12.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:12.128+0000 7f562aa00700 1 --2- 172.21.15.62:0/2598106956 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f562c102470 0x7f562c198210 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f561c00b5c0 tx=0x7f561c00be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:01:12.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:12.128+0000 7f5623400700 1 -- 172.21.15.62:0/2598106956 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f561c01d070 con 0x7f562c102470 2024-04-20T22:01:12.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:12.129+0000 7f5623400700 1 -- 172.21.15.62:0/2598106956 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f561c00f460 con 0x7f562c102470 2024-04-20T22:01:12.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:12.129+0000 7f5631600700 1 -- 172.21.15.62:0/2598106956 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f562c19ce40 con 0x7f562c102470 2024-04-20T22:01:12.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:12.129+0000 7f5623400700 1 -- 172.21.15.62:0/2598106956 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f561c017610 con 0x7f562c102470 2024-04-20T22:01:12.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:12.129+0000 7f5631600700 1 -- 172.21.15.62:0/2598106956 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f562c19d330 con 0x7f562c102470 2024-04-20T22:01:12.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:12.130+0000 7f5631600700 1 -- 172.21.15.62:0/2598106956 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f562c10a9e0 con 0x7f562c102470 2024-04-20T22:01:12.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:12.131+0000 7f5623400700 1 -- 172.21.15.62:0/2598106956 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f561c00f5d0 con 0x7f562c102470 2024-04-20T22:01:12.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:12.132+0000 7f5623400700 1 --2- 172.21.15.62:0/2598106956 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f561006ecc0 0x7f5610071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:01:12.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:12.132+0000 7f5623400700 1 -- 172.21.15.62:0/2598106956 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f561c090510 con 0x7f562c102470 2024-04-20T22:01:12.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:12.132+0000 7f562a000700 1 --2- 172.21.15.62:0/2598106956 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f561006ecc0 0x7f5610071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:01:12.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:12.133+0000 7f562a000700 1 --2- 172.21.15.62:0/2598106956 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f561006ecc0 0x7f5610071170 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f5618006fd0 tx=0x7f5618008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:01:12.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:12.136+0000 7f5623400700 1 -- 172.21.15.62:0/2598106956 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f561c059220 con 0x7f562c102470 2024-04-20T22:01:12.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:12.379+0000 7f5631600700 1 -- 172.21.15.62:0/2598106956 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f562c199610 con 0x7f561006ecc0 2024-04-20T22:01:12.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:12.403+0000 7f5623400700 1 -- 172.21.15.62:0/2598106956 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f562c199610 con 0x7f561006ecc0 2024-04-20T22:01:12.403 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T22:01:12.403 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (27m) 3m ago 40m 15.3M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T22:01:12.403 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (41m) 3m ago 41m 10.4M - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T22:01:12.403 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (39m) 3m ago 39m 10.1M - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T22:01:12.403 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (23m) 3m ago 41m 6660k - 19.0.0-3133-ga8c84f59 76bc6b91b726 2825d5ba11c6 2024-04-20T22:01:12.403 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (23m) 3m ago 39m 6665k - 19.0.0-3133-ga8c84f59 76bc6b91b726 20f746fa1bba 2024-04-20T22:01:12.403 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (26m) 3m ago 40m 73.5M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T22:01:12.403 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (18m) 3m ago 33m 18.5M - 19.0.0-3133-ga8c84f59 76bc6b91b726 de80bc90a683 2024-04-20T22:01:12.404 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (19m) 3m ago 33m 18.7M - 19.0.0-3133-ga8c84f59 76bc6b91b726 cd793b7ee98e 2024-04-20T22:01:12.404 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (3m) 3m ago 33m 11.0M - 19.0.0-3133-ga8c84f59 76bc6b91b726 0e7b0bc4d1ab 2024-04-20T22:01:12.404 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (33m) 3m ago 33m 18.6M - 18.2.0 dc2bc1663786 f2eec2ee8951 2024-04-20T22:01:12.404 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (29m) 3m ago 42m 513M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T22:01:12.404 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (28m) 3m ago 39m 459M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T22:01:12.404 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (24m) 3m ago 42m 71.7M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 019eb1479485 2024-04-20T22:01:12.404 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (24m) 3m ago 39m 51.0M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 55aac560aab6 2024-04-20T22:01:12.404 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (28m) 3m ago 41m 8472k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T22:01:12.404 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (28m) 3m ago 39m 8048k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T22:01:12.404 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (22m) 3m ago 38m 91.2M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 ee2bb901f26e 2024-04-20T22:01:12.404 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (21m) 3m ago 37m 94.6M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 9075152d4162 2024-04-20T22:01:12.404 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (21m) 3m ago 37m 78.1M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 2d4f3e38dc0e 2024-04-20T22:01:12.404 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (20m) 3m ago 36m 92.6M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 8f97b17ae0eb 2024-04-20T22:01:12.404 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (20m) 3m ago 36m 95.6M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 653d8079967d 2024-04-20T22:01:12.404 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (19m) 3m ago 35m 89.2M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 596dded14c12 2024-04-20T22:01:12.405 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (27m) 3m ago 40m 82.1M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T22:01:12.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:12.406+0000 7f5631600700 1 -- 172.21.15.62:0/2598106956 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f561006ecc0 msgr2=0x7f5610071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:01:12.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:12.406+0000 7f5631600700 1 --2- 172.21.15.62:0/2598106956 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f561006ecc0 0x7f5610071170 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f5618006fd0 tx=0x7f5618008040 comp rx=0 tx=0).stop 2024-04-20T22:01:12.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:12.406+0000 7f5631600700 1 -- 172.21.15.62:0/2598106956 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f562c102470 msgr2=0x7f562c198210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:01:12.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:12.406+0000 7f5631600700 1 --2- 172.21.15.62:0/2598106956 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f562c102470 0x7f562c198210 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f561c00b5c0 tx=0x7f561c00be00 comp rx=0 tx=0).stop 2024-04-20T22:01:12.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:12.406+0000 7f5631600700 1 -- 172.21.15.62:0/2598106956 shutdown_connections 2024-04-20T22:01:12.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:12.406+0000 7f5631600700 1 --2- 172.21.15.62:0/2598106956 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f561006ecc0 0x7f5610071170 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:01:12.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:12.406+0000 7f5631600700 1 --2- 172.21.15.62:0/2598106956 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f562c102470 0x7f562c198210 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:01:12.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:12.406+0000 7f5631600700 1 --2- 172.21.15.62:0/2598106956 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f562c108470 0x7f562c198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:01:12.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:12.407+0000 7f5631600700 1 -- 172.21.15.62:0/2598106956 >> 172.21.15.62:0/2598106956 conn(0x7f562c0fdf90 msgr2=0x7f562c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:01:12.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:12.407+0000 7f5631600700 1 -- 172.21.15.62:0/2598106956 shutdown_connections 2024-04-20T22:01:12.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:12.407+0000 7f5631600700 1 -- 172.21.15.62:0/2598106956 wait complete. 2024-04-20T22:01:12.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:12.568+0000 7fb87f600700 1 -- 172.21.15.62:0/2281065653 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb878108470 msgr2=0x7fb878108840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:01:12.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:12.568+0000 7fb87f600700 1 --2- 172.21.15.62:0/2281065653 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb878108470 0x7fb878108840 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fb86c009910 tx=0x7fb86c009c20 comp rx=0 tx=0).stop 2024-04-20T22:01:12.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:12.569+0000 7fb87f600700 1 -- 172.21.15.62:0/2281065653 shutdown_connections 2024-04-20T22:01:12.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:12.569+0000 7fb87f600700 1 --2- 172.21.15.62:0/2281065653 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb878102470 0x7fb8781028e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:01:12.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:12.569+0000 7fb87f600700 1 --2- 172.21.15.62:0/2281065653 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb878108470 0x7fb878108840 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:01:12.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:12.569+0000 7fb87f600700 1 -- 172.21.15.62:0/2281065653 >> 172.21.15.62:0/2281065653 conn(0x7fb8780fdf90 msgr2=0x7fb8781003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:01:12.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:12.569+0000 7fb87f600700 1 -- 172.21.15.62:0/2281065653 shutdown_connections 2024-04-20T22:01:12.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:12.569+0000 7fb87f600700 1 -- 172.21.15.62:0/2281065653 wait complete. 2024-04-20T22:01:12.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:12.570+0000 7fb87f600700 1 Processor -- start 2024-04-20T22:01:12.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:12.570+0000 7fb87f600700 1 -- start start 2024-04-20T22:01:12.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:12.570+0000 7fb87f600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb878102470 0x7fb8781981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:01:12.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:12.570+0000 7fb87f600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb878108470 0x7fb878198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:01:12.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:12.570+0000 7fb87f600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb878198e10 con 0x7fb878108470 2024-04-20T22:01:12.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:12.570+0000 7fb87f600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb87819cba0 con 0x7fb878102470 2024-04-20T22:01:12.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:12.571+0000 7fb877e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb878108470 0x7fb878198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:01:12.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:12.571+0000 7fb877e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb878108470 0x7fb878198730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35558/0 (socket says 172.21.15.62:35558) 2024-04-20T22:01:12.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:12.571+0000 7fb877e00700 1 -- 172.21.15.62:0/2372141257 learned_addr learned my addr 172.21.15.62:0/2372141257 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:01:12.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:12.571+0000 7fb87cc00700 1 --2- 172.21.15.62:0/2372141257 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb878102470 0x7fb8781981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:01:12.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:12.571+0000 7fb877e00700 1 -- 172.21.15.62:0/2372141257 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb878102470 msgr2=0x7fb8781981f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:01:12.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:12.571+0000 7fb877e00700 1 --2- 172.21.15.62:0/2372141257 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb878102470 0x7fb8781981f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:01:12.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:12.571+0000 7fb877e00700 1 -- 172.21.15.62:0/2372141257 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb86c0095f0 con 0x7fb878108470 2024-04-20T22:01:12.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:12.572+0000 7fb877e00700 1 --2- 172.21.15.62:0/2372141257 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb878108470 0x7fb878198730 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fb86400c8a0 tx=0x7fb86400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:01:12.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:12.572+0000 7fb875600700 1 -- 172.21.15.62:0/2372141257 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb86400cea0 con 0x7fb878108470 2024-04-20T22:01:12.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:12.572+0000 7fb875600700 1 -- 172.21.15.62:0/2372141257 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb864004830 con 0x7fb878108470 2024-04-20T22:01:12.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:12.572+0000 7fb875600700 1 -- 172.21.15.62:0/2372141257 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb8640055b0 con 0x7fb878108470 2024-04-20T22:01:12.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:12.572+0000 7fb87f600700 1 -- 172.21.15.62:0/2372141257 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb87819ce80 con 0x7fb878108470 2024-04-20T22:01:12.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:12.572+0000 7fb87f600700 1 -- 172.21.15.62:0/2372141257 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb87819d3d0 con 0x7fb878108470 2024-04-20T22:01:12.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:12.573+0000 7fb87f600700 1 -- 172.21.15.62:0/2372141257 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb87810a9e0 con 0x7fb878108470 2024-04-20T22:01:12.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:12.574+0000 7fb875600700 1 -- 172.21.15.62:0/2372141257 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fb8640049a0 con 0x7fb878108470 2024-04-20T22:01:12.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:12.576+0000 7fb875600700 1 --2- 172.21.15.62:0/2372141257 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb86006ecc0 0x7fb860071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:01:12.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:12.576+0000 7fb875600700 1 -- 172.21.15.62:0/2372141257 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fb86408eaf0 con 0x7fb878108470 2024-04-20T22:01:12.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:12.576+0000 7fb87cc00700 1 --2- 172.21.15.62:0/2372141257 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb86006ecc0 0x7fb860071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:01:12.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:12.577+0000 7fb87cc00700 1 --2- 172.21.15.62:0/2372141257 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb86006ecc0 0x7fb860071170 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fb86c006010 tx=0x7fb86c01a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:01:12.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:12.581+0000 7fb875600700 1 -- 172.21.15.62:0/2372141257 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fb864057880 con 0x7fb878108470 2024-04-20T22:01:12.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:12.916+0000 7fb87f600700 1 -- 172.21.15.62:0/2372141257 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fb87804f2a0 con 0x7fb878108470 2024-04-20T22:01:12.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:12.917+0000 7fb875600700 1 -- 172.21.15.62:0/2372141257 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7fb86401d070 con 0x7fb878108470 2024-04-20T22:01:12.917 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:01:12.917 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T22:01:12.917 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:01:12.917 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:01:12.917 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T22:01:12.917 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:01:12.917 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:01:12.917 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T22:01:12.917 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 6 2024-04-20T22:01:12.917 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:01:12.918 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T22:01:12.918 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 1, 2024-04-20T22:01:12.918 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 3 2024-04-20T22:01:12.918 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:01:12.918 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T22:01:12.918 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 1, 2024-04-20T22:01:12.918 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 13 2024-04-20T22:01:12.918 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T22:01:12.918 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:01:12.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:12.920+0000 7fb87f600700 1 -- 172.21.15.62:0/2372141257 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb86006ecc0 msgr2=0x7fb860071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:01:12.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:12.920+0000 7fb87f600700 1 --2- 172.21.15.62:0/2372141257 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb86006ecc0 0x7fb860071170 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fb86c006010 tx=0x7fb86c01a040 comp rx=0 tx=0).stop 2024-04-20T22:01:12.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:12.920+0000 7fb87f600700 1 -- 172.21.15.62:0/2372141257 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb878108470 msgr2=0x7fb878198730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:01:12.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:12.920+0000 7fb87f600700 1 --2- 172.21.15.62:0/2372141257 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb878108470 0x7fb878198730 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fb86400c8a0 tx=0x7fb86400cc60 comp rx=0 tx=0).stop 2024-04-20T22:01:12.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:12.920+0000 7fb87f600700 1 -- 172.21.15.62:0/2372141257 shutdown_connections 2024-04-20T22:01:12.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:12.920+0000 7fb87f600700 1 --2- 172.21.15.62:0/2372141257 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb86006ecc0 0x7fb860071170 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:01:12.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:12.920+0000 7fb87f600700 1 --2- 172.21.15.62:0/2372141257 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb878102470 0x7fb8781981f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:01:12.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:12.920+0000 7fb87f600700 1 --2- 172.21.15.62:0/2372141257 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb878108470 0x7fb878198730 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:01:12.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:12.920+0000 7fb87f600700 1 -- 172.21.15.62:0/2372141257 >> 172.21.15.62:0/2372141257 conn(0x7fb8780fdf90 msgr2=0x7fb8781002c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:01:12.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:12.920+0000 7fb87f600700 1 -- 172.21.15.62:0/2372141257 shutdown_connections 2024-04-20T22:01:12.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:12.920+0000 7fb87f600700 1 -- 172.21.15.62:0/2372141257 wait complete. 2024-04-20T22:01:13.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.079+0000 7f961c600700 1 -- 172.21.15.62:0/2403290797 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f961410c8f0 msgr2=0x7f961410ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:01:13.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.079+0000 7f961c600700 1 --2- 172.21.15.62:0/2403290797 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f961410c8f0 0x7f961410ccc0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f9608009910 tx=0x7f9608009c20 comp rx=0 tx=0).stop 2024-04-20T22:01:13.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.079+0000 7f961c600700 1 -- 172.21.15.62:0/2403290797 shutdown_connections 2024-04-20T22:01:13.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.079+0000 7f961c600700 1 --2- 172.21.15.62:0/2403290797 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9614106b80 0x7f9614106ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:01:13.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.079+0000 7f961c600700 1 --2- 172.21.15.62:0/2403290797 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f961410c8f0 0x7f961410ccc0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:01:13.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.079+0000 7f961c600700 1 -- 172.21.15.62:0/2403290797 >> 172.21.15.62:0/2403290797 conn(0x7f9614076060 msgr2=0x7f9614078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:01:13.079 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:01:12 smithi062 ceph-mon[76322]: pgmap v778: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:01:13.080 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:01:12 smithi062 ceph-mon[76322]: from='client.35404 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:01:13.080 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:01:12 smithi062 ceph-mon[76322]: from='client.35408 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:01:13.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.079+0000 7f961c600700 1 -- 172.21.15.62:0/2403290797 shutdown_connections 2024-04-20T22:01:13.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.079+0000 7f961c600700 1 -- 172.21.15.62:0/2403290797 wait complete. 2024-04-20T22:01:13.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.080+0000 7f961c600700 1 Processor -- start 2024-04-20T22:01:13.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.080+0000 7f961c600700 1 -- start start 2024-04-20T22:01:13.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.081+0000 7f961c600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9614106b80 0x7f961419c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:01:13.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.081+0000 7f961c600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f961410c8f0 0x7f961419ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:01:13.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.081+0000 7f961c600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f961419d090 con 0x7f961410c8f0 2024-04-20T22:01:13.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.081+0000 7f961c600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f961419d1d0 con 0x7f9614106b80 2024-04-20T22:01:13.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.081+0000 7f9619200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f961410c8f0 0x7f961419ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:01:13.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.081+0000 7f9619200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f961410c8f0 0x7f961419ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35584/0 (socket says 172.21.15.62:35584) 2024-04-20T22:01:13.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.081+0000 7f9619200700 1 -- 172.21.15.62:0/2766190890 learned_addr learned my addr 172.21.15.62:0/2766190890 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:01:13.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.082+0000 7f9619c00700 1 --2- 172.21.15.62:0/2766190890 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9614106b80 0x7f961419c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:01:13.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.082+0000 7f9619200700 1 -- 172.21.15.62:0/2766190890 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9614106b80 msgr2=0x7f961419c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:01:13.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.082+0000 7f9619200700 1 --2- 172.21.15.62:0/2766190890 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9614106b80 0x7f961419c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:01:13.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.082+0000 7f9619200700 1 -- 172.21.15.62:0/2766190890 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96080095f0 con 0x7f961410c8f0 2024-04-20T22:01:13.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.082+0000 7f9619200700 1 --2- 172.21.15.62:0/2766190890 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f961410c8f0 0x7f961419ca40 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f960000c930 tx=0x7f960000ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:01:13.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.082+0000 7f9612000700 1 -- 172.21.15.62:0/2766190890 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9600007ad0 con 0x7f961410c8f0 2024-04-20T22:01:13.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.083+0000 7f9612000700 1 -- 172.21.15.62:0/2766190890 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9600007c30 con 0x7f961410c8f0 2024-04-20T22:01:13.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.083+0000 7f9612000700 1 -- 172.21.15.62:0/2766190890 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9600018610 con 0x7f961410c8f0 2024-04-20T22:01:13.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.083+0000 7f961c600700 1 -- 172.21.15.62:0/2766190890 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96141a1020 con 0x7f961410c8f0 2024-04-20T22:01:13.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.083+0000 7f961c600700 1 -- 172.21.15.62:0/2766190890 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96141a1570 con 0x7f961410c8f0 2024-04-20T22:01:13.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.084+0000 7f961c600700 1 -- 172.21.15.62:0/2766190890 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9614109b20 con 0x7f961410c8f0 2024-04-20T22:01:13.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.085+0000 7f9612000700 1 -- 172.21.15.62:0/2766190890 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f960000f450 con 0x7f961410c8f0 2024-04-20T22:01:13.085 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.086+0000 7f9612000700 1 --2- 172.21.15.62:0/2766190890 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f95fc06ecc0 0x7f95fc071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:01:13.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.086+0000 7f9612000700 1 -- 172.21.15.62:0/2766190890 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f960008eff0 con 0x7f961410c8f0 2024-04-20T22:01:13.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.086+0000 7f9619c00700 1 --2- 172.21.15.62:0/2766190890 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f95fc06ecc0 0x7f95fc071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:01:13.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.087+0000 7f9619c00700 1 --2- 172.21.15.62:0/2766190890 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f95fc06ecc0 0x7f95fc071170 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f9608006010 tx=0x7f960801a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:01:13.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.091+0000 7f9612000700 1 -- 172.21.15.62:0/2766190890 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f9600057de0 con 0x7f961410c8f0 2024-04-20T22:01:13.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:01:12 smithi189 ceph-mon[57022]: pgmap v778: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:01:13.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:01:12 smithi189 ceph-mon[57022]: from='client.35404 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:01:13.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:01:12 smithi189 ceph-mon[57022]: from='client.35408 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:01:13.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.370+0000 7f961c600700 1 -- 172.21.15.62:0/2766190890 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f961404f2a0 con 0x7f961410c8f0 2024-04-20T22:01:13.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.371+0000 7f9612000700 1 -- 172.21.15.62:0/2766190890 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 27 v27) v1 ==== 76+0+1888 (secure 0 0 0) 0x7f960005b4a0 con 0x7f961410c8f0 2024-04-20T22:01:13.371 INFO:teuthology.orchestra.run.smithi062.stdout:e27 2024-04-20T22:01:13.372 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T22:01:13.372 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:01:13.372 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T22:01:13.372 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:01:13.372 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T22:01:13.372 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T22:01:13.372 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 20 2024-04-20T22:01:13.372 INFO:teuthology.orchestra.run.smithi062.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-20T22:01:13.372 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T22:01:13.372 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T21:42:01.559061+0000 2024-04-20T22:01:13.372 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T22:01:13.372 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T22:01:13.372 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T22:01:13.372 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T22:01:13.373 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T22:01:13.373 INFO:teuthology.orchestra.run.smithi062.stdout:max_xattr_size 65536 2024-04-20T22:01:13.373 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T22:01:13.373 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T22:01:13.373 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 81 2024-04-20T22:01:13.373 INFO:teuthology.orchestra.run.smithi062.stdout:snap_rstat 1 2024-04-20T22:01:13.373 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:01:13.373 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 2 2024-04-20T22:01:13.373 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1 2024-04-20T22:01:13.373 INFO:teuthology.orchestra.run.smithi062.stdout:up {} 2024-04-20T22:01:13.373 INFO:teuthology.orchestra.run.smithi062.stdout:failed 0,1 2024-04-20T22:01:13.373 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T22:01:13.373 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T22:01:13.373 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T22:01:13.373 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T22:01:13.373 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data enabled 2024-04-20T22:01:13.374 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T22:01:13.374 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T22:01:13.374 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 1 2024-04-20T22:01:13.374 INFO:teuthology.orchestra.run.smithi062.stdout:qdb_cluster leader: 0 members: 2024-04-20T22:01:13.374 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:01:13.374 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:01:13.374 INFO:teuthology.orchestra.run.smithi062.stdout:Standby daemons: 2024-04-20T22:01:13.374 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:01:13.374 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ryqblb{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6826/4113916205,v1:172.21.15.62:6827/4113916205] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:01:13.374 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ozizwi{-1:34476} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6828/3801975165,v1:172.21.15.62:6829/3801975165] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:01:13.374 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.xbgspw{-1:44285} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6826/3996040526,v1:172.21.15.189:6827/3996040526] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T22:01:13.374 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.dvimer{-1:44727} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6824/3576699232,v1:172.21.15.189:6825/3576699232] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:01:13.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.376+0000 7f961c600700 1 -- 172.21.15.62:0/2766190890 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f95fc06ecc0 msgr2=0x7f95fc071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:01:13.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.376+0000 7f961c600700 1 --2- 172.21.15.62:0/2766190890 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f95fc06ecc0 0x7f95fc071170 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f9608006010 tx=0x7f960801a040 comp rx=0 tx=0).stop 2024-04-20T22:01:13.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.376+0000 7f961c600700 1 -- 172.21.15.62:0/2766190890 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f961410c8f0 msgr2=0x7f961419ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:01:13.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.376+0000 7f961c600700 1 --2- 172.21.15.62:0/2766190890 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f961410c8f0 0x7f961419ca40 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f960000c930 tx=0x7f960000ccf0 comp rx=0 tx=0).stop 2024-04-20T22:01:13.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.376+0000 7f961c600700 1 -- 172.21.15.62:0/2766190890 shutdown_connections 2024-04-20T22:01:13.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.376+0000 7f961c600700 1 --2- 172.21.15.62:0/2766190890 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f95fc06ecc0 0x7f95fc071170 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:01:13.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.376+0000 7f961c600700 1 --2- 172.21.15.62:0/2766190890 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9614106b80 0x7f961419c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:01:13.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.376+0000 7f961c600700 1 --2- 172.21.15.62:0/2766190890 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f961410c8f0 0x7f961419ca40 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:01:13.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.376+0000 7f961c600700 1 -- 172.21.15.62:0/2766190890 >> 172.21.15.62:0/2766190890 conn(0x7f9614076060 msgr2=0x7f9614077aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:01:13.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.376+0000 7f961c600700 1 -- 172.21.15.62:0/2766190890 shutdown_connections 2024-04-20T22:01:13.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.376+0000 7f961c600700 1 -- 172.21.15.62:0/2766190890 wait complete. 2024-04-20T22:01:13.379 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 27 2024-04-20T22:01:13.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.542+0000 7f0cf5200700 1 -- 172.21.15.62:0/3336955964 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0cf0102470 msgr2=0x7f0cf01028e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:01:13.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.542+0000 7f0cf5200700 1 --2- 172.21.15.62:0/3336955964 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0cf0102470 0x7f0cf01028e0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f0cdc009910 tx=0x7f0cdc009c20 comp rx=0 tx=0).stop 2024-04-20T22:01:13.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.542+0000 7f0cf5200700 1 -- 172.21.15.62:0/3336955964 shutdown_connections 2024-04-20T22:01:13.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.542+0000 7f0cf5200700 1 --2- 172.21.15.62:0/3336955964 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0cf0102470 0x7f0cf01028e0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:01:13.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.542+0000 7f0cf5200700 1 --2- 172.21.15.62:0/3336955964 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0cf0108470 0x7f0cf0108840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:01:13.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.542+0000 7f0cf5200700 1 -- 172.21.15.62:0/3336955964 >> 172.21.15.62:0/3336955964 conn(0x7f0cf00fdf90 msgr2=0x7f0cf01003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:01:13.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.542+0000 7f0cf5200700 1 -- 172.21.15.62:0/3336955964 shutdown_connections 2024-04-20T22:01:13.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.542+0000 7f0cf5200700 1 -- 172.21.15.62:0/3336955964 wait complete. 2024-04-20T22:01:13.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.543+0000 7f0cf5200700 1 Processor -- start 2024-04-20T22:01:13.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.543+0000 7f0cf5200700 1 -- start start 2024-04-20T22:01:13.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.544+0000 7f0cf5200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0cf0102470 0x7f0cf01981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:01:13.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.544+0000 7f0cf5200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0cf0108470 0x7f0cf0198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:01:13.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.544+0000 7f0cf5200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf0198e10 con 0x7f0cf0108470 2024-04-20T22:01:13.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.544+0000 7f0cf5200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf019cba0 con 0x7f0cf0102470 2024-04-20T22:01:13.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.544+0000 7f0ced400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0cf0108470 0x7f0cf0198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:01:13.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.544+0000 7f0ced400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0cf0108470 0x7f0cf0198730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35600/0 (socket says 172.21.15.62:35600) 2024-04-20T22:01:13.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.544+0000 7f0ced400700 1 -- 172.21.15.62:0/85084259 learned_addr learned my addr 172.21.15.62:0/85084259 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:01:13.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.544+0000 7f0cede00700 1 --2- 172.21.15.62:0/85084259 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0cf0102470 0x7f0cf01981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:01:13.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.545+0000 7f0ced400700 1 -- 172.21.15.62:0/85084259 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0cf0102470 msgr2=0x7f0cf01981f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:01:13.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.545+0000 7f0ced400700 1 --2- 172.21.15.62:0/85084259 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0cf0102470 0x7f0cf01981f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:01:13.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.545+0000 7f0ced400700 1 -- 172.21.15.62:0/85084259 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cdc0095f0 con 0x7f0cf0108470 2024-04-20T22:01:13.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.545+0000 7f0ced400700 1 --2- 172.21.15.62:0/85084259 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0cf0108470 0x7f0cf0198730 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f0cdc0092e0 tx=0x7f0cdc00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:01:13.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.545+0000 7f0ce6a00700 1 -- 172.21.15.62:0/85084259 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0cdc01d070 con 0x7f0cf0108470 2024-04-20T22:01:13.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.545+0000 7f0ce6a00700 1 -- 172.21.15.62:0/85084259 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0cdc005e00 con 0x7f0cf0108470 2024-04-20T22:01:13.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.545+0000 7f0ce6a00700 1 -- 172.21.15.62:0/85084259 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0cdc017400 con 0x7f0cf0108470 2024-04-20T22:01:13.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.545+0000 7f0cf5200700 1 -- 172.21.15.62:0/85084259 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cf019ce20 con 0x7f0cf0108470 2024-04-20T22:01:13.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.545+0000 7f0cf5200700 1 -- 172.21.15.62:0/85084259 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cf019d310 con 0x7f0cf0108470 2024-04-20T22:01:13.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.547+0000 7f0cf5200700 1 -- 172.21.15.62:0/85084259 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cf010a970 con 0x7f0cf0108470 2024-04-20T22:01:13.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.548+0000 7f0ce6a00700 1 -- 172.21.15.62:0/85084259 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f0cdc00f460 con 0x7f0cf0108470 2024-04-20T22:01:13.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.549+0000 7f0ce6a00700 1 --2- 172.21.15.62:0/85084259 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f0cd406ecc0 0x7f0cd4071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:01:13.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.549+0000 7f0ce6a00700 1 -- 172.21.15.62:0/85084259 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f0cdc090330 con 0x7f0cf0108470 2024-04-20T22:01:13.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.549+0000 7f0cede00700 1 --2- 172.21.15.62:0/85084259 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f0cd406ecc0 0x7f0cd4071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:01:13.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.550+0000 7f0cede00700 1 --2- 172.21.15.62:0/85084259 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f0cd406ecc0 0x7f0cd4071170 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f0ce0005950 tx=0x7f0ce00058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:01:13.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.554+0000 7f0ce6a00700 1 -- 172.21.15.62:0/85084259 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f0cdc0590c0 con 0x7f0cf0108470 2024-04-20T22:01:13.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.815+0000 7f0cf5200700 1 -- 172.21.15.62:0/85084259 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0cf01995c0 con 0x7f0cd406ecc0 2024-04-20T22:01:13.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.818+0000 7f0ce6a00700 1 -- 172.21.15.62:0/85084259 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f0cf01995c0 con 0x7f0cd406ecc0 2024-04-20T22:01:13.817 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:01:13.818 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T22:01:13.818 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T22:01:13.818 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-20T22:01:13.818 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T22:01:13.818 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr", 2024-04-20T22:01:13.818 INFO:teuthology.orchestra.run.smithi062.stdout: "crash", 2024-04-20T22:01:13.818 INFO:teuthology.orchestra.run.smithi062.stdout: "mon", 2024-04-20T22:01:13.818 INFO:teuthology.orchestra.run.smithi062.stdout: "osd" 2024-04-20T22:01:13.818 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T22:01:13.818 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "15/23 daemons upgraded", 2024-04-20T22:01:13.818 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading mds daemons", 2024-04-20T22:01:13.818 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T22:01:13.818 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:01:13.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.821+0000 7f0cf5200700 1 -- 172.21.15.62:0/85084259 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f0cd406ecc0 msgr2=0x7f0cd4071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:01:13.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.821+0000 7f0cf5200700 1 --2- 172.21.15.62:0/85084259 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f0cd406ecc0 0x7f0cd4071170 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f0ce0005950 tx=0x7f0ce00058e0 comp rx=0 tx=0).stop 2024-04-20T22:01:13.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.821+0000 7f0cf5200700 1 -- 172.21.15.62:0/85084259 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0cf0108470 msgr2=0x7f0cf0198730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:01:13.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.821+0000 7f0cf5200700 1 --2- 172.21.15.62:0/85084259 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0cf0108470 0x7f0cf0198730 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f0cdc0092e0 tx=0x7f0cdc00be20 comp rx=0 tx=0).stop 2024-04-20T22:01:13.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.821+0000 7f0cf5200700 1 -- 172.21.15.62:0/85084259 shutdown_connections 2024-04-20T22:01:13.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.821+0000 7f0cf5200700 1 --2- 172.21.15.62:0/85084259 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f0cd406ecc0 0x7f0cd4071170 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:01:13.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.821+0000 7f0cf5200700 1 --2- 172.21.15.62:0/85084259 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0cf0102470 0x7f0cf01981f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:01:13.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.821+0000 7f0cf5200700 1 --2- 172.21.15.62:0/85084259 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0cf0108470 0x7f0cf0198730 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:01:13.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.821+0000 7f0cf5200700 1 -- 172.21.15.62:0/85084259 >> 172.21.15.62:0/85084259 conn(0x7f0cf00fdf90 msgr2=0x7f0cf00ff3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:01:13.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.821+0000 7f0cf5200700 1 -- 172.21.15.62:0/85084259 shutdown_connections 2024-04-20T22:01:13.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.822+0000 7f0cf5200700 1 -- 172.21.15.62:0/85084259 wait complete. 2024-04-20T22:01:13.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.969+0000 7fcd81000700 1 -- 172.21.15.62:0/3863722685 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcd7c069140 msgr2=0x7fcd7c104eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:01:13.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.969+0000 7fcd81000700 1 --2- 172.21.15.62:0/3863722685 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcd7c069140 0x7fcd7c104eb0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fcd68009910 tx=0x7fcd68009c20 comp rx=0 tx=0).stop 2024-04-20T22:01:13.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.969+0000 7fcd81000700 1 -- 172.21.15.62:0/3863722685 shutdown_connections 2024-04-20T22:01:13.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.969+0000 7fcd81000700 1 --2- 172.21.15.62:0/3863722685 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcd7c069140 0x7fcd7c104eb0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:01:13.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.969+0000 7fcd81000700 1 --2- 172.21.15.62:0/3863722685 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcd7c0687a0 0x7fcd7c068b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:01:13.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.969+0000 7fcd81000700 1 -- 172.21.15.62:0/3863722685 >> 172.21.15.62:0/3863722685 conn(0x7fcd7c075130 msgr2=0x7fcd7c075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:01:13.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.969+0000 7fcd81000700 1 -- 172.21.15.62:0/3863722685 shutdown_connections 2024-04-20T22:01:13.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.969+0000 7fcd81000700 1 -- 172.21.15.62:0/3863722685 wait complete. 2024-04-20T22:01:13.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.970+0000 7fcd81000700 1 Processor -- start 2024-04-20T22:01:13.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.970+0000 7fcd81000700 1 -- start start 2024-04-20T22:01:13.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.971+0000 7fcd81000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcd7c0687a0 0x7fcd7c198180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:01:13.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.971+0000 7fcd81000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcd7c069140 0x7fcd7c1986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:01:13.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.971+0000 7fcd81000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd7c198d10 con 0x7fcd7c069140 2024-04-20T22:01:13.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.971+0000 7fcd81000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd7c198e50 con 0x7fcd7c0687a0 2024-04-20T22:01:13.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.971+0000 7fcd79400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcd7c069140 0x7fcd7c1986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:01:13.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.971+0000 7fcd79400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcd7c069140 0x7fcd7c1986c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35626/0 (socket says 172.21.15.62:35626) 2024-04-20T22:01:13.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.971+0000 7fcd79400700 1 -- 172.21.15.62:0/4160973004 learned_addr learned my addr 172.21.15.62:0/4160973004 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:01:13.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.971+0000 7fcd79e00700 1 --2- 172.21.15.62:0/4160973004 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcd7c0687a0 0x7fcd7c198180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:01:13.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.971+0000 7fcd79400700 1 -- 172.21.15.62:0/4160973004 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcd7c0687a0 msgr2=0x7fcd7c198180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:01:13.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.971+0000 7fcd79400700 1 --2- 172.21.15.62:0/4160973004 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcd7c0687a0 0x7fcd7c198180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:01:13.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.971+0000 7fcd79400700 1 -- 172.21.15.62:0/4160973004 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd680095f0 con 0x7fcd7c069140 2024-04-20T22:01:13.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.971+0000 7fcd79e00700 1 --2- 172.21.15.62:0/4160973004 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcd7c0687a0 0x7fcd7c198180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T22:01:13.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.972+0000 7fcd79400700 1 --2- 172.21.15.62:0/4160973004 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcd7c069140 0x7fcd7c1986c0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fcd68005f50 tx=0x7fcd6800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:01:13.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.972+0000 7fcd72a00700 1 -- 172.21.15.62:0/4160973004 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcd6801d070 con 0x7fcd7c069140 2024-04-20T22:01:13.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.972+0000 7fcd72a00700 1 -- 172.21.15.62:0/4160973004 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcd6800f460 con 0x7fcd7c069140 2024-04-20T22:01:13.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.972+0000 7fcd72a00700 1 -- 172.21.15.62:0/4160973004 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcd68017630 con 0x7fcd7c069140 2024-04-20T22:01:13.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.972+0000 7fcd81000700 1 -- 172.21.15.62:0/4160973004 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd7c19cc40 con 0x7fcd7c069140 2024-04-20T22:01:13.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.972+0000 7fcd81000700 1 -- 172.21.15.62:0/4160973004 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd7c19d130 con 0x7fcd7c069140 2024-04-20T22:01:13.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.973+0000 7fcd81000700 1 -- 172.21.15.62:0/4160973004 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd7c04f2a0 con 0x7fcd7c069140 2024-04-20T22:01:13.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.974+0000 7fcd72a00700 1 -- 172.21.15.62:0/4160973004 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fcd6800f5d0 con 0x7fcd7c069140 2024-04-20T22:01:13.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.975+0000 7fcd72a00700 1 --2- 172.21.15.62:0/4160973004 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fcd6006ecc0 0x7fcd60071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:01:13.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.975+0000 7fcd72a00700 1 -- 172.21.15.62:0/4160973004 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fcd68090420 con 0x7fcd7c069140 2024-04-20T22:01:13.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.976+0000 7fcd79e00700 1 --2- 172.21.15.62:0/4160973004 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fcd6006ecc0 0x7fcd60071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:01:13.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.976+0000 7fcd79e00700 1 --2- 172.21.15.62:0/4160973004 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fcd6006ecc0 0x7fcd60071170 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fcd6c005fd0 tx=0x7fcd6c005e20 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:01:13.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:13.980+0000 7fcd72a00700 1 -- 172.21.15.62:0/4160973004 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fcd680591b0 con 0x7fcd7c069140 2024-04-20T22:01:14.075 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:01:13 smithi062 ceph-mon[76322]: from='client.35412 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:01:14.075 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:01:13 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/2372141257' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:01:14.075 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:01:13 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/2766190890' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:01:14.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:01:13 smithi189 ceph-mon[57022]: from='client.35412 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:01:14.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:01:13 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/2372141257' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:01:14.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:01:13 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/2766190890' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:01:14.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:14.301+0000 7fcd81000700 1 -- 172.21.15.62:0/4160973004 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fcd7c1994e0 con 0x7fcd7c069140 2024-04-20T22:01:14.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:14.302+0000 7fcd72a00700 1 -- 172.21.15.62:0/4160973004 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7fcd6805c870 con 0x7fcd7c069140 2024-04-20T22:01:14.302 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-20T22:01:14.302 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-20T22:01:14.302 INFO:teuthology.orchestra.run.smithi062.stdout: host smithi062 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T22:01:14.303 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-20T22:01:14.303 INFO:teuthology.orchestra.run.smithi062.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi062 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpq7y7oa_r:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:01:14.303 INFO:teuthology.orchestra.run.smithi062.stdout:Traceback (most recent call last): 2024-04-20T22:01:14.303 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-20T22:01:14.303 INFO:teuthology.orchestra.run.smithi062.stdout: return _run_code(code, main_globals, None, 2024-04-20T22:01:14.303 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-20T22:01:14.303 INFO:teuthology.orchestra.run.smithi062.stdout: exec(code, run_globals) 2024-04-20T22:01:14.303 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5618, in 2024-04-20T22:01:14.303 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5606, in main 2024-04-20T22:01:14.303 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 408, in _infer_config 2024-04-20T22:01:14.303 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 323, in _infer_fsid 2024-04-20T22:01:14.303 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 436, in _infer_image 2024-04-20T22:01:14.304 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 310, in _validate_fsid 2024-04-20T22:01:14.304 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 3359, in command_ceph_volume 2024-04-20T22:01:14.304 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-20T22:01:14.304 INFO:teuthology.orchestra.run.smithi062.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi062 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpq7y7oa_r:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:01:14.304 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-20T22:01:14.304 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is degraded 2024-04-20T22:01:14.304 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T22:01:14.304 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T22:01:14.304 INFO:teuthology.orchestra.run.smithi062.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-20T22:01:14.304 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-20T22:01:14.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:14.305+0000 7fcd81000700 1 -- 172.21.15.62:0/4160973004 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fcd6006ecc0 msgr2=0x7fcd60071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:01:14.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:14.305+0000 7fcd81000700 1 --2- 172.21.15.62:0/4160973004 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fcd6006ecc0 0x7fcd60071170 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fcd6c005fd0 tx=0x7fcd6c005e20 comp rx=0 tx=0).stop 2024-04-20T22:01:14.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:14.305+0000 7fcd81000700 1 -- 172.21.15.62:0/4160973004 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcd7c069140 msgr2=0x7fcd7c1986c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:01:14.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:14.305+0000 7fcd81000700 1 --2- 172.21.15.62:0/4160973004 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcd7c069140 0x7fcd7c1986c0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fcd68005f50 tx=0x7fcd6800be20 comp rx=0 tx=0).stop 2024-04-20T22:01:14.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:14.305+0000 7fcd81000700 1 -- 172.21.15.62:0/4160973004 shutdown_connections 2024-04-20T22:01:14.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:14.305+0000 7fcd81000700 1 --2- 172.21.15.62:0/4160973004 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fcd6006ecc0 0x7fcd60071170 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:01:14.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:14.305+0000 7fcd81000700 1 --2- 172.21.15.62:0/4160973004 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcd7c0687a0 0x7fcd7c198180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:01:14.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:14.305+0000 7fcd81000700 1 --2- 172.21.15.62:0/4160973004 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcd7c069140 0x7fcd7c1986c0 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:01:14.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:14.305+0000 7fcd81000700 1 -- 172.21.15.62:0/4160973004 >> 172.21.15.62:0/4160973004 conn(0x7fcd7c075130 msgr2=0x7fcd7c101e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:01:14.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:14.305+0000 7fcd81000700 1 -- 172.21.15.62:0/4160973004 shutdown_connections 2024-04-20T22:01:14.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:14.305+0000 7fcd81000700 1 -- 172.21.15.62:0/4160973004 wait complete. 2024-04-20T22:01:15.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:01:14 smithi189 ceph-mon[57022]: pgmap v779: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:01:15.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:01:14 smithi189 ceph-mon[57022]: from='client.35424 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:01:15.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:01:14 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/4160973004' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:01:15.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:01:14 smithi062 ceph-mon[76322]: pgmap v779: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:01:15.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:01:14 smithi062 ceph-mon[76322]: from='client.35424 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:01:15.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:01:14 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/4160973004' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:01:17.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:01:16 smithi189 ceph-mon[57022]: pgmap v780: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:01:17.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:01:16 smithi062 ceph-mon[76322]: pgmap v780: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:01:19.232 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:01:18 smithi062 ceph-mon[76322]: pgmap v781: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:01:19.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:01:18 smithi189 ceph-mon[57022]: pgmap v781: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:01:20.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:01:19 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:01:20.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:01:19 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:01:21.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:01:20 smithi189 ceph-mon[57022]: pgmap v782: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:01:21.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:01:20 smithi062 ceph-mon[76322]: pgmap v782: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:01:23.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:01:22 smithi189 ceph-mon[57022]: pgmap v783: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:01:23.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:01:22 smithi062 ceph-mon[76322]: pgmap v783: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:01:25.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:01:24 smithi189 ceph-mon[57022]: pgmap v784: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:01:25.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:01:24 smithi062 ceph-mon[76322]: pgmap v784: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:01:27.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:01:26 smithi189 ceph-mon[57022]: pgmap v785: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:01:27.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:01:26 smithi062 ceph-mon[76322]: pgmap v785: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:01:29.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:01:28 smithi189 ceph-mon[57022]: pgmap v786: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:01:29.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:01:28 smithi062 ceph-mon[76322]: pgmap v786: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:01:31.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:01:30 smithi189 ceph-mon[57022]: pgmap v787: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:01:31.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:01:30 smithi062 ceph-mon[76322]: pgmap v787: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:01:33.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:01:32 smithi189 ceph-mon[57022]: pgmap v788: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:01:33.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:01:32 smithi062 ceph-mon[76322]: pgmap v788: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:01:35.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:01:34 smithi062 ceph-mon[76322]: pgmap v789: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:01:35.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:01:34 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:01:35.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:01:34 smithi189 ceph-mon[57022]: pgmap v789: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:01:35.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:01:34 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:01:37.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:01:36 smithi189 ceph-mon[57022]: pgmap v790: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:01:37.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:01:36 smithi062 ceph-mon[76322]: pgmap v790: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:01:39.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:01:38 smithi189 ceph-mon[57022]: pgmap v791: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:01:39.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:01:38 smithi062 ceph-mon[76322]: pgmap v791: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:01:41.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:01:40 smithi189 ceph-mon[57022]: pgmap v792: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:01:41.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:01:40 smithi062 ceph-mon[76322]: pgmap v792: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:01:43.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:01:42 smithi189 ceph-mon[57022]: pgmap v793: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:01:43.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:01:42 smithi062 ceph-mon[76322]: pgmap v793: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:01:44.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:44.481+0000 7fa186a00700 1 -- 172.21.15.62:0/92605551 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa180073a00 msgr2=0x7fa18010c580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:01:44.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:44.481+0000 7fa186a00700 1 --2- 172.21.15.62:0/92605551 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa180073a00 0x7fa18010c580 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fa168009910 tx=0x7fa168009c20 comp rx=0 tx=0).stop 2024-04-20T22:01:44.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:44.481+0000 7fa186a00700 1 -- 172.21.15.62:0/92605551 shutdown_connections 2024-04-20T22:01:44.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:44.481+0000 7fa186a00700 1 --2- 172.21.15.62:0/92605551 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa180073a00 0x7fa18010c580 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:01:44.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:44.481+0000 7fa186a00700 1 --2- 172.21.15.62:0/92605551 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1800730f0 0x7fa1800734c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:01:44.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:44.481+0000 7fa186a00700 1 -- 172.21.15.62:0/92605551 >> 172.21.15.62:0/92605551 conn(0x7fa1800fbda0 msgr2=0x7fa1800fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:01:44.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:44.482+0000 7fa186a00700 1 -- 172.21.15.62:0/92605551 shutdown_connections 2024-04-20T22:01:44.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:44.482+0000 7fa186a00700 1 -- 172.21.15.62:0/92605551 wait complete. 2024-04-20T22:01:44.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:44.483+0000 7fa186a00700 1 Processor -- start 2024-04-20T22:01:44.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:44.483+0000 7fa186a00700 1 -- start start 2024-04-20T22:01:44.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:44.483+0000 7fa186a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1800730f0 0x7fa180197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:01:44.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:44.483+0000 7fa186a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa180073a00 0x7fa180198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:01:44.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:44.483+0000 7fa186a00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa18019c970 con 0x7fa1800730f0 2024-04-20T22:01:44.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:44.483+0000 7fa186a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa18019cae0 con 0x7fa180073a00 2024-04-20T22:01:44.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:44.484+0000 7fa17fe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1800730f0 0x7fa180197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:01:44.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:44.484+0000 7fa17fe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1800730f0 0x7fa180197fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51930/0 (socket says 172.21.15.62:51930) 2024-04-20T22:01:44.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:44.484+0000 7fa17fe00700 1 -- 172.21.15.62:0/1431881214 learned_addr learned my addr 172.21.15.62:0/1431881214 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:01:44.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:44.484+0000 7fa177e00700 1 --2- 172.21.15.62:0/1431881214 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa180073a00 0x7fa180198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:01:44.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:44.484+0000 7fa17fe00700 1 -- 172.21.15.62:0/1431881214 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa180073a00 msgr2=0x7fa180198630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:01:44.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:44.484+0000 7fa17fe00700 1 --2- 172.21.15.62:0/1431881214 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa180073a00 0x7fa180198630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:01:44.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:44.484+0000 7fa17fe00700 1 -- 172.21.15.62:0/1431881214 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1680095f0 con 0x7fa1800730f0 2024-04-20T22:01:44.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:44.484+0000 7fa17fe00700 1 --2- 172.21.15.62:0/1431881214 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1800730f0 0x7fa180197fc0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fa17000c8a0 tx=0x7fa17000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:01:44.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:44.485+0000 7fa17d600700 1 -- 172.21.15.62:0/1431881214 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa17000cea0 con 0x7fa1800730f0 2024-04-20T22:01:44.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:44.485+0000 7fa17d600700 1 -- 172.21.15.62:0/1431881214 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa170004830 con 0x7fa1800730f0 2024-04-20T22:01:44.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:44.485+0000 7fa186a00700 1 -- 172.21.15.62:0/1431881214 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa18019cdc0 con 0x7fa1800730f0 2024-04-20T22:01:44.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:44.485+0000 7fa17d600700 1 -- 172.21.15.62:0/1431881214 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa1700055b0 con 0x7fa1800730f0 2024-04-20T22:01:44.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:44.485+0000 7fa186a00700 1 -- 172.21.15.62:0/1431881214 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa18019d310 con 0x7fa1800730f0 2024-04-20T22:01:44.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:44.486+0000 7fa186a00700 1 -- 172.21.15.62:0/1431881214 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa180109c80 con 0x7fa1800730f0 2024-04-20T22:01:44.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:44.487+0000 7fa17d600700 1 -- 172.21.15.62:0/1431881214 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fa1700049a0 con 0x7fa1800730f0 2024-04-20T22:01:44.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:44.488+0000 7fa17d600700 1 --2- 172.21.15.62:0/1431881214 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa16406ecc0 0x7fa164071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:01:44.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:44.488+0000 7fa17d600700 1 -- 172.21.15.62:0/1431881214 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fa17008ebf0 con 0x7fa1800730f0 2024-04-20T22:01:44.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:44.489+0000 7fa177e00700 1 --2- 172.21.15.62:0/1431881214 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa16406ecc0 0x7fa164071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:01:44.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:44.490+0000 7fa177e00700 1 --2- 172.21.15.62:0/1431881214 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa16406ecc0 0x7fa164071170 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fa168006010 tx=0x7fa16801a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:01:44.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:44.494+0000 7fa17d600700 1 -- 172.21.15.62:0/1431881214 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fa170057900 con 0x7fa1800730f0 2024-04-20T22:01:44.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:44.751+0000 7fa186a00700 1 -- 172.21.15.62:0/1431881214 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa180199400 con 0x7fa16406ecc0 2024-04-20T22:01:44.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:44.754+0000 7fa17d600700 1 -- 172.21.15.62:0/1431881214 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fa180199400 con 0x7fa16406ecc0 2024-04-20T22:01:44.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:44.757+0000 7fa186a00700 1 -- 172.21.15.62:0/1431881214 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa16406ecc0 msgr2=0x7fa164071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:01:44.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:44.757+0000 7fa186a00700 1 --2- 172.21.15.62:0/1431881214 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa16406ecc0 0x7fa164071170 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fa168006010 tx=0x7fa16801a040 comp rx=0 tx=0).stop 2024-04-20T22:01:44.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:44.757+0000 7fa186a00700 1 -- 172.21.15.62:0/1431881214 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1800730f0 msgr2=0x7fa180197fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:01:44.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:44.757+0000 7fa186a00700 1 --2- 172.21.15.62:0/1431881214 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1800730f0 0x7fa180197fc0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fa17000c8a0 tx=0x7fa17000cc60 comp rx=0 tx=0).stop 2024-04-20T22:01:44.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:44.758+0000 7fa186a00700 1 -- 172.21.15.62:0/1431881214 shutdown_connections 2024-04-20T22:01:44.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:44.758+0000 7fa186a00700 1 --2- 172.21.15.62:0/1431881214 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa16406ecc0 0x7fa164071170 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:01:44.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:44.758+0000 7fa186a00700 1 --2- 172.21.15.62:0/1431881214 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1800730f0 0x7fa180197fc0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:01:44.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:44.758+0000 7fa186a00700 1 --2- 172.21.15.62:0/1431881214 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa180073a00 0x7fa180198630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:01:44.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:44.758+0000 7fa186a00700 1 -- 172.21.15.62:0/1431881214 >> 172.21.15.62:0/1431881214 conn(0x7fa1800fbda0 msgr2=0x7fa180106dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:01:44.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:44.758+0000 7fa186a00700 1 -- 172.21.15.62:0/1431881214 shutdown_connections 2024-04-20T22:01:44.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:44.758+0000 7fa186a00700 1 -- 172.21.15.62:0/1431881214 wait complete. 2024-04-20T22:01:44.773 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T22:01:44.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:44.922+0000 7f72b5e00700 1 -- 172.21.15.62:0/1707644997 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72b0073090 msgr2=0x7f72b0073460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:01:44.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:44.922+0000 7f72b5e00700 1 --2- 172.21.15.62:0/1707644997 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72b0073090 0x7f72b0073460 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f72a0009940 tx=0x7f72a0009c50 comp rx=0 tx=0).stop 2024-04-20T22:01:44.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:44.922+0000 7f72b5e00700 1 -- 172.21.15.62:0/1707644997 shutdown_connections 2024-04-20T22:01:44.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:44.922+0000 7f72b5e00700 1 --2- 172.21.15.62:0/1707644997 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72b00739a0 0x7f72b010c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:01:44.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:44.922+0000 7f72b5e00700 1 --2- 172.21.15.62:0/1707644997 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72b0073090 0x7f72b0073460 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:01:44.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:44.922+0000 7f72b5e00700 1 -- 172.21.15.62:0/1707644997 >> 172.21.15.62:0/1707644997 conn(0x7f72b00fbda0 msgr2=0x7f72b00fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:01:44.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:44.922+0000 7f72b5e00700 1 -- 172.21.15.62:0/1707644997 shutdown_connections 2024-04-20T22:01:44.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:44.922+0000 7f72b5e00700 1 -- 172.21.15.62:0/1707644997 wait complete. 2024-04-20T22:01:44.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:44.923+0000 7f72b5e00700 1 Processor -- start 2024-04-20T22:01:44.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:44.923+0000 7f72b5e00700 1 -- start start 2024-04-20T22:01:44.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:44.924+0000 7f72b5e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72b0073090 0x7f72b0198180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:01:44.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:44.924+0000 7f72b5e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72b00739a0 0x7f72b01986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:01:44.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:44.924+0000 7f72b5e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72b0198da0 con 0x7f72b00739a0 2024-04-20T22:01:44.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:44.924+0000 7f72b5e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72b019cb30 con 0x7f72b0073090 2024-04-20T22:01:44.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:44.924+0000 7f72ae000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72b00739a0 0x7f72b01986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:01:44.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:44.924+0000 7f72ae000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72b00739a0 0x7f72b01986c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51946/0 (socket says 172.21.15.62:51946) 2024-04-20T22:01:44.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:44.924+0000 7f72ae000700 1 -- 172.21.15.62:0/1233615621 learned_addr learned my addr 172.21.15.62:0/1233615621 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:01:44.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:44.925+0000 7f72aea00700 1 --2- 172.21.15.62:0/1233615621 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72b0073090 0x7f72b0198180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:01:44.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:44.925+0000 7f72ae000700 1 -- 172.21.15.62:0/1233615621 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72b0073090 msgr2=0x7f72b0198180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:01:44.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:44.925+0000 7f72ae000700 1 --2- 172.21.15.62:0/1233615621 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72b0073090 0x7f72b0198180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:01:44.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:44.925+0000 7f72ae000700 1 -- 172.21.15.62:0/1233615621 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72a00095f0 con 0x7f72b00739a0 2024-04-20T22:01:44.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:44.925+0000 7f72ae000700 1 --2- 172.21.15.62:0/1233615621 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72b00739a0 0x7f72b01986c0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f729c00c8a0 tx=0x7f729c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:01:44.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:44.927+0000 7f72b4a00700 1 -- 172.21.15.62:0/1233615621 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f729c00cea0 con 0x7f72b00739a0 2024-04-20T22:01:44.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:44.927+0000 7f72b4a00700 1 -- 172.21.15.62:0/1233615621 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f729c004830 con 0x7f72b00739a0 2024-04-20T22:01:44.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:44.927+0000 7f72b4a00700 1 -- 172.21.15.62:0/1233615621 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f729c0055b0 con 0x7f72b00739a0 2024-04-20T22:01:44.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:44.927+0000 7f72b5e00700 1 -- 172.21.15.62:0/1233615621 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72b019ce10 con 0x7f72b00739a0 2024-04-20T22:01:44.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:44.927+0000 7f72b5e00700 1 -- 172.21.15.62:0/1233615621 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72b019d360 con 0x7f72b00739a0 2024-04-20T22:01:44.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:44.928+0000 7f72b5e00700 1 -- 172.21.15.62:0/1233615621 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72b0109ce0 con 0x7f72b00739a0 2024-04-20T22:01:44.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:44.929+0000 7f72b4a00700 1 -- 172.21.15.62:0/1233615621 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f729c0049a0 con 0x7f72b00739a0 2024-04-20T22:01:44.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:44.930+0000 7f72b4a00700 1 --2- 172.21.15.62:0/1233615621 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f729406ecc0 0x7f7294071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:01:44.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:44.930+0000 7f72b4a00700 1 -- 172.21.15.62:0/1233615621 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f729c08eaf0 con 0x7f72b00739a0 2024-04-20T22:01:44.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:44.930+0000 7f72aea00700 1 --2- 172.21.15.62:0/1233615621 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f729406ecc0 0x7f7294071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:01:44.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:44.931+0000 7f72aea00700 1 --2- 172.21.15.62:0/1233615621 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f729406ecc0 0x7f7294071170 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f72a0006010 tx=0x7f72a0005a90 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:01:44.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:44.935+0000 7f72b4a00700 1 -- 172.21.15.62:0/1233615621 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f729c057880 con 0x7f72b00739a0 2024-04-20T22:01:45.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:01:44 smithi062 ceph-mon[76322]: pgmap v794: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:01:45.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:45.183+0000 7f72b5e00700 1 -- 172.21.15.62:0/1233615621 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f72b01995f0 con 0x7f729406ecc0 2024-04-20T22:01:45.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:45.186+0000 7f72b4a00700 1 -- 172.21.15.62:0/1233615621 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f72b01995f0 con 0x7f729406ecc0 2024-04-20T22:01:45.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:45.189+0000 7f72b5e00700 1 -- 172.21.15.62:0/1233615621 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f729406ecc0 msgr2=0x7f7294071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:01:45.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:45.189+0000 7f72b5e00700 1 --2- 172.21.15.62:0/1233615621 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f729406ecc0 0x7f7294071170 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f72a0006010 tx=0x7f72a0005a90 comp rx=0 tx=0).stop 2024-04-20T22:01:45.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:45.189+0000 7f72b5e00700 1 -- 172.21.15.62:0/1233615621 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72b00739a0 msgr2=0x7f72b01986c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:01:45.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:45.189+0000 7f72b5e00700 1 --2- 172.21.15.62:0/1233615621 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72b00739a0 0x7f72b01986c0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f729c00c8a0 tx=0x7f729c00cc60 comp rx=0 tx=0).stop 2024-04-20T22:01:45.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:45.189+0000 7f72b5e00700 1 -- 172.21.15.62:0/1233615621 shutdown_connections 2024-04-20T22:01:45.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:45.189+0000 7f72b5e00700 1 --2- 172.21.15.62:0/1233615621 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f729406ecc0 0x7f7294071170 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:01:45.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:45.189+0000 7f72b5e00700 1 --2- 172.21.15.62:0/1233615621 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72b0073090 0x7f72b0198180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:01:45.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:45.189+0000 7f72b5e00700 1 --2- 172.21.15.62:0/1233615621 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72b00739a0 0x7f72b01986c0 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:01:45.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:45.189+0000 7f72b5e00700 1 -- 172.21.15.62:0/1233615621 >> 172.21.15.62:0/1233615621 conn(0x7f72b00fbda0 msgr2=0x7f72b0106e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:01:45.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:45.190+0000 7f72b5e00700 1 -- 172.21.15.62:0/1233615621 shutdown_connections 2024-04-20T22:01:45.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:45.190+0000 7f72b5e00700 1 -- 172.21.15.62:0/1233615621 wait complete. 2024-04-20T22:01:45.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:01:44 smithi189 ceph-mon[57022]: pgmap v794: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:01:45.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:45.353+0000 7f03be800700 1 -- 172.21.15.62:0/3211694853 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f03b810c8f0 msgr2=0x7f03b810ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:01:45.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:45.353+0000 7f03be800700 1 --2- 172.21.15.62:0/3211694853 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f03b810c8f0 0x7f03b810ccc0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f03a8009910 tx=0x7f03a8009c20 comp rx=0 tx=0).stop 2024-04-20T22:01:45.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:45.353+0000 7f03be800700 1 -- 172.21.15.62:0/3211694853 shutdown_connections 2024-04-20T22:01:45.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:45.353+0000 7f03be800700 1 --2- 172.21.15.62:0/3211694853 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03b8106b80 0x7f03b8106ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:01:45.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:45.353+0000 7f03be800700 1 --2- 172.21.15.62:0/3211694853 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f03b810c8f0 0x7f03b810ccc0 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:01:45.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:45.353+0000 7f03be800700 1 -- 172.21.15.62:0/3211694853 >> 172.21.15.62:0/3211694853 conn(0x7f03b8076060 msgr2=0x7f03b8078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:01:45.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:45.353+0000 7f03be800700 1 -- 172.21.15.62:0/3211694853 shutdown_connections 2024-04-20T22:01:45.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:45.353+0000 7f03be800700 1 -- 172.21.15.62:0/3211694853 wait complete. 2024-04-20T22:01:45.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:45.354+0000 7f03be800700 1 Processor -- start 2024-04-20T22:01:45.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:45.355+0000 7f03be800700 1 -- start start 2024-04-20T22:01:45.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:45.355+0000 7f03be800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03b8106b80 0x7f03b819c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:01:45.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:45.355+0000 7f03be800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f03b810c8f0 0x7f03b819ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:01:45.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:45.355+0000 7f03be800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03b819d120 con 0x7f03b810c8f0 2024-04-20T22:01:45.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:45.355+0000 7f03be800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03b81a0eb0 con 0x7f03b8106b80 2024-04-20T22:01:45.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:45.355+0000 7f03b6a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f03b810c8f0 0x7f03b819ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:01:45.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:45.355+0000 7f03b6a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f03b810c8f0 0x7f03b819ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51960/0 (socket says 172.21.15.62:51960) 2024-04-20T22:01:45.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:45.356+0000 7f03b6a00700 1 -- 172.21.15.62:0/2417918449 learned_addr learned my addr 172.21.15.62:0/2417918449 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:01:45.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:45.356+0000 7f03b7400700 1 --2- 172.21.15.62:0/2417918449 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03b8106b80 0x7f03b819c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:01:45.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:45.356+0000 7f03b6a00700 1 -- 172.21.15.62:0/2417918449 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03b8106b80 msgr2=0x7f03b819c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:01:45.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:45.356+0000 7f03b6a00700 1 --2- 172.21.15.62:0/2417918449 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03b8106b80 0x7f03b819c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:01:45.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:45.356+0000 7f03b6a00700 1 -- 172.21.15.62:0/2417918449 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03a80095f0 con 0x7f03b810c8f0 2024-04-20T22:01:45.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:45.356+0000 7f03b7400700 1 --2- 172.21.15.62:0/2417918449 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03b8106b80 0x7f03b819c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T22:01:45.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:45.356+0000 7f03b6a00700 1 --2- 172.21.15.62:0/2417918449 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f03b810c8f0 0x7f03b819ca40 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f03a400d8d0 tx=0x7f03a400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:01:45.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:45.357+0000 7f03afe00700 1 -- 172.21.15.62:0/2417918449 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f03a4009940 con 0x7f03b810c8f0 2024-04-20T22:01:45.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:45.357+0000 7f03afe00700 1 -- 172.21.15.62:0/2417918449 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f03a4010460 con 0x7f03b810c8f0 2024-04-20T22:01:45.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:45.357+0000 7f03afe00700 1 -- 172.21.15.62:0/2417918449 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f03a400f5f0 con 0x7f03b810c8f0 2024-04-20T22:01:45.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:45.357+0000 7f03be800700 1 -- 172.21.15.62:0/2417918449 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03b81a1190 con 0x7f03b810c8f0 2024-04-20T22:01:45.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:45.357+0000 7f03be800700 1 -- 172.21.15.62:0/2417918449 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03b81a16e0 con 0x7f03b810c8f0 2024-04-20T22:01:45.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:45.358+0000 7f03be800700 1 -- 172.21.15.62:0/2417918449 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03b8109b20 con 0x7f03b810c8f0 2024-04-20T22:01:45.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:45.360+0000 7f03afe00700 1 -- 172.21.15.62:0/2417918449 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f03a40105d0 con 0x7f03b810c8f0 2024-04-20T22:01:45.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:45.361+0000 7f03afe00700 1 --2- 172.21.15.62:0/2417918449 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f039c06ecc0 0x7f039c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:01:45.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:45.361+0000 7f03afe00700 1 -- 172.21.15.62:0/2417918449 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f03a408ee40 con 0x7f03b810c8f0 2024-04-20T22:01:45.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:45.361+0000 7f03b7400700 1 --2- 172.21.15.62:0/2417918449 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f039c06ecc0 0x7f039c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:01:45.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:45.362+0000 7f03b7400700 1 --2- 172.21.15.62:0/2417918449 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f039c06ecc0 0x7f039c071170 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f03a800b5c0 tx=0x7f03a8005960 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:01:45.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:45.366+0000 7f03afe00700 1 -- 172.21.15.62:0/2417918449 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f03a4057bd0 con 0x7f03b810c8f0 2024-04-20T22:01:45.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:45.611+0000 7f03be800700 1 -- 172.21.15.62:0/2417918449 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f03b81a19c0 con 0x7f039c06ecc0 2024-04-20T22:01:45.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:45.624+0000 7f03afe00700 1 -- 172.21.15.62:0/2417918449 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f03b81a19c0 con 0x7f039c06ecc0 2024-04-20T22:01:45.625 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T22:01:45.625 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (27m) 3m ago 41m 15.3M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T22:01:45.625 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (41m) 3m ago 41m 10.4M - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T22:01:45.625 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (40m) 3m ago 40m 10.1M - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T22:01:45.625 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (23m) 3m ago 41m 6660k - 19.0.0-3133-ga8c84f59 76bc6b91b726 2825d5ba11c6 2024-04-20T22:01:45.626 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (23m) 3m ago 40m 6665k - 19.0.0-3133-ga8c84f59 76bc6b91b726 20f746fa1bba 2024-04-20T22:01:45.626 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (26m) 3m ago 41m 73.5M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T22:01:45.626 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (19m) 3m ago 34m 18.5M - 19.0.0-3133-ga8c84f59 76bc6b91b726 de80bc90a683 2024-04-20T22:01:45.626 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (19m) 3m ago 34m 18.7M - 19.0.0-3133-ga8c84f59 76bc6b91b726 cd793b7ee98e 2024-04-20T22:01:45.626 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (3m) 3m ago 34m 11.0M - 19.0.0-3133-ga8c84f59 76bc6b91b726 0e7b0bc4d1ab 2024-04-20T22:01:45.626 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (34m) 3m ago 34m 18.6M - 18.2.0 dc2bc1663786 f2eec2ee8951 2024-04-20T22:01:45.627 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (30m) 3m ago 43m 513M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T22:01:45.627 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (29m) 3m ago 40m 459M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T22:01:45.627 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (25m) 3m ago 43m 71.7M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 019eb1479485 2024-04-20T22:01:45.627 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (24m) 3m ago 40m 51.0M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 55aac560aab6 2024-04-20T22:01:45.627 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (28m) 3m ago 41m 8472k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T22:01:45.627 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (28m) 3m ago 40m 8048k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T22:01:45.627 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (23m) 3m ago 39m 91.2M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 ee2bb901f26e 2024-04-20T22:01:45.628 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (22m) 3m ago 38m 94.6M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 9075152d4162 2024-04-20T22:01:45.628 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (21m) 3m ago 37m 78.1M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 2d4f3e38dc0e 2024-04-20T22:01:45.628 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (21m) 3m ago 37m 92.6M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 8f97b17ae0eb 2024-04-20T22:01:45.628 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (20m) 3m ago 36m 95.6M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 653d8079967d 2024-04-20T22:01:45.628 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (20m) 3m ago 36m 89.2M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 596dded14c12 2024-04-20T22:01:45.628 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (28m) 3m ago 40m 82.1M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T22:01:45.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:45.627+0000 7f03be800700 1 -- 172.21.15.62:0/2417918449 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f039c06ecc0 msgr2=0x7f039c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:01:45.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:45.627+0000 7f03be800700 1 --2- 172.21.15.62:0/2417918449 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f039c06ecc0 0x7f039c071170 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f03a800b5c0 tx=0x7f03a8005960 comp rx=0 tx=0).stop 2024-04-20T22:01:45.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:45.627+0000 7f03be800700 1 -- 172.21.15.62:0/2417918449 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f03b810c8f0 msgr2=0x7f03b819ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:01:45.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:45.627+0000 7f03be800700 1 --2- 172.21.15.62:0/2417918449 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f03b810c8f0 0x7f03b819ca40 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f03a400d8d0 tx=0x7f03a400dc90 comp rx=0 tx=0).stop 2024-04-20T22:01:45.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:45.627+0000 7f03be800700 1 -- 172.21.15.62:0/2417918449 shutdown_connections 2024-04-20T22:01:45.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:45.627+0000 7f03be800700 1 --2- 172.21.15.62:0/2417918449 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f039c06ecc0 0x7f039c071170 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:01:45.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:45.627+0000 7f03be800700 1 --2- 172.21.15.62:0/2417918449 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03b8106b80 0x7f03b819c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:01:45.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:45.627+0000 7f03be800700 1 --2- 172.21.15.62:0/2417918449 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f03b810c8f0 0x7f03b819ca40 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:01:45.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:45.627+0000 7f03be800700 1 -- 172.21.15.62:0/2417918449 >> 172.21.15.62:0/2417918449 conn(0x7f03b8076060 msgr2=0x7f03b8077aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:01:45.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:45.627+0000 7f03be800700 1 -- 172.21.15.62:0/2417918449 shutdown_connections 2024-04-20T22:01:45.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:45.628+0000 7f03be800700 1 -- 172.21.15.62:0/2417918449 wait complete. 2024-04-20T22:01:45.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:45.789+0000 7ff1c5800700 1 -- 172.21.15.62:0/4214633931 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1c0108470 msgr2=0x7ff1c0108840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:01:45.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:45.789+0000 7ff1c5800700 1 --2- 172.21.15.62:0/4214633931 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1c0108470 0x7ff1c0108840 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7ff1b0009910 tx=0x7ff1b0009c20 comp rx=0 tx=0).stop 2024-04-20T22:01:45.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:45.790+0000 7ff1c5800700 1 -- 172.21.15.62:0/4214633931 shutdown_connections 2024-04-20T22:01:45.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:45.790+0000 7ff1c5800700 1 --2- 172.21.15.62:0/4214633931 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1c0102470 0x7ff1c01028e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:01:45.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:45.790+0000 7ff1c5800700 1 --2- 172.21.15.62:0/4214633931 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1c0108470 0x7ff1c0108840 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:01:45.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:45.790+0000 7ff1c5800700 1 -- 172.21.15.62:0/4214633931 >> 172.21.15.62:0/4214633931 conn(0x7ff1c00fdf90 msgr2=0x7ff1c01003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:01:45.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:45.790+0000 7ff1c5800700 1 -- 172.21.15.62:0/4214633931 shutdown_connections 2024-04-20T22:01:45.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:45.790+0000 7ff1c5800700 1 -- 172.21.15.62:0/4214633931 wait complete. 2024-04-20T22:01:45.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:45.791+0000 7ff1c5800700 1 Processor -- start 2024-04-20T22:01:45.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:45.791+0000 7ff1c5800700 1 -- start start 2024-04-20T22:01:45.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:45.791+0000 7ff1c5800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1c0102470 0x7ff1c0198210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:01:45.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:45.791+0000 7ff1c5800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1c0108470 0x7ff1c0198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:01:45.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:45.791+0000 7ff1c5800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1c0198e30 con 0x7ff1c0102470 2024-04-20T22:01:45.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:45.791+0000 7ff1c5800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1c019cbc0 con 0x7ff1c0108470 2024-04-20T22:01:45.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:45.792+0000 7ff1bea00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1c0102470 0x7ff1c0198210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:01:45.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:45.792+0000 7ff1bea00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1c0102470 0x7ff1c0198210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51974/0 (socket says 172.21.15.62:51974) 2024-04-20T22:01:45.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:45.792+0000 7ff1bea00700 1 -- 172.21.15.62:0/692631899 learned_addr learned my addr 172.21.15.62:0/692631899 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:01:45.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:45.792+0000 7ff1be000700 1 --2- 172.21.15.62:0/692631899 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1c0108470 0x7ff1c0198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:01:45.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:45.792+0000 7ff1bea00700 1 -- 172.21.15.62:0/692631899 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1c0108470 msgr2=0x7ff1c0198750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:01:45.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:45.792+0000 7ff1bea00700 1 --2- 172.21.15.62:0/692631899 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1c0108470 0x7ff1c0198750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:01:45.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:45.792+0000 7ff1bea00700 1 -- 172.21.15.62:0/692631899 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1b00095f0 con 0x7ff1c0102470 2024-04-20T22:01:45.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:45.793+0000 7ff1bea00700 1 --2- 172.21.15.62:0/692631899 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1c0102470 0x7ff1c0198210 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7ff1b000b5c0 tx=0x7ff1b000be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:01:45.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:45.793+0000 7ff1b7400700 1 -- 172.21.15.62:0/692631899 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff1b001d070 con 0x7ff1c0102470 2024-04-20T22:01:45.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:45.793+0000 7ff1b7400700 1 -- 172.21.15.62:0/692631899 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff1b000f460 con 0x7ff1c0102470 2024-04-20T22:01:45.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:45.793+0000 7ff1b7400700 1 -- 172.21.15.62:0/692631899 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff1b0017610 con 0x7ff1c0102470 2024-04-20T22:01:45.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:45.793+0000 7ff1c5800700 1 -- 172.21.15.62:0/692631899 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1c019ce40 con 0x7ff1c0102470 2024-04-20T22:01:45.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:45.793+0000 7ff1c5800700 1 -- 172.21.15.62:0/692631899 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1c019d330 con 0x7ff1c0102470 2024-04-20T22:01:45.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:45.794+0000 7ff1c5800700 1 -- 172.21.15.62:0/692631899 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1c010a9e0 con 0x7ff1c0102470 2024-04-20T22:01:45.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:45.795+0000 7ff1b7400700 1 -- 172.21.15.62:0/692631899 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff1b000f5d0 con 0x7ff1c0102470 2024-04-20T22:01:45.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:45.796+0000 7ff1b7400700 1 --2- 172.21.15.62:0/692631899 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff1a406ecc0 0x7ff1a4071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:01:45.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:45.796+0000 7ff1b7400700 1 -- 172.21.15.62:0/692631899 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7ff1b0090440 con 0x7ff1c0102470 2024-04-20T22:01:45.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:45.797+0000 7ff1be000700 1 --2- 172.21.15.62:0/692631899 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff1a406ecc0 0x7ff1a4071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:01:45.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:45.797+0000 7ff1be000700 1 --2- 172.21.15.62:0/692631899 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff1a406ecc0 0x7ff1a4071170 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7ff1ac006fd0 tx=0x7ff1ac008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:01:45.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:45.801+0000 7ff1b7400700 1 -- 172.21.15.62:0/692631899 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7ff1b00591d0 con 0x7ff1c0102470 2024-04-20T22:01:46.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:46.129+0000 7ff1c5800700 1 -- 172.21.15.62:0/692631899 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7ff1c004f2a0 con 0x7ff1c0102470 2024-04-20T22:01:46.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:46.130+0000 7ff1b7400700 1 -- 172.21.15.62:0/692631899 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7ff1b005c890 con 0x7ff1c0102470 2024-04-20T22:01:46.130 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:01:46.130 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T22:01:46.130 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:01:46.130 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:01:46.131 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T22:01:46.131 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:01:46.131 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:01:46.131 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T22:01:46.131 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 6 2024-04-20T22:01:46.131 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:01:46.131 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T22:01:46.131 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 1, 2024-04-20T22:01:46.131 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 3 2024-04-20T22:01:46.131 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:01:46.131 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T22:01:46.131 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 1, 2024-04-20T22:01:46.131 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 13 2024-04-20T22:01:46.131 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T22:01:46.132 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:01:46.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:46.133+0000 7ff1c5800700 1 -- 172.21.15.62:0/692631899 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff1a406ecc0 msgr2=0x7ff1a4071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:01:46.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:46.133+0000 7ff1c5800700 1 --2- 172.21.15.62:0/692631899 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff1a406ecc0 0x7ff1a4071170 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7ff1ac006fd0 tx=0x7ff1ac008040 comp rx=0 tx=0).stop 2024-04-20T22:01:46.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:46.133+0000 7ff1c5800700 1 -- 172.21.15.62:0/692631899 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1c0102470 msgr2=0x7ff1c0198210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:01:46.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:46.133+0000 7ff1c5800700 1 --2- 172.21.15.62:0/692631899 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1c0102470 0x7ff1c0198210 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7ff1b000b5c0 tx=0x7ff1b000be00 comp rx=0 tx=0).stop 2024-04-20T22:01:46.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:46.133+0000 7ff1c5800700 1 -- 172.21.15.62:0/692631899 shutdown_connections 2024-04-20T22:01:46.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:46.133+0000 7ff1c5800700 1 --2- 172.21.15.62:0/692631899 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff1a406ecc0 0x7ff1a4071170 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:01:46.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:46.133+0000 7ff1c5800700 1 --2- 172.21.15.62:0/692631899 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1c0102470 0x7ff1c0198210 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:01:46.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:46.133+0000 7ff1c5800700 1 --2- 172.21.15.62:0/692631899 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1c0108470 0x7ff1c0198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:01:46.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:46.133+0000 7ff1c5800700 1 -- 172.21.15.62:0/692631899 >> 172.21.15.62:0/692631899 conn(0x7ff1c00fdf90 msgr2=0x7ff1c00febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:01:46.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:46.133+0000 7ff1c5800700 1 -- 172.21.15.62:0/692631899 shutdown_connections 2024-04-20T22:01:46.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:46.133+0000 7ff1c5800700 1 -- 172.21.15.62:0/692631899 wait complete. 2024-04-20T22:01:46.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:01:45 smithi189 ceph-mon[57022]: from='client.35432 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:01:46.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:01:45 smithi189 ceph-mon[57022]: from='client.35436 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:01:46.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:01:45 smithi062 ceph-mon[76322]: from='client.35432 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:01:46.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:01:45 smithi062 ceph-mon[76322]: from='client.35436 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:01:46.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:46.292+0000 7f1e8aa00700 1 -- 172.21.15.62:0/2168936735 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1e840687a0 msgr2=0x7f1e84068b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:01:46.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:46.292+0000 7f1e8aa00700 1 --2- 172.21.15.62:0/2168936735 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1e840687a0 0x7f1e84068b70 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f1e7400b0a0 tx=0x7f1e7400b3b0 comp rx=0 tx=0).stop 2024-04-20T22:01:46.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:46.293+0000 7f1e8aa00700 1 -- 172.21.15.62:0/2168936735 shutdown_connections 2024-04-20T22:01:46.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:46.293+0000 7f1e8aa00700 1 --2- 172.21.15.62:0/2168936735 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1e84069140 0x7f1e84104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:01:46.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:46.293+0000 7f1e8aa00700 1 --2- 172.21.15.62:0/2168936735 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1e840687a0 0x7f1e84068b70 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:01:46.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:46.293+0000 7f1e8aa00700 1 -- 172.21.15.62:0/2168936735 >> 172.21.15.62:0/2168936735 conn(0x7f1e84075130 msgr2=0x7f1e84075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:01:46.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:46.293+0000 7f1e8aa00700 1 -- 172.21.15.62:0/2168936735 shutdown_connections 2024-04-20T22:01:46.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:46.293+0000 7f1e8aa00700 1 -- 172.21.15.62:0/2168936735 wait complete. 2024-04-20T22:01:46.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:46.294+0000 7f1e8aa00700 1 Processor -- start 2024-04-20T22:01:46.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:46.294+0000 7f1e8aa00700 1 -- start start 2024-04-20T22:01:46.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:46.294+0000 7f1e8aa00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1e840687a0 0x7f1e84195fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:01:46.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:46.294+0000 7f1e8aa00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1e84069140 0x7f1e84196520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:01:46.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:46.294+0000 7f1e8aa00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e84196c00 con 0x7f1e840687a0 2024-04-20T22:01:46.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:46.294+0000 7f1e8aa00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e8419a990 con 0x7f1e84069140 2024-04-20T22:01:46.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:46.295+0000 7f1e83e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1e840687a0 0x7f1e84195fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:01:46.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:46.295+0000 7f1e83e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1e840687a0 0x7f1e84195fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51986/0 (socket says 172.21.15.62:51986) 2024-04-20T22:01:46.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:46.295+0000 7f1e83e00700 1 -- 172.21.15.62:0/2440053751 learned_addr learned my addr 172.21.15.62:0/2440053751 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:01:46.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:46.295+0000 7f1e83e00700 1 -- 172.21.15.62:0/2440053751 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1e84069140 msgr2=0x7f1e84196520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:01:46.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:46.295+0000 7f1e83400700 1 --2- 172.21.15.62:0/2440053751 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1e84069140 0x7f1e84196520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:01:46.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:46.295+0000 7f1e83e00700 1 --2- 172.21.15.62:0/2440053751 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1e84069140 0x7f1e84196520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:01:46.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:46.295+0000 7f1e83e00700 1 -- 172.21.15.62:0/2440053751 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e74009d00 con 0x7f1e840687a0 2024-04-20T22:01:46.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:46.295+0000 7f1e83400700 1 --2- 172.21.15.62:0/2440053751 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1e84069140 0x7f1e84196520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T22:01:46.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:46.295+0000 7f1e83e00700 1 --2- 172.21.15.62:0/2440053751 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1e840687a0 0x7f1e84195fe0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f1e7400b070 tx=0x7f1e7400cdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:01:46.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:46.296+0000 7f1e80c00700 1 -- 172.21.15.62:0/2440053751 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1e7400e040 con 0x7f1e840687a0 2024-04-20T22:01:46.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:46.296+0000 7f1e80c00700 1 -- 172.21.15.62:0/2440053751 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1e74003e60 con 0x7f1e840687a0 2024-04-20T22:01:46.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:46.296+0000 7f1e8aa00700 1 -- 172.21.15.62:0/2440053751 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e8419ac10 con 0x7f1e840687a0 2024-04-20T22:01:46.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:46.296+0000 7f1e80c00700 1 -- 172.21.15.62:0/2440053751 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1e7401d910 con 0x7f1e840687a0 2024-04-20T22:01:46.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:46.296+0000 7f1e8aa00700 1 -- 172.21.15.62:0/2440053751 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e8419b100 con 0x7f1e840687a0 2024-04-20T22:01:46.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:46.297+0000 7f1e8aa00700 1 -- 172.21.15.62:0/2440053751 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e84108790 con 0x7f1e840687a0 2024-04-20T22:01:46.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:46.298+0000 7f1e80c00700 1 -- 172.21.15.62:0/2440053751 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f1e74004820 con 0x7f1e840687a0 2024-04-20T22:01:46.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:46.299+0000 7f1e80c00700 1 --2- 172.21.15.62:0/2440053751 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f1e6806ecc0 0x7f1e68071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:01:46.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:46.299+0000 7f1e80c00700 1 -- 172.21.15.62:0/2440053751 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f1e740907a0 con 0x7f1e840687a0 2024-04-20T22:01:46.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:46.300+0000 7f1e83400700 1 --2- 172.21.15.62:0/2440053751 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f1e6806ecc0 0x7f1e68071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:01:46.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:46.300+0000 7f1e83400700 1 --2- 172.21.15.62:0/2440053751 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f1e6806ecc0 0x7f1e68071170 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f1e84197600 tx=0x7f1e70008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:01:46.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:46.304+0000 7f1e80c00700 1 -- 172.21.15.62:0/2440053751 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f1e74059530 con 0x7f1e840687a0 2024-04-20T22:01:46.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:46.581+0000 7f1e8aa00700 1 -- 172.21.15.62:0/2440053751 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f1e84197440 con 0x7f1e840687a0 2024-04-20T22:01:46.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:46.582+0000 7f1e80c00700 1 -- 172.21.15.62:0/2440053751 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 27 v27) v1 ==== 76+0+1888 (secure 0 0 0) 0x7f1e7405cbf0 con 0x7f1e840687a0 2024-04-20T22:01:46.583 INFO:teuthology.orchestra.run.smithi062.stdout:e27 2024-04-20T22:01:46.583 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T22:01:46.583 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:01:46.583 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T22:01:46.583 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:01:46.583 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T22:01:46.583 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T22:01:46.583 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 20 2024-04-20T22:01:46.583 INFO:teuthology.orchestra.run.smithi062.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-20T22:01:46.584 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T22:01:46.584 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T21:42:01.559061+0000 2024-04-20T22:01:46.584 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T22:01:46.584 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T22:01:46.584 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T22:01:46.584 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T22:01:46.584 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T22:01:46.584 INFO:teuthology.orchestra.run.smithi062.stdout:max_xattr_size 65536 2024-04-20T22:01:46.584 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T22:01:46.585 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T22:01:46.585 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 81 2024-04-20T22:01:46.585 INFO:teuthology.orchestra.run.smithi062.stdout:snap_rstat 1 2024-04-20T22:01:46.585 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:01:46.585 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 2 2024-04-20T22:01:46.585 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1 2024-04-20T22:01:46.585 INFO:teuthology.orchestra.run.smithi062.stdout:up {} 2024-04-20T22:01:46.586 INFO:teuthology.orchestra.run.smithi062.stdout:failed 0,1 2024-04-20T22:01:46.586 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T22:01:46.586 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T22:01:46.586 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T22:01:46.586 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T22:01:46.586 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data enabled 2024-04-20T22:01:46.586 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T22:01:46.587 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T22:01:46.587 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 1 2024-04-20T22:01:46.587 INFO:teuthology.orchestra.run.smithi062.stdout:qdb_cluster leader: 0 members: 2024-04-20T22:01:46.587 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:01:46.587 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:01:46.587 INFO:teuthology.orchestra.run.smithi062.stdout:Standby daemons: 2024-04-20T22:01:46.587 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:01:46.588 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ryqblb{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6826/4113916205,v1:172.21.15.62:6827/4113916205] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:01:46.588 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ozizwi{-1:34476} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6828/3801975165,v1:172.21.15.62:6829/3801975165] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:01:46.588 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.xbgspw{-1:44285} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6826/3996040526,v1:172.21.15.189:6827/3996040526] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T22:01:46.589 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.dvimer{-1:44727} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6824/3576699232,v1:172.21.15.189:6825/3576699232] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:01:46.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:46.586+0000 7f1e8aa00700 1 -- 172.21.15.62:0/2440053751 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f1e6806ecc0 msgr2=0x7f1e68071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:01:46.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:46.586+0000 7f1e8aa00700 1 --2- 172.21.15.62:0/2440053751 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f1e6806ecc0 0x7f1e68071170 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f1e84197600 tx=0x7f1e70008040 comp rx=0 tx=0).stop 2024-04-20T22:01:46.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:46.586+0000 7f1e8aa00700 1 -- 172.21.15.62:0/2440053751 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1e840687a0 msgr2=0x7f1e84195fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:01:46.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:46.586+0000 7f1e8aa00700 1 --2- 172.21.15.62:0/2440053751 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1e840687a0 0x7f1e84195fe0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f1e7400b070 tx=0x7f1e7400cdd0 comp rx=0 tx=0).stop 2024-04-20T22:01:46.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:46.587+0000 7f1e8aa00700 1 -- 172.21.15.62:0/2440053751 shutdown_connections 2024-04-20T22:01:46.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:46.587+0000 7f1e8aa00700 1 --2- 172.21.15.62:0/2440053751 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f1e6806ecc0 0x7f1e68071170 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:01:46.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:46.587+0000 7f1e8aa00700 1 --2- 172.21.15.62:0/2440053751 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1e840687a0 0x7f1e84195fe0 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:01:46.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:46.587+0000 7f1e8aa00700 1 --2- 172.21.15.62:0/2440053751 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1e84069140 0x7f1e84196520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:01:46.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:46.587+0000 7f1e8aa00700 1 -- 172.21.15.62:0/2440053751 >> 172.21.15.62:0/2440053751 conn(0x7f1e84075130 msgr2=0x7f1e840fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:01:46.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:46.587+0000 7f1e8aa00700 1 -- 172.21.15.62:0/2440053751 shutdown_connections 2024-04-20T22:01:46.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:46.587+0000 7f1e8aa00700 1 -- 172.21.15.62:0/2440053751 wait complete. 2024-04-20T22:01:46.591 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 27 2024-04-20T22:01:46.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:46.750+0000 7f6f22800700 1 -- 172.21.15.62:0/2230066466 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6f1c106b80 msgr2=0x7f6f1c106ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:01:46.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:46.750+0000 7f6f22800700 1 --2- 172.21.15.62:0/2230066466 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6f1c106b80 0x7f6f1c106ff0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f6f08009910 tx=0x7f6f08009c20 comp rx=0 tx=0).stop 2024-04-20T22:01:46.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:46.750+0000 7f6f22800700 1 -- 172.21.15.62:0/2230066466 shutdown_connections 2024-04-20T22:01:46.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:46.750+0000 7f6f22800700 1 --2- 172.21.15.62:0/2230066466 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6f1c106b80 0x7f6f1c106ff0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:01:46.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:46.750+0000 7f6f22800700 1 --2- 172.21.15.62:0/2230066466 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f1c10c8f0 0x7f6f1c10ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:01:46.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:46.750+0000 7f6f22800700 1 -- 172.21.15.62:0/2230066466 >> 172.21.15.62:0/2230066466 conn(0x7f6f1c076020 msgr2=0x7f6f1c078430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:01:46.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:46.751+0000 7f6f22800700 1 -- 172.21.15.62:0/2230066466 shutdown_connections 2024-04-20T22:01:46.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:46.751+0000 7f6f22800700 1 -- 172.21.15.62:0/2230066466 wait complete. 2024-04-20T22:01:46.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:46.752+0000 7f6f22800700 1 Processor -- start 2024-04-20T22:01:46.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:46.752+0000 7f6f22800700 1 -- start start 2024-04-20T22:01:46.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:46.752+0000 7f6f22800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f1c106b80 0x7f6f1c19c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:01:46.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:46.752+0000 7f6f22800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6f1c10c8f0 0x7f6f1c19c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:01:46.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:46.752+0000 7f6f22800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f1c19d070 con 0x7f6f1c10c8f0 2024-04-20T22:01:46.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:46.752+0000 7f6f22800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f1c1a0e00 con 0x7f6f1c106b80 2024-04-20T22:01:46.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:46.752+0000 7f6f1aa00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6f1c10c8f0 0x7f6f1c19c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:01:46.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:46.753+0000 7f6f1aa00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6f1c10c8f0 0x7f6f1c19c990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52002/0 (socket says 172.21.15.62:52002) 2024-04-20T22:01:46.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:46.753+0000 7f6f1aa00700 1 -- 172.21.15.62:0/481442817 learned_addr learned my addr 172.21.15.62:0/481442817 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:01:46.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:46.753+0000 7f6f1b400700 1 --2- 172.21.15.62:0/481442817 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f1c106b80 0x7f6f1c19c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:01:46.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:46.753+0000 7f6f1aa00700 1 -- 172.21.15.62:0/481442817 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f1c106b80 msgr2=0x7f6f1c19c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:01:46.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:46.753+0000 7f6f1aa00700 1 --2- 172.21.15.62:0/481442817 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f1c106b80 0x7f6f1c19c450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:01:46.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:46.753+0000 7f6f1aa00700 1 -- 172.21.15.62:0/481442817 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f080095f0 con 0x7f6f1c10c8f0 2024-04-20T22:01:46.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:46.753+0000 7f6f1aa00700 1 --2- 172.21.15.62:0/481442817 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6f1c10c8f0 0x7f6f1c19c990 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f6f0800f690 tx=0x7f6f0800f770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:01:46.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:46.754+0000 7f6f13e00700 1 -- 172.21.15.62:0/481442817 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6f0801d070 con 0x7f6f1c10c8f0 2024-04-20T22:01:46.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:46.754+0000 7f6f13e00700 1 -- 172.21.15.62:0/481442817 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6f080050d0 con 0x7f6f1c10c8f0 2024-04-20T22:01:46.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:46.754+0000 7f6f13e00700 1 -- 172.21.15.62:0/481442817 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6f08022470 con 0x7f6f1c10c8f0 2024-04-20T22:01:46.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:46.754+0000 7f6f22800700 1 -- 172.21.15.62:0/481442817 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f1c1a1080 con 0x7f6f1c10c8f0 2024-04-20T22:01:46.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:46.754+0000 7f6f22800700 1 -- 172.21.15.62:0/481442817 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f1c1a1570 con 0x7f6f1c10c8f0 2024-04-20T22:01:46.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:46.755+0000 7f6f22800700 1 -- 172.21.15.62:0/481442817 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f1c04f2a0 con 0x7f6f1c10c8f0 2024-04-20T22:01:46.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:46.756+0000 7f6f13e00700 1 -- 172.21.15.62:0/481442817 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f6f08003680 con 0x7f6f1c10c8f0 2024-04-20T22:01:46.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:46.757+0000 7f6f13e00700 1 --2- 172.21.15.62:0/481442817 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6f0006ecc0 0x7f6f00071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:01:46.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:46.757+0000 7f6f13e00700 1 -- 172.21.15.62:0/481442817 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f6f080902d0 con 0x7f6f1c10c8f0 2024-04-20T22:01:46.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:46.757+0000 7f6f1b400700 1 --2- 172.21.15.62:0/481442817 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6f0006ecc0 0x7f6f00071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:01:46.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:46.758+0000 7f6f1b400700 1 --2- 172.21.15.62:0/481442817 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6f0006ecc0 0x7f6f00071170 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f6f0c009bf0 tx=0x7f6f0c009260 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:01:46.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:46.762+0000 7f6f13e00700 1 -- 172.21.15.62:0/481442817 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f6f08059060 con 0x7f6f1c10c8f0 2024-04-20T22:01:47.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:47.008+0000 7f6f22800700 1 -- 172.21.15.62:0/481442817 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6f1c1a1980 con 0x7f6f0006ecc0 2024-04-20T22:01:47.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:47.010+0000 7f6f13e00700 1 -- 172.21.15.62:0/481442817 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f6f1c1a1980 con 0x7f6f0006ecc0 2024-04-20T22:01:47.010 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:01:47.010 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T22:01:47.010 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T22:01:47.010 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-20T22:01:47.010 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T22:01:47.010 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr", 2024-04-20T22:01:47.010 INFO:teuthology.orchestra.run.smithi062.stdout: "crash", 2024-04-20T22:01:47.010 INFO:teuthology.orchestra.run.smithi062.stdout: "mon", 2024-04-20T22:01:47.010 INFO:teuthology.orchestra.run.smithi062.stdout: "osd" 2024-04-20T22:01:47.010 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T22:01:47.011 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "15/23 daemons upgraded", 2024-04-20T22:01:47.011 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading mds daemons", 2024-04-20T22:01:47.011 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T22:01:47.011 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:01:47.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:47.013+0000 7f6f22800700 1 -- 172.21.15.62:0/481442817 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6f0006ecc0 msgr2=0x7f6f00071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:01:47.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:47.013+0000 7f6f22800700 1 --2- 172.21.15.62:0/481442817 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6f0006ecc0 0x7f6f00071170 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f6f0c009bf0 tx=0x7f6f0c009260 comp rx=0 tx=0).stop 2024-04-20T22:01:47.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:47.013+0000 7f6f22800700 1 -- 172.21.15.62:0/481442817 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6f1c10c8f0 msgr2=0x7f6f1c19c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:01:47.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:47.013+0000 7f6f22800700 1 --2- 172.21.15.62:0/481442817 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6f1c10c8f0 0x7f6f1c19c990 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f6f0800f690 tx=0x7f6f0800f770 comp rx=0 tx=0).stop 2024-04-20T22:01:47.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:47.013+0000 7f6f22800700 1 -- 172.21.15.62:0/481442817 shutdown_connections 2024-04-20T22:01:47.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:47.013+0000 7f6f22800700 1 --2- 172.21.15.62:0/481442817 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6f0006ecc0 0x7f6f00071170 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:01:47.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:47.013+0000 7f6f22800700 1 --2- 172.21.15.62:0/481442817 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f1c106b80 0x7f6f1c19c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:01:47.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:47.013+0000 7f6f22800700 1 --2- 172.21.15.62:0/481442817 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6f1c10c8f0 0x7f6f1c19c990 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:01:47.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:47.013+0000 7f6f22800700 1 -- 172.21.15.62:0/481442817 >> 172.21.15.62:0/481442817 conn(0x7f6f1c076020 msgr2=0x7f6f1c077a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:01:47.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:47.014+0000 7f6f22800700 1 -- 172.21.15.62:0/481442817 shutdown_connections 2024-04-20T22:01:47.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:47.014+0000 7f6f22800700 1 -- 172.21.15.62:0/481442817 wait complete. 2024-04-20T22:01:47.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:47.168+0000 7fde0ae00700 1 -- 172.21.15.62:0/1938759806 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fde040687a0 msgr2=0x7fde04068b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:01:47.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:47.168+0000 7fde0ae00700 1 --2- 172.21.15.62:0/1938759806 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fde040687a0 0x7fde04068b70 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fddf4009910 tx=0x7fddf4009c20 comp rx=0 tx=0).stop 2024-04-20T22:01:47.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:47.169+0000 7fde0ae00700 1 -- 172.21.15.62:0/1938759806 shutdown_connections 2024-04-20T22:01:47.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:47.169+0000 7fde0ae00700 1 --2- 172.21.15.62:0/1938759806 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde04069140 0x7fde04104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:01:47.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:47.169+0000 7fde0ae00700 1 --2- 172.21.15.62:0/1938759806 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fde040687a0 0x7fde04068b70 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:01:47.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:47.169+0000 7fde0ae00700 1 -- 172.21.15.62:0/1938759806 >> 172.21.15.62:0/1938759806 conn(0x7fde04075130 msgr2=0x7fde04075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:01:47.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:47.169+0000 7fde0ae00700 1 -- 172.21.15.62:0/1938759806 shutdown_connections 2024-04-20T22:01:47.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:47.169+0000 7fde0ae00700 1 -- 172.21.15.62:0/1938759806 wait complete. 2024-04-20T22:01:47.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:47.170+0000 7fde0ae00700 1 Processor -- start 2024-04-20T22:01:47.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:47.170+0000 7fde0ae00700 1 -- start start 2024-04-20T22:01:47.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:47.170+0000 7fde0ae00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fde040687a0 0x7fde040ffca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:01:47.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:47.170+0000 7fde0ae00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde04069140 0x7fde041001e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:01:47.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:47.170+0000 7fde0ae00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde04100720 con 0x7fde040687a0 2024-04-20T22:01:47.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:47.170+0000 7fde0ae00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde04100860 con 0x7fde04069140 2024-04-20T22:01:47.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:47.171+0000 7fde03e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fde040687a0 0x7fde040ffca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:01:47.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:47.171+0000 7fde03e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fde040687a0 0x7fde040ffca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52028/0 (socket says 172.21.15.62:52028) 2024-04-20T22:01:47.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:47.171+0000 7fde03e00700 1 -- 172.21.15.62:0/1250206704 learned_addr learned my addr 172.21.15.62:0/1250206704 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:01:47.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:47.171+0000 7fde03400700 1 --2- 172.21.15.62:0/1250206704 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde04069140 0x7fde041001e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:01:47.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:47.171+0000 7fde03e00700 1 -- 172.21.15.62:0/1250206704 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde04069140 msgr2=0x7fde041001e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:01:47.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:47.171+0000 7fde03e00700 1 --2- 172.21.15.62:0/1250206704 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde04069140 0x7fde041001e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:01:47.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:47.171+0000 7fde03e00700 1 -- 172.21.15.62:0/1250206704 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddf40095f0 con 0x7fde040687a0 2024-04-20T22:01:47.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:47.172+0000 7fde03e00700 1 --2- 172.21.15.62:0/1250206704 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fde040687a0 0x7fde040ffca0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fddf4006010 tx=0x7fddf400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:01:47.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:47.172+0000 7fde00c00700 1 -- 172.21.15.62:0/1250206704 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fddf401d070 con 0x7fde040687a0 2024-04-20T22:01:47.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:47.172+0000 7fde00c00700 1 -- 172.21.15.62:0/1250206704 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fddf400f460 con 0x7fde040687a0 2024-04-20T22:01:47.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:47.172+0000 7fde00c00700 1 -- 172.21.15.62:0/1250206704 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fddf4017630 con 0x7fde040687a0 2024-04-20T22:01:47.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:47.172+0000 7fde0ae00700 1 -- 172.21.15.62:0/1250206704 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde04100ae0 con 0x7fde040687a0 2024-04-20T22:01:47.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:47.172+0000 7fde0ae00700 1 -- 172.21.15.62:0/1250206704 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde041a23e0 con 0x7fde040687a0 2024-04-20T22:01:47.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:47.173+0000 7fde0ae00700 1 -- 172.21.15.62:0/1250206704 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde04108790 con 0x7fde040687a0 2024-04-20T22:01:47.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:47.174+0000 7fde00c00700 1 -- 172.21.15.62:0/1250206704 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fddf400f5d0 con 0x7fde040687a0 2024-04-20T22:01:47.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:47.175+0000 7fde00c00700 1 --2- 172.21.15.62:0/1250206704 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fdde806ecc0 0x7fdde8071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:01:47.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:47.175+0000 7fde00c00700 1 -- 172.21.15.62:0/1250206704 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fddf4090320 con 0x7fde040687a0 2024-04-20T22:01:47.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:47.176+0000 7fde03400700 1 --2- 172.21.15.62:0/1250206704 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fdde806ecc0 0x7fdde8071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:01:47.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:47.176+0000 7fde03400700 1 --2- 172.21.15.62:0/1250206704 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fdde806ecc0 0x7fdde8071170 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fddf0005950 tx=0x7fddf00058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:01:47.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:47.180+0000 7fde00c00700 1 -- 172.21.15.62:0/1250206704 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fddf40590b0 con 0x7fde040687a0 2024-04-20T22:01:47.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:01:46 smithi189 ceph-mon[57022]: pgmap v795: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:01:47.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:01:46 smithi189 ceph-mon[57022]: from='client.35440 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:01:47.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:01:46 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/692631899' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:01:47.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:01:46 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/2440053751' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:01:47.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:01:46 smithi062 ceph-mon[76322]: pgmap v795: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:01:47.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:01:46 smithi062 ceph-mon[76322]: from='client.35440 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:01:47.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:01:46 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/692631899' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:01:47.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:01:46 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/2440053751' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:01:47.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:47.498+0000 7fde0ae00700 1 -- 172.21.15.62:0/1250206704 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fde0404f2a0 con 0x7fde040687a0 2024-04-20T22:01:47.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:47.499+0000 7fde00c00700 1 -- 172.21.15.62:0/1250206704 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7fddf405c770 con 0x7fde040687a0 2024-04-20T22:01:47.499 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-20T22:01:47.499 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-20T22:01:47.499 INFO:teuthology.orchestra.run.smithi062.stdout: host smithi062 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T22:01:47.499 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-20T22:01:47.499 INFO:teuthology.orchestra.run.smithi062.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi062 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpq7y7oa_r:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:01:47.499 INFO:teuthology.orchestra.run.smithi062.stdout:Traceback (most recent call last): 2024-04-20T22:01:47.499 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-20T22:01:47.499 INFO:teuthology.orchestra.run.smithi062.stdout: return _run_code(code, main_globals, None, 2024-04-20T22:01:47.500 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-20T22:01:47.500 INFO:teuthology.orchestra.run.smithi062.stdout: exec(code, run_globals) 2024-04-20T22:01:47.500 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5618, in 2024-04-20T22:01:47.500 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5606, in main 2024-04-20T22:01:47.500 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 408, in _infer_config 2024-04-20T22:01:47.500 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 323, in _infer_fsid 2024-04-20T22:01:47.500 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 436, in _infer_image 2024-04-20T22:01:47.500 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 310, in _validate_fsid 2024-04-20T22:01:47.500 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 3359, in command_ceph_volume 2024-04-20T22:01:47.500 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-20T22:01:47.500 INFO:teuthology.orchestra.run.smithi062.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi062 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpq7y7oa_r:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:01:47.500 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-20T22:01:47.500 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is degraded 2024-04-20T22:01:47.500 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T22:01:47.501 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T22:01:47.501 INFO:teuthology.orchestra.run.smithi062.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-20T22:01:47.501 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-20T22:01:47.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:47.502+0000 7fde0ae00700 1 -- 172.21.15.62:0/1250206704 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fdde806ecc0 msgr2=0x7fdde8071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:01:47.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:47.502+0000 7fde0ae00700 1 --2- 172.21.15.62:0/1250206704 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fdde806ecc0 0x7fdde8071170 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fddf0005950 tx=0x7fddf00058e0 comp rx=0 tx=0).stop 2024-04-20T22:01:47.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:47.502+0000 7fde0ae00700 1 -- 172.21.15.62:0/1250206704 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fde040687a0 msgr2=0x7fde040ffca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:01:47.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:47.502+0000 7fde0ae00700 1 --2- 172.21.15.62:0/1250206704 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fde040687a0 0x7fde040ffca0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fddf4006010 tx=0x7fddf400be20 comp rx=0 tx=0).stop 2024-04-20T22:01:47.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:47.502+0000 7fde0ae00700 1 -- 172.21.15.62:0/1250206704 shutdown_connections 2024-04-20T22:01:47.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:47.502+0000 7fde0ae00700 1 --2- 172.21.15.62:0/1250206704 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fdde806ecc0 0x7fdde8071170 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:01:47.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:47.502+0000 7fde0ae00700 1 --2- 172.21.15.62:0/1250206704 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fde040687a0 0x7fde040ffca0 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:01:47.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:47.502+0000 7fde0ae00700 1 --2- 172.21.15.62:0/1250206704 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde04069140 0x7fde041001e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:01:47.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:47.502+0000 7fde0ae00700 1 -- 172.21.15.62:0/1250206704 >> 172.21.15.62:0/1250206704 conn(0x7fde04075130 msgr2=0x7fde040fe7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:01:47.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:47.502+0000 7fde0ae00700 1 -- 172.21.15.62:0/1250206704 shutdown_connections 2024-04-20T22:01:47.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:01:47.503+0000 7fde0ae00700 1 -- 172.21.15.62:0/1250206704 wait complete. 2024-04-20T22:01:48.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:01:47 smithi189 ceph-mon[57022]: from='client.35452 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:01:48.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:01:47 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/1250206704' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:01:48.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:01:47 smithi062 ceph-mon[76322]: from='client.35452 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:01:48.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:01:47 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/1250206704' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:01:49.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:01:48 smithi189 ceph-mon[57022]: pgmap v796: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:01:49.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:01:48 smithi062 ceph-mon[76322]: pgmap v796: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:01:50.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:01:49 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:01:50.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:01:49 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:01:51.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:01:50 smithi189 ceph-mon[57022]: pgmap v797: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:01:51.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:01:50 smithi062 ceph-mon[76322]: pgmap v797: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:01:53.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:01:52 smithi189 ceph-mon[57022]: pgmap v798: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:01:53.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:01:52 smithi062 ceph-mon[76322]: pgmap v798: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:01:55.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:01:54 smithi189 ceph-mon[57022]: pgmap v799: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:01:55.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:01:54 smithi062 ceph-mon[76322]: pgmap v799: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:01:57.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:01:56 smithi062 ceph-mon[76322]: pgmap v800: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:01:57.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:01:56 smithi189 ceph-mon[57022]: pgmap v800: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:01:59.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:01:58 smithi062 ceph-mon[76322]: pgmap v801: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:01:59.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:01:58 smithi189 ceph-mon[57022]: pgmap v801: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:02:01.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:02:01 smithi062 ceph-mon[76322]: pgmap v802: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:02:01.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:02:01 smithi189 ceph-mon[57022]: pgmap v802: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:02:03.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:02:03 smithi189 ceph-mon[57022]: pgmap v803: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:02:03.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:02:03 smithi062 ceph-mon[76322]: pgmap v803: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:02:05.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:02:05 smithi062 ceph-mon[76322]: pgmap v804: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:02:05.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:02:05 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:02:05.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:02:05 smithi189 ceph-mon[57022]: pgmap v804: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:02:05.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:02:05 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:02:07.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:02:07 smithi062 ceph-mon[76322]: pgmap v805: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:02:07.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:02:07 smithi189 ceph-mon[57022]: pgmap v805: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:02:09.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:02:09 smithi062 ceph-mon[76322]: pgmap v806: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:02:09.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:02:09 smithi189 ceph-mon[57022]: pgmap v806: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:02:11.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:02:11 smithi062 ceph-mon[76322]: pgmap v807: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:02:11.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:02:11 smithi189 ceph-mon[57022]: pgmap v807: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:02:13.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:02:13 smithi189 ceph-mon[57022]: pgmap v808: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:02:13.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:02:13 smithi062 ceph-mon[76322]: pgmap v808: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:02:15.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:02:15 smithi189 ceph-mon[57022]: pgmap v809: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:02:15.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:02:15 smithi062 ceph-mon[76322]: pgmap v809: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:02:17.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:02:17 smithi189 ceph-mon[57022]: pgmap v810: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:02:17.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:02:17 smithi062 ceph-mon[76322]: pgmap v810: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:02:17.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:17.679+0000 7f4d31e00700 1 -- 172.21.15.62:0/1467375417 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d2c101d30 msgr2=0x7f4d2c10a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:02:17.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:17.679+0000 7f4d31e00700 1 --2- 172.21.15.62:0/1467375417 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d2c101d30 0x7f4d2c10a230 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f4d18009910 tx=0x7f4d18009c20 comp rx=0 tx=0).stop 2024-04-20T22:02:17.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:17.680+0000 7f4d31e00700 1 -- 172.21.15.62:0/1467375417 shutdown_connections 2024-04-20T22:02:17.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:17.680+0000 7f4d31e00700 1 --2- 172.21.15.62:0/1467375417 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d2c101d30 0x7f4d2c10a230 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:02:17.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:17.680+0000 7f4d31e00700 1 --2- 172.21.15.62:0/1467375417 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4d2c101420 0x7f4d2c1017f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:02:17.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:17.680+0000 7f4d31e00700 1 -- 172.21.15.62:0/1467375417 >> 172.21.15.62:0/1467375417 conn(0x7f4d2c0fac80 msgr2=0x7f4d2c0fd090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:02:17.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:17.680+0000 7f4d31e00700 1 -- 172.21.15.62:0/1467375417 shutdown_connections 2024-04-20T22:02:17.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:17.680+0000 7f4d31e00700 1 -- 172.21.15.62:0/1467375417 wait complete. 2024-04-20T22:02:17.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:17.681+0000 7f4d31e00700 1 Processor -- start 2024-04-20T22:02:17.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:17.681+0000 7f4d31e00700 1 -- start start 2024-04-20T22:02:17.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:17.681+0000 7f4d31e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4d2c101420 0x7f4d2c0ff080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:02:17.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:17.681+0000 7f4d31e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d2c101d30 0x7f4d2c0ff5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:02:17.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:17.681+0000 7f4d31e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d2c0ffb00 con 0x7f4d2c101d30 2024-04-20T22:02:17.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:17.681+0000 7f4d31e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d2c0ffc40 con 0x7f4d2c101420 2024-04-20T22:02:17.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:17.682+0000 7f4d2a000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d2c101d30 0x7f4d2c0ff5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:02:17.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:17.682+0000 7f4d2a000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d2c101d30 0x7f4d2c0ff5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:38044/0 (socket says 172.21.15.62:38044) 2024-04-20T22:02:17.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:17.682+0000 7f4d2a000700 1 -- 172.21.15.62:0/3170846096 learned_addr learned my addr 172.21.15.62:0/3170846096 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:02:17.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:17.682+0000 7f4d2a000700 1 -- 172.21.15.62:0/3170846096 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4d2c101420 msgr2=0x7f4d2c0ff080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:02:17.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:17.682+0000 7f4d2a000700 1 --2- 172.21.15.62:0/3170846096 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4d2c101420 0x7f4d2c0ff080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:02:17.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:17.682+0000 7f4d2a000700 1 -- 172.21.15.62:0/3170846096 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d180095f0 con 0x7f4d2c101d30 2024-04-20T22:02:17.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:17.683+0000 7f4d2a000700 1 --2- 172.21.15.62:0/3170846096 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d2c101d30 0x7f4d2c0ff5c0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f4d1800b5c0 tx=0x7f4d1800be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:02:17.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:17.683+0000 7f4d30a00700 1 -- 172.21.15.62:0/3170846096 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4d1801d070 con 0x7f4d2c101d30 2024-04-20T22:02:17.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:17.683+0000 7f4d30a00700 1 -- 172.21.15.62:0/3170846096 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4d1800f460 con 0x7f4d2c101d30 2024-04-20T22:02:17.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:17.683+0000 7f4d31e00700 1 -- 172.21.15.62:0/3170846096 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d2c0ffec0 con 0x7f4d2c101d30 2024-04-20T22:02:17.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:17.683+0000 7f4d30a00700 1 -- 172.21.15.62:0/3170846096 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4d18017610 con 0x7f4d2c101d30 2024-04-20T22:02:17.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:17.683+0000 7f4d31e00700 1 -- 172.21.15.62:0/3170846096 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d2c1a2460 con 0x7f4d2c101d30 2024-04-20T22:02:17.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:17.684+0000 7f4d31e00700 1 -- 172.21.15.62:0/3170846096 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d2c1078a0 con 0x7f4d2c101d30 2024-04-20T22:02:17.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:17.685+0000 7f4d30a00700 1 -- 172.21.15.62:0/3170846096 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f4d1800f5d0 con 0x7f4d2c101d30 2024-04-20T22:02:17.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:17.686+0000 7f4d30a00700 1 --2- 172.21.15.62:0/3170846096 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f4d1006ecc0 0x7f4d10071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:02:17.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:17.686+0000 7f4d30a00700 1 -- 172.21.15.62:0/3170846096 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f4d18090420 con 0x7f4d2c101d30 2024-04-20T22:02:17.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:17.687+0000 7f4d2aa00700 1 --2- 172.21.15.62:0/3170846096 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f4d1006ecc0 0x7f4d10071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:02:17.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:17.687+0000 7f4d2aa00700 1 --2- 172.21.15.62:0/3170846096 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f4d1006ecc0 0x7f4d10071170 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f4d1c0097b0 tx=0x7f4d1c006d20 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:02:17.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:17.692+0000 7f4d30a00700 1 -- 172.21.15.62:0/3170846096 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f4d180591b0 con 0x7f4d2c101d30 2024-04-20T22:02:17.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:17.946+0000 7f4d31e00700 1 -- 172.21.15.62:0/3170846096 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4d2c061190 con 0x7f4d1006ecc0 2024-04-20T22:02:17.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:17.949+0000 7f4d30a00700 1 -- 172.21.15.62:0/3170846096 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f4d2c061190 con 0x7f4d1006ecc0 2024-04-20T22:02:17.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:17.952+0000 7f4d31e00700 1 -- 172.21.15.62:0/3170846096 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f4d1006ecc0 msgr2=0x7f4d10071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:02:17.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:17.952+0000 7f4d31e00700 1 --2- 172.21.15.62:0/3170846096 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f4d1006ecc0 0x7f4d10071170 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f4d1c0097b0 tx=0x7f4d1c006d20 comp rx=0 tx=0).stop 2024-04-20T22:02:17.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:17.952+0000 7f4d31e00700 1 -- 172.21.15.62:0/3170846096 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d2c101d30 msgr2=0x7f4d2c0ff5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:02:17.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:17.952+0000 7f4d31e00700 1 --2- 172.21.15.62:0/3170846096 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d2c101d30 0x7f4d2c0ff5c0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f4d1800b5c0 tx=0x7f4d1800be00 comp rx=0 tx=0).stop 2024-04-20T22:02:17.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:17.952+0000 7f4d31e00700 1 -- 172.21.15.62:0/3170846096 shutdown_connections 2024-04-20T22:02:17.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:17.952+0000 7f4d31e00700 1 --2- 172.21.15.62:0/3170846096 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f4d1006ecc0 0x7f4d10071170 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:02:17.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:17.952+0000 7f4d31e00700 1 --2- 172.21.15.62:0/3170846096 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4d2c101420 0x7f4d2c0ff080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:02:17.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:17.952+0000 7f4d31e00700 1 --2- 172.21.15.62:0/3170846096 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d2c101d30 0x7f4d2c0ff5c0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:02:17.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:17.952+0000 7f4d31e00700 1 -- 172.21.15.62:0/3170846096 >> 172.21.15.62:0/3170846096 conn(0x7f4d2c0fac80 msgr2=0x7f4d2c1049e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:02:17.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:17.953+0000 7f4d31e00700 1 -- 172.21.15.62:0/3170846096 shutdown_connections 2024-04-20T22:02:17.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:17.953+0000 7f4d31e00700 1 -- 172.21.15.62:0/3170846096 wait complete. 2024-04-20T22:02:17.969 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T22:02:18.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.114+0000 7fe425a00700 1 -- 172.21.15.62:0/2511489070 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe4201026a0 msgr2=0x7fe420102a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:02:18.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.114+0000 7fe425a00700 1 --2- 172.21.15.62:0/2511489070 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe4201026a0 0x7fe420102a70 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fe410009910 tx=0x7fe410009c20 comp rx=0 tx=0).stop 2024-04-20T22:02:18.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.114+0000 7fe425a00700 1 -- 172.21.15.62:0/2511489070 shutdown_connections 2024-04-20T22:02:18.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.114+0000 7fe425a00700 1 --2- 172.21.15.62:0/2511489070 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe420103040 0x7fe420107010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:02:18.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.114+0000 7fe425a00700 1 --2- 172.21.15.62:0/2511489070 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe4201026a0 0x7fe420102a70 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:02:18.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.114+0000 7fe425a00700 1 -- 172.21.15.62:0/2511489070 >> 172.21.15.62:0/2511489070 conn(0x7fe4200fdf40 msgr2=0x7fe420100350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:02:18.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.114+0000 7fe425a00700 1 -- 172.21.15.62:0/2511489070 shutdown_connections 2024-04-20T22:02:18.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.114+0000 7fe425a00700 1 -- 172.21.15.62:0/2511489070 wait complete. 2024-04-20T22:02:18.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.115+0000 7fe425a00700 1 Processor -- start 2024-04-20T22:02:18.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.115+0000 7fe425a00700 1 -- start start 2024-04-20T22:02:18.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.116+0000 7fe425a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe4201026a0 0x7fe420198200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:02:18.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.116+0000 7fe425a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe420103040 0x7fe420198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:02:18.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.116+0000 7fe425a00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe420198e20 con 0x7fe4201026a0 2024-04-20T22:02:18.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.116+0000 7fe425a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe42019cbb0 con 0x7fe420103040 2024-04-20T22:02:18.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.116+0000 7fe41ea00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe4201026a0 0x7fe420198200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:02:18.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.116+0000 7fe41ea00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe4201026a0 0x7fe420198200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:38068/0 (socket says 172.21.15.62:38068) 2024-04-20T22:02:18.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.116+0000 7fe41ea00700 1 -- 172.21.15.62:0/1847890977 learned_addr learned my addr 172.21.15.62:0/1847890977 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:02:18.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.117+0000 7fe41ea00700 1 -- 172.21.15.62:0/1847890977 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe420103040 msgr2=0x7fe420198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:02:18.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.117+0000 7fe41ea00700 1 --2- 172.21.15.62:0/1847890977 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe420103040 0x7fe420198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:02:18.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.117+0000 7fe41ea00700 1 -- 172.21.15.62:0/1847890977 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4100095f0 con 0x7fe4201026a0 2024-04-20T22:02:18.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.117+0000 7fe41ea00700 1 --2- 172.21.15.62:0/1847890977 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe4201026a0 0x7fe420198200 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fe410006010 tx=0x7fe41000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:02:18.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.117+0000 7fe417400700 1 -- 172.21.15.62:0/1847890977 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe41001d070 con 0x7fe4201026a0 2024-04-20T22:02:18.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.117+0000 7fe417400700 1 -- 172.21.15.62:0/1847890977 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe410004fb0 con 0x7fe4201026a0 2024-04-20T22:02:18.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.117+0000 7fe425a00700 1 -- 172.21.15.62:0/1847890977 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe42019ce30 con 0x7fe4201026a0 2024-04-20T22:02:18.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.118+0000 7fe417400700 1 -- 172.21.15.62:0/1847890977 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe410022ea0 con 0x7fe4201026a0 2024-04-20T22:02:18.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.118+0000 7fe425a00700 1 -- 172.21.15.62:0/1847890977 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe42019d320 con 0x7fe4201026a0 2024-04-20T22:02:18.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.119+0000 7fe425a00700 1 -- 172.21.15.62:0/1847890977 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe42010a9b0 con 0x7fe4201026a0 2024-04-20T22:02:18.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.120+0000 7fe417400700 1 -- 172.21.15.62:0/1847890977 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe410022470 con 0x7fe4201026a0 2024-04-20T22:02:18.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.121+0000 7fe417400700 1 --2- 172.21.15.62:0/1847890977 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe40406ecc0 0x7fe404071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:02:18.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.121+0000 7fe417400700 1 -- 172.21.15.62:0/1847890977 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fe4100904a0 con 0x7fe4201026a0 2024-04-20T22:02:18.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.121+0000 7fe41e000700 1 --2- 172.21.15.62:0/1847890977 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe40406ecc0 0x7fe404071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:02:18.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.122+0000 7fe41e000700 1 --2- 172.21.15.62:0/1847890977 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe40406ecc0 0x7fe404071170 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fe420199820 tx=0x7fe40c006d20 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:02:18.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.126+0000 7fe417400700 1 -- 172.21.15.62:0/1847890977 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fe410059260 con 0x7fe4201026a0 2024-04-20T22:02:18.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.373+0000 7fe425a00700 1 -- 172.21.15.62:0/1847890977 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe42019d600 con 0x7fe40406ecc0 2024-04-20T22:02:18.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.376+0000 7fe417400700 1 -- 172.21.15.62:0/1847890977 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fe42019d600 con 0x7fe40406ecc0 2024-04-20T22:02:18.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.378+0000 7fe425a00700 1 -- 172.21.15.62:0/1847890977 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe40406ecc0 msgr2=0x7fe404071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:02:18.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.378+0000 7fe425a00700 1 --2- 172.21.15.62:0/1847890977 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe40406ecc0 0x7fe404071170 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fe420199820 tx=0x7fe40c006d20 comp rx=0 tx=0).stop 2024-04-20T22:02:18.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.378+0000 7fe425a00700 1 -- 172.21.15.62:0/1847890977 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe4201026a0 msgr2=0x7fe420198200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:02:18.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.378+0000 7fe425a00700 1 --2- 172.21.15.62:0/1847890977 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe4201026a0 0x7fe420198200 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fe410006010 tx=0x7fe41000be20 comp rx=0 tx=0).stop 2024-04-20T22:02:18.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.378+0000 7fe425a00700 1 -- 172.21.15.62:0/1847890977 shutdown_connections 2024-04-20T22:02:18.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.378+0000 7fe425a00700 1 --2- 172.21.15.62:0/1847890977 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe40406ecc0 0x7fe404071170 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:02:18.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.378+0000 7fe425a00700 1 --2- 172.21.15.62:0/1847890977 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe4201026a0 0x7fe420198200 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:02:18.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.378+0000 7fe425a00700 1 --2- 172.21.15.62:0/1847890977 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe420103040 0x7fe420198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:02:18.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.378+0000 7fe425a00700 1 -- 172.21.15.62:0/1847890977 >> 172.21.15.62:0/1847890977 conn(0x7fe4200fdf40 msgr2=0x7fe4200ff8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:02:18.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.379+0000 7fe425a00700 1 -- 172.21.15.62:0/1847890977 shutdown_connections 2024-04-20T22:02:18.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.379+0000 7fe425a00700 1 -- 172.21.15.62:0/1847890977 wait complete. 2024-04-20T22:02:18.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.538+0000 7fd7d9800700 1 -- 172.21.15.62:0/1508226499 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd7d4102470 msgr2=0x7fd7d41028e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:02:18.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.538+0000 7fd7d9800700 1 --2- 172.21.15.62:0/1508226499 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd7d4102470 0x7fd7d41028e0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fd7c0009910 tx=0x7fd7c0009c20 comp rx=0 tx=0).stop 2024-04-20T22:02:18.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.538+0000 7fd7d9800700 1 -- 172.21.15.62:0/1508226499 shutdown_connections 2024-04-20T22:02:18.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.538+0000 7fd7d9800700 1 --2- 172.21.15.62:0/1508226499 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd7d4102470 0x7fd7d41028e0 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:02:18.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.538+0000 7fd7d9800700 1 --2- 172.21.15.62:0/1508226499 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd7d4108470 0x7fd7d4108840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:02:18.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.538+0000 7fd7d9800700 1 -- 172.21.15.62:0/1508226499 >> 172.21.15.62:0/1508226499 conn(0x7fd7d40fdf90 msgr2=0x7fd7d41003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:02:18.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.538+0000 7fd7d9800700 1 -- 172.21.15.62:0/1508226499 shutdown_connections 2024-04-20T22:02:18.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.538+0000 7fd7d9800700 1 -- 172.21.15.62:0/1508226499 wait complete. 2024-04-20T22:02:18.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.540+0000 7fd7d9800700 1 Processor -- start 2024-04-20T22:02:18.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.540+0000 7fd7d9800700 1 -- start start 2024-04-20T22:02:18.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.540+0000 7fd7d9800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd7d4102470 0x7fd7d41981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:02:18.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.540+0000 7fd7d9800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd7d4108470 0x7fd7d4198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:02:18.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.540+0000 7fd7d9800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7d4198d80 con 0x7fd7d4108470 2024-04-20T22:02:18.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.540+0000 7fd7d9800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7d4198ec0 con 0x7fd7d4102470 2024-04-20T22:02:18.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.541+0000 7fd7d2000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd7d4108470 0x7fd7d4198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:02:18.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.541+0000 7fd7d2000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd7d4108470 0x7fd7d4198730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:38072/0 (socket says 172.21.15.62:38072) 2024-04-20T22:02:18.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.541+0000 7fd7d2000700 1 -- 172.21.15.62:0/144857182 learned_addr learned my addr 172.21.15.62:0/144857182 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:02:18.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.541+0000 7fd7d2a00700 1 --2- 172.21.15.62:0/144857182 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd7d4102470 0x7fd7d41981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:02:18.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.541+0000 7fd7d2000700 1 -- 172.21.15.62:0/144857182 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd7d4102470 msgr2=0x7fd7d41981f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:02:18.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.541+0000 7fd7d2000700 1 --2- 172.21.15.62:0/144857182 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd7d4102470 0x7fd7d41981f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:02:18.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.541+0000 7fd7d2000700 1 -- 172.21.15.62:0/144857182 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7c00095f0 con 0x7fd7d4108470 2024-04-20T22:02:18.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.542+0000 7fd7d2000700 1 --2- 172.21.15.62:0/144857182 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd7d4108470 0x7fd7d4198730 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fd7c0006010 tx=0x7fd7c000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:02:18.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.542+0000 7fd7cb400700 1 -- 172.21.15.62:0/144857182 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd7c001d070 con 0x7fd7d4108470 2024-04-20T22:02:18.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.542+0000 7fd7cb400700 1 -- 172.21.15.62:0/144857182 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd7c0004fb0 con 0x7fd7d4108470 2024-04-20T22:02:18.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.542+0000 7fd7cb400700 1 -- 172.21.15.62:0/144857182 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd7c0022ea0 con 0x7fd7d4108470 2024-04-20T22:02:18.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.542+0000 7fd7d9800700 1 -- 172.21.15.62:0/144857182 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7d419ccb0 con 0x7fd7d4108470 2024-04-20T22:02:18.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.542+0000 7fd7d9800700 1 -- 172.21.15.62:0/144857182 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7d419d1a0 con 0x7fd7d4108470 2024-04-20T22:02:18.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.543+0000 7fd7d9800700 1 -- 172.21.15.62:0/144857182 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7d410a970 con 0x7fd7d4108470 2024-04-20T22:02:18.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.544+0000 7fd7cb400700 1 -- 172.21.15.62:0/144857182 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fd7c0022470 con 0x7fd7d4108470 2024-04-20T22:02:18.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.545+0000 7fd7cb400700 1 --2- 172.21.15.62:0/144857182 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd7b806ecc0 0x7fd7b8071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:02:18.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.545+0000 7fd7cb400700 1 -- 172.21.15.62:0/144857182 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fd7c00903c0 con 0x7fd7d4108470 2024-04-20T22:02:18.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.546+0000 7fd7d2a00700 1 --2- 172.21.15.62:0/144857182 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd7b806ecc0 0x7fd7b8071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:02:18.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.546+0000 7fd7d2a00700 1 --2- 172.21.15.62:0/144857182 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd7b806ecc0 0x7fd7b8071170 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fd7c4005ea0 tx=0x7fd7c4005e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:02:18.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.550+0000 7fd7cb400700 1 -- 172.21.15.62:0/144857182 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fd7c0059200 con 0x7fd7d4108470 2024-04-20T22:02:18.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.795+0000 7fd7d9800700 1 -- 172.21.15.62:0/144857182 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fd7d4199630 con 0x7fd7b806ecc0 2024-04-20T22:02:18.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.828+0000 7fd7cb400700 1 -- 172.21.15.62:0/144857182 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fd7d4199630 con 0x7fd7b806ecc0 2024-04-20T22:02:18.828 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T22:02:18.828 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (28m) 4m ago 42m 15.3M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T22:02:18.828 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (42m) 4m ago 42m 10.4M - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T22:02:18.828 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (40m) 4m ago 40m 10.1M - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T22:02:18.828 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (24m) 4m ago 42m 6660k - 19.0.0-3133-ga8c84f59 76bc6b91b726 2825d5ba11c6 2024-04-20T22:02:18.829 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (24m) 4m ago 40m 6665k - 19.0.0-3133-ga8c84f59 76bc6b91b726 20f746fa1bba 2024-04-20T22:02:18.829 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (27m) 4m ago 41m 73.5M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T22:02:18.829 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (19m) 4m ago 34m 18.5M - 19.0.0-3133-ga8c84f59 76bc6b91b726 de80bc90a683 2024-04-20T22:02:18.829 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (20m) 4m ago 34m 18.7M - 19.0.0-3133-ga8c84f59 76bc6b91b726 cd793b7ee98e 2024-04-20T22:02:18.829 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (4m) 4m ago 34m 11.0M - 19.0.0-3133-ga8c84f59 76bc6b91b726 0e7b0bc4d1ab 2024-04-20T22:02:18.829 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (34m) 4m ago 34m 18.6M - 18.2.0 dc2bc1663786 f2eec2ee8951 2024-04-20T22:02:18.829 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (31m) 4m ago 43m 513M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T22:02:18.829 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (29m) 4m ago 40m 459M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T22:02:18.829 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (25m) 4m ago 43m 71.7M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 019eb1479485 2024-04-20T22:02:18.829 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (25m) 4m ago 40m 51.0M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 55aac560aab6 2024-04-20T22:02:18.829 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (29m) 4m ago 42m 8472k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T22:02:18.829 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (29m) 4m ago 40m 8048k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T22:02:18.829 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (23m) 4m ago 39m 91.2M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 ee2bb901f26e 2024-04-20T22:02:18.829 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (23m) 4m ago 39m 94.6M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 9075152d4162 2024-04-20T22:02:18.830 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (22m) 4m ago 38m 78.1M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 2d4f3e38dc0e 2024-04-20T22:02:18.830 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (21m) 4m ago 37m 92.6M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 8f97b17ae0eb 2024-04-20T22:02:18.830 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (21m) 4m ago 37m 95.6M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 653d8079967d 2024-04-20T22:02:18.830 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (20m) 4m ago 36m 89.2M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 596dded14c12 2024-04-20T22:02:18.830 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (28m) 4m ago 41m 82.1M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T22:02:18.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.831+0000 7fd7d9800700 1 -- 172.21.15.62:0/144857182 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd7b806ecc0 msgr2=0x7fd7b8071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:02:18.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.831+0000 7fd7d9800700 1 --2- 172.21.15.62:0/144857182 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd7b806ecc0 0x7fd7b8071170 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fd7c4005ea0 tx=0x7fd7c4005e30 comp rx=0 tx=0).stop 2024-04-20T22:02:18.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.831+0000 7fd7d9800700 1 -- 172.21.15.62:0/144857182 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd7d4108470 msgr2=0x7fd7d4198730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:02:18.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.831+0000 7fd7d9800700 1 --2- 172.21.15.62:0/144857182 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd7d4108470 0x7fd7d4198730 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fd7c0006010 tx=0x7fd7c000be20 comp rx=0 tx=0).stop 2024-04-20T22:02:18.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.831+0000 7fd7d9800700 1 -- 172.21.15.62:0/144857182 shutdown_connections 2024-04-20T22:02:18.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.831+0000 7fd7d9800700 1 --2- 172.21.15.62:0/144857182 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd7b806ecc0 0x7fd7b8071170 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:02:18.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.831+0000 7fd7d9800700 1 --2- 172.21.15.62:0/144857182 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd7d4102470 0x7fd7d41981f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:02:18.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.831+0000 7fd7d9800700 1 --2- 172.21.15.62:0/144857182 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd7d4108470 0x7fd7d4198730 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:02:18.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.831+0000 7fd7d9800700 1 -- 172.21.15.62:0/144857182 >> 172.21.15.62:0/144857182 conn(0x7fd7d40fdf90 msgr2=0x7fd7d40ff3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:02:18.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.832+0000 7fd7d9800700 1 -- 172.21.15.62:0/144857182 shutdown_connections 2024-04-20T22:02:18.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.832+0000 7fd7d9800700 1 -- 172.21.15.62:0/144857182 wait complete. 2024-04-20T22:02:18.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.994+0000 7fe38d000700 1 -- 172.21.15.62:0/122301581 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe388068540 msgr2=0x7fe388068910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:02:18.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.994+0000 7fe38d000700 1 --2- 172.21.15.62:0/122301581 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe388068540 0x7fe388068910 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fe378009940 tx=0x7fe378009c50 comp rx=0 tx=0).stop 2024-04-20T22:02:18.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.994+0000 7fe38d000700 1 -- 172.21.15.62:0/122301581 shutdown_connections 2024-04-20T22:02:18.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.994+0000 7fe38d000700 1 --2- 172.21.15.62:0/122301581 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe388068ee0 0x7fe388110a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:02:18.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.994+0000 7fe38d000700 1 --2- 172.21.15.62:0/122301581 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe388068540 0x7fe388068910 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:02:18.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.994+0000 7fe38d000700 1 -- 172.21.15.62:0/122301581 >> 172.21.15.62:0/122301581 conn(0x7fe388075130 msgr2=0x7fe388075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:02:18.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.995+0000 7fe38d000700 1 -- 172.21.15.62:0/122301581 shutdown_connections 2024-04-20T22:02:18.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.995+0000 7fe38d000700 1 -- 172.21.15.62:0/122301581 wait complete. 2024-04-20T22:02:18.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.996+0000 7fe38d000700 1 Processor -- start 2024-04-20T22:02:18.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.996+0000 7fe38d000700 1 -- start start 2024-04-20T22:02:18.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.997+0000 7fe38d000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe388068540 0x7fe38819c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:02:18.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.997+0000 7fe38d000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe388068ee0 0x7fe38819cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:02:18.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.997+0000 7fe38d000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe38819d220 con 0x7fe388068540 2024-04-20T22:02:18.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.997+0000 7fe38d000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3881a0fb0 con 0x7fe388068ee0 2024-04-20T22:02:18.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.997+0000 7fe385e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe388068540 0x7fe38819c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:02:18.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.997+0000 7fe385e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe388068540 0x7fe38819c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:38088/0 (socket says 172.21.15.62:38088) 2024-04-20T22:02:18.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.997+0000 7fe385e00700 1 -- 172.21.15.62:0/11287848 learned_addr learned my addr 172.21.15.62:0/11287848 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:02:18.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.998+0000 7fe385e00700 1 -- 172.21.15.62:0/11287848 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe388068ee0 msgr2=0x7fe38819cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:02:18.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.998+0000 7fe385e00700 1 --2- 172.21.15.62:0/11287848 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe388068ee0 0x7fe38819cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:02:18.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.998+0000 7fe385e00700 1 -- 172.21.15.62:0/11287848 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3780095f0 con 0x7fe388068540 2024-04-20T22:02:18.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.998+0000 7fe385e00700 1 --2- 172.21.15.62:0/11287848 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe388068540 0x7fe38819c600 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fe378005620 tx=0x7fe378005700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:02:18.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.998+0000 7fe37ea00700 1 -- 172.21.15.62:0/11287848 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe37801d070 con 0x7fe388068540 2024-04-20T22:02:18.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.998+0000 7fe37ea00700 1 -- 172.21.15.62:0/11287848 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe37800f460 con 0x7fe388068540 2024-04-20T22:02:18.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.998+0000 7fe38d000700 1 -- 172.21.15.62:0/11287848 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3881a1230 con 0x7fe388068540 2024-04-20T22:02:18.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.998+0000 7fe37ea00700 1 -- 172.21.15.62:0/11287848 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe378021640 con 0x7fe388068540 2024-04-20T22:02:18.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:18.999+0000 7fe38d000700 1 -- 172.21.15.62:0/11287848 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3881a1780 con 0x7fe388068540 2024-04-20T22:02:19.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.000+0000 7fe38d000700 1 -- 172.21.15.62:0/11287848 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe388101410 con 0x7fe388068540 2024-04-20T22:02:19.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.001+0000 7fe37ea00700 1 -- 172.21.15.62:0/11287848 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe37800f5d0 con 0x7fe388068540 2024-04-20T22:02:19.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.001+0000 7fe37ea00700 1 --2- 172.21.15.62:0/11287848 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe36c06ecc0 0x7fe36c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:02:19.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.002+0000 7fe37ea00700 1 -- 172.21.15.62:0/11287848 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fe378090350 con 0x7fe388068540 2024-04-20T22:02:19.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.002+0000 7fe385400700 1 --2- 172.21.15.62:0/11287848 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe36c06ecc0 0x7fe36c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:02:19.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.003+0000 7fe385400700 1 --2- 172.21.15.62:0/11287848 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe36c06ecc0 0x7fe36c071170 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fe38819dc20 tx=0x7fe374006d20 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:02:19.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.006+0000 7fe37ea00700 1 -- 172.21.15.62:0/11287848 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fe378059060 con 0x7fe388068540 2024-04-20T22:02:19.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.334+0000 7fe38d000700 1 -- 172.21.15.62:0/11287848 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fe38819da00 con 0x7fe388068540 2024-04-20T22:02:19.334 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:02:19 smithi062 ceph-mon[76322]: pgmap v811: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:02:19.334 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:02:19 smithi062 ceph-mon[76322]: from='client.35460 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:02:19.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.335+0000 7fe37ea00700 1 -- 172.21.15.62:0/11287848 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7fe37805c720 con 0x7fe388068540 2024-04-20T22:02:19.335 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:02:19.335 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T22:02:19.335 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:02:19.335 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:02:19.335 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T22:02:19.335 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:02:19.335 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:02:19.335 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T22:02:19.335 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 6 2024-04-20T22:02:19.335 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:02:19.335 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T22:02:19.335 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 1, 2024-04-20T22:02:19.335 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 3 2024-04-20T22:02:19.336 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:02:19.336 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T22:02:19.336 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 1, 2024-04-20T22:02:19.336 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 13 2024-04-20T22:02:19.336 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T22:02:19.336 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:02:19.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.337+0000 7fe38d000700 1 -- 172.21.15.62:0/11287848 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe36c06ecc0 msgr2=0x7fe36c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:02:19.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.337+0000 7fe38d000700 1 --2- 172.21.15.62:0/11287848 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe36c06ecc0 0x7fe36c071170 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fe38819dc20 tx=0x7fe374006d20 comp rx=0 tx=0).stop 2024-04-20T22:02:19.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.337+0000 7fe38d000700 1 -- 172.21.15.62:0/11287848 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe388068540 msgr2=0x7fe38819c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:02:19.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.338+0000 7fe38d000700 1 --2- 172.21.15.62:0/11287848 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe388068540 0x7fe38819c600 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fe378005620 tx=0x7fe378005700 comp rx=0 tx=0).stop 2024-04-20T22:02:19.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.338+0000 7fe38d000700 1 -- 172.21.15.62:0/11287848 shutdown_connections 2024-04-20T22:02:19.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.338+0000 7fe38d000700 1 --2- 172.21.15.62:0/11287848 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe36c06ecc0 0x7fe36c071170 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:02:19.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.338+0000 7fe38d000700 1 --2- 172.21.15.62:0/11287848 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe388068540 0x7fe38819c600 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:02:19.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.338+0000 7fe38d000700 1 --2- 172.21.15.62:0/11287848 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe388068ee0 0x7fe38819cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:02:19.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.338+0000 7fe38d000700 1 -- 172.21.15.62:0/11287848 >> 172.21.15.62:0/11287848 conn(0x7fe388075130 msgr2=0x7fe38810f070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:02:19.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.338+0000 7fe38d000700 1 -- 172.21.15.62:0/11287848 shutdown_connections 2024-04-20T22:02:19.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.338+0000 7fe38d000700 1 -- 172.21.15.62:0/11287848 wait complete. 2024-04-20T22:02:19.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:02:19 smithi189 ceph-mon[57022]: pgmap v811: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:02:19.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:02:19 smithi189 ceph-mon[57022]: from='client.35460 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:02:19.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.489+0000 7f7d17800700 1 -- 172.21.15.62:0/1508456503 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d1010c8f0 msgr2=0x7f7d1010ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:02:19.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.489+0000 7f7d17800700 1 --2- 172.21.15.62:0/1508456503 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d1010c8f0 0x7f7d1010ccc0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f7d04009910 tx=0x7f7d04009c20 comp rx=0 tx=0).stop 2024-04-20T22:02:19.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.489+0000 7f7d17800700 1 -- 172.21.15.62:0/1508456503 shutdown_connections 2024-04-20T22:02:19.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.489+0000 7f7d17800700 1 --2- 172.21.15.62:0/1508456503 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d10106b80 0x7f7d10106ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:02:19.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.489+0000 7f7d17800700 1 --2- 172.21.15.62:0/1508456503 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d1010c8f0 0x7f7d1010ccc0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:02:19.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.489+0000 7f7d17800700 1 -- 172.21.15.62:0/1508456503 >> 172.21.15.62:0/1508456503 conn(0x7f7d10076060 msgr2=0x7f7d10078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:02:19.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.489+0000 7f7d17800700 1 -- 172.21.15.62:0/1508456503 shutdown_connections 2024-04-20T22:02:19.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.489+0000 7f7d17800700 1 -- 172.21.15.62:0/1508456503 wait complete. 2024-04-20T22:02:19.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.491+0000 7f7d17800700 1 Processor -- start 2024-04-20T22:02:19.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.491+0000 7f7d17800700 1 -- start start 2024-04-20T22:02:19.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.491+0000 7f7d17800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d10106b80 0x7f7d1019c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:02:19.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.491+0000 7f7d17800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d1010c8f0 0x7f7d1019ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:02:19.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.491+0000 7f7d17800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d1019d120 con 0x7f7d1010c8f0 2024-04-20T22:02:19.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.491+0000 7f7d17800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d101a0eb0 con 0x7f7d10106b80 2024-04-20T22:02:19.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.491+0000 7f7d0fe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d1010c8f0 0x7f7d1019ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:02:19.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.491+0000 7f7d0fe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d1010c8f0 0x7f7d1019ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:38116/0 (socket says 172.21.15.62:38116) 2024-04-20T22:02:19.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.491+0000 7f7d0fe00700 1 -- 172.21.15.62:0/1406278730 learned_addr learned my addr 172.21.15.62:0/1406278730 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:02:19.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.492+0000 7f7d0fe00700 1 -- 172.21.15.62:0/1406278730 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d10106b80 msgr2=0x7f7d1019c500 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-20T22:02:19.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.492+0000 7f7d0fe00700 1 --2- 172.21.15.62:0/1406278730 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d10106b80 0x7f7d1019c500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:02:19.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.492+0000 7f7d0fe00700 1 -- 172.21.15.62:0/1406278730 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d040095f0 con 0x7f7d1010c8f0 2024-04-20T22:02:19.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.492+0000 7f7d0fe00700 1 --2- 172.21.15.62:0/1406278730 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d1010c8f0 0x7f7d1019ca40 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f7cfc00d8d0 tx=0x7f7cfc00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:02:19.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.492+0000 7f7d0d600700 1 -- 172.21.15.62:0/1406278730 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7cfc009940 con 0x7f7d1010c8f0 2024-04-20T22:02:19.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.493+0000 7f7d0d600700 1 -- 172.21.15.62:0/1406278730 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7cfc010460 con 0x7f7d1010c8f0 2024-04-20T22:02:19.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.493+0000 7f7d0d600700 1 -- 172.21.15.62:0/1406278730 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7cfc00f5f0 con 0x7f7d1010c8f0 2024-04-20T22:02:19.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.493+0000 7f7d17800700 1 -- 172.21.15.62:0/1406278730 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d101a1190 con 0x7f7d1010c8f0 2024-04-20T22:02:19.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.493+0000 7f7d17800700 1 -- 172.21.15.62:0/1406278730 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d101a16e0 con 0x7f7d1010c8f0 2024-04-20T22:02:19.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.494+0000 7f7d17800700 1 -- 172.21.15.62:0/1406278730 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d10109b20 con 0x7f7d1010c8f0 2024-04-20T22:02:19.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.495+0000 7f7d0d600700 1 -- 172.21.15.62:0/1406278730 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f7cfc0105d0 con 0x7f7d1010c8f0 2024-04-20T22:02:19.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.496+0000 7f7d0d600700 1 --2- 172.21.15.62:0/1406278730 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7cf806ecc0 0x7f7cf8071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:02:19.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.496+0000 7f7d0d600700 1 -- 172.21.15.62:0/1406278730 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f7cfc08ee10 con 0x7f7d1010c8f0 2024-04-20T22:02:19.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.496+0000 7f7d14e00700 1 --2- 172.21.15.62:0/1406278730 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7cf806ecc0 0x7f7cf8071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:02:19.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.497+0000 7f7d14e00700 1 --2- 172.21.15.62:0/1406278730 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7cf806ecc0 0x7f7cf8071170 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f7d04000c00 tx=0x7f7d04005fb0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:02:19.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.501+0000 7f7d0d600700 1 -- 172.21.15.62:0/1406278730 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f7cfc057ba0 con 0x7f7d1010c8f0 2024-04-20T22:02:19.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.790+0000 7f7d17800700 1 -- 172.21.15.62:0/1406278730 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f7d1004f2a0 con 0x7f7d1010c8f0 2024-04-20T22:02:19.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.791+0000 7f7d0d600700 1 -- 172.21.15.62:0/1406278730 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 27 v27) v1 ==== 76+0+1888 (secure 0 0 0) 0x7f7cfc016770 con 0x7f7d1010c8f0 2024-04-20T22:02:19.792 INFO:teuthology.orchestra.run.smithi062.stdout:e27 2024-04-20T22:02:19.792 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T22:02:19.792 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:02:19.792 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T22:02:19.792 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:02:19.792 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T22:02:19.792 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T22:02:19.792 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 20 2024-04-20T22:02:19.792 INFO:teuthology.orchestra.run.smithi062.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-20T22:02:19.793 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T22:02:19.793 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T21:42:01.559061+0000 2024-04-20T22:02:19.793 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T22:02:19.793 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T22:02:19.793 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T22:02:19.793 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T22:02:19.793 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T22:02:19.793 INFO:teuthology.orchestra.run.smithi062.stdout:max_xattr_size 65536 2024-04-20T22:02:19.793 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T22:02:19.793 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T22:02:19.793 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 81 2024-04-20T22:02:19.793 INFO:teuthology.orchestra.run.smithi062.stdout:snap_rstat 1 2024-04-20T22:02:19.793 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:02:19.794 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 2 2024-04-20T22:02:19.794 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1 2024-04-20T22:02:19.794 INFO:teuthology.orchestra.run.smithi062.stdout:up {} 2024-04-20T22:02:19.794 INFO:teuthology.orchestra.run.smithi062.stdout:failed 0,1 2024-04-20T22:02:19.794 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T22:02:19.794 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T22:02:19.794 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T22:02:19.794 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T22:02:19.794 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data enabled 2024-04-20T22:02:19.794 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T22:02:19.794 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T22:02:19.795 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 1 2024-04-20T22:02:19.795 INFO:teuthology.orchestra.run.smithi062.stdout:qdb_cluster leader: 0 members: 2024-04-20T22:02:19.795 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:02:19.795 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:02:19.795 INFO:teuthology.orchestra.run.smithi062.stdout:Standby daemons: 2024-04-20T22:02:19.795 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:02:19.795 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ryqblb{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6826/4113916205,v1:172.21.15.62:6827/4113916205] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:02:19.795 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ozizwi{-1:34476} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6828/3801975165,v1:172.21.15.62:6829/3801975165] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:02:19.795 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.xbgspw{-1:44285} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6826/3996040526,v1:172.21.15.189:6827/3996040526] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T22:02:19.795 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.dvimer{-1:44727} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6824/3576699232,v1:172.21.15.189:6825/3576699232] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:02:19.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.795+0000 7f7d17800700 1 -- 172.21.15.62:0/1406278730 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7cf806ecc0 msgr2=0x7f7cf8071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:02:19.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.795+0000 7f7d17800700 1 --2- 172.21.15.62:0/1406278730 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7cf806ecc0 0x7f7cf8071170 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f7d04000c00 tx=0x7f7d04005fb0 comp rx=0 tx=0).stop 2024-04-20T22:02:19.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.795+0000 7f7d17800700 1 -- 172.21.15.62:0/1406278730 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d1010c8f0 msgr2=0x7f7d1019ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:02:19.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.795+0000 7f7d17800700 1 --2- 172.21.15.62:0/1406278730 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d1010c8f0 0x7f7d1019ca40 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f7cfc00d8d0 tx=0x7f7cfc00dc90 comp rx=0 tx=0).stop 2024-04-20T22:02:19.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.795+0000 7f7d17800700 1 -- 172.21.15.62:0/1406278730 shutdown_connections 2024-04-20T22:02:19.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.795+0000 7f7d17800700 1 --2- 172.21.15.62:0/1406278730 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7cf806ecc0 0x7f7cf8071170 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:02:19.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.795+0000 7f7d17800700 1 --2- 172.21.15.62:0/1406278730 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d10106b80 0x7f7d1019c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:02:19.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.795+0000 7f7d17800700 1 --2- 172.21.15.62:0/1406278730 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d1010c8f0 0x7f7d1019ca40 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:02:19.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.795+0000 7f7d17800700 1 -- 172.21.15.62:0/1406278730 >> 172.21.15.62:0/1406278730 conn(0x7f7d10076060 msgr2=0x7f7d10077aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:02:19.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.796+0000 7f7d17800700 1 -- 172.21.15.62:0/1406278730 shutdown_connections 2024-04-20T22:02:19.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.796+0000 7f7d17800700 1 -- 172.21.15.62:0/1406278730 wait complete. 2024-04-20T22:02:19.797 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 27 2024-04-20T22:02:19.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.959+0000 7faabc400700 1 -- 172.21.15.62:0/463705911 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faab4106b80 msgr2=0x7faab4106ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:02:19.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.959+0000 7faabc400700 1 --2- 172.21.15.62:0/463705911 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faab4106b80 0x7faab4106ff0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7faaa0009910 tx=0x7faaa0009c20 comp rx=0 tx=0).stop 2024-04-20T22:02:19.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.959+0000 7faabc400700 1 -- 172.21.15.62:0/463705911 shutdown_connections 2024-04-20T22:02:19.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.959+0000 7faabc400700 1 --2- 172.21.15.62:0/463705911 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faab4106b80 0x7faab4106ff0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:02:19.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.959+0000 7faabc400700 1 --2- 172.21.15.62:0/463705911 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faab410c8f0 0x7faab410ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:02:19.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.959+0000 7faabc400700 1 -- 172.21.15.62:0/463705911 >> 172.21.15.62:0/463705911 conn(0x7faab4076060 msgr2=0x7faab4078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:02:19.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.959+0000 7faabc400700 1 -- 172.21.15.62:0/463705911 shutdown_connections 2024-04-20T22:02:19.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.959+0000 7faabc400700 1 -- 172.21.15.62:0/463705911 wait complete. 2024-04-20T22:02:19.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.960+0000 7faabc400700 1 Processor -- start 2024-04-20T22:02:19.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.960+0000 7faabc400700 1 -- start start 2024-04-20T22:02:19.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.961+0000 7faabc400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faab4106b80 0x7faab419c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:02:19.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.961+0000 7faabc400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faab410c8f0 0x7faab419c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:02:19.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.961+0000 7faabc400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faab419d060 con 0x7faab410c8f0 2024-04-20T22:02:19.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.961+0000 7faabc400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faab41a0df0 con 0x7faab4106b80 2024-04-20T22:02:19.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.961+0000 7faab9000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faab410c8f0 0x7faab419c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:02:19.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.961+0000 7faab9000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faab410c8f0 0x7faab419c980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:38134/0 (socket says 172.21.15.62:38134) 2024-04-20T22:02:19.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.961+0000 7faab9000700 1 -- 172.21.15.62:0/3760811820 learned_addr learned my addr 172.21.15.62:0/3760811820 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:02:19.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.962+0000 7faab9a00700 1 --2- 172.21.15.62:0/3760811820 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faab4106b80 0x7faab419c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:02:19.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.962+0000 7faab9000700 1 -- 172.21.15.62:0/3760811820 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faab4106b80 msgr2=0x7faab419c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:02:19.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.962+0000 7faab9000700 1 --2- 172.21.15.62:0/3760811820 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faab4106b80 0x7faab419c440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:02:19.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.962+0000 7faab9000700 1 -- 172.21.15.62:0/3760811820 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaa00095f0 con 0x7faab410c8f0 2024-04-20T22:02:19.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.962+0000 7faab9000700 1 --2- 172.21.15.62:0/3760811820 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faab410c8f0 0x7faab419c980 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7faaa00092e0 tx=0x7faaa000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:02:19.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.962+0000 7faab2000700 1 -- 172.21.15.62:0/3760811820 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faaa001d070 con 0x7faab410c8f0 2024-04-20T22:02:19.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.962+0000 7faab2000700 1 -- 172.21.15.62:0/3760811820 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7faaa0005e00 con 0x7faab410c8f0 2024-04-20T22:02:19.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.962+0000 7faab2000700 1 -- 172.21.15.62:0/3760811820 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faaa0017400 con 0x7faab410c8f0 2024-04-20T22:02:19.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.962+0000 7faabc400700 1 -- 172.21.15.62:0/3760811820 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faab41a1070 con 0x7faab410c8f0 2024-04-20T22:02:19.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.963+0000 7faabc400700 1 -- 172.21.15.62:0/3760811820 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faab41a1560 con 0x7faab410c8f0 2024-04-20T22:02:19.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.964+0000 7faabc400700 1 -- 172.21.15.62:0/3760811820 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faab410ecc0 con 0x7faab410c8f0 2024-04-20T22:02:19.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.965+0000 7faab2000700 1 -- 172.21.15.62:0/3760811820 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7faaa000f460 con 0x7faab410c8f0 2024-04-20T22:02:19.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.966+0000 7faab2000700 1 --2- 172.21.15.62:0/3760811820 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7faa9c06ecc0 0x7faa9c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:02:19.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.967+0000 7faab2000700 1 -- 172.21.15.62:0/3760811820 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7faaa0090330 con 0x7faab410c8f0 2024-04-20T22:02:19.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.967+0000 7faab9a00700 1 --2- 172.21.15.62:0/3760811820 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7faa9c06ecc0 0x7faa9c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:02:19.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.967+0000 7faab9a00700 1 --2- 172.21.15.62:0/3760811820 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7faa9c06ecc0 0x7faa9c071170 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7faaa8005950 tx=0x7faaa80058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:02:19.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:19.972+0000 7faab2000700 1 -- 172.21.15.62:0/3760811820 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7faaa00590c0 con 0x7faab410c8f0 2024-04-20T22:02:20.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:20.232+0000 7faabc400700 1 -- 172.21.15.62:0/3760811820 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7faab402cca0 con 0x7faa9c06ecc0 2024-04-20T22:02:20.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:20.235+0000 7faab2000700 1 -- 172.21.15.62:0/3760811820 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7faab402cca0 con 0x7faa9c06ecc0 2024-04-20T22:02:20.234 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:02:20.235 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T22:02:20.235 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T22:02:20.235 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-20T22:02:20.235 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T22:02:20.235 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr", 2024-04-20T22:02:20.235 INFO:teuthology.orchestra.run.smithi062.stdout: "crash", 2024-04-20T22:02:20.235 INFO:teuthology.orchestra.run.smithi062.stdout: "mon", 2024-04-20T22:02:20.235 INFO:teuthology.orchestra.run.smithi062.stdout: "osd" 2024-04-20T22:02:20.235 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T22:02:20.235 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "15/23 daemons upgraded", 2024-04-20T22:02:20.235 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading mds daemons", 2024-04-20T22:02:20.235 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T22:02:20.235 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:02:20.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:20.238+0000 7faabc400700 1 -- 172.21.15.62:0/3760811820 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7faa9c06ecc0 msgr2=0x7faa9c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:02:20.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:20.238+0000 7faabc400700 1 --2- 172.21.15.62:0/3760811820 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7faa9c06ecc0 0x7faa9c071170 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7faaa8005950 tx=0x7faaa80058e0 comp rx=0 tx=0).stop 2024-04-20T22:02:20.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:20.238+0000 7faabc400700 1 -- 172.21.15.62:0/3760811820 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faab410c8f0 msgr2=0x7faab419c980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:02:20.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:20.238+0000 7faabc400700 1 --2- 172.21.15.62:0/3760811820 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faab410c8f0 0x7faab419c980 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7faaa00092e0 tx=0x7faaa000be20 comp rx=0 tx=0).stop 2024-04-20T22:02:20.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:20.238+0000 7faabc400700 1 -- 172.21.15.62:0/3760811820 shutdown_connections 2024-04-20T22:02:20.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:20.238+0000 7faabc400700 1 --2- 172.21.15.62:0/3760811820 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7faa9c06ecc0 0x7faa9c071170 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:02:20.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:20.238+0000 7faabc400700 1 --2- 172.21.15.62:0/3760811820 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faab4106b80 0x7faab419c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:02:20.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:20.238+0000 7faabc400700 1 --2- 172.21.15.62:0/3760811820 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faab410c8f0 0x7faab419c980 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:02:20.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:20.238+0000 7faabc400700 1 -- 172.21.15.62:0/3760811820 >> 172.21.15.62:0/3760811820 conn(0x7faab4076060 msgr2=0x7faab4077ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:02:20.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:20.238+0000 7faabc400700 1 -- 172.21.15.62:0/3760811820 shutdown_connections 2024-04-20T22:02:20.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:20.238+0000 7faabc400700 1 -- 172.21.15.62:0/3760811820 wait complete. 2024-04-20T22:02:20.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:20.406+0000 7efdc5e00700 1 -- 172.21.15.62:0/2866757373 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efdc01085b0 msgr2=0x7efdc0108980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:02:20.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:20.406+0000 7efdc5e00700 1 --2- 172.21.15.62:0/2866757373 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efdc01085b0 0x7efdc0108980 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7efdb0009910 tx=0x7efdb0009c20 comp rx=0 tx=0).stop 2024-04-20T22:02:20.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:20.406+0000 7efdc5e00700 1 -- 172.21.15.62:0/2866757373 shutdown_connections 2024-04-20T22:02:20.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:20.406+0000 7efdc5e00700 1 --2- 172.21.15.62:0/2866757373 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efdc0068540 0x7efdc00689b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:02:20.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:20.406+0000 7efdc5e00700 1 --2- 172.21.15.62:0/2866757373 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efdc01085b0 0x7efdc0108980 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:02:20.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:20.406+0000 7efdc5e00700 1 -- 172.21.15.62:0/2866757373 >> 172.21.15.62:0/2866757373 conn(0x7efdc0075130 msgr2=0x7efdc0075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:02:20.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:20.406+0000 7efdc5e00700 1 -- 172.21.15.62:0/2866757373 shutdown_connections 2024-04-20T22:02:20.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:20.406+0000 7efdc5e00700 1 -- 172.21.15.62:0/2866757373 wait complete. 2024-04-20T22:02:20.406 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:02:20 smithi062 ceph-mon[76322]: from='client.35464 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:02:20.406 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:02:20 smithi062 ceph-mon[76322]: from='client.35468 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:02:20.407 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:02:20 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/11287848' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:02:20.407 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:02:20 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:02:20.407 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:02:20 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/1406278730' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:02:20.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:20.407+0000 7efdc5e00700 1 Processor -- start 2024-04-20T22:02:20.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:20.407+0000 7efdc5e00700 1 -- start start 2024-04-20T22:02:20.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:20.408+0000 7efdc5e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efdc0068540 0x7efdc01980e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:02:20.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:20.408+0000 7efdc5e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efdc01085b0 0x7efdc0198620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:02:20.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:20.408+0000 7efdc5e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdc0198d00 con 0x7efdc0068540 2024-04-20T22:02:20.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:20.408+0000 7efdc5e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdc019ca90 con 0x7efdc01085b0 2024-04-20T22:02:20.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:20.408+0000 7efdbea00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efdc0068540 0x7efdc01980e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:02:20.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:20.408+0000 7efdbea00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efdc0068540 0x7efdc01980e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:38152/0 (socket says 172.21.15.62:38152) 2024-04-20T22:02:20.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:20.408+0000 7efdbea00700 1 -- 172.21.15.62:0/2558543794 learned_addr learned my addr 172.21.15.62:0/2558543794 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:02:20.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:20.409+0000 7efdbe000700 1 --2- 172.21.15.62:0/2558543794 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efdc01085b0 0x7efdc0198620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:02:20.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:20.409+0000 7efdbea00700 1 -- 172.21.15.62:0/2558543794 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efdc01085b0 msgr2=0x7efdc0198620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:02:20.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:20.409+0000 7efdbea00700 1 --2- 172.21.15.62:0/2558543794 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efdc01085b0 0x7efdc0198620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:02:20.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:20.409+0000 7efdbea00700 1 -- 172.21.15.62:0/2558543794 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdb00095f0 con 0x7efdc0068540 2024-04-20T22:02:20.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:20.409+0000 7efdbea00700 1 --2- 172.21.15.62:0/2558543794 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efdc0068540 0x7efdc01980e0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7efdb000bd70 tx=0x7efdb000be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:02:20.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:20.410+0000 7efdc4a00700 1 -- 172.21.15.62:0/2558543794 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efdb001d070 con 0x7efdc0068540 2024-04-20T22:02:20.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:20.410+0000 7efdc5e00700 1 -- 172.21.15.62:0/2558543794 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdc019cd10 con 0x7efdc0068540 2024-04-20T22:02:20.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:20.410+0000 7efdc4a00700 1 -- 172.21.15.62:0/2558543794 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efdb0005dc0 con 0x7efdc0068540 2024-04-20T22:02:20.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:20.410+0000 7efdc4a00700 1 -- 172.21.15.62:0/2558543794 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efdb0017400 con 0x7efdc0068540 2024-04-20T22:02:20.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:20.410+0000 7efdc5e00700 1 -- 172.21.15.62:0/2558543794 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdc019d200 con 0x7efdc0068540 2024-04-20T22:02:20.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:20.411+0000 7efdc5e00700 1 -- 172.21.15.62:0/2558543794 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdc010aa00 con 0x7efdc0068540 2024-04-20T22:02:20.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:20.413+0000 7efdc4a00700 1 -- 172.21.15.62:0/2558543794 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7efdb000f7f0 con 0x7efdc0068540 2024-04-20T22:02:20.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:20.414+0000 7efdc4a00700 1 --2- 172.21.15.62:0/2558543794 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7efda406ecc0 0x7efda4071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:02:20.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:20.414+0000 7efdc4a00700 1 -- 172.21.15.62:0/2558543794 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7efdb00905c0 con 0x7efdc0068540 2024-04-20T22:02:20.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:20.415+0000 7efdbe000700 1 --2- 172.21.15.62:0/2558543794 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7efda406ecc0 0x7efda4071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:02:20.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:20.415+0000 7efdbe000700 1 --2- 172.21.15.62:0/2558543794 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7efda406ecc0 0x7efda4071170 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7efdac005950 tx=0x7efdac0058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:02:20.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:20.419+0000 7efdc4a00700 1 -- 172.21.15.62:0/2558543794 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7efdb0059350 con 0x7efdc0068540 2024-04-20T22:02:20.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:02:20 smithi189 ceph-mon[57022]: from='client.35464 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:02:20.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:02:20 smithi189 ceph-mon[57022]: from='client.35468 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:02:20.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:02:20 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/11287848' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:02:20.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:02:20 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:02:20.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:02:20 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/1406278730' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:02:20.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:20.746+0000 7efdc5e00700 1 -- 172.21.15.62:0/2558543794 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7efdc004f2a0 con 0x7efdc0068540 2024-04-20T22:02:20.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:20.747+0000 7efdc4a00700 1 -- 172.21.15.62:0/2558543794 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7efdb005ca10 con 0x7efdc0068540 2024-04-20T22:02:20.747 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-20T22:02:20.747 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-20T22:02:20.747 INFO:teuthology.orchestra.run.smithi062.stdout: host smithi062 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T22:02:20.747 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-20T22:02:20.747 INFO:teuthology.orchestra.run.smithi062.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi062 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpq7y7oa_r:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:02:20.747 INFO:teuthology.orchestra.run.smithi062.stdout:Traceback (most recent call last): 2024-04-20T22:02:20.747 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-20T22:02:20.747 INFO:teuthology.orchestra.run.smithi062.stdout: return _run_code(code, main_globals, None, 2024-04-20T22:02:20.747 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-20T22:02:20.748 INFO:teuthology.orchestra.run.smithi062.stdout: exec(code, run_globals) 2024-04-20T22:02:20.748 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5618, in 2024-04-20T22:02:20.748 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5606, in main 2024-04-20T22:02:20.748 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 408, in _infer_config 2024-04-20T22:02:20.748 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 323, in _infer_fsid 2024-04-20T22:02:20.748 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 436, in _infer_image 2024-04-20T22:02:20.748 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 310, in _validate_fsid 2024-04-20T22:02:20.748 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 3359, in command_ceph_volume 2024-04-20T22:02:20.748 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-20T22:02:20.748 INFO:teuthology.orchestra.run.smithi062.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi062 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpq7y7oa_r:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:02:20.748 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-20T22:02:20.748 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is degraded 2024-04-20T22:02:20.748 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T22:02:20.748 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T22:02:20.748 INFO:teuthology.orchestra.run.smithi062.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-20T22:02:20.749 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-20T22:02:20.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:20.749+0000 7efdc5e00700 1 -- 172.21.15.62:0/2558543794 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7efda406ecc0 msgr2=0x7efda4071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:02:20.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:20.750+0000 7efdc5e00700 1 --2- 172.21.15.62:0/2558543794 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7efda406ecc0 0x7efda4071170 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7efdac005950 tx=0x7efdac0058e0 comp rx=0 tx=0).stop 2024-04-20T22:02:20.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:20.750+0000 7efdc5e00700 1 -- 172.21.15.62:0/2558543794 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efdc0068540 msgr2=0x7efdc01980e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:02:20.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:20.750+0000 7efdc5e00700 1 --2- 172.21.15.62:0/2558543794 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efdc0068540 0x7efdc01980e0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7efdb000bd70 tx=0x7efdb000be50 comp rx=0 tx=0).stop 2024-04-20T22:02:20.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:20.750+0000 7efdc5e00700 1 -- 172.21.15.62:0/2558543794 shutdown_connections 2024-04-20T22:02:20.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:20.750+0000 7efdc5e00700 1 --2- 172.21.15.62:0/2558543794 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7efda406ecc0 0x7efda4071170 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:02:20.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:20.750+0000 7efdc5e00700 1 --2- 172.21.15.62:0/2558543794 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efdc0068540 0x7efdc01980e0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:02:20.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:20.750+0000 7efdc5e00700 1 --2- 172.21.15.62:0/2558543794 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efdc01085b0 0x7efdc0198620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:02:20.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:20.750+0000 7efdc5e00700 1 -- 172.21.15.62:0/2558543794 >> 172.21.15.62:0/2558543794 conn(0x7efdc0075130 msgr2=0x7efdc0106430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:02:20.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:20.750+0000 7efdc5e00700 1 -- 172.21.15.62:0/2558543794 shutdown_connections 2024-04-20T22:02:20.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:20.750+0000 7efdc5e00700 1 -- 172.21.15.62:0/2558543794 wait complete. 2024-04-20T22:02:21.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:02:21 smithi189 ceph-mon[57022]: pgmap v812: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:02:21.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:02:21 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/2558543794' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:02:21.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:02:21 smithi062 ceph-mon[76322]: pgmap v812: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:02:21.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:02:21 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/2558543794' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:02:22.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:02:22 smithi189 ceph-mon[57022]: from='client.35480 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:02:22.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:02:22 smithi062 ceph-mon[76322]: from='client.35480 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:02:23.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:02:23 smithi189 ceph-mon[57022]: pgmap v813: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:02:23.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:02:23 smithi062 ceph-mon[76322]: pgmap v813: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:02:25.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:02:25 smithi189 ceph-mon[57022]: pgmap v814: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:02:25.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:02:25 smithi062 ceph-mon[76322]: pgmap v814: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:02:27.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:02:27 smithi189 ceph-mon[57022]: pgmap v815: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:02:27.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:02:27 smithi062 ceph-mon[76322]: pgmap v815: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:02:29.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:02:29 smithi189 ceph-mon[57022]: pgmap v816: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:02:29.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:02:29 smithi062 ceph-mon[76322]: pgmap v816: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:02:31.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:02:31 smithi189 ceph-mon[57022]: pgmap v817: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:02:31.531 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:02:31 smithi062 ceph-mon[76322]: pgmap v817: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:02:33.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:02:33 smithi189 ceph-mon[57022]: pgmap v818: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:02:33.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:02:33 smithi062 ceph-mon[76322]: pgmap v818: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:02:35.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:02:35 smithi189 ceph-mon[57022]: pgmap v819: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:02:35.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:02:35 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:02:35.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:02:35 smithi062 ceph-mon[76322]: pgmap v819: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:02:35.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:02:35 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:02:37.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:02:37 smithi189 ceph-mon[57022]: pgmap v820: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:02:37.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:02:37 smithi062 ceph-mon[76322]: pgmap v820: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:02:39.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:02:39 smithi189 ceph-mon[57022]: pgmap v821: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:02:39.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:02:39 smithi062 ceph-mon[76322]: pgmap v821: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:02:41.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:02:41 smithi189 ceph-mon[57022]: pgmap v822: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:02:41.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:02:41 smithi062 ceph-mon[76322]: pgmap v822: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:02:43.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:02:43 smithi189 ceph-mon[57022]: pgmap v823: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:02:43.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:02:43 smithi062 ceph-mon[76322]: pgmap v823: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:02:45.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:02:45 smithi189 ceph-mon[57022]: pgmap v824: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:02:45.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:02:45 smithi062 ceph-mon[76322]: pgmap v824: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:02:47.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:02:47 smithi189 ceph-mon[57022]: pgmap v825: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:02:47.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:02:47 smithi062 ceph-mon[76322]: pgmap v825: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:02:49.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:02:49 smithi189 ceph-mon[57022]: pgmap v826: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:02:49.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:02:49 smithi062 ceph-mon[76322]: pgmap v826: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:02:50.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:02:50 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:02:50.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:02:50 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:02:50.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:50.925+0000 7f8182c00700 1 -- 172.21.15.62:0/2970813444 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f817c100e80 msgr2=0x7f817c104e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:02:50.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:50.925+0000 7f8182c00700 1 --2- 172.21.15.62:0/2970813444 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f817c100e80 0x7f817c104e50 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f8168009910 tx=0x7f8168009c20 comp rx=0 tx=0).stop 2024-04-20T22:02:50.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:50.926+0000 7f8182c00700 1 -- 172.21.15.62:0/2970813444 shutdown_connections 2024-04-20T22:02:50.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:50.926+0000 7f8182c00700 1 --2- 172.21.15.62:0/2970813444 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f817c100e80 0x7f817c104e50 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:02:50.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:50.926+0000 7f8182c00700 1 --2- 172.21.15.62:0/2970813444 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f817c1004e0 0x7f817c1008b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:02:50.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:50.926+0000 7f8182c00700 1 -- 172.21.15.62:0/2970813444 >> 172.21.15.62:0/2970813444 conn(0x7f817c0fbda0 msgr2=0x7f817c0fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:02:50.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:50.926+0000 7f8182c00700 1 -- 172.21.15.62:0/2970813444 shutdown_connections 2024-04-20T22:02:50.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:50.926+0000 7f8182c00700 1 -- 172.21.15.62:0/2970813444 wait complete. 2024-04-20T22:02:50.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:50.927+0000 7f8182c00700 1 Processor -- start 2024-04-20T22:02:50.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:50.927+0000 7f8182c00700 1 -- start start 2024-04-20T22:02:50.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:50.927+0000 7f8182c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f817c1004e0 0x7f817c195ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:02:50.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:50.927+0000 7f8182c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f817c100e80 0x7f817c196530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:02:50.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:50.927+0000 7f8182c00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f817c196c10 con 0x7f817c1004e0 2024-04-20T22:02:50.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:50.927+0000 7f8182c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f817c19a9a0 con 0x7f817c100e80 2024-04-20T22:02:50.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:50.928+0000 7f817be00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f817c1004e0 0x7f817c195ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:02:50.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:50.928+0000 7f817be00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f817c1004e0 0x7f817c195ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:57034/0 (socket says 172.21.15.62:57034) 2024-04-20T22:02:50.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:50.928+0000 7f817be00700 1 -- 172.21.15.62:0/1497744072 learned_addr learned my addr 172.21.15.62:0/1497744072 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:02:50.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:50.928+0000 7f817b400700 1 --2- 172.21.15.62:0/1497744072 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f817c100e80 0x7f817c196530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:02:50.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:50.928+0000 7f817be00700 1 -- 172.21.15.62:0/1497744072 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f817c100e80 msgr2=0x7f817c196530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:02:50.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:50.928+0000 7f817be00700 1 --2- 172.21.15.62:0/1497744072 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f817c100e80 0x7f817c196530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:02:50.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:50.928+0000 7f817be00700 1 -- 172.21.15.62:0/1497744072 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81680095f0 con 0x7f817c1004e0 2024-04-20T22:02:50.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:50.929+0000 7f817be00700 1 --2- 172.21.15.62:0/1497744072 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f817c1004e0 0x7f817c195ff0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f816c00b4b0 tx=0x7f816c00b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:02:50.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:50.929+0000 7f8178c00700 1 -- 172.21.15.62:0/1497744072 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f816c011800 con 0x7f817c1004e0 2024-04-20T22:02:50.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:50.929+0000 7f8178c00700 1 -- 172.21.15.62:0/1497744072 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f816c011e20 con 0x7f817c1004e0 2024-04-20T22:02:50.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:50.929+0000 7f8182c00700 1 -- 172.21.15.62:0/1497744072 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f817c19ac80 con 0x7f817c1004e0 2024-04-20T22:02:50.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:50.929+0000 7f8178c00700 1 -- 172.21.15.62:0/1497744072 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f816c00f2e0 con 0x7f817c1004e0 2024-04-20T22:02:50.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:50.929+0000 7f8182c00700 1 -- 172.21.15.62:0/1497744072 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f817c19b1d0 con 0x7f817c1004e0 2024-04-20T22:02:50.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:50.930+0000 7f8182c00700 1 -- 172.21.15.62:0/1497744072 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f817c108780 con 0x7f817c1004e0 2024-04-20T22:02:50.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:50.931+0000 7f8178c00700 1 -- 172.21.15.62:0/1497744072 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f816c010380 con 0x7f817c1004e0 2024-04-20T22:02:50.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:50.932+0000 7f8178c00700 1 --2- 172.21.15.62:0/1497744072 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f816006ecc0 0x7f8160071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:02:50.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:50.932+0000 7f8178c00700 1 -- 172.21.15.62:0/1497744072 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f816c08eaa0 con 0x7f817c1004e0 2024-04-20T22:02:50.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:50.933+0000 7f817b400700 1 --2- 172.21.15.62:0/1497744072 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f816006ecc0 0x7f8160071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:02:50.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:50.933+0000 7f817b400700 1 --2- 172.21.15.62:0/1497744072 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f816006ecc0 0x7f8160071170 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f817c197610 tx=0x7f816801a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:02:50.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:50.938+0000 7f8178c00700 1 -- 172.21.15.62:0/1497744072 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f816c0577b0 con 0x7f817c1004e0 2024-04-20T22:02:51.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:51.195+0000 7f8182c00700 1 -- 172.21.15.62:0/1497744072 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f817c19b450 con 0x7f816006ecc0 2024-04-20T22:02:51.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:51.197+0000 7f8178c00700 1 -- 172.21.15.62:0/1497744072 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f817c19b450 con 0x7f816006ecc0 2024-04-20T22:02:51.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:51.200+0000 7f8182c00700 1 -- 172.21.15.62:0/1497744072 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f816006ecc0 msgr2=0x7f8160071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:02:51.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:51.200+0000 7f8182c00700 1 --2- 172.21.15.62:0/1497744072 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f816006ecc0 0x7f8160071170 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f817c197610 tx=0x7f816801a040 comp rx=0 tx=0).stop 2024-04-20T22:02:51.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:51.200+0000 7f8182c00700 1 -- 172.21.15.62:0/1497744072 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f817c1004e0 msgr2=0x7f817c195ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:02:51.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:51.200+0000 7f8182c00700 1 --2- 172.21.15.62:0/1497744072 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f817c1004e0 0x7f817c195ff0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f816c00b4b0 tx=0x7f816c00b870 comp rx=0 tx=0).stop 2024-04-20T22:02:51.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:51.200+0000 7f8182c00700 1 -- 172.21.15.62:0/1497744072 shutdown_connections 2024-04-20T22:02:51.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:51.200+0000 7f8182c00700 1 --2- 172.21.15.62:0/1497744072 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f816006ecc0 0x7f8160071170 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:02:51.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:51.200+0000 7f8182c00700 1 --2- 172.21.15.62:0/1497744072 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f817c1004e0 0x7f817c195ff0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:02:51.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:51.200+0000 7f8182c00700 1 --2- 172.21.15.62:0/1497744072 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f817c100e80 0x7f817c196530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:02:51.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:51.200+0000 7f8182c00700 1 -- 172.21.15.62:0/1497744072 >> 172.21.15.62:0/1497744072 conn(0x7f817c0fbda0 msgr2=0x7f817c0fd520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:02:51.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:51.200+0000 7f8182c00700 1 -- 172.21.15.62:0/1497744072 shutdown_connections 2024-04-20T22:02:51.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:51.200+0000 7f8182c00700 1 -- 172.21.15.62:0/1497744072 wait complete. 2024-04-20T22:02:51.217 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T22:02:51.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:51.360+0000 7fa16dc00700 1 -- 172.21.15.62:0/4088888817 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa168104130 msgr2=0x7fa168104500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:02:51.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:51.360+0000 7fa16dc00700 1 --2- 172.21.15.62:0/4088888817 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa168104130 0x7fa168104500 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fa158009910 tx=0x7fa158009c20 comp rx=0 tx=0).stop 2024-04-20T22:02:51.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:51.360+0000 7fa16dc00700 1 -- 172.21.15.62:0/4088888817 shutdown_connections 2024-04-20T22:02:51.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:51.360+0000 7fa16dc00700 1 --2- 172.21.15.62:0/4088888817 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1680ffbd0 0x7fa168100040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:02:51.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:51.360+0000 7fa16dc00700 1 --2- 172.21.15.62:0/4088888817 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa168104130 0x7fa168104500 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:02:51.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:51.360+0000 7fa16dc00700 1 -- 172.21.15.62:0/4088888817 >> 172.21.15.62:0/4088888817 conn(0x7fa168075130 msgr2=0x7fa168075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:02:51.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:51.360+0000 7fa16dc00700 1 -- 172.21.15.62:0/4088888817 shutdown_connections 2024-04-20T22:02:51.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:51.361+0000 7fa16dc00700 1 -- 172.21.15.62:0/4088888817 wait complete. 2024-04-20T22:02:51.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:51.362+0000 7fa16dc00700 1 Processor -- start 2024-04-20T22:02:51.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:51.362+0000 7fa16dc00700 1 -- start start 2024-04-20T22:02:51.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:51.362+0000 7fa16dc00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1680ffbd0 0x7fa168198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:02:51.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:51.362+0000 7fa16dc00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa168104130 0x7fa168198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:02:51.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:51.362+0000 7fa16dc00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa168198c50 con 0x7fa1680ffbd0 2024-04-20T22:02:51.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:51.362+0000 7fa16dc00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa16819c9e0 con 0x7fa168104130 2024-04-20T22:02:51.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:51.363+0000 7fa166a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1680ffbd0 0x7fa168198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:02:51.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:51.363+0000 7fa166a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1680ffbd0 0x7fa168198030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40816/0 (socket says 172.21.15.62:40816) 2024-04-20T22:02:51.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:51.363+0000 7fa166a00700 1 -- 172.21.15.62:0/1454479320 learned_addr learned my addr 172.21.15.62:0/1454479320 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:02:51.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:51.363+0000 7fa166a00700 1 -- 172.21.15.62:0/1454479320 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa168104130 msgr2=0x7fa168198570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:02:51.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:51.363+0000 7fa166a00700 1 --2- 172.21.15.62:0/1454479320 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa168104130 0x7fa168198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:02:51.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:51.363+0000 7fa166a00700 1 -- 172.21.15.62:0/1454479320 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1580095f0 con 0x7fa1680ffbd0 2024-04-20T22:02:51.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:51.363+0000 7fa166a00700 1 --2- 172.21.15.62:0/1454479320 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1680ffbd0 0x7fa168198030 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fa158006010 tx=0x7fa15800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:02:51.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:51.364+0000 7fa15f400700 1 -- 172.21.15.62:0/1454479320 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa15801d070 con 0x7fa1680ffbd0 2024-04-20T22:02:51.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:51.364+0000 7fa15f400700 1 -- 172.21.15.62:0/1454479320 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa158004fb0 con 0x7fa1680ffbd0 2024-04-20T22:02:51.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:51.364+0000 7fa15f400700 1 -- 172.21.15.62:0/1454479320 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa15800f480 con 0x7fa1680ffbd0 2024-04-20T22:02:51.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:51.364+0000 7fa16dc00700 1 -- 172.21.15.62:0/1454479320 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa16819cc60 con 0x7fa1680ffbd0 2024-04-20T22:02:51.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:51.364+0000 7fa16dc00700 1 -- 172.21.15.62:0/1454479320 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa16819d150 con 0x7fa1680ffbd0 2024-04-20T22:02:51.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:51.365+0000 7fa16dc00700 1 -- 172.21.15.62:0/1454479320 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa168103510 con 0x7fa1680ffbd0 2024-04-20T22:02:51.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:51.366+0000 7fa15f400700 1 -- 172.21.15.62:0/1454479320 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fa158022890 con 0x7fa1680ffbd0 2024-04-20T22:02:51.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:51.367+0000 7fa15f400700 1 --2- 172.21.15.62:0/1454479320 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa14c06ecc0 0x7fa14c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:02:51.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:51.367+0000 7fa15f400700 1 -- 172.21.15.62:0/1454479320 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fa1580904f0 con 0x7fa1680ffbd0 2024-04-20T22:02:51.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:51.367+0000 7fa166000700 1 --2- 172.21.15.62:0/1454479320 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa14c06ecc0 0x7fa14c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:02:51.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:51.368+0000 7fa166000700 1 --2- 172.21.15.62:0/1454479320 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa14c06ecc0 0x7fa14c071170 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fa168199650 tx=0x7fa154005c80 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:02:51.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:51.372+0000 7fa15f400700 1 -- 172.21.15.62:0/1454479320 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fa158059330 con 0x7fa1680ffbd0 2024-04-20T22:02:51.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:02:51 smithi189 ceph-mon[57022]: pgmap v827: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:02:51.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:02:51 smithi062 ceph-mon[76322]: pgmap v827: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:02:51.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:51.635+0000 7fa16dc00700 1 -- 172.21.15.62:0/1454479320 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa16819d480 con 0x7fa14c06ecc0 2024-04-20T22:02:51.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:51.637+0000 7fa15f400700 1 -- 172.21.15.62:0/1454479320 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fa16819d480 con 0x7fa14c06ecc0 2024-04-20T22:02:51.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:51.639+0000 7fa16dc00700 1 -- 172.21.15.62:0/1454479320 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa14c06ecc0 msgr2=0x7fa14c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:02:51.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:51.639+0000 7fa16dc00700 1 --2- 172.21.15.62:0/1454479320 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa14c06ecc0 0x7fa14c071170 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fa168199650 tx=0x7fa154005c80 comp rx=0 tx=0).stop 2024-04-20T22:02:51.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:51.639+0000 7fa16dc00700 1 -- 172.21.15.62:0/1454479320 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1680ffbd0 msgr2=0x7fa168198030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:02:51.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:51.639+0000 7fa16dc00700 1 --2- 172.21.15.62:0/1454479320 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1680ffbd0 0x7fa168198030 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fa158006010 tx=0x7fa15800be20 comp rx=0 tx=0).stop 2024-04-20T22:02:51.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:51.639+0000 7fa16dc00700 1 -- 172.21.15.62:0/1454479320 shutdown_connections 2024-04-20T22:02:51.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:51.639+0000 7fa16dc00700 1 --2- 172.21.15.62:0/1454479320 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa14c06ecc0 0x7fa14c071170 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:02:51.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:51.639+0000 7fa16dc00700 1 --2- 172.21.15.62:0/1454479320 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1680ffbd0 0x7fa168198030 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:02:51.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:51.639+0000 7fa16dc00700 1 --2- 172.21.15.62:0/1454479320 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa168104130 0x7fa168198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:02:51.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:51.639+0000 7fa16dc00700 1 -- 172.21.15.62:0/1454479320 >> 172.21.15.62:0/1454479320 conn(0x7fa168075130 msgr2=0x7fa1681097c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:02:51.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:51.640+0000 7fa16dc00700 1 -- 172.21.15.62:0/1454479320 shutdown_connections 2024-04-20T22:02:51.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:51.640+0000 7fa16dc00700 1 -- 172.21.15.62:0/1454479320 wait complete. 2024-04-20T22:02:51.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:51.804+0000 7fd8bb400700 1 -- 172.21.15.62:0/1751969862 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd8b4100e80 msgr2=0x7fd8b4104e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:02:51.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:51.804+0000 7fd8bb400700 1 --2- 172.21.15.62:0/1751969862 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd8b4100e80 0x7fd8b4104e50 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fd8a0009910 tx=0x7fd8a0009c20 comp rx=0 tx=0).stop 2024-04-20T22:02:51.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:51.804+0000 7fd8bb400700 1 -- 172.21.15.62:0/1751969862 shutdown_connections 2024-04-20T22:02:51.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:51.804+0000 7fd8bb400700 1 --2- 172.21.15.62:0/1751969862 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd8b4100e80 0x7fd8b4104e50 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:02:51.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:51.804+0000 7fd8bb400700 1 --2- 172.21.15.62:0/1751969862 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8b41004e0 0x7fd8b41008b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:02:51.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:51.804+0000 7fd8bb400700 1 -- 172.21.15.62:0/1751969862 >> 172.21.15.62:0/1751969862 conn(0x7fd8b40fbda0 msgr2=0x7fd8b40fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:02:51.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:51.805+0000 7fd8bb400700 1 -- 172.21.15.62:0/1751969862 shutdown_connections 2024-04-20T22:02:51.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:51.805+0000 7fd8bb400700 1 -- 172.21.15.62:0/1751969862 wait complete. 2024-04-20T22:02:51.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:51.806+0000 7fd8bb400700 1 Processor -- start 2024-04-20T22:02:51.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:51.806+0000 7fd8bb400700 1 -- start start 2024-04-20T22:02:51.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:51.806+0000 7fd8bb400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd8b41004e0 0x7fd8b4195f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:02:51.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:51.806+0000 7fd8bb400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8b4100e80 0x7fd8b4196450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:02:51.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:51.806+0000 7fd8bb400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b4196b30 con 0x7fd8b41004e0 2024-04-20T22:02:51.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:51.806+0000 7fd8bb400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b419a8c0 con 0x7fd8b4100e80 2024-04-20T22:02:51.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:51.807+0000 7fd8b8a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd8b41004e0 0x7fd8b4195f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:02:51.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:51.807+0000 7fd8b8a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd8b41004e0 0x7fd8b4195f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40826/0 (socket says 172.21.15.62:40826) 2024-04-20T22:02:51.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:51.807+0000 7fd8b8a00700 1 -- 172.21.15.62:0/637571963 learned_addr learned my addr 172.21.15.62:0/637571963 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:02:51.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:51.807+0000 7fd8b3e00700 1 --2- 172.21.15.62:0/637571963 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8b4100e80 0x7fd8b4196450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:02:51.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:51.807+0000 7fd8b8a00700 1 -- 172.21.15.62:0/637571963 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8b4100e80 msgr2=0x7fd8b4196450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:02:51.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:51.807+0000 7fd8b8a00700 1 --2- 172.21.15.62:0/637571963 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8b4100e80 0x7fd8b4196450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:02:51.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:51.807+0000 7fd8b8a00700 1 -- 172.21.15.62:0/637571963 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8a00095f0 con 0x7fd8b41004e0 2024-04-20T22:02:51.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:51.808+0000 7fd8b8a00700 1 --2- 172.21.15.62:0/637571963 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd8b41004e0 0x7fd8b4195f10 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fd8a800c8a0 tx=0x7fd8a800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:02:51.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:51.808+0000 7fd8b1600700 1 -- 172.21.15.62:0/637571963 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd8a800cea0 con 0x7fd8b41004e0 2024-04-20T22:02:51.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:51.808+0000 7fd8b1600700 1 -- 172.21.15.62:0/637571963 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd8a8004830 con 0x7fd8b41004e0 2024-04-20T22:02:51.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:51.808+0000 7fd8bb400700 1 -- 172.21.15.62:0/637571963 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8b419aba0 con 0x7fd8b41004e0 2024-04-20T22:02:51.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:51.808+0000 7fd8b1600700 1 -- 172.21.15.62:0/637571963 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd8a80055b0 con 0x7fd8b41004e0 2024-04-20T22:02:51.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:51.808+0000 7fd8bb400700 1 -- 172.21.15.62:0/637571963 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8b419b0f0 con 0x7fd8b41004e0 2024-04-20T22:02:51.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:51.809+0000 7fd8bb400700 1 -- 172.21.15.62:0/637571963 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8b4108850 con 0x7fd8b41004e0 2024-04-20T22:02:51.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:51.810+0000 7fd8b1600700 1 -- 172.21.15.62:0/637571963 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fd8a80049a0 con 0x7fd8b41004e0 2024-04-20T22:02:51.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:51.811+0000 7fd8b1600700 1 --2- 172.21.15.62:0/637571963 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd89c06ecc0 0x7fd89c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:02:51.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:51.811+0000 7fd8b1600700 1 -- 172.21.15.62:0/637571963 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fd8a808eaf0 con 0x7fd8b41004e0 2024-04-20T22:02:51.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:51.812+0000 7fd8b3e00700 1 --2- 172.21.15.62:0/637571963 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd89c06ecc0 0x7fd89c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:02:51.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:51.812+0000 7fd8b3e00700 1 --2- 172.21.15.62:0/637571963 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd89c06ecc0 0x7fd89c071170 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fd8a0005950 tx=0x7fd8a00058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:02:51.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:51.816+0000 7fd8b1600700 1 -- 172.21.15.62:0/637571963 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fd8a8057880 con 0x7fd8b41004e0 2024-04-20T22:02:52.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:52.062+0000 7fd8bb400700 1 -- 172.21.15.62:0/637571963 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fd8b419b3c0 con 0x7fd89c06ecc0 2024-04-20T22:02:52.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:52.096+0000 7fd8b1600700 1 -- 172.21.15.62:0/637571963 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fd8b419b3c0 con 0x7fd89c06ecc0 2024-04-20T22:02:52.095 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T22:02:52.095 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (28m) 4m ago 42m 15.3M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T22:02:52.095 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (42m) 4m ago 42m 10.4M - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T22:02:52.095 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (41m) 4m ago 41m 10.1M - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T22:02:52.095 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (24m) 4m ago 42m 6660k - 19.0.0-3133-ga8c84f59 76bc6b91b726 2825d5ba11c6 2024-04-20T22:02:52.095 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (24m) 4m ago 41m 6665k - 19.0.0-3133-ga8c84f59 76bc6b91b726 20f746fa1bba 2024-04-20T22:02:52.095 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (28m) 4m ago 42m 73.5M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T22:02:52.095 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (20m) 4m ago 35m 18.5M - 19.0.0-3133-ga8c84f59 76bc6b91b726 de80bc90a683 2024-04-20T22:02:52.095 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (20m) 4m ago 35m 18.7M - 19.0.0-3133-ga8c84f59 76bc6b91b726 cd793b7ee98e 2024-04-20T22:02:52.096 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (4m) 4m ago 35m 11.0M - 19.0.0-3133-ga8c84f59 76bc6b91b726 0e7b0bc4d1ab 2024-04-20T22:02:52.096 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (35m) 4m ago 35m 18.6M - 18.2.0 dc2bc1663786 f2eec2ee8951 2024-04-20T22:02:52.096 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (31m) 4m ago 44m 513M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T22:02:52.096 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (30m) 4m ago 41m 459M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T22:02:52.096 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (26m) 4m ago 44m 71.7M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 019eb1479485 2024-04-20T22:02:52.096 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (25m) 4m ago 41m 51.0M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 55aac560aab6 2024-04-20T22:02:52.096 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (29m) 4m ago 42m 8472k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T22:02:52.096 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (29m) 4m ago 41m 8048k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T22:02:52.096 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (24m) 4m ago 40m 91.2M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 ee2bb901f26e 2024-04-20T22:02:52.096 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (23m) 4m ago 39m 94.6M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 9075152d4162 2024-04-20T22:02:52.096 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (23m) 4m ago 38m 78.1M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 2d4f3e38dc0e 2024-04-20T22:02:52.096 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (22m) 4m ago 38m 92.6M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 8f97b17ae0eb 2024-04-20T22:02:52.096 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (21m) 4m ago 37m 95.6M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 653d8079967d 2024-04-20T22:02:52.096 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (21m) 4m ago 37m 89.2M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 596dded14c12 2024-04-20T22:02:52.096 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (29m) 4m ago 41m 82.1M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T22:02:52.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:52.097+0000 7fd8bb400700 1 -- 172.21.15.62:0/637571963 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd89c06ecc0 msgr2=0x7fd89c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:02:52.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:52.097+0000 7fd8bb400700 1 --2- 172.21.15.62:0/637571963 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd89c06ecc0 0x7fd89c071170 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fd8a0005950 tx=0x7fd8a00058e0 comp rx=0 tx=0).stop 2024-04-20T22:02:52.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:52.098+0000 7fd8bb400700 1 -- 172.21.15.62:0/637571963 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd8b41004e0 msgr2=0x7fd8b4195f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:02:52.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:52.098+0000 7fd8bb400700 1 --2- 172.21.15.62:0/637571963 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd8b41004e0 0x7fd8b4195f10 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fd8a800c8a0 tx=0x7fd8a800cc60 comp rx=0 tx=0).stop 2024-04-20T22:02:52.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:52.098+0000 7fd8bb400700 1 -- 172.21.15.62:0/637571963 shutdown_connections 2024-04-20T22:02:52.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:52.098+0000 7fd8bb400700 1 --2- 172.21.15.62:0/637571963 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd89c06ecc0 0x7fd89c071170 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:02:52.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:52.098+0000 7fd8bb400700 1 --2- 172.21.15.62:0/637571963 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd8b41004e0 0x7fd8b4195f10 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:02:52.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:52.098+0000 7fd8bb400700 1 --2- 172.21.15.62:0/637571963 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8b4100e80 0x7fd8b4196450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:02:52.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:52.098+0000 7fd8bb400700 1 -- 172.21.15.62:0/637571963 >> 172.21.15.62:0/637571963 conn(0x7fd8b40fbda0 msgr2=0x7fd8b40fe180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:02:52.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:52.098+0000 7fd8bb400700 1 -- 172.21.15.62:0/637571963 shutdown_connections 2024-04-20T22:02:52.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:52.098+0000 7fd8bb400700 1 -- 172.21.15.62:0/637571963 wait complete. 2024-04-20T22:02:52.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:52.253+0000 7fab03800700 1 -- 172.21.15.62:0/686329374 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faafc073090 msgr2=0x7faafc073460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:02:52.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:52.253+0000 7fab03800700 1 --2- 172.21.15.62:0/686329374 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faafc073090 0x7faafc073460 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7faaf0009940 tx=0x7faaf0009c50 comp rx=0 tx=0).stop 2024-04-20T22:02:52.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:52.253+0000 7fab03800700 1 -- 172.21.15.62:0/686329374 shutdown_connections 2024-04-20T22:02:52.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:52.253+0000 7fab03800700 1 --2- 172.21.15.62:0/686329374 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faafc0739a0 0x7faafc10c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:02:52.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:52.253+0000 7fab03800700 1 --2- 172.21.15.62:0/686329374 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faafc073090 0x7faafc073460 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:02:52.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:52.253+0000 7fab03800700 1 -- 172.21.15.62:0/686329374 >> 172.21.15.62:0/686329374 conn(0x7faafc0fbda0 msgr2=0x7faafc0fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:02:52.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:52.253+0000 7fab03800700 1 -- 172.21.15.62:0/686329374 shutdown_connections 2024-04-20T22:02:52.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:52.253+0000 7fab03800700 1 -- 172.21.15.62:0/686329374 wait complete. 2024-04-20T22:02:52.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:52.254+0000 7fab03800700 1 Processor -- start 2024-04-20T22:02:52.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:52.254+0000 7fab03800700 1 -- start start 2024-04-20T22:02:52.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:52.255+0000 7fab03800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faafc073090 0x7faafc1981c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:02:52.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:52.255+0000 7fab03800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faafc0739a0 0x7faafc198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:02:52.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:52.255+0000 7fab03800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faafc198de0 con 0x7faafc0739a0 2024-04-20T22:02:52.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:52.255+0000 7fab03800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faafc19cb70 con 0x7faafc073090 2024-04-20T22:02:52.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:52.255+0000 7faafbe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faafc0739a0 0x7faafc198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:02:52.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:52.255+0000 7faafbe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faafc0739a0 0x7faafc198700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40836/0 (socket says 172.21.15.62:40836) 2024-04-20T22:02:52.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:52.255+0000 7faafbe00700 1 -- 172.21.15.62:0/1461070285 learned_addr learned my addr 172.21.15.62:0/1461070285 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:02:52.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:52.255+0000 7fab00e00700 1 --2- 172.21.15.62:0/1461070285 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faafc073090 0x7faafc1981c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:02:52.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:52.255+0000 7faafbe00700 1 -- 172.21.15.62:0/1461070285 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faafc073090 msgr2=0x7faafc1981c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:02:52.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:52.255+0000 7faafbe00700 1 --2- 172.21.15.62:0/1461070285 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faafc073090 0x7faafc1981c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:02:52.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:52.255+0000 7faafbe00700 1 -- 172.21.15.62:0/1461070285 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaf00095f0 con 0x7faafc0739a0 2024-04-20T22:02:52.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:52.256+0000 7fab00e00700 1 --2- 172.21.15.62:0/1461070285 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faafc073090 0x7faafc1981c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T22:02:52.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:52.256+0000 7faafbe00700 1 --2- 172.21.15.62:0/1461070285 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faafc0739a0 0x7faafc198700 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7faae800b4b0 tx=0x7faae800b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:02:52.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:52.256+0000 7faaf9600700 1 -- 172.21.15.62:0/1461070285 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faae8011800 con 0x7faafc0739a0 2024-04-20T22:02:52.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:52.256+0000 7faaf9600700 1 -- 172.21.15.62:0/1461070285 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7faae8011e20 con 0x7faafc0739a0 2024-04-20T22:02:52.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:52.256+0000 7faaf9600700 1 -- 172.21.15.62:0/1461070285 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faae800f2e0 con 0x7faafc0739a0 2024-04-20T22:02:52.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:52.256+0000 7fab03800700 1 -- 172.21.15.62:0/1461070285 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faafc19ce50 con 0x7faafc0739a0 2024-04-20T22:02:52.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:52.256+0000 7fab03800700 1 -- 172.21.15.62:0/1461070285 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faafc19d3a0 con 0x7faafc0739a0 2024-04-20T22:02:52.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:52.258+0000 7fab03800700 1 -- 172.21.15.62:0/1461070285 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faafc109ce0 con 0x7faafc0739a0 2024-04-20T22:02:52.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:52.259+0000 7faaf9600700 1 -- 172.21.15.62:0/1461070285 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7faae8010380 con 0x7faafc0739a0 2024-04-20T22:02:52.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:52.260+0000 7faaf9600700 1 --2- 172.21.15.62:0/1461070285 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7faae406ecc0 0x7faae4071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:02:52.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:52.260+0000 7faaf9600700 1 -- 172.21.15.62:0/1461070285 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7faae808ea30 con 0x7faafc0739a0 2024-04-20T22:02:52.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:52.260+0000 7fab00e00700 1 --2- 172.21.15.62:0/1461070285 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7faae406ecc0 0x7faae4071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:02:52.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:52.261+0000 7fab00e00700 1 --2- 172.21.15.62:0/1461070285 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7faae406ecc0 0x7faae4071170 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7faaf0000c00 tx=0x7faaf0005fb0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:02:52.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:52.264+0000 7faaf9600700 1 -- 172.21.15.62:0/1461070285 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7faae8057870 con 0x7faafc0739a0 2024-04-20T22:02:52.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:02:52 smithi189 ceph-mon[57022]: from='client.35488 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:02:52.537 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:02:52 smithi062 ceph-mon[76322]: from='client.35488 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:02:52.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:52.586+0000 7fab03800700 1 -- 172.21.15.62:0/1461070285 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7faafc04f2a0 con 0x7faafc0739a0 2024-04-20T22:02:52.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:52.587+0000 7faaf9600700 1 -- 172.21.15.62:0/1461070285 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7faae805af30 con 0x7faafc0739a0 2024-04-20T22:02:52.587 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:02:52.587 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T22:02:52.587 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:02:52.587 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:02:52.587 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T22:02:52.587 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:02:52.587 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:02:52.587 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T22:02:52.587 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 6 2024-04-20T22:02:52.587 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:02:52.587 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T22:02:52.587 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 1, 2024-04-20T22:02:52.587 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 3 2024-04-20T22:02:52.587 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:02:52.587 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T22:02:52.587 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 1, 2024-04-20T22:02:52.588 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 13 2024-04-20T22:02:52.588 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T22:02:52.588 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:02:52.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:52.591+0000 7fab03800700 1 -- 172.21.15.62:0/1461070285 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7faae406ecc0 msgr2=0x7faae4071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:02:52.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:52.591+0000 7fab03800700 1 --2- 172.21.15.62:0/1461070285 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7faae406ecc0 0x7faae4071170 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7faaf0000c00 tx=0x7faaf0005fb0 comp rx=0 tx=0).stop 2024-04-20T22:02:52.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:52.591+0000 7fab03800700 1 -- 172.21.15.62:0/1461070285 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faafc0739a0 msgr2=0x7faafc198700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:02:52.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:52.591+0000 7fab03800700 1 --2- 172.21.15.62:0/1461070285 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faafc0739a0 0x7faafc198700 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7faae800b4b0 tx=0x7faae800b870 comp rx=0 tx=0).stop 2024-04-20T22:02:52.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:52.591+0000 7fab03800700 1 -- 172.21.15.62:0/1461070285 shutdown_connections 2024-04-20T22:02:52.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:52.591+0000 7fab03800700 1 --2- 172.21.15.62:0/1461070285 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7faae406ecc0 0x7faae4071170 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:02:52.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:52.591+0000 7fab03800700 1 --2- 172.21.15.62:0/1461070285 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faafc073090 0x7faafc1981c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:02:52.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:52.591+0000 7fab03800700 1 --2- 172.21.15.62:0/1461070285 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faafc0739a0 0x7faafc198700 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:02:52.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:52.591+0000 7fab03800700 1 -- 172.21.15.62:0/1461070285 >> 172.21.15.62:0/1461070285 conn(0x7faafc0fbda0 msgr2=0x7faafc106e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:02:52.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:52.591+0000 7fab03800700 1 -- 172.21.15.62:0/1461070285 shutdown_connections 2024-04-20T22:02:52.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:52.591+0000 7fab03800700 1 -- 172.21.15.62:0/1461070285 wait complete. 2024-04-20T22:02:52.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:52.746+0000 7fc935e00700 1 -- 172.21.15.62:0/1320887753 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc930102470 msgr2=0x7fc9301028e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:02:52.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:52.746+0000 7fc935e00700 1 --2- 172.21.15.62:0/1320887753 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc930102470 0x7fc9301028e0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fc91c009910 tx=0x7fc91c009c20 comp rx=0 tx=0).stop 2024-04-20T22:02:52.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:52.746+0000 7fc935e00700 1 -- 172.21.15.62:0/1320887753 shutdown_connections 2024-04-20T22:02:52.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:52.746+0000 7fc935e00700 1 --2- 172.21.15.62:0/1320887753 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc930102470 0x7fc9301028e0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:02:52.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:52.746+0000 7fc935e00700 1 --2- 172.21.15.62:0/1320887753 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc930108470 0x7fc930108840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:02:52.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:52.746+0000 7fc935e00700 1 -- 172.21.15.62:0/1320887753 >> 172.21.15.62:0/1320887753 conn(0x7fc9300fdf90 msgr2=0x7fc9301003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:02:52.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:52.746+0000 7fc935e00700 1 -- 172.21.15.62:0/1320887753 shutdown_connections 2024-04-20T22:02:52.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:52.746+0000 7fc935e00700 1 -- 172.21.15.62:0/1320887753 wait complete. 2024-04-20T22:02:52.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:52.747+0000 7fc935e00700 1 Processor -- start 2024-04-20T22:02:52.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:52.748+0000 7fc935e00700 1 -- start start 2024-04-20T22:02:52.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:52.748+0000 7fc935e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc930102470 0x7fc930198200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:02:52.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:52.748+0000 7fc935e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc930108470 0x7fc930198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:02:52.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:52.748+0000 7fc935e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc930198e20 con 0x7fc930102470 2024-04-20T22:02:52.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:52.748+0000 7fc935e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc93019cb60 con 0x7fc930108470 2024-04-20T22:02:52.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:52.748+0000 7fc92ea00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc930102470 0x7fc930198200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:02:52.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:52.749+0000 7fc92ea00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc930102470 0x7fc930198200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40864/0 (socket says 172.21.15.62:40864) 2024-04-20T22:02:52.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:52.749+0000 7fc92ea00700 1 -- 172.21.15.62:0/3030124894 learned_addr learned my addr 172.21.15.62:0/3030124894 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:02:52.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:52.749+0000 7fc92e000700 1 --2- 172.21.15.62:0/3030124894 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc930108470 0x7fc930198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:02:52.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:52.749+0000 7fc92ea00700 1 -- 172.21.15.62:0/3030124894 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc930108470 msgr2=0x7fc930198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:02:52.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:52.749+0000 7fc92ea00700 1 --2- 172.21.15.62:0/3030124894 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc930108470 0x7fc930198740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:02:52.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:52.749+0000 7fc92ea00700 1 -- 172.21.15.62:0/3030124894 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc91c0095f0 con 0x7fc930102470 2024-04-20T22:02:52.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:52.749+0000 7fc92ea00700 1 --2- 172.21.15.62:0/3030124894 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc930102470 0x7fc930198200 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fc92000b4b0 tx=0x7fc92000b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:02:52.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:52.750+0000 7fc934a00700 1 -- 172.21.15.62:0/3030124894 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc920011800 con 0x7fc930102470 2024-04-20T22:02:52.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:52.750+0000 7fc934a00700 1 -- 172.21.15.62:0/3030124894 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc920011e20 con 0x7fc930102470 2024-04-20T22:02:52.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:52.750+0000 7fc935e00700 1 -- 172.21.15.62:0/3030124894 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc93019ce40 con 0x7fc930102470 2024-04-20T22:02:52.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:52.750+0000 7fc934a00700 1 -- 172.21.15.62:0/3030124894 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc92000f2e0 con 0x7fc930102470 2024-04-20T22:02:52.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:52.750+0000 7fc935e00700 1 -- 172.21.15.62:0/3030124894 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc93019d390 con 0x7fc930102470 2024-04-20T22:02:52.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:52.751+0000 7fc935e00700 1 -- 172.21.15.62:0/3030124894 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc93010aa00 con 0x7fc930102470 2024-04-20T22:02:52.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:52.752+0000 7fc934a00700 1 -- 172.21.15.62:0/3030124894 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fc920010380 con 0x7fc930102470 2024-04-20T22:02:52.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:52.753+0000 7fc934a00700 1 --2- 172.21.15.62:0/3030124894 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc91406ecc0 0x7fc914071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:02:52.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:52.754+0000 7fc934a00700 1 -- 172.21.15.62:0/3030124894 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fc92008ead0 con 0x7fc930102470 2024-04-20T22:02:52.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:52.754+0000 7fc92e000700 1 --2- 172.21.15.62:0/3030124894 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc91406ecc0 0x7fc914071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:02:52.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:52.755+0000 7fc92e000700 1 --2- 172.21.15.62:0/3030124894 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc91406ecc0 0x7fc914071170 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fc9301997d0 tx=0x7fc91c01a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:02:52.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:52.759+0000 7fc934a00700 1 -- 172.21.15.62:0/3030124894 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fc9200577e0 con 0x7fc930102470 2024-04-20T22:02:53.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.043+0000 7fc935e00700 1 -- 172.21.15.62:0/3030124894 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fc93004f2a0 con 0x7fc930102470 2024-04-20T22:02:53.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.044+0000 7fc934a00700 1 -- 172.21.15.62:0/3030124894 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 27 v27) v1 ==== 76+0+1888 (secure 0 0 0) 0x7fc92005aea0 con 0x7fc930102470 2024-04-20T22:02:53.043 INFO:teuthology.orchestra.run.smithi062.stdout:e27 2024-04-20T22:02:53.043 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T22:02:53.043 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:02:53.043 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T22:02:53.043 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:02:53.044 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T22:02:53.044 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T22:02:53.044 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 20 2024-04-20T22:02:53.044 INFO:teuthology.orchestra.run.smithi062.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-20T22:02:53.044 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T22:02:53.044 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T21:42:01.559061+0000 2024-04-20T22:02:53.044 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T22:02:53.044 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T22:02:53.044 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T22:02:53.044 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T22:02:53.044 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T22:02:53.044 INFO:teuthology.orchestra.run.smithi062.stdout:max_xattr_size 65536 2024-04-20T22:02:53.044 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T22:02:53.044 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T22:02:53.044 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 81 2024-04-20T22:02:53.044 INFO:teuthology.orchestra.run.smithi062.stdout:snap_rstat 1 2024-04-20T22:02:53.044 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:02:53.045 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 2 2024-04-20T22:02:53.045 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1 2024-04-20T22:02:53.045 INFO:teuthology.orchestra.run.smithi062.stdout:up {} 2024-04-20T22:02:53.045 INFO:teuthology.orchestra.run.smithi062.stdout:failed 0,1 2024-04-20T22:02:53.045 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T22:02:53.045 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T22:02:53.045 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T22:02:53.045 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T22:02:53.045 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data enabled 2024-04-20T22:02:53.045 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T22:02:53.045 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T22:02:53.045 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 1 2024-04-20T22:02:53.045 INFO:teuthology.orchestra.run.smithi062.stdout:qdb_cluster leader: 0 members: 2024-04-20T22:02:53.045 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:02:53.045 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:02:53.045 INFO:teuthology.orchestra.run.smithi062.stdout:Standby daemons: 2024-04-20T22:02:53.045 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:02:53.046 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ryqblb{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6826/4113916205,v1:172.21.15.62:6827/4113916205] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:02:53.046 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ozizwi{-1:34476} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6828/3801975165,v1:172.21.15.62:6829/3801975165] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:02:53.046 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.xbgspw{-1:44285} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6826/3996040526,v1:172.21.15.189:6827/3996040526] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T22:02:53.046 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.dvimer{-1:44727} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6824/3576699232,v1:172.21.15.189:6825/3576699232] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:02:53.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.048+0000 7fc935e00700 1 -- 172.21.15.62:0/3030124894 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc91406ecc0 msgr2=0x7fc914071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:02:53.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.048+0000 7fc935e00700 1 --2- 172.21.15.62:0/3030124894 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc91406ecc0 0x7fc914071170 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fc9301997d0 tx=0x7fc91c01a040 comp rx=0 tx=0).stop 2024-04-20T22:02:53.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.048+0000 7fc935e00700 1 -- 172.21.15.62:0/3030124894 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc930102470 msgr2=0x7fc930198200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:02:53.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.048+0000 7fc935e00700 1 --2- 172.21.15.62:0/3030124894 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc930102470 0x7fc930198200 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fc92000b4b0 tx=0x7fc92000b870 comp rx=0 tx=0).stop 2024-04-20T22:02:53.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.048+0000 7fc935e00700 1 -- 172.21.15.62:0/3030124894 shutdown_connections 2024-04-20T22:02:53.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.048+0000 7fc935e00700 1 --2- 172.21.15.62:0/3030124894 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc91406ecc0 0x7fc914071170 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:02:53.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.048+0000 7fc935e00700 1 --2- 172.21.15.62:0/3030124894 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc930102470 0x7fc930198200 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:02:53.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.048+0000 7fc935e00700 1 --2- 172.21.15.62:0/3030124894 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc930108470 0x7fc930198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:02:53.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.048+0000 7fc935e00700 1 -- 172.21.15.62:0/3030124894 >> 172.21.15.62:0/3030124894 conn(0x7fc9300fdf90 msgr2=0x7fc9300febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:02:53.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.048+0000 7fc935e00700 1 -- 172.21.15.62:0/3030124894 shutdown_connections 2024-04-20T22:02:53.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.048+0000 7fc935e00700 1 -- 172.21.15.62:0/3030124894 wait complete. 2024-04-20T22:02:53.048 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 27 2024-04-20T22:02:53.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.211+0000 7fe75f200700 1 -- 172.21.15.62:0/3304140953 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe7600730f0 msgr2=0x7fe7600734c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:02:53.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.211+0000 7fe75f200700 1 --2- 172.21.15.62:0/3304140953 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe7600730f0 0x7fe7600734c0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fe750009910 tx=0x7fe750009c20 comp rx=0 tx=0).stop 2024-04-20T22:02:53.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.212+0000 7fe75f200700 1 -- 172.21.15.62:0/3304140953 shutdown_connections 2024-04-20T22:02:53.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.212+0000 7fe75f200700 1 --2- 172.21.15.62:0/3304140953 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe760073a00 0x7fe76010c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:02:53.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.212+0000 7fe75f200700 1 --2- 172.21.15.62:0/3304140953 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe7600730f0 0x7fe7600734c0 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:02:53.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.212+0000 7fe75f200700 1 -- 172.21.15.62:0/3304140953 >> 172.21.15.62:0/3304140953 conn(0x7fe7600fbb10 msgr2=0x7fe7600fdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:02:53.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.212+0000 7fe75f200700 1 -- 172.21.15.62:0/3304140953 shutdown_connections 2024-04-20T22:02:53.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.212+0000 7fe75f200700 1 -- 172.21.15.62:0/3304140953 wait complete. 2024-04-20T22:02:53.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.213+0000 7fe75f200700 1 Processor -- start 2024-04-20T22:02:53.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.213+0000 7fe75f200700 1 -- start start 2024-04-20T22:02:53.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.213+0000 7fe75f200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe7600730f0 0x7fe76019dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:02:53.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.213+0000 7fe75f200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe760073a00 0x7fe76019dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:02:53.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.213+0000 7fe75f200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe76019e680 con 0x7fe760073a00 2024-04-20T22:02:53.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.213+0000 7fe75f200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe760197b30 con 0x7fe7600730f0 2024-04-20T22:02:53.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.214+0000 7fe757e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe760073a00 0x7fe76019dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:02:53.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.214+0000 7fe757e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe760073a00 0x7fe76019dff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40876/0 (socket says 172.21.15.62:40876) 2024-04-20T22:02:53.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.214+0000 7fe757e00700 1 -- 172.21.15.62:0/2271416213 learned_addr learned my addr 172.21.15.62:0/2271416213 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:02:53.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.214+0000 7fe75de00700 1 --2- 172.21.15.62:0/2271416213 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe7600730f0 0x7fe76019dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:02:53.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.214+0000 7fe757e00700 1 -- 172.21.15.62:0/2271416213 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe7600730f0 msgr2=0x7fe76019dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:02:53.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.214+0000 7fe757e00700 1 --2- 172.21.15.62:0/2271416213 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe7600730f0 0x7fe76019dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:02:53.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.214+0000 7fe757e00700 1 -- 172.21.15.62:0/2271416213 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7500095f0 con 0x7fe760073a00 2024-04-20T22:02:53.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.214+0000 7fe75de00700 1 --2- 172.21.15.62:0/2271416213 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe7600730f0 0x7fe76019dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T22:02:53.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.215+0000 7fe757e00700 1 --2- 172.21.15.62:0/2271416213 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe760073a00 0x7fe76019dff0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fe74800c930 tx=0x7fe74800ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:02:53.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.215+0000 7fe756a00700 1 -- 172.21.15.62:0/2271416213 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe748007ad0 con 0x7fe760073a00 2024-04-20T22:02:53.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.215+0000 7fe756a00700 1 -- 172.21.15.62:0/2271416213 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe748007c30 con 0x7fe760073a00 2024-04-20T22:02:53.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.215+0000 7fe756a00700 1 -- 172.21.15.62:0/2271416213 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe748018610 con 0x7fe760073a00 2024-04-20T22:02:53.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.215+0000 7fe75f200700 1 -- 172.21.15.62:0/2271416213 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe760197e10 con 0x7fe760073a00 2024-04-20T22:02:53.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.215+0000 7fe75f200700 1 -- 172.21.15.62:0/2271416213 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe760198310 con 0x7fe760073a00 2024-04-20T22:02:53.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.216+0000 7fe75f200700 1 -- 172.21.15.62:0/2271416213 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe760109a30 con 0x7fe760073a00 2024-04-20T22:02:53.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.217+0000 7fe756a00700 1 -- 172.21.15.62:0/2271416213 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe74800f450 con 0x7fe760073a00 2024-04-20T22:02:53.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.218+0000 7fe756a00700 1 --2- 172.21.15.62:0/2271416213 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe74406ecc0 0x7fe744071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:02:53.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.218+0000 7fe756a00700 1 -- 172.21.15.62:0/2271416213 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fe74808eff0 con 0x7fe760073a00 2024-04-20T22:02:53.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.219+0000 7fe75de00700 1 --2- 172.21.15.62:0/2271416213 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe74406ecc0 0x7fe744071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:02:53.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.219+0000 7fe75de00700 1 --2- 172.21.15.62:0/2271416213 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe74406ecc0 0x7fe744071170 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fe750006010 tx=0x7fe75000b540 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:02:53.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.223+0000 7fe756a00700 1 -- 172.21.15.62:0/2271416213 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fe74805b910 con 0x7fe760073a00 2024-04-20T22:02:53.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.481+0000 7fe75f200700 1 -- 172.21.15.62:0/2271416213 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe7601985e0 con 0x7fe74406ecc0 2024-04-20T22:02:53.481 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:02:53 smithi062 ceph-mon[76322]: pgmap v828: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:02:53.481 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:02:53 smithi062 ceph-mon[76322]: from='client.35492 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:02:53.481 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:02:53 smithi062 ceph-mon[76322]: from='client.35496 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:02:53.481 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:02:53 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/1461070285' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:02:53.481 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:02:53 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/3030124894' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:02:53.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.484+0000 7fe756a00700 1 -- 172.21.15.62:0/2271416213 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fe7601985e0 con 0x7fe74406ecc0 2024-04-20T22:02:53.483 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:02:53.483 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T22:02:53.484 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T22:02:53.484 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-20T22:02:53.484 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T22:02:53.484 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr", 2024-04-20T22:02:53.484 INFO:teuthology.orchestra.run.smithi062.stdout: "crash", 2024-04-20T22:02:53.484 INFO:teuthology.orchestra.run.smithi062.stdout: "mon", 2024-04-20T22:02:53.484 INFO:teuthology.orchestra.run.smithi062.stdout: "osd" 2024-04-20T22:02:53.484 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T22:02:53.484 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "15/23 daemons upgraded", 2024-04-20T22:02:53.484 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading mds daemons", 2024-04-20T22:02:53.484 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T22:02:53.484 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:02:53.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.487+0000 7fe75f200700 1 -- 172.21.15.62:0/2271416213 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe74406ecc0 msgr2=0x7fe744071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:02:53.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.487+0000 7fe75f200700 1 --2- 172.21.15.62:0/2271416213 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe74406ecc0 0x7fe744071170 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fe750006010 tx=0x7fe75000b540 comp rx=0 tx=0).stop 2024-04-20T22:02:53.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.487+0000 7fe75f200700 1 -- 172.21.15.62:0/2271416213 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe760073a00 msgr2=0x7fe76019dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:02:53.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.487+0000 7fe75f200700 1 --2- 172.21.15.62:0/2271416213 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe760073a00 0x7fe76019dff0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fe74800c930 tx=0x7fe74800ccf0 comp rx=0 tx=0).stop 2024-04-20T22:02:53.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.488+0000 7fe75f200700 1 -- 172.21.15.62:0/2271416213 shutdown_connections 2024-04-20T22:02:53.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.488+0000 7fe75f200700 1 --2- 172.21.15.62:0/2271416213 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe74406ecc0 0x7fe744071170 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:02:53.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.488+0000 7fe75f200700 1 --2- 172.21.15.62:0/2271416213 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe7600730f0 0x7fe76019dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:02:53.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.488+0000 7fe75f200700 1 --2- 172.21.15.62:0/2271416213 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe760073a00 0x7fe76019dff0 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:02:53.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.488+0000 7fe75f200700 1 -- 172.21.15.62:0/2271416213 >> 172.21.15.62:0/2271416213 conn(0x7fe7600fbb10 msgr2=0x7fe760106b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:02:53.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.488+0000 7fe75f200700 1 -- 172.21.15.62:0/2271416213 shutdown_connections 2024-04-20T22:02:53.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.488+0000 7fe75f200700 1 -- 172.21.15.62:0/2271416213 wait complete. 2024-04-20T22:02:53.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:02:53 smithi189 ceph-mon[57022]: pgmap v828: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:02:53.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:02:53 smithi189 ceph-mon[57022]: from='client.35492 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:02:53.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:02:53 smithi189 ceph-mon[57022]: from='client.35496 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:02:53.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:02:53 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/1461070285' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:02:53.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:02:53 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/3030124894' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:02:53.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.647+0000 7f0917200700 1 -- 172.21.15.62:0/1651769799 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0918100c30 msgr2=0x7f0918104c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:02:53.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.647+0000 7f0917200700 1 --2- 172.21.15.62:0/1651769799 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0918100c30 0x7f0918104c00 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f0900009910 tx=0x7f0900009c20 comp rx=0 tx=0).stop 2024-04-20T22:02:53.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.647+0000 7f0917200700 1 -- 172.21.15.62:0/1651769799 shutdown_connections 2024-04-20T22:02:53.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.647+0000 7f0917200700 1 --2- 172.21.15.62:0/1651769799 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0918100c30 0x7f0918104c00 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:02:53.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.647+0000 7f0917200700 1 --2- 172.21.15.62:0/1651769799 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0918100290 0x7f0918100660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:02:53.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.647+0000 7f0917200700 1 -- 172.21.15.62:0/1651769799 >> 172.21.15.62:0/1651769799 conn(0x7f09180fbb10 msgr2=0x7f09180fdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:02:53.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.647+0000 7f0917200700 1 -- 172.21.15.62:0/1651769799 shutdown_connections 2024-04-20T22:02:53.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.647+0000 7f0917200700 1 -- 172.21.15.62:0/1651769799 wait complete. 2024-04-20T22:02:53.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.648+0000 7f0917200700 1 Processor -- start 2024-04-20T22:02:53.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.648+0000 7f0917200700 1 -- start start 2024-04-20T22:02:53.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.649+0000 7f0917200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0918100290 0x7f0918197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:02:53.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.649+0000 7f0917200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0918100c30 0x7f0918198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:02:53.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.649+0000 7f0917200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0918198b30 con 0x7f0918100c30 2024-04-20T22:02:53.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.649+0000 7f0917200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f091819c8c0 con 0x7f0918100290 2024-04-20T22:02:53.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.649+0000 7f090d400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0918100c30 0x7f0918198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:02:53.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.649+0000 7f090d400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0918100c30 0x7f0918198450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40898/0 (socket says 172.21.15.62:40898) 2024-04-20T22:02:53.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.649+0000 7f090d400700 1 -- 172.21.15.62:0/1881588712 learned_addr learned my addr 172.21.15.62:0/1881588712 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:02:53.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.649+0000 7f0915e00700 1 --2- 172.21.15.62:0/1881588712 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0918100290 0x7f0918197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:02:53.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.649+0000 7f090d400700 1 -- 172.21.15.62:0/1881588712 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0918100290 msgr2=0x7f0918197f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:02:53.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.649+0000 7f090d400700 1 --2- 172.21.15.62:0/1881588712 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0918100290 0x7f0918197f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:02:53.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.649+0000 7f090d400700 1 -- 172.21.15.62:0/1881588712 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09000095f0 con 0x7f0918100c30 2024-04-20T22:02:53.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.649+0000 7f0915e00700 1 --2- 172.21.15.62:0/1881588712 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0918100290 0x7f0918197f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T22:02:53.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.650+0000 7f090d400700 1 --2- 172.21.15.62:0/1881588712 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0918100c30 0x7f0918198450 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f090000bda0 tx=0x7f090000be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:02:53.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.650+0000 7f090f400700 1 -- 172.21.15.62:0/1881588712 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f090001d070 con 0x7f0918100c30 2024-04-20T22:02:53.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.650+0000 7f090f400700 1 -- 172.21.15.62:0/1881588712 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0900004fb0 con 0x7f0918100c30 2024-04-20T22:02:53.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.650+0000 7f0917200700 1 -- 172.21.15.62:0/1881588712 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f091819cb40 con 0x7f0918100c30 2024-04-20T22:02:53.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.650+0000 7f090f400700 1 -- 172.21.15.62:0/1881588712 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f090000f460 con 0x7f0918100c30 2024-04-20T22:02:53.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.650+0000 7f0917200700 1 -- 172.21.15.62:0/1881588712 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f091819d030 con 0x7f0918100c30 2024-04-20T22:02:53.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.651+0000 7f0917200700 1 -- 172.21.15.62:0/1881588712 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0918104380 con 0x7f0918100c30 2024-04-20T22:02:53.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.653+0000 7f090f400700 1 -- 172.21.15.62:0/1881588712 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f0900022800 con 0x7f0918100c30 2024-04-20T22:02:53.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.653+0000 7f090f400700 1 --2- 172.21.15.62:0/1881588712 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f08fc06ecc0 0x7f08fc071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:02:53.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.654+0000 7f090f400700 1 -- 172.21.15.62:0/1881588712 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f09000903a0 con 0x7f0918100c30 2024-04-20T22:02:53.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.654+0000 7f0915e00700 1 --2- 172.21.15.62:0/1881588712 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f08fc06ecc0 0x7f08fc071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:02:53.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.654+0000 7f0915e00700 1 --2- 172.21.15.62:0/1881588712 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f08fc06ecc0 0x7f08fc071170 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f0908007eb0 tx=0x7f09080073c0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:02:53.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.658+0000 7f090f400700 1 -- 172.21.15.62:0/1881588712 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f0900059320 con 0x7f0918100c30 2024-04-20T22:02:53.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.972+0000 7f0917200700 1 -- 172.21.15.62:0/1881588712 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f0918062380 con 0x7f0918100c30 2024-04-20T22:02:53.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.973+0000 7f090f400700 1 -- 172.21.15.62:0/1881588712 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7f090005c9e0 con 0x7f0918100c30 2024-04-20T22:02:53.973 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-20T22:02:53.973 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-20T22:02:53.973 INFO:teuthology.orchestra.run.smithi062.stdout: host smithi062 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T22:02:53.973 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-20T22:02:53.973 INFO:teuthology.orchestra.run.smithi062.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi062 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpq7y7oa_r:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:02:53.973 INFO:teuthology.orchestra.run.smithi062.stdout:Traceback (most recent call last): 2024-04-20T22:02:53.973 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-20T22:02:53.973 INFO:teuthology.orchestra.run.smithi062.stdout: return _run_code(code, main_globals, None, 2024-04-20T22:02:53.973 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-20T22:02:53.973 INFO:teuthology.orchestra.run.smithi062.stdout: exec(code, run_globals) 2024-04-20T22:02:53.974 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5618, in 2024-04-20T22:02:53.974 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5606, in main 2024-04-20T22:02:53.974 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 408, in _infer_config 2024-04-20T22:02:53.974 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 323, in _infer_fsid 2024-04-20T22:02:53.974 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 436, in _infer_image 2024-04-20T22:02:53.974 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 310, in _validate_fsid 2024-04-20T22:02:53.974 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 3359, in command_ceph_volume 2024-04-20T22:02:53.974 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-20T22:02:53.974 INFO:teuthology.orchestra.run.smithi062.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi062 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpq7y7oa_r:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:02:53.974 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-20T22:02:53.974 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is degraded 2024-04-20T22:02:53.974 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T22:02:53.974 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T22:02:53.974 INFO:teuthology.orchestra.run.smithi062.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-20T22:02:53.975 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-20T22:02:53.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.976+0000 7f0917200700 1 -- 172.21.15.62:0/1881588712 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f08fc06ecc0 msgr2=0x7f08fc071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:02:53.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.976+0000 7f0917200700 1 --2- 172.21.15.62:0/1881588712 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f08fc06ecc0 0x7f08fc071170 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f0908007eb0 tx=0x7f09080073c0 comp rx=0 tx=0).stop 2024-04-20T22:02:53.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.976+0000 7f0917200700 1 -- 172.21.15.62:0/1881588712 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0918100c30 msgr2=0x7f0918198450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:02:53.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.976+0000 7f0917200700 1 --2- 172.21.15.62:0/1881588712 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0918100c30 0x7f0918198450 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f090000bda0 tx=0x7f090000be80 comp rx=0 tx=0).stop 2024-04-20T22:02:53.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.976+0000 7f0917200700 1 -- 172.21.15.62:0/1881588712 shutdown_connections 2024-04-20T22:02:53.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.976+0000 7f0917200700 1 --2- 172.21.15.62:0/1881588712 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f08fc06ecc0 0x7f08fc071170 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:02:53.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.976+0000 7f0917200700 1 --2- 172.21.15.62:0/1881588712 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0918100290 0x7f0918197f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:02:53.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.976+0000 7f0917200700 1 --2- 172.21.15.62:0/1881588712 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0918100c30 0x7f0918198450 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:02:53.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.976+0000 7f0917200700 1 -- 172.21.15.62:0/1881588712 >> 172.21.15.62:0/1881588712 conn(0x7f09180fbb10 msgr2=0x7f09180fd220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:02:53.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.976+0000 7f0917200700 1 -- 172.21.15.62:0/1881588712 shutdown_connections 2024-04-20T22:02:53.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:02:53.976+0000 7f0917200700 1 -- 172.21.15.62:0/1881588712 wait complete. 2024-04-20T22:02:54.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:02:54 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/1881588712' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:02:54.483 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:02:54 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/1881588712' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:02:55.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:02:55 smithi189 ceph-mon[57022]: pgmap v829: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:02:55.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:02:55 smithi189 ceph-mon[57022]: from='client.35508 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:02:55.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:02:55 smithi062 ceph-mon[76322]: pgmap v829: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:02:55.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:02:55 smithi062 ceph-mon[76322]: from='client.35508 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:02:56.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:02:56 smithi189 ceph-mon[57022]: pgmap v830: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:02:56.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:02:56 smithi062 ceph-mon[76322]: pgmap v830: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:02:58.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:02:58 smithi189 ceph-mon[57022]: pgmap v831: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:02:58.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:02:58 smithi062 ceph-mon[76322]: pgmap v831: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:03:00.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:03:00 smithi189 ceph-mon[57022]: pgmap v832: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:03:01.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:03:00 smithi062 ceph-mon[76322]: pgmap v832: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:03:02.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:03:02 smithi062 ceph-mon[76322]: pgmap v833: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:03:02.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:03:02 smithi189 ceph-mon[57022]: pgmap v833: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:03:04.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:03:04 smithi189 ceph-mon[57022]: pgmap v834: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:03:04.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:03:04 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:03:05.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:03:04 smithi062 ceph-mon[76322]: pgmap v834: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:03:05.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:03:04 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:03:06.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:03:06 smithi189 ceph-mon[57022]: pgmap v835: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:03:07.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:03:06 smithi062 ceph-mon[76322]: pgmap v835: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:03:08.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:03:08 smithi189 ceph-mon[57022]: pgmap v836: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:03:09.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:03:08 smithi062 ceph-mon[76322]: pgmap v836: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:03:10.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:03:10 smithi189 ceph-mon[57022]: pgmap v837: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:03:11.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:03:10 smithi062 ceph-mon[76322]: pgmap v837: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:03:12.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:03:12 smithi189 ceph-mon[57022]: pgmap v838: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:03:13.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:03:12 smithi062 ceph-mon[76322]: pgmap v838: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:03:14.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:03:14 smithi189 ceph-mon[57022]: pgmap v839: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:03:15.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:03:14 smithi062 ceph-mon[76322]: pgmap v839: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:03:16.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:03:16 smithi189 ceph-mon[57022]: pgmap v840: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:03:17.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:03:16 smithi062 ceph-mon[76322]: pgmap v840: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:03:18.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:03:18 smithi189 ceph-mon[57022]: pgmap v841: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:03:19.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:03:18 smithi062 ceph-mon[76322]: pgmap v841: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:03:19.990 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:03:19 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:03:20.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:03:19 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:03:20.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:03:20 smithi189 ceph-mon[57022]: pgmap v842: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:03:21.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:03:20 smithi062 ceph-mon[76322]: pgmap v842: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:03:22.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:03:22 smithi189 ceph-mon[57022]: pgmap v843: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:03:23.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:03:22 smithi062 ceph-mon[76322]: pgmap v843: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:03:24.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:24.151+0000 7f6970c00700 1 -- 172.21.15.62:0/3778211479 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f696c10c8f0 msgr2=0x7f696c10ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:03:24.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:24.151+0000 7f6970c00700 1 --2- 172.21.15.62:0/3778211479 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f696c10c8f0 0x7f696c10ccc0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f695c009910 tx=0x7f695c009c20 comp rx=0 tx=0).stop 2024-04-20T22:03:24.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:24.152+0000 7f6970c00700 1 -- 172.21.15.62:0/3778211479 shutdown_connections 2024-04-20T22:03:24.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:24.152+0000 7f6970c00700 1 --2- 172.21.15.62:0/3778211479 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f696c106b80 0x7f696c106ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:03:24.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:24.152+0000 7f6970c00700 1 --2- 172.21.15.62:0/3778211479 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f696c10c8f0 0x7f696c10ccc0 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:03:24.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:24.152+0000 7f6970c00700 1 -- 172.21.15.62:0/3778211479 >> 172.21.15.62:0/3778211479 conn(0x7f696c076060 msgr2=0x7f696c078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:03:24.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:24.152+0000 7f6970c00700 1 -- 172.21.15.62:0/3778211479 shutdown_connections 2024-04-20T22:03:24.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:24.152+0000 7f6970c00700 1 -- 172.21.15.62:0/3778211479 wait complete. 2024-04-20T22:03:24.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:24.153+0000 7f6970c00700 1 Processor -- start 2024-04-20T22:03:24.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:24.153+0000 7f6970c00700 1 -- start start 2024-04-20T22:03:24.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:24.153+0000 7f6970c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f696c106b80 0x7f696c19c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:03:24.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:24.153+0000 7f6970c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f696c10c8f0 0x7f696c19ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:03:24.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:24.153+0000 7f6970c00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f696c19d120 con 0x7f696c10c8f0 2024-04-20T22:03:24.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:24.153+0000 7f6970c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f696c1a0eb0 con 0x7f696c106b80 2024-04-20T22:03:24.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:24.154+0000 7f6963e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f696c10c8f0 0x7f696c19ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:03:24.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:24.154+0000 7f6963e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f696c10c8f0 0x7f696c19ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47372/0 (socket says 172.21.15.62:47372) 2024-04-20T22:03:24.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:24.154+0000 7f6963e00700 1 -- 172.21.15.62:0/2221072434 learned_addr learned my addr 172.21.15.62:0/2221072434 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:03:24.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:24.154+0000 7f6969e00700 1 --2- 172.21.15.62:0/2221072434 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f696c106b80 0x7f696c19c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:03:24.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:24.154+0000 7f6963e00700 1 -- 172.21.15.62:0/2221072434 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f696c106b80 msgr2=0x7f696c19c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:03:24.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:24.154+0000 7f6963e00700 1 --2- 172.21.15.62:0/2221072434 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f696c106b80 0x7f696c19c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:03:24.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:24.154+0000 7f6963e00700 1 -- 172.21.15.62:0/2221072434 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f695c0095f0 con 0x7f696c10c8f0 2024-04-20T22:03:24.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:24.155+0000 7f6963e00700 1 --2- 172.21.15.62:0/2221072434 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f696c10c8f0 0x7f696c19ca40 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f695400d8d0 tx=0x7f695400dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:03:24.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:24.155+0000 7f6962a00700 1 -- 172.21.15.62:0/2221072434 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f69540098a0 con 0x7f696c10c8f0 2024-04-20T22:03:24.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:24.155+0000 7f6962a00700 1 -- 172.21.15.62:0/2221072434 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6954010460 con 0x7f696c10c8f0 2024-04-20T22:03:24.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:24.155+0000 7f6970c00700 1 -- 172.21.15.62:0/2221072434 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f696c1a1190 con 0x7f696c10c8f0 2024-04-20T22:03:24.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:24.155+0000 7f6962a00700 1 -- 172.21.15.62:0/2221072434 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f695400f5f0 con 0x7f696c10c8f0 2024-04-20T22:03:24.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:24.155+0000 7f6970c00700 1 -- 172.21.15.62:0/2221072434 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f696c1a16e0 con 0x7f696c10c8f0 2024-04-20T22:03:24.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:24.157+0000 7f6970c00700 1 -- 172.21.15.62:0/2221072434 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f696c109b20 con 0x7f696c10c8f0 2024-04-20T22:03:24.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:24.158+0000 7f6962a00700 1 -- 172.21.15.62:0/2221072434 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f6954009a00 con 0x7f696c10c8f0 2024-04-20T22:03:24.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:24.159+0000 7f6962a00700 1 --2- 172.21.15.62:0/2221072434 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f695006ecc0 0x7f6950071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:03:24.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:24.160+0000 7f6962a00700 1 -- 172.21.15.62:0/2221072434 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f695408edd0 con 0x7f696c10c8f0 2024-04-20T22:03:24.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:24.160+0000 7f6969e00700 1 --2- 172.21.15.62:0/2221072434 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f695006ecc0 0x7f6950071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:03:24.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:24.161+0000 7f6969e00700 1 --2- 172.21.15.62:0/2221072434 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f695006ecc0 0x7f6950071170 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f695c006010 tx=0x7f695c01a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:03:24.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:24.165+0000 7f6962a00700 1 -- 172.21.15.62:0/2221072434 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f6954057ae0 con 0x7f696c10c8f0 2024-04-20T22:03:24.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:24.425+0000 7f6970c00700 1 -- 172.21.15.62:0/2221072434 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f696c1a1970 con 0x7f695006ecc0 2024-04-20T22:03:24.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:24.428+0000 7f6962a00700 1 -- 172.21.15.62:0/2221072434 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f696c1a1970 con 0x7f695006ecc0 2024-04-20T22:03:24.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:24.431+0000 7f6970c00700 1 -- 172.21.15.62:0/2221072434 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f695006ecc0 msgr2=0x7f6950071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:03:24.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:24.431+0000 7f6970c00700 1 --2- 172.21.15.62:0/2221072434 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f695006ecc0 0x7f6950071170 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f695c006010 tx=0x7f695c01a040 comp rx=0 tx=0).stop 2024-04-20T22:03:24.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:24.431+0000 7f6970c00700 1 -- 172.21.15.62:0/2221072434 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f696c10c8f0 msgr2=0x7f696c19ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:03:24.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:24.431+0000 7f6970c00700 1 --2- 172.21.15.62:0/2221072434 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f696c10c8f0 0x7f696c19ca40 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f695400d8d0 tx=0x7f695400dbe0 comp rx=0 tx=0).stop 2024-04-20T22:03:24.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:24.431+0000 7f6970c00700 1 -- 172.21.15.62:0/2221072434 shutdown_connections 2024-04-20T22:03:24.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:24.431+0000 7f6970c00700 1 --2- 172.21.15.62:0/2221072434 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f695006ecc0 0x7f6950071170 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:03:24.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:24.431+0000 7f6970c00700 1 --2- 172.21.15.62:0/2221072434 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f696c106b80 0x7f696c19c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:03:24.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:24.431+0000 7f6970c00700 1 --2- 172.21.15.62:0/2221072434 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f696c10c8f0 0x7f696c19ca40 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:03:24.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:24.431+0000 7f6970c00700 1 -- 172.21.15.62:0/2221072434 >> 172.21.15.62:0/2221072434 conn(0x7f696c076060 msgr2=0x7f696c077aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:03:24.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:24.432+0000 7f6970c00700 1 -- 172.21.15.62:0/2221072434 shutdown_connections 2024-04-20T22:03:24.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:24.432+0000 7f6970c00700 1 -- 172.21.15.62:0/2221072434 wait complete. 2024-04-20T22:03:24.449 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T22:03:24.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:24.627+0000 7f5451e00700 1 -- 172.21.15.62:0/1450751093 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f544c10c8f0 msgr2=0x7f544c10ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:03:24.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:24.627+0000 7f5451e00700 1 --2- 172.21.15.62:0/1450751093 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f544c10c8f0 0x7f544c10ccc0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f543c009910 tx=0x7f543c009c20 comp rx=0 tx=0).stop 2024-04-20T22:03:24.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:24.627+0000 7f5451e00700 1 -- 172.21.15.62:0/1450751093 shutdown_connections 2024-04-20T22:03:24.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:24.627+0000 7f5451e00700 1 --2- 172.21.15.62:0/1450751093 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f544c106b80 0x7f544c106ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:03:24.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:24.627+0000 7f5451e00700 1 --2- 172.21.15.62:0/1450751093 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f544c10c8f0 0x7f544c10ccc0 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:03:24.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:24.627+0000 7f5451e00700 1 -- 172.21.15.62:0/1450751093 >> 172.21.15.62:0/1450751093 conn(0x7f544c076060 msgr2=0x7f544c078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:03:24.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:24.627+0000 7f5451e00700 1 -- 172.21.15.62:0/1450751093 shutdown_connections 2024-04-20T22:03:24.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:24.627+0000 7f5451e00700 1 -- 172.21.15.62:0/1450751093 wait complete. 2024-04-20T22:03:24.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:24.628+0000 7f5451e00700 1 Processor -- start 2024-04-20T22:03:24.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:24.629+0000 7f5451e00700 1 -- start start 2024-04-20T22:03:24.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:24.629+0000 7f5451e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f544c106b80 0x7f544c19c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:03:24.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:24.629+0000 7f5451e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f544c10c8f0 0x7f544c19ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:03:24.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:24.629+0000 7f5451e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f544c19d120 con 0x7f544c106b80 2024-04-20T22:03:24.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:24.629+0000 7f5451e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f544c1a0eb0 con 0x7f544c10c8f0 2024-04-20T22:03:24.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:24.629+0000 7f544aa00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f544c106b80 0x7f544c19c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:03:24.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:24.629+0000 7f544aa00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f544c106b80 0x7f544c19c500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47388/0 (socket says 172.21.15.62:47388) 2024-04-20T22:03:24.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:24.629+0000 7f544aa00700 1 -- 172.21.15.62:0/138128091 learned_addr learned my addr 172.21.15.62:0/138128091 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:03:24.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:24.630+0000 7f544a000700 1 --2- 172.21.15.62:0/138128091 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f544c10c8f0 0x7f544c19ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:03:24.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:24.630+0000 7f544aa00700 1 -- 172.21.15.62:0/138128091 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f544c10c8f0 msgr2=0x7f544c19ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:03:24.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:24.630+0000 7f544aa00700 1 --2- 172.21.15.62:0/138128091 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f544c10c8f0 0x7f544c19ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:03:24.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:24.630+0000 7f544aa00700 1 -- 172.21.15.62:0/138128091 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f543c0095f0 con 0x7f544c106b80 2024-04-20T22:03:24.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:24.630+0000 7f544aa00700 1 --2- 172.21.15.62:0/138128091 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f544c106b80 0x7f544c19c500 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f543c006010 tx=0x7f543c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:03:24.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:24.631+0000 7f5450a00700 1 -- 172.21.15.62:0/138128091 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f543c01d070 con 0x7f544c106b80 2024-04-20T22:03:24.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:24.631+0000 7f5450a00700 1 -- 172.21.15.62:0/138128091 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f543c004fb0 con 0x7f544c106b80 2024-04-20T22:03:24.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:24.631+0000 7f5450a00700 1 -- 172.21.15.62:0/138128091 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f543c022ea0 con 0x7f544c106b80 2024-04-20T22:03:24.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:24.631+0000 7f5451e00700 1 -- 172.21.15.62:0/138128091 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f544c1a1130 con 0x7f544c106b80 2024-04-20T22:03:24.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:24.631+0000 7f5451e00700 1 -- 172.21.15.62:0/138128091 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f544c1a1620 con 0x7f544c106b80 2024-04-20T22:03:24.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:24.633+0000 7f5451e00700 1 -- 172.21.15.62:0/138128091 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f544c109b20 con 0x7f544c106b80 2024-04-20T22:03:24.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:24.634+0000 7f5450a00700 1 -- 172.21.15.62:0/138128091 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f543c022470 con 0x7f544c106b80 2024-04-20T22:03:24.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:24.635+0000 7f5450a00700 1 --2- 172.21.15.62:0/138128091 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f543006ecc0 0x7f5430071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:03:24.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:24.635+0000 7f5450a00700 1 -- 172.21.15.62:0/138128091 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f543c0903c0 con 0x7f544c106b80 2024-04-20T22:03:24.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:24.635+0000 7f544a000700 1 --2- 172.21.15.62:0/138128091 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f543006ecc0 0x7f5430071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:03:24.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:24.636+0000 7f544a000700 1 --2- 172.21.15.62:0/138128091 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f543006ecc0 0x7f5430071170 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f5438005fd0 tx=0x7f543800a560 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:03:24.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:24.640+0000 7f5450a00700 1 -- 172.21.15.62:0/138128091 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f543c059200 con 0x7f544c106b80 2024-04-20T22:03:24.680 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:03:24 smithi062 ceph-mon[76322]: pgmap v844: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:03:24.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:24.899+0000 7f5451e00700 1 -- 172.21.15.62:0/138128091 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f544c1a1890 con 0x7f543006ecc0 2024-04-20T22:03:24.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:24.901+0000 7f5450a00700 1 -- 172.21.15.62:0/138128091 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f544c1a1890 con 0x7f543006ecc0 2024-04-20T22:03:24.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:24.904+0000 7f5451e00700 1 -- 172.21.15.62:0/138128091 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f543006ecc0 msgr2=0x7f5430071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:03:24.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:24.904+0000 7f5451e00700 1 --2- 172.21.15.62:0/138128091 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f543006ecc0 0x7f5430071170 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f5438005fd0 tx=0x7f543800a560 comp rx=0 tx=0).stop 2024-04-20T22:03:24.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:24.904+0000 7f5451e00700 1 -- 172.21.15.62:0/138128091 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f544c106b80 msgr2=0x7f544c19c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:03:24.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:24.904+0000 7f5451e00700 1 --2- 172.21.15.62:0/138128091 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f544c106b80 0x7f544c19c500 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f543c006010 tx=0x7f543c00be20 comp rx=0 tx=0).stop 2024-04-20T22:03:24.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:24.904+0000 7f5451e00700 1 -- 172.21.15.62:0/138128091 shutdown_connections 2024-04-20T22:03:24.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:24.904+0000 7f5451e00700 1 --2- 172.21.15.62:0/138128091 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f543006ecc0 0x7f5430071170 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:03:24.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:24.904+0000 7f5451e00700 1 --2- 172.21.15.62:0/138128091 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f544c106b80 0x7f544c19c500 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:03:24.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:24.904+0000 7f5451e00700 1 --2- 172.21.15.62:0/138128091 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f544c10c8f0 0x7f544c19ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:03:24.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:24.904+0000 7f5451e00700 1 -- 172.21.15.62:0/138128091 >> 172.21.15.62:0/138128091 conn(0x7f544c076060 msgr2=0x7f544c077aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:03:24.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:24.905+0000 7f5451e00700 1 -- 172.21.15.62:0/138128091 shutdown_connections 2024-04-20T22:03:24.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:24.905+0000 7f5451e00700 1 -- 172.21.15.62:0/138128091 wait complete. 2024-04-20T22:03:24.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:03:24 smithi189 ceph-mon[57022]: pgmap v844: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:03:25.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:25.072+0000 7f2a33e00700 1 -- 172.21.15.62:0/3297763962 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2a2c102470 msgr2=0x7f2a2c1028e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:03:25.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:25.072+0000 7f2a33e00700 1 --2- 172.21.15.62:0/3297763962 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2a2c102470 0x7f2a2c1028e0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f2a18009910 tx=0x7f2a18009c20 comp rx=0 tx=0).stop 2024-04-20T22:03:25.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:25.072+0000 7f2a33e00700 1 -- 172.21.15.62:0/3297763962 shutdown_connections 2024-04-20T22:03:25.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:25.072+0000 7f2a33e00700 1 --2- 172.21.15.62:0/3297763962 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2a2c102470 0x7f2a2c1028e0 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:03:25.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:25.072+0000 7f2a33e00700 1 --2- 172.21.15.62:0/3297763962 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a2c108470 0x7f2a2c108840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:03:25.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:25.072+0000 7f2a33e00700 1 -- 172.21.15.62:0/3297763962 >> 172.21.15.62:0/3297763962 conn(0x7f2a2c0fdf90 msgr2=0x7f2a2c1003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:03:25.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:25.073+0000 7f2a33e00700 1 -- 172.21.15.62:0/3297763962 shutdown_connections 2024-04-20T22:03:25.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:25.073+0000 7f2a33e00700 1 -- 172.21.15.62:0/3297763962 wait complete. 2024-04-20T22:03:25.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:25.074+0000 7f2a33e00700 1 Processor -- start 2024-04-20T22:03:25.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:25.074+0000 7f2a33e00700 1 -- start start 2024-04-20T22:03:25.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:25.074+0000 7f2a33e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2a2c102470 0x7f2a2c198190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:03:25.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:25.074+0000 7f2a33e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a2c108470 0x7f2a2c1986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:03:25.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:25.074+0000 7f2a33e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a2c198db0 con 0x7f2a2c102470 2024-04-20T22:03:25.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:25.074+0000 7f2a33e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a2c19cb40 con 0x7f2a2c108470 2024-04-20T22:03:25.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:25.075+0000 7f2a31400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2a2c102470 0x7f2a2c198190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:03:25.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:25.075+0000 7f2a31400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2a2c102470 0x7f2a2c198190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47406/0 (socket says 172.21.15.62:47406) 2024-04-20T22:03:25.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:25.075+0000 7f2a31400700 1 -- 172.21.15.62:0/2115994539 learned_addr learned my addr 172.21.15.62:0/2115994539 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:03:25.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:25.075+0000 7f2a30a00700 1 --2- 172.21.15.62:0/2115994539 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a2c108470 0x7f2a2c1986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:03:25.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:25.075+0000 7f2a31400700 1 -- 172.21.15.62:0/2115994539 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a2c108470 msgr2=0x7f2a2c1986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:03:25.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:25.075+0000 7f2a31400700 1 --2- 172.21.15.62:0/2115994539 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a2c108470 0x7f2a2c1986d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:03:25.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:25.075+0000 7f2a31400700 1 -- 172.21.15.62:0/2115994539 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a180095f0 con 0x7f2a2c102470 2024-04-20T22:03:25.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:25.075+0000 7f2a31400700 1 --2- 172.21.15.62:0/2115994539 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2a2c102470 0x7f2a2c198190 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f2a2000b4b0 tx=0x7f2a2000b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:03:25.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:25.076+0000 7f2a2a000700 1 -- 172.21.15.62:0/2115994539 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2a20011800 con 0x7f2a2c102470 2024-04-20T22:03:25.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:25.076+0000 7f2a2a000700 1 -- 172.21.15.62:0/2115994539 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2a20011e20 con 0x7f2a2c102470 2024-04-20T22:03:25.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:25.076+0000 7f2a33e00700 1 -- 172.21.15.62:0/2115994539 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a2c19ce20 con 0x7f2a2c102470 2024-04-20T22:03:25.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:25.076+0000 7f2a2a000700 1 -- 172.21.15.62:0/2115994539 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2a2000f2e0 con 0x7f2a2c102470 2024-04-20T22:03:25.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:25.076+0000 7f2a33e00700 1 -- 172.21.15.62:0/2115994539 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a2c19d370 con 0x7f2a2c102470 2024-04-20T22:03:25.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:25.077+0000 7f2a33e00700 1 -- 172.21.15.62:0/2115994539 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a2c10aa00 con 0x7f2a2c102470 2024-04-20T22:03:25.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:25.078+0000 7f2a2a000700 1 -- 172.21.15.62:0/2115994539 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f2a20010380 con 0x7f2a2c102470 2024-04-20T22:03:25.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:25.079+0000 7f2a2a000700 1 --2- 172.21.15.62:0/2115994539 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2a1406ecc0 0x7f2a14071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:03:25.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:25.079+0000 7f2a2a000700 1 -- 172.21.15.62:0/2115994539 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f2a2008ea30 con 0x7f2a2c102470 2024-04-20T22:03:25.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:25.080+0000 7f2a30a00700 1 --2- 172.21.15.62:0/2115994539 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2a1406ecc0 0x7f2a14071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:03:25.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:25.080+0000 7f2a30a00700 1 --2- 172.21.15.62:0/2115994539 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2a1406ecc0 0x7f2a14071170 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f2a18006010 tx=0x7f2a1800b540 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:03:25.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:25.084+0000 7f2a2a000700 1 -- 172.21.15.62:0/2115994539 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f2a20057870 con 0x7f2a2c102470 2024-04-20T22:03:25.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:25.318+0000 7f2a33e00700 1 -- 172.21.15.62:0/2115994539 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f2a2c19d640 con 0x7f2a1406ecc0 2024-04-20T22:03:25.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:25.349+0000 7f2a2a000700 1 -- 172.21.15.62:0/2115994539 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f2a2c19d640 con 0x7f2a1406ecc0 2024-04-20T22:03:25.349 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T22:03:25.349 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (29m) 5m ago 43m 15.3M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T22:03:25.349 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (43m) 5m ago 43m 10.4M - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T22:03:25.349 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (42m) 5m ago 42m 10.1M - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T22:03:25.349 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (25m) 5m ago 43m 6660k - 19.0.0-3133-ga8c84f59 76bc6b91b726 2825d5ba11c6 2024-04-20T22:03:25.349 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (25m) 5m ago 42m 6665k - 19.0.0-3133-ga8c84f59 76bc6b91b726 20f746fa1bba 2024-04-20T22:03:25.349 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (28m) 5m ago 42m 73.5M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T22:03:25.350 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (20m) 5m ago 35m 18.5M - 19.0.0-3133-ga8c84f59 76bc6b91b726 de80bc90a683 2024-04-20T22:03:25.350 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (21m) 5m ago 35m 18.7M - 19.0.0-3133-ga8c84f59 76bc6b91b726 cd793b7ee98e 2024-04-20T22:03:25.350 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (5m) 5m ago 35m 11.0M - 19.0.0-3133-ga8c84f59 76bc6b91b726 0e7b0bc4d1ab 2024-04-20T22:03:25.350 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (35m) 5m ago 35m 18.6M - 18.2.0 dc2bc1663786 f2eec2ee8951 2024-04-20T22:03:25.350 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (32m) 5m ago 44m 513M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T22:03:25.350 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (30m) 5m ago 41m 459M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T22:03:25.350 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (27m) 5m ago 45m 71.7M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 019eb1479485 2024-04-20T22:03:25.350 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (26m) 5m ago 41m 51.0M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 55aac560aab6 2024-04-20T22:03:25.350 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (30m) 5m ago 43m 8472k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T22:03:25.350 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (30m) 5m ago 41m 8048k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T22:03:25.350 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (24m) 5m ago 40m 91.2M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 ee2bb901f26e 2024-04-20T22:03:25.350 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (24m) 5m ago 40m 94.6M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 9075152d4162 2024-04-20T22:03:25.350 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (23m) 5m ago 39m 78.1M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 2d4f3e38dc0e 2024-04-20T22:03:25.350 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (23m) 5m ago 38m 92.6M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 8f97b17ae0eb 2024-04-20T22:03:25.351 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (22m) 5m ago 38m 95.6M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 653d8079967d 2024-04-20T22:03:25.351 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (21m) 5m ago 37m 89.2M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 596dded14c12 2024-04-20T22:03:25.351 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (29m) 5m ago 42m 82.1M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T22:03:25.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:25.352+0000 7f2a33e00700 1 -- 172.21.15.62:0/2115994539 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2a1406ecc0 msgr2=0x7f2a14071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:03:25.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:25.352+0000 7f2a33e00700 1 --2- 172.21.15.62:0/2115994539 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2a1406ecc0 0x7f2a14071170 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f2a18006010 tx=0x7f2a1800b540 comp rx=0 tx=0).stop 2024-04-20T22:03:25.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:25.352+0000 7f2a33e00700 1 -- 172.21.15.62:0/2115994539 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2a2c102470 msgr2=0x7f2a2c198190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:03:25.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:25.352+0000 7f2a33e00700 1 --2- 172.21.15.62:0/2115994539 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2a2c102470 0x7f2a2c198190 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f2a2000b4b0 tx=0x7f2a2000b870 comp rx=0 tx=0).stop 2024-04-20T22:03:25.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:25.353+0000 7f2a33e00700 1 -- 172.21.15.62:0/2115994539 shutdown_connections 2024-04-20T22:03:25.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:25.353+0000 7f2a33e00700 1 --2- 172.21.15.62:0/2115994539 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2a1406ecc0 0x7f2a14071170 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:03:25.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:25.353+0000 7f2a33e00700 1 --2- 172.21.15.62:0/2115994539 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2a2c102470 0x7f2a2c198190 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:03:25.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:25.353+0000 7f2a33e00700 1 --2- 172.21.15.62:0/2115994539 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a2c108470 0x7f2a2c1986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:03:25.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:25.353+0000 7f2a33e00700 1 -- 172.21.15.62:0/2115994539 >> 172.21.15.62:0/2115994539 conn(0x7f2a2c0fdf90 msgr2=0x7f2a2c0feb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:03:25.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:25.353+0000 7f2a33e00700 1 -- 172.21.15.62:0/2115994539 shutdown_connections 2024-04-20T22:03:25.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:25.353+0000 7f2a33e00700 1 -- 172.21.15.62:0/2115994539 wait complete. 2024-04-20T22:03:25.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:25.515+0000 7f0801c00700 1 -- 172.21.15.62:0/2756036103 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f07fc1026a0 msgr2=0x7f07fc102a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:03:25.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:25.515+0000 7f0801c00700 1 --2- 172.21.15.62:0/2756036103 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f07fc1026a0 0x7f07fc102a70 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f07ec009910 tx=0x7f07ec009c20 comp rx=0 tx=0).stop 2024-04-20T22:03:25.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:25.515+0000 7f0801c00700 1 -- 172.21.15.62:0/2756036103 shutdown_connections 2024-04-20T22:03:25.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:25.515+0000 7f0801c00700 1 --2- 172.21.15.62:0/2756036103 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07fc103040 0x7f07fc107010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:03:25.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:25.515+0000 7f0801c00700 1 --2- 172.21.15.62:0/2756036103 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f07fc1026a0 0x7f07fc102a70 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:03:25.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:25.515+0000 7f0801c00700 1 -- 172.21.15.62:0/2756036103 >> 172.21.15.62:0/2756036103 conn(0x7f07fc0fdf40 msgr2=0x7f07fc100350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:03:25.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:25.515+0000 7f0801c00700 1 -- 172.21.15.62:0/2756036103 shutdown_connections 2024-04-20T22:03:25.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:25.515+0000 7f0801c00700 1 -- 172.21.15.62:0/2756036103 wait complete. 2024-04-20T22:03:25.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:25.516+0000 7f0801c00700 1 Processor -- start 2024-04-20T22:03:25.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:25.516+0000 7f0801c00700 1 -- start start 2024-04-20T22:03:25.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:25.517+0000 7f0801c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f07fc1026a0 0x7f07fc198130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:03:25.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:25.517+0000 7f0801c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07fc103040 0x7f07fc198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:03:25.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:25.517+0000 7f0801c00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07fc198d50 con 0x7f07fc1026a0 2024-04-20T22:03:25.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:25.517+0000 7f0801c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07fc19cae0 con 0x7f07fc103040 2024-04-20T22:03:25.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:25.517+0000 7f07faa00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f07fc1026a0 0x7f07fc198130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:03:25.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:25.517+0000 7f07faa00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f07fc1026a0 0x7f07fc198130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47412/0 (socket says 172.21.15.62:47412) 2024-04-20T22:03:25.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:25.517+0000 7f07faa00700 1 -- 172.21.15.62:0/2838206343 learned_addr learned my addr 172.21.15.62:0/2838206343 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:03:25.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:25.518+0000 7f07fa000700 1 --2- 172.21.15.62:0/2838206343 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07fc103040 0x7f07fc198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:03:25.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:25.518+0000 7f07faa00700 1 -- 172.21.15.62:0/2838206343 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07fc103040 msgr2=0x7f07fc198670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:03:25.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:25.518+0000 7f07faa00700 1 --2- 172.21.15.62:0/2838206343 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07fc103040 0x7f07fc198670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:03:25.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:25.518+0000 7f07faa00700 1 -- 172.21.15.62:0/2838206343 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07ec0095f0 con 0x7f07fc1026a0 2024-04-20T22:03:25.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:25.518+0000 7f07fa000700 1 --2- 172.21.15.62:0/2838206343 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07fc103040 0x7f07fc198670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T22:03:25.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:25.518+0000 7f07faa00700 1 --2- 172.21.15.62:0/2838206343 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f07fc1026a0 0x7f07fc198130 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f07ec0092e0 tx=0x7f07ec00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:03:25.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:25.518+0000 7f07f3400700 1 -- 172.21.15.62:0/2838206343 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f07ec01d070 con 0x7f07fc1026a0 2024-04-20T22:03:25.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:25.519+0000 7f07f3400700 1 -- 172.21.15.62:0/2838206343 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f07ec005e00 con 0x7f07fc1026a0 2024-04-20T22:03:25.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:25.519+0000 7f07f3400700 1 -- 172.21.15.62:0/2838206343 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f07ec017400 con 0x7f07fc1026a0 2024-04-20T22:03:25.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:25.519+0000 7f0801c00700 1 -- 172.21.15.62:0/2838206343 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07fc19cd60 con 0x7f07fc1026a0 2024-04-20T22:03:25.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:25.519+0000 7f0801c00700 1 -- 172.21.15.62:0/2838206343 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07fc19d250 con 0x7f07fc1026a0 2024-04-20T22:03:25.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:25.520+0000 7f0801c00700 1 -- 172.21.15.62:0/2838206343 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07fc10a9b0 con 0x7f07fc1026a0 2024-04-20T22:03:25.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:25.521+0000 7f07f3400700 1 -- 172.21.15.62:0/2838206343 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f07ec00f460 con 0x7f07fc1026a0 2024-04-20T22:03:25.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:25.522+0000 7f07f3400700 1 --2- 172.21.15.62:0/2838206343 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f07e006ecc0 0x7f07e0071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:03:25.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:25.522+0000 7f07f3400700 1 -- 172.21.15.62:0/2838206343 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f07ec090330 con 0x7f07fc1026a0 2024-04-20T22:03:25.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:25.523+0000 7f07fa000700 1 --2- 172.21.15.62:0/2838206343 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f07e006ecc0 0x7f07e0071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:03:25.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:25.523+0000 7f07fa000700 1 --2- 172.21.15.62:0/2838206343 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f07e006ecc0 0x7f07e0071170 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f07e8005fd0 tx=0x7f07e8005ee0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:03:25.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:25.527+0000 7f07f3400700 1 -- 172.21.15.62:0/2838206343 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f07ec0590c0 con 0x7f07fc1026a0 2024-04-20T22:03:25.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:03:25 smithi189 ceph-mon[57022]: from='client.35516 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:03:25.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:03:25 smithi189 ceph-mon[57022]: from='client.35520 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:03:25.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:25.854+0000 7f0801c00700 1 -- 172.21.15.62:0/2838206343 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f07fc10a390 con 0x7f07fc1026a0 2024-04-20T22:03:25.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:25.854+0000 7f07f3400700 1 -- 172.21.15.62:0/2838206343 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7f07ec05c780 con 0x7f07fc1026a0 2024-04-20T22:03:25.855 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:03:25.855 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T22:03:25.855 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:03:25.855 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:03:25.855 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T22:03:25.855 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:03:25.855 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:03:25.855 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T22:03:25.855 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 6 2024-04-20T22:03:25.855 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:03:25.855 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T22:03:25.855 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 1, 2024-04-20T22:03:25.856 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 3 2024-04-20T22:03:25.856 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:03:25.856 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T22:03:25.856 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 1, 2024-04-20T22:03:25.856 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 13 2024-04-20T22:03:25.856 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T22:03:25.856 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:03:25.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:25.857+0000 7f0801c00700 1 -- 172.21.15.62:0/2838206343 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f07e006ecc0 msgr2=0x7f07e0071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:03:25.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:25.857+0000 7f0801c00700 1 --2- 172.21.15.62:0/2838206343 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f07e006ecc0 0x7f07e0071170 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f07e8005fd0 tx=0x7f07e8005ee0 comp rx=0 tx=0).stop 2024-04-20T22:03:25.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:25.857+0000 7f0801c00700 1 -- 172.21.15.62:0/2838206343 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f07fc1026a0 msgr2=0x7f07fc198130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:03:25.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:25.857+0000 7f0801c00700 1 --2- 172.21.15.62:0/2838206343 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f07fc1026a0 0x7f07fc198130 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f07ec0092e0 tx=0x7f07ec00be20 comp rx=0 tx=0).stop 2024-04-20T22:03:25.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:25.858+0000 7f0801c00700 1 -- 172.21.15.62:0/2838206343 shutdown_connections 2024-04-20T22:03:25.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:25.858+0000 7f0801c00700 1 --2- 172.21.15.62:0/2838206343 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f07e006ecc0 0x7f07e0071170 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:03:25.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:25.858+0000 7f0801c00700 1 --2- 172.21.15.62:0/2838206343 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f07fc1026a0 0x7f07fc198130 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:03:25.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:25.858+0000 7f0801c00700 1 --2- 172.21.15.62:0/2838206343 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07fc103040 0x7f07fc198670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:03:25.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:25.858+0000 7f0801c00700 1 -- 172.21.15.62:0/2838206343 >> 172.21.15.62:0/2838206343 conn(0x7f07fc0fdf40 msgr2=0x7f07fc0ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:03:25.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:25.858+0000 7f0801c00700 1 -- 172.21.15.62:0/2838206343 shutdown_connections 2024-04-20T22:03:25.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:25.858+0000 7f0801c00700 1 -- 172.21.15.62:0/2838206343 wait complete. 2024-04-20T22:03:26.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.018+0000 7fc039400700 1 -- 172.21.15.62:0/3982187912 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc034100e80 msgr2=0x7fc034104e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:03:26.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.018+0000 7fc039400700 1 --2- 172.21.15.62:0/3982187912 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc034100e80 0x7fc034104e50 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fc01c009910 tx=0x7fc01c009c20 comp rx=0 tx=0).stop 2024-04-20T22:03:26.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.018+0000 7fc039400700 1 -- 172.21.15.62:0/3982187912 shutdown_connections 2024-04-20T22:03:26.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.018+0000 7fc039400700 1 --2- 172.21.15.62:0/3982187912 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc034100e80 0x7fc034104e50 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:03:26.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.018+0000 7fc039400700 1 --2- 172.21.15.62:0/3982187912 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc0341004e0 0x7fc0341008b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:03:26.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.018+0000 7fc039400700 1 -- 172.21.15.62:0/3982187912 >> 172.21.15.62:0/3982187912 conn(0x7fc0340fbda0 msgr2=0x7fc0340fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:03:26.019 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:03:25 smithi062 ceph-mon[76322]: from='client.35516 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:03:26.019 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:03:25 smithi062 ceph-mon[76322]: from='client.35520 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:03:26.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.019+0000 7fc039400700 1 -- 172.21.15.62:0/3982187912 shutdown_connections 2024-04-20T22:03:26.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.019+0000 7fc039400700 1 -- 172.21.15.62:0/3982187912 wait complete. 2024-04-20T22:03:26.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.020+0000 7fc039400700 1 Processor -- start 2024-04-20T22:03:26.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.020+0000 7fc039400700 1 -- start start 2024-04-20T22:03:26.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.020+0000 7fc039400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc0341004e0 0x7fc0341981b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:03:26.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.021+0000 7fc039400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc034100e80 0x7fc0341986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:03:26.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.021+0000 7fc039400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc034198dd0 con 0x7fc0341004e0 2024-04-20T22:03:26.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.021+0000 7fc039400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc03419cb60 con 0x7fc034100e80 2024-04-20T22:03:26.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.021+0000 7fc032800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc0341004e0 0x7fc0341981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:03:26.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.021+0000 7fc032800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc0341004e0 0x7fc0341981b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47430/0 (socket says 172.21.15.62:47430) 2024-04-20T22:03:26.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.021+0000 7fc032800700 1 -- 172.21.15.62:0/4292152745 learned_addr learned my addr 172.21.15.62:0/4292152745 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:03:26.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.021+0000 7fc02be00700 1 --2- 172.21.15.62:0/4292152745 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc034100e80 0x7fc0341986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:03:26.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.021+0000 7fc032800700 1 -- 172.21.15.62:0/4292152745 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc034100e80 msgr2=0x7fc0341986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:03:26.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.021+0000 7fc032800700 1 --2- 172.21.15.62:0/4292152745 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc034100e80 0x7fc0341986f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:03:26.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.021+0000 7fc032800700 1 -- 172.21.15.62:0/4292152745 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc01c0095f0 con 0x7fc0341004e0 2024-04-20T22:03:26.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.022+0000 7fc032800700 1 --2- 172.21.15.62:0/4292152745 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc0341004e0 0x7fc0341981b0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fc02400b4b0 tx=0x7fc02400b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:03:26.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.022+0000 7fc02b400700 1 -- 172.21.15.62:0/4292152745 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc024011800 con 0x7fc0341004e0 2024-04-20T22:03:26.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.022+0000 7fc02b400700 1 -- 172.21.15.62:0/4292152745 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc024011e20 con 0x7fc0341004e0 2024-04-20T22:03:26.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.022+0000 7fc02b400700 1 -- 172.21.15.62:0/4292152745 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc02400f2e0 con 0x7fc0341004e0 2024-04-20T22:03:26.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.022+0000 7fc039400700 1 -- 172.21.15.62:0/4292152745 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc03419ce40 con 0x7fc0341004e0 2024-04-20T22:03:26.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.023+0000 7fc039400700 1 -- 172.21.15.62:0/4292152745 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc03419d390 con 0x7fc0341004e0 2024-04-20T22:03:26.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.024+0000 7fc039400700 1 -- 172.21.15.62:0/4292152745 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0341045d0 con 0x7fc0341004e0 2024-04-20T22:03:26.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.025+0000 7fc02b400700 1 -- 172.21.15.62:0/4292152745 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fc024010380 con 0x7fc0341004e0 2024-04-20T22:03:26.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.026+0000 7fc02b400700 1 --2- 172.21.15.62:0/4292152745 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc01806ecc0 0x7fc018071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:03:26.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.026+0000 7fc02b400700 1 -- 172.21.15.62:0/4292152745 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fc02408e620 con 0x7fc0341004e0 2024-04-20T22:03:26.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.026+0000 7fc02be00700 1 --2- 172.21.15.62:0/4292152745 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc01806ecc0 0x7fc018071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:03:26.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.027+0000 7fc02be00700 1 --2- 172.21.15.62:0/4292152745 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc01806ecc0 0x7fc018071170 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fc01c006010 tx=0x7fc01c005960 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:03:26.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.031+0000 7fc02b400700 1 -- 172.21.15.62:0/4292152745 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fc024057450 con 0x7fc0341004e0 2024-04-20T22:03:26.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.302+0000 7fc039400700 1 -- 172.21.15.62:0/4292152745 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fc03404f2a0 con 0x7fc0341004e0 2024-04-20T22:03:26.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.302+0000 7fc02b400700 1 -- 172.21.15.62:0/4292152745 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 27 v27) v1 ==== 76+0+1888 (secure 0 0 0) 0x7fc024015770 con 0x7fc0341004e0 2024-04-20T22:03:26.303 INFO:teuthology.orchestra.run.smithi062.stdout:e27 2024-04-20T22:03:26.303 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T22:03:26.303 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:03:26.303 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T22:03:26.303 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:03:26.303 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T22:03:26.303 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T22:03:26.303 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 20 2024-04-20T22:03:26.304 INFO:teuthology.orchestra.run.smithi062.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-20T22:03:26.304 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T22:03:26.304 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T21:42:01.559061+0000 2024-04-20T22:03:26.304 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T22:03:26.304 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T22:03:26.304 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T22:03:26.304 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T22:03:26.304 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T22:03:26.304 INFO:teuthology.orchestra.run.smithi062.stdout:max_xattr_size 65536 2024-04-20T22:03:26.304 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T22:03:26.304 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T22:03:26.304 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 81 2024-04-20T22:03:26.304 INFO:teuthology.orchestra.run.smithi062.stdout:snap_rstat 1 2024-04-20T22:03:26.304 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:03:26.304 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 2 2024-04-20T22:03:26.304 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1 2024-04-20T22:03:26.305 INFO:teuthology.orchestra.run.smithi062.stdout:up {} 2024-04-20T22:03:26.305 INFO:teuthology.orchestra.run.smithi062.stdout:failed 0,1 2024-04-20T22:03:26.305 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T22:03:26.305 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T22:03:26.305 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T22:03:26.305 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T22:03:26.305 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data enabled 2024-04-20T22:03:26.305 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T22:03:26.305 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T22:03:26.305 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 1 2024-04-20T22:03:26.305 INFO:teuthology.orchestra.run.smithi062.stdout:qdb_cluster leader: 0 members: 2024-04-20T22:03:26.305 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:03:26.305 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:03:26.305 INFO:teuthology.orchestra.run.smithi062.stdout:Standby daemons: 2024-04-20T22:03:26.306 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:03:26.306 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ryqblb{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6826/4113916205,v1:172.21.15.62:6827/4113916205] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:03:26.306 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ozizwi{-1:34476} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6828/3801975165,v1:172.21.15.62:6829/3801975165] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:03:26.306 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.xbgspw{-1:44285} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6826/3996040526,v1:172.21.15.189:6827/3996040526] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T22:03:26.306 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.dvimer{-1:44727} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6824/3576699232,v1:172.21.15.189:6825/3576699232] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:03:26.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.305+0000 7fc039400700 1 -- 172.21.15.62:0/4292152745 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc01806ecc0 msgr2=0x7fc018071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:03:26.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.305+0000 7fc039400700 1 --2- 172.21.15.62:0/4292152745 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc01806ecc0 0x7fc018071170 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fc01c006010 tx=0x7fc01c005960 comp rx=0 tx=0).stop 2024-04-20T22:03:26.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.305+0000 7fc039400700 1 -- 172.21.15.62:0/4292152745 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc0341004e0 msgr2=0x7fc0341981b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:03:26.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.305+0000 7fc039400700 1 --2- 172.21.15.62:0/4292152745 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc0341004e0 0x7fc0341981b0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fc02400b4b0 tx=0x7fc02400b870 comp rx=0 tx=0).stop 2024-04-20T22:03:26.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.306+0000 7fc039400700 1 -- 172.21.15.62:0/4292152745 shutdown_connections 2024-04-20T22:03:26.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.306+0000 7fc039400700 1 --2- 172.21.15.62:0/4292152745 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc01806ecc0 0x7fc018071170 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:03:26.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.306+0000 7fc039400700 1 --2- 172.21.15.62:0/4292152745 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc0341004e0 0x7fc0341981b0 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:03:26.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.306+0000 7fc039400700 1 --2- 172.21.15.62:0/4292152745 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc034100e80 0x7fc0341986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:03:26.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.306+0000 7fc039400700 1 -- 172.21.15.62:0/4292152745 >> 172.21.15.62:0/4292152745 conn(0x7fc0340fbda0 msgr2=0x7fc0340fd520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:03:26.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.306+0000 7fc039400700 1 -- 172.21.15.62:0/4292152745 shutdown_connections 2024-04-20T22:03:26.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.306+0000 7fc039400700 1 -- 172.21.15.62:0/4292152745 wait complete. 2024-04-20T22:03:26.307 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 27 2024-04-20T22:03:26.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.469+0000 7fcbdaa00700 1 -- 172.21.15.62:0/3617906618 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcbd4100e80 msgr2=0x7fcbd4104e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:03:26.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.469+0000 7fcbdaa00700 1 --2- 172.21.15.62:0/3617906618 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcbd4100e80 0x7fcbd4104e50 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fcbbc009910 tx=0x7fcbbc009c20 comp rx=0 tx=0).stop 2024-04-20T22:03:26.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.469+0000 7fcbdaa00700 1 -- 172.21.15.62:0/3617906618 shutdown_connections 2024-04-20T22:03:26.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.469+0000 7fcbdaa00700 1 --2- 172.21.15.62:0/3617906618 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcbd4100e80 0x7fcbd4104e50 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:03:26.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.469+0000 7fcbdaa00700 1 --2- 172.21.15.62:0/3617906618 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcbd41004e0 0x7fcbd41008b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:03:26.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.469+0000 7fcbdaa00700 1 -- 172.21.15.62:0/3617906618 >> 172.21.15.62:0/3617906618 conn(0x7fcbd40fbda0 msgr2=0x7fcbd40fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:03:26.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.469+0000 7fcbdaa00700 1 -- 172.21.15.62:0/3617906618 shutdown_connections 2024-04-20T22:03:26.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.469+0000 7fcbdaa00700 1 -- 172.21.15.62:0/3617906618 wait complete. 2024-04-20T22:03:26.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.470+0000 7fcbdaa00700 1 Processor -- start 2024-04-20T22:03:26.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.471+0000 7fcbdaa00700 1 -- start start 2024-04-20T22:03:26.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.471+0000 7fcbdaa00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcbd41004e0 0x7fcbd4073050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:03:26.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.471+0000 7fcbdaa00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcbd4100e80 0x7fcbd4073590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:03:26.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.471+0000 7fcbdaa00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbd4074d70 con 0x7fcbd4100e80 2024-04-20T22:03:26.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.471+0000 7fcbdaa00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbd4074ee0 con 0x7fcbd41004e0 2024-04-20T22:03:26.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.471+0000 7fcbcb400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcbd4100e80 0x7fcbd4073590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:03:26.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.471+0000 7fcbcb400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcbd4100e80 0x7fcbd4073590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47462/0 (socket says 172.21.15.62:47462) 2024-04-20T22:03:26.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.471+0000 7fcbcb400700 1 -- 172.21.15.62:0/205204209 learned_addr learned my addr 172.21.15.62:0/205204209 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:03:26.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.472+0000 7fcbd3e00700 1 --2- 172.21.15.62:0/205204209 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcbd41004e0 0x7fcbd4073050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:03:26.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.472+0000 7fcbcb400700 1 -- 172.21.15.62:0/205204209 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcbd41004e0 msgr2=0x7fcbd4073050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:03:26.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.472+0000 7fcbcb400700 1 --2- 172.21.15.62:0/205204209 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcbd41004e0 0x7fcbd4073050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:03:26.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.472+0000 7fcbcb400700 1 -- 172.21.15.62:0/205204209 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbbc0095f0 con 0x7fcbd4100e80 2024-04-20T22:03:26.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.472+0000 7fcbcb400700 1 --2- 172.21.15.62:0/205204209 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcbd4100e80 0x7fcbd4073590 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fcbbc005f50 tx=0x7fcbbc00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:03:26.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.472+0000 7fcbd1600700 1 -- 172.21.15.62:0/205204209 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcbbc01d070 con 0x7fcbd4100e80 2024-04-20T22:03:26.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.472+0000 7fcbd1600700 1 -- 172.21.15.62:0/205204209 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcbbc00f460 con 0x7fcbd4100e80 2024-04-20T22:03:26.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.472+0000 7fcbd1600700 1 -- 172.21.15.62:0/205204209 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcbbc017630 con 0x7fcbd4100e80 2024-04-20T22:03:26.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.472+0000 7fcbdaa00700 1 -- 172.21.15.62:0/205204209 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbd4073ad0 con 0x7fcbd4100e80 2024-04-20T22:03:26.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.473+0000 7fcbdaa00700 1 -- 172.21.15.62:0/205204209 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbd41a68d0 con 0x7fcbd4100e80 2024-04-20T22:03:26.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.474+0000 7fcbdaa00700 1 -- 172.21.15.62:0/205204209 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbd4108850 con 0x7fcbd4100e80 2024-04-20T22:03:26.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.475+0000 7fcbd1600700 1 -- 172.21.15.62:0/205204209 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fcbbc00f5d0 con 0x7fcbd4100e80 2024-04-20T22:03:26.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.476+0000 7fcbd1600700 1 --2- 172.21.15.62:0/205204209 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fcbb806ecc0 0x7fcbb8071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:03:26.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.476+0000 7fcbd1600700 1 -- 172.21.15.62:0/205204209 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fcbbc090420 con 0x7fcbd4100e80 2024-04-20T22:03:26.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.476+0000 7fcbd3e00700 1 --2- 172.21.15.62:0/205204209 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fcbb806ecc0 0x7fcbb8071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:03:26.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.477+0000 7fcbd3e00700 1 --2- 172.21.15.62:0/205204209 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fcbb806ecc0 0x7fcbb8071170 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fcbc400d260 tx=0x7fcbc4009040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:03:26.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.480+0000 7fcbd1600700 1 -- 172.21.15.62:0/205204209 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fcbbc0591b0 con 0x7fcbd4100e80 2024-04-20T22:03:26.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.739+0000 7fcbdaa00700 1 -- 172.21.15.62:0/205204209 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fcbd4061190 con 0x7fcbb806ecc0 2024-04-20T22:03:26.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.741+0000 7fcbd1600700 1 -- 172.21.15.62:0/205204209 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fcbd4061190 con 0x7fcbb806ecc0 2024-04-20T22:03:26.741 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:03:26.741 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T22:03:26.741 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T22:03:26.741 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-20T22:03:26.741 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T22:03:26.741 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr", 2024-04-20T22:03:26.741 INFO:teuthology.orchestra.run.smithi062.stdout: "crash", 2024-04-20T22:03:26.741 INFO:teuthology.orchestra.run.smithi062.stdout: "mon", 2024-04-20T22:03:26.742 INFO:teuthology.orchestra.run.smithi062.stdout: "osd" 2024-04-20T22:03:26.742 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T22:03:26.742 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "15/23 daemons upgraded", 2024-04-20T22:03:26.742 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading mds daemons", 2024-04-20T22:03:26.742 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T22:03:26.742 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:03:26.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.744+0000 7fcbdaa00700 1 -- 172.21.15.62:0/205204209 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fcbb806ecc0 msgr2=0x7fcbb8071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:03:26.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.744+0000 7fcbdaa00700 1 --2- 172.21.15.62:0/205204209 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fcbb806ecc0 0x7fcbb8071170 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fcbc400d260 tx=0x7fcbc4009040 comp rx=0 tx=0).stop 2024-04-20T22:03:26.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.744+0000 7fcbdaa00700 1 -- 172.21.15.62:0/205204209 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcbd4100e80 msgr2=0x7fcbd4073590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:03:26.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.744+0000 7fcbdaa00700 1 --2- 172.21.15.62:0/205204209 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcbd4100e80 0x7fcbd4073590 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fcbbc005f50 tx=0x7fcbbc00be20 comp rx=0 tx=0).stop 2024-04-20T22:03:26.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.745+0000 7fcbdaa00700 1 -- 172.21.15.62:0/205204209 shutdown_connections 2024-04-20T22:03:26.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.745+0000 7fcbdaa00700 1 --2- 172.21.15.62:0/205204209 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fcbb806ecc0 0x7fcbb8071170 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:03:26.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.745+0000 7fcbdaa00700 1 --2- 172.21.15.62:0/205204209 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcbd41004e0 0x7fcbd4073050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:03:26.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.745+0000 7fcbdaa00700 1 --2- 172.21.15.62:0/205204209 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcbd4100e80 0x7fcbd4073590 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:03:26.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.745+0000 7fcbdaa00700 1 -- 172.21.15.62:0/205204209 >> 172.21.15.62:0/205204209 conn(0x7fcbd40fbda0 msgr2=0x7fcbd40fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:03:26.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.745+0000 7fcbdaa00700 1 -- 172.21.15.62:0/205204209 shutdown_connections 2024-04-20T22:03:26.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.745+0000 7fcbdaa00700 1 -- 172.21.15.62:0/205204209 wait complete. 2024-04-20T22:03:26.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.903+0000 7f0b60a00700 1 -- 172.21.15.62:0/2415493994 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0b5c106b80 msgr2=0x7f0b5c106ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:03:26.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.903+0000 7f0b60a00700 1 --2- 172.21.15.62:0/2415493994 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0b5c106b80 0x7f0b5c106ff0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f0b48009910 tx=0x7f0b48009c20 comp rx=0 tx=0).stop 2024-04-20T22:03:26.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.903+0000 7f0b60a00700 1 -- 172.21.15.62:0/2415493994 shutdown_connections 2024-04-20T22:03:26.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.903+0000 7f0b60a00700 1 --2- 172.21.15.62:0/2415493994 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0b5c106b80 0x7f0b5c106ff0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:03:26.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.903+0000 7f0b60a00700 1 --2- 172.21.15.62:0/2415493994 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b5c10c8f0 0x7f0b5c10ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:03:26.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.903+0000 7f0b60a00700 1 -- 172.21.15.62:0/2415493994 >> 172.21.15.62:0/2415493994 conn(0x7f0b5c076060 msgr2=0x7f0b5c078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:03:26.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.903+0000 7f0b60a00700 1 -- 172.21.15.62:0/2415493994 shutdown_connections 2024-04-20T22:03:26.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.903+0000 7f0b60a00700 1 -- 172.21.15.62:0/2415493994 wait complete. 2024-04-20T22:03:26.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.904+0000 7f0b60a00700 1 Processor -- start 2024-04-20T22:03:26.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.904+0000 7f0b60a00700 1 -- start start 2024-04-20T22:03:26.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.905+0000 7f0b60a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b5c106b80 0x7f0b5c19c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:03:26.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.905+0000 7f0b60a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0b5c10c8f0 0x7f0b5c19c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:03:26.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.905+0000 7f0b60a00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b5c19d060 con 0x7f0b5c10c8f0 2024-04-20T22:03:26.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.905+0000 7f0b60a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b5c1a0df0 con 0x7f0b5c106b80 2024-04-20T22:03:26.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.905+0000 7f0b59400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0b5c10c8f0 0x7f0b5c19c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:03:26.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.905+0000 7f0b59400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0b5c10c8f0 0x7f0b5c19c980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47476/0 (socket says 172.21.15.62:47476) 2024-04-20T22:03:26.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.905+0000 7f0b59400700 1 -- 172.21.15.62:0/2186273965 learned_addr learned my addr 172.21.15.62:0/2186273965 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:03:26.905 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:03:26 smithi062 ceph-mon[76322]: from='client.35524 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:03:26.905 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:03:26 smithi062 ceph-mon[76322]: pgmap v845: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:03:26.905 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:03:26 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/2838206343' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:03:26.905 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:03:26 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/4292152745' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:03:26.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.905+0000 7f0b59e00700 1 --2- 172.21.15.62:0/2186273965 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b5c106b80 0x7f0b5c19c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:03:26.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.905+0000 7f0b59400700 1 -- 172.21.15.62:0/2186273965 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b5c106b80 msgr2=0x7f0b5c19c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:03:26.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.905+0000 7f0b59400700 1 --2- 172.21.15.62:0/2186273965 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b5c106b80 0x7f0b5c19c440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:03:26.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.905+0000 7f0b59400700 1 -- 172.21.15.62:0/2186273965 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b480095f0 con 0x7f0b5c10c8f0 2024-04-20T22:03:26.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.906+0000 7f0b59400700 1 --2- 172.21.15.62:0/2186273965 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0b5c10c8f0 0x7f0b5c19c980 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f0b48006010 tx=0x7f0b4800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:03:26.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.906+0000 7f0b52a00700 1 -- 172.21.15.62:0/2186273965 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0b4801d070 con 0x7f0b5c10c8f0 2024-04-20T22:03:26.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.906+0000 7f0b52a00700 1 -- 172.21.15.62:0/2186273965 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0b48004fb0 con 0x7f0b5c10c8f0 2024-04-20T22:03:26.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.906+0000 7f0b52a00700 1 -- 172.21.15.62:0/2186273965 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0b4800f690 con 0x7f0b5c10c8f0 2024-04-20T22:03:26.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.906+0000 7f0b60a00700 1 -- 172.21.15.62:0/2186273965 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b5c1a1070 con 0x7f0b5c10c8f0 2024-04-20T22:03:26.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.906+0000 7f0b60a00700 1 -- 172.21.15.62:0/2186273965 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b5c1a1560 con 0x7f0b5c10c8f0 2024-04-20T22:03:26.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.908+0000 7f0b60a00700 1 -- 172.21.15.62:0/2186273965 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b5c10ecc0 con 0x7f0b5c10c8f0 2024-04-20T22:03:26.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.909+0000 7f0b52a00700 1 -- 172.21.15.62:0/2186273965 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f0b480228a0 con 0x7f0b5c10c8f0 2024-04-20T22:03:26.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.909+0000 7f0b52a00700 1 --2- 172.21.15.62:0/2186273965 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f0b4006ecc0 0x7f0b40071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:03:26.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.910+0000 7f0b52a00700 1 -- 172.21.15.62:0/2186273965 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f0b48090770 con 0x7f0b5c10c8f0 2024-04-20T22:03:26.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.910+0000 7f0b59e00700 1 --2- 172.21.15.62:0/2186273965 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f0b4006ecc0 0x7f0b40071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:03:26.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.911+0000 7f0b59e00700 1 --2- 172.21.15.62:0/2186273965 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f0b4006ecc0 0x7f0b40071170 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f0b4c005fd0 tx=0x7f0b4c00a560 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:03:26.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:26.916+0000 7f0b52a00700 1 -- 172.21.15.62:0/2186273965 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f0b48059480 con 0x7f0b5c10c8f0 2024-04-20T22:03:26.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:03:26 smithi189 ceph-mon[57022]: from='client.35524 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:03:26.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:03:26 smithi189 ceph-mon[57022]: pgmap v845: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:03:26.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:03:26 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/2838206343' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:03:26.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:03:26 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/4292152745' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:03:27.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:27.240+0000 7f0b60a00700 1 -- 172.21.15.62:0/2186273965 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f0b5c19d840 con 0x7f0b5c10c8f0 2024-04-20T22:03:27.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:27.241+0000 7f0b52a00700 1 -- 172.21.15.62:0/2186273965 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7f0b4805cb40 con 0x7f0b5c10c8f0 2024-04-20T22:03:27.242 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-20T22:03:27.242 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-20T22:03:27.242 INFO:teuthology.orchestra.run.smithi062.stdout: host smithi062 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T22:03:27.242 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-20T22:03:27.242 INFO:teuthology.orchestra.run.smithi062.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi062 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpq7y7oa_r:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:03:27.242 INFO:teuthology.orchestra.run.smithi062.stdout:Traceback (most recent call last): 2024-04-20T22:03:27.242 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-20T22:03:27.242 INFO:teuthology.orchestra.run.smithi062.stdout: return _run_code(code, main_globals, None, 2024-04-20T22:03:27.242 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-20T22:03:27.242 INFO:teuthology.orchestra.run.smithi062.stdout: exec(code, run_globals) 2024-04-20T22:03:27.242 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5618, in 2024-04-20T22:03:27.242 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5606, in main 2024-04-20T22:03:27.242 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 408, in _infer_config 2024-04-20T22:03:27.243 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 323, in _infer_fsid 2024-04-20T22:03:27.243 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 436, in _infer_image 2024-04-20T22:03:27.243 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 310, in _validate_fsid 2024-04-20T22:03:27.243 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 3359, in command_ceph_volume 2024-04-20T22:03:27.243 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-20T22:03:27.243 INFO:teuthology.orchestra.run.smithi062.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi062 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpq7y7oa_r:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:03:27.243 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-20T22:03:27.243 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is degraded 2024-04-20T22:03:27.243 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T22:03:27.243 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T22:03:27.243 INFO:teuthology.orchestra.run.smithi062.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-20T22:03:27.243 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-20T22:03:27.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:27.244+0000 7f0b60a00700 1 -- 172.21.15.62:0/2186273965 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f0b4006ecc0 msgr2=0x7f0b40071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:03:27.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:27.244+0000 7f0b60a00700 1 --2- 172.21.15.62:0/2186273965 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f0b4006ecc0 0x7f0b40071170 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f0b4c005fd0 tx=0x7f0b4c00a560 comp rx=0 tx=0).stop 2024-04-20T22:03:27.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:27.244+0000 7f0b60a00700 1 -- 172.21.15.62:0/2186273965 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0b5c10c8f0 msgr2=0x7f0b5c19c980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:03:27.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:27.244+0000 7f0b60a00700 1 --2- 172.21.15.62:0/2186273965 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0b5c10c8f0 0x7f0b5c19c980 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f0b48006010 tx=0x7f0b4800be20 comp rx=0 tx=0).stop 2024-04-20T22:03:27.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:27.245+0000 7f0b60a00700 1 -- 172.21.15.62:0/2186273965 shutdown_connections 2024-04-20T22:03:27.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:27.245+0000 7f0b60a00700 1 --2- 172.21.15.62:0/2186273965 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f0b4006ecc0 0x7f0b40071170 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:03:27.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:27.245+0000 7f0b60a00700 1 --2- 172.21.15.62:0/2186273965 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b5c106b80 0x7f0b5c19c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:03:27.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:27.245+0000 7f0b60a00700 1 --2- 172.21.15.62:0/2186273965 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0b5c10c8f0 0x7f0b5c19c980 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:03:27.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:27.245+0000 7f0b60a00700 1 -- 172.21.15.62:0/2186273965 >> 172.21.15.62:0/2186273965 conn(0x7f0b5c076060 msgr2=0x7f0b5c077ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:03:27.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:27.245+0000 7f0b60a00700 1 -- 172.21.15.62:0/2186273965 shutdown_connections 2024-04-20T22:03:27.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:27.245+0000 7f0b60a00700 1 -- 172.21.15.62:0/2186273965 wait complete. 2024-04-20T22:03:27.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:03:27 smithi189 ceph-mon[57022]: from='client.35536 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:03:27.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:03:27 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/2186273965' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:03:28.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:03:27 smithi062 ceph-mon[76322]: from='client.35536 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:03:28.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:03:27 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/2186273965' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:03:28.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:03:28 smithi189 ceph-mon[57022]: pgmap v846: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:03:29.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:03:28 smithi062 ceph-mon[76322]: pgmap v846: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:03:30.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:03:30 smithi189 ceph-mon[57022]: pgmap v847: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:03:31.038 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:03:30 smithi062 ceph-mon[76322]: pgmap v847: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:03:32.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:03:32 smithi189 ceph-mon[57022]: pgmap v848: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:03:33.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:03:32 smithi062 ceph-mon[76322]: pgmap v848: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:03:34.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:03:34 smithi189 ceph-mon[57022]: pgmap v849: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:03:34.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:03:34 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:03:35.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:03:34 smithi062 ceph-mon[76322]: pgmap v849: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:03:35.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:03:34 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:03:36.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:03:36 smithi189 ceph-mon[57022]: pgmap v850: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:03:37.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:03:36 smithi062 ceph-mon[76322]: pgmap v850: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:03:38.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:03:38 smithi189 ceph-mon[57022]: pgmap v851: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:03:39.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:03:38 smithi062 ceph-mon[76322]: pgmap v851: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:03:40.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:03:40 smithi189 ceph-mon[57022]: pgmap v852: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:03:41.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:03:40 smithi062 ceph-mon[76322]: pgmap v852: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:03:42.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:03:42 smithi189 ceph-mon[57022]: pgmap v853: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:03:43.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:03:42 smithi062 ceph-mon[76322]: pgmap v853: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:03:44.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:03:44 smithi189 ceph-mon[57022]: pgmap v854: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:03:45.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:03:44 smithi062 ceph-mon[76322]: pgmap v854: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:03:47.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:03:46 smithi062 ceph-mon[76322]: pgmap v855: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:03:47.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:03:46 smithi189 ceph-mon[57022]: pgmap v855: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:03:49.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:03:48 smithi062 ceph-mon[76322]: pgmap v856: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:03:49.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:03:48 smithi189 ceph-mon[57022]: pgmap v856: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:03:50.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:03:49 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:03:50.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:03:49 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:03:51.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:03:50 smithi062 ceph-mon[76322]: pgmap v857: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:03:51.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:03:50 smithi189 ceph-mon[57022]: pgmap v857: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:03:53.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:03:52 smithi062 ceph-mon[76322]: pgmap v858: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:03:53.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:03:52 smithi189 ceph-mon[57022]: pgmap v858: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:03:55.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:03:54 smithi062 ceph-mon[76322]: pgmap v859: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:03:55.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:03:54 smithi189 ceph-mon[57022]: pgmap v859: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:03:57.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:03:56 smithi062 ceph-mon[76322]: pgmap v860: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:03:57.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:03:56 smithi189 ceph-mon[57022]: pgmap v860: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:03:57.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:57.423+0000 7fe991400700 1 -- 172.21.15.62:0/978518057 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe98c1004e0 msgr2=0x7fe98c1008b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:03:57.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:57.423+0000 7fe991400700 1 --2- 172.21.15.62:0/978518057 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe98c1004e0 0x7fe98c1008b0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fe97c009910 tx=0x7fe97c009c20 comp rx=0 tx=0).stop 2024-04-20T22:03:57.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:57.423+0000 7fe991400700 1 -- 172.21.15.62:0/978518057 shutdown_connections 2024-04-20T22:03:57.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:57.423+0000 7fe991400700 1 --2- 172.21.15.62:0/978518057 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe98c100e80 0x7fe98c104e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:03:57.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:57.423+0000 7fe991400700 1 --2- 172.21.15.62:0/978518057 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe98c1004e0 0x7fe98c1008b0 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:03:57.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:57.424+0000 7fe991400700 1 -- 172.21.15.62:0/978518057 >> 172.21.15.62:0/978518057 conn(0x7fe98c0fbda0 msgr2=0x7fe98c0fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:03:57.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:57.424+0000 7fe991400700 1 -- 172.21.15.62:0/978518057 shutdown_connections 2024-04-20T22:03:57.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:57.424+0000 7fe991400700 1 -- 172.21.15.62:0/978518057 wait complete. 2024-04-20T22:03:57.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:57.425+0000 7fe991400700 1 Processor -- start 2024-04-20T22:03:57.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:57.425+0000 7fe991400700 1 -- start start 2024-04-20T22:03:57.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:57.425+0000 7fe991400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe98c1004e0 0x7fe98c073050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:03:57.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:57.425+0000 7fe991400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe98c100e80 0x7fe98c073590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:03:57.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:57.425+0000 7fe991400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe98c073ad0 con 0x7fe98c1004e0 2024-04-20T22:03:57.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:57.425+0000 7fe991400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe98c073c10 con 0x7fe98c100e80 2024-04-20T22:03:57.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:57.426+0000 7fe98a800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe98c1004e0 0x7fe98c073050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:03:57.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:57.426+0000 7fe98a800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe98c1004e0 0x7fe98c073050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44510/0 (socket says 172.21.15.62:44510) 2024-04-20T22:03:57.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:57.426+0000 7fe98a800700 1 -- 172.21.15.62:0/4150430899 learned_addr learned my addr 172.21.15.62:0/4150430899 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:03:57.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:57.426+0000 7fe98a800700 1 -- 172.21.15.62:0/4150430899 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe98c100e80 msgr2=0x7fe98c073590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:03:57.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:57.426+0000 7fe98a800700 1 --2- 172.21.15.62:0/4150430899 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe98c100e80 0x7fe98c073590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:03:57.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:57.426+0000 7fe98a800700 1 -- 172.21.15.62:0/4150430899 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe97c0095f0 con 0x7fe98c1004e0 2024-04-20T22:03:57.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:57.426+0000 7fe98a800700 1 --2- 172.21.15.62:0/4150430899 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe98c1004e0 0x7fe98c073050 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fe97c006010 tx=0x7fe97c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:03:57.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:57.427+0000 7fe983400700 1 -- 172.21.15.62:0/4150430899 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe97c01d070 con 0x7fe98c1004e0 2024-04-20T22:03:57.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:57.427+0000 7fe983400700 1 -- 172.21.15.62:0/4150430899 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe97c004fb0 con 0x7fe98c1004e0 2024-04-20T22:03:57.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:57.427+0000 7fe983400700 1 -- 172.21.15.62:0/4150430899 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe97c022ea0 con 0x7fe98c1004e0 2024-04-20T22:03:57.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:57.427+0000 7fe991400700 1 -- 172.21.15.62:0/4150430899 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe98c073e90 con 0x7fe98c1004e0 2024-04-20T22:03:57.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:57.427+0000 7fe991400700 1 -- 172.21.15.62:0/4150430899 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe98c1a2430 con 0x7fe98c1004e0 2024-04-20T22:03:57.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:57.428+0000 7fe991400700 1 -- 172.21.15.62:0/4150430899 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe98c1087f0 con 0x7fe98c1004e0 2024-04-20T22:03:57.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:57.429+0000 7fe983400700 1 -- 172.21.15.62:0/4150430899 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe97c022470 con 0x7fe98c1004e0 2024-04-20T22:03:57.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:57.430+0000 7fe983400700 1 --2- 172.21.15.62:0/4150430899 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe97006ecc0 0x7fe970071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:03:57.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:57.430+0000 7fe983400700 1 -- 172.21.15.62:0/4150430899 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fe97c0903c0 con 0x7fe98c1004e0 2024-04-20T22:03:57.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:57.430+0000 7fe983e00700 1 --2- 172.21.15.62:0/4150430899 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe97006ecc0 0x7fe970071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:03:57.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:57.431+0000 7fe983e00700 1 --2- 172.21.15.62:0/4150430899 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe97006ecc0 0x7fe970071170 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fe98c074840 tx=0x7fe974005c80 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:03:57.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:57.435+0000 7fe983400700 1 -- 172.21.15.62:0/4150430899 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fe97c059200 con 0x7fe98c1004e0 2024-04-20T22:03:57.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:57.699+0000 7fe991400700 1 -- 172.21.15.62:0/4150430899 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe98c061190 con 0x7fe97006ecc0 2024-04-20T22:03:57.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:57.702+0000 7fe983400700 1 -- 172.21.15.62:0/4150430899 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fe98c061190 con 0x7fe97006ecc0 2024-04-20T22:03:57.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:57.705+0000 7fe991400700 1 -- 172.21.15.62:0/4150430899 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe97006ecc0 msgr2=0x7fe970071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:03:57.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:57.705+0000 7fe991400700 1 --2- 172.21.15.62:0/4150430899 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe97006ecc0 0x7fe970071170 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fe98c074840 tx=0x7fe974005c80 comp rx=0 tx=0).stop 2024-04-20T22:03:57.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:57.705+0000 7fe991400700 1 -- 172.21.15.62:0/4150430899 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe98c1004e0 msgr2=0x7fe98c073050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:03:57.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:57.705+0000 7fe991400700 1 --2- 172.21.15.62:0/4150430899 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe98c1004e0 0x7fe98c073050 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fe97c006010 tx=0x7fe97c00be20 comp rx=0 tx=0).stop 2024-04-20T22:03:57.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:57.705+0000 7fe991400700 1 -- 172.21.15.62:0/4150430899 shutdown_connections 2024-04-20T22:03:57.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:57.705+0000 7fe991400700 1 --2- 172.21.15.62:0/4150430899 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe97006ecc0 0x7fe970071170 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:03:57.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:57.705+0000 7fe991400700 1 --2- 172.21.15.62:0/4150430899 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe98c1004e0 0x7fe98c073050 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:03:57.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:57.705+0000 7fe991400700 1 --2- 172.21.15.62:0/4150430899 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe98c100e80 0x7fe98c073590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:03:57.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:57.705+0000 7fe991400700 1 -- 172.21.15.62:0/4150430899 >> 172.21.15.62:0/4150430899 conn(0x7fe98c0fbda0 msgr2=0x7fe98c0fd3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:03:57.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:57.705+0000 7fe991400700 1 -- 172.21.15.62:0/4150430899 shutdown_connections 2024-04-20T22:03:57.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:57.705+0000 7fe991400700 1 -- 172.21.15.62:0/4150430899 wait complete. 2024-04-20T22:03:57.721 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T22:03:57.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:57.875+0000 7f308e200700 1 -- 172.21.15.62:0/2737012989 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f30881004a0 msgr2=0x7f3088100870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:03:57.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:57.875+0000 7f308e200700 1 --2- 172.21.15.62:0/2737012989 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f30881004a0 0x7f3088100870 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f3078009910 tx=0x7f3078009c20 comp rx=0 tx=0).stop 2024-04-20T22:03:57.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:57.875+0000 7f308e200700 1 -- 172.21.15.62:0/2737012989 shutdown_connections 2024-04-20T22:03:57.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:57.875+0000 7f308e200700 1 --2- 172.21.15.62:0/2737012989 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3088100e40 0x7f3088104e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:03:57.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:57.875+0000 7f308e200700 1 --2- 172.21.15.62:0/2737012989 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f30881004a0 0x7f3088100870 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:03:57.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:57.875+0000 7f308e200700 1 -- 172.21.15.62:0/2737012989 >> 172.21.15.62:0/2737012989 conn(0x7f30880fbd40 msgr2=0x7f30880fe150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:03:57.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:57.875+0000 7f308e200700 1 -- 172.21.15.62:0/2737012989 shutdown_connections 2024-04-20T22:03:57.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:57.875+0000 7f308e200700 1 -- 172.21.15.62:0/2737012989 wait complete. 2024-04-20T22:03:57.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:57.876+0000 7f308e200700 1 Processor -- start 2024-04-20T22:03:57.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:57.876+0000 7f308e200700 1 -- start start 2024-04-20T22:03:57.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:57.877+0000 7f308e200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30881004a0 0x7f3088073050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:03:57.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:57.877+0000 7f308e200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3088100e40 0x7f3088073590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:03:57.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:57.877+0000 7f308e200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3088073ad0 con 0x7f3088100e40 2024-04-20T22:03:57.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:57.877+0000 7f308e200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3088073c10 con 0x7f30881004a0 2024-04-20T22:03:57.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:57.877+0000 7f3086a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3088100e40 0x7f3088073590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:03:57.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:57.877+0000 7f3086a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3088100e40 0x7f3088073590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44536/0 (socket says 172.21.15.62:44536) 2024-04-20T22:03:57.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:57.877+0000 7f3086a00700 1 -- 172.21.15.62:0/297469367 learned_addr learned my addr 172.21.15.62:0/297469367 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:03:57.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:57.877+0000 7f3087400700 1 --2- 172.21.15.62:0/297469367 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30881004a0 0x7f3088073050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:03:57.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:57.878+0000 7f3086a00700 1 -- 172.21.15.62:0/297469367 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30881004a0 msgr2=0x7f3088073050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:03:57.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:57.878+0000 7f3086a00700 1 --2- 172.21.15.62:0/297469367 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30881004a0 0x7f3088073050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:03:57.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:57.878+0000 7f3086a00700 1 -- 172.21.15.62:0/297469367 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30780095f0 con 0x7f3088100e40 2024-04-20T22:03:57.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:57.878+0000 7f3086a00700 1 --2- 172.21.15.62:0/297469367 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3088100e40 0x7f3088073590 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f307400c8f0 tx=0x7f307400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:03:57.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:57.878+0000 7f307fe00700 1 -- 172.21.15.62:0/297469367 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3074007a50 con 0x7f3088100e40 2024-04-20T22:03:57.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:57.878+0000 7f307fe00700 1 -- 172.21.15.62:0/297469367 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3074007bb0 con 0x7f3088100e40 2024-04-20T22:03:57.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:57.878+0000 7f307fe00700 1 -- 172.21.15.62:0/297469367 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3074018630 con 0x7f3088100e40 2024-04-20T22:03:57.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:57.878+0000 7f308e200700 1 -- 172.21.15.62:0/297469367 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30881a20a0 con 0x7f3088100e40 2024-04-20T22:03:57.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:57.879+0000 7f308e200700 1 -- 172.21.15.62:0/297469367 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30881a24b0 con 0x7f3088100e40 2024-04-20T22:03:57.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:57.880+0000 7f308e200700 1 -- 172.21.15.62:0/297469367 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30881087b0 con 0x7f3088100e40 2024-04-20T22:03:57.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:57.881+0000 7f307fe00700 1 -- 172.21.15.62:0/297469367 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f3074007d20 con 0x7f3088100e40 2024-04-20T22:03:57.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:57.882+0000 7f307fe00700 1 --2- 172.21.15.62:0/297469367 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f306c06ecc0 0x7f306c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:03:57.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:57.882+0000 7f307fe00700 1 -- 172.21.15.62:0/297469367 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f307408eef0 con 0x7f3088100e40 2024-04-20T22:03:57.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:57.882+0000 7f3087400700 1 --2- 172.21.15.62:0/297469367 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f306c06ecc0 0x7f306c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:03:57.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:57.883+0000 7f3087400700 1 --2- 172.21.15.62:0/297469367 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f306c06ecc0 0x7f306c071170 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f3078006010 tx=0x7f307801a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:03:57.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:57.887+0000 7f307fe00700 1 -- 172.21.15.62:0/297469367 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f3074057c00 con 0x7f3088100e40 2024-04-20T22:03:58.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:58.142+0000 7f308e200700 1 -- 172.21.15.62:0/297469367 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3088061190 con 0x7f306c06ecc0 2024-04-20T22:03:58.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:58.145+0000 7f307fe00700 1 -- 172.21.15.62:0/297469367 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f3088061190 con 0x7f306c06ecc0 2024-04-20T22:03:58.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:58.147+0000 7f308e200700 1 -- 172.21.15.62:0/297469367 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f306c06ecc0 msgr2=0x7f306c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:03:58.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:58.147+0000 7f308e200700 1 --2- 172.21.15.62:0/297469367 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f306c06ecc0 0x7f306c071170 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f3078006010 tx=0x7f307801a040 comp rx=0 tx=0).stop 2024-04-20T22:03:58.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:58.147+0000 7f308e200700 1 -- 172.21.15.62:0/297469367 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3088100e40 msgr2=0x7f3088073590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:03:58.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:58.147+0000 7f308e200700 1 --2- 172.21.15.62:0/297469367 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3088100e40 0x7f3088073590 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f307400c8f0 tx=0x7f307400ccb0 comp rx=0 tx=0).stop 2024-04-20T22:03:58.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:58.148+0000 7f308e200700 1 -- 172.21.15.62:0/297469367 shutdown_connections 2024-04-20T22:03:58.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:58.148+0000 7f308e200700 1 --2- 172.21.15.62:0/297469367 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f306c06ecc0 0x7f306c071170 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:03:58.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:58.148+0000 7f308e200700 1 --2- 172.21.15.62:0/297469367 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30881004a0 0x7f3088073050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:03:58.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:58.148+0000 7f308e200700 1 --2- 172.21.15.62:0/297469367 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3088100e40 0x7f3088073590 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:03:58.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:58.148+0000 7f308e200700 1 -- 172.21.15.62:0/297469367 >> 172.21.15.62:0/297469367 conn(0x7f30880fbd40 msgr2=0x7f30880fd380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:03:58.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:58.148+0000 7f308e200700 1 -- 172.21.15.62:0/297469367 shutdown_connections 2024-04-20T22:03:58.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:58.148+0000 7f308e200700 1 -- 172.21.15.62:0/297469367 wait complete. 2024-04-20T22:03:58.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:58.313+0000 7f236c600700 1 -- 172.21.15.62:0/2151248555 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2364101c90 msgr2=0x7f236410a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:03:58.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:58.313+0000 7f236c600700 1 --2- 172.21.15.62:0/2151248555 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2364101c90 0x7f236410a2d0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f2350009910 tx=0x7f2350009c20 comp rx=0 tx=0).stop 2024-04-20T22:03:58.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:58.313+0000 7f236c600700 1 -- 172.21.15.62:0/2151248555 shutdown_connections 2024-04-20T22:03:58.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:58.313+0000 7f236c600700 1 --2- 172.21.15.62:0/2151248555 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2364101c90 0x7f236410a2d0 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:03:58.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:58.313+0000 7f236c600700 1 --2- 172.21.15.62:0/2151248555 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23641012f0 0x7f23641016c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:03:58.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:58.313+0000 7f236c600700 1 -- 172.21.15.62:0/2151248555 >> 172.21.15.62:0/2151248555 conn(0x7f23640fac80 msgr2=0x7f23640fd090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:03:58.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:58.313+0000 7f236c600700 1 -- 172.21.15.62:0/2151248555 shutdown_connections 2024-04-20T22:03:58.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:58.313+0000 7f236c600700 1 -- 172.21.15.62:0/2151248555 wait complete. 2024-04-20T22:03:58.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:58.315+0000 7f236c600700 1 Processor -- start 2024-04-20T22:03:58.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:58.315+0000 7f236c600700 1 -- start start 2024-04-20T22:03:58.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:58.315+0000 7f236c600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f23641012f0 0x7f23641980e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:03:58.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:58.315+0000 7f236c600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2364101c90 0x7f2364198620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:03:58.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:58.315+0000 7f236c600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2364198c70 con 0x7f23641012f0 2024-04-20T22:03:58.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:58.315+0000 7f236c600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2364198db0 con 0x7f2364101c90 2024-04-20T22:03:58.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:58.316+0000 7f2369c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f23641012f0 0x7f23641980e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:03:58.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:58.316+0000 7f2369c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f23641012f0 0x7f23641980e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44558/0 (socket says 172.21.15.62:44558) 2024-04-20T22:03:58.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:58.316+0000 7f2369c00700 1 -- 172.21.15.62:0/4222643140 learned_addr learned my addr 172.21.15.62:0/4222643140 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:03:58.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:58.316+0000 7f2369200700 1 --2- 172.21.15.62:0/4222643140 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2364101c90 0x7f2364198620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:03:58.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:58.316+0000 7f2369c00700 1 -- 172.21.15.62:0/4222643140 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2364101c90 msgr2=0x7f2364198620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:03:58.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:58.316+0000 7f2369c00700 1 --2- 172.21.15.62:0/4222643140 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2364101c90 0x7f2364198620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:03:58.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:58.316+0000 7f2369c00700 1 -- 172.21.15.62:0/4222643140 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23500095f0 con 0x7f23641012f0 2024-04-20T22:03:58.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:58.317+0000 7f2369c00700 1 --2- 172.21.15.62:0/4222643140 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f23641012f0 0x7f23641980e0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f235800c930 tx=0x7f235800ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:03:58.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:58.317+0000 7f2362000700 1 -- 172.21.15.62:0/4222643140 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2358007ad0 con 0x7f23641012f0 2024-04-20T22:03:58.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:58.317+0000 7f2362000700 1 -- 172.21.15.62:0/4222643140 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2358007c30 con 0x7f23641012f0 2024-04-20T22:03:58.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:58.317+0000 7f2362000700 1 -- 172.21.15.62:0/4222643140 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2358018610 con 0x7f23641012f0 2024-04-20T22:03:58.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:58.317+0000 7f236c600700 1 -- 172.21.15.62:0/4222643140 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f236419cc00 con 0x7f23641012f0 2024-04-20T22:03:58.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:58.317+0000 7f236c600700 1 -- 172.21.15.62:0/4222643140 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f236419d150 con 0x7f23641012f0 2024-04-20T22:03:58.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:58.319+0000 7f236c600700 1 -- 172.21.15.62:0/4222643140 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2364109e30 con 0x7f23641012f0 2024-04-20T22:03:58.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:58.320+0000 7f2362000700 1 -- 172.21.15.62:0/4222643140 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f235800f450 con 0x7f23641012f0 2024-04-20T22:03:58.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:58.321+0000 7f2362000700 1 --2- 172.21.15.62:0/4222643140 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f234c06ecc0 0x7f234c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:03:58.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:58.321+0000 7f2362000700 1 -- 172.21.15.62:0/4222643140 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f235808efe0 con 0x7f23641012f0 2024-04-20T22:03:58.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:58.321+0000 7f2369200700 1 --2- 172.21.15.62:0/4222643140 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f234c06ecc0 0x7f234c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:03:58.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:58.322+0000 7f2369200700 1 --2- 172.21.15.62:0/4222643140 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f234c06ecc0 0x7f234c071170 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f2364199700 tx=0x7f2350005940 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:03:58.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:58.325+0000 7f2362000700 1 -- 172.21.15.62:0/4222643140 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f2358057d90 con 0x7f23641012f0 2024-04-20T22:03:58.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:58.564+0000 7f236c600700 1 -- 172.21.15.62:0/4222643140 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f236419d430 con 0x7f234c06ecc0 2024-04-20T22:03:58.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:58.583+0000 7f2362000700 1 -- 172.21.15.62:0/4222643140 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f236419d430 con 0x7f234c06ecc0 2024-04-20T22:03:58.583 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T22:03:58.583 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (29m) 5m ago 43m 15.3M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T22:03:58.583 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (44m) 5m ago 44m 10.4M - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T22:03:58.583 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (42m) 5m ago 42m 10.1M - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T22:03:58.583 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (25m) 5m ago 43m 6660k - 19.0.0-3133-ga8c84f59 76bc6b91b726 2825d5ba11c6 2024-04-20T22:03:58.583 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (25m) 5m ago 42m 6665k - 19.0.0-3133-ga8c84f59 76bc6b91b726 20f746fa1bba 2024-04-20T22:03:58.583 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (29m) 5m ago 43m 73.5M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T22:03:58.583 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (21m) 5m ago 36m 18.5M - 19.0.0-3133-ga8c84f59 76bc6b91b726 de80bc90a683 2024-04-20T22:03:58.583 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (21m) 5m ago 36m 18.7M - 19.0.0-3133-ga8c84f59 76bc6b91b726 cd793b7ee98e 2024-04-20T22:03:58.583 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (6m) 5m ago 36m 11.0M - 19.0.0-3133-ga8c84f59 76bc6b91b726 0e7b0bc4d1ab 2024-04-20T22:03:58.583 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (36m) 5m ago 36m 18.6M - 18.2.0 dc2bc1663786 f2eec2ee8951 2024-04-20T22:03:58.583 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (32m) 5m ago 45m 513M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T22:03:58.583 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (31m) 5m ago 42m 459M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T22:03:58.583 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (27m) 5m ago 45m 71.7M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 019eb1479485 2024-04-20T22:03:58.583 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (26m) 5m ago 42m 51.0M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 55aac560aab6 2024-04-20T22:03:58.584 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (30m) 5m ago 43m 8472k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T22:03:58.584 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (30m) 5m ago 42m 8048k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T22:03:58.584 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (25m) 5m ago 41m 91.2M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 ee2bb901f26e 2024-04-20T22:03:58.584 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (24m) 5m ago 40m 94.6M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 9075152d4162 2024-04-20T22:03:58.584 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (24m) 5m ago 40m 78.1M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 2d4f3e38dc0e 2024-04-20T22:03:58.584 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (23m) 5m ago 39m 92.6M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 8f97b17ae0eb 2024-04-20T22:03:58.584 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (22m) 5m ago 38m 95.6M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 653d8079967d 2024-04-20T22:03:58.584 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (22m) 5m ago 38m 89.2M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 596dded14c12 2024-04-20T22:03:58.584 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (30m) 5m ago 43m 82.1M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T22:03:58.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:58.587+0000 7f236c600700 1 -- 172.21.15.62:0/4222643140 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f234c06ecc0 msgr2=0x7f234c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:03:58.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:58.587+0000 7f236c600700 1 --2- 172.21.15.62:0/4222643140 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f234c06ecc0 0x7f234c071170 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f2364199700 tx=0x7f2350005940 comp rx=0 tx=0).stop 2024-04-20T22:03:58.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:58.587+0000 7f236c600700 1 -- 172.21.15.62:0/4222643140 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f23641012f0 msgr2=0x7f23641980e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:03:58.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:58.587+0000 7f236c600700 1 --2- 172.21.15.62:0/4222643140 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f23641012f0 0x7f23641980e0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f235800c930 tx=0x7f235800ccf0 comp rx=0 tx=0).stop 2024-04-20T22:03:58.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:58.587+0000 7f236c600700 1 -- 172.21.15.62:0/4222643140 shutdown_connections 2024-04-20T22:03:58.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:58.587+0000 7f236c600700 1 --2- 172.21.15.62:0/4222643140 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f234c06ecc0 0x7f234c071170 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:03:58.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:58.587+0000 7f236c600700 1 --2- 172.21.15.62:0/4222643140 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f23641012f0 0x7f23641980e0 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:03:58.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:58.587+0000 7f236c600700 1 --2- 172.21.15.62:0/4222643140 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2364101c90 0x7f2364198620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:03:58.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:58.587+0000 7f236c600700 1 -- 172.21.15.62:0/4222643140 >> 172.21.15.62:0/4222643140 conn(0x7f23640fac80 msgr2=0x7f23640ff8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:03:58.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:58.588+0000 7f236c600700 1 -- 172.21.15.62:0/4222643140 shutdown_connections 2024-04-20T22:03:58.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:58.588+0000 7f236c600700 1 -- 172.21.15.62:0/4222643140 wait complete. 2024-04-20T22:03:58.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:58.741+0000 7f6dc8000700 1 -- 172.21.15.62:0/227225163 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6dc010c8f0 msgr2=0x7f6dc010ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:03:58.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:58.741+0000 7f6dc8000700 1 --2- 172.21.15.62:0/227225163 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6dc010c8f0 0x7f6dc010ccc0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f6db4009910 tx=0x7f6db4009c20 comp rx=0 tx=0).stop 2024-04-20T22:03:58.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:58.741+0000 7f6dc8000700 1 -- 172.21.15.62:0/227225163 shutdown_connections 2024-04-20T22:03:58.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:58.741+0000 7f6dc8000700 1 --2- 172.21.15.62:0/227225163 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6dc0106b80 0x7f6dc0106ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:03:58.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:58.741+0000 7f6dc8000700 1 --2- 172.21.15.62:0/227225163 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6dc010c8f0 0x7f6dc010ccc0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:03:58.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:58.741+0000 7f6dc8000700 1 -- 172.21.15.62:0/227225163 >> 172.21.15.62:0/227225163 conn(0x7f6dc0076020 msgr2=0x7f6dc0078430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:03:58.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:58.741+0000 7f6dc8000700 1 -- 172.21.15.62:0/227225163 shutdown_connections 2024-04-20T22:03:58.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:58.741+0000 7f6dc8000700 1 -- 172.21.15.62:0/227225163 wait complete. 2024-04-20T22:03:58.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:58.742+0000 7f6dc8000700 1 Processor -- start 2024-04-20T22:03:58.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:58.742+0000 7f6dc8000700 1 -- start start 2024-04-20T22:03:58.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:58.742+0000 7f6dc8000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6dc0106b80 0x7f6dc019c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:03:58.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:58.742+0000 7f6dc8000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6dc010c8f0 0x7f6dc019ca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:03:58.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:58.742+0000 7f6dc8000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dc019d0f0 con 0x7f6dc0106b80 2024-04-20T22:03:58.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:58.742+0000 7f6dc8000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dc01a0e80 con 0x7f6dc010c8f0 2024-04-20T22:03:58.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:58.743+0000 7f6dc5600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6dc0106b80 0x7f6dc019c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:03:58.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:58.743+0000 7f6dc5600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6dc0106b80 0x7f6dc019c4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44566/0 (socket says 172.21.15.62:44566) 2024-04-20T22:03:58.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:58.743+0000 7f6dc5600700 1 -- 172.21.15.62:0/2025894678 learned_addr learned my addr 172.21.15.62:0/2025894678 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:03:58.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:58.743+0000 7f6dc5600700 1 -- 172.21.15.62:0/2025894678 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6dc010c8f0 msgr2=0x7f6dc019ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:03:58.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:58.743+0000 7f6dc5600700 1 --2- 172.21.15.62:0/2025894678 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6dc010c8f0 0x7f6dc019ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:03:58.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:58.743+0000 7f6dc5600700 1 -- 172.21.15.62:0/2025894678 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6db40095f0 con 0x7f6dc0106b80 2024-04-20T22:03:58.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:58.743+0000 7f6dc5600700 1 --2- 172.21.15.62:0/2025894678 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6dc0106b80 0x7f6dc019c4d0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f6db4006010 tx=0x7f6db400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:03:58.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:58.744+0000 7f6dbe000700 1 -- 172.21.15.62:0/2025894678 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6db401d070 con 0x7f6dc0106b80 2024-04-20T22:03:58.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:58.744+0000 7f6dbe000700 1 -- 172.21.15.62:0/2025894678 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6db4004fb0 con 0x7f6dc0106b80 2024-04-20T22:03:58.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:58.744+0000 7f6dbe000700 1 -- 172.21.15.62:0/2025894678 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6db400f690 con 0x7f6dc0106b80 2024-04-20T22:03:58.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:58.744+0000 7f6dc8000700 1 -- 172.21.15.62:0/2025894678 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6dc01a1100 con 0x7f6dc0106b80 2024-04-20T22:03:58.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:58.744+0000 7f6dc8000700 1 -- 172.21.15.62:0/2025894678 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6dc01a15f0 con 0x7f6dc0106b80 2024-04-20T22:03:58.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:58.745+0000 7f6dc8000700 1 -- 172.21.15.62:0/2025894678 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6dc004f2a0 con 0x7f6dc0106b80 2024-04-20T22:03:58.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:58.746+0000 7f6dbe000700 1 -- 172.21.15.62:0/2025894678 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f6db40228a0 con 0x7f6dc0106b80 2024-04-20T22:03:58.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:58.747+0000 7f6dbe000700 1 --2- 172.21.15.62:0/2025894678 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6da806ecc0 0x7f6da8071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:03:58.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:58.747+0000 7f6dbe000700 1 -- 172.21.15.62:0/2025894678 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f6db4090700 con 0x7f6dc0106b80 2024-04-20T22:03:58.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:58.748+0000 7f6dc4c00700 1 --2- 172.21.15.62:0/2025894678 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6da806ecc0 0x7f6da8071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:03:58.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:58.748+0000 7f6dc4c00700 1 --2- 172.21.15.62:0/2025894678 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6da806ecc0 0x7f6da8071170 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f6dc019daf0 tx=0x7f6dac006d20 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:03:58.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:58.752+0000 7f6dbe000700 1 -- 172.21.15.62:0/2025894678 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f6db4059490 con 0x7f6dc0106b80 2024-04-20T22:03:59.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:03:58 smithi062 ceph-mon[76322]: pgmap v861: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:03:59.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:03:58 smithi062 ceph-mon[76322]: from='client.35544 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:03:59.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:03:58 smithi062 ceph-mon[76322]: from='client.35548 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:03:59.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.074+0000 7f6dc8000700 1 -- 172.21.15.62:0/2025894678 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f6dc01a1a20 con 0x7f6dc0106b80 2024-04-20T22:03:59.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.076+0000 7f6dbe000700 1 -- 172.21.15.62:0/2025894678 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7f6db405cb50 con 0x7f6dc0106b80 2024-04-20T22:03:59.074 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:03:59.075 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T22:03:59.075 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:03:59.075 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:03:59.075 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T22:03:59.075 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:03:59.075 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:03:59.075 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T22:03:59.075 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 6 2024-04-20T22:03:59.075 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:03:59.075 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T22:03:59.075 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 1, 2024-04-20T22:03:59.075 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 3 2024-04-20T22:03:59.075 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:03:59.075 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T22:03:59.076 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 1, 2024-04-20T22:03:59.076 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 13 2024-04-20T22:03:59.076 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T22:03:59.076 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:03:59.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.078+0000 7f6dc8000700 1 -- 172.21.15.62:0/2025894678 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6da806ecc0 msgr2=0x7f6da8071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:03:59.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.078+0000 7f6dc8000700 1 --2- 172.21.15.62:0/2025894678 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6da806ecc0 0x7f6da8071170 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f6dc019daf0 tx=0x7f6dac006d20 comp rx=0 tx=0).stop 2024-04-20T22:03:59.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.078+0000 7f6dc8000700 1 -- 172.21.15.62:0/2025894678 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6dc0106b80 msgr2=0x7f6dc019c4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:03:59.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.078+0000 7f6dc8000700 1 --2- 172.21.15.62:0/2025894678 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6dc0106b80 0x7f6dc019c4d0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f6db4006010 tx=0x7f6db400be20 comp rx=0 tx=0).stop 2024-04-20T22:03:59.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.078+0000 7f6dc8000700 1 -- 172.21.15.62:0/2025894678 shutdown_connections 2024-04-20T22:03:59.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.078+0000 7f6dc8000700 1 --2- 172.21.15.62:0/2025894678 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6da806ecc0 0x7f6da8071170 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:03:59.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.078+0000 7f6dc8000700 1 --2- 172.21.15.62:0/2025894678 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6dc0106b80 0x7f6dc019c4d0 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:03:59.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.078+0000 7f6dc8000700 1 --2- 172.21.15.62:0/2025894678 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6dc010c8f0 0x7f6dc019ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:03:59.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.078+0000 7f6dc8000700 1 -- 172.21.15.62:0/2025894678 >> 172.21.15.62:0/2025894678 conn(0x7f6dc0076020 msgr2=0x7f6dc00779e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:03:59.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.079+0000 7f6dc8000700 1 -- 172.21.15.62:0/2025894678 shutdown_connections 2024-04-20T22:03:59.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.079+0000 7f6dc8000700 1 -- 172.21.15.62:0/2025894678 wait complete. 2024-04-20T22:03:59.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.236+0000 7fe6ba000700 1 -- 172.21.15.62:0/575280629 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6b4100e80 msgr2=0x7fe6b4104e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:03:59.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.236+0000 7fe6ba000700 1 --2- 172.21.15.62:0/575280629 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6b4100e80 0x7fe6b4104e50 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fe69c009910 tx=0x7fe69c009c20 comp rx=0 tx=0).stop 2024-04-20T22:03:59.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.236+0000 7fe6ba000700 1 -- 172.21.15.62:0/575280629 shutdown_connections 2024-04-20T22:03:59.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.236+0000 7fe6ba000700 1 --2- 172.21.15.62:0/575280629 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6b4100e80 0x7fe6b4104e50 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:03:59.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.236+0000 7fe6ba000700 1 --2- 172.21.15.62:0/575280629 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe6b41004e0 0x7fe6b41008b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:03:59.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.236+0000 7fe6ba000700 1 -- 172.21.15.62:0/575280629 >> 172.21.15.62:0/575280629 conn(0x7fe6b40fbda0 msgr2=0x7fe6b40fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:03:59.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.236+0000 7fe6ba000700 1 -- 172.21.15.62:0/575280629 shutdown_connections 2024-04-20T22:03:59.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.236+0000 7fe6ba000700 1 -- 172.21.15.62:0/575280629 wait complete. 2024-04-20T22:03:59.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.237+0000 7fe6ba000700 1 Processor -- start 2024-04-20T22:03:59.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.237+0000 7fe6ba000700 1 -- start start 2024-04-20T22:03:59.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.238+0000 7fe6ba000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe6b41004e0 0x7fe6b4198250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:03:59.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.238+0000 7fe6ba000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6b4100e80 0x7fe6b4198790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:03:59.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.238+0000 7fe6ba000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b4198e70 con 0x7fe6b4100e80 2024-04-20T22:03:59.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.238+0000 7fe6ba000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b419cc00 con 0x7fe6b41004e0 2024-04-20T22:03:59.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.238+0000 7fe6aaa00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6b4100e80 0x7fe6b4198790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:03:59.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.238+0000 7fe6aaa00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6b4100e80 0x7fe6b4198790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44580/0 (socket says 172.21.15.62:44580) 2024-04-20T22:03:59.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.238+0000 7fe6aaa00700 1 -- 172.21.15.62:0/1061875792 learned_addr learned my addr 172.21.15.62:0/1061875792 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:03:59.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:03:58 smithi189 ceph-mon[57022]: pgmap v861: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:03:59.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:03:58 smithi189 ceph-mon[57022]: from='client.35544 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:03:59.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:03:58 smithi189 ceph-mon[57022]: from='client.35548 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:03:59.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.239+0000 7fe6aaa00700 1 -- 172.21.15.62:0/1061875792 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe6b41004e0 msgr2=0x7fe6b4198250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:03:59.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.239+0000 7fe6aaa00700 1 --2- 172.21.15.62:0/1061875792 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe6b41004e0 0x7fe6b4198250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:03:59.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.239+0000 7fe6aaa00700 1 -- 172.21.15.62:0/1061875792 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe69c0095f0 con 0x7fe6b4100e80 2024-04-20T22:03:59.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.239+0000 7fe6aaa00700 1 --2- 172.21.15.62:0/1061875792 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6b4100e80 0x7fe6b4198790 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fe69c005f50 tx=0x7fe69c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:03:59.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.239+0000 7fe6b0c00700 1 -- 172.21.15.62:0/1061875792 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe69c01d070 con 0x7fe6b4100e80 2024-04-20T22:03:59.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.239+0000 7fe6b0c00700 1 -- 172.21.15.62:0/1061875792 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe69c00f460 con 0x7fe6b4100e80 2024-04-20T22:03:59.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.239+0000 7fe6b0c00700 1 -- 172.21.15.62:0/1061875792 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe69c017630 con 0x7fe6b4100e80 2024-04-20T22:03:59.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.239+0000 7fe6ba000700 1 -- 172.21.15.62:0/1061875792 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6b419ce80 con 0x7fe6b4100e80 2024-04-20T22:03:59.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.240+0000 7fe6ba000700 1 -- 172.21.15.62:0/1061875792 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6b419d370 con 0x7fe6b4100e80 2024-04-20T22:03:59.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.241+0000 7fe6ba000700 1 -- 172.21.15.62:0/1061875792 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6b41045d0 con 0x7fe6b4100e80 2024-04-20T22:03:59.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.242+0000 7fe6b0c00700 1 -- 172.21.15.62:0/1061875792 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe69c00f5d0 con 0x7fe6b4100e80 2024-04-20T22:03:59.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.242+0000 7fe6b0c00700 1 --2- 172.21.15.62:0/1061875792 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe69806ecc0 0x7fe698071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:03:59.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.243+0000 7fe6b0c00700 1 -- 172.21.15.62:0/1061875792 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fe69c090420 con 0x7fe6b4100e80 2024-04-20T22:03:59.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.243+0000 7fe6b3400700 1 --2- 172.21.15.62:0/1061875792 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe69806ecc0 0x7fe698071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:03:59.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.243+0000 7fe6b3400700 1 --2- 172.21.15.62:0/1061875792 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe69806ecc0 0x7fe698071170 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fe6a40097b0 tx=0x7fe6a4006d20 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:03:59.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.247+0000 7fe6b0c00700 1 -- 172.21.15.62:0/1061875792 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fe69c0591b0 con 0x7fe6b4100e80 2024-04-20T22:03:59.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.540+0000 7fe6ba000700 1 -- 172.21.15.62:0/1061875792 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fe6b4199650 con 0x7fe6b4100e80 2024-04-20T22:03:59.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.541+0000 7fe6b0c00700 1 -- 172.21.15.62:0/1061875792 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 27 v27) v1 ==== 76+0+1888 (secure 0 0 0) 0x7fe69c05c870 con 0x7fe6b4100e80 2024-04-20T22:03:59.541 INFO:teuthology.orchestra.run.smithi062.stdout:e27 2024-04-20T22:03:59.541 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T22:03:59.541 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:03:59.541 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T22:03:59.541 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:03:59.541 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T22:03:59.541 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T22:03:59.541 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 20 2024-04-20T22:03:59.542 INFO:teuthology.orchestra.run.smithi062.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-20T22:03:59.542 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T22:03:59.542 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T21:42:01.559061+0000 2024-04-20T22:03:59.542 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T22:03:59.542 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T22:03:59.542 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T22:03:59.542 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T22:03:59.542 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T22:03:59.542 INFO:teuthology.orchestra.run.smithi062.stdout:max_xattr_size 65536 2024-04-20T22:03:59.542 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T22:03:59.542 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T22:03:59.542 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 81 2024-04-20T22:03:59.542 INFO:teuthology.orchestra.run.smithi062.stdout:snap_rstat 1 2024-04-20T22:03:59.542 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:03:59.542 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 2 2024-04-20T22:03:59.542 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1 2024-04-20T22:03:59.543 INFO:teuthology.orchestra.run.smithi062.stdout:up {} 2024-04-20T22:03:59.543 INFO:teuthology.orchestra.run.smithi062.stdout:failed 0,1 2024-04-20T22:03:59.543 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T22:03:59.543 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T22:03:59.543 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T22:03:59.543 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T22:03:59.543 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data enabled 2024-04-20T22:03:59.543 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T22:03:59.543 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T22:03:59.543 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 1 2024-04-20T22:03:59.543 INFO:teuthology.orchestra.run.smithi062.stdout:qdb_cluster leader: 0 members: 2024-04-20T22:03:59.543 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:03:59.543 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:03:59.543 INFO:teuthology.orchestra.run.smithi062.stdout:Standby daemons: 2024-04-20T22:03:59.543 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:03:59.543 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ryqblb{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6826/4113916205,v1:172.21.15.62:6827/4113916205] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:03:59.543 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ozizwi{-1:34476} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6828/3801975165,v1:172.21.15.62:6829/3801975165] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:03:59.544 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.xbgspw{-1:44285} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6826/3996040526,v1:172.21.15.189:6827/3996040526] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T22:03:59.544 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.dvimer{-1:44727} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6824/3576699232,v1:172.21.15.189:6825/3576699232] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:03:59.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.545+0000 7fe6ba000700 1 -- 172.21.15.62:0/1061875792 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe69806ecc0 msgr2=0x7fe698071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:03:59.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.545+0000 7fe6ba000700 1 --2- 172.21.15.62:0/1061875792 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe69806ecc0 0x7fe698071170 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fe6a40097b0 tx=0x7fe6a4006d20 comp rx=0 tx=0).stop 2024-04-20T22:03:59.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.545+0000 7fe6ba000700 1 -- 172.21.15.62:0/1061875792 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6b4100e80 msgr2=0x7fe6b4198790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:03:59.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.545+0000 7fe6ba000700 1 --2- 172.21.15.62:0/1061875792 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6b4100e80 0x7fe6b4198790 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fe69c005f50 tx=0x7fe69c00be20 comp rx=0 tx=0).stop 2024-04-20T22:03:59.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.546+0000 7fe6ba000700 1 -- 172.21.15.62:0/1061875792 shutdown_connections 2024-04-20T22:03:59.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.546+0000 7fe6ba000700 1 --2- 172.21.15.62:0/1061875792 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe69806ecc0 0x7fe698071170 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:03:59.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.546+0000 7fe6ba000700 1 --2- 172.21.15.62:0/1061875792 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe6b41004e0 0x7fe6b4198250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:03:59.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.546+0000 7fe6ba000700 1 --2- 172.21.15.62:0/1061875792 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6b4100e80 0x7fe6b4198790 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:03:59.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.546+0000 7fe6ba000700 1 -- 172.21.15.62:0/1061875792 >> 172.21.15.62:0/1061875792 conn(0x7fe6b40fbda0 msgr2=0x7fe6b40fd500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:03:59.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.546+0000 7fe6ba000700 1 -- 172.21.15.62:0/1061875792 shutdown_connections 2024-04-20T22:03:59.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.546+0000 7fe6ba000700 1 -- 172.21.15.62:0/1061875792 wait complete. 2024-04-20T22:03:59.546 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 27 2024-04-20T22:03:59.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.704+0000 7f5940200700 1 -- 172.21.15.62:0/1629402842 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f59380ffbd0 msgr2=0x7f5938100040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:03:59.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.704+0000 7f5940200700 1 --2- 172.21.15.62:0/1629402842 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f59380ffbd0 0x7f5938100040 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f5924009910 tx=0x7f5924009c20 comp rx=0 tx=0).stop 2024-04-20T22:03:59.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.704+0000 7f5940200700 1 -- 172.21.15.62:0/1629402842 shutdown_connections 2024-04-20T22:03:59.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.704+0000 7f5940200700 1 --2- 172.21.15.62:0/1629402842 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f59380ffbd0 0x7f5938100040 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:03:59.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.704+0000 7f5940200700 1 --2- 172.21.15.62:0/1629402842 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5938104130 0x7f5938104500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:03:59.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.704+0000 7f5940200700 1 -- 172.21.15.62:0/1629402842 >> 172.21.15.62:0/1629402842 conn(0x7f5938075130 msgr2=0x7f5938075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:03:59.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.704+0000 7f5940200700 1 -- 172.21.15.62:0/1629402842 shutdown_connections 2024-04-20T22:03:59.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.704+0000 7f5940200700 1 -- 172.21.15.62:0/1629402842 wait complete. 2024-04-20T22:03:59.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.705+0000 7f5940200700 1 Processor -- start 2024-04-20T22:03:59.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.706+0000 7f5940200700 1 -- start start 2024-04-20T22:03:59.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.706+0000 7f5940200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f59380ffbd0 0x7f5938195e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:03:59.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.706+0000 7f5940200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5938104130 0x7f5938196390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:03:59.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.706+0000 7f5940200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5938196a70 con 0x7f5938104130 2024-04-20T22:03:59.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.706+0000 7f5940200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f593819a800 con 0x7f59380ffbd0 2024-04-20T22:03:59.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.706+0000 7f593ce00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5938104130 0x7f5938196390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:03:59.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.706+0000 7f593ce00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5938104130 0x7f5938196390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44604/0 (socket says 172.21.15.62:44604) 2024-04-20T22:03:59.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.706+0000 7f593ce00700 1 -- 172.21.15.62:0/1940117565 learned_addr learned my addr 172.21.15.62:0/1940117565 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:03:59.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.707+0000 7f593ce00700 1 -- 172.21.15.62:0/1940117565 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f59380ffbd0 msgr2=0x7f5938195e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:03:59.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.707+0000 7f593d800700 1 --2- 172.21.15.62:0/1940117565 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f59380ffbd0 0x7f5938195e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:03:59.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.707+0000 7f593ce00700 1 --2- 172.21.15.62:0/1940117565 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f59380ffbd0 0x7f5938195e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:03:59.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.707+0000 7f593ce00700 1 -- 172.21.15.62:0/1940117565 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59240095f0 con 0x7f5938104130 2024-04-20T22:03:59.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.707+0000 7f593d800700 1 --2- 172.21.15.62:0/1940117565 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f59380ffbd0 0x7f5938195e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T22:03:59.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.707+0000 7f593ce00700 1 --2- 172.21.15.62:0/1940117565 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5938104130 0x7f5938196390 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f59240092e0 tx=0x7f592400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:03:59.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.708+0000 7f5936000700 1 -- 172.21.15.62:0/1940117565 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f592401d070 con 0x7f5938104130 2024-04-20T22:03:59.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.708+0000 7f5936000700 1 -- 172.21.15.62:0/1940117565 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f592400f460 con 0x7f5938104130 2024-04-20T22:03:59.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.708+0000 7f5936000700 1 -- 172.21.15.62:0/1940117565 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5924005e20 con 0x7f5938104130 2024-04-20T22:03:59.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.708+0000 7f5940200700 1 -- 172.21.15.62:0/1940117565 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f593819aa80 con 0x7f5938104130 2024-04-20T22:03:59.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.708+0000 7f5940200700 1 -- 172.21.15.62:0/1940117565 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f593819af70 con 0x7f5938104130 2024-04-20T22:03:59.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.709+0000 7f5940200700 1 -- 172.21.15.62:0/1940117565 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f593804f2a0 con 0x7f5938104130 2024-04-20T22:03:59.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.710+0000 7f5936000700 1 -- 172.21.15.62:0/1940117565 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f592400f5d0 con 0x7f5938104130 2024-04-20T22:03:59.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.711+0000 7f5936000700 1 --2- 172.21.15.62:0/1940117565 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f592006ecc0 0x7f5920071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:03:59.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.711+0000 7f5936000700 1 -- 172.21.15.62:0/1940117565 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f5924090350 con 0x7f5938104130 2024-04-20T22:03:59.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.711+0000 7f593d800700 1 --2- 172.21.15.62:0/1940117565 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f592006ecc0 0x7f5920071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:03:59.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.712+0000 7f593d800700 1 --2- 172.21.15.62:0/1940117565 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f592006ecc0 0x7f5920071170 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f592c005fd0 tx=0x7f592c005ee0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:03:59.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.716+0000 7f5936000700 1 -- 172.21.15.62:0/1940117565 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f59240591a0 con 0x7f5938104130 2024-04-20T22:03:59.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.961+0000 7f5940200700 1 -- 172.21.15.62:0/1940117565 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f593819b380 con 0x7f592006ecc0 2024-04-20T22:03:59.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.964+0000 7f5936000700 1 -- 172.21.15.62:0/1940117565 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f593819b380 con 0x7f592006ecc0 2024-04-20T22:03:59.963 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:03:59.963 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T22:03:59.963 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T22:03:59.963 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-20T22:03:59.963 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T22:03:59.963 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr", 2024-04-20T22:03:59.963 INFO:teuthology.orchestra.run.smithi062.stdout: "crash", 2024-04-20T22:03:59.963 INFO:teuthology.orchestra.run.smithi062.stdout: "mon", 2024-04-20T22:03:59.963 INFO:teuthology.orchestra.run.smithi062.stdout: "osd" 2024-04-20T22:03:59.963 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T22:03:59.963 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "15/23 daemons upgraded", 2024-04-20T22:03:59.964 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading mds daemons", 2024-04-20T22:03:59.964 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T22:03:59.964 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:03:59.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.967+0000 7f5940200700 1 -- 172.21.15.62:0/1940117565 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f592006ecc0 msgr2=0x7f5920071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:03:59.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.967+0000 7f5940200700 1 --2- 172.21.15.62:0/1940117565 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f592006ecc0 0x7f5920071170 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f592c005fd0 tx=0x7f592c005ee0 comp rx=0 tx=0).stop 2024-04-20T22:03:59.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.967+0000 7f5940200700 1 -- 172.21.15.62:0/1940117565 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5938104130 msgr2=0x7f5938196390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:03:59.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.967+0000 7f5940200700 1 --2- 172.21.15.62:0/1940117565 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5938104130 0x7f5938196390 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f59240092e0 tx=0x7f592400be20 comp rx=0 tx=0).stop 2024-04-20T22:03:59.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.967+0000 7f5940200700 1 -- 172.21.15.62:0/1940117565 shutdown_connections 2024-04-20T22:03:59.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.967+0000 7f5940200700 1 --2- 172.21.15.62:0/1940117565 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f592006ecc0 0x7f5920071170 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:03:59.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.967+0000 7f5940200700 1 --2- 172.21.15.62:0/1940117565 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f59380ffbd0 0x7f5938195e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:03:59.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.967+0000 7f5940200700 1 --2- 172.21.15.62:0/1940117565 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5938104130 0x7f5938196390 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:03:59.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.967+0000 7f5940200700 1 -- 172.21.15.62:0/1940117565 >> 172.21.15.62:0/1940117565 conn(0x7f5938075130 msgr2=0x7f59380fe870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:03:59.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.967+0000 7f5940200700 1 -- 172.21.15.62:0/1940117565 shutdown_connections 2024-04-20T22:03:59.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:03:59.967+0000 7f5940200700 1 -- 172.21.15.62:0/1940117565 wait complete. 2024-04-20T22:04:00.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:00.117+0000 7f5e9e200700 1 -- 172.21.15.62:0/2429472549 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e98100e60 msgr2=0x7f5e98104e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:04:00.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:00.117+0000 7f5e9e200700 1 --2- 172.21.15.62:0/2429472549 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e98100e60 0x7f5e98104e30 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f5e84009910 tx=0x7f5e84009c20 comp rx=0 tx=0).stop 2024-04-20T22:04:00.116 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:03:59 smithi062 ceph-mon[76322]: from='client.35552 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:04:00.116 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:03:59 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/2025894678' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:04:00.116 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:03:59 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/1061875792' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:04:00.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:00.117+0000 7f5e9e200700 1 -- 172.21.15.62:0/2429472549 shutdown_connections 2024-04-20T22:04:00.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:00.117+0000 7f5e9e200700 1 --2- 172.21.15.62:0/2429472549 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e98100e60 0x7f5e98104e30 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:04:00.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:00.117+0000 7f5e9e200700 1 --2- 172.21.15.62:0/2429472549 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e981004c0 0x7f5e98100890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:04:00.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:00.117+0000 7f5e9e200700 1 -- 172.21.15.62:0/2429472549 >> 172.21.15.62:0/2429472549 conn(0x7f5e980fbd80 msgr2=0x7f5e980fe190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:04:00.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:00.118+0000 7f5e9e200700 1 -- 172.21.15.62:0/2429472549 shutdown_connections 2024-04-20T22:04:00.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:00.118+0000 7f5e9e200700 1 -- 172.21.15.62:0/2429472549 wait complete. 2024-04-20T22:04:00.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:00.119+0000 7f5e9e200700 1 Processor -- start 2024-04-20T22:04:00.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:00.119+0000 7f5e9e200700 1 -- start start 2024-04-20T22:04:00.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:00.119+0000 7f5e9e200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e981004c0 0x7f5e98195ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:04:00.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:00.119+0000 7f5e9e200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e98100e60 0x7f5e98196430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:04:00.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:00.119+0000 7f5e9e200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e98196b10 con 0x7f5e981004c0 2024-04-20T22:04:00.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:00.119+0000 7f5e9e200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e9819a8a0 con 0x7f5e98100e60 2024-04-20T22:04:00.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:00.119+0000 7f5e97400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e981004c0 0x7f5e98195ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:04:00.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:00.120+0000 7f5e97400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e981004c0 0x7f5e98195ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44622/0 (socket says 172.21.15.62:44622) 2024-04-20T22:04:00.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:00.120+0000 7f5e97400700 1 -- 172.21.15.62:0/942726662 learned_addr learned my addr 172.21.15.62:0/942726662 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:04:00.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:00.120+0000 7f5e96a00700 1 --2- 172.21.15.62:0/942726662 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e98100e60 0x7f5e98196430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:04:00.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:00.120+0000 7f5e97400700 1 -- 172.21.15.62:0/942726662 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e98100e60 msgr2=0x7f5e98196430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:04:00.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:00.120+0000 7f5e97400700 1 --2- 172.21.15.62:0/942726662 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e98100e60 0x7f5e98196430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:04:00.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:00.120+0000 7f5e97400700 1 -- 172.21.15.62:0/942726662 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e840095f0 con 0x7f5e981004c0 2024-04-20T22:04:00.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:00.120+0000 7f5e96a00700 1 --2- 172.21.15.62:0/942726662 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e98100e60 0x7f5e98196430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T22:04:00.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:00.120+0000 7f5e97400700 1 --2- 172.21.15.62:0/942726662 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e981004c0 0x7f5e98195ef0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f5e8800d8d0 tx=0x7f5e8800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:04:00.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:00.121+0000 7f5e8fe00700 1 -- 172.21.15.62:0/942726662 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5e88009940 con 0x7f5e981004c0 2024-04-20T22:04:00.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:00.121+0000 7f5e8fe00700 1 -- 172.21.15.62:0/942726662 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5e88010460 con 0x7f5e981004c0 2024-04-20T22:04:00.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:00.121+0000 7f5e9e200700 1 -- 172.21.15.62:0/942726662 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e9819ab80 con 0x7f5e981004c0 2024-04-20T22:04:00.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:00.121+0000 7f5e8fe00700 1 -- 172.21.15.62:0/942726662 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5e8800f5f0 con 0x7f5e981004c0 2024-04-20T22:04:00.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:00.121+0000 7f5e9e200700 1 -- 172.21.15.62:0/942726662 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e9819b0d0 con 0x7f5e981004c0 2024-04-20T22:04:00.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:00.122+0000 7f5e9e200700 1 -- 172.21.15.62:0/942726662 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e98108830 con 0x7f5e981004c0 2024-04-20T22:04:00.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:00.123+0000 7f5e8fe00700 1 -- 172.21.15.62:0/942726662 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f5e880105d0 con 0x7f5e981004c0 2024-04-20T22:04:00.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:00.124+0000 7f5e8fe00700 1 --2- 172.21.15.62:0/942726662 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5e7c06ecc0 0x7f5e7c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:04:00.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:00.124+0000 7f5e8fe00700 1 -- 172.21.15.62:0/942726662 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f5e8808ef20 con 0x7f5e981004c0 2024-04-20T22:04:00.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:00.125+0000 7f5e96a00700 1 --2- 172.21.15.62:0/942726662 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5e7c06ecc0 0x7f5e7c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:04:00.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:00.125+0000 7f5e96a00700 1 --2- 172.21.15.62:0/942726662 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5e7c06ecc0 0x7f5e7c071170 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f5e84006010 tx=0x7f5e8400b540 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:04:00.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:00.129+0000 7f5e8fe00700 1 -- 172.21.15.62:0/942726662 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f5e88057c30 con 0x7f5e981004c0 2024-04-20T22:04:00.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:03:59 smithi189 ceph-mon[57022]: from='client.35552 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:04:00.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:03:59 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/2025894678' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:04:00.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:03:59 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/1061875792' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:04:00.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:00.451+0000 7f5e9e200700 1 -- 172.21.15.62:0/942726662 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f5e9804f2a0 con 0x7f5e981004c0 2024-04-20T22:04:00.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:00.452+0000 7f5e8fe00700 1 -- 172.21.15.62:0/942726662 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7f5e8805b2f0 con 0x7f5e981004c0 2024-04-20T22:04:00.452 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-20T22:04:00.452 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-20T22:04:00.452 INFO:teuthology.orchestra.run.smithi062.stdout: host smithi062 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T22:04:00.452 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-20T22:04:00.452 INFO:teuthology.orchestra.run.smithi062.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi062 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpq7y7oa_r:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:04:00.452 INFO:teuthology.orchestra.run.smithi062.stdout:Traceback (most recent call last): 2024-04-20T22:04:00.453 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-20T22:04:00.453 INFO:teuthology.orchestra.run.smithi062.stdout: return _run_code(code, main_globals, None, 2024-04-20T22:04:00.453 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-20T22:04:00.453 INFO:teuthology.orchestra.run.smithi062.stdout: exec(code, run_globals) 2024-04-20T22:04:00.453 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5618, in 2024-04-20T22:04:00.453 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5606, in main 2024-04-20T22:04:00.453 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 408, in _infer_config 2024-04-20T22:04:00.453 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 323, in _infer_fsid 2024-04-20T22:04:00.453 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 436, in _infer_image 2024-04-20T22:04:00.453 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 310, in _validate_fsid 2024-04-20T22:04:00.453 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 3359, in command_ceph_volume 2024-04-20T22:04:00.453 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-20T22:04:00.453 INFO:teuthology.orchestra.run.smithi062.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi062 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpq7y7oa_r:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:04:00.453 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-20T22:04:00.453 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is degraded 2024-04-20T22:04:00.453 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T22:04:00.454 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T22:04:00.454 INFO:teuthology.orchestra.run.smithi062.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-20T22:04:00.454 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-20T22:04:00.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:00.455+0000 7f5e9e200700 1 -- 172.21.15.62:0/942726662 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5e7c06ecc0 msgr2=0x7f5e7c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:04:00.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:00.455+0000 7f5e9e200700 1 --2- 172.21.15.62:0/942726662 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5e7c06ecc0 0x7f5e7c071170 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f5e84006010 tx=0x7f5e8400b540 comp rx=0 tx=0).stop 2024-04-20T22:04:00.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:00.455+0000 7f5e9e200700 1 -- 172.21.15.62:0/942726662 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e981004c0 msgr2=0x7f5e98195ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:04:00.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:00.455+0000 7f5e9e200700 1 --2- 172.21.15.62:0/942726662 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e981004c0 0x7f5e98195ef0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f5e8800d8d0 tx=0x7f5e8800dc90 comp rx=0 tx=0).stop 2024-04-20T22:04:00.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:00.455+0000 7f5e9e200700 1 -- 172.21.15.62:0/942726662 shutdown_connections 2024-04-20T22:04:00.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:00.455+0000 7f5e9e200700 1 --2- 172.21.15.62:0/942726662 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5e7c06ecc0 0x7f5e7c071170 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:04:00.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:00.455+0000 7f5e9e200700 1 --2- 172.21.15.62:0/942726662 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e981004c0 0x7f5e98195ef0 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:04:00.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:00.455+0000 7f5e9e200700 1 --2- 172.21.15.62:0/942726662 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e98100e60 0x7f5e98196430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:04:00.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:00.455+0000 7f5e9e200700 1 -- 172.21.15.62:0/942726662 >> 172.21.15.62:0/942726662 conn(0x7f5e980fbd80 msgr2=0x7f5e980fe170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:04:00.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:00.455+0000 7f5e9e200700 1 -- 172.21.15.62:0/942726662 shutdown_connections 2024-04-20T22:04:00.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:00.455+0000 7f5e9e200700 1 -- 172.21.15.62:0/942726662 wait complete. 2024-04-20T22:04:01.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:04:00 smithi189 ceph-mon[57022]: pgmap v862: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:04:01.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:04:00 smithi189 ceph-mon[57022]: from='client.35564 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:04:01.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:04:00 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/942726662' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:04:01.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:04:00 smithi062 ceph-mon[76322]: pgmap v862: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:04:01.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:04:00 smithi062 ceph-mon[76322]: from='client.35564 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:04:01.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:04:00 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/942726662' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:04:03.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:04:02 smithi189 ceph-mon[57022]: pgmap v863: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:04:03.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:04:02 smithi062 ceph-mon[76322]: pgmap v863: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:04:05.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:04:04 smithi189 ceph-mon[57022]: pgmap v864: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:04:05.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:04:04 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:04:05.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:04:04 smithi062 ceph-mon[76322]: pgmap v864: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:04:05.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:04:04 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:04:06.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:04:06 smithi189 ceph-mon[57022]: pgmap v865: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:04:07.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:04:06 smithi062 ceph-mon[76322]: pgmap v865: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:04:09.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:04:08 smithi189 ceph-mon[57022]: pgmap v866: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:04:09.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:04:08 smithi062 ceph-mon[76322]: pgmap v866: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:04:11.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:04:10 smithi189 ceph-mon[57022]: pgmap v867: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:04:11.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:04:10 smithi062 ceph-mon[76322]: pgmap v867: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:04:13.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:04:12 smithi189 ceph-mon[57022]: pgmap v868: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:04:13.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:04:12 smithi062 ceph-mon[76322]: pgmap v868: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:04:15.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:04:14 smithi062 ceph-mon[76322]: pgmap v869: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:04:15.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:04:14 smithi189 ceph-mon[57022]: pgmap v869: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:04:17.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:04:16 smithi189 ceph-mon[57022]: pgmap v870: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:04:17.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:04:16 smithi062 ceph-mon[76322]: pgmap v870: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:04:19.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:04:18 smithi189 ceph-mon[57022]: pgmap v871: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:04:19.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:04:18 smithi062 ceph-mon[76322]: pgmap v871: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:04:20.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:04:19 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:04:20.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:04:19 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:04:21.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:04:20 smithi189 ceph-mon[57022]: pgmap v872: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:04:21.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:04:20 smithi062 ceph-mon[76322]: pgmap v872: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:04:23.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:04:22 smithi189 ceph-mon[57022]: pgmap v873: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:04:23.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:04:22 smithi062 ceph-mon[76322]: pgmap v873: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:04:25.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:04:24 smithi189 ceph-mon[57022]: pgmap v874: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:04:25.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:04:24 smithi062 ceph-mon[76322]: pgmap v874: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:04:27.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:04:26 smithi189 ceph-mon[57022]: pgmap v875: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:04:27.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:04:26 smithi062 ceph-mon[76322]: pgmap v875: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:04:29.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:04:28 smithi189 ceph-mon[57022]: pgmap v876: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:04:29.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:04:28 smithi062 ceph-mon[76322]: pgmap v876: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:04:30.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:30.633+0000 7fefad000700 1 -- 172.21.15.62:0/3142601338 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fefa8068540 msgr2=0x7fefa80689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:04:30.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:30.633+0000 7fefad000700 1 --2- 172.21.15.62:0/3142601338 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fefa8068540 0x7fefa80689b0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fef94009910 tx=0x7fef94009c20 comp rx=0 tx=0).stop 2024-04-20T22:04:30.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:30.634+0000 7fefad000700 1 -- 172.21.15.62:0/3142601338 shutdown_connections 2024-04-20T22:04:30.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:30.634+0000 7fefad000700 1 --2- 172.21.15.62:0/3142601338 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fefa8068540 0x7fefa80689b0 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:04:30.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:30.634+0000 7fefad000700 1 --2- 172.21.15.62:0/3142601338 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fefa81085b0 0x7fefa8108980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:04:30.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:30.634+0000 7fefad000700 1 -- 172.21.15.62:0/3142601338 >> 172.21.15.62:0/3142601338 conn(0x7fefa8075130 msgr2=0x7fefa8075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:04:30.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:30.634+0000 7fefad000700 1 -- 172.21.15.62:0/3142601338 shutdown_connections 2024-04-20T22:04:30.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:30.634+0000 7fefad000700 1 -- 172.21.15.62:0/3142601338 wait complete. 2024-04-20T22:04:30.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:30.635+0000 7fefad000700 1 Processor -- start 2024-04-20T22:04:30.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:30.635+0000 7fefad000700 1 -- start start 2024-04-20T22:04:30.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:30.635+0000 7fefad000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fefa8068540 0x7fefa81980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:04:30.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:30.635+0000 7fefad000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fefa81085b0 0x7fefa81985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:04:30.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:30.635+0000 7fefad000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefa8198cd0 con 0x7fefa81085b0 2024-04-20T22:04:30.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:30.635+0000 7fefad000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefa819ca60 con 0x7fefa8068540 2024-04-20T22:04:30.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:30.636+0000 7fefa5400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fefa81085b0 0x7fefa81985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:04:30.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:30.636+0000 7fefa5400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fefa81085b0 0x7fefa81985f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52804/0 (socket says 172.21.15.62:52804) 2024-04-20T22:04:30.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:30.636+0000 7fefa5400700 1 -- 172.21.15.62:0/2187903321 learned_addr learned my addr 172.21.15.62:0/2187903321 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:04:30.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:30.636+0000 7fefa5e00700 1 --2- 172.21.15.62:0/2187903321 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fefa8068540 0x7fefa81980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:04:30.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:30.636+0000 7fefa5400700 1 -- 172.21.15.62:0/2187903321 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fefa8068540 msgr2=0x7fefa81980b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:04:30.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:30.636+0000 7fefa5400700 1 --2- 172.21.15.62:0/2187903321 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fefa8068540 0x7fefa81980b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:04:30.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:30.636+0000 7fefa5400700 1 -- 172.21.15.62:0/2187903321 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef940095f0 con 0x7fefa81085b0 2024-04-20T22:04:30.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:30.637+0000 7fefa5400700 1 --2- 172.21.15.62:0/2187903321 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fefa81085b0 0x7fefa81985f0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fef94006010 tx=0x7fef9400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:04:30.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:30.637+0000 7fef9ea00700 1 -- 172.21.15.62:0/2187903321 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fef9401d070 con 0x7fefa81085b0 2024-04-20T22:04:30.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:30.637+0000 7fef9ea00700 1 -- 172.21.15.62:0/2187903321 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fef94004fb0 con 0x7fefa81085b0 2024-04-20T22:04:30.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:30.637+0000 7fefad000700 1 -- 172.21.15.62:0/2187903321 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefa819cce0 con 0x7fefa81085b0 2024-04-20T22:04:30.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:30.637+0000 7fefad000700 1 -- 172.21.15.62:0/2187903321 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefa819d1d0 con 0x7fefa81085b0 2024-04-20T22:04:30.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:30.637+0000 7fef9ea00700 1 -- 172.21.15.62:0/2187903321 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fef94022ea0 con 0x7fefa81085b0 2024-04-20T22:04:30.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:30.638+0000 7fefad000700 1 -- 172.21.15.62:0/2187903321 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefa810aa00 con 0x7fefa81085b0 2024-04-20T22:04:30.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:30.640+0000 7fef9ea00700 1 -- 172.21.15.62:0/2187903321 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fef94022470 con 0x7fefa81085b0 2024-04-20T22:04:30.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:30.641+0000 7fef9ea00700 1 --2- 172.21.15.62:0/2187903321 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fef8c06ecc0 0x7fef8c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:04:30.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:30.641+0000 7fef9ea00700 1 -- 172.21.15.62:0/2187903321 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fef94090470 con 0x7fefa81085b0 2024-04-20T22:04:30.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:30.641+0000 7fefa5e00700 1 --2- 172.21.15.62:0/2187903321 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fef8c06ecc0 0x7fef8c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:04:30.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:30.642+0000 7fefa5e00700 1 --2- 172.21.15.62:0/2187903321 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fef8c06ecc0 0x7fef8c071170 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fef98005fd0 tx=0x7fef98005dd0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:04:30.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:30.646+0000 7fef9ea00700 1 -- 172.21.15.62:0/2187903321 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fef94059230 con 0x7fefa81085b0 2024-04-20T22:04:30.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:30.904+0000 7fefad000700 1 -- 172.21.15.62:0/2187903321 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fefa819d440 con 0x7fef8c06ecc0 2024-04-20T22:04:30.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:30.908+0000 7fef9ea00700 1 -- 172.21.15.62:0/2187903321 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fefa819d440 con 0x7fef8c06ecc0 2024-04-20T22:04:30.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:30.911+0000 7fefad000700 1 -- 172.21.15.62:0/2187903321 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fef8c06ecc0 msgr2=0x7fef8c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:04:30.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:30.911+0000 7fefad000700 1 --2- 172.21.15.62:0/2187903321 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fef8c06ecc0 0x7fef8c071170 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fef98005fd0 tx=0x7fef98005dd0 comp rx=0 tx=0).stop 2024-04-20T22:04:30.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:30.911+0000 7fefad000700 1 -- 172.21.15.62:0/2187903321 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fefa81085b0 msgr2=0x7fefa81985f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:04:30.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:30.911+0000 7fefad000700 1 --2- 172.21.15.62:0/2187903321 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fefa81085b0 0x7fefa81985f0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fef94006010 tx=0x7fef9400be20 comp rx=0 tx=0).stop 2024-04-20T22:04:30.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:30.911+0000 7fefad000700 1 -- 172.21.15.62:0/2187903321 shutdown_connections 2024-04-20T22:04:30.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:30.911+0000 7fefad000700 1 --2- 172.21.15.62:0/2187903321 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fef8c06ecc0 0x7fef8c071170 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:04:30.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:30.911+0000 7fefad000700 1 --2- 172.21.15.62:0/2187903321 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fefa8068540 0x7fefa81980b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:04:30.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:30.911+0000 7fefad000700 1 --2- 172.21.15.62:0/2187903321 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fefa81085b0 0x7fefa81985f0 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:04:30.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:30.911+0000 7fefad000700 1 -- 172.21.15.62:0/2187903321 >> 172.21.15.62:0/2187903321 conn(0x7fefa8075130 msgr2=0x7fefa8106390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:04:30.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:30.912+0000 7fefad000700 1 -- 172.21.15.62:0/2187903321 shutdown_connections 2024-04-20T22:04:30.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:30.912+0000 7fefad000700 1 -- 172.21.15.62:0/2187903321 wait complete. 2024-04-20T22:04:30.929 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T22:04:31.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.080+0000 7fae33200700 1 -- 172.21.15.62:0/2022558347 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fae2c10c8f0 msgr2=0x7fae2c10ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:04:31.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.080+0000 7fae33200700 1 --2- 172.21.15.62:0/2022558347 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fae2c10c8f0 0x7fae2c10ccc0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fae1c009910 tx=0x7fae1c009c20 comp rx=0 tx=0).stop 2024-04-20T22:04:31.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.080+0000 7fae33200700 1 -- 172.21.15.62:0/2022558347 shutdown_connections 2024-04-20T22:04:31.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.080+0000 7fae33200700 1 --2- 172.21.15.62:0/2022558347 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae2c106b80 0x7fae2c106ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:04:31.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.080+0000 7fae33200700 1 --2- 172.21.15.62:0/2022558347 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fae2c10c8f0 0x7fae2c10ccc0 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:04:31.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.080+0000 7fae33200700 1 -- 172.21.15.62:0/2022558347 >> 172.21.15.62:0/2022558347 conn(0x7fae2c076060 msgr2=0x7fae2c078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:04:31.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.081+0000 7fae33200700 1 -- 172.21.15.62:0/2022558347 shutdown_connections 2024-04-20T22:04:31.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.081+0000 7fae33200700 1 -- 172.21.15.62:0/2022558347 wait complete. 2024-04-20T22:04:31.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.082+0000 7fae33200700 1 Processor -- start 2024-04-20T22:04:31.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.082+0000 7fae33200700 1 -- start start 2024-04-20T22:04:31.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.082+0000 7fae33200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fae2c106b80 0x7fae2c19c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:04:31.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.082+0000 7fae33200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae2c10c8f0 0x7fae2c19ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:04:31.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.082+0000 7fae33200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae2c19d120 con 0x7fae2c106b80 2024-04-20T22:04:31.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.082+0000 7fae33200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae2c1a0eb0 con 0x7fae2c10c8f0 2024-04-20T22:04:31.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.083+0000 7fae2be00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fae2c106b80 0x7fae2c19c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:04:31.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.083+0000 7fae2be00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fae2c106b80 0x7fae2c19c500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52826/0 (socket says 172.21.15.62:52826) 2024-04-20T22:04:31.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.083+0000 7fae2be00700 1 -- 172.21.15.62:0/2429942756 learned_addr learned my addr 172.21.15.62:0/2429942756 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:04:31.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.083+0000 7fae2b400700 1 --2- 172.21.15.62:0/2429942756 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae2c10c8f0 0x7fae2c19ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:04:31.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.083+0000 7fae2be00700 1 -- 172.21.15.62:0/2429942756 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae2c10c8f0 msgr2=0x7fae2c19ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:04:31.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.083+0000 7fae2be00700 1 --2- 172.21.15.62:0/2429942756 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae2c10c8f0 0x7fae2c19ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:04:31.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.083+0000 7fae2be00700 1 -- 172.21.15.62:0/2429942756 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae1c0095f0 con 0x7fae2c106b80 2024-04-20T22:04:31.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.084+0000 7fae2be00700 1 --2- 172.21.15.62:0/2429942756 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fae2c106b80 0x7fae2c19c500 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fae1c0092e0 tx=0x7fae1c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:04:31.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.084+0000 7fae28c00700 1 -- 172.21.15.62:0/2429942756 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fae1c01d070 con 0x7fae2c106b80 2024-04-20T22:04:31.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.084+0000 7fae28c00700 1 -- 172.21.15.62:0/2429942756 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fae1c005e00 con 0x7fae2c106b80 2024-04-20T22:04:31.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.084+0000 7fae28c00700 1 -- 172.21.15.62:0/2429942756 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fae1c017400 con 0x7fae2c106b80 2024-04-20T22:04:31.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.084+0000 7fae33200700 1 -- 172.21.15.62:0/2429942756 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae2c1a1130 con 0x7fae2c106b80 2024-04-20T22:04:31.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.084+0000 7fae33200700 1 -- 172.21.15.62:0/2429942756 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae2c1a1620 con 0x7fae2c106b80 2024-04-20T22:04:31.085 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.085+0000 7fae33200700 1 -- 172.21.15.62:0/2429942756 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae2c109b20 con 0x7fae2c106b80 2024-04-20T22:04:31.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.086+0000 7fae28c00700 1 -- 172.21.15.62:0/2429942756 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fae1c00f460 con 0x7fae2c106b80 2024-04-20T22:04:31.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.087+0000 7fae28c00700 1 --2- 172.21.15.62:0/2429942756 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fae1006ecc0 0x7fae10071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:04:31.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.088+0000 7fae28c00700 1 -- 172.21.15.62:0/2429942756 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fae1c090330 con 0x7fae2c106b80 2024-04-20T22:04:31.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.088+0000 7fae2b400700 1 --2- 172.21.15.62:0/2429942756 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fae1006ecc0 0x7fae10071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:04:31.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.088+0000 7fae2b400700 1 --2- 172.21.15.62:0/2429942756 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fae1006ecc0 0x7fae10071170 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fae18005950 tx=0x7fae180058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:04:31.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.093+0000 7fae28c00700 1 -- 172.21.15.62:0/2429942756 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fae1c0590c0 con 0x7fae2c106b80 2024-04-20T22:04:31.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:04:30 smithi189 ceph-mon[57022]: pgmap v877: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:04:31.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:04:30 smithi062 ceph-mon[76322]: pgmap v877: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:04:31.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.349+0000 7fae33200700 1 -- 172.21.15.62:0/2429942756 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fae2c19d970 con 0x7fae1006ecc0 2024-04-20T22:04:31.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.351+0000 7fae28c00700 1 -- 172.21.15.62:0/2429942756 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fae2c19d970 con 0x7fae1006ecc0 2024-04-20T22:04:31.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.354+0000 7fae33200700 1 -- 172.21.15.62:0/2429942756 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fae1006ecc0 msgr2=0x7fae10071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:04:31.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.354+0000 7fae33200700 1 --2- 172.21.15.62:0/2429942756 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fae1006ecc0 0x7fae10071170 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fae18005950 tx=0x7fae180058e0 comp rx=0 tx=0).stop 2024-04-20T22:04:31.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.354+0000 7fae33200700 1 -- 172.21.15.62:0/2429942756 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fae2c106b80 msgr2=0x7fae2c19c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:04:31.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.354+0000 7fae33200700 1 --2- 172.21.15.62:0/2429942756 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fae2c106b80 0x7fae2c19c500 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fae1c0092e0 tx=0x7fae1c00be20 comp rx=0 tx=0).stop 2024-04-20T22:04:31.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.354+0000 7fae33200700 1 -- 172.21.15.62:0/2429942756 shutdown_connections 2024-04-20T22:04:31.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.354+0000 7fae33200700 1 --2- 172.21.15.62:0/2429942756 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fae1006ecc0 0x7fae10071170 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:04:31.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.354+0000 7fae33200700 1 --2- 172.21.15.62:0/2429942756 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fae2c106b80 0x7fae2c19c500 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:04:31.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.354+0000 7fae33200700 1 --2- 172.21.15.62:0/2429942756 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae2c10c8f0 0x7fae2c19ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:04:31.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.354+0000 7fae33200700 1 -- 172.21.15.62:0/2429942756 >> 172.21.15.62:0/2429942756 conn(0x7fae2c076060 msgr2=0x7fae2c077aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:04:31.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.355+0000 7fae33200700 1 -- 172.21.15.62:0/2429942756 shutdown_connections 2024-04-20T22:04:31.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.355+0000 7fae33200700 1 -- 172.21.15.62:0/2429942756 wait complete. 2024-04-20T22:04:31.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.519+0000 7ff108000700 1 -- 172.21.15.62:0/3529908568 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff100108470 msgr2=0x7ff100108840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:04:31.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.519+0000 7ff108000700 1 --2- 172.21.15.62:0/3529908568 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff100108470 0x7ff100108840 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7ff0f4009910 tx=0x7ff0f4009c20 comp rx=0 tx=0).stop 2024-04-20T22:04:31.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.520+0000 7ff108000700 1 -- 172.21.15.62:0/3529908568 shutdown_connections 2024-04-20T22:04:31.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.520+0000 7ff108000700 1 --2- 172.21.15.62:0/3529908568 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff100102470 0x7ff1001028e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:04:31.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.520+0000 7ff108000700 1 --2- 172.21.15.62:0/3529908568 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff100108470 0x7ff100108840 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:04:31.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.520+0000 7ff108000700 1 -- 172.21.15.62:0/3529908568 >> 172.21.15.62:0/3529908568 conn(0x7ff1000fdf90 msgr2=0x7ff1001003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:04:31.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.520+0000 7ff108000700 1 -- 172.21.15.62:0/3529908568 shutdown_connections 2024-04-20T22:04:31.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.520+0000 7ff108000700 1 -- 172.21.15.62:0/3529908568 wait complete. 2024-04-20T22:04:31.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.521+0000 7ff108000700 1 Processor -- start 2024-04-20T22:04:31.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.521+0000 7ff108000700 1 -- start start 2024-04-20T22:04:31.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.521+0000 7ff108000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff100102470 0x7ff1001981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:04:31.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.521+0000 7ff108000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff100108470 0x7ff100198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:04:31.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.521+0000 7ff108000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff100198e10 con 0x7ff100102470 2024-04-20T22:04:31.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.521+0000 7ff108000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff10019cba0 con 0x7ff100108470 2024-04-20T22:04:31.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.522+0000 7ff105600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff100102470 0x7ff1001981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:04:31.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.522+0000 7ff105600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff100102470 0x7ff1001981f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49106/0 (socket says 172.21.15.62:49106) 2024-04-20T22:04:31.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.522+0000 7ff105600700 1 -- 172.21.15.62:0/2642407050 learned_addr learned my addr 172.21.15.62:0/2642407050 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:04:31.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.522+0000 7ff104c00700 1 --2- 172.21.15.62:0/2642407050 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff100108470 0x7ff100198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:04:31.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.523+0000 7ff105600700 1 -- 172.21.15.62:0/2642407050 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff100108470 msgr2=0x7ff100198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:04:31.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.523+0000 7ff105600700 1 --2- 172.21.15.62:0/2642407050 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff100108470 0x7ff100198730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:04:31.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.523+0000 7ff105600700 1 -- 172.21.15.62:0/2642407050 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0f40095f0 con 0x7ff100102470 2024-04-20T22:04:31.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.523+0000 7ff105600700 1 --2- 172.21.15.62:0/2642407050 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff100102470 0x7ff1001981f0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7ff0f4006010 tx=0x7ff0f400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:04:31.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.524+0000 7ff0fe000700 1 -- 172.21.15.62:0/2642407050 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff0f401d070 con 0x7ff100102470 2024-04-20T22:04:31.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.524+0000 7ff0fe000700 1 -- 172.21.15.62:0/2642407050 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff0f4005dc0 con 0x7ff100102470 2024-04-20T22:04:31.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.524+0000 7ff0fe000700 1 -- 172.21.15.62:0/2642407050 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff0f4017400 con 0x7ff100102470 2024-04-20T22:04:31.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.524+0000 7ff108000700 1 -- 172.21.15.62:0/2642407050 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff10019ce20 con 0x7ff100102470 2024-04-20T22:04:31.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.524+0000 7ff108000700 1 -- 172.21.15.62:0/2642407050 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff10019d310 con 0x7ff100102470 2024-04-20T22:04:31.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.525+0000 7ff108000700 1 -- 172.21.15.62:0/2642407050 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff10010a9e0 con 0x7ff100102470 2024-04-20T22:04:31.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.526+0000 7ff0fe000700 1 -- 172.21.15.62:0/2642407050 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff0f4003a60 con 0x7ff100102470 2024-04-20T22:04:31.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.527+0000 7ff0fe000700 1 --2- 172.21.15.62:0/2642407050 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff0e806ecc0 0x7ff0e8071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:04:31.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.527+0000 7ff0fe000700 1 -- 172.21.15.62:0/2642407050 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7ff0f4090280 con 0x7ff100102470 2024-04-20T22:04:31.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.527+0000 7ff104c00700 1 --2- 172.21.15.62:0/2642407050 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff0e806ecc0 0x7ff0e8071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:04:31.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.528+0000 7ff104c00700 1 --2- 172.21.15.62:0/2642407050 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff0e806ecc0 0x7ff0e8071170 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7ff100199810 tx=0x7ff0ec0073d0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:04:31.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.532+0000 7ff0fe000700 1 -- 172.21.15.62:0/2642407050 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7ff0f4059010 con 0x7ff100102470 2024-04-20T22:04:31.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.795+0000 7ff108000700 1 -- 172.21.15.62:0/2642407050 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7ff1001995c0 con 0x7ff0e806ecc0 2024-04-20T22:04:31.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.813+0000 7ff0fe000700 1 -- 172.21.15.62:0/2642407050 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7ff1001995c0 con 0x7ff0e806ecc0 2024-04-20T22:04:31.813 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T22:04:31.813 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (30m) 6m ago 44m 15.3M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T22:04:31.813 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (44m) 6m ago 44m 10.4M - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T22:04:31.813 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (43m) 6m ago 43m 10.1M - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T22:04:31.813 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (26m) 6m ago 44m 6660k - 19.0.0-3133-ga8c84f59 76bc6b91b726 2825d5ba11c6 2024-04-20T22:04:31.813 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (26m) 6m ago 43m 6665k - 19.0.0-3133-ga8c84f59 76bc6b91b726 20f746fa1bba 2024-04-20T22:04:31.813 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (29m) 6m ago 43m 73.5M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T22:04:31.813 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (21m) 6m ago 36m 18.5M - 19.0.0-3133-ga8c84f59 76bc6b91b726 de80bc90a683 2024-04-20T22:04:31.813 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (22m) 6m ago 37m 18.7M - 19.0.0-3133-ga8c84f59 76bc6b91b726 cd793b7ee98e 2024-04-20T22:04:31.813 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (6m) 6m ago 37m 11.0M - 19.0.0-3133-ga8c84f59 76bc6b91b726 0e7b0bc4d1ab 2024-04-20T22:04:31.813 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (36m) 6m ago 36m 18.6M - 18.2.0 dc2bc1663786 f2eec2ee8951 2024-04-20T22:04:31.813 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (33m) 6m ago 46m 513M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T22:04:31.813 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (31m) 6m ago 42m 459M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T22:04:31.813 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (28m) 6m ago 46m 71.7M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 019eb1479485 2024-04-20T22:04:31.814 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (27m) 6m ago 42m 51.0M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 55aac560aab6 2024-04-20T22:04:31.814 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (31m) 6m ago 44m 8472k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T22:04:31.814 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (31m) 6m ago 43m 8048k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T22:04:31.814 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (25m) 6m ago 41m 91.2M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 ee2bb901f26e 2024-04-20T22:04:31.814 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (25m) 6m ago 41m 94.6M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 9075152d4162 2024-04-20T22:04:31.814 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (24m) 6m ago 40m 78.1M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 2d4f3e38dc0e 2024-04-20T22:04:31.814 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (24m) 6m ago 40m 92.6M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 8f97b17ae0eb 2024-04-20T22:04:31.814 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (23m) 6m ago 39m 95.6M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 653d8079967d 2024-04-20T22:04:31.814 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (22m) 6m ago 38m 89.2M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 596dded14c12 2024-04-20T22:04:31.814 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (30m) 6m ago 43m 82.1M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T22:04:31.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.816+0000 7ff108000700 1 -- 172.21.15.62:0/2642407050 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff0e806ecc0 msgr2=0x7ff0e8071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:04:31.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.816+0000 7ff108000700 1 --2- 172.21.15.62:0/2642407050 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff0e806ecc0 0x7ff0e8071170 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7ff100199810 tx=0x7ff0ec0073d0 comp rx=0 tx=0).stop 2024-04-20T22:04:31.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.816+0000 7ff108000700 1 -- 172.21.15.62:0/2642407050 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff100102470 msgr2=0x7ff1001981f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:04:31.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.816+0000 7ff108000700 1 --2- 172.21.15.62:0/2642407050 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff100102470 0x7ff1001981f0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7ff0f4006010 tx=0x7ff0f400bd70 comp rx=0 tx=0).stop 2024-04-20T22:04:31.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.816+0000 7ff108000700 1 -- 172.21.15.62:0/2642407050 shutdown_connections 2024-04-20T22:04:31.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.816+0000 7ff108000700 1 --2- 172.21.15.62:0/2642407050 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff0e806ecc0 0x7ff0e8071170 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:04:31.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.816+0000 7ff108000700 1 --2- 172.21.15.62:0/2642407050 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff100102470 0x7ff1001981f0 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:04:31.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.816+0000 7ff108000700 1 --2- 172.21.15.62:0/2642407050 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff100108470 0x7ff100198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:04:31.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.816+0000 7ff108000700 1 -- 172.21.15.62:0/2642407050 >> 172.21.15.62:0/2642407050 conn(0x7ff1000fdf90 msgr2=0x7ff1001002c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:04:31.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.816+0000 7ff108000700 1 -- 172.21.15.62:0/2642407050 shutdown_connections 2024-04-20T22:04:31.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.816+0000 7ff108000700 1 -- 172.21.15.62:0/2642407050 wait complete. 2024-04-20T22:04:31.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.979+0000 7fdd58600700 1 -- 172.21.15.62:0/1835870712 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd50100e80 msgr2=0x7fdd50104e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:04:31.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.979+0000 7fdd58600700 1 --2- 172.21.15.62:0/1835870712 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd50100e80 0x7fdd50104e50 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fdd3c009910 tx=0x7fdd3c009c20 comp rx=0 tx=0).stop 2024-04-20T22:04:31.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.979+0000 7fdd58600700 1 -- 172.21.15.62:0/1835870712 shutdown_connections 2024-04-20T22:04:31.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.979+0000 7fdd58600700 1 --2- 172.21.15.62:0/1835870712 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd50100e80 0x7fdd50104e50 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:04:31.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.979+0000 7fdd58600700 1 --2- 172.21.15.62:0/1835870712 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd501004e0 0x7fdd501008b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:04:31.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.979+0000 7fdd58600700 1 -- 172.21.15.62:0/1835870712 >> 172.21.15.62:0/1835870712 conn(0x7fdd500fbda0 msgr2=0x7fdd500fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:04:31.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.979+0000 7fdd58600700 1 -- 172.21.15.62:0/1835870712 shutdown_connections 2024-04-20T22:04:31.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.979+0000 7fdd58600700 1 -- 172.21.15.62:0/1835870712 wait complete. 2024-04-20T22:04:31.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.980+0000 7fdd58600700 1 Processor -- start 2024-04-20T22:04:31.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.981+0000 7fdd58600700 1 -- start start 2024-04-20T22:04:31.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.981+0000 7fdd58600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd501004e0 0x7fdd50198110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:04:31.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.981+0000 7fdd58600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd50100e80 0x7fdd50198650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:04:31.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.981+0000 7fdd58600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd50198d30 con 0x7fdd50100e80 2024-04-20T22:04:31.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.981+0000 7fdd58600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd5019cac0 con 0x7fdd501004e0 2024-04-20T22:04:31.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.981+0000 7fdd55200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd50100e80 0x7fdd50198650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:04:31.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.981+0000 7fdd55200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd50100e80 0x7fdd50198650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49124/0 (socket says 172.21.15.62:49124) 2024-04-20T22:04:31.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.981+0000 7fdd55200700 1 -- 172.21.15.62:0/3755479646 learned_addr learned my addr 172.21.15.62:0/3755479646 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:04:31.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.982+0000 7fdd55c00700 1 --2- 172.21.15.62:0/3755479646 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd501004e0 0x7fdd50198110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:04:31.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.982+0000 7fdd55200700 1 -- 172.21.15.62:0/3755479646 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd501004e0 msgr2=0x7fdd50198110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:04:31.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.982+0000 7fdd55200700 1 --2- 172.21.15.62:0/3755479646 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd501004e0 0x7fdd50198110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:04:31.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.982+0000 7fdd55200700 1 -- 172.21.15.62:0/3755479646 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd3c0095f0 con 0x7fdd50100e80 2024-04-20T22:04:31.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.982+0000 7fdd55200700 1 --2- 172.21.15.62:0/3755479646 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd50100e80 0x7fdd50198650 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fdd3c0092e0 tx=0x7fdd3c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:04:31.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.982+0000 7fdd4e000700 1 -- 172.21.15.62:0/3755479646 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdd3c01d070 con 0x7fdd50100e80 2024-04-20T22:04:31.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.982+0000 7fdd4e000700 1 -- 172.21.15.62:0/3755479646 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdd3c005dc0 con 0x7fdd50100e80 2024-04-20T22:04:31.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.982+0000 7fdd4e000700 1 -- 172.21.15.62:0/3755479646 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdd3c017400 con 0x7fdd50100e80 2024-04-20T22:04:31.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.982+0000 7fdd58600700 1 -- 172.21.15.62:0/3755479646 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd5019cd40 con 0x7fdd50100e80 2024-04-20T22:04:31.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.983+0000 7fdd58600700 1 -- 172.21.15.62:0/3755479646 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd5019d230 con 0x7fdd50100e80 2024-04-20T22:04:31.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.984+0000 7fdd58600700 1 -- 172.21.15.62:0/3755479646 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd501045d0 con 0x7fdd50100e80 2024-04-20T22:04:31.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.985+0000 7fdd4e000700 1 -- 172.21.15.62:0/3755479646 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fdd3c00f460 con 0x7fdd50100e80 2024-04-20T22:04:31.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.986+0000 7fdd4e000700 1 --2- 172.21.15.62:0/3755479646 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fdd3806ecc0 0x7fdd38071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:04:31.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.986+0000 7fdd4e000700 1 -- 172.21.15.62:0/3755479646 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fdd3c090360 con 0x7fdd50100e80 2024-04-20T22:04:31.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.986+0000 7fdd55c00700 1 --2- 172.21.15.62:0/3755479646 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fdd3806ecc0 0x7fdd38071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:04:31.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.987+0000 7fdd55c00700 1 --2- 172.21.15.62:0/3755479646 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fdd3806ecc0 0x7fdd38071170 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fdd44009c10 tx=0x7fdd44009040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:04:31.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:31.991+0000 7fdd4e000700 1 -- 172.21.15.62:0/3755479646 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fdd3c0590f0 con 0x7fdd50100e80 2024-04-20T22:04:32.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:04:31 smithi189 ceph-mon[57022]: from='client.35572 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:04:32.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:04:31 smithi062 ceph-mon[76322]: from='client.35572 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:04:32.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:32.319+0000 7fdd58600700 1 -- 172.21.15.62:0/3755479646 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fdd5004f2a0 con 0x7fdd50100e80 2024-04-20T22:04:32.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:32.320+0000 7fdd4e000700 1 -- 172.21.15.62:0/3755479646 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7fdd3c05c7b0 con 0x7fdd50100e80 2024-04-20T22:04:32.320 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:04:32.320 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T22:04:32.320 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:04:32.320 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:04:32.320 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T22:04:32.320 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:04:32.320 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:04:32.320 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T22:04:32.320 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 6 2024-04-20T22:04:32.320 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:04:32.321 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T22:04:32.321 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 1, 2024-04-20T22:04:32.321 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 3 2024-04-20T22:04:32.321 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:04:32.321 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T22:04:32.321 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 1, 2024-04-20T22:04:32.321 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 13 2024-04-20T22:04:32.321 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T22:04:32.321 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:04:32.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:32.322+0000 7fdd58600700 1 -- 172.21.15.62:0/3755479646 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fdd3806ecc0 msgr2=0x7fdd38071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:04:32.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:32.322+0000 7fdd58600700 1 --2- 172.21.15.62:0/3755479646 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fdd3806ecc0 0x7fdd38071170 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fdd44009c10 tx=0x7fdd44009040 comp rx=0 tx=0).stop 2024-04-20T22:04:32.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:32.322+0000 7fdd58600700 1 -- 172.21.15.62:0/3755479646 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd50100e80 msgr2=0x7fdd50198650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:04:32.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:32.322+0000 7fdd58600700 1 --2- 172.21.15.62:0/3755479646 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd50100e80 0x7fdd50198650 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fdd3c0092e0 tx=0x7fdd3c00be20 comp rx=0 tx=0).stop 2024-04-20T22:04:32.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:32.323+0000 7fdd58600700 1 -- 172.21.15.62:0/3755479646 shutdown_connections 2024-04-20T22:04:32.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:32.323+0000 7fdd58600700 1 --2- 172.21.15.62:0/3755479646 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fdd3806ecc0 0x7fdd38071170 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:04:32.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:32.323+0000 7fdd58600700 1 --2- 172.21.15.62:0/3755479646 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd501004e0 0x7fdd50198110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:04:32.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:32.323+0000 7fdd58600700 1 --2- 172.21.15.62:0/3755479646 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd50100e80 0x7fdd50198650 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:04:32.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:32.323+0000 7fdd58600700 1 -- 172.21.15.62:0/3755479646 >> 172.21.15.62:0/3755479646 conn(0x7fdd500fbda0 msgr2=0x7fdd500fe150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:04:32.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:32.323+0000 7fdd58600700 1 -- 172.21.15.62:0/3755479646 shutdown_connections 2024-04-20T22:04:32.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:32.323+0000 7fdd58600700 1 -- 172.21.15.62:0/3755479646 wait complete. 2024-04-20T22:04:32.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:32.473+0000 7ff31b800700 1 -- 172.21.15.62:0/396034107 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff314100e80 msgr2=0x7ff314104e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:04:32.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:32.473+0000 7ff31b800700 1 --2- 172.21.15.62:0/396034107 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff314100e80 0x7ff314104e50 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7ff300009910 tx=0x7ff300009c20 comp rx=0 tx=0).stop 2024-04-20T22:04:32.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:32.474+0000 7ff31b800700 1 -- 172.21.15.62:0/396034107 shutdown_connections 2024-04-20T22:04:32.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:32.474+0000 7ff31b800700 1 --2- 172.21.15.62:0/396034107 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff314100e80 0x7ff314104e50 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:04:32.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:32.474+0000 7ff31b800700 1 --2- 172.21.15.62:0/396034107 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3141004e0 0x7ff3141008b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:04:32.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:32.474+0000 7ff31b800700 1 -- 172.21.15.62:0/396034107 >> 172.21.15.62:0/396034107 conn(0x7ff3140fbda0 msgr2=0x7ff3140fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:04:32.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:32.474+0000 7ff31b800700 1 -- 172.21.15.62:0/396034107 shutdown_connections 2024-04-20T22:04:32.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:32.474+0000 7ff31b800700 1 -- 172.21.15.62:0/396034107 wait complete. 2024-04-20T22:04:32.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:32.475+0000 7ff31b800700 1 Processor -- start 2024-04-20T22:04:32.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:32.475+0000 7ff31b800700 1 -- start start 2024-04-20T22:04:32.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:32.475+0000 7ff31b800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3141004e0 0x7ff3141980f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:04:32.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:32.475+0000 7ff31b800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff314100e80 0x7ff314198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:04:32.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:32.475+0000 7ff31b800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff314198d10 con 0x7ff314100e80 2024-04-20T22:04:32.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:32.475+0000 7ff31b800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff31419caa0 con 0x7ff3141004e0 2024-04-20T22:04:32.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:32.476+0000 7ff313e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff314100e80 0x7ff314198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:04:32.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:32.476+0000 7ff313e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff314100e80 0x7ff314198630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49148/0 (socket says 172.21.15.62:49148) 2024-04-20T22:04:32.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:32.476+0000 7ff313e00700 1 -- 172.21.15.62:0/3143710006 learned_addr learned my addr 172.21.15.62:0/3143710006 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:04:32.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:32.476+0000 7ff313e00700 1 -- 172.21.15.62:0/3143710006 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3141004e0 msgr2=0x7ff3141980f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:04:32.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:32.476+0000 7ff318e00700 1 --2- 172.21.15.62:0/3143710006 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3141004e0 0x7ff3141980f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:04:32.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:32.476+0000 7ff313e00700 1 --2- 172.21.15.62:0/3143710006 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3141004e0 0x7ff3141980f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:04:32.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:32.476+0000 7ff313e00700 1 -- 172.21.15.62:0/3143710006 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3000095f0 con 0x7ff314100e80 2024-04-20T22:04:32.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:32.476+0000 7ff313e00700 1 --2- 172.21.15.62:0/3143710006 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff314100e80 0x7ff314198630 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7ff30000bda0 tx=0x7ff30000be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:04:32.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:32.477+0000 7ff311600700 1 -- 172.21.15.62:0/3143710006 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff30001d070 con 0x7ff314100e80 2024-04-20T22:04:32.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:32.477+0000 7ff311600700 1 -- 172.21.15.62:0/3143710006 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff300004fb0 con 0x7ff314100e80 2024-04-20T22:04:32.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:32.477+0000 7ff311600700 1 -- 172.21.15.62:0/3143710006 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff30000f460 con 0x7ff314100e80 2024-04-20T22:04:32.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:32.477+0000 7ff31b800700 1 -- 172.21.15.62:0/3143710006 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff31419cd20 con 0x7ff314100e80 2024-04-20T22:04:32.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:32.477+0000 7ff31b800700 1 -- 172.21.15.62:0/3143710006 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff31419d210 con 0x7ff314100e80 2024-04-20T22:04:32.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:32.478+0000 7ff31b800700 1 -- 172.21.15.62:0/3143710006 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3141045d0 con 0x7ff314100e80 2024-04-20T22:04:32.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:32.479+0000 7ff311600700 1 -- 172.21.15.62:0/3143710006 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff300022800 con 0x7ff314100e80 2024-04-20T22:04:32.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:32.480+0000 7ff311600700 1 --2- 172.21.15.62:0/3143710006 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff2fc06ecc0 0x7ff2fc071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:04:32.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:32.480+0000 7ff311600700 1 -- 172.21.15.62:0/3143710006 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7ff300090390 con 0x7ff314100e80 2024-04-20T22:04:32.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:32.480+0000 7ff318e00700 1 --2- 172.21.15.62:0/3143710006 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff2fc06ecc0 0x7ff2fc071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:04:32.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:32.481+0000 7ff318e00700 1 --2- 172.21.15.62:0/3143710006 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff2fc06ecc0 0x7ff2fc071170 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7ff308005950 tx=0x7ff308005cf0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:04:32.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:32.485+0000 7ff311600700 1 -- 172.21.15.62:0/3143710006 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7ff3000592f0 con 0x7ff314100e80 2024-04-20T22:04:32.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:32.765+0000 7ff31b800700 1 -- 172.21.15.62:0/3143710006 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7ff31402cc30 con 0x7ff314100e80 2024-04-20T22:04:32.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:32.766+0000 7ff311600700 1 -- 172.21.15.62:0/3143710006 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 27 v27) v1 ==== 76+0+1888 (secure 0 0 0) 0x7ff300027770 con 0x7ff314100e80 2024-04-20T22:04:32.766 INFO:teuthology.orchestra.run.smithi062.stdout:e27 2024-04-20T22:04:32.766 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T22:04:32.766 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:04:32.766 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T22:04:32.766 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:04:32.766 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T22:04:32.766 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T22:04:32.766 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 20 2024-04-20T22:04:32.766 INFO:teuthology.orchestra.run.smithi062.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-20T22:04:32.767 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T22:04:32.767 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T21:42:01.559061+0000 2024-04-20T22:04:32.767 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T22:04:32.767 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T22:04:32.767 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T22:04:32.767 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T22:04:32.767 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T22:04:32.767 INFO:teuthology.orchestra.run.smithi062.stdout:max_xattr_size 65536 2024-04-20T22:04:32.767 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T22:04:32.767 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T22:04:32.767 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 81 2024-04-20T22:04:32.767 INFO:teuthology.orchestra.run.smithi062.stdout:snap_rstat 1 2024-04-20T22:04:32.768 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:04:32.768 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 2 2024-04-20T22:04:32.768 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1 2024-04-20T22:04:32.768 INFO:teuthology.orchestra.run.smithi062.stdout:up {} 2024-04-20T22:04:32.768 INFO:teuthology.orchestra.run.smithi062.stdout:failed 0,1 2024-04-20T22:04:32.768 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T22:04:32.768 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T22:04:32.768 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T22:04:32.768 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T22:04:32.768 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data enabled 2024-04-20T22:04:32.768 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T22:04:32.768 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T22:04:32.769 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 1 2024-04-20T22:04:32.769 INFO:teuthology.orchestra.run.smithi062.stdout:qdb_cluster leader: 0 members: 2024-04-20T22:04:32.769 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:04:32.769 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:04:32.769 INFO:teuthology.orchestra.run.smithi062.stdout:Standby daemons: 2024-04-20T22:04:32.769 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:04:32.769 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ryqblb{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6826/4113916205,v1:172.21.15.62:6827/4113916205] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:04:32.769 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ozizwi{-1:34476} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6828/3801975165,v1:172.21.15.62:6829/3801975165] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:04:32.769 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.xbgspw{-1:44285} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6826/3996040526,v1:172.21.15.189:6827/3996040526] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T22:04:32.769 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.dvimer{-1:44727} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6824/3576699232,v1:172.21.15.189:6825/3576699232] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:04:32.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:32.769+0000 7ff31b800700 1 -- 172.21.15.62:0/3143710006 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff2fc06ecc0 msgr2=0x7ff2fc071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:04:32.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:32.769+0000 7ff31b800700 1 --2- 172.21.15.62:0/3143710006 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff2fc06ecc0 0x7ff2fc071170 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7ff308005950 tx=0x7ff308005cf0 comp rx=0 tx=0).stop 2024-04-20T22:04:32.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:32.769+0000 7ff31b800700 1 -- 172.21.15.62:0/3143710006 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff314100e80 msgr2=0x7ff314198630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:04:32.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:32.769+0000 7ff31b800700 1 --2- 172.21.15.62:0/3143710006 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff314100e80 0x7ff314198630 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7ff30000bda0 tx=0x7ff30000be80 comp rx=0 tx=0).stop 2024-04-20T22:04:32.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:32.769+0000 7ff31b800700 1 -- 172.21.15.62:0/3143710006 shutdown_connections 2024-04-20T22:04:32.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:32.769+0000 7ff31b800700 1 --2- 172.21.15.62:0/3143710006 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff2fc06ecc0 0x7ff2fc071170 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:04:32.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:32.769+0000 7ff31b800700 1 --2- 172.21.15.62:0/3143710006 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3141004e0 0x7ff3141980f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:04:32.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:32.769+0000 7ff31b800700 1 --2- 172.21.15.62:0/3143710006 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff314100e80 0x7ff314198630 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:04:32.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:32.769+0000 7ff31b800700 1 -- 172.21.15.62:0/3143710006 >> 172.21.15.62:0/3143710006 conn(0x7ff3140fbda0 msgr2=0x7ff3140fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:04:32.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:32.769+0000 7ff31b800700 1 -- 172.21.15.62:0/3143710006 shutdown_connections 2024-04-20T22:04:32.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:32.769+0000 7ff31b800700 1 -- 172.21.15.62:0/3143710006 wait complete. 2024-04-20T22:04:32.771 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 27 2024-04-20T22:04:32.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:32.924+0000 7fdda4400700 1 -- 172.21.15.62:0/2119631243 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd9c101000 msgr2=0x7fdd9c1013d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:04:32.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:32.924+0000 7fdda4400700 1 --2- 172.21.15.62:0/2119631243 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd9c101000 0x7fdd9c1013d0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fdd9000b0a0 tx=0x7fdd9000b3b0 comp rx=0 tx=0).stop 2024-04-20T22:04:32.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:32.924+0000 7fdda4400700 1 -- 172.21.15.62:0/2119631243 shutdown_connections 2024-04-20T22:04:32.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:32.924+0000 7fdda4400700 1 --2- 172.21.15.62:0/2119631243 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd9c068540 0x7fdd9c0689b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:04:32.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:32.924+0000 7fdda4400700 1 --2- 172.21.15.62:0/2119631243 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd9c101000 0x7fdd9c1013d0 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:04:32.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:32.924+0000 7fdda4400700 1 -- 172.21.15.62:0/2119631243 >> 172.21.15.62:0/2119631243 conn(0x7fdd9c075130 msgr2=0x7fdd9c075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:04:32.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:32.924+0000 7fdda4400700 1 -- 172.21.15.62:0/2119631243 shutdown_connections 2024-04-20T22:04:32.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:32.924+0000 7fdda4400700 1 -- 172.21.15.62:0/2119631243 wait complete. 2024-04-20T22:04:32.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:32.925+0000 7fdda4400700 1 Processor -- start 2024-04-20T22:04:32.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:32.925+0000 7fdda4400700 1 -- start start 2024-04-20T22:04:32.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:32.926+0000 7fdda4400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd9c068540 0x7fdd9c1980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:04:32.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:32.926+0000 7fdda4400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd9c101000 0x7fdd9c1985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:04:32.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:32.926+0000 7fdda4400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd9c198cd0 con 0x7fdd9c101000 2024-04-20T22:04:32.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:32.926+0000 7fdda4400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd9c19ca60 con 0x7fdd9c068540 2024-04-20T22:04:32.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:32.926+0000 7fdda1000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd9c101000 0x7fdd9c1985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:04:32.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:32.926+0000 7fdda1000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd9c101000 0x7fdd9c1985f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49162/0 (socket says 172.21.15.62:49162) 2024-04-20T22:04:32.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:32.926+0000 7fdda1000700 1 -- 172.21.15.62:0/4225375815 learned_addr learned my addr 172.21.15.62:0/4225375815 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:04:32.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:32.926+0000 7fdda1a00700 1 --2- 172.21.15.62:0/4225375815 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd9c068540 0x7fdd9c1980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:04:32.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:32.927+0000 7fdda1000700 1 -- 172.21.15.62:0/4225375815 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd9c068540 msgr2=0x7fdd9c1980b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:04:32.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:32.927+0000 7fdda1000700 1 --2- 172.21.15.62:0/4225375815 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd9c068540 0x7fdd9c1980b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:04:32.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:32.927+0000 7fdda1000700 1 -- 172.21.15.62:0/4225375815 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd90009d00 con 0x7fdd9c101000 2024-04-20T22:04:32.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:32.927+0000 7fdda1000700 1 --2- 172.21.15.62:0/4225375815 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd9c101000 0x7fdd9c1985f0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fdd8800c8a0 tx=0x7fdd8800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:04:32.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:32.929+0000 7fdd9a000700 1 -- 172.21.15.62:0/4225375815 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdd8800cea0 con 0x7fdd9c101000 2024-04-20T22:04:32.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:32.929+0000 7fdd9a000700 1 -- 172.21.15.62:0/4225375815 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdd88004830 con 0x7fdd9c101000 2024-04-20T22:04:32.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:32.929+0000 7fdda4400700 1 -- 172.21.15.62:0/4225375815 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd9c19cd40 con 0x7fdd9c101000 2024-04-20T22:04:32.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:32.929+0000 7fdd9a000700 1 -- 172.21.15.62:0/4225375815 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdd880055b0 con 0x7fdd9c101000 2024-04-20T22:04:32.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:32.929+0000 7fdda4400700 1 -- 172.21.15.62:0/4225375815 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd9c19d290 con 0x7fdd9c101000 2024-04-20T22:04:32.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:32.930+0000 7fdda4400700 1 -- 172.21.15.62:0/4225375815 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd9c04f2a0 con 0x7fdd9c101000 2024-04-20T22:04:32.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:32.931+0000 7fdd9a000700 1 -- 172.21.15.62:0/4225375815 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fdd880049a0 con 0x7fdd9c101000 2024-04-20T22:04:32.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:32.932+0000 7fdd9a000700 1 --2- 172.21.15.62:0/4225375815 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fdd8406ecc0 0x7fdd84071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:04:32.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:32.932+0000 7fdd9a000700 1 -- 172.21.15.62:0/4225375815 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fdd8808eaf0 con 0x7fdd9c101000 2024-04-20T22:04:32.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:32.933+0000 7fdda1a00700 1 --2- 172.21.15.62:0/4225375815 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fdd8406ecc0 0x7fdd84071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:04:32.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:32.933+0000 7fdda1a00700 1 --2- 172.21.15.62:0/4225375815 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fdd8406ecc0 0x7fdd84071170 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fdd90009480 tx=0x7fdd9000b7f0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:04:32.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:32.937+0000 7fdd9a000700 1 -- 172.21.15.62:0/4225375815 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fdd88057880 con 0x7fdd9c101000 2024-04-20T22:04:33.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:04:32 smithi062 ceph-mon[76322]: from='client.35576 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:04:33.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:04:32 smithi062 ceph-mon[76322]: pgmap v878: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:04:33.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:04:32 smithi062 ceph-mon[76322]: from='client.35580 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:04:33.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:04:32 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/3755479646' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:04:33.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:04:32 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/3143710006' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:04:33.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:33.183+0000 7fdda4400700 1 -- 172.21.15.62:0/4225375815 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fdd9c19d710 con 0x7fdd8406ecc0 2024-04-20T22:04:33.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:33.186+0000 7fdd9a000700 1 -- 172.21.15.62:0/4225375815 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fdd9c19d710 con 0x7fdd8406ecc0 2024-04-20T22:04:33.186 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:04:33.186 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T22:04:33.186 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T22:04:33.186 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-20T22:04:33.186 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T22:04:33.186 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr", 2024-04-20T22:04:33.186 INFO:teuthology.orchestra.run.smithi062.stdout: "crash", 2024-04-20T22:04:33.186 INFO:teuthology.orchestra.run.smithi062.stdout: "mon", 2024-04-20T22:04:33.186 INFO:teuthology.orchestra.run.smithi062.stdout: "osd" 2024-04-20T22:04:33.187 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T22:04:33.187 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "15/23 daemons upgraded", 2024-04-20T22:04:33.187 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading mds daemons", 2024-04-20T22:04:33.187 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T22:04:33.187 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:04:33.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:33.189+0000 7fdda4400700 1 -- 172.21.15.62:0/4225375815 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fdd8406ecc0 msgr2=0x7fdd84071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:04:33.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:33.189+0000 7fdda4400700 1 --2- 172.21.15.62:0/4225375815 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fdd8406ecc0 0x7fdd84071170 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fdd90009480 tx=0x7fdd9000b7f0 comp rx=0 tx=0).stop 2024-04-20T22:04:33.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:33.189+0000 7fdda4400700 1 -- 172.21.15.62:0/4225375815 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd9c101000 msgr2=0x7fdd9c1985f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:04:33.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:33.189+0000 7fdda4400700 1 --2- 172.21.15.62:0/4225375815 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd9c101000 0x7fdd9c1985f0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fdd8800c8a0 tx=0x7fdd8800cc60 comp rx=0 tx=0).stop 2024-04-20T22:04:33.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:33.189+0000 7fdda4400700 1 -- 172.21.15.62:0/4225375815 shutdown_connections 2024-04-20T22:04:33.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:33.189+0000 7fdda4400700 1 --2- 172.21.15.62:0/4225375815 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fdd8406ecc0 0x7fdd84071170 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:04:33.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:33.189+0000 7fdda4400700 1 --2- 172.21.15.62:0/4225375815 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd9c068540 0x7fdd9c1980b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:04:33.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:33.189+0000 7fdda4400700 1 --2- 172.21.15.62:0/4225375815 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd9c101000 0x7fdd9c1985f0 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:04:33.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:33.189+0000 7fdda4400700 1 -- 172.21.15.62:0/4225375815 >> 172.21.15.62:0/4225375815 conn(0x7fdd9c075130 msgr2=0x7fdd9c0fdb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:04:33.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:33.189+0000 7fdda4400700 1 -- 172.21.15.62:0/4225375815 shutdown_connections 2024-04-20T22:04:33.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:33.189+0000 7fdda4400700 1 -- 172.21.15.62:0/4225375815 wait complete. 2024-04-20T22:04:33.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:04:32 smithi189 ceph-mon[57022]: from='client.35576 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:04:33.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:04:32 smithi189 ceph-mon[57022]: pgmap v878: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:04:33.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:04:32 smithi189 ceph-mon[57022]: from='client.35580 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:04:33.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:04:32 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/3755479646' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:04:33.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:04:32 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/3143710006' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:04:33.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:33.345+0000 7f0fdf400700 1 -- 172.21.15.62:0/673593131 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0fd8106b80 msgr2=0x7f0fd8106ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:04:33.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:33.345+0000 7f0fdf400700 1 --2- 172.21.15.62:0/673593131 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0fd8106b80 0x7f0fd8106ff0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f0fc4009910 tx=0x7f0fc4009c20 comp rx=0 tx=0).stop 2024-04-20T22:04:33.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:33.345+0000 7f0fdf400700 1 -- 172.21.15.62:0/673593131 shutdown_connections 2024-04-20T22:04:33.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:33.345+0000 7f0fdf400700 1 --2- 172.21.15.62:0/673593131 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0fd8106b80 0x7f0fd8106ff0 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:04:33.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:33.345+0000 7f0fdf400700 1 --2- 172.21.15.62:0/673593131 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0fd810c8f0 0x7f0fd810ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:04:33.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:33.345+0000 7f0fdf400700 1 -- 172.21.15.62:0/673593131 >> 172.21.15.62:0/673593131 conn(0x7f0fd8076060 msgr2=0x7f0fd8078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:04:33.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:33.345+0000 7f0fdf400700 1 -- 172.21.15.62:0/673593131 shutdown_connections 2024-04-20T22:04:33.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:33.345+0000 7f0fdf400700 1 -- 172.21.15.62:0/673593131 wait complete. 2024-04-20T22:04:33.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:33.346+0000 7f0fdf400700 1 Processor -- start 2024-04-20T22:04:33.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:33.346+0000 7f0fdf400700 1 -- start start 2024-04-20T22:04:33.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:33.347+0000 7f0fdf400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0fd8106b80 0x7f0fd819c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:04:33.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:33.347+0000 7f0fdf400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0fd810c8f0 0x7f0fd819ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:04:33.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:33.347+0000 7f0fdf400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd819d100 con 0x7f0fd8106b80 2024-04-20T22:04:33.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:33.347+0000 7f0fdf400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd81a0e90 con 0x7f0fd810c8f0 2024-04-20T22:04:33.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:33.347+0000 7f0fdca00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0fd8106b80 0x7f0fd819c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:04:33.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:33.347+0000 7f0fdca00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0fd8106b80 0x7f0fd819c4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49182/0 (socket says 172.21.15.62:49182) 2024-04-20T22:04:33.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:33.347+0000 7f0fdca00700 1 -- 172.21.15.62:0/2623286579 learned_addr learned my addr 172.21.15.62:0/2623286579 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:04:33.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:33.348+0000 7f0fdca00700 1 -- 172.21.15.62:0/2623286579 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0fd810c8f0 msgr2=0x7f0fd819ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:04:33.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:33.348+0000 7f0fdca00700 1 --2- 172.21.15.62:0/2623286579 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0fd810c8f0 0x7f0fd819ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:04:33.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:33.348+0000 7f0fdca00700 1 -- 172.21.15.62:0/2623286579 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fc40095f0 con 0x7f0fd8106b80 2024-04-20T22:04:33.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:33.348+0000 7f0fdca00700 1 --2- 172.21.15.62:0/2623286579 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0fd8106b80 0x7f0fd819c4e0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f0fcc00b4b0 tx=0x7f0fcc00b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:04:33.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:33.348+0000 7f0fd5600700 1 -- 172.21.15.62:0/2623286579 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0fcc011800 con 0x7f0fd8106b80 2024-04-20T22:04:33.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:33.348+0000 7f0fd5600700 1 -- 172.21.15.62:0/2623286579 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0fcc011e20 con 0x7f0fd8106b80 2024-04-20T22:04:33.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:33.348+0000 7f0fdf400700 1 -- 172.21.15.62:0/2623286579 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fd81a1170 con 0x7f0fd8106b80 2024-04-20T22:04:33.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:33.348+0000 7f0fd5600700 1 -- 172.21.15.62:0/2623286579 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0fcc00f2e0 con 0x7f0fd8106b80 2024-04-20T22:04:33.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:33.348+0000 7f0fdf400700 1 -- 172.21.15.62:0/2623286579 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fd81a16c0 con 0x7f0fd8106b80 2024-04-20T22:04:33.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:33.349+0000 7f0fdf400700 1 -- 172.21.15.62:0/2623286579 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fd8109b20 con 0x7f0fd8106b80 2024-04-20T22:04:33.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:33.350+0000 7f0fd5600700 1 -- 172.21.15.62:0/2623286579 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f0fcc010380 con 0x7f0fd8106b80 2024-04-20T22:04:33.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:33.351+0000 7f0fd5600700 1 --2- 172.21.15.62:0/2623286579 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f0fc006ecc0 0x7f0fc0071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:04:33.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:33.352+0000 7f0fd5600700 1 -- 172.21.15.62:0/2623286579 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f0fcc08ea30 con 0x7f0fd8106b80 2024-04-20T22:04:33.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:33.352+0000 7f0fd7e00700 1 --2- 172.21.15.62:0/2623286579 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f0fc006ecc0 0x7f0fc0071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:04:33.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:33.352+0000 7f0fd7e00700 1 --2- 172.21.15.62:0/2623286579 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f0fc006ecc0 0x7f0fc0071170 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f0fc400b5c0 tx=0x7f0fc4005fd0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:04:33.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:33.356+0000 7f0fd5600700 1 -- 172.21.15.62:0/2623286579 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f0fcc057870 con 0x7f0fd8106b80 2024-04-20T22:04:33.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:33.675+0000 7f0fdf400700 1 -- 172.21.15.62:0/2623286579 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f0fd804f2a0 con 0x7f0fd8106b80 2024-04-20T22:04:33.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:33.676+0000 7f0fd5600700 1 -- 172.21.15.62:0/2623286579 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7f0fcc05af30 con 0x7f0fd8106b80 2024-04-20T22:04:33.676 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-20T22:04:33.676 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-20T22:04:33.676 INFO:teuthology.orchestra.run.smithi062.stdout: host smithi062 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T22:04:33.676 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-20T22:04:33.676 INFO:teuthology.orchestra.run.smithi062.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi062 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpq7y7oa_r:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:04:33.676 INFO:teuthology.orchestra.run.smithi062.stdout:Traceback (most recent call last): 2024-04-20T22:04:33.676 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-20T22:04:33.676 INFO:teuthology.orchestra.run.smithi062.stdout: return _run_code(code, main_globals, None, 2024-04-20T22:04:33.676 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-20T22:04:33.677 INFO:teuthology.orchestra.run.smithi062.stdout: exec(code, run_globals) 2024-04-20T22:04:33.677 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5618, in 2024-04-20T22:04:33.677 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5606, in main 2024-04-20T22:04:33.677 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 408, in _infer_config 2024-04-20T22:04:33.677 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 323, in _infer_fsid 2024-04-20T22:04:33.677 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 436, in _infer_image 2024-04-20T22:04:33.677 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 310, in _validate_fsid 2024-04-20T22:04:33.677 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 3359, in command_ceph_volume 2024-04-20T22:04:33.677 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-20T22:04:33.677 INFO:teuthology.orchestra.run.smithi062.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi062 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpq7y7oa_r:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:04:33.677 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-20T22:04:33.677 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is degraded 2024-04-20T22:04:33.677 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T22:04:33.677 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T22:04:33.677 INFO:teuthology.orchestra.run.smithi062.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-20T22:04:33.678 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-20T22:04:33.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:33.678+0000 7f0fdf400700 1 -- 172.21.15.62:0/2623286579 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f0fc006ecc0 msgr2=0x7f0fc0071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:04:33.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:33.678+0000 7f0fdf400700 1 --2- 172.21.15.62:0/2623286579 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f0fc006ecc0 0x7f0fc0071170 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f0fc400b5c0 tx=0x7f0fc4005fd0 comp rx=0 tx=0).stop 2024-04-20T22:04:33.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:33.678+0000 7f0fdf400700 1 -- 172.21.15.62:0/2623286579 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0fd8106b80 msgr2=0x7f0fd819c4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:04:33.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:33.678+0000 7f0fdf400700 1 --2- 172.21.15.62:0/2623286579 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0fd8106b80 0x7f0fd819c4e0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f0fcc00b4b0 tx=0x7f0fcc00b870 comp rx=0 tx=0).stop 2024-04-20T22:04:33.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:33.679+0000 7f0fdf400700 1 -- 172.21.15.62:0/2623286579 shutdown_connections 2024-04-20T22:04:33.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:33.679+0000 7f0fdf400700 1 --2- 172.21.15.62:0/2623286579 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f0fc006ecc0 0x7f0fc0071170 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:04:33.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:33.679+0000 7f0fdf400700 1 --2- 172.21.15.62:0/2623286579 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0fd8106b80 0x7f0fd819c4e0 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:04:33.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:33.679+0000 7f0fdf400700 1 --2- 172.21.15.62:0/2623286579 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0fd810c8f0 0x7f0fd819ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:04:33.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:33.679+0000 7f0fdf400700 1 -- 172.21.15.62:0/2623286579 >> 172.21.15.62:0/2623286579 conn(0x7f0fd8076060 msgr2=0x7f0fd8077a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:04:33.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:33.679+0000 7f0fdf400700 1 -- 172.21.15.62:0/2623286579 shutdown_connections 2024-04-20T22:04:33.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:04:33.679+0000 7f0fdf400700 1 -- 172.21.15.62:0/2623286579 wait complete. 2024-04-20T22:04:34.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:04:33 smithi189 ceph-mon[57022]: from='client.35592 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:04:34.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:04:33 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/2623286579' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:04:34.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:04:33 smithi062 ceph-mon[76322]: from='client.35592 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:04:34.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:04:33 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/2623286579' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:04:35.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:04:34 smithi189 ceph-mon[57022]: pgmap v879: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:04:35.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:04:34 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:04:35.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:04:34 smithi062 ceph-mon[76322]: pgmap v879: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:04:35.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:04:34 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:04:37.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:04:36 smithi189 ceph-mon[57022]: pgmap v880: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:04:37.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:04:36 smithi062 ceph-mon[76322]: pgmap v880: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:04:39.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:04:38 smithi189 ceph-mon[57022]: pgmap v881: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:04:39.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:04:38 smithi062 ceph-mon[76322]: pgmap v881: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:04:41.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:04:40 smithi189 ceph-mon[57022]: pgmap v882: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:04:41.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:04:40 smithi062 ceph-mon[76322]: pgmap v882: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:04:43.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:04:42 smithi189 ceph-mon[57022]: pgmap v883: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:04:43.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:04:42 smithi062 ceph-mon[76322]: pgmap v883: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:04:45.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:04:44 smithi189 ceph-mon[57022]: pgmap v884: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:04:45.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:04:44 smithi062 ceph-mon[76322]: pgmap v884: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:04:47.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:04:46 smithi189 ceph-mon[57022]: pgmap v885: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:04:47.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:04:46 smithi062 ceph-mon[76322]: pgmap v885: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:04:49.271 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:04:48 smithi062 ceph-mon[76322]: pgmap v886: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:04:49.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:04:48 smithi189 ceph-mon[57022]: pgmap v886: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:04:50.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:04:50 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:04:50.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:04:50 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:04:51.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:04:51 smithi062 ceph-mon[76322]: pgmap v887: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:04:51.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:04:51 smithi189 ceph-mon[57022]: pgmap v887: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:04:53.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:04:53 smithi062 ceph-mon[76322]: pgmap v888: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:04:53.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:04:53 smithi189 ceph-mon[57022]: pgmap v888: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:04:55.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:04:55 smithi062 ceph-mon[76322]: pgmap v889: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:04:55.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:04:55 smithi189 ceph-mon[57022]: pgmap v889: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:04:57.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:04:57 smithi062 ceph-mon[76322]: pgmap v890: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:04:57.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:04:57 smithi189 ceph-mon[57022]: pgmap v890: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:04:59.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:04:59 smithi062 ceph-mon[76322]: pgmap v891: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:04:59.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:04:59 smithi189 ceph-mon[57022]: pgmap v891: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:05:01.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:05:01 smithi189 ceph-mon[57022]: pgmap v892: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:05:01.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:05:01 smithi062 ceph-mon[76322]: pgmap v892: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:05:03.468 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:05:03 smithi189 ceph-mon[57022]: pgmap v893: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:05:03.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:05:03 smithi062 ceph-mon[76322]: pgmap v893: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:05:03.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:03.856+0000 7f58a0000700 1 -- 172.21.15.62:0/2008856593 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5898073090 msgr2=0x7f5898073460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:05:03.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:03.856+0000 7f58a0000700 1 --2- 172.21.15.62:0/2008856593 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5898073090 0x7f5898073460 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f588c009940 tx=0x7f588c009c50 comp rx=0 tx=0).stop 2024-04-20T22:05:03.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:03.856+0000 7f58a0000700 1 -- 172.21.15.62:0/2008856593 shutdown_connections 2024-04-20T22:05:03.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:03.856+0000 7f58a0000700 1 --2- 172.21.15.62:0/2008856593 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f58980739a0 0x7f589810c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:05:03.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:03.856+0000 7f58a0000700 1 --2- 172.21.15.62:0/2008856593 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5898073090 0x7f5898073460 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:05:03.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:03.856+0000 7f58a0000700 1 -- 172.21.15.62:0/2008856593 >> 172.21.15.62:0/2008856593 conn(0x7f58980fbda0 msgr2=0x7f58980fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:05:03.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:03.856+0000 7f58a0000700 1 -- 172.21.15.62:0/2008856593 shutdown_connections 2024-04-20T22:05:03.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:03.856+0000 7f58a0000700 1 -- 172.21.15.62:0/2008856593 wait complete. 2024-04-20T22:05:03.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:03.857+0000 7f58a0000700 1 Processor -- start 2024-04-20T22:05:03.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:03.858+0000 7f58a0000700 1 -- start start 2024-04-20T22:05:03.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:03.858+0000 7f58a0000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5898073090 0x7f5898198180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:05:03.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:03.858+0000 7f58a0000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f58980739a0 0x7f58981986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:05:03.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:03.858+0000 7f58a0000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5898198da0 con 0x7f5898073090 2024-04-20T22:05:03.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:03.858+0000 7f58a0000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f589819cb30 con 0x7f58980739a0 2024-04-20T22:05:03.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:03.858+0000 7f589d600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5898073090 0x7f5898198180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:05:03.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:03.858+0000 7f589d600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5898073090 0x7f5898198180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:36516/0 (socket says 172.21.15.62:36516) 2024-04-20T22:05:03.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:03.858+0000 7f589d600700 1 -- 172.21.15.62:0/3621024839 learned_addr learned my addr 172.21.15.62:0/3621024839 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:05:03.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:03.859+0000 7f589d600700 1 -- 172.21.15.62:0/3621024839 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f58980739a0 msgr2=0x7f58981986c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:05:03.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:03.859+0000 7f589cc00700 1 --2- 172.21.15.62:0/3621024839 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f58980739a0 0x7f58981986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:05:03.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:03.859+0000 7f589d600700 1 --2- 172.21.15.62:0/3621024839 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f58980739a0 0x7f58981986c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:05:03.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:03.859+0000 7f589d600700 1 -- 172.21.15.62:0/3621024839 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f588c0095f0 con 0x7f5898073090 2024-04-20T22:05:03.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:03.859+0000 7f589cc00700 1 --2- 172.21.15.62:0/3621024839 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f58980739a0 0x7f58981986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T22:05:03.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:03.859+0000 7f589d600700 1 --2- 172.21.15.62:0/3621024839 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5898073090 0x7f5898198180 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f588c00b870 tx=0x7f588c0056a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:05:03.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:03.860+0000 7f5896000700 1 -- 172.21.15.62:0/3621024839 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f588c01d070 con 0x7f5898073090 2024-04-20T22:05:03.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:03.860+0000 7f58a0000700 1 -- 172.21.15.62:0/3621024839 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f589819cdb0 con 0x7f5898073090 2024-04-20T22:05:03.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:03.860+0000 7f5896000700 1 -- 172.21.15.62:0/3621024839 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f588c00f460 con 0x7f5898073090 2024-04-20T22:05:03.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:03.860+0000 7f5896000700 1 -- 172.21.15.62:0/3621024839 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f588c021640 con 0x7f5898073090 2024-04-20T22:05:03.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:03.860+0000 7f58a0000700 1 -- 172.21.15.62:0/3621024839 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f589819d2a0 con 0x7f5898073090 2024-04-20T22:05:03.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:03.861+0000 7f58a0000700 1 -- 172.21.15.62:0/3621024839 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5898109ce0 con 0x7f5898073090 2024-04-20T22:05:03.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:03.862+0000 7f5896000700 1 -- 172.21.15.62:0/3621024839 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f588c00fa60 con 0x7f5898073090 2024-04-20T22:05:03.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:03.863+0000 7f5896000700 1 --2- 172.21.15.62:0/3621024839 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f588006ecc0 0x7f5880071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:05:03.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:03.863+0000 7f5896000700 1 -- 172.21.15.62:0/3621024839 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f588c0901d0 con 0x7f5898073090 2024-04-20T22:05:03.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:03.863+0000 7f589cc00700 1 --2- 172.21.15.62:0/3621024839 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f588006ecc0 0x7f5880071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:05:03.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:03.864+0000 7f589cc00700 1 --2- 172.21.15.62:0/3621024839 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f588006ecc0 0x7f5880071170 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f5884005fd0 tx=0x7f5884005dc0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:05:03.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:03.868+0000 7f5896000700 1 -- 172.21.15.62:0/3621024839 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f588c058ee0 con 0x7f5898073090 2024-04-20T22:05:04.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.122+0000 7f58a0000700 1 -- 172.21.15.62:0/3621024839 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f589819d510 con 0x7f588006ecc0 2024-04-20T22:05:04.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.124+0000 7f5896000700 1 -- 172.21.15.62:0/3621024839 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f589819d510 con 0x7f588006ecc0 2024-04-20T22:05:04.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.127+0000 7f58a0000700 1 -- 172.21.15.62:0/3621024839 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f588006ecc0 msgr2=0x7f5880071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:05:04.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.127+0000 7f58a0000700 1 --2- 172.21.15.62:0/3621024839 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f588006ecc0 0x7f5880071170 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f5884005fd0 tx=0x7f5884005dc0 comp rx=0 tx=0).stop 2024-04-20T22:05:04.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.127+0000 7f58a0000700 1 -- 172.21.15.62:0/3621024839 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5898073090 msgr2=0x7f5898198180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:05:04.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.127+0000 7f58a0000700 1 --2- 172.21.15.62:0/3621024839 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5898073090 0x7f5898198180 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f588c00b870 tx=0x7f588c0056a0 comp rx=0 tx=0).stop 2024-04-20T22:05:04.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.127+0000 7f58a0000700 1 -- 172.21.15.62:0/3621024839 shutdown_connections 2024-04-20T22:05:04.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.127+0000 7f58a0000700 1 --2- 172.21.15.62:0/3621024839 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f588006ecc0 0x7f5880071170 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:05:04.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.127+0000 7f58a0000700 1 --2- 172.21.15.62:0/3621024839 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5898073090 0x7f5898198180 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:05:04.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.127+0000 7f58a0000700 1 --2- 172.21.15.62:0/3621024839 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f58980739a0 0x7f58981986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:05:04.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.127+0000 7f58a0000700 1 -- 172.21.15.62:0/3621024839 >> 172.21.15.62:0/3621024839 conn(0x7f58980fbda0 msgr2=0x7f5898106e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:05:04.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.128+0000 7f58a0000700 1 -- 172.21.15.62:0/3621024839 shutdown_connections 2024-04-20T22:05:04.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.128+0000 7f58a0000700 1 -- 172.21.15.62:0/3621024839 wait complete. 2024-04-20T22:05:04.146 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T22:05:04.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.282+0000 7fc8aee00700 1 -- 172.21.15.62:0/637248574 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8a810c8f0 msgr2=0x7fc8a810ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:05:04.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.282+0000 7fc8aee00700 1 --2- 172.21.15.62:0/637248574 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8a810c8f0 0x7fc8a810ccc0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fc898009910 tx=0x7fc898009c20 comp rx=0 tx=0).stop 2024-04-20T22:05:04.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.282+0000 7fc8aee00700 1 -- 172.21.15.62:0/637248574 shutdown_connections 2024-04-20T22:05:04.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.282+0000 7fc8aee00700 1 --2- 172.21.15.62:0/637248574 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc8a8106b80 0x7fc8a8106ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:05:04.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.282+0000 7fc8aee00700 1 --2- 172.21.15.62:0/637248574 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8a810c8f0 0x7fc8a810ccc0 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:05:04.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.282+0000 7fc8aee00700 1 -- 172.21.15.62:0/637248574 >> 172.21.15.62:0/637248574 conn(0x7fc8a8076060 msgr2=0x7fc8a8078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:05:04.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.282+0000 7fc8aee00700 1 -- 172.21.15.62:0/637248574 shutdown_connections 2024-04-20T22:05:04.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.282+0000 7fc8aee00700 1 -- 172.21.15.62:0/637248574 wait complete. 2024-04-20T22:05:04.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.283+0000 7fc8aee00700 1 Processor -- start 2024-04-20T22:05:04.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.284+0000 7fc8aee00700 1 -- start start 2024-04-20T22:05:04.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.284+0000 7fc8aee00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc8a8106b80 0x7fc8a819c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:05:04.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.284+0000 7fc8aee00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8a810c8f0 0x7fc8a819ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:05:04.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.284+0000 7fc8aee00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8a819d100 con 0x7fc8a810c8f0 2024-04-20T22:05:04.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.284+0000 7fc8aee00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8a81a0e90 con 0x7fc8a8106b80 2024-04-20T22:05:04.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.284+0000 7fc8a7400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8a810c8f0 0x7fc8a819ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:05:04.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.284+0000 7fc8a7400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8a810c8f0 0x7fc8a819ca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:36530/0 (socket says 172.21.15.62:36530) 2024-04-20T22:05:04.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.284+0000 7fc8a7400700 1 -- 172.21.15.62:0/835049460 learned_addr learned my addr 172.21.15.62:0/835049460 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:05:04.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.285+0000 7fc8a7e00700 1 --2- 172.21.15.62:0/835049460 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc8a8106b80 0x7fc8a819c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:05:04.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.285+0000 7fc8a7400700 1 -- 172.21.15.62:0/835049460 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc8a8106b80 msgr2=0x7fc8a819c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:05:04.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.285+0000 7fc8a7400700 1 --2- 172.21.15.62:0/835049460 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc8a8106b80 0x7fc8a819c4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:05:04.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.285+0000 7fc8a7400700 1 -- 172.21.15.62:0/835049460 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8980095f0 con 0x7fc8a810c8f0 2024-04-20T22:05:04.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.285+0000 7fc8a7400700 1 --2- 172.21.15.62:0/835049460 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8a810c8f0 0x7fc8a819ca20 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fc89400b4b0 tx=0x7fc89400b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:05:04.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.285+0000 7fc8a4c00700 1 -- 172.21.15.62:0/835049460 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc894011800 con 0x7fc8a810c8f0 2024-04-20T22:05:04.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.286+0000 7fc8a4c00700 1 -- 172.21.15.62:0/835049460 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc894011e20 con 0x7fc8a810c8f0 2024-04-20T22:05:04.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.286+0000 7fc8aee00700 1 -- 172.21.15.62:0/835049460 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8a81a1170 con 0x7fc8a810c8f0 2024-04-20T22:05:04.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.286+0000 7fc8a4c00700 1 -- 172.21.15.62:0/835049460 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc89400f2e0 con 0x7fc8a810c8f0 2024-04-20T22:05:04.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.286+0000 7fc8aee00700 1 -- 172.21.15.62:0/835049460 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8a81a16c0 con 0x7fc8a810c8f0 2024-04-20T22:05:04.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.287+0000 7fc8aee00700 1 -- 172.21.15.62:0/835049460 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8a8109b20 con 0x7fc8a810c8f0 2024-04-20T22:05:04.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.288+0000 7fc8a4c00700 1 -- 172.21.15.62:0/835049460 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fc894010380 con 0x7fc8a810c8f0 2024-04-20T22:05:04.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.289+0000 7fc8a4c00700 1 --2- 172.21.15.62:0/835049460 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc88c06ecc0 0x7fc88c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:05:04.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.289+0000 7fc8a4c00700 1 -- 172.21.15.62:0/835049460 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fc89408ea30 con 0x7fc8a810c8f0 2024-04-20T22:05:04.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.289+0000 7fc8a7e00700 1 --2- 172.21.15.62:0/835049460 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc88c06ecc0 0x7fc88c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:05:04.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.290+0000 7fc8a7e00700 1 --2- 172.21.15.62:0/835049460 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc88c06ecc0 0x7fc88c071170 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fc898006010 tx=0x7fc89801a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:05:04.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.294+0000 7fc8a4c00700 1 -- 172.21.15.62:0/835049460 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fc894057870 con 0x7fc8a810c8f0 2024-04-20T22:05:04.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.551+0000 7fc8aee00700 1 -- 172.21.15.62:0/835049460 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc8a81a1990 con 0x7fc88c06ecc0 2024-04-20T22:05:04.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.555+0000 7fc8a4c00700 1 -- 172.21.15.62:0/835049460 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fc8a81a1990 con 0x7fc88c06ecc0 2024-04-20T22:05:04.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.558+0000 7fc8aee00700 1 -- 172.21.15.62:0/835049460 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc88c06ecc0 msgr2=0x7fc88c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:05:04.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.558+0000 7fc8aee00700 1 --2- 172.21.15.62:0/835049460 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc88c06ecc0 0x7fc88c071170 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fc898006010 tx=0x7fc89801a040 comp rx=0 tx=0).stop 2024-04-20T22:05:04.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.558+0000 7fc8aee00700 1 -- 172.21.15.62:0/835049460 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8a810c8f0 msgr2=0x7fc8a819ca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:05:04.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.558+0000 7fc8aee00700 1 --2- 172.21.15.62:0/835049460 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8a810c8f0 0x7fc8a819ca20 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fc89400b4b0 tx=0x7fc89400b870 comp rx=0 tx=0).stop 2024-04-20T22:05:04.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.558+0000 7fc8aee00700 1 -- 172.21.15.62:0/835049460 shutdown_connections 2024-04-20T22:05:04.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.558+0000 7fc8aee00700 1 --2- 172.21.15.62:0/835049460 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc88c06ecc0 0x7fc88c071170 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:05:04.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.558+0000 7fc8aee00700 1 --2- 172.21.15.62:0/835049460 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc8a8106b80 0x7fc8a819c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:05:04.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.558+0000 7fc8aee00700 1 --2- 172.21.15.62:0/835049460 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8a810c8f0 0x7fc8a819ca20 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:05:04.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.558+0000 7fc8aee00700 1 -- 172.21.15.62:0/835049460 >> 172.21.15.62:0/835049460 conn(0x7fc8a8076060 msgr2=0x7fc8a8077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:05:04.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.558+0000 7fc8aee00700 1 -- 172.21.15.62:0/835049460 shutdown_connections 2024-04-20T22:05:04.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.558+0000 7fc8aee00700 1 -- 172.21.15.62:0/835049460 wait complete. 2024-04-20T22:05:04.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.715+0000 7f79f8c00700 1 -- 172.21.15.62:0/1446583488 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f79f410c8f0 msgr2=0x7f79f410ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:05:04.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.715+0000 7f79f8c00700 1 --2- 172.21.15.62:0/1446583488 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f79f410c8f0 0x7f79f410ccc0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f79e4009910 tx=0x7f79e4009c20 comp rx=0 tx=0).stop 2024-04-20T22:05:04.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.715+0000 7f79f8c00700 1 -- 172.21.15.62:0/1446583488 shutdown_connections 2024-04-20T22:05:04.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.715+0000 7f79f8c00700 1 --2- 172.21.15.62:0/1446583488 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f79f4106b80 0x7f79f4106ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:05:04.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.715+0000 7f79f8c00700 1 --2- 172.21.15.62:0/1446583488 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f79f410c8f0 0x7f79f410ccc0 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:05:04.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.715+0000 7f79f8c00700 1 -- 172.21.15.62:0/1446583488 >> 172.21.15.62:0/1446583488 conn(0x7f79f4076060 msgr2=0x7f79f4078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:05:04.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.715+0000 7f79f8c00700 1 -- 172.21.15.62:0/1446583488 shutdown_connections 2024-04-20T22:05:04.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.715+0000 7f79f8c00700 1 -- 172.21.15.62:0/1446583488 wait complete. 2024-04-20T22:05:04.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.716+0000 7f79f8c00700 1 Processor -- start 2024-04-20T22:05:04.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.716+0000 7f79f8c00700 1 -- start start 2024-04-20T22:05:04.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.717+0000 7f79f8c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f79f4106b80 0x7f79f419c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:05:04.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.717+0000 7f79f8c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f79f410c8f0 0x7f79f419ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:05:04.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.717+0000 7f79f8c00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79f419d100 con 0x7f79f410c8f0 2024-04-20T22:05:04.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.717+0000 7f79f8c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79f41a0e90 con 0x7f79f4106b80 2024-04-20T22:05:04.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.717+0000 7f79f1400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f79f410c8f0 0x7f79f419ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:05:04.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.717+0000 7f79f1400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f79f410c8f0 0x7f79f419ca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:36554/0 (socket says 172.21.15.62:36554) 2024-04-20T22:05:04.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.717+0000 7f79f1400700 1 -- 172.21.15.62:0/473727249 learned_addr learned my addr 172.21.15.62:0/473727249 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:05:04.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.717+0000 7f79f1e00700 1 --2- 172.21.15.62:0/473727249 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f79f4106b80 0x7f79f419c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:05:04.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.717+0000 7f79f1400700 1 -- 172.21.15.62:0/473727249 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f79f4106b80 msgr2=0x7f79f419c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:05:04.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.717+0000 7f79f1400700 1 --2- 172.21.15.62:0/473727249 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f79f4106b80 0x7f79f419c4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:05:04.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.717+0000 7f79f1400700 1 -- 172.21.15.62:0/473727249 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79e40095f0 con 0x7f79f410c8f0 2024-04-20T22:05:04.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.718+0000 7f79f1400700 1 --2- 172.21.15.62:0/473727249 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f79f410c8f0 0x7f79f419ca20 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f79e000b4b0 tx=0x7f79e000b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:05:04.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.718+0000 7f79eaa00700 1 -- 172.21.15.62:0/473727249 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f79e0011800 con 0x7f79f410c8f0 2024-04-20T22:05:04.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.718+0000 7f79eaa00700 1 -- 172.21.15.62:0/473727249 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f79e0011e20 con 0x7f79f410c8f0 2024-04-20T22:05:04.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.718+0000 7f79f8c00700 1 -- 172.21.15.62:0/473727249 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79f41a1170 con 0x7f79f410c8f0 2024-04-20T22:05:04.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.718+0000 7f79eaa00700 1 -- 172.21.15.62:0/473727249 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f79e000f2e0 con 0x7f79f410c8f0 2024-04-20T22:05:04.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.718+0000 7f79f8c00700 1 -- 172.21.15.62:0/473727249 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79f41a16c0 con 0x7f79f410c8f0 2024-04-20T22:05:04.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.719+0000 7f79f8c00700 1 -- 172.21.15.62:0/473727249 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79f4109b20 con 0x7f79f410c8f0 2024-04-20T22:05:04.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.720+0000 7f79eaa00700 1 -- 172.21.15.62:0/473727249 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f79e0010380 con 0x7f79f410c8f0 2024-04-20T22:05:04.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.721+0000 7f79eaa00700 1 --2- 172.21.15.62:0/473727249 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f79d806ecc0 0x7f79d8071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:05:04.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.721+0000 7f79eaa00700 1 -- 172.21.15.62:0/473727249 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f79e008ea30 con 0x7f79f410c8f0 2024-04-20T22:05:04.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.722+0000 7f79f1e00700 1 --2- 172.21.15.62:0/473727249 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f79d806ecc0 0x7f79d8071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:05:04.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.722+0000 7f79f1e00700 1 --2- 172.21.15.62:0/473727249 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f79d806ecc0 0x7f79d8071170 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f79e4000c00 tx=0x7f79e4005fd0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:05:04.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.727+0000 7f79eaa00700 1 -- 172.21.15.62:0/473727249 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f79e0057870 con 0x7f79f410c8f0 2024-04-20T22:05:04.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.975+0000 7f79f8c00700 1 -- 172.21.15.62:0/473727249 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f79f41a1990 con 0x7f79d806ecc0 2024-04-20T22:05:04.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.988+0000 7f79eaa00700 1 -- 172.21.15.62:0/473727249 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f79f41a1990 con 0x7f79d806ecc0 2024-04-20T22:05:04.987 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T22:05:04.987 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (30m) 7m ago 44m 15.3M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T22:05:04.988 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (45m) 7m ago 45m 10.4M - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T22:05:04.988 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (43m) 7m ago 43m 10.1M - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T22:05:04.988 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (27m) 7m ago 45m 6660k - 19.0.0-3133-ga8c84f59 76bc6b91b726 2825d5ba11c6 2024-04-20T22:05:04.988 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (26m) 7m ago 43m 6665k - 19.0.0-3133-ga8c84f59 76bc6b91b726 20f746fa1bba 2024-04-20T22:05:04.988 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (30m) 7m ago 44m 73.5M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T22:05:04.988 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (22m) 7m ago 37m 18.5M - 19.0.0-3133-ga8c84f59 76bc6b91b726 de80bc90a683 2024-04-20T22:05:04.988 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (22m) 7m ago 37m 18.7M - 19.0.0-3133-ga8c84f59 76bc6b91b726 cd793b7ee98e 2024-04-20T22:05:04.988 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (7m) 7m ago 37m 11.0M - 19.0.0-3133-ga8c84f59 76bc6b91b726 0e7b0bc4d1ab 2024-04-20T22:05:04.988 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (37m) 7m ago 37m 18.6M - 18.2.0 dc2bc1663786 f2eec2ee8951 2024-04-20T22:05:04.988 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (33m) 7m ago 46m 513M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T22:05:04.988 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (32m) 7m ago 43m 459M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T22:05:04.988 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (28m) 7m ago 46m 71.7M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 019eb1479485 2024-04-20T22:05:04.988 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (28m) 7m ago 43m 51.0M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 55aac560aab6 2024-04-20T22:05:04.988 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (32m) 7m ago 44m 8472k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T22:05:04.989 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (31m) 7m ago 43m 8048k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T22:05:04.989 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (26m) 7m ago 42m 91.2M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 ee2bb901f26e 2024-04-20T22:05:04.989 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (25m) 7m ago 41m 94.6M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 9075152d4162 2024-04-20T22:05:04.989 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (25m) 7m ago 41m 78.1M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 2d4f3e38dc0e 2024-04-20T22:05:04.989 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (24m) 7m ago 40m 92.6M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 8f97b17ae0eb 2024-04-20T22:05:04.989 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (24m) 7m ago 40m 95.6M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 653d8079967d 2024-04-20T22:05:04.989 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (23m) 7m ago 39m 89.2M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 596dded14c12 2024-04-20T22:05:04.989 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (31m) 7m ago 44m 82.1M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T22:05:04.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.991+0000 7f79f8c00700 1 -- 172.21.15.62:0/473727249 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f79d806ecc0 msgr2=0x7f79d8071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:05:04.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.991+0000 7f79f8c00700 1 --2- 172.21.15.62:0/473727249 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f79d806ecc0 0x7f79d8071170 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f79e4000c00 tx=0x7f79e4005fd0 comp rx=0 tx=0).stop 2024-04-20T22:05:04.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.991+0000 7f79f8c00700 1 -- 172.21.15.62:0/473727249 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f79f410c8f0 msgr2=0x7f79f419ca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:05:04.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.991+0000 7f79f8c00700 1 --2- 172.21.15.62:0/473727249 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f79f410c8f0 0x7f79f419ca20 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f79e000b4b0 tx=0x7f79e000b870 comp rx=0 tx=0).stop 2024-04-20T22:05:04.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.991+0000 7f79f8c00700 1 -- 172.21.15.62:0/473727249 shutdown_connections 2024-04-20T22:05:04.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.991+0000 7f79f8c00700 1 --2- 172.21.15.62:0/473727249 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f79d806ecc0 0x7f79d8071170 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:05:04.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.991+0000 7f79f8c00700 1 --2- 172.21.15.62:0/473727249 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f79f4106b80 0x7f79f419c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:05:04.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.991+0000 7f79f8c00700 1 --2- 172.21.15.62:0/473727249 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f79f410c8f0 0x7f79f419ca20 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:05:04.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.991+0000 7f79f8c00700 1 -- 172.21.15.62:0/473727249 >> 172.21.15.62:0/473727249 conn(0x7f79f4076060 msgr2=0x7f79f4077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:05:04.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.991+0000 7f79f8c00700 1 -- 172.21.15.62:0/473727249 shutdown_connections 2024-04-20T22:05:04.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:04.991+0000 7f79f8c00700 1 -- 172.21.15.62:0/473727249 wait complete. 2024-04-20T22:05:05.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:05.140+0000 7f9782400700 1 -- 172.21.15.62:0/657053502 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f977c0739e0 msgr2=0x7f977c10c580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:05:05.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:05.140+0000 7f9782400700 1 --2- 172.21.15.62:0/657053502 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f977c0739e0 0x7f977c10c580 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f9768009910 tx=0x7f9768009c20 comp rx=0 tx=0).stop 2024-04-20T22:05:05.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:05.140+0000 7f9782400700 1 -- 172.21.15.62:0/657053502 shutdown_connections 2024-04-20T22:05:05.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:05.140+0000 7f9782400700 1 --2- 172.21.15.62:0/657053502 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f977c0739e0 0x7f977c10c580 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:05:05.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:05.140+0000 7f9782400700 1 --2- 172.21.15.62:0/657053502 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f977c0730d0 0x7f977c0734a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:05:05.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:05.140+0000 7f9782400700 1 -- 172.21.15.62:0/657053502 >> 172.21.15.62:0/657053502 conn(0x7f977c0fbd80 msgr2=0x7f977c0fe190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:05:05.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:05.140+0000 7f9782400700 1 -- 172.21.15.62:0/657053502 shutdown_connections 2024-04-20T22:05:05.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:05.140+0000 7f9782400700 1 -- 172.21.15.62:0/657053502 wait complete. 2024-04-20T22:05:05.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:05.141+0000 7f9782400700 1 Processor -- start 2024-04-20T22:05:05.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:05.142+0000 7f9782400700 1 -- start start 2024-04-20T22:05:05.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:05.142+0000 7f9782400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f977c0730d0 0x7f977c1981c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:05:05.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:05.142+0000 7f9782400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f977c0739e0 0x7f977c198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:05:05.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:05.142+0000 7f9782400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f977c198de0 con 0x7f977c0730d0 2024-04-20T22:05:05.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:05.142+0000 7f9782400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f977c19cb70 con 0x7f977c0739e0 2024-04-20T22:05:05.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:05.142+0000 7f977b400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f977c0730d0 0x7f977c1981c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:05:05.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:05.142+0000 7f977b400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f977c0730d0 0x7f977c1981c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:36568/0 (socket says 172.21.15.62:36568) 2024-04-20T22:05:05.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:05.142+0000 7f977b400700 1 -- 172.21.15.62:0/779881331 learned_addr learned my addr 172.21.15.62:0/779881331 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:05:05.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:05.143+0000 7f977b400700 1 -- 172.21.15.62:0/779881331 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f977c0739e0 msgr2=0x7f977c198700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:05:05.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:05.143+0000 7f977b400700 1 --2- 172.21.15.62:0/779881331 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f977c0739e0 0x7f977c198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:05:05.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:05.143+0000 7f977b400700 1 -- 172.21.15.62:0/779881331 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97680095f0 con 0x7f977c0730d0 2024-04-20T22:05:05.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:05.143+0000 7f977b400700 1 --2- 172.21.15.62:0/779881331 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f977c0730d0 0x7f977c1981c0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f976c00d900 tx=0x7f976c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:05:05.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:05.143+0000 7f9773e00700 1 -- 172.21.15.62:0/779881331 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f976c004a00 con 0x7f977c0730d0 2024-04-20T22:05:05.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:05.143+0000 7f9773e00700 1 -- 172.21.15.62:0/779881331 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f976c0054e0 con 0x7f977c0730d0 2024-04-20T22:05:05.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:05.143+0000 7f9782400700 1 -- 172.21.15.62:0/779881331 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f977c19ce50 con 0x7f977c0730d0 2024-04-20T22:05:05.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:05.143+0000 7f9773e00700 1 -- 172.21.15.62:0/779881331 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f976c009d60 con 0x7f977c0730d0 2024-04-20T22:05:05.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:05.144+0000 7f9782400700 1 -- 172.21.15.62:0/779881331 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f977c19d3a0 con 0x7f977c0730d0 2024-04-20T22:05:05.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:05.145+0000 7f9782400700 1 -- 172.21.15.62:0/779881331 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f977c109c80 con 0x7f977c0730d0 2024-04-20T22:05:05.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:05.146+0000 7f9773e00700 1 -- 172.21.15.62:0/779881331 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f976c005020 con 0x7f977c0730d0 2024-04-20T22:05:05.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:05.147+0000 7f9773e00700 1 --2- 172.21.15.62:0/779881331 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f976006ecc0 0x7f9760071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:05:05.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:05.147+0000 7f9773e00700 1 -- 172.21.15.62:0/779881331 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f976c08f0d0 con 0x7f977c0730d0 2024-04-20T22:05:05.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:05.147+0000 7f977aa00700 1 --2- 172.21.15.62:0/779881331 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f976006ecc0 0x7f9760071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:05:05.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:05.148+0000 7f977aa00700 1 --2- 172.21.15.62:0/779881331 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f976006ecc0 0x7f9760071170 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f9768009fd0 tx=0x7f976800b560 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:05:05.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:05.152+0000 7f9773e00700 1 -- 172.21.15.62:0/779881331 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f976c057de0 con 0x7f977c0730d0 2024-04-20T22:05:05.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:05:05 smithi062 ceph-mon[76322]: pgmap v894: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:05:05.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:05:05 smithi062 ceph-mon[76322]: from='client.35600 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:05:05.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:05:05 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:05:05.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:05.475+0000 7f9782400700 1 -- 172.21.15.62:0/779881331 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f977c1995f0 con 0x7f977c0730d0 2024-04-20T22:05:05.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:05.476+0000 7f9773e00700 1 -- 172.21.15.62:0/779881331 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7f976c05b4a0 con 0x7f977c0730d0 2024-04-20T22:05:05.476 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:05:05.476 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T22:05:05.476 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:05:05.476 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:05:05.476 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T22:05:05.476 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:05:05.476 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:05:05.477 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T22:05:05.477 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 6 2024-04-20T22:05:05.477 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:05:05.477 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T22:05:05.477 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 1, 2024-04-20T22:05:05.477 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 3 2024-04-20T22:05:05.477 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:05:05.477 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T22:05:05.477 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 1, 2024-04-20T22:05:05.477 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 13 2024-04-20T22:05:05.477 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T22:05:05.477 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:05:05.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:05.479+0000 7f9782400700 1 -- 172.21.15.62:0/779881331 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f976006ecc0 msgr2=0x7f9760071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:05:05.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:05.479+0000 7f9782400700 1 --2- 172.21.15.62:0/779881331 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f976006ecc0 0x7f9760071170 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f9768009fd0 tx=0x7f976800b560 comp rx=0 tx=0).stop 2024-04-20T22:05:05.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:05.479+0000 7f9782400700 1 -- 172.21.15.62:0/779881331 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f977c0730d0 msgr2=0x7f977c1981c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:05:05.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:05.479+0000 7f9782400700 1 --2- 172.21.15.62:0/779881331 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f977c0730d0 0x7f977c1981c0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f976c00d900 tx=0x7f976c00dcc0 comp rx=0 tx=0).stop 2024-04-20T22:05:05.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:05.479+0000 7f9782400700 1 -- 172.21.15.62:0/779881331 shutdown_connections 2024-04-20T22:05:05.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:05.479+0000 7f9782400700 1 --2- 172.21.15.62:0/779881331 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f976006ecc0 0x7f9760071170 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:05:05.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:05.479+0000 7f9782400700 1 --2- 172.21.15.62:0/779881331 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f977c0730d0 0x7f977c1981c0 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:05:05.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:05.479+0000 7f9782400700 1 --2- 172.21.15.62:0/779881331 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f977c0739e0 0x7f977c198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:05:05.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:05.479+0000 7f9782400700 1 -- 172.21.15.62:0/779881331 >> 172.21.15.62:0/779881331 conn(0x7f977c0fbd80 msgr2=0x7f977c106dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:05:05.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:05.479+0000 7f9782400700 1 -- 172.21.15.62:0/779881331 shutdown_connections 2024-04-20T22:05:05.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:05.479+0000 7f9782400700 1 -- 172.21.15.62:0/779881331 wait complete. 2024-04-20T22:05:05.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:05:05 smithi189 ceph-mon[57022]: pgmap v894: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:05:05.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:05:05 smithi189 ceph-mon[57022]: from='client.35600 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:05:05.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:05:05 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:05:05.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:05.641+0000 7f2b34200700 1 -- 172.21.15.62:0/2152081079 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2b2c100e80 msgr2=0x7f2b2c104e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:05:05.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:05.641+0000 7f2b34200700 1 --2- 172.21.15.62:0/2152081079 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2b2c100e80 0x7f2b2c104e50 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f2b18009910 tx=0x7f2b18009c20 comp rx=0 tx=0).stop 2024-04-20T22:05:05.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:05.642+0000 7f2b34200700 1 -- 172.21.15.62:0/2152081079 shutdown_connections 2024-04-20T22:05:05.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:05.642+0000 7f2b34200700 1 --2- 172.21.15.62:0/2152081079 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2b2c100e80 0x7f2b2c104e50 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:05:05.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:05.642+0000 7f2b34200700 1 --2- 172.21.15.62:0/2152081079 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2b2c1004e0 0x7f2b2c1008b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:05:05.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:05.642+0000 7f2b34200700 1 -- 172.21.15.62:0/2152081079 >> 172.21.15.62:0/2152081079 conn(0x7f2b2c0fbda0 msgr2=0x7f2b2c0fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:05:05.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:05.642+0000 7f2b34200700 1 -- 172.21.15.62:0/2152081079 shutdown_connections 2024-04-20T22:05:05.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:05.642+0000 7f2b34200700 1 -- 172.21.15.62:0/2152081079 wait complete. 2024-04-20T22:05:05.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:05.644+0000 7f2b34200700 1 Processor -- start 2024-04-20T22:05:05.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:05.644+0000 7f2b34200700 1 -- start start 2024-04-20T22:05:05.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:05.644+0000 7f2b34200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2b2c1004e0 0x7f2b2c198130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:05:05.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:05.644+0000 7f2b34200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2b2c100e80 0x7f2b2c198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:05:05.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:05.645+0000 7f2b34200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b2c198d50 con 0x7f2b2c100e80 2024-04-20T22:05:05.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:05.645+0000 7f2b34200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b2c19cae0 con 0x7f2b2c1004e0 2024-04-20T22:05:05.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:05.645+0000 7f2b30e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2b2c100e80 0x7f2b2c198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:05:05.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:05.645+0000 7f2b30e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2b2c100e80 0x7f2b2c198670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:36584/0 (socket says 172.21.15.62:36584) 2024-04-20T22:05:05.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:05.645+0000 7f2b30e00700 1 -- 172.21.15.62:0/4185932859 learned_addr learned my addr 172.21.15.62:0/4185932859 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:05:05.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:05.645+0000 7f2b31800700 1 --2- 172.21.15.62:0/4185932859 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2b2c1004e0 0x7f2b2c198130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:05:05.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:05.646+0000 7f2b30e00700 1 -- 172.21.15.62:0/4185932859 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2b2c1004e0 msgr2=0x7f2b2c198130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:05:05.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:05.646+0000 7f2b30e00700 1 --2- 172.21.15.62:0/4185932859 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2b2c1004e0 0x7f2b2c198130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:05:05.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:05.646+0000 7f2b30e00700 1 -- 172.21.15.62:0/4185932859 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b180095f0 con 0x7f2b2c100e80 2024-04-20T22:05:05.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:05.646+0000 7f2b30e00700 1 --2- 172.21.15.62:0/4185932859 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2b2c100e80 0x7f2b2c198670 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f2b1800bda0 tx=0x7f2b1800bdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:05:05.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:05.646+0000 7f2b2a000700 1 -- 172.21.15.62:0/4185932859 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2b1801d070 con 0x7f2b2c100e80 2024-04-20T22:05:05.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:05.646+0000 7f2b2a000700 1 -- 172.21.15.62:0/4185932859 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2b1800f460 con 0x7f2b2c100e80 2024-04-20T22:05:05.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:05.646+0000 7f2b34200700 1 -- 172.21.15.62:0/4185932859 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b2c19cd60 con 0x7f2b2c100e80 2024-04-20T22:05:05.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:05.647+0000 7f2b2a000700 1 -- 172.21.15.62:0/4185932859 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2b18017630 con 0x7f2b2c100e80 2024-04-20T22:05:05.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:05.647+0000 7f2b34200700 1 -- 172.21.15.62:0/4185932859 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b2c19d250 con 0x7f2b2c100e80 2024-04-20T22:05:05.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:05.648+0000 7f2b34200700 1 -- 172.21.15.62:0/4185932859 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b2c1045d0 con 0x7f2b2c100e80 2024-04-20T22:05:05.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:05.649+0000 7f2b2a000700 1 -- 172.21.15.62:0/4185932859 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f2b1800fab0 con 0x7f2b2c100e80 2024-04-20T22:05:05.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:05.650+0000 7f2b2a000700 1 --2- 172.21.15.62:0/4185932859 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2b1406ecc0 0x7f2b14071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:05:05.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:05.651+0000 7f2b2a000700 1 -- 172.21.15.62:0/4185932859 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f2b180903e0 con 0x7f2b2c100e80 2024-04-20T22:05:05.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:05.651+0000 7f2b31800700 1 --2- 172.21.15.62:0/4185932859 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2b1406ecc0 0x7f2b14071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:05:05.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:05.652+0000 7f2b31800700 1 --2- 172.21.15.62:0/4185932859 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2b1406ecc0 0x7f2b14071170 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f2b20005950 tx=0x7f2b2000a300 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:05:05.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:05.656+0000 7f2b2a000700 1 -- 172.21.15.62:0/4185932859 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f2b180590f0 con 0x7f2b2c100e80 2024-04-20T22:05:05.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:05.937+0000 7f2b34200700 1 -- 172.21.15.62:0/4185932859 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f2b2c04f2a0 con 0x7f2b2c100e80 2024-04-20T22:05:05.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:05.937+0000 7f2b2a000700 1 -- 172.21.15.62:0/4185932859 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 27 v27) v1 ==== 76+0+1888 (secure 0 0 0) 0x7f2b1805c7b0 con 0x7f2b2c100e80 2024-04-20T22:05:05.938 INFO:teuthology.orchestra.run.smithi062.stdout:e27 2024-04-20T22:05:05.938 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T22:05:05.938 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:05:05.938 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T22:05:05.938 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:05:05.939 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T22:05:05.939 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T22:05:05.939 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 20 2024-04-20T22:05:05.939 INFO:teuthology.orchestra.run.smithi062.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-20T22:05:05.939 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T22:05:05.939 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T21:42:01.559061+0000 2024-04-20T22:05:05.939 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T22:05:05.939 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T22:05:05.939 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T22:05:05.939 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T22:05:05.939 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T22:05:05.939 INFO:teuthology.orchestra.run.smithi062.stdout:max_xattr_size 65536 2024-04-20T22:05:05.939 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T22:05:05.939 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T22:05:05.940 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 81 2024-04-20T22:05:05.940 INFO:teuthology.orchestra.run.smithi062.stdout:snap_rstat 1 2024-04-20T22:05:05.940 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:05:05.940 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 2 2024-04-20T22:05:05.940 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1 2024-04-20T22:05:05.940 INFO:teuthology.orchestra.run.smithi062.stdout:up {} 2024-04-20T22:05:05.940 INFO:teuthology.orchestra.run.smithi062.stdout:failed 0,1 2024-04-20T22:05:05.940 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T22:05:05.940 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T22:05:05.940 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T22:05:05.941 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T22:05:05.941 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data enabled 2024-04-20T22:05:05.941 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T22:05:05.941 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T22:05:05.941 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 1 2024-04-20T22:05:05.941 INFO:teuthology.orchestra.run.smithi062.stdout:qdb_cluster leader: 0 members: 2024-04-20T22:05:05.941 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:05:05.941 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:05:05.941 INFO:teuthology.orchestra.run.smithi062.stdout:Standby daemons: 2024-04-20T22:05:05.941 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:05:05.942 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ryqblb{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6826/4113916205,v1:172.21.15.62:6827/4113916205] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:05:05.942 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ozizwi{-1:34476} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6828/3801975165,v1:172.21.15.62:6829/3801975165] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:05:05.942 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.xbgspw{-1:44285} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6826/3996040526,v1:172.21.15.189:6827/3996040526] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T22:05:05.942 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.dvimer{-1:44727} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6824/3576699232,v1:172.21.15.189:6825/3576699232] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:05:05.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:05.940+0000 7f2b34200700 1 -- 172.21.15.62:0/4185932859 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2b1406ecc0 msgr2=0x7f2b14071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:05:05.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:05.940+0000 7f2b34200700 1 --2- 172.21.15.62:0/4185932859 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2b1406ecc0 0x7f2b14071170 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f2b20005950 tx=0x7f2b2000a300 comp rx=0 tx=0).stop 2024-04-20T22:05:05.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:05.940+0000 7f2b34200700 1 -- 172.21.15.62:0/4185932859 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2b2c100e80 msgr2=0x7f2b2c198670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:05:05.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:05.940+0000 7f2b34200700 1 --2- 172.21.15.62:0/4185932859 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2b2c100e80 0x7f2b2c198670 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f2b1800bda0 tx=0x7f2b1800bdd0 comp rx=0 tx=0).stop 2024-04-20T22:05:05.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:05.940+0000 7f2b34200700 1 -- 172.21.15.62:0/4185932859 shutdown_connections 2024-04-20T22:05:05.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:05.940+0000 7f2b34200700 1 --2- 172.21.15.62:0/4185932859 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2b1406ecc0 0x7f2b14071170 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:05:05.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:05.940+0000 7f2b34200700 1 --2- 172.21.15.62:0/4185932859 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2b2c1004e0 0x7f2b2c198130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:05:05.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:05.940+0000 7f2b34200700 1 --2- 172.21.15.62:0/4185932859 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2b2c100e80 0x7f2b2c198670 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:05:05.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:05.940+0000 7f2b34200700 1 -- 172.21.15.62:0/4185932859 >> 172.21.15.62:0/4185932859 conn(0x7f2b2c0fbda0 msgr2=0x7f2b2c0fe160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:05:05.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:05.941+0000 7f2b34200700 1 -- 172.21.15.62:0/4185932859 shutdown_connections 2024-04-20T22:05:05.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:05.941+0000 7f2b34200700 1 -- 172.21.15.62:0/4185932859 wait complete. 2024-04-20T22:05:05.944 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 27 2024-04-20T22:05:06.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:06.111+0000 7f5a46800700 1 -- 172.21.15.62:0/1526094569 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5a400687a0 msgr2=0x7f5a40068b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:05:06.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:06.111+0000 7f5a46800700 1 --2- 172.21.15.62:0/1526094569 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5a400687a0 0x7f5a40068b70 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f5a30009910 tx=0x7f5a30009c20 comp rx=0 tx=0).stop 2024-04-20T22:05:06.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:06.111+0000 7f5a46800700 1 -- 172.21.15.62:0/1526094569 shutdown_connections 2024-04-20T22:05:06.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:06.111+0000 7f5a46800700 1 --2- 172.21.15.62:0/1526094569 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5a40069140 0x7f5a40104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:05:06.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:06.111+0000 7f5a46800700 1 --2- 172.21.15.62:0/1526094569 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5a400687a0 0x7f5a40068b70 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:05:06.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:06.111+0000 7f5a46800700 1 -- 172.21.15.62:0/1526094569 >> 172.21.15.62:0/1526094569 conn(0x7f5a40075130 msgr2=0x7f5a40075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:05:06.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:06.112+0000 7f5a46800700 1 -- 172.21.15.62:0/1526094569 shutdown_connections 2024-04-20T22:05:06.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:06.112+0000 7f5a46800700 1 -- 172.21.15.62:0/1526094569 wait complete. 2024-04-20T22:05:06.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:06.113+0000 7f5a46800700 1 Processor -- start 2024-04-20T22:05:06.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:06.113+0000 7f5a46800700 1 -- start start 2024-04-20T22:05:06.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:06.113+0000 7f5a46800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5a400687a0 0x7f5a40195ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:05:06.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:06.113+0000 7f5a46800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5a40069140 0x7f5a40196420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:05:06.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:06.113+0000 7f5a46800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a40196b00 con 0x7f5a400687a0 2024-04-20T22:05:06.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:06.113+0000 7f5a46800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a4019a890 con 0x7f5a40069140 2024-04-20T22:05:06.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:06.113+0000 7f5a3f400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5a400687a0 0x7f5a40195ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:05:06.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:06.114+0000 7f5a3f400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5a400687a0 0x7f5a40195ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:36598/0 (socket says 172.21.15.62:36598) 2024-04-20T22:05:06.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:06.114+0000 7f5a3f400700 1 -- 172.21.15.62:0/2571991402 learned_addr learned my addr 172.21.15.62:0/2571991402 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:05:06.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:06.114+0000 7f5a3ea00700 1 --2- 172.21.15.62:0/2571991402 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5a40069140 0x7f5a40196420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:05:06.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:06.114+0000 7f5a3f400700 1 -- 172.21.15.62:0/2571991402 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5a40069140 msgr2=0x7f5a40196420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:05:06.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:06.114+0000 7f5a3f400700 1 --2- 172.21.15.62:0/2571991402 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5a40069140 0x7f5a40196420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:05:06.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:06.114+0000 7f5a3f400700 1 -- 172.21.15.62:0/2571991402 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a300095f0 con 0x7f5a400687a0 2024-04-20T22:05:06.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:06.114+0000 7f5a3ea00700 1 --2- 172.21.15.62:0/2571991402 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5a40069140 0x7f5a40196420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T22:05:06.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:06.114+0000 7f5a3f400700 1 --2- 172.21.15.62:0/2571991402 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5a400687a0 0x7f5a40195ee0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f5a3000f690 tx=0x7f5a3000f770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:05:06.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:06.115+0000 7f5a37e00700 1 -- 172.21.15.62:0/2571991402 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5a3001d070 con 0x7f5a400687a0 2024-04-20T22:05:06.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:06.115+0000 7f5a37e00700 1 -- 172.21.15.62:0/2571991402 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5a300050d0 con 0x7f5a400687a0 2024-04-20T22:05:06.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:06.115+0000 7f5a37e00700 1 -- 172.21.15.62:0/2571991402 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5a30022470 con 0x7f5a400687a0 2024-04-20T22:05:06.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:06.115+0000 7f5a46800700 1 -- 172.21.15.62:0/2571991402 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a4019ab10 con 0x7f5a400687a0 2024-04-20T22:05:06.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:06.115+0000 7f5a46800700 1 -- 172.21.15.62:0/2571991402 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a4019b000 con 0x7f5a400687a0 2024-04-20T22:05:06.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:06.116+0000 7f5a46800700 1 -- 172.21.15.62:0/2571991402 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a40108790 con 0x7f5a400687a0 2024-04-20T22:05:06.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:06.117+0000 7f5a37e00700 1 -- 172.21.15.62:0/2571991402 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f5a30003680 con 0x7f5a400687a0 2024-04-20T22:05:06.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:06.118+0000 7f5a37e00700 1 --2- 172.21.15.62:0/2571991402 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5a2406ecc0 0x7f5a24071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:05:06.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:06.118+0000 7f5a37e00700 1 -- 172.21.15.62:0/2571991402 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f5a300902e0 con 0x7f5a400687a0 2024-04-20T22:05:06.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:06.118+0000 7f5a3ea00700 1 --2- 172.21.15.62:0/2571991402 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5a2406ecc0 0x7f5a24071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:05:06.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:06.119+0000 7f5a3ea00700 1 --2- 172.21.15.62:0/2571991402 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5a2406ecc0 0x7f5a24071170 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f5a40197500 tx=0x7f5a2c009260 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:05:06.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:06.123+0000 7f5a37e00700 1 -- 172.21.15.62:0/2571991402 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f5a30059070 con 0x7f5a400687a0 2024-04-20T22:05:06.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:05:06 smithi062 ceph-mon[76322]: from='client.35604 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:05:06.299 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:05:06 smithi062 ceph-mon[76322]: from='client.35608 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:05:06.299 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:05:06 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/779881331' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:05:06.299 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:05:06 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/4185932859' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:05:06.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:06.387+0000 7f5a46800700 1 -- 172.21.15.62:0/2571991402 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5a4019b270 con 0x7f5a2406ecc0 2024-04-20T22:05:06.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:06.390+0000 7f5a37e00700 1 -- 172.21.15.62:0/2571991402 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f5a4019b270 con 0x7f5a2406ecc0 2024-04-20T22:05:06.390 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:05:06.390 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T22:05:06.390 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T22:05:06.390 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-20T22:05:06.390 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T22:05:06.390 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr", 2024-04-20T22:05:06.391 INFO:teuthology.orchestra.run.smithi062.stdout: "crash", 2024-04-20T22:05:06.391 INFO:teuthology.orchestra.run.smithi062.stdout: "mon", 2024-04-20T22:05:06.391 INFO:teuthology.orchestra.run.smithi062.stdout: "osd" 2024-04-20T22:05:06.391 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T22:05:06.391 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "15/23 daemons upgraded", 2024-04-20T22:05:06.391 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading mds daemons", 2024-04-20T22:05:06.391 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T22:05:06.391 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:05:06.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:06.393+0000 7f5a46800700 1 -- 172.21.15.62:0/2571991402 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5a2406ecc0 msgr2=0x7f5a24071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:05:06.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:06.393+0000 7f5a46800700 1 --2- 172.21.15.62:0/2571991402 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5a2406ecc0 0x7f5a24071170 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f5a40197500 tx=0x7f5a2c009260 comp rx=0 tx=0).stop 2024-04-20T22:05:06.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:06.393+0000 7f5a46800700 1 -- 172.21.15.62:0/2571991402 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5a400687a0 msgr2=0x7f5a40195ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:05:06.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:06.393+0000 7f5a46800700 1 --2- 172.21.15.62:0/2571991402 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5a400687a0 0x7f5a40195ee0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f5a3000f690 tx=0x7f5a3000f770 comp rx=0 tx=0).stop 2024-04-20T22:05:06.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:06.393+0000 7f5a46800700 1 -- 172.21.15.62:0/2571991402 shutdown_connections 2024-04-20T22:05:06.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:06.393+0000 7f5a46800700 1 --2- 172.21.15.62:0/2571991402 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5a2406ecc0 0x7f5a24071170 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:05:06.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:06.393+0000 7f5a46800700 1 --2- 172.21.15.62:0/2571991402 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5a400687a0 0x7f5a40195ee0 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:05:06.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:06.393+0000 7f5a46800700 1 --2- 172.21.15.62:0/2571991402 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5a40069140 0x7f5a40196420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:05:06.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:06.393+0000 7f5a46800700 1 -- 172.21.15.62:0/2571991402 >> 172.21.15.62:0/2571991402 conn(0x7f5a40075130 msgr2=0x7f5a400fe7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:05:06.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:06.394+0000 7f5a46800700 1 -- 172.21.15.62:0/2571991402 shutdown_connections 2024-04-20T22:05:06.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:06.394+0000 7f5a46800700 1 -- 172.21.15.62:0/2571991402 wait complete. 2024-04-20T22:05:06.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:05:06 smithi189 ceph-mon[57022]: from='client.35604 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:05:06.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:05:06 smithi189 ceph-mon[57022]: from='client.35608 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:05:06.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:05:06 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/779881331' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:05:06.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:05:06 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/4185932859' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:05:06.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:06.557+0000 7f12ae800700 1 -- 172.21.15.62:0/3024291521 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12a8102470 msgr2=0x7f12a81028e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:05:06.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:06.557+0000 7f12ae800700 1 --2- 172.21.15.62:0/3024291521 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12a8102470 0x7f12a81028e0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f1294009910 tx=0x7f1294009c20 comp rx=0 tx=0).stop 2024-04-20T22:05:06.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:06.557+0000 7f12ae800700 1 -- 172.21.15.62:0/3024291521 shutdown_connections 2024-04-20T22:05:06.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:06.557+0000 7f12ae800700 1 --2- 172.21.15.62:0/3024291521 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12a8102470 0x7f12a81028e0 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:05:06.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:06.557+0000 7f12ae800700 1 --2- 172.21.15.62:0/3024291521 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12a8108470 0x7f12a8108840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:05:06.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:06.557+0000 7f12ae800700 1 -- 172.21.15.62:0/3024291521 >> 172.21.15.62:0/3024291521 conn(0x7f12a80fdf90 msgr2=0x7f12a81003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:05:06.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:06.557+0000 7f12ae800700 1 -- 172.21.15.62:0/3024291521 shutdown_connections 2024-04-20T22:05:06.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:06.557+0000 7f12ae800700 1 -- 172.21.15.62:0/3024291521 wait complete. 2024-04-20T22:05:06.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:06.559+0000 7f12ae800700 1 Processor -- start 2024-04-20T22:05:06.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:06.559+0000 7f12ae800700 1 -- start start 2024-04-20T22:05:06.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:06.559+0000 7f12ae800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12a8102470 0x7f12a81981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:05:06.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:06.559+0000 7f12ae800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12a8108470 0x7f12a8198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:05:06.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:06.559+0000 7f12ae800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12a8198e10 con 0x7f12a8102470 2024-04-20T22:05:06.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:06.559+0000 7f12ae800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12a819cba0 con 0x7f12a8108470 2024-04-20T22:05:06.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:06.560+0000 7f12a7400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12a8102470 0x7f12a81981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:05:06.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:06.560+0000 7f12a7400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12a8102470 0x7f12a81981f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:36620/0 (socket says 172.21.15.62:36620) 2024-04-20T22:05:06.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:06.560+0000 7f12a7400700 1 -- 172.21.15.62:0/4016000862 learned_addr learned my addr 172.21.15.62:0/4016000862 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:05:06.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:06.560+0000 7f12a7400700 1 -- 172.21.15.62:0/4016000862 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12a8108470 msgr2=0x7f12a8198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:05:06.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:06.560+0000 7f12a7400700 1 --2- 172.21.15.62:0/4016000862 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12a8108470 0x7f12a8198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:05:06.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:06.560+0000 7f12a7400700 1 -- 172.21.15.62:0/4016000862 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12940095f0 con 0x7f12a8102470 2024-04-20T22:05:06.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:06.560+0000 7f12a7400700 1 --2- 172.21.15.62:0/4016000862 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12a8102470 0x7f12a81981f0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f129800d8d0 tx=0x7f129800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:05:06.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:06.561+0000 7f129fe00700 1 -- 172.21.15.62:0/4016000862 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1298009940 con 0x7f12a8102470 2024-04-20T22:05:06.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:06.561+0000 7f129fe00700 1 -- 172.21.15.62:0/4016000862 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1298010460 con 0x7f12a8102470 2024-04-20T22:05:06.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:06.561+0000 7f12ae800700 1 -- 172.21.15.62:0/4016000862 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12a819ce80 con 0x7f12a8102470 2024-04-20T22:05:06.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:06.561+0000 7f129fe00700 1 -- 172.21.15.62:0/4016000862 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f129800f5f0 con 0x7f12a8102470 2024-04-20T22:05:06.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:06.561+0000 7f12ae800700 1 -- 172.21.15.62:0/4016000862 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12a819d3d0 con 0x7f12a8102470 2024-04-20T22:05:06.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:06.562+0000 7f12ae800700 1 -- 172.21.15.62:0/4016000862 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12a810a970 con 0x7f12a8102470 2024-04-20T22:05:06.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:06.563+0000 7f129fe00700 1 -- 172.21.15.62:0/4016000862 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f12980105d0 con 0x7f12a8102470 2024-04-20T22:05:06.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:06.564+0000 7f129fe00700 1 --2- 172.21.15.62:0/4016000862 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f128c06ecc0 0x7f128c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:05:06.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:06.564+0000 7f129fe00700 1 -- 172.21.15.62:0/4016000862 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f129808ee10 con 0x7f12a8102470 2024-04-20T22:05:06.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:06.565+0000 7f12a6a00700 1 --2- 172.21.15.62:0/4016000862 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f128c06ecc0 0x7f128c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:05:06.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:06.565+0000 7f12a6a00700 1 --2- 172.21.15.62:0/4016000862 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f128c06ecc0 0x7f128c071170 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f129400b5c0 tx=0x7f1294005960 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:05:06.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:06.569+0000 7f129fe00700 1 -- 172.21.15.62:0/4016000862 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f1298057ba0 con 0x7f12a8102470 2024-04-20T22:05:06.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:06.897+0000 7f12ae800700 1 -- 172.21.15.62:0/4016000862 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f12a804f2a0 con 0x7f12a8102470 2024-04-20T22:05:06.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:06.898+0000 7f129fe00700 1 -- 172.21.15.62:0/4016000862 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7f129805b260 con 0x7f12a8102470 2024-04-20T22:05:06.898 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-20T22:05:06.898 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-20T22:05:06.898 INFO:teuthology.orchestra.run.smithi062.stdout: host smithi062 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T22:05:06.898 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-20T22:05:06.898 INFO:teuthology.orchestra.run.smithi062.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi062 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpq7y7oa_r:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:05:06.898 INFO:teuthology.orchestra.run.smithi062.stdout:Traceback (most recent call last): 2024-04-20T22:05:06.898 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-20T22:05:06.898 INFO:teuthology.orchestra.run.smithi062.stdout: return _run_code(code, main_globals, None, 2024-04-20T22:05:06.899 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-20T22:05:06.899 INFO:teuthology.orchestra.run.smithi062.stdout: exec(code, run_globals) 2024-04-20T22:05:06.899 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5618, in 2024-04-20T22:05:06.899 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5606, in main 2024-04-20T22:05:06.899 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 408, in _infer_config 2024-04-20T22:05:06.899 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 323, in _infer_fsid 2024-04-20T22:05:06.899 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 436, in _infer_image 2024-04-20T22:05:06.899 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 310, in _validate_fsid 2024-04-20T22:05:06.899 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 3359, in command_ceph_volume 2024-04-20T22:05:06.899 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-20T22:05:06.899 INFO:teuthology.orchestra.run.smithi062.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi062 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpq7y7oa_r:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:05:06.899 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-20T22:05:06.899 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is degraded 2024-04-20T22:05:06.899 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T22:05:06.900 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T22:05:06.900 INFO:teuthology.orchestra.run.smithi062.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-20T22:05:06.900 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-20T22:05:06.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:06.901+0000 7f12ae800700 1 -- 172.21.15.62:0/4016000862 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f128c06ecc0 msgr2=0x7f128c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:05:06.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:06.901+0000 7f12ae800700 1 --2- 172.21.15.62:0/4016000862 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f128c06ecc0 0x7f128c071170 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f129400b5c0 tx=0x7f1294005960 comp rx=0 tx=0).stop 2024-04-20T22:05:06.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:06.901+0000 7f12ae800700 1 -- 172.21.15.62:0/4016000862 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12a8102470 msgr2=0x7f12a81981f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:05:06.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:06.901+0000 7f12ae800700 1 --2- 172.21.15.62:0/4016000862 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12a8102470 0x7f12a81981f0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f129800d8d0 tx=0x7f129800dc90 comp rx=0 tx=0).stop 2024-04-20T22:05:06.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:06.901+0000 7f12ae800700 1 -- 172.21.15.62:0/4016000862 shutdown_connections 2024-04-20T22:05:06.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:06.901+0000 7f12ae800700 1 --2- 172.21.15.62:0/4016000862 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f128c06ecc0 0x7f128c071170 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:05:06.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:06.901+0000 7f12ae800700 1 --2- 172.21.15.62:0/4016000862 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12a8102470 0x7f12a81981f0 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:05:06.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:06.901+0000 7f12ae800700 1 --2- 172.21.15.62:0/4016000862 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12a8108470 0x7f12a8198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:05:06.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:06.901+0000 7f12ae800700 1 -- 172.21.15.62:0/4016000862 >> 172.21.15.62:0/4016000862 conn(0x7f12a80fdf90 msgr2=0x7f12a80ff3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:05:06.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:06.902+0000 7f12ae800700 1 -- 172.21.15.62:0/4016000862 shutdown_connections 2024-04-20T22:05:06.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:06.902+0000 7f12ae800700 1 -- 172.21.15.62:0/4016000862 wait complete. 2024-04-20T22:05:07.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:05:07 smithi062 ceph-mon[76322]: pgmap v895: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:05:07.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:05:07 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/4016000862' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:05:07.389 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:05:07 smithi189 ceph-mon[57022]: pgmap v895: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:05:07.389 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:05:07 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/4016000862' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:05:08.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:05:08 smithi189 ceph-mon[57022]: from='client.35620 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:05:08.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:05:08 smithi062 ceph-mon[76322]: from='client.35620 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:05:09.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:05:09 smithi189 ceph-mon[57022]: pgmap v896: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:05:09.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:05:09 smithi062 ceph-mon[76322]: pgmap v896: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:05:11.465 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:05:11 smithi062 ceph-mon[76322]: pgmap v897: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:05:11.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:05:11 smithi189 ceph-mon[57022]: pgmap v897: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:05:13.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:05:13 smithi189 ceph-mon[57022]: pgmap v898: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:05:13.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:05:13 smithi062 ceph-mon[76322]: pgmap v898: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:05:15.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:05:15 smithi189 ceph-mon[57022]: pgmap v899: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:05:15.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:05:15 smithi062 ceph-mon[76322]: pgmap v899: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:05:17.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:05:17 smithi189 ceph-mon[57022]: pgmap v900: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:05:17.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:05:17 smithi062 ceph-mon[76322]: pgmap v900: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:05:19.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:05:19 smithi189 ceph-mon[57022]: pgmap v901: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:05:19.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:05:19 smithi062 ceph-mon[76322]: pgmap v901: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:05:20.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:05:20 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:05:20.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:05:20 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:05:21.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:05:21 smithi189 ceph-mon[57022]: pgmap v902: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:05:21.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:05:21 smithi062 ceph-mon[76322]: pgmap v902: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:05:23.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:05:23 smithi189 ceph-mon[57022]: pgmap v903: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:05:23.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:05:23 smithi062 ceph-mon[76322]: pgmap v903: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:05:25.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:05:25 smithi189 ceph-mon[57022]: pgmap v904: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:05:25.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:05:25 smithi062 ceph-mon[76322]: pgmap v904: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:05:27.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:05:27 smithi062 ceph-mon[76322]: pgmap v905: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:05:27.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:05:27 smithi189 ceph-mon[57022]: pgmap v905: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:05:29.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:05:29 smithi189 ceph-mon[57022]: pgmap v906: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:05:29.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:05:29 smithi062 ceph-mon[76322]: pgmap v906: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:05:31.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:05:31 smithi189 ceph-mon[57022]: pgmap v907: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:05:31.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:05:31 smithi062 ceph-mon[76322]: pgmap v907: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:05:33.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:05:33 smithi189 ceph-mon[57022]: pgmap v908: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:05:33.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:05:33 smithi062 ceph-mon[76322]: pgmap v908: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:05:35.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:05:35 smithi189 ceph-mon[57022]: pgmap v909: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:05:35.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:05:35 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:05:35.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:05:35 smithi062 ceph-mon[76322]: pgmap v909: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:05:35.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:05:35 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:05:37.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.078+0000 7f5be0c00700 1 -- 172.21.15.62:0/1399467641 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5bdc10c8f0 msgr2=0x7f5bdc10ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:05:37.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.078+0000 7f5be0c00700 1 --2- 172.21.15.62:0/1399467641 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5bdc10c8f0 0x7f5bdc10ccc0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f5bcc009910 tx=0x7f5bcc009c20 comp rx=0 tx=0).stop 2024-04-20T22:05:37.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.078+0000 7f5be0c00700 1 -- 172.21.15.62:0/1399467641 shutdown_connections 2024-04-20T22:05:37.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.078+0000 7f5be0c00700 1 --2- 172.21.15.62:0/1399467641 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5bdc106b80 0x7f5bdc106ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:05:37.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.078+0000 7f5be0c00700 1 --2- 172.21.15.62:0/1399467641 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5bdc10c8f0 0x7f5bdc10ccc0 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:05:37.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.078+0000 7f5be0c00700 1 -- 172.21.15.62:0/1399467641 >> 172.21.15.62:0/1399467641 conn(0x7f5bdc076060 msgr2=0x7f5bdc078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:05:37.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.078+0000 7f5be0c00700 1 -- 172.21.15.62:0/1399467641 shutdown_connections 2024-04-20T22:05:37.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.078+0000 7f5be0c00700 1 -- 172.21.15.62:0/1399467641 wait complete. 2024-04-20T22:05:37.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.079+0000 7f5be0c00700 1 Processor -- start 2024-04-20T22:05:37.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.080+0000 7f5be0c00700 1 -- start start 2024-04-20T22:05:37.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.080+0000 7f5be0c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5bdc106b80 0x7f5bdc19c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:05:37.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.080+0000 7f5be0c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5bdc10c8f0 0x7f5bdc19ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:05:37.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.080+0000 7f5be0c00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bdc19d120 con 0x7f5bdc106b80 2024-04-20T22:05:37.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.080+0000 7f5be0c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bdc1a0eb0 con 0x7f5bdc10c8f0 2024-04-20T22:05:37.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.081+0000 7f5bd9e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5bdc106b80 0x7f5bdc19c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:05:37.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.081+0000 7f5bd9e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5bdc106b80 0x7f5bdc19c500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49200/0 (socket says 172.21.15.62:49200) 2024-04-20T22:05:37.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.081+0000 7f5bd9e00700 1 -- 172.21.15.62:0/3103647018 learned_addr learned my addr 172.21.15.62:0/3103647018 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:05:37.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.081+0000 7f5bd9e00700 1 -- 172.21.15.62:0/3103647018 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5bdc10c8f0 msgr2=0x7f5bdc19ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:05:37.085 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.081+0000 7f5bd9e00700 1 --2- 172.21.15.62:0/3103647018 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5bdc10c8f0 0x7f5bdc19ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:05:37.085 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.081+0000 7f5bd9e00700 1 -- 172.21.15.62:0/3103647018 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bcc0095f0 con 0x7f5bdc106b80 2024-04-20T22:05:37.085 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.081+0000 7f5bd9e00700 1 --2- 172.21.15.62:0/3103647018 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5bdc106b80 0x7f5bdc19c500 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f5bcc0092e0 tx=0x7f5bcc00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:05:37.085 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.082+0000 7f5bd2a00700 1 -- 172.21.15.62:0/3103647018 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5bcc01d070 con 0x7f5bdc106b80 2024-04-20T22:05:37.085 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.082+0000 7f5bd2a00700 1 -- 172.21.15.62:0/3103647018 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5bcc005e00 con 0x7f5bdc106b80 2024-04-20T22:05:37.085 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.082+0000 7f5bd2a00700 1 -- 172.21.15.62:0/3103647018 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5bcc017400 con 0x7f5bdc106b80 2024-04-20T22:05:37.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.082+0000 7f5be0c00700 1 -- 172.21.15.62:0/3103647018 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bdc1a1130 con 0x7f5bdc106b80 2024-04-20T22:05:37.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.082+0000 7f5be0c00700 1 -- 172.21.15.62:0/3103647018 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bdc1a15d0 con 0x7f5bdc106b80 2024-04-20T22:05:37.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.083+0000 7f5be0c00700 1 -- 172.21.15.62:0/3103647018 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bdc109b20 con 0x7f5bdc106b80 2024-04-20T22:05:37.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.084+0000 7f5bd2a00700 1 -- 172.21.15.62:0/3103647018 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f5bcc00f460 con 0x7f5bdc106b80 2024-04-20T22:05:37.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.085+0000 7f5bd2a00700 1 --2- 172.21.15.62:0/3103647018 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5bc006ecc0 0x7f5bc0071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:05:37.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.086+0000 7f5bd2a00700 1 -- 172.21.15.62:0/3103647018 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f5bcc090380 con 0x7f5bdc106b80 2024-04-20T22:05:37.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.086+0000 7f5bd9400700 1 --2- 172.21.15.62:0/3103647018 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5bc006ecc0 0x7f5bc0071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:05:37.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.087+0000 7f5bd9400700 1 --2- 172.21.15.62:0/3103647018 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5bc006ecc0 0x7f5bc0071170 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f5bc8005fd0 tx=0x7f5bc8005f60 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:05:37.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.091+0000 7f5bd2a00700 1 -- 172.21.15.62:0/3103647018 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f5bcc059130 con 0x7f5bdc106b80 2024-04-20T22:05:37.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.364+0000 7f5be0c00700 1 -- 172.21.15.62:0/3103647018 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5bdc19d970 con 0x7f5bc006ecc0 2024-04-20T22:05:37.364 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:05:37 smithi062 ceph-mon[76322]: pgmap v910: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:05:37.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.366+0000 7f5bd2a00700 1 -- 172.21.15.62:0/3103647018 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f5bdc19d970 con 0x7f5bc006ecc0 2024-04-20T22:05:37.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.369+0000 7f5be0c00700 1 -- 172.21.15.62:0/3103647018 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5bc006ecc0 msgr2=0x7f5bc0071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:05:37.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.369+0000 7f5be0c00700 1 --2- 172.21.15.62:0/3103647018 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5bc006ecc0 0x7f5bc0071170 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f5bc8005fd0 tx=0x7f5bc8005f60 comp rx=0 tx=0).stop 2024-04-20T22:05:37.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.369+0000 7f5be0c00700 1 -- 172.21.15.62:0/3103647018 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5bdc106b80 msgr2=0x7f5bdc19c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:05:37.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.369+0000 7f5be0c00700 1 --2- 172.21.15.62:0/3103647018 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5bdc106b80 0x7f5bdc19c500 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f5bcc0092e0 tx=0x7f5bcc00be20 comp rx=0 tx=0).stop 2024-04-20T22:05:37.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.370+0000 7f5be0c00700 1 -- 172.21.15.62:0/3103647018 shutdown_connections 2024-04-20T22:05:37.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.370+0000 7f5be0c00700 1 --2- 172.21.15.62:0/3103647018 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5bc006ecc0 0x7f5bc0071170 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:05:37.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.370+0000 7f5be0c00700 1 --2- 172.21.15.62:0/3103647018 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5bdc106b80 0x7f5bdc19c500 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:05:37.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.370+0000 7f5be0c00700 1 --2- 172.21.15.62:0/3103647018 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5bdc10c8f0 0x7f5bdc19ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:05:37.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.370+0000 7f5be0c00700 1 -- 172.21.15.62:0/3103647018 >> 172.21.15.62:0/3103647018 conn(0x7f5bdc076060 msgr2=0x7f5bdc077aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:05:37.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.370+0000 7f5be0c00700 1 -- 172.21.15.62:0/3103647018 shutdown_connections 2024-04-20T22:05:37.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.370+0000 7f5be0c00700 1 -- 172.21.15.62:0/3103647018 wait complete. 2024-04-20T22:05:37.389 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T22:05:37.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:05:37 smithi189 ceph-mon[57022]: pgmap v910: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:05:37.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.537+0000 7f9583e00700 1 -- 172.21.15.62:0/2909813399 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9584101c50 msgr2=0x7f95841020c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:05:37.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.537+0000 7f9583e00700 1 --2- 172.21.15.62:0/2909813399 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9584101c50 0x7f95841020c0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f9570009910 tx=0x7f9570009c20 comp rx=0 tx=0).stop 2024-04-20T22:05:37.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.538+0000 7f9583e00700 1 -- 172.21.15.62:0/2909813399 shutdown_connections 2024-04-20T22:05:37.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.538+0000 7f9583e00700 1 --2- 172.21.15.62:0/2909813399 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9584101c50 0x7f95841020c0 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:05:37.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.538+0000 7f9583e00700 1 --2- 172.21.15.62:0/2909813399 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f95841003f0 0x7f95841007c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:05:37.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.538+0000 7f9583e00700 1 -- 172.21.15.62:0/2909813399 >> 172.21.15.62:0/2909813399 conn(0x7f95840fbbf0 msgr2=0x7f95840fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:05:37.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.538+0000 7f9583e00700 1 -- 172.21.15.62:0/2909813399 shutdown_connections 2024-04-20T22:05:37.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.538+0000 7f9583e00700 1 -- 172.21.15.62:0/2909813399 wait complete. 2024-04-20T22:05:37.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.539+0000 7f9583e00700 1 Processor -- start 2024-04-20T22:05:37.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.539+0000 7f9583e00700 1 -- start start 2024-04-20T22:05:37.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.540+0000 7f9583e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f95841003f0 0x7f9584197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:05:37.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.540+0000 7f9583e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9584101c50 0x7f9584198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:05:37.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.540+0000 7f9583e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9584198c10 con 0x7f9584101c50 2024-04-20T22:05:37.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.540+0000 7f9583e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f958419c9a0 con 0x7f95841003f0 2024-04-20T22:05:37.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.540+0000 7f9582000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9584101c50 0x7f9584198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:05:37.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.540+0000 7f9582000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9584101c50 0x7f9584198530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49232/0 (socket says 172.21.15.62:49232) 2024-04-20T22:05:37.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.540+0000 7f9582000700 1 -- 172.21.15.62:0/1169820134 learned_addr learned my addr 172.21.15.62:0/1169820134 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:05:37.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.540+0000 7f9582a00700 1 --2- 172.21.15.62:0/1169820134 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f95841003f0 0x7f9584197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:05:37.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.541+0000 7f9582000700 1 -- 172.21.15.62:0/1169820134 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f95841003f0 msgr2=0x7f9584197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:05:37.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.541+0000 7f9582000700 1 --2- 172.21.15.62:0/1169820134 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f95841003f0 0x7f9584197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:05:37.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.541+0000 7f9582000700 1 -- 172.21.15.62:0/1169820134 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95700095f0 con 0x7f9584101c50 2024-04-20T22:05:37.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.541+0000 7f9582000700 1 --2- 172.21.15.62:0/1169820134 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9584101c50 0x7f9584198530 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f9570005b40 tx=0x7f957000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:05:37.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.541+0000 7f9588a00700 1 -- 172.21.15.62:0/1169820134 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f957001d070 con 0x7f9584101c50 2024-04-20T22:05:37.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.541+0000 7f9588a00700 1 -- 172.21.15.62:0/1169820134 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f95700048e0 con 0x7f9584101c50 2024-04-20T22:05:37.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.541+0000 7f9583e00700 1 -- 172.21.15.62:0/1169820134 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f958419cc20 con 0x7f9584101c50 2024-04-20T22:05:37.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.541+0000 7f9588a00700 1 -- 172.21.15.62:0/1169820134 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9570003b60 con 0x7f9584101c50 2024-04-20T22:05:37.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.541+0000 7f9583e00700 1 -- 172.21.15.62:0/1169820134 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f958419d130 con 0x7f9584101c50 2024-04-20T22:05:37.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.543+0000 7f9583e00700 1 -- 172.21.15.62:0/1169820134 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9584062380 con 0x7f9584101c50 2024-04-20T22:05:37.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.544+0000 7f9588a00700 1 -- 172.21.15.62:0/1169820134 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f9570004a50 con 0x7f9584101c50 2024-04-20T22:05:37.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.544+0000 7f9588a00700 1 --2- 172.21.15.62:0/1169820134 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f956806ecc0 0x7f9568071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:05:37.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.545+0000 7f9588a00700 1 -- 172.21.15.62:0/1169820134 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f9570090460 con 0x7f9584101c50 2024-04-20T22:05:37.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.545+0000 7f9582a00700 1 --2- 172.21.15.62:0/1169820134 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f956806ecc0 0x7f9568071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:05:37.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.545+0000 7f9582a00700 1 --2- 172.21.15.62:0/1169820134 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f956806ecc0 0x7f9568071170 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f9574009910 tx=0x7f9574008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:05:37.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.550+0000 7f9588a00700 1 -- 172.21.15.62:0/1169820134 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f9570059310 con 0x7f9584101c50 2024-04-20T22:05:37.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.811+0000 7f9583e00700 1 -- 172.21.15.62:0/1169820134 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f958419d500 con 0x7f956806ecc0 2024-04-20T22:05:37.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.813+0000 7f9588a00700 1 -- 172.21.15.62:0/1169820134 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f958419d500 con 0x7f956806ecc0 2024-04-20T22:05:37.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.816+0000 7f9583e00700 1 -- 172.21.15.62:0/1169820134 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f956806ecc0 msgr2=0x7f9568071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:05:37.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.816+0000 7f9583e00700 1 --2- 172.21.15.62:0/1169820134 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f956806ecc0 0x7f9568071170 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f9574009910 tx=0x7f9574008040 comp rx=0 tx=0).stop 2024-04-20T22:05:37.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.816+0000 7f9583e00700 1 -- 172.21.15.62:0/1169820134 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9584101c50 msgr2=0x7f9584198530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:05:37.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.816+0000 7f9583e00700 1 --2- 172.21.15.62:0/1169820134 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9584101c50 0x7f9584198530 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f9570005b40 tx=0x7f957000be20 comp rx=0 tx=0).stop 2024-04-20T22:05:37.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.816+0000 7f9583e00700 1 -- 172.21.15.62:0/1169820134 shutdown_connections 2024-04-20T22:05:37.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.816+0000 7f9583e00700 1 --2- 172.21.15.62:0/1169820134 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f956806ecc0 0x7f9568071170 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:05:37.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.816+0000 7f9583e00700 1 --2- 172.21.15.62:0/1169820134 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f95841003f0 0x7f9584197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:05:37.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.816+0000 7f9583e00700 1 --2- 172.21.15.62:0/1169820134 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9584101c50 0x7f9584198530 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:05:37.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.816+0000 7f9583e00700 1 -- 172.21.15.62:0/1169820134 >> 172.21.15.62:0/1169820134 conn(0x7f95840fbbf0 msgr2=0x7f9584072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:05:37.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.817+0000 7f9583e00700 1 -- 172.21.15.62:0/1169820134 shutdown_connections 2024-04-20T22:05:37.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.817+0000 7f9583e00700 1 -- 172.21.15.62:0/1169820134 wait complete. 2024-04-20T22:05:37.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.981+0000 7fc0d5800700 1 -- 172.21.15.62:0/2240396804 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc0d00687a0 msgr2=0x7fc0d0068b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:05:37.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.981+0000 7fc0d5800700 1 --2- 172.21.15.62:0/2240396804 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc0d00687a0 0x7fc0d0068b70 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fc0c0009910 tx=0x7fc0c0009c20 comp rx=0 tx=0).stop 2024-04-20T22:05:37.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.981+0000 7fc0d5800700 1 -- 172.21.15.62:0/2240396804 shutdown_connections 2024-04-20T22:05:37.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.981+0000 7fc0d5800700 1 --2- 172.21.15.62:0/2240396804 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc0d0069140 0x7fc0d0104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:05:37.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.981+0000 7fc0d5800700 1 --2- 172.21.15.62:0/2240396804 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc0d00687a0 0x7fc0d0068b70 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:05:37.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.981+0000 7fc0d5800700 1 -- 172.21.15.62:0/2240396804 >> 172.21.15.62:0/2240396804 conn(0x7fc0d0075130 msgr2=0x7fc0d0075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:05:37.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.981+0000 7fc0d5800700 1 -- 172.21.15.62:0/2240396804 shutdown_connections 2024-04-20T22:05:37.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.981+0000 7fc0d5800700 1 -- 172.21.15.62:0/2240396804 wait complete. 2024-04-20T22:05:37.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.982+0000 7fc0d5800700 1 Processor -- start 2024-04-20T22:05:37.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.982+0000 7fc0d5800700 1 -- start start 2024-04-20T22:05:37.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.983+0000 7fc0d5800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc0d00687a0 0x7fc0d00ffca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:05:37.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.983+0000 7fc0d5800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc0d0069140 0x7fc0d01001e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:05:37.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.983+0000 7fc0d5800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0d0100720 con 0x7fc0d0069140 2024-04-20T22:05:37.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.983+0000 7fc0d5800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0d0100860 con 0x7fc0d00687a0 2024-04-20T22:05:37.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.983+0000 7fc0ce000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc0d0069140 0x7fc0d01001e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:05:37.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.983+0000 7fc0ce000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc0d0069140 0x7fc0d01001e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49254/0 (socket says 172.21.15.62:49254) 2024-04-20T22:05:37.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.983+0000 7fc0ce000700 1 -- 172.21.15.62:0/285126561 learned_addr learned my addr 172.21.15.62:0/285126561 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:05:37.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.984+0000 7fc0cea00700 1 --2- 172.21.15.62:0/285126561 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc0d00687a0 0x7fc0d00ffca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:05:37.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.984+0000 7fc0ce000700 1 -- 172.21.15.62:0/285126561 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc0d00687a0 msgr2=0x7fc0d00ffca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:05:37.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.984+0000 7fc0ce000700 1 --2- 172.21.15.62:0/285126561 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc0d00687a0 0x7fc0d00ffca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:05:37.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.984+0000 7fc0ce000700 1 -- 172.21.15.62:0/285126561 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0c00095f0 con 0x7fc0d0069140 2024-04-20T22:05:37.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.984+0000 7fc0ce000700 1 --2- 172.21.15.62:0/285126561 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc0d0069140 0x7fc0d01001e0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fc0bc00c8f0 tx=0x7fc0bc00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:05:37.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.984+0000 7fc0c7400700 1 -- 172.21.15.62:0/285126561 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc0bc007a50 con 0x7fc0d0069140 2024-04-20T22:05:37.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.984+0000 7fc0c7400700 1 -- 172.21.15.62:0/285126561 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc0bc007bb0 con 0x7fc0d0069140 2024-04-20T22:05:37.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.984+0000 7fc0c7400700 1 -- 172.21.15.62:0/285126561 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc0bc018630 con 0x7fc0d0069140 2024-04-20T22:05:37.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.984+0000 7fc0d5800700 1 -- 172.21.15.62:0/285126561 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0d01a20d0 con 0x7fc0d0069140 2024-04-20T22:05:37.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.984+0000 7fc0d5800700 1 -- 172.21.15.62:0/285126561 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0d01a24e0 con 0x7fc0d0069140 2024-04-20T22:05:37.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.986+0000 7fc0d5800700 1 -- 172.21.15.62:0/285126561 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0d0108820 con 0x7fc0d0069140 2024-04-20T22:05:37.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.987+0000 7fc0c7400700 1 -- 172.21.15.62:0/285126561 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fc0bc007d20 con 0x7fc0d0069140 2024-04-20T22:05:37.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.988+0000 7fc0c7400700 1 --2- 172.21.15.62:0/285126561 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc0b406ecc0 0x7fc0b4071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:05:37.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.988+0000 7fc0c7400700 1 -- 172.21.15.62:0/285126561 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fc0bc08ee00 con 0x7fc0d0069140 2024-04-20T22:05:37.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.988+0000 7fc0cea00700 1 --2- 172.21.15.62:0/285126561 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc0b406ecc0 0x7fc0b4071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:05:37.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.989+0000 7fc0cea00700 1 --2- 172.21.15.62:0/285126561 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc0b406ecc0 0x7fc0b4071170 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fc0c0000c00 tx=0x7fc0c0005fd0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:05:37.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:37.992+0000 7fc0c7400700 1 -- 172.21.15.62:0/285126561 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fc0bc057b90 con 0x7fc0d0069140 2024-04-20T22:05:38.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:38.235+0000 7fc0d5800700 1 -- 172.21.15.62:0/285126561 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fc0d0061190 con 0x7fc0b406ecc0 2024-04-20T22:05:38.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:38.248+0000 7fc0c7400700 1 -- 172.21.15.62:0/285126561 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fc0d0061190 con 0x7fc0b406ecc0 2024-04-20T22:05:38.248 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T22:05:38.248 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (31m) 7m ago 45m 15.3M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T22:05:38.248 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (45m) 7m ago 45m 10.4M - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T22:05:38.248 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (44m) 7m ago 44m 10.1M - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T22:05:38.248 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (27m) 7m ago 45m 6660k - 19.0.0-3133-ga8c84f59 76bc6b91b726 2825d5ba11c6 2024-04-20T22:05:38.248 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (27m) 7m ago 44m 6665k - 19.0.0-3133-ga8c84f59 76bc6b91b726 20f746fa1bba 2024-04-20T22:05:38.248 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (30m) 7m ago 44m 73.5M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T22:05:38.249 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (22m) 7m ago 38m 18.5M - 19.0.0-3133-ga8c84f59 76bc6b91b726 de80bc90a683 2024-04-20T22:05:38.249 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (23m) 7m ago 38m 18.7M - 19.0.0-3133-ga8c84f59 76bc6b91b726 cd793b7ee98e 2024-04-20T22:05:38.249 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (7m) 7m ago 38m 11.0M - 19.0.0-3133-ga8c84f59 76bc6b91b726 0e7b0bc4d1ab 2024-04-20T22:05:38.249 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (37m) 7m ago 37m 18.6M - 18.2.0 dc2bc1663786 f2eec2ee8951 2024-04-20T22:05:38.249 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (34m) 7m ago 47m 513M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T22:05:38.249 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (33m) 7m ago 44m 459M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T22:05:38.249 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (29m) 7m ago 47m 71.7M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 019eb1479485 2024-04-20T22:05:38.249 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (28m) 7m ago 43m 51.0M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 55aac560aab6 2024-04-20T22:05:38.249 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (32m) 7m ago 45m 8472k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T22:05:38.249 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (32m) 7m ago 44m 8048k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T22:05:38.249 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (26m) 7m ago 42m 91.2M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 ee2bb901f26e 2024-04-20T22:05:38.249 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (26m) 7m ago 42m 94.6M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 9075152d4162 2024-04-20T22:05:38.250 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (25m) 7m ago 41m 78.1M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 2d4f3e38dc0e 2024-04-20T22:05:38.250 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (25m) 7m ago 41m 92.6M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 8f97b17ae0eb 2024-04-20T22:05:38.250 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (24m) 7m ago 40m 95.6M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 653d8079967d 2024-04-20T22:05:38.250 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (23m) 7m ago 40m 89.2M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 596dded14c12 2024-04-20T22:05:38.250 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (32m) 7m ago 44m 82.1M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T22:05:38.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:38.251+0000 7fc0d5800700 1 -- 172.21.15.62:0/285126561 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc0b406ecc0 msgr2=0x7fc0b4071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:05:38.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:38.251+0000 7fc0d5800700 1 --2- 172.21.15.62:0/285126561 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc0b406ecc0 0x7fc0b4071170 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fc0c0000c00 tx=0x7fc0c0005fd0 comp rx=0 tx=0).stop 2024-04-20T22:05:38.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:38.251+0000 7fc0d5800700 1 -- 172.21.15.62:0/285126561 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc0d0069140 msgr2=0x7fc0d01001e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:05:38.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:38.251+0000 7fc0d5800700 1 --2- 172.21.15.62:0/285126561 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc0d0069140 0x7fc0d01001e0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fc0bc00c8f0 tx=0x7fc0bc00ccb0 comp rx=0 tx=0).stop 2024-04-20T22:05:38.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:38.251+0000 7fc0d5800700 1 -- 172.21.15.62:0/285126561 shutdown_connections 2024-04-20T22:05:38.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:38.251+0000 7fc0d5800700 1 --2- 172.21.15.62:0/285126561 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc0b406ecc0 0x7fc0b4071170 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:05:38.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:38.251+0000 7fc0d5800700 1 --2- 172.21.15.62:0/285126561 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc0d00687a0 0x7fc0d00ffca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:05:38.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:38.251+0000 7fc0d5800700 1 --2- 172.21.15.62:0/285126561 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc0d0069140 0x7fc0d01001e0 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:05:38.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:38.251+0000 7fc0d5800700 1 -- 172.21.15.62:0/285126561 >> 172.21.15.62:0/285126561 conn(0x7fc0d0075130 msgr2=0x7fc0d00ff4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:05:38.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:38.251+0000 7fc0d5800700 1 -- 172.21.15.62:0/285126561 shutdown_connections 2024-04-20T22:05:38.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:38.252+0000 7fc0d5800700 1 -- 172.21.15.62:0/285126561 wait complete. 2024-04-20T22:05:38.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:38.408+0000 7f2c4a000700 1 -- 172.21.15.62:0/3585385554 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c44075280 msgr2=0x7f2c44075650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:05:38.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:38.408+0000 7f2c4a000700 1 --2- 172.21.15.62:0/3585385554 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c44075280 0x7f2c44075650 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f2c34009910 tx=0x7f2c34009c20 comp rx=0 tx=0).stop 2024-04-20T22:05:38.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:38.408+0000 7f2c4a000700 1 -- 172.21.15.62:0/3585385554 shutdown_connections 2024-04-20T22:05:38.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:38.408+0000 7f2c4a000700 1 --2- 172.21.15.62:0/3585385554 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c44075b90 0x7f2c44110bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:05:38.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:38.408+0000 7f2c4a000700 1 --2- 172.21.15.62:0/3585385554 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c44075280 0x7f2c44075650 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:05:38.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:38.408+0000 7f2c4a000700 1 -- 172.21.15.62:0/3585385554 >> 172.21.15.62:0/3585385554 conn(0x7f2c440fdf90 msgr2=0x7f2c441003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:05:38.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:38.409+0000 7f2c4a000700 1 -- 172.21.15.62:0/3585385554 shutdown_connections 2024-04-20T22:05:38.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:38.409+0000 7f2c4a000700 1 -- 172.21.15.62:0/3585385554 wait complete. 2024-04-20T22:05:38.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:38.410+0000 7f2c4a000700 1 Processor -- start 2024-04-20T22:05:38.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:38.410+0000 7f2c4a000700 1 -- start start 2024-04-20T22:05:38.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:38.411+0000 7f2c4a000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c44075280 0x7f2c4419c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:05:38.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:38.411+0000 7f2c4a000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c44075b90 0x7f2c4419c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:05:38.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:38.411+0000 7f2c4a000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c4419d050 con 0x7f2c44075280 2024-04-20T22:05:38.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:38.411+0000 7f2c4a000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c441a0de0 con 0x7f2c44075b90 2024-04-20T22:05:38.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:38.411+0000 7f2c43400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c44075280 0x7f2c4419c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:05:38.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:38.411+0000 7f2c43400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c44075280 0x7f2c4419c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49264/0 (socket says 172.21.15.62:49264) 2024-04-20T22:05:38.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:38.411+0000 7f2c43400700 1 -- 172.21.15.62:0/3556928366 learned_addr learned my addr 172.21.15.62:0/3556928366 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:05:38.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:38.411+0000 7f2c42a00700 1 --2- 172.21.15.62:0/3556928366 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c44075b90 0x7f2c4419c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:05:38.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:38.412+0000 7f2c43400700 1 -- 172.21.15.62:0/3556928366 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c44075b90 msgr2=0x7f2c4419c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:05:38.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:38.412+0000 7f2c43400700 1 --2- 172.21.15.62:0/3556928366 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c44075b90 0x7f2c4419c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:05:38.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:38.412+0000 7f2c43400700 1 -- 172.21.15.62:0/3556928366 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c340095f0 con 0x7f2c44075280 2024-04-20T22:05:38.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:38.412+0000 7f2c43400700 1 --2- 172.21.15.62:0/3556928366 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c44075280 0x7f2c4419c430 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f2c34005b40 tx=0x7f2c3400b660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:05:38.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:38.412+0000 7f2c3be00700 1 -- 172.21.15.62:0/3556928366 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2c3401d070 con 0x7f2c44075280 2024-04-20T22:05:38.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:38.412+0000 7f2c3be00700 1 -- 172.21.15.62:0/3556928366 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2c340052a0 con 0x7f2c44075280 2024-04-20T22:05:38.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:38.412+0000 7f2c4a000700 1 -- 172.21.15.62:0/3556928366 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c441a1060 con 0x7f2c44075280 2024-04-20T22:05:38.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:38.412+0000 7f2c3be00700 1 -- 172.21.15.62:0/3556928366 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2c34022470 con 0x7f2c44075280 2024-04-20T22:05:38.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:38.413+0000 7f2c4a000700 1 -- 172.21.15.62:0/3556928366 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c441a1550 con 0x7f2c44075280 2024-04-20T22:05:38.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:38.414+0000 7f2c4a000700 1 -- 172.21.15.62:0/3556928366 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c4410e340 con 0x7f2c44075280 2024-04-20T22:05:38.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:38.415+0000 7f2c3be00700 1 -- 172.21.15.62:0/3556928366 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f2c3400f850 con 0x7f2c44075280 2024-04-20T22:05:38.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:38.417+0000 7f2c3be00700 1 --2- 172.21.15.62:0/3556928366 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2c2806ecc0 0x7f2c28071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:05:38.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:38.417+0000 7f2c3be00700 1 -- 172.21.15.62:0/3556928366 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f2c340902f0 con 0x7f2c44075280 2024-04-20T22:05:38.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:38.417+0000 7f2c42a00700 1 --2- 172.21.15.62:0/3556928366 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2c2806ecc0 0x7f2c28071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:05:38.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:38.418+0000 7f2c42a00700 1 --2- 172.21.15.62:0/3556928366 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2c2806ecc0 0x7f2c28071170 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f2c4419da50 tx=0x7f2c30009260 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:05:38.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:38.421+0000 7f2c3be00700 1 -- 172.21.15.62:0/3556928366 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f2c34059100 con 0x7f2c44075280 2024-04-20T22:05:38.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:38.745+0000 7f2c4a000700 1 -- 172.21.15.62:0/3556928366 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f2c4404f2a0 con 0x7f2c44075280 2024-04-20T22:05:38.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:38.746+0000 7f2c3be00700 1 -- 172.21.15.62:0/3556928366 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7f2c3405c7c0 con 0x7f2c44075280 2024-04-20T22:05:38.746 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:05:38.746 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T22:05:38.746 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:05:38.746 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:05:38.746 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T22:05:38.746 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:05:38.747 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:05:38.747 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T22:05:38.747 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 6 2024-04-20T22:05:38.747 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:05:38.747 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T22:05:38.747 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 1, 2024-04-20T22:05:38.747 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 3 2024-04-20T22:05:38.747 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:05:38.747 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T22:05:38.747 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 1, 2024-04-20T22:05:38.747 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 13 2024-04-20T22:05:38.747 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T22:05:38.747 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:05:38.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:38.749+0000 7f2c4a000700 1 -- 172.21.15.62:0/3556928366 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2c2806ecc0 msgr2=0x7f2c28071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:05:38.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:38.749+0000 7f2c4a000700 1 --2- 172.21.15.62:0/3556928366 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2c2806ecc0 0x7f2c28071170 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f2c4419da50 tx=0x7f2c30009260 comp rx=0 tx=0).stop 2024-04-20T22:05:38.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:38.749+0000 7f2c4a000700 1 -- 172.21.15.62:0/3556928366 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c44075280 msgr2=0x7f2c4419c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:05:38.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:38.749+0000 7f2c4a000700 1 --2- 172.21.15.62:0/3556928366 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c44075280 0x7f2c4419c430 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f2c34005b40 tx=0x7f2c3400b660 comp rx=0 tx=0).stop 2024-04-20T22:05:38.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:38.749+0000 7f2c4a000700 1 -- 172.21.15.62:0/3556928366 shutdown_connections 2024-04-20T22:05:38.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:38.749+0000 7f2c4a000700 1 --2- 172.21.15.62:0/3556928366 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2c2806ecc0 0x7f2c28071170 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:05:38.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:38.750+0000 7f2c4a000700 1 --2- 172.21.15.62:0/3556928366 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c44075280 0x7f2c4419c430 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:05:38.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:38.750+0000 7f2c4a000700 1 --2- 172.21.15.62:0/3556928366 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c44075b90 0x7f2c4419c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:05:38.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:38.750+0000 7f2c4a000700 1 -- 172.21.15.62:0/3556928366 >> 172.21.15.62:0/3556928366 conn(0x7f2c440fdf90 msgr2=0x7f2c440ffc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:05:38.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:38.750+0000 7f2c4a000700 1 -- 172.21.15.62:0/3556928366 shutdown_connections 2024-04-20T22:05:38.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:38.750+0000 7f2c4a000700 1 -- 172.21.15.62:0/3556928366 wait complete. 2024-04-20T22:05:38.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:38.913+0000 7faac9000700 1 -- 172.21.15.62:0/870239418 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faac4068540 msgr2=0x7faac40689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:05:38.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:38.913+0000 7faac9000700 1 --2- 172.21.15.62:0/870239418 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faac4068540 0x7faac40689b0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7faab0009910 tx=0x7faab0009c20 comp rx=0 tx=0).stop 2024-04-20T22:05:38.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:38.913+0000 7faac9000700 1 -- 172.21.15.62:0/870239418 shutdown_connections 2024-04-20T22:05:38.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:38.913+0000 7faac9000700 1 --2- 172.21.15.62:0/870239418 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faac4068540 0x7faac40689b0 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:05:38.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:38.913+0000 7faac9000700 1 --2- 172.21.15.62:0/870239418 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faac4106320 0x7faac41066f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:05:38.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:38.913+0000 7faac9000700 1 -- 172.21.15.62:0/870239418 >> 172.21.15.62:0/870239418 conn(0x7faac4075130 msgr2=0x7faac4075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:05:38.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:38.913+0000 7faac9000700 1 -- 172.21.15.62:0/870239418 shutdown_connections 2024-04-20T22:05:38.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:38.913+0000 7faac9000700 1 -- 172.21.15.62:0/870239418 wait complete. 2024-04-20T22:05:38.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:38.914+0000 7faac9000700 1 Processor -- start 2024-04-20T22:05:38.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:38.914+0000 7faac9000700 1 -- start start 2024-04-20T22:05:38.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:38.915+0000 7faac9000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faac4068540 0x7faac4198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:05:38.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:38.915+0000 7faac9000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faac4106320 0x7faac4198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:05:38.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:38.915+0000 7faac9000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faac4198c40 con 0x7faac4106320 2024-04-20T22:05:38.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:38.915+0000 7faac9000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faac419c9d0 con 0x7faac4068540 2024-04-20T22:05:38.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:38.915+0000 7faac1400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faac4106320 0x7faac4198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:05:38.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:38.915+0000 7faac1400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faac4106320 0x7faac4198560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49278/0 (socket says 172.21.15.62:49278) 2024-04-20T22:05:38.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:38.915+0000 7faac1400700 1 -- 172.21.15.62:0/2820714529 learned_addr learned my addr 172.21.15.62:0/2820714529 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:05:38.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:38.915+0000 7faac1e00700 1 --2- 172.21.15.62:0/2820714529 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faac4068540 0x7faac4198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:05:38.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:38.916+0000 7faac1400700 1 -- 172.21.15.62:0/2820714529 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faac4068540 msgr2=0x7faac4198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:05:38.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:38.916+0000 7faac1400700 1 --2- 172.21.15.62:0/2820714529 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faac4068540 0x7faac4198020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:05:38.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:38.916+0000 7faac1400700 1 -- 172.21.15.62:0/2820714529 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faab00095f0 con 0x7faac4106320 2024-04-20T22:05:38.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:38.916+0000 7faac1400700 1 --2- 172.21.15.62:0/2820714529 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faac4106320 0x7faac4198560 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7faab000b5c0 tx=0x7faab000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:05:38.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:38.916+0000 7faabaa00700 1 -- 172.21.15.62:0/2820714529 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faab001d070 con 0x7faac4106320 2024-04-20T22:05:38.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:38.916+0000 7faabaa00700 1 -- 172.21.15.62:0/2820714529 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7faab0005e00 con 0x7faac4106320 2024-04-20T22:05:38.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:38.916+0000 7faac9000700 1 -- 172.21.15.62:0/2820714529 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faac419cc50 con 0x7faac4106320 2024-04-20T22:05:38.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:38.916+0000 7faabaa00700 1 -- 172.21.15.62:0/2820714529 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faab0017400 con 0x7faac4106320 2024-04-20T22:05:38.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:38.917+0000 7faac9000700 1 -- 172.21.15.62:0/2820714529 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faac419d140 con 0x7faac4106320 2024-04-20T22:05:38.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:38.918+0000 7faac9000700 1 -- 172.21.15.62:0/2820714529 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faac404f2a0 con 0x7faac4106320 2024-04-20T22:05:38.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:38.919+0000 7faabaa00700 1 -- 172.21.15.62:0/2820714529 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7faab000f460 con 0x7faac4106320 2024-04-20T22:05:38.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:38.920+0000 7faabaa00700 1 --2- 172.21.15.62:0/2820714529 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7faaa806ecc0 0x7faaa8071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:05:38.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:38.920+0000 7faabaa00700 1 -- 172.21.15.62:0/2820714529 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7faab0090370 con 0x7faac4106320 2024-04-20T22:05:38.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:38.920+0000 7faac1e00700 1 --2- 172.21.15.62:0/2820714529 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7faaa806ecc0 0x7faaa8071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:05:38.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:38.921+0000 7faac1e00700 1 --2- 172.21.15.62:0/2820714529 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7faaa806ecc0 0x7faaa8071170 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7faab4005950 tx=0x7faab40058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:05:38.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:38.925+0000 7faabaa00700 1 -- 172.21.15.62:0/2820714529 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7faab0059100 con 0x7faac4106320 2024-04-20T22:05:39.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:39.200+0000 7faac9000700 1 -- 172.21.15.62:0/2820714529 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7faac4199380 con 0x7faac4106320 2024-04-20T22:05:39.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:39.201+0000 7faabaa00700 1 -- 172.21.15.62:0/2820714529 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 27 v27) v1 ==== 76+0+1888 (secure 0 0 0) 0x7faab005c7c0 con 0x7faac4106320 2024-04-20T22:05:39.201 INFO:teuthology.orchestra.run.smithi062.stdout:e27 2024-04-20T22:05:39.202 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T22:05:39.202 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:05:39.202 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T22:05:39.202 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:05:39.202 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T22:05:39.202 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T22:05:39.202 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 20 2024-04-20T22:05:39.202 INFO:teuthology.orchestra.run.smithi062.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-20T22:05:39.202 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T22:05:39.202 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T21:42:01.559061+0000 2024-04-20T22:05:39.202 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T22:05:39.202 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T22:05:39.202 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T22:05:39.202 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T22:05:39.203 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T22:05:39.203 INFO:teuthology.orchestra.run.smithi062.stdout:max_xattr_size 65536 2024-04-20T22:05:39.203 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T22:05:39.203 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T22:05:39.203 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 81 2024-04-20T22:05:39.203 INFO:teuthology.orchestra.run.smithi062.stdout:snap_rstat 1 2024-04-20T22:05:39.203 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:05:39.203 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 2 2024-04-20T22:05:39.203 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1 2024-04-20T22:05:39.203 INFO:teuthology.orchestra.run.smithi062.stdout:up {} 2024-04-20T22:05:39.203 INFO:teuthology.orchestra.run.smithi062.stdout:failed 0,1 2024-04-20T22:05:39.203 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T22:05:39.203 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T22:05:39.203 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T22:05:39.203 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T22:05:39.204 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data enabled 2024-04-20T22:05:39.204 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T22:05:39.204 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T22:05:39.204 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 1 2024-04-20T22:05:39.204 INFO:teuthology.orchestra.run.smithi062.stdout:qdb_cluster leader: 0 members: 2024-04-20T22:05:39.204 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:05:39.204 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:05:39.204 INFO:teuthology.orchestra.run.smithi062.stdout:Standby daemons: 2024-04-20T22:05:39.204 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:05:39.204 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ryqblb{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6826/4113916205,v1:172.21.15.62:6827/4113916205] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:05:39.204 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ozizwi{-1:34476} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6828/3801975165,v1:172.21.15.62:6829/3801975165] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:05:39.204 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.xbgspw{-1:44285} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6826/3996040526,v1:172.21.15.189:6827/3996040526] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T22:05:39.204 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.dvimer{-1:44727} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6824/3576699232,v1:172.21.15.189:6825/3576699232] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:05:39.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:39.204+0000 7faac9000700 1 -- 172.21.15.62:0/2820714529 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7faaa806ecc0 msgr2=0x7faaa8071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:05:39.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:39.204+0000 7faac9000700 1 --2- 172.21.15.62:0/2820714529 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7faaa806ecc0 0x7faaa8071170 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7faab4005950 tx=0x7faab40058e0 comp rx=0 tx=0).stop 2024-04-20T22:05:39.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:39.204+0000 7faac9000700 1 -- 172.21.15.62:0/2820714529 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faac4106320 msgr2=0x7faac4198560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:05:39.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:39.204+0000 7faac9000700 1 --2- 172.21.15.62:0/2820714529 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faac4106320 0x7faac4198560 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7faab000b5c0 tx=0x7faab000be20 comp rx=0 tx=0).stop 2024-04-20T22:05:39.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:39.204+0000 7faac9000700 1 -- 172.21.15.62:0/2820714529 shutdown_connections 2024-04-20T22:05:39.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:39.204+0000 7faac9000700 1 --2- 172.21.15.62:0/2820714529 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7faaa806ecc0 0x7faaa8071170 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:05:39.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:39.204+0000 7faac9000700 1 --2- 172.21.15.62:0/2820714529 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faac4068540 0x7faac4198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:05:39.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:39.204+0000 7faac9000700 1 --2- 172.21.15.62:0/2820714529 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faac4106320 0x7faac4198560 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:05:39.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:39.204+0000 7faac9000700 1 -- 172.21.15.62:0/2820714529 >> 172.21.15.62:0/2820714529 conn(0x7faac4075130 msgr2=0x7faac40ffff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:05:39.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:39.204+0000 7faac9000700 1 -- 172.21.15.62:0/2820714529 shutdown_connections 2024-04-20T22:05:39.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:39.205+0000 7faac9000700 1 -- 172.21.15.62:0/2820714529 wait complete. 2024-04-20T22:05:39.206 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 27 2024-04-20T22:05:39.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:39.366+0000 7f3811200700 1 -- 172.21.15.62:0/4067937778 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f380c073a00 msgr2=0x7f380c110d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:05:39.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:39.366+0000 7f3811200700 1 --2- 172.21.15.62:0/4067937778 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f380c073a00 0x7f380c110d50 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f37f8009910 tx=0x7f37f8009c20 comp rx=0 tx=0).stop 2024-04-20T22:05:39.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:39.367+0000 7f3811200700 1 -- 172.21.15.62:0/4067937778 shutdown_connections 2024-04-20T22:05:39.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:39.367+0000 7f3811200700 1 --2- 172.21.15.62:0/4067937778 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f380c073a00 0x7f380c110d50 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:05:39.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:39.367+0000 7f3811200700 1 --2- 172.21.15.62:0/4067937778 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f380c0730f0 0x7f380c0734c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:05:39.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:39.367+0000 7f3811200700 1 -- 172.21.15.62:0/4067937778 >> 172.21.15.62:0/4067937778 conn(0x7f380c0fbda0 msgr2=0x7f380c0fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:05:39.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:39.367+0000 7f3811200700 1 -- 172.21.15.62:0/4067937778 shutdown_connections 2024-04-20T22:05:39.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:39.367+0000 7f3811200700 1 -- 172.21.15.62:0/4067937778 wait complete. 2024-04-20T22:05:39.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:39.368+0000 7f3811200700 1 Processor -- start 2024-04-20T22:05:39.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:39.368+0000 7f3811200700 1 -- start start 2024-04-20T22:05:39.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:39.368+0000 7f3811200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f380c0730f0 0x7f380c19c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:05:39.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:39.368+0000 7f3811200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f380c073a00 0x7f380c19c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:05:39.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:39.368+0000 7f3811200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f380c19d050 con 0x7f380c073a00 2024-04-20T22:05:39.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:39.368+0000 7f3811200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f380c1a0de0 con 0x7f380c0730f0 2024-04-20T22:05:39.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:39.369+0000 7f3809400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f380c073a00 0x7f380c19c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:05:39.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:39.369+0000 7f3809400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f380c073a00 0x7f380c19c970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49300/0 (socket says 172.21.15.62:49300) 2024-04-20T22:05:39.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:39.369+0000 7f3809400700 1 -- 172.21.15.62:0/843137009 learned_addr learned my addr 172.21.15.62:0/843137009 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:05:39.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:39.369+0000 7f3809e00700 1 --2- 172.21.15.62:0/843137009 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f380c0730f0 0x7f380c19c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:05:39.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:39.369+0000 7f3809400700 1 -- 172.21.15.62:0/843137009 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f380c0730f0 msgr2=0x7f380c19c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:05:39.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:39.369+0000 7f3809400700 1 --2- 172.21.15.62:0/843137009 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f380c0730f0 0x7f380c19c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:05:39.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:39.369+0000 7f3809400700 1 -- 172.21.15.62:0/843137009 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37f80095f0 con 0x7f380c073a00 2024-04-20T22:05:39.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:39.370+0000 7f3809400700 1 --2- 172.21.15.62:0/843137009 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f380c073a00 0x7f380c19c970 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f37f80092e0 tx=0x7f37f800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:05:39.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:39.370+0000 7f3802a00700 1 -- 172.21.15.62:0/843137009 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f37f801d070 con 0x7f380c073a00 2024-04-20T22:05:39.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:39.370+0000 7f3802a00700 1 -- 172.21.15.62:0/843137009 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f37f8005e00 con 0x7f380c073a00 2024-04-20T22:05:39.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:39.370+0000 7f3811200700 1 -- 172.21.15.62:0/843137009 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f380c1a10c0 con 0x7f380c073a00 2024-04-20T22:05:39.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:39.370+0000 7f3802a00700 1 -- 172.21.15.62:0/843137009 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f37f8017400 con 0x7f380c073a00 2024-04-20T22:05:39.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:39.370+0000 7f3811200700 1 -- 172.21.15.62:0/843137009 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f380c1a1610 con 0x7f380c073a00 2024-04-20T22:05:39.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:39.371+0000 7f3811200700 1 -- 172.21.15.62:0/843137009 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f380c10e4d0 con 0x7f380c073a00 2024-04-20T22:05:39.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:39.372+0000 7f3802a00700 1 -- 172.21.15.62:0/843137009 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f37f800f460 con 0x7f380c073a00 2024-04-20T22:05:39.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:39.373+0000 7f3802a00700 1 --2- 172.21.15.62:0/843137009 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f37f006ecc0 0x7f37f0071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:05:39.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:39.373+0000 7f3802a00700 1 -- 172.21.15.62:0/843137009 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f37f8090330 con 0x7f380c073a00 2024-04-20T22:05:39.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:39.374+0000 7f3809e00700 1 --2- 172.21.15.62:0/843137009 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f37f006ecc0 0x7f37f0071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:05:39.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:39.374+0000 7f3809e00700 1 --2- 172.21.15.62:0/843137009 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f37f006ecc0 0x7f37f0071170 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f37fc005950 tx=0x7f37fc0058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:05:39.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:39.378+0000 7f3802a00700 1 -- 172.21.15.62:0/843137009 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f37f80590e0 con 0x7f380c073a00 2024-04-20T22:05:39.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:05:39 smithi189 ceph-mon[57022]: from='client.35628 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:05:39.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:05:39 smithi189 ceph-mon[57022]: pgmap v911: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:05:39.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:05:39 smithi189 ceph-mon[57022]: from='client.35632 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:05:39.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:05:39 smithi189 ceph-mon[57022]: from='client.35636 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:05:39.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:05:39 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/3556928366' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:05:39.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:05:39 smithi062 ceph-mon[76322]: from='client.35628 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:05:39.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:05:39 smithi062 ceph-mon[76322]: pgmap v911: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:05:39.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:05:39 smithi062 ceph-mon[76322]: from='client.35632 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:05:39.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:05:39 smithi062 ceph-mon[76322]: from='client.35636 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:05:39.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:05:39 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/3556928366' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:05:39.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:39.645+0000 7f3811200700 1 -- 172.21.15.62:0/843137009 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f380c1a18a0 con 0x7f37f006ecc0 2024-04-20T22:05:39.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:39.648+0000 7f3802a00700 1 -- 172.21.15.62:0/843137009 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f380c1a18a0 con 0x7f37f006ecc0 2024-04-20T22:05:39.648 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:05:39.648 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T22:05:39.648 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T22:05:39.648 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-20T22:05:39.648 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T22:05:39.648 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr", 2024-04-20T22:05:39.648 INFO:teuthology.orchestra.run.smithi062.stdout: "crash", 2024-04-20T22:05:39.648 INFO:teuthology.orchestra.run.smithi062.stdout: "mon", 2024-04-20T22:05:39.648 INFO:teuthology.orchestra.run.smithi062.stdout: "osd" 2024-04-20T22:05:39.648 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T22:05:39.649 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "15/23 daemons upgraded", 2024-04-20T22:05:39.649 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading mds daemons", 2024-04-20T22:05:39.649 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T22:05:39.649 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:05:39.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:39.651+0000 7f3811200700 1 -- 172.21.15.62:0/843137009 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f37f006ecc0 msgr2=0x7f37f0071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:05:39.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:39.651+0000 7f3811200700 1 --2- 172.21.15.62:0/843137009 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f37f006ecc0 0x7f37f0071170 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f37fc005950 tx=0x7f37fc0058e0 comp rx=0 tx=0).stop 2024-04-20T22:05:39.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:39.651+0000 7f3811200700 1 -- 172.21.15.62:0/843137009 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f380c073a00 msgr2=0x7f380c19c970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:05:39.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:39.651+0000 7f3811200700 1 --2- 172.21.15.62:0/843137009 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f380c073a00 0x7f380c19c970 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f37f80092e0 tx=0x7f37f800be20 comp rx=0 tx=0).stop 2024-04-20T22:05:39.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:39.651+0000 7f3811200700 1 -- 172.21.15.62:0/843137009 shutdown_connections 2024-04-20T22:05:39.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:39.651+0000 7f3811200700 1 --2- 172.21.15.62:0/843137009 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f37f006ecc0 0x7f37f0071170 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:05:39.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:39.651+0000 7f3811200700 1 --2- 172.21.15.62:0/843137009 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f380c0730f0 0x7f380c19c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:05:39.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:39.651+0000 7f3811200700 1 --2- 172.21.15.62:0/843137009 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f380c073a00 0x7f380c19c970 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:05:39.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:39.651+0000 7f3811200700 1 -- 172.21.15.62:0/843137009 >> 172.21.15.62:0/843137009 conn(0x7f380c0fbda0 msgr2=0x7f380c102860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:05:39.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:39.651+0000 7f3811200700 1 -- 172.21.15.62:0/843137009 shutdown_connections 2024-04-20T22:05:39.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:39.652+0000 7f3811200700 1 -- 172.21.15.62:0/843137009 wait complete. 2024-04-20T22:05:39.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:39.819+0000 7f12b8000700 1 -- 172.21.15.62:0/500965943 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12b0108470 msgr2=0x7f12b0108840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:05:39.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:39.819+0000 7f12b8000700 1 --2- 172.21.15.62:0/500965943 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12b0108470 0x7f12b0108840 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f12a4009910 tx=0x7f12a4009c20 comp rx=0 tx=0).stop 2024-04-20T22:05:39.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:39.819+0000 7f12b8000700 1 -- 172.21.15.62:0/500965943 shutdown_connections 2024-04-20T22:05:39.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:39.819+0000 7f12b8000700 1 --2- 172.21.15.62:0/500965943 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12b0102470 0x7f12b01028e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:05:39.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:39.820+0000 7f12b8000700 1 --2- 172.21.15.62:0/500965943 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12b0108470 0x7f12b0108840 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:05:39.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:39.820+0000 7f12b8000700 1 -- 172.21.15.62:0/500965943 >> 172.21.15.62:0/500965943 conn(0x7f12b00fdf90 msgr2=0x7f12b01003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:05:39.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:39.820+0000 7f12b8000700 1 -- 172.21.15.62:0/500965943 shutdown_connections 2024-04-20T22:05:39.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:39.820+0000 7f12b8000700 1 -- 172.21.15.62:0/500965943 wait complete. 2024-04-20T22:05:39.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:39.821+0000 7f12b8000700 1 Processor -- start 2024-04-20T22:05:39.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:39.822+0000 7f12b8000700 1 -- start start 2024-04-20T22:05:39.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:39.822+0000 7f12b8000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12b0102470 0x7f12b01981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:05:39.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:39.822+0000 7f12b8000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12b0108470 0x7f12b0198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:05:39.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:39.822+0000 7f12b8000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12b0198e10 con 0x7f12b0102470 2024-04-20T22:05:39.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:39.822+0000 7f12b8000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12b019cba0 con 0x7f12b0108470 2024-04-20T22:05:39.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:39.822+0000 7f12b5600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12b0102470 0x7f12b01981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:05:39.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:39.822+0000 7f12b5600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12b0102470 0x7f12b01981f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49320/0 (socket says 172.21.15.62:49320) 2024-04-20T22:05:39.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:39.822+0000 7f12b5600700 1 -- 172.21.15.62:0/2032345171 learned_addr learned my addr 172.21.15.62:0/2032345171 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:05:39.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:39.823+0000 7f12b5600700 1 -- 172.21.15.62:0/2032345171 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12b0108470 msgr2=0x7f12b0198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:05:39.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:39.823+0000 7f12b5600700 1 --2- 172.21.15.62:0/2032345171 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12b0108470 0x7f12b0198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:05:39.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:39.823+0000 7f12b5600700 1 -- 172.21.15.62:0/2032345171 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12a40095f0 con 0x7f12b0102470 2024-04-20T22:05:39.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:39.823+0000 7f12b5600700 1 --2- 172.21.15.62:0/2032345171 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12b0102470 0x7f12b01981f0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f12a400b5c0 tx=0x7f12a400be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:05:39.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:39.823+0000 7f12ae000700 1 -- 172.21.15.62:0/2032345171 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f12a401d070 con 0x7f12b0102470 2024-04-20T22:05:39.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:39.823+0000 7f12ae000700 1 -- 172.21.15.62:0/2032345171 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f12a400f460 con 0x7f12b0102470 2024-04-20T22:05:39.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:39.824+0000 7f12ae000700 1 -- 172.21.15.62:0/2032345171 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f12a4017610 con 0x7f12b0102470 2024-04-20T22:05:39.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:39.824+0000 7f12b8000700 1 -- 172.21.15.62:0/2032345171 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12b019ce20 con 0x7f12b0102470 2024-04-20T22:05:39.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:39.824+0000 7f12b8000700 1 -- 172.21.15.62:0/2032345171 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12b019d310 con 0x7f12b0102470 2024-04-20T22:05:39.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:39.825+0000 7f12b8000700 1 -- 172.21.15.62:0/2032345171 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12b010a9e0 con 0x7f12b0102470 2024-04-20T22:05:39.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:39.826+0000 7f12ae000700 1 -- 172.21.15.62:0/2032345171 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f12a400f5d0 con 0x7f12b0102470 2024-04-20T22:05:39.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:39.827+0000 7f12ae000700 1 --2- 172.21.15.62:0/2032345171 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f129806ecc0 0x7f1298071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:05:39.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:39.827+0000 7f12ae000700 1 -- 172.21.15.62:0/2032345171 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f12a4090440 con 0x7f12b0102470 2024-04-20T22:05:39.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:39.827+0000 7f12b4c00700 1 --2- 172.21.15.62:0/2032345171 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f129806ecc0 0x7f1298071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:05:39.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:39.828+0000 7f12b4c00700 1 --2- 172.21.15.62:0/2032345171 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f129806ecc0 0x7f1298071170 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f12b0199810 tx=0x7f129c005c80 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:05:39.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:39.833+0000 7f12ae000700 1 -- 172.21.15.62:0/2032345171 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f12a40591d0 con 0x7f12b0102470 2024-04-20T22:05:40.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:40.169+0000 7f12b8000700 1 -- 172.21.15.62:0/2032345171 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f12b0199580 con 0x7f12b0102470 2024-04-20T22:05:40.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:40.170+0000 7f12ae000700 1 -- 172.21.15.62:0/2032345171 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7f12a405c890 con 0x7f12b0102470 2024-04-20T22:05:40.170 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-20T22:05:40.170 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-20T22:05:40.171 INFO:teuthology.orchestra.run.smithi062.stdout: host smithi062 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T22:05:40.171 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-20T22:05:40.171 INFO:teuthology.orchestra.run.smithi062.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi062 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpq7y7oa_r:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:05:40.171 INFO:teuthology.orchestra.run.smithi062.stdout:Traceback (most recent call last): 2024-04-20T22:05:40.171 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-20T22:05:40.171 INFO:teuthology.orchestra.run.smithi062.stdout: return _run_code(code, main_globals, None, 2024-04-20T22:05:40.171 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-20T22:05:40.171 INFO:teuthology.orchestra.run.smithi062.stdout: exec(code, run_globals) 2024-04-20T22:05:40.171 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5618, in 2024-04-20T22:05:40.171 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5606, in main 2024-04-20T22:05:40.171 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 408, in _infer_config 2024-04-20T22:05:40.171 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 323, in _infer_fsid 2024-04-20T22:05:40.171 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 436, in _infer_image 2024-04-20T22:05:40.171 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 310, in _validate_fsid 2024-04-20T22:05:40.172 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 3359, in command_ceph_volume 2024-04-20T22:05:40.172 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-20T22:05:40.172 INFO:teuthology.orchestra.run.smithi062.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi062 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpq7y7oa_r:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:05:40.172 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-20T22:05:40.172 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is degraded 2024-04-20T22:05:40.172 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T22:05:40.172 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T22:05:40.172 INFO:teuthology.orchestra.run.smithi062.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-20T22:05:40.172 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-20T22:05:40.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:40.174+0000 7f12b8000700 1 -- 172.21.15.62:0/2032345171 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f129806ecc0 msgr2=0x7f1298071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:05:40.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:40.174+0000 7f12b8000700 1 --2- 172.21.15.62:0/2032345171 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f129806ecc0 0x7f1298071170 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f12b0199810 tx=0x7f129c005c80 comp rx=0 tx=0).stop 2024-04-20T22:05:40.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:40.174+0000 7f12b8000700 1 -- 172.21.15.62:0/2032345171 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12b0102470 msgr2=0x7f12b01981f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:05:40.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:40.174+0000 7f12b8000700 1 --2- 172.21.15.62:0/2032345171 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12b0102470 0x7f12b01981f0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f12a400b5c0 tx=0x7f12a400be00 comp rx=0 tx=0).stop 2024-04-20T22:05:40.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:40.174+0000 7f12b8000700 1 -- 172.21.15.62:0/2032345171 shutdown_connections 2024-04-20T22:05:40.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:40.174+0000 7f12b8000700 1 --2- 172.21.15.62:0/2032345171 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f129806ecc0 0x7f1298071170 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:05:40.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:40.174+0000 7f12b8000700 1 --2- 172.21.15.62:0/2032345171 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12b0102470 0x7f12b01981f0 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:05:40.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:40.174+0000 7f12b8000700 1 --2- 172.21.15.62:0/2032345171 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12b0108470 0x7f12b0198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:05:40.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:40.174+0000 7f12b8000700 1 -- 172.21.15.62:0/2032345171 >> 172.21.15.62:0/2032345171 conn(0x7f12b00fdf90 msgr2=0x7f12b01002c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:05:40.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:40.174+0000 7f12b8000700 1 -- 172.21.15.62:0/2032345171 shutdown_connections 2024-04-20T22:05:40.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:05:40.174+0000 7f12b8000700 1 -- 172.21.15.62:0/2032345171 wait complete. 2024-04-20T22:05:40.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:05:40 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/2820714529' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:05:40.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:05:40 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/2032345171' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:05:40.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:05:40 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/2820714529' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:05:40.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:05:40 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/2032345171' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:05:41.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:05:41 smithi189 ceph-mon[57022]: pgmap v912: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:05:41.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:05:41 smithi189 ceph-mon[57022]: from='client.35648 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:05:41.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:05:41 smithi062 ceph-mon[76322]: pgmap v912: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:05:41.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:05:41 smithi062 ceph-mon[76322]: from='client.35648 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:05:43.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:05:43 smithi189 ceph-mon[57022]: pgmap v913: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:05:43.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:05:43 smithi062 ceph-mon[76322]: pgmap v913: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:05:45.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:05:45 smithi189 ceph-mon[57022]: pgmap v914: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:05:45.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:05:45 smithi062 ceph-mon[76322]: pgmap v914: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:05:47.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:05:47 smithi189 ceph-mon[57022]: pgmap v915: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:05:47.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:05:47 smithi062 ceph-mon[76322]: pgmap v915: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:05:49.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:05:49 smithi189 ceph-mon[57022]: pgmap v916: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:05:49.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:05:49 smithi062 ceph-mon[76322]: pgmap v916: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:05:50.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:05:50 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:05:50.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:05:50 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:05:51.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:05:51 smithi062 ceph-mon[76322]: pgmap v917: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:05:51.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:05:51 smithi189 ceph-mon[57022]: pgmap v917: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:05:53.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:05:53 smithi062 ceph-mon[76322]: pgmap v918: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:05:53.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:05:53 smithi189 ceph-mon[57022]: pgmap v918: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:05:54.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:05:54 smithi189 ceph-mon[57022]: pgmap v919: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:05:54.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:05:54 smithi062 ceph-mon[76322]: pgmap v919: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:05:56.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:05:56 smithi189 ceph-mon[57022]: pgmap v920: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:05:57.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:05:56 smithi062 ceph-mon[76322]: pgmap v920: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:05:58.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:05:58 smithi189 ceph-mon[57022]: pgmap v921: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:05:59.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:05:58 smithi062 ceph-mon[76322]: pgmap v921: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:06:00.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:06:00 smithi189 ceph-mon[57022]: pgmap v922: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:06:01.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:06:00 smithi062 ceph-mon[76322]: pgmap v922: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:06:02.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:06:02 smithi189 ceph-mon[57022]: pgmap v923: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:06:03.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:06:02 smithi062 ceph-mon[76322]: pgmap v923: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:06:04.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:06:04 smithi189 ceph-mon[57022]: pgmap v924: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:06:04.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:06:04 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:06:05.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:06:04 smithi062 ceph-mon[76322]: pgmap v924: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:06:05.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:06:04 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:06:06.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:06:06 smithi189 ceph-mon[57022]: pgmap v925: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:06:07.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:06:06 smithi062 ceph-mon[76322]: pgmap v925: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:06:08.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:06:08 smithi189 ceph-mon[57022]: pgmap v926: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:06:09.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:06:08 smithi062 ceph-mon[76322]: pgmap v926: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:06:10.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:10.356+0000 7ff115e00700 1 -- 172.21.15.62:0/2184782141 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff110102e30 msgr2=0x7ff110103200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:06:10.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:10.356+0000 7ff115e00700 1 --2- 172.21.15.62:0/2184782141 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff110102e30 0x7ff110103200 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7ff0f4009910 tx=0x7ff0f4009c20 comp rx=0 tx=0).stop 2024-04-20T22:06:10.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:10.356+0000 7ff115e00700 1 -- 172.21.15.62:0/2184782141 shutdown_connections 2024-04-20T22:06:10.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:10.356+0000 7ff115e00700 1 --2- 172.21.15.62:0/2184782141 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1100ff9e0 0x7ff1100ffe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:06:10.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:10.356+0000 7ff115e00700 1 --2- 172.21.15.62:0/2184782141 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff110102e30 0x7ff110103200 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:06:10.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:10.356+0000 7ff115e00700 1 -- 172.21.15.62:0/2184782141 >> 172.21.15.62:0/2184782141 conn(0x7ff110074fa0 msgr2=0x7ff1100753a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:06:10.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:10.356+0000 7ff115e00700 1 -- 172.21.15.62:0/2184782141 shutdown_connections 2024-04-20T22:06:10.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:10.357+0000 7ff115e00700 1 -- 172.21.15.62:0/2184782141 wait complete. 2024-04-20T22:06:10.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:10.358+0000 7ff115e00700 1 Processor -- start 2024-04-20T22:06:10.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:10.358+0000 7ff115e00700 1 -- start start 2024-04-20T22:06:10.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:10.358+0000 7ff115e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1100ff9e0 0x7ff110198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:06:10.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:10.358+0000 7ff115e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff110198540 0x7ff11019c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:06:10.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:10.358+0000 7ff115e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff110198b50 con 0x7ff110198540 2024-04-20T22:06:10.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:10.358+0000 7ff115e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff110198cc0 con 0x7ff1100ff9e0 2024-04-20T22:06:10.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:10.359+0000 7ff107e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff110198540 0x7ff11019c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:06:10.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:10.359+0000 7ff107e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff110198540 0x7ff11019c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:57356/0 (socket says 172.21.15.62:57356) 2024-04-20T22:06:10.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:10.359+0000 7ff107e00700 1 -- 172.21.15.62:0/2468045085 learned_addr learned my addr 172.21.15.62:0/2468045085 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:06:10.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:10.359+0000 7ff114a00700 1 --2- 172.21.15.62:0/2468045085 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1100ff9e0 0x7ff110198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:06:10.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:10.359+0000 7ff107e00700 1 -- 172.21.15.62:0/2468045085 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1100ff9e0 msgr2=0x7ff110198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:06:10.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:10.359+0000 7ff107e00700 1 --2- 172.21.15.62:0/2468045085 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1100ff9e0 0x7ff110198000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:06:10.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:10.359+0000 7ff107e00700 1 -- 172.21.15.62:0/2468045085 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0f40095f0 con 0x7ff110198540 2024-04-20T22:06:10.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:10.359+0000 7ff107e00700 1 --2- 172.21.15.62:0/2468045085 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff110198540 0x7ff11019c9b0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7ff0fc00d930 tx=0x7ff0fc00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:06:10.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:10.360+0000 7ff10e000700 1 -- 172.21.15.62:0/2468045085 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff0fc004a00 con 0x7ff110198540 2024-04-20T22:06:10.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:10.360+0000 7ff10e000700 1 -- 172.21.15.62:0/2468045085 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff0fc0054e0 con 0x7ff110198540 2024-04-20T22:06:10.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:10.360+0000 7ff115e00700 1 -- 172.21.15.62:0/2468045085 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff11019cfb0 con 0x7ff110198540 2024-04-20T22:06:10.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:10.360+0000 7ff10e000700 1 -- 172.21.15.62:0/2468045085 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff0fc009d00 con 0x7ff110198540 2024-04-20T22:06:10.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:10.360+0000 7ff115e00700 1 -- 172.21.15.62:0/2468045085 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff11019d580 con 0x7ff110198540 2024-04-20T22:06:10.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:10.361+0000 7ff115e00700 1 -- 172.21.15.62:0/2468045085 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff11019d800 con 0x7ff110198540 2024-04-20T22:06:10.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:10.362+0000 7ff10e000700 1 -- 172.21.15.62:0/2468045085 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff0fc005020 con 0x7ff110198540 2024-04-20T22:06:10.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:10.364+0000 7ff10e000700 1 --2- 172.21.15.62:0/2468045085 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff0ec06ecc0 0x7ff0ec071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:06:10.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:10.364+0000 7ff10e000700 1 -- 172.21.15.62:0/2468045085 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7ff0fc08eee0 con 0x7ff110198540 2024-04-20T22:06:10.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:10.364+0000 7ff114a00700 1 --2- 172.21.15.62:0/2468045085 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff0ec06ecc0 0x7ff0ec071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:06:10.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:10.365+0000 7ff114a00700 1 --2- 172.21.15.62:0/2468045085 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff0ec06ecc0 0x7ff0ec071170 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7ff0f400b5c0 tx=0x7ff0f4005fd0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:06:10.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:10.369+0000 7ff10e000700 1 -- 172.21.15.62:0/2468045085 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7ff0fc057bf0 con 0x7ff110198540 2024-04-20T22:06:10.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:10.624+0000 7ff115e00700 1 -- 172.21.15.62:0/2468045085 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff110100e90 con 0x7ff0ec06ecc0 2024-04-20T22:06:10.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:10.627+0000 7ff10e000700 1 -- 172.21.15.62:0/2468045085 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7ff110100e90 con 0x7ff0ec06ecc0 2024-04-20T22:06:10.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:10.630+0000 7ff115e00700 1 -- 172.21.15.62:0/2468045085 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff0ec06ecc0 msgr2=0x7ff0ec071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:06:10.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:10.630+0000 7ff115e00700 1 --2- 172.21.15.62:0/2468045085 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff0ec06ecc0 0x7ff0ec071170 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7ff0f400b5c0 tx=0x7ff0f4005fd0 comp rx=0 tx=0).stop 2024-04-20T22:06:10.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:10.630+0000 7ff115e00700 1 -- 172.21.15.62:0/2468045085 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff110198540 msgr2=0x7ff11019c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:06:10.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:10.630+0000 7ff115e00700 1 --2- 172.21.15.62:0/2468045085 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff110198540 0x7ff11019c9b0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7ff0fc00d930 tx=0x7ff0fc00dcf0 comp rx=0 tx=0).stop 2024-04-20T22:06:10.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:10.630+0000 7ff115e00700 1 -- 172.21.15.62:0/2468045085 shutdown_connections 2024-04-20T22:06:10.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:10.630+0000 7ff115e00700 1 --2- 172.21.15.62:0/2468045085 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff0ec06ecc0 0x7ff0ec071170 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:06:10.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:10.630+0000 7ff115e00700 1 --2- 172.21.15.62:0/2468045085 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1100ff9e0 0x7ff110198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:06:10.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:10.630+0000 7ff115e00700 1 --2- 172.21.15.62:0/2468045085 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff110198540 0x7ff11019c9b0 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:06:10.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:10.630+0000 7ff115e00700 1 -- 172.21.15.62:0/2468045085 >> 172.21.15.62:0/2468045085 conn(0x7ff110074fa0 msgr2=0x7ff110068640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:06:10.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:10.630+0000 7ff115e00700 1 -- 172.21.15.62:0/2468045085 shutdown_connections 2024-04-20T22:06:10.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:10.631+0000 7ff115e00700 1 -- 172.21.15.62:0/2468045085 wait complete. 2024-04-20T22:06:10.646 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T22:06:10.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:10.793+0000 7fdd9b000700 1 -- 172.21.15.62:0/1080071291 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd9410c8f0 msgr2=0x7fdd9410ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:06:10.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:10.793+0000 7fdd9b000700 1 --2- 172.21.15.62:0/1080071291 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd9410c8f0 0x7fdd9410ccc0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fdd84009910 tx=0x7fdd84009c20 comp rx=0 tx=0).stop 2024-04-20T22:06:10.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:10.794+0000 7fdd9b000700 1 -- 172.21.15.62:0/1080071291 shutdown_connections 2024-04-20T22:06:10.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:10.794+0000 7fdd9b000700 1 --2- 172.21.15.62:0/1080071291 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd94106b80 0x7fdd94106ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:06:10.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:10.794+0000 7fdd9b000700 1 --2- 172.21.15.62:0/1080071291 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd9410c8f0 0x7fdd9410ccc0 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:06:10.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:10.794+0000 7fdd9b000700 1 -- 172.21.15.62:0/1080071291 >> 172.21.15.62:0/1080071291 conn(0x7fdd94076060 msgr2=0x7fdd94078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:06:10.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:10.794+0000 7fdd9b000700 1 -- 172.21.15.62:0/1080071291 shutdown_connections 2024-04-20T22:06:10.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:10.794+0000 7fdd9b000700 1 -- 172.21.15.62:0/1080071291 wait complete. 2024-04-20T22:06:10.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:10.795+0000 7fdd9b000700 1 Processor -- start 2024-04-20T22:06:10.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:10.795+0000 7fdd9b000700 1 -- start start 2024-04-20T22:06:10.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:10.795+0000 7fdd9b000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd94106b80 0x7fdd9419c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:06:10.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:10.795+0000 7fdd9b000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd9410c8f0 0x7fdd9419ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:06:10.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:10.795+0000 7fdd9b000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd9419d100 con 0x7fdd94106b80 2024-04-20T22:06:10.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:10.795+0000 7fdd9b000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd941a0e90 con 0x7fdd9410c8f0 2024-04-20T22:06:10.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:10.796+0000 7fdd93e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd94106b80 0x7fdd9419c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:06:10.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:10.796+0000 7fdd93e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd94106b80 0x7fdd9419c4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:57372/0 (socket says 172.21.15.62:57372) 2024-04-20T22:06:10.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:10.796+0000 7fdd93e00700 1 -- 172.21.15.62:0/2162067870 learned_addr learned my addr 172.21.15.62:0/2162067870 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:06:10.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:10.796+0000 7fdd93400700 1 --2- 172.21.15.62:0/2162067870 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd9410c8f0 0x7fdd9419ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:06:10.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:10.796+0000 7fdd93e00700 1 -- 172.21.15.62:0/2162067870 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd9410c8f0 msgr2=0x7fdd9419ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:06:10.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:10.796+0000 7fdd93e00700 1 --2- 172.21.15.62:0/2162067870 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd9410c8f0 0x7fdd9419ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:06:10.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:10.796+0000 7fdd93e00700 1 -- 172.21.15.62:0/2162067870 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd840095f0 con 0x7fdd94106b80 2024-04-20T22:06:10.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:10.796+0000 7fdd93400700 1 --2- 172.21.15.62:0/2162067870 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd9410c8f0 0x7fdd9419ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T22:06:10.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:10.796+0000 7fdd93e00700 1 --2- 172.21.15.62:0/2162067870 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd94106b80 0x7fdd9419c4e0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fdd8400b5c0 tx=0x7fdd8400be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:06:10.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:10.797+0000 7fdd90c00700 1 -- 172.21.15.62:0/2162067870 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdd8401d070 con 0x7fdd94106b80 2024-04-20T22:06:10.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:10.797+0000 7fdd90c00700 1 -- 172.21.15.62:0/2162067870 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdd8400f460 con 0x7fdd94106b80 2024-04-20T22:06:10.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:10.797+0000 7fdd90c00700 1 -- 172.21.15.62:0/2162067870 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdd84017610 con 0x7fdd94106b80 2024-04-20T22:06:10.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:10.797+0000 7fdd9b000700 1 -- 172.21.15.62:0/2162067870 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd941a1110 con 0x7fdd94106b80 2024-04-20T22:06:10.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:10.797+0000 7fdd9b000700 1 -- 172.21.15.62:0/2162067870 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd941a1600 con 0x7fdd94106b80 2024-04-20T22:06:10.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:10.798+0000 7fdd9b000700 1 -- 172.21.15.62:0/2162067870 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd94109b20 con 0x7fdd94106b80 2024-04-20T22:06:10.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:10.799+0000 7fdd90c00700 1 -- 172.21.15.62:0/2162067870 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fdd8400f5d0 con 0x7fdd94106b80 2024-04-20T22:06:10.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:10.800+0000 7fdd90c00700 1 --2- 172.21.15.62:0/2162067870 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fdd7806ecc0 0x7fdd78071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:06:10.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:10.800+0000 7fdd90c00700 1 -- 172.21.15.62:0/2162067870 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fdd84090450 con 0x7fdd94106b80 2024-04-20T22:06:10.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:10.801+0000 7fdd93400700 1 --2- 172.21.15.62:0/2162067870 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fdd7806ecc0 0x7fdd78071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:06:10.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:10.801+0000 7fdd93400700 1 --2- 172.21.15.62:0/2162067870 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fdd7806ecc0 0x7fdd78071170 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fdd9419db00 tx=0x7fdd80008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:06:10.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:10.805+0000 7fdd90c00700 1 -- 172.21.15.62:0/2162067870 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fdd840591e0 con 0x7fdd94106b80 2024-04-20T22:06:10.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:06:10 smithi189 ceph-mon[57022]: pgmap v927: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:06:11.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:06:10 smithi062 ceph-mon[76322]: pgmap v927: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:06:11.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:11.063+0000 7fdd9b000700 1 -- 172.21.15.62:0/2162067870 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fdd9402cca0 con 0x7fdd7806ecc0 2024-04-20T22:06:11.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:11.065+0000 7fdd90c00700 1 -- 172.21.15.62:0/2162067870 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fdd9402cca0 con 0x7fdd7806ecc0 2024-04-20T22:06:11.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:11.067+0000 7fdd9b000700 1 -- 172.21.15.62:0/2162067870 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fdd7806ecc0 msgr2=0x7fdd78071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:06:11.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:11.067+0000 7fdd9b000700 1 --2- 172.21.15.62:0/2162067870 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fdd7806ecc0 0x7fdd78071170 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fdd9419db00 tx=0x7fdd80008040 comp rx=0 tx=0).stop 2024-04-20T22:06:11.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:11.067+0000 7fdd9b000700 1 -- 172.21.15.62:0/2162067870 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd94106b80 msgr2=0x7fdd9419c4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:06:11.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:11.068+0000 7fdd9b000700 1 --2- 172.21.15.62:0/2162067870 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd94106b80 0x7fdd9419c4e0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fdd8400b5c0 tx=0x7fdd8400be00 comp rx=0 tx=0).stop 2024-04-20T22:06:11.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:11.068+0000 7fdd9b000700 1 -- 172.21.15.62:0/2162067870 shutdown_connections 2024-04-20T22:06:11.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:11.068+0000 7fdd9b000700 1 --2- 172.21.15.62:0/2162067870 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fdd7806ecc0 0x7fdd78071170 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:06:11.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:11.068+0000 7fdd9b000700 1 --2- 172.21.15.62:0/2162067870 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd94106b80 0x7fdd9419c4e0 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:06:11.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:11.068+0000 7fdd9b000700 1 --2- 172.21.15.62:0/2162067870 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd9410c8f0 0x7fdd9419ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:06:11.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:11.068+0000 7fdd9b000700 1 -- 172.21.15.62:0/2162067870 >> 172.21.15.62:0/2162067870 conn(0x7fdd94076060 msgr2=0x7fdd94077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:06:11.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:11.068+0000 7fdd9b000700 1 -- 172.21.15.62:0/2162067870 shutdown_connections 2024-04-20T22:06:11.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:11.068+0000 7fdd9b000700 1 -- 172.21.15.62:0/2162067870 wait complete. 2024-04-20T22:06:11.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:11.216+0000 7f5461200700 1 -- 172.21.15.62:0/2062870164 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f545c106b80 msgr2=0x7f545c106ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:06:11.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:11.216+0000 7f5461200700 1 --2- 172.21.15.62:0/2062870164 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f545c106b80 0x7f545c106ff0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f5448009910 tx=0x7f5448009c20 comp rx=0 tx=0).stop 2024-04-20T22:06:11.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:11.216+0000 7f5461200700 1 -- 172.21.15.62:0/2062870164 shutdown_connections 2024-04-20T22:06:11.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:11.216+0000 7f5461200700 1 --2- 172.21.15.62:0/2062870164 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f545c106b80 0x7f545c106ff0 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:06:11.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:11.216+0000 7f5461200700 1 --2- 172.21.15.62:0/2062870164 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f545c10c8f0 0x7f545c10ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:06:11.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:11.216+0000 7f5461200700 1 -- 172.21.15.62:0/2062870164 >> 172.21.15.62:0/2062870164 conn(0x7f545c076060 msgr2=0x7f545c078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:06:11.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:11.216+0000 7f5461200700 1 -- 172.21.15.62:0/2062870164 shutdown_connections 2024-04-20T22:06:11.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:11.216+0000 7f5461200700 1 -- 172.21.15.62:0/2062870164 wait complete. 2024-04-20T22:06:11.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:11.217+0000 7f5461200700 1 Processor -- start 2024-04-20T22:06:11.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:11.218+0000 7f5461200700 1 -- start start 2024-04-20T22:06:11.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:11.218+0000 7f5461200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f545c106b80 0x7f545c19c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:06:11.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:11.218+0000 7f5461200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f545c10c8f0 0x7f545c19c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:06:11.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:11.218+0000 7f5461200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f545c19d060 con 0x7f545c106b80 2024-04-20T22:06:11.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:11.218+0000 7f5461200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f545c1a0df0 con 0x7f545c10c8f0 2024-04-20T22:06:11.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:11.218+0000 7f5459e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f545c106b80 0x7f545c19c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:06:11.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:11.218+0000 7f5459e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f545c106b80 0x7f545c19c440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:57382/0 (socket says 172.21.15.62:57382) 2024-04-20T22:06:11.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:11.218+0000 7f5459e00700 1 -- 172.21.15.62:0/724274343 learned_addr learned my addr 172.21.15.62:0/724274343 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:06:11.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:11.219+0000 7f5459e00700 1 -- 172.21.15.62:0/724274343 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f545c10c8f0 msgr2=0x7f545c19c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:06:11.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:11.219+0000 7f5459e00700 1 --2- 172.21.15.62:0/724274343 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f545c10c8f0 0x7f545c19c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:06:11.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:11.219+0000 7f5459e00700 1 -- 172.21.15.62:0/724274343 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54480095f0 con 0x7f545c106b80 2024-04-20T22:06:11.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:11.219+0000 7f5459e00700 1 --2- 172.21.15.62:0/724274343 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f545c106b80 0x7f545c19c440 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f544c00c8a0 tx=0x7f544c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:06:11.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:11.219+0000 7f5452a00700 1 -- 172.21.15.62:0/724274343 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f544c00cea0 con 0x7f545c106b80 2024-04-20T22:06:11.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:11.219+0000 7f5452a00700 1 -- 172.21.15.62:0/724274343 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f544c004830 con 0x7f545c106b80 2024-04-20T22:06:11.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:11.219+0000 7f5461200700 1 -- 172.21.15.62:0/724274343 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f545c1a10d0 con 0x7f545c106b80 2024-04-20T22:06:11.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:11.219+0000 7f5452a00700 1 -- 172.21.15.62:0/724274343 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f544c0055b0 con 0x7f545c106b80 2024-04-20T22:06:11.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:11.220+0000 7f5461200700 1 -- 172.21.15.62:0/724274343 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f545c1a1620 con 0x7f545c106b80 2024-04-20T22:06:11.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:11.221+0000 7f5461200700 1 -- 172.21.15.62:0/724274343 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f545c10ecc0 con 0x7f545c106b80 2024-04-20T22:06:11.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:11.222+0000 7f5452a00700 1 -- 172.21.15.62:0/724274343 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f544c0049a0 con 0x7f545c106b80 2024-04-20T22:06:11.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:11.223+0000 7f5452a00700 1 --2- 172.21.15.62:0/724274343 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f544006ecc0 0x7f5440071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:06:11.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:11.223+0000 7f5452a00700 1 -- 172.21.15.62:0/724274343 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f544c08eaf0 con 0x7f545c106b80 2024-04-20T22:06:11.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:11.223+0000 7f5459400700 1 --2- 172.21.15.62:0/724274343 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f544006ecc0 0x7f5440071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:06:11.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:11.224+0000 7f5459400700 1 --2- 172.21.15.62:0/724274343 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f544006ecc0 0x7f5440071170 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f544800b5c0 tx=0x7f5448005960 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:06:11.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:11.227+0000 7f5452a00700 1 -- 172.21.15.62:0/724274343 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f544c057880 con 0x7f545c106b80 2024-04-20T22:06:11.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:11.465+0000 7f5461200700 1 -- 172.21.15.62:0/724274343 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f545c1a18f0 con 0x7f544006ecc0 2024-04-20T22:06:11.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:11.498+0000 7f5452a00700 1 -- 172.21.15.62:0/724274343 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f545c1a18f0 con 0x7f544006ecc0 2024-04-20T22:06:11.498 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T22:06:11.499 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (32m) 8m ago 45m 15.3M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T22:06:11.499 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (46m) 8m ago 46m 10.4M - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T22:06:11.499 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (44m) 8m ago 44m 10.1M - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T22:06:11.499 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (28m) 8m ago 46m 6660k - 19.0.0-3133-ga8c84f59 76bc6b91b726 2825d5ba11c6 2024-04-20T22:06:11.499 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (27m) 8m ago 44m 6665k - 19.0.0-3133-ga8c84f59 76bc6b91b726 20f746fa1bba 2024-04-20T22:06:11.499 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (31m) 8m ago 45m 73.5M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T22:06:11.499 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (23m) 8m ago 38m 18.5M - 19.0.0-3133-ga8c84f59 76bc6b91b726 de80bc90a683 2024-04-20T22:06:11.499 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (23m) 8m ago 38m 18.7M - 19.0.0-3133-ga8c84f59 76bc6b91b726 cd793b7ee98e 2024-04-20T22:06:11.499 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (8m) 8m ago 38m 11.0M - 19.0.0-3133-ga8c84f59 76bc6b91b726 0e7b0bc4d1ab 2024-04-20T22:06:11.499 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (38m) 8m ago 38m 18.6M - 18.2.0 dc2bc1663786 f2eec2ee8951 2024-04-20T22:06:11.499 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (34m) 8m ago 47m 513M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T22:06:11.499 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (33m) 8m ago 44m 459M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T22:06:11.499 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (29m) 8m ago 47m 71.7M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 019eb1479485 2024-04-20T22:06:11.499 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (29m) 8m ago 44m 51.0M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 55aac560aab6 2024-04-20T22:06:11.500 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (33m) 8m ago 46m 8472k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T22:06:11.500 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (33m) 8m ago 44m 8048k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T22:06:11.500 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (27m) 8m ago 43m 91.2M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 ee2bb901f26e 2024-04-20T22:06:11.500 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (26m) 8m ago 42m 94.6M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 9075152d4162 2024-04-20T22:06:11.500 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (26m) 8m ago 42m 78.1M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 2d4f3e38dc0e 2024-04-20T22:06:11.500 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (25m) 8m ago 41m 92.6M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 8f97b17ae0eb 2024-04-20T22:06:11.500 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (25m) 8m ago 41m 95.6M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 653d8079967d 2024-04-20T22:06:11.500 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (24m) 8m ago 40m 89.2M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 596dded14c12 2024-04-20T22:06:11.500 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (32m) 8m ago 45m 82.1M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T22:06:11.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:11.501+0000 7f5461200700 1 -- 172.21.15.62:0/724274343 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f544006ecc0 msgr2=0x7f5440071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:06:11.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:11.501+0000 7f5461200700 1 --2- 172.21.15.62:0/724274343 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f544006ecc0 0x7f5440071170 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f544800b5c0 tx=0x7f5448005960 comp rx=0 tx=0).stop 2024-04-20T22:06:11.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:11.501+0000 7f5461200700 1 -- 172.21.15.62:0/724274343 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f545c106b80 msgr2=0x7f545c19c440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:06:11.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:11.501+0000 7f5461200700 1 --2- 172.21.15.62:0/724274343 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f545c106b80 0x7f545c19c440 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f544c00c8a0 tx=0x7f544c00cc60 comp rx=0 tx=0).stop 2024-04-20T22:06:11.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:11.501+0000 7f5461200700 1 -- 172.21.15.62:0/724274343 shutdown_connections 2024-04-20T22:06:11.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:11.501+0000 7f5461200700 1 --2- 172.21.15.62:0/724274343 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f544006ecc0 0x7f5440071170 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:06:11.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:11.501+0000 7f5461200700 1 --2- 172.21.15.62:0/724274343 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f545c106b80 0x7f545c19c440 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:06:11.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:11.501+0000 7f5461200700 1 --2- 172.21.15.62:0/724274343 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f545c10c8f0 0x7f545c19c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:06:11.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:11.501+0000 7f5461200700 1 -- 172.21.15.62:0/724274343 >> 172.21.15.62:0/724274343 conn(0x7f545c076060 msgr2=0x7f545c077ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:06:11.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:11.501+0000 7f5461200700 1 -- 172.21.15.62:0/724274343 shutdown_connections 2024-04-20T22:06:11.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:11.502+0000 7f5461200700 1 -- 172.21.15.62:0/724274343 wait complete. 2024-04-20T22:06:11.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:11.663+0000 7f7490000700 1 -- 172.21.15.62:0/3760825511 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7488100e80 msgr2=0x7f7488104e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:06:11.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:11.663+0000 7f7490000700 1 --2- 172.21.15.62:0/3760825511 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7488100e80 0x7f7488104e50 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f7474009910 tx=0x7f7474009c20 comp rx=0 tx=0).stop 2024-04-20T22:06:11.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:11.664+0000 7f7490000700 1 -- 172.21.15.62:0/3760825511 shutdown_connections 2024-04-20T22:06:11.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:11.664+0000 7f7490000700 1 --2- 172.21.15.62:0/3760825511 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7488100e80 0x7f7488104e50 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:06:11.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:11.664+0000 7f7490000700 1 --2- 172.21.15.62:0/3760825511 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f74881004e0 0x7f74881008b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:06:11.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:11.664+0000 7f7490000700 1 -- 172.21.15.62:0/3760825511 >> 172.21.15.62:0/3760825511 conn(0x7f74880fbda0 msgr2=0x7f74880fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:06:11.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:11.664+0000 7f7490000700 1 -- 172.21.15.62:0/3760825511 shutdown_connections 2024-04-20T22:06:11.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:11.664+0000 7f7490000700 1 -- 172.21.15.62:0/3760825511 wait complete. 2024-04-20T22:06:11.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:11.665+0000 7f7490000700 1 Processor -- start 2024-04-20T22:06:11.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:11.665+0000 7f7490000700 1 -- start start 2024-04-20T22:06:11.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:11.666+0000 7f7490000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f74881004e0 0x7f74881980f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:06:11.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:11.666+0000 7f7490000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7488100e80 0x7f7488198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:06:11.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:11.666+0000 7f7490000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7488198c80 con 0x7f74881004e0 2024-04-20T22:06:11.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:11.666+0000 7f7490000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7488198dc0 con 0x7f7488100e80 2024-04-20T22:06:11.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:11.666+0000 7f748d600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f74881004e0 0x7f74881980f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:06:11.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:11.666+0000 7f748d600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f74881004e0 0x7f74881980f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:46896/0 (socket says 172.21.15.62:46896) 2024-04-20T22:06:11.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:11.666+0000 7f748d600700 1 -- 172.21.15.62:0/4109452414 learned_addr learned my addr 172.21.15.62:0/4109452414 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:06:11.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:11.666+0000 7f748cc00700 1 --2- 172.21.15.62:0/4109452414 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7488100e80 0x7f7488198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:06:11.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:11.667+0000 7f748d600700 1 -- 172.21.15.62:0/4109452414 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7488100e80 msgr2=0x7f7488198630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:06:11.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:11.667+0000 7f748d600700 1 --2- 172.21.15.62:0/4109452414 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7488100e80 0x7f7488198630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:06:11.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:11.667+0000 7f748d600700 1 -- 172.21.15.62:0/4109452414 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74740095f0 con 0x7f74881004e0 2024-04-20T22:06:11.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:11.667+0000 7f748d600700 1 --2- 172.21.15.62:0/4109452414 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f74881004e0 0x7f74881980f0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f747c00c930 tx=0x7f747c00ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:06:11.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:11.667+0000 7f7486000700 1 -- 172.21.15.62:0/4109452414 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f747c007ad0 con 0x7f74881004e0 2024-04-20T22:06:11.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:11.667+0000 7f7486000700 1 -- 172.21.15.62:0/4109452414 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f747c007c30 con 0x7f74881004e0 2024-04-20T22:06:11.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:11.667+0000 7f7490000700 1 -- 172.21.15.62:0/4109452414 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f748819cc10 con 0x7f74881004e0 2024-04-20T22:06:11.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:11.667+0000 7f7486000700 1 -- 172.21.15.62:0/4109452414 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f747c018610 con 0x7f74881004e0 2024-04-20T22:06:11.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:11.667+0000 7f7490000700 1 -- 172.21.15.62:0/4109452414 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f748819d160 con 0x7f74881004e0 2024-04-20T22:06:11.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:11.669+0000 7f7490000700 1 -- 172.21.15.62:0/4109452414 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7468005320 con 0x7f74881004e0 2024-04-20T22:06:11.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:11.670+0000 7f7486000700 1 -- 172.21.15.62:0/4109452414 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f747c018830 con 0x7f74881004e0 2024-04-20T22:06:11.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:11.671+0000 7f7486000700 1 --2- 172.21.15.62:0/4109452414 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f747006ecc0 0x7f7470071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:06:11.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:11.671+0000 7f7486000700 1 -- 172.21.15.62:0/4109452414 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f747c08efb0 con 0x7f74881004e0 2024-04-20T22:06:11.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:11.671+0000 7f748cc00700 1 --2- 172.21.15.62:0/4109452414 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f747006ecc0 0x7f7470071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:06:11.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:11.672+0000 7f748cc00700 1 --2- 172.21.15.62:0/4109452414 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f747006ecc0 0x7f7470071170 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f7488199710 tx=0x7f747401a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:06:11.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:11.676+0000 7f7486000700 1 -- 172.21.15.62:0/4109452414 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f747c0c3ff0 con 0x7f74881004e0 2024-04-20T22:06:11.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:06:11 smithi062 ceph-mon[76322]: from='client.35656 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:06:11.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:06:11 smithi062 ceph-mon[76322]: from='client.35660 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:06:11.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:06:11 smithi189 ceph-mon[57022]: from='client.35656 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:06:11.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:06:11 smithi189 ceph-mon[57022]: from='client.35660 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:06:12.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.004+0000 7f7490000700 1 -- 172.21.15.62:0/4109452414 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f7468006200 con 0x7f74881004e0 2024-04-20T22:06:12.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.006+0000 7f7486000700 1 -- 172.21.15.62:0/4109452414 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7f747c057d40 con 0x7f74881004e0 2024-04-20T22:06:12.006 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:06:12.006 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T22:06:12.006 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:06:12.006 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:06:12.006 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T22:06:12.006 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:06:12.006 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:06:12.006 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T22:06:12.006 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 6 2024-04-20T22:06:12.006 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:06:12.006 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T22:06:12.006 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 1, 2024-04-20T22:06:12.006 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 3 2024-04-20T22:06:12.007 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:06:12.007 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T22:06:12.007 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 1, 2024-04-20T22:06:12.007 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 13 2024-04-20T22:06:12.007 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T22:06:12.007 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:06:12.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.008+0000 7f7490000700 1 -- 172.21.15.62:0/4109452414 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f747006ecc0 msgr2=0x7f7470071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:06:12.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.008+0000 7f7490000700 1 --2- 172.21.15.62:0/4109452414 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f747006ecc0 0x7f7470071170 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f7488199710 tx=0x7f747401a040 comp rx=0 tx=0).stop 2024-04-20T22:06:12.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.008+0000 7f7490000700 1 -- 172.21.15.62:0/4109452414 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f74881004e0 msgr2=0x7f74881980f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:06:12.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.008+0000 7f7490000700 1 --2- 172.21.15.62:0/4109452414 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f74881004e0 0x7f74881980f0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f747c00c930 tx=0x7f747c00ccf0 comp rx=0 tx=0).stop 2024-04-20T22:06:12.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.009+0000 7f7490000700 1 -- 172.21.15.62:0/4109452414 shutdown_connections 2024-04-20T22:06:12.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.009+0000 7f7490000700 1 --2- 172.21.15.62:0/4109452414 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f747006ecc0 0x7f7470071170 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:06:12.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.009+0000 7f7490000700 1 --2- 172.21.15.62:0/4109452414 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f74881004e0 0x7f74881980f0 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:06:12.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.009+0000 7f7490000700 1 --2- 172.21.15.62:0/4109452414 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7488100e80 0x7f7488198630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:06:12.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.009+0000 7f7490000700 1 -- 172.21.15.62:0/4109452414 >> 172.21.15.62:0/4109452414 conn(0x7f74880fbda0 msgr2=0x7f74880fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:06:12.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.009+0000 7f7490000700 1 -- 172.21.15.62:0/4109452414 shutdown_connections 2024-04-20T22:06:12.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.009+0000 7f7490000700 1 -- 172.21.15.62:0/4109452414 wait complete. 2024-04-20T22:06:12.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.169+0000 7fa59e800700 1 -- 172.21.15.62:0/2383928660 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa598075b90 msgr2=0x7fa598110bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:06:12.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.169+0000 7fa59e800700 1 --2- 172.21.15.62:0/2383928660 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa598075b90 0x7fa598110bd0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fa584009910 tx=0x7fa584009c20 comp rx=0 tx=0).stop 2024-04-20T22:06:12.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.170+0000 7fa59e800700 1 -- 172.21.15.62:0/2383928660 shutdown_connections 2024-04-20T22:06:12.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.170+0000 7fa59e800700 1 --2- 172.21.15.62:0/2383928660 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa598075b90 0x7fa598110bd0 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:06:12.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.170+0000 7fa59e800700 1 --2- 172.21.15.62:0/2383928660 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa598075280 0x7fa598075650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:06:12.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.170+0000 7fa59e800700 1 -- 172.21.15.62:0/2383928660 >> 172.21.15.62:0/2383928660 conn(0x7fa5980fdf90 msgr2=0x7fa5981003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:06:12.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.170+0000 7fa59e800700 1 -- 172.21.15.62:0/2383928660 shutdown_connections 2024-04-20T22:06:12.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.170+0000 7fa59e800700 1 -- 172.21.15.62:0/2383928660 wait complete. 2024-04-20T22:06:12.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.171+0000 7fa59e800700 1 Processor -- start 2024-04-20T22:06:12.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.171+0000 7fa59e800700 1 -- start start 2024-04-20T22:06:12.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.171+0000 7fa59e800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa598075280 0x7fa59819c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:06:12.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.171+0000 7fa59e800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa598075b90 0x7fa59819cb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:06:12.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.171+0000 7fa59e800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa59819d1f0 con 0x7fa598075b90 2024-04-20T22:06:12.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.171+0000 7fa59e800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5981a0f80 con 0x7fa598075280 2024-04-20T22:06:12.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.172+0000 7fa596a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa598075b90 0x7fa59819cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:06:12.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.172+0000 7fa596a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa598075b90 0x7fa59819cb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:46908/0 (socket says 172.21.15.62:46908) 2024-04-20T22:06:12.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.172+0000 7fa596a00700 1 -- 172.21.15.62:0/1445730231 learned_addr learned my addr 172.21.15.62:0/1445730231 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:06:12.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.172+0000 7fa597400700 1 --2- 172.21.15.62:0/1445730231 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa598075280 0x7fa59819c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:06:12.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.172+0000 7fa596a00700 1 -- 172.21.15.62:0/1445730231 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa598075280 msgr2=0x7fa59819c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:06:12.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.172+0000 7fa596a00700 1 --2- 172.21.15.62:0/1445730231 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa598075280 0x7fa59819c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:06:12.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.172+0000 7fa596a00700 1 -- 172.21.15.62:0/1445730231 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5840095f0 con 0x7fa598075b90 2024-04-20T22:06:12.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.173+0000 7fa596a00700 1 --2- 172.21.15.62:0/1445730231 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa598075b90 0x7fa59819cb10 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fa584006010 tx=0x7fa5840056a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:06:12.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.173+0000 7fa58fe00700 1 -- 172.21.15.62:0/1445730231 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa58401d070 con 0x7fa598075b90 2024-04-20T22:06:12.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.173+0000 7fa58fe00700 1 -- 172.21.15.62:0/1445730231 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa58400f460 con 0x7fa598075b90 2024-04-20T22:06:12.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.173+0000 7fa58fe00700 1 -- 172.21.15.62:0/1445730231 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa584021640 con 0x7fa598075b90 2024-04-20T22:06:12.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.173+0000 7fa59e800700 1 -- 172.21.15.62:0/1445730231 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5981a1200 con 0x7fa598075b90 2024-04-20T22:06:12.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.173+0000 7fa59e800700 1 -- 172.21.15.62:0/1445730231 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5981a1700 con 0x7fa598075b90 2024-04-20T22:06:12.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.175+0000 7fa59e800700 1 -- 172.21.15.62:0/1445730231 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa59810e340 con 0x7fa598075b90 2024-04-20T22:06:12.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.176+0000 7fa58fe00700 1 -- 172.21.15.62:0/1445730231 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fa58400f5d0 con 0x7fa598075b90 2024-04-20T22:06:12.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.177+0000 7fa58fe00700 1 --2- 172.21.15.62:0/1445730231 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa57c06ecc0 0x7fa57c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:06:12.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.177+0000 7fa58fe00700 1 -- 172.21.15.62:0/1445730231 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fa584090270 con 0x7fa598075b90 2024-04-20T22:06:12.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.177+0000 7fa597400700 1 --2- 172.21.15.62:0/1445730231 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa57c06ecc0 0x7fa57c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:06:12.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.178+0000 7fa597400700 1 --2- 172.21.15.62:0/1445730231 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa57c06ecc0 0x7fa57c071170 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fa588006fd0 tx=0x7fa588008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:06:12.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.182+0000 7fa58fe00700 1 -- 172.21.15.62:0/1445730231 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fa584058f80 con 0x7fa598075b90 2024-04-20T22:06:12.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.453+0000 7fa59e800700 1 -- 172.21.15.62:0/1445730231 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fa59804f2a0 con 0x7fa598075b90 2024-04-20T22:06:12.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.454+0000 7fa58fe00700 1 -- 172.21.15.62:0/1445730231 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 27 v27) v1 ==== 76+0+1888 (secure 0 0 0) 0x7fa58405c640 con 0x7fa598075b90 2024-04-20T22:06:12.454 INFO:teuthology.orchestra.run.smithi062.stdout:e27 2024-04-20T22:06:12.455 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T22:06:12.455 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:06:12.455 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T22:06:12.455 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:06:12.455 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T22:06:12.455 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T22:06:12.455 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 20 2024-04-20T22:06:12.455 INFO:teuthology.orchestra.run.smithi062.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-20T22:06:12.455 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T22:06:12.455 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T21:42:01.559061+0000 2024-04-20T22:06:12.455 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T22:06:12.455 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T22:06:12.455 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T22:06:12.456 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T22:06:12.456 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T22:06:12.456 INFO:teuthology.orchestra.run.smithi062.stdout:max_xattr_size 65536 2024-04-20T22:06:12.456 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T22:06:12.456 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T22:06:12.456 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 81 2024-04-20T22:06:12.456 INFO:teuthology.orchestra.run.smithi062.stdout:snap_rstat 1 2024-04-20T22:06:12.456 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:06:12.456 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 2 2024-04-20T22:06:12.456 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1 2024-04-20T22:06:12.456 INFO:teuthology.orchestra.run.smithi062.stdout:up {} 2024-04-20T22:06:12.456 INFO:teuthology.orchestra.run.smithi062.stdout:failed 0,1 2024-04-20T22:06:12.456 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T22:06:12.456 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T22:06:12.456 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T22:06:12.457 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T22:06:12.457 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data enabled 2024-04-20T22:06:12.457 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T22:06:12.457 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T22:06:12.457 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 1 2024-04-20T22:06:12.457 INFO:teuthology.orchestra.run.smithi062.stdout:qdb_cluster leader: 0 members: 2024-04-20T22:06:12.457 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:06:12.457 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:06:12.457 INFO:teuthology.orchestra.run.smithi062.stdout:Standby daemons: 2024-04-20T22:06:12.457 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:06:12.457 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ryqblb{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6826/4113916205,v1:172.21.15.62:6827/4113916205] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:06:12.457 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ozizwi{-1:34476} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6828/3801975165,v1:172.21.15.62:6829/3801975165] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:06:12.457 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.xbgspw{-1:44285} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6826/3996040526,v1:172.21.15.189:6827/3996040526] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T22:06:12.457 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.dvimer{-1:44727} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6824/3576699232,v1:172.21.15.189:6825/3576699232] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:06:12.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.457+0000 7fa59e800700 1 -- 172.21.15.62:0/1445730231 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa57c06ecc0 msgr2=0x7fa57c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:06:12.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.457+0000 7fa59e800700 1 --2- 172.21.15.62:0/1445730231 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa57c06ecc0 0x7fa57c071170 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fa588006fd0 tx=0x7fa588008040 comp rx=0 tx=0).stop 2024-04-20T22:06:12.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.457+0000 7fa59e800700 1 -- 172.21.15.62:0/1445730231 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa598075b90 msgr2=0x7fa59819cb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:06:12.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.457+0000 7fa59e800700 1 --2- 172.21.15.62:0/1445730231 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa598075b90 0x7fa59819cb10 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fa584006010 tx=0x7fa5840056a0 comp rx=0 tx=0).stop 2024-04-20T22:06:12.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.457+0000 7fa59e800700 1 -- 172.21.15.62:0/1445730231 shutdown_connections 2024-04-20T22:06:12.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.458+0000 7fa59e800700 1 --2- 172.21.15.62:0/1445730231 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa57c06ecc0 0x7fa57c071170 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:06:12.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.458+0000 7fa59e800700 1 --2- 172.21.15.62:0/1445730231 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa598075280 0x7fa59819c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:06:12.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.458+0000 7fa59e800700 1 --2- 172.21.15.62:0/1445730231 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa598075b90 0x7fa59819cb10 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:06:12.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.458+0000 7fa59e800700 1 -- 172.21.15.62:0/1445730231 >> 172.21.15.62:0/1445730231 conn(0x7fa5980fdf90 msgr2=0x7fa5980febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:06:12.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.458+0000 7fa59e800700 1 -- 172.21.15.62:0/1445730231 shutdown_connections 2024-04-20T22:06:12.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.458+0000 7fa59e800700 1 -- 172.21.15.62:0/1445730231 wait complete. 2024-04-20T22:06:12.459 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 27 2024-04-20T22:06:12.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.617+0000 7f5d4d800700 1 -- 172.21.15.62:0/2272013356 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d48106b80 msgr2=0x7f5d48106ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:06:12.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.617+0000 7f5d4d800700 1 --2- 172.21.15.62:0/2272013356 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d48106b80 0x7f5d48106ff0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f5d34009910 tx=0x7f5d34009c20 comp rx=0 tx=0).stop 2024-04-20T22:06:12.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.617+0000 7f5d4d800700 1 -- 172.21.15.62:0/2272013356 shutdown_connections 2024-04-20T22:06:12.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.617+0000 7f5d4d800700 1 --2- 172.21.15.62:0/2272013356 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d48106b80 0x7f5d48106ff0 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:06:12.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.617+0000 7f5d4d800700 1 --2- 172.21.15.62:0/2272013356 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d4810c8f0 0x7f5d4810ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:06:12.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.617+0000 7f5d4d800700 1 -- 172.21.15.62:0/2272013356 >> 172.21.15.62:0/2272013356 conn(0x7f5d48076060 msgr2=0x7f5d48078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:06:12.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.617+0000 7f5d4d800700 1 -- 172.21.15.62:0/2272013356 shutdown_connections 2024-04-20T22:06:12.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.618+0000 7f5d4d800700 1 -- 172.21.15.62:0/2272013356 wait complete. 2024-04-20T22:06:12.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.619+0000 7f5d4d800700 1 Processor -- start 2024-04-20T22:06:12.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.619+0000 7f5d4d800700 1 -- start start 2024-04-20T22:06:12.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.620+0000 7f5d4d800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d48106b80 0x7f5d4819c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:06:12.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.620+0000 7f5d4d800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d4810c8f0 0x7f5d4819c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:06:12.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.620+0000 7f5d4d800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d4819d090 con 0x7f5d4810c8f0 2024-04-20T22:06:12.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.620+0000 7f5d4d800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d481a0e20 con 0x7f5d48106b80 2024-04-20T22:06:12.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.620+0000 7f5d46000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d4810c8f0 0x7f5d4819c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:06:12.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.620+0000 7f5d46000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d4810c8f0 0x7f5d4819c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:46926/0 (socket says 172.21.15.62:46926) 2024-04-20T22:06:12.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.620+0000 7f5d46000700 1 -- 172.21.15.62:0/817306695 learned_addr learned my addr 172.21.15.62:0/817306695 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:06:12.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.620+0000 7f5d46a00700 1 --2- 172.21.15.62:0/817306695 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d48106b80 0x7f5d4819c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:06:12.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.620+0000 7f5d46000700 1 -- 172.21.15.62:0/817306695 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d48106b80 msgr2=0x7f5d4819c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:06:12.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.620+0000 7f5d46000700 1 --2- 172.21.15.62:0/817306695 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d48106b80 0x7f5d4819c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:06:12.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.620+0000 7f5d46000700 1 -- 172.21.15.62:0/817306695 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d340095f0 con 0x7f5d4810c8f0 2024-04-20T22:06:12.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.621+0000 7f5d46000700 1 --2- 172.21.15.62:0/817306695 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d4810c8f0 0x7f5d4819c9b0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f5d34006010 tx=0x7f5d3400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:06:12.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.621+0000 7f5d3f400700 1 -- 172.21.15.62:0/817306695 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5d3401d070 con 0x7f5d4810c8f0 2024-04-20T22:06:12.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.621+0000 7f5d3f400700 1 -- 172.21.15.62:0/817306695 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5d3400f460 con 0x7f5d4810c8f0 2024-04-20T22:06:12.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.621+0000 7f5d3f400700 1 -- 172.21.15.62:0/817306695 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5d34017630 con 0x7f5d4810c8f0 2024-04-20T22:06:12.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.621+0000 7f5d4d800700 1 -- 172.21.15.62:0/817306695 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d481a10a0 con 0x7f5d4810c8f0 2024-04-20T22:06:12.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.621+0000 7f5d4d800700 1 -- 172.21.15.62:0/817306695 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d481a1590 con 0x7f5d4810c8f0 2024-04-20T22:06:12.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.622+0000 7f5d4d800700 1 -- 172.21.15.62:0/817306695 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d4810ecc0 con 0x7f5d4810c8f0 2024-04-20T22:06:12.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.624+0000 7f5d3f400700 1 -- 172.21.15.62:0/817306695 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f5d3400f5d0 con 0x7f5d4810c8f0 2024-04-20T22:06:12.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.624+0000 7f5d3f400700 1 --2- 172.21.15.62:0/817306695 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5d2c06ecc0 0x7f5d2c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:06:12.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.625+0000 7f5d3f400700 1 -- 172.21.15.62:0/817306695 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f5d34090290 con 0x7f5d4810c8f0 2024-04-20T22:06:12.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.625+0000 7f5d46a00700 1 --2- 172.21.15.62:0/817306695 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5d2c06ecc0 0x7f5d2c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:06:12.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.626+0000 7f5d46a00700 1 --2- 172.21.15.62:0/817306695 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5d2c06ecc0 0x7f5d2c071170 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f5d38005950 tx=0x7f5d380058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:06:12.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.630+0000 7f5d3f400700 1 -- 172.21.15.62:0/817306695 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f5d340590c0 con 0x7f5d4810c8f0 2024-04-20T22:06:12.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:06:12 smithi062 ceph-mon[76322]: from='client.35664 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:06:12.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:06:12 smithi062 ceph-mon[76322]: pgmap v928: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:06:12.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:06:12 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/4109452414' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:06:12.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:06:12 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/1445730231' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:06:12.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.885+0000 7f5d4d800700 1 -- 172.21.15.62:0/817306695 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5d4802cca0 con 0x7f5d2c06ecc0 2024-04-20T22:06:12.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.888+0000 7f5d3f400700 1 -- 172.21.15.62:0/817306695 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f5d4802cca0 con 0x7f5d2c06ecc0 2024-04-20T22:06:12.888 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:06:12.888 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T22:06:12.888 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T22:06:12.888 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-20T22:06:12.888 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T22:06:12.888 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr", 2024-04-20T22:06:12.888 INFO:teuthology.orchestra.run.smithi062.stdout: "crash", 2024-04-20T22:06:12.888 INFO:teuthology.orchestra.run.smithi062.stdout: "mon", 2024-04-20T22:06:12.888 INFO:teuthology.orchestra.run.smithi062.stdout: "osd" 2024-04-20T22:06:12.889 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T22:06:12.889 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "15/23 daemons upgraded", 2024-04-20T22:06:12.889 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading mds daemons", 2024-04-20T22:06:12.889 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T22:06:12.889 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:06:12.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.891+0000 7f5d4d800700 1 -- 172.21.15.62:0/817306695 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5d2c06ecc0 msgr2=0x7f5d2c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:06:12.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.891+0000 7f5d4d800700 1 --2- 172.21.15.62:0/817306695 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5d2c06ecc0 0x7f5d2c071170 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f5d38005950 tx=0x7f5d380058e0 comp rx=0 tx=0).stop 2024-04-20T22:06:12.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.891+0000 7f5d4d800700 1 -- 172.21.15.62:0/817306695 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d4810c8f0 msgr2=0x7f5d4819c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:06:12.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.891+0000 7f5d4d800700 1 --2- 172.21.15.62:0/817306695 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d4810c8f0 0x7f5d4819c9b0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f5d34006010 tx=0x7f5d3400be20 comp rx=0 tx=0).stop 2024-04-20T22:06:12.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.891+0000 7f5d4d800700 1 -- 172.21.15.62:0/817306695 shutdown_connections 2024-04-20T22:06:12.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.891+0000 7f5d4d800700 1 --2- 172.21.15.62:0/817306695 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5d2c06ecc0 0x7f5d2c071170 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:06:12.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.891+0000 7f5d4d800700 1 --2- 172.21.15.62:0/817306695 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d48106b80 0x7f5d4819c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:06:12.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.891+0000 7f5d4d800700 1 --2- 172.21.15.62:0/817306695 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d4810c8f0 0x7f5d4819c9b0 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:06:12.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.891+0000 7f5d4d800700 1 -- 172.21.15.62:0/817306695 >> 172.21.15.62:0/817306695 conn(0x7f5d48076060 msgr2=0x7f5d48077c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:06:12.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.891+0000 7f5d4d800700 1 -- 172.21.15.62:0/817306695 shutdown_connections 2024-04-20T22:06:12.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:12.891+0000 7f5d4d800700 1 -- 172.21.15.62:0/817306695 wait complete. 2024-04-20T22:06:12.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:06:12 smithi189 ceph-mon[57022]: from='client.35664 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:06:12.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:06:12 smithi189 ceph-mon[57022]: pgmap v928: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:06:12.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:06:12 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/4109452414' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:06:12.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:06:12 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/1445730231' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:06:13.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:13.058+0000 7f87dc400700 1 -- 172.21.15.62:0/3074334694 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f87d40730f0 msgr2=0x7f87d40734c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:06:13.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:13.058+0000 7f87dc400700 1 --2- 172.21.15.62:0/3074334694 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f87d40730f0 0x7f87d40734c0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f87c8009910 tx=0x7f87c8009c20 comp rx=0 tx=0).stop 2024-04-20T22:06:13.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:13.058+0000 7f87dc400700 1 -- 172.21.15.62:0/3074334694 shutdown_connections 2024-04-20T22:06:13.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:13.058+0000 7f87dc400700 1 --2- 172.21.15.62:0/3074334694 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87d4073a00 0x7f87d410c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:06:13.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:13.058+0000 7f87dc400700 1 --2- 172.21.15.62:0/3074334694 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f87d40730f0 0x7f87d40734c0 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:06:13.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:13.058+0000 7f87dc400700 1 -- 172.21.15.62:0/3074334694 >> 172.21.15.62:0/3074334694 conn(0x7f87d40fbda0 msgr2=0x7f87d40fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:06:13.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:13.058+0000 7f87dc400700 1 -- 172.21.15.62:0/3074334694 shutdown_connections 2024-04-20T22:06:13.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:13.058+0000 7f87dc400700 1 -- 172.21.15.62:0/3074334694 wait complete. 2024-04-20T22:06:13.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:13.059+0000 7f87dc400700 1 Processor -- start 2024-04-20T22:06:13.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:13.059+0000 7f87dc400700 1 -- start start 2024-04-20T22:06:13.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:13.060+0000 7f87dc400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f87d40730f0 0x7f87d4197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:06:13.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:13.060+0000 7f87dc400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87d4073a00 0x7f87d4198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:06:13.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:13.060+0000 7f87dc400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87d419c970 con 0x7f87d40730f0 2024-04-20T22:06:13.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:13.060+0000 7f87dc400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87d419cae0 con 0x7f87d4073a00 2024-04-20T22:06:13.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:13.060+0000 7f87d9a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f87d40730f0 0x7f87d4197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:06:13.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:13.060+0000 7f87d9a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f87d40730f0 0x7f87d4197fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:46946/0 (socket says 172.21.15.62:46946) 2024-04-20T22:06:13.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:13.060+0000 7f87d9a00700 1 -- 172.21.15.62:0/1467977613 learned_addr learned my addr 172.21.15.62:0/1467977613 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:06:13.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:13.061+0000 7f87d9000700 1 --2- 172.21.15.62:0/1467977613 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87d4073a00 0x7f87d4198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:06:13.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:13.061+0000 7f87d9a00700 1 -- 172.21.15.62:0/1467977613 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87d4073a00 msgr2=0x7f87d4198630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:06:13.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:13.061+0000 7f87d9a00700 1 --2- 172.21.15.62:0/1467977613 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87d4073a00 0x7f87d4198630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:06:13.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:13.061+0000 7f87d9a00700 1 -- 172.21.15.62:0/1467977613 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87c80095f0 con 0x7f87d40730f0 2024-04-20T22:06:13.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:13.061+0000 7f87d9a00700 1 --2- 172.21.15.62:0/1467977613 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f87d40730f0 0x7f87d4197fc0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f87c800b5c0 tx=0x7f87c800be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:06:13.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:13.061+0000 7f87d2000700 1 -- 172.21.15.62:0/1467977613 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f87c801d070 con 0x7f87d40730f0 2024-04-20T22:06:13.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:13.061+0000 7f87d2000700 1 -- 172.21.15.62:0/1467977613 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f87c800f460 con 0x7f87d40730f0 2024-04-20T22:06:13.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:13.061+0000 7f87d2000700 1 -- 172.21.15.62:0/1467977613 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f87c8017630 con 0x7f87d40730f0 2024-04-20T22:06:13.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:13.062+0000 7f87dc400700 1 -- 172.21.15.62:0/1467977613 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87d419cdc0 con 0x7f87d40730f0 2024-04-20T22:06:13.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:13.062+0000 7f87dc400700 1 -- 172.21.15.62:0/1467977613 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87d419d310 con 0x7f87d40730f0 2024-04-20T22:06:13.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:13.063+0000 7f87dc400700 1 -- 172.21.15.62:0/1467977613 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87d4109c80 con 0x7f87d40730f0 2024-04-20T22:06:13.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:13.064+0000 7f87d2000700 1 -- 172.21.15.62:0/1467977613 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f87c800f5d0 con 0x7f87d40730f0 2024-04-20T22:06:13.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:13.065+0000 7f87d2000700 1 --2- 172.21.15.62:0/1467977613 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f87bc06ecc0 0x7f87bc071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:06:13.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:13.065+0000 7f87d2000700 1 -- 172.21.15.62:0/1467977613 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f87c80903c0 con 0x7f87d40730f0 2024-04-20T22:06:13.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:13.065+0000 7f87d9000700 1 --2- 172.21.15.62:0/1467977613 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f87bc06ecc0 0x7f87bc071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:06:13.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:13.066+0000 7f87d9000700 1 --2- 172.21.15.62:0/1467977613 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f87bc06ecc0 0x7f87bc071170 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f87c0005fd0 tx=0x7f87c000a560 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:06:13.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:13.070+0000 7f87d2000700 1 -- 172.21.15.62:0/1467977613 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f87c8059150 con 0x7f87d40730f0 2024-04-20T22:06:13.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:13.398+0000 7f87dc400700 1 -- 172.21.15.62:0/1467977613 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f87d404f2a0 con 0x7f87d40730f0 2024-04-20T22:06:13.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:13.399+0000 7f87d2000700 1 -- 172.21.15.62:0/1467977613 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7f87c805c810 con 0x7f87d40730f0 2024-04-20T22:06:13.398 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-20T22:06:13.399 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-20T22:06:13.399 INFO:teuthology.orchestra.run.smithi062.stdout: host smithi062 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T22:06:13.399 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-20T22:06:13.399 INFO:teuthology.orchestra.run.smithi062.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi062 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpq7y7oa_r:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:06:13.399 INFO:teuthology.orchestra.run.smithi062.stdout:Traceback (most recent call last): 2024-04-20T22:06:13.399 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-20T22:06:13.399 INFO:teuthology.orchestra.run.smithi062.stdout: return _run_code(code, main_globals, None, 2024-04-20T22:06:13.399 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-20T22:06:13.399 INFO:teuthology.orchestra.run.smithi062.stdout: exec(code, run_globals) 2024-04-20T22:06:13.399 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5618, in 2024-04-20T22:06:13.399 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5606, in main 2024-04-20T22:06:13.399 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 408, in _infer_config 2024-04-20T22:06:13.399 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 323, in _infer_fsid 2024-04-20T22:06:13.399 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 436, in _infer_image 2024-04-20T22:06:13.399 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 310, in _validate_fsid 2024-04-20T22:06:13.400 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 3359, in command_ceph_volume 2024-04-20T22:06:13.400 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-20T22:06:13.400 INFO:teuthology.orchestra.run.smithi062.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi062 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpq7y7oa_r:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:06:13.400 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-20T22:06:13.400 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is degraded 2024-04-20T22:06:13.400 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T22:06:13.400 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T22:06:13.400 INFO:teuthology.orchestra.run.smithi062.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-20T22:06:13.400 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-20T22:06:13.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:13.401+0000 7f87dc400700 1 -- 172.21.15.62:0/1467977613 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f87bc06ecc0 msgr2=0x7f87bc071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:06:13.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:13.401+0000 7f87dc400700 1 --2- 172.21.15.62:0/1467977613 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f87bc06ecc0 0x7f87bc071170 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f87c0005fd0 tx=0x7f87c000a560 comp rx=0 tx=0).stop 2024-04-20T22:06:13.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:13.402+0000 7f87dc400700 1 -- 172.21.15.62:0/1467977613 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f87d40730f0 msgr2=0x7f87d4197fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:06:13.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:13.402+0000 7f87dc400700 1 --2- 172.21.15.62:0/1467977613 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f87d40730f0 0x7f87d4197fc0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f87c800b5c0 tx=0x7f87c800be00 comp rx=0 tx=0).stop 2024-04-20T22:06:13.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:13.402+0000 7f87dc400700 1 -- 172.21.15.62:0/1467977613 shutdown_connections 2024-04-20T22:06:13.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:13.402+0000 7f87dc400700 1 --2- 172.21.15.62:0/1467977613 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f87bc06ecc0 0x7f87bc071170 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:06:13.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:13.402+0000 7f87dc400700 1 --2- 172.21.15.62:0/1467977613 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f87d40730f0 0x7f87d4197fc0 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:06:13.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:13.402+0000 7f87dc400700 1 --2- 172.21.15.62:0/1467977613 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87d4073a00 0x7f87d4198630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:06:13.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:13.402+0000 7f87dc400700 1 -- 172.21.15.62:0/1467977613 >> 172.21.15.62:0/1467977613 conn(0x7f87d40fbda0 msgr2=0x7f87d4106dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:06:13.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:13.402+0000 7f87dc400700 1 -- 172.21.15.62:0/1467977613 shutdown_connections 2024-04-20T22:06:13.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:13.402+0000 7f87dc400700 1 -- 172.21.15.62:0/1467977613 wait complete. 2024-04-20T22:06:13.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:06:13 smithi189 ceph-mon[57022]: from='client.35676 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:06:13.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:06:13 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/1467977613' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:06:14.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:06:13 smithi062 ceph-mon[76322]: from='client.35676 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:06:14.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:06:13 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/1467977613' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:06:14.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:06:14 smithi189 ceph-mon[57022]: pgmap v929: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:06:15.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:06:14 smithi062 ceph-mon[76322]: pgmap v929: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:06:16.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:06:16 smithi189 ceph-mon[57022]: pgmap v930: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:06:17.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:06:16 smithi062 ceph-mon[76322]: pgmap v930: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:06:18.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:06:18 smithi062 ceph-mon[76322]: pgmap v931: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:06:18.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:06:18 smithi189 ceph-mon[57022]: pgmap v931: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:06:19.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:06:19 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:06:20.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:06:19 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:06:20.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:06:20 smithi189 ceph-mon[57022]: pgmap v932: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:06:21.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:06:20 smithi062 ceph-mon[76322]: pgmap v932: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:06:23.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:06:22 smithi062 ceph-mon[76322]: pgmap v933: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:06:23.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:06:22 smithi189 ceph-mon[57022]: pgmap v933: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:06:25.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:06:24 smithi062 ceph-mon[76322]: pgmap v934: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:06:25.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:06:24 smithi189 ceph-mon[57022]: pgmap v934: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:06:27.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:06:26 smithi062 ceph-mon[76322]: pgmap v935: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:06:27.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:06:26 smithi189 ceph-mon[57022]: pgmap v935: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:06:28.860 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:06:28 smithi062 ceph-mon[76322]: pgmap v936: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:06:29.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:06:28 smithi189 ceph-mon[57022]: pgmap v936: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:06:31.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:06:30 smithi062 ceph-mon[76322]: pgmap v937: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:06:31.180 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:06:30 smithi189 ceph-mon[57022]: pgmap v937: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:06:33.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:06:32 smithi062 ceph-mon[76322]: pgmap v938: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:06:33.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:06:32 smithi189 ceph-mon[57022]: pgmap v938: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:06:35.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:06:34 smithi062 ceph-mon[76322]: pgmap v939: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:06:35.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:06:34 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:06:35.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:06:34 smithi189 ceph-mon[57022]: pgmap v939: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:06:35.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:06:34 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:06:37.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:06:36 smithi062 ceph-mon[76322]: pgmap v940: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:06:37.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:06:36 smithi189 ceph-mon[57022]: pgmap v940: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:06:39.042 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:06:38 smithi189 ceph-mon[57022]: pgmap v941: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:06:39.187 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:06:38 smithi062 ceph-mon[76322]: pgmap v941: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:06:41.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:06:40 smithi189 ceph-mon[57022]: pgmap v942: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:06:41.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:06:40 smithi062 ceph-mon[76322]: pgmap v942: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:06:43.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:06:42 smithi189 ceph-mon[57022]: pgmap v943: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:06:43.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:06:42 smithi062 ceph-mon[76322]: pgmap v943: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:06:43.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:43.592+0000 7fd36e000700 1 -- 172.21.15.62:0/834625600 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3680687a0 msgr2=0x7fd368068b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:06:43.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:43.592+0000 7fd36e000700 1 --2- 172.21.15.62:0/834625600 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3680687a0 0x7fd368068b70 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fd358009910 tx=0x7fd358009c20 comp rx=0 tx=0).stop 2024-04-20T22:06:43.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:43.592+0000 7fd36e000700 1 -- 172.21.15.62:0/834625600 shutdown_connections 2024-04-20T22:06:43.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:43.592+0000 7fd36e000700 1 --2- 172.21.15.62:0/834625600 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd368069140 0x7fd368104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:06:43.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:43.592+0000 7fd36e000700 1 --2- 172.21.15.62:0/834625600 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3680687a0 0x7fd368068b70 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:06:43.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:43.592+0000 7fd36e000700 1 -- 172.21.15.62:0/834625600 >> 172.21.15.62:0/834625600 conn(0x7fd368075130 msgr2=0x7fd368075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:06:43.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:43.592+0000 7fd36e000700 1 -- 172.21.15.62:0/834625600 shutdown_connections 2024-04-20T22:06:43.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:43.592+0000 7fd36e000700 1 -- 172.21.15.62:0/834625600 wait complete. 2024-04-20T22:06:43.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:43.593+0000 7fd36e000700 1 Processor -- start 2024-04-20T22:06:43.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:43.594+0000 7fd36e000700 1 -- start start 2024-04-20T22:06:43.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:43.594+0000 7fd36e000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3680687a0 0x7fd3681980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:06:43.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:43.594+0000 7fd36e000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd368069140 0x7fd3681985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:06:43.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:43.594+0000 7fd36e000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd368198c40 con 0x7fd368069140 2024-04-20T22:06:43.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:43.594+0000 7fd36e000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd368198d80 con 0x7fd3680687a0 2024-04-20T22:06:43.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:43.594+0000 7fd366a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd368069140 0x7fd3681985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:06:43.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:43.595+0000 7fd366a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd368069140 0x7fd3681985f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34180/0 (socket says 172.21.15.62:34180) 2024-04-20T22:06:43.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:43.595+0000 7fd366a00700 1 -- 172.21.15.62:0/3960324293 learned_addr learned my addr 172.21.15.62:0/3960324293 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:06:43.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:43.595+0000 7fd367400700 1 --2- 172.21.15.62:0/3960324293 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3680687a0 0x7fd3681980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:06:43.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:43.595+0000 7fd366a00700 1 -- 172.21.15.62:0/3960324293 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3680687a0 msgr2=0x7fd3681980b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:06:43.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:43.595+0000 7fd366a00700 1 --2- 172.21.15.62:0/3960324293 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3680687a0 0x7fd3681980b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:06:43.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:43.595+0000 7fd366a00700 1 -- 172.21.15.62:0/3960324293 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3580095f0 con 0x7fd368069140 2024-04-20T22:06:43.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:43.595+0000 7fd367400700 1 --2- 172.21.15.62:0/3960324293 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3680687a0 0x7fd3681980b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-20T22:06:43.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:43.595+0000 7fd366a00700 1 --2- 172.21.15.62:0/3960324293 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd368069140 0x7fd3681985f0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fd35400c960 tx=0x7fd35400cd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:06:43.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:43.596+0000 7fd35fe00700 1 -- 172.21.15.62:0/3960324293 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd354007a70 con 0x7fd368069140 2024-04-20T22:06:43.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:43.596+0000 7fd35fe00700 1 -- 172.21.15.62:0/3960324293 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd354007bd0 con 0x7fd368069140 2024-04-20T22:06:43.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:43.596+0000 7fd36e000700 1 -- 172.21.15.62:0/3960324293 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd36819cbd0 con 0x7fd368069140 2024-04-20T22:06:43.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:43.596+0000 7fd35fe00700 1 -- 172.21.15.62:0/3960324293 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd3540186c0 con 0x7fd368069140 2024-04-20T22:06:43.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:43.596+0000 7fd36e000700 1 -- 172.21.15.62:0/3960324293 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd36819d120 con 0x7fd368069140 2024-04-20T22:06:43.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:43.597+0000 7fd36e000700 1 -- 172.21.15.62:0/3960324293 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd36804f2a0 con 0x7fd368069140 2024-04-20T22:06:43.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:43.598+0000 7fd35fe00700 1 -- 172.21.15.62:0/3960324293 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fd35400faa0 con 0x7fd368069140 2024-04-20T22:06:43.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:43.599+0000 7fd35fe00700 1 --2- 172.21.15.62:0/3960324293 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd34c06ecc0 0x7fd34c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:06:43.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:43.599+0000 7fd35fe00700 1 -- 172.21.15.62:0/3960324293 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fd35408eea0 con 0x7fd368069140 2024-04-20T22:06:43.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:43.599+0000 7fd367400700 1 --2- 172.21.15.62:0/3960324293 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd34c06ecc0 0x7fd34c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:06:43.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:43.600+0000 7fd367400700 1 --2- 172.21.15.62:0/3960324293 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd34c06ecc0 0x7fd34c071170 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fd35800b5c0 tx=0x7fd35801a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:06:43.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:43.604+0000 7fd35fe00700 1 -- 172.21.15.62:0/3960324293 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fd35405b760 con 0x7fd368069140 2024-04-20T22:06:43.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:43.860+0000 7fd36e000700 1 -- 172.21.15.62:0/3960324293 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd36819d490 con 0x7fd34c06ecc0 2024-04-20T22:06:43.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:43.862+0000 7fd35fe00700 1 -- 172.21.15.62:0/3960324293 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fd36819d490 con 0x7fd34c06ecc0 2024-04-20T22:06:43.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:43.865+0000 7fd36e000700 1 -- 172.21.15.62:0/3960324293 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd34c06ecc0 msgr2=0x7fd34c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:06:43.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:43.865+0000 7fd36e000700 1 --2- 172.21.15.62:0/3960324293 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd34c06ecc0 0x7fd34c071170 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fd35800b5c0 tx=0x7fd35801a040 comp rx=0 tx=0).stop 2024-04-20T22:06:43.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:43.865+0000 7fd36e000700 1 -- 172.21.15.62:0/3960324293 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd368069140 msgr2=0x7fd3681985f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:06:43.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:43.865+0000 7fd36e000700 1 --2- 172.21.15.62:0/3960324293 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd368069140 0x7fd3681985f0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fd35400c960 tx=0x7fd35400cd20 comp rx=0 tx=0).stop 2024-04-20T22:06:43.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:43.866+0000 7fd36e000700 1 -- 172.21.15.62:0/3960324293 shutdown_connections 2024-04-20T22:06:43.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:43.866+0000 7fd36e000700 1 --2- 172.21.15.62:0/3960324293 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd34c06ecc0 0x7fd34c071170 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:06:43.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:43.866+0000 7fd36e000700 1 --2- 172.21.15.62:0/3960324293 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3680687a0 0x7fd3681980b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:06:43.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:43.866+0000 7fd36e000700 1 --2- 172.21.15.62:0/3960324293 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd368069140 0x7fd3681985f0 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:06:43.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:43.866+0000 7fd36e000700 1 -- 172.21.15.62:0/3960324293 >> 172.21.15.62:0/3960324293 conn(0x7fd368075130 msgr2=0x7fd368101e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:06:43.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:43.866+0000 7fd36e000700 1 -- 172.21.15.62:0/3960324293 shutdown_connections 2024-04-20T22:06:43.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:43.866+0000 7fd36e000700 1 -- 172.21.15.62:0/3960324293 wait complete. 2024-04-20T22:06:43.882 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T22:06:44.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.035+0000 7f3608c00700 1 -- 172.21.15.62:0/3286435403 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f360410c8f0 msgr2=0x7f360410ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:06:44.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.035+0000 7f3608c00700 1 --2- 172.21.15.62:0/3286435403 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f360410c8f0 0x7f360410ccc0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f35f4009910 tx=0x7f35f4009c20 comp rx=0 tx=0).stop 2024-04-20T22:06:44.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.035+0000 7f3608c00700 1 -- 172.21.15.62:0/3286435403 shutdown_connections 2024-04-20T22:06:44.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.035+0000 7f3608c00700 1 --2- 172.21.15.62:0/3286435403 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3604106b80 0x7f3604106ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:06:44.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.035+0000 7f3608c00700 1 --2- 172.21.15.62:0/3286435403 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f360410c8f0 0x7f360410ccc0 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:06:44.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.035+0000 7f3608c00700 1 -- 172.21.15.62:0/3286435403 >> 172.21.15.62:0/3286435403 conn(0x7f3604076060 msgr2=0x7f3604078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:06:44.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.035+0000 7f3608c00700 1 -- 172.21.15.62:0/3286435403 shutdown_connections 2024-04-20T22:06:44.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.035+0000 7f3608c00700 1 -- 172.21.15.62:0/3286435403 wait complete. 2024-04-20T22:06:44.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.037+0000 7f3608c00700 1 Processor -- start 2024-04-20T22:06:44.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.037+0000 7f3608c00700 1 -- start start 2024-04-20T22:06:44.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.037+0000 7f3608c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3604106b80 0x7f360419c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:06:44.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.037+0000 7f3608c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f360410c8f0 0x7f360419ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:06:44.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.037+0000 7f3608c00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f360419d120 con 0x7f360410c8f0 2024-04-20T22:06:44.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.037+0000 7f3608c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36041a0eb0 con 0x7f3604106b80 2024-04-20T22:06:44.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.038+0000 7f35fbe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f360410c8f0 0x7f360419ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:06:44.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.038+0000 7f35fbe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f360410c8f0 0x7f360419ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34198/0 (socket says 172.21.15.62:34198) 2024-04-20T22:06:44.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.038+0000 7f35fbe00700 1 -- 172.21.15.62:0/574970737 learned_addr learned my addr 172.21.15.62:0/574970737 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:06:44.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.038+0000 7f3601e00700 1 --2- 172.21.15.62:0/574970737 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3604106b80 0x7f360419c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:06:44.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.038+0000 7f35fbe00700 1 -- 172.21.15.62:0/574970737 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3604106b80 msgr2=0x7f360419c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:06:44.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.038+0000 7f35fbe00700 1 --2- 172.21.15.62:0/574970737 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3604106b80 0x7f360419c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:06:44.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.038+0000 7f35fbe00700 1 -- 172.21.15.62:0/574970737 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35f40095f0 con 0x7f360410c8f0 2024-04-20T22:06:44.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.039+0000 7f35fbe00700 1 --2- 172.21.15.62:0/574970737 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f360410c8f0 0x7f360419ca40 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f35ec00d8d0 tx=0x7f35ec00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:06:44.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.039+0000 7f35faa00700 1 -- 172.21.15.62:0/574970737 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f35ec009940 con 0x7f360410c8f0 2024-04-20T22:06:44.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.039+0000 7f35faa00700 1 -- 172.21.15.62:0/574970737 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f35ec010460 con 0x7f360410c8f0 2024-04-20T22:06:44.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.039+0000 7f3608c00700 1 -- 172.21.15.62:0/574970737 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36041a1190 con 0x7f360410c8f0 2024-04-20T22:06:44.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.039+0000 7f35faa00700 1 -- 172.21.15.62:0/574970737 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f35ec00f5f0 con 0x7f360410c8f0 2024-04-20T22:06:44.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.039+0000 7f3608c00700 1 -- 172.21.15.62:0/574970737 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36041a16e0 con 0x7f360410c8f0 2024-04-20T22:06:44.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.040+0000 7f3608c00700 1 -- 172.21.15.62:0/574970737 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3604109b20 con 0x7f360410c8f0 2024-04-20T22:06:44.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.041+0000 7f35faa00700 1 -- 172.21.15.62:0/574970737 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f35ec0105d0 con 0x7f360410c8f0 2024-04-20T22:06:44.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.043+0000 7f35faa00700 1 --2- 172.21.15.62:0/574970737 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f35e806ecc0 0x7f35e8071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:06:44.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.043+0000 7f35faa00700 1 -- 172.21.15.62:0/574970737 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f35ec08ee10 con 0x7f360410c8f0 2024-04-20T22:06:44.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.043+0000 7f3601e00700 1 --2- 172.21.15.62:0/574970737 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f35e806ecc0 0x7f35e8071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:06:44.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.044+0000 7f3601e00700 1 --2- 172.21.15.62:0/574970737 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f35e806ecc0 0x7f35e8071170 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f35f4006010 tx=0x7f35f401a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:06:44.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.048+0000 7f35faa00700 1 -- 172.21.15.62:0/574970737 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f35ec057ba0 con 0x7f360410c8f0 2024-04-20T22:06:44.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.299+0000 7f3608c00700 1 -- 172.21.15.62:0/574970737 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f36041a1950 con 0x7f35e806ecc0 2024-04-20T22:06:44.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.301+0000 7f35faa00700 1 -- 172.21.15.62:0/574970737 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f36041a1950 con 0x7f35e806ecc0 2024-04-20T22:06:44.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.304+0000 7f3608c00700 1 -- 172.21.15.62:0/574970737 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f35e806ecc0 msgr2=0x7f35e8071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:06:44.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.304+0000 7f3608c00700 1 --2- 172.21.15.62:0/574970737 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f35e806ecc0 0x7f35e8071170 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f35f4006010 tx=0x7f35f401a040 comp rx=0 tx=0).stop 2024-04-20T22:06:44.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.304+0000 7f3608c00700 1 -- 172.21.15.62:0/574970737 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f360410c8f0 msgr2=0x7f360419ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:06:44.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.304+0000 7f3608c00700 1 --2- 172.21.15.62:0/574970737 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f360410c8f0 0x7f360419ca40 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f35ec00d8d0 tx=0x7f35ec00dc90 comp rx=0 tx=0).stop 2024-04-20T22:06:44.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.304+0000 7f3608c00700 1 -- 172.21.15.62:0/574970737 shutdown_connections 2024-04-20T22:06:44.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.304+0000 7f3608c00700 1 --2- 172.21.15.62:0/574970737 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f35e806ecc0 0x7f35e8071170 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:06:44.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.304+0000 7f3608c00700 1 --2- 172.21.15.62:0/574970737 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3604106b80 0x7f360419c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:06:44.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.304+0000 7f3608c00700 1 --2- 172.21.15.62:0/574970737 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f360410c8f0 0x7f360419ca40 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:06:44.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.304+0000 7f3608c00700 1 -- 172.21.15.62:0/574970737 >> 172.21.15.62:0/574970737 conn(0x7f3604076060 msgr2=0x7f3604077aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:06:44.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.305+0000 7f3608c00700 1 -- 172.21.15.62:0/574970737 shutdown_connections 2024-04-20T22:06:44.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.305+0000 7f3608c00700 1 -- 172.21.15.62:0/574970737 wait complete. 2024-04-20T22:06:44.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.463+0000 7f054b600700 1 -- 172.21.15.62:0/2557218301 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f05440687a0 msgr2=0x7f0544068b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:06:44.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.463+0000 7f054b600700 1 --2- 172.21.15.62:0/2557218301 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f05440687a0 0x7f0544068b70 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f0538009910 tx=0x7f0538009c20 comp rx=0 tx=0).stop 2024-04-20T22:06:44.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.463+0000 7f054b600700 1 -- 172.21.15.62:0/2557218301 shutdown_connections 2024-04-20T22:06:44.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.463+0000 7f054b600700 1 --2- 172.21.15.62:0/2557218301 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0544069140 0x7f0544104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:06:44.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.463+0000 7f054b600700 1 --2- 172.21.15.62:0/2557218301 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f05440687a0 0x7f0544068b70 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:06:44.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.463+0000 7f054b600700 1 -- 172.21.15.62:0/2557218301 >> 172.21.15.62:0/2557218301 conn(0x7f0544075130 msgr2=0x7f0544075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:06:44.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.464+0000 7f054b600700 1 -- 172.21.15.62:0/2557218301 shutdown_connections 2024-04-20T22:06:44.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.464+0000 7f054b600700 1 -- 172.21.15.62:0/2557218301 wait complete. 2024-04-20T22:06:44.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.465+0000 7f054b600700 1 Processor -- start 2024-04-20T22:06:44.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.465+0000 7f054b600700 1 -- start start 2024-04-20T22:06:44.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.465+0000 7f054b600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f05440687a0 0x7f05441980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:06:44.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.465+0000 7f054b600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0544069140 0x7f05441985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:06:44.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.465+0000 7f054b600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0544198cd0 con 0x7f05440687a0 2024-04-20T22:06:44.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.465+0000 7f054b600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f054419ca60 con 0x7f0544069140 2024-04-20T22:06:44.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.466+0000 7f0548c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f05440687a0 0x7f05441980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:06:44.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.466+0000 7f0548c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f05440687a0 0x7f05441980b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34220/0 (socket says 172.21.15.62:34220) 2024-04-20T22:06:44.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.466+0000 7f0548c00700 1 -- 172.21.15.62:0/2015374573 learned_addr learned my addr 172.21.15.62:0/2015374573 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:06:44.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.466+0000 7f0548c00700 1 -- 172.21.15.62:0/2015374573 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0544069140 msgr2=0x7f05441985f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:06:44.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.466+0000 7f0548c00700 1 --2- 172.21.15.62:0/2015374573 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0544069140 0x7f05441985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:06:44.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.466+0000 7f0548c00700 1 -- 172.21.15.62:0/2015374573 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05380095f0 con 0x7f05440687a0 2024-04-20T22:06:44.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.467+0000 7f0548c00700 1 --2- 172.21.15.62:0/2015374573 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f05440687a0 0x7f05441980b0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f053800b5c0 tx=0x7f053800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:06:44.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.467+0000 7f0541600700 1 -- 172.21.15.62:0/2015374573 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f053801d070 con 0x7f05440687a0 2024-04-20T22:06:44.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.467+0000 7f0541600700 1 -- 172.21.15.62:0/2015374573 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0538005e00 con 0x7f05440687a0 2024-04-20T22:06:44.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.467+0000 7f054b600700 1 -- 172.21.15.62:0/2015374573 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f054419cce0 con 0x7f05440687a0 2024-04-20T22:06:44.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.467+0000 7f0541600700 1 -- 172.21.15.62:0/2015374573 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0538017400 con 0x7f05440687a0 2024-04-20T22:06:44.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.467+0000 7f054b600700 1 -- 172.21.15.62:0/2015374573 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f054419d1d0 con 0x7f05440687a0 2024-04-20T22:06:44.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.468+0000 7f054b600700 1 -- 172.21.15.62:0/2015374573 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f054404f2a0 con 0x7f05440687a0 2024-04-20T22:06:44.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.469+0000 7f0541600700 1 -- 172.21.15.62:0/2015374573 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f053800f460 con 0x7f05440687a0 2024-04-20T22:06:44.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.470+0000 7f0541600700 1 --2- 172.21.15.62:0/2015374573 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f052c06ecc0 0x7f052c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:06:44.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.471+0000 7f0541600700 1 -- 172.21.15.62:0/2015374573 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f0538090460 con 0x7f05440687a0 2024-04-20T22:06:44.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.471+0000 7f0543e00700 1 --2- 172.21.15.62:0/2015374573 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f052c06ecc0 0x7f052c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:06:44.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.472+0000 7f0543e00700 1 --2- 172.21.15.62:0/2015374573 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f052c06ecc0 0x7f052c071170 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f0530005950 tx=0x7f05300058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:06:44.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.475+0000 7f0541600700 1 -- 172.21.15.62:0/2015374573 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f0538059170 con 0x7f05440687a0 2024-04-20T22:06:44.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.720+0000 7f054b600700 1 -- 172.21.15.62:0/2015374573 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f054419d5e0 con 0x7f052c06ecc0 2024-04-20T22:06:44.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.739+0000 7f0541600700 1 -- 172.21.15.62:0/2015374573 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f054419d5e0 con 0x7f052c06ecc0 2024-04-20T22:06:44.739 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T22:06:44.739 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (32m) 8m ago 46m 15.3M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T22:06:44.739 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (46m) 8m ago 46m 10.4M - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T22:06:44.739 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (45m) 8m ago 45m 10.1M - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T22:06:44.739 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (28m) 8m ago 46m 6660k - 19.0.0-3133-ga8c84f59 76bc6b91b726 2825d5ba11c6 2024-04-20T22:06:44.739 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (28m) 8m ago 45m 6665k - 19.0.0-3133-ga8c84f59 76bc6b91b726 20f746fa1bba 2024-04-20T22:06:44.739 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (31m) 8m ago 46m 73.5M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T22:06:44.739 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (24m) 8m ago 39m 18.5M - 19.0.0-3133-ga8c84f59 76bc6b91b726 de80bc90a683 2024-04-20T22:06:44.739 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (24m) 8m ago 39m 18.7M - 19.0.0-3133-ga8c84f59 76bc6b91b726 cd793b7ee98e 2024-04-20T22:06:44.739 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (8m) 8m ago 39m 11.0M - 19.0.0-3133-ga8c84f59 76bc6b91b726 0e7b0bc4d1ab 2024-04-20T22:06:44.739 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (39m) 8m ago 39m 18.6M - 18.2.0 dc2bc1663786 f2eec2ee8951 2024-04-20T22:06:44.739 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (35m) 8m ago 48m 513M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T22:06:44.739 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (34m) 8m ago 45m 459M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T22:06:44.740 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (30m) 8m ago 48m 71.7M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 019eb1479485 2024-04-20T22:06:44.740 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (29m) 8m ago 45m 51.0M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 55aac560aab6 2024-04-20T22:06:44.740 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (33m) 8m ago 46m 8472k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T22:06:44.740 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (33m) 8m ago 45m 8048k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T22:06:44.740 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (28m) 8m ago 44m 91.2M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 ee2bb901f26e 2024-04-20T22:06:44.740 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (27m) 8m ago 43m 94.6M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 9075152d4162 2024-04-20T22:06:44.740 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (26m) 8m ago 42m 78.1M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 2d4f3e38dc0e 2024-04-20T22:06:44.740 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (26m) 8m ago 42m 92.6M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 8f97b17ae0eb 2024-04-20T22:06:44.740 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (25m) 8m ago 41m 95.6M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 653d8079967d 2024-04-20T22:06:44.740 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (24m) 8m ago 41m 89.2M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 596dded14c12 2024-04-20T22:06:44.741 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (33m) 8m ago 45m 82.1M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T22:06:44.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.741+0000 7f054b600700 1 -- 172.21.15.62:0/2015374573 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f052c06ecc0 msgr2=0x7f052c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:06:44.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.741+0000 7f054b600700 1 --2- 172.21.15.62:0/2015374573 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f052c06ecc0 0x7f052c071170 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f0530005950 tx=0x7f05300058e0 comp rx=0 tx=0).stop 2024-04-20T22:06:44.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.741+0000 7f054b600700 1 -- 172.21.15.62:0/2015374573 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f05440687a0 msgr2=0x7f05441980b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:06:44.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.741+0000 7f054b600700 1 --2- 172.21.15.62:0/2015374573 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f05440687a0 0x7f05441980b0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f053800b5c0 tx=0x7f053800be20 comp rx=0 tx=0).stop 2024-04-20T22:06:44.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.742+0000 7f054b600700 1 -- 172.21.15.62:0/2015374573 shutdown_connections 2024-04-20T22:06:44.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.742+0000 7f054b600700 1 --2- 172.21.15.62:0/2015374573 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f052c06ecc0 0x7f052c071170 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:06:44.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.742+0000 7f054b600700 1 --2- 172.21.15.62:0/2015374573 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f05440687a0 0x7f05441980b0 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:06:44.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.742+0000 7f054b600700 1 --2- 172.21.15.62:0/2015374573 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0544069140 0x7f05441985f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:06:44.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.742+0000 7f054b600700 1 -- 172.21.15.62:0/2015374573 >> 172.21.15.62:0/2015374573 conn(0x7f0544075130 msgr2=0x7f0544101e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:06:44.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.742+0000 7f054b600700 1 -- 172.21.15.62:0/2015374573 shutdown_connections 2024-04-20T22:06:44.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.742+0000 7f054b600700 1 -- 172.21.15.62:0/2015374573 wait complete. 2024-04-20T22:06:44.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.892+0000 7f7b17600700 1 -- 172.21.15.62:0/3553874355 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7b10106b80 msgr2=0x7f7b10106ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:06:44.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.892+0000 7f7b17600700 1 --2- 172.21.15.62:0/3553874355 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7b10106b80 0x7f7b10106ff0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f7b00009910 tx=0x7f7b00009c20 comp rx=0 tx=0).stop 2024-04-20T22:06:44.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.892+0000 7f7b17600700 1 -- 172.21.15.62:0/3553874355 shutdown_connections 2024-04-20T22:06:44.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.892+0000 7f7b17600700 1 --2- 172.21.15.62:0/3553874355 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7b10106b80 0x7f7b10106ff0 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:06:44.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.892+0000 7f7b17600700 1 --2- 172.21.15.62:0/3553874355 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7b1010c8f0 0x7f7b1010ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:06:44.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.892+0000 7f7b17600700 1 -- 172.21.15.62:0/3553874355 >> 172.21.15.62:0/3553874355 conn(0x7f7b10076060 msgr2=0x7f7b10078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:06:44.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.892+0000 7f7b17600700 1 -- 172.21.15.62:0/3553874355 shutdown_connections 2024-04-20T22:06:44.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.892+0000 7f7b17600700 1 -- 172.21.15.62:0/3553874355 wait complete. 2024-04-20T22:06:44.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.893+0000 7f7b17600700 1 Processor -- start 2024-04-20T22:06:44.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.893+0000 7f7b17600700 1 -- start start 2024-04-20T22:06:44.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.894+0000 7f7b17600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7b10106b80 0x7f7b1019c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:06:44.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.894+0000 7f7b17600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7b1010c8f0 0x7f7b1019c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:06:44.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.894+0000 7f7b17600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b1019cfd0 con 0x7f7b10106b80 2024-04-20T22:06:44.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.894+0000 7f7b17600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b1019d110 con 0x7f7b1010c8f0 2024-04-20T22:06:44.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.894+0000 7f7b14c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7b10106b80 0x7f7b1019c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:06:44.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.894+0000 7f7b14c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7b10106b80 0x7f7b1019c440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34232/0 (socket says 172.21.15.62:34232) 2024-04-20T22:06:44.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.894+0000 7f7b14c00700 1 -- 172.21.15.62:0/1634672374 learned_addr learned my addr 172.21.15.62:0/1634672374 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:06:44.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.894+0000 7f7b07e00700 1 --2- 172.21.15.62:0/1634672374 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7b1010c8f0 0x7f7b1019c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:06:44.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.894+0000 7f7b14c00700 1 -- 172.21.15.62:0/1634672374 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7b1010c8f0 msgr2=0x7f7b1019c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:06:44.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.894+0000 7f7b14c00700 1 --2- 172.21.15.62:0/1634672374 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7b1010c8f0 0x7f7b1019c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:06:44.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.894+0000 7f7b14c00700 1 -- 172.21.15.62:0/1634672374 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b000095f0 con 0x7f7b10106b80 2024-04-20T22:06:44.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.895+0000 7f7b14c00700 1 --2- 172.21.15.62:0/1634672374 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7b10106b80 0x7f7b1019c440 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f7af800c930 tx=0x7f7af800ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:06:44.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.895+0000 7f7b0e000700 1 -- 172.21.15.62:0/1634672374 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7af8007ad0 con 0x7f7b10106b80 2024-04-20T22:06:44.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.895+0000 7f7b0e000700 1 -- 172.21.15.62:0/1634672374 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7af8007c30 con 0x7f7b10106b80 2024-04-20T22:06:44.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.895+0000 7f7b0e000700 1 -- 172.21.15.62:0/1634672374 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7af8018610 con 0x7f7b10106b80 2024-04-20T22:06:44.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.895+0000 7f7b17600700 1 -- 172.21.15.62:0/1634672374 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b101a0f60 con 0x7f7b10106b80 2024-04-20T22:06:44.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.895+0000 7f7b17600700 1 -- 172.21.15.62:0/1634672374 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b101a1460 con 0x7f7b10106b80 2024-04-20T22:06:44.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.896+0000 7f7b17600700 1 -- 172.21.15.62:0/1634672374 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b1010ecc0 con 0x7f7b10106b80 2024-04-20T22:06:44.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.897+0000 7f7b0e000700 1 -- 172.21.15.62:0/1634672374 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f7af800f450 con 0x7f7b10106b80 2024-04-20T22:06:44.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.898+0000 7f7b0e000700 1 --2- 172.21.15.62:0/1634672374 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7af006ecc0 0x7f7af0071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:06:44.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.898+0000 7f7b0e000700 1 -- 172.21.15.62:0/1634672374 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f7af808eff0 con 0x7f7b10106b80 2024-04-20T22:06:44.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.898+0000 7f7b07e00700 1 --2- 172.21.15.62:0/1634672374 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7af006ecc0 0x7f7af0071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:06:44.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.899+0000 7f7b07e00700 1 --2- 172.21.15.62:0/1634672374 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7af006ecc0 0x7f7af0071170 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f7b1019da60 tx=0x7f7b0001a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:06:44.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:44.903+0000 7f7b0e000700 1 -- 172.21.15.62:0/1634672374 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f7af8057de0 con 0x7f7b10106b80 2024-04-20T22:06:45.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:06:44 smithi062 ceph-mon[76322]: pgmap v944: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:06:45.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:06:44 smithi062 ceph-mon[76322]: from='client.35684 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:06:45.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:45.221+0000 7f7b17600700 1 -- 172.21.15.62:0/1634672374 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f7b1010e6a0 con 0x7f7b10106b80 2024-04-20T22:06:45.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:45.222+0000 7f7b0e000700 1 -- 172.21.15.62:0/1634672374 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7f7af805b4a0 con 0x7f7b10106b80 2024-04-20T22:06:45.223 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:06:45.223 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T22:06:45.223 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:06:45.223 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:06:45.223 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T22:06:45.223 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:06:45.223 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:06:45.223 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T22:06:45.224 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 6 2024-04-20T22:06:45.224 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:06:45.224 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T22:06:45.224 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 1, 2024-04-20T22:06:45.224 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 3 2024-04-20T22:06:45.224 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:06:45.224 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T22:06:45.224 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 1, 2024-04-20T22:06:45.224 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 13 2024-04-20T22:06:45.225 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T22:06:45.225 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:06:45.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:45.225+0000 7f7b17600700 1 -- 172.21.15.62:0/1634672374 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7af006ecc0 msgr2=0x7f7af0071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:06:45.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:45.225+0000 7f7b17600700 1 --2- 172.21.15.62:0/1634672374 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7af006ecc0 0x7f7af0071170 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f7b1019da60 tx=0x7f7b0001a040 comp rx=0 tx=0).stop 2024-04-20T22:06:45.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:45.225+0000 7f7b17600700 1 -- 172.21.15.62:0/1634672374 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7b10106b80 msgr2=0x7f7b1019c440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:06:45.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:45.225+0000 7f7b17600700 1 --2- 172.21.15.62:0/1634672374 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7b10106b80 0x7f7b1019c440 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f7af800c930 tx=0x7f7af800ccf0 comp rx=0 tx=0).stop 2024-04-20T22:06:45.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:45.225+0000 7f7b17600700 1 -- 172.21.15.62:0/1634672374 shutdown_connections 2024-04-20T22:06:45.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:45.225+0000 7f7b17600700 1 --2- 172.21.15.62:0/1634672374 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7af006ecc0 0x7f7af0071170 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:06:45.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:45.225+0000 7f7b17600700 1 --2- 172.21.15.62:0/1634672374 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7b10106b80 0x7f7b1019c440 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:06:45.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:45.225+0000 7f7b17600700 1 --2- 172.21.15.62:0/1634672374 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7b1010c8f0 0x7f7b1019c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:06:45.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:45.225+0000 7f7b17600700 1 -- 172.21.15.62:0/1634672374 >> 172.21.15.62:0/1634672374 conn(0x7f7b10076060 msgr2=0x7f7b10077ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:06:45.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:45.225+0000 7f7b17600700 1 -- 172.21.15.62:0/1634672374 shutdown_connections 2024-04-20T22:06:45.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:45.225+0000 7f7b17600700 1 -- 172.21.15.62:0/1634672374 wait complete. 2024-04-20T22:06:45.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:06:44 smithi189 ceph-mon[57022]: pgmap v944: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:06:45.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:06:44 smithi189 ceph-mon[57022]: from='client.35684 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:06:45.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:45.390+0000 7f2cca200700 1 -- 172.21.15.62:0/3533693790 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2cc4102e70 msgr2=0x7f2cc4106e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:06:45.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:45.390+0000 7f2cca200700 1 --2- 172.21.15.62:0/3533693790 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2cc4102e70 0x7f2cc4106e40 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f2cb0009910 tx=0x7f2cb0009c20 comp rx=0 tx=0).stop 2024-04-20T22:06:45.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:45.390+0000 7f2cca200700 1 -- 172.21.15.62:0/3533693790 shutdown_connections 2024-04-20T22:06:45.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:45.390+0000 7f2cca200700 1 --2- 172.21.15.62:0/3533693790 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2cc4102e70 0x7f2cc4106e40 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:06:45.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:45.390+0000 7f2cca200700 1 --2- 172.21.15.62:0/3533693790 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2cc41024d0 0x7f2cc41028a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:06:45.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:45.390+0000 7f2cca200700 1 -- 172.21.15.62:0/3533693790 >> 172.21.15.62:0/3533693790 conn(0x7f2cc40fdd90 msgr2=0x7f2cc41001a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:06:45.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:45.390+0000 7f2cca200700 1 -- 172.21.15.62:0/3533693790 shutdown_connections 2024-04-20T22:06:45.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:45.390+0000 7f2cca200700 1 -- 172.21.15.62:0/3533693790 wait complete. 2024-04-20T22:06:45.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:45.391+0000 7f2cca200700 1 Processor -- start 2024-04-20T22:06:45.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:45.392+0000 7f2cca200700 1 -- start start 2024-04-20T22:06:45.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:45.392+0000 7f2cca200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2cc41024d0 0x7f2cc4197e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:06:45.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:45.392+0000 7f2cca200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2cc4102e70 0x7f2cc4198350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:06:45.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:45.392+0000 7f2cca200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc41989a0 con 0x7f2cc41024d0 2024-04-20T22:06:45.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:45.392+0000 7f2cca200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc4198ae0 con 0x7f2cc4102e70 2024-04-20T22:06:45.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:45.392+0000 7f2cc8e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2cc41024d0 0x7f2cc4197e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:06:45.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:45.392+0000 7f2cc8e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2cc41024d0 0x7f2cc4197e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34258/0 (socket says 172.21.15.62:34258) 2024-04-20T22:06:45.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:45.392+0000 7f2cc8e00700 1 -- 172.21.15.62:0/3145466806 learned_addr learned my addr 172.21.15.62:0/3145466806 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:06:45.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:45.393+0000 7f2cc8e00700 1 -- 172.21.15.62:0/3145466806 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2cc4102e70 msgr2=0x7f2cc4198350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:06:45.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:45.393+0000 7f2cc8e00700 1 --2- 172.21.15.62:0/3145466806 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2cc4102e70 0x7f2cc4198350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:06:45.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:45.393+0000 7f2cc8e00700 1 -- 172.21.15.62:0/3145466806 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cb00095f0 con 0x7f2cc41024d0 2024-04-20T22:06:45.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:45.393+0000 7f2cc8e00700 1 --2- 172.21.15.62:0/3145466806 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2cc41024d0 0x7f2cc4197e10 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f2cb800c930 tx=0x7f2cb800ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:06:45.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:45.393+0000 7f2cc1600700 1 -- 172.21.15.62:0/3145466806 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2cb8007ad0 con 0x7f2cc41024d0 2024-04-20T22:06:45.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:45.394+0000 7f2cc1600700 1 -- 172.21.15.62:0/3145466806 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2cb8007c30 con 0x7f2cc41024d0 2024-04-20T22:06:45.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:45.394+0000 7f2cc1600700 1 -- 172.21.15.62:0/3145466806 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2cb8018610 con 0x7f2cc41024d0 2024-04-20T22:06:45.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:45.394+0000 7f2cca200700 1 -- 172.21.15.62:0/3145466806 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cc419c930 con 0x7f2cc41024d0 2024-04-20T22:06:45.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:45.394+0000 7f2cca200700 1 -- 172.21.15.62:0/3145466806 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cc419ce80 con 0x7f2cc41024d0 2024-04-20T22:06:45.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:45.395+0000 7f2cca200700 1 -- 172.21.15.62:0/3145466806 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cc410a840 con 0x7f2cc41024d0 2024-04-20T22:06:45.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:45.396+0000 7f2cc1600700 1 -- 172.21.15.62:0/3145466806 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f2cb800f450 con 0x7f2cc41024d0 2024-04-20T22:06:45.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:45.398+0000 7f2cc1600700 1 --2- 172.21.15.62:0/3145466806 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2cac06ecc0 0x7f2cac071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:06:45.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:45.398+0000 7f2cc1600700 1 -- 172.21.15.62:0/3145466806 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f2cb808efb0 con 0x7f2cc41024d0 2024-04-20T22:06:45.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:45.398+0000 7f2cc3e00700 1 --2- 172.21.15.62:0/3145466806 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2cac06ecc0 0x7f2cac071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:06:45.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:45.399+0000 7f2cc3e00700 1 --2- 172.21.15.62:0/3145466806 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2cac06ecc0 0x7f2cac071170 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f2cb000b5c0 tx=0x7f2cb0005fd0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:06:45.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:45.403+0000 7f2cc1600700 1 -- 172.21.15.62:0/3145466806 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f2cb8057de0 con 0x7f2cc41024d0 2024-04-20T22:06:45.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:45.687+0000 7f2cca200700 1 -- 172.21.15.62:0/3145466806 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f2cc4062380 con 0x7f2cc41024d0 2024-04-20T22:06:45.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:45.688+0000 7f2cc1600700 1 -- 172.21.15.62:0/3145466806 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 27 v27) v1 ==== 76+0+1888 (secure 0 0 0) 0x7f2cb805b4a0 con 0x7f2cc41024d0 2024-04-20T22:06:45.689 INFO:teuthology.orchestra.run.smithi062.stdout:e27 2024-04-20T22:06:45.689 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T22:06:45.689 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:06:45.689 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T22:06:45.689 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:06:45.689 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T22:06:45.689 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T22:06:45.689 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 20 2024-04-20T22:06:45.689 INFO:teuthology.orchestra.run.smithi062.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-20T22:06:45.689 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T22:06:45.689 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T21:42:01.559061+0000 2024-04-20T22:06:45.689 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T22:06:45.689 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T22:06:45.690 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T22:06:45.690 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T22:06:45.690 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T22:06:45.690 INFO:teuthology.orchestra.run.smithi062.stdout:max_xattr_size 65536 2024-04-20T22:06:45.690 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T22:06:45.690 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T22:06:45.690 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 81 2024-04-20T22:06:45.690 INFO:teuthology.orchestra.run.smithi062.stdout:snap_rstat 1 2024-04-20T22:06:45.690 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:06:45.690 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 2 2024-04-20T22:06:45.690 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1 2024-04-20T22:06:45.690 INFO:teuthology.orchestra.run.smithi062.stdout:up {} 2024-04-20T22:06:45.690 INFO:teuthology.orchestra.run.smithi062.stdout:failed 0,1 2024-04-20T22:06:45.690 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T22:06:45.691 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T22:06:45.691 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T22:06:45.691 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T22:06:45.691 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data enabled 2024-04-20T22:06:45.691 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T22:06:45.691 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T22:06:45.691 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 1 2024-04-20T22:06:45.691 INFO:teuthology.orchestra.run.smithi062.stdout:qdb_cluster leader: 0 members: 2024-04-20T22:06:45.691 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:06:45.691 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:06:45.691 INFO:teuthology.orchestra.run.smithi062.stdout:Standby daemons: 2024-04-20T22:06:45.691 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:06:45.691 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ryqblb{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6826/4113916205,v1:172.21.15.62:6827/4113916205] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:06:45.691 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ozizwi{-1:34476} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6828/3801975165,v1:172.21.15.62:6829/3801975165] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:06:45.691 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.xbgspw{-1:44285} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6826/3996040526,v1:172.21.15.189:6827/3996040526] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T22:06:45.692 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.dvimer{-1:44727} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6824/3576699232,v1:172.21.15.189:6825/3576699232] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:06:45.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:45.691+0000 7f2cca200700 1 -- 172.21.15.62:0/3145466806 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2cac06ecc0 msgr2=0x7f2cac071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:06:45.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:45.691+0000 7f2cca200700 1 --2- 172.21.15.62:0/3145466806 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2cac06ecc0 0x7f2cac071170 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f2cb000b5c0 tx=0x7f2cb0005fd0 comp rx=0 tx=0).stop 2024-04-20T22:06:45.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:45.691+0000 7f2cca200700 1 -- 172.21.15.62:0/3145466806 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2cc41024d0 msgr2=0x7f2cc4197e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:06:45.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:45.691+0000 7f2cca200700 1 --2- 172.21.15.62:0/3145466806 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2cc41024d0 0x7f2cc4197e10 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f2cb800c930 tx=0x7f2cb800ccf0 comp rx=0 tx=0).stop 2024-04-20T22:06:45.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:45.691+0000 7f2cca200700 1 -- 172.21.15.62:0/3145466806 shutdown_connections 2024-04-20T22:06:45.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:45.691+0000 7f2cca200700 1 --2- 172.21.15.62:0/3145466806 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2cac06ecc0 0x7f2cac071170 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:06:45.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:45.691+0000 7f2cca200700 1 --2- 172.21.15.62:0/3145466806 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2cc41024d0 0x7f2cc4197e10 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:06:45.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:45.691+0000 7f2cca200700 1 --2- 172.21.15.62:0/3145466806 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2cc4102e70 0x7f2cc4198350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:06:45.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:45.691+0000 7f2cca200700 1 -- 172.21.15.62:0/3145466806 >> 172.21.15.62:0/3145466806 conn(0x7f2cc40fdd90 msgr2=0x7f2cc4100180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:06:45.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:45.692+0000 7f2cca200700 1 -- 172.21.15.62:0/3145466806 shutdown_connections 2024-04-20T22:06:45.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:45.692+0000 7f2cca200700 1 -- 172.21.15.62:0/3145466806 wait complete. 2024-04-20T22:06:45.693 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 27 2024-04-20T22:06:45.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:45.849+0000 7f01a5800700 1 -- 172.21.15.62:0/2843396180 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f01a010c8f0 msgr2=0x7f01a010ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:06:45.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:45.849+0000 7f01a5800700 1 --2- 172.21.15.62:0/2843396180 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f01a010c8f0 0x7f01a010ccc0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f0190009910 tx=0x7f0190009c20 comp rx=0 tx=0).stop 2024-04-20T22:06:45.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:45.849+0000 7f01a5800700 1 -- 172.21.15.62:0/2843396180 shutdown_connections 2024-04-20T22:06:45.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:45.849+0000 7f01a5800700 1 --2- 172.21.15.62:0/2843396180 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f01a0106b80 0x7f01a0106ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:06:45.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:45.849+0000 7f01a5800700 1 --2- 172.21.15.62:0/2843396180 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f01a010c8f0 0x7f01a010ccc0 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:06:45.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:45.849+0000 7f01a5800700 1 -- 172.21.15.62:0/2843396180 >> 172.21.15.62:0/2843396180 conn(0x7f01a0076060 msgr2=0x7f01a0078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:06:45.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:45.849+0000 7f01a5800700 1 -- 172.21.15.62:0/2843396180 shutdown_connections 2024-04-20T22:06:45.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:45.849+0000 7f01a5800700 1 -- 172.21.15.62:0/2843396180 wait complete. 2024-04-20T22:06:45.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:45.850+0000 7f01a5800700 1 Processor -- start 2024-04-20T22:06:45.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:45.850+0000 7f01a5800700 1 -- start start 2024-04-20T22:06:45.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:45.851+0000 7f01a5800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f01a0106b80 0x7f01a019c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:06:45.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:45.851+0000 7f01a5800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f01a010c8f0 0x7f01a019ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:06:45.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:45.851+0000 7f01a5800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01a019d100 con 0x7f01a0106b80 2024-04-20T22:06:45.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:45.851+0000 7f01a5800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01a01a0e90 con 0x7f01a010c8f0 2024-04-20T22:06:45.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:45.851+0000 7f019ea00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f01a0106b80 0x7f01a019c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:06:45.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:45.851+0000 7f019ea00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f01a0106b80 0x7f01a019c4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34268/0 (socket says 172.21.15.62:34268) 2024-04-20T22:06:45.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:45.851+0000 7f019ea00700 1 -- 172.21.15.62:0/918670837 learned_addr learned my addr 172.21.15.62:0/918670837 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:06:45.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:45.851+0000 7f019ea00700 1 -- 172.21.15.62:0/918670837 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f01a010c8f0 msgr2=0x7f01a019ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:06:45.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:45.851+0000 7f019ea00700 1 --2- 172.21.15.62:0/918670837 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f01a010c8f0 0x7f01a019ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:06:45.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:45.852+0000 7f019ea00700 1 -- 172.21.15.62:0/918670837 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01900095f0 con 0x7f01a0106b80 2024-04-20T22:06:45.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:45.852+0000 7f019ea00700 1 --2- 172.21.15.62:0/918670837 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f01a0106b80 0x7f01a019c4e0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f0190006010 tx=0x7f019000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:06:45.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:45.852+0000 7f0197400700 1 -- 172.21.15.62:0/918670837 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f019001d070 con 0x7f01a0106b80 2024-04-20T22:06:45.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:45.852+0000 7f0197400700 1 -- 172.21.15.62:0/918670837 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0190004fb0 con 0x7f01a0106b80 2024-04-20T22:06:45.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:45.852+0000 7f0197400700 1 -- 172.21.15.62:0/918670837 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f019000f690 con 0x7f01a0106b80 2024-04-20T22:06:45.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:45.852+0000 7f01a5800700 1 -- 172.21.15.62:0/918670837 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01a01a1110 con 0x7f01a0106b80 2024-04-20T22:06:45.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:45.852+0000 7f01a5800700 1 -- 172.21.15.62:0/918670837 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01a01a1600 con 0x7f01a0106b80 2024-04-20T22:06:45.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:45.853+0000 7f01a5800700 1 -- 172.21.15.62:0/918670837 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01a0109b20 con 0x7f01a0106b80 2024-04-20T22:06:45.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:45.854+0000 7f0197400700 1 -- 172.21.15.62:0/918670837 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f01900228a0 con 0x7f01a0106b80 2024-04-20T22:06:45.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:45.855+0000 7f0197400700 1 --2- 172.21.15.62:0/918670837 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f018406ecc0 0x7f0184071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:06:45.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:45.855+0000 7f0197400700 1 -- 172.21.15.62:0/918670837 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f0190090700 con 0x7f01a0106b80 2024-04-20T22:06:45.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:45.856+0000 7f019e000700 1 --2- 172.21.15.62:0/918670837 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f018406ecc0 0x7f0184071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:06:45.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:45.856+0000 7f019e000700 1 --2- 172.21.15.62:0/918670837 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f018406ecc0 0x7f0184071170 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f01a019db00 tx=0x7f018c006d20 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:06:45.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:45.860+0000 7f0197400700 1 -- 172.21.15.62:0/918670837 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f0190059490 con 0x7f01a0106b80 2024-04-20T22:06:46.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:06:45 smithi062 ceph-mon[76322]: from='client.35688 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:06:46.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:06:45 smithi062 ceph-mon[76322]: from='client.35692 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:06:46.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:06:45 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/1634672374' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:06:46.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:06:45 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/3145466806' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:06:46.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:46.113+0000 7f01a5800700 1 -- 172.21.15.62:0/918670837 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f01a002ccd0 con 0x7f018406ecc0 2024-04-20T22:06:46.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:46.116+0000 7f0197400700 1 -- 172.21.15.62:0/918670837 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f01a002ccd0 con 0x7f018406ecc0 2024-04-20T22:06:46.116 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:06:46.116 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T22:06:46.116 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T22:06:46.116 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-20T22:06:46.116 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T22:06:46.116 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr", 2024-04-20T22:06:46.116 INFO:teuthology.orchestra.run.smithi062.stdout: "crash", 2024-04-20T22:06:46.116 INFO:teuthology.orchestra.run.smithi062.stdout: "mon", 2024-04-20T22:06:46.116 INFO:teuthology.orchestra.run.smithi062.stdout: "osd" 2024-04-20T22:06:46.116 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T22:06:46.116 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "15/23 daemons upgraded", 2024-04-20T22:06:46.116 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading mds daemons", 2024-04-20T22:06:46.117 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T22:06:46.117 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:06:46.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:46.119+0000 7f01a5800700 1 -- 172.21.15.62:0/918670837 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f018406ecc0 msgr2=0x7f0184071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:06:46.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:46.119+0000 7f01a5800700 1 --2- 172.21.15.62:0/918670837 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f018406ecc0 0x7f0184071170 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f01a019db00 tx=0x7f018c006d20 comp rx=0 tx=0).stop 2024-04-20T22:06:46.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:46.119+0000 7f01a5800700 1 -- 172.21.15.62:0/918670837 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f01a0106b80 msgr2=0x7f01a019c4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:06:46.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:46.119+0000 7f01a5800700 1 --2- 172.21.15.62:0/918670837 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f01a0106b80 0x7f01a019c4e0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f0190006010 tx=0x7f019000be20 comp rx=0 tx=0).stop 2024-04-20T22:06:46.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:46.119+0000 7f01a5800700 1 -- 172.21.15.62:0/918670837 shutdown_connections 2024-04-20T22:06:46.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:46.119+0000 7f01a5800700 1 --2- 172.21.15.62:0/918670837 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f018406ecc0 0x7f0184071170 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:06:46.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:46.119+0000 7f01a5800700 1 --2- 172.21.15.62:0/918670837 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f01a0106b80 0x7f01a019c4e0 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:06:46.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:46.119+0000 7f01a5800700 1 --2- 172.21.15.62:0/918670837 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f01a010c8f0 0x7f01a019ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:06:46.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:46.119+0000 7f01a5800700 1 -- 172.21.15.62:0/918670837 >> 172.21.15.62:0/918670837 conn(0x7f01a0076060 msgr2=0x7f01a0077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:06:46.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:46.119+0000 7f01a5800700 1 -- 172.21.15.62:0/918670837 shutdown_connections 2024-04-20T22:06:46.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:46.120+0000 7f01a5800700 1 -- 172.21.15.62:0/918670837 wait complete. 2024-04-20T22:06:46.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:06:45 smithi189 ceph-mon[57022]: from='client.35688 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:06:46.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:06:45 smithi189 ceph-mon[57022]: from='client.35692 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:06:46.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:06:45 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/1634672374' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:06:46.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:06:45 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/3145466806' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:06:46.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:46.279+0000 7ff030a00700 1 -- 172.21.15.62:0/2139215366 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff02c073070 msgr2=0x7ff02c073440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:06:46.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:46.279+0000 7ff030a00700 1 --2- 172.21.15.62:0/2139215366 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff02c073070 0x7ff02c073440 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7ff01c009940 tx=0x7ff01c009c50 comp rx=0 tx=0).stop 2024-04-20T22:06:46.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:46.279+0000 7ff030a00700 1 -- 172.21.15.62:0/2139215366 shutdown_connections 2024-04-20T22:06:46.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:46.279+0000 7ff030a00700 1 --2- 172.21.15.62:0/2139215366 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff02c073980 0x7ff02c10c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:06:46.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:46.279+0000 7ff030a00700 1 --2- 172.21.15.62:0/2139215366 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff02c073070 0x7ff02c073440 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:06:46.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:46.279+0000 7ff030a00700 1 -- 172.21.15.62:0/2139215366 >> 172.21.15.62:0/2139215366 conn(0x7ff02c0fbda0 msgr2=0x7ff02c0fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:06:46.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:46.279+0000 7ff030a00700 1 -- 172.21.15.62:0/2139215366 shutdown_connections 2024-04-20T22:06:46.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:46.279+0000 7ff030a00700 1 -- 172.21.15.62:0/2139215366 wait complete. 2024-04-20T22:06:46.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:46.280+0000 7ff030a00700 1 Processor -- start 2024-04-20T22:06:46.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:46.280+0000 7ff030a00700 1 -- start start 2024-04-20T22:06:46.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:46.281+0000 7ff030a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff02c073070 0x7ff02c198160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:06:46.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:46.281+0000 7ff030a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff02c073980 0x7ff02c1986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:06:46.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:46.281+0000 7ff030a00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff02c198d80 con 0x7ff02c073980 2024-04-20T22:06:46.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:46.281+0000 7ff030a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff02c19cb10 con 0x7ff02c073070 2024-04-20T22:06:46.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:46.281+0000 7ff023e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff02c073980 0x7ff02c1986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:06:46.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:46.281+0000 7ff023e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff02c073980 0x7ff02c1986a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34286/0 (socket says 172.21.15.62:34286) 2024-04-20T22:06:46.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:46.281+0000 7ff023e00700 1 -- 172.21.15.62:0/2823385132 learned_addr learned my addr 172.21.15.62:0/2823385132 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:06:46.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:46.281+0000 7ff029e00700 1 --2- 172.21.15.62:0/2823385132 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff02c073070 0x7ff02c198160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:06:46.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:46.282+0000 7ff023e00700 1 -- 172.21.15.62:0/2823385132 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff02c073070 msgr2=0x7ff02c198160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:06:46.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:46.282+0000 7ff023e00700 1 --2- 172.21.15.62:0/2823385132 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff02c073070 0x7ff02c198160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:06:46.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:46.282+0000 7ff023e00700 1 -- 172.21.15.62:0/2823385132 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff01c0095f0 con 0x7ff02c073980 2024-04-20T22:06:46.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:46.282+0000 7ff023e00700 1 --2- 172.21.15.62:0/2823385132 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff02c073980 0x7ff02c1986a0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7ff01400d900 tx=0x7ff01400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:06:46.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:46.282+0000 7ff022a00700 1 -- 172.21.15.62:0/2823385132 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff014004a00 con 0x7ff02c073980 2024-04-20T22:06:46.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:46.282+0000 7ff022a00700 1 -- 172.21.15.62:0/2823385132 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff0140054e0 con 0x7ff02c073980 2024-04-20T22:06:46.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:46.282+0000 7ff022a00700 1 -- 172.21.15.62:0/2823385132 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff014009d60 con 0x7ff02c073980 2024-04-20T22:06:46.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:46.282+0000 7ff030a00700 1 -- 172.21.15.62:0/2823385132 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff02c19cdf0 con 0x7ff02c073980 2024-04-20T22:06:46.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:46.282+0000 7ff030a00700 1 -- 172.21.15.62:0/2823385132 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff02c19d340 con 0x7ff02c073980 2024-04-20T22:06:46.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:46.283+0000 7ff030a00700 1 -- 172.21.15.62:0/2823385132 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff02c109d00 con 0x7ff02c073980 2024-04-20T22:06:46.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:46.285+0000 7ff022a00700 1 -- 172.21.15.62:0/2823385132 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff014005020 con 0x7ff02c073980 2024-04-20T22:06:46.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:46.285+0000 7ff022a00700 1 --2- 172.21.15.62:0/2823385132 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff01006ecc0 0x7ff010071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:06:46.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:46.286+0000 7ff022a00700 1 -- 172.21.15.62:0/2823385132 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7ff01408efe0 con 0x7ff02c073980 2024-04-20T22:06:46.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:46.286+0000 7ff029e00700 1 --2- 172.21.15.62:0/2823385132 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff01006ecc0 0x7ff010071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:06:46.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:46.286+0000 7ff029e00700 1 --2- 172.21.15.62:0/2823385132 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff01006ecc0 0x7ff010071170 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7ff01c005b40 tx=0x7ff01c005ab0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:06:46.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:46.290+0000 7ff022a00700 1 -- 172.21.15.62:0/2823385132 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7ff0140c3ff0 con 0x7ff02c073980 2024-04-20T22:06:46.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:46.601+0000 7ff030a00700 1 -- 172.21.15.62:0/2823385132 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7ff02c04f2a0 con 0x7ff02c073980 2024-04-20T22:06:46.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:46.602+0000 7ff022a00700 1 -- 172.21.15.62:0/2823385132 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7ff014017020 con 0x7ff02c073980 2024-04-20T22:06:46.602 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-20T22:06:46.603 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-20T22:06:46.603 INFO:teuthology.orchestra.run.smithi062.stdout: host smithi062 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T22:06:46.603 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-20T22:06:46.603 INFO:teuthology.orchestra.run.smithi062.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi062 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpq7y7oa_r:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:06:46.603 INFO:teuthology.orchestra.run.smithi062.stdout:Traceback (most recent call last): 2024-04-20T22:06:46.603 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-20T22:06:46.603 INFO:teuthology.orchestra.run.smithi062.stdout: return _run_code(code, main_globals, None, 2024-04-20T22:06:46.603 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-20T22:06:46.603 INFO:teuthology.orchestra.run.smithi062.stdout: exec(code, run_globals) 2024-04-20T22:06:46.603 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5618, in 2024-04-20T22:06:46.603 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5606, in main 2024-04-20T22:06:46.603 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 408, in _infer_config 2024-04-20T22:06:46.603 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 323, in _infer_fsid 2024-04-20T22:06:46.603 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 436, in _infer_image 2024-04-20T22:06:46.604 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 310, in _validate_fsid 2024-04-20T22:06:46.604 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 3359, in command_ceph_volume 2024-04-20T22:06:46.604 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-20T22:06:46.604 INFO:teuthology.orchestra.run.smithi062.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi062 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpq7y7oa_r:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:06:46.604 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-20T22:06:46.604 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is degraded 2024-04-20T22:06:46.604 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T22:06:46.604 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T22:06:46.604 INFO:teuthology.orchestra.run.smithi062.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-20T22:06:46.604 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-20T22:06:46.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:46.605+0000 7ff030a00700 1 -- 172.21.15.62:0/2823385132 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff01006ecc0 msgr2=0x7ff010071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:06:46.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:46.605+0000 7ff030a00700 1 --2- 172.21.15.62:0/2823385132 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff01006ecc0 0x7ff010071170 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7ff01c005b40 tx=0x7ff01c005ab0 comp rx=0 tx=0).stop 2024-04-20T22:06:46.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:46.605+0000 7ff030a00700 1 -- 172.21.15.62:0/2823385132 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff02c073980 msgr2=0x7ff02c1986a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:06:46.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:46.605+0000 7ff030a00700 1 --2- 172.21.15.62:0/2823385132 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff02c073980 0x7ff02c1986a0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7ff01400d900 tx=0x7ff01400dcc0 comp rx=0 tx=0).stop 2024-04-20T22:06:46.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:46.605+0000 7ff030a00700 1 -- 172.21.15.62:0/2823385132 shutdown_connections 2024-04-20T22:06:46.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:46.605+0000 7ff030a00700 1 --2- 172.21.15.62:0/2823385132 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff01006ecc0 0x7ff010071170 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:06:46.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:46.605+0000 7ff030a00700 1 --2- 172.21.15.62:0/2823385132 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff02c073070 0x7ff02c198160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:06:46.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:46.605+0000 7ff030a00700 1 --2- 172.21.15.62:0/2823385132 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff02c073980 0x7ff02c1986a0 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:06:46.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:46.605+0000 7ff030a00700 1 -- 172.21.15.62:0/2823385132 >> 172.21.15.62:0/2823385132 conn(0x7ff02c0fbda0 msgr2=0x7ff02c106e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:06:46.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:46.605+0000 7ff030a00700 1 -- 172.21.15.62:0/2823385132 shutdown_connections 2024-04-20T22:06:46.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:06:46.605+0000 7ff030a00700 1 -- 172.21.15.62:0/2823385132 wait complete. 2024-04-20T22:06:46.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:06:46 smithi189 ceph-mon[57022]: pgmap v945: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:06:46.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:06:46 smithi189 ceph-mon[57022]: from='client.35704 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:06:46.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:06:46 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/2823385132' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:06:47.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:06:46 smithi062 ceph-mon[76322]: pgmap v945: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:06:47.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:06:46 smithi062 ceph-mon[76322]: from='client.35704 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:06:47.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:06:46 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/2823385132' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:06:49.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:06:48 smithi189 ceph-mon[57022]: pgmap v946: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:06:49.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:06:48 smithi062 ceph-mon[76322]: pgmap v946: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:06:50.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:06:49 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:06:50.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:06:49 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:06:51.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:06:50 smithi189 ceph-mon[57022]: pgmap v947: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:06:51.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:06:50 smithi062 ceph-mon[76322]: pgmap v947: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:06:53.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:06:52 smithi189 ceph-mon[57022]: pgmap v948: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:06:53.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:06:52 smithi062 ceph-mon[76322]: pgmap v948: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:06:55.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:06:54 smithi062 ceph-mon[76322]: pgmap v949: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:06:55.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:06:54 smithi189 ceph-mon[57022]: pgmap v949: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:06:57.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:06:56 smithi189 ceph-mon[57022]: pgmap v950: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:06:57.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:06:56 smithi062 ceph-mon[76322]: pgmap v950: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:06:59.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:06:58 smithi189 ceph-mon[57022]: pgmap v951: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:06:59.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:06:58 smithi062 ceph-mon[76322]: pgmap v951: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:07:01.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:07:00 smithi189 ceph-mon[57022]: pgmap v952: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:07:01.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:07:00 smithi062 ceph-mon[76322]: pgmap v952: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:07:03.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:07:02 smithi189 ceph-mon[57022]: pgmap v953: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:07:03.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:07:02 smithi062 ceph-mon[76322]: pgmap v953: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:07:05.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:07:04 smithi189 ceph-mon[57022]: pgmap v954: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:07:05.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:07:04 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:07:05.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:07:04 smithi062 ceph-mon[76322]: pgmap v954: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:07:05.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:07:04 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:07:07.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:07:06 smithi189 ceph-mon[57022]: pgmap v955: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:07:07.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:07:06 smithi062 ceph-mon[76322]: pgmap v955: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:07:09.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:07:08 smithi189 ceph-mon[57022]: pgmap v956: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:07:09.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:07:08 smithi062 ceph-mon[76322]: pgmap v956: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:07:11.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:07:10 smithi189 ceph-mon[57022]: pgmap v957: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:07:11.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:07:10 smithi062 ceph-mon[76322]: pgmap v957: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:07:13.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:07:12 smithi189 ceph-mon[57022]: pgmap v958: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:07:13.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:07:12 smithi062 ceph-mon[76322]: pgmap v958: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:07:15.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:07:14 smithi189 ceph-mon[57022]: pgmap v959: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:07:15.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:07:14 smithi062 ceph-mon[76322]: pgmap v959: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:07:16.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:16.783+0000 7fec86600700 1 -- 172.21.15.62:0/3967362790 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fec8010c8f0 msgr2=0x7fec8010ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:07:16.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:16.783+0000 7fec86600700 1 --2- 172.21.15.62:0/3967362790 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fec8010c8f0 0x7fec8010ccc0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fec70009910 tx=0x7fec70009c20 comp rx=0 tx=0).stop 2024-04-20T22:07:16.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:16.784+0000 7fec86600700 1 -- 172.21.15.62:0/3967362790 shutdown_connections 2024-04-20T22:07:16.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:16.784+0000 7fec86600700 1 --2- 172.21.15.62:0/3967362790 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec80106b80 0x7fec80106ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:07:16.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:16.784+0000 7fec86600700 1 --2- 172.21.15.62:0/3967362790 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fec8010c8f0 0x7fec8010ccc0 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:07:16.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:16.784+0000 7fec86600700 1 -- 172.21.15.62:0/3967362790 >> 172.21.15.62:0/3967362790 conn(0x7fec80076060 msgr2=0x7fec80078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:07:16.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:16.784+0000 7fec86600700 1 -- 172.21.15.62:0/3967362790 shutdown_connections 2024-04-20T22:07:16.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:16.784+0000 7fec86600700 1 -- 172.21.15.62:0/3967362790 wait complete. 2024-04-20T22:07:16.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:16.785+0000 7fec86600700 1 Processor -- start 2024-04-20T22:07:16.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:16.785+0000 7fec86600700 1 -- start start 2024-04-20T22:07:16.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:16.785+0000 7fec86600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fec80106b80 0x7fec8019c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:07:16.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:16.785+0000 7fec86600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec8010c8f0 0x7fec8019ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:07:16.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:16.786+0000 7fec86600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec8019d120 con 0x7fec80106b80 2024-04-20T22:07:16.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:16.786+0000 7fec86600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec801a0eb0 con 0x7fec8010c8f0 2024-04-20T22:07:16.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:16.786+0000 7fec7f400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fec80106b80 0x7fec8019c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:07:16.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:16.786+0000 7fec7f400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fec80106b80 0x7fec8019c500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40010/0 (socket says 172.21.15.62:40010) 2024-04-20T22:07:16.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:16.786+0000 7fec7f400700 1 -- 172.21.15.62:0/4060628319 learned_addr learned my addr 172.21.15.62:0/4060628319 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:07:16.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:16.786+0000 7fec7ea00700 1 --2- 172.21.15.62:0/4060628319 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec8010c8f0 0x7fec8019ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:07:16.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:16.786+0000 7fec7f400700 1 -- 172.21.15.62:0/4060628319 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec8010c8f0 msgr2=0x7fec8019ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:07:16.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:16.786+0000 7fec7f400700 1 --2- 172.21.15.62:0/4060628319 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec8010c8f0 0x7fec8019ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:07:16.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:16.786+0000 7fec7f400700 1 -- 172.21.15.62:0/4060628319 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec700095f0 con 0x7fec80106b80 2024-04-20T22:07:16.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:16.787+0000 7fec7f400700 1 --2- 172.21.15.62:0/4060628319 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fec80106b80 0x7fec8019c500 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7fec70006010 tx=0x7fec7000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:07:16.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:16.787+0000 7fec77e00700 1 -- 172.21.15.62:0/4060628319 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fec7001d070 con 0x7fec80106b80 2024-04-20T22:07:16.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:16.787+0000 7fec77e00700 1 -- 172.21.15.62:0/4060628319 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fec70004fb0 con 0x7fec80106b80 2024-04-20T22:07:16.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:16.787+0000 7fec86600700 1 -- 172.21.15.62:0/4060628319 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec801a1130 con 0x7fec80106b80 2024-04-20T22:07:16.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:16.787+0000 7fec77e00700 1 -- 172.21.15.62:0/4060628319 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fec70022ea0 con 0x7fec80106b80 2024-04-20T22:07:16.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:16.787+0000 7fec86600700 1 -- 172.21.15.62:0/4060628319 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec801a1620 con 0x7fec80106b80 2024-04-20T22:07:16.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:16.788+0000 7fec86600700 1 -- 172.21.15.62:0/4060628319 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec80109b20 con 0x7fec80106b80 2024-04-20T22:07:16.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:16.789+0000 7fec77e00700 1 -- 172.21.15.62:0/4060628319 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fec70022470 con 0x7fec80106b80 2024-04-20T22:07:16.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:16.791+0000 7fec77e00700 1 --2- 172.21.15.62:0/4060628319 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fec6406ecc0 0x7fec64071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:07:16.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:16.791+0000 7fec77e00700 1 -- 172.21.15.62:0/4060628319 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fec700903f0 con 0x7fec80106b80 2024-04-20T22:07:16.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:16.791+0000 7fec7ea00700 1 --2- 172.21.15.62:0/4060628319 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fec6406ecc0 0x7fec64071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:07:16.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:16.792+0000 7fec7ea00700 1 --2- 172.21.15.62:0/4060628319 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fec6406ecc0 0x7fec64071170 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fec6c005fd0 tx=0x7fec6c00a560 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:07:16.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:16.796+0000 7fec77e00700 1 -- 172.21.15.62:0/4060628319 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fec70059230 con 0x7fec80106b80 2024-04-20T22:07:17.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:07:16 smithi062 ceph-mon[76322]: pgmap v960: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:07:17.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.053+0000 7fec86600700 1 -- 172.21.15.62:0/4060628319 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fec801a1950 con 0x7fec6406ecc0 2024-04-20T22:07:17.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.056+0000 7fec77e00700 1 -- 172.21.15.62:0/4060628319 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fec801a1950 con 0x7fec6406ecc0 2024-04-20T22:07:17.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.059+0000 7fec86600700 1 -- 172.21.15.62:0/4060628319 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fec6406ecc0 msgr2=0x7fec64071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:07:17.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.059+0000 7fec86600700 1 --2- 172.21.15.62:0/4060628319 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fec6406ecc0 0x7fec64071170 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fec6c005fd0 tx=0x7fec6c00a560 comp rx=0 tx=0).stop 2024-04-20T22:07:17.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.059+0000 7fec86600700 1 -- 172.21.15.62:0/4060628319 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fec80106b80 msgr2=0x7fec8019c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:07:17.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.059+0000 7fec86600700 1 --2- 172.21.15.62:0/4060628319 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fec80106b80 0x7fec8019c500 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7fec70006010 tx=0x7fec7000be20 comp rx=0 tx=0).stop 2024-04-20T22:07:17.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.059+0000 7fec86600700 1 -- 172.21.15.62:0/4060628319 shutdown_connections 2024-04-20T22:07:17.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.059+0000 7fec86600700 1 --2- 172.21.15.62:0/4060628319 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fec6406ecc0 0x7fec64071170 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:07:17.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.059+0000 7fec86600700 1 --2- 172.21.15.62:0/4060628319 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fec80106b80 0x7fec8019c500 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:07:17.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.059+0000 7fec86600700 1 --2- 172.21.15.62:0/4060628319 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec8010c8f0 0x7fec8019ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:07:17.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.059+0000 7fec86600700 1 -- 172.21.15.62:0/4060628319 >> 172.21.15.62:0/4060628319 conn(0x7fec80076060 msgr2=0x7fec80077aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:07:17.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.060+0000 7fec86600700 1 -- 172.21.15.62:0/4060628319 shutdown_connections 2024-04-20T22:07:17.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.060+0000 7fec86600700 1 -- 172.21.15.62:0/4060628319 wait complete. 2024-04-20T22:07:17.078 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T22:07:17.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.225+0000 7fe4fc600700 1 -- 172.21.15.62:0/3290313983 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe4f410c8f0 msgr2=0x7fe4f410ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:07:17.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.225+0000 7fe4fc600700 1 --2- 172.21.15.62:0/3290313983 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe4f410c8f0 0x7fe4f410ccc0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fe4e8009910 tx=0x7fe4e8009c20 comp rx=0 tx=0).stop 2024-04-20T22:07:17.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.225+0000 7fe4fc600700 1 -- 172.21.15.62:0/3290313983 shutdown_connections 2024-04-20T22:07:17.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.225+0000 7fe4fc600700 1 --2- 172.21.15.62:0/3290313983 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe4f4106b80 0x7fe4f4106ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:07:17.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.225+0000 7fe4fc600700 1 --2- 172.21.15.62:0/3290313983 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe4f410c8f0 0x7fe4f410ccc0 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:07:17.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.225+0000 7fe4fc600700 1 -- 172.21.15.62:0/3290313983 >> 172.21.15.62:0/3290313983 conn(0x7fe4f4076020 msgr2=0x7fe4f4078430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:07:17.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.225+0000 7fe4fc600700 1 -- 172.21.15.62:0/3290313983 shutdown_connections 2024-04-20T22:07:17.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.225+0000 7fe4fc600700 1 -- 172.21.15.62:0/3290313983 wait complete. 2024-04-20T22:07:17.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.226+0000 7fe4fc600700 1 Processor -- start 2024-04-20T22:07:17.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.227+0000 7fe4fc600700 1 -- start start 2024-04-20T22:07:17.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.227+0000 7fe4fc600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe4f4106b80 0x7fe4f419c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:07:17.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.227+0000 7fe4fc600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe4f410c8f0 0x7fe4f419c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:07:17.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.227+0000 7fe4fc600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4f419cf60 con 0x7fe4f4106b80 2024-04-20T22:07:17.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.227+0000 7fe4fc600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4f419d0a0 con 0x7fe4f410c8f0 2024-04-20T22:07:17.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.228+0000 7fe4f9c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe4f4106b80 0x7fe4f419c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:07:17.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.228+0000 7fe4f9c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe4f4106b80 0x7fe4f419c3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40022/0 (socket says 172.21.15.62:40022) 2024-04-20T22:07:17.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.228+0000 7fe4f9c00700 1 -- 172.21.15.62:0/1581251143 learned_addr learned my addr 172.21.15.62:0/1581251143 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:07:17.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.228+0000 7fe4f9c00700 1 -- 172.21.15.62:0/1581251143 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe4f410c8f0 msgr2=0x7fe4f419c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:07:17.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.228+0000 7fe4f9c00700 1 --2- 172.21.15.62:0/1581251143 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe4f410c8f0 0x7fe4f419c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:07:17.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.228+0000 7fe4f9c00700 1 -- 172.21.15.62:0/1581251143 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4e80095f0 con 0x7fe4f4106b80 2024-04-20T22:07:17.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.229+0000 7fe4f9c00700 1 --2- 172.21.15.62:0/1581251143 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe4f4106b80 0x7fe4f419c3d0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fe4e800b5c0 tx=0x7fe4e800be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:07:17.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.229+0000 7fe4f2000700 1 -- 172.21.15.62:0/1581251143 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe4e801d070 con 0x7fe4f4106b80 2024-04-20T22:07:17.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.229+0000 7fe4f2000700 1 -- 172.21.15.62:0/1581251143 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe4e800f460 con 0x7fe4f4106b80 2024-04-20T22:07:17.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.229+0000 7fe4f2000700 1 -- 172.21.15.62:0/1581251143 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe4e8017610 con 0x7fe4f4106b80 2024-04-20T22:07:17.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.229+0000 7fe4fc600700 1 -- 172.21.15.62:0/1581251143 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4f41a0e90 con 0x7fe4f4106b80 2024-04-20T22:07:17.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.229+0000 7fe4fc600700 1 -- 172.21.15.62:0/1581251143 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4f41a1380 con 0x7fe4f4106b80 2024-04-20T22:07:17.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.230+0000 7fe4fc600700 1 -- 172.21.15.62:0/1581251143 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4f404f2a0 con 0x7fe4f4106b80 2024-04-20T22:07:17.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.231+0000 7fe4f2000700 1 -- 172.21.15.62:0/1581251143 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe4e800f5d0 con 0x7fe4f4106b80 2024-04-20T22:07:17.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.232+0000 7fe4f2000700 1 --2- 172.21.15.62:0/1581251143 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe4dc06ecc0 0x7fe4dc071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:07:17.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.232+0000 7fe4f2000700 1 -- 172.21.15.62:0/1581251143 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fe4e8090450 con 0x7fe4f4106b80 2024-04-20T22:07:17.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.233+0000 7fe4f9200700 1 --2- 172.21.15.62:0/1581251143 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe4dc06ecc0 0x7fe4dc071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:07:17.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.233+0000 7fe4f9200700 1 --2- 172.21.15.62:0/1581251143 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe4dc06ecc0 0x7fe4dc071170 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fe4e0005fd0 tx=0x7fe4e0005d00 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:07:17.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.237+0000 7fe4f2000700 1 -- 172.21.15.62:0/1581251143 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fe4e80591e0 con 0x7fe4f4106b80 2024-04-20T22:07:17.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:07:16 smithi189 ceph-mon[57022]: pgmap v960: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:07:17.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.497+0000 7fe4fc600700 1 -- 172.21.15.62:0/1581251143 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe4f41a17f0 con 0x7fe4dc06ecc0 2024-04-20T22:07:17.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.500+0000 7fe4f2000700 1 -- 172.21.15.62:0/1581251143 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fe4f41a17f0 con 0x7fe4dc06ecc0 2024-04-20T22:07:17.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.502+0000 7fe4fc600700 1 -- 172.21.15.62:0/1581251143 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe4dc06ecc0 msgr2=0x7fe4dc071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:07:17.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.502+0000 7fe4fc600700 1 --2- 172.21.15.62:0/1581251143 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe4dc06ecc0 0x7fe4dc071170 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fe4e0005fd0 tx=0x7fe4e0005d00 comp rx=0 tx=0).stop 2024-04-20T22:07:17.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.502+0000 7fe4fc600700 1 -- 172.21.15.62:0/1581251143 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe4f4106b80 msgr2=0x7fe4f419c3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:07:17.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.502+0000 7fe4fc600700 1 --2- 172.21.15.62:0/1581251143 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe4f4106b80 0x7fe4f419c3d0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fe4e800b5c0 tx=0x7fe4e800be00 comp rx=0 tx=0).stop 2024-04-20T22:07:17.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.503+0000 7fe4fc600700 1 -- 172.21.15.62:0/1581251143 shutdown_connections 2024-04-20T22:07:17.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.503+0000 7fe4fc600700 1 --2- 172.21.15.62:0/1581251143 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe4dc06ecc0 0x7fe4dc071170 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:07:17.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.503+0000 7fe4fc600700 1 --2- 172.21.15.62:0/1581251143 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe4f4106b80 0x7fe4f419c3d0 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:07:17.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.503+0000 7fe4fc600700 1 --2- 172.21.15.62:0/1581251143 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe4f410c8f0 0x7fe4f419c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:07:17.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.503+0000 7fe4fc600700 1 -- 172.21.15.62:0/1581251143 >> 172.21.15.62:0/1581251143 conn(0x7fe4f4076020 msgr2=0x7fe4f4077940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:07:17.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.503+0000 7fe4fc600700 1 -- 172.21.15.62:0/1581251143 shutdown_connections 2024-04-20T22:07:17.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.503+0000 7fe4fc600700 1 -- 172.21.15.62:0/1581251143 wait complete. 2024-04-20T22:07:17.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.666+0000 7fc45ca00700 1 -- 172.21.15.62:0/1476567423 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc458106b80 msgr2=0x7fc458106ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:07:17.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.666+0000 7fc45ca00700 1 --2- 172.21.15.62:0/1476567423 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc458106b80 0x7fc458106ff0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fc444009910 tx=0x7fc444009c20 comp rx=0 tx=0).stop 2024-04-20T22:07:17.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.666+0000 7fc45ca00700 1 -- 172.21.15.62:0/1476567423 shutdown_connections 2024-04-20T22:07:17.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.666+0000 7fc45ca00700 1 --2- 172.21.15.62:0/1476567423 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc458106b80 0x7fc458106ff0 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:07:17.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.666+0000 7fc45ca00700 1 --2- 172.21.15.62:0/1476567423 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc45810c8f0 0x7fc45810ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:07:17.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.666+0000 7fc45ca00700 1 -- 172.21.15.62:0/1476567423 >> 172.21.15.62:0/1476567423 conn(0x7fc458076020 msgr2=0x7fc458078430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:07:17.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.666+0000 7fc45ca00700 1 -- 172.21.15.62:0/1476567423 shutdown_connections 2024-04-20T22:07:17.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.666+0000 7fc45ca00700 1 -- 172.21.15.62:0/1476567423 wait complete. 2024-04-20T22:07:17.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.667+0000 7fc45ca00700 1 Processor -- start 2024-04-20T22:07:17.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.667+0000 7fc45ca00700 1 -- start start 2024-04-20T22:07:17.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.668+0000 7fc45ca00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc458106b80 0x7fc45819c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:07:17.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.668+0000 7fc45ca00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc45810c8f0 0x7fc45819c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:07:17.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.668+0000 7fc45ca00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc45819d0a0 con 0x7fc45810c8f0 2024-04-20T22:07:17.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.668+0000 7fc45ca00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4581a0e30 con 0x7fc458106b80 2024-04-20T22:07:17.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.668+0000 7fc455400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc45810c8f0 0x7fc45819c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:07:17.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.668+0000 7fc455400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc45810c8f0 0x7fc45819c9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40044/0 (socket says 172.21.15.62:40044) 2024-04-20T22:07:17.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.668+0000 7fc455400700 1 -- 172.21.15.62:0/2286050214 learned_addr learned my addr 172.21.15.62:0/2286050214 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:07:17.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.669+0000 7fc455e00700 1 --2- 172.21.15.62:0/2286050214 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc458106b80 0x7fc45819c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:07:17.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.669+0000 7fc455400700 1 -- 172.21.15.62:0/2286050214 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc458106b80 msgr2=0x7fc45819c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:07:17.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.669+0000 7fc455400700 1 --2- 172.21.15.62:0/2286050214 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc458106b80 0x7fc45819c480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:07:17.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.669+0000 7fc455400700 1 -- 172.21.15.62:0/2286050214 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4440095f0 con 0x7fc45810c8f0 2024-04-20T22:07:17.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.669+0000 7fc455400700 1 --2- 172.21.15.62:0/2286050214 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc45810c8f0 0x7fc45819c9c0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fc444006010 tx=0x7fc44400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:07:17.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.669+0000 7fc44ea00700 1 -- 172.21.15.62:0/2286050214 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc44401d070 con 0x7fc45810c8f0 2024-04-20T22:07:17.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.669+0000 7fc44ea00700 1 -- 172.21.15.62:0/2286050214 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc44400f460 con 0x7fc45810c8f0 2024-04-20T22:07:17.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.669+0000 7fc44ea00700 1 -- 172.21.15.62:0/2286050214 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc444017630 con 0x7fc45810c8f0 2024-04-20T22:07:17.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.669+0000 7fc45ca00700 1 -- 172.21.15.62:0/2286050214 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4581a10b0 con 0x7fc45810c8f0 2024-04-20T22:07:17.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.669+0000 7fc45ca00700 1 -- 172.21.15.62:0/2286050214 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4581a1550 con 0x7fc45810c8f0 2024-04-20T22:07:17.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.670+0000 7fc45ca00700 1 -- 172.21.15.62:0/2286050214 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc45810ed60 con 0x7fc45810c8f0 2024-04-20T22:07:17.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.672+0000 7fc44ea00700 1 -- 172.21.15.62:0/2286050214 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fc44400f5d0 con 0x7fc45810c8f0 2024-04-20T22:07:17.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.673+0000 7fc44ea00700 1 --2- 172.21.15.62:0/2286050214 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc43c06ecc0 0x7fc43c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:07:17.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.673+0000 7fc44ea00700 1 -- 172.21.15.62:0/2286050214 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fc444090350 con 0x7fc45810c8f0 2024-04-20T22:07:17.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.673+0000 7fc455e00700 1 --2- 172.21.15.62:0/2286050214 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc43c06ecc0 0x7fc43c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:07:17.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.674+0000 7fc455e00700 1 --2- 172.21.15.62:0/2286050214 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc43c06ecc0 0x7fc43c071170 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fc448005950 tx=0x7fc4480058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:07:17.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.678+0000 7fc44ea00700 1 -- 172.21.15.62:0/2286050214 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fc4440590e0 con 0x7fc45810c8f0 2024-04-20T22:07:17.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.920+0000 7fc45ca00700 1 -- 172.21.15.62:0/2286050214 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fc45802cca0 con 0x7fc43c06ecc0 2024-04-20T22:07:17.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.933+0000 7fc44ea00700 1 -- 172.21.15.62:0/2286050214 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fc45802cca0 con 0x7fc43c06ecc0 2024-04-20T22:07:17.933 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T22:07:17.933 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (33m) 9m ago 46m 15.3M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T22:07:17.933 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (47m) 9m ago 47m 10.4M - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T22:07:17.933 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (45m) 9m ago 45m 10.1M - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T22:07:17.933 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (29m) 9m ago 47m 6660k - 19.0.0-3133-ga8c84f59 76bc6b91b726 2825d5ba11c6 2024-04-20T22:07:17.933 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (29m) 9m ago 45m 6665k - 19.0.0-3133-ga8c84f59 76bc6b91b726 20f746fa1bba 2024-04-20T22:07:17.934 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (32m) 9m ago 46m 73.5M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T22:07:17.934 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (24m) 9m ago 39m 18.5M - 19.0.0-3133-ga8c84f59 76bc6b91b726 de80bc90a683 2024-04-20T22:07:17.934 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (25m) 9m ago 39m 18.7M - 19.0.0-3133-ga8c84f59 76bc6b91b726 cd793b7ee98e 2024-04-20T22:07:17.934 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (9m) 9m ago 39m 11.0M - 19.0.0-3133-ga8c84f59 76bc6b91b726 0e7b0bc4d1ab 2024-04-20T22:07:17.934 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (39m) 9m ago 39m 18.6M - 18.2.0 dc2bc1663786 f2eec2ee8951 2024-04-20T22:07:17.934 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (36m) 9m ago 48m 513M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T22:07:17.934 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (34m) 9m ago 45m 459M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T22:07:17.934 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (30m) 9m ago 48m 71.7M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 019eb1479485 2024-04-20T22:07:17.934 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (30m) 9m ago 45m 51.0M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 55aac560aab6 2024-04-20T22:07:17.934 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (34m) 9m ago 47m 8472k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T22:07:17.934 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (34m) 9m ago 45m 8048k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T22:07:17.934 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (28m) 9m ago 44m 91.2M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 ee2bb901f26e 2024-04-20T22:07:17.934 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (28m) 9m ago 44m 94.6M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 9075152d4162 2024-04-20T22:07:17.935 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (27m) 9m ago 43m 78.1M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 2d4f3e38dc0e 2024-04-20T22:07:17.935 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (26m) 9m ago 42m 92.6M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 8f97b17ae0eb 2024-04-20T22:07:17.935 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (26m) 9m ago 42m 95.6M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 653d8079967d 2024-04-20T22:07:17.935 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (25m) 9m ago 41m 89.2M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 596dded14c12 2024-04-20T22:07:17.935 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (33m) 9m ago 46m 82.1M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T22:07:17.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.936+0000 7fc45ca00700 1 -- 172.21.15.62:0/2286050214 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc43c06ecc0 msgr2=0x7fc43c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:07:17.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.936+0000 7fc45ca00700 1 --2- 172.21.15.62:0/2286050214 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc43c06ecc0 0x7fc43c071170 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fc448005950 tx=0x7fc4480058e0 comp rx=0 tx=0).stop 2024-04-20T22:07:17.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.936+0000 7fc45ca00700 1 -- 172.21.15.62:0/2286050214 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc45810c8f0 msgr2=0x7fc45819c9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:07:17.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.936+0000 7fc45ca00700 1 --2- 172.21.15.62:0/2286050214 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc45810c8f0 0x7fc45819c9c0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fc444006010 tx=0x7fc44400be20 comp rx=0 tx=0).stop 2024-04-20T22:07:17.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.936+0000 7fc45ca00700 1 -- 172.21.15.62:0/2286050214 shutdown_connections 2024-04-20T22:07:17.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.936+0000 7fc45ca00700 1 --2- 172.21.15.62:0/2286050214 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc43c06ecc0 0x7fc43c071170 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:07:17.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.936+0000 7fc45ca00700 1 --2- 172.21.15.62:0/2286050214 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc458106b80 0x7fc45819c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:07:17.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.936+0000 7fc45ca00700 1 --2- 172.21.15.62:0/2286050214 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc45810c8f0 0x7fc45819c9c0 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:07:17.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.936+0000 7fc45ca00700 1 -- 172.21.15.62:0/2286050214 >> 172.21.15.62:0/2286050214 conn(0x7fc458076020 msgr2=0x7fc458077a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:07:17.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.936+0000 7fc45ca00700 1 -- 172.21.15.62:0/2286050214 shutdown_connections 2024-04-20T22:07:17.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:17.936+0000 7fc45ca00700 1 -- 172.21.15.62:0/2286050214 wait complete. 2024-04-20T22:07:18.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:18.096+0000 7f63f9a00700 1 -- 172.21.15.62:0/1778098727 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f63f4102460 msgr2=0x7f63f41028d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:07:18.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:18.096+0000 7f63f9a00700 1 --2- 172.21.15.62:0/1778098727 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f63f4102460 0x7f63f41028d0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f63e0009910 tx=0x7f63e0009c20 comp rx=0 tx=0).stop 2024-04-20T22:07:18.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:18.097+0000 7f63f9a00700 1 -- 172.21.15.62:0/1778098727 shutdown_connections 2024-04-20T22:07:18.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:18.097+0000 7f63f9a00700 1 --2- 172.21.15.62:0/1778098727 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f63f4102460 0x7f63f41028d0 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:07:18.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:18.097+0000 7f63f9a00700 1 --2- 172.21.15.62:0/1778098727 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63f4108460 0x7f63f4108830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:07:18.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:18.097+0000 7f63f9a00700 1 -- 172.21.15.62:0/1778098727 >> 172.21.15.62:0/1778098727 conn(0x7f63f40fdf40 msgr2=0x7f63f4100350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:07:18.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:18.097+0000 7f63f9a00700 1 -- 172.21.15.62:0/1778098727 shutdown_connections 2024-04-20T22:07:18.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:18.097+0000 7f63f9a00700 1 -- 172.21.15.62:0/1778098727 wait complete. 2024-04-20T22:07:18.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:18.098+0000 7f63f9a00700 1 Processor -- start 2024-04-20T22:07:18.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:18.098+0000 7f63f9a00700 1 -- start start 2024-04-20T22:07:18.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:18.099+0000 7f63f9a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63f4102460 0x7f63f4198190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:07:18.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:18.099+0000 7f63f9a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f63f4108460 0x7f63f41986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:07:18.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:18.099+0000 7f63f9a00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63f4198db0 con 0x7f63f4108460 2024-04-20T22:07:18.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:18.099+0000 7f63f9a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63f419cb40 con 0x7f63f4102460 2024-04-20T22:07:18.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:18.099+0000 7f63f2000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f63f4108460 0x7f63f41986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:07:18.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:18.099+0000 7f63f2000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f63f4108460 0x7f63f41986d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40060/0 (socket says 172.21.15.62:40060) 2024-04-20T22:07:18.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:18.099+0000 7f63f2000700 1 -- 172.21.15.62:0/4146659750 learned_addr learned my addr 172.21.15.62:0/4146659750 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:07:18.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:18.100+0000 7f63f2a00700 1 --2- 172.21.15.62:0/4146659750 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63f4102460 0x7f63f4198190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:07:18.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:18.100+0000 7f63f2000700 1 -- 172.21.15.62:0/4146659750 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63f4102460 msgr2=0x7f63f4198190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:07:18.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:18.100+0000 7f63f2000700 1 --2- 172.21.15.62:0/4146659750 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63f4102460 0x7f63f4198190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:07:18.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:18.100+0000 7f63f2000700 1 -- 172.21.15.62:0/4146659750 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63e00095f0 con 0x7f63f4108460 2024-04-20T22:07:18.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:18.100+0000 7f63f2000700 1 --2- 172.21.15.62:0/4146659750 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f63f4108460 0x7f63f41986d0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f63e000b5c0 tx=0x7f63e000be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:07:18.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:18.100+0000 7f63eb400700 1 -- 172.21.15.62:0/4146659750 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f63e001d070 con 0x7f63f4108460 2024-04-20T22:07:18.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:18.100+0000 7f63eb400700 1 -- 172.21.15.62:0/4146659750 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f63e000f460 con 0x7f63f4108460 2024-04-20T22:07:18.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:18.100+0000 7f63f9a00700 1 -- 172.21.15.62:0/4146659750 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63f419cdc0 con 0x7f63f4108460 2024-04-20T22:07:18.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:18.100+0000 7f63eb400700 1 -- 172.21.15.62:0/4146659750 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f63e0017610 con 0x7f63f4108460 2024-04-20T22:07:18.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:18.101+0000 7f63f9a00700 1 -- 172.21.15.62:0/4146659750 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63f419d2b0 con 0x7f63f4108460 2024-04-20T22:07:18.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:18.102+0000 7f63f9a00700 1 -- 172.21.15.62:0/4146659750 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63f410a9f0 con 0x7f63f4108460 2024-04-20T22:07:18.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:18.103+0000 7f63eb400700 1 -- 172.21.15.62:0/4146659750 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f63e000f5d0 con 0x7f63f4108460 2024-04-20T22:07:18.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:18.104+0000 7f63eb400700 1 --2- 172.21.15.62:0/4146659750 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f63d806ecc0 0x7f63d8071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:07:18.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:18.104+0000 7f63eb400700 1 -- 172.21.15.62:0/4146659750 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f63e0090510 con 0x7f63f4108460 2024-04-20T22:07:18.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:18.104+0000 7f63f2a00700 1 --2- 172.21.15.62:0/4146659750 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f63d806ecc0 0x7f63d8071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:07:18.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:18.105+0000 7f63f2a00700 1 --2- 172.21.15.62:0/4146659750 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f63d806ecc0 0x7f63d8071170 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f63e4006fd0 tx=0x7f63e4008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:07:18.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:18.109+0000 7f63eb400700 1 -- 172.21.15.62:0/4146659750 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f63e0059220 con 0x7f63f4108460 2024-04-20T22:07:18.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:07:17 smithi189 ceph-mon[57022]: from='client.35712 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:07:18.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:07:17 smithi062 ceph-mon[76322]: from='client.35712 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:07:18.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:18.425+0000 7f63f9a00700 1 -- 172.21.15.62:0/4146659750 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f63f402cc30 con 0x7f63f4108460 2024-04-20T22:07:18.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:18.426+0000 7f63eb400700 1 -- 172.21.15.62:0/4146659750 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7f63e005c8e0 con 0x7f63f4108460 2024-04-20T22:07:18.426 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:07:18.426 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T22:07:18.426 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:07:18.426 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:07:18.426 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T22:07:18.426 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:07:18.427 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:07:18.427 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T22:07:18.427 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 6 2024-04-20T22:07:18.427 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:07:18.427 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T22:07:18.427 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 1, 2024-04-20T22:07:18.427 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 3 2024-04-20T22:07:18.427 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:07:18.427 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T22:07:18.427 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 1, 2024-04-20T22:07:18.427 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 13 2024-04-20T22:07:18.427 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T22:07:18.427 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:07:18.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:18.429+0000 7f63f9a00700 1 -- 172.21.15.62:0/4146659750 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f63d806ecc0 msgr2=0x7f63d8071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:07:18.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:18.429+0000 7f63f9a00700 1 --2- 172.21.15.62:0/4146659750 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f63d806ecc0 0x7f63d8071170 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f63e4006fd0 tx=0x7f63e4008040 comp rx=0 tx=0).stop 2024-04-20T22:07:18.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:18.429+0000 7f63f9a00700 1 -- 172.21.15.62:0/4146659750 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f63f4108460 msgr2=0x7f63f41986d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:07:18.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:18.429+0000 7f63f9a00700 1 --2- 172.21.15.62:0/4146659750 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f63f4108460 0x7f63f41986d0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f63e000b5c0 tx=0x7f63e000be00 comp rx=0 tx=0).stop 2024-04-20T22:07:18.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:18.429+0000 7f63f9a00700 1 -- 172.21.15.62:0/4146659750 shutdown_connections 2024-04-20T22:07:18.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:18.429+0000 7f63f9a00700 1 --2- 172.21.15.62:0/4146659750 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f63d806ecc0 0x7f63d8071170 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:07:18.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:18.429+0000 7f63f9a00700 1 --2- 172.21.15.62:0/4146659750 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63f4102460 0x7f63f4198190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:07:18.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:18.429+0000 7f63f9a00700 1 --2- 172.21.15.62:0/4146659750 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f63f4108460 0x7f63f41986d0 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:07:18.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:18.429+0000 7f63f9a00700 1 -- 172.21.15.62:0/4146659750 >> 172.21.15.62:0/4146659750 conn(0x7f63f40fdf40 msgr2=0x7f63f40fe9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:07:18.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:18.429+0000 7f63f9a00700 1 -- 172.21.15.62:0/4146659750 shutdown_connections 2024-04-20T22:07:18.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:18.429+0000 7f63f9a00700 1 -- 172.21.15.62:0/4146659750 wait complete. 2024-04-20T22:07:18.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:18.585+0000 7f34de000700 1 -- 172.21.15.62:0/2672164027 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f34d80739a0 msgr2=0x7f34d810c5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:07:18.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:18.585+0000 7f34de000700 1 --2- 172.21.15.62:0/2672164027 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f34d80739a0 0x7f34d810c5e0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f34c0009940 tx=0x7f34c0009c50 comp rx=0 tx=0).stop 2024-04-20T22:07:18.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:18.585+0000 7f34de000700 1 -- 172.21.15.62:0/2672164027 shutdown_connections 2024-04-20T22:07:18.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:18.585+0000 7f34de000700 1 --2- 172.21.15.62:0/2672164027 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f34d80739a0 0x7f34d810c5e0 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:07:18.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:18.585+0000 7f34de000700 1 --2- 172.21.15.62:0/2672164027 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f34d8073090 0x7f34d8073460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:07:18.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:18.585+0000 7f34de000700 1 -- 172.21.15.62:0/2672164027 >> 172.21.15.62:0/2672164027 conn(0x7f34d80fbda0 msgr2=0x7f34d80fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:07:18.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:18.585+0000 7f34de000700 1 -- 172.21.15.62:0/2672164027 shutdown_connections 2024-04-20T22:07:18.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:18.585+0000 7f34de000700 1 -- 172.21.15.62:0/2672164027 wait complete. 2024-04-20T22:07:18.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:18.586+0000 7f34de000700 1 Processor -- start 2024-04-20T22:07:18.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:18.586+0000 7f34de000700 1 -- start start 2024-04-20T22:07:18.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:18.587+0000 7f34de000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f34d8073090 0x7f34d81981b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:07:18.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:18.587+0000 7f34de000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f34d80739a0 0x7f34d81986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:07:18.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:18.587+0000 7f34de000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34d8198dd0 con 0x7f34d8073090 2024-04-20T22:07:18.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:18.587+0000 7f34de000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34d819cb60 con 0x7f34d80739a0 2024-04-20T22:07:18.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:18.587+0000 7f34d7400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f34d8073090 0x7f34d81981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:07:18.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:18.587+0000 7f34d7400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f34d8073090 0x7f34d81981b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40068/0 (socket says 172.21.15.62:40068) 2024-04-20T22:07:18.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:18.587+0000 7f34d7400700 1 -- 172.21.15.62:0/2149283160 learned_addr learned my addr 172.21.15.62:0/2149283160 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:07:18.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:18.587+0000 7f34d7400700 1 -- 172.21.15.62:0/2149283160 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f34d80739a0 msgr2=0x7f34d81986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:07:18.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:18.587+0000 7f34cea00700 1 --2- 172.21.15.62:0/2149283160 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f34d80739a0 0x7f34d81986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:07:18.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:18.588+0000 7f34d7400700 1 --2- 172.21.15.62:0/2149283160 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f34d80739a0 0x7f34d81986f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:07:18.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:18.588+0000 7f34d7400700 1 -- 172.21.15.62:0/2149283160 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34c00095f0 con 0x7f34d8073090 2024-04-20T22:07:18.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:18.588+0000 7f34cea00700 1 --2- 172.21.15.62:0/2149283160 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f34d80739a0 0x7f34d81986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T22:07:18.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:18.588+0000 7f34d7400700 1 --2- 172.21.15.62:0/2149283160 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f34d8073090 0x7f34d81981b0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f34c800ef10 tx=0x7f34c800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:07:18.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:18.588+0000 7f34d4c00700 1 -- 172.21.15.62:0/2149283160 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f34c800ca50 con 0x7f34d8073090 2024-04-20T22:07:18.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:18.588+0000 7f34d4c00700 1 -- 172.21.15.62:0/2149283160 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f34c800f040 con 0x7f34d8073090 2024-04-20T22:07:18.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:18.588+0000 7f34de000700 1 -- 172.21.15.62:0/2149283160 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34d819cde0 con 0x7f34d8073090 2024-04-20T22:07:18.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:18.588+0000 7f34d4c00700 1 -- 172.21.15.62:0/2149283160 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f34c80146b0 con 0x7f34d8073090 2024-04-20T22:07:18.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:18.588+0000 7f34de000700 1 -- 172.21.15.62:0/2149283160 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34d819d330 con 0x7f34d8073090 2024-04-20T22:07:18.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:18.590+0000 7f34de000700 1 -- 172.21.15.62:0/2149283160 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34d8109ce0 con 0x7f34d8073090 2024-04-20T22:07:18.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:18.591+0000 7f34d4c00700 1 -- 172.21.15.62:0/2149283160 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f34c800cbb0 con 0x7f34d8073090 2024-04-20T22:07:18.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:18.592+0000 7f34d4c00700 1 --2- 172.21.15.62:0/2149283160 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f34bc06ecc0 0x7f34bc071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:07:18.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:18.592+0000 7f34d4c00700 1 -- 172.21.15.62:0/2149283160 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f34c808f240 con 0x7f34d8073090 2024-04-20T22:07:18.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:18.592+0000 7f34cea00700 1 --2- 172.21.15.62:0/2149283160 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f34bc06ecc0 0x7f34bc071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:07:18.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:18.593+0000 7f34cea00700 1 --2- 172.21.15.62:0/2149283160 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f34bc06ecc0 0x7f34bc071170 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f34c0000c00 tx=0x7f34c00058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:07:18.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:18.597+0000 7f34d4c00700 1 -- 172.21.15.62:0/2149283160 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f34c8057fd0 con 0x7f34d8073090 2024-04-20T22:07:18.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:18.869+0000 7f34de000700 1 -- 172.21.15.62:0/2149283160 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f34d81995e0 con 0x7f34d8073090 2024-04-20T22:07:18.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:18.870+0000 7f34d4c00700 1 -- 172.21.15.62:0/2149283160 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 27 v27) v1 ==== 76+0+1888 (secure 0 0 0) 0x7f34c805b690 con 0x7f34d8073090 2024-04-20T22:07:18.871 INFO:teuthology.orchestra.run.smithi062.stdout:e27 2024-04-20T22:07:18.871 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T22:07:18.871 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:07:18.871 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T22:07:18.871 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:07:18.871 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T22:07:18.871 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T22:07:18.871 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 20 2024-04-20T22:07:18.871 INFO:teuthology.orchestra.run.smithi062.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-20T22:07:18.871 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T22:07:18.871 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T21:42:01.559061+0000 2024-04-20T22:07:18.871 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T22:07:18.871 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T22:07:18.872 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T22:07:18.872 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T22:07:18.872 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T22:07:18.872 INFO:teuthology.orchestra.run.smithi062.stdout:max_xattr_size 65536 2024-04-20T22:07:18.872 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T22:07:18.872 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T22:07:18.872 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 81 2024-04-20T22:07:18.872 INFO:teuthology.orchestra.run.smithi062.stdout:snap_rstat 1 2024-04-20T22:07:18.872 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:07:18.872 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 2 2024-04-20T22:07:18.872 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1 2024-04-20T22:07:18.872 INFO:teuthology.orchestra.run.smithi062.stdout:up {} 2024-04-20T22:07:18.872 INFO:teuthology.orchestra.run.smithi062.stdout:failed 0,1 2024-04-20T22:07:18.872 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T22:07:18.872 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T22:07:18.873 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T22:07:18.873 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T22:07:18.873 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data enabled 2024-04-20T22:07:18.873 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T22:07:18.873 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T22:07:18.873 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 1 2024-04-20T22:07:18.873 INFO:teuthology.orchestra.run.smithi062.stdout:qdb_cluster leader: 0 members: 2024-04-20T22:07:18.873 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:07:18.873 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:07:18.873 INFO:teuthology.orchestra.run.smithi062.stdout:Standby daemons: 2024-04-20T22:07:18.873 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:07:18.873 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ryqblb{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6826/4113916205,v1:172.21.15.62:6827/4113916205] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:07:18.873 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ozizwi{-1:34476} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6828/3801975165,v1:172.21.15.62:6829/3801975165] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:07:18.873 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.xbgspw{-1:44285} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6826/3996040526,v1:172.21.15.189:6827/3996040526] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T22:07:18.873 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.dvimer{-1:44727} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6824/3576699232,v1:172.21.15.189:6825/3576699232] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:07:18.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:18.874+0000 7f34de000700 1 -- 172.21.15.62:0/2149283160 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f34bc06ecc0 msgr2=0x7f34bc071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:07:18.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:18.874+0000 7f34de000700 1 --2- 172.21.15.62:0/2149283160 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f34bc06ecc0 0x7f34bc071170 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f34c0000c00 tx=0x7f34c00058e0 comp rx=0 tx=0).stop 2024-04-20T22:07:18.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:18.874+0000 7f34de000700 1 -- 172.21.15.62:0/2149283160 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f34d8073090 msgr2=0x7f34d81981b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:07:18.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:18.874+0000 7f34de000700 1 --2- 172.21.15.62:0/2149283160 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f34d8073090 0x7f34d81981b0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f34c800ef10 tx=0x7f34c800c3a0 comp rx=0 tx=0).stop 2024-04-20T22:07:18.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:18.874+0000 7f34de000700 1 -- 172.21.15.62:0/2149283160 shutdown_connections 2024-04-20T22:07:18.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:18.874+0000 7f34de000700 1 --2- 172.21.15.62:0/2149283160 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f34bc06ecc0 0x7f34bc071170 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:07:18.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:18.874+0000 7f34de000700 1 --2- 172.21.15.62:0/2149283160 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f34d8073090 0x7f34d81981b0 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:07:18.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:18.874+0000 7f34de000700 1 --2- 172.21.15.62:0/2149283160 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f34d80739a0 0x7f34d81986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:07:18.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:18.874+0000 7f34de000700 1 -- 172.21.15.62:0/2149283160 >> 172.21.15.62:0/2149283160 conn(0x7f34d80fbda0 msgr2=0x7f34d8106e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:07:18.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:18.875+0000 7f34de000700 1 -- 172.21.15.62:0/2149283160 shutdown_connections 2024-04-20T22:07:18.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:18.875+0000 7f34de000700 1 -- 172.21.15.62:0/2149283160 wait complete. 2024-04-20T22:07:18.876 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 27 2024-04-20T22:07:19.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:19.033+0000 7f6653600700 1 -- 172.21.15.62:0/4259575089 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f664c0ff210 msgr2=0x7f664c0ff5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:07:19.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:19.033+0000 7f6653600700 1 --2- 172.21.15.62:0/4259575089 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f664c0ff210 0x7f664c0ff5e0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f6640009910 tx=0x7f6640009c20 comp rx=0 tx=0).stop 2024-04-20T22:07:19.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:19.034+0000 7f6653600700 1 -- 172.21.15.62:0/4259575089 shutdown_connections 2024-04-20T22:07:19.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:19.034+0000 7f6653600700 1 --2- 172.21.15.62:0/4259575089 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f664c0ffb20 0x7f664c104000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:07:19.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:19.034+0000 7f6653600700 1 --2- 172.21.15.62:0/4259575089 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f664c0ff210 0x7f664c0ff5e0 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:07:19.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:19.034+0000 7f6653600700 1 -- 172.21.15.62:0/4259575089 >> 172.21.15.62:0/4259575089 conn(0x7f664c0fac80 msgr2=0x7f664c0fd090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:07:19.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:19.034+0000 7f6653600700 1 -- 172.21.15.62:0/4259575089 shutdown_connections 2024-04-20T22:07:19.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:19.034+0000 7f6653600700 1 -- 172.21.15.62:0/4259575089 wait complete. 2024-04-20T22:07:19.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:19.035+0000 7f6653600700 1 Processor -- start 2024-04-20T22:07:19.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:19.035+0000 7f6653600700 1 -- start start 2024-04-20T22:07:19.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:19.035+0000 7f6653600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f664c0ff210 0x7f664c19c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:07:19.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:19.035+0000 7f6653600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f664c0ffb20 0x7f664c19c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:07:19.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:19.035+0000 7f6653600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f664c19d060 con 0x7f664c0ffb20 2024-04-20T22:07:19.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:19.035+0000 7f6653600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f664c1a0df0 con 0x7f664c0ff210 2024-04-20T22:07:19.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:19.036+0000 7f664be00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f664c0ffb20 0x7f664c19c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:07:19.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:19.036+0000 7f664be00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f664c0ffb20 0x7f664c19c980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40088/0 (socket says 172.21.15.62:40088) 2024-04-20T22:07:19.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:19.036+0000 7f664be00700 1 -- 172.21.15.62:0/3206063729 learned_addr learned my addr 172.21.15.62:0/3206063729 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:07:19.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:19.036+0000 7f6650c00700 1 --2- 172.21.15.62:0/3206063729 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f664c0ff210 0x7f664c19c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:07:19.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:19.036+0000 7f664be00700 1 -- 172.21.15.62:0/3206063729 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f664c0ff210 msgr2=0x7f664c19c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:07:19.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:19.036+0000 7f664be00700 1 --2- 172.21.15.62:0/3206063729 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f664c0ff210 0x7f664c19c440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:07:19.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:19.036+0000 7f664be00700 1 -- 172.21.15.62:0/3206063729 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66400095f0 con 0x7f664c0ffb20 2024-04-20T22:07:19.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:19.037+0000 7f664be00700 1 --2- 172.21.15.62:0/3206063729 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f664c0ffb20 0x7f664c19c980 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f663800c8a0 tx=0x7f663800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:07:19.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:19.037+0000 7f6649600700 1 -- 172.21.15.62:0/3206063729 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f663800cea0 con 0x7f664c0ffb20 2024-04-20T22:07:19.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:19.037+0000 7f6649600700 1 -- 172.21.15.62:0/3206063729 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6638004830 con 0x7f664c0ffb20 2024-04-20T22:07:19.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:19.037+0000 7f6649600700 1 -- 172.21.15.62:0/3206063729 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f66380055b0 con 0x7f664c0ffb20 2024-04-20T22:07:19.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:19.037+0000 7f6653600700 1 -- 172.21.15.62:0/3206063729 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f664c1a10d0 con 0x7f664c0ffb20 2024-04-20T22:07:19.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:19.037+0000 7f6653600700 1 -- 172.21.15.62:0/3206063729 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f664c1a1620 con 0x7f664c0ffb20 2024-04-20T22:07:19.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:19.038+0000 7f6653600700 1 -- 172.21.15.62:0/3206063729 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f664c101730 con 0x7f664c0ffb20 2024-04-20T22:07:19.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:19.039+0000 7f6649600700 1 -- 172.21.15.62:0/3206063729 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f66380049a0 con 0x7f664c0ffb20 2024-04-20T22:07:19.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:19.041+0000 7f6649600700 1 --2- 172.21.15.62:0/3206063729 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f663406ecc0 0x7f6634071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:07:19.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:19.041+0000 7f6649600700 1 -- 172.21.15.62:0/3206063729 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f663808eaf0 con 0x7f664c0ffb20 2024-04-20T22:07:19.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:19.041+0000 7f6650c00700 1 --2- 172.21.15.62:0/3206063729 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f663406ecc0 0x7f6634071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:07:19.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:19.042+0000 7f6650c00700 1 --2- 172.21.15.62:0/3206063729 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f663406ecc0 0x7f6634071170 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f6640006010 tx=0x7f664001a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:07:19.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:19.046+0000 7f6649600700 1 -- 172.21.15.62:0/3206063729 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f6638057880 con 0x7f664c0ffb20 2024-04-20T22:07:19.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:07:18 smithi189 ceph-mon[57022]: from='client.35716 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:07:19.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:07:18 smithi189 ceph-mon[57022]: pgmap v961: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:07:19.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:07:18 smithi189 ceph-mon[57022]: from='client.35720 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:07:19.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:07:18 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/4146659750' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:07:19.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:07:18 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/2149283160' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:07:19.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:07:18 smithi062 ceph-mon[76322]: from='client.35716 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:07:19.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:07:18 smithi062 ceph-mon[76322]: pgmap v961: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:07:19.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:07:18 smithi062 ceph-mon[76322]: from='client.35720 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:07:19.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:07:18 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/4146659750' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:07:19.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:07:18 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/2149283160' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:07:19.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:19.305+0000 7f6653600700 1 -- 172.21.15.62:0/3206063729 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f664c1a1960 con 0x7f663406ecc0 2024-04-20T22:07:19.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:19.307+0000 7f6649600700 1 -- 172.21.15.62:0/3206063729 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f664c1a1960 con 0x7f663406ecc0 2024-04-20T22:07:19.307 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:07:19.307 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T22:07:19.307 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T22:07:19.307 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-20T22:07:19.307 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T22:07:19.307 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr", 2024-04-20T22:07:19.307 INFO:teuthology.orchestra.run.smithi062.stdout: "crash", 2024-04-20T22:07:19.308 INFO:teuthology.orchestra.run.smithi062.stdout: "mon", 2024-04-20T22:07:19.308 INFO:teuthology.orchestra.run.smithi062.stdout: "osd" 2024-04-20T22:07:19.308 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T22:07:19.308 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "15/23 daemons upgraded", 2024-04-20T22:07:19.308 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading mds daemons", 2024-04-20T22:07:19.308 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T22:07:19.308 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:07:19.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:19.310+0000 7f6653600700 1 -- 172.21.15.62:0/3206063729 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f663406ecc0 msgr2=0x7f6634071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:07:19.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:19.310+0000 7f6653600700 1 --2- 172.21.15.62:0/3206063729 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f663406ecc0 0x7f6634071170 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f6640006010 tx=0x7f664001a040 comp rx=0 tx=0).stop 2024-04-20T22:07:19.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:19.310+0000 7f6653600700 1 -- 172.21.15.62:0/3206063729 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f664c0ffb20 msgr2=0x7f664c19c980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:07:19.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:19.310+0000 7f6653600700 1 --2- 172.21.15.62:0/3206063729 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f664c0ffb20 0x7f664c19c980 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f663800c8a0 tx=0x7f663800cc60 comp rx=0 tx=0).stop 2024-04-20T22:07:19.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:19.310+0000 7f6653600700 1 -- 172.21.15.62:0/3206063729 shutdown_connections 2024-04-20T22:07:19.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:19.310+0000 7f6653600700 1 --2- 172.21.15.62:0/3206063729 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f663406ecc0 0x7f6634071170 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:07:19.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:19.310+0000 7f6653600700 1 --2- 172.21.15.62:0/3206063729 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f664c0ff210 0x7f664c19c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:07:19.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:19.310+0000 7f6653600700 1 --2- 172.21.15.62:0/3206063729 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f664c0ffb20 0x7f664c19c980 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:07:19.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:19.310+0000 7f6653600700 1 -- 172.21.15.62:0/3206063729 >> 172.21.15.62:0/3206063729 conn(0x7f664c0fac80 msgr2=0x7f664c110680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:07:19.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:19.311+0000 7f6653600700 1 -- 172.21.15.62:0/3206063729 shutdown_connections 2024-04-20T22:07:19.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:19.311+0000 7f6653600700 1 -- 172.21.15.62:0/3206063729 wait complete. 2024-04-20T22:07:19.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:19.469+0000 7fcc98c00700 1 -- 172.21.15.62:0/3321095685 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcc94100e80 msgr2=0x7fcc94104e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:07:19.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:19.469+0000 7fcc98c00700 1 --2- 172.21.15.62:0/3321095685 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcc94100e80 0x7fcc94104e50 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fcc7c009910 tx=0x7fcc7c009c20 comp rx=0 tx=0).stop 2024-04-20T22:07:19.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:19.470+0000 7fcc98c00700 1 -- 172.21.15.62:0/3321095685 shutdown_connections 2024-04-20T22:07:19.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:19.470+0000 7fcc98c00700 1 --2- 172.21.15.62:0/3321095685 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcc94100e80 0x7fcc94104e50 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:07:19.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:19.470+0000 7fcc98c00700 1 --2- 172.21.15.62:0/3321095685 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc941004e0 0x7fcc941008b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:07:19.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:19.470+0000 7fcc98c00700 1 -- 172.21.15.62:0/3321095685 >> 172.21.15.62:0/3321095685 conn(0x7fcc940fbda0 msgr2=0x7fcc940fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:07:19.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:19.470+0000 7fcc98c00700 1 -- 172.21.15.62:0/3321095685 shutdown_connections 2024-04-20T22:07:19.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:19.470+0000 7fcc98c00700 1 -- 172.21.15.62:0/3321095685 wait complete. 2024-04-20T22:07:19.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:19.471+0000 7fcc98c00700 1 Processor -- start 2024-04-20T22:07:19.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:19.471+0000 7fcc98c00700 1 -- start start 2024-04-20T22:07:19.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:19.471+0000 7fcc98c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc941004e0 0x7fcc94073050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:07:19.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:19.471+0000 7fcc98c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcc94100e80 0x7fcc94073590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:07:19.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:19.471+0000 7fcc98c00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc94073ad0 con 0x7fcc94100e80 2024-04-20T22:07:19.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:19.471+0000 7fcc98c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc94073c10 con 0x7fcc941004e0 2024-04-20T22:07:19.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:19.472+0000 7fcc89400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcc94100e80 0x7fcc94073590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:07:19.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:19.472+0000 7fcc89400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcc94100e80 0x7fcc94073590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40102/0 (socket says 172.21.15.62:40102) 2024-04-20T22:07:19.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:19.472+0000 7fcc89400700 1 -- 172.21.15.62:0/3802935300 learned_addr learned my addr 172.21.15.62:0/3802935300 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:07:19.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:19.472+0000 7fcc91e00700 1 --2- 172.21.15.62:0/3802935300 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc941004e0 0x7fcc94073050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:07:19.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:19.472+0000 7fcc89400700 1 -- 172.21.15.62:0/3802935300 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc941004e0 msgr2=0x7fcc94073050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:07:19.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:19.472+0000 7fcc89400700 1 --2- 172.21.15.62:0/3802935300 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc941004e0 0x7fcc94073050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:07:19.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:19.472+0000 7fcc89400700 1 -- 172.21.15.62:0/3802935300 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc7c0095f0 con 0x7fcc94100e80 2024-04-20T22:07:19.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:19.472+0000 7fcc91e00700 1 --2- 172.21.15.62:0/3802935300 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc941004e0 0x7fcc94073050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T22:07:19.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:19.473+0000 7fcc89400700 1 --2- 172.21.15.62:0/3802935300 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcc94100e80 0x7fcc94073590 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fcc7c00bda0 tx=0x7fcc7c00be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:07:19.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:19.473+0000 7fcc8b400700 1 -- 172.21.15.62:0/3802935300 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcc7c01d070 con 0x7fcc94100e80 2024-04-20T22:07:19.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:19.473+0000 7fcc8b400700 1 -- 172.21.15.62:0/3802935300 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcc7c00f460 con 0x7fcc94100e80 2024-04-20T22:07:19.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:19.473+0000 7fcc98c00700 1 -- 172.21.15.62:0/3802935300 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc94073e90 con 0x7fcc94100e80 2024-04-20T22:07:19.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:19.473+0000 7fcc8b400700 1 -- 172.21.15.62:0/3802935300 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcc7c017630 con 0x7fcc94100e80 2024-04-20T22:07:19.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:19.473+0000 7fcc98c00700 1 -- 172.21.15.62:0/3802935300 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc941a24a0 con 0x7fcc94100e80 2024-04-20T22:07:19.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:19.474+0000 7fcc98c00700 1 -- 172.21.15.62:0/3802935300 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc94108760 con 0x7fcc94100e80 2024-04-20T22:07:19.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:19.475+0000 7fcc8b400700 1 -- 172.21.15.62:0/3802935300 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fcc7c00fa40 con 0x7fcc94100e80 2024-04-20T22:07:19.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:19.476+0000 7fcc8b400700 1 --2- 172.21.15.62:0/3802935300 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fcc7806ecc0 0x7fcc78071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:07:19.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:19.476+0000 7fcc8b400700 1 -- 172.21.15.62:0/3802935300 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fcc7c090340 con 0x7fcc94100e80 2024-04-20T22:07:19.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:19.477+0000 7fcc91e00700 1 --2- 172.21.15.62:0/3802935300 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fcc7806ecc0 0x7fcc78071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:07:19.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:19.477+0000 7fcc91e00700 1 --2- 172.21.15.62:0/3802935300 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fcc7806ecc0 0x7fcc78071170 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fcc84005fd0 tx=0x7fcc84005ee0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:07:19.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:19.481+0000 7fcc8b400700 1 -- 172.21.15.62:0/3802935300 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fcc7c059050 con 0x7fcc94100e80 2024-04-20T22:07:19.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:19.808+0000 7fcc98c00700 1 -- 172.21.15.62:0/3802935300 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fcc940745b0 con 0x7fcc94100e80 2024-04-20T22:07:19.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:19.809+0000 7fcc8b400700 1 -- 172.21.15.62:0/3802935300 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7fcc7c05c710 con 0x7fcc94100e80 2024-04-20T22:07:19.809 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-20T22:07:19.809 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-20T22:07:19.809 INFO:teuthology.orchestra.run.smithi062.stdout: host smithi062 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T22:07:19.809 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-20T22:07:19.809 INFO:teuthology.orchestra.run.smithi062.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi062 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpq7y7oa_r:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:07:19.809 INFO:teuthology.orchestra.run.smithi062.stdout:Traceback (most recent call last): 2024-04-20T22:07:19.810 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-20T22:07:19.810 INFO:teuthology.orchestra.run.smithi062.stdout: return _run_code(code, main_globals, None, 2024-04-20T22:07:19.810 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-20T22:07:19.810 INFO:teuthology.orchestra.run.smithi062.stdout: exec(code, run_globals) 2024-04-20T22:07:19.810 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5618, in 2024-04-20T22:07:19.810 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5606, in main 2024-04-20T22:07:19.810 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 408, in _infer_config 2024-04-20T22:07:19.810 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 323, in _infer_fsid 2024-04-20T22:07:19.810 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 436, in _infer_image 2024-04-20T22:07:19.810 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 310, in _validate_fsid 2024-04-20T22:07:19.810 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 3359, in command_ceph_volume 2024-04-20T22:07:19.810 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-20T22:07:19.810 INFO:teuthology.orchestra.run.smithi062.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi062 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpq7y7oa_r:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:07:19.810 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-20T22:07:19.811 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is degraded 2024-04-20T22:07:19.811 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T22:07:19.811 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T22:07:19.811 INFO:teuthology.orchestra.run.smithi062.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-20T22:07:19.811 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-20T22:07:19.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:19.811+0000 7fcc98c00700 1 -- 172.21.15.62:0/3802935300 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fcc7806ecc0 msgr2=0x7fcc78071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:07:19.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:19.811+0000 7fcc98c00700 1 --2- 172.21.15.62:0/3802935300 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fcc7806ecc0 0x7fcc78071170 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fcc84005fd0 tx=0x7fcc84005ee0 comp rx=0 tx=0).stop 2024-04-20T22:07:19.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:19.811+0000 7fcc98c00700 1 -- 172.21.15.62:0/3802935300 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcc94100e80 msgr2=0x7fcc94073590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:07:19.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:19.811+0000 7fcc98c00700 1 --2- 172.21.15.62:0/3802935300 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcc94100e80 0x7fcc94073590 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fcc7c00bda0 tx=0x7fcc7c00be80 comp rx=0 tx=0).stop 2024-04-20T22:07:19.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:19.812+0000 7fcc98c00700 1 -- 172.21.15.62:0/3802935300 shutdown_connections 2024-04-20T22:07:19.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:19.812+0000 7fcc98c00700 1 --2- 172.21.15.62:0/3802935300 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fcc7806ecc0 0x7fcc78071170 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:07:19.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:19.812+0000 7fcc98c00700 1 --2- 172.21.15.62:0/3802935300 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc941004e0 0x7fcc94073050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:07:19.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:19.812+0000 7fcc98c00700 1 --2- 172.21.15.62:0/3802935300 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcc94100e80 0x7fcc94073590 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:07:19.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:19.812+0000 7fcc98c00700 1 -- 172.21.15.62:0/3802935300 >> 172.21.15.62:0/3802935300 conn(0x7fcc940fbda0 msgr2=0x7fcc941046c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:07:19.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:19.812+0000 7fcc98c00700 1 -- 172.21.15.62:0/3802935300 shutdown_connections 2024-04-20T22:07:19.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:19.812+0000 7fcc98c00700 1 -- 172.21.15.62:0/3802935300 wait complete. 2024-04-20T22:07:20.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:07:19 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:07:20.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:07:19 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/3802935300' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:07:20.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:07:19 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:07:20.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:07:19 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/3802935300' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:07:21.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:07:20 smithi189 ceph-mon[57022]: from='client.35732 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:07:21.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:07:20 smithi189 ceph-mon[57022]: pgmap v962: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:07:21.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:07:20 smithi062 ceph-mon[76322]: from='client.35732 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:07:21.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:07:20 smithi062 ceph-mon[76322]: pgmap v962: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:07:23.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:07:22 smithi189 ceph-mon[57022]: pgmap v963: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:07:23.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:07:22 smithi062 ceph-mon[76322]: pgmap v963: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:07:25.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:07:24 smithi189 ceph-mon[57022]: pgmap v964: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:07:25.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:07:24 smithi062 ceph-mon[76322]: pgmap v964: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:07:27.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:07:26 smithi062 ceph-mon[76322]: pgmap v965: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:07:27.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:07:26 smithi189 ceph-mon[57022]: pgmap v965: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:07:29.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:07:28 smithi062 ceph-mon[76322]: pgmap v966: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:07:29.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:07:29 smithi189 ceph-mon[57022]: pgmap v966: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:07:31.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:07:31 smithi062 ceph-mon[76322]: pgmap v967: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:07:31.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:07:31 smithi189 ceph-mon[57022]: pgmap v967: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:07:33.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:07:33 smithi062 ceph-mon[76322]: pgmap v968: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:07:33.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:07:33 smithi189 ceph-mon[57022]: pgmap v968: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:07:35.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:07:35 smithi062 ceph-mon[76322]: pgmap v969: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:07:35.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:07:35 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:07:35.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:07:35 smithi189 ceph-mon[57022]: pgmap v969: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:07:35.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:07:35 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:07:37.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:07:37 smithi062 ceph-mon[76322]: pgmap v970: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:07:37.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:07:37 smithi189 ceph-mon[57022]: pgmap v970: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:07:39.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:07:39 smithi062 ceph-mon[76322]: pgmap v971: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:07:39.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:07:39 smithi189 ceph-mon[57022]: pgmap v971: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:07:41.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:07:41 smithi189 ceph-mon[57022]: pgmap v972: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:07:41.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:07:41 smithi062 ceph-mon[76322]: pgmap v972: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:07:43.408 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:07:43 smithi189 ceph-mon[57022]: pgmap v973: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:07:43.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:07:43 smithi062 ceph-mon[76322]: pgmap v973: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:07:45.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:07:45 smithi189 ceph-mon[57022]: pgmap v974: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:07:45.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:07:45 smithi062 ceph-mon[76322]: pgmap v974: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:07:47.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:07:47 smithi189 ceph-mon[57022]: pgmap v975: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:07:47.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:07:47 smithi062 ceph-mon[76322]: pgmap v975: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:07:49.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:07:49 smithi189 ceph-mon[57022]: pgmap v976: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:07:49.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:07:49 smithi062 ceph-mon[76322]: pgmap v976: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:07:49.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:49.990+0000 7f811ba00700 1 -- 172.21.15.62:0/380736493 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f81140ffbd0 msgr2=0x7f8114100040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:07:49.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:49.990+0000 7f811ba00700 1 --2- 172.21.15.62:0/380736493 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f81140ffbd0 0x7f8114100040 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f8100009910 tx=0x7f8100009c20 comp rx=0 tx=0).stop 2024-04-20T22:07:49.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:49.990+0000 7f811ba00700 1 -- 172.21.15.62:0/380736493 shutdown_connections 2024-04-20T22:07:49.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:49.990+0000 7f811ba00700 1 --2- 172.21.15.62:0/380736493 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f81140ffbd0 0x7f8114100040 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:07:49.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:49.990+0000 7f811ba00700 1 --2- 172.21.15.62:0/380736493 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8114104130 0x7f8114104500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:07:49.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:49.990+0000 7f811ba00700 1 -- 172.21.15.62:0/380736493 >> 172.21.15.62:0/380736493 conn(0x7f8114075130 msgr2=0x7f8114075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:07:49.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:49.990+0000 7f811ba00700 1 -- 172.21.15.62:0/380736493 shutdown_connections 2024-04-20T22:07:49.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:49.991+0000 7f811ba00700 1 -- 172.21.15.62:0/380736493 wait complete. 2024-04-20T22:07:49.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:49.992+0000 7f811ba00700 1 Processor -- start 2024-04-20T22:07:49.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:49.992+0000 7f811ba00700 1 -- start start 2024-04-20T22:07:49.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:49.992+0000 7f811ba00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f81140ffbd0 0x7f8114198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:07:49.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:49.992+0000 7f811ba00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8114104130 0x7f8114198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:07:49.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:49.992+0000 7f811ba00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8114198c40 con 0x7f8114104130 2024-04-20T22:07:49.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:49.992+0000 7f811ba00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f811419c9d0 con 0x7f81140ffbd0 2024-04-20T22:07:49.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:49.992+0000 7f8113e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8114104130 0x7f8114198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:07:49.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:49.993+0000 7f8113e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8114104130 0x7f8114198560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54174/0 (socket says 172.21.15.62:54174) 2024-04-20T22:07:49.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:49.993+0000 7f8113e00700 1 -- 172.21.15.62:0/4195026347 learned_addr learned my addr 172.21.15.62:0/4195026347 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:07:49.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:49.993+0000 7f8119000700 1 --2- 172.21.15.62:0/4195026347 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f81140ffbd0 0x7f8114198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:07:49.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:49.993+0000 7f8113e00700 1 -- 172.21.15.62:0/4195026347 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f81140ffbd0 msgr2=0x7f8114198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:07:49.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:49.993+0000 7f8113e00700 1 --2- 172.21.15.62:0/4195026347 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f81140ffbd0 0x7f8114198020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:07:49.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:49.993+0000 7f8113e00700 1 -- 172.21.15.62:0/4195026347 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81000095f0 con 0x7f8114104130 2024-04-20T22:07:49.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:49.993+0000 7f8113e00700 1 --2- 172.21.15.62:0/4195026347 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8114104130 0x7f8114198560 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f8100006010 tx=0x7f810000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:07:49.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:49.994+0000 7f8111600700 1 -- 172.21.15.62:0/4195026347 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f810001d070 con 0x7f8114104130 2024-04-20T22:07:49.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:49.994+0000 7f8111600700 1 -- 172.21.15.62:0/4195026347 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f810000f460 con 0x7f8114104130 2024-04-20T22:07:49.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:49.994+0000 7f8111600700 1 -- 172.21.15.62:0/4195026347 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8100017630 con 0x7f8114104130 2024-04-20T22:07:49.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:49.994+0000 7f811ba00700 1 -- 172.21.15.62:0/4195026347 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f811419cc50 con 0x7f8114104130 2024-04-20T22:07:49.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:49.994+0000 7f811ba00700 1 -- 172.21.15.62:0/4195026347 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f811419d0f0 con 0x7f8114104130 2024-04-20T22:07:49.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:49.995+0000 7f811ba00700 1 -- 172.21.15.62:0/4195026347 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f811404f2a0 con 0x7f8114104130 2024-04-20T22:07:49.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:49.996+0000 7f8111600700 1 -- 172.21.15.62:0/4195026347 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f810000f5d0 con 0x7f8114104130 2024-04-20T22:07:49.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:49.997+0000 7f8111600700 1 --2- 172.21.15.62:0/4195026347 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f80fc06ecc0 0x7f80fc071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:07:49.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:49.997+0000 7f8111600700 1 -- 172.21.15.62:0/4195026347 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f8100090330 con 0x7f8114104130 2024-04-20T22:07:49.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:49.997+0000 7f8119000700 1 --2- 172.21.15.62:0/4195026347 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f80fc06ecc0 0x7f80fc071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:07:49.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:49.998+0000 7f8119000700 1 --2- 172.21.15.62:0/4195026347 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f80fc06ecc0 0x7f80fc071170 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f8108005950 tx=0x7f81080058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:07:50.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:50.003+0000 7f8111600700 1 -- 172.21.15.62:0/4195026347 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f8100059180 con 0x7f8114104130 2024-04-20T22:07:50.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:50.263+0000 7f811ba00700 1 -- 172.21.15.62:0/4195026347 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f811419d420 con 0x7f80fc06ecc0 2024-04-20T22:07:50.263 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:07:50 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:07:50.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:50.266+0000 7f8111600700 1 -- 172.21.15.62:0/4195026347 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f811419d420 con 0x7f80fc06ecc0 2024-04-20T22:07:50.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:50.269+0000 7f811ba00700 1 -- 172.21.15.62:0/4195026347 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f80fc06ecc0 msgr2=0x7f80fc071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:07:50.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:50.269+0000 7f811ba00700 1 --2- 172.21.15.62:0/4195026347 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f80fc06ecc0 0x7f80fc071170 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f8108005950 tx=0x7f81080058e0 comp rx=0 tx=0).stop 2024-04-20T22:07:50.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:50.269+0000 7f811ba00700 1 -- 172.21.15.62:0/4195026347 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8114104130 msgr2=0x7f8114198560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:07:50.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:50.269+0000 7f811ba00700 1 --2- 172.21.15.62:0/4195026347 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8114104130 0x7f8114198560 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f8100006010 tx=0x7f810000be20 comp rx=0 tx=0).stop 2024-04-20T22:07:50.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:50.269+0000 7f811ba00700 1 -- 172.21.15.62:0/4195026347 shutdown_connections 2024-04-20T22:07:50.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:50.269+0000 7f811ba00700 1 --2- 172.21.15.62:0/4195026347 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f80fc06ecc0 0x7f80fc071170 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:07:50.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:50.269+0000 7f811ba00700 1 --2- 172.21.15.62:0/4195026347 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f81140ffbd0 0x7f8114198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:07:50.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:50.269+0000 7f811ba00700 1 --2- 172.21.15.62:0/4195026347 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8114104130 0x7f8114198560 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:07:50.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:50.269+0000 7f811ba00700 1 -- 172.21.15.62:0/4195026347 >> 172.21.15.62:0/4195026347 conn(0x7f8114075130 msgr2=0x7f81140fea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:07:50.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:50.270+0000 7f811ba00700 1 -- 172.21.15.62:0/4195026347 shutdown_connections 2024-04-20T22:07:50.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:50.270+0000 7f811ba00700 1 -- 172.21.15.62:0/4195026347 wait complete. 2024-04-20T22:07:50.286 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T22:07:50.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:50.435+0000 7ff7aae00700 1 -- 172.21.15.62:0/1359499832 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff7a4108470 msgr2=0x7ff7a4108840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:07:50.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:50.435+0000 7ff7aae00700 1 --2- 172.21.15.62:0/1359499832 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff7a4108470 0x7ff7a4108840 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7ff794009910 tx=0x7ff794009c20 comp rx=0 tx=0).stop 2024-04-20T22:07:50.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:50.435+0000 7ff7aae00700 1 -- 172.21.15.62:0/1359499832 shutdown_connections 2024-04-20T22:07:50.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:50.435+0000 7ff7aae00700 1 --2- 172.21.15.62:0/1359499832 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff7a4102470 0x7ff7a41028e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:07:50.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:50.435+0000 7ff7aae00700 1 --2- 172.21.15.62:0/1359499832 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff7a4108470 0x7ff7a4108840 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:07:50.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:50.435+0000 7ff7aae00700 1 -- 172.21.15.62:0/1359499832 >> 172.21.15.62:0/1359499832 conn(0x7ff7a40fdf90 msgr2=0x7ff7a41003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:07:50.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:50.435+0000 7ff7aae00700 1 -- 172.21.15.62:0/1359499832 shutdown_connections 2024-04-20T22:07:50.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:50.435+0000 7ff7aae00700 1 -- 172.21.15.62:0/1359499832 wait complete. 2024-04-20T22:07:50.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:50.436+0000 7ff7aae00700 1 Processor -- start 2024-04-20T22:07:50.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:50.436+0000 7ff7aae00700 1 -- start start 2024-04-20T22:07:50.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:50.437+0000 7ff7aae00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff7a4102470 0x7ff7a41981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:07:50.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:50.437+0000 7ff7aae00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff7a4108470 0x7ff7a4198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:07:50.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:50.437+0000 7ff7aae00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7a4198e10 con 0x7ff7a4108470 2024-04-20T22:07:50.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:50.437+0000 7ff7aae00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7a419cb30 con 0x7ff7a4102470 2024-04-20T22:07:50.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:50.437+0000 7ff7a3400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff7a4108470 0x7ff7a4198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:07:50.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:50.437+0000 7ff7a3400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff7a4108470 0x7ff7a4198730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54190/0 (socket says 172.21.15.62:54190) 2024-04-20T22:07:50.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:50.437+0000 7ff7a3400700 1 -- 172.21.15.62:0/2680862131 learned_addr learned my addr 172.21.15.62:0/2680862131 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:07:50.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:50.437+0000 7ff7a3e00700 1 --2- 172.21.15.62:0/2680862131 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff7a4102470 0x7ff7a41981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:07:50.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:50.437+0000 7ff7a3400700 1 -- 172.21.15.62:0/2680862131 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff7a4102470 msgr2=0x7ff7a41981f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:07:50.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:50.437+0000 7ff7a3400700 1 --2- 172.21.15.62:0/2680862131 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff7a4102470 0x7ff7a41981f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:07:50.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:50.437+0000 7ff7a3400700 1 -- 172.21.15.62:0/2680862131 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7940095f0 con 0x7ff7a4108470 2024-04-20T22:07:50.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:50.438+0000 7ff7a3400700 1 --2- 172.21.15.62:0/2680862131 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff7a4108470 0x7ff7a4198730 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7ff79000d8d0 tx=0x7ff79000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:07:50.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:50.438+0000 7ff7a0c00700 1 -- 172.21.15.62:0/2680862131 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff790009940 con 0x7ff7a4108470 2024-04-20T22:07:50.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:50.438+0000 7ff7aae00700 1 -- 172.21.15.62:0/2680862131 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7a419ce10 con 0x7ff7a4108470 2024-04-20T22:07:50.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:50.438+0000 7ff7a0c00700 1 -- 172.21.15.62:0/2680862131 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff790010460 con 0x7ff7a4108470 2024-04-20T22:07:50.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:50.438+0000 7ff7aae00700 1 -- 172.21.15.62:0/2680862131 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7a419d360 con 0x7ff7a4108470 2024-04-20T22:07:50.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:50.438+0000 7ff7a0c00700 1 -- 172.21.15.62:0/2680862131 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff79001b620 con 0x7ff7a4108470 2024-04-20T22:07:50.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:50.439+0000 7ff7aae00700 1 -- 172.21.15.62:0/2680862131 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7a410a9e0 con 0x7ff7a4108470 2024-04-20T22:07:50.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:50.441+0000 7ff7a0c00700 1 -- 172.21.15.62:0/2680862131 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff7900105d0 con 0x7ff7a4108470 2024-04-20T22:07:50.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:50.441+0000 7ff7a0c00700 1 --2- 172.21.15.62:0/2680862131 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff78806ecc0 0x7ff788071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:07:50.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:50.442+0000 7ff7a0c00700 1 -- 172.21.15.62:0/2680862131 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7ff79008f230 con 0x7ff7a4108470 2024-04-20T22:07:50.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:50.442+0000 7ff7a3e00700 1 --2- 172.21.15.62:0/2680862131 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff78806ecc0 0x7ff788071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:07:50.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:50.443+0000 7ff7a3e00700 1 --2- 172.21.15.62:0/2680862131 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff78806ecc0 0x7ff788071170 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7ff79400b5c0 tx=0x7ff79401a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:07:50.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:50.447+0000 7ff7a0c00700 1 -- 172.21.15.62:0/2680862131 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7ff790057f40 con 0x7ff7a4108470 2024-04-20T22:07:50.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:07:50 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:07:50.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:50.703+0000 7ff7aae00700 1 -- 172.21.15.62:0/2680862131 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff7a4199550 con 0x7ff78806ecc0 2024-04-20T22:07:50.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:50.705+0000 7ff7a0c00700 1 -- 172.21.15.62:0/2680862131 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7ff7a4199550 con 0x7ff78806ecc0 2024-04-20T22:07:50.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:50.708+0000 7ff7aae00700 1 -- 172.21.15.62:0/2680862131 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff78806ecc0 msgr2=0x7ff788071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:07:50.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:50.708+0000 7ff7aae00700 1 --2- 172.21.15.62:0/2680862131 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff78806ecc0 0x7ff788071170 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7ff79400b5c0 tx=0x7ff79401a040 comp rx=0 tx=0).stop 2024-04-20T22:07:50.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:50.708+0000 7ff7aae00700 1 -- 172.21.15.62:0/2680862131 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff7a4108470 msgr2=0x7ff7a4198730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:07:50.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:50.708+0000 7ff7aae00700 1 --2- 172.21.15.62:0/2680862131 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff7a4108470 0x7ff7a4198730 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7ff79000d8d0 tx=0x7ff79000dc90 comp rx=0 tx=0).stop 2024-04-20T22:07:50.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:50.708+0000 7ff7aae00700 1 -- 172.21.15.62:0/2680862131 shutdown_connections 2024-04-20T22:07:50.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:50.708+0000 7ff7aae00700 1 --2- 172.21.15.62:0/2680862131 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff78806ecc0 0x7ff788071170 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:07:50.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:50.708+0000 7ff7aae00700 1 --2- 172.21.15.62:0/2680862131 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff7a4102470 0x7ff7a41981f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:07:50.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:50.708+0000 7ff7aae00700 1 --2- 172.21.15.62:0/2680862131 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff7a4108470 0x7ff7a4198730 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:07:50.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:50.708+0000 7ff7aae00700 1 -- 172.21.15.62:0/2680862131 >> 172.21.15.62:0/2680862131 conn(0x7ff7a40fdf90 msgr2=0x7ff7a41002c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:07:50.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:50.709+0000 7ff7aae00700 1 -- 172.21.15.62:0/2680862131 shutdown_connections 2024-04-20T22:07:50.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:50.709+0000 7ff7aae00700 1 -- 172.21.15.62:0/2680862131 wait complete. 2024-04-20T22:07:50.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:50.874+0000 7ff002400700 1 -- 172.21.15.62:0/4284837662 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feffc10c8f0 msgr2=0x7feffc10ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:07:50.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:50.874+0000 7ff002400700 1 --2- 172.21.15.62:0/4284837662 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feffc10c8f0 0x7feffc10ccc0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fefec009910 tx=0x7fefec009c20 comp rx=0 tx=0).stop 2024-04-20T22:07:50.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:50.875+0000 7ff002400700 1 -- 172.21.15.62:0/4284837662 shutdown_connections 2024-04-20T22:07:50.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:50.875+0000 7ff002400700 1 --2- 172.21.15.62:0/4284837662 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feffc106b80 0x7feffc106ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:07:50.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:50.875+0000 7ff002400700 1 --2- 172.21.15.62:0/4284837662 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feffc10c8f0 0x7feffc10ccc0 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:07:50.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:50.875+0000 7ff002400700 1 -- 172.21.15.62:0/4284837662 >> 172.21.15.62:0/4284837662 conn(0x7feffc076060 msgr2=0x7feffc078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:07:50.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:50.875+0000 7ff002400700 1 -- 172.21.15.62:0/4284837662 shutdown_connections 2024-04-20T22:07:50.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:50.875+0000 7ff002400700 1 -- 172.21.15.62:0/4284837662 wait complete. 2024-04-20T22:07:50.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:50.876+0000 7ff002400700 1 Processor -- start 2024-04-20T22:07:50.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:50.876+0000 7ff002400700 1 -- start start 2024-04-20T22:07:50.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:50.877+0000 7ff002400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feffc106b80 0x7feffc19c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:07:50.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:50.877+0000 7ff002400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feffc10c8f0 0x7feffc19ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:07:50.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:50.877+0000 7ff002400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feffc19d120 con 0x7feffc10c8f0 2024-04-20T22:07:50.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:50.877+0000 7ff002400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feffc1a0eb0 con 0x7feffc106b80 2024-04-20T22:07:50.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:50.877+0000 7feffaa00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feffc10c8f0 0x7feffc19ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:07:50.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:50.877+0000 7feffaa00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feffc10c8f0 0x7feffc19ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54210/0 (socket says 172.21.15.62:54210) 2024-04-20T22:07:50.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:50.877+0000 7feffaa00700 1 -- 172.21.15.62:0/2370994403 learned_addr learned my addr 172.21.15.62:0/2370994403 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:07:50.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:50.877+0000 7feffb400700 1 --2- 172.21.15.62:0/2370994403 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feffc106b80 0x7feffc19c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:07:50.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:50.877+0000 7feffaa00700 1 -- 172.21.15.62:0/2370994403 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feffc106b80 msgr2=0x7feffc19c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:07:50.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:50.877+0000 7feffaa00700 1 --2- 172.21.15.62:0/2370994403 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feffc106b80 0x7feffc19c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:07:50.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:50.877+0000 7feffaa00700 1 -- 172.21.15.62:0/2370994403 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefec0095f0 con 0x7feffc10c8f0 2024-04-20T22:07:50.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:50.878+0000 7feffaa00700 1 --2- 172.21.15.62:0/2370994403 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feffc10c8f0 0x7feffc19ca40 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fefe800d8d0 tx=0x7fefe800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:07:50.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:50.878+0000 7feff3e00700 1 -- 172.21.15.62:0/2370994403 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fefe8009940 con 0x7feffc10c8f0 2024-04-20T22:07:50.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:50.878+0000 7feff3e00700 1 -- 172.21.15.62:0/2370994403 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fefe8010460 con 0x7feffc10c8f0 2024-04-20T22:07:50.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:50.878+0000 7ff002400700 1 -- 172.21.15.62:0/2370994403 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feffc1a1190 con 0x7feffc10c8f0 2024-04-20T22:07:50.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:50.878+0000 7feff3e00700 1 -- 172.21.15.62:0/2370994403 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fefe800f5f0 con 0x7feffc10c8f0 2024-04-20T22:07:50.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:50.878+0000 7ff002400700 1 -- 172.21.15.62:0/2370994403 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feffc1a16e0 con 0x7feffc10c8f0 2024-04-20T22:07:50.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:50.879+0000 7ff002400700 1 -- 172.21.15.62:0/2370994403 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feffc109b20 con 0x7feffc10c8f0 2024-04-20T22:07:50.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:50.880+0000 7feff3e00700 1 -- 172.21.15.62:0/2370994403 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fefe80105d0 con 0x7feffc10c8f0 2024-04-20T22:07:50.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:50.881+0000 7feff3e00700 1 --2- 172.21.15.62:0/2370994403 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fefe006ecc0 0x7fefe0071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:07:50.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:50.881+0000 7feff3e00700 1 -- 172.21.15.62:0/2370994403 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fefe808ef10 con 0x7feffc10c8f0 2024-04-20T22:07:50.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:50.882+0000 7feffb400700 1 --2- 172.21.15.62:0/2370994403 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fefe006ecc0 0x7fefe0071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:07:50.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:50.882+0000 7feffb400700 1 --2- 172.21.15.62:0/2370994403 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fefe006ecc0 0x7fefe0071170 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fefec006010 tx=0x7fefec01a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:07:50.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:50.887+0000 7feff3e00700 1 -- 172.21.15.62:0/2370994403 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fefe8057c20 con 0x7feffc10c8f0 2024-04-20T22:07:51.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:51.133+0000 7ff002400700 1 -- 172.21.15.62:0/2370994403 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7feffc19d970 con 0x7fefe006ecc0 2024-04-20T22:07:51.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:51.153+0000 7feff3e00700 1 -- 172.21.15.62:0/2370994403 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7feffc19d970 con 0x7fefe006ecc0 2024-04-20T22:07:51.154 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T22:07:51.154 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (33m) 9m ago 47m 15.3M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T22:07:51.154 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (47m) 9m ago 47m 10.4M - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T22:07:51.154 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (46m) 9m ago 46m 10.1M - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T22:07:51.154 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (29m) 9m ago 47m 6660k - 19.0.0-3133-ga8c84f59 76bc6b91b726 2825d5ba11c6 2024-04-20T22:07:51.154 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (29m) 9m ago 46m 6665k - 19.0.0-3133-ga8c84f59 76bc6b91b726 20f746fa1bba 2024-04-20T22:07:51.154 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (33m) 9m ago 47m 73.5M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T22:07:51.154 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (25m) 9m ago 40m 18.5M - 19.0.0-3133-ga8c84f59 76bc6b91b726 de80bc90a683 2024-04-20T22:07:51.154 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (25m) 9m ago 40m 18.7M - 19.0.0-3133-ga8c84f59 76bc6b91b726 cd793b7ee98e 2024-04-20T22:07:51.154 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (9m) 9m ago 40m 11.0M - 19.0.0-3133-ga8c84f59 76bc6b91b726 0e7b0bc4d1ab 2024-04-20T22:07:51.154 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (40m) 9m ago 40m 18.6M - 18.2.0 dc2bc1663786 f2eec2ee8951 2024-04-20T22:07:51.154 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (36m) 9m ago 49m 513M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T22:07:51.155 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (35m) 9m ago 46m 459M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T22:07:51.155 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (31m) 9m ago 49m 71.7M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 019eb1479485 2024-04-20T22:07:51.155 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (30m) 9m ago 46m 51.0M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 55aac560aab6 2024-04-20T22:07:51.155 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (34m) 9m ago 47m 8472k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T22:07:51.155 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (34m) 9m ago 46m 8048k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T22:07:51.155 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (29m) 9m ago 45m 91.2M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 ee2bb901f26e 2024-04-20T22:07:51.155 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (28m) 9m ago 44m 94.6M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 9075152d4162 2024-04-20T22:07:51.155 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (28m) 9m ago 43m 78.1M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 2d4f3e38dc0e 2024-04-20T22:07:51.155 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (27m) 9m ago 43m 92.6M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 8f97b17ae0eb 2024-04-20T22:07:51.155 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (26m) 9m ago 42m 95.6M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 653d8079967d 2024-04-20T22:07:51.155 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (26m) 9m ago 42m 89.2M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 596dded14c12 2024-04-20T22:07:51.155 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (34m) 9m ago 46m 82.1M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T22:07:51.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:51.157+0000 7ff002400700 1 -- 172.21.15.62:0/2370994403 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fefe006ecc0 msgr2=0x7fefe0071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:07:51.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:51.157+0000 7ff002400700 1 --2- 172.21.15.62:0/2370994403 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fefe006ecc0 0x7fefe0071170 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fefec006010 tx=0x7fefec01a040 comp rx=0 tx=0).stop 2024-04-20T22:07:51.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:51.157+0000 7ff002400700 1 -- 172.21.15.62:0/2370994403 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feffc10c8f0 msgr2=0x7feffc19ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:07:51.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:51.157+0000 7ff002400700 1 --2- 172.21.15.62:0/2370994403 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feffc10c8f0 0x7feffc19ca40 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fefe800d8d0 tx=0x7fefe800dc90 comp rx=0 tx=0).stop 2024-04-20T22:07:51.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:51.157+0000 7ff002400700 1 -- 172.21.15.62:0/2370994403 shutdown_connections 2024-04-20T22:07:51.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:51.157+0000 7ff002400700 1 --2- 172.21.15.62:0/2370994403 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fefe006ecc0 0x7fefe0071170 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:07:51.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:51.157+0000 7ff002400700 1 --2- 172.21.15.62:0/2370994403 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feffc106b80 0x7feffc19c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:07:51.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:51.157+0000 7ff002400700 1 --2- 172.21.15.62:0/2370994403 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feffc10c8f0 0x7feffc19ca40 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:07:51.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:51.157+0000 7ff002400700 1 -- 172.21.15.62:0/2370994403 >> 172.21.15.62:0/2370994403 conn(0x7feffc076060 msgr2=0x7feffc077aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:07:51.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:51.158+0000 7ff002400700 1 -- 172.21.15.62:0/2370994403 shutdown_connections 2024-04-20T22:07:51.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:51.158+0000 7ff002400700 1 -- 172.21.15.62:0/2370994403 wait complete. 2024-04-20T22:07:51.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:51.319+0000 7f9c0be00700 1 -- 172.21.15.62:0/1143911272 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9c04075b90 msgr2=0x7f9c04110bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:07:51.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:51.319+0000 7f9c0be00700 1 --2- 172.21.15.62:0/1143911272 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9c04075b90 0x7f9c04110bd0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f9bf0009910 tx=0x7f9bf0009c20 comp rx=0 tx=0).stop 2024-04-20T22:07:51.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:51.319+0000 7f9c0be00700 1 -- 172.21.15.62:0/1143911272 shutdown_connections 2024-04-20T22:07:51.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:51.319+0000 7f9c0be00700 1 --2- 172.21.15.62:0/1143911272 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9c04075b90 0x7f9c04110bd0 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:07:51.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:51.319+0000 7f9c0be00700 1 --2- 172.21.15.62:0/1143911272 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9c04075280 0x7f9c04075650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:07:51.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:51.319+0000 7f9c0be00700 1 -- 172.21.15.62:0/1143911272 >> 172.21.15.62:0/1143911272 conn(0x7f9c040fdf90 msgr2=0x7f9c041003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:07:51.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:51.319+0000 7f9c0be00700 1 -- 172.21.15.62:0/1143911272 shutdown_connections 2024-04-20T22:07:51.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:51.319+0000 7f9c0be00700 1 -- 172.21.15.62:0/1143911272 wait complete. 2024-04-20T22:07:51.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:51.320+0000 7f9c0be00700 1 Processor -- start 2024-04-20T22:07:51.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:51.320+0000 7f9c0be00700 1 -- start start 2024-04-20T22:07:51.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:51.321+0000 7f9c0be00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9c04075280 0x7f9c0419c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:07:51.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:51.321+0000 7f9c0be00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9c04075b90 0x7f9c0419ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:07:51.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:51.321+0000 7f9c0be00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c0419d100 con 0x7f9c04075280 2024-04-20T22:07:51.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:51.321+0000 7f9c0be00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c041a0e90 con 0x7f9c04075b90 2024-04-20T22:07:51.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:51.321+0000 7f9c09400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9c04075280 0x7f9c0419c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:07:51.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:51.321+0000 7f9c09400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9c04075280 0x7f9c0419c4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54222/0 (socket says 172.21.15.62:54222) 2024-04-20T22:07:51.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:51.321+0000 7f9c09400700 1 -- 172.21.15.62:0/349064892 learned_addr learned my addr 172.21.15.62:0/349064892 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:07:51.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:51.321+0000 7f9c08a00700 1 --2- 172.21.15.62:0/349064892 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9c04075b90 0x7f9c0419ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:07:51.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:51.322+0000 7f9c09400700 1 -- 172.21.15.62:0/349064892 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9c04075b90 msgr2=0x7f9c0419ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:07:51.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:51.322+0000 7f9c09400700 1 --2- 172.21.15.62:0/349064892 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9c04075b90 0x7f9c0419ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:07:51.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:51.322+0000 7f9c09400700 1 -- 172.21.15.62:0/349064892 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bf00095f0 con 0x7f9c04075280 2024-04-20T22:07:51.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:51.322+0000 7f9c09400700 1 --2- 172.21.15.62:0/349064892 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9c04075280 0x7f9c0419c4e0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f9bf800ada0 tx=0x7f9bf800c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:07:51.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:51.322+0000 7f9c02000700 1 -- 172.21.15.62:0/349064892 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9bf800f070 con 0x7f9c04075280 2024-04-20T22:07:51.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:51.322+0000 7f9c02000700 1 -- 172.21.15.62:0/349064892 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9bf8007bf0 con 0x7f9c04075280 2024-04-20T22:07:51.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:51.322+0000 7f9c02000700 1 -- 172.21.15.62:0/349064892 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9bf8005690 con 0x7f9c04075280 2024-04-20T22:07:51.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:51.322+0000 7f9c0be00700 1 -- 172.21.15.62:0/349064892 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c041a1170 con 0x7f9c04075280 2024-04-20T22:07:51.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:51.323+0000 7f9c0be00700 1 -- 172.21.15.62:0/349064892 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c041a16c0 con 0x7f9c04075280 2024-04-20T22:07:51.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:51.324+0000 7f9c0be00700 1 -- 172.21.15.62:0/349064892 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c0410e340 con 0x7f9c04075280 2024-04-20T22:07:51.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:51.325+0000 7f9c02000700 1 -- 172.21.15.62:0/349064892 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f9bf8005850 con 0x7f9c04075280 2024-04-20T22:07:51.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:51.326+0000 7f9c02000700 1 --2- 172.21.15.62:0/349064892 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9bec06ecc0 0x7f9bec071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:07:51.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:51.326+0000 7f9c02000700 1 -- 172.21.15.62:0/349064892 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f9bf808e860 con 0x7f9c04075280 2024-04-20T22:07:51.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:51.326+0000 7f9c08a00700 1 --2- 172.21.15.62:0/349064892 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9bec06ecc0 0x7f9bec071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:07:51.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:51.327+0000 7f9c08a00700 1 --2- 172.21.15.62:0/349064892 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9bec06ecc0 0x7f9bec071170 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f9c0419db00 tx=0x7f9bf000b540 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:07:51.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:51.331+0000 7f9c02000700 1 -- 172.21.15.62:0/349064892 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f9bf80575f0 con 0x7f9c04075280 2024-04-20T22:07:51.331 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:07:51 smithi062 ceph-mon[76322]: pgmap v977: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:07:51.331 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:07:51 smithi062 ceph-mon[76322]: from='client.35740 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:07:51.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:07:51 smithi189 ceph-mon[57022]: pgmap v977: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:07:51.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:07:51 smithi189 ceph-mon[57022]: from='client.35740 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:07:51.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:51.659+0000 7f9c0be00700 1 -- 172.21.15.62:0/349064892 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f9c0419d910 con 0x7f9c04075280 2024-04-20T22:07:51.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:51.660+0000 7f9c02000700 1 -- 172.21.15.62:0/349064892 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7f9bf805acb0 con 0x7f9c04075280 2024-04-20T22:07:51.661 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:07:51.661 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T22:07:51.661 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:07:51.661 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:07:51.661 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T22:07:51.661 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:07:51.661 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:07:51.661 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T22:07:51.661 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 6 2024-04-20T22:07:51.661 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:07:51.662 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T22:07:51.662 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 1, 2024-04-20T22:07:51.662 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 3 2024-04-20T22:07:51.662 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:07:51.662 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T22:07:51.662 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 1, 2024-04-20T22:07:51.662 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 13 2024-04-20T22:07:51.662 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T22:07:51.662 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:07:51.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:51.664+0000 7f9c0be00700 1 -- 172.21.15.62:0/349064892 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9bec06ecc0 msgr2=0x7f9bec071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:07:51.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:51.664+0000 7f9c0be00700 1 --2- 172.21.15.62:0/349064892 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9bec06ecc0 0x7f9bec071170 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f9c0419db00 tx=0x7f9bf000b540 comp rx=0 tx=0).stop 2024-04-20T22:07:51.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:51.664+0000 7f9c0be00700 1 -- 172.21.15.62:0/349064892 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9c04075280 msgr2=0x7f9c0419c4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:07:51.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:51.664+0000 7f9c0be00700 1 --2- 172.21.15.62:0/349064892 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9c04075280 0x7f9c0419c4e0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f9bf800ada0 tx=0x7f9bf800c330 comp rx=0 tx=0).stop 2024-04-20T22:07:51.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:51.664+0000 7f9c0be00700 1 -- 172.21.15.62:0/349064892 shutdown_connections 2024-04-20T22:07:51.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:51.664+0000 7f9c0be00700 1 --2- 172.21.15.62:0/349064892 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9bec06ecc0 0x7f9bec071170 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:07:51.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:51.664+0000 7f9c0be00700 1 --2- 172.21.15.62:0/349064892 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9c04075280 0x7f9c0419c4e0 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:07:51.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:51.664+0000 7f9c0be00700 1 --2- 172.21.15.62:0/349064892 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9c04075b90 0x7f9c0419ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:07:51.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:51.664+0000 7f9c0be00700 1 -- 172.21.15.62:0/349064892 >> 172.21.15.62:0/349064892 conn(0x7f9c040fdf90 msgr2=0x7f9c040febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:07:51.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:51.664+0000 7f9c0be00700 1 -- 172.21.15.62:0/349064892 shutdown_connections 2024-04-20T22:07:51.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:51.665+0000 7f9c0be00700 1 -- 172.21.15.62:0/349064892 wait complete. 2024-04-20T22:07:51.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:51.828+0000 7f7b87200700 1 -- 172.21.15.62:0/2689156392 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7b881061b0 msgr2=0x7f7b88106580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:07:51.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:51.828+0000 7f7b87200700 1 --2- 172.21.15.62:0/2689156392 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7b881061b0 0x7f7b88106580 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f7b78009910 tx=0x7f7b78009c20 comp rx=0 tx=0).stop 2024-04-20T22:07:51.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:51.828+0000 7f7b87200700 1 -- 172.21.15.62:0/2689156392 shutdown_connections 2024-04-20T22:07:51.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:51.828+0000 7f7b87200700 1 --2- 172.21.15.62:0/2689156392 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7b88068540 0x7f7b880689b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:07:51.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:51.828+0000 7f7b87200700 1 --2- 172.21.15.62:0/2689156392 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7b881061b0 0x7f7b88106580 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:07:51.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:51.828+0000 7f7b87200700 1 -- 172.21.15.62:0/2689156392 >> 172.21.15.62:0/2689156392 conn(0x7f7b88074fa0 msgr2=0x7f7b880753a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:07:51.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:51.828+0000 7f7b87200700 1 -- 172.21.15.62:0/2689156392 shutdown_connections 2024-04-20T22:07:51.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:51.828+0000 7f7b87200700 1 -- 172.21.15.62:0/2689156392 wait complete. 2024-04-20T22:07:51.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:51.830+0000 7f7b87200700 1 Processor -- start 2024-04-20T22:07:51.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:51.830+0000 7f7b87200700 1 -- start start 2024-04-20T22:07:51.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:51.830+0000 7f7b87200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7b88068540 0x7f7b88197d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:07:51.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:51.830+0000 7f7b87200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7b881061b0 0x7f7b881982a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:07:51.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:51.830+0000 7f7b87200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b88198980 con 0x7f7b88068540 2024-04-20T22:07:51.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:51.830+0000 7f7b87200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b8819c710 con 0x7f7b881061b0 2024-04-20T22:07:51.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:51.830+0000 7f7b85e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7b88068540 0x7f7b88197d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:07:51.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:51.830+0000 7f7b85e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7b88068540 0x7f7b88197d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35334/0 (socket says 172.21.15.62:35334) 2024-04-20T22:07:51.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:51.830+0000 7f7b85e00700 1 -- 172.21.15.62:0/3743444342 learned_addr learned my addr 172.21.15.62:0/3743444342 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:07:51.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:51.831+0000 7f7b85e00700 1 -- 172.21.15.62:0/3743444342 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7b881061b0 msgr2=0x7f7b881982a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:07:51.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:51.831+0000 7f7b85e00700 1 --2- 172.21.15.62:0/3743444342 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7b881061b0 0x7f7b881982a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:07:51.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:51.831+0000 7f7b85e00700 1 -- 172.21.15.62:0/3743444342 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b780095f0 con 0x7f7b88068540 2024-04-20T22:07:51.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:51.831+0000 7f7b85e00700 1 --2- 172.21.15.62:0/3743444342 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7b88068540 0x7f7b88197d60 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f7b78006010 tx=0x7f7b7800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:07:51.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:51.831+0000 7f7b7ea00700 1 -- 172.21.15.62:0/3743444342 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b7801d070 con 0x7f7b88068540 2024-04-20T22:07:51.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:51.832+0000 7f7b87200700 1 -- 172.21.15.62:0/3743444342 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b8819c990 con 0x7f7b88068540 2024-04-20T22:07:51.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:51.832+0000 7f7b7ea00700 1 -- 172.21.15.62:0/3743444342 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7b78004fb0 con 0x7f7b88068540 2024-04-20T22:07:51.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:51.832+0000 7f7b7ea00700 1 -- 172.21.15.62:0/3743444342 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b78022ea0 con 0x7f7b88068540 2024-04-20T22:07:51.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:51.832+0000 7f7b87200700 1 -- 172.21.15.62:0/3743444342 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b8819cf00 con 0x7f7b88068540 2024-04-20T22:07:51.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:51.833+0000 7f7b87200700 1 -- 172.21.15.62:0/3743444342 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b88062380 con 0x7f7b88068540 2024-04-20T22:07:51.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:51.834+0000 7f7b7ea00700 1 -- 172.21.15.62:0/3743444342 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f7b78022470 con 0x7f7b88068540 2024-04-20T22:07:51.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:51.835+0000 7f7b7ea00700 1 --2- 172.21.15.62:0/3743444342 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7b6c06ecc0 0x7f7b6c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:07:51.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:51.835+0000 7f7b7ea00700 1 -- 172.21.15.62:0/3743444342 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f7b78090490 con 0x7f7b88068540 2024-04-20T22:07:51.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:51.835+0000 7f7b85400700 1 --2- 172.21.15.62:0/3743444342 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7b6c06ecc0 0x7f7b6c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:07:51.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:51.836+0000 7f7b85400700 1 --2- 172.21.15.62:0/3743444342 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7b6c06ecc0 0x7f7b6c071170 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f7b88199380 tx=0x7f7b74005c80 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:07:51.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:51.840+0000 7f7b7ea00700 1 -- 172.21.15.62:0/3743444342 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f7b78059250 con 0x7f7b88068540 2024-04-20T22:07:52.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:52.123+0000 7f7b87200700 1 -- 172.21.15.62:0/3743444342 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f7b8819d330 con 0x7f7b88068540 2024-04-20T22:07:52.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:52.124+0000 7f7b7ea00700 1 -- 172.21.15.62:0/3743444342 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 27 v27) v1 ==== 76+0+1888 (secure 0 0 0) 0x7f7b78027770 con 0x7f7b88068540 2024-04-20T22:07:52.125 INFO:teuthology.orchestra.run.smithi062.stdout:e27 2024-04-20T22:07:52.125 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T22:07:52.125 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:07:52.125 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T22:07:52.125 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:07:52.125 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T22:07:52.125 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T22:07:52.125 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 20 2024-04-20T22:07:52.125 INFO:teuthology.orchestra.run.smithi062.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-20T22:07:52.125 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T22:07:52.125 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T21:42:01.559061+0000 2024-04-20T22:07:52.125 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T22:07:52.126 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T22:07:52.126 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T22:07:52.126 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T22:07:52.126 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T22:07:52.126 INFO:teuthology.orchestra.run.smithi062.stdout:max_xattr_size 65536 2024-04-20T22:07:52.126 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T22:07:52.126 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T22:07:52.126 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 81 2024-04-20T22:07:52.126 INFO:teuthology.orchestra.run.smithi062.stdout:snap_rstat 1 2024-04-20T22:07:52.126 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:07:52.126 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 2 2024-04-20T22:07:52.126 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1 2024-04-20T22:07:52.126 INFO:teuthology.orchestra.run.smithi062.stdout:up {} 2024-04-20T22:07:52.126 INFO:teuthology.orchestra.run.smithi062.stdout:failed 0,1 2024-04-20T22:07:52.127 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T22:07:52.127 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T22:07:52.127 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T22:07:52.127 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T22:07:52.127 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data enabled 2024-04-20T22:07:52.127 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T22:07:52.127 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T22:07:52.127 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 1 2024-04-20T22:07:52.127 INFO:teuthology.orchestra.run.smithi062.stdout:qdb_cluster leader: 0 members: 2024-04-20T22:07:52.127 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:07:52.127 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:07:52.127 INFO:teuthology.orchestra.run.smithi062.stdout:Standby daemons: 2024-04-20T22:07:52.127 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:07:52.127 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ryqblb{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6826/4113916205,v1:172.21.15.62:6827/4113916205] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:07:52.127 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ozizwi{-1:34476} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6828/3801975165,v1:172.21.15.62:6829/3801975165] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:07:52.127 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.xbgspw{-1:44285} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6826/3996040526,v1:172.21.15.189:6827/3996040526] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T22:07:52.128 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.dvimer{-1:44727} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6824/3576699232,v1:172.21.15.189:6825/3576699232] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:07:52.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:52.128+0000 7f7b87200700 1 -- 172.21.15.62:0/3743444342 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7b6c06ecc0 msgr2=0x7f7b6c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:07:52.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:52.128+0000 7f7b87200700 1 --2- 172.21.15.62:0/3743444342 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7b6c06ecc0 0x7f7b6c071170 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f7b88199380 tx=0x7f7b74005c80 comp rx=0 tx=0).stop 2024-04-20T22:07:52.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:52.128+0000 7f7b87200700 1 -- 172.21.15.62:0/3743444342 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7b88068540 msgr2=0x7f7b88197d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:07:52.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:52.128+0000 7f7b87200700 1 --2- 172.21.15.62:0/3743444342 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7b88068540 0x7f7b88197d60 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f7b78006010 tx=0x7f7b7800be20 comp rx=0 tx=0).stop 2024-04-20T22:07:52.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:52.129+0000 7f7b87200700 1 -- 172.21.15.62:0/3743444342 shutdown_connections 2024-04-20T22:07:52.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:52.129+0000 7f7b87200700 1 --2- 172.21.15.62:0/3743444342 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7b6c06ecc0 0x7f7b6c071170 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:07:52.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:52.129+0000 7f7b87200700 1 --2- 172.21.15.62:0/3743444342 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7b88068540 0x7f7b88197d60 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:07:52.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:52.129+0000 7f7b87200700 1 --2- 172.21.15.62:0/3743444342 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7b881061b0 0x7f7b881982a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:07:52.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:52.129+0000 7f7b87200700 1 -- 172.21.15.62:0/3743444342 >> 172.21.15.62:0/3743444342 conn(0x7f7b88074fa0 msgr2=0x7f7b880fe740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:07:52.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:52.129+0000 7f7b87200700 1 -- 172.21.15.62:0/3743444342 shutdown_connections 2024-04-20T22:07:52.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:52.129+0000 7f7b87200700 1 -- 172.21.15.62:0/3743444342 wait complete. 2024-04-20T22:07:52.130 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 27 2024-04-20T22:07:52.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:52.280+0000 7f90e4400700 1 -- 172.21.15.62:0/1551168242 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90dc10c8f0 msgr2=0x7f90dc10ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:07:52.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:52.280+0000 7f90e4400700 1 --2- 172.21.15.62:0/1551168242 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90dc10c8f0 0x7f90dc10ccc0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f90d0009910 tx=0x7f90d0009c20 comp rx=0 tx=0).stop 2024-04-20T22:07:52.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:52.280+0000 7f90e4400700 1 -- 172.21.15.62:0/1551168242 shutdown_connections 2024-04-20T22:07:52.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:52.280+0000 7f90e4400700 1 --2- 172.21.15.62:0/1551168242 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f90dc106b80 0x7f90dc106ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:07:52.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:52.280+0000 7f90e4400700 1 --2- 172.21.15.62:0/1551168242 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90dc10c8f0 0x7f90dc10ccc0 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:07:52.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:52.280+0000 7f90e4400700 1 -- 172.21.15.62:0/1551168242 >> 172.21.15.62:0/1551168242 conn(0x7f90dc076060 msgr2=0x7f90dc078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:07:52.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:52.280+0000 7f90e4400700 1 -- 172.21.15.62:0/1551168242 shutdown_connections 2024-04-20T22:07:52.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:52.280+0000 7f90e4400700 1 -- 172.21.15.62:0/1551168242 wait complete. 2024-04-20T22:07:52.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:52.282+0000 7f90e4400700 1 Processor -- start 2024-04-20T22:07:52.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:52.282+0000 7f90e4400700 1 -- start start 2024-04-20T22:07:52.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:52.282+0000 7f90e4400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90dc106b80 0x7f90dc19c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:07:52.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:52.283+0000 7f90e4400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f90dc10c8f0 0x7f90dc19ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:07:52.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:52.283+0000 7f90e4400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90dc19d120 con 0x7f90dc106b80 2024-04-20T22:07:52.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:52.283+0000 7f90e4400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90dc1a0eb0 con 0x7f90dc10c8f0 2024-04-20T22:07:52.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:52.283+0000 7f90e1a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90dc106b80 0x7f90dc19c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:07:52.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:52.283+0000 7f90e1a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90dc106b80 0x7f90dc19c500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35360/0 (socket says 172.21.15.62:35360) 2024-04-20T22:07:52.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:52.283+0000 7f90e1a00700 1 -- 172.21.15.62:0/2069003280 learned_addr learned my addr 172.21.15.62:0/2069003280 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:07:52.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:52.283+0000 7f90e1000700 1 --2- 172.21.15.62:0/2069003280 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f90dc10c8f0 0x7f90dc19ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:07:52.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:52.283+0000 7f90e1a00700 1 -- 172.21.15.62:0/2069003280 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f90dc10c8f0 msgr2=0x7f90dc19ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:07:52.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:52.283+0000 7f90e1a00700 1 --2- 172.21.15.62:0/2069003280 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f90dc10c8f0 0x7f90dc19ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:07:52.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:52.283+0000 7f90e1a00700 1 -- 172.21.15.62:0/2069003280 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90d00095f0 con 0x7f90dc106b80 2024-04-20T22:07:52.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:52.283+0000 7f90e1000700 1 --2- 172.21.15.62:0/2069003280 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f90dc10c8f0 0x7f90dc19ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T22:07:52.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:52.284+0000 7f90e1a00700 1 --2- 172.21.15.62:0/2069003280 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90dc106b80 0x7f90dc19c500 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f90d00092e0 tx=0x7f90d000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:07:52.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:52.284+0000 7f90da000700 1 -- 172.21.15.62:0/2069003280 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f90d001d070 con 0x7f90dc106b80 2024-04-20T22:07:52.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:52.284+0000 7f90da000700 1 -- 172.21.15.62:0/2069003280 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f90d0005e00 con 0x7f90dc106b80 2024-04-20T22:07:52.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:52.284+0000 7f90da000700 1 -- 172.21.15.62:0/2069003280 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f90d0017400 con 0x7f90dc106b80 2024-04-20T22:07:52.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:52.284+0000 7f90e4400700 1 -- 172.21.15.62:0/2069003280 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90dc1a1130 con 0x7f90dc106b80 2024-04-20T22:07:52.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:52.284+0000 7f90e4400700 1 -- 172.21.15.62:0/2069003280 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90dc1a1620 con 0x7f90dc106b80 2024-04-20T22:07:52.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:52.285+0000 7f90e4400700 1 -- 172.21.15.62:0/2069003280 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90dc109b20 con 0x7f90dc106b80 2024-04-20T22:07:52.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:52.287+0000 7f90da000700 1 -- 172.21.15.62:0/2069003280 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f90d000f460 con 0x7f90dc106b80 2024-04-20T22:07:52.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:52.287+0000 7f90da000700 1 --2- 172.21.15.62:0/2069003280 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f90c406ecc0 0x7f90c4071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:07:52.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:52.288+0000 7f90da000700 1 -- 172.21.15.62:0/2069003280 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f90d0090330 con 0x7f90dc106b80 2024-04-20T22:07:52.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:52.288+0000 7f90e1000700 1 --2- 172.21.15.62:0/2069003280 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f90c406ecc0 0x7f90c4071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:07:52.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:52.289+0000 7f90e1000700 1 --2- 172.21.15.62:0/2069003280 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f90c406ecc0 0x7f90c4071170 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f90c8005fd0 tx=0x7f90c8005ee0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:07:52.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:52.293+0000 7f90da000700 1 -- 172.21.15.62:0/2069003280 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f90d00590c0 con 0x7f90dc106b80 2024-04-20T22:07:52.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:07:52 smithi189 ceph-mon[57022]: from='client.35744 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:07:52.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:07:52 smithi189 ceph-mon[57022]: from='client.35748 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:07:52.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:07:52 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/349064892' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:07:52.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:07:52 smithi062 ceph-mon[76322]: from='client.35744 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:07:52.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:07:52 smithi062 ceph-mon[76322]: from='client.35748 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:07:52.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:07:52 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/349064892' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:07:52.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:52.555+0000 7f90e4400700 1 -- 172.21.15.62:0/2069003280 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f90dc1a1950 con 0x7f90c406ecc0 2024-04-20T22:07:52.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:52.558+0000 7f90da000700 1 -- 172.21.15.62:0/2069003280 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f90dc1a1950 con 0x7f90c406ecc0 2024-04-20T22:07:52.558 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:07:52.558 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T22:07:52.558 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T22:07:52.558 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-20T22:07:52.558 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T22:07:52.559 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr", 2024-04-20T22:07:52.559 INFO:teuthology.orchestra.run.smithi062.stdout: "crash", 2024-04-20T22:07:52.559 INFO:teuthology.orchestra.run.smithi062.stdout: "mon", 2024-04-20T22:07:52.559 INFO:teuthology.orchestra.run.smithi062.stdout: "osd" 2024-04-20T22:07:52.559 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T22:07:52.559 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "15/23 daemons upgraded", 2024-04-20T22:07:52.559 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading mds daemons", 2024-04-20T22:07:52.559 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T22:07:52.559 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:07:52.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:52.561+0000 7f90e4400700 1 -- 172.21.15.62:0/2069003280 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f90c406ecc0 msgr2=0x7f90c4071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:07:52.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:52.561+0000 7f90e4400700 1 --2- 172.21.15.62:0/2069003280 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f90c406ecc0 0x7f90c4071170 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f90c8005fd0 tx=0x7f90c8005ee0 comp rx=0 tx=0).stop 2024-04-20T22:07:52.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:52.561+0000 7f90e4400700 1 -- 172.21.15.62:0/2069003280 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90dc106b80 msgr2=0x7f90dc19c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:07:52.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:52.561+0000 7f90e4400700 1 --2- 172.21.15.62:0/2069003280 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90dc106b80 0x7f90dc19c500 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f90d00092e0 tx=0x7f90d000be20 comp rx=0 tx=0).stop 2024-04-20T22:07:52.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:52.562+0000 7f90e4400700 1 -- 172.21.15.62:0/2069003280 shutdown_connections 2024-04-20T22:07:52.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:52.562+0000 7f90e4400700 1 --2- 172.21.15.62:0/2069003280 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f90c406ecc0 0x7f90c4071170 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:07:52.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:52.562+0000 7f90e4400700 1 --2- 172.21.15.62:0/2069003280 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90dc106b80 0x7f90dc19c500 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:07:52.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:52.562+0000 7f90e4400700 1 --2- 172.21.15.62:0/2069003280 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f90dc10c8f0 0x7f90dc19ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:07:52.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:52.562+0000 7f90e4400700 1 -- 172.21.15.62:0/2069003280 >> 172.21.15.62:0/2069003280 conn(0x7f90dc076060 msgr2=0x7f90dc077aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:07:52.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:52.562+0000 7f90e4400700 1 -- 172.21.15.62:0/2069003280 shutdown_connections 2024-04-20T22:07:52.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:52.562+0000 7f90e4400700 1 -- 172.21.15.62:0/2069003280 wait complete. 2024-04-20T22:07:52.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:52.727+0000 7f50ce400700 1 -- 172.21.15.62:0/1194588550 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f50c81061b0 msgr2=0x7f50c8106580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:07:52.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:52.727+0000 7f50ce400700 1 --2- 172.21.15.62:0/1194588550 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f50c81061b0 0x7f50c8106580 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f50ac009910 tx=0x7f50ac009c20 comp rx=0 tx=0).stop 2024-04-20T22:07:52.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:52.727+0000 7f50ce400700 1 -- 172.21.15.62:0/1194588550 shutdown_connections 2024-04-20T22:07:52.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:52.727+0000 7f50ce400700 1 --2- 172.21.15.62:0/1194588550 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50c8068540 0x7f50c80689b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:07:52.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:52.727+0000 7f50ce400700 1 --2- 172.21.15.62:0/1194588550 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f50c81061b0 0x7f50c8106580 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:07:52.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:52.727+0000 7f50ce400700 1 -- 172.21.15.62:0/1194588550 >> 172.21.15.62:0/1194588550 conn(0x7f50c8074fa0 msgr2=0x7f50c80753a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:07:52.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:52.727+0000 7f50ce400700 1 -- 172.21.15.62:0/1194588550 shutdown_connections 2024-04-20T22:07:52.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:52.727+0000 7f50ce400700 1 -- 172.21.15.62:0/1194588550 wait complete. 2024-04-20T22:07:52.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:52.728+0000 7f50ce400700 1 Processor -- start 2024-04-20T22:07:52.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:52.728+0000 7f50ce400700 1 -- start start 2024-04-20T22:07:52.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:52.729+0000 7f50ce400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f50c8068540 0x7f50c8197d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:07:52.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:52.729+0000 7f50ce400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50c81061b0 0x7f50c81982a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:07:52.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:52.729+0000 7f50ce400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50c8198980 con 0x7f50c8068540 2024-04-20T22:07:52.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:52.729+0000 7f50ce400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50c819c710 con 0x7f50c81061b0 2024-04-20T22:07:52.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:52.729+0000 7f50cd000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f50c8068540 0x7f50c8197d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:07:52.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:52.729+0000 7f50cd000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f50c8068540 0x7f50c8197d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35374/0 (socket says 172.21.15.62:35374) 2024-04-20T22:07:52.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:52.729+0000 7f50cd000700 1 -- 172.21.15.62:0/1936147678 learned_addr learned my addr 172.21.15.62:0/1936147678 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:07:52.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:52.730+0000 7f50bfe00700 1 --2- 172.21.15.62:0/1936147678 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50c81061b0 0x7f50c81982a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:07:52.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:52.730+0000 7f50cd000700 1 -- 172.21.15.62:0/1936147678 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50c81061b0 msgr2=0x7f50c81982a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:07:52.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:52.730+0000 7f50cd000700 1 --2- 172.21.15.62:0/1936147678 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50c81061b0 0x7f50c81982a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:07:52.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:52.730+0000 7f50cd000700 1 -- 172.21.15.62:0/1936147678 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50ac0095f0 con 0x7f50c8068540 2024-04-20T22:07:52.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:52.730+0000 7f50cd000700 1 --2- 172.21.15.62:0/1936147678 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f50c8068540 0x7f50c8197d60 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f50ac0092e0 tx=0x7f50ac00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:07:52.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:52.730+0000 7f50c6000700 1 -- 172.21.15.62:0/1936147678 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f50ac01d070 con 0x7f50c8068540 2024-04-20T22:07:52.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:52.730+0000 7f50c6000700 1 -- 172.21.15.62:0/1936147678 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f50ac005e00 con 0x7f50c8068540 2024-04-20T22:07:52.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:52.730+0000 7f50c6000700 1 -- 172.21.15.62:0/1936147678 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f50ac017400 con 0x7f50c8068540 2024-04-20T22:07:52.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:52.730+0000 7f50ce400700 1 -- 172.21.15.62:0/1936147678 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50c819c990 con 0x7f50c8068540 2024-04-20T22:07:52.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:52.731+0000 7f50ce400700 1 -- 172.21.15.62:0/1936147678 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50c819ceb0 con 0x7f50c8068540 2024-04-20T22:07:52.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:52.732+0000 7f50ce400700 1 -- 172.21.15.62:0/1936147678 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50c8062380 con 0x7f50c8068540 2024-04-20T22:07:52.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:52.733+0000 7f50c6000700 1 -- 172.21.15.62:0/1936147678 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f50ac00f460 con 0x7f50c8068540 2024-04-20T22:07:52.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:52.734+0000 7f50c6000700 1 --2- 172.21.15.62:0/1936147678 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f50a406ecc0 0x7f50a4071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:07:52.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:52.734+0000 7f50c6000700 1 -- 172.21.15.62:0/1936147678 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f50ac090630 con 0x7f50c8068540 2024-04-20T22:07:52.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:52.734+0000 7f50bfe00700 1 --2- 172.21.15.62:0/1936147678 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f50a406ecc0 0x7f50a4071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:07:52.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:52.735+0000 7f50bfe00700 1 --2- 172.21.15.62:0/1936147678 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f50a406ecc0 0x7f50a4071170 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f50b4005950 tx=0x7f50b40058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:07:52.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:52.738+0000 7f50c6000700 1 -- 172.21.15.62:0/1936147678 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f50ac0593c0 con 0x7f50c8068540 2024-04-20T22:07:53.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:53.064+0000 7f50ce400700 1 -- 172.21.15.62:0/1936147678 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f50c819d2c0 con 0x7f50c8068540 2024-04-20T22:07:53.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:53.065+0000 7f50c6000700 1 -- 172.21.15.62:0/1936147678 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7f50ac05ca80 con 0x7f50c8068540 2024-04-20T22:07:53.065 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-20T22:07:53.065 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-20T22:07:53.065 INFO:teuthology.orchestra.run.smithi062.stdout: host smithi062 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T22:07:53.065 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-20T22:07:53.065 INFO:teuthology.orchestra.run.smithi062.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi062 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpq7y7oa_r:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:07:53.065 INFO:teuthology.orchestra.run.smithi062.stdout:Traceback (most recent call last): 2024-04-20T22:07:53.065 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-20T22:07:53.066 INFO:teuthology.orchestra.run.smithi062.stdout: return _run_code(code, main_globals, None, 2024-04-20T22:07:53.066 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-20T22:07:53.066 INFO:teuthology.orchestra.run.smithi062.stdout: exec(code, run_globals) 2024-04-20T22:07:53.066 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5618, in 2024-04-20T22:07:53.066 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5606, in main 2024-04-20T22:07:53.066 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 408, in _infer_config 2024-04-20T22:07:53.066 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 323, in _infer_fsid 2024-04-20T22:07:53.066 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 436, in _infer_image 2024-04-20T22:07:53.066 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 310, in _validate_fsid 2024-04-20T22:07:53.066 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 3359, in command_ceph_volume 2024-04-20T22:07:53.066 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-20T22:07:53.066 INFO:teuthology.orchestra.run.smithi062.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi062 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpq7y7oa_r:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:07:53.066 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-20T22:07:53.066 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is degraded 2024-04-20T22:07:53.067 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T22:07:53.067 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T22:07:53.067 INFO:teuthology.orchestra.run.smithi062.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-20T22:07:53.067 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-20T22:07:53.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:53.069+0000 7f50ce400700 1 -- 172.21.15.62:0/1936147678 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f50a406ecc0 msgr2=0x7f50a4071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:07:53.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:53.069+0000 7f50ce400700 1 --2- 172.21.15.62:0/1936147678 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f50a406ecc0 0x7f50a4071170 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f50b4005950 tx=0x7f50b40058e0 comp rx=0 tx=0).stop 2024-04-20T22:07:53.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:53.069+0000 7f50ce400700 1 -- 172.21.15.62:0/1936147678 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f50c8068540 msgr2=0x7f50c8197d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:07:53.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:53.069+0000 7f50ce400700 1 --2- 172.21.15.62:0/1936147678 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f50c8068540 0x7f50c8197d60 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f50ac0092e0 tx=0x7f50ac00be20 comp rx=0 tx=0).stop 2024-04-20T22:07:53.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:53.069+0000 7f50ce400700 1 -- 172.21.15.62:0/1936147678 shutdown_connections 2024-04-20T22:07:53.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:53.069+0000 7f50ce400700 1 --2- 172.21.15.62:0/1936147678 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f50a406ecc0 0x7f50a4071170 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:07:53.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:53.069+0000 7f50ce400700 1 --2- 172.21.15.62:0/1936147678 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f50c8068540 0x7f50c8197d60 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:07:53.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:53.069+0000 7f50ce400700 1 --2- 172.21.15.62:0/1936147678 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50c81061b0 0x7f50c81982a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:07:53.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:53.069+0000 7f50ce400700 1 -- 172.21.15.62:0/1936147678 >> 172.21.15.62:0/1936147678 conn(0x7f50c8074fa0 msgr2=0x7f50c80fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:07:53.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:53.069+0000 7f50ce400700 1 -- 172.21.15.62:0/1936147678 shutdown_connections 2024-04-20T22:07:53.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:07:53.069+0000 7f50ce400700 1 -- 172.21.15.62:0/1936147678 wait complete. 2024-04-20T22:07:53.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:07:53 smithi189 ceph-mon[57022]: pgmap v978: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:07:53.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:07:53 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/3743444342' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:07:53.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:07:53 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/1936147678' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:07:53.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:07:53 smithi062 ceph-mon[76322]: pgmap v978: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:07:53.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:07:53 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/3743444342' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:07:53.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:07:53 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/1936147678' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:07:54.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:07:54 smithi189 ceph-mon[57022]: from='client.35760 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:07:54.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:07:54 smithi062 ceph-mon[76322]: from='client.35760 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:07:55.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:07:55 smithi189 ceph-mon[57022]: pgmap v979: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:07:55.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:07:55 smithi062 ceph-mon[76322]: pgmap v979: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:07:57.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:07:57 smithi189 ceph-mon[57022]: pgmap v980: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:07:57.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:07:57 smithi062 ceph-mon[76322]: pgmap v980: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:07:59.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:07:59 smithi189 ceph-mon[57022]: pgmap v981: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:07:59.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:07:59 smithi062 ceph-mon[76322]: pgmap v981: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:08:01.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:08:01 smithi189 ceph-mon[57022]: pgmap v982: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:08:01.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:08:01 smithi062 ceph-mon[76322]: pgmap v982: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:08:03.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:08:03 smithi189 ceph-mon[57022]: pgmap v983: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:08:03.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:08:03 smithi062 ceph-mon[76322]: pgmap v983: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:08:05.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:08:05 smithi189 ceph-mon[57022]: pgmap v984: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:08:05.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:08:05 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:08:05.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:08:05 smithi062 ceph-mon[76322]: pgmap v984: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:08:05.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:08:05 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:08:07.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:08:07 smithi062 ceph-mon[76322]: pgmap v985: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:08:07.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:08:07 smithi189 ceph-mon[57022]: pgmap v985: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:08:09.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:08:09 smithi189 ceph-mon[57022]: pgmap v986: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:08:09.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:08:09 smithi062 ceph-mon[76322]: pgmap v986: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:08:11.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:08:11 smithi189 ceph-mon[57022]: pgmap v987: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:08:11.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:08:11 smithi062 ceph-mon[76322]: pgmap v987: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:08:13.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:08:13 smithi189 ceph-mon[57022]: pgmap v988: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:08:13.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:08:13 smithi062 ceph-mon[76322]: pgmap v988: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:08:15.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:08:15 smithi189 ceph-mon[57022]: pgmap v989: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:08:15.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:08:15 smithi062 ceph-mon[76322]: pgmap v989: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:08:17.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:08:17 smithi189 ceph-mon[57022]: pgmap v990: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:08:17.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:08:17 smithi062 ceph-mon[76322]: pgmap v990: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:08:19.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:08:19 smithi189 ceph-mon[57022]: pgmap v991: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:08:19.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:08:19 smithi062 ceph-mon[76322]: pgmap v991: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:08:20.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:08:20 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:08:20.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:08:20 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:08:21.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:08:21 smithi189 ceph-mon[57022]: pgmap v992: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:08:21.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:08:21 smithi062 ceph-mon[76322]: pgmap v992: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:08:23.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:23.246+0000 7f9ed1a00700 1 -- 172.21.15.62:0/639771912 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ecc102470 msgr2=0x7f9ecc1028e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:08:23.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:23.246+0000 7f9ed1a00700 1 --2- 172.21.15.62:0/639771912 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ecc102470 0x7f9ecc1028e0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f9eb4009910 tx=0x7f9eb4009c20 comp rx=0 tx=0).stop 2024-04-20T22:08:23.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:23.246+0000 7f9ed1a00700 1 -- 172.21.15.62:0/639771912 shutdown_connections 2024-04-20T22:08:23.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:23.246+0000 7f9ed1a00700 1 --2- 172.21.15.62:0/639771912 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ecc102470 0x7f9ecc1028e0 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:08:23.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:23.246+0000 7f9ed1a00700 1 --2- 172.21.15.62:0/639771912 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ecc108470 0x7f9ecc108840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:08:23.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:23.246+0000 7f9ed1a00700 1 -- 172.21.15.62:0/639771912 >> 172.21.15.62:0/639771912 conn(0x7f9ecc0fdf90 msgr2=0x7f9ecc1003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:08:23.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:23.246+0000 7f9ed1a00700 1 -- 172.21.15.62:0/639771912 shutdown_connections 2024-04-20T22:08:23.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:23.246+0000 7f9ed1a00700 1 -- 172.21.15.62:0/639771912 wait complete. 2024-04-20T22:08:23.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:23.248+0000 7f9ed1a00700 1 Processor -- start 2024-04-20T22:08:23.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:23.248+0000 7f9ed1a00700 1 -- start start 2024-04-20T22:08:23.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:23.249+0000 7f9ed1a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ecc102470 0x7f9ecc1981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:08:23.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:23.249+0000 7f9ed1a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ecc108470 0x7f9ecc198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:08:23.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:23.249+0000 7f9ed1a00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ecc198d80 con 0x7f9ecc108470 2024-04-20T22:08:23.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:23.249+0000 7f9ed1a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ecc198ec0 con 0x7f9ecc102470 2024-04-20T22:08:23.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:23.249+0000 7f9ec3e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ecc108470 0x7f9ecc198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:08:23.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:23.249+0000 7f9ec3e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ecc108470 0x7f9ecc198730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56666/0 (socket says 172.21.15.62:56666) 2024-04-20T22:08:23.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:23.249+0000 7f9ec3e00700 1 -- 172.21.15.62:0/1975826346 learned_addr learned my addr 172.21.15.62:0/1975826346 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:08:23.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:23.249+0000 7f9ecaa00700 1 --2- 172.21.15.62:0/1975826346 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ecc102470 0x7f9ecc1981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:08:23.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:23.249+0000 7f9ec3e00700 1 -- 172.21.15.62:0/1975826346 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ecc102470 msgr2=0x7f9ecc1981f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:08:23.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:23.249+0000 7f9ec3e00700 1 --2- 172.21.15.62:0/1975826346 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ecc102470 0x7f9ecc1981f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:08:23.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:23.249+0000 7f9ec3e00700 1 -- 172.21.15.62:0/1975826346 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9eb40095f0 con 0x7f9ecc108470 2024-04-20T22:08:23.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:23.250+0000 7f9ec3e00700 1 --2- 172.21.15.62:0/1975826346 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ecc108470 0x7f9ecc198730 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f9eb400b5c0 tx=0x7f9eb400be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:08:23.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:23.250+0000 7f9ec3400700 1 -- 172.21.15.62:0/1975826346 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9eb401d070 con 0x7f9ecc108470 2024-04-20T22:08:23.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:23.250+0000 7f9ec3400700 1 -- 172.21.15.62:0/1975826346 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9eb400f460 con 0x7f9ecc108470 2024-04-20T22:08:23.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:23.250+0000 7f9ed1a00700 1 -- 172.21.15.62:0/1975826346 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ecc19ccb0 con 0x7f9ecc108470 2024-04-20T22:08:23.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:23.250+0000 7f9ec3400700 1 -- 172.21.15.62:0/1975826346 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9eb4017610 con 0x7f9ecc108470 2024-04-20T22:08:23.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:23.250+0000 7f9ed1a00700 1 -- 172.21.15.62:0/1975826346 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ecc19d1a0 con 0x7f9ecc108470 2024-04-20T22:08:23.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:23.252+0000 7f9ed1a00700 1 -- 172.21.15.62:0/1975826346 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ecc10a970 con 0x7f9ecc108470 2024-04-20T22:08:23.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:23.253+0000 7f9ec3400700 1 -- 172.21.15.62:0/1975826346 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f9eb400f5d0 con 0x7f9ecc108470 2024-04-20T22:08:23.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:23.254+0000 7f9ec3400700 1 --2- 172.21.15.62:0/1975826346 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9eac06ecc0 0x7f9eac071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:08:23.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:23.254+0000 7f9ec3400700 1 -- 172.21.15.62:0/1975826346 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f9eb4090440 con 0x7f9ecc108470 2024-04-20T22:08:23.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:23.254+0000 7f9ecaa00700 1 --2- 172.21.15.62:0/1975826346 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9eac06ecc0 0x7f9eac071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:08:23.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:23.255+0000 7f9ecaa00700 1 --2- 172.21.15.62:0/1975826346 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9eac06ecc0 0x7f9eac071170 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f9ebc005ea0 tx=0x7f9ebc005e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:08:23.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:23.259+0000 7f9ec3400700 1 -- 172.21.15.62:0/1975826346 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f9eb40591d0 con 0x7f9ecc108470 2024-04-20T22:08:23.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:08:23 smithi189 ceph-mon[57022]: pgmap v993: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:08:23.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:23.514+0000 7f9ed1a00700 1 -- 172.21.15.62:0/1975826346 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9ecc199550 con 0x7f9eac06ecc0 2024-04-20T22:08:23.515 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:08:23 smithi062 ceph-mon[76322]: pgmap v993: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:08:23.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:23.518+0000 7f9ec3400700 1 -- 172.21.15.62:0/1975826346 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f9ecc199550 con 0x7f9eac06ecc0 2024-04-20T22:08:23.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:23.521+0000 7f9ed1a00700 1 -- 172.21.15.62:0/1975826346 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9eac06ecc0 msgr2=0x7f9eac071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:08:23.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:23.521+0000 7f9ed1a00700 1 --2- 172.21.15.62:0/1975826346 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9eac06ecc0 0x7f9eac071170 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f9ebc005ea0 tx=0x7f9ebc005e30 comp rx=0 tx=0).stop 2024-04-20T22:08:23.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:23.521+0000 7f9ed1a00700 1 -- 172.21.15.62:0/1975826346 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ecc108470 msgr2=0x7f9ecc198730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:08:23.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:23.521+0000 7f9ed1a00700 1 --2- 172.21.15.62:0/1975826346 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ecc108470 0x7f9ecc198730 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f9eb400b5c0 tx=0x7f9eb400be00 comp rx=0 tx=0).stop 2024-04-20T22:08:23.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:23.521+0000 7f9ed1a00700 1 -- 172.21.15.62:0/1975826346 shutdown_connections 2024-04-20T22:08:23.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:23.521+0000 7f9ed1a00700 1 --2- 172.21.15.62:0/1975826346 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9eac06ecc0 0x7f9eac071170 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:08:23.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:23.521+0000 7f9ed1a00700 1 --2- 172.21.15.62:0/1975826346 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ecc102470 0x7f9ecc1981f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:08:23.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:23.521+0000 7f9ed1a00700 1 --2- 172.21.15.62:0/1975826346 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ecc108470 0x7f9ecc198730 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:08:23.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:23.521+0000 7f9ed1a00700 1 -- 172.21.15.62:0/1975826346 >> 172.21.15.62:0/1975826346 conn(0x7f9ecc0fdf90 msgr2=0x7f9ecc0ff3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:08:23.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:23.521+0000 7f9ed1a00700 1 -- 172.21.15.62:0/1975826346 shutdown_connections 2024-04-20T22:08:23.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:23.521+0000 7f9ed1a00700 1 -- 172.21.15.62:0/1975826346 wait complete. 2024-04-20T22:08:23.538 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T22:08:23.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:23.684+0000 7ff384800700 1 -- 172.21.15.62:0/1027731507 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff37c106b80 msgr2=0x7ff37c106ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:08:23.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:23.684+0000 7ff384800700 1 --2- 172.21.15.62:0/1027731507 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff37c106b80 0x7ff37c106ff0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7ff36c009910 tx=0x7ff36c009c20 comp rx=0 tx=0).stop 2024-04-20T22:08:23.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:23.684+0000 7ff384800700 1 -- 172.21.15.62:0/1027731507 shutdown_connections 2024-04-20T22:08:23.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:23.684+0000 7ff384800700 1 --2- 172.21.15.62:0/1027731507 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff37c106b80 0x7ff37c106ff0 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:08:23.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:23.684+0000 7ff384800700 1 --2- 172.21.15.62:0/1027731507 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff37c10c8f0 0x7ff37c10ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:08:23.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:23.684+0000 7ff384800700 1 -- 172.21.15.62:0/1027731507 >> 172.21.15.62:0/1027731507 conn(0x7ff37c076060 msgr2=0x7ff37c078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:08:23.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:23.684+0000 7ff384800700 1 -- 172.21.15.62:0/1027731507 shutdown_connections 2024-04-20T22:08:23.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:23.684+0000 7ff384800700 1 -- 172.21.15.62:0/1027731507 wait complete. 2024-04-20T22:08:23.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:23.685+0000 7ff384800700 1 Processor -- start 2024-04-20T22:08:23.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:23.685+0000 7ff384800700 1 -- start start 2024-04-20T22:08:23.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:23.685+0000 7ff384800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff37c106b80 0x7ff37c19c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:08:23.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:23.686+0000 7ff384800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff37c10c8f0 0x7ff37c19c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:08:23.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:23.686+0000 7ff384800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff37c19d060 con 0x7ff37c106b80 2024-04-20T22:08:23.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:23.686+0000 7ff384800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff37c1a0df0 con 0x7ff37c10c8f0 2024-04-20T22:08:23.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:23.686+0000 7ff381e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff37c106b80 0x7ff37c19c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:08:23.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:23.686+0000 7ff381e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff37c106b80 0x7ff37c19c440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56672/0 (socket says 172.21.15.62:56672) 2024-04-20T22:08:23.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:23.686+0000 7ff381e00700 1 -- 172.21.15.62:0/402620525 learned_addr learned my addr 172.21.15.62:0/402620525 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:08:23.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:23.686+0000 7ff381400700 1 --2- 172.21.15.62:0/402620525 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff37c10c8f0 0x7ff37c19c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:08:23.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:23.686+0000 7ff381e00700 1 -- 172.21.15.62:0/402620525 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff37c10c8f0 msgr2=0x7ff37c19c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:08:23.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:23.686+0000 7ff381e00700 1 --2- 172.21.15.62:0/402620525 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff37c10c8f0 0x7ff37c19c980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:08:23.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:23.686+0000 7ff381e00700 1 -- 172.21.15.62:0/402620525 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff36c0095f0 con 0x7ff37c106b80 2024-04-20T22:08:23.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:23.687+0000 7ff381e00700 1 --2- 172.21.15.62:0/402620525 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff37c106b80 0x7ff37c19c440 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7ff37000c8a0 tx=0x7ff37000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:08:23.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:23.687+0000 7ff376a00700 1 -- 172.21.15.62:0/402620525 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff37000cea0 con 0x7ff37c106b80 2024-04-20T22:08:23.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:23.687+0000 7ff376a00700 1 -- 172.21.15.62:0/402620525 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff370004830 con 0x7ff37c106b80 2024-04-20T22:08:23.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:23.687+0000 7ff384800700 1 -- 172.21.15.62:0/402620525 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff37c1a10d0 con 0x7ff37c106b80 2024-04-20T22:08:23.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:23.687+0000 7ff376a00700 1 -- 172.21.15.62:0/402620525 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff3700055b0 con 0x7ff37c106b80 2024-04-20T22:08:23.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:23.687+0000 7ff384800700 1 -- 172.21.15.62:0/402620525 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff37c1a1620 con 0x7ff37c106b80 2024-04-20T22:08:23.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:23.688+0000 7ff384800700 1 -- 172.21.15.62:0/402620525 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff37c10ecc0 con 0x7ff37c106b80 2024-04-20T22:08:23.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:23.690+0000 7ff376a00700 1 -- 172.21.15.62:0/402620525 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff3700049a0 con 0x7ff37c106b80 2024-04-20T22:08:23.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:23.691+0000 7ff376a00700 1 --2- 172.21.15.62:0/402620525 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff36406ecc0 0x7ff364071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:08:23.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:23.691+0000 7ff376a00700 1 -- 172.21.15.62:0/402620525 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7ff37008eaf0 con 0x7ff37c106b80 2024-04-20T22:08:23.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:23.691+0000 7ff381400700 1 --2- 172.21.15.62:0/402620525 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff36406ecc0 0x7ff364071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:08:23.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:23.692+0000 7ff381400700 1 --2- 172.21.15.62:0/402620525 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff36406ecc0 0x7ff364071170 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7ff36c00b5c0 tx=0x7ff36c005960 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:08:23.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:23.696+0000 7ff376a00700 1 -- 172.21.15.62:0/402620525 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7ff370057880 con 0x7ff37c106b80 2024-04-20T22:08:23.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:23.950+0000 7ff384800700 1 -- 172.21.15.62:0/402620525 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff37c1a18f0 con 0x7ff36406ecc0 2024-04-20T22:08:23.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:23.954+0000 7ff376a00700 1 -- 172.21.15.62:0/402620525 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7ff37c1a18f0 con 0x7ff36406ecc0 2024-04-20T22:08:23.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:23.957+0000 7ff384800700 1 -- 172.21.15.62:0/402620525 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff36406ecc0 msgr2=0x7ff364071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:08:23.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:23.957+0000 7ff384800700 1 --2- 172.21.15.62:0/402620525 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff36406ecc0 0x7ff364071170 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7ff36c00b5c0 tx=0x7ff36c005960 comp rx=0 tx=0).stop 2024-04-20T22:08:23.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:23.957+0000 7ff384800700 1 -- 172.21.15.62:0/402620525 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff37c106b80 msgr2=0x7ff37c19c440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:08:23.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:23.957+0000 7ff384800700 1 --2- 172.21.15.62:0/402620525 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff37c106b80 0x7ff37c19c440 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7ff37000c8a0 tx=0x7ff37000cc60 comp rx=0 tx=0).stop 2024-04-20T22:08:23.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:23.957+0000 7ff384800700 1 -- 172.21.15.62:0/402620525 shutdown_connections 2024-04-20T22:08:23.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:23.957+0000 7ff384800700 1 --2- 172.21.15.62:0/402620525 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff36406ecc0 0x7ff364071170 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:08:23.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:23.957+0000 7ff384800700 1 --2- 172.21.15.62:0/402620525 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff37c106b80 0x7ff37c19c440 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:08:23.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:23.957+0000 7ff384800700 1 --2- 172.21.15.62:0/402620525 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff37c10c8f0 0x7ff37c19c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:08:23.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:23.957+0000 7ff384800700 1 -- 172.21.15.62:0/402620525 >> 172.21.15.62:0/402620525 conn(0x7ff37c076060 msgr2=0x7ff37c077ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:08:23.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:23.957+0000 7ff384800700 1 -- 172.21.15.62:0/402620525 shutdown_connections 2024-04-20T22:08:23.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:23.957+0000 7ff384800700 1 -- 172.21.15.62:0/402620525 wait complete. 2024-04-20T22:08:24.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:24.111+0000 7f1eeb200700 1 -- 172.21.15.62:0/2930100339 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1eec073980 msgr2=0x7f1eec10c3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:08:24.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:24.111+0000 7f1eeb200700 1 --2- 172.21.15.62:0/2930100339 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1eec073980 0x7f1eec10c3e0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f1ed4009940 tx=0x7f1ed4009c50 comp rx=0 tx=0).stop 2024-04-20T22:08:24.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:24.111+0000 7f1eeb200700 1 -- 172.21.15.62:0/2930100339 shutdown_connections 2024-04-20T22:08:24.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:24.111+0000 7f1eeb200700 1 --2- 172.21.15.62:0/2930100339 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1eec073980 0x7f1eec10c3e0 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:08:24.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:24.111+0000 7f1eeb200700 1 --2- 172.21.15.62:0/2930100339 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1eec073070 0x7f1eec073440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:08:24.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:24.111+0000 7f1eeb200700 1 -- 172.21.15.62:0/2930100339 >> 172.21.15.62:0/2930100339 conn(0x7f1eec0fbb60 msgr2=0x7f1eec0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:08:24.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:24.112+0000 7f1eeb200700 1 -- 172.21.15.62:0/2930100339 shutdown_connections 2024-04-20T22:08:24.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:24.112+0000 7f1eeb200700 1 -- 172.21.15.62:0/2930100339 wait complete. 2024-04-20T22:08:24.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:24.113+0000 7f1eeb200700 1 Processor -- start 2024-04-20T22:08:24.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:24.113+0000 7f1eeb200700 1 -- start start 2024-04-20T22:08:24.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:24.113+0000 7f1eeb200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1eec073070 0x7f1eec197e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:08:24.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:24.113+0000 7f1eeb200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1eec073980 0x7f1eec198340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:08:24.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:24.113+0000 7f1eeb200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eec198990 con 0x7f1eec073070 2024-04-20T22:08:24.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:24.113+0000 7f1eeb200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eec198ad0 con 0x7f1eec073980 2024-04-20T22:08:24.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:24.114+0000 7f1ee9e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1eec073070 0x7f1eec197e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:08:24.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:24.114+0000 7f1ee9e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1eec073070 0x7f1eec197e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56686/0 (socket says 172.21.15.62:56686) 2024-04-20T22:08:24.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:24.114+0000 7f1ee9e00700 1 -- 172.21.15.62:0/960043559 learned_addr learned my addr 172.21.15.62:0/960043559 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:08:24.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:24.114+0000 7f1ee1400700 1 --2- 172.21.15.62:0/960043559 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1eec073980 0x7f1eec198340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:08:24.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:24.114+0000 7f1ee9e00700 1 -- 172.21.15.62:0/960043559 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1eec073980 msgr2=0x7f1eec198340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:08:24.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:24.114+0000 7f1ee9e00700 1 --2- 172.21.15.62:0/960043559 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1eec073980 0x7f1eec198340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:08:24.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:24.114+0000 7f1ee9e00700 1 -- 172.21.15.62:0/960043559 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ed40095f0 con 0x7f1eec073070 2024-04-20T22:08:24.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:24.114+0000 7f1ee9e00700 1 --2- 172.21.15.62:0/960043559 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1eec073070 0x7f1eec197e00 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f1edc00cfd0 tx=0x7f1edc00a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:08:24.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:24.115+0000 7f1ee3400700 1 -- 172.21.15.62:0/960043559 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1edc00ad70 con 0x7f1eec073070 2024-04-20T22:08:24.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:24.115+0000 7f1ee3400700 1 -- 172.21.15.62:0/960043559 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1edc0128e0 con 0x7f1eec073070 2024-04-20T22:08:24.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:24.115+0000 7f1eeb200700 1 -- 172.21.15.62:0/960043559 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1eec19c920 con 0x7f1eec073070 2024-04-20T22:08:24.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:24.115+0000 7f1ee3400700 1 -- 172.21.15.62:0/960043559 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1edc011880 con 0x7f1eec073070 2024-04-20T22:08:24.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:24.115+0000 7f1eeb200700 1 -- 172.21.15.62:0/960043559 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1eec19ce70 con 0x7f1eec073070 2024-04-20T22:08:24.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:24.116+0000 7f1eeb200700 1 -- 172.21.15.62:0/960043559 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1eec109ae0 con 0x7f1eec073070 2024-04-20T22:08:24.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:24.117+0000 7f1ee3400700 1 -- 172.21.15.62:0/960043559 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f1edc012420 con 0x7f1eec073070 2024-04-20T22:08:24.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:24.118+0000 7f1ee3400700 1 --2- 172.21.15.62:0/960043559 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f1ed006ecc0 0x7f1ed0071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:08:24.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:24.118+0000 7f1ee3400700 1 -- 172.21.15.62:0/960043559 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f1edc08ef40 con 0x7f1eec073070 2024-04-20T22:08:24.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:24.118+0000 7f1ee1400700 1 --2- 172.21.15.62:0/960043559 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f1ed006ecc0 0x7f1ed0071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:08:24.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:24.119+0000 7f1ee1400700 1 --2- 172.21.15.62:0/960043559 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f1ed006ecc0 0x7f1ed0071170 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f1ed400b870 tx=0x7f1ed4005fb0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:08:24.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:24.123+0000 7f1ee3400700 1 -- 172.21.15.62:0/960043559 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f1edc057cd0 con 0x7f1eec073070 2024-04-20T22:08:24.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:24.361+0000 7f1eeb200700 1 -- 172.21.15.62:0/960043559 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f1eec19d140 con 0x7f1ed006ecc0 2024-04-20T22:08:24.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:24.394+0000 7f1ee3400700 1 -- 172.21.15.62:0/960043559 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f1eec19d140 con 0x7f1ed006ecc0 2024-04-20T22:08:24.395 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T22:08:24.395 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (34m) 10m ago 48m 15.3M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T22:08:24.395 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (48m) 10m ago 48m 10.4M - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T22:08:24.395 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (47m) 10m ago 47m 10.1M - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T22:08:24.395 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (30m) 10m ago 48m 6660k - 19.0.0-3133-ga8c84f59 76bc6b91b726 2825d5ba11c6 2024-04-20T22:08:24.395 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (30m) 10m ago 46m 6665k - 19.0.0-3133-ga8c84f59 76bc6b91b726 20f746fa1bba 2024-04-20T22:08:24.395 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (33m) 10m ago 47m 73.5M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T22:08:24.395 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (25m) 10m ago 40m 18.5M - 19.0.0-3133-ga8c84f59 76bc6b91b726 de80bc90a683 2024-04-20T22:08:24.395 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (26m) 10m ago 40m 18.7M - 19.0.0-3133-ga8c84f59 76bc6b91b726 cd793b7ee98e 2024-04-20T22:08:24.395 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (10m) 10m ago 40m 11.0M - 19.0.0-3133-ga8c84f59 76bc6b91b726 0e7b0bc4d1ab 2024-04-20T22:08:24.395 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (40m) 10m ago 40m 18.6M - 18.2.0 dc2bc1663786 f2eec2ee8951 2024-04-20T22:08:24.395 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (37m) 10m ago 49m 513M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T22:08:24.395 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (35m) 10m ago 46m 459M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T22:08:24.395 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (32m) 10m ago 50m 71.7M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 019eb1479485 2024-04-20T22:08:24.396 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (31m) 10m ago 46m 51.0M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 55aac560aab6 2024-04-20T22:08:24.396 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (35m) 10m ago 48m 8472k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T22:08:24.396 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (35m) 10m ago 46m 8048k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T22:08:24.396 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (29m) 10m ago 45m 91.2M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 ee2bb901f26e 2024-04-20T22:08:24.396 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (29m) 10m ago 45m 94.6M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 9075152d4162 2024-04-20T22:08:24.396 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (28m) 10m ago 44m 78.1M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 2d4f3e38dc0e 2024-04-20T22:08:24.396 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (28m) 10m ago 43m 92.6M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 8f97b17ae0eb 2024-04-20T22:08:24.396 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (27m) 10m ago 43m 95.6M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 653d8079967d 2024-04-20T22:08:24.396 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (26m) 10m ago 42m 89.2M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 596dded14c12 2024-04-20T22:08:24.396 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (34m) 10m ago 47m 82.1M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T22:08:24.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:24.397+0000 7f1eeb200700 1 -- 172.21.15.62:0/960043559 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f1ed006ecc0 msgr2=0x7f1ed0071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:08:24.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:24.397+0000 7f1eeb200700 1 --2- 172.21.15.62:0/960043559 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f1ed006ecc0 0x7f1ed0071170 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f1ed400b870 tx=0x7f1ed4005fb0 comp rx=0 tx=0).stop 2024-04-20T22:08:24.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:24.397+0000 7f1eeb200700 1 -- 172.21.15.62:0/960043559 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1eec073070 msgr2=0x7f1eec197e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:08:24.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:24.397+0000 7f1eeb200700 1 --2- 172.21.15.62:0/960043559 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1eec073070 0x7f1eec197e00 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f1edc00cfd0 tx=0x7f1edc00a340 comp rx=0 tx=0).stop 2024-04-20T22:08:24.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:24.397+0000 7f1eeb200700 1 -- 172.21.15.62:0/960043559 shutdown_connections 2024-04-20T22:08:24.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:24.397+0000 7f1eeb200700 1 --2- 172.21.15.62:0/960043559 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f1ed006ecc0 0x7f1ed0071170 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:08:24.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:24.397+0000 7f1eeb200700 1 --2- 172.21.15.62:0/960043559 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1eec073070 0x7f1eec197e00 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:08:24.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:24.397+0000 7f1eeb200700 1 --2- 172.21.15.62:0/960043559 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1eec073980 0x7f1eec198340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:08:24.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:24.397+0000 7f1eeb200700 1 -- 172.21.15.62:0/960043559 >> 172.21.15.62:0/960043559 conn(0x7f1eec0fbb60 msgr2=0x7f1eec106c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:08:24.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:24.397+0000 7f1eeb200700 1 -- 172.21.15.62:0/960043559 shutdown_connections 2024-04-20T22:08:24.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:24.398+0000 7f1eeb200700 1 -- 172.21.15.62:0/960043559 wait complete. 2024-04-20T22:08:24.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:24.558+0000 7f0669c00700 1 -- 172.21.15.62:0/3611020977 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0664103040 msgr2=0x7f0664107010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:08:24.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:24.558+0000 7f0669c00700 1 --2- 172.21.15.62:0/3611020977 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0664103040 0x7f0664107010 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f064c009910 tx=0x7f064c009c20 comp rx=0 tx=0).stop 2024-04-20T22:08:24.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:24.559+0000 7f0669c00700 1 -- 172.21.15.62:0/3611020977 shutdown_connections 2024-04-20T22:08:24.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:24.559+0000 7f0669c00700 1 --2- 172.21.15.62:0/3611020977 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0664103040 0x7f0664107010 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:08:24.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:24.559+0000 7f0669c00700 1 --2- 172.21.15.62:0/3611020977 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06641026a0 0x7f0664102a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:08:24.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:24.559+0000 7f0669c00700 1 -- 172.21.15.62:0/3611020977 >> 172.21.15.62:0/3611020977 conn(0x7f06640fdf40 msgr2=0x7f0664100350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:08:24.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:24.559+0000 7f0669c00700 1 -- 172.21.15.62:0/3611020977 shutdown_connections 2024-04-20T22:08:24.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:24.559+0000 7f0669c00700 1 -- 172.21.15.62:0/3611020977 wait complete. 2024-04-20T22:08:24.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:24.560+0000 7f0669c00700 1 Processor -- start 2024-04-20T22:08:24.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:24.560+0000 7f0669c00700 1 -- start start 2024-04-20T22:08:24.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:24.560+0000 7f0669c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06641026a0 0x7f0664198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:08:24.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:24.560+0000 7f0669c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0664103040 0x7f0664198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:08:24.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:24.560+0000 7f0669c00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0664198c70 con 0x7f0664103040 2024-04-20T22:08:24.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:24.560+0000 7f0669c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f066419ca00 con 0x7f06641026a0 2024-04-20T22:08:24.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:24.561+0000 7f065be00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0664103040 0x7f0664198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:08:24.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:24.561+0000 7f065be00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0664103040 0x7f0664198590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56712/0 (socket says 172.21.15.62:56712) 2024-04-20T22:08:24.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:24.561+0000 7f065be00700 1 -- 172.21.15.62:0/1624973788 learned_addr learned my addr 172.21.15.62:0/1624973788 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:08:24.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:24.561+0000 7f0662a00700 1 --2- 172.21.15.62:0/1624973788 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06641026a0 0x7f0664198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:08:24.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:24.561+0000 7f065be00700 1 -- 172.21.15.62:0/1624973788 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06641026a0 msgr2=0x7f0664198050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:08:24.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:24.561+0000 7f065be00700 1 --2- 172.21.15.62:0/1624973788 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06641026a0 0x7f0664198050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:08:24.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:24.561+0000 7f065be00700 1 -- 172.21.15.62:0/1624973788 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f064c0095f0 con 0x7f0664103040 2024-04-20T22:08:24.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:24.561+0000 7f0662a00700 1 --2- 172.21.15.62:0/1624973788 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06641026a0 0x7f0664198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T22:08:24.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:24.562+0000 7f065be00700 1 --2- 172.21.15.62:0/1624973788 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0664103040 0x7f0664198590 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f064c00bda0 tx=0x7f064c00be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:08:24.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:24.562+0000 7f065b400700 1 -- 172.21.15.62:0/1624973788 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f064c01d070 con 0x7f0664103040 2024-04-20T22:08:24.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:24.562+0000 7f065b400700 1 -- 172.21.15.62:0/1624973788 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f064c004fb0 con 0x7f0664103040 2024-04-20T22:08:24.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:24.562+0000 7f065b400700 1 -- 172.21.15.62:0/1624973788 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f064c00f460 con 0x7f0664103040 2024-04-20T22:08:24.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:24.562+0000 7f0669c00700 1 -- 172.21.15.62:0/1624973788 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f066419cc80 con 0x7f0664103040 2024-04-20T22:08:24.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:24.562+0000 7f0669c00700 1 -- 172.21.15.62:0/1624973788 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f066419d170 con 0x7f0664103040 2024-04-20T22:08:24.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:24.563+0000 7f0669c00700 1 -- 172.21.15.62:0/1624973788 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f066410aa10 con 0x7f0664103040 2024-04-20T22:08:24.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:24.564+0000 7f065b400700 1 -- 172.21.15.62:0/1624973788 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f064c022800 con 0x7f0664103040 2024-04-20T22:08:24.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:24.565+0000 7f065b400700 1 --2- 172.21.15.62:0/1624973788 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f064806ecc0 0x7f0648071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:08:24.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:24.565+0000 7f065b400700 1 -- 172.21.15.62:0/1624973788 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f064c090390 con 0x7f0664103040 2024-04-20T22:08:24.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:24.566+0000 7f0662a00700 1 --2- 172.21.15.62:0/1624973788 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f064806ecc0 0x7f0648071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:08:24.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:24.566+0000 7f0662a00700 1 --2- 172.21.15.62:0/1624973788 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f064806ecc0 0x7f0648071170 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f0654005fd0 tx=0x7f0654005dc0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:08:24.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:24.570+0000 7f065b400700 1 -- 172.21.15.62:0/1624973788 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f064c0592f0 con 0x7f0664103040 2024-04-20T22:08:24.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:24.908+0000 7f0669c00700 1 -- 172.21.15.62:0/1624973788 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f066404f2a0 con 0x7f0664103040 2024-04-20T22:08:24.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:24.909+0000 7f065b400700 1 -- 172.21.15.62:0/1624973788 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7f064c05c9b0 con 0x7f0664103040 2024-04-20T22:08:24.909 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:08:24.909 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T22:08:24.909 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:08:24.909 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:08:24.909 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T22:08:24.909 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:08:24.909 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:08:24.909 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T22:08:24.910 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 6 2024-04-20T22:08:24.910 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:08:24.910 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T22:08:24.910 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 1, 2024-04-20T22:08:24.910 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 3 2024-04-20T22:08:24.910 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:08:24.910 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T22:08:24.910 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 1, 2024-04-20T22:08:24.910 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 13 2024-04-20T22:08:24.910 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T22:08:24.910 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:08:24.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:24.912+0000 7f0669c00700 1 -- 172.21.15.62:0/1624973788 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f064806ecc0 msgr2=0x7f0648071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:08:24.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:24.912+0000 7f0669c00700 1 --2- 172.21.15.62:0/1624973788 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f064806ecc0 0x7f0648071170 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f0654005fd0 tx=0x7f0654005dc0 comp rx=0 tx=0).stop 2024-04-20T22:08:24.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:24.912+0000 7f0669c00700 1 -- 172.21.15.62:0/1624973788 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0664103040 msgr2=0x7f0664198590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:08:24.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:24.912+0000 7f0669c00700 1 --2- 172.21.15.62:0/1624973788 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0664103040 0x7f0664198590 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f064c00bda0 tx=0x7f064c00be80 comp rx=0 tx=0).stop 2024-04-20T22:08:24.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:24.912+0000 7f0669c00700 1 -- 172.21.15.62:0/1624973788 shutdown_connections 2024-04-20T22:08:24.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:24.912+0000 7f0669c00700 1 --2- 172.21.15.62:0/1624973788 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f064806ecc0 0x7f0648071170 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:08:24.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:24.912+0000 7f0669c00700 1 --2- 172.21.15.62:0/1624973788 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06641026a0 0x7f0664198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:08:24.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:24.912+0000 7f0669c00700 1 --2- 172.21.15.62:0/1624973788 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0664103040 0x7f0664198590 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:08:24.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:24.912+0000 7f0669c00700 1 -- 172.21.15.62:0/1624973788 >> 172.21.15.62:0/1624973788 conn(0x7f06640fdf40 msgr2=0x7f0664100320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:08:24.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:24.912+0000 7f0669c00700 1 -- 172.21.15.62:0/1624973788 shutdown_connections 2024-04-20T22:08:24.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:24.912+0000 7f0669c00700 1 -- 172.21.15.62:0/1624973788 wait complete. 2024-04-20T22:08:25.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.074+0000 7f9079600700 1 -- 172.21.15.62:0/17737791 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9074103040 msgr2=0x7f9074107010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:08:25.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.074+0000 7f9079600700 1 --2- 172.21.15.62:0/17737791 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9074103040 0x7f9074107010 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f9060009910 tx=0x7f9060009c20 comp rx=0 tx=0).stop 2024-04-20T22:08:25.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.074+0000 7f9079600700 1 -- 172.21.15.62:0/17737791 shutdown_connections 2024-04-20T22:08:25.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.074+0000 7f9079600700 1 --2- 172.21.15.62:0/17737791 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9074103040 0x7f9074107010 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:08:25.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.074+0000 7f9079600700 1 --2- 172.21.15.62:0/17737791 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f90741026a0 0x7f9074102a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:08:25.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.074+0000 7f9079600700 1 -- 172.21.15.62:0/17737791 >> 172.21.15.62:0/17737791 conn(0x7f90740fdf40 msgr2=0x7f9074100350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:08:25.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.074+0000 7f9079600700 1 -- 172.21.15.62:0/17737791 shutdown_connections 2024-04-20T22:08:25.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.074+0000 7f9079600700 1 -- 172.21.15.62:0/17737791 wait complete. 2024-04-20T22:08:25.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.075+0000 7f9079600700 1 Processor -- start 2024-04-20T22:08:25.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.075+0000 7f9079600700 1 -- start start 2024-04-20T22:08:25.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.076+0000 7f9079600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90741026a0 0x7f9074198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:08:25.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.076+0000 7f9079600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9074103040 0x7f9074198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:08:25.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.076+0000 7f9079600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9074198bc0 con 0x7f90741026a0 2024-04-20T22:08:25.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.076+0000 7f9079600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9074198d00 con 0x7f9074103040 2024-04-20T22:08:25.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.076+0000 7f9072a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90741026a0 0x7f9074198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:08:25.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.076+0000 7f9072a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90741026a0 0x7f9074198030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56730/0 (socket says 172.21.15.62:56730) 2024-04-20T22:08:25.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.076+0000 7f9072a00700 1 -- 172.21.15.62:0/1037525077 learned_addr learned my addr 172.21.15.62:0/1037525077 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:08:25.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.076+0000 7f9072a00700 1 -- 172.21.15.62:0/1037525077 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9074103040 msgr2=0x7f9074198570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:08:25.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.076+0000 7f9072000700 1 --2- 172.21.15.62:0/1037525077 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9074103040 0x7f9074198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:08:25.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.077+0000 7f9072a00700 1 --2- 172.21.15.62:0/1037525077 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9074103040 0x7f9074198570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:08:25.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.077+0000 7f9072a00700 1 -- 172.21.15.62:0/1037525077 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90600095f0 con 0x7f90741026a0 2024-04-20T22:08:25.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.077+0000 7f9072000700 1 --2- 172.21.15.62:0/1037525077 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9074103040 0x7f9074198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T22:08:25.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.077+0000 7f9072a00700 1 --2- 172.21.15.62:0/1037525077 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90741026a0 0x7f9074198030 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f906400c930 tx=0x7f906400ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:08:25.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.077+0000 7f906b400700 1 -- 172.21.15.62:0/1037525077 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9064007ad0 con 0x7f90741026a0 2024-04-20T22:08:25.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.077+0000 7f906b400700 1 -- 172.21.15.62:0/1037525077 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9064007c30 con 0x7f90741026a0 2024-04-20T22:08:25.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.077+0000 7f906b400700 1 -- 172.21.15.62:0/1037525077 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9064018610 con 0x7f90741026a0 2024-04-20T22:08:25.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.077+0000 7f9079600700 1 -- 172.21.15.62:0/1037525077 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f907419cb50 con 0x7f90741026a0 2024-04-20T22:08:25.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.077+0000 7f9079600700 1 -- 172.21.15.62:0/1037525077 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f907419d050 con 0x7f90741026a0 2024-04-20T22:08:25.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.079+0000 7f9079600700 1 -- 172.21.15.62:0/1037525077 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f907410a920 con 0x7f90741026a0 2024-04-20T22:08:25.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.080+0000 7f906b400700 1 -- 172.21.15.62:0/1037525077 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f906400f450 con 0x7f90741026a0 2024-04-20T22:08:25.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.081+0000 7f906b400700 1 --2- 172.21.15.62:0/1037525077 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f905806ecc0 0x7f9058071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:08:25.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.081+0000 7f906b400700 1 -- 172.21.15.62:0/1037525077 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f906408eff0 con 0x7f90741026a0 2024-04-20T22:08:25.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.081+0000 7f9072000700 1 --2- 172.21.15.62:0/1037525077 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f905806ecc0 0x7f9058071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:08:25.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.082+0000 7f9072000700 1 --2- 172.21.15.62:0/1037525077 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f905806ecc0 0x7f9058071170 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f9060006010 tx=0x7f906000b540 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:08:25.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.086+0000 7f906b400700 1 -- 172.21.15.62:0/1037525077 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f9064057de0 con 0x7f90741026a0 2024-04-20T22:08:25.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.371+0000 7f9079600700 1 -- 172.21.15.62:0/1037525077 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f907404f2a0 con 0x7f90741026a0 2024-04-20T22:08:25.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.372+0000 7f906b400700 1 -- 172.21.15.62:0/1037525077 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 27 v27) v1 ==== 76+0+1888 (secure 0 0 0) 0x7f906405b4a0 con 0x7f90741026a0 2024-04-20T22:08:25.372 INFO:teuthology.orchestra.run.smithi062.stdout:e27 2024-04-20T22:08:25.373 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T22:08:25.373 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:08:25.373 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T22:08:25.373 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:08:25.373 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T22:08:25.373 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T22:08:25.373 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 20 2024-04-20T22:08:25.373 INFO:teuthology.orchestra.run.smithi062.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-20T22:08:25.373 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T22:08:25.373 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T21:42:01.559061+0000 2024-04-20T22:08:25.373 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T22:08:25.373 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T22:08:25.373 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T22:08:25.373 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T22:08:25.374 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T22:08:25.374 INFO:teuthology.orchestra.run.smithi062.stdout:max_xattr_size 65536 2024-04-20T22:08:25.374 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T22:08:25.374 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T22:08:25.374 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 81 2024-04-20T22:08:25.374 INFO:teuthology.orchestra.run.smithi062.stdout:snap_rstat 1 2024-04-20T22:08:25.374 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:08:25.374 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 2 2024-04-20T22:08:25.374 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1 2024-04-20T22:08:25.374 INFO:teuthology.orchestra.run.smithi062.stdout:up {} 2024-04-20T22:08:25.374 INFO:teuthology.orchestra.run.smithi062.stdout:failed 0,1 2024-04-20T22:08:25.374 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T22:08:25.375 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T22:08:25.375 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T22:08:25.375 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T22:08:25.375 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data enabled 2024-04-20T22:08:25.375 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T22:08:25.375 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T22:08:25.375 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 1 2024-04-20T22:08:25.375 INFO:teuthology.orchestra.run.smithi062.stdout:qdb_cluster leader: 0 members: 2024-04-20T22:08:25.375 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:08:25.375 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:08:25.375 INFO:teuthology.orchestra.run.smithi062.stdout:Standby daemons: 2024-04-20T22:08:25.375 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:08:25.375 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ryqblb{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6826/4113916205,v1:172.21.15.62:6827/4113916205] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:08:25.375 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ozizwi{-1:34476} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6828/3801975165,v1:172.21.15.62:6829/3801975165] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:08:25.375 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.xbgspw{-1:44285} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6826/3996040526,v1:172.21.15.189:6827/3996040526] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T22:08:25.376 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.dvimer{-1:44727} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6824/3576699232,v1:172.21.15.189:6825/3576699232] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:08:25.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.375+0000 7f9079600700 1 -- 172.21.15.62:0/1037525077 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f905806ecc0 msgr2=0x7f9058071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:08:25.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.375+0000 7f9079600700 1 --2- 172.21.15.62:0/1037525077 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f905806ecc0 0x7f9058071170 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f9060006010 tx=0x7f906000b540 comp rx=0 tx=0).stop 2024-04-20T22:08:25.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.375+0000 7f9079600700 1 -- 172.21.15.62:0/1037525077 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90741026a0 msgr2=0x7f9074198030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:08:25.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.375+0000 7f9079600700 1 --2- 172.21.15.62:0/1037525077 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90741026a0 0x7f9074198030 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f906400c930 tx=0x7f906400ccf0 comp rx=0 tx=0).stop 2024-04-20T22:08:25.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.375+0000 7f9079600700 1 -- 172.21.15.62:0/1037525077 shutdown_connections 2024-04-20T22:08:25.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.375+0000 7f9079600700 1 --2- 172.21.15.62:0/1037525077 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f905806ecc0 0x7f9058071170 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:08:25.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.375+0000 7f9079600700 1 --2- 172.21.15.62:0/1037525077 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90741026a0 0x7f9074198030 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:08:25.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.375+0000 7f9079600700 1 --2- 172.21.15.62:0/1037525077 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9074103040 0x7f9074198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:08:25.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.375+0000 7f9079600700 1 -- 172.21.15.62:0/1037525077 >> 172.21.15.62:0/1037525077 conn(0x7f90740fdf40 msgr2=0x7f9074106880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:08:25.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.375+0000 7f9079600700 1 -- 172.21.15.62:0/1037525077 shutdown_connections 2024-04-20T22:08:25.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.375+0000 7f9079600700 1 -- 172.21.15.62:0/1037525077 wait complete. 2024-04-20T22:08:25.377 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 27 2024-04-20T22:08:25.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:08:25 smithi189 ceph-mon[57022]: from='client.35768 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:08:25.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:08:25 smithi189 ceph-mon[57022]: pgmap v994: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:08:25.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:08:25 smithi189 ceph-mon[57022]: from='client.35772 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:08:25.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:08:25 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/1624973788' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:08:25.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:08:25 smithi062 ceph-mon[76322]: from='client.35768 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:08:25.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:08:25 smithi062 ceph-mon[76322]: pgmap v994: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:08:25.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:08:25 smithi062 ceph-mon[76322]: from='client.35772 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:08:25.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:08:25 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/1624973788' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:08:25.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.537+0000 7fee37200700 1 -- 172.21.15.62:0/1192809644 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee301004e0 msgr2=0x7fee301008b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:08:25.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.537+0000 7fee37200700 1 --2- 172.21.15.62:0/1192809644 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee301004e0 0x7fee301008b0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fee20009910 tx=0x7fee20009c20 comp rx=0 tx=0).stop 2024-04-20T22:08:25.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.537+0000 7fee37200700 1 -- 172.21.15.62:0/1192809644 shutdown_connections 2024-04-20T22:08:25.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.537+0000 7fee37200700 1 --2- 172.21.15.62:0/1192809644 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee30100e80 0x7fee30104e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:08:25.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.537+0000 7fee37200700 1 --2- 172.21.15.62:0/1192809644 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee301004e0 0x7fee301008b0 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:08:25.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.537+0000 7fee37200700 1 -- 172.21.15.62:0/1192809644 >> 172.21.15.62:0/1192809644 conn(0x7fee300fbda0 msgr2=0x7fee300fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:08:25.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.538+0000 7fee37200700 1 -- 172.21.15.62:0/1192809644 shutdown_connections 2024-04-20T22:08:25.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.538+0000 7fee37200700 1 -- 172.21.15.62:0/1192809644 wait complete. 2024-04-20T22:08:25.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.539+0000 7fee37200700 1 Processor -- start 2024-04-20T22:08:25.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.539+0000 7fee37200700 1 -- start start 2024-04-20T22:08:25.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.539+0000 7fee37200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee301004e0 0x7fee30195f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:08:25.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.539+0000 7fee37200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee30100e80 0x7fee301964d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:08:25.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.539+0000 7fee37200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee30196bb0 con 0x7fee301004e0 2024-04-20T22:08:25.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.539+0000 7fee37200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee3019a940 con 0x7fee30100e80 2024-04-20T22:08:25.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.540+0000 7fee2fe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee301004e0 0x7fee30195f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:08:25.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.540+0000 7fee2fe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee301004e0 0x7fee30195f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56752/0 (socket says 172.21.15.62:56752) 2024-04-20T22:08:25.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.540+0000 7fee2fe00700 1 -- 172.21.15.62:0/3471101431 learned_addr learned my addr 172.21.15.62:0/3471101431 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:08:25.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.540+0000 7fee27e00700 1 --2- 172.21.15.62:0/3471101431 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee30100e80 0x7fee301964d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:08:25.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.540+0000 7fee2fe00700 1 -- 172.21.15.62:0/3471101431 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee30100e80 msgr2=0x7fee301964d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:08:25.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.540+0000 7fee2fe00700 1 --2- 172.21.15.62:0/3471101431 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee30100e80 0x7fee301964d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:08:25.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.540+0000 7fee2fe00700 1 -- 172.21.15.62:0/3471101431 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee200095f0 con 0x7fee301004e0 2024-04-20T22:08:25.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.540+0000 7fee27e00700 1 --2- 172.21.15.62:0/3471101431 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee30100e80 0x7fee301964d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T22:08:25.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.541+0000 7fee2fe00700 1 --2- 172.21.15.62:0/3471101431 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee301004e0 0x7fee30195f90 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fee2000b5c0 tx=0x7fee2000be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:08:25.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.541+0000 7fee2d600700 1 -- 172.21.15.62:0/3471101431 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fee2001d070 con 0x7fee301004e0 2024-04-20T22:08:25.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.541+0000 7fee2d600700 1 -- 172.21.15.62:0/3471101431 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fee2000f460 con 0x7fee301004e0 2024-04-20T22:08:25.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.541+0000 7fee2d600700 1 -- 172.21.15.62:0/3471101431 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fee20017610 con 0x7fee301004e0 2024-04-20T22:08:25.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.541+0000 7fee37200700 1 -- 172.21.15.62:0/3471101431 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee3019abc0 con 0x7fee301004e0 2024-04-20T22:08:25.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.541+0000 7fee37200700 1 -- 172.21.15.62:0/3471101431 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee3019b0b0 con 0x7fee301004e0 2024-04-20T22:08:25.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.543+0000 7fee37200700 1 -- 172.21.15.62:0/3471101431 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee301087f0 con 0x7fee301004e0 2024-04-20T22:08:25.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.544+0000 7fee2d600700 1 -- 172.21.15.62:0/3471101431 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fee2000f5d0 con 0x7fee301004e0 2024-04-20T22:08:25.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.545+0000 7fee2d600700 1 --2- 172.21.15.62:0/3471101431 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fee1406ecc0 0x7fee14071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:08:25.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.545+0000 7fee2d600700 1 -- 172.21.15.62:0/3471101431 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fee20090480 con 0x7fee301004e0 2024-04-20T22:08:25.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.546+0000 7fee27e00700 1 --2- 172.21.15.62:0/3471101431 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fee1406ecc0 0x7fee14071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:08:25.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.546+0000 7fee27e00700 1 --2- 172.21.15.62:0/3471101431 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fee1406ecc0 0x7fee14071170 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fee301975b0 tx=0x7fee18008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:08:25.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.550+0000 7fee2d600700 1 -- 172.21.15.62:0/3471101431 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fee20059230 con 0x7fee301004e0 2024-04-20T22:08:25.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.806+0000 7fee37200700 1 -- 172.21.15.62:0/3471101431 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fee3002cca0 con 0x7fee1406ecc0 2024-04-20T22:08:25.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.809+0000 7fee2d600700 1 -- 172.21.15.62:0/3471101431 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fee3002cca0 con 0x7fee1406ecc0 2024-04-20T22:08:25.809 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:08:25.809 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T22:08:25.809 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T22:08:25.809 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-20T22:08:25.809 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T22:08:25.809 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr", 2024-04-20T22:08:25.809 INFO:teuthology.orchestra.run.smithi062.stdout: "crash", 2024-04-20T22:08:25.809 INFO:teuthology.orchestra.run.smithi062.stdout: "mon", 2024-04-20T22:08:25.809 INFO:teuthology.orchestra.run.smithi062.stdout: "osd" 2024-04-20T22:08:25.809 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T22:08:25.810 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "15/23 daemons upgraded", 2024-04-20T22:08:25.810 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading mds daemons", 2024-04-20T22:08:25.810 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T22:08:25.810 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:08:25.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.812+0000 7fee37200700 1 -- 172.21.15.62:0/3471101431 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fee1406ecc0 msgr2=0x7fee14071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:08:25.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.812+0000 7fee37200700 1 --2- 172.21.15.62:0/3471101431 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fee1406ecc0 0x7fee14071170 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fee301975b0 tx=0x7fee18008040 comp rx=0 tx=0).stop 2024-04-20T22:08:25.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.812+0000 7fee37200700 1 -- 172.21.15.62:0/3471101431 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee301004e0 msgr2=0x7fee30195f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:08:25.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.812+0000 7fee37200700 1 --2- 172.21.15.62:0/3471101431 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee301004e0 0x7fee30195f90 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fee2000b5c0 tx=0x7fee2000be00 comp rx=0 tx=0).stop 2024-04-20T22:08:25.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.813+0000 7fee37200700 1 -- 172.21.15.62:0/3471101431 shutdown_connections 2024-04-20T22:08:25.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.813+0000 7fee37200700 1 --2- 172.21.15.62:0/3471101431 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fee1406ecc0 0x7fee14071170 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:08:25.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.813+0000 7fee37200700 1 --2- 172.21.15.62:0/3471101431 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee301004e0 0x7fee30195f90 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:08:25.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.813+0000 7fee37200700 1 --2- 172.21.15.62:0/3471101431 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee30100e80 0x7fee301964d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:08:25.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.813+0000 7fee37200700 1 -- 172.21.15.62:0/3471101431 >> 172.21.15.62:0/3471101431 conn(0x7fee300fbda0 msgr2=0x7fee300fd470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:08:25.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.813+0000 7fee37200700 1 -- 172.21.15.62:0/3471101431 shutdown_connections 2024-04-20T22:08:25.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.813+0000 7fee37200700 1 -- 172.21.15.62:0/3471101431 wait complete. 2024-04-20T22:08:25.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.978+0000 7f7f42c00700 1 -- 172.21.15.62:0/3626730350 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f3c068540 msgr2=0x7f7f3c0689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:08:25.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.978+0000 7f7f42c00700 1 --2- 172.21.15.62:0/3626730350 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f3c068540 0x7f7f3c0689b0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f7f28009910 tx=0x7f7f28009c20 comp rx=0 tx=0).stop 2024-04-20T22:08:25.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.978+0000 7f7f42c00700 1 -- 172.21.15.62:0/3626730350 shutdown_connections 2024-04-20T22:08:25.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.978+0000 7f7f42c00700 1 --2- 172.21.15.62:0/3626730350 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f3c068540 0x7f7f3c0689b0 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:08:25.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.978+0000 7f7f42c00700 1 --2- 172.21.15.62:0/3626730350 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f3c106320 0x7f7f3c1066f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:08:25.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.978+0000 7f7f42c00700 1 -- 172.21.15.62:0/3626730350 >> 172.21.15.62:0/3626730350 conn(0x7f7f3c075130 msgr2=0x7f7f3c075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:08:25.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.979+0000 7f7f42c00700 1 -- 172.21.15.62:0/3626730350 shutdown_connections 2024-04-20T22:08:25.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.979+0000 7f7f42c00700 1 -- 172.21.15.62:0/3626730350 wait complete. 2024-04-20T22:08:25.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.980+0000 7f7f42c00700 1 Processor -- start 2024-04-20T22:08:25.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.980+0000 7f7f42c00700 1 -- start start 2024-04-20T22:08:25.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.980+0000 7f7f42c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f3c068540 0x7f7f3c0ffca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:08:25.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.980+0000 7f7f42c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f3c106320 0x7f7f3c1001e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:08:25.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.980+0000 7f7f42c00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f3c1007b0 con 0x7f7f3c068540 2024-04-20T22:08:25.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.980+0000 7f7f42c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f3c100920 con 0x7f7f3c106320 2024-04-20T22:08:25.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.981+0000 7f7f3be00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f3c068540 0x7f7f3c0ffca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:08:25.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.981+0000 7f7f3be00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f3c068540 0x7f7f3c0ffca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56768/0 (socket says 172.21.15.62:56768) 2024-04-20T22:08:25.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.981+0000 7f7f3be00700 1 -- 172.21.15.62:0/3299759966 learned_addr learned my addr 172.21.15.62:0/3299759966 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:08:25.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.981+0000 7f7f3b400700 1 --2- 172.21.15.62:0/3299759966 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f3c106320 0x7f7f3c1001e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:08:25.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.981+0000 7f7f3be00700 1 -- 172.21.15.62:0/3299759966 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f3c106320 msgr2=0x7f7f3c1001e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:08:25.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.981+0000 7f7f3be00700 1 --2- 172.21.15.62:0/3299759966 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f3c106320 0x7f7f3c1001e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:08:25.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.981+0000 7f7f3be00700 1 -- 172.21.15.62:0/3299759966 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f280095f0 con 0x7f7f3c068540 2024-04-20T22:08:25.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.981+0000 7f7f3be00700 1 --2- 172.21.15.62:0/3299759966 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f3c068540 0x7f7f3c0ffca0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f7f2c00c8a0 tx=0x7f7f2c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:08:25.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.982+0000 7f7f38c00700 1 -- 172.21.15.62:0/3299759966 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7f2c00cea0 con 0x7f7f3c068540 2024-04-20T22:08:25.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.982+0000 7f7f38c00700 1 -- 172.21.15.62:0/3299759966 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7f2c004830 con 0x7f7f3c068540 2024-04-20T22:08:25.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.982+0000 7f7f42c00700 1 -- 172.21.15.62:0/3299759966 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f3c1a2030 con 0x7f7f3c068540 2024-04-20T22:08:25.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.982+0000 7f7f38c00700 1 -- 172.21.15.62:0/3299759966 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7f2c0055b0 con 0x7f7f3c068540 2024-04-20T22:08:25.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.982+0000 7f7f42c00700 1 -- 172.21.15.62:0/3299759966 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f3c1a24a0 con 0x7f7f3c068540 2024-04-20T22:08:25.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.983+0000 7f7f42c00700 1 -- 172.21.15.62:0/3299759966 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f3c108770 con 0x7f7f3c068540 2024-04-20T22:08:25.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.984+0000 7f7f38c00700 1 -- 172.21.15.62:0/3299759966 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f7f2c0049a0 con 0x7f7f3c068540 2024-04-20T22:08:25.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.985+0000 7f7f38c00700 1 --2- 172.21.15.62:0/3299759966 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7f2006ecc0 0x7f7f20071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:08:25.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.985+0000 7f7f38c00700 1 -- 172.21.15.62:0/3299759966 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f7f2c08eaf0 con 0x7f7f3c068540 2024-04-20T22:08:25.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.986+0000 7f7f3b400700 1 --2- 172.21.15.62:0/3299759966 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7f2006ecc0 0x7f7f20071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:08:25.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.986+0000 7f7f3b400700 1 --2- 172.21.15.62:0/3299759966 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7f2006ecc0 0x7f7f20071170 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f7f28006010 tx=0x7f7f2800b540 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:08:25.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:25.990+0000 7f7f38c00700 1 -- 172.21.15.62:0/3299759966 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f7f2c057880 con 0x7f7f3c068540 2024-04-20T22:08:26.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:26.319+0000 7f7f42c00700 1 -- 172.21.15.62:0/3299759966 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f7f3c108150 con 0x7f7f3c068540 2024-04-20T22:08:26.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:26.320+0000 7f7f38c00700 1 -- 172.21.15.62:0/3299759966 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7f7f2c01d090 con 0x7f7f3c068540 2024-04-20T22:08:26.321 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-20T22:08:26.321 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-20T22:08:26.321 INFO:teuthology.orchestra.run.smithi062.stdout: host smithi062 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T22:08:26.321 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-20T22:08:26.321 INFO:teuthology.orchestra.run.smithi062.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi062 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpq7y7oa_r:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:08:26.321 INFO:teuthology.orchestra.run.smithi062.stdout:Traceback (most recent call last): 2024-04-20T22:08:26.321 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-20T22:08:26.321 INFO:teuthology.orchestra.run.smithi062.stdout: return _run_code(code, main_globals, None, 2024-04-20T22:08:26.321 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-20T22:08:26.322 INFO:teuthology.orchestra.run.smithi062.stdout: exec(code, run_globals) 2024-04-20T22:08:26.322 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5618, in 2024-04-20T22:08:26.322 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5606, in main 2024-04-20T22:08:26.322 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 408, in _infer_config 2024-04-20T22:08:26.322 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 323, in _infer_fsid 2024-04-20T22:08:26.322 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 436, in _infer_image 2024-04-20T22:08:26.322 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 310, in _validate_fsid 2024-04-20T22:08:26.322 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 3359, in command_ceph_volume 2024-04-20T22:08:26.322 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-20T22:08:26.322 INFO:teuthology.orchestra.run.smithi062.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi062 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpq7y7oa_r:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:08:26.322 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-20T22:08:26.322 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is degraded 2024-04-20T22:08:26.322 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T22:08:26.322 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T22:08:26.323 INFO:teuthology.orchestra.run.smithi062.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-20T22:08:26.323 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-20T22:08:26.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:26.323+0000 7f7f42c00700 1 -- 172.21.15.62:0/3299759966 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7f2006ecc0 msgr2=0x7f7f20071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:08:26.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:26.323+0000 7f7f42c00700 1 --2- 172.21.15.62:0/3299759966 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7f2006ecc0 0x7f7f20071170 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f7f28006010 tx=0x7f7f2800b540 comp rx=0 tx=0).stop 2024-04-20T22:08:26.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:26.323+0000 7f7f42c00700 1 -- 172.21.15.62:0/3299759966 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f3c068540 msgr2=0x7f7f3c0ffca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:08:26.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:26.323+0000 7f7f42c00700 1 --2- 172.21.15.62:0/3299759966 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f3c068540 0x7f7f3c0ffca0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f7f2c00c8a0 tx=0x7f7f2c00cc60 comp rx=0 tx=0).stop 2024-04-20T22:08:26.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:26.323+0000 7f7f42c00700 1 -- 172.21.15.62:0/3299759966 shutdown_connections 2024-04-20T22:08:26.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:26.323+0000 7f7f42c00700 1 --2- 172.21.15.62:0/3299759966 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7f2006ecc0 0x7f7f20071170 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:08:26.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:26.323+0000 7f7f42c00700 1 --2- 172.21.15.62:0/3299759966 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f3c068540 0x7f7f3c0ffca0 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:08:26.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:26.323+0000 7f7f42c00700 1 --2- 172.21.15.62:0/3299759966 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f3c106320 0x7f7f3c1001e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:08:26.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:26.323+0000 7f7f42c00700 1 -- 172.21.15.62:0/3299759966 >> 172.21.15.62:0/3299759966 conn(0x7f7f3c075130 msgr2=0x7f7f3c0febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:08:26.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:26.323+0000 7f7f42c00700 1 -- 172.21.15.62:0/3299759966 shutdown_connections 2024-04-20T22:08:26.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:26.323+0000 7f7f42c00700 1 -- 172.21.15.62:0/3299759966 wait complete. 2024-04-20T22:08:26.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:08:26 smithi189 ceph-mon[57022]: from='client.35776 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:08:26.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:08:26 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/1037525077' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:08:26.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:08:26 smithi062 ceph-mon[76322]: from='client.35776 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:08:26.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:08:26 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/1037525077' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:08:27.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:08:27 smithi189 ceph-mon[57022]: pgmap v995: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:08:27.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:08:27 smithi189 ceph-mon[57022]: from='client.35788 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:08:27.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:08:27 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/3299759966' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:08:27.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:08:27 smithi062 ceph-mon[76322]: pgmap v995: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:08:27.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:08:27 smithi062 ceph-mon[76322]: from='client.35788 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:08:27.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:08:27 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/3299759966' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:08:29.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:08:29 smithi062 ceph-mon[76322]: pgmap v996: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:08:29.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:08:29 smithi189 ceph-mon[57022]: pgmap v996: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:08:31.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:08:31 smithi062 ceph-mon[76322]: pgmap v997: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:08:31.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:08:31 smithi189 ceph-mon[57022]: pgmap v997: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:08:33.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:08:33 smithi062 ceph-mon[76322]: pgmap v998: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:08:33.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:08:33 smithi189 ceph-mon[57022]: pgmap v998: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:08:35.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:08:35 smithi062 ceph-mon[76322]: pgmap v999: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:08:35.537 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:08:35 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:08:35.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:08:35 smithi189 ceph-mon[57022]: pgmap v999: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:08:35.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:08:35 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:08:37.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:08:37 smithi062 ceph-mon[76322]: pgmap v1000: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:08:37.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:08:37 smithi189 ceph-mon[57022]: pgmap v1000: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:08:39.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:08:39 smithi062 ceph-mon[76322]: pgmap v1001: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:08:39.540 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:08:39 smithi189 ceph-mon[57022]: pgmap v1001: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:08:41.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:08:41 smithi062 ceph-mon[76322]: pgmap v1002: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:08:41.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:08:41 smithi189 ceph-mon[57022]: pgmap v1002: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:08:43.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:08:43 smithi062 ceph-mon[76322]: pgmap v1003: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:08:43.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:08:43 smithi189 ceph-mon[57022]: pgmap v1003: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:08:44.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:08:44 smithi189 ceph-mon[57022]: pgmap v1004: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:08:44.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:08:44 smithi062 ceph-mon[76322]: pgmap v1004: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:08:46.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:08:46 smithi189 ceph-mon[57022]: pgmap v1005: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:08:47.037 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:08:46 smithi062 ceph-mon[76322]: pgmap v1005: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:08:48.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:08:48 smithi189 ceph-mon[57022]: pgmap v1006: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:08:49.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:08:48 smithi062 ceph-mon[76322]: pgmap v1006: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:08:49.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:08:49 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:08:50.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:08:49 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:08:50.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:08:50 smithi189 ceph-mon[57022]: pgmap v1007: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:08:51.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:08:50 smithi062 ceph-mon[76322]: pgmap v1007: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:08:53.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:08:52 smithi062 ceph-mon[76322]: pgmap v1008: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:08:53.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:08:52 smithi189 ceph-mon[57022]: pgmap v1008: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:08:55.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:08:54 smithi062 ceph-mon[76322]: pgmap v1009: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:08:55.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:08:54 smithi189 ceph-mon[57022]: pgmap v1009: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:08:56.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:56.499+0000 7fd611a00700 1 -- 172.21.15.62:0/1206848848 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd60c10c8f0 msgr2=0x7fd60c10ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:08:56.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:56.499+0000 7fd611a00700 1 --2- 172.21.15.62:0/1206848848 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd60c10c8f0 0x7fd60c10ccc0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc009910 tx=0x7fd5fc009c20 comp rx=0 tx=0).stop 2024-04-20T22:08:56.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:56.500+0000 7fd611a00700 1 -- 172.21.15.62:0/1206848848 shutdown_connections 2024-04-20T22:08:56.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:56.500+0000 7fd611a00700 1 --2- 172.21.15.62:0/1206848848 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd60c106b80 0x7fd60c106ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:08:56.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:56.500+0000 7fd611a00700 1 --2- 172.21.15.62:0/1206848848 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd60c10c8f0 0x7fd60c10ccc0 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:08:56.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:56.500+0000 7fd611a00700 1 -- 172.21.15.62:0/1206848848 >> 172.21.15.62:0/1206848848 conn(0x7fd60c076060 msgr2=0x7fd60c078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:08:56.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:56.500+0000 7fd611a00700 1 -- 172.21.15.62:0/1206848848 shutdown_connections 2024-04-20T22:08:56.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:56.500+0000 7fd611a00700 1 -- 172.21.15.62:0/1206848848 wait complete. 2024-04-20T22:08:56.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:56.501+0000 7fd611a00700 1 Processor -- start 2024-04-20T22:08:56.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:56.501+0000 7fd611a00700 1 -- start start 2024-04-20T22:08:56.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:56.502+0000 7fd611a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd60c106b80 0x7fd60c19c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:08:56.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:56.502+0000 7fd611a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd60c10c8f0 0x7fd60c19ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:08:56.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:56.502+0000 7fd611a00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd60c19d100 con 0x7fd60c10c8f0 2024-04-20T22:08:56.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:56.502+0000 7fd611a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd60c1a0e90 con 0x7fd60c106b80 2024-04-20T22:08:56.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:56.502+0000 7fd60a000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd60c10c8f0 0x7fd60c19ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:08:56.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:56.502+0000 7fd60a000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd60c10c8f0 0x7fd60c19ca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56274/0 (socket says 172.21.15.62:56274) 2024-04-20T22:08:56.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:56.502+0000 7fd60a000700 1 -- 172.21.15.62:0/4033273564 learned_addr learned my addr 172.21.15.62:0/4033273564 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:08:56.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:56.503+0000 7fd60aa00700 1 --2- 172.21.15.62:0/4033273564 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd60c106b80 0x7fd60c19c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:08:56.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:56.503+0000 7fd60a000700 1 -- 172.21.15.62:0/4033273564 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd60c106b80 msgr2=0x7fd60c19c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:08:56.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:56.503+0000 7fd60a000700 1 --2- 172.21.15.62:0/4033273564 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd60c106b80 0x7fd60c19c4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:08:56.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:56.503+0000 7fd60a000700 1 -- 172.21.15.62:0/4033273564 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5fc0095f0 con 0x7fd60c10c8f0 2024-04-20T22:08:56.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:56.503+0000 7fd60aa00700 1 --2- 172.21.15.62:0/4033273564 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd60c106b80 0x7fd60c19c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T22:08:56.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:56.503+0000 7fd60a000700 1 --2- 172.21.15.62:0/4033273564 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd60c10c8f0 0x7fd60c19ca20 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fd5f800b4b0 tx=0x7fd5f800b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:08:56.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:56.503+0000 7fd603400700 1 -- 172.21.15.62:0/4033273564 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd5f8011800 con 0x7fd60c10c8f0 2024-04-20T22:08:56.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:56.504+0000 7fd603400700 1 -- 172.21.15.62:0/4033273564 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd5f8011e20 con 0x7fd60c10c8f0 2024-04-20T22:08:56.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:56.504+0000 7fd611a00700 1 -- 172.21.15.62:0/4033273564 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd60c1a1170 con 0x7fd60c10c8f0 2024-04-20T22:08:56.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:56.504+0000 7fd603400700 1 -- 172.21.15.62:0/4033273564 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd5f800f2e0 con 0x7fd60c10c8f0 2024-04-20T22:08:56.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:56.504+0000 7fd611a00700 1 -- 172.21.15.62:0/4033273564 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd60c1a16c0 con 0x7fd60c10c8f0 2024-04-20T22:08:56.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:56.505+0000 7fd611a00700 1 -- 172.21.15.62:0/4033273564 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd60c109b20 con 0x7fd60c10c8f0 2024-04-20T22:08:56.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:56.506+0000 7fd603400700 1 -- 172.21.15.62:0/4033273564 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fd5f8010380 con 0x7fd60c10c8f0 2024-04-20T22:08:56.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:56.507+0000 7fd603400700 1 --2- 172.21.15.62:0/4033273564 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd5f006ecc0 0x7fd5f0071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:08:56.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:56.507+0000 7fd603400700 1 -- 172.21.15.62:0/4033273564 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fd5f808ea30 con 0x7fd60c10c8f0 2024-04-20T22:08:56.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:56.507+0000 7fd60aa00700 1 --2- 172.21.15.62:0/4033273564 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd5f006ecc0 0x7fd5f0071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:08:56.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:56.508+0000 7fd60aa00700 1 --2- 172.21.15.62:0/4033273564 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd5f006ecc0 0x7fd5f0071170 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc006010 tx=0x7fd5fc00b540 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:08:56.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:56.512+0000 7fd603400700 1 -- 172.21.15.62:0/4033273564 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fd5f8057870 con 0x7fd60c10c8f0 2024-04-20T22:08:56.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:56.769+0000 7fd611a00700 1 -- 172.21.15.62:0/4033273564 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd60c1a1990 con 0x7fd5f006ecc0 2024-04-20T22:08:56.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:56.772+0000 7fd603400700 1 -- 172.21.15.62:0/4033273564 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fd60c1a1990 con 0x7fd5f006ecc0 2024-04-20T22:08:56.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:56.775+0000 7fd611a00700 1 -- 172.21.15.62:0/4033273564 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd5f006ecc0 msgr2=0x7fd5f0071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:08:56.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:56.775+0000 7fd611a00700 1 --2- 172.21.15.62:0/4033273564 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd5f006ecc0 0x7fd5f0071170 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc006010 tx=0x7fd5fc00b540 comp rx=0 tx=0).stop 2024-04-20T22:08:56.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:56.775+0000 7fd611a00700 1 -- 172.21.15.62:0/4033273564 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd60c10c8f0 msgr2=0x7fd60c19ca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:08:56.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:56.775+0000 7fd611a00700 1 --2- 172.21.15.62:0/4033273564 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd60c10c8f0 0x7fd60c19ca20 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fd5f800b4b0 tx=0x7fd5f800b870 comp rx=0 tx=0).stop 2024-04-20T22:08:56.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:56.775+0000 7fd611a00700 1 -- 172.21.15.62:0/4033273564 shutdown_connections 2024-04-20T22:08:56.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:56.775+0000 7fd611a00700 1 --2- 172.21.15.62:0/4033273564 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd5f006ecc0 0x7fd5f0071170 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:08:56.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:56.775+0000 7fd611a00700 1 --2- 172.21.15.62:0/4033273564 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd60c106b80 0x7fd60c19c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:08:56.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:56.775+0000 7fd611a00700 1 --2- 172.21.15.62:0/4033273564 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd60c10c8f0 0x7fd60c19ca20 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:08:56.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:56.775+0000 7fd611a00700 1 -- 172.21.15.62:0/4033273564 >> 172.21.15.62:0/4033273564 conn(0x7fd60c076060 msgr2=0x7fd60c077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:08:56.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:56.776+0000 7fd611a00700 1 -- 172.21.15.62:0/4033273564 shutdown_connections 2024-04-20T22:08:56.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:56.776+0000 7fd611a00700 1 -- 172.21.15.62:0/4033273564 wait complete. 2024-04-20T22:08:56.792 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T22:08:56.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:56.947+0000 7f336ec00700 1 -- 172.21.15.62:0/999140446 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3368100290 msgr2=0x7f3368100660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:08:56.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:56.947+0000 7f336ec00700 1 --2- 172.21.15.62:0/999140446 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3368100290 0x7f3368100660 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f335c009910 tx=0x7f335c009c20 comp rx=0 tx=0).stop 2024-04-20T22:08:56.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:56.947+0000 7f336ec00700 1 -- 172.21.15.62:0/999140446 shutdown_connections 2024-04-20T22:08:56.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:56.947+0000 7f336ec00700 1 --2- 172.21.15.62:0/999140446 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3368100c30 0x7f3368104c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:08:56.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:56.947+0000 7f336ec00700 1 --2- 172.21.15.62:0/999140446 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3368100290 0x7f3368100660 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:08:56.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:56.947+0000 7f336ec00700 1 -- 172.21.15.62:0/999140446 >> 172.21.15.62:0/999140446 conn(0x7f33680fbb10 msgr2=0x7f33680fdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:08:56.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:56.947+0000 7f336ec00700 1 -- 172.21.15.62:0/999140446 shutdown_connections 2024-04-20T22:08:56.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:56.947+0000 7f336ec00700 1 -- 172.21.15.62:0/999140446 wait complete. 2024-04-20T22:08:56.949 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:56.948+0000 7f336ec00700 1 Processor -- start 2024-04-20T22:08:56.949 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:56.949+0000 7f336ec00700 1 -- start start 2024-04-20T22:08:56.949 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:56.949+0000 7f336ec00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3368100290 0x7f3368197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:08:56.949 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:56.949+0000 7f336ec00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3368100c30 0x7f33681984f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:08:56.949 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:56.949+0000 7f336ec00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3368198bd0 con 0x7f3368100290 2024-04-20T22:08:56.949 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:56.949+0000 7f336ec00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f336819c8c0 con 0x7f3368100c30 2024-04-20T22:08:56.949 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:56.949+0000 7f336d800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3368100290 0x7f3368197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:08:56.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:56.949+0000 7f336d800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3368100290 0x7f3368197fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56290/0 (socket says 172.21.15.62:56290) 2024-04-20T22:08:56.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:56.949+0000 7f336d800700 1 -- 172.21.15.62:0/657689758 learned_addr learned my addr 172.21.15.62:0/657689758 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:08:56.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:56.950+0000 7f336ce00700 1 --2- 172.21.15.62:0/657689758 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3368100c30 0x7f33681984f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:08:56.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:56.950+0000 7f336d800700 1 -- 172.21.15.62:0/657689758 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3368100c30 msgr2=0x7f33681984f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:08:56.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:56.950+0000 7f336d800700 1 --2- 172.21.15.62:0/657689758 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3368100c30 0x7f33681984f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:08:56.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:56.950+0000 7f336d800700 1 -- 172.21.15.62:0/657689758 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f335c0095f0 con 0x7f3368100290 2024-04-20T22:08:56.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:56.950+0000 7f336d800700 1 --2- 172.21.15.62:0/657689758 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3368100290 0x7f3368197fb0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f335c006010 tx=0x7f335c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:08:56.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:56.951+0000 7f3366000700 1 -- 172.21.15.62:0/657689758 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f335c01d070 con 0x7f3368100290 2024-04-20T22:08:56.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:56.951+0000 7f3366000700 1 -- 172.21.15.62:0/657689758 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f335c004fb0 con 0x7f3368100290 2024-04-20T22:08:56.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:56.951+0000 7f3366000700 1 -- 172.21.15.62:0/657689758 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f335c022ea0 con 0x7f3368100290 2024-04-20T22:08:56.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:56.951+0000 7f336ec00700 1 -- 172.21.15.62:0/657689758 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f336819cb40 con 0x7f3368100290 2024-04-20T22:08:56.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:56.951+0000 7f336ec00700 1 -- 172.21.15.62:0/657689758 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f336819d030 con 0x7f3368100290 2024-04-20T22:08:56.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:56.952+0000 7f336ec00700 1 -- 172.21.15.62:0/657689758 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3368104380 con 0x7f3368100290 2024-04-20T22:08:56.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:56.953+0000 7f3366000700 1 -- 172.21.15.62:0/657689758 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f335c022470 con 0x7f3368100290 2024-04-20T22:08:56.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:56.954+0000 7f3366000700 1 --2- 172.21.15.62:0/657689758 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f335006ecc0 0x7f3350071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:08:56.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:56.954+0000 7f3366000700 1 -- 172.21.15.62:0/657689758 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f335c0903c0 con 0x7f3368100290 2024-04-20T22:08:56.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:56.954+0000 7f336ce00700 1 --2- 172.21.15.62:0/657689758 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f335006ecc0 0x7f3350071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:08:56.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:56.955+0000 7f336ce00700 1 --2- 172.21.15.62:0/657689758 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f335006ecc0 0x7f3350071170 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f3354006fd0 tx=0x7f3354008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:08:56.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:56.959+0000 7f3366000700 1 -- 172.21.15.62:0/657689758 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f335c059200 con 0x7f3368100290 2024-04-20T22:08:57.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:08:56 smithi062 ceph-mon[76322]: pgmap v1010: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:08:57.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:57.218+0000 7f336ec00700 1 -- 172.21.15.62:0/657689758 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f336819d360 con 0x7f335006ecc0 2024-04-20T22:08:57.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:57.220+0000 7f3366000700 1 -- 172.21.15.62:0/657689758 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f336819d360 con 0x7f335006ecc0 2024-04-20T22:08:57.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:57.223+0000 7f336ec00700 1 -- 172.21.15.62:0/657689758 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f335006ecc0 msgr2=0x7f3350071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:08:57.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:57.223+0000 7f336ec00700 1 --2- 172.21.15.62:0/657689758 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f335006ecc0 0x7f3350071170 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f3354006fd0 tx=0x7f3354008040 comp rx=0 tx=0).stop 2024-04-20T22:08:57.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:57.223+0000 7f336ec00700 1 -- 172.21.15.62:0/657689758 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3368100290 msgr2=0x7f3368197fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:08:57.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:57.223+0000 7f336ec00700 1 --2- 172.21.15.62:0/657689758 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3368100290 0x7f3368197fb0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f335c006010 tx=0x7f335c00be20 comp rx=0 tx=0).stop 2024-04-20T22:08:57.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:57.223+0000 7f336ec00700 1 -- 172.21.15.62:0/657689758 shutdown_connections 2024-04-20T22:08:57.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:57.223+0000 7f336ec00700 1 --2- 172.21.15.62:0/657689758 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f335006ecc0 0x7f3350071170 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:08:57.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:57.224+0000 7f336ec00700 1 --2- 172.21.15.62:0/657689758 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3368100290 0x7f3368197fb0 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:08:57.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:57.224+0000 7f336ec00700 1 --2- 172.21.15.62:0/657689758 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3368100c30 0x7f33681984f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:08:57.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:57.224+0000 7f336ec00700 1 -- 172.21.15.62:0/657689758 >> 172.21.15.62:0/657689758 conn(0x7f33680fbb10 msgr2=0x7f33680fd2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:08:57.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:57.224+0000 7f336ec00700 1 -- 172.21.15.62:0/657689758 shutdown_connections 2024-04-20T22:08:57.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:57.224+0000 7f336ec00700 1 -- 172.21.15.62:0/657689758 wait complete. 2024-04-20T22:08:57.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:08:56 smithi189 ceph-mon[57022]: pgmap v1010: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:08:57.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:57.391+0000 7ffa7de00700 1 -- 172.21.15.62:0/3423956199 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa780730f0 msgr2=0x7ffa780734c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:08:57.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:57.391+0000 7ffa7de00700 1 --2- 172.21.15.62:0/3423956199 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa780730f0 0x7ffa780734c0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7ffa68009910 tx=0x7ffa68009c20 comp rx=0 tx=0).stop 2024-04-20T22:08:57.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:57.391+0000 7ffa7de00700 1 -- 172.21.15.62:0/3423956199 shutdown_connections 2024-04-20T22:08:57.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:57.391+0000 7ffa7de00700 1 --2- 172.21.15.62:0/3423956199 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffa78073a00 0x7ffa7810c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:08:57.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:57.391+0000 7ffa7de00700 1 --2- 172.21.15.62:0/3423956199 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa780730f0 0x7ffa780734c0 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:08:57.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:57.391+0000 7ffa7de00700 1 -- 172.21.15.62:0/3423956199 >> 172.21.15.62:0/3423956199 conn(0x7ffa780fbe10 msgr2=0x7ffa780fe220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:08:57.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:57.391+0000 7ffa7de00700 1 -- 172.21.15.62:0/3423956199 shutdown_connections 2024-04-20T22:08:57.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:57.391+0000 7ffa7de00700 1 -- 172.21.15.62:0/3423956199 wait complete. 2024-04-20T22:08:57.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:57.392+0000 7ffa7de00700 1 Processor -- start 2024-04-20T22:08:57.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:57.392+0000 7ffa7de00700 1 -- start start 2024-04-20T22:08:57.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:57.392+0000 7ffa7de00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffa780730f0 0x7ffa78197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:08:57.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:57.393+0000 7ffa7de00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa78073a00 0x7ffa781984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:08:57.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:57.393+0000 7ffa7de00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa78198b80 con 0x7ffa78073a00 2024-04-20T22:08:57.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:57.393+0000 7ffa7de00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa7819c910 con 0x7ffa780730f0 2024-04-20T22:08:57.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:57.393+0000 7ffa6fe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa78073a00 0x7ffa781984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:08:57.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:57.393+0000 7ffa6fe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa78073a00 0x7ffa781984a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56318/0 (socket says 172.21.15.62:56318) 2024-04-20T22:08:57.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:57.393+0000 7ffa6fe00700 1 -- 172.21.15.62:0/2212837110 learned_addr learned my addr 172.21.15.62:0/2212837110 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:08:57.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:57.393+0000 7ffa76a00700 1 --2- 172.21.15.62:0/2212837110 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffa780730f0 0x7ffa78197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:08:57.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:57.393+0000 7ffa6fe00700 1 -- 172.21.15.62:0/2212837110 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffa780730f0 msgr2=0x7ffa78197f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:08:57.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:57.393+0000 7ffa6fe00700 1 --2- 172.21.15.62:0/2212837110 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffa780730f0 0x7ffa78197f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:08:57.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:57.393+0000 7ffa6fe00700 1 -- 172.21.15.62:0/2212837110 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa680095f0 con 0x7ffa78073a00 2024-04-20T22:08:57.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:57.394+0000 7ffa6fe00700 1 --2- 172.21.15.62:0/2212837110 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa78073a00 0x7ffa781984a0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7ffa6000c930 tx=0x7ffa6000ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:08:57.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:57.394+0000 7ffa7ca00700 1 -- 172.21.15.62:0/2212837110 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffa60007ad0 con 0x7ffa78073a00 2024-04-20T22:08:57.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:57.394+0000 7ffa7ca00700 1 -- 172.21.15.62:0/2212837110 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ffa60007c30 con 0x7ffa78073a00 2024-04-20T22:08:57.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:57.394+0000 7ffa7ca00700 1 -- 172.21.15.62:0/2212837110 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffa60018610 con 0x7ffa78073a00 2024-04-20T22:08:57.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:57.394+0000 7ffa7de00700 1 -- 172.21.15.62:0/2212837110 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa7819cbf0 con 0x7ffa78073a00 2024-04-20T22:08:57.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:57.394+0000 7ffa7de00700 1 -- 172.21.15.62:0/2212837110 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa7819d140 con 0x7ffa78073a00 2024-04-20T22:08:57.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:57.395+0000 7ffa7de00700 1 -- 172.21.15.62:0/2212837110 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa78109cd0 con 0x7ffa78073a00 2024-04-20T22:08:57.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:57.396+0000 7ffa7ca00700 1 -- 172.21.15.62:0/2212837110 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ffa6000f450 con 0x7ffa78073a00 2024-04-20T22:08:57.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:57.397+0000 7ffa7ca00700 1 --2- 172.21.15.62:0/2212837110 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ffa5c06ecc0 0x7ffa5c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:08:57.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:57.398+0000 7ffa7ca00700 1 -- 172.21.15.62:0/2212837110 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7ffa6008eff0 con 0x7ffa78073a00 2024-04-20T22:08:57.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:57.398+0000 7ffa76a00700 1 --2- 172.21.15.62:0/2212837110 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ffa5c06ecc0 0x7ffa5c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:08:57.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:57.398+0000 7ffa76a00700 1 --2- 172.21.15.62:0/2212837110 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ffa5c06ecc0 0x7ffa5c071170 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7ffa68000c00 tx=0x7ffa68005fd0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:08:57.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:57.402+0000 7ffa7ca00700 1 -- 172.21.15.62:0/2212837110 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7ffa60057de0 con 0x7ffa78073a00 2024-04-20T22:08:57.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:57.660+0000 7ffa7de00700 1 -- 172.21.15.62:0/2212837110 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7ffa7819d3c0 con 0x7ffa5c06ecc0 2024-04-20T22:08:57.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:57.673+0000 7ffa7ca00700 1 -- 172.21.15.62:0/2212837110 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7ffa7819d3c0 con 0x7ffa5c06ecc0 2024-04-20T22:08:57.674 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T22:08:57.674 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (34m) 10m ago 48m 15.3M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T22:08:57.674 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (49m) 10m ago 49m 10.4M - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T22:08:57.674 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (47m) 10m ago 47m 10.1M - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T22:08:57.674 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (30m) 10m ago 48m 6660k - 19.0.0-3133-ga8c84f59 76bc6b91b726 2825d5ba11c6 2024-04-20T22:08:57.674 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (30m) 10m ago 47m 6665k - 19.0.0-3133-ga8c84f59 76bc6b91b726 20f746fa1bba 2024-04-20T22:08:57.674 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (34m) 10m ago 48m 73.5M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T22:08:57.674 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (26m) 10m ago 41m 18.5M - 19.0.0-3133-ga8c84f59 76bc6b91b726 de80bc90a683 2024-04-20T22:08:57.674 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (26m) 10m ago 41m 18.7M - 19.0.0-3133-ga8c84f59 76bc6b91b726 cd793b7ee98e 2024-04-20T22:08:57.674 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (11m) 10m ago 41m 11.0M - 19.0.0-3133-ga8c84f59 76bc6b91b726 0e7b0bc4d1ab 2024-04-20T22:08:57.675 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (41m) 10m ago 41m 18.6M - 18.2.0 dc2bc1663786 f2eec2ee8951 2024-04-20T22:08:57.675 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (37m) 10m ago 50m 513M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T22:08:57.675 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (36m) 10m ago 47m 459M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T22:08:57.675 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (32m) 10m ago 50m 71.7M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 019eb1479485 2024-04-20T22:08:57.675 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (31m) 10m ago 47m 51.0M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 55aac560aab6 2024-04-20T22:08:57.675 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (35m) 10m ago 48m 8472k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T22:08:57.675 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (35m) 10m ago 47m 8048k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T22:08:57.675 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (30m) 10m ago 46m 91.2M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 ee2bb901f26e 2024-04-20T22:08:57.675 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (29m) 10m ago 45m 94.6M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 9075152d4162 2024-04-20T22:08:57.675 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (29m) 10m ago 45m 78.1M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 2d4f3e38dc0e 2024-04-20T22:08:57.675 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (28m) 10m ago 44m 92.6M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 8f97b17ae0eb 2024-04-20T22:08:57.675 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (27m) 10m ago 43m 95.6M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 653d8079967d 2024-04-20T22:08:57.676 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (27m) 10m ago 43m 89.2M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 596dded14c12 2024-04-20T22:08:57.676 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (35m) 10m ago 48m 82.1M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T22:08:57.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:57.676+0000 7ffa7de00700 1 -- 172.21.15.62:0/2212837110 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ffa5c06ecc0 msgr2=0x7ffa5c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:08:57.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:57.676+0000 7ffa7de00700 1 --2- 172.21.15.62:0/2212837110 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ffa5c06ecc0 0x7ffa5c071170 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7ffa68000c00 tx=0x7ffa68005fd0 comp rx=0 tx=0).stop 2024-04-20T22:08:57.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:57.676+0000 7ffa7de00700 1 -- 172.21.15.62:0/2212837110 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa78073a00 msgr2=0x7ffa781984a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:08:57.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:57.676+0000 7ffa7de00700 1 --2- 172.21.15.62:0/2212837110 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa78073a00 0x7ffa781984a0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7ffa6000c930 tx=0x7ffa6000ccf0 comp rx=0 tx=0).stop 2024-04-20T22:08:57.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:57.676+0000 7ffa7de00700 1 -- 172.21.15.62:0/2212837110 shutdown_connections 2024-04-20T22:08:57.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:57.676+0000 7ffa7de00700 1 --2- 172.21.15.62:0/2212837110 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ffa5c06ecc0 0x7ffa5c071170 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:08:57.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:57.676+0000 7ffa7de00700 1 --2- 172.21.15.62:0/2212837110 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffa780730f0 0x7ffa78197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:08:57.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:57.676+0000 7ffa7de00700 1 --2- 172.21.15.62:0/2212837110 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa78073a00 0x7ffa781984a0 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:08:57.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:57.676+0000 7ffa7de00700 1 -- 172.21.15.62:0/2212837110 >> 172.21.15.62:0/2212837110 conn(0x7ffa780fbe10 msgr2=0x7ffa78106e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:08:57.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:57.677+0000 7ffa7de00700 1 -- 172.21.15.62:0/2212837110 shutdown_connections 2024-04-20T22:08:57.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:57.677+0000 7ffa7de00700 1 -- 172.21.15.62:0/2212837110 wait complete. 2024-04-20T22:08:57.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:57.846+0000 7fdc95200700 1 -- 172.21.15.62:0/2924208150 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc901004e0 msgr2=0x7fdc901008b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:08:57.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:57.846+0000 7fdc95200700 1 --2- 172.21.15.62:0/2924208150 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc901004e0 0x7fdc901008b0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fdc8000b0a0 tx=0x7fdc8000b3b0 comp rx=0 tx=0).stop 2024-04-20T22:08:57.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:57.847+0000 7fdc95200700 1 -- 172.21.15.62:0/2924208150 shutdown_connections 2024-04-20T22:08:57.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:57.847+0000 7fdc95200700 1 --2- 172.21.15.62:0/2924208150 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc90100e80 0x7fdc90104e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:08:57.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:57.847+0000 7fdc95200700 1 --2- 172.21.15.62:0/2924208150 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc901004e0 0x7fdc901008b0 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:08:57.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:57.847+0000 7fdc95200700 1 -- 172.21.15.62:0/2924208150 >> 172.21.15.62:0/2924208150 conn(0x7fdc900fbda0 msgr2=0x7fdc900fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:08:57.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:57.847+0000 7fdc95200700 1 -- 172.21.15.62:0/2924208150 shutdown_connections 2024-04-20T22:08:57.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:57.847+0000 7fdc95200700 1 -- 172.21.15.62:0/2924208150 wait complete. 2024-04-20T22:08:57.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:57.848+0000 7fdc95200700 1 Processor -- start 2024-04-20T22:08:57.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:57.848+0000 7fdc95200700 1 -- start start 2024-04-20T22:08:57.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:57.848+0000 7fdc95200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc901004e0 0x7fdc90198260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:08:57.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:57.848+0000 7fdc95200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc90100e80 0x7fdc901987a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:08:57.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:57.848+0000 7fdc95200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc90198e80 con 0x7fdc90100e80 2024-04-20T22:08:57.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:57.848+0000 7fdc95200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc9019cc10 con 0x7fdc901004e0 2024-04-20T22:08:57.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:57.849+0000 7fdc85400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc90100e80 0x7fdc901987a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:08:57.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:57.849+0000 7fdc85400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc90100e80 0x7fdc901987a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56338/0 (socket says 172.21.15.62:56338) 2024-04-20T22:08:57.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:57.849+0000 7fdc85400700 1 -- 172.21.15.62:0/170244113 learned_addr learned my addr 172.21.15.62:0/170244113 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:08:57.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:57.849+0000 7fdc8de00700 1 --2- 172.21.15.62:0/170244113 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc901004e0 0x7fdc90198260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:08:57.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:57.849+0000 7fdc85400700 1 -- 172.21.15.62:0/170244113 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc901004e0 msgr2=0x7fdc90198260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:08:57.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:57.849+0000 7fdc85400700 1 --2- 172.21.15.62:0/170244113 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc901004e0 0x7fdc90198260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:08:57.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:57.849+0000 7fdc85400700 1 -- 172.21.15.62:0/170244113 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc80009d00 con 0x7fdc90100e80 2024-04-20T22:08:57.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:57.850+0000 7fdc85400700 1 --2- 172.21.15.62:0/170244113 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc90100e80 0x7fdc901987a0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7fdc7800b480 tx=0x7fdc7800b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:08:57.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:57.850+0000 7fdc87400700 1 -- 172.21.15.62:0/170244113 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdc780118c0 con 0x7fdc90100e80 2024-04-20T22:08:57.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:57.850+0000 7fdc87400700 1 -- 172.21.15.62:0/170244113 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdc78010380 con 0x7fdc90100e80 2024-04-20T22:08:57.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:57.850+0000 7fdc95200700 1 -- 172.21.15.62:0/170244113 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc9019cef0 con 0x7fdc90100e80 2024-04-20T22:08:57.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:57.850+0000 7fdc87400700 1 -- 172.21.15.62:0/170244113 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdc7800f4f0 con 0x7fdc90100e80 2024-04-20T22:08:57.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:57.850+0000 7fdc95200700 1 -- 172.21.15.62:0/170244113 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc9019d440 con 0x7fdc90100e80 2024-04-20T22:08:57.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:57.851+0000 7fdc95200700 1 -- 172.21.15.62:0/170244113 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc901045d0 con 0x7fdc90100e80 2024-04-20T22:08:57.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:57.852+0000 7fdc87400700 1 -- 172.21.15.62:0/170244113 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fdc780104f0 con 0x7fdc90100e80 2024-04-20T22:08:57.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:57.854+0000 7fdc87400700 1 --2- 172.21.15.62:0/170244113 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fdc7406ecc0 0x7fdc74071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:08:57.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:57.854+0000 7fdc87400700 1 -- 172.21.15.62:0/170244113 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fdc7808ea70 con 0x7fdc90100e80 2024-04-20T22:08:57.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:57.854+0000 7fdc8de00700 1 --2- 172.21.15.62:0/170244113 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fdc7406ecc0 0x7fdc74071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:08:57.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:57.855+0000 7fdc8de00700 1 --2- 172.21.15.62:0/170244113 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fdc7406ecc0 0x7fdc74071170 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fdc80009480 tx=0x7fdc8000b7f0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:08:57.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:57.859+0000 7fdc87400700 1 -- 172.21.15.62:0/170244113 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fdc78057800 con 0x7fdc90100e80 2024-04-20T22:08:58.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:08:57 smithi062 ceph-mon[76322]: from='client.35796 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:08:58.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:08:57 smithi062 ceph-mon[76322]: from='client.35800 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:08:58.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:58.186+0000 7fdc95200700 1 -- 172.21.15.62:0/170244113 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fdc9019d710 con 0x7fdc90100e80 2024-04-20T22:08:58.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:58.188+0000 7fdc87400700 1 -- 172.21.15.62:0/170244113 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7fdc7805aec0 con 0x7fdc90100e80 2024-04-20T22:08:58.187 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:08:58.188 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T22:08:58.188 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:08:58.188 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:08:58.188 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T22:08:58.188 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:08:58.188 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:08:58.188 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T22:08:58.188 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 6 2024-04-20T22:08:58.188 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:08:58.188 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T22:08:58.188 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 1, 2024-04-20T22:08:58.188 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 3 2024-04-20T22:08:58.188 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:08:58.188 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T22:08:58.188 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 1, 2024-04-20T22:08:58.189 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 13 2024-04-20T22:08:58.189 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T22:08:58.189 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:08:58.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:58.191+0000 7fdc95200700 1 -- 172.21.15.62:0/170244113 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fdc7406ecc0 msgr2=0x7fdc74071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:08:58.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:58.191+0000 7fdc95200700 1 --2- 172.21.15.62:0/170244113 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fdc7406ecc0 0x7fdc74071170 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fdc80009480 tx=0x7fdc8000b7f0 comp rx=0 tx=0).stop 2024-04-20T22:08:58.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:58.191+0000 7fdc95200700 1 -- 172.21.15.62:0/170244113 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc90100e80 msgr2=0x7fdc901987a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:08:58.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:58.191+0000 7fdc95200700 1 --2- 172.21.15.62:0/170244113 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc90100e80 0x7fdc901987a0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7fdc7800b480 tx=0x7fdc7800b840 comp rx=0 tx=0).stop 2024-04-20T22:08:58.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:58.191+0000 7fdc95200700 1 -- 172.21.15.62:0/170244113 shutdown_connections 2024-04-20T22:08:58.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:58.191+0000 7fdc95200700 1 --2- 172.21.15.62:0/170244113 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fdc7406ecc0 0x7fdc74071170 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:08:58.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:58.191+0000 7fdc95200700 1 --2- 172.21.15.62:0/170244113 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc901004e0 0x7fdc90198260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:08:58.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:58.191+0000 7fdc95200700 1 --2- 172.21.15.62:0/170244113 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc90100e80 0x7fdc901987a0 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:08:58.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:58.191+0000 7fdc95200700 1 -- 172.21.15.62:0/170244113 >> 172.21.15.62:0/170244113 conn(0x7fdc900fbda0 msgr2=0x7fdc900fd480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:08:58.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:58.191+0000 7fdc95200700 1 -- 172.21.15.62:0/170244113 shutdown_connections 2024-04-20T22:08:58.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:58.191+0000 7fdc95200700 1 -- 172.21.15.62:0/170244113 wait complete. 2024-04-20T22:08:58.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:08:57 smithi189 ceph-mon[57022]: from='client.35796 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:08:58.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:08:57 smithi189 ceph-mon[57022]: from='client.35800 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:08:58.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:58.352+0000 7fba64400700 1 -- 172.21.15.62:0/738984233 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba5c106b80 msgr2=0x7fba5c106ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:08:58.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:58.352+0000 7fba64400700 1 --2- 172.21.15.62:0/738984233 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba5c106b80 0x7fba5c106ff0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7fba48009910 tx=0x7fba48009c20 comp rx=0 tx=0).stop 2024-04-20T22:08:58.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:58.352+0000 7fba64400700 1 -- 172.21.15.62:0/738984233 shutdown_connections 2024-04-20T22:08:58.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:58.352+0000 7fba64400700 1 --2- 172.21.15.62:0/738984233 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba5c106b80 0x7fba5c106ff0 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:08:58.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:58.352+0000 7fba64400700 1 --2- 172.21.15.62:0/738984233 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba5c10c8f0 0x7fba5c10ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:08:58.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:58.352+0000 7fba64400700 1 -- 172.21.15.62:0/738984233 >> 172.21.15.62:0/738984233 conn(0x7fba5c076060 msgr2=0x7fba5c078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:08:58.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:58.352+0000 7fba64400700 1 -- 172.21.15.62:0/738984233 shutdown_connections 2024-04-20T22:08:58.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:58.353+0000 7fba64400700 1 -- 172.21.15.62:0/738984233 wait complete. 2024-04-20T22:08:58.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:58.354+0000 7fba64400700 1 Processor -- start 2024-04-20T22:08:58.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:58.354+0000 7fba64400700 1 -- start start 2024-04-20T22:08:58.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:58.354+0000 7fba64400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba5c106b80 0x7fba5c19c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:08:58.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:58.354+0000 7fba64400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba5c10c8f0 0x7fba5c19c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:08:58.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:58.354+0000 7fba64400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba5c19cfd0 con 0x7fba5c10c8f0 2024-04-20T22:08:58.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:58.354+0000 7fba64400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba5c19d110 con 0x7fba5c106b80 2024-04-20T22:08:58.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:58.355+0000 7fba61000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba5c10c8f0 0x7fba5c19c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:08:58.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:58.355+0000 7fba61000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba5c10c8f0 0x7fba5c19c980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56348/0 (socket says 172.21.15.62:56348) 2024-04-20T22:08:58.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:58.355+0000 7fba61000700 1 -- 172.21.15.62:0/4068493669 learned_addr learned my addr 172.21.15.62:0/4068493669 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:08:58.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:58.355+0000 7fba61a00700 1 --2- 172.21.15.62:0/4068493669 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba5c106b80 0x7fba5c19c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:08:58.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:58.355+0000 7fba61000700 1 -- 172.21.15.62:0/4068493669 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba5c106b80 msgr2=0x7fba5c19c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:08:58.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:58.355+0000 7fba61000700 1 --2- 172.21.15.62:0/4068493669 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba5c106b80 0x7fba5c19c440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:08:58.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:58.355+0000 7fba61000700 1 -- 172.21.15.62:0/4068493669 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba480095f0 con 0x7fba5c10c8f0 2024-04-20T22:08:58.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:58.356+0000 7fba61000700 1 --2- 172.21.15.62:0/4068493669 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba5c10c8f0 0x7fba5c19c980 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7fba4800b5c0 tx=0x7fba4800be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:08:58.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:58.356+0000 7fba5a000700 1 -- 172.21.15.62:0/4068493669 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fba4801d070 con 0x7fba5c10c8f0 2024-04-20T22:08:58.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:58.356+0000 7fba5a000700 1 -- 172.21.15.62:0/4068493669 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fba4800f460 con 0x7fba5c10c8f0 2024-04-20T22:08:58.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:58.356+0000 7fba5a000700 1 -- 172.21.15.62:0/4068493669 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fba48017610 con 0x7fba5c10c8f0 2024-04-20T22:08:58.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:58.356+0000 7fba64400700 1 -- 172.21.15.62:0/4068493669 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba5c1a0f00 con 0x7fba5c10c8f0 2024-04-20T22:08:58.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:58.356+0000 7fba64400700 1 -- 172.21.15.62:0/4068493669 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba5c1a13f0 con 0x7fba5c10c8f0 2024-04-20T22:08:58.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:58.358+0000 7fba64400700 1 -- 172.21.15.62:0/4068493669 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba5c10ecc0 con 0x7fba5c10c8f0 2024-04-20T22:08:58.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:58.358+0000 7fba5a000700 1 -- 172.21.15.62:0/4068493669 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fba4800f5d0 con 0x7fba5c10c8f0 2024-04-20T22:08:58.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:58.359+0000 7fba5a000700 1 --2- 172.21.15.62:0/4068493669 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fba4406ecc0 0x7fba44071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:08:58.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:58.360+0000 7fba5a000700 1 -- 172.21.15.62:0/4068493669 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fba48090440 con 0x7fba5c10c8f0 2024-04-20T22:08:58.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:58.360+0000 7fba61a00700 1 --2- 172.21.15.62:0/4068493669 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fba4406ecc0 0x7fba44071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:08:58.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:58.361+0000 7fba61a00700 1 --2- 172.21.15.62:0/4068493669 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fba4406ecc0 0x7fba44071170 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fba50005ea0 tx=0x7fba50005e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:08:58.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:58.364+0000 7fba5a000700 1 -- 172.21.15.62:0/4068493669 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fba480591d0 con 0x7fba5c10c8f0 2024-04-20T22:08:58.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:58.663+0000 7fba64400700 1 -- 172.21.15.62:0/4068493669 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fba5c19d840 con 0x7fba5c10c8f0 2024-04-20T22:08:58.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:58.665+0000 7fba5a000700 1 -- 172.21.15.62:0/4068493669 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 27 v27) v1 ==== 76+0+1888 (secure 0 0 0) 0x7fba4805c890 con 0x7fba5c10c8f0 2024-04-20T22:08:58.666 INFO:teuthology.orchestra.run.smithi062.stdout:e27 2024-04-20T22:08:58.666 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T22:08:58.667 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:08:58.667 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T22:08:58.667 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:08:58.667 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T22:08:58.667 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T22:08:58.667 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 20 2024-04-20T22:08:58.667 INFO:teuthology.orchestra.run.smithi062.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-20T22:08:58.667 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T22:08:58.667 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T21:42:01.559061+0000 2024-04-20T22:08:58.667 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T22:08:58.667 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T22:08:58.667 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T22:08:58.667 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T22:08:58.668 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T22:08:58.668 INFO:teuthology.orchestra.run.smithi062.stdout:max_xattr_size 65536 2024-04-20T22:08:58.668 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T22:08:58.668 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T22:08:58.668 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 81 2024-04-20T22:08:58.668 INFO:teuthology.orchestra.run.smithi062.stdout:snap_rstat 1 2024-04-20T22:08:58.668 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:08:58.668 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 2 2024-04-20T22:08:58.668 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1 2024-04-20T22:08:58.668 INFO:teuthology.orchestra.run.smithi062.stdout:up {} 2024-04-20T22:08:58.668 INFO:teuthology.orchestra.run.smithi062.stdout:failed 0,1 2024-04-20T22:08:58.668 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T22:08:58.668 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T22:08:58.668 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T22:08:58.669 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T22:08:58.669 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data enabled 2024-04-20T22:08:58.669 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T22:08:58.669 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T22:08:58.669 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 1 2024-04-20T22:08:58.669 INFO:teuthology.orchestra.run.smithi062.stdout:qdb_cluster leader: 0 members: 2024-04-20T22:08:58.669 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:08:58.669 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:08:58.669 INFO:teuthology.orchestra.run.smithi062.stdout:Standby daemons: 2024-04-20T22:08:58.669 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:08:58.669 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ryqblb{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6826/4113916205,v1:172.21.15.62:6827/4113916205] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:08:58.669 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ozizwi{-1:34476} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6828/3801975165,v1:172.21.15.62:6829/3801975165] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:08:58.669 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.xbgspw{-1:44285} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6826/3996040526,v1:172.21.15.189:6827/3996040526] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T22:08:58.669 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.dvimer{-1:44727} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6824/3576699232,v1:172.21.15.189:6825/3576699232] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:08:58.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:58.668+0000 7fba64400700 1 -- 172.21.15.62:0/4068493669 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fba4406ecc0 msgr2=0x7fba44071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:08:58.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:58.668+0000 7fba64400700 1 --2- 172.21.15.62:0/4068493669 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fba4406ecc0 0x7fba44071170 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fba50005ea0 tx=0x7fba50005e30 comp rx=0 tx=0).stop 2024-04-20T22:08:58.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:58.668+0000 7fba64400700 1 -- 172.21.15.62:0/4068493669 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba5c10c8f0 msgr2=0x7fba5c19c980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:08:58.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:58.668+0000 7fba64400700 1 --2- 172.21.15.62:0/4068493669 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba5c10c8f0 0x7fba5c19c980 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7fba4800b5c0 tx=0x7fba4800be00 comp rx=0 tx=0).stop 2024-04-20T22:08:58.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:58.668+0000 7fba64400700 1 -- 172.21.15.62:0/4068493669 shutdown_connections 2024-04-20T22:08:58.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:58.668+0000 7fba64400700 1 --2- 172.21.15.62:0/4068493669 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fba4406ecc0 0x7fba44071170 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:08:58.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:58.668+0000 7fba64400700 1 --2- 172.21.15.62:0/4068493669 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba5c106b80 0x7fba5c19c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:08:58.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:58.668+0000 7fba64400700 1 --2- 172.21.15.62:0/4068493669 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba5c10c8f0 0x7fba5c19c980 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:08:58.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:58.669+0000 7fba64400700 1 -- 172.21.15.62:0/4068493669 >> 172.21.15.62:0/4068493669 conn(0x7fba5c076060 msgr2=0x7fba5c077ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:08:58.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:58.669+0000 7fba64400700 1 -- 172.21.15.62:0/4068493669 shutdown_connections 2024-04-20T22:08:58.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:58.669+0000 7fba64400700 1 -- 172.21.15.62:0/4068493669 wait complete. 2024-04-20T22:08:58.671 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 27 2024-04-20T22:08:58.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:58.838+0000 7f30c7200700 1 -- 172.21.15.62:0/4153622336 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f30c8069140 msgr2=0x7f30c8108090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:08:58.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:58.838+0000 7f30c7200700 1 --2- 172.21.15.62:0/4153622336 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f30c8069140 0x7f30c8108090 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f30b4009910 tx=0x7f30b4009c20 comp rx=0 tx=0).stop 2024-04-20T22:08:58.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:58.838+0000 7f30c7200700 1 -- 172.21.15.62:0/4153622336 shutdown_connections 2024-04-20T22:08:58.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:58.838+0000 7f30c7200700 1 --2- 172.21.15.62:0/4153622336 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f30c8069140 0x7f30c8108090 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:08:58.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:58.838+0000 7f30c7200700 1 --2- 172.21.15.62:0/4153622336 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30c80687a0 0x7f30c8068b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:08:58.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:58.838+0000 7f30c7200700 1 -- 172.21.15.62:0/4153622336 >> 172.21.15.62:0/4153622336 conn(0x7f30c8074fa0 msgr2=0x7f30c80753a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:08:58.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:58.839+0000 7f30c7200700 1 -- 172.21.15.62:0/4153622336 shutdown_connections 2024-04-20T22:08:58.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:58.839+0000 7f30c7200700 1 -- 172.21.15.62:0/4153622336 wait complete. 2024-04-20T22:08:58.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:58.840+0000 7f30c7200700 1 Processor -- start 2024-04-20T22:08:58.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:58.840+0000 7f30c7200700 1 -- start start 2024-04-20T22:08:58.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:58.840+0000 7f30c7200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f30c80687a0 0x7f30c8197e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:08:58.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:58.840+0000 7f30c7200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30c8069140 0x7f30c8198370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:08:58.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:58.840+0000 7f30c7200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30c8198a50 con 0x7f30c80687a0 2024-04-20T22:08:58.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:58.840+0000 7f30c7200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30c819c7e0 con 0x7f30c8069140 2024-04-20T22:08:58.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:58.841+0000 7f30c5e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f30c80687a0 0x7f30c8197e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:08:58.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:58.841+0000 7f30c5e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f30c80687a0 0x7f30c8197e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56352/0 (socket says 172.21.15.62:56352) 2024-04-20T22:08:58.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:58.841+0000 7f30c5e00700 1 -- 172.21.15.62:0/2535167388 learned_addr learned my addr 172.21.15.62:0/2535167388 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:08:58.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:58.841+0000 7f30c5400700 1 --2- 172.21.15.62:0/2535167388 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30c8069140 0x7f30c8198370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:08:58.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:58.841+0000 7f30c5e00700 1 -- 172.21.15.62:0/2535167388 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30c8069140 msgr2=0x7f30c8198370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:08:58.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:58.841+0000 7f30c5e00700 1 --2- 172.21.15.62:0/2535167388 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30c8069140 0x7f30c8198370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:08:58.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:58.841+0000 7f30c5e00700 1 -- 172.21.15.62:0/2535167388 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30b40095f0 con 0x7f30c80687a0 2024-04-20T22:08:58.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:58.842+0000 7f30c5e00700 1 --2- 172.21.15.62:0/2535167388 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f30c80687a0 0x7f30c8197e30 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f30b800c8f0 tx=0x7f30b800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:08:58.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:58.842+0000 7f30bea00700 1 -- 172.21.15.62:0/2535167388 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f30b8007a70 con 0x7f30c80687a0 2024-04-20T22:08:58.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:58.842+0000 7f30bea00700 1 -- 172.21.15.62:0/2535167388 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f30b8007bd0 con 0x7f30c80687a0 2024-04-20T22:08:58.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:58.842+0000 7f30c7200700 1 -- 172.21.15.62:0/2535167388 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30c819cac0 con 0x7f30c80687a0 2024-04-20T22:08:58.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:58.842+0000 7f30bea00700 1 -- 172.21.15.62:0/2535167388 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f30b800f680 con 0x7f30c80687a0 2024-04-20T22:08:58.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:58.842+0000 7f30c7200700 1 -- 172.21.15.62:0/2535167388 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30c819d010 con 0x7f30c80687a0 2024-04-20T22:08:58.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:58.843+0000 7f30c7200700 1 -- 172.21.15.62:0/2535167388 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30c810a810 con 0x7f30c80687a0 2024-04-20T22:08:58.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:58.844+0000 7f30bea00700 1 -- 172.21.15.62:0/2535167388 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f30b8007d40 con 0x7f30c80687a0 2024-04-20T22:08:58.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:58.845+0000 7f30bea00700 1 --2- 172.21.15.62:0/2535167388 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f30ac06ecc0 0x7f30ac071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:08:58.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:58.846+0000 7f30bea00700 1 -- 172.21.15.62:0/2535167388 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f30b8013070 con 0x7f30c80687a0 2024-04-20T22:08:58.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:58.846+0000 7f30c5400700 1 --2- 172.21.15.62:0/2535167388 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f30ac06ecc0 0x7f30ac071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:08:58.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:58.847+0000 7f30c5400700 1 --2- 172.21.15.62:0/2535167388 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f30ac06ecc0 0x7f30ac071170 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f30b400b5c0 tx=0x7f30b4005fd0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:08:58.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:58.851+0000 7f30bea00700 1 -- 172.21.15.62:0/2535167388 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f30b8057e90 con 0x7f30c80687a0 2024-04-20T22:08:59.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:08:58 smithi062 ceph-mon[76322]: from='client.35804 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:08:59.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:08:58 smithi062 ceph-mon[76322]: pgmap v1011: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:08:59.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:08:58 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/170244113' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:08:59.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:08:58 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/4068493669' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:08:59.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:59.112+0000 7f30c7200700 1 -- 172.21.15.62:0/2535167388 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f30c819d300 con 0x7f30ac06ecc0 2024-04-20T22:08:59.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:59.114+0000 7f30bea00700 1 -- 172.21.15.62:0/2535167388 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f30c819d300 con 0x7f30ac06ecc0 2024-04-20T22:08:59.114 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:08:59.114 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T22:08:59.114 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T22:08:59.114 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-20T22:08:59.114 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T22:08:59.114 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr", 2024-04-20T22:08:59.114 INFO:teuthology.orchestra.run.smithi062.stdout: "crash", 2024-04-20T22:08:59.114 INFO:teuthology.orchestra.run.smithi062.stdout: "mon", 2024-04-20T22:08:59.114 INFO:teuthology.orchestra.run.smithi062.stdout: "osd" 2024-04-20T22:08:59.115 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T22:08:59.115 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "15/23 daemons upgraded", 2024-04-20T22:08:59.115 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading mds daemons", 2024-04-20T22:08:59.115 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T22:08:59.115 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:08:59.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:59.117+0000 7f30c7200700 1 -- 172.21.15.62:0/2535167388 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f30ac06ecc0 msgr2=0x7f30ac071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:08:59.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:59.117+0000 7f30c7200700 1 --2- 172.21.15.62:0/2535167388 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f30ac06ecc0 0x7f30ac071170 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f30b400b5c0 tx=0x7f30b4005fd0 comp rx=0 tx=0).stop 2024-04-20T22:08:59.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:59.117+0000 7f30c7200700 1 -- 172.21.15.62:0/2535167388 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f30c80687a0 msgr2=0x7f30c8197e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:08:59.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:59.117+0000 7f30c7200700 1 --2- 172.21.15.62:0/2535167388 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f30c80687a0 0x7f30c8197e30 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f30b800c8f0 tx=0x7f30b800ccb0 comp rx=0 tx=0).stop 2024-04-20T22:08:59.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:59.117+0000 7f30c7200700 1 -- 172.21.15.62:0/2535167388 shutdown_connections 2024-04-20T22:08:59.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:59.117+0000 7f30c7200700 1 --2- 172.21.15.62:0/2535167388 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f30ac06ecc0 0x7f30ac071170 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:08:59.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:59.117+0000 7f30c7200700 1 --2- 172.21.15.62:0/2535167388 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f30c80687a0 0x7f30c8197e30 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:08:59.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:59.117+0000 7f30c7200700 1 --2- 172.21.15.62:0/2535167388 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30c8069140 0x7f30c8198370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:08:59.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:59.117+0000 7f30c7200700 1 -- 172.21.15.62:0/2535167388 >> 172.21.15.62:0/2535167388 conn(0x7f30c8074fa0 msgr2=0x7f30c8106450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:08:59.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:59.118+0000 7f30c7200700 1 -- 172.21.15.62:0/2535167388 shutdown_connections 2024-04-20T22:08:59.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:59.118+0000 7f30c7200700 1 -- 172.21.15.62:0/2535167388 wait complete. 2024-04-20T22:08:59.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:08:58 smithi189 ceph-mon[57022]: from='client.35804 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:08:59.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:08:58 smithi189 ceph-mon[57022]: pgmap v1011: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:08:59.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:08:58 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/170244113' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:08:59.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:08:58 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/4068493669' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:08:59.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:59.271+0000 7effe9600700 1 -- 172.21.15.62:0/35996121 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7effe41004e0 msgr2=0x7effe41008b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:08:59.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:59.271+0000 7effe9600700 1 --2- 172.21.15.62:0/35996121 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7effe41004e0 0x7effe41008b0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7effd4009910 tx=0x7effd4009c20 comp rx=0 tx=0).stop 2024-04-20T22:08:59.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:59.271+0000 7effe9600700 1 -- 172.21.15.62:0/35996121 shutdown_connections 2024-04-20T22:08:59.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:59.271+0000 7effe9600700 1 --2- 172.21.15.62:0/35996121 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7effe4100e80 0x7effe4104e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:08:59.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:59.271+0000 7effe9600700 1 --2- 172.21.15.62:0/35996121 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7effe41004e0 0x7effe41008b0 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:08:59.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:59.271+0000 7effe9600700 1 -- 172.21.15.62:0/35996121 >> 172.21.15.62:0/35996121 conn(0x7effe40fbda0 msgr2=0x7effe40fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:08:59.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:59.271+0000 7effe9600700 1 -- 172.21.15.62:0/35996121 shutdown_connections 2024-04-20T22:08:59.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:59.271+0000 7effe9600700 1 -- 172.21.15.62:0/35996121 wait complete. 2024-04-20T22:08:59.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:59.272+0000 7effe9600700 1 Processor -- start 2024-04-20T22:08:59.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:59.272+0000 7effe9600700 1 -- start start 2024-04-20T22:08:59.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:59.273+0000 7effe9600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7effe41004e0 0x7effe41980f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:08:59.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:59.273+0000 7effe9600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7effe4100e80 0x7effe4198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:08:59.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:59.273+0000 7effe9600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effe4198d10 con 0x7effe41004e0 2024-04-20T22:08:59.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:59.273+0000 7effe9600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effe419caa0 con 0x7effe4100e80 2024-04-20T22:08:59.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:59.273+0000 7effe2a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7effe41004e0 0x7effe41980f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:08:59.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:59.273+0000 7effe2a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7effe41004e0 0x7effe41980f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56376/0 (socket says 172.21.15.62:56376) 2024-04-20T22:08:59.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:59.273+0000 7effe2a00700 1 -- 172.21.15.62:0/1778008646 learned_addr learned my addr 172.21.15.62:0/1778008646 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:08:59.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:59.273+0000 7effe2a00700 1 -- 172.21.15.62:0/1778008646 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7effe4100e80 msgr2=0x7effe4198630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:08:59.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:59.273+0000 7effe2a00700 1 --2- 172.21.15.62:0/1778008646 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7effe4100e80 0x7effe4198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:08:59.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:59.273+0000 7effe2a00700 1 -- 172.21.15.62:0/1778008646 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effd40095f0 con 0x7effe41004e0 2024-04-20T22:08:59.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:59.274+0000 7effe2a00700 1 --2- 172.21.15.62:0/1778008646 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7effe41004e0 0x7effe41980f0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7effd4006010 tx=0x7effd400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:08:59.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:59.274+0000 7effdb400700 1 -- 172.21.15.62:0/1778008646 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7effd401d070 con 0x7effe41004e0 2024-04-20T22:08:59.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:59.274+0000 7effdb400700 1 -- 172.21.15.62:0/1778008646 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7effd400f460 con 0x7effe41004e0 2024-04-20T22:08:59.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:59.274+0000 7effe9600700 1 -- 172.21.15.62:0/1778008646 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effe419cd20 con 0x7effe41004e0 2024-04-20T22:08:59.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:59.274+0000 7effdb400700 1 -- 172.21.15.62:0/1778008646 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7effd4017630 con 0x7effe41004e0 2024-04-20T22:08:59.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:59.274+0000 7effe9600700 1 -- 172.21.15.62:0/1778008646 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effe419d210 con 0x7effe41004e0 2024-04-20T22:08:59.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:59.275+0000 7effe9600700 1 -- 172.21.15.62:0/1778008646 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effe41045d0 con 0x7effe41004e0 2024-04-20T22:08:59.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:59.277+0000 7effdb400700 1 -- 172.21.15.62:0/1778008646 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7effd400f5d0 con 0x7effe41004e0 2024-04-20T22:08:59.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:59.277+0000 7effdb400700 1 --2- 172.21.15.62:0/1778008646 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7effc806ecc0 0x7effc8071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:08:59.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:59.278+0000 7effdb400700 1 -- 172.21.15.62:0/1778008646 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7effd4090330 con 0x7effe41004e0 2024-04-20T22:08:59.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:59.278+0000 7effe2000700 1 --2- 172.21.15.62:0/1778008646 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7effc806ecc0 0x7effc8071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:08:59.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:59.278+0000 7effe2000700 1 --2- 172.21.15.62:0/1778008646 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7effc806ecc0 0x7effc8071170 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7effd0005950 tx=0x7effd00058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:08:59.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:59.282+0000 7effdb400700 1 -- 172.21.15.62:0/1778008646 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7effd40590c0 con 0x7effe41004e0 2024-04-20T22:08:59.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:59.609+0000 7effe9600700 1 -- 172.21.15.62:0/1778008646 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7effe404f2a0 con 0x7effe41004e0 2024-04-20T22:08:59.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:59.610+0000 7effdb400700 1 -- 172.21.15.62:0/1778008646 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7effd405c780 con 0x7effe41004e0 2024-04-20T22:08:59.610 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-20T22:08:59.610 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-20T22:08:59.611 INFO:teuthology.orchestra.run.smithi062.stdout: host smithi062 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T22:08:59.611 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-20T22:08:59.611 INFO:teuthology.orchestra.run.smithi062.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi062 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpq7y7oa_r:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:08:59.611 INFO:teuthology.orchestra.run.smithi062.stdout:Traceback (most recent call last): 2024-04-20T22:08:59.611 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-20T22:08:59.611 INFO:teuthology.orchestra.run.smithi062.stdout: return _run_code(code, main_globals, None, 2024-04-20T22:08:59.611 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-20T22:08:59.611 INFO:teuthology.orchestra.run.smithi062.stdout: exec(code, run_globals) 2024-04-20T22:08:59.611 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5618, in 2024-04-20T22:08:59.611 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5606, in main 2024-04-20T22:08:59.611 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 408, in _infer_config 2024-04-20T22:08:59.611 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 323, in _infer_fsid 2024-04-20T22:08:59.611 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 436, in _infer_image 2024-04-20T22:08:59.611 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 310, in _validate_fsid 2024-04-20T22:08:59.611 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 3359, in command_ceph_volume 2024-04-20T22:08:59.612 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-20T22:08:59.612 INFO:teuthology.orchestra.run.smithi062.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi062 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpq7y7oa_r:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:08:59.612 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-20T22:08:59.612 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is degraded 2024-04-20T22:08:59.612 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T22:08:59.612 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T22:08:59.612 INFO:teuthology.orchestra.run.smithi062.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-20T22:08:59.612 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-20T22:08:59.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:59.613+0000 7effe9600700 1 -- 172.21.15.62:0/1778008646 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7effc806ecc0 msgr2=0x7effc8071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:08:59.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:59.613+0000 7effe9600700 1 --2- 172.21.15.62:0/1778008646 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7effc806ecc0 0x7effc8071170 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7effd0005950 tx=0x7effd00058e0 comp rx=0 tx=0).stop 2024-04-20T22:08:59.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:59.613+0000 7effe9600700 1 -- 172.21.15.62:0/1778008646 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7effe41004e0 msgr2=0x7effe41980f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:08:59.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:59.613+0000 7effe9600700 1 --2- 172.21.15.62:0/1778008646 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7effe41004e0 0x7effe41980f0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7effd4006010 tx=0x7effd400be20 comp rx=0 tx=0).stop 2024-04-20T22:08:59.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:59.613+0000 7effe9600700 1 -- 172.21.15.62:0/1778008646 shutdown_connections 2024-04-20T22:08:59.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:59.613+0000 7effe9600700 1 --2- 172.21.15.62:0/1778008646 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7effc806ecc0 0x7effc8071170 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:08:59.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:59.613+0000 7effe9600700 1 --2- 172.21.15.62:0/1778008646 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7effe41004e0 0x7effe41980f0 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:08:59.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:59.613+0000 7effe9600700 1 --2- 172.21.15.62:0/1778008646 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7effe4100e80 0x7effe4198630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:08:59.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:59.613+0000 7effe9600700 1 -- 172.21.15.62:0/1778008646 >> 172.21.15.62:0/1778008646 conn(0x7effe40fbda0 msgr2=0x7effe40fe160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:08:59.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:59.613+0000 7effe9600700 1 -- 172.21.15.62:0/1778008646 shutdown_connections 2024-04-20T22:08:59.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:08:59.613+0000 7effe9600700 1 -- 172.21.15.62:0/1778008646 wait complete. 2024-04-20T22:09:00.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:08:59 smithi062 ceph-mon[76322]: from='client.35816 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:09:00.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:08:59 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/1778008646' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:09:00.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:08:59 smithi189 ceph-mon[57022]: from='client.35816 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:09:00.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:08:59 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/1778008646' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:09:01.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:09:00 smithi062 ceph-mon[76322]: pgmap v1012: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:09:01.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:09:00 smithi189 ceph-mon[57022]: pgmap v1012: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:09:03.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:09:02 smithi189 ceph-mon[57022]: pgmap v1013: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:09:03.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:09:02 smithi062 ceph-mon[76322]: pgmap v1013: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:09:05.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:09:04 smithi189 ceph-mon[57022]: pgmap v1014: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:09:05.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:09:04 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:09:05.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:09:04 smithi062 ceph-mon[76322]: pgmap v1014: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:09:05.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:09:04 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:09:07.003 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:09:06 smithi189 ceph-mon[57022]: pgmap v1015: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:09:07.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:09:06 smithi062 ceph-mon[76322]: pgmap v1015: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:09:09.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:09:08 smithi189 ceph-mon[57022]: pgmap v1016: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:09:09.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:09:08 smithi062 ceph-mon[76322]: pgmap v1016: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:09:11.105 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:09:10 smithi189 ceph-mon[57022]: pgmap v1017: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:09:11.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:09:10 smithi062 ceph-mon[76322]: pgmap v1017: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:09:13.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:09:12 smithi189 ceph-mon[57022]: pgmap v1018: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:09:13.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:09:12 smithi062 ceph-mon[76322]: pgmap v1018: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:09:15.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:09:14 smithi189 ceph-mon[57022]: pgmap v1019: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:09:15.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:09:14 smithi062 ceph-mon[76322]: pgmap v1019: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:09:17.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:09:16 smithi189 ceph-mon[57022]: pgmap v1020: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:09:17.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:09:16 smithi062 ceph-mon[76322]: pgmap v1020: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:09:19.227 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:09:18 smithi062 ceph-mon[76322]: pgmap v1021: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:09:19.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:09:18 smithi189 ceph-mon[57022]: pgmap v1021: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:09:20.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:09:19 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:09:20.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:09:19 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:09:21.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:09:20 smithi189 ceph-mon[57022]: pgmap v1022: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:09:21.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:09:20 smithi062 ceph-mon[76322]: pgmap v1022: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:09:23.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:09:22 smithi189 ceph-mon[57022]: pgmap v1023: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:09:23.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:09:22 smithi062 ceph-mon[76322]: pgmap v1023: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:09:25.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:09:24 smithi189 ceph-mon[57022]: pgmap v1024: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:09:25.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:09:24 smithi062 ceph-mon[76322]: pgmap v1024: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:09:27.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:09:26 smithi189 ceph-mon[57022]: pgmap v1025: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:09:27.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:09:26 smithi062 ceph-mon[76322]: pgmap v1025: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:09:29.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:09:28 smithi189 ceph-mon[57022]: pgmap v1026: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:09:29.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:09:28 smithi062 ceph-mon[76322]: pgmap v1026: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:09:29.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:29.794+0000 7f4bf9a00700 1 -- 172.21.15.62:0/1782603030 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4bf4103040 msgr2=0x7f4bf4107010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:09:29.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:29.794+0000 7f4bf9a00700 1 --2- 172.21.15.62:0/1782603030 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4bf4103040 0x7f4bf4107010 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f4bdc009910 tx=0x7f4bdc009c20 comp rx=0 tx=0).stop 2024-04-20T22:09:29.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:29.794+0000 7f4bf9a00700 1 -- 172.21.15.62:0/1782603030 shutdown_connections 2024-04-20T22:09:29.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:29.794+0000 7f4bf9a00700 1 --2- 172.21.15.62:0/1782603030 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4bf4103040 0x7f4bf4107010 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:09:29.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:29.794+0000 7f4bf9a00700 1 --2- 172.21.15.62:0/1782603030 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4bf41026a0 0x7f4bf4102a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:09:29.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:29.794+0000 7f4bf9a00700 1 -- 172.21.15.62:0/1782603030 >> 172.21.15.62:0/1782603030 conn(0x7f4bf40fdf40 msgr2=0x7f4bf4100350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:09:29.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:29.794+0000 7f4bf9a00700 1 -- 172.21.15.62:0/1782603030 shutdown_connections 2024-04-20T22:09:29.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:29.794+0000 7f4bf9a00700 1 -- 172.21.15.62:0/1782603030 wait complete. 2024-04-20T22:09:29.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:29.795+0000 7f4bf9a00700 1 Processor -- start 2024-04-20T22:09:29.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:29.796+0000 7f4bf9a00700 1 -- start start 2024-04-20T22:09:29.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:29.796+0000 7f4bf9a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4bf41026a0 0x7f4bf4198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:09:29.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:29.796+0000 7f4bf9a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4bf4103040 0x7f4bf4198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:09:29.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:29.796+0000 7f4bf9a00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bf4198c70 con 0x7f4bf41026a0 2024-04-20T22:09:29.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:29.796+0000 7f4bf9a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bf419ca00 con 0x7f4bf4103040 2024-04-20T22:09:29.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:29.796+0000 7f4bf2a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4bf41026a0 0x7f4bf4198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:09:29.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:29.796+0000 7f4bf2a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4bf41026a0 0x7f4bf4198050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43386/0 (socket says 172.21.15.62:43386) 2024-04-20T22:09:29.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:29.796+0000 7f4bf2a00700 1 -- 172.21.15.62:0/1110966178 learned_addr learned my addr 172.21.15.62:0/1110966178 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:09:29.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:29.797+0000 7f4bebe00700 1 --2- 172.21.15.62:0/1110966178 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4bf4103040 0x7f4bf4198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:09:29.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:29.797+0000 7f4bf2a00700 1 -- 172.21.15.62:0/1110966178 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4bf4103040 msgr2=0x7f4bf4198590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:09:29.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:29.797+0000 7f4bf2a00700 1 --2- 172.21.15.62:0/1110966178 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4bf4103040 0x7f4bf4198590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:09:29.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:29.797+0000 7f4bf2a00700 1 -- 172.21.15.62:0/1110966178 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bdc0095f0 con 0x7f4bf41026a0 2024-04-20T22:09:29.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:29.797+0000 7f4bebe00700 1 --2- 172.21.15.62:0/1110966178 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4bf4103040 0x7f4bf4198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T22:09:29.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:29.797+0000 7f4bf2a00700 1 --2- 172.21.15.62:0/1110966178 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4bf41026a0 0x7f4bf4198050 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f4be400c8a0 tx=0x7f4be400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:09:29.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:29.798+0000 7f4beb400700 1 -- 172.21.15.62:0/1110966178 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4be400cea0 con 0x7f4bf41026a0 2024-04-20T22:09:29.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:29.798+0000 7f4beb400700 1 -- 172.21.15.62:0/1110966178 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4be4004830 con 0x7f4bf41026a0 2024-04-20T22:09:29.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:29.798+0000 7f4bf9a00700 1 -- 172.21.15.62:0/1110966178 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bf419cce0 con 0x7f4bf41026a0 2024-04-20T22:09:29.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:29.798+0000 7f4beb400700 1 -- 172.21.15.62:0/1110966178 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4be40055b0 con 0x7f4bf41026a0 2024-04-20T22:09:29.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:29.798+0000 7f4bf9a00700 1 -- 172.21.15.62:0/1110966178 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bf419d230 con 0x7f4bf41026a0 2024-04-20T22:09:29.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:29.799+0000 7f4bf9a00700 1 -- 172.21.15.62:0/1110966178 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bf410aa10 con 0x7f4bf41026a0 2024-04-20T22:09:29.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:29.800+0000 7f4beb400700 1 -- 172.21.15.62:0/1110966178 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f4be40049a0 con 0x7f4bf41026a0 2024-04-20T22:09:29.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:29.801+0000 7f4beb400700 1 --2- 172.21.15.62:0/1110966178 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f4bd806ecc0 0x7f4bd8071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:09:29.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:29.801+0000 7f4beb400700 1 -- 172.21.15.62:0/1110966178 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f4be408ebf0 con 0x7f4bf41026a0 2024-04-20T22:09:29.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:29.801+0000 7f4bebe00700 1 --2- 172.21.15.62:0/1110966178 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f4bd806ecc0 0x7f4bd8071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:09:29.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:29.802+0000 7f4bebe00700 1 --2- 172.21.15.62:0/1110966178 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f4bd806ecc0 0x7f4bd8071170 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f4bdc006010 tx=0x7f4bdc005960 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:09:29.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:29.806+0000 7f4beb400700 1 -- 172.21.15.62:0/1110966178 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f4be4057900 con 0x7f4bf41026a0 2024-04-20T22:09:30.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.066+0000 7f4bf9a00700 1 -- 172.21.15.62:0/1110966178 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4bf419d4b0 con 0x7f4bd806ecc0 2024-04-20T22:09:30.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.068+0000 7f4beb400700 1 -- 172.21.15.62:0/1110966178 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f4bf419d4b0 con 0x7f4bd806ecc0 2024-04-20T22:09:30.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.071+0000 7f4bf9a00700 1 -- 172.21.15.62:0/1110966178 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f4bd806ecc0 msgr2=0x7f4bd8071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:09:30.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.071+0000 7f4bf9a00700 1 --2- 172.21.15.62:0/1110966178 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f4bd806ecc0 0x7f4bd8071170 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f4bdc006010 tx=0x7f4bdc005960 comp rx=0 tx=0).stop 2024-04-20T22:09:30.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.071+0000 7f4bf9a00700 1 -- 172.21.15.62:0/1110966178 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4bf41026a0 msgr2=0x7f4bf4198050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:09:30.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.071+0000 7f4bf9a00700 1 --2- 172.21.15.62:0/1110966178 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4bf41026a0 0x7f4bf4198050 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f4be400c8a0 tx=0x7f4be400cc60 comp rx=0 tx=0).stop 2024-04-20T22:09:30.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.071+0000 7f4bf9a00700 1 -- 172.21.15.62:0/1110966178 shutdown_connections 2024-04-20T22:09:30.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.071+0000 7f4bf9a00700 1 --2- 172.21.15.62:0/1110966178 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f4bd806ecc0 0x7f4bd8071170 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:09:30.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.071+0000 7f4bf9a00700 1 --2- 172.21.15.62:0/1110966178 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4bf41026a0 0x7f4bf4198050 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:09:30.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.071+0000 7f4bf9a00700 1 --2- 172.21.15.62:0/1110966178 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4bf4103040 0x7f4bf4198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:09:30.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.071+0000 7f4bf9a00700 1 -- 172.21.15.62:0/1110966178 >> 172.21.15.62:0/1110966178 conn(0x7f4bf40fdf40 msgr2=0x7f4bf4100320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:09:30.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.072+0000 7f4bf9a00700 1 -- 172.21.15.62:0/1110966178 shutdown_connections 2024-04-20T22:09:30.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.072+0000 7f4bf9a00700 1 -- 172.21.15.62:0/1110966178 wait complete. 2024-04-20T22:09:30.088 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T22:09:30.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.234+0000 7f01d5a00700 1 -- 172.21.15.62:0/1498519388 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f01d00730f0 msgr2=0x7f01d00734c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:09:30.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.234+0000 7f01d5a00700 1 --2- 172.21.15.62:0/1498519388 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f01d00730f0 0x7f01d00734c0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f01c0009910 tx=0x7f01c0009c20 comp rx=0 tx=0).stop 2024-04-20T22:09:30.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.234+0000 7f01d5a00700 1 -- 172.21.15.62:0/1498519388 shutdown_connections 2024-04-20T22:09:30.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.234+0000 7f01d5a00700 1 --2- 172.21.15.62:0/1498519388 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f01d0073a00 0x7f01d010c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:09:30.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.234+0000 7f01d5a00700 1 --2- 172.21.15.62:0/1498519388 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f01d00730f0 0x7f01d00734c0 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:09:30.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.234+0000 7f01d5a00700 1 -- 172.21.15.62:0/1498519388 >> 172.21.15.62:0/1498519388 conn(0x7f01d00fbda0 msgr2=0x7f01d00fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:09:30.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.235+0000 7f01d5a00700 1 -- 172.21.15.62:0/1498519388 shutdown_connections 2024-04-20T22:09:30.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.235+0000 7f01d5a00700 1 -- 172.21.15.62:0/1498519388 wait complete. 2024-04-20T22:09:30.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.236+0000 7f01d5a00700 1 Processor -- start 2024-04-20T22:09:30.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.236+0000 7f01d5a00700 1 -- start start 2024-04-20T22:09:30.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.236+0000 7f01d5a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f01d00730f0 0x7f01d0197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:09:30.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.236+0000 7f01d5a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f01d0073a00 0x7f01d0198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:09:30.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.236+0000 7f01d5a00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01d019c970 con 0x7f01d0073a00 2024-04-20T22:09:30.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.236+0000 7f01d5a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01d019cae0 con 0x7f01d00730f0 2024-04-20T22:09:30.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.237+0000 7f01ce000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f01d0073a00 0x7f01d0198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:09:30.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.237+0000 7f01ce000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f01d0073a00 0x7f01d0198630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43400/0 (socket says 172.21.15.62:43400) 2024-04-20T22:09:30.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.237+0000 7f01ce000700 1 -- 172.21.15.62:0/921680210 learned_addr learned my addr 172.21.15.62:0/921680210 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:09:30.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.237+0000 7f01cea00700 1 --2- 172.21.15.62:0/921680210 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f01d00730f0 0x7f01d0197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:09:30.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.237+0000 7f01ce000700 1 -- 172.21.15.62:0/921680210 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f01d00730f0 msgr2=0x7f01d0197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:09:30.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.237+0000 7f01ce000700 1 --2- 172.21.15.62:0/921680210 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f01d00730f0 0x7f01d0197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:09:30.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.237+0000 7f01ce000700 1 -- 172.21.15.62:0/921680210 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01c00095f0 con 0x7f01d0073a00 2024-04-20T22:09:30.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.238+0000 7f01ce000700 1 --2- 172.21.15.62:0/921680210 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f01d0073a00 0x7f01d0198630 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f01bc00c8a0 tx=0x7f01bc00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:09:30.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.238+0000 7f01c7400700 1 -- 172.21.15.62:0/921680210 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f01bc00cea0 con 0x7f01d0073a00 2024-04-20T22:09:30.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.238+0000 7f01c7400700 1 -- 172.21.15.62:0/921680210 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f01bc004830 con 0x7f01d0073a00 2024-04-20T22:09:30.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.238+0000 7f01d5a00700 1 -- 172.21.15.62:0/921680210 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01d019cdc0 con 0x7f01d0073a00 2024-04-20T22:09:30.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.238+0000 7f01c7400700 1 -- 172.21.15.62:0/921680210 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f01bc0055b0 con 0x7f01d0073a00 2024-04-20T22:09:30.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.238+0000 7f01d5a00700 1 -- 172.21.15.62:0/921680210 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01d019d310 con 0x7f01d0073a00 2024-04-20T22:09:30.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.240+0000 7f01d5a00700 1 -- 172.21.15.62:0/921680210 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01d0109c80 con 0x7f01d0073a00 2024-04-20T22:09:30.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.240+0000 7f01c7400700 1 -- 172.21.15.62:0/921680210 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f01bc0049a0 con 0x7f01d0073a00 2024-04-20T22:09:30.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.241+0000 7f01c7400700 1 --2- 172.21.15.62:0/921680210 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f01b406ecc0 0x7f01b4071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:09:30.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.242+0000 7f01c7400700 1 -- 172.21.15.62:0/921680210 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f01bc08ebf0 con 0x7f01d0073a00 2024-04-20T22:09:30.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.242+0000 7f01cea00700 1 --2- 172.21.15.62:0/921680210 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f01b406ecc0 0x7f01b4071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:09:30.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.242+0000 7f01cea00700 1 --2- 172.21.15.62:0/921680210 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f01b406ecc0 0x7f01b4071170 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f01c0006010 tx=0x7f01c001a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:09:30.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.246+0000 7f01c7400700 1 -- 172.21.15.62:0/921680210 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f01bc057900 con 0x7f01d0073a00 2024-04-20T22:09:30.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.506+0000 7f01d5a00700 1 -- 172.21.15.62:0/921680210 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f01d0199390 con 0x7f01b406ecc0 2024-04-20T22:09:30.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.509+0000 7f01c7400700 1 -- 172.21.15.62:0/921680210 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f01d0199390 con 0x7f01b406ecc0 2024-04-20T22:09:30.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.513+0000 7f01d5a00700 1 -- 172.21.15.62:0/921680210 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f01b406ecc0 msgr2=0x7f01b4071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:09:30.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.513+0000 7f01d5a00700 1 --2- 172.21.15.62:0/921680210 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f01b406ecc0 0x7f01b4071170 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f01c0006010 tx=0x7f01c001a040 comp rx=0 tx=0).stop 2024-04-20T22:09:30.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.513+0000 7f01d5a00700 1 -- 172.21.15.62:0/921680210 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f01d0073a00 msgr2=0x7f01d0198630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:09:30.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.513+0000 7f01d5a00700 1 --2- 172.21.15.62:0/921680210 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f01d0073a00 0x7f01d0198630 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f01bc00c8a0 tx=0x7f01bc00cc60 comp rx=0 tx=0).stop 2024-04-20T22:09:30.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.513+0000 7f01d5a00700 1 -- 172.21.15.62:0/921680210 shutdown_connections 2024-04-20T22:09:30.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.513+0000 7f01d5a00700 1 --2- 172.21.15.62:0/921680210 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f01b406ecc0 0x7f01b4071170 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:09:30.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.513+0000 7f01d5a00700 1 --2- 172.21.15.62:0/921680210 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f01d00730f0 0x7f01d0197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:09:30.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.513+0000 7f01d5a00700 1 --2- 172.21.15.62:0/921680210 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f01d0073a00 0x7f01d0198630 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:09:30.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.513+0000 7f01d5a00700 1 -- 172.21.15.62:0/921680210 >> 172.21.15.62:0/921680210 conn(0x7f01d00fbda0 msgr2=0x7f01d0106dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:09:30.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.513+0000 7f01d5a00700 1 -- 172.21.15.62:0/921680210 shutdown_connections 2024-04-20T22:09:30.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.513+0000 7f01d5a00700 1 -- 172.21.15.62:0/921680210 wait complete. 2024-04-20T22:09:30.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.674+0000 7f1dd3000700 1 -- 172.21.15.62:0/4041569920 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1dcc068540 msgr2=0x7f1dcc0689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:09:30.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.674+0000 7f1dd3000700 1 --2- 172.21.15.62:0/4041569920 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1dcc068540 0x7f1dcc0689b0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f1db4009910 tx=0x7f1db4009c20 comp rx=0 tx=0).stop 2024-04-20T22:09:30.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.674+0000 7f1dd3000700 1 -- 172.21.15.62:0/4041569920 shutdown_connections 2024-04-20T22:09:30.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.674+0000 7f1dd3000700 1 --2- 172.21.15.62:0/4041569920 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1dcc068540 0x7f1dcc0689b0 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:09:30.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.675+0000 7f1dd3000700 1 --2- 172.21.15.62:0/4041569920 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1dcc101000 0x7f1dcc1013d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:09:30.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.675+0000 7f1dd3000700 1 -- 172.21.15.62:0/4041569920 >> 172.21.15.62:0/4041569920 conn(0x7f1dcc075130 msgr2=0x7f1dcc075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:09:30.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.675+0000 7f1dd3000700 1 -- 172.21.15.62:0/4041569920 shutdown_connections 2024-04-20T22:09:30.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.675+0000 7f1dd3000700 1 -- 172.21.15.62:0/4041569920 wait complete. 2024-04-20T22:09:30.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.676+0000 7f1dd3000700 1 Processor -- start 2024-04-20T22:09:30.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.676+0000 7f1dd3000700 1 -- start start 2024-04-20T22:09:30.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.676+0000 7f1dd3000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1dcc068540 0x7f1dcc198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:09:30.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.676+0000 7f1dd3000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1dcc101000 0x7f1dcc1985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:09:30.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.676+0000 7f1dd3000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dcc198c90 con 0x7f1dcc068540 2024-04-20T22:09:30.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.676+0000 7f1dd3000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dcc19ca20 con 0x7f1dcc101000 2024-04-20T22:09:30.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.677+0000 7f1dcbe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1dcc068540 0x7f1dcc198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:09:30.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.677+0000 7f1dcbe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1dcc068540 0x7f1dcc198070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43418/0 (socket says 172.21.15.62:43418) 2024-04-20T22:09:30.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.677+0000 7f1dcbe00700 1 -- 172.21.15.62:0/1207414026 learned_addr learned my addr 172.21.15.62:0/1207414026 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:09:30.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.677+0000 7f1dc3400700 1 --2- 172.21.15.62:0/1207414026 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1dcc101000 0x7f1dcc1985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:09:30.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.677+0000 7f1dcbe00700 1 -- 172.21.15.62:0/1207414026 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1dcc101000 msgr2=0x7f1dcc1985b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:09:30.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.677+0000 7f1dcbe00700 1 --2- 172.21.15.62:0/1207414026 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1dcc101000 0x7f1dcc1985b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:09:30.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.677+0000 7f1dcbe00700 1 -- 172.21.15.62:0/1207414026 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1db40095f0 con 0x7f1dcc068540 2024-04-20T22:09:30.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.678+0000 7f1dcbe00700 1 --2- 172.21.15.62:0/1207414026 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1dcc068540 0x7f1dcc198070 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f1dbc00ada0 tx=0x7f1dbc00c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:09:30.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.678+0000 7f1dc9600700 1 -- 172.21.15.62:0/1207414026 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1dbc00f070 con 0x7f1dcc068540 2024-04-20T22:09:30.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.678+0000 7f1dc9600700 1 -- 172.21.15.62:0/1207414026 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1dbc007bf0 con 0x7f1dcc068540 2024-04-20T22:09:30.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.678+0000 7f1dd3000700 1 -- 172.21.15.62:0/1207414026 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1dcc19cd00 con 0x7f1dcc068540 2024-04-20T22:09:30.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.678+0000 7f1dc9600700 1 -- 172.21.15.62:0/1207414026 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1dbc005690 con 0x7f1dcc068540 2024-04-20T22:09:30.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.678+0000 7f1dd3000700 1 -- 172.21.15.62:0/1207414026 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dcc19d250 con 0x7f1dcc068540 2024-04-20T22:09:30.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.679+0000 7f1dd3000700 1 -- 172.21.15.62:0/1207414026 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1dcc04f2a0 con 0x7f1dcc068540 2024-04-20T22:09:30.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.680+0000 7f1dc9600700 1 -- 172.21.15.62:0/1207414026 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f1dbc005850 con 0x7f1dcc068540 2024-04-20T22:09:30.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.681+0000 7f1dc9600700 1 --2- 172.21.15.62:0/1207414026 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f1db006ecc0 0x7f1db0071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:09:30.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.682+0000 7f1dc9600700 1 -- 172.21.15.62:0/1207414026 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f1dbc08e860 con 0x7f1dcc068540 2024-04-20T22:09:30.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.682+0000 7f1dc3400700 1 --2- 172.21.15.62:0/1207414026 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f1db006ecc0 0x7f1db0071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:09:30.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.683+0000 7f1dc3400700 1 --2- 172.21.15.62:0/1207414026 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f1db006ecc0 0x7f1db0071170 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f1db4005f50 tx=0x7f1db401a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:09:30.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.686+0000 7f1dc9600700 1 -- 172.21.15.62:0/1207414026 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f1dbc0575f0 con 0x7f1dcc068540 2024-04-20T22:09:30.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.933+0000 7f1dd3000700 1 -- 172.21.15.62:0/1207414026 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f1dcc19d5d0 con 0x7f1db006ecc0 2024-04-20T22:09:30.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.949+0000 7f1dc9600700 1 -- 172.21.15.62:0/1207414026 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f1dcc19d5d0 con 0x7f1db006ecc0 2024-04-20T22:09:30.949 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T22:09:30.949 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (35m) 11m ago 49m 15.3M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T22:09:30.949 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (49m) 11m ago 49m 10.4M - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T22:09:30.949 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (48m) 11m ago 48m 10.1M - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T22:09:30.950 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (31m) 11m ago 49m 6660k - 19.0.0-3133-ga8c84f59 76bc6b91b726 2825d5ba11c6 2024-04-20T22:09:30.950 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (31m) 11m ago 48m 6665k - 19.0.0-3133-ga8c84f59 76bc6b91b726 20f746fa1bba 2024-04-20T22:09:30.950 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (34m) 11m ago 48m 73.5M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T22:09:30.950 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (26m) 11m ago 41m 18.5M - 19.0.0-3133-ga8c84f59 76bc6b91b726 de80bc90a683 2024-04-20T22:09:30.950 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (27m) 11m ago 42m 18.7M - 19.0.0-3133-ga8c84f59 76bc6b91b726 cd793b7ee98e 2024-04-20T22:09:30.950 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (11m) 11m ago 42m 11.0M - 19.0.0-3133-ga8c84f59 76bc6b91b726 0e7b0bc4d1ab 2024-04-20T22:09:30.950 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (41m) 11m ago 41m 18.6M - 18.2.0 dc2bc1663786 f2eec2ee8951 2024-04-20T22:09:30.950 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (38m) 11m ago 51m 513M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T22:09:30.951 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (36m) 11m ago 47m 459M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T22:09:30.951 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (33m) 11m ago 51m 71.7M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 019eb1479485 2024-04-20T22:09:30.951 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (32m) 11m ago 47m 51.0M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 55aac560aab6 2024-04-20T22:09:30.952 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (36m) 11m ago 49m 8472k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T22:09:30.952 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (36m) 11m ago 47m 8048k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T22:09:30.952 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (30m) 11m ago 46m 91.2M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 ee2bb901f26e 2024-04-20T22:09:30.952 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (30m) 11m ago 46m 94.6M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 9075152d4162 2024-04-20T22:09:30.952 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (29m) 11m ago 45m 78.1M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 2d4f3e38dc0e 2024-04-20T22:09:30.952 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (29m) 11m ago 45m 92.6M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 8f97b17ae0eb 2024-04-20T22:09:30.952 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (28m) 11m ago 44m 95.6M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 653d8079967d 2024-04-20T22:09:30.952 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (27m) 11m ago 43m 89.2M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 596dded14c12 2024-04-20T22:09:30.952 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (35m) 11m ago 48m 82.1M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T22:09:30.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.952+0000 7f1dd3000700 1 -- 172.21.15.62:0/1207414026 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f1db006ecc0 msgr2=0x7f1db0071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:09:30.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.952+0000 7f1dd3000700 1 --2- 172.21.15.62:0/1207414026 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f1db006ecc0 0x7f1db0071170 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f1db4005f50 tx=0x7f1db401a040 comp rx=0 tx=0).stop 2024-04-20T22:09:30.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.952+0000 7f1dd3000700 1 -- 172.21.15.62:0/1207414026 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1dcc068540 msgr2=0x7f1dcc198070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:09:30.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.952+0000 7f1dd3000700 1 --2- 172.21.15.62:0/1207414026 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1dcc068540 0x7f1dcc198070 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f1dbc00ada0 tx=0x7f1dbc00c330 comp rx=0 tx=0).stop 2024-04-20T22:09:30.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.952+0000 7f1dd3000700 1 -- 172.21.15.62:0/1207414026 shutdown_connections 2024-04-20T22:09:30.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.952+0000 7f1dd3000700 1 --2- 172.21.15.62:0/1207414026 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f1db006ecc0 0x7f1db0071170 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:09:30.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.952+0000 7f1dd3000700 1 --2- 172.21.15.62:0/1207414026 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1dcc068540 0x7f1dcc198070 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:09:30.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.952+0000 7f1dd3000700 1 --2- 172.21.15.62:0/1207414026 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1dcc101000 0x7f1dcc1985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:09:30.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.952+0000 7f1dd3000700 1 -- 172.21.15.62:0/1207414026 >> 172.21.15.62:0/1207414026 conn(0x7f1dcc075130 msgr2=0x7f1dcc0fdb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:09:30.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.952+0000 7f1dd3000700 1 -- 172.21.15.62:0/1207414026 shutdown_connections 2024-04-20T22:09:30.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:30.952+0000 7f1dd3000700 1 -- 172.21.15.62:0/1207414026 wait complete. 2024-04-20T22:09:31.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:31.115+0000 7f6517e00700 1 -- 172.21.15.62:0/881246635 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f65181024d0 msgr2=0x7f65181028a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:09:31.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:31.115+0000 7f6517e00700 1 --2- 172.21.15.62:0/881246635 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f65181024d0 0x7f65181028a0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f6508009910 tx=0x7f6508009c20 comp rx=0 tx=0).stop 2024-04-20T22:09:31.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:31.115+0000 7f6517e00700 1 -- 172.21.15.62:0/881246635 shutdown_connections 2024-04-20T22:09:31.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:31.115+0000 7f6517e00700 1 --2- 172.21.15.62:0/881246635 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6518102e70 0x7f6518106e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:09:31.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:31.115+0000 7f6517e00700 1 --2- 172.21.15.62:0/881246635 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f65181024d0 0x7f65181028a0 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:09:31.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:31.115+0000 7f6517e00700 1 -- 172.21.15.62:0/881246635 >> 172.21.15.62:0/881246635 conn(0x7f65180fdd90 msgr2=0x7f65181001a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:09:31.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:31.115+0000 7f6517e00700 1 -- 172.21.15.62:0/881246635 shutdown_connections 2024-04-20T22:09:31.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:31.115+0000 7f6517e00700 1 -- 172.21.15.62:0/881246635 wait complete. 2024-04-20T22:09:31.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:31.116+0000 7f6517e00700 1 Processor -- start 2024-04-20T22:09:31.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:31.117+0000 7f6517e00700 1 -- start start 2024-04-20T22:09:31.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:31.117+0000 7f6517e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f65181024d0 0x7f6518197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:09:31.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:31.117+0000 7f6517e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6518102e70 0x7f6518198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:09:31.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:31.117+0000 7f6517e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6518198b50 con 0x7f65181024d0 2024-04-20T22:09:31.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:31.117+0000 7f6517e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f651819c8e0 con 0x7f6518102e70 2024-04-20T22:09:31.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:31.117+0000 7f6516a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f65181024d0 0x7f6518197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:09:31.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:31.117+0000 7f6516a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f65181024d0 0x7f6518197f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43430/0 (socket says 172.21.15.62:43430) 2024-04-20T22:09:31.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:31.117+0000 7f6516a00700 1 -- 172.21.15.62:0/2752528894 learned_addr learned my addr 172.21.15.62:0/2752528894 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:09:31.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:31.118+0000 7f6516000700 1 --2- 172.21.15.62:0/2752528894 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6518102e70 0x7f6518198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:09:31.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:31.118+0000 7f6516a00700 1 -- 172.21.15.62:0/2752528894 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6518102e70 msgr2=0x7f6518198470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:09:31.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:31.118+0000 7f6516a00700 1 --2- 172.21.15.62:0/2752528894 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6518102e70 0x7f6518198470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:09:31.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:31.118+0000 7f6516a00700 1 -- 172.21.15.62:0/2752528894 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65080095f0 con 0x7f65181024d0 2024-04-20T22:09:31.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:31.118+0000 7f6516a00700 1 --2- 172.21.15.62:0/2752528894 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f65181024d0 0x7f6518197f30 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f6508006010 tx=0x7f650800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:09:31.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:31.118+0000 7f650f400700 1 -- 172.21.15.62:0/2752528894 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f650801d070 con 0x7f65181024d0 2024-04-20T22:09:31.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:31.118+0000 7f650f400700 1 -- 172.21.15.62:0/2752528894 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6508004fb0 con 0x7f65181024d0 2024-04-20T22:09:31.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:31.118+0000 7f650f400700 1 -- 172.21.15.62:0/2752528894 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6508022ea0 con 0x7f65181024d0 2024-04-20T22:09:31.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:31.118+0000 7f6517e00700 1 -- 172.21.15.62:0/2752528894 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f651819cb60 con 0x7f65181024d0 2024-04-20T22:09:31.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:31.119+0000 7f6517e00700 1 -- 172.21.15.62:0/2752528894 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f651819d050 con 0x7f65181024d0 2024-04-20T22:09:31.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:31.120+0000 7f6517e00700 1 -- 172.21.15.62:0/2752528894 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f651810a7e0 con 0x7f65181024d0 2024-04-20T22:09:31.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:31.121+0000 7f650f400700 1 -- 172.21.15.62:0/2752528894 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f6508022470 con 0x7f65181024d0 2024-04-20T22:09:31.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:31.122+0000 7f650f400700 1 --2- 172.21.15.62:0/2752528894 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f64fc06ecc0 0x7f64fc071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:09:31.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:31.122+0000 7f650f400700 1 -- 172.21.15.62:0/2752528894 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f65080903c0 con 0x7f65181024d0 2024-04-20T22:09:31.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:31.122+0000 7f6516000700 1 --2- 172.21.15.62:0/2752528894 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f64fc06ecc0 0x7f64fc071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:09:31.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:31.123+0000 7f6516000700 1 --2- 172.21.15.62:0/2752528894 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f64fc06ecc0 0x7f64fc071170 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f6504006fd0 tx=0x7f6504008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:09:31.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:31.127+0000 7f650f400700 1 -- 172.21.15.62:0/2752528894 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f6508059200 con 0x7f65181024d0 2024-04-20T22:09:31.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:09:30 smithi189 ceph-mon[57022]: pgmap v1027: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:09:31.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:09:30 smithi189 ceph-mon[57022]: from='client.35824 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:09:31.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:09:30 smithi062 ceph-mon[76322]: pgmap v1027: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:09:31.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:09:30 smithi062 ceph-mon[76322]: from='client.35824 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:09:31.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:31.445+0000 7f6517e00700 1 -- 172.21.15.62:0/2752528894 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f6518062380 con 0x7f65181024d0 2024-04-20T22:09:31.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:31.446+0000 7f650f400700 1 -- 172.21.15.62:0/2752528894 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7f650805c8c0 con 0x7f65181024d0 2024-04-20T22:09:31.446 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:09:31.446 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T22:09:31.446 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:09:31.446 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:09:31.446 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T22:09:31.447 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:09:31.447 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:09:31.447 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T22:09:31.447 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 6 2024-04-20T22:09:31.447 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:09:31.447 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T22:09:31.447 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 1, 2024-04-20T22:09:31.447 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 3 2024-04-20T22:09:31.447 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:09:31.447 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T22:09:31.447 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 1, 2024-04-20T22:09:31.447 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 13 2024-04-20T22:09:31.447 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T22:09:31.447 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:09:31.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:31.449+0000 7f6517e00700 1 -- 172.21.15.62:0/2752528894 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f64fc06ecc0 msgr2=0x7f64fc071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:09:31.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:31.449+0000 7f6517e00700 1 --2- 172.21.15.62:0/2752528894 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f64fc06ecc0 0x7f64fc071170 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f6504006fd0 tx=0x7f6504008040 comp rx=0 tx=0).stop 2024-04-20T22:09:31.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:31.449+0000 7f6517e00700 1 -- 172.21.15.62:0/2752528894 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f65181024d0 msgr2=0x7f6518197f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:09:31.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:31.449+0000 7f6517e00700 1 --2- 172.21.15.62:0/2752528894 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f65181024d0 0x7f6518197f30 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f6508006010 tx=0x7f650800be20 comp rx=0 tx=0).stop 2024-04-20T22:09:31.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:31.449+0000 7f6517e00700 1 -- 172.21.15.62:0/2752528894 shutdown_connections 2024-04-20T22:09:31.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:31.449+0000 7f6517e00700 1 --2- 172.21.15.62:0/2752528894 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f64fc06ecc0 0x7f64fc071170 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:09:31.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:31.449+0000 7f6517e00700 1 --2- 172.21.15.62:0/2752528894 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f65181024d0 0x7f6518197f30 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:09:31.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:31.449+0000 7f6517e00700 1 --2- 172.21.15.62:0/2752528894 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6518102e70 0x7f6518198470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:09:31.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:31.449+0000 7f6517e00700 1 -- 172.21.15.62:0/2752528894 >> 172.21.15.62:0/2752528894 conn(0x7f65180fdd90 msgr2=0x7f65180ff580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:09:31.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:31.449+0000 7f6517e00700 1 -- 172.21.15.62:0/2752528894 shutdown_connections 2024-04-20T22:09:31.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:31.449+0000 7f6517e00700 1 -- 172.21.15.62:0/2752528894 wait complete. 2024-04-20T22:09:31.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:31.610+0000 7fe8da400700 1 -- 172.21.15.62:0/3351736489 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe8d4108470 msgr2=0x7fe8d4108840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:09:31.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:31.610+0000 7fe8da400700 1 --2- 172.21.15.62:0/3351736489 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe8d4108470 0x7fe8d4108840 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7fe8c4009910 tx=0x7fe8c4009c20 comp rx=0 tx=0).stop 2024-04-20T22:09:31.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:31.610+0000 7fe8da400700 1 -- 172.21.15.62:0/3351736489 shutdown_connections 2024-04-20T22:09:31.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:31.610+0000 7fe8da400700 1 --2- 172.21.15.62:0/3351736489 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8d4102470 0x7fe8d41028e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:09:31.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:31.610+0000 7fe8da400700 1 --2- 172.21.15.62:0/3351736489 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe8d4108470 0x7fe8d4108840 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:09:31.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:31.610+0000 7fe8da400700 1 -- 172.21.15.62:0/3351736489 >> 172.21.15.62:0/3351736489 conn(0x7fe8d40fdf90 msgr2=0x7fe8d41003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:09:31.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:31.610+0000 7fe8da400700 1 -- 172.21.15.62:0/3351736489 shutdown_connections 2024-04-20T22:09:31.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:31.610+0000 7fe8da400700 1 -- 172.21.15.62:0/3351736489 wait complete. 2024-04-20T22:09:31.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:31.611+0000 7fe8da400700 1 Processor -- start 2024-04-20T22:09:31.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:31.612+0000 7fe8da400700 1 -- start start 2024-04-20T22:09:31.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:31.612+0000 7fe8da400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe8d4102470 0x7fe8d4198210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:09:31.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:31.612+0000 7fe8da400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8d4108470 0x7fe8d4198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:09:31.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:31.612+0000 7fe8da400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8d4198e30 con 0x7fe8d4102470 2024-04-20T22:09:31.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:31.612+0000 7fe8da400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8d419cbc0 con 0x7fe8d4108470 2024-04-20T22:09:31.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:31.612+0000 7fe8d3400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe8d4102470 0x7fe8d4198210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:09:31.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:31.612+0000 7fe8d3400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe8d4102470 0x7fe8d4198210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59576/0 (socket says 172.21.15.62:59576) 2024-04-20T22:09:31.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:31.612+0000 7fe8d3400700 1 -- 172.21.15.62:0/3423108887 learned_addr learned my addr 172.21.15.62:0/3423108887 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:09:31.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:31.613+0000 7fe8d3400700 1 -- 172.21.15.62:0/3423108887 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8d4108470 msgr2=0x7fe8d4198750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:09:31.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:31.613+0000 7fe8d3400700 1 --2- 172.21.15.62:0/3423108887 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8d4108470 0x7fe8d4198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:09:31.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:31.613+0000 7fe8d3400700 1 -- 172.21.15.62:0/3423108887 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8c40095f0 con 0x7fe8d4102470 2024-04-20T22:09:31.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:31.613+0000 7fe8d3400700 1 --2- 172.21.15.62:0/3423108887 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe8d4102470 0x7fe8d4198210 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fe8c400b5c0 tx=0x7fe8c400be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:09:31.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:31.613+0000 7fe8cbe00700 1 -- 172.21.15.62:0/3423108887 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe8c401d070 con 0x7fe8d4102470 2024-04-20T22:09:31.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:31.613+0000 7fe8cbe00700 1 -- 172.21.15.62:0/3423108887 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe8c400f460 con 0x7fe8d4102470 2024-04-20T22:09:31.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:31.613+0000 7fe8cbe00700 1 -- 172.21.15.62:0/3423108887 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe8c4017610 con 0x7fe8d4102470 2024-04-20T22:09:31.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:31.613+0000 7fe8da400700 1 -- 172.21.15.62:0/3423108887 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8d419ce40 con 0x7fe8d4102470 2024-04-20T22:09:31.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:31.614+0000 7fe8da400700 1 -- 172.21.15.62:0/3423108887 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8d419d330 con 0x7fe8d4102470 2024-04-20T22:09:31.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:31.615+0000 7fe8da400700 1 -- 172.21.15.62:0/3423108887 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8d410a9e0 con 0x7fe8d4102470 2024-04-20T22:09:31.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:31.616+0000 7fe8cbe00700 1 -- 172.21.15.62:0/3423108887 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe8c400f5d0 con 0x7fe8d4102470 2024-04-20T22:09:31.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:31.617+0000 7fe8cbe00700 1 --2- 172.21.15.62:0/3423108887 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe8b806ecc0 0x7fe8b8071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:09:31.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:31.617+0000 7fe8cbe00700 1 -- 172.21.15.62:0/3423108887 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fe8c4090450 con 0x7fe8d4102470 2024-04-20T22:09:31.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:31.617+0000 7fe8d2a00700 1 --2- 172.21.15.62:0/3423108887 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe8b806ecc0 0x7fe8b8071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:09:31.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:31.618+0000 7fe8d2a00700 1 --2- 172.21.15.62:0/3423108887 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe8b806ecc0 0x7fe8b8071170 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fe8d4199830 tx=0x7fe8c0006d20 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:09:31.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:31.621+0000 7fe8cbe00700 1 -- 172.21.15.62:0/3423108887 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fe8c40591e0 con 0x7fe8d4102470 2024-04-20T22:09:31.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:31.905+0000 7fe8da400700 1 -- 172.21.15.62:0/3423108887 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fe8d404f2a0 con 0x7fe8d4102470 2024-04-20T22:09:31.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:31.910+0000 7fe8cbe00700 1 -- 172.21.15.62:0/3423108887 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 27 v27) v1 ==== 76+0+1888 (secure 0 0 0) 0x7fe8c405c8a0 con 0x7fe8d4102470 2024-04-20T22:09:31.910 INFO:teuthology.orchestra.run.smithi062.stdout:e27 2024-04-20T22:09:31.910 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T22:09:31.910 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:09:31.910 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T22:09:31.910 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:09:31.910 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T22:09:31.910 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T22:09:31.910 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 20 2024-04-20T22:09:31.911 INFO:teuthology.orchestra.run.smithi062.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-20T22:09:31.911 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T22:09:31.911 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T21:42:01.559061+0000 2024-04-20T22:09:31.911 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T22:09:31.911 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T22:09:31.911 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T22:09:31.911 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T22:09:31.911 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T22:09:31.911 INFO:teuthology.orchestra.run.smithi062.stdout:max_xattr_size 65536 2024-04-20T22:09:31.911 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T22:09:31.911 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T22:09:31.911 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 81 2024-04-20T22:09:31.911 INFO:teuthology.orchestra.run.smithi062.stdout:snap_rstat 1 2024-04-20T22:09:31.911 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:09:31.911 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 2 2024-04-20T22:09:31.911 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1 2024-04-20T22:09:31.912 INFO:teuthology.orchestra.run.smithi062.stdout:up {} 2024-04-20T22:09:31.912 INFO:teuthology.orchestra.run.smithi062.stdout:failed 0,1 2024-04-20T22:09:31.912 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T22:09:31.912 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T22:09:31.912 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T22:09:31.912 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T22:09:31.912 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data enabled 2024-04-20T22:09:31.912 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T22:09:31.912 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T22:09:31.912 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 1 2024-04-20T22:09:31.912 INFO:teuthology.orchestra.run.smithi062.stdout:qdb_cluster leader: 0 members: 2024-04-20T22:09:31.912 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:09:31.912 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:09:31.912 INFO:teuthology.orchestra.run.smithi062.stdout:Standby daemons: 2024-04-20T22:09:31.912 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:09:31.913 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ryqblb{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6826/4113916205,v1:172.21.15.62:6827/4113916205] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:09:31.913 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ozizwi{-1:34476} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6828/3801975165,v1:172.21.15.62:6829/3801975165] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:09:31.913 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.xbgspw{-1:44285} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6826/3996040526,v1:172.21.15.189:6827/3996040526] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T22:09:31.913 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.dvimer{-1:44727} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6824/3576699232,v1:172.21.15.189:6825/3576699232] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:09:31.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:31.913+0000 7fe8da400700 1 -- 172.21.15.62:0/3423108887 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe8b806ecc0 msgr2=0x7fe8b8071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:09:31.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:31.913+0000 7fe8da400700 1 --2- 172.21.15.62:0/3423108887 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe8b806ecc0 0x7fe8b8071170 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fe8d4199830 tx=0x7fe8c0006d20 comp rx=0 tx=0).stop 2024-04-20T22:09:31.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:31.913+0000 7fe8da400700 1 -- 172.21.15.62:0/3423108887 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe8d4102470 msgr2=0x7fe8d4198210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:09:31.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:31.913+0000 7fe8da400700 1 --2- 172.21.15.62:0/3423108887 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe8d4102470 0x7fe8d4198210 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fe8c400b5c0 tx=0x7fe8c400be00 comp rx=0 tx=0).stop 2024-04-20T22:09:31.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:31.913+0000 7fe8da400700 1 -- 172.21.15.62:0/3423108887 shutdown_connections 2024-04-20T22:09:31.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:31.913+0000 7fe8da400700 1 --2- 172.21.15.62:0/3423108887 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe8b806ecc0 0x7fe8b8071170 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:09:31.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:31.913+0000 7fe8da400700 1 --2- 172.21.15.62:0/3423108887 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe8d4102470 0x7fe8d4198210 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:09:31.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:31.913+0000 7fe8da400700 1 --2- 172.21.15.62:0/3423108887 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8d4108470 0x7fe8d4198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:09:31.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:31.913+0000 7fe8da400700 1 -- 172.21.15.62:0/3423108887 >> 172.21.15.62:0/3423108887 conn(0x7fe8d40fdf90 msgr2=0x7fe8d40febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:09:31.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:31.913+0000 7fe8da400700 1 -- 172.21.15.62:0/3423108887 shutdown_connections 2024-04-20T22:09:31.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:31.914+0000 7fe8da400700 1 -- 172.21.15.62:0/3423108887 wait complete. 2024-04-20T22:09:31.914 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 27 2024-04-20T22:09:32.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:32.079+0000 7ff7cfa00700 1 -- 172.21.15.62:0/3102184560 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff7c8108470 msgr2=0x7ff7c8108840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:09:32.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:32.079+0000 7ff7cfa00700 1 --2- 172.21.15.62:0/3102184560 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff7c8108470 0x7ff7c8108840 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7ff7bc009910 tx=0x7ff7bc009c20 comp rx=0 tx=0).stop 2024-04-20T22:09:32.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:32.079+0000 7ff7cfa00700 1 -- 172.21.15.62:0/3102184560 shutdown_connections 2024-04-20T22:09:32.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:32.079+0000 7ff7cfa00700 1 --2- 172.21.15.62:0/3102184560 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff7c8102470 0x7ff7c81028e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:09:32.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:32.079+0000 7ff7cfa00700 1 --2- 172.21.15.62:0/3102184560 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff7c8108470 0x7ff7c8108840 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:09:32.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:32.079+0000 7ff7cfa00700 1 -- 172.21.15.62:0/3102184560 >> 172.21.15.62:0/3102184560 conn(0x7ff7c80fdf90 msgr2=0x7ff7c81003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:09:32.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:32.079+0000 7ff7cfa00700 1 -- 172.21.15.62:0/3102184560 shutdown_connections 2024-04-20T22:09:32.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:32.079+0000 7ff7cfa00700 1 -- 172.21.15.62:0/3102184560 wait complete. 2024-04-20T22:09:32.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:32.080+0000 7ff7cfa00700 1 Processor -- start 2024-04-20T22:09:32.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:32.081+0000 7ff7cfa00700 1 -- start start 2024-04-20T22:09:32.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:32.081+0000 7ff7cfa00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff7c8102470 0x7ff7c8198210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:09:32.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:32.081+0000 7ff7cfa00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff7c8108470 0x7ff7c8198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:09:32.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:32.081+0000 7ff7cfa00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7c8198e30 con 0x7ff7c8102470 2024-04-20T22:09:32.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:32.081+0000 7ff7cfa00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7c819cbc0 con 0x7ff7c8108470 2024-04-20T22:09:32.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:32.081+0000 7ff7cd000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff7c8102470 0x7ff7c8198210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:09:32.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:32.081+0000 7ff7cd000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff7c8102470 0x7ff7c8198210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59590/0 (socket says 172.21.15.62:59590) 2024-04-20T22:09:32.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:32.081+0000 7ff7cd000700 1 -- 172.21.15.62:0/1407887799 learned_addr learned my addr 172.21.15.62:0/1407887799 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:09:32.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:32.082+0000 7ff7c7e00700 1 --2- 172.21.15.62:0/1407887799 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff7c8108470 0x7ff7c8198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:09:32.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:32.082+0000 7ff7cd000700 1 -- 172.21.15.62:0/1407887799 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff7c8108470 msgr2=0x7ff7c8198750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:09:32.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:32.082+0000 7ff7cd000700 1 --2- 172.21.15.62:0/1407887799 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff7c8108470 0x7ff7c8198750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:09:32.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:32.082+0000 7ff7cd000700 1 -- 172.21.15.62:0/1407887799 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7bc0095f0 con 0x7ff7c8102470 2024-04-20T22:09:32.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:32.082+0000 7ff7cd000700 1 --2- 172.21.15.62:0/1407887799 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff7c8102470 0x7ff7c8198210 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7ff7bc00b5c0 tx=0x7ff7bc00be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:09:32.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:32.083+0000 7ff7c5600700 1 -- 172.21.15.62:0/1407887799 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff7bc01d070 con 0x7ff7c8102470 2024-04-20T22:09:32.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:32.083+0000 7ff7c5600700 1 -- 172.21.15.62:0/1407887799 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff7bc00f460 con 0x7ff7c8102470 2024-04-20T22:09:32.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:32.083+0000 7ff7c5600700 1 -- 172.21.15.62:0/1407887799 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff7bc017610 con 0x7ff7c8102470 2024-04-20T22:09:32.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:32.083+0000 7ff7cfa00700 1 -- 172.21.15.62:0/1407887799 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7c819ce40 con 0x7ff7c8102470 2024-04-20T22:09:32.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:32.083+0000 7ff7cfa00700 1 -- 172.21.15.62:0/1407887799 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7c819d330 con 0x7ff7c8102470 2024-04-20T22:09:32.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:32.084+0000 7ff7cfa00700 1 -- 172.21.15.62:0/1407887799 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7c810a9e0 con 0x7ff7c8102470 2024-04-20T22:09:32.085 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:32.085+0000 7ff7c5600700 1 -- 172.21.15.62:0/1407887799 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff7bc00f5d0 con 0x7ff7c8102470 2024-04-20T22:09:32.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:32.086+0000 7ff7c5600700 1 --2- 172.21.15.62:0/1407887799 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff7b006ecc0 0x7ff7b0071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:09:32.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:32.086+0000 7ff7c5600700 1 -- 172.21.15.62:0/1407887799 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7ff7bc090450 con 0x7ff7c8102470 2024-04-20T22:09:32.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:32.087+0000 7ff7c7e00700 1 --2- 172.21.15.62:0/1407887799 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff7b006ecc0 0x7ff7b0071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:09:32.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:32.087+0000 7ff7c7e00700 1 --2- 172.21.15.62:0/1407887799 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff7b006ecc0 0x7ff7b0071170 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7ff7b4006fd0 tx=0x7ff7b4008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:09:32.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:32.091+0000 7ff7c5600700 1 -- 172.21.15.62:0/1407887799 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7ff7bc0591e0 con 0x7ff7c8102470 2024-04-20T22:09:32.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:09:31 smithi189 ceph-mon[57022]: from='client.35828 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:09:32.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:09:31 smithi189 ceph-mon[57022]: from='client.35832 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:09:32.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:09:31 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/2752528894' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:09:32.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:09:31 smithi062 ceph-mon[76322]: from='client.35828 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:09:32.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:09:31 smithi062 ceph-mon[76322]: from='client.35832 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:09:32.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:09:31 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/2752528894' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:09:32.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:32.360+0000 7ff7cfa00700 1 -- 172.21.15.62:0/1407887799 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff7c819d5a0 con 0x7ff7b006ecc0 2024-04-20T22:09:32.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:32.362+0000 7ff7c5600700 1 -- 172.21.15.62:0/1407887799 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7ff7c819d5a0 con 0x7ff7b006ecc0 2024-04-20T22:09:32.362 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:09:32.363 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T22:09:32.363 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T22:09:32.363 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-20T22:09:32.363 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T22:09:32.363 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr", 2024-04-20T22:09:32.363 INFO:teuthology.orchestra.run.smithi062.stdout: "crash", 2024-04-20T22:09:32.363 INFO:teuthology.orchestra.run.smithi062.stdout: "mon", 2024-04-20T22:09:32.363 INFO:teuthology.orchestra.run.smithi062.stdout: "osd" 2024-04-20T22:09:32.363 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T22:09:32.363 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "15/23 daemons upgraded", 2024-04-20T22:09:32.363 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading mds daemons", 2024-04-20T22:09:32.363 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T22:09:32.363 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:09:32.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:32.365+0000 7ff7cfa00700 1 -- 172.21.15.62:0/1407887799 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff7b006ecc0 msgr2=0x7ff7b0071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:09:32.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:32.365+0000 7ff7cfa00700 1 --2- 172.21.15.62:0/1407887799 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff7b006ecc0 0x7ff7b0071170 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7ff7b4006fd0 tx=0x7ff7b4008040 comp rx=0 tx=0).stop 2024-04-20T22:09:32.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:32.365+0000 7ff7cfa00700 1 -- 172.21.15.62:0/1407887799 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff7c8102470 msgr2=0x7ff7c8198210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:09:32.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:32.365+0000 7ff7cfa00700 1 --2- 172.21.15.62:0/1407887799 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff7c8102470 0x7ff7c8198210 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7ff7bc00b5c0 tx=0x7ff7bc00be00 comp rx=0 tx=0).stop 2024-04-20T22:09:32.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:32.365+0000 7ff7cfa00700 1 -- 172.21.15.62:0/1407887799 shutdown_connections 2024-04-20T22:09:32.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:32.365+0000 7ff7cfa00700 1 --2- 172.21.15.62:0/1407887799 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff7b006ecc0 0x7ff7b0071170 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:09:32.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:32.365+0000 7ff7cfa00700 1 --2- 172.21.15.62:0/1407887799 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff7c8102470 0x7ff7c8198210 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:09:32.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:32.365+0000 7ff7cfa00700 1 --2- 172.21.15.62:0/1407887799 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff7c8108470 0x7ff7c8198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:09:32.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:32.366+0000 7ff7cfa00700 1 -- 172.21.15.62:0/1407887799 >> 172.21.15.62:0/1407887799 conn(0x7ff7c80fdf90 msgr2=0x7ff7c80febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:09:32.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:32.366+0000 7ff7cfa00700 1 -- 172.21.15.62:0/1407887799 shutdown_connections 2024-04-20T22:09:32.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:32.366+0000 7ff7cfa00700 1 -- 172.21.15.62:0/1407887799 wait complete. 2024-04-20T22:09:32.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:32.525+0000 7f8ad0a00700 1 -- 172.21.15.62:0/3773645296 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8acc108470 msgr2=0x7f8acc108840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:09:32.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:32.525+0000 7f8ad0a00700 1 --2- 172.21.15.62:0/3773645296 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8acc108470 0x7f8acc108840 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f8abc009910 tx=0x7f8abc009c20 comp rx=0 tx=0).stop 2024-04-20T22:09:32.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:32.525+0000 7f8ad0a00700 1 -- 172.21.15.62:0/3773645296 shutdown_connections 2024-04-20T22:09:32.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:32.525+0000 7f8ad0a00700 1 --2- 172.21.15.62:0/3773645296 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8acc102470 0x7f8acc1028e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:09:32.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:32.525+0000 7f8ad0a00700 1 --2- 172.21.15.62:0/3773645296 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8acc108470 0x7f8acc108840 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:09:32.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:32.525+0000 7f8ad0a00700 1 -- 172.21.15.62:0/3773645296 >> 172.21.15.62:0/3773645296 conn(0x7f8acc0fdf90 msgr2=0x7f8acc1003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:09:32.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:32.526+0000 7f8ad0a00700 1 -- 172.21.15.62:0/3773645296 shutdown_connections 2024-04-20T22:09:32.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:32.526+0000 7f8ad0a00700 1 -- 172.21.15.62:0/3773645296 wait complete. 2024-04-20T22:09:32.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:32.527+0000 7f8ad0a00700 1 Processor -- start 2024-04-20T22:09:32.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:32.527+0000 7f8ad0a00700 1 -- start start 2024-04-20T22:09:32.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:32.528+0000 7f8ad0a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8acc102470 0x7f8acc198230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:09:32.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:32.528+0000 7f8ad0a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8acc108470 0x7f8acc198770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:09:32.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:32.528+0000 7f8ad0a00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8acc198dc0 con 0x7f8acc108470 2024-04-20T22:09:32.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:32.528+0000 7f8ad0a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8acc198f00 con 0x7f8acc102470 2024-04-20T22:09:32.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:32.528+0000 7f8ac9400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8acc108470 0x7f8acc198770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:09:32.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:32.528+0000 7f8ac9400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8acc108470 0x7f8acc198770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59604/0 (socket says 172.21.15.62:59604) 2024-04-20T22:09:32.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:32.528+0000 7f8ac9400700 1 -- 172.21.15.62:0/2799382304 learned_addr learned my addr 172.21.15.62:0/2799382304 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:09:32.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:32.529+0000 7f8ac9e00700 1 --2- 172.21.15.62:0/2799382304 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8acc102470 0x7f8acc198230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:09:32.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:32.529+0000 7f8ac9400700 1 -- 172.21.15.62:0/2799382304 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8acc102470 msgr2=0x7f8acc198230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:09:32.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:32.529+0000 7f8ac9400700 1 --2- 172.21.15.62:0/2799382304 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8acc102470 0x7f8acc198230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:09:32.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:32.529+0000 7f8ac9400700 1 -- 172.21.15.62:0/2799382304 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8abc0095f0 con 0x7f8acc108470 2024-04-20T22:09:32.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:32.529+0000 7f8ac9e00700 1 --2- 172.21.15.62:0/2799382304 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8acc102470 0x7f8acc198230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T22:09:32.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:32.529+0000 7f8ac9400700 1 --2- 172.21.15.62:0/2799382304 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8acc108470 0x7f8acc198770 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f8ab800c930 tx=0x7f8ab800ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:09:32.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:32.529+0000 7f8ac2a00700 1 -- 172.21.15.62:0/2799382304 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8ab8007ad0 con 0x7f8acc108470 2024-04-20T22:09:32.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:32.529+0000 7f8ac2a00700 1 -- 172.21.15.62:0/2799382304 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8ab8007c30 con 0x7f8acc108470 2024-04-20T22:09:32.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:32.529+0000 7f8ac2a00700 1 -- 172.21.15.62:0/2799382304 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8ab8018610 con 0x7f8acc108470 2024-04-20T22:09:32.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:32.529+0000 7f8ad0a00700 1 -- 172.21.15.62:0/2799382304 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8acc19cd50 con 0x7f8acc108470 2024-04-20T22:09:32.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:32.530+0000 7f8ad0a00700 1 -- 172.21.15.62:0/2799382304 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8acc19d250 con 0x7f8acc108470 2024-04-20T22:09:32.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:32.531+0000 7f8ad0a00700 1 -- 172.21.15.62:0/2799382304 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8acc10a9e0 con 0x7f8acc108470 2024-04-20T22:09:32.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:32.532+0000 7f8ac2a00700 1 -- 172.21.15.62:0/2799382304 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f8ab800f450 con 0x7f8acc108470 2024-04-20T22:09:32.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:32.534+0000 7f8ac2a00700 1 --2- 172.21.15.62:0/2799382304 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f8ab006ecc0 0x7f8ab0071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:09:32.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:32.534+0000 7f8ac2a00700 1 -- 172.21.15.62:0/2799382304 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f8ab808eff0 con 0x7f8acc108470 2024-04-20T22:09:32.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:32.534+0000 7f8ac9e00700 1 --2- 172.21.15.62:0/2799382304 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f8ab006ecc0 0x7f8ab0071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:09:32.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:32.535+0000 7f8ac9e00700 1 --2- 172.21.15.62:0/2799382304 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f8ab006ecc0 0x7f8ab0071170 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f8abc006010 tx=0x7f8abc00b540 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:09:32.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:32.540+0000 7f8ac2a00700 1 -- 172.21.15.62:0/2799382304 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f8ab8057de0 con 0x7f8acc108470 2024-04-20T22:09:32.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:32.871+0000 7f8ad0a00700 1 -- 172.21.15.62:0/2799382304 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f8acc04f2a0 con 0x7f8acc108470 2024-04-20T22:09:32.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:32.872+0000 7f8ac2a00700 1 -- 172.21.15.62:0/2799382304 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7f8ab805b4a0 con 0x7f8acc108470 2024-04-20T22:09:32.873 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-20T22:09:32.873 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-20T22:09:32.873 INFO:teuthology.orchestra.run.smithi062.stdout: host smithi062 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T22:09:32.873 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-20T22:09:32.873 INFO:teuthology.orchestra.run.smithi062.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi062 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpq7y7oa_r:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:09:32.873 INFO:teuthology.orchestra.run.smithi062.stdout:Traceback (most recent call last): 2024-04-20T22:09:32.873 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-20T22:09:32.873 INFO:teuthology.orchestra.run.smithi062.stdout: return _run_code(code, main_globals, None, 2024-04-20T22:09:32.873 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-20T22:09:32.874 INFO:teuthology.orchestra.run.smithi062.stdout: exec(code, run_globals) 2024-04-20T22:09:32.874 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5618, in 2024-04-20T22:09:32.874 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5606, in main 2024-04-20T22:09:32.874 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 408, in _infer_config 2024-04-20T22:09:32.874 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 323, in _infer_fsid 2024-04-20T22:09:32.874 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 436, in _infer_image 2024-04-20T22:09:32.874 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 310, in _validate_fsid 2024-04-20T22:09:32.874 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 3359, in command_ceph_volume 2024-04-20T22:09:32.874 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-20T22:09:32.874 INFO:teuthology.orchestra.run.smithi062.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi062 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpq7y7oa_r:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:09:32.874 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-20T22:09:32.874 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is degraded 2024-04-20T22:09:32.874 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T22:09:32.874 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T22:09:32.875 INFO:teuthology.orchestra.run.smithi062.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-20T22:09:32.875 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-20T22:09:32.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:32.876+0000 7f8ad0a00700 1 -- 172.21.15.62:0/2799382304 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f8ab006ecc0 msgr2=0x7f8ab0071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:09:32.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:32.876+0000 7f8ad0a00700 1 --2- 172.21.15.62:0/2799382304 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f8ab006ecc0 0x7f8ab0071170 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f8abc006010 tx=0x7f8abc00b540 comp rx=0 tx=0).stop 2024-04-20T22:09:32.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:32.876+0000 7f8ad0a00700 1 -- 172.21.15.62:0/2799382304 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8acc108470 msgr2=0x7f8acc198770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:09:32.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:32.876+0000 7f8ad0a00700 1 --2- 172.21.15.62:0/2799382304 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8acc108470 0x7f8acc198770 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f8ab800c930 tx=0x7f8ab800ccf0 comp rx=0 tx=0).stop 2024-04-20T22:09:32.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:32.876+0000 7f8ad0a00700 1 -- 172.21.15.62:0/2799382304 shutdown_connections 2024-04-20T22:09:32.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:32.876+0000 7f8ad0a00700 1 --2- 172.21.15.62:0/2799382304 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f8ab006ecc0 0x7f8ab0071170 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:09:32.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:32.876+0000 7f8ad0a00700 1 --2- 172.21.15.62:0/2799382304 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8acc102470 0x7f8acc198230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:09:32.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:32.876+0000 7f8ad0a00700 1 --2- 172.21.15.62:0/2799382304 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8acc108470 0x7f8acc198770 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:09:32.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:32.876+0000 7f8ad0a00700 1 -- 172.21.15.62:0/2799382304 >> 172.21.15.62:0/2799382304 conn(0x7f8acc0fdf90 msgr2=0x7f8acc0ff300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:09:32.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:32.876+0000 7f8ad0a00700 1 -- 172.21.15.62:0/2799382304 shutdown_connections 2024-04-20T22:09:32.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:09:32.877+0000 7f8ad0a00700 1 -- 172.21.15.62:0/2799382304 wait complete. 2024-04-20T22:09:33.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:09:32 smithi189 ceph-mon[57022]: pgmap v1028: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:09:33.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:09:32 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/3423108887' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:09:33.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:09:32 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/2799382304' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:09:33.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:09:32 smithi062 ceph-mon[76322]: pgmap v1028: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:09:33.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:09:32 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/3423108887' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:09:33.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:09:32 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/2799382304' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:09:34.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:09:33 smithi189 ceph-mon[57022]: from='client.35844 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:09:34.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:09:33 smithi062 ceph-mon[76322]: from='client.35844 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:09:35.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:09:34 smithi062 ceph-mon[76322]: pgmap v1029: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:09:35.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:09:34 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:09:35.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:09:34 smithi189 ceph-mon[57022]: pgmap v1029: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:09:35.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:09:34 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:09:37.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:09:36 smithi189 ceph-mon[57022]: pgmap v1030: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:09:37.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:09:36 smithi062 ceph-mon[76322]: pgmap v1030: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:09:39.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:09:38 smithi189 ceph-mon[57022]: pgmap v1031: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:09:39.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:09:38 smithi062 ceph-mon[76322]: pgmap v1031: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:09:41.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:09:40 smithi189 ceph-mon[57022]: pgmap v1032: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:09:41.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:09:40 smithi062 ceph-mon[76322]: pgmap v1032: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:09:43.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:09:42 smithi189 ceph-mon[57022]: pgmap v1033: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:09:43.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:09:42 smithi062 ceph-mon[76322]: pgmap v1033: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:09:45.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:09:44 smithi189 ceph-mon[57022]: pgmap v1034: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:09:45.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:09:44 smithi062 ceph-mon[76322]: pgmap v1034: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:09:47.224 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:09:46 smithi189 ceph-mon[57022]: pgmap v1035: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:09:47.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:09:46 smithi062 ceph-mon[76322]: pgmap v1035: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:09:49.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:09:48 smithi189 ceph-mon[57022]: pgmap v1036: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:09:49.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:09:48 smithi062 ceph-mon[76322]: pgmap v1036: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:09:50.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:09:49 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:09:50.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:09:49 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:09:51.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:09:50 smithi189 ceph-mon[57022]: pgmap v1037: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:09:51.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:09:50 smithi062 ceph-mon[76322]: pgmap v1037: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:09:53.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:09:52 smithi062 ceph-mon[76322]: pgmap v1038: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:09:53.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:09:52 smithi189 ceph-mon[57022]: pgmap v1038: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:09:55.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:09:54 smithi062 ceph-mon[76322]: pgmap v1039: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:09:55.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:09:54 smithi189 ceph-mon[57022]: pgmap v1039: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:09:57.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:09:57 smithi062 ceph-mon[76322]: pgmap v1040: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:09:57.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:09:57 smithi189 ceph-mon[57022]: pgmap v1040: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:09:59.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:09:59 smithi062 ceph-mon[76322]: pgmap v1041: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:09:59.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:09:59 smithi189 ceph-mon[57022]: pgmap v1041: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:10:00.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:10:00 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-mon-smithi062[76318]: 2024-04-20T22:09:59.999+0000 7f6157e00700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-20T22:10:00.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:10:00 smithi062 ceph-mon[76322]: overall HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-20T22:10:00.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:10:00 smithi189 ceph-mon[57022]: overall HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-20T22:10:01.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:10:01 smithi062 ceph-mon[76322]: pgmap v1042: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:10:01.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:10:01 smithi189 ceph-mon[57022]: pgmap v1042: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:10:03.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.057+0000 7f455fe00700 1 -- 172.21.15.62:0/3499463691 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4560100290 msgr2=0x7f4560100660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:10:03.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.057+0000 7f455fe00700 1 --2- 172.21.15.62:0/3499463691 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4560100290 0x7f4560100660 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f455000b0a0 tx=0x7f455000b3b0 comp rx=0 tx=0).stop 2024-04-20T22:10:03.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.057+0000 7f455fe00700 1 -- 172.21.15.62:0/3499463691 shutdown_connections 2024-04-20T22:10:03.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.057+0000 7f455fe00700 1 --2- 172.21.15.62:0/3499463691 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4560100c30 0x7f4560104c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:10:03.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.057+0000 7f455fe00700 1 --2- 172.21.15.62:0/3499463691 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4560100290 0x7f4560100660 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:10:03.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.057+0000 7f455fe00700 1 -- 172.21.15.62:0/3499463691 >> 172.21.15.62:0/3499463691 conn(0x7f45600fbb10 msgr2=0x7f45600fdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:10:03.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.057+0000 7f455fe00700 1 -- 172.21.15.62:0/3499463691 shutdown_connections 2024-04-20T22:10:03.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.057+0000 7f455fe00700 1 -- 172.21.15.62:0/3499463691 wait complete. 2024-04-20T22:10:03.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.058+0000 7f455fe00700 1 Processor -- start 2024-04-20T22:10:03.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.059+0000 7f455fe00700 1 -- start start 2024-04-20T22:10:03.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.059+0000 7f455fe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4560100290 0x7f4560197e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:10:03.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.059+0000 7f455fe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4560100c30 0x7f45601983d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:10:03.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.059+0000 7f455fe00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4560198ab0 con 0x7f4560100290 2024-04-20T22:10:03.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.059+0000 7f455fe00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f456019c840 con 0x7f4560100c30 2024-04-20T22:10:03.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.059+0000 7f455ea00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4560100290 0x7f4560197e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:10:03.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.059+0000 7f455ea00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4560100290 0x7f4560197e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37698/0 (socket says 172.21.15.62:37698) 2024-04-20T22:10:03.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.059+0000 7f455ea00700 1 -- 172.21.15.62:0/2333787366 learned_addr learned my addr 172.21.15.62:0/2333787366 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:10:03.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.060+0000 7f454fe00700 1 --2- 172.21.15.62:0/2333787366 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4560100c30 0x7f45601983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:10:03.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.060+0000 7f455ea00700 1 -- 172.21.15.62:0/2333787366 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4560100c30 msgr2=0x7f45601983d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:10:03.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.060+0000 7f455ea00700 1 --2- 172.21.15.62:0/2333787366 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4560100c30 0x7f45601983d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:10:03.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.060+0000 7f455ea00700 1 -- 172.21.15.62:0/2333787366 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4550009d00 con 0x7f4560100290 2024-04-20T22:10:03.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.060+0000 7f454fe00700 1 --2- 172.21.15.62:0/2333787366 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4560100c30 0x7f45601983d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T22:10:03.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.060+0000 7f455ea00700 1 --2- 172.21.15.62:0/2333787366 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4560100290 0x7f4560197e90 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f455000b070 tx=0x7f455000cdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:10:03.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.060+0000 7f4557e00700 1 -- 172.21.15.62:0/2333787366 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f455000e040 con 0x7f4560100290 2024-04-20T22:10:03.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.061+0000 7f4557e00700 1 -- 172.21.15.62:0/2333787366 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4550003de0 con 0x7f4560100290 2024-04-20T22:10:03.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.061+0000 7f455fe00700 1 -- 172.21.15.62:0/2333787366 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f456019cac0 con 0x7f4560100290 2024-04-20T22:10:03.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.061+0000 7f4557e00700 1 -- 172.21.15.62:0/2333787366 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f455001d8e0 con 0x7f4560100290 2024-04-20T22:10:03.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.061+0000 7f455fe00700 1 -- 172.21.15.62:0/2333787366 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f456019cfb0 con 0x7f4560100290 2024-04-20T22:10:03.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.062+0000 7f455fe00700 1 -- 172.21.15.62:0/2333787366 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4560104380 con 0x7f4560100290 2024-04-20T22:10:03.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.063+0000 7f4557e00700 1 -- 172.21.15.62:0/2333787366 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f4550004040 con 0x7f4560100290 2024-04-20T22:10:03.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.064+0000 7f4557e00700 1 --2- 172.21.15.62:0/2333787366 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f454006ecc0 0x7f4540071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:10:03.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.064+0000 7f4557e00700 1 -- 172.21.15.62:0/2333787366 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f45500906b0 con 0x7f4560100290 2024-04-20T22:10:03.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.065+0000 7f454fe00700 1 --2- 172.21.15.62:0/2333787366 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f454006ecc0 0x7f4540071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:10:03.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.065+0000 7f454fe00700 1 --2- 172.21.15.62:0/2333787366 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f454006ecc0 0x7f4540071170 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f4548005fd0 tx=0x7f4548005ee0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:10:03.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.070+0000 7f4557e00700 1 -- 172.21.15.62:0/2333787366 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f4550059460 con 0x7f4560100290 2024-04-20T22:10:03.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:10:03 smithi062 ceph-mon[76322]: pgmap v1043: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:10:03.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.325+0000 7f455fe00700 1 -- 172.21.15.62:0/2333787366 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f456019d1d0 con 0x7f454006ecc0 2024-04-20T22:10:03.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.328+0000 7f4557e00700 1 -- 172.21.15.62:0/2333787366 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f456019d1d0 con 0x7f454006ecc0 2024-04-20T22:10:03.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.331+0000 7f455fe00700 1 -- 172.21.15.62:0/2333787366 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f454006ecc0 msgr2=0x7f4540071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:10:03.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.331+0000 7f455fe00700 1 --2- 172.21.15.62:0/2333787366 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f454006ecc0 0x7f4540071170 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f4548005fd0 tx=0x7f4548005ee0 comp rx=0 tx=0).stop 2024-04-20T22:10:03.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.331+0000 7f455fe00700 1 -- 172.21.15.62:0/2333787366 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4560100290 msgr2=0x7f4560197e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:10:03.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.331+0000 7f455fe00700 1 --2- 172.21.15.62:0/2333787366 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4560100290 0x7f4560197e90 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f455000b070 tx=0x7f455000cdd0 comp rx=0 tx=0).stop 2024-04-20T22:10:03.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.331+0000 7f455fe00700 1 -- 172.21.15.62:0/2333787366 shutdown_connections 2024-04-20T22:10:03.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.331+0000 7f455fe00700 1 --2- 172.21.15.62:0/2333787366 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f454006ecc0 0x7f4540071170 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:10:03.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.331+0000 7f455fe00700 1 --2- 172.21.15.62:0/2333787366 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4560100290 0x7f4560197e90 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:10:03.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.331+0000 7f455fe00700 1 --2- 172.21.15.62:0/2333787366 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4560100c30 0x7f45601983d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:10:03.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.331+0000 7f455fe00700 1 -- 172.21.15.62:0/2333787366 >> 172.21.15.62:0/2333787366 conn(0x7f45600fbb10 msgr2=0x7f45600fd1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:10:03.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.331+0000 7f455fe00700 1 -- 172.21.15.62:0/2333787366 shutdown_connections 2024-04-20T22:10:03.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.331+0000 7f455fe00700 1 -- 172.21.15.62:0/2333787366 wait complete. 2024-04-20T22:10:03.346 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T22:10:03.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:10:03 smithi189 ceph-mon[57022]: pgmap v1043: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:10:03.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.496+0000 7fb35ea00700 1 -- 172.21.15.62:0/769137459 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb358068ee0 msgr2=0x7fb358110a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:10:03.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.496+0000 7fb35ea00700 1 --2- 172.21.15.62:0/769137459 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb358068ee0 0x7fb358110a10 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fb340009940 tx=0x7fb340009c50 comp rx=0 tx=0).stop 2024-04-20T22:10:03.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.496+0000 7fb35ea00700 1 -- 172.21.15.62:0/769137459 shutdown_connections 2024-04-20T22:10:03.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.496+0000 7fb35ea00700 1 --2- 172.21.15.62:0/769137459 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb358068ee0 0x7fb358110a10 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:10:03.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.496+0000 7fb35ea00700 1 --2- 172.21.15.62:0/769137459 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb358068540 0x7fb358068910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:10:03.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.496+0000 7fb35ea00700 1 -- 172.21.15.62:0/769137459 >> 172.21.15.62:0/769137459 conn(0x7fb358075130 msgr2=0x7fb358075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:10:03.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.497+0000 7fb35ea00700 1 -- 172.21.15.62:0/769137459 shutdown_connections 2024-04-20T22:10:03.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.497+0000 7fb35ea00700 1 -- 172.21.15.62:0/769137459 wait complete. 2024-04-20T22:10:03.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.498+0000 7fb35ea00700 1 Processor -- start 2024-04-20T22:10:03.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.498+0000 7fb35ea00700 1 -- start start 2024-04-20T22:10:03.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.499+0000 7fb35ea00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb358068540 0x7fb35819c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:10:03.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.499+0000 7fb35ea00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb358068ee0 0x7fb35819c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:10:03.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.499+0000 7fb35ea00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb35819d080 con 0x7fb358068ee0 2024-04-20T22:10:03.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.499+0000 7fb35ea00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3581a0e10 con 0x7fb358068540 2024-04-20T22:10:03.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.499+0000 7fb34fe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb358068ee0 0x7fb35819c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:10:03.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.499+0000 7fb34fe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb358068ee0 0x7fb35819c9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37718/0 (socket says 172.21.15.62:37718) 2024-04-20T22:10:03.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.499+0000 7fb34fe00700 1 -- 172.21.15.62:0/2021833629 learned_addr learned my addr 172.21.15.62:0/2021833629 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:10:03.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.499+0000 7fb357e00700 1 --2- 172.21.15.62:0/2021833629 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb358068540 0x7fb35819c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:10:03.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.500+0000 7fb34fe00700 1 -- 172.21.15.62:0/2021833629 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb358068540 msgr2=0x7fb35819c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:10:03.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.500+0000 7fb34fe00700 1 --2- 172.21.15.62:0/2021833629 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb358068540 0x7fb35819c460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:10:03.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.500+0000 7fb34fe00700 1 -- 172.21.15.62:0/2021833629 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3400095f0 con 0x7fb358068ee0 2024-04-20T22:10:03.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.500+0000 7fb34fe00700 1 --2- 172.21.15.62:0/2021833629 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb358068ee0 0x7fb35819c9a0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7fb3400055f0 tx=0x7fb3400056d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:10:03.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.501+0000 7fb355600700 1 -- 172.21.15.62:0/2021833629 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb34001d070 con 0x7fb358068ee0 2024-04-20T22:10:03.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.501+0000 7fb355600700 1 -- 172.21.15.62:0/2021833629 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb34000f460 con 0x7fb358068ee0 2024-04-20T22:10:03.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.501+0000 7fb35ea00700 1 -- 172.21.15.62:0/2021833629 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3581a1090 con 0x7fb358068ee0 2024-04-20T22:10:03.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.501+0000 7fb355600700 1 -- 172.21.15.62:0/2021833629 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb340021640 con 0x7fb358068ee0 2024-04-20T22:10:03.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.501+0000 7fb35ea00700 1 -- 172.21.15.62:0/2021833629 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3581a15e0 con 0x7fb358068ee0 2024-04-20T22:10:03.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.502+0000 7fb35ea00700 1 -- 172.21.15.62:0/2021833629 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb358101410 con 0x7fb358068ee0 2024-04-20T22:10:03.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.503+0000 7fb355600700 1 -- 172.21.15.62:0/2021833629 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fb34000fab0 con 0x7fb358068ee0 2024-04-20T22:10:03.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.504+0000 7fb355600700 1 --2- 172.21.15.62:0/2021833629 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb33c06ecc0 0x7fb33c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:10:03.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.504+0000 7fb355600700 1 -- 172.21.15.62:0/2021833629 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fb3400900f0 con 0x7fb358068ee0 2024-04-20T22:10:03.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.505+0000 7fb357e00700 1 --2- 172.21.15.62:0/2021833629 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb33c06ecc0 0x7fb33c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:10:03.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.505+0000 7fb357e00700 1 --2- 172.21.15.62:0/2021833629 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb33c06ecc0 0x7fb33c071170 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fb348005950 tx=0x7fb3480058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:10:03.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.510+0000 7fb355600700 1 -- 172.21.15.62:0/2021833629 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fb340058ea0 con 0x7fb358068ee0 2024-04-20T22:10:03.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.771+0000 7fb35ea00700 1 -- 172.21.15.62:0/2021833629 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb3581a1800 con 0x7fb33c06ecc0 2024-04-20T22:10:03.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.774+0000 7fb355600700 1 -- 172.21.15.62:0/2021833629 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fb3581a1800 con 0x7fb33c06ecc0 2024-04-20T22:10:03.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.777+0000 7fb35ea00700 1 -- 172.21.15.62:0/2021833629 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb33c06ecc0 msgr2=0x7fb33c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:10:03.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.777+0000 7fb35ea00700 1 --2- 172.21.15.62:0/2021833629 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb33c06ecc0 0x7fb33c071170 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fb348005950 tx=0x7fb3480058e0 comp rx=0 tx=0).stop 2024-04-20T22:10:03.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.777+0000 7fb35ea00700 1 -- 172.21.15.62:0/2021833629 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb358068ee0 msgr2=0x7fb35819c9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:10:03.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.777+0000 7fb35ea00700 1 --2- 172.21.15.62:0/2021833629 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb358068ee0 0x7fb35819c9a0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7fb3400055f0 tx=0x7fb3400056d0 comp rx=0 tx=0).stop 2024-04-20T22:10:03.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.777+0000 7fb35ea00700 1 -- 172.21.15.62:0/2021833629 shutdown_connections 2024-04-20T22:10:03.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.777+0000 7fb35ea00700 1 --2- 172.21.15.62:0/2021833629 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb33c06ecc0 0x7fb33c071170 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:10:03.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.777+0000 7fb35ea00700 1 --2- 172.21.15.62:0/2021833629 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb358068540 0x7fb35819c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:10:03.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.777+0000 7fb35ea00700 1 --2- 172.21.15.62:0/2021833629 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb358068ee0 0x7fb35819c9a0 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:10:03.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.777+0000 7fb35ea00700 1 -- 172.21.15.62:0/2021833629 >> 172.21.15.62:0/2021833629 conn(0x7fb358075130 msgr2=0x7fb35810eed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:10:03.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.778+0000 7fb35ea00700 1 -- 172.21.15.62:0/2021833629 shutdown_connections 2024-04-20T22:10:03.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.778+0000 7fb35ea00700 1 -- 172.21.15.62:0/2021833629 wait complete. 2024-04-20T22:10:03.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.938+0000 7ff1c7a00700 1 -- 172.21.15.62:0/1810211297 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1c0102470 msgr2=0x7ff1c01028e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:10:03.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.938+0000 7ff1c7a00700 1 --2- 172.21.15.62:0/1810211297 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1c0102470 0x7ff1c01028e0 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7ff1ac009910 tx=0x7ff1ac009c20 comp rx=0 tx=0).stop 2024-04-20T22:10:03.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.938+0000 7ff1c7a00700 1 -- 172.21.15.62:0/1810211297 shutdown_connections 2024-04-20T22:10:03.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.938+0000 7ff1c7a00700 1 --2- 172.21.15.62:0/1810211297 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1c0102470 0x7ff1c01028e0 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:10:03.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.938+0000 7ff1c7a00700 1 --2- 172.21.15.62:0/1810211297 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1c0108470 0x7ff1c0108840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:10:03.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.938+0000 7ff1c7a00700 1 -- 172.21.15.62:0/1810211297 >> 172.21.15.62:0/1810211297 conn(0x7ff1c00fdf90 msgr2=0x7ff1c01003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:10:03.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.938+0000 7ff1c7a00700 1 -- 172.21.15.62:0/1810211297 shutdown_connections 2024-04-20T22:10:03.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.938+0000 7ff1c7a00700 1 -- 172.21.15.62:0/1810211297 wait complete. 2024-04-20T22:10:03.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.939+0000 7ff1c7a00700 1 Processor -- start 2024-04-20T22:10:03.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.939+0000 7ff1c7a00700 1 -- start start 2024-04-20T22:10:03.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.940+0000 7ff1c7a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1c0102470 0x7ff1c01981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:10:03.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.940+0000 7ff1c7a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1c0108470 0x7ff1c0198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:10:03.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.940+0000 7ff1c7a00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1c0198e10 con 0x7ff1c0108470 2024-04-20T22:10:03.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.940+0000 7ff1c7a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1c019cba0 con 0x7ff1c0102470 2024-04-20T22:10:03.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.940+0000 7ff1bfe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1c0108470 0x7ff1c0198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:10:03.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.940+0000 7ff1bfe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1c0108470 0x7ff1c0198730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37734/0 (socket says 172.21.15.62:37734) 2024-04-20T22:10:03.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.940+0000 7ff1bfe00700 1 -- 172.21.15.62:0/1620750500 learned_addr learned my addr 172.21.15.62:0/1620750500 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:10:03.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.940+0000 7ff1c5000700 1 --2- 172.21.15.62:0/1620750500 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1c0102470 0x7ff1c01981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:10:03.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.941+0000 7ff1bfe00700 1 -- 172.21.15.62:0/1620750500 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1c0102470 msgr2=0x7ff1c01981f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:10:03.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.941+0000 7ff1bfe00700 1 --2- 172.21.15.62:0/1620750500 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1c0102470 0x7ff1c01981f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:10:03.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.941+0000 7ff1bfe00700 1 -- 172.21.15.62:0/1620750500 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1ac0095f0 con 0x7ff1c0108470 2024-04-20T22:10:03.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.941+0000 7ff1c5000700 1 --2- 172.21.15.62:0/1620750500 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1c0102470 0x7ff1c01981f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-20T22:10:03.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.941+0000 7ff1bfe00700 1 --2- 172.21.15.62:0/1620750500 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1c0108470 0x7ff1c0198730 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7ff1ac006010 tx=0x7ff1ac00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:10:03.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.941+0000 7ff1bd600700 1 -- 172.21.15.62:0/1620750500 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff1ac01d070 con 0x7ff1c0108470 2024-04-20T22:10:03.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.941+0000 7ff1bd600700 1 -- 172.21.15.62:0/1620750500 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff1ac00f460 con 0x7ff1c0108470 2024-04-20T22:10:03.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.941+0000 7ff1c7a00700 1 -- 172.21.15.62:0/1620750500 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1c019ce20 con 0x7ff1c0108470 2024-04-20T22:10:03.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.941+0000 7ff1bd600700 1 -- 172.21.15.62:0/1620750500 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff1ac017630 con 0x7ff1c0108470 2024-04-20T22:10:03.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.942+0000 7ff1c7a00700 1 -- 172.21.15.62:0/1620750500 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1c019d310 con 0x7ff1c0108470 2024-04-20T22:10:03.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.943+0000 7ff1c7a00700 1 -- 172.21.15.62:0/1620750500 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1c010a970 con 0x7ff1c0108470 2024-04-20T22:10:03.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.944+0000 7ff1bd600700 1 -- 172.21.15.62:0/1620750500 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff1ac00f5d0 con 0x7ff1c0108470 2024-04-20T22:10:03.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.945+0000 7ff1bd600700 1 --2- 172.21.15.62:0/1620750500 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff1a806ecc0 0x7ff1a8071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:10:03.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.945+0000 7ff1bd600700 1 -- 172.21.15.62:0/1620750500 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7ff1ac090390 con 0x7ff1c0108470 2024-04-20T22:10:03.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.945+0000 7ff1c5000700 1 --2- 172.21.15.62:0/1620750500 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff1a806ecc0 0x7ff1a8071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:10:03.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.946+0000 7ff1c5000700 1 --2- 172.21.15.62:0/1620750500 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff1a806ecc0 0x7ff1a8071170 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7ff1b4005950 tx=0x7ff1b40058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:10:03.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:03.950+0000 7ff1bd600700 1 -- 172.21.15.62:0/1620750500 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7ff1ac0590a0 con 0x7ff1c0108470 2024-04-20T22:10:04.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:04.201+0000 7ff1c7a00700 1 -- 172.21.15.62:0/1620750500 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7ff1c019d530 con 0x7ff1a806ecc0 2024-04-20T22:10:04.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:04.232+0000 7ff1bd600700 1 -- 172.21.15.62:0/1620750500 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7ff1c019d530 con 0x7ff1a806ecc0 2024-04-20T22:10:04.232 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T22:10:04.232 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (35m) 11m ago 49m 15.3M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T22:10:04.232 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (50m) 11m ago 50m 10.4M - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T22:10:04.232 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (48m) 12m ago 48m 10.1M - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T22:10:04.233 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (32m) 11m ago 50m 6660k - 19.0.0-3133-ga8c84f59 76bc6b91b726 2825d5ba11c6 2024-04-20T22:10:04.233 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (31m) 12m ago 48m 6665k - 19.0.0-3133-ga8c84f59 76bc6b91b726 20f746fa1bba 2024-04-20T22:10:04.233 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (35m) 11m ago 49m 73.5M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T22:10:04.233 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (27m) 11m ago 42m 18.5M - 19.0.0-3133-ga8c84f59 76bc6b91b726 de80bc90a683 2024-04-20T22:10:04.233 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (27m) 11m ago 42m 18.7M - 19.0.0-3133-ga8c84f59 76bc6b91b726 cd793b7ee98e 2024-04-20T22:10:04.233 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (12m) 12m ago 42m 11.0M - 19.0.0-3133-ga8c84f59 76bc6b91b726 0e7b0bc4d1ab 2024-04-20T22:10:04.233 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (42m) 12m ago 42m 18.6M - 18.2.0 dc2bc1663786 f2eec2ee8951 2024-04-20T22:10:04.233 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (38m) 11m ago 51m 513M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T22:10:04.233 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (37m) 12m ago 48m 459M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T22:10:04.233 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (33m) 11m ago 51m 71.7M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 019eb1479485 2024-04-20T22:10:04.233 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (33m) 12m ago 48m 51.0M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 55aac560aab6 2024-04-20T22:10:04.233 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (37m) 11m ago 49m 8472k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T22:10:04.233 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (36m) 12m ago 48m 8048k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T22:10:04.234 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (31m) 11m ago 47m 91.2M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 ee2bb901f26e 2024-04-20T22:10:04.234 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (30m) 11m ago 46m 94.6M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 9075152d4162 2024-04-20T22:10:04.234 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (30m) 11m ago 46m 78.1M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 2d4f3e38dc0e 2024-04-20T22:10:04.234 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (29m) 12m ago 45m 92.6M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 8f97b17ae0eb 2024-04-20T22:10:04.234 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (29m) 12m ago 45m 95.6M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 653d8079967d 2024-04-20T22:10:04.234 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (28m) 12m ago 44m 89.2M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 596dded14c12 2024-04-20T22:10:04.234 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (36m) 11m ago 49m 82.1M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T22:10:04.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:04.236+0000 7ff1c7a00700 1 -- 172.21.15.62:0/1620750500 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff1a806ecc0 msgr2=0x7ff1a8071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:10:04.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:04.236+0000 7ff1c7a00700 1 --2- 172.21.15.62:0/1620750500 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff1a806ecc0 0x7ff1a8071170 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7ff1b4005950 tx=0x7ff1b40058e0 comp rx=0 tx=0).stop 2024-04-20T22:10:04.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:04.236+0000 7ff1c7a00700 1 -- 172.21.15.62:0/1620750500 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1c0108470 msgr2=0x7ff1c0198730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:10:04.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:04.236+0000 7ff1c7a00700 1 --2- 172.21.15.62:0/1620750500 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1c0108470 0x7ff1c0198730 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7ff1ac006010 tx=0x7ff1ac00be20 comp rx=0 tx=0).stop 2024-04-20T22:10:04.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:04.236+0000 7ff1c7a00700 1 -- 172.21.15.62:0/1620750500 shutdown_connections 2024-04-20T22:10:04.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:04.236+0000 7ff1c7a00700 1 --2- 172.21.15.62:0/1620750500 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff1a806ecc0 0x7ff1a8071170 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:10:04.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:04.236+0000 7ff1c7a00700 1 --2- 172.21.15.62:0/1620750500 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1c0102470 0x7ff1c01981f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:10:04.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:04.236+0000 7ff1c7a00700 1 --2- 172.21.15.62:0/1620750500 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1c0108470 0x7ff1c0198730 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:10:04.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:04.236+0000 7ff1c7a00700 1 -- 172.21.15.62:0/1620750500 >> 172.21.15.62:0/1620750500 conn(0x7ff1c00fdf90 msgr2=0x7ff1c00ff3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:10:04.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:04.237+0000 7ff1c7a00700 1 -- 172.21.15.62:0/1620750500 shutdown_connections 2024-04-20T22:10:04.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:04.237+0000 7ff1c7a00700 1 -- 172.21.15.62:0/1620750500 wait complete. 2024-04-20T22:10:04.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:04.398+0000 7fab1f800700 1 -- 172.21.15.62:0/3451486929 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fab18108470 msgr2=0x7fab18108840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:10:04.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:04.398+0000 7fab1f800700 1 --2- 172.21.15.62:0/3451486929 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fab18108470 0x7fab18108840 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7faafc009910 tx=0x7faafc009c20 comp rx=0 tx=0).stop 2024-04-20T22:10:04.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:04.398+0000 7fab1f800700 1 -- 172.21.15.62:0/3451486929 shutdown_connections 2024-04-20T22:10:04.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:04.398+0000 7fab1f800700 1 --2- 172.21.15.62:0/3451486929 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fab18102470 0x7fab181028e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:10:04.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:04.398+0000 7fab1f800700 1 --2- 172.21.15.62:0/3451486929 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fab18108470 0x7fab18108840 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:10:04.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:04.398+0000 7fab1f800700 1 -- 172.21.15.62:0/3451486929 >> 172.21.15.62:0/3451486929 conn(0x7fab180fdf90 msgr2=0x7fab181003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:10:04.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:04.398+0000 7fab1f800700 1 -- 172.21.15.62:0/3451486929 shutdown_connections 2024-04-20T22:10:04.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:04.398+0000 7fab1f800700 1 -- 172.21.15.62:0/3451486929 wait complete. 2024-04-20T22:10:04.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:04.399+0000 7fab1f800700 1 Processor -- start 2024-04-20T22:10:04.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:04.399+0000 7fab1f800700 1 -- start start 2024-04-20T22:10:04.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:04.400+0000 7fab1f800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fab18102470 0x7fab181981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:10:04.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:04.400+0000 7fab1f800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fab18108470 0x7fab18198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:10:04.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:04.400+0000 7fab1f800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab18198d80 con 0x7fab18108470 2024-04-20T22:10:04.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:04.400+0000 7fab1f800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab18198ec0 con 0x7fab18102470 2024-04-20T22:10:04.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:04.400+0000 7fab0fe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fab18108470 0x7fab18198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:10:04.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:04.400+0000 7fab0fe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fab18108470 0x7fab18198730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37748/0 (socket says 172.21.15.62:37748) 2024-04-20T22:10:04.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:04.400+0000 7fab0fe00700 1 -- 172.21.15.62:0/3232733501 learned_addr learned my addr 172.21.15.62:0/3232733501 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:10:04.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:04.400+0000 7fab1ce00700 1 --2- 172.21.15.62:0/3232733501 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fab18102470 0x7fab181981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:10:04.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:04.401+0000 7fab0fe00700 1 -- 172.21.15.62:0/3232733501 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fab18102470 msgr2=0x7fab181981f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:10:04.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:04.401+0000 7fab0fe00700 1 --2- 172.21.15.62:0/3232733501 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fab18102470 0x7fab181981f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:10:04.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:04.401+0000 7fab0fe00700 1 -- 172.21.15.62:0/3232733501 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faafc0095f0 con 0x7fab18108470 2024-04-20T22:10:04.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:04.401+0000 7fab0fe00700 1 --2- 172.21.15.62:0/3232733501 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fab18108470 0x7fab18198730 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7fab0400c930 tx=0x7fab0400ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:10:04.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:04.401+0000 7fab16000700 1 -- 172.21.15.62:0/3232733501 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fab04007ad0 con 0x7fab18108470 2024-04-20T22:10:04.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:04.401+0000 7fab16000700 1 -- 172.21.15.62:0/3232733501 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fab04007c30 con 0x7fab18108470 2024-04-20T22:10:04.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:04.401+0000 7fab1f800700 1 -- 172.21.15.62:0/3232733501 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab1819cd10 con 0x7fab18108470 2024-04-20T22:10:04.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:04.401+0000 7fab16000700 1 -- 172.21.15.62:0/3232733501 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fab04018610 con 0x7fab18108470 2024-04-20T22:10:04.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:04.401+0000 7fab1f800700 1 -- 172.21.15.62:0/3232733501 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab1819d260 con 0x7fab18108470 2024-04-20T22:10:04.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:04.403+0000 7fab1f800700 1 -- 172.21.15.62:0/3232733501 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab1810a9e0 con 0x7fab18108470 2024-04-20T22:10:04.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:04.404+0000 7fab16000700 1 -- 172.21.15.62:0/3232733501 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fab0400f450 con 0x7fab18108470 2024-04-20T22:10:04.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:04.405+0000 7fab16000700 1 --2- 172.21.15.62:0/3232733501 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7faaf406ecc0 0x7faaf4071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:10:04.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:04.405+0000 7fab16000700 1 -- 172.21.15.62:0/3232733501 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fab0408f140 con 0x7fab18108470 2024-04-20T22:10:04.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:04.405+0000 7fab1ce00700 1 --2- 172.21.15.62:0/3232733501 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7faaf406ecc0 0x7faaf4071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:10:04.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:04.406+0000 7fab1ce00700 1 --2- 172.21.15.62:0/3232733501 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7faaf406ecc0 0x7faaf4071170 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7faafc000c00 tx=0x7faafc005fd0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:10:04.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:04.410+0000 7fab16000700 1 -- 172.21.15.62:0/3232733501 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fab04057e50 con 0x7fab18108470 2024-04-20T22:10:04.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:04.748+0000 7fab1f800700 1 -- 172.21.15.62:0/3232733501 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fab18199550 con 0x7fab18108470 2024-04-20T22:10:04.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:04.748+0000 7fab16000700 1 -- 172.21.15.62:0/3232733501 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7fab0405b510 con 0x7fab18108470 2024-04-20T22:10:04.748 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:10:04.748 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T22:10:04.748 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:10:04.748 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:10:04.748 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T22:10:04.748 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:10:04.748 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:10:04.748 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T22:10:04.748 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 6 2024-04-20T22:10:04.748 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:10:04.748 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T22:10:04.749 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 1, 2024-04-20T22:10:04.749 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 3 2024-04-20T22:10:04.749 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:10:04.749 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T22:10:04.749 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 1, 2024-04-20T22:10:04.749 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 13 2024-04-20T22:10:04.749 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T22:10:04.749 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:10:04.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:04.751+0000 7fab1f800700 1 -- 172.21.15.62:0/3232733501 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7faaf406ecc0 msgr2=0x7faaf4071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:10:04.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:04.751+0000 7fab1f800700 1 --2- 172.21.15.62:0/3232733501 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7faaf406ecc0 0x7faaf4071170 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7faafc000c00 tx=0x7faafc005fd0 comp rx=0 tx=0).stop 2024-04-20T22:10:04.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:04.751+0000 7fab1f800700 1 -- 172.21.15.62:0/3232733501 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fab18108470 msgr2=0x7fab18198730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:10:04.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:04.751+0000 7fab1f800700 1 --2- 172.21.15.62:0/3232733501 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fab18108470 0x7fab18198730 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7fab0400c930 tx=0x7fab0400ccf0 comp rx=0 tx=0).stop 2024-04-20T22:10:04.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:04.751+0000 7fab1f800700 1 -- 172.21.15.62:0/3232733501 shutdown_connections 2024-04-20T22:10:04.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:04.751+0000 7fab1f800700 1 --2- 172.21.15.62:0/3232733501 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7faaf406ecc0 0x7faaf4071170 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:10:04.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:04.751+0000 7fab1f800700 1 --2- 172.21.15.62:0/3232733501 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fab18102470 0x7fab181981f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:10:04.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:04.751+0000 7fab1f800700 1 --2- 172.21.15.62:0/3232733501 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fab18108470 0x7fab18198730 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:10:04.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:04.751+0000 7fab1f800700 1 -- 172.21.15.62:0/3232733501 >> 172.21.15.62:0/3232733501 conn(0x7fab180fdf90 msgr2=0x7fab181002c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:10:04.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:04.752+0000 7fab1f800700 1 -- 172.21.15.62:0/3232733501 shutdown_connections 2024-04-20T22:10:04.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:04.752+0000 7fab1f800700 1 -- 172.21.15.62:0/3232733501 wait complete. 2024-04-20T22:10:04.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:04.913+0000 7f3013000700 1 -- 172.21.15.62:0/1499765001 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f300c10c8f0 msgr2=0x7f300c10ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:10:04.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:04.913+0000 7f3013000700 1 --2- 172.21.15.62:0/1499765001 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f300c10c8f0 0x7f300c10ccc0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f2ffc009910 tx=0x7f2ffc009c20 comp rx=0 tx=0).stop 2024-04-20T22:10:04.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:04.914+0000 7f3013000700 1 -- 172.21.15.62:0/1499765001 shutdown_connections 2024-04-20T22:10:04.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:04.914+0000 7f3013000700 1 --2- 172.21.15.62:0/1499765001 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f300c106b80 0x7f300c106ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:10:04.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:04.914+0000 7f3013000700 1 --2- 172.21.15.62:0/1499765001 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f300c10c8f0 0x7f300c10ccc0 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:10:04.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:04.914+0000 7f3013000700 1 -- 172.21.15.62:0/1499765001 >> 172.21.15.62:0/1499765001 conn(0x7f300c076060 msgr2=0x7f300c078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:10:04.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:04.914+0000 7f3013000700 1 -- 172.21.15.62:0/1499765001 shutdown_connections 2024-04-20T22:10:04.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:04.914+0000 7f3013000700 1 -- 172.21.15.62:0/1499765001 wait complete. 2024-04-20T22:10:04.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:04.915+0000 7f3013000700 1 Processor -- start 2024-04-20T22:10:04.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:04.915+0000 7f3013000700 1 -- start start 2024-04-20T22:10:04.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:04.915+0000 7f3013000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f300c106b80 0x7f300c19c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:10:04.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:04.915+0000 7f3013000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f300c10c8f0 0x7f300c19ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:10:04.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:04.915+0000 7f3013000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f300c19d120 con 0x7f300c106b80 2024-04-20T22:10:04.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:04.915+0000 7f3013000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f300c1a0eb0 con 0x7f300c10c8f0 2024-04-20T22:10:04.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:04.916+0000 7f300be00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f300c106b80 0x7f300c19c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:10:04.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:04.916+0000 7f300be00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f300c106b80 0x7f300c19c500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37758/0 (socket says 172.21.15.62:37758) 2024-04-20T22:10:04.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:04.916+0000 7f300be00700 1 -- 172.21.15.62:0/518372735 learned_addr learned my addr 172.21.15.62:0/518372735 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:10:04.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:04.916+0000 7f300b400700 1 --2- 172.21.15.62:0/518372735 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f300c10c8f0 0x7f300c19ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:10:04.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:04.916+0000 7f300be00700 1 -- 172.21.15.62:0/518372735 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f300c10c8f0 msgr2=0x7f300c19ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:10:04.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:04.916+0000 7f300be00700 1 --2- 172.21.15.62:0/518372735 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f300c10c8f0 0x7f300c19ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:10:04.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:04.916+0000 7f300be00700 1 -- 172.21.15.62:0/518372735 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ffc0095f0 con 0x7f300c106b80 2024-04-20T22:10:04.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:04.917+0000 7f300be00700 1 --2- 172.21.15.62:0/518372735 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f300c106b80 0x7f300c19c500 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f2ffc00f690 tx=0x7f2ffc00f770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:10:04.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:04.917+0000 7f3008c00700 1 -- 172.21.15.62:0/518372735 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2ffc01d070 con 0x7f300c106b80 2024-04-20T22:10:04.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:04.917+0000 7f3008c00700 1 -- 172.21.15.62:0/518372735 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2ffc0050d0 con 0x7f300c106b80 2024-04-20T22:10:04.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:04.917+0000 7f3008c00700 1 -- 172.21.15.62:0/518372735 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2ffc022470 con 0x7f300c106b80 2024-04-20T22:10:04.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:04.917+0000 7f3013000700 1 -- 172.21.15.62:0/518372735 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f300c1a1130 con 0x7f300c106b80 2024-04-20T22:10:04.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:04.917+0000 7f3013000700 1 -- 172.21.15.62:0/518372735 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f300c1a1620 con 0x7f300c106b80 2024-04-20T22:10:04.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:04.918+0000 7f3013000700 1 -- 172.21.15.62:0/518372735 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f300c109b20 con 0x7f300c106b80 2024-04-20T22:10:04.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:04.919+0000 7f3008c00700 1 -- 172.21.15.62:0/518372735 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f2ffc003680 con 0x7f300c106b80 2024-04-20T22:10:04.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:04.920+0000 7f3008c00700 1 --2- 172.21.15.62:0/518372735 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2ff006ecc0 0x7f2ff0071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:10:04.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:04.920+0000 7f3008c00700 1 -- 172.21.15.62:0/518372735 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f2ffc0903d0 con 0x7f300c106b80 2024-04-20T22:10:04.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:04.921+0000 7f300b400700 1 --2- 172.21.15.62:0/518372735 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2ff006ecc0 0x7f2ff0071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:10:04.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:04.921+0000 7f300b400700 1 --2- 172.21.15.62:0/518372735 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2ff006ecc0 0x7f2ff0071170 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f300c19db20 tx=0x7f2ff8009260 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:10:04.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:04.925+0000 7f3008c00700 1 -- 172.21.15.62:0/518372735 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f2ffc0590e0 con 0x7f300c106b80 2024-04-20T22:10:05.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:05.221+0000 7f3013000700 1 -- 172.21.15.62:0/518372735 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f300c19d960 con 0x7f300c106b80 2024-04-20T22:10:05.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:05.222+0000 7f3008c00700 1 -- 172.21.15.62:0/518372735 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 27 v27) v1 ==== 76+0+1888 (secure 0 0 0) 0x7f2ffc05c7a0 con 0x7f300c106b80 2024-04-20T22:10:05.221 INFO:teuthology.orchestra.run.smithi062.stdout:e27 2024-04-20T22:10:05.221 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T22:10:05.221 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:10:05.221 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T22:10:05.221 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:10:05.221 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T22:10:05.221 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T22:10:05.221 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 20 2024-04-20T22:10:05.221 INFO:teuthology.orchestra.run.smithi062.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-20T22:10:05.221 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T22:10:05.221 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T21:42:01.559061+0000 2024-04-20T22:10:05.221 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T22:10:05.221 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T22:10:05.222 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T22:10:05.222 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T22:10:05.222 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T22:10:05.222 INFO:teuthology.orchestra.run.smithi062.stdout:max_xattr_size 65536 2024-04-20T22:10:05.222 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T22:10:05.222 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T22:10:05.222 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 81 2024-04-20T22:10:05.222 INFO:teuthology.orchestra.run.smithi062.stdout:snap_rstat 1 2024-04-20T22:10:05.222 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:10:05.223 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 2 2024-04-20T22:10:05.223 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1 2024-04-20T22:10:05.223 INFO:teuthology.orchestra.run.smithi062.stdout:up {} 2024-04-20T22:10:05.223 INFO:teuthology.orchestra.run.smithi062.stdout:failed 0,1 2024-04-20T22:10:05.223 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T22:10:05.223 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T22:10:05.223 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T22:10:05.224 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T22:10:05.224 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data enabled 2024-04-20T22:10:05.224 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T22:10:05.224 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T22:10:05.224 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 1 2024-04-20T22:10:05.224 INFO:teuthology.orchestra.run.smithi062.stdout:qdb_cluster leader: 0 members: 2024-04-20T22:10:05.224 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:10:05.225 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:10:05.225 INFO:teuthology.orchestra.run.smithi062.stdout:Standby daemons: 2024-04-20T22:10:05.225 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:10:05.225 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ryqblb{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6826/4113916205,v1:172.21.15.62:6827/4113916205] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:10:05.225 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ozizwi{-1:34476} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6828/3801975165,v1:172.21.15.62:6829/3801975165] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:10:05.225 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.xbgspw{-1:44285} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6826/3996040526,v1:172.21.15.189:6827/3996040526] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T22:10:05.226 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.dvimer{-1:44727} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6824/3576699232,v1:172.21.15.189:6825/3576699232] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:10:05.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:05.224+0000 7f3013000700 1 -- 172.21.15.62:0/518372735 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2ff006ecc0 msgr2=0x7f2ff0071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:10:05.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:05.224+0000 7f3013000700 1 --2- 172.21.15.62:0/518372735 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2ff006ecc0 0x7f2ff0071170 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f300c19db20 tx=0x7f2ff8009260 comp rx=0 tx=0).stop 2024-04-20T22:10:05.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:05.225+0000 7f3013000700 1 -- 172.21.15.62:0/518372735 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f300c106b80 msgr2=0x7f300c19c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:10:05.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:05.225+0000 7f3013000700 1 --2- 172.21.15.62:0/518372735 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f300c106b80 0x7f300c19c500 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f2ffc00f690 tx=0x7f2ffc00f770 comp rx=0 tx=0).stop 2024-04-20T22:10:05.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:05.225+0000 7f3013000700 1 -- 172.21.15.62:0/518372735 shutdown_connections 2024-04-20T22:10:05.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:05.225+0000 7f3013000700 1 --2- 172.21.15.62:0/518372735 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2ff006ecc0 0x7f2ff0071170 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:10:05.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:05.225+0000 7f3013000700 1 --2- 172.21.15.62:0/518372735 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f300c106b80 0x7f300c19c500 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:10:05.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:05.225+0000 7f3013000700 1 --2- 172.21.15.62:0/518372735 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f300c10c8f0 0x7f300c19ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:10:05.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:05.225+0000 7f3013000700 1 -- 172.21.15.62:0/518372735 >> 172.21.15.62:0/518372735 conn(0x7f300c076060 msgr2=0x7f300c077aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:10:05.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:05.225+0000 7f3013000700 1 -- 172.21.15.62:0/518372735 shutdown_connections 2024-04-20T22:10:05.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:05.225+0000 7f3013000700 1 -- 172.21.15.62:0/518372735 wait complete. 2024-04-20T22:10:05.229 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 27 2024-04-20T22:10:05.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:05.384+0000 7fa6ecc00700 1 -- 172.21.15.62:0/2826707758 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa6e81024c0 msgr2=0x7fa6e8102890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:10:05.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:05.384+0000 7fa6ecc00700 1 --2- 172.21.15.62:0/2826707758 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa6e81024c0 0x7fa6e8102890 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7fa6d8009910 tx=0x7fa6d8009c20 comp rx=0 tx=0).stop 2024-04-20T22:10:05.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:05.385+0000 7fa6ecc00700 1 -- 172.21.15.62:0/2826707758 shutdown_connections 2024-04-20T22:10:05.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:05.385+0000 7fa6ecc00700 1 --2- 172.21.15.62:0/2826707758 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa6e8102e60 0x7fa6e8106e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:10:05.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:05.385+0000 7fa6ecc00700 1 --2- 172.21.15.62:0/2826707758 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa6e81024c0 0x7fa6e8102890 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:10:05.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:05.385+0000 7fa6ecc00700 1 -- 172.21.15.62:0/2826707758 >> 172.21.15.62:0/2826707758 conn(0x7fa6e80fdd20 msgr2=0x7fa6e8100130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:10:05.384 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:10:05 smithi062 ceph-mon[76322]: from='client.35852 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:10:05.384 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:10:05 smithi062 ceph-mon[76322]: pgmap v1044: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:10:05.384 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:10:05 smithi062 ceph-mon[76322]: from='client.35856 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:10:05.385 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:10:05 smithi062 ceph-mon[76322]: from='client.35860 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:10:05.385 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:10:05 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:10:05.385 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:10:05 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/3232733501' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:10:05.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:05.385+0000 7fa6ecc00700 1 -- 172.21.15.62:0/2826707758 shutdown_connections 2024-04-20T22:10:05.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:05.385+0000 7fa6ecc00700 1 -- 172.21.15.62:0/2826707758 wait complete. 2024-04-20T22:10:05.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:05.387+0000 7fa6ecc00700 1 Processor -- start 2024-04-20T22:10:05.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:05.387+0000 7fa6ecc00700 1 -- start start 2024-04-20T22:10:05.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:05.387+0000 7fa6ecc00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa6e81024c0 0x7fa6e8197e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:10:05.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:05.387+0000 7fa6ecc00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa6e8102e60 0x7fa6e8198350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:10:05.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:05.387+0000 7fa6ecc00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e8198a30 con 0x7fa6e81024c0 2024-04-20T22:10:05.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:05.387+0000 7fa6ecc00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e819c7c0 con 0x7fa6e8102e60 2024-04-20T22:10:05.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:05.388+0000 7fa6e7400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa6e81024c0 0x7fa6e8197e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:10:05.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:05.388+0000 7fa6e7400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa6e81024c0 0x7fa6e8197e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37786/0 (socket says 172.21.15.62:37786) 2024-04-20T22:10:05.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:05.388+0000 7fa6e7400700 1 -- 172.21.15.62:0/407909011 learned_addr learned my addr 172.21.15.62:0/407909011 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:10:05.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:05.388+0000 7fa6e6a00700 1 --2- 172.21.15.62:0/407909011 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa6e8102e60 0x7fa6e8198350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:10:05.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:05.388+0000 7fa6e7400700 1 -- 172.21.15.62:0/407909011 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa6e8102e60 msgr2=0x7fa6e8198350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:10:05.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:05.388+0000 7fa6e7400700 1 --2- 172.21.15.62:0/407909011 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa6e8102e60 0x7fa6e8198350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:10:05.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:05.388+0000 7fa6e7400700 1 -- 172.21.15.62:0/407909011 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6d80095f0 con 0x7fa6e81024c0 2024-04-20T22:10:05.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:05.389+0000 7fa6e7400700 1 --2- 172.21.15.62:0/407909011 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa6e81024c0 0x7fa6e8197e10 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fa6d800b5c0 tx=0x7fa6d800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:10:05.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:05.389+0000 7fa6dfe00700 1 -- 172.21.15.62:0/407909011 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa6d801d070 con 0x7fa6e81024c0 2024-04-20T22:10:05.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:05.389+0000 7fa6ecc00700 1 -- 172.21.15.62:0/407909011 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6e819ca40 con 0x7fa6e81024c0 2024-04-20T22:10:05.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:05.389+0000 7fa6dfe00700 1 -- 172.21.15.62:0/407909011 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa6d8005e00 con 0x7fa6e81024c0 2024-04-20T22:10:05.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:05.389+0000 7fa6ecc00700 1 -- 172.21.15.62:0/407909011 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6e819cf30 con 0x7fa6e81024c0 2024-04-20T22:10:05.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:05.390+0000 7fa6dfe00700 1 -- 172.21.15.62:0/407909011 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa6d8017400 con 0x7fa6e81024c0 2024-04-20T22:10:05.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:05.391+0000 7fa6ecc00700 1 -- 172.21.15.62:0/407909011 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6c8005320 con 0x7fa6e81024c0 2024-04-20T22:10:05.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:05.392+0000 7fa6dfe00700 1 -- 172.21.15.62:0/407909011 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fa6d8003a60 con 0x7fa6e81024c0 2024-04-20T22:10:05.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:05.393+0000 7fa6dfe00700 1 --2- 172.21.15.62:0/407909011 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa6cc06ecc0 0x7fa6cc071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:10:05.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:05.393+0000 7fa6dfe00700 1 -- 172.21.15.62:0/407909011 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fa6d8090150 con 0x7fa6e81024c0 2024-04-20T22:10:05.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:05.394+0000 7fa6e6a00700 1 --2- 172.21.15.62:0/407909011 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa6cc06ecc0 0x7fa6cc071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:10:05.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:05.394+0000 7fa6e6a00700 1 --2- 172.21.15.62:0/407909011 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa6cc06ecc0 0x7fa6cc071170 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fa6e8199430 tx=0x7fa6d400b410 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:10:05.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:05.398+0000 7fa6dfe00700 1 -- 172.21.15.62:0/407909011 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fa6d8058f80 con 0x7fa6e81024c0 2024-04-20T22:10:05.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:10:05 smithi189 ceph-mon[57022]: from='client.35852 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:10:05.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:10:05 smithi189 ceph-mon[57022]: pgmap v1044: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:10:05.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:10:05 smithi189 ceph-mon[57022]: from='client.35856 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:10:05.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:10:05 smithi189 ceph-mon[57022]: from='client.35860 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:10:05.489 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:10:05 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:10:05.489 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:10:05 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/3232733501' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:10:05.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:05.641+0000 7fa6ecc00700 1 -- 172.21.15.62:0/407909011 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa6c8000bf0 con 0x7fa6cc06ecc0 2024-04-20T22:10:05.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:05.644+0000 7fa6dfe00700 1 -- 172.21.15.62:0/407909011 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fa6c8000bf0 con 0x7fa6cc06ecc0 2024-04-20T22:10:05.643 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:10:05.643 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T22:10:05.643 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T22:10:05.644 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-20T22:10:05.644 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T22:10:05.644 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr", 2024-04-20T22:10:05.644 INFO:teuthology.orchestra.run.smithi062.stdout: "crash", 2024-04-20T22:10:05.644 INFO:teuthology.orchestra.run.smithi062.stdout: "mon", 2024-04-20T22:10:05.644 INFO:teuthology.orchestra.run.smithi062.stdout: "osd" 2024-04-20T22:10:05.644 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T22:10:05.644 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "15/23 daemons upgraded", 2024-04-20T22:10:05.644 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading mds daemons", 2024-04-20T22:10:05.644 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T22:10:05.644 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:10:05.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:05.647+0000 7fa6ecc00700 1 -- 172.21.15.62:0/407909011 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa6cc06ecc0 msgr2=0x7fa6cc071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:10:05.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:05.647+0000 7fa6ecc00700 1 --2- 172.21.15.62:0/407909011 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa6cc06ecc0 0x7fa6cc071170 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fa6e8199430 tx=0x7fa6d400b410 comp rx=0 tx=0).stop 2024-04-20T22:10:05.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:05.647+0000 7fa6ecc00700 1 -- 172.21.15.62:0/407909011 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa6e81024c0 msgr2=0x7fa6e8197e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:10:05.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:05.647+0000 7fa6ecc00700 1 --2- 172.21.15.62:0/407909011 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa6e81024c0 0x7fa6e8197e10 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fa6d800b5c0 tx=0x7fa6d800bd70 comp rx=0 tx=0).stop 2024-04-20T22:10:05.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:05.647+0000 7fa6ecc00700 1 -- 172.21.15.62:0/407909011 shutdown_connections 2024-04-20T22:10:05.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:05.647+0000 7fa6ecc00700 1 --2- 172.21.15.62:0/407909011 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa6cc06ecc0 0x7fa6cc071170 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:10:05.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:05.647+0000 7fa6ecc00700 1 --2- 172.21.15.62:0/407909011 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa6e81024c0 0x7fa6e8197e10 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:10:05.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:05.647+0000 7fa6ecc00700 1 --2- 172.21.15.62:0/407909011 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa6e8102e60 0x7fa6e8198350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:10:05.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:05.647+0000 7fa6ecc00700 1 -- 172.21.15.62:0/407909011 >> 172.21.15.62:0/407909011 conn(0x7fa6e80fdd20 msgr2=0x7fa6e80ff3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:10:05.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:05.648+0000 7fa6ecc00700 1 -- 172.21.15.62:0/407909011 shutdown_connections 2024-04-20T22:10:05.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:05.648+0000 7fa6ecc00700 1 -- 172.21.15.62:0/407909011 wait complete. 2024-04-20T22:10:05.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:05.809+0000 7fc8e3a00700 1 -- 172.21.15.62:0/364312773 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8dc106b80 msgr2=0x7fc8dc106ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:10:05.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:05.809+0000 7fc8e3a00700 1 --2- 172.21.15.62:0/364312773 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8dc106b80 0x7fc8dc106ff0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fc8c8009910 tx=0x7fc8c8009c20 comp rx=0 tx=0).stop 2024-04-20T22:10:05.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:05.809+0000 7fc8e3a00700 1 -- 172.21.15.62:0/364312773 shutdown_connections 2024-04-20T22:10:05.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:05.809+0000 7fc8e3a00700 1 --2- 172.21.15.62:0/364312773 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8dc106b80 0x7fc8dc106ff0 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:10:05.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:05.809+0000 7fc8e3a00700 1 --2- 172.21.15.62:0/364312773 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc8dc10c8f0 0x7fc8dc10ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:10:05.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:05.809+0000 7fc8e3a00700 1 -- 172.21.15.62:0/364312773 >> 172.21.15.62:0/364312773 conn(0x7fc8dc076060 msgr2=0x7fc8dc078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:10:05.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:05.809+0000 7fc8e3a00700 1 -- 172.21.15.62:0/364312773 shutdown_connections 2024-04-20T22:10:05.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:05.809+0000 7fc8e3a00700 1 -- 172.21.15.62:0/364312773 wait complete. 2024-04-20T22:10:05.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:05.810+0000 7fc8e3a00700 1 Processor -- start 2024-04-20T22:10:05.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:05.810+0000 7fc8e3a00700 1 -- start start 2024-04-20T22:10:05.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:05.811+0000 7fc8e3a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8dc106b80 0x7fc8dc19c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:10:05.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:05.811+0000 7fc8e3a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc8dc10c8f0 0x7fc8dc19c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:10:05.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:05.811+0000 7fc8e3a00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8dc19cfd0 con 0x7fc8dc106b80 2024-04-20T22:10:05.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:05.811+0000 7fc8e3a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8dc19d110 con 0x7fc8dc10c8f0 2024-04-20T22:10:05.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:05.811+0000 7fc8e1000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8dc106b80 0x7fc8dc19c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:10:05.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:05.811+0000 7fc8e1000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8dc106b80 0x7fc8dc19c440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37806/0 (socket says 172.21.15.62:37806) 2024-04-20T22:10:05.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:05.811+0000 7fc8e1000700 1 -- 172.21.15.62:0/920354143 learned_addr learned my addr 172.21.15.62:0/920354143 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:10:05.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:05.811+0000 7fc8dbe00700 1 --2- 172.21.15.62:0/920354143 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc8dc10c8f0 0x7fc8dc19c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:10:05.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:05.811+0000 7fc8e1000700 1 -- 172.21.15.62:0/920354143 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc8dc10c8f0 msgr2=0x7fc8dc19c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:10:05.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:05.811+0000 7fc8e1000700 1 --2- 172.21.15.62:0/920354143 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc8dc10c8f0 0x7fc8dc19c980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:10:05.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:05.811+0000 7fc8e1000700 1 -- 172.21.15.62:0/920354143 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8c80095f0 con 0x7fc8dc106b80 2024-04-20T22:10:05.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:05.812+0000 7fc8e1000700 1 --2- 172.21.15.62:0/920354143 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8dc106b80 0x7fc8dc19c440 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fc8d000c930 tx=0x7fc8d000ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:10:05.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:05.812+0000 7fc8d9600700 1 -- 172.21.15.62:0/920354143 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc8d0007ad0 con 0x7fc8dc106b80 2024-04-20T22:10:05.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:05.812+0000 7fc8e3a00700 1 -- 172.21.15.62:0/920354143 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8dc1a0f60 con 0x7fc8dc106b80 2024-04-20T22:10:05.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:05.812+0000 7fc8d9600700 1 -- 172.21.15.62:0/920354143 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc8d0007c30 con 0x7fc8dc106b80 2024-04-20T22:10:05.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:05.812+0000 7fc8d9600700 1 -- 172.21.15.62:0/920354143 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc8d0018610 con 0x7fc8dc106b80 2024-04-20T22:10:05.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:05.812+0000 7fc8e3a00700 1 -- 172.21.15.62:0/920354143 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8dc1a14b0 con 0x7fc8dc106b80 2024-04-20T22:10:05.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:05.813+0000 7fc8e3a00700 1 -- 172.21.15.62:0/920354143 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8dc10ecc0 con 0x7fc8dc106b80 2024-04-20T22:10:05.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:05.814+0000 7fc8d9600700 1 -- 172.21.15.62:0/920354143 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fc8d000f450 con 0x7fc8dc106b80 2024-04-20T22:10:05.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:05.815+0000 7fc8d9600700 1 --2- 172.21.15.62:0/920354143 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc8c406ecc0 0x7fc8c4071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:10:05.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:05.815+0000 7fc8d9600700 1 -- 172.21.15.62:0/920354143 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fc8d008f120 con 0x7fc8dc106b80 2024-04-20T22:10:05.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:05.815+0000 7fc8dbe00700 1 --2- 172.21.15.62:0/920354143 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc8c406ecc0 0x7fc8c4071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:10:05.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:05.816+0000 7fc8dbe00700 1 --2- 172.21.15.62:0/920354143 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc8c406ecc0 0x7fc8c4071170 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fc8c8006010 tx=0x7fc8c800b540 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:10:05.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:05.820+0000 7fc8d9600700 1 -- 172.21.15.62:0/920354143 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fc8d0057e30 con 0x7fc8dc106b80 2024-04-20T22:10:06.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:06.211+0000 7fc8e3a00700 1 -- 172.21.15.62:0/920354143 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fc8dc10e6a0 con 0x7fc8dc106b80 2024-04-20T22:10:06.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:06.212+0000 7fc8d9600700 1 -- 172.21.15.62:0/920354143 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7fc8d005b4f0 con 0x7fc8dc106b80 2024-04-20T22:10:06.212 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-20T22:10:06.212 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-20T22:10:06.212 INFO:teuthology.orchestra.run.smithi062.stdout: host smithi062 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T22:10:06.212 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-20T22:10:06.212 INFO:teuthology.orchestra.run.smithi062.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi062 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpq7y7oa_r:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:10:06.213 INFO:teuthology.orchestra.run.smithi062.stdout:Traceback (most recent call last): 2024-04-20T22:10:06.213 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-20T22:10:06.213 INFO:teuthology.orchestra.run.smithi062.stdout: return _run_code(code, main_globals, None, 2024-04-20T22:10:06.213 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-20T22:10:06.213 INFO:teuthology.orchestra.run.smithi062.stdout: exec(code, run_globals) 2024-04-20T22:10:06.213 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5618, in 2024-04-20T22:10:06.213 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5606, in main 2024-04-20T22:10:06.213 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 408, in _infer_config 2024-04-20T22:10:06.213 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 323, in _infer_fsid 2024-04-20T22:10:06.213 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 436, in _infer_image 2024-04-20T22:10:06.213 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 310, in _validate_fsid 2024-04-20T22:10:06.213 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 3359, in command_ceph_volume 2024-04-20T22:10:06.214 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-20T22:10:06.214 INFO:teuthology.orchestra.run.smithi062.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi062 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpq7y7oa_r:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:10:06.214 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-20T22:10:06.214 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is degraded 2024-04-20T22:10:06.214 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T22:10:06.214 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T22:10:06.214 INFO:teuthology.orchestra.run.smithi062.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-20T22:10:06.214 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-20T22:10:06.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:06.215+0000 7fc8e3a00700 1 -- 172.21.15.62:0/920354143 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc8c406ecc0 msgr2=0x7fc8c4071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:10:06.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:06.215+0000 7fc8e3a00700 1 --2- 172.21.15.62:0/920354143 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc8c406ecc0 0x7fc8c4071170 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fc8c8006010 tx=0x7fc8c800b540 comp rx=0 tx=0).stop 2024-04-20T22:10:06.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:06.215+0000 7fc8e3a00700 1 -- 172.21.15.62:0/920354143 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8dc106b80 msgr2=0x7fc8dc19c440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:10:06.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:06.215+0000 7fc8e3a00700 1 --2- 172.21.15.62:0/920354143 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8dc106b80 0x7fc8dc19c440 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fc8d000c930 tx=0x7fc8d000ccf0 comp rx=0 tx=0).stop 2024-04-20T22:10:06.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:06.216+0000 7fc8e3a00700 1 -- 172.21.15.62:0/920354143 shutdown_connections 2024-04-20T22:10:06.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:06.216+0000 7fc8e3a00700 1 --2- 172.21.15.62:0/920354143 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc8c406ecc0 0x7fc8c4071170 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:10:06.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:06.216+0000 7fc8e3a00700 1 --2- 172.21.15.62:0/920354143 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8dc106b80 0x7fc8dc19c440 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:10:06.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:06.216+0000 7fc8e3a00700 1 --2- 172.21.15.62:0/920354143 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc8dc10c8f0 0x7fc8dc19c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:10:06.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:06.216+0000 7fc8e3a00700 1 -- 172.21.15.62:0/920354143 >> 172.21.15.62:0/920354143 conn(0x7fc8dc076060 msgr2=0x7fc8dc077ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:10:06.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:06.216+0000 7fc8e3a00700 1 -- 172.21.15.62:0/920354143 shutdown_connections 2024-04-20T22:10:06.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:06.216+0000 7fc8e3a00700 1 -- 172.21.15.62:0/920354143 wait complete. 2024-04-20T22:10:06.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:10:06 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/518372735' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:10:06.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:10:06 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/518372735' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:10:07.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:10:07 smithi189 ceph-mon[57022]: from='client.35872 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:10:07.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:10:07 smithi189 ceph-mon[57022]: pgmap v1045: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:10:07.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:10:07 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/920354143' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:10:07.538 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:10:07 smithi062 ceph-mon[76322]: from='client.35872 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:10:07.538 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:10:07 smithi062 ceph-mon[76322]: pgmap v1045: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:10:07.538 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:10:07 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/920354143' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:10:09.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:10:09 smithi189 ceph-mon[57022]: pgmap v1046: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:10:09.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:10:09 smithi062 ceph-mon[76322]: pgmap v1046: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:10:11.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:10:11 smithi189 ceph-mon[57022]: pgmap v1047: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:10:11.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:10:11 smithi062 ceph-mon[76322]: pgmap v1047: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:10:13.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:10:13 smithi189 ceph-mon[57022]: pgmap v1048: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:10:13.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:10:13 smithi062 ceph-mon[76322]: pgmap v1048: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:10:15.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:10:15 smithi189 ceph-mon[57022]: pgmap v1049: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:10:15.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:10:15 smithi062 ceph-mon[76322]: pgmap v1049: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:10:17.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:10:17 smithi189 ceph-mon[57022]: pgmap v1050: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:10:17.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:10:17 smithi062 ceph-mon[76322]: pgmap v1050: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:10:19.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:10:19 smithi189 ceph-mon[57022]: pgmap v1051: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:10:19.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:10:19 smithi062 ceph-mon[76322]: pgmap v1051: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:10:20.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:10:20 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:10:20.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:10:20 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:10:21.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:10:21 smithi189 ceph-mon[57022]: pgmap v1052: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:10:21.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:10:21 smithi062 ceph-mon[76322]: pgmap v1052: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:10:23.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:10:23 smithi189 ceph-mon[57022]: pgmap v1053: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:10:23.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:10:23 smithi062 ceph-mon[76322]: pgmap v1053: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:10:25.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:10:25 smithi189 ceph-mon[57022]: pgmap v1054: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:10:25.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:10:25 smithi062 ceph-mon[76322]: pgmap v1054: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:10:27.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:10:27 smithi189 ceph-mon[57022]: pgmap v1055: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:10:27.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:10:27 smithi062 ceph-mon[76322]: pgmap v1055: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:10:29.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:10:29 smithi189 ceph-mon[57022]: pgmap v1056: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:10:29.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:10:29 smithi062 ceph-mon[76322]: pgmap v1056: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:10:31.443 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:10:31 smithi062 ceph-mon[76322]: pgmap v1057: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:10:31.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:10:31 smithi189 ceph-mon[57022]: pgmap v1057: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:10:33.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:10:33 smithi189 ceph-mon[57022]: pgmap v1058: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:10:33.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:10:33 smithi062 ceph-mon[76322]: pgmap v1058: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:10:35.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:10:35 smithi189 ceph-mon[57022]: pgmap v1059: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:10:35.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:10:35 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:10:35.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:10:35 smithi062 ceph-mon[76322]: pgmap v1059: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:10:35.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:10:35 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:10:36.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:36.395+0000 7f638e400700 1 -- 172.21.15.62:0/2261962826 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6388108470 msgr2=0x7f6388108840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:10:36.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:36.395+0000 7f638e400700 1 --2- 172.21.15.62:0/2261962826 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6388108470 0x7f6388108840 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f6378009910 tx=0x7f6378009c20 comp rx=0 tx=0).stop 2024-04-20T22:10:36.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:36.395+0000 7f638e400700 1 -- 172.21.15.62:0/2261962826 shutdown_connections 2024-04-20T22:10:36.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:36.395+0000 7f638e400700 1 --2- 172.21.15.62:0/2261962826 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6388102470 0x7f63881028e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:10:36.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:36.395+0000 7f638e400700 1 --2- 172.21.15.62:0/2261962826 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6388108470 0x7f6388108840 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:10:36.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:36.395+0000 7f638e400700 1 -- 172.21.15.62:0/2261962826 >> 172.21.15.62:0/2261962826 conn(0x7f63880fdf90 msgr2=0x7f63881003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:10:36.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:36.396+0000 7f638e400700 1 -- 172.21.15.62:0/2261962826 shutdown_connections 2024-04-20T22:10:36.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:36.396+0000 7f638e400700 1 -- 172.21.15.62:0/2261962826 wait complete. 2024-04-20T22:10:36.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:36.397+0000 7f638e400700 1 Processor -- start 2024-04-20T22:10:36.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:36.397+0000 7f638e400700 1 -- start start 2024-04-20T22:10:36.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:36.397+0000 7f638e400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6388102470 0x7f63881981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:10:36.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:36.397+0000 7f638e400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6388108470 0x7f6388198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:10:36.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:36.397+0000 7f638e400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6388198e10 con 0x7f6388108470 2024-04-20T22:10:36.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:36.397+0000 7f638e400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f638819cb30 con 0x7f6388102470 2024-04-20T22:10:36.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:36.398+0000 7f6386a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6388108470 0x7f6388198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:10:36.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:36.398+0000 7f6386a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6388108470 0x7f6388198730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45738/0 (socket says 172.21.15.62:45738) 2024-04-20T22:10:36.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:36.398+0000 7f6386a00700 1 -- 172.21.15.62:0/1285994926 learned_addr learned my addr 172.21.15.62:0/1285994926 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:10:36.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:36.398+0000 7f6387400700 1 --2- 172.21.15.62:0/1285994926 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6388102470 0x7f63881981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:10:36.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:36.398+0000 7f6386a00700 1 -- 172.21.15.62:0/1285994926 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6388102470 msgr2=0x7f63881981f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:10:36.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:36.398+0000 7f6386a00700 1 --2- 172.21.15.62:0/1285994926 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6388102470 0x7f63881981f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:10:36.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:36.398+0000 7f6386a00700 1 -- 172.21.15.62:0/1285994926 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63780095f0 con 0x7f6388108470 2024-04-20T22:10:36.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:36.399+0000 7f6386a00700 1 --2- 172.21.15.62:0/1285994926 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6388108470 0x7f6388198730 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f637400c8a0 tx=0x7f637400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:10:36.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:36.399+0000 7f637fe00700 1 -- 172.21.15.62:0/1285994926 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f637400cea0 con 0x7f6388108470 2024-04-20T22:10:36.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:36.399+0000 7f637fe00700 1 -- 172.21.15.62:0/1285994926 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6374004830 con 0x7f6388108470 2024-04-20T22:10:36.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:36.399+0000 7f638e400700 1 -- 172.21.15.62:0/1285994926 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f638819ce10 con 0x7f6388108470 2024-04-20T22:10:36.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:36.399+0000 7f637fe00700 1 -- 172.21.15.62:0/1285994926 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f63740055b0 con 0x7f6388108470 2024-04-20T22:10:36.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:36.399+0000 7f638e400700 1 -- 172.21.15.62:0/1285994926 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f638819d360 con 0x7f6388108470 2024-04-20T22:10:36.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:36.400+0000 7f638e400700 1 -- 172.21.15.62:0/1285994926 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f638810a9e0 con 0x7f6388108470 2024-04-20T22:10:36.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:36.401+0000 7f637fe00700 1 -- 172.21.15.62:0/1285994926 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f63740049a0 con 0x7f6388108470 2024-04-20T22:10:36.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:36.402+0000 7f637fe00700 1 --2- 172.21.15.62:0/1285994926 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f636c06ecc0 0x7f636c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:10:36.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:36.403+0000 7f637fe00700 1 -- 172.21.15.62:0/1285994926 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f637408ebf0 con 0x7f6388108470 2024-04-20T22:10:36.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:36.403+0000 7f6387400700 1 --2- 172.21.15.62:0/1285994926 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f636c06ecc0 0x7f636c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:10:36.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:36.403+0000 7f6387400700 1 --2- 172.21.15.62:0/1285994926 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f636c06ecc0 0x7f636c071170 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f637800b5c0 tx=0x7f637801a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:10:36.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:36.408+0000 7f637fe00700 1 -- 172.21.15.62:0/1285994926 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f6374057900 con 0x7f6388108470 2024-04-20T22:10:36.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:36.682+0000 7f638e400700 1 -- 172.21.15.62:0/1285994926 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f63881995c0 con 0x7f636c06ecc0 2024-04-20T22:10:36.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:36.685+0000 7f637fe00700 1 -- 172.21.15.62:0/1285994926 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f63881995c0 con 0x7f636c06ecc0 2024-04-20T22:10:36.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:36.688+0000 7f638e400700 1 -- 172.21.15.62:0/1285994926 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f636c06ecc0 msgr2=0x7f636c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:10:36.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:36.688+0000 7f638e400700 1 --2- 172.21.15.62:0/1285994926 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f636c06ecc0 0x7f636c071170 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f637800b5c0 tx=0x7f637801a040 comp rx=0 tx=0).stop 2024-04-20T22:10:36.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:36.688+0000 7f638e400700 1 -- 172.21.15.62:0/1285994926 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6388108470 msgr2=0x7f6388198730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:10:36.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:36.688+0000 7f638e400700 1 --2- 172.21.15.62:0/1285994926 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6388108470 0x7f6388198730 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f637400c8a0 tx=0x7f637400cc60 comp rx=0 tx=0).stop 2024-04-20T22:10:36.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:36.688+0000 7f638e400700 1 -- 172.21.15.62:0/1285994926 shutdown_connections 2024-04-20T22:10:36.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:36.688+0000 7f638e400700 1 --2- 172.21.15.62:0/1285994926 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f636c06ecc0 0x7f636c071170 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:10:36.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:36.688+0000 7f638e400700 1 --2- 172.21.15.62:0/1285994926 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6388102470 0x7f63881981f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:10:36.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:36.688+0000 7f638e400700 1 --2- 172.21.15.62:0/1285994926 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6388108470 0x7f6388198730 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:10:36.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:36.688+0000 7f638e400700 1 -- 172.21.15.62:0/1285994926 >> 172.21.15.62:0/1285994926 conn(0x7f63880fdf90 msgr2=0x7f63881002c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:10:36.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:36.688+0000 7f638e400700 1 -- 172.21.15.62:0/1285994926 shutdown_connections 2024-04-20T22:10:36.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:36.688+0000 7f638e400700 1 -- 172.21.15.62:0/1285994926 wait complete. 2024-04-20T22:10:36.705 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T22:10:36.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:36.850+0000 7f6449200700 1 -- 172.21.15.62:0/955766978 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6444102470 msgr2=0x7f64441028e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:10:36.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:36.850+0000 7f6449200700 1 --2- 172.21.15.62:0/955766978 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6444102470 0x7f64441028e0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f6430009910 tx=0x7f6430009c20 comp rx=0 tx=0).stop 2024-04-20T22:10:36.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:36.851+0000 7f6449200700 1 -- 172.21.15.62:0/955766978 shutdown_connections 2024-04-20T22:10:36.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:36.851+0000 7f6449200700 1 --2- 172.21.15.62:0/955766978 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6444102470 0x7f64441028e0 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:10:36.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:36.851+0000 7f6449200700 1 --2- 172.21.15.62:0/955766978 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6444108470 0x7f6444108840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:10:36.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:36.851+0000 7f6449200700 1 -- 172.21.15.62:0/955766978 >> 172.21.15.62:0/955766978 conn(0x7f64440fdf90 msgr2=0x7f64441003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:10:36.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:36.851+0000 7f6449200700 1 -- 172.21.15.62:0/955766978 shutdown_connections 2024-04-20T22:10:36.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:36.851+0000 7f6449200700 1 -- 172.21.15.62:0/955766978 wait complete. 2024-04-20T22:10:36.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:36.852+0000 7f6449200700 1 Processor -- start 2024-04-20T22:10:36.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:36.852+0000 7f6449200700 1 -- start start 2024-04-20T22:10:36.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:36.852+0000 7f6449200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6444102470 0x7f6444198270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:10:36.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:36.852+0000 7f6449200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6444108470 0x7f64441987b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:10:36.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:36.852+0000 7f6449200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6444198e90 con 0x7f6444108470 2024-04-20T22:10:36.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:36.852+0000 7f6449200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f644419cc20 con 0x7f6444102470 2024-04-20T22:10:36.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:36.853+0000 7f6441400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6444108470 0x7f64441987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:10:36.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:36.853+0000 7f6441400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6444108470 0x7f64441987b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45766/0 (socket says 172.21.15.62:45766) 2024-04-20T22:10:36.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:36.853+0000 7f6441400700 1 -- 172.21.15.62:0/3863344589 learned_addr learned my addr 172.21.15.62:0/3863344589 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:10:36.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:36.853+0000 7f6441e00700 1 --2- 172.21.15.62:0/3863344589 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6444102470 0x7f6444198270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:10:36.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:36.853+0000 7f6441400700 1 -- 172.21.15.62:0/3863344589 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6444102470 msgr2=0x7f6444198270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:10:36.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:36.853+0000 7f6441400700 1 --2- 172.21.15.62:0/3863344589 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6444102470 0x7f6444198270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:10:36.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:36.853+0000 7f6441400700 1 -- 172.21.15.62:0/3863344589 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64300095f0 con 0x7f6444108470 2024-04-20T22:10:36.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:36.853+0000 7f6441400700 1 --2- 172.21.15.62:0/3863344589 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6444108470 0x7f64441987b0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f643000b5c0 tx=0x7f643000be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:10:36.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:36.854+0000 7f643aa00700 1 -- 172.21.15.62:0/3863344589 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f643001d070 con 0x7f6444108470 2024-04-20T22:10:36.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:36.854+0000 7f643aa00700 1 -- 172.21.15.62:0/3863344589 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f643000f460 con 0x7f6444108470 2024-04-20T22:10:36.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:36.854+0000 7f6449200700 1 -- 172.21.15.62:0/3863344589 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f644419cea0 con 0x7f6444108470 2024-04-20T22:10:36.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:36.854+0000 7f643aa00700 1 -- 172.21.15.62:0/3863344589 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6430017610 con 0x7f6444108470 2024-04-20T22:10:36.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:36.854+0000 7f6449200700 1 -- 172.21.15.62:0/3863344589 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f644419d390 con 0x7f6444108470 2024-04-20T22:10:36.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:36.855+0000 7f6449200700 1 -- 172.21.15.62:0/3863344589 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f644410a970 con 0x7f6444108470 2024-04-20T22:10:36.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:36.856+0000 7f643aa00700 1 -- 172.21.15.62:0/3863344589 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f643000f5d0 con 0x7f6444108470 2024-04-20T22:10:36.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:36.857+0000 7f643aa00700 1 --2- 172.21.15.62:0/3863344589 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f642806ecc0 0x7f6428071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:10:36.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:36.857+0000 7f643aa00700 1 -- 172.21.15.62:0/3863344589 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f6430090450 con 0x7f6444108470 2024-04-20T22:10:36.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:36.858+0000 7f6441e00700 1 --2- 172.21.15.62:0/3863344589 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f642806ecc0 0x7f6428071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:10:36.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:36.858+0000 7f6441e00700 1 --2- 172.21.15.62:0/3863344589 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f642806ecc0 0x7f6428071170 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f6434005fd0 tx=0x7f643400a560 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:10:36.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:36.862+0000 7f643aa00700 1 -- 172.21.15.62:0/3863344589 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f64300591e0 con 0x7f6444108470 2024-04-20T22:10:37.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:37.125+0000 7f6449200700 1 -- 172.21.15.62:0/3863344589 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f644419d600 con 0x7f642806ecc0 2024-04-20T22:10:37.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:37.127+0000 7f643aa00700 1 -- 172.21.15.62:0/3863344589 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f644419d600 con 0x7f642806ecc0 2024-04-20T22:10:37.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:37.130+0000 7f6449200700 1 -- 172.21.15.62:0/3863344589 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f642806ecc0 msgr2=0x7f6428071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:10:37.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:37.130+0000 7f6449200700 1 --2- 172.21.15.62:0/3863344589 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f642806ecc0 0x7f6428071170 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f6434005fd0 tx=0x7f643400a560 comp rx=0 tx=0).stop 2024-04-20T22:10:37.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:37.130+0000 7f6449200700 1 -- 172.21.15.62:0/3863344589 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6444108470 msgr2=0x7f64441987b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:10:37.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:37.130+0000 7f6449200700 1 --2- 172.21.15.62:0/3863344589 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6444108470 0x7f64441987b0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f643000b5c0 tx=0x7f643000be00 comp rx=0 tx=0).stop 2024-04-20T22:10:37.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:37.131+0000 7f6449200700 1 -- 172.21.15.62:0/3863344589 shutdown_connections 2024-04-20T22:10:37.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:37.131+0000 7f6449200700 1 --2- 172.21.15.62:0/3863344589 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f642806ecc0 0x7f6428071170 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:10:37.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:37.131+0000 7f6449200700 1 --2- 172.21.15.62:0/3863344589 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6444102470 0x7f6444198270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:10:37.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:37.131+0000 7f6449200700 1 --2- 172.21.15.62:0/3863344589 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6444108470 0x7f64441987b0 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:10:37.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:37.131+0000 7f6449200700 1 -- 172.21.15.62:0/3863344589 >> 172.21.15.62:0/3863344589 conn(0x7f64440fdf90 msgr2=0x7f64440ff410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:10:37.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:37.131+0000 7f6449200700 1 -- 172.21.15.62:0/3863344589 shutdown_connections 2024-04-20T22:10:37.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:37.131+0000 7f6449200700 1 -- 172.21.15.62:0/3863344589 wait complete. 2024-04-20T22:10:37.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:37.296+0000 7fee0fc00700 1 -- 172.21.15.62:0/2252714100 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee08068540 msgr2=0x7fee080689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:10:37.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:37.296+0000 7fee0fc00700 1 --2- 172.21.15.62:0/2252714100 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee08068540 0x7fee080689b0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7fedf4009910 tx=0x7fedf4009c20 comp rx=0 tx=0).stop 2024-04-20T22:10:37.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:37.296+0000 7fee0fc00700 1 -- 172.21.15.62:0/2252714100 shutdown_connections 2024-04-20T22:10:37.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:37.296+0000 7fee0fc00700 1 --2- 172.21.15.62:0/2252714100 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee08068540 0x7fee080689b0 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:10:37.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:37.296+0000 7fee0fc00700 1 --2- 172.21.15.62:0/2252714100 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee08106320 0x7fee081066f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:10:37.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:37.296+0000 7fee0fc00700 1 -- 172.21.15.62:0/2252714100 >> 172.21.15.62:0/2252714100 conn(0x7fee08075130 msgr2=0x7fee08075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:10:37.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:37.296+0000 7fee0fc00700 1 -- 172.21.15.62:0/2252714100 shutdown_connections 2024-04-20T22:10:37.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:37.297+0000 7fee0fc00700 1 -- 172.21.15.62:0/2252714100 wait complete. 2024-04-20T22:10:37.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:37.298+0000 7fee0fc00700 1 Processor -- start 2024-04-20T22:10:37.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:37.298+0000 7fee0fc00700 1 -- start start 2024-04-20T22:10:37.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:37.298+0000 7fee0fc00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee08068540 0x7fee08195e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:10:37.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:37.298+0000 7fee0fc00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee08106320 0x7fee081963a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:10:37.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:37.298+0000 7fee0fc00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee081969f0 con 0x7fee08068540 2024-04-20T22:10:37.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:37.298+0000 7fee0fc00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee08196b30 con 0x7fee08106320 2024-04-20T22:10:37.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:37.298+0000 7fee0d200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee08068540 0x7fee08195e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:10:37.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:37.299+0000 7fee0d200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee08068540 0x7fee08195e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45792/0 (socket says 172.21.15.62:45792) 2024-04-20T22:10:37.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:37.299+0000 7fee0d200700 1 -- 172.21.15.62:0/2741328244 learned_addr learned my addr 172.21.15.62:0/2741328244 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:10:37.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:37.299+0000 7fee07e00700 1 --2- 172.21.15.62:0/2741328244 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee08106320 0x7fee081963a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:10:37.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:37.299+0000 7fee0d200700 1 -- 172.21.15.62:0/2741328244 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee08106320 msgr2=0x7fee081963a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:10:37.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:37.299+0000 7fee0d200700 1 --2- 172.21.15.62:0/2741328244 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee08106320 0x7fee081963a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:10:37.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:37.299+0000 7fee0d200700 1 -- 172.21.15.62:0/2741328244 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedf40095f0 con 0x7fee08068540 2024-04-20T22:10:37.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:37.299+0000 7fee0d200700 1 --2- 172.21.15.62:0/2741328244 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee08068540 0x7fee08195e60 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7fedfc00c930 tx=0x7fedfc00ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:10:37.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:37.300+0000 7fee05600700 1 -- 172.21.15.62:0/2741328244 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fedfc007ad0 con 0x7fee08068540 2024-04-20T22:10:37.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:37.300+0000 7fee05600700 1 -- 172.21.15.62:0/2741328244 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fedfc007c30 con 0x7fee08068540 2024-04-20T22:10:37.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:37.300+0000 7fee0fc00700 1 -- 172.21.15.62:0/2741328244 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee0819a980 con 0x7fee08068540 2024-04-20T22:10:37.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:37.300+0000 7fee05600700 1 -- 172.21.15.62:0/2741328244 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fedfc018610 con 0x7fee08068540 2024-04-20T22:10:37.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:37.300+0000 7fee0fc00700 1 -- 172.21.15.62:0/2741328244 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee0819aed0 con 0x7fee08068540 2024-04-20T22:10:37.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:37.301+0000 7fee0fc00700 1 -- 172.21.15.62:0/2741328244 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee08108770 con 0x7fee08068540 2024-04-20T22:10:37.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:37.302+0000 7fee05600700 1 -- 172.21.15.62:0/2741328244 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fedfc00f450 con 0x7fee08068540 2024-04-20T22:10:37.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:37.303+0000 7fee05600700 1 --2- 172.21.15.62:0/2741328244 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fedf006ecc0 0x7fedf0071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:10:37.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:37.303+0000 7fee05600700 1 -- 172.21.15.62:0/2741328244 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fedfc08eff0 con 0x7fee08068540 2024-04-20T22:10:37.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:37.304+0000 7fee07e00700 1 --2- 172.21.15.62:0/2741328244 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fedf006ecc0 0x7fedf0071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:10:37.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:37.304+0000 7fee07e00700 1 --2- 172.21.15.62:0/2741328244 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fedf006ecc0 0x7fedf0071170 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fee08197480 tx=0x7fedf401a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:10:37.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:37.308+0000 7fee05600700 1 -- 172.21.15.62:0/2741328244 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fedfc057de0 con 0x7fee08068540 2024-04-20T22:10:37.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:10:37 smithi189 ceph-mon[57022]: pgmap v1060: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:10:37.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:10:37 smithi062 ceph-mon[76322]: pgmap v1060: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:10:37.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:37.546+0000 7fee0fc00700 1 -- 172.21.15.62:0/2741328244 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fee0819b150 con 0x7fedf006ecc0 2024-04-20T22:10:37.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:37.579+0000 7fee05600700 1 -- 172.21.15.62:0/2741328244 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fee0819b150 con 0x7fedf006ecc0 2024-04-20T22:10:37.580 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T22:10:37.580 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (36m) 12m ago 50m 15.3M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T22:10:37.580 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (50m) 12m ago 50m 10.4M - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T22:10:37.580 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (49m) 12m ago 49m 10.1M - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T22:10:37.580 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (32m) 12m ago 50m 6660k - 19.0.0-3133-ga8c84f59 76bc6b91b726 2825d5ba11c6 2024-04-20T22:10:37.580 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (32m) 12m ago 49m 6665k - 19.0.0-3133-ga8c84f59 76bc6b91b726 20f746fa1bba 2024-04-20T22:10:37.580 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (35m) 12m ago 49m 73.5M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T22:10:37.580 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (27m) 12m ago 43m 18.5M - 19.0.0-3133-ga8c84f59 76bc6b91b726 de80bc90a683 2024-04-20T22:10:37.580 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (28m) 12m ago 43m 18.7M - 19.0.0-3133-ga8c84f59 76bc6b91b726 cd793b7ee98e 2024-04-20T22:10:37.580 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (12m) 12m ago 43m 11.0M - 19.0.0-3133-ga8c84f59 76bc6b91b726 0e7b0bc4d1ab 2024-04-20T22:10:37.580 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (42m) 12m ago 42m 18.6M - 18.2.0 dc2bc1663786 f2eec2ee8951 2024-04-20T22:10:37.580 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (39m) 12m ago 52m 513M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T22:10:37.580 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (38m) 12m ago 49m 459M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T22:10:37.581 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (34m) 12m ago 52m 71.7M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 019eb1479485 2024-04-20T22:10:37.581 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (33m) 12m ago 48m 51.0M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 55aac560aab6 2024-04-20T22:10:37.581 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (37m) 12m ago 50m 8472k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T22:10:37.581 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (37m) 12m ago 49m 8048k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T22:10:37.581 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (31m) 12m ago 47m 91.2M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 ee2bb901f26e 2024-04-20T22:10:37.581 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (31m) 12m ago 47m 94.6M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 9075152d4162 2024-04-20T22:10:37.581 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (30m) 12m ago 46m 78.1M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 2d4f3e38dc0e 2024-04-20T22:10:37.581 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (30m) 12m ago 46m 92.6M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 8f97b17ae0eb 2024-04-20T22:10:37.581 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (29m) 12m ago 45m 95.6M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 653d8079967d 2024-04-20T22:10:37.581 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (28m) 12m ago 45m 89.2M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 596dded14c12 2024-04-20T22:10:37.581 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (37m) 12m ago 49m 82.1M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T22:10:37.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:37.582+0000 7fee0fc00700 1 -- 172.21.15.62:0/2741328244 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fedf006ecc0 msgr2=0x7fedf0071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:10:37.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:37.582+0000 7fee0fc00700 1 --2- 172.21.15.62:0/2741328244 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fedf006ecc0 0x7fedf0071170 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fee08197480 tx=0x7fedf401a040 comp rx=0 tx=0).stop 2024-04-20T22:10:37.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:37.582+0000 7fee0fc00700 1 -- 172.21.15.62:0/2741328244 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee08068540 msgr2=0x7fee08195e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:10:37.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:37.582+0000 7fee0fc00700 1 --2- 172.21.15.62:0/2741328244 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee08068540 0x7fee08195e60 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7fedfc00c930 tx=0x7fedfc00ccf0 comp rx=0 tx=0).stop 2024-04-20T22:10:37.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:37.582+0000 7fee0fc00700 1 -- 172.21.15.62:0/2741328244 shutdown_connections 2024-04-20T22:10:37.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:37.582+0000 7fee0fc00700 1 --2- 172.21.15.62:0/2741328244 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fedf006ecc0 0x7fedf0071170 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:10:37.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:37.582+0000 7fee0fc00700 1 --2- 172.21.15.62:0/2741328244 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee08068540 0x7fee08195e60 secure :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7fedfc00c930 tx=0x7fedfc00ccf0 comp rx=0 tx=0).stop 2024-04-20T22:10:37.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:37.582+0000 7fee0fc00700 1 --2- 172.21.15.62:0/2741328244 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee08106320 0x7fee081963a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:10:37.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:37.582+0000 7fee0fc00700 1 -- 172.21.15.62:0/2741328244 >> 172.21.15.62:0/2741328244 conn(0x7fee08075130 msgr2=0x7fee080febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:10:37.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:37.583+0000 7fee0fc00700 1 -- 172.21.15.62:0/2741328244 shutdown_connections 2024-04-20T22:10:37.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:37.583+0000 7fee0fc00700 1 -- 172.21.15.62:0/2741328244 wait complete. 2024-04-20T22:10:37.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:37.744+0000 7f24adc00700 1 -- 172.21.15.62:0/477288405 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f24a80687a0 msgr2=0x7f24a8068b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:10:37.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:37.744+0000 7f24adc00700 1 --2- 172.21.15.62:0/477288405 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f24a80687a0 0x7f24a8068b70 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f2498009910 tx=0x7f2498009c20 comp rx=0 tx=0).stop 2024-04-20T22:10:37.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:37.744+0000 7f24adc00700 1 -- 172.21.15.62:0/477288405 shutdown_connections 2024-04-20T22:10:37.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:37.744+0000 7f24adc00700 1 --2- 172.21.15.62:0/477288405 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f24a8069140 0x7f24a8104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:10:37.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:37.744+0000 7f24adc00700 1 --2- 172.21.15.62:0/477288405 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f24a80687a0 0x7f24a8068b70 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:10:37.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:37.744+0000 7f24adc00700 1 -- 172.21.15.62:0/477288405 >> 172.21.15.62:0/477288405 conn(0x7f24a8075130 msgr2=0x7f24a8075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:10:37.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:37.744+0000 7f24adc00700 1 -- 172.21.15.62:0/477288405 shutdown_connections 2024-04-20T22:10:37.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:37.745+0000 7f24adc00700 1 -- 172.21.15.62:0/477288405 wait complete. 2024-04-20T22:10:37.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:37.746+0000 7f24adc00700 1 Processor -- start 2024-04-20T22:10:37.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:37.746+0000 7f24adc00700 1 -- start start 2024-04-20T22:10:37.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:37.746+0000 7f24adc00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f24a80687a0 0x7f24a81980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:10:37.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:37.746+0000 7f24adc00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f24a8069140 0x7f24a81985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:10:37.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:37.746+0000 7f24adc00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24a8198cd0 con 0x7f24a8069140 2024-04-20T22:10:37.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:37.746+0000 7f24adc00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24a819ca60 con 0x7f24a80687a0 2024-04-20T22:10:37.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:37.746+0000 7f24a6000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f24a8069140 0x7f24a81985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:10:37.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:37.746+0000 7f24a6000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f24a8069140 0x7f24a81985f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45802/0 (socket says 172.21.15.62:45802) 2024-04-20T22:10:37.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:37.746+0000 7f24a6000700 1 -- 172.21.15.62:0/514388077 learned_addr learned my addr 172.21.15.62:0/514388077 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:10:37.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:37.747+0000 7f24a6a00700 1 --2- 172.21.15.62:0/514388077 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f24a80687a0 0x7f24a81980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:10:37.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:37.747+0000 7f24a6000700 1 -- 172.21.15.62:0/514388077 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f24a80687a0 msgr2=0x7f24a81980b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:10:37.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:37.747+0000 7f24a6000700 1 --2- 172.21.15.62:0/514388077 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f24a80687a0 0x7f24a81980b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:10:37.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:37.747+0000 7f24a6000700 1 -- 172.21.15.62:0/514388077 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24980095f0 con 0x7f24a8069140 2024-04-20T22:10:37.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:37.747+0000 7f24a6000700 1 --2- 172.21.15.62:0/514388077 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f24a8069140 0x7f24a81985f0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f249400d900 tx=0x7f249400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:10:37.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:37.748+0000 7f249f400700 1 -- 172.21.15.62:0/514388077 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2494009900 con 0x7f24a8069140 2024-04-20T22:10:37.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:37.748+0000 7f249f400700 1 -- 172.21.15.62:0/514388077 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2494010460 con 0x7f24a8069140 2024-04-20T22:10:37.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:37.748+0000 7f24adc00700 1 -- 172.21.15.62:0/514388077 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24a819cd40 con 0x7f24a8069140 2024-04-20T22:10:37.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:37.748+0000 7f249f400700 1 -- 172.21.15.62:0/514388077 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f249400f5f0 con 0x7f24a8069140 2024-04-20T22:10:37.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:37.748+0000 7f24adc00700 1 -- 172.21.15.62:0/514388077 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24a819d290 con 0x7f24a8069140 2024-04-20T22:10:37.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:37.749+0000 7f24adc00700 1 -- 172.21.15.62:0/514388077 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24a804f2a0 con 0x7f24a8069140 2024-04-20T22:10:37.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:37.750+0000 7f249f400700 1 -- 172.21.15.62:0/514388077 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f24940105d0 con 0x7f24a8069140 2024-04-20T22:10:37.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:37.751+0000 7f249f400700 1 --2- 172.21.15.62:0/514388077 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f248c06ecc0 0x7f248c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:10:37.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:37.751+0000 7f249f400700 1 -- 172.21.15.62:0/514388077 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f249408ed50 con 0x7f24a8069140 2024-04-20T22:10:37.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:37.751+0000 7f24a6a00700 1 --2- 172.21.15.62:0/514388077 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f248c06ecc0 0x7f248c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:10:37.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:37.752+0000 7f24a6a00700 1 --2- 172.21.15.62:0/514388077 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f248c06ecc0 0x7f248c071170 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f2498006010 tx=0x7f249801a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:10:37.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:37.756+0000 7f249f400700 1 -- 172.21.15.62:0/514388077 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f2494057a60 con 0x7f24a8069140 2024-04-20T22:10:38.085 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.086+0000 7f24adc00700 1 -- 172.21.15.62:0/514388077 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f24a8199410 con 0x7f24a8069140 2024-04-20T22:10:38.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.087+0000 7f249f400700 1 -- 172.21.15.62:0/514388077 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7f249405b120 con 0x7f24a8069140 2024-04-20T22:10:38.087 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:10:38.087 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T22:10:38.087 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:10:38.087 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:10:38.087 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T22:10:38.088 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:10:38.088 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:10:38.088 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T22:10:38.088 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 6 2024-04-20T22:10:38.088 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:10:38.088 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T22:10:38.088 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 1, 2024-04-20T22:10:38.088 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 3 2024-04-20T22:10:38.088 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:10:38.088 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T22:10:38.088 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 1, 2024-04-20T22:10:38.088 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 13 2024-04-20T22:10:38.088 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T22:10:38.088 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:10:38.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.090+0000 7f24adc00700 1 -- 172.21.15.62:0/514388077 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f248c06ecc0 msgr2=0x7f248c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:10:38.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.090+0000 7f24adc00700 1 --2- 172.21.15.62:0/514388077 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f248c06ecc0 0x7f248c071170 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f2498006010 tx=0x7f249801a040 comp rx=0 tx=0).stop 2024-04-20T22:10:38.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.090+0000 7f24adc00700 1 -- 172.21.15.62:0/514388077 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f24a8069140 msgr2=0x7f24a81985f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:10:38.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.090+0000 7f24adc00700 1 --2- 172.21.15.62:0/514388077 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f24a8069140 0x7f24a81985f0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f249400d900 tx=0x7f249400dcc0 comp rx=0 tx=0).stop 2024-04-20T22:10:38.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.091+0000 7f24adc00700 1 -- 172.21.15.62:0/514388077 shutdown_connections 2024-04-20T22:10:38.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.091+0000 7f24adc00700 1 --2- 172.21.15.62:0/514388077 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f248c06ecc0 0x7f248c071170 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:10:38.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.091+0000 7f24adc00700 1 --2- 172.21.15.62:0/514388077 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f24a80687a0 0x7f24a81980b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:10:38.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.091+0000 7f24adc00700 1 --2- 172.21.15.62:0/514388077 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f24a8069140 0x7f24a81985f0 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:10:38.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.091+0000 7f24adc00700 1 -- 172.21.15.62:0/514388077 >> 172.21.15.62:0/514388077 conn(0x7f24a8075130 msgr2=0x7f24a8101e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:10:38.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.091+0000 7f24adc00700 1 -- 172.21.15.62:0/514388077 shutdown_connections 2024-04-20T22:10:38.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.091+0000 7f24adc00700 1 -- 172.21.15.62:0/514388077 wait complete. 2024-04-20T22:10:38.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.257+0000 7fae0d400700 1 -- 172.21.15.62:0/2708939951 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae080ff800 msgr2=0x7fae0810c4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:10:38.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.257+0000 7fae0d400700 1 --2- 172.21.15.62:0/2708939951 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae080ff800 0x7fae0810c4d0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fadf4009910 tx=0x7fadf4009c20 comp rx=0 tx=0).stop 2024-04-20T22:10:38.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.257+0000 7fae0d400700 1 -- 172.21.15.62:0/2708939951 shutdown_connections 2024-04-20T22:10:38.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.257+0000 7fae0d400700 1 --2- 172.21.15.62:0/2708939951 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae080ff800 0x7fae0810c4d0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:10:38.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.257+0000 7fae0d400700 1 --2- 172.21.15.62:0/2708939951 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fae080fee60 0x7fae080ff230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:10:38.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.257+0000 7fae0d400700 1 -- 172.21.15.62:0/2708939951 >> 172.21.15.62:0/2708939951 conn(0x7fae080faad0 msgr2=0x7fae080fcee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:10:38.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.257+0000 7fae0d400700 1 -- 172.21.15.62:0/2708939951 shutdown_connections 2024-04-20T22:10:38.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.257+0000 7fae0d400700 1 -- 172.21.15.62:0/2708939951 wait complete. 2024-04-20T22:10:38.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.258+0000 7fae0d400700 1 Processor -- start 2024-04-20T22:10:38.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.258+0000 7fae0d400700 1 -- start start 2024-04-20T22:10:38.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.259+0000 7fae0d400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fae080fee60 0x7fae08197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:10:38.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.259+0000 7fae0d400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae080ff800 0x7fae08198440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:10:38.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.259+0000 7fae0d400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae08198b20 con 0x7fae080fee60 2024-04-20T22:10:38.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.259+0000 7fae0d400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae0819c8b0 con 0x7fae080ff800 2024-04-20T22:10:38.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.259+0000 7fae07e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fae080fee60 0x7fae08197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:10:38.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.259+0000 7fae07e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fae080fee60 0x7fae08197f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45810/0 (socket says 172.21.15.62:45810) 2024-04-20T22:10:38.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.259+0000 7fae07e00700 1 -- 172.21.15.62:0/939545463 learned_addr learned my addr 172.21.15.62:0/939545463 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:10:38.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.260+0000 7fae07400700 1 --2- 172.21.15.62:0/939545463 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae080ff800 0x7fae08198440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:10:38.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.260+0000 7fae07e00700 1 -- 172.21.15.62:0/939545463 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae080ff800 msgr2=0x7fae08198440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:10:38.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.260+0000 7fae07e00700 1 --2- 172.21.15.62:0/939545463 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae080ff800 0x7fae08198440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:10:38.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.260+0000 7fae07e00700 1 -- 172.21.15.62:0/939545463 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadf40095f0 con 0x7fae080fee60 2024-04-20T22:10:38.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.260+0000 7fae07400700 1 --2- 172.21.15.62:0/939545463 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae080ff800 0x7fae08198440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T22:10:38.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.260+0000 7fae07e00700 1 --2- 172.21.15.62:0/939545463 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fae080fee60 0x7fae08197f00 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7fadf800e920 tx=0x7fadf800ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:10:38.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.260+0000 7fae04c00700 1 -- 172.21.15.62:0/939545463 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fadf8009790 con 0x7fae080fee60 2024-04-20T22:10:38.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.260+0000 7fae04c00700 1 -- 172.21.15.62:0/939545463 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fadf800cba0 con 0x7fae080fee60 2024-04-20T22:10:38.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.260+0000 7fae0d400700 1 -- 172.21.15.62:0/939545463 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae0819cb90 con 0x7fae080fee60 2024-04-20T22:10:38.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.260+0000 7fae04c00700 1 -- 172.21.15.62:0/939545463 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fadf8018650 con 0x7fae080fee60 2024-04-20T22:10:38.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.261+0000 7fae0d400700 1 -- 172.21.15.62:0/939545463 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae0819d0e0 con 0x7fae080fee60 2024-04-20T22:10:38.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.262+0000 7fae0d400700 1 -- 172.21.15.62:0/939545463 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae08109c40 con 0x7fae080fee60 2024-04-20T22:10:38.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.263+0000 7fae04c00700 1 -- 172.21.15.62:0/939545463 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fadf800cd10 con 0x7fae080fee60 2024-04-20T22:10:38.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.264+0000 7fae04c00700 1 --2- 172.21.15.62:0/939545463 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fadec06ecc0 0x7fadec071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:10:38.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.264+0000 7fae04c00700 1 -- 172.21.15.62:0/939545463 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fadf8014070 con 0x7fae080fee60 2024-04-20T22:10:38.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.264+0000 7fae07400700 1 --2- 172.21.15.62:0/939545463 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fadec06ecc0 0x7fadec071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:10:38.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.265+0000 7fae07400700 1 --2- 172.21.15.62:0/939545463 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fadec06ecc0 0x7fadec071170 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fadf4005850 tx=0x7fadf4005fb0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:10:38.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.268+0000 7fae04c00700 1 -- 172.21.15.62:0/939545463 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fadf8053ff0 con 0x7fae080fee60 2024-04-20T22:10:38.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:10:38 smithi189 ceph-mon[57022]: from='client.35880 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:10:38.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:10:38 smithi189 ceph-mon[57022]: from='client.35884 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:10:38.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:10:38 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/514388077' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:10:38.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:10:38 smithi062 ceph-mon[76322]: from='client.35880 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:10:38.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:10:38 smithi062 ceph-mon[76322]: from='client.35884 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:10:38.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:10:38 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/514388077' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:10:38.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.545+0000 7fae0d400700 1 -- 172.21.15.62:0/939545463 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fae0819d350 con 0x7fae080fee60 2024-04-20T22:10:38.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.546+0000 7fae04c00700 1 -- 172.21.15.62:0/939545463 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 27 v27) v1 ==== 76+0+1888 (secure 0 0 0) 0x7fadf805bad0 con 0x7fae080fee60 2024-04-20T22:10:38.547 INFO:teuthology.orchestra.run.smithi062.stdout:e27 2024-04-20T22:10:38.547 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T22:10:38.547 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:10:38.547 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T22:10:38.547 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:10:38.547 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T22:10:38.547 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T22:10:38.547 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 20 2024-04-20T22:10:38.547 INFO:teuthology.orchestra.run.smithi062.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-20T22:10:38.547 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T22:10:38.547 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T21:42:01.559061+0000 2024-04-20T22:10:38.548 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T22:10:38.548 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T22:10:38.548 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T22:10:38.548 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T22:10:38.548 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T22:10:38.548 INFO:teuthology.orchestra.run.smithi062.stdout:max_xattr_size 65536 2024-04-20T22:10:38.548 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T22:10:38.548 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T22:10:38.548 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 81 2024-04-20T22:10:38.548 INFO:teuthology.orchestra.run.smithi062.stdout:snap_rstat 1 2024-04-20T22:10:38.548 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:10:38.548 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 2 2024-04-20T22:10:38.548 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1 2024-04-20T22:10:38.548 INFO:teuthology.orchestra.run.smithi062.stdout:up {} 2024-04-20T22:10:38.549 INFO:teuthology.orchestra.run.smithi062.stdout:failed 0,1 2024-04-20T22:10:38.549 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T22:10:38.549 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T22:10:38.549 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T22:10:38.549 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T22:10:38.549 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data enabled 2024-04-20T22:10:38.549 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T22:10:38.549 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T22:10:38.549 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 1 2024-04-20T22:10:38.549 INFO:teuthology.orchestra.run.smithi062.stdout:qdb_cluster leader: 0 members: 2024-04-20T22:10:38.549 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:10:38.549 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:10:38.549 INFO:teuthology.orchestra.run.smithi062.stdout:Standby daemons: 2024-04-20T22:10:38.549 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:10:38.550 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ryqblb{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6826/4113916205,v1:172.21.15.62:6827/4113916205] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:10:38.550 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ozizwi{-1:34476} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6828/3801975165,v1:172.21.15.62:6829/3801975165] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:10:38.550 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.xbgspw{-1:44285} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6826/3996040526,v1:172.21.15.189:6827/3996040526] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T22:10:38.550 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.dvimer{-1:44727} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6824/3576699232,v1:172.21.15.189:6825/3576699232] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:10:38.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.549+0000 7fae0d400700 1 -- 172.21.15.62:0/939545463 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fadec06ecc0 msgr2=0x7fadec071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:10:38.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.549+0000 7fae0d400700 1 --2- 172.21.15.62:0/939545463 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fadec06ecc0 0x7fadec071170 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fadf4005850 tx=0x7fadf4005fb0 comp rx=0 tx=0).stop 2024-04-20T22:10:38.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.549+0000 7fae0d400700 1 -- 172.21.15.62:0/939545463 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fae080fee60 msgr2=0x7fae08197f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:10:38.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.549+0000 7fae0d400700 1 --2- 172.21.15.62:0/939545463 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fae080fee60 0x7fae08197f00 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7fadf800e920 tx=0x7fadf800ece0 comp rx=0 tx=0).stop 2024-04-20T22:10:38.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.550+0000 7fae0d400700 1 -- 172.21.15.62:0/939545463 shutdown_connections 2024-04-20T22:10:38.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.550+0000 7fae0d400700 1 --2- 172.21.15.62:0/939545463 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fadec06ecc0 0x7fadec071170 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:10:38.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.550+0000 7fae0d400700 1 --2- 172.21.15.62:0/939545463 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fae080fee60 0x7fae08197f00 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:10:38.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.550+0000 7fae0d400700 1 --2- 172.21.15.62:0/939545463 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae080ff800 0x7fae08198440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:10:38.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.550+0000 7fae0d400700 1 -- 172.21.15.62:0/939545463 >> 172.21.15.62:0/939545463 conn(0x7fae080faad0 msgr2=0x7fae080fc000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:10:38.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.550+0000 7fae0d400700 1 -- 172.21.15.62:0/939545463 shutdown_connections 2024-04-20T22:10:38.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.550+0000 7fae0d400700 1 -- 172.21.15.62:0/939545463 wait complete. 2024-04-20T22:10:38.552 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 27 2024-04-20T22:10:38.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.704+0000 7efe5b600700 1 -- 172.21.15.62:0/3894892723 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efe540739a0 msgr2=0x7efe5410c5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:10:38.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.704+0000 7efe5b600700 1 --2- 172.21.15.62:0/3894892723 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efe540739a0 0x7efe5410c5e0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7efe3c009940 tx=0x7efe3c009c50 comp rx=0 tx=0).stop 2024-04-20T22:10:38.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.704+0000 7efe5b600700 1 -- 172.21.15.62:0/3894892723 shutdown_connections 2024-04-20T22:10:38.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.704+0000 7efe5b600700 1 --2- 172.21.15.62:0/3894892723 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efe540739a0 0x7efe5410c5e0 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:10:38.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.704+0000 7efe5b600700 1 --2- 172.21.15.62:0/3894892723 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efe54073090 0x7efe54073460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:10:38.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.704+0000 7efe5b600700 1 -- 172.21.15.62:0/3894892723 >> 172.21.15.62:0/3894892723 conn(0x7efe540fbda0 msgr2=0x7efe540fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:10:38.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.704+0000 7efe5b600700 1 -- 172.21.15.62:0/3894892723 shutdown_connections 2024-04-20T22:10:38.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.704+0000 7efe5b600700 1 -- 172.21.15.62:0/3894892723 wait complete. 2024-04-20T22:10:38.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.705+0000 7efe5b600700 1 Processor -- start 2024-04-20T22:10:38.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.706+0000 7efe5b600700 1 -- start start 2024-04-20T22:10:38.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.706+0000 7efe5b600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efe54073090 0x7efe541981a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:10:38.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.706+0000 7efe5b600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efe540739a0 0x7efe541986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:10:38.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.706+0000 7efe5b600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe54198dc0 con 0x7efe540739a0 2024-04-20T22:10:38.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.706+0000 7efe5b600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe5419cb50 con 0x7efe54073090 2024-04-20T22:10:38.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.706+0000 7efe4be00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efe540739a0 0x7efe541986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:10:38.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.706+0000 7efe4be00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efe540739a0 0x7efe541986e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45828/0 (socket says 172.21.15.62:45828) 2024-04-20T22:10:38.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.706+0000 7efe4be00700 1 -- 172.21.15.62:0/3458929014 learned_addr learned my addr 172.21.15.62:0/3458929014 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:10:38.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.707+0000 7efe58c00700 1 --2- 172.21.15.62:0/3458929014 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efe54073090 0x7efe541981a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:10:38.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.707+0000 7efe4be00700 1 -- 172.21.15.62:0/3458929014 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efe54073090 msgr2=0x7efe541981a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:10:38.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.707+0000 7efe4be00700 1 --2- 172.21.15.62:0/3458929014 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efe54073090 0x7efe541981a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:10:38.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.707+0000 7efe4be00700 1 -- 172.21.15.62:0/3458929014 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe3c0095f0 con 0x7efe540739a0 2024-04-20T22:10:38.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.707+0000 7efe58c00700 1 --2- 172.21.15.62:0/3458929014 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efe54073090 0x7efe541981a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T22:10:38.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.707+0000 7efe4be00700 1 --2- 172.21.15.62:0/3458929014 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efe540739a0 0x7efe541986e0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7efe3c0055f0 tx=0x7efe3c0056d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:10:38.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.708+0000 7efe52000700 1 -- 172.21.15.62:0/3458929014 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efe3c01d070 con 0x7efe540739a0 2024-04-20T22:10:38.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.708+0000 7efe52000700 1 -- 172.21.15.62:0/3458929014 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efe3c00f460 con 0x7efe540739a0 2024-04-20T22:10:38.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.708+0000 7efe5b600700 1 -- 172.21.15.62:0/3458929014 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe5419cdd0 con 0x7efe540739a0 2024-04-20T22:10:38.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.708+0000 7efe52000700 1 -- 172.21.15.62:0/3458929014 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efe3c021640 con 0x7efe540739a0 2024-04-20T22:10:38.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.708+0000 7efe5b600700 1 -- 172.21.15.62:0/3458929014 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe5419d2c0 con 0x7efe540739a0 2024-04-20T22:10:38.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.709+0000 7efe5b600700 1 -- 172.21.15.62:0/3458929014 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe54109ce0 con 0x7efe540739a0 2024-04-20T22:10:38.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.710+0000 7efe52000700 1 -- 172.21.15.62:0/3458929014 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7efe3c00f5d0 con 0x7efe540739a0 2024-04-20T22:10:38.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.711+0000 7efe52000700 1 --2- 172.21.15.62:0/3458929014 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7efe3806ecc0 0x7efe38071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:10:38.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.711+0000 7efe52000700 1 -- 172.21.15.62:0/3458929014 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7efe3c090230 con 0x7efe540739a0 2024-04-20T22:10:38.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.711+0000 7efe58c00700 1 --2- 172.21.15.62:0/3458929014 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7efe3806ecc0 0x7efe38071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:10:38.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.712+0000 7efe58c00700 1 --2- 172.21.15.62:0/3458929014 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7efe3806ecc0 0x7efe38071170 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7efe44007900 tx=0x7efe44008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:10:38.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.715+0000 7efe52000700 1 -- 172.21.15.62:0/3458929014 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7efe3c058f40 con 0x7efe540739a0 2024-04-20T22:10:38.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.965+0000 7efe5b600700 1 -- 172.21.15.62:0/3458929014 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7efe5419d5a0 con 0x7efe3806ecc0 2024-04-20T22:10:38.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.967+0000 7efe52000700 1 -- 172.21.15.62:0/3458929014 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7efe5419d5a0 con 0x7efe3806ecc0 2024-04-20T22:10:38.968 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:10:38.968 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T22:10:38.968 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T22:10:38.968 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-20T22:10:38.968 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T22:10:38.968 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr", 2024-04-20T22:10:38.969 INFO:teuthology.orchestra.run.smithi062.stdout: "crash", 2024-04-20T22:10:38.969 INFO:teuthology.orchestra.run.smithi062.stdout: "mon", 2024-04-20T22:10:38.969 INFO:teuthology.orchestra.run.smithi062.stdout: "osd" 2024-04-20T22:10:38.969 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T22:10:38.969 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "15/23 daemons upgraded", 2024-04-20T22:10:38.969 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading mds daemons", 2024-04-20T22:10:38.969 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T22:10:38.970 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:10:38.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.970+0000 7efe5b600700 1 -- 172.21.15.62:0/3458929014 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7efe3806ecc0 msgr2=0x7efe38071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:10:38.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.970+0000 7efe5b600700 1 --2- 172.21.15.62:0/3458929014 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7efe3806ecc0 0x7efe38071170 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7efe44007900 tx=0x7efe44008040 comp rx=0 tx=0).stop 2024-04-20T22:10:38.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.970+0000 7efe5b600700 1 -- 172.21.15.62:0/3458929014 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efe540739a0 msgr2=0x7efe541986e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:10:38.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.970+0000 7efe5b600700 1 --2- 172.21.15.62:0/3458929014 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efe540739a0 0x7efe541986e0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7efe3c0055f0 tx=0x7efe3c0056d0 comp rx=0 tx=0).stop 2024-04-20T22:10:38.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.970+0000 7efe5b600700 1 -- 172.21.15.62:0/3458929014 shutdown_connections 2024-04-20T22:10:38.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.970+0000 7efe5b600700 1 --2- 172.21.15.62:0/3458929014 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7efe3806ecc0 0x7efe38071170 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:10:38.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.970+0000 7efe5b600700 1 --2- 172.21.15.62:0/3458929014 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efe54073090 0x7efe541981a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:10:38.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.970+0000 7efe5b600700 1 --2- 172.21.15.62:0/3458929014 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efe540739a0 0x7efe541986e0 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:10:38.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.970+0000 7efe5b600700 1 -- 172.21.15.62:0/3458929014 >> 172.21.15.62:0/3458929014 conn(0x7efe540fbda0 msgr2=0x7efe54106e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:10:38.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.970+0000 7efe5b600700 1 -- 172.21.15.62:0/3458929014 shutdown_connections 2024-04-20T22:10:38.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:38.971+0000 7efe5b600700 1 -- 172.21.15.62:0/3458929014 wait complete. 2024-04-20T22:10:39.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:39.127+0000 7f4185e00700 1 -- 172.21.15.62:0/3343186306 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4180108470 msgr2=0x7f4180108840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:10:39.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:39.127+0000 7f4185e00700 1 --2- 172.21.15.62:0/3343186306 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4180108470 0x7f4180108840 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f4170009910 tx=0x7f4170009c20 comp rx=0 tx=0).stop 2024-04-20T22:10:39.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:39.127+0000 7f4185e00700 1 -- 172.21.15.62:0/3343186306 shutdown_connections 2024-04-20T22:10:39.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:39.127+0000 7f4185e00700 1 --2- 172.21.15.62:0/3343186306 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4180102470 0x7f41801028e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:10:39.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:39.127+0000 7f4185e00700 1 --2- 172.21.15.62:0/3343186306 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4180108470 0x7f4180108840 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:10:39.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:39.127+0000 7f4185e00700 1 -- 172.21.15.62:0/3343186306 >> 172.21.15.62:0/3343186306 conn(0x7f41800fdf90 msgr2=0x7f41801003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:10:39.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:39.127+0000 7f4185e00700 1 -- 172.21.15.62:0/3343186306 shutdown_connections 2024-04-20T22:10:39.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:39.128+0000 7f4185e00700 1 -- 172.21.15.62:0/3343186306 wait complete. 2024-04-20T22:10:39.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:39.128+0000 7f4185e00700 1 Processor -- start 2024-04-20T22:10:39.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:39.129+0000 7f4185e00700 1 -- start start 2024-04-20T22:10:39.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:39.129+0000 7f4185e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4180102470 0x7f4180198210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:10:39.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:39.129+0000 7f4185e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4180108470 0x7f4180198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:10:39.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:39.129+0000 7f4185e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4180198e30 con 0x7f4180102470 2024-04-20T22:10:39.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:39.129+0000 7f4185e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f418019cbc0 con 0x7f4180108470 2024-04-20T22:10:39.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:39.129+0000 7f417ea00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4180102470 0x7f4180198210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:10:39.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:39.129+0000 7f417ea00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4180102470 0x7f4180198210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45850/0 (socket says 172.21.15.62:45850) 2024-04-20T22:10:39.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:39.129+0000 7f417ea00700 1 -- 172.21.15.62:0/3514006912 learned_addr learned my addr 172.21.15.62:0/3514006912 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:10:39.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:39.130+0000 7f417ea00700 1 -- 172.21.15.62:0/3514006912 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4180108470 msgr2=0x7f4180198750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:10:39.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:39.130+0000 7f417ea00700 1 --2- 172.21.15.62:0/3514006912 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4180108470 0x7f4180198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:10:39.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:39.130+0000 7f417ea00700 1 -- 172.21.15.62:0/3514006912 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41700095f0 con 0x7f4180102470 2024-04-20T22:10:39.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:39.130+0000 7f417ea00700 1 --2- 172.21.15.62:0/3514006912 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4180102470 0x7f4180198210 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f417000b5c0 tx=0x7f417000be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:10:39.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:39.130+0000 7f4184a00700 1 -- 172.21.15.62:0/3514006912 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f417001d070 con 0x7f4180102470 2024-04-20T22:10:39.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:39.130+0000 7f4184a00700 1 -- 172.21.15.62:0/3514006912 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f417000f460 con 0x7f4180102470 2024-04-20T22:10:39.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:39.131+0000 7f4185e00700 1 -- 172.21.15.62:0/3514006912 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f418019ce40 con 0x7f4180102470 2024-04-20T22:10:39.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:39.131+0000 7f4184a00700 1 -- 172.21.15.62:0/3514006912 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4170017610 con 0x7f4180102470 2024-04-20T22:10:39.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:39.131+0000 7f4185e00700 1 -- 172.21.15.62:0/3514006912 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f418019d330 con 0x7f4180102470 2024-04-20T22:10:39.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:39.132+0000 7f4185e00700 1 -- 172.21.15.62:0/3514006912 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f418010a9e0 con 0x7f4180102470 2024-04-20T22:10:39.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:39.133+0000 7f4184a00700 1 -- 172.21.15.62:0/3514006912 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f417000f5d0 con 0x7f4180102470 2024-04-20T22:10:39.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:39.134+0000 7f4184a00700 1 --2- 172.21.15.62:0/3514006912 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f416406ecc0 0x7f4164071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:10:39.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:39.134+0000 7f4184a00700 1 -- 172.21.15.62:0/3514006912 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f4170090450 con 0x7f4180102470 2024-04-20T22:10:39.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:39.134+0000 7f4176000700 1 --2- 172.21.15.62:0/3514006912 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f416406ecc0 0x7f4164071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:10:39.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:39.135+0000 7f4176000700 1 --2- 172.21.15.62:0/3514006912 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f416406ecc0 0x7f4164071170 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f4180199830 tx=0x7f4168009250 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:10:39.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:39.139+0000 7f4184a00700 1 -- 172.21.15.62:0/3514006912 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f41700591e0 con 0x7f4180102470 2024-04-20T22:10:39.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:10:39 smithi062 ceph-mon[76322]: from='client.35888 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:10:39.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:10:39 smithi062 ceph-mon[76322]: pgmap v1061: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:10:39.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:10:39 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/939545463' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:10:39.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:39.463+0000 7f4185e00700 1 -- 172.21.15.62:0/3514006912 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f418004f2a0 con 0x7f4180102470 2024-04-20T22:10:39.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:39.463+0000 7f4184a00700 1 -- 172.21.15.62:0/3514006912 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7f417005c8a0 con 0x7f4180102470 2024-04-20T22:10:39.464 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-20T22:10:39.464 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-20T22:10:39.464 INFO:teuthology.orchestra.run.smithi062.stdout: host smithi062 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T22:10:39.464 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-20T22:10:39.464 INFO:teuthology.orchestra.run.smithi062.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi062 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpq7y7oa_r:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:10:39.464 INFO:teuthology.orchestra.run.smithi062.stdout:Traceback (most recent call last): 2024-04-20T22:10:39.464 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-20T22:10:39.464 INFO:teuthology.orchestra.run.smithi062.stdout: return _run_code(code, main_globals, None, 2024-04-20T22:10:39.464 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-20T22:10:39.464 INFO:teuthology.orchestra.run.smithi062.stdout: exec(code, run_globals) 2024-04-20T22:10:39.465 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5618, in 2024-04-20T22:10:39.465 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5606, in main 2024-04-20T22:10:39.465 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 408, in _infer_config 2024-04-20T22:10:39.465 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 323, in _infer_fsid 2024-04-20T22:10:39.465 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 436, in _infer_image 2024-04-20T22:10:39.465 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 310, in _validate_fsid 2024-04-20T22:10:39.465 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 3359, in command_ceph_volume 2024-04-20T22:10:39.465 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-20T22:10:39.465 INFO:teuthology.orchestra.run.smithi062.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi062 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpq7y7oa_r:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:10:39.465 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-20T22:10:39.465 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is degraded 2024-04-20T22:10:39.465 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T22:10:39.465 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T22:10:39.465 INFO:teuthology.orchestra.run.smithi062.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-20T22:10:39.465 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-20T22:10:39.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:39.466+0000 7f4185e00700 1 -- 172.21.15.62:0/3514006912 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f416406ecc0 msgr2=0x7f4164071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:10:39.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:39.467+0000 7f4185e00700 1 --2- 172.21.15.62:0/3514006912 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f416406ecc0 0x7f4164071170 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f4180199830 tx=0x7f4168009250 comp rx=0 tx=0).stop 2024-04-20T22:10:39.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:39.467+0000 7f4185e00700 1 -- 172.21.15.62:0/3514006912 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4180102470 msgr2=0x7f4180198210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:10:39.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:39.467+0000 7f4185e00700 1 --2- 172.21.15.62:0/3514006912 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4180102470 0x7f4180198210 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f417000b5c0 tx=0x7f417000be00 comp rx=0 tx=0).stop 2024-04-20T22:10:39.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:39.467+0000 7f4185e00700 1 -- 172.21.15.62:0/3514006912 shutdown_connections 2024-04-20T22:10:39.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:39.467+0000 7f4185e00700 1 --2- 172.21.15.62:0/3514006912 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f416406ecc0 0x7f4164071170 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:10:39.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:39.467+0000 7f4185e00700 1 --2- 172.21.15.62:0/3514006912 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4180102470 0x7f4180198210 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:10:39.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:39.467+0000 7f4185e00700 1 --2- 172.21.15.62:0/3514006912 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4180108470 0x7f4180198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:10:39.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:39.467+0000 7f4185e00700 1 -- 172.21.15.62:0/3514006912 >> 172.21.15.62:0/3514006912 conn(0x7f41800fdf90 msgr2=0x7f41800febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:10:39.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:39.467+0000 7f4185e00700 1 -- 172.21.15.62:0/3514006912 shutdown_connections 2024-04-20T22:10:39.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:10:39.467+0000 7f4185e00700 1 -- 172.21.15.62:0/3514006912 wait complete. 2024-04-20T22:10:39.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:10:39 smithi189 ceph-mon[57022]: from='client.35888 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:10:39.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:10:39 smithi189 ceph-mon[57022]: pgmap v1061: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:10:39.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:10:39 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/939545463' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:10:40.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:10:40 smithi189 ceph-mon[57022]: from='client.35900 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:10:40.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:10:40 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/3514006912' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:10:40.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:10:40 smithi062 ceph-mon[76322]: from='client.35900 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:10:40.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:10:40 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/3514006912' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:10:41.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:10:41 smithi189 ceph-mon[57022]: pgmap v1062: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:10:41.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:10:41 smithi062 ceph-mon[76322]: pgmap v1062: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:10:43.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:10:43 smithi189 ceph-mon[57022]: pgmap v1063: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:10:43.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:10:43 smithi062 ceph-mon[76322]: pgmap v1063: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:10:45.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:10:45 smithi189 ceph-mon[57022]: pgmap v1064: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:10:45.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:10:45 smithi062 ceph-mon[76322]: pgmap v1064: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:10:47.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:10:47 smithi189 ceph-mon[57022]: pgmap v1065: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:10:47.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:10:47 smithi062 ceph-mon[76322]: pgmap v1065: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:10:49.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:10:49 smithi189 ceph-mon[57022]: pgmap v1066: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:10:49.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:10:49 smithi062 ceph-mon[76322]: pgmap v1066: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:10:50.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:10:50 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:10:50.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:10:50 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:10:51.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:10:51 smithi189 ceph-mon[57022]: pgmap v1067: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:10:51.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:10:51 smithi062 ceph-mon[76322]: pgmap v1067: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:10:53.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:10:53 smithi189 ceph-mon[57022]: pgmap v1068: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:10:53.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:10:53 smithi062 ceph-mon[76322]: pgmap v1068: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:10:55.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:10:55 smithi062 ceph-mon[76322]: pgmap v1069: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:10:55.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:10:55 smithi189 ceph-mon[57022]: pgmap v1069: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:10:57.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:10:57 smithi062 ceph-mon[76322]: pgmap v1070: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:10:57.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:10:57 smithi189 ceph-mon[57022]: pgmap v1070: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:10:59.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:10:59 smithi062 ceph-mon[76322]: pgmap v1071: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:10:59.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:10:59 smithi189 ceph-mon[57022]: pgmap v1071: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:11:01.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:11:01 smithi062 ceph-mon[76322]: pgmap v1072: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:11:01.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:11:01 smithi189 ceph-mon[57022]: pgmap v1072: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:11:03.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:11:03 smithi062 ceph-mon[76322]: pgmap v1073: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:11:03.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:11:03 smithi189 ceph-mon[57022]: pgmap v1073: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:11:05.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:11:05 smithi062 ceph-mon[76322]: pgmap v1074: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:11:05.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:11:05 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:11:05.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:11:05 smithi189 ceph-mon[57022]: pgmap v1074: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:11:05.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:11:05 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:11:07.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:11:07 smithi062 ceph-mon[76322]: pgmap v1075: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:11:07.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:11:07 smithi189 ceph-mon[57022]: pgmap v1075: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:11:09.541 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:11:09 smithi189 ceph-mon[57022]: pgmap v1076: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:11:09.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:09.645+0000 7f5402a00700 1 -- 172.21.15.62:0/1719426159 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f53fc101d30 msgr2=0x7f53fc10a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:11:09.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:09.645+0000 7f5402a00700 1 --2- 172.21.15.62:0/1719426159 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f53fc101d30 0x7f53fc10a230 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f53e8009910 tx=0x7f53e8009c20 comp rx=0 tx=0).stop 2024-04-20T22:11:09.644 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:11:09 smithi062 ceph-mon[76322]: pgmap v1076: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:11:09.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:09.645+0000 7f5402a00700 1 -- 172.21.15.62:0/1719426159 shutdown_connections 2024-04-20T22:11:09.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:09.645+0000 7f5402a00700 1 --2- 172.21.15.62:0/1719426159 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f53fc101d30 0x7f53fc10a230 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:11:09.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:09.645+0000 7f5402a00700 1 --2- 172.21.15.62:0/1719426159 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53fc101420 0x7f53fc1017f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:11:09.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:09.645+0000 7f5402a00700 1 -- 172.21.15.62:0/1719426159 >> 172.21.15.62:0/1719426159 conn(0x7f53fc0fac80 msgr2=0x7f53fc0fd090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:11:09.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:09.645+0000 7f5402a00700 1 -- 172.21.15.62:0/1719426159 shutdown_connections 2024-04-20T22:11:09.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:09.645+0000 7f5402a00700 1 -- 172.21.15.62:0/1719426159 wait complete. 2024-04-20T22:11:09.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:09.646+0000 7f5402a00700 1 Processor -- start 2024-04-20T22:11:09.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:09.646+0000 7f5402a00700 1 -- start start 2024-04-20T22:11:09.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:09.647+0000 7f5402a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f53fc101420 0x7f53fc195e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:11:09.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:09.647+0000 7f5402a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53fc101d30 0x7f53fc1963a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:11:09.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:09.647+0000 7f5402a00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53fc196a80 con 0x7f53fc101420 2024-04-20T22:11:09.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:09.647+0000 7f5402a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53fc19a810 con 0x7f53fc101d30 2024-04-20T22:11:09.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:09.647+0000 7f53fbe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f53fc101420 0x7f53fc195e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:11:09.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:09.647+0000 7f53fbe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f53fc101420 0x7f53fc195e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34522/0 (socket says 172.21.15.62:34522) 2024-04-20T22:11:09.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:09.647+0000 7f53fbe00700 1 -- 172.21.15.62:0/2327307108 learned_addr learned my addr 172.21.15.62:0/2327307108 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:11:09.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:09.647+0000 7f53fb400700 1 --2- 172.21.15.62:0/2327307108 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53fc101d30 0x7f53fc1963a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:11:09.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:09.648+0000 7f53fbe00700 1 -- 172.21.15.62:0/2327307108 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53fc101d30 msgr2=0x7f53fc1963a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:11:09.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:09.648+0000 7f53fbe00700 1 --2- 172.21.15.62:0/2327307108 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53fc101d30 0x7f53fc1963a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:11:09.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:09.648+0000 7f53fbe00700 1 -- 172.21.15.62:0/2327307108 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53e80095f0 con 0x7f53fc101420 2024-04-20T22:11:09.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:09.648+0000 7f53fb400700 1 --2- 172.21.15.62:0/2327307108 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53fc101d30 0x7f53fc1963a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T22:11:09.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:09.648+0000 7f53fbe00700 1 --2- 172.21.15.62:0/2327307108 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f53fc101420 0x7f53fc195e60 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f53ec00d8d0 tx=0x7f53ec00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:11:09.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:09.648+0000 7f53f8c00700 1 -- 172.21.15.62:0/2327307108 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f53ec009940 con 0x7f53fc101420 2024-04-20T22:11:09.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:09.648+0000 7f53f8c00700 1 -- 172.21.15.62:0/2327307108 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f53ec010460 con 0x7f53fc101420 2024-04-20T22:11:09.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:09.648+0000 7f53f8c00700 1 -- 172.21.15.62:0/2327307108 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f53ec00f5f0 con 0x7f53fc101420 2024-04-20T22:11:09.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:09.648+0000 7f5402a00700 1 -- 172.21.15.62:0/2327307108 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53fc19aaf0 con 0x7f53fc101420 2024-04-20T22:11:09.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:09.649+0000 7f5402a00700 1 -- 172.21.15.62:0/2327307108 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53fc19b040 con 0x7f53fc101420 2024-04-20T22:11:09.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:09.650+0000 7f5402a00700 1 -- 172.21.15.62:0/2327307108 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53fc1078a0 con 0x7f53fc101420 2024-04-20T22:11:09.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:09.651+0000 7f53f8c00700 1 -- 172.21.15.62:0/2327307108 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f53ec0105d0 con 0x7f53fc101420 2024-04-20T22:11:09.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:09.652+0000 7f53f8c00700 1 --2- 172.21.15.62:0/2327307108 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f53e006ecc0 0x7f53e0071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:11:09.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:09.652+0000 7f53f8c00700 1 -- 172.21.15.62:0/2327307108 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f53ec08ee10 con 0x7f53fc101420 2024-04-20T22:11:09.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:09.653+0000 7f53fb400700 1 --2- 172.21.15.62:0/2327307108 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f53e006ecc0 0x7f53e0071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:11:09.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:09.653+0000 7f53fb400700 1 --2- 172.21.15.62:0/2327307108 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f53e006ecc0 0x7f53e0071170 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f53e8006010 tx=0x7f53e800b540 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:11:09.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:09.658+0000 7f53f8c00700 1 -- 172.21.15.62:0/2327307108 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f53ec057ba0 con 0x7f53fc101420 2024-04-20T22:11:09.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:09.917+0000 7f5402a00700 1 -- 172.21.15.62:0/2327307108 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f53fc19b2b0 con 0x7f53e006ecc0 2024-04-20T22:11:09.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:09.920+0000 7f53f8c00700 1 -- 172.21.15.62:0/2327307108 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f53fc19b2b0 con 0x7f53e006ecc0 2024-04-20T22:11:09.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:09.923+0000 7f5402a00700 1 -- 172.21.15.62:0/2327307108 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f53e006ecc0 msgr2=0x7f53e0071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:11:09.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:09.923+0000 7f5402a00700 1 --2- 172.21.15.62:0/2327307108 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f53e006ecc0 0x7f53e0071170 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f53e8006010 tx=0x7f53e800b540 comp rx=0 tx=0).stop 2024-04-20T22:11:09.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:09.923+0000 7f5402a00700 1 -- 172.21.15.62:0/2327307108 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f53fc101420 msgr2=0x7f53fc195e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:11:09.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:09.923+0000 7f5402a00700 1 --2- 172.21.15.62:0/2327307108 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f53fc101420 0x7f53fc195e60 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f53ec00d8d0 tx=0x7f53ec00dc90 comp rx=0 tx=0).stop 2024-04-20T22:11:09.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:09.923+0000 7f5402a00700 1 -- 172.21.15.62:0/2327307108 shutdown_connections 2024-04-20T22:11:09.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:09.923+0000 7f5402a00700 1 --2- 172.21.15.62:0/2327307108 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f53e006ecc0 0x7f53e0071170 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:11:09.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:09.923+0000 7f5402a00700 1 --2- 172.21.15.62:0/2327307108 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f53fc101420 0x7f53fc195e60 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:11:09.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:09.923+0000 7f5402a00700 1 --2- 172.21.15.62:0/2327307108 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53fc101d30 0x7f53fc1963a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:11:09.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:09.923+0000 7f5402a00700 1 -- 172.21.15.62:0/2327307108 >> 172.21.15.62:0/2327307108 conn(0x7f53fc0fac80 msgr2=0x7f53fc1049e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:11:09.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:09.923+0000 7f5402a00700 1 -- 172.21.15.62:0/2327307108 shutdown_connections 2024-04-20T22:11:09.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:09.924+0000 7f5402a00700 1 -- 172.21.15.62:0/2327307108 wait complete. 2024-04-20T22:11:09.939 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T22:11:10.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.093+0000 7f3b69000700 1 -- 172.21.15.62:0/3885110150 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3b64068540 msgr2=0x7f3b640689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:11:10.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.093+0000 7f3b69000700 1 --2- 172.21.15.62:0/3885110150 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3b64068540 0x7f3b640689b0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f3b50009910 tx=0x7f3b50009c20 comp rx=0 tx=0).stop 2024-04-20T22:11:10.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.093+0000 7f3b69000700 1 -- 172.21.15.62:0/3885110150 shutdown_connections 2024-04-20T22:11:10.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.093+0000 7f3b69000700 1 --2- 172.21.15.62:0/3885110150 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3b64068540 0x7f3b640689b0 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:11:10.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.093+0000 7f3b69000700 1 --2- 172.21.15.62:0/3885110150 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b64106200 0x7f3b641065d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:11:10.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.093+0000 7f3b69000700 1 -- 172.21.15.62:0/3885110150 >> 172.21.15.62:0/3885110150 conn(0x7f3b64074fa0 msgr2=0x7f3b640753a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:11:10.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.093+0000 7f3b69000700 1 -- 172.21.15.62:0/3885110150 shutdown_connections 2024-04-20T22:11:10.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.093+0000 7f3b69000700 1 -- 172.21.15.62:0/3885110150 wait complete. 2024-04-20T22:11:10.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.094+0000 7f3b69000700 1 Processor -- start 2024-04-20T22:11:10.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.094+0000 7f3b69000700 1 -- start start 2024-04-20T22:11:10.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.095+0000 7f3b69000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3b64068540 0x7f3b6419b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:11:10.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.095+0000 7f3b69000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b64106200 0x7f3b6419be70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:11:10.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.095+0000 7f3b69000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b6419c470 con 0x7f3b64068540 2024-04-20T22:11:10.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.095+0000 7f3b69000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b641959b0 con 0x7f3b64106200 2024-04-20T22:11:10.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.095+0000 7f3b63400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3b64068540 0x7f3b6419b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:11:10.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.095+0000 7f3b63400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3b64068540 0x7f3b6419b930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34538/0 (socket says 172.21.15.62:34538) 2024-04-20T22:11:10.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.095+0000 7f3b63400700 1 -- 172.21.15.62:0/3407369370 learned_addr learned my addr 172.21.15.62:0/3407369370 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:11:10.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.096+0000 7f3b62a00700 1 --2- 172.21.15.62:0/3407369370 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b64106200 0x7f3b6419be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:11:10.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.096+0000 7f3b63400700 1 -- 172.21.15.62:0/3407369370 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b64106200 msgr2=0x7f3b6419be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:11:10.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.096+0000 7f3b63400700 1 --2- 172.21.15.62:0/3407369370 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b64106200 0x7f3b6419be70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:11:10.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.096+0000 7f3b63400700 1 -- 172.21.15.62:0/3407369370 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b500095f0 con 0x7f3b64068540 2024-04-20T22:11:10.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.096+0000 7f3b62a00700 1 --2- 172.21.15.62:0/3407369370 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b64106200 0x7f3b6419be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T22:11:10.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.096+0000 7f3b63400700 1 --2- 172.21.15.62:0/3407369370 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3b64068540 0x7f3b6419b930 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f3b5400c930 tx=0x7f3b5400ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:11:10.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.096+0000 7f3b5be00700 1 -- 172.21.15.62:0/3407369370 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3b54007ad0 con 0x7f3b64068540 2024-04-20T22:11:10.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.096+0000 7f3b5be00700 1 -- 172.21.15.62:0/3407369370 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3b54007c30 con 0x7f3b64068540 2024-04-20T22:11:10.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.096+0000 7f3b69000700 1 -- 172.21.15.62:0/3407369370 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b64195c90 con 0x7f3b64068540 2024-04-20T22:11:10.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.096+0000 7f3b5be00700 1 -- 172.21.15.62:0/3407369370 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3b54018610 con 0x7f3b64068540 2024-04-20T22:11:10.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.097+0000 7f3b69000700 1 -- 172.21.15.62:0/3407369370 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b64196260 con 0x7f3b64068540 2024-04-20T22:11:10.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.098+0000 7f3b69000700 1 -- 172.21.15.62:0/3407369370 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b64062380 con 0x7f3b64068540 2024-04-20T22:11:10.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.099+0000 7f3b5be00700 1 -- 172.21.15.62:0/3407369370 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f3b5400f450 con 0x7f3b64068540 2024-04-20T22:11:10.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.100+0000 7f3b5be00700 1 --2- 172.21.15.62:0/3407369370 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3b4806ecc0 0x7f3b48071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:11:10.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.100+0000 7f3b5be00700 1 -- 172.21.15.62:0/3407369370 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f3b5408f080 con 0x7f3b64068540 2024-04-20T22:11:10.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.100+0000 7f3b62a00700 1 --2- 172.21.15.62:0/3407369370 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3b4806ecc0 0x7f3b48071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:11:10.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.101+0000 7f3b62a00700 1 --2- 172.21.15.62:0/3407369370 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3b4806ecc0 0x7f3b48071170 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f3b50006010 tx=0x7f3b5000b540 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:11:10.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.105+0000 7f3b5be00700 1 -- 172.21.15.62:0/3407369370 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f3b5405b960 con 0x7f3b64068540 2024-04-20T22:11:10.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.367+0000 7f3b69000700 1 -- 172.21.15.62:0/3407369370 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3b64196550 con 0x7f3b4806ecc0 2024-04-20T22:11:10.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.370+0000 7f3b5be00700 1 -- 172.21.15.62:0/3407369370 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f3b64196550 con 0x7f3b4806ecc0 2024-04-20T22:11:10.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.373+0000 7f3b69000700 1 -- 172.21.15.62:0/3407369370 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3b4806ecc0 msgr2=0x7f3b48071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:11:10.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.373+0000 7f3b69000700 1 --2- 172.21.15.62:0/3407369370 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3b4806ecc0 0x7f3b48071170 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f3b50006010 tx=0x7f3b5000b540 comp rx=0 tx=0).stop 2024-04-20T22:11:10.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.373+0000 7f3b69000700 1 -- 172.21.15.62:0/3407369370 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3b64068540 msgr2=0x7f3b6419b930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:11:10.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.373+0000 7f3b69000700 1 --2- 172.21.15.62:0/3407369370 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3b64068540 0x7f3b6419b930 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f3b5400c930 tx=0x7f3b5400ccf0 comp rx=0 tx=0).stop 2024-04-20T22:11:10.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.373+0000 7f3b69000700 1 -- 172.21.15.62:0/3407369370 shutdown_connections 2024-04-20T22:11:10.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.373+0000 7f3b69000700 1 --2- 172.21.15.62:0/3407369370 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3b4806ecc0 0x7f3b48071170 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:11:10.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.373+0000 7f3b69000700 1 --2- 172.21.15.62:0/3407369370 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3b64068540 0x7f3b6419b930 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:11:10.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.373+0000 7f3b69000700 1 --2- 172.21.15.62:0/3407369370 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b64106200 0x7f3b6419be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:11:10.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.373+0000 7f3b69000700 1 -- 172.21.15.62:0/3407369370 >> 172.21.15.62:0/3407369370 conn(0x7f3b64074fa0 msgr2=0x7f3b640fe700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:11:10.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.373+0000 7f3b69000700 1 -- 172.21.15.62:0/3407369370 shutdown_connections 2024-04-20T22:11:10.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.373+0000 7f3b69000700 1 -- 172.21.15.62:0/3407369370 wait complete. 2024-04-20T22:11:10.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.530+0000 7f631f200700 1 -- 172.21.15.62:0/1611775358 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6320102d40 msgr2=0x7f6320106d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:11:10.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.530+0000 7f631f200700 1 --2- 172.21.15.62:0/1611775358 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6320102d40 0x7f6320106d10 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f630c009910 tx=0x7f630c009c20 comp rx=0 tx=0).stop 2024-04-20T22:11:10.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.531+0000 7f631f200700 1 -- 172.21.15.62:0/1611775358 shutdown_connections 2024-04-20T22:11:10.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.531+0000 7f631f200700 1 --2- 172.21.15.62:0/1611775358 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6320102d40 0x7f6320106d10 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:11:10.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.531+0000 7f631f200700 1 --2- 172.21.15.62:0/1611775358 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63201023a0 0x7f6320102770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:11:10.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.531+0000 7f631f200700 1 -- 172.21.15.62:0/1611775358 >> 172.21.15.62:0/1611775358 conn(0x7f63200fdc80 msgr2=0x7f6320100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:11:10.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.531+0000 7f631f200700 1 -- 172.21.15.62:0/1611775358 shutdown_connections 2024-04-20T22:11:10.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.531+0000 7f631f200700 1 -- 172.21.15.62:0/1611775358 wait complete. 2024-04-20T22:11:10.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.532+0000 7f631f200700 1 Processor -- start 2024-04-20T22:11:10.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.532+0000 7f631f200700 1 -- start start 2024-04-20T22:11:10.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.532+0000 7f631f200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63201023a0 0x7f6320197d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:11:10.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.532+0000 7f631f200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6320102d40 0x7f63201982b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:11:10.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.532+0000 7f631f200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6320198990 con 0x7f6320102d40 2024-04-20T22:11:10.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.532+0000 7f631f200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f632019c720 con 0x7f63201023a0 2024-04-20T22:11:10.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.533+0000 7f631d400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6320102d40 0x7f63201982b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:11:10.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.533+0000 7f631d400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6320102d40 0x7f63201982b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34570/0 (socket says 172.21.15.62:34570) 2024-04-20T22:11:10.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.533+0000 7f631d400700 1 -- 172.21.15.62:0/282505356 learned_addr learned my addr 172.21.15.62:0/282505356 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:11:10.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.533+0000 7f631de00700 1 --2- 172.21.15.62:0/282505356 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63201023a0 0x7f6320197d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:11:10.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.533+0000 7f631d400700 1 -- 172.21.15.62:0/282505356 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63201023a0 msgr2=0x7f6320197d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:11:10.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.533+0000 7f631d400700 1 --2- 172.21.15.62:0/282505356 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63201023a0 0x7f6320197d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:11:10.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.533+0000 7f631d400700 1 -- 172.21.15.62:0/282505356 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f630c0095f0 con 0x7f6320102d40 2024-04-20T22:11:10.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.533+0000 7f631d400700 1 --2- 172.21.15.62:0/282505356 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6320102d40 0x7f63201982b0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f630c0092e0 tx=0x7f630c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:11:10.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.534+0000 7f6316a00700 1 -- 172.21.15.62:0/282505356 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f630c01d070 con 0x7f6320102d40 2024-04-20T22:11:10.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.534+0000 7f6316a00700 1 -- 172.21.15.62:0/282505356 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f630c005dc0 con 0x7f6320102d40 2024-04-20T22:11:10.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.534+0000 7f631f200700 1 -- 172.21.15.62:0/282505356 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f632019c9a0 con 0x7f6320102d40 2024-04-20T22:11:10.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.534+0000 7f6316a00700 1 -- 172.21.15.62:0/282505356 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f630c017400 con 0x7f6320102d40 2024-04-20T22:11:10.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.534+0000 7f631f200700 1 -- 172.21.15.62:0/282505356 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f632019ce90 con 0x7f6320102d40 2024-04-20T22:11:10.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.535+0000 7f631f200700 1 -- 172.21.15.62:0/282505356 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6320062380 con 0x7f6320102d40 2024-04-20T22:11:10.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.536+0000 7f6316a00700 1 -- 172.21.15.62:0/282505356 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f630c00f460 con 0x7f6320102d40 2024-04-20T22:11:10.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.537+0000 7f6316a00700 1 --2- 172.21.15.62:0/282505356 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f630406ecc0 0x7f6304071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:11:10.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.537+0000 7f6316a00700 1 -- 172.21.15.62:0/282505356 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f630c090420 con 0x7f6320102d40 2024-04-20T22:11:10.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.537+0000 7f631de00700 1 --2- 172.21.15.62:0/282505356 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f630406ecc0 0x7f6304071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:11:10.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.538+0000 7f631de00700 1 --2- 172.21.15.62:0/282505356 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f630406ecc0 0x7f6304071170 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f6310005950 tx=0x7f63100058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:11:10.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.542+0000 7f6316a00700 1 -- 172.21.15.62:0/282505356 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f630c059130 con 0x7f6320102d40 2024-04-20T22:11:10.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.786+0000 7f631f200700 1 -- 172.21.15.62:0/282505356 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f632019d370 con 0x7f630406ecc0 2024-04-20T22:11:10.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.819+0000 7f6316a00700 1 -- 172.21.15.62:0/282505356 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f632019d370 con 0x7f630406ecc0 2024-04-20T22:11:10.820 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T22:11:10.820 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (37m) 13m ago 50m 15.3M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T22:11:10.820 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (51m) 13m ago 51m 10.4M - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T22:11:10.820 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (49m) 13m ago 49m 10.1M - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T22:11:10.820 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (33m) 13m ago 51m 6660k - 19.0.0-3133-ga8c84f59 76bc6b91b726 2825d5ba11c6 2024-04-20T22:11:10.820 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (32m) 13m ago 49m 6665k - 19.0.0-3133-ga8c84f59 76bc6b91b726 20f746fa1bba 2024-04-20T22:11:10.820 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (36m) 13m ago 50m 73.5M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T22:11:10.820 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (28m) 13m ago 43m 18.5M - 19.0.0-3133-ga8c84f59 76bc6b91b726 de80bc90a683 2024-04-20T22:11:10.821 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (28m) 13m ago 43m 18.7M - 19.0.0-3133-ga8c84f59 76bc6b91b726 cd793b7ee98e 2024-04-20T22:11:10.821 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (13m) 13m ago 43m 11.0M - 19.0.0-3133-ga8c84f59 76bc6b91b726 0e7b0bc4d1ab 2024-04-20T22:11:10.821 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (43m) 13m ago 43m 18.6M - 18.2.0 dc2bc1663786 f2eec2ee8951 2024-04-20T22:11:10.821 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (39m) 13m ago 52m 513M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T22:11:10.821 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (38m) 13m ago 49m 459M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T22:11:10.821 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (34m) 13m ago 52m 71.7M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 019eb1479485 2024-04-20T22:11:10.821 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (34m) 13m ago 49m 51.0M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 55aac560aab6 2024-04-20T22:11:10.821 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (38m) 13m ago 51m 8472k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T22:11:10.821 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (38m) 13m ago 49m 8048k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T22:11:10.821 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (32m) 13m ago 48m 91.2M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 ee2bb901f26e 2024-04-20T22:11:10.821 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (31m) 13m ago 47m 94.6M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 9075152d4162 2024-04-20T22:11:10.821 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (31m) 13m ago 47m 78.1M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 2d4f3e38dc0e 2024-04-20T22:11:10.821 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (30m) 13m ago 46m 92.6M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 8f97b17ae0eb 2024-04-20T22:11:10.821 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (30m) 13m ago 46m 95.6M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 653d8079967d 2024-04-20T22:11:10.821 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (29m) 13m ago 45m 89.2M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 596dded14c12 2024-04-20T22:11:10.822 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (37m) 13m ago 50m 82.1M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T22:11:10.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.823+0000 7f631f200700 1 -- 172.21.15.62:0/282505356 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f630406ecc0 msgr2=0x7f6304071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:11:10.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.823+0000 7f631f200700 1 --2- 172.21.15.62:0/282505356 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f630406ecc0 0x7f6304071170 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f6310005950 tx=0x7f63100058e0 comp rx=0 tx=0).stop 2024-04-20T22:11:10.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.823+0000 7f631f200700 1 -- 172.21.15.62:0/282505356 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6320102d40 msgr2=0x7f63201982b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:11:10.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.823+0000 7f631f200700 1 --2- 172.21.15.62:0/282505356 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6320102d40 0x7f63201982b0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f630c0092e0 tx=0x7f630c00be20 comp rx=0 tx=0).stop 2024-04-20T22:11:10.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.824+0000 7f631f200700 1 -- 172.21.15.62:0/282505356 shutdown_connections 2024-04-20T22:11:10.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.824+0000 7f631f200700 1 --2- 172.21.15.62:0/282505356 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f630406ecc0 0x7f6304071170 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:11:10.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.824+0000 7f631f200700 1 --2- 172.21.15.62:0/282505356 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63201023a0 0x7f6320197d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:11:10.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.824+0000 7f631f200700 1 --2- 172.21.15.62:0/282505356 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6320102d40 0x7f63201982b0 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:11:10.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.824+0000 7f631f200700 1 -- 172.21.15.62:0/282505356 >> 172.21.15.62:0/282505356 conn(0x7f63200fdc80 msgr2=0x7f63200ff250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:11:10.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.824+0000 7f631f200700 1 -- 172.21.15.62:0/282505356 shutdown_connections 2024-04-20T22:11:10.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.824+0000 7f631f200700 1 -- 172.21.15.62:0/282505356 wait complete. 2024-04-20T22:11:10.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.984+0000 7f90baa00700 1 -- 172.21.15.62:0/1803032876 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90b41004e0 msgr2=0x7f90b41008b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:11:10.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.984+0000 7f90baa00700 1 --2- 172.21.15.62:0/1803032876 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90b41004e0 0x7f90b41008b0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f90a4009910 tx=0x7f90a4009c20 comp rx=0 tx=0).stop 2024-04-20T22:11:10.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.984+0000 7f90baa00700 1 -- 172.21.15.62:0/1803032876 shutdown_connections 2024-04-20T22:11:10.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.984+0000 7f90baa00700 1 --2- 172.21.15.62:0/1803032876 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f90b4100e80 0x7f90b4104e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:11:10.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.984+0000 7f90baa00700 1 --2- 172.21.15.62:0/1803032876 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90b41004e0 0x7f90b41008b0 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:11:10.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.984+0000 7f90baa00700 1 -- 172.21.15.62:0/1803032876 >> 172.21.15.62:0/1803032876 conn(0x7f90b40fbda0 msgr2=0x7f90b40fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:11:10.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.984+0000 7f90baa00700 1 -- 172.21.15.62:0/1803032876 shutdown_connections 2024-04-20T22:11:10.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.984+0000 7f90baa00700 1 -- 172.21.15.62:0/1803032876 wait complete. 2024-04-20T22:11:10.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.985+0000 7f90baa00700 1 Processor -- start 2024-04-20T22:11:10.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.986+0000 7f90baa00700 1 -- start start 2024-04-20T22:11:10.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.986+0000 7f90baa00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f90b41004e0 0x7f90b41980f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:11:10.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.986+0000 7f90baa00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90b4100e80 0x7f90b4198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:11:10.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.986+0000 7f90baa00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90b4198d10 con 0x7f90b4100e80 2024-04-20T22:11:10.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.986+0000 7f90baa00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90b419caa0 con 0x7f90b41004e0 2024-04-20T22:11:10.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.986+0000 7f90b3400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90b4100e80 0x7f90b4198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:11:10.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.986+0000 7f90b3400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90b4100e80 0x7f90b4198630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34586/0 (socket says 172.21.15.62:34586) 2024-04-20T22:11:10.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.986+0000 7f90b3400700 1 -- 172.21.15.62:0/854203483 learned_addr learned my addr 172.21.15.62:0/854203483 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:11:10.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.987+0000 7f90b3e00700 1 --2- 172.21.15.62:0/854203483 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f90b41004e0 0x7f90b41980f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:11:10.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.987+0000 7f90b3400700 1 -- 172.21.15.62:0/854203483 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f90b41004e0 msgr2=0x7f90b41980f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:11:10.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.987+0000 7f90b3400700 1 --2- 172.21.15.62:0/854203483 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f90b41004e0 0x7f90b41980f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:11:10.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.987+0000 7f90b3400700 1 -- 172.21.15.62:0/854203483 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90a40095f0 con 0x7f90b4100e80 2024-04-20T22:11:10.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.987+0000 7f90b3400700 1 --2- 172.21.15.62:0/854203483 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90b4100e80 0x7f90b4198630 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f90a000c8f0 tx=0x7f90a000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:11:10.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.987+0000 7f90b0c00700 1 -- 172.21.15.62:0/854203483 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f90a0007a50 con 0x7f90b4100e80 2024-04-20T22:11:10.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.988+0000 7f90b0c00700 1 -- 172.21.15.62:0/854203483 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f90a0007bb0 con 0x7f90b4100e80 2024-04-20T22:11:10.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.988+0000 7f90b0c00700 1 -- 172.21.15.62:0/854203483 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f90a0018630 con 0x7f90b4100e80 2024-04-20T22:11:10.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.988+0000 7f90baa00700 1 -- 172.21.15.62:0/854203483 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90b419cd80 con 0x7f90b4100e80 2024-04-20T22:11:10.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.988+0000 7f90baa00700 1 -- 172.21.15.62:0/854203483 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90b419d2d0 con 0x7f90b4100e80 2024-04-20T22:11:10.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.989+0000 7f90baa00700 1 -- 172.21.15.62:0/854203483 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90b41045d0 con 0x7f90b4100e80 2024-04-20T22:11:10.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.990+0000 7f90b0c00700 1 -- 172.21.15.62:0/854203483 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f90a0007d20 con 0x7f90b4100e80 2024-04-20T22:11:10.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.991+0000 7f90b0c00700 1 --2- 172.21.15.62:0/854203483 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f909806ecc0 0x7f9098071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:11:10.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.991+0000 7f90b0c00700 1 -- 172.21.15.62:0/854203483 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f90a008ee00 con 0x7f90b4100e80 2024-04-20T22:11:10.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.991+0000 7f90b3e00700 1 --2- 172.21.15.62:0/854203483 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f909806ecc0 0x7f9098071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:11:10.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.992+0000 7f90b3e00700 1 --2- 172.21.15.62:0/854203483 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f909806ecc0 0x7f9098071170 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f90a4006010 tx=0x7f90a401a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:11:10.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:10.996+0000 7f90b0c00700 1 -- 172.21.15.62:0/854203483 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f90a0057b90 con 0x7f90b4100e80 2024-04-20T22:11:11.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:11.322+0000 7f90baa00700 1 -- 172.21.15.62:0/854203483 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f90b419d5a0 con 0x7f90b4100e80 2024-04-20T22:11:11.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:11.323+0000 7f90b0c00700 1 -- 172.21.15.62:0/854203483 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7f90a005b250 con 0x7f90b4100e80 2024-04-20T22:11:11.324 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:11:11.324 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T22:11:11.324 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:11:11.324 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:11:11.324 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T22:11:11.324 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:11:11.324 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:11:11.324 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T22:11:11.324 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 6 2024-04-20T22:11:11.324 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:11:11.324 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T22:11:11.324 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 1, 2024-04-20T22:11:11.324 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 3 2024-04-20T22:11:11.325 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:11:11.325 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T22:11:11.325 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 1, 2024-04-20T22:11:11.325 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 13 2024-04-20T22:11:11.325 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T22:11:11.325 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:11:11.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:11.327+0000 7f90baa00700 1 -- 172.21.15.62:0/854203483 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f909806ecc0 msgr2=0x7f9098071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:11:11.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:11.327+0000 7f90baa00700 1 --2- 172.21.15.62:0/854203483 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f909806ecc0 0x7f9098071170 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f90a4006010 tx=0x7f90a401a040 comp rx=0 tx=0).stop 2024-04-20T22:11:11.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:11.327+0000 7f90baa00700 1 -- 172.21.15.62:0/854203483 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90b4100e80 msgr2=0x7f90b4198630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:11:11.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:11.327+0000 7f90baa00700 1 --2- 172.21.15.62:0/854203483 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90b4100e80 0x7f90b4198630 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f90a000c8f0 tx=0x7f90a000ccb0 comp rx=0 tx=0).stop 2024-04-20T22:11:11.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:11.327+0000 7f90baa00700 1 -- 172.21.15.62:0/854203483 shutdown_connections 2024-04-20T22:11:11.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:11.327+0000 7f90baa00700 1 --2- 172.21.15.62:0/854203483 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f909806ecc0 0x7f9098071170 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:11:11.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:11.327+0000 7f90baa00700 1 --2- 172.21.15.62:0/854203483 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f90b41004e0 0x7f90b41980f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:11:11.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:11.327+0000 7f90baa00700 1 --2- 172.21.15.62:0/854203483 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90b4100e80 0x7f90b4198630 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:11:11.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:11.327+0000 7f90baa00700 1 -- 172.21.15.62:0/854203483 >> 172.21.15.62:0/854203483 conn(0x7f90b40fbda0 msgr2=0x7f90b40fe160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:11:11.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:11.327+0000 7f90baa00700 1 -- 172.21.15.62:0/854203483 shutdown_connections 2024-04-20T22:11:11.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:11.327+0000 7f90baa00700 1 -- 172.21.15.62:0/854203483 wait complete. 2024-04-20T22:11:11.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:11.483+0000 7f05bb200700 1 -- 172.21.15.62:0/1121714447 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f05b4100e80 msgr2=0x7f05b4104e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:11:11.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:11.483+0000 7f05bb200700 1 --2- 172.21.15.62:0/1121714447 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f05b4100e80 0x7f05b4104e50 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f059c009910 tx=0x7f059c009c20 comp rx=0 tx=0).stop 2024-04-20T22:11:11.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:11.483+0000 7f05bb200700 1 -- 172.21.15.62:0/1121714447 shutdown_connections 2024-04-20T22:11:11.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:11.483+0000 7f05bb200700 1 --2- 172.21.15.62:0/1121714447 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f05b4100e80 0x7f05b4104e50 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:11:11.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:11.483+0000 7f05bb200700 1 --2- 172.21.15.62:0/1121714447 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05b41004e0 0x7f05b41008b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:11:11.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:11.483+0000 7f05bb200700 1 -- 172.21.15.62:0/1121714447 >> 172.21.15.62:0/1121714447 conn(0x7f05b40fbda0 msgr2=0x7f05b40fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:11:11.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:11.483+0000 7f05bb200700 1 -- 172.21.15.62:0/1121714447 shutdown_connections 2024-04-20T22:11:11.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:11.483+0000 7f05bb200700 1 -- 172.21.15.62:0/1121714447 wait complete. 2024-04-20T22:11:11.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:11.484+0000 7f05bb200700 1 Processor -- start 2024-04-20T22:11:11.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:11.485+0000 7f05bb200700 1 -- start start 2024-04-20T22:11:11.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:11.485+0000 7f05bb200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f05b41004e0 0x7f05b4073050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:11:11.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:11.485+0000 7f05bb200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05b4100e80 0x7f05b4073590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:11:11.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:11.485+0000 7f05bb200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05b4073ad0 con 0x7f05b41004e0 2024-04-20T22:11:11.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:11.485+0000 7f05bb200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05b4073c10 con 0x7f05b4100e80 2024-04-20T22:11:11.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:11.485+0000 7f05b3e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f05b41004e0 0x7f05b4073050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:11:11.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:11.485+0000 7f05b3e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f05b41004e0 0x7f05b4073050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:39748/0 (socket says 172.21.15.62:39748) 2024-04-20T22:11:11.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:11.485+0000 7f05b3e00700 1 -- 172.21.15.62:0/817446537 learned_addr learned my addr 172.21.15.62:0/817446537 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:11:11.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:11.486+0000 7f05ab400700 1 --2- 172.21.15.62:0/817446537 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05b4100e80 0x7f05b4073590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:11:11.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:11.486+0000 7f05b3e00700 1 -- 172.21.15.62:0/817446537 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05b4100e80 msgr2=0x7f05b4073590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:11:11.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:11.486+0000 7f05b3e00700 1 --2- 172.21.15.62:0/817446537 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05b4100e80 0x7f05b4073590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:11:11.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:11.486+0000 7f05b3e00700 1 -- 172.21.15.62:0/817446537 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f059c0095f0 con 0x7f05b41004e0 2024-04-20T22:11:11.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:11.486+0000 7f05ab400700 1 --2- 172.21.15.62:0/817446537 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05b4100e80 0x7f05b4073590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T22:11:11.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:11.486+0000 7f05b3e00700 1 --2- 172.21.15.62:0/817446537 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f05b41004e0 0x7f05b4073050 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f05a400ada0 tx=0x7f05a400c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:11:11.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:11.486+0000 7f05b1600700 1 -- 172.21.15.62:0/817446537 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f05a400f070 con 0x7f05b41004e0 2024-04-20T22:11:11.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:11.486+0000 7f05b1600700 1 -- 172.21.15.62:0/817446537 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f05a4007bf0 con 0x7f05b41004e0 2024-04-20T22:11:11.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:11.486+0000 7f05b1600700 1 -- 172.21.15.62:0/817446537 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f05a4005690 con 0x7f05b41004e0 2024-04-20T22:11:11.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:11.486+0000 7f05bb200700 1 -- 172.21.15.62:0/817446537 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05b41a2100 con 0x7f05b41004e0 2024-04-20T22:11:11.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:11.487+0000 7f05bb200700 1 -- 172.21.15.62:0/817446537 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05b41a24b0 con 0x7f05b41004e0 2024-04-20T22:11:11.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:11.488+0000 7f05bb200700 1 -- 172.21.15.62:0/817446537 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05b4108850 con 0x7f05b41004e0 2024-04-20T22:11:11.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:11.489+0000 7f05b1600700 1 -- 172.21.15.62:0/817446537 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f05a4005900 con 0x7f05b41004e0 2024-04-20T22:11:11.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:11.490+0000 7f05b1600700 1 --2- 172.21.15.62:0/817446537 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f059806ecc0 0x7f0598071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:11:11.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:11.490+0000 7f05b1600700 1 -- 172.21.15.62:0/817446537 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f05a408e920 con 0x7f05b41004e0 2024-04-20T22:11:11.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:11.490+0000 7f05ab400700 1 --2- 172.21.15.62:0/817446537 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f059806ecc0 0x7f0598071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:11:11.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:11.491+0000 7f05ab400700 1 --2- 172.21.15.62:0/817446537 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f059806ecc0 0x7f0598071170 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f059c006010 tx=0x7f059c00b540 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:11:11.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:11.496+0000 7f05b1600700 1 -- 172.21.15.62:0/817446537 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f05a4057630 con 0x7f05b41004e0 2024-04-20T22:11:11.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:11:11 smithi189 ceph-mon[57022]: pgmap v1077: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:11:11.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:11:11 smithi189 ceph-mon[57022]: from='client.35908 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:11:11.776 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:11:11 smithi062 ceph-mon[76322]: pgmap v1077: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:11:11.776 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:11:11 smithi062 ceph-mon[76322]: from='client.35908 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:11:11.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:11.776+0000 7f05bb200700 1 -- 172.21.15.62:0/817446537 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f05b4108230 con 0x7f05b41004e0 2024-04-20T22:11:11.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:11.777+0000 7f05b1600700 1 -- 172.21.15.62:0/817446537 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 27 v27) v1 ==== 76+0+1888 (secure 0 0 0) 0x7f05a405acf0 con 0x7f05b41004e0 2024-04-20T22:11:11.778 INFO:teuthology.orchestra.run.smithi062.stdout:e27 2024-04-20T22:11:11.778 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T22:11:11.778 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:11:11.778 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T22:11:11.779 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:11:11.779 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T22:11:11.779 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T22:11:11.779 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 20 2024-04-20T22:11:11.779 INFO:teuthology.orchestra.run.smithi062.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-20T22:11:11.779 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T22:11:11.780 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T21:42:01.559061+0000 2024-04-20T22:11:11.780 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T22:11:11.780 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T22:11:11.780 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T22:11:11.780 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T22:11:11.780 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T22:11:11.780 INFO:teuthology.orchestra.run.smithi062.stdout:max_xattr_size 65536 2024-04-20T22:11:11.780 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T22:11:11.780 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T22:11:11.780 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 81 2024-04-20T22:11:11.780 INFO:teuthology.orchestra.run.smithi062.stdout:snap_rstat 1 2024-04-20T22:11:11.781 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:11:11.781 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 2 2024-04-20T22:11:11.781 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1 2024-04-20T22:11:11.781 INFO:teuthology.orchestra.run.smithi062.stdout:up {} 2024-04-20T22:11:11.781 INFO:teuthology.orchestra.run.smithi062.stdout:failed 0,1 2024-04-20T22:11:11.781 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T22:11:11.781 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T22:11:11.782 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T22:11:11.782 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T22:11:11.782 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data enabled 2024-04-20T22:11:11.782 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T22:11:11.782 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T22:11:11.782 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 1 2024-04-20T22:11:11.782 INFO:teuthology.orchestra.run.smithi062.stdout:qdb_cluster leader: 0 members: 2024-04-20T22:11:11.782 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:11:11.782 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:11:11.782 INFO:teuthology.orchestra.run.smithi062.stdout:Standby daemons: 2024-04-20T22:11:11.782 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:11:11.782 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ryqblb{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6826/4113916205,v1:172.21.15.62:6827/4113916205] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:11:11.782 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ozizwi{-1:34476} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6828/3801975165,v1:172.21.15.62:6829/3801975165] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:11:11.782 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.xbgspw{-1:44285} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6826/3996040526,v1:172.21.15.189:6827/3996040526] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T22:11:11.783 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.dvimer{-1:44727} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6824/3576699232,v1:172.21.15.189:6825/3576699232] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:11:11.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:11.780+0000 7f05bb200700 1 -- 172.21.15.62:0/817446537 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f059806ecc0 msgr2=0x7f0598071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:11:11.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:11.780+0000 7f05bb200700 1 --2- 172.21.15.62:0/817446537 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f059806ecc0 0x7f0598071170 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f059c006010 tx=0x7f059c00b540 comp rx=0 tx=0).stop 2024-04-20T22:11:11.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:11.780+0000 7f05bb200700 1 -- 172.21.15.62:0/817446537 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f05b41004e0 msgr2=0x7f05b4073050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:11:11.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:11.780+0000 7f05bb200700 1 --2- 172.21.15.62:0/817446537 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f05b41004e0 0x7f05b4073050 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f05a400ada0 tx=0x7f05a400c330 comp rx=0 tx=0).stop 2024-04-20T22:11:11.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:11.780+0000 7f05bb200700 1 -- 172.21.15.62:0/817446537 shutdown_connections 2024-04-20T22:11:11.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:11.780+0000 7f05bb200700 1 --2- 172.21.15.62:0/817446537 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f059806ecc0 0x7f0598071170 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:11:11.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:11.780+0000 7f05bb200700 1 --2- 172.21.15.62:0/817446537 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f05b41004e0 0x7f05b4073050 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:11:11.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:11.780+0000 7f05bb200700 1 --2- 172.21.15.62:0/817446537 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05b4100e80 0x7f05b4073590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:11:11.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:11.780+0000 7f05bb200700 1 -- 172.21.15.62:0/817446537 >> 172.21.15.62:0/817446537 conn(0x7f05b40fbda0 msgr2=0x7f05b40fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:11:11.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:11.780+0000 7f05bb200700 1 -- 172.21.15.62:0/817446537 shutdown_connections 2024-04-20T22:11:11.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:11.780+0000 7f05bb200700 1 -- 172.21.15.62:0/817446537 wait complete. 2024-04-20T22:11:11.786 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 27 2024-04-20T22:11:11.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:11.937+0000 7f80eea00700 1 -- 172.21.15.62:0/150500179 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f80e8101230 msgr2=0x7f80e8101600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:11:11.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:11.937+0000 7f80eea00700 1 --2- 172.21.15.62:0/150500179 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f80e8101230 0x7f80e8101600 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f80dc009910 tx=0x7f80dc009c20 comp rx=0 tx=0).stop 2024-04-20T22:11:11.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:11.938+0000 7f80eea00700 1 -- 172.21.15.62:0/150500179 shutdown_connections 2024-04-20T22:11:11.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:11.938+0000 7f80eea00700 1 --2- 172.21.15.62:0/150500179 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f80e8101b40 0x7f80e810a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:11:11.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:11.938+0000 7f80eea00700 1 --2- 172.21.15.62:0/150500179 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f80e8101230 0x7f80e8101600 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:11:11.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:11.938+0000 7f80eea00700 1 -- 172.21.15.62:0/150500179 >> 172.21.15.62:0/150500179 conn(0x7f80e80faa90 msgr2=0x7f80e80fcea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:11:11.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:11.938+0000 7f80eea00700 1 -- 172.21.15.62:0/150500179 shutdown_connections 2024-04-20T22:11:11.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:11.938+0000 7f80eea00700 1 -- 172.21.15.62:0/150500179 wait complete. 2024-04-20T22:11:11.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:11.939+0000 7f80eea00700 1 Processor -- start 2024-04-20T22:11:11.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:11.939+0000 7f80eea00700 1 -- start start 2024-04-20T22:11:11.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:11.940+0000 7f80eea00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f80e8101230 0x7f80e819b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:11:11.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:11.940+0000 7f80eea00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f80e8101b40 0x7f80e819bf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:11:11.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:11.940+0000 7f80eea00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80e819c5c0 con 0x7f80e8101230 2024-04-20T22:11:11.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:11.940+0000 7f80eea00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80e8195a70 con 0x7f80e8101b40 2024-04-20T22:11:11.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:11.940+0000 7f80ed600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f80e8101230 0x7f80e819b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:11:11.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:11.940+0000 7f80ed600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f80e8101230 0x7f80e819b9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:39760/0 (socket says 172.21.15.62:39760) 2024-04-20T22:11:11.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:11.940+0000 7f80ed600700 1 -- 172.21.15.62:0/3624204979 learned_addr learned my addr 172.21.15.62:0/3624204979 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:11:11.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:11.940+0000 7f80ecc00700 1 --2- 172.21.15.62:0/3624204979 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f80e8101b40 0x7f80e819bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:11:11.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:11.940+0000 7f80ed600700 1 -- 172.21.15.62:0/3624204979 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f80e8101b40 msgr2=0x7f80e819bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:11:11.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:11.940+0000 7f80ed600700 1 --2- 172.21.15.62:0/3624204979 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f80e8101b40 0x7f80e819bf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:11:11.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:11.940+0000 7f80ed600700 1 -- 172.21.15.62:0/3624204979 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80dc0095f0 con 0x7f80e8101230 2024-04-20T22:11:11.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:11.941+0000 7f80ed600700 1 --2- 172.21.15.62:0/3624204979 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f80e8101230 0x7f80e819b9f0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f80dc006010 tx=0x7f80dc00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:11:11.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:11.941+0000 7f80e6000700 1 -- 172.21.15.62:0/3624204979 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f80dc01d070 con 0x7f80e8101230 2024-04-20T22:11:11.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:11.941+0000 7f80e6000700 1 -- 172.21.15.62:0/3624204979 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f80dc00f460 con 0x7f80e8101230 2024-04-20T22:11:11.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:11.941+0000 7f80e6000700 1 -- 172.21.15.62:0/3624204979 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f80dc017630 con 0x7f80e8101230 2024-04-20T22:11:11.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:11.941+0000 7f80eea00700 1 -- 172.21.15.62:0/3624204979 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80e8195cf0 con 0x7f80e8101230 2024-04-20T22:11:11.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:11.941+0000 7f80eea00700 1 -- 172.21.15.62:0/3624204979 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80e81961e0 con 0x7f80e8101230 2024-04-20T22:11:11.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:11.942+0000 7f80eea00700 1 -- 172.21.15.62:0/3624204979 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80e8107710 con 0x7f80e8101230 2024-04-20T22:11:11.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:11.943+0000 7f80e6000700 1 -- 172.21.15.62:0/3624204979 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f80dc00f5d0 con 0x7f80e8101230 2024-04-20T22:11:11.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:11.944+0000 7f80e6000700 1 --2- 172.21.15.62:0/3624204979 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f80d006ecc0 0x7f80d0071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:11:11.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:11.944+0000 7f80e6000700 1 -- 172.21.15.62:0/3624204979 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f80dc0903b0 con 0x7f80e8101230 2024-04-20T22:11:11.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:11.944+0000 7f80ecc00700 1 --2- 172.21.15.62:0/3624204979 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f80d006ecc0 0x7f80d0071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:11:11.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:11.945+0000 7f80ecc00700 1 --2- 172.21.15.62:0/3624204979 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f80d006ecc0 0x7f80d0071170 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f80d4005950 tx=0x7f80d40058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:11:11.949 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:11.949+0000 7f80e6000700 1 -- 172.21.15.62:0/3624204979 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f80dc05cbf0 con 0x7f80e8101230 2024-04-20T22:11:12.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:12.205+0000 7f80eea00700 1 -- 172.21.15.62:0/3624204979 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f80e81964c0 con 0x7f80d006ecc0 2024-04-20T22:11:12.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:12.208+0000 7f80e6000700 1 -- 172.21.15.62:0/3624204979 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f80e81964c0 con 0x7f80d006ecc0 2024-04-20T22:11:12.208 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:11:12.208 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T22:11:12.208 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T22:11:12.208 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-20T22:11:12.208 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T22:11:12.208 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr", 2024-04-20T22:11:12.208 INFO:teuthology.orchestra.run.smithi062.stdout: "crash", 2024-04-20T22:11:12.208 INFO:teuthology.orchestra.run.smithi062.stdout: "mon", 2024-04-20T22:11:12.208 INFO:teuthology.orchestra.run.smithi062.stdout: "osd" 2024-04-20T22:11:12.209 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T22:11:12.209 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "15/23 daemons upgraded", 2024-04-20T22:11:12.209 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading mds daemons", 2024-04-20T22:11:12.209 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T22:11:12.209 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:11:12.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:12.211+0000 7f80eea00700 1 -- 172.21.15.62:0/3624204979 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f80d006ecc0 msgr2=0x7f80d0071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:11:12.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:12.211+0000 7f80eea00700 1 --2- 172.21.15.62:0/3624204979 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f80d006ecc0 0x7f80d0071170 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f80d4005950 tx=0x7f80d40058e0 comp rx=0 tx=0).stop 2024-04-20T22:11:12.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:12.211+0000 7f80eea00700 1 -- 172.21.15.62:0/3624204979 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f80e8101230 msgr2=0x7f80e819b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:11:12.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:12.211+0000 7f80eea00700 1 --2- 172.21.15.62:0/3624204979 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f80e8101230 0x7f80e819b9f0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f80dc006010 tx=0x7f80dc00be20 comp rx=0 tx=0).stop 2024-04-20T22:11:12.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:12.211+0000 7f80eea00700 1 -- 172.21.15.62:0/3624204979 shutdown_connections 2024-04-20T22:11:12.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:12.211+0000 7f80eea00700 1 --2- 172.21.15.62:0/3624204979 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f80d006ecc0 0x7f80d0071170 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:11:12.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:12.211+0000 7f80eea00700 1 --2- 172.21.15.62:0/3624204979 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f80e8101230 0x7f80e819b9f0 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:11:12.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:12.211+0000 7f80eea00700 1 --2- 172.21.15.62:0/3624204979 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f80e8101b40 0x7f80e819bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:11:12.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:12.211+0000 7f80eea00700 1 -- 172.21.15.62:0/3624204979 >> 172.21.15.62:0/3624204979 conn(0x7f80e80faa90 msgr2=0x7f80e8104850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:11:12.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:12.211+0000 7f80eea00700 1 -- 172.21.15.62:0/3624204979 shutdown_connections 2024-04-20T22:11:12.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:12.211+0000 7f80eea00700 1 -- 172.21.15.62:0/3624204979 wait complete. 2024-04-20T22:11:12.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:12.376+0000 7f5794c00700 1 -- 172.21.15.62:0/3810885928 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5790106b80 msgr2=0x7f5790106ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:11:12.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:12.376+0000 7f5794c00700 1 --2- 172.21.15.62:0/3810885928 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5790106b80 0x7f5790106ff0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f577c009910 tx=0x7f577c009c20 comp rx=0 tx=0).stop 2024-04-20T22:11:12.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:12.376+0000 7f5794c00700 1 -- 172.21.15.62:0/3810885928 shutdown_connections 2024-04-20T22:11:12.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:12.376+0000 7f5794c00700 1 --2- 172.21.15.62:0/3810885928 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5790106b80 0x7f5790106ff0 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:11:12.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:12.376+0000 7f5794c00700 1 --2- 172.21.15.62:0/3810885928 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f579010c660 0x7f579010ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:11:12.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:12.376+0000 7f5794c00700 1 -- 172.21.15.62:0/3810885928 >> 172.21.15.62:0/3810885928 conn(0x7f5790075e90 msgr2=0x7f57900782a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:11:12.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:12.377+0000 7f5794c00700 1 -- 172.21.15.62:0/3810885928 shutdown_connections 2024-04-20T22:11:12.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:12.377+0000 7f5794c00700 1 -- 172.21.15.62:0/3810885928 wait complete. 2024-04-20T22:11:12.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:12.378+0000 7f5794c00700 1 Processor -- start 2024-04-20T22:11:12.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:12.378+0000 7f5794c00700 1 -- start start 2024-04-20T22:11:12.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:12.378+0000 7f5794c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5790106b80 0x7f579019c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:11:12.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:12.378+0000 7f5794c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f579010c660 0x7f579019c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:11:12.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:12.378+0000 7f5794c00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f579019ce00 con 0x7f5790106b80 2024-04-20T22:11:12.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:12.378+0000 7f5794c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57901a0b90 con 0x7f579010c660 2024-04-20T22:11:12.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:12.379+0000 7f578f400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5790106b80 0x7f579019c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:11:12.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:12.379+0000 7f578f400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5790106b80 0x7f579019c1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:39774/0 (socket says 172.21.15.62:39774) 2024-04-20T22:11:12.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:12.379+0000 7f578f400700 1 -- 172.21.15.62:0/3415859427 learned_addr learned my addr 172.21.15.62:0/3415859427 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:11:12.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:12.379+0000 7f578ea00700 1 --2- 172.21.15.62:0/3415859427 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f579010c660 0x7f579019c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:11:12.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:12.379+0000 7f578f400700 1 -- 172.21.15.62:0/3415859427 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f579010c660 msgr2=0x7f579019c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:11:12.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:12.379+0000 7f578f400700 1 --2- 172.21.15.62:0/3415859427 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f579010c660 0x7f579019c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:11:12.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:12.379+0000 7f578f400700 1 -- 172.21.15.62:0/3415859427 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f577c0095f0 con 0x7f5790106b80 2024-04-20T22:11:12.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:12.379+0000 7f578ea00700 1 --2- 172.21.15.62:0/3415859427 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f579010c660 0x7f579019c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T22:11:12.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:12.380+0000 7f578f400700 1 --2- 172.21.15.62:0/3415859427 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5790106b80 0x7f579019c1e0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f578000d8d0 tx=0x7f578000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:11:12.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:12.380+0000 7f5787e00700 1 -- 172.21.15.62:0/3415859427 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5780009940 con 0x7f5790106b80 2024-04-20T22:11:12.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:12.380+0000 7f5787e00700 1 -- 172.21.15.62:0/3415859427 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5780010460 con 0x7f5790106b80 2024-04-20T22:11:12.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:12.380+0000 7f5794c00700 1 -- 172.21.15.62:0/3415859427 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57901a0e70 con 0x7f5790106b80 2024-04-20T22:11:12.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:12.380+0000 7f5787e00700 1 -- 172.21.15.62:0/3415859427 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f578000f5f0 con 0x7f5790106b80 2024-04-20T22:11:12.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:12.380+0000 7f5794c00700 1 -- 172.21.15.62:0/3415859427 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57901a13c0 con 0x7f5790106b80 2024-04-20T22:11:12.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:12.381+0000 7f5794c00700 1 -- 172.21.15.62:0/3415859427 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f579010eb40 con 0x7f5790106b80 2024-04-20T22:11:12.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:12.383+0000 7f5787e00700 1 -- 172.21.15.62:0/3415859427 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f57800105d0 con 0x7f5790106b80 2024-04-20T22:11:12.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:12.384+0000 7f5787e00700 1 --2- 172.21.15.62:0/3415859427 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f577406ecc0 0x7f5774071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:11:12.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:12.384+0000 7f5787e00700 1 -- 172.21.15.62:0/3415859427 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f578008ef20 con 0x7f5790106b80 2024-04-20T22:11:12.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:12.384+0000 7f578ea00700 1 --2- 172.21.15.62:0/3415859427 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f577406ecc0 0x7f5774071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:11:12.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:12.385+0000 7f578ea00700 1 --2- 172.21.15.62:0/3415859427 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f577406ecc0 0x7f5774071170 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f577c00b5c0 tx=0x7f577c0053a0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:11:12.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:12.389+0000 7f5787e00700 1 -- 172.21.15.62:0/3415859427 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f5780057c30 con 0x7f5790106b80 2024-04-20T22:11:12.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:11:12 smithi062 ceph-mon[76322]: from='client.35912 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:11:12.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:11:12 smithi062 ceph-mon[76322]: from='client.35916 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:11:12.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:11:12 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/854203483' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:11:12.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:11:12 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/817446537' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:11:12.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:12.727+0000 7f5794c00700 1 -- 172.21.15.62:0/3415859427 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f5790062380 con 0x7f5790106b80 2024-04-20T22:11:12.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:12.728+0000 7f5787e00700 1 -- 172.21.15.62:0/3415859427 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7f578005b2f0 con 0x7f5790106b80 2024-04-20T22:11:12.728 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-20T22:11:12.728 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-20T22:11:12.728 INFO:teuthology.orchestra.run.smithi062.stdout: host smithi062 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T22:11:12.728 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-20T22:11:12.728 INFO:teuthology.orchestra.run.smithi062.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi062 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpq7y7oa_r:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:11:12.728 INFO:teuthology.orchestra.run.smithi062.stdout:Traceback (most recent call last): 2024-04-20T22:11:12.728 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-20T22:11:12.728 INFO:teuthology.orchestra.run.smithi062.stdout: return _run_code(code, main_globals, None, 2024-04-20T22:11:12.728 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-20T22:11:12.728 INFO:teuthology.orchestra.run.smithi062.stdout: exec(code, run_globals) 2024-04-20T22:11:12.729 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5618, in 2024-04-20T22:11:12.729 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5606, in main 2024-04-20T22:11:12.729 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 408, in _infer_config 2024-04-20T22:11:12.729 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 323, in _infer_fsid 2024-04-20T22:11:12.729 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 436, in _infer_image 2024-04-20T22:11:12.729 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 310, in _validate_fsid 2024-04-20T22:11:12.729 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 3359, in command_ceph_volume 2024-04-20T22:11:12.729 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-20T22:11:12.729 INFO:teuthology.orchestra.run.smithi062.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi062 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpq7y7oa_r:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:11:12.729 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-20T22:11:12.729 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is degraded 2024-04-20T22:11:12.729 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T22:11:12.729 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T22:11:12.729 INFO:teuthology.orchestra.run.smithi062.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-20T22:11:12.730 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-20T22:11:12.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:12.730+0000 7f5794c00700 1 -- 172.21.15.62:0/3415859427 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f577406ecc0 msgr2=0x7f5774071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:11:12.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:12.730+0000 7f5794c00700 1 --2- 172.21.15.62:0/3415859427 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f577406ecc0 0x7f5774071170 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f577c00b5c0 tx=0x7f577c0053a0 comp rx=0 tx=0).stop 2024-04-20T22:11:12.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:12.730+0000 7f5794c00700 1 -- 172.21.15.62:0/3415859427 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5790106b80 msgr2=0x7f579019c1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:11:12.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:12.730+0000 7f5794c00700 1 --2- 172.21.15.62:0/3415859427 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5790106b80 0x7f579019c1e0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f578000d8d0 tx=0x7f578000dc90 comp rx=0 tx=0).stop 2024-04-20T22:11:12.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:12.731+0000 7f5794c00700 1 -- 172.21.15.62:0/3415859427 shutdown_connections 2024-04-20T22:11:12.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:12.731+0000 7f5794c00700 1 --2- 172.21.15.62:0/3415859427 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f577406ecc0 0x7f5774071170 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:11:12.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:12.731+0000 7f5794c00700 1 --2- 172.21.15.62:0/3415859427 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5790106b80 0x7f579019c1e0 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:11:12.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:12.731+0000 7f5794c00700 1 --2- 172.21.15.62:0/3415859427 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f579010c660 0x7f579019c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:11:12.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:12.731+0000 7f5794c00700 1 -- 172.21.15.62:0/3415859427 >> 172.21.15.62:0/3415859427 conn(0x7f5790075e90 msgr2=0x7f57900778d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:11:12.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:12.731+0000 7f5794c00700 1 -- 172.21.15.62:0/3415859427 shutdown_connections 2024-04-20T22:11:12.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:12.731+0000 7f5794c00700 1 -- 172.21.15.62:0/3415859427 wait complete. 2024-04-20T22:11:12.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:11:12 smithi189 ceph-mon[57022]: from='client.35912 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:11:12.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:11:12 smithi189 ceph-mon[57022]: from='client.35916 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:11:12.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:11:12 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/854203483' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:11:12.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:11:12 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/817446537' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:11:13.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:11:13 smithi189 ceph-mon[57022]: pgmap v1078: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:11:13.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:11:13 smithi189 ceph-mon[57022]: from='client.35928 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:11:13.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:11:13 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/3415859427' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:11:13.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:11:13 smithi062 ceph-mon[76322]: pgmap v1078: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:11:13.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:11:13 smithi062 ceph-mon[76322]: from='client.35928 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:11:13.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:11:13 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/3415859427' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:11:14.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:11:14 smithi189 ceph-mon[57022]: pgmap v1079: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:11:14.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:11:14 smithi062 ceph-mon[76322]: pgmap v1079: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:11:17.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:11:16 smithi189 ceph-mon[57022]: pgmap v1080: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:11:17.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:11:16 smithi062 ceph-mon[76322]: pgmap v1080: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:11:19.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:11:18 smithi189 ceph-mon[57022]: pgmap v1081: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:11:19.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:11:18 smithi062 ceph-mon[76322]: pgmap v1081: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:11:20.076 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:11:19 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:11:20.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:11:19 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:11:21.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:11:20 smithi189 ceph-mon[57022]: pgmap v1082: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:11:21.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:11:20 smithi062 ceph-mon[76322]: pgmap v1082: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:11:23.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:11:22 smithi189 ceph-mon[57022]: pgmap v1083: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:11:23.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:11:22 smithi062 ceph-mon[76322]: pgmap v1083: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:11:25.235 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:11:24 smithi189 ceph-mon[57022]: pgmap v1084: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:11:25.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:11:24 smithi062 ceph-mon[76322]: pgmap v1084: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:11:27.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:11:26 smithi189 ceph-mon[57022]: pgmap v1085: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:11:27.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:11:26 smithi062 ceph-mon[76322]: pgmap v1085: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:11:29.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:11:28 smithi189 ceph-mon[57022]: pgmap v1086: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:11:29.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:11:28 smithi062 ceph-mon[76322]: pgmap v1086: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:11:31.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:11:30 smithi189 ceph-mon[57022]: pgmap v1087: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:11:31.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:11:30 smithi062 ceph-mon[76322]: pgmap v1087: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:11:33.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:11:32 smithi189 ceph-mon[57022]: pgmap v1088: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:11:33.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:11:32 smithi062 ceph-mon[76322]: pgmap v1088: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:11:35.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:11:34 smithi189 ceph-mon[57022]: pgmap v1089: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:11:35.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:11:34 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:11:35.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:11:34 smithi062 ceph-mon[76322]: pgmap v1089: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:11:35.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:11:34 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:11:37.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:11:36 smithi189 ceph-mon[57022]: pgmap v1090: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:11:37.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:11:36 smithi062 ceph-mon[76322]: pgmap v1090: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:11:39.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:11:38 smithi189 ceph-mon[57022]: pgmap v1091: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:11:39.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:11:38 smithi062 ceph-mon[76322]: pgmap v1091: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:11:40.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:11:40 smithi189 ceph-mon[57022]: pgmap v1092: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:11:41.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:11:40 smithi062 ceph-mon[76322]: pgmap v1092: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:11:42.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:42.910+0000 7f5183e00700 1 -- 172.21.15.62:0/320366805 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f517c100e80 msgr2=0x7f517c104e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:11:42.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:42.910+0000 7f5183e00700 1 --2- 172.21.15.62:0/320366805 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f517c100e80 0x7f517c104e50 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f5168009910 tx=0x7f5168009c20 comp rx=0 tx=0).stop 2024-04-20T22:11:42.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:42.910+0000 7f5183e00700 1 -- 172.21.15.62:0/320366805 shutdown_connections 2024-04-20T22:11:42.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:42.910+0000 7f5183e00700 1 --2- 172.21.15.62:0/320366805 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f517c100e80 0x7f517c104e50 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:11:42.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:42.910+0000 7f5183e00700 1 --2- 172.21.15.62:0/320366805 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f517c1004e0 0x7f517c1008b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:11:42.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:42.910+0000 7f5183e00700 1 -- 172.21.15.62:0/320366805 >> 172.21.15.62:0/320366805 conn(0x7f517c0fbda0 msgr2=0x7f517c0fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:11:42.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:42.910+0000 7f5183e00700 1 -- 172.21.15.62:0/320366805 shutdown_connections 2024-04-20T22:11:42.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:42.910+0000 7f5183e00700 1 -- 172.21.15.62:0/320366805 wait complete. 2024-04-20T22:11:42.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:42.911+0000 7f5183e00700 1 Processor -- start 2024-04-20T22:11:42.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:42.911+0000 7f5183e00700 1 -- start start 2024-04-20T22:11:42.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:42.912+0000 7f5183e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f517c1004e0 0x7f517c073050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:11:42.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:42.912+0000 7f5183e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f517c100e80 0x7f517c073590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:11:42.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:42.912+0000 7f5183e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f517c073ad0 con 0x7f517c100e80 2024-04-20T22:11:42.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:42.912+0000 7f5183e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f517c073c10 con 0x7f517c1004e0 2024-04-20T22:11:42.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:42.912+0000 7f5180a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f517c100e80 0x7f517c073590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:11:42.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:42.912+0000 7f5180a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f517c100e80 0x7f517c073590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55922/0 (socket says 172.21.15.62:55922) 2024-04-20T22:11:42.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:42.912+0000 7f5180a00700 1 -- 172.21.15.62:0/289033636 learned_addr learned my addr 172.21.15.62:0/289033636 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:11:42.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:42.913+0000 7f5181400700 1 --2- 172.21.15.62:0/289033636 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f517c1004e0 0x7f517c073050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:11:42.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:42.913+0000 7f5180a00700 1 -- 172.21.15.62:0/289033636 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f517c1004e0 msgr2=0x7f517c073050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:11:42.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:42.913+0000 7f5180a00700 1 --2- 172.21.15.62:0/289033636 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f517c1004e0 0x7f517c073050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:11:42.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:42.913+0000 7f5180a00700 1 -- 172.21.15.62:0/289033636 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51680095f0 con 0x7f517c100e80 2024-04-20T22:11:42.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:42.913+0000 7f5180a00700 1 --2- 172.21.15.62:0/289033636 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f517c100e80 0x7f517c073590 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f516800bda0 tx=0x7f516800be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:11:42.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:42.913+0000 7f517a000700 1 -- 172.21.15.62:0/289033636 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f516801d070 con 0x7f517c100e80 2024-04-20T22:11:42.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:42.913+0000 7f5183e00700 1 -- 172.21.15.62:0/289033636 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f517c073e90 con 0x7f517c100e80 2024-04-20T22:11:42.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:42.913+0000 7f517a000700 1 -- 172.21.15.62:0/289033636 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5168004fb0 con 0x7f517c100e80 2024-04-20T22:11:42.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:42.914+0000 7f517a000700 1 -- 172.21.15.62:0/289033636 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f516800f480 con 0x7f517c100e80 2024-04-20T22:11:42.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:42.914+0000 7f5183e00700 1 -- 172.21.15.62:0/289033636 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f517c1a23c0 con 0x7f517c100e80 2024-04-20T22:11:42.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:42.915+0000 7f5183e00700 1 -- 172.21.15.62:0/289033636 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f517c108850 con 0x7f517c100e80 2024-04-20T22:11:42.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:42.916+0000 7f517a000700 1 -- 172.21.15.62:0/289033636 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f5168022870 con 0x7f517c100e80 2024-04-20T22:11:42.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:42.917+0000 7f517a000700 1 --2- 172.21.15.62:0/289033636 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f516406ecc0 0x7f5164071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:11:42.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:42.917+0000 7f517a000700 1 -- 172.21.15.62:0/289033636 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f5168090370 con 0x7f517c100e80 2024-04-20T22:11:42.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:42.917+0000 7f5181400700 1 --2- 172.21.15.62:0/289033636 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f516406ecc0 0x7f5164071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:11:42.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:42.918+0000 7f5181400700 1 --2- 172.21.15.62:0/289033636 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f516406ecc0 0x7f5164071170 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f5170005ea0 tx=0x7f5170005e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:11:42.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:42.922+0000 7f517a000700 1 -- 172.21.15.62:0/289033636 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f51680591b0 con 0x7f517c100e80 2024-04-20T22:11:43.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:43.178+0000 7f5183e00700 1 -- 172.21.15.62:0/289033636 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f517c074690 con 0x7f516406ecc0 2024-04-20T22:11:43.178 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:11:42 smithi062 ceph-mon[76322]: pgmap v1093: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:11:43.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:43.181+0000 7f517a000700 1 -- 172.21.15.62:0/289033636 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f517c074690 con 0x7f516406ecc0 2024-04-20T22:11:43.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:43.184+0000 7f5183e00700 1 -- 172.21.15.62:0/289033636 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f516406ecc0 msgr2=0x7f5164071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:11:43.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:43.185+0000 7f5183e00700 1 --2- 172.21.15.62:0/289033636 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f516406ecc0 0x7f5164071170 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f5170005ea0 tx=0x7f5170005e30 comp rx=0 tx=0).stop 2024-04-20T22:11:43.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:43.185+0000 7f5183e00700 1 -- 172.21.15.62:0/289033636 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f517c100e80 msgr2=0x7f517c073590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:11:43.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:43.185+0000 7f5183e00700 1 --2- 172.21.15.62:0/289033636 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f517c100e80 0x7f517c073590 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f516800bda0 tx=0x7f516800be80 comp rx=0 tx=0).stop 2024-04-20T22:11:43.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:43.185+0000 7f5183e00700 1 -- 172.21.15.62:0/289033636 shutdown_connections 2024-04-20T22:11:43.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:43.185+0000 7f5183e00700 1 --2- 172.21.15.62:0/289033636 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f516406ecc0 0x7f5164071170 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:11:43.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:43.185+0000 7f5183e00700 1 --2- 172.21.15.62:0/289033636 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f517c1004e0 0x7f517c073050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:11:43.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:43.185+0000 7f5183e00700 1 --2- 172.21.15.62:0/289033636 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f517c100e80 0x7f517c073590 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:11:43.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:43.185+0000 7f5183e00700 1 -- 172.21.15.62:0/289033636 >> 172.21.15.62:0/289033636 conn(0x7f517c0fbda0 msgr2=0x7f517c0fe180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:11:43.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:43.185+0000 7f5183e00700 1 -- 172.21.15.62:0/289033636 shutdown_connections 2024-04-20T22:11:43.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:43.185+0000 7f5183e00700 1 -- 172.21.15.62:0/289033636 wait complete. 2024-04-20T22:11:43.201 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T22:11:43.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:11:42 smithi189 ceph-mon[57022]: pgmap v1093: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:11:43.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:43.352+0000 7f32cfc00700 1 -- 172.21.15.62:0/3885120917 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f32c810c8f0 msgr2=0x7f32c810ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:11:43.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:43.352+0000 7f32cfc00700 1 --2- 172.21.15.62:0/3885120917 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f32c810c8f0 0x7f32c810ccc0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f32bc009910 tx=0x7f32bc009c20 comp rx=0 tx=0).stop 2024-04-20T22:11:43.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:43.352+0000 7f32cfc00700 1 -- 172.21.15.62:0/3885120917 shutdown_connections 2024-04-20T22:11:43.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:43.352+0000 7f32cfc00700 1 --2- 172.21.15.62:0/3885120917 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f32c8106b80 0x7f32c8106ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:11:43.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:43.352+0000 7f32cfc00700 1 --2- 172.21.15.62:0/3885120917 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f32c810c8f0 0x7f32c810ccc0 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:11:43.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:43.352+0000 7f32cfc00700 1 -- 172.21.15.62:0/3885120917 >> 172.21.15.62:0/3885120917 conn(0x7f32c8076060 msgr2=0x7f32c8078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:11:43.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:43.352+0000 7f32cfc00700 1 -- 172.21.15.62:0/3885120917 shutdown_connections 2024-04-20T22:11:43.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:43.352+0000 7f32cfc00700 1 -- 172.21.15.62:0/3885120917 wait complete. 2024-04-20T22:11:43.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:43.353+0000 7f32cfc00700 1 Processor -- start 2024-04-20T22:11:43.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:43.354+0000 7f32cfc00700 1 -- start start 2024-04-20T22:11:43.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:43.354+0000 7f32cfc00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f32c8106b80 0x7f32c819c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:11:43.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:43.354+0000 7f32cfc00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f32c810c8f0 0x7f32c819ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:11:43.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:43.354+0000 7f32cfc00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32c819d120 con 0x7f32c8106b80 2024-04-20T22:11:43.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:43.354+0000 7f32cfc00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32c81a0eb0 con 0x7f32c810c8f0 2024-04-20T22:11:43.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:43.354+0000 7f32cd200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f32c8106b80 0x7f32c819c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:11:43.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:43.354+0000 7f32cd200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f32c8106b80 0x7f32c819c500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55942/0 (socket says 172.21.15.62:55942) 2024-04-20T22:11:43.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:43.354+0000 7f32cd200700 1 -- 172.21.15.62:0/1552235569 learned_addr learned my addr 172.21.15.62:0/1552235569 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:11:43.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:43.355+0000 7f32cd200700 1 -- 172.21.15.62:0/1552235569 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f32c810c8f0 msgr2=0x7f32c819ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:11:43.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:43.355+0000 7f32cd200700 1 --2- 172.21.15.62:0/1552235569 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f32c810c8f0 0x7f32c819ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:11:43.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:43.355+0000 7f32cd200700 1 -- 172.21.15.62:0/1552235569 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32bc0095f0 con 0x7f32c8106b80 2024-04-20T22:11:43.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:43.355+0000 7f32cd200700 1 --2- 172.21.15.62:0/1552235569 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f32c8106b80 0x7f32c819c500 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f32bc0092e0 tx=0x7f32bc00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:11:43.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:43.355+0000 7f32c5600700 1 -- 172.21.15.62:0/1552235569 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f32bc01d070 con 0x7f32c8106b80 2024-04-20T22:11:43.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:43.355+0000 7f32c5600700 1 -- 172.21.15.62:0/1552235569 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f32bc005e00 con 0x7f32c8106b80 2024-04-20T22:11:43.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:43.355+0000 7f32c5600700 1 -- 172.21.15.62:0/1552235569 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f32bc017400 con 0x7f32c8106b80 2024-04-20T22:11:43.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:43.355+0000 7f32cfc00700 1 -- 172.21.15.62:0/1552235569 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32c81a1130 con 0x7f32c8106b80 2024-04-20T22:11:43.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:43.356+0000 7f32cfc00700 1 -- 172.21.15.62:0/1552235569 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32c81a1620 con 0x7f32c8106b80 2024-04-20T22:11:43.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:43.357+0000 7f32cfc00700 1 -- 172.21.15.62:0/1552235569 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32c8109b20 con 0x7f32c8106b80 2024-04-20T22:11:43.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:43.358+0000 7f32c5600700 1 -- 172.21.15.62:0/1552235569 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f32bc00f460 con 0x7f32c8106b80 2024-04-20T22:11:43.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:43.359+0000 7f32c5600700 1 --2- 172.21.15.62:0/1552235569 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f32b006ecc0 0x7f32b0071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:11:43.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:43.359+0000 7f32c5600700 1 -- 172.21.15.62:0/1552235569 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f32bc090420 con 0x7f32c8106b80 2024-04-20T22:11:43.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:43.359+0000 7f32c7e00700 1 --2- 172.21.15.62:0/1552235569 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f32b006ecc0 0x7f32b0071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:11:43.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:43.360+0000 7f32c7e00700 1 --2- 172.21.15.62:0/1552235569 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f32b006ecc0 0x7f32b0071170 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f32b4005950 tx=0x7f32b40058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:11:43.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:43.364+0000 7f32c5600700 1 -- 172.21.15.62:0/1552235569 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f32bc059130 con 0x7f32c8106b80 2024-04-20T22:11:43.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:43.648+0000 7f32cfc00700 1 -- 172.21.15.62:0/1552235569 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f32c81a1840 con 0x7f32b006ecc0 2024-04-20T22:11:43.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:43.650+0000 7f32c5600700 1 -- 172.21.15.62:0/1552235569 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f32c81a1840 con 0x7f32b006ecc0 2024-04-20T22:11:43.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:43.653+0000 7f32cfc00700 1 -- 172.21.15.62:0/1552235569 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f32b006ecc0 msgr2=0x7f32b0071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:11:43.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:43.653+0000 7f32cfc00700 1 --2- 172.21.15.62:0/1552235569 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f32b006ecc0 0x7f32b0071170 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f32b4005950 tx=0x7f32b40058e0 comp rx=0 tx=0).stop 2024-04-20T22:11:43.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:43.653+0000 7f32cfc00700 1 -- 172.21.15.62:0/1552235569 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f32c8106b80 msgr2=0x7f32c819c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:11:43.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:43.653+0000 7f32cfc00700 1 --2- 172.21.15.62:0/1552235569 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f32c8106b80 0x7f32c819c500 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f32bc0092e0 tx=0x7f32bc00be20 comp rx=0 tx=0).stop 2024-04-20T22:11:43.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:43.653+0000 7f32cfc00700 1 -- 172.21.15.62:0/1552235569 shutdown_connections 2024-04-20T22:11:43.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:43.653+0000 7f32cfc00700 1 --2- 172.21.15.62:0/1552235569 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f32b006ecc0 0x7f32b0071170 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:11:43.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:43.653+0000 7f32cfc00700 1 --2- 172.21.15.62:0/1552235569 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f32c8106b80 0x7f32c819c500 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:11:43.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:43.653+0000 7f32cfc00700 1 --2- 172.21.15.62:0/1552235569 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f32c810c8f0 0x7f32c819ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:11:43.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:43.653+0000 7f32cfc00700 1 -- 172.21.15.62:0/1552235569 >> 172.21.15.62:0/1552235569 conn(0x7f32c8076060 msgr2=0x7f32c8077aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:11:43.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:43.653+0000 7f32cfc00700 1 -- 172.21.15.62:0/1552235569 shutdown_connections 2024-04-20T22:11:43.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:43.654+0000 7f32cfc00700 1 -- 172.21.15.62:0/1552235569 wait complete. 2024-04-20T22:11:43.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:43.822+0000 7fd61fc00700 1 -- 172.21.15.62:0/3133558477 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6181004e0 msgr2=0x7fd6181008b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:11:43.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:43.822+0000 7fd61fc00700 1 --2- 172.21.15.62:0/3133558477 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6181004e0 0x7fd6181008b0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fd60c009910 tx=0x7fd60c009c20 comp rx=0 tx=0).stop 2024-04-20T22:11:43.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:43.823+0000 7fd61fc00700 1 -- 172.21.15.62:0/3133558477 shutdown_connections 2024-04-20T22:11:43.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:43.823+0000 7fd61fc00700 1 --2- 172.21.15.62:0/3133558477 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd618100e80 0x7fd618104e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:11:43.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:43.823+0000 7fd61fc00700 1 --2- 172.21.15.62:0/3133558477 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6181004e0 0x7fd6181008b0 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:11:43.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:43.823+0000 7fd61fc00700 1 -- 172.21.15.62:0/3133558477 >> 172.21.15.62:0/3133558477 conn(0x7fd6180fbda0 msgr2=0x7fd6180fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:11:43.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:43.823+0000 7fd61fc00700 1 -- 172.21.15.62:0/3133558477 shutdown_connections 2024-04-20T22:11:43.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:43.823+0000 7fd61fc00700 1 -- 172.21.15.62:0/3133558477 wait complete. 2024-04-20T22:11:43.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:43.824+0000 7fd61fc00700 1 Processor -- start 2024-04-20T22:11:43.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:43.824+0000 7fd61fc00700 1 -- start start 2024-04-20T22:11:43.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:43.824+0000 7fd61fc00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6181004e0 0x7fd6181980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:11:43.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:43.824+0000 7fd61fc00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd618100e80 0x7fd6181985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:11:43.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:43.824+0000 7fd61fc00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd618198c30 con 0x7fd6181004e0 2024-04-20T22:11:43.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:43.824+0000 7fd61fc00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd618198d70 con 0x7fd618100e80 2024-04-20T22:11:43.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:43.825+0000 7fd61d200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6181004e0 0x7fd6181980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:11:43.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:43.825+0000 7fd61d200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6181004e0 0x7fd6181980a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55948/0 (socket says 172.21.15.62:55948) 2024-04-20T22:11:43.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:43.825+0000 7fd61d200700 1 -- 172.21.15.62:0/2832324943 learned_addr learned my addr 172.21.15.62:0/2832324943 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:11:43.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:43.825+0000 7fd617e00700 1 --2- 172.21.15.62:0/2832324943 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd618100e80 0x7fd6181985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:11:43.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:43.825+0000 7fd61d200700 1 -- 172.21.15.62:0/2832324943 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd618100e80 msgr2=0x7fd6181985e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:11:43.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:43.825+0000 7fd61d200700 1 --2- 172.21.15.62:0/2832324943 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd618100e80 0x7fd6181985e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:11:43.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:43.825+0000 7fd61d200700 1 -- 172.21.15.62:0/2832324943 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd60c0095f0 con 0x7fd6181004e0 2024-04-20T22:11:43.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:43.826+0000 7fd61d200700 1 --2- 172.21.15.62:0/2832324943 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6181004e0 0x7fd6181980a0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7fd60c00b5c0 tx=0x7fd60c00be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:11:43.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:43.826+0000 7fd615600700 1 -- 172.21.15.62:0/2832324943 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd60c01d070 con 0x7fd6181004e0 2024-04-20T22:11:43.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:43.826+0000 7fd615600700 1 -- 172.21.15.62:0/2832324943 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd60c00f460 con 0x7fd6181004e0 2024-04-20T22:11:43.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:43.826+0000 7fd61fc00700 1 -- 172.21.15.62:0/2832324943 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd61819cb60 con 0x7fd6181004e0 2024-04-20T22:11:43.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:43.826+0000 7fd615600700 1 -- 172.21.15.62:0/2832324943 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd60c017610 con 0x7fd6181004e0 2024-04-20T22:11:43.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:43.826+0000 7fd61fc00700 1 -- 172.21.15.62:0/2832324943 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd61819d050 con 0x7fd6181004e0 2024-04-20T22:11:43.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:43.828+0000 7fd61fc00700 1 -- 172.21.15.62:0/2832324943 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd61804f2a0 con 0x7fd6181004e0 2024-04-20T22:11:43.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:43.828+0000 7fd615600700 1 -- 172.21.15.62:0/2832324943 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fd60c00f5d0 con 0x7fd6181004e0 2024-04-20T22:11:43.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:43.830+0000 7fd615600700 1 --2- 172.21.15.62:0/2832324943 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd60006ecc0 0x7fd600071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:11:43.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:43.830+0000 7fd615600700 1 -- 172.21.15.62:0/2832324943 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fd60c090510 con 0x7fd6181004e0 2024-04-20T22:11:43.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:43.830+0000 7fd617e00700 1 --2- 172.21.15.62:0/2832324943 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd60006ecc0 0x7fd600071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:11:43.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:43.831+0000 7fd617e00700 1 --2- 172.21.15.62:0/2832324943 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd60006ecc0 0x7fd600071170 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fd604005ea0 tx=0x7fd604005e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:11:43.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:43.835+0000 7fd615600700 1 -- 172.21.15.62:0/2832324943 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fd60c059220 con 0x7fd6181004e0 2024-04-20T22:11:44.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:11:43 smithi062 ceph-mon[76322]: from='client.35936 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:11:44.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:44.081+0000 7fd61fc00700 1 -- 172.21.15.62:0/2832324943 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fd6181040c0 con 0x7fd60006ecc0 2024-04-20T22:11:44.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:44.096+0000 7fd615600700 1 -- 172.21.15.62:0/2832324943 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fd6181040c0 con 0x7fd60006ecc0 2024-04-20T22:11:44.096 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T22:11:44.096 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (37m) 13m ago 51m 15.3M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T22:11:44.096 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (51m) 13m ago 51m 10.4M - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T22:11:44.096 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (50m) 13m ago 50m 10.1M - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T22:11:44.096 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (33m) 13m ago 51m 6660k - 19.0.0-3133-ga8c84f59 76bc6b91b726 2825d5ba11c6 2024-04-20T22:11:44.096 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (33m) 13m ago 50m 6665k - 19.0.0-3133-ga8c84f59 76bc6b91b726 20f746fa1bba 2024-04-20T22:11:44.097 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (36m) 13m ago 51m 73.5M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T22:11:44.097 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (29m) 13m ago 44m 18.5M - 19.0.0-3133-ga8c84f59 76bc6b91b726 de80bc90a683 2024-04-20T22:11:44.097 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (29m) 13m ago 44m 18.7M - 19.0.0-3133-ga8c84f59 76bc6b91b726 cd793b7ee98e 2024-04-20T22:11:44.097 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (13m) 13m ago 44m 11.0M - 19.0.0-3133-ga8c84f59 76bc6b91b726 0e7b0bc4d1ab 2024-04-20T22:11:44.097 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (44m) 13m ago 44m 18.6M - 18.2.0 dc2bc1663786 f2eec2ee8951 2024-04-20T22:11:44.097 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (40m) 13m ago 53m 513M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T22:11:44.097 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (39m) 13m ago 50m 459M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T22:11:44.097 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (35m) 13m ago 53m 71.7M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 019eb1479485 2024-04-20T22:11:44.097 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (34m) 13m ago 50m 51.0M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 55aac560aab6 2024-04-20T22:11:44.097 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (38m) 13m ago 51m 8472k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T22:11:44.097 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (38m) 13m ago 50m 8048k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T22:11:44.097 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (33m) 13m ago 49m 91.2M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 ee2bb901f26e 2024-04-20T22:11:44.097 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (32m) 13m ago 48m 94.6M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 9075152d4162 2024-04-20T22:11:44.097 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (31m) 13m ago 47m 78.1M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 2d4f3e38dc0e 2024-04-20T22:11:44.097 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (31m) 13m ago 47m 92.6M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 8f97b17ae0eb 2024-04-20T22:11:44.098 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (30m) 13m ago 46m 95.6M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 653d8079967d 2024-04-20T22:11:44.098 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (29m) 13m ago 46m 89.2M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 596dded14c12 2024-04-20T22:11:44.098 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (38m) 13m ago 50m 82.1M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T22:11:44.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:44.099+0000 7fd61fc00700 1 -- 172.21.15.62:0/2832324943 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd60006ecc0 msgr2=0x7fd600071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:11:44.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:44.099+0000 7fd61fc00700 1 --2- 172.21.15.62:0/2832324943 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd60006ecc0 0x7fd600071170 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fd604005ea0 tx=0x7fd604005e30 comp rx=0 tx=0).stop 2024-04-20T22:11:44.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:44.099+0000 7fd61fc00700 1 -- 172.21.15.62:0/2832324943 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6181004e0 msgr2=0x7fd6181980a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:11:44.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:44.099+0000 7fd61fc00700 1 --2- 172.21.15.62:0/2832324943 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6181004e0 0x7fd6181980a0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7fd60c00b5c0 tx=0x7fd60c00be00 comp rx=0 tx=0).stop 2024-04-20T22:11:44.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:44.099+0000 7fd61fc00700 1 -- 172.21.15.62:0/2832324943 shutdown_connections 2024-04-20T22:11:44.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:44.099+0000 7fd61fc00700 1 --2- 172.21.15.62:0/2832324943 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd60006ecc0 0x7fd600071170 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:11:44.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:44.099+0000 7fd61fc00700 1 --2- 172.21.15.62:0/2832324943 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6181004e0 0x7fd6181980a0 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:11:44.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:44.099+0000 7fd61fc00700 1 --2- 172.21.15.62:0/2832324943 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd618100e80 0x7fd6181985e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:11:44.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:44.099+0000 7fd61fc00700 1 -- 172.21.15.62:0/2832324943 >> 172.21.15.62:0/2832324943 conn(0x7fd6180fbda0 msgr2=0x7fd6180fd3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:11:44.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:44.099+0000 7fd61fc00700 1 -- 172.21.15.62:0/2832324943 shutdown_connections 2024-04-20T22:11:44.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:44.100+0000 7fd61fc00700 1 -- 172.21.15.62:0/2832324943 wait complete. 2024-04-20T22:11:44.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:11:43 smithi189 ceph-mon[57022]: from='client.35936 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:11:44.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:44.264+0000 7f35cf400700 1 -- 172.21.15.62:0/3255855316 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f35c8102470 msgr2=0x7f35c81028e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:11:44.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:44.264+0000 7f35cf400700 1 --2- 172.21.15.62:0/3255855316 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f35c8102470 0x7f35c81028e0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f35b4009910 tx=0x7f35b4009c20 comp rx=0 tx=0).stop 2024-04-20T22:11:44.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:44.264+0000 7f35cf400700 1 -- 172.21.15.62:0/3255855316 shutdown_connections 2024-04-20T22:11:44.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:44.264+0000 7f35cf400700 1 --2- 172.21.15.62:0/3255855316 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f35c8102470 0x7f35c81028e0 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:11:44.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:44.264+0000 7f35cf400700 1 --2- 172.21.15.62:0/3255855316 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f35c8108470 0x7f35c8108840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:11:44.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:44.264+0000 7f35cf400700 1 -- 172.21.15.62:0/3255855316 >> 172.21.15.62:0/3255855316 conn(0x7f35c80fdf90 msgr2=0x7f35c81003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:11:44.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:44.264+0000 7f35cf400700 1 -- 172.21.15.62:0/3255855316 shutdown_connections 2024-04-20T22:11:44.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:44.264+0000 7f35cf400700 1 -- 172.21.15.62:0/3255855316 wait complete. 2024-04-20T22:11:44.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:44.265+0000 7f35cf400700 1 Processor -- start 2024-04-20T22:11:44.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:44.265+0000 7f35cf400700 1 -- start start 2024-04-20T22:11:44.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:44.266+0000 7f35cf400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f35c8102470 0x7f35c81982d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:11:44.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:44.266+0000 7f35cf400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f35c8108470 0x7f35c8198810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:11:44.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:44.266+0000 7f35cf400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35c8198ef0 con 0x7f35c8102470 2024-04-20T22:11:44.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:44.266+0000 7f35cf400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35c819cc80 con 0x7f35c8108470 2024-04-20T22:11:44.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:44.266+0000 7f35cca00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f35c8102470 0x7f35c81982d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:11:44.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:44.266+0000 7f35cca00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f35c8102470 0x7f35c81982d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55956/0 (socket says 172.21.15.62:55956) 2024-04-20T22:11:44.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:44.266+0000 7f35cca00700 1 -- 172.21.15.62:0/1096373089 learned_addr learned my addr 172.21.15.62:0/1096373089 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:11:44.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:44.266+0000 7f35c7e00700 1 --2- 172.21.15.62:0/1096373089 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f35c8108470 0x7f35c8198810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:11:44.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:44.267+0000 7f35cca00700 1 -- 172.21.15.62:0/1096373089 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f35c8108470 msgr2=0x7f35c8198810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:11:44.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:44.267+0000 7f35cca00700 1 --2- 172.21.15.62:0/1096373089 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f35c8108470 0x7f35c8198810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:11:44.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:44.267+0000 7f35cca00700 1 -- 172.21.15.62:0/1096373089 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35b40095f0 con 0x7f35c8102470 2024-04-20T22:11:44.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:44.267+0000 7f35cca00700 1 --2- 172.21.15.62:0/1096373089 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f35c8102470 0x7f35c81982d0 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f35bc00d900 tx=0x7f35bc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:11:44.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:44.267+0000 7f35c5600700 1 -- 172.21.15.62:0/1096373089 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f35bc004a00 con 0x7f35c8102470 2024-04-20T22:11:44.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:44.267+0000 7f35c5600700 1 -- 172.21.15.62:0/1096373089 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f35bc0054e0 con 0x7f35c8102470 2024-04-20T22:11:44.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:44.267+0000 7f35c5600700 1 -- 172.21.15.62:0/1096373089 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f35bc009d60 con 0x7f35c8102470 2024-04-20T22:11:44.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:44.267+0000 7f35cf400700 1 -- 172.21.15.62:0/1096373089 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35c819cf60 con 0x7f35c8102470 2024-04-20T22:11:44.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:44.267+0000 7f35cf400700 1 -- 172.21.15.62:0/1096373089 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35c819d4b0 con 0x7f35c8102470 2024-04-20T22:11:44.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:44.269+0000 7f35cf400700 1 -- 172.21.15.62:0/1096373089 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35c810a970 con 0x7f35c8102470 2024-04-20T22:11:44.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:44.270+0000 7f35c5600700 1 -- 172.21.15.62:0/1096373089 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f35bc005020 con 0x7f35c8102470 2024-04-20T22:11:44.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:44.271+0000 7f35c5600700 1 --2- 172.21.15.62:0/1096373089 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f35b006ecc0 0x7f35b0071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:11:44.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:44.271+0000 7f35c5600700 1 -- 172.21.15.62:0/1096373089 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f35bc08efe0 con 0x7f35c8102470 2024-04-20T22:11:44.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:44.271+0000 7f35c7e00700 1 --2- 172.21.15.62:0/1096373089 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f35b006ecc0 0x7f35b0071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:11:44.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:44.272+0000 7f35c7e00700 1 --2- 172.21.15.62:0/1096373089 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f35b006ecc0 0x7f35b0071170 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f35c81998f0 tx=0x7f35b401a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:11:44.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:44.277+0000 7f35c5600700 1 -- 172.21.15.62:0/1096373089 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f35bc057dd0 con 0x7f35c8102470 2024-04-20T22:11:44.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:44.608+0000 7f35cf400700 1 -- 172.21.15.62:0/1096373089 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f35c8199700 con 0x7f35c8102470 2024-04-20T22:11:44.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:44.609+0000 7f35c5600700 1 -- 172.21.15.62:0/1096373089 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7f35bc017340 con 0x7f35c8102470 2024-04-20T22:11:44.609 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:11:44.609 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T22:11:44.610 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:11:44.610 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:11:44.610 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T22:11:44.610 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:11:44.610 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:11:44.610 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T22:11:44.610 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 6 2024-04-20T22:11:44.610 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:11:44.610 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T22:11:44.610 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 1, 2024-04-20T22:11:44.610 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 3 2024-04-20T22:11:44.610 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:11:44.610 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T22:11:44.610 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 1, 2024-04-20T22:11:44.610 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 13 2024-04-20T22:11:44.611 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T22:11:44.611 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:11:44.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:44.612+0000 7f35cf400700 1 -- 172.21.15.62:0/1096373089 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f35b006ecc0 msgr2=0x7f35b0071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:11:44.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:44.612+0000 7f35cf400700 1 --2- 172.21.15.62:0/1096373089 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f35b006ecc0 0x7f35b0071170 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f35c81998f0 tx=0x7f35b401a040 comp rx=0 tx=0).stop 2024-04-20T22:11:44.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:44.612+0000 7f35cf400700 1 -- 172.21.15.62:0/1096373089 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f35c8102470 msgr2=0x7f35c81982d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:11:44.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:44.612+0000 7f35cf400700 1 --2- 172.21.15.62:0/1096373089 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f35c8102470 0x7f35c81982d0 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f35bc00d900 tx=0x7f35bc00dcc0 comp rx=0 tx=0).stop 2024-04-20T22:11:44.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:44.612+0000 7f35cf400700 1 -- 172.21.15.62:0/1096373089 shutdown_connections 2024-04-20T22:11:44.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:44.612+0000 7f35cf400700 1 --2- 172.21.15.62:0/1096373089 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f35b006ecc0 0x7f35b0071170 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:11:44.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:44.612+0000 7f35cf400700 1 --2- 172.21.15.62:0/1096373089 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f35c8102470 0x7f35c81982d0 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:11:44.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:44.612+0000 7f35cf400700 1 --2- 172.21.15.62:0/1096373089 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f35c8108470 0x7f35c8198810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:11:44.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:44.612+0000 7f35cf400700 1 -- 172.21.15.62:0/1096373089 >> 172.21.15.62:0/1096373089 conn(0x7f35c80fdf90 msgr2=0x7f35c80ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:11:44.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:44.612+0000 7f35cf400700 1 -- 172.21.15.62:0/1096373089 shutdown_connections 2024-04-20T22:11:44.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:44.612+0000 7f35cf400700 1 -- 172.21.15.62:0/1096373089 wait complete. 2024-04-20T22:11:44.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:44.777+0000 7fab3ba00700 1 -- 172.21.15.62:0/2027078452 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fab3410c8f0 msgr2=0x7fab3410ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:11:44.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:44.777+0000 7fab3ba00700 1 --2- 172.21.15.62:0/2027078452 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fab3410c8f0 0x7fab3410ccc0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7fab28009910 tx=0x7fab28009c20 comp rx=0 tx=0).stop 2024-04-20T22:11:44.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:44.778+0000 7fab3ba00700 1 -- 172.21.15.62:0/2027078452 shutdown_connections 2024-04-20T22:11:44.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:44.778+0000 7fab3ba00700 1 --2- 172.21.15.62:0/2027078452 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fab34106b80 0x7fab34106ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:11:44.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:44.778+0000 7fab3ba00700 1 --2- 172.21.15.62:0/2027078452 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fab3410c8f0 0x7fab3410ccc0 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:11:44.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:44.778+0000 7fab3ba00700 1 -- 172.21.15.62:0/2027078452 >> 172.21.15.62:0/2027078452 conn(0x7fab34076060 msgr2=0x7fab34078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:11:44.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:44.778+0000 7fab3ba00700 1 -- 172.21.15.62:0/2027078452 shutdown_connections 2024-04-20T22:11:44.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:44.778+0000 7fab3ba00700 1 -- 172.21.15.62:0/2027078452 wait complete. 2024-04-20T22:11:44.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:44.779+0000 7fab3ba00700 1 Processor -- start 2024-04-20T22:11:44.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:44.779+0000 7fab3ba00700 1 -- start start 2024-04-20T22:11:44.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:44.780+0000 7fab3ba00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fab34106b80 0x7fab3419c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:11:44.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:44.780+0000 7fab3ba00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fab3410c8f0 0x7fab3419ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:11:44.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:44.780+0000 7fab3ba00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab3419d120 con 0x7fab3410c8f0 2024-04-20T22:11:44.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:44.780+0000 7fab3ba00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab341a0eb0 con 0x7fab34106b80 2024-04-20T22:11:44.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:44.780+0000 7fab33e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fab3410c8f0 0x7fab3419ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:11:44.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:44.780+0000 7fab33e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fab3410c8f0 0x7fab3419ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55970/0 (socket says 172.21.15.62:55970) 2024-04-20T22:11:44.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:44.780+0000 7fab33e00700 1 -- 172.21.15.62:0/3229792570 learned_addr learned my addr 172.21.15.62:0/3229792570 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:11:44.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:44.781+0000 7fab39000700 1 --2- 172.21.15.62:0/3229792570 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fab34106b80 0x7fab3419c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:11:44.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:44.781+0000 7fab33e00700 1 -- 172.21.15.62:0/3229792570 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fab34106b80 msgr2=0x7fab3419c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:11:44.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:44.781+0000 7fab33e00700 1 --2- 172.21.15.62:0/3229792570 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fab34106b80 0x7fab3419c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:11:44.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:44.781+0000 7fab33e00700 1 -- 172.21.15.62:0/3229792570 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab280095f0 con 0x7fab3410c8f0 2024-04-20T22:11:44.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:44.781+0000 7fab39000700 1 --2- 172.21.15.62:0/3229792570 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fab34106b80 0x7fab3419c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T22:11:44.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:44.781+0000 7fab33e00700 1 --2- 172.21.15.62:0/3229792570 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fab3410c8f0 0x7fab3419ca40 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7fab2000d8d0 tx=0x7fab2000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:11:44.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:44.781+0000 7fab31600700 1 -- 172.21.15.62:0/3229792570 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fab20009940 con 0x7fab3410c8f0 2024-04-20T22:11:44.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:44.782+0000 7fab31600700 1 -- 172.21.15.62:0/3229792570 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fab20010460 con 0x7fab3410c8f0 2024-04-20T22:11:44.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:44.782+0000 7fab31600700 1 -- 172.21.15.62:0/3229792570 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fab2000f5f0 con 0x7fab3410c8f0 2024-04-20T22:11:44.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:44.782+0000 7fab3ba00700 1 -- 172.21.15.62:0/3229792570 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab341a1190 con 0x7fab3410c8f0 2024-04-20T22:11:44.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:44.782+0000 7fab3ba00700 1 -- 172.21.15.62:0/3229792570 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab341a16e0 con 0x7fab3410c8f0 2024-04-20T22:11:44.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:44.783+0000 7fab3ba00700 1 -- 172.21.15.62:0/3229792570 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab34109b20 con 0x7fab3410c8f0 2024-04-20T22:11:44.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:44.784+0000 7fab31600700 1 -- 172.21.15.62:0/3229792570 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fab200105d0 con 0x7fab3410c8f0 2024-04-20T22:11:44.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:44.785+0000 7fab31600700 1 --2- 172.21.15.62:0/3229792570 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fab1c06ecc0 0x7fab1c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:11:44.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:44.785+0000 7fab31600700 1 -- 172.21.15.62:0/3229792570 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fab2008ee10 con 0x7fab3410c8f0 2024-04-20T22:11:44.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:44.785+0000 7fab39000700 1 --2- 172.21.15.62:0/3229792570 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fab1c06ecc0 0x7fab1c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:11:44.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:44.786+0000 7fab39000700 1 --2- 172.21.15.62:0/3229792570 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fab1c06ecc0 0x7fab1c071170 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fab28006010 tx=0x7fab2800b540 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:11:44.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:44.791+0000 7fab31600700 1 -- 172.21.15.62:0/3229792570 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fab20057ba0 con 0x7fab3410c8f0 2024-04-20T22:11:45.085 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:45.084+0000 7fab3ba00700 1 -- 172.21.15.62:0/3229792570 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fab3404f2a0 con 0x7fab3410c8f0 2024-04-20T22:11:45.085 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:45.086+0000 7fab31600700 1 -- 172.21.15.62:0/3229792570 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 27 v27) v1 ==== 76+0+1888 (secure 0 0 0) 0x7fab20016770 con 0x7fab3410c8f0 2024-04-20T22:11:45.086 INFO:teuthology.orchestra.run.smithi062.stdout:e27 2024-04-20T22:11:45.086 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T22:11:45.087 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:11:45.087 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T22:11:45.087 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:11:45.087 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T22:11:45.087 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T22:11:45.087 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 20 2024-04-20T22:11:45.087 INFO:teuthology.orchestra.run.smithi062.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-20T22:11:45.087 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T22:11:45.087 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T21:42:01.559061+0000 2024-04-20T22:11:45.087 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T22:11:45.087 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T22:11:45.087 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T22:11:45.087 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T22:11:45.088 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T22:11:45.088 INFO:teuthology.orchestra.run.smithi062.stdout:max_xattr_size 65536 2024-04-20T22:11:45.088 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T22:11:45.088 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T22:11:45.088 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 81 2024-04-20T22:11:45.088 INFO:teuthology.orchestra.run.smithi062.stdout:snap_rstat 1 2024-04-20T22:11:45.088 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:11:45.088 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 2 2024-04-20T22:11:45.088 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1 2024-04-20T22:11:45.088 INFO:teuthology.orchestra.run.smithi062.stdout:up {} 2024-04-20T22:11:45.088 INFO:teuthology.orchestra.run.smithi062.stdout:failed 0,1 2024-04-20T22:11:45.088 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T22:11:45.089 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T22:11:45.089 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T22:11:45.089 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T22:11:45.089 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data enabled 2024-04-20T22:11:45.089 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T22:11:45.089 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T22:11:45.089 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 1 2024-04-20T22:11:45.089 INFO:teuthology.orchestra.run.smithi062.stdout:qdb_cluster leader: 0 members: 2024-04-20T22:11:45.089 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:11:45.089 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:11:45.089 INFO:teuthology.orchestra.run.smithi062.stdout:Standby daemons: 2024-04-20T22:11:45.089 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:11:45.090 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ryqblb{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6826/4113916205,v1:172.21.15.62:6827/4113916205] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:11:45.090 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ozizwi{-1:34476} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6828/3801975165,v1:172.21.15.62:6829/3801975165] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:11:45.090 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.xbgspw{-1:44285} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6826/3996040526,v1:172.21.15.189:6827/3996040526] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T22:11:45.090 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.dvimer{-1:44727} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6824/3576699232,v1:172.21.15.189:6825/3576699232] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:11:45.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:45.090+0000 7fab3ba00700 1 -- 172.21.15.62:0/3229792570 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fab1c06ecc0 msgr2=0x7fab1c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:11:45.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:45.090+0000 7fab3ba00700 1 --2- 172.21.15.62:0/3229792570 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fab1c06ecc0 0x7fab1c071170 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fab28006010 tx=0x7fab2800b540 comp rx=0 tx=0).stop 2024-04-20T22:11:45.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:45.090+0000 7fab3ba00700 1 -- 172.21.15.62:0/3229792570 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fab3410c8f0 msgr2=0x7fab3419ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:11:45.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:45.090+0000 7fab3ba00700 1 --2- 172.21.15.62:0/3229792570 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fab3410c8f0 0x7fab3419ca40 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7fab2000d8d0 tx=0x7fab2000dc90 comp rx=0 tx=0).stop 2024-04-20T22:11:45.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:45.090+0000 7fab3ba00700 1 -- 172.21.15.62:0/3229792570 shutdown_connections 2024-04-20T22:11:45.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:45.090+0000 7fab3ba00700 1 --2- 172.21.15.62:0/3229792570 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fab1c06ecc0 0x7fab1c071170 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:11:45.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:45.090+0000 7fab3ba00700 1 --2- 172.21.15.62:0/3229792570 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fab34106b80 0x7fab3419c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:11:45.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:45.090+0000 7fab3ba00700 1 --2- 172.21.15.62:0/3229792570 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fab3410c8f0 0x7fab3419ca40 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:11:45.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:45.090+0000 7fab3ba00700 1 -- 172.21.15.62:0/3229792570 >> 172.21.15.62:0/3229792570 conn(0x7fab34076060 msgr2=0x7fab34077aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:11:45.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:45.090+0000 7fab3ba00700 1 -- 172.21.15.62:0/3229792570 shutdown_connections 2024-04-20T22:11:45.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:45.091+0000 7fab3ba00700 1 -- 172.21.15.62:0/3229792570 wait complete. 2024-04-20T22:11:45.092 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 27 2024-04-20T22:11:45.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:11:44 smithi189 ceph-mon[57022]: from='client.35940 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:11:45.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:11:44 smithi189 ceph-mon[57022]: pgmap v1094: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:11:45.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:11:44 smithi189 ceph-mon[57022]: from='client.35944 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:11:45.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:11:44 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/1096373089' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:11:45.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:45.256+0000 7f464c200700 1 -- 172.21.15.62:0/414390262 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f464410c8f0 msgr2=0x7f464410ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:11:45.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:45.256+0000 7f464c200700 1 --2- 172.21.15.62:0/414390262 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f464410c8f0 0x7f464410ccc0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f4638009910 tx=0x7f4638009c20 comp rx=0 tx=0).stop 2024-04-20T22:11:45.257 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:11:44 smithi062 ceph-mon[76322]: from='client.35940 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:11:45.257 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:11:44 smithi062 ceph-mon[76322]: pgmap v1094: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:11:45.257 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:11:44 smithi062 ceph-mon[76322]: from='client.35944 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:11:45.257 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:11:44 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/1096373089' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:11:45.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:45.256+0000 7f464c200700 1 -- 172.21.15.62:0/414390262 shutdown_connections 2024-04-20T22:11:45.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:45.256+0000 7f464c200700 1 --2- 172.21.15.62:0/414390262 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4644106b80 0x7f4644106ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:11:45.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:45.256+0000 7f464c200700 1 --2- 172.21.15.62:0/414390262 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f464410c8f0 0x7f464410ccc0 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:11:45.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:45.256+0000 7f464c200700 1 -- 172.21.15.62:0/414390262 >> 172.21.15.62:0/414390262 conn(0x7f4644076060 msgr2=0x7f4644078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:11:45.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:45.257+0000 7f464c200700 1 -- 172.21.15.62:0/414390262 shutdown_connections 2024-04-20T22:11:45.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:45.257+0000 7f464c200700 1 -- 172.21.15.62:0/414390262 wait complete. 2024-04-20T22:11:45.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:45.258+0000 7f464c200700 1 Processor -- start 2024-04-20T22:11:45.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:45.258+0000 7f464c200700 1 -- start start 2024-04-20T22:11:45.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:45.258+0000 7f464c200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4644106b80 0x7f464419c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:11:45.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:45.258+0000 7f464c200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f464410c8f0 0x7f464419ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:11:45.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:45.258+0000 7f464c200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f464419d120 con 0x7f4644106b80 2024-04-20T22:11:45.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:45.258+0000 7f464c200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46441a0eb0 con 0x7f464410c8f0 2024-04-20T22:11:45.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:45.259+0000 7f4649800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4644106b80 0x7f464419c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:11:45.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:45.259+0000 7f4649800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4644106b80 0x7f464419c500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56000/0 (socket says 172.21.15.62:56000) 2024-04-20T22:11:45.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:45.259+0000 7f4649800700 1 -- 172.21.15.62:0/3516512393 learned_addr learned my addr 172.21.15.62:0/3516512393 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:11:45.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:45.259+0000 7f4648e00700 1 --2- 172.21.15.62:0/3516512393 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f464410c8f0 0x7f464419ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:11:45.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:45.259+0000 7f4649800700 1 -- 172.21.15.62:0/3516512393 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f464410c8f0 msgr2=0x7f464419ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:11:45.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:45.259+0000 7f4649800700 1 --2- 172.21.15.62:0/3516512393 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f464410c8f0 0x7f464419ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:11:45.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:45.259+0000 7f4649800700 1 -- 172.21.15.62:0/3516512393 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46380095f0 con 0x7f4644106b80 2024-04-20T22:11:45.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:45.260+0000 7f4649800700 1 --2- 172.21.15.62:0/3516512393 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4644106b80 0x7f464419c500 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f4638006010 tx=0x7f463800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:11:45.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:45.260+0000 7f4642000700 1 -- 172.21.15.62:0/3516512393 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f463801d070 con 0x7f4644106b80 2024-04-20T22:11:45.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:45.260+0000 7f4642000700 1 -- 172.21.15.62:0/3516512393 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4638004fb0 con 0x7f4644106b80 2024-04-20T22:11:45.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:45.260+0000 7f464c200700 1 -- 172.21.15.62:0/3516512393 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46441a1130 con 0x7f4644106b80 2024-04-20T22:11:45.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:45.260+0000 7f4642000700 1 -- 172.21.15.62:0/3516512393 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f463800f690 con 0x7f4644106b80 2024-04-20T22:11:45.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:45.260+0000 7f464c200700 1 -- 172.21.15.62:0/3516512393 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46441a1620 con 0x7f4644106b80 2024-04-20T22:11:45.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:45.261+0000 7f464c200700 1 -- 172.21.15.62:0/3516512393 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4644109b20 con 0x7f4644106b80 2024-04-20T22:11:45.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:45.263+0000 7f4642000700 1 -- 172.21.15.62:0/3516512393 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f46380228a0 con 0x7f4644106b80 2024-04-20T22:11:45.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:45.264+0000 7f4642000700 1 --2- 172.21.15.62:0/3516512393 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f462c06ecc0 0x7f462c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:11:45.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:45.264+0000 7f4642000700 1 -- 172.21.15.62:0/3516512393 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f4638090700 con 0x7f4644106b80 2024-04-20T22:11:45.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:45.264+0000 7f4648e00700 1 --2- 172.21.15.62:0/3516512393 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f462c06ecc0 0x7f462c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:11:45.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:45.265+0000 7f4648e00700 1 --2- 172.21.15.62:0/3516512393 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f462c06ecc0 0x7f462c071170 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f4630005fd0 tx=0x7f463000a560 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:11:45.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:45.269+0000 7f4642000700 1 -- 172.21.15.62:0/3516512393 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f4638059490 con 0x7f4644106b80 2024-04-20T22:11:45.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:45.520+0000 7f464c200700 1 -- 172.21.15.62:0/3516512393 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f46441a1890 con 0x7f462c06ecc0 2024-04-20T22:11:45.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:45.522+0000 7f4642000700 1 -- 172.21.15.62:0/3516512393 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f46441a1890 con 0x7f462c06ecc0 2024-04-20T22:11:45.522 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:11:45.522 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T22:11:45.523 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T22:11:45.523 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-20T22:11:45.523 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T22:11:45.523 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr", 2024-04-20T22:11:45.523 INFO:teuthology.orchestra.run.smithi062.stdout: "crash", 2024-04-20T22:11:45.523 INFO:teuthology.orchestra.run.smithi062.stdout: "mon", 2024-04-20T22:11:45.523 INFO:teuthology.orchestra.run.smithi062.stdout: "osd" 2024-04-20T22:11:45.523 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T22:11:45.523 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "15/23 daemons upgraded", 2024-04-20T22:11:45.523 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading mds daemons", 2024-04-20T22:11:45.523 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T22:11:45.523 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:11:45.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:45.525+0000 7f464c200700 1 -- 172.21.15.62:0/3516512393 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f462c06ecc0 msgr2=0x7f462c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:11:45.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:45.525+0000 7f464c200700 1 --2- 172.21.15.62:0/3516512393 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f462c06ecc0 0x7f462c071170 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f4630005fd0 tx=0x7f463000a560 comp rx=0 tx=0).stop 2024-04-20T22:11:45.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:45.525+0000 7f464c200700 1 -- 172.21.15.62:0/3516512393 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4644106b80 msgr2=0x7f464419c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:11:45.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:45.525+0000 7f464c200700 1 --2- 172.21.15.62:0/3516512393 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4644106b80 0x7f464419c500 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f4638006010 tx=0x7f463800be20 comp rx=0 tx=0).stop 2024-04-20T22:11:45.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:45.525+0000 7f464c200700 1 -- 172.21.15.62:0/3516512393 shutdown_connections 2024-04-20T22:11:45.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:45.525+0000 7f464c200700 1 --2- 172.21.15.62:0/3516512393 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f462c06ecc0 0x7f462c071170 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:11:45.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:45.525+0000 7f464c200700 1 --2- 172.21.15.62:0/3516512393 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4644106b80 0x7f464419c500 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:11:45.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:45.525+0000 7f464c200700 1 --2- 172.21.15.62:0/3516512393 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f464410c8f0 0x7f464419ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:11:45.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:45.525+0000 7f464c200700 1 -- 172.21.15.62:0/3516512393 >> 172.21.15.62:0/3516512393 conn(0x7f4644076060 msgr2=0x7f4644077aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:11:45.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:45.526+0000 7f464c200700 1 -- 172.21.15.62:0/3516512393 shutdown_connections 2024-04-20T22:11:45.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:45.526+0000 7f464c200700 1 -- 172.21.15.62:0/3516512393 wait complete. 2024-04-20T22:11:45.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:45.684+0000 7f8682200700 1 -- 172.21.15.62:0/2300386193 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f867c102470 msgr2=0x7f867c1028e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:11:45.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:45.684+0000 7f8682200700 1 --2- 172.21.15.62:0/2300386193 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f867c102470 0x7f867c1028e0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f8668009910 tx=0x7f8668009c20 comp rx=0 tx=0).stop 2024-04-20T22:11:45.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:45.684+0000 7f8682200700 1 -- 172.21.15.62:0/2300386193 shutdown_connections 2024-04-20T22:11:45.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:45.684+0000 7f8682200700 1 --2- 172.21.15.62:0/2300386193 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f867c102470 0x7f867c1028e0 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:11:45.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:45.684+0000 7f8682200700 1 --2- 172.21.15.62:0/2300386193 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f867c108470 0x7f867c108840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:11:45.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:45.684+0000 7f8682200700 1 -- 172.21.15.62:0/2300386193 >> 172.21.15.62:0/2300386193 conn(0x7f867c0fdf90 msgr2=0x7f867c1003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:11:45.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:45.684+0000 7f8682200700 1 -- 172.21.15.62:0/2300386193 shutdown_connections 2024-04-20T22:11:45.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:45.684+0000 7f8682200700 1 -- 172.21.15.62:0/2300386193 wait complete. 2024-04-20T22:11:45.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:45.685+0000 7f8682200700 1 Processor -- start 2024-04-20T22:11:45.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:45.685+0000 7f8682200700 1 -- start start 2024-04-20T22:11:45.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:45.686+0000 7f8682200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f867c102470 0x7f867c1981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:11:45.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:45.686+0000 7f8682200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f867c108470 0x7f867c198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:11:45.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:45.686+0000 7f8682200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f867c198d80 con 0x7f867c102470 2024-04-20T22:11:45.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:45.686+0000 7f8682200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f867c198ec0 con 0x7f867c108470 2024-04-20T22:11:45.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:45.686+0000 7f867b400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f867c102470 0x7f867c1981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:11:45.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:45.686+0000 7f867b400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f867c102470 0x7f867c1981f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56018/0 (socket says 172.21.15.62:56018) 2024-04-20T22:11:45.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:45.686+0000 7f867b400700 1 -- 172.21.15.62:0/3927970294 learned_addr learned my addr 172.21.15.62:0/3927970294 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:11:45.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:45.687+0000 7f867aa00700 1 --2- 172.21.15.62:0/3927970294 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f867c108470 0x7f867c198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:11:45.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:45.687+0000 7f867b400700 1 -- 172.21.15.62:0/3927970294 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f867c108470 msgr2=0x7f867c198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:11:45.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:45.687+0000 7f867b400700 1 --2- 172.21.15.62:0/3927970294 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f867c108470 0x7f867c198730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:11:45.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:45.687+0000 7f867b400700 1 -- 172.21.15.62:0/3927970294 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86680095f0 con 0x7f867c102470 2024-04-20T22:11:45.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:45.687+0000 7f867b400700 1 --2- 172.21.15.62:0/3927970294 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f867c102470 0x7f867c1981f0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f866c00c930 tx=0x7f866c00ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:11:45.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:45.687+0000 7f8673e00700 1 -- 172.21.15.62:0/3927970294 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f866c007ad0 con 0x7f867c102470 2024-04-20T22:11:45.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:45.687+0000 7f8673e00700 1 -- 172.21.15.62:0/3927970294 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f866c007c30 con 0x7f867c102470 2024-04-20T22:11:45.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:45.687+0000 7f8682200700 1 -- 172.21.15.62:0/3927970294 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f867c19cd10 con 0x7f867c102470 2024-04-20T22:11:45.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:45.687+0000 7f8673e00700 1 -- 172.21.15.62:0/3927970294 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f866c018610 con 0x7f867c102470 2024-04-20T22:11:45.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:45.687+0000 7f8682200700 1 -- 172.21.15.62:0/3927970294 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f867c19d260 con 0x7f867c102470 2024-04-20T22:11:45.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:45.689+0000 7f8682200700 1 -- 172.21.15.62:0/3927970294 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f867c10a970 con 0x7f867c102470 2024-04-20T22:11:45.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:45.690+0000 7f8673e00700 1 -- 172.21.15.62:0/3927970294 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f866c00f450 con 0x7f867c102470 2024-04-20T22:11:45.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:45.690+0000 7f8673e00700 1 --2- 172.21.15.62:0/3927970294 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f866006ecc0 0x7f8660071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:11:45.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:45.691+0000 7f8673e00700 1 -- 172.21.15.62:0/3927970294 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f866c08ef50 con 0x7f867c102470 2024-04-20T22:11:45.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:45.691+0000 7f867aa00700 1 --2- 172.21.15.62:0/3927970294 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f866006ecc0 0x7f8660071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:11:45.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:45.691+0000 7f867aa00700 1 --2- 172.21.15.62:0/3927970294 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f866006ecc0 0x7f8660071170 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f8668000c00 tx=0x7f8668005fd0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:11:45.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:45.695+0000 7f8673e00700 1 -- 172.21.15.62:0/3927970294 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f866c0c3ff0 con 0x7f867c102470 2024-04-20T22:11:46.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:46.008+0000 7f8682200700 1 -- 172.21.15.62:0/3927970294 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f867c04f2a0 con 0x7f867c102470 2024-04-20T22:11:46.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:46.009+0000 7f8673e00700 1 -- 172.21.15.62:0/3927970294 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7f866c057e20 con 0x7f867c102470 2024-04-20T22:11:46.009 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-20T22:11:46.009 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-20T22:11:46.009 INFO:teuthology.orchestra.run.smithi062.stdout: host smithi062 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T22:11:46.009 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-20T22:11:46.009 INFO:teuthology.orchestra.run.smithi062.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi062 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpq7y7oa_r:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:11:46.009 INFO:teuthology.orchestra.run.smithi062.stdout:Traceback (most recent call last): 2024-04-20T22:11:46.010 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-20T22:11:46.010 INFO:teuthology.orchestra.run.smithi062.stdout: return _run_code(code, main_globals, None, 2024-04-20T22:11:46.010 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-20T22:11:46.010 INFO:teuthology.orchestra.run.smithi062.stdout: exec(code, run_globals) 2024-04-20T22:11:46.010 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5618, in 2024-04-20T22:11:46.010 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5606, in main 2024-04-20T22:11:46.010 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 408, in _infer_config 2024-04-20T22:11:46.010 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 323, in _infer_fsid 2024-04-20T22:11:46.010 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 436, in _infer_image 2024-04-20T22:11:46.010 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 310, in _validate_fsid 2024-04-20T22:11:46.010 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 3359, in command_ceph_volume 2024-04-20T22:11:46.010 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-20T22:11:46.010 INFO:teuthology.orchestra.run.smithi062.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi062 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpq7y7oa_r:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:11:46.010 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-20T22:11:46.011 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is degraded 2024-04-20T22:11:46.011 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T22:11:46.011 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T22:11:46.011 INFO:teuthology.orchestra.run.smithi062.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-20T22:11:46.011 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-20T22:11:46.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:46.012+0000 7f8682200700 1 -- 172.21.15.62:0/3927970294 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f866006ecc0 msgr2=0x7f8660071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:11:46.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:46.012+0000 7f8682200700 1 --2- 172.21.15.62:0/3927970294 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f866006ecc0 0x7f8660071170 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f8668000c00 tx=0x7f8668005fd0 comp rx=0 tx=0).stop 2024-04-20T22:11:46.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:46.012+0000 7f8682200700 1 -- 172.21.15.62:0/3927970294 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f867c102470 msgr2=0x7f867c1981f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:11:46.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:46.012+0000 7f8682200700 1 --2- 172.21.15.62:0/3927970294 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f867c102470 0x7f867c1981f0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f866c00c930 tx=0x7f866c00ccf0 comp rx=0 tx=0).stop 2024-04-20T22:11:46.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:46.012+0000 7f8682200700 1 -- 172.21.15.62:0/3927970294 shutdown_connections 2024-04-20T22:11:46.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:46.012+0000 7f8682200700 1 --2- 172.21.15.62:0/3927970294 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f866006ecc0 0x7f8660071170 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:11:46.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:46.012+0000 7f8682200700 1 --2- 172.21.15.62:0/3927970294 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f867c102470 0x7f867c1981f0 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:11:46.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:46.012+0000 7f8682200700 1 --2- 172.21.15.62:0/3927970294 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f867c108470 0x7f867c198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:11:46.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:46.012+0000 7f8682200700 1 -- 172.21.15.62:0/3927970294 >> 172.21.15.62:0/3927970294 conn(0x7f867c0fdf90 msgr2=0x7f867c0ff3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:11:46.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:46.013+0000 7f8682200700 1 -- 172.21.15.62:0/3927970294 shutdown_connections 2024-04-20T22:11:46.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:11:46.013+0000 7f8682200700 1 -- 172.21.15.62:0/3927970294 wait complete. 2024-04-20T22:11:46.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:11:45 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/3229792570' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:11:46.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:11:45 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/3229792570' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:11:47.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:11:46 smithi189 ceph-mon[57022]: from='client.35956 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:11:47.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:11:46 smithi189 ceph-mon[57022]: pgmap v1095: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:11:47.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:11:46 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/3927970294' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:11:47.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:11:46 smithi062 ceph-mon[76322]: from='client.35956 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:11:47.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:11:46 smithi062 ceph-mon[76322]: pgmap v1095: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:11:47.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:11:46 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/3927970294' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:11:49.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:11:48 smithi189 ceph-mon[57022]: pgmap v1096: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:11:49.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:11:48 smithi062 ceph-mon[76322]: pgmap v1096: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:11:50.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:11:49 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:11:50.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:11:49 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:11:51.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:11:50 smithi189 ceph-mon[57022]: pgmap v1097: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:11:51.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:11:50 smithi062 ceph-mon[76322]: pgmap v1097: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:11:53.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:11:52 smithi189 ceph-mon[57022]: pgmap v1098: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:11:53.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:11:52 smithi062 ceph-mon[76322]: pgmap v1098: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:11:55.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:11:54 smithi189 ceph-mon[57022]: pgmap v1099: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:11:55.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:11:54 smithi062 ceph-mon[76322]: pgmap v1099: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:11:57.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:11:56 smithi189 ceph-mon[57022]: pgmap v1100: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:11:57.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:11:56 smithi062 ceph-mon[76322]: pgmap v1100: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:11:59.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:11:58 smithi189 ceph-mon[57022]: pgmap v1101: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:11:59.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:11:58 smithi062 ceph-mon[76322]: pgmap v1101: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:12:01.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:12:00 smithi189 ceph-mon[57022]: pgmap v1102: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:12:01.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:12:00 smithi062 ceph-mon[76322]: pgmap v1102: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:12:03.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:12:02 smithi189 ceph-mon[57022]: pgmap v1103: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:12:03.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:12:02 smithi062 ceph-mon[76322]: pgmap v1103: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:12:05.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:12:04 smithi189 ceph-mon[57022]: pgmap v1104: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:12:05.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:12:04 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:12:05.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:12:04 smithi062 ceph-mon[76322]: pgmap v1104: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:12:05.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:12:04 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:12:07.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:12:06 smithi189 ceph-mon[57022]: pgmap v1105: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:12:07.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:12:06 smithi062 ceph-mon[76322]: pgmap v1105: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:12:09.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:12:08 smithi062 ceph-mon[76322]: pgmap v1106: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:12:09.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:12:08 smithi189 ceph-mon[57022]: pgmap v1106: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:12:11.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:12:10 smithi062 ceph-mon[76322]: pgmap v1107: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:12:11.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:12:11 smithi189 ceph-mon[57022]: pgmap v1107: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:12:13.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:12:13 smithi062 ceph-mon[76322]: pgmap v1108: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:12:13.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:12:13 smithi189 ceph-mon[57022]: pgmap v1108: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:12:15.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:12:15 smithi062 ceph-mon[76322]: pgmap v1109: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:12:15.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:12:15 smithi189 ceph-mon[57022]: pgmap v1109: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:12:16.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:16.187+0000 7f8214c00700 1 -- 172.21.15.62:0/986164615 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f82101004e0 msgr2=0x7f82101008b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:12:16.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:16.187+0000 7f8214c00700 1 --2- 172.21.15.62:0/986164615 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f82101004e0 0x7f82101008b0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f8200009910 tx=0x7f8200009c20 comp rx=0 tx=0).stop 2024-04-20T22:12:16.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:16.187+0000 7f8214c00700 1 -- 172.21.15.62:0/986164615 shutdown_connections 2024-04-20T22:12:16.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:16.187+0000 7f8214c00700 1 --2- 172.21.15.62:0/986164615 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8210100e80 0x7f8210104e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:12:16.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:16.187+0000 7f8214c00700 1 --2- 172.21.15.62:0/986164615 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f82101004e0 0x7f82101008b0 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:12:16.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:16.187+0000 7f8214c00700 1 -- 172.21.15.62:0/986164615 >> 172.21.15.62:0/986164615 conn(0x7f82100fbda0 msgr2=0x7f82100fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:12:16.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:16.187+0000 7f8214c00700 1 -- 172.21.15.62:0/986164615 shutdown_connections 2024-04-20T22:12:16.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:16.187+0000 7f8214c00700 1 -- 172.21.15.62:0/986164615 wait complete. 2024-04-20T22:12:16.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:16.188+0000 7f8214c00700 1 Processor -- start 2024-04-20T22:12:16.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:16.188+0000 7f8214c00700 1 -- start start 2024-04-20T22:12:16.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:16.189+0000 7f8214c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f82101004e0 0x7f82101980d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:12:16.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:16.189+0000 7f8214c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8210100e80 0x7f8210198610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:12:16.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:16.189+0000 7f8214c00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8210198cf0 con 0x7f8210100e80 2024-04-20T22:12:16.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:16.189+0000 7f8214c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f821019ca80 con 0x7f82101004e0 2024-04-20T22:12:16.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:16.189+0000 7f8205400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8210100e80 0x7f8210198610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:12:16.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:16.189+0000 7f8205400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8210100e80 0x7f8210198610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33264/0 (socket says 172.21.15.62:33264) 2024-04-20T22:12:16.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:16.189+0000 7f8205400700 1 -- 172.21.15.62:0/2358715890 learned_addr learned my addr 172.21.15.62:0/2358715890 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:12:16.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:16.189+0000 7f820de00700 1 --2- 172.21.15.62:0/2358715890 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f82101004e0 0x7f82101980d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:12:16.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:16.189+0000 7f8205400700 1 -- 172.21.15.62:0/2358715890 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f82101004e0 msgr2=0x7f82101980d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:12:16.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:16.189+0000 7f8205400700 1 --2- 172.21.15.62:0/2358715890 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f82101004e0 0x7f82101980d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:12:16.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:16.189+0000 7f8205400700 1 -- 172.21.15.62:0/2358715890 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82000095f0 con 0x7f8210100e80 2024-04-20T22:12:16.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:16.190+0000 7f8205400700 1 --2- 172.21.15.62:0/2358715890 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8210100e80 0x7f8210198610 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f81f800d900 tx=0x7f81f800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:12:16.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:16.190+0000 7f8207400700 1 -- 172.21.15.62:0/2358715890 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f81f8009900 con 0x7f8210100e80 2024-04-20T22:12:16.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:16.190+0000 7f8207400700 1 -- 172.21.15.62:0/2358715890 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f81f8010460 con 0x7f8210100e80 2024-04-20T22:12:16.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:16.190+0000 7f8214c00700 1 -- 172.21.15.62:0/2358715890 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f821019cd60 con 0x7f8210100e80 2024-04-20T22:12:16.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:16.190+0000 7f8207400700 1 -- 172.21.15.62:0/2358715890 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f81f800f5f0 con 0x7f8210100e80 2024-04-20T22:12:16.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:16.190+0000 7f8214c00700 1 -- 172.21.15.62:0/2358715890 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f821019d2b0 con 0x7f8210100e80 2024-04-20T22:12:16.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:16.191+0000 7f8214c00700 1 -- 172.21.15.62:0/2358715890 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82101045d0 con 0x7f8210100e80 2024-04-20T22:12:16.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:16.193+0000 7f8207400700 1 -- 172.21.15.62:0/2358715890 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f81f80105d0 con 0x7f8210100e80 2024-04-20T22:12:16.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:16.193+0000 7f8207400700 1 --2- 172.21.15.62:0/2358715890 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f81f406ecc0 0x7f81f4071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:12:16.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:16.194+0000 7f8207400700 1 -- 172.21.15.62:0/2358715890 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f81f808eca0 con 0x7f8210100e80 2024-04-20T22:12:16.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:16.194+0000 7f820de00700 1 --2- 172.21.15.62:0/2358715890 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f81f406ecc0 0x7f81f4071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:12:16.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:16.195+0000 7f820de00700 1 --2- 172.21.15.62:0/2358715890 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f81f406ecc0 0x7f81f4071170 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f8200006010 tx=0x7f820001a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:12:16.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:16.199+0000 7f8207400700 1 -- 172.21.15.62:0/2358715890 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f81f8057a30 con 0x7f8210100e80 2024-04-20T22:12:16.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:16.459+0000 7f8214c00700 1 -- 172.21.15.62:0/2358715890 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f821019d520 con 0x7f81f406ecc0 2024-04-20T22:12:16.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:16.461+0000 7f8207400700 1 -- 172.21.15.62:0/2358715890 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f821019d520 con 0x7f81f406ecc0 2024-04-20T22:12:16.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:16.464+0000 7f8214c00700 1 -- 172.21.15.62:0/2358715890 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f81f406ecc0 msgr2=0x7f81f4071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:12:16.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:16.464+0000 7f8214c00700 1 --2- 172.21.15.62:0/2358715890 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f81f406ecc0 0x7f81f4071170 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f8200006010 tx=0x7f820001a040 comp rx=0 tx=0).stop 2024-04-20T22:12:16.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:16.464+0000 7f8214c00700 1 -- 172.21.15.62:0/2358715890 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8210100e80 msgr2=0x7f8210198610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:12:16.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:16.464+0000 7f8214c00700 1 --2- 172.21.15.62:0/2358715890 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8210100e80 0x7f8210198610 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f81f800d900 tx=0x7f81f800dcc0 comp rx=0 tx=0).stop 2024-04-20T22:12:16.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:16.465+0000 7f8214c00700 1 -- 172.21.15.62:0/2358715890 shutdown_connections 2024-04-20T22:12:16.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:16.465+0000 7f8214c00700 1 --2- 172.21.15.62:0/2358715890 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f81f406ecc0 0x7f81f4071170 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:12:16.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:16.465+0000 7f8214c00700 1 --2- 172.21.15.62:0/2358715890 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f82101004e0 0x7f82101980d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:12:16.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:16.465+0000 7f8214c00700 1 --2- 172.21.15.62:0/2358715890 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8210100e80 0x7f8210198610 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:12:16.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:16.465+0000 7f8214c00700 1 -- 172.21.15.62:0/2358715890 >> 172.21.15.62:0/2358715890 conn(0x7f82100fbda0 msgr2=0x7f82100fe150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:12:16.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:16.465+0000 7f8214c00700 1 -- 172.21.15.62:0/2358715890 shutdown_connections 2024-04-20T22:12:16.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:16.465+0000 7f8214c00700 1 -- 172.21.15.62:0/2358715890 wait complete. 2024-04-20T22:12:16.482 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T22:12:16.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:16.640+0000 7fc879600700 1 -- 172.21.15.62:0/2325565350 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8741061b0 msgr2=0x7fc874106580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:12:16.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:16.640+0000 7fc879600700 1 --2- 172.21.15.62:0/2325565350 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8741061b0 0x7fc874106580 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7fc864009910 tx=0x7fc864009c20 comp rx=0 tx=0).stop 2024-04-20T22:12:16.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:16.640+0000 7fc879600700 1 -- 172.21.15.62:0/2325565350 shutdown_connections 2024-04-20T22:12:16.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:16.640+0000 7fc879600700 1 --2- 172.21.15.62:0/2325565350 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc874068540 0x7fc8740689b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:12:16.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:16.640+0000 7fc879600700 1 --2- 172.21.15.62:0/2325565350 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8741061b0 0x7fc874106580 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:12:16.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:16.640+0000 7fc879600700 1 -- 172.21.15.62:0/2325565350 >> 172.21.15.62:0/2325565350 conn(0x7fc874074fa0 msgr2=0x7fc8740753a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:12:16.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:16.640+0000 7fc879600700 1 -- 172.21.15.62:0/2325565350 shutdown_connections 2024-04-20T22:12:16.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:16.640+0000 7fc879600700 1 -- 172.21.15.62:0/2325565350 wait complete. 2024-04-20T22:12:16.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:16.641+0000 7fc879600700 1 Processor -- start 2024-04-20T22:12:16.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:16.642+0000 7fc879600700 1 -- start start 2024-04-20T22:12:16.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:16.642+0000 7fc879600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc874068540 0x7fc874197e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:12:16.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:16.642+0000 7fc879600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8741061b0 0x7fc8741983a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:12:16.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:16.642+0000 7fc879600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc874198a80 con 0x7fc8741061b0 2024-04-20T22:12:16.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:16.642+0000 7fc879600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc87419c810 con 0x7fc874068540 2024-04-20T22:12:16.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:16.643+0000 7fc86b400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8741061b0 0x7fc8741983a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:12:16.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:16.643+0000 7fc86b400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8741061b0 0x7fc8741983a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33276/0 (socket says 172.21.15.62:33276) 2024-04-20T22:12:16.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:16.643+0000 7fc86b400700 1 -- 172.21.15.62:0/2659176137 learned_addr learned my addr 172.21.15.62:0/2659176137 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:12:16.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:16.643+0000 7fc873e00700 1 --2- 172.21.15.62:0/2659176137 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc874068540 0x7fc874197e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:12:16.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:16.643+0000 7fc86b400700 1 -- 172.21.15.62:0/2659176137 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc874068540 msgr2=0x7fc874197e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:12:16.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:16.643+0000 7fc86b400700 1 --2- 172.21.15.62:0/2659176137 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc874068540 0x7fc874197e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:12:16.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:16.643+0000 7fc86b400700 1 -- 172.21.15.62:0/2659176137 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8640095f0 con 0x7fc8741061b0 2024-04-20T22:12:16.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:16.644+0000 7fc86b400700 1 --2- 172.21.15.62:0/2659176137 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8741061b0 0x7fc8741983a0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7fc85c00b4b0 tx=0x7fc85c00b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:12:16.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:16.644+0000 7fc871600700 1 -- 172.21.15.62:0/2659176137 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc85c011800 con 0x7fc8741061b0 2024-04-20T22:12:16.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:16.644+0000 7fc871600700 1 -- 172.21.15.62:0/2659176137 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc85c011e20 con 0x7fc8741061b0 2024-04-20T22:12:16.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:16.644+0000 7fc871600700 1 -- 172.21.15.62:0/2659176137 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc85c00f2e0 con 0x7fc8741061b0 2024-04-20T22:12:16.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:16.644+0000 7fc879600700 1 -- 172.21.15.62:0/2659176137 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc87419caf0 con 0x7fc8741061b0 2024-04-20T22:12:16.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:16.644+0000 7fc879600700 1 -- 172.21.15.62:0/2659176137 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc87419d0c0 con 0x7fc8741061b0 2024-04-20T22:12:16.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:16.645+0000 7fc879600700 1 -- 172.21.15.62:0/2659176137 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc874062380 con 0x7fc8741061b0 2024-04-20T22:12:16.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:16.646+0000 7fc871600700 1 -- 172.21.15.62:0/2659176137 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fc85c010380 con 0x7fc8741061b0 2024-04-20T22:12:16.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:16.647+0000 7fc871600700 1 --2- 172.21.15.62:0/2659176137 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc85806ecc0 0x7fc858071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:12:16.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:16.648+0000 7fc871600700 1 -- 172.21.15.62:0/2659176137 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fc85c08ea30 con 0x7fc8741061b0 2024-04-20T22:12:16.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:16.648+0000 7fc873e00700 1 --2- 172.21.15.62:0/2659176137 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc85806ecc0 0x7fc858071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:12:16.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:16.648+0000 7fc873e00700 1 --2- 172.21.15.62:0/2659176137 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc85806ecc0 0x7fc858071170 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fc864006010 tx=0x7fc86401a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:12:16.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:16.652+0000 7fc871600700 1 -- 172.21.15.62:0/2659176137 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fc85c057870 con 0x7fc8741061b0 2024-04-20T22:12:16.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:16.918+0000 7fc879600700 1 -- 172.21.15.62:0/2659176137 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc87419d540 con 0x7fc85806ecc0 2024-04-20T22:12:16.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:16.920+0000 7fc871600700 1 -- 172.21.15.62:0/2659176137 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fc87419d540 con 0x7fc85806ecc0 2024-04-20T22:12:16.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:16.923+0000 7fc879600700 1 -- 172.21.15.62:0/2659176137 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc85806ecc0 msgr2=0x7fc858071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:12:16.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:16.923+0000 7fc879600700 1 --2- 172.21.15.62:0/2659176137 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc85806ecc0 0x7fc858071170 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fc864006010 tx=0x7fc86401a040 comp rx=0 tx=0).stop 2024-04-20T22:12:16.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:16.923+0000 7fc879600700 1 -- 172.21.15.62:0/2659176137 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8741061b0 msgr2=0x7fc8741983a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:12:16.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:16.923+0000 7fc879600700 1 --2- 172.21.15.62:0/2659176137 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8741061b0 0x7fc8741983a0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7fc85c00b4b0 tx=0x7fc85c00b870 comp rx=0 tx=0).stop 2024-04-20T22:12:16.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:16.924+0000 7fc879600700 1 -- 172.21.15.62:0/2659176137 shutdown_connections 2024-04-20T22:12:16.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:16.924+0000 7fc879600700 1 --2- 172.21.15.62:0/2659176137 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc85806ecc0 0x7fc858071170 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:12:16.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:16.924+0000 7fc879600700 1 --2- 172.21.15.62:0/2659176137 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc874068540 0x7fc874197e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:12:16.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:16.924+0000 7fc879600700 1 --2- 172.21.15.62:0/2659176137 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8741061b0 0x7fc8741983a0 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:12:16.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:16.924+0000 7fc879600700 1 -- 172.21.15.62:0/2659176137 >> 172.21.15.62:0/2659176137 conn(0x7fc874074fa0 msgr2=0x7fc8740fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:12:16.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:16.924+0000 7fc879600700 1 -- 172.21.15.62:0/2659176137 shutdown_connections 2024-04-20T22:12:16.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:16.924+0000 7fc879600700 1 -- 172.21.15.62:0/2659176137 wait complete. 2024-04-20T22:12:17.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:17.091+0000 7f9e74a00700 1 -- 172.21.15.62:0/1340522796 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9e640054b0 con 0x7f9e7010c8f0 2024-04-20T22:12:17.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:17.091+0000 7f9e78800700 1 -- 172.21.15.62:0/1340522796 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e7010c8f0 msgr2=0x7f9e7010ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:12:17.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:17.091+0000 7f9e78800700 1 --2- 172.21.15.62:0/1340522796 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e7010c8f0 0x7f9e7010ccc0 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f9e64009910 tx=0x7f9e64009c20 comp rx=0 tx=0).stop 2024-04-20T22:12:17.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:17.091+0000 7f9e78800700 1 -- 172.21.15.62:0/1340522796 shutdown_connections 2024-04-20T22:12:17.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:17.091+0000 7f9e78800700 1 --2- 172.21.15.62:0/1340522796 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e70106b80 0x7f9e70106ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:12:17.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:17.091+0000 7f9e78800700 1 --2- 172.21.15.62:0/1340522796 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e7010c8f0 0x7f9e7010ccc0 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:12:17.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:17.091+0000 7f9e78800700 1 -- 172.21.15.62:0/1340522796 >> 172.21.15.62:0/1340522796 conn(0x7f9e70076060 msgr2=0x7f9e70078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:12:17.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:17.092+0000 7f9e78800700 1 -- 172.21.15.62:0/1340522796 shutdown_connections 2024-04-20T22:12:17.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:17.092+0000 7f9e78800700 1 -- 172.21.15.62:0/1340522796 wait complete. 2024-04-20T22:12:17.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:17.093+0000 7f9e78800700 1 Processor -- start 2024-04-20T22:12:17.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:17.093+0000 7f9e78800700 1 -- start start 2024-04-20T22:12:17.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:17.093+0000 7f9e78800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e70106b80 0x7f9e7019c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:12:17.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:17.093+0000 7f9e78800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e7019cba0 0x7f9e701a1010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:12:17.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:17.093+0000 7f9e78800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e7019d1b0 con 0x7f9e7019cba0 2024-04-20T22:12:17.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:17.093+0000 7f9e78800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e7019d320 con 0x7f9e70106b80 2024-04-20T22:12:17.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:17.094+0000 7f9e75400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e7019cba0 0x7f9e701a1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:12:17.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:17.094+0000 7f9e75400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e7019cba0 0x7f9e701a1010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33290/0 (socket says 172.21.15.62:33290) 2024-04-20T22:12:17.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:17.094+0000 7f9e75400700 1 -- 172.21.15.62:0/2693407421 learned_addr learned my addr 172.21.15.62:0/2693407421 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:12:17.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:17.094+0000 7f9e75e00700 1 --2- 172.21.15.62:0/2693407421 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e70106b80 0x7f9e7019c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:12:17.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:17.094+0000 7f9e75400700 1 -- 172.21.15.62:0/2693407421 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e70106b80 msgr2=0x7f9e7019c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:12:17.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:17.094+0000 7f9e75400700 1 --2- 172.21.15.62:0/2693407421 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e70106b80 0x7f9e7019c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:12:17.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:17.094+0000 7f9e75400700 1 -- 172.21.15.62:0/2693407421 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e640095f0 con 0x7f9e7019cba0 2024-04-20T22:12:17.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:17.095+0000 7f9e75400700 1 --2- 172.21.15.62:0/2693407421 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e7019cba0 0x7f9e701a1010 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f9e6000d900 tx=0x7f9e6000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:12:17.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:17.095+0000 7f9e6aa00700 1 -- 172.21.15.62:0/2693407421 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9e60009b10 con 0x7f9e7019cba0 2024-04-20T22:12:17.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:17.095+0000 7f9e6aa00700 1 -- 172.21.15.62:0/2693407421 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9e6000dea0 con 0x7f9e7019cba0 2024-04-20T22:12:17.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:17.095+0000 7f9e6aa00700 1 -- 172.21.15.62:0/2693407421 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9e6000f5f0 con 0x7f9e7019cba0 2024-04-20T22:12:17.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:17.095+0000 7f9e78800700 1 -- 172.21.15.62:0/2693407421 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e701a1610 con 0x7f9e7019cba0 2024-04-20T22:12:17.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:17.095+0000 7f9e78800700 1 -- 172.21.15.62:0/2693407421 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e701a1b60 con 0x7f9e7019cba0 2024-04-20T22:12:17.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:17.096+0000 7f9e78800700 1 -- 172.21.15.62:0/2693407421 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e70109b20 con 0x7f9e7019cba0 2024-04-20T22:12:17.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:17.097+0000 7f9e6aa00700 1 -- 172.21.15.62:0/2693407421 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f9e60009c70 con 0x7f9e7019cba0 2024-04-20T22:12:17.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:17.099+0000 7f9e6aa00700 1 --2- 172.21.15.62:0/2693407421 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9e5806ecc0 0x7f9e58071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:12:17.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:17.099+0000 7f9e6aa00700 1 -- 172.21.15.62:0/2693407421 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f9e6008ed40 con 0x7f9e7019cba0 2024-04-20T22:12:17.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:17.099+0000 7f9e75e00700 1 --2- 172.21.15.62:0/2693407421 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9e5806ecc0 0x7f9e58071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:12:17.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:17.100+0000 7f9e75e00700 1 --2- 172.21.15.62:0/2693407421 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9e5806ecc0 0x7f9e58071170 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f9e64009fd0 tx=0x7f9e6401a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:12:17.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:17.104+0000 7f9e6aa00700 1 -- 172.21.15.62:0/2693407421 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f9e60057ad0 con 0x7f9e7019cba0 2024-04-20T22:12:17.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:12:17 smithi189 ceph-mon[57022]: pgmap v1110: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:12:17.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:12:17 smithi062 ceph-mon[76322]: pgmap v1110: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:12:17.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:17.347+0000 7f9e78800700 1 -- 172.21.15.62:0/2693407421 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f9e7019dad0 con 0x7f9e5806ecc0 2024-04-20T22:12:17.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:17.364+0000 7f9e6aa00700 1 -- 172.21.15.62:0/2693407421 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f9e7019dad0 con 0x7f9e5806ecc0 2024-04-20T22:12:17.365 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T22:12:17.365 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (38m) 14m ago 51m 15.3M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T22:12:17.365 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (52m) 14m ago 52m 10.4M - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T22:12:17.365 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (50m) 14m ago 50m 10.1M - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T22:12:17.365 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (34m) 14m ago 52m 6660k - 19.0.0-3133-ga8c84f59 76bc6b91b726 2825d5ba11c6 2024-04-20T22:12:17.365 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (34m) 14m ago 50m 6665k - 19.0.0-3133-ga8c84f59 76bc6b91b726 20f746fa1bba 2024-04-20T22:12:17.365 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (37m) 14m ago 51m 73.5M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T22:12:17.365 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (29m) 14m ago 44m 18.5M - 19.0.0-3133-ga8c84f59 76bc6b91b726 de80bc90a683 2024-04-20T22:12:17.365 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (30m) 14m ago 44m 18.7M - 19.0.0-3133-ga8c84f59 76bc6b91b726 cd793b7ee98e 2024-04-20T22:12:17.365 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (14m) 14m ago 44m 11.0M - 19.0.0-3133-ga8c84f59 76bc6b91b726 0e7b0bc4d1ab 2024-04-20T22:12:17.365 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (44m) 14m ago 44m 18.6M - 18.2.0 dc2bc1663786 f2eec2ee8951 2024-04-20T22:12:17.365 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (41m) 14m ago 53m 513M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T22:12:17.366 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (39m) 14m ago 50m 459M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T22:12:17.366 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (35m) 14m ago 53m 71.7M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 019eb1479485 2024-04-20T22:12:17.366 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (35m) 14m ago 50m 51.0M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 55aac560aab6 2024-04-20T22:12:17.367 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (39m) 14m ago 52m 8472k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T22:12:17.367 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (39m) 14m ago 50m 8048k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T22:12:17.367 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (33m) 14m ago 49m 91.2M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 ee2bb901f26e 2024-04-20T22:12:17.367 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (33m) 14m ago 49m 94.6M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 9075152d4162 2024-04-20T22:12:17.367 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (32m) 14m ago 48m 78.1M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 2d4f3e38dc0e 2024-04-20T22:12:17.367 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (31m) 14m ago 47m 92.6M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 8f97b17ae0eb 2024-04-20T22:12:17.367 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (31m) 14m ago 47m 95.6M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 653d8079967d 2024-04-20T22:12:17.367 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (30m) 14m ago 46m 89.2M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 596dded14c12 2024-04-20T22:12:17.367 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (38m) 14m ago 51m 82.1M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T22:12:17.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:17.367+0000 7f9e78800700 1 -- 172.21.15.62:0/2693407421 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9e5806ecc0 msgr2=0x7f9e58071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:12:17.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:17.367+0000 7f9e78800700 1 --2- 172.21.15.62:0/2693407421 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9e5806ecc0 0x7f9e58071170 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f9e64009fd0 tx=0x7f9e6401a040 comp rx=0 tx=0).stop 2024-04-20T22:12:17.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:17.367+0000 7f9e78800700 1 -- 172.21.15.62:0/2693407421 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e7019cba0 msgr2=0x7f9e701a1010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:12:17.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:17.367+0000 7f9e78800700 1 --2- 172.21.15.62:0/2693407421 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e7019cba0 0x7f9e701a1010 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f9e6000d900 tx=0x7f9e6000dcc0 comp rx=0 tx=0).stop 2024-04-20T22:12:17.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:17.367+0000 7f9e78800700 1 -- 172.21.15.62:0/2693407421 shutdown_connections 2024-04-20T22:12:17.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:17.367+0000 7f9e78800700 1 --2- 172.21.15.62:0/2693407421 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9e5806ecc0 0x7f9e58071170 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:12:17.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:17.367+0000 7f9e78800700 1 --2- 172.21.15.62:0/2693407421 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e70106b80 0x7f9e7019c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:12:17.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:17.367+0000 7f9e78800700 1 --2- 172.21.15.62:0/2693407421 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e7019cba0 0x7f9e701a1010 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:12:17.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:17.367+0000 7f9e78800700 1 -- 172.21.15.62:0/2693407421 >> 172.21.15.62:0/2693407421 conn(0x7f9e70076060 msgr2=0x7f9e70077cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:12:17.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:17.368+0000 7f9e78800700 1 -- 172.21.15.62:0/2693407421 shutdown_connections 2024-04-20T22:12:17.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:17.368+0000 7f9e78800700 1 -- 172.21.15.62:0/2693407421 wait complete. 2024-04-20T22:12:17.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:17.533+0000 7f75c9e00700 1 -- 172.21.15.62:0/2596206912 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f75c41026a0 msgr2=0x7f75c4102a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:12:17.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:17.533+0000 7f75c9e00700 1 --2- 172.21.15.62:0/2596206912 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f75c41026a0 0x7f75c4102a70 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f75b4009910 tx=0x7f75b4009c20 comp rx=0 tx=0).stop 2024-04-20T22:12:17.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:17.534+0000 7f75c9e00700 1 -- 172.21.15.62:0/2596206912 shutdown_connections 2024-04-20T22:12:17.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:17.534+0000 7f75c9e00700 1 --2- 172.21.15.62:0/2596206912 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75c4103040 0x7f75c4107010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:12:17.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:17.534+0000 7f75c9e00700 1 --2- 172.21.15.62:0/2596206912 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f75c41026a0 0x7f75c4102a70 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:12:17.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:17.534+0000 7f75c9e00700 1 -- 172.21.15.62:0/2596206912 >> 172.21.15.62:0/2596206912 conn(0x7f75c40fdf40 msgr2=0x7f75c4100350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:12:17.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:17.534+0000 7f75c9e00700 1 -- 172.21.15.62:0/2596206912 shutdown_connections 2024-04-20T22:12:17.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:17.534+0000 7f75c9e00700 1 -- 172.21.15.62:0/2596206912 wait complete. 2024-04-20T22:12:17.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:17.535+0000 7f75c9e00700 1 Processor -- start 2024-04-20T22:12:17.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:17.535+0000 7f75c9e00700 1 -- start start 2024-04-20T22:12:17.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:17.535+0000 7f75c9e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f75c41026a0 0x7f75c4198130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:12:17.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:17.535+0000 7f75c9e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75c4103040 0x7f75c4198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:12:17.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:17.535+0000 7f75c9e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75c4198d50 con 0x7f75c41026a0 2024-04-20T22:12:17.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:17.535+0000 7f75c9e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75c419cae0 con 0x7f75c4103040 2024-04-20T22:12:17.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:17.536+0000 7f75c2a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f75c41026a0 0x7f75c4198130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:12:17.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:17.536+0000 7f75c2a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f75c41026a0 0x7f75c4198130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33304/0 (socket says 172.21.15.62:33304) 2024-04-20T22:12:17.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:17.536+0000 7f75c2a00700 1 -- 172.21.15.62:0/1413725026 learned_addr learned my addr 172.21.15.62:0/1413725026 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:12:17.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:17.536+0000 7f75c2000700 1 --2- 172.21.15.62:0/1413725026 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75c4103040 0x7f75c4198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:12:17.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:17.536+0000 7f75c2a00700 1 -- 172.21.15.62:0/1413725026 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75c4103040 msgr2=0x7f75c4198670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:12:17.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:17.536+0000 7f75c2a00700 1 --2- 172.21.15.62:0/1413725026 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75c4103040 0x7f75c4198670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:12:17.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:17.536+0000 7f75c2a00700 1 -- 172.21.15.62:0/1413725026 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75b40095f0 con 0x7f75c41026a0 2024-04-20T22:12:17.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:17.537+0000 7f75c2a00700 1 --2- 172.21.15.62:0/1413725026 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f75c41026a0 0x7f75c4198130 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f75b40092e0 tx=0x7f75b400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:12:17.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:17.537+0000 7f75c8a00700 1 -- 172.21.15.62:0/1413725026 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f75b401d070 con 0x7f75c41026a0 2024-04-20T22:12:17.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:17.537+0000 7f75c8a00700 1 -- 172.21.15.62:0/1413725026 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f75b4005e00 con 0x7f75c41026a0 2024-04-20T22:12:17.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:17.537+0000 7f75c9e00700 1 -- 172.21.15.62:0/1413725026 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75c419cd60 con 0x7f75c41026a0 2024-04-20T22:12:17.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:17.537+0000 7f75c8a00700 1 -- 172.21.15.62:0/1413725026 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f75b4017400 con 0x7f75c41026a0 2024-04-20T22:12:17.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:17.537+0000 7f75c9e00700 1 -- 172.21.15.62:0/1413725026 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75c419d250 con 0x7f75c41026a0 2024-04-20T22:12:17.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:17.538+0000 7f75c9e00700 1 -- 172.21.15.62:0/1413725026 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75c410a9b0 con 0x7f75c41026a0 2024-04-20T22:12:17.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:17.539+0000 7f75c8a00700 1 -- 172.21.15.62:0/1413725026 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f75b400f460 con 0x7f75c41026a0 2024-04-20T22:12:17.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:17.540+0000 7f75c8a00700 1 --2- 172.21.15.62:0/1413725026 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f75a806ecc0 0x7f75a8071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:12:17.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:17.540+0000 7f75c8a00700 1 -- 172.21.15.62:0/1413725026 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f75b4090430 con 0x7f75c41026a0 2024-04-20T22:12:17.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:17.541+0000 7f75c2000700 1 --2- 172.21.15.62:0/1413725026 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f75a806ecc0 0x7f75a8071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:12:17.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:17.541+0000 7f75c2000700 1 --2- 172.21.15.62:0/1413725026 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f75a806ecc0 0x7f75a8071170 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f75b0005950 tx=0x7f75b00058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:12:17.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:17.545+0000 7f75c8a00700 1 -- 172.21.15.62:0/1413725026 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f75b4059140 con 0x7f75c41026a0 2024-04-20T22:12:17.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:17.866+0000 7f75c9e00700 1 -- 172.21.15.62:0/1413725026 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f75c404f2a0 con 0x7f75c41026a0 2024-04-20T22:12:17.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:17.867+0000 7f75c8a00700 1 -- 172.21.15.62:0/1413725026 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7f75b405c800 con 0x7f75c41026a0 2024-04-20T22:12:17.868 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:12:17.868 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T22:12:17.868 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:12:17.869 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:12:17.869 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T22:12:17.869 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:12:17.869 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:12:17.869 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T22:12:17.869 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 6 2024-04-20T22:12:17.869 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:12:17.869 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T22:12:17.869 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 1, 2024-04-20T22:12:17.869 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 3 2024-04-20T22:12:17.869 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:12:17.869 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T22:12:17.869 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 1, 2024-04-20T22:12:17.869 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 13 2024-04-20T22:12:17.870 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T22:12:17.870 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:12:17.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:17.870+0000 7f75c9e00700 1 -- 172.21.15.62:0/1413725026 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f75a806ecc0 msgr2=0x7f75a8071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:12:17.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:17.870+0000 7f75c9e00700 1 --2- 172.21.15.62:0/1413725026 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f75a806ecc0 0x7f75a8071170 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f75b0005950 tx=0x7f75b00058e0 comp rx=0 tx=0).stop 2024-04-20T22:12:17.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:17.870+0000 7f75c9e00700 1 -- 172.21.15.62:0/1413725026 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f75c41026a0 msgr2=0x7f75c4198130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:12:17.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:17.870+0000 7f75c9e00700 1 --2- 172.21.15.62:0/1413725026 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f75c41026a0 0x7f75c4198130 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f75b40092e0 tx=0x7f75b400be20 comp rx=0 tx=0).stop 2024-04-20T22:12:17.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:17.870+0000 7f75c9e00700 1 -- 172.21.15.62:0/1413725026 shutdown_connections 2024-04-20T22:12:17.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:17.870+0000 7f75c9e00700 1 --2- 172.21.15.62:0/1413725026 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f75a806ecc0 0x7f75a8071170 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:12:17.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:17.870+0000 7f75c9e00700 1 --2- 172.21.15.62:0/1413725026 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f75c41026a0 0x7f75c4198130 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:12:17.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:17.870+0000 7f75c9e00700 1 --2- 172.21.15.62:0/1413725026 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75c4103040 0x7f75c4198670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:12:17.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:17.870+0000 7f75c9e00700 1 -- 172.21.15.62:0/1413725026 >> 172.21.15.62:0/1413725026 conn(0x7f75c40fdf40 msgr2=0x7f75c40ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:12:17.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:17.871+0000 7f75c9e00700 1 -- 172.21.15.62:0/1413725026 shutdown_connections 2024-04-20T22:12:17.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:17.871+0000 7f75c9e00700 1 -- 172.21.15.62:0/1413725026 wait complete. 2024-04-20T22:12:18.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.036+0000 7f159b000700 1 -- 172.21.15.62:0/2613501466 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1594068540 msgr2=0x7f15940689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:12:18.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.036+0000 7f159b000700 1 --2- 172.21.15.62:0/2613501466 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1594068540 0x7f15940689b0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f1580009910 tx=0x7f1580009c20 comp rx=0 tx=0).stop 2024-04-20T22:12:18.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.036+0000 7f159b000700 1 -- 172.21.15.62:0/2613501466 shutdown_connections 2024-04-20T22:12:18.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.036+0000 7f159b000700 1 --2- 172.21.15.62:0/2613501466 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1594068540 0x7f15940689b0 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:12:18.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.036+0000 7f159b000700 1 --2- 172.21.15.62:0/2613501466 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f15941061b0 0x7f1594106580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:12:18.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.036+0000 7f159b000700 1 -- 172.21.15.62:0/2613501466 >> 172.21.15.62:0/2613501466 conn(0x7f1594074fa0 msgr2=0x7f15940753a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:12:18.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.036+0000 7f159b000700 1 -- 172.21.15.62:0/2613501466 shutdown_connections 2024-04-20T22:12:18.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.036+0000 7f159b000700 1 -- 172.21.15.62:0/2613501466 wait complete. 2024-04-20T22:12:18.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.037+0000 7f159b000700 1 Processor -- start 2024-04-20T22:12:18.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.037+0000 7f159b000700 1 -- start start 2024-04-20T22:12:18.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.038+0000 7f159b000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1594068540 0x7f1594197d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:12:18.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.038+0000 7f159b000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f15941061b0 0x7f15941982a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:12:18.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.038+0000 7f159b000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1594198980 con 0x7f15941061b0 2024-04-20T22:12:18.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.038+0000 7f159b000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f159419c710 con 0x7f1594068540 2024-04-20T22:12:18.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.038+0000 7f1599200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f15941061b0 0x7f15941982a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:12:18.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.038+0000 7f1599200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f15941061b0 0x7f15941982a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33322/0 (socket says 172.21.15.62:33322) 2024-04-20T22:12:18.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.038+0000 7f1599200700 1 -- 172.21.15.62:0/1638338973 learned_addr learned my addr 172.21.15.62:0/1638338973 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:12:18.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.038+0000 7f1599c00700 1 --2- 172.21.15.62:0/1638338973 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1594068540 0x7f1594197d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:12:18.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.039+0000 7f1599200700 1 -- 172.21.15.62:0/1638338973 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1594068540 msgr2=0x7f1594197d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:12:18.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.039+0000 7f1599200700 1 --2- 172.21.15.62:0/1638338973 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1594068540 0x7f1594197d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:12:18.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.039+0000 7f1599200700 1 -- 172.21.15.62:0/1638338973 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15800095f0 con 0x7f15941061b0 2024-04-20T22:12:18.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.039+0000 7f1599200700 1 --2- 172.21.15.62:0/1638338973 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f15941061b0 0x7f15941982a0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f158000f690 tx=0x7f158000f770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:12:18.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.039+0000 7f1592000700 1 -- 172.21.15.62:0/1638338973 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f158001d070 con 0x7f15941061b0 2024-04-20T22:12:18.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.039+0000 7f1592000700 1 -- 172.21.15.62:0/1638338973 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f15800050d0 con 0x7f15941061b0 2024-04-20T22:12:18.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.039+0000 7f1592000700 1 -- 172.21.15.62:0/1638338973 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1580022470 con 0x7f15941061b0 2024-04-20T22:12:18.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.039+0000 7f159b000700 1 -- 172.21.15.62:0/1638338973 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f159419c990 con 0x7f15941061b0 2024-04-20T22:12:18.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.040+0000 7f159b000700 1 -- 172.21.15.62:0/1638338973 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f159419ceb0 con 0x7f15941061b0 2024-04-20T22:12:18.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.041+0000 7f159b000700 1 -- 172.21.15.62:0/1638338973 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1594100a20 con 0x7f15941061b0 2024-04-20T22:12:18.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.042+0000 7f1592000700 1 -- 172.21.15.62:0/1638338973 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f1580003680 con 0x7f15941061b0 2024-04-20T22:12:18.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.043+0000 7f1592000700 1 --2- 172.21.15.62:0/1638338973 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f157c06ecc0 0x7f157c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:12:18.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.043+0000 7f1592000700 1 -- 172.21.15.62:0/1638338973 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f1580090370 con 0x7f15941061b0 2024-04-20T22:12:18.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.043+0000 7f1599c00700 1 --2- 172.21.15.62:0/1638338973 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f157c06ecc0 0x7f157c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:12:18.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.044+0000 7f1599c00700 1 --2- 172.21.15.62:0/1638338973 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f157c06ecc0 0x7f157c071170 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f1588009bf0 tx=0x7f1588009260 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:12:18.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.048+0000 7f1592000700 1 -- 172.21.15.62:0/1638338973 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f1580059080 con 0x7f15941061b0 2024-04-20T22:12:18.143 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:12:18 smithi062 ceph-mon[76322]: from='client.35964 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:12:18.143 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:12:18 smithi062 ceph-mon[76322]: from='client.35968 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:12:18.143 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:12:18 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/1413725026' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:12:18.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.350+0000 7f159b000700 1 -- 172.21.15.62:0/1638338973 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f159419d1e0 con 0x7f15941061b0 2024-04-20T22:12:18.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.351+0000 7f1592000700 1 -- 172.21.15.62:0/1638338973 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 27 v27) v1 ==== 76+0+1888 (secure 0 0 0) 0x7f158005c740 con 0x7f15941061b0 2024-04-20T22:12:18.351 INFO:teuthology.orchestra.run.smithi062.stdout:e27 2024-04-20T22:12:18.351 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T22:12:18.351 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:12:18.351 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T22:12:18.352 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:12:18.352 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T22:12:18.352 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T22:12:18.352 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 20 2024-04-20T22:12:18.352 INFO:teuthology.orchestra.run.smithi062.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-20T22:12:18.352 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T22:12:18.352 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T21:42:01.559061+0000 2024-04-20T22:12:18.352 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T22:12:18.352 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T22:12:18.352 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T22:12:18.352 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T22:12:18.352 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T22:12:18.352 INFO:teuthology.orchestra.run.smithi062.stdout:max_xattr_size 65536 2024-04-20T22:12:18.352 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T22:12:18.352 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T22:12:18.353 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 81 2024-04-20T22:12:18.353 INFO:teuthology.orchestra.run.smithi062.stdout:snap_rstat 1 2024-04-20T22:12:18.353 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:12:18.353 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 2 2024-04-20T22:12:18.353 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1 2024-04-20T22:12:18.353 INFO:teuthology.orchestra.run.smithi062.stdout:up {} 2024-04-20T22:12:18.353 INFO:teuthology.orchestra.run.smithi062.stdout:failed 0,1 2024-04-20T22:12:18.353 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T22:12:18.353 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T22:12:18.353 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T22:12:18.353 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T22:12:18.353 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data enabled 2024-04-20T22:12:18.353 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T22:12:18.353 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T22:12:18.353 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 1 2024-04-20T22:12:18.354 INFO:teuthology.orchestra.run.smithi062.stdout:qdb_cluster leader: 0 members: 2024-04-20T22:12:18.354 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:12:18.354 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:12:18.354 INFO:teuthology.orchestra.run.smithi062.stdout:Standby daemons: 2024-04-20T22:12:18.354 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:12:18.354 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ryqblb{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6826/4113916205,v1:172.21.15.62:6827/4113916205] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:12:18.354 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ozizwi{-1:34476} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6828/3801975165,v1:172.21.15.62:6829/3801975165] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:12:18.354 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.xbgspw{-1:44285} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6826/3996040526,v1:172.21.15.189:6827/3996040526] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T22:12:18.354 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.dvimer{-1:44727} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6824/3576699232,v1:172.21.15.189:6825/3576699232] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:12:18.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.354+0000 7f159b000700 1 -- 172.21.15.62:0/1638338973 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f157c06ecc0 msgr2=0x7f157c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:12:18.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.354+0000 7f159b000700 1 --2- 172.21.15.62:0/1638338973 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f157c06ecc0 0x7f157c071170 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f1588009bf0 tx=0x7f1588009260 comp rx=0 tx=0).stop 2024-04-20T22:12:18.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.354+0000 7f159b000700 1 -- 172.21.15.62:0/1638338973 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f15941061b0 msgr2=0x7f15941982a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:12:18.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.354+0000 7f159b000700 1 --2- 172.21.15.62:0/1638338973 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f15941061b0 0x7f15941982a0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f158000f690 tx=0x7f158000f770 comp rx=0 tx=0).stop 2024-04-20T22:12:18.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.354+0000 7f159b000700 1 -- 172.21.15.62:0/1638338973 shutdown_connections 2024-04-20T22:12:18.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.354+0000 7f159b000700 1 --2- 172.21.15.62:0/1638338973 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f157c06ecc0 0x7f157c071170 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:12:18.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.354+0000 7f159b000700 1 --2- 172.21.15.62:0/1638338973 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1594068540 0x7f1594197d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:12:18.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.354+0000 7f159b000700 1 --2- 172.21.15.62:0/1638338973 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f15941061b0 0x7f15941982a0 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:12:18.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.354+0000 7f159b000700 1 -- 172.21.15.62:0/1638338973 >> 172.21.15.62:0/1638338973 conn(0x7f1594074fa0 msgr2=0x7f159410a5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:12:18.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.354+0000 7f159b000700 1 -- 172.21.15.62:0/1638338973 shutdown_connections 2024-04-20T22:12:18.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.355+0000 7f159b000700 1 -- 172.21.15.62:0/1638338973 wait complete. 2024-04-20T22:12:18.356 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 27 2024-04-20T22:12:18.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:12:18 smithi189 ceph-mon[57022]: from='client.35964 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:12:18.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:12:18 smithi189 ceph-mon[57022]: from='client.35968 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:12:18.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:12:18 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/1413725026' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:12:18.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.512+0000 7fa40e600700 1 -- 172.21.15.62:0/3709407288 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa408073090 msgr2=0x7fa408073460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:12:18.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.512+0000 7fa40e600700 1 --2- 172.21.15.62:0/3709407288 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa408073090 0x7fa408073460 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7fa3f8009940 tx=0x7fa3f8009c50 comp rx=0 tx=0).stop 2024-04-20T22:12:18.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.512+0000 7fa40e600700 1 -- 172.21.15.62:0/3709407288 shutdown_connections 2024-04-20T22:12:18.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.512+0000 7fa40e600700 1 --2- 172.21.15.62:0/3709407288 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4080739a0 0x7fa40810c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:12:18.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.512+0000 7fa40e600700 1 --2- 172.21.15.62:0/3709407288 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa408073090 0x7fa408073460 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:12:18.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.512+0000 7fa40e600700 1 -- 172.21.15.62:0/3709407288 >> 172.21.15.62:0/3709407288 conn(0x7fa4080fbda0 msgr2=0x7fa4080fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:12:18.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.512+0000 7fa40e600700 1 -- 172.21.15.62:0/3709407288 shutdown_connections 2024-04-20T22:12:18.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.512+0000 7fa40e600700 1 -- 172.21.15.62:0/3709407288 wait complete. 2024-04-20T22:12:18.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.514+0000 7fa40e600700 1 Processor -- start 2024-04-20T22:12:18.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.514+0000 7fa40e600700 1 -- start start 2024-04-20T22:12:18.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.515+0000 7fa40e600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa408073090 0x7fa4081981c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:12:18.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.515+0000 7fa40e600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4080739a0 0x7fa408198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:12:18.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.515+0000 7fa40e600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa408198de0 con 0x7fa408073090 2024-04-20T22:12:18.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.515+0000 7fa40e600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa40819cb70 con 0x7fa4080739a0 2024-04-20T22:12:18.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.515+0000 7fa407400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa408073090 0x7fa4081981c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:12:18.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.515+0000 7fa407400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa408073090 0x7fa4081981c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33332/0 (socket says 172.21.15.62:33332) 2024-04-20T22:12:18.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.515+0000 7fa407400700 1 -- 172.21.15.62:0/3296293145 learned_addr learned my addr 172.21.15.62:0/3296293145 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:12:18.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.515+0000 7fa406a00700 1 --2- 172.21.15.62:0/3296293145 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4080739a0 0x7fa408198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:12:18.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.515+0000 7fa407400700 1 -- 172.21.15.62:0/3296293145 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4080739a0 msgr2=0x7fa408198700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:12:18.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.516+0000 7fa407400700 1 --2- 172.21.15.62:0/3296293145 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4080739a0 0x7fa408198700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:12:18.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.516+0000 7fa407400700 1 -- 172.21.15.62:0/3296293145 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3f80095f0 con 0x7fa408073090 2024-04-20T22:12:18.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.516+0000 7fa406a00700 1 --2- 172.21.15.62:0/3296293145 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4080739a0 0x7fa408198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T22:12:18.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.516+0000 7fa407400700 1 --2- 172.21.15.62:0/3296293145 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa408073090 0x7fa4081981c0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7fa3f8005620 tx=0x7fa3f8005700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:12:18.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.516+0000 7fa3ffe00700 1 -- 172.21.15.62:0/3296293145 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa3f801d070 con 0x7fa408073090 2024-04-20T22:12:18.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.516+0000 7fa3ffe00700 1 -- 172.21.15.62:0/3296293145 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa3f800f460 con 0x7fa408073090 2024-04-20T22:12:18.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.516+0000 7fa40e600700 1 -- 172.21.15.62:0/3296293145 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa40819cdf0 con 0x7fa408073090 2024-04-20T22:12:18.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.516+0000 7fa3ffe00700 1 -- 172.21.15.62:0/3296293145 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa3f8021640 con 0x7fa408073090 2024-04-20T22:12:18.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.516+0000 7fa40e600700 1 -- 172.21.15.62:0/3296293145 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa40819d2e0 con 0x7fa408073090 2024-04-20T22:12:18.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.517+0000 7fa40e600700 1 -- 172.21.15.62:0/3296293145 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa408109ce0 con 0x7fa408073090 2024-04-20T22:12:18.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.519+0000 7fa3ffe00700 1 -- 172.21.15.62:0/3296293145 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fa3f800fa40 con 0x7fa408073090 2024-04-20T22:12:18.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.520+0000 7fa3ffe00700 1 --2- 172.21.15.62:0/3296293145 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa3ec06ecc0 0x7fa3ec071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:12:18.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.520+0000 7fa3ffe00700 1 -- 172.21.15.62:0/3296293145 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fa3f8090180 con 0x7fa408073090 2024-04-20T22:12:18.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.520+0000 7fa406a00700 1 --2- 172.21.15.62:0/3296293145 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa3ec06ecc0 0x7fa3ec071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:12:18.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.521+0000 7fa406a00700 1 --2- 172.21.15.62:0/3296293145 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa3ec06ecc0 0x7fa3ec071170 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fa4081997e0 tx=0x7fa3f4008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:12:18.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.525+0000 7fa3ffe00700 1 -- 172.21.15.62:0/3296293145 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fa3f8058f10 con 0x7fa408073090 2024-04-20T22:12:18.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.770+0000 7fa40e600700 1 -- 172.21.15.62:0/3296293145 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa40819d610 con 0x7fa3ec06ecc0 2024-04-20T22:12:18.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.773+0000 7fa3ffe00700 1 -- 172.21.15.62:0/3296293145 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fa40819d610 con 0x7fa3ec06ecc0 2024-04-20T22:12:18.773 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:12:18.773 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T22:12:18.773 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T22:12:18.773 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-20T22:12:18.773 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T22:12:18.773 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr", 2024-04-20T22:12:18.773 INFO:teuthology.orchestra.run.smithi062.stdout: "crash", 2024-04-20T22:12:18.773 INFO:teuthology.orchestra.run.smithi062.stdout: "mon", 2024-04-20T22:12:18.773 INFO:teuthology.orchestra.run.smithi062.stdout: "osd" 2024-04-20T22:12:18.773 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T22:12:18.774 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "15/23 daemons upgraded", 2024-04-20T22:12:18.774 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading mds daemons", 2024-04-20T22:12:18.774 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T22:12:18.774 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:12:18.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.775+0000 7fa40e600700 1 -- 172.21.15.62:0/3296293145 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa3ec06ecc0 msgr2=0x7fa3ec071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:12:18.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.775+0000 7fa40e600700 1 --2- 172.21.15.62:0/3296293145 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa3ec06ecc0 0x7fa3ec071170 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fa4081997e0 tx=0x7fa3f4008040 comp rx=0 tx=0).stop 2024-04-20T22:12:18.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.775+0000 7fa40e600700 1 -- 172.21.15.62:0/3296293145 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa408073090 msgr2=0x7fa4081981c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:12:18.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.775+0000 7fa40e600700 1 --2- 172.21.15.62:0/3296293145 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa408073090 0x7fa4081981c0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7fa3f8005620 tx=0x7fa3f8005700 comp rx=0 tx=0).stop 2024-04-20T22:12:18.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.775+0000 7fa40e600700 1 -- 172.21.15.62:0/3296293145 shutdown_connections 2024-04-20T22:12:18.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.775+0000 7fa40e600700 1 --2- 172.21.15.62:0/3296293145 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa3ec06ecc0 0x7fa3ec071170 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:12:18.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.776+0000 7fa40e600700 1 --2- 172.21.15.62:0/3296293145 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa408073090 0x7fa4081981c0 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:12:18.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.776+0000 7fa40e600700 1 --2- 172.21.15.62:0/3296293145 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4080739a0 0x7fa408198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:12:18.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.776+0000 7fa40e600700 1 -- 172.21.15.62:0/3296293145 >> 172.21.15.62:0/3296293145 conn(0x7fa4080fbda0 msgr2=0x7fa408106e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:12:18.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.776+0000 7fa40e600700 1 -- 172.21.15.62:0/3296293145 shutdown_connections 2024-04-20T22:12:18.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.776+0000 7fa40e600700 1 -- 172.21.15.62:0/3296293145 wait complete. 2024-04-20T22:12:18.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.936+0000 7f1976a00700 1 -- 172.21.15.62:0/1780457388 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f197010c8f0 msgr2=0x7f197010ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:12:18.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.936+0000 7f1976a00700 1 --2- 172.21.15.62:0/1780457388 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f197010c8f0 0x7f197010ccc0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f1960009910 tx=0x7f1960009c20 comp rx=0 tx=0).stop 2024-04-20T22:12:18.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.937+0000 7f1976a00700 1 -- 172.21.15.62:0/1780457388 shutdown_connections 2024-04-20T22:12:18.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.937+0000 7f1976a00700 1 --2- 172.21.15.62:0/1780457388 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1970106b80 0x7f1970106ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:12:18.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.937+0000 7f1976a00700 1 --2- 172.21.15.62:0/1780457388 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f197010c8f0 0x7f197010ccc0 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:12:18.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.937+0000 7f1976a00700 1 -- 172.21.15.62:0/1780457388 >> 172.21.15.62:0/1780457388 conn(0x7f1970076060 msgr2=0x7f1970078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:12:18.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.937+0000 7f1976a00700 1 -- 172.21.15.62:0/1780457388 shutdown_connections 2024-04-20T22:12:18.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.937+0000 7f1976a00700 1 -- 172.21.15.62:0/1780457388 wait complete. 2024-04-20T22:12:18.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.938+0000 7f1976a00700 1 Processor -- start 2024-04-20T22:12:18.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.938+0000 7f1976a00700 1 -- start start 2024-04-20T22:12:18.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.938+0000 7f1976a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1970106b80 0x7f197019c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:12:18.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.938+0000 7f1976a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f197010c8f0 0x7f197019ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:12:18.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.938+0000 7f1976a00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f197019d120 con 0x7f1970106b80 2024-04-20T22:12:18.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.938+0000 7f1976a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19701a0eb0 con 0x7f197010c8f0 2024-04-20T22:12:18.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.939+0000 7f196fe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1970106b80 0x7f197019c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:12:18.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.939+0000 7f196fe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1970106b80 0x7f197019c500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33352/0 (socket says 172.21.15.62:33352) 2024-04-20T22:12:18.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.939+0000 7f196fe00700 1 -- 172.21.15.62:0/3775735954 learned_addr learned my addr 172.21.15.62:0/3775735954 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:12:18.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.939+0000 7f196f400700 1 --2- 172.21.15.62:0/3775735954 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f197010c8f0 0x7f197019ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:12:18.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.939+0000 7f196fe00700 1 -- 172.21.15.62:0/3775735954 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f197010c8f0 msgr2=0x7f197019ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:12:18.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.939+0000 7f196fe00700 1 --2- 172.21.15.62:0/3775735954 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f197010c8f0 0x7f197019ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:12:18.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.939+0000 7f196fe00700 1 -- 172.21.15.62:0/3775735954 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19600095f0 con 0x7f1970106b80 2024-04-20T22:12:18.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.939+0000 7f196fe00700 1 --2- 172.21.15.62:0/3775735954 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1970106b80 0x7f197019c500 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f1960006010 tx=0x7f196000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:12:18.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.940+0000 7f196cc00700 1 -- 172.21.15.62:0/3775735954 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f196001d070 con 0x7f1970106b80 2024-04-20T22:12:18.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.940+0000 7f196cc00700 1 -- 172.21.15.62:0/3775735954 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f196000f460 con 0x7f1970106b80 2024-04-20T22:12:18.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.940+0000 7f196cc00700 1 -- 172.21.15.62:0/3775735954 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1960017630 con 0x7f1970106b80 2024-04-20T22:12:18.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.940+0000 7f1976a00700 1 -- 172.21.15.62:0/3775735954 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19701a1130 con 0x7f1970106b80 2024-04-20T22:12:18.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.940+0000 7f1976a00700 1 -- 172.21.15.62:0/3775735954 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19701a15d0 con 0x7f1970106b80 2024-04-20T22:12:18.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.941+0000 7f1976a00700 1 -- 172.21.15.62:0/3775735954 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1970109b20 con 0x7f1970106b80 2024-04-20T22:12:18.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.942+0000 7f196cc00700 1 -- 172.21.15.62:0/3775735954 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f196000f5d0 con 0x7f1970106b80 2024-04-20T22:12:18.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.943+0000 7f196cc00700 1 --2- 172.21.15.62:0/3775735954 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f195406ecc0 0x7f1954071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:12:18.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.943+0000 7f196cc00700 1 -- 172.21.15.62:0/3775735954 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f1960090330 con 0x7f1970106b80 2024-04-20T22:12:18.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.944+0000 7f196f400700 1 --2- 172.21.15.62:0/3775735954 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f195406ecc0 0x7f1954071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:12:18.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.944+0000 7f196f400700 1 --2- 172.21.15.62:0/3775735954 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f195406ecc0 0x7f1954071170 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f195c005950 tx=0x7f195c0058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:12:18.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:18.948+0000 7f196cc00700 1 -- 172.21.15.62:0/3775735954 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f19600590c0 con 0x7f1970106b80 2024-04-20T22:12:19.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:19.276+0000 7f1976a00700 1 -- 172.21.15.62:0/3775735954 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f197004f2a0 con 0x7f1970106b80 2024-04-20T22:12:19.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:19.277+0000 7f196cc00700 1 -- 172.21.15.62:0/3775735954 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7f196005c780 con 0x7f1970106b80 2024-04-20T22:12:19.278 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-20T22:12:19.278 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-20T22:12:19.278 INFO:teuthology.orchestra.run.smithi062.stdout: host smithi062 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T22:12:19.278 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-20T22:12:19.278 INFO:teuthology.orchestra.run.smithi062.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi062 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpq7y7oa_r:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:12:19.278 INFO:teuthology.orchestra.run.smithi062.stdout:Traceback (most recent call last): 2024-04-20T22:12:19.278 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-20T22:12:19.278 INFO:teuthology.orchestra.run.smithi062.stdout: return _run_code(code, main_globals, None, 2024-04-20T22:12:19.278 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-20T22:12:19.279 INFO:teuthology.orchestra.run.smithi062.stdout: exec(code, run_globals) 2024-04-20T22:12:19.279 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5618, in 2024-04-20T22:12:19.279 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5606, in main 2024-04-20T22:12:19.279 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 408, in _infer_config 2024-04-20T22:12:19.279 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 323, in _infer_fsid 2024-04-20T22:12:19.279 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 436, in _infer_image 2024-04-20T22:12:19.279 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 310, in _validate_fsid 2024-04-20T22:12:19.279 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 3359, in command_ceph_volume 2024-04-20T22:12:19.279 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-20T22:12:19.279 INFO:teuthology.orchestra.run.smithi062.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi062 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpq7y7oa_r:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:12:19.279 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-20T22:12:19.279 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is degraded 2024-04-20T22:12:19.279 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T22:12:19.280 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T22:12:19.280 INFO:teuthology.orchestra.run.smithi062.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-20T22:12:19.280 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-20T22:12:19.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:19.279+0000 7f1976a00700 1 -- 172.21.15.62:0/3775735954 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f195406ecc0 msgr2=0x7f1954071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:12:19.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:19.279+0000 7f1976a00700 1 --2- 172.21.15.62:0/3775735954 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f195406ecc0 0x7f1954071170 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f195c005950 tx=0x7f195c0058e0 comp rx=0 tx=0).stop 2024-04-20T22:12:19.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:19.279+0000 7f1976a00700 1 -- 172.21.15.62:0/3775735954 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1970106b80 msgr2=0x7f197019c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:12:19.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:19.279+0000 7f1976a00700 1 --2- 172.21.15.62:0/3775735954 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1970106b80 0x7f197019c500 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f1960006010 tx=0x7f196000be20 comp rx=0 tx=0).stop 2024-04-20T22:12:19.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:19.279+0000 7f1976a00700 1 -- 172.21.15.62:0/3775735954 shutdown_connections 2024-04-20T22:12:19.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:19.279+0000 7f1976a00700 1 --2- 172.21.15.62:0/3775735954 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f195406ecc0 0x7f1954071170 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:12:19.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:19.279+0000 7f1976a00700 1 --2- 172.21.15.62:0/3775735954 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1970106b80 0x7f197019c500 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:12:19.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:19.279+0000 7f1976a00700 1 --2- 172.21.15.62:0/3775735954 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f197010c8f0 0x7f197019ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:12:19.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:19.279+0000 7f1976a00700 1 -- 172.21.15.62:0/3775735954 >> 172.21.15.62:0/3775735954 conn(0x7f1970076060 msgr2=0x7f1970077aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:12:19.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:19.280+0000 7f1976a00700 1 -- 172.21.15.62:0/3775735954 shutdown_connections 2024-04-20T22:12:19.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:19.280+0000 7f1976a00700 1 -- 172.21.15.62:0/3775735954 wait complete. 2024-04-20T22:12:19.281 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:12:19 smithi062 ceph-mon[76322]: from='client.35972 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:12:19.282 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:12:19 smithi062 ceph-mon[76322]: pgmap v1111: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:12:19.282 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:12:19 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/1638338973' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:12:19.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:12:19 smithi189 ceph-mon[57022]: from='client.35972 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:12:19.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:12:19 smithi189 ceph-mon[57022]: pgmap v1111: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:12:19.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:12:19 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/1638338973' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:12:20.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:12:20 smithi062 ceph-mon[76322]: from='client.35984 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:12:20.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:12:20 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/3775735954' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:12:20.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:12:20 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:12:20.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:12:20 smithi189 ceph-mon[57022]: from='client.35984 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:12:20.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:12:20 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/3775735954' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:12:20.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:12:20 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:12:21.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:12:21 smithi189 ceph-mon[57022]: pgmap v1112: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:12:21.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:12:21 smithi062 ceph-mon[76322]: pgmap v1112: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:12:23.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:12:23 smithi189 ceph-mon[57022]: pgmap v1113: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:12:23.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:12:23 smithi062 ceph-mon[76322]: pgmap v1113: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:12:25.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:12:25 smithi062 ceph-mon[76322]: pgmap v1114: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:12:25.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:12:25 smithi189 ceph-mon[57022]: pgmap v1114: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:12:27.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:12:27 smithi189 ceph-mon[57022]: pgmap v1115: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:12:27.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:12:27 smithi062 ceph-mon[76322]: pgmap v1115: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:12:29.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:12:29 smithi189 ceph-mon[57022]: pgmap v1116: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:12:29.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:12:29 smithi062 ceph-mon[76322]: pgmap v1116: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:12:31.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:12:31 smithi189 ceph-mon[57022]: pgmap v1117: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:12:31.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:12:31 smithi062 ceph-mon[76322]: pgmap v1117: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:12:33.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:12:33 smithi189 ceph-mon[57022]: pgmap v1118: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:12:33.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:12:33 smithi062 ceph-mon[76322]: pgmap v1118: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:12:35.490 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:12:35 smithi189 ceph-mon[57022]: pgmap v1119: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:12:35.490 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:12:35 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:12:35.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:12:35 smithi062 ceph-mon[76322]: pgmap v1119: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:12:35.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:12:35 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:12:37.455 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:12:37 smithi189 ceph-mon[57022]: pgmap v1120: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:12:37.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:12:37 smithi062 ceph-mon[76322]: pgmap v1120: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:12:39.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:12:39 smithi189 ceph-mon[57022]: pgmap v1121: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:12:39.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:12:39 smithi062 ceph-mon[76322]: pgmap v1121: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:12:41.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:12:41 smithi189 ceph-mon[57022]: pgmap v1122: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:12:41.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:12:41 smithi062 ceph-mon[76322]: pgmap v1122: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:12:43.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:12:43 smithi189 ceph-mon[57022]: pgmap v1123: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:12:43.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:12:43 smithi062 ceph-mon[76322]: pgmap v1123: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:12:45.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:12:45 smithi189 ceph-mon[57022]: pgmap v1124: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:12:45.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:12:45 smithi062 ceph-mon[76322]: pgmap v1124: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:12:47.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:12:47 smithi189 ceph-mon[57022]: pgmap v1125: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:12:47.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:12:47 smithi062 ceph-mon[76322]: pgmap v1125: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:12:49.457 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:12:49 smithi062 ceph-mon[76322]: pgmap v1126: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:12:49.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:49.457+0000 7fe9db200700 1 -- 172.21.15.62:0/71496774 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9d4100290 msgr2=0x7fe9d4100660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:12:49.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:49.457+0000 7fe9db200700 1 --2- 172.21.15.62:0/71496774 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9d4100290 0x7fe9d4100660 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7fe9c8009910 tx=0x7fe9c8009c20 comp rx=0 tx=0).stop 2024-04-20T22:12:49.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:49.457+0000 7fe9db200700 1 -- 172.21.15.62:0/71496774 shutdown_connections 2024-04-20T22:12:49.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:49.457+0000 7fe9db200700 1 --2- 172.21.15.62:0/71496774 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe9d4100c30 0x7fe9d4104c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:12:49.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:49.457+0000 7fe9db200700 1 --2- 172.21.15.62:0/71496774 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9d4100290 0x7fe9d4100660 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:12:49.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:49.457+0000 7fe9db200700 1 -- 172.21.15.62:0/71496774 >> 172.21.15.62:0/71496774 conn(0x7fe9d40fbb10 msgr2=0x7fe9d40fdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:12:49.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:49.458+0000 7fe9db200700 1 -- 172.21.15.62:0/71496774 shutdown_connections 2024-04-20T22:12:49.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:49.458+0000 7fe9db200700 1 -- 172.21.15.62:0/71496774 wait complete. 2024-04-20T22:12:49.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:49.459+0000 7fe9db200700 1 Processor -- start 2024-04-20T22:12:49.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:49.459+0000 7fe9db200700 1 -- start start 2024-04-20T22:12:49.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:49.460+0000 7fe9db200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9d4100290 0x7fe9d4193820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:12:49.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:49.460+0000 7fe9db200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe9d4100c30 0x7fe9d4193d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:12:49.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:49.460+0000 7fe9db200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9d4196910 con 0x7fe9d4100290 2024-04-20T22:12:49.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:49.460+0000 7fe9db200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9d4196a80 con 0x7fe9d4100c30 2024-04-20T22:12:49.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:49.460+0000 7fe9d9e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9d4100290 0x7fe9d4193820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:12:49.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:49.460+0000 7fe9d9e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9d4100290 0x7fe9d4193820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:57900/0 (socket says 172.21.15.62:57900) 2024-04-20T22:12:49.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:49.460+0000 7fe9d9e00700 1 -- 172.21.15.62:0/378276810 learned_addr learned my addr 172.21.15.62:0/378276810 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:12:49.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:49.461+0000 7fe9d9400700 1 --2- 172.21.15.62:0/378276810 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe9d4100c30 0x7fe9d4193d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:12:49.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:49.461+0000 7fe9d9e00700 1 -- 172.21.15.62:0/378276810 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe9d4100c30 msgr2=0x7fe9d4193d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:12:49.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:49.461+0000 7fe9d9e00700 1 --2- 172.21.15.62:0/378276810 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe9d4100c30 0x7fe9d4193d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:12:49.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:49.461+0000 7fe9d9e00700 1 -- 172.21.15.62:0/378276810 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9c80095f0 con 0x7fe9d4100290 2024-04-20T22:12:49.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:49.461+0000 7fe9d9e00700 1 --2- 172.21.15.62:0/378276810 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9d4100290 0x7fe9d4193820 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7fe9c8006010 tx=0x7fe9c800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:12:49.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:49.461+0000 7fe9cea00700 1 -- 172.21.15.62:0/378276810 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe9c801d070 con 0x7fe9d4100290 2024-04-20T22:12:49.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:49.461+0000 7fe9cea00700 1 -- 172.21.15.62:0/378276810 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe9c8004fb0 con 0x7fe9d4100290 2024-04-20T22:12:49.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:49.461+0000 7fe9cea00700 1 -- 172.21.15.62:0/378276810 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe9c800f690 con 0x7fe9d4100290 2024-04-20T22:12:49.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:49.462+0000 7fe9db200700 1 -- 172.21.15.62:0/378276810 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9d41942a0 con 0x7fe9d4100290 2024-04-20T22:12:49.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:49.462+0000 7fe9db200700 1 -- 172.21.15.62:0/378276810 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9d41a6560 con 0x7fe9d4100290 2024-04-20T22:12:49.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:49.463+0000 7fe9db200700 1 -- 172.21.15.62:0/378276810 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9d4104380 con 0x7fe9d4100290 2024-04-20T22:12:49.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:49.464+0000 7fe9cea00700 1 -- 172.21.15.62:0/378276810 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe9c80228a0 con 0x7fe9d4100290 2024-04-20T22:12:49.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:49.465+0000 7fe9cea00700 1 --2- 172.21.15.62:0/378276810 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe9bc06ecc0 0x7fe9bc071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:12:49.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:49.466+0000 7fe9cea00700 1 -- 172.21.15.62:0/378276810 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fe9c8090700 con 0x7fe9d4100290 2024-04-20T22:12:49.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:49.466+0000 7fe9d9400700 1 --2- 172.21.15.62:0/378276810 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe9bc06ecc0 0x7fe9bc071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:12:49.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:49.467+0000 7fe9d9400700 1 --2- 172.21.15.62:0/378276810 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe9bc06ecc0 0x7fe9bc071170 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fe9c4005ea0 tx=0x7fe9c4005e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:12:49.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:49.471+0000 7fe9cea00700 1 -- 172.21.15.62:0/378276810 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fe9c8059490 con 0x7fe9d4100290 2024-04-20T22:12:49.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:12:49 smithi189 ceph-mon[57022]: pgmap v1126: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:12:49.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:49.732+0000 7fe9db200700 1 -- 172.21.15.62:0/378276810 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe9d4061190 con 0x7fe9bc06ecc0 2024-04-20T22:12:49.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:49.735+0000 7fe9cea00700 1 -- 172.21.15.62:0/378276810 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fe9d4061190 con 0x7fe9bc06ecc0 2024-04-20T22:12:49.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:49.738+0000 7fe9db200700 1 -- 172.21.15.62:0/378276810 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe9bc06ecc0 msgr2=0x7fe9bc071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:12:49.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:49.738+0000 7fe9db200700 1 --2- 172.21.15.62:0/378276810 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe9bc06ecc0 0x7fe9bc071170 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fe9c4005ea0 tx=0x7fe9c4005e30 comp rx=0 tx=0).stop 2024-04-20T22:12:49.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:49.738+0000 7fe9db200700 1 -- 172.21.15.62:0/378276810 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9d4100290 msgr2=0x7fe9d4193820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:12:49.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:49.738+0000 7fe9db200700 1 --2- 172.21.15.62:0/378276810 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9d4100290 0x7fe9d4193820 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7fe9c8006010 tx=0x7fe9c800be20 comp rx=0 tx=0).stop 2024-04-20T22:12:49.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:49.738+0000 7fe9db200700 1 -- 172.21.15.62:0/378276810 shutdown_connections 2024-04-20T22:12:49.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:49.738+0000 7fe9db200700 1 --2- 172.21.15.62:0/378276810 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe9bc06ecc0 0x7fe9bc071170 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:12:49.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:49.738+0000 7fe9db200700 1 --2- 172.21.15.62:0/378276810 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9d4100290 0x7fe9d4193820 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:12:49.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:49.738+0000 7fe9db200700 1 --2- 172.21.15.62:0/378276810 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe9d4100c30 0x7fe9d4193d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:12:49.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:49.738+0000 7fe9db200700 1 -- 172.21.15.62:0/378276810 >> 172.21.15.62:0/378276810 conn(0x7fe9d40fbb10 msgr2=0x7fe9d40fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:12:49.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:49.739+0000 7fe9db200700 1 -- 172.21.15.62:0/378276810 shutdown_connections 2024-04-20T22:12:49.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:49.739+0000 7fe9db200700 1 -- 172.21.15.62:0/378276810 wait complete. 2024-04-20T22:12:49.755 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T22:12:49.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:49.908+0000 7f0c1f200700 1 -- 172.21.15.62:0/3078881484 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c18106b80 msgr2=0x7f0c18106ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:12:49.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:49.908+0000 7f0c1f200700 1 --2- 172.21.15.62:0/3078881484 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c18106b80 0x7f0c18106ff0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f0c08009910 tx=0x7f0c08009c20 comp rx=0 tx=0).stop 2024-04-20T22:12:49.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:49.908+0000 7f0c1f200700 1 -- 172.21.15.62:0/3078881484 shutdown_connections 2024-04-20T22:12:49.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:49.908+0000 7f0c1f200700 1 --2- 172.21.15.62:0/3078881484 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c18106b80 0x7f0c18106ff0 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:12:49.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:49.908+0000 7f0c1f200700 1 --2- 172.21.15.62:0/3078881484 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c1810c660 0x7f0c1810ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:12:49.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:49.908+0000 7f0c1f200700 1 -- 172.21.15.62:0/3078881484 >> 172.21.15.62:0/3078881484 conn(0x7f0c18075e90 msgr2=0x7f0c180782a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:12:49.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:49.908+0000 7f0c1f200700 1 -- 172.21.15.62:0/3078881484 shutdown_connections 2024-04-20T22:12:49.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:49.908+0000 7f0c1f200700 1 -- 172.21.15.62:0/3078881484 wait complete. 2024-04-20T22:12:49.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:49.909+0000 7f0c1f200700 1 Processor -- start 2024-04-20T22:12:49.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:49.909+0000 7f0c1f200700 1 -- start start 2024-04-20T22:12:49.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:49.910+0000 7f0c1f200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c18106b80 0x7f0c1819c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:12:49.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:49.910+0000 7f0c1f200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c1810c660 0x7f0c1819c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:12:49.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:49.910+0000 7f0c1f200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c1819ce00 con 0x7f0c18106b80 2024-04-20T22:12:49.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:49.910+0000 7f0c1f200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c181a0b90 con 0x7f0c1810c660 2024-04-20T22:12:49.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:49.910+0000 7f0c1de00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c18106b80 0x7f0c1819c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:12:49.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:49.910+0000 7f0c1de00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c18106b80 0x7f0c1819c1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:57926/0 (socket says 172.21.15.62:57926) 2024-04-20T22:12:49.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:49.910+0000 7f0c1de00700 1 -- 172.21.15.62:0/3428434124 learned_addr learned my addr 172.21.15.62:0/3428434124 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:12:49.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:49.910+0000 7f0c1d400700 1 --2- 172.21.15.62:0/3428434124 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c1810c660 0x7f0c1819c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:12:49.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:49.910+0000 7f0c1de00700 1 -- 172.21.15.62:0/3428434124 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c1810c660 msgr2=0x7f0c1819c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:12:49.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:49.911+0000 7f0c1de00700 1 --2- 172.21.15.62:0/3428434124 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c1810c660 0x7f0c1819c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:12:49.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:49.911+0000 7f0c1de00700 1 -- 172.21.15.62:0/3428434124 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c080095f0 con 0x7f0c18106b80 2024-04-20T22:12:49.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:49.911+0000 7f0c1d400700 1 --2- 172.21.15.62:0/3428434124 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c1810c660 0x7f0c1819c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T22:12:49.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:49.911+0000 7f0c1de00700 1 --2- 172.21.15.62:0/3428434124 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c18106b80 0x7f0c1819c1e0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f0c0c00d8d0 tx=0x7f0c0c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:12:49.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:49.911+0000 7f0c12a00700 1 -- 172.21.15.62:0/3428434124 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0c0c009940 con 0x7f0c18106b80 2024-04-20T22:12:49.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:49.911+0000 7f0c12a00700 1 -- 172.21.15.62:0/3428434124 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0c0c010460 con 0x7f0c18106b80 2024-04-20T22:12:49.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:49.911+0000 7f0c12a00700 1 -- 172.21.15.62:0/3428434124 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0c0c00f5f0 con 0x7f0c18106b80 2024-04-20T22:12:49.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:49.911+0000 7f0c1f200700 1 -- 172.21.15.62:0/3428434124 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c181a0e70 con 0x7f0c18106b80 2024-04-20T22:12:49.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:49.911+0000 7f0c1f200700 1 -- 172.21.15.62:0/3428434124 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c181a13c0 con 0x7f0c18106b80 2024-04-20T22:12:49.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:49.913+0000 7f0c1f200700 1 -- 172.21.15.62:0/3428434124 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c1810eb40 con 0x7f0c18106b80 2024-04-20T22:12:49.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:49.914+0000 7f0c12a00700 1 -- 172.21.15.62:0/3428434124 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f0c0c0105d0 con 0x7f0c18106b80 2024-04-20T22:12:49.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:49.915+0000 7f0c12a00700 1 --2- 172.21.15.62:0/3428434124 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f0c0006ecc0 0x7f0c00071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:12:49.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:49.915+0000 7f0c12a00700 1 -- 172.21.15.62:0/3428434124 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f0c0c08ef20 con 0x7f0c18106b80 2024-04-20T22:12:49.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:49.915+0000 7f0c1d400700 1 --2- 172.21.15.62:0/3428434124 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f0c0006ecc0 0x7f0c00071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:12:49.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:49.916+0000 7f0c1d400700 1 --2- 172.21.15.62:0/3428434124 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f0c0006ecc0 0x7f0c00071170 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f0c08006010 tx=0x7f0c0800b540 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:12:49.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:49.920+0000 7f0c12a00700 1 -- 172.21.15.62:0/3428434124 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f0c0c057c30 con 0x7f0c18106b80 2024-04-20T22:12:50.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:50.184+0000 7f0c1f200700 1 -- 172.21.15.62:0/3428434124 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0c181a15e0 con 0x7f0c0006ecc0 2024-04-20T22:12:50.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:50.187+0000 7f0c12a00700 1 -- 172.21.15.62:0/3428434124 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f0c181a15e0 con 0x7f0c0006ecc0 2024-04-20T22:12:50.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:50.190+0000 7f0c1f200700 1 -- 172.21.15.62:0/3428434124 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f0c0006ecc0 msgr2=0x7f0c00071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:12:50.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:50.190+0000 7f0c1f200700 1 --2- 172.21.15.62:0/3428434124 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f0c0006ecc0 0x7f0c00071170 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f0c08006010 tx=0x7f0c0800b540 comp rx=0 tx=0).stop 2024-04-20T22:12:50.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:50.190+0000 7f0c1f200700 1 -- 172.21.15.62:0/3428434124 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c18106b80 msgr2=0x7f0c1819c1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:12:50.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:50.190+0000 7f0c1f200700 1 --2- 172.21.15.62:0/3428434124 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c18106b80 0x7f0c1819c1e0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f0c0c00d8d0 tx=0x7f0c0c00dc90 comp rx=0 tx=0).stop 2024-04-20T22:12:50.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:50.190+0000 7f0c1f200700 1 -- 172.21.15.62:0/3428434124 shutdown_connections 2024-04-20T22:12:50.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:50.190+0000 7f0c1f200700 1 --2- 172.21.15.62:0/3428434124 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f0c0006ecc0 0x7f0c00071170 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:12:50.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:50.190+0000 7f0c1f200700 1 --2- 172.21.15.62:0/3428434124 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c18106b80 0x7f0c1819c1e0 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:12:50.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:50.190+0000 7f0c1f200700 1 --2- 172.21.15.62:0/3428434124 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c1810c660 0x7f0c1819c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:12:50.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:50.190+0000 7f0c1f200700 1 -- 172.21.15.62:0/3428434124 >> 172.21.15.62:0/3428434124 conn(0x7f0c18075e90 msgr2=0x7f0c180778d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:12:50.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:50.190+0000 7f0c1f200700 1 -- 172.21.15.62:0/3428434124 shutdown_connections 2024-04-20T22:12:50.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:50.190+0000 7f0c1f200700 1 -- 172.21.15.62:0/3428434124 wait complete. 2024-04-20T22:12:50.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:50.353+0000 7fc853000700 1 -- 172.21.15.62:0/1727525788 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc84c106b80 msgr2=0x7fc84c106ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:12:50.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:50.353+0000 7fc853000700 1 --2- 172.21.15.62:0/1727525788 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc84c106b80 0x7fc84c106ff0 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7fc838009910 tx=0x7fc838009c20 comp rx=0 tx=0).stop 2024-04-20T22:12:50.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:50.353+0000 7fc853000700 1 -- 172.21.15.62:0/1727525788 shutdown_connections 2024-04-20T22:12:50.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:50.353+0000 7fc853000700 1 --2- 172.21.15.62:0/1727525788 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc84c106b80 0x7fc84c106ff0 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:12:50.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:50.353+0000 7fc853000700 1 --2- 172.21.15.62:0/1727525788 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc84c10c660 0x7fc84c10ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:12:50.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:50.353+0000 7fc853000700 1 -- 172.21.15.62:0/1727525788 >> 172.21.15.62:0/1727525788 conn(0x7fc84c075e90 msgr2=0x7fc84c0782a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:12:50.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:50.353+0000 7fc853000700 1 -- 172.21.15.62:0/1727525788 shutdown_connections 2024-04-20T22:12:50.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:50.353+0000 7fc853000700 1 -- 172.21.15.62:0/1727525788 wait complete. 2024-04-20T22:12:50.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:50.355+0000 7fc853000700 1 Processor -- start 2024-04-20T22:12:50.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:50.355+0000 7fc853000700 1 -- start start 2024-04-20T22:12:50.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:50.355+0000 7fc853000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc84c106b80 0x7fc84c19c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:12:50.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:50.355+0000 7fc853000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc84c10c660 0x7fc84c19c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:12:50.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:50.355+0000 7fc853000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc84c19ce00 con 0x7fc84c106b80 2024-04-20T22:12:50.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:50.355+0000 7fc853000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc84c1a0b90 con 0x7fc84c10c660 2024-04-20T22:12:50.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:50.356+0000 7fc851c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc84c106b80 0x7fc84c19c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:12:50.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:50.356+0000 7fc851c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc84c106b80 0x7fc84c19c1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:57942/0 (socket says 172.21.15.62:57942) 2024-04-20T22:12:50.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:50.356+0000 7fc851c00700 1 -- 172.21.15.62:0/661186242 learned_addr learned my addr 172.21.15.62:0/661186242 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:12:50.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:50.356+0000 7fc851200700 1 --2- 172.21.15.62:0/661186242 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc84c10c660 0x7fc84c19c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:12:50.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:50.356+0000 7fc851c00700 1 -- 172.21.15.62:0/661186242 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc84c10c660 msgr2=0x7fc84c19c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:12:50.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:50.356+0000 7fc851c00700 1 --2- 172.21.15.62:0/661186242 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc84c10c660 0x7fc84c19c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:12:50.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:50.356+0000 7fc851c00700 1 -- 172.21.15.62:0/661186242 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8380095f0 con 0x7fc84c106b80 2024-04-20T22:12:50.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:50.356+0000 7fc851c00700 1 --2- 172.21.15.62:0/661186242 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc84c106b80 0x7fc84c19c1e0 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7fc84000c8a0 tx=0x7fc84000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:12:50.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:50.357+0000 7fc84a000700 1 -- 172.21.15.62:0/661186242 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc84000cea0 con 0x7fc84c106b80 2024-04-20T22:12:50.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:50.357+0000 7fc84a000700 1 -- 172.21.15.62:0/661186242 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc840004830 con 0x7fc84c106b80 2024-04-20T22:12:50.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:50.357+0000 7fc853000700 1 -- 172.21.15.62:0/661186242 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc84c1a0e70 con 0x7fc84c106b80 2024-04-20T22:12:50.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:50.357+0000 7fc84a000700 1 -- 172.21.15.62:0/661186242 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc8400055b0 con 0x7fc84c106b80 2024-04-20T22:12:50.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:50.357+0000 7fc853000700 1 -- 172.21.15.62:0/661186242 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc84c1a13c0 con 0x7fc84c106b80 2024-04-20T22:12:50.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:50.358+0000 7fc853000700 1 -- 172.21.15.62:0/661186242 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc84c10eb40 con 0x7fc84c106b80 2024-04-20T22:12:50.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:50.359+0000 7fc84a000700 1 -- 172.21.15.62:0/661186242 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fc8400049a0 con 0x7fc84c106b80 2024-04-20T22:12:50.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:50.360+0000 7fc84a000700 1 --2- 172.21.15.62:0/661186242 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc83406ecc0 0x7fc834071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:12:50.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:50.360+0000 7fc84a000700 1 -- 172.21.15.62:0/661186242 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fc84008eaf0 con 0x7fc84c106b80 2024-04-20T22:12:50.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:50.360+0000 7fc851200700 1 --2- 172.21.15.62:0/661186242 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc83406ecc0 0x7fc834071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:12:50.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:50.361+0000 7fc851200700 1 --2- 172.21.15.62:0/661186242 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc83406ecc0 0x7fc834071170 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fc84c19d800 tx=0x7fc83801a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:12:50.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:50.365+0000 7fc84a000700 1 -- 172.21.15.62:0/661186242 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fc840057880 con 0x7fc84c106b80 2024-04-20T22:12:50.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:12:50 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:12:50.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:12:50 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:12:50.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:50.602+0000 7fc853000700 1 -- 172.21.15.62:0/661186242 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fc84c1a1690 con 0x7fc83406ecc0 2024-04-20T22:12:50.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:50.634+0000 7fc84a000700 1 -- 172.21.15.62:0/661186242 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fc84c1a1690 con 0x7fc83406ecc0 2024-04-20T22:12:50.635 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T22:12:50.635 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (38m) 14m ago 52m 15.3M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T22:12:50.635 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (52m) 14m ago 52m 10.4M - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T22:12:50.635 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (51m) 14m ago 51m 10.1M - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T22:12:50.635 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (34m) 14m ago 52m 6660k - 19.0.0-3133-ga8c84f59 76bc6b91b726 2825d5ba11c6 2024-04-20T22:12:50.635 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (34m) 14m ago 51m 6665k - 19.0.0-3133-ga8c84f59 76bc6b91b726 20f746fa1bba 2024-04-20T22:12:50.635 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (38m) 14m ago 52m 73.5M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T22:12:50.635 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (30m) 14m ago 45m 18.5M - 19.0.0-3133-ga8c84f59 76bc6b91b726 de80bc90a683 2024-04-20T22:12:50.635 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (30m) 14m ago 45m 18.7M - 19.0.0-3133-ga8c84f59 76bc6b91b726 cd793b7ee98e 2024-04-20T22:12:50.635 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (14m) 14m ago 45m 11.0M - 19.0.0-3133-ga8c84f59 76bc6b91b726 0e7b0bc4d1ab 2024-04-20T22:12:50.635 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (45m) 14m ago 45m 18.6M - 18.2.0 dc2bc1663786 f2eec2ee8951 2024-04-20T22:12:50.635 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (41m) 14m ago 54m 513M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T22:12:50.635 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (40m) 14m ago 51m 459M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T22:12:50.636 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (36m) 14m ago 54m 71.7M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 019eb1479485 2024-04-20T22:12:50.636 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (35m) 14m ago 51m 51.0M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 55aac560aab6 2024-04-20T22:12:50.636 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (39m) 14m ago 52m 8472k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T22:12:50.636 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (39m) 14m ago 51m 8048k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T22:12:50.636 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (34m) 14m ago 50m 91.2M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 ee2bb901f26e 2024-04-20T22:12:50.636 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (33m) 14m ago 49m 94.6M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 9075152d4162 2024-04-20T22:12:50.636 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (33m) 14m ago 48m 78.1M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 2d4f3e38dc0e 2024-04-20T22:12:50.636 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (32m) 14m ago 48m 92.6M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 8f97b17ae0eb 2024-04-20T22:12:50.636 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (31m) 14m ago 47m 95.6M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 653d8079967d 2024-04-20T22:12:50.636 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (31m) 14m ago 47m 89.2M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 596dded14c12 2024-04-20T22:12:50.636 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (39m) 14m ago 51m 82.1M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T22:12:50.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:50.637+0000 7fc853000700 1 -- 172.21.15.62:0/661186242 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc83406ecc0 msgr2=0x7fc834071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:12:50.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:50.637+0000 7fc853000700 1 --2- 172.21.15.62:0/661186242 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc83406ecc0 0x7fc834071170 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fc84c19d800 tx=0x7fc83801a040 comp rx=0 tx=0).stop 2024-04-20T22:12:50.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:50.637+0000 7fc853000700 1 -- 172.21.15.62:0/661186242 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc84c106b80 msgr2=0x7fc84c19c1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:12:50.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:50.637+0000 7fc853000700 1 --2- 172.21.15.62:0/661186242 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc84c106b80 0x7fc84c19c1e0 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7fc84000c8a0 tx=0x7fc84000cc60 comp rx=0 tx=0).stop 2024-04-20T22:12:50.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:50.638+0000 7fc853000700 1 -- 172.21.15.62:0/661186242 shutdown_connections 2024-04-20T22:12:50.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:50.638+0000 7fc853000700 1 --2- 172.21.15.62:0/661186242 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc83406ecc0 0x7fc834071170 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:12:50.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:50.638+0000 7fc853000700 1 --2- 172.21.15.62:0/661186242 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc84c106b80 0x7fc84c19c1e0 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:12:50.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:50.638+0000 7fc853000700 1 --2- 172.21.15.62:0/661186242 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc84c10c660 0x7fc84c19c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:12:50.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:50.638+0000 7fc853000700 1 -- 172.21.15.62:0/661186242 >> 172.21.15.62:0/661186242 conn(0x7fc84c075e90 msgr2=0x7fc84c0778d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:12:50.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:50.638+0000 7fc853000700 1 -- 172.21.15.62:0/661186242 shutdown_connections 2024-04-20T22:12:50.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:50.638+0000 7fc853000700 1 -- 172.21.15.62:0/661186242 wait complete. 2024-04-20T22:12:50.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:50.797+0000 7f2954800700 1 -- 172.21.15.62:0/2759405397 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f294c108470 msgr2=0x7f294c108840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:12:50.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:50.797+0000 7f2954800700 1 --2- 172.21.15.62:0/2759405397 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f294c108470 0x7f294c108840 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f2940009910 tx=0x7f2940009c20 comp rx=0 tx=0).stop 2024-04-20T22:12:50.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:50.798+0000 7f2954800700 1 -- 172.21.15.62:0/2759405397 shutdown_connections 2024-04-20T22:12:50.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:50.798+0000 7f2954800700 1 --2- 172.21.15.62:0/2759405397 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f294c102470 0x7f294c1028e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:12:50.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:50.798+0000 7f2954800700 1 --2- 172.21.15.62:0/2759405397 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f294c108470 0x7f294c108840 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:12:50.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:50.798+0000 7f2954800700 1 -- 172.21.15.62:0/2759405397 >> 172.21.15.62:0/2759405397 conn(0x7f294c0fdf90 msgr2=0x7f294c1003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:12:50.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:50.798+0000 7f2954800700 1 -- 172.21.15.62:0/2759405397 shutdown_connections 2024-04-20T22:12:50.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:50.798+0000 7f2954800700 1 -- 172.21.15.62:0/2759405397 wait complete. 2024-04-20T22:12:50.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:50.799+0000 7f2954800700 1 Processor -- start 2024-04-20T22:12:50.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:50.799+0000 7f2954800700 1 -- start start 2024-04-20T22:12:50.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:50.799+0000 7f2954800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f294c102470 0x7f294c1981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:12:50.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:50.799+0000 7f2954800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f294c108470 0x7f294c198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:12:50.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:50.800+0000 7f2954800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f294c198e10 con 0x7f294c102470 2024-04-20T22:12:50.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:50.800+0000 7f2954800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f294c19cba0 con 0x7f294c108470 2024-04-20T22:12:50.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:50.800+0000 7f2951e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f294c102470 0x7f294c1981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:12:50.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:50.800+0000 7f2951e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f294c102470 0x7f294c1981f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:57956/0 (socket says 172.21.15.62:57956) 2024-04-20T22:12:50.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:50.800+0000 7f2951e00700 1 -- 172.21.15.62:0/3244515701 learned_addr learned my addr 172.21.15.62:0/3244515701 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:12:50.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:50.800+0000 7f2951400700 1 --2- 172.21.15.62:0/3244515701 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f294c108470 0x7f294c198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:12:50.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:50.800+0000 7f2951e00700 1 -- 172.21.15.62:0/3244515701 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f294c108470 msgr2=0x7f294c198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:12:50.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:50.800+0000 7f2951e00700 1 --2- 172.21.15.62:0/3244515701 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f294c108470 0x7f294c198730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:12:50.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:50.800+0000 7f2951e00700 1 -- 172.21.15.62:0/3244515701 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29400095f0 con 0x7f294c102470 2024-04-20T22:12:50.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:50.801+0000 7f2951e00700 1 --2- 172.21.15.62:0/3244515701 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f294c102470 0x7f294c1981f0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f2940006010 tx=0x7f294000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:12:50.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:50.801+0000 7f2946a00700 1 -- 172.21.15.62:0/3244515701 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f294001d070 con 0x7f294c102470 2024-04-20T22:12:50.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:50.801+0000 7f2946a00700 1 -- 172.21.15.62:0/3244515701 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f294000f460 con 0x7f294c102470 2024-04-20T22:12:50.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:50.801+0000 7f2946a00700 1 -- 172.21.15.62:0/3244515701 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2940017610 con 0x7f294c102470 2024-04-20T22:12:50.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:50.801+0000 7f2954800700 1 -- 172.21.15.62:0/3244515701 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f294c19ce20 con 0x7f294c102470 2024-04-20T22:12:50.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:50.801+0000 7f2954800700 1 -- 172.21.15.62:0/3244515701 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f294c19d2c0 con 0x7f294c102470 2024-04-20T22:12:50.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:50.802+0000 7f2954800700 1 -- 172.21.15.62:0/3244515701 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f294c10a9e0 con 0x7f294c102470 2024-04-20T22:12:50.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:50.803+0000 7f2946a00700 1 -- 172.21.15.62:0/3244515701 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f294000f5d0 con 0x7f294c102470 2024-04-20T22:12:50.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:50.804+0000 7f2946a00700 1 --2- 172.21.15.62:0/3244515701 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f293406ecc0 0x7f2934071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:12:50.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:50.805+0000 7f2946a00700 1 -- 172.21.15.62:0/3244515701 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f2940090350 con 0x7f294c102470 2024-04-20T22:12:50.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:50.805+0000 7f2951400700 1 --2- 172.21.15.62:0/3244515701 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f293406ecc0 0x7f2934071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:12:50.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:50.805+0000 7f2951400700 1 --2- 172.21.15.62:0/3244515701 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f293406ecc0 0x7f2934071170 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f293c005950 tx=0x7f293c0058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:12:50.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:50.809+0000 7f2946a00700 1 -- 172.21.15.62:0/3244515701 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f29400591a0 con 0x7f294c102470 2024-04-20T22:12:51.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:51.173+0000 7f2954800700 1 -- 172.21.15.62:0/3244515701 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f294c199620 con 0x7f294c102470 2024-04-20T22:12:51.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:51.174+0000 7f2946a00700 1 -- 172.21.15.62:0/3244515701 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7f294005c860 con 0x7f294c102470 2024-04-20T22:12:51.174 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:12:51.174 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T22:12:51.174 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:12:51.174 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:12:51.174 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T22:12:51.174 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:12:51.174 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:12:51.174 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T22:12:51.174 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 6 2024-04-20T22:12:51.174 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:12:51.174 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T22:12:51.174 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 1, 2024-04-20T22:12:51.174 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 3 2024-04-20T22:12:51.175 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:12:51.175 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T22:12:51.175 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 1, 2024-04-20T22:12:51.175 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 13 2024-04-20T22:12:51.175 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T22:12:51.175 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:12:51.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:51.177+0000 7f2954800700 1 -- 172.21.15.62:0/3244515701 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f293406ecc0 msgr2=0x7f2934071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:12:51.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:51.177+0000 7f2954800700 1 --2- 172.21.15.62:0/3244515701 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f293406ecc0 0x7f2934071170 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f293c005950 tx=0x7f293c0058e0 comp rx=0 tx=0).stop 2024-04-20T22:12:51.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:51.177+0000 7f2954800700 1 -- 172.21.15.62:0/3244515701 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f294c102470 msgr2=0x7f294c1981f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:12:51.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:51.177+0000 7f2954800700 1 --2- 172.21.15.62:0/3244515701 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f294c102470 0x7f294c1981f0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f2940006010 tx=0x7f294000be20 comp rx=0 tx=0).stop 2024-04-20T22:12:51.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:51.177+0000 7f2954800700 1 -- 172.21.15.62:0/3244515701 shutdown_connections 2024-04-20T22:12:51.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:51.177+0000 7f2954800700 1 --2- 172.21.15.62:0/3244515701 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f293406ecc0 0x7f2934071170 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:12:51.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:51.177+0000 7f2954800700 1 --2- 172.21.15.62:0/3244515701 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f294c102470 0x7f294c1981f0 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:12:51.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:51.177+0000 7f2954800700 1 --2- 172.21.15.62:0/3244515701 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f294c108470 0x7f294c198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:12:51.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:51.177+0000 7f2954800700 1 -- 172.21.15.62:0/3244515701 >> 172.21.15.62:0/3244515701 conn(0x7f294c0fdf90 msgr2=0x7f294c1002c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:12:51.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:51.177+0000 7f2954800700 1 -- 172.21.15.62:0/3244515701 shutdown_connections 2024-04-20T22:12:51.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:51.177+0000 7f2954800700 1 -- 172.21.15.62:0/3244515701 wait complete. 2024-04-20T22:12:51.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:51.330+0000 7f2955000700 1 -- 172.21.15.62:0/2591153359 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f29500ff3c0 msgr2=0x7f29500ff790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:12:51.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:51.330+0000 7f2955000700 1 --2- 172.21.15.62:0/2591153359 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f29500ff3c0 0x7f29500ff790 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f2940009910 tx=0x7f2940009c20 comp rx=0 tx=0).stop 2024-04-20T22:12:51.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:51.330+0000 7f2955000700 1 -- 172.21.15.62:0/2591153359 shutdown_connections 2024-04-20T22:12:51.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:51.330+0000 7f2955000700 1 --2- 172.21.15.62:0/2591153359 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f29500ffd60 0x7f2950103d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:12:51.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:51.330+0000 7f2955000700 1 --2- 172.21.15.62:0/2591153359 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f29500ff3c0 0x7f29500ff790 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:12:51.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:51.330+0000 7f2955000700 1 -- 172.21.15.62:0/2591153359 >> 172.21.15.62:0/2591153359 conn(0x7f29500fac80 msgr2=0x7f29500fd090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:12:51.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:51.330+0000 7f2955000700 1 -- 172.21.15.62:0/2591153359 shutdown_connections 2024-04-20T22:12:51.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:51.330+0000 7f2955000700 1 -- 172.21.15.62:0/2591153359 wait complete. 2024-04-20T22:12:51.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:51.331+0000 7f2955000700 1 Processor -- start 2024-04-20T22:12:51.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:51.332+0000 7f2955000700 1 -- start start 2024-04-20T22:12:51.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:51.332+0000 7f2955000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f29500ff3c0 0x7f29501981a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:12:51.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:51.332+0000 7f2955000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f29500ffd60 0x7f29501986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:12:51.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:51.332+0000 7f2955000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2950198dc0 con 0x7f29500ff3c0 2024-04-20T22:12:51.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:51.332+0000 7f2955000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f295019cb50 con 0x7f29500ffd60 2024-04-20T22:12:51.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:51.332+0000 7f294de00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f29500ff3c0 0x7f29501981a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:12:51.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:51.332+0000 7f294de00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f29500ff3c0 0x7f29501981a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43250/0 (socket says 172.21.15.62:43250) 2024-04-20T22:12:51.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:51.332+0000 7f294de00700 1 -- 172.21.15.62:0/847612540 learned_addr learned my addr 172.21.15.62:0/847612540 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:12:51.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:51.333+0000 7f294d400700 1 --2- 172.21.15.62:0/847612540 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f29500ffd60 0x7f29501986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:12:51.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:51.333+0000 7f294de00700 1 -- 172.21.15.62:0/847612540 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f29500ffd60 msgr2=0x7f29501986e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:12:51.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:51.333+0000 7f294de00700 1 --2- 172.21.15.62:0/847612540 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f29500ffd60 0x7f29501986e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:12:51.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:51.333+0000 7f294de00700 1 -- 172.21.15.62:0/847612540 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29400095f0 con 0x7f29500ff3c0 2024-04-20T22:12:51.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:51.333+0000 7f294de00700 1 --2- 172.21.15.62:0/847612540 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f29500ff3c0 0x7f29501981a0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f294000b5c0 tx=0x7f294000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:12:51.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:51.333+0000 7f2946a00700 1 -- 172.21.15.62:0/847612540 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f294001d070 con 0x7f29500ff3c0 2024-04-20T22:12:51.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:51.334+0000 7f2946a00700 1 -- 172.21.15.62:0/847612540 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2940005e00 con 0x7f29500ff3c0 2024-04-20T22:12:51.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:51.334+0000 7f2946a00700 1 -- 172.21.15.62:0/847612540 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2940017400 con 0x7f29500ff3c0 2024-04-20T22:12:51.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:51.334+0000 7f2955000700 1 -- 172.21.15.62:0/847612540 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f295019cdd0 con 0x7f29500ff3c0 2024-04-20T22:12:51.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:51.334+0000 7f2955000700 1 -- 172.21.15.62:0/847612540 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f295019d270 con 0x7f29500ff3c0 2024-04-20T22:12:51.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:51.335+0000 7f2955000700 1 -- 172.21.15.62:0/847612540 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f295004f2a0 con 0x7f29500ff3c0 2024-04-20T22:12:51.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:51.336+0000 7f2946a00700 1 -- 172.21.15.62:0/847612540 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f294000f460 con 0x7f29500ff3c0 2024-04-20T22:12:51.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:51.337+0000 7f2946a00700 1 --2- 172.21.15.62:0/847612540 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f293406ecc0 0x7f2934071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:12:51.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:51.337+0000 7f2946a00700 1 -- 172.21.15.62:0/847612540 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f2940090480 con 0x7f29500ff3c0 2024-04-20T22:12:51.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:51.337+0000 7f294d400700 1 --2- 172.21.15.62:0/847612540 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f293406ecc0 0x7f2934071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:12:51.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:51.338+0000 7f294d400700 1 --2- 172.21.15.62:0/847612540 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f293406ecc0 0x7f2934071170 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f293c005950 tx=0x7f293c0058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:12:51.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:51.342+0000 7f2946a00700 1 -- 172.21.15.62:0/847612540 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f2940059190 con 0x7f29500ff3c0 2024-04-20T22:12:51.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:12:51 smithi189 ceph-mon[57022]: from='client.35992 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:12:51.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:12:51 smithi189 ceph-mon[57022]: pgmap v1127: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:12:51.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:12:51 smithi189 ceph-mon[57022]: from='client.35996 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:12:51.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:12:51 smithi062 ceph-mon[76322]: from='client.35992 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:12:51.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:12:51 smithi062 ceph-mon[76322]: pgmap v1127: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:12:51.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:12:51 smithi062 ceph-mon[76322]: from='client.35996 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:12:51.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:51.619+0000 7f2955000700 1 -- 172.21.15.62:0/847612540 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f295019d580 con 0x7f29500ff3c0 2024-04-20T22:12:51.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:51.620+0000 7f2946a00700 1 -- 172.21.15.62:0/847612540 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 27 v27) v1 ==== 76+0+1888 (secure 0 0 0) 0x7f294005c850 con 0x7f29500ff3c0 2024-04-20T22:12:51.620 INFO:teuthology.orchestra.run.smithi062.stdout:e27 2024-04-20T22:12:51.620 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T22:12:51.620 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:12:51.620 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T22:12:51.620 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:12:51.620 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T22:12:51.620 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T22:12:51.620 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 20 2024-04-20T22:12:51.620 INFO:teuthology.orchestra.run.smithi062.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-20T22:12:51.621 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T22:12:51.621 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T21:42:01.559061+0000 2024-04-20T22:12:51.621 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T22:12:51.621 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T22:12:51.621 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T22:12:51.621 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T22:12:51.621 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T22:12:51.621 INFO:teuthology.orchestra.run.smithi062.stdout:max_xattr_size 65536 2024-04-20T22:12:51.621 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T22:12:51.621 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T22:12:51.621 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 81 2024-04-20T22:12:51.621 INFO:teuthology.orchestra.run.smithi062.stdout:snap_rstat 1 2024-04-20T22:12:51.621 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:12:51.622 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 2 2024-04-20T22:12:51.622 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1 2024-04-20T22:12:51.622 INFO:teuthology.orchestra.run.smithi062.stdout:up {} 2024-04-20T22:12:51.622 INFO:teuthology.orchestra.run.smithi062.stdout:failed 0,1 2024-04-20T22:12:51.622 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T22:12:51.622 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T22:12:51.622 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T22:12:51.622 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T22:12:51.622 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data enabled 2024-04-20T22:12:51.622 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T22:12:51.622 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T22:12:51.622 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 1 2024-04-20T22:12:51.622 INFO:teuthology.orchestra.run.smithi062.stdout:qdb_cluster leader: 0 members: 2024-04-20T22:12:51.622 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:12:51.623 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:12:51.623 INFO:teuthology.orchestra.run.smithi062.stdout:Standby daemons: 2024-04-20T22:12:51.623 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:12:51.623 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ryqblb{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6826/4113916205,v1:172.21.15.62:6827/4113916205] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:12:51.623 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ozizwi{-1:34476} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6828/3801975165,v1:172.21.15.62:6829/3801975165] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:12:51.623 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.xbgspw{-1:44285} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6826/3996040526,v1:172.21.15.189:6827/3996040526] compat {c=[1],r=[1],i=[7ff]}] 2024-04-20T22:12:51.623 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.dvimer{-1:44727} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6824/3576699232,v1:172.21.15.189:6825/3576699232] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:12:51.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:51.622+0000 7f2955000700 1 -- 172.21.15.62:0/847612540 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f293406ecc0 msgr2=0x7f2934071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:12:51.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:51.622+0000 7f2955000700 1 --2- 172.21.15.62:0/847612540 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f293406ecc0 0x7f2934071170 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f293c005950 tx=0x7f293c0058e0 comp rx=0 tx=0).stop 2024-04-20T22:12:51.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:51.622+0000 7f2955000700 1 -- 172.21.15.62:0/847612540 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f29500ff3c0 msgr2=0x7f29501981a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:12:51.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:51.622+0000 7f2955000700 1 --2- 172.21.15.62:0/847612540 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f29500ff3c0 0x7f29501981a0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f294000b5c0 tx=0x7f294000be20 comp rx=0 tx=0).stop 2024-04-20T22:12:51.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:51.623+0000 7f2955000700 1 -- 172.21.15.62:0/847612540 shutdown_connections 2024-04-20T22:12:51.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:51.623+0000 7f2955000700 1 --2- 172.21.15.62:0/847612540 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f293406ecc0 0x7f2934071170 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:12:51.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:51.623+0000 7f2955000700 1 --2- 172.21.15.62:0/847612540 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f29500ff3c0 0x7f29501981a0 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:12:51.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:51.623+0000 7f2955000700 1 --2- 172.21.15.62:0/847612540 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f29500ffd60 0x7f29501986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:12:51.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:51.623+0000 7f2955000700 1 -- 172.21.15.62:0/847612540 >> 172.21.15.62:0/847612540 conn(0x7f29500fac80 msgr2=0x7f2950068e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:12:51.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:51.623+0000 7f2955000700 1 -- 172.21.15.62:0/847612540 shutdown_connections 2024-04-20T22:12:51.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:51.623+0000 7f2955000700 1 -- 172.21.15.62:0/847612540 wait complete. 2024-04-20T22:12:51.625 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 27 2024-04-20T22:12:51.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:51.779+0000 7f4877e00700 1 -- 172.21.15.62:0/315262499 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f487810c660 msgr2=0x7f487810ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:12:51.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:51.779+0000 7f4877e00700 1 --2- 172.21.15.62:0/315262499 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f487810c660 0x7f487810ca30 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f4868009910 tx=0x7f4868009c20 comp rx=0 tx=0).stop 2024-04-20T22:12:51.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:51.779+0000 7f4877e00700 1 -- 172.21.15.62:0/315262499 shutdown_connections 2024-04-20T22:12:51.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:51.779+0000 7f4877e00700 1 --2- 172.21.15.62:0/315262499 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4878106b80 0x7f4878106ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:12:51.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:51.779+0000 7f4877e00700 1 --2- 172.21.15.62:0/315262499 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f487810c660 0x7f487810ca30 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:12:51.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:51.779+0000 7f4877e00700 1 -- 172.21.15.62:0/315262499 >> 172.21.15.62:0/315262499 conn(0x7f4878075e90 msgr2=0x7f48780782a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:12:51.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:51.779+0000 7f4877e00700 1 -- 172.21.15.62:0/315262499 shutdown_connections 2024-04-20T22:12:51.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:51.779+0000 7f4877e00700 1 -- 172.21.15.62:0/315262499 wait complete. 2024-04-20T22:12:51.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:51.780+0000 7f4877e00700 1 Processor -- start 2024-04-20T22:12:51.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:51.780+0000 7f4877e00700 1 -- start start 2024-04-20T22:12:51.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:51.781+0000 7f4877e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4878106b80 0x7f487819c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:12:51.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:51.781+0000 7f4877e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f487810c660 0x7f487819c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:12:51.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:51.781+0000 7f4877e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f487819cea0 con 0x7f4878106b80 2024-04-20T22:12:51.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:51.781+0000 7f4877e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48781a0c30 con 0x7f487810c660 2024-04-20T22:12:51.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:51.781+0000 7f4876a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4878106b80 0x7f487819c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:12:51.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:51.781+0000 7f4876a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4878106b80 0x7f487819c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43266/0 (socket says 172.21.15.62:43266) 2024-04-20T22:12:51.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:51.781+0000 7f4876a00700 1 -- 172.21.15.62:0/4269306106 learned_addr learned my addr 172.21.15.62:0/4269306106 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:12:51.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:51.782+0000 7f4876000700 1 --2- 172.21.15.62:0/4269306106 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f487810c660 0x7f487819c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:12:51.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:51.782+0000 7f4876a00700 1 -- 172.21.15.62:0/4269306106 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f487810c660 msgr2=0x7f487819c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:12:51.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:51.782+0000 7f4876a00700 1 --2- 172.21.15.62:0/4269306106 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f487810c660 0x7f487819c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:12:51.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:51.782+0000 7f4876a00700 1 -- 172.21.15.62:0/4269306106 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48680095f0 con 0x7f4878106b80 2024-04-20T22:12:51.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:51.782+0000 7f4876a00700 1 --2- 172.21.15.62:0/4269306106 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4878106b80 0x7f487819c280 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f48680092e0 tx=0x7f486800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:12:51.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:51.782+0000 7f486f400700 1 -- 172.21.15.62:0/4269306106 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f486801d070 con 0x7f4878106b80 2024-04-20T22:12:51.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:51.783+0000 7f486f400700 1 -- 172.21.15.62:0/4269306106 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4868005e00 con 0x7f4878106b80 2024-04-20T22:12:51.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:51.783+0000 7f4877e00700 1 -- 172.21.15.62:0/4269306106 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48781a0eb0 con 0x7f4878106b80 2024-04-20T22:12:51.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:51.783+0000 7f486f400700 1 -- 172.21.15.62:0/4269306106 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4868017400 con 0x7f4878106b80 2024-04-20T22:12:51.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:51.783+0000 7f4877e00700 1 -- 172.21.15.62:0/4269306106 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48781a13a0 con 0x7f4878106b80 2024-04-20T22:12:51.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:51.784+0000 7f4877e00700 1 -- 172.21.15.62:0/4269306106 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f487810eb60 con 0x7f4878106b80 2024-04-20T22:12:51.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:51.785+0000 7f486f400700 1 -- 172.21.15.62:0/4269306106 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f486800f460 con 0x7f4878106b80 2024-04-20T22:12:51.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:51.786+0000 7f486f400700 1 --2- 172.21.15.62:0/4269306106 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f485c06ecc0 0x7f485c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:12:51.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:51.786+0000 7f486f400700 1 -- 172.21.15.62:0/4269306106 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f4868090430 con 0x7f4878106b80 2024-04-20T22:12:51.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:51.786+0000 7f4876000700 1 --2- 172.21.15.62:0/4269306106 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f485c06ecc0 0x7f485c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:12:51.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:51.787+0000 7f4876000700 1 --2- 172.21.15.62:0/4269306106 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f485c06ecc0 0x7f485c071170 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f4864005950 tx=0x7f48640058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:12:51.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:51.791+0000 7f486f400700 1 -- 172.21.15.62:0/4269306106 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f4868059140 con 0x7f4878106b80 2024-04-20T22:12:52.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:52.054+0000 7f4877e00700 1 -- 172.21.15.62:0/4269306106 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f48781a15c0 con 0x7f485c06ecc0 2024-04-20T22:12:52.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:52.056+0000 7f486f400700 1 -- 172.21.15.62:0/4269306106 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f48781a15c0 con 0x7f485c06ecc0 2024-04-20T22:12:52.057 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:12:52.057 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T22:12:52.057 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T22:12:52.057 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-20T22:12:52.057 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T22:12:52.057 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr", 2024-04-20T22:12:52.057 INFO:teuthology.orchestra.run.smithi062.stdout: "crash", 2024-04-20T22:12:52.057 INFO:teuthology.orchestra.run.smithi062.stdout: "mon", 2024-04-20T22:12:52.057 INFO:teuthology.orchestra.run.smithi062.stdout: "osd" 2024-04-20T22:12:52.058 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T22:12:52.058 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "15/23 daemons upgraded", 2024-04-20T22:12:52.058 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading mds daemons", 2024-04-20T22:12:52.058 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T22:12:52.058 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:12:52.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:52.060+0000 7f4877e00700 1 -- 172.21.15.62:0/4269306106 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f485c06ecc0 msgr2=0x7f485c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:12:52.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:52.060+0000 7f4877e00700 1 --2- 172.21.15.62:0/4269306106 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f485c06ecc0 0x7f485c071170 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f4864005950 tx=0x7f48640058e0 comp rx=0 tx=0).stop 2024-04-20T22:12:52.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:52.060+0000 7f4877e00700 1 -- 172.21.15.62:0/4269306106 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4878106b80 msgr2=0x7f487819c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:12:52.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:52.060+0000 7f4877e00700 1 --2- 172.21.15.62:0/4269306106 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4878106b80 0x7f487819c280 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f48680092e0 tx=0x7f486800be20 comp rx=0 tx=0).stop 2024-04-20T22:12:52.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:52.060+0000 7f4877e00700 1 -- 172.21.15.62:0/4269306106 shutdown_connections 2024-04-20T22:12:52.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:52.060+0000 7f4877e00700 1 --2- 172.21.15.62:0/4269306106 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f485c06ecc0 0x7f485c071170 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:12:52.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:52.060+0000 7f4877e00700 1 --2- 172.21.15.62:0/4269306106 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4878106b80 0x7f487819c280 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:12:52.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:52.060+0000 7f4877e00700 1 --2- 172.21.15.62:0/4269306106 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f487810c660 0x7f487819c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:12:52.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:52.060+0000 7f4877e00700 1 -- 172.21.15.62:0/4269306106 >> 172.21.15.62:0/4269306106 conn(0x7f4878075e90 msgr2=0x7f4878077980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:12:52.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:52.060+0000 7f4877e00700 1 -- 172.21.15.62:0/4269306106 shutdown_connections 2024-04-20T22:12:52.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:52.060+0000 7f4877e00700 1 -- 172.21.15.62:0/4269306106 wait complete. 2024-04-20T22:12:52.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:52.220+0000 7facb1600700 1 -- 172.21.15.62:0/1769350247 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7facac0ffb20 msgr2=0x7facac104000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:12:52.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:52.220+0000 7facb1600700 1 --2- 172.21.15.62:0/1769350247 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7facac0ffb20 0x7facac104000 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7fac94009910 tx=0x7fac94009c20 comp rx=0 tx=0).stop 2024-04-20T22:12:52.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:52.221+0000 7facb1600700 1 -- 172.21.15.62:0/1769350247 shutdown_connections 2024-04-20T22:12:52.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:52.221+0000 7facb1600700 1 --2- 172.21.15.62:0/1769350247 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7facac0ffb20 0x7facac104000 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:12:52.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:52.221+0000 7facb1600700 1 --2- 172.21.15.62:0/1769350247 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7facac0ff210 0x7facac0ff5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:12:52.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:52.221+0000 7facb1600700 1 -- 172.21.15.62:0/1769350247 >> 172.21.15.62:0/1769350247 conn(0x7facac0fac80 msgr2=0x7facac0fd090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:12:52.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:52.221+0000 7facb1600700 1 -- 172.21.15.62:0/1769350247 shutdown_connections 2024-04-20T22:12:52.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:52.221+0000 7facb1600700 1 -- 172.21.15.62:0/1769350247 wait complete. 2024-04-20T22:12:52.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:52.222+0000 7facb1600700 1 Processor -- start 2024-04-20T22:12:52.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:52.222+0000 7facb1600700 1 -- start start 2024-04-20T22:12:52.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:52.223+0000 7facb1600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7facac0ff210 0x7facac19c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:12:52.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:52.223+0000 7facb1600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7facac0ffb20 0x7facac19ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:12:52.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:52.223+0000 7facb1600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facac19d110 con 0x7facac0ff210 2024-04-20T22:12:52.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:52.223+0000 7facb1600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facac1a0ea0 con 0x7facac0ffb20 2024-04-20T22:12:52.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:52.223+0000 7facaaa00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7facac0ff210 0x7facac19c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:12:52.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:52.223+0000 7facaaa00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7facac0ff210 0x7facac19c4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43280/0 (socket says 172.21.15.62:43280) 2024-04-20T22:12:52.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:52.223+0000 7facaaa00700 1 -- 172.21.15.62:0/3612647350 learned_addr learned my addr 172.21.15.62:0/3612647350 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:12:52.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:52.223+0000 7faca2000700 1 --2- 172.21.15.62:0/3612647350 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7facac0ffb20 0x7facac19ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:12:52.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:52.223+0000 7facaaa00700 1 -- 172.21.15.62:0/3612647350 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7facac0ffb20 msgr2=0x7facac19ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:12:52.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:52.223+0000 7facaaa00700 1 --2- 172.21.15.62:0/3612647350 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7facac0ffb20 0x7facac19ca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:12:52.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:52.223+0000 7facaaa00700 1 -- 172.21.15.62:0/3612647350 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac940095f0 con 0x7facac0ff210 2024-04-20T22:12:52.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:52.224+0000 7facaaa00700 1 --2- 172.21.15.62:0/3612647350 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7facac0ff210 0x7facac19c4f0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7fac9c0060b0 tx=0x7fac9c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:12:52.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:52.224+0000 7faca3e00700 1 -- 172.21.15.62:0/3612647350 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fac9c00dda0 con 0x7facac0ff210 2024-04-20T22:12:52.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:52.224+0000 7faca3e00700 1 -- 172.21.15.62:0/3612647350 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fac9c00f040 con 0x7facac0ff210 2024-04-20T22:12:52.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:52.224+0000 7faca3e00700 1 -- 172.21.15.62:0/3612647350 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fac9c014790 con 0x7facac0ff210 2024-04-20T22:12:52.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:52.224+0000 7facb1600700 1 -- 172.21.15.62:0/3612647350 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facac1a1180 con 0x7facac0ff210 2024-04-20T22:12:52.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:52.224+0000 7facb1600700 1 -- 172.21.15.62:0/3612647350 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facac1a16d0 con 0x7facac0ff210 2024-04-20T22:12:52.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:52.226+0000 7facb1600700 1 -- 172.21.15.62:0/3612647350 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facac101690 con 0x7facac0ff210 2024-04-20T22:12:52.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:52.226+0000 7faca3e00700 1 -- 172.21.15.62:0/3612647350 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fac9c0148f0 con 0x7facac0ff210 2024-04-20T22:12:52.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:52.227+0000 7faca3e00700 1 --2- 172.21.15.62:0/3612647350 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fac9006ecc0 0x7fac90071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:12:52.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:52.228+0000 7faca3e00700 1 -- 172.21.15.62:0/3612647350 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fac9c08e8d0 con 0x7facac0ff210 2024-04-20T22:12:52.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:52.228+0000 7faca2000700 1 --2- 172.21.15.62:0/3612647350 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fac9006ecc0 0x7fac90071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:12:52.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:52.228+0000 7faca2000700 1 --2- 172.21.15.62:0/3612647350 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fac9006ecc0 0x7fac90071170 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7facac19db10 tx=0x7fac9401a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:12:52.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:52.232+0000 7faca3e00700 1 -- 172.21.15.62:0/3612647350 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fac9c057710 con 0x7facac0ff210 2024-04-20T22:12:52.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:12:52 smithi189 ceph-mon[57022]: from='client.36000 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:12:52.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:12:52 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/3244515701' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:12:52.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:12:52 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/847612540' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:12:52.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:12:52 smithi062 ceph-mon[76322]: from='client.36000 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:12:52.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:12:52 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/3244515701' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:12:52.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:12:52 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/847612540' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:12:52.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:52.553+0000 7facb1600700 1 -- 172.21.15.62:0/3612647350 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7facac1a19a0 con 0x7facac0ff210 2024-04-20T22:12:52.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:52.554+0000 7faca3e00700 1 -- 172.21.15.62:0/3612647350 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7fac9c08ce60 con 0x7facac0ff210 2024-04-20T22:12:52.554 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-20T22:12:52.554 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-20T22:12:52.554 INFO:teuthology.orchestra.run.smithi062.stdout: host smithi062 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T22:12:52.554 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-20T22:12:52.554 INFO:teuthology.orchestra.run.smithi062.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi062 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpq7y7oa_r:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:12:52.554 INFO:teuthology.orchestra.run.smithi062.stdout:Traceback (most recent call last): 2024-04-20T22:12:52.555 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-20T22:12:52.555 INFO:teuthology.orchestra.run.smithi062.stdout: return _run_code(code, main_globals, None, 2024-04-20T22:12:52.555 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-20T22:12:52.555 INFO:teuthology.orchestra.run.smithi062.stdout: exec(code, run_globals) 2024-04-20T22:12:52.555 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5618, in 2024-04-20T22:12:52.555 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5606, in main 2024-04-20T22:12:52.555 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 408, in _infer_config 2024-04-20T22:12:52.555 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 323, in _infer_fsid 2024-04-20T22:12:52.555 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 436, in _infer_image 2024-04-20T22:12:52.555 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 310, in _validate_fsid 2024-04-20T22:12:52.555 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 3359, in command_ceph_volume 2024-04-20T22:12:52.555 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-20T22:12:52.555 INFO:teuthology.orchestra.run.smithi062.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi062 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpq7y7oa_r:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:12:52.555 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-20T22:12:52.555 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is degraded 2024-04-20T22:12:52.556 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T22:12:52.556 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T22:12:52.556 INFO:teuthology.orchestra.run.smithi062.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-20T22:12:52.556 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-20T22:12:52.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:52.556+0000 7facb1600700 1 -- 172.21.15.62:0/3612647350 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fac9006ecc0 msgr2=0x7fac90071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:12:52.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:52.556+0000 7facb1600700 1 --2- 172.21.15.62:0/3612647350 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fac9006ecc0 0x7fac90071170 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7facac19db10 tx=0x7fac9401a040 comp rx=0 tx=0).stop 2024-04-20T22:12:52.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:52.556+0000 7facb1600700 1 -- 172.21.15.62:0/3612647350 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7facac0ff210 msgr2=0x7facac19c4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:12:52.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:52.556+0000 7facb1600700 1 --2- 172.21.15.62:0/3612647350 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7facac0ff210 0x7facac19c4f0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7fac9c0060b0 tx=0x7fac9c00d3b0 comp rx=0 tx=0).stop 2024-04-20T22:12:52.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:52.557+0000 7facb1600700 1 -- 172.21.15.62:0/3612647350 shutdown_connections 2024-04-20T22:12:52.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:52.557+0000 7facb1600700 1 --2- 172.21.15.62:0/3612647350 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fac9006ecc0 0x7fac90071170 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:12:52.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:52.557+0000 7facb1600700 1 --2- 172.21.15.62:0/3612647350 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7facac0ff210 0x7facac19c4f0 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:12:52.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:52.557+0000 7facb1600700 1 --2- 172.21.15.62:0/3612647350 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7facac0ffb20 0x7facac19ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:12:52.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:52.557+0000 7facb1600700 1 -- 172.21.15.62:0/3612647350 >> 172.21.15.62:0/3612647350 conn(0x7facac0fac80 msgr2=0x7facac110680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:12:52.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:52.557+0000 7facb1600700 1 -- 172.21.15.62:0/3612647350 shutdown_connections 2024-04-20T22:12:52.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:12:52.557+0000 7facb1600700 1 -- 172.21.15.62:0/3612647350 wait complete. 2024-04-20T22:12:53.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:12:53 smithi189 ceph-mon[57022]: pgmap v1128: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:12:53.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:12:53 smithi189 ceph-mon[57022]: from='client.36012 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:12:53.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:12:53 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/3612647350' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:12:53.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:12:53 smithi062 ceph-mon[76322]: pgmap v1128: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:12:53.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:12:53 smithi062 ceph-mon[76322]: from='client.36012 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:12:53.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:12:53 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/3612647350' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:12:55.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:12:55 smithi189 ceph-mon[57022]: pgmap v1129: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:12:55.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:12:55 smithi062 ceph-mon[76322]: pgmap v1129: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:12:57.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:12:57 smithi189 ceph-mon[57022]: pgmap v1130: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:12:57.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:12:57 smithi062 ceph-mon[76322]: pgmap v1130: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:12:59.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:12:59 smithi189 ceph-mon[57022]: pgmap v1131: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:12:59.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:12:59 smithi062 ceph-mon[76322]: pgmap v1131: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:13:01.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:13:01 smithi189 ceph-mon[57022]: pgmap v1132: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:13:01.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:13:01 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T22:13:01.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:13:01 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T22:13:01.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:13:01 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:13:01.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:13:01 smithi062 ceph-mon[76322]: pgmap v1132: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:13:01.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:13:01 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T22:13:01.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:13:01 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T22:13:01.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:13:01 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:13:02.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:13:02 smithi189 ceph-mon[57022]: pgmap v1133: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:13:02.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:13:02 smithi189 ceph-mon[57022]: pgmap v1134: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:13:02.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:13:02 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T22:13:02.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:13:02 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:13:02.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:13:02 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:13:02.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:13:02 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:13:02.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:13:02 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:13:02.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:13:02 smithi062 ceph-mon[76322]: pgmap v1133: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:13:02.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:13:02 smithi062 ceph-mon[76322]: pgmap v1134: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:13:02.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:13:02 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T22:13:02.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:13:02 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:13:02.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:13:02 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:13:02.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:13:02 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:13:02.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:13:02 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:13:04.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:13:04 smithi189 ceph-mon[57022]: pgmap v1135: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:13:04.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:13:04 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:13:04.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:13:04 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi189.xbgspw", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-20T22:13:04.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:13:04 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T22:13:04.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:13:04 smithi062 ceph-mon[76322]: pgmap v1135: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:13:04.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:13:04 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:13:04.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:13:04 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi189.xbgspw", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-20T22:13:04.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:13:04 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T22:13:05.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:13:05 smithi189 ceph-mon[57022]: Upgrade: Updating mds.cephfs.smithi189.xbgspw 2024-04-20T22:13:05.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:13:05 smithi189 ceph-mon[57022]: Deploying daemon mds.cephfs.smithi189.xbgspw on smithi189 2024-04-20T22:13:05.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:13:05 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:13:05.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:13:05 smithi062 ceph-mon[76322]: Upgrade: Updating mds.cephfs.smithi189.xbgspw 2024-04-20T22:13:05.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:13:05 smithi062 ceph-mon[76322]: Deploying daemon mds.cephfs.smithi189.xbgspw on smithi189 2024-04-20T22:13:05.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:13:05 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:13:06.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:13:06 smithi189 ceph-mon[57022]: pgmap v1136: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:13:06.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:13:06 smithi062 ceph-mon[76322]: pgmap v1136: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:13:08.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:13:08 smithi189 ceph-mon[57022]: pgmap v1137: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:13:08.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:13:08 smithi189 ceph-mon[57022]: osdmap e85: 6 total, 6 up, 6 in 2024-04-20T22:13:08.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:13:08 smithi189 ceph-mon[57022]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-04-20T22:13:08.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:13:08 smithi062 ceph-mon[76322]: pgmap v1137: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:13:08.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:13:08 smithi062 ceph-mon[76322]: osdmap e85: 6 total, 6 up, 6 in 2024-04-20T22:13:08.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:13:08 smithi062 ceph-mon[76322]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-04-20T22:13:10.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:13:10 smithi062 ceph-mon[76322]: pgmap v1139: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:13:10.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:13:10 smithi189 ceph-mon[57022]: pgmap v1139: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:13:12.505 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:13:12 smithi189 ceph-mon[57022]: pgmap v1140: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:13:12.505 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:13:12 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:13:12.505 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:13:12 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:13:12.505 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:13:12 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T22:13:12.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:13:12 smithi062 ceph-mon[76322]: pgmap v1140: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:13:12.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:13:12 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:13:12.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:13:12 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:13:12.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:13:12 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T22:13:13.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:13:13 smithi062 ceph-mon[76322]: mds.? [v2:172.21.15.189:6826/212905792,v1:172.21.15.189:6827/212905792] up:boot 2024-04-20T22:13:13.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:13:13 smithi062 ceph-mon[76322]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-20T22:13:13.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:13:13 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi189.xbgspw"}]: dispatch 2024-04-20T22:13:13.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:13:13 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:13:13.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:13:13 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:13:13.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:13:13 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:13:13.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:13:13 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:13:13.559 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:13:13 smithi189 ceph-mon[57022]: mds.? [v2:172.21.15.189:6826/212905792,v1:172.21.15.189:6827/212905792] up:boot 2024-04-20T22:13:13.559 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:13:13 smithi189 ceph-mon[57022]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-20T22:13:13.559 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:13:13 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi189.xbgspw"}]: dispatch 2024-04-20T22:13:13.559 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:13:13 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:13:13.559 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:13:13 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:13:13.560 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:13:13 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:13:13.560 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:13:13 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:13:14.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:13:14 smithi062 ceph-mon[76322]: pgmap v1141: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:13:14.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:13:14 smithi189 ceph-mon[57022]: pgmap v1141: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:13:16.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:13:16 smithi189 ceph-mon[57022]: pgmap v1142: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:13:16.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:13:16 smithi062 ceph-mon[76322]: pgmap v1142: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:13:18.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:13:18 smithi062 ceph-mon[76322]: pgmap v1143: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:13:18.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:13:18 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:13:18.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:13:18 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:13:18.705 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:13:18 smithi189 ceph-mon[57022]: pgmap v1143: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:13:18.705 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:13:18 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:13:18.705 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:13:18 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:13:20.522 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:13:20 smithi189 ceph-mon[57022]: pgmap v1144: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:13:20.522 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:13:20 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:13:20.522 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:13:20 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:13:20.522 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:13:20 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:13:20.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:13:20 smithi062 ceph-mon[76322]: pgmap v1144: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:13:20.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:13:20 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:13:20.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:13:20 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:13:20.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:13:20 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:13:21.849 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:13:21 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:13:21.849 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:13:21 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:13:21.849 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:13:21 smithi189 ceph-mon[57022]: pgmap v1145: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:13:22.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:13:21 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:13:22.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:13:21 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:13:22.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:13:21 smithi062 ceph-mon[76322]: pgmap v1145: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:13:22.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:22.739+0000 7eff16000700 1 -- 172.21.15.62:0/1039245109 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff10069140 msgr2=0x7eff10104eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:13:22.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:22.739+0000 7eff16000700 1 --2- 172.21.15.62:0/1039245109 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff10069140 0x7eff10104eb0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7efef8009910 tx=0x7efef8009c20 comp rx=0 tx=0).stop 2024-04-20T22:13:22.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:22.739+0000 7eff16000700 1 -- 172.21.15.62:0/1039245109 shutdown_connections 2024-04-20T22:13:22.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:22.739+0000 7eff16000700 1 --2- 172.21.15.62:0/1039245109 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff10069140 0x7eff10104eb0 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:13:22.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:22.739+0000 7eff16000700 1 --2- 172.21.15.62:0/1039245109 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff100687a0 0x7eff10068b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:13:22.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:22.739+0000 7eff16000700 1 -- 172.21.15.62:0/1039245109 >> 172.21.15.62:0/1039245109 conn(0x7eff10075130 msgr2=0x7eff10075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:13:22.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:22.739+0000 7eff16000700 1 -- 172.21.15.62:0/1039245109 shutdown_connections 2024-04-20T22:13:22.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:22.739+0000 7eff16000700 1 -- 172.21.15.62:0/1039245109 wait complete. 2024-04-20T22:13:22.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:22.740+0000 7eff16000700 1 Processor -- start 2024-04-20T22:13:22.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:22.740+0000 7eff16000700 1 -- start start 2024-04-20T22:13:22.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:22.741+0000 7eff16000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff100687a0 0x7eff10198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:13:22.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:22.741+0000 7eff16000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff10069140 0x7eff101985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:13:22.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:22.741+0000 7eff16000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff10198c80 con 0x7eff100687a0 2024-04-20T22:13:22.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:22.741+0000 7eff16000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff1019ca10 con 0x7eff10069140 2024-04-20T22:13:22.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:22.741+0000 7eff0f400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff100687a0 0x7eff10198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:13:22.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:22.741+0000 7eff0f400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff100687a0 0x7eff10198060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41532/0 (socket says 172.21.15.62:41532) 2024-04-20T22:13:22.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:22.741+0000 7eff0f400700 1 -- 172.21.15.62:0/4014729712 learned_addr learned my addr 172.21.15.62:0/4014729712 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:13:22.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:22.741+0000 7eff07e00700 1 --2- 172.21.15.62:0/4014729712 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff10069140 0x7eff101985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:13:22.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:22.742+0000 7eff0f400700 1 -- 172.21.15.62:0/4014729712 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff10069140 msgr2=0x7eff101985a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:13:22.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:22.742+0000 7eff0f400700 1 --2- 172.21.15.62:0/4014729712 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff10069140 0x7eff101985a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:13:22.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:22.742+0000 7eff0f400700 1 -- 172.21.15.62:0/4014729712 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efef80095f0 con 0x7eff100687a0 2024-04-20T22:13:22.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:22.742+0000 7eff0f400700 1 --2- 172.21.15.62:0/4014729712 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff100687a0 0x7eff10198060 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7eff0000c8f0 tx=0x7eff0000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:13:22.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:22.742+0000 7eff0cc00700 1 -- 172.21.15.62:0/4014729712 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7eff00007a70 con 0x7eff100687a0 2024-04-20T22:13:22.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:22.743+0000 7eff16000700 1 -- 172.21.15.62:0/4014729712 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff1019ccf0 con 0x7eff100687a0 2024-04-20T22:13:22.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:22.743+0000 7eff0cc00700 1 -- 172.21.15.62:0/4014729712 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7eff00007bd0 con 0x7eff100687a0 2024-04-20T22:13:22.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:22.743+0000 7eff0cc00700 1 -- 172.21.15.62:0/4014729712 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7eff00018630 con 0x7eff100687a0 2024-04-20T22:13:22.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:22.743+0000 7eff16000700 1 -- 172.21.15.62:0/4014729712 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff1019d240 con 0x7eff100687a0 2024-04-20T22:13:22.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:22.744+0000 7eff16000700 1 -- 172.21.15.62:0/4014729712 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff1004f2a0 con 0x7eff100687a0 2024-04-20T22:13:22.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:22.745+0000 7eff0cc00700 1 -- 172.21.15.62:0/4014729712 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7eff0000f450 con 0x7eff100687a0 2024-04-20T22:13:22.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:22.746+0000 7eff0cc00700 1 --2- 172.21.15.62:0/4014729712 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7efef406ecc0 0x7efef4071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:13:22.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:22.746+0000 7eff0cc00700 1 -- 172.21.15.62:0/4014729712 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7eff0008ee00 con 0x7eff100687a0 2024-04-20T22:13:22.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:22.747+0000 7eff07e00700 1 --2- 172.21.15.62:0/4014729712 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7efef406ecc0 0x7efef4071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:13:22.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:22.748+0000 7eff07e00700 1 --2- 172.21.15.62:0/4014729712 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7efef406ecc0 0x7efef4071170 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7eff10199680 tx=0x7efef801a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:13:22.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:22.753+0000 7eff0cc00700 1 -- 172.21.15.62:0/4014729712 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7eff00057c60 con 0x7eff100687a0 2024-04-20T22:13:23.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.015+0000 7eff16000700 1 -- 172.21.15.62:0/4014729712 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7eff1019d590 con 0x7efef406ecc0 2024-04-20T22:13:23.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.017+0000 7eff0cc00700 1 -- 172.21.15.62:0/4014729712 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7eff1019d590 con 0x7efef406ecc0 2024-04-20T22:13:23.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.019+0000 7eff16000700 1 -- 172.21.15.62:0/4014729712 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7efef406ecc0 msgr2=0x7efef4071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:13:23.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.019+0000 7eff16000700 1 --2- 172.21.15.62:0/4014729712 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7efef406ecc0 0x7efef4071170 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7eff10199680 tx=0x7efef801a040 comp rx=0 tx=0).stop 2024-04-20T22:13:23.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.019+0000 7eff16000700 1 -- 172.21.15.62:0/4014729712 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff100687a0 msgr2=0x7eff10198060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:13:23.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.019+0000 7eff16000700 1 --2- 172.21.15.62:0/4014729712 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff100687a0 0x7eff10198060 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7eff0000c8f0 tx=0x7eff0000ccb0 comp rx=0 tx=0).stop 2024-04-20T22:13:23.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.020+0000 7eff16000700 1 -- 172.21.15.62:0/4014729712 shutdown_connections 2024-04-20T22:13:23.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.020+0000 7eff16000700 1 --2- 172.21.15.62:0/4014729712 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7efef406ecc0 0x7efef4071170 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:13:23.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.020+0000 7eff16000700 1 --2- 172.21.15.62:0/4014729712 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff100687a0 0x7eff10198060 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:13:23.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.020+0000 7eff16000700 1 --2- 172.21.15.62:0/4014729712 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff10069140 0x7eff101985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:13:23.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.020+0000 7eff16000700 1 -- 172.21.15.62:0/4014729712 >> 172.21.15.62:0/4014729712 conn(0x7eff10075130 msgr2=0x7eff100fe6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:13:23.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.020+0000 7eff16000700 1 -- 172.21.15.62:0/4014729712 shutdown_connections 2024-04-20T22:13:23.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.020+0000 7eff16000700 1 -- 172.21.15.62:0/4014729712 wait complete. 2024-04-20T22:13:23.037 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T22:13:23.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.174+0000 7f284ca00700 1 -- 172.21.15.62:0/912954114 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2848073090 msgr2=0x7f2848073460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:13:23.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.174+0000 7f284ca00700 1 --2- 172.21.15.62:0/912954114 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2848073090 0x7f2848073460 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f2838009940 tx=0x7f2838009c50 comp rx=0 tx=0).stop 2024-04-20T22:13:23.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.174+0000 7f284ca00700 1 -- 172.21.15.62:0/912954114 shutdown_connections 2024-04-20T22:13:23.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.174+0000 7f284ca00700 1 --2- 172.21.15.62:0/912954114 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f28480739a0 0x7f284810c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:13:23.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.174+0000 7f284ca00700 1 --2- 172.21.15.62:0/912954114 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2848073090 0x7f2848073460 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:13:23.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.174+0000 7f284ca00700 1 -- 172.21.15.62:0/912954114 >> 172.21.15.62:0/912954114 conn(0x7f28480fbda0 msgr2=0x7f28480fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:13:23.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.174+0000 7f284ca00700 1 -- 172.21.15.62:0/912954114 shutdown_connections 2024-04-20T22:13:23.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.174+0000 7f284ca00700 1 -- 172.21.15.62:0/912954114 wait complete. 2024-04-20T22:13:23.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.175+0000 7f284ca00700 1 Processor -- start 2024-04-20T22:13:23.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.175+0000 7f284ca00700 1 -- start start 2024-04-20T22:13:23.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.176+0000 7f284ca00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2848073090 0x7f2848198180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:13:23.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.176+0000 7f284ca00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f28480739a0 0x7f28481986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:13:23.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.176+0000 7f284ca00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2848198da0 con 0x7f28480739a0 2024-04-20T22:13:23.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.176+0000 7f284ca00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f284819cb30 con 0x7f2848073090 2024-04-20T22:13:23.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.176+0000 7f2845400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f28480739a0 0x7f28481986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:13:23.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.176+0000 7f2845400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f28480739a0 0x7f28481986c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41542/0 (socket says 172.21.15.62:41542) 2024-04-20T22:13:23.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.176+0000 7f2845400700 1 -- 172.21.15.62:0/2735637586 learned_addr learned my addr 172.21.15.62:0/2735637586 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:13:23.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.176+0000 7f2845e00700 1 --2- 172.21.15.62:0/2735637586 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2848073090 0x7f2848198180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:13:23.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.177+0000 7f2845400700 1 -- 172.21.15.62:0/2735637586 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2848073090 msgr2=0x7f2848198180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:13:23.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.177+0000 7f2845400700 1 --2- 172.21.15.62:0/2735637586 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2848073090 0x7f2848198180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:13:23.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.177+0000 7f2845400700 1 -- 172.21.15.62:0/2735637586 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28380095f0 con 0x7f28480739a0 2024-04-20T22:13:23.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.177+0000 7f2845400700 1 --2- 172.21.15.62:0/2735637586 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f28480739a0 0x7f28481986c0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f283400c8a0 tx=0x7f283400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:13:23.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.177+0000 7f283ea00700 1 -- 172.21.15.62:0/2735637586 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f283400cea0 con 0x7f28480739a0 2024-04-20T22:13:23.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.177+0000 7f284ca00700 1 -- 172.21.15.62:0/2735637586 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f284819ce10 con 0x7f28480739a0 2024-04-20T22:13:23.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.177+0000 7f283ea00700 1 -- 172.21.15.62:0/2735637586 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2834004830 con 0x7f28480739a0 2024-04-20T22:13:23.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.177+0000 7f283ea00700 1 -- 172.21.15.62:0/2735637586 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f28340055d0 con 0x7f28480739a0 2024-04-20T22:13:23.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.177+0000 7f284ca00700 1 -- 172.21.15.62:0/2735637586 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f284819d360 con 0x7f28480739a0 2024-04-20T22:13:23.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.179+0000 7f284ca00700 1 -- 172.21.15.62:0/2735637586 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2848109ce0 con 0x7f28480739a0 2024-04-20T22:13:23.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.180+0000 7f283ea00700 1 -- 172.21.15.62:0/2735637586 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f28340049a0 con 0x7f28480739a0 2024-04-20T22:13:23.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.181+0000 7f283ea00700 1 --2- 172.21.15.62:0/2735637586 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f282c06ecc0 0x7f282c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:13:23.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.182+0000 7f283ea00700 1 -- 172.21.15.62:0/2735637586 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f283408ef20 con 0x7f28480739a0 2024-04-20T22:13:23.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.182+0000 7f2845e00700 1 --2- 172.21.15.62:0/2735637586 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f282c06ecc0 0x7f282c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:13:23.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.182+0000 7f2845e00700 1 --2- 172.21.15.62:0/2735637586 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f282c06ecc0 0x7f282c071170 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f2838005cd0 tx=0x7f2838005c00 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:13:23.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.185+0000 7f283ea00700 1 -- 172.21.15.62:0/2735637586 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f2834057c30 con 0x7f28480739a0 2024-04-20T22:13:23.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.449+0000 7f284ca00700 1 -- 172.21.15.62:0/2735637586 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f284819d5e0 con 0x7f282c06ecc0 2024-04-20T22:13:23.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.451+0000 7f283ea00700 1 -- 172.21.15.62:0/2735637586 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f284819d5e0 con 0x7f282c06ecc0 2024-04-20T22:13:23.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.455+0000 7f284ca00700 1 -- 172.21.15.62:0/2735637586 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f282c06ecc0 msgr2=0x7f282c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:13:23.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.455+0000 7f284ca00700 1 --2- 172.21.15.62:0/2735637586 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f282c06ecc0 0x7f282c071170 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f2838005cd0 tx=0x7f2838005c00 comp rx=0 tx=0).stop 2024-04-20T22:13:23.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.455+0000 7f284ca00700 1 -- 172.21.15.62:0/2735637586 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f28480739a0 msgr2=0x7f28481986c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:13:23.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.455+0000 7f284ca00700 1 --2- 172.21.15.62:0/2735637586 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f28480739a0 0x7f28481986c0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f283400c8a0 tx=0x7f283400cc60 comp rx=0 tx=0).stop 2024-04-20T22:13:23.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.455+0000 7f284ca00700 1 -- 172.21.15.62:0/2735637586 shutdown_connections 2024-04-20T22:13:23.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.455+0000 7f284ca00700 1 --2- 172.21.15.62:0/2735637586 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f282c06ecc0 0x7f282c071170 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:13:23.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.455+0000 7f284ca00700 1 --2- 172.21.15.62:0/2735637586 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2848073090 0x7f2848198180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:13:23.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.455+0000 7f284ca00700 1 --2- 172.21.15.62:0/2735637586 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f28480739a0 0x7f28481986c0 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:13:23.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.455+0000 7f284ca00700 1 -- 172.21.15.62:0/2735637586 >> 172.21.15.62:0/2735637586 conn(0x7f28480fbda0 msgr2=0x7f2848106e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:13:23.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.455+0000 7f284ca00700 1 -- 172.21.15.62:0/2735637586 shutdown_connections 2024-04-20T22:13:23.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.456+0000 7f284ca00700 1 -- 172.21.15.62:0/2735637586 wait complete. 2024-04-20T22:13:23.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.623+0000 7f9608c00700 1 -- 172.21.15.62:0/1967372973 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f96040739a0 msgr2=0x7f960410c5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:13:23.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.623+0000 7f9608c00700 1 --2- 172.21.15.62:0/1967372973 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f96040739a0 0x7f960410c5e0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f95f0009940 tx=0x7f95f0009c50 comp rx=0 tx=0).stop 2024-04-20T22:13:23.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.623+0000 7f9608c00700 1 -- 172.21.15.62:0/1967372973 shutdown_connections 2024-04-20T22:13:23.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.623+0000 7f9608c00700 1 --2- 172.21.15.62:0/1967372973 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f96040739a0 0x7f960410c5e0 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:13:23.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.623+0000 7f9608c00700 1 --2- 172.21.15.62:0/1967372973 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9604073090 0x7f9604073460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:13:23.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.623+0000 7f9608c00700 1 -- 172.21.15.62:0/1967372973 >> 172.21.15.62:0/1967372973 conn(0x7f96040fbda0 msgr2=0x7f96040fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:13:23.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.623+0000 7f9608c00700 1 -- 172.21.15.62:0/1967372973 shutdown_connections 2024-04-20T22:13:23.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.623+0000 7f9608c00700 1 -- 172.21.15.62:0/1967372973 wait complete. 2024-04-20T22:13:23.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.624+0000 7f9608c00700 1 Processor -- start 2024-04-20T22:13:23.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.624+0000 7f9608c00700 1 -- start start 2024-04-20T22:13:23.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.625+0000 7f9608c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9604073090 0x7f96041981b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:13:23.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.625+0000 7f9608c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96040739a0 0x7f96041986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:13:23.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.625+0000 7f9608c00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9604198dd0 con 0x7f9604073090 2024-04-20T22:13:23.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.625+0000 7f9608c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f960419cb60 con 0x7f96040739a0 2024-04-20T22:13:23.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.625+0000 7f9601e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9604073090 0x7f96041981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:13:23.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.625+0000 7f9601e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9604073090 0x7f96041981b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41560/0 (socket says 172.21.15.62:41560) 2024-04-20T22:13:23.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.625+0000 7f9601e00700 1 -- 172.21.15.62:0/3356497392 learned_addr learned my addr 172.21.15.62:0/3356497392 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:13:23.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.625+0000 7f9601400700 1 --2- 172.21.15.62:0/3356497392 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96040739a0 0x7f96041986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:13:23.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.626+0000 7f9601e00700 1 -- 172.21.15.62:0/3356497392 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96040739a0 msgr2=0x7f96041986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:13:23.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.626+0000 7f9601e00700 1 --2- 172.21.15.62:0/3356497392 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96040739a0 0x7f96041986f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:13:23.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.626+0000 7f9601e00700 1 -- 172.21.15.62:0/3356497392 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95f00095f0 con 0x7f9604073090 2024-04-20T22:13:23.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.626+0000 7f9601e00700 1 --2- 172.21.15.62:0/3356497392 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9604073090 0x7f96041981b0 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f95f400d8d0 tx=0x7f95f400dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:13:23.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.626+0000 7f95faa00700 1 -- 172.21.15.62:0/3356497392 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f95f40098a0 con 0x7f9604073090 2024-04-20T22:13:23.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.626+0000 7f95faa00700 1 -- 172.21.15.62:0/3356497392 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f95f4010460 con 0x7f9604073090 2024-04-20T22:13:23.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.626+0000 7f9608c00700 1 -- 172.21.15.62:0/3356497392 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f960419ce40 con 0x7f9604073090 2024-04-20T22:13:23.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.626+0000 7f95faa00700 1 -- 172.21.15.62:0/3356497392 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f95f400f5f0 con 0x7f9604073090 2024-04-20T22:13:23.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.626+0000 7f9608c00700 1 -- 172.21.15.62:0/3356497392 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f960419d360 con 0x7f9604073090 2024-04-20T22:13:23.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.628+0000 7f9608c00700 1 -- 172.21.15.62:0/3356497392 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9604109ce0 con 0x7f9604073090 2024-04-20T22:13:23.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.629+0000 7f95faa00700 1 -- 172.21.15.62:0/3356497392 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f95f4009a00 con 0x7f9604073090 2024-04-20T22:13:23.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.630+0000 7f95faa00700 1 --2- 172.21.15.62:0/3356497392 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f95e806ecc0 0x7f95e8071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:13:23.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.630+0000 7f95faa00700 1 -- 172.21.15.62:0/3356497392 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f95f408edd0 con 0x7f9604073090 2024-04-20T22:13:23.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.631+0000 7f9601400700 1 --2- 172.21.15.62:0/3356497392 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f95e806ecc0 0x7f95e8071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:13:23.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.632+0000 7f9601400700 1 --2- 172.21.15.62:0/3356497392 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f95e806ecc0 0x7f95e8071170 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f95f0009910 tx=0x7f95f0005ab0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:13:23.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.636+0000 7f95faa00700 1 -- 172.21.15.62:0/3356497392 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f95f4057ae0 con 0x7f9604073090 2024-04-20T22:13:23.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.912+0000 7f9608c00700 1 -- 172.21.15.62:0/3356497392 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f9604199670 con 0x7f95e806ecc0 2024-04-20T22:13:23.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.948+0000 7f95faa00700 1 -- 172.21.15.62:0/3356497392 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f9604199670 con 0x7f95e806ecc0 2024-04-20T22:13:23.949 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T22:13:23.949 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (39m) 3s ago 53m 14.6M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T22:13:23.949 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (53m) 3s ago 53m 11.6M - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T22:13:23.949 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (52m) 6s ago 52m 11.2M - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T22:13:23.949 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (35m) 3s ago 53m 6660k - 19.0.0-3133-ga8c84f59 76bc6b91b726 2825d5ba11c6 2024-04-20T22:13:23.949 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (35m) 6s ago 51m 6665k - 19.0.0-3133-ga8c84f59 76bc6b91b726 20f746fa1bba 2024-04-20T22:13:23.949 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (38m) 3s ago 52m 75.5M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T22:13:23.949 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (30m) 3s ago 45m 22.9M - 19.0.0-3133-ga8c84f59 76bc6b91b726 de80bc90a683 2024-04-20T22:13:23.949 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (31m) 3s ago 45m 23.0M - 19.0.0-3133-ga8c84f59 76bc6b91b726 cd793b7ee98e 2024-04-20T22:13:23.949 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (15m) 6s ago 45m 18.5M - 19.0.0-3133-ga8c84f59 76bc6b91b726 0e7b0bc4d1ab 2024-04-20T22:13:23.949 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (12s) 6s ago 45m 11.4M - 19.0.0-3133-ga8c84f59 76bc6b91b726 0af732ac9e06 2024-04-20T22:13:23.949 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (42m) 3s ago 54m 517M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T22:13:23.949 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (40m) 6s ago 51m 460M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T22:13:23.950 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (37m) 3s ago 55m 86.3M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 019eb1479485 2024-04-20T22:13:23.950 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (36m) 6s ago 51m 58.0M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 55aac560aab6 2024-04-20T22:13:23.950 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (40m) 3s ago 53m 8766k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T22:13:23.950 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (40m) 6s ago 51m 8711k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T22:13:23.950 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (34m) 3s ago 50m 91.3M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 ee2bb901f26e 2024-04-20T22:13:23.950 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (34m) 3s ago 50m 94.9M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 9075152d4162 2024-04-20T22:13:23.950 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (33m) 3s ago 49m 78.5M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 2d4f3e38dc0e 2024-04-20T22:13:23.950 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (33m) 6s ago 48m 94.1M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 8f97b17ae0eb 2024-04-20T22:13:23.950 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (32m) 6s ago 48m 96.1M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 653d8079967d 2024-04-20T22:13:23.950 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (31m) 6s ago 47m 90.9M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 596dded14c12 2024-04-20T22:13:23.950 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (39m) 3s ago 52m 90.4M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T22:13:23.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.951+0000 7f9608c00700 1 -- 172.21.15.62:0/3356497392 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f95e806ecc0 msgr2=0x7f95e8071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:13:23.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.951+0000 7f9608c00700 1 --2- 172.21.15.62:0/3356497392 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f95e806ecc0 0x7f95e8071170 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f95f0009910 tx=0x7f95f0005ab0 comp rx=0 tx=0).stop 2024-04-20T22:13:23.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.951+0000 7f9608c00700 1 -- 172.21.15.62:0/3356497392 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9604073090 msgr2=0x7f96041981b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:13:23.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.951+0000 7f9608c00700 1 --2- 172.21.15.62:0/3356497392 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9604073090 0x7f96041981b0 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f95f400d8d0 tx=0x7f95f400dbe0 comp rx=0 tx=0).stop 2024-04-20T22:13:23.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.952+0000 7f9608c00700 1 -- 172.21.15.62:0/3356497392 shutdown_connections 2024-04-20T22:13:23.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.952+0000 7f9608c00700 1 --2- 172.21.15.62:0/3356497392 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f95e806ecc0 0x7f95e8071170 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:13:23.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.952+0000 7f9608c00700 1 --2- 172.21.15.62:0/3356497392 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9604073090 0x7f96041981b0 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:13:23.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.952+0000 7f9608c00700 1 --2- 172.21.15.62:0/3356497392 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96040739a0 0x7f96041986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:13:23.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.952+0000 7f9608c00700 1 -- 172.21.15.62:0/3356497392 >> 172.21.15.62:0/3356497392 conn(0x7f96040fbda0 msgr2=0x7f9604106e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:13:23.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.952+0000 7f9608c00700 1 -- 172.21.15.62:0/3356497392 shutdown_connections 2024-04-20T22:13:23.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:23.952+0000 7f9608c00700 1 -- 172.21.15.62:0/3356497392 wait complete. 2024-04-20T22:13:24.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:24.117+0000 7f98b6600700 1 -- 172.21.15.62:0/1920544805 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f98b01004e0 msgr2=0x7f98b01008b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:13:24.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:24.117+0000 7f98b6600700 1 --2- 172.21.15.62:0/1920544805 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f98b01004e0 0x7f98b01008b0 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f98a0009910 tx=0x7f98a0009c20 comp rx=0 tx=0).stop 2024-04-20T22:13:24.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:24.117+0000 7f98b6600700 1 -- 172.21.15.62:0/1920544805 shutdown_connections 2024-04-20T22:13:24.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:24.117+0000 7f98b6600700 1 --2- 172.21.15.62:0/1920544805 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98b0100e80 0x7f98b0104e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:13:24.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:24.117+0000 7f98b6600700 1 --2- 172.21.15.62:0/1920544805 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f98b01004e0 0x7f98b01008b0 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:13:24.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:24.117+0000 7f98b6600700 1 -- 172.21.15.62:0/1920544805 >> 172.21.15.62:0/1920544805 conn(0x7f98b00fbda0 msgr2=0x7f98b00fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:13:24.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:24.117+0000 7f98b6600700 1 -- 172.21.15.62:0/1920544805 shutdown_connections 2024-04-20T22:13:24.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:24.117+0000 7f98b6600700 1 -- 172.21.15.62:0/1920544805 wait complete. 2024-04-20T22:13:24.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:24.118+0000 7f98b6600700 1 Processor -- start 2024-04-20T22:13:24.118 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:13:23 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:13:24.118 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:13:23 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:13:24.119 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:13:23 smithi062 ceph-mon[76322]: from='client.36022 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:13:24.119 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:13:23 smithi062 ceph-mon[76322]: pgmap v1146: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:13:24.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:24.118+0000 7f98b6600700 1 -- start start 2024-04-20T22:13:24.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:24.119+0000 7f98b6600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f98b01004e0 0x7f98b0195fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:13:24.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:24.119+0000 7f98b6600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98b0100e80 0x7f98b01964f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:13:24.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:24.119+0000 7f98b6600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98b0196bd0 con 0x7f98b01004e0 2024-04-20T22:13:24.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:24.119+0000 7f98b6600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98b019a960 con 0x7f98b0100e80 2024-04-20T22:13:24.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:24.119+0000 7f98af400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f98b01004e0 0x7f98b0195fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:13:24.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:24.119+0000 7f98af400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f98b01004e0 0x7f98b0195fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41578/0 (socket says 172.21.15.62:41578) 2024-04-20T22:13:24.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:24.119+0000 7f98af400700 1 -- 172.21.15.62:0/3470267089 learned_addr learned my addr 172.21.15.62:0/3470267089 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:13:24.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:24.119+0000 7f98af400700 1 -- 172.21.15.62:0/3470267089 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98b0100e80 msgr2=0x7f98b01964f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:13:24.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:24.119+0000 7f98a7e00700 1 --2- 172.21.15.62:0/3470267089 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98b0100e80 0x7f98b01964f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:13:24.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:24.119+0000 7f98af400700 1 --2- 172.21.15.62:0/3470267089 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98b0100e80 0x7f98b01964f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:13:24.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:24.119+0000 7f98af400700 1 -- 172.21.15.62:0/3470267089 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98a00095f0 con 0x7f98b01004e0 2024-04-20T22:13:24.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:24.120+0000 7f98a7e00700 1 --2- 172.21.15.62:0/3470267089 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98b0100e80 0x7f98b01964f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T22:13:24.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:24.120+0000 7f98af400700 1 --2- 172.21.15.62:0/3470267089 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f98b01004e0 0x7f98b0195fb0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f98a000b5c0 tx=0x7f98a000be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:13:24.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:24.120+0000 7f98acc00700 1 -- 172.21.15.62:0/3470267089 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f98a001d070 con 0x7f98b01004e0 2024-04-20T22:13:24.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:24.120+0000 7f98acc00700 1 -- 172.21.15.62:0/3470267089 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f98a000f460 con 0x7f98b01004e0 2024-04-20T22:13:24.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:24.120+0000 7f98b6600700 1 -- 172.21.15.62:0/3470267089 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98b019abe0 con 0x7f98b01004e0 2024-04-20T22:13:24.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:24.120+0000 7f98acc00700 1 -- 172.21.15.62:0/3470267089 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f98a0017610 con 0x7f98b01004e0 2024-04-20T22:13:24.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:24.120+0000 7f98b6600700 1 -- 172.21.15.62:0/3470267089 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98b019b0d0 con 0x7f98b01004e0 2024-04-20T22:13:24.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:24.121+0000 7f98b6600700 1 -- 172.21.15.62:0/3470267089 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98b01087f0 con 0x7f98b01004e0 2024-04-20T22:13:24.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:24.122+0000 7f98acc00700 1 -- 172.21.15.62:0/3470267089 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f98a000f5d0 con 0x7f98b01004e0 2024-04-20T22:13:24.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:24.123+0000 7f98acc00700 1 --2- 172.21.15.62:0/3470267089 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f989406ecc0 0x7f9894071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:13:24.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:24.123+0000 7f98acc00700 1 -- 172.21.15.62:0/3470267089 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f98a0090510 con 0x7f98b01004e0 2024-04-20T22:13:24.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:24.124+0000 7f98a7e00700 1 --2- 172.21.15.62:0/3470267089 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f989406ecc0 0x7f9894071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:13:24.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:24.124+0000 7f98a7e00700 1 --2- 172.21.15.62:0/3470267089 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f989406ecc0 0x7f9894071170 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f98b01975d0 tx=0x7f9898008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:13:24.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:24.130+0000 7f98acc00700 1 -- 172.21.15.62:0/3470267089 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f98a0059220 con 0x7f98b01004e0 2024-04-20T22:13:24.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:13:23 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:13:24.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:13:23 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:13:24.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:13:23 smithi189 ceph-mon[57022]: from='client.36022 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:13:24.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:13:23 smithi189 ceph-mon[57022]: pgmap v1146: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:13:24.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:24.472+0000 7f98b6600700 1 -- 172.21.15.62:0/3470267089 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f98b0197410 con 0x7f98b01004e0 2024-04-20T22:13:24.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:24.474+0000 7f98acc00700 1 -- 172.21.15.62:0/3470267089 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f98a005c8e0 con 0x7f98b01004e0 2024-04-20T22:13:24.474 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:13:24.474 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T22:13:24.474 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:13:24.474 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:13:24.474 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T22:13:24.474 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:13:24.474 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:13:24.474 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T22:13:24.474 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 6 2024-04-20T22:13:24.474 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:13:24.475 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T22:13:24.475 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 4 2024-04-20T22:13:24.475 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:13:24.475 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T22:13:24.475 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 14 2024-04-20T22:13:24.475 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T22:13:24.475 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:13:24.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:24.476+0000 7f98b6600700 1 -- 172.21.15.62:0/3470267089 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f989406ecc0 msgr2=0x7f9894071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:13:24.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:24.476+0000 7f98b6600700 1 --2- 172.21.15.62:0/3470267089 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f989406ecc0 0x7f9894071170 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f98b01975d0 tx=0x7f9898008040 comp rx=0 tx=0).stop 2024-04-20T22:13:24.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:24.476+0000 7f98b6600700 1 -- 172.21.15.62:0/3470267089 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f98b01004e0 msgr2=0x7f98b0195fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:13:24.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:24.476+0000 7f98b6600700 1 --2- 172.21.15.62:0/3470267089 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f98b01004e0 0x7f98b0195fb0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f98a000b5c0 tx=0x7f98a000be00 comp rx=0 tx=0).stop 2024-04-20T22:13:24.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:24.477+0000 7f98b6600700 1 -- 172.21.15.62:0/3470267089 shutdown_connections 2024-04-20T22:13:24.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:24.477+0000 7f98b6600700 1 --2- 172.21.15.62:0/3470267089 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f989406ecc0 0x7f9894071170 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:13:24.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:24.477+0000 7f98b6600700 1 --2- 172.21.15.62:0/3470267089 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f98b01004e0 0x7f98b0195fb0 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:13:24.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:24.477+0000 7f98b6600700 1 --2- 172.21.15.62:0/3470267089 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98b0100e80 0x7f98b01964f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:13:24.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:24.477+0000 7f98b6600700 1 -- 172.21.15.62:0/3470267089 >> 172.21.15.62:0/3470267089 conn(0x7f98b00fbda0 msgr2=0x7f98b00fd480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:13:24.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:24.477+0000 7f98b6600700 1 -- 172.21.15.62:0/3470267089 shutdown_connections 2024-04-20T22:13:24.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:24.477+0000 7f98b6600700 1 -- 172.21.15.62:0/3470267089 wait complete. 2024-04-20T22:13:24.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:24.642+0000 7fcf78c00700 1 -- 172.21.15.62:0/3668364407 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf74100d30 msgr2=0x7fcf74104d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:13:24.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:24.642+0000 7fcf78c00700 1 --2- 172.21.15.62:0/3668364407 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf74100d30 0x7fcf74104d00 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7fcf5c00b0a0 tx=0x7fcf5c00b3b0 comp rx=0 tx=0).stop 2024-04-20T22:13:24.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:24.642+0000 7fcf78c00700 1 -- 172.21.15.62:0/3668364407 shutdown_connections 2024-04-20T22:13:24.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:24.642+0000 7fcf78c00700 1 --2- 172.21.15.62:0/3668364407 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf74100d30 0x7fcf74104d00 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:13:24.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:24.642+0000 7fcf78c00700 1 --2- 172.21.15.62:0/3668364407 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcf74100390 0x7fcf74100760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:13:24.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:24.642+0000 7fcf78c00700 1 -- 172.21.15.62:0/3668364407 >> 172.21.15.62:0/3668364407 conn(0x7fcf740fbc30 msgr2=0x7fcf740fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:13:24.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:24.642+0000 7fcf78c00700 1 -- 172.21.15.62:0/3668364407 shutdown_connections 2024-04-20T22:13:24.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:24.643+0000 7fcf78c00700 1 -- 172.21.15.62:0/3668364407 wait complete. 2024-04-20T22:13:24.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:24.643+0000 7fcf78c00700 1 Processor -- start 2024-04-20T22:13:24.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:24.644+0000 7fcf78c00700 1 -- start start 2024-04-20T22:13:24.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:24.644+0000 7fcf78c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf74100390 0x7fcf74197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:13:24.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:24.644+0000 7fcf78c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcf74100d30 0x7fcf741984e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:13:24.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:24.644+0000 7fcf78c00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf74198bc0 con 0x7fcf74100390 2024-04-20T22:13:24.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:24.644+0000 7fcf78c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf7419c950 con 0x7fcf74100d30 2024-04-20T22:13:24.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:24.644+0000 7fcf71e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf74100390 0x7fcf74197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:13:24.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:24.644+0000 7fcf71e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf74100390 0x7fcf74197fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41592/0 (socket says 172.21.15.62:41592) 2024-04-20T22:13:24.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:24.644+0000 7fcf71e00700 1 -- 172.21.15.62:0/887363946 learned_addr learned my addr 172.21.15.62:0/887363946 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:13:24.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:24.645+0000 7fcf71e00700 1 -- 172.21.15.62:0/887363946 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcf74100d30 msgr2=0x7fcf741984e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:13:24.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:24.645+0000 7fcf71e00700 1 --2- 172.21.15.62:0/887363946 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcf74100d30 0x7fcf741984e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:13:24.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:24.645+0000 7fcf71e00700 1 -- 172.21.15.62:0/887363946 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf5c009d00 con 0x7fcf74100390 2024-04-20T22:13:24.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:24.645+0000 7fcf71e00700 1 --2- 172.21.15.62:0/887363946 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf74100390 0x7fcf74197fa0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7fcf6400d8d0 tx=0x7fcf6400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:13:24.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:24.646+0000 7fcf6b400700 1 -- 172.21.15.62:0/887363946 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcf64009940 con 0x7fcf74100390 2024-04-20T22:13:24.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:24.646+0000 7fcf6b400700 1 -- 172.21.15.62:0/887363946 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcf64010460 con 0x7fcf74100390 2024-04-20T22:13:24.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:24.646+0000 7fcf78c00700 1 -- 172.21.15.62:0/887363946 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf7419cc30 con 0x7fcf74100390 2024-04-20T22:13:24.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:24.646+0000 7fcf6b400700 1 -- 172.21.15.62:0/887363946 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcf6400f5f0 con 0x7fcf74100390 2024-04-20T22:13:24.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:24.646+0000 7fcf78c00700 1 -- 172.21.15.62:0/887363946 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf7419d180 con 0x7fcf74100390 2024-04-20T22:13:24.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:24.647+0000 7fcf78c00700 1 -- 172.21.15.62:0/887363946 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf74104480 con 0x7fcf74100390 2024-04-20T22:13:24.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:24.648+0000 7fcf6b400700 1 -- 172.21.15.62:0/887363946 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fcf640105d0 con 0x7fcf74100390 2024-04-20T22:13:24.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:24.650+0000 7fcf6b400700 1 --2- 172.21.15.62:0/887363946 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fcf5806ecc0 0x7fcf58071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:13:24.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:24.650+0000 7fcf6b400700 1 -- 172.21.15.62:0/887363946 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fcf6408ee10 con 0x7fcf74100390 2024-04-20T22:13:24.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:24.650+0000 7fcf69400700 1 --2- 172.21.15.62:0/887363946 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fcf5806ecc0 0x7fcf58071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:13:24.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:24.651+0000 7fcf69400700 1 --2- 172.21.15.62:0/887363946 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fcf5806ecc0 0x7fcf58071170 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fcf5c00b830 tx=0x7fcf5c006000 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:13:24.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:24.655+0000 7fcf6b400700 1 -- 172.21.15.62:0/887363946 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fcf64057ba0 con 0x7fcf74100390 2024-04-20T22:13:24.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:24.969+0000 7fcf78c00700 1 -- 172.21.15.62:0/887363946 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fcf7404f2a0 con 0x7fcf74100390 2024-04-20T22:13:24.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:24.970+0000 7fcf6b400700 1 -- 172.21.15.62:0/887363946 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 29 v29) v1 ==== 76+0+1887 (secure 0 0 0) 0x7fcf64016770 con 0x7fcf74100390 2024-04-20T22:13:24.971 INFO:teuthology.orchestra.run.smithi062.stdout:e29 2024-04-20T22:13:24.971 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T22:13:24.971 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:13:24.971 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T22:13:24.971 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:13:24.971 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T22:13:24.971 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T22:13:24.971 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 20 2024-04-20T22:13:24.971 INFO:teuthology.orchestra.run.smithi062.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-20T22:13:24.971 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T22:13:24.971 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T21:42:01.559061+0000 2024-04-20T22:13:24.972 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T22:13:24.972 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T22:13:24.972 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T22:13:24.972 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T22:13:24.972 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T22:13:24.972 INFO:teuthology.orchestra.run.smithi062.stdout:max_xattr_size 65536 2024-04-20T22:13:24.972 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T22:13:24.972 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T22:13:24.972 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 81 2024-04-20T22:13:24.972 INFO:teuthology.orchestra.run.smithi062.stdout:snap_rstat 1 2024-04-20T22:13:24.972 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:13:24.972 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 2 2024-04-20T22:13:24.972 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1 2024-04-20T22:13:24.972 INFO:teuthology.orchestra.run.smithi062.stdout:up {} 2024-04-20T22:13:24.972 INFO:teuthology.orchestra.run.smithi062.stdout:failed 0,1 2024-04-20T22:13:24.973 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T22:13:24.973 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T22:13:24.973 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T22:13:24.973 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T22:13:24.973 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data enabled 2024-04-20T22:13:24.973 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T22:13:24.973 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T22:13:24.973 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 1 2024-04-20T22:13:24.973 INFO:teuthology.orchestra.run.smithi062.stdout:qdb_cluster leader: 0 members: 2024-04-20T22:13:24.973 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:13:24.973 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:13:24.973 INFO:teuthology.orchestra.run.smithi062.stdout:Standby daemons: 2024-04-20T22:13:24.973 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:13:24.973 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ryqblb{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6826/4113916205,v1:172.21.15.62:6827/4113916205] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:13:24.973 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ozizwi{-1:34476} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6828/3801975165,v1:172.21.15.62:6829/3801975165] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:13:24.973 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.dvimer{-1:44727} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6824/3576699232,v1:172.21.15.189:6825/3576699232] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:13:24.974 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.xbgspw{-1:45075} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6826/212905792,v1:172.21.15.189:6827/212905792] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:13:24.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:24.973+0000 7fcf78c00700 1 -- 172.21.15.62:0/887363946 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fcf5806ecc0 msgr2=0x7fcf58071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:13:24.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:24.973+0000 7fcf78c00700 1 --2- 172.21.15.62:0/887363946 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fcf5806ecc0 0x7fcf58071170 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fcf5c00b830 tx=0x7fcf5c006000 comp rx=0 tx=0).stop 2024-04-20T22:13:24.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:24.973+0000 7fcf78c00700 1 -- 172.21.15.62:0/887363946 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf74100390 msgr2=0x7fcf74197fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:13:24.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:24.973+0000 7fcf78c00700 1 --2- 172.21.15.62:0/887363946 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf74100390 0x7fcf74197fa0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7fcf6400d8d0 tx=0x7fcf6400dc90 comp rx=0 tx=0).stop 2024-04-20T22:13:24.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:24.973+0000 7fcf78c00700 1 -- 172.21.15.62:0/887363946 shutdown_connections 2024-04-20T22:13:24.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:24.973+0000 7fcf78c00700 1 --2- 172.21.15.62:0/887363946 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fcf5806ecc0 0x7fcf58071170 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:13:24.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:24.973+0000 7fcf78c00700 1 --2- 172.21.15.62:0/887363946 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf74100390 0x7fcf74197fa0 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:13:24.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:24.973+0000 7fcf78c00700 1 --2- 172.21.15.62:0/887363946 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcf74100d30 0x7fcf741984e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:13:24.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:24.973+0000 7fcf78c00700 1 -- 172.21.15.62:0/887363946 >> 172.21.15.62:0/887363946 conn(0x7fcf740fbc30 msgr2=0x7fcf740fd1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:13:24.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:24.974+0000 7fcf78c00700 1 -- 172.21.15.62:0/887363946 shutdown_connections 2024-04-20T22:13:24.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:24.974+0000 7fcf78c00700 1 -- 172.21.15.62:0/887363946 wait complete. 2024-04-20T22:13:24.975 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 29 2024-04-20T22:13:25.134 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:13:24 smithi062 ceph-mon[76322]: from='client.36026 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:13:25.134 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:13:24 smithi062 ceph-mon[76322]: from='client.36030 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:13:25.134 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:13:24 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/3470267089' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:13:25.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:25.144+0000 7f3956c00700 1 -- 172.21.15.62:0/1532947144 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3950103c90 msgr2=0x7f3950104060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:13:25.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:25.144+0000 7f3956c00700 1 --2- 172.21.15.62:0/1532947144 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3950103c90 0x7f3950104060 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f3944009910 tx=0x7f3944009c20 comp rx=0 tx=0).stop 2024-04-20T22:13:25.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:25.144+0000 7f3956c00700 1 -- 172.21.15.62:0/1532947144 shutdown_connections 2024-04-20T22:13:25.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:25.144+0000 7f3956c00700 1 --2- 172.21.15.62:0/1532947144 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39501045a0 0x7f3950068540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:13:25.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:25.144+0000 7f3956c00700 1 --2- 172.21.15.62:0/1532947144 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3950103c90 0x7f3950104060 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:13:25.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:25.144+0000 7f3956c00700 1 -- 172.21.15.62:0/1532947144 >> 172.21.15.62:0/1532947144 conn(0x7f3950074fa0 msgr2=0x7f39500753a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:13:25.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:25.144+0000 7f3956c00700 1 -- 172.21.15.62:0/1532947144 shutdown_connections 2024-04-20T22:13:25.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:25.144+0000 7f3956c00700 1 -- 172.21.15.62:0/1532947144 wait complete. 2024-04-20T22:13:25.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:25.146+0000 7f3956c00700 1 Processor -- start 2024-04-20T22:13:25.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:25.146+0000 7f3956c00700 1 -- start start 2024-04-20T22:13:25.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:25.147+0000 7f3956c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3950103c90 0x7f395019c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:13:25.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:25.147+0000 7f3956c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39501045a0 0x7f395019c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:13:25.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:25.147+0000 7f3956c00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f395019cd60 con 0x7f39501045a0 2024-04-20T22:13:25.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:25.147+0000 7f3956c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39501a0af0 con 0x7f3950103c90 2024-04-20T22:13:25.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:25.147+0000 7f3954e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39501045a0 0x7f395019c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:13:25.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:25.147+0000 7f3954e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39501045a0 0x7f395019c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41612/0 (socket says 172.21.15.62:41612) 2024-04-20T22:13:25.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:25.147+0000 7f3954e00700 1 -- 172.21.15.62:0/774895307 learned_addr learned my addr 172.21.15.62:0/774895307 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:13:25.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:25.147+0000 7f3955800700 1 --2- 172.21.15.62:0/774895307 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3950103c90 0x7f395019c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:13:25.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:25.148+0000 7f3954e00700 1 -- 172.21.15.62:0/774895307 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3950103c90 msgr2=0x7f395019c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:13:25.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:25.148+0000 7f3954e00700 1 --2- 172.21.15.62:0/774895307 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3950103c90 0x7f395019c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:13:25.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:25.148+0000 7f3954e00700 1 -- 172.21.15.62:0/774895307 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39440095f0 con 0x7f39501045a0 2024-04-20T22:13:25.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:25.148+0000 7f3955800700 1 --2- 172.21.15.62:0/774895307 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3950103c90 0x7f395019c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-20T22:13:25.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:25.148+0000 7f3954e00700 1 --2- 172.21.15.62:0/774895307 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39501045a0 0x7f395019c680 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f393c00d900 tx=0x7f393c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:13:25.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:25.149+0000 7f394e000700 1 -- 172.21.15.62:0/774895307 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f393c0041f0 con 0x7f39501045a0 2024-04-20T22:13:25.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:25.149+0000 7f394e000700 1 -- 172.21.15.62:0/774895307 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f393c004cf0 con 0x7f39501045a0 2024-04-20T22:13:25.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:25.149+0000 7f3956c00700 1 -- 172.21.15.62:0/774895307 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39501a0dd0 con 0x7f39501045a0 2024-04-20T22:13:25.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:25.149+0000 7f394e000700 1 -- 172.21.15.62:0/774895307 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f393c00b750 con 0x7f39501045a0 2024-04-20T22:13:25.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:25.149+0000 7f3956c00700 1 -- 172.21.15.62:0/774895307 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39501a1320 con 0x7f39501045a0 2024-04-20T22:13:25.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:25.150+0000 7f3956c00700 1 -- 172.21.15.62:0/774895307 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3950062380 con 0x7f39501045a0 2024-04-20T22:13:25.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:25.152+0000 7f394e000700 1 -- 172.21.15.62:0/774895307 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f393c004350 con 0x7f39501045a0 2024-04-20T22:13:25.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:25.153+0000 7f394e000700 1 --2- 172.21.15.62:0/774895307 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f393806ecc0 0x7f3938071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:13:25.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:25.153+0000 7f394e000700 1 -- 172.21.15.62:0/774895307 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f393c08f050 con 0x7f39501045a0 2024-04-20T22:13:25.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:25.153+0000 7f3955800700 1 --2- 172.21.15.62:0/774895307 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f393806ecc0 0x7f3938071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:13:25.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:25.154+0000 7f3955800700 1 --2- 172.21.15.62:0/774895307 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f393806ecc0 0x7f3938071170 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f394400b5c0 tx=0x7f394401a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:13:25.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:25.163+0000 7f394e000700 1 -- 172.21.15.62:0/774895307 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f393c057de0 con 0x7f39501045a0 2024-04-20T22:13:25.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:13:24 smithi189 ceph-mon[57022]: from='client.36026 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:13:25.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:13:24 smithi189 ceph-mon[57022]: from='client.36030 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:13:25.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:13:24 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/3470267089' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:13:25.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:25.430+0000 7f3956c00700 1 -- 172.21.15.62:0/774895307 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f39501a1850 con 0x7f393806ecc0 2024-04-20T22:13:25.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:25.432+0000 7f394e000700 1 -- 172.21.15.62:0/774895307 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f39501a1850 con 0x7f393806ecc0 2024-04-20T22:13:25.432 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:13:25.433 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T22:13:25.433 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T22:13:25.433 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-20T22:13:25.433 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T22:13:25.433 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr", 2024-04-20T22:13:25.433 INFO:teuthology.orchestra.run.smithi062.stdout: "osd", 2024-04-20T22:13:25.433 INFO:teuthology.orchestra.run.smithi062.stdout: "mds", 2024-04-20T22:13:25.433 INFO:teuthology.orchestra.run.smithi062.stdout: "crash", 2024-04-20T22:13:25.433 INFO:teuthology.orchestra.run.smithi062.stdout: "mon" 2024-04-20T22:13:25.433 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T22:13:25.433 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "16/23 daemons upgraded", 2024-04-20T22:13:25.433 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading mds daemons", 2024-04-20T22:13:25.433 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T22:13:25.433 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:13:25.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:25.435+0000 7f3956c00700 1 -- 172.21.15.62:0/774895307 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f393806ecc0 msgr2=0x7f3938071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:13:25.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:25.435+0000 7f3956c00700 1 --2- 172.21.15.62:0/774895307 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f393806ecc0 0x7f3938071170 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f394400b5c0 tx=0x7f394401a040 comp rx=0 tx=0).stop 2024-04-20T22:13:25.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:25.435+0000 7f3956c00700 1 -- 172.21.15.62:0/774895307 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39501045a0 msgr2=0x7f395019c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:13:25.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:25.435+0000 7f3956c00700 1 --2- 172.21.15.62:0/774895307 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39501045a0 0x7f395019c680 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f393c00d900 tx=0x7f393c00dcc0 comp rx=0 tx=0).stop 2024-04-20T22:13:25.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:25.436+0000 7f3956c00700 1 -- 172.21.15.62:0/774895307 shutdown_connections 2024-04-20T22:13:25.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:25.436+0000 7f3956c00700 1 --2- 172.21.15.62:0/774895307 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f393806ecc0 0x7f3938071170 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:13:25.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:25.436+0000 7f3956c00700 1 --2- 172.21.15.62:0/774895307 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3950103c90 0x7f395019c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:13:25.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:25.436+0000 7f3956c00700 1 --2- 172.21.15.62:0/774895307 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39501045a0 0x7f395019c680 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:13:25.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:25.436+0000 7f3956c00700 1 -- 172.21.15.62:0/774895307 >> 172.21.15.62:0/774895307 conn(0x7f3950074fa0 msgr2=0x7f395010f6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:13:25.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:25.436+0000 7f3956c00700 1 -- 172.21.15.62:0/774895307 shutdown_connections 2024-04-20T22:13:25.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:25.436+0000 7f3956c00700 1 -- 172.21.15.62:0/774895307 wait complete. 2024-04-20T22:13:25.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:25.599+0000 7fe731600700 1 -- 172.21.15.62:0/3841713725 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe72c10c660 msgr2=0x7fe72c10ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:13:25.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:25.599+0000 7fe731600700 1 --2- 172.21.15.62:0/3841713725 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe72c10c660 0x7fe72c10ca30 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7fe71c009910 tx=0x7fe71c009c20 comp rx=0 tx=0).stop 2024-04-20T22:13:25.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:25.600+0000 7fe731600700 1 -- 172.21.15.62:0/3841713725 shutdown_connections 2024-04-20T22:13:25.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:25.600+0000 7fe731600700 1 --2- 172.21.15.62:0/3841713725 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe72c106b80 0x7fe72c106ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:13:25.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:25.600+0000 7fe731600700 1 --2- 172.21.15.62:0/3841713725 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe72c10c660 0x7fe72c10ca30 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:13:25.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:25.600+0000 7fe731600700 1 -- 172.21.15.62:0/3841713725 >> 172.21.15.62:0/3841713725 conn(0x7fe72c075e90 msgr2=0x7fe72c0782a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:13:25.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:25.600+0000 7fe731600700 1 -- 172.21.15.62:0/3841713725 shutdown_connections 2024-04-20T22:13:25.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:25.600+0000 7fe731600700 1 -- 172.21.15.62:0/3841713725 wait complete. 2024-04-20T22:13:25.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:25.601+0000 7fe731600700 1 Processor -- start 2024-04-20T22:13:25.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:25.601+0000 7fe731600700 1 -- start start 2024-04-20T22:13:25.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:25.601+0000 7fe731600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe72c106b80 0x7fe72c19c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:13:25.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:25.601+0000 7fe731600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe72c10c660 0x7fe72c19c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:13:25.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:25.601+0000 7fe731600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe72c19cf70 con 0x7fe72c106b80 2024-04-20T22:13:25.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:25.601+0000 7fe731600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe72c1a0d00 con 0x7fe72c10c660 2024-04-20T22:13:25.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:25.602+0000 7fe72be00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe72c106b80 0x7fe72c19c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:13:25.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:25.602+0000 7fe72be00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe72c106b80 0x7fe72c19c350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41630/0 (socket says 172.21.15.62:41630) 2024-04-20T22:13:25.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:25.602+0000 7fe72be00700 1 -- 172.21.15.62:0/3062471345 learned_addr learned my addr 172.21.15.62:0/3062471345 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:13:25.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:25.602+0000 7fe72be00700 1 -- 172.21.15.62:0/3062471345 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe72c10c660 msgr2=0x7fe72c19c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:13:25.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:25.602+0000 7fe72be00700 1 --2- 172.21.15.62:0/3062471345 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe72c10c660 0x7fe72c19c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:13:25.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:25.602+0000 7fe72be00700 1 -- 172.21.15.62:0/3062471345 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe71c0095f0 con 0x7fe72c106b80 2024-04-20T22:13:25.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:25.602+0000 7fe72be00700 1 --2- 172.21.15.62:0/3062471345 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe72c106b80 0x7fe72c19c350 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7fe71c00b5c0 tx=0x7fe71c00be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:13:25.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:25.603+0000 7fe728c00700 1 -- 172.21.15.62:0/3062471345 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe71c01d070 con 0x7fe72c106b80 2024-04-20T22:13:25.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:25.603+0000 7fe728c00700 1 -- 172.21.15.62:0/3062471345 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe71c00f460 con 0x7fe72c106b80 2024-04-20T22:13:25.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:25.603+0000 7fe731600700 1 -- 172.21.15.62:0/3062471345 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe72c1a0f80 con 0x7fe72c106b80 2024-04-20T22:13:25.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:25.603+0000 7fe728c00700 1 -- 172.21.15.62:0/3062471345 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe71c017610 con 0x7fe72c106b80 2024-04-20T22:13:25.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:25.603+0000 7fe731600700 1 -- 172.21.15.62:0/3062471345 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe72c1a1470 con 0x7fe72c106b80 2024-04-20T22:13:25.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:25.604+0000 7fe731600700 1 -- 172.21.15.62:0/3062471345 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe72c10eb60 con 0x7fe72c106b80 2024-04-20T22:13:25.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:25.606+0000 7fe728c00700 1 -- 172.21.15.62:0/3062471345 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe71c00f5d0 con 0x7fe72c106b80 2024-04-20T22:13:25.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:25.606+0000 7fe728c00700 1 --2- 172.21.15.62:0/3062471345 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe71006ecc0 0x7fe710071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:13:25.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:25.607+0000 7fe728c00700 1 -- 172.21.15.62:0/3062471345 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fe71c090660 con 0x7fe72c106b80 2024-04-20T22:13:25.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:25.607+0000 7fe72b400700 1 --2- 172.21.15.62:0/3062471345 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe71006ecc0 0x7fe710071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:13:25.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:25.607+0000 7fe72b400700 1 --2- 172.21.15.62:0/3062471345 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe71006ecc0 0x7fe710071170 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fe72c19d970 tx=0x7fe718006d20 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:13:25.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:25.610+0000 7fe728c00700 1 -- 172.21.15.62:0/3062471345 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fe71c059470 con 0x7fe72c106b80 2024-04-20T22:13:25.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:25.961+0000 7fe731600700 1 -- 172.21.15.62:0/3062471345 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fe72c062380 con 0x7fe72c106b80 2024-04-20T22:13:25.961 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:13:25 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/887363946' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:13:25.962 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:13:25 smithi062 ceph-mon[76322]: pgmap v1147: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:13:25.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:25.962+0000 7fe728c00700 1 -- 172.21.15.62:0/3062471345 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7fe71c05cb30 con 0x7fe72c106b80 2024-04-20T22:13:25.963 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-20T22:13:25.963 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-20T22:13:25.963 INFO:teuthology.orchestra.run.smithi062.stdout: host smithi189 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi189/config 2024-04-20T22:13:25.963 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-20T22:13:25.963 INFO:teuthology.orchestra.run.smithi062.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi189 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp11670v2q:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:13:25.963 INFO:teuthology.orchestra.run.smithi062.stdout:Traceback (most recent call last): 2024-04-20T22:13:25.963 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-20T22:13:25.963 INFO:teuthology.orchestra.run.smithi062.stdout: return _run_code(code, main_globals, None, 2024-04-20T22:13:25.963 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-20T22:13:25.963 INFO:teuthology.orchestra.run.smithi062.stdout: exec(code, run_globals) 2024-04-20T22:13:25.963 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5618, in 2024-04-20T22:13:25.963 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5606, in main 2024-04-20T22:13:25.964 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 408, in _infer_config 2024-04-20T22:13:25.964 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 323, in _infer_fsid 2024-04-20T22:13:25.964 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 436, in _infer_image 2024-04-20T22:13:25.964 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 310, in _validate_fsid 2024-04-20T22:13:25.964 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 3359, in command_ceph_volume 2024-04-20T22:13:25.964 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-20T22:13:25.964 INFO:teuthology.orchestra.run.smithi062.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi189 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp11670v2q:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:13:25.964 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-20T22:13:25.964 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is degraded 2024-04-20T22:13:25.964 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T22:13:25.964 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T22:13:25.964 INFO:teuthology.orchestra.run.smithi062.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-20T22:13:25.964 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-20T22:13:25.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:25.965+0000 7fe731600700 1 -- 172.21.15.62:0/3062471345 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe71006ecc0 msgr2=0x7fe710071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:13:25.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:25.965+0000 7fe731600700 1 --2- 172.21.15.62:0/3062471345 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe71006ecc0 0x7fe710071170 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fe72c19d970 tx=0x7fe718006d20 comp rx=0 tx=0).stop 2024-04-20T22:13:25.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:25.965+0000 7fe731600700 1 -- 172.21.15.62:0/3062471345 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe72c106b80 msgr2=0x7fe72c19c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:13:25.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:25.965+0000 7fe731600700 1 --2- 172.21.15.62:0/3062471345 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe72c106b80 0x7fe72c19c350 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7fe71c00b5c0 tx=0x7fe71c00be00 comp rx=0 tx=0).stop 2024-04-20T22:13:25.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:25.966+0000 7fe731600700 1 -- 172.21.15.62:0/3062471345 shutdown_connections 2024-04-20T22:13:25.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:25.966+0000 7fe731600700 1 --2- 172.21.15.62:0/3062471345 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe71006ecc0 0x7fe710071170 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:13:25.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:25.966+0000 7fe731600700 1 --2- 172.21.15.62:0/3062471345 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe72c106b80 0x7fe72c19c350 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:13:25.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:25.966+0000 7fe731600700 1 --2- 172.21.15.62:0/3062471345 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe72c10c660 0x7fe72c19c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:13:25.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:25.966+0000 7fe731600700 1 -- 172.21.15.62:0/3062471345 >> 172.21.15.62:0/3062471345 conn(0x7fe72c075e90 msgr2=0x7fe72c077a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:13:25.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:25.966+0000 7fe731600700 1 -- 172.21.15.62:0/3062471345 shutdown_connections 2024-04-20T22:13:25.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:25.966+0000 7fe731600700 1 -- 172.21.15.62:0/3062471345 wait complete. 2024-04-20T22:13:26.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:13:25 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/887363946' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:13:26.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:13:25 smithi189 ceph-mon[57022]: pgmap v1147: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:13:27.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:13:26 smithi062 ceph-mon[76322]: from='client.36042 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:13:27.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:13:26 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/3062471345' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:13:27.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:13:26 smithi189 ceph-mon[57022]: from='client.36042 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:13:27.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:13:26 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/3062471345' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:13:28.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:13:27 smithi062 ceph-mon[76322]: pgmap v1148: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:13:28.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:13:27 smithi189 ceph-mon[57022]: pgmap v1148: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:13:30.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:13:30 smithi189 ceph-mon[57022]: pgmap v1149: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:13:30.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:13:30 smithi062 ceph-mon[76322]: pgmap v1149: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:13:32.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:13:32 smithi062 ceph-mon[76322]: pgmap v1150: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:13:32.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:13:32 smithi189 ceph-mon[57022]: pgmap v1150: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:13:34.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:13:34 smithi189 ceph-mon[57022]: pgmap v1151: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:13:34.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:13:34 smithi062 ceph-mon[76322]: pgmap v1151: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:13:35.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:13:35 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:13:35.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:13:35 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:13:36.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:13:36 smithi062 ceph-mon[76322]: pgmap v1152: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:13:36.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:13:36 smithi189 ceph-mon[57022]: pgmap v1152: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:13:38.511 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:13:38 smithi062 ceph-mon[76322]: pgmap v1153: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:13:38.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:13:38 smithi189 ceph-mon[57022]: pgmap v1153: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:13:40.530 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:13:40 smithi189 ceph-mon[57022]: pgmap v1154: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:13:40.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:13:40 smithi062 ceph-mon[76322]: pgmap v1154: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:13:42.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:13:42 smithi189 ceph-mon[57022]: pgmap v1155: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:13:42.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:13:42 smithi062 ceph-mon[76322]: pgmap v1155: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:13:44.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:13:44 smithi189 ceph-mon[57022]: pgmap v1156: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:13:44.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:13:44 smithi062 ceph-mon[76322]: pgmap v1156: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:13:46.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:13:46 smithi189 ceph-mon[57022]: pgmap v1157: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:13:46.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:13:46 smithi062 ceph-mon[76322]: pgmap v1157: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:13:48.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:13:48 smithi062 ceph-mon[76322]: pgmap v1158: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:13:48.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:13:48 smithi189 ceph-mon[57022]: pgmap v1158: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:13:50.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:13:50 smithi189 ceph-mon[57022]: pgmap v1159: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:13:50.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:13:50 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:13:50.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:13:50 smithi062 ceph-mon[76322]: pgmap v1159: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:13:50.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:13:50 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:13:52.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:13:52 smithi189 ceph-mon[57022]: pgmap v1160: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:13:52.599 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:13:52 smithi062 ceph-mon[76322]: pgmap v1160: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:13:54.668 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:13:54 smithi189 ceph-mon[57022]: pgmap v1161: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:13:54.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:13:54 smithi062 ceph-mon[76322]: pgmap v1161: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:13:56.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:56.144+0000 7fa0fbe00700 1 -- 172.21.15.62:0/405686786 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0f4103040 msgr2=0x7fa0f4107010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:13:56.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:56.144+0000 7fa0fbe00700 1 --2- 172.21.15.62:0/405686786 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0f4103040 0x7fa0f4107010 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7fa0e0009910 tx=0x7fa0e0009c20 comp rx=0 tx=0).stop 2024-04-20T22:13:56.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:56.144+0000 7fa0fbe00700 1 -- 172.21.15.62:0/405686786 shutdown_connections 2024-04-20T22:13:56.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:56.144+0000 7fa0fbe00700 1 --2- 172.21.15.62:0/405686786 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0f4103040 0x7fa0f4107010 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:13:56.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:56.144+0000 7fa0fbe00700 1 --2- 172.21.15.62:0/405686786 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0f41026a0 0x7fa0f4102a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:13:56.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:56.144+0000 7fa0fbe00700 1 -- 172.21.15.62:0/405686786 >> 172.21.15.62:0/405686786 conn(0x7fa0f40fdf40 msgr2=0x7fa0f4100350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:13:56.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:56.144+0000 7fa0fbe00700 1 -- 172.21.15.62:0/405686786 shutdown_connections 2024-04-20T22:13:56.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:56.144+0000 7fa0fbe00700 1 -- 172.21.15.62:0/405686786 wait complete. 2024-04-20T22:13:56.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:56.146+0000 7fa0fbe00700 1 Processor -- start 2024-04-20T22:13:56.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:56.146+0000 7fa0fbe00700 1 -- start start 2024-04-20T22:13:56.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:56.146+0000 7fa0fbe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0f41026a0 0x7fa0f4198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:13:56.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:56.146+0000 7fa0fbe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0f4103040 0x7fa0f4198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:13:56.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:56.146+0000 7fa0fbe00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0f4198c70 con 0x7fa0f4103040 2024-04-20T22:13:56.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:56.146+0000 7fa0fbe00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0f419ca00 con 0x7fa0f41026a0 2024-04-20T22:13:56.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:56.146+0000 7fa0f8a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0f4103040 0x7fa0f4198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:13:56.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:56.147+0000 7fa0f8a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0f4103040 0x7fa0f4198590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37172/0 (socket says 172.21.15.62:37172) 2024-04-20T22:13:56.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:56.147+0000 7fa0f8a00700 1 -- 172.21.15.62:0/2504626105 learned_addr learned my addr 172.21.15.62:0/2504626105 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:13:56.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:56.147+0000 7fa0f8a00700 1 -- 172.21.15.62:0/2504626105 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0f41026a0 msgr2=0x7fa0f4198050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:13:56.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:56.147+0000 7fa0f9400700 1 --2- 172.21.15.62:0/2504626105 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0f41026a0 0x7fa0f4198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:13:56.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:56.147+0000 7fa0f8a00700 1 --2- 172.21.15.62:0/2504626105 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0f41026a0 0x7fa0f4198050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:13:56.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:56.147+0000 7fa0f8a00700 1 -- 172.21.15.62:0/2504626105 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0e00095f0 con 0x7fa0f4103040 2024-04-20T22:13:56.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:56.147+0000 7fa0f9400700 1 --2- 172.21.15.62:0/2504626105 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0f41026a0 0x7fa0f4198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T22:13:56.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:56.147+0000 7fa0f8a00700 1 --2- 172.21.15.62:0/2504626105 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0f4103040 0x7fa0f4198590 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fa0e00092e0 tx=0x7fa0e000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:13:56.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:56.148+0000 7fa0f2000700 1 -- 172.21.15.62:0/2504626105 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa0e001d070 con 0x7fa0f4103040 2024-04-20T22:13:56.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:56.148+0000 7fa0f2000700 1 -- 172.21.15.62:0/2504626105 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa0e0005dc0 con 0x7fa0f4103040 2024-04-20T22:13:56.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:56.148+0000 7fa0fbe00700 1 -- 172.21.15.62:0/2504626105 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0f419cc80 con 0x7fa0f4103040 2024-04-20T22:13:56.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:56.148+0000 7fa0f2000700 1 -- 172.21.15.62:0/2504626105 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa0e0017400 con 0x7fa0f4103040 2024-04-20T22:13:56.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:56.148+0000 7fa0fbe00700 1 -- 172.21.15.62:0/2504626105 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0f419d170 con 0x7fa0f4103040 2024-04-20T22:13:56.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:56.149+0000 7fa0fbe00700 1 -- 172.21.15.62:0/2504626105 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0f410aa10 con 0x7fa0f4103040 2024-04-20T22:13:56.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:56.150+0000 7fa0f2000700 1 -- 172.21.15.62:0/2504626105 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fa0e000f460 con 0x7fa0f4103040 2024-04-20T22:13:56.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:56.152+0000 7fa0f2000700 1 --2- 172.21.15.62:0/2504626105 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa0dc06ecc0 0x7fa0dc071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:13:56.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:56.152+0000 7fa0f2000700 1 -- 172.21.15.62:0/2504626105 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fa0e0090360 con 0x7fa0f4103040 2024-04-20T22:13:56.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:56.152+0000 7fa0f9400700 1 --2- 172.21.15.62:0/2504626105 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa0dc06ecc0 0x7fa0dc071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:13:56.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:56.153+0000 7fa0f9400700 1 --2- 172.21.15.62:0/2504626105 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa0dc06ecc0 0x7fa0dc071170 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fa0e8005fd0 tx=0x7fa0e8005ee0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:13:56.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:56.157+0000 7fa0f2000700 1 -- 172.21.15.62:0/2504626105 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fa0e00590f0 con 0x7fa0f4103040 2024-04-20T22:13:56.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:56.416+0000 7fa0fbe00700 1 -- 172.21.15.62:0/2504626105 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa0f419d4a0 con 0x7fa0dc06ecc0 2024-04-20T22:13:56.416 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:13:56 smithi062 ceph-mon[76322]: pgmap v1162: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:13:56.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:56.420+0000 7fa0f2000700 1 -- 172.21.15.62:0/2504626105 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fa0f419d4a0 con 0x7fa0dc06ecc0 2024-04-20T22:13:56.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:56.423+0000 7fa0fbe00700 1 -- 172.21.15.62:0/2504626105 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa0dc06ecc0 msgr2=0x7fa0dc071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:13:56.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:56.423+0000 7fa0fbe00700 1 --2- 172.21.15.62:0/2504626105 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa0dc06ecc0 0x7fa0dc071170 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fa0e8005fd0 tx=0x7fa0e8005ee0 comp rx=0 tx=0).stop 2024-04-20T22:13:56.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:56.423+0000 7fa0fbe00700 1 -- 172.21.15.62:0/2504626105 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0f4103040 msgr2=0x7fa0f4198590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:13:56.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:56.423+0000 7fa0fbe00700 1 --2- 172.21.15.62:0/2504626105 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0f4103040 0x7fa0f4198590 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fa0e00092e0 tx=0x7fa0e000be20 comp rx=0 tx=0).stop 2024-04-20T22:13:56.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:56.423+0000 7fa0fbe00700 1 -- 172.21.15.62:0/2504626105 shutdown_connections 2024-04-20T22:13:56.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:56.423+0000 7fa0fbe00700 1 --2- 172.21.15.62:0/2504626105 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa0dc06ecc0 0x7fa0dc071170 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:13:56.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:56.423+0000 7fa0fbe00700 1 --2- 172.21.15.62:0/2504626105 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0f41026a0 0x7fa0f4198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:13:56.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:56.423+0000 7fa0fbe00700 1 --2- 172.21.15.62:0/2504626105 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0f4103040 0x7fa0f4198590 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:13:56.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:56.423+0000 7fa0fbe00700 1 -- 172.21.15.62:0/2504626105 >> 172.21.15.62:0/2504626105 conn(0x7fa0f40fdf40 msgr2=0x7fa0f4100320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:13:56.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:56.423+0000 7fa0fbe00700 1 -- 172.21.15.62:0/2504626105 shutdown_connections 2024-04-20T22:13:56.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:56.424+0000 7fa0fbe00700 1 -- 172.21.15.62:0/2504626105 wait complete. 2024-04-20T22:13:56.440 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T22:13:56.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:56.585+0000 7fd3b3a00700 1 -- 172.21.15.62:0/4016720770 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3ac10c8f0 msgr2=0x7fd3ac10ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:13:56.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:56.585+0000 7fd3b3a00700 1 --2- 172.21.15.62:0/4016720770 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3ac10c8f0 0x7fd3ac10ccc0 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7fd3a0009910 tx=0x7fd3a0009c20 comp rx=0 tx=0).stop 2024-04-20T22:13:56.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:56.585+0000 7fd3b3a00700 1 -- 172.21.15.62:0/4016720770 shutdown_connections 2024-04-20T22:13:56.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:56.585+0000 7fd3b3a00700 1 --2- 172.21.15.62:0/4016720770 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3ac106b80 0x7fd3ac106ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:13:56.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:56.585+0000 7fd3b3a00700 1 --2- 172.21.15.62:0/4016720770 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3ac10c8f0 0x7fd3ac10ccc0 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:13:56.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:56.585+0000 7fd3b3a00700 1 -- 172.21.15.62:0/4016720770 >> 172.21.15.62:0/4016720770 conn(0x7fd3ac076060 msgr2=0x7fd3ac078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:13:56.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:56.585+0000 7fd3b3a00700 1 -- 172.21.15.62:0/4016720770 shutdown_connections 2024-04-20T22:13:56.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:56.585+0000 7fd3b3a00700 1 -- 172.21.15.62:0/4016720770 wait complete. 2024-04-20T22:13:56.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:56.586+0000 7fd3b3a00700 1 Processor -- start 2024-04-20T22:13:56.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:56.586+0000 7fd3b3a00700 1 -- start start 2024-04-20T22:13:56.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:56.587+0000 7fd3b3a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3ac106b80 0x7fd3ac19c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:13:56.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:56.587+0000 7fd3b3a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3ac10c8f0 0x7fd3ac19ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:13:56.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:56.587+0000 7fd3b3a00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3ac19d120 con 0x7fd3ac106b80 2024-04-20T22:13:56.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:56.587+0000 7fd3b3a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3ac1a0eb0 con 0x7fd3ac10c8f0 2024-04-20T22:13:56.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:56.587+0000 7fd3b1000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3ac106b80 0x7fd3ac19c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:13:56.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:56.587+0000 7fd3b1000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3ac106b80 0x7fd3ac19c500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37198/0 (socket says 172.21.15.62:37198) 2024-04-20T22:13:56.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:56.587+0000 7fd3b1000700 1 -- 172.21.15.62:0/583414213 learned_addr learned my addr 172.21.15.62:0/583414213 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:13:56.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:56.587+0000 7fd3abe00700 1 --2- 172.21.15.62:0/583414213 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3ac10c8f0 0x7fd3ac19ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:13:56.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:56.587+0000 7fd3b1000700 1 -- 172.21.15.62:0/583414213 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3ac10c8f0 msgr2=0x7fd3ac19ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:13:56.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:56.587+0000 7fd3b1000700 1 --2- 172.21.15.62:0/583414213 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3ac10c8f0 0x7fd3ac19ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:13:56.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:56.587+0000 7fd3b1000700 1 -- 172.21.15.62:0/583414213 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3a00095f0 con 0x7fd3ac106b80 2024-04-20T22:13:56.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:56.588+0000 7fd3b1000700 1 --2- 172.21.15.62:0/583414213 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3ac106b80 0x7fd3ac19c500 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7fd3a000b5c0 tx=0x7fd3a000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:13:56.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:56.588+0000 7fd3a9600700 1 -- 172.21.15.62:0/583414213 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd3a001d070 con 0x7fd3ac106b80 2024-04-20T22:13:56.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:56.588+0000 7fd3a9600700 1 -- 172.21.15.62:0/583414213 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd3a0005e00 con 0x7fd3ac106b80 2024-04-20T22:13:56.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:56.588+0000 7fd3a9600700 1 -- 172.21.15.62:0/583414213 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd3a0017400 con 0x7fd3ac106b80 2024-04-20T22:13:56.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:56.588+0000 7fd3b3a00700 1 -- 172.21.15.62:0/583414213 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3ac1a1130 con 0x7fd3ac106b80 2024-04-20T22:13:56.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:56.588+0000 7fd3b3a00700 1 -- 172.21.15.62:0/583414213 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3ac1a1620 con 0x7fd3ac106b80 2024-04-20T22:13:56.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:56.589+0000 7fd3b3a00700 1 -- 172.21.15.62:0/583414213 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3ac109b20 con 0x7fd3ac106b80 2024-04-20T22:13:56.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:56.591+0000 7fd3a9600700 1 -- 172.21.15.62:0/583414213 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fd3a000f460 con 0x7fd3ac106b80 2024-04-20T22:13:56.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:56.591+0000 7fd3a9600700 1 --2- 172.21.15.62:0/583414213 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd39406ecc0 0x7fd394071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:13:56.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:56.592+0000 7fd3a9600700 1 -- 172.21.15.62:0/583414213 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fd3a0090370 con 0x7fd3ac106b80 2024-04-20T22:13:56.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:56.592+0000 7fd3abe00700 1 --2- 172.21.15.62:0/583414213 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd39406ecc0 0x7fd394071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:13:56.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:56.593+0000 7fd3abe00700 1 --2- 172.21.15.62:0/583414213 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd39406ecc0 0x7fd394071170 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fd398005950 tx=0x7fd3980058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:13:56.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:56.597+0000 7fd3a9600700 1 -- 172.21.15.62:0/583414213 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fd3a0059100 con 0x7fd3ac106b80 2024-04-20T22:13:56.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:13:56 smithi189 ceph-mon[57022]: pgmap v1162: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:13:56.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:56.851+0000 7fd3b3a00700 1 -- 172.21.15.62:0/583414213 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd3ac1a1950 con 0x7fd39406ecc0 2024-04-20T22:13:56.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:56.853+0000 7fd3a9600700 1 -- 172.21.15.62:0/583414213 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fd3ac1a1950 con 0x7fd39406ecc0 2024-04-20T22:13:56.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:56.856+0000 7fd3b3a00700 1 -- 172.21.15.62:0/583414213 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd39406ecc0 msgr2=0x7fd394071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:13:56.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:56.856+0000 7fd3b3a00700 1 --2- 172.21.15.62:0/583414213 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd39406ecc0 0x7fd394071170 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fd398005950 tx=0x7fd3980058e0 comp rx=0 tx=0).stop 2024-04-20T22:13:56.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:56.856+0000 7fd3b3a00700 1 -- 172.21.15.62:0/583414213 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3ac106b80 msgr2=0x7fd3ac19c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:13:56.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:56.856+0000 7fd3b3a00700 1 --2- 172.21.15.62:0/583414213 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3ac106b80 0x7fd3ac19c500 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7fd3a000b5c0 tx=0x7fd3a000be20 comp rx=0 tx=0).stop 2024-04-20T22:13:56.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:56.856+0000 7fd3b3a00700 1 -- 172.21.15.62:0/583414213 shutdown_connections 2024-04-20T22:13:56.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:56.856+0000 7fd3b3a00700 1 --2- 172.21.15.62:0/583414213 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd39406ecc0 0x7fd394071170 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:13:56.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:56.857+0000 7fd3b3a00700 1 --2- 172.21.15.62:0/583414213 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3ac106b80 0x7fd3ac19c500 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:13:56.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:56.857+0000 7fd3b3a00700 1 --2- 172.21.15.62:0/583414213 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3ac10c8f0 0x7fd3ac19ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:13:56.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:56.857+0000 7fd3b3a00700 1 -- 172.21.15.62:0/583414213 >> 172.21.15.62:0/583414213 conn(0x7fd3ac076060 msgr2=0x7fd3ac077aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:13:56.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:56.857+0000 7fd3b3a00700 1 -- 172.21.15.62:0/583414213 shutdown_connections 2024-04-20T22:13:56.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:56.857+0000 7fd3b3a00700 1 -- 172.21.15.62:0/583414213 wait complete. 2024-04-20T22:13:57.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.014+0000 7f6c85c00700 1 -- 172.21.15.62:0/4222920110 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c80101c30 msgr2=0x7f6c801020a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:13:57.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.014+0000 7f6c85c00700 1 --2- 172.21.15.62:0/4222920110 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c80101c30 0x7f6c801020a0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c009910 tx=0x7f6c6c009c20 comp rx=0 tx=0).stop 2024-04-20T22:13:57.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.014+0000 7f6c85c00700 1 -- 172.21.15.62:0/4222920110 shutdown_connections 2024-04-20T22:13:57.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.014+0000 7f6c85c00700 1 --2- 172.21.15.62:0/4222920110 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c80101c30 0x7f6c801020a0 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:13:57.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.015+0000 7f6c85c00700 1 --2- 172.21.15.62:0/4222920110 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c801003d0 0x7f6c801007a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:13:57.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.015+0000 7f6c85c00700 1 -- 172.21.15.62:0/4222920110 >> 172.21.15.62:0/4222920110 conn(0x7f6c800fbbd0 msgr2=0x7f6c800fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:13:57.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.015+0000 7f6c85c00700 1 -- 172.21.15.62:0/4222920110 shutdown_connections 2024-04-20T22:13:57.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.015+0000 7f6c85c00700 1 -- 172.21.15.62:0/4222920110 wait complete. 2024-04-20T22:13:57.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.016+0000 7f6c85c00700 1 Processor -- start 2024-04-20T22:13:57.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.016+0000 7f6c85c00700 1 -- start start 2024-04-20T22:13:57.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.016+0000 7f6c85c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c801003d0 0x7f6c80197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:13:57.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.016+0000 7f6c85c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c80101c30 0x7f6c80198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:13:57.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.016+0000 7f6c85c00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c80198b30 con 0x7f6c80101c30 2024-04-20T22:13:57.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.016+0000 7f6c85c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c8019c8c0 con 0x7f6c801003d0 2024-04-20T22:13:57.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.017+0000 7f6c7f400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c80101c30 0x7f6c80198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:13:57.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.017+0000 7f6c7f400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c80101c30 0x7f6c80198450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37218/0 (socket says 172.21.15.62:37218) 2024-04-20T22:13:57.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.017+0000 7f6c7f400700 1 -- 172.21.15.62:0/4265665591 learned_addr learned my addr 172.21.15.62:0/4265665591 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:13:57.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.017+0000 7f6c7fe00700 1 --2- 172.21.15.62:0/4265665591 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c801003d0 0x7f6c80197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:13:57.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.017+0000 7f6c7f400700 1 -- 172.21.15.62:0/4265665591 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c801003d0 msgr2=0x7f6c80197f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:13:57.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.017+0000 7f6c7f400700 1 --2- 172.21.15.62:0/4265665591 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c801003d0 0x7f6c80197f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:13:57.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.017+0000 7f6c7f400700 1 -- 172.21.15.62:0/4265665591 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c6c0095f0 con 0x7f6c80101c30 2024-04-20T22:13:57.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.017+0000 7f6c7f400700 1 --2- 172.21.15.62:0/4265665591 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c80101c30 0x7f6c80198450 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c006010 tx=0x7f6c6c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:13:57.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.018+0000 7f6c7cc00700 1 -- 172.21.15.62:0/4265665591 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6c6c01d070 con 0x7f6c80101c30 2024-04-20T22:13:57.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.018+0000 7f6c7cc00700 1 -- 172.21.15.62:0/4265665591 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6c6c00f460 con 0x7f6c80101c30 2024-04-20T22:13:57.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.018+0000 7f6c85c00700 1 -- 172.21.15.62:0/4265665591 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c8019cb40 con 0x7f6c80101c30 2024-04-20T22:13:57.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.018+0000 7f6c7cc00700 1 -- 172.21.15.62:0/4265665591 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6c6c017610 con 0x7f6c80101c30 2024-04-20T22:13:57.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.018+0000 7f6c85c00700 1 -- 172.21.15.62:0/4265665591 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c8019d030 con 0x7f6c80101c30 2024-04-20T22:13:57.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.019+0000 7f6c85c00700 1 -- 172.21.15.62:0/4265665591 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c80062380 con 0x7f6c80101c30 2024-04-20T22:13:57.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.020+0000 7f6c7cc00700 1 -- 172.21.15.62:0/4265665591 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f6c6c00f5d0 con 0x7f6c80101c30 2024-04-20T22:13:57.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.021+0000 7f6c7cc00700 1 --2- 172.21.15.62:0/4265665591 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6c6406ecc0 0x7f6c64071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:13:57.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.021+0000 7f6c7cc00700 1 -- 172.21.15.62:0/4265665591 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f6c6c0903e0 con 0x7f6c80101c30 2024-04-20T22:13:57.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.021+0000 7f6c7fe00700 1 --2- 172.21.15.62:0/4265665591 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6c6406ecc0 0x7f6c64071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:13:57.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.022+0000 7f6c7fe00700 1 --2- 172.21.15.62:0/4265665591 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6c6406ecc0 0x7f6c64071170 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f6c70005950 tx=0x7f6c700058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:13:57.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.026+0000 7f6c7cc00700 1 -- 172.21.15.62:0/4265665591 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f6c6c0591b0 con 0x7f6c80101c30 2024-04-20T22:13:57.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.261+0000 7f6c85c00700 1 -- 172.21.15.62:0/4265665591 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f6c8019d360 con 0x7f6c6406ecc0 2024-04-20T22:13:57.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.280+0000 7f6c7cc00700 1 -- 172.21.15.62:0/4265665591 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f6c8019d360 con 0x7f6c6406ecc0 2024-04-20T22:13:57.281 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T22:13:57.281 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (39m) 36s ago 53m 14.6M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T22:13:57.281 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (54m) 36s ago 54m 11.6M - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T22:13:57.281 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (52m) 39s ago 52m 11.2M - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T22:13:57.281 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (35m) 36s ago 53m 6660k - 19.0.0-3133-ga8c84f59 76bc6b91b726 2825d5ba11c6 2024-04-20T22:13:57.281 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (35m) 39s ago 52m 6665k - 19.0.0-3133-ga8c84f59 76bc6b91b726 20f746fa1bba 2024-04-20T22:13:57.281 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (39m) 36s ago 53m 75.5M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T22:13:57.281 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (31m) 36s ago 46m 22.9M - 19.0.0-3133-ga8c84f59 76bc6b91b726 de80bc90a683 2024-04-20T22:13:57.281 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (31m) 36s ago 46m 23.0M - 19.0.0-3133-ga8c84f59 76bc6b91b726 cd793b7ee98e 2024-04-20T22:13:57.282 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (16m) 39s ago 46m 18.5M - 19.0.0-3133-ga8c84f59 76bc6b91b726 0e7b0bc4d1ab 2024-04-20T22:13:57.282 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (45s) 39s ago 46m 11.4M - 19.0.0-3133-ga8c84f59 76bc6b91b726 0af732ac9e06 2024-04-20T22:13:57.282 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (42m) 36s ago 55m 517M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T22:13:57.282 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (41m) 39s ago 52m 460M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T22:13:57.282 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (37m) 36s ago 55m 86.3M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 019eb1479485 2024-04-20T22:13:57.282 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (36m) 39s ago 52m 58.0M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 55aac560aab6 2024-04-20T22:13:57.282 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (40m) 36s ago 53m 8766k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T22:13:57.282 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (40m) 39s ago 52m 8711k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T22:13:57.282 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (35m) 36s ago 51m 91.3M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 ee2bb901f26e 2024-04-20T22:13:57.283 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (34m) 36s ago 50m 94.9M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 9075152d4162 2024-04-20T22:13:57.283 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (34m) 36s ago 50m 78.5M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 2d4f3e38dc0e 2024-04-20T22:13:57.283 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (33m) 39s ago 49m 94.1M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 8f97b17ae0eb 2024-04-20T22:13:57.283 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (32m) 39s ago 48m 96.1M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 653d8079967d 2024-04-20T22:13:57.283 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (32m) 39s ago 48m 90.9M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 596dded14c12 2024-04-20T22:13:57.283 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (40m) 36s ago 53m 90.4M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T22:13:57.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.283+0000 7f6c85c00700 1 -- 172.21.15.62:0/4265665591 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6c6406ecc0 msgr2=0x7f6c64071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:13:57.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.283+0000 7f6c85c00700 1 --2- 172.21.15.62:0/4265665591 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6c6406ecc0 0x7f6c64071170 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f6c70005950 tx=0x7f6c700058e0 comp rx=0 tx=0).stop 2024-04-20T22:13:57.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.283+0000 7f6c85c00700 1 -- 172.21.15.62:0/4265665591 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c80101c30 msgr2=0x7f6c80198450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:13:57.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.283+0000 7f6c85c00700 1 --2- 172.21.15.62:0/4265665591 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c80101c30 0x7f6c80198450 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c006010 tx=0x7f6c6c00be20 comp rx=0 tx=0).stop 2024-04-20T22:13:57.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.283+0000 7f6c85c00700 1 -- 172.21.15.62:0/4265665591 shutdown_connections 2024-04-20T22:13:57.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.283+0000 7f6c85c00700 1 --2- 172.21.15.62:0/4265665591 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6c6406ecc0 0x7f6c64071170 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:13:57.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.283+0000 7f6c85c00700 1 --2- 172.21.15.62:0/4265665591 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c801003d0 0x7f6c80197f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:13:57.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.283+0000 7f6c85c00700 1 --2- 172.21.15.62:0/4265665591 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c80101c30 0x7f6c80198450 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:13:57.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.283+0000 7f6c85c00700 1 -- 172.21.15.62:0/4265665591 >> 172.21.15.62:0/4265665591 conn(0x7f6c800fbbd0 msgr2=0x7f6c80073000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:13:57.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.283+0000 7f6c85c00700 1 -- 172.21.15.62:0/4265665591 shutdown_connections 2024-04-20T22:13:57.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.284+0000 7f6c85c00700 1 -- 172.21.15.62:0/4265665591 wait complete. 2024-04-20T22:13:57.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.444+0000 7f77b3c00700 1 -- 172.21.15.62:0/2710834579 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f77ac1026a0 msgr2=0x7f77ac102a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:13:57.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.444+0000 7f77b3c00700 1 --2- 172.21.15.62:0/2710834579 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f77ac1026a0 0x7f77ac102a70 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f77a0009910 tx=0x7f77a0009c20 comp rx=0 tx=0).stop 2024-04-20T22:13:57.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.444+0000 7f77b3c00700 1 -- 172.21.15.62:0/2710834579 shutdown_connections 2024-04-20T22:13:57.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.444+0000 7f77b3c00700 1 --2- 172.21.15.62:0/2710834579 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f77ac103040 0x7f77ac107010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:13:57.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.444+0000 7f77b3c00700 1 --2- 172.21.15.62:0/2710834579 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f77ac1026a0 0x7f77ac102a70 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:13:57.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.444+0000 7f77b3c00700 1 -- 172.21.15.62:0/2710834579 >> 172.21.15.62:0/2710834579 conn(0x7f77ac0fdf40 msgr2=0x7f77ac100350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:13:57.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.445+0000 7f77b3c00700 1 -- 172.21.15.62:0/2710834579 shutdown_connections 2024-04-20T22:13:57.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.445+0000 7f77b3c00700 1 -- 172.21.15.62:0/2710834579 wait complete. 2024-04-20T22:13:57.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.446+0000 7f77b3c00700 1 Processor -- start 2024-04-20T22:13:57.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.446+0000 7f77b3c00700 1 -- start start 2024-04-20T22:13:57.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.447+0000 7f77b3c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f77ac1026a0 0x7f77ac198200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:13:57.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.447+0000 7f77b3c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f77ac103040 0x7f77ac198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:13:57.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.447+0000 7f77b3c00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77ac198e20 con 0x7f77ac103040 2024-04-20T22:13:57.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.447+0000 7f77b3c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77ac19cbb0 con 0x7f77ac1026a0 2024-04-20T22:13:57.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.447+0000 7f77abe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f77ac103040 0x7f77ac198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:13:57.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.447+0000 7f77abe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f77ac103040 0x7f77ac198740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37232/0 (socket says 172.21.15.62:37232) 2024-04-20T22:13:57.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.447+0000 7f77abe00700 1 -- 172.21.15.62:0/4185661073 learned_addr learned my addr 172.21.15.62:0/4185661073 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:13:57.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.447+0000 7f77b1200700 1 --2- 172.21.15.62:0/4185661073 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f77ac1026a0 0x7f77ac198200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:13:57.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.448+0000 7f77abe00700 1 -- 172.21.15.62:0/4185661073 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f77ac1026a0 msgr2=0x7f77ac198200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:13:57.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.448+0000 7f77abe00700 1 --2- 172.21.15.62:0/4185661073 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f77ac1026a0 0x7f77ac198200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:13:57.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.448+0000 7f77abe00700 1 -- 172.21.15.62:0/4185661073 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77a00095f0 con 0x7f77ac103040 2024-04-20T22:13:57.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.448+0000 7f77b1200700 1 --2- 172.21.15.62:0/4185661073 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f77ac1026a0 0x7f77ac198200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-20T22:13:57.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.448+0000 7f77abe00700 1 --2- 172.21.15.62:0/4185661073 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f77ac103040 0x7f77ac198740 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f779800b480 tx=0x7f779800b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:13:57.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.448+0000 7f77a9600700 1 -- 172.21.15.62:0/4185661073 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f77980118c0 con 0x7f77ac103040 2024-04-20T22:13:57.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.448+0000 7f77a9600700 1 -- 172.21.15.62:0/4185661073 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7798010380 con 0x7f77ac103040 2024-04-20T22:13:57.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.448+0000 7f77a9600700 1 -- 172.21.15.62:0/4185661073 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f779800f4f0 con 0x7f77ac103040 2024-04-20T22:13:57.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.448+0000 7f77b3c00700 1 -- 172.21.15.62:0/4185661073 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77ac19ce90 con 0x7f77ac103040 2024-04-20T22:13:57.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.448+0000 7f77b3c00700 1 -- 172.21.15.62:0/4185661073 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77ac19d3e0 con 0x7f77ac103040 2024-04-20T22:13:57.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.450+0000 7f77b3c00700 1 -- 172.21.15.62:0/4185661073 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77ac10a9b0 con 0x7f77ac103040 2024-04-20T22:13:57.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.451+0000 7f77a9600700 1 -- 172.21.15.62:0/4185661073 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f77980104f0 con 0x7f77ac103040 2024-04-20T22:13:57.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.451+0000 7f77a9600700 1 --2- 172.21.15.62:0/4185661073 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f779406ecc0 0x7f7794071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:13:57.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.452+0000 7f77a9600700 1 -- 172.21.15.62:0/4185661073 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f779808ea70 con 0x7f77ac103040 2024-04-20T22:13:57.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.452+0000 7f77b1200700 1 --2- 172.21.15.62:0/4185661073 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f779406ecc0 0x7f7794071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:13:57.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.453+0000 7f77b1200700 1 --2- 172.21.15.62:0/4185661073 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f779406ecc0 0x7f7794071170 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f77a0005100 tx=0x7f77a001a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:13:57.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.457+0000 7f77a9600700 1 -- 172.21.15.62:0/4185661073 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f7798057800 con 0x7f77ac103040 2024-04-20T22:13:57.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.777+0000 7f77b3c00700 1 -- 172.21.15.62:0/4185661073 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f77ac19d6b0 con 0x7f77ac103040 2024-04-20T22:13:57.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.779+0000 7f77a9600700 1 -- 172.21.15.62:0/4185661073 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f779805aec0 con 0x7f77ac103040 2024-04-20T22:13:57.779 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:13:57.779 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T22:13:57.779 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:13:57.779 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:13:57.779 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T22:13:57.779 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:13:57.780 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:13:57.780 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T22:13:57.780 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 6 2024-04-20T22:13:57.780 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:13:57.780 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T22:13:57.780 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 4 2024-04-20T22:13:57.780 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:13:57.780 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T22:13:57.780 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 14 2024-04-20T22:13:57.780 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T22:13:57.780 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:13:57.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.782+0000 7f77b3c00700 1 -- 172.21.15.62:0/4185661073 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f779406ecc0 msgr2=0x7f7794071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:13:57.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.782+0000 7f77b3c00700 1 --2- 172.21.15.62:0/4185661073 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f779406ecc0 0x7f7794071170 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f77a0005100 tx=0x7f77a001a040 comp rx=0 tx=0).stop 2024-04-20T22:13:57.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.782+0000 7f77b3c00700 1 -- 172.21.15.62:0/4185661073 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f77ac103040 msgr2=0x7f77ac198740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:13:57.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.782+0000 7f77b3c00700 1 --2- 172.21.15.62:0/4185661073 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f77ac103040 0x7f77ac198740 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f779800b480 tx=0x7f779800b840 comp rx=0 tx=0).stop 2024-04-20T22:13:57.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.782+0000 7f77b3c00700 1 -- 172.21.15.62:0/4185661073 shutdown_connections 2024-04-20T22:13:57.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.782+0000 7f77b3c00700 1 --2- 172.21.15.62:0/4185661073 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f779406ecc0 0x7f7794071170 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:13:57.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.782+0000 7f77b3c00700 1 --2- 172.21.15.62:0/4185661073 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f77ac1026a0 0x7f77ac198200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:13:57.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.782+0000 7f77b3c00700 1 --2- 172.21.15.62:0/4185661073 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f77ac103040 0x7f77ac198740 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:13:57.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.782+0000 7f77b3c00700 1 -- 172.21.15.62:0/4185661073 >> 172.21.15.62:0/4185661073 conn(0x7f77ac0fdf40 msgr2=0x7f77ac0ff8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:13:57.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.782+0000 7f77b3c00700 1 -- 172.21.15.62:0/4185661073 shutdown_connections 2024-04-20T22:13:57.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.782+0000 7f77b3c00700 1 -- 172.21.15.62:0/4185661073 wait complete. 2024-04-20T22:13:57.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.934+0000 7f5bed400700 1 -- 172.21.15.62:0/236127213 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5be8100e80 msgr2=0x7f5be8104e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:13:57.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.934+0000 7f5bed400700 1 --2- 172.21.15.62:0/236127213 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5be8100e80 0x7f5be8104e50 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f5bd0009910 tx=0x7f5bd0009c20 comp rx=0 tx=0).stop 2024-04-20T22:13:57.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.934+0000 7f5bed400700 1 -- 172.21.15.62:0/236127213 shutdown_connections 2024-04-20T22:13:57.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.934+0000 7f5bed400700 1 --2- 172.21.15.62:0/236127213 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5be8100e80 0x7f5be8104e50 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:13:57.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.934+0000 7f5bed400700 1 --2- 172.21.15.62:0/236127213 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5be81004e0 0x7f5be81008b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:13:57.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.934+0000 7f5bed400700 1 -- 172.21.15.62:0/236127213 >> 172.21.15.62:0/236127213 conn(0x7f5be80fbda0 msgr2=0x7f5be80fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:13:57.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.934+0000 7f5bed400700 1 -- 172.21.15.62:0/236127213 shutdown_connections 2024-04-20T22:13:57.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.934+0000 7f5bed400700 1 -- 172.21.15.62:0/236127213 wait complete. 2024-04-20T22:13:57.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.936+0000 7f5bed400700 1 Processor -- start 2024-04-20T22:13:57.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.936+0000 7f5bed400700 1 -- start start 2024-04-20T22:13:57.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.936+0000 7f5bed400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5be81004e0 0x7f5be8198130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:13:57.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.936+0000 7f5bed400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5be8100e80 0x7f5be8198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:13:57.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.936+0000 7f5bed400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5be8198d50 con 0x7f5be81004e0 2024-04-20T22:13:57.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.936+0000 7f5bed400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5be819cae0 con 0x7f5be8100e80 2024-04-20T22:13:57.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.936+0000 7f5be6800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5be81004e0 0x7f5be8198130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:13:57.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.936+0000 7f5be6800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5be81004e0 0x7f5be8198130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37248/0 (socket says 172.21.15.62:37248) 2024-04-20T22:13:57.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.936+0000 7f5be6800700 1 -- 172.21.15.62:0/2367117367 learned_addr learned my addr 172.21.15.62:0/2367117367 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:13:57.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.937+0000 7f5bdfe00700 1 --2- 172.21.15.62:0/2367117367 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5be8100e80 0x7f5be8198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:13:57.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.937+0000 7f5be6800700 1 -- 172.21.15.62:0/2367117367 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5be8100e80 msgr2=0x7f5be8198670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:13:57.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.937+0000 7f5be6800700 1 --2- 172.21.15.62:0/2367117367 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5be8100e80 0x7f5be8198670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:13:57.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.937+0000 7f5be6800700 1 -- 172.21.15.62:0/2367117367 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bd00095f0 con 0x7f5be81004e0 2024-04-20T22:13:57.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.937+0000 7f5be6800700 1 --2- 172.21.15.62:0/2367117367 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5be81004e0 0x7f5be8198130 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f5bd800c8a0 tx=0x7f5bd800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:13:57.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.937+0000 7f5bdf400700 1 -- 172.21.15.62:0/2367117367 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5bd800cea0 con 0x7f5be81004e0 2024-04-20T22:13:57.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.938+0000 7f5bdf400700 1 -- 172.21.15.62:0/2367117367 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5bd8004830 con 0x7f5be81004e0 2024-04-20T22:13:57.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.938+0000 7f5bdf400700 1 -- 172.21.15.62:0/2367117367 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5bd80055b0 con 0x7f5be81004e0 2024-04-20T22:13:57.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.938+0000 7f5bed400700 1 -- 172.21.15.62:0/2367117367 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5be819cdc0 con 0x7f5be81004e0 2024-04-20T22:13:57.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.938+0000 7f5bed400700 1 -- 172.21.15.62:0/2367117367 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5be819d2c0 con 0x7f5be81004e0 2024-04-20T22:13:57.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.939+0000 7f5bed400700 1 -- 172.21.15.62:0/2367117367 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5be81045d0 con 0x7f5be81004e0 2024-04-20T22:13:57.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.940+0000 7f5bdf400700 1 -- 172.21.15.62:0/2367117367 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f5bd80049a0 con 0x7f5be81004e0 2024-04-20T22:13:57.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.941+0000 7f5bdf400700 1 --2- 172.21.15.62:0/2367117367 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5bcc06ecc0 0x7f5bcc071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:13:57.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.941+0000 7f5bdf400700 1 -- 172.21.15.62:0/2367117367 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f5bd808ebf0 con 0x7f5be81004e0 2024-04-20T22:13:57.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.941+0000 7f5bdfe00700 1 --2- 172.21.15.62:0/2367117367 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5bcc06ecc0 0x7f5bcc071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:13:57.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.942+0000 7f5bdfe00700 1 --2- 172.21.15.62:0/2367117367 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5bcc06ecc0 0x7f5bcc071170 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f5bd0000c00 tx=0x7f5bd0005fd0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:13:57.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:57.946+0000 7f5bdf400700 1 -- 172.21.15.62:0/2367117367 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f5bd8057900 con 0x7f5be81004e0 2024-04-20T22:13:58.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:58.227+0000 7f5bed400700 1 -- 172.21.15.62:0/2367117367 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f5be819d590 con 0x7f5be81004e0 2024-04-20T22:13:58.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:58.229+0000 7f5bdf400700 1 -- 172.21.15.62:0/2367117367 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 29 v29) v1 ==== 76+0+1887 (secure 0 0 0) 0x7f5bd801d770 con 0x7f5be81004e0 2024-04-20T22:13:58.229 INFO:teuthology.orchestra.run.smithi062.stdout:e29 2024-04-20T22:13:58.229 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T22:13:58.229 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:13:58.230 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T22:13:58.230 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:13:58.230 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T22:13:58.230 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T22:13:58.230 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 20 2024-04-20T22:13:58.230 INFO:teuthology.orchestra.run.smithi062.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-20T22:13:58.230 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T22:13:58.230 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T21:42:01.559061+0000 2024-04-20T22:13:58.230 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T22:13:58.230 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T22:13:58.230 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T22:13:58.230 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T22:13:58.230 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T22:13:58.231 INFO:teuthology.orchestra.run.smithi062.stdout:max_xattr_size 65536 2024-04-20T22:13:58.231 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T22:13:58.231 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T22:13:58.231 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 81 2024-04-20T22:13:58.231 INFO:teuthology.orchestra.run.smithi062.stdout:snap_rstat 1 2024-04-20T22:13:58.231 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:13:58.231 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 2 2024-04-20T22:13:58.231 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1 2024-04-20T22:13:58.231 INFO:teuthology.orchestra.run.smithi062.stdout:up {} 2024-04-20T22:13:58.231 INFO:teuthology.orchestra.run.smithi062.stdout:failed 0,1 2024-04-20T22:13:58.231 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T22:13:58.231 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T22:13:58.231 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T22:13:58.232 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T22:13:58.232 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data enabled 2024-04-20T22:13:58.232 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T22:13:58.232 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T22:13:58.232 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 1 2024-04-20T22:13:58.232 INFO:teuthology.orchestra.run.smithi062.stdout:qdb_cluster leader: 0 members: 2024-04-20T22:13:58.232 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:13:58.232 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:13:58.232 INFO:teuthology.orchestra.run.smithi062.stdout:Standby daemons: 2024-04-20T22:13:58.232 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:13:58.232 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ryqblb{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6826/4113916205,v1:172.21.15.62:6827/4113916205] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:13:58.232 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ozizwi{-1:34476} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6828/3801975165,v1:172.21.15.62:6829/3801975165] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:13:58.232 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.dvimer{-1:44727} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6824/3576699232,v1:172.21.15.189:6825/3576699232] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:13:58.232 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.xbgspw{-1:45075} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6826/212905792,v1:172.21.15.189:6827/212905792] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:13:58.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:58.232+0000 7f5bed400700 1 -- 172.21.15.62:0/2367117367 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5bcc06ecc0 msgr2=0x7f5bcc071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:13:58.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:58.232+0000 7f5bed400700 1 --2- 172.21.15.62:0/2367117367 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5bcc06ecc0 0x7f5bcc071170 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f5bd0000c00 tx=0x7f5bd0005fd0 comp rx=0 tx=0).stop 2024-04-20T22:13:58.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:58.232+0000 7f5bed400700 1 -- 172.21.15.62:0/2367117367 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5be81004e0 msgr2=0x7f5be8198130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:13:58.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:58.232+0000 7f5bed400700 1 --2- 172.21.15.62:0/2367117367 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5be81004e0 0x7f5be8198130 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f5bd800c8a0 tx=0x7f5bd800cc60 comp rx=0 tx=0).stop 2024-04-20T22:13:58.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:58.232+0000 7f5bed400700 1 -- 172.21.15.62:0/2367117367 shutdown_connections 2024-04-20T22:13:58.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:58.232+0000 7f5bed400700 1 --2- 172.21.15.62:0/2367117367 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5bcc06ecc0 0x7f5bcc071170 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:13:58.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:58.232+0000 7f5bed400700 1 --2- 172.21.15.62:0/2367117367 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5be81004e0 0x7f5be8198130 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:13:58.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:58.232+0000 7f5bed400700 1 --2- 172.21.15.62:0/2367117367 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5be8100e80 0x7f5be8198670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:13:58.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:58.232+0000 7f5bed400700 1 -- 172.21.15.62:0/2367117367 >> 172.21.15.62:0/2367117367 conn(0x7f5be80fbda0 msgr2=0x7f5be80fe160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:13:58.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:58.232+0000 7f5bed400700 1 -- 172.21.15.62:0/2367117367 shutdown_connections 2024-04-20T22:13:58.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:58.232+0000 7f5bed400700 1 -- 172.21.15.62:0/2367117367 wait complete. 2024-04-20T22:13:58.235 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 29 2024-04-20T22:13:58.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:58.387+0000 7ffb84e00700 1 -- 172.21.15.62:0/1799129380 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffb80073090 msgr2=0x7ffb80073460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:13:58.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:58.387+0000 7ffb84e00700 1 --2- 172.21.15.62:0/1799129380 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffb80073090 0x7ffb80073460 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7ffb70009940 tx=0x7ffb70009c50 comp rx=0 tx=0).stop 2024-04-20T22:13:58.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:58.387+0000 7ffb84e00700 1 -- 172.21.15.62:0/1799129380 shutdown_connections 2024-04-20T22:13:58.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:58.387+0000 7ffb84e00700 1 --2- 172.21.15.62:0/1799129380 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb800739a0 0x7ffb8010c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:13:58.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:58.387+0000 7ffb84e00700 1 --2- 172.21.15.62:0/1799129380 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffb80073090 0x7ffb80073460 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:13:58.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:58.387+0000 7ffb84e00700 1 -- 172.21.15.62:0/1799129380 >> 172.21.15.62:0/1799129380 conn(0x7ffb800fbda0 msgr2=0x7ffb800fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:13:58.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:58.387+0000 7ffb84e00700 1 -- 172.21.15.62:0/1799129380 shutdown_connections 2024-04-20T22:13:58.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:58.387+0000 7ffb84e00700 1 -- 172.21.15.62:0/1799129380 wait complete. 2024-04-20T22:13:58.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:58.388+0000 7ffb84e00700 1 Processor -- start 2024-04-20T22:13:58.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:58.389+0000 7ffb84e00700 1 -- start start 2024-04-20T22:13:58.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:58.389+0000 7ffb84e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffb80073090 0x7ffb80198180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:13:58.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:58.389+0000 7ffb84e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb800739a0 0x7ffb801986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:13:58.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:58.389+0000 7ffb84e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb80198da0 con 0x7ffb80073090 2024-04-20T22:13:58.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:58.389+0000 7ffb84e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb8019cb30 con 0x7ffb800739a0 2024-04-20T22:13:58.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:58.389+0000 7ffb7de00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffb80073090 0x7ffb80198180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:13:58.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:58.389+0000 7ffb7de00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffb80073090 0x7ffb80198180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37270/0 (socket says 172.21.15.62:37270) 2024-04-20T22:13:58.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:58.389+0000 7ffb7de00700 1 -- 172.21.15.62:0/3390091566 learned_addr learned my addr 172.21.15.62:0/3390091566 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:13:58.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:58.390+0000 7ffb77e00700 1 --2- 172.21.15.62:0/3390091566 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb800739a0 0x7ffb801986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:13:58.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:58.390+0000 7ffb7de00700 1 -- 172.21.15.62:0/3390091566 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb800739a0 msgr2=0x7ffb801986c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:13:58.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:58.390+0000 7ffb7de00700 1 --2- 172.21.15.62:0/3390091566 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb800739a0 0x7ffb801986c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:13:58.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:58.390+0000 7ffb7de00700 1 -- 172.21.15.62:0/3390091566 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb700095f0 con 0x7ffb80073090 2024-04-20T22:13:58.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:58.390+0000 7ffb7de00700 1 --2- 172.21.15.62:0/3390091566 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffb80073090 0x7ffb80198180 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7ffb70005620 tx=0x7ffb70005700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:13:58.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:58.391+0000 7ffb76a00700 1 -- 172.21.15.62:0/3390091566 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffb7001d070 con 0x7ffb80073090 2024-04-20T22:13:58.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:58.391+0000 7ffb76a00700 1 -- 172.21.15.62:0/3390091566 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ffb7000f460 con 0x7ffb80073090 2024-04-20T22:13:58.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:58.391+0000 7ffb84e00700 1 -- 172.21.15.62:0/3390091566 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb8019cdb0 con 0x7ffb80073090 2024-04-20T22:13:58.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:58.391+0000 7ffb76a00700 1 -- 172.21.15.62:0/3390091566 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffb70021640 con 0x7ffb80073090 2024-04-20T22:13:58.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:58.391+0000 7ffb84e00700 1 -- 172.21.15.62:0/3390091566 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb8019d2a0 con 0x7ffb80073090 2024-04-20T22:13:58.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:58.392+0000 7ffb84e00700 1 -- 172.21.15.62:0/3390091566 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb80109ce0 con 0x7ffb80073090 2024-04-20T22:13:58.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:58.394+0000 7ffb76a00700 1 -- 172.21.15.62:0/3390091566 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ffb7000fa40 con 0x7ffb80073090 2024-04-20T22:13:58.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:58.395+0000 7ffb76a00700 1 --2- 172.21.15.62:0/3390091566 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ffb6406ecc0 0x7ffb64071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:13:58.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:58.395+0000 7ffb76a00700 1 -- 172.21.15.62:0/3390091566 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7ffb70090280 con 0x7ffb80073090 2024-04-20T22:13:58.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:58.395+0000 7ffb77e00700 1 --2- 172.21.15.62:0/3390091566 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ffb6406ecc0 0x7ffb64071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:13:58.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:58.396+0000 7ffb77e00700 1 --2- 172.21.15.62:0/3390091566 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ffb6406ecc0 0x7ffb64071170 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7ffb68005fd0 tx=0x7ffb6800a560 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:13:58.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:58.400+0000 7ffb76a00700 1 -- 172.21.15.62:0/3390091566 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7ffb70058f90 con 0x7ffb80073090 2024-04-20T22:13:58.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:58.649+0000 7ffb84e00700 1 -- 172.21.15.62:0/3390091566 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ffb80199610 con 0x7ffb6406ecc0 2024-04-20T22:13:58.649 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:13:58 smithi062 ceph-mon[76322]: from='client.36050 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:13:58.649 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:13:58 smithi062 ceph-mon[76322]: from='client.36054 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:13:58.649 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:13:58 smithi062 ceph-mon[76322]: pgmap v1163: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:13:58.649 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:13:58 smithi062 ceph-mon[76322]: from='client.36058 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:13:58.649 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:13:58 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/4185661073' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:13:58.650 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:13:58 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/2367117367' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:13:58.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:58.651+0000 7ffb76a00700 1 -- 172.21.15.62:0/3390091566 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7ffb80199610 con 0x7ffb6406ecc0 2024-04-20T22:13:58.651 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:13:58.651 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T22:13:58.651 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T22:13:58.651 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-20T22:13:58.651 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T22:13:58.651 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr", 2024-04-20T22:13:58.651 INFO:teuthology.orchestra.run.smithi062.stdout: "osd", 2024-04-20T22:13:58.651 INFO:teuthology.orchestra.run.smithi062.stdout: "mds", 2024-04-20T22:13:58.651 INFO:teuthology.orchestra.run.smithi062.stdout: "crash", 2024-04-20T22:13:58.652 INFO:teuthology.orchestra.run.smithi062.stdout: "mon" 2024-04-20T22:13:58.652 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T22:13:58.652 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "16/23 daemons upgraded", 2024-04-20T22:13:58.652 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading mds daemons", 2024-04-20T22:13:58.652 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T22:13:58.652 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:13:58.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:58.654+0000 7ffb84e00700 1 -- 172.21.15.62:0/3390091566 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ffb6406ecc0 msgr2=0x7ffb64071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:13:58.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:58.654+0000 7ffb84e00700 1 --2- 172.21.15.62:0/3390091566 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ffb6406ecc0 0x7ffb64071170 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7ffb68005fd0 tx=0x7ffb6800a560 comp rx=0 tx=0).stop 2024-04-20T22:13:58.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:58.654+0000 7ffb84e00700 1 -- 172.21.15.62:0/3390091566 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffb80073090 msgr2=0x7ffb80198180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:13:58.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:58.654+0000 7ffb84e00700 1 --2- 172.21.15.62:0/3390091566 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffb80073090 0x7ffb80198180 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7ffb70005620 tx=0x7ffb70005700 comp rx=0 tx=0).stop 2024-04-20T22:13:58.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:58.655+0000 7ffb84e00700 1 -- 172.21.15.62:0/3390091566 shutdown_connections 2024-04-20T22:13:58.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:58.655+0000 7ffb84e00700 1 --2- 172.21.15.62:0/3390091566 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ffb6406ecc0 0x7ffb64071170 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:13:58.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:58.655+0000 7ffb84e00700 1 --2- 172.21.15.62:0/3390091566 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffb80073090 0x7ffb80198180 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:13:58.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:58.655+0000 7ffb84e00700 1 --2- 172.21.15.62:0/3390091566 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb800739a0 0x7ffb801986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:13:58.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:58.655+0000 7ffb84e00700 1 -- 172.21.15.62:0/3390091566 >> 172.21.15.62:0/3390091566 conn(0x7ffb800fbda0 msgr2=0x7ffb80106e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:13:58.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:58.655+0000 7ffb84e00700 1 -- 172.21.15.62:0/3390091566 shutdown_connections 2024-04-20T22:13:58.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:58.655+0000 7ffb84e00700 1 -- 172.21.15.62:0/3390091566 wait complete. 2024-04-20T22:13:58.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:13:58 smithi189 ceph-mon[57022]: from='client.36050 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:13:58.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:13:58 smithi189 ceph-mon[57022]: from='client.36054 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:13:58.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:13:58 smithi189 ceph-mon[57022]: pgmap v1163: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:13:58.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:13:58 smithi189 ceph-mon[57022]: from='client.36058 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:13:58.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:13:58 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/4185661073' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:13:58.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:13:58 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/2367117367' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:13:58.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:58.812+0000 7f19b8200700 1 -- 172.21.15.62:0/213398425 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f19b0100e80 msgr2=0x7f19b0104e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:13:58.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:58.812+0000 7f19b8200700 1 --2- 172.21.15.62:0/213398425 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f19b0100e80 0x7f19b0104e50 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f199c009910 tx=0x7f199c009c20 comp rx=0 tx=0).stop 2024-04-20T22:13:58.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:58.812+0000 7f19b8200700 1 -- 172.21.15.62:0/213398425 shutdown_connections 2024-04-20T22:13:58.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:58.812+0000 7f19b8200700 1 --2- 172.21.15.62:0/213398425 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f19b0100e80 0x7f19b0104e50 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:13:58.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:58.812+0000 7f19b8200700 1 --2- 172.21.15.62:0/213398425 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19b01004e0 0x7f19b01008b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:13:58.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:58.812+0000 7f19b8200700 1 -- 172.21.15.62:0/213398425 >> 172.21.15.62:0/213398425 conn(0x7f19b00fbda0 msgr2=0x7f19b00fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:13:58.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:58.813+0000 7f19b8200700 1 -- 172.21.15.62:0/213398425 shutdown_connections 2024-04-20T22:13:58.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:58.813+0000 7f19b8200700 1 -- 172.21.15.62:0/213398425 wait complete. 2024-04-20T22:13:58.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:58.814+0000 7f19b8200700 1 Processor -- start 2024-04-20T22:13:58.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:58.814+0000 7f19b8200700 1 -- start start 2024-04-20T22:13:58.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:58.815+0000 7f19b8200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19b01004e0 0x7f19b0073050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:13:58.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:58.815+0000 7f19b8200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f19b0100e80 0x7f19b0073590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:13:58.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:58.815+0000 7f19b8200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19b0073ad0 con 0x7f19b0100e80 2024-04-20T22:13:58.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:58.815+0000 7f19b8200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19b0073c10 con 0x7f19b01004e0 2024-04-20T22:13:58.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:58.815+0000 7f19b4e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f19b0100e80 0x7f19b0073590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:13:58.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:58.815+0000 7f19b4e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f19b0100e80 0x7f19b0073590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37292/0 (socket says 172.21.15.62:37292) 2024-04-20T22:13:58.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:58.815+0000 7f19b4e00700 1 -- 172.21.15.62:0/1932758962 learned_addr learned my addr 172.21.15.62:0/1932758962 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:13:58.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:58.816+0000 7f19b5800700 1 --2- 172.21.15.62:0/1932758962 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19b01004e0 0x7f19b0073050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:13:58.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:58.816+0000 7f19b4e00700 1 -- 172.21.15.62:0/1932758962 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19b01004e0 msgr2=0x7f19b0073050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:13:58.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:58.816+0000 7f19b4e00700 1 --2- 172.21.15.62:0/1932758962 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19b01004e0 0x7f19b0073050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:13:58.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:58.816+0000 7f19b4e00700 1 -- 172.21.15.62:0/1932758962 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f199c0095f0 con 0x7f19b0100e80 2024-04-20T22:13:58.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:58.816+0000 7f19b4e00700 1 --2- 172.21.15.62:0/1932758962 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f19b0100e80 0x7f19b0073590 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f199c00bda0 tx=0x7f199c00be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:13:58.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:58.817+0000 7f19ae000700 1 -- 172.21.15.62:0/1932758962 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f199c01d070 con 0x7f19b0100e80 2024-04-20T22:13:58.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:58.817+0000 7f19ae000700 1 -- 172.21.15.62:0/1932758962 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f199c00f460 con 0x7f19b0100e80 2024-04-20T22:13:58.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:58.817+0000 7f19b8200700 1 -- 172.21.15.62:0/1932758962 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19b0073e90 con 0x7f19b0100e80 2024-04-20T22:13:58.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:58.817+0000 7f19ae000700 1 -- 172.21.15.62:0/1932758962 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f199c017610 con 0x7f19b0100e80 2024-04-20T22:13:58.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:58.817+0000 7f19b8200700 1 -- 172.21.15.62:0/1932758962 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19b01a2430 con 0x7f19b0100e80 2024-04-20T22:13:58.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:58.818+0000 7f19b8200700 1 -- 172.21.15.62:0/1932758962 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19b0108850 con 0x7f19b0100e80 2024-04-20T22:13:58.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:58.819+0000 7f19ae000700 1 -- 172.21.15.62:0/1932758962 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f199c00fa40 con 0x7f19b0100e80 2024-04-20T22:13:58.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:58.821+0000 7f19ae000700 1 --2- 172.21.15.62:0/1932758962 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f199806ecc0 0x7f1998071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:13:58.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:58.821+0000 7f19ae000700 1 -- 172.21.15.62:0/1932758962 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f199c0902c0 con 0x7f19b0100e80 2024-04-20T22:13:58.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:58.821+0000 7f19b5800700 1 --2- 172.21.15.62:0/1932758962 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f199806ecc0 0x7f1998071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:13:58.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:58.822+0000 7f19b5800700 1 --2- 172.21.15.62:0/1932758962 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f199806ecc0 0x7f1998071170 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f19a4005950 tx=0x7f19a40058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:13:58.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:58.825+0000 7f19ae000700 1 -- 172.21.15.62:0/1932758962 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f199c059050 con 0x7f19b0100e80 2024-04-20T22:13:59.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:59.155+0000 7f19b8200700 1 -- 172.21.15.62:0/1932758962 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f19b004f2a0 con 0x7f19b0100e80 2024-04-20T22:13:59.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:59.156+0000 7f19ae000700 1 -- 172.21.15.62:0/1932758962 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7f199c05c710 con 0x7f19b0100e80 2024-04-20T22:13:59.156 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-20T22:13:59.156 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-20T22:13:59.157 INFO:teuthology.orchestra.run.smithi062.stdout: host smithi189 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi189/config 2024-04-20T22:13:59.157 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-20T22:13:59.157 INFO:teuthology.orchestra.run.smithi062.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi189 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp11670v2q:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:13:59.157 INFO:teuthology.orchestra.run.smithi062.stdout:Traceback (most recent call last): 2024-04-20T22:13:59.157 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-20T22:13:59.157 INFO:teuthology.orchestra.run.smithi062.stdout: return _run_code(code, main_globals, None, 2024-04-20T22:13:59.157 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-20T22:13:59.157 INFO:teuthology.orchestra.run.smithi062.stdout: exec(code, run_globals) 2024-04-20T22:13:59.157 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5618, in 2024-04-20T22:13:59.157 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5606, in main 2024-04-20T22:13:59.157 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 408, in _infer_config 2024-04-20T22:13:59.157 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 323, in _infer_fsid 2024-04-20T22:13:59.157 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 436, in _infer_image 2024-04-20T22:13:59.158 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 310, in _validate_fsid 2024-04-20T22:13:59.158 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 3359, in command_ceph_volume 2024-04-20T22:13:59.158 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-20T22:13:59.158 INFO:teuthology.orchestra.run.smithi062.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi189 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp11670v2q:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:13:59.158 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-20T22:13:59.158 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is degraded 2024-04-20T22:13:59.158 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T22:13:59.158 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T22:13:59.158 INFO:teuthology.orchestra.run.smithi062.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-20T22:13:59.158 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-20T22:13:59.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:59.159+0000 7f19b8200700 1 -- 172.21.15.62:0/1932758962 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f199806ecc0 msgr2=0x7f1998071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:13:59.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:59.159+0000 7f19b8200700 1 --2- 172.21.15.62:0/1932758962 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f199806ecc0 0x7f1998071170 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f19a4005950 tx=0x7f19a40058e0 comp rx=0 tx=0).stop 2024-04-20T22:13:59.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:59.159+0000 7f19b8200700 1 -- 172.21.15.62:0/1932758962 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f19b0100e80 msgr2=0x7f19b0073590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:13:59.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:59.159+0000 7f19b8200700 1 --2- 172.21.15.62:0/1932758962 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f19b0100e80 0x7f19b0073590 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f199c00bda0 tx=0x7f199c00be80 comp rx=0 tx=0).stop 2024-04-20T22:13:59.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:59.159+0000 7f19b8200700 1 -- 172.21.15.62:0/1932758962 shutdown_connections 2024-04-20T22:13:59.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:59.159+0000 7f19b8200700 1 --2- 172.21.15.62:0/1932758962 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f199806ecc0 0x7f1998071170 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:13:59.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:59.159+0000 7f19b8200700 1 --2- 172.21.15.62:0/1932758962 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19b01004e0 0x7f19b0073050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:13:59.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:59.159+0000 7f19b8200700 1 --2- 172.21.15.62:0/1932758962 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f19b0100e80 0x7f19b0073590 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:13:59.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:59.159+0000 7f19b8200700 1 -- 172.21.15.62:0/1932758962 >> 172.21.15.62:0/1932758962 conn(0x7f19b00fbda0 msgr2=0x7f19b00fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:13:59.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:59.160+0000 7f19b8200700 1 -- 172.21.15.62:0/1932758962 shutdown_connections 2024-04-20T22:13:59.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:13:59.160+0000 7f19b8200700 1 -- 172.21.15.62:0/1932758962 wait complete. 2024-04-20T22:13:59.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:13:59 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/1932758962' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:13:59.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:13:59 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/1932758962' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:14:00.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:14:00 smithi189 ceph-mon[57022]: from='client.36070 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:14:00.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:14:00 smithi189 ceph-mon[57022]: pgmap v1164: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:14:00.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:14:00 smithi062 ceph-mon[76322]: from='client.36070 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:14:00.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:14:00 smithi062 ceph-mon[76322]: pgmap v1164: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:14:02.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:14:02 smithi189 ceph-mon[57022]: pgmap v1165: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:14:02.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:14:02 smithi062 ceph-mon[76322]: pgmap v1165: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:14:03.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:14:03 smithi189 ceph-mon[57022]: pgmap v1166: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:14:03.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:14:03 smithi062 ceph-mon[76322]: pgmap v1166: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:14:04.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:14:04 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:14:04.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:14:04 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:14:05.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:14:05 smithi062 ceph-mon[76322]: pgmap v1167: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:14:05.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:14:05 smithi189 ceph-mon[57022]: pgmap v1167: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:14:08.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:14:08 smithi062 ceph-mon[76322]: pgmap v1168: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:14:08.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:14:08 smithi189 ceph-mon[57022]: pgmap v1168: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:14:10.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:14:10 smithi189 ceph-mon[57022]: pgmap v1169: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:14:10.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:14:10 smithi062 ceph-mon[76322]: pgmap v1169: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:14:12.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:14:12 smithi062 ceph-mon[76322]: pgmap v1170: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:14:12.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:14:12 smithi189 ceph-mon[57022]: pgmap v1170: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:14:14.539 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:14:14 smithi062 ceph-mon[76322]: pgmap v1171: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:14:14.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:14:14 smithi189 ceph-mon[57022]: pgmap v1171: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:14:16.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:14:16 smithi062 ceph-mon[76322]: pgmap v1172: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:14:16.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:14:16 smithi189 ceph-mon[57022]: pgmap v1172: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:14:18.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:14:18 smithi062 ceph-mon[76322]: pgmap v1173: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:14:18.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:14:18 smithi189 ceph-mon[57022]: pgmap v1173: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:14:20.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:14:20 smithi189 ceph-mon[57022]: pgmap v1174: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:14:20.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:14:20 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:14:20.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:14:20 smithi062 ceph-mon[76322]: pgmap v1174: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:14:20.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:14:20 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:14:22.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:14:22 smithi189 ceph-mon[57022]: pgmap v1175: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:14:22.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:14:22 smithi062 ceph-mon[76322]: pgmap v1175: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:14:24.739 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:14:24 smithi189 ceph-mon[57022]: pgmap v1176: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:14:24.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:14:24 smithi062 ceph-mon[76322]: pgmap v1176: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:14:25.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:14:25 smithi189 ceph-mon[57022]: pgmap v1177: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:14:26.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:14:25 smithi062 ceph-mon[76322]: pgmap v1177: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:14:28.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:14:28 smithi062 ceph-mon[76322]: pgmap v1178: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:14:28.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:14:28 smithi189 ceph-mon[57022]: pgmap v1178: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:14:29.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:29.336+0000 7f20a8c00700 1 -- 172.21.15.62:0/923186100 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f20a4101d30 msgr2=0x7f20a410a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:14:29.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:29.336+0000 7f20a8c00700 1 --2- 172.21.15.62:0/923186100 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f20a4101d30 0x7f20a410a230 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f2090009910 tx=0x7f2090009c20 comp rx=0 tx=0).stop 2024-04-20T22:14:29.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:29.337+0000 7f20a8c00700 1 -- 172.21.15.62:0/923186100 shutdown_connections 2024-04-20T22:14:29.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:29.337+0000 7f20a8c00700 1 --2- 172.21.15.62:0/923186100 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f20a4101d30 0x7f20a410a230 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:14:29.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:29.337+0000 7f20a8c00700 1 --2- 172.21.15.62:0/923186100 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f20a4101420 0x7f20a41017f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:14:29.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:29.337+0000 7f20a8c00700 1 -- 172.21.15.62:0/923186100 >> 172.21.15.62:0/923186100 conn(0x7f20a40fac80 msgr2=0x7f20a40fd090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:14:29.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:29.337+0000 7f20a8c00700 1 -- 172.21.15.62:0/923186100 shutdown_connections 2024-04-20T22:14:29.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:29.337+0000 7f20a8c00700 1 -- 172.21.15.62:0/923186100 wait complete. 2024-04-20T22:14:29.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:29.338+0000 7f20a8c00700 1 Processor -- start 2024-04-20T22:14:29.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:29.338+0000 7f20a8c00700 1 -- start start 2024-04-20T22:14:29.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:29.339+0000 7f20a8c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f20a4101420 0x7f20a4195e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:14:29.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:29.339+0000 7f20a8c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f20a4101d30 0x7f20a41963a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:14:29.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:29.339+0000 7f20a8c00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20a41969f0 con 0x7f20a4101420 2024-04-20T22:14:29.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:29.339+0000 7f20a8c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20a4196b30 con 0x7f20a4101d30 2024-04-20T22:14:29.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:29.339+0000 7f20a1e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f20a4101420 0x7f20a4195e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:14:29.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:29.339+0000 7f20a1e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f20a4101420 0x7f20a4195e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48594/0 (socket says 172.21.15.62:48594) 2024-04-20T22:14:29.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:29.339+0000 7f20a1e00700 1 -- 172.21.15.62:0/641243026 learned_addr learned my addr 172.21.15.62:0/641243026 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:14:29.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:29.340+0000 7f20a1400700 1 --2- 172.21.15.62:0/641243026 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f20a4101d30 0x7f20a41963a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:14:29.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:29.340+0000 7f20a1e00700 1 -- 172.21.15.62:0/641243026 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f20a4101d30 msgr2=0x7f20a41963a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:14:29.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:29.340+0000 7f20a1e00700 1 --2- 172.21.15.62:0/641243026 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f20a4101d30 0x7f20a41963a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:14:29.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:29.340+0000 7f20a1e00700 1 -- 172.21.15.62:0/641243026 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20900095f0 con 0x7f20a4101420 2024-04-20T22:14:29.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:29.340+0000 7f20a1e00700 1 --2- 172.21.15.62:0/641243026 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f20a4101420 0x7f20a4195e60 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f209400c930 tx=0x7f209400ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:14:29.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:29.341+0000 7f209aa00700 1 -- 172.21.15.62:0/641243026 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2094007ad0 con 0x7f20a4101420 2024-04-20T22:14:29.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:29.341+0000 7f209aa00700 1 -- 172.21.15.62:0/641243026 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2094007c30 con 0x7f20a4101420 2024-04-20T22:14:29.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:29.341+0000 7f20a8c00700 1 -- 172.21.15.62:0/641243026 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20a419a980 con 0x7f20a4101420 2024-04-20T22:14:29.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:29.341+0000 7f209aa00700 1 -- 172.21.15.62:0/641243026 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2094018610 con 0x7f20a4101420 2024-04-20T22:14:29.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:29.341+0000 7f20a8c00700 1 -- 172.21.15.62:0/641243026 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20a419aed0 con 0x7f20a4101420 2024-04-20T22:14:29.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:29.342+0000 7f20a8c00700 1 -- 172.21.15.62:0/641243026 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20a41078a0 con 0x7f20a4101420 2024-04-20T22:14:29.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:29.343+0000 7f209aa00700 1 -- 172.21.15.62:0/641243026 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f209400f450 con 0x7f20a4101420 2024-04-20T22:14:29.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:29.344+0000 7f209aa00700 1 --2- 172.21.15.62:0/641243026 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f208806ecc0 0x7f2088071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:14:29.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:29.345+0000 7f209aa00700 1 -- 172.21.15.62:0/641243026 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f209408eff0 con 0x7f20a4101420 2024-04-20T22:14:29.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:29.345+0000 7f20a1400700 1 --2- 172.21.15.62:0/641243026 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f208806ecc0 0x7f2088071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:14:29.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:29.346+0000 7f20a1400700 1 --2- 172.21.15.62:0/641243026 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f208806ecc0 0x7f2088071170 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f20a4197480 tx=0x7f209001a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:14:29.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:29.350+0000 7f209aa00700 1 -- 172.21.15.62:0/641243026 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f2094057de0 con 0x7f20a4101420 2024-04-20T22:14:29.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:29.604+0000 7f20a8c00700 1 -- 172.21.15.62:0/641243026 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f20a419b150 con 0x7f208806ecc0 2024-04-20T22:14:29.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:29.607+0000 7f209aa00700 1 -- 172.21.15.62:0/641243026 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f20a419b150 con 0x7f208806ecc0 2024-04-20T22:14:29.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:29.610+0000 7f20a8c00700 1 -- 172.21.15.62:0/641243026 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f208806ecc0 msgr2=0x7f2088071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:14:29.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:29.610+0000 7f20a8c00700 1 --2- 172.21.15.62:0/641243026 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f208806ecc0 0x7f2088071170 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f20a4197480 tx=0x7f209001a040 comp rx=0 tx=0).stop 2024-04-20T22:14:29.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:29.610+0000 7f20a8c00700 1 -- 172.21.15.62:0/641243026 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f20a4101420 msgr2=0x7f20a4195e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:14:29.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:29.610+0000 7f20a8c00700 1 --2- 172.21.15.62:0/641243026 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f20a4101420 0x7f20a4195e60 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f209400c930 tx=0x7f209400ccf0 comp rx=0 tx=0).stop 2024-04-20T22:14:29.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:29.610+0000 7f20a8c00700 1 -- 172.21.15.62:0/641243026 shutdown_connections 2024-04-20T22:14:29.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:29.610+0000 7f20a8c00700 1 --2- 172.21.15.62:0/641243026 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f208806ecc0 0x7f2088071170 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:14:29.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:29.610+0000 7f20a8c00700 1 --2- 172.21.15.62:0/641243026 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f20a4101420 0x7f20a4195e60 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:14:29.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:29.610+0000 7f20a8c00700 1 --2- 172.21.15.62:0/641243026 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f20a4101d30 0x7f20a41963a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:14:29.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:29.611+0000 7f20a8c00700 1 -- 172.21.15.62:0/641243026 >> 172.21.15.62:0/641243026 conn(0x7f20a40fac80 msgr2=0x7f20a41049e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:14:29.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:29.611+0000 7f20a8c00700 1 -- 172.21.15.62:0/641243026 shutdown_connections 2024-04-20T22:14:29.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:29.611+0000 7f20a8c00700 1 -- 172.21.15.62:0/641243026 wait complete. 2024-04-20T22:14:29.628 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T22:14:29.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:29.778+0000 7fc6b2e00700 1 -- 172.21.15.62:0/1831243366 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc6ac1004e0 msgr2=0x7fc6ac1008b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:14:29.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:29.778+0000 7fc6b2e00700 1 --2- 172.21.15.62:0/1831243366 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc6ac1004e0 0x7fc6ac1008b0 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7fc69c009910 tx=0x7fc69c009c20 comp rx=0 tx=0).stop 2024-04-20T22:14:29.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:29.778+0000 7fc6b2e00700 1 -- 172.21.15.62:0/1831243366 shutdown_connections 2024-04-20T22:14:29.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:29.778+0000 7fc6b2e00700 1 --2- 172.21.15.62:0/1831243366 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6ac100e80 0x7fc6ac104e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:14:29.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:29.778+0000 7fc6b2e00700 1 --2- 172.21.15.62:0/1831243366 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc6ac1004e0 0x7fc6ac1008b0 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:14:29.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:29.778+0000 7fc6b2e00700 1 -- 172.21.15.62:0/1831243366 >> 172.21.15.62:0/1831243366 conn(0x7fc6ac0fbda0 msgr2=0x7fc6ac0fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:14:29.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:29.778+0000 7fc6b2e00700 1 -- 172.21.15.62:0/1831243366 shutdown_connections 2024-04-20T22:14:29.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:29.778+0000 7fc6b2e00700 1 -- 172.21.15.62:0/1831243366 wait complete. 2024-04-20T22:14:29.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:29.779+0000 7fc6b2e00700 1 Processor -- start 2024-04-20T22:14:29.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:29.780+0000 7fc6b2e00700 1 -- start start 2024-04-20T22:14:29.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:29.780+0000 7fc6b2e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc6ac1004e0 0x7fc6ac073050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:14:29.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:29.780+0000 7fc6b2e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6ac100e80 0x7fc6ac073590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:14:29.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:29.780+0000 7fc6b2e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6ac073ad0 con 0x7fc6ac1004e0 2024-04-20T22:14:29.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:29.780+0000 7fc6b2e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6ac073c10 con 0x7fc6ac100e80 2024-04-20T22:14:29.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:29.780+0000 7fc6abe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc6ac1004e0 0x7fc6ac073050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:14:29.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:29.781+0000 7fc6abe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc6ac1004e0 0x7fc6ac073050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48620/0 (socket says 172.21.15.62:48620) 2024-04-20T22:14:29.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:29.781+0000 7fc6abe00700 1 -- 172.21.15.62:0/2828517700 learned_addr learned my addr 172.21.15.62:0/2828517700 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:14:29.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:29.781+0000 7fc6abe00700 1 -- 172.21.15.62:0/2828517700 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6ac100e80 msgr2=0x7fc6ac073590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:14:29.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:29.781+0000 7fc6ab400700 1 --2- 172.21.15.62:0/2828517700 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6ac100e80 0x7fc6ac073590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:14:29.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:29.781+0000 7fc6abe00700 1 --2- 172.21.15.62:0/2828517700 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6ac100e80 0x7fc6ac073590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:14:29.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:29.781+0000 7fc6abe00700 1 -- 172.21.15.62:0/2828517700 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc69c0095f0 con 0x7fc6ac1004e0 2024-04-20T22:14:29.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:29.781+0000 7fc6ab400700 1 --2- 172.21.15.62:0/2828517700 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6ac100e80 0x7fc6ac073590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T22:14:29.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:29.781+0000 7fc6abe00700 1 --2- 172.21.15.62:0/2828517700 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc6ac1004e0 0x7fc6ac073050 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7fc69c00b5c0 tx=0x7fc69c00be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:14:29.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:29.782+0000 7fc6a8c00700 1 -- 172.21.15.62:0/2828517700 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc69c01d070 con 0x7fc6ac1004e0 2024-04-20T22:14:29.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:29.782+0000 7fc6a8c00700 1 -- 172.21.15.62:0/2828517700 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc69c00f460 con 0x7fc6ac1004e0 2024-04-20T22:14:29.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:29.782+0000 7fc6b2e00700 1 -- 172.21.15.62:0/2828517700 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6ac073e90 con 0x7fc6ac1004e0 2024-04-20T22:14:29.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:29.782+0000 7fc6a8c00700 1 -- 172.21.15.62:0/2828517700 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc69c017610 con 0x7fc6ac1004e0 2024-04-20T22:14:29.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:29.782+0000 7fc6b2e00700 1 -- 172.21.15.62:0/2828517700 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6ac1a23a0 con 0x7fc6ac1004e0 2024-04-20T22:14:29.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:29.783+0000 7fc6b2e00700 1 -- 172.21.15.62:0/2828517700 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6ac1087f0 con 0x7fc6ac1004e0 2024-04-20T22:14:29.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:29.784+0000 7fc6a8c00700 1 -- 172.21.15.62:0/2828517700 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fc69c00f5d0 con 0x7fc6ac1004e0 2024-04-20T22:14:29.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:29.785+0000 7fc6a8c00700 1 --2- 172.21.15.62:0/2828517700 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc69006ecc0 0x7fc690071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:14:29.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:29.785+0000 7fc6a8c00700 1 -- 172.21.15.62:0/2828517700 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fc69c090440 con 0x7fc6ac1004e0 2024-04-20T22:14:29.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:29.786+0000 7fc6ab400700 1 --2- 172.21.15.62:0/2828517700 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc69006ecc0 0x7fc690071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:14:29.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:29.786+0000 7fc6ab400700 1 --2- 172.21.15.62:0/2828517700 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc69006ecc0 0x7fc690071170 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fc698005fd0 tx=0x7fc698005e20 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:14:29.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:29.790+0000 7fc6a8c00700 1 -- 172.21.15.62:0/2828517700 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fc69c0591d0 con 0x7fc6ac1004e0 2024-04-20T22:14:30.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.048+0000 7fc6b2e00700 1 -- 172.21.15.62:0/2828517700 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc6ac061190 con 0x7fc69006ecc0 2024-04-20T22:14:30.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.050+0000 7fc6a8c00700 1 -- 172.21.15.62:0/2828517700 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fc6ac061190 con 0x7fc69006ecc0 2024-04-20T22:14:30.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.053+0000 7fc6b2e00700 1 -- 172.21.15.62:0/2828517700 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc69006ecc0 msgr2=0x7fc690071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:14:30.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.053+0000 7fc6b2e00700 1 --2- 172.21.15.62:0/2828517700 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc69006ecc0 0x7fc690071170 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fc698005fd0 tx=0x7fc698005e20 comp rx=0 tx=0).stop 2024-04-20T22:14:30.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.053+0000 7fc6b2e00700 1 -- 172.21.15.62:0/2828517700 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc6ac1004e0 msgr2=0x7fc6ac073050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:14:30.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.053+0000 7fc6b2e00700 1 --2- 172.21.15.62:0/2828517700 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc6ac1004e0 0x7fc6ac073050 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7fc69c00b5c0 tx=0x7fc69c00be00 comp rx=0 tx=0).stop 2024-04-20T22:14:30.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.053+0000 7fc6b2e00700 1 -- 172.21.15.62:0/2828517700 shutdown_connections 2024-04-20T22:14:30.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.053+0000 7fc6b2e00700 1 --2- 172.21.15.62:0/2828517700 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc69006ecc0 0x7fc690071170 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:14:30.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.053+0000 7fc6b2e00700 1 --2- 172.21.15.62:0/2828517700 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc6ac1004e0 0x7fc6ac073050 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:14:30.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.053+0000 7fc6b2e00700 1 --2- 172.21.15.62:0/2828517700 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6ac100e80 0x7fc6ac073590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:14:30.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.053+0000 7fc6b2e00700 1 -- 172.21.15.62:0/2828517700 >> 172.21.15.62:0/2828517700 conn(0x7fc6ac0fbda0 msgr2=0x7fc6ac0fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:14:30.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.054+0000 7fc6b2e00700 1 -- 172.21.15.62:0/2828517700 shutdown_connections 2024-04-20T22:14:30.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.054+0000 7fc6b2e00700 1 -- 172.21.15.62:0/2828517700 wait complete. 2024-04-20T22:14:30.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.217+0000 7f14f7e00700 1 -- 172.21.15.62:0/4234162664 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f14f8100c70 msgr2=0x7f14f8104c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:14:30.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.217+0000 7f14f7e00700 1 --2- 172.21.15.62:0/4234162664 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f14f8100c70 0x7f14f8104c40 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f14e4009910 tx=0x7f14e4009c20 comp rx=0 tx=0).stop 2024-04-20T22:14:30.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.217+0000 7f14f7e00700 1 -- 172.21.15.62:0/4234162664 shutdown_connections 2024-04-20T22:14:30.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.217+0000 7f14f7e00700 1 --2- 172.21.15.62:0/4234162664 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f14f8100c70 0x7f14f8104c40 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:14:30.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.217+0000 7f14f7e00700 1 --2- 172.21.15.62:0/4234162664 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f14f81002d0 0x7f14f81006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:14:30.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.217+0000 7f14f7e00700 1 -- 172.21.15.62:0/4234162664 >> 172.21.15.62:0/4234162664 conn(0x7f14f80fbbb0 msgr2=0x7f14f80fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:14:30.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.217+0000 7f14f7e00700 1 -- 172.21.15.62:0/4234162664 shutdown_connections 2024-04-20T22:14:30.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.218+0000 7f14f7e00700 1 -- 172.21.15.62:0/4234162664 wait complete. 2024-04-20T22:14:30.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.219+0000 7f14f7e00700 1 Processor -- start 2024-04-20T22:14:30.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.219+0000 7f14f7e00700 1 -- start start 2024-04-20T22:14:30.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.219+0000 7f14f7e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f14f81002d0 0x7f14f8073050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:14:30.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.219+0000 7f14f7e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f14f8073590 0x7f14f8073a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:14:30.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.219+0000 7f14f7e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14f81a2060 con 0x7f14f81002d0 2024-04-20T22:14:30.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.219+0000 7f14f7e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14f81a21a0 con 0x7f14f8073590 2024-04-20T22:14:30.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.220+0000 7f14f6a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f14f81002d0 0x7f14f8073050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:14:30.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.220+0000 7f14f6a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f14f81002d0 0x7f14f8073050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48642/0 (socket says 172.21.15.62:48642) 2024-04-20T22:14:30.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.220+0000 7f14f6a00700 1 -- 172.21.15.62:0/3891061979 learned_addr learned my addr 172.21.15.62:0/3891061979 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:14:30.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.220+0000 7f14f6000700 1 --2- 172.21.15.62:0/3891061979 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f14f8073590 0x7f14f8073a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:14:30.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.220+0000 7f14f6a00700 1 -- 172.21.15.62:0/3891061979 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f14f8073590 msgr2=0x7f14f8073a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:14:30.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.220+0000 7f14f6a00700 1 --2- 172.21.15.62:0/3891061979 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f14f8073590 0x7f14f8073a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:14:30.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.220+0000 7f14f6a00700 1 -- 172.21.15.62:0/3891061979 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14e40095f0 con 0x7f14f81002d0 2024-04-20T22:14:30.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.220+0000 7f14f6a00700 1 --2- 172.21.15.62:0/3891061979 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f14f81002d0 0x7f14f8073050 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f14e800d900 tx=0x7f14e800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:14:30.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.221+0000 7f14ef400700 1 -- 172.21.15.62:0/3891061979 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f14e8009b10 con 0x7f14f81002d0 2024-04-20T22:14:30.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.221+0000 7f14ef400700 1 -- 172.21.15.62:0/3891061979 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f14e800b6e0 con 0x7f14f81002d0 2024-04-20T22:14:30.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.221+0000 7f14ef400700 1 -- 172.21.15.62:0/3891061979 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f14e800f5f0 con 0x7f14f81002d0 2024-04-20T22:14:30.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.221+0000 7f14f7e00700 1 -- 172.21.15.62:0/3891061979 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14f81a22e0 con 0x7f14f81002d0 2024-04-20T22:14:30.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.221+0000 7f14f7e00700 1 -- 172.21.15.62:0/3891061979 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14f81a2830 con 0x7f14f81002d0 2024-04-20T22:14:30.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.222+0000 7f14f7e00700 1 -- 172.21.15.62:0/3891061979 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14f8108640 con 0x7f14f81002d0 2024-04-20T22:14:30.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.223+0000 7f14ef400700 1 -- 172.21.15.62:0/3891061979 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f14e8009c70 con 0x7f14f81002d0 2024-04-20T22:14:30.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.224+0000 7f14ef400700 1 --2- 172.21.15.62:0/3891061979 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f14dc06ecc0 0x7f14dc071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:14:30.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.224+0000 7f14ef400700 1 -- 172.21.15.62:0/3891061979 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f14e808ee50 con 0x7f14f81002d0 2024-04-20T22:14:30.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.224+0000 7f14f6000700 1 --2- 172.21.15.62:0/3891061979 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f14dc06ecc0 0x7f14dc071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:14:30.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.225+0000 7f14f6000700 1 --2- 172.21.15.62:0/3891061979 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f14dc06ecc0 0x7f14dc071170 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f14e4000c00 tx=0x7f14e401a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:14:30.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.230+0000 7f14ef400700 1 -- 172.21.15.62:0/3891061979 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f14e8057b60 con 0x7f14f81002d0 2024-04-20T22:14:30.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.465+0000 7f14f7e00700 1 -- 172.21.15.62:0/3891061979 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f14f8061190 con 0x7f14dc06ecc0 2024-04-20T22:14:30.465 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:14:30 smithi062 ceph-mon[76322]: pgmap v1179: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:14:30.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.482+0000 7f14ef400700 1 -- 172.21.15.62:0/3891061979 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f14f8061190 con 0x7f14dc06ecc0 2024-04-20T22:14:30.482 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T22:14:30.482 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (40m) 69s ago 54m 14.6M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T22:14:30.483 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (54m) 69s ago 54m 11.6M - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T22:14:30.483 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (53m) 73s ago 53m 11.2M - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T22:14:30.483 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (36m) 69s ago 54m 6660k - 19.0.0-3133-ga8c84f59 76bc6b91b726 2825d5ba11c6 2024-04-20T22:14:30.483 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (36m) 73s ago 53m 6665k - 19.0.0-3133-ga8c84f59 76bc6b91b726 20f746fa1bba 2024-04-20T22:14:30.483 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (39m) 69s ago 53m 75.5M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T22:14:30.483 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (31m) 69s ago 46m 22.9M - 19.0.0-3133-ga8c84f59 76bc6b91b726 de80bc90a683 2024-04-20T22:14:30.483 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (32m) 69s ago 47m 23.0M - 19.0.0-3133-ga8c84f59 76bc6b91b726 cd793b7ee98e 2024-04-20T22:14:30.483 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (16m) 73s ago 47m 18.5M - 19.0.0-3133-ga8c84f59 76bc6b91b726 0e7b0bc4d1ab 2024-04-20T22:14:30.483 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (79s) 73s ago 46m 11.4M - 19.0.0-3133-ga8c84f59 76bc6b91b726 0af732ac9e06 2024-04-20T22:14:30.483 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (43m) 69s ago 56m 517M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T22:14:30.483 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (41m) 73s ago 52m 460M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T22:14:30.483 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (38m) 69s ago 56m 86.3M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 019eb1479485 2024-04-20T22:14:30.483 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (37m) 73s ago 52m 58.0M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 55aac560aab6 2024-04-20T22:14:30.483 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (41m) 69s ago 54m 8766k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T22:14:30.483 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (41m) 73s ago 52m 8711k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T22:14:30.484 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (35m) 69s ago 51m 91.3M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 ee2bb901f26e 2024-04-20T22:14:30.484 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (35m) 69s ago 51m 94.9M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 9075152d4162 2024-04-20T22:14:30.484 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (34m) 69s ago 50m 78.5M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 2d4f3e38dc0e 2024-04-20T22:14:30.484 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (34m) 73s ago 50m 94.1M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 8f97b17ae0eb 2024-04-20T22:14:30.484 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (33m) 73s ago 49m 96.1M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 653d8079967d 2024-04-20T22:14:30.484 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (32m) 73s ago 48m 90.9M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 596dded14c12 2024-04-20T22:14:30.484 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (40m) 69s ago 53m 90.4M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T22:14:30.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.485+0000 7f14f7e00700 1 -- 172.21.15.62:0/3891061979 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f14dc06ecc0 msgr2=0x7f14dc071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:14:30.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.485+0000 7f14f7e00700 1 --2- 172.21.15.62:0/3891061979 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f14dc06ecc0 0x7f14dc071170 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f14e4000c00 tx=0x7f14e401a040 comp rx=0 tx=0).stop 2024-04-20T22:14:30.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.485+0000 7f14f7e00700 1 -- 172.21.15.62:0/3891061979 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f14f81002d0 msgr2=0x7f14f8073050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:14:30.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.485+0000 7f14f7e00700 1 --2- 172.21.15.62:0/3891061979 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f14f81002d0 0x7f14f8073050 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f14e800d900 tx=0x7f14e800dcc0 comp rx=0 tx=0).stop 2024-04-20T22:14:30.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.485+0000 7f14f7e00700 1 -- 172.21.15.62:0/3891061979 shutdown_connections 2024-04-20T22:14:30.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.485+0000 7f14f7e00700 1 --2- 172.21.15.62:0/3891061979 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f14dc06ecc0 0x7f14dc071170 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:14:30.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.485+0000 7f14f7e00700 1 --2- 172.21.15.62:0/3891061979 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f14f81002d0 0x7f14f8073050 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:14:30.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.485+0000 7f14f7e00700 1 --2- 172.21.15.62:0/3891061979 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f14f8073590 0x7f14f8073a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:14:30.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.485+0000 7f14f7e00700 1 -- 172.21.15.62:0/3891061979 >> 172.21.15.62:0/3891061979 conn(0x7f14f80fbbb0 msgr2=0x7f14f80fd280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:14:30.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.486+0000 7f14f7e00700 1 -- 172.21.15.62:0/3891061979 shutdown_connections 2024-04-20T22:14:30.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.486+0000 7f14f7e00700 1 -- 172.21.15.62:0/3891061979 wait complete. 2024-04-20T22:14:30.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.635+0000 7f04d5e00700 1 -- 172.21.15.62:0/2355067632 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04d0106b80 msgr2=0x7f04d0106ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:14:30.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.635+0000 7f04d5e00700 1 --2- 172.21.15.62:0/2355067632 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04d0106b80 0x7f04d0106ff0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f04bc009910 tx=0x7f04bc009c20 comp rx=0 tx=0).stop 2024-04-20T22:14:30.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.635+0000 7f04d5e00700 1 -- 172.21.15.62:0/2355067632 shutdown_connections 2024-04-20T22:14:30.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.635+0000 7f04d5e00700 1 --2- 172.21.15.62:0/2355067632 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04d0106b80 0x7f04d0106ff0 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:14:30.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.635+0000 7f04d5e00700 1 --2- 172.21.15.62:0/2355067632 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f04d010c8f0 0x7f04d010ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:14:30.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.635+0000 7f04d5e00700 1 -- 172.21.15.62:0/2355067632 >> 172.21.15.62:0/2355067632 conn(0x7f04d0076060 msgr2=0x7f04d0078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:14:30.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.635+0000 7f04d5e00700 1 -- 172.21.15.62:0/2355067632 shutdown_connections 2024-04-20T22:14:30.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.635+0000 7f04d5e00700 1 -- 172.21.15.62:0/2355067632 wait complete. 2024-04-20T22:14:30.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.636+0000 7f04d5e00700 1 Processor -- start 2024-04-20T22:14:30.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.636+0000 7f04d5e00700 1 -- start start 2024-04-20T22:14:30.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.636+0000 7f04d5e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04d0106b80 0x7f04d019c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:14:30.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.637+0000 7f04d5e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f04d010c8f0 0x7f04d019c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:14:30.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.637+0000 7f04d5e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04d019d060 con 0x7f04d0106b80 2024-04-20T22:14:30.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.637+0000 7f04d5e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04d01a0df0 con 0x7f04d010c8f0 2024-04-20T22:14:30.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.637+0000 7f04cea00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04d0106b80 0x7f04d019c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:14:30.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.637+0000 7f04cea00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04d0106b80 0x7f04d019c440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48652/0 (socket says 172.21.15.62:48652) 2024-04-20T22:14:30.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.637+0000 7f04cea00700 1 -- 172.21.15.62:0/1147559675 learned_addr learned my addr 172.21.15.62:0/1147559675 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:14:30.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.637+0000 7f04ce000700 1 --2- 172.21.15.62:0/1147559675 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f04d010c8f0 0x7f04d019c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:14:30.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.637+0000 7f04cea00700 1 -- 172.21.15.62:0/1147559675 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f04d010c8f0 msgr2=0x7f04d019c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:14:30.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.637+0000 7f04cea00700 1 --2- 172.21.15.62:0/1147559675 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f04d010c8f0 0x7f04d019c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:14:30.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.637+0000 7f04cea00700 1 -- 172.21.15.62:0/1147559675 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04bc0095f0 con 0x7f04d0106b80 2024-04-20T22:14:30.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.638+0000 7f04cea00700 1 --2- 172.21.15.62:0/1147559675 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04d0106b80 0x7f04d019c440 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f04c000e920 tx=0x7f04c000ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:14:30.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.638+0000 7f04d4a00700 1 -- 172.21.15.62:0/1147559675 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f04c0009790 con 0x7f04d0106b80 2024-04-20T22:14:30.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.638+0000 7f04d4a00700 1 -- 172.21.15.62:0/1147559675 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f04c000cba0 con 0x7f04d0106b80 2024-04-20T22:14:30.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.638+0000 7f04d5e00700 1 -- 172.21.15.62:0/1147559675 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04d01a10d0 con 0x7f04d0106b80 2024-04-20T22:14:30.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.638+0000 7f04d4a00700 1 -- 172.21.15.62:0/1147559675 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f04c0018650 con 0x7f04d0106b80 2024-04-20T22:14:30.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.638+0000 7f04d5e00700 1 -- 172.21.15.62:0/1147559675 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04d01a1620 con 0x7f04d0106b80 2024-04-20T22:14:30.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.639+0000 7f04d5e00700 1 -- 172.21.15.62:0/1147559675 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04d010ecc0 con 0x7f04d0106b80 2024-04-20T22:14:30.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.640+0000 7f04d4a00700 1 -- 172.21.15.62:0/1147559675 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f04c000cd10 con 0x7f04d0106b80 2024-04-20T22:14:30.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.641+0000 7f04d4a00700 1 --2- 172.21.15.62:0/1147559675 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f04b406ecc0 0x7f04b4071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:14:30.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.641+0000 7f04d4a00700 1 -- 172.21.15.62:0/1147559675 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f04c0014070 con 0x7f04d0106b80 2024-04-20T22:14:30.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.642+0000 7f04ce000700 1 --2- 172.21.15.62:0/1147559675 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f04b406ecc0 0x7f04b4071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:14:30.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.642+0000 7f04ce000700 1 --2- 172.21.15.62:0/1147559675 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f04b406ecc0 0x7f04b4071170 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f04bc006010 tx=0x7f04bc005a90 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:14:30.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.646+0000 7f04d4a00700 1 -- 172.21.15.62:0/1147559675 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f04c0053ff0 con 0x7f04d0106b80 2024-04-20T22:14:30.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:14:30 smithi189 ceph-mon[57022]: pgmap v1179: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:14:30.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.972+0000 7f04d5e00700 1 -- 172.21.15.62:0/1147559675 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f04d01a1840 con 0x7f04d0106b80 2024-04-20T22:14:30.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.973+0000 7f04d4a00700 1 -- 172.21.15.62:0/1147559675 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f04c005bb40 con 0x7f04d0106b80 2024-04-20T22:14:30.973 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:14:30.973 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T22:14:30.973 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:14:30.974 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:14:30.974 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T22:14:30.974 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:14:30.974 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:14:30.974 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T22:14:30.974 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 6 2024-04-20T22:14:30.974 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:14:30.974 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T22:14:30.974 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 4 2024-04-20T22:14:30.974 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:14:30.974 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T22:14:30.974 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 14 2024-04-20T22:14:30.974 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T22:14:30.974 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:14:30.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.976+0000 7f04d5e00700 1 -- 172.21.15.62:0/1147559675 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f04b406ecc0 msgr2=0x7f04b4071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:14:30.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.976+0000 7f04d5e00700 1 --2- 172.21.15.62:0/1147559675 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f04b406ecc0 0x7f04b4071170 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f04bc006010 tx=0x7f04bc005a90 comp rx=0 tx=0).stop 2024-04-20T22:14:30.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.976+0000 7f04d5e00700 1 -- 172.21.15.62:0/1147559675 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04d0106b80 msgr2=0x7f04d019c440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:14:30.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.976+0000 7f04d5e00700 1 --2- 172.21.15.62:0/1147559675 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04d0106b80 0x7f04d019c440 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f04c000e920 tx=0x7f04c000ece0 comp rx=0 tx=0).stop 2024-04-20T22:14:30.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.977+0000 7f04d5e00700 1 -- 172.21.15.62:0/1147559675 shutdown_connections 2024-04-20T22:14:30.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.977+0000 7f04d5e00700 1 --2- 172.21.15.62:0/1147559675 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f04b406ecc0 0x7f04b4071170 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:14:30.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.977+0000 7f04d5e00700 1 --2- 172.21.15.62:0/1147559675 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04d0106b80 0x7f04d019c440 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:14:30.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.977+0000 7f04d5e00700 1 --2- 172.21.15.62:0/1147559675 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f04d010c8f0 0x7f04d019c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:14:30.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.977+0000 7f04d5e00700 1 -- 172.21.15.62:0/1147559675 >> 172.21.15.62:0/1147559675 conn(0x7f04d0076060 msgr2=0x7f04d0077ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:14:30.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.977+0000 7f04d5e00700 1 -- 172.21.15.62:0/1147559675 shutdown_connections 2024-04-20T22:14:30.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:30.977+0000 7f04d5e00700 1 -- 172.21.15.62:0/1147559675 wait complete. 2024-04-20T22:14:31.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:31.126+0000 7fbdfc600700 1 -- 172.21.15.62:0/2268415960 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbdf4102470 msgr2=0x7fbdf41028e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:14:31.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:31.126+0000 7fbdfc600700 1 --2- 172.21.15.62:0/2268415960 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbdf4102470 0x7fbdf41028e0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7fbde0009910 tx=0x7fbde0009c20 comp rx=0 tx=0).stop 2024-04-20T22:14:31.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:31.126+0000 7fbdfc600700 1 -- 172.21.15.62:0/2268415960 shutdown_connections 2024-04-20T22:14:31.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:31.126+0000 7fbdfc600700 1 --2- 172.21.15.62:0/2268415960 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbdf4102470 0x7fbdf41028e0 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:14:31.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:31.126+0000 7fbdfc600700 1 --2- 172.21.15.62:0/2268415960 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbdf4108470 0x7fbdf4108840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:14:31.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:31.126+0000 7fbdfc600700 1 -- 172.21.15.62:0/2268415960 >> 172.21.15.62:0/2268415960 conn(0x7fbdf40fdf90 msgr2=0x7fbdf41003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:14:31.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:31.126+0000 7fbdfc600700 1 -- 172.21.15.62:0/2268415960 shutdown_connections 2024-04-20T22:14:31.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:31.126+0000 7fbdfc600700 1 -- 172.21.15.62:0/2268415960 wait complete. 2024-04-20T22:14:31.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:31.127+0000 7fbdfc600700 1 Processor -- start 2024-04-20T22:14:31.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:31.127+0000 7fbdfc600700 1 -- start start 2024-04-20T22:14:31.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:31.128+0000 7fbdfc600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbdf4102470 0x7fbdf4198200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:14:31.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:31.128+0000 7fbdfc600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbdf4108470 0x7fbdf4198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:14:31.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:31.128+0000 7fbdfc600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf4198e20 con 0x7fbdf4108470 2024-04-20T22:14:31.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:31.128+0000 7fbdfc600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf419cb60 con 0x7fbdf4102470 2024-04-20T22:14:31.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:31.128+0000 7fbdf9200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbdf4108470 0x7fbdf4198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:14:31.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:31.128+0000 7fbdf9200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbdf4108470 0x7fbdf4198740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48664/0 (socket says 172.21.15.62:48664) 2024-04-20T22:14:31.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:31.128+0000 7fbdf9200700 1 -- 172.21.15.62:0/2182910826 learned_addr learned my addr 172.21.15.62:0/2182910826 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:14:31.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:31.128+0000 7fbdf9c00700 1 --2- 172.21.15.62:0/2182910826 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbdf4102470 0x7fbdf4198200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:14:31.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:31.129+0000 7fbdf9200700 1 -- 172.21.15.62:0/2182910826 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbdf4102470 msgr2=0x7fbdf4198200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:14:31.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:31.129+0000 7fbdf9200700 1 --2- 172.21.15.62:0/2182910826 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbdf4102470 0x7fbdf4198200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:14:31.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:31.129+0000 7fbdf9200700 1 -- 172.21.15.62:0/2182910826 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbde00095f0 con 0x7fbdf4108470 2024-04-20T22:14:31.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:31.129+0000 7fbdf9200700 1 --2- 172.21.15.62:0/2182910826 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbdf4108470 0x7fbdf4198740 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7fbde000b5c0 tx=0x7fbde000be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:14:31.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:31.129+0000 7fbdf2000700 1 -- 172.21.15.62:0/2182910826 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbde001d070 con 0x7fbdf4108470 2024-04-20T22:14:31.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:31.129+0000 7fbdf2000700 1 -- 172.21.15.62:0/2182910826 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbde000f460 con 0x7fbdf4108470 2024-04-20T22:14:31.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:31.129+0000 7fbdfc600700 1 -- 172.21.15.62:0/2182910826 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdf419cde0 con 0x7fbdf4108470 2024-04-20T22:14:31.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:31.129+0000 7fbdf2000700 1 -- 172.21.15.62:0/2182910826 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbde0017610 con 0x7fbdf4108470 2024-04-20T22:14:31.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:31.130+0000 7fbdfc600700 1 -- 172.21.15.62:0/2182910826 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdf419d2d0 con 0x7fbdf4108470 2024-04-20T22:14:31.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:31.130+0000 7fbdfc600700 1 -- 172.21.15.62:0/2182910826 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdf410aa00 con 0x7fbdf4108470 2024-04-20T22:14:31.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:31.132+0000 7fbdf2000700 1 -- 172.21.15.62:0/2182910826 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fbde000f5d0 con 0x7fbdf4108470 2024-04-20T22:14:31.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:31.132+0000 7fbdf2000700 1 --2- 172.21.15.62:0/2182910826 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fbddc06ecc0 0x7fbddc071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:14:31.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:31.133+0000 7fbdf2000700 1 -- 172.21.15.62:0/2182910826 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fbde0090450 con 0x7fbdf4108470 2024-04-20T22:14:31.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:31.133+0000 7fbdf9c00700 1 --2- 172.21.15.62:0/2182910826 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fbddc06ecc0 0x7fbddc071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:14:31.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:31.133+0000 7fbdf9c00700 1 --2- 172.21.15.62:0/2182910826 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fbddc06ecc0 0x7fbddc071170 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fbde8006fd0 tx=0x7fbde8008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:14:31.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:31.137+0000 7fbdf2000700 1 -- 172.21.15.62:0/2182910826 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fbde00591e0 con 0x7fbdf4108470 2024-04-20T22:14:31.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:31.408+0000 7fbdfc600700 1 -- 172.21.15.62:0/2182910826 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fbdf404f2a0 con 0x7fbdf4108470 2024-04-20T22:14:31.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:31.409+0000 7fbdf2000700 1 -- 172.21.15.62:0/2182910826 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 29 v29) v1 ==== 76+0+1887 (secure 0 0 0) 0x7fbde005c8a0 con 0x7fbdf4108470 2024-04-20T22:14:31.410 INFO:teuthology.orchestra.run.smithi062.stdout:e29 2024-04-20T22:14:31.410 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T22:14:31.410 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:14:31.410 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T22:14:31.410 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:14:31.410 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T22:14:31.410 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T22:14:31.410 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 20 2024-04-20T22:14:31.410 INFO:teuthology.orchestra.run.smithi062.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-20T22:14:31.410 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T22:14:31.410 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T21:42:01.559061+0000 2024-04-20T22:14:31.411 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T22:14:31.411 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T22:14:31.411 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T22:14:31.411 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T22:14:31.411 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T22:14:31.411 INFO:teuthology.orchestra.run.smithi062.stdout:max_xattr_size 65536 2024-04-20T22:14:31.411 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T22:14:31.411 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T22:14:31.411 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 81 2024-04-20T22:14:31.411 INFO:teuthology.orchestra.run.smithi062.stdout:snap_rstat 1 2024-04-20T22:14:31.411 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:14:31.411 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 2 2024-04-20T22:14:31.411 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1 2024-04-20T22:14:31.411 INFO:teuthology.orchestra.run.smithi062.stdout:up {} 2024-04-20T22:14:31.411 INFO:teuthology.orchestra.run.smithi062.stdout:failed 0,1 2024-04-20T22:14:31.412 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T22:14:31.412 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T22:14:31.412 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T22:14:31.412 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T22:14:31.412 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data enabled 2024-04-20T22:14:31.412 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T22:14:31.412 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T22:14:31.412 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 1 2024-04-20T22:14:31.412 INFO:teuthology.orchestra.run.smithi062.stdout:qdb_cluster leader: 0 members: 2024-04-20T22:14:31.412 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:14:31.412 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:14:31.412 INFO:teuthology.orchestra.run.smithi062.stdout:Standby daemons: 2024-04-20T22:14:31.412 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:14:31.412 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ryqblb{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6826/4113916205,v1:172.21.15.62:6827/4113916205] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:14:31.412 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ozizwi{-1:34476} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6828/3801975165,v1:172.21.15.62:6829/3801975165] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:14:31.413 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.dvimer{-1:44727} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6824/3576699232,v1:172.21.15.189:6825/3576699232] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:14:31.413 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.xbgspw{-1:45075} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6826/212905792,v1:172.21.15.189:6827/212905792] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:14:31.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:31.412+0000 7fbdfc600700 1 -- 172.21.15.62:0/2182910826 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fbddc06ecc0 msgr2=0x7fbddc071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:14:31.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:31.412+0000 7fbdfc600700 1 --2- 172.21.15.62:0/2182910826 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fbddc06ecc0 0x7fbddc071170 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fbde8006fd0 tx=0x7fbde8008040 comp rx=0 tx=0).stop 2024-04-20T22:14:31.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:31.412+0000 7fbdfc600700 1 -- 172.21.15.62:0/2182910826 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbdf4108470 msgr2=0x7fbdf4198740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:14:31.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:31.412+0000 7fbdfc600700 1 --2- 172.21.15.62:0/2182910826 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbdf4108470 0x7fbdf4198740 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7fbde000b5c0 tx=0x7fbde000be00 comp rx=0 tx=0).stop 2024-04-20T22:14:31.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:31.412+0000 7fbdfc600700 1 -- 172.21.15.62:0/2182910826 shutdown_connections 2024-04-20T22:14:31.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:31.412+0000 7fbdfc600700 1 --2- 172.21.15.62:0/2182910826 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fbddc06ecc0 0x7fbddc071170 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:14:31.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:31.412+0000 7fbdfc600700 1 --2- 172.21.15.62:0/2182910826 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbdf4102470 0x7fbdf4198200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:14:31.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:31.412+0000 7fbdfc600700 1 --2- 172.21.15.62:0/2182910826 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbdf4108470 0x7fbdf4198740 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:14:31.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:31.412+0000 7fbdfc600700 1 -- 172.21.15.62:0/2182910826 >> 172.21.15.62:0/2182910826 conn(0x7fbdf40fdf90 msgr2=0x7fbdf40febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:14:31.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:31.413+0000 7fbdfc600700 1 -- 172.21.15.62:0/2182910826 shutdown_connections 2024-04-20T22:14:31.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:31.413+0000 7fbdfc600700 1 -- 172.21.15.62:0/2182910826 wait complete. 2024-04-20T22:14:31.414 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 29 2024-04-20T22:14:31.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:14:31 smithi062 ceph-mon[76322]: from='client.36078 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:14:31.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:14:31 smithi062 ceph-mon[76322]: from='client.36082 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:14:31.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:14:31 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/1147559675' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:14:31.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:31.564+0000 7fa960200700 1 -- 172.21.15.62:0/2131507858 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa958068540 msgr2=0x7fa9580689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:14:31.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:31.564+0000 7fa960200700 1 --2- 172.21.15.62:0/2131507858 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa958068540 0x7fa9580689b0 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7fa944009910 tx=0x7fa944009c20 comp rx=0 tx=0).stop 2024-04-20T22:14:31.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:31.564+0000 7fa960200700 1 -- 172.21.15.62:0/2131507858 shutdown_connections 2024-04-20T22:14:31.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:31.564+0000 7fa960200700 1 --2- 172.21.15.62:0/2131507858 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa958068540 0x7fa9580689b0 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:14:31.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:31.564+0000 7fa960200700 1 --2- 172.21.15.62:0/2131507858 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa958106320 0x7fa9581066f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:14:31.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:31.564+0000 7fa960200700 1 -- 172.21.15.62:0/2131507858 >> 172.21.15.62:0/2131507858 conn(0x7fa958075130 msgr2=0x7fa958075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:14:31.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:31.564+0000 7fa960200700 1 -- 172.21.15.62:0/2131507858 shutdown_connections 2024-04-20T22:14:31.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:31.565+0000 7fa960200700 1 -- 172.21.15.62:0/2131507858 wait complete. 2024-04-20T22:14:31.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:31.566+0000 7fa960200700 1 Processor -- start 2024-04-20T22:14:31.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:31.566+0000 7fa960200700 1 -- start start 2024-04-20T22:14:31.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:31.566+0000 7fa960200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa958068540 0x7fa958195f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:14:31.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:31.566+0000 7fa960200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa958106320 0x7fa9581964a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:14:31.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:31.566+0000 7fa960200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa958196b80 con 0x7fa958068540 2024-04-20T22:14:31.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:31.566+0000 7fa960200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa95819a910 con 0x7fa958106320 2024-04-20T22:14:31.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:31.566+0000 7fa95d800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa958068540 0x7fa958195f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:14:31.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:31.567+0000 7fa95d800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa958068540 0x7fa958195f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44234/0 (socket says 172.21.15.62:44234) 2024-04-20T22:14:31.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:31.567+0000 7fa95d800700 1 -- 172.21.15.62:0/631627539 learned_addr learned my addr 172.21.15.62:0/631627539 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:14:31.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:31.567+0000 7fa95ce00700 1 --2- 172.21.15.62:0/631627539 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa958106320 0x7fa9581964a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:14:31.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:31.567+0000 7fa95d800700 1 -- 172.21.15.62:0/631627539 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa958106320 msgr2=0x7fa9581964a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:14:31.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:31.567+0000 7fa95d800700 1 --2- 172.21.15.62:0/631627539 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa958106320 0x7fa9581964a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:14:31.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:31.567+0000 7fa95d800700 1 -- 172.21.15.62:0/631627539 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9440095f0 con 0x7fa958068540 2024-04-20T22:14:31.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:31.567+0000 7fa95ce00700 1 --2- 172.21.15.62:0/631627539 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa958106320 0x7fa9581964a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T22:14:31.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:31.567+0000 7fa95d800700 1 --2- 172.21.15.62:0/631627539 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa958068540 0x7fa958195f60 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7fa94c00b4b0 tx=0x7fa94c00b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:14:31.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:31.568+0000 7fa956000700 1 -- 172.21.15.62:0/631627539 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa94c011800 con 0x7fa958068540 2024-04-20T22:14:31.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:31.568+0000 7fa960200700 1 -- 172.21.15.62:0/631627539 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa95819abf0 con 0x7fa958068540 2024-04-20T22:14:31.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:31.568+0000 7fa956000700 1 -- 172.21.15.62:0/631627539 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa94c011e20 con 0x7fa958068540 2024-04-20T22:14:31.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:31.568+0000 7fa956000700 1 -- 172.21.15.62:0/631627539 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa94c00f2e0 con 0x7fa958068540 2024-04-20T22:14:31.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:31.568+0000 7fa960200700 1 -- 172.21.15.62:0/631627539 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa95819b140 con 0x7fa958068540 2024-04-20T22:14:31.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:31.569+0000 7fa960200700 1 -- 172.21.15.62:0/631627539 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa958108770 con 0x7fa958068540 2024-04-20T22:14:31.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:31.572+0000 7fa956000700 1 -- 172.21.15.62:0/631627539 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fa94c010380 con 0x7fa958068540 2024-04-20T22:14:31.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:31.573+0000 7fa956000700 1 --2- 172.21.15.62:0/631627539 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa94006ecc0 0x7fa940071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:14:31.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:31.573+0000 7fa956000700 1 -- 172.21.15.62:0/631627539 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fa94c08eaa0 con 0x7fa958068540 2024-04-20T22:14:31.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:31.573+0000 7fa95ce00700 1 --2- 172.21.15.62:0/631627539 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa94006ecc0 0x7fa940071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:14:31.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:31.574+0000 7fa95ce00700 1 --2- 172.21.15.62:0/631627539 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa94006ecc0 0x7fa940071170 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fa94400b5c0 tx=0x7fa944005960 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:14:31.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:31.578+0000 7fa956000700 1 -- 172.21.15.62:0/631627539 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fa94c0577b0 con 0x7fa958068540 2024-04-20T22:14:31.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:14:31 smithi189 ceph-mon[57022]: from='client.36078 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:14:31.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:14:31 smithi189 ceph-mon[57022]: from='client.36082 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:14:31.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:14:31 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/1147559675' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:14:31.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:31.834+0000 7fa960200700 1 -- 172.21.15.62:0/631627539 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa95819b420 con 0x7fa94006ecc0 2024-04-20T22:14:31.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:31.836+0000 7fa956000700 1 -- 172.21.15.62:0/631627539 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fa95819b420 con 0x7fa94006ecc0 2024-04-20T22:14:31.837 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:14:31.837 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T22:14:31.837 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T22:14:31.837 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-20T22:14:31.837 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T22:14:31.837 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr", 2024-04-20T22:14:31.837 INFO:teuthology.orchestra.run.smithi062.stdout: "osd", 2024-04-20T22:14:31.837 INFO:teuthology.orchestra.run.smithi062.stdout: "mds", 2024-04-20T22:14:31.837 INFO:teuthology.orchestra.run.smithi062.stdout: "crash", 2024-04-20T22:14:31.838 INFO:teuthology.orchestra.run.smithi062.stdout: "mon" 2024-04-20T22:14:31.838 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T22:14:31.838 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "16/23 daemons upgraded", 2024-04-20T22:14:31.838 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading mds daemons", 2024-04-20T22:14:31.838 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T22:14:31.838 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:14:31.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:31.840+0000 7fa960200700 1 -- 172.21.15.62:0/631627539 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa94006ecc0 msgr2=0x7fa940071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:14:31.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:31.840+0000 7fa960200700 1 --2- 172.21.15.62:0/631627539 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa94006ecc0 0x7fa940071170 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fa94400b5c0 tx=0x7fa944005960 comp rx=0 tx=0).stop 2024-04-20T22:14:31.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:31.840+0000 7fa960200700 1 -- 172.21.15.62:0/631627539 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa958068540 msgr2=0x7fa958195f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:14:31.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:31.840+0000 7fa960200700 1 --2- 172.21.15.62:0/631627539 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa958068540 0x7fa958195f60 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7fa94c00b4b0 tx=0x7fa94c00b870 comp rx=0 tx=0).stop 2024-04-20T22:14:31.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:31.840+0000 7fa960200700 1 -- 172.21.15.62:0/631627539 shutdown_connections 2024-04-20T22:14:31.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:31.840+0000 7fa960200700 1 --2- 172.21.15.62:0/631627539 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa94006ecc0 0x7fa940071170 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:14:31.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:31.840+0000 7fa960200700 1 --2- 172.21.15.62:0/631627539 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa958068540 0x7fa958195f60 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:14:31.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:31.840+0000 7fa960200700 1 --2- 172.21.15.62:0/631627539 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa958106320 0x7fa9581964a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:14:31.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:31.840+0000 7fa960200700 1 -- 172.21.15.62:0/631627539 >> 172.21.15.62:0/631627539 conn(0x7fa958075130 msgr2=0x7fa9580fecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:14:31.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:31.840+0000 7fa960200700 1 -- 172.21.15.62:0/631627539 shutdown_connections 2024-04-20T22:14:31.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:31.840+0000 7fa960200700 1 -- 172.21.15.62:0/631627539 wait complete. 2024-04-20T22:14:32.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:32.001+0000 7f18f3200700 1 -- 172.21.15.62:0/825705741 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f18f4068540 msgr2=0x7f18f40689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:14:32.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:32.001+0000 7f18f3200700 1 --2- 172.21.15.62:0/825705741 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f18f4068540 0x7f18f40689b0 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f18dc009910 tx=0x7f18dc009c20 comp rx=0 tx=0).stop 2024-04-20T22:14:32.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:32.002+0000 7f18f3200700 1 -- 172.21.15.62:0/825705741 shutdown_connections 2024-04-20T22:14:32.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:32.002+0000 7f18f3200700 1 --2- 172.21.15.62:0/825705741 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f18f4068540 0x7f18f40689b0 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:14:32.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:32.002+0000 7f18f3200700 1 --2- 172.21.15.62:0/825705741 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18f41061b0 0x7f18f4106580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:14:32.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:32.002+0000 7f18f3200700 1 -- 172.21.15.62:0/825705741 >> 172.21.15.62:0/825705741 conn(0x7f18f4074fa0 msgr2=0x7f18f40753a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:14:32.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:32.002+0000 7f18f3200700 1 -- 172.21.15.62:0/825705741 shutdown_connections 2024-04-20T22:14:32.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:32.002+0000 7f18f3200700 1 -- 172.21.15.62:0/825705741 wait complete. 2024-04-20T22:14:32.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:32.003+0000 7f18f3200700 1 Processor -- start 2024-04-20T22:14:32.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:32.003+0000 7f18f3200700 1 -- start start 2024-04-20T22:14:32.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:32.003+0000 7f18f3200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f18f4068540 0x7f18f419b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:14:32.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:32.003+0000 7f18f3200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18f41061b0 0x7f18f419be90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:14:32.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:32.003+0000 7f18f3200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18f419c520 con 0x7f18f4068540 2024-04-20T22:14:32.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:32.003+0000 7f18f3200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18f41959d0 con 0x7f18f41061b0 2024-04-20T22:14:32.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:32.004+0000 7f18f1e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f18f4068540 0x7f18f419b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:14:32.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:32.004+0000 7f18f1e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f18f4068540 0x7f18f419b950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44258/0 (socket says 172.21.15.62:44258) 2024-04-20T22:14:32.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:32.004+0000 7f18f1e00700 1 -- 172.21.15.62:0/191548642 learned_addr learned my addr 172.21.15.62:0/191548642 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:14:32.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:32.004+0000 7f18e9400700 1 --2- 172.21.15.62:0/191548642 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18f41061b0 0x7f18f419be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:14:32.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:32.004+0000 7f18f1e00700 1 -- 172.21.15.62:0/191548642 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18f41061b0 msgr2=0x7f18f419be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:14:32.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:32.004+0000 7f18f1e00700 1 --2- 172.21.15.62:0/191548642 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18f41061b0 0x7f18f419be90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:14:32.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:32.004+0000 7f18f1e00700 1 -- 172.21.15.62:0/191548642 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18dc0095f0 con 0x7f18f4068540 2024-04-20T22:14:32.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:32.005+0000 7f18f1e00700 1 --2- 172.21.15.62:0/191548642 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f18f4068540 0x7f18f419b950 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f18e400c8a0 tx=0x7f18e400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:14:32.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:32.005+0000 7f18eb400700 1 -- 172.21.15.62:0/191548642 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f18e400cea0 con 0x7f18f4068540 2024-04-20T22:14:32.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:32.005+0000 7f18f3200700 1 -- 172.21.15.62:0/191548642 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18f4195cb0 con 0x7f18f4068540 2024-04-20T22:14:32.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:32.005+0000 7f18eb400700 1 -- 172.21.15.62:0/191548642 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f18e4004830 con 0x7f18f4068540 2024-04-20T22:14:32.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:32.005+0000 7f18eb400700 1 -- 172.21.15.62:0/191548642 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f18e40055b0 con 0x7f18f4068540 2024-04-20T22:14:32.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:32.005+0000 7f18f3200700 1 -- 172.21.15.62:0/191548642 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18f4196280 con 0x7f18f4068540 2024-04-20T22:14:32.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:32.006+0000 7f18f3200700 1 -- 172.21.15.62:0/191548642 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18f4062380 con 0x7f18f4068540 2024-04-20T22:14:32.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:32.007+0000 7f18eb400700 1 -- 172.21.15.62:0/191548642 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f18e40049a0 con 0x7f18f4068540 2024-04-20T22:14:32.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:32.008+0000 7f18eb400700 1 --2- 172.21.15.62:0/191548642 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f18d406ecc0 0x7f18d4071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:14:32.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:32.008+0000 7f18eb400700 1 -- 172.21.15.62:0/191548642 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f18e408ebf0 con 0x7f18f4068540 2024-04-20T22:14:32.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:32.008+0000 7f18e9400700 1 --2- 172.21.15.62:0/191548642 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f18d406ecc0 0x7f18d4071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:14:32.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:32.009+0000 7f18e9400700 1 --2- 172.21.15.62:0/191548642 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f18d406ecc0 0x7f18d4071170 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f18f410abe0 tx=0x7f18dc01a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:14:32.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:32.013+0000 7f18eb400700 1 -- 172.21.15.62:0/191548642 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f18e405b430 con 0x7f18f4068540 2024-04-20T22:14:32.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:32.345+0000 7f18f3200700 1 -- 172.21.15.62:0/191548642 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f18f4196ee0 con 0x7f18f4068540 2024-04-20T22:14:32.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:32.346+0000 7f18eb400700 1 -- 172.21.15.62:0/191548642 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7f18e4004c50 con 0x7f18f4068540 2024-04-20T22:14:32.346 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-20T22:14:32.346 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-20T22:14:32.347 INFO:teuthology.orchestra.run.smithi062.stdout: host smithi189 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi189/config 2024-04-20T22:14:32.347 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-20T22:14:32.347 INFO:teuthology.orchestra.run.smithi062.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi189 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp11670v2q:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:14:32.347 INFO:teuthology.orchestra.run.smithi062.stdout:Traceback (most recent call last): 2024-04-20T22:14:32.347 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-20T22:14:32.347 INFO:teuthology.orchestra.run.smithi062.stdout: return _run_code(code, main_globals, None, 2024-04-20T22:14:32.347 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-20T22:14:32.347 INFO:teuthology.orchestra.run.smithi062.stdout: exec(code, run_globals) 2024-04-20T22:14:32.347 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5618, in 2024-04-20T22:14:32.347 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5606, in main 2024-04-20T22:14:32.347 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 408, in _infer_config 2024-04-20T22:14:32.347 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 323, in _infer_fsid 2024-04-20T22:14:32.348 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 436, in _infer_image 2024-04-20T22:14:32.348 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 310, in _validate_fsid 2024-04-20T22:14:32.348 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 3359, in command_ceph_volume 2024-04-20T22:14:32.348 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-20T22:14:32.348 INFO:teuthology.orchestra.run.smithi062.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi189 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp11670v2q:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:14:32.348 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-20T22:14:32.348 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is degraded 2024-04-20T22:14:32.348 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T22:14:32.348 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T22:14:32.348 INFO:teuthology.orchestra.run.smithi062.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-20T22:14:32.348 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-20T22:14:32.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:32.349+0000 7f18f3200700 1 -- 172.21.15.62:0/191548642 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f18d406ecc0 msgr2=0x7f18d4071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:14:32.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:32.349+0000 7f18f3200700 1 --2- 172.21.15.62:0/191548642 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f18d406ecc0 0x7f18d4071170 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f18f410abe0 tx=0x7f18dc01a040 comp rx=0 tx=0).stop 2024-04-20T22:14:32.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:32.349+0000 7f18f3200700 1 -- 172.21.15.62:0/191548642 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f18f4068540 msgr2=0x7f18f419b950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:14:32.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:32.349+0000 7f18f3200700 1 --2- 172.21.15.62:0/191548642 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f18f4068540 0x7f18f419b950 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f18e400c8a0 tx=0x7f18e400cc60 comp rx=0 tx=0).stop 2024-04-20T22:14:32.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:32.350+0000 7f18f3200700 1 -- 172.21.15.62:0/191548642 shutdown_connections 2024-04-20T22:14:32.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:32.350+0000 7f18f3200700 1 --2- 172.21.15.62:0/191548642 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f18d406ecc0 0x7f18d4071170 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:14:32.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:32.350+0000 7f18f3200700 1 --2- 172.21.15.62:0/191548642 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f18f4068540 0x7f18f419b950 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:14:32.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:32.350+0000 7f18f3200700 1 --2- 172.21.15.62:0/191548642 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18f41061b0 0x7f18f419be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:14:32.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:32.350+0000 7f18f3200700 1 -- 172.21.15.62:0/191548642 >> 172.21.15.62:0/191548642 conn(0x7f18f4074fa0 msgr2=0x7f18f40fe640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:14:32.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:32.350+0000 7f18f3200700 1 -- 172.21.15.62:0/191548642 shutdown_connections 2024-04-20T22:14:32.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:14:32.350+0000 7f18f3200700 1 -- 172.21.15.62:0/191548642 wait complete. 2024-04-20T22:14:32.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:14:32 smithi062 ceph-mon[76322]: from='client.36086 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:14:32.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:14:32 smithi062 ceph-mon[76322]: pgmap v1180: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:14:32.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:14:32 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/2182910826' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:14:32.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:14:32 smithi189 ceph-mon[57022]: from='client.36086 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:14:32.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:14:32 smithi189 ceph-mon[57022]: pgmap v1180: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:14:32.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:14:32 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/2182910826' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:14:33.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:14:33 smithi189 ceph-mon[57022]: from='client.36098 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:14:33.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:14:33 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/191548642' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:14:33.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:14:33 smithi062 ceph-mon[76322]: from='client.36098 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:14:33.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:14:33 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/191548642' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:14:34.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:14:34 smithi189 ceph-mon[57022]: pgmap v1181: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:14:34.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:14:34 smithi062 ceph-mon[76322]: pgmap v1181: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:14:35.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:14:35 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:14:35.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:14:35 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:14:36.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:14:36 smithi189 ceph-mon[57022]: pgmap v1182: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:14:36.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:14:36 smithi062 ceph-mon[76322]: pgmap v1182: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:14:38.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:14:38 smithi062 ceph-mon[76322]: pgmap v1183: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:14:38.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:14:38 smithi189 ceph-mon[57022]: pgmap v1183: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:14:40.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:14:40 smithi189 ceph-mon[57022]: pgmap v1184: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:14:40.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:14:40 smithi062 ceph-mon[76322]: pgmap v1184: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:14:42.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:14:42 smithi189 ceph-mon[57022]: pgmap v1185: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:14:42.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:14:42 smithi062 ceph-mon[76322]: pgmap v1185: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:14:44.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:14:44 smithi062 ceph-mon[76322]: pgmap v1186: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:14:44.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:14:44 smithi189 ceph-mon[57022]: pgmap v1186: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:14:46.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:14:46 smithi189 ceph-mon[57022]: pgmap v1187: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:14:46.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:14:46 smithi062 ceph-mon[76322]: pgmap v1187: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:14:48.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:14:48 smithi062 ceph-mon[76322]: pgmap v1188: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:14:48.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:14:48 smithi189 ceph-mon[57022]: pgmap v1188: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:14:49.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:14:49 smithi062 ceph-mon[76322]: pgmap v1189: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:14:49.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:14:49 smithi189 ceph-mon[57022]: pgmap v1189: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:14:50.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:14:50 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:14:50.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:14:50 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:14:51.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:14:51 smithi189 ceph-mon[57022]: pgmap v1190: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:14:51.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:14:51 smithi062 ceph-mon[76322]: pgmap v1190: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:14:54.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:14:54 smithi062 ceph-mon[76322]: pgmap v1191: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:14:54.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:14:54 smithi189 ceph-mon[57022]: pgmap v1191: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:14:56.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:14:56 smithi062 ceph-mon[76322]: pgmap v1192: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:14:56.596 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:14:56 smithi189 ceph-mon[57022]: pgmap v1192: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:14:58.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:14:58 smithi062 ceph-mon[76322]: pgmap v1193: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:14:58.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:14:58 smithi189 ceph-mon[57022]: pgmap v1193: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:15:00.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:15:00 smithi189 ceph-mon[57022]: pgmap v1194: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:15:00.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:15:00 smithi062 ceph-mon[76322]: pgmap v1194: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:15:02.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:02.529+0000 7fed99800700 1 -- 172.21.15.62:0/2646090104 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fed94102470 msgr2=0x7fed941028e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:15:02.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:02.529+0000 7fed99800700 1 --2- 172.21.15.62:0/2646090104 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fed94102470 0x7fed941028e0 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7fed7c009910 tx=0x7fed7c009c20 comp rx=0 tx=0).stop 2024-04-20T22:15:02.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:02.529+0000 7fed99800700 1 -- 172.21.15.62:0/2646090104 shutdown_connections 2024-04-20T22:15:02.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:02.529+0000 7fed99800700 1 --2- 172.21.15.62:0/2646090104 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fed94102470 0x7fed941028e0 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:15:02.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:02.529+0000 7fed99800700 1 --2- 172.21.15.62:0/2646090104 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fed94108470 0x7fed94108840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:15:02.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:02.529+0000 7fed99800700 1 -- 172.21.15.62:0/2646090104 >> 172.21.15.62:0/2646090104 conn(0x7fed940fdf90 msgr2=0x7fed941003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:15:02.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:02.529+0000 7fed99800700 1 -- 172.21.15.62:0/2646090104 shutdown_connections 2024-04-20T22:15:02.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:02.529+0000 7fed99800700 1 -- 172.21.15.62:0/2646090104 wait complete. 2024-04-20T22:15:02.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:02.531+0000 7fed99800700 1 Processor -- start 2024-04-20T22:15:02.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:02.531+0000 7fed99800700 1 -- start start 2024-04-20T22:15:02.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:02.531+0000 7fed99800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fed94102470 0x7fed94198190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:15:02.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:02.531+0000 7fed99800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fed94108470 0x7fed941986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:15:02.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:02.531+0000 7fed99800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed94198db0 con 0x7fed94108470 2024-04-20T22:15:02.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:02.531+0000 7fed99800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed9419cb40 con 0x7fed94102470 2024-04-20T22:15:02.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:02.532+0000 7fed8be00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fed94108470 0x7fed941986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:15:02.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:02.532+0000 7fed8be00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fed94108470 0x7fed941986d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47182/0 (socket says 172.21.15.62:47182) 2024-04-20T22:15:02.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:02.532+0000 7fed8be00700 1 -- 172.21.15.62:0/2172253735 learned_addr learned my addr 172.21.15.62:0/2172253735 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:15:02.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:02.532+0000 7fed92a00700 1 --2- 172.21.15.62:0/2172253735 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fed94102470 0x7fed94198190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:15:02.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:02.532+0000 7fed8be00700 1 -- 172.21.15.62:0/2172253735 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fed94102470 msgr2=0x7fed94198190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:15:02.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:02.532+0000 7fed8be00700 1 --2- 172.21.15.62:0/2172253735 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fed94102470 0x7fed94198190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:15:02.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:02.532+0000 7fed8be00700 1 -- 172.21.15.62:0/2172253735 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed7c0095f0 con 0x7fed94108470 2024-04-20T22:15:02.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:02.532+0000 7fed92a00700 1 --2- 172.21.15.62:0/2172253735 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fed94102470 0x7fed94198190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T22:15:02.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:02.532+0000 7fed8be00700 1 --2- 172.21.15.62:0/2172253735 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fed94108470 0x7fed941986d0 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7fed7c006010 tx=0x7fed7c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:15:02.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:02.533+0000 7fed8b400700 1 -- 172.21.15.62:0/2172253735 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fed7c01d070 con 0x7fed94108470 2024-04-20T22:15:02.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:02.533+0000 7fed8b400700 1 -- 172.21.15.62:0/2172253735 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fed7c004fb0 con 0x7fed94108470 2024-04-20T22:15:02.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:02.533+0000 7fed99800700 1 -- 172.21.15.62:0/2172253735 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed9419cdc0 con 0x7fed94108470 2024-04-20T22:15:02.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:02.533+0000 7fed8b400700 1 -- 172.21.15.62:0/2172253735 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fed7c022ea0 con 0x7fed94108470 2024-04-20T22:15:02.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:02.533+0000 7fed99800700 1 -- 172.21.15.62:0/2172253735 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed9419d2b0 con 0x7fed94108470 2024-04-20T22:15:02.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:02.534+0000 7fed99800700 1 -- 172.21.15.62:0/2172253735 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed9410aa00 con 0x7fed94108470 2024-04-20T22:15:02.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:02.535+0000 7fed8b400700 1 -- 172.21.15.62:0/2172253735 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fed7c022470 con 0x7fed94108470 2024-04-20T22:15:02.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:02.536+0000 7fed8b400700 1 --2- 172.21.15.62:0/2172253735 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fed7806ecc0 0x7fed78071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:15:02.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:02.536+0000 7fed8b400700 1 -- 172.21.15.62:0/2172253735 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fed7c0904a0 con 0x7fed94108470 2024-04-20T22:15:02.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:02.536+0000 7fed92a00700 1 --2- 172.21.15.62:0/2172253735 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fed7806ecc0 0x7fed78071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:15:02.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:02.537+0000 7fed92a00700 1 --2- 172.21.15.62:0/2172253735 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fed7806ecc0 0x7fed78071170 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fed84007900 tx=0x7fed84008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:15:02.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:02.542+0000 7fed8b400700 1 -- 172.21.15.62:0/2172253735 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fed7c059260 con 0x7fed94108470 2024-04-20T22:15:02.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:15:02 smithi189 ceph-mon[57022]: pgmap v1195: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:15:02.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:15:02 smithi062 ceph-mon[76322]: pgmap v1195: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:15:02.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:02.800+0000 7fed99800700 1 -- 172.21.15.62:0/2172253735 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fed9419d4d0 con 0x7fed7806ecc0 2024-04-20T22:15:02.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:02.803+0000 7fed8b400700 1 -- 172.21.15.62:0/2172253735 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fed9419d4d0 con 0x7fed7806ecc0 2024-04-20T22:15:02.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:02.805+0000 7fed99800700 1 -- 172.21.15.62:0/2172253735 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fed7806ecc0 msgr2=0x7fed78071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:15:02.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:02.805+0000 7fed99800700 1 --2- 172.21.15.62:0/2172253735 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fed7806ecc0 0x7fed78071170 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fed84007900 tx=0x7fed84008040 comp rx=0 tx=0).stop 2024-04-20T22:15:02.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:02.805+0000 7fed99800700 1 -- 172.21.15.62:0/2172253735 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fed94108470 msgr2=0x7fed941986d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:15:02.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:02.805+0000 7fed99800700 1 --2- 172.21.15.62:0/2172253735 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fed94108470 0x7fed941986d0 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7fed7c006010 tx=0x7fed7c00be20 comp rx=0 tx=0).stop 2024-04-20T22:15:02.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:02.806+0000 7fed99800700 1 -- 172.21.15.62:0/2172253735 shutdown_connections 2024-04-20T22:15:02.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:02.806+0000 7fed99800700 1 --2- 172.21.15.62:0/2172253735 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fed7806ecc0 0x7fed78071170 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:15:02.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:02.806+0000 7fed99800700 1 --2- 172.21.15.62:0/2172253735 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fed94102470 0x7fed94198190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:15:02.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:02.806+0000 7fed99800700 1 --2- 172.21.15.62:0/2172253735 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fed94108470 0x7fed941986d0 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:15:02.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:02.806+0000 7fed99800700 1 -- 172.21.15.62:0/2172253735 >> 172.21.15.62:0/2172253735 conn(0x7fed940fdf90 msgr2=0x7fed940feb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:15:02.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:02.806+0000 7fed99800700 1 -- 172.21.15.62:0/2172253735 shutdown_connections 2024-04-20T22:15:02.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:02.806+0000 7fed99800700 1 -- 172.21.15.62:0/2172253735 wait complete. 2024-04-20T22:15:02.822 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T22:15:02.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:02.974+0000 7fd55fe00700 1 -- 172.21.15.62:0/734510370 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd5581026a0 msgr2=0x7fd558102a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:15:02.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:02.974+0000 7fd55fe00700 1 --2- 172.21.15.62:0/734510370 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd5581026a0 0x7fd558102a70 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7fd54c009910 tx=0x7fd54c009c20 comp rx=0 tx=0).stop 2024-04-20T22:15:02.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:02.974+0000 7fd55fe00700 1 -- 172.21.15.62:0/734510370 shutdown_connections 2024-04-20T22:15:02.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:02.974+0000 7fd55fe00700 1 --2- 172.21.15.62:0/734510370 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd558103040 0x7fd558107010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:15:02.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:02.974+0000 7fd55fe00700 1 --2- 172.21.15.62:0/734510370 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd5581026a0 0x7fd558102a70 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:15:02.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:02.974+0000 7fd55fe00700 1 -- 172.21.15.62:0/734510370 >> 172.21.15.62:0/734510370 conn(0x7fd5580fdf40 msgr2=0x7fd558100350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:15:02.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:02.974+0000 7fd55fe00700 1 -- 172.21.15.62:0/734510370 shutdown_connections 2024-04-20T22:15:02.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:02.974+0000 7fd55fe00700 1 -- 172.21.15.62:0/734510370 wait complete. 2024-04-20T22:15:02.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:02.975+0000 7fd55fe00700 1 Processor -- start 2024-04-20T22:15:02.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:02.975+0000 7fd55fe00700 1 -- start start 2024-04-20T22:15:02.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:02.976+0000 7fd55fe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd5581026a0 0x7fd558198130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:15:02.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:02.976+0000 7fd55fe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd558103040 0x7fd558198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:15:02.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:02.976+0000 7fd55fe00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd558198d50 con 0x7fd558103040 2024-04-20T22:15:02.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:02.976+0000 7fd55fe00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd55819cae0 con 0x7fd5581026a0 2024-04-20T22:15:02.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:02.976+0000 7fd55ca00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd558103040 0x7fd558198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:15:02.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:02.976+0000 7fd55ca00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd558103040 0x7fd558198670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47200/0 (socket says 172.21.15.62:47200) 2024-04-20T22:15:02.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:02.976+0000 7fd55ca00700 1 -- 172.21.15.62:0/4201710417 learned_addr learned my addr 172.21.15.62:0/4201710417 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:15:02.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:02.976+0000 7fd55d400700 1 --2- 172.21.15.62:0/4201710417 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd5581026a0 0x7fd558198130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:15:02.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:02.977+0000 7fd55ca00700 1 -- 172.21.15.62:0/4201710417 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd5581026a0 msgr2=0x7fd558198130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:15:02.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:02.977+0000 7fd55ca00700 1 --2- 172.21.15.62:0/4201710417 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd5581026a0 0x7fd558198130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:15:02.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:02.977+0000 7fd55ca00700 1 -- 172.21.15.62:0/4201710417 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd54c0095f0 con 0x7fd558103040 2024-04-20T22:15:02.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:02.977+0000 7fd55d400700 1 --2- 172.21.15.62:0/4201710417 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd5581026a0 0x7fd558198130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T22:15:02.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:02.977+0000 7fd55ca00700 1 --2- 172.21.15.62:0/4201710417 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd558103040 0x7fd558198670 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fd54400d900 tx=0x7fd54400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:15:02.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:02.977+0000 7fd556000700 1 -- 172.21.15.62:0/4201710417 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd544009900 con 0x7fd558103040 2024-04-20T22:15:02.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:02.977+0000 7fd556000700 1 -- 172.21.15.62:0/4201710417 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd544010460 con 0x7fd558103040 2024-04-20T22:15:02.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:02.977+0000 7fd556000700 1 -- 172.21.15.62:0/4201710417 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd54400f5f0 con 0x7fd558103040 2024-04-20T22:15:02.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:02.977+0000 7fd55fe00700 1 -- 172.21.15.62:0/4201710417 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd55819cdc0 con 0x7fd558103040 2024-04-20T22:15:02.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:02.977+0000 7fd55fe00700 1 -- 172.21.15.62:0/4201710417 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd55819d310 con 0x7fd558103040 2024-04-20T22:15:02.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:02.979+0000 7fd55fe00700 1 -- 172.21.15.62:0/4201710417 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd55810a9b0 con 0x7fd558103040 2024-04-20T22:15:02.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:02.980+0000 7fd556000700 1 -- 172.21.15.62:0/4201710417 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fd5440105d0 con 0x7fd558103040 2024-04-20T22:15:02.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:02.981+0000 7fd556000700 1 --2- 172.21.15.62:0/4201710417 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd54006ecc0 0x7fd540071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:15:02.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:02.981+0000 7fd556000700 1 -- 172.21.15.62:0/4201710417 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fd54408eca0 con 0x7fd558103040 2024-04-20T22:15:02.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:02.981+0000 7fd55d400700 1 --2- 172.21.15.62:0/4201710417 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd54006ecc0 0x7fd540071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:15:02.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:02.982+0000 7fd55d400700 1 --2- 172.21.15.62:0/4201710417 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd54006ecc0 0x7fd540071170 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fd54c00b5c0 tx=0x7fd54c0053a0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:15:02.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:02.985+0000 7fd556000700 1 -- 172.21.15.62:0/4201710417 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fd544057a30 con 0x7fd558103040 2024-04-20T22:15:03.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:03.243+0000 7fd55fe00700 1 -- 172.21.15.62:0/4201710417 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd55819d580 con 0x7fd54006ecc0 2024-04-20T22:15:03.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:03.245+0000 7fd556000700 1 -- 172.21.15.62:0/4201710417 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fd55819d580 con 0x7fd54006ecc0 2024-04-20T22:15:03.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:03.247+0000 7fd55fe00700 1 -- 172.21.15.62:0/4201710417 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd54006ecc0 msgr2=0x7fd540071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:15:03.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:03.247+0000 7fd55fe00700 1 --2- 172.21.15.62:0/4201710417 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd54006ecc0 0x7fd540071170 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fd54c00b5c0 tx=0x7fd54c0053a0 comp rx=0 tx=0).stop 2024-04-20T22:15:03.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:03.247+0000 7fd55fe00700 1 -- 172.21.15.62:0/4201710417 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd558103040 msgr2=0x7fd558198670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:15:03.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:03.247+0000 7fd55fe00700 1 --2- 172.21.15.62:0/4201710417 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd558103040 0x7fd558198670 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fd54400d900 tx=0x7fd54400dcc0 comp rx=0 tx=0).stop 2024-04-20T22:15:03.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:03.247+0000 7fd55fe00700 1 -- 172.21.15.62:0/4201710417 shutdown_connections 2024-04-20T22:15:03.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:03.247+0000 7fd55fe00700 1 --2- 172.21.15.62:0/4201710417 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd54006ecc0 0x7fd540071170 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:15:03.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:03.247+0000 7fd55fe00700 1 --2- 172.21.15.62:0/4201710417 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd5581026a0 0x7fd558198130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:15:03.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:03.248+0000 7fd55fe00700 1 --2- 172.21.15.62:0/4201710417 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd558103040 0x7fd558198670 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:15:03.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:03.248+0000 7fd55fe00700 1 -- 172.21.15.62:0/4201710417 >> 172.21.15.62:0/4201710417 conn(0x7fd5580fdf40 msgr2=0x7fd5580ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:15:03.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:03.248+0000 7fd55fe00700 1 -- 172.21.15.62:0/4201710417 shutdown_connections 2024-04-20T22:15:03.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:03.248+0000 7fd55fe00700 1 -- 172.21.15.62:0/4201710417 wait complete. 2024-04-20T22:15:03.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:03.413+0000 7f450c200700 1 -- 172.21.15.62:0/1645191168 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f45040739a0 msgr2=0x7f450410c5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:15:03.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:03.413+0000 7f450c200700 1 --2- 172.21.15.62:0/1645191168 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f45040739a0 0x7f450410c5e0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f44f0009940 tx=0x7f44f0009c50 comp rx=0 tx=0).stop 2024-04-20T22:15:03.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:03.414+0000 7f450c200700 1 -- 172.21.15.62:0/1645191168 shutdown_connections 2024-04-20T22:15:03.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:03.414+0000 7f450c200700 1 --2- 172.21.15.62:0/1645191168 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f45040739a0 0x7f450410c5e0 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:15:03.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:03.414+0000 7f450c200700 1 --2- 172.21.15.62:0/1645191168 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4504073090 0x7f4504073460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:15:03.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:03.414+0000 7f450c200700 1 -- 172.21.15.62:0/1645191168 >> 172.21.15.62:0/1645191168 conn(0x7f45040fbda0 msgr2=0x7f45040fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:15:03.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:03.414+0000 7f450c200700 1 -- 172.21.15.62:0/1645191168 shutdown_connections 2024-04-20T22:15:03.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:03.414+0000 7f450c200700 1 -- 172.21.15.62:0/1645191168 wait complete. 2024-04-20T22:15:03.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:03.415+0000 7f450c200700 1 Processor -- start 2024-04-20T22:15:03.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:03.415+0000 7f450c200700 1 -- start start 2024-04-20T22:15:03.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:03.416+0000 7f450c200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4504073090 0x7f45041981b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:15:03.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:03.416+0000 7f450c200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45040739a0 0x7f45041986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:15:03.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:03.416+0000 7f450c200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4504198d40 con 0x7f4504073090 2024-04-20T22:15:03.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:03.416+0000 7f450c200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4504198e80 con 0x7f45040739a0 2024-04-20T22:15:03.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:03.416+0000 7f4509800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4504073090 0x7f45041981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:15:03.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:03.416+0000 7f4509800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4504073090 0x7f45041981b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47224/0 (socket says 172.21.15.62:47224) 2024-04-20T22:15:03.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:03.416+0000 7f4509800700 1 -- 172.21.15.62:0/4274756328 learned_addr learned my addr 172.21.15.62:0/4274756328 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:15:03.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:03.417+0000 7f4508e00700 1 --2- 172.21.15.62:0/4274756328 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45040739a0 0x7f45041986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:15:03.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:03.417+0000 7f4509800700 1 -- 172.21.15.62:0/4274756328 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45040739a0 msgr2=0x7f45041986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:15:03.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:03.417+0000 7f4509800700 1 --2- 172.21.15.62:0/4274756328 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45040739a0 0x7f45041986f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:15:03.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:03.417+0000 7f4509800700 1 -- 172.21.15.62:0/4274756328 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44f00095f0 con 0x7f4504073090 2024-04-20T22:15:03.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:03.417+0000 7f4509800700 1 --2- 172.21.15.62:0/4274756328 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4504073090 0x7f45041981b0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f44f800c930 tx=0x7f44f800ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:15:03.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:03.418+0000 7f4502000700 1 -- 172.21.15.62:0/4274756328 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f44f8007ad0 con 0x7f4504073090 2024-04-20T22:15:03.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:03.418+0000 7f4502000700 1 -- 172.21.15.62:0/4274756328 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f44f8007c30 con 0x7f4504073090 2024-04-20T22:15:03.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:03.418+0000 7f4502000700 1 -- 172.21.15.62:0/4274756328 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f44f8018610 con 0x7f4504073090 2024-04-20T22:15:03.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:03.418+0000 7f450c200700 1 -- 172.21.15.62:0/4274756328 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f450419ccd0 con 0x7f4504073090 2024-04-20T22:15:03.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:03.418+0000 7f450c200700 1 -- 172.21.15.62:0/4274756328 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f450419d1a0 con 0x7f4504073090 2024-04-20T22:15:03.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:03.419+0000 7f450c200700 1 -- 172.21.15.62:0/4274756328 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4504109ce0 con 0x7f4504073090 2024-04-20T22:15:03.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:03.420+0000 7f4502000700 1 -- 172.21.15.62:0/4274756328 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f44f800f450 con 0x7f4504073090 2024-04-20T22:15:03.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:03.421+0000 7f4502000700 1 --2- 172.21.15.62:0/4274756328 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f44ec06ecc0 0x7f44ec071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:15:03.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:03.421+0000 7f4502000700 1 -- 172.21.15.62:0/4274756328 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f44f808eff0 con 0x7f4504073090 2024-04-20T22:15:03.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:03.421+0000 7f4508e00700 1 --2- 172.21.15.62:0/4274756328 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f44ec06ecc0 0x7f44ec071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:15:03.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:03.422+0000 7f4508e00700 1 --2- 172.21.15.62:0/4274756328 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f44ec06ecc0 0x7f44ec071170 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f44f0006010 tx=0x7f44f0005a90 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:15:03.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:03.426+0000 7f4502000700 1 -- 172.21.15.62:0/4274756328 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f44f8057de0 con 0x7f4504073090 2024-04-20T22:15:03.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:03.679+0000 7f450c200700 1 -- 172.21.15.62:0/4274756328 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f4504199650 con 0x7f44ec06ecc0 2024-04-20T22:15:03.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:03.713+0000 7f4502000700 1 -- 172.21.15.62:0/4274756328 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7f4504199650 con 0x7f44ec06ecc0 2024-04-20T22:15:03.713 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T22:15:03.714 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (40m) 103s ago 54m 14.6M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T22:15:03.714 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (55m) 103s ago 55m 11.6M - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T22:15:03.714 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (53m) 106s ago 53m 11.2M - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T22:15:03.714 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (37m) 103s ago 55m 6660k - 19.0.0-3133-ga8c84f59 76bc6b91b726 2825d5ba11c6 2024-04-20T22:15:03.714 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (36m) 106s ago 53m 6665k - 19.0.0-3133-ga8c84f59 76bc6b91b726 20f746fa1bba 2024-04-20T22:15:03.714 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (40m) 103s ago 54m 75.5M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T22:15:03.714 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (32m) 103s ago 47m 22.9M - 19.0.0-3133-ga8c84f59 76bc6b91b726 de80bc90a683 2024-04-20T22:15:03.714 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (32m) 103s ago 47m 23.0M - 19.0.0-3133-ga8c84f59 76bc6b91b726 cd793b7ee98e 2024-04-20T22:15:03.714 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (17m) 106s ago 47m 18.5M - 19.0.0-3133-ga8c84f59 76bc6b91b726 0e7b0bc4d1ab 2024-04-20T22:15:03.714 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (112s) 106s ago 47m 11.4M - 19.0.0-3133-ga8c84f59 76bc6b91b726 0af732ac9e06 2024-04-20T22:15:03.714 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (43m) 103s ago 56m 517M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T22:15:03.714 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (42m) 106s ago 53m 460M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T22:15:03.714 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (38m) 103s ago 56m 86.3M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 019eb1479485 2024-04-20T22:15:03.714 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (38m) 106s ago 53m 58.0M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 55aac560aab6 2024-04-20T22:15:03.715 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (42m) 103s ago 54m 8766k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T22:15:03.715 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (41m) 106s ago 53m 8711k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T22:15:03.715 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (36m) 103s ago 52m 91.3M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 ee2bb901f26e 2024-04-20T22:15:03.715 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (35m) 103s ago 51m 94.9M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 9075152d4162 2024-04-20T22:15:03.715 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (35m) 103s ago 51m 78.5M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 2d4f3e38dc0e 2024-04-20T22:15:03.715 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (34m) 106s ago 50m 94.1M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 8f97b17ae0eb 2024-04-20T22:15:03.715 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (34m) 106s ago 50m 96.1M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 653d8079967d 2024-04-20T22:15:03.715 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (33m) 106s ago 49m 90.9M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 596dded14c12 2024-04-20T22:15:03.715 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (41m) 103s ago 54m 90.4M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T22:15:03.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:03.717+0000 7f450c200700 1 -- 172.21.15.62:0/4274756328 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f44ec06ecc0 msgr2=0x7f44ec071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:15:03.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:03.717+0000 7f450c200700 1 --2- 172.21.15.62:0/4274756328 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f44ec06ecc0 0x7f44ec071170 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f44f0006010 tx=0x7f44f0005a90 comp rx=0 tx=0).stop 2024-04-20T22:15:03.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:03.717+0000 7f450c200700 1 -- 172.21.15.62:0/4274756328 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4504073090 msgr2=0x7f45041981b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:15:03.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:03.717+0000 7f450c200700 1 --2- 172.21.15.62:0/4274756328 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4504073090 0x7f45041981b0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f44f800c930 tx=0x7f44f800ccf0 comp rx=0 tx=0).stop 2024-04-20T22:15:03.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:03.717+0000 7f450c200700 1 -- 172.21.15.62:0/4274756328 shutdown_connections 2024-04-20T22:15:03.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:03.717+0000 7f450c200700 1 --2- 172.21.15.62:0/4274756328 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f44ec06ecc0 0x7f44ec071170 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:15:03.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:03.717+0000 7f450c200700 1 --2- 172.21.15.62:0/4274756328 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4504073090 0x7f45041981b0 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:15:03.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:03.717+0000 7f450c200700 1 --2- 172.21.15.62:0/4274756328 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45040739a0 0x7f45041986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:15:03.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:03.717+0000 7f450c200700 1 -- 172.21.15.62:0/4274756328 >> 172.21.15.62:0/4274756328 conn(0x7f45040fbda0 msgr2=0x7f4504106e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:15:03.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:03.718+0000 7f450c200700 1 -- 172.21.15.62:0/4274756328 shutdown_connections 2024-04-20T22:15:03.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:03.718+0000 7f450c200700 1 -- 172.21.15.62:0/4274756328 wait complete. 2024-04-20T22:15:03.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:03.882+0000 7f9ca3200700 1 -- 172.21.15.62:0/618313280 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9c9c108470 msgr2=0x7f9c9c108840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:15:03.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:03.882+0000 7f9ca3200700 1 --2- 172.21.15.62:0/618313280 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9c9c108470 0x7f9c9c108840 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f9c8c009910 tx=0x7f9c8c009c20 comp rx=0 tx=0).stop 2024-04-20T22:15:03.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:03.882+0000 7f9ca3200700 1 -- 172.21.15.62:0/618313280 shutdown_connections 2024-04-20T22:15:03.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:03.882+0000 7f9ca3200700 1 --2- 172.21.15.62:0/618313280 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9c9c102470 0x7f9c9c1028e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:15:03.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:03.882+0000 7f9ca3200700 1 --2- 172.21.15.62:0/618313280 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9c9c108470 0x7f9c9c108840 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:15:03.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:03.882+0000 7f9ca3200700 1 -- 172.21.15.62:0/618313280 >> 172.21.15.62:0/618313280 conn(0x7f9c9c0fdf90 msgr2=0x7f9c9c1003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:15:03.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:03.882+0000 7f9ca3200700 1 -- 172.21.15.62:0/618313280 shutdown_connections 2024-04-20T22:15:03.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:03.882+0000 7f9ca3200700 1 -- 172.21.15.62:0/618313280 wait complete. 2024-04-20T22:15:03.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:03.883+0000 7f9ca3200700 1 Processor -- start 2024-04-20T22:15:03.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:03.884+0000 7f9ca3200700 1 -- start start 2024-04-20T22:15:03.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:03.884+0000 7f9ca3200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9c9c102470 0x7f9c9c1981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:15:03.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:03.884+0000 7f9ca3200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9c9c108470 0x7f9c9c198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:15:03.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:03.884+0000 7f9ca3200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c9c198d80 con 0x7f9c9c108470 2024-04-20T22:15:03.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:03.884+0000 7f9ca3200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c9c198ec0 con 0x7f9c9c102470 2024-04-20T22:15:03.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:03.884+0000 7f9c9b400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9c9c108470 0x7f9c9c198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:15:03.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:03.885+0000 7f9c9b400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9c9c108470 0x7f9c9c198730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47242/0 (socket says 172.21.15.62:47242) 2024-04-20T22:15:03.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:03.885+0000 7f9c9b400700 1 -- 172.21.15.62:0/1279502060 learned_addr learned my addr 172.21.15.62:0/1279502060 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:15:03.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:03.885+0000 7f9c9be00700 1 --2- 172.21.15.62:0/1279502060 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9c9c102470 0x7f9c9c1981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:15:03.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:03.885+0000 7f9c9b400700 1 -- 172.21.15.62:0/1279502060 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9c9c102470 msgr2=0x7f9c9c1981f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:15:03.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:03.885+0000 7f9c9b400700 1 --2- 172.21.15.62:0/1279502060 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9c9c102470 0x7f9c9c1981f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:15:03.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:03.885+0000 7f9c9b400700 1 -- 172.21.15.62:0/1279502060 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c8c0095f0 con 0x7f9c9c108470 2024-04-20T22:15:03.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:03.885+0000 7f9c9b400700 1 --2- 172.21.15.62:0/1279502060 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9c9c108470 0x7f9c9c198730 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f9c8800c930 tx=0x7f9c8800ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:15:03.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:03.885+0000 7f9c98c00700 1 -- 172.21.15.62:0/1279502060 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9c88007ad0 con 0x7f9c9c108470 2024-04-20T22:15:03.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:03.886+0000 7f9c98c00700 1 -- 172.21.15.62:0/1279502060 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9c88007c30 con 0x7f9c9c108470 2024-04-20T22:15:03.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:03.886+0000 7f9ca3200700 1 -- 172.21.15.62:0/1279502060 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c9c19cd10 con 0x7f9c9c108470 2024-04-20T22:15:03.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:03.886+0000 7f9ca3200700 1 -- 172.21.15.62:0/1279502060 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c9c19d260 con 0x7f9c9c108470 2024-04-20T22:15:03.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:03.886+0000 7f9c98c00700 1 -- 172.21.15.62:0/1279502060 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9c88018610 con 0x7f9c9c108470 2024-04-20T22:15:03.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:03.887+0000 7f9ca3200700 1 -- 172.21.15.62:0/1279502060 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c9c10a9e0 con 0x7f9c9c108470 2024-04-20T22:15:03.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:03.888+0000 7f9c98c00700 1 -- 172.21.15.62:0/1279502060 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f9c8800f450 con 0x7f9c9c108470 2024-04-20T22:15:03.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:03.889+0000 7f9c98c00700 1 --2- 172.21.15.62:0/1279502060 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9c8006ecc0 0x7f9c80071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:15:03.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:03.889+0000 7f9c98c00700 1 -- 172.21.15.62:0/1279502060 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f9c8808f0d0 con 0x7f9c9c108470 2024-04-20T22:15:03.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:03.890+0000 7f9c9be00700 1 --2- 172.21.15.62:0/1279502060 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9c8006ecc0 0x7f9c80071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:15:03.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:03.890+0000 7f9c9be00700 1 --2- 172.21.15.62:0/1279502060 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9c8006ecc0 0x7f9c80071170 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f9c8c000c00 tx=0x7f9c8c005fd0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:15:03.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:03.894+0000 7f9c98c00700 1 -- 172.21.15.62:0/1279502060 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f9c88057de0 con 0x7f9c9c108470 2024-04-20T22:15:04.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:04.222+0000 7f9ca3200700 1 -- 172.21.15.62:0/1279502060 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f9c9c04f2a0 con 0x7f9c9c108470 2024-04-20T22:15:04.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:04.223+0000 7f9c98c00700 1 -- 172.21.15.62:0/1279502060 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f9c8805b4a0 con 0x7f9c9c108470 2024-04-20T22:15:04.223 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:15:04.223 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T22:15:04.223 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:15:04.223 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:15:04.223 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T22:15:04.223 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:15:04.223 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:15:04.223 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T22:15:04.224 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 6 2024-04-20T22:15:04.224 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:15:04.224 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T22:15:04.224 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 4 2024-04-20T22:15:04.224 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:15:04.224 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T22:15:04.224 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 14 2024-04-20T22:15:04.224 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T22:15:04.224 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:15:04.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:04.226+0000 7f9ca3200700 1 -- 172.21.15.62:0/1279502060 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9c8006ecc0 msgr2=0x7f9c80071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:15:04.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:04.226+0000 7f9ca3200700 1 --2- 172.21.15.62:0/1279502060 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9c8006ecc0 0x7f9c80071170 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f9c8c000c00 tx=0x7f9c8c005fd0 comp rx=0 tx=0).stop 2024-04-20T22:15:04.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:04.226+0000 7f9ca3200700 1 -- 172.21.15.62:0/1279502060 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9c9c108470 msgr2=0x7f9c9c198730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:15:04.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:04.226+0000 7f9ca3200700 1 --2- 172.21.15.62:0/1279502060 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9c9c108470 0x7f9c9c198730 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f9c8800c930 tx=0x7f9c8800ccf0 comp rx=0 tx=0).stop 2024-04-20T22:15:04.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:04.226+0000 7f9ca3200700 1 -- 172.21.15.62:0/1279502060 shutdown_connections 2024-04-20T22:15:04.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:04.226+0000 7f9ca3200700 1 --2- 172.21.15.62:0/1279502060 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9c8006ecc0 0x7f9c80071170 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:15:04.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:04.226+0000 7f9ca3200700 1 --2- 172.21.15.62:0/1279502060 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9c9c102470 0x7f9c9c1981f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:15:04.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:04.226+0000 7f9ca3200700 1 --2- 172.21.15.62:0/1279502060 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9c9c108470 0x7f9c9c198730 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:15:04.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:04.226+0000 7f9ca3200700 1 -- 172.21.15.62:0/1279502060 >> 172.21.15.62:0/1279502060 conn(0x7f9c9c0fdf90 msgr2=0x7f9c9c1002c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:15:04.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:04.226+0000 7f9ca3200700 1 -- 172.21.15.62:0/1279502060 shutdown_connections 2024-04-20T22:15:04.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:04.227+0000 7f9ca3200700 1 -- 172.21.15.62:0/1279502060 wait complete. 2024-04-20T22:15:04.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:04.375+0000 7fc537200700 1 -- 172.21.15.62:0/2216970704 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc538106b80 msgr2=0x7fc538106ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:15:04.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:04.375+0000 7fc537200700 1 --2- 172.21.15.62:0/2216970704 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc538106b80 0x7fc538106ff0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7fc524009910 tx=0x7fc524009c20 comp rx=0 tx=0).stop 2024-04-20T22:15:04.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:04.376+0000 7fc537200700 1 -- 172.21.15.62:0/2216970704 shutdown_connections 2024-04-20T22:15:04.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:04.376+0000 7fc537200700 1 --2- 172.21.15.62:0/2216970704 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc538106b80 0x7fc538106ff0 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:15:04.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:04.376+0000 7fc537200700 1 --2- 172.21.15.62:0/2216970704 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc53810c660 0x7fc53810ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:15:04.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:04.376+0000 7fc537200700 1 -- 172.21.15.62:0/2216970704 >> 172.21.15.62:0/2216970704 conn(0x7fc538075e90 msgr2=0x7fc5380782a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:15:04.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:04.376+0000 7fc537200700 1 -- 172.21.15.62:0/2216970704 shutdown_connections 2024-04-20T22:15:04.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:04.376+0000 7fc537200700 1 -- 172.21.15.62:0/2216970704 wait complete. 2024-04-20T22:15:04.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:04.377+0000 7fc537200700 1 Processor -- start 2024-04-20T22:15:04.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:04.377+0000 7fc537200700 1 -- start start 2024-04-20T22:15:04.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:04.378+0000 7fc537200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc538106b80 0x7fc53819c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:15:04.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:04.378+0000 7fc537200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc53810c660 0x7fc53819c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:15:04.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:04.378+0000 7fc537200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc53819cf60 con 0x7fc538106b80 2024-04-20T22:15:04.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:04.378+0000 7fc537200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5381a0cf0 con 0x7fc53810c660 2024-04-20T22:15:04.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:04.378+0000 7fc535e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc538106b80 0x7fc53819c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:15:04.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:04.378+0000 7fc535e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc538106b80 0x7fc53819c340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47260/0 (socket says 172.21.15.62:47260) 2024-04-20T22:15:04.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:04.378+0000 7fc535e00700 1 -- 172.21.15.62:0/3921420416 learned_addr learned my addr 172.21.15.62:0/3921420416 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:15:04.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:04.378+0000 7fc535e00700 1 -- 172.21.15.62:0/3921420416 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc53810c660 msgr2=0x7fc53819c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:15:04.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:04.378+0000 7fc535400700 1 --2- 172.21.15.62:0/3921420416 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc53810c660 0x7fc53819c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:15:04.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:04.379+0000 7fc535e00700 1 --2- 172.21.15.62:0/3921420416 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc53810c660 0x7fc53819c880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:15:04.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:04.379+0000 7fc535e00700 1 -- 172.21.15.62:0/3921420416 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5240095f0 con 0x7fc538106b80 2024-04-20T22:15:04.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:04.379+0000 7fc535e00700 1 --2- 172.21.15.62:0/3921420416 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc538106b80 0x7fc53819c340 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7fc52800b4b0 tx=0x7fc52800b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:15:04.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:04.379+0000 7fc52ea00700 1 -- 172.21.15.62:0/3921420416 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc528011800 con 0x7fc538106b80 2024-04-20T22:15:04.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:04.379+0000 7fc537200700 1 -- 172.21.15.62:0/3921420416 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5381a0fd0 con 0x7fc538106b80 2024-04-20T22:15:04.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:04.379+0000 7fc52ea00700 1 -- 172.21.15.62:0/3921420416 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc528011e20 con 0x7fc538106b80 2024-04-20T22:15:04.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:04.379+0000 7fc52ea00700 1 -- 172.21.15.62:0/3921420416 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc52800f2e0 con 0x7fc538106b80 2024-04-20T22:15:04.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:04.379+0000 7fc537200700 1 -- 172.21.15.62:0/3921420416 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5381a1520 con 0x7fc538106b80 2024-04-20T22:15:04.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:04.380+0000 7fc537200700 1 -- 172.21.15.62:0/3921420416 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc53810eb40 con 0x7fc538106b80 2024-04-20T22:15:04.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:04.381+0000 7fc52ea00700 1 -- 172.21.15.62:0/3921420416 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fc528010380 con 0x7fc538106b80 2024-04-20T22:15:04.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:04.382+0000 7fc52ea00700 1 --2- 172.21.15.62:0/3921420416 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc51c06ecc0 0x7fc51c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:15:04.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:04.383+0000 7fc52ea00700 1 -- 172.21.15.62:0/3921420416 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fc52808e9c0 con 0x7fc538106b80 2024-04-20T22:15:04.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:04.383+0000 7fc535400700 1 --2- 172.21.15.62:0/3921420416 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc51c06ecc0 0x7fc51c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:15:04.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:04.383+0000 7fc535400700 1 --2- 172.21.15.62:0/3921420416 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc51c06ecc0 0x7fc51c071170 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fc52400b5c0 tx=0x7fc524005960 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:15:04.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:04.387+0000 7fc52ea00700 1 -- 172.21.15.62:0/3921420416 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fc528057750 con 0x7fc538106b80 2024-04-20T22:15:04.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:15:04 smithi062 ceph-mon[76322]: from='client.36106 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:15:04.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:15:04 smithi062 ceph-mon[76322]: pgmap v1196: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:15:04.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:15:04 smithi062 ceph-mon[76322]: from='client.36110 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:15:04.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:15:04 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/1279502060' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:15:04.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:04.668+0000 7fc537200700 1 -- 172.21.15.62:0/3921420416 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fc5381a17f0 con 0x7fc538106b80 2024-04-20T22:15:04.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:04.669+0000 7fc52ea00700 1 -- 172.21.15.62:0/3921420416 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 29 v29) v1 ==== 76+0+1887 (secure 0 0 0) 0x7fc52805ae10 con 0x7fc538106b80 2024-04-20T22:15:04.670 INFO:teuthology.orchestra.run.smithi062.stdout:e29 2024-04-20T22:15:04.670 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T22:15:04.670 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:15:04.670 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T22:15:04.670 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:15:04.670 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T22:15:04.670 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T22:15:04.671 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 20 2024-04-20T22:15:04.671 INFO:teuthology.orchestra.run.smithi062.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-20T22:15:04.671 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T22:15:04.671 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T21:42:01.559061+0000 2024-04-20T22:15:04.671 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T22:15:04.671 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T22:15:04.671 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T22:15:04.671 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T22:15:04.671 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T22:15:04.671 INFO:teuthology.orchestra.run.smithi062.stdout:max_xattr_size 65536 2024-04-20T22:15:04.671 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T22:15:04.671 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T22:15:04.671 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 81 2024-04-20T22:15:04.671 INFO:teuthology.orchestra.run.smithi062.stdout:snap_rstat 1 2024-04-20T22:15:04.672 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:15:04.672 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 2 2024-04-20T22:15:04.672 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1 2024-04-20T22:15:04.672 INFO:teuthology.orchestra.run.smithi062.stdout:up {} 2024-04-20T22:15:04.672 INFO:teuthology.orchestra.run.smithi062.stdout:failed 0,1 2024-04-20T22:15:04.672 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T22:15:04.672 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T22:15:04.672 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T22:15:04.672 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T22:15:04.672 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data enabled 2024-04-20T22:15:04.672 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T22:15:04.672 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T22:15:04.672 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 1 2024-04-20T22:15:04.672 INFO:teuthology.orchestra.run.smithi062.stdout:qdb_cluster leader: 0 members: 2024-04-20T22:15:04.672 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:15:04.672 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:15:04.673 INFO:teuthology.orchestra.run.smithi062.stdout:Standby daemons: 2024-04-20T22:15:04.673 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:15:04.673 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ryqblb{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6826/4113916205,v1:172.21.15.62:6827/4113916205] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:15:04.673 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ozizwi{-1:34476} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6828/3801975165,v1:172.21.15.62:6829/3801975165] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:15:04.673 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.dvimer{-1:44727} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6824/3576699232,v1:172.21.15.189:6825/3576699232] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:15:04.673 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.xbgspw{-1:45075} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6826/212905792,v1:172.21.15.189:6827/212905792] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:15:04.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:04.673+0000 7fc537200700 1 -- 172.21.15.62:0/3921420416 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc51c06ecc0 msgr2=0x7fc51c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:15:04.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:04.673+0000 7fc537200700 1 --2- 172.21.15.62:0/3921420416 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc51c06ecc0 0x7fc51c071170 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fc52400b5c0 tx=0x7fc524005960 comp rx=0 tx=0).stop 2024-04-20T22:15:04.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:04.673+0000 7fc537200700 1 -- 172.21.15.62:0/3921420416 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc538106b80 msgr2=0x7fc53819c340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:15:04.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:04.673+0000 7fc537200700 1 --2- 172.21.15.62:0/3921420416 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc538106b80 0x7fc53819c340 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7fc52800b4b0 tx=0x7fc52800b870 comp rx=0 tx=0).stop 2024-04-20T22:15:04.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:04.673+0000 7fc537200700 1 -- 172.21.15.62:0/3921420416 shutdown_connections 2024-04-20T22:15:04.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:04.673+0000 7fc537200700 1 --2- 172.21.15.62:0/3921420416 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc51c06ecc0 0x7fc51c071170 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:15:04.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:04.673+0000 7fc537200700 1 --2- 172.21.15.62:0/3921420416 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc538106b80 0x7fc53819c340 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:15:04.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:04.673+0000 7fc537200700 1 --2- 172.21.15.62:0/3921420416 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc53810c660 0x7fc53819c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:15:04.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:04.673+0000 7fc537200700 1 -- 172.21.15.62:0/3921420416 >> 172.21.15.62:0/3921420416 conn(0x7fc538075e90 msgr2=0x7fc538077b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:15:04.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:04.673+0000 7fc537200700 1 -- 172.21.15.62:0/3921420416 shutdown_connections 2024-04-20T22:15:04.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:04.674+0000 7fc537200700 1 -- 172.21.15.62:0/3921420416 wait complete. 2024-04-20T22:15:04.675 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 29 2024-04-20T22:15:04.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:15:04 smithi189 ceph-mon[57022]: from='client.36106 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:15:04.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:15:04 smithi189 ceph-mon[57022]: pgmap v1196: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:15:04.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:15:04 smithi189 ceph-mon[57022]: from='client.36110 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:15:04.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:15:04 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/1279502060' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:15:04.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:04.821+0000 7fca6b200700 1 -- 172.21.15.62:0/1525032144 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fca64102e70 msgr2=0x7fca64106e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:15:04.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:04.821+0000 7fca6b200700 1 --2- 172.21.15.62:0/1525032144 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fca64102e70 0x7fca64106e40 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7fca54009910 tx=0x7fca54009c20 comp rx=0 tx=0).stop 2024-04-20T22:15:04.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:04.821+0000 7fca6b200700 1 -- 172.21.15.62:0/1525032144 shutdown_connections 2024-04-20T22:15:04.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:04.821+0000 7fca6b200700 1 --2- 172.21.15.62:0/1525032144 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fca64102e70 0x7fca64106e40 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:15:04.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:04.821+0000 7fca6b200700 1 --2- 172.21.15.62:0/1525032144 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca641024d0 0x7fca641028a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:15:04.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:04.821+0000 7fca6b200700 1 -- 172.21.15.62:0/1525032144 >> 172.21.15.62:0/1525032144 conn(0x7fca640fdd90 msgr2=0x7fca641001a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:15:04.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:04.822+0000 7fca6b200700 1 -- 172.21.15.62:0/1525032144 shutdown_connections 2024-04-20T22:15:04.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:04.822+0000 7fca6b200700 1 -- 172.21.15.62:0/1525032144 wait complete. 2024-04-20T22:15:04.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:04.823+0000 7fca6b200700 1 Processor -- start 2024-04-20T22:15:04.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:04.823+0000 7fca6b200700 1 -- start start 2024-04-20T22:15:04.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:04.823+0000 7fca6b200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fca641024d0 0x7fca64197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:15:04.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:04.823+0000 7fca6b200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca64102e70 0x7fca641984c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:15:04.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:04.823+0000 7fca6b200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca64198ba0 con 0x7fca641024d0 2024-04-20T22:15:04.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:04.823+0000 7fca6b200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca6419c930 con 0x7fca64102e70 2024-04-20T22:15:04.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:04.823+0000 7fca69e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fca641024d0 0x7fca64197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:15:04.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:04.824+0000 7fca69e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fca641024d0 0x7fca64197f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47276/0 (socket says 172.21.15.62:47276) 2024-04-20T22:15:04.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:04.824+0000 7fca69e00700 1 -- 172.21.15.62:0/506086859 learned_addr learned my addr 172.21.15.62:0/506086859 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:15:04.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:04.824+0000 7fca69e00700 1 -- 172.21.15.62:0/506086859 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca64102e70 msgr2=0x7fca641984c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:15:04.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:04.824+0000 7fca69e00700 1 --2- 172.21.15.62:0/506086859 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca64102e70 0x7fca641984c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:15:04.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:04.824+0000 7fca69e00700 1 -- 172.21.15.62:0/506086859 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca540095f0 con 0x7fca641024d0 2024-04-20T22:15:04.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:04.824+0000 7fca69e00700 1 --2- 172.21.15.62:0/506086859 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fca641024d0 0x7fca64197f80 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7fca5800b4b0 tx=0x7fca5800b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:15:04.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:04.824+0000 7fca5ea00700 1 -- 172.21.15.62:0/506086859 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fca58011800 con 0x7fca641024d0 2024-04-20T22:15:04.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:04.824+0000 7fca6b200700 1 -- 172.21.15.62:0/506086859 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca6419cc10 con 0x7fca641024d0 2024-04-20T22:15:04.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:04.825+0000 7fca5ea00700 1 -- 172.21.15.62:0/506086859 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fca58011e20 con 0x7fca641024d0 2024-04-20T22:15:04.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:04.825+0000 7fca5ea00700 1 -- 172.21.15.62:0/506086859 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fca5800f2e0 con 0x7fca641024d0 2024-04-20T22:15:04.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:04.825+0000 7fca6b200700 1 -- 172.21.15.62:0/506086859 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca6419d160 con 0x7fca641024d0 2024-04-20T22:15:04.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:04.826+0000 7fca6b200700 1 -- 172.21.15.62:0/506086859 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca6410a770 con 0x7fca641024d0 2024-04-20T22:15:04.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:04.827+0000 7fca5ea00700 1 -- 172.21.15.62:0/506086859 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fca58010380 con 0x7fca641024d0 2024-04-20T22:15:04.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:04.828+0000 7fca5ea00700 1 --2- 172.21.15.62:0/506086859 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fca4c06ecc0 0x7fca4c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:15:04.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:04.828+0000 7fca5ea00700 1 -- 172.21.15.62:0/506086859 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fca5808eaa0 con 0x7fca641024d0 2024-04-20T22:15:04.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:04.828+0000 7fca69400700 1 --2- 172.21.15.62:0/506086859 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fca4c06ecc0 0x7fca4c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:15:04.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:04.829+0000 7fca69400700 1 --2- 172.21.15.62:0/506086859 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fca4c06ecc0 0x7fca4c071170 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fca5400b5c0 tx=0x7fca54005fd0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:15:04.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:04.833+0000 7fca5ea00700 1 -- 172.21.15.62:0/506086859 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fca580577b0 con 0x7fca641024d0 2024-04-20T22:15:05.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:05.100+0000 7fca6b200700 1 -- 172.21.15.62:0/506086859 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fca6419d480 con 0x7fca4c06ecc0 2024-04-20T22:15:05.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:05.103+0000 7fca5ea00700 1 -- 172.21.15.62:0/506086859 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fca6419d480 con 0x7fca4c06ecc0 2024-04-20T22:15:05.103 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:15:05.103 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T22:15:05.103 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T22:15:05.103 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-20T22:15:05.103 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T22:15:05.103 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr", 2024-04-20T22:15:05.103 INFO:teuthology.orchestra.run.smithi062.stdout: "osd", 2024-04-20T22:15:05.103 INFO:teuthology.orchestra.run.smithi062.stdout: "mds", 2024-04-20T22:15:05.103 INFO:teuthology.orchestra.run.smithi062.stdout: "crash", 2024-04-20T22:15:05.103 INFO:teuthology.orchestra.run.smithi062.stdout: "mon" 2024-04-20T22:15:05.103 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T22:15:05.104 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "16/23 daemons upgraded", 2024-04-20T22:15:05.104 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading mds daemons", 2024-04-20T22:15:05.104 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T22:15:05.104 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:15:05.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:05.105+0000 7fca6b200700 1 -- 172.21.15.62:0/506086859 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fca4c06ecc0 msgr2=0x7fca4c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:15:05.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:05.105+0000 7fca6b200700 1 --2- 172.21.15.62:0/506086859 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fca4c06ecc0 0x7fca4c071170 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fca5400b5c0 tx=0x7fca54005fd0 comp rx=0 tx=0).stop 2024-04-20T22:15:05.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:05.106+0000 7fca6b200700 1 -- 172.21.15.62:0/506086859 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fca641024d0 msgr2=0x7fca64197f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:15:05.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:05.106+0000 7fca6b200700 1 --2- 172.21.15.62:0/506086859 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fca641024d0 0x7fca64197f80 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7fca5800b4b0 tx=0x7fca5800b870 comp rx=0 tx=0).stop 2024-04-20T22:15:05.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:05.106+0000 7fca6b200700 1 -- 172.21.15.62:0/506086859 shutdown_connections 2024-04-20T22:15:05.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:05.106+0000 7fca6b200700 1 --2- 172.21.15.62:0/506086859 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fca4c06ecc0 0x7fca4c071170 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:15:05.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:05.106+0000 7fca6b200700 1 --2- 172.21.15.62:0/506086859 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fca641024d0 0x7fca64197f80 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:15:05.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:05.106+0000 7fca6b200700 1 --2- 172.21.15.62:0/506086859 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca64102e70 0x7fca641984c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:15:05.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:05.106+0000 7fca6b200700 1 -- 172.21.15.62:0/506086859 >> 172.21.15.62:0/506086859 conn(0x7fca640fdd90 msgr2=0x7fca640ff4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:15:05.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:05.106+0000 7fca6b200700 1 -- 172.21.15.62:0/506086859 shutdown_connections 2024-04-20T22:15:05.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:05.106+0000 7fca6b200700 1 -- 172.21.15.62:0/506086859 wait complete. 2024-04-20T22:15:05.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:05.268+0000 7f8d75a00700 1 -- 172.21.15.62:0/788806374 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d70100e80 msgr2=0x7f8d70104e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:15:05.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:05.268+0000 7f8d75a00700 1 --2- 172.21.15.62:0/788806374 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d70100e80 0x7f8d70104e50 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f8d58009910 tx=0x7f8d58009c20 comp rx=0 tx=0).stop 2024-04-20T22:15:05.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:05.268+0000 7f8d75a00700 1 -- 172.21.15.62:0/788806374 shutdown_connections 2024-04-20T22:15:05.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:05.268+0000 7f8d75a00700 1 --2- 172.21.15.62:0/788806374 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d70100e80 0x7f8d70104e50 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:15:05.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:05.268+0000 7f8d75a00700 1 --2- 172.21.15.62:0/788806374 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d701004e0 0x7f8d701008b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:15:05.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:05.268+0000 7f8d75a00700 1 -- 172.21.15.62:0/788806374 >> 172.21.15.62:0/788806374 conn(0x7f8d700fbda0 msgr2=0x7f8d700fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:15:05.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:05.268+0000 7f8d75a00700 1 -- 172.21.15.62:0/788806374 shutdown_connections 2024-04-20T22:15:05.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:05.268+0000 7f8d75a00700 1 -- 172.21.15.62:0/788806374 wait complete. 2024-04-20T22:15:05.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:05.269+0000 7f8d75a00700 1 Processor -- start 2024-04-20T22:15:05.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:05.270+0000 7f8d75a00700 1 -- start start 2024-04-20T22:15:05.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:05.270+0000 7f8d75a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d701004e0 0x7f8d701981b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:15:05.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:05.270+0000 7f8d75a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d70100e80 0x7f8d701986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:15:05.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:05.270+0000 7f8d75a00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d70198dd0 con 0x7f8d70100e80 2024-04-20T22:15:05.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:05.270+0000 7f8d75a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d7019cb60 con 0x7f8d701004e0 2024-04-20T22:15:05.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:05.270+0000 7f8d66000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d70100e80 0x7f8d701986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:15:05.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:05.270+0000 7f8d66000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d70100e80 0x7f8d701986f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47294/0 (socket says 172.21.15.62:47294) 2024-04-20T22:15:05.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:05.270+0000 7f8d66000700 1 -- 172.21.15.62:0/3955952531 learned_addr learned my addr 172.21.15.62:0/3955952531 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:15:05.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:05.271+0000 7f8d66000700 1 -- 172.21.15.62:0/3955952531 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d701004e0 msgr2=0x7f8d701981b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:15:05.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:05.271+0000 7f8d66000700 1 --2- 172.21.15.62:0/3955952531 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d701004e0 0x7f8d701981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:15:05.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:05.271+0000 7f8d66000700 1 -- 172.21.15.62:0/3955952531 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d580095f0 con 0x7f8d70100e80 2024-04-20T22:15:05.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:05.271+0000 7f8d66000700 1 --2- 172.21.15.62:0/3955952531 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d70100e80 0x7f8d701986f0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f8d5800bda0 tx=0x7f8d5800be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:15:05.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:05.271+0000 7f8d67e00700 1 -- 172.21.15.62:0/3955952531 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8d5801d070 con 0x7f8d70100e80 2024-04-20T22:15:05.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:05.271+0000 7f8d67e00700 1 -- 172.21.15.62:0/3955952531 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8d58004fb0 con 0x7f8d70100e80 2024-04-20T22:15:05.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:05.272+0000 7f8d67e00700 1 -- 172.21.15.62:0/3955952531 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8d5800f460 con 0x7f8d70100e80 2024-04-20T22:15:05.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:05.272+0000 7f8d75a00700 1 -- 172.21.15.62:0/3955952531 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d7019cde0 con 0x7f8d70100e80 2024-04-20T22:15:05.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:05.272+0000 7f8d75a00700 1 -- 172.21.15.62:0/3955952531 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d7019d2d0 con 0x7f8d70100e80 2024-04-20T22:15:05.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:05.273+0000 7f8d75a00700 1 -- 172.21.15.62:0/3955952531 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d701045d0 con 0x7f8d70100e80 2024-04-20T22:15:05.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:05.274+0000 7f8d67e00700 1 -- 172.21.15.62:0/3955952531 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f8d58022800 con 0x7f8d70100e80 2024-04-20T22:15:05.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:05.275+0000 7f8d67e00700 1 --2- 172.21.15.62:0/3955952531 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f8d5406ecc0 0x7f8d54071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:15:05.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:05.275+0000 7f8d67e00700 1 -- 172.21.15.62:0/3955952531 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f8d58090440 con 0x7f8d70100e80 2024-04-20T22:15:05.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:05.275+0000 7f8d6ea00700 1 --2- 172.21.15.62:0/3955952531 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f8d5406ecc0 0x7f8d54071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:15:05.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:05.276+0000 7f8d6ea00700 1 --2- 172.21.15.62:0/3955952531 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f8d5406ecc0 0x7f8d54071170 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f8d600097b0 tx=0x7f8d60006d20 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:15:05.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:05.280+0000 7f8d67e00700 1 -- 172.21.15.62:0/3955952531 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f8d58059320 con 0x7f8d70100e80 2024-04-20T22:15:05.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:15:05 smithi062 ceph-mon[76322]: from='client.36114 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:15:05.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:15:05 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:15:05.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:15:05 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/3921420416' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:15:05.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:05.609+0000 7f8d75a00700 1 -- 172.21.15.62:0/3955952531 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f8d7004f2a0 con 0x7f8d70100e80 2024-04-20T22:15:05.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:05.610+0000 7f8d67e00700 1 -- 172.21.15.62:0/3955952531 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7f8d5805c9e0 con 0x7f8d70100e80 2024-04-20T22:15:05.610 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-20T22:15:05.610 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-20T22:15:05.610 INFO:teuthology.orchestra.run.smithi062.stdout: host smithi189 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi189/config 2024-04-20T22:15:05.610 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-20T22:15:05.610 INFO:teuthology.orchestra.run.smithi062.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi189 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp11670v2q:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:15:05.611 INFO:teuthology.orchestra.run.smithi062.stdout:Traceback (most recent call last): 2024-04-20T22:15:05.611 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-20T22:15:05.611 INFO:teuthology.orchestra.run.smithi062.stdout: return _run_code(code, main_globals, None, 2024-04-20T22:15:05.611 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-20T22:15:05.611 INFO:teuthology.orchestra.run.smithi062.stdout: exec(code, run_globals) 2024-04-20T22:15:05.611 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5618, in 2024-04-20T22:15:05.611 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5606, in main 2024-04-20T22:15:05.611 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 408, in _infer_config 2024-04-20T22:15:05.611 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 323, in _infer_fsid 2024-04-20T22:15:05.611 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 436, in _infer_image 2024-04-20T22:15:05.611 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 310, in _validate_fsid 2024-04-20T22:15:05.611 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 3359, in command_ceph_volume 2024-04-20T22:15:05.611 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-20T22:15:05.611 INFO:teuthology.orchestra.run.smithi062.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi189 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp11670v2q:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:15:05.611 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-20T22:15:05.612 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is degraded 2024-04-20T22:15:05.612 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T22:15:05.612 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T22:15:05.612 INFO:teuthology.orchestra.run.smithi062.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-20T22:15:05.612 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-20T22:15:05.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:05.612+0000 7f8d75a00700 1 -- 172.21.15.62:0/3955952531 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f8d5406ecc0 msgr2=0x7f8d54071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:15:05.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:05.612+0000 7f8d75a00700 1 --2- 172.21.15.62:0/3955952531 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f8d5406ecc0 0x7f8d54071170 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f8d600097b0 tx=0x7f8d60006d20 comp rx=0 tx=0).stop 2024-04-20T22:15:05.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:05.612+0000 7f8d75a00700 1 -- 172.21.15.62:0/3955952531 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d70100e80 msgr2=0x7f8d701986f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:15:05.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:05.612+0000 7f8d75a00700 1 --2- 172.21.15.62:0/3955952531 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d70100e80 0x7f8d701986f0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f8d5800bda0 tx=0x7f8d5800be80 comp rx=0 tx=0).stop 2024-04-20T22:15:05.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:05.612+0000 7f8d75a00700 1 -- 172.21.15.62:0/3955952531 shutdown_connections 2024-04-20T22:15:05.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:05.612+0000 7f8d75a00700 1 --2- 172.21.15.62:0/3955952531 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f8d5406ecc0 0x7f8d54071170 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:15:05.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:05.612+0000 7f8d75a00700 1 --2- 172.21.15.62:0/3955952531 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d701004e0 0x7f8d701981b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:15:05.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:05.612+0000 7f8d75a00700 1 --2- 172.21.15.62:0/3955952531 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d70100e80 0x7f8d701986f0 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:15:05.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:05.612+0000 7f8d75a00700 1 -- 172.21.15.62:0/3955952531 >> 172.21.15.62:0/3955952531 conn(0x7f8d700fbda0 msgr2=0x7f8d700fd520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:15:05.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:05.613+0000 7f8d75a00700 1 -- 172.21.15.62:0/3955952531 shutdown_connections 2024-04-20T22:15:05.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:05.613+0000 7f8d75a00700 1 -- 172.21.15.62:0/3955952531 wait complete. 2024-04-20T22:15:05.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:15:05 smithi189 ceph-mon[57022]: from='client.36114 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:15:05.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:15:05 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:15:05.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:15:05 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/3921420416' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:15:06.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:15:06 smithi189 ceph-mon[57022]: from='client.36126 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:15:06.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:15:06 smithi189 ceph-mon[57022]: pgmap v1197: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:15:06.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:15:06 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/3955952531' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:15:06.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:15:06 smithi062 ceph-mon[76322]: from='client.36126 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:15:06.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:15:06 smithi062 ceph-mon[76322]: pgmap v1197: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:15:06.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:15:06 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/3955952531' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:15:08.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:15:08 smithi062 ceph-mon[76322]: pgmap v1198: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:15:08.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:15:08 smithi189 ceph-mon[57022]: pgmap v1198: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:15:10.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:15:10 smithi189 ceph-mon[57022]: pgmap v1199: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:15:10.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:15:10 smithi062 ceph-mon[76322]: pgmap v1199: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:15:12.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:15:12 smithi189 ceph-mon[57022]: pgmap v1200: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:15:12.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:15:12 smithi062 ceph-mon[76322]: pgmap v1200: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:15:14.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:15:14 smithi189 ceph-mon[57022]: pgmap v1201: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:15:14.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:15:14 smithi062 ceph-mon[76322]: pgmap v1201: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:15:16.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:15:16 smithi189 ceph-mon[57022]: pgmap v1202: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:15:16.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:15:16 smithi062 ceph-mon[76322]: pgmap v1202: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:15:18.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:15:18 smithi062 ceph-mon[76322]: pgmap v1203: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:15:18.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:15:18 smithi189 ceph-mon[57022]: pgmap v1203: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:15:20.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:15:20 smithi189 ceph-mon[57022]: pgmap v1204: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:15:20.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:15:20 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:15:20.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:15:20 smithi062 ceph-mon[76322]: pgmap v1204: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:15:20.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:15:20 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:15:21.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:15:21 smithi189 ceph-mon[57022]: pgmap v1205: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:15:21.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:15:21 smithi062 ceph-mon[76322]: pgmap v1205: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:15:24.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:15:24 smithi189 ceph-mon[57022]: pgmap v1206: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:15:24.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:15:24 smithi062 ceph-mon[76322]: pgmap v1206: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:15:26.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:15:26 smithi189 ceph-mon[57022]: pgmap v1207: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:15:26.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:15:26 smithi062 ceph-mon[76322]: pgmap v1207: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:15:28.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:15:28 smithi062 ceph-mon[76322]: pgmap v1208: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:15:28.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:15:28 smithi189 ceph-mon[57022]: pgmap v1208: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:15:30.661 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:15:30 smithi062 ceph-mon[76322]: pgmap v1209: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:15:30.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:15:30 smithi189 ceph-mon[57022]: pgmap v1209: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:15:32.670 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:15:32 smithi189 ceph-mon[57022]: pgmap v1210: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:15:32.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:15:32 smithi062 ceph-mon[76322]: pgmap v1210: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:15:34.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:15:34 smithi189 ceph-mon[57022]: pgmap v1211: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:15:34.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:15:34 smithi062 ceph-mon[76322]: pgmap v1211: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:15:35.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:15:35 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:15:35.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:15:35 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:15:35.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:35.790+0000 7f3ae3e00700 1 -- 172.21.15.62:0/2319102589 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3ae4068540 msgr2=0x7f3ae40689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:15:35.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:35.790+0000 7f3ae3e00700 1 --2- 172.21.15.62:0/2319102589 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3ae4068540 0x7f3ae40689b0 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f3acc009910 tx=0x7f3acc009c20 comp rx=0 tx=0).stop 2024-04-20T22:15:35.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:35.790+0000 7f3ae3e00700 1 -- 172.21.15.62:0/2319102589 shutdown_connections 2024-04-20T22:15:35.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:35.790+0000 7f3ae3e00700 1 --2- 172.21.15.62:0/2319102589 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3ae4068540 0x7f3ae40689b0 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:15:35.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:35.790+0000 7f3ae3e00700 1 --2- 172.21.15.62:0/2319102589 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ae41061b0 0x7f3ae4106580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:15:35.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:35.790+0000 7f3ae3e00700 1 -- 172.21.15.62:0/2319102589 >> 172.21.15.62:0/2319102589 conn(0x7f3ae4074fa0 msgr2=0x7f3ae40753a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:15:35.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:35.790+0000 7f3ae3e00700 1 -- 172.21.15.62:0/2319102589 shutdown_connections 2024-04-20T22:15:35.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:35.790+0000 7f3ae3e00700 1 -- 172.21.15.62:0/2319102589 wait complete. 2024-04-20T22:15:35.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:35.791+0000 7f3ae3e00700 1 Processor -- start 2024-04-20T22:15:35.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:35.791+0000 7f3ae3e00700 1 -- start start 2024-04-20T22:15:35.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:35.792+0000 7f3ae3e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3ae4068540 0x7f3ae4197d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:15:35.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:35.792+0000 7f3ae3e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ae41061b0 0x7f3ae41982a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:15:35.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:35.792+0000 7f3ae3e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ae4198980 con 0x7f3ae4068540 2024-04-20T22:15:35.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:35.792+0000 7f3ae3e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ae419c710 con 0x7f3ae41061b0 2024-04-20T22:15:35.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:35.792+0000 7f3ae2a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3ae4068540 0x7f3ae4197d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:15:35.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:35.792+0000 7f3ae2a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3ae4068540 0x7f3ae4197d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34262/0 (socket says 172.21.15.62:34262) 2024-04-20T22:15:35.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:35.792+0000 7f3ae2a00700 1 -- 172.21.15.62:0/3501688750 learned_addr learned my addr 172.21.15.62:0/3501688750 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:15:35.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:35.793+0000 7f3adbe00700 1 --2- 172.21.15.62:0/3501688750 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ae41061b0 0x7f3ae41982a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:15:35.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:35.793+0000 7f3ae2a00700 1 -- 172.21.15.62:0/3501688750 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ae41061b0 msgr2=0x7f3ae41982a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:15:35.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:35.793+0000 7f3ae2a00700 1 --2- 172.21.15.62:0/3501688750 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ae41061b0 0x7f3ae41982a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:15:35.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:35.793+0000 7f3ae2a00700 1 -- 172.21.15.62:0/3501688750 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3acc0095f0 con 0x7f3ae4068540 2024-04-20T22:15:35.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:35.793+0000 7f3ae2a00700 1 --2- 172.21.15.62:0/3501688750 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3ae4068540 0x7f3ae4197d60 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f3ad400d900 tx=0x7f3ad400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:15:35.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:35.793+0000 7f3ae8a00700 1 -- 172.21.15.62:0/3501688750 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3ad4004a00 con 0x7f3ae4068540 2024-04-20T22:15:35.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:35.793+0000 7f3ae8a00700 1 -- 172.21.15.62:0/3501688750 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3ad40054e0 con 0x7f3ae4068540 2024-04-20T22:15:35.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:35.793+0000 7f3ae8a00700 1 -- 172.21.15.62:0/3501688750 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3ad4009d60 con 0x7f3ae4068540 2024-04-20T22:15:35.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:35.793+0000 7f3ae3e00700 1 -- 172.21.15.62:0/3501688750 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ae419c9f0 con 0x7f3ae4068540 2024-04-20T22:15:35.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:35.794+0000 7f3ae3e00700 1 -- 172.21.15.62:0/3501688750 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ae419cfc0 con 0x7f3ae4068540 2024-04-20T22:15:35.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:35.795+0000 7f3ae3e00700 1 -- 172.21.15.62:0/3501688750 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ae4100a20 con 0x7f3ae4068540 2024-04-20T22:15:35.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:35.796+0000 7f3ae8a00700 1 -- 172.21.15.62:0/3501688750 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f3ad4005020 con 0x7f3ae4068540 2024-04-20T22:15:35.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:35.797+0000 7f3ae8a00700 1 --2- 172.21.15.62:0/3501688750 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3ac806ecc0 0x7f3ac8071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:15:35.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:35.797+0000 7f3ae8a00700 1 -- 172.21.15.62:0/3501688750 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f3ad408efe0 con 0x7f3ae4068540 2024-04-20T22:15:35.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:35.797+0000 7f3adbe00700 1 --2- 172.21.15.62:0/3501688750 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3ac806ecc0 0x7f3ac8071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:15:35.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:35.798+0000 7f3adbe00700 1 --2- 172.21.15.62:0/3501688750 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3ac806ecc0 0x7f3ac8071170 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f3ae4199380 tx=0x7f3acc01a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:15:35.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:35.802+0000 7f3ae8a00700 1 -- 172.21.15.62:0/3501688750 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f3ad4057dd0 con 0x7f3ae4068540 2024-04-20T22:15:36.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.064+0000 7f3ae3e00700 1 -- 172.21.15.62:0/3501688750 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3ae419d2a0 con 0x7f3ac806ecc0 2024-04-20T22:15:36.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.069+0000 7f3ae8a00700 1 -- 172.21.15.62:0/3501688750 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f3ae419d2a0 con 0x7f3ac806ecc0 2024-04-20T22:15:36.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.072+0000 7f3ae3e00700 1 -- 172.21.15.62:0/3501688750 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3ac806ecc0 msgr2=0x7f3ac8071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:15:36.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.072+0000 7f3ae3e00700 1 --2- 172.21.15.62:0/3501688750 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3ac806ecc0 0x7f3ac8071170 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f3ae4199380 tx=0x7f3acc01a040 comp rx=0 tx=0).stop 2024-04-20T22:15:36.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.072+0000 7f3ae3e00700 1 -- 172.21.15.62:0/3501688750 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3ae4068540 msgr2=0x7f3ae4197d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:15:36.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.072+0000 7f3ae3e00700 1 --2- 172.21.15.62:0/3501688750 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3ae4068540 0x7f3ae4197d60 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f3ad400d900 tx=0x7f3ad400dcc0 comp rx=0 tx=0).stop 2024-04-20T22:15:36.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.073+0000 7f3ae3e00700 1 -- 172.21.15.62:0/3501688750 shutdown_connections 2024-04-20T22:15:36.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.073+0000 7f3ae3e00700 1 --2- 172.21.15.62:0/3501688750 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3ac806ecc0 0x7f3ac8071170 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:15:36.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.073+0000 7f3ae3e00700 1 --2- 172.21.15.62:0/3501688750 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3ae4068540 0x7f3ae4197d60 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:15:36.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.073+0000 7f3ae3e00700 1 --2- 172.21.15.62:0/3501688750 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ae41061b0 0x7f3ae41982a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:15:36.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.073+0000 7f3ae3e00700 1 -- 172.21.15.62:0/3501688750 >> 172.21.15.62:0/3501688750 conn(0x7f3ae4074fa0 msgr2=0x7f3ae410a5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:15:36.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.073+0000 7f3ae3e00700 1 -- 172.21.15.62:0/3501688750 shutdown_connections 2024-04-20T22:15:36.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.073+0000 7f3ae3e00700 1 -- 172.21.15.62:0/3501688750 wait complete. 2024-04-20T22:15:36.090 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T22:15:36.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.239+0000 7f9ceea00700 1 -- 172.21.15.62:0/2263483696 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ce8073090 msgr2=0x7f9ce8073460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:15:36.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.239+0000 7f9ceea00700 1 --2- 172.21.15.62:0/2263483696 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ce8073090 0x7f9ce8073460 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f9cd800b0d0 tx=0x7f9cd800b3e0 comp rx=0 tx=0).stop 2024-04-20T22:15:36.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.240+0000 7f9ceea00700 1 -- 172.21.15.62:0/2263483696 shutdown_connections 2024-04-20T22:15:36.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.240+0000 7f9ceea00700 1 --2- 172.21.15.62:0/2263483696 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ce80739a0 0x7f9ce810c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:15:36.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.240+0000 7f9ceea00700 1 --2- 172.21.15.62:0/2263483696 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ce8073090 0x7f9ce8073460 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:15:36.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.240+0000 7f9ceea00700 1 -- 172.21.15.62:0/2263483696 >> 172.21.15.62:0/2263483696 conn(0x7f9ce80fbda0 msgr2=0x7f9ce80fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:15:36.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.240+0000 7f9ceea00700 1 -- 172.21.15.62:0/2263483696 shutdown_connections 2024-04-20T22:15:36.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.240+0000 7f9ceea00700 1 -- 172.21.15.62:0/2263483696 wait complete. 2024-04-20T22:15:36.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.241+0000 7f9ceea00700 1 Processor -- start 2024-04-20T22:15:36.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.241+0000 7f9ceea00700 1 -- start start 2024-04-20T22:15:36.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.242+0000 7f9ceea00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ce8073090 0x7f9ce8198220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:15:36.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.242+0000 7f9ceea00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ce80739a0 0x7f9ce8198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:15:36.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.242+0000 7f9ceea00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ce8198e40 con 0x7f9ce8073090 2024-04-20T22:15:36.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.242+0000 7f9ceea00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ce819cbd0 con 0x7f9ce80739a0 2024-04-20T22:15:36.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.242+0000 7f9ce7e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ce8073090 0x7f9ce8198220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:15:36.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.242+0000 7f9ce7e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ce8073090 0x7f9ce8198220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34284/0 (socket says 172.21.15.62:34284) 2024-04-20T22:15:36.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.242+0000 7f9ce7e00700 1 -- 172.21.15.62:0/3392748843 learned_addr learned my addr 172.21.15.62:0/3392748843 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:15:36.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.242+0000 7f9cdf400700 1 --2- 172.21.15.62:0/3392748843 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ce80739a0 0x7f9ce8198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:15:36.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.242+0000 7f9ce7e00700 1 -- 172.21.15.62:0/3392748843 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ce80739a0 msgr2=0x7f9ce8198760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:15:36.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.242+0000 7f9ce7e00700 1 --2- 172.21.15.62:0/3392748843 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ce80739a0 0x7f9ce8198760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:15:36.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.242+0000 7f9ce7e00700 1 -- 172.21.15.62:0/3392748843 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cd8009d00 con 0x7f9ce8073090 2024-04-20T22:15:36.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.243+0000 7f9ce7e00700 1 --2- 172.21.15.62:0/3392748843 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ce8073090 0x7f9ce8198220 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f9cd8007be0 tx=0x7f9cd8007cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:15:36.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.243+0000 7f9ce5600700 1 -- 172.21.15.62:0/3392748843 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9cd800e040 con 0x7f9ce8073090 2024-04-20T22:15:36.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.243+0000 7f9ce5600700 1 -- 172.21.15.62:0/3392748843 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9cd80096a0 con 0x7f9ce8073090 2024-04-20T22:15:36.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.243+0000 7f9ce5600700 1 -- 172.21.15.62:0/3392748843 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9cd801db10 con 0x7f9ce8073090 2024-04-20T22:15:36.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.243+0000 7f9ceea00700 1 -- 172.21.15.62:0/3392748843 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ce819ce50 con 0x7f9ce8073090 2024-04-20T22:15:36.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.243+0000 7f9ceea00700 1 -- 172.21.15.62:0/3392748843 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ce819d340 con 0x7f9ce8073090 2024-04-20T22:15:36.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.245+0000 7f9ceea00700 1 -- 172.21.15.62:0/3392748843 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ce8109ce0 con 0x7f9ce8073090 2024-04-20T22:15:36.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.245+0000 7f9ce5600700 1 -- 172.21.15.62:0/3392748843 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f9cd8012640 con 0x7f9ce8073090 2024-04-20T22:15:36.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.246+0000 7f9ce5600700 1 --2- 172.21.15.62:0/3392748843 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9ccc06ecc0 0x7f9ccc071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:15:36.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.246+0000 7f9ce5600700 1 -- 172.21.15.62:0/3392748843 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f9cd80905c0 con 0x7f9ce8073090 2024-04-20T22:15:36.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.247+0000 7f9cdf400700 1 --2- 172.21.15.62:0/3392748843 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9ccc06ecc0 0x7f9ccc071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:15:36.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.247+0000 7f9cdf400700 1 --2- 172.21.15.62:0/3392748843 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9ccc06ecc0 0x7f9ccc071170 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f9cd0005950 tx=0x7f9cd00058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:15:36.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.251+0000 7f9ce5600700 1 -- 172.21.15.62:0/3392748843 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f9cd80592d0 con 0x7f9ce8073090 2024-04-20T22:15:36.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.515+0000 7f9ceea00700 1 -- 172.21.15.62:0/3392748843 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9ce8199640 con 0x7f9ccc06ecc0 2024-04-20T22:15:36.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.518+0000 7f9ce5600700 1 -- 172.21.15.62:0/3392748843 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f9ce8199640 con 0x7f9ccc06ecc0 2024-04-20T22:15:36.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.521+0000 7f9ceea00700 1 -- 172.21.15.62:0/3392748843 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9ccc06ecc0 msgr2=0x7f9ccc071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:15:36.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.521+0000 7f9ceea00700 1 --2- 172.21.15.62:0/3392748843 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9ccc06ecc0 0x7f9ccc071170 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f9cd0005950 tx=0x7f9cd00058e0 comp rx=0 tx=0).stop 2024-04-20T22:15:36.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.521+0000 7f9ceea00700 1 -- 172.21.15.62:0/3392748843 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ce8073090 msgr2=0x7f9ce8198220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:15:36.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.521+0000 7f9ceea00700 1 --2- 172.21.15.62:0/3392748843 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ce8073090 0x7f9ce8198220 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f9cd8007be0 tx=0x7f9cd8007cc0 comp rx=0 tx=0).stop 2024-04-20T22:15:36.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.521+0000 7f9ceea00700 1 -- 172.21.15.62:0/3392748843 shutdown_connections 2024-04-20T22:15:36.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.521+0000 7f9ceea00700 1 --2- 172.21.15.62:0/3392748843 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9ccc06ecc0 0x7f9ccc071170 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:15:36.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.521+0000 7f9ceea00700 1 --2- 172.21.15.62:0/3392748843 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ce8073090 0x7f9ce8198220 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:15:36.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.521+0000 7f9ceea00700 1 --2- 172.21.15.62:0/3392748843 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ce80739a0 0x7f9ce8198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:15:36.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.521+0000 7f9ceea00700 1 -- 172.21.15.62:0/3392748843 >> 172.21.15.62:0/3392748843 conn(0x7f9ce80fbda0 msgr2=0x7f9ce8106e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:15:36.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.521+0000 7f9ceea00700 1 -- 172.21.15.62:0/3392748843 shutdown_connections 2024-04-20T22:15:36.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.521+0000 7f9ceea00700 1 -- 172.21.15.62:0/3392748843 wait complete. 2024-04-20T22:15:36.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.687+0000 7f7f6d400700 1 -- 172.21.15.62:0/1129425997 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f68102470 msgr2=0x7f7f681028e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:15:36.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.687+0000 7f7f6d400700 1 --2- 172.21.15.62:0/1129425997 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f68102470 0x7f7f681028e0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f7f54009910 tx=0x7f7f54009c20 comp rx=0 tx=0).stop 2024-04-20T22:15:36.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.687+0000 7f7f6d400700 1 -- 172.21.15.62:0/1129425997 shutdown_connections 2024-04-20T22:15:36.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.687+0000 7f7f6d400700 1 --2- 172.21.15.62:0/1129425997 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f68102470 0x7f7f681028e0 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:15:36.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.687+0000 7f7f6d400700 1 --2- 172.21.15.62:0/1129425997 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f68108470 0x7f7f68108840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:15:36.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.687+0000 7f7f6d400700 1 -- 172.21.15.62:0/1129425997 >> 172.21.15.62:0/1129425997 conn(0x7f7f680fdf90 msgr2=0x7f7f681003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:15:36.688 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:15:36 smithi062 ceph-mon[76322]: pgmap v1212: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:15:36.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.687+0000 7f7f6d400700 1 -- 172.21.15.62:0/1129425997 shutdown_connections 2024-04-20T22:15:36.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.687+0000 7f7f6d400700 1 -- 172.21.15.62:0/1129425997 wait complete. 2024-04-20T22:15:36.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.688+0000 7f7f6d400700 1 Processor -- start 2024-04-20T22:15:36.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.689+0000 7f7f6d400700 1 -- start start 2024-04-20T22:15:36.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.689+0000 7f7f6d400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f68102470 0x7f7f681981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:15:36.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.689+0000 7f7f6d400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f68108470 0x7f7f68198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:15:36.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.689+0000 7f7f6d400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f68198d80 con 0x7f7f68108470 2024-04-20T22:15:36.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.689+0000 7f7f6d400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f68198ec0 con 0x7f7f68102470 2024-04-20T22:15:36.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.690+0000 7f7f65e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f68108470 0x7f7f68198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:15:36.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.690+0000 7f7f65e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f68108470 0x7f7f68198730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34294/0 (socket says 172.21.15.62:34294) 2024-04-20T22:15:36.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.690+0000 7f7f65e00700 1 -- 172.21.15.62:0/2344656288 learned_addr learned my addr 172.21.15.62:0/2344656288 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:15:36.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.690+0000 7f7f66800700 1 --2- 172.21.15.62:0/2344656288 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f68102470 0x7f7f681981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:15:36.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.690+0000 7f7f65e00700 1 -- 172.21.15.62:0/2344656288 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f68102470 msgr2=0x7f7f681981f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:15:36.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.690+0000 7f7f65e00700 1 --2- 172.21.15.62:0/2344656288 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f68102470 0x7f7f681981f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:15:36.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.690+0000 7f7f65e00700 1 -- 172.21.15.62:0/2344656288 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f540095f0 con 0x7f7f68108470 2024-04-20T22:15:36.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.691+0000 7f7f65e00700 1 --2- 172.21.15.62:0/2344656288 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f68108470 0x7f7f68198730 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f7f54006010 tx=0x7f7f5400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:15:36.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.691+0000 7f7f5f400700 1 -- 172.21.15.62:0/2344656288 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7f5401d070 con 0x7f7f68108470 2024-04-20T22:15:36.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.691+0000 7f7f5f400700 1 -- 172.21.15.62:0/2344656288 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7f54004fb0 con 0x7f7f68108470 2024-04-20T22:15:36.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.691+0000 7f7f6d400700 1 -- 172.21.15.62:0/2344656288 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f6819ccb0 con 0x7f7f68108470 2024-04-20T22:15:36.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.691+0000 7f7f5f400700 1 -- 172.21.15.62:0/2344656288 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7f54022ea0 con 0x7f7f68108470 2024-04-20T22:15:36.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.691+0000 7f7f6d400700 1 -- 172.21.15.62:0/2344656288 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f6819d1a0 con 0x7f7f68108470 2024-04-20T22:15:36.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.693+0000 7f7f6d400700 1 -- 172.21.15.62:0/2344656288 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f6810a970 con 0x7f7f68108470 2024-04-20T22:15:36.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.694+0000 7f7f5f400700 1 -- 172.21.15.62:0/2344656288 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f7f54022470 con 0x7f7f68108470 2024-04-20T22:15:36.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.694+0000 7f7f5f400700 1 --2- 172.21.15.62:0/2344656288 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7f4c06ecc0 0x7f7f4c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:15:36.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.695+0000 7f7f5f400700 1 -- 172.21.15.62:0/2344656288 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f7f540903f0 con 0x7f7f68108470 2024-04-20T22:15:36.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.695+0000 7f7f66800700 1 --2- 172.21.15.62:0/2344656288 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7f4c06ecc0 0x7f7f4c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:15:36.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.696+0000 7f7f66800700 1 --2- 172.21.15.62:0/2344656288 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7f4c06ecc0 0x7f7f4c071170 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f7f58005ea0 tx=0x7f7f58005e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:15:36.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.699+0000 7f7f5f400700 1 -- 172.21.15.62:0/2344656288 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f7f54059230 con 0x7f7f68108470 2024-04-20T22:15:36.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:15:36 smithi189 ceph-mon[57022]: pgmap v1212: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:15:36.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.952+0000 7f7f6d400700 1 -- 172.21.15.62:0/2344656288 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f7f6819d410 con 0x7f7f4c06ecc0 2024-04-20T22:15:36.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.972+0000 7f7f5f400700 1 -- 172.21.15.62:0/2344656288 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f7f6819d410 con 0x7f7f4c06ecc0 2024-04-20T22:15:36.972 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T22:15:36.972 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (41m) 2m ago 55m 14.6M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T22:15:36.972 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (55m) 2m ago 55m 11.6M - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T22:15:36.972 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (54m) 2m ago 54m 11.2M - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T22:15:36.972 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (37m) 2m ago 55m 6660k - 19.0.0-3133-ga8c84f59 76bc6b91b726 2825d5ba11c6 2024-04-20T22:15:36.972 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (37m) 2m ago 54m 6665k - 19.0.0-3133-ga8c84f59 76bc6b91b726 20f746fa1bba 2024-04-20T22:15:36.972 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (40m) 2m ago 54m 75.5M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T22:15:36.972 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (32m) 2m ago 48m 22.9M - 19.0.0-3133-ga8c84f59 76bc6b91b726 de80bc90a683 2024-04-20T22:15:36.972 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (33m) 2m ago 48m 23.0M - 19.0.0-3133-ga8c84f59 76bc6b91b726 cd793b7ee98e 2024-04-20T22:15:36.973 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (17m) 2m ago 48m 18.5M - 19.0.0-3133-ga8c84f59 76bc6b91b726 0e7b0bc4d1ab 2024-04-20T22:15:36.973 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (2m) 2m ago 47m 11.4M - 19.0.0-3133-ga8c84f59 76bc6b91b726 0af732ac9e06 2024-04-20T22:15:36.973 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (44m) 2m ago 57m 517M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T22:15:36.973 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (43m) 2m ago 54m 460M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T22:15:36.973 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (39m) 2m ago 57m 86.3M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 019eb1479485 2024-04-20T22:15:36.973 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (38m) 2m ago 53m 58.0M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 55aac560aab6 2024-04-20T22:15:36.973 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (42m) 2m ago 55m 8766k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T22:15:36.973 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (42m) 2m ago 54m 8711k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T22:15:36.973 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (36m) 2m ago 52m 91.3M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 ee2bb901f26e 2024-04-20T22:15:36.973 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (36m) 2m ago 52m 94.9M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 9075152d4162 2024-04-20T22:15:36.973 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (35m) 2m ago 51m 78.5M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 2d4f3e38dc0e 2024-04-20T22:15:36.973 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (35m) 2m ago 51m 94.1M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 8f97b17ae0eb 2024-04-20T22:15:36.973 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (34m) 2m ago 50m 96.1M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 653d8079967d 2024-04-20T22:15:36.973 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (33m) 2m ago 50m 90.9M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 596dded14c12 2024-04-20T22:15:36.973 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (42m) 2m ago 54m 90.4M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T22:15:36.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.975+0000 7f7f6d400700 1 -- 172.21.15.62:0/2344656288 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7f4c06ecc0 msgr2=0x7f7f4c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:15:36.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.975+0000 7f7f6d400700 1 --2- 172.21.15.62:0/2344656288 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7f4c06ecc0 0x7f7f4c071170 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f7f58005ea0 tx=0x7f7f58005e30 comp rx=0 tx=0).stop 2024-04-20T22:15:36.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.975+0000 7f7f6d400700 1 -- 172.21.15.62:0/2344656288 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f68108470 msgr2=0x7f7f68198730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:15:36.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.975+0000 7f7f6d400700 1 --2- 172.21.15.62:0/2344656288 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f68108470 0x7f7f68198730 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f7f54006010 tx=0x7f7f5400be20 comp rx=0 tx=0).stop 2024-04-20T22:15:36.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.975+0000 7f7f6d400700 1 -- 172.21.15.62:0/2344656288 shutdown_connections 2024-04-20T22:15:36.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.975+0000 7f7f6d400700 1 --2- 172.21.15.62:0/2344656288 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7f4c06ecc0 0x7f7f4c071170 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:15:36.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.975+0000 7f7f6d400700 1 --2- 172.21.15.62:0/2344656288 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f68102470 0x7f7f681981f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:15:36.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.975+0000 7f7f6d400700 1 --2- 172.21.15.62:0/2344656288 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f68108470 0x7f7f68198730 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:15:36.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.975+0000 7f7f6d400700 1 -- 172.21.15.62:0/2344656288 >> 172.21.15.62:0/2344656288 conn(0x7f7f680fdf90 msgr2=0x7f7f680ff3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:15:36.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.975+0000 7f7f6d400700 1 -- 172.21.15.62:0/2344656288 shutdown_connections 2024-04-20T22:15:36.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:36.976+0000 7f7f6d400700 1 -- 172.21.15.62:0/2344656288 wait complete. 2024-04-20T22:15:37.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:37.142+0000 7f919be00700 1 -- 172.21.15.62:0/3309028939 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91941004e0 msgr2=0x7f91941008b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:15:37.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:37.142+0000 7f919be00700 1 --2- 172.21.15.62:0/3309028939 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91941004e0 0x7f91941008b0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f9188009910 tx=0x7f9188009c20 comp rx=0 tx=0).stop 2024-04-20T22:15:37.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:37.142+0000 7f919be00700 1 -- 172.21.15.62:0/3309028939 shutdown_connections 2024-04-20T22:15:37.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:37.142+0000 7f919be00700 1 --2- 172.21.15.62:0/3309028939 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9194100e80 0x7f9194104e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:15:37.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:37.142+0000 7f919be00700 1 --2- 172.21.15.62:0/3309028939 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91941004e0 0x7f91941008b0 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:15:37.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:37.142+0000 7f919be00700 1 -- 172.21.15.62:0/3309028939 >> 172.21.15.62:0/3309028939 conn(0x7f91940fbda0 msgr2=0x7f91940fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:15:37.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:37.142+0000 7f919be00700 1 -- 172.21.15.62:0/3309028939 shutdown_connections 2024-04-20T22:15:37.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:37.143+0000 7f919be00700 1 -- 172.21.15.62:0/3309028939 wait complete. 2024-04-20T22:15:37.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:37.144+0000 7f919be00700 1 Processor -- start 2024-04-20T22:15:37.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:37.144+0000 7f919be00700 1 -- start start 2024-04-20T22:15:37.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:37.145+0000 7f919be00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91941004e0 0x7f9194195ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:15:37.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:37.145+0000 7f919be00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9194100e80 0x7f9194196430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:15:37.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:37.145+0000 7f919be00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9194196b10 con 0x7f9194100e80 2024-04-20T22:15:37.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:37.145+0000 7f919be00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f919419a8a0 con 0x7f91941004e0 2024-04-20T22:15:37.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:37.145+0000 7f9198a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9194100e80 0x7f9194196430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:15:37.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:37.145+0000 7f9198a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9194100e80 0x7f9194196430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34310/0 (socket says 172.21.15.62:34310) 2024-04-20T22:15:37.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:37.145+0000 7f9198a00700 1 -- 172.21.15.62:0/631629507 learned_addr learned my addr 172.21.15.62:0/631629507 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:15:37.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:37.145+0000 7f9199400700 1 --2- 172.21.15.62:0/631629507 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91941004e0 0x7f9194195ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:15:37.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:37.146+0000 7f9198a00700 1 -- 172.21.15.62:0/631629507 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91941004e0 msgr2=0x7f9194195ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:15:37.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:37.146+0000 7f9198a00700 1 --2- 172.21.15.62:0/631629507 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91941004e0 0x7f9194195ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:15:37.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:37.146+0000 7f9198a00700 1 -- 172.21.15.62:0/631629507 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91880095f0 con 0x7f9194100e80 2024-04-20T22:15:37.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:37.146+0000 7f9199400700 1 --2- 172.21.15.62:0/631629507 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91941004e0 0x7f9194195ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-20T22:15:37.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:37.146+0000 7f9198a00700 1 --2- 172.21.15.62:0/631629507 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9194100e80 0x7f9194196430 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f918000d900 tx=0x7f918000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:15:37.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:37.146+0000 7f9192000700 1 -- 172.21.15.62:0/631629507 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9180009900 con 0x7f9194100e80 2024-04-20T22:15:37.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:37.146+0000 7f919be00700 1 -- 172.21.15.62:0/631629507 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f919419ab80 con 0x7f9194100e80 2024-04-20T22:15:37.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:37.146+0000 7f9192000700 1 -- 172.21.15.62:0/631629507 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9180010460 con 0x7f9194100e80 2024-04-20T22:15:37.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:37.146+0000 7f9192000700 1 -- 172.21.15.62:0/631629507 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f918000f5f0 con 0x7f9194100e80 2024-04-20T22:15:37.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:37.146+0000 7f919be00700 1 -- 172.21.15.62:0/631629507 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f919419b0d0 con 0x7f9194100e80 2024-04-20T22:15:37.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:37.148+0000 7f919be00700 1 -- 172.21.15.62:0/631629507 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91941087f0 con 0x7f9194100e80 2024-04-20T22:15:37.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:37.149+0000 7f9192000700 1 -- 172.21.15.62:0/631629507 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f91800105d0 con 0x7f9194100e80 2024-04-20T22:15:37.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:37.150+0000 7f9192000700 1 --2- 172.21.15.62:0/631629507 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f917c06ecc0 0x7f917c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:15:37.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:37.150+0000 7f9192000700 1 -- 172.21.15.62:0/631629507 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f918008ed60 con 0x7f9194100e80 2024-04-20T22:15:37.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:37.150+0000 7f9199400700 1 --2- 172.21.15.62:0/631629507 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f917c06ecc0 0x7f917c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:15:37.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:37.151+0000 7f9199400700 1 --2- 172.21.15.62:0/631629507 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f917c06ecc0 0x7f917c071170 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f9188005b40 tx=0x7f918801a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:15:37.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:37.155+0000 7f9192000700 1 -- 172.21.15.62:0/631629507 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f9180093070 con 0x7f9194100e80 2024-04-20T22:15:37.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:37.481+0000 7f919be00700 1 -- 172.21.15.62:0/631629507 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f919419b2f0 con 0x7f9194100e80 2024-04-20T22:15:37.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:37.482+0000 7f9192000700 1 -- 172.21.15.62:0/631629507 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f9180057a70 con 0x7f9194100e80 2024-04-20T22:15:37.482 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:15:37.483 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T22:15:37.483 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:15:37.483 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:15:37.483 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T22:15:37.483 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:15:37.483 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:15:37.483 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T22:15:37.483 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 6 2024-04-20T22:15:37.483 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:15:37.483 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T22:15:37.483 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 4 2024-04-20T22:15:37.483 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:15:37.483 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T22:15:37.484 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 14 2024-04-20T22:15:37.484 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T22:15:37.484 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:15:37.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:37.485+0000 7f919be00700 1 -- 172.21.15.62:0/631629507 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f917c06ecc0 msgr2=0x7f917c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:15:37.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:37.486+0000 7f919be00700 1 --2- 172.21.15.62:0/631629507 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f917c06ecc0 0x7f917c071170 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f9188005b40 tx=0x7f918801a040 comp rx=0 tx=0).stop 2024-04-20T22:15:37.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:37.486+0000 7f919be00700 1 -- 172.21.15.62:0/631629507 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9194100e80 msgr2=0x7f9194196430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:15:37.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:37.486+0000 7f919be00700 1 --2- 172.21.15.62:0/631629507 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9194100e80 0x7f9194196430 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f918000d900 tx=0x7f918000dcc0 comp rx=0 tx=0).stop 2024-04-20T22:15:37.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:37.486+0000 7f919be00700 1 -- 172.21.15.62:0/631629507 shutdown_connections 2024-04-20T22:15:37.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:37.486+0000 7f919be00700 1 --2- 172.21.15.62:0/631629507 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f917c06ecc0 0x7f917c071170 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:15:37.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:37.486+0000 7f919be00700 1 --2- 172.21.15.62:0/631629507 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91941004e0 0x7f9194195ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:15:37.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:37.486+0000 7f919be00700 1 --2- 172.21.15.62:0/631629507 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9194100e80 0x7f9194196430 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:15:37.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:37.486+0000 7f919be00700 1 -- 172.21.15.62:0/631629507 >> 172.21.15.62:0/631629507 conn(0x7f91940fbda0 msgr2=0x7f91940fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:15:37.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:37.486+0000 7f919be00700 1 -- 172.21.15.62:0/631629507 shutdown_connections 2024-04-20T22:15:37.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:37.486+0000 7f919be00700 1 -- 172.21.15.62:0/631629507 wait complete. 2024-04-20T22:15:37.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:37.641+0000 7fdf6a000700 1 -- 172.21.15.62:0/3411700002 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdf64102470 msgr2=0x7fdf641028e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:15:37.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:37.641+0000 7fdf6a000700 1 --2- 172.21.15.62:0/3411700002 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdf64102470 0x7fdf641028e0 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7fdf4c009910 tx=0x7fdf4c009c20 comp rx=0 tx=0).stop 2024-04-20T22:15:37.642 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:15:37 smithi062 ceph-mon[76322]: from='client.36134 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:15:37.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:37.641+0000 7fdf6a000700 1 -- 172.21.15.62:0/3411700002 shutdown_connections 2024-04-20T22:15:37.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:37.641+0000 7fdf6a000700 1 --2- 172.21.15.62:0/3411700002 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdf64102470 0x7fdf641028e0 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:15:37.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:37.641+0000 7fdf6a000700 1 --2- 172.21.15.62:0/3411700002 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdf64108470 0x7fdf64108840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:15:37.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:37.641+0000 7fdf6a000700 1 -- 172.21.15.62:0/3411700002 >> 172.21.15.62:0/3411700002 conn(0x7fdf640fdf90 msgr2=0x7fdf641003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:15:37.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:37.641+0000 7fdf6a000700 1 -- 172.21.15.62:0/3411700002 shutdown_connections 2024-04-20T22:15:37.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:37.642+0000 7fdf6a000700 1 -- 172.21.15.62:0/3411700002 wait complete. 2024-04-20T22:15:37.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:37.643+0000 7fdf6a000700 1 Processor -- start 2024-04-20T22:15:37.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:37.643+0000 7fdf6a000700 1 -- start start 2024-04-20T22:15:37.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:37.643+0000 7fdf6a000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdf64102470 0x7fdf64198270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:15:37.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:37.643+0000 7fdf6a000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdf64108470 0x7fdf641987b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:15:37.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:37.643+0000 7fdf6a000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf64198e90 con 0x7fdf64102470 2024-04-20T22:15:37.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:37.643+0000 7fdf6a000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf6419cc20 con 0x7fdf64108470 2024-04-20T22:15:37.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:37.644+0000 7fdf63400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdf64102470 0x7fdf64198270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:15:37.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:37.644+0000 7fdf63400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdf64102470 0x7fdf64198270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34328/0 (socket says 172.21.15.62:34328) 2024-04-20T22:15:37.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:37.644+0000 7fdf63400700 1 -- 172.21.15.62:0/4101534152 learned_addr learned my addr 172.21.15.62:0/4101534152 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:15:37.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:37.644+0000 7fdf5aa00700 1 --2- 172.21.15.62:0/4101534152 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdf64108470 0x7fdf641987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:15:37.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:37.644+0000 7fdf63400700 1 -- 172.21.15.62:0/4101534152 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdf64108470 msgr2=0x7fdf641987b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:15:37.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:37.644+0000 7fdf63400700 1 --2- 172.21.15.62:0/4101534152 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdf64108470 0x7fdf641987b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:15:37.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:37.644+0000 7fdf63400700 1 -- 172.21.15.62:0/4101534152 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf4c0095f0 con 0x7fdf64102470 2024-04-20T22:15:37.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:37.645+0000 7fdf63400700 1 --2- 172.21.15.62:0/4101534152 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdf64102470 0x7fdf64198270 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7fdf5400c8a0 tx=0x7fdf5400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:15:37.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:37.645+0000 7fdf60c00700 1 -- 172.21.15.62:0/4101534152 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdf5400cea0 con 0x7fdf64102470 2024-04-20T22:15:37.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:37.645+0000 7fdf60c00700 1 -- 172.21.15.62:0/4101534152 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdf54004830 con 0x7fdf64102470 2024-04-20T22:15:37.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:37.645+0000 7fdf60c00700 1 -- 172.21.15.62:0/4101534152 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdf540055b0 con 0x7fdf64102470 2024-04-20T22:15:37.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:37.645+0000 7fdf6a000700 1 -- 172.21.15.62:0/4101534152 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf6419cf00 con 0x7fdf64102470 2024-04-20T22:15:37.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:37.645+0000 7fdf6a000700 1 -- 172.21.15.62:0/4101534152 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf6419d450 con 0x7fdf64102470 2024-04-20T22:15:37.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:37.646+0000 7fdf6a000700 1 -- 172.21.15.62:0/4101534152 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf6410a970 con 0x7fdf64102470 2024-04-20T22:15:37.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:37.647+0000 7fdf60c00700 1 -- 172.21.15.62:0/4101534152 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fdf540049a0 con 0x7fdf64102470 2024-04-20T22:15:37.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:37.648+0000 7fdf60c00700 1 --2- 172.21.15.62:0/4101534152 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fdf4806ed10 0x7fdf480711c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:15:37.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:37.649+0000 7fdf60c00700 1 -- 172.21.15.62:0/4101534152 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fdf5408eaf0 con 0x7fdf64102470 2024-04-20T22:15:37.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:37.649+0000 7fdf5aa00700 1 --2- 172.21.15.62:0/4101534152 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fdf4806ed10 0x7fdf480711c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:15:37.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:37.650+0000 7fdf5aa00700 1 --2- 172.21.15.62:0/4101534152 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fdf4806ed10 0x7fdf480711c0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fdf64199890 tx=0x7fdf4c01a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:15:37.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:37.653+0000 7fdf60c00700 1 -- 172.21.15.62:0/4101534152 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fdf54057880 con 0x7fdf64102470 2024-04-20T22:15:37.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:15:37 smithi189 ceph-mon[57022]: from='client.36134 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:15:37.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:37.937+0000 7fdf6a000700 1 -- 172.21.15.62:0/4101534152 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fdf641996a0 con 0x7fdf64102470 2024-04-20T22:15:37.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:37.938+0000 7fdf60c00700 1 -- 172.21.15.62:0/4101534152 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 29 v29) v1 ==== 76+0+1887 (secure 0 0 0) 0x7fdf5401d070 con 0x7fdf64102470 2024-04-20T22:15:37.939 INFO:teuthology.orchestra.run.smithi062.stdout:e29 2024-04-20T22:15:37.939 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T22:15:37.939 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:15:37.939 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T22:15:37.940 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:15:37.940 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T22:15:37.940 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T22:15:37.940 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 20 2024-04-20T22:15:37.940 INFO:teuthology.orchestra.run.smithi062.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-20T22:15:37.940 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T22:15:37.940 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T21:42:01.559061+0000 2024-04-20T22:15:37.940 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T22:15:37.940 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T22:15:37.940 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T22:15:37.940 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T22:15:37.940 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T22:15:37.940 INFO:teuthology.orchestra.run.smithi062.stdout:max_xattr_size 65536 2024-04-20T22:15:37.940 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T22:15:37.941 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T22:15:37.941 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 81 2024-04-20T22:15:37.941 INFO:teuthology.orchestra.run.smithi062.stdout:snap_rstat 1 2024-04-20T22:15:37.941 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:15:37.941 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 2 2024-04-20T22:15:37.941 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1 2024-04-20T22:15:37.941 INFO:teuthology.orchestra.run.smithi062.stdout:up {} 2024-04-20T22:15:37.941 INFO:teuthology.orchestra.run.smithi062.stdout:failed 0,1 2024-04-20T22:15:37.941 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T22:15:37.941 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T22:15:37.941 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T22:15:37.941 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T22:15:37.942 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data enabled 2024-04-20T22:15:37.942 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T22:15:37.942 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T22:15:37.942 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 1 2024-04-20T22:15:37.942 INFO:teuthology.orchestra.run.smithi062.stdout:qdb_cluster leader: 0 members: 2024-04-20T22:15:37.942 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:15:37.942 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:15:37.942 INFO:teuthology.orchestra.run.smithi062.stdout:Standby daemons: 2024-04-20T22:15:37.942 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:15:37.942 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ryqblb{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6826/4113916205,v1:172.21.15.62:6827/4113916205] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:15:37.942 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ozizwi{-1:34476} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6828/3801975165,v1:172.21.15.62:6829/3801975165] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:15:37.942 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.dvimer{-1:44727} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6824/3576699232,v1:172.21.15.189:6825/3576699232] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:15:37.942 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.xbgspw{-1:45075} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6826/212905792,v1:172.21.15.189:6827/212905792] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:15:37.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:37.941+0000 7fdf6a000700 1 -- 172.21.15.62:0/4101534152 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fdf4806ed10 msgr2=0x7fdf480711c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:15:37.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:37.941+0000 7fdf6a000700 1 --2- 172.21.15.62:0/4101534152 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fdf4806ed10 0x7fdf480711c0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fdf64199890 tx=0x7fdf4c01a040 comp rx=0 tx=0).stop 2024-04-20T22:15:37.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:37.942+0000 7fdf6a000700 1 -- 172.21.15.62:0/4101534152 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdf64102470 msgr2=0x7fdf64198270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:15:37.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:37.942+0000 7fdf6a000700 1 --2- 172.21.15.62:0/4101534152 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdf64102470 0x7fdf64198270 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7fdf5400c8a0 tx=0x7fdf5400cc60 comp rx=0 tx=0).stop 2024-04-20T22:15:37.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:37.942+0000 7fdf6a000700 1 -- 172.21.15.62:0/4101534152 shutdown_connections 2024-04-20T22:15:37.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:37.942+0000 7fdf6a000700 1 --2- 172.21.15.62:0/4101534152 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fdf4806ed10 0x7fdf480711c0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:15:37.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:37.942+0000 7fdf6a000700 1 --2- 172.21.15.62:0/4101534152 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdf64102470 0x7fdf64198270 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:15:37.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:37.942+0000 7fdf6a000700 1 --2- 172.21.15.62:0/4101534152 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdf64108470 0x7fdf641987b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:15:37.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:37.942+0000 7fdf6a000700 1 -- 172.21.15.62:0/4101534152 >> 172.21.15.62:0/4101534152 conn(0x7fdf640fdf90 msgr2=0x7fdf640ff410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:15:37.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:37.942+0000 7fdf6a000700 1 -- 172.21.15.62:0/4101534152 shutdown_connections 2024-04-20T22:15:37.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:37.942+0000 7fdf6a000700 1 -- 172.21.15.62:0/4101534152 wait complete. 2024-04-20T22:15:37.944 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 29 2024-04-20T22:15:38.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:38.100+0000 7feb80e00700 1 -- 172.21.15.62:0/3448066667 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb7c1025b0 msgr2=0x7feb7c102980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:15:38.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:38.100+0000 7feb80e00700 1 --2- 172.21.15.62:0/3448066667 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb7c1025b0 0x7feb7c102980 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7feb6c009910 tx=0x7feb6c009c20 comp rx=0 tx=0).stop 2024-04-20T22:15:38.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:38.100+0000 7feb80e00700 1 -- 172.21.15.62:0/3448066667 shutdown_connections 2024-04-20T22:15:38.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:38.100+0000 7feb80e00700 1 --2- 172.21.15.62:0/3448066667 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb7c102f50 0x7feb7c106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:15:38.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:38.100+0000 7feb80e00700 1 --2- 172.21.15.62:0/3448066667 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb7c1025b0 0x7feb7c102980 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:15:38.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:38.100+0000 7feb80e00700 1 -- 172.21.15.62:0/3448066667 >> 172.21.15.62:0/3448066667 conn(0x7feb7c0fde50 msgr2=0x7feb7c100260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:15:38.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:38.100+0000 7feb80e00700 1 -- 172.21.15.62:0/3448066667 shutdown_connections 2024-04-20T22:15:38.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:38.100+0000 7feb80e00700 1 -- 172.21.15.62:0/3448066667 wait complete. 2024-04-20T22:15:38.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:38.101+0000 7feb80e00700 1 Processor -- start 2024-04-20T22:15:38.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:38.101+0000 7feb80e00700 1 -- start start 2024-04-20T22:15:38.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:38.102+0000 7feb80e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb7c1025b0 0x7feb7c197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:15:38.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:38.102+0000 7feb80e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb7c102f50 0x7feb7c198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:15:38.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:38.102+0000 7feb80e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb7c198c00 con 0x7feb7c1025b0 2024-04-20T22:15:38.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:38.102+0000 7feb80e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb7c19c990 con 0x7feb7c102f50 2024-04-20T22:15:38.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:38.102+0000 7feb79e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb7c1025b0 0x7feb7c197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:15:38.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:38.102+0000 7feb79e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb7c1025b0 0x7feb7c197fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34340/0 (socket says 172.21.15.62:34340) 2024-04-20T22:15:38.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:38.102+0000 7feb79e00700 1 -- 172.21.15.62:0/670951467 learned_addr learned my addr 172.21.15.62:0/670951467 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:15:38.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:38.102+0000 7feb79400700 1 --2- 172.21.15.62:0/670951467 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb7c102f50 0x7feb7c198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:15:38.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:38.102+0000 7feb79e00700 1 -- 172.21.15.62:0/670951467 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb7c102f50 msgr2=0x7feb7c198520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:15:38.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:38.102+0000 7feb79e00700 1 --2- 172.21.15.62:0/670951467 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb7c102f50 0x7feb7c198520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:15:38.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:38.102+0000 7feb79e00700 1 -- 172.21.15.62:0/670951467 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb6c0095f0 con 0x7feb7c1025b0 2024-04-20T22:15:38.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:38.102+0000 7feb79400700 1 --2- 172.21.15.62:0/670951467 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb7c102f50 0x7feb7c198520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T22:15:38.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:38.103+0000 7feb79e00700 1 --2- 172.21.15.62:0/670951467 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb7c1025b0 0x7feb7c197fe0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7feb6c006010 tx=0x7feb6c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:15:38.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:38.103+0000 7feb72a00700 1 -- 172.21.15.62:0/670951467 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feb6c01d070 con 0x7feb7c1025b0 2024-04-20T22:15:38.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:38.103+0000 7feb72a00700 1 -- 172.21.15.62:0/670951467 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7feb6c004fb0 con 0x7feb7c1025b0 2024-04-20T22:15:38.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:38.103+0000 7feb72a00700 1 -- 172.21.15.62:0/670951467 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feb6c022ea0 con 0x7feb7c1025b0 2024-04-20T22:15:38.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:38.103+0000 7feb80e00700 1 -- 172.21.15.62:0/670951467 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb7c19cc10 con 0x7feb7c1025b0 2024-04-20T22:15:38.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:38.103+0000 7feb80e00700 1 -- 172.21.15.62:0/670951467 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb7c19d100 con 0x7feb7c1025b0 2024-04-20T22:15:38.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:38.104+0000 7feb80e00700 1 -- 172.21.15.62:0/670951467 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb7c10a8c0 con 0x7feb7c1025b0 2024-04-20T22:15:38.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:38.105+0000 7feb72a00700 1 -- 172.21.15.62:0/670951467 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7feb6c022470 con 0x7feb7c1025b0 2024-04-20T22:15:38.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:38.107+0000 7feb72a00700 1 --2- 172.21.15.62:0/670951467 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7feb6006ecc0 0x7feb60071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:15:38.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:38.107+0000 7feb72a00700 1 -- 172.21.15.62:0/670951467 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7feb6c0903c0 con 0x7feb7c1025b0 2024-04-20T22:15:38.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:38.107+0000 7feb79400700 1 --2- 172.21.15.62:0/670951467 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7feb6006ecc0 0x7feb60071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:15:38.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:38.108+0000 7feb79400700 1 --2- 172.21.15.62:0/670951467 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7feb6006ecc0 0x7feb60071170 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7feb7c199600 tx=0x7feb68008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:15:38.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:38.112+0000 7feb72a00700 1 -- 172.21.15.62:0/670951467 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7feb6c059200 con 0x7feb7c1025b0 2024-04-20T22:15:38.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:38.369+0000 7feb80e00700 1 -- 172.21.15.62:0/670951467 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7feb7c19d4a0 con 0x7feb6006ecc0 2024-04-20T22:15:38.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:38.371+0000 7feb72a00700 1 -- 172.21.15.62:0/670951467 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7feb7c19d4a0 con 0x7feb6006ecc0 2024-04-20T22:15:38.371 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:15:38.371 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T22:15:38.371 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T22:15:38.371 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-20T22:15:38.372 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T22:15:38.372 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr", 2024-04-20T22:15:38.372 INFO:teuthology.orchestra.run.smithi062.stdout: "osd", 2024-04-20T22:15:38.372 INFO:teuthology.orchestra.run.smithi062.stdout: "mds", 2024-04-20T22:15:38.372 INFO:teuthology.orchestra.run.smithi062.stdout: "crash", 2024-04-20T22:15:38.372 INFO:teuthology.orchestra.run.smithi062.stdout: "mon" 2024-04-20T22:15:38.372 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T22:15:38.372 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "16/23 daemons upgraded", 2024-04-20T22:15:38.372 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading mds daemons", 2024-04-20T22:15:38.372 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T22:15:38.372 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:15:38.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:38.375+0000 7feb80e00700 1 -- 172.21.15.62:0/670951467 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7feb6006ecc0 msgr2=0x7feb60071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:15:38.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:38.375+0000 7feb80e00700 1 --2- 172.21.15.62:0/670951467 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7feb6006ecc0 0x7feb60071170 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7feb7c199600 tx=0x7feb68008040 comp rx=0 tx=0).stop 2024-04-20T22:15:38.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:38.375+0000 7feb80e00700 1 -- 172.21.15.62:0/670951467 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb7c1025b0 msgr2=0x7feb7c197fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:15:38.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:38.375+0000 7feb80e00700 1 --2- 172.21.15.62:0/670951467 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb7c1025b0 0x7feb7c197fe0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7feb6c006010 tx=0x7feb6c00be20 comp rx=0 tx=0).stop 2024-04-20T22:15:38.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:38.375+0000 7feb80e00700 1 -- 172.21.15.62:0/670951467 shutdown_connections 2024-04-20T22:15:38.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:38.375+0000 7feb80e00700 1 --2- 172.21.15.62:0/670951467 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7feb6006ecc0 0x7feb60071170 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:15:38.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:38.375+0000 7feb80e00700 1 --2- 172.21.15.62:0/670951467 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb7c1025b0 0x7feb7c197fe0 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:15:38.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:38.375+0000 7feb80e00700 1 --2- 172.21.15.62:0/670951467 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb7c102f50 0x7feb7c198520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:15:38.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:38.375+0000 7feb80e00700 1 -- 172.21.15.62:0/670951467 >> 172.21.15.62:0/670951467 conn(0x7feb7c0fde50 msgr2=0x7feb7c0ff4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:15:38.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:38.375+0000 7feb80e00700 1 -- 172.21.15.62:0/670951467 shutdown_connections 2024-04-20T22:15:38.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:38.375+0000 7feb80e00700 1 -- 172.21.15.62:0/670951467 wait complete. 2024-04-20T22:15:38.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:38.540+0000 7fdb5b000700 1 -- 172.21.15.62:0/2594422420 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdb540ff210 msgr2=0x7fdb540ff5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:15:38.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:38.540+0000 7fdb5b000700 1 --2- 172.21.15.62:0/2594422420 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdb540ff210 0x7fdb540ff5e0 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7fdb44009910 tx=0x7fdb44009c20 comp rx=0 tx=0).stop 2024-04-20T22:15:38.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:38.541+0000 7fdb5b000700 1 -- 172.21.15.62:0/2594422420 shutdown_connections 2024-04-20T22:15:38.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:38.541+0000 7fdb5b000700 1 --2- 172.21.15.62:0/2594422420 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb540ffb20 0x7fdb54104000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:15:38.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:38.541+0000 7fdb5b000700 1 --2- 172.21.15.62:0/2594422420 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdb540ff210 0x7fdb540ff5e0 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:15:38.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:38.541+0000 7fdb5b000700 1 -- 172.21.15.62:0/2594422420 >> 172.21.15.62:0/2594422420 conn(0x7fdb540fac80 msgr2=0x7fdb540fd090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:15:38.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:38.541+0000 7fdb5b000700 1 -- 172.21.15.62:0/2594422420 shutdown_connections 2024-04-20T22:15:38.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:38.541+0000 7fdb5b000700 1 -- 172.21.15.62:0/2594422420 wait complete. 2024-04-20T22:15:38.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:38.542+0000 7fdb5b000700 1 Processor -- start 2024-04-20T22:15:38.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:38.542+0000 7fdb5b000700 1 -- start start 2024-04-20T22:15:38.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:38.543+0000 7fdb5b000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdb540ff210 0x7fdb5419c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:15:38.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:38.543+0000 7fdb5b000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb540ffb20 0x7fdb5419c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:15:38.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:38.543+0000 7fdb5b000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb5419d070 con 0x7fdb540ff210 2024-04-20T22:15:38.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:38.543+0000 7fdb5b000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb541a0e00 con 0x7fdb540ffb20 2024-04-20T22:15:38.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:38.543+0000 7fdb53e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdb540ff210 0x7fdb5419c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:15:38.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:38.543+0000 7fdb53e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdb540ff210 0x7fdb5419c450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34358/0 (socket says 172.21.15.62:34358) 2024-04-20T22:15:38.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:38.543+0000 7fdb53e00700 1 -- 172.21.15.62:0/135092127 learned_addr learned my addr 172.21.15.62:0/135092127 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:15:38.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:38.543+0000 7fdb53400700 1 --2- 172.21.15.62:0/135092127 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb540ffb20 0x7fdb5419c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:15:38.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:38.544+0000 7fdb53e00700 1 -- 172.21.15.62:0/135092127 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb540ffb20 msgr2=0x7fdb5419c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:15:38.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:38.544+0000 7fdb53e00700 1 --2- 172.21.15.62:0/135092127 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb540ffb20 0x7fdb5419c990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:15:38.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:38.544+0000 7fdb53e00700 1 -- 172.21.15.62:0/135092127 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb440095f0 con 0x7fdb540ff210 2024-04-20T22:15:38.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:38.544+0000 7fdb53e00700 1 --2- 172.21.15.62:0/135092127 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdb540ff210 0x7fdb5419c450 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7fdb440092e0 tx=0x7fdb4400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:15:38.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:38.544+0000 7fdb50c00700 1 -- 172.21.15.62:0/135092127 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdb4401d070 con 0x7fdb540ff210 2024-04-20T22:15:38.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:38.544+0000 7fdb50c00700 1 -- 172.21.15.62:0/135092127 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdb44005e00 con 0x7fdb540ff210 2024-04-20T22:15:38.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:38.544+0000 7fdb50c00700 1 -- 172.21.15.62:0/135092127 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdb44017400 con 0x7fdb540ff210 2024-04-20T22:15:38.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:38.544+0000 7fdb5b000700 1 -- 172.21.15.62:0/135092127 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb541a1080 con 0x7fdb540ff210 2024-04-20T22:15:38.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:38.544+0000 7fdb5b000700 1 -- 172.21.15.62:0/135092127 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb541a1570 con 0x7fdb540ff210 2024-04-20T22:15:38.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:38.546+0000 7fdb5b000700 1 -- 172.21.15.62:0/135092127 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb54101690 con 0x7fdb540ff210 2024-04-20T22:15:38.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:38.547+0000 7fdb50c00700 1 -- 172.21.15.62:0/135092127 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fdb4400f460 con 0x7fdb540ff210 2024-04-20T22:15:38.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:38.548+0000 7fdb50c00700 1 --2- 172.21.15.62:0/135092127 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fdb3806ecc0 0x7fdb38071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:15:38.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:38.548+0000 7fdb50c00700 1 -- 172.21.15.62:0/135092127 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fdb44090290 con 0x7fdb540ff210 2024-04-20T22:15:38.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:38.548+0000 7fdb53400700 1 --2- 172.21.15.62:0/135092127 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fdb3806ecc0 0x7fdb38071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:15:38.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:38.549+0000 7fdb53400700 1 --2- 172.21.15.62:0/135092127 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fdb3806ecc0 0x7fdb38071170 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fdb40005950 tx=0x7fdb400058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:15:38.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:38.553+0000 7fdb50c00700 1 -- 172.21.15.62:0/135092127 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fdb440590c0 con 0x7fdb540ff210 2024-04-20T22:15:38.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:15:38 smithi189 ceph-mon[57022]: from='client.36138 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:15:38.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:15:38 smithi189 ceph-mon[57022]: from='client.36142 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:15:38.739 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:15:38 smithi189 ceph-mon[57022]: pgmap v1213: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:15:38.739 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:15:38 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/631629507' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:15:38.739 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:15:38 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/4101534152' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:15:38.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:15:38 smithi062 ceph-mon[76322]: from='client.36138 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:15:38.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:15:38 smithi062 ceph-mon[76322]: from='client.36142 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:15:38.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:15:38 smithi062 ceph-mon[76322]: pgmap v1213: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:15:38.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:15:38 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/631629507' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:15:38.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:15:38 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/4101534152' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:15:38.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:38.887+0000 7fdb5b000700 1 -- 172.21.15.62:0/135092127 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fdb5404f2a0 con 0x7fdb540ff210 2024-04-20T22:15:38.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:38.888+0000 7fdb50c00700 1 -- 172.21.15.62:0/135092127 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7fdb4405c780 con 0x7fdb540ff210 2024-04-20T22:15:38.889 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-20T22:15:38.889 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-20T22:15:38.889 INFO:teuthology.orchestra.run.smithi062.stdout: host smithi189 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi189/config 2024-04-20T22:15:38.889 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-20T22:15:38.889 INFO:teuthology.orchestra.run.smithi062.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi189 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp11670v2q:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:15:38.889 INFO:teuthology.orchestra.run.smithi062.stdout:Traceback (most recent call last): 2024-04-20T22:15:38.889 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-20T22:15:38.889 INFO:teuthology.orchestra.run.smithi062.stdout: return _run_code(code, main_globals, None, 2024-04-20T22:15:38.889 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-20T22:15:38.889 INFO:teuthology.orchestra.run.smithi062.stdout: exec(code, run_globals) 2024-04-20T22:15:38.890 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5618, in 2024-04-20T22:15:38.890 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5606, in main 2024-04-20T22:15:38.890 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 408, in _infer_config 2024-04-20T22:15:38.890 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 323, in _infer_fsid 2024-04-20T22:15:38.890 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 436, in _infer_image 2024-04-20T22:15:38.890 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 310, in _validate_fsid 2024-04-20T22:15:38.890 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 3359, in command_ceph_volume 2024-04-20T22:15:38.890 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-20T22:15:38.890 INFO:teuthology.orchestra.run.smithi062.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi189 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp11670v2q:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:15:38.890 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-20T22:15:38.890 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is degraded 2024-04-20T22:15:38.890 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T22:15:38.891 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T22:15:38.891 INFO:teuthology.orchestra.run.smithi062.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-20T22:15:38.891 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-20T22:15:38.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:38.891+0000 7fdb5b000700 1 -- 172.21.15.62:0/135092127 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fdb3806ecc0 msgr2=0x7fdb38071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:15:38.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:38.891+0000 7fdb5b000700 1 --2- 172.21.15.62:0/135092127 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fdb3806ecc0 0x7fdb38071170 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fdb40005950 tx=0x7fdb400058e0 comp rx=0 tx=0).stop 2024-04-20T22:15:38.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:38.891+0000 7fdb5b000700 1 -- 172.21.15.62:0/135092127 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdb540ff210 msgr2=0x7fdb5419c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:15:38.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:38.891+0000 7fdb5b000700 1 --2- 172.21.15.62:0/135092127 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdb540ff210 0x7fdb5419c450 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7fdb440092e0 tx=0x7fdb4400be20 comp rx=0 tx=0).stop 2024-04-20T22:15:38.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:38.891+0000 7fdb5b000700 1 -- 172.21.15.62:0/135092127 shutdown_connections 2024-04-20T22:15:38.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:38.891+0000 7fdb5b000700 1 --2- 172.21.15.62:0/135092127 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fdb3806ecc0 0x7fdb38071170 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:15:38.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:38.891+0000 7fdb5b000700 1 --2- 172.21.15.62:0/135092127 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdb540ff210 0x7fdb5419c450 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:15:38.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:38.891+0000 7fdb5b000700 1 --2- 172.21.15.62:0/135092127 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb540ffb20 0x7fdb5419c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:15:38.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:38.891+0000 7fdb5b000700 1 -- 172.21.15.62:0/135092127 >> 172.21.15.62:0/135092127 conn(0x7fdb540fac80 msgr2=0x7fdb54110680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:15:38.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:38.892+0000 7fdb5b000700 1 -- 172.21.15.62:0/135092127 shutdown_connections 2024-04-20T22:15:38.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:15:38.892+0000 7fdb5b000700 1 -- 172.21.15.62:0/135092127 wait complete. 2024-04-20T22:15:39.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:15:39 smithi189 ceph-mon[57022]: from='client.36154 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:15:39.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:15:39 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/135092127' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:15:39.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:15:39 smithi062 ceph-mon[76322]: from='client.36154 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:15:39.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:15:39 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/135092127' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:15:40.639 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:15:40 smithi189 ceph-mon[57022]: pgmap v1214: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:15:40.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:15:40 smithi062 ceph-mon[76322]: pgmap v1214: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:15:42.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:15:42 smithi189 ceph-mon[57022]: pgmap v1215: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:15:42.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:15:42 smithi062 ceph-mon[76322]: pgmap v1215: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:15:43.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:15:43 smithi189 ceph-mon[57022]: pgmap v1216: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:15:43.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:15:43 smithi062 ceph-mon[76322]: pgmap v1216: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:15:46.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:15:46 smithi062 ceph-mon[76322]: pgmap v1217: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:15:46.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:15:46 smithi189 ceph-mon[57022]: pgmap v1217: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:15:48.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:15:48 smithi189 ceph-mon[57022]: pgmap v1218: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:15:48.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:15:48 smithi062 ceph-mon[76322]: pgmap v1218: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:15:50.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:15:50 smithi189 ceph-mon[57022]: pgmap v1219: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:15:50.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:15:50 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:15:50.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:15:50 smithi062 ceph-mon[76322]: pgmap v1219: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:15:50.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:15:50 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:15:52.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:15:52 smithi189 ceph-mon[57022]: pgmap v1220: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:15:52.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:15:52 smithi062 ceph-mon[76322]: pgmap v1220: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:15:54.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:15:54 smithi189 ceph-mon[57022]: pgmap v1221: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:15:54.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:15:54 smithi062 ceph-mon[76322]: pgmap v1221: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:15:56.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:15:56 smithi189 ceph-mon[57022]: pgmap v1222: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:15:56.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:15:56 smithi062 ceph-mon[76322]: pgmap v1222: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:15:58.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:15:58 smithi062 ceph-mon[76322]: pgmap v1223: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:15:58.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:15:58 smithi189 ceph-mon[57022]: pgmap v1223: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:16:00.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:16:00 smithi189 ceph-mon[57022]: pgmap v1224: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:16:00.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:16:00 smithi062 ceph-mon[76322]: pgmap v1224: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:16:02.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:16:02 smithi189 ceph-mon[57022]: pgmap v1225: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:16:02.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:16:02 smithi062 ceph-mon[76322]: pgmap v1225: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:16:04.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:16:04 smithi189 ceph-mon[57022]: pgmap v1226: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:16:04.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:16:04 smithi062 ceph-mon[76322]: pgmap v1226: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:16:05.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:16:05 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:16:05.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:16:05 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:16:06.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:16:06 smithi189 ceph-mon[57022]: pgmap v1227: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:16:06.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:16:06 smithi062 ceph-mon[76322]: pgmap v1227: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:16:07.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:16:07 smithi189 ceph-mon[57022]: pgmap v1228: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:16:07.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:16:07 smithi062 ceph-mon[76322]: pgmap v1228: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:16:09.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.070+0000 7fccb0200700 1 -- 172.21.15.62:0/1509664141 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcca8069140 msgr2=0x7fcca8104eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:16:09.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.070+0000 7fccb0200700 1 --2- 172.21.15.62:0/1509664141 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcca8069140 0x7fcca8104eb0 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7fcc94009910 tx=0x7fcc94009c20 comp rx=0 tx=0).stop 2024-04-20T22:16:09.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.070+0000 7fccb0200700 1 -- 172.21.15.62:0/1509664141 shutdown_connections 2024-04-20T22:16:09.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.070+0000 7fccb0200700 1 --2- 172.21.15.62:0/1509664141 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcca8069140 0x7fcca8104eb0 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:16:09.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.070+0000 7fccb0200700 1 --2- 172.21.15.62:0/1509664141 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcca80687a0 0x7fcca8068b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:16:09.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.070+0000 7fccb0200700 1 -- 172.21.15.62:0/1509664141 >> 172.21.15.62:0/1509664141 conn(0x7fcca8075130 msgr2=0x7fcca8075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:16:09.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.070+0000 7fccb0200700 1 -- 172.21.15.62:0/1509664141 shutdown_connections 2024-04-20T22:16:09.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.070+0000 7fccb0200700 1 -- 172.21.15.62:0/1509664141 wait complete. 2024-04-20T22:16:09.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.071+0000 7fccb0200700 1 Processor -- start 2024-04-20T22:16:09.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.072+0000 7fccb0200700 1 -- start start 2024-04-20T22:16:09.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.072+0000 7fccb0200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcca80687a0 0x7fcca8195e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:16:09.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.072+0000 7fccb0200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcca8069140 0x7fcca8196370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:16:09.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.072+0000 7fccb0200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcca8196a50 con 0x7fcca8069140 2024-04-20T22:16:09.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.072+0000 7fccb0200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcca819a7e0 con 0x7fcca80687a0 2024-04-20T22:16:09.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.072+0000 7fccace00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcca8069140 0x7fcca8196370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:16:09.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.073+0000 7fccace00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcca8069140 0x7fcca8196370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52998/0 (socket says 172.21.15.62:52998) 2024-04-20T22:16:09.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.073+0000 7fccace00700 1 -- 172.21.15.62:0/2481164882 learned_addr learned my addr 172.21.15.62:0/2481164882 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:16:09.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.073+0000 7fccad800700 1 --2- 172.21.15.62:0/2481164882 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcca80687a0 0x7fcca8195e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:16:09.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.073+0000 7fccace00700 1 -- 172.21.15.62:0/2481164882 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcca80687a0 msgr2=0x7fcca8195e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:16:09.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.073+0000 7fccace00700 1 --2- 172.21.15.62:0/2481164882 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcca80687a0 0x7fcca8195e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:16:09.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.073+0000 7fccace00700 1 -- 172.21.15.62:0/2481164882 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc940095f0 con 0x7fcca8069140 2024-04-20T22:16:09.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.073+0000 7fccace00700 1 --2- 172.21.15.62:0/2481164882 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcca8069140 0x7fcca8196370 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7fcc9400b5c0 tx=0x7fcc9400be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:16:09.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.074+0000 7fcca6000700 1 -- 172.21.15.62:0/2481164882 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcc9401d070 con 0x7fcca8069140 2024-04-20T22:16:09.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.074+0000 7fcca6000700 1 -- 172.21.15.62:0/2481164882 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcc9400f460 con 0x7fcca8069140 2024-04-20T22:16:09.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.074+0000 7fccb0200700 1 -- 172.21.15.62:0/2481164882 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcca819aa60 con 0x7fcca8069140 2024-04-20T22:16:09.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.074+0000 7fcca6000700 1 -- 172.21.15.62:0/2481164882 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcc94017610 con 0x7fcca8069140 2024-04-20T22:16:09.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.074+0000 7fccb0200700 1 -- 172.21.15.62:0/2481164882 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcca819af50 con 0x7fcca8069140 2024-04-20T22:16:09.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.075+0000 7fccb0200700 1 -- 172.21.15.62:0/2481164882 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcca8108800 con 0x7fcca8069140 2024-04-20T22:16:09.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.076+0000 7fcca6000700 1 -- 172.21.15.62:0/2481164882 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fcc9400f5d0 con 0x7fcca8069140 2024-04-20T22:16:09.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.077+0000 7fcca6000700 1 --2- 172.21.15.62:0/2481164882 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fcc9006ecc0 0x7fcc90071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:16:09.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.077+0000 7fcca6000700 1 -- 172.21.15.62:0/2481164882 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fcc94090510 con 0x7fcca8069140 2024-04-20T22:16:09.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.077+0000 7fccad800700 1 --2- 172.21.15.62:0/2481164882 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fcc9006ecc0 0x7fcc90071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:16:09.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.078+0000 7fccad800700 1 --2- 172.21.15.62:0/2481164882 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fcc9006ecc0 0x7fcc90071170 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fcc9c005fd0 tx=0x7fcc9c005dd0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:16:09.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.082+0000 7fcca6000700 1 -- 172.21.15.62:0/2481164882 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fcc94059220 con 0x7fcca8069140 2024-04-20T22:16:09.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.344+0000 7fccb0200700 1 -- 172.21.15.62:0/2481164882 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fcca802cc80 con 0x7fcc9006ecc0 2024-04-20T22:16:09.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.348+0000 7fcca6000700 1 -- 172.21.15.62:0/2481164882 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fcca802cc80 con 0x7fcc9006ecc0 2024-04-20T22:16:09.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.351+0000 7fccb0200700 1 -- 172.21.15.62:0/2481164882 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fcc9006ecc0 msgr2=0x7fcc90071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:16:09.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.351+0000 7fccb0200700 1 --2- 172.21.15.62:0/2481164882 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fcc9006ecc0 0x7fcc90071170 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fcc9c005fd0 tx=0x7fcc9c005dd0 comp rx=0 tx=0).stop 2024-04-20T22:16:09.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.351+0000 7fccb0200700 1 -- 172.21.15.62:0/2481164882 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcca8069140 msgr2=0x7fcca8196370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:16:09.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.351+0000 7fccb0200700 1 --2- 172.21.15.62:0/2481164882 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcca8069140 0x7fcca8196370 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7fcc9400b5c0 tx=0x7fcc9400be00 comp rx=0 tx=0).stop 2024-04-20T22:16:09.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.351+0000 7fccb0200700 1 -- 172.21.15.62:0/2481164882 shutdown_connections 2024-04-20T22:16:09.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.351+0000 7fccb0200700 1 --2- 172.21.15.62:0/2481164882 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fcc9006ecc0 0x7fcc90071170 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:16:09.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.351+0000 7fccb0200700 1 --2- 172.21.15.62:0/2481164882 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcca80687a0 0x7fcca8195e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:16:09.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.351+0000 7fccb0200700 1 --2- 172.21.15.62:0/2481164882 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcca8069140 0x7fcca8196370 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:16:09.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.351+0000 7fccb0200700 1 -- 172.21.15.62:0/2481164882 >> 172.21.15.62:0/2481164882 conn(0x7fcca8075130 msgr2=0x7fcca80fe740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:16:09.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.351+0000 7fccb0200700 1 -- 172.21.15.62:0/2481164882 shutdown_connections 2024-04-20T22:16:09.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.351+0000 7fccb0200700 1 -- 172.21.15.62:0/2481164882 wait complete. 2024-04-20T22:16:09.369 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T22:16:09.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.524+0000 7eff0ea00700 1 -- 172.21.15.62:0/1847541061 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff08101230 msgr2=0x7eff08101600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:16:09.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.524+0000 7eff0ea00700 1 --2- 172.21.15.62:0/1847541061 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff08101230 0x7eff08101600 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7efefc009910 tx=0x7efefc009c20 comp rx=0 tx=0).stop 2024-04-20T22:16:09.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.524+0000 7eff0ea00700 1 -- 172.21.15.62:0/1847541061 shutdown_connections 2024-04-20T22:16:09.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.524+0000 7eff0ea00700 1 --2- 172.21.15.62:0/1847541061 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff08101b40 0x7eff0810a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:16:09.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.524+0000 7eff0ea00700 1 --2- 172.21.15.62:0/1847541061 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff08101230 0x7eff08101600 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:16:09.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.524+0000 7eff0ea00700 1 -- 172.21.15.62:0/1847541061 >> 172.21.15.62:0/1847541061 conn(0x7eff080faa90 msgr2=0x7eff080fcea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:16:09.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.524+0000 7eff0ea00700 1 -- 172.21.15.62:0/1847541061 shutdown_connections 2024-04-20T22:16:09.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.524+0000 7eff0ea00700 1 -- 172.21.15.62:0/1847541061 wait complete. 2024-04-20T22:16:09.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.526+0000 7eff0ea00700 1 Processor -- start 2024-04-20T22:16:09.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.526+0000 7eff0ea00700 1 -- start start 2024-04-20T22:16:09.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.527+0000 7eff0ea00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff08101230 0x7eff080fee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:16:09.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.527+0000 7eff0ea00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff08101b40 0x7eff080ff3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:16:09.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.527+0000 7eff0ea00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff080ff910 con 0x7eff08101b40 2024-04-20T22:16:09.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.527+0000 7eff0ea00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff080ffa50 con 0x7eff08101230 2024-04-20T22:16:09.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.527+0000 7eff0cc00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff08101b40 0x7eff080ff3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:16:09.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.527+0000 7eff0cc00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff08101b40 0x7eff080ff3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53010/0 (socket says 172.21.15.62:53010) 2024-04-20T22:16:09.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.527+0000 7eff0cc00700 1 -- 172.21.15.62:0/807929064 learned_addr learned my addr 172.21.15.62:0/807929064 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:16:09.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.527+0000 7eff0d600700 1 --2- 172.21.15.62:0/807929064 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff08101230 0x7eff080fee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:16:09.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.528+0000 7eff0cc00700 1 -- 172.21.15.62:0/807929064 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff08101230 msgr2=0x7eff080fee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:16:09.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.528+0000 7eff0cc00700 1 --2- 172.21.15.62:0/807929064 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff08101230 0x7eff080fee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:16:09.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.528+0000 7eff0cc00700 1 -- 172.21.15.62:0/807929064 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efefc0095f0 con 0x7eff08101b40 2024-04-20T22:16:09.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.528+0000 7eff0cc00700 1 --2- 172.21.15.62:0/807929064 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff08101b40 0x7eff080ff3d0 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7efef400c930 tx=0x7efef400ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:16:09.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.528+0000 7eff06000700 1 -- 172.21.15.62:0/807929064 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efef4007ad0 con 0x7eff08101b40 2024-04-20T22:16:09.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.528+0000 7eff06000700 1 -- 172.21.15.62:0/807929064 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efef4007c30 con 0x7eff08101b40 2024-04-20T22:16:09.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.528+0000 7eff0ea00700 1 -- 172.21.15.62:0/807929064 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff081a1df0 con 0x7eff08101b40 2024-04-20T22:16:09.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.528+0000 7eff06000700 1 -- 172.21.15.62:0/807929064 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efef4018610 con 0x7eff08101b40 2024-04-20T22:16:09.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.528+0000 7eff0ea00700 1 -- 172.21.15.62:0/807929064 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff081a2150 con 0x7eff08101b40 2024-04-20T22:16:09.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.530+0000 7eff0ea00700 1 -- 172.21.15.62:0/807929064 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff08107710 con 0x7eff08101b40 2024-04-20T22:16:09.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.531+0000 7eff06000700 1 -- 172.21.15.62:0/807929064 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7efef400f450 con 0x7eff08101b40 2024-04-20T22:16:09.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.532+0000 7eff06000700 1 --2- 172.21.15.62:0/807929064 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7efef006ecc0 0x7efef0071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:16:09.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.532+0000 7eff06000700 1 -- 172.21.15.62:0/807929064 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7efef408f0e0 con 0x7eff08101b40 2024-04-20T22:16:09.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.532+0000 7eff0d600700 1 --2- 172.21.15.62:0/807929064 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7efef006ecc0 0x7efef0071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:16:09.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.533+0000 7eff0d600700 1 --2- 172.21.15.62:0/807929064 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7efef006ecc0 0x7efef0071170 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7efefc006010 tx=0x7efefc01a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:16:09.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.537+0000 7eff06000700 1 -- 172.21.15.62:0/807929064 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7efef4057e90 con 0x7eff08101b40 2024-04-20T22:16:09.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.798+0000 7eff0ea00700 1 -- 172.21.15.62:0/807929064 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7eff08061190 con 0x7efef006ecc0 2024-04-20T22:16:09.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.801+0000 7eff06000700 1 -- 172.21.15.62:0/807929064 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7eff08061190 con 0x7efef006ecc0 2024-04-20T22:16:09.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.805+0000 7eff0ea00700 1 -- 172.21.15.62:0/807929064 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7efef006ecc0 msgr2=0x7efef0071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:16:09.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.805+0000 7eff0ea00700 1 --2- 172.21.15.62:0/807929064 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7efef006ecc0 0x7efef0071170 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7efefc006010 tx=0x7efefc01a040 comp rx=0 tx=0).stop 2024-04-20T22:16:09.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.805+0000 7eff0ea00700 1 -- 172.21.15.62:0/807929064 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff08101b40 msgr2=0x7eff080ff3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:16:09.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.805+0000 7eff0ea00700 1 --2- 172.21.15.62:0/807929064 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff08101b40 0x7eff080ff3d0 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7efef400c930 tx=0x7efef400ccf0 comp rx=0 tx=0).stop 2024-04-20T22:16:09.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.805+0000 7eff0ea00700 1 -- 172.21.15.62:0/807929064 shutdown_connections 2024-04-20T22:16:09.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.805+0000 7eff0ea00700 1 --2- 172.21.15.62:0/807929064 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7efef006ecc0 0x7efef0071170 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:16:09.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.805+0000 7eff0ea00700 1 --2- 172.21.15.62:0/807929064 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff08101230 0x7eff080fee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:16:09.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.805+0000 7eff0ea00700 1 --2- 172.21.15.62:0/807929064 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff08101b40 0x7eff080ff3d0 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:16:09.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.805+0000 7eff0ea00700 1 -- 172.21.15.62:0/807929064 >> 172.21.15.62:0/807929064 conn(0x7eff080faa90 msgr2=0x7eff08104850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:16:09.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.805+0000 7eff0ea00700 1 -- 172.21.15.62:0/807929064 shutdown_connections 2024-04-20T22:16:09.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.805+0000 7eff0ea00700 1 -- 172.21.15.62:0/807929064 wait complete. 2024-04-20T22:16:09.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.972+0000 7fbdbfc00700 1 -- 172.21.15.62:0/2042227790 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbdb80ff0d0 msgr2=0x7fbdb80ff4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:16:09.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.972+0000 7fbdbfc00700 1 --2- 172.21.15.62:0/2042227790 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbdb80ff0d0 0x7fbdb80ff4a0 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7fbdac009910 tx=0x7fbdac009c20 comp rx=0 tx=0).stop 2024-04-20T22:16:09.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.972+0000 7fbdbfc00700 1 -- 172.21.15.62:0/2042227790 shutdown_connections 2024-04-20T22:16:09.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.972+0000 7fbdbfc00700 1 --2- 172.21.15.62:0/2042227790 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbdb80ff9e0 0x7fbdb8103ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:16:09.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.972+0000 7fbdbfc00700 1 --2- 172.21.15.62:0/2042227790 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbdb80ff0d0 0x7fbdb80ff4a0 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:16:09.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.972+0000 7fbdbfc00700 1 -- 172.21.15.62:0/2042227790 >> 172.21.15.62:0/2042227790 conn(0x7fbdb80fab20 msgr2=0x7fbdb80fcf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:16:09.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.972+0000 7fbdbfc00700 1 -- 172.21.15.62:0/2042227790 shutdown_connections 2024-04-20T22:16:09.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.972+0000 7fbdbfc00700 1 -- 172.21.15.62:0/2042227790 wait complete. 2024-04-20T22:16:09.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.973+0000 7fbdbfc00700 1 Processor -- start 2024-04-20T22:16:09.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.973+0000 7fbdbfc00700 1 -- start start 2024-04-20T22:16:09.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.974+0000 7fbdbfc00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbdb80ff0d0 0x7fbdb819c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:16:09.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.974+0000 7fbdbfc00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbdb80ff9e0 0x7fbdb819c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:16:09.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.974+0000 7fbdbfc00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdb819cea0 con 0x7fbdb80ff9e0 2024-04-20T22:16:09.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.974+0000 7fbdbfc00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdb81a0c30 con 0x7fbdb80ff0d0 2024-04-20T22:16:09.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.974+0000 7fbdb7e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbdb80ff9e0 0x7fbdb819c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:16:09.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.974+0000 7fbdb7e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbdb80ff9e0 0x7fbdb819c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53028/0 (socket says 172.21.15.62:53028) 2024-04-20T22:16:09.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.974+0000 7fbdb7e00700 1 -- 172.21.15.62:0/2081692067 learned_addr learned my addr 172.21.15.62:0/2081692067 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:16:09.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.974+0000 7fbdbd200700 1 --2- 172.21.15.62:0/2081692067 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbdb80ff0d0 0x7fbdb819c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:16:09.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.975+0000 7fbdb7e00700 1 -- 172.21.15.62:0/2081692067 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbdb80ff0d0 msgr2=0x7fbdb819c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:16:09.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.975+0000 7fbdb7e00700 1 --2- 172.21.15.62:0/2081692067 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbdb80ff0d0 0x7fbdb819c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:16:09.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.975+0000 7fbdb7e00700 1 -- 172.21.15.62:0/2081692067 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdac0095f0 con 0x7fbdb80ff9e0 2024-04-20T22:16:09.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.975+0000 7fbdb7e00700 1 --2- 172.21.15.62:0/2081692067 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbdb80ff9e0 0x7fbdb819c7c0 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7fbda400d8d0 tx=0x7fbda400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:16:09.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.975+0000 7fbdb5600700 1 -- 172.21.15.62:0/2081692067 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbda4009940 con 0x7fbdb80ff9e0 2024-04-20T22:16:09.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.975+0000 7fbdb5600700 1 -- 172.21.15.62:0/2081692067 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbda4010460 con 0x7fbdb80ff9e0 2024-04-20T22:16:09.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.975+0000 7fbdb5600700 1 -- 172.21.15.62:0/2081692067 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbda400f5f0 con 0x7fbdb80ff9e0 2024-04-20T22:16:09.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.975+0000 7fbdbfc00700 1 -- 172.21.15.62:0/2081692067 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdb81a0f10 con 0x7fbdb80ff9e0 2024-04-20T22:16:09.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.975+0000 7fbdbfc00700 1 -- 172.21.15.62:0/2081692067 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdb81a1460 con 0x7fbdb80ff9e0 2024-04-20T22:16:09.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.977+0000 7fbdbfc00700 1 -- 172.21.15.62:0/2081692067 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdb81015f0 con 0x7fbdb80ff9e0 2024-04-20T22:16:09.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.978+0000 7fbdb5600700 1 -- 172.21.15.62:0/2081692067 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fbda40105d0 con 0x7fbdb80ff9e0 2024-04-20T22:16:09.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.979+0000 7fbdb5600700 1 --2- 172.21.15.62:0/2081692067 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fbda006ecc0 0x7fbda0071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:16:09.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.979+0000 7fbdb5600700 1 -- 172.21.15.62:0/2081692067 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fbda408ee10 con 0x7fbdb80ff9e0 2024-04-20T22:16:09.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.979+0000 7fbdbd200700 1 --2- 172.21.15.62:0/2081692067 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fbda006ecc0 0x7fbda0071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:16:09.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.980+0000 7fbdbd200700 1 --2- 172.21.15.62:0/2081692067 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fbda006ecc0 0x7fbda0071170 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fbdac006010 tx=0x7fbdac01a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:16:09.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:09.984+0000 7fbdb5600700 1 -- 172.21.15.62:0/2081692067 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fbda4057ba0 con 0x7fbdb80ff9e0 2024-04-20T22:16:10.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:10.228+0000 7fbdbfc00700 1 -- 172.21.15.62:0/2081692067 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fbdb81a17b0 con 0x7fbda006ecc0 2024-04-20T22:16:10.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:10.262+0000 7fbdb5600700 1 -- 172.21.15.62:0/2081692067 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fbdb81a17b0 con 0x7fbda006ecc0 2024-04-20T22:16:10.263 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T22:16:10.263 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (41m) 2m ago 55m 14.6M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T22:16:10.263 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (56m) 2m ago 56m 11.6M - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T22:16:10.263 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (54m) 2m ago 54m 11.2M - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T22:16:10.263 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (38m) 2m ago 56m 6660k - 19.0.0-3133-ga8c84f59 76bc6b91b726 2825d5ba11c6 2024-04-20T22:16:10.263 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (37m) 2m ago 54m 6665k - 19.0.0-3133-ga8c84f59 76bc6b91b726 20f746fa1bba 2024-04-20T22:16:10.263 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (41m) 2m ago 55m 75.5M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T22:16:10.263 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (33m) 2m ago 48m 22.9M - 19.0.0-3133-ga8c84f59 76bc6b91b726 de80bc90a683 2024-04-20T22:16:10.264 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (33m) 2m ago 48m 23.0M - 19.0.0-3133-ga8c84f59 76bc6b91b726 cd793b7ee98e 2024-04-20T22:16:10.264 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (18m) 2m ago 48m 18.5M - 19.0.0-3133-ga8c84f59 76bc6b91b726 0e7b0bc4d1ab 2024-04-20T22:16:10.264 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (2m) 2m ago 48m 11.4M - 19.0.0-3133-ga8c84f59 76bc6b91b726 0af732ac9e06 2024-04-20T22:16:10.264 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (44m) 2m ago 57m 517M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T22:16:10.264 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (43m) 2m ago 54m 460M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T22:16:10.264 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (39m) 2m ago 57m 86.3M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 019eb1479485 2024-04-20T22:16:10.264 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (39m) 2m ago 54m 58.0M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 55aac560aab6 2024-04-20T22:16:10.264 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (43m) 2m ago 56m 8766k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T22:16:10.264 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (43m) 2m ago 54m 8711k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T22:16:10.264 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (37m) 2m ago 53m 91.3M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 ee2bb901f26e 2024-04-20T22:16:10.264 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (36m) 2m ago 52m 94.9M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 9075152d4162 2024-04-20T22:16:10.264 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (36m) 2m ago 52m 78.5M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 2d4f3e38dc0e 2024-04-20T22:16:10.264 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (35m) 2m ago 51m 94.1M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 8f97b17ae0eb 2024-04-20T22:16:10.264 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (35m) 2m ago 51m 96.1M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 653d8079967d 2024-04-20T22:16:10.264 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (34m) 2m ago 50m 90.9M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 596dded14c12 2024-04-20T22:16:10.265 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (42m) 2m ago 55m 90.4M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T22:16:10.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:10.266+0000 7fbdbfc00700 1 -- 172.21.15.62:0/2081692067 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fbda006ecc0 msgr2=0x7fbda0071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:16:10.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:10.266+0000 7fbdbfc00700 1 --2- 172.21.15.62:0/2081692067 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fbda006ecc0 0x7fbda0071170 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fbdac006010 tx=0x7fbdac01a040 comp rx=0 tx=0).stop 2024-04-20T22:16:10.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:10.266+0000 7fbdbfc00700 1 -- 172.21.15.62:0/2081692067 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbdb80ff9e0 msgr2=0x7fbdb819c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:16:10.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:10.266+0000 7fbdbfc00700 1 --2- 172.21.15.62:0/2081692067 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbdb80ff9e0 0x7fbdb819c7c0 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7fbda400d8d0 tx=0x7fbda400dc90 comp rx=0 tx=0).stop 2024-04-20T22:16:10.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:10.266+0000 7fbdbfc00700 1 -- 172.21.15.62:0/2081692067 shutdown_connections 2024-04-20T22:16:10.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:10.266+0000 7fbdbfc00700 1 --2- 172.21.15.62:0/2081692067 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fbda006ecc0 0x7fbda0071170 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:16:10.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:10.266+0000 7fbdbfc00700 1 --2- 172.21.15.62:0/2081692067 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbdb80ff0d0 0x7fbdb819c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:16:10.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:10.266+0000 7fbdbfc00700 1 --2- 172.21.15.62:0/2081692067 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbdb80ff9e0 0x7fbdb819c7c0 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:16:10.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:10.266+0000 7fbdbfc00700 1 -- 172.21.15.62:0/2081692067 >> 172.21.15.62:0/2081692067 conn(0x7fbdb80fab20 msgr2=0x7fbdb8110540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:16:10.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:10.266+0000 7fbdbfc00700 1 -- 172.21.15.62:0/2081692067 shutdown_connections 2024-04-20T22:16:10.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:10.267+0000 7fbdbfc00700 1 -- 172.21.15.62:0/2081692067 wait complete. 2024-04-20T22:16:10.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:10.436+0000 7fe64ca00700 1 -- 172.21.15.62:0/3147812265 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe648101b20 msgr2=0x7fe64810a020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:16:10.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:10.436+0000 7fe64ca00700 1 --2- 172.21.15.62:0/3147812265 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe648101b20 0x7fe64810a020 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7fe634009910 tx=0x7fe634009c20 comp rx=0 tx=0).stop 2024-04-20T22:16:10.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:10.436+0000 7fe64ca00700 1 -- 172.21.15.62:0/3147812265 shutdown_connections 2024-04-20T22:16:10.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:10.436+0000 7fe64ca00700 1 --2- 172.21.15.62:0/3147812265 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe648101b20 0x7fe64810a020 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:16:10.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:10.436+0000 7fe64ca00700 1 --2- 172.21.15.62:0/3147812265 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe648101210 0x7fe6481015e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:16:10.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:10.436+0000 7fe64ca00700 1 -- 172.21.15.62:0/3147812265 >> 172.21.15.62:0/3147812265 conn(0x7fe6480faa30 msgr2=0x7fe6480fce40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:16:10.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:10.436+0000 7fe64ca00700 1 -- 172.21.15.62:0/3147812265 shutdown_connections 2024-04-20T22:16:10.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:10.436+0000 7fe64ca00700 1 -- 172.21.15.62:0/3147812265 wait complete. 2024-04-20T22:16:10.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:10.437+0000 7fe64ca00700 1 Processor -- start 2024-04-20T22:16:10.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:10.437+0000 7fe64ca00700 1 -- start start 2024-04-20T22:16:10.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:10.438+0000 7fe64ca00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe648101210 0x7fe648197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:16:10.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:10.438+0000 7fe64ca00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe648101b20 0x7fe648198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:16:10.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:10.438+0000 7fe64ca00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe648198b00 con 0x7fe648101210 2024-04-20T22:16:10.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:10.438+0000 7fe64ca00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe64819c890 con 0x7fe648101b20 2024-04-20T22:16:10.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:10.438+0000 7fe646800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe648101210 0x7fe648197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:16:10.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:10.438+0000 7fe646800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe648101210 0x7fe648197ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53032/0 (socket says 172.21.15.62:53032) 2024-04-20T22:16:10.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:10.438+0000 7fe646800700 1 -- 172.21.15.62:0/3407675323 learned_addr learned my addr 172.21.15.62:0/3407675323 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:16:10.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:10.439+0000 7fe645e00700 1 --2- 172.21.15.62:0/3407675323 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe648101b20 0x7fe648198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:16:10.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:10.439+0000 7fe646800700 1 -- 172.21.15.62:0/3407675323 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe648101b20 msgr2=0x7fe648198420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:16:10.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:10.439+0000 7fe646800700 1 --2- 172.21.15.62:0/3407675323 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe648101b20 0x7fe648198420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:16:10.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:10.439+0000 7fe646800700 1 -- 172.21.15.62:0/3407675323 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6340095f0 con 0x7fe648101210 2024-04-20T22:16:10.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:10.439+0000 7fe646800700 1 --2- 172.21.15.62:0/3407675323 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe648101210 0x7fe648197ee0 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7fe63800b4b0 tx=0x7fe63800b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:16:10.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:10.439+0000 7fe63f400700 1 -- 172.21.15.62:0/3407675323 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe638011800 con 0x7fe648101210 2024-04-20T22:16:10.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:10.439+0000 7fe63f400700 1 -- 172.21.15.62:0/3407675323 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe638011e20 con 0x7fe648101210 2024-04-20T22:16:10.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:10.439+0000 7fe63f400700 1 -- 172.21.15.62:0/3407675323 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe63800f2e0 con 0x7fe648101210 2024-04-20T22:16:10.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:10.439+0000 7fe64ca00700 1 -- 172.21.15.62:0/3407675323 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe64819cb70 con 0x7fe648101210 2024-04-20T22:16:10.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:10.440+0000 7fe64ca00700 1 -- 172.21.15.62:0/3407675323 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe64819d0c0 con 0x7fe648101210 2024-04-20T22:16:10.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:10.441+0000 7fe64ca00700 1 -- 172.21.15.62:0/3407675323 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe648109af0 con 0x7fe648101210 2024-04-20T22:16:10.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:10.442+0000 7fe63f400700 1 -- 172.21.15.62:0/3407675323 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe638010380 con 0x7fe648101210 2024-04-20T22:16:10.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:10.443+0000 7fe63f400700 1 --2- 172.21.15.62:0/3407675323 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe62c06ecc0 0x7fe62c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:16:10.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:10.443+0000 7fe63f400700 1 -- 172.21.15.62:0/3407675323 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fe63808eaa0 con 0x7fe648101210 2024-04-20T22:16:10.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:10.443+0000 7fe645e00700 1 --2- 172.21.15.62:0/3407675323 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe62c06ecc0 0x7fe62c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:16:10.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:10.444+0000 7fe645e00700 1 --2- 172.21.15.62:0/3407675323 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe62c06ecc0 0x7fe62c071170 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fe648199500 tx=0x7fe63401a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:16:10.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:10.448+0000 7fe63f400700 1 -- 172.21.15.62:0/3407675323 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fe6380577b0 con 0x7fe648101210 2024-04-20T22:16:10.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:16:10 smithi062 ceph-mon[76322]: pgmap v1229: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:16:10.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:16:10 smithi062 ceph-mon[76322]: from='client.36162 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:16:10.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:16:10 smithi189 ceph-mon[57022]: pgmap v1229: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:16:10.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:16:10 smithi189 ceph-mon[57022]: from='client.36162 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:16:10.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:10.785+0000 7fe64ca00700 1 -- 172.21.15.62:0/3407675323 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fe64819d340 con 0x7fe648101210 2024-04-20T22:16:10.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:10.787+0000 7fe63f400700 1 -- 172.21.15.62:0/3407675323 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7fe63805ae70 con 0x7fe648101210 2024-04-20T22:16:10.787 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:16:10.787 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T22:16:10.787 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:16:10.787 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:16:10.787 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T22:16:10.787 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:16:10.787 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:16:10.788 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T22:16:10.788 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 6 2024-04-20T22:16:10.788 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:16:10.788 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T22:16:10.788 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 4 2024-04-20T22:16:10.788 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:16:10.788 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T22:16:10.788 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 14 2024-04-20T22:16:10.788 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T22:16:10.788 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:16:10.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:10.790+0000 7fe64ca00700 1 -- 172.21.15.62:0/3407675323 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe62c06ecc0 msgr2=0x7fe62c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:16:10.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:10.790+0000 7fe64ca00700 1 --2- 172.21.15.62:0/3407675323 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe62c06ecc0 0x7fe62c071170 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fe648199500 tx=0x7fe63401a040 comp rx=0 tx=0).stop 2024-04-20T22:16:10.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:10.790+0000 7fe64ca00700 1 -- 172.21.15.62:0/3407675323 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe648101210 msgr2=0x7fe648197ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:16:10.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:10.790+0000 7fe64ca00700 1 --2- 172.21.15.62:0/3407675323 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe648101210 0x7fe648197ee0 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7fe63800b4b0 tx=0x7fe63800b870 comp rx=0 tx=0).stop 2024-04-20T22:16:10.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:10.791+0000 7fe64ca00700 1 -- 172.21.15.62:0/3407675323 shutdown_connections 2024-04-20T22:16:10.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:10.791+0000 7fe64ca00700 1 --2- 172.21.15.62:0/3407675323 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe62c06ecc0 0x7fe62c071170 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:16:10.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:10.791+0000 7fe64ca00700 1 --2- 172.21.15.62:0/3407675323 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe648101210 0x7fe648197ee0 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:16:10.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:10.791+0000 7fe64ca00700 1 --2- 172.21.15.62:0/3407675323 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe648101b20 0x7fe648198420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:16:10.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:10.791+0000 7fe64ca00700 1 -- 172.21.15.62:0/3407675323 >> 172.21.15.62:0/3407675323 conn(0x7fe6480faa30 msgr2=0x7fe6480ff610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:16:10.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:10.791+0000 7fe64ca00700 1 -- 172.21.15.62:0/3407675323 shutdown_connections 2024-04-20T22:16:10.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:10.791+0000 7fe64ca00700 1 -- 172.21.15.62:0/3407675323 wait complete. 2024-04-20T22:16:10.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:10.957+0000 7f044ee00700 1 -- 172.21.15.62:0/2606220910 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0448073c20 msgr2=0x7f044810c3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:16:10.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:10.957+0000 7f044ee00700 1 --2- 172.21.15.62:0/2606220910 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0448073c20 0x7f044810c3f0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f0434009910 tx=0x7f0434009c20 comp rx=0 tx=0).stop 2024-04-20T22:16:10.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:10.957+0000 7f044ee00700 1 -- 172.21.15.62:0/2606220910 shutdown_connections 2024-04-20T22:16:10.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:10.957+0000 7f044ee00700 1 --2- 172.21.15.62:0/2606220910 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0448073c20 0x7f044810c3f0 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:16:10.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:10.957+0000 7f044ee00700 1 --2- 172.21.15.62:0/2606220910 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0448073310 0x7f04480736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:16:10.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:10.957+0000 7f044ee00700 1 -- 172.21.15.62:0/2606220910 >> 172.21.15.62:0/2606220910 conn(0x7f04480fbbd0 msgr2=0x7f04480fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:16:10.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:10.957+0000 7f044ee00700 1 -- 172.21.15.62:0/2606220910 shutdown_connections 2024-04-20T22:16:10.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:10.957+0000 7f044ee00700 1 -- 172.21.15.62:0/2606220910 wait complete. 2024-04-20T22:16:10.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:10.958+0000 7f044ee00700 1 Processor -- start 2024-04-20T22:16:10.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:10.959+0000 7f044ee00700 1 -- start start 2024-04-20T22:16:10.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:10.959+0000 7f044ee00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0448073310 0x7f0448197e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:16:10.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:10.959+0000 7f044ee00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0448073c20 0x7f04481983b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:16:10.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:10.959+0000 7f044ee00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0448198a90 con 0x7f0448073c20 2024-04-20T22:16:10.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:10.959+0000 7f044ee00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f044819c820 con 0x7f0448073310 2024-04-20T22:16:10.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:10.959+0000 7f044d000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0448073c20 0x7f04481983b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:16:10.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:10.959+0000 7f044d000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0448073c20 0x7f04481983b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53054/0 (socket says 172.21.15.62:53054) 2024-04-20T22:16:10.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:10.959+0000 7f044d000700 1 -- 172.21.15.62:0/788427409 learned_addr learned my addr 172.21.15.62:0/788427409 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:16:10.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:10.960+0000 7f044da00700 1 --2- 172.21.15.62:0/788427409 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0448073310 0x7f0448197e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:16:10.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:10.960+0000 7f044d000700 1 -- 172.21.15.62:0/788427409 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0448073310 msgr2=0x7f0448197e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:16:10.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:10.960+0000 7f044d000700 1 --2- 172.21.15.62:0/788427409 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0448073310 0x7f0448197e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:16:10.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:10.960+0000 7f044d000700 1 -- 172.21.15.62:0/788427409 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04340095f0 con 0x7f0448073c20 2024-04-20T22:16:10.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:10.960+0000 7f044d000700 1 --2- 172.21.15.62:0/788427409 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0448073c20 0x7f04481983b0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f0434006010 tx=0x7f043400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:16:10.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:10.961+0000 7f0446000700 1 -- 172.21.15.62:0/788427409 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f043401d070 con 0x7f0448073c20 2024-04-20T22:16:10.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:10.961+0000 7f0446000700 1 -- 172.21.15.62:0/788427409 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f043400f460 con 0x7f0448073c20 2024-04-20T22:16:10.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:10.961+0000 7f0446000700 1 -- 172.21.15.62:0/788427409 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0434017630 con 0x7f0448073c20 2024-04-20T22:16:10.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:10.961+0000 7f044ee00700 1 -- 172.21.15.62:0/788427409 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f044819cb00 con 0x7f0448073c20 2024-04-20T22:16:10.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:10.961+0000 7f044ee00700 1 -- 172.21.15.62:0/788427409 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f044819d050 con 0x7f0448073c20 2024-04-20T22:16:10.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:10.962+0000 7f044ee00700 1 -- 172.21.15.62:0/788427409 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0448109af0 con 0x7f0448073c20 2024-04-20T22:16:10.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:10.963+0000 7f0446000700 1 -- 172.21.15.62:0/788427409 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f043400f5d0 con 0x7f0448073c20 2024-04-20T22:16:10.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:10.964+0000 7f0446000700 1 --2- 172.21.15.62:0/788427409 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f043006ecc0 0x7f0430071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:16:10.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:10.964+0000 7f0446000700 1 -- 172.21.15.62:0/788427409 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f04340902c0 con 0x7f0448073c20 2024-04-20T22:16:10.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:10.964+0000 7f044da00700 1 --2- 172.21.15.62:0/788427409 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f043006ecc0 0x7f0430071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:16:10.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:10.965+0000 7f044da00700 1 --2- 172.21.15.62:0/788427409 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f043006ecc0 0x7f0430071170 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f043c005950 tx=0x7f043c0058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:16:10.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:10.969+0000 7f0446000700 1 -- 172.21.15.62:0/788427409 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f0434059050 con 0x7f0448073c20 2024-04-20T22:16:11.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:11.251+0000 7f044ee00700 1 -- 172.21.15.62:0/788427409 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f0448062380 con 0x7f0448073c20 2024-04-20T22:16:11.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:11.252+0000 7f0446000700 1 -- 172.21.15.62:0/788427409 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 29 v29) v1 ==== 76+0+1887 (secure 0 0 0) 0x7f043405c710 con 0x7f0448073c20 2024-04-20T22:16:11.254 INFO:teuthology.orchestra.run.smithi062.stdout:e29 2024-04-20T22:16:11.254 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T22:16:11.254 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:16:11.254 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T22:16:11.254 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:16:11.254 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T22:16:11.254 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T22:16:11.254 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 20 2024-04-20T22:16:11.254 INFO:teuthology.orchestra.run.smithi062.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-20T22:16:11.254 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T22:16:11.254 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T21:42:01.559061+0000 2024-04-20T22:16:11.255 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T22:16:11.255 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T22:16:11.255 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T22:16:11.255 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T22:16:11.255 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T22:16:11.255 INFO:teuthology.orchestra.run.smithi062.stdout:max_xattr_size 65536 2024-04-20T22:16:11.255 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T22:16:11.255 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T22:16:11.255 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 81 2024-04-20T22:16:11.255 INFO:teuthology.orchestra.run.smithi062.stdout:snap_rstat 1 2024-04-20T22:16:11.255 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:16:11.255 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 2 2024-04-20T22:16:11.255 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1 2024-04-20T22:16:11.255 INFO:teuthology.orchestra.run.smithi062.stdout:up {} 2024-04-20T22:16:11.255 INFO:teuthology.orchestra.run.smithi062.stdout:failed 0,1 2024-04-20T22:16:11.256 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T22:16:11.256 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T22:16:11.256 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T22:16:11.256 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T22:16:11.256 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data enabled 2024-04-20T22:16:11.256 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T22:16:11.256 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T22:16:11.256 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 1 2024-04-20T22:16:11.256 INFO:teuthology.orchestra.run.smithi062.stdout:qdb_cluster leader: 0 members: 2024-04-20T22:16:11.256 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:16:11.256 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:16:11.256 INFO:teuthology.orchestra.run.smithi062.stdout:Standby daemons: 2024-04-20T22:16:11.256 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:16:11.256 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ryqblb{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6826/4113916205,v1:172.21.15.62:6827/4113916205] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:16:11.256 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ozizwi{-1:34476} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6828/3801975165,v1:172.21.15.62:6829/3801975165] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:16:11.256 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.dvimer{-1:44727} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6824/3576699232,v1:172.21.15.189:6825/3576699232] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:16:11.257 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.xbgspw{-1:45075} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6826/212905792,v1:172.21.15.189:6827/212905792] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:16:11.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:11.255+0000 7f044ee00700 1 -- 172.21.15.62:0/788427409 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f043006ecc0 msgr2=0x7f0430071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:16:11.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:11.255+0000 7f044ee00700 1 --2- 172.21.15.62:0/788427409 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f043006ecc0 0x7f0430071170 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f043c005950 tx=0x7f043c0058e0 comp rx=0 tx=0).stop 2024-04-20T22:16:11.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:11.255+0000 7f044ee00700 1 -- 172.21.15.62:0/788427409 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0448073c20 msgr2=0x7f04481983b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:16:11.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:11.255+0000 7f044ee00700 1 --2- 172.21.15.62:0/788427409 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0448073c20 0x7f04481983b0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f0434006010 tx=0x7f043400be20 comp rx=0 tx=0).stop 2024-04-20T22:16:11.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:11.255+0000 7f044ee00700 1 -- 172.21.15.62:0/788427409 shutdown_connections 2024-04-20T22:16:11.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:11.255+0000 7f044ee00700 1 --2- 172.21.15.62:0/788427409 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f043006ecc0 0x7f0430071170 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:16:11.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:11.255+0000 7f044ee00700 1 --2- 172.21.15.62:0/788427409 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0448073310 0x7f0448197e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:16:11.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:11.255+0000 7f044ee00700 1 --2- 172.21.15.62:0/788427409 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0448073c20 0x7f04481983b0 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:16:11.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:11.255+0000 7f044ee00700 1 -- 172.21.15.62:0/788427409 >> 172.21.15.62:0/788427409 conn(0x7f04480fbbd0 msgr2=0x7f0448106c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:16:11.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:11.256+0000 7f044ee00700 1 -- 172.21.15.62:0/788427409 shutdown_connections 2024-04-20T22:16:11.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:11.256+0000 7f044ee00700 1 -- 172.21.15.62:0/788427409 wait complete. 2024-04-20T22:16:11.258 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 29 2024-04-20T22:16:11.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:11.418+0000 7f6bb4400700 1 -- 172.21.15.62:0/2978877955 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6bac0739c0 msgr2=0x7f6bac10c5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:16:11.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:11.418+0000 7f6bb4400700 1 --2- 172.21.15.62:0/2978877955 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6bac0739c0 0x7f6bac10c5c0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f6b98009940 tx=0x7f6b98009c50 comp rx=0 tx=0).stop 2024-04-20T22:16:11.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:11.418+0000 7f6bb4400700 1 -- 172.21.15.62:0/2978877955 shutdown_connections 2024-04-20T22:16:11.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:11.418+0000 7f6bb4400700 1 --2- 172.21.15.62:0/2978877955 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6bac0739c0 0x7f6bac10c5c0 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:16:11.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:11.418+0000 7f6bb4400700 1 --2- 172.21.15.62:0/2978877955 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6bac0730b0 0x7f6bac073480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:16:11.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:11.418+0000 7f6bb4400700 1 -- 172.21.15.62:0/2978877955 >> 172.21.15.62:0/2978877955 conn(0x7f6bac0fbda0 msgr2=0x7f6bac0fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:16:11.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:11.418+0000 7f6bb4400700 1 -- 172.21.15.62:0/2978877955 shutdown_connections 2024-04-20T22:16:11.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:11.418+0000 7f6bb4400700 1 -- 172.21.15.62:0/2978877955 wait complete. 2024-04-20T22:16:11.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:11.420+0000 7f6bb4400700 1 Processor -- start 2024-04-20T22:16:11.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:11.420+0000 7f6bb4400700 1 -- start start 2024-04-20T22:16:11.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:11.420+0000 7f6bb4400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6bac0730b0 0x7f6bac100220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:16:11.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:11.420+0000 7f6bb4400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6bac0739c0 0x7f6bac100760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:16:11.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:11.420+0000 7f6bb4400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bac1043b0 con 0x7f6bac0730b0 2024-04-20T22:16:11.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:11.420+0000 7f6bb4400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bac100ca0 con 0x7f6bac0739c0 2024-04-20T22:16:11.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:11.420+0000 7f6bb1a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6bac0730b0 0x7f6bac100220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:16:11.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:11.420+0000 7f6bb1a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6bac0730b0 0x7f6bac100220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56918/0 (socket says 172.21.15.62:56918) 2024-04-20T22:16:11.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:11.420+0000 7f6bb1a00700 1 -- 172.21.15.62:0/4080633021 learned_addr learned my addr 172.21.15.62:0/4080633021 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:16:11.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:11.420+0000 7f6bb1a00700 1 -- 172.21.15.62:0/4080633021 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6bac0739c0 msgr2=0x7f6bac100760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:16:11.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:11.421+0000 7f6bb1000700 1 --2- 172.21.15.62:0/4080633021 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6bac0739c0 0x7f6bac100760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:16:11.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:11.421+0000 7f6bb1a00700 1 --2- 172.21.15.62:0/4080633021 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6bac0739c0 0x7f6bac100760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:16:11.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:11.421+0000 7f6bb1a00700 1 -- 172.21.15.62:0/4080633021 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b980095f0 con 0x7f6bac0730b0 2024-04-20T22:16:11.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:11.421+0000 7f6bb1000700 1 --2- 172.21.15.62:0/4080633021 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6bac0739c0 0x7f6bac100760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T22:16:11.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:11.421+0000 7f6bb1a00700 1 --2- 172.21.15.62:0/4080633021 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6bac0730b0 0x7f6bac100220 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f6ba000d8d0 tx=0x7f6ba000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:16:11.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:11.421+0000 7f6baa000700 1 -- 172.21.15.62:0/4080633021 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6ba0009940 con 0x7f6bac0730b0 2024-04-20T22:16:11.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:11.421+0000 7f6baa000700 1 -- 172.21.15.62:0/4080633021 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6ba0010460 con 0x7f6bac0730b0 2024-04-20T22:16:11.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:11.421+0000 7f6bb4400700 1 -- 172.21.15.62:0/4080633021 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bac100f80 con 0x7f6bac0730b0 2024-04-20T22:16:11.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:11.421+0000 7f6baa000700 1 -- 172.21.15.62:0/4080633021 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6ba000f5f0 con 0x7f6bac0730b0 2024-04-20T22:16:11.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:11.421+0000 7f6bb4400700 1 -- 172.21.15.62:0/4080633021 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bac1a6a60 con 0x7f6bac0730b0 2024-04-20T22:16:11.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:11.423+0000 7f6bb4400700 1 -- 172.21.15.62:0/4080633021 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bac109cc0 con 0x7f6bac0730b0 2024-04-20T22:16:11.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:11.425+0000 7f6baa000700 1 -- 172.21.15.62:0/4080633021 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f6ba00105d0 con 0x7f6bac0730b0 2024-04-20T22:16:11.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:11.425+0000 7f6baa000700 1 --2- 172.21.15.62:0/4080633021 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6b9406ecc0 0x7f6b94071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:16:11.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:11.426+0000 7f6baa000700 1 -- 172.21.15.62:0/4080633021 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f6ba008ede0 con 0x7f6bac0730b0 2024-04-20T22:16:11.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:11.426+0000 7f6bb1000700 1 --2- 172.21.15.62:0/4080633021 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6b9406ecc0 0x7f6b94071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:16:11.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:11.426+0000 7f6bb1000700 1 --2- 172.21.15.62:0/4080633021 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6b9406ecc0 0x7f6b94071170 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f6bac101a10 tx=0x7f6b98005fd0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:16:11.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:11.429+0000 7f6baa000700 1 -- 172.21.15.62:0/4080633021 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f6ba0057b70 con 0x7f6bac0730b0 2024-04-20T22:16:11.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:16:11 smithi062 ceph-mon[76322]: from='client.36166 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:16:11.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:16:11 smithi062 ceph-mon[76322]: from='client.36170 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:16:11.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:16:11 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/3407675323' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:16:11.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:16:11 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/788427409' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:16:11.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:11.690+0000 7f6bb4400700 1 -- 172.21.15.62:0/4080633021 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6bac061190 con 0x7f6b9406ecc0 2024-04-20T22:16:11.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:11.692+0000 7f6baa000700 1 -- 172.21.15.62:0/4080633021 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f6bac061190 con 0x7f6b9406ecc0 2024-04-20T22:16:11.691 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:16:11.691 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T22:16:11.692 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T22:16:11.692 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-20T22:16:11.692 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T22:16:11.692 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr", 2024-04-20T22:16:11.692 INFO:teuthology.orchestra.run.smithi062.stdout: "osd", 2024-04-20T22:16:11.692 INFO:teuthology.orchestra.run.smithi062.stdout: "mds", 2024-04-20T22:16:11.692 INFO:teuthology.orchestra.run.smithi062.stdout: "crash", 2024-04-20T22:16:11.692 INFO:teuthology.orchestra.run.smithi062.stdout: "mon" 2024-04-20T22:16:11.692 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T22:16:11.692 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "16/23 daemons upgraded", 2024-04-20T22:16:11.692 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading mds daemons", 2024-04-20T22:16:11.692 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T22:16:11.692 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:16:11.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:11.695+0000 7f6bb4400700 1 -- 172.21.15.62:0/4080633021 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6b9406ecc0 msgr2=0x7f6b94071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:16:11.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:11.695+0000 7f6bb4400700 1 --2- 172.21.15.62:0/4080633021 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6b9406ecc0 0x7f6b94071170 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f6bac101a10 tx=0x7f6b98005fd0 comp rx=0 tx=0).stop 2024-04-20T22:16:11.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:11.695+0000 7f6bb4400700 1 -- 172.21.15.62:0/4080633021 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6bac0730b0 msgr2=0x7f6bac100220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:16:11.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:11.695+0000 7f6bb4400700 1 --2- 172.21.15.62:0/4080633021 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6bac0730b0 0x7f6bac100220 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f6ba000d8d0 tx=0x7f6ba000dc90 comp rx=0 tx=0).stop 2024-04-20T22:16:11.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:11.696+0000 7f6bb4400700 1 -- 172.21.15.62:0/4080633021 shutdown_connections 2024-04-20T22:16:11.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:11.696+0000 7f6bb4400700 1 --2- 172.21.15.62:0/4080633021 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6b9406ecc0 0x7f6b94071170 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:16:11.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:11.696+0000 7f6bb4400700 1 --2- 172.21.15.62:0/4080633021 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6bac0730b0 0x7f6bac100220 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:16:11.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:11.696+0000 7f6bb4400700 1 --2- 172.21.15.62:0/4080633021 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6bac0739c0 0x7f6bac100760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:16:11.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:11.696+0000 7f6bb4400700 1 -- 172.21.15.62:0/4080633021 >> 172.21.15.62:0/4080633021 conn(0x7f6bac0fbda0 msgr2=0x7f6bac106e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:16:11.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:11.696+0000 7f6bb4400700 1 -- 172.21.15.62:0/4080633021 shutdown_connections 2024-04-20T22:16:11.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:11.696+0000 7f6bb4400700 1 -- 172.21.15.62:0/4080633021 wait complete. 2024-04-20T22:16:11.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:16:11 smithi189 ceph-mon[57022]: from='client.36166 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:16:11.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:16:11 smithi189 ceph-mon[57022]: from='client.36170 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:16:11.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:16:11 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/3407675323' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:16:11.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:16:11 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/788427409' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:16:11.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:11.863+0000 7f4f12800700 1 -- 172.21.15.62:0/175832757 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f0c106b80 msgr2=0x7f4f0c106ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:16:11.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:11.863+0000 7f4f12800700 1 --2- 172.21.15.62:0/175832757 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f0c106b80 0x7f4f0c106ff0 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f4ef4009910 tx=0x7f4ef4009c20 comp rx=0 tx=0).stop 2024-04-20T22:16:11.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:11.863+0000 7f4f12800700 1 -- 172.21.15.62:0/175832757 shutdown_connections 2024-04-20T22:16:11.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:11.863+0000 7f4f12800700 1 --2- 172.21.15.62:0/175832757 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f0c106b80 0x7f4f0c106ff0 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:16:11.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:11.863+0000 7f4f12800700 1 --2- 172.21.15.62:0/175832757 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f0c10c8f0 0x7f4f0c10ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:16:11.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:11.863+0000 7f4f12800700 1 -- 172.21.15.62:0/175832757 >> 172.21.15.62:0/175832757 conn(0x7f4f0c076060 msgr2=0x7f4f0c078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:16:11.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:11.863+0000 7f4f12800700 1 -- 172.21.15.62:0/175832757 shutdown_connections 2024-04-20T22:16:11.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:11.863+0000 7f4f12800700 1 -- 172.21.15.62:0/175832757 wait complete. 2024-04-20T22:16:11.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:11.864+0000 7f4f12800700 1 Processor -- start 2024-04-20T22:16:11.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:11.864+0000 7f4f12800700 1 -- start start 2024-04-20T22:16:11.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:11.865+0000 7f4f12800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f0c106b80 0x7f4f0c19c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:16:11.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:11.865+0000 7f4f12800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f0c10c8f0 0x7f4f0c19c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:16:11.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:11.865+0000 7f4f12800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f0c19d060 con 0x7f4f0c10c8f0 2024-04-20T22:16:11.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:11.865+0000 7f4f12800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f0c1a0df0 con 0x7f4f0c106b80 2024-04-20T22:16:11.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:11.865+0000 7f4f02a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f0c10c8f0 0x7f4f0c19c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:16:11.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:11.865+0000 7f4f02a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f0c10c8f0 0x7f4f0c19c980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56944/0 (socket says 172.21.15.62:56944) 2024-04-20T22:16:11.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:11.865+0000 7f4f02a00700 1 -- 172.21.15.62:0/1643359414 learned_addr learned my addr 172.21.15.62:0/1643359414 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:16:11.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:11.865+0000 7f4f0b400700 1 --2- 172.21.15.62:0/1643359414 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f0c106b80 0x7f4f0c19c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:16:11.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:11.866+0000 7f4f02a00700 1 -- 172.21.15.62:0/1643359414 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f0c106b80 msgr2=0x7f4f0c19c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:16:11.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:11.866+0000 7f4f02a00700 1 --2- 172.21.15.62:0/1643359414 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f0c106b80 0x7f4f0c19c440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:16:11.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:11.866+0000 7f4f02a00700 1 -- 172.21.15.62:0/1643359414 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ef40095f0 con 0x7f4f0c10c8f0 2024-04-20T22:16:11.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:11.866+0000 7f4f0b400700 1 --2- 172.21.15.62:0/1643359414 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f0c106b80 0x7f4f0c19c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T22:16:11.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:11.866+0000 7f4f02a00700 1 --2- 172.21.15.62:0/1643359414 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f0c10c8f0 0x7f4f0c19c980 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f4ef400b5c0 tx=0x7f4ef400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:16:11.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:11.866+0000 7f4f08c00700 1 -- 172.21.15.62:0/1643359414 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4ef401d070 con 0x7f4f0c10c8f0 2024-04-20T22:16:11.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:11.866+0000 7f4f12800700 1 -- 172.21.15.62:0/1643359414 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f0c1a1070 con 0x7f4f0c10c8f0 2024-04-20T22:16:11.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:11.866+0000 7f4f08c00700 1 -- 172.21.15.62:0/1643359414 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4ef4005e00 con 0x7f4f0c10c8f0 2024-04-20T22:16:11.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:11.866+0000 7f4f08c00700 1 -- 172.21.15.62:0/1643359414 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4ef4017400 con 0x7f4f0c10c8f0 2024-04-20T22:16:11.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:11.866+0000 7f4f12800700 1 -- 172.21.15.62:0/1643359414 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f0c1a1560 con 0x7f4f0c10c8f0 2024-04-20T22:16:11.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:11.868+0000 7f4f12800700 1 -- 172.21.15.62:0/1643359414 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f0c10ecc0 con 0x7f4f0c10c8f0 2024-04-20T22:16:11.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:11.869+0000 7f4f08c00700 1 -- 172.21.15.62:0/1643359414 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f4ef400f460 con 0x7f4f0c10c8f0 2024-04-20T22:16:11.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:11.870+0000 7f4f08c00700 1 --2- 172.21.15.62:0/1643359414 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f4ef006ecc0 0x7f4ef0071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:16:11.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:11.870+0000 7f4f08c00700 1 -- 172.21.15.62:0/1643359414 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f4ef40903e0 con 0x7f4f0c10c8f0 2024-04-20T22:16:11.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:11.870+0000 7f4f0b400700 1 --2- 172.21.15.62:0/1643359414 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f4ef006ecc0 0x7f4ef0071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:16:11.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:11.871+0000 7f4f0b400700 1 --2- 172.21.15.62:0/1643359414 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f4ef006ecc0 0x7f4ef0071170 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f4efc0094f0 tx=0x7f4efc009040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:16:11.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:11.875+0000 7f4f08c00700 1 -- 172.21.15.62:0/1643359414 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f4ef4059190 con 0x7f4f0c10c8f0 2024-04-20T22:16:12.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:12.204+0000 7f4f12800700 1 -- 172.21.15.62:0/1643359414 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f4f0c19d840 con 0x7f4f0c10c8f0 2024-04-20T22:16:12.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:12.205+0000 7f4f08c00700 1 -- 172.21.15.62:0/1643359414 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7f4ef405c850 con 0x7f4f0c10c8f0 2024-04-20T22:16:12.204 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-20T22:16:12.205 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-20T22:16:12.205 INFO:teuthology.orchestra.run.smithi062.stdout: host smithi189 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi189/config 2024-04-20T22:16:12.205 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-20T22:16:12.205 INFO:teuthology.orchestra.run.smithi062.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi189 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp11670v2q:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:16:12.205 INFO:teuthology.orchestra.run.smithi062.stdout:Traceback (most recent call last): 2024-04-20T22:16:12.205 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-20T22:16:12.205 INFO:teuthology.orchestra.run.smithi062.stdout: return _run_code(code, main_globals, None, 2024-04-20T22:16:12.205 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-20T22:16:12.205 INFO:teuthology.orchestra.run.smithi062.stdout: exec(code, run_globals) 2024-04-20T22:16:12.205 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5618, in 2024-04-20T22:16:12.205 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5606, in main 2024-04-20T22:16:12.205 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 408, in _infer_config 2024-04-20T22:16:12.205 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 323, in _infer_fsid 2024-04-20T22:16:12.205 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 436, in _infer_image 2024-04-20T22:16:12.206 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 310, in _validate_fsid 2024-04-20T22:16:12.206 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 3359, in command_ceph_volume 2024-04-20T22:16:12.206 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-20T22:16:12.206 INFO:teuthology.orchestra.run.smithi062.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi189 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp11670v2q:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:16:12.206 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-20T22:16:12.206 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is degraded 2024-04-20T22:16:12.206 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T22:16:12.206 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T22:16:12.206 INFO:teuthology.orchestra.run.smithi062.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-20T22:16:12.206 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-20T22:16:12.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:12.208+0000 7f4f12800700 1 -- 172.21.15.62:0/1643359414 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f4ef006ecc0 msgr2=0x7f4ef0071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:16:12.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:12.208+0000 7f4f12800700 1 --2- 172.21.15.62:0/1643359414 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f4ef006ecc0 0x7f4ef0071170 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f4efc0094f0 tx=0x7f4efc009040 comp rx=0 tx=0).stop 2024-04-20T22:16:12.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:12.208+0000 7f4f12800700 1 -- 172.21.15.62:0/1643359414 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f0c10c8f0 msgr2=0x7f4f0c19c980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:16:12.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:12.208+0000 7f4f12800700 1 --2- 172.21.15.62:0/1643359414 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f0c10c8f0 0x7f4f0c19c980 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f4ef400b5c0 tx=0x7f4ef400be20 comp rx=0 tx=0).stop 2024-04-20T22:16:12.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:12.208+0000 7f4f12800700 1 -- 172.21.15.62:0/1643359414 shutdown_connections 2024-04-20T22:16:12.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:12.208+0000 7f4f12800700 1 --2- 172.21.15.62:0/1643359414 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f4ef006ecc0 0x7f4ef0071170 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:16:12.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:12.208+0000 7f4f12800700 1 --2- 172.21.15.62:0/1643359414 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f0c106b80 0x7f4f0c19c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:16:12.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:12.208+0000 7f4f12800700 1 --2- 172.21.15.62:0/1643359414 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f0c10c8f0 0x7f4f0c19c980 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:16:12.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:12.208+0000 7f4f12800700 1 -- 172.21.15.62:0/1643359414 >> 172.21.15.62:0/1643359414 conn(0x7f4f0c076060 msgr2=0x7f4f0c077ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:16:12.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:12.208+0000 7f4f12800700 1 -- 172.21.15.62:0/1643359414 shutdown_connections 2024-04-20T22:16:12.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:12.208+0000 7f4f12800700 1 -- 172.21.15.62:0/1643359414 wait complete. 2024-04-20T22:16:12.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:16:12 smithi189 ceph-mon[57022]: pgmap v1230: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:16:12.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:16:12 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/1643359414' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:16:12.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:16:12 smithi062 ceph-mon[76322]: pgmap v1230: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:16:12.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:16:12 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/1643359414' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:16:13.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:16:13 smithi189 ceph-mon[57022]: from='client.36182 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:16:13.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:16:13 smithi062 ceph-mon[76322]: from='client.36182 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:16:14.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:16:14 smithi189 ceph-mon[57022]: pgmap v1231: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:16:14.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:16:14 smithi062 ceph-mon[76322]: pgmap v1231: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:16:16.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:16:16 smithi189 ceph-mon[57022]: pgmap v1232: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:16:16.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:16:16 smithi062 ceph-mon[76322]: pgmap v1232: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:16:18.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:16:18 smithi062 ceph-mon[76322]: pgmap v1233: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:16:18.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:16:18 smithi189 ceph-mon[57022]: pgmap v1233: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:16:20.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:16:20 smithi189 ceph-mon[57022]: pgmap v1234: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:16:20.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:16:20 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:16:20.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:16:20 smithi062 ceph-mon[76322]: pgmap v1234: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:16:20.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:16:20 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:16:22.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:16:22 smithi062 ceph-mon[76322]: pgmap v1235: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:16:22.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:16:22 smithi189 ceph-mon[57022]: pgmap v1235: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:16:24.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:16:24 smithi189 ceph-mon[57022]: pgmap v1236: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:16:24.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:16:24 smithi062 ceph-mon[76322]: pgmap v1236: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:16:26.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:16:26 smithi189 ceph-mon[57022]: pgmap v1237: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:16:26.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:16:26 smithi062 ceph-mon[76322]: pgmap v1237: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:16:27.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:16:27 smithi189 ceph-mon[57022]: pgmap v1238: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:16:27.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:16:27 smithi062 ceph-mon[76322]: pgmap v1238: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:16:30.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:16:30 smithi189 ceph-mon[57022]: pgmap v1239: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:16:30.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:16:30 smithi062 ceph-mon[76322]: pgmap v1239: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:16:32.610 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:16:32 smithi062 ceph-mon[76322]: pgmap v1240: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:16:32.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:16:32 smithi189 ceph-mon[57022]: pgmap v1240: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:16:34.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:16:34 smithi189 ceph-mon[57022]: pgmap v1241: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:16:34.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:16:34 smithi062 ceph-mon[76322]: pgmap v1241: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:16:35.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:16:35 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:16:35.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:16:35 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:16:36.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:16:36 smithi189 ceph-mon[57022]: pgmap v1242: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:16:36.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:16:36 smithi062 ceph-mon[76322]: pgmap v1242: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:16:38.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:16:38 smithi062 ceph-mon[76322]: pgmap v1243: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:16:38.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:16:38 smithi189 ceph-mon[57022]: pgmap v1243: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:16:40.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:16:40 smithi189 ceph-mon[57022]: pgmap v1244: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:16:40.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:16:40 smithi062 ceph-mon[76322]: pgmap v1244: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:16:42.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:42.385+0000 7fce9b200700 1 -- 172.21.15.62:0/1812200430 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce9c0ff9e0 msgr2=0x7fce9c0ffe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:16:42.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:42.385+0000 7fce9b200700 1 --2- 172.21.15.62:0/1812200430 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce9c0ff9e0 0x7fce9c0ffe50 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7fce88009910 tx=0x7fce88009c20 comp rx=0 tx=0).stop 2024-04-20T22:16:42.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:42.385+0000 7fce9b200700 1 -- 172.21.15.62:0/1812200430 shutdown_connections 2024-04-20T22:16:42.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:42.385+0000 7fce9b200700 1 --2- 172.21.15.62:0/1812200430 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce9c0ff9e0 0x7fce9c0ffe50 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:16:42.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:42.385+0000 7fce9b200700 1 --2- 172.21.15.62:0/1812200430 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce9c102e30 0x7fce9c103200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:16:42.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:42.385+0000 7fce9b200700 1 -- 172.21.15.62:0/1812200430 >> 172.21.15.62:0/1812200430 conn(0x7fce9c074fa0 msgr2=0x7fce9c0753a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:16:42.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:42.385+0000 7fce9b200700 1 -- 172.21.15.62:0/1812200430 shutdown_connections 2024-04-20T22:16:42.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:42.385+0000 7fce9b200700 1 -- 172.21.15.62:0/1812200430 wait complete. 2024-04-20T22:16:42.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:42.386+0000 7fce9b200700 1 Processor -- start 2024-04-20T22:16:42.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:42.386+0000 7fce9b200700 1 -- start start 2024-04-20T22:16:42.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:42.387+0000 7fce9b200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce9c0ff9e0 0x7fce9c19b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:16:42.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:42.387+0000 7fce9b200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce9c102e30 0x7fce9c19be40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:16:42.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:42.387+0000 7fce9b200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce9c19c4d0 con 0x7fce9c0ff9e0 2024-04-20T22:16:42.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:42.387+0000 7fce9b200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce9c195980 con 0x7fce9c102e30 2024-04-20T22:16:42.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:42.387+0000 7fce99e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce9c0ff9e0 0x7fce9c19b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:16:42.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:42.387+0000 7fce99e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce9c0ff9e0 0x7fce9c19b900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:36440/0 (socket says 172.21.15.62:36440) 2024-04-20T22:16:42.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:42.387+0000 7fce99e00700 1 -- 172.21.15.62:0/2654449867 learned_addr learned my addr 172.21.15.62:0/2654449867 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:16:42.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:42.387+0000 7fce99400700 1 --2- 172.21.15.62:0/2654449867 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce9c102e30 0x7fce9c19be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:16:42.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:42.387+0000 7fce99e00700 1 -- 172.21.15.62:0/2654449867 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce9c102e30 msgr2=0x7fce9c19be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:16:42.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:42.387+0000 7fce99e00700 1 --2- 172.21.15.62:0/2654449867 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce9c102e30 0x7fce9c19be40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:16:42.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:42.387+0000 7fce99e00700 1 -- 172.21.15.62:0/2654449867 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce880095f0 con 0x7fce9c0ff9e0 2024-04-20T22:16:42.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:42.388+0000 7fce99e00700 1 --2- 172.21.15.62:0/2654449867 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce9c0ff9e0 0x7fce9c19b900 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7fce8c00d8d0 tx=0x7fce8c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:16:42.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:42.388+0000 7fce92a00700 1 -- 172.21.15.62:0/2654449867 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce8c009940 con 0x7fce9c0ff9e0 2024-04-20T22:16:42.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:42.388+0000 7fce92a00700 1 -- 172.21.15.62:0/2654449867 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fce8c010460 con 0x7fce9c0ff9e0 2024-04-20T22:16:42.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:42.388+0000 7fce92a00700 1 -- 172.21.15.62:0/2654449867 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce8c00f5f0 con 0x7fce9c0ff9e0 2024-04-20T22:16:42.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:42.388+0000 7fce9b200700 1 -- 172.21.15.62:0/2654449867 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce9c195c60 con 0x7fce9c0ff9e0 2024-04-20T22:16:42.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:42.388+0000 7fce9b200700 1 -- 172.21.15.62:0/2654449867 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce9c196230 con 0x7fce9c0ff9e0 2024-04-20T22:16:42.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:42.390+0000 7fce9b200700 1 -- 172.21.15.62:0/2654449867 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce9c1084a0 con 0x7fce9c0ff9e0 2024-04-20T22:16:42.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:42.391+0000 7fce92a00700 1 -- 172.21.15.62:0/2654449867 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fce8c0105d0 con 0x7fce9c0ff9e0 2024-04-20T22:16:42.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:42.392+0000 7fce92a00700 1 --2- 172.21.15.62:0/2654449867 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fce8006ecc0 0x7fce80071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:16:42.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:42.392+0000 7fce92a00700 1 -- 172.21.15.62:0/2654449867 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fce8c08ed70 con 0x7fce9c0ff9e0 2024-04-20T22:16:42.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:42.392+0000 7fce99400700 1 --2- 172.21.15.62:0/2654449867 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fce8006ecc0 0x7fce80071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:16:42.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:42.393+0000 7fce99400700 1 --2- 172.21.15.62:0/2654449867 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fce8006ecc0 0x7fce80071170 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fce88000c00 tx=0x7fce88005fd0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:16:42.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:42.397+0000 7fce92a00700 1 -- 172.21.15.62:0/2654449867 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fce8c05b6d0 con 0x7fce9c0ff9e0 2024-04-20T22:16:42.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:42.655+0000 7fce9b200700 1 -- 172.21.15.62:0/2654449867 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fce9c1964a0 con 0x7fce8006ecc0 2024-04-20T22:16:42.655 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:16:42 smithi062 ceph-mon[76322]: pgmap v1245: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:16:42.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:42.657+0000 7fce92a00700 1 -- 172.21.15.62:0/2654449867 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fce9c1964a0 con 0x7fce8006ecc0 2024-04-20T22:16:42.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:42.661+0000 7fce9b200700 1 -- 172.21.15.62:0/2654449867 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fce8006ecc0 msgr2=0x7fce80071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:16:42.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:42.661+0000 7fce9b200700 1 --2- 172.21.15.62:0/2654449867 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fce8006ecc0 0x7fce80071170 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fce88000c00 tx=0x7fce88005fd0 comp rx=0 tx=0).stop 2024-04-20T22:16:42.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:42.661+0000 7fce9b200700 1 -- 172.21.15.62:0/2654449867 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce9c0ff9e0 msgr2=0x7fce9c19b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:16:42.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:42.661+0000 7fce9b200700 1 --2- 172.21.15.62:0/2654449867 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce9c0ff9e0 0x7fce9c19b900 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7fce8c00d8d0 tx=0x7fce8c00dc90 comp rx=0 tx=0).stop 2024-04-20T22:16:42.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:42.661+0000 7fce9b200700 1 -- 172.21.15.62:0/2654449867 shutdown_connections 2024-04-20T22:16:42.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:42.661+0000 7fce9b200700 1 --2- 172.21.15.62:0/2654449867 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fce8006ecc0 0x7fce80071170 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:16:42.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:42.661+0000 7fce9b200700 1 --2- 172.21.15.62:0/2654449867 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce9c0ff9e0 0x7fce9c19b900 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:16:42.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:42.661+0000 7fce9b200700 1 --2- 172.21.15.62:0/2654449867 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce9c102e30 0x7fce9c19be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:16:42.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:42.661+0000 7fce9b200700 1 -- 172.21.15.62:0/2654449867 >> 172.21.15.62:0/2654449867 conn(0x7fce9c074fa0 msgr2=0x7fce9c101ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:16:42.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:42.661+0000 7fce9b200700 1 -- 172.21.15.62:0/2654449867 shutdown_connections 2024-04-20T22:16:42.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:42.661+0000 7fce9b200700 1 -- 172.21.15.62:0/2654449867 wait complete. 2024-04-20T22:16:42.678 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T22:16:42.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:16:42 smithi189 ceph-mon[57022]: pgmap v1245: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:16:42.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:42.830+0000 7f453f600700 1 -- 172.21.15.62:0/3656589741 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4538106b80 msgr2=0x7f4538106ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:16:42.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:42.830+0000 7f453f600700 1 --2- 172.21.15.62:0/3656589741 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4538106b80 0x7f4538106ff0 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f4524009910 tx=0x7f4524009c20 comp rx=0 tx=0).stop 2024-04-20T22:16:42.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:42.830+0000 7f453f600700 1 -- 172.21.15.62:0/3656589741 shutdown_connections 2024-04-20T22:16:42.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:42.830+0000 7f453f600700 1 --2- 172.21.15.62:0/3656589741 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4538106b80 0x7f4538106ff0 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:16:42.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:42.830+0000 7f453f600700 1 --2- 172.21.15.62:0/3656589741 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f453810c8f0 0x7f453810ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:16:42.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:42.830+0000 7f453f600700 1 -- 172.21.15.62:0/3656589741 >> 172.21.15.62:0/3656589741 conn(0x7f4538076020 msgr2=0x7f4538078430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:16:42.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:42.830+0000 7f453f600700 1 -- 172.21.15.62:0/3656589741 shutdown_connections 2024-04-20T22:16:42.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:42.830+0000 7f453f600700 1 -- 172.21.15.62:0/3656589741 wait complete. 2024-04-20T22:16:42.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:42.831+0000 7f453f600700 1 Processor -- start 2024-04-20T22:16:42.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:42.831+0000 7f453f600700 1 -- start start 2024-04-20T22:16:42.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:42.832+0000 7f453f600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4538106b80 0x7f453819c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:16:42.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:42.832+0000 7f453f600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f453810c8f0 0x7f453819c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:16:42.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:42.832+0000 7f453f600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f453819d070 con 0x7f453810c8f0 2024-04-20T22:16:42.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:42.832+0000 7f453f600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45381a0e00 con 0x7f4538106b80 2024-04-20T22:16:42.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:42.832+0000 7f4537e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f453810c8f0 0x7f453819c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:16:42.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:42.832+0000 7f4537e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f453810c8f0 0x7f453819c990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:36456/0 (socket says 172.21.15.62:36456) 2024-04-20T22:16:42.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:42.832+0000 7f4537e00700 1 -- 172.21.15.62:0/4150825408 learned_addr learned my addr 172.21.15.62:0/4150825408 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:16:42.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:42.833+0000 7f453cc00700 1 --2- 172.21.15.62:0/4150825408 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4538106b80 0x7f453819c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:16:42.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:42.833+0000 7f4537e00700 1 -- 172.21.15.62:0/4150825408 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4538106b80 msgr2=0x7f453819c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:16:42.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:42.833+0000 7f4537e00700 1 --2- 172.21.15.62:0/4150825408 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4538106b80 0x7f453819c450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:16:42.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:42.833+0000 7f4537e00700 1 -- 172.21.15.62:0/4150825408 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45240095f0 con 0x7f453810c8f0 2024-04-20T22:16:42.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:42.833+0000 7f4537e00700 1 --2- 172.21.15.62:0/4150825408 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f453810c8f0 0x7f453819c990 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f4524006010 tx=0x7f452400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:16:42.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:42.833+0000 7f4535600700 1 -- 172.21.15.62:0/4150825408 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f452401d070 con 0x7f453810c8f0 2024-04-20T22:16:42.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:42.833+0000 7f4535600700 1 -- 172.21.15.62:0/4150825408 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f452400f460 con 0x7f453810c8f0 2024-04-20T22:16:42.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:42.833+0000 7f4535600700 1 -- 172.21.15.62:0/4150825408 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4524017630 con 0x7f453810c8f0 2024-04-20T22:16:42.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:42.833+0000 7f453f600700 1 -- 172.21.15.62:0/4150825408 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45381a1080 con 0x7f453810c8f0 2024-04-20T22:16:42.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:42.834+0000 7f453f600700 1 -- 172.21.15.62:0/4150825408 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45381a1570 con 0x7f453810c8f0 2024-04-20T22:16:42.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:42.835+0000 7f453f600700 1 -- 172.21.15.62:0/4150825408 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f453804f2a0 con 0x7f453810c8f0 2024-04-20T22:16:42.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:42.836+0000 7f4535600700 1 -- 172.21.15.62:0/4150825408 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f452400f5d0 con 0x7f453810c8f0 2024-04-20T22:16:42.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:42.837+0000 7f4535600700 1 --2- 172.21.15.62:0/4150825408 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f452006ecc0 0x7f4520071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:16:42.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:42.837+0000 7f4535600700 1 -- 172.21.15.62:0/4150825408 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f4524090350 con 0x7f453810c8f0 2024-04-20T22:16:42.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:42.837+0000 7f453cc00700 1 --2- 172.21.15.62:0/4150825408 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f452006ecc0 0x7f4520071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:16:42.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:42.838+0000 7f453cc00700 1 --2- 172.21.15.62:0/4150825408 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f452006ecc0 0x7f4520071170 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f452c005950 tx=0x7f452c0058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:16:42.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:42.842+0000 7f4535600700 1 -- 172.21.15.62:0/4150825408 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f45240590e0 con 0x7f453810c8f0 2024-04-20T22:16:43.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:43.096+0000 7f453f600700 1 -- 172.21.15.62:0/4150825408 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f45381a1930 con 0x7f452006ecc0 2024-04-20T22:16:43.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:43.099+0000 7f4535600700 1 -- 172.21.15.62:0/4150825408 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f45381a1930 con 0x7f452006ecc0 2024-04-20T22:16:43.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:43.101+0000 7f453f600700 1 -- 172.21.15.62:0/4150825408 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f452006ecc0 msgr2=0x7f4520071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:16:43.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:43.101+0000 7f453f600700 1 --2- 172.21.15.62:0/4150825408 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f452006ecc0 0x7f4520071170 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f452c005950 tx=0x7f452c0058e0 comp rx=0 tx=0).stop 2024-04-20T22:16:43.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:43.101+0000 7f453f600700 1 -- 172.21.15.62:0/4150825408 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f453810c8f0 msgr2=0x7f453819c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:16:43.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:43.101+0000 7f453f600700 1 --2- 172.21.15.62:0/4150825408 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f453810c8f0 0x7f453819c990 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f4524006010 tx=0x7f452400be20 comp rx=0 tx=0).stop 2024-04-20T22:16:43.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:43.101+0000 7f453f600700 1 -- 172.21.15.62:0/4150825408 shutdown_connections 2024-04-20T22:16:43.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:43.101+0000 7f453f600700 1 --2- 172.21.15.62:0/4150825408 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f452006ecc0 0x7f4520071170 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:16:43.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:43.101+0000 7f453f600700 1 --2- 172.21.15.62:0/4150825408 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4538106b80 0x7f453819c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:16:43.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:43.101+0000 7f453f600700 1 --2- 172.21.15.62:0/4150825408 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f453810c8f0 0x7f453819c990 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:16:43.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:43.101+0000 7f453f600700 1 -- 172.21.15.62:0/4150825408 >> 172.21.15.62:0/4150825408 conn(0x7f4538076020 msgr2=0x7f4538077a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:16:43.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:43.102+0000 7f453f600700 1 -- 172.21.15.62:0/4150825408 shutdown_connections 2024-04-20T22:16:43.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:43.102+0000 7f453f600700 1 -- 172.21.15.62:0/4150825408 wait complete. 2024-04-20T22:16:43.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:43.252+0000 7f9749a00700 1 -- 172.21.15.62:0/2421013017 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9744100c30 msgr2=0x7f9744104c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:16:43.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:43.252+0000 7f9749a00700 1 --2- 172.21.15.62:0/2421013017 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9744100c30 0x7f9744104c00 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f972c009910 tx=0x7f972c009c20 comp rx=0 tx=0).stop 2024-04-20T22:16:43.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:43.252+0000 7f9749a00700 1 -- 172.21.15.62:0/2421013017 shutdown_connections 2024-04-20T22:16:43.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:43.252+0000 7f9749a00700 1 --2- 172.21.15.62:0/2421013017 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9744100c30 0x7f9744104c00 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:16:43.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:43.252+0000 7f9749a00700 1 --2- 172.21.15.62:0/2421013017 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9744100290 0x7f9744100660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:16:43.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:43.252+0000 7f9749a00700 1 -- 172.21.15.62:0/2421013017 >> 172.21.15.62:0/2421013017 conn(0x7f97440fbb10 msgr2=0x7f97440fdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:16:43.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:43.252+0000 7f9749a00700 1 -- 172.21.15.62:0/2421013017 shutdown_connections 2024-04-20T22:16:43.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:43.252+0000 7f9749a00700 1 -- 172.21.15.62:0/2421013017 wait complete. 2024-04-20T22:16:43.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:43.253+0000 7f9749a00700 1 Processor -- start 2024-04-20T22:16:43.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:43.253+0000 7f9749a00700 1 -- start start 2024-04-20T22:16:43.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:43.253+0000 7f9749a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9744100290 0x7f9744073050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:16:43.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:43.254+0000 7f9749a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9744100c30 0x7f9744073590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:16:43.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:43.254+0000 7f9749a00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9744073ad0 con 0x7f9744100c30 2024-04-20T22:16:43.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:43.254+0000 7f9749a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9744073c10 con 0x7f9744100290 2024-04-20T22:16:43.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:43.254+0000 7f973b400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9744100c30 0x7f9744073590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:16:43.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:43.254+0000 7f973b400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9744100c30 0x7f9744073590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:36474/0 (socket says 172.21.15.62:36474) 2024-04-20T22:16:43.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:43.254+0000 7f973b400700 1 -- 172.21.15.62:0/509906708 learned_addr learned my addr 172.21.15.62:0/509906708 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:16:43.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:43.254+0000 7f9743e00700 1 --2- 172.21.15.62:0/509906708 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9744100290 0x7f9744073050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:16:43.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:43.254+0000 7f973b400700 1 -- 172.21.15.62:0/509906708 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9744100290 msgr2=0x7f9744073050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:16:43.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:43.254+0000 7f973b400700 1 --2- 172.21.15.62:0/509906708 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9744100290 0x7f9744073050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:16:43.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:43.254+0000 7f973b400700 1 -- 172.21.15.62:0/509906708 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f972c0095f0 con 0x7f9744100c30 2024-04-20T22:16:43.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:43.255+0000 7f973b400700 1 --2- 172.21.15.62:0/509906708 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9744100c30 0x7f9744073590 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f972c00bda0 tx=0x7f972c00be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:16:43.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:43.255+0000 7f9741600700 1 -- 172.21.15.62:0/509906708 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f972c01d070 con 0x7f9744100c30 2024-04-20T22:16:43.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:43.255+0000 7f9741600700 1 -- 172.21.15.62:0/509906708 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f972c004fb0 con 0x7f9744100c30 2024-04-20T22:16:43.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:43.255+0000 7f9749a00700 1 -- 172.21.15.62:0/509906708 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9744073e90 con 0x7f9744100c30 2024-04-20T22:16:43.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:43.255+0000 7f9741600700 1 -- 172.21.15.62:0/509906708 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f972c00f460 con 0x7f9744100c30 2024-04-20T22:16:43.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:43.255+0000 7f9749a00700 1 -- 172.21.15.62:0/509906708 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97441a21a0 con 0x7f9744100c30 2024-04-20T22:16:43.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:43.257+0000 7f9749a00700 1 -- 172.21.15.62:0/509906708 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9744108600 con 0x7f9744100c30 2024-04-20T22:16:43.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:43.258+0000 7f9741600700 1 -- 172.21.15.62:0/509906708 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f972c022800 con 0x7f9744100c30 2024-04-20T22:16:43.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:43.259+0000 7f9741600700 1 --2- 172.21.15.62:0/509906708 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f972806ecc0 0x7f9728071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:16:43.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:43.259+0000 7f9741600700 1 -- 172.21.15.62:0/509906708 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f972c0903f0 con 0x7f9744100c30 2024-04-20T22:16:43.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:43.259+0000 7f9743e00700 1 --2- 172.21.15.62:0/509906708 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f972806ecc0 0x7f9728071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:16:43.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:43.260+0000 7f9743e00700 1 --2- 172.21.15.62:0/509906708 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f972806ecc0 0x7f9728071170 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f9734005fd0 tx=0x7f973400a560 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:16:43.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:43.263+0000 7f9741600700 1 -- 172.21.15.62:0/509906708 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f972c0592d0 con 0x7f9744100c30 2024-04-20T22:16:43.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:43.509+0000 7f9749a00700 1 -- 172.21.15.62:0/509906708 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f9744061190 con 0x7f972806ecc0 2024-04-20T22:16:43.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:43.542+0000 7f9741600700 1 -- 172.21.15.62:0/509906708 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f9744061190 con 0x7f972806ecc0 2024-04-20T22:16:43.543 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T22:16:43.543 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (42m) 3m ago 56m 14.6M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T22:16:43.543 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (56m) 3m ago 56m 11.6M - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T22:16:43.543 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (55m) 3m ago 55m 11.2M - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T22:16:43.543 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (38m) 3m ago 56m 6660k - 19.0.0-3133-ga8c84f59 76bc6b91b726 2825d5ba11c6 2024-04-20T22:16:43.543 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (38m) 3m ago 55m 6665k - 19.0.0-3133-ga8c84f59 76bc6b91b726 20f746fa1bba 2024-04-20T22:16:43.543 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (41m) 3m ago 56m 75.5M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T22:16:43.543 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (34m) 3m ago 49m 22.9M - 19.0.0-3133-ga8c84f59 76bc6b91b726 de80bc90a683 2024-04-20T22:16:43.543 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (34m) 3m ago 49m 23.0M - 19.0.0-3133-ga8c84f59 76bc6b91b726 cd793b7ee98e 2024-04-20T22:16:43.543 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (18m) 3m ago 49m 18.5M - 19.0.0-3133-ga8c84f59 76bc6b91b726 0e7b0bc4d1ab 2024-04-20T22:16:43.544 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (3m) 3m ago 49m 11.4M - 19.0.0-3133-ga8c84f59 76bc6b91b726 0af732ac9e06 2024-04-20T22:16:43.544 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (45m) 3m ago 58m 517M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T22:16:43.544 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (44m) 3m ago 55m 460M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T22:16:43.544 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (40m) 3m ago 58m 86.3M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 019eb1479485 2024-04-20T22:16:43.544 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (39m) 3m ago 55m 58.0M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 55aac560aab6 2024-04-20T22:16:43.544 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (43m) 3m ago 56m 8766k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T22:16:43.544 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (43m) 3m ago 55m 8711k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T22:16:43.544 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (37m) 3m ago 54m 91.3M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 ee2bb901f26e 2024-04-20T22:16:43.544 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (37m) 3m ago 53m 94.9M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 9075152d4162 2024-04-20T22:16:43.544 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (36m) 3m ago 52m 78.5M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 2d4f3e38dc0e 2024-04-20T22:16:43.544 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (36m) 3m ago 52m 94.1M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 8f97b17ae0eb 2024-04-20T22:16:43.544 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (35m) 3m ago 51m 96.1M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 653d8079967d 2024-04-20T22:16:43.544 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (34m) 3m ago 51m 90.9M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 596dded14c12 2024-04-20T22:16:43.544 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (43m) 3m ago 55m 90.4M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T22:16:43.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:43.545+0000 7f9749a00700 1 -- 172.21.15.62:0/509906708 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f972806ecc0 msgr2=0x7f9728071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:16:43.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:43.545+0000 7f9749a00700 1 --2- 172.21.15.62:0/509906708 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f972806ecc0 0x7f9728071170 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f9734005fd0 tx=0x7f973400a560 comp rx=0 tx=0).stop 2024-04-20T22:16:43.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:43.545+0000 7f9749a00700 1 -- 172.21.15.62:0/509906708 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9744100c30 msgr2=0x7f9744073590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:16:43.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:43.545+0000 7f9749a00700 1 --2- 172.21.15.62:0/509906708 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9744100c30 0x7f9744073590 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f972c00bda0 tx=0x7f972c00be80 comp rx=0 tx=0).stop 2024-04-20T22:16:43.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:43.545+0000 7f9749a00700 1 -- 172.21.15.62:0/509906708 shutdown_connections 2024-04-20T22:16:43.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:43.545+0000 7f9749a00700 1 --2- 172.21.15.62:0/509906708 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f972806ecc0 0x7f9728071170 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:16:43.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:43.545+0000 7f9749a00700 1 --2- 172.21.15.62:0/509906708 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9744100290 0x7f9744073050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:16:43.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:43.545+0000 7f9749a00700 1 --2- 172.21.15.62:0/509906708 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9744100c30 0x7f9744073590 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:16:43.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:43.545+0000 7f9749a00700 1 -- 172.21.15.62:0/509906708 >> 172.21.15.62:0/509906708 conn(0x7f97440fbb10 msgr2=0x7f97440fd120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:16:43.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:43.546+0000 7f9749a00700 1 -- 172.21.15.62:0/509906708 shutdown_connections 2024-04-20T22:16:43.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:43.546+0000 7f9749a00700 1 -- 172.21.15.62:0/509906708 wait complete. 2024-04-20T22:16:43.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:43.714+0000 7f9dcbe00700 1 -- 172.21.15.62:0/512205012 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9dcc106b80 msgr2=0x7f9dcc106ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:16:43.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:43.714+0000 7f9dcbe00700 1 --2- 172.21.15.62:0/512205012 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9dcc106b80 0x7f9dcc106ff0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f9db8009910 tx=0x7f9db8009c20 comp rx=0 tx=0).stop 2024-04-20T22:16:43.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:43.714+0000 7f9dcbe00700 1 -- 172.21.15.62:0/512205012 shutdown_connections 2024-04-20T22:16:43.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:43.714+0000 7f9dcbe00700 1 --2- 172.21.15.62:0/512205012 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9dcc106b80 0x7f9dcc106ff0 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:16:43.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:43.714+0000 7f9dcbe00700 1 --2- 172.21.15.62:0/512205012 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9dcc10c660 0x7f9dcc10ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:16:43.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:43.714+0000 7f9dcbe00700 1 -- 172.21.15.62:0/512205012 >> 172.21.15.62:0/512205012 conn(0x7f9dcc075e90 msgr2=0x7f9dcc0782a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:16:43.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:43.714+0000 7f9dcbe00700 1 -- 172.21.15.62:0/512205012 shutdown_connections 2024-04-20T22:16:43.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:43.714+0000 7f9dcbe00700 1 -- 172.21.15.62:0/512205012 wait complete. 2024-04-20T22:16:43.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:43.715+0000 7f9dcbe00700 1 Processor -- start 2024-04-20T22:16:43.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:43.715+0000 7f9dcbe00700 1 -- start start 2024-04-20T22:16:43.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:43.715+0000 7f9dcbe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9dcc106b80 0x7f9dcc19c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:16:43.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:43.715+0000 7f9dcbe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9dcc10c660 0x7f9dcc19c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:16:43.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:43.715+0000 7f9dcbe00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dcc19ce00 con 0x7f9dcc10c660 2024-04-20T22:16:43.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:43.715+0000 7f9dcbe00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dcc1a0b90 con 0x7f9dcc106b80 2024-04-20T22:16:43.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:43.716+0000 7f9dca000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9dcc10c660 0x7f9dcc19c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:16:43.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:43.716+0000 7f9dca000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9dcc10c660 0x7f9dcc19c720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:36496/0 (socket says 172.21.15.62:36496) 2024-04-20T22:16:43.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:43.716+0000 7f9dca000700 1 -- 172.21.15.62:0/3459510735 learned_addr learned my addr 172.21.15.62:0/3459510735 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:16:43.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:43.716+0000 7f9dcaa00700 1 --2- 172.21.15.62:0/3459510735 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9dcc106b80 0x7f9dcc19c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:16:43.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:43.716+0000 7f9dca000700 1 -- 172.21.15.62:0/3459510735 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9dcc106b80 msgr2=0x7f9dcc19c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:16:43.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:43.716+0000 7f9dca000700 1 --2- 172.21.15.62:0/3459510735 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9dcc106b80 0x7f9dcc19c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:16:43.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:43.716+0000 7f9dca000700 1 -- 172.21.15.62:0/3459510735 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9db80095f0 con 0x7f9dcc10c660 2024-04-20T22:16:43.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:43.716+0000 7f9dca000700 1 --2- 172.21.15.62:0/3459510735 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9dcc10c660 0x7f9dcc19c720 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f9db800b5c0 tx=0x7f9db800be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:16:43.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:43.717+0000 7f9dc3400700 1 -- 172.21.15.62:0/3459510735 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9db801d070 con 0x7f9dcc10c660 2024-04-20T22:16:43.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:43.717+0000 7f9dc3400700 1 -- 172.21.15.62:0/3459510735 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9db800f460 con 0x7f9dcc10c660 2024-04-20T22:16:43.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:43.717+0000 7f9dc3400700 1 -- 172.21.15.62:0/3459510735 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9db8017610 con 0x7f9dcc10c660 2024-04-20T22:16:43.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:43.717+0000 7f9dcbe00700 1 -- 172.21.15.62:0/3459510735 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dcc1a0e10 con 0x7f9dcc10c660 2024-04-20T22:16:43.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:43.717+0000 7f9dcbe00700 1 -- 172.21.15.62:0/3459510735 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dcc1a1300 con 0x7f9dcc10c660 2024-04-20T22:16:43.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:43.718+0000 7f9dcbe00700 1 -- 172.21.15.62:0/3459510735 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9dcc10eb40 con 0x7f9dcc10c660 2024-04-20T22:16:43.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:43.719+0000 7f9dc3400700 1 -- 172.21.15.62:0/3459510735 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f9db800f5d0 con 0x7f9dcc10c660 2024-04-20T22:16:43.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:43.720+0000 7f9dc3400700 1 --2- 172.21.15.62:0/3459510735 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9db006ecc0 0x7f9db0071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:16:43.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:43.720+0000 7f9dc3400700 1 -- 172.21.15.62:0/3459510735 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f9db8090440 con 0x7f9dcc10c660 2024-04-20T22:16:43.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:43.720+0000 7f9dcaa00700 1 --2- 172.21.15.62:0/3459510735 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9db006ecc0 0x7f9db0071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:16:43.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:43.721+0000 7f9dcaa00700 1 --2- 172.21.15.62:0/3459510735 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9db006ecc0 0x7f9db0071170 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f9dbc005fd0 tx=0x7f9dbc00a560 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:16:43.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:43.725+0000 7f9dc3400700 1 -- 172.21.15.62:0/3459510735 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f9db80591d0 con 0x7f9dcc10c660 2024-04-20T22:16:44.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.044+0000 7f9dcbe00700 1 -- 172.21.15.62:0/3459510735 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f9dcc062380 con 0x7f9dcc10c660 2024-04-20T22:16:44.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.045+0000 7f9dc3400700 1 -- 172.21.15.62:0/3459510735 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f9db805c890 con 0x7f9dcc10c660 2024-04-20T22:16:44.045 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:16:44.045 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T22:16:44.045 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:16:44.045 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:16:44.045 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T22:16:44.045 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:16:44.045 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:16:44.045 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T22:16:44.045 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 6 2024-04-20T22:16:44.045 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:16:44.046 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T22:16:44.046 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 4 2024-04-20T22:16:44.046 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:16:44.046 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T22:16:44.046 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 14 2024-04-20T22:16:44.046 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T22:16:44.046 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:16:44.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.047+0000 7f9dcbe00700 1 -- 172.21.15.62:0/3459510735 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9db006ecc0 msgr2=0x7f9db0071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:16:44.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.047+0000 7f9dcbe00700 1 --2- 172.21.15.62:0/3459510735 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9db006ecc0 0x7f9db0071170 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f9dbc005fd0 tx=0x7f9dbc00a560 comp rx=0 tx=0).stop 2024-04-20T22:16:44.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.047+0000 7f9dcbe00700 1 -- 172.21.15.62:0/3459510735 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9dcc10c660 msgr2=0x7f9dcc19c720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:16:44.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.047+0000 7f9dcbe00700 1 --2- 172.21.15.62:0/3459510735 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9dcc10c660 0x7f9dcc19c720 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f9db800b5c0 tx=0x7f9db800be00 comp rx=0 tx=0).stop 2024-04-20T22:16:44.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.048+0000 7f9dcbe00700 1 -- 172.21.15.62:0/3459510735 shutdown_connections 2024-04-20T22:16:44.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.048+0000 7f9dcbe00700 1 --2- 172.21.15.62:0/3459510735 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9db006ecc0 0x7f9db0071170 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:16:44.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.048+0000 7f9dcbe00700 1 --2- 172.21.15.62:0/3459510735 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9dcc106b80 0x7f9dcc19c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:16:44.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.048+0000 7f9dcbe00700 1 --2- 172.21.15.62:0/3459510735 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9dcc10c660 0x7f9dcc19c720 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:16:44.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.048+0000 7f9dcbe00700 1 -- 172.21.15.62:0/3459510735 >> 172.21.15.62:0/3459510735 conn(0x7f9dcc075e90 msgr2=0x7f9dcc0778d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:16:44.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.048+0000 7f9dcbe00700 1 -- 172.21.15.62:0/3459510735 shutdown_connections 2024-04-20T22:16:44.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.048+0000 7f9dcbe00700 1 -- 172.21.15.62:0/3459510735 wait complete. 2024-04-20T22:16:44.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.203+0000 7f0b2bc00700 1 -- 172.21.15.62:0/2759582535 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0b24100e80 msgr2=0x7f0b24104e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:16:44.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.203+0000 7f0b2bc00700 1 --2- 172.21.15.62:0/2759582535 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0b24100e80 0x7f0b24104e50 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f0b0c009910 tx=0x7f0b0c009c20 comp rx=0 tx=0).stop 2024-04-20T22:16:44.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.203+0000 7f0b2bc00700 1 -- 172.21.15.62:0/2759582535 shutdown_connections 2024-04-20T22:16:44.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.203+0000 7f0b2bc00700 1 --2- 172.21.15.62:0/2759582535 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0b24100e80 0x7f0b24104e50 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:16:44.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.203+0000 7f0b2bc00700 1 --2- 172.21.15.62:0/2759582535 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b241004e0 0x7f0b241008b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:16:44.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.203+0000 7f0b2bc00700 1 -- 172.21.15.62:0/2759582535 >> 172.21.15.62:0/2759582535 conn(0x7f0b240fbda0 msgr2=0x7f0b240fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:16:44.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.203+0000 7f0b2bc00700 1 -- 172.21.15.62:0/2759582535 shutdown_connections 2024-04-20T22:16:44.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.204+0000 7f0b2bc00700 1 -- 172.21.15.62:0/2759582535 wait complete. 2024-04-20T22:16:44.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.205+0000 7f0b2bc00700 1 Processor -- start 2024-04-20T22:16:44.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.205+0000 7f0b2bc00700 1 -- start start 2024-04-20T22:16:44.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.205+0000 7f0b2bc00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b241004e0 0x7f0b24198130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:16:44.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.205+0000 7f0b2bc00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0b24100e80 0x7f0b24198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:16:44.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.205+0000 7f0b2bc00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b24198d50 con 0x7f0b24100e80 2024-04-20T22:16:44.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.205+0000 7f0b2bc00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b2419cae0 con 0x7f0b241004e0 2024-04-20T22:16:44.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.205+0000 7f0b1be00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0b24100e80 0x7f0b24198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:16:44.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.206+0000 7f0b1be00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0b24100e80 0x7f0b24198670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:36512/0 (socket says 172.21.15.62:36512) 2024-04-20T22:16:44.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.206+0000 7f0b1be00700 1 -- 172.21.15.62:0/1551301756 learned_addr learned my addr 172.21.15.62:0/1551301756 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:16:44.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.206+0000 7f0b29200700 1 --2- 172.21.15.62:0/1551301756 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b241004e0 0x7f0b24198130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:16:44.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.206+0000 7f0b1be00700 1 -- 172.21.15.62:0/1551301756 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b241004e0 msgr2=0x7f0b24198130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:16:44.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.206+0000 7f0b1be00700 1 --2- 172.21.15.62:0/1551301756 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b241004e0 0x7f0b24198130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:16:44.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.206+0000 7f0b1be00700 1 -- 172.21.15.62:0/1551301756 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b0c0095f0 con 0x7f0b24100e80 2024-04-20T22:16:44.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.206+0000 7f0b1be00700 1 --2- 172.21.15.62:0/1551301756 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0b24100e80 0x7f0b24198670 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f0b0c00bda0 tx=0x7f0b0c00be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:16:44.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.207+0000 7f0b22000700 1 -- 172.21.15.62:0/1551301756 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0b0c01d070 con 0x7f0b24100e80 2024-04-20T22:16:44.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.207+0000 7f0b22000700 1 -- 172.21.15.62:0/1551301756 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0b0c004fb0 con 0x7f0b24100e80 2024-04-20T22:16:44.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.207+0000 7f0b22000700 1 -- 172.21.15.62:0/1551301756 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0b0c00f480 con 0x7f0b24100e80 2024-04-20T22:16:44.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.207+0000 7f0b2bc00700 1 -- 172.21.15.62:0/1551301756 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b2419cd60 con 0x7f0b24100e80 2024-04-20T22:16:44.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.207+0000 7f0b2bc00700 1 -- 172.21.15.62:0/1551301756 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b2419d250 con 0x7f0b24100e80 2024-04-20T22:16:44.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.208+0000 7f0b2bc00700 1 -- 172.21.15.62:0/1551301756 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b241045d0 con 0x7f0b24100e80 2024-04-20T22:16:44.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.209+0000 7f0b22000700 1 -- 172.21.15.62:0/1551301756 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f0b0c022870 con 0x7f0b24100e80 2024-04-20T22:16:44.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.210+0000 7f0b22000700 1 --2- 172.21.15.62:0/1551301756 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f0b0806ecc0 0x7f0b08071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:16:44.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.210+0000 7f0b22000700 1 -- 172.21.15.62:0/1551301756 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f0b0c090370 con 0x7f0b24100e80 2024-04-20T22:16:44.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.210+0000 7f0b29200700 1 --2- 172.21.15.62:0/1551301756 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f0b0806ecc0 0x7f0b08071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:16:44.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.211+0000 7f0b29200700 1 --2- 172.21.15.62:0/1551301756 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f0b0806ecc0 0x7f0b08071170 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f0b14005fd0 tx=0x7f0b1400a560 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:16:44.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.215+0000 7f0b22000700 1 -- 172.21.15.62:0/1551301756 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f0b0c0591b0 con 0x7f0b24100e80 2024-04-20T22:16:44.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.502+0000 7f0b2bc00700 1 -- 172.21.15.62:0/1551301756 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f0b24199530 con 0x7f0b24100e80 2024-04-20T22:16:44.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.502+0000 7f0b22000700 1 -- 172.21.15.62:0/1551301756 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 29 v29) v1 ==== 76+0+1887 (secure 0 0 0) 0x7f0b0c05c870 con 0x7f0b24100e80 2024-04-20T22:16:44.503 INFO:teuthology.orchestra.run.smithi062.stdout:e29 2024-04-20T22:16:44.503 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T22:16:44.503 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:16:44.503 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T22:16:44.503 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:16:44.503 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T22:16:44.503 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T22:16:44.503 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 20 2024-04-20T22:16:44.503 INFO:teuthology.orchestra.run.smithi062.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-20T22:16:44.504 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T22:16:44.504 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T21:42:01.559061+0000 2024-04-20T22:16:44.504 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T22:16:44.504 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T22:16:44.504 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T22:16:44.504 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T22:16:44.504 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T22:16:44.504 INFO:teuthology.orchestra.run.smithi062.stdout:max_xattr_size 65536 2024-04-20T22:16:44.504 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T22:16:44.504 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T22:16:44.504 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 81 2024-04-20T22:16:44.504 INFO:teuthology.orchestra.run.smithi062.stdout:snap_rstat 1 2024-04-20T22:16:44.504 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:16:44.504 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 2 2024-04-20T22:16:44.505 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1 2024-04-20T22:16:44.505 INFO:teuthology.orchestra.run.smithi062.stdout:up {} 2024-04-20T22:16:44.505 INFO:teuthology.orchestra.run.smithi062.stdout:failed 0,1 2024-04-20T22:16:44.505 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T22:16:44.505 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T22:16:44.505 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T22:16:44.505 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T22:16:44.505 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data enabled 2024-04-20T22:16:44.505 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T22:16:44.505 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T22:16:44.505 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 1 2024-04-20T22:16:44.505 INFO:teuthology.orchestra.run.smithi062.stdout:qdb_cluster leader: 0 members: 2024-04-20T22:16:44.505 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:16:44.505 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:16:44.505 INFO:teuthology.orchestra.run.smithi062.stdout:Standby daemons: 2024-04-20T22:16:44.506 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:16:44.506 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ryqblb{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6826/4113916205,v1:172.21.15.62:6827/4113916205] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:16:44.506 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ozizwi{-1:34476} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6828/3801975165,v1:172.21.15.62:6829/3801975165] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:16:44.506 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.dvimer{-1:44727} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6824/3576699232,v1:172.21.15.189:6825/3576699232] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:16:44.506 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.xbgspw{-1:45075} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6826/212905792,v1:172.21.15.189:6827/212905792] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:16:44.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.506+0000 7f0b2bc00700 1 -- 172.21.15.62:0/1551301756 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f0b0806ecc0 msgr2=0x7f0b08071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:16:44.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.506+0000 7f0b2bc00700 1 --2- 172.21.15.62:0/1551301756 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f0b0806ecc0 0x7f0b08071170 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f0b14005fd0 tx=0x7f0b1400a560 comp rx=0 tx=0).stop 2024-04-20T22:16:44.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.506+0000 7f0b2bc00700 1 -- 172.21.15.62:0/1551301756 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0b24100e80 msgr2=0x7f0b24198670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:16:44.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.506+0000 7f0b2bc00700 1 --2- 172.21.15.62:0/1551301756 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0b24100e80 0x7f0b24198670 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f0b0c00bda0 tx=0x7f0b0c00be80 comp rx=0 tx=0).stop 2024-04-20T22:16:44.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.507+0000 7f0b2bc00700 1 -- 172.21.15.62:0/1551301756 shutdown_connections 2024-04-20T22:16:44.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.507+0000 7f0b2bc00700 1 --2- 172.21.15.62:0/1551301756 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f0b0806ecc0 0x7f0b08071170 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:16:44.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.507+0000 7f0b2bc00700 1 --2- 172.21.15.62:0/1551301756 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b241004e0 0x7f0b24198130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:16:44.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.507+0000 7f0b2bc00700 1 --2- 172.21.15.62:0/1551301756 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0b24100e80 0x7f0b24198670 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:16:44.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.507+0000 7f0b2bc00700 1 -- 172.21.15.62:0/1551301756 >> 172.21.15.62:0/1551301756 conn(0x7f0b240fbda0 msgr2=0x7f0b240fe160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:16:44.508 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.507+0000 7f0b2bc00700 1 -- 172.21.15.62:0/1551301756 shutdown_connections 2024-04-20T22:16:44.508 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.507+0000 7f0b2bc00700 1 -- 172.21.15.62:0/1551301756 wait complete. 2024-04-20T22:16:44.508 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 29 2024-04-20T22:16:44.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.667+0000 7f9f91e00700 1 -- 172.21.15.62:0/2375660107 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9f8c106b80 msgr2=0x7f9f8c106ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:16:44.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.667+0000 7f9f91e00700 1 --2- 172.21.15.62:0/2375660107 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9f8c106b80 0x7f9f8c106ff0 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f9f78009910 tx=0x7f9f78009c20 comp rx=0 tx=0).stop 2024-04-20T22:16:44.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.668+0000 7f9f91e00700 1 -- 172.21.15.62:0/2375660107 shutdown_connections 2024-04-20T22:16:44.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.668+0000 7f9f91e00700 1 --2- 172.21.15.62:0/2375660107 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9f8c106b80 0x7f9f8c106ff0 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:16:44.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.668+0000 7f9f91e00700 1 --2- 172.21.15.62:0/2375660107 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f8c10c8f0 0x7f9f8c10ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:16:44.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.668+0000 7f9f91e00700 1 -- 172.21.15.62:0/2375660107 >> 172.21.15.62:0/2375660107 conn(0x7f9f8c076060 msgr2=0x7f9f8c078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:16:44.669 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:16:44 smithi062 ceph-mon[76322]: from='client.36190 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:16:44.669 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:16:44 smithi062 ceph-mon[76322]: from='client.36194 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:16:44.669 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:16:44 smithi062 ceph-mon[76322]: pgmap v1246: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:16:44.669 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:16:44 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/3459510735' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:16:44.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.668+0000 7f9f91e00700 1 -- 172.21.15.62:0/2375660107 shutdown_connections 2024-04-20T22:16:44.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.668+0000 7f9f91e00700 1 -- 172.21.15.62:0/2375660107 wait complete. 2024-04-20T22:16:44.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.669+0000 7f9f91e00700 1 Processor -- start 2024-04-20T22:16:44.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.669+0000 7f9f91e00700 1 -- start start 2024-04-20T22:16:44.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.670+0000 7f9f91e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9f8c106b80 0x7f9f8c19c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:16:44.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.670+0000 7f9f91e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f8c10c8f0 0x7f9f8c19c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:16:44.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.670+0000 7f9f91e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f8c19d060 con 0x7f9f8c106b80 2024-04-20T22:16:44.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.670+0000 7f9f91e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f8c1a0df0 con 0x7f9f8c10c8f0 2024-04-20T22:16:44.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.670+0000 7f9f8aa00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9f8c106b80 0x7f9f8c19c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:16:44.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.670+0000 7f9f8aa00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9f8c106b80 0x7f9f8c19c440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:36524/0 (socket says 172.21.15.62:36524) 2024-04-20T22:16:44.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.670+0000 7f9f8aa00700 1 -- 172.21.15.62:0/2810639796 learned_addr learned my addr 172.21.15.62:0/2810639796 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:16:44.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.670+0000 7f9f8a000700 1 --2- 172.21.15.62:0/2810639796 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f8c10c8f0 0x7f9f8c19c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:16:44.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.670+0000 7f9f8aa00700 1 -- 172.21.15.62:0/2810639796 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f8c10c8f0 msgr2=0x7f9f8c19c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:16:44.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.670+0000 7f9f8aa00700 1 --2- 172.21.15.62:0/2810639796 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f8c10c8f0 0x7f9f8c19c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:16:44.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.671+0000 7f9f8aa00700 1 -- 172.21.15.62:0/2810639796 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f780095f0 con 0x7f9f8c106b80 2024-04-20T22:16:44.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.671+0000 7f9f8aa00700 1 --2- 172.21.15.62:0/2810639796 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9f8c106b80 0x7f9f8c19c440 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f9f7c00c8a0 tx=0x7f9f7c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:16:44.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.671+0000 7f9f90a00700 1 -- 172.21.15.62:0/2810639796 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9f7c00cea0 con 0x7f9f8c106b80 2024-04-20T22:16:44.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.671+0000 7f9f91e00700 1 -- 172.21.15.62:0/2810639796 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f8c1a10d0 con 0x7f9f8c106b80 2024-04-20T22:16:44.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.671+0000 7f9f90a00700 1 -- 172.21.15.62:0/2810639796 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9f7c004830 con 0x7f9f8c106b80 2024-04-20T22:16:44.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.671+0000 7f9f90a00700 1 -- 172.21.15.62:0/2810639796 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9f7c0055b0 con 0x7f9f8c106b80 2024-04-20T22:16:44.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.671+0000 7f9f91e00700 1 -- 172.21.15.62:0/2810639796 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f8c1a1620 con 0x7f9f8c106b80 2024-04-20T22:16:44.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.673+0000 7f9f91e00700 1 -- 172.21.15.62:0/2810639796 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f8c10ecc0 con 0x7f9f8c106b80 2024-04-20T22:16:44.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.674+0000 7f9f90a00700 1 -- 172.21.15.62:0/2810639796 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f9f7c0049a0 con 0x7f9f8c106b80 2024-04-20T22:16:44.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.675+0000 7f9f90a00700 1 --2- 172.21.15.62:0/2810639796 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9f7006ecc0 0x7f9f70071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:16:44.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.675+0000 7f9f90a00700 1 -- 172.21.15.62:0/2810639796 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f9f7c08ebf0 con 0x7f9f8c106b80 2024-04-20T22:16:44.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.675+0000 7f9f8a000700 1 --2- 172.21.15.62:0/2810639796 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9f7006ecc0 0x7f9f70071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:16:44.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.676+0000 7f9f8a000700 1 --2- 172.21.15.62:0/2810639796 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9f7006ecc0 0x7f9f70071170 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f9f8c19da60 tx=0x7f9f7801a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:16:44.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.680+0000 7f9f90a00700 1 -- 172.21.15.62:0/2810639796 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f9f7c057900 con 0x7f9f8c106b80 2024-04-20T22:16:44.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:16:44 smithi189 ceph-mon[57022]: from='client.36190 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:16:44.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:16:44 smithi189 ceph-mon[57022]: from='client.36194 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:16:44.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:16:44 smithi189 ceph-mon[57022]: pgmap v1246: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:16:44.739 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:16:44 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/3459510735' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:16:44.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.937+0000 7f9f91e00700 1 -- 172.21.15.62:0/2810639796 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9f8c1a18a0 con 0x7f9f7006ecc0 2024-04-20T22:16:44.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.940+0000 7f9f90a00700 1 -- 172.21.15.62:0/2810639796 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f9f8c1a18a0 con 0x7f9f7006ecc0 2024-04-20T22:16:44.941 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:16:44.941 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T22:16:44.941 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T22:16:44.941 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-20T22:16:44.941 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T22:16:44.941 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr", 2024-04-20T22:16:44.941 INFO:teuthology.orchestra.run.smithi062.stdout: "osd", 2024-04-20T22:16:44.941 INFO:teuthology.orchestra.run.smithi062.stdout: "mds", 2024-04-20T22:16:44.941 INFO:teuthology.orchestra.run.smithi062.stdout: "crash", 2024-04-20T22:16:44.941 INFO:teuthology.orchestra.run.smithi062.stdout: "mon" 2024-04-20T22:16:44.941 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T22:16:44.942 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "16/23 daemons upgraded", 2024-04-20T22:16:44.942 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading mds daemons", 2024-04-20T22:16:44.942 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T22:16:44.942 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:16:44.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.943+0000 7f9f91e00700 1 -- 172.21.15.62:0/2810639796 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9f7006ecc0 msgr2=0x7f9f70071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:16:44.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.943+0000 7f9f91e00700 1 --2- 172.21.15.62:0/2810639796 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9f7006ecc0 0x7f9f70071170 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f9f8c19da60 tx=0x7f9f7801a040 comp rx=0 tx=0).stop 2024-04-20T22:16:44.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.943+0000 7f9f91e00700 1 -- 172.21.15.62:0/2810639796 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9f8c106b80 msgr2=0x7f9f8c19c440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:16:44.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.943+0000 7f9f91e00700 1 --2- 172.21.15.62:0/2810639796 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9f8c106b80 0x7f9f8c19c440 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f9f7c00c8a0 tx=0x7f9f7c00cc60 comp rx=0 tx=0).stop 2024-04-20T22:16:44.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.944+0000 7f9f91e00700 1 -- 172.21.15.62:0/2810639796 shutdown_connections 2024-04-20T22:16:44.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.944+0000 7f9f91e00700 1 --2- 172.21.15.62:0/2810639796 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9f7006ecc0 0x7f9f70071170 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:16:44.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.944+0000 7f9f91e00700 1 --2- 172.21.15.62:0/2810639796 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9f8c106b80 0x7f9f8c19c440 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:16:44.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.944+0000 7f9f91e00700 1 --2- 172.21.15.62:0/2810639796 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f8c10c8f0 0x7f9f8c19c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:16:44.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.944+0000 7f9f91e00700 1 -- 172.21.15.62:0/2810639796 >> 172.21.15.62:0/2810639796 conn(0x7f9f8c076060 msgr2=0x7f9f8c077ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:16:44.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.944+0000 7f9f91e00700 1 -- 172.21.15.62:0/2810639796 shutdown_connections 2024-04-20T22:16:44.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:44.944+0000 7f9f91e00700 1 -- 172.21.15.62:0/2810639796 wait complete. 2024-04-20T22:16:45.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:45.110+0000 7f6042800700 1 -- 172.21.15.62:0/2494094638 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f603c1003e0 msgr2=0x7f603c1007b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:16:45.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:45.110+0000 7f6042800700 1 --2- 172.21.15.62:0/2494094638 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f603c1003e0 0x7f603c1007b0 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f6030009910 tx=0x7f6030009c20 comp rx=0 tx=0).stop 2024-04-20T22:16:45.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:45.111+0000 7f6042800700 1 -- 172.21.15.62:0/2494094638 shutdown_connections 2024-04-20T22:16:45.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:45.111+0000 7f6042800700 1 --2- 172.21.15.62:0/2494094638 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f603c101c40 0x7f603c1020b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:16:45.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:45.111+0000 7f6042800700 1 --2- 172.21.15.62:0/2494094638 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f603c1003e0 0x7f603c1007b0 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:16:45.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:45.111+0000 7f6042800700 1 -- 172.21.15.62:0/2494094638 >> 172.21.15.62:0/2494094638 conn(0x7f603c0fbbc0 msgr2=0x7f603c0fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:16:45.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:45.111+0000 7f6042800700 1 -- 172.21.15.62:0/2494094638 shutdown_connections 2024-04-20T22:16:45.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:45.111+0000 7f6042800700 1 -- 172.21.15.62:0/2494094638 wait complete. 2024-04-20T22:16:45.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:45.112+0000 7f6042800700 1 Processor -- start 2024-04-20T22:16:45.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:45.112+0000 7f6042800700 1 -- start start 2024-04-20T22:16:45.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:45.112+0000 7f6042800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f603c1003e0 0x7f603c073050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:16:45.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:45.112+0000 7f6042800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f603c101c40 0x7f603c073590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:16:45.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:45.112+0000 7f6042800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f603c073ad0 con 0x7f603c101c40 2024-04-20T22:16:45.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:45.112+0000 7f6042800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f603c073c10 con 0x7f603c1003e0 2024-04-20T22:16:45.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:45.113+0000 7f6040a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f603c101c40 0x7f603c073590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:16:45.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:45.113+0000 7f6040a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f603c101c40 0x7f603c073590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:36548/0 (socket says 172.21.15.62:36548) 2024-04-20T22:16:45.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:45.113+0000 7f6040a00700 1 -- 172.21.15.62:0/2714588553 learned_addr learned my addr 172.21.15.62:0/2714588553 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:16:45.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:45.113+0000 7f6041400700 1 --2- 172.21.15.62:0/2714588553 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f603c1003e0 0x7f603c073050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:16:45.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:45.113+0000 7f6040a00700 1 -- 172.21.15.62:0/2714588553 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f603c1003e0 msgr2=0x7f603c073050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:16:45.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:45.113+0000 7f6040a00700 1 --2- 172.21.15.62:0/2714588553 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f603c1003e0 0x7f603c073050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:16:45.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:45.113+0000 7f6040a00700 1 -- 172.21.15.62:0/2714588553 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60300095f0 con 0x7f603c101c40 2024-04-20T22:16:45.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:45.114+0000 7f6040a00700 1 --2- 172.21.15.62:0/2714588553 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f603c101c40 0x7f603c073590 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f602800c930 tx=0x7f602800ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:16:45.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:45.114+0000 7f603a000700 1 -- 172.21.15.62:0/2714588553 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6028007ad0 con 0x7f603c101c40 2024-04-20T22:16:45.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:45.114+0000 7f603a000700 1 -- 172.21.15.62:0/2714588553 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6028007c30 con 0x7f603c101c40 2024-04-20T22:16:45.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:45.114+0000 7f6042800700 1 -- 172.21.15.62:0/2714588553 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f603c1a1ef0 con 0x7f603c101c40 2024-04-20T22:16:45.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:45.114+0000 7f603a000700 1 -- 172.21.15.62:0/2714588553 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6028018610 con 0x7f603c101c40 2024-04-20T22:16:45.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:45.114+0000 7f6042800700 1 -- 172.21.15.62:0/2714588553 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f603c1a2250 con 0x7f603c101c40 2024-04-20T22:16:45.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:45.115+0000 7f6042800700 1 -- 172.21.15.62:0/2714588553 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f603c062380 con 0x7f603c101c40 2024-04-20T22:16:45.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:45.117+0000 7f603a000700 1 -- 172.21.15.62:0/2714588553 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f602800f450 con 0x7f603c101c40 2024-04-20T22:16:45.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:45.118+0000 7f603a000700 1 --2- 172.21.15.62:0/2714588553 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f602406ecc0 0x7f6024071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:16:45.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:45.118+0000 7f603a000700 1 -- 172.21.15.62:0/2714588553 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f602808f1c0 con 0x7f603c101c40 2024-04-20T22:16:45.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:45.118+0000 7f6041400700 1 --2- 172.21.15.62:0/2714588553 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f602406ecc0 0x7f6024071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:16:45.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:45.119+0000 7f6041400700 1 --2- 172.21.15.62:0/2714588553 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f602406ecc0 0x7f6024071170 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f6030006010 tx=0x7f603001a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:16:45.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:45.122+0000 7f603a000700 1 -- 172.21.15.62:0/2714588553 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f6028057fd0 con 0x7f603c101c40 2024-04-20T22:16:45.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:45.447+0000 7f6042800700 1 -- 172.21.15.62:0/2714588553 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f603c074580 con 0x7f603c101c40 2024-04-20T22:16:45.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:45.448+0000 7f603a000700 1 -- 172.21.15.62:0/2714588553 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7f602805b690 con 0x7f603c101c40 2024-04-20T22:16:45.449 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-20T22:16:45.449 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-20T22:16:45.449 INFO:teuthology.orchestra.run.smithi062.stdout: host smithi189 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi189/config 2024-04-20T22:16:45.449 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-20T22:16:45.449 INFO:teuthology.orchestra.run.smithi062.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi189 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp11670v2q:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:16:45.449 INFO:teuthology.orchestra.run.smithi062.stdout:Traceback (most recent call last): 2024-04-20T22:16:45.449 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-20T22:16:45.449 INFO:teuthology.orchestra.run.smithi062.stdout: return _run_code(code, main_globals, None, 2024-04-20T22:16:45.449 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-20T22:16:45.449 INFO:teuthology.orchestra.run.smithi062.stdout: exec(code, run_globals) 2024-04-20T22:16:45.449 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5618, in 2024-04-20T22:16:45.449 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5606, in main 2024-04-20T22:16:45.449 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 408, in _infer_config 2024-04-20T22:16:45.450 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 323, in _infer_fsid 2024-04-20T22:16:45.450 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 436, in _infer_image 2024-04-20T22:16:45.450 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 310, in _validate_fsid 2024-04-20T22:16:45.450 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 3359, in command_ceph_volume 2024-04-20T22:16:45.450 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-20T22:16:45.450 INFO:teuthology.orchestra.run.smithi062.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi189 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp11670v2q:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:16:45.450 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-20T22:16:45.450 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is degraded 2024-04-20T22:16:45.450 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T22:16:45.450 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T22:16:45.450 INFO:teuthology.orchestra.run.smithi062.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-20T22:16:45.450 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-20T22:16:45.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:45.451+0000 7f6042800700 1 -- 172.21.15.62:0/2714588553 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f602406ecc0 msgr2=0x7f6024071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:16:45.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:45.451+0000 7f6042800700 1 --2- 172.21.15.62:0/2714588553 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f602406ecc0 0x7f6024071170 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f6030006010 tx=0x7f603001a040 comp rx=0 tx=0).stop 2024-04-20T22:16:45.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:45.451+0000 7f6042800700 1 -- 172.21.15.62:0/2714588553 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f603c101c40 msgr2=0x7f603c073590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:16:45.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:45.451+0000 7f6042800700 1 --2- 172.21.15.62:0/2714588553 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f603c101c40 0x7f603c073590 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f602800c930 tx=0x7f602800ccf0 comp rx=0 tx=0).stop 2024-04-20T22:16:45.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:45.451+0000 7f6042800700 1 -- 172.21.15.62:0/2714588553 shutdown_connections 2024-04-20T22:16:45.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:45.451+0000 7f6042800700 1 --2- 172.21.15.62:0/2714588553 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f602406ecc0 0x7f6024071170 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:16:45.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:45.451+0000 7f6042800700 1 --2- 172.21.15.62:0/2714588553 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f603c1003e0 0x7f603c073050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:16:45.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:45.451+0000 7f6042800700 1 --2- 172.21.15.62:0/2714588553 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f603c101c40 0x7f603c073590 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:16:45.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:45.451+0000 7f6042800700 1 -- 172.21.15.62:0/2714588553 >> 172.21.15.62:0/2714588553 conn(0x7f603c0fbbc0 msgr2=0x7f603c0fc420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:16:45.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:45.451+0000 7f6042800700 1 -- 172.21.15.62:0/2714588553 shutdown_connections 2024-04-20T22:16:45.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:16:45.452+0000 7f6042800700 1 -- 172.21.15.62:0/2714588553 wait complete. 2024-04-20T22:16:45.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:16:45 smithi189 ceph-mon[57022]: from='client.36198 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:16:45.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:16:45 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/1551301756' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:16:45.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:16:45 smithi062 ceph-mon[76322]: from='client.36198 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:16:45.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:16:45 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/1551301756' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:16:46.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:16:46 smithi189 ceph-mon[57022]: from='client.36210 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:16:46.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:16:46 smithi189 ceph-mon[57022]: pgmap v1247: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:16:46.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:16:46 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/2714588553' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:16:46.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:16:46 smithi062 ceph-mon[76322]: from='client.36210 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:16:46.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:16:46 smithi062 ceph-mon[76322]: pgmap v1247: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:16:46.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:16:46 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/2714588553' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:16:47.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:16:47 smithi189 ceph-mon[57022]: pgmap v1248: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:16:47.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:16:47 smithi062 ceph-mon[76322]: pgmap v1248: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:16:50.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:16:50 smithi189 ceph-mon[57022]: pgmap v1249: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:16:50.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:16:50 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:16:50.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:16:50 smithi062 ceph-mon[76322]: pgmap v1249: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:16:50.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:16:50 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:16:52.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:16:52 smithi189 ceph-mon[57022]: pgmap v1250: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:16:52.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:16:52 smithi062 ceph-mon[76322]: pgmap v1250: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:16:54.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:16:54 smithi189 ceph-mon[57022]: pgmap v1251: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:16:54.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:16:54 smithi062 ceph-mon[76322]: pgmap v1251: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:16:56.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:16:56 smithi189 ceph-mon[57022]: pgmap v1252: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:16:56.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:16:56 smithi062 ceph-mon[76322]: pgmap v1252: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:16:58.439 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:16:58 smithi062 ceph-mon[76322]: pgmap v1253: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:16:58.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:16:58 smithi189 ceph-mon[57022]: pgmap v1253: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:17:00.740 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:17:00 smithi189 ceph-mon[57022]: pgmap v1254: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:17:00.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:17:00 smithi062 ceph-mon[76322]: pgmap v1254: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:17:02.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:17:02 smithi189 ceph-mon[57022]: pgmap v1255: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:17:02.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:17:02 smithi062 ceph-mon[76322]: pgmap v1255: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:17:04.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:17:04 smithi189 ceph-mon[57022]: pgmap v1256: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:17:04.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:17:04 smithi062 ceph-mon[76322]: pgmap v1256: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:17:05.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:17:05 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:17:05.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:17:05 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:17:06.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:17:06 smithi189 ceph-mon[57022]: pgmap v1257: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:17:06.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:17:06 smithi062 ceph-mon[76322]: pgmap v1257: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:17:07.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:17:07 smithi189 ceph-mon[57022]: pgmap v1258: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:17:07.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:17:07 smithi062 ceph-mon[76322]: pgmap v1258: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:17:10.698 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:17:10 smithi189 ceph-mon[57022]: pgmap v1259: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:17:10.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:17:10 smithi062 ceph-mon[76322]: pgmap v1259: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:17:12.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:17:12 smithi189 ceph-mon[57022]: pgmap v1260: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:17:12.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:17:12 smithi062 ceph-mon[76322]: pgmap v1260: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:17:14.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:17:14 smithi189 ceph-mon[57022]: pgmap v1261: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:17:14.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:17:14 smithi062 ceph-mon[76322]: pgmap v1261: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:17:15.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:15.632+0000 7f2485c00700 1 -- 172.21.15.62:0/1258843714 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f24800730f0 msgr2=0x7f24800734c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:17:15.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:15.632+0000 7f2485c00700 1 --2- 172.21.15.62:0/1258843714 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f24800730f0 0x7f24800734c0 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f2470009910 tx=0x7f2470009c20 comp rx=0 tx=0).stop 2024-04-20T22:17:15.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:15.632+0000 7f2485c00700 1 -- 172.21.15.62:0/1258843714 shutdown_connections 2024-04-20T22:17:15.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:15.632+0000 7f2485c00700 1 --2- 172.21.15.62:0/1258843714 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2480073a00 0x7f2480110d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:17:15.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:15.632+0000 7f2485c00700 1 --2- 172.21.15.62:0/1258843714 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f24800730f0 0x7f24800734c0 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:17:15.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:15.632+0000 7f2485c00700 1 -- 172.21.15.62:0/1258843714 >> 172.21.15.62:0/1258843714 conn(0x7f24800fbda0 msgr2=0x7f24800fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:17:15.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:15.632+0000 7f2485c00700 1 -- 172.21.15.62:0/1258843714 shutdown_connections 2024-04-20T22:17:15.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:15.632+0000 7f2485c00700 1 -- 172.21.15.62:0/1258843714 wait complete. 2024-04-20T22:17:15.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:15.634+0000 7f2485c00700 1 Processor -- start 2024-04-20T22:17:15.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:15.634+0000 7f2485c00700 1 -- start start 2024-04-20T22:17:15.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:15.634+0000 7f2485c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f24800730f0 0x7f248019c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:17:15.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:15.634+0000 7f2485c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2480073a00 0x7f248019c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:17:15.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:15.634+0000 7f2485c00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f248019d000 con 0x7f2480073a00 2024-04-20T22:17:15.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:15.634+0000 7f2485c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24801a0d90 con 0x7f24800730f0 2024-04-20T22:17:15.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:15.635+0000 7f2476000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2480073a00 0x7f248019c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:17:15.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:15.635+0000 7f2476000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2480073a00 0x7f248019c920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:32852/0 (socket says 172.21.15.62:32852) 2024-04-20T22:17:15.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:15.635+0000 7f2476000700 1 -- 172.21.15.62:0/1811072379 learned_addr learned my addr 172.21.15.62:0/1811072379 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:17:15.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:15.635+0000 7f247ea00700 1 --2- 172.21.15.62:0/1811072379 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f24800730f0 0x7f248019c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:17:15.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:15.635+0000 7f2476000700 1 -- 172.21.15.62:0/1811072379 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f24800730f0 msgr2=0x7f248019c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:17:15.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:15.635+0000 7f2476000700 1 --2- 172.21.15.62:0/1811072379 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f24800730f0 0x7f248019c3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:17:15.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:15.635+0000 7f2476000700 1 -- 172.21.15.62:0/1811072379 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24700095f0 con 0x7f2480073a00 2024-04-20T22:17:15.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:15.635+0000 7f2476000700 1 --2- 172.21.15.62:0/1811072379 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2480073a00 0x7f248019c920 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f246800d8d0 tx=0x7f246800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:17:15.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:15.636+0000 7f2477e00700 1 -- 172.21.15.62:0/1811072379 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2468009940 con 0x7f2480073a00 2024-04-20T22:17:15.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:15.636+0000 7f2477e00700 1 -- 172.21.15.62:0/1811072379 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2468010460 con 0x7f2480073a00 2024-04-20T22:17:15.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:15.636+0000 7f2477e00700 1 -- 172.21.15.62:0/1811072379 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f246800f5f0 con 0x7f2480073a00 2024-04-20T22:17:15.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:15.636+0000 7f2485c00700 1 -- 172.21.15.62:0/1811072379 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24801a1070 con 0x7f2480073a00 2024-04-20T22:17:15.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:15.636+0000 7f2485c00700 1 -- 172.21.15.62:0/1811072379 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24801a15c0 con 0x7f2480073a00 2024-04-20T22:17:15.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:15.637+0000 7f2485c00700 1 -- 172.21.15.62:0/1811072379 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f248010e4d0 con 0x7f2480073a00 2024-04-20T22:17:15.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:15.638+0000 7f2477e00700 1 -- 172.21.15.62:0/1811072379 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f24680105d0 con 0x7f2480073a00 2024-04-20T22:17:15.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:15.639+0000 7f2477e00700 1 --2- 172.21.15.62:0/1811072379 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f246406ecc0 0x7f2464071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:17:15.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:15.639+0000 7f2477e00700 1 -- 172.21.15.62:0/1811072379 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f246808ed70 con 0x7f2480073a00 2024-04-20T22:17:15.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:15.640+0000 7f247ea00700 1 --2- 172.21.15.62:0/1811072379 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f246406ecc0 0x7f2464071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:17:15.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:15.641+0000 7f247ea00700 1 --2- 172.21.15.62:0/1811072379 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f246406ecc0 0x7f2464071170 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f2470006010 tx=0x7f247001a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:17:15.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:15.645+0000 7f2477e00700 1 -- 172.21.15.62:0/1811072379 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f2468057ba0 con 0x7f2480073a00 2024-04-20T22:17:15.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:15.903+0000 7f2485c00700 1 -- 172.21.15.62:0/1811072379 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f24801a18a0 con 0x7f246406ecc0 2024-04-20T22:17:15.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:15.906+0000 7f2477e00700 1 -- 172.21.15.62:0/1811072379 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f24801a18a0 con 0x7f246406ecc0 2024-04-20T22:17:15.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:15.909+0000 7f2485c00700 1 -- 172.21.15.62:0/1811072379 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f246406ecc0 msgr2=0x7f2464071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:17:15.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:15.909+0000 7f2485c00700 1 --2- 172.21.15.62:0/1811072379 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f246406ecc0 0x7f2464071170 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f2470006010 tx=0x7f247001a040 comp rx=0 tx=0).stop 2024-04-20T22:17:15.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:15.909+0000 7f2485c00700 1 -- 172.21.15.62:0/1811072379 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2480073a00 msgr2=0x7f248019c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:17:15.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:15.909+0000 7f2485c00700 1 --2- 172.21.15.62:0/1811072379 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2480073a00 0x7f248019c920 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f246800d8d0 tx=0x7f246800dc90 comp rx=0 tx=0).stop 2024-04-20T22:17:15.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:15.909+0000 7f2485c00700 1 -- 172.21.15.62:0/1811072379 shutdown_connections 2024-04-20T22:17:15.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:15.909+0000 7f2485c00700 1 --2- 172.21.15.62:0/1811072379 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f246406ecc0 0x7f2464071170 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:17:15.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:15.909+0000 7f2485c00700 1 --2- 172.21.15.62:0/1811072379 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f24800730f0 0x7f248019c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:17:15.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:15.909+0000 7f2485c00700 1 --2- 172.21.15.62:0/1811072379 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2480073a00 0x7f248019c920 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:17:15.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:15.909+0000 7f2485c00700 1 -- 172.21.15.62:0/1811072379 >> 172.21.15.62:0/1811072379 conn(0x7f24800fbda0 msgr2=0x7f2480102860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:17:15.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:15.909+0000 7f2485c00700 1 -- 172.21.15.62:0/1811072379 shutdown_connections 2024-04-20T22:17:15.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:15.909+0000 7f2485c00700 1 -- 172.21.15.62:0/1811072379 wait complete. 2024-04-20T22:17:15.925 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T22:17:16.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:16.119+0000 7fc9bba00700 1 -- 172.21.15.62:0/610854694 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9b4100e80 msgr2=0x7fc9b4104e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:17:16.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:16.119+0000 7fc9bba00700 1 --2- 172.21.15.62:0/610854694 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9b4100e80 0x7fc9b4104e50 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7fc99c009910 tx=0x7fc99c009c20 comp rx=0 tx=0).stop 2024-04-20T22:17:16.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:16.119+0000 7fc9bba00700 1 -- 172.21.15.62:0/610854694 shutdown_connections 2024-04-20T22:17:16.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:16.119+0000 7fc9bba00700 1 --2- 172.21.15.62:0/610854694 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9b4100e80 0x7fc9b4104e50 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:17:16.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:16.119+0000 7fc9bba00700 1 --2- 172.21.15.62:0/610854694 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc9b41004e0 0x7fc9b41008b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:17:16.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:16.119+0000 7fc9bba00700 1 -- 172.21.15.62:0/610854694 >> 172.21.15.62:0/610854694 conn(0x7fc9b40fbda0 msgr2=0x7fc9b40fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:17:16.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:16.120+0000 7fc9bba00700 1 -- 172.21.15.62:0/610854694 shutdown_connections 2024-04-20T22:17:16.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:16.120+0000 7fc9bba00700 1 -- 172.21.15.62:0/610854694 wait complete. 2024-04-20T22:17:16.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:16.121+0000 7fc9bba00700 1 Processor -- start 2024-04-20T22:17:16.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:16.121+0000 7fc9bba00700 1 -- start start 2024-04-20T22:17:16.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:16.122+0000 7fc9bba00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc9b41004e0 0x7fc9b41981c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:17:16.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:16.122+0000 7fc9bba00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9b4100e80 0x7fc9b4198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:17:16.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:16.122+0000 7fc9bba00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9b4198de0 con 0x7fc9b4100e80 2024-04-20T22:17:16.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:16.122+0000 7fc9bba00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9b419cb70 con 0x7fc9b41004e0 2024-04-20T22:17:16.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:16.122+0000 7fc9abe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9b4100e80 0x7fc9b4198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:17:16.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:16.122+0000 7fc9abe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9b4100e80 0x7fc9b4198700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:32856/0 (socket says 172.21.15.62:32856) 2024-04-20T22:17:16.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:16.122+0000 7fc9abe00700 1 -- 172.21.15.62:0/2927036938 learned_addr learned my addr 172.21.15.62:0/2927036938 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:17:16.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:16.122+0000 7fc9b9000700 1 --2- 172.21.15.62:0/2927036938 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc9b41004e0 0x7fc9b41981c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:17:16.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:16.122+0000 7fc9abe00700 1 -- 172.21.15.62:0/2927036938 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc9b41004e0 msgr2=0x7fc9b41981c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:17:16.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:16.122+0000 7fc9abe00700 1 --2- 172.21.15.62:0/2927036938 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc9b41004e0 0x7fc9b41981c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:17:16.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:16.122+0000 7fc9abe00700 1 -- 172.21.15.62:0/2927036938 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc99c0095f0 con 0x7fc9b4100e80 2024-04-20T22:17:16.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:16.123+0000 7fc9abe00700 1 --2- 172.21.15.62:0/2927036938 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9b4100e80 0x7fc9b4198700 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7fc99c00bda0 tx=0x7fc99c00be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:17:16.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:16.123+0000 7fc9b2000700 1 -- 172.21.15.62:0/2927036938 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc99c01d070 con 0x7fc9b4100e80 2024-04-20T22:17:16.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:16.123+0000 7fc9b2000700 1 -- 172.21.15.62:0/2927036938 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc99c004fb0 con 0x7fc9b4100e80 2024-04-20T22:17:16.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:16.123+0000 7fc9b2000700 1 -- 172.21.15.62:0/2927036938 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc99c00f460 con 0x7fc9b4100e80 2024-04-20T22:17:16.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:16.123+0000 7fc9bba00700 1 -- 172.21.15.62:0/2927036938 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9b419cdf0 con 0x7fc9b4100e80 2024-04-20T22:17:16.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:16.123+0000 7fc9bba00700 1 -- 172.21.15.62:0/2927036938 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9b419d2e0 con 0x7fc9b4100e80 2024-04-20T22:17:16.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:16.124+0000 7fc9bba00700 1 -- 172.21.15.62:0/2927036938 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9b41045d0 con 0x7fc9b4100e80 2024-04-20T22:17:16.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:16.125+0000 7fc9b2000700 1 -- 172.21.15.62:0/2927036938 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fc99c022800 con 0x7fc9b4100e80 2024-04-20T22:17:16.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:16.126+0000 7fc9b2000700 1 --2- 172.21.15.62:0/2927036938 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc99806ecc0 0x7fc998071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:17:16.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:16.127+0000 7fc9b2000700 1 -- 172.21.15.62:0/2927036938 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fc99c090440 con 0x7fc9b4100e80 2024-04-20T22:17:16.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:16.127+0000 7fc9b9000700 1 --2- 172.21.15.62:0/2927036938 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc99806ecc0 0x7fc998071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:17:16.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:16.128+0000 7fc9b9000700 1 --2- 172.21.15.62:0/2927036938 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc99806ecc0 0x7fc998071170 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fc9a4006fd0 tx=0x7fc9a4008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:17:16.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:16.131+0000 7fc9b2000700 1 -- 172.21.15.62:0/2927036938 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fc99c059320 con 0x7fc9b4100e80 2024-04-20T22:17:16.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:16.393+0000 7fc9bba00700 1 -- 172.21.15.62:0/2927036938 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc9b419d500 con 0x7fc99806ecc0 2024-04-20T22:17:16.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:16.395+0000 7fc9b2000700 1 -- 172.21.15.62:0/2927036938 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fc9b419d500 con 0x7fc99806ecc0 2024-04-20T22:17:16.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:16.398+0000 7fc9bba00700 1 -- 172.21.15.62:0/2927036938 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc99806ecc0 msgr2=0x7fc998071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:17:16.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:16.398+0000 7fc9bba00700 1 --2- 172.21.15.62:0/2927036938 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc99806ecc0 0x7fc998071170 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fc9a4006fd0 tx=0x7fc9a4008040 comp rx=0 tx=0).stop 2024-04-20T22:17:16.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:16.398+0000 7fc9bba00700 1 -- 172.21.15.62:0/2927036938 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9b4100e80 msgr2=0x7fc9b4198700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:17:16.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:16.398+0000 7fc9bba00700 1 --2- 172.21.15.62:0/2927036938 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9b4100e80 0x7fc9b4198700 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7fc99c00bda0 tx=0x7fc99c00be80 comp rx=0 tx=0).stop 2024-04-20T22:17:16.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:16.399+0000 7fc9bba00700 1 -- 172.21.15.62:0/2927036938 shutdown_connections 2024-04-20T22:17:16.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:16.399+0000 7fc9bba00700 1 --2- 172.21.15.62:0/2927036938 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc99806ecc0 0x7fc998071170 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:17:16.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:16.399+0000 7fc9bba00700 1 --2- 172.21.15.62:0/2927036938 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc9b41004e0 0x7fc9b41981c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:17:16.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:16.399+0000 7fc9bba00700 1 --2- 172.21.15.62:0/2927036938 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9b4100e80 0x7fc9b4198700 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:17:16.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:16.399+0000 7fc9bba00700 1 -- 172.21.15.62:0/2927036938 >> 172.21.15.62:0/2927036938 conn(0x7fc9b40fbda0 msgr2=0x7fc9b40fd4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:17:16.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:16.399+0000 7fc9bba00700 1 -- 172.21.15.62:0/2927036938 shutdown_connections 2024-04-20T22:17:16.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:16.399+0000 7fc9bba00700 1 -- 172.21.15.62:0/2927036938 wait complete. 2024-04-20T22:17:16.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:16.569+0000 7f9458200700 1 -- 172.21.15.62:0/2678005570 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9450075280 msgr2=0x7f9450075650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:17:16.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:16.569+0000 7f9458200700 1 --2- 172.21.15.62:0/2678005570 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9450075280 0x7f9450075650 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f9444009910 tx=0x7f9444009c20 comp rx=0 tx=0).stop 2024-04-20T22:17:16.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:16.569+0000 7f9458200700 1 -- 172.21.15.62:0/2678005570 shutdown_connections 2024-04-20T22:17:16.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:16.569+0000 7f9458200700 1 --2- 172.21.15.62:0/2678005570 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9450075b90 0x7f9450110bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:17:16.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:16.569+0000 7f9458200700 1 --2- 172.21.15.62:0/2678005570 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9450075280 0x7f9450075650 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:17:16.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:16.569+0000 7f9458200700 1 -- 172.21.15.62:0/2678005570 >> 172.21.15.62:0/2678005570 conn(0x7f94500fdf90 msgr2=0x7f94501003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:17:16.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:16.570+0000 7f9458200700 1 -- 172.21.15.62:0/2678005570 shutdown_connections 2024-04-20T22:17:16.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:16.570+0000 7f9458200700 1 -- 172.21.15.62:0/2678005570 wait complete. 2024-04-20T22:17:16.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:16.571+0000 7f9458200700 1 Processor -- start 2024-04-20T22:17:16.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:16.571+0000 7f9458200700 1 -- start start 2024-04-20T22:17:16.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:16.571+0000 7f9458200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9450075280 0x7f945019c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:17:16.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:16.572+0000 7f9458200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9450075b90 0x7f945019c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:17:16.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:16.572+0000 7f9458200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f945019cfc0 con 0x7f9450075280 2024-04-20T22:17:16.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:16.572+0000 7f9458200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f945019d100 con 0x7f9450075b90 2024-04-20T22:17:16.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:16.572+0000 7f9455800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9450075280 0x7f945019c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:17:16.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:16.572+0000 7f9455800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9450075280 0x7f945019c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:32876/0 (socket says 172.21.15.62:32876) 2024-04-20T22:17:16.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:16.572+0000 7f9455800700 1 -- 172.21.15.62:0/2728821120 learned_addr learned my addr 172.21.15.62:0/2728821120 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:17:16.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:16.572+0000 7f9454e00700 1 --2- 172.21.15.62:0/2728821120 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9450075b90 0x7f945019c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:17:16.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:16.572+0000 7f9455800700 1 -- 172.21.15.62:0/2728821120 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9450075b90 msgr2=0x7f945019c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:17:16.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:16.572+0000 7f9455800700 1 --2- 172.21.15.62:0/2728821120 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9450075b90 0x7f945019c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:17:16.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:16.572+0000 7f9455800700 1 -- 172.21.15.62:0/2728821120 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94440095f0 con 0x7f9450075280 2024-04-20T22:17:16.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:16.573+0000 7f9455800700 1 --2- 172.21.15.62:0/2728821120 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9450075280 0x7f945019c430 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f9444006010 tx=0x7f944400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:17:16.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:16.573+0000 7f944e000700 1 -- 172.21.15.62:0/2728821120 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f944401d070 con 0x7f9450075280 2024-04-20T22:17:16.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:16.573+0000 7f944e000700 1 -- 172.21.15.62:0/2728821120 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9444004fb0 con 0x7f9450075280 2024-04-20T22:17:16.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:16.573+0000 7f944e000700 1 -- 172.21.15.62:0/2728821120 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f944400f690 con 0x7f9450075280 2024-04-20T22:17:16.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:16.573+0000 7f9458200700 1 -- 172.21.15.62:0/2728821120 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94501a0ef0 con 0x7f9450075280 2024-04-20T22:17:16.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:16.574+0000 7f9458200700 1 -- 172.21.15.62:0/2728821120 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94501a1440 con 0x7f9450075280 2024-04-20T22:17:16.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:16.575+0000 7f9458200700 1 -- 172.21.15.62:0/2728821120 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f945010e340 con 0x7f9450075280 2024-04-20T22:17:16.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:16.576+0000 7f944e000700 1 -- 172.21.15.62:0/2728821120 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f94440228a0 con 0x7f9450075280 2024-04-20T22:17:16.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:16.577+0000 7f944e000700 1 --2- 172.21.15.62:0/2728821120 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f943806ecc0 0x7f9438071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:17:16.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:16.577+0000 7f944e000700 1 -- 172.21.15.62:0/2728821120 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f9444090630 con 0x7f9450075280 2024-04-20T22:17:16.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:16.577+0000 7f9454e00700 1 --2- 172.21.15.62:0/2728821120 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f943806ecc0 0x7f9438071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:17:16.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:16.578+0000 7f9454e00700 1 --2- 172.21.15.62:0/2728821120 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f943806ecc0 0x7f9438071170 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f943c005ea0 tx=0x7f943c005e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:17:16.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:16.582+0000 7f944e000700 1 -- 172.21.15.62:0/2728821120 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f94440593c0 con 0x7f9450075280 2024-04-20T22:17:16.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:17:16 smithi189 ceph-mon[57022]: pgmap v1262: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:17:16.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:17:16 smithi062 ceph-mon[76322]: pgmap v1262: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:17:16.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:16.829+0000 7f9458200700 1 -- 172.21.15.62:0/2728821120 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f945002cca0 con 0x7f943806ecc0 2024-04-20T22:17:16.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:16.842+0000 7f944e000700 1 -- 172.21.15.62:0/2728821120 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f945002cca0 con 0x7f943806ecc0 2024-04-20T22:17:16.842 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T22:17:16.842 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (43m) 3m ago 56m 14.6M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T22:17:16.842 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (57m) 3m ago 57m 11.6M - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T22:17:16.842 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (55m) 3m ago 55m 11.2M - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T22:17:16.842 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (39m) 3m ago 57m 6660k - 19.0.0-3133-ga8c84f59 76bc6b91b726 2825d5ba11c6 2024-04-20T22:17:16.842 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (39m) 3m ago 55m 6665k - 19.0.0-3133-ga8c84f59 76bc6b91b726 20f746fa1bba 2024-04-20T22:17:16.842 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (42m) 3m ago 56m 75.5M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T22:17:16.842 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (34m) 3m ago 49m 22.9M - 19.0.0-3133-ga8c84f59 76bc6b91b726 de80bc90a683 2024-04-20T22:17:16.843 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (35m) 3m ago 49m 23.0M - 19.0.0-3133-ga8c84f59 76bc6b91b726 cd793b7ee98e 2024-04-20T22:17:16.843 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (19m) 3m ago 49m 18.5M - 19.0.0-3133-ga8c84f59 76bc6b91b726 0e7b0bc4d1ab 2024-04-20T22:17:16.843 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (4m) 3m ago 49m 11.4M - 19.0.0-3133-ga8c84f59 76bc6b91b726 0af732ac9e06 2024-04-20T22:17:16.843 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (46m) 3m ago 58m 517M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T22:17:16.843 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (44m) 3m ago 55m 460M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T22:17:16.843 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (40m) 3m ago 58m 86.3M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 019eb1479485 2024-04-20T22:17:16.843 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (40m) 3m ago 55m 58.0M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 55aac560aab6 2024-04-20T22:17:16.843 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (44m) 3m ago 57m 8766k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T22:17:16.843 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (44m) 3m ago 55m 8711k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T22:17:16.843 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (38m) 3m ago 54m 91.3M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 ee2bb901f26e 2024-04-20T22:17:16.843 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (38m) 3m ago 54m 94.9M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 9075152d4162 2024-04-20T22:17:16.843 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (37m) 3m ago 53m 78.5M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 2d4f3e38dc0e 2024-04-20T22:17:16.843 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (36m) 3m ago 52m 94.1M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 8f97b17ae0eb 2024-04-20T22:17:16.843 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (36m) 3m ago 52m 96.1M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 653d8079967d 2024-04-20T22:17:16.843 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (35m) 3m ago 51m 90.9M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 596dded14c12 2024-04-20T22:17:16.843 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (43m) 3m ago 56m 90.4M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T22:17:16.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:16.845+0000 7f9458200700 1 -- 172.21.15.62:0/2728821120 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f943806ecc0 msgr2=0x7f9438071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:17:16.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:16.845+0000 7f9458200700 1 --2- 172.21.15.62:0/2728821120 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f943806ecc0 0x7f9438071170 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f943c005ea0 tx=0x7f943c005e30 comp rx=0 tx=0).stop 2024-04-20T22:17:16.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:16.845+0000 7f9458200700 1 -- 172.21.15.62:0/2728821120 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9450075280 msgr2=0x7f945019c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:17:16.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:16.845+0000 7f9458200700 1 --2- 172.21.15.62:0/2728821120 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9450075280 0x7f945019c430 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f9444006010 tx=0x7f944400be20 comp rx=0 tx=0).stop 2024-04-20T22:17:16.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:16.845+0000 7f9458200700 1 -- 172.21.15.62:0/2728821120 shutdown_connections 2024-04-20T22:17:16.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:16.845+0000 7f9458200700 1 --2- 172.21.15.62:0/2728821120 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f943806ecc0 0x7f9438071170 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:17:16.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:16.845+0000 7f9458200700 1 --2- 172.21.15.62:0/2728821120 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9450075280 0x7f945019c430 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:17:16.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:16.845+0000 7f9458200700 1 --2- 172.21.15.62:0/2728821120 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9450075b90 0x7f945019c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:17:16.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:16.845+0000 7f9458200700 1 -- 172.21.15.62:0/2728821120 >> 172.21.15.62:0/2728821120 conn(0x7f94500fdf90 msgr2=0x7f94500ffc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:17:16.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:16.846+0000 7f9458200700 1 -- 172.21.15.62:0/2728821120 shutdown_connections 2024-04-20T22:17:16.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:16.846+0000 7f9458200700 1 -- 172.21.15.62:0/2728821120 wait complete. 2024-04-20T22:17:17.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:17.010+0000 7f3593600700 1 -- 172.21.15.62:0/3922630972 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f358c10c8f0 msgr2=0x7f358c10ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:17:17.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:17.010+0000 7f3593600700 1 --2- 172.21.15.62:0/3922630972 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f358c10c8f0 0x7f358c10ccc0 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f3580009910 tx=0x7f3580009c20 comp rx=0 tx=0).stop 2024-04-20T22:17:17.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:17.011+0000 7f3593600700 1 -- 172.21.15.62:0/3922630972 shutdown_connections 2024-04-20T22:17:17.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:17.011+0000 7f3593600700 1 --2- 172.21.15.62:0/3922630972 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f358c106b80 0x7f358c106ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:17:17.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:17.011+0000 7f3593600700 1 --2- 172.21.15.62:0/3922630972 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f358c10c8f0 0x7f358c10ccc0 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:17:17.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:17.011+0000 7f3593600700 1 -- 172.21.15.62:0/3922630972 >> 172.21.15.62:0/3922630972 conn(0x7f358c076060 msgr2=0x7f358c078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:17:17.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:17.011+0000 7f3593600700 1 -- 172.21.15.62:0/3922630972 shutdown_connections 2024-04-20T22:17:17.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:17.011+0000 7f3593600700 1 -- 172.21.15.62:0/3922630972 wait complete. 2024-04-20T22:17:17.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:17.012+0000 7f3593600700 1 Processor -- start 2024-04-20T22:17:17.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:17.012+0000 7f3593600700 1 -- start start 2024-04-20T22:17:17.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:17.013+0000 7f3593600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f358c106b80 0x7f358c19c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:17:17.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:17.013+0000 7f3593600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f358c10c8f0 0x7f358c19ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:17:17.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:17.013+0000 7f3593600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f358c19d120 con 0x7f358c10c8f0 2024-04-20T22:17:17.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:17.013+0000 7f3593600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f358c1a0eb0 con 0x7f358c106b80 2024-04-20T22:17:17.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:17.013+0000 7f358be00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f358c10c8f0 0x7f358c19ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:17:17.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:17.013+0000 7f358be00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f358c10c8f0 0x7f358c19ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:32890/0 (socket says 172.21.15.62:32890) 2024-04-20T22:17:17.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:17.013+0000 7f358be00700 1 -- 172.21.15.62:0/774363725 learned_addr learned my addr 172.21.15.62:0/774363725 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:17:17.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:17.013+0000 7f3590c00700 1 --2- 172.21.15.62:0/774363725 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f358c106b80 0x7f358c19c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:17:17.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:17.013+0000 7f358be00700 1 -- 172.21.15.62:0/774363725 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f358c106b80 msgr2=0x7f358c19c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:17:17.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:17.013+0000 7f358be00700 1 --2- 172.21.15.62:0/774363725 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f358c106b80 0x7f358c19c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:17:17.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:17.013+0000 7f358be00700 1 -- 172.21.15.62:0/774363725 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35800095f0 con 0x7f358c10c8f0 2024-04-20T22:17:17.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:17.014+0000 7f358be00700 1 --2- 172.21.15.62:0/774363725 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f358c10c8f0 0x7f358c19ca40 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f357800d8d0 tx=0x7f357800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:17:17.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:17.014+0000 7f3589600700 1 -- 172.21.15.62:0/774363725 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3578009940 con 0x7f358c10c8f0 2024-04-20T22:17:17.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:17.014+0000 7f3589600700 1 -- 172.21.15.62:0/774363725 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3578010460 con 0x7f358c10c8f0 2024-04-20T22:17:17.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:17.014+0000 7f3593600700 1 -- 172.21.15.62:0/774363725 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f358c1a1190 con 0x7f358c10c8f0 2024-04-20T22:17:17.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:17.014+0000 7f3589600700 1 -- 172.21.15.62:0/774363725 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f357800f5f0 con 0x7f358c10c8f0 2024-04-20T22:17:17.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:17.014+0000 7f3593600700 1 -- 172.21.15.62:0/774363725 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f358c1a16e0 con 0x7f358c10c8f0 2024-04-20T22:17:17.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:17.015+0000 7f3593600700 1 -- 172.21.15.62:0/774363725 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f358c109b20 con 0x7f358c10c8f0 2024-04-20T22:17:17.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:17.017+0000 7f3589600700 1 -- 172.21.15.62:0/774363725 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f35780105d0 con 0x7f358c10c8f0 2024-04-20T22:17:17.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:17.018+0000 7f3589600700 1 --2- 172.21.15.62:0/774363725 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f357406ecc0 0x7f3574071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:17:17.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:17.018+0000 7f3589600700 1 -- 172.21.15.62:0/774363725 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f357808ee80 con 0x7f358c10c8f0 2024-04-20T22:17:17.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:17.018+0000 7f3590c00700 1 --2- 172.21.15.62:0/774363725 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f357406ecc0 0x7f3574071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:17:17.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:17.019+0000 7f3590c00700 1 --2- 172.21.15.62:0/774363725 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f357406ecc0 0x7f3574071170 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f3580006010 tx=0x7f358001a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:17:17.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:17.023+0000 7f3589600700 1 -- 172.21.15.62:0/774363725 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f3578057c30 con 0x7f358c10c8f0 2024-04-20T22:17:17.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:17.361+0000 7f3593600700 1 -- 172.21.15.62:0/774363725 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f358c04f2a0 con 0x7f358c10c8f0 2024-04-20T22:17:17.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:17.363+0000 7f3589600700 1 -- 172.21.15.62:0/774363725 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f357805b2f0 con 0x7f358c10c8f0 2024-04-20T22:17:17.362 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:17:17.362 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T22:17:17.362 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:17:17.362 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:17:17.362 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T22:17:17.362 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:17:17.362 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:17:17.362 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T22:17:17.362 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 6 2024-04-20T22:17:17.362 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:17:17.363 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T22:17:17.363 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 4 2024-04-20T22:17:17.363 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:17:17.363 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T22:17:17.363 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 14 2024-04-20T22:17:17.363 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T22:17:17.363 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:17:17.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:17.365+0000 7f3593600700 1 -- 172.21.15.62:0/774363725 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f357406ecc0 msgr2=0x7f3574071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:17:17.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:17.365+0000 7f3593600700 1 --2- 172.21.15.62:0/774363725 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f357406ecc0 0x7f3574071170 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f3580006010 tx=0x7f358001a040 comp rx=0 tx=0).stop 2024-04-20T22:17:17.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:17.365+0000 7f3593600700 1 -- 172.21.15.62:0/774363725 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f358c10c8f0 msgr2=0x7f358c19ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:17:17.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:17.365+0000 7f3593600700 1 --2- 172.21.15.62:0/774363725 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f358c10c8f0 0x7f358c19ca40 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f357800d8d0 tx=0x7f357800dc90 comp rx=0 tx=0).stop 2024-04-20T22:17:17.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:17.365+0000 7f3593600700 1 -- 172.21.15.62:0/774363725 shutdown_connections 2024-04-20T22:17:17.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:17.365+0000 7f3593600700 1 --2- 172.21.15.62:0/774363725 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f357406ecc0 0x7f3574071170 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:17:17.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:17.365+0000 7f3593600700 1 --2- 172.21.15.62:0/774363725 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f358c106b80 0x7f358c19c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:17:17.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:17.365+0000 7f3593600700 1 --2- 172.21.15.62:0/774363725 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f358c10c8f0 0x7f358c19ca40 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:17:17.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:17.366+0000 7f3593600700 1 -- 172.21.15.62:0/774363725 >> 172.21.15.62:0/774363725 conn(0x7f358c076060 msgr2=0x7f358c077aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:17:17.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:17.366+0000 7f3593600700 1 -- 172.21.15.62:0/774363725 shutdown_connections 2024-04-20T22:17:17.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:17.366+0000 7f3593600700 1 -- 172.21.15.62:0/774363725 wait complete. 2024-04-20T22:17:17.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:17.532+0000 7f9671c00700 1 -- 172.21.15.62:0/1818866372 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f966c0739a0 msgr2=0x7f966c10c3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:17:17.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:17.532+0000 7f9671c00700 1 --2- 172.21.15.62:0/1818866372 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f966c0739a0 0x7f966c10c3c0 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f9658009940 tx=0x7f9658009c50 comp rx=0 tx=0).stop 2024-04-20T22:17:17.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:17.532+0000 7f9671c00700 1 -- 172.21.15.62:0/1818866372 shutdown_connections 2024-04-20T22:17:17.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:17.532+0000 7f9671c00700 1 --2- 172.21.15.62:0/1818866372 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f966c0739a0 0x7f966c10c3c0 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:17:17.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:17.532+0000 7f9671c00700 1 --2- 172.21.15.62:0/1818866372 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f966c073090 0x7f966c073460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:17:17.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:17.532+0000 7f9671c00700 1 -- 172.21.15.62:0/1818866372 >> 172.21.15.62:0/1818866372 conn(0x7f966c0fbb60 msgr2=0x7f966c0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:17:17.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:17.532+0000 7f9671c00700 1 -- 172.21.15.62:0/1818866372 shutdown_connections 2024-04-20T22:17:17.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:17.532+0000 7f9671c00700 1 -- 172.21.15.62:0/1818866372 wait complete. 2024-04-20T22:17:17.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:17.533+0000 7f9671c00700 1 Processor -- start 2024-04-20T22:17:17.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:17.534+0000 7f9671c00700 1 -- start start 2024-04-20T22:17:17.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:17.534+0000 7f9671c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f966c073090 0x7f966c197e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:17:17.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:17.534+0000 7f9671c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f966c0739a0 0x7f966c198380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:17:17.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:17.534+0000 7f9671c00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f966c198a60 con 0x7f966c073090 2024-04-20T22:17:17.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:17.534+0000 7f9671c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f966c19c7f0 con 0x7f966c0739a0 2024-04-20T22:17:17.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:17.534+0000 7f966be00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f966c073090 0x7f966c197e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:17:17.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:17.534+0000 7f966be00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f966c073090 0x7f966c197e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:32904/0 (socket says 172.21.15.62:32904) 2024-04-20T22:17:17.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:17.534+0000 7f966be00700 1 -- 172.21.15.62:0/1145812501 learned_addr learned my addr 172.21.15.62:0/1145812501 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:17:17.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:17.535+0000 7f966be00700 1 -- 172.21.15.62:0/1145812501 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f966c0739a0 msgr2=0x7f966c198380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:17:17.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:17.535+0000 7f966be00700 1 --2- 172.21.15.62:0/1145812501 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f966c0739a0 0x7f966c198380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:17:17.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:17.535+0000 7f966be00700 1 -- 172.21.15.62:0/1145812501 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96580095f0 con 0x7f966c073090 2024-04-20T22:17:17.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:17.535+0000 7f966be00700 1 --2- 172.21.15.62:0/1145812501 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f966c073090 0x7f966c197e40 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f965c00c8a0 tx=0x7f965c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:17:17.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:17.535+0000 7f9668c00700 1 -- 172.21.15.62:0/1145812501 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f965c00cea0 con 0x7f966c073090 2024-04-20T22:17:17.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:17.535+0000 7f9668c00700 1 -- 172.21.15.62:0/1145812501 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f965c004830 con 0x7f966c073090 2024-04-20T22:17:17.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:17.535+0000 7f9668c00700 1 -- 172.21.15.62:0/1145812501 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f965c0055b0 con 0x7f966c073090 2024-04-20T22:17:17.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:17.535+0000 7f9671c00700 1 -- 172.21.15.62:0/1145812501 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f966c19cad0 con 0x7f966c073090 2024-04-20T22:17:17.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:17.536+0000 7f9671c00700 1 -- 172.21.15.62:0/1145812501 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f966c19d020 con 0x7f966c073090 2024-04-20T22:17:17.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:17.537+0000 7f9671c00700 1 -- 172.21.15.62:0/1145812501 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f966c109ac0 con 0x7f966c073090 2024-04-20T22:17:17.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:17.538+0000 7f9668c00700 1 -- 172.21.15.62:0/1145812501 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f965c0049a0 con 0x7f966c073090 2024-04-20T22:17:17.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:17.539+0000 7f9668c00700 1 --2- 172.21.15.62:0/1145812501 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f965006ecc0 0x7f9650071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:17:17.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:17.539+0000 7f9668c00700 1 -- 172.21.15.62:0/1145812501 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f965c08eaf0 con 0x7f966c073090 2024-04-20T22:17:17.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:17.540+0000 7f966b400700 1 --2- 172.21.15.62:0/1145812501 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f965006ecc0 0x7f9650071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:17:17.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:17.540+0000 7f966b400700 1 --2- 172.21.15.62:0/1145812501 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f965006ecc0 0x7f9650071170 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f9658006010 tx=0x7f9658000bc0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:17:17.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:17.545+0000 7f9668c00700 1 -- 172.21.15.62:0/1145812501 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f965c057880 con 0x7f966c073090 2024-04-20T22:17:17.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:17:17 smithi189 ceph-mon[57022]: from='client.36218 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:17:17.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:17:17 smithi189 ceph-mon[57022]: from='client.36222 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:17:17.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:17:17 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/774363725' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:17:17.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:17:17 smithi062 ceph-mon[76322]: from='client.36218 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:17:17.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:17:17 smithi062 ceph-mon[76322]: from='client.36222 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:17:17.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:17:17 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/774363725' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:17:17.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:17.838+0000 7f9671c00700 1 -- 172.21.15.62:0/1145812501 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f966c19d2f0 con 0x7f966c073090 2024-04-20T22:17:17.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:17.839+0000 7f9668c00700 1 -- 172.21.15.62:0/1145812501 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 29 v29) v1 ==== 76+0+1887 (secure 0 0 0) 0x7f965c01d070 con 0x7f966c073090 2024-04-20T22:17:17.839 INFO:teuthology.orchestra.run.smithi062.stdout:e29 2024-04-20T22:17:17.839 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T22:17:17.839 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:17:17.839 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T22:17:17.839 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:17:17.839 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T22:17:17.839 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T22:17:17.839 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 20 2024-04-20T22:17:17.839 INFO:teuthology.orchestra.run.smithi062.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-20T22:17:17.839 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T22:17:17.839 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T21:42:01.559061+0000 2024-04-20T22:17:17.839 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T22:17:17.839 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T22:17:17.840 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T22:17:17.840 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T22:17:17.840 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T22:17:17.840 INFO:teuthology.orchestra.run.smithi062.stdout:max_xattr_size 65536 2024-04-20T22:17:17.840 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T22:17:17.840 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T22:17:17.840 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 81 2024-04-20T22:17:17.840 INFO:teuthology.orchestra.run.smithi062.stdout:snap_rstat 1 2024-04-20T22:17:17.840 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:17:17.840 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 2 2024-04-20T22:17:17.840 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1 2024-04-20T22:17:17.840 INFO:teuthology.orchestra.run.smithi062.stdout:up {} 2024-04-20T22:17:17.841 INFO:teuthology.orchestra.run.smithi062.stdout:failed 0,1 2024-04-20T22:17:17.841 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T22:17:17.841 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T22:17:17.841 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T22:17:17.841 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T22:17:17.841 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data enabled 2024-04-20T22:17:17.841 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T22:17:17.841 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T22:17:17.841 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 1 2024-04-20T22:17:17.841 INFO:teuthology.orchestra.run.smithi062.stdout:qdb_cluster leader: 0 members: 2024-04-20T22:17:17.841 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:17:17.841 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:17:17.841 INFO:teuthology.orchestra.run.smithi062.stdout:Standby daemons: 2024-04-20T22:17:17.841 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:17:17.842 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ryqblb{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6826/4113916205,v1:172.21.15.62:6827/4113916205] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:17:17.842 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ozizwi{-1:34476} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6828/3801975165,v1:172.21.15.62:6829/3801975165] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:17:17.842 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.dvimer{-1:44727} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6824/3576699232,v1:172.21.15.189:6825/3576699232] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:17:17.842 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.xbgspw{-1:45075} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6826/212905792,v1:172.21.15.189:6827/212905792] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:17:17.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:17.842+0000 7f9671c00700 1 -- 172.21.15.62:0/1145812501 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f965006ecc0 msgr2=0x7f9650071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:17:17.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:17.842+0000 7f9671c00700 1 --2- 172.21.15.62:0/1145812501 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f965006ecc0 0x7f9650071170 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f9658006010 tx=0x7f9658000bc0 comp rx=0 tx=0).stop 2024-04-20T22:17:17.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:17.842+0000 7f9671c00700 1 -- 172.21.15.62:0/1145812501 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f966c073090 msgr2=0x7f966c197e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:17:17.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:17.842+0000 7f9671c00700 1 --2- 172.21.15.62:0/1145812501 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f966c073090 0x7f966c197e40 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f965c00c8a0 tx=0x7f965c00cc60 comp rx=0 tx=0).stop 2024-04-20T22:17:17.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:17.842+0000 7f9671c00700 1 -- 172.21.15.62:0/1145812501 shutdown_connections 2024-04-20T22:17:17.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:17.842+0000 7f9671c00700 1 --2- 172.21.15.62:0/1145812501 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f965006ecc0 0x7f9650071170 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:17:17.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:17.842+0000 7f9671c00700 1 --2- 172.21.15.62:0/1145812501 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f966c073090 0x7f966c197e40 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:17:17.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:17.842+0000 7f9671c00700 1 --2- 172.21.15.62:0/1145812501 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f966c0739a0 0x7f966c198380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:17:17.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:17.842+0000 7f9671c00700 1 -- 172.21.15.62:0/1145812501 >> 172.21.15.62:0/1145812501 conn(0x7f966c0fbb60 msgr2=0x7f966c106c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:17:17.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:17.842+0000 7f9671c00700 1 -- 172.21.15.62:0/1145812501 shutdown_connections 2024-04-20T22:17:17.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:17.843+0000 7f9671c00700 1 -- 172.21.15.62:0/1145812501 wait complete. 2024-04-20T22:17:17.844 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 29 2024-04-20T22:17:18.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:18.010+0000 7f9262e00700 1 -- 172.21.15.62:0/124848316 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f925c106b80 msgr2=0x7f925c106ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:17:18.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:18.010+0000 7f9262e00700 1 --2- 172.21.15.62:0/124848316 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f925c106b80 0x7f925c106ff0 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f9244009910 tx=0x7f9244009c20 comp rx=0 tx=0).stop 2024-04-20T22:17:18.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:18.010+0000 7f9262e00700 1 -- 172.21.15.62:0/124848316 shutdown_connections 2024-04-20T22:17:18.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:18.010+0000 7f9262e00700 1 --2- 172.21.15.62:0/124848316 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f925c106b80 0x7f925c106ff0 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:17:18.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:18.010+0000 7f9262e00700 1 --2- 172.21.15.62:0/124848316 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f925c10c8f0 0x7f925c10ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:17:18.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:18.010+0000 7f9262e00700 1 -- 172.21.15.62:0/124848316 >> 172.21.15.62:0/124848316 conn(0x7f925c076060 msgr2=0x7f925c078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:17:18.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:18.010+0000 7f9262e00700 1 -- 172.21.15.62:0/124848316 shutdown_connections 2024-04-20T22:17:18.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:18.010+0000 7f9262e00700 1 -- 172.21.15.62:0/124848316 wait complete. 2024-04-20T22:17:18.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:18.011+0000 7f9262e00700 1 Processor -- start 2024-04-20T22:17:18.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:18.011+0000 7f9262e00700 1 -- start start 2024-04-20T22:17:18.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:18.012+0000 7f9262e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f925c106b80 0x7f925c19c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:17:18.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:18.012+0000 7f9262e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f925c10c8f0 0x7f925c19c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:17:18.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:18.012+0000 7f9262e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f925c19cfc0 con 0x7f925c10c8f0 2024-04-20T22:17:18.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:18.012+0000 7f9262e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f925c1a0d50 con 0x7f925c106b80 2024-04-20T22:17:18.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:18.012+0000 7f9253e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f925c10c8f0 0x7f925c19c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:17:18.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:18.012+0000 7f9253e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f925c10c8f0 0x7f925c19c8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:32916/0 (socket says 172.21.15.62:32916) 2024-04-20T22:17:18.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:18.012+0000 7f9253e00700 1 -- 172.21.15.62:0/984004522 learned_addr learned my addr 172.21.15.62:0/984004522 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:17:18.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:18.013+0000 7f925be00700 1 --2- 172.21.15.62:0/984004522 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f925c106b80 0x7f925c19c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:17:18.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:18.013+0000 7f9253e00700 1 -- 172.21.15.62:0/984004522 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f925c106b80 msgr2=0x7f925c19c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:17:18.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:18.013+0000 7f9253e00700 1 --2- 172.21.15.62:0/984004522 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f925c106b80 0x7f925c19c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:17:18.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:18.013+0000 7f9253e00700 1 -- 172.21.15.62:0/984004522 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92440095f0 con 0x7f925c10c8f0 2024-04-20T22:17:18.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:18.013+0000 7f9253e00700 1 --2- 172.21.15.62:0/984004522 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f925c10c8f0 0x7f925c19c8e0 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f92440092e0 tx=0x7f924400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:17:18.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:18.013+0000 7f9259600700 1 -- 172.21.15.62:0/984004522 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f924401d070 con 0x7f925c10c8f0 2024-04-20T22:17:18.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:18.013+0000 7f9259600700 1 -- 172.21.15.62:0/984004522 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9244005e00 con 0x7f925c10c8f0 2024-04-20T22:17:18.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:18.013+0000 7f9262e00700 1 -- 172.21.15.62:0/984004522 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f925c1a0fd0 con 0x7f925c10c8f0 2024-04-20T22:17:18.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:18.014+0000 7f9259600700 1 -- 172.21.15.62:0/984004522 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9244017400 con 0x7f925c10c8f0 2024-04-20T22:17:18.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:18.014+0000 7f9262e00700 1 -- 172.21.15.62:0/984004522 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f925c1a14c0 con 0x7f925c10c8f0 2024-04-20T22:17:18.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:18.015+0000 7f9262e00700 1 -- 172.21.15.62:0/984004522 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f925c10ecc0 con 0x7f925c10c8f0 2024-04-20T22:17:18.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:18.016+0000 7f9259600700 1 -- 172.21.15.62:0/984004522 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f924400f460 con 0x7f925c10c8f0 2024-04-20T22:17:18.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:18.017+0000 7f9259600700 1 --2- 172.21.15.62:0/984004522 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f924006ecc0 0x7f9240071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:17:18.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:18.017+0000 7f9259600700 1 -- 172.21.15.62:0/984004522 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f9244090430 con 0x7f925c10c8f0 2024-04-20T22:17:18.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:18.017+0000 7f925be00700 1 --2- 172.21.15.62:0/984004522 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f924006ecc0 0x7f9240071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:17:18.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:18.018+0000 7f925be00700 1 --2- 172.21.15.62:0/984004522 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f924006ecc0 0x7f9240071170 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f924c005950 tx=0x7f924c0058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:17:18.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:18.022+0000 7f9259600700 1 -- 172.21.15.62:0/984004522 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f9244059140 con 0x7f925c10c8f0 2024-04-20T22:17:18.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:18.292+0000 7f9262e00700 1 -- 172.21.15.62:0/984004522 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f925c02cd10 con 0x7f924006ecc0 2024-04-20T22:17:18.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:18.295+0000 7f9259600700 1 -- 172.21.15.62:0/984004522 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f925c02cd10 con 0x7f924006ecc0 2024-04-20T22:17:18.294 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:17:18.295 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T22:17:18.295 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T22:17:18.295 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-20T22:17:18.295 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T22:17:18.295 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr", 2024-04-20T22:17:18.295 INFO:teuthology.orchestra.run.smithi062.stdout: "osd", 2024-04-20T22:17:18.295 INFO:teuthology.orchestra.run.smithi062.stdout: "mds", 2024-04-20T22:17:18.295 INFO:teuthology.orchestra.run.smithi062.stdout: "crash", 2024-04-20T22:17:18.295 INFO:teuthology.orchestra.run.smithi062.stdout: "mon" 2024-04-20T22:17:18.295 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T22:17:18.295 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "16/23 daemons upgraded", 2024-04-20T22:17:18.295 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading mds daemons", 2024-04-20T22:17:18.295 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T22:17:18.295 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:17:18.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:18.298+0000 7f9262e00700 1 -- 172.21.15.62:0/984004522 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f924006ecc0 msgr2=0x7f9240071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:17:18.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:18.299+0000 7f9262e00700 1 --2- 172.21.15.62:0/984004522 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f924006ecc0 0x7f9240071170 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f924c005950 tx=0x7f924c0058e0 comp rx=0 tx=0).stop 2024-04-20T22:17:18.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:18.299+0000 7f9262e00700 1 -- 172.21.15.62:0/984004522 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f925c10c8f0 msgr2=0x7f925c19c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:17:18.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:18.299+0000 7f9262e00700 1 --2- 172.21.15.62:0/984004522 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f925c10c8f0 0x7f925c19c8e0 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f92440092e0 tx=0x7f924400be20 comp rx=0 tx=0).stop 2024-04-20T22:17:18.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:18.299+0000 7f9262e00700 1 -- 172.21.15.62:0/984004522 shutdown_connections 2024-04-20T22:17:18.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:18.299+0000 7f9262e00700 1 --2- 172.21.15.62:0/984004522 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f924006ecc0 0x7f9240071170 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:17:18.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:18.299+0000 7f9262e00700 1 --2- 172.21.15.62:0/984004522 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f925c106b80 0x7f925c19c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:17:18.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:18.299+0000 7f9262e00700 1 --2- 172.21.15.62:0/984004522 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f925c10c8f0 0x7f925c19c8e0 secure :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f92440092e0 tx=0x7f924400be20 comp rx=0 tx=0).stop 2024-04-20T22:17:18.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:18.299+0000 7f9262e00700 1 -- 172.21.15.62:0/984004522 >> 172.21.15.62:0/984004522 conn(0x7f925c076060 msgr2=0x7f925c077b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:17:18.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:18.299+0000 7f9262e00700 1 -- 172.21.15.62:0/984004522 shutdown_connections 2024-04-20T22:17:18.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:18.299+0000 7f9262e00700 1 -- 172.21.15.62:0/984004522 wait complete. 2024-04-20T22:17:18.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:18.461+0000 7f3ce9e00700 1 -- 172.21.15.62:0/3144880625 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3ce4104130 msgr2=0x7f3ce4104500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:17:18.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:18.461+0000 7f3ce9e00700 1 --2- 172.21.15.62:0/3144880625 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3ce4104130 0x7f3ce4104500 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f3cd400b0a0 tx=0x7f3cd400b3b0 comp rx=0 tx=0).stop 2024-04-20T22:17:18.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:18.461+0000 7f3ce9e00700 1 -- 172.21.15.62:0/3144880625 shutdown_connections 2024-04-20T22:17:18.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:18.461+0000 7f3ce9e00700 1 --2- 172.21.15.62:0/3144880625 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ce40ffbd0 0x7f3ce4100040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:17:18.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:18.461+0000 7f3ce9e00700 1 --2- 172.21.15.62:0/3144880625 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3ce4104130 0x7f3ce4104500 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:17:18.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:18.461+0000 7f3ce9e00700 1 -- 172.21.15.62:0/3144880625 >> 172.21.15.62:0/3144880625 conn(0x7f3ce4075130 msgr2=0x7f3ce4075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:17:18.461 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:17:18 smithi062 ceph-mon[76322]: from='client.36224 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:17:18.462 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:17:18 smithi062 ceph-mon[76322]: pgmap v1263: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:17:18.462 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:17:18 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/1145812501' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:17:18.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:18.462+0000 7f3ce9e00700 1 -- 172.21.15.62:0/3144880625 shutdown_connections 2024-04-20T22:17:18.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:18.462+0000 7f3ce9e00700 1 -- 172.21.15.62:0/3144880625 wait complete. 2024-04-20T22:17:18.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:18.463+0000 7f3ce9e00700 1 Processor -- start 2024-04-20T22:17:18.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:18.463+0000 7f3ce9e00700 1 -- start start 2024-04-20T22:17:18.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:18.463+0000 7f3ce9e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3ce40ffbd0 0x7f3ce41980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:17:18.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:18.464+0000 7f3ce9e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ce4104130 0x7f3ce41985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:17:18.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:18.464+0000 7f3ce9e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ce4198cd0 con 0x7f3ce40ffbd0 2024-04-20T22:17:18.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:18.464+0000 7f3ce9e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ce419ca60 con 0x7f3ce4104130 2024-04-20T22:17:18.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:18.464+0000 7f3ce2a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3ce40ffbd0 0x7f3ce41980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:17:18.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:18.464+0000 7f3ce2a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3ce40ffbd0 0x7f3ce41980b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:32934/0 (socket says 172.21.15.62:32934) 2024-04-20T22:17:18.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:18.464+0000 7f3ce2a00700 1 -- 172.21.15.62:0/4070815497 learned_addr learned my addr 172.21.15.62:0/4070815497 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:17:18.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:18.464+0000 7f3cda000700 1 --2- 172.21.15.62:0/4070815497 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ce4104130 0x7f3ce41985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:17:18.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:18.464+0000 7f3ce2a00700 1 -- 172.21.15.62:0/4070815497 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ce4104130 msgr2=0x7f3ce41985f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:17:18.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:18.464+0000 7f3ce2a00700 1 --2- 172.21.15.62:0/4070815497 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ce4104130 0x7f3ce41985f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:17:18.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:18.464+0000 7f3ce2a00700 1 -- 172.21.15.62:0/4070815497 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cd4009d00 con 0x7f3ce40ffbd0 2024-04-20T22:17:18.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:18.465+0000 7f3ce2a00700 1 --2- 172.21.15.62:0/4070815497 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3ce40ffbd0 0x7f3ce41980b0 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f3cd400b070 tx=0x7f3cd400bd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:17:18.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:18.465+0000 7f3ce8a00700 1 -- 172.21.15.62:0/4070815497 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3cd400e040 con 0x7f3ce40ffbd0 2024-04-20T22:17:18.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:18.465+0000 7f3ce8a00700 1 -- 172.21.15.62:0/4070815497 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3cd4003c10 con 0x7f3ce40ffbd0 2024-04-20T22:17:18.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:18.465+0000 7f3ce8a00700 1 -- 172.21.15.62:0/4070815497 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3cd401d6a0 con 0x7f3ce40ffbd0 2024-04-20T22:17:18.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:18.465+0000 7f3ce9e00700 1 -- 172.21.15.62:0/4070815497 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ce419cce0 con 0x7f3ce40ffbd0 2024-04-20T22:17:18.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:18.465+0000 7f3ce9e00700 1 -- 172.21.15.62:0/4070815497 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ce419d1d0 con 0x7f3ce40ffbd0 2024-04-20T22:17:18.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:18.467+0000 7f3ce9e00700 1 -- 172.21.15.62:0/4070815497 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ce4103510 con 0x7f3ce40ffbd0 2024-04-20T22:17:18.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:18.468+0000 7f3ce8a00700 1 -- 172.21.15.62:0/4070815497 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f3cd4019070 con 0x7f3ce40ffbd0 2024-04-20T22:17:18.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:18.469+0000 7f3ce8a00700 1 --2- 172.21.15.62:0/4070815497 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3cc806ecc0 0x7f3cc8071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:17:18.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:18.469+0000 7f3ce8a00700 1 -- 172.21.15.62:0/4070815497 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f3cd4090370 con 0x7f3ce40ffbd0 2024-04-20T22:17:18.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:18.470+0000 7f3cda000700 1 --2- 172.21.15.62:0/4070815497 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3cc806ecc0 0x7f3cc8071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:17:18.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:18.470+0000 7f3cda000700 1 --2- 172.21.15.62:0/4070815497 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3cc806ecc0 0x7f3cc8071170 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f3ce41996d0 tx=0x7f3ccc009260 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:17:18.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:18.474+0000 7f3ce8a00700 1 -- 172.21.15.62:0/4070815497 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f3cd4059100 con 0x7f3ce40ffbd0 2024-04-20T22:17:18.516 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:17:18 smithi189 ceph-mon[57022]: from='client.36224 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:17:18.516 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:17:18 smithi189 ceph-mon[57022]: pgmap v1263: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:17:18.516 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:17:18 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/1145812501' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:17:18.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:18.806+0000 7f3ce9e00700 1 -- 172.21.15.62:0/4070815497 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f3ce404f2a0 con 0x7f3ce40ffbd0 2024-04-20T22:17:18.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:18.807+0000 7f3ce8a00700 1 -- 172.21.15.62:0/4070815497 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7f3cd4003d80 con 0x7f3ce40ffbd0 2024-04-20T22:17:18.807 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-20T22:17:18.807 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-20T22:17:18.807 INFO:teuthology.orchestra.run.smithi062.stdout: host smithi189 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi189/config 2024-04-20T22:17:18.807 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-20T22:17:18.807 INFO:teuthology.orchestra.run.smithi062.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi189 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp11670v2q:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:17:18.807 INFO:teuthology.orchestra.run.smithi062.stdout:Traceback (most recent call last): 2024-04-20T22:17:18.807 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-20T22:17:18.807 INFO:teuthology.orchestra.run.smithi062.stdout: return _run_code(code, main_globals, None, 2024-04-20T22:17:18.807 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-20T22:17:18.807 INFO:teuthology.orchestra.run.smithi062.stdout: exec(code, run_globals) 2024-04-20T22:17:18.808 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5618, in 2024-04-20T22:17:18.808 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5606, in main 2024-04-20T22:17:18.808 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 408, in _infer_config 2024-04-20T22:17:18.808 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 323, in _infer_fsid 2024-04-20T22:17:18.808 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 436, in _infer_image 2024-04-20T22:17:18.808 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 310, in _validate_fsid 2024-04-20T22:17:18.808 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 3359, in command_ceph_volume 2024-04-20T22:17:18.808 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-20T22:17:18.808 INFO:teuthology.orchestra.run.smithi062.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi189 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp11670v2q:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:17:18.808 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-20T22:17:18.808 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is degraded 2024-04-20T22:17:18.808 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T22:17:18.808 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T22:17:18.808 INFO:teuthology.orchestra.run.smithi062.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-20T22:17:18.808 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-20T22:17:18.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:18.810+0000 7f3ce9e00700 1 -- 172.21.15.62:0/4070815497 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3cc806ecc0 msgr2=0x7f3cc8071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:17:18.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:18.810+0000 7f3ce9e00700 1 --2- 172.21.15.62:0/4070815497 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3cc806ecc0 0x7f3cc8071170 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f3ce41996d0 tx=0x7f3ccc009260 comp rx=0 tx=0).stop 2024-04-20T22:17:18.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:18.810+0000 7f3ce9e00700 1 -- 172.21.15.62:0/4070815497 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3ce40ffbd0 msgr2=0x7f3ce41980b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:17:18.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:18.810+0000 7f3ce9e00700 1 --2- 172.21.15.62:0/4070815497 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3ce40ffbd0 0x7f3ce41980b0 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f3cd400b070 tx=0x7f3cd400bd10 comp rx=0 tx=0).stop 2024-04-20T22:17:18.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:18.810+0000 7f3ce9e00700 1 -- 172.21.15.62:0/4070815497 shutdown_connections 2024-04-20T22:17:18.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:18.810+0000 7f3ce9e00700 1 --2- 172.21.15.62:0/4070815497 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3cc806ecc0 0x7f3cc8071170 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:17:18.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:18.810+0000 7f3ce9e00700 1 --2- 172.21.15.62:0/4070815497 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3ce40ffbd0 0x7f3ce41980b0 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:17:18.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:18.810+0000 7f3ce9e00700 1 --2- 172.21.15.62:0/4070815497 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ce4104130 0x7f3ce41985f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:17:18.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:18.810+0000 7f3ce9e00700 1 -- 172.21.15.62:0/4070815497 >> 172.21.15.62:0/4070815497 conn(0x7f3ce4075130 msgr2=0x7f3ce41097b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:17:18.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:18.810+0000 7f3ce9e00700 1 -- 172.21.15.62:0/4070815497 shutdown_connections 2024-04-20T22:17:18.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:18.810+0000 7f3ce9e00700 1 -- 172.21.15.62:0/4070815497 wait complete. 2024-04-20T22:17:19.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:17:19 smithi189 ceph-mon[57022]: from='client.36236 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:17:19.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:17:19 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/4070815497' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:17:19.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:17:19 smithi062 ceph-mon[76322]: from='client.36236 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:17:19.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:17:19 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/4070815497' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:17:20.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:17:20 smithi189 ceph-mon[57022]: pgmap v1264: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:17:20.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:17:20 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:17:20.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:17:20 smithi062 ceph-mon[76322]: pgmap v1264: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:17:20.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:17:20 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:17:21.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:17:21 smithi189 ceph-mon[57022]: pgmap v1265: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:17:21.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:17:21 smithi062 ceph-mon[76322]: pgmap v1265: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:17:24.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:17:24 smithi062 ceph-mon[76322]: pgmap v1266: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:17:24.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:17:24 smithi189 ceph-mon[57022]: pgmap v1266: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:17:26.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:17:26 smithi189 ceph-mon[57022]: pgmap v1267: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:17:26.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:17:26 smithi062 ceph-mon[76322]: pgmap v1267: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:17:28.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:17:28 smithi062 ceph-mon[76322]: pgmap v1268: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:17:28.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:17:28 smithi189 ceph-mon[57022]: pgmap v1268: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:17:30.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:17:30 smithi189 ceph-mon[57022]: pgmap v1269: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:17:30.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:17:30 smithi062 ceph-mon[76322]: pgmap v1269: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:17:32.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:17:32 smithi189 ceph-mon[57022]: pgmap v1270: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:17:32.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:17:32 smithi062 ceph-mon[76322]: pgmap v1270: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:17:34.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:17:34 smithi189 ceph-mon[57022]: pgmap v1271: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:17:34.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:17:34 smithi062 ceph-mon[76322]: pgmap v1271: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:17:35.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:17:35 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:17:35.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:17:35 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:17:36.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:17:36 smithi189 ceph-mon[57022]: pgmap v1272: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:17:36.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:17:36 smithi062 ceph-mon[76322]: pgmap v1272: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:17:38.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:17:38 smithi189 ceph-mon[57022]: pgmap v1273: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:17:38.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:17:38 smithi062 ceph-mon[76322]: pgmap v1273: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:17:39.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:17:39 smithi189 ceph-mon[57022]: pgmap v1274: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:17:39.757 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:17:39 smithi062 ceph-mon[76322]: pgmap v1274: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:17:42.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:17:42 smithi189 ceph-mon[57022]: pgmap v1275: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:17:42.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:17:42 smithi062 ceph-mon[76322]: pgmap v1275: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:17:44.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:17:44 smithi189 ceph-mon[57022]: pgmap v1276: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:17:44.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:17:44 smithi062 ceph-mon[76322]: pgmap v1276: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:17:46.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:17:46 smithi189 ceph-mon[57022]: pgmap v1277: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:17:46.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:17:46 smithi062 ceph-mon[76322]: pgmap v1277: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:17:48.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:17:48 smithi062 ceph-mon[76322]: pgmap v1278: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:17:48.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:17:48 smithi189 ceph-mon[57022]: pgmap v1278: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:17:48.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:48.987+0000 7fd3fb800700 1 -- 172.21.15.62:0/1109644532 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3f41085b0 msgr2=0x7fd3f4108980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:17:48.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:48.987+0000 7fd3fb800700 1 --2- 172.21.15.62:0/1109644532 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3f41085b0 0x7fd3f4108980 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7fd3e8009910 tx=0x7fd3e8009c20 comp rx=0 tx=0).stop 2024-04-20T22:17:48.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:48.987+0000 7fd3fb800700 1 -- 172.21.15.62:0/1109644532 shutdown_connections 2024-04-20T22:17:48.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:48.987+0000 7fd3fb800700 1 --2- 172.21.15.62:0/1109644532 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3f4068540 0x7fd3f40689b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:17:48.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:48.987+0000 7fd3fb800700 1 --2- 172.21.15.62:0/1109644532 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3f41085b0 0x7fd3f4108980 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:17:48.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:48.987+0000 7fd3fb800700 1 -- 172.21.15.62:0/1109644532 >> 172.21.15.62:0/1109644532 conn(0x7fd3f4075130 msgr2=0x7fd3f4075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:17:48.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:48.987+0000 7fd3fb800700 1 -- 172.21.15.62:0/1109644532 shutdown_connections 2024-04-20T22:17:48.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:48.987+0000 7fd3fb800700 1 -- 172.21.15.62:0/1109644532 wait complete. 2024-04-20T22:17:48.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:48.989+0000 7fd3fb800700 1 Processor -- start 2024-04-20T22:17:48.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:48.989+0000 7fd3fb800700 1 -- start start 2024-04-20T22:17:48.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:48.989+0000 7fd3fb800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3f4068540 0x7fd3f41980e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:17:48.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:48.990+0000 7fd3fb800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3f41085b0 0x7fd3f4198620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:17:48.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:48.990+0000 7fd3fb800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f4198d00 con 0x7fd3f41085b0 2024-04-20T22:17:48.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:48.990+0000 7fd3fb800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f419ca90 con 0x7fd3f4068540 2024-04-20T22:17:48.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:48.990+0000 7fd3f3e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3f41085b0 0x7fd3f4198620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:17:48.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:48.990+0000 7fd3f3e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3f41085b0 0x7fd3f4198620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44850/0 (socket says 172.21.15.62:44850) 2024-04-20T22:17:48.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:48.990+0000 7fd3f3e00700 1 -- 172.21.15.62:0/3891580355 learned_addr learned my addr 172.21.15.62:0/3891580355 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:17:48.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:48.990+0000 7fd3f8e00700 1 --2- 172.21.15.62:0/3891580355 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3f4068540 0x7fd3f41980e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:17:48.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:48.990+0000 7fd3f3e00700 1 -- 172.21.15.62:0/3891580355 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3f4068540 msgr2=0x7fd3f41980e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:17:48.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:48.990+0000 7fd3f3e00700 1 --2- 172.21.15.62:0/3891580355 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3f4068540 0x7fd3f41980e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:17:48.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:48.991+0000 7fd3f3e00700 1 -- 172.21.15.62:0/3891580355 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3e80095f0 con 0x7fd3f41085b0 2024-04-20T22:17:48.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:48.991+0000 7fd3f3e00700 1 --2- 172.21.15.62:0/3891580355 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3f41085b0 0x7fd3f4198620 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7fd3e000c8a0 tx=0x7fd3e000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:17:48.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:48.991+0000 7fd3f1600700 1 -- 172.21.15.62:0/3891580355 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd3e000cea0 con 0x7fd3f41085b0 2024-04-20T22:17:48.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:48.991+0000 7fd3f1600700 1 -- 172.21.15.62:0/3891580355 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd3e0004830 con 0x7fd3f41085b0 2024-04-20T22:17:48.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:48.991+0000 7fd3fb800700 1 -- 172.21.15.62:0/3891580355 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3f419cd70 con 0x7fd3f41085b0 2024-04-20T22:17:48.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:48.991+0000 7fd3f1600700 1 -- 172.21.15.62:0/3891580355 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd3e00055b0 con 0x7fd3f41085b0 2024-04-20T22:17:48.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:48.992+0000 7fd3fb800700 1 -- 172.21.15.62:0/3891580355 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3f419d2c0 con 0x7fd3f41085b0 2024-04-20T22:17:48.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:48.993+0000 7fd3fb800700 1 -- 172.21.15.62:0/3891580355 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3f410aa00 con 0x7fd3f41085b0 2024-04-20T22:17:48.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:48.994+0000 7fd3f1600700 1 -- 172.21.15.62:0/3891580355 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fd3e00049a0 con 0x7fd3f41085b0 2024-04-20T22:17:48.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:48.995+0000 7fd3f1600700 1 --2- 172.21.15.62:0/3891580355 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd3dc06ecc0 0x7fd3dc071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:17:48.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:48.995+0000 7fd3f1600700 1 -- 172.21.15.62:0/3891580355 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fd3e008eaf0 con 0x7fd3f41085b0 2024-04-20T22:17:48.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:48.995+0000 7fd3f8e00700 1 --2- 172.21.15.62:0/3891580355 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd3dc06ecc0 0x7fd3dc071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:17:48.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:48.996+0000 7fd3f8e00700 1 --2- 172.21.15.62:0/3891580355 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd3dc06ecc0 0x7fd3dc071170 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fd3e8005f50 tx=0x7fd3e801a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:17:49.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:49.000+0000 7fd3f1600700 1 -- 172.21.15.62:0/3891580355 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fd3e0057880 con 0x7fd3f41085b0 2024-04-20T22:17:49.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:49.263+0000 7fd3fb800700 1 -- 172.21.15.62:0/3891580355 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd3f419d600 con 0x7fd3dc06ecc0 2024-04-20T22:17:49.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:49.266+0000 7fd3f1600700 1 -- 172.21.15.62:0/3891580355 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fd3f419d600 con 0x7fd3dc06ecc0 2024-04-20T22:17:49.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:49.269+0000 7fd3fb800700 1 -- 172.21.15.62:0/3891580355 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd3dc06ecc0 msgr2=0x7fd3dc071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:17:49.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:49.269+0000 7fd3fb800700 1 --2- 172.21.15.62:0/3891580355 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd3dc06ecc0 0x7fd3dc071170 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fd3e8005f50 tx=0x7fd3e801a040 comp rx=0 tx=0).stop 2024-04-20T22:17:49.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:49.269+0000 7fd3fb800700 1 -- 172.21.15.62:0/3891580355 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3f41085b0 msgr2=0x7fd3f4198620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:17:49.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:49.269+0000 7fd3fb800700 1 --2- 172.21.15.62:0/3891580355 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3f41085b0 0x7fd3f4198620 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7fd3e000c8a0 tx=0x7fd3e000cc60 comp rx=0 tx=0).stop 2024-04-20T22:17:49.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:49.269+0000 7fd3fb800700 1 -- 172.21.15.62:0/3891580355 shutdown_connections 2024-04-20T22:17:49.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:49.269+0000 7fd3fb800700 1 --2- 172.21.15.62:0/3891580355 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd3dc06ecc0 0x7fd3dc071170 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:17:49.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:49.269+0000 7fd3fb800700 1 --2- 172.21.15.62:0/3891580355 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3f4068540 0x7fd3f41980e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:17:49.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:49.269+0000 7fd3fb800700 1 --2- 172.21.15.62:0/3891580355 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3f41085b0 0x7fd3f4198620 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:17:49.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:49.269+0000 7fd3fb800700 1 -- 172.21.15.62:0/3891580355 >> 172.21.15.62:0/3891580355 conn(0x7fd3f4075130 msgr2=0x7fd3f4106430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:17:49.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:49.269+0000 7fd3fb800700 1 -- 172.21.15.62:0/3891580355 shutdown_connections 2024-04-20T22:17:49.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:49.270+0000 7fd3fb800700 1 -- 172.21.15.62:0/3891580355 wait complete. 2024-04-20T22:17:49.286 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T22:17:49.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:49.440+0000 7f7ff5400700 1 -- 172.21.15.62:0/3657684151 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ff0100290 msgr2=0x7f7ff0100660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:17:49.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:49.440+0000 7f7ff5400700 1 --2- 172.21.15.62:0/3657684151 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ff0100290 0x7f7ff0100660 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f7fe0009910 tx=0x7f7fe0009c20 comp rx=0 tx=0).stop 2024-04-20T22:17:49.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:49.440+0000 7f7ff5400700 1 -- 172.21.15.62:0/3657684151 shutdown_connections 2024-04-20T22:17:49.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:49.440+0000 7f7ff5400700 1 --2- 172.21.15.62:0/3657684151 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ff0100c30 0x7f7ff0104c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:17:49.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:49.440+0000 7f7ff5400700 1 --2- 172.21.15.62:0/3657684151 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ff0100290 0x7f7ff0100660 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:17:49.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:49.440+0000 7f7ff5400700 1 -- 172.21.15.62:0/3657684151 >> 172.21.15.62:0/3657684151 conn(0x7f7ff00fbb10 msgr2=0x7f7ff00fdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:17:49.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:49.440+0000 7f7ff5400700 1 -- 172.21.15.62:0/3657684151 shutdown_connections 2024-04-20T22:17:49.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:49.440+0000 7f7ff5400700 1 -- 172.21.15.62:0/3657684151 wait complete. 2024-04-20T22:17:49.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:49.442+0000 7f7ff5400700 1 Processor -- start 2024-04-20T22:17:49.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:49.442+0000 7f7ff5400700 1 -- start start 2024-04-20T22:17:49.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:49.442+0000 7f7ff5400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ff0100290 0x7f7ff0197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:17:49.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:49.442+0000 7f7ff5400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ff0100c30 0x7f7ff0198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:17:49.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:49.442+0000 7f7ff5400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ff0198b50 con 0x7f7ff0100c30 2024-04-20T22:17:49.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:49.442+0000 7f7ff5400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ff019c8e0 con 0x7f7ff0100290 2024-04-20T22:17:49.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:49.443+0000 7f7fef400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ff0100c30 0x7f7ff0198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:17:49.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:49.443+0000 7f7fef400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ff0100c30 0x7f7ff0198470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44876/0 (socket says 172.21.15.62:44876) 2024-04-20T22:17:49.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:49.443+0000 7f7fef400700 1 -- 172.21.15.62:0/150198729 learned_addr learned my addr 172.21.15.62:0/150198729 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:17:49.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:49.443+0000 7f7fefe00700 1 --2- 172.21.15.62:0/150198729 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ff0100290 0x7f7ff0197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:17:49.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:49.443+0000 7f7fef400700 1 -- 172.21.15.62:0/150198729 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ff0100290 msgr2=0x7f7ff0197f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:17:49.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:49.443+0000 7f7fef400700 1 --2- 172.21.15.62:0/150198729 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ff0100290 0x7f7ff0197f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:17:49.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:49.443+0000 7f7fef400700 1 -- 172.21.15.62:0/150198729 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fe00095f0 con 0x7f7ff0100c30 2024-04-20T22:17:49.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:49.444+0000 7f7fef400700 1 --2- 172.21.15.62:0/150198729 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ff0100c30 0x7f7ff0198470 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f7fdc00b480 tx=0x7f7fdc00b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:17:49.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:49.444+0000 7f7fecc00700 1 -- 172.21.15.62:0/150198729 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7fdc0118c0 con 0x7f7ff0100c30 2024-04-20T22:17:49.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:49.444+0000 7f7fecc00700 1 -- 172.21.15.62:0/150198729 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7fdc010380 con 0x7f7ff0100c30 2024-04-20T22:17:49.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:49.444+0000 7f7ff5400700 1 -- 172.21.15.62:0/150198729 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ff019cbc0 con 0x7f7ff0100c30 2024-04-20T22:17:49.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:49.444+0000 7f7fecc00700 1 -- 172.21.15.62:0/150198729 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7fdc00f4f0 con 0x7f7ff0100c30 2024-04-20T22:17:49.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:49.444+0000 7f7ff5400700 1 -- 172.21.15.62:0/150198729 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ff019d110 con 0x7f7ff0100c30 2024-04-20T22:17:49.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:49.446+0000 7f7ff5400700 1 -- 172.21.15.62:0/150198729 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ff0104380 con 0x7f7ff0100c30 2024-04-20T22:17:49.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:49.446+0000 7f7fecc00700 1 -- 172.21.15.62:0/150198729 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f7fdc00f760 con 0x7f7ff0100c30 2024-04-20T22:17:49.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:49.450+0000 7f7fecc00700 1 --2- 172.21.15.62:0/150198729 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7fd406ecc0 0x7f7fd4071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:17:49.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:49.450+0000 7f7fefe00700 1 --2- 172.21.15.62:0/150198729 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7fd406ecc0 0x7f7fd4071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:17:49.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:49.451+0000 7f7fecc00700 1 -- 172.21.15.62:0/150198729 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f7fdc08e9f0 con 0x7f7ff0100c30 2024-04-20T22:17:49.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:49.451+0000 7f7fefe00700 1 --2- 172.21.15.62:0/150198729 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7fd406ecc0 0x7f7fd4071170 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f7fe0006010 tx=0x7f7fe001a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:17:49.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:49.453+0000 7f7fecc00700 1 -- 172.21.15.62:0/150198729 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f7fdc090e60 con 0x7f7ff0100c30 2024-04-20T22:17:49.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:49.724+0000 7f7ff5400700 1 -- 172.21.15.62:0/150198729 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7ff019d4a0 con 0x7f7fd406ecc0 2024-04-20T22:17:49.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:49.727+0000 7f7fecc00700 1 -- 172.21.15.62:0/150198729 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f7ff019d4a0 con 0x7f7fd406ecc0 2024-04-20T22:17:49.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:49.729+0000 7f7ff5400700 1 -- 172.21.15.62:0/150198729 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7fd406ecc0 msgr2=0x7f7fd4071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:17:49.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:49.729+0000 7f7ff5400700 1 --2- 172.21.15.62:0/150198729 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7fd406ecc0 0x7f7fd4071170 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f7fe0006010 tx=0x7f7fe001a040 comp rx=0 tx=0).stop 2024-04-20T22:17:49.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:49.730+0000 7f7ff5400700 1 -- 172.21.15.62:0/150198729 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ff0100c30 msgr2=0x7f7ff0198470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:17:49.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:49.730+0000 7f7ff5400700 1 --2- 172.21.15.62:0/150198729 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ff0100c30 0x7f7ff0198470 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f7fdc00b480 tx=0x7f7fdc00b840 comp rx=0 tx=0).stop 2024-04-20T22:17:49.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:49.730+0000 7f7ff5400700 1 -- 172.21.15.62:0/150198729 shutdown_connections 2024-04-20T22:17:49.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:49.730+0000 7f7ff5400700 1 --2- 172.21.15.62:0/150198729 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7fd406ecc0 0x7f7fd4071170 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:17:49.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:49.730+0000 7f7ff5400700 1 --2- 172.21.15.62:0/150198729 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ff0100290 0x7f7ff0197f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:17:49.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:49.730+0000 7f7ff5400700 1 --2- 172.21.15.62:0/150198729 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ff0100c30 0x7f7ff0198470 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:17:49.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:49.730+0000 7f7ff5400700 1 -- 172.21.15.62:0/150198729 >> 172.21.15.62:0/150198729 conn(0x7f7ff00fbb10 msgr2=0x7f7ff00fd260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:17:49.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:49.730+0000 7f7ff5400700 1 -- 172.21.15.62:0/150198729 shutdown_connections 2024-04-20T22:17:49.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:49.730+0000 7f7ff5400700 1 -- 172.21.15.62:0/150198729 wait complete. 2024-04-20T22:17:49.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:49.889+0000 7f9101800700 1 -- 172.21.15.62:0/3035137319 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90fc068540 msgr2=0x7f90fc0689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:17:49.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:49.889+0000 7f9101800700 1 --2- 172.21.15.62:0/3035137319 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90fc068540 0x7f90fc0689b0 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f90e8009910 tx=0x7f90e8009c20 comp rx=0 tx=0).stop 2024-04-20T22:17:49.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:49.889+0000 7f9101800700 1 -- 172.21.15.62:0/3035137319 shutdown_connections 2024-04-20T22:17:49.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:49.889+0000 7f9101800700 1 --2- 172.21.15.62:0/3035137319 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90fc068540 0x7f90fc0689b0 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:17:49.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:49.889+0000 7f9101800700 1 --2- 172.21.15.62:0/3035137319 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f90fc1085b0 0x7f90fc108980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:17:49.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:49.889+0000 7f9101800700 1 -- 172.21.15.62:0/3035137319 >> 172.21.15.62:0/3035137319 conn(0x7f90fc075130 msgr2=0x7f90fc075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:17:49.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:49.889+0000 7f9101800700 1 -- 172.21.15.62:0/3035137319 shutdown_connections 2024-04-20T22:17:49.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:49.890+0000 7f9101800700 1 -- 172.21.15.62:0/3035137319 wait complete. 2024-04-20T22:17:49.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:49.890+0000 7f9101800700 1 Processor -- start 2024-04-20T22:17:49.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:49.891+0000 7f9101800700 1 -- start start 2024-04-20T22:17:49.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:49.891+0000 7f9101800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90fc068540 0x7f90fc1980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:17:49.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:49.891+0000 7f9101800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f90fc1085b0 0x7f90fc1985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:17:49.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:49.891+0000 7f9101800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90fc198cd0 con 0x7f90fc068540 2024-04-20T22:17:49.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:49.891+0000 7f9101800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90fc19ca60 con 0x7f90fc1085b0 2024-04-20T22:17:49.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:49.891+0000 7f90faa00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90fc068540 0x7f90fc1980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:17:49.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:49.891+0000 7f90faa00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90fc068540 0x7f90fc1980b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44890/0 (socket says 172.21.15.62:44890) 2024-04-20T22:17:49.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:49.891+0000 7f90faa00700 1 -- 172.21.15.62:0/25568104 learned_addr learned my addr 172.21.15.62:0/25568104 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:17:49.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:49.892+0000 7f90faa00700 1 -- 172.21.15.62:0/25568104 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f90fc1085b0 msgr2=0x7f90fc1985f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:17:49.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:49.892+0000 7f90fa000700 1 --2- 172.21.15.62:0/25568104 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f90fc1085b0 0x7f90fc1985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:17:49.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:49.892+0000 7f90faa00700 1 --2- 172.21.15.62:0/25568104 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f90fc1085b0 0x7f90fc1985f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:17:49.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:49.892+0000 7f90faa00700 1 -- 172.21.15.62:0/25568104 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90e80095f0 con 0x7f90fc068540 2024-04-20T22:17:49.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:49.892+0000 7f90faa00700 1 --2- 172.21.15.62:0/25568104 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90fc068540 0x7f90fc1980b0 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f90ec00c8f0 tx=0x7f90ec00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:17:49.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:49.892+0000 7f90f3400700 1 -- 172.21.15.62:0/25568104 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f90ec007a70 con 0x7f90fc068540 2024-04-20T22:17:49.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:49.892+0000 7f90f3400700 1 -- 172.21.15.62:0/25568104 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f90ec007bd0 con 0x7f90fc068540 2024-04-20T22:17:49.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:49.892+0000 7f90f3400700 1 -- 172.21.15.62:0/25568104 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f90ec018630 con 0x7f90fc068540 2024-04-20T22:17:49.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:49.892+0000 7f9101800700 1 -- 172.21.15.62:0/25568104 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90fc19cd40 con 0x7f90fc068540 2024-04-20T22:17:49.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:49.893+0000 7f9101800700 1 -- 172.21.15.62:0/25568104 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90fc19d290 con 0x7f90fc068540 2024-04-20T22:17:49.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:49.894+0000 7f9101800700 1 -- 172.21.15.62:0/25568104 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90fc10aa00 con 0x7f90fc068540 2024-04-20T22:17:49.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:49.895+0000 7f90f3400700 1 -- 172.21.15.62:0/25568104 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f90ec00f450 con 0x7f90fc068540 2024-04-20T22:17:49.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:49.896+0000 7f90f3400700 1 --2- 172.21.15.62:0/25568104 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f90e006ecc0 0x7f90e0071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:17:49.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:49.896+0000 7f90f3400700 1 -- 172.21.15.62:0/25568104 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f90ec08e9c0 con 0x7f90fc068540 2024-04-20T22:17:49.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:49.896+0000 7f90fa000700 1 --2- 172.21.15.62:0/25568104 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f90e006ecc0 0x7f90e0071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:17:49.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:49.897+0000 7f90fa000700 1 --2- 172.21.15.62:0/25568104 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f90e006ecc0 0x7f90e0071170 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f90e8000c00 tx=0x7f90e8005fd0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:17:49.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:49.901+0000 7f90f3400700 1 -- 172.21.15.62:0/25568104 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f90ec057940 con 0x7f90fc068540 2024-04-20T22:17:50.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:50.149+0000 7f9101800700 1 -- 172.21.15.62:0/25568104 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f90fc19d510 con 0x7f90e006ecc0 2024-04-20T22:17:50.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:50.165+0000 7f90f3400700 1 -- 172.21.15.62:0/25568104 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f90fc19d510 con 0x7f90e006ecc0 2024-04-20T22:17:50.166 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T22:17:50.166 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (43m) 4m ago 57m 14.6M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T22:17:50.166 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (57m) 4m ago 57m 11.6M - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T22:17:50.166 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (56m) 4m ago 56m 11.2M - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T22:17:50.166 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (39m) 4m ago 57m 6660k - 19.0.0-3133-ga8c84f59 76bc6b91b726 2825d5ba11c6 2024-04-20T22:17:50.166 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (39m) 4m ago 56m 6665k - 19.0.0-3133-ga8c84f59 76bc6b91b726 20f746fa1bba 2024-04-20T22:17:50.166 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (43m) 4m ago 57m 75.5M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T22:17:50.166 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (35m) 4m ago 50m 22.9M - 19.0.0-3133-ga8c84f59 76bc6b91b726 de80bc90a683 2024-04-20T22:17:50.166 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (35m) 4m ago 50m 23.0M - 19.0.0-3133-ga8c84f59 76bc6b91b726 cd793b7ee98e 2024-04-20T22:17:50.167 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (19m) 4m ago 50m 18.5M - 19.0.0-3133-ga8c84f59 76bc6b91b726 0e7b0bc4d1ab 2024-04-20T22:17:50.167 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (4m) 4m ago 50m 11.4M - 19.0.0-3133-ga8c84f59 76bc6b91b726 0af732ac9e06 2024-04-20T22:17:50.167 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (46m) 4m ago 59m 517M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T22:17:50.167 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (45m) 4m ago 56m 460M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T22:17:50.167 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (41m) 4m ago 59m 86.3M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 019eb1479485 2024-04-20T22:17:50.167 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (40m) 4m ago 56m 58.0M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 55aac560aab6 2024-04-20T22:17:50.167 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (44m) 4m ago 57m 8766k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T22:17:50.167 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (44m) 4m ago 56m 8711k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T22:17:50.167 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (39m) 4m ago 55m 91.3M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 ee2bb901f26e 2024-04-20T22:17:50.167 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (38m) 4m ago 54m 94.9M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 9075152d4162 2024-04-20T22:17:50.167 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (38m) 4m ago 53m 78.5M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 2d4f3e38dc0e 2024-04-20T22:17:50.167 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (37m) 4m ago 53m 94.1M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 8f97b17ae0eb 2024-04-20T22:17:50.167 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (36m) 4m ago 52m 96.1M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 653d8079967d 2024-04-20T22:17:50.167 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (36m) 4m ago 52m 90.9M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 596dded14c12 2024-04-20T22:17:50.167 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (44m) 4m ago 56m 90.4M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T22:17:50.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:50.168+0000 7f9101800700 1 -- 172.21.15.62:0/25568104 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f90e006ecc0 msgr2=0x7f90e0071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:17:50.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:50.168+0000 7f9101800700 1 --2- 172.21.15.62:0/25568104 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f90e006ecc0 0x7f90e0071170 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f90e8000c00 tx=0x7f90e8005fd0 comp rx=0 tx=0).stop 2024-04-20T22:17:50.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:50.168+0000 7f9101800700 1 -- 172.21.15.62:0/25568104 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90fc068540 msgr2=0x7f90fc1980b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:17:50.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:50.168+0000 7f9101800700 1 --2- 172.21.15.62:0/25568104 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90fc068540 0x7f90fc1980b0 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f90ec00c8f0 tx=0x7f90ec00ccb0 comp rx=0 tx=0).stop 2024-04-20T22:17:50.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:50.168+0000 7f9101800700 1 -- 172.21.15.62:0/25568104 shutdown_connections 2024-04-20T22:17:50.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:50.168+0000 7f9101800700 1 --2- 172.21.15.62:0/25568104 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f90e006ecc0 0x7f90e0071170 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:17:50.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:50.168+0000 7f9101800700 1 --2- 172.21.15.62:0/25568104 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90fc068540 0x7f90fc1980b0 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:17:50.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:50.168+0000 7f9101800700 1 --2- 172.21.15.62:0/25568104 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f90fc1085b0 0x7f90fc1985f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:17:50.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:50.168+0000 7f9101800700 1 -- 172.21.15.62:0/25568104 >> 172.21.15.62:0/25568104 conn(0x7f90fc075130 msgr2=0x7f90fc106390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:17:50.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:50.169+0000 7f9101800700 1 -- 172.21.15.62:0/25568104 shutdown_connections 2024-04-20T22:17:50.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:50.169+0000 7f9101800700 1 -- 172.21.15.62:0/25568104 wait complete. 2024-04-20T22:17:50.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:50.331+0000 7f8d36400700 1 -- 172.21.15.62:0/3723653695 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d300730f0 msgr2=0x7f8d300734c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:17:50.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:50.331+0000 7f8d36400700 1 --2- 172.21.15.62:0/3723653695 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d300730f0 0x7f8d300734c0 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7f8d20009910 tx=0x7f8d20009c20 comp rx=0 tx=0).stop 2024-04-20T22:17:50.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:50.332+0000 7f8d36400700 1 -- 172.21.15.62:0/3723653695 shutdown_connections 2024-04-20T22:17:50.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:50.332+0000 7f8d36400700 1 --2- 172.21.15.62:0/3723653695 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d30073a00 0x7f8d3010c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:17:50.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:50.332+0000 7f8d36400700 1 --2- 172.21.15.62:0/3723653695 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d300730f0 0x7f8d300734c0 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:17:50.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:50.332+0000 7f8d36400700 1 -- 172.21.15.62:0/3723653695 >> 172.21.15.62:0/3723653695 conn(0x7f8d300fbda0 msgr2=0x7f8d300fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:17:50.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:50.332+0000 7f8d36400700 1 -- 172.21.15.62:0/3723653695 shutdown_connections 2024-04-20T22:17:50.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:50.332+0000 7f8d36400700 1 -- 172.21.15.62:0/3723653695 wait complete. 2024-04-20T22:17:50.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:50.333+0000 7f8d36400700 1 Processor -- start 2024-04-20T22:17:50.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:50.333+0000 7f8d36400700 1 -- start start 2024-04-20T22:17:50.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:50.333+0000 7f8d36400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d300730f0 0x7f8d30197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:17:50.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:50.333+0000 7f8d36400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d30073a00 0x7f8d30198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:17:50.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:50.333+0000 7f8d36400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d3019c970 con 0x7f8d300730f0 2024-04-20T22:17:50.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:50.333+0000 7f8d36400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d3019cae0 con 0x7f8d30073a00 2024-04-20T22:17:50.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:50.333+0000 7f8d2f400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d300730f0 0x7f8d30197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:17:50.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:50.333+0000 7f8d2f400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d300730f0 0x7f8d30197fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44904/0 (socket says 172.21.15.62:44904) 2024-04-20T22:17:50.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:50.334+0000 7f8d2f400700 1 -- 172.21.15.62:0/3621484258 learned_addr learned my addr 172.21.15.62:0/3621484258 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:17:50.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:50.334+0000 7f8d2f400700 1 -- 172.21.15.62:0/3621484258 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d30073a00 msgr2=0x7f8d30198630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:17:50.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:50.334+0000 7f8d2ea00700 1 --2- 172.21.15.62:0/3621484258 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d30073a00 0x7f8d30198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:17:50.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:50.334+0000 7f8d2f400700 1 --2- 172.21.15.62:0/3621484258 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d30073a00 0x7f8d30198630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:17:50.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:50.334+0000 7f8d2f400700 1 -- 172.21.15.62:0/3621484258 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d200095f0 con 0x7f8d300730f0 2024-04-20T22:17:50.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:50.334+0000 7f8d2ea00700 1 --2- 172.21.15.62:0/3621484258 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d30073a00 0x7f8d30198630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T22:17:50.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:50.334+0000 7f8d2f400700 1 --2- 172.21.15.62:0/3621484258 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d300730f0 0x7f8d30197fc0 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f8d20006010 tx=0x7f8d2000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:17:50.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:50.335+0000 7f8d27e00700 1 -- 172.21.15.62:0/3621484258 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8d2001d070 con 0x7f8d300730f0 2024-04-20T22:17:50.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:50.335+0000 7f8d27e00700 1 -- 172.21.15.62:0/3621484258 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8d20004fb0 con 0x7f8d300730f0 2024-04-20T22:17:50.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:50.335+0000 7f8d27e00700 1 -- 172.21.15.62:0/3621484258 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8d20022ea0 con 0x7f8d300730f0 2024-04-20T22:17:50.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:50.335+0000 7f8d36400700 1 -- 172.21.15.62:0/3621484258 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d3019cdc0 con 0x7f8d300730f0 2024-04-20T22:17:50.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:50.335+0000 7f8d36400700 1 -- 172.21.15.62:0/3621484258 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d3019d310 con 0x7f8d300730f0 2024-04-20T22:17:50.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:50.336+0000 7f8d36400700 1 -- 172.21.15.62:0/3621484258 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d30109c80 con 0x7f8d300730f0 2024-04-20T22:17:50.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:50.337+0000 7f8d27e00700 1 -- 172.21.15.62:0/3621484258 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f8d20022470 con 0x7f8d300730f0 2024-04-20T22:17:50.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:50.338+0000 7f8d27e00700 1 --2- 172.21.15.62:0/3621484258 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f8d1406ecc0 0x7f8d14071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:17:50.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:50.339+0000 7f8d27e00700 1 -- 172.21.15.62:0/3621484258 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f8d200903c0 con 0x7f8d300730f0 2024-04-20T22:17:50.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:50.339+0000 7f8d2ea00700 1 --2- 172.21.15.62:0/3621484258 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f8d1406ecc0 0x7f8d14071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:17:50.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:50.339+0000 7f8d2ea00700 1 --2- 172.21.15.62:0/3621484258 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f8d1406ecc0 0x7f8d14071170 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f8d1c005fd0 tx=0x7f8d1c005dc0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:17:50.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:50.343+0000 7f8d27e00700 1 -- 172.21.15.62:0/3621484258 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f8d20059200 con 0x7f8d300730f0 2024-04-20T22:17:50.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:50.670+0000 7f8d36400700 1 -- 172.21.15.62:0/3621484258 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f8d3004f2a0 con 0x7f8d300730f0 2024-04-20T22:17:50.671 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:17:50 smithi062 ceph-mon[76322]: from='client.36244 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:17:50.671 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:17:50 smithi062 ceph-mon[76322]: pgmap v1279: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:17:50.671 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:17:50 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:17:50.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:50.671+0000 7f8d27e00700 1 -- 172.21.15.62:0/3621484258 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f8d2005c8c0 con 0x7f8d300730f0 2024-04-20T22:17:50.672 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:17:50.672 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T22:17:50.672 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:17:50.672 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:17:50.672 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T22:17:50.672 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:17:50.672 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:17:50.672 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T22:17:50.672 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 6 2024-04-20T22:17:50.672 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:17:50.673 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T22:17:50.673 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 4 2024-04-20T22:17:50.673 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:17:50.673 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T22:17:50.673 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 14 2024-04-20T22:17:50.673 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T22:17:50.673 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:17:50.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:50.674+0000 7f8d36400700 1 -- 172.21.15.62:0/3621484258 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f8d1406ecc0 msgr2=0x7f8d14071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:17:50.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:50.674+0000 7f8d36400700 1 --2- 172.21.15.62:0/3621484258 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f8d1406ecc0 0x7f8d14071170 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f8d1c005fd0 tx=0x7f8d1c005dc0 comp rx=0 tx=0).stop 2024-04-20T22:17:50.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:50.674+0000 7f8d36400700 1 -- 172.21.15.62:0/3621484258 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d300730f0 msgr2=0x7f8d30197fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:17:50.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:50.674+0000 7f8d36400700 1 --2- 172.21.15.62:0/3621484258 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d300730f0 0x7f8d30197fc0 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f8d20006010 tx=0x7f8d2000be20 comp rx=0 tx=0).stop 2024-04-20T22:17:50.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:50.674+0000 7f8d36400700 1 -- 172.21.15.62:0/3621484258 shutdown_connections 2024-04-20T22:17:50.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:50.674+0000 7f8d36400700 1 --2- 172.21.15.62:0/3621484258 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f8d1406ecc0 0x7f8d14071170 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:17:50.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:50.674+0000 7f8d36400700 1 --2- 172.21.15.62:0/3621484258 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d300730f0 0x7f8d30197fc0 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:17:50.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:50.674+0000 7f8d36400700 1 --2- 172.21.15.62:0/3621484258 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d30073a00 0x7f8d30198630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:17:50.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:50.674+0000 7f8d36400700 1 -- 172.21.15.62:0/3621484258 >> 172.21.15.62:0/3621484258 conn(0x7f8d300fbda0 msgr2=0x7f8d30106dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:17:50.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:50.675+0000 7f8d36400700 1 -- 172.21.15.62:0/3621484258 shutdown_connections 2024-04-20T22:17:50.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:50.675+0000 7f8d36400700 1 -- 172.21.15.62:0/3621484258 wait complete. 2024-04-20T22:17:50.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:17:50 smithi189 ceph-mon[57022]: from='client.36244 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:17:50.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:17:50 smithi189 ceph-mon[57022]: pgmap v1279: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:17:50.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:17:50 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:17:50.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:50.835+0000 7f18efa00700 1 -- 172.21.15.62:0/2121849617 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f18e8073090 msgr2=0x7f18e8073460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:17:50.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:50.835+0000 7f18efa00700 1 --2- 172.21.15.62:0/2121849617 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f18e8073090 0x7f18e8073460 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f18d8009940 tx=0x7f18d8009c50 comp rx=0 tx=0).stop 2024-04-20T22:17:50.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:50.836+0000 7f18efa00700 1 -- 172.21.15.62:0/2121849617 shutdown_connections 2024-04-20T22:17:50.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:50.836+0000 7f18efa00700 1 --2- 172.21.15.62:0/2121849617 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18e80739a0 0x7f18e810c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:17:50.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:50.836+0000 7f18efa00700 1 --2- 172.21.15.62:0/2121849617 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f18e8073090 0x7f18e8073460 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:17:50.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:50.836+0000 7f18efa00700 1 -- 172.21.15.62:0/2121849617 >> 172.21.15.62:0/2121849617 conn(0x7f18e80fbda0 msgr2=0x7f18e80fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:17:50.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:50.836+0000 7f18efa00700 1 -- 172.21.15.62:0/2121849617 shutdown_connections 2024-04-20T22:17:50.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:50.836+0000 7f18efa00700 1 -- 172.21.15.62:0/2121849617 wait complete. 2024-04-20T22:17:50.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:50.837+0000 7f18efa00700 1 Processor -- start 2024-04-20T22:17:50.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:50.837+0000 7f18efa00700 1 -- start start 2024-04-20T22:17:50.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:50.838+0000 7f18efa00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18e8073090 0x7f18e81981c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:17:50.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:50.838+0000 7f18efa00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f18e80739a0 0x7f18e8198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:17:50.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:50.838+0000 7f18efa00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18e8198de0 con 0x7f18e80739a0 2024-04-20T22:17:50.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:50.838+0000 7f18efa00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18e819cb70 con 0x7f18e8073090 2024-04-20T22:17:50.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:50.838+0000 7f18dfe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f18e80739a0 0x7f18e8198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:17:50.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:50.838+0000 7f18dfe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f18e80739a0 0x7f18e8198700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44912/0 (socket says 172.21.15.62:44912) 2024-04-20T22:17:50.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:50.838+0000 7f18dfe00700 1 -- 172.21.15.62:0/4132227426 learned_addr learned my addr 172.21.15.62:0/4132227426 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:17:50.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:50.838+0000 7f18ed000700 1 --2- 172.21.15.62:0/4132227426 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18e8073090 0x7f18e81981c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:17:50.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:50.839+0000 7f18dfe00700 1 -- 172.21.15.62:0/4132227426 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18e8073090 msgr2=0x7f18e81981c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:17:50.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:50.839+0000 7f18dfe00700 1 --2- 172.21.15.62:0/4132227426 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18e8073090 0x7f18e81981c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:17:50.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:50.839+0000 7f18dfe00700 1 -- 172.21.15.62:0/4132227426 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18d80095f0 con 0x7f18e80739a0 2024-04-20T22:17:50.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:50.839+0000 7f18dfe00700 1 --2- 172.21.15.62:0/4132227426 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f18e80739a0 0x7f18e8198700 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f18d000b4b0 tx=0x7f18d000b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:17:50.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:50.839+0000 7f18e6000700 1 -- 172.21.15.62:0/4132227426 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f18d0011800 con 0x7f18e80739a0 2024-04-20T22:17:50.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:50.839+0000 7f18e6000700 1 -- 172.21.15.62:0/4132227426 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f18d0011e20 con 0x7f18e80739a0 2024-04-20T22:17:50.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:50.839+0000 7f18efa00700 1 -- 172.21.15.62:0/4132227426 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18e819ce50 con 0x7f18e80739a0 2024-04-20T22:17:50.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:50.839+0000 7f18e6000700 1 -- 172.21.15.62:0/4132227426 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f18d000f2e0 con 0x7f18e80739a0 2024-04-20T22:17:50.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:50.839+0000 7f18efa00700 1 -- 172.21.15.62:0/4132227426 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18e819d3a0 con 0x7f18e80739a0 2024-04-20T22:17:50.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:50.841+0000 7f18efa00700 1 -- 172.21.15.62:0/4132227426 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18e8109ce0 con 0x7f18e80739a0 2024-04-20T22:17:50.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:50.842+0000 7f18e6000700 1 -- 172.21.15.62:0/4132227426 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f18d0010380 con 0x7f18e80739a0 2024-04-20T22:17:50.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:50.843+0000 7f18e6000700 1 --2- 172.21.15.62:0/4132227426 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f18cc06ecc0 0x7f18cc071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:17:50.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:50.843+0000 7f18e6000700 1 -- 172.21.15.62:0/4132227426 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f18d008ea30 con 0x7f18e80739a0 2024-04-20T22:17:50.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:50.843+0000 7f18ed000700 1 --2- 172.21.15.62:0/4132227426 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f18cc06ecc0 0x7f18cc071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:17:50.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:50.844+0000 7f18ed000700 1 --2- 172.21.15.62:0/4132227426 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f18cc06ecc0 0x7f18cc071170 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f18d8006010 tx=0x7f18d8005a90 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:17:50.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:50.847+0000 7f18e6000700 1 -- 172.21.15.62:0/4132227426 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f18d0057870 con 0x7f18e80739a0 2024-04-20T22:17:51.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:51.128+0000 7f18efa00700 1 -- 172.21.15.62:0/4132227426 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f18e819d670 con 0x7f18e80739a0 2024-04-20T22:17:51.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:51.129+0000 7f18e6000700 1 -- 172.21.15.62:0/4132227426 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 29 v29) v1 ==== 76+0+1887 (secure 0 0 0) 0x7f18d005af30 con 0x7f18e80739a0 2024-04-20T22:17:51.130 INFO:teuthology.orchestra.run.smithi062.stdout:e29 2024-04-20T22:17:51.130 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T22:17:51.130 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:17:51.130 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T22:17:51.130 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:17:51.130 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T22:17:51.130 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T22:17:51.130 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 20 2024-04-20T22:17:51.130 INFO:teuthology.orchestra.run.smithi062.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-20T22:17:51.130 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T22:17:51.131 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T21:42:01.559061+0000 2024-04-20T22:17:51.131 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T22:17:51.131 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T22:17:51.131 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T22:17:51.131 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T22:17:51.131 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T22:17:51.131 INFO:teuthology.orchestra.run.smithi062.stdout:max_xattr_size 65536 2024-04-20T22:17:51.131 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T22:17:51.131 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T22:17:51.131 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 81 2024-04-20T22:17:51.131 INFO:teuthology.orchestra.run.smithi062.stdout:snap_rstat 1 2024-04-20T22:17:51.131 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:17:51.131 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 2 2024-04-20T22:17:51.131 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1 2024-04-20T22:17:51.131 INFO:teuthology.orchestra.run.smithi062.stdout:up {} 2024-04-20T22:17:51.131 INFO:teuthology.orchestra.run.smithi062.stdout:failed 0,1 2024-04-20T22:17:51.132 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T22:17:51.132 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T22:17:51.132 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T22:17:51.132 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T22:17:51.132 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data enabled 2024-04-20T22:17:51.132 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T22:17:51.132 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T22:17:51.132 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 1 2024-04-20T22:17:51.132 INFO:teuthology.orchestra.run.smithi062.stdout:qdb_cluster leader: 0 members: 2024-04-20T22:17:51.132 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:17:51.132 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:17:51.132 INFO:teuthology.orchestra.run.smithi062.stdout:Standby daemons: 2024-04-20T22:17:51.132 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:17:51.132 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ryqblb{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6826/4113916205,v1:172.21.15.62:6827/4113916205] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:17:51.132 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ozizwi{-1:34476} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6828/3801975165,v1:172.21.15.62:6829/3801975165] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:17:51.132 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.dvimer{-1:44727} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6824/3576699232,v1:172.21.15.189:6825/3576699232] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:17:51.133 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.xbgspw{-1:45075} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6826/212905792,v1:172.21.15.189:6827/212905792] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:17:51.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:51.132+0000 7f18efa00700 1 -- 172.21.15.62:0/4132227426 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f18cc06ecc0 msgr2=0x7f18cc071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:17:51.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:51.132+0000 7f18efa00700 1 --2- 172.21.15.62:0/4132227426 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f18cc06ecc0 0x7f18cc071170 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f18d8006010 tx=0x7f18d8005a90 comp rx=0 tx=0).stop 2024-04-20T22:17:51.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:51.132+0000 7f18efa00700 1 -- 172.21.15.62:0/4132227426 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f18e80739a0 msgr2=0x7f18e8198700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:17:51.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:51.132+0000 7f18efa00700 1 --2- 172.21.15.62:0/4132227426 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f18e80739a0 0x7f18e8198700 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f18d000b4b0 tx=0x7f18d000b870 comp rx=0 tx=0).stop 2024-04-20T22:17:51.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:51.132+0000 7f18efa00700 1 -- 172.21.15.62:0/4132227426 shutdown_connections 2024-04-20T22:17:51.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:51.132+0000 7f18efa00700 1 --2- 172.21.15.62:0/4132227426 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f18cc06ecc0 0x7f18cc071170 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:17:51.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:51.132+0000 7f18efa00700 1 --2- 172.21.15.62:0/4132227426 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18e8073090 0x7f18e81981c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:17:51.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:51.132+0000 7f18efa00700 1 --2- 172.21.15.62:0/4132227426 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f18e80739a0 0x7f18e8198700 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:17:51.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:51.132+0000 7f18efa00700 1 -- 172.21.15.62:0/4132227426 >> 172.21.15.62:0/4132227426 conn(0x7f18e80fbda0 msgr2=0x7f18e8106e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:17:51.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:51.132+0000 7f18efa00700 1 -- 172.21.15.62:0/4132227426 shutdown_connections 2024-04-20T22:17:51.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:51.132+0000 7f18efa00700 1 -- 172.21.15.62:0/4132227426 wait complete. 2024-04-20T22:17:51.134 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 29 2024-04-20T22:17:51.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:51.283+0000 7fa569600700 1 -- 172.21.15.62:0/3287958667 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa55c0054b0 con 0x7fa56c108470 2024-04-20T22:17:51.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:51.284+0000 7fa571c00700 1 -- 172.21.15.62:0/3287958667 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa56c108470 msgr2=0x7fa56c108840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:17:51.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:51.284+0000 7fa571c00700 1 --2- 172.21.15.62:0/3287958667 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa56c108470 0x7fa56c108840 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7fa55c009910 tx=0x7fa55c009c20 comp rx=0 tx=0).stop 2024-04-20T22:17:51.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:51.284+0000 7fa571c00700 1 -- 172.21.15.62:0/3287958667 shutdown_connections 2024-04-20T22:17:51.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:51.284+0000 7fa571c00700 1 --2- 172.21.15.62:0/3287958667 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa56c102470 0x7fa56c1028e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:17:51.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:51.284+0000 7fa571c00700 1 --2- 172.21.15.62:0/3287958667 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa56c108470 0x7fa56c108840 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:17:51.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:51.284+0000 7fa571c00700 1 -- 172.21.15.62:0/3287958667 >> 172.21.15.62:0/3287958667 conn(0x7fa56c0fdf90 msgr2=0x7fa56c1003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:17:51.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:51.284+0000 7fa571c00700 1 -- 172.21.15.62:0/3287958667 shutdown_connections 2024-04-20T22:17:51.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:51.284+0000 7fa571c00700 1 -- 172.21.15.62:0/3287958667 wait complete. 2024-04-20T22:17:51.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:51.285+0000 7fa571c00700 1 Processor -- start 2024-04-20T22:17:51.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:51.285+0000 7fa571c00700 1 -- start start 2024-04-20T22:17:51.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:51.285+0000 7fa571c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa56c102470 0x7fa56c198210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:17:51.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:51.285+0000 7fa571c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa56c108470 0x7fa56c198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:17:51.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:51.285+0000 7fa571c00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa56c198e30 con 0x7fa56c102470 2024-04-20T22:17:51.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:51.285+0000 7fa571c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa56c19cbc0 con 0x7fa56c108470 2024-04-20T22:17:51.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:51.286+0000 7fa56aa00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa56c102470 0x7fa56c198210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:17:51.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:51.286+0000 7fa56aa00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa56c102470 0x7fa56c198210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44936/0 (socket says 172.21.15.62:44936) 2024-04-20T22:17:51.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:51.286+0000 7fa56aa00700 1 -- 172.21.15.62:0/1306544369 learned_addr learned my addr 172.21.15.62:0/1306544369 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:17:51.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:51.286+0000 7fa56a000700 1 --2- 172.21.15.62:0/1306544369 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa56c108470 0x7fa56c198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:17:51.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:51.286+0000 7fa56aa00700 1 -- 172.21.15.62:0/1306544369 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa56c108470 msgr2=0x7fa56c198750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:17:51.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:51.286+0000 7fa56aa00700 1 --2- 172.21.15.62:0/1306544369 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa56c108470 0x7fa56c198750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:17:51.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:51.286+0000 7fa56aa00700 1 -- 172.21.15.62:0/1306544369 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa55c0095f0 con 0x7fa56c102470 2024-04-20T22:17:51.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:51.287+0000 7fa56aa00700 1 --2- 172.21.15.62:0/1306544369 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa56c102470 0x7fa56c198210 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7fa55c006010 tx=0x7fa55c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:17:51.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:51.287+0000 7fa563400700 1 -- 172.21.15.62:0/1306544369 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa55c01d070 con 0x7fa56c102470 2024-04-20T22:17:51.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:51.287+0000 7fa563400700 1 -- 172.21.15.62:0/1306544369 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa55c004fb0 con 0x7fa56c102470 2024-04-20T22:17:51.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:51.287+0000 7fa563400700 1 -- 172.21.15.62:0/1306544369 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa55c022ea0 con 0x7fa56c102470 2024-04-20T22:17:51.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:51.287+0000 7fa571c00700 1 -- 172.21.15.62:0/1306544369 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa56c19ce40 con 0x7fa56c102470 2024-04-20T22:17:51.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:51.287+0000 7fa571c00700 1 -- 172.21.15.62:0/1306544369 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa56c19d330 con 0x7fa56c102470 2024-04-20T22:17:51.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:51.288+0000 7fa571c00700 1 -- 172.21.15.62:0/1306544369 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa56c10a9e0 con 0x7fa56c102470 2024-04-20T22:17:51.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:51.289+0000 7fa563400700 1 -- 172.21.15.62:0/1306544369 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fa55c022470 con 0x7fa56c102470 2024-04-20T22:17:51.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:51.290+0000 7fa563400700 1 --2- 172.21.15.62:0/1306544369 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa55006ecc0 0x7fa550071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:17:51.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:51.290+0000 7fa563400700 1 -- 172.21.15.62:0/1306544369 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fa55c0903f0 con 0x7fa56c102470 2024-04-20T22:17:51.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:51.290+0000 7fa56a000700 1 --2- 172.21.15.62:0/1306544369 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa55006ecc0 0x7fa550071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:17:51.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:51.291+0000 7fa56a000700 1 --2- 172.21.15.62:0/1306544369 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa55006ecc0 0x7fa550071170 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fa558006fd0 tx=0x7fa558008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:17:51.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:51.295+0000 7fa563400700 1 -- 172.21.15.62:0/1306544369 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fa55c059230 con 0x7fa56c102470 2024-04-20T22:17:51.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:51.543+0000 7fa571c00700 1 -- 172.21.15.62:0/1306544369 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa56c1995e0 con 0x7fa55006ecc0 2024-04-20T22:17:51.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:51.545+0000 7fa563400700 1 -- 172.21.15.62:0/1306544369 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fa56c1995e0 con 0x7fa55006ecc0 2024-04-20T22:17:51.545 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:17:51.545 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T22:17:51.545 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T22:17:51.545 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-20T22:17:51.545 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T22:17:51.545 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr", 2024-04-20T22:17:51.545 INFO:teuthology.orchestra.run.smithi062.stdout: "osd", 2024-04-20T22:17:51.545 INFO:teuthology.orchestra.run.smithi062.stdout: "mds", 2024-04-20T22:17:51.545 INFO:teuthology.orchestra.run.smithi062.stdout: "crash", 2024-04-20T22:17:51.545 INFO:teuthology.orchestra.run.smithi062.stdout: "mon" 2024-04-20T22:17:51.545 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T22:17:51.546 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "16/23 daemons upgraded", 2024-04-20T22:17:51.546 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading mds daemons", 2024-04-20T22:17:51.546 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T22:17:51.546 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:17:51.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:51.547+0000 7fa571c00700 1 -- 172.21.15.62:0/1306544369 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa55006ecc0 msgr2=0x7fa550071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:17:51.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:51.547+0000 7fa571c00700 1 --2- 172.21.15.62:0/1306544369 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa55006ecc0 0x7fa550071170 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fa558006fd0 tx=0x7fa558008040 comp rx=0 tx=0).stop 2024-04-20T22:17:51.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:51.547+0000 7fa571c00700 1 -- 172.21.15.62:0/1306544369 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa56c102470 msgr2=0x7fa56c198210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:17:51.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:51.547+0000 7fa571c00700 1 --2- 172.21.15.62:0/1306544369 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa56c102470 0x7fa56c198210 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7fa55c006010 tx=0x7fa55c00be20 comp rx=0 tx=0).stop 2024-04-20T22:17:51.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:51.547+0000 7fa571c00700 1 -- 172.21.15.62:0/1306544369 shutdown_connections 2024-04-20T22:17:51.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:51.547+0000 7fa571c00700 1 --2- 172.21.15.62:0/1306544369 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa55006ecc0 0x7fa550071170 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:17:51.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:51.547+0000 7fa571c00700 1 --2- 172.21.15.62:0/1306544369 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa56c102470 0x7fa56c198210 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:17:51.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:51.547+0000 7fa571c00700 1 --2- 172.21.15.62:0/1306544369 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa56c108470 0x7fa56c198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:17:51.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:51.548+0000 7fa571c00700 1 -- 172.21.15.62:0/1306544369 >> 172.21.15.62:0/1306544369 conn(0x7fa56c0fdf90 msgr2=0x7fa56c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:17:51.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:51.548+0000 7fa571c00700 1 -- 172.21.15.62:0/1306544369 shutdown_connections 2024-04-20T22:17:51.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:51.548+0000 7fa571c00700 1 -- 172.21.15.62:0/1306544369 wait complete. 2024-04-20T22:17:51.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:51.704+0000 7f4eee200700 1 -- 172.21.15.62:0/1520617204 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4ee81002d0 msgr2=0x7f4ee81006a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:17:51.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:51.704+0000 7f4eee200700 1 --2- 172.21.15.62:0/1520617204 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4ee81002d0 0x7f4ee81006a0 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f4edc009910 tx=0x7f4edc009c20 comp rx=0 tx=0).stop 2024-04-20T22:17:51.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:51.704+0000 7f4eee200700 1 -- 172.21.15.62:0/1520617204 shutdown_connections 2024-04-20T22:17:51.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:51.705+0000 7f4eee200700 1 --2- 172.21.15.62:0/1520617204 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ee8100c70 0x7f4ee8104c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:17:51.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:51.705+0000 7f4eee200700 1 --2- 172.21.15.62:0/1520617204 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4ee81002d0 0x7f4ee81006a0 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:17:51.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:51.705+0000 7f4eee200700 1 -- 172.21.15.62:0/1520617204 >> 172.21.15.62:0/1520617204 conn(0x7f4ee80fbbb0 msgr2=0x7f4ee80fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:17:51.705 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:17:51 smithi062 ceph-mon[76322]: from='client.36248 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:17:51.705 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:17:51 smithi062 ceph-mon[76322]: from='client.36252 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:17:51.705 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:17:51 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/3621484258' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:17:51.705 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:17:51 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/4132227426' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:17:51.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:51.705+0000 7f4eee200700 1 -- 172.21.15.62:0/1520617204 shutdown_connections 2024-04-20T22:17:51.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:51.705+0000 7f4eee200700 1 -- 172.21.15.62:0/1520617204 wait complete. 2024-04-20T22:17:51.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:51.706+0000 7f4eee200700 1 Processor -- start 2024-04-20T22:17:51.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:51.707+0000 7f4eee200700 1 -- start start 2024-04-20T22:17:51.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:51.707+0000 7f4eee200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4ee81002d0 0x7f4ee8197ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:17:51.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:51.707+0000 7f4eee200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ee8100c70 0x7f4ee8198400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:17:51.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:51.707+0000 7f4eee200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ee8198ae0 con 0x7f4ee81002d0 2024-04-20T22:17:51.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:51.707+0000 7f4eee200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ee819c870 con 0x7f4ee8100c70 2024-04-20T22:17:51.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:51.707+0000 7f4eece00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4ee81002d0 0x7f4ee8197ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:17:51.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:51.707+0000 7f4eece00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4ee81002d0 0x7f4ee8197ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53158/0 (socket says 172.21.15.62:53158) 2024-04-20T22:17:51.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:51.707+0000 7f4eece00700 1 -- 172.21.15.62:0/208122529 learned_addr learned my addr 172.21.15.62:0/208122529 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:17:51.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:51.708+0000 7f4ee7e00700 1 --2- 172.21.15.62:0/208122529 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ee8100c70 0x7f4ee8198400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:17:51.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:51.708+0000 7f4eece00700 1 -- 172.21.15.62:0/208122529 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ee8100c70 msgr2=0x7f4ee8198400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:17:51.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:51.708+0000 7f4eece00700 1 --2- 172.21.15.62:0/208122529 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ee8100c70 0x7f4ee8198400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:17:51.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:51.708+0000 7f4eece00700 1 -- 172.21.15.62:0/208122529 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4edc0095f0 con 0x7f4ee81002d0 2024-04-20T22:17:51.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:51.708+0000 7f4eece00700 1 --2- 172.21.15.62:0/208122529 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4ee81002d0 0x7f4ee8197ec0 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f4edc0092e0 tx=0x7f4edc00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:17:51.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:51.709+0000 7f4ee5600700 1 -- 172.21.15.62:0/208122529 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4edc01d070 con 0x7f4ee81002d0 2024-04-20T22:17:51.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:51.709+0000 7f4eee200700 1 -- 172.21.15.62:0/208122529 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ee819caf0 con 0x7f4ee81002d0 2024-04-20T22:17:51.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:51.709+0000 7f4ee5600700 1 -- 172.21.15.62:0/208122529 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4edc005e00 con 0x7f4ee81002d0 2024-04-20T22:17:51.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:51.709+0000 7f4ee5600700 1 -- 172.21.15.62:0/208122529 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4edc017400 con 0x7f4ee81002d0 2024-04-20T22:17:51.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:51.709+0000 7f4eee200700 1 -- 172.21.15.62:0/208122529 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ee819cfe0 con 0x7f4ee81002d0 2024-04-20T22:17:51.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:51.710+0000 7f4eee200700 1 -- 172.21.15.62:0/208122529 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ee81043c0 con 0x7f4ee81002d0 2024-04-20T22:17:51.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:51.711+0000 7f4ee5600700 1 -- 172.21.15.62:0/208122529 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f4edc00f460 con 0x7f4ee81002d0 2024-04-20T22:17:51.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:51.712+0000 7f4ee5600700 1 --2- 172.21.15.62:0/208122529 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f4ed006ecc0 0x7f4ed0071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:17:51.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:51.712+0000 7f4ee5600700 1 -- 172.21.15.62:0/208122529 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f4edc0903f0 con 0x7f4ee81002d0 2024-04-20T22:17:51.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:51.713+0000 7f4ee7e00700 1 --2- 172.21.15.62:0/208122529 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f4ed006ecc0 0x7f4ed0071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:17:51.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:51.713+0000 7f4ee7e00700 1 --2- 172.21.15.62:0/208122529 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f4ed006ecc0 0x7f4ed0071170 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f4ed4005950 tx=0x7f4ed40058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:17:51.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:51.717+0000 7f4ee5600700 1 -- 172.21.15.62:0/208122529 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f4edc059100 con 0x7f4ee81002d0 2024-04-20T22:17:51.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:17:51 smithi189 ceph-mon[57022]: from='client.36248 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:17:51.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:17:51 smithi189 ceph-mon[57022]: from='client.36252 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:17:51.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:17:51 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/3621484258' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:17:51.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:17:51 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/4132227426' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:17:52.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:52.045+0000 7f4eee200700 1 -- 172.21.15.62:0/208122529 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f4ee8062380 con 0x7f4ee81002d0 2024-04-20T22:17:52.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:52.046+0000 7f4ee5600700 1 -- 172.21.15.62:0/208122529 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7f4ee8062380 con 0x7f4ee81002d0 2024-04-20T22:17:52.046 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-20T22:17:52.046 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-20T22:17:52.047 INFO:teuthology.orchestra.run.smithi062.stdout: host smithi189 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi189/config 2024-04-20T22:17:52.047 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-20T22:17:52.047 INFO:teuthology.orchestra.run.smithi062.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi189 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp11670v2q:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:17:52.047 INFO:teuthology.orchestra.run.smithi062.stdout:Traceback (most recent call last): 2024-04-20T22:17:52.047 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-20T22:17:52.047 INFO:teuthology.orchestra.run.smithi062.stdout: return _run_code(code, main_globals, None, 2024-04-20T22:17:52.047 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-20T22:17:52.047 INFO:teuthology.orchestra.run.smithi062.stdout: exec(code, run_globals) 2024-04-20T22:17:52.047 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5618, in 2024-04-20T22:17:52.047 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5606, in main 2024-04-20T22:17:52.047 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 408, in _infer_config 2024-04-20T22:17:52.047 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 323, in _infer_fsid 2024-04-20T22:17:52.047 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 436, in _infer_image 2024-04-20T22:17:52.047 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 310, in _validate_fsid 2024-04-20T22:17:52.048 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 3359, in command_ceph_volume 2024-04-20T22:17:52.048 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-20T22:17:52.048 INFO:teuthology.orchestra.run.smithi062.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi189 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp11670v2q:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:17:52.048 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-20T22:17:52.048 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is degraded 2024-04-20T22:17:52.048 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T22:17:52.048 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T22:17:52.048 INFO:teuthology.orchestra.run.smithi062.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-20T22:17:52.048 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-20T22:17:52.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:52.049+0000 7f4eee200700 1 -- 172.21.15.62:0/208122529 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f4ed006ecc0 msgr2=0x7f4ed0071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:17:52.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:52.049+0000 7f4eee200700 1 --2- 172.21.15.62:0/208122529 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f4ed006ecc0 0x7f4ed0071170 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f4ed4005950 tx=0x7f4ed40058e0 comp rx=0 tx=0).stop 2024-04-20T22:17:52.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:52.049+0000 7f4eee200700 1 -- 172.21.15.62:0/208122529 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4ee81002d0 msgr2=0x7f4ee8197ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:17:52.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:52.049+0000 7f4eee200700 1 --2- 172.21.15.62:0/208122529 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4ee81002d0 0x7f4ee8197ec0 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f4edc0092e0 tx=0x7f4edc00be20 comp rx=0 tx=0).stop 2024-04-20T22:17:52.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:52.049+0000 7f4eee200700 1 -- 172.21.15.62:0/208122529 shutdown_connections 2024-04-20T22:17:52.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:52.049+0000 7f4eee200700 1 --2- 172.21.15.62:0/208122529 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f4ed006ecc0 0x7f4ed0071170 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:17:52.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:52.049+0000 7f4eee200700 1 --2- 172.21.15.62:0/208122529 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4ee81002d0 0x7f4ee8197ec0 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:17:52.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:52.049+0000 7f4eee200700 1 --2- 172.21.15.62:0/208122529 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ee8100c70 0x7f4ee8198400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:17:52.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:52.049+0000 7f4eee200700 1 -- 172.21.15.62:0/208122529 >> 172.21.15.62:0/208122529 conn(0x7f4ee80fbbb0 msgr2=0x7f4ee80fdf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:17:52.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:52.049+0000 7f4eee200700 1 -- 172.21.15.62:0/208122529 shutdown_connections 2024-04-20T22:17:52.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:17:52.050+0000 7f4eee200700 1 -- 172.21.15.62:0/208122529 wait complete. 2024-04-20T22:17:52.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:17:52 smithi189 ceph-mon[57022]: pgmap v1280: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:17:52.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:17:52 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/208122529' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:17:52.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:17:52 smithi062 ceph-mon[76322]: pgmap v1280: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:17:52.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:17:52 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/208122529' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:17:53.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:17:53 smithi189 ceph-mon[57022]: from='client.36264 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:17:53.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:17:53 smithi062 ceph-mon[76322]: from='client.36264 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:17:54.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:17:54 smithi189 ceph-mon[57022]: pgmap v1281: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:17:54.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:17:54 smithi062 ceph-mon[76322]: pgmap v1281: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:17:55.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:17:55 smithi062 ceph-mon[76322]: pgmap v1282: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:17:55.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:17:55 smithi189 ceph-mon[57022]: pgmap v1282: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:17:58.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:17:58 smithi062 ceph-mon[76322]: pgmap v1283: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:17:58.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:17:58 smithi189 ceph-mon[57022]: pgmap v1283: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:18:00.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:18:00 smithi189 ceph-mon[57022]: pgmap v1284: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:18:00.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:18:00 smithi062 ceph-mon[76322]: pgmap v1284: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:18:02.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:18:02 smithi189 ceph-mon[57022]: pgmap v1285: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:18:02.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:18:02 smithi062 ceph-mon[76322]: pgmap v1285: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:18:04.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:18:04 smithi189 ceph-mon[57022]: pgmap v1286: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:18:04.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:18:04 smithi062 ceph-mon[76322]: pgmap v1286: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:18:05.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:18:05 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:18:05.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:18:05 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:18:06.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:18:06 smithi189 ceph-mon[57022]: pgmap v1287: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:18:06.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:18:06 smithi062 ceph-mon[76322]: pgmap v1287: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:18:07.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:18:07 smithi189 ceph-mon[57022]: pgmap v1288: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:18:07.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:18:07 smithi062 ceph-mon[76322]: pgmap v1288: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:18:10.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:18:10 smithi189 ceph-mon[57022]: pgmap v1289: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:18:10.741 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:18:10 smithi062 ceph-mon[76322]: pgmap v1289: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:18:12.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:18:12 smithi189 ceph-mon[57022]: pgmap v1290: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:18:12.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:18:12 smithi062 ceph-mon[76322]: pgmap v1290: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:18:14.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:18:14 smithi189 ceph-mon[57022]: pgmap v1291: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:18:14.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:18:14 smithi062 ceph-mon[76322]: pgmap v1291: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:18:16.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:18:16 smithi189 ceph-mon[57022]: pgmap v1292: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:18:16.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:18:16 smithi062 ceph-mon[76322]: pgmap v1292: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:18:18.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:18:18 smithi189 ceph-mon[57022]: pgmap v1293: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:18:18.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:18:18 smithi062 ceph-mon[76322]: pgmap v1293: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:18:19.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:18:19 smithi189 ceph-mon[57022]: pgmap v1294: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:18:19.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:18:19 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:18:19.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:18:19 smithi062 ceph-mon[76322]: pgmap v1294: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:18:19.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:18:19 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:18:22.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:22.225+0000 7fa0b9200700 1 -- 172.21.15.62:0/3565370422 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0b41004e0 msgr2=0x7fa0b41008b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:18:22.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:22.225+0000 7fa0b9200700 1 --2- 172.21.15.62:0/3565370422 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0b41004e0 0x7fa0b41008b0 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7fa0a4009910 tx=0x7fa0a4009c20 comp rx=0 tx=0).stop 2024-04-20T22:18:22.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:22.226+0000 7fa0b9200700 1 -- 172.21.15.62:0/3565370422 shutdown_connections 2024-04-20T22:18:22.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:22.226+0000 7fa0b9200700 1 --2- 172.21.15.62:0/3565370422 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0b4100e80 0x7fa0b4104e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:18:22.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:22.226+0000 7fa0b9200700 1 --2- 172.21.15.62:0/3565370422 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0b41004e0 0x7fa0b41008b0 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:18:22.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:22.226+0000 7fa0b9200700 1 -- 172.21.15.62:0/3565370422 >> 172.21.15.62:0/3565370422 conn(0x7fa0b40fbda0 msgr2=0x7fa0b40fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:18:22.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:22.226+0000 7fa0b9200700 1 -- 172.21.15.62:0/3565370422 shutdown_connections 2024-04-20T22:18:22.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:22.226+0000 7fa0b9200700 1 -- 172.21.15.62:0/3565370422 wait complete. 2024-04-20T22:18:22.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:22.227+0000 7fa0b9200700 1 Processor -- start 2024-04-20T22:18:22.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:22.227+0000 7fa0b9200700 1 -- start start 2024-04-20T22:18:22.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:22.227+0000 7fa0b9200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0b41004e0 0x7fa0b4073050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:18:22.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:22.228+0000 7fa0b9200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0b4100e80 0x7fa0b4073590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:18:22.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:22.228+0000 7fa0b9200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b4073ad0 con 0x7fa0b4100e80 2024-04-20T22:18:22.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:22.228+0000 7fa0b9200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b4073c10 con 0x7fa0b41004e0 2024-04-20T22:18:22.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:22.228+0000 7fa0b1400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0b4100e80 0x7fa0b4073590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:18:22.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:22.228+0000 7fa0b1400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0b4100e80 0x7fa0b4073590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35650/0 (socket says 172.21.15.62:35650) 2024-04-20T22:18:22.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:22.228+0000 7fa0b1400700 1 -- 172.21.15.62:0/2037405659 learned_addr learned my addr 172.21.15.62:0/2037405659 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:18:22.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:22.228+0000 7fa0b1e00700 1 --2- 172.21.15.62:0/2037405659 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0b41004e0 0x7fa0b4073050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:18:22.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:22.228+0000 7fa0b1400700 1 -- 172.21.15.62:0/2037405659 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0b41004e0 msgr2=0x7fa0b4073050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:18:22.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:22.228+0000 7fa0b1400700 1 --2- 172.21.15.62:0/2037405659 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0b41004e0 0x7fa0b4073050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:18:22.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:22.228+0000 7fa0b1400700 1 -- 172.21.15.62:0/2037405659 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0a40095f0 con 0x7fa0b4100e80 2024-04-20T22:18:22.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:22.229+0000 7fa0b1400700 1 --2- 172.21.15.62:0/2037405659 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0b4100e80 0x7fa0b4073590 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7fa0a000b480 tx=0x7fa0a000b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:18:22.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:22.229+0000 7fa0aaa00700 1 -- 172.21.15.62:0/2037405659 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa0a00118c0 con 0x7fa0b4100e80 2024-04-20T22:18:22.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:22.229+0000 7fa0aaa00700 1 -- 172.21.15.62:0/2037405659 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa0a0010380 con 0x7fa0b4100e80 2024-04-20T22:18:22.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:22.229+0000 7fa0aaa00700 1 -- 172.21.15.62:0/2037405659 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa0a000f4f0 con 0x7fa0b4100e80 2024-04-20T22:18:22.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:22.229+0000 7fa0b9200700 1 -- 172.21.15.62:0/2037405659 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0b41a2180 con 0x7fa0b4100e80 2024-04-20T22:18:22.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:22.229+0000 7fa0b9200700 1 -- 172.21.15.62:0/2037405659 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0b41a2590 con 0x7fa0b4100e80 2024-04-20T22:18:22.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:22.230+0000 7fa0b9200700 1 -- 172.21.15.62:0/2037405659 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0b41087f0 con 0x7fa0b4100e80 2024-04-20T22:18:22.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:22.231+0000 7fa0aaa00700 1 -- 172.21.15.62:0/2037405659 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fa0a00104f0 con 0x7fa0b4100e80 2024-04-20T22:18:22.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:22.232+0000 7fa0aaa00700 1 --2- 172.21.15.62:0/2037405659 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa09806ecc0 0x7fa098071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:18:22.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:22.232+0000 7fa0aaa00700 1 -- 172.21.15.62:0/2037405659 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fa0a008ea70 con 0x7fa0b4100e80 2024-04-20T22:18:22.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:22.233+0000 7fa0b1e00700 1 --2- 172.21.15.62:0/2037405659 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa09806ecc0 0x7fa098071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:18:22.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:22.233+0000 7fa0b1e00700 1 --2- 172.21.15.62:0/2037405659 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa09806ecc0 0x7fa098071170 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fa0a4006010 tx=0x7fa0a401a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:18:22.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:22.238+0000 7fa0aaa00700 1 -- 172.21.15.62:0/2037405659 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fa0a0057800 con 0x7fa0b4100e80 2024-04-20T22:18:22.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:22.495+0000 7fa0b9200700 1 -- 172.21.15.62:0/2037405659 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa0b4061190 con 0x7fa09806ecc0 2024-04-20T22:18:22.496 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:18:22 smithi062 ceph-mon[76322]: pgmap v1295: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:18:22.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:22.498+0000 7fa0aaa00700 1 -- 172.21.15.62:0/2037405659 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fa0b4061190 con 0x7fa09806ecc0 2024-04-20T22:18:22.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:22.501+0000 7fa0b9200700 1 -- 172.21.15.62:0/2037405659 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa09806ecc0 msgr2=0x7fa098071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:18:22.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:22.501+0000 7fa0b9200700 1 --2- 172.21.15.62:0/2037405659 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa09806ecc0 0x7fa098071170 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fa0a4006010 tx=0x7fa0a401a040 comp rx=0 tx=0).stop 2024-04-20T22:18:22.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:22.501+0000 7fa0b9200700 1 -- 172.21.15.62:0/2037405659 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0b4100e80 msgr2=0x7fa0b4073590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:18:22.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:22.501+0000 7fa0b9200700 1 --2- 172.21.15.62:0/2037405659 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0b4100e80 0x7fa0b4073590 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7fa0a000b480 tx=0x7fa0a000b840 comp rx=0 tx=0).stop 2024-04-20T22:18:22.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:22.501+0000 7fa0b9200700 1 -- 172.21.15.62:0/2037405659 shutdown_connections 2024-04-20T22:18:22.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:22.501+0000 7fa0b9200700 1 --2- 172.21.15.62:0/2037405659 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa09806ecc0 0x7fa098071170 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:18:22.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:22.501+0000 7fa0b9200700 1 --2- 172.21.15.62:0/2037405659 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0b41004e0 0x7fa0b4073050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:18:22.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:22.501+0000 7fa0b9200700 1 --2- 172.21.15.62:0/2037405659 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0b4100e80 0x7fa0b4073590 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:18:22.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:22.501+0000 7fa0b9200700 1 -- 172.21.15.62:0/2037405659 >> 172.21.15.62:0/2037405659 conn(0x7fa0b40fbda0 msgr2=0x7fa0b40fd480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:18:22.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:22.501+0000 7fa0b9200700 1 -- 172.21.15.62:0/2037405659 shutdown_connections 2024-04-20T22:18:22.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:22.502+0000 7fa0b9200700 1 -- 172.21.15.62:0/2037405659 wait complete. 2024-04-20T22:18:22.518 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T22:18:22.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:22.667+0000 7f6eb2400700 1 -- 172.21.15.62:0/3980827487 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6eac102e70 msgr2=0x7f6eac106e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:18:22.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:22.667+0000 7f6eb2400700 1 --2- 172.21.15.62:0/3980827487 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6eac102e70 0x7f6eac106e40 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f6e98009910 tx=0x7f6e98009c20 comp rx=0 tx=0).stop 2024-04-20T22:18:22.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:22.667+0000 7f6eb2400700 1 -- 172.21.15.62:0/3980827487 shutdown_connections 2024-04-20T22:18:22.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:22.667+0000 7f6eb2400700 1 --2- 172.21.15.62:0/3980827487 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6eac102e70 0x7f6eac106e40 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:18:22.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:22.667+0000 7f6eb2400700 1 --2- 172.21.15.62:0/3980827487 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6eac1024d0 0x7f6eac1028a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:18:22.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:22.667+0000 7f6eb2400700 1 -- 172.21.15.62:0/3980827487 >> 172.21.15.62:0/3980827487 conn(0x7f6eac0fdd90 msgr2=0x7f6eac1001a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:18:22.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:22.667+0000 7f6eb2400700 1 -- 172.21.15.62:0/3980827487 shutdown_connections 2024-04-20T22:18:22.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:22.667+0000 7f6eb2400700 1 -- 172.21.15.62:0/3980827487 wait complete. 2024-04-20T22:18:22.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:22.669+0000 7f6eb2400700 1 Processor -- start 2024-04-20T22:18:22.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:22.669+0000 7f6eb2400700 1 -- start start 2024-04-20T22:18:22.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:22.670+0000 7f6eb2400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6eac1024d0 0x7f6eac197e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:18:22.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:22.670+0000 7f6eb2400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6eac102e70 0x7f6eac198350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:18:22.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:22.670+0000 7f6eb2400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6eac198a30 con 0x7f6eac1024d0 2024-04-20T22:18:22.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:22.670+0000 7f6eb2400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6eac19c7c0 con 0x7f6eac102e70 2024-04-20T22:18:22.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:22.670+0000 7f6eb1000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6eac1024d0 0x7f6eac197e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:18:22.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:22.670+0000 7f6eb1000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6eac1024d0 0x7f6eac197e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35678/0 (socket says 172.21.15.62:35678) 2024-04-20T22:18:22.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:22.670+0000 7f6eb1000700 1 -- 172.21.15.62:0/776156914 learned_addr learned my addr 172.21.15.62:0/776156914 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:18:22.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:22.670+0000 7f6eabe00700 1 --2- 172.21.15.62:0/776156914 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6eac102e70 0x7f6eac198350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:18:22.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:22.670+0000 7f6eb1000700 1 -- 172.21.15.62:0/776156914 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6eac102e70 msgr2=0x7f6eac198350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:18:22.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:22.671+0000 7f6eb1000700 1 --2- 172.21.15.62:0/776156914 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6eac102e70 0x7f6eac198350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:18:22.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:22.671+0000 7f6eb1000700 1 -- 172.21.15.62:0/776156914 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e980095f0 con 0x7f6eac1024d0 2024-04-20T22:18:22.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:22.671+0000 7f6eb1000700 1 --2- 172.21.15.62:0/776156914 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6eac1024d0 0x7f6eac197e10 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f6ea00060b0 tx=0x7f6ea000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:18:22.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:22.671+0000 7f6ea9600700 1 -- 172.21.15.62:0/776156914 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6ea000dda0 con 0x7f6eac1024d0 2024-04-20T22:18:22.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:22.671+0000 7f6ea9600700 1 -- 172.21.15.62:0/776156914 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6ea000f040 con 0x7f6eac1024d0 2024-04-20T22:18:22.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:22.671+0000 7f6ea9600700 1 -- 172.21.15.62:0/776156914 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6ea0014790 con 0x7f6eac1024d0 2024-04-20T22:18:22.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:22.671+0000 7f6eb2400700 1 -- 172.21.15.62:0/776156914 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6eac19caa0 con 0x7f6eac1024d0 2024-04-20T22:18:22.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:22.671+0000 7f6eb2400700 1 -- 172.21.15.62:0/776156914 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6eac19cff0 con 0x7f6eac1024d0 2024-04-20T22:18:22.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:22.673+0000 7f6eb2400700 1 -- 172.21.15.62:0/776156914 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6eac10a840 con 0x7f6eac1024d0 2024-04-20T22:18:22.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:22.674+0000 7f6ea9600700 1 -- 172.21.15.62:0/776156914 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f6ea00148f0 con 0x7f6eac1024d0 2024-04-20T22:18:22.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:22.675+0000 7f6ea9600700 1 --2- 172.21.15.62:0/776156914 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6e9406ecc0 0x7f6e94071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:18:22.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:22.675+0000 7f6ea9600700 1 -- 172.21.15.62:0/776156914 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f6ea008e8d0 con 0x7f6eac1024d0 2024-04-20T22:18:22.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:22.675+0000 7f6eabe00700 1 --2- 172.21.15.62:0/776156914 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6e9406ecc0 0x7f6e94071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:18:22.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:22.676+0000 7f6eabe00700 1 --2- 172.21.15.62:0/776156914 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6e9406ecc0 0x7f6e94071170 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f6eac199430 tx=0x7f6e9801a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:18:22.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:22.680+0000 7f6ea9600700 1 -- 172.21.15.62:0/776156914 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f6ea0057710 con 0x7f6eac1024d0 2024-04-20T22:18:22.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:18:22 smithi189 ceph-mon[57022]: pgmap v1295: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:18:22.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:22.938+0000 7f6eb2400700 1 -- 172.21.15.62:0/776156914 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6eac19d2c0 con 0x7f6e9406ecc0 2024-04-20T22:18:22.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:22.940+0000 7f6ea9600700 1 -- 172.21.15.62:0/776156914 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f6eac19d2c0 con 0x7f6e9406ecc0 2024-04-20T22:18:22.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:22.943+0000 7f6eb2400700 1 -- 172.21.15.62:0/776156914 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6e9406ecc0 msgr2=0x7f6e94071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:18:22.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:22.943+0000 7f6eb2400700 1 --2- 172.21.15.62:0/776156914 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6e9406ecc0 0x7f6e94071170 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f6eac199430 tx=0x7f6e9801a040 comp rx=0 tx=0).stop 2024-04-20T22:18:22.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:22.943+0000 7f6eb2400700 1 -- 172.21.15.62:0/776156914 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6eac1024d0 msgr2=0x7f6eac197e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:18:22.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:22.943+0000 7f6eb2400700 1 --2- 172.21.15.62:0/776156914 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6eac1024d0 0x7f6eac197e10 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f6ea00060b0 tx=0x7f6ea000d3b0 comp rx=0 tx=0).stop 2024-04-20T22:18:22.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:22.944+0000 7f6eb2400700 1 -- 172.21.15.62:0/776156914 shutdown_connections 2024-04-20T22:18:22.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:22.944+0000 7f6eb2400700 1 --2- 172.21.15.62:0/776156914 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6e9406ecc0 0x7f6e94071170 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:18:22.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:22.944+0000 7f6eb2400700 1 --2- 172.21.15.62:0/776156914 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6eac1024d0 0x7f6eac197e10 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:18:22.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:22.944+0000 7f6eb2400700 1 --2- 172.21.15.62:0/776156914 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6eac102e70 0x7f6eac198350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:18:22.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:22.944+0000 7f6eb2400700 1 -- 172.21.15.62:0/776156914 >> 172.21.15.62:0/776156914 conn(0x7f6eac0fdd90 msgr2=0x7f6eac100180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:18:22.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:22.944+0000 7f6eb2400700 1 -- 172.21.15.62:0/776156914 shutdown_connections 2024-04-20T22:18:22.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:22.944+0000 7f6eb2400700 1 -- 172.21.15.62:0/776156914 wait complete. 2024-04-20T22:18:23.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:23.111+0000 7fe2b8800700 1 -- 172.21.15.62:0/3037391429 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2b0106b80 msgr2=0x7fe2b0106ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:18:23.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:23.111+0000 7fe2b8800700 1 --2- 172.21.15.62:0/3037391429 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2b0106b80 0x7fe2b0106ff0 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7fe2a0009910 tx=0x7fe2a0009c20 comp rx=0 tx=0).stop 2024-04-20T22:18:23.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:23.111+0000 7fe2b8800700 1 -- 172.21.15.62:0/3037391429 shutdown_connections 2024-04-20T22:18:23.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:23.111+0000 7fe2b8800700 1 --2- 172.21.15.62:0/3037391429 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2b0106b80 0x7fe2b0106ff0 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:18:23.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:23.111+0000 7fe2b8800700 1 --2- 172.21.15.62:0/3037391429 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe2b010c8f0 0x7fe2b010ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:18:23.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:23.111+0000 7fe2b8800700 1 -- 172.21.15.62:0/3037391429 >> 172.21.15.62:0/3037391429 conn(0x7fe2b0076020 msgr2=0x7fe2b0078430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:18:23.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:23.111+0000 7fe2b8800700 1 -- 172.21.15.62:0/3037391429 shutdown_connections 2024-04-20T22:18:23.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:23.111+0000 7fe2b8800700 1 -- 172.21.15.62:0/3037391429 wait complete. 2024-04-20T22:18:23.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:23.112+0000 7fe2b8800700 1 Processor -- start 2024-04-20T22:18:23.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:23.112+0000 7fe2b8800700 1 -- start start 2024-04-20T22:18:23.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:23.113+0000 7fe2b8800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2b0106b80 0x7fe2b019c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:18:23.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:23.113+0000 7fe2b8800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe2b010c8f0 0x7fe2b019c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:18:23.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:23.113+0000 7fe2b8800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2b019cfe0 con 0x7fe2b0106b80 2024-04-20T22:18:23.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:23.113+0000 7fe2b8800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2b019d120 con 0x7fe2b010c8f0 2024-04-20T22:18:23.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:23.113+0000 7fe2b5e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2b0106b80 0x7fe2b019c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:18:23.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:23.113+0000 7fe2b5e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2b0106b80 0x7fe2b019c450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35702/0 (socket says 172.21.15.62:35702) 2024-04-20T22:18:23.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:23.113+0000 7fe2b5e00700 1 -- 172.21.15.62:0/2164443538 learned_addr learned my addr 172.21.15.62:0/2164443538 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:18:23.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:23.114+0000 7fe2b5400700 1 --2- 172.21.15.62:0/2164443538 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe2b010c8f0 0x7fe2b019c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:18:23.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:23.114+0000 7fe2b5e00700 1 -- 172.21.15.62:0/2164443538 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe2b010c8f0 msgr2=0x7fe2b019c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:18:23.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:23.114+0000 7fe2b5e00700 1 --2- 172.21.15.62:0/2164443538 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe2b010c8f0 0x7fe2b019c990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:18:23.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:23.114+0000 7fe2b5e00700 1 -- 172.21.15.62:0/2164443538 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2a00095f0 con 0x7fe2b0106b80 2024-04-20T22:18:23.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:23.114+0000 7fe2b5e00700 1 --2- 172.21.15.62:0/2164443538 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2b0106b80 0x7fe2b019c450 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7fe2a400c930 tx=0x7fe2a400ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:18:23.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:23.114+0000 7fe2aaa00700 1 -- 172.21.15.62:0/2164443538 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe2a4007ad0 con 0x7fe2b0106b80 2024-04-20T22:18:23.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:23.114+0000 7fe2aaa00700 1 -- 172.21.15.62:0/2164443538 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe2a4007c30 con 0x7fe2b0106b80 2024-04-20T22:18:23.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:23.114+0000 7fe2b8800700 1 -- 172.21.15.62:0/2164443538 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2b01a0f70 con 0x7fe2b0106b80 2024-04-20T22:18:23.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:23.114+0000 7fe2aaa00700 1 -- 172.21.15.62:0/2164443538 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe2a4018610 con 0x7fe2b0106b80 2024-04-20T22:18:23.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:23.114+0000 7fe2b8800700 1 -- 172.21.15.62:0/2164443538 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2b01a14c0 con 0x7fe2b0106b80 2024-04-20T22:18:23.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:23.116+0000 7fe2b8800700 1 -- 172.21.15.62:0/2164443538 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2b004f2a0 con 0x7fe2b0106b80 2024-04-20T22:18:23.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:23.117+0000 7fe2aaa00700 1 -- 172.21.15.62:0/2164443538 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe2a400f450 con 0x7fe2b0106b80 2024-04-20T22:18:23.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:23.118+0000 7fe2aaa00700 1 --2- 172.21.15.62:0/2164443538 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe29806ecc0 0x7fe298071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:18:23.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:23.118+0000 7fe2aaa00700 1 -- 172.21.15.62:0/2164443538 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fe2a408eff0 con 0x7fe2b0106b80 2024-04-20T22:18:23.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:23.118+0000 7fe2b5400700 1 --2- 172.21.15.62:0/2164443538 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe29806ecc0 0x7fe298071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:18:23.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:23.119+0000 7fe2b5400700 1 --2- 172.21.15.62:0/2164443538 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe29806ecc0 0x7fe298071170 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fe2a0000c00 tx=0x7fe2a0005fd0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:18:23.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:23.123+0000 7fe2aaa00700 1 -- 172.21.15.62:0/2164443538 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fe2a4057de0 con 0x7fe2b0106b80 2024-04-20T22:18:23.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:23.366+0000 7fe2b8800700 1 -- 172.21.15.62:0/2164443538 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fe2b01a18f0 con 0x7fe29806ecc0 2024-04-20T22:18:23.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:23.379+0000 7fe2aaa00700 1 -- 172.21.15.62:0/2164443538 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fe2b01a18f0 con 0x7fe29806ecc0 2024-04-20T22:18:23.379 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T22:18:23.379 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (44m) 5m ago 58m 14.6M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T22:18:23.379 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (58m) 5m ago 58m 11.6M - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T22:18:23.379 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (57m) 5m ago 57m 11.2M - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T22:18:23.379 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (40m) 5m ago 58m 6660k - 19.0.0-3133-ga8c84f59 76bc6b91b726 2825d5ba11c6 2024-04-20T22:18:23.379 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (40m) 5m ago 56m 6665k - 19.0.0-3133-ga8c84f59 76bc6b91b726 20f746fa1bba 2024-04-20T22:18:23.380 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (43m) 5m ago 57m 75.5M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T22:18:23.380 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (35m) 5m ago 50m 22.9M - 19.0.0-3133-ga8c84f59 76bc6b91b726 de80bc90a683 2024-04-20T22:18:23.380 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (36m) 5m ago 50m 23.0M - 19.0.0-3133-ga8c84f59 76bc6b91b726 cd793b7ee98e 2024-04-20T22:18:23.380 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (20m) 5m ago 50m 18.5M - 19.0.0-3133-ga8c84f59 76bc6b91b726 0e7b0bc4d1ab 2024-04-20T22:18:23.380 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (5m) 5m ago 50m 11.4M - 19.0.0-3133-ga8c84f59 76bc6b91b726 0af732ac9e06 2024-04-20T22:18:23.380 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (47m) 5m ago 59m 517M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T22:18:23.380 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (45m) 5m ago 56m 460M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T22:18:23.380 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (41m) 5m ago 60m 86.3M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 019eb1479485 2024-04-20T22:18:23.380 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (41m) 5m ago 56m 58.0M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 55aac560aab6 2024-04-20T22:18:23.380 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (45m) 5m ago 58m 8766k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T22:18:23.380 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (45m) 5m ago 56m 8711k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T22:18:23.380 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (39m) 5m ago 55m 91.3M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 ee2bb901f26e 2024-04-20T22:18:23.380 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (39m) 5m ago 55m 94.9M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 9075152d4162 2024-04-20T22:18:23.380 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (38m) 5m ago 54m 78.5M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 2d4f3e38dc0e 2024-04-20T22:18:23.381 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (38m) 5m ago 53m 94.1M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 8f97b17ae0eb 2024-04-20T22:18:23.381 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (37m) 5m ago 53m 96.1M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 653d8079967d 2024-04-20T22:18:23.381 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (36m) 5m ago 52m 90.9M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 596dded14c12 2024-04-20T22:18:23.381 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (44m) 5m ago 57m 90.4M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T22:18:23.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:23.382+0000 7fe2b8800700 1 -- 172.21.15.62:0/2164443538 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe29806ecc0 msgr2=0x7fe298071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:18:23.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:23.382+0000 7fe2b8800700 1 --2- 172.21.15.62:0/2164443538 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe29806ecc0 0x7fe298071170 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fe2a0000c00 tx=0x7fe2a0005fd0 comp rx=0 tx=0).stop 2024-04-20T22:18:23.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:23.382+0000 7fe2b8800700 1 -- 172.21.15.62:0/2164443538 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2b0106b80 msgr2=0x7fe2b019c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:18:23.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:23.382+0000 7fe2b8800700 1 --2- 172.21.15.62:0/2164443538 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2b0106b80 0x7fe2b019c450 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7fe2a400c930 tx=0x7fe2a400ccf0 comp rx=0 tx=0).stop 2024-04-20T22:18:23.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:23.383+0000 7fe2b8800700 1 -- 172.21.15.62:0/2164443538 shutdown_connections 2024-04-20T22:18:23.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:23.383+0000 7fe2b8800700 1 --2- 172.21.15.62:0/2164443538 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe29806ecc0 0x7fe298071170 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:18:23.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:23.383+0000 7fe2b8800700 1 --2- 172.21.15.62:0/2164443538 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2b0106b80 0x7fe2b019c450 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:18:23.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:23.383+0000 7fe2b8800700 1 --2- 172.21.15.62:0/2164443538 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe2b010c8f0 0x7fe2b019c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:18:23.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:23.383+0000 7fe2b8800700 1 -- 172.21.15.62:0/2164443538 >> 172.21.15.62:0/2164443538 conn(0x7fe2b0076020 msgr2=0x7fe2b0077a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:18:23.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:23.383+0000 7fe2b8800700 1 -- 172.21.15.62:0/2164443538 shutdown_connections 2024-04-20T22:18:23.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:23.383+0000 7fe2b8800700 1 -- 172.21.15.62:0/2164443538 wait complete. 2024-04-20T22:18:23.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:23.550+0000 7fd39e600700 1 -- 172.21.15.62:0/4045369183 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3981025f0 msgr2=0x7fd3981029c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:18:23.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:23.550+0000 7fd39e600700 1 --2- 172.21.15.62:0/4045369183 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3981025f0 0x7fd3981029c0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7fd38c009910 tx=0x7fd38c009c20 comp rx=0 tx=0).stop 2024-04-20T22:18:23.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:23.551+0000 7fd39e600700 1 -- 172.21.15.62:0/4045369183 shutdown_connections 2024-04-20T22:18:23.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:23.551+0000 7fd39e600700 1 --2- 172.21.15.62:0/4045369183 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd398103e50 0x7fd3981042c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:18:23.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:23.551+0000 7fd39e600700 1 --2- 172.21.15.62:0/4045369183 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3981025f0 0x7fd3981029c0 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:18:23.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:23.551+0000 7fd39e600700 1 -- 172.21.15.62:0/4045369183 >> 172.21.15.62:0/4045369183 conn(0x7fd3980fddb0 msgr2=0x7fd3981001c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:18:23.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:23.551+0000 7fd39e600700 1 -- 172.21.15.62:0/4045369183 shutdown_connections 2024-04-20T22:18:23.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:23.551+0000 7fd39e600700 1 -- 172.21.15.62:0/4045369183 wait complete. 2024-04-20T22:18:23.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:23.552+0000 7fd39e600700 1 Processor -- start 2024-04-20T22:18:23.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:23.552+0000 7fd39e600700 1 -- start start 2024-04-20T22:18:23.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:23.552+0000 7fd39e600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3981025f0 0x7fd398197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:18:23.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:23.552+0000 7fd39e600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd398103e50 0x7fd3981984e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:18:23.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:23.552+0000 7fd39e600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd398198bc0 con 0x7fd398103e50 2024-04-20T22:18:23.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:23.552+0000 7fd39e600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd39819c950 con 0x7fd3981025f0 2024-04-20T22:18:23.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:23.553+0000 7fd397e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd398103e50 0x7fd3981984e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:18:23.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:23.553+0000 7fd397e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd398103e50 0x7fd3981984e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35728/0 (socket says 172.21.15.62:35728) 2024-04-20T22:18:23.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:23.553+0000 7fd397e00700 1 -- 172.21.15.62:0/169209739 learned_addr learned my addr 172.21.15.62:0/169209739 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:18:23.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:23.553+0000 7fd39d200700 1 --2- 172.21.15.62:0/169209739 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3981025f0 0x7fd398197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:18:23.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:23.553+0000 7fd397e00700 1 -- 172.21.15.62:0/169209739 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3981025f0 msgr2=0x7fd398197fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:18:23.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:23.553+0000 7fd397e00700 1 --2- 172.21.15.62:0/169209739 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3981025f0 0x7fd398197fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:18:23.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:23.553+0000 7fd397e00700 1 -- 172.21.15.62:0/169209739 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd38c0095f0 con 0x7fd398103e50 2024-04-20T22:18:23.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:23.554+0000 7fd397e00700 1 --2- 172.21.15.62:0/169209739 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd398103e50 0x7fd3981984e0 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7fd38400b4b0 tx=0x7fd38400b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:18:23.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:23.554+0000 7fd395600700 1 -- 172.21.15.62:0/169209739 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd384011800 con 0x7fd398103e50 2024-04-20T22:18:23.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:23.554+0000 7fd395600700 1 -- 172.21.15.62:0/169209739 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd384011e20 con 0x7fd398103e50 2024-04-20T22:18:23.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:23.554+0000 7fd39e600700 1 -- 172.21.15.62:0/169209739 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd39819cc30 con 0x7fd398103e50 2024-04-20T22:18:23.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:23.554+0000 7fd395600700 1 -- 172.21.15.62:0/169209739 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd38400f2e0 con 0x7fd398103e50 2024-04-20T22:18:23.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:23.554+0000 7fd39e600700 1 -- 172.21.15.62:0/169209739 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd39819d200 con 0x7fd398103e50 2024-04-20T22:18:23.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:23.555+0000 7fd39e600700 1 -- 172.21.15.62:0/169209739 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd398062380 con 0x7fd398103e50 2024-04-20T22:18:23.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:23.556+0000 7fd395600700 1 -- 172.21.15.62:0/169209739 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fd384010380 con 0x7fd398103e50 2024-04-20T22:18:23.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:23.557+0000 7fd395600700 1 --2- 172.21.15.62:0/169209739 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd38006ecc0 0x7fd380071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:18:23.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:23.557+0000 7fd395600700 1 -- 172.21.15.62:0/169209739 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fd38408ea30 con 0x7fd398103e50 2024-04-20T22:18:23.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:23.558+0000 7fd39d200700 1 --2- 172.21.15.62:0/169209739 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd38006ecc0 0x7fd380071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:18:23.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:23.558+0000 7fd39d200700 1 --2- 172.21.15.62:0/169209739 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd38006ecc0 0x7fd380071170 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fd38c006010 tx=0x7fd38c01a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:18:23.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:23.562+0000 7fd395600700 1 -- 172.21.15.62:0/169209739 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fd384057870 con 0x7fd398103e50 2024-04-20T22:18:23.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:23.889+0000 7fd39e600700 1 -- 172.21.15.62:0/169209739 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fd39819d4d0 con 0x7fd398103e50 2024-04-20T22:18:23.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:23.889+0000 7fd395600700 1 -- 172.21.15.62:0/169209739 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7fd38405af30 con 0x7fd398103e50 2024-04-20T22:18:23.890 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:18:23.890 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T22:18:23.890 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:18:23.890 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:18:23.890 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T22:18:23.890 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:18:23.890 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:18:23.890 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T22:18:23.890 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 6 2024-04-20T22:18:23.890 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:18:23.890 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T22:18:23.891 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 4 2024-04-20T22:18:23.891 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:18:23.891 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T22:18:23.891 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 14 2024-04-20T22:18:23.891 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T22:18:23.891 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:18:23.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:23.892+0000 7fd39e600700 1 -- 172.21.15.62:0/169209739 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd38006ecc0 msgr2=0x7fd380071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:18:23.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:23.892+0000 7fd39e600700 1 --2- 172.21.15.62:0/169209739 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd38006ecc0 0x7fd380071170 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fd38c006010 tx=0x7fd38c01a040 comp rx=0 tx=0).stop 2024-04-20T22:18:23.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:23.892+0000 7fd39e600700 1 -- 172.21.15.62:0/169209739 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd398103e50 msgr2=0x7fd3981984e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:18:23.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:23.892+0000 7fd39e600700 1 --2- 172.21.15.62:0/169209739 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd398103e50 0x7fd3981984e0 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7fd38400b4b0 tx=0x7fd38400b870 comp rx=0 tx=0).stop 2024-04-20T22:18:23.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:23.892+0000 7fd39e600700 1 -- 172.21.15.62:0/169209739 shutdown_connections 2024-04-20T22:18:23.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:23.892+0000 7fd39e600700 1 --2- 172.21.15.62:0/169209739 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd38006ecc0 0x7fd380071170 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:18:23.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:23.892+0000 7fd39e600700 1 --2- 172.21.15.62:0/169209739 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3981025f0 0x7fd398197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:18:23.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:23.892+0000 7fd39e600700 1 --2- 172.21.15.62:0/169209739 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd398103e50 0x7fd3981984e0 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:18:23.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:23.892+0000 7fd39e600700 1 -- 172.21.15.62:0/169209739 >> 172.21.15.62:0/169209739 conn(0x7fd3980fddb0 msgr2=0x7fd3980fe7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:18:23.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:23.893+0000 7fd39e600700 1 -- 172.21.15.62:0/169209739 shutdown_connections 2024-04-20T22:18:23.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:23.893+0000 7fd39e600700 1 -- 172.21.15.62:0/169209739 wait complete. 2024-04-20T22:18:24.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.053+0000 7f9ec8a00700 1 -- 172.21.15.62:0/2859030086 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ec4101b40 msgr2=0x7f9ec410a040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:18:24.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.053+0000 7f9ec8a00700 1 --2- 172.21.15.62:0/2859030086 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ec4101b40 0x7f9ec410a040 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f9eb0009910 tx=0x7f9eb0009c20 comp rx=0 tx=0).stop 2024-04-20T22:18:24.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.054+0000 7f9ec8a00700 1 -- 172.21.15.62:0/2859030086 shutdown_connections 2024-04-20T22:18:24.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.054+0000 7f9ec8a00700 1 --2- 172.21.15.62:0/2859030086 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ec4101b40 0x7f9ec410a040 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:18:24.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.054+0000 7f9ec8a00700 1 --2- 172.21.15.62:0/2859030086 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ec4101230 0x7f9ec4101600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:18:24.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.054+0000 7f9ec8a00700 1 -- 172.21.15.62:0/2859030086 >> 172.21.15.62:0/2859030086 conn(0x7f9ec40faa90 msgr2=0x7f9ec40fcea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:18:24.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.054+0000 7f9ec8a00700 1 -- 172.21.15.62:0/2859030086 shutdown_connections 2024-04-20T22:18:24.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.054+0000 7f9ec8a00700 1 -- 172.21.15.62:0/2859030086 wait complete. 2024-04-20T22:18:24.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.055+0000 7f9ec8a00700 1 Processor -- start 2024-04-20T22:18:24.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.055+0000 7f9ec8a00700 1 -- start start 2024-04-20T22:18:24.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.055+0000 7f9ec8a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ec4101230 0x7f9ec419b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:18:24.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.055+0000 7f9ec8a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ec4101b40 0x7f9ec419bef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:18:24.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.055+0000 7f9ec8a00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ec419c580 con 0x7f9ec4101b40 2024-04-20T22:18:24.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.055+0000 7f9ec8a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ec4195a30 con 0x7f9ec4101230 2024-04-20T22:18:24.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.056+0000 7f9ec2a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ec4101b40 0x7f9ec419bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:18:24.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.056+0000 7f9ec2a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ec4101b40 0x7f9ec419bef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35748/0 (socket says 172.21.15.62:35748) 2024-04-20T22:18:24.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.056+0000 7f9ec2a00700 1 -- 172.21.15.62:0/3083507389 learned_addr learned my addr 172.21.15.62:0/3083507389 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:18:24.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.056+0000 7f9ec3400700 1 --2- 172.21.15.62:0/3083507389 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ec4101230 0x7f9ec419b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:18:24.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.056+0000 7f9ec2a00700 1 -- 172.21.15.62:0/3083507389 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ec4101230 msgr2=0x7f9ec419b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:18:24.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.056+0000 7f9ec2a00700 1 --2- 172.21.15.62:0/3083507389 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ec4101230 0x7f9ec419b9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:18:24.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.056+0000 7f9ec2a00700 1 -- 172.21.15.62:0/3083507389 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9eb00095f0 con 0x7f9ec4101b40 2024-04-20T22:18:24.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.057+0000 7f9ec2a00700 1 --2- 172.21.15.62:0/3083507389 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ec4101b40 0x7f9ec419bef0 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f9eb0006010 tx=0x7f9eb000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:18:24.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.057+0000 7f9ebbe00700 1 -- 172.21.15.62:0/3083507389 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9eb001d070 con 0x7f9ec4101b40 2024-04-20T22:18:24.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.057+0000 7f9ebbe00700 1 -- 172.21.15.62:0/3083507389 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9eb0004fb0 con 0x7f9ec4101b40 2024-04-20T22:18:24.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.057+0000 7f9ec8a00700 1 -- 172.21.15.62:0/3083507389 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ec4195cb0 con 0x7f9ec4101b40 2024-04-20T22:18:24.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.057+0000 7f9ebbe00700 1 -- 172.21.15.62:0/3083507389 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9eb000f690 con 0x7f9ec4101b40 2024-04-20T22:18:24.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.057+0000 7f9ec8a00700 1 -- 172.21.15.62:0/3083507389 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ec41961a0 con 0x7f9ec4101b40 2024-04-20T22:18:24.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.058+0000 7f9ec8a00700 1 -- 172.21.15.62:0/3083507389 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ec41076b0 con 0x7f9ec4101b40 2024-04-20T22:18:24.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.059+0000 7f9ebbe00700 1 -- 172.21.15.62:0/3083507389 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f9eb00228a0 con 0x7f9ec4101b40 2024-04-20T22:18:24.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.061+0000 7f9ebbe00700 1 --2- 172.21.15.62:0/3083507389 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9ea806ecc0 0x7f9ea8071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:18:24.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.061+0000 7f9ebbe00700 1 -- 172.21.15.62:0/3083507389 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f9eb0090770 con 0x7f9ec4101b40 2024-04-20T22:18:24.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.061+0000 7f9ec3400700 1 --2- 172.21.15.62:0/3083507389 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9ea806ecc0 0x7f9ea8071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:18:24.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.062+0000 7f9ec3400700 1 --2- 172.21.15.62:0/3083507389 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9ea806ecc0 0x7f9ea8071170 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f9eb40060b0 tx=0x7f9eb4007b80 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:18:24.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.066+0000 7f9ebbe00700 1 -- 172.21.15.62:0/3083507389 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f9eb005cfb0 con 0x7f9ec4101b40 2024-04-20T22:18:24.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.352+0000 7f9ec8a00700 1 -- 172.21.15.62:0/3083507389 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f9ec4062380 con 0x7f9ec4101b40 2024-04-20T22:18:24.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.353+0000 7f9ebbe00700 1 -- 172.21.15.62:0/3083507389 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 29 v29) v1 ==== 76+0+1887 (secure 0 0 0) 0x7f9eb005cb40 con 0x7f9ec4101b40 2024-04-20T22:18:24.354 INFO:teuthology.orchestra.run.smithi062.stdout:e29 2024-04-20T22:18:24.354 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T22:18:24.354 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:18:24.354 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T22:18:24.354 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:18:24.354 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T22:18:24.354 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T22:18:24.355 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 20 2024-04-20T22:18:24.355 INFO:teuthology.orchestra.run.smithi062.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-20T22:18:24.355 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T22:18:24.355 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T21:42:01.559061+0000 2024-04-20T22:18:24.355 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T22:18:24.355 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T22:18:24.355 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T22:18:24.355 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T22:18:24.355 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T22:18:24.355 INFO:teuthology.orchestra.run.smithi062.stdout:max_xattr_size 65536 2024-04-20T22:18:24.355 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T22:18:24.355 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T22:18:24.355 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 81 2024-04-20T22:18:24.355 INFO:teuthology.orchestra.run.smithi062.stdout:snap_rstat 1 2024-04-20T22:18:24.355 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:18:24.356 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 2 2024-04-20T22:18:24.356 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1 2024-04-20T22:18:24.356 INFO:teuthology.orchestra.run.smithi062.stdout:up {} 2024-04-20T22:18:24.356 INFO:teuthology.orchestra.run.smithi062.stdout:failed 0,1 2024-04-20T22:18:24.356 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T22:18:24.356 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T22:18:24.356 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T22:18:24.356 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T22:18:24.356 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data enabled 2024-04-20T22:18:24.356 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T22:18:24.356 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T22:18:24.356 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 1 2024-04-20T22:18:24.357 INFO:teuthology.orchestra.run.smithi062.stdout:qdb_cluster leader: 0 members: 2024-04-20T22:18:24.357 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:18:24.357 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:18:24.357 INFO:teuthology.orchestra.run.smithi062.stdout:Standby daemons: 2024-04-20T22:18:24.357 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:18:24.357 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ryqblb{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6826/4113916205,v1:172.21.15.62:6827/4113916205] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:18:24.357 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ozizwi{-1:34476} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6828/3801975165,v1:172.21.15.62:6829/3801975165] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:18:24.357 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.dvimer{-1:44727} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6824/3576699232,v1:172.21.15.189:6825/3576699232] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:18:24.357 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.xbgspw{-1:45075} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6826/212905792,v1:172.21.15.189:6827/212905792] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:18:24.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.356+0000 7f9ec8a00700 1 -- 172.21.15.62:0/3083507389 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9ea806ecc0 msgr2=0x7f9ea8071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:18:24.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.356+0000 7f9ec8a00700 1 --2- 172.21.15.62:0/3083507389 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9ea806ecc0 0x7f9ea8071170 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f9eb40060b0 tx=0x7f9eb4007b80 comp rx=0 tx=0).stop 2024-04-20T22:18:24.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.356+0000 7f9ec8a00700 1 -- 172.21.15.62:0/3083507389 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ec4101b40 msgr2=0x7f9ec419bef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:18:24.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.356+0000 7f9ec8a00700 1 --2- 172.21.15.62:0/3083507389 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ec4101b40 0x7f9ec419bef0 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f9eb0006010 tx=0x7f9eb000be20 comp rx=0 tx=0).stop 2024-04-20T22:18:24.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.356+0000 7f9ec8a00700 1 -- 172.21.15.62:0/3083507389 shutdown_connections 2024-04-20T22:18:24.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.356+0000 7f9ec8a00700 1 --2- 172.21.15.62:0/3083507389 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9ea806ecc0 0x7f9ea8071170 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:18:24.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.356+0000 7f9ec8a00700 1 --2- 172.21.15.62:0/3083507389 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ec4101230 0x7f9ec419b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:18:24.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.356+0000 7f9ec8a00700 1 --2- 172.21.15.62:0/3083507389 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ec4101b40 0x7f9ec419bef0 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:18:24.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.356+0000 7f9ec8a00700 1 -- 172.21.15.62:0/3083507389 >> 172.21.15.62:0/3083507389 conn(0x7f9ec40faa90 msgr2=0x7f9ec41047f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:18:24.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.357+0000 7f9ec8a00700 1 -- 172.21.15.62:0/3083507389 shutdown_connections 2024-04-20T22:18:24.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.357+0000 7f9ec8a00700 1 -- 172.21.15.62:0/3083507389 wait complete. 2024-04-20T22:18:24.359 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 29 2024-04-20T22:18:24.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.514+0000 7feb0ce00700 1 -- 172.21.15.62:0/3572734379 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb081002d0 msgr2=0x7feb081006a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:18:24.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.514+0000 7feb0ce00700 1 --2- 172.21.15.62:0/3572734379 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb081002d0 0x7feb081006a0 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7feaf8009910 tx=0x7feaf8009c20 comp rx=0 tx=0).stop 2024-04-20T22:18:24.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.514+0000 7feb0ce00700 1 -- 172.21.15.62:0/3572734379 shutdown_connections 2024-04-20T22:18:24.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.514+0000 7feb0ce00700 1 --2- 172.21.15.62:0/3572734379 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb08100c70 0x7feb08104c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:18:24.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.514+0000 7feb0ce00700 1 --2- 172.21.15.62:0/3572734379 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb081002d0 0x7feb081006a0 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:18:24.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.514+0000 7feb0ce00700 1 -- 172.21.15.62:0/3572734379 >> 172.21.15.62:0/3572734379 conn(0x7feb080fbbb0 msgr2=0x7feb080fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:18:24.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.515+0000 7feb0ce00700 1 -- 172.21.15.62:0/3572734379 shutdown_connections 2024-04-20T22:18:24.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.515+0000 7feb0ce00700 1 -- 172.21.15.62:0/3572734379 wait complete. 2024-04-20T22:18:24.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.516+0000 7feb0ce00700 1 Processor -- start 2024-04-20T22:18:24.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.516+0000 7feb0ce00700 1 -- start start 2024-04-20T22:18:24.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.516+0000 7feb0ce00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb081002d0 0x7feb08195cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:18:24.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.516+0000 7feb0ce00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb08100c70 0x7feb08196210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:18:24.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.516+0000 7feb0ce00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb08196810 con 0x7feb081002d0 2024-04-20T22:18:24.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.516+0000 7feb0ce00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb08196980 con 0x7feb08100c70 2024-04-20T22:18:24.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.517+0000 7feb07400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb081002d0 0x7feb08195cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:18:24.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.517+0000 7feb07400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb081002d0 0x7feb08195cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35768/0 (socket says 172.21.15.62:35768) 2024-04-20T22:18:24.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.517+0000 7feb07400700 1 -- 172.21.15.62:0/248685935 learned_addr learned my addr 172.21.15.62:0/248685935 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:18:24.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.517+0000 7feb07400700 1 -- 172.21.15.62:0/248685935 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb08100c70 msgr2=0x7feb08196210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:18:24.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.517+0000 7feb07400700 1 --2- 172.21.15.62:0/248685935 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb08100c70 0x7feb08196210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:18:24.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.517+0000 7feb07400700 1 -- 172.21.15.62:0/248685935 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaf80095f0 con 0x7feb081002d0 2024-04-20T22:18:24.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.517+0000 7feb07400700 1 --2- 172.21.15.62:0/248685935 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb081002d0 0x7feb08195cd0 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7feaf800b5c0 tx=0x7feaf800be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:18:24.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.517+0000 7feb04c00700 1 -- 172.21.15.62:0/248685935 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feaf801d070 con 0x7feb081002d0 2024-04-20T22:18:24.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.518+0000 7feb04c00700 1 -- 172.21.15.62:0/248685935 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7feaf800f460 con 0x7feb081002d0 2024-04-20T22:18:24.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.518+0000 7feb0ce00700 1 -- 172.21.15.62:0/248685935 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb081996c0 con 0x7feb081002d0 2024-04-20T22:18:24.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.518+0000 7feb04c00700 1 -- 172.21.15.62:0/248685935 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feaf8017610 con 0x7feb081002d0 2024-04-20T22:18:24.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.518+0000 7feb0ce00700 1 -- 172.21.15.62:0/248685935 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb08199bb0 con 0x7feb081002d0 2024-04-20T22:18:24.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.519+0000 7feb0ce00700 1 -- 172.21.15.62:0/248685935 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb08062380 con 0x7feb081002d0 2024-04-20T22:18:24.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.520+0000 7feb04c00700 1 -- 172.21.15.62:0/248685935 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7feaf800f5d0 con 0x7feb081002d0 2024-04-20T22:18:24.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.521+0000 7feb04c00700 1 --2- 172.21.15.62:0/248685935 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7feaec06ecc0 0x7feaec071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:18:24.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.521+0000 7feb04c00700 1 -- 172.21.15.62:0/248685935 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7feaf8090510 con 0x7feb081002d0 2024-04-20T22:18:24.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.521+0000 7feaffe00700 1 --2- 172.21.15.62:0/248685935 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7feaec06ecc0 0x7feaec071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:18:24.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.522+0000 7feaffe00700 1 --2- 172.21.15.62:0/248685935 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7feaec06ecc0 0x7feaec071170 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7feaf0005fd0 tx=0x7feaf0005d00 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:18:24.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.526+0000 7feb04c00700 1 -- 172.21.15.62:0/248685935 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7feaf8059220 con 0x7feb081002d0 2024-04-20T22:18:24.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:18:24 smithi189 ceph-mon[57022]: from='client.36272 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:18:24.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:18:24 smithi189 ceph-mon[57022]: from='client.36276 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:18:24.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:18:24 smithi189 ceph-mon[57022]: pgmap v1296: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:18:24.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:18:24 smithi189 ceph-mon[57022]: from='client.36280 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:18:24.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:18:24 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/169209739' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:18:24.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:18:24 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/3083507389' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:18:24.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.783+0000 7feb0ce00700 1 -- 172.21.15.62:0/248685935 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7feb08108170 con 0x7feaec06ecc0 2024-04-20T22:18:24.783 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:18:24 smithi062 ceph-mon[76322]: from='client.36272 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:18:24.783 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:18:24 smithi062 ceph-mon[76322]: from='client.36276 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:18:24.783 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:18:24 smithi062 ceph-mon[76322]: pgmap v1296: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:18:24.784 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:18:24 smithi062 ceph-mon[76322]: from='client.36280 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:18:24.784 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:18:24 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/169209739' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:18:24.784 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:18:24 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/3083507389' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:18:24.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.785+0000 7feb04c00700 1 -- 172.21.15.62:0/248685935 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7feb08108170 con 0x7feaec06ecc0 2024-04-20T22:18:24.785 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:18:24.785 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T22:18:24.786 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T22:18:24.786 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-20T22:18:24.786 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T22:18:24.786 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr", 2024-04-20T22:18:24.786 INFO:teuthology.orchestra.run.smithi062.stdout: "osd", 2024-04-20T22:18:24.786 INFO:teuthology.orchestra.run.smithi062.stdout: "mds", 2024-04-20T22:18:24.786 INFO:teuthology.orchestra.run.smithi062.stdout: "crash", 2024-04-20T22:18:24.786 INFO:teuthology.orchestra.run.smithi062.stdout: "mon" 2024-04-20T22:18:24.786 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T22:18:24.786 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "16/23 daemons upgraded", 2024-04-20T22:18:24.786 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading mds daemons", 2024-04-20T22:18:24.786 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T22:18:24.786 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:18:24.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.788+0000 7feb0ce00700 1 -- 172.21.15.62:0/248685935 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7feaec06ecc0 msgr2=0x7feaec071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:18:24.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.788+0000 7feb0ce00700 1 --2- 172.21.15.62:0/248685935 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7feaec06ecc0 0x7feaec071170 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7feaf0005fd0 tx=0x7feaf0005d00 comp rx=0 tx=0).stop 2024-04-20T22:18:24.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.788+0000 7feb0ce00700 1 -- 172.21.15.62:0/248685935 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb081002d0 msgr2=0x7feb08195cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:18:24.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.788+0000 7feb0ce00700 1 --2- 172.21.15.62:0/248685935 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb081002d0 0x7feb08195cd0 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7feaf800b5c0 tx=0x7feaf800be00 comp rx=0 tx=0).stop 2024-04-20T22:18:24.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.788+0000 7feb0ce00700 1 -- 172.21.15.62:0/248685935 shutdown_connections 2024-04-20T22:18:24.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.788+0000 7feb0ce00700 1 --2- 172.21.15.62:0/248685935 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7feaec06ecc0 0x7feaec071170 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:18:24.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.788+0000 7feb0ce00700 1 --2- 172.21.15.62:0/248685935 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feb081002d0 0x7feb08195cd0 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:18:24.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.788+0000 7feb0ce00700 1 --2- 172.21.15.62:0/248685935 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb08100c70 0x7feb08196210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:18:24.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.788+0000 7feb0ce00700 1 -- 172.21.15.62:0/248685935 >> 172.21.15.62:0/248685935 conn(0x7feb080fbbb0 msgr2=0x7feb080fdfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:18:24.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.789+0000 7feb0ce00700 1 -- 172.21.15.62:0/248685935 shutdown_connections 2024-04-20T22:18:24.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.789+0000 7feb0ce00700 1 -- 172.21.15.62:0/248685935 wait complete. 2024-04-20T22:18:24.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.951+0000 7fa011400700 1 -- 172.21.15.62:0/3210522714 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa00c0687a0 msgr2=0x7fa00c068b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:18:24.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.951+0000 7fa011400700 1 --2- 172.21.15.62:0/3210522714 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa00c0687a0 0x7fa00c068b70 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f9ffc009910 tx=0x7f9ffc009c20 comp rx=0 tx=0).stop 2024-04-20T22:18:24.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.951+0000 7fa011400700 1 -- 172.21.15.62:0/3210522714 shutdown_connections 2024-04-20T22:18:24.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.951+0000 7fa011400700 1 --2- 172.21.15.62:0/3210522714 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa00c069140 0x7fa00c104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:18:24.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.951+0000 7fa011400700 1 --2- 172.21.15.62:0/3210522714 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa00c0687a0 0x7fa00c068b70 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:18:24.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.951+0000 7fa011400700 1 -- 172.21.15.62:0/3210522714 >> 172.21.15.62:0/3210522714 conn(0x7fa00c075130 msgr2=0x7fa00c075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:18:24.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.952+0000 7fa011400700 1 -- 172.21.15.62:0/3210522714 shutdown_connections 2024-04-20T22:18:24.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.952+0000 7fa011400700 1 -- 172.21.15.62:0/3210522714 wait complete. 2024-04-20T22:18:24.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.953+0000 7fa011400700 1 Processor -- start 2024-04-20T22:18:24.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.953+0000 7fa011400700 1 -- start start 2024-04-20T22:18:24.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.953+0000 7fa011400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa00c0687a0 0x7fa00c195f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:18:24.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.953+0000 7fa011400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa00c069140 0x7fa00c1964a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:18:24.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.953+0000 7fa011400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa00c196b80 con 0x7fa00c069140 2024-04-20T22:18:24.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.953+0000 7fa011400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa00c19a910 con 0x7fa00c0687a0 2024-04-20T22:18:24.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.953+0000 7fa001e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa00c069140 0x7fa00c1964a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:18:24.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.953+0000 7fa001e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa00c069140 0x7fa00c1964a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35786/0 (socket says 172.21.15.62:35786) 2024-04-20T22:18:24.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.953+0000 7fa001e00700 1 -- 172.21.15.62:0/3038225454 learned_addr learned my addr 172.21.15.62:0/3038225454 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:18:24.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.954+0000 7fa00a800700 1 --2- 172.21.15.62:0/3038225454 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa00c0687a0 0x7fa00c195f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:18:24.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.954+0000 7fa001e00700 1 -- 172.21.15.62:0/3038225454 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa00c0687a0 msgr2=0x7fa00c195f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:18:24.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.954+0000 7fa001e00700 1 --2- 172.21.15.62:0/3038225454 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa00c0687a0 0x7fa00c195f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:18:24.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.954+0000 7fa001e00700 1 -- 172.21.15.62:0/3038225454 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ffc0095f0 con 0x7fa00c069140 2024-04-20T22:18:24.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.954+0000 7fa001e00700 1 --2- 172.21.15.62:0/3038225454 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa00c069140 0x7fa00c1964a0 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f9ff400b480 tx=0x7f9ff400b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:18:24.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.954+0000 7fa003e00700 1 -- 172.21.15.62:0/3038225454 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9ff40118c0 con 0x7fa00c069140 2024-04-20T22:18:24.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.954+0000 7fa003e00700 1 -- 172.21.15.62:0/3038225454 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9ff4010380 con 0x7fa00c069140 2024-04-20T22:18:24.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.954+0000 7fa003e00700 1 -- 172.21.15.62:0/3038225454 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9ff400f4f0 con 0x7fa00c069140 2024-04-20T22:18:24.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.954+0000 7fa011400700 1 -- 172.21.15.62:0/3038225454 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa00c19abf0 con 0x7fa00c069140 2024-04-20T22:18:24.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.955+0000 7fa011400700 1 -- 172.21.15.62:0/3038225454 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa00c19b140 con 0x7fa00c069140 2024-04-20T22:18:24.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.956+0000 7fa011400700 1 -- 172.21.15.62:0/3038225454 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa00c108770 con 0x7fa00c069140 2024-04-20T22:18:24.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.957+0000 7fa003e00700 1 -- 172.21.15.62:0/3038225454 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f9ff40104f0 con 0x7fa00c069140 2024-04-20T22:18:24.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.958+0000 7fa003e00700 1 --2- 172.21.15.62:0/3038225454 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9ff006ecc0 0x7f9ff0071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:18:24.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.959+0000 7fa003e00700 1 -- 172.21.15.62:0/3038225454 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f9ff408eb30 con 0x7fa00c069140 2024-04-20T22:18:24.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.959+0000 7fa00a800700 1 --2- 172.21.15.62:0/3038225454 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9ff006ecc0 0x7f9ff0071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:18:24.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.959+0000 7fa00a800700 1 --2- 172.21.15.62:0/3038225454 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9ff006ecc0 0x7f9ff0071170 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f9ffc000c00 tx=0x7f9ffc005fd0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:18:24.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:24.964+0000 7fa003e00700 1 -- 172.21.15.62:0/3038225454 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f9ff4057840 con 0x7fa00c069140 2024-04-20T22:18:25.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:25.276+0000 7fa011400700 1 -- 172.21.15.62:0/3038225454 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fa00c19b3c0 con 0x7fa00c069140 2024-04-20T22:18:25.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:25.277+0000 7fa003e00700 1 -- 172.21.15.62:0/3038225454 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7f9ff405af00 con 0x7fa00c069140 2024-04-20T22:18:25.278 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-20T22:18:25.278 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-20T22:18:25.278 INFO:teuthology.orchestra.run.smithi062.stdout: host smithi189 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi189/config 2024-04-20T22:18:25.278 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-20T22:18:25.278 INFO:teuthology.orchestra.run.smithi062.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi189 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp11670v2q:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:18:25.278 INFO:teuthology.orchestra.run.smithi062.stdout:Traceback (most recent call last): 2024-04-20T22:18:25.278 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-20T22:18:25.278 INFO:teuthology.orchestra.run.smithi062.stdout: return _run_code(code, main_globals, None, 2024-04-20T22:18:25.278 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-20T22:18:25.278 INFO:teuthology.orchestra.run.smithi062.stdout: exec(code, run_globals) 2024-04-20T22:18:25.278 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5618, in 2024-04-20T22:18:25.278 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5606, in main 2024-04-20T22:18:25.279 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 408, in _infer_config 2024-04-20T22:18:25.279 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 323, in _infer_fsid 2024-04-20T22:18:25.279 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 436, in _infer_image 2024-04-20T22:18:25.279 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 310, in _validate_fsid 2024-04-20T22:18:25.279 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 3359, in command_ceph_volume 2024-04-20T22:18:25.279 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-20T22:18:25.279 INFO:teuthology.orchestra.run.smithi062.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi189 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp11670v2q:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:18:25.279 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-20T22:18:25.279 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is degraded 2024-04-20T22:18:25.279 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T22:18:25.279 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T22:18:25.279 INFO:teuthology.orchestra.run.smithi062.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-20T22:18:25.279 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-20T22:18:25.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:25.280+0000 7fa011400700 1 -- 172.21.15.62:0/3038225454 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9ff006ecc0 msgr2=0x7f9ff0071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:18:25.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:25.280+0000 7fa011400700 1 --2- 172.21.15.62:0/3038225454 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9ff006ecc0 0x7f9ff0071170 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f9ffc000c00 tx=0x7f9ffc005fd0 comp rx=0 tx=0).stop 2024-04-20T22:18:25.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:25.280+0000 7fa011400700 1 -- 172.21.15.62:0/3038225454 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa00c069140 msgr2=0x7fa00c1964a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:18:25.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:25.280+0000 7fa011400700 1 --2- 172.21.15.62:0/3038225454 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa00c069140 0x7fa00c1964a0 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f9ff400b480 tx=0x7f9ff400b840 comp rx=0 tx=0).stop 2024-04-20T22:18:25.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:25.281+0000 7fa011400700 1 -- 172.21.15.62:0/3038225454 shutdown_connections 2024-04-20T22:18:25.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:25.281+0000 7fa011400700 1 --2- 172.21.15.62:0/3038225454 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9ff006ecc0 0x7f9ff0071170 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:18:25.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:25.281+0000 7fa011400700 1 --2- 172.21.15.62:0/3038225454 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa00c0687a0 0x7fa00c195f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:18:25.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:25.281+0000 7fa011400700 1 --2- 172.21.15.62:0/3038225454 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa00c069140 0x7fa00c1964a0 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:18:25.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:25.281+0000 7fa011400700 1 -- 172.21.15.62:0/3038225454 >> 172.21.15.62:0/3038225454 conn(0x7fa00c075130 msgr2=0x7fa00c0fe7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:18:25.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:25.281+0000 7fa011400700 1 -- 172.21.15.62:0/3038225454 shutdown_connections 2024-04-20T22:18:25.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:25.281+0000 7fa011400700 1 -- 172.21.15.62:0/3038225454 wait complete. 2024-04-20T22:18:26.234 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:18:25 smithi062 ceph-mon[76322]: from='client.36292 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:18:26.234 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:18:25 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/3038225454' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:18:26.234 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:18:25 smithi062 ceph-mon[76322]: pgmap v1297: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:18:26.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:18:25 smithi189 ceph-mon[57022]: from='client.36292 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:18:26.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:18:25 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/3038225454' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:18:26.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:18:25 smithi189 ceph-mon[57022]: pgmap v1297: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:18:28.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:18:28 smithi062 ceph-mon[76322]: pgmap v1298: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:18:28.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:18:28 smithi189 ceph-mon[57022]: pgmap v1298: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:18:30.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:18:30 smithi189 ceph-mon[57022]: pgmap v1299: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:18:30.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:18:30 smithi062 ceph-mon[76322]: pgmap v1299: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:18:32.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:18:32 smithi189 ceph-mon[57022]: pgmap v1300: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:18:32.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:18:32 smithi062 ceph-mon[76322]: pgmap v1300: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:18:34.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:18:34 smithi189 ceph-mon[57022]: pgmap v1301: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:18:34.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:18:34 smithi062 ceph-mon[76322]: pgmap v1301: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:18:35.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:18:35 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:18:35.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:18:35 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:18:36.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:18:36 smithi189 ceph-mon[57022]: pgmap v1302: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:18:36.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:18:36 smithi062 ceph-mon[76322]: pgmap v1302: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:18:37.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:18:37 smithi189 ceph-mon[57022]: pgmap v1303: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:18:37.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:18:37 smithi062 ceph-mon[76322]: pgmap v1303: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:18:40.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:18:40 smithi189 ceph-mon[57022]: pgmap v1304: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:18:40.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:18:40 smithi062 ceph-mon[76322]: pgmap v1304: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:18:42.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:18:42 smithi189 ceph-mon[57022]: pgmap v1305: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:18:42.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:18:42 smithi062 ceph-mon[76322]: pgmap v1305: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:18:44.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:18:44 smithi189 ceph-mon[57022]: pgmap v1306: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:18:44.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:18:44 smithi062 ceph-mon[76322]: pgmap v1306: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:18:46.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:18:46 smithi189 ceph-mon[57022]: pgmap v1307: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:18:46.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:18:46 smithi062 ceph-mon[76322]: pgmap v1307: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:18:47.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:18:47 smithi189 ceph-mon[57022]: pgmap v1308: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:18:47.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:18:47 smithi062 ceph-mon[76322]: pgmap v1308: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:18:50.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:18:50 smithi189 ceph-mon[57022]: pgmap v1309: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:18:50.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:18:50 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:18:50.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:18:50 smithi062 ceph-mon[76322]: pgmap v1309: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:18:50.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:18:50 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:18:52.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:18:52 smithi189 ceph-mon[57022]: pgmap v1310: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:18:52.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:18:52 smithi062 ceph-mon[76322]: pgmap v1310: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:18:54.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:18:54 smithi189 ceph-mon[57022]: pgmap v1311: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:18:54.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:18:54 smithi062 ceph-mon[76322]: pgmap v1311: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:18:55.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:55.460+0000 7f9f60c00700 1 -- 172.21.15.62:0/1063859173 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9f5c106b80 msgr2=0x7f9f5c106ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:18:55.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:55.460+0000 7f9f60c00700 1 --2- 172.21.15.62:0/1063859173 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9f5c106b80 0x7f9f5c106ff0 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f9f48009910 tx=0x7f9f48009c20 comp rx=0 tx=0).stop 2024-04-20T22:18:55.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:55.460+0000 7f9f60c00700 1 -- 172.21.15.62:0/1063859173 shutdown_connections 2024-04-20T22:18:55.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:55.460+0000 7f9f60c00700 1 --2- 172.21.15.62:0/1063859173 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9f5c106b80 0x7f9f5c106ff0 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:18:55.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:55.460+0000 7f9f60c00700 1 --2- 172.21.15.62:0/1063859173 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f5c10c8f0 0x7f9f5c10ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:18:55.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:55.460+0000 7f9f60c00700 1 -- 172.21.15.62:0/1063859173 >> 172.21.15.62:0/1063859173 conn(0x7f9f5c076060 msgr2=0x7f9f5c078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:18:55.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:55.460+0000 7f9f60c00700 1 -- 172.21.15.62:0/1063859173 shutdown_connections 2024-04-20T22:18:55.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:55.460+0000 7f9f60c00700 1 -- 172.21.15.62:0/1063859173 wait complete. 2024-04-20T22:18:55.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:55.462+0000 7f9f60c00700 1 Processor -- start 2024-04-20T22:18:55.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:55.462+0000 7f9f60c00700 1 -- start start 2024-04-20T22:18:55.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:55.462+0000 7f9f60c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9f5c106b80 0x7f9f5c19c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:18:55.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:55.462+0000 7f9f60c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f5c10c8f0 0x7f9f5c19c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:18:55.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:55.462+0000 7f9f60c00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f5c19d060 con 0x7f9f5c106b80 2024-04-20T22:18:55.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:55.462+0000 7f9f60c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f5c1a0df0 con 0x7f9f5c10c8f0 2024-04-20T22:18:55.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:55.463+0000 7f9f59e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9f5c106b80 0x7f9f5c19c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:18:55.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:55.463+0000 7f9f59e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9f5c106b80 0x7f9f5c19c440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35034/0 (socket says 172.21.15.62:35034) 2024-04-20T22:18:55.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:55.463+0000 7f9f59e00700 1 -- 172.21.15.62:0/3594019936 learned_addr learned my addr 172.21.15.62:0/3594019936 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:18:55.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:55.463+0000 7f9f59e00700 1 -- 172.21.15.62:0/3594019936 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f5c10c8f0 msgr2=0x7f9f5c19c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:18:55.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:55.463+0000 7f9f59e00700 1 --2- 172.21.15.62:0/3594019936 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f5c10c8f0 0x7f9f5c19c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:18:55.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:55.463+0000 7f9f59e00700 1 -- 172.21.15.62:0/3594019936 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f480095f0 con 0x7f9f5c106b80 2024-04-20T22:18:55.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:55.464+0000 7f9f59e00700 1 --2- 172.21.15.62:0/3594019936 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9f5c106b80 0x7f9f5c19c440 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f9f4c00c8a0 tx=0x7f9f4c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:18:55.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:55.464+0000 7f9f52a00700 1 -- 172.21.15.62:0/3594019936 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9f4c00cea0 con 0x7f9f5c106b80 2024-04-20T22:18:55.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:55.464+0000 7f9f52a00700 1 -- 172.21.15.62:0/3594019936 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9f4c004830 con 0x7f9f5c106b80 2024-04-20T22:18:55.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:55.464+0000 7f9f52a00700 1 -- 172.21.15.62:0/3594019936 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9f4c0055b0 con 0x7f9f5c106b80 2024-04-20T22:18:55.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:55.464+0000 7f9f60c00700 1 -- 172.21.15.62:0/3594019936 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f5c1a10d0 con 0x7f9f5c106b80 2024-04-20T22:18:55.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:55.464+0000 7f9f60c00700 1 -- 172.21.15.62:0/3594019936 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f5c1a1620 con 0x7f9f5c106b80 2024-04-20T22:18:55.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:55.465+0000 7f9f60c00700 1 -- 172.21.15.62:0/3594019936 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f5c10ecc0 con 0x7f9f5c106b80 2024-04-20T22:18:55.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:55.466+0000 7f9f52a00700 1 -- 172.21.15.62:0/3594019936 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f9f4c0049a0 con 0x7f9f5c106b80 2024-04-20T22:18:55.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:55.467+0000 7f9f52a00700 1 --2- 172.21.15.62:0/3594019936 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9f4006ecc0 0x7f9f40071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:18:55.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:55.467+0000 7f9f52a00700 1 -- 172.21.15.62:0/3594019936 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f9f4c08ebf0 con 0x7f9f5c106b80 2024-04-20T22:18:55.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:55.468+0000 7f9f59400700 1 --2- 172.21.15.62:0/3594019936 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9f4006ecc0 0x7f9f40071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:18:55.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:55.468+0000 7f9f59400700 1 --2- 172.21.15.62:0/3594019936 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9f4006ecc0 0x7f9f40071170 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f9f48006010 tx=0x7f9f4800b540 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:18:55.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:55.473+0000 7f9f52a00700 1 -- 172.21.15.62:0/3594019936 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f9f4c057900 con 0x7f9f5c106b80 2024-04-20T22:18:55.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:55.732+0000 7f9f60c00700 1 -- 172.21.15.62:0/3594019936 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9f5c1a18a0 con 0x7f9f4006ecc0 2024-04-20T22:18:55.733 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:18:55 smithi062 ceph-mon[76322]: pgmap v1312: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:18:55.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:55.734+0000 7f9f52a00700 1 -- 172.21.15.62:0/3594019936 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f9f5c1a18a0 con 0x7f9f4006ecc0 2024-04-20T22:18:55.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:55.737+0000 7f9f60c00700 1 -- 172.21.15.62:0/3594019936 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9f4006ecc0 msgr2=0x7f9f40071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:18:55.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:55.737+0000 7f9f60c00700 1 --2- 172.21.15.62:0/3594019936 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9f4006ecc0 0x7f9f40071170 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f9f48006010 tx=0x7f9f4800b540 comp rx=0 tx=0).stop 2024-04-20T22:18:55.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:55.737+0000 7f9f60c00700 1 -- 172.21.15.62:0/3594019936 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9f5c106b80 msgr2=0x7f9f5c19c440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:18:55.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:55.737+0000 7f9f60c00700 1 --2- 172.21.15.62:0/3594019936 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9f5c106b80 0x7f9f5c19c440 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f9f4c00c8a0 tx=0x7f9f4c00cc60 comp rx=0 tx=0).stop 2024-04-20T22:18:55.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:55.737+0000 7f9f60c00700 1 -- 172.21.15.62:0/3594019936 shutdown_connections 2024-04-20T22:18:55.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:18:55 smithi189 ceph-mon[57022]: pgmap v1312: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:18:55.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:55.737+0000 7f9f60c00700 1 --2- 172.21.15.62:0/3594019936 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9f4006ecc0 0x7f9f40071170 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:18:55.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:55.737+0000 7f9f60c00700 1 --2- 172.21.15.62:0/3594019936 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9f5c106b80 0x7f9f5c19c440 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:18:55.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:55.737+0000 7f9f60c00700 1 --2- 172.21.15.62:0/3594019936 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f5c10c8f0 0x7f9f5c19c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:18:55.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:55.737+0000 7f9f60c00700 1 -- 172.21.15.62:0/3594019936 >> 172.21.15.62:0/3594019936 conn(0x7f9f5c076060 msgr2=0x7f9f5c077ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:18:55.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:55.738+0000 7f9f60c00700 1 -- 172.21.15.62:0/3594019936 shutdown_connections 2024-04-20T22:18:55.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:55.738+0000 7f9f60c00700 1 -- 172.21.15.62:0/3594019936 wait complete. 2024-04-20T22:18:55.755 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T22:18:55.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:55.898+0000 7f21e8a00700 1 -- 172.21.15.62:0/1826096599 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f21e4106b80 msgr2=0x7f21e4106ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:18:55.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:55.898+0000 7f21e8a00700 1 --2- 172.21.15.62:0/1826096599 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f21e4106b80 0x7f21e4106ff0 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f21d0009910 tx=0x7f21d0009c20 comp rx=0 tx=0).stop 2024-04-20T22:18:55.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:55.899+0000 7f21e8a00700 1 -- 172.21.15.62:0/1826096599 shutdown_connections 2024-04-20T22:18:55.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:55.899+0000 7f21e8a00700 1 --2- 172.21.15.62:0/1826096599 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f21e4106b80 0x7f21e4106ff0 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:18:55.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:55.899+0000 7f21e8a00700 1 --2- 172.21.15.62:0/1826096599 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21e410c660 0x7f21e410ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:18:55.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:55.899+0000 7f21e8a00700 1 -- 172.21.15.62:0/1826096599 >> 172.21.15.62:0/1826096599 conn(0x7f21e4075e90 msgr2=0x7f21e40782a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:18:55.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:55.899+0000 7f21e8a00700 1 -- 172.21.15.62:0/1826096599 shutdown_connections 2024-04-20T22:18:55.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:55.899+0000 7f21e8a00700 1 -- 172.21.15.62:0/1826096599 wait complete. 2024-04-20T22:18:55.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:55.901+0000 7f21e8a00700 1 Processor -- start 2024-04-20T22:18:55.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:55.901+0000 7f21e8a00700 1 -- start start 2024-04-20T22:18:55.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:55.901+0000 7f21e8a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21e4106b80 0x7f21e419c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:18:55.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:55.901+0000 7f21e8a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f21e410c660 0x7f21e419c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:18:55.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:55.901+0000 7f21e8a00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21e419ce00 con 0x7f21e410c660 2024-04-20T22:18:55.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:55.901+0000 7f21e8a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21e41a0b90 con 0x7f21e4106b80 2024-04-20T22:18:55.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:55.902+0000 7f21e1e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f21e410c660 0x7f21e419c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:18:55.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:55.902+0000 7f21e1e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f21e410c660 0x7f21e419c720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35056/0 (socket says 172.21.15.62:35056) 2024-04-20T22:18:55.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:55.902+0000 7f21e1e00700 1 -- 172.21.15.62:0/1005397910 learned_addr learned my addr 172.21.15.62:0/1005397910 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:18:55.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:55.902+0000 7f21e2800700 1 --2- 172.21.15.62:0/1005397910 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21e4106b80 0x7f21e419c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:18:55.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:55.902+0000 7f21e1e00700 1 -- 172.21.15.62:0/1005397910 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21e4106b80 msgr2=0x7f21e419c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:18:55.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:55.902+0000 7f21e1e00700 1 --2- 172.21.15.62:0/1005397910 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21e4106b80 0x7f21e419c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:18:55.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:55.902+0000 7f21e1e00700 1 -- 172.21.15.62:0/1005397910 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21d00095f0 con 0x7f21e410c660 2024-04-20T22:18:55.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:55.902+0000 7f21e1e00700 1 --2- 172.21.15.62:0/1005397910 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f21e410c660 0x7f21e419c720 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f21d00092e0 tx=0x7f21d000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:18:55.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:55.903+0000 7f21db400700 1 -- 172.21.15.62:0/1005397910 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f21d001d070 con 0x7f21e410c660 2024-04-20T22:18:55.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:55.903+0000 7f21db400700 1 -- 172.21.15.62:0/1005397910 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f21d0005e00 con 0x7f21e410c660 2024-04-20T22:18:55.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:55.903+0000 7f21db400700 1 -- 172.21.15.62:0/1005397910 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f21d0017400 con 0x7f21e410c660 2024-04-20T22:18:55.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:55.903+0000 7f21e8a00700 1 -- 172.21.15.62:0/1005397910 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21e41a0e10 con 0x7f21e410c660 2024-04-20T22:18:55.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:55.903+0000 7f21e8a00700 1 -- 172.21.15.62:0/1005397910 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21e41a1300 con 0x7f21e410c660 2024-04-20T22:18:55.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:55.904+0000 7f21e8a00700 1 -- 172.21.15.62:0/1005397910 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21e410eb40 con 0x7f21e410c660 2024-04-20T22:18:55.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:55.905+0000 7f21db400700 1 -- 172.21.15.62:0/1005397910 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f21d000f460 con 0x7f21e410c660 2024-04-20T22:18:55.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:55.906+0000 7f21db400700 1 --2- 172.21.15.62:0/1005397910 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f21c806ecc0 0x7f21c8071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:18:55.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:55.907+0000 7f21db400700 1 -- 172.21.15.62:0/1005397910 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f21d0090330 con 0x7f21e410c660 2024-04-20T22:18:55.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:55.907+0000 7f21e2800700 1 --2- 172.21.15.62:0/1005397910 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f21c806ecc0 0x7f21c8071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:18:55.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:55.908+0000 7f21e2800700 1 --2- 172.21.15.62:0/1005397910 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f21c806ecc0 0x7f21c8071170 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f21d4005950 tx=0x7f21d40058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:18:55.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:55.912+0000 7f21db400700 1 -- 172.21.15.62:0/1005397910 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f21d00590c0 con 0x7f21e410c660 2024-04-20T22:18:56.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:56.175+0000 7f21e8a00700 1 -- 172.21.15.62:0/1005397910 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f21e402ccf0 con 0x7f21c806ecc0 2024-04-20T22:18:56.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:56.178+0000 7f21db400700 1 -- 172.21.15.62:0/1005397910 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f21e402ccf0 con 0x7f21c806ecc0 2024-04-20T22:18:56.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:56.180+0000 7f21e8a00700 1 -- 172.21.15.62:0/1005397910 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f21c806ecc0 msgr2=0x7f21c8071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:18:56.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:56.180+0000 7f21e8a00700 1 --2- 172.21.15.62:0/1005397910 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f21c806ecc0 0x7f21c8071170 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f21d4005950 tx=0x7f21d40058e0 comp rx=0 tx=0).stop 2024-04-20T22:18:56.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:56.180+0000 7f21e8a00700 1 -- 172.21.15.62:0/1005397910 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f21e410c660 msgr2=0x7f21e419c720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:18:56.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:56.180+0000 7f21e8a00700 1 --2- 172.21.15.62:0/1005397910 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f21e410c660 0x7f21e419c720 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f21d00092e0 tx=0x7f21d000be20 comp rx=0 tx=0).stop 2024-04-20T22:18:56.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:56.181+0000 7f21e8a00700 1 -- 172.21.15.62:0/1005397910 shutdown_connections 2024-04-20T22:18:56.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:56.181+0000 7f21e8a00700 1 --2- 172.21.15.62:0/1005397910 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f21c806ecc0 0x7f21c8071170 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:18:56.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:56.181+0000 7f21e8a00700 1 --2- 172.21.15.62:0/1005397910 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21e4106b80 0x7f21e419c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:18:56.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:56.181+0000 7f21e8a00700 1 --2- 172.21.15.62:0/1005397910 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f21e410c660 0x7f21e419c720 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:18:56.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:56.181+0000 7f21e8a00700 1 -- 172.21.15.62:0/1005397910 >> 172.21.15.62:0/1005397910 conn(0x7f21e4075e90 msgr2=0x7f21e40778d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:18:56.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:56.181+0000 7f21e8a00700 1 -- 172.21.15.62:0/1005397910 shutdown_connections 2024-04-20T22:18:56.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:56.181+0000 7f21e8a00700 1 -- 172.21.15.62:0/1005397910 wait complete. 2024-04-20T22:18:56.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:56.342+0000 7fabfac00700 1 -- 172.21.15.62:0/929550486 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fabf410c8f0 msgr2=0x7fabf410ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:18:56.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:56.342+0000 7fabfac00700 1 --2- 172.21.15.62:0/929550486 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fabf410c8f0 0x7fabf410ccc0 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7fabe4009910 tx=0x7fabe4009c20 comp rx=0 tx=0).stop 2024-04-20T22:18:56.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:56.342+0000 7fabfac00700 1 -- 172.21.15.62:0/929550486 shutdown_connections 2024-04-20T22:18:56.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:56.342+0000 7fabfac00700 1 --2- 172.21.15.62:0/929550486 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fabf4106b80 0x7fabf4106ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:18:56.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:56.342+0000 7fabfac00700 1 --2- 172.21.15.62:0/929550486 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fabf410c8f0 0x7fabf410ccc0 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:18:56.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:56.342+0000 7fabfac00700 1 -- 172.21.15.62:0/929550486 >> 172.21.15.62:0/929550486 conn(0x7fabf4076060 msgr2=0x7fabf4078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:18:56.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:56.342+0000 7fabfac00700 1 -- 172.21.15.62:0/929550486 shutdown_connections 2024-04-20T22:18:56.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:56.342+0000 7fabfac00700 1 -- 172.21.15.62:0/929550486 wait complete. 2024-04-20T22:18:56.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:56.343+0000 7fabfac00700 1 Processor -- start 2024-04-20T22:18:56.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:56.343+0000 7fabfac00700 1 -- start start 2024-04-20T22:18:56.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:56.344+0000 7fabfac00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fabf4106b80 0x7fabf419c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:18:56.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:56.344+0000 7fabfac00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fabf410c8f0 0x7fabf419ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:18:56.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:56.344+0000 7fabfac00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabf419d120 con 0x7fabf4106b80 2024-04-20T22:18:56.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:56.344+0000 7fabfac00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabf41a0eb0 con 0x7fabf410c8f0 2024-04-20T22:18:56.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:56.344+0000 7fabf3e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fabf4106b80 0x7fabf419c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:18:56.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:56.344+0000 7fabf3e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fabf4106b80 0x7fabf419c500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35064/0 (socket says 172.21.15.62:35064) 2024-04-20T22:18:56.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:56.344+0000 7fabf3e00700 1 -- 172.21.15.62:0/3682879493 learned_addr learned my addr 172.21.15.62:0/3682879493 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:18:56.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:56.344+0000 7fabebe00700 1 --2- 172.21.15.62:0/3682879493 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fabf410c8f0 0x7fabf419ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:18:56.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:56.344+0000 7fabf3e00700 1 -- 172.21.15.62:0/3682879493 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fabf410c8f0 msgr2=0x7fabf419ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:18:56.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:56.344+0000 7fabf3e00700 1 --2- 172.21.15.62:0/3682879493 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fabf410c8f0 0x7fabf419ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:18:56.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:56.344+0000 7fabf3e00700 1 -- 172.21.15.62:0/3682879493 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabe40095f0 con 0x7fabf4106b80 2024-04-20T22:18:56.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:56.345+0000 7fabf3e00700 1 --2- 172.21.15.62:0/3682879493 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fabf4106b80 0x7fabf419c500 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7fabe4000c00 tx=0x7fabe400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:18:56.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:56.345+0000 7fabf1600700 1 -- 172.21.15.62:0/3682879493 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fabe401d070 con 0x7fabf4106b80 2024-04-20T22:18:56.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:56.345+0000 7fabf1600700 1 -- 172.21.15.62:0/3682879493 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fabe4005dc0 con 0x7fabf4106b80 2024-04-20T22:18:56.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:56.345+0000 7fabf1600700 1 -- 172.21.15.62:0/3682879493 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fabe4017400 con 0x7fabf4106b80 2024-04-20T22:18:56.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:56.345+0000 7fabfac00700 1 -- 172.21.15.62:0/3682879493 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabf41a1130 con 0x7fabf4106b80 2024-04-20T22:18:56.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:56.345+0000 7fabfac00700 1 -- 172.21.15.62:0/3682879493 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabf41a1620 con 0x7fabf4106b80 2024-04-20T22:18:56.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:56.347+0000 7fabfac00700 1 -- 172.21.15.62:0/3682879493 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabf4109b20 con 0x7fabf4106b80 2024-04-20T22:18:56.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:56.347+0000 7fabf1600700 1 -- 172.21.15.62:0/3682879493 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fabe400f7f0 con 0x7fabf4106b80 2024-04-20T22:18:56.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:56.348+0000 7fabf1600700 1 --2- 172.21.15.62:0/3682879493 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fabd806ecc0 0x7fabd8071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:18:56.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:56.348+0000 7fabf1600700 1 -- 172.21.15.62:0/3682879493 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fabe40902b0 con 0x7fabf4106b80 2024-04-20T22:18:56.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:56.349+0000 7fabebe00700 1 --2- 172.21.15.62:0/3682879493 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fabd806ecc0 0x7fabd8071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:18:56.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:56.349+0000 7fabebe00700 1 --2- 172.21.15.62:0/3682879493 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fabd806ecc0 0x7fabd8071170 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fabdc005950 tx=0x7fabdc0058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:18:56.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:56.353+0000 7fabf1600700 1 -- 172.21.15.62:0/3682879493 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fabe40590e0 con 0x7fabf4106b80 2024-04-20T22:18:56.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:56.597+0000 7fabfac00700 1 -- 172.21.15.62:0/3682879493 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fabf419d990 con 0x7fabd806ecc0 2024-04-20T22:18:56.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:56.631+0000 7fabf1600700 1 -- 172.21.15.62:0/3682879493 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fabf419d990 con 0x7fabd806ecc0 2024-04-20T22:18:56.631 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T22:18:56.631 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (44m) 5m ago 58m 14.6M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T22:18:56.631 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (59m) 5m ago 59m 11.6M - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T22:18:56.631 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (57m) 5m ago 57m 11.2M - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T22:18:56.631 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (40m) 5m ago 58m 6660k - 19.0.0-3133-ga8c84f59 76bc6b91b726 2825d5ba11c6 2024-04-20T22:18:56.631 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (40m) 5m ago 57m 6665k - 19.0.0-3133-ga8c84f59 76bc6b91b726 20f746fa1bba 2024-04-20T22:18:56.631 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (44m) 5m ago 58m 75.5M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T22:18:56.631 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (36m) 5m ago 51m 22.9M - 19.0.0-3133-ga8c84f59 76bc6b91b726 de80bc90a683 2024-04-20T22:18:56.632 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (36m) 5m ago 51m 23.0M - 19.0.0-3133-ga8c84f59 76bc6b91b726 cd793b7ee98e 2024-04-20T22:18:56.632 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (21m) 5m ago 51m 18.5M - 19.0.0-3133-ga8c84f59 76bc6b91b726 0e7b0bc4d1ab 2024-04-20T22:18:56.632 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (5m) 5m ago 51m 11.4M - 19.0.0-3133-ga8c84f59 76bc6b91b726 0af732ac9e06 2024-04-20T22:18:56.632 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (47m) 5m ago 60m 517M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T22:18:56.632 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (46m) 5m ago 57m 460M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T22:18:56.632 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (42m) 5m ago 60m 86.3M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 019eb1479485 2024-04-20T22:18:56.632 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (41m) 5m ago 57m 58.0M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 55aac560aab6 2024-04-20T22:18:56.632 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (45m) 5m ago 58m 8766k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T22:18:56.632 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (45m) 5m ago 57m 8711k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T22:18:56.632 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (40m) 5m ago 56m 91.3M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 ee2bb901f26e 2024-04-20T22:18:56.632 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (39m) 5m ago 55m 94.9M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 9075152d4162 2024-04-20T22:18:56.632 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (39m) 5m ago 55m 78.5M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 2d4f3e38dc0e 2024-04-20T22:18:56.632 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (38m) 5m ago 54m 94.1M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 8f97b17ae0eb 2024-04-20T22:18:56.632 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (37m) 5m ago 53m 96.1M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 653d8079967d 2024-04-20T22:18:56.633 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (37m) 5m ago 53m 90.9M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 596dded14c12 2024-04-20T22:18:56.633 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (45m) 5m ago 58m 90.4M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T22:18:56.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:56.634+0000 7fabfac00700 1 -- 172.21.15.62:0/3682879493 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fabd806ecc0 msgr2=0x7fabd8071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:18:56.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:56.634+0000 7fabfac00700 1 --2- 172.21.15.62:0/3682879493 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fabd806ecc0 0x7fabd8071170 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fabdc005950 tx=0x7fabdc0058e0 comp rx=0 tx=0).stop 2024-04-20T22:18:56.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:56.635+0000 7fabfac00700 1 -- 172.21.15.62:0/3682879493 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fabf4106b80 msgr2=0x7fabf419c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:18:56.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:56.635+0000 7fabfac00700 1 --2- 172.21.15.62:0/3682879493 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fabf4106b80 0x7fabf419c500 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7fabe4000c00 tx=0x7fabe400be20 comp rx=0 tx=0).stop 2024-04-20T22:18:56.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:56.635+0000 7fabfac00700 1 -- 172.21.15.62:0/3682879493 shutdown_connections 2024-04-20T22:18:56.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:56.635+0000 7fabfac00700 1 --2- 172.21.15.62:0/3682879493 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fabd806ecc0 0x7fabd8071170 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:18:56.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:56.635+0000 7fabfac00700 1 --2- 172.21.15.62:0/3682879493 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fabf4106b80 0x7fabf419c500 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:18:56.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:56.635+0000 7fabfac00700 1 --2- 172.21.15.62:0/3682879493 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fabf410c8f0 0x7fabf419ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:18:56.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:56.635+0000 7fabfac00700 1 -- 172.21.15.62:0/3682879493 >> 172.21.15.62:0/3682879493 conn(0x7fabf4076060 msgr2=0x7fabf4077aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:18:56.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:56.635+0000 7fabfac00700 1 -- 172.21.15.62:0/3682879493 shutdown_connections 2024-04-20T22:18:56.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:56.635+0000 7fabfac00700 1 -- 172.21.15.62:0/3682879493 wait complete. 2024-04-20T22:18:56.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:18:56 smithi189 ceph-mon[57022]: from='client.36300 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:18:56.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:18:56 smithi189 ceph-mon[57022]: from='client.36304 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:18:56.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:18:56 smithi062 ceph-mon[76322]: from='client.36300 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:18:56.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:18:56 smithi062 ceph-mon[76322]: from='client.36304 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:18:56.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:56.793+0000 7fceecc00700 1 -- 172.21.15.62:0/288409517 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcee8102470 msgr2=0x7fcee81028e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:18:56.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:56.793+0000 7fceecc00700 1 --2- 172.21.15.62:0/288409517 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcee8102470 0x7fcee81028e0 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7fced4009910 tx=0x7fced4009c20 comp rx=0 tx=0).stop 2024-04-20T22:18:56.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:56.794+0000 7fceecc00700 1 -- 172.21.15.62:0/288409517 shutdown_connections 2024-04-20T22:18:56.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:56.794+0000 7fceecc00700 1 --2- 172.21.15.62:0/288409517 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcee8102470 0x7fcee81028e0 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:18:56.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:56.794+0000 7fceecc00700 1 --2- 172.21.15.62:0/288409517 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcee8108470 0x7fcee8108840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:18:56.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:56.794+0000 7fceecc00700 1 -- 172.21.15.62:0/288409517 >> 172.21.15.62:0/288409517 conn(0x7fcee80fdf90 msgr2=0x7fcee81003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:18:56.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:56.794+0000 7fceecc00700 1 -- 172.21.15.62:0/288409517 shutdown_connections 2024-04-20T22:18:56.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:56.794+0000 7fceecc00700 1 -- 172.21.15.62:0/288409517 wait complete. 2024-04-20T22:18:56.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:56.795+0000 7fceecc00700 1 Processor -- start 2024-04-20T22:18:56.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:56.795+0000 7fceecc00700 1 -- start start 2024-04-20T22:18:56.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:56.795+0000 7fceecc00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcee8102470 0x7fcee8198270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:18:56.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:56.795+0000 7fceecc00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcee8108470 0x7fcee81987b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:18:56.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:56.795+0000 7fceecc00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcee8198e00 con 0x7fcee8108470 2024-04-20T22:18:56.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:56.795+0000 7fceecc00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcee8198f40 con 0x7fcee8102470 2024-04-20T22:18:56.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:56.796+0000 7fcee5400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcee8108470 0x7fcee81987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:18:56.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:56.796+0000 7fcee5400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcee8108470 0x7fcee81987b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35086/0 (socket says 172.21.15.62:35086) 2024-04-20T22:18:56.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:56.796+0000 7fcee5400700 1 -- 172.21.15.62:0/899504946 learned_addr learned my addr 172.21.15.62:0/899504946 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:18:56.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:56.796+0000 7fcee5400700 1 -- 172.21.15.62:0/899504946 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcee8102470 msgr2=0x7fcee8198270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:18:56.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:56.796+0000 7fcee5e00700 1 --2- 172.21.15.62:0/899504946 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcee8102470 0x7fcee8198270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:18:56.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:56.796+0000 7fcee5400700 1 --2- 172.21.15.62:0/899504946 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcee8102470 0x7fcee8198270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:18:56.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:56.796+0000 7fcee5400700 1 -- 172.21.15.62:0/899504946 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fced40095f0 con 0x7fcee8108470 2024-04-20T22:18:56.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:56.796+0000 7fcee5e00700 1 --2- 172.21.15.62:0/899504946 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcee8102470 0x7fcee8198270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T22:18:56.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:56.797+0000 7fcee5400700 1 --2- 172.21.15.62:0/899504946 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcee8108470 0x7fcee81987b0 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7fced4006010 tx=0x7fced400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:18:56.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:56.797+0000 7fcedea00700 1 -- 172.21.15.62:0/899504946 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fced401d070 con 0x7fcee8108470 2024-04-20T22:18:56.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:56.797+0000 7fcedea00700 1 -- 172.21.15.62:0/899504946 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fced4004fb0 con 0x7fcee8108470 2024-04-20T22:18:56.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:56.797+0000 7fceecc00700 1 -- 172.21.15.62:0/899504946 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcee819cce0 con 0x7fcee8108470 2024-04-20T22:18:56.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:56.797+0000 7fcedea00700 1 -- 172.21.15.62:0/899504946 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fced4022ea0 con 0x7fcee8108470 2024-04-20T22:18:56.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:56.797+0000 7fceecc00700 1 -- 172.21.15.62:0/899504946 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcee819d1d0 con 0x7fcee8108470 2024-04-20T22:18:56.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:56.799+0000 7fceecc00700 1 -- 172.21.15.62:0/899504946 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcee810a970 con 0x7fcee8108470 2024-04-20T22:18:56.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:56.799+0000 7fcedea00700 1 -- 172.21.15.62:0/899504946 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fced4022470 con 0x7fcee8108470 2024-04-20T22:18:56.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:56.800+0000 7fcedea00700 1 --2- 172.21.15.62:0/899504946 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fcecc06ecc0 0x7fcecc071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:18:56.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:56.801+0000 7fcedea00700 1 -- 172.21.15.62:0/899504946 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fced4090320 con 0x7fcee8108470 2024-04-20T22:18:56.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:56.801+0000 7fcee5e00700 1 --2- 172.21.15.62:0/899504946 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fcecc06ecc0 0x7fcecc071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:18:56.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:56.802+0000 7fcee5e00700 1 --2- 172.21.15.62:0/899504946 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fcecc06ecc0 0x7fcecc071170 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fced8005fd0 tx=0x7fced8005e20 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:18:56.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:56.805+0000 7fcedea00700 1 -- 172.21.15.62:0/899504946 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fced4059200 con 0x7fcee8108470 2024-04-20T22:18:57.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:57.139+0000 7fceecc00700 1 -- 172.21.15.62:0/899504946 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fcee8199580 con 0x7fcee8108470 2024-04-20T22:18:57.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:57.141+0000 7fcedea00700 1 -- 172.21.15.62:0/899504946 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7fced405c8c0 con 0x7fcee8108470 2024-04-20T22:18:57.141 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:18:57.141 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T22:18:57.141 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:18:57.141 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:18:57.141 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T22:18:57.141 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:18:57.141 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:18:57.141 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T22:18:57.142 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 6 2024-04-20T22:18:57.142 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:18:57.142 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T22:18:57.142 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 4 2024-04-20T22:18:57.142 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:18:57.142 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T22:18:57.142 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 14 2024-04-20T22:18:57.142 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T22:18:57.142 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:18:57.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:57.143+0000 7fceecc00700 1 -- 172.21.15.62:0/899504946 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fcecc06ecc0 msgr2=0x7fcecc071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:18:57.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:57.143+0000 7fceecc00700 1 --2- 172.21.15.62:0/899504946 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fcecc06ecc0 0x7fcecc071170 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fced8005fd0 tx=0x7fced8005e20 comp rx=0 tx=0).stop 2024-04-20T22:18:57.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:57.143+0000 7fceecc00700 1 -- 172.21.15.62:0/899504946 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcee8108470 msgr2=0x7fcee81987b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:18:57.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:57.143+0000 7fceecc00700 1 --2- 172.21.15.62:0/899504946 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcee8108470 0x7fcee81987b0 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7fced4006010 tx=0x7fced400be20 comp rx=0 tx=0).stop 2024-04-20T22:18:57.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:57.144+0000 7fceecc00700 1 -- 172.21.15.62:0/899504946 shutdown_connections 2024-04-20T22:18:57.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:57.144+0000 7fceecc00700 1 --2- 172.21.15.62:0/899504946 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fcecc06ecc0 0x7fcecc071170 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:18:57.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:57.144+0000 7fceecc00700 1 --2- 172.21.15.62:0/899504946 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcee8102470 0x7fcee8198270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:18:57.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:57.144+0000 7fceecc00700 1 --2- 172.21.15.62:0/899504946 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcee8108470 0x7fcee81987b0 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:18:57.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:57.144+0000 7fceecc00700 1 -- 172.21.15.62:0/899504946 >> 172.21.15.62:0/899504946 conn(0x7fcee80fdf90 msgr2=0x7fcee80ff470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:18:57.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:57.144+0000 7fceecc00700 1 -- 172.21.15.62:0/899504946 shutdown_connections 2024-04-20T22:18:57.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:57.144+0000 7fceecc00700 1 -- 172.21.15.62:0/899504946 wait complete. 2024-04-20T22:18:57.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:57.296+0000 7fbccfc00700 1 -- 172.21.15.62:0/1333226177 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbcc81004e0 msgr2=0x7fbcc81008b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:18:57.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:57.296+0000 7fbccfc00700 1 --2- 172.21.15.62:0/1333226177 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbcc81004e0 0x7fbcc81008b0 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7fbcbc009910 tx=0x7fbcbc009c20 comp rx=0 tx=0).stop 2024-04-20T22:18:57.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:57.297+0000 7fbccfc00700 1 -- 172.21.15.62:0/1333226177 shutdown_connections 2024-04-20T22:18:57.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:57.297+0000 7fbccfc00700 1 --2- 172.21.15.62:0/1333226177 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbcc8100e80 0x7fbcc8104e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:18:57.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:57.297+0000 7fbccfc00700 1 --2- 172.21.15.62:0/1333226177 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbcc81004e0 0x7fbcc81008b0 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:18:57.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:57.297+0000 7fbccfc00700 1 -- 172.21.15.62:0/1333226177 >> 172.21.15.62:0/1333226177 conn(0x7fbcc80fbda0 msgr2=0x7fbcc80fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:18:57.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:57.297+0000 7fbccfc00700 1 -- 172.21.15.62:0/1333226177 shutdown_connections 2024-04-20T22:18:57.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:57.297+0000 7fbccfc00700 1 -- 172.21.15.62:0/1333226177 wait complete. 2024-04-20T22:18:57.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:57.298+0000 7fbccfc00700 1 Processor -- start 2024-04-20T22:18:57.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:57.298+0000 7fbccfc00700 1 -- start start 2024-04-20T22:18:57.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:57.298+0000 7fbccfc00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbcc81004e0 0x7fbcc81980d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:18:57.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:57.299+0000 7fbccfc00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbcc8100e80 0x7fbcc8198610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:18:57.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:57.299+0000 7fbccfc00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcc8198cf0 con 0x7fbcc8100e80 2024-04-20T22:18:57.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:57.299+0000 7fbccfc00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcc819ca80 con 0x7fbcc81004e0 2024-04-20T22:18:57.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:57.299+0000 7fbcc7e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbcc8100e80 0x7fbcc8198610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:18:57.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:57.299+0000 7fbcc7e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbcc8100e80 0x7fbcc8198610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35102/0 (socket says 172.21.15.62:35102) 2024-04-20T22:18:57.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:57.299+0000 7fbcc7e00700 1 -- 172.21.15.62:0/2738754788 learned_addr learned my addr 172.21.15.62:0/2738754788 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:18:57.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:57.299+0000 7fbccd200700 1 --2- 172.21.15.62:0/2738754788 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbcc81004e0 0x7fbcc81980d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:18:57.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:57.299+0000 7fbcc7e00700 1 -- 172.21.15.62:0/2738754788 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbcc81004e0 msgr2=0x7fbcc81980d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:18:57.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:57.299+0000 7fbcc7e00700 1 --2- 172.21.15.62:0/2738754788 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbcc81004e0 0x7fbcc81980d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:18:57.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:57.299+0000 7fbcc7e00700 1 -- 172.21.15.62:0/2738754788 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcbc0095f0 con 0x7fbcc8100e80 2024-04-20T22:18:57.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:57.300+0000 7fbcc7e00700 1 --2- 172.21.15.62:0/2738754788 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbcc8100e80 0x7fbcc8198610 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7fbcb400c8f0 tx=0x7fbcb400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:18:57.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:57.300+0000 7fbcc5600700 1 -- 172.21.15.62:0/2738754788 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbcb4007a50 con 0x7fbcc8100e80 2024-04-20T22:18:57.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:57.300+0000 7fbcc5600700 1 -- 172.21.15.62:0/2738754788 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbcb4007bb0 con 0x7fbcc8100e80 2024-04-20T22:18:57.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:57.300+0000 7fbcc5600700 1 -- 172.21.15.62:0/2738754788 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbcb4018630 con 0x7fbcc8100e80 2024-04-20T22:18:57.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:57.300+0000 7fbccfc00700 1 -- 172.21.15.62:0/2738754788 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcc819cd60 con 0x7fbcc8100e80 2024-04-20T22:18:57.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:57.300+0000 7fbccfc00700 1 -- 172.21.15.62:0/2738754788 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcc819d2b0 con 0x7fbcc8100e80 2024-04-20T22:18:57.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:57.301+0000 7fbccfc00700 1 -- 172.21.15.62:0/2738754788 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcc81045d0 con 0x7fbcc8100e80 2024-04-20T22:18:57.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:57.303+0000 7fbcc5600700 1 -- 172.21.15.62:0/2738754788 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fbcb4007d20 con 0x7fbcc8100e80 2024-04-20T22:18:57.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:57.303+0000 7fbcc5600700 1 --2- 172.21.15.62:0/2738754788 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fbcb006ecc0 0x7fbcb0071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:18:57.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:57.304+0000 7fbcc5600700 1 -- 172.21.15.62:0/2738754788 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fbcb408ee00 con 0x7fbcc8100e80 2024-04-20T22:18:57.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:57.304+0000 7fbccd200700 1 --2- 172.21.15.62:0/2738754788 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fbcb006ecc0 0x7fbcb0071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:18:57.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:57.305+0000 7fbccd200700 1 --2- 172.21.15.62:0/2738754788 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fbcb006ecc0 0x7fbcb0071170 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fbcbc006010 tx=0x7fbcbc01a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:18:57.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:57.309+0000 7fbcc5600700 1 -- 172.21.15.62:0/2738754788 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fbcb4057b90 con 0x7fbcc8100e80 2024-04-20T22:18:57.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:57.591+0000 7fbccfc00700 1 -- 172.21.15.62:0/2738754788 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fbcc819d580 con 0x7fbcc8100e80 2024-04-20T22:18:57.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:57.592+0000 7fbcc5600700 1 -- 172.21.15.62:0/2738754788 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 29 v29) v1 ==== 76+0+1887 (secure 0 0 0) 0x7fbcb405b250 con 0x7fbcc8100e80 2024-04-20T22:18:57.593 INFO:teuthology.orchestra.run.smithi062.stdout:e29 2024-04-20T22:18:57.593 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T22:18:57.593 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:18:57.593 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T22:18:57.593 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:18:57.593 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T22:18:57.593 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T22:18:57.593 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 20 2024-04-20T22:18:57.593 INFO:teuthology.orchestra.run.smithi062.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-20T22:18:57.593 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T22:18:57.593 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T21:42:01.559061+0000 2024-04-20T22:18:57.593 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T22:18:57.593 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T22:18:57.594 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T22:18:57.594 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T22:18:57.594 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T22:18:57.594 INFO:teuthology.orchestra.run.smithi062.stdout:max_xattr_size 65536 2024-04-20T22:18:57.594 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T22:18:57.594 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T22:18:57.594 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 81 2024-04-20T22:18:57.594 INFO:teuthology.orchestra.run.smithi062.stdout:snap_rstat 1 2024-04-20T22:18:57.594 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:18:57.594 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 2 2024-04-20T22:18:57.594 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1 2024-04-20T22:18:57.594 INFO:teuthology.orchestra.run.smithi062.stdout:up {} 2024-04-20T22:18:57.594 INFO:teuthology.orchestra.run.smithi062.stdout:failed 0,1 2024-04-20T22:18:57.594 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T22:18:57.595 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T22:18:57.595 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T22:18:57.595 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T22:18:57.595 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data enabled 2024-04-20T22:18:57.595 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T22:18:57.595 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T22:18:57.595 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 1 2024-04-20T22:18:57.595 INFO:teuthology.orchestra.run.smithi062.stdout:qdb_cluster leader: 0 members: 2024-04-20T22:18:57.595 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:18:57.595 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:18:57.595 INFO:teuthology.orchestra.run.smithi062.stdout:Standby daemons: 2024-04-20T22:18:57.595 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:18:57.595 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ryqblb{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6826/4113916205,v1:172.21.15.62:6827/4113916205] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:18:57.595 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ozizwi{-1:34476} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6828/3801975165,v1:172.21.15.62:6829/3801975165] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:18:57.595 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.dvimer{-1:44727} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6824/3576699232,v1:172.21.15.189:6825/3576699232] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:18:57.596 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.xbgspw{-1:45075} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6826/212905792,v1:172.21.15.189:6827/212905792] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:18:57.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:57.594+0000 7fbccfc00700 1 -- 172.21.15.62:0/2738754788 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fbcb006ecc0 msgr2=0x7fbcb0071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:18:57.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:57.594+0000 7fbccfc00700 1 --2- 172.21.15.62:0/2738754788 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fbcb006ecc0 0x7fbcb0071170 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fbcbc006010 tx=0x7fbcbc01a040 comp rx=0 tx=0).stop 2024-04-20T22:18:57.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:57.594+0000 7fbccfc00700 1 -- 172.21.15.62:0/2738754788 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbcc8100e80 msgr2=0x7fbcc8198610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:18:57.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:57.594+0000 7fbccfc00700 1 --2- 172.21.15.62:0/2738754788 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbcc8100e80 0x7fbcc8198610 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7fbcb400c8f0 tx=0x7fbcb400ccb0 comp rx=0 tx=0).stop 2024-04-20T22:18:57.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:57.595+0000 7fbccfc00700 1 -- 172.21.15.62:0/2738754788 shutdown_connections 2024-04-20T22:18:57.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:57.595+0000 7fbccfc00700 1 --2- 172.21.15.62:0/2738754788 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fbcb006ecc0 0x7fbcb0071170 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:18:57.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:57.595+0000 7fbccfc00700 1 --2- 172.21.15.62:0/2738754788 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbcc81004e0 0x7fbcc81980d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:18:57.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:57.595+0000 7fbccfc00700 1 --2- 172.21.15.62:0/2738754788 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbcc8100e80 0x7fbcc8198610 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:18:57.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:57.595+0000 7fbccfc00700 1 -- 172.21.15.62:0/2738754788 >> 172.21.15.62:0/2738754788 conn(0x7fbcc80fbda0 msgr2=0x7fbcc80fe150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:18:57.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:57.595+0000 7fbccfc00700 1 -- 172.21.15.62:0/2738754788 shutdown_connections 2024-04-20T22:18:57.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:57.595+0000 7fbccfc00700 1 -- 172.21.15.62:0/2738754788 wait complete. 2024-04-20T22:18:57.597 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 29 2024-04-20T22:18:57.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:57.755+0000 7fe98e200700 1 -- 172.21.15.62:0/874674943 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe988101210 msgr2=0x7fe9881015e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:18:57.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:57.755+0000 7fe98e200700 1 --2- 172.21.15.62:0/874674943 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe988101210 0x7fe9881015e0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7fe97c009910 tx=0x7fe97c009c20 comp rx=0 tx=0).stop 2024-04-20T22:18:57.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:57.755+0000 7fe98e200700 1 -- 172.21.15.62:0/874674943 shutdown_connections 2024-04-20T22:18:57.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:57.755+0000 7fe98e200700 1 --2- 172.21.15.62:0/874674943 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe988101b20 0x7fe98810a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:18:57.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:57.755+0000 7fe98e200700 1 --2- 172.21.15.62:0/874674943 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe988101210 0x7fe9881015e0 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:18:57.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:57.755+0000 7fe98e200700 1 -- 172.21.15.62:0/874674943 >> 172.21.15.62:0/874674943 conn(0x7fe9880faa30 msgr2=0x7fe9880fce40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:18:57.755 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:18:57 smithi062 ceph-mon[76322]: from='client.36308 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:18:57.755 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:18:57 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/899504946' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:18:57.755 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:18:57 smithi062 ceph-mon[76322]: pgmap v1313: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:18:57.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:57.755+0000 7fe98e200700 1 -- 172.21.15.62:0/874674943 shutdown_connections 2024-04-20T22:18:57.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:57.755+0000 7fe98e200700 1 -- 172.21.15.62:0/874674943 wait complete. 2024-04-20T22:18:57.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:57.757+0000 7fe98e200700 1 Processor -- start 2024-04-20T22:18:57.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:57.757+0000 7fe98e200700 1 -- start start 2024-04-20T22:18:57.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:57.758+0000 7fe98e200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe988101210 0x7fe9880fee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:18:57.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:57.758+0000 7fe98e200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe988101b20 0x7fe9880ff3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:18:57.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:57.758+0000 7fe98e200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9880ff8f0 con 0x7fe988101210 2024-04-20T22:18:57.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:57.758+0000 7fe98e200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9880ffa30 con 0x7fe988101b20 2024-04-20T22:18:57.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:57.758+0000 7fe98ce00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe988101210 0x7fe9880fee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:18:57.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:57.759+0000 7fe98ce00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe988101210 0x7fe9880fee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35114/0 (socket says 172.21.15.62:35114) 2024-04-20T22:18:57.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:57.759+0000 7fe98ce00700 1 -- 172.21.15.62:0/1037211719 learned_addr learned my addr 172.21.15.62:0/1037211719 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:18:57.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:57.759+0000 7fe987e00700 1 --2- 172.21.15.62:0/1037211719 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe988101b20 0x7fe9880ff3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:18:57.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:57.759+0000 7fe98ce00700 1 -- 172.21.15.62:0/1037211719 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe988101b20 msgr2=0x7fe9880ff3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:18:57.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:57.759+0000 7fe98ce00700 1 --2- 172.21.15.62:0/1037211719 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe988101b20 0x7fe9880ff3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:18:57.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:57.759+0000 7fe98ce00700 1 -- 172.21.15.62:0/1037211719 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe97c0095f0 con 0x7fe988101210 2024-04-20T22:18:57.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:57.760+0000 7fe98ce00700 1 --2- 172.21.15.62:0/1037211719 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe988101210 0x7fe9880fee70 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7fe97c006010 tx=0x7fe97c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:18:57.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:57.760+0000 7fe985600700 1 -- 172.21.15.62:0/1037211719 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe97c01d070 con 0x7fe988101210 2024-04-20T22:18:57.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:57.760+0000 7fe985600700 1 -- 172.21.15.62:0/1037211719 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe97c00f460 con 0x7fe988101210 2024-04-20T22:18:57.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:57.760+0000 7fe98e200700 1 -- 172.21.15.62:0/1037211719 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9880ffcb0 con 0x7fe988101210 2024-04-20T22:18:57.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:57.760+0000 7fe985600700 1 -- 172.21.15.62:0/1037211719 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe97c017630 con 0x7fe988101210 2024-04-20T22:18:57.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:57.760+0000 7fe98e200700 1 -- 172.21.15.62:0/1037211719 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9881a20d0 con 0x7fe988101210 2024-04-20T22:18:57.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:57.762+0000 7fe98e200700 1 -- 172.21.15.62:0/1037211719 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9881076f0 con 0x7fe988101210 2024-04-20T22:18:57.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:57.762+0000 7fe985600700 1 -- 172.21.15.62:0/1037211719 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe97c00f5d0 con 0x7fe988101210 2024-04-20T22:18:57.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:57.763+0000 7fe985600700 1 --2- 172.21.15.62:0/1037211719 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe97006ecc0 0x7fe970071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:18:57.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:57.764+0000 7fe985600700 1 -- 172.21.15.62:0/1037211719 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fe97c090330 con 0x7fe988101210 2024-04-20T22:18:57.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:57.764+0000 7fe987e00700 1 --2- 172.21.15.62:0/1037211719 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe97006ecc0 0x7fe970071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:18:57.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:57.765+0000 7fe987e00700 1 --2- 172.21.15.62:0/1037211719 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe97006ecc0 0x7fe970071170 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fe974005950 tx=0x7fe9740058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:18:57.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:57.768+0000 7fe985600700 1 -- 172.21.15.62:0/1037211719 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fe97c0590c0 con 0x7fe988101210 2024-04-20T22:18:57.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:18:57 smithi189 ceph-mon[57022]: from='client.36308 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:18:57.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:18:57 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/899504946' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:18:57.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:18:57 smithi189 ceph-mon[57022]: pgmap v1313: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:18:58.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:58.023+0000 7fe98e200700 1 -- 172.21.15.62:0/1037211719 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe988061190 con 0x7fe97006ecc0 2024-04-20T22:18:58.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:58.024+0000 7fe985600700 1 -- 172.21.15.62:0/1037211719 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fe988061190 con 0x7fe97006ecc0 2024-04-20T22:18:58.024 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:18:58.024 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T22:18:58.024 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T22:18:58.024 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-20T22:18:58.024 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T22:18:58.025 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr", 2024-04-20T22:18:58.025 INFO:teuthology.orchestra.run.smithi062.stdout: "osd", 2024-04-20T22:18:58.025 INFO:teuthology.orchestra.run.smithi062.stdout: "mds", 2024-04-20T22:18:58.025 INFO:teuthology.orchestra.run.smithi062.stdout: "crash", 2024-04-20T22:18:58.025 INFO:teuthology.orchestra.run.smithi062.stdout: "mon" 2024-04-20T22:18:58.025 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T22:18:58.025 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "16/23 daemons upgraded", 2024-04-20T22:18:58.025 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading mds daemons", 2024-04-20T22:18:58.025 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T22:18:58.025 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:18:58.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:58.027+0000 7fe98e200700 1 -- 172.21.15.62:0/1037211719 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe97006ecc0 msgr2=0x7fe970071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:18:58.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:58.027+0000 7fe98e200700 1 --2- 172.21.15.62:0/1037211719 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe97006ecc0 0x7fe970071170 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fe974005950 tx=0x7fe9740058e0 comp rx=0 tx=0).stop 2024-04-20T22:18:58.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:58.027+0000 7fe98e200700 1 -- 172.21.15.62:0/1037211719 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe988101210 msgr2=0x7fe9880fee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:18:58.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:58.027+0000 7fe98e200700 1 --2- 172.21.15.62:0/1037211719 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe988101210 0x7fe9880fee70 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7fe97c006010 tx=0x7fe97c00be20 comp rx=0 tx=0).stop 2024-04-20T22:18:58.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:58.027+0000 7fe98e200700 1 -- 172.21.15.62:0/1037211719 shutdown_connections 2024-04-20T22:18:58.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:58.027+0000 7fe98e200700 1 --2- 172.21.15.62:0/1037211719 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe97006ecc0 0x7fe970071170 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:18:58.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:58.027+0000 7fe98e200700 1 --2- 172.21.15.62:0/1037211719 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe988101210 0x7fe9880fee70 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:18:58.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:58.027+0000 7fe98e200700 1 --2- 172.21.15.62:0/1037211719 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe988101b20 0x7fe9880ff3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:18:58.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:58.027+0000 7fe98e200700 1 -- 172.21.15.62:0/1037211719 >> 172.21.15.62:0/1037211719 conn(0x7fe9880faa30 msgr2=0x7fe988104830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:18:58.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:58.027+0000 7fe98e200700 1 -- 172.21.15.62:0/1037211719 shutdown_connections 2024-04-20T22:18:58.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:58.027+0000 7fe98e200700 1 -- 172.21.15.62:0/1037211719 wait complete. 2024-04-20T22:18:58.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:58.184+0000 7fa65f600700 1 -- 172.21.15.62:0/4261441284 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa6581026a0 msgr2=0x7fa658102a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:18:58.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:58.184+0000 7fa65f600700 1 --2- 172.21.15.62:0/4261441284 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa6581026a0 0x7fa658102a70 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7fa64c009910 tx=0x7fa64c009c20 comp rx=0 tx=0).stop 2024-04-20T22:18:58.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:58.184+0000 7fa65f600700 1 -- 172.21.15.62:0/4261441284 shutdown_connections 2024-04-20T22:18:58.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:58.184+0000 7fa65f600700 1 --2- 172.21.15.62:0/4261441284 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa658103040 0x7fa658107010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:18:58.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:58.184+0000 7fa65f600700 1 --2- 172.21.15.62:0/4261441284 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa6581026a0 0x7fa658102a70 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:18:58.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:58.184+0000 7fa65f600700 1 -- 172.21.15.62:0/4261441284 >> 172.21.15.62:0/4261441284 conn(0x7fa6580fdf40 msgr2=0x7fa658100350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:18:58.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:58.184+0000 7fa65f600700 1 -- 172.21.15.62:0/4261441284 shutdown_connections 2024-04-20T22:18:58.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:58.184+0000 7fa65f600700 1 -- 172.21.15.62:0/4261441284 wait complete. 2024-04-20T22:18:58.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:58.186+0000 7fa65f600700 1 Processor -- start 2024-04-20T22:18:58.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:58.186+0000 7fa65f600700 1 -- start start 2024-04-20T22:18:58.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:58.187+0000 7fa65f600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa6581026a0 0x7fa658198130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:18:58.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:58.187+0000 7fa65f600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa658103040 0x7fa658198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:18:58.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:58.187+0000 7fa65f600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa658198cc0 con 0x7fa6581026a0 2024-04-20T22:18:58.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:58.187+0000 7fa65f600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa658198e00 con 0x7fa658103040 2024-04-20T22:18:58.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:58.187+0000 7fa65cc00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa6581026a0 0x7fa658198130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:18:58.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:58.187+0000 7fa65cc00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa6581026a0 0x7fa658198130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35136/0 (socket says 172.21.15.62:35136) 2024-04-20T22:18:58.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:58.187+0000 7fa65cc00700 1 -- 172.21.15.62:0/215559152 learned_addr learned my addr 172.21.15.62:0/215559152 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:18:58.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:58.188+0000 7fa657e00700 1 --2- 172.21.15.62:0/215559152 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa658103040 0x7fa658198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:18:58.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:58.188+0000 7fa65cc00700 1 -- 172.21.15.62:0/215559152 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa658103040 msgr2=0x7fa658198670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:18:58.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:58.188+0000 7fa65cc00700 1 --2- 172.21.15.62:0/215559152 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa658103040 0x7fa658198670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:18:58.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:58.188+0000 7fa65cc00700 1 -- 172.21.15.62:0/215559152 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa64c0095f0 con 0x7fa6581026a0 2024-04-20T22:18:58.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:58.188+0000 7fa65cc00700 1 --2- 172.21.15.62:0/215559152 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa6581026a0 0x7fa658198130 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7fa64c006010 tx=0x7fa64c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:18:58.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:58.188+0000 7fa655600700 1 -- 172.21.15.62:0/215559152 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa64c01d070 con 0x7fa6581026a0 2024-04-20T22:18:58.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:58.189+0000 7fa65f600700 1 -- 172.21.15.62:0/215559152 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa65819cbf0 con 0x7fa6581026a0 2024-04-20T22:18:58.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:58.189+0000 7fa655600700 1 -- 172.21.15.62:0/215559152 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa64c004fb0 con 0x7fa6581026a0 2024-04-20T22:18:58.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:58.189+0000 7fa65f600700 1 -- 172.21.15.62:0/215559152 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa65819d0e0 con 0x7fa6581026a0 2024-04-20T22:18:58.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:58.189+0000 7fa655600700 1 -- 172.21.15.62:0/215559152 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa64c00f630 con 0x7fa6581026a0 2024-04-20T22:18:58.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:58.190+0000 7fa65f600700 1 -- 172.21.15.62:0/215559152 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa65810a9b0 con 0x7fa6581026a0 2024-04-20T22:18:58.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:58.191+0000 7fa655600700 1 -- 172.21.15.62:0/215559152 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fa64c0228a0 con 0x7fa6581026a0 2024-04-20T22:18:58.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:58.192+0000 7fa655600700 1 --2- 172.21.15.62:0/215559152 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa64006ecc0 0x7fa640071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:18:58.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:58.192+0000 7fa655600700 1 -- 172.21.15.62:0/215559152 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fa64c090510 con 0x7fa6581026a0 2024-04-20T22:18:58.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:58.192+0000 7fa657e00700 1 --2- 172.21.15.62:0/215559152 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa64006ecc0 0x7fa640071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:18:58.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:58.193+0000 7fa657e00700 1 --2- 172.21.15.62:0/215559152 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa64006ecc0 0x7fa640071170 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fa644005ea0 tx=0x7fa644005e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:18:58.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:58.197+0000 7fa655600700 1 -- 172.21.15.62:0/215559152 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fa64c0592d0 con 0x7fa6581026a0 2024-04-20T22:18:58.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:58.530+0000 7fa65f600700 1 -- 172.21.15.62:0/215559152 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fa65804f2a0 con 0x7fa6581026a0 2024-04-20T22:18:58.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:58.530+0000 7fa655600700 1 -- 172.21.15.62:0/215559152 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7fa64c05c990 con 0x7fa6581026a0 2024-04-20T22:18:58.531 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-20T22:18:58.531 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-20T22:18:58.531 INFO:teuthology.orchestra.run.smithi062.stdout: host smithi189 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi189/config 2024-04-20T22:18:58.531 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-20T22:18:58.531 INFO:teuthology.orchestra.run.smithi062.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi189 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp11670v2q:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:18:58.531 INFO:teuthology.orchestra.run.smithi062.stdout:Traceback (most recent call last): 2024-04-20T22:18:58.531 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-20T22:18:58.531 INFO:teuthology.orchestra.run.smithi062.stdout: return _run_code(code, main_globals, None, 2024-04-20T22:18:58.531 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-20T22:18:58.531 INFO:teuthology.orchestra.run.smithi062.stdout: exec(code, run_globals) 2024-04-20T22:18:58.531 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5618, in 2024-04-20T22:18:58.531 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5606, in main 2024-04-20T22:18:58.531 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 408, in _infer_config 2024-04-20T22:18:58.532 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 323, in _infer_fsid 2024-04-20T22:18:58.532 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 436, in _infer_image 2024-04-20T22:18:58.532 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 310, in _validate_fsid 2024-04-20T22:18:58.532 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 3359, in command_ceph_volume 2024-04-20T22:18:58.532 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-20T22:18:58.532 INFO:teuthology.orchestra.run.smithi062.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi189 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp11670v2q:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:18:58.532 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-20T22:18:58.532 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is degraded 2024-04-20T22:18:58.532 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T22:18:58.532 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T22:18:58.532 INFO:teuthology.orchestra.run.smithi062.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-20T22:18:58.532 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-20T22:18:58.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:58.533+0000 7fa65f600700 1 -- 172.21.15.62:0/215559152 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa64006ecc0 msgr2=0x7fa640071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:18:58.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:58.533+0000 7fa65f600700 1 --2- 172.21.15.62:0/215559152 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa64006ecc0 0x7fa640071170 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fa644005ea0 tx=0x7fa644005e30 comp rx=0 tx=0).stop 2024-04-20T22:18:58.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:58.533+0000 7fa65f600700 1 -- 172.21.15.62:0/215559152 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa6581026a0 msgr2=0x7fa658198130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:18:58.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:58.533+0000 7fa65f600700 1 --2- 172.21.15.62:0/215559152 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa6581026a0 0x7fa658198130 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7fa64c006010 tx=0x7fa64c00be20 comp rx=0 tx=0).stop 2024-04-20T22:18:58.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:58.533+0000 7fa65f600700 1 -- 172.21.15.62:0/215559152 shutdown_connections 2024-04-20T22:18:58.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:58.533+0000 7fa65f600700 1 --2- 172.21.15.62:0/215559152 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa64006ecc0 0x7fa640071170 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:18:58.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:58.533+0000 7fa65f600700 1 --2- 172.21.15.62:0/215559152 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa6581026a0 0x7fa658198130 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:18:58.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:58.533+0000 7fa65f600700 1 --2- 172.21.15.62:0/215559152 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa658103040 0x7fa658198670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:18:58.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:58.533+0000 7fa65f600700 1 -- 172.21.15.62:0/215559152 >> 172.21.15.62:0/215559152 conn(0x7fa6580fdf40 msgr2=0x7fa6580ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:18:58.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:58.533+0000 7fa65f600700 1 -- 172.21.15.62:0/215559152 shutdown_connections 2024-04-20T22:18:58.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:18:58.534+0000 7fa65f600700 1 -- 172.21.15.62:0/215559152 wait complete. 2024-04-20T22:18:58.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:18:58 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/2738754788' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:18:58.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:18:58 smithi062 ceph-mon[76322]: from='client.36320 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:18:58.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:18:58 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/2738754788' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:18:58.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:18:58 smithi189 ceph-mon[57022]: from='client.36320 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:18:59.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:18:59 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/215559152' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:18:59.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:18:59 smithi062 ceph-mon[76322]: pgmap v1314: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:18:59.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:18:59 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/215559152' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:18:59.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:18:59 smithi189 ceph-mon[57022]: pgmap v1314: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:19:02.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:19:02 smithi189 ceph-mon[57022]: pgmap v1315: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:19:02.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:19:02 smithi062 ceph-mon[76322]: pgmap v1315: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:19:04.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:19:04 smithi189 ceph-mon[57022]: pgmap v1316: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:19:04.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:19:04 smithi062 ceph-mon[76322]: pgmap v1316: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:19:05.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:19:05 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:19:05.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:19:05 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:19:06.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:19:06 smithi189 ceph-mon[57022]: pgmap v1317: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:19:06.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:19:06 smithi062 ceph-mon[76322]: pgmap v1317: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:19:07.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:19:07 smithi189 ceph-mon[57022]: pgmap v1318: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:19:07.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:19:07 smithi062 ceph-mon[76322]: pgmap v1318: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:19:10.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:19:10 smithi189 ceph-mon[57022]: pgmap v1319: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:19:10.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:19:10 smithi062 ceph-mon[76322]: pgmap v1319: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:19:12.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:19:12 smithi189 ceph-mon[57022]: pgmap v1320: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:19:12.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:19:12 smithi062 ceph-mon[76322]: pgmap v1320: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:19:13.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:19:13 smithi189 ceph-mon[57022]: pgmap v1321: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:19:13.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:19:13 smithi062 ceph-mon[76322]: pgmap v1321: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:19:16.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:19:15 smithi189 ceph-mon[57022]: pgmap v1322: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:19:16.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:19:15 smithi062 ceph-mon[76322]: pgmap v1322: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:19:18.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:19:18 smithi189 ceph-mon[57022]: pgmap v1323: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:19:18.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:19:18 smithi062 ceph-mon[76322]: pgmap v1323: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:19:20.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:19:20 smithi189 ceph-mon[57022]: pgmap v1324: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:19:20.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:19:20 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:19:20.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:19:20 smithi062 ceph-mon[76322]: pgmap v1324: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:19:20.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:19:20 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:19:21.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:19:21 smithi189 ceph-mon[57022]: pgmap v1325: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:19:21.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:19:21 smithi062 ceph-mon[76322]: pgmap v1325: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:19:24.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:19:24 smithi189 ceph-mon[57022]: pgmap v1326: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:19:24.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:19:24 smithi062 ceph-mon[76322]: pgmap v1326: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:19:26.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:19:26 smithi189 ceph-mon[57022]: pgmap v1327: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:19:26.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:19:26 smithi062 ceph-mon[76322]: pgmap v1327: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:19:27.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:19:27 smithi189 ceph-mon[57022]: pgmap v1328: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:19:27.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:19:27 smithi062 ceph-mon[76322]: pgmap v1328: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:19:28.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:28.711+0000 7f52c0600700 1 -- 172.21.15.62:0/2102235773 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f52b8106b80 msgr2=0x7f52b8106ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:19:28.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:28.711+0000 7f52c0600700 1 --2- 172.21.15.62:0/2102235773 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f52b8106b80 0x7f52b8106ff0 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f52a4009910 tx=0x7f52a4009c20 comp rx=0 tx=0).stop 2024-04-20T22:19:28.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:28.711+0000 7f52c0600700 1 -- 172.21.15.62:0/2102235773 shutdown_connections 2024-04-20T22:19:28.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:28.711+0000 7f52c0600700 1 --2- 172.21.15.62:0/2102235773 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f52b8106b80 0x7f52b8106ff0 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:19:28.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:28.711+0000 7f52c0600700 1 --2- 172.21.15.62:0/2102235773 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f52b810c8f0 0x7f52b810ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:19:28.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:28.711+0000 7f52c0600700 1 -- 172.21.15.62:0/2102235773 >> 172.21.15.62:0/2102235773 conn(0x7f52b8076060 msgr2=0x7f52b8078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:19:28.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:28.711+0000 7f52c0600700 1 -- 172.21.15.62:0/2102235773 shutdown_connections 2024-04-20T22:19:28.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:28.711+0000 7f52c0600700 1 -- 172.21.15.62:0/2102235773 wait complete. 2024-04-20T22:19:28.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:28.712+0000 7f52c0600700 1 Processor -- start 2024-04-20T22:19:28.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:28.712+0000 7f52c0600700 1 -- start start 2024-04-20T22:19:28.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:28.713+0000 7f52c0600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f52b8106b80 0x7f52b819c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:19:28.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:28.713+0000 7f52c0600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f52b810c8f0 0x7f52b819c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:19:28.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:28.713+0000 7f52c0600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52b819d060 con 0x7f52b810c8f0 2024-04-20T22:19:28.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:28.713+0000 7f52c0600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52b81a0df0 con 0x7f52b8106b80 2024-04-20T22:19:28.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:28.713+0000 7f52bd200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f52b810c8f0 0x7f52b819c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:19:28.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:28.713+0000 7f52bd200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f52b810c8f0 0x7f52b819c980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47890/0 (socket says 172.21.15.62:47890) 2024-04-20T22:19:28.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:28.713+0000 7f52bd200700 1 -- 172.21.15.62:0/115873496 learned_addr learned my addr 172.21.15.62:0/115873496 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:19:28.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:28.713+0000 7f52bdc00700 1 --2- 172.21.15.62:0/115873496 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f52b8106b80 0x7f52b819c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:19:28.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:28.713+0000 7f52bd200700 1 -- 172.21.15.62:0/115873496 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f52b8106b80 msgr2=0x7f52b819c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:19:28.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:28.713+0000 7f52bd200700 1 --2- 172.21.15.62:0/115873496 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f52b8106b80 0x7f52b819c440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:19:28.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:28.714+0000 7f52bd200700 1 -- 172.21.15.62:0/115873496 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52a40095f0 con 0x7f52b810c8f0 2024-04-20T22:19:28.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:28.714+0000 7f52bd200700 1 --2- 172.21.15.62:0/115873496 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f52b810c8f0 0x7f52b819c980 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f52a4006010 tx=0x7f52a400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:19:28.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:28.714+0000 7f52b6000700 1 -- 172.21.15.62:0/115873496 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f52a401d070 con 0x7f52b810c8f0 2024-04-20T22:19:28.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:28.714+0000 7f52b6000700 1 -- 172.21.15.62:0/115873496 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f52a400f460 con 0x7f52b810c8f0 2024-04-20T22:19:28.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:28.714+0000 7f52c0600700 1 -- 172.21.15.62:0/115873496 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52b81a1070 con 0x7f52b810c8f0 2024-04-20T22:19:28.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:28.714+0000 7f52b6000700 1 -- 172.21.15.62:0/115873496 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f52a4017630 con 0x7f52b810c8f0 2024-04-20T22:19:28.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:28.714+0000 7f52c0600700 1 -- 172.21.15.62:0/115873496 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52b81a1560 con 0x7f52b810c8f0 2024-04-20T22:19:28.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:28.716+0000 7f52c0600700 1 -- 172.21.15.62:0/115873496 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52b810ecc0 con 0x7f52b810c8f0 2024-04-20T22:19:28.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:28.717+0000 7f52b6000700 1 -- 172.21.15.62:0/115873496 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f52a400f5d0 con 0x7f52b810c8f0 2024-04-20T22:19:28.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:28.718+0000 7f52b6000700 1 --2- 172.21.15.62:0/115873496 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f52a006ecc0 0x7f52a0071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:19:28.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:28.718+0000 7f52b6000700 1 -- 172.21.15.62:0/115873496 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f52a4090290 con 0x7f52b810c8f0 2024-04-20T22:19:28.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:28.718+0000 7f52bdc00700 1 --2- 172.21.15.62:0/115873496 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f52a006ecc0 0x7f52a0071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:19:28.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:28.719+0000 7f52bdc00700 1 --2- 172.21.15.62:0/115873496 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f52a006ecc0 0x7f52a0071170 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f52ac005950 tx=0x7f52ac0058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:19:28.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:28.723+0000 7f52b6000700 1 -- 172.21.15.62:0/115873496 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f52a40590c0 con 0x7f52b810c8f0 2024-04-20T22:19:28.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:28.986+0000 7f52c0600700 1 -- 172.21.15.62:0/115873496 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f52b81a1780 con 0x7f52a006ecc0 2024-04-20T22:19:28.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:28.988+0000 7f52b6000700 1 -- 172.21.15.62:0/115873496 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f52b81a1780 con 0x7f52a006ecc0 2024-04-20T22:19:28.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:28.991+0000 7f52c0600700 1 -- 172.21.15.62:0/115873496 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f52a006ecc0 msgr2=0x7f52a0071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:19:28.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:28.991+0000 7f52c0600700 1 --2- 172.21.15.62:0/115873496 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f52a006ecc0 0x7f52a0071170 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f52ac005950 tx=0x7f52ac0058e0 comp rx=0 tx=0).stop 2024-04-20T22:19:28.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:28.991+0000 7f52c0600700 1 -- 172.21.15.62:0/115873496 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f52b810c8f0 msgr2=0x7f52b819c980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:19:28.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:28.991+0000 7f52c0600700 1 --2- 172.21.15.62:0/115873496 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f52b810c8f0 0x7f52b819c980 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f52a4006010 tx=0x7f52a400be20 comp rx=0 tx=0).stop 2024-04-20T22:19:28.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:28.991+0000 7f52c0600700 1 -- 172.21.15.62:0/115873496 shutdown_connections 2024-04-20T22:19:28.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:28.991+0000 7f52c0600700 1 --2- 172.21.15.62:0/115873496 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f52a006ecc0 0x7f52a0071170 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:19:28.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:28.991+0000 7f52c0600700 1 --2- 172.21.15.62:0/115873496 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f52b8106b80 0x7f52b819c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:19:28.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:28.991+0000 7f52c0600700 1 --2- 172.21.15.62:0/115873496 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f52b810c8f0 0x7f52b819c980 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:19:28.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:28.991+0000 7f52c0600700 1 -- 172.21.15.62:0/115873496 >> 172.21.15.62:0/115873496 conn(0x7f52b8076060 msgr2=0x7f52b8077ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:19:28.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:28.992+0000 7f52c0600700 1 -- 172.21.15.62:0/115873496 shutdown_connections 2024-04-20T22:19:28.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:28.992+0000 7f52c0600700 1 -- 172.21.15.62:0/115873496 wait complete. 2024-04-20T22:19:29.008 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T22:19:29.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:29.159+0000 7f24a4400700 1 -- 172.21.15.62:0/2016174917 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f249c10c8f0 msgr2=0x7f249c10ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:19:29.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:29.159+0000 7f24a4400700 1 --2- 172.21.15.62:0/2016174917 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f249c10c8f0 0x7f249c10ccc0 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f2490009910 tx=0x7f2490009c20 comp rx=0 tx=0).stop 2024-04-20T22:19:29.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:29.159+0000 7f24a4400700 1 -- 172.21.15.62:0/2016174917 shutdown_connections 2024-04-20T22:19:29.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:29.159+0000 7f24a4400700 1 --2- 172.21.15.62:0/2016174917 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f249c106b80 0x7f249c106ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:19:29.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:29.159+0000 7f24a4400700 1 --2- 172.21.15.62:0/2016174917 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f249c10c8f0 0x7f249c10ccc0 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:19:29.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:29.159+0000 7f24a4400700 1 -- 172.21.15.62:0/2016174917 >> 172.21.15.62:0/2016174917 conn(0x7f249c076060 msgr2=0x7f249c078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:19:29.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:29.159+0000 7f24a4400700 1 -- 172.21.15.62:0/2016174917 shutdown_connections 2024-04-20T22:19:29.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:29.159+0000 7f24a4400700 1 -- 172.21.15.62:0/2016174917 wait complete. 2024-04-20T22:19:29.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:29.160+0000 7f24a4400700 1 Processor -- start 2024-04-20T22:19:29.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:29.161+0000 7f24a4400700 1 -- start start 2024-04-20T22:19:29.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:29.161+0000 7f24a4400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f249c106b80 0x7f249c19c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:19:29.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:29.161+0000 7f24a4400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f249c10c8f0 0x7f249c19ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:19:29.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:29.161+0000 7f24a4400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f249c19d100 con 0x7f249c10c8f0 2024-04-20T22:19:29.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:29.161+0000 7f24a4400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f249c1a0e90 con 0x7f249c106b80 2024-04-20T22:19:29.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:29.161+0000 7f24a1000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f249c10c8f0 0x7f249c19ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:19:29.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:29.162+0000 7f24a1000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f249c10c8f0 0x7f249c19ca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47902/0 (socket says 172.21.15.62:47902) 2024-04-20T22:19:29.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:29.162+0000 7f24a1000700 1 -- 172.21.15.62:0/2588310842 learned_addr learned my addr 172.21.15.62:0/2588310842 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:19:29.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:29.162+0000 7f24a1a00700 1 --2- 172.21.15.62:0/2588310842 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f249c106b80 0x7f249c19c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:19:29.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:29.162+0000 7f24a1000700 1 -- 172.21.15.62:0/2588310842 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f249c106b80 msgr2=0x7f249c19c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:19:29.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:29.162+0000 7f24a1000700 1 --2- 172.21.15.62:0/2588310842 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f249c106b80 0x7f249c19c4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:19:29.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:29.162+0000 7f24a1000700 1 -- 172.21.15.62:0/2588310842 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24900095f0 con 0x7f249c10c8f0 2024-04-20T22:19:29.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:29.162+0000 7f24a1000700 1 --2- 172.21.15.62:0/2588310842 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f249c10c8f0 0x7f249c19ca20 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f248800b4b0 tx=0x7f248800b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:19:29.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:29.163+0000 7f249a000700 1 -- 172.21.15.62:0/2588310842 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2488011800 con 0x7f249c10c8f0 2024-04-20T22:19:29.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:29.163+0000 7f249a000700 1 -- 172.21.15.62:0/2588310842 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2488011e20 con 0x7f249c10c8f0 2024-04-20T22:19:29.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:29.163+0000 7f24a4400700 1 -- 172.21.15.62:0/2588310842 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f249c1a1170 con 0x7f249c10c8f0 2024-04-20T22:19:29.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:29.163+0000 7f249a000700 1 -- 172.21.15.62:0/2588310842 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f248800f2e0 con 0x7f249c10c8f0 2024-04-20T22:19:29.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:29.163+0000 7f24a4400700 1 -- 172.21.15.62:0/2588310842 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f249c1a16c0 con 0x7f249c10c8f0 2024-04-20T22:19:29.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:29.164+0000 7f24a4400700 1 -- 172.21.15.62:0/2588310842 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f249c109b20 con 0x7f249c10c8f0 2024-04-20T22:19:29.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:29.165+0000 7f249a000700 1 -- 172.21.15.62:0/2588310842 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f2488010380 con 0x7f249c10c8f0 2024-04-20T22:19:29.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:29.166+0000 7f249a000700 1 --2- 172.21.15.62:0/2588310842 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f248406ecc0 0x7f2484071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:19:29.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:29.166+0000 7f249a000700 1 -- 172.21.15.62:0/2588310842 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f248808ea30 con 0x7f249c10c8f0 2024-04-20T22:19:29.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:29.166+0000 7f24a1a00700 1 --2- 172.21.15.62:0/2588310842 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f248406ecc0 0x7f2484071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:19:29.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:29.167+0000 7f24a1a00700 1 --2- 172.21.15.62:0/2588310842 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f248406ecc0 0x7f2484071170 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f2490006010 tx=0x7f249001a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:19:29.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:29.172+0000 7f249a000700 1 -- 172.21.15.62:0/2588310842 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f2488057870 con 0x7f249c10c8f0 2024-04-20T22:19:29.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:29.429+0000 7f24a4400700 1 -- 172.21.15.62:0/2588310842 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f249c1a1990 con 0x7f248406ecc0 2024-04-20T22:19:29.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:29.432+0000 7f249a000700 1 -- 172.21.15.62:0/2588310842 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f249c1a1990 con 0x7f248406ecc0 2024-04-20T22:19:29.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:29.434+0000 7f24a4400700 1 -- 172.21.15.62:0/2588310842 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f248406ecc0 msgr2=0x7f2484071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:19:29.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:29.434+0000 7f24a4400700 1 --2- 172.21.15.62:0/2588310842 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f248406ecc0 0x7f2484071170 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f2490006010 tx=0x7f249001a040 comp rx=0 tx=0).stop 2024-04-20T22:19:29.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:29.434+0000 7f24a4400700 1 -- 172.21.15.62:0/2588310842 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f249c10c8f0 msgr2=0x7f249c19ca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:19:29.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:29.434+0000 7f24a4400700 1 --2- 172.21.15.62:0/2588310842 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f249c10c8f0 0x7f249c19ca20 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f248800b4b0 tx=0x7f248800b870 comp rx=0 tx=0).stop 2024-04-20T22:19:29.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:29.434+0000 7f24a4400700 1 -- 172.21.15.62:0/2588310842 shutdown_connections 2024-04-20T22:19:29.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:29.434+0000 7f24a4400700 1 --2- 172.21.15.62:0/2588310842 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f248406ecc0 0x7f2484071170 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:19:29.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:29.434+0000 7f24a4400700 1 --2- 172.21.15.62:0/2588310842 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f249c106b80 0x7f249c19c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:19:29.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:29.434+0000 7f24a4400700 1 --2- 172.21.15.62:0/2588310842 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f249c10c8f0 0x7f249c19ca20 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:19:29.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:29.434+0000 7f24a4400700 1 -- 172.21.15.62:0/2588310842 >> 172.21.15.62:0/2588310842 conn(0x7f249c076060 msgr2=0x7f249c077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:19:29.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:29.435+0000 7f24a4400700 1 -- 172.21.15.62:0/2588310842 shutdown_connections 2024-04-20T22:19:29.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:29.435+0000 7f24a4400700 1 -- 172.21.15.62:0/2588310842 wait complete. 2024-04-20T22:19:29.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:29.593+0000 7f6a69000700 1 -- 172.21.15.62:0/1438822987 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6a64073a00 msgr2=0x7f6a6410c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:19:29.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:29.593+0000 7f6a69000700 1 --2- 172.21.15.62:0/1438822987 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6a64073a00 0x7f6a6410c330 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f6a50009910 tx=0x7f6a50009c20 comp rx=0 tx=0).stop 2024-04-20T22:19:29.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:29.593+0000 7f6a69000700 1 -- 172.21.15.62:0/1438822987 shutdown_connections 2024-04-20T22:19:29.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:29.593+0000 7f6a69000700 1 --2- 172.21.15.62:0/1438822987 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6a64073a00 0x7f6a6410c330 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:19:29.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:29.593+0000 7f6a69000700 1 --2- 172.21.15.62:0/1438822987 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a640730f0 0x7f6a640734c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:19:29.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:29.593+0000 7f6a69000700 1 -- 172.21.15.62:0/1438822987 >> 172.21.15.62:0/1438822987 conn(0x7f6a640fbb10 msgr2=0x7f6a640fdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:19:29.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:29.594+0000 7f6a69000700 1 -- 172.21.15.62:0/1438822987 shutdown_connections 2024-04-20T22:19:29.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:29.594+0000 7f6a69000700 1 -- 172.21.15.62:0/1438822987 wait complete. 2024-04-20T22:19:29.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:29.595+0000 7f6a69000700 1 Processor -- start 2024-04-20T22:19:29.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:29.595+0000 7f6a69000700 1 -- start start 2024-04-20T22:19:29.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:29.595+0000 7f6a69000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a640730f0 0x7f6a6419dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:19:29.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:29.595+0000 7f6a69000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6a64073a00 0x7f6a6419e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:19:29.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:29.595+0000 7f6a69000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a6419e690 con 0x7f6a64073a00 2024-04-20T22:19:29.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:29.595+0000 7f6a69000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a64197b40 con 0x7f6a640730f0 2024-04-20T22:19:29.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:29.596+0000 7f6a62a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6a64073a00 0x7f6a6419e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:19:29.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:29.596+0000 7f6a62a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6a64073a00 0x7f6a6419e000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47924/0 (socket says 172.21.15.62:47924) 2024-04-20T22:19:29.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:29.596+0000 7f6a62a00700 1 -- 172.21.15.62:0/4118029486 learned_addr learned my addr 172.21.15.62:0/4118029486 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:19:29.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:29.596+0000 7f6a63400700 1 --2- 172.21.15.62:0/4118029486 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a640730f0 0x7f6a6419dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:19:29.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:29.596+0000 7f6a62a00700 1 -- 172.21.15.62:0/4118029486 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a640730f0 msgr2=0x7f6a6419dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:19:29.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:29.596+0000 7f6a62a00700 1 --2- 172.21.15.62:0/4118029486 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a640730f0 0x7f6a6419dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:19:29.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:29.596+0000 7f6a62a00700 1 -- 172.21.15.62:0/4118029486 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a500095f0 con 0x7f6a64073a00 2024-04-20T22:19:29.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:29.596+0000 7f6a62a00700 1 --2- 172.21.15.62:0/4118029486 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6a64073a00 0x7f6a6419e000 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f6a50006010 tx=0x7f6a5000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:19:29.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:29.597+0000 7f6a5be00700 1 -- 172.21.15.62:0/4118029486 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6a5001d070 con 0x7f6a64073a00 2024-04-20T22:19:29.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:29.597+0000 7f6a5be00700 1 -- 172.21.15.62:0/4118029486 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6a50004fb0 con 0x7f6a64073a00 2024-04-20T22:19:29.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:29.597+0000 7f6a5be00700 1 -- 172.21.15.62:0/4118029486 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6a50022ea0 con 0x7f6a64073a00 2024-04-20T22:19:29.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:29.597+0000 7f6a69000700 1 -- 172.21.15.62:0/4118029486 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a64197e20 con 0x7f6a64073a00 2024-04-20T22:19:29.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:29.597+0000 7f6a69000700 1 -- 172.21.15.62:0/4118029486 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a64198370 con 0x7f6a64073a00 2024-04-20T22:19:29.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:29.598+0000 7f6a69000700 1 -- 172.21.15.62:0/4118029486 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a64109a30 con 0x7f6a64073a00 2024-04-20T22:19:29.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:29.599+0000 7f6a5be00700 1 -- 172.21.15.62:0/4118029486 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f6a50022470 con 0x7f6a64073a00 2024-04-20T22:19:29.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:29.600+0000 7f6a5be00700 1 --2- 172.21.15.62:0/4118029486 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6a4806ecc0 0x7f6a48071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:19:29.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:29.600+0000 7f6a5be00700 1 -- 172.21.15.62:0/4118029486 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f6a50090440 con 0x7f6a64073a00 2024-04-20T22:19:29.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:29.600+0000 7f6a63400700 1 --2- 172.21.15.62:0/4118029486 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6a4806ecc0 0x7f6a48071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:19:29.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:29.601+0000 7f6a63400700 1 --2- 172.21.15.62:0/4118029486 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6a4806ecc0 0x7f6a48071170 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f6a54005ea0 tx=0x7f6a54005e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:19:29.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:29.605+0000 7f6a5be00700 1 -- 172.21.15.62:0/4118029486 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f6a5005cd30 con 0x7f6a64073a00 2024-04-20T22:19:29.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:29.850+0000 7f6a69000700 1 -- 172.21.15.62:0/4118029486 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f6a64198650 con 0x7f6a4806ecc0 2024-04-20T22:19:29.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:29.883+0000 7f6a5be00700 1 -- 172.21.15.62:0/4118029486 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f6a64198650 con 0x7f6a4806ecc0 2024-04-20T22:19:29.884 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T22:19:29.884 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (45m) 6m ago 59m 14.6M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T22:19:29.884 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (59m) 6m ago 59m 11.6M - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T22:19:29.884 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (58m) 6m ago 58m 11.2M - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T22:19:29.884 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (41m) 6m ago 59m 6660k - 19.0.0-3133-ga8c84f59 76bc6b91b726 2825d5ba11c6 2024-04-20T22:19:29.884 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (41m) 6m ago 58m 6665k - 19.0.0-3133-ga8c84f59 76bc6b91b726 20f746fa1bba 2024-04-20T22:19:29.884 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (44m) 6m ago 58m 75.5M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T22:19:29.885 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (36m) 6m ago 51m 22.9M - 19.0.0-3133-ga8c84f59 76bc6b91b726 de80bc90a683 2024-04-20T22:19:29.885 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (37m) 6m ago 52m 23.0M - 19.0.0-3133-ga8c84f59 76bc6b91b726 cd793b7ee98e 2024-04-20T22:19:29.885 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (21m) 6m ago 52m 18.5M - 19.0.0-3133-ga8c84f59 76bc6b91b726 0e7b0bc4d1ab 2024-04-20T22:19:29.885 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (6m) 6m ago 51m 11.4M - 19.0.0-3133-ga8c84f59 76bc6b91b726 0af732ac9e06 2024-04-20T22:19:29.885 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (48m) 6m ago 61m 517M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T22:19:29.885 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (46m) 6m ago 57m 460M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T22:19:29.885 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (43m) 6m ago 61m 86.3M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 019eb1479485 2024-04-20T22:19:29.885 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (42m) 6m ago 57m 58.0M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 55aac560aab6 2024-04-20T22:19:29.885 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (46m) 6m ago 59m 8766k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T22:19:29.885 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (46m) 6m ago 57m 8711k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T22:19:29.885 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (40m) 6m ago 56m 91.3M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 ee2bb901f26e 2024-04-20T22:19:29.885 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (40m) 6m ago 56m 94.9M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 9075152d4162 2024-04-20T22:19:29.885 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (39m) 6m ago 55m 78.5M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 2d4f3e38dc0e 2024-04-20T22:19:29.886 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (39m) 6m ago 55m 94.1M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 8f97b17ae0eb 2024-04-20T22:19:29.886 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (38m) 6m ago 54m 96.1M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 653d8079967d 2024-04-20T22:19:29.886 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (37m) 6m ago 53m 90.9M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 596dded14c12 2024-04-20T22:19:29.886 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (45m) 6m ago 58m 90.4M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T22:19:29.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:29.887+0000 7f6a69000700 1 -- 172.21.15.62:0/4118029486 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6a4806ecc0 msgr2=0x7f6a48071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:19:29.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:29.887+0000 7f6a69000700 1 --2- 172.21.15.62:0/4118029486 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6a4806ecc0 0x7f6a48071170 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f6a54005ea0 tx=0x7f6a54005e30 comp rx=0 tx=0).stop 2024-04-20T22:19:29.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:29.887+0000 7f6a69000700 1 -- 172.21.15.62:0/4118029486 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6a64073a00 msgr2=0x7f6a6419e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:19:29.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:29.887+0000 7f6a69000700 1 --2- 172.21.15.62:0/4118029486 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6a64073a00 0x7f6a6419e000 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f6a50006010 tx=0x7f6a5000be20 comp rx=0 tx=0).stop 2024-04-20T22:19:29.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:29.887+0000 7f6a69000700 1 -- 172.21.15.62:0/4118029486 shutdown_connections 2024-04-20T22:19:29.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:29.887+0000 7f6a69000700 1 --2- 172.21.15.62:0/4118029486 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6a4806ecc0 0x7f6a48071170 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:19:29.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:29.887+0000 7f6a69000700 1 --2- 172.21.15.62:0/4118029486 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a640730f0 0x7f6a6419dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:19:29.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:29.887+0000 7f6a69000700 1 --2- 172.21.15.62:0/4118029486 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6a64073a00 0x7f6a6419e000 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:19:29.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:29.887+0000 7f6a69000700 1 -- 172.21.15.62:0/4118029486 >> 172.21.15.62:0/4118029486 conn(0x7f6a640fbb10 msgr2=0x7f6a64106b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:19:29.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:29.887+0000 7f6a69000700 1 -- 172.21.15.62:0/4118029486 shutdown_connections 2024-04-20T22:19:29.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:29.888+0000 7f6a69000700 1 -- 172.21.15.62:0/4118029486 wait complete. 2024-04-20T22:19:30.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:30.051+0000 7f8a53200700 1 -- 172.21.15.62:0/1571052544 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8a4c073a00 msgr2=0x7f8a4c10c580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:19:30.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:30.051+0000 7f8a53200700 1 --2- 172.21.15.62:0/1571052544 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8a4c073a00 0x7f8a4c10c580 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f8a38009910 tx=0x7f8a38009c20 comp rx=0 tx=0).stop 2024-04-20T22:19:30.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:30.051+0000 7f8a53200700 1 -- 172.21.15.62:0/1571052544 shutdown_connections 2024-04-20T22:19:30.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:30.051+0000 7f8a53200700 1 --2- 172.21.15.62:0/1571052544 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8a4c073a00 0x7f8a4c10c580 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:19:30.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:30.051+0000 7f8a53200700 1 --2- 172.21.15.62:0/1571052544 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8a4c0730f0 0x7f8a4c0734c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:19:30.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:30.051+0000 7f8a53200700 1 -- 172.21.15.62:0/1571052544 >> 172.21.15.62:0/1571052544 conn(0x7f8a4c0fbda0 msgr2=0x7f8a4c0fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:19:30.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:30.051+0000 7f8a53200700 1 -- 172.21.15.62:0/1571052544 shutdown_connections 2024-04-20T22:19:30.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:30.051+0000 7f8a53200700 1 -- 172.21.15.62:0/1571052544 wait complete. 2024-04-20T22:19:30.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:30.052+0000 7f8a53200700 1 Processor -- start 2024-04-20T22:19:30.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:30.052+0000 7f8a53200700 1 -- start start 2024-04-20T22:19:30.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:30.052+0000 7f8a53200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8a4c0730f0 0x7f8a4c197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:19:30.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:30.053+0000 7f8a53200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8a4c073a00 0x7f8a4c198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:19:30.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:30.053+0000 7f8a53200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a4c198be0 con 0x7f8a4c073a00 2024-04-20T22:19:30.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:30.053+0000 7f8a53200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a4c19c970 con 0x7f8a4c0730f0 2024-04-20T22:19:30.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:30.053+0000 7f8a4b400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8a4c073a00 0x7f8a4c198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:19:30.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:30.053+0000 7f8a4b400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8a4c073a00 0x7f8a4c198500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47946/0 (socket says 172.21.15.62:47946) 2024-04-20T22:19:30.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:30.053+0000 7f8a4b400700 1 -- 172.21.15.62:0/3065813476 learned_addr learned my addr 172.21.15.62:0/3065813476 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:19:30.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:30.053+0000 7f8a4be00700 1 --2- 172.21.15.62:0/3065813476 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8a4c0730f0 0x7f8a4c197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:19:30.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:30.053+0000 7f8a4b400700 1 -- 172.21.15.62:0/3065813476 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8a4c0730f0 msgr2=0x7f8a4c197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:19:30.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:30.053+0000 7f8a4b400700 1 --2- 172.21.15.62:0/3065813476 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8a4c0730f0 0x7f8a4c197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:19:30.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:30.053+0000 7f8a4b400700 1 -- 172.21.15.62:0/3065813476 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a380095f0 con 0x7f8a4c073a00 2024-04-20T22:19:30.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:30.054+0000 7f8a4b400700 1 --2- 172.21.15.62:0/3065813476 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8a4c073a00 0x7f8a4c198500 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f8a3800b5c0 tx=0x7f8a3800be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:19:30.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:30.054+0000 7f8a48c00700 1 -- 172.21.15.62:0/3065813476 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8a3801d070 con 0x7f8a4c073a00 2024-04-20T22:19:30.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:30.054+0000 7f8a53200700 1 -- 172.21.15.62:0/3065813476 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a4c19cc50 con 0x7f8a4c073a00 2024-04-20T22:19:30.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:30.054+0000 7f8a48c00700 1 -- 172.21.15.62:0/3065813476 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8a3800f460 con 0x7f8a4c073a00 2024-04-20T22:19:30.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:30.054+0000 7f8a48c00700 1 -- 172.21.15.62:0/3065813476 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8a38017630 con 0x7f8a4c073a00 2024-04-20T22:19:30.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:30.054+0000 7f8a53200700 1 -- 172.21.15.62:0/3065813476 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a4c19d1a0 con 0x7f8a4c073a00 2024-04-20T22:19:30.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:30.056+0000 7f8a53200700 1 -- 172.21.15.62:0/3065813476 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a4c109c80 con 0x7f8a4c073a00 2024-04-20T22:19:30.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:30.057+0000 7f8a48c00700 1 -- 172.21.15.62:0/3065813476 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f8a3800f5d0 con 0x7f8a4c073a00 2024-04-20T22:19:30.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:30.058+0000 7f8a48c00700 1 --2- 172.21.15.62:0/3065813476 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f8a3006ecc0 0x7f8a30071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:19:30.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:30.058+0000 7f8a48c00700 1 -- 172.21.15.62:0/3065813476 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f8a38090490 con 0x7f8a4c073a00 2024-04-20T22:19:30.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:30.058+0000 7f8a4be00700 1 --2- 172.21.15.62:0/3065813476 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f8a3006ecc0 0x7f8a30071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:19:30.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:30.059+0000 7f8a4be00700 1 --2- 172.21.15.62:0/3065813476 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f8a3006ecc0 0x7f8a30071170 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c005ea0 tx=0x7f8a3c005e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:19:30.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:30.063+0000 7f8a48c00700 1 -- 172.21.15.62:0/3065813476 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f8a380591a0 con 0x7f8a4c073a00 2024-04-20T22:19:30.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:30.396+0000 7f8a53200700 1 -- 172.21.15.62:0/3065813476 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f8a4c1993f0 con 0x7f8a4c073a00 2024-04-20T22:19:30.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:30.397+0000 7f8a48c00700 1 -- 172.21.15.62:0/3065813476 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f8a3805c860 con 0x7f8a4c073a00 2024-04-20T22:19:30.397 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:19:30.397 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T22:19:30.397 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:19:30.397 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:19:30.398 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T22:19:30.398 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:19:30.398 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:19:30.398 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T22:19:30.398 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 6 2024-04-20T22:19:30.398 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:19:30.398 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T22:19:30.398 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 4 2024-04-20T22:19:30.398 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:19:30.398 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T22:19:30.398 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 14 2024-04-20T22:19:30.398 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T22:19:30.398 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:19:30.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:30.400+0000 7f8a53200700 1 -- 172.21.15.62:0/3065813476 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f8a3006ecc0 msgr2=0x7f8a30071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:19:30.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:30.400+0000 7f8a53200700 1 --2- 172.21.15.62:0/3065813476 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f8a3006ecc0 0x7f8a30071170 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c005ea0 tx=0x7f8a3c005e30 comp rx=0 tx=0).stop 2024-04-20T22:19:30.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:30.400+0000 7f8a53200700 1 -- 172.21.15.62:0/3065813476 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8a4c073a00 msgr2=0x7f8a4c198500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:19:30.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:30.400+0000 7f8a53200700 1 --2- 172.21.15.62:0/3065813476 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8a4c073a00 0x7f8a4c198500 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f8a3800b5c0 tx=0x7f8a3800be00 comp rx=0 tx=0).stop 2024-04-20T22:19:30.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:30.400+0000 7f8a53200700 1 -- 172.21.15.62:0/3065813476 shutdown_connections 2024-04-20T22:19:30.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:30.400+0000 7f8a53200700 1 --2- 172.21.15.62:0/3065813476 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f8a3006ecc0 0x7f8a30071170 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:19:30.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:30.400+0000 7f8a53200700 1 --2- 172.21.15.62:0/3065813476 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8a4c0730f0 0x7f8a4c197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:19:30.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:30.400+0000 7f8a53200700 1 --2- 172.21.15.62:0/3065813476 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8a4c073a00 0x7f8a4c198500 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:19:30.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:30.400+0000 7f8a53200700 1 -- 172.21.15.62:0/3065813476 >> 172.21.15.62:0/3065813476 conn(0x7f8a4c0fbda0 msgr2=0x7f8a4c106dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:19:30.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:30.401+0000 7f8a53200700 1 -- 172.21.15.62:0/3065813476 shutdown_connections 2024-04-20T22:19:30.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:30.401+0000 7f8a53200700 1 -- 172.21.15.62:0/3065813476 wait complete. 2024-04-20T22:19:30.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:30.556+0000 7f014a800700 1 -- 172.21.15.62:0/3385349536 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f01441011f0 msgr2=0x7f01441015c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:19:30.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:30.556+0000 7f014a800700 1 --2- 172.21.15.62:0/3385349536 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f01441011f0 0x7f01441015c0 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f0138009910 tx=0x7f0138009c20 comp rx=0 tx=0).stop 2024-04-20T22:19:30.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:30.556+0000 7f014a800700 1 -- 172.21.15.62:0/3385349536 shutdown_connections 2024-04-20T22:19:30.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:30.556+0000 7f014a800700 1 --2- 172.21.15.62:0/3385349536 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0144101b00 0x7f014410a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:19:30.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:30.556+0000 7f014a800700 1 --2- 172.21.15.62:0/3385349536 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f01441011f0 0x7f01441015c0 unknown :-1 s=CLOSED pgs=1135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:19:30.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:30.556+0000 7f014a800700 1 -- 172.21.15.62:0/3385349536 >> 172.21.15.62:0/3385349536 conn(0x7f01440faa10 msgr2=0x7f01440fce20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:19:30.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:30.556+0000 7f014a800700 1 -- 172.21.15.62:0/3385349536 shutdown_connections 2024-04-20T22:19:30.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:30.556+0000 7f014a800700 1 -- 172.21.15.62:0/3385349536 wait complete. 2024-04-20T22:19:30.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:30.557+0000 7f014a800700 1 Processor -- start 2024-04-20T22:19:30.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:30.558+0000 7f014a800700 1 -- start start 2024-04-20T22:19:30.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:30.558+0000 7f014a800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f01441011f0 0x7f014419b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:19:30.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:30.558+0000 7f014a800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0144101b00 0x7f014419beb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:19:30.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:30.558+0000 7f014a800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f014419c4d0 con 0x7f01441011f0 2024-04-20T22:19:30.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:30.558+0000 7f014a800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0144195a60 con 0x7f0144101b00 2024-04-20T22:19:30.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:30.558+0000 7f0149400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f01441011f0 0x7f014419b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:19:30.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:30.558+0000 7f0149400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f01441011f0 0x7f014419b970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47960/0 (socket says 172.21.15.62:47960) 2024-04-20T22:19:30.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:30.558+0000 7f0149400700 1 -- 172.21.15.62:0/3490241342 learned_addr learned my addr 172.21.15.62:0/3490241342 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:19:30.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:30.559+0000 7f0149400700 1 -- 172.21.15.62:0/3490241342 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0144101b00 msgr2=0x7f014419beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:19:30.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:30.559+0000 7f0149400700 1 --2- 172.21.15.62:0/3490241342 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0144101b00 0x7f014419beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:19:30.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:30.559+0000 7f0149400700 1 -- 172.21.15.62:0/3490241342 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01380095f0 con 0x7f01441011f0 2024-04-20T22:19:30.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:30.559+0000 7f0149400700 1 --2- 172.21.15.62:0/3490241342 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f01441011f0 0x7f014419b970 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f0138005f50 tx=0x7f013800f740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:19:30.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:30.559+0000 7f0142000700 1 -- 172.21.15.62:0/3490241342 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f013801d070 con 0x7f01441011f0 2024-04-20T22:19:30.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:30.559+0000 7f0142000700 1 -- 172.21.15.62:0/3490241342 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f013800b610 con 0x7f01441011f0 2024-04-20T22:19:30.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:30.559+0000 7f014a800700 1 -- 172.21.15.62:0/3490241342 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0144195ce0 con 0x7f01441011f0 2024-04-20T22:19:30.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:30.559+0000 7f0142000700 1 -- 172.21.15.62:0/3490241342 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0138022470 con 0x7f01441011f0 2024-04-20T22:19:30.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:30.559+0000 7f014a800700 1 -- 172.21.15.62:0/3490241342 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01441961d0 con 0x7f01441011f0 2024-04-20T22:19:30.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:30.561+0000 7f014a800700 1 -- 172.21.15.62:0/3490241342 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01441076a0 con 0x7f01441011f0 2024-04-20T22:19:30.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:30.562+0000 7f0142000700 1 -- 172.21.15.62:0/3490241342 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f013800fca0 con 0x7f01441011f0 2024-04-20T22:19:30.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:30.563+0000 7f0142000700 1 --2- 172.21.15.62:0/3490241342 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f012c06ecc0 0x7f012c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:19:30.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:30.563+0000 7f0142000700 1 -- 172.21.15.62:0/3490241342 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f0138090370 con 0x7f01441011f0 2024-04-20T22:19:30.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:30.563+0000 7f0148a00700 1 --2- 172.21.15.62:0/3490241342 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f012c06ecc0 0x7f012c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:19:30.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:30.564+0000 7f0148a00700 1 --2- 172.21.15.62:0/3490241342 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f012c06ecc0 0x7f012c071170 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f0130005950 tx=0x7f01300058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:19:30.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:30.568+0000 7f0142000700 1 -- 172.21.15.62:0/3490241342 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f013805cbb0 con 0x7f01441011f0 2024-04-20T22:19:30.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:19:30 smithi189 ceph-mon[57022]: from='client.36328 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:19:30.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:19:30 smithi189 ceph-mon[57022]: pgmap v1329: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:19:30.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:19:30 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/3065813476' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:19:30.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:19:30 smithi062 ceph-mon[76322]: from='client.36328 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:19:30.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:19:30 smithi062 ceph-mon[76322]: pgmap v1329: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:19:30.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:19:30 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/3065813476' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:19:30.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:30.852+0000 7f014a800700 1 -- 172.21.15.62:0/3490241342 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f014402cc30 con 0x7f01441011f0 2024-04-20T22:19:30.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:30.853+0000 7f0142000700 1 -- 172.21.15.62:0/3490241342 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 29 v29) v1 ==== 76+0+1887 (secure 0 0 0) 0x7f013805c740 con 0x7f01441011f0 2024-04-20T22:19:30.854 INFO:teuthology.orchestra.run.smithi062.stdout:e29 2024-04-20T22:19:30.854 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T22:19:30.854 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:19:30.854 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T22:19:30.854 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:19:30.854 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T22:19:30.854 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T22:19:30.855 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 20 2024-04-20T22:19:30.855 INFO:teuthology.orchestra.run.smithi062.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-20T22:19:30.855 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T22:19:30.855 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T21:42:01.559061+0000 2024-04-20T22:19:30.855 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T22:19:30.855 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T22:19:30.855 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T22:19:30.855 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T22:19:30.855 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T22:19:30.855 INFO:teuthology.orchestra.run.smithi062.stdout:max_xattr_size 65536 2024-04-20T22:19:30.855 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T22:19:30.855 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T22:19:30.855 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 81 2024-04-20T22:19:30.855 INFO:teuthology.orchestra.run.smithi062.stdout:snap_rstat 1 2024-04-20T22:19:30.855 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:19:30.856 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 2 2024-04-20T22:19:30.856 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1 2024-04-20T22:19:30.856 INFO:teuthology.orchestra.run.smithi062.stdout:up {} 2024-04-20T22:19:30.856 INFO:teuthology.orchestra.run.smithi062.stdout:failed 0,1 2024-04-20T22:19:30.856 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T22:19:30.856 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T22:19:30.856 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T22:19:30.856 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T22:19:30.856 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data enabled 2024-04-20T22:19:30.856 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T22:19:30.856 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T22:19:30.856 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 1 2024-04-20T22:19:30.856 INFO:teuthology.orchestra.run.smithi062.stdout:qdb_cluster leader: 0 members: 2024-04-20T22:19:30.856 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:19:30.856 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:19:30.857 INFO:teuthology.orchestra.run.smithi062.stdout:Standby daemons: 2024-04-20T22:19:30.857 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:19:30.857 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ryqblb{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6826/4113916205,v1:172.21.15.62:6827/4113916205] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:19:30.857 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ozizwi{-1:34476} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6828/3801975165,v1:172.21.15.62:6829/3801975165] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:19:30.857 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.dvimer{-1:44727} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6824/3576699232,v1:172.21.15.189:6825/3576699232] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:19:30.857 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.xbgspw{-1:45075} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6826/212905792,v1:172.21.15.189:6827/212905792] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:19:30.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:30.856+0000 7f014a800700 1 -- 172.21.15.62:0/3490241342 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f012c06ecc0 msgr2=0x7f012c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:19:30.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:30.856+0000 7f014a800700 1 --2- 172.21.15.62:0/3490241342 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f012c06ecc0 0x7f012c071170 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f0130005950 tx=0x7f01300058e0 comp rx=0 tx=0).stop 2024-04-20T22:19:30.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:30.856+0000 7f014a800700 1 -- 172.21.15.62:0/3490241342 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f01441011f0 msgr2=0x7f014419b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:19:30.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:30.856+0000 7f014a800700 1 --2- 172.21.15.62:0/3490241342 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f01441011f0 0x7f014419b970 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f0138005f50 tx=0x7f013800f740 comp rx=0 tx=0).stop 2024-04-20T22:19:30.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:30.856+0000 7f014a800700 1 -- 172.21.15.62:0/3490241342 shutdown_connections 2024-04-20T22:19:30.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:30.856+0000 7f014a800700 1 --2- 172.21.15.62:0/3490241342 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f012c06ecc0 0x7f012c071170 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:19:30.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:30.856+0000 7f014a800700 1 --2- 172.21.15.62:0/3490241342 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f01441011f0 0x7f014419b970 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:19:30.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:30.856+0000 7f014a800700 1 --2- 172.21.15.62:0/3490241342 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0144101b00 0x7f014419beb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:19:30.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:30.856+0000 7f014a800700 1 -- 172.21.15.62:0/3490241342 >> 172.21.15.62:0/3490241342 conn(0x7f01440faa10 msgr2=0x7f01441047e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:19:30.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:30.856+0000 7f014a800700 1 -- 172.21.15.62:0/3490241342 shutdown_connections 2024-04-20T22:19:30.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:30.856+0000 7f014a800700 1 -- 172.21.15.62:0/3490241342 wait complete. 2024-04-20T22:19:30.859 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 29 2024-04-20T22:19:31.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:31.015+0000 7f9d41600700 1 -- 172.21.15.62:0/2957144970 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9d3c108460 msgr2=0x7f9d3c108830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:19:31.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:31.015+0000 7f9d41600700 1 --2- 172.21.15.62:0/2957144970 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9d3c108460 0x7f9d3c108830 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f9d2c009910 tx=0x7f9d2c009c20 comp rx=0 tx=0).stop 2024-04-20T22:19:31.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:31.015+0000 7f9d41600700 1 -- 172.21.15.62:0/2957144970 shutdown_connections 2024-04-20T22:19:31.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:31.015+0000 7f9d41600700 1 --2- 172.21.15.62:0/2957144970 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9d3c102460 0x7f9d3c1028d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:19:31.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:31.015+0000 7f9d41600700 1 --2- 172.21.15.62:0/2957144970 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9d3c108460 0x7f9d3c108830 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:19:31.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:31.015+0000 7f9d41600700 1 -- 172.21.15.62:0/2957144970 >> 172.21.15.62:0/2957144970 conn(0x7f9d3c0fdf60 msgr2=0x7f9d3c100370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:19:31.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:31.015+0000 7f9d41600700 1 -- 172.21.15.62:0/2957144970 shutdown_connections 2024-04-20T22:19:31.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:31.015+0000 7f9d41600700 1 -- 172.21.15.62:0/2957144970 wait complete. 2024-04-20T22:19:31.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:31.016+0000 7f9d41600700 1 Processor -- start 2024-04-20T22:19:31.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:31.017+0000 7f9d41600700 1 -- start start 2024-04-20T22:19:31.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:31.017+0000 7f9d41600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9d3c102460 0x7f9d3c1981a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:19:31.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:31.017+0000 7f9d41600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9d3c108460 0x7f9d3c1986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:19:31.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:31.017+0000 7f9d41600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d3c198dc0 con 0x7f9d3c108460 2024-04-20T22:19:31.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:31.017+0000 7f9d41600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d3c19cb50 con 0x7f9d3c102460 2024-04-20T22:19:31.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:31.017+0000 7f9d3a000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9d3c108460 0x7f9d3c1986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:19:31.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:31.017+0000 7f9d3a000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9d3c108460 0x7f9d3c1986e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47976/0 (socket says 172.21.15.62:47976) 2024-04-20T22:19:31.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:31.017+0000 7f9d3a000700 1 -- 172.21.15.62:0/645619354 learned_addr learned my addr 172.21.15.62:0/645619354 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:19:31.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:31.018+0000 7f9d3aa00700 1 --2- 172.21.15.62:0/645619354 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9d3c102460 0x7f9d3c1981a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:19:31.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:31.018+0000 7f9d3a000700 1 -- 172.21.15.62:0/645619354 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9d3c102460 msgr2=0x7f9d3c1981a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:19:31.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:31.018+0000 7f9d3a000700 1 --2- 172.21.15.62:0/645619354 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9d3c102460 0x7f9d3c1981a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:19:31.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:31.018+0000 7f9d3a000700 1 -- 172.21.15.62:0/645619354 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d2c0095f0 con 0x7f9d3c108460 2024-04-20T22:19:31.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:31.018+0000 7f9d3a000700 1 --2- 172.21.15.62:0/645619354 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9d3c108460 0x7f9d3c1986e0 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f9d2800b4b0 tx=0x7f9d2800b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:19:31.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:31.018+0000 7f9d33400700 1 -- 172.21.15.62:0/645619354 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9d28011800 con 0x7f9d3c108460 2024-04-20T22:19:31.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:31.018+0000 7f9d33400700 1 -- 172.21.15.62:0/645619354 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9d28011e20 con 0x7f9d3c108460 2024-04-20T22:19:31.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:31.018+0000 7f9d41600700 1 -- 172.21.15.62:0/645619354 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d3c19ce30 con 0x7f9d3c108460 2024-04-20T22:19:31.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:31.018+0000 7f9d33400700 1 -- 172.21.15.62:0/645619354 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9d2800f2e0 con 0x7f9d3c108460 2024-04-20T22:19:31.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:31.019+0000 7f9d41600700 1 -- 172.21.15.62:0/645619354 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d3c19d380 con 0x7f9d3c108460 2024-04-20T22:19:31.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:31.020+0000 7f9d41600700 1 -- 172.21.15.62:0/645619354 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d3c10a9d0 con 0x7f9d3c108460 2024-04-20T22:19:31.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:31.021+0000 7f9d33400700 1 -- 172.21.15.62:0/645619354 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f9d28010380 con 0x7f9d3c108460 2024-04-20T22:19:31.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:31.022+0000 7f9d33400700 1 --2- 172.21.15.62:0/645619354 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9d2006ecc0 0x7f9d20071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:19:31.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:31.022+0000 7f9d33400700 1 -- 172.21.15.62:0/645619354 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f9d2808ea30 con 0x7f9d3c108460 2024-04-20T22:19:31.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:31.022+0000 7f9d3aa00700 1 --2- 172.21.15.62:0/645619354 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9d2006ecc0 0x7f9d20071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:19:31.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:31.023+0000 7f9d3aa00700 1 --2- 172.21.15.62:0/645619354 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9d2006ecc0 0x7f9d20071170 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f9d2c006010 tx=0x7f9d2c01a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:19:31.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:31.027+0000 7f9d33400700 1 -- 172.21.15.62:0/645619354 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f9d28057870 con 0x7f9d3c108460 2024-04-20T22:19:31.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:31.277+0000 7f9d41600700 1 -- 172.21.15.62:0/645619354 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9d3c19d6c0 con 0x7f9d2006ecc0 2024-04-20T22:19:31.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:31.281+0000 7f9d33400700 1 -- 172.21.15.62:0/645619354 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f9d3c19d6c0 con 0x7f9d2006ecc0 2024-04-20T22:19:31.281 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:19:31.281 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T22:19:31.281 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T22:19:31.282 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-20T22:19:31.282 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T22:19:31.282 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr", 2024-04-20T22:19:31.282 INFO:teuthology.orchestra.run.smithi062.stdout: "osd", 2024-04-20T22:19:31.282 INFO:teuthology.orchestra.run.smithi062.stdout: "mds", 2024-04-20T22:19:31.282 INFO:teuthology.orchestra.run.smithi062.stdout: "crash", 2024-04-20T22:19:31.282 INFO:teuthology.orchestra.run.smithi062.stdout: "mon" 2024-04-20T22:19:31.282 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T22:19:31.282 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "16/23 daemons upgraded", 2024-04-20T22:19:31.282 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading mds daemons", 2024-04-20T22:19:31.282 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T22:19:31.282 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:19:31.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:31.285+0000 7f9d41600700 1 -- 172.21.15.62:0/645619354 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9d2006ecc0 msgr2=0x7f9d20071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:19:31.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:31.285+0000 7f9d41600700 1 --2- 172.21.15.62:0/645619354 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9d2006ecc0 0x7f9d20071170 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f9d2c006010 tx=0x7f9d2c01a040 comp rx=0 tx=0).stop 2024-04-20T22:19:31.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:31.285+0000 7f9d41600700 1 -- 172.21.15.62:0/645619354 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9d3c108460 msgr2=0x7f9d3c1986e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:19:31.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:31.285+0000 7f9d41600700 1 --2- 172.21.15.62:0/645619354 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9d3c108460 0x7f9d3c1986e0 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f9d2800b4b0 tx=0x7f9d2800b870 comp rx=0 tx=0).stop 2024-04-20T22:19:31.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:31.285+0000 7f9d41600700 1 -- 172.21.15.62:0/645619354 shutdown_connections 2024-04-20T22:19:31.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:31.285+0000 7f9d41600700 1 --2- 172.21.15.62:0/645619354 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9d2006ecc0 0x7f9d20071170 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:19:31.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:31.285+0000 7f9d41600700 1 --2- 172.21.15.62:0/645619354 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9d3c102460 0x7f9d3c1981a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:19:31.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:31.285+0000 7f9d41600700 1 --2- 172.21.15.62:0/645619354 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9d3c108460 0x7f9d3c1986e0 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:19:31.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:31.285+0000 7f9d41600700 1 -- 172.21.15.62:0/645619354 >> 172.21.15.62:0/645619354 conn(0x7f9d3c0fdf60 msgr2=0x7f9d3c0feb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:19:31.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:31.285+0000 7f9d41600700 1 -- 172.21.15.62:0/645619354 shutdown_connections 2024-04-20T22:19:31.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:31.285+0000 7f9d41600700 1 -- 172.21.15.62:0/645619354 wait complete. 2024-04-20T22:19:31.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:31.448+0000 7f9ced200700 1 -- 172.21.15.62:0/2992386679 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ce80739a0 msgr2=0x7f9ce810c5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:19:31.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:31.448+0000 7f9ced200700 1 --2- 172.21.15.62:0/2992386679 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ce80739a0 0x7f9ce810c5e0 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f9cd4009940 tx=0x7f9cd4009c50 comp rx=0 tx=0).stop 2024-04-20T22:19:31.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:31.448+0000 7f9ced200700 1 -- 172.21.15.62:0/2992386679 shutdown_connections 2024-04-20T22:19:31.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:31.448+0000 7f9ced200700 1 --2- 172.21.15.62:0/2992386679 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ce80739a0 0x7f9ce810c5e0 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:19:31.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:31.448+0000 7f9ced200700 1 --2- 172.21.15.62:0/2992386679 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ce8073090 0x7f9ce8073460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:19:31.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:31.448+0000 7f9ced200700 1 -- 172.21.15.62:0/2992386679 >> 172.21.15.62:0/2992386679 conn(0x7f9ce80fbda0 msgr2=0x7f9ce80fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:19:31.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:31.448+0000 7f9ced200700 1 -- 172.21.15.62:0/2992386679 shutdown_connections 2024-04-20T22:19:31.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:31.448+0000 7f9ced200700 1 -- 172.21.15.62:0/2992386679 wait complete. 2024-04-20T22:19:31.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:31.449+0000 7f9ced200700 1 Processor -- start 2024-04-20T22:19:31.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:31.450+0000 7f9ced200700 1 -- start start 2024-04-20T22:19:31.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:31.450+0000 7f9ced200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ce8073090 0x7f9ce81981a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:19:31.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:31.450+0000 7f9ced200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ce80739a0 0x7f9ce81986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:19:31.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:31.450+0000 7f9ced200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ce8198dc0 con 0x7f9ce8073090 2024-04-20T22:19:31.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:31.450+0000 7f9ced200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ce819cb50 con 0x7f9ce80739a0 2024-04-20T22:19:31.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:31.450+0000 7f9ce5e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ce8073090 0x7f9ce81981a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:19:31.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:31.450+0000 7f9ce5e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ce8073090 0x7f9ce81981a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:32780/0 (socket says 172.21.15.62:32780) 2024-04-20T22:19:31.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:31.450+0000 7f9ce5e00700 1 -- 172.21.15.62:0/395756627 learned_addr learned my addr 172.21.15.62:0/395756627 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:19:31.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:31.451+0000 7f9ce5e00700 1 -- 172.21.15.62:0/395756627 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ce80739a0 msgr2=0x7f9ce81986e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:19:31.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:31.451+0000 7f9ce5400700 1 --2- 172.21.15.62:0/395756627 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ce80739a0 0x7f9ce81986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:19:31.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:31.451+0000 7f9ce5e00700 1 --2- 172.21.15.62:0/395756627 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ce80739a0 0x7f9ce81986e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:19:31.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:31.451+0000 7f9ce5e00700 1 -- 172.21.15.62:0/395756627 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cd40095f0 con 0x7f9ce8073090 2024-04-20T22:19:31.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:31.451+0000 7f9ce5400700 1 --2- 172.21.15.62:0/395756627 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ce80739a0 0x7f9ce81986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T22:19:31.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:31.451+0000 7f9ce5e00700 1 --2- 172.21.15.62:0/395756627 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ce8073090 0x7f9ce81981a0 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f9cd800ba50 tx=0x7f9cd800be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:19:31.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:31.451+0000 7f9cdea00700 1 -- 172.21.15.62:0/395756627 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9cd800ca00 con 0x7f9ce8073090 2024-04-20T22:19:31.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:31.451+0000 7f9cdea00700 1 -- 172.21.15.62:0/395756627 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9cd800cb60 con 0x7f9ce8073090 2024-04-20T22:19:31.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:31.451+0000 7f9ced200700 1 -- 172.21.15.62:0/395756627 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ce819ce30 con 0x7f9ce8073090 2024-04-20T22:19:31.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:31.452+0000 7f9cdea00700 1 -- 172.21.15.62:0/395756627 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9cd80075a0 con 0x7f9ce8073090 2024-04-20T22:19:31.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:31.452+0000 7f9ced200700 1 -- 172.21.15.62:0/395756627 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ce819d380 con 0x7f9ce8073090 2024-04-20T22:19:31.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:31.453+0000 7f9ced200700 1 -- 172.21.15.62:0/395756627 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ce8109ce0 con 0x7f9ce8073090 2024-04-20T22:19:31.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:31.454+0000 7f9cdea00700 1 -- 172.21.15.62:0/395756627 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f9cd800ccd0 con 0x7f9ce8073090 2024-04-20T22:19:31.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:31.455+0000 7f9cdea00700 1 --2- 172.21.15.62:0/395756627 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9ccc06ecc0 0x7f9ccc071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:19:31.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:31.455+0000 7f9cdea00700 1 -- 172.21.15.62:0/395756627 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f9cd8006e40 con 0x7f9ce8073090 2024-04-20T22:19:31.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:31.455+0000 7f9ce5400700 1 --2- 172.21.15.62:0/395756627 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9ccc06ecc0 0x7f9ccc071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:19:31.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:31.456+0000 7f9ce5400700 1 --2- 172.21.15.62:0/395756627 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9ccc06ecc0 0x7f9ccc071170 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f9ce81997c0 tx=0x7f9cd4005fd0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:19:31.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:31.460+0000 7f9cdea00700 1 -- 172.21.15.62:0/395756627 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f9cd80573f0 con 0x7f9ce8073090 2024-04-20T22:19:31.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:19:31 smithi062 ceph-mon[76322]: from='client.36332 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:19:31.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:19:31 smithi062 ceph-mon[76322]: from='client.36336 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:19:31.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:19:31 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/3490241342' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:19:31.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:19:31 smithi189 ceph-mon[57022]: from='client.36332 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:19:31.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:19:31 smithi189 ceph-mon[57022]: from='client.36336 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:19:31.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:19:31 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/3490241342' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:19:31.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:31.786+0000 7f9ced200700 1 -- 172.21.15.62:0/395756627 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f9ce804f2a0 con 0x7f9ce8073090 2024-04-20T22:19:31.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:31.787+0000 7f9cdea00700 1 -- 172.21.15.62:0/395756627 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7f9cd805aab0 con 0x7f9ce8073090 2024-04-20T22:19:31.787 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-20T22:19:31.787 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-20T22:19:31.787 INFO:teuthology.orchestra.run.smithi062.stdout: host smithi189 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi189/config 2024-04-20T22:19:31.787 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-20T22:19:31.787 INFO:teuthology.orchestra.run.smithi062.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi189 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp11670v2q:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:19:31.788 INFO:teuthology.orchestra.run.smithi062.stdout:Traceback (most recent call last): 2024-04-20T22:19:31.788 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-20T22:19:31.788 INFO:teuthology.orchestra.run.smithi062.stdout: return _run_code(code, main_globals, None, 2024-04-20T22:19:31.788 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-20T22:19:31.788 INFO:teuthology.orchestra.run.smithi062.stdout: exec(code, run_globals) 2024-04-20T22:19:31.788 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5618, in 2024-04-20T22:19:31.788 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5606, in main 2024-04-20T22:19:31.788 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 408, in _infer_config 2024-04-20T22:19:31.788 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 323, in _infer_fsid 2024-04-20T22:19:31.788 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 436, in _infer_image 2024-04-20T22:19:31.788 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 310, in _validate_fsid 2024-04-20T22:19:31.788 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 3359, in command_ceph_volume 2024-04-20T22:19:31.788 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-20T22:19:31.789 INFO:teuthology.orchestra.run.smithi062.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi189 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp11670v2q:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:19:31.789 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-20T22:19:31.789 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is degraded 2024-04-20T22:19:31.789 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T22:19:31.789 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T22:19:31.789 INFO:teuthology.orchestra.run.smithi062.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-20T22:19:31.789 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-20T22:19:31.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:31.789+0000 7f9ced200700 1 -- 172.21.15.62:0/395756627 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9ccc06ecc0 msgr2=0x7f9ccc071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:19:31.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:31.789+0000 7f9ced200700 1 --2- 172.21.15.62:0/395756627 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9ccc06ecc0 0x7f9ccc071170 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f9ce81997c0 tx=0x7f9cd4005fd0 comp rx=0 tx=0).stop 2024-04-20T22:19:31.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:31.789+0000 7f9ced200700 1 -- 172.21.15.62:0/395756627 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ce8073090 msgr2=0x7f9ce81981a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:19:31.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:31.789+0000 7f9ced200700 1 --2- 172.21.15.62:0/395756627 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ce8073090 0x7f9ce81981a0 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f9cd800ba50 tx=0x7f9cd800be10 comp rx=0 tx=0).stop 2024-04-20T22:19:31.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:31.789+0000 7f9ced200700 1 -- 172.21.15.62:0/395756627 shutdown_connections 2024-04-20T22:19:31.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:31.789+0000 7f9ced200700 1 --2- 172.21.15.62:0/395756627 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9ccc06ecc0 0x7f9ccc071170 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:19:31.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:31.789+0000 7f9ced200700 1 --2- 172.21.15.62:0/395756627 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ce8073090 0x7f9ce81981a0 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:19:31.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:31.789+0000 7f9ced200700 1 --2- 172.21.15.62:0/395756627 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ce80739a0 0x7f9ce81986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:19:31.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:31.789+0000 7f9ced200700 1 -- 172.21.15.62:0/395756627 >> 172.21.15.62:0/395756627 conn(0x7f9ce80fbda0 msgr2=0x7f9ce8106e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:19:31.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:31.790+0000 7f9ced200700 1 -- 172.21.15.62:0/395756627 shutdown_connections 2024-04-20T22:19:31.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:19:31.790+0000 7f9ced200700 1 -- 172.21.15.62:0/395756627 wait complete. 2024-04-20T22:19:32.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:19:32 smithi189 ceph-mon[57022]: from='client.36348 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:19:32.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:19:32 smithi189 ceph-mon[57022]: pgmap v1330: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:19:32.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:19:32 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/395756627' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:19:32.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:19:32 smithi062 ceph-mon[76322]: from='client.36348 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:19:32.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:19:32 smithi062 ceph-mon[76322]: pgmap v1330: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:19:32.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:19:32 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/395756627' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:19:33.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:19:33 smithi189 ceph-mon[57022]: pgmap v1331: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:19:33.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:19:33 smithi062 ceph-mon[76322]: pgmap v1331: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:19:34.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:19:34 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:19:34.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:19:34 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:19:35.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:19:35 smithi189 ceph-mon[57022]: pgmap v1332: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:19:35.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:19:35 smithi062 ceph-mon[76322]: pgmap v1332: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:19:38.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:19:38 smithi189 ceph-mon[57022]: pgmap v1333: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:19:38.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:19:38 smithi062 ceph-mon[76322]: pgmap v1333: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:19:39.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:19:39 smithi189 ceph-mon[57022]: pgmap v1334: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:19:39.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:19:39 smithi062 ceph-mon[76322]: pgmap v1334: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:19:42.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:19:42 smithi189 ceph-mon[57022]: pgmap v1335: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:19:42.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:19:42 smithi062 ceph-mon[76322]: pgmap v1335: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:19:43.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:19:43 smithi189 ceph-mon[57022]: pgmap v1336: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:19:43.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:19:43 smithi062 ceph-mon[76322]: pgmap v1336: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:19:45.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:19:45 smithi189 ceph-mon[57022]: pgmap v1337: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:19:45.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:19:45 smithi062 ceph-mon[76322]: pgmap v1337: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:19:47.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:19:47 smithi189 ceph-mon[57022]: pgmap v1338: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:19:47.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:19:47 smithi062 ceph-mon[76322]: pgmap v1338: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:19:49.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:19:49 smithi189 ceph-mon[57022]: pgmap v1339: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:19:49.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:19:49 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:19:49.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:19:49 smithi062 ceph-mon[76322]: pgmap v1339: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:19:49.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:19:49 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:19:51.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:19:51 smithi189 ceph-mon[57022]: pgmap v1340: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:19:51.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:19:51 smithi062 ceph-mon[76322]: pgmap v1340: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:19:53.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:19:53 smithi189 ceph-mon[57022]: pgmap v1341: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:19:53.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:19:53 smithi062 ceph-mon[76322]: pgmap v1341: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:19:55.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:19:55 smithi189 ceph-mon[57022]: pgmap v1342: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:19:55.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:19:55 smithi062 ceph-mon[76322]: pgmap v1342: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:19:57.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:19:57 smithi189 ceph-mon[57022]: pgmap v1343: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:19:57.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:19:57 smithi062 ceph-mon[76322]: pgmap v1343: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:19:59.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:19:59 smithi189 ceph-mon[57022]: pgmap v1344: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:19:59.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:19:59 smithi062 ceph-mon[76322]: pgmap v1344: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:20:00.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:20:00 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-mon-smithi062[76318]: 2024-04-20T22:19:59.999+0000 7f6157e00700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-20T22:20:00.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:20:00 smithi189 ceph-mon[57022]: overall HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-20T22:20:00.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:20:00 smithi062 ceph-mon[76322]: overall HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-20T22:20:01.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:20:01 smithi062 ceph-mon[76322]: pgmap v1345: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:20:01.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:01.967+0000 7f44a2600700 1 -- 172.21.15.62:0/1446886733 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f449c102e70 msgr2=0x7f449c106e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:20:01.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:01.967+0000 7f44a2600700 1 --2- 172.21.15.62:0/1446886733 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f449c102e70 0x7f449c106e40 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f4488009910 tx=0x7f4488009c20 comp rx=0 tx=0).stop 2024-04-20T22:20:01.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:01.967+0000 7f44a2600700 1 -- 172.21.15.62:0/1446886733 shutdown_connections 2024-04-20T22:20:01.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:01.967+0000 7f44a2600700 1 --2- 172.21.15.62:0/1446886733 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f449c102e70 0x7f449c106e40 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:20:01.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:01.967+0000 7f44a2600700 1 --2- 172.21.15.62:0/1446886733 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f449c1024d0 0x7f449c1028a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:20:01.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:01.967+0000 7f44a2600700 1 -- 172.21.15.62:0/1446886733 >> 172.21.15.62:0/1446886733 conn(0x7f449c0fdd90 msgr2=0x7f449c1001a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:20:01.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:01.967+0000 7f44a2600700 1 -- 172.21.15.62:0/1446886733 shutdown_connections 2024-04-20T22:20:01.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:01.967+0000 7f44a2600700 1 -- 172.21.15.62:0/1446886733 wait complete. 2024-04-20T22:20:01.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:01.969+0000 7f44a2600700 1 Processor -- start 2024-04-20T22:20:01.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:01.969+0000 7f44a2600700 1 -- start start 2024-04-20T22:20:01.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:01.969+0000 7f44a2600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f449c1024d0 0x7f449c197e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:20:01.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:01.969+0000 7f44a2600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f449c102e70 0x7f449c198350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:20:01.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:01.969+0000 7f44a2600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f449c198a30 con 0x7f449c1024d0 2024-04-20T22:20:01.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:01.969+0000 7f44a2600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f449c19c7c0 con 0x7f449c102e70 2024-04-20T22:20:01.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:01.970+0000 7f44a1200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f449c1024d0 0x7f449c197e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:20:01.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:01.970+0000 7f44a1200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f449c1024d0 0x7f449c197e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49866/0 (socket says 172.21.15.62:49866) 2024-04-20T22:20:01.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:01.970+0000 7f44a1200700 1 -- 172.21.15.62:0/3343574855 learned_addr learned my addr 172.21.15.62:0/3343574855 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:20:01.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:01.970+0000 7f449be00700 1 --2- 172.21.15.62:0/3343574855 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f449c102e70 0x7f449c198350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:20:01.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:01.970+0000 7f44a1200700 1 -- 172.21.15.62:0/3343574855 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f449c102e70 msgr2=0x7f449c198350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:20:01.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:01.970+0000 7f44a1200700 1 --2- 172.21.15.62:0/3343574855 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f449c102e70 0x7f449c198350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:20:01.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:01.970+0000 7f44a1200700 1 -- 172.21.15.62:0/3343574855 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44880095f0 con 0x7f449c1024d0 2024-04-20T22:20:01.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:01.971+0000 7f44a1200700 1 --2- 172.21.15.62:0/3343574855 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f449c1024d0 0x7f449c197e10 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f449000d8d0 tx=0x7f449000dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:20:01.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:01.971+0000 7f4499600700 1 -- 172.21.15.62:0/3343574855 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f44900098a0 con 0x7f449c1024d0 2024-04-20T22:20:01.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:01.971+0000 7f4499600700 1 -- 172.21.15.62:0/3343574855 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4490010460 con 0x7f449c1024d0 2024-04-20T22:20:01.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:01.971+0000 7f4499600700 1 -- 172.21.15.62:0/3343574855 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f449000f5f0 con 0x7f449c1024d0 2024-04-20T22:20:01.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:01.972+0000 7f44a2600700 1 -- 172.21.15.62:0/3343574855 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f449c19caa0 con 0x7f449c1024d0 2024-04-20T22:20:01.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:01.972+0000 7f44a2600700 1 -- 172.21.15.62:0/3343574855 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f449c19cff0 con 0x7f449c1024d0 2024-04-20T22:20:01.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:01.973+0000 7f44a2600700 1 -- 172.21.15.62:0/3343574855 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f449c10a840 con 0x7f449c1024d0 2024-04-20T22:20:01.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:01.974+0000 7f4499600700 1 -- 172.21.15.62:0/3343574855 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f4490009a00 con 0x7f449c1024d0 2024-04-20T22:20:01.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:01.975+0000 7f4499600700 1 --2- 172.21.15.62:0/3343574855 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f448406ecc0 0x7f4484071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:20:01.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:01.975+0000 7f4499600700 1 -- 172.21.15.62:0/3343574855 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f449008ed10 con 0x7f449c1024d0 2024-04-20T22:20:01.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:01.976+0000 7f449be00700 1 --2- 172.21.15.62:0/3343574855 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f448406ecc0 0x7f4484071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:20:01.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:01.977+0000 7f449be00700 1 --2- 172.21.15.62:0/3343574855 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f448406ecc0 0x7f4484071170 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f449c199430 tx=0x7f448801a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:20:01.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:01.980+0000 7f4499600700 1 -- 172.21.15.62:0/3343574855 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f4490057bd0 con 0x7f449c1024d0 2024-04-20T22:20:01.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:20:01 smithi189 ceph-mon[57022]: pgmap v1345: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:20:02.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:02.241+0000 7f44a2600700 1 -- 172.21.15.62:0/3343574855 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f449c19d2d0 con 0x7f448406ecc0 2024-04-20T22:20:02.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:02.243+0000 7f4499600700 1 -- 172.21.15.62:0/3343574855 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f449c19d2d0 con 0x7f448406ecc0 2024-04-20T22:20:02.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:02.245+0000 7f44a2600700 1 -- 172.21.15.62:0/3343574855 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f448406ecc0 msgr2=0x7f4484071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:20:02.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:02.245+0000 7f44a2600700 1 --2- 172.21.15.62:0/3343574855 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f448406ecc0 0x7f4484071170 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f449c199430 tx=0x7f448801a040 comp rx=0 tx=0).stop 2024-04-20T22:20:02.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:02.245+0000 7f44a2600700 1 -- 172.21.15.62:0/3343574855 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f449c1024d0 msgr2=0x7f449c197e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:20:02.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:02.245+0000 7f44a2600700 1 --2- 172.21.15.62:0/3343574855 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f449c1024d0 0x7f449c197e10 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f449000d8d0 tx=0x7f449000dbe0 comp rx=0 tx=0).stop 2024-04-20T22:20:02.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:02.246+0000 7f44a2600700 1 -- 172.21.15.62:0/3343574855 shutdown_connections 2024-04-20T22:20:02.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:02.246+0000 7f44a2600700 1 --2- 172.21.15.62:0/3343574855 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f448406ecc0 0x7f4484071170 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:20:02.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:02.246+0000 7f44a2600700 1 --2- 172.21.15.62:0/3343574855 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f449c1024d0 0x7f449c197e10 unknown :-1 s=CLOSED pgs=1142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:20:02.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:02.246+0000 7f44a2600700 1 --2- 172.21.15.62:0/3343574855 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f449c102e70 0x7f449c198350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:20:02.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:02.246+0000 7f44a2600700 1 -- 172.21.15.62:0/3343574855 >> 172.21.15.62:0/3343574855 conn(0x7f449c0fdd90 msgr2=0x7f449c100180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:20:02.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:02.246+0000 7f44a2600700 1 -- 172.21.15.62:0/3343574855 shutdown_connections 2024-04-20T22:20:02.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:02.246+0000 7f44a2600700 1 -- 172.21.15.62:0/3343574855 wait complete. 2024-04-20T22:20:02.263 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T22:20:02.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:02.412+0000 7fc477a00700 1 -- 172.21.15.62:0/2305210652 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc470073a00 msgr2=0x7fc470110d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:20:02.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:02.412+0000 7fc477a00700 1 --2- 172.21.15.62:0/2305210652 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc470073a00 0x7fc470110d50 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7fc45c009910 tx=0x7fc45c009c20 comp rx=0 tx=0).stop 2024-04-20T22:20:02.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:02.412+0000 7fc477a00700 1 -- 172.21.15.62:0/2305210652 shutdown_connections 2024-04-20T22:20:02.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:02.412+0000 7fc477a00700 1 --2- 172.21.15.62:0/2305210652 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc470073a00 0x7fc470110d50 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:20:02.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:02.412+0000 7fc477a00700 1 --2- 172.21.15.62:0/2305210652 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4700730f0 0x7fc4700734c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:20:02.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:02.412+0000 7fc477a00700 1 -- 172.21.15.62:0/2305210652 >> 172.21.15.62:0/2305210652 conn(0x7fc4700fbda0 msgr2=0x7fc4700fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:20:02.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:02.413+0000 7fc477a00700 1 -- 172.21.15.62:0/2305210652 shutdown_connections 2024-04-20T22:20:02.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:02.413+0000 7fc477a00700 1 -- 172.21.15.62:0/2305210652 wait complete. 2024-04-20T22:20:02.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:02.414+0000 7fc477a00700 1 Processor -- start 2024-04-20T22:20:02.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:02.414+0000 7fc477a00700 1 -- start start 2024-04-20T22:20:02.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:02.414+0000 7fc477a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4700730f0 0x7fc47019c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:20:02.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:02.414+0000 7fc477a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc470073a00 0x7fc47019c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:20:02.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:02.414+0000 7fc477a00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc47019d040 con 0x7fc470073a00 2024-04-20T22:20:02.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:02.414+0000 7fc477a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4701a0dd0 con 0x7fc4700730f0 2024-04-20T22:20:02.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:02.414+0000 7fc46fe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc470073a00 0x7fc47019c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:20:02.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:02.415+0000 7fc46fe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc470073a00 0x7fc47019c960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49882/0 (socket says 172.21.15.62:49882) 2024-04-20T22:20:02.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:02.415+0000 7fc46fe00700 1 -- 172.21.15.62:0/2501961593 learned_addr learned my addr 172.21.15.62:0/2501961593 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:20:02.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:02.415+0000 7fc46fe00700 1 -- 172.21.15.62:0/2501961593 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4700730f0 msgr2=0x7fc47019c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:20:02.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:02.415+0000 7fc46fe00700 1 --2- 172.21.15.62:0/2501961593 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4700730f0 0x7fc47019c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:20:02.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:02.415+0000 7fc46fe00700 1 -- 172.21.15.62:0/2501961593 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc45c0095f0 con 0x7fc470073a00 2024-04-20T22:20:02.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:02.415+0000 7fc46fe00700 1 --2- 172.21.15.62:0/2501961593 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc470073a00 0x7fc47019c960 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7fc45c00b5c0 tx=0x7fc45c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:20:02.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:02.416+0000 7fc46d600700 1 -- 172.21.15.62:0/2501961593 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc45c01d070 con 0x7fc470073a00 2024-04-20T22:20:02.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:02.416+0000 7fc46d600700 1 -- 172.21.15.62:0/2501961593 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc45c005e00 con 0x7fc470073a00 2024-04-20T22:20:02.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:02.416+0000 7fc46d600700 1 -- 172.21.15.62:0/2501961593 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc45c017400 con 0x7fc470073a00 2024-04-20T22:20:02.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:02.416+0000 7fc477a00700 1 -- 172.21.15.62:0/2501961593 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4701a10b0 con 0x7fc470073a00 2024-04-20T22:20:02.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:02.416+0000 7fc477a00700 1 -- 172.21.15.62:0/2501961593 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4701a15b0 con 0x7fc470073a00 2024-04-20T22:20:02.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:02.417+0000 7fc477a00700 1 -- 172.21.15.62:0/2501961593 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc47010e4d0 con 0x7fc470073a00 2024-04-20T22:20:02.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:02.418+0000 7fc46d600700 1 -- 172.21.15.62:0/2501961593 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fc45c00f460 con 0x7fc470073a00 2024-04-20T22:20:02.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:02.419+0000 7fc46d600700 1 --2- 172.21.15.62:0/2501961593 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc45806ecc0 0x7fc458071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:20:02.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:02.419+0000 7fc46d600700 1 -- 172.21.15.62:0/2501961593 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fc45c0903f0 con 0x7fc470073a00 2024-04-20T22:20:02.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:02.419+0000 7fc475000700 1 --2- 172.21.15.62:0/2501961593 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc45806ecc0 0x7fc458071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:20:02.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:02.420+0000 7fc475000700 1 --2- 172.21.15.62:0/2501961593 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc45806ecc0 0x7fc458071170 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fc464005950 tx=0x7fc4640058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:20:02.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:02.424+0000 7fc46d600700 1 -- 172.21.15.62:0/2501961593 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fc45c059100 con 0x7fc470073a00 2024-04-20T22:20:02.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:02.673+0000 7fc477a00700 1 -- 172.21.15.62:0/2501961593 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc4701a1890 con 0x7fc45806ecc0 2024-04-20T22:20:02.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:02.675+0000 7fc46d600700 1 -- 172.21.15.62:0/2501961593 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fc4701a1890 con 0x7fc45806ecc0 2024-04-20T22:20:02.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:02.678+0000 7fc477a00700 1 -- 172.21.15.62:0/2501961593 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc45806ecc0 msgr2=0x7fc458071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:20:02.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:02.678+0000 7fc477a00700 1 --2- 172.21.15.62:0/2501961593 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc45806ecc0 0x7fc458071170 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fc464005950 tx=0x7fc4640058e0 comp rx=0 tx=0).stop 2024-04-20T22:20:02.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:02.678+0000 7fc477a00700 1 -- 172.21.15.62:0/2501961593 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc470073a00 msgr2=0x7fc47019c960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:20:02.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:02.678+0000 7fc477a00700 1 --2- 172.21.15.62:0/2501961593 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc470073a00 0x7fc47019c960 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7fc45c00b5c0 tx=0x7fc45c00be20 comp rx=0 tx=0).stop 2024-04-20T22:20:02.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:02.678+0000 7fc477a00700 1 -- 172.21.15.62:0/2501961593 shutdown_connections 2024-04-20T22:20:02.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:02.678+0000 7fc477a00700 1 --2- 172.21.15.62:0/2501961593 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc45806ecc0 0x7fc458071170 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:20:02.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:02.678+0000 7fc477a00700 1 --2- 172.21.15.62:0/2501961593 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4700730f0 0x7fc47019c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:20:02.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:02.678+0000 7fc477a00700 1 --2- 172.21.15.62:0/2501961593 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc470073a00 0x7fc47019c960 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:20:02.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:02.678+0000 7fc477a00700 1 -- 172.21.15.62:0/2501961593 >> 172.21.15.62:0/2501961593 conn(0x7fc4700fbda0 msgr2=0x7fc470102860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:20:02.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:02.678+0000 7fc477a00700 1 -- 172.21.15.62:0/2501961593 shutdown_connections 2024-04-20T22:20:02.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:02.678+0000 7fc477a00700 1 -- 172.21.15.62:0/2501961593 wait complete. 2024-04-20T22:20:02.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:20:02 smithi062 ceph-mon[76322]: from='client.36356 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:20:02.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:02.842+0000 7f4b94800700 1 -- 172.21.15.62:0/3603661989 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4b8c069140 msgr2=0x7f4b8c104eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:20:02.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:02.842+0000 7f4b94800700 1 --2- 172.21.15.62:0/3603661989 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4b8c069140 0x7f4b8c104eb0 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f4b7c009910 tx=0x7f4b7c009c20 comp rx=0 tx=0).stop 2024-04-20T22:20:02.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:02.843+0000 7f4b94800700 1 -- 172.21.15.62:0/3603661989 shutdown_connections 2024-04-20T22:20:02.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:02.843+0000 7f4b94800700 1 --2- 172.21.15.62:0/3603661989 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4b8c069140 0x7f4b8c104eb0 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:20:02.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:02.843+0000 7f4b94800700 1 --2- 172.21.15.62:0/3603661989 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b8c0687a0 0x7f4b8c068b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:20:02.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:02.843+0000 7f4b94800700 1 -- 172.21.15.62:0/3603661989 >> 172.21.15.62:0/3603661989 conn(0x7f4b8c075130 msgr2=0x7f4b8c075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:20:02.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:02.843+0000 7f4b94800700 1 -- 172.21.15.62:0/3603661989 shutdown_connections 2024-04-20T22:20:02.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:02.843+0000 7f4b94800700 1 -- 172.21.15.62:0/3603661989 wait complete. 2024-04-20T22:20:02.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:02.844+0000 7f4b94800700 1 Processor -- start 2024-04-20T22:20:02.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:02.844+0000 7f4b94800700 1 -- start start 2024-04-20T22:20:02.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:02.844+0000 7f4b94800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4b8c0687a0 0x7f4b8c0ffca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:20:02.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:02.844+0000 7f4b94800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b8c069140 0x7f4b8c1001e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:20:02.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:02.844+0000 7f4b94800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b8c100720 con 0x7f4b8c0687a0 2024-04-20T22:20:02.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:02.844+0000 7f4b94800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b8c100860 con 0x7f4b8c069140 2024-04-20T22:20:02.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:02.845+0000 7f4b91e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4b8c0687a0 0x7f4b8c0ffca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:20:02.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:02.845+0000 7f4b91e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4b8c0687a0 0x7f4b8c0ffca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49894/0 (socket says 172.21.15.62:49894) 2024-04-20T22:20:02.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:02.845+0000 7f4b91e00700 1 -- 172.21.15.62:0/4179560350 learned_addr learned my addr 172.21.15.62:0/4179560350 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:20:02.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:02.845+0000 7f4b91400700 1 --2- 172.21.15.62:0/4179560350 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b8c069140 0x7f4b8c1001e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:20:02.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:02.846+0000 7f4b91e00700 1 -- 172.21.15.62:0/4179560350 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b8c069140 msgr2=0x7f4b8c1001e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:20:02.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:02.846+0000 7f4b91e00700 1 --2- 172.21.15.62:0/4179560350 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b8c069140 0x7f4b8c1001e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:20:02.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:02.846+0000 7f4b91e00700 1 -- 172.21.15.62:0/4179560350 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b7c0095f0 con 0x7f4b8c0687a0 2024-04-20T22:20:02.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:02.846+0000 7f4b91e00700 1 --2- 172.21.15.62:0/4179560350 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4b8c0687a0 0x7f4b8c0ffca0 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f4b8000d8d0 tx=0x7f4b8000dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:20:02.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:02.846+0000 7f4b86a00700 1 -- 172.21.15.62:0/4179560350 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4b800098a0 con 0x7f4b8c0687a0 2024-04-20T22:20:02.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:02.846+0000 7f4b86a00700 1 -- 172.21.15.62:0/4179560350 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4b80010460 con 0x7f4b8c0687a0 2024-04-20T22:20:02.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:02.846+0000 7f4b94800700 1 -- 172.21.15.62:0/4179560350 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b8c1a2030 con 0x7f4b8c0687a0 2024-04-20T22:20:02.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:02.847+0000 7f4b86a00700 1 -- 172.21.15.62:0/4179560350 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4b8000f5f0 con 0x7f4b8c0687a0 2024-04-20T22:20:02.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:02.847+0000 7f4b94800700 1 -- 172.21.15.62:0/4179560350 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b8c1a2440 con 0x7f4b8c0687a0 2024-04-20T22:20:02.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:02.848+0000 7f4b94800700 1 -- 172.21.15.62:0/4179560350 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b8c108750 con 0x7f4b8c0687a0 2024-04-20T22:20:02.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:02.849+0000 7f4b86a00700 1 -- 172.21.15.62:0/4179560350 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f4b80009a00 con 0x7f4b8c0687a0 2024-04-20T22:20:02.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:02.850+0000 7f4b86a00700 1 --2- 172.21.15.62:0/4179560350 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f4b7406ecc0 0x7f4b74071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:20:02.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:02.850+0000 7f4b86a00700 1 -- 172.21.15.62:0/4179560350 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f4b8008edd0 con 0x7f4b8c0687a0 2024-04-20T22:20:02.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:02.850+0000 7f4b91400700 1 --2- 172.21.15.62:0/4179560350 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f4b7406ecc0 0x7f4b74071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:20:02.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:02.851+0000 7f4b91400700 1 --2- 172.21.15.62:0/4179560350 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f4b7406ecc0 0x7f4b74071170 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f4b8c101490 tx=0x7f4b7c01a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:20:02.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:02.855+0000 7f4b86a00700 1 -- 172.21.15.62:0/4179560350 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f4b80057ae0 con 0x7f4b8c0687a0 2024-04-20T22:20:02.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:20:02 smithi189 ceph-mon[57022]: from='client.36356 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:20:03.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:03.100+0000 7f4b94800700 1 -- 172.21.15.62:0/4179560350 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f4b8c101310 con 0x7f4b7406ecc0 2024-04-20T22:20:03.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:03.112+0000 7f4b86a00700 1 -- 172.21.15.62:0/4179560350 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f4b7c01e040 con 0x7f4b7406ecc0 2024-04-20T22:20:03.112 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T22:20:03.112 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (45m) 6m ago 59m 14.6M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T22:20:03.113 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (60m) 6m ago 60m 11.6M - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T22:20:03.113 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (58m) 6m ago 58m 11.2M - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T22:20:03.113 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (41m) 6m ago 60m 6660k - 19.0.0-3133-ga8c84f59 76bc6b91b726 2825d5ba11c6 2024-04-20T22:20:03.113 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (41m) 6m ago 58m 6665k - 19.0.0-3133-ga8c84f59 76bc6b91b726 20f746fa1bba 2024-04-20T22:20:03.113 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (45m) 6m ago 59m 75.5M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T22:20:03.113 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (37m) 6m ago 52m 22.9M - 19.0.0-3133-ga8c84f59 76bc6b91b726 de80bc90a683 2024-04-20T22:20:03.113 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (37m) 6m ago 52m 23.0M - 19.0.0-3133-ga8c84f59 76bc6b91b726 cd793b7ee98e 2024-04-20T22:20:03.113 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (22m) 6m ago 52m 18.5M - 19.0.0-3133-ga8c84f59 76bc6b91b726 0e7b0bc4d1ab 2024-04-20T22:20:03.113 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (6m) 6m ago 52m 11.4M - 19.0.0-3133-ga8c84f59 76bc6b91b726 0af732ac9e06 2024-04-20T22:20:03.113 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (48m) 6m ago 61m 517M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T22:20:03.113 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (47m) 6m ago 58m 460M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T22:20:03.113 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (43m) 6m ago 61m 86.3M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 019eb1479485 2024-04-20T22:20:03.113 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (43m) 6m ago 58m 58.0M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 55aac560aab6 2024-04-20T22:20:03.113 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (47m) 6m ago 59m 8766k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T22:20:03.113 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (46m) 6m ago 58m 8711k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T22:20:03.114 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (41m) 6m ago 57m 91.3M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 ee2bb901f26e 2024-04-20T22:20:03.114 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (40m) 6m ago 56m 94.9M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 9075152d4162 2024-04-20T22:20:03.114 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (40m) 6m ago 56m 78.5M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 2d4f3e38dc0e 2024-04-20T22:20:03.114 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (39m) 6m ago 55m 94.1M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 8f97b17ae0eb 2024-04-20T22:20:03.114 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (39m) 6m ago 55m 96.1M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 653d8079967d 2024-04-20T22:20:03.114 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (38m) 6m ago 54m 90.9M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 596dded14c12 2024-04-20T22:20:03.114 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (46m) 6m ago 59m 90.4M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T22:20:03.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:03.115+0000 7f4b94800700 1 -- 172.21.15.62:0/4179560350 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f4b7406ecc0 msgr2=0x7f4b74071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:20:03.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:03.115+0000 7f4b94800700 1 --2- 172.21.15.62:0/4179560350 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f4b7406ecc0 0x7f4b74071170 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f4b8c101490 tx=0x7f4b7c01a040 comp rx=0 tx=0).stop 2024-04-20T22:20:03.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:03.115+0000 7f4b94800700 1 -- 172.21.15.62:0/4179560350 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4b8c0687a0 msgr2=0x7f4b8c0ffca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:20:03.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:03.115+0000 7f4b94800700 1 --2- 172.21.15.62:0/4179560350 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4b8c0687a0 0x7f4b8c0ffca0 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f4b8000d8d0 tx=0x7f4b8000dbe0 comp rx=0 tx=0).stop 2024-04-20T22:20:03.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:03.115+0000 7f4b94800700 1 -- 172.21.15.62:0/4179560350 shutdown_connections 2024-04-20T22:20:03.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:03.115+0000 7f4b94800700 1 --2- 172.21.15.62:0/4179560350 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f4b7406ecc0 0x7f4b74071170 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:20:03.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:03.115+0000 7f4b94800700 1 --2- 172.21.15.62:0/4179560350 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4b8c0687a0 0x7f4b8c0ffca0 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:20:03.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:03.115+0000 7f4b94800700 1 --2- 172.21.15.62:0/4179560350 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b8c069140 0x7f4b8c1001e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:20:03.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:03.115+0000 7f4b94800700 1 -- 172.21.15.62:0/4179560350 >> 172.21.15.62:0/4179560350 conn(0x7f4b8c075130 msgr2=0x7f4b8c104720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:20:03.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:03.115+0000 7f4b94800700 1 -- 172.21.15.62:0/4179560350 shutdown_connections 2024-04-20T22:20:03.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:03.115+0000 7f4b94800700 1 -- 172.21.15.62:0/4179560350 wait complete. 2024-04-20T22:20:03.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:03.277+0000 7f45d4400700 1 -- 172.21.15.62:0/646417376 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f45cc10c8f0 msgr2=0x7f45cc10ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:20:03.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:03.277+0000 7f45d4400700 1 --2- 172.21.15.62:0/646417376 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f45cc10c8f0 0x7f45cc10ccc0 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f45c0009910 tx=0x7f45c0009c20 comp rx=0 tx=0).stop 2024-04-20T22:20:03.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:03.277+0000 7f45d4400700 1 -- 172.21.15.62:0/646417376 shutdown_connections 2024-04-20T22:20:03.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:03.277+0000 7f45d4400700 1 --2- 172.21.15.62:0/646417376 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45cc106b80 0x7f45cc106ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:20:03.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:03.277+0000 7f45d4400700 1 --2- 172.21.15.62:0/646417376 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f45cc10c8f0 0x7f45cc10ccc0 unknown :-1 s=CLOSED pgs=1147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:20:03.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:03.277+0000 7f45d4400700 1 -- 172.21.15.62:0/646417376 >> 172.21.15.62:0/646417376 conn(0x7f45cc076060 msgr2=0x7f45cc078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:20:03.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:03.277+0000 7f45d4400700 1 -- 172.21.15.62:0/646417376 shutdown_connections 2024-04-20T22:20:03.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:03.277+0000 7f45d4400700 1 -- 172.21.15.62:0/646417376 wait complete. 2024-04-20T22:20:03.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:03.278+0000 7f45d4400700 1 Processor -- start 2024-04-20T22:20:03.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:03.279+0000 7f45d4400700 1 -- start start 2024-04-20T22:20:03.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:03.279+0000 7f45d4400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45cc106b80 0x7f45cc19c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:20:03.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:03.279+0000 7f45d4400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f45cc10c8f0 0x7f45cc19ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:20:03.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:03.279+0000 7f45d4400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45cc19d120 con 0x7f45cc10c8f0 2024-04-20T22:20:03.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:03.279+0000 7f45d4400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45cc1a0eb0 con 0x7f45cc106b80 2024-04-20T22:20:03.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:03.279+0000 7f45d1000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f45cc10c8f0 0x7f45cc19ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:20:03.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:03.279+0000 7f45d1000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f45cc10c8f0 0x7f45cc19ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49910/0 (socket says 172.21.15.62:49910) 2024-04-20T22:20:03.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:03.279+0000 7f45d1000700 1 -- 172.21.15.62:0/758683208 learned_addr learned my addr 172.21.15.62:0/758683208 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:20:03.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:03.280+0000 7f45d1a00700 1 --2- 172.21.15.62:0/758683208 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45cc106b80 0x7f45cc19c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:20:03.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:03.280+0000 7f45d1000700 1 -- 172.21.15.62:0/758683208 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45cc106b80 msgr2=0x7f45cc19c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:20:03.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:03.280+0000 7f45d1000700 1 --2- 172.21.15.62:0/758683208 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45cc106b80 0x7f45cc19c500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:20:03.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:03.280+0000 7f45d1000700 1 -- 172.21.15.62:0/758683208 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45c00095f0 con 0x7f45cc10c8f0 2024-04-20T22:20:03.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:03.280+0000 7f45d1000700 1 --2- 172.21.15.62:0/758683208 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f45cc10c8f0 0x7f45cc19ca40 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f45b800d8d0 tx=0x7f45b800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:20:03.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:03.280+0000 7f45ca000700 1 -- 172.21.15.62:0/758683208 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f45b8009940 con 0x7f45cc10c8f0 2024-04-20T22:20:03.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:03.280+0000 7f45ca000700 1 -- 172.21.15.62:0/758683208 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f45b8010460 con 0x7f45cc10c8f0 2024-04-20T22:20:03.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:03.280+0000 7f45d4400700 1 -- 172.21.15.62:0/758683208 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45cc1a1190 con 0x7f45cc10c8f0 2024-04-20T22:20:03.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:03.281+0000 7f45ca000700 1 -- 172.21.15.62:0/758683208 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f45b800f5f0 con 0x7f45cc10c8f0 2024-04-20T22:20:03.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:03.281+0000 7f45d4400700 1 -- 172.21.15.62:0/758683208 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45cc1a16e0 con 0x7f45cc10c8f0 2024-04-20T22:20:03.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:03.282+0000 7f45d4400700 1 -- 172.21.15.62:0/758683208 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45cc109b20 con 0x7f45cc10c8f0 2024-04-20T22:20:03.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:03.283+0000 7f45ca000700 1 -- 172.21.15.62:0/758683208 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f45b80105d0 con 0x7f45cc10c8f0 2024-04-20T22:20:03.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:03.284+0000 7f45ca000700 1 --2- 172.21.15.62:0/758683208 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f45b406ecc0 0x7f45b4071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:20:03.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:03.284+0000 7f45ca000700 1 -- 172.21.15.62:0/758683208 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f45b808ef20 con 0x7f45cc10c8f0 2024-04-20T22:20:03.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:03.285+0000 7f45d1a00700 1 --2- 172.21.15.62:0/758683208 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f45b406ecc0 0x7f45b4071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:20:03.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:03.285+0000 7f45d1a00700 1 --2- 172.21.15.62:0/758683208 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f45b406ecc0 0x7f45b4071170 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f45c0000c00 tx=0x7f45c0005fd0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:20:03.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:03.289+0000 7f45ca000700 1 -- 172.21.15.62:0/758683208 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f45b8057c30 con 0x7f45cc10c8f0 2024-04-20T22:20:03.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:03.629+0000 7f45d4400700 1 -- 172.21.15.62:0/758683208 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f45cc04f2a0 con 0x7f45cc10c8f0 2024-04-20T22:20:03.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:03.630+0000 7f45ca000700 1 -- 172.21.15.62:0/758683208 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f45b805b2f0 con 0x7f45cc10c8f0 2024-04-20T22:20:03.631 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:20:03.631 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T22:20:03.631 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:20:03.631 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:20:03.631 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T22:20:03.632 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:20:03.632 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:20:03.632 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T22:20:03.632 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 6 2024-04-20T22:20:03.632 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:20:03.632 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T22:20:03.632 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 4 2024-04-20T22:20:03.633 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:20:03.633 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T22:20:03.633 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 14 2024-04-20T22:20:03.633 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T22:20:03.633 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:20:03.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:03.633+0000 7f45d4400700 1 -- 172.21.15.62:0/758683208 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f45b406ecc0 msgr2=0x7f45b4071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:20:03.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:03.633+0000 7f45d4400700 1 --2- 172.21.15.62:0/758683208 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f45b406ecc0 0x7f45b4071170 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f45c0000c00 tx=0x7f45c0005fd0 comp rx=0 tx=0).stop 2024-04-20T22:20:03.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:03.633+0000 7f45d4400700 1 -- 172.21.15.62:0/758683208 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f45cc10c8f0 msgr2=0x7f45cc19ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:20:03.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:03.633+0000 7f45d4400700 1 --2- 172.21.15.62:0/758683208 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f45cc10c8f0 0x7f45cc19ca40 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f45b800d8d0 tx=0x7f45b800dc90 comp rx=0 tx=0).stop 2024-04-20T22:20:03.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:03.633+0000 7f45d4400700 1 -- 172.21.15.62:0/758683208 shutdown_connections 2024-04-20T22:20:03.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:03.633+0000 7f45d4400700 1 --2- 172.21.15.62:0/758683208 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f45b406ecc0 0x7f45b4071170 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:20:03.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:03.633+0000 7f45d4400700 1 --2- 172.21.15.62:0/758683208 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45cc106b80 0x7f45cc19c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:20:03.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:03.634+0000 7f45d4400700 1 --2- 172.21.15.62:0/758683208 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f45cc10c8f0 0x7f45cc19ca40 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:20:03.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:03.634+0000 7f45d4400700 1 -- 172.21.15.62:0/758683208 >> 172.21.15.62:0/758683208 conn(0x7f45cc076060 msgr2=0x7f45cc077aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:20:03.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:03.634+0000 7f45d4400700 1 -- 172.21.15.62:0/758683208 shutdown_connections 2024-04-20T22:20:03.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:03.634+0000 7f45d4400700 1 -- 172.21.15.62:0/758683208 wait complete. 2024-04-20T22:20:03.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:20:03 smithi062 ceph-mon[76322]: from='client.36360 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:20:03.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:20:03 smithi062 ceph-mon[76322]: from='client.36364 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:20:03.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:20:03 smithi062 ceph-mon[76322]: pgmap v1346: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:20:03.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:03.803+0000 7f5dc0800700 1 -- 172.21.15.62:0/1771479371 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5db81004e0 msgr2=0x7f5db81008b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:20:03.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:03.803+0000 7f5dc0800700 1 --2- 172.21.15.62:0/1771479371 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5db81004e0 0x7f5db81008b0 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f5dac009910 tx=0x7f5dac009c20 comp rx=0 tx=0).stop 2024-04-20T22:20:03.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:03.803+0000 7f5dc0800700 1 -- 172.21.15.62:0/1771479371 shutdown_connections 2024-04-20T22:20:03.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:03.803+0000 7f5dc0800700 1 --2- 172.21.15.62:0/1771479371 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5db8100e80 0x7f5db8104e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:20:03.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:03.803+0000 7f5dc0800700 1 --2- 172.21.15.62:0/1771479371 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5db81004e0 0x7f5db81008b0 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:20:03.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:03.803+0000 7f5dc0800700 1 -- 172.21.15.62:0/1771479371 >> 172.21.15.62:0/1771479371 conn(0x7f5db80fbda0 msgr2=0x7f5db80fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:20:03.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:03.803+0000 7f5dc0800700 1 -- 172.21.15.62:0/1771479371 shutdown_connections 2024-04-20T22:20:03.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:03.803+0000 7f5dc0800700 1 -- 172.21.15.62:0/1771479371 wait complete. 2024-04-20T22:20:03.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:03.804+0000 7f5dc0800700 1 Processor -- start 2024-04-20T22:20:03.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:03.804+0000 7f5dc0800700 1 -- start start 2024-04-20T22:20:03.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:03.805+0000 7f5dc0800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5db81004e0 0x7f5db81980f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:20:03.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:03.805+0000 7f5dc0800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5db8100e80 0x7f5db8198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:20:03.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:03.805+0000 7f5dc0800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5db8198d10 con 0x7f5db81004e0 2024-04-20T22:20:03.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:03.805+0000 7f5dc0800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5db819caa0 con 0x7f5db8100e80 2024-04-20T22:20:03.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:03.805+0000 7f5dbde00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5db81004e0 0x7f5db81980f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:20:03.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:03.805+0000 7f5dbde00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5db81004e0 0x7f5db81980f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49916/0 (socket says 172.21.15.62:49916) 2024-04-20T22:20:03.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:03.805+0000 7f5dbde00700 1 -- 172.21.15.62:0/2856704261 learned_addr learned my addr 172.21.15.62:0/2856704261 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:20:03.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:03.806+0000 7f5dbd400700 1 --2- 172.21.15.62:0/2856704261 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5db8100e80 0x7f5db8198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:20:03.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:03.806+0000 7f5dbde00700 1 -- 172.21.15.62:0/2856704261 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5db8100e80 msgr2=0x7f5db8198630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:20:03.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:03.806+0000 7f5dbde00700 1 --2- 172.21.15.62:0/2856704261 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5db8100e80 0x7f5db8198630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:20:03.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:03.806+0000 7f5dbde00700 1 -- 172.21.15.62:0/2856704261 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dac0095f0 con 0x7f5db81004e0 2024-04-20T22:20:03.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:03.806+0000 7f5dbde00700 1 --2- 172.21.15.62:0/2856704261 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5db81004e0 0x7f5db81980f0 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f5dac0092e0 tx=0x7f5dac00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:20:03.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:03.806+0000 7f5db2a00700 1 -- 172.21.15.62:0/2856704261 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5dac01d070 con 0x7f5db81004e0 2024-04-20T22:20:03.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:03.806+0000 7f5db2a00700 1 -- 172.21.15.62:0/2856704261 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5dac005e00 con 0x7f5db81004e0 2024-04-20T22:20:03.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:03.806+0000 7f5db2a00700 1 -- 172.21.15.62:0/2856704261 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5dac017400 con 0x7f5db81004e0 2024-04-20T22:20:03.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:03.806+0000 7f5dc0800700 1 -- 172.21.15.62:0/2856704261 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5db819cd20 con 0x7f5db81004e0 2024-04-20T22:20:03.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:03.807+0000 7f5dc0800700 1 -- 172.21.15.62:0/2856704261 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5db819d210 con 0x7f5db81004e0 2024-04-20T22:20:03.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:03.808+0000 7f5dc0800700 1 -- 172.21.15.62:0/2856704261 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5db81045d0 con 0x7f5db81004e0 2024-04-20T22:20:03.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:03.809+0000 7f5db2a00700 1 -- 172.21.15.62:0/2856704261 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f5dac00f460 con 0x7f5db81004e0 2024-04-20T22:20:03.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:03.810+0000 7f5db2a00700 1 --2- 172.21.15.62:0/2856704261 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5da006ecc0 0x7f5da0071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:20:03.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:03.810+0000 7f5db2a00700 1 -- 172.21.15.62:0/2856704261 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f5dac090330 con 0x7f5db81004e0 2024-04-20T22:20:03.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:03.810+0000 7f5dbd400700 1 --2- 172.21.15.62:0/2856704261 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5da006ecc0 0x7f5da0071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:20:03.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:03.811+0000 7f5dbd400700 1 --2- 172.21.15.62:0/2856704261 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5da006ecc0 0x7f5da0071170 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f5da8005950 tx=0x7f5da80058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:20:03.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:03.815+0000 7f5db2a00700 1 -- 172.21.15.62:0/2856704261 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f5dac0590c0 con 0x7f5db81004e0 2024-04-20T22:20:03.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:20:03 smithi189 ceph-mon[57022]: from='client.36360 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:20:03.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:20:03 smithi189 ceph-mon[57022]: from='client.36364 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:20:03.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:20:03 smithi189 ceph-mon[57022]: pgmap v1346: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:20:04.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:04.104+0000 7f5dc0800700 1 -- 172.21.15.62:0/2856704261 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f5db804f2a0 con 0x7f5db81004e0 2024-04-20T22:20:04.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:04.105+0000 7f5db2a00700 1 -- 172.21.15.62:0/2856704261 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 29 v29) v1 ==== 76+0+1887 (secure 0 0 0) 0x7f5dac05c780 con 0x7f5db81004e0 2024-04-20T22:20:04.106 INFO:teuthology.orchestra.run.smithi062.stdout:e29 2024-04-20T22:20:04.106 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T22:20:04.107 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:20:04.107 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T22:20:04.107 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:20:04.107 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T22:20:04.107 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T22:20:04.107 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 20 2024-04-20T22:20:04.107 INFO:teuthology.orchestra.run.smithi062.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-20T22:20:04.107 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T22:20:04.107 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T21:42:01.559061+0000 2024-04-20T22:20:04.107 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T22:20:04.107 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T22:20:04.107 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T22:20:04.108 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T22:20:04.108 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T22:20:04.108 INFO:teuthology.orchestra.run.smithi062.stdout:max_xattr_size 65536 2024-04-20T22:20:04.108 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T22:20:04.108 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T22:20:04.108 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 81 2024-04-20T22:20:04.108 INFO:teuthology.orchestra.run.smithi062.stdout:snap_rstat 1 2024-04-20T22:20:04.108 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:20:04.108 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 2 2024-04-20T22:20:04.108 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1 2024-04-20T22:20:04.108 INFO:teuthology.orchestra.run.smithi062.stdout:up {} 2024-04-20T22:20:04.108 INFO:teuthology.orchestra.run.smithi062.stdout:failed 0,1 2024-04-20T22:20:04.108 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T22:20:04.108 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T22:20:04.108 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T22:20:04.109 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T22:20:04.109 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data enabled 2024-04-20T22:20:04.109 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T22:20:04.109 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T22:20:04.109 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 1 2024-04-20T22:20:04.109 INFO:teuthology.orchestra.run.smithi062.stdout:qdb_cluster leader: 0 members: 2024-04-20T22:20:04.109 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:20:04.109 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:20:04.109 INFO:teuthology.orchestra.run.smithi062.stdout:Standby daemons: 2024-04-20T22:20:04.109 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:20:04.109 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ryqblb{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6826/4113916205,v1:172.21.15.62:6827/4113916205] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:20:04.109 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ozizwi{-1:34476} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6828/3801975165,v1:172.21.15.62:6829/3801975165] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:20:04.109 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.dvimer{-1:44727} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6824/3576699232,v1:172.21.15.189:6825/3576699232] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:20:04.109 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.xbgspw{-1:45075} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6826/212905792,v1:172.21.15.189:6827/212905792] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:20:04.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:04.110+0000 7f5dc0800700 1 -- 172.21.15.62:0/2856704261 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5da006ecc0 msgr2=0x7f5da0071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:20:04.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:04.110+0000 7f5dc0800700 1 --2- 172.21.15.62:0/2856704261 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5da006ecc0 0x7f5da0071170 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f5da8005950 tx=0x7f5da80058e0 comp rx=0 tx=0).stop 2024-04-20T22:20:04.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:04.110+0000 7f5dc0800700 1 -- 172.21.15.62:0/2856704261 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5db81004e0 msgr2=0x7f5db81980f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:20:04.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:04.110+0000 7f5dc0800700 1 --2- 172.21.15.62:0/2856704261 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5db81004e0 0x7f5db81980f0 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f5dac0092e0 tx=0x7f5dac00be20 comp rx=0 tx=0).stop 2024-04-20T22:20:04.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:04.110+0000 7f5dc0800700 1 -- 172.21.15.62:0/2856704261 shutdown_connections 2024-04-20T22:20:04.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:04.110+0000 7f5dc0800700 1 --2- 172.21.15.62:0/2856704261 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5da006ecc0 0x7f5da0071170 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:20:04.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:04.110+0000 7f5dc0800700 1 --2- 172.21.15.62:0/2856704261 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5db81004e0 0x7f5db81980f0 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:20:04.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:04.110+0000 7f5dc0800700 1 --2- 172.21.15.62:0/2856704261 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5db8100e80 0x7f5db8198630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:20:04.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:04.110+0000 7f5dc0800700 1 -- 172.21.15.62:0/2856704261 >> 172.21.15.62:0/2856704261 conn(0x7f5db80fbda0 msgr2=0x7f5db80fe160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:20:04.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:04.110+0000 7f5dc0800700 1 -- 172.21.15.62:0/2856704261 shutdown_connections 2024-04-20T22:20:04.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:04.110+0000 7f5dc0800700 1 -- 172.21.15.62:0/2856704261 wait complete. 2024-04-20T22:20:04.112 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 29 2024-04-20T22:20:04.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:04.273+0000 7fa577800700 1 -- 172.21.15.62:0/1620176564 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa570106320 msgr2=0x7fa5701066f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:20:04.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:04.273+0000 7fa577800700 1 --2- 172.21.15.62:0/1620176564 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa570106320 0x7fa5701066f0 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7fa564009910 tx=0x7fa564009c20 comp rx=0 tx=0).stop 2024-04-20T22:20:04.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:04.273+0000 7fa577800700 1 -- 172.21.15.62:0/1620176564 shutdown_connections 2024-04-20T22:20:04.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:04.273+0000 7fa577800700 1 --2- 172.21.15.62:0/1620176564 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa570068540 0x7fa5700689b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:20:04.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:04.273+0000 7fa577800700 1 --2- 172.21.15.62:0/1620176564 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa570106320 0x7fa5701066f0 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:20:04.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:04.273+0000 7fa577800700 1 -- 172.21.15.62:0/1620176564 >> 172.21.15.62:0/1620176564 conn(0x7fa570075130 msgr2=0x7fa570075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:20:04.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:04.273+0000 7fa577800700 1 -- 172.21.15.62:0/1620176564 shutdown_connections 2024-04-20T22:20:04.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:04.273+0000 7fa577800700 1 -- 172.21.15.62:0/1620176564 wait complete. 2024-04-20T22:20:04.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:04.274+0000 7fa577800700 1 Processor -- start 2024-04-20T22:20:04.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:04.275+0000 7fa577800700 1 -- start start 2024-04-20T22:20:04.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:04.275+0000 7fa577800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa570068540 0x7fa570198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:20:04.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:04.275+0000 7fa577800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa570106320 0x7fa5701985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:20:04.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:04.275+0000 7fa577800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa570198bf0 con 0x7fa570106320 2024-04-20T22:20:04.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:04.275+0000 7fa577800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa570198d30 con 0x7fa570068540 2024-04-20T22:20:04.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:04.275+0000 7fa56fe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa570106320 0x7fa5701985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:20:04.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:04.275+0000 7fa56fe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa570106320 0x7fa5701985a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49946/0 (socket says 172.21.15.62:49946) 2024-04-20T22:20:04.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:04.276+0000 7fa56fe00700 1 -- 172.21.15.62:0/2834639854 learned_addr learned my addr 172.21.15.62:0/2834639854 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:20:04.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:04.276+0000 7fa574e00700 1 --2- 172.21.15.62:0/2834639854 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa570068540 0x7fa570198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:20:04.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:04.276+0000 7fa56fe00700 1 -- 172.21.15.62:0/2834639854 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa570068540 msgr2=0x7fa570198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:20:04.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:04.276+0000 7fa56fe00700 1 --2- 172.21.15.62:0/2834639854 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa570068540 0x7fa570198060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:20:04.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:04.276+0000 7fa56fe00700 1 -- 172.21.15.62:0/2834639854 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5640095f0 con 0x7fa570106320 2024-04-20T22:20:04.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:04.276+0000 7fa56fe00700 1 --2- 172.21.15.62:0/2834639854 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa570106320 0x7fa5701985a0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7fa55c00c930 tx=0x7fa55c00ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:20:04.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:04.277+0000 7fa56d600700 1 -- 172.21.15.62:0/2834639854 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa55c007ad0 con 0x7fa570106320 2024-04-20T22:20:04.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:04.277+0000 7fa56d600700 1 -- 172.21.15.62:0/2834639854 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa55c007c30 con 0x7fa570106320 2024-04-20T22:20:04.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:04.277+0000 7fa577800700 1 -- 172.21.15.62:0/2834639854 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa57019cb80 con 0x7fa570106320 2024-04-20T22:20:04.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:04.277+0000 7fa56d600700 1 -- 172.21.15.62:0/2834639854 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa55c018610 con 0x7fa570106320 2024-04-20T22:20:04.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:04.277+0000 7fa577800700 1 -- 172.21.15.62:0/2834639854 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa57019d0d0 con 0x7fa570106320 2024-04-20T22:20:04.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:04.278+0000 7fa577800700 1 -- 172.21.15.62:0/2834639854 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa57004f2a0 con 0x7fa570106320 2024-04-20T22:20:04.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:04.279+0000 7fa56d600700 1 -- 172.21.15.62:0/2834639854 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fa55c00f450 con 0x7fa570106320 2024-04-20T22:20:04.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:04.280+0000 7fa56d600700 1 --2- 172.21.15.62:0/2834639854 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa55806ecc0 0x7fa558071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:20:04.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:04.280+0000 7fa56d600700 1 -- 172.21.15.62:0/2834639854 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fa55c08f120 con 0x7fa570106320 2024-04-20T22:20:04.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:04.281+0000 7fa574e00700 1 --2- 172.21.15.62:0/2834639854 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa55806ecc0 0x7fa558071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:20:04.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:04.281+0000 7fa574e00700 1 --2- 172.21.15.62:0/2834639854 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa55806ecc0 0x7fa558071170 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fa564006010 tx=0x7fa56401a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:20:04.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:04.285+0000 7fa56d600700 1 -- 172.21.15.62:0/2834639854 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fa55c057e30 con 0x7fa570106320 2024-04-20T22:20:04.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:04.551+0000 7fa577800700 1 -- 172.21.15.62:0/2834639854 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa57019d430 con 0x7fa55806ecc0 2024-04-20T22:20:04.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:04.554+0000 7fa56d600700 1 -- 172.21.15.62:0/2834639854 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fa57019d430 con 0x7fa55806ecc0 2024-04-20T22:20:04.554 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:20:04.554 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T22:20:04.555 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T22:20:04.555 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-20T22:20:04.555 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T22:20:04.555 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr", 2024-04-20T22:20:04.555 INFO:teuthology.orchestra.run.smithi062.stdout: "osd", 2024-04-20T22:20:04.555 INFO:teuthology.orchestra.run.smithi062.stdout: "mds", 2024-04-20T22:20:04.555 INFO:teuthology.orchestra.run.smithi062.stdout: "crash", 2024-04-20T22:20:04.555 INFO:teuthology.orchestra.run.smithi062.stdout: "mon" 2024-04-20T22:20:04.555 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T22:20:04.555 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "16/23 daemons upgraded", 2024-04-20T22:20:04.555 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading mds daemons", 2024-04-20T22:20:04.555 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T22:20:04.555 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:20:04.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:04.557+0000 7fa577800700 1 -- 172.21.15.62:0/2834639854 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa55806ecc0 msgr2=0x7fa558071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:20:04.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:04.557+0000 7fa577800700 1 --2- 172.21.15.62:0/2834639854 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa55806ecc0 0x7fa558071170 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fa564006010 tx=0x7fa56401a040 comp rx=0 tx=0).stop 2024-04-20T22:20:04.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:04.557+0000 7fa577800700 1 -- 172.21.15.62:0/2834639854 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa570106320 msgr2=0x7fa5701985a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:20:04.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:04.557+0000 7fa577800700 1 --2- 172.21.15.62:0/2834639854 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa570106320 0x7fa5701985a0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7fa55c00c930 tx=0x7fa55c00ccf0 comp rx=0 tx=0).stop 2024-04-20T22:20:04.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:04.557+0000 7fa577800700 1 -- 172.21.15.62:0/2834639854 shutdown_connections 2024-04-20T22:20:04.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:04.557+0000 7fa577800700 1 --2- 172.21.15.62:0/2834639854 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa55806ecc0 0x7fa558071170 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:20:04.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:04.557+0000 7fa577800700 1 --2- 172.21.15.62:0/2834639854 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa570068540 0x7fa570198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:20:04.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:04.557+0000 7fa577800700 1 --2- 172.21.15.62:0/2834639854 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa570106320 0x7fa5701985a0 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:20:04.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:04.557+0000 7fa577800700 1 -- 172.21.15.62:0/2834639854 >> 172.21.15.62:0/2834639854 conn(0x7fa570075130 msgr2=0x7fa5700fffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:20:04.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:04.557+0000 7fa577800700 1 -- 172.21.15.62:0/2834639854 shutdown_connections 2024-04-20T22:20:04.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:04.557+0000 7fa577800700 1 -- 172.21.15.62:0/2834639854 wait complete. 2024-04-20T22:20:04.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:04.726+0000 7f3973200700 1 -- 172.21.15.62:0/3247198275 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39741011f0 msgr2=0x7f39741015c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:20:04.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:04.726+0000 7f3973200700 1 --2- 172.21.15.62:0/3247198275 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39741011f0 0x7f39741015c0 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f396400b0a0 tx=0x7f396400b3b0 comp rx=0 tx=0).stop 2024-04-20T22:20:04.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:04.726+0000 7f3973200700 1 -- 172.21.15.62:0/3247198275 shutdown_connections 2024-04-20T22:20:04.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:04.726+0000 7f3973200700 1 --2- 172.21.15.62:0/3247198275 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3974101b00 0x7f397410a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:20:04.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:04.726+0000 7f3973200700 1 --2- 172.21.15.62:0/3247198275 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39741011f0 0x7f39741015c0 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:20:04.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:04.726+0000 7f3973200700 1 -- 172.21.15.62:0/3247198275 >> 172.21.15.62:0/3247198275 conn(0x7f39740faa10 msgr2=0x7f39740fce20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:20:04.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:04.726+0000 7f3973200700 1 -- 172.21.15.62:0/3247198275 shutdown_connections 2024-04-20T22:20:04.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:04.726+0000 7f3973200700 1 -- 172.21.15.62:0/3247198275 wait complete. 2024-04-20T22:20:04.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:04.727+0000 7f3973200700 1 Processor -- start 2024-04-20T22:20:04.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:04.728+0000 7f3973200700 1 -- start start 2024-04-20T22:20:04.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:04.728+0000 7f3973200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39741011f0 0x7f397419b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:20:04.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:04.728+0000 7f3973200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3974101b00 0x7f397419bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:20:04.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:04.728+0000 7f3973200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f397419c5b0 con 0x7f3974101b00 2024-04-20T22:20:04.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:04.728+0000 7f3973200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3974195ab0 con 0x7f39741011f0 2024-04-20T22:20:04.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:04.728+0000 7f3969400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3974101b00 0x7f397419bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:20:04.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:04.728+0000 7f3969400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3974101b00 0x7f397419bf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49956/0 (socket says 172.21.15.62:49956) 2024-04-20T22:20:04.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:04.728+0000 7f3969400700 1 -- 172.21.15.62:0/2794564423 learned_addr learned my addr 172.21.15.62:0/2794564423 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:20:04.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:04.729+0000 7f3971e00700 1 --2- 172.21.15.62:0/2794564423 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39741011f0 0x7f397419b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:20:04.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:04.729+0000 7f3969400700 1 -- 172.21.15.62:0/2794564423 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39741011f0 msgr2=0x7f397419b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:20:04.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:04.729+0000 7f3969400700 1 --2- 172.21.15.62:0/2794564423 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39741011f0 0x7f397419b9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:20:04.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:04.729+0000 7f3969400700 1 -- 172.21.15.62:0/2794564423 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3964009d00 con 0x7f3974101b00 2024-04-20T22:20:04.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:04.729+0000 7f3971e00700 1 --2- 172.21.15.62:0/2794564423 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39741011f0 0x7f397419b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T22:20:04.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:04.729+0000 7f3969400700 1 --2- 172.21.15.62:0/2794564423 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3974101b00 0x7f397419bf20 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f395c00d8d0 tx=0x7f395c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:20:04.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:04.729+0000 7f396b400700 1 -- 172.21.15.62:0/2794564423 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f395c009940 con 0x7f3974101b00 2024-04-20T22:20:04.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:04.730+0000 7f396b400700 1 -- 172.21.15.62:0/2794564423 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f395c010460 con 0x7f3974101b00 2024-04-20T22:20:04.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:04.730+0000 7f3973200700 1 -- 172.21.15.62:0/2794564423 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3974195d90 con 0x7f3974101b00 2024-04-20T22:20:04.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:04.730+0000 7f396b400700 1 -- 172.21.15.62:0/2794564423 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f395c00f5f0 con 0x7f3974101b00 2024-04-20T22:20:04.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:04.730+0000 7f3973200700 1 -- 172.21.15.62:0/2794564423 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39741962e0 con 0x7f3974101b00 2024-04-20T22:20:04.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:04.731+0000 7f3973200700 1 -- 172.21.15.62:0/2794564423 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39741076d0 con 0x7f3974101b00 2024-04-20T22:20:04.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:04.732+0000 7f396b400700 1 -- 172.21.15.62:0/2794564423 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f395c0105d0 con 0x7f3974101b00 2024-04-20T22:20:04.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:04.733+0000 7f396b400700 1 --2- 172.21.15.62:0/2794564423 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f395806ecc0 0x7f3958071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:20:04.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:04.734+0000 7f396b400700 1 -- 172.21.15.62:0/2794564423 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f395c08ee10 con 0x7f3974101b00 2024-04-20T22:20:04.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:04.734+0000 7f3971e00700 1 --2- 172.21.15.62:0/2794564423 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f395806ecc0 0x7f3958071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:20:04.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:04.734+0000 7f3971e00700 1 --2- 172.21.15.62:0/2794564423 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f395806ecc0 0x7f3958071170 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f3964009480 tx=0x7f3964000f40 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:20:04.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:04.738+0000 7f396b400700 1 -- 172.21.15.62:0/2794564423 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f395c05b6d0 con 0x7f3974101b00 2024-04-20T22:20:04.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:20:04 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/758683208' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:20:04.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:20:04 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/2856704261' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:20:04.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:20:04 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:20:04.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:20:04 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/758683208' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:20:04.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:20:04 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/2856704261' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:20:04.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:20:04 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:20:05.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:05.074+0000 7f3973200700 1 -- 172.21.15.62:0/2794564423 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f3974062380 con 0x7f3974101b00 2024-04-20T22:20:05.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:05.075+0000 7f396b400700 1 -- 172.21.15.62:0/2794564423 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7f395c05b260 con 0x7f3974101b00 2024-04-20T22:20:05.076 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-20T22:20:05.076 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-20T22:20:05.076 INFO:teuthology.orchestra.run.smithi062.stdout: host smithi189 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi189/config 2024-04-20T22:20:05.076 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-20T22:20:05.076 INFO:teuthology.orchestra.run.smithi062.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi189 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp11670v2q:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:20:05.076 INFO:teuthology.orchestra.run.smithi062.stdout:Traceback (most recent call last): 2024-04-20T22:20:05.076 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-20T22:20:05.076 INFO:teuthology.orchestra.run.smithi062.stdout: return _run_code(code, main_globals, None, 2024-04-20T22:20:05.076 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-20T22:20:05.076 INFO:teuthology.orchestra.run.smithi062.stdout: exec(code, run_globals) 2024-04-20T22:20:05.076 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5618, in 2024-04-20T22:20:05.077 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5606, in main 2024-04-20T22:20:05.077 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 408, in _infer_config 2024-04-20T22:20:05.077 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 323, in _infer_fsid 2024-04-20T22:20:05.077 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 436, in _infer_image 2024-04-20T22:20:05.077 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 310, in _validate_fsid 2024-04-20T22:20:05.077 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 3359, in command_ceph_volume 2024-04-20T22:20:05.077 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-20T22:20:05.077 INFO:teuthology.orchestra.run.smithi062.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi189 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp11670v2q:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:20:05.077 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-20T22:20:05.077 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is degraded 2024-04-20T22:20:05.077 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T22:20:05.077 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T22:20:05.077 INFO:teuthology.orchestra.run.smithi062.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-20T22:20:05.077 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-20T22:20:05.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:05.078+0000 7f3973200700 1 -- 172.21.15.62:0/2794564423 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f395806ecc0 msgr2=0x7f3958071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:20:05.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:05.078+0000 7f3973200700 1 --2- 172.21.15.62:0/2794564423 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f395806ecc0 0x7f3958071170 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f3964009480 tx=0x7f3964000f40 comp rx=0 tx=0).stop 2024-04-20T22:20:05.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:05.078+0000 7f3973200700 1 -- 172.21.15.62:0/2794564423 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3974101b00 msgr2=0x7f397419bf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:20:05.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:05.078+0000 7f3973200700 1 --2- 172.21.15.62:0/2794564423 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3974101b00 0x7f397419bf20 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f395c00d8d0 tx=0x7f395c00dc90 comp rx=0 tx=0).stop 2024-04-20T22:20:05.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:05.079+0000 7f3973200700 1 -- 172.21.15.62:0/2794564423 shutdown_connections 2024-04-20T22:20:05.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:05.079+0000 7f3973200700 1 --2- 172.21.15.62:0/2794564423 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f395806ecc0 0x7f3958071170 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:20:05.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:05.079+0000 7f3973200700 1 --2- 172.21.15.62:0/2794564423 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39741011f0 0x7f397419b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:20:05.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:05.079+0000 7f3973200700 1 --2- 172.21.15.62:0/2794564423 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3974101b00 0x7f397419bf20 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:20:05.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:05.079+0000 7f3973200700 1 -- 172.21.15.62:0/2794564423 >> 172.21.15.62:0/2794564423 conn(0x7f39740faa10 msgr2=0x7f3974104810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:20:05.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:05.079+0000 7f3973200700 1 -- 172.21.15.62:0/2794564423 shutdown_connections 2024-04-20T22:20:05.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:05.079+0000 7f3973200700 1 -- 172.21.15.62:0/2794564423 wait complete. 2024-04-20T22:20:05.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:20:05 smithi062 ceph-mon[76322]: from='client.36376 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:20:05.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:20:05 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/2794564423' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:20:05.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:20:05 smithi062 ceph-mon[76322]: pgmap v1347: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:20:05.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:20:05 smithi189 ceph-mon[57022]: from='client.36376 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:20:05.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:20:05 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/2794564423' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:20:05.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:20:05 smithi189 ceph-mon[57022]: pgmap v1347: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:20:07.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:20:07 smithi189 ceph-mon[57022]: pgmap v1348: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:20:07.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:20:07 smithi062 ceph-mon[76322]: pgmap v1348: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:20:09.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:20:09 smithi189 ceph-mon[57022]: pgmap v1349: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:20:09.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:20:09 smithi062 ceph-mon[76322]: pgmap v1349: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:20:11.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:20:11 smithi189 ceph-mon[57022]: pgmap v1350: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:20:11.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:20:11 smithi062 ceph-mon[76322]: pgmap v1350: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:20:13.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:20:13 smithi189 ceph-mon[57022]: pgmap v1351: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:20:13.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:20:13 smithi062 ceph-mon[76322]: pgmap v1351: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:20:15.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:20:15 smithi189 ceph-mon[57022]: pgmap v1352: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:20:15.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:20:15 smithi062 ceph-mon[76322]: pgmap v1352: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:20:17.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:20:17 smithi189 ceph-mon[57022]: pgmap v1353: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:20:17.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:20:17 smithi062 ceph-mon[76322]: pgmap v1353: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:20:19.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:20:19 smithi189 ceph-mon[57022]: pgmap v1354: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:20:19.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:20:19 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:20:19.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:20:19 smithi062 ceph-mon[76322]: pgmap v1354: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:20:19.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:20:19 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:20:21.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:20:21 smithi189 ceph-mon[57022]: pgmap v1355: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:20:21.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:20:21 smithi062 ceph-mon[76322]: pgmap v1355: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:20:23.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:20:23 smithi189 ceph-mon[57022]: pgmap v1356: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:20:23.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:20:23 smithi062 ceph-mon[76322]: pgmap v1356: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:20:25.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:20:25 smithi189 ceph-mon[57022]: pgmap v1357: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:20:25.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:20:25 smithi062 ceph-mon[76322]: pgmap v1357: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:20:27.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:20:27 smithi062 ceph-mon[76322]: pgmap v1358: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:20:27.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:20:27 smithi189 ceph-mon[57022]: pgmap v1358: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:20:29.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:20:29 smithi062 ceph-mon[76322]: pgmap v1359: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:20:29.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:20:29 smithi189 ceph-mon[57022]: pgmap v1359: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:20:31.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:20:31 smithi062 ceph-mon[76322]: pgmap v1360: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:20:31.896 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:20:31 smithi189 ceph-mon[57022]: pgmap v1360: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:20:33.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:20:33 smithi062 ceph-mon[76322]: pgmap v1361: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:20:33.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:20:33 smithi189 ceph-mon[57022]: pgmap v1361: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:20:34.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:20:34 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:20:34.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:20:34 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:20:35.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:35.258+0000 7f2dc6000700 1 -- 172.21.15.62:0/1561282409 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2dc0102e70 msgr2=0x7f2dc0106e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:20:35.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:35.258+0000 7f2dc6000700 1 --2- 172.21.15.62:0/1561282409 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2dc0102e70 0x7f2dc0106e40 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f2dac009910 tx=0x7f2dac009c20 comp rx=0 tx=0).stop 2024-04-20T22:20:35.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:35.258+0000 7f2dc6000700 1 -- 172.21.15.62:0/1561282409 shutdown_connections 2024-04-20T22:20:35.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:35.258+0000 7f2dc6000700 1 --2- 172.21.15.62:0/1561282409 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2dc0102e70 0x7f2dc0106e40 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:20:35.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:35.258+0000 7f2dc6000700 1 --2- 172.21.15.62:0/1561282409 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2dc01024d0 0x7f2dc01028a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:20:35.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:35.258+0000 7f2dc6000700 1 -- 172.21.15.62:0/1561282409 >> 172.21.15.62:0/1561282409 conn(0x7f2dc00fdd90 msgr2=0x7f2dc01001a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:20:35.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:35.258+0000 7f2dc6000700 1 -- 172.21.15.62:0/1561282409 shutdown_connections 2024-04-20T22:20:35.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:35.258+0000 7f2dc6000700 1 -- 172.21.15.62:0/1561282409 wait complete. 2024-04-20T22:20:35.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:35.259+0000 7f2dc6000700 1 Processor -- start 2024-04-20T22:20:35.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:35.260+0000 7f2dc6000700 1 -- start start 2024-04-20T22:20:35.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:35.260+0000 7f2dc6000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2dc01024d0 0x7f2dc0197e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:20:35.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:35.260+0000 7f2dc6000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2dc0102e70 0x7f2dc0198350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:20:35.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:35.260+0000 7f2dc6000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dc0198a30 con 0x7f2dc0102e70 2024-04-20T22:20:35.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:35.260+0000 7f2dc6000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dc019c7c0 con 0x7f2dc01024d0 2024-04-20T22:20:35.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:35.260+0000 7f2dbfe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2dc0102e70 0x7f2dc0198350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:20:35.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:35.260+0000 7f2dbfe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2dc0102e70 0x7f2dc0198350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34908/0 (socket says 172.21.15.62:34908) 2024-04-20T22:20:35.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:35.260+0000 7f2dbfe00700 1 -- 172.21.15.62:0/3860342411 learned_addr learned my addr 172.21.15.62:0/3860342411 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:20:35.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:35.261+0000 7f2dc4c00700 1 --2- 172.21.15.62:0/3860342411 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2dc01024d0 0x7f2dc0197e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:20:35.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:35.261+0000 7f2dbfe00700 1 -- 172.21.15.62:0/3860342411 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2dc01024d0 msgr2=0x7f2dc0197e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:20:35.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:35.261+0000 7f2dbfe00700 1 --2- 172.21.15.62:0/3860342411 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2dc01024d0 0x7f2dc0197e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:20:35.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:35.261+0000 7f2dbfe00700 1 -- 172.21.15.62:0/3860342411 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dac0095f0 con 0x7f2dc0102e70 2024-04-20T22:20:35.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:35.261+0000 7f2dbfe00700 1 --2- 172.21.15.62:0/3860342411 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2dc0102e70 0x7f2dc0198350 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f2dac0092e0 tx=0x7f2dac00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:20:35.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:35.261+0000 7f2dbd600700 1 -- 172.21.15.62:0/3860342411 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2dac01d070 con 0x7f2dc0102e70 2024-04-20T22:20:35.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:35.261+0000 7f2dbd600700 1 -- 172.21.15.62:0/3860342411 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2dac005dc0 con 0x7f2dc0102e70 2024-04-20T22:20:35.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:35.261+0000 7f2dc6000700 1 -- 172.21.15.62:0/3860342411 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dc019ca40 con 0x7f2dc0102e70 2024-04-20T22:20:35.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:35.261+0000 7f2dbd600700 1 -- 172.21.15.62:0/3860342411 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2dac017400 con 0x7f2dc0102e70 2024-04-20T22:20:35.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:35.262+0000 7f2dc6000700 1 -- 172.21.15.62:0/3860342411 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dc019cf30 con 0x7f2dc0102e70 2024-04-20T22:20:35.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:35.263+0000 7f2dc6000700 1 -- 172.21.15.62:0/3860342411 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2dc010a840 con 0x7f2dc0102e70 2024-04-20T22:20:35.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:35.264+0000 7f2dbd600700 1 -- 172.21.15.62:0/3860342411 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f2dac00f460 con 0x7f2dc0102e70 2024-04-20T22:20:35.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:35.265+0000 7f2dbd600700 1 --2- 172.21.15.62:0/3860342411 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2da806ecc0 0x7f2da8071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:20:35.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:35.265+0000 7f2dbd600700 1 -- 172.21.15.62:0/3860342411 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f2dac090420 con 0x7f2dc0102e70 2024-04-20T22:20:35.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:35.265+0000 7f2dc4c00700 1 --2- 172.21.15.62:0/3860342411 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2da806ecc0 0x7f2da8071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:20:35.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:35.266+0000 7f2dc4c00700 1 --2- 172.21.15.62:0/3860342411 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2da806ecc0 0x7f2da8071170 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f2db4005950 tx=0x7f2db40058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:20:35.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:35.271+0000 7f2dbd600700 1 -- 172.21.15.62:0/3860342411 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f2dac059130 con 0x7f2dc0102e70 2024-04-20T22:20:35.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:20:35 smithi062 ceph-mon[76322]: pgmap v1362: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:20:35.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:35.536+0000 7f2dc6000700 1 -- 172.21.15.62:0/3860342411 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2dc002cca0 con 0x7f2da806ecc0 2024-04-20T22:20:35.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:35.539+0000 7f2dbd600700 1 -- 172.21.15.62:0/3860342411 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f2dc002cca0 con 0x7f2da806ecc0 2024-04-20T22:20:35.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:35.542+0000 7f2dc6000700 1 -- 172.21.15.62:0/3860342411 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2da806ecc0 msgr2=0x7f2da8071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:20:35.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:35.542+0000 7f2dc6000700 1 --2- 172.21.15.62:0/3860342411 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2da806ecc0 0x7f2da8071170 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f2db4005950 tx=0x7f2db40058e0 comp rx=0 tx=0).stop 2024-04-20T22:20:35.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:35.542+0000 7f2dc6000700 1 -- 172.21.15.62:0/3860342411 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2dc0102e70 msgr2=0x7f2dc0198350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:20:35.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:35.542+0000 7f2dc6000700 1 --2- 172.21.15.62:0/3860342411 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2dc0102e70 0x7f2dc0198350 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f2dac0092e0 tx=0x7f2dac00be20 comp rx=0 tx=0).stop 2024-04-20T22:20:35.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:35.542+0000 7f2dc6000700 1 -- 172.21.15.62:0/3860342411 shutdown_connections 2024-04-20T22:20:35.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:35.542+0000 7f2dc6000700 1 --2- 172.21.15.62:0/3860342411 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2da806ecc0 0x7f2da8071170 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:20:35.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:35.542+0000 7f2dc6000700 1 --2- 172.21.15.62:0/3860342411 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2dc01024d0 0x7f2dc0197e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:20:35.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:35.542+0000 7f2dc6000700 1 --2- 172.21.15.62:0/3860342411 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2dc0102e70 0x7f2dc0198350 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:20:35.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:35.542+0000 7f2dc6000700 1 -- 172.21.15.62:0/3860342411 >> 172.21.15.62:0/3860342411 conn(0x7f2dc00fdd90 msgr2=0x7f2dc0100180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:20:35.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:35.543+0000 7f2dc6000700 1 -- 172.21.15.62:0/3860342411 shutdown_connections 2024-04-20T22:20:35.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:35.543+0000 7f2dc6000700 1 -- 172.21.15.62:0/3860342411 wait complete. 2024-04-20T22:20:35.559 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T22:20:35.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:35.713+0000 7f4af5600700 1 -- 172.21.15.62:0/3480176342 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4af0102470 msgr2=0x7f4af01028e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:20:35.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:35.713+0000 7f4af5600700 1 --2- 172.21.15.62:0/3480176342 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4af0102470 0x7f4af01028e0 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f4ad8009910 tx=0x7f4ad8009c20 comp rx=0 tx=0).stop 2024-04-20T22:20:35.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:35.713+0000 7f4af5600700 1 -- 172.21.15.62:0/3480176342 shutdown_connections 2024-04-20T22:20:35.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:35.713+0000 7f4af5600700 1 --2- 172.21.15.62:0/3480176342 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4af0102470 0x7f4af01028e0 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:20:35.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:35.713+0000 7f4af5600700 1 --2- 172.21.15.62:0/3480176342 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4af0108470 0x7f4af0108840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:20:35.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:35.713+0000 7f4af5600700 1 -- 172.21.15.62:0/3480176342 >> 172.21.15.62:0/3480176342 conn(0x7f4af00fdf90 msgr2=0x7f4af01003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:20:35.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:35.713+0000 7f4af5600700 1 -- 172.21.15.62:0/3480176342 shutdown_connections 2024-04-20T22:20:35.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:35.713+0000 7f4af5600700 1 -- 172.21.15.62:0/3480176342 wait complete. 2024-04-20T22:20:35.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:35.715+0000 7f4af5600700 1 Processor -- start 2024-04-20T22:20:35.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:35.715+0000 7f4af5600700 1 -- start start 2024-04-20T22:20:35.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:35.715+0000 7f4af5600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4af0102470 0x7f4af0198270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:20:35.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:35.715+0000 7f4af5600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4af0108470 0x7f4af01987b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:20:35.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:35.715+0000 7f4af5600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4af0198e90 con 0x7f4af0102470 2024-04-20T22:20:35.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:35.715+0000 7f4af5600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4af019cc20 con 0x7f4af0108470 2024-04-20T22:20:35.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:35.715+0000 7f4aeea00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4af0102470 0x7f4af0198270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:20:35.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:35.715+0000 7f4aeea00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4af0102470 0x7f4af0198270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34922/0 (socket says 172.21.15.62:34922) 2024-04-20T22:20:35.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:35.716+0000 7f4aeea00700 1 -- 172.21.15.62:0/2517689921 learned_addr learned my addr 172.21.15.62:0/2517689921 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:20:35.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:35.716+0000 7f4ae7e00700 1 --2- 172.21.15.62:0/2517689921 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4af0108470 0x7f4af01987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:20:35.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:35.716+0000 7f4aeea00700 1 -- 172.21.15.62:0/2517689921 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4af0108470 msgr2=0x7f4af01987b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:20:35.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:35.716+0000 7f4aeea00700 1 --2- 172.21.15.62:0/2517689921 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4af0108470 0x7f4af01987b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:20:35.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:35.716+0000 7f4aeea00700 1 -- 172.21.15.62:0/2517689921 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ad80095f0 con 0x7f4af0102470 2024-04-20T22:20:35.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:35.716+0000 7f4aeea00700 1 --2- 172.21.15.62:0/2517689921 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4af0102470 0x7f4af0198270 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f4ae000d8d0 tx=0x7f4ae000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:20:35.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:35.717+0000 7f4ae7400700 1 -- 172.21.15.62:0/2517689921 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4ae0009940 con 0x7f4af0102470 2024-04-20T22:20:35.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:35.717+0000 7f4ae7400700 1 -- 172.21.15.62:0/2517689921 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4ae0010460 con 0x7f4af0102470 2024-04-20T22:20:35.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:35.717+0000 7f4ae7400700 1 -- 172.21.15.62:0/2517689921 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4ae000f5f0 con 0x7f4af0102470 2024-04-20T22:20:35.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:35.717+0000 7f4af5600700 1 -- 172.21.15.62:0/2517689921 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4af019cf00 con 0x7f4af0102470 2024-04-20T22:20:35.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:35.717+0000 7f4af5600700 1 -- 172.21.15.62:0/2517689921 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4af019d450 con 0x7f4af0102470 2024-04-20T22:20:35.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:35.718+0000 7f4af5600700 1 -- 172.21.15.62:0/2517689921 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4af010a970 con 0x7f4af0102470 2024-04-20T22:20:35.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:35.719+0000 7f4ae7400700 1 -- 172.21.15.62:0/2517689921 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f4ae00105d0 con 0x7f4af0102470 2024-04-20T22:20:35.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:35.720+0000 7f4ae7400700 1 --2- 172.21.15.62:0/2517689921 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f4ad406ecc0 0x7f4ad4071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:20:35.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:35.720+0000 7f4ae7400700 1 -- 172.21.15.62:0/2517689921 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f4ae008ee10 con 0x7f4af0102470 2024-04-20T22:20:35.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:35.720+0000 7f4ae7e00700 1 --2- 172.21.15.62:0/2517689921 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f4ad406ecc0 0x7f4ad4071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:20:35.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:35.721+0000 7f4ae7e00700 1 --2- 172.21.15.62:0/2517689921 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f4ad406ecc0 0x7f4ad4071170 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f4ad8006010 tx=0x7f4ad800b540 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:20:35.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:35.725+0000 7f4ae7400700 1 -- 172.21.15.62:0/2517689921 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f4ae0057ba0 con 0x7f4af0102470 2024-04-20T22:20:35.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:35.980+0000 7f4af5600700 1 -- 172.21.15.62:0/2517689921 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4af019d6c0 con 0x7f4ad406ecc0 2024-04-20T22:20:35.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:35.983+0000 7f4ae7400700 1 -- 172.21.15.62:0/2517689921 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f4af019d6c0 con 0x7f4ad406ecc0 2024-04-20T22:20:35.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:35.986+0000 7f4af5600700 1 -- 172.21.15.62:0/2517689921 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f4ad406ecc0 msgr2=0x7f4ad4071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:20:35.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:35.987+0000 7f4af5600700 1 --2- 172.21.15.62:0/2517689921 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f4ad406ecc0 0x7f4ad4071170 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f4ad8006010 tx=0x7f4ad800b540 comp rx=0 tx=0).stop 2024-04-20T22:20:35.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:35.987+0000 7f4af5600700 1 -- 172.21.15.62:0/2517689921 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4af0102470 msgr2=0x7f4af0198270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:20:35.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:35.987+0000 7f4af5600700 1 --2- 172.21.15.62:0/2517689921 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4af0102470 0x7f4af0198270 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f4ae000d8d0 tx=0x7f4ae000dc90 comp rx=0 tx=0).stop 2024-04-20T22:20:35.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:35.987+0000 7f4af5600700 1 -- 172.21.15.62:0/2517689921 shutdown_connections 2024-04-20T22:20:35.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:35.987+0000 7f4af5600700 1 --2- 172.21.15.62:0/2517689921 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f4ad406ecc0 0x7f4ad4071170 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:20:35.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:35.987+0000 7f4af5600700 1 --2- 172.21.15.62:0/2517689921 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4af0102470 0x7f4af0198270 unknown :-1 s=CLOSED pgs=1158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:20:35.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:35.987+0000 7f4af5600700 1 --2- 172.21.15.62:0/2517689921 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4af0108470 0x7f4af01987b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:20:35.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:35.987+0000 7f4af5600700 1 -- 172.21.15.62:0/2517689921 >> 172.21.15.62:0/2517689921 conn(0x7f4af00fdf90 msgr2=0x7f4af00ff410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:20:35.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:35.987+0000 7f4af5600700 1 -- 172.21.15.62:0/2517689921 shutdown_connections 2024-04-20T22:20:35.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:35.987+0000 7f4af5600700 1 -- 172.21.15.62:0/2517689921 wait complete. 2024-04-20T22:20:35.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:20:35 smithi189 ceph-mon[57022]: pgmap v1362: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:20:36.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:36.152+0000 7ff923200700 1 -- 172.21.15.62:0/345302098 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff924103e50 msgr2=0x7ff9241042c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:20:36.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:36.152+0000 7ff923200700 1 --2- 172.21.15.62:0/345302098 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff924103e50 0x7ff9241042c0 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7ff910009910 tx=0x7ff910009c20 comp rx=0 tx=0).stop 2024-04-20T22:20:36.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:36.152+0000 7ff923200700 1 -- 172.21.15.62:0/345302098 shutdown_connections 2024-04-20T22:20:36.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:36.152+0000 7ff923200700 1 --2- 172.21.15.62:0/345302098 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff924103e50 0x7ff9241042c0 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:20:36.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:36.152+0000 7ff923200700 1 --2- 172.21.15.62:0/345302098 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff9241025f0 0x7ff9241029c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:20:36.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:36.152+0000 7ff923200700 1 -- 172.21.15.62:0/345302098 >> 172.21.15.62:0/345302098 conn(0x7ff9240fddb0 msgr2=0x7ff9241001c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:20:36.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:36.152+0000 7ff923200700 1 -- 172.21.15.62:0/345302098 shutdown_connections 2024-04-20T22:20:36.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:36.152+0000 7ff923200700 1 -- 172.21.15.62:0/345302098 wait complete. 2024-04-20T22:20:36.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:36.153+0000 7ff923200700 1 Processor -- start 2024-04-20T22:20:36.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:36.153+0000 7ff923200700 1 -- start start 2024-04-20T22:20:36.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:36.154+0000 7ff923200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff9241025f0 0x7ff924197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:20:36.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:36.154+0000 7ff923200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff924103e50 0x7ff924198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:20:36.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:36.154+0000 7ff923200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff924198af0 con 0x7ff924103e50 2024-04-20T22:20:36.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:36.154+0000 7ff923200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff92419c880 con 0x7ff9241025f0 2024-04-20T22:20:36.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:36.154+0000 7ff921400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff924103e50 0x7ff924198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:20:36.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:36.154+0000 7ff921400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff924103e50 0x7ff924198410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34940/0 (socket says 172.21.15.62:34940) 2024-04-20T22:20:36.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:36.154+0000 7ff921400700 1 -- 172.21.15.62:0/3227024310 learned_addr learned my addr 172.21.15.62:0/3227024310 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:20:36.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:36.154+0000 7ff921e00700 1 --2- 172.21.15.62:0/3227024310 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff9241025f0 0x7ff924197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:20:36.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:36.154+0000 7ff921400700 1 -- 172.21.15.62:0/3227024310 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff9241025f0 msgr2=0x7ff924197ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:20:36.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:36.154+0000 7ff921400700 1 --2- 172.21.15.62:0/3227024310 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff9241025f0 0x7ff924197ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:20:36.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:36.154+0000 7ff921400700 1 -- 172.21.15.62:0/3227024310 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9100095f0 con 0x7ff924103e50 2024-04-20T22:20:36.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:36.155+0000 7ff921400700 1 --2- 172.21.15.62:0/3227024310 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff924103e50 0x7ff924198410 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7ff910006010 tx=0x7ff91000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:20:36.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:36.155+0000 7ff91aa00700 1 -- 172.21.15.62:0/3227024310 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff91001d070 con 0x7ff924103e50 2024-04-20T22:20:36.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:36.155+0000 7ff91aa00700 1 -- 172.21.15.62:0/3227024310 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff910004fb0 con 0x7ff924103e50 2024-04-20T22:20:36.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:36.155+0000 7ff91aa00700 1 -- 172.21.15.62:0/3227024310 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff91000f690 con 0x7ff924103e50 2024-04-20T22:20:36.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:36.155+0000 7ff923200700 1 -- 172.21.15.62:0/3227024310 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff92419cb00 con 0x7ff924103e50 2024-04-20T22:20:36.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:36.155+0000 7ff923200700 1 -- 172.21.15.62:0/3227024310 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff92419d070 con 0x7ff924103e50 2024-04-20T22:20:36.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:36.156+0000 7ff923200700 1 -- 172.21.15.62:0/3227024310 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff924062380 con 0x7ff924103e50 2024-04-20T22:20:36.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:36.158+0000 7ff91aa00700 1 -- 172.21.15.62:0/3227024310 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff9100228a0 con 0x7ff924103e50 2024-04-20T22:20:36.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:36.158+0000 7ff91aa00700 1 --2- 172.21.15.62:0/3227024310 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff90806ecc0 0x7ff908071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:20:36.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:36.159+0000 7ff91aa00700 1 -- 172.21.15.62:0/3227024310 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7ff910090700 con 0x7ff924103e50 2024-04-20T22:20:36.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:36.159+0000 7ff921e00700 1 --2- 172.21.15.62:0/3227024310 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff90806ecc0 0x7ff908071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:20:36.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:36.159+0000 7ff921e00700 1 --2- 172.21.15.62:0/3227024310 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff90806ecc0 0x7ff908071170 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7ff914005fd0 tx=0x7ff91400a560 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:20:36.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:36.163+0000 7ff91aa00700 1 -- 172.21.15.62:0/3227024310 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7ff910059490 con 0x7ff924103e50 2024-04-20T22:20:36.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:36.408+0000 7ff923200700 1 -- 172.21.15.62:0/3227024310 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7ff92419d410 con 0x7ff90806ecc0 2024-04-20T22:20:36.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:36.439+0000 7ff91aa00700 1 -- 172.21.15.62:0/3227024310 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7ff92419d410 con 0x7ff90806ecc0 2024-04-20T22:20:36.439 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T22:20:36.439 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (46m) 7m ago 60m 14.6M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T22:20:36.439 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (60m) 7m ago 60m 11.6M - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T22:20:36.439 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (59m) 7m ago 59m 11.2M - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T22:20:36.440 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (42m) 7m ago 60m 6660k - 19.0.0-3133-ga8c84f59 76bc6b91b726 2825d5ba11c6 2024-04-20T22:20:36.440 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (42m) 7m ago 59m 6665k - 19.0.0-3133-ga8c84f59 76bc6b91b726 20f746fa1bba 2024-04-20T22:20:36.440 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (45m) 7m ago 59m 75.5M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T22:20:36.440 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (37m) 7m ago 53m 22.9M - 19.0.0-3133-ga8c84f59 76bc6b91b726 de80bc90a683 2024-04-20T22:20:36.440 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (38m) 7m ago 53m 23.0M - 19.0.0-3133-ga8c84f59 76bc6b91b726 cd793b7ee98e 2024-04-20T22:20:36.440 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (22m) 7m ago 53m 18.5M - 19.0.0-3133-ga8c84f59 76bc6b91b726 0e7b0bc4d1ab 2024-04-20T22:20:36.440 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (7m) 7m ago 52m 11.4M - 19.0.0-3133-ga8c84f59 76bc6b91b726 0af732ac9e06 2024-04-20T22:20:36.440 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (49m) 7m ago 62m 517M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T22:20:36.440 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (48m) 7m ago 59m 460M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T22:20:36.440 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (44m) 7m ago 62m 86.3M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 019eb1479485 2024-04-20T22:20:36.440 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (43m) 7m ago 58m 58.0M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 55aac560aab6 2024-04-20T22:20:36.441 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (47m) 7m ago 60m 8766k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T22:20:36.441 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (47m) 7m ago 59m 8711k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T22:20:36.441 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (41m) 7m ago 57m 91.3M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 ee2bb901f26e 2024-04-20T22:20:36.441 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (41m) 7m ago 57m 94.9M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 9075152d4162 2024-04-20T22:20:36.441 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (40m) 7m ago 56m 78.5M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 2d4f3e38dc0e 2024-04-20T22:20:36.441 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (40m) 7m ago 56m 94.1M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 8f97b17ae0eb 2024-04-20T22:20:36.441 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (39m) 7m ago 55m 96.1M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 653d8079967d 2024-04-20T22:20:36.441 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (38m) 7m ago 54m 90.9M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 596dded14c12 2024-04-20T22:20:36.441 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (47m) 7m ago 59m 90.4M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T22:20:36.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:36.441+0000 7ff923200700 1 -- 172.21.15.62:0/3227024310 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff90806ecc0 msgr2=0x7ff908071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:20:36.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:36.441+0000 7ff923200700 1 --2- 172.21.15.62:0/3227024310 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff90806ecc0 0x7ff908071170 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7ff914005fd0 tx=0x7ff91400a560 comp rx=0 tx=0).stop 2024-04-20T22:20:36.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:36.441+0000 7ff923200700 1 -- 172.21.15.62:0/3227024310 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff924103e50 msgr2=0x7ff924198410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:20:36.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:36.441+0000 7ff923200700 1 --2- 172.21.15.62:0/3227024310 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff924103e50 0x7ff924198410 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7ff910006010 tx=0x7ff91000be20 comp rx=0 tx=0).stop 2024-04-20T22:20:36.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:36.442+0000 7ff923200700 1 -- 172.21.15.62:0/3227024310 shutdown_connections 2024-04-20T22:20:36.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:36.442+0000 7ff923200700 1 --2- 172.21.15.62:0/3227024310 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff90806ecc0 0x7ff908071170 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:20:36.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:36.442+0000 7ff923200700 1 --2- 172.21.15.62:0/3227024310 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff9241025f0 0x7ff924197ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:20:36.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:36.442+0000 7ff923200700 1 --2- 172.21.15.62:0/3227024310 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff924103e50 0x7ff924198410 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:20:36.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:36.442+0000 7ff923200700 1 -- 172.21.15.62:0/3227024310 >> 172.21.15.62:0/3227024310 conn(0x7ff9240fddb0 msgr2=0x7ff9240fe680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:20:36.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:36.442+0000 7ff923200700 1 -- 172.21.15.62:0/3227024310 shutdown_connections 2024-04-20T22:20:36.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:36.442+0000 7ff923200700 1 -- 172.21.15.62:0/3227024310 wait complete. 2024-04-20T22:20:36.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:36.598+0000 7f2e80a00700 1 -- 172.21.15.62:0/1458857092 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e7c0739a0 msgr2=0x7f2e7c10c5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:20:36.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:36.598+0000 7f2e80a00700 1 --2- 172.21.15.62:0/1458857092 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e7c0739a0 0x7f2e7c10c5e0 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f2e68009940 tx=0x7f2e68009c50 comp rx=0 tx=0).stop 2024-04-20T22:20:36.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:36.599+0000 7f2e80a00700 1 -- 172.21.15.62:0/1458857092 shutdown_connections 2024-04-20T22:20:36.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:36.599+0000 7f2e80a00700 1 --2- 172.21.15.62:0/1458857092 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e7c0739a0 0x7f2e7c10c5e0 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:20:36.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:36.599+0000 7f2e80a00700 1 --2- 172.21.15.62:0/1458857092 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e7c073090 0x7f2e7c073460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:20:36.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:36.599+0000 7f2e80a00700 1 -- 172.21.15.62:0/1458857092 >> 172.21.15.62:0/1458857092 conn(0x7f2e7c0fbda0 msgr2=0x7f2e7c0fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:20:36.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:36.599+0000 7f2e80a00700 1 -- 172.21.15.62:0/1458857092 shutdown_connections 2024-04-20T22:20:36.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:36.599+0000 7f2e80a00700 1 -- 172.21.15.62:0/1458857092 wait complete. 2024-04-20T22:20:36.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:36.600+0000 7f2e80a00700 1 Processor -- start 2024-04-20T22:20:36.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:36.600+0000 7f2e80a00700 1 -- start start 2024-04-20T22:20:36.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:36.600+0000 7f2e80a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e7c073090 0x7f2e7c198220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:20:36.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:36.600+0000 7f2e80a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e7c0739a0 0x7f2e7c198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:20:36.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:36.600+0000 7f2e80a00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e7c198e40 con 0x7f2e7c073090 2024-04-20T22:20:36.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:36.600+0000 7f2e80a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e7c19cbd0 con 0x7f2e7c0739a0 2024-04-20T22:20:36.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:36.601+0000 7f2e79e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e7c073090 0x7f2e7c198220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:20:36.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:36.601+0000 7f2e79e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e7c073090 0x7f2e7c198220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34958/0 (socket says 172.21.15.62:34958) 2024-04-20T22:20:36.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:36.601+0000 7f2e79e00700 1 -- 172.21.15.62:0/3228821770 learned_addr learned my addr 172.21.15.62:0/3228821770 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:20:36.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:36.601+0000 7f2e79400700 1 --2- 172.21.15.62:0/3228821770 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e7c0739a0 0x7f2e7c198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:20:36.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:36.601+0000 7f2e79e00700 1 -- 172.21.15.62:0/3228821770 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e7c0739a0 msgr2=0x7f2e7c198760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:20:36.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:36.601+0000 7f2e79e00700 1 --2- 172.21.15.62:0/3228821770 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e7c0739a0 0x7f2e7c198760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:20:36.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:36.601+0000 7f2e79e00700 1 -- 172.21.15.62:0/3228821770 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e680095f0 con 0x7f2e7c073090 2024-04-20T22:20:36.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:36.602+0000 7f2e79e00700 1 --2- 172.21.15.62:0/3228821770 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e7c073090 0x7f2e7c198220 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f2e6c00d8d0 tx=0x7f2e6c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:20:36.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:36.602+0000 7f2e72a00700 1 -- 172.21.15.62:0/3228821770 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2e6c009940 con 0x7f2e7c073090 2024-04-20T22:20:36.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:36.602+0000 7f2e72a00700 1 -- 172.21.15.62:0/3228821770 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2e6c010460 con 0x7f2e7c073090 2024-04-20T22:20:36.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:36.602+0000 7f2e72a00700 1 -- 172.21.15.62:0/3228821770 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2e6c00f5f0 con 0x7f2e7c073090 2024-04-20T22:20:36.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:36.602+0000 7f2e80a00700 1 -- 172.21.15.62:0/3228821770 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e7c19ceb0 con 0x7f2e7c073090 2024-04-20T22:20:36.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:36.602+0000 7f2e80a00700 1 -- 172.21.15.62:0/3228821770 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e7c19d400 con 0x7f2e7c073090 2024-04-20T22:20:36.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:36.603+0000 7f2e80a00700 1 -- 172.21.15.62:0/3228821770 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e7c109ce0 con 0x7f2e7c073090 2024-04-20T22:20:36.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:36.604+0000 7f2e72a00700 1 -- 172.21.15.62:0/3228821770 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f2e6c0105d0 con 0x7f2e7c073090 2024-04-20T22:20:36.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:36.605+0000 7f2e72a00700 1 --2- 172.21.15.62:0/3228821770 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2e6006ecc0 0x7f2e60071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:20:36.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:36.605+0000 7f2e72a00700 1 -- 172.21.15.62:0/3228821770 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f2e6c08ee10 con 0x7f2e7c073090 2024-04-20T22:20:36.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:36.605+0000 7f2e79400700 1 --2- 172.21.15.62:0/3228821770 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2e6006ecc0 0x7f2e60071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:20:36.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:36.606+0000 7f2e79400700 1 --2- 172.21.15.62:0/3228821770 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2e6006ecc0 0x7f2e60071170 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f2e68005b40 tx=0x7f2e68005a90 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:20:36.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:36.610+0000 7f2e72a00700 1 -- 172.21.15.62:0/3228821770 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f2e6c057ba0 con 0x7f2e7c073090 2024-04-20T22:20:36.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:20:36 smithi062 ceph-mon[76322]: from='client.36384 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:20:36.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:20:36 smithi062 ceph-mon[76322]: from='client.36388 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:20:36.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:20:36 smithi062 ceph-mon[76322]: from='client.36392 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:20:36.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:36.937+0000 7f2e80a00700 1 -- 172.21.15.62:0/3228821770 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f2e7c199650 con 0x7f2e7c073090 2024-04-20T22:20:36.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:36.938+0000 7f2e72a00700 1 -- 172.21.15.62:0/3228821770 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f2e6c05b260 con 0x7f2e7c073090 2024-04-20T22:20:36.938 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:20:36.938 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T22:20:36.939 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:20:36.939 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:20:36.939 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T22:20:36.939 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:20:36.939 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:20:36.939 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T22:20:36.939 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 6 2024-04-20T22:20:36.939 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:20:36.939 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T22:20:36.939 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 4 2024-04-20T22:20:36.939 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:20:36.939 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T22:20:36.940 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 14 2024-04-20T22:20:36.940 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T22:20:36.940 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:20:36.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:36.941+0000 7f2e80a00700 1 -- 172.21.15.62:0/3228821770 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2e6006ecc0 msgr2=0x7f2e60071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:20:36.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:36.941+0000 7f2e80a00700 1 --2- 172.21.15.62:0/3228821770 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2e6006ecc0 0x7f2e60071170 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f2e68005b40 tx=0x7f2e68005a90 comp rx=0 tx=0).stop 2024-04-20T22:20:36.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:36.941+0000 7f2e80a00700 1 -- 172.21.15.62:0/3228821770 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e7c073090 msgr2=0x7f2e7c198220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:20:36.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:36.941+0000 7f2e80a00700 1 --2- 172.21.15.62:0/3228821770 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e7c073090 0x7f2e7c198220 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f2e6c00d8d0 tx=0x7f2e6c00dc90 comp rx=0 tx=0).stop 2024-04-20T22:20:36.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:36.942+0000 7f2e80a00700 1 -- 172.21.15.62:0/3228821770 shutdown_connections 2024-04-20T22:20:36.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:36.942+0000 7f2e80a00700 1 --2- 172.21.15.62:0/3228821770 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2e6006ecc0 0x7f2e60071170 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:20:36.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:36.942+0000 7f2e80a00700 1 --2- 172.21.15.62:0/3228821770 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e7c073090 0x7f2e7c198220 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:20:36.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:36.942+0000 7f2e80a00700 1 --2- 172.21.15.62:0/3228821770 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e7c0739a0 0x7f2e7c198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:20:36.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:36.942+0000 7f2e80a00700 1 -- 172.21.15.62:0/3228821770 >> 172.21.15.62:0/3228821770 conn(0x7f2e7c0fbda0 msgr2=0x7f2e7c106e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:20:36.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:36.942+0000 7f2e80a00700 1 -- 172.21.15.62:0/3228821770 shutdown_connections 2024-04-20T22:20:36.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:36.942+0000 7f2e80a00700 1 -- 172.21.15.62:0/3228821770 wait complete. 2024-04-20T22:20:36.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:20:36 smithi189 ceph-mon[57022]: from='client.36384 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:20:36.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:20:36 smithi189 ceph-mon[57022]: from='client.36388 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:20:36.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:20:36 smithi189 ceph-mon[57022]: from='client.36392 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:20:37.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.100+0000 7f2776600700 1 -- 172.21.15.62:0/2677307704 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2770073980 msgr2=0x7f277010c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:20:37.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.100+0000 7f2776600700 1 --2- 172.21.15.62:0/2677307704 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2770073980 0x7f277010c600 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f275c009940 tx=0x7f275c009c50 comp rx=0 tx=0).stop 2024-04-20T22:20:37.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.100+0000 7f2776600700 1 -- 172.21.15.62:0/2677307704 shutdown_connections 2024-04-20T22:20:37.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.100+0000 7f2776600700 1 --2- 172.21.15.62:0/2677307704 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2770073980 0x7f277010c600 unknown :-1 s=CLOSED pgs=1163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:20:37.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.100+0000 7f2776600700 1 --2- 172.21.15.62:0/2677307704 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2770073070 0x7f2770073440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:20:37.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.100+0000 7f2776600700 1 -- 172.21.15.62:0/2677307704 >> 172.21.15.62:0/2677307704 conn(0x7f27700fbda0 msgr2=0x7f27700fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:20:37.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.100+0000 7f2776600700 1 -- 172.21.15.62:0/2677307704 shutdown_connections 2024-04-20T22:20:37.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.100+0000 7f2776600700 1 -- 172.21.15.62:0/2677307704 wait complete. 2024-04-20T22:20:37.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.101+0000 7f2776600700 1 Processor -- start 2024-04-20T22:20:37.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.101+0000 7f2776600700 1 -- start start 2024-04-20T22:20:37.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.102+0000 7f2776600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2770073070 0x7f27701981b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:20:37.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.102+0000 7f2776600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2770073980 0x7f27701986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:20:37.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.102+0000 7f2776600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2770198dd0 con 0x7f2770073070 2024-04-20T22:20:37.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.102+0000 7f2776600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f277019cb60 con 0x7f2770073980 2024-04-20T22:20:37.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.102+0000 7f276f400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2770073070 0x7f27701981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:20:37.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.102+0000 7f276f400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2770073070 0x7f27701981b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34968/0 (socket says 172.21.15.62:34968) 2024-04-20T22:20:37.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.102+0000 7f276f400700 1 -- 172.21.15.62:0/4261694412 learned_addr learned my addr 172.21.15.62:0/4261694412 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:20:37.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.103+0000 7f276ea00700 1 --2- 172.21.15.62:0/4261694412 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2770073980 0x7f27701986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:20:37.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.103+0000 7f276f400700 1 -- 172.21.15.62:0/4261694412 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2770073980 msgr2=0x7f27701986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:20:37.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.103+0000 7f276f400700 1 --2- 172.21.15.62:0/4261694412 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2770073980 0x7f27701986f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:20:37.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.103+0000 7f276f400700 1 -- 172.21.15.62:0/4261694412 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f275c0095f0 con 0x7f2770073070 2024-04-20T22:20:37.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.103+0000 7f276f400700 1 --2- 172.21.15.62:0/4261694412 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2770073070 0x7f27701981b0 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f276000b4b0 tx=0x7f276000b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:20:37.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.103+0000 7f2767e00700 1 -- 172.21.15.62:0/4261694412 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2760011800 con 0x7f2770073070 2024-04-20T22:20:37.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.103+0000 7f2767e00700 1 -- 172.21.15.62:0/4261694412 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2760011e20 con 0x7f2770073070 2024-04-20T22:20:37.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.103+0000 7f2776600700 1 -- 172.21.15.62:0/4261694412 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f277019ce40 con 0x7f2770073070 2024-04-20T22:20:37.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.103+0000 7f2767e00700 1 -- 172.21.15.62:0/4261694412 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f276000f2e0 con 0x7f2770073070 2024-04-20T22:20:37.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.103+0000 7f2776600700 1 -- 172.21.15.62:0/4261694412 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f277019d390 con 0x7f2770073070 2024-04-20T22:20:37.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.105+0000 7f2776600700 1 -- 172.21.15.62:0/4261694412 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2770109d00 con 0x7f2770073070 2024-04-20T22:20:37.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.105+0000 7f2767e00700 1 -- 172.21.15.62:0/4261694412 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f2760010380 con 0x7f2770073070 2024-04-20T22:20:37.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.106+0000 7f2767e00700 1 --2- 172.21.15.62:0/4261694412 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f275406ecc0 0x7f2754071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:20:37.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.106+0000 7f2767e00700 1 -- 172.21.15.62:0/4261694412 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f276008eaa0 con 0x7f2770073070 2024-04-20T22:20:37.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.107+0000 7f276ea00700 1 --2- 172.21.15.62:0/4261694412 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f275406ecc0 0x7f2754071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:20:37.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.107+0000 7f276ea00700 1 --2- 172.21.15.62:0/4261694412 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f275406ecc0 0x7f2754071170 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f275c00b870 tx=0x7f275c005fb0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:20:37.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.111+0000 7f2767e00700 1 -- 172.21.15.62:0/4261694412 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f27600577b0 con 0x7f2770073070 2024-04-20T22:20:37.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.382+0000 7f2776600700 1 -- 172.21.15.62:0/4261694412 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f277004f2a0 con 0x7f2770073070 2024-04-20T22:20:37.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.383+0000 7f2767e00700 1 -- 172.21.15.62:0/4261694412 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 29 v29) v1 ==== 76+0+1887 (secure 0 0 0) 0x7f276005ae70 con 0x7f2770073070 2024-04-20T22:20:37.383 INFO:teuthology.orchestra.run.smithi062.stdout:e29 2024-04-20T22:20:37.383 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T22:20:37.383 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:20:37.383 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T22:20:37.383 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:20:37.383 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T22:20:37.383 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T22:20:37.383 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 20 2024-04-20T22:20:37.384 INFO:teuthology.orchestra.run.smithi062.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-20T22:20:37.384 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T22:20:37.384 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T21:42:01.559061+0000 2024-04-20T22:20:37.384 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T22:20:37.384 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T22:20:37.384 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T22:20:37.384 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T22:20:37.384 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T22:20:37.384 INFO:teuthology.orchestra.run.smithi062.stdout:max_xattr_size 65536 2024-04-20T22:20:37.384 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T22:20:37.384 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T22:20:37.384 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 81 2024-04-20T22:20:37.384 INFO:teuthology.orchestra.run.smithi062.stdout:snap_rstat 1 2024-04-20T22:20:37.384 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:20:37.385 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 2 2024-04-20T22:20:37.385 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1 2024-04-20T22:20:37.385 INFO:teuthology.orchestra.run.smithi062.stdout:up {} 2024-04-20T22:20:37.385 INFO:teuthology.orchestra.run.smithi062.stdout:failed 0,1 2024-04-20T22:20:37.385 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T22:20:37.385 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T22:20:37.385 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T22:20:37.385 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T22:20:37.385 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data enabled 2024-04-20T22:20:37.385 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T22:20:37.385 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T22:20:37.386 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 1 2024-04-20T22:20:37.386 INFO:teuthology.orchestra.run.smithi062.stdout:qdb_cluster leader: 0 members: 2024-04-20T22:20:37.386 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:20:37.386 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:20:37.386 INFO:teuthology.orchestra.run.smithi062.stdout:Standby daemons: 2024-04-20T22:20:37.386 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:20:37.386 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ryqblb{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6826/4113916205,v1:172.21.15.62:6827/4113916205] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:20:37.386 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ozizwi{-1:34476} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6828/3801975165,v1:172.21.15.62:6829/3801975165] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:20:37.386 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.dvimer{-1:44727} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6824/3576699232,v1:172.21.15.189:6825/3576699232] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:20:37.386 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.xbgspw{-1:45075} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6826/212905792,v1:172.21.15.189:6827/212905792] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:20:37.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.385+0000 7f2776600700 1 -- 172.21.15.62:0/4261694412 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f275406ecc0 msgr2=0x7f2754071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:20:37.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.385+0000 7f2776600700 1 --2- 172.21.15.62:0/4261694412 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f275406ecc0 0x7f2754071170 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f275c00b870 tx=0x7f275c005fb0 comp rx=0 tx=0).stop 2024-04-20T22:20:37.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.385+0000 7f2776600700 1 -- 172.21.15.62:0/4261694412 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2770073070 msgr2=0x7f27701981b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:20:37.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.385+0000 7f2776600700 1 --2- 172.21.15.62:0/4261694412 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2770073070 0x7f27701981b0 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f276000b4b0 tx=0x7f276000b870 comp rx=0 tx=0).stop 2024-04-20T22:20:37.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.385+0000 7f2776600700 1 -- 172.21.15.62:0/4261694412 shutdown_connections 2024-04-20T22:20:37.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.385+0000 7f2776600700 1 --2- 172.21.15.62:0/4261694412 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f275406ecc0 0x7f2754071170 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:20:37.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.385+0000 7f2776600700 1 --2- 172.21.15.62:0/4261694412 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2770073070 0x7f27701981b0 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:20:37.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.385+0000 7f2776600700 1 --2- 172.21.15.62:0/4261694412 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2770073980 0x7f27701986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:20:37.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.385+0000 7f2776600700 1 -- 172.21.15.62:0/4261694412 >> 172.21.15.62:0/4261694412 conn(0x7f27700fbda0 msgr2=0x7f2770106e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:20:37.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.386+0000 7f2776600700 1 -- 172.21.15.62:0/4261694412 shutdown_connections 2024-04-20T22:20:37.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.386+0000 7f2776600700 1 -- 172.21.15.62:0/4261694412 wait complete. 2024-04-20T22:20:37.388 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 29 2024-04-20T22:20:37.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.551+0000 7fcaca400700 1 -- 172.21.15.62:0/411489886 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcac4106b80 msgr2=0x7fcac4106ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:20:37.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.551+0000 7fcaca400700 1 --2- 172.21.15.62:0/411489886 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcac4106b80 0x7fcac4106ff0 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7fcab0009910 tx=0x7fcab0009c20 comp rx=0 tx=0).stop 2024-04-20T22:20:37.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.551+0000 7fcaca400700 1 -- 172.21.15.62:0/411489886 shutdown_connections 2024-04-20T22:20:37.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.551+0000 7fcaca400700 1 --2- 172.21.15.62:0/411489886 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcac4106b80 0x7fcac4106ff0 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:20:37.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.551+0000 7fcaca400700 1 --2- 172.21.15.62:0/411489886 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcac410c8f0 0x7fcac410ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:20:37.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.551+0000 7fcaca400700 1 -- 172.21.15.62:0/411489886 >> 172.21.15.62:0/411489886 conn(0x7fcac4076060 msgr2=0x7fcac4078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:20:37.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.551+0000 7fcaca400700 1 -- 172.21.15.62:0/411489886 shutdown_connections 2024-04-20T22:20:37.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.551+0000 7fcaca400700 1 -- 172.21.15.62:0/411489886 wait complete. 2024-04-20T22:20:37.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.552+0000 7fcaca400700 1 Processor -- start 2024-04-20T22:20:37.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.552+0000 7fcaca400700 1 -- start start 2024-04-20T22:20:37.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.553+0000 7fcaca400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcac4106b80 0x7fcac419c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:20:37.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.553+0000 7fcaca400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcac410c8f0 0x7fcac419c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:20:37.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.553+0000 7fcaca400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcac419d060 con 0x7fcac410c8f0 2024-04-20T22:20:37.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.553+0000 7fcaca400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcac41a0df0 con 0x7fcac4106b80 2024-04-20T22:20:37.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.553+0000 7fcac2a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcac410c8f0 0x7fcac419c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:20:37.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.553+0000 7fcac2a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcac410c8f0 0x7fcac419c980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34982/0 (socket says 172.21.15.62:34982) 2024-04-20T22:20:37.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.553+0000 7fcac2a00700 1 -- 172.21.15.62:0/767501121 learned_addr learned my addr 172.21.15.62:0/767501121 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:20:37.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.553+0000 7fcac3400700 1 --2- 172.21.15.62:0/767501121 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcac4106b80 0x7fcac419c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:20:37.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.553+0000 7fcac2a00700 1 -- 172.21.15.62:0/767501121 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcac4106b80 msgr2=0x7fcac419c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:20:37.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.553+0000 7fcac2a00700 1 --2- 172.21.15.62:0/767501121 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcac4106b80 0x7fcac419c440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:20:37.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.553+0000 7fcac2a00700 1 -- 172.21.15.62:0/767501121 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcab00095f0 con 0x7fcac410c8f0 2024-04-20T22:20:37.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.554+0000 7fcac2a00700 1 --2- 172.21.15.62:0/767501121 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcac410c8f0 0x7fcac419c980 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7fcab000b5c0 tx=0x7fcab000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:20:37.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.554+0000 7fcabbe00700 1 -- 172.21.15.62:0/767501121 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcab001d070 con 0x7fcac410c8f0 2024-04-20T22:20:37.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.554+0000 7fcabbe00700 1 -- 172.21.15.62:0/767501121 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcab0005e00 con 0x7fcac410c8f0 2024-04-20T22:20:37.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.554+0000 7fcaca400700 1 -- 172.21.15.62:0/767501121 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcac41a1070 con 0x7fcac410c8f0 2024-04-20T22:20:37.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.554+0000 7fcabbe00700 1 -- 172.21.15.62:0/767501121 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcab0017400 con 0x7fcac410c8f0 2024-04-20T22:20:37.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.554+0000 7fcaca400700 1 -- 172.21.15.62:0/767501121 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcac41a1560 con 0x7fcac410c8f0 2024-04-20T22:20:37.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.555+0000 7fcaca400700 1 -- 172.21.15.62:0/767501121 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcac410ecc0 con 0x7fcac410c8f0 2024-04-20T22:20:37.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.556+0000 7fcabbe00700 1 -- 172.21.15.62:0/767501121 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fcab000f460 con 0x7fcac410c8f0 2024-04-20T22:20:37.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.557+0000 7fcabbe00700 1 --2- 172.21.15.62:0/767501121 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fcaa806ecc0 0x7fcaa8071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:20:37.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.558+0000 7fcabbe00700 1 -- 172.21.15.62:0/767501121 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fcab0090370 con 0x7fcac410c8f0 2024-04-20T22:20:37.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.558+0000 7fcac3400700 1 --2- 172.21.15.62:0/767501121 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fcaa806ecc0 0x7fcaa8071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:20:37.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.559+0000 7fcac3400700 1 --2- 172.21.15.62:0/767501121 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fcaa806ecc0 0x7fcaa8071170 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fcab4005950 tx=0x7fcab40058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:20:37.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.563+0000 7fcabbe00700 1 -- 172.21.15.62:0/767501121 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fcab0059100 con 0x7fcac410c8f0 2024-04-20T22:20:37.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:20:37 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/3228821770' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:20:37.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:20:37 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/4261694412' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:20:37.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:20:37 smithi062 ceph-mon[76322]: pgmap v1363: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:20:37.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.816+0000 7fcaca400700 1 -- 172.21.15.62:0/767501121 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fcac419d8b0 con 0x7fcaa806ecc0 2024-04-20T22:20:37.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.818+0000 7fcabbe00700 1 -- 172.21.15.62:0/767501121 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fcac419d8b0 con 0x7fcaa806ecc0 2024-04-20T22:20:37.819 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:20:37.819 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T22:20:37.819 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T22:20:37.819 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-20T22:20:37.819 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T22:20:37.819 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr", 2024-04-20T22:20:37.819 INFO:teuthology.orchestra.run.smithi062.stdout: "osd", 2024-04-20T22:20:37.819 INFO:teuthology.orchestra.run.smithi062.stdout: "mds", 2024-04-20T22:20:37.819 INFO:teuthology.orchestra.run.smithi062.stdout: "crash", 2024-04-20T22:20:37.820 INFO:teuthology.orchestra.run.smithi062.stdout: "mon" 2024-04-20T22:20:37.820 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T22:20:37.820 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "16/23 daemons upgraded", 2024-04-20T22:20:37.820 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading mds daemons", 2024-04-20T22:20:37.820 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T22:20:37.820 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:20:37.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.821+0000 7fcaca400700 1 -- 172.21.15.62:0/767501121 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fcaa806ecc0 msgr2=0x7fcaa8071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:20:37.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.821+0000 7fcaca400700 1 --2- 172.21.15.62:0/767501121 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fcaa806ecc0 0x7fcaa8071170 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fcab4005950 tx=0x7fcab40058e0 comp rx=0 tx=0).stop 2024-04-20T22:20:37.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.821+0000 7fcaca400700 1 -- 172.21.15.62:0/767501121 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcac410c8f0 msgr2=0x7fcac419c980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:20:37.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.821+0000 7fcaca400700 1 --2- 172.21.15.62:0/767501121 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcac410c8f0 0x7fcac419c980 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7fcab000b5c0 tx=0x7fcab000be20 comp rx=0 tx=0).stop 2024-04-20T22:20:37.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.822+0000 7fcaca400700 1 -- 172.21.15.62:0/767501121 shutdown_connections 2024-04-20T22:20:37.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.822+0000 7fcaca400700 1 --2- 172.21.15.62:0/767501121 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fcaa806ecc0 0x7fcaa8071170 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:20:37.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.822+0000 7fcaca400700 1 --2- 172.21.15.62:0/767501121 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcac4106b80 0x7fcac419c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:20:37.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.822+0000 7fcaca400700 1 --2- 172.21.15.62:0/767501121 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcac410c8f0 0x7fcac419c980 unknown :-1 s=CLOSED pgs=1166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:20:37.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.822+0000 7fcaca400700 1 -- 172.21.15.62:0/767501121 >> 172.21.15.62:0/767501121 conn(0x7fcac4076060 msgr2=0x7fcac4077ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:20:37.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.822+0000 7fcaca400700 1 -- 172.21.15.62:0/767501121 shutdown_connections 2024-04-20T22:20:37.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.822+0000 7fcaca400700 1 -- 172.21.15.62:0/767501121 wait complete. 2024-04-20T22:20:37.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:20:37 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/3228821770' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:20:37.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:20:37 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/4261694412' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:20:37.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:20:37 smithi189 ceph-mon[57022]: pgmap v1363: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:20:37.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.990+0000 7f7232600700 1 -- 172.21.15.62:0/1114057125 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f722c103e50 msgr2=0x7f722c1042c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:20:37.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.990+0000 7f7232600700 1 --2- 172.21.15.62:0/1114057125 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f722c103e50 0x7f722c1042c0 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f7218009910 tx=0x7f7218009c20 comp rx=0 tx=0).stop 2024-04-20T22:20:37.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.990+0000 7f7232600700 1 -- 172.21.15.62:0/1114057125 shutdown_connections 2024-04-20T22:20:37.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.990+0000 7f7232600700 1 --2- 172.21.15.62:0/1114057125 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f722c103e50 0x7f722c1042c0 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:20:37.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.990+0000 7f7232600700 1 --2- 172.21.15.62:0/1114057125 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f722c1025f0 0x7f722c1029c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:20:37.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.990+0000 7f7232600700 1 -- 172.21.15.62:0/1114057125 >> 172.21.15.62:0/1114057125 conn(0x7f722c0fddb0 msgr2=0x7f722c1001c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:20:37.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.991+0000 7f7232600700 1 -- 172.21.15.62:0/1114057125 shutdown_connections 2024-04-20T22:20:37.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.991+0000 7f7232600700 1 -- 172.21.15.62:0/1114057125 wait complete. 2024-04-20T22:20:37.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.992+0000 7f7232600700 1 Processor -- start 2024-04-20T22:20:37.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.992+0000 7f7232600700 1 -- start start 2024-04-20T22:20:37.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.992+0000 7f7232600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f722c1025f0 0x7f722c197ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:20:37.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.992+0000 7f7232600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f722c103e50 0x7f722c1983e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:20:37.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.992+0000 7f7232600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f722c198ac0 con 0x7f722c103e50 2024-04-20T22:20:37.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.992+0000 7f7232600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f722c19c850 con 0x7f722c1025f0 2024-04-20T22:20:37.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.992+0000 7f722be00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f722c103e50 0x7f722c1983e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:20:37.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.993+0000 7f722be00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f722c103e50 0x7f722c1983e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34994/0 (socket says 172.21.15.62:34994) 2024-04-20T22:20:37.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.993+0000 7f722be00700 1 -- 172.21.15.62:0/790460313 learned_addr learned my addr 172.21.15.62:0/790460313 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:20:37.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.993+0000 7f7231200700 1 --2- 172.21.15.62:0/790460313 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f722c1025f0 0x7f722c197ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:20:37.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.993+0000 7f722be00700 1 -- 172.21.15.62:0/790460313 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f722c1025f0 msgr2=0x7f722c197ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:20:37.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.993+0000 7f722be00700 1 --2- 172.21.15.62:0/790460313 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f722c1025f0 0x7f722c197ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:20:37.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.993+0000 7f722be00700 1 -- 172.21.15.62:0/790460313 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72180095f0 con 0x7f722c103e50 2024-04-20T22:20:37.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.993+0000 7f722be00700 1 --2- 172.21.15.62:0/790460313 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f722c103e50 0x7f722c1983e0 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f72180092e0 tx=0x7f721800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:20:37.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.994+0000 7f7229600700 1 -- 172.21.15.62:0/790460313 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f721801d070 con 0x7f722c103e50 2024-04-20T22:20:37.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.994+0000 7f7229600700 1 -- 172.21.15.62:0/790460313 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7218005e00 con 0x7f722c103e50 2024-04-20T22:20:37.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.994+0000 7f7232600700 1 -- 172.21.15.62:0/790460313 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f722c19cad0 con 0x7f722c103e50 2024-04-20T22:20:37.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.994+0000 7f7229600700 1 -- 172.21.15.62:0/790460313 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7218017400 con 0x7f722c103e50 2024-04-20T22:20:37.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.994+0000 7f7232600700 1 -- 172.21.15.62:0/790460313 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f722c19d040 con 0x7f722c103e50 2024-04-20T22:20:37.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.995+0000 7f7232600700 1 -- 172.21.15.62:0/790460313 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f722c062380 con 0x7f722c103e50 2024-04-20T22:20:37.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.996+0000 7f7229600700 1 -- 172.21.15.62:0/790460313 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f721800f460 con 0x7f722c103e50 2024-04-20T22:20:37.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.997+0000 7f7229600700 1 --2- 172.21.15.62:0/790460313 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f721406ecc0 0x7f7214071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:20:37.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.997+0000 7f7229600700 1 -- 172.21.15.62:0/790460313 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f7218090330 con 0x7f722c103e50 2024-04-20T22:20:37.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.997+0000 7f7231200700 1 --2- 172.21.15.62:0/790460313 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f721406ecc0 0x7f7214071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:20:37.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:37.998+0000 7f7231200700 1 --2- 172.21.15.62:0/790460313 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f721406ecc0 0x7f7214071170 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f7220005950 tx=0x7f72200058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:20:38.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:38.002+0000 7f7229600700 1 -- 172.21.15.62:0/790460313 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f72180590c0 con 0x7f722c103e50 2024-04-20T22:20:38.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:38.334+0000 7f7232600700 1 -- 172.21.15.62:0/790460313 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f722c19d2b0 con 0x7f722c103e50 2024-04-20T22:20:38.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:38.335+0000 7f7229600700 1 -- 172.21.15.62:0/790460313 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7f721805c780 con 0x7f722c103e50 2024-04-20T22:20:38.336 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-20T22:20:38.336 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-20T22:20:38.336 INFO:teuthology.orchestra.run.smithi062.stdout: host smithi189 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi189/config 2024-04-20T22:20:38.336 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-20T22:20:38.336 INFO:teuthology.orchestra.run.smithi062.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi189 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp11670v2q:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:20:38.336 INFO:teuthology.orchestra.run.smithi062.stdout:Traceback (most recent call last): 2024-04-20T22:20:38.336 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-20T22:20:38.336 INFO:teuthology.orchestra.run.smithi062.stdout: return _run_code(code, main_globals, None, 2024-04-20T22:20:38.336 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-20T22:20:38.337 INFO:teuthology.orchestra.run.smithi062.stdout: exec(code, run_globals) 2024-04-20T22:20:38.337 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5618, in 2024-04-20T22:20:38.337 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5606, in main 2024-04-20T22:20:38.337 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 408, in _infer_config 2024-04-20T22:20:38.337 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 323, in _infer_fsid 2024-04-20T22:20:38.337 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 436, in _infer_image 2024-04-20T22:20:38.337 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 310, in _validate_fsid 2024-04-20T22:20:38.337 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 3359, in command_ceph_volume 2024-04-20T22:20:38.337 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-20T22:20:38.337 INFO:teuthology.orchestra.run.smithi062.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi189 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp11670v2q:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:20:38.337 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-20T22:20:38.337 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is degraded 2024-04-20T22:20:38.337 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T22:20:38.338 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T22:20:38.338 INFO:teuthology.orchestra.run.smithi062.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-20T22:20:38.338 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-20T22:20:38.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:38.338+0000 7f7232600700 1 -- 172.21.15.62:0/790460313 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f721406ecc0 msgr2=0x7f7214071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:20:38.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:38.338+0000 7f7232600700 1 --2- 172.21.15.62:0/790460313 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f721406ecc0 0x7f7214071170 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f7220005950 tx=0x7f72200058e0 comp rx=0 tx=0).stop 2024-04-20T22:20:38.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:38.338+0000 7f7232600700 1 -- 172.21.15.62:0/790460313 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f722c103e50 msgr2=0x7f722c1983e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:20:38.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:38.338+0000 7f7232600700 1 --2- 172.21.15.62:0/790460313 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f722c103e50 0x7f722c1983e0 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f72180092e0 tx=0x7f721800be20 comp rx=0 tx=0).stop 2024-04-20T22:20:38.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:38.338+0000 7f7232600700 1 -- 172.21.15.62:0/790460313 shutdown_connections 2024-04-20T22:20:38.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:38.338+0000 7f7232600700 1 --2- 172.21.15.62:0/790460313 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f721406ecc0 0x7f7214071170 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:20:38.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:38.338+0000 7f7232600700 1 --2- 172.21.15.62:0/790460313 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f722c1025f0 0x7f722c197ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:20:38.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:38.338+0000 7f7232600700 1 --2- 172.21.15.62:0/790460313 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f722c103e50 0x7f722c1983e0 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:20:38.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:38.338+0000 7f7232600700 1 -- 172.21.15.62:0/790460313 >> 172.21.15.62:0/790460313 conn(0x7f722c0fddb0 msgr2=0x7f722c0fe710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:20:38.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:38.339+0000 7f7232600700 1 -- 172.21.15.62:0/790460313 shutdown_connections 2024-04-20T22:20:38.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:20:38.339+0000 7f7232600700 1 -- 172.21.15.62:0/790460313 wait complete. 2024-04-20T22:20:38.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:20:38 smithi062 ceph-mon[76322]: from='client.36404 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:20:38.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:20:38 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/790460313' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:20:38.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:20:38 smithi189 ceph-mon[57022]: from='client.36404 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:20:38.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:20:38 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/790460313' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:20:39.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:20:39 smithi062 ceph-mon[76322]: pgmap v1364: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:20:39.990 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:20:39 smithi189 ceph-mon[57022]: pgmap v1364: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:20:41.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:20:41 smithi062 ceph-mon[76322]: pgmap v1365: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:20:41.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:20:41 smithi189 ceph-mon[57022]: pgmap v1365: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:20:43.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:20:43 smithi062 ceph-mon[76322]: pgmap v1366: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:20:43.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:20:43 smithi189 ceph-mon[57022]: pgmap v1366: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:20:45.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:20:45 smithi062 ceph-mon[76322]: pgmap v1367: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:20:45.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:20:45 smithi189 ceph-mon[57022]: pgmap v1367: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:20:47.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:20:47 smithi189 ceph-mon[57022]: pgmap v1368: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:20:47.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:20:47 smithi062 ceph-mon[76322]: pgmap v1368: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:20:49.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:20:49 smithi062 ceph-mon[76322]: pgmap v1369: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:20:49.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:20:49 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:20:49.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:20:49 smithi189 ceph-mon[57022]: pgmap v1369: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:20:49.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:20:49 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:20:51.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:20:51 smithi189 ceph-mon[57022]: pgmap v1370: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:20:51.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:20:51 smithi062 ceph-mon[76322]: pgmap v1370: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:20:53.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:20:53 smithi062 ceph-mon[76322]: pgmap v1371: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:20:53.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:20:53 smithi189 ceph-mon[57022]: pgmap v1371: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:20:55.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:20:55 smithi062 ceph-mon[76322]: pgmap v1372: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:20:55.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:20:55 smithi189 ceph-mon[57022]: pgmap v1372: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:20:57.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:20:57 smithi062 ceph-mon[76322]: pgmap v1373: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:20:57.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:20:57 smithi189 ceph-mon[57022]: pgmap v1373: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:20:59.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:20:59 smithi062 ceph-mon[76322]: pgmap v1374: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:20:59.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:20:59 smithi189 ceph-mon[57022]: pgmap v1374: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:21:01.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:21:01 smithi062 ceph-mon[76322]: pgmap v1375: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:21:01.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:21:01 smithi189 ceph-mon[57022]: pgmap v1375: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:21:03.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:21:03 smithi062 ceph-mon[76322]: pgmap v1376: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:21:03.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:21:03 smithi189 ceph-mon[57022]: pgmap v1376: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:21:04.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:21:04 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:21:04.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:21:04 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:21:05.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:21:05 smithi062 ceph-mon[76322]: pgmap v1377: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:21:05.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:21:05 smithi189 ceph-mon[57022]: pgmap v1377: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:21:07.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:21:07 smithi062 ceph-mon[76322]: pgmap v1378: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:21:07.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:21:07 smithi189 ceph-mon[57022]: pgmap v1378: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:21:08.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:08.518+0000 7f0dccc00700 1 -- 172.21.15.62:0/653188841 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0dc81004e0 msgr2=0x7f0dc81008b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:21:08.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:08.518+0000 7f0dccc00700 1 --2- 172.21.15.62:0/653188841 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0dc81004e0 0x7f0dc81008b0 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f0db8009910 tx=0x7f0db8009c20 comp rx=0 tx=0).stop 2024-04-20T22:21:08.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:08.518+0000 7f0dccc00700 1 -- 172.21.15.62:0/653188841 shutdown_connections 2024-04-20T22:21:08.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:08.518+0000 7f0dccc00700 1 --2- 172.21.15.62:0/653188841 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0dc8100e80 0x7f0dc8104e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:21:08.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:08.518+0000 7f0dccc00700 1 --2- 172.21.15.62:0/653188841 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0dc81004e0 0x7f0dc81008b0 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:21:08.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:08.518+0000 7f0dccc00700 1 -- 172.21.15.62:0/653188841 >> 172.21.15.62:0/653188841 conn(0x7f0dc80fbda0 msgr2=0x7f0dc80fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:21:08.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:08.519+0000 7f0dccc00700 1 -- 172.21.15.62:0/653188841 shutdown_connections 2024-04-20T22:21:08.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:08.519+0000 7f0dccc00700 1 -- 172.21.15.62:0/653188841 wait complete. 2024-04-20T22:21:08.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:08.520+0000 7f0dccc00700 1 Processor -- start 2024-04-20T22:21:08.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:08.520+0000 7f0dccc00700 1 -- start start 2024-04-20T22:21:08.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:08.520+0000 7f0dccc00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0dc81004e0 0x7f0dc8073050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:21:08.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:08.520+0000 7f0dccc00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0dc8100e80 0x7f0dc8073590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:21:08.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:08.520+0000 7f0dccc00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dc8073ad0 con 0x7f0dc8100e80 2024-04-20T22:21:08.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:08.520+0000 7f0dccc00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dc8073c10 con 0x7f0dc81004e0 2024-04-20T22:21:08.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:08.521+0000 7f0dbfe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0dc8100e80 0x7f0dc8073590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:21:08.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:08.521+0000 7f0dbfe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0dc8100e80 0x7f0dc8073590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51630/0 (socket says 172.21.15.62:51630) 2024-04-20T22:21:08.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:08.521+0000 7f0dbfe00700 1 -- 172.21.15.62:0/218940485 learned_addr learned my addr 172.21.15.62:0/218940485 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:21:08.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:08.521+0000 7f0dbfe00700 1 -- 172.21.15.62:0/218940485 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0dc81004e0 msgr2=0x7f0dc8073050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:21:08.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:08.521+0000 7f0dc5e00700 1 --2- 172.21.15.62:0/218940485 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0dc81004e0 0x7f0dc8073050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:21:08.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:08.521+0000 7f0dbfe00700 1 --2- 172.21.15.62:0/218940485 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0dc81004e0 0x7f0dc8073050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:21:08.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:08.521+0000 7f0dbfe00700 1 -- 172.21.15.62:0/218940485 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0db80095f0 con 0x7f0dc8100e80 2024-04-20T22:21:08.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:08.522+0000 7f0dbfe00700 1 --2- 172.21.15.62:0/218940485 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0dc8100e80 0x7f0dc8073590 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f0db000b480 tx=0x7f0db000b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:21:08.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:08.522+0000 7f0dbea00700 1 -- 172.21.15.62:0/218940485 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0db00118c0 con 0x7f0dc8100e80 2024-04-20T22:21:08.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:08.522+0000 7f0dbea00700 1 -- 172.21.15.62:0/218940485 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0db0010380 con 0x7f0dc8100e80 2024-04-20T22:21:08.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:08.522+0000 7f0dccc00700 1 -- 172.21.15.62:0/218940485 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0dc81a2160 con 0x7f0dc8100e80 2024-04-20T22:21:08.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:08.522+0000 7f0dbea00700 1 -- 172.21.15.62:0/218940485 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0db000f4f0 con 0x7f0dc8100e80 2024-04-20T22:21:08.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:08.522+0000 7f0dccc00700 1 -- 172.21.15.62:0/218940485 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0dc81a2570 con 0x7f0dc8100e80 2024-04-20T22:21:08.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:08.523+0000 7f0dccc00700 1 -- 172.21.15.62:0/218940485 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0dc81087f0 con 0x7f0dc8100e80 2024-04-20T22:21:08.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:08.524+0000 7f0dbea00700 1 -- 172.21.15.62:0/218940485 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f0db00104f0 con 0x7f0dc8100e80 2024-04-20T22:21:08.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:08.525+0000 7f0dbea00700 1 --2- 172.21.15.62:0/218940485 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f0dac06ecc0 0x7f0dac071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:21:08.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:08.525+0000 7f0dbea00700 1 -- 172.21.15.62:0/218940485 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f0db008ea70 con 0x7f0dc8100e80 2024-04-20T22:21:08.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:08.526+0000 7f0dc5e00700 1 --2- 172.21.15.62:0/218940485 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f0dac06ecc0 0x7f0dac071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:21:08.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:08.526+0000 7f0dc5e00700 1 --2- 172.21.15.62:0/218940485 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f0dac06ecc0 0x7f0dac071170 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f0db8006010 tx=0x7f0db800b540 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:21:08.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:08.531+0000 7f0dbea00700 1 -- 172.21.15.62:0/218940485 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f0db0057800 con 0x7f0dc8100e80 2024-04-20T22:21:08.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:08.791+0000 7f0dccc00700 1 -- 172.21.15.62:0/218940485 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0dc8061190 con 0x7f0dac06ecc0 2024-04-20T22:21:08.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:08.795+0000 7f0dbea00700 1 -- 172.21.15.62:0/218940485 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f0dc8061190 con 0x7f0dac06ecc0 2024-04-20T22:21:08.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:08.797+0000 7f0dccc00700 1 -- 172.21.15.62:0/218940485 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f0dac06ecc0 msgr2=0x7f0dac071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:21:08.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:08.797+0000 7f0dccc00700 1 --2- 172.21.15.62:0/218940485 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f0dac06ecc0 0x7f0dac071170 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f0db8006010 tx=0x7f0db800b540 comp rx=0 tx=0).stop 2024-04-20T22:21:08.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:08.797+0000 7f0dccc00700 1 -- 172.21.15.62:0/218940485 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0dc8100e80 msgr2=0x7f0dc8073590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:21:08.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:08.797+0000 7f0dccc00700 1 --2- 172.21.15.62:0/218940485 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0dc8100e80 0x7f0dc8073590 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f0db000b480 tx=0x7f0db000b840 comp rx=0 tx=0).stop 2024-04-20T22:21:08.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:08.798+0000 7f0dccc00700 1 -- 172.21.15.62:0/218940485 shutdown_connections 2024-04-20T22:21:08.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:08.798+0000 7f0dccc00700 1 --2- 172.21.15.62:0/218940485 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f0dac06ecc0 0x7f0dac071170 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:21:08.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:08.798+0000 7f0dccc00700 1 --2- 172.21.15.62:0/218940485 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0dc81004e0 0x7f0dc8073050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:21:08.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:08.798+0000 7f0dccc00700 1 --2- 172.21.15.62:0/218940485 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0dc8100e80 0x7f0dc8073590 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:21:08.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:08.798+0000 7f0dccc00700 1 -- 172.21.15.62:0/218940485 >> 172.21.15.62:0/218940485 conn(0x7f0dc80fbda0 msgr2=0x7f0dc80fd470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:21:08.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:08.798+0000 7f0dccc00700 1 -- 172.21.15.62:0/218940485 shutdown_connections 2024-04-20T22:21:08.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:08.798+0000 7f0dccc00700 1 -- 172.21.15.62:0/218940485 wait complete. 2024-04-20T22:21:08.814 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T22:21:08.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:08.963+0000 7faf41400700 1 -- 172.21.15.62:0/161139685 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faf3c10c8f0 msgr2=0x7faf3c10ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:21:08.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:08.963+0000 7faf41400700 1 --2- 172.21.15.62:0/161139685 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faf3c10c8f0 0x7faf3c10ccc0 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7faf2c009910 tx=0x7faf2c009c20 comp rx=0 tx=0).stop 2024-04-20T22:21:08.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:08.964+0000 7faf41400700 1 -- 172.21.15.62:0/161139685 shutdown_connections 2024-04-20T22:21:08.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:08.964+0000 7faf41400700 1 --2- 172.21.15.62:0/161139685 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faf3c106b80 0x7faf3c106ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:21:08.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:08.964+0000 7faf41400700 1 --2- 172.21.15.62:0/161139685 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faf3c10c8f0 0x7faf3c10ccc0 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:21:08.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:08.964+0000 7faf41400700 1 -- 172.21.15.62:0/161139685 >> 172.21.15.62:0/161139685 conn(0x7faf3c076060 msgr2=0x7faf3c078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:21:08.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:08.964+0000 7faf41400700 1 -- 172.21.15.62:0/161139685 shutdown_connections 2024-04-20T22:21:08.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:08.964+0000 7faf41400700 1 -- 172.21.15.62:0/161139685 wait complete. 2024-04-20T22:21:08.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:08.965+0000 7faf41400700 1 Processor -- start 2024-04-20T22:21:08.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:08.965+0000 7faf41400700 1 -- start start 2024-04-20T22:21:08.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:08.966+0000 7faf41400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faf3c106b80 0x7faf3c19c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:21:08.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:08.966+0000 7faf41400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faf3c10c8f0 0x7faf3c19ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:21:08.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:08.966+0000 7faf41400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf3c19d100 con 0x7faf3c106b80 2024-04-20T22:21:08.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:08.966+0000 7faf41400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf3c1a0e90 con 0x7faf3c10c8f0 2024-04-20T22:21:08.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:08.966+0000 7faf3a800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faf3c106b80 0x7faf3c19c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:21:08.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:08.966+0000 7faf3a800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faf3c106b80 0x7faf3c19c4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51642/0 (socket says 172.21.15.62:51642) 2024-04-20T22:21:08.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:08.966+0000 7faf3a800700 1 -- 172.21.15.62:0/1771254294 learned_addr learned my addr 172.21.15.62:0/1771254294 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:21:08.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:08.966+0000 7faf39e00700 1 --2- 172.21.15.62:0/1771254294 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faf3c10c8f0 0x7faf3c19ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:21:08.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:08.967+0000 7faf3a800700 1 -- 172.21.15.62:0/1771254294 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faf3c10c8f0 msgr2=0x7faf3c19ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:21:08.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:08.967+0000 7faf3a800700 1 --2- 172.21.15.62:0/1771254294 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faf3c10c8f0 0x7faf3c19ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:21:08.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:08.967+0000 7faf3a800700 1 -- 172.21.15.62:0/1771254294 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf2c0095f0 con 0x7faf3c106b80 2024-04-20T22:21:08.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:08.967+0000 7faf39e00700 1 --2- 172.21.15.62:0/1771254294 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faf3c10c8f0 0x7faf3c19ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T22:21:08.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:08.967+0000 7faf3a800700 1 --2- 172.21.15.62:0/1771254294 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faf3c106b80 0x7faf3c19c4e0 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7faf2c006010 tx=0x7faf2c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:21:08.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:08.967+0000 7faf33400700 1 -- 172.21.15.62:0/1771254294 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faf2c01d070 con 0x7faf3c106b80 2024-04-20T22:21:08.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:08.967+0000 7faf33400700 1 -- 172.21.15.62:0/1771254294 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7faf2c004fb0 con 0x7faf3c106b80 2024-04-20T22:21:08.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:08.967+0000 7faf33400700 1 -- 172.21.15.62:0/1771254294 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faf2c022ea0 con 0x7faf3c106b80 2024-04-20T22:21:08.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:08.967+0000 7faf41400700 1 -- 172.21.15.62:0/1771254294 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf3c1a1110 con 0x7faf3c106b80 2024-04-20T22:21:08.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:08.968+0000 7faf41400700 1 -- 172.21.15.62:0/1771254294 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf3c1a1600 con 0x7faf3c106b80 2024-04-20T22:21:08.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:08.969+0000 7faf41400700 1 -- 172.21.15.62:0/1771254294 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf3c109b20 con 0x7faf3c106b80 2024-04-20T22:21:08.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:08.970+0000 7faf33400700 1 -- 172.21.15.62:0/1771254294 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7faf2c022470 con 0x7faf3c106b80 2024-04-20T22:21:08.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:08.971+0000 7faf33400700 1 --2- 172.21.15.62:0/1771254294 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7faf2006ecc0 0x7faf20071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:21:08.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:08.971+0000 7faf33400700 1 -- 172.21.15.62:0/1771254294 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7faf2c0904a0 con 0x7faf3c106b80 2024-04-20T22:21:08.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:08.971+0000 7faf39e00700 1 --2- 172.21.15.62:0/1771254294 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7faf2006ecc0 0x7faf20071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:21:08.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:08.972+0000 7faf39e00700 1 --2- 172.21.15.62:0/1771254294 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7faf2006ecc0 0x7faf20071170 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7faf3c19db00 tx=0x7faf28008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:21:08.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:08.976+0000 7faf33400700 1 -- 172.21.15.62:0/1771254294 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7faf2c059260 con 0x7faf3c106b80 2024-04-20T22:21:09.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:09.232+0000 7faf41400700 1 -- 172.21.15.62:0/1771254294 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7faf3c1a18e0 con 0x7faf2006ecc0 2024-04-20T22:21:09.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:09.235+0000 7faf33400700 1 -- 172.21.15.62:0/1771254294 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7faf3c1a18e0 con 0x7faf2006ecc0 2024-04-20T22:21:09.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:09.238+0000 7faf41400700 1 -- 172.21.15.62:0/1771254294 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7faf2006ecc0 msgr2=0x7faf20071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:21:09.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:09.238+0000 7faf41400700 1 --2- 172.21.15.62:0/1771254294 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7faf2006ecc0 0x7faf20071170 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7faf3c19db00 tx=0x7faf28008040 comp rx=0 tx=0).stop 2024-04-20T22:21:09.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:09.238+0000 7faf41400700 1 -- 172.21.15.62:0/1771254294 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faf3c106b80 msgr2=0x7faf3c19c4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:21:09.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:09.238+0000 7faf41400700 1 --2- 172.21.15.62:0/1771254294 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faf3c106b80 0x7faf3c19c4e0 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7faf2c006010 tx=0x7faf2c00be20 comp rx=0 tx=0).stop 2024-04-20T22:21:09.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:09.238+0000 7faf41400700 1 -- 172.21.15.62:0/1771254294 shutdown_connections 2024-04-20T22:21:09.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:09.238+0000 7faf41400700 1 --2- 172.21.15.62:0/1771254294 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7faf2006ecc0 0x7faf20071170 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:21:09.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:09.238+0000 7faf41400700 1 --2- 172.21.15.62:0/1771254294 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faf3c106b80 0x7faf3c19c4e0 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:21:09.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:09.238+0000 7faf41400700 1 --2- 172.21.15.62:0/1771254294 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faf3c10c8f0 0x7faf3c19ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:21:09.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:09.238+0000 7faf41400700 1 -- 172.21.15.62:0/1771254294 >> 172.21.15.62:0/1771254294 conn(0x7faf3c076060 msgr2=0x7faf3c077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:21:09.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:09.238+0000 7faf41400700 1 -- 172.21.15.62:0/1771254294 shutdown_connections 2024-04-20T22:21:09.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:09.238+0000 7faf41400700 1 -- 172.21.15.62:0/1771254294 wait complete. 2024-04-20T22:21:09.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:09.397+0000 7f4b45000700 1 -- 172.21.15.62:0/472679403 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4b40106b80 msgr2=0x7f4b40106ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:21:09.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:09.397+0000 7f4b45000700 1 --2- 172.21.15.62:0/472679403 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4b40106b80 0x7f4b40106ff0 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f4b2c009910 tx=0x7f4b2c009c20 comp rx=0 tx=0).stop 2024-04-20T22:21:09.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:09.398+0000 7f4b45000700 1 -- 172.21.15.62:0/472679403 shutdown_connections 2024-04-20T22:21:09.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:09.398+0000 7f4b45000700 1 --2- 172.21.15.62:0/472679403 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4b40106b80 0x7f4b40106ff0 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:21:09.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:09.398+0000 7f4b45000700 1 --2- 172.21.15.62:0/472679403 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b4010c660 0x7f4b4010ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:21:09.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:09.398+0000 7f4b45000700 1 -- 172.21.15.62:0/472679403 >> 172.21.15.62:0/472679403 conn(0x7f4b40075e90 msgr2=0x7f4b400782a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:21:09.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:09.398+0000 7f4b45000700 1 -- 172.21.15.62:0/472679403 shutdown_connections 2024-04-20T22:21:09.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:09.398+0000 7f4b45000700 1 -- 172.21.15.62:0/472679403 wait complete. 2024-04-20T22:21:09.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:09.399+0000 7f4b45000700 1 Processor -- start 2024-04-20T22:21:09.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:09.399+0000 7f4b45000700 1 -- start start 2024-04-20T22:21:09.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:09.399+0000 7f4b45000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4b40106b80 0x7f4b4019c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:21:09.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:09.399+0000 7f4b45000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b4010c660 0x7f4b4019c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:21:09.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:09.399+0000 7f4b45000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b4019ce00 con 0x7f4b40106b80 2024-04-20T22:21:09.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:09.399+0000 7f4b45000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b401a0b90 con 0x7f4b4010c660 2024-04-20T22:21:09.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:09.400+0000 7f4b3f400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4b40106b80 0x7f4b4019c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:21:09.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:09.400+0000 7f4b3f400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4b40106b80 0x7f4b4019c1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51650/0 (socket says 172.21.15.62:51650) 2024-04-20T22:21:09.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:09.400+0000 7f4b3f400700 1 -- 172.21.15.62:0/2239219295 learned_addr learned my addr 172.21.15.62:0/2239219295 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:21:09.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:09.400+0000 7f4b3f400700 1 -- 172.21.15.62:0/2239219295 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b4010c660 msgr2=0x7f4b4019c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:21:09.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:09.400+0000 7f4b3f400700 1 --2- 172.21.15.62:0/2239219295 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b4010c660 0x7f4b4019c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:21:09.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:09.400+0000 7f4b3f400700 1 -- 172.21.15.62:0/2239219295 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b2c0095f0 con 0x7f4b40106b80 2024-04-20T22:21:09.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:09.401+0000 7f4b3f400700 1 --2- 172.21.15.62:0/2239219295 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4b40106b80 0x7f4b4019c1e0 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f4b3000d8d0 tx=0x7f4b3000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:21:09.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:09.401+0000 7f4b37e00700 1 -- 172.21.15.62:0/2239219295 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4b30009940 con 0x7f4b40106b80 2024-04-20T22:21:09.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:09.401+0000 7f4b37e00700 1 -- 172.21.15.62:0/2239219295 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4b30010460 con 0x7f4b40106b80 2024-04-20T22:21:09.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:09.401+0000 7f4b45000700 1 -- 172.21.15.62:0/2239219295 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b401a0e70 con 0x7f4b40106b80 2024-04-20T22:21:09.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:09.401+0000 7f4b37e00700 1 -- 172.21.15.62:0/2239219295 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4b3000f5f0 con 0x7f4b40106b80 2024-04-20T22:21:09.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:09.401+0000 7f4b45000700 1 -- 172.21.15.62:0/2239219295 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b401a13c0 con 0x7f4b40106b80 2024-04-20T22:21:09.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:09.402+0000 7f4b45000700 1 -- 172.21.15.62:0/2239219295 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b4010eb40 con 0x7f4b40106b80 2024-04-20T22:21:09.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:09.403+0000 7f4b37e00700 1 -- 172.21.15.62:0/2239219295 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f4b300105d0 con 0x7f4b40106b80 2024-04-20T22:21:09.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:09.405+0000 7f4b37e00700 1 --2- 172.21.15.62:0/2239219295 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f4b2406ecc0 0x7f4b24071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:21:09.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:09.405+0000 7f4b37e00700 1 -- 172.21.15.62:0/2239219295 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f4b3008ef20 con 0x7f4b40106b80 2024-04-20T22:21:09.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:09.405+0000 7f4b3ea00700 1 --2- 172.21.15.62:0/2239219295 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f4b2406ecc0 0x7f4b24071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:21:09.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:09.406+0000 7f4b3ea00700 1 --2- 172.21.15.62:0/2239219295 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f4b2406ecc0 0x7f4b24071170 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f4b2c00b5c0 tx=0x7f4b2c005960 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:21:09.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:09.410+0000 7f4b37e00700 1 -- 172.21.15.62:0/2239219295 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f4b30057c30 con 0x7f4b40106b80 2024-04-20T22:21:09.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:09.658+0000 7f4b45000700 1 -- 172.21.15.62:0/2239219295 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f4b401a1650 con 0x7f4b2406ecc0 2024-04-20T22:21:09.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:09.675+0000 7f4b37e00700 1 -- 172.21.15.62:0/2239219295 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f4b401a1650 con 0x7f4b2406ecc0 2024-04-20T22:21:09.675 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T22:21:09.675 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (46m) 7m ago 60m 14.6M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T22:21:09.675 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (61m) 7m ago 61m 11.6M - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T22:21:09.675 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (59m) 7m ago 59m 11.2M - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T22:21:09.675 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (43m) 7m ago 61m 6660k - 19.0.0-3133-ga8c84f59 76bc6b91b726 2825d5ba11c6 2024-04-20T22:21:09.675 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (42m) 7m ago 59m 6665k - 19.0.0-3133-ga8c84f59 76bc6b91b726 20f746fa1bba 2024-04-20T22:21:09.676 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (46m) 7m ago 60m 75.5M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T22:21:09.676 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (38m) 7m ago 53m 22.9M - 19.0.0-3133-ga8c84f59 76bc6b91b726 de80bc90a683 2024-04-20T22:21:09.676 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (38m) 7m ago 53m 23.0M - 19.0.0-3133-ga8c84f59 76bc6b91b726 cd793b7ee98e 2024-04-20T22:21:09.676 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (23m) 7m ago 53m 18.5M - 19.0.0-3133-ga8c84f59 76bc6b91b726 0e7b0bc4d1ab 2024-04-20T22:21:09.676 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (7m) 7m ago 53m 11.4M - 19.0.0-3133-ga8c84f59 76bc6b91b726 0af732ac9e06 2024-04-20T22:21:09.676 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (49m) 7m ago 62m 517M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T22:21:09.676 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (48m) 7m ago 59m 460M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T22:21:09.676 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (44m) 7m ago 62m 86.3M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 019eb1479485 2024-04-20T22:21:09.676 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (44m) 7m ago 59m 58.0M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 55aac560aab6 2024-04-20T22:21:09.676 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (48m) 7m ago 61m 8766k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T22:21:09.676 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (48m) 7m ago 59m 8711k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T22:21:09.676 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (42m) 7m ago 58m 91.3M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 ee2bb901f26e 2024-04-20T22:21:09.676 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (41m) 7m ago 57m 94.9M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 9075152d4162 2024-04-20T22:21:09.676 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (41m) 7m ago 57m 78.5M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 2d4f3e38dc0e 2024-04-20T22:21:09.676 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (40m) 7m ago 56m 94.1M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 8f97b17ae0eb 2024-04-20T22:21:09.676 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (40m) 7m ago 56m 96.1M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 653d8079967d 2024-04-20T22:21:09.677 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (39m) 7m ago 55m 90.9M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 596dded14c12 2024-04-20T22:21:09.677 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (47m) 7m ago 60m 90.4M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T22:21:09.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:09.678+0000 7f4b45000700 1 -- 172.21.15.62:0/2239219295 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f4b2406ecc0 msgr2=0x7f4b24071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:21:09.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:09.678+0000 7f4b45000700 1 --2- 172.21.15.62:0/2239219295 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f4b2406ecc0 0x7f4b24071170 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f4b2c00b5c0 tx=0x7f4b2c005960 comp rx=0 tx=0).stop 2024-04-20T22:21:09.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:09.678+0000 7f4b45000700 1 -- 172.21.15.62:0/2239219295 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4b40106b80 msgr2=0x7f4b4019c1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:21:09.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:09.678+0000 7f4b45000700 1 --2- 172.21.15.62:0/2239219295 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4b40106b80 0x7f4b4019c1e0 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f4b3000d8d0 tx=0x7f4b3000dc90 comp rx=0 tx=0).stop 2024-04-20T22:21:09.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:09.679+0000 7f4b45000700 1 -- 172.21.15.62:0/2239219295 shutdown_connections 2024-04-20T22:21:09.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:09.679+0000 7f4b45000700 1 --2- 172.21.15.62:0/2239219295 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f4b2406ecc0 0x7f4b24071170 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:21:09.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:09.679+0000 7f4b45000700 1 --2- 172.21.15.62:0/2239219295 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4b40106b80 0x7f4b4019c1e0 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:21:09.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:09.679+0000 7f4b45000700 1 --2- 172.21.15.62:0/2239219295 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b4010c660 0x7f4b4019c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:21:09.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:09.679+0000 7f4b45000700 1 -- 172.21.15.62:0/2239219295 >> 172.21.15.62:0/2239219295 conn(0x7f4b40075e90 msgr2=0x7f4b400778d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:21:09.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:09.679+0000 7f4b45000700 1 -- 172.21.15.62:0/2239219295 shutdown_connections 2024-04-20T22:21:09.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:09.679+0000 7f4b45000700 1 -- 172.21.15.62:0/2239219295 wait complete. 2024-04-20T22:21:09.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:21:09 smithi062 ceph-mon[76322]: from='client.36412 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:21:09.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:21:09 smithi062 ceph-mon[76322]: from='client.36416 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:21:09.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:21:09 smithi062 ceph-mon[76322]: pgmap v1379: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:21:09.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:09.841+0000 7fe16ee00700 1 -- 172.21.15.62:0/3998435069 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe168101420 msgr2=0x7fe1681017f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:21:09.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:09.841+0000 7fe16ee00700 1 --2- 172.21.15.62:0/3998435069 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe168101420 0x7fe1681017f0 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7fe158009910 tx=0x7fe158009c20 comp rx=0 tx=0).stop 2024-04-20T22:21:09.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:09.841+0000 7fe16ee00700 1 -- 172.21.15.62:0/3998435069 shutdown_connections 2024-04-20T22:21:09.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:09.841+0000 7fe16ee00700 1 --2- 172.21.15.62:0/3998435069 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe168101d30 0x7fe16810a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:21:09.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:09.841+0000 7fe16ee00700 1 --2- 172.21.15.62:0/3998435069 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe168101420 0x7fe1681017f0 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:21:09.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:09.841+0000 7fe16ee00700 1 -- 172.21.15.62:0/3998435069 >> 172.21.15.62:0/3998435069 conn(0x7fe1680fac80 msgr2=0x7fe1680fd090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:21:09.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:09.841+0000 7fe16ee00700 1 -- 172.21.15.62:0/3998435069 shutdown_connections 2024-04-20T22:21:09.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:09.841+0000 7fe16ee00700 1 -- 172.21.15.62:0/3998435069 wait complete. 2024-04-20T22:21:09.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:09.842+0000 7fe16ee00700 1 Processor -- start 2024-04-20T22:21:09.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:09.842+0000 7fe16ee00700 1 -- start start 2024-04-20T22:21:09.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:09.842+0000 7fe16ee00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe168101420 0x7fe168198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:21:09.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:09.843+0000 7fe16ee00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe168101d30 0x7fe1681985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:21:09.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:09.843+0000 7fe16ee00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe168198c10 con 0x7fe168101420 2024-04-20T22:21:09.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:09.843+0000 7fe16ee00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe168198d50 con 0x7fe168101d30 2024-04-20T22:21:09.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:09.843+0000 7fe167e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe168101420 0x7fe168198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:21:09.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:09.843+0000 7fe167e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe168101420 0x7fe168198080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51670/0 (socket says 172.21.15.62:51670) 2024-04-20T22:21:09.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:09.843+0000 7fe167e00700 1 -- 172.21.15.62:0/3313185739 learned_addr learned my addr 172.21.15.62:0/3313185739 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:21:09.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:09.843+0000 7fe167e00700 1 -- 172.21.15.62:0/3313185739 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe168101d30 msgr2=0x7fe1681985c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:21:09.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:09.843+0000 7fe167e00700 1 --2- 172.21.15.62:0/3313185739 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe168101d30 0x7fe1681985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:21:09.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:09.843+0000 7fe167e00700 1 -- 172.21.15.62:0/3313185739 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1580095f0 con 0x7fe168101420 2024-04-20T22:21:09.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:09.844+0000 7fe167e00700 1 --2- 172.21.15.62:0/3313185739 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe168101420 0x7fe168198080 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7fe158006010 tx=0x7fe15800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:21:09.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:09.844+0000 7fe164c00700 1 -- 172.21.15.62:0/3313185739 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe15801d070 con 0x7fe168101420 2024-04-20T22:21:09.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:09.844+0000 7fe164c00700 1 -- 172.21.15.62:0/3313185739 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe158004fb0 con 0x7fe168101420 2024-04-20T22:21:09.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:09.844+0000 7fe164c00700 1 -- 172.21.15.62:0/3313185739 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe158022ea0 con 0x7fe168101420 2024-04-20T22:21:09.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:09.844+0000 7fe16ee00700 1 -- 172.21.15.62:0/3313185739 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe16819cb40 con 0x7fe168101420 2024-04-20T22:21:09.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:09.844+0000 7fe16ee00700 1 -- 172.21.15.62:0/3313185739 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe16819d030 con 0x7fe168101420 2024-04-20T22:21:09.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:09.845+0000 7fe16ee00700 1 -- 172.21.15.62:0/3313185739 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe168109d60 con 0x7fe168101420 2024-04-20T22:21:09.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:09.846+0000 7fe164c00700 1 -- 172.21.15.62:0/3313185739 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe158022470 con 0x7fe168101420 2024-04-20T22:21:09.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:09.847+0000 7fe164c00700 1 --2- 172.21.15.62:0/3313185739 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe14c06ecc0 0x7fe14c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:21:09.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:09.847+0000 7fe164c00700 1 -- 172.21.15.62:0/3313185739 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fe1580904a0 con 0x7fe168101420 2024-04-20T22:21:09.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:09.847+0000 7fe167400700 1 --2- 172.21.15.62:0/3313185739 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe14c06ecc0 0x7fe14c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:21:09.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:09.848+0000 7fe167400700 1 --2- 172.21.15.62:0/3313185739 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe14c06ecc0 0x7fe14c071170 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fe154005fd0 tx=0x7fe154005d00 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:21:09.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:09.852+0000 7fe164c00700 1 -- 172.21.15.62:0/3313185739 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fe158059260 con 0x7fe168101420 2024-04-20T22:21:09.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:21:09 smithi189 ceph-mon[57022]: from='client.36412 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:21:09.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:21:09 smithi189 ceph-mon[57022]: from='client.36416 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:21:09.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:21:09 smithi189 ceph-mon[57022]: pgmap v1379: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:21:10.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:10.184+0000 7fe16ee00700 1 -- 172.21.15.62:0/3313185739 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fe1681994f0 con 0x7fe168101420 2024-04-20T22:21:10.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:10.185+0000 7fe164c00700 1 -- 172.21.15.62:0/3313185739 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7fe158022720 con 0x7fe168101420 2024-04-20T22:21:10.185 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:21:10.185 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T22:21:10.185 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:21:10.185 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:21:10.186 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T22:21:10.186 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:21:10.186 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:21:10.186 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T22:21:10.186 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 6 2024-04-20T22:21:10.186 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:21:10.186 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T22:21:10.186 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 4 2024-04-20T22:21:10.186 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:21:10.186 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T22:21:10.186 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 14 2024-04-20T22:21:10.186 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T22:21:10.186 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:21:10.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:10.190+0000 7fe16ee00700 1 -- 172.21.15.62:0/3313185739 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe14c06ecc0 msgr2=0x7fe14c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:21:10.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:10.190+0000 7fe16ee00700 1 --2- 172.21.15.62:0/3313185739 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe14c06ecc0 0x7fe14c071170 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fe154005fd0 tx=0x7fe154005d00 comp rx=0 tx=0).stop 2024-04-20T22:21:10.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:10.190+0000 7fe16ee00700 1 -- 172.21.15.62:0/3313185739 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe168101420 msgr2=0x7fe168198080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:21:10.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:10.190+0000 7fe16ee00700 1 --2- 172.21.15.62:0/3313185739 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe168101420 0x7fe168198080 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7fe158006010 tx=0x7fe15800be20 comp rx=0 tx=0).stop 2024-04-20T22:21:10.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:10.190+0000 7fe16ee00700 1 -- 172.21.15.62:0/3313185739 shutdown_connections 2024-04-20T22:21:10.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:10.190+0000 7fe16ee00700 1 --2- 172.21.15.62:0/3313185739 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe14c06ecc0 0x7fe14c071170 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:21:10.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:10.190+0000 7fe16ee00700 1 --2- 172.21.15.62:0/3313185739 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe168101420 0x7fe168198080 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:21:10.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:10.190+0000 7fe16ee00700 1 --2- 172.21.15.62:0/3313185739 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe168101d30 0x7fe1681985c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:21:10.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:10.190+0000 7fe16ee00700 1 -- 172.21.15.62:0/3313185739 >> 172.21.15.62:0/3313185739 conn(0x7fe1680fac80 msgr2=0x7fe1680ff840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:21:10.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:10.190+0000 7fe16ee00700 1 -- 172.21.15.62:0/3313185739 shutdown_connections 2024-04-20T22:21:10.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:10.190+0000 7fe16ee00700 1 -- 172.21.15.62:0/3313185739 wait complete. 2024-04-20T22:21:10.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:10.354+0000 7f442ae00700 1 -- 172.21.15.62:0/6266542 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4424100e80 msgr2=0x7f4424104e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:21:10.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:10.354+0000 7f442ae00700 1 --2- 172.21.15.62:0/6266542 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4424100e80 0x7f4424104e50 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f4410009910 tx=0x7f4410009c20 comp rx=0 tx=0).stop 2024-04-20T22:21:10.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:10.355+0000 7f442ae00700 1 -- 172.21.15.62:0/6266542 shutdown_connections 2024-04-20T22:21:10.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:10.355+0000 7f442ae00700 1 --2- 172.21.15.62:0/6266542 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4424100e80 0x7f4424104e50 unknown :-1 s=CLOSED pgs=1177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:21:10.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:10.355+0000 7f442ae00700 1 --2- 172.21.15.62:0/6266542 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f44241004e0 0x7f44241008b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:21:10.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:10.355+0000 7f442ae00700 1 -- 172.21.15.62:0/6266542 >> 172.21.15.62:0/6266542 conn(0x7f44240fbda0 msgr2=0x7f44240fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:21:10.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:10.355+0000 7f442ae00700 1 -- 172.21.15.62:0/6266542 shutdown_connections 2024-04-20T22:21:10.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:10.355+0000 7f442ae00700 1 -- 172.21.15.62:0/6266542 wait complete. 2024-04-20T22:21:10.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:10.356+0000 7f442ae00700 1 Processor -- start 2024-04-20T22:21:10.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:10.356+0000 7f442ae00700 1 -- start start 2024-04-20T22:21:10.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:10.356+0000 7f442ae00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f44241004e0 0x7f44241980f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:21:10.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:10.356+0000 7f442ae00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4424100e80 0x7f4424198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:21:10.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:10.356+0000 7f442ae00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4424198d10 con 0x7f44241004e0 2024-04-20T22:21:10.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:10.356+0000 7f442ae00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f442419caa0 con 0x7f4424100e80 2024-04-20T22:21:10.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:10.357+0000 7f4423e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f44241004e0 0x7f44241980f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:21:10.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:10.357+0000 7f4423e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f44241004e0 0x7f44241980f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51702/0 (socket says 172.21.15.62:51702) 2024-04-20T22:21:10.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:10.357+0000 7f4423e00700 1 -- 172.21.15.62:0/4146587967 learned_addr learned my addr 172.21.15.62:0/4146587967 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:21:10.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:10.357+0000 7f4423400700 1 --2- 172.21.15.62:0/4146587967 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4424100e80 0x7f4424198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:21:10.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:10.357+0000 7f4423e00700 1 -- 172.21.15.62:0/4146587967 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4424100e80 msgr2=0x7f4424198630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:21:10.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:10.357+0000 7f4423e00700 1 --2- 172.21.15.62:0/4146587967 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4424100e80 0x7f4424198630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:21:10.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:10.357+0000 7f4423e00700 1 -- 172.21.15.62:0/4146587967 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44100095f0 con 0x7f44241004e0 2024-04-20T22:21:10.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:10.357+0000 7f4423400700 1 --2- 172.21.15.62:0/4146587967 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4424100e80 0x7f4424198630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T22:21:10.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:10.358+0000 7f4423e00700 1 --2- 172.21.15.62:0/4146587967 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f44241004e0 0x7f44241980f0 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f441400c8a0 tx=0x7f441400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:21:10.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:10.358+0000 7f4420c00700 1 -- 172.21.15.62:0/4146587967 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f441400cea0 con 0x7f44241004e0 2024-04-20T22:21:10.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:10.358+0000 7f442ae00700 1 -- 172.21.15.62:0/4146587967 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f442419cd80 con 0x7f44241004e0 2024-04-20T22:21:10.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:10.358+0000 7f4420c00700 1 -- 172.21.15.62:0/4146587967 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4414004830 con 0x7f44241004e0 2024-04-20T22:21:10.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:10.358+0000 7f4420c00700 1 -- 172.21.15.62:0/4146587967 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f44140055d0 con 0x7f44241004e0 2024-04-20T22:21:10.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:10.358+0000 7f442ae00700 1 -- 172.21.15.62:0/4146587967 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f442419d2d0 con 0x7f44241004e0 2024-04-20T22:21:10.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:10.359+0000 7f442ae00700 1 -- 172.21.15.62:0/4146587967 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44241045d0 con 0x7f44241004e0 2024-04-20T22:21:10.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:10.360+0000 7f4420c00700 1 -- 172.21.15.62:0/4146587967 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f44140049a0 con 0x7f44241004e0 2024-04-20T22:21:10.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:10.361+0000 7f4420c00700 1 --2- 172.21.15.62:0/4146587967 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f440806ecc0 0x7f4408071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:21:10.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:10.361+0000 7f4420c00700 1 -- 172.21.15.62:0/4146587967 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f441408ef20 con 0x7f44241004e0 2024-04-20T22:21:10.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:10.362+0000 7f4423400700 1 --2- 172.21.15.62:0/4146587967 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f440806ecc0 0x7f4408071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:21:10.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:10.362+0000 7f4423400700 1 --2- 172.21.15.62:0/4146587967 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f440806ecc0 0x7f4408071170 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f4410006010 tx=0x7f4410005960 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:21:10.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:10.367+0000 7f4420c00700 1 -- 172.21.15.62:0/4146587967 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f4414057c30 con 0x7f44241004e0 2024-04-20T22:21:10.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:10.659+0000 7f442ae00700 1 -- 172.21.15.62:0/4146587967 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f442419d5a0 con 0x7f44241004e0 2024-04-20T22:21:10.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:10.660+0000 7f4420c00700 1 -- 172.21.15.62:0/4146587967 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 29 v29) v1 ==== 76+0+1887 (secure 0 0 0) 0x7f4414004c50 con 0x7f44241004e0 2024-04-20T22:21:10.661 INFO:teuthology.orchestra.run.smithi062.stdout:e29 2024-04-20T22:21:10.661 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T22:21:10.661 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:21:10.661 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T22:21:10.661 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:21:10.661 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T22:21:10.661 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T22:21:10.661 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 20 2024-04-20T22:21:10.661 INFO:teuthology.orchestra.run.smithi062.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-20T22:21:10.662 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T22:21:10.662 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T21:42:01.559061+0000 2024-04-20T22:21:10.662 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T22:21:10.662 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T22:21:10.662 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T22:21:10.662 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T22:21:10.662 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T22:21:10.662 INFO:teuthology.orchestra.run.smithi062.stdout:max_xattr_size 65536 2024-04-20T22:21:10.662 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T22:21:10.662 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T22:21:10.662 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 81 2024-04-20T22:21:10.662 INFO:teuthology.orchestra.run.smithi062.stdout:snap_rstat 1 2024-04-20T22:21:10.662 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:21:10.662 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 2 2024-04-20T22:21:10.663 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1 2024-04-20T22:21:10.663 INFO:teuthology.orchestra.run.smithi062.stdout:up {} 2024-04-20T22:21:10.663 INFO:teuthology.orchestra.run.smithi062.stdout:failed 0,1 2024-04-20T22:21:10.663 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T22:21:10.663 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T22:21:10.663 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T22:21:10.663 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T22:21:10.663 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data enabled 2024-04-20T22:21:10.663 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T22:21:10.663 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T22:21:10.663 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 1 2024-04-20T22:21:10.663 INFO:teuthology.orchestra.run.smithi062.stdout:qdb_cluster leader: 0 members: 2024-04-20T22:21:10.663 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:21:10.663 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:21:10.663 INFO:teuthology.orchestra.run.smithi062.stdout:Standby daemons: 2024-04-20T22:21:10.663 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:21:10.664 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ryqblb{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6826/4113916205,v1:172.21.15.62:6827/4113916205] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:21:10.664 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ozizwi{-1:34476} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6828/3801975165,v1:172.21.15.62:6829/3801975165] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:21:10.664 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.dvimer{-1:44727} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6824/3576699232,v1:172.21.15.189:6825/3576699232] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:21:10.664 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.xbgspw{-1:45075} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6826/212905792,v1:172.21.15.189:6827/212905792] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:21:10.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:10.664+0000 7f442ae00700 1 -- 172.21.15.62:0/4146587967 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f440806ecc0 msgr2=0x7f4408071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:21:10.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:10.664+0000 7f442ae00700 1 --2- 172.21.15.62:0/4146587967 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f440806ecc0 0x7f4408071170 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f4410006010 tx=0x7f4410005960 comp rx=0 tx=0).stop 2024-04-20T22:21:10.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:10.664+0000 7f442ae00700 1 -- 172.21.15.62:0/4146587967 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f44241004e0 msgr2=0x7f44241980f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:21:10.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:10.664+0000 7f442ae00700 1 --2- 172.21.15.62:0/4146587967 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f44241004e0 0x7f44241980f0 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f441400c8a0 tx=0x7f441400cc60 comp rx=0 tx=0).stop 2024-04-20T22:21:10.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:10.664+0000 7f442ae00700 1 -- 172.21.15.62:0/4146587967 shutdown_connections 2024-04-20T22:21:10.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:10.664+0000 7f442ae00700 1 --2- 172.21.15.62:0/4146587967 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f440806ecc0 0x7f4408071170 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:21:10.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:10.664+0000 7f442ae00700 1 --2- 172.21.15.62:0/4146587967 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f44241004e0 0x7f44241980f0 unknown :-1 s=CLOSED pgs=1178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:21:10.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:10.664+0000 7f442ae00700 1 --2- 172.21.15.62:0/4146587967 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4424100e80 0x7f4424198630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:21:10.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:10.664+0000 7f442ae00700 1 -- 172.21.15.62:0/4146587967 >> 172.21.15.62:0/4146587967 conn(0x7f44240fbda0 msgr2=0x7f44240fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:21:10.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:10.665+0000 7f442ae00700 1 -- 172.21.15.62:0/4146587967 shutdown_connections 2024-04-20T22:21:10.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:10.665+0000 7f442ae00700 1 -- 172.21.15.62:0/4146587967 wait complete. 2024-04-20T22:21:10.666 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 29 2024-04-20T22:21:10.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:21:10 smithi062 ceph-mon[76322]: from='client.36420 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:21:10.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:21:10 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/3313185739' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:21:10.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:10.829+0000 7f47afe00700 1 -- 172.21.15.62:0/4222518386 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f47b0103e50 msgr2=0x7f47b01042c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:21:10.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:10.829+0000 7f47afe00700 1 --2- 172.21.15.62:0/4222518386 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f47b0103e50 0x7f47b01042c0 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f479c009910 tx=0x7f479c009c20 comp rx=0 tx=0).stop 2024-04-20T22:21:10.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:10.830+0000 7f47afe00700 1 -- 172.21.15.62:0/4222518386 shutdown_connections 2024-04-20T22:21:10.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:10.830+0000 7f47afe00700 1 --2- 172.21.15.62:0/4222518386 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f47b0103e50 0x7f47b01042c0 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:21:10.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:10.830+0000 7f47afe00700 1 --2- 172.21.15.62:0/4222518386 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f47b01025f0 0x7f47b01029c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:21:10.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:10.830+0000 7f47afe00700 1 -- 172.21.15.62:0/4222518386 >> 172.21.15.62:0/4222518386 conn(0x7f47b00fddb0 msgr2=0x7f47b01001c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:21:10.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:10.830+0000 7f47afe00700 1 -- 172.21.15.62:0/4222518386 shutdown_connections 2024-04-20T22:21:10.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:10.830+0000 7f47afe00700 1 -- 172.21.15.62:0/4222518386 wait complete. 2024-04-20T22:21:10.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:10.831+0000 7f47afe00700 1 Processor -- start 2024-04-20T22:21:10.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:10.831+0000 7f47afe00700 1 -- start start 2024-04-20T22:21:10.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:10.831+0000 7f47afe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f47b01025f0 0x7f47b0197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:21:10.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:10.831+0000 7f47afe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f47b0103e50 0x7f47b0198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:21:10.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:10.831+0000 7f47afe00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47b0198af0 con 0x7f47b01025f0 2024-04-20T22:21:10.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:10.831+0000 7f47afe00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47b019c880 con 0x7f47b0103e50 2024-04-20T22:21:10.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:10.832+0000 7f47aea00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f47b01025f0 0x7f47b0197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:21:10.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:10.832+0000 7f47aea00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f47b01025f0 0x7f47b0197ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51708/0 (socket says 172.21.15.62:51708) 2024-04-20T22:21:10.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:10.832+0000 7f47aea00700 1 -- 172.21.15.62:0/1085682677 learned_addr learned my addr 172.21.15.62:0/1085682677 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:21:10.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:10.832+0000 7f47aea00700 1 -- 172.21.15.62:0/1085682677 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f47b0103e50 msgr2=0x7f47b0198410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:21:10.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:10.832+0000 7f47aea00700 1 --2- 172.21.15.62:0/1085682677 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f47b0103e50 0x7f47b0198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:21:10.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:10.832+0000 7f47aea00700 1 -- 172.21.15.62:0/1085682677 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f479c0095f0 con 0x7f47b01025f0 2024-04-20T22:21:10.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:10.832+0000 7f47aea00700 1 --2- 172.21.15.62:0/1085682677 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f47b01025f0 0x7f47b0197ed0 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f47a000d8d0 tx=0x7f47a000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:21:10.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:10.833+0000 7f47a7400700 1 -- 172.21.15.62:0/1085682677 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f47a0009940 con 0x7f47b01025f0 2024-04-20T22:21:10.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:10.833+0000 7f47a7400700 1 -- 172.21.15.62:0/1085682677 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f47a0010460 con 0x7f47b01025f0 2024-04-20T22:21:10.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:10.833+0000 7f47afe00700 1 -- 172.21.15.62:0/1085682677 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47b019cb60 con 0x7f47b01025f0 2024-04-20T22:21:10.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:10.833+0000 7f47a7400700 1 -- 172.21.15.62:0/1085682677 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f47a000f5f0 con 0x7f47b01025f0 2024-04-20T22:21:10.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:10.833+0000 7f47afe00700 1 -- 172.21.15.62:0/1085682677 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47b019d130 con 0x7f47b01025f0 2024-04-20T22:21:10.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:10.834+0000 7f47afe00700 1 -- 172.21.15.62:0/1085682677 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47b0062380 con 0x7f47b01025f0 2024-04-20T22:21:10.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:10.835+0000 7f47a7400700 1 -- 172.21.15.62:0/1085682677 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f47a00105d0 con 0x7f47b01025f0 2024-04-20T22:21:10.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:10.836+0000 7f47a7400700 1 --2- 172.21.15.62:0/1085682677 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f479406ecc0 0x7f4794071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:21:10.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:10.836+0000 7f47a7400700 1 -- 172.21.15.62:0/1085682677 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f47a008ee10 con 0x7f47b01025f0 2024-04-20T22:21:10.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:10.836+0000 7f47ae000700 1 --2- 172.21.15.62:0/1085682677 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f479406ecc0 0x7f4794071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:21:10.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:10.837+0000 7f47ae000700 1 --2- 172.21.15.62:0/1085682677 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f479406ecc0 0x7f4794071170 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f479c006010 tx=0x7f479c00b540 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:21:10.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:10.841+0000 7f47a7400700 1 -- 172.21.15.62:0/1085682677 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f47a0057ba0 con 0x7f47b01025f0 2024-04-20T22:21:10.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:21:10 smithi189 ceph-mon[57022]: from='client.36420 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:21:10.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:21:10 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/3313185739' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:21:11.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:11.086+0000 7f47afe00700 1 -- 172.21.15.62:0/1085682677 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f47b019d510 con 0x7f479406ecc0 2024-04-20T22:21:11.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:11.089+0000 7f47a7400700 1 -- 172.21.15.62:0/1085682677 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f47b019d510 con 0x7f479406ecc0 2024-04-20T22:21:11.090 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:21:11.090 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T22:21:11.090 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T22:21:11.090 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-20T22:21:11.090 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T22:21:11.090 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr", 2024-04-20T22:21:11.090 INFO:teuthology.orchestra.run.smithi062.stdout: "osd", 2024-04-20T22:21:11.090 INFO:teuthology.orchestra.run.smithi062.stdout: "mds", 2024-04-20T22:21:11.090 INFO:teuthology.orchestra.run.smithi062.stdout: "crash", 2024-04-20T22:21:11.090 INFO:teuthology.orchestra.run.smithi062.stdout: "mon" 2024-04-20T22:21:11.090 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T22:21:11.091 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "16/23 daemons upgraded", 2024-04-20T22:21:11.091 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading mds daemons", 2024-04-20T22:21:11.091 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T22:21:11.091 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:21:11.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:11.094+0000 7f47afe00700 1 -- 172.21.15.62:0/1085682677 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f479406ecc0 msgr2=0x7f4794071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:21:11.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:11.094+0000 7f47afe00700 1 --2- 172.21.15.62:0/1085682677 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f479406ecc0 0x7f4794071170 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f479c006010 tx=0x7f479c00b540 comp rx=0 tx=0).stop 2024-04-20T22:21:11.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:11.094+0000 7f47afe00700 1 -- 172.21.15.62:0/1085682677 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f47b01025f0 msgr2=0x7f47b0197ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:21:11.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:11.094+0000 7f47afe00700 1 --2- 172.21.15.62:0/1085682677 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f47b01025f0 0x7f47b0197ed0 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f47a000d8d0 tx=0x7f47a000dc90 comp rx=0 tx=0).stop 2024-04-20T22:21:11.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:11.094+0000 7f47afe00700 1 -- 172.21.15.62:0/1085682677 shutdown_connections 2024-04-20T22:21:11.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:11.094+0000 7f47afe00700 1 --2- 172.21.15.62:0/1085682677 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f479406ecc0 0x7f4794071170 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:21:11.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:11.094+0000 7f47afe00700 1 --2- 172.21.15.62:0/1085682677 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f47b01025f0 0x7f47b0197ed0 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:21:11.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:11.094+0000 7f47afe00700 1 --2- 172.21.15.62:0/1085682677 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f47b0103e50 0x7f47b0198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:21:11.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:11.094+0000 7f47afe00700 1 -- 172.21.15.62:0/1085682677 >> 172.21.15.62:0/1085682677 conn(0x7f47b00fddb0 msgr2=0x7f47b00fe680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:21:11.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:11.094+0000 7f47afe00700 1 -- 172.21.15.62:0/1085682677 shutdown_connections 2024-04-20T22:21:11.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:11.094+0000 7f47afe00700 1 -- 172.21.15.62:0/1085682677 wait complete. 2024-04-20T22:21:11.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:11.257+0000 7f5e85400700 1 -- 172.21.15.62:0/544685418 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e80073a00 msgr2=0x7f5e80110b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:21:11.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:11.257+0000 7f5e85400700 1 --2- 172.21.15.62:0/544685418 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e80073a00 0x7f5e80110b40 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7f5e6c009910 tx=0x7f5e6c009c20 comp rx=0 tx=0).stop 2024-04-20T22:21:11.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:11.257+0000 7f5e85400700 1 -- 172.21.15.62:0/544685418 shutdown_connections 2024-04-20T22:21:11.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:11.257+0000 7f5e85400700 1 --2- 172.21.15.62:0/544685418 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e80073a00 0x7f5e80110b40 unknown :-1 s=CLOSED pgs=1181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:21:11.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:11.257+0000 7f5e85400700 1 --2- 172.21.15.62:0/544685418 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e800730f0 0x7f5e800734c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:21:11.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:11.257+0000 7f5e85400700 1 -- 172.21.15.62:0/544685418 >> 172.21.15.62:0/544685418 conn(0x7f5e800fbb70 msgr2=0x7f5e800fdf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:21:11.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:11.257+0000 7f5e85400700 1 -- 172.21.15.62:0/544685418 shutdown_connections 2024-04-20T22:21:11.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:11.257+0000 7f5e85400700 1 -- 172.21.15.62:0/544685418 wait complete. 2024-04-20T22:21:11.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:11.258+0000 7f5e85400700 1 Processor -- start 2024-04-20T22:21:11.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:11.259+0000 7f5e85400700 1 -- start start 2024-04-20T22:21:11.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:11.259+0000 7f5e85400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e800730f0 0x7f5e8019c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:21:11.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:11.259+0000 7f5e85400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e80073a00 0x7f5e8019c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:21:11.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:11.259+0000 7f5e85400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e8019cde0 con 0x7f5e800730f0 2024-04-20T22:21:11.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:11.259+0000 7f5e85400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e801a0b20 con 0x7f5e80073a00 2024-04-20T22:21:11.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:11.259+0000 7f5e7fe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e800730f0 0x7f5e8019c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:21:11.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:11.259+0000 7f5e7fe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e800730f0 0x7f5e8019c1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51716/0 (socket says 172.21.15.62:51716) 2024-04-20T22:21:11.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:11.259+0000 7f5e7fe00700 1 -- 172.21.15.62:0/1313503001 learned_addr learned my addr 172.21.15.62:0/1313503001 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:21:11.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:11.260+0000 7f5e7f400700 1 --2- 172.21.15.62:0/1313503001 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e80073a00 0x7f5e8019c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:21:11.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:11.260+0000 7f5e7fe00700 1 -- 172.21.15.62:0/1313503001 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e80073a00 msgr2=0x7f5e8019c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:21:11.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:11.260+0000 7f5e7fe00700 1 --2- 172.21.15.62:0/1313503001 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e80073a00 0x7f5e8019c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:21:11.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:11.260+0000 7f5e7fe00700 1 -- 172.21.15.62:0/1313503001 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e6c0095f0 con 0x7f5e800730f0 2024-04-20T22:21:11.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:11.260+0000 7f5e7fe00700 1 --2- 172.21.15.62:0/1313503001 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e800730f0 0x7f5e8019c1c0 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f5e7000d8d0 tx=0x7f5e7000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:21:11.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:11.260+0000 7f5e7cc00700 1 -- 172.21.15.62:0/1313503001 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5e70009940 con 0x7f5e800730f0 2024-04-20T22:21:11.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:11.260+0000 7f5e7cc00700 1 -- 172.21.15.62:0/1313503001 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5e70010460 con 0x7f5e800730f0 2024-04-20T22:21:11.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:11.261+0000 7f5e7cc00700 1 -- 172.21.15.62:0/1313503001 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5e7000f5f0 con 0x7f5e800730f0 2024-04-20T22:21:11.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:11.261+0000 7f5e85400700 1 -- 172.21.15.62:0/1313503001 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e801a0e00 con 0x7f5e800730f0 2024-04-20T22:21:11.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:11.261+0000 7f5e85400700 1 -- 172.21.15.62:0/1313503001 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e801a1350 con 0x7f5e800730f0 2024-04-20T22:21:11.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:11.262+0000 7f5e85400700 1 -- 172.21.15.62:0/1313503001 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e8010e2c0 con 0x7f5e800730f0 2024-04-20T22:21:11.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:11.263+0000 7f5e7cc00700 1 -- 172.21.15.62:0/1313503001 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f5e700105d0 con 0x7f5e800730f0 2024-04-20T22:21:11.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:11.264+0000 7f5e7cc00700 1 --2- 172.21.15.62:0/1313503001 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5e6406ecc0 0x7f5e64071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:21:11.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:11.264+0000 7f5e7cc00700 1 -- 172.21.15.62:0/1313503001 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f5e7008ee10 con 0x7f5e800730f0 2024-04-20T22:21:11.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:11.264+0000 7f5e7f400700 1 --2- 172.21.15.62:0/1313503001 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5e6406ecc0 0x7f5e64071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:21:11.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:11.265+0000 7f5e7f400700 1 --2- 172.21.15.62:0/1313503001 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5e6406ecc0 0x7f5e64071170 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f5e8019d790 tx=0x7f5e6c01a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:21:11.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:11.269+0000 7f5e7cc00700 1 -- 172.21.15.62:0/1313503001 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f5e70057ba0 con 0x7f5e800730f0 2024-04-20T22:21:11.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:11.620+0000 7f5e85400700 1 -- 172.21.15.62:0/1313503001 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f5e80062380 con 0x7f5e800730f0 2024-04-20T22:21:11.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:11.621+0000 7f5e7cc00700 1 -- 172.21.15.62:0/1313503001 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7f5e7005b260 con 0x7f5e800730f0 2024-04-20T22:21:11.622 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-20T22:21:11.622 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-20T22:21:11.622 INFO:teuthology.orchestra.run.smithi062.stdout: host smithi189 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi189/config 2024-04-20T22:21:11.622 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-20T22:21:11.622 INFO:teuthology.orchestra.run.smithi062.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi189 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp11670v2q:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:21:11.622 INFO:teuthology.orchestra.run.smithi062.stdout:Traceback (most recent call last): 2024-04-20T22:21:11.622 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-20T22:21:11.622 INFO:teuthology.orchestra.run.smithi062.stdout: return _run_code(code, main_globals, None, 2024-04-20T22:21:11.622 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-20T22:21:11.623 INFO:teuthology.orchestra.run.smithi062.stdout: exec(code, run_globals) 2024-04-20T22:21:11.623 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5618, in 2024-04-20T22:21:11.623 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5606, in main 2024-04-20T22:21:11.623 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 408, in _infer_config 2024-04-20T22:21:11.623 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 323, in _infer_fsid 2024-04-20T22:21:11.623 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 436, in _infer_image 2024-04-20T22:21:11.623 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 310, in _validate_fsid 2024-04-20T22:21:11.623 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 3359, in command_ceph_volume 2024-04-20T22:21:11.623 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-20T22:21:11.623 INFO:teuthology.orchestra.run.smithi062.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi189 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp11670v2q:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:21:11.623 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-20T22:21:11.623 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is degraded 2024-04-20T22:21:11.623 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T22:21:11.623 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T22:21:11.624 INFO:teuthology.orchestra.run.smithi062.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-20T22:21:11.624 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-20T22:21:11.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:11.624+0000 7f5e85400700 1 -- 172.21.15.62:0/1313503001 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5e6406ecc0 msgr2=0x7f5e64071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:21:11.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:11.624+0000 7f5e85400700 1 --2- 172.21.15.62:0/1313503001 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5e6406ecc0 0x7f5e64071170 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f5e8019d790 tx=0x7f5e6c01a040 comp rx=0 tx=0).stop 2024-04-20T22:21:11.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:11.624+0000 7f5e85400700 1 -- 172.21.15.62:0/1313503001 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e800730f0 msgr2=0x7f5e8019c1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:21:11.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:11.624+0000 7f5e85400700 1 --2- 172.21.15.62:0/1313503001 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e800730f0 0x7f5e8019c1c0 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f5e7000d8d0 tx=0x7f5e7000dc90 comp rx=0 tx=0).stop 2024-04-20T22:21:11.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:11.624+0000 7f5e85400700 1 -- 172.21.15.62:0/1313503001 shutdown_connections 2024-04-20T22:21:11.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:11.624+0000 7f5e85400700 1 --2- 172.21.15.62:0/1313503001 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5e6406ecc0 0x7f5e64071170 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:21:11.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:11.624+0000 7f5e85400700 1 --2- 172.21.15.62:0/1313503001 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e800730f0 0x7f5e8019c1c0 unknown :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:21:11.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:11.624+0000 7f5e85400700 1 --2- 172.21.15.62:0/1313503001 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e80073a00 0x7f5e8019c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:21:11.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:11.624+0000 7f5e85400700 1 -- 172.21.15.62:0/1313503001 >> 172.21.15.62:0/1313503001 conn(0x7f5e800fbb70 msgr2=0x7f5e80102650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:21:11.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:11.624+0000 7f5e85400700 1 -- 172.21.15.62:0/1313503001 shutdown_connections 2024-04-20T22:21:11.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:11.624+0000 7f5e85400700 1 -- 172.21.15.62:0/1313503001 wait complete. 2024-04-20T22:21:11.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:21:11 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/4146587967' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:21:11.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:21:11 smithi062 ceph-mon[76322]: from='client.36432 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:21:11.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:21:11 smithi062 ceph-mon[76322]: pgmap v1380: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:21:11.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:21:11 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/4146587967' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:21:11.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:21:11 smithi189 ceph-mon[57022]: from='client.36432 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:21:11.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:21:11 smithi189 ceph-mon[57022]: pgmap v1380: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:21:12.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:21:12 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/1313503001' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:21:12.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:21:12 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/1313503001' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:21:13.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:21:13 smithi062 ceph-mon[76322]: pgmap v1381: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:21:13.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:21:13 smithi189 ceph-mon[57022]: pgmap v1381: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:21:15.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:21:15 smithi062 ceph-mon[76322]: pgmap v1382: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:21:15.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:21:15 smithi189 ceph-mon[57022]: pgmap v1382: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:21:17.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:21:17 smithi062 ceph-mon[76322]: pgmap v1383: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:21:17.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:21:17 smithi189 ceph-mon[57022]: pgmap v1383: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:21:19.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:21:19 smithi062 ceph-mon[76322]: pgmap v1384: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:21:19.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:21:19 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:21:19.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:21:19 smithi189 ceph-mon[57022]: pgmap v1384: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:21:19.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:21:19 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:21:21.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:21:21 smithi062 ceph-mon[76322]: pgmap v1385: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:21:21.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:21:21 smithi189 ceph-mon[57022]: pgmap v1385: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:21:23.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:21:23 smithi062 ceph-mon[76322]: pgmap v1386: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:21:23.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:21:23 smithi189 ceph-mon[57022]: pgmap v1386: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:21:25.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:21:25 smithi062 ceph-mon[76322]: pgmap v1387: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:21:25.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:21:25 smithi189 ceph-mon[57022]: pgmap v1387: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:21:27.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:21:27 smithi062 ceph-mon[76322]: pgmap v1388: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:21:27.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:21:27 smithi189 ceph-mon[57022]: pgmap v1388: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:21:29.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:21:29 smithi062 ceph-mon[76322]: pgmap v1389: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:21:29.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:21:29 smithi189 ceph-mon[57022]: pgmap v1389: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:21:31.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:21:31 smithi062 ceph-mon[76322]: pgmap v1390: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:21:31.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:21:31 smithi189 ceph-mon[57022]: pgmap v1390: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:21:33.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:21:33 smithi062 ceph-mon[76322]: pgmap v1391: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:21:33.816 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:21:33 smithi189 ceph-mon[57022]: pgmap v1391: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:21:34.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:21:34 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:21:34.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:21:34 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:21:35.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:21:35 smithi062 ceph-mon[76322]: pgmap v1392: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:21:35.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:21:35 smithi189 ceph-mon[57022]: pgmap v1392: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:21:37.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:21:37 smithi062 ceph-mon[76322]: pgmap v1393: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:21:37.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:21:37 smithi189 ceph-mon[57022]: pgmap v1393: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:21:39.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:21:39 smithi062 ceph-mon[76322]: pgmap v1394: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:21:39.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:21:39 smithi189 ceph-mon[57022]: pgmap v1394: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:21:41.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:21:41 smithi062 ceph-mon[76322]: pgmap v1395: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:21:41.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:41.801+0000 7fde89800700 1 -- 172.21.15.62:0/1301253833 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fde841004e0 msgr2=0x7fde841008b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:21:41.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:41.801+0000 7fde89800700 1 --2- 172.21.15.62:0/1301253833 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fde841004e0 0x7fde841008b0 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7fde74009910 tx=0x7fde74009c20 comp rx=0 tx=0).stop 2024-04-20T22:21:41.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:41.802+0000 7fde89800700 1 -- 172.21.15.62:0/1301253833 shutdown_connections 2024-04-20T22:21:41.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:41.802+0000 7fde89800700 1 --2- 172.21.15.62:0/1301253833 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde84100e80 0x7fde84104e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:21:41.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:41.802+0000 7fde89800700 1 --2- 172.21.15.62:0/1301253833 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fde841004e0 0x7fde841008b0 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:21:41.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:41.802+0000 7fde89800700 1 -- 172.21.15.62:0/1301253833 >> 172.21.15.62:0/1301253833 conn(0x7fde840fbda0 msgr2=0x7fde840fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:21:41.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:41.802+0000 7fde89800700 1 -- 172.21.15.62:0/1301253833 shutdown_connections 2024-04-20T22:21:41.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:41.802+0000 7fde89800700 1 -- 172.21.15.62:0/1301253833 wait complete. 2024-04-20T22:21:41.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:41.803+0000 7fde89800700 1 Processor -- start 2024-04-20T22:21:41.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:41.803+0000 7fde89800700 1 -- start start 2024-04-20T22:21:41.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:41.803+0000 7fde89800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde841004e0 0x7fde841980d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:21:41.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:41.803+0000 7fde89800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fde84100e80 0x7fde84198610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:21:41.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:41.803+0000 7fde89800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde84198c60 con 0x7fde84100e80 2024-04-20T22:21:41.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:41.803+0000 7fde89800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde84198da0 con 0x7fde841004e0 2024-04-20T22:21:41.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:41.804+0000 7fde7be00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fde84100e80 0x7fde84198610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:21:41.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:41.804+0000 7fde7be00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fde84100e80 0x7fde84198610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44956/0 (socket says 172.21.15.62:44956) 2024-04-20T22:21:41.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:41.804+0000 7fde7be00700 1 -- 172.21.15.62:0/3166831848 learned_addr learned my addr 172.21.15.62:0/3166831848 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:21:41.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:41.804+0000 7fde82a00700 1 --2- 172.21.15.62:0/3166831848 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde841004e0 0x7fde841980d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:21:41.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:41.804+0000 7fde7be00700 1 -- 172.21.15.62:0/3166831848 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde841004e0 msgr2=0x7fde841980d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:21:41.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:41.804+0000 7fde7be00700 1 --2- 172.21.15.62:0/3166831848 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde841004e0 0x7fde841980d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:21:41.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:41.804+0000 7fde7be00700 1 -- 172.21.15.62:0/3166831848 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde740095f0 con 0x7fde84100e80 2024-04-20T22:21:41.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:41.805+0000 7fde7be00700 1 --2- 172.21.15.62:0/3166831848 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fde84100e80 0x7fde84198610 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7fde6c00c960 tx=0x7fde6c00cd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:21:41.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:41.805+0000 7fde7b400700 1 -- 172.21.15.62:0/3166831848 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fde6c007a70 con 0x7fde84100e80 2024-04-20T22:21:41.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:41.805+0000 7fde7b400700 1 -- 172.21.15.62:0/3166831848 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fde6c007bd0 con 0x7fde84100e80 2024-04-20T22:21:41.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:41.805+0000 7fde89800700 1 -- 172.21.15.62:0/3166831848 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde8419cbf0 con 0x7fde84100e80 2024-04-20T22:21:41.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:41.805+0000 7fde7b400700 1 -- 172.21.15.62:0/3166831848 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fde6c0186c0 con 0x7fde84100e80 2024-04-20T22:21:41.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:41.805+0000 7fde89800700 1 -- 172.21.15.62:0/3166831848 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde8419d140 con 0x7fde84100e80 2024-04-20T22:21:41.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:41.806+0000 7fde89800700 1 -- 172.21.15.62:0/3166831848 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde841045d0 con 0x7fde84100e80 2024-04-20T22:21:41.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:41.807+0000 7fde7b400700 1 -- 172.21.15.62:0/3166831848 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fde6c00faa0 con 0x7fde84100e80 2024-04-20T22:21:41.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:41.809+0000 7fde7b400700 1 --2- 172.21.15.62:0/3166831848 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fde6806ecc0 0x7fde68071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:21:41.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:41.809+0000 7fde7b400700 1 -- 172.21.15.62:0/3166831848 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fde6c08eea0 con 0x7fde84100e80 2024-04-20T22:21:41.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:41.809+0000 7fde82a00700 1 --2- 172.21.15.62:0/3166831848 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fde6806ecc0 0x7fde68071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:21:41.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:41.810+0000 7fde82a00700 1 --2- 172.21.15.62:0/3166831848 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fde6806ecc0 0x7fde68071170 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fde74006010 tx=0x7fde7401a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:21:41.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:41.814+0000 7fde7b400700 1 -- 172.21.15.62:0/3166831848 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fde6c05b760 con 0x7fde84100e80 2024-04-20T22:21:41.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:21:41 smithi189 ceph-mon[57022]: pgmap v1395: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:21:42.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.074+0000 7fde89800700 1 -- 172.21.15.62:0/3166831848 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fde8402cca0 con 0x7fde6806ecc0 2024-04-20T22:21:42.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.077+0000 7fde7b400700 1 -- 172.21.15.62:0/3166831848 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fde8402cca0 con 0x7fde6806ecc0 2024-04-20T22:21:42.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.080+0000 7fde89800700 1 -- 172.21.15.62:0/3166831848 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fde6806ecc0 msgr2=0x7fde68071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:21:42.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.080+0000 7fde89800700 1 --2- 172.21.15.62:0/3166831848 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fde6806ecc0 0x7fde68071170 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fde74006010 tx=0x7fde7401a040 comp rx=0 tx=0).stop 2024-04-20T22:21:42.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.080+0000 7fde89800700 1 -- 172.21.15.62:0/3166831848 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fde84100e80 msgr2=0x7fde84198610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:21:42.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.080+0000 7fde89800700 1 --2- 172.21.15.62:0/3166831848 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fde84100e80 0x7fde84198610 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7fde6c00c960 tx=0x7fde6c00cd20 comp rx=0 tx=0).stop 2024-04-20T22:21:42.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.081+0000 7fde89800700 1 -- 172.21.15.62:0/3166831848 shutdown_connections 2024-04-20T22:21:42.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.081+0000 7fde89800700 1 --2- 172.21.15.62:0/3166831848 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fde6806ecc0 0x7fde68071170 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:21:42.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.081+0000 7fde89800700 1 --2- 172.21.15.62:0/3166831848 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde841004e0 0x7fde841980d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:21:42.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.081+0000 7fde89800700 1 --2- 172.21.15.62:0/3166831848 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fde84100e80 0x7fde84198610 unknown :-1 s=CLOSED pgs=1184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:21:42.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.081+0000 7fde89800700 1 -- 172.21.15.62:0/3166831848 >> 172.21.15.62:0/3166831848 conn(0x7fde840fbda0 msgr2=0x7fde840fe150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:21:42.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.081+0000 7fde89800700 1 -- 172.21.15.62:0/3166831848 shutdown_connections 2024-04-20T22:21:42.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.081+0000 7fde89800700 1 -- 172.21.15.62:0/3166831848 wait complete. 2024-04-20T22:21:42.098 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T22:21:42.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.252+0000 7f0c34400700 1 -- 172.21.15.62:0/3543396624 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c2c1038b0 msgr2=0x7f0c2c103c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:21:42.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.252+0000 7f0c34400700 1 --2- 172.21.15.62:0/3543396624 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c2c1038b0 0x7f0c2c103c80 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f0c2000c210 tx=0x7f0c2000c520 comp rx=0 tx=0).stop 2024-04-20T22:21:42.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.253+0000 7f0c34400700 1 -- 172.21.15.62:0/3543396624 shutdown_connections 2024-04-20T22:21:42.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.253+0000 7f0c34400700 1 --2- 172.21.15.62:0/3543396624 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c2c0ff350 0x7f0c2c0ff7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:21:42.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.253+0000 7f0c34400700 1 --2- 172.21.15.62:0/3543396624 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c2c1038b0 0x7f0c2c103c80 unknown :-1 s=CLOSED pgs=1185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:21:42.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.253+0000 7f0c34400700 1 -- 172.21.15.62:0/3543396624 >> 172.21.15.62:0/3543396624 conn(0x7f0c2c0748b0 msgr2=0x7f0c2c074cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:21:42.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.253+0000 7f0c34400700 1 -- 172.21.15.62:0/3543396624 shutdown_connections 2024-04-20T22:21:42.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.253+0000 7f0c34400700 1 -- 172.21.15.62:0/3543396624 wait complete. 2024-04-20T22:21:42.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.254+0000 7f0c34400700 1 Processor -- start 2024-04-20T22:21:42.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.254+0000 7f0c34400700 1 -- start start 2024-04-20T22:21:42.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.255+0000 7f0c34400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c2c0ff350 0x7f0c2c198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:21:42.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.255+0000 7f0c34400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c2c1038b0 0x7f0c2c1985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:21:42.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.255+0000 7f0c34400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c2c198cb0 con 0x7f0c2c0ff350 2024-04-20T22:21:42.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.255+0000 7f0c34400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c2c19ca40 con 0x7f0c2c1038b0 2024-04-20T22:21:42.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.255+0000 7f0c31a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c2c0ff350 0x7f0c2c198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:21:42.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.255+0000 7f0c31a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c2c0ff350 0x7f0c2c198090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44972/0 (socket says 172.21.15.62:44972) 2024-04-20T22:21:42.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.255+0000 7f0c31a00700 1 -- 172.21.15.62:0/3597140647 learned_addr learned my addr 172.21.15.62:0/3597140647 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:21:42.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.255+0000 7f0c31000700 1 --2- 172.21.15.62:0/3597140647 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c2c1038b0 0x7f0c2c1985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:21:42.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.255+0000 7f0c31a00700 1 -- 172.21.15.62:0/3597140647 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c2c1038b0 msgr2=0x7f0c2c1985d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:21:42.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.255+0000 7f0c31a00700 1 --2- 172.21.15.62:0/3597140647 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c2c1038b0 0x7f0c2c1985d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:21:42.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.255+0000 7f0c31a00700 1 -- 172.21.15.62:0/3597140647 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c2000ae10 con 0x7f0c2c0ff350 2024-04-20T22:21:42.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.255+0000 7f0c31000700 1 --2- 172.21.15.62:0/3597140647 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c2c1038b0 0x7f0c2c1985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T22:21:42.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.256+0000 7f0c31a00700 1 --2- 172.21.15.62:0/3597140647 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c2c0ff350 0x7f0c2c198090 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f0c20016040 tx=0x7f0c20008dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:21:42.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.256+0000 7f0c2a000700 1 -- 172.21.15.62:0/3597140647 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0c2000f070 con 0x7f0c2c0ff350 2024-04-20T22:21:42.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.256+0000 7f0c2a000700 1 -- 172.21.15.62:0/3597140647 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0c20008420 con 0x7f0c2c0ff350 2024-04-20T22:21:42.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.256+0000 7f0c2a000700 1 -- 172.21.15.62:0/3597140647 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0c2001e9a0 con 0x7f0c2c0ff350 2024-04-20T22:21:42.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.256+0000 7f0c34400700 1 -- 172.21.15.62:0/3597140647 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c2c19ccc0 con 0x7f0c2c0ff350 2024-04-20T22:21:42.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.256+0000 7f0c34400700 1 -- 172.21.15.62:0/3597140647 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c2c19d210 con 0x7f0c2c0ff350 2024-04-20T22:21:42.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.257+0000 7f0c34400700 1 -- 172.21.15.62:0/3597140647 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c2c192040 con 0x7f0c2c0ff350 2024-04-20T22:21:42.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.259+0000 7f0c2a000700 1 -- 172.21.15.62:0/3597140647 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f0c200074a0 con 0x7f0c2c0ff350 2024-04-20T22:21:42.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.260+0000 7f0c2a000700 1 --2- 172.21.15.62:0/3597140647 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f0c1406ecc0 0x7f0c14071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:21:42.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.260+0000 7f0c2a000700 1 -- 172.21.15.62:0/3597140647 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f0c200904e0 con 0x7f0c2c0ff350 2024-04-20T22:21:42.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.260+0000 7f0c31000700 1 --2- 172.21.15.62:0/3597140647 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f0c1406ecc0 0x7f0c14071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:21:42.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.261+0000 7f0c31000700 1 --2- 172.21.15.62:0/3597140647 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f0c1406ecc0 0x7f0c14071170 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f0c18005fd0 tx=0x7f0c18005ee0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:21:42.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.265+0000 7f0c2a000700 1 -- 172.21.15.62:0/3597140647 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f0c20059270 con 0x7f0c2c0ff350 2024-04-20T22:21:42.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.523+0000 7f0c34400700 1 -- 172.21.15.62:0/3597140647 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0c2c19d540 con 0x7f0c1406ecc0 2024-04-20T22:21:42.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.526+0000 7f0c2a000700 1 -- 172.21.15.62:0/3597140647 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f0c2c19d540 con 0x7f0c1406ecc0 2024-04-20T22:21:42.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.529+0000 7f0c34400700 1 -- 172.21.15.62:0/3597140647 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f0c1406ecc0 msgr2=0x7f0c14071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:21:42.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.529+0000 7f0c34400700 1 --2- 172.21.15.62:0/3597140647 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f0c1406ecc0 0x7f0c14071170 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f0c18005fd0 tx=0x7f0c18005ee0 comp rx=0 tx=0).stop 2024-04-20T22:21:42.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.529+0000 7f0c34400700 1 -- 172.21.15.62:0/3597140647 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c2c0ff350 msgr2=0x7f0c2c198090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:21:42.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.529+0000 7f0c34400700 1 --2- 172.21.15.62:0/3597140647 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c2c0ff350 0x7f0c2c198090 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f0c20016040 tx=0x7f0c20008dd0 comp rx=0 tx=0).stop 2024-04-20T22:21:42.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.529+0000 7f0c34400700 1 -- 172.21.15.62:0/3597140647 shutdown_connections 2024-04-20T22:21:42.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.529+0000 7f0c34400700 1 --2- 172.21.15.62:0/3597140647 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f0c1406ecc0 0x7f0c14071170 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:21:42.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.529+0000 7f0c34400700 1 --2- 172.21.15.62:0/3597140647 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c2c0ff350 0x7f0c2c198090 unknown :-1 s=CLOSED pgs=1186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:21:42.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.529+0000 7f0c34400700 1 --2- 172.21.15.62:0/3597140647 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c2c1038b0 0x7f0c2c1985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:21:42.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.529+0000 7f0c34400700 1 -- 172.21.15.62:0/3597140647 >> 172.21.15.62:0/3597140647 conn(0x7f0c2c0748b0 msgr2=0x7f0c2c0fea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:21:42.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.530+0000 7f0c34400700 1 -- 172.21.15.62:0/3597140647 shutdown_connections 2024-04-20T22:21:42.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.530+0000 7f0c34400700 1 -- 172.21.15.62:0/3597140647 wait complete. 2024-04-20T22:21:42.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.698+0000 7fbd5ea00700 1 -- 172.21.15.62:0/2386032081 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd58102470 msgr2=0x7fbd581028e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:21:42.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.698+0000 7fbd5ea00700 1 --2- 172.21.15.62:0/2386032081 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd58102470 0x7fbd581028e0 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7fbd4000b0a0 tx=0x7fbd4000b3b0 comp rx=0 tx=0).stop 2024-04-20T22:21:42.699 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:21:42 smithi062 ceph-mon[76322]: from='client.36440 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:21:42.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.698+0000 7fbd5ea00700 1 -- 172.21.15.62:0/2386032081 shutdown_connections 2024-04-20T22:21:42.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.698+0000 7fbd5ea00700 1 --2- 172.21.15.62:0/2386032081 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd58102470 0x7fbd581028e0 unknown :-1 s=CLOSED pgs=1187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:21:42.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.698+0000 7fbd5ea00700 1 --2- 172.21.15.62:0/2386032081 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd58108470 0x7fbd58108840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:21:42.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.698+0000 7fbd5ea00700 1 -- 172.21.15.62:0/2386032081 >> 172.21.15.62:0/2386032081 conn(0x7fbd580fdf90 msgr2=0x7fbd581003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:21:42.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.699+0000 7fbd5ea00700 1 -- 172.21.15.62:0/2386032081 shutdown_connections 2024-04-20T22:21:42.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.699+0000 7fbd5ea00700 1 -- 172.21.15.62:0/2386032081 wait complete. 2024-04-20T22:21:42.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.700+0000 7fbd5ea00700 1 Processor -- start 2024-04-20T22:21:42.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.700+0000 7fbd5ea00700 1 -- start start 2024-04-20T22:21:42.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.700+0000 7fbd5ea00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd58102470 0x7fbd58198270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:21:42.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.700+0000 7fbd5ea00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd58108470 0x7fbd581987b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:21:42.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.700+0000 7fbd5ea00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd58198e90 con 0x7fbd58108470 2024-04-20T22:21:42.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.700+0000 7fbd5ea00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd5819cc20 con 0x7fbd58102470 2024-04-20T22:21:42.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.701+0000 7fbd4f400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd58108470 0x7fbd581987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:21:42.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.701+0000 7fbd4f400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd58108470 0x7fbd581987b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44988/0 (socket says 172.21.15.62:44988) 2024-04-20T22:21:42.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.701+0000 7fbd4f400700 1 -- 172.21.15.62:0/139317261 learned_addr learned my addr 172.21.15.62:0/139317261 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:21:42.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.701+0000 7fbd57e00700 1 --2- 172.21.15.62:0/139317261 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd58102470 0x7fbd58198270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:21:42.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.701+0000 7fbd4f400700 1 -- 172.21.15.62:0/139317261 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd58102470 msgr2=0x7fbd58198270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:21:42.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.701+0000 7fbd4f400700 1 --2- 172.21.15.62:0/139317261 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd58102470 0x7fbd58198270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:21:42.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.701+0000 7fbd4f400700 1 -- 172.21.15.62:0/139317261 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd40009d00 con 0x7fbd58108470 2024-04-20T22:21:42.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.702+0000 7fbd4f400700 1 --2- 172.21.15.62:0/139317261 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd58108470 0x7fbd581987b0 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7fbd4000b070 tx=0x7fbd4000cdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:21:42.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.702+0000 7fbd55600700 1 -- 172.21.15.62:0/139317261 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbd4000e040 con 0x7fbd58108470 2024-04-20T22:21:42.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.702+0000 7fbd55600700 1 -- 172.21.15.62:0/139317261 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbd40003de0 con 0x7fbd58108470 2024-04-20T22:21:42.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.702+0000 7fbd55600700 1 -- 172.21.15.62:0/139317261 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbd4001d8e0 con 0x7fbd58108470 2024-04-20T22:21:42.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.702+0000 7fbd5ea00700 1 -- 172.21.15.62:0/139317261 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd5819cea0 con 0x7fbd58108470 2024-04-20T22:21:42.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.702+0000 7fbd5ea00700 1 -- 172.21.15.62:0/139317261 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd5819d390 con 0x7fbd58108470 2024-04-20T22:21:42.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.703+0000 7fbd5ea00700 1 -- 172.21.15.62:0/139317261 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd5810a970 con 0x7fbd58108470 2024-04-20T22:21:42.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.705+0000 7fbd55600700 1 -- 172.21.15.62:0/139317261 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fbd40004040 con 0x7fbd58108470 2024-04-20T22:21:42.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.705+0000 7fbd55600700 1 --2- 172.21.15.62:0/139317261 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fbd3c06ecc0 0x7fbd3c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:21:42.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.706+0000 7fbd55600700 1 -- 172.21.15.62:0/139317261 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fbd400905b0 con 0x7fbd58108470 2024-04-20T22:21:42.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.706+0000 7fbd57e00700 1 --2- 172.21.15.62:0/139317261 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fbd3c06ecc0 0x7fbd3c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:21:42.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.707+0000 7fbd57e00700 1 --2- 172.21.15.62:0/139317261 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fbd3c06ecc0 0x7fbd3c071170 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fbd48005950 tx=0x7fbd480058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:21:42.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.711+0000 7fbd55600700 1 -- 172.21.15.62:0/139317261 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fbd400593e0 con 0x7fbd58108470 2024-04-20T22:21:42.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.956+0000 7fbd5ea00700 1 -- 172.21.15.62:0/139317261 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fbd5819d6c0 con 0x7fbd3c06ecc0 2024-04-20T22:21:42.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.980+0000 7fbd55600700 1 -- 172.21.15.62:0/139317261 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fbd5819d6c0 con 0x7fbd3c06ecc0 2024-04-20T22:21:42.980 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T22:21:42.980 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (47m) 8m ago 61m 14.6M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T22:21:42.980 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (61m) 8m ago 61m 11.6M - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T22:21:42.980 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (60m) 8m ago 60m 11.2M - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T22:21:42.980 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (43m) 8m ago 61m 6660k - 19.0.0-3133-ga8c84f59 76bc6b91b726 2825d5ba11c6 2024-04-20T22:21:42.981 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (43m) 8m ago 60m 6665k - 19.0.0-3133-ga8c84f59 76bc6b91b726 20f746fa1bba 2024-04-20T22:21:42.981 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (46m) 8m ago 61m 75.5M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T22:21:42.981 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (39m) 8m ago 54m 22.9M - 19.0.0-3133-ga8c84f59 76bc6b91b726 de80bc90a683 2024-04-20T22:21:42.981 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (39m) 8m ago 54m 23.0M - 19.0.0-3133-ga8c84f59 76bc6b91b726 cd793b7ee98e 2024-04-20T22:21:42.981 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (23m) 8m ago 54m 18.5M - 19.0.0-3133-ga8c84f59 76bc6b91b726 0e7b0bc4d1ab 2024-04-20T22:21:42.981 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (8m) 8m ago 54m 11.4M - 19.0.0-3133-ga8c84f59 76bc6b91b726 0af732ac9e06 2024-04-20T22:21:42.981 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (50m) 8m ago 63m 517M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T22:21:42.981 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (49m) 8m ago 60m 460M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T22:21:42.981 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (45m) 8m ago 63m 86.3M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 019eb1479485 2024-04-20T22:21:42.981 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (44m) 8m ago 59m 58.0M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 55aac560aab6 2024-04-20T22:21:42.981 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (48m) 8m ago 61m 8766k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T22:21:42.981 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (48m) 8m ago 60m 8711k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T22:21:42.982 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (42m) 8m ago 59m 91.3M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 ee2bb901f26e 2024-04-20T22:21:42.982 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (42m) 8m ago 58m 94.9M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 9075152d4162 2024-04-20T22:21:42.982 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (41m) 8m ago 57m 78.5M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 2d4f3e38dc0e 2024-04-20T22:21:42.982 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (41m) 8m ago 57m 94.1M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 8f97b17ae0eb 2024-04-20T22:21:42.982 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (40m) 8m ago 56m 96.1M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 653d8079967d 2024-04-20T22:21:42.982 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (39m) 8m ago 56m 90.9M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 596dded14c12 2024-04-20T22:21:42.982 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (48m) 8m ago 60m 90.4M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T22:21:42.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.983+0000 7fbd5ea00700 1 -- 172.21.15.62:0/139317261 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fbd3c06ecc0 msgr2=0x7fbd3c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:21:42.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.983+0000 7fbd5ea00700 1 --2- 172.21.15.62:0/139317261 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fbd3c06ecc0 0x7fbd3c071170 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fbd48005950 tx=0x7fbd480058e0 comp rx=0 tx=0).stop 2024-04-20T22:21:42.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.983+0000 7fbd5ea00700 1 -- 172.21.15.62:0/139317261 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd58108470 msgr2=0x7fbd581987b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:21:42.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.983+0000 7fbd5ea00700 1 --2- 172.21.15.62:0/139317261 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd58108470 0x7fbd581987b0 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7fbd4000b070 tx=0x7fbd4000cdd0 comp rx=0 tx=0).stop 2024-04-20T22:21:42.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.983+0000 7fbd5ea00700 1 -- 172.21.15.62:0/139317261 shutdown_connections 2024-04-20T22:21:42.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.983+0000 7fbd5ea00700 1 --2- 172.21.15.62:0/139317261 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fbd3c06ecc0 0x7fbd3c071170 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:21:42.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.983+0000 7fbd5ea00700 1 --2- 172.21.15.62:0/139317261 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd58102470 0x7fbd58198270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:21:42.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.983+0000 7fbd5ea00700 1 --2- 172.21.15.62:0/139317261 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd58108470 0x7fbd581987b0 unknown :-1 s=CLOSED pgs=1188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:21:42.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.983+0000 7fbd5ea00700 1 -- 172.21.15.62:0/139317261 >> 172.21.15.62:0/139317261 conn(0x7fbd580fdf90 msgr2=0x7fbd580ff410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:21:42.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.983+0000 7fbd5ea00700 1 -- 172.21.15.62:0/139317261 shutdown_connections 2024-04-20T22:21:42.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:42.983+0000 7fbd5ea00700 1 -- 172.21.15.62:0/139317261 wait complete. 2024-04-20T22:21:42.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:21:42 smithi189 ceph-mon[57022]: from='client.36440 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:21:43.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:43.147+0000 7f12b5200700 1 -- 172.21.15.62:0/1551385087 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12b0100e80 msgr2=0x7f12b0104e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:21:43.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:43.147+0000 7f12b5200700 1 --2- 172.21.15.62:0/1551385087 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12b0100e80 0x7f12b0104e50 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f129c009910 tx=0x7f129c009c20 comp rx=0 tx=0).stop 2024-04-20T22:21:43.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:43.147+0000 7f12b5200700 1 -- 172.21.15.62:0/1551385087 shutdown_connections 2024-04-20T22:21:43.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:43.147+0000 7f12b5200700 1 --2- 172.21.15.62:0/1551385087 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12b0100e80 0x7f12b0104e50 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:21:43.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:43.147+0000 7f12b5200700 1 --2- 172.21.15.62:0/1551385087 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12b01004e0 0x7f12b01008b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:21:43.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:43.147+0000 7f12b5200700 1 -- 172.21.15.62:0/1551385087 >> 172.21.15.62:0/1551385087 conn(0x7f12b00fbda0 msgr2=0x7f12b00fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:21:43.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:43.147+0000 7f12b5200700 1 -- 172.21.15.62:0/1551385087 shutdown_connections 2024-04-20T22:21:43.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:43.147+0000 7f12b5200700 1 -- 172.21.15.62:0/1551385087 wait complete. 2024-04-20T22:21:43.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:43.148+0000 7f12b5200700 1 Processor -- start 2024-04-20T22:21:43.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:43.148+0000 7f12b5200700 1 -- start start 2024-04-20T22:21:43.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:43.149+0000 7f12b5200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12b01004e0 0x7f12b0198110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:21:43.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:43.149+0000 7f12b5200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12b0100e80 0x7f12b0198650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:21:43.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:43.149+0000 7f12b5200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12b0198d30 con 0x7f12b01004e0 2024-04-20T22:21:43.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:43.149+0000 7f12b5200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12b019cac0 con 0x7f12b0100e80 2024-04-20T22:21:43.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:43.149+0000 7f12ade00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12b01004e0 0x7f12b0198110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:21:43.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:43.149+0000 7f12ade00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12b01004e0 0x7f12b0198110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45006/0 (socket says 172.21.15.62:45006) 2024-04-20T22:21:43.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:43.149+0000 7f12ade00700 1 -- 172.21.15.62:0/2003022732 learned_addr learned my addr 172.21.15.62:0/2003022732 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:21:43.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:43.150+0000 7f12ad400700 1 --2- 172.21.15.62:0/2003022732 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12b0100e80 0x7f12b0198650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:21:43.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:43.150+0000 7f12ade00700 1 -- 172.21.15.62:0/2003022732 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12b0100e80 msgr2=0x7f12b0198650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:21:43.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:43.150+0000 7f12ade00700 1 --2- 172.21.15.62:0/2003022732 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12b0100e80 0x7f12b0198650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:21:43.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:43.150+0000 7f12ade00700 1 -- 172.21.15.62:0/2003022732 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f129c0095f0 con 0x7f12b01004e0 2024-04-20T22:21:43.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:43.150+0000 7f12ade00700 1 --2- 172.21.15.62:0/2003022732 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12b01004e0 0x7f12b0198110 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f12a000d8d0 tx=0x7f12a000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:21:43.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:43.150+0000 7f12a6a00700 1 -- 172.21.15.62:0/2003022732 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f12a0009940 con 0x7f12b01004e0 2024-04-20T22:21:43.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:43.150+0000 7f12a6a00700 1 -- 172.21.15.62:0/2003022732 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f12a0010460 con 0x7f12b01004e0 2024-04-20T22:21:43.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:43.151+0000 7f12a6a00700 1 -- 172.21.15.62:0/2003022732 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f12a000f5f0 con 0x7f12b01004e0 2024-04-20T22:21:43.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:43.151+0000 7f12b5200700 1 -- 172.21.15.62:0/2003022732 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12b019cda0 con 0x7f12b01004e0 2024-04-20T22:21:43.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:43.151+0000 7f12b5200700 1 -- 172.21.15.62:0/2003022732 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12b019d2f0 con 0x7f12b01004e0 2024-04-20T22:21:43.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:43.152+0000 7f12b5200700 1 -- 172.21.15.62:0/2003022732 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12b01045d0 con 0x7f12b01004e0 2024-04-20T22:21:43.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:43.153+0000 7f12a6a00700 1 -- 172.21.15.62:0/2003022732 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f12a00105d0 con 0x7f12b01004e0 2024-04-20T22:21:43.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:43.154+0000 7f12a6a00700 1 --2- 172.21.15.62:0/2003022732 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f129406ecc0 0x7f1294071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:21:43.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:43.154+0000 7f12a6a00700 1 -- 172.21.15.62:0/2003022732 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f12a008ee10 con 0x7f12b01004e0 2024-04-20T22:21:43.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:43.154+0000 7f12ad400700 1 --2- 172.21.15.62:0/2003022732 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f129406ecc0 0x7f1294071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:21:43.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:43.155+0000 7f12ad400700 1 --2- 172.21.15.62:0/2003022732 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f129406ecc0 0x7f1294071170 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f129c000c00 tx=0x7f129c005fd0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:21:43.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:43.159+0000 7f12a6a00700 1 -- 172.21.15.62:0/2003022732 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f12a0057ba0 con 0x7f12b01004e0 2024-04-20T22:21:43.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:43.498+0000 7f12b5200700 1 -- 172.21.15.62:0/2003022732 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f12b019d560 con 0x7f12b01004e0 2024-04-20T22:21:43.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:43.498+0000 7f12a6a00700 1 -- 172.21.15.62:0/2003022732 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f12a005b260 con 0x7f12b01004e0 2024-04-20T22:21:43.499 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:21:43.499 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T22:21:43.499 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:21:43.499 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:21:43.499 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T22:21:43.499 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:21:43.499 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:21:43.499 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T22:21:43.499 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 6 2024-04-20T22:21:43.499 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:21:43.499 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T22:21:43.499 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 4 2024-04-20T22:21:43.499 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:21:43.499 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T22:21:43.499 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 14 2024-04-20T22:21:43.500 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T22:21:43.500 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:21:43.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:43.502+0000 7f12b5200700 1 -- 172.21.15.62:0/2003022732 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f129406ecc0 msgr2=0x7f1294071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:21:43.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:43.502+0000 7f12b5200700 1 --2- 172.21.15.62:0/2003022732 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f129406ecc0 0x7f1294071170 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f129c000c00 tx=0x7f129c005fd0 comp rx=0 tx=0).stop 2024-04-20T22:21:43.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:43.502+0000 7f12b5200700 1 -- 172.21.15.62:0/2003022732 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12b01004e0 msgr2=0x7f12b0198110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:21:43.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:43.502+0000 7f12b5200700 1 --2- 172.21.15.62:0/2003022732 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12b01004e0 0x7f12b0198110 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f12a000d8d0 tx=0x7f12a000dc90 comp rx=0 tx=0).stop 2024-04-20T22:21:43.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:43.502+0000 7f12b5200700 1 -- 172.21.15.62:0/2003022732 shutdown_connections 2024-04-20T22:21:43.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:43.502+0000 7f12b5200700 1 --2- 172.21.15.62:0/2003022732 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f129406ecc0 0x7f1294071170 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:21:43.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:43.502+0000 7f12b5200700 1 --2- 172.21.15.62:0/2003022732 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12b01004e0 0x7f12b0198110 unknown :-1 s=CLOSED pgs=1190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:21:43.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:43.502+0000 7f12b5200700 1 --2- 172.21.15.62:0/2003022732 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12b0100e80 0x7f12b0198650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:21:43.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:43.502+0000 7f12b5200700 1 -- 172.21.15.62:0/2003022732 >> 172.21.15.62:0/2003022732 conn(0x7f12b00fbda0 msgr2=0x7f12b00fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:21:43.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:43.502+0000 7f12b5200700 1 -- 172.21.15.62:0/2003022732 shutdown_connections 2024-04-20T22:21:43.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:43.502+0000 7f12b5200700 1 -- 172.21.15.62:0/2003022732 wait complete. 2024-04-20T22:21:43.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:43.659+0000 7fb98ee00700 1 -- 172.21.15.62:0/3686791627 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb988106b80 msgr2=0x7fb988106ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:21:43.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:43.659+0000 7fb98ee00700 1 --2- 172.21.15.62:0/3686791627 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb988106b80 0x7fb988106ff0 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7fb974009910 tx=0x7fb974009c20 comp rx=0 tx=0).stop 2024-04-20T22:21:43.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:43.659+0000 7fb98ee00700 1 -- 172.21.15.62:0/3686791627 shutdown_connections 2024-04-20T22:21:43.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:43.659+0000 7fb98ee00700 1 --2- 172.21.15.62:0/3686791627 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb988106b80 0x7fb988106ff0 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:21:43.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:43.659+0000 7fb98ee00700 1 --2- 172.21.15.62:0/3686791627 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb98810c8f0 0x7fb98810ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:21:43.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:43.659+0000 7fb98ee00700 1 -- 172.21.15.62:0/3686791627 >> 172.21.15.62:0/3686791627 conn(0x7fb988076020 msgr2=0x7fb988078430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:21:43.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:43.659+0000 7fb98ee00700 1 -- 172.21.15.62:0/3686791627 shutdown_connections 2024-04-20T22:21:43.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:43.659+0000 7fb98ee00700 1 -- 172.21.15.62:0/3686791627 wait complete. 2024-04-20T22:21:43.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:43.660+0000 7fb98ee00700 1 Processor -- start 2024-04-20T22:21:43.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:43.660+0000 7fb98ee00700 1 -- start start 2024-04-20T22:21:43.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:43.661+0000 7fb98ee00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb988106b80 0x7fb98819c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:21:43.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:43.661+0000 7fb98ee00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb98810c8f0 0x7fb98819c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:21:43.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:43.661+0000 7fb98ee00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb98819d070 con 0x7fb98810c8f0 2024-04-20T22:21:43.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:43.661+0000 7fb98ee00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9881a0e00 con 0x7fb988106b80 2024-04-20T22:21:43.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:43.661+0000 7fb987400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb98810c8f0 0x7fb98819c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:21:43.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:43.661+0000 7fb987400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb98810c8f0 0x7fb98819c990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45022/0 (socket says 172.21.15.62:45022) 2024-04-20T22:21:43.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:43.661+0000 7fb987400700 1 -- 172.21.15.62:0/205300743 learned_addr learned my addr 172.21.15.62:0/205300743 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:21:43.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:43.661+0000 7fb987e00700 1 --2- 172.21.15.62:0/205300743 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb988106b80 0x7fb98819c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:21:43.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:43.662+0000 7fb987400700 1 -- 172.21.15.62:0/205300743 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb988106b80 msgr2=0x7fb98819c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:21:43.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:43.662+0000 7fb987400700 1 --2- 172.21.15.62:0/205300743 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb988106b80 0x7fb98819c450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:21:43.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:43.662+0000 7fb987400700 1 -- 172.21.15.62:0/205300743 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9740095f0 con 0x7fb98810c8f0 2024-04-20T22:21:43.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:43.662+0000 7fb987400700 1 --2- 172.21.15.62:0/205300743 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb98810c8f0 0x7fb98819c990 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7fb9740092e0 tx=0x7fb97400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:21:43.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:43.662+0000 7fb984c00700 1 -- 172.21.15.62:0/205300743 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb97401d070 con 0x7fb98810c8f0 2024-04-20T22:21:43.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:43.662+0000 7fb984c00700 1 -- 172.21.15.62:0/205300743 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb974005e00 con 0x7fb98810c8f0 2024-04-20T22:21:43.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:43.662+0000 7fb984c00700 1 -- 172.21.15.62:0/205300743 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb974017400 con 0x7fb98810c8f0 2024-04-20T22:21:43.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:43.662+0000 7fb98ee00700 1 -- 172.21.15.62:0/205300743 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9881a1080 con 0x7fb98810c8f0 2024-04-20T22:21:43.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:43.662+0000 7fb98ee00700 1 -- 172.21.15.62:0/205300743 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9881a1570 con 0x7fb98810c8f0 2024-04-20T22:21:43.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:43.664+0000 7fb98ee00700 1 -- 172.21.15.62:0/205300743 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb98804f2a0 con 0x7fb98810c8f0 2024-04-20T22:21:43.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:43.665+0000 7fb984c00700 1 -- 172.21.15.62:0/205300743 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fb97400f460 con 0x7fb98810c8f0 2024-04-20T22:21:43.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:43.665+0000 7fb984c00700 1 --2- 172.21.15.62:0/205300743 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb96c06ecc0 0x7fb96c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:21:43.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:43.666+0000 7fb984c00700 1 -- 172.21.15.62:0/205300743 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fb974090330 con 0x7fb98810c8f0 2024-04-20T22:21:43.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:43.666+0000 7fb987e00700 1 --2- 172.21.15.62:0/205300743 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb96c06ecc0 0x7fb96c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:21:43.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:43.667+0000 7fb987e00700 1 --2- 172.21.15.62:0/205300743 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb96c06ecc0 0x7fb96c071170 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fb978005950 tx=0x7fb9780058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:21:43.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:43.671+0000 7fb984c00700 1 -- 172.21.15.62:0/205300743 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fb9740590c0 con 0x7fb98810c8f0 2024-04-20T22:21:43.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:21:43 smithi062 ceph-mon[76322]: from='client.36444 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:21:43.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:21:43 smithi062 ceph-mon[76322]: from='client.36448 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:21:43.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:21:43 smithi062 ceph-mon[76322]: pgmap v1396: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:21:43.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:21:43 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/2003022732' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:21:43.890 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:21:43 smithi189 ceph-mon[57022]: from='client.36444 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:21:43.890 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:21:43 smithi189 ceph-mon[57022]: from='client.36448 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:21:43.890 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:21:43 smithi189 ceph-mon[57022]: pgmap v1396: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:21:43.890 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:21:43 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/2003022732' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:21:43.949 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:43.949+0000 7fb98ee00700 1 -- 172.21.15.62:0/205300743 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fb98819d7b0 con 0x7fb98810c8f0 2024-04-20T22:21:43.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:43.950+0000 7fb984c00700 1 -- 172.21.15.62:0/205300743 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 29 v29) v1 ==== 76+0+1887 (secure 0 0 0) 0x7fb97405c780 con 0x7fb98810c8f0 2024-04-20T22:21:43.951 INFO:teuthology.orchestra.run.smithi062.stdout:e29 2024-04-20T22:21:43.951 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T22:21:43.951 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:21:43.951 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T22:21:43.951 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:21:43.952 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T22:21:43.952 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T22:21:43.952 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 20 2024-04-20T22:21:43.952 INFO:teuthology.orchestra.run.smithi062.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-20T22:21:43.952 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T22:21:43.952 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T21:42:01.559061+0000 2024-04-20T22:21:43.952 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T22:21:43.952 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T22:21:43.952 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T22:21:43.952 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T22:21:43.952 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T22:21:43.952 INFO:teuthology.orchestra.run.smithi062.stdout:max_xattr_size 65536 2024-04-20T22:21:43.952 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T22:21:43.952 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T22:21:43.952 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 81 2024-04-20T22:21:43.952 INFO:teuthology.orchestra.run.smithi062.stdout:snap_rstat 1 2024-04-20T22:21:43.953 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:21:43.953 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 2 2024-04-20T22:21:43.953 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1 2024-04-20T22:21:43.953 INFO:teuthology.orchestra.run.smithi062.stdout:up {} 2024-04-20T22:21:43.953 INFO:teuthology.orchestra.run.smithi062.stdout:failed 0,1 2024-04-20T22:21:43.953 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T22:21:43.953 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T22:21:43.953 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T22:21:43.953 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T22:21:43.953 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data enabled 2024-04-20T22:21:43.953 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T22:21:43.953 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T22:21:43.953 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 1 2024-04-20T22:21:43.953 INFO:teuthology.orchestra.run.smithi062.stdout:qdb_cluster leader: 0 members: 2024-04-20T22:21:43.953 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:21:43.953 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:21:43.953 INFO:teuthology.orchestra.run.smithi062.stdout:Standby daemons: 2024-04-20T22:21:43.953 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:21:43.954 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ryqblb{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6826/4113916205,v1:172.21.15.62:6827/4113916205] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:21:43.954 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ozizwi{-1:34476} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6828/3801975165,v1:172.21.15.62:6829/3801975165] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:21:43.954 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.dvimer{-1:44727} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6824/3576699232,v1:172.21.15.189:6825/3576699232] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:21:43.954 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.xbgspw{-1:45075} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6826/212905792,v1:172.21.15.189:6827/212905792] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:21:43.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:43.953+0000 7fb98ee00700 1 -- 172.21.15.62:0/205300743 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb96c06ecc0 msgr2=0x7fb96c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:21:43.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:43.953+0000 7fb98ee00700 1 --2- 172.21.15.62:0/205300743 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb96c06ecc0 0x7fb96c071170 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fb978005950 tx=0x7fb9780058e0 comp rx=0 tx=0).stop 2024-04-20T22:21:43.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:43.953+0000 7fb98ee00700 1 -- 172.21.15.62:0/205300743 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb98810c8f0 msgr2=0x7fb98819c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:21:43.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:43.953+0000 7fb98ee00700 1 --2- 172.21.15.62:0/205300743 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb98810c8f0 0x7fb98819c990 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7fb9740092e0 tx=0x7fb97400be20 comp rx=0 tx=0).stop 2024-04-20T22:21:43.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:43.953+0000 7fb98ee00700 1 -- 172.21.15.62:0/205300743 shutdown_connections 2024-04-20T22:21:43.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:43.953+0000 7fb98ee00700 1 --2- 172.21.15.62:0/205300743 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb96c06ecc0 0x7fb96c071170 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:21:43.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:43.953+0000 7fb98ee00700 1 --2- 172.21.15.62:0/205300743 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb988106b80 0x7fb98819c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:21:43.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:43.953+0000 7fb98ee00700 1 --2- 172.21.15.62:0/205300743 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb98810c8f0 0x7fb98819c990 unknown :-1 s=CLOSED pgs=1192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:21:43.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:43.953+0000 7fb98ee00700 1 -- 172.21.15.62:0/205300743 >> 172.21.15.62:0/205300743 conn(0x7fb988076020 msgr2=0x7fb988077a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:21:43.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:43.953+0000 7fb98ee00700 1 -- 172.21.15.62:0/205300743 shutdown_connections 2024-04-20T22:21:43.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:43.953+0000 7fb98ee00700 1 -- 172.21.15.62:0/205300743 wait complete. 2024-04-20T22:21:43.955 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 29 2024-04-20T22:21:44.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:44.117+0000 7fa02f800700 1 -- 172.21.15.62:0/136485575 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa028106b80 msgr2=0x7fa028106ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:21:44.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:44.117+0000 7fa02f800700 1 --2- 172.21.15.62:0/136485575 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa028106b80 0x7fa028106ff0 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7fa014009910 tx=0x7fa014009c20 comp rx=0 tx=0).stop 2024-04-20T22:21:44.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:44.117+0000 7fa02f800700 1 -- 172.21.15.62:0/136485575 shutdown_connections 2024-04-20T22:21:44.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:44.117+0000 7fa02f800700 1 --2- 172.21.15.62:0/136485575 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa028106b80 0x7fa028106ff0 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:21:44.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:44.117+0000 7fa02f800700 1 --2- 172.21.15.62:0/136485575 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa02810c8f0 0x7fa02810ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:21:44.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:44.117+0000 7fa02f800700 1 -- 172.21.15.62:0/136485575 >> 172.21.15.62:0/136485575 conn(0x7fa028076060 msgr2=0x7fa028078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:21:44.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:44.117+0000 7fa02f800700 1 -- 172.21.15.62:0/136485575 shutdown_connections 2024-04-20T22:21:44.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:44.117+0000 7fa02f800700 1 -- 172.21.15.62:0/136485575 wait complete. 2024-04-20T22:21:44.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:44.118+0000 7fa02f800700 1 Processor -- start 2024-04-20T22:21:44.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:44.118+0000 7fa02f800700 1 -- start start 2024-04-20T22:21:44.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:44.119+0000 7fa02f800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa028106b80 0x7fa02819c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:21:44.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:44.119+0000 7fa02f800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa02810c8f0 0x7fa02819c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:21:44.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:44.119+0000 7fa02f800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa02819d060 con 0x7fa02810c8f0 2024-04-20T22:21:44.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:44.119+0000 7fa02f800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0281a0df0 con 0x7fa028106b80 2024-04-20T22:21:44.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:44.119+0000 7fa027e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa02810c8f0 0x7fa02819c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:21:44.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:44.119+0000 7fa027e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa02810c8f0 0x7fa02819c980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45038/0 (socket says 172.21.15.62:45038) 2024-04-20T22:21:44.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:44.119+0000 7fa027e00700 1 -- 172.21.15.62:0/2875947560 learned_addr learned my addr 172.21.15.62:0/2875947560 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:21:44.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:44.119+0000 7fa02ce00700 1 --2- 172.21.15.62:0/2875947560 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa028106b80 0x7fa02819c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:21:44.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:44.119+0000 7fa027e00700 1 -- 172.21.15.62:0/2875947560 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa028106b80 msgr2=0x7fa02819c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:21:44.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:44.119+0000 7fa027e00700 1 --2- 172.21.15.62:0/2875947560 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa028106b80 0x7fa02819c440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:21:44.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:44.119+0000 7fa027e00700 1 -- 172.21.15.62:0/2875947560 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0140095f0 con 0x7fa02810c8f0 2024-04-20T22:21:44.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:44.120+0000 7fa027e00700 1 --2- 172.21.15.62:0/2875947560 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa02810c8f0 0x7fa02819c980 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7fa01400b5c0 tx=0x7fa01400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:21:44.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:44.120+0000 7fa025600700 1 -- 172.21.15.62:0/2875947560 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa01401d070 con 0x7fa02810c8f0 2024-04-20T22:21:44.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:44.120+0000 7fa025600700 1 -- 172.21.15.62:0/2875947560 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa014005e00 con 0x7fa02810c8f0 2024-04-20T22:21:44.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:44.120+0000 7fa02f800700 1 -- 172.21.15.62:0/2875947560 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0281a1070 con 0x7fa02810c8f0 2024-04-20T22:21:44.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:44.120+0000 7fa025600700 1 -- 172.21.15.62:0/2875947560 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa014017400 con 0x7fa02810c8f0 2024-04-20T22:21:44.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:44.120+0000 7fa02f800700 1 -- 172.21.15.62:0/2875947560 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0281a1560 con 0x7fa02810c8f0 2024-04-20T22:21:44.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:44.122+0000 7fa02f800700 1 -- 172.21.15.62:0/2875947560 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa02810ecc0 con 0x7fa02810c8f0 2024-04-20T22:21:44.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:44.123+0000 7fa025600700 1 -- 172.21.15.62:0/2875947560 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fa01400f460 con 0x7fa02810c8f0 2024-04-20T22:21:44.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:44.123+0000 7fa025600700 1 --2- 172.21.15.62:0/2875947560 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa01006ecc0 0x7fa010071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:21:44.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:44.124+0000 7fa025600700 1 -- 172.21.15.62:0/2875947560 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fa014090460 con 0x7fa02810c8f0 2024-04-20T22:21:44.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:44.124+0000 7fa02ce00700 1 --2- 172.21.15.62:0/2875947560 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa01006ecc0 0x7fa010071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:21:44.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:44.125+0000 7fa02ce00700 1 --2- 172.21.15.62:0/2875947560 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa01006ecc0 0x7fa010071170 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fa01c005950 tx=0x7fa01c0058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:21:44.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:44.129+0000 7fa025600700 1 -- 172.21.15.62:0/2875947560 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fa014059170 con 0x7fa02810c8f0 2024-04-20T22:21:44.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:44.388+0000 7fa02f800700 1 -- 172.21.15.62:0/2875947560 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa02802cca0 con 0x7fa01006ecc0 2024-04-20T22:21:44.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:44.391+0000 7fa025600700 1 -- 172.21.15.62:0/2875947560 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fa02802cca0 con 0x7fa01006ecc0 2024-04-20T22:21:44.391 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:21:44.391 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T22:21:44.391 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T22:21:44.391 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-20T22:21:44.391 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T22:21:44.391 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr", 2024-04-20T22:21:44.391 INFO:teuthology.orchestra.run.smithi062.stdout: "osd", 2024-04-20T22:21:44.391 INFO:teuthology.orchestra.run.smithi062.stdout: "mds", 2024-04-20T22:21:44.392 INFO:teuthology.orchestra.run.smithi062.stdout: "crash", 2024-04-20T22:21:44.392 INFO:teuthology.orchestra.run.smithi062.stdout: "mon" 2024-04-20T22:21:44.392 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T22:21:44.392 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "16/23 daemons upgraded", 2024-04-20T22:21:44.392 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading mds daemons", 2024-04-20T22:21:44.392 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T22:21:44.392 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:21:44.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:44.393+0000 7fa02f800700 1 -- 172.21.15.62:0/2875947560 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa01006ecc0 msgr2=0x7fa010071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:21:44.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:44.393+0000 7fa02f800700 1 --2- 172.21.15.62:0/2875947560 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa01006ecc0 0x7fa010071170 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fa01c005950 tx=0x7fa01c0058e0 comp rx=0 tx=0).stop 2024-04-20T22:21:44.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:44.393+0000 7fa02f800700 1 -- 172.21.15.62:0/2875947560 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa02810c8f0 msgr2=0x7fa02819c980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:21:44.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:44.393+0000 7fa02f800700 1 --2- 172.21.15.62:0/2875947560 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa02810c8f0 0x7fa02819c980 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7fa01400b5c0 tx=0x7fa01400be20 comp rx=0 tx=0).stop 2024-04-20T22:21:44.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:44.394+0000 7fa02f800700 1 -- 172.21.15.62:0/2875947560 shutdown_connections 2024-04-20T22:21:44.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:44.394+0000 7fa02f800700 1 --2- 172.21.15.62:0/2875947560 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa01006ecc0 0x7fa010071170 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:21:44.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:44.394+0000 7fa02f800700 1 --2- 172.21.15.62:0/2875947560 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa028106b80 0x7fa02819c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:21:44.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:44.394+0000 7fa02f800700 1 --2- 172.21.15.62:0/2875947560 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa02810c8f0 0x7fa02819c980 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:21:44.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:44.394+0000 7fa02f800700 1 -- 172.21.15.62:0/2875947560 >> 172.21.15.62:0/2875947560 conn(0x7fa028076060 msgr2=0x7fa028077ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:21:44.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:44.394+0000 7fa02f800700 1 -- 172.21.15.62:0/2875947560 shutdown_connections 2024-04-20T22:21:44.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:44.394+0000 7fa02f800700 1 -- 172.21.15.62:0/2875947560 wait complete. 2024-04-20T22:21:44.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:44.560+0000 7fbc53800700 1 -- 172.21.15.62:0/2858307661 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbc4c108470 msgr2=0x7fbc4c108840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:21:44.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:44.560+0000 7fbc53800700 1 --2- 172.21.15.62:0/2858307661 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbc4c108470 0x7fbc4c108840 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7fbc40009910 tx=0x7fbc40009c20 comp rx=0 tx=0).stop 2024-04-20T22:21:44.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:44.560+0000 7fbc53800700 1 -- 172.21.15.62:0/2858307661 shutdown_connections 2024-04-20T22:21:44.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:44.560+0000 7fbc53800700 1 --2- 172.21.15.62:0/2858307661 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc4c102470 0x7fbc4c1028e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:21:44.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:44.560+0000 7fbc53800700 1 --2- 172.21.15.62:0/2858307661 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbc4c108470 0x7fbc4c108840 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:21:44.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:44.560+0000 7fbc53800700 1 -- 172.21.15.62:0/2858307661 >> 172.21.15.62:0/2858307661 conn(0x7fbc4c0fdf90 msgr2=0x7fbc4c1003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:21:44.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:44.560+0000 7fbc53800700 1 -- 172.21.15.62:0/2858307661 shutdown_connections 2024-04-20T22:21:44.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:44.561+0000 7fbc53800700 1 -- 172.21.15.62:0/2858307661 wait complete. 2024-04-20T22:21:44.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:44.562+0000 7fbc53800700 1 Processor -- start 2024-04-20T22:21:44.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:44.562+0000 7fbc53800700 1 -- start start 2024-04-20T22:21:44.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:44.563+0000 7fbc53800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbc4c102470 0x7fbc4c1981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:21:44.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:44.563+0000 7fbc53800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc4c108470 0x7fbc4c198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:21:44.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:44.563+0000 7fbc53800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc4c198e10 con 0x7fbc4c102470 2024-04-20T22:21:44.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:44.563+0000 7fbc53800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc4c19cba0 con 0x7fbc4c108470 2024-04-20T22:21:44.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:44.563+0000 7fbc50e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbc4c102470 0x7fbc4c1981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:21:44.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:44.563+0000 7fbc50e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbc4c102470 0x7fbc4c1981f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45044/0 (socket says 172.21.15.62:45044) 2024-04-20T22:21:44.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:44.563+0000 7fbc50e00700 1 -- 172.21.15.62:0/1837431778 learned_addr learned my addr 172.21.15.62:0/1837431778 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:21:44.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:44.564+0000 7fbc4be00700 1 --2- 172.21.15.62:0/1837431778 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc4c108470 0x7fbc4c198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:21:44.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:44.564+0000 7fbc50e00700 1 -- 172.21.15.62:0/1837431778 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc4c108470 msgr2=0x7fbc4c198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:21:44.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:44.564+0000 7fbc50e00700 1 --2- 172.21.15.62:0/1837431778 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc4c108470 0x7fbc4c198730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:21:44.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:44.564+0000 7fbc50e00700 1 -- 172.21.15.62:0/1837431778 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc400095f0 con 0x7fbc4c102470 2024-04-20T22:21:44.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:44.564+0000 7fbc50e00700 1 --2- 172.21.15.62:0/1837431778 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbc4c102470 0x7fbc4c1981f0 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7fbc400092e0 tx=0x7fbc4000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:21:44.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:44.564+0000 7fbc49600700 1 -- 172.21.15.62:0/1837431778 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbc4001d070 con 0x7fbc4c102470 2024-04-20T22:21:44.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:44.564+0000 7fbc49600700 1 -- 172.21.15.62:0/1837431778 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbc40005e00 con 0x7fbc4c102470 2024-04-20T22:21:44.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:44.564+0000 7fbc49600700 1 -- 172.21.15.62:0/1837431778 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbc40017400 con 0x7fbc4c102470 2024-04-20T22:21:44.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:44.564+0000 7fbc53800700 1 -- 172.21.15.62:0/1837431778 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc4c19ce20 con 0x7fbc4c102470 2024-04-20T22:21:44.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:44.565+0000 7fbc53800700 1 -- 172.21.15.62:0/1837431778 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc4c19d310 con 0x7fbc4c102470 2024-04-20T22:21:44.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:44.566+0000 7fbc53800700 1 -- 172.21.15.62:0/1837431778 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc4c10a9e0 con 0x7fbc4c102470 2024-04-20T22:21:44.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:44.567+0000 7fbc49600700 1 -- 172.21.15.62:0/1837431778 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fbc4000f460 con 0x7fbc4c102470 2024-04-20T22:21:44.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:44.568+0000 7fbc49600700 1 --2- 172.21.15.62:0/1837431778 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fbc3406ecc0 0x7fbc34071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:21:44.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:44.568+0000 7fbc49600700 1 -- 172.21.15.62:0/1837431778 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fbc40090330 con 0x7fbc4c102470 2024-04-20T22:21:44.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:44.569+0000 7fbc4be00700 1 --2- 172.21.15.62:0/1837431778 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fbc3406ecc0 0x7fbc34071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:21:44.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:44.569+0000 7fbc4be00700 1 --2- 172.21.15.62:0/1837431778 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fbc3406ecc0 0x7fbc34071170 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fbc38005950 tx=0x7fbc380058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:21:44.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:44.574+0000 7fbc49600700 1 -- 172.21.15.62:0/1837431778 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fbc400590c0 con 0x7fbc4c102470 2024-04-20T22:21:44.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:21:44 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/205300743' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:21:44.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:21:44 smithi062 ceph-mon[76322]: from='client.36460 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:21:44.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:44.905+0000 7fbc53800700 1 -- 172.21.15.62:0/1837431778 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fbc4c1995f0 con 0x7fbc4c102470 2024-04-20T22:21:44.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:44.906+0000 7fbc49600700 1 -- 172.21.15.62:0/1837431778 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7fbc4005c780 con 0x7fbc4c102470 2024-04-20T22:21:44.906 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-20T22:21:44.906 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-20T22:21:44.906 INFO:teuthology.orchestra.run.smithi062.stdout: host smithi189 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi189/config 2024-04-20T22:21:44.906 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-20T22:21:44.906 INFO:teuthology.orchestra.run.smithi062.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi189 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp11670v2q:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:21:44.906 INFO:teuthology.orchestra.run.smithi062.stdout:Traceback (most recent call last): 2024-04-20T22:21:44.906 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-20T22:21:44.906 INFO:teuthology.orchestra.run.smithi062.stdout: return _run_code(code, main_globals, None, 2024-04-20T22:21:44.906 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-20T22:21:44.906 INFO:teuthology.orchestra.run.smithi062.stdout: exec(code, run_globals) 2024-04-20T22:21:44.906 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5618, in 2024-04-20T22:21:44.907 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5606, in main 2024-04-20T22:21:44.907 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 408, in _infer_config 2024-04-20T22:21:44.907 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 323, in _infer_fsid 2024-04-20T22:21:44.907 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 436, in _infer_image 2024-04-20T22:21:44.907 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 310, in _validate_fsid 2024-04-20T22:21:44.907 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 3359, in command_ceph_volume 2024-04-20T22:21:44.907 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-20T22:21:44.907 INFO:teuthology.orchestra.run.smithi062.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi189 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp11670v2q:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:21:44.907 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-20T22:21:44.907 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is degraded 2024-04-20T22:21:44.907 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T22:21:44.907 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T22:21:44.907 INFO:teuthology.orchestra.run.smithi062.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-20T22:21:44.907 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-20T22:21:44.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:44.909+0000 7fbc53800700 1 -- 172.21.15.62:0/1837431778 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fbc3406ecc0 msgr2=0x7fbc34071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:21:44.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:44.909+0000 7fbc53800700 1 --2- 172.21.15.62:0/1837431778 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fbc3406ecc0 0x7fbc34071170 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fbc38005950 tx=0x7fbc380058e0 comp rx=0 tx=0).stop 2024-04-20T22:21:44.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:44.909+0000 7fbc53800700 1 -- 172.21.15.62:0/1837431778 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbc4c102470 msgr2=0x7fbc4c1981f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:21:44.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:44.909+0000 7fbc53800700 1 --2- 172.21.15.62:0/1837431778 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbc4c102470 0x7fbc4c1981f0 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7fbc400092e0 tx=0x7fbc4000be20 comp rx=0 tx=0).stop 2024-04-20T22:21:44.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:44.909+0000 7fbc53800700 1 -- 172.21.15.62:0/1837431778 shutdown_connections 2024-04-20T22:21:44.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:44.909+0000 7fbc53800700 1 --2- 172.21.15.62:0/1837431778 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fbc3406ecc0 0x7fbc34071170 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:21:44.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:44.909+0000 7fbc53800700 1 --2- 172.21.15.62:0/1837431778 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbc4c102470 0x7fbc4c1981f0 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:21:44.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:44.909+0000 7fbc53800700 1 --2- 172.21.15.62:0/1837431778 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc4c108470 0x7fbc4c198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:21:44.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:44.909+0000 7fbc53800700 1 -- 172.21.15.62:0/1837431778 >> 172.21.15.62:0/1837431778 conn(0x7fbc4c0fdf90 msgr2=0x7fbc4c1002c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:21:44.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:44.909+0000 7fbc53800700 1 -- 172.21.15.62:0/1837431778 shutdown_connections 2024-04-20T22:21:44.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:21:44.909+0000 7fbc53800700 1 -- 172.21.15.62:0/1837431778 wait complete. 2024-04-20T22:21:44.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:21:44 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/205300743' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:21:44.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:21:44 smithi189 ceph-mon[57022]: from='client.36460 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:21:45.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:21:45 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/1837431778' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:21:45.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:21:45 smithi062 ceph-mon[76322]: pgmap v1397: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:21:45.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:21:45 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/1837431778' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:21:45.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:21:45 smithi189 ceph-mon[57022]: pgmap v1397: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:21:47.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:21:47 smithi062 ceph-mon[76322]: pgmap v1398: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:21:47.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:21:47 smithi189 ceph-mon[57022]: pgmap v1398: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:21:49.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:21:49 smithi189 ceph-mon[57022]: pgmap v1399: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:21:49.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:21:49 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:21:49.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:21:49 smithi062 ceph-mon[76322]: pgmap v1399: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:21:49.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:21:49 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:21:51.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:21:51 smithi062 ceph-mon[76322]: pgmap v1400: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:21:51.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:21:51 smithi189 ceph-mon[57022]: pgmap v1400: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:21:53.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:21:53 smithi062 ceph-mon[76322]: pgmap v1401: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:21:53.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:21:53 smithi189 ceph-mon[57022]: pgmap v1401: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:21:55.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:21:55 smithi062 ceph-mon[76322]: pgmap v1402: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:21:55.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:21:55 smithi189 ceph-mon[57022]: pgmap v1402: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:21:57.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:21:57 smithi062 ceph-mon[76322]: pgmap v1403: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:21:57.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:21:57 smithi189 ceph-mon[57022]: pgmap v1403: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:21:59.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:21:59 smithi062 ceph-mon[76322]: pgmap v1404: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:21:59.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:21:59 smithi189 ceph-mon[57022]: pgmap v1404: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:22:01.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:22:01 smithi062 ceph-mon[76322]: pgmap v1405: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:22:01.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:22:01 smithi189 ceph-mon[57022]: pgmap v1405: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:22:03.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:22:03 smithi062 ceph-mon[76322]: pgmap v1406: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:22:03.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:22:03 smithi189 ceph-mon[57022]: pgmap v1406: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:22:04.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:22:04 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:22:04.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:22:04 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:22:05.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:22:05 smithi062 ceph-mon[76322]: pgmap v1407: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:22:05.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:22:05 smithi189 ceph-mon[57022]: pgmap v1407: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:22:07.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:22:07 smithi062 ceph-mon[76322]: pgmap v1408: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:22:07.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:22:07 smithi189 ceph-mon[57022]: pgmap v1408: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:22:09.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:22:09 smithi062 ceph-mon[76322]: pgmap v1409: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:22:09.893 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:22:09 smithi189 ceph-mon[57022]: pgmap v1409: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:22:11.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:22:11 smithi062 ceph-mon[76322]: pgmap v1410: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:22:11.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:22:11 smithi189 ceph-mon[57022]: pgmap v1410: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:22:13.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:22:13 smithi062 ceph-mon[76322]: pgmap v1411: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:22:13.790 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:22:13 smithi189 ceph-mon[57022]: pgmap v1411: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:22:15.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.086+0000 7fd4e0200700 1 -- 172.21.15.62:0/655444514 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd4d80739a0 msgr2=0x7fd4d810c5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:22:15.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.086+0000 7fd4e0200700 1 --2- 172.21.15.62:0/655444514 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd4d80739a0 0x7fd4d810c5e0 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7fd4c4009940 tx=0x7fd4c4009c50 comp rx=0 tx=0).stop 2024-04-20T22:22:15.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.087+0000 7fd4e0200700 1 -- 172.21.15.62:0/655444514 shutdown_connections 2024-04-20T22:22:15.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.087+0000 7fd4e0200700 1 --2- 172.21.15.62:0/655444514 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd4d80739a0 0x7fd4d810c5e0 unknown :-1 s=CLOSED pgs=1197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:22:15.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.087+0000 7fd4e0200700 1 --2- 172.21.15.62:0/655444514 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd4d8073090 0x7fd4d8073460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:22:15.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.087+0000 7fd4e0200700 1 -- 172.21.15.62:0/655444514 >> 172.21.15.62:0/655444514 conn(0x7fd4d80fbda0 msgr2=0x7fd4d80fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:22:15.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.087+0000 7fd4e0200700 1 -- 172.21.15.62:0/655444514 shutdown_connections 2024-04-20T22:22:15.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.087+0000 7fd4e0200700 1 -- 172.21.15.62:0/655444514 wait complete. 2024-04-20T22:22:15.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.088+0000 7fd4e0200700 1 Processor -- start 2024-04-20T22:22:15.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.088+0000 7fd4e0200700 1 -- start start 2024-04-20T22:22:15.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.088+0000 7fd4e0200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd4d8073090 0x7fd4d8198220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:22:15.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.088+0000 7fd4e0200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd4d80739a0 0x7fd4d8198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:22:15.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.088+0000 7fd4e0200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4d8198e40 con 0x7fd4d80739a0 2024-04-20T22:22:15.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.088+0000 7fd4e0200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4d819cbd0 con 0x7fd4d8073090 2024-04-20T22:22:15.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.089+0000 7fd4dce00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd4d80739a0 0x7fd4d8198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:22:15.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.089+0000 7fd4dce00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd4d80739a0 0x7fd4d8198760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59078/0 (socket says 172.21.15.62:59078) 2024-04-20T22:22:15.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.089+0000 7fd4dce00700 1 -- 172.21.15.62:0/1440373812 learned_addr learned my addr 172.21.15.62:0/1440373812 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:22:15.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.089+0000 7fd4dd800700 1 --2- 172.21.15.62:0/1440373812 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd4d8073090 0x7fd4d8198220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:22:15.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.089+0000 7fd4dce00700 1 -- 172.21.15.62:0/1440373812 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd4d8073090 msgr2=0x7fd4d8198220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:22:15.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.089+0000 7fd4dce00700 1 --2- 172.21.15.62:0/1440373812 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd4d8073090 0x7fd4d8198220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:22:15.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.089+0000 7fd4dce00700 1 -- 172.21.15.62:0/1440373812 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4c40095f0 con 0x7fd4d80739a0 2024-04-20T22:22:15.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.090+0000 7fd4dce00700 1 --2- 172.21.15.62:0/1440373812 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd4d80739a0 0x7fd4d8198760 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7fd4c400b870 tx=0x7fd4c40056a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:22:15.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.090+0000 7fd4d6000700 1 -- 172.21.15.62:0/1440373812 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd4c401d070 con 0x7fd4d80739a0 2024-04-20T22:22:15.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.090+0000 7fd4d6000700 1 -- 172.21.15.62:0/1440373812 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd4c400f460 con 0x7fd4d80739a0 2024-04-20T22:22:15.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.090+0000 7fd4e0200700 1 -- 172.21.15.62:0/1440373812 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4d819ce50 con 0x7fd4d80739a0 2024-04-20T22:22:15.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.090+0000 7fd4d6000700 1 -- 172.21.15.62:0/1440373812 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd4c4021640 con 0x7fd4d80739a0 2024-04-20T22:22:15.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.090+0000 7fd4e0200700 1 -- 172.21.15.62:0/1440373812 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4d819d340 con 0x7fd4d80739a0 2024-04-20T22:22:15.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.091+0000 7fd4e0200700 1 -- 172.21.15.62:0/1440373812 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4d8109ce0 con 0x7fd4d80739a0 2024-04-20T22:22:15.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.092+0000 7fd4d6000700 1 -- 172.21.15.62:0/1440373812 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fd4c400fa60 con 0x7fd4d80739a0 2024-04-20T22:22:15.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.093+0000 7fd4d6000700 1 --2- 172.21.15.62:0/1440373812 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd4c006ecc0 0x7fd4c0071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:22:15.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.093+0000 7fd4d6000700 1 -- 172.21.15.62:0/1440373812 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fd4c40901d0 con 0x7fd4d80739a0 2024-04-20T22:22:15.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.094+0000 7fd4dd800700 1 --2- 172.21.15.62:0/1440373812 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd4c006ecc0 0x7fd4c0071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:22:15.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.094+0000 7fd4dd800700 1 --2- 172.21.15.62:0/1440373812 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd4c006ecc0 0x7fd4c0071170 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fd4cc005fd0 tx=0x7fd4cc00a560 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:22:15.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.099+0000 7fd4d6000700 1 -- 172.21.15.62:0/1440373812 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fd4c4058ee0 con 0x7fd4d80739a0 2024-04-20T22:22:15.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.359+0000 7fd4e0200700 1 -- 172.21.15.62:0/1440373812 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd4d8199580 con 0x7fd4c006ecc0 2024-04-20T22:22:15.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.361+0000 7fd4d6000700 1 -- 172.21.15.62:0/1440373812 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fd4d8199580 con 0x7fd4c006ecc0 2024-04-20T22:22:15.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.364+0000 7fd4e0200700 1 -- 172.21.15.62:0/1440373812 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd4c006ecc0 msgr2=0x7fd4c0071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:22:15.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.364+0000 7fd4e0200700 1 --2- 172.21.15.62:0/1440373812 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd4c006ecc0 0x7fd4c0071170 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fd4cc005fd0 tx=0x7fd4cc00a560 comp rx=0 tx=0).stop 2024-04-20T22:22:15.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.364+0000 7fd4e0200700 1 -- 172.21.15.62:0/1440373812 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd4d80739a0 msgr2=0x7fd4d8198760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:22:15.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.364+0000 7fd4e0200700 1 --2- 172.21.15.62:0/1440373812 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd4d80739a0 0x7fd4d8198760 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7fd4c400b870 tx=0x7fd4c40056a0 comp rx=0 tx=0).stop 2024-04-20T22:22:15.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.364+0000 7fd4e0200700 1 -- 172.21.15.62:0/1440373812 shutdown_connections 2024-04-20T22:22:15.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.364+0000 7fd4e0200700 1 --2- 172.21.15.62:0/1440373812 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd4c006ecc0 0x7fd4c0071170 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:22:15.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.364+0000 7fd4e0200700 1 --2- 172.21.15.62:0/1440373812 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd4d8073090 0x7fd4d8198220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:22:15.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.364+0000 7fd4e0200700 1 --2- 172.21.15.62:0/1440373812 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd4d80739a0 0x7fd4d8198760 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:22:15.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.364+0000 7fd4e0200700 1 -- 172.21.15.62:0/1440373812 >> 172.21.15.62:0/1440373812 conn(0x7fd4d80fbda0 msgr2=0x7fd4d8106e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:22:15.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.364+0000 7fd4e0200700 1 -- 172.21.15.62:0/1440373812 shutdown_connections 2024-04-20T22:22:15.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.364+0000 7fd4e0200700 1 -- 172.21.15.62:0/1440373812 wait complete. 2024-04-20T22:22:15.381 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T22:22:15.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.526+0000 7f24cfc00700 1 -- 172.21.15.62:0/3427144229 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f24c8100e80 msgr2=0x7f24c8104e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:22:15.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.526+0000 7f24cfc00700 1 --2- 172.21.15.62:0/3427144229 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f24c8100e80 0x7f24c8104e50 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f24b4009910 tx=0x7f24b4009c20 comp rx=0 tx=0).stop 2024-04-20T22:22:15.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.526+0000 7f24cfc00700 1 -- 172.21.15.62:0/3427144229 shutdown_connections 2024-04-20T22:22:15.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.526+0000 7f24cfc00700 1 --2- 172.21.15.62:0/3427144229 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f24c8100e80 0x7f24c8104e50 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:22:15.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.526+0000 7f24cfc00700 1 --2- 172.21.15.62:0/3427144229 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f24c81004e0 0x7f24c81008b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:22:15.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.526+0000 7f24cfc00700 1 -- 172.21.15.62:0/3427144229 >> 172.21.15.62:0/3427144229 conn(0x7f24c80fbda0 msgr2=0x7f24c80fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:22:15.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.526+0000 7f24cfc00700 1 -- 172.21.15.62:0/3427144229 shutdown_connections 2024-04-20T22:22:15.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.526+0000 7f24cfc00700 1 -- 172.21.15.62:0/3427144229 wait complete. 2024-04-20T22:22:15.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.528+0000 7f24cfc00700 1 Processor -- start 2024-04-20T22:22:15.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.528+0000 7f24cfc00700 1 -- start start 2024-04-20T22:22:15.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.528+0000 7f24cfc00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f24c81004e0 0x7f24c8198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:22:15.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.528+0000 7f24cfc00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f24c8100e80 0x7f24c81985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:22:15.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.528+0000 7f24cfc00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24c8198ca0 con 0x7f24c81004e0 2024-04-20T22:22:15.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.528+0000 7f24cfc00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24c819ca30 con 0x7f24c8100e80 2024-04-20T22:22:15.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.529+0000 7f24cd200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f24c81004e0 0x7f24c8198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:22:15.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.529+0000 7f24cd200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f24c81004e0 0x7f24c8198080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59096/0 (socket says 172.21.15.62:59096) 2024-04-20T22:22:15.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.529+0000 7f24cd200700 1 -- 172.21.15.62:0/3975385430 learned_addr learned my addr 172.21.15.62:0/3975385430 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:22:15.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.529+0000 7f24c7e00700 1 --2- 172.21.15.62:0/3975385430 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f24c8100e80 0x7f24c81985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:22:15.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.529+0000 7f24cd200700 1 -- 172.21.15.62:0/3975385430 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f24c8100e80 msgr2=0x7f24c81985c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:22:15.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.529+0000 7f24cd200700 1 --2- 172.21.15.62:0/3975385430 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f24c8100e80 0x7f24c81985c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:22:15.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.529+0000 7f24cd200700 1 -- 172.21.15.62:0/3975385430 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24b40095f0 con 0x7f24c81004e0 2024-04-20T22:22:15.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.530+0000 7f24cd200700 1 --2- 172.21.15.62:0/3975385430 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f24c81004e0 0x7f24c8198080 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f24bc00d8d0 tx=0x7f24bc00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:22:15.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.530+0000 7f24c5600700 1 -- 172.21.15.62:0/3975385430 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f24bc009940 con 0x7f24c81004e0 2024-04-20T22:22:15.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.530+0000 7f24c5600700 1 -- 172.21.15.62:0/3975385430 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f24bc010460 con 0x7f24c81004e0 2024-04-20T22:22:15.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.530+0000 7f24cfc00700 1 -- 172.21.15.62:0/3975385430 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24c819cd10 con 0x7f24c81004e0 2024-04-20T22:22:15.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.530+0000 7f24c5600700 1 -- 172.21.15.62:0/3975385430 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f24bc00f5f0 con 0x7f24c81004e0 2024-04-20T22:22:15.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.530+0000 7f24cfc00700 1 -- 172.21.15.62:0/3975385430 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24c819d260 con 0x7f24c81004e0 2024-04-20T22:22:15.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.531+0000 7f24cfc00700 1 -- 172.21.15.62:0/3975385430 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24c81045d0 con 0x7f24c81004e0 2024-04-20T22:22:15.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.533+0000 7f24c5600700 1 -- 172.21.15.62:0/3975385430 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f24bc0105d0 con 0x7f24c81004e0 2024-04-20T22:22:15.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.533+0000 7f24c5600700 1 --2- 172.21.15.62:0/3975385430 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f24b006ecc0 0x7f24b0071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:22:15.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.534+0000 7f24c5600700 1 -- 172.21.15.62:0/3975385430 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f24bc08ee10 con 0x7f24c81004e0 2024-04-20T22:22:15.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.534+0000 7f24c7e00700 1 --2- 172.21.15.62:0/3975385430 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f24b006ecc0 0x7f24b0071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:22:15.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.535+0000 7f24c7e00700 1 --2- 172.21.15.62:0/3975385430 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f24b006ecc0 0x7f24b0071170 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f24c81996a0 tx=0x7f24b401a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:22:15.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.539+0000 7f24c5600700 1 -- 172.21.15.62:0/3975385430 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f24bc057ba0 con 0x7f24c81004e0 2024-04-20T22:22:15.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:22:15 smithi062 ceph-mon[76322]: from='client.36468 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:22:15.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:22:15 smithi062 ceph-mon[76322]: pgmap v1412: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:22:15.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.790+0000 7f24cfc00700 1 -- 172.21.15.62:0/3975385430 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f24c819d4d0 con 0x7f24b006ecc0 2024-04-20T22:22:15.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.792+0000 7f24c5600700 1 -- 172.21.15.62:0/3975385430 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f24c819d4d0 con 0x7f24b006ecc0 2024-04-20T22:22:15.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.795+0000 7f24cfc00700 1 -- 172.21.15.62:0/3975385430 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f24b006ecc0 msgr2=0x7f24b0071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:22:15.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.795+0000 7f24cfc00700 1 --2- 172.21.15.62:0/3975385430 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f24b006ecc0 0x7f24b0071170 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f24c81996a0 tx=0x7f24b401a040 comp rx=0 tx=0).stop 2024-04-20T22:22:15.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.795+0000 7f24cfc00700 1 -- 172.21.15.62:0/3975385430 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f24c81004e0 msgr2=0x7f24c8198080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:22:15.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.795+0000 7f24cfc00700 1 --2- 172.21.15.62:0/3975385430 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f24c81004e0 0x7f24c8198080 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f24bc00d8d0 tx=0x7f24bc00dc90 comp rx=0 tx=0).stop 2024-04-20T22:22:15.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.795+0000 7f24cfc00700 1 -- 172.21.15.62:0/3975385430 shutdown_connections 2024-04-20T22:22:15.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.795+0000 7f24cfc00700 1 --2- 172.21.15.62:0/3975385430 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f24b006ecc0 0x7f24b0071170 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:22:15.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.795+0000 7f24cfc00700 1 --2- 172.21.15.62:0/3975385430 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f24c81004e0 0x7f24c8198080 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:22:15.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.795+0000 7f24cfc00700 1 --2- 172.21.15.62:0/3975385430 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f24c8100e80 0x7f24c81985c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:22:15.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.795+0000 7f24cfc00700 1 -- 172.21.15.62:0/3975385430 >> 172.21.15.62:0/3975385430 conn(0x7f24c80fbda0 msgr2=0x7f24c80fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:22:15.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.795+0000 7f24cfc00700 1 -- 172.21.15.62:0/3975385430 shutdown_connections 2024-04-20T22:22:15.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.796+0000 7f24cfc00700 1 -- 172.21.15.62:0/3975385430 wait complete. 2024-04-20T22:22:15.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.956+0000 7f40f1400700 1 -- 172.21.15.62:0/172391512 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40ec103e50 msgr2=0x7f40ec1042c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:22:15.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.956+0000 7f40f1400700 1 --2- 172.21.15.62:0/172391512 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40ec103e50 0x7f40ec1042c0 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f40d8009910 tx=0x7f40d8009c20 comp rx=0 tx=0).stop 2024-04-20T22:22:15.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.956+0000 7f40f1400700 1 -- 172.21.15.62:0/172391512 shutdown_connections 2024-04-20T22:22:15.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.956+0000 7f40f1400700 1 --2- 172.21.15.62:0/172391512 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40ec103e50 0x7f40ec1042c0 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:22:15.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.956+0000 7f40f1400700 1 --2- 172.21.15.62:0/172391512 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f40ec1025f0 0x7f40ec1029c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:22:15.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.956+0000 7f40f1400700 1 -- 172.21.15.62:0/172391512 >> 172.21.15.62:0/172391512 conn(0x7f40ec0fddb0 msgr2=0x7f40ec1001c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:22:15.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.956+0000 7f40f1400700 1 -- 172.21.15.62:0/172391512 shutdown_connections 2024-04-20T22:22:15.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.956+0000 7f40f1400700 1 -- 172.21.15.62:0/172391512 wait complete. 2024-04-20T22:22:15.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.957+0000 7f40f1400700 1 Processor -- start 2024-04-20T22:22:15.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.957+0000 7f40f1400700 1 -- start start 2024-04-20T22:22:15.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.958+0000 7f40f1400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40ec1025f0 0x7f40ec197ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:22:15.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.958+0000 7f40f1400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f40ec103e50 0x7f40ec1983e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:22:15.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.958+0000 7f40f1400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40ec198ac0 con 0x7f40ec1025f0 2024-04-20T22:22:15.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.958+0000 7f40f1400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40ec19c850 con 0x7f40ec103e50 2024-04-20T22:22:15.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.958+0000 7f40ebe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40ec1025f0 0x7f40ec197ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:22:15.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.958+0000 7f40ebe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40ec1025f0 0x7f40ec197ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59110/0 (socket says 172.21.15.62:59110) 2024-04-20T22:22:15.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.958+0000 7f40ebe00700 1 -- 172.21.15.62:0/933762419 learned_addr learned my addr 172.21.15.62:0/933762419 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:22:15.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.958+0000 7f40eb400700 1 --2- 172.21.15.62:0/933762419 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f40ec103e50 0x7f40ec1983e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:22:15.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.958+0000 7f40ebe00700 1 -- 172.21.15.62:0/933762419 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f40ec103e50 msgr2=0x7f40ec1983e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:22:15.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.958+0000 7f40ebe00700 1 --2- 172.21.15.62:0/933762419 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f40ec103e50 0x7f40ec1983e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:22:15.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.958+0000 7f40ebe00700 1 -- 172.21.15.62:0/933762419 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40d80095f0 con 0x7f40ec1025f0 2024-04-20T22:22:15.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.959+0000 7f40ebe00700 1 --2- 172.21.15.62:0/933762419 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40ec1025f0 0x7f40ec197ea0 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f40dc00d8d0 tx=0x7f40dc00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:22:15.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.959+0000 7f40e8c00700 1 -- 172.21.15.62:0/933762419 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f40dc009940 con 0x7f40ec1025f0 2024-04-20T22:22:15.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.959+0000 7f40e8c00700 1 -- 172.21.15.62:0/933762419 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f40dc010460 con 0x7f40ec1025f0 2024-04-20T22:22:15.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.959+0000 7f40f1400700 1 -- 172.21.15.62:0/933762419 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40ec19cb30 con 0x7f40ec1025f0 2024-04-20T22:22:15.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.959+0000 7f40e8c00700 1 -- 172.21.15.62:0/933762419 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f40dc00f5f0 con 0x7f40ec1025f0 2024-04-20T22:22:15.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.959+0000 7f40f1400700 1 -- 172.21.15.62:0/933762419 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40ec19d100 con 0x7f40ec1025f0 2024-04-20T22:22:15.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.960+0000 7f40f1400700 1 -- 172.21.15.62:0/933762419 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40ec062380 con 0x7f40ec1025f0 2024-04-20T22:22:15.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.961+0000 7f40e8c00700 1 -- 172.21.15.62:0/933762419 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f40dc0105d0 con 0x7f40ec1025f0 2024-04-20T22:22:15.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.963+0000 7f40e8c00700 1 --2- 172.21.15.62:0/933762419 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f40d006ecc0 0x7f40d0071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:22:15.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.963+0000 7f40e8c00700 1 -- 172.21.15.62:0/933762419 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f40dc08ef20 con 0x7f40ec1025f0 2024-04-20T22:22:15.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.963+0000 7f40eb400700 1 --2- 172.21.15.62:0/933762419 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f40d006ecc0 0x7f40d0071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:22:15.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.964+0000 7f40eb400700 1 --2- 172.21.15.62:0/933762419 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f40d006ecc0 0x7f40d0071170 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f40ec1994c0 tx=0x7f40d801a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:22:15.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:15.968+0000 7f40e8c00700 1 -- 172.21.15.62:0/933762419 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f40dc057c30 con 0x7f40ec1025f0 2024-04-20T22:22:15.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:22:15 smithi189 ceph-mon[57022]: from='client.36468 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:22:15.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:22:15 smithi189 ceph-mon[57022]: pgmap v1412: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:22:16.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:16.214+0000 7f40f1400700 1 -- 172.21.15.62:0/933762419 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f40ec19d400 con 0x7f40d006ecc0 2024-04-20T22:22:16.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:16.247+0000 7f40e8c00700 1 -- 172.21.15.62:0/933762419 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f40ec19d400 con 0x7f40d006ecc0 2024-04-20T22:22:16.247 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T22:22:16.248 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (48m) 8m ago 61m 14.6M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T22:22:16.248 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (62m) 8m ago 62m 11.6M - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T22:22:16.248 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (60m) 8m ago 60m 11.2M - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T22:22:16.248 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (44m) 8m ago 62m 6660k - 19.0.0-3133-ga8c84f59 76bc6b91b726 2825d5ba11c6 2024-04-20T22:22:16.248 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (44m) 8m ago 60m 6665k - 19.0.0-3133-ga8c84f59 76bc6b91b726 20f746fa1bba 2024-04-20T22:22:16.248 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (47m) 8m ago 61m 75.5M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T22:22:16.248 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (39m) 8m ago 54m 22.9M - 19.0.0-3133-ga8c84f59 76bc6b91b726 de80bc90a683 2024-04-20T22:22:16.248 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (40m) 8m ago 54m 23.0M - 19.0.0-3133-ga8c84f59 76bc6b91b726 cd793b7ee98e 2024-04-20T22:22:16.248 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (24m) 8m ago 54m 18.5M - 19.0.0-3133-ga8c84f59 76bc6b91b726 0e7b0bc4d1ab 2024-04-20T22:22:16.248 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (9m) 8m ago 54m 11.4M - 19.0.0-3133-ga8c84f59 76bc6b91b726 0af732ac9e06 2024-04-20T22:22:16.248 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (50m) 8m ago 63m 517M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T22:22:16.248 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (49m) 8m ago 60m 460M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T22:22:16.248 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (45m) 8m ago 63m 86.3M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 019eb1479485 2024-04-20T22:22:16.249 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (45m) 8m ago 60m 58.0M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 55aac560aab6 2024-04-20T22:22:16.249 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (49m) 8m ago 62m 8766k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T22:22:16.249 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (49m) 8m ago 60m 8711k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T22:22:16.249 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (43m) 8m ago 59m 91.3M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 ee2bb901f26e 2024-04-20T22:22:16.249 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (43m) 8m ago 59m 94.9M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 9075152d4162 2024-04-20T22:22:16.249 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (42m) 8m ago 58m 78.5M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 2d4f3e38dc0e 2024-04-20T22:22:16.249 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (41m) 8m ago 57m 94.1M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 8f97b17ae0eb 2024-04-20T22:22:16.249 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (41m) 8m ago 57m 96.1M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 653d8079967d 2024-04-20T22:22:16.249 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (40m) 8m ago 56m 90.9M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 596dded14c12 2024-04-20T22:22:16.249 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (48m) 8m ago 61m 90.4M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T22:22:16.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:16.250+0000 7f40f1400700 1 -- 172.21.15.62:0/933762419 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f40d006ecc0 msgr2=0x7f40d0071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:22:16.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:16.250+0000 7f40f1400700 1 --2- 172.21.15.62:0/933762419 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f40d006ecc0 0x7f40d0071170 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f40ec1994c0 tx=0x7f40d801a040 comp rx=0 tx=0).stop 2024-04-20T22:22:16.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:16.250+0000 7f40f1400700 1 -- 172.21.15.62:0/933762419 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40ec1025f0 msgr2=0x7f40ec197ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:22:16.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:16.250+0000 7f40f1400700 1 --2- 172.21.15.62:0/933762419 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40ec1025f0 0x7f40ec197ea0 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f40dc00d8d0 tx=0x7f40dc00dc90 comp rx=0 tx=0).stop 2024-04-20T22:22:16.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:16.250+0000 7f40f1400700 1 -- 172.21.15.62:0/933762419 shutdown_connections 2024-04-20T22:22:16.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:16.250+0000 7f40f1400700 1 --2- 172.21.15.62:0/933762419 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f40d006ecc0 0x7f40d0071170 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:22:16.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:16.250+0000 7f40f1400700 1 --2- 172.21.15.62:0/933762419 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40ec1025f0 0x7f40ec197ea0 unknown :-1 s=CLOSED pgs=1202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:22:16.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:16.250+0000 7f40f1400700 1 --2- 172.21.15.62:0/933762419 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f40ec103e50 0x7f40ec1983e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:22:16.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:16.250+0000 7f40f1400700 1 -- 172.21.15.62:0/933762419 >> 172.21.15.62:0/933762419 conn(0x7f40ec0fddb0 msgr2=0x7f40ec0fe710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:22:16.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:16.251+0000 7f40f1400700 1 -- 172.21.15.62:0/933762419 shutdown_connections 2024-04-20T22:22:16.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:16.251+0000 7f40f1400700 1 -- 172.21.15.62:0/933762419 wait complete. 2024-04-20T22:22:16.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:16.397+0000 7f81dce00700 1 -- 172.21.15.62:0/3592244561 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f81d8106b80 msgr2=0x7f81d8106ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:22:16.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:16.397+0000 7f81dce00700 1 --2- 172.21.15.62:0/3592244561 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f81d8106b80 0x7f81d8106ff0 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f81c4009910 tx=0x7f81c4009c20 comp rx=0 tx=0).stop 2024-04-20T22:22:16.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:16.397+0000 7f81dce00700 1 -- 172.21.15.62:0/3592244561 shutdown_connections 2024-04-20T22:22:16.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:16.397+0000 7f81dce00700 1 --2- 172.21.15.62:0/3592244561 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f81d8106b80 0x7f81d8106ff0 unknown :-1 s=CLOSED pgs=1203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:22:16.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:16.397+0000 7f81dce00700 1 --2- 172.21.15.62:0/3592244561 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f81d810c8f0 0x7f81d810ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:22:16.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:16.397+0000 7f81dce00700 1 -- 172.21.15.62:0/3592244561 >> 172.21.15.62:0/3592244561 conn(0x7f81d8076060 msgr2=0x7f81d8078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:22:16.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:16.398+0000 7f81dce00700 1 -- 172.21.15.62:0/3592244561 shutdown_connections 2024-04-20T22:22:16.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:16.398+0000 7f81dce00700 1 -- 172.21.15.62:0/3592244561 wait complete. 2024-04-20T22:22:16.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:16.399+0000 7f81dce00700 1 Processor -- start 2024-04-20T22:22:16.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:16.399+0000 7f81dce00700 1 -- start start 2024-04-20T22:22:16.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:16.399+0000 7f81dce00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f81d8106b80 0x7f81d819c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:22:16.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:16.399+0000 7f81dce00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f81d810c8f0 0x7f81d819ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:22:16.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:16.399+0000 7f81dce00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81d819d130 con 0x7f81d8106b80 2024-04-20T22:22:16.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:16.399+0000 7f81dce00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81d81a0ec0 con 0x7f81d810c8f0 2024-04-20T22:22:16.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:16.400+0000 7f81d5e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f81d8106b80 0x7f81d819c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:22:16.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:16.400+0000 7f81d5e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f81d8106b80 0x7f81d819c510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59120/0 (socket says 172.21.15.62:59120) 2024-04-20T22:22:16.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:16.400+0000 7f81d5e00700 1 -- 172.21.15.62:0/3925762327 learned_addr learned my addr 172.21.15.62:0/3925762327 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:22:16.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:16.400+0000 7f81d5e00700 1 -- 172.21.15.62:0/3925762327 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f81d810c8f0 msgr2=0x7f81d819ca50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-20T22:22:16.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:16.400+0000 7f81d5e00700 1 --2- 172.21.15.62:0/3925762327 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f81d810c8f0 0x7f81d819ca50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:22:16.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:16.400+0000 7f81d5e00700 1 -- 172.21.15.62:0/3925762327 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81c40095f0 con 0x7f81d8106b80 2024-04-20T22:22:16.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:16.400+0000 7f81d5e00700 1 --2- 172.21.15.62:0/3925762327 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f81d8106b80 0x7f81d819c510 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f81c800b4d0 tx=0x7f81c800b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:22:16.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:16.401+0000 7f81cea00700 1 -- 172.21.15.62:0/3925762327 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f81c8011800 con 0x7f81d8106b80 2024-04-20T22:22:16.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:16.401+0000 7f81cea00700 1 -- 172.21.15.62:0/3925762327 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f81c8011e20 con 0x7f81d8106b80 2024-04-20T22:22:16.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:16.401+0000 7f81dce00700 1 -- 172.21.15.62:0/3925762327 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81d81a11a0 con 0x7f81d8106b80 2024-04-20T22:22:16.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:16.401+0000 7f81cea00700 1 -- 172.21.15.62:0/3925762327 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f81c800f2e0 con 0x7f81d8106b80 2024-04-20T22:22:16.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:16.401+0000 7f81dce00700 1 -- 172.21.15.62:0/3925762327 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81d81a16f0 con 0x7f81d8106b80 2024-04-20T22:22:16.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:16.402+0000 7f81dce00700 1 -- 172.21.15.62:0/3925762327 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81d8109b20 con 0x7f81d8106b80 2024-04-20T22:22:16.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:16.403+0000 7f81cea00700 1 -- 172.21.15.62:0/3925762327 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f81c8010380 con 0x7f81d8106b80 2024-04-20T22:22:16.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:16.404+0000 7f81cea00700 1 --2- 172.21.15.62:0/3925762327 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f81bc06ecc0 0x7f81bc071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:22:16.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:16.404+0000 7f81cea00700 1 -- 172.21.15.62:0/3925762327 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f81c808e980 con 0x7f81d8106b80 2024-04-20T22:22:16.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:16.404+0000 7f81d5400700 1 --2- 172.21.15.62:0/3925762327 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f81bc06ecc0 0x7f81bc071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:22:16.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:16.405+0000 7f81d5400700 1 --2- 172.21.15.62:0/3925762327 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f81bc06ecc0 0x7f81bc071170 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f81c4000c00 tx=0x7f81c4005fb0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:22:16.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:16.409+0000 7f81cea00700 1 -- 172.21.15.62:0/3925762327 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f81c8057710 con 0x7f81d8106b80 2024-04-20T22:22:16.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:16.727+0000 7f81dce00700 1 -- 172.21.15.62:0/3925762327 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f81d804f2a0 con 0x7f81d8106b80 2024-04-20T22:22:16.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:16.728+0000 7f81cea00700 1 -- 172.21.15.62:0/3925762327 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f81c805add0 con 0x7f81d8106b80 2024-04-20T22:22:16.728 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:22:16.728 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T22:22:16.729 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:22:16.729 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:22:16.729 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T22:22:16.729 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:22:16.729 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:22:16.729 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T22:22:16.729 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 6 2024-04-20T22:22:16.729 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:22:16.729 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T22:22:16.729 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 4 2024-04-20T22:22:16.729 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:22:16.730 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T22:22:16.730 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 14 2024-04-20T22:22:16.730 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T22:22:16.730 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:22:16.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:16.731+0000 7f81dce00700 1 -- 172.21.15.62:0/3925762327 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f81bc06ecc0 msgr2=0x7f81bc071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:22:16.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:16.731+0000 7f81dce00700 1 --2- 172.21.15.62:0/3925762327 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f81bc06ecc0 0x7f81bc071170 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f81c4000c00 tx=0x7f81c4005fb0 comp rx=0 tx=0).stop 2024-04-20T22:22:16.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:16.731+0000 7f81dce00700 1 -- 172.21.15.62:0/3925762327 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f81d8106b80 msgr2=0x7f81d819c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:22:16.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:16.731+0000 7f81dce00700 1 --2- 172.21.15.62:0/3925762327 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f81d8106b80 0x7f81d819c510 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f81c800b4d0 tx=0x7f81c800b890 comp rx=0 tx=0).stop 2024-04-20T22:22:16.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:16.731+0000 7f81dce00700 1 -- 172.21.15.62:0/3925762327 shutdown_connections 2024-04-20T22:22:16.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:16.731+0000 7f81dce00700 1 --2- 172.21.15.62:0/3925762327 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f81bc06ecc0 0x7f81bc071170 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:22:16.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:16.731+0000 7f81dce00700 1 --2- 172.21.15.62:0/3925762327 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f81d8106b80 0x7f81d819c510 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:22:16.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:16.731+0000 7f81dce00700 1 --2- 172.21.15.62:0/3925762327 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f81d810c8f0 0x7f81d819ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:22:16.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:16.731+0000 7f81dce00700 1 -- 172.21.15.62:0/3925762327 >> 172.21.15.62:0/3925762327 conn(0x7f81d8076060 msgr2=0x7f81d8077a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:22:16.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:16.732+0000 7f81dce00700 1 -- 172.21.15.62:0/3925762327 shutdown_connections 2024-04-20T22:22:16.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:16.732+0000 7f81dce00700 1 -- 172.21.15.62:0/3925762327 wait complete. 2024-04-20T22:22:16.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:22:16 smithi062 ceph-mon[76322]: from='client.36472 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:22:16.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:22:16 smithi062 ceph-mon[76322]: from='client.36476 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:22:16.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:16.894+0000 7f2fd6200700 1 -- 172.21.15.62:0/9377690 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2fd01061b0 msgr2=0x7f2fd0106580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:22:16.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:16.894+0000 7f2fd6200700 1 --2- 172.21.15.62:0/9377690 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2fd01061b0 0x7f2fd0106580 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f2fb8009910 tx=0x7f2fb8009c20 comp rx=0 tx=0).stop 2024-04-20T22:22:16.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:16.895+0000 7f2fd6200700 1 -- 172.21.15.62:0/9377690 shutdown_connections 2024-04-20T22:22:16.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:16.895+0000 7f2fd6200700 1 --2- 172.21.15.62:0/9377690 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2fd0068540 0x7f2fd00689b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:22:16.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:16.895+0000 7f2fd6200700 1 --2- 172.21.15.62:0/9377690 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2fd01061b0 0x7f2fd0106580 unknown :-1 s=CLOSED pgs=1205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:22:16.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:16.895+0000 7f2fd6200700 1 -- 172.21.15.62:0/9377690 >> 172.21.15.62:0/9377690 conn(0x7f2fd0074fa0 msgr2=0x7f2fd00753a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:22:16.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:16.895+0000 7f2fd6200700 1 -- 172.21.15.62:0/9377690 shutdown_connections 2024-04-20T22:22:16.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:16.895+0000 7f2fd6200700 1 -- 172.21.15.62:0/9377690 wait complete. 2024-04-20T22:22:16.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:16.896+0000 7f2fd6200700 1 Processor -- start 2024-04-20T22:22:16.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:16.896+0000 7f2fd6200700 1 -- start start 2024-04-20T22:22:16.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:16.896+0000 7f2fd6200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2fd0068540 0x7f2fd0197d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:22:16.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:16.897+0000 7f2fd6200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2fd01061b0 0x7f2fd01982a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:22:16.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:16.897+0000 7f2fd6200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fd0198980 con 0x7f2fd0068540 2024-04-20T22:22:16.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:16.897+0000 7f2fd6200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fd019c710 con 0x7f2fd01061b0 2024-04-20T22:22:16.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:16.897+0000 7f2fd4e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2fd0068540 0x7f2fd0197d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:22:16.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:16.897+0000 7f2fd4e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2fd0068540 0x7f2fd0197d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59130/0 (socket says 172.21.15.62:59130) 2024-04-20T22:22:16.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:16.897+0000 7f2fd4e00700 1 -- 172.21.15.62:0/3593700589 learned_addr learned my addr 172.21.15.62:0/3593700589 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:22:16.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:16.897+0000 7f2fd4e00700 1 -- 172.21.15.62:0/3593700589 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2fd01061b0 msgr2=0x7f2fd01982a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:22:16.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:16.897+0000 7f2fd4e00700 1 --2- 172.21.15.62:0/3593700589 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2fd01061b0 0x7f2fd01982a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:22:16.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:16.897+0000 7f2fd4e00700 1 -- 172.21.15.62:0/3593700589 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fb80095f0 con 0x7f2fd0068540 2024-04-20T22:22:16.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:16.898+0000 7f2fd4e00700 1 --2- 172.21.15.62:0/3593700589 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2fd0068540 0x7f2fd0197d60 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f2fb800b5c0 tx=0x7f2fb800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:22:16.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:16.898+0000 7f2fce000700 1 -- 172.21.15.62:0/3593700589 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2fb801d070 con 0x7f2fd0068540 2024-04-20T22:22:16.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:16.898+0000 7f2fce000700 1 -- 172.21.15.62:0/3593700589 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2fb8005e00 con 0x7f2fd0068540 2024-04-20T22:22:16.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:16.898+0000 7f2fce000700 1 -- 172.21.15.62:0/3593700589 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2fb8017400 con 0x7f2fd0068540 2024-04-20T22:22:16.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:16.898+0000 7f2fd6200700 1 -- 172.21.15.62:0/3593700589 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fd019c990 con 0x7f2fd0068540 2024-04-20T22:22:16.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:16.898+0000 7f2fd6200700 1 -- 172.21.15.62:0/3593700589 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fd019cf00 con 0x7f2fd0068540 2024-04-20T22:22:16.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:16.899+0000 7f2fd6200700 1 -- 172.21.15.62:0/3593700589 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fd0100eb0 con 0x7f2fd0068540 2024-04-20T22:22:16.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:16.900+0000 7f2fce000700 1 -- 172.21.15.62:0/3593700589 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f2fb800f460 con 0x7f2fd0068540 2024-04-20T22:22:16.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:16.901+0000 7f2fce000700 1 --2- 172.21.15.62:0/3593700589 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2fb006ecc0 0x7f2fb0071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:22:16.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:16.901+0000 7f2fce000700 1 -- 172.21.15.62:0/3593700589 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f2fb8090480 con 0x7f2fd0068540 2024-04-20T22:22:16.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:16.902+0000 7f2fc7e00700 1 --2- 172.21.15.62:0/3593700589 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2fb006ecc0 0x7f2fb0071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:22:16.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:16.902+0000 7f2fc7e00700 1 --2- 172.21.15.62:0/3593700589 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2fb006ecc0 0x7f2fb0071170 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f2fc0005950 tx=0x7f2fc00058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:22:16.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:16.906+0000 7f2fce000700 1 -- 172.21.15.62:0/3593700589 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f2fb8059190 con 0x7f2fd0068540 2024-04-20T22:22:16.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:22:16 smithi189 ceph-mon[57022]: from='client.36472 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:22:16.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:22:16 smithi189 ceph-mon[57022]: from='client.36476 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:22:17.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:17.191+0000 7f2fd6200700 1 -- 172.21.15.62:0/3593700589 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f2fd019d1e0 con 0x7f2fd0068540 2024-04-20T22:22:17.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:17.192+0000 7f2fce000700 1 -- 172.21.15.62:0/3593700589 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 29 v29) v1 ==== 76+0+1887 (secure 0 0 0) 0x7f2fb805c850 con 0x7f2fd0068540 2024-04-20T22:22:17.193 INFO:teuthology.orchestra.run.smithi062.stdout:e29 2024-04-20T22:22:17.193 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T22:22:17.193 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:22:17.193 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T22:22:17.194 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:22:17.194 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T22:22:17.194 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T22:22:17.194 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 20 2024-04-20T22:22:17.194 INFO:teuthology.orchestra.run.smithi062.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-20T22:22:17.194 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T22:22:17.194 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T21:42:01.559061+0000 2024-04-20T22:22:17.194 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T22:22:17.194 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T22:22:17.194 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T22:22:17.194 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T22:22:17.194 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T22:22:17.194 INFO:teuthology.orchestra.run.smithi062.stdout:max_xattr_size 65536 2024-04-20T22:22:17.194 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T22:22:17.195 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T22:22:17.195 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 81 2024-04-20T22:22:17.195 INFO:teuthology.orchestra.run.smithi062.stdout:snap_rstat 1 2024-04-20T22:22:17.195 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:22:17.195 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 2 2024-04-20T22:22:17.195 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1 2024-04-20T22:22:17.195 INFO:teuthology.orchestra.run.smithi062.stdout:up {} 2024-04-20T22:22:17.195 INFO:teuthology.orchestra.run.smithi062.stdout:failed 0,1 2024-04-20T22:22:17.195 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T22:22:17.195 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T22:22:17.195 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T22:22:17.195 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T22:22:17.195 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data enabled 2024-04-20T22:22:17.195 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T22:22:17.196 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T22:22:17.196 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 1 2024-04-20T22:22:17.196 INFO:teuthology.orchestra.run.smithi062.stdout:qdb_cluster leader: 0 members: 2024-04-20T22:22:17.196 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:22:17.196 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:22:17.196 INFO:teuthology.orchestra.run.smithi062.stdout:Standby daemons: 2024-04-20T22:22:17.196 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:22:17.196 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ryqblb{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6826/4113916205,v1:172.21.15.62:6827/4113916205] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:22:17.196 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ozizwi{-1:34476} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6828/3801975165,v1:172.21.15.62:6829/3801975165] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:22:17.196 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.dvimer{-1:44727} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6824/3576699232,v1:172.21.15.189:6825/3576699232] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:22:17.196 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.xbgspw{-1:45075} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6826/212905792,v1:172.21.15.189:6827/212905792] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:22:17.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:17.196+0000 7f2fd6200700 1 -- 172.21.15.62:0/3593700589 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2fb006ecc0 msgr2=0x7f2fb0071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:22:17.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:17.196+0000 7f2fd6200700 1 --2- 172.21.15.62:0/3593700589 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2fb006ecc0 0x7f2fb0071170 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f2fc0005950 tx=0x7f2fc00058e0 comp rx=0 tx=0).stop 2024-04-20T22:22:17.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:17.196+0000 7f2fd6200700 1 -- 172.21.15.62:0/3593700589 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2fd0068540 msgr2=0x7f2fd0197d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:22:17.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:17.196+0000 7f2fd6200700 1 --2- 172.21.15.62:0/3593700589 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2fd0068540 0x7f2fd0197d60 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f2fb800b5c0 tx=0x7f2fb800be20 comp rx=0 tx=0).stop 2024-04-20T22:22:17.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:17.196+0000 7f2fd6200700 1 -- 172.21.15.62:0/3593700589 shutdown_connections 2024-04-20T22:22:17.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:17.196+0000 7f2fd6200700 1 --2- 172.21.15.62:0/3593700589 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2fb006ecc0 0x7f2fb0071170 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:22:17.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:17.196+0000 7f2fd6200700 1 --2- 172.21.15.62:0/3593700589 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2fd0068540 0x7f2fd0197d60 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:22:17.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:17.196+0000 7f2fd6200700 1 --2- 172.21.15.62:0/3593700589 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2fd01061b0 0x7f2fd01982a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:22:17.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:17.196+0000 7f2fd6200700 1 -- 172.21.15.62:0/3593700589 >> 172.21.15.62:0/3593700589 conn(0x7f2fd0074fa0 msgr2=0x7f2fd0107100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:22:17.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:17.196+0000 7f2fd6200700 1 -- 172.21.15.62:0/3593700589 shutdown_connections 2024-04-20T22:22:17.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:17.197+0000 7f2fd6200700 1 -- 172.21.15.62:0/3593700589 wait complete. 2024-04-20T22:22:17.198 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 29 2024-04-20T22:22:17.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:17.357+0000 7f7520000700 1 -- 172.21.15.62:0/203701048 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f75180687a0 msgr2=0x7f7518068b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:22:17.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:17.357+0000 7f7520000700 1 --2- 172.21.15.62:0/203701048 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f75180687a0 0x7f7518068b70 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f750c009910 tx=0x7f750c009c20 comp rx=0 tx=0).stop 2024-04-20T22:22:17.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:17.358+0000 7f7520000700 1 -- 172.21.15.62:0/203701048 shutdown_connections 2024-04-20T22:22:17.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:17.358+0000 7f7520000700 1 --2- 172.21.15.62:0/203701048 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7518069140 0x7f7518104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:22:17.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:17.358+0000 7f7520000700 1 --2- 172.21.15.62:0/203701048 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f75180687a0 0x7f7518068b70 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:22:17.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:17.358+0000 7f7520000700 1 -- 172.21.15.62:0/203701048 >> 172.21.15.62:0/203701048 conn(0x7f7518075130 msgr2=0x7f7518075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:22:17.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:17.358+0000 7f7520000700 1 -- 172.21.15.62:0/203701048 shutdown_connections 2024-04-20T22:22:17.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:17.358+0000 7f7520000700 1 -- 172.21.15.62:0/203701048 wait complete. 2024-04-20T22:22:17.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:17.359+0000 7f7520000700 1 Processor -- start 2024-04-20T22:22:17.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:17.359+0000 7f7520000700 1 -- start start 2024-04-20T22:22:17.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:17.360+0000 7f7520000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75180687a0 0x7f75180ffca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:22:17.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:17.360+0000 7f7520000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7518069140 0x7f75181001e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:22:17.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:17.360+0000 7f7520000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7518100720 con 0x7f7518069140 2024-04-20T22:22:17.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:17.360+0000 7f7520000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7518100860 con 0x7f75180687a0 2024-04-20T22:22:17.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:17.360+0000 7f751cc00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7518069140 0x7f75181001e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:22:17.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:17.360+0000 7f751cc00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7518069140 0x7f75181001e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59144/0 (socket says 172.21.15.62:59144) 2024-04-20T22:22:17.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:17.360+0000 7f751cc00700 1 -- 172.21.15.62:0/3112676000 learned_addr learned my addr 172.21.15.62:0/3112676000 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:22:17.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:17.360+0000 7f751d600700 1 --2- 172.21.15.62:0/3112676000 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75180687a0 0x7f75180ffca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:22:17.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:17.361+0000 7f751cc00700 1 -- 172.21.15.62:0/3112676000 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75180687a0 msgr2=0x7f75180ffca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:22:17.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:17.361+0000 7f751cc00700 1 --2- 172.21.15.62:0/3112676000 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75180687a0 0x7f75180ffca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:22:17.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:17.361+0000 7f751cc00700 1 -- 172.21.15.62:0/3112676000 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f750c0095f0 con 0x7f7518069140 2024-04-20T22:22:17.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:17.361+0000 7f751cc00700 1 --2- 172.21.15.62:0/3112676000 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7518069140 0x7f75181001e0 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f750400d900 tx=0x7f750400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:22:17.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:17.361+0000 7f7516000700 1 -- 172.21.15.62:0/3112676000 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7504009900 con 0x7f7518069140 2024-04-20T22:22:17.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:17.361+0000 7f7516000700 1 -- 172.21.15.62:0/3112676000 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7504010460 con 0x7f7518069140 2024-04-20T22:22:17.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:17.361+0000 7f7516000700 1 -- 172.21.15.62:0/3112676000 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f750400f5f0 con 0x7f7518069140 2024-04-20T22:22:17.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:17.361+0000 7f7520000700 1 -- 172.21.15.62:0/3112676000 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75181a20b0 con 0x7f7518069140 2024-04-20T22:22:17.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:17.361+0000 7f7520000700 1 -- 172.21.15.62:0/3112676000 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75181a24c0 con 0x7f7518069140 2024-04-20T22:22:17.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:17.362+0000 7f7520000700 1 -- 172.21.15.62:0/3112676000 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7518108770 con 0x7f7518069140 2024-04-20T22:22:17.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:17.364+0000 7f7516000700 1 -- 172.21.15.62:0/3112676000 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f75040105d0 con 0x7f7518069140 2024-04-20T22:22:17.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:17.364+0000 7f7516000700 1 --2- 172.21.15.62:0/3112676000 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f750006ecc0 0x7f7500071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:22:17.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:17.365+0000 7f7516000700 1 -- 172.21.15.62:0/3112676000 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f750408eca0 con 0x7f7518069140 2024-04-20T22:22:17.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:17.365+0000 7f751d600700 1 --2- 172.21.15.62:0/3112676000 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f750006ecc0 0x7f7500071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:22:17.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:17.366+0000 7f751d600700 1 --2- 172.21.15.62:0/3112676000 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f750006ecc0 0x7f7500071170 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f750c006010 tx=0x7f750c01a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:22:17.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:17.369+0000 7f7516000700 1 -- 172.21.15.62:0/3112676000 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f7504057a30 con 0x7f7518069140 2024-04-20T22:22:17.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:17.626+0000 7f7520000700 1 -- 172.21.15.62:0/3112676000 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7518061190 con 0x7f750006ecc0 2024-04-20T22:22:17.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:17.629+0000 7f7516000700 1 -- 172.21.15.62:0/3112676000 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f7518061190 con 0x7f750006ecc0 2024-04-20T22:22:17.629 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:22:17.629 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T22:22:17.630 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T22:22:17.630 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-20T22:22:17.630 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T22:22:17.630 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr", 2024-04-20T22:22:17.630 INFO:teuthology.orchestra.run.smithi062.stdout: "osd", 2024-04-20T22:22:17.630 INFO:teuthology.orchestra.run.smithi062.stdout: "mds", 2024-04-20T22:22:17.630 INFO:teuthology.orchestra.run.smithi062.stdout: "crash", 2024-04-20T22:22:17.630 INFO:teuthology.orchestra.run.smithi062.stdout: "mon" 2024-04-20T22:22:17.630 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T22:22:17.630 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "16/23 daemons upgraded", 2024-04-20T22:22:17.630 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading mds daemons", 2024-04-20T22:22:17.630 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T22:22:17.630 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:22:17.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:17.632+0000 7f7520000700 1 -- 172.21.15.62:0/3112676000 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f750006ecc0 msgr2=0x7f7500071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:22:17.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:17.632+0000 7f7520000700 1 --2- 172.21.15.62:0/3112676000 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f750006ecc0 0x7f7500071170 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f750c006010 tx=0x7f750c01a040 comp rx=0 tx=0).stop 2024-04-20T22:22:17.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:17.632+0000 7f7520000700 1 -- 172.21.15.62:0/3112676000 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7518069140 msgr2=0x7f75181001e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:22:17.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:17.632+0000 7f7520000700 1 --2- 172.21.15.62:0/3112676000 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7518069140 0x7f75181001e0 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f750400d900 tx=0x7f750400dcc0 comp rx=0 tx=0).stop 2024-04-20T22:22:17.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:17.632+0000 7f7520000700 1 -- 172.21.15.62:0/3112676000 shutdown_connections 2024-04-20T22:22:17.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:17.632+0000 7f7520000700 1 --2- 172.21.15.62:0/3112676000 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f750006ecc0 0x7f7500071170 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:22:17.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:17.632+0000 7f7520000700 1 --2- 172.21.15.62:0/3112676000 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75180687a0 0x7f75180ffca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:22:17.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:17.632+0000 7f7520000700 1 --2- 172.21.15.62:0/3112676000 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7518069140 0x7f75181001e0 unknown :-1 s=CLOSED pgs=1208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:22:17.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:17.632+0000 7f7520000700 1 -- 172.21.15.62:0/3112676000 >> 172.21.15.62:0/3112676000 conn(0x7f7518075130 msgr2=0x7f75180fe7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:22:17.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:17.632+0000 7f7520000700 1 -- 172.21.15.62:0/3112676000 shutdown_connections 2024-04-20T22:22:17.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:17.633+0000 7f7520000700 1 -- 172.21.15.62:0/3112676000 wait complete. 2024-04-20T22:22:17.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:22:17 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/3925762327' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:22:17.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:22:17 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/3593700589' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:22:17.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:22:17 smithi062 ceph-mon[76322]: pgmap v1413: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:22:17.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:17.793+0000 7f1be8600700 1 -- 172.21.15.62:0/3249684467 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1be0100e80 msgr2=0x7f1be0104e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:22:17.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:17.793+0000 7f1be8600700 1 --2- 172.21.15.62:0/3249684467 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1be0100e80 0x7f1be0104e50 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f1bcc009910 tx=0x7f1bcc009c20 comp rx=0 tx=0).stop 2024-04-20T22:22:17.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:17.793+0000 7f1be8600700 1 -- 172.21.15.62:0/3249684467 shutdown_connections 2024-04-20T22:22:17.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:17.793+0000 7f1be8600700 1 --2- 172.21.15.62:0/3249684467 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1be0100e80 0x7f1be0104e50 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:22:17.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:17.793+0000 7f1be8600700 1 --2- 172.21.15.62:0/3249684467 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1be01004e0 0x7f1be01008b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:22:17.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:17.793+0000 7f1be8600700 1 -- 172.21.15.62:0/3249684467 >> 172.21.15.62:0/3249684467 conn(0x7f1be00fbda0 msgr2=0x7f1be00fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:22:17.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:17.793+0000 7f1be8600700 1 -- 172.21.15.62:0/3249684467 shutdown_connections 2024-04-20T22:22:17.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:17.794+0000 7f1be8600700 1 -- 172.21.15.62:0/3249684467 wait complete. 2024-04-20T22:22:17.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:17.795+0000 7f1be8600700 1 Processor -- start 2024-04-20T22:22:17.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:17.795+0000 7f1be8600700 1 -- start start 2024-04-20T22:22:17.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:17.795+0000 7f1be8600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1be01004e0 0x7f1be01980f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:22:17.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:17.795+0000 7f1be8600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1be0100e80 0x7f1be0198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:22:17.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:17.795+0000 7f1be8600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1be0198d10 con 0x7f1be01004e0 2024-04-20T22:22:17.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:17.795+0000 7f1be8600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1be019caa0 con 0x7f1be0100e80 2024-04-20T22:22:17.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:17.796+0000 7f1be5c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1be01004e0 0x7f1be01980f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:22:17.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:17.796+0000 7f1be5c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1be01004e0 0x7f1be01980f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59170/0 (socket says 172.21.15.62:59170) 2024-04-20T22:22:17.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:17.796+0000 7f1be5c00700 1 -- 172.21.15.62:0/3548643627 learned_addr learned my addr 172.21.15.62:0/3548643627 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:22:17.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:17.796+0000 7f1be5200700 1 --2- 172.21.15.62:0/3548643627 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1be0100e80 0x7f1be0198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:22:17.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:17.796+0000 7f1be5c00700 1 -- 172.21.15.62:0/3548643627 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1be0100e80 msgr2=0x7f1be0198630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:22:17.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:17.796+0000 7f1be5c00700 1 --2- 172.21.15.62:0/3548643627 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1be0100e80 0x7f1be0198630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:22:17.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:17.796+0000 7f1be5c00700 1 -- 172.21.15.62:0/3548643627 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bcc0095f0 con 0x7f1be01004e0 2024-04-20T22:22:17.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:17.796+0000 7f1be5200700 1 --2- 172.21.15.62:0/3548643627 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1be0100e80 0x7f1be0198630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-20T22:22:17.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:17.797+0000 7f1be5c00700 1 --2- 172.21.15.62:0/3548643627 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1be01004e0 0x7f1be01980f0 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f1bd400c8a0 tx=0x7f1bd400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:22:17.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:17.797+0000 7f1bde000700 1 -- 172.21.15.62:0/3548643627 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1bd400cea0 con 0x7f1be01004e0 2024-04-20T22:22:17.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:17.797+0000 7f1bde000700 1 -- 172.21.15.62:0/3548643627 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1bd4004830 con 0x7f1be01004e0 2024-04-20T22:22:17.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:17.797+0000 7f1bde000700 1 -- 172.21.15.62:0/3548643627 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1bd40055b0 con 0x7f1be01004e0 2024-04-20T22:22:17.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:17.797+0000 7f1be8600700 1 -- 172.21.15.62:0/3548643627 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1be019cd80 con 0x7f1be01004e0 2024-04-20T22:22:17.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:17.797+0000 7f1be8600700 1 -- 172.21.15.62:0/3548643627 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1be019d2d0 con 0x7f1be01004e0 2024-04-20T22:22:17.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:17.799+0000 7f1be8600700 1 -- 172.21.15.62:0/3548643627 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1be01045d0 con 0x7f1be01004e0 2024-04-20T22:22:17.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:17.800+0000 7f1bde000700 1 -- 172.21.15.62:0/3548643627 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f1bd40049a0 con 0x7f1be01004e0 2024-04-20T22:22:17.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:17.801+0000 7f1bde000700 1 --2- 172.21.15.62:0/3548643627 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f1bc806ecc0 0x7f1bc8071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:22:17.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:17.801+0000 7f1bde000700 1 -- 172.21.15.62:0/3548643627 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f1bd408eaf0 con 0x7f1be01004e0 2024-04-20T22:22:17.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:17.801+0000 7f1be5200700 1 --2- 172.21.15.62:0/3548643627 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f1bc806ecc0 0x7f1bc8071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:22:17.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:17.802+0000 7f1be5200700 1 --2- 172.21.15.62:0/3548643627 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f1bc806ecc0 0x7f1bc8071170 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f1be0199710 tx=0x7f1bcc01a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:22:17.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:17.807+0000 7f1bde000700 1 -- 172.21.15.62:0/3548643627 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f1bd4057880 con 0x7f1be01004e0 2024-04-20T22:22:17.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:22:17 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/3925762327' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:22:17.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:22:17 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/3593700589' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:22:17.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:22:17 smithi189 ceph-mon[57022]: pgmap v1413: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:22:18.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:18.141+0000 7f1be8600700 1 -- 172.21.15.62:0/3548643627 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f1be004f2a0 con 0x7f1be01004e0 2024-04-20T22:22:18.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:18.142+0000 7f1bde000700 1 -- 172.21.15.62:0/3548643627 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7f1bd401d090 con 0x7f1be01004e0 2024-04-20T22:22:18.143 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-20T22:22:18.144 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-20T22:22:18.144 INFO:teuthology.orchestra.run.smithi062.stdout: host smithi189 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi189/config 2024-04-20T22:22:18.144 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-20T22:22:18.144 INFO:teuthology.orchestra.run.smithi062.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi189 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp11670v2q:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:22:18.144 INFO:teuthology.orchestra.run.smithi062.stdout:Traceback (most recent call last): 2024-04-20T22:22:18.144 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-20T22:22:18.144 INFO:teuthology.orchestra.run.smithi062.stdout: return _run_code(code, main_globals, None, 2024-04-20T22:22:18.144 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-20T22:22:18.144 INFO:teuthology.orchestra.run.smithi062.stdout: exec(code, run_globals) 2024-04-20T22:22:18.144 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5618, in 2024-04-20T22:22:18.144 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5606, in main 2024-04-20T22:22:18.144 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 408, in _infer_config 2024-04-20T22:22:18.145 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 323, in _infer_fsid 2024-04-20T22:22:18.145 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 436, in _infer_image 2024-04-20T22:22:18.145 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 310, in _validate_fsid 2024-04-20T22:22:18.145 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 3359, in command_ceph_volume 2024-04-20T22:22:18.145 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-20T22:22:18.145 INFO:teuthology.orchestra.run.smithi062.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi189 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp11670v2q:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:22:18.145 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-20T22:22:18.145 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is degraded 2024-04-20T22:22:18.145 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T22:22:18.145 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T22:22:18.145 INFO:teuthology.orchestra.run.smithi062.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-20T22:22:18.145 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-20T22:22:18.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:18.146+0000 7f1be8600700 1 -- 172.21.15.62:0/3548643627 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f1bc806ecc0 msgr2=0x7f1bc8071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:22:18.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:18.146+0000 7f1be8600700 1 --2- 172.21.15.62:0/3548643627 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f1bc806ecc0 0x7f1bc8071170 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f1be0199710 tx=0x7f1bcc01a040 comp rx=0 tx=0).stop 2024-04-20T22:22:18.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:18.146+0000 7f1be8600700 1 -- 172.21.15.62:0/3548643627 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1be01004e0 msgr2=0x7f1be01980f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:22:18.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:18.146+0000 7f1be8600700 1 --2- 172.21.15.62:0/3548643627 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1be01004e0 0x7f1be01980f0 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f1bd400c8a0 tx=0x7f1bd400cc60 comp rx=0 tx=0).stop 2024-04-20T22:22:18.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:18.146+0000 7f1be8600700 1 -- 172.21.15.62:0/3548643627 shutdown_connections 2024-04-20T22:22:18.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:18.146+0000 7f1be8600700 1 --2- 172.21.15.62:0/3548643627 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f1bc806ecc0 0x7f1bc8071170 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:22:18.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:18.146+0000 7f1be8600700 1 --2- 172.21.15.62:0/3548643627 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1be01004e0 0x7f1be01980f0 unknown :-1 s=CLOSED pgs=1210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:22:18.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:18.146+0000 7f1be8600700 1 --2- 172.21.15.62:0/3548643627 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1be0100e80 0x7f1be0198630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:22:18.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:18.146+0000 7f1be8600700 1 -- 172.21.15.62:0/3548643627 >> 172.21.15.62:0/3548643627 conn(0x7f1be00fbda0 msgr2=0x7f1be00fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:22:18.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:18.147+0000 7f1be8600700 1 -- 172.21.15.62:0/3548643627 shutdown_connections 2024-04-20T22:22:18.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:18.147+0000 7f1be8600700 1 -- 172.21.15.62:0/3548643627 wait complete. 2024-04-20T22:22:18.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:22:18 smithi062 ceph-mon[76322]: from='client.36488 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:22:18.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:22:18 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/3548643627' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:22:18.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:22:18 smithi189 ceph-mon[57022]: from='client.36488 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:22:18.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:22:18 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/3548643627' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:22:19.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:22:19 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:22:19.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:22:19 smithi189 ceph-mon[57022]: pgmap v1414: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:22:20.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:22:19 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:22:20.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:22:19 smithi062 ceph-mon[76322]: pgmap v1414: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:22:21.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:22:21 smithi062 ceph-mon[76322]: pgmap v1415: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:22:21.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:22:21 smithi189 ceph-mon[57022]: pgmap v1415: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:22:23.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:22:23 smithi062 ceph-mon[76322]: pgmap v1416: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:22:23.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:22:23 smithi189 ceph-mon[57022]: pgmap v1416: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:22:26.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:22:25 smithi062 ceph-mon[76322]: pgmap v1417: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:22:26.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:22:25 smithi189 ceph-mon[57022]: pgmap v1417: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:22:27.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:22:27 smithi062 ceph-mon[76322]: pgmap v1418: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:22:27.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:22:27 smithi189 ceph-mon[57022]: pgmap v1418: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:22:29.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:22:29 smithi062 ceph-mon[76322]: pgmap v1419: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:22:29.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:22:29 smithi189 ceph-mon[57022]: pgmap v1419: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:22:31.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:22:31 smithi062 ceph-mon[76322]: pgmap v1420: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:22:31.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:22:31 smithi189 ceph-mon[57022]: pgmap v1420: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:22:33.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:22:33 smithi062 ceph-mon[76322]: pgmap v1421: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:22:33.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:22:33 smithi189 ceph-mon[57022]: pgmap v1421: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:22:34.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:22:34 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:22:34.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:22:34 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:22:35.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:22:35 smithi062 ceph-mon[76322]: pgmap v1422: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:22:35.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:22:35 smithi189 ceph-mon[57022]: pgmap v1422: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:22:37.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:22:37 smithi062 ceph-mon[76322]: pgmap v1423: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:22:37.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:22:37 smithi189 ceph-mon[57022]: pgmap v1423: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:22:39.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:22:39 smithi062 ceph-mon[76322]: pgmap v1424: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:22:39.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:22:39 smithi189 ceph-mon[57022]: pgmap v1424: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:22:41.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:22:41 smithi062 ceph-mon[76322]: pgmap v1425: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:22:41.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:22:41 smithi189 ceph-mon[57022]: pgmap v1425: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:22:44.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:22:44 smithi062 ceph-mon[76322]: pgmap v1426: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:22:44.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:22:44 smithi189 ceph-mon[57022]: pgmap v1426: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:22:46.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:22:46 smithi062 ceph-mon[76322]: pgmap v1427: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:22:46.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:22:46 smithi189 ceph-mon[57022]: pgmap v1427: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:22:48.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:48.326+0000 7f6741e00700 1 -- 172.21.15.62:0/1584514260 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f673c108470 msgr2=0x7f673c108840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:22:48.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:48.326+0000 7f6741e00700 1 --2- 172.21.15.62:0/1584514260 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f673c108470 0x7f673c108840 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f672c00b0a0 tx=0x7f672c00b3b0 comp rx=0 tx=0).stop 2024-04-20T22:22:48.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:48.326+0000 7f6741e00700 1 -- 172.21.15.62:0/1584514260 shutdown_connections 2024-04-20T22:22:48.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:48.326+0000 7f6741e00700 1 --2- 172.21.15.62:0/1584514260 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f673c102470 0x7f673c1028e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:22:48.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:48.326+0000 7f6741e00700 1 --2- 172.21.15.62:0/1584514260 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f673c108470 0x7f673c108840 unknown :-1 s=CLOSED pgs=1211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:22:48.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:48.326+0000 7f6741e00700 1 -- 172.21.15.62:0/1584514260 >> 172.21.15.62:0/1584514260 conn(0x7f673c0fdf90 msgr2=0x7f673c1003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:22:48.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:48.326+0000 7f6741e00700 1 -- 172.21.15.62:0/1584514260 shutdown_connections 2024-04-20T22:22:48.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:48.326+0000 7f6741e00700 1 -- 172.21.15.62:0/1584514260 wait complete. 2024-04-20T22:22:48.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:48.327+0000 7f6741e00700 1 Processor -- start 2024-04-20T22:22:48.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:48.327+0000 7f6741e00700 1 -- start start 2024-04-20T22:22:48.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:48.328+0000 7f6741e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f673c102470 0x7f673c198230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:22:48.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:48.328+0000 7f6741e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f673c108470 0x7f673c198770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:22:48.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:48.328+0000 7f6741e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f673c198e50 con 0x7f673c102470 2024-04-20T22:22:48.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:48.328+0000 7f6741e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f673c19cbe0 con 0x7f673c108470 2024-04-20T22:22:48.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:48.328+0000 7f673aa00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f673c102470 0x7f673c198230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:22:48.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:48.328+0000 7f673aa00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f673c102470 0x7f673c198230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42002/0 (socket says 172.21.15.62:42002) 2024-04-20T22:22:48.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:48.328+0000 7f673aa00700 1 -- 172.21.15.62:0/4026216492 learned_addr learned my addr 172.21.15.62:0/4026216492 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:22:48.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:48.328+0000 7f6732000700 1 --2- 172.21.15.62:0/4026216492 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f673c108470 0x7f673c198770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:22:48.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:48.328+0000 7f673aa00700 1 -- 172.21.15.62:0/4026216492 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f673c108470 msgr2=0x7f673c198770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:22:48.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:48.328+0000 7f673aa00700 1 --2- 172.21.15.62:0/4026216492 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f673c108470 0x7f673c198770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:22:48.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:48.329+0000 7f673aa00700 1 -- 172.21.15.62:0/4026216492 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f672c009d00 con 0x7f673c102470 2024-04-20T22:22:48.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:48.329+0000 7f6732000700 1 --2- 172.21.15.62:0/4026216492 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f673c108470 0x7f673c198770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T22:22:48.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:48.329+0000 7f673aa00700 1 --2- 172.21.15.62:0/4026216492 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f673c102470 0x7f673c198230 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f672c00b070 tx=0x7f672c00cdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:22:48.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:48.329+0000 7f6740a00700 1 -- 172.21.15.62:0/4026216492 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f672c00e040 con 0x7f673c102470 2024-04-20T22:22:48.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:48.329+0000 7f6740a00700 1 -- 172.21.15.62:0/4026216492 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f672c003e60 con 0x7f673c102470 2024-04-20T22:22:48.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:48.329+0000 7f6741e00700 1 -- 172.21.15.62:0/4026216492 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f673c19ce60 con 0x7f673c102470 2024-04-20T22:22:48.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:48.329+0000 7f6740a00700 1 -- 172.21.15.62:0/4026216492 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f672c01d910 con 0x7f673c102470 2024-04-20T22:22:48.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:48.329+0000 7f6741e00700 1 -- 172.21.15.62:0/4026216492 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f673c19d350 con 0x7f673c102470 2024-04-20T22:22:48.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:48.331+0000 7f6741e00700 1 -- 172.21.15.62:0/4026216492 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f673c10a9e0 con 0x7f673c102470 2024-04-20T22:22:48.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:48.332+0000 7f6740a00700 1 -- 172.21.15.62:0/4026216492 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f672c004820 con 0x7f673c102470 2024-04-20T22:22:48.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:48.332+0000 7f6740a00700 1 --2- 172.21.15.62:0/4026216492 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f672006ecc0 0x7f6720071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:22:48.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:48.333+0000 7f6740a00700 1 -- 172.21.15.62:0/4026216492 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f672c090860 con 0x7f673c102470 2024-04-20T22:22:48.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:48.333+0000 7f6732000700 1 --2- 172.21.15.62:0/4026216492 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f672006ecc0 0x7f6720071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:22:48.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:48.334+0000 7f6732000700 1 --2- 172.21.15.62:0/4026216492 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f672006ecc0 0x7f6720071170 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f6724005fd0 tx=0x7f6724005dc0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:22:48.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:48.338+0000 7f6740a00700 1 -- 172.21.15.62:0/4026216492 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f672c059570 con 0x7f673c102470 2024-04-20T22:22:48.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:48.600+0000 7f6741e00700 1 -- 172.21.15.62:0/4026216492 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f673c1996d0 con 0x7f672006ecc0 2024-04-20T22:22:48.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:48.602+0000 7f6740a00700 1 -- 172.21.15.62:0/4026216492 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f67240086c0 con 0x7f672006ecc0 2024-04-20T22:22:48.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:48.605+0000 7f6741e00700 1 -- 172.21.15.62:0/4026216492 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f672006ecc0 msgr2=0x7f6720071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:22:48.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:48.605+0000 7f6741e00700 1 --2- 172.21.15.62:0/4026216492 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f672006ecc0 0x7f6720071170 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f6724005fd0 tx=0x7f6724005dc0 comp rx=0 tx=0).stop 2024-04-20T22:22:48.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:48.605+0000 7f6741e00700 1 -- 172.21.15.62:0/4026216492 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f673c102470 msgr2=0x7f673c198230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:22:48.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:48.605+0000 7f6741e00700 1 --2- 172.21.15.62:0/4026216492 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f673c102470 0x7f673c198230 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f672c00b070 tx=0x7f672c00cdd0 comp rx=0 tx=0).stop 2024-04-20T22:22:48.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:48.606+0000 7f6741e00700 1 -- 172.21.15.62:0/4026216492 shutdown_connections 2024-04-20T22:22:48.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:48.606+0000 7f6741e00700 1 --2- 172.21.15.62:0/4026216492 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f672006ecc0 0x7f6720071170 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:22:48.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:48.606+0000 7f6741e00700 1 --2- 172.21.15.62:0/4026216492 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f673c102470 0x7f673c198230 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:22:48.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:48.606+0000 7f6741e00700 1 --2- 172.21.15.62:0/4026216492 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f673c108470 0x7f673c198770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:22:48.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:48.606+0000 7f6741e00700 1 -- 172.21.15.62:0/4026216492 >> 172.21.15.62:0/4026216492 conn(0x7f673c0fdf90 msgr2=0x7f673c100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:22:48.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:48.606+0000 7f6741e00700 1 -- 172.21.15.62:0/4026216492 shutdown_connections 2024-04-20T22:22:48.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:48.606+0000 7f6741e00700 1 -- 172.21.15.62:0/4026216492 wait complete. 2024-04-20T22:22:48.622 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T22:22:48.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:48.772+0000 7f1791400700 1 -- 172.21.15.62:0/3653276869 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f178c102470 msgr2=0x7f178c1028e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:22:48.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:48.772+0000 7f1791400700 1 --2- 172.21.15.62:0/3653276869 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f178c102470 0x7f178c1028e0 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7f1778009910 tx=0x7f1778009c20 comp rx=0 tx=0).stop 2024-04-20T22:22:48.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:48.772+0000 7f1791400700 1 -- 172.21.15.62:0/3653276869 shutdown_connections 2024-04-20T22:22:48.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:48.772+0000 7f1791400700 1 --2- 172.21.15.62:0/3653276869 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f178c102470 0x7f178c1028e0 unknown :-1 s=CLOSED pgs=1213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:22:48.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:48.772+0000 7f1791400700 1 --2- 172.21.15.62:0/3653276869 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f178c108470 0x7f178c108840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:22:48.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:48.772+0000 7f1791400700 1 -- 172.21.15.62:0/3653276869 >> 172.21.15.62:0/3653276869 conn(0x7f178c0fdf90 msgr2=0x7f178c1003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:22:48.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:48.772+0000 7f1791400700 1 -- 172.21.15.62:0/3653276869 shutdown_connections 2024-04-20T22:22:48.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:48.772+0000 7f1791400700 1 -- 172.21.15.62:0/3653276869 wait complete. 2024-04-20T22:22:48.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:48.773+0000 7f1791400700 1 Processor -- start 2024-04-20T22:22:48.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:48.773+0000 7f1791400700 1 -- start start 2024-04-20T22:22:48.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:48.774+0000 7f1791400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f178c102470 0x7f178c1981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:22:48.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:48.774+0000 7f1791400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f178c108470 0x7f178c198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:22:48.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:48.774+0000 7f1791400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f178c198e10 con 0x7f178c108470 2024-04-20T22:22:48.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:48.774+0000 7f1791400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f178c19cba0 con 0x7f178c102470 2024-04-20T22:22:48.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:48.774+0000 7f1789e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f178c108470 0x7f178c198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:22:48.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:48.774+0000 7f1789e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f178c108470 0x7f178c198730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42006/0 (socket says 172.21.15.62:42006) 2024-04-20T22:22:48.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:48.774+0000 7f1789e00700 1 -- 172.21.15.62:0/337577033 learned_addr learned my addr 172.21.15.62:0/337577033 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:22:48.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:48.774+0000 7f178a800700 1 --2- 172.21.15.62:0/337577033 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f178c102470 0x7f178c1981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:22:48.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:48.774+0000 7f1789e00700 1 -- 172.21.15.62:0/337577033 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f178c102470 msgr2=0x7f178c1981f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:22:48.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:48.774+0000 7f1789e00700 1 --2- 172.21.15.62:0/337577033 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f178c102470 0x7f178c1981f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:22:48.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:48.775+0000 7f1789e00700 1 -- 172.21.15.62:0/337577033 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17780095f0 con 0x7f178c108470 2024-04-20T22:22:48.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:48.775+0000 7f1789e00700 1 --2- 172.21.15.62:0/337577033 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f178c108470 0x7f178c198730 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f17780092e0 tx=0x7f177800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:22:48.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:48.775+0000 7f1783400700 1 -- 172.21.15.62:0/337577033 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f177801d070 con 0x7f178c108470 2024-04-20T22:22:48.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:48.775+0000 7f1783400700 1 -- 172.21.15.62:0/337577033 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1778005e00 con 0x7f178c108470 2024-04-20T22:22:48.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:48.775+0000 7f1783400700 1 -- 172.21.15.62:0/337577033 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1778017400 con 0x7f178c108470 2024-04-20T22:22:48.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:48.775+0000 7f1791400700 1 -- 172.21.15.62:0/337577033 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f178c19ce20 con 0x7f178c108470 2024-04-20T22:22:48.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:48.775+0000 7f1791400700 1 -- 172.21.15.62:0/337577033 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f178c19d310 con 0x7f178c108470 2024-04-20T22:22:48.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:48.777+0000 7f1791400700 1 -- 172.21.15.62:0/337577033 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f178c10a970 con 0x7f178c108470 2024-04-20T22:22:48.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:48.778+0000 7f1783400700 1 -- 172.21.15.62:0/337577033 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f177800f460 con 0x7f178c108470 2024-04-20T22:22:48.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:48.779+0000 7f1783400700 1 --2- 172.21.15.62:0/337577033 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f177006ecc0 0x7f1770071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:22:48.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:48.779+0000 7f1783400700 1 -- 172.21.15.62:0/337577033 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f1778090330 con 0x7f178c108470 2024-04-20T22:22:48.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:48.779+0000 7f178a800700 1 --2- 172.21.15.62:0/337577033 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f177006ecc0 0x7f1770071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:22:48.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:48.780+0000 7f178a800700 1 --2- 172.21.15.62:0/337577033 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f177006ecc0 0x7f1770071170 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f177c005950 tx=0x7f177c0058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:22:48.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:48.784+0000 7f1783400700 1 -- 172.21.15.62:0/337577033 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f17780590c0 con 0x7f178c108470 2024-04-20T22:22:48.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:22:48 smithi062 ceph-mon[76322]: pgmap v1428: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:22:48.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:22:48 smithi189 ceph-mon[57022]: pgmap v1428: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:22:49.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:49.048+0000 7f1791400700 1 -- 172.21.15.62:0/337577033 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f178c19d580 con 0x7f177006ecc0 2024-04-20T22:22:49.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:49.050+0000 7f1783400700 1 -- 172.21.15.62:0/337577033 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f178c19d580 con 0x7f177006ecc0 2024-04-20T22:22:49.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:49.053+0000 7f1791400700 1 -- 172.21.15.62:0/337577033 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f177006ecc0 msgr2=0x7f1770071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:22:49.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:49.053+0000 7f1791400700 1 --2- 172.21.15.62:0/337577033 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f177006ecc0 0x7f1770071170 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f177c005950 tx=0x7f177c0058e0 comp rx=0 tx=0).stop 2024-04-20T22:22:49.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:49.053+0000 7f1791400700 1 -- 172.21.15.62:0/337577033 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f178c108470 msgr2=0x7f178c198730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:22:49.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:49.053+0000 7f1791400700 1 --2- 172.21.15.62:0/337577033 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f178c108470 0x7f178c198730 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f17780092e0 tx=0x7f177800be20 comp rx=0 tx=0).stop 2024-04-20T22:22:49.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:49.053+0000 7f1791400700 1 -- 172.21.15.62:0/337577033 shutdown_connections 2024-04-20T22:22:49.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:49.054+0000 7f1791400700 1 --2- 172.21.15.62:0/337577033 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f177006ecc0 0x7f1770071170 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:22:49.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:49.054+0000 7f1791400700 1 --2- 172.21.15.62:0/337577033 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f178c102470 0x7f178c1981f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:22:49.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:49.054+0000 7f1791400700 1 --2- 172.21.15.62:0/337577033 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f178c108470 0x7f178c198730 unknown :-1 s=CLOSED pgs=1214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:22:49.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:49.054+0000 7f1791400700 1 -- 172.21.15.62:0/337577033 >> 172.21.15.62:0/337577033 conn(0x7f178c0fdf90 msgr2=0x7f178c0ff3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:22:49.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:49.054+0000 7f1791400700 1 -- 172.21.15.62:0/337577033 shutdown_connections 2024-04-20T22:22:49.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:49.054+0000 7f1791400700 1 -- 172.21.15.62:0/337577033 wait complete. 2024-04-20T22:22:49.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:49.213+0000 7fd3fbe00700 1 -- 172.21.15.62:0/2849033123 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3fc073a00 msgr2=0x7fd3fc10c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:22:49.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:49.213+0000 7fd3fbe00700 1 --2- 172.21.15.62:0/2849033123 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3fc073a00 0x7fd3fc10c330 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7fd3e4009910 tx=0x7fd3e4009c20 comp rx=0 tx=0).stop 2024-04-20T22:22:49.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:49.213+0000 7fd3fbe00700 1 -- 172.21.15.62:0/2849033123 shutdown_connections 2024-04-20T22:22:49.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:49.213+0000 7fd3fbe00700 1 --2- 172.21.15.62:0/2849033123 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3fc073a00 0x7fd3fc10c330 unknown :-1 s=CLOSED pgs=1215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:22:49.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:49.213+0000 7fd3fbe00700 1 --2- 172.21.15.62:0/2849033123 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3fc0730f0 0x7fd3fc0734c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:22:49.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:49.213+0000 7fd3fbe00700 1 -- 172.21.15.62:0/2849033123 >> 172.21.15.62:0/2849033123 conn(0x7fd3fc0fbb10 msgr2=0x7fd3fc0fdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:22:49.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:49.213+0000 7fd3fbe00700 1 -- 172.21.15.62:0/2849033123 shutdown_connections 2024-04-20T22:22:49.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:49.213+0000 7fd3fbe00700 1 -- 172.21.15.62:0/2849033123 wait complete. 2024-04-20T22:22:49.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:49.214+0000 7fd3fbe00700 1 Processor -- start 2024-04-20T22:22:49.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:49.215+0000 7fd3fbe00700 1 -- start start 2024-04-20T22:22:49.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:49.215+0000 7fd3fbe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3fc0730f0 0x7fd3fc19dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:22:49.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:49.215+0000 7fd3fbe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3fc073a00 0x7fd3fc19e130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:22:49.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:49.215+0000 7fd3fbe00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3fc197b40 con 0x7fd3fc073a00 2024-04-20T22:22:49.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:49.215+0000 7fd3fbe00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3fc197cb0 con 0x7fd3fc0730f0 2024-04-20T22:22:49.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:49.215+0000 7fd3f3e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3fc073a00 0x7fd3fc19e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:22:49.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:49.215+0000 7fd3f3e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3fc073a00 0x7fd3fc19e130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42030/0 (socket says 172.21.15.62:42030) 2024-04-20T22:22:49.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:49.215+0000 7fd3f3e00700 1 -- 172.21.15.62:0/3348999628 learned_addr learned my addr 172.21.15.62:0/3348999628 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:22:49.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:49.216+0000 7fd3faa00700 1 --2- 172.21.15.62:0/3348999628 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3fc0730f0 0x7fd3fc19dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:22:49.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:49.216+0000 7fd3f3e00700 1 -- 172.21.15.62:0/3348999628 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3fc0730f0 msgr2=0x7fd3fc19dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:22:49.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:49.216+0000 7fd3f3e00700 1 --2- 172.21.15.62:0/3348999628 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3fc0730f0 0x7fd3fc19dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:22:49.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:49.216+0000 7fd3f3e00700 1 -- 172.21.15.62:0/3348999628 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3e40095f0 con 0x7fd3fc073a00 2024-04-20T22:22:49.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:49.216+0000 7fd3faa00700 1 --2- 172.21.15.62:0/3348999628 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3fc0730f0 0x7fd3fc19dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T22:22:49.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:49.216+0000 7fd3f3e00700 1 --2- 172.21.15.62:0/3348999628 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3fc073a00 0x7fd3fc19e130 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7fd3e400b5c0 tx=0x7fd3e400be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:22:49.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:49.216+0000 7fd3f3400700 1 -- 172.21.15.62:0/3348999628 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd3e401d070 con 0x7fd3fc073a00 2024-04-20T22:22:49.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:49.217+0000 7fd3f3400700 1 -- 172.21.15.62:0/3348999628 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd3e400f460 con 0x7fd3fc073a00 2024-04-20T22:22:49.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:49.217+0000 7fd3f3400700 1 -- 172.21.15.62:0/3348999628 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd3e4017630 con 0x7fd3fc073a00 2024-04-20T22:22:49.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:49.217+0000 7fd3fbe00700 1 -- 172.21.15.62:0/3348999628 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3fc197f90 con 0x7fd3fc073a00 2024-04-20T22:22:49.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:49.217+0000 7fd3fbe00700 1 -- 172.21.15.62:0/3348999628 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3fc1984e0 con 0x7fd3fc073a00 2024-04-20T22:22:49.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:49.218+0000 7fd3fbe00700 1 -- 172.21.15.62:0/3348999628 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3fc109a30 con 0x7fd3fc073a00 2024-04-20T22:22:49.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:49.219+0000 7fd3f3400700 1 -- 172.21.15.62:0/3348999628 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fd3e400f5d0 con 0x7fd3fc073a00 2024-04-20T22:22:49.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:49.220+0000 7fd3f3400700 1 --2- 172.21.15.62:0/3348999628 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd3e006ecc0 0x7fd3e0071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:22:49.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:49.220+0000 7fd3f3400700 1 -- 172.21.15.62:0/3348999628 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fd3e40903c0 con 0x7fd3fc073a00 2024-04-20T22:22:49.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:49.220+0000 7fd3faa00700 1 --2- 172.21.15.62:0/3348999628 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd3e006ecc0 0x7fd3e0071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:22:49.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:49.221+0000 7fd3faa00700 1 --2- 172.21.15.62:0/3348999628 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd3e006ecc0 0x7fd3e0071170 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fd3ec005fd0 tx=0x7fd3ec005dc0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:22:49.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:49.225+0000 7fd3f3400700 1 -- 172.21.15.62:0/3348999628 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fd3e405cc80 con 0x7fd3fc073a00 2024-04-20T22:22:49.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:49.470+0000 7fd3fbe00700 1 -- 172.21.15.62:0/3348999628 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fd3fc199050 con 0x7fd3e006ecc0 2024-04-20T22:22:49.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:49.498+0000 7fd3f3400700 1 -- 172.21.15.62:0/3348999628 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fd3fc199050 con 0x7fd3e006ecc0 2024-04-20T22:22:49.498 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T22:22:49.498 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (48m) 9m ago 62m 14.6M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T22:22:49.498 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (62m) 9m ago 62m 11.6M - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T22:22:49.499 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (61m) 9m ago 61m 11.2M - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T22:22:49.499 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (44m) 9m ago 62m 6660k - 19.0.0-3133-ga8c84f59 76bc6b91b726 2825d5ba11c6 2024-04-20T22:22:49.499 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (44m) 9m ago 61m 6665k - 19.0.0-3133-ga8c84f59 76bc6b91b726 20f746fa1bba 2024-04-20T22:22:49.499 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (48m) 9m ago 62m 75.5M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T22:22:49.499 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (40m) 9m ago 55m 22.9M - 19.0.0-3133-ga8c84f59 76bc6b91b726 de80bc90a683 2024-04-20T22:22:49.499 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (40m) 9m ago 55m 23.0M - 19.0.0-3133-ga8c84f59 76bc6b91b726 cd793b7ee98e 2024-04-20T22:22:49.499 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (24m) 9m ago 55m 18.5M - 19.0.0-3133-ga8c84f59 76bc6b91b726 0e7b0bc4d1ab 2024-04-20T22:22:49.499 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (9m) 9m ago 55m 11.4M - 19.0.0-3133-ga8c84f59 76bc6b91b726 0af732ac9e06 2024-04-20T22:22:49.499 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (51m) 9m ago 64m 517M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T22:22:49.499 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (50m) 9m ago 61m 460M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T22:22:49.499 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (46m) 9m ago 64m 86.3M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 019eb1479485 2024-04-20T22:22:49.499 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (45m) 9m ago 61m 58.0M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 55aac560aab6 2024-04-20T22:22:49.499 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (49m) 9m ago 62m 8766k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T22:22:49.499 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (49m) 9m ago 61m 8711k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T22:22:49.500 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (44m) 9m ago 60m 91.3M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 ee2bb901f26e 2024-04-20T22:22:49.500 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (43m) 9m ago 59m 94.9M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 9075152d4162 2024-04-20T22:22:49.500 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (43m) 9m ago 58m 78.5M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 2d4f3e38dc0e 2024-04-20T22:22:49.500 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (42m) 9m ago 58m 94.1M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 8f97b17ae0eb 2024-04-20T22:22:49.500 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (41m) 9m ago 57m 96.1M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 653d8079967d 2024-04-20T22:22:49.500 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (41m) 9m ago 57m 90.9M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 596dded14c12 2024-04-20T22:22:49.500 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (49m) 9m ago 61m 90.4M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T22:22:49.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:49.501+0000 7fd3fbe00700 1 -- 172.21.15.62:0/3348999628 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd3e006ecc0 msgr2=0x7fd3e0071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:22:49.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:49.501+0000 7fd3fbe00700 1 --2- 172.21.15.62:0/3348999628 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd3e006ecc0 0x7fd3e0071170 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fd3ec005fd0 tx=0x7fd3ec005dc0 comp rx=0 tx=0).stop 2024-04-20T22:22:49.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:49.501+0000 7fd3fbe00700 1 -- 172.21.15.62:0/3348999628 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3fc073a00 msgr2=0x7fd3fc19e130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:22:49.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:49.501+0000 7fd3fbe00700 1 --2- 172.21.15.62:0/3348999628 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3fc073a00 0x7fd3fc19e130 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7fd3e400b5c0 tx=0x7fd3e400be00 comp rx=0 tx=0).stop 2024-04-20T22:22:49.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:49.501+0000 7fd3fbe00700 1 -- 172.21.15.62:0/3348999628 shutdown_connections 2024-04-20T22:22:49.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:49.501+0000 7fd3fbe00700 1 --2- 172.21.15.62:0/3348999628 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd3e006ecc0 0x7fd3e0071170 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:22:49.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:49.501+0000 7fd3fbe00700 1 --2- 172.21.15.62:0/3348999628 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3fc0730f0 0x7fd3fc19dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:22:49.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:49.501+0000 7fd3fbe00700 1 --2- 172.21.15.62:0/3348999628 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3fc073a00 0x7fd3fc19e130 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:22:49.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:49.501+0000 7fd3fbe00700 1 -- 172.21.15.62:0/3348999628 >> 172.21.15.62:0/3348999628 conn(0x7fd3fc0fbb10 msgr2=0x7fd3fc106b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:22:49.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:49.501+0000 7fd3fbe00700 1 -- 172.21.15.62:0/3348999628 shutdown_connections 2024-04-20T22:22:49.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:49.501+0000 7fd3fbe00700 1 -- 172.21.15.62:0/3348999628 wait complete. 2024-04-20T22:22:49.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:49.670+0000 7f5fb8e00700 1 -- 172.21.15.62:0/4054491019 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5fb4103040 msgr2=0x7f5fb4107010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:22:49.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:49.670+0000 7f5fb8e00700 1 --2- 172.21.15.62:0/4054491019 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5fb4103040 0x7f5fb4107010 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f5f9c009910 tx=0x7f5f9c009c20 comp rx=0 tx=0).stop 2024-04-20T22:22:49.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:49.670+0000 7f5fb8e00700 1 -- 172.21.15.62:0/4054491019 shutdown_connections 2024-04-20T22:22:49.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:49.670+0000 7f5fb8e00700 1 --2- 172.21.15.62:0/4054491019 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5fb4103040 0x7f5fb4107010 unknown :-1 s=CLOSED pgs=1217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:22:49.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:49.670+0000 7f5fb8e00700 1 --2- 172.21.15.62:0/4054491019 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5fb41026a0 0x7f5fb4102a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:22:49.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:49.670+0000 7f5fb8e00700 1 -- 172.21.15.62:0/4054491019 >> 172.21.15.62:0/4054491019 conn(0x7f5fb40fdf40 msgr2=0x7f5fb4100350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:22:49.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:49.671+0000 7f5fb8e00700 1 -- 172.21.15.62:0/4054491019 shutdown_connections 2024-04-20T22:22:49.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:49.671+0000 7f5fb8e00700 1 -- 172.21.15.62:0/4054491019 wait complete. 2024-04-20T22:22:49.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:49.672+0000 7f5fb8e00700 1 Processor -- start 2024-04-20T22:22:49.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:49.672+0000 7f5fb8e00700 1 -- start start 2024-04-20T22:22:49.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:49.672+0000 7f5fb8e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5fb41026a0 0x7f5fb4198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:22:49.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:49.672+0000 7f5fb8e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5fb4103040 0x7f5fb4198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:22:49.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:49.672+0000 7f5fb8e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb4198bc0 con 0x7f5fb41026a0 2024-04-20T22:22:49.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:49.672+0000 7f5fb8e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb4198d00 con 0x7f5fb4103040 2024-04-20T22:22:49.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:49.672+0000 7f5fb1e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5fb41026a0 0x7f5fb4198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:22:49.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:49.673+0000 7f5fb1e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5fb41026a0 0x7f5fb4198030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42056/0 (socket says 172.21.15.62:42056) 2024-04-20T22:22:49.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:49.673+0000 7f5fb1e00700 1 -- 172.21.15.62:0/1558949219 learned_addr learned my addr 172.21.15.62:0/1558949219 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:22:49.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:49.673+0000 7f5fabe00700 1 --2- 172.21.15.62:0/1558949219 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5fb4103040 0x7f5fb4198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:22:49.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:49.673+0000 7f5fb1e00700 1 -- 172.21.15.62:0/1558949219 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5fb4103040 msgr2=0x7f5fb4198570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:22:49.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:49.673+0000 7f5fb1e00700 1 --2- 172.21.15.62:0/1558949219 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5fb4103040 0x7f5fb4198570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:22:49.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:49.673+0000 7f5fb1e00700 1 -- 172.21.15.62:0/1558949219 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f9c0095f0 con 0x7f5fb41026a0 2024-04-20T22:22:49.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:49.673+0000 7f5fb1e00700 1 --2- 172.21.15.62:0/1558949219 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5fb41026a0 0x7f5fb4198030 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f5fa400c930 tx=0x7f5fa400ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:22:49.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:49.674+0000 7f5faaa00700 1 -- 172.21.15.62:0/1558949219 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5fa4007ad0 con 0x7f5fb41026a0 2024-04-20T22:22:49.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:49.674+0000 7f5faaa00700 1 -- 172.21.15.62:0/1558949219 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5fa4007c30 con 0x7f5fb41026a0 2024-04-20T22:22:49.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:49.674+0000 7f5faaa00700 1 -- 172.21.15.62:0/1558949219 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5fa4018610 con 0x7f5fb41026a0 2024-04-20T22:22:49.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:49.674+0000 7f5fb8e00700 1 -- 172.21.15.62:0/1558949219 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fb419cb50 con 0x7f5fb41026a0 2024-04-20T22:22:49.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:49.674+0000 7f5fb8e00700 1 -- 172.21.15.62:0/1558949219 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fb419d0a0 con 0x7f5fb41026a0 2024-04-20T22:22:49.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:49.675+0000 7f5fb8e00700 1 -- 172.21.15.62:0/1558949219 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fb410a920 con 0x7f5fb41026a0 2024-04-20T22:22:49.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:49.676+0000 7f5faaa00700 1 -- 172.21.15.62:0/1558949219 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f5fa400f450 con 0x7f5fb41026a0 2024-04-20T22:22:49.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:49.677+0000 7f5faaa00700 1 --2- 172.21.15.62:0/1558949219 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5f9806ecc0 0x7f5f98071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:22:49.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:49.677+0000 7f5faaa00700 1 -- 172.21.15.62:0/1558949219 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f5fa408eff0 con 0x7f5fb41026a0 2024-04-20T22:22:49.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:49.677+0000 7f5fabe00700 1 --2- 172.21.15.62:0/1558949219 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5f9806ecc0 0x7f5f98071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:22:49.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:49.678+0000 7f5fabe00700 1 --2- 172.21.15.62:0/1558949219 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5f9806ecc0 0x7f5f98071170 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f5fb4199650 tx=0x7f5f9c01a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:22:49.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:49.682+0000 7f5faaa00700 1 -- 172.21.15.62:0/1558949219 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f5fa4057de0 con 0x7f5fb41026a0 2024-04-20T22:22:49.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:22:49 smithi062 ceph-mon[76322]: from='client.36496 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:22:49.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:22:49 smithi062 ceph-mon[76322]: from='client.36500 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:22:49.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:22:49 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:22:49.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:22:49 smithi189 ceph-mon[57022]: from='client.36496 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:22:49.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:22:49 smithi189 ceph-mon[57022]: from='client.36500 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:22:49.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:22:49 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:22:50.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.011+0000 7f5fb8e00700 1 -- 172.21.15.62:0/1558949219 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f5fb410a300 con 0x7f5fb41026a0 2024-04-20T22:22:50.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.012+0000 7f5faaa00700 1 -- 172.21.15.62:0/1558949219 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f5fa405b4a0 con 0x7f5fb41026a0 2024-04-20T22:22:50.012 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:22:50.012 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T22:22:50.013 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:22:50.013 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:22:50.013 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T22:22:50.013 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:22:50.013 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:22:50.013 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T22:22:50.013 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 6 2024-04-20T22:22:50.013 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:22:50.013 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T22:22:50.013 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 4 2024-04-20T22:22:50.013 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:22:50.013 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T22:22:50.013 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 14 2024-04-20T22:22:50.013 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T22:22:50.013 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:22:50.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.015+0000 7f5fb8e00700 1 -- 172.21.15.62:0/1558949219 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5f9806ecc0 msgr2=0x7f5f98071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:22:50.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.015+0000 7f5fb8e00700 1 --2- 172.21.15.62:0/1558949219 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5f9806ecc0 0x7f5f98071170 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f5fb4199650 tx=0x7f5f9c01a040 comp rx=0 tx=0).stop 2024-04-20T22:22:50.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.015+0000 7f5fb8e00700 1 -- 172.21.15.62:0/1558949219 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5fb41026a0 msgr2=0x7f5fb4198030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:22:50.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.015+0000 7f5fb8e00700 1 --2- 172.21.15.62:0/1558949219 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5fb41026a0 0x7f5fb4198030 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f5fa400c930 tx=0x7f5fa400ccf0 comp rx=0 tx=0).stop 2024-04-20T22:22:50.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.015+0000 7f5fb8e00700 1 -- 172.21.15.62:0/1558949219 shutdown_connections 2024-04-20T22:22:50.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.015+0000 7f5fb8e00700 1 --2- 172.21.15.62:0/1558949219 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5f9806ecc0 0x7f5f98071170 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:22:50.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.015+0000 7f5fb8e00700 1 --2- 172.21.15.62:0/1558949219 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5fb41026a0 0x7f5fb4198030 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:22:50.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.015+0000 7f5fb8e00700 1 --2- 172.21.15.62:0/1558949219 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5fb4103040 0x7f5fb4198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:22:50.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.015+0000 7f5fb8e00700 1 -- 172.21.15.62:0/1558949219 >> 172.21.15.62:0/1558949219 conn(0x7f5fb40fdf40 msgr2=0x7f5fb4106880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:22:50.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.015+0000 7f5fb8e00700 1 -- 172.21.15.62:0/1558949219 shutdown_connections 2024-04-20T22:22:50.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.016+0000 7f5fb8e00700 1 -- 172.21.15.62:0/1558949219 wait complete. 2024-04-20T22:22:50.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.164+0000 7f9260c00700 1 -- 172.21.15.62:0/3681641267 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f925c100e80 msgr2=0x7f925c104e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:22:50.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.164+0000 7f9260c00700 1 --2- 172.21.15.62:0/3681641267 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f925c100e80 0x7f925c104e50 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f9248009910 tx=0x7f9248009c20 comp rx=0 tx=0).stop 2024-04-20T22:22:50.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.164+0000 7f9260c00700 1 -- 172.21.15.62:0/3681641267 shutdown_connections 2024-04-20T22:22:50.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.164+0000 7f9260c00700 1 --2- 172.21.15.62:0/3681641267 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f925c100e80 0x7f925c104e50 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:22:50.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.164+0000 7f9260c00700 1 --2- 172.21.15.62:0/3681641267 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f925c1004e0 0x7f925c1008b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:22:50.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.164+0000 7f9260c00700 1 -- 172.21.15.62:0/3681641267 >> 172.21.15.62:0/3681641267 conn(0x7f925c0fbda0 msgr2=0x7f925c0fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:22:50.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.165+0000 7f9260c00700 1 -- 172.21.15.62:0/3681641267 shutdown_connections 2024-04-20T22:22:50.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.165+0000 7f9260c00700 1 -- 172.21.15.62:0/3681641267 wait complete. 2024-04-20T22:22:50.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.166+0000 7f9260c00700 1 Processor -- start 2024-04-20T22:22:50.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.166+0000 7f9260c00700 1 -- start start 2024-04-20T22:22:50.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.166+0000 7f9260c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f925c1004e0 0x7f925c073050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:22:50.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.166+0000 7f9260c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f925c100e80 0x7f925c073590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:22:50.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.166+0000 7f9260c00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f925c073ad0 con 0x7f925c1004e0 2024-04-20T22:22:50.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.166+0000 7f9260c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f925c073c10 con 0x7f925c100e80 2024-04-20T22:22:50.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.167+0000 7f9259e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f925c1004e0 0x7f925c073050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:22:50.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.167+0000 7f9259e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f925c1004e0 0x7f925c073050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42074/0 (socket says 172.21.15.62:42074) 2024-04-20T22:22:50.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.167+0000 7f9259e00700 1 -- 172.21.15.62:0/1531634258 learned_addr learned my addr 172.21.15.62:0/1531634258 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:22:50.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.167+0000 7f9259400700 1 --2- 172.21.15.62:0/1531634258 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f925c100e80 0x7f925c073590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:22:50.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.167+0000 7f9259e00700 1 -- 172.21.15.62:0/1531634258 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f925c100e80 msgr2=0x7f925c073590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:22:50.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.167+0000 7f9259e00700 1 --2- 172.21.15.62:0/1531634258 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f925c100e80 0x7f925c073590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:22:50.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.167+0000 7f9259e00700 1 -- 172.21.15.62:0/1531634258 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92480095f0 con 0x7f925c1004e0 2024-04-20T22:22:50.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.167+0000 7f9259400700 1 --2- 172.21.15.62:0/1531634258 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f925c100e80 0x7f925c073590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T22:22:50.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.167+0000 7f9259e00700 1 --2- 172.21.15.62:0/1531634258 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f925c1004e0 0x7f925c073050 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f924c00b4b0 tx=0x7f924c00b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:22:50.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.168+0000 7f9252a00700 1 -- 172.21.15.62:0/1531634258 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f924c011800 con 0x7f925c1004e0 2024-04-20T22:22:50.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.168+0000 7f9252a00700 1 -- 172.21.15.62:0/1531634258 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f924c011e20 con 0x7f925c1004e0 2024-04-20T22:22:50.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.168+0000 7f9252a00700 1 -- 172.21.15.62:0/1531634258 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f924c00f2e0 con 0x7f925c1004e0 2024-04-20T22:22:50.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.168+0000 7f9260c00700 1 -- 172.21.15.62:0/1531634258 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f925c1a21c0 con 0x7f925c1004e0 2024-04-20T22:22:50.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.168+0000 7f9260c00700 1 -- 172.21.15.62:0/1531634258 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f925c1a25d0 con 0x7f925c1004e0 2024-04-20T22:22:50.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.169+0000 7f9260c00700 1 -- 172.21.15.62:0/1531634258 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f925c108780 con 0x7f925c1004e0 2024-04-20T22:22:50.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.170+0000 7f9252a00700 1 -- 172.21.15.62:0/1531634258 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f924c010380 con 0x7f925c1004e0 2024-04-20T22:22:50.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.171+0000 7f9252a00700 1 --2- 172.21.15.62:0/1531634258 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f924006ecc0 0x7f9240071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:22:50.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.171+0000 7f9252a00700 1 -- 172.21.15.62:0/1531634258 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f924c08ea30 con 0x7f925c1004e0 2024-04-20T22:22:50.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.171+0000 7f9259400700 1 --2- 172.21.15.62:0/1531634258 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f924006ecc0 0x7f9240071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:22:50.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.172+0000 7f9259400700 1 --2- 172.21.15.62:0/1531634258 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f924006ecc0 0x7f9240071170 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f9248006010 tx=0x7f9248005960 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:22:50.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.177+0000 7f9252a00700 1 -- 172.21.15.62:0/1531634258 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f924c057870 con 0x7f925c1004e0 2024-04-20T22:22:50.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.453+0000 7f9260c00700 1 -- 172.21.15.62:0/1531634258 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f925c108160 con 0x7f925c1004e0 2024-04-20T22:22:50.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.454+0000 7f9252a00700 1 -- 172.21.15.62:0/1531634258 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 29 v29) v1 ==== 76+0+1887 (secure 0 0 0) 0x7f924c05af30 con 0x7f925c1004e0 2024-04-20T22:22:50.455 INFO:teuthology.orchestra.run.smithi062.stdout:e29 2024-04-20T22:22:50.455 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T22:22:50.455 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:22:50.455 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T22:22:50.455 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:22:50.455 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T22:22:50.455 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T22:22:50.455 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 20 2024-04-20T22:22:50.455 INFO:teuthology.orchestra.run.smithi062.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-20T22:22:50.455 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T22:22:50.455 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T21:42:01.559061+0000 2024-04-20T22:22:50.455 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T22:22:50.456 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T22:22:50.456 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T22:22:50.456 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T22:22:50.456 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T22:22:50.456 INFO:teuthology.orchestra.run.smithi062.stdout:max_xattr_size 65536 2024-04-20T22:22:50.456 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T22:22:50.456 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T22:22:50.456 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 81 2024-04-20T22:22:50.456 INFO:teuthology.orchestra.run.smithi062.stdout:snap_rstat 1 2024-04-20T22:22:50.456 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:22:50.456 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 2 2024-04-20T22:22:50.456 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1 2024-04-20T22:22:50.456 INFO:teuthology.orchestra.run.smithi062.stdout:up {} 2024-04-20T22:22:50.456 INFO:teuthology.orchestra.run.smithi062.stdout:failed 0,1 2024-04-20T22:22:50.456 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T22:22:50.456 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T22:22:50.457 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T22:22:50.457 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T22:22:50.457 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data enabled 2024-04-20T22:22:50.457 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T22:22:50.457 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T22:22:50.457 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 1 2024-04-20T22:22:50.457 INFO:teuthology.orchestra.run.smithi062.stdout:qdb_cluster leader: 0 members: 2024-04-20T22:22:50.457 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:22:50.457 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:22:50.457 INFO:teuthology.orchestra.run.smithi062.stdout:Standby daemons: 2024-04-20T22:22:50.457 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:22:50.457 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ryqblb{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6826/4113916205,v1:172.21.15.62:6827/4113916205] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:22:50.457 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ozizwi{-1:34476} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6828/3801975165,v1:172.21.15.62:6829/3801975165] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:22:50.457 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.dvimer{-1:44727} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6824/3576699232,v1:172.21.15.189:6825/3576699232] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:22:50.457 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.xbgspw{-1:45075} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6826/212905792,v1:172.21.15.189:6827/212905792] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:22:50.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.456+0000 7f9260c00700 1 -- 172.21.15.62:0/1531634258 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f924006ecc0 msgr2=0x7f9240071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:22:50.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.456+0000 7f9260c00700 1 --2- 172.21.15.62:0/1531634258 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f924006ecc0 0x7f9240071170 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f9248006010 tx=0x7f9248005960 comp rx=0 tx=0).stop 2024-04-20T22:22:50.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.456+0000 7f9260c00700 1 -- 172.21.15.62:0/1531634258 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f925c1004e0 msgr2=0x7f925c073050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:22:50.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.456+0000 7f9260c00700 1 --2- 172.21.15.62:0/1531634258 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f925c1004e0 0x7f925c073050 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f924c00b4b0 tx=0x7f924c00b870 comp rx=0 tx=0).stop 2024-04-20T22:22:50.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.456+0000 7f9260c00700 1 -- 172.21.15.62:0/1531634258 shutdown_connections 2024-04-20T22:22:50.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.456+0000 7f9260c00700 1 --2- 172.21.15.62:0/1531634258 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f924006ecc0 0x7f9240071170 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:22:50.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.456+0000 7f9260c00700 1 --2- 172.21.15.62:0/1531634258 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f925c1004e0 0x7f925c073050 unknown :-1 s=CLOSED pgs=1220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:22:50.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.456+0000 7f9260c00700 1 --2- 172.21.15.62:0/1531634258 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f925c100e80 0x7f925c073590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:22:50.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.456+0000 7f9260c00700 1 -- 172.21.15.62:0/1531634258 >> 172.21.15.62:0/1531634258 conn(0x7f925c0fbda0 msgr2=0x7f925c0fd520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:22:50.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.457+0000 7f9260c00700 1 -- 172.21.15.62:0/1531634258 shutdown_connections 2024-04-20T22:22:50.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.457+0000 7f9260c00700 1 -- 172.21.15.62:0/1531634258 wait complete. 2024-04-20T22:22:50.459 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 29 2024-04-20T22:22:50.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.615+0000 7ff424c00700 1 -- 172.21.15.62:0/1286891743 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff420101420 msgr2=0x7ff4201017f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:22:50.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.615+0000 7ff424c00700 1 --2- 172.21.15.62:0/1286891743 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff420101420 0x7ff4201017f0 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7ff410009910 tx=0x7ff410009c20 comp rx=0 tx=0).stop 2024-04-20T22:22:50.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.616+0000 7ff424c00700 1 -- 172.21.15.62:0/1286891743 shutdown_connections 2024-04-20T22:22:50.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.616+0000 7ff424c00700 1 --2- 172.21.15.62:0/1286891743 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff420101d30 0x7ff42010a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:22:50.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.616+0000 7ff424c00700 1 --2- 172.21.15.62:0/1286891743 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff420101420 0x7ff4201017f0 unknown :-1 s=CLOSED pgs=1221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:22:50.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.616+0000 7ff424c00700 1 -- 172.21.15.62:0/1286891743 >> 172.21.15.62:0/1286891743 conn(0x7ff4200fac80 msgr2=0x7ff4200fd090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:22:50.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.616+0000 7ff424c00700 1 -- 172.21.15.62:0/1286891743 shutdown_connections 2024-04-20T22:22:50.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.616+0000 7ff424c00700 1 -- 172.21.15.62:0/1286891743 wait complete. 2024-04-20T22:22:50.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.617+0000 7ff424c00700 1 Processor -- start 2024-04-20T22:22:50.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.617+0000 7ff424c00700 1 -- start start 2024-04-20T22:22:50.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.618+0000 7ff424c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff420101420 0x7ff420198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:22:50.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.618+0000 7ff424c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff420101d30 0x7ff4201985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:22:50.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.618+0000 7ff424c00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff420198c90 con 0x7ff420101420 2024-04-20T22:22:50.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.618+0000 7ff424c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff42019ca20 con 0x7ff420101d30 2024-04-20T22:22:50.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.618+0000 7ff41de00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff420101420 0x7ff420198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:22:50.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.618+0000 7ff41de00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff420101420 0x7ff420198070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42084/0 (socket says 172.21.15.62:42084) 2024-04-20T22:22:50.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.618+0000 7ff41de00700 1 -- 172.21.15.62:0/1664584804 learned_addr learned my addr 172.21.15.62:0/1664584804 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:22:50.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.618+0000 7ff41d400700 1 --2- 172.21.15.62:0/1664584804 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff420101d30 0x7ff4201985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:22:50.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.618+0000 7ff41de00700 1 -- 172.21.15.62:0/1664584804 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff420101d30 msgr2=0x7ff4201985b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:22:50.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.618+0000 7ff41de00700 1 --2- 172.21.15.62:0/1664584804 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff420101d30 0x7ff4201985b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:22:50.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.618+0000 7ff41de00700 1 -- 172.21.15.62:0/1664584804 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4100095f0 con 0x7ff420101420 2024-04-20T22:22:50.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.618+0000 7ff41d400700 1 --2- 172.21.15.62:0/1664584804 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff420101d30 0x7ff4201985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T22:22:50.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.619+0000 7ff41de00700 1 --2- 172.21.15.62:0/1664584804 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff420101420 0x7ff420198070 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7ff410006010 tx=0x7ff41000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:22:50.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.619+0000 7ff416a00700 1 -- 172.21.15.62:0/1664584804 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff41001d070 con 0x7ff420101420 2024-04-20T22:22:50.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.619+0000 7ff416a00700 1 -- 172.21.15.62:0/1664584804 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff41000f460 con 0x7ff420101420 2024-04-20T22:22:50.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.619+0000 7ff424c00700 1 -- 172.21.15.62:0/1664584804 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff42019cca0 con 0x7ff420101420 2024-04-20T22:22:50.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.619+0000 7ff416a00700 1 -- 172.21.15.62:0/1664584804 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff410017630 con 0x7ff420101420 2024-04-20T22:22:50.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.619+0000 7ff424c00700 1 -- 172.21.15.62:0/1664584804 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff42019d190 con 0x7ff420101420 2024-04-20T22:22:50.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.620+0000 7ff424c00700 1 -- 172.21.15.62:0/1664584804 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff420109cd0 con 0x7ff420101420 2024-04-20T22:22:50.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.621+0000 7ff416a00700 1 -- 172.21.15.62:0/1664584804 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff41000f5d0 con 0x7ff420101420 2024-04-20T22:22:50.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.622+0000 7ff416a00700 1 --2- 172.21.15.62:0/1664584804 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff40406ecc0 0x7ff404071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:22:50.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.623+0000 7ff416a00700 1 -- 172.21.15.62:0/1664584804 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7ff4100903a0 con 0x7ff420101420 2024-04-20T22:22:50.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.623+0000 7ff41d400700 1 --2- 172.21.15.62:0/1664584804 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff40406ecc0 0x7ff404071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:22:50.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.624+0000 7ff41d400700 1 --2- 172.21.15.62:0/1664584804 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff40406ecc0 0x7ff404071170 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7ff40c005fd0 tx=0x7ff40c005ee0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:22:50.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.627+0000 7ff416a00700 1 -- 172.21.15.62:0/1664584804 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7ff4100590b0 con 0x7ff420101420 2024-04-20T22:22:50.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:22:50 smithi062 ceph-mon[76322]: pgmap v1429: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:22:50.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:22:50 smithi062 ceph-mon[76322]: from='client.36504 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:22:50.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:22:50 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/1558949219' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:22:50.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:22:50 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/1531634258' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:22:50.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.881+0000 7ff424c00700 1 -- 172.21.15.62:0/1664584804 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff42002cca0 con 0x7ff40406ecc0 2024-04-20T22:22:50.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.883+0000 7ff416a00700 1 -- 172.21.15.62:0/1664584804 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7ff42002cca0 con 0x7ff40406ecc0 2024-04-20T22:22:50.883 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:22:50.883 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T22:22:50.883 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T22:22:50.884 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-20T22:22:50.884 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T22:22:50.884 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr", 2024-04-20T22:22:50.884 INFO:teuthology.orchestra.run.smithi062.stdout: "osd", 2024-04-20T22:22:50.884 INFO:teuthology.orchestra.run.smithi062.stdout: "mds", 2024-04-20T22:22:50.884 INFO:teuthology.orchestra.run.smithi062.stdout: "crash", 2024-04-20T22:22:50.884 INFO:teuthology.orchestra.run.smithi062.stdout: "mon" 2024-04-20T22:22:50.884 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T22:22:50.884 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "16/23 daemons upgraded", 2024-04-20T22:22:50.884 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading mds daemons", 2024-04-20T22:22:50.884 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T22:22:50.884 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:22:50.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.886+0000 7ff424c00700 1 -- 172.21.15.62:0/1664584804 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff40406ecc0 msgr2=0x7ff404071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:22:50.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.886+0000 7ff424c00700 1 --2- 172.21.15.62:0/1664584804 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff40406ecc0 0x7ff404071170 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7ff40c005fd0 tx=0x7ff40c005ee0 comp rx=0 tx=0).stop 2024-04-20T22:22:50.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.886+0000 7ff424c00700 1 -- 172.21.15.62:0/1664584804 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff420101420 msgr2=0x7ff420198070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:22:50.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.886+0000 7ff424c00700 1 --2- 172.21.15.62:0/1664584804 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff420101420 0x7ff420198070 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7ff410006010 tx=0x7ff41000be20 comp rx=0 tx=0).stop 2024-04-20T22:22:50.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.886+0000 7ff424c00700 1 -- 172.21.15.62:0/1664584804 shutdown_connections 2024-04-20T22:22:50.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.886+0000 7ff424c00700 1 --2- 172.21.15.62:0/1664584804 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff40406ecc0 0x7ff404071170 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:22:50.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.886+0000 7ff424c00700 1 --2- 172.21.15.62:0/1664584804 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff420101420 0x7ff420198070 unknown :-1 s=CLOSED pgs=1222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:22:50.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.886+0000 7ff424c00700 1 --2- 172.21.15.62:0/1664584804 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff420101d30 0x7ff4201985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:22:50.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.886+0000 7ff424c00700 1 -- 172.21.15.62:0/1664584804 >> 172.21.15.62:0/1664584804 conn(0x7ff4200fac80 msgr2=0x7ff420106e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:22:50.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.886+0000 7ff424c00700 1 -- 172.21.15.62:0/1664584804 shutdown_connections 2024-04-20T22:22:50.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:50.886+0000 7ff424c00700 1 -- 172.21.15.62:0/1664584804 wait complete. 2024-04-20T22:22:50.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:22:50 smithi189 ceph-mon[57022]: pgmap v1429: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:22:50.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:22:50 smithi189 ceph-mon[57022]: from='client.36504 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:22:50.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:22:50 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/1558949219' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:22:50.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:22:50 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/1531634258' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:22:51.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:51.046+0000 7f97eb400700 1 -- 172.21.15.62:0/1909965079 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f97e4100e80 msgr2=0x7f97e4104e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:22:51.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:51.046+0000 7f97eb400700 1 --2- 172.21.15.62:0/1909965079 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f97e4100e80 0x7f97e4104e50 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f97d0009910 tx=0x7f97d0009c20 comp rx=0 tx=0).stop 2024-04-20T22:22:51.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:51.046+0000 7f97eb400700 1 -- 172.21.15.62:0/1909965079 shutdown_connections 2024-04-20T22:22:51.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:51.046+0000 7f97eb400700 1 --2- 172.21.15.62:0/1909965079 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f97e4100e80 0x7f97e4104e50 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:22:51.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:51.046+0000 7f97eb400700 1 --2- 172.21.15.62:0/1909965079 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97e41004e0 0x7f97e41008b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:22:51.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:51.046+0000 7f97eb400700 1 -- 172.21.15.62:0/1909965079 >> 172.21.15.62:0/1909965079 conn(0x7f97e40fbda0 msgr2=0x7f97e40fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:22:51.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:51.046+0000 7f97eb400700 1 -- 172.21.15.62:0/1909965079 shutdown_connections 2024-04-20T22:22:51.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:51.046+0000 7f97eb400700 1 -- 172.21.15.62:0/1909965079 wait complete. 2024-04-20T22:22:51.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:51.047+0000 7f97eb400700 1 Processor -- start 2024-04-20T22:22:51.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:51.047+0000 7f97eb400700 1 -- start start 2024-04-20T22:22:51.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:51.048+0000 7f97eb400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97e41004e0 0x7f97e4198130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:22:51.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:51.048+0000 7f97eb400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f97e4100e80 0x7f97e4198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:22:51.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:51.048+0000 7f97eb400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97e4198d50 con 0x7f97e4100e80 2024-04-20T22:22:51.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:51.048+0000 7f97eb400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97e419cae0 con 0x7f97e41004e0 2024-04-20T22:22:51.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:51.048+0000 7f97e3e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f97e4100e80 0x7f97e4198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:22:51.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:51.048+0000 7f97e3e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f97e4100e80 0x7f97e4198670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42102/0 (socket says 172.21.15.62:42102) 2024-04-20T22:22:51.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:51.048+0000 7f97e3e00700 1 -- 172.21.15.62:0/3360763082 learned_addr learned my addr 172.21.15.62:0/3360763082 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:22:51.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:51.049+0000 7f97e8a00700 1 --2- 172.21.15.62:0/3360763082 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97e41004e0 0x7f97e4198130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:22:51.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:51.049+0000 7f97e3e00700 1 -- 172.21.15.62:0/3360763082 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97e41004e0 msgr2=0x7f97e4198130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:22:51.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:51.049+0000 7f97e3e00700 1 --2- 172.21.15.62:0/3360763082 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97e41004e0 0x7f97e4198130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:22:51.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:51.049+0000 7f97e3e00700 1 -- 172.21.15.62:0/3360763082 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97d00095f0 con 0x7f97e4100e80 2024-04-20T22:22:51.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:51.049+0000 7f97e3e00700 1 --2- 172.21.15.62:0/3360763082 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f97e4100e80 0x7f97e4198670 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f97d000bd70 tx=0x7f97d000be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:22:51.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:51.049+0000 7f97e1600700 1 -- 172.21.15.62:0/3360763082 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f97d001d070 con 0x7f97e4100e80 2024-04-20T22:22:51.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:51.049+0000 7f97e1600700 1 -- 172.21.15.62:0/3360763082 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f97d0005dc0 con 0x7f97e4100e80 2024-04-20T22:22:51.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:51.049+0000 7f97eb400700 1 -- 172.21.15.62:0/3360763082 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97e419cd60 con 0x7f97e4100e80 2024-04-20T22:22:51.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:51.049+0000 7f97e1600700 1 -- 172.21.15.62:0/3360763082 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f97d0017400 con 0x7f97e4100e80 2024-04-20T22:22:51.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:51.049+0000 7f97eb400700 1 -- 172.21.15.62:0/3360763082 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97e419d250 con 0x7f97e4100e80 2024-04-20T22:22:51.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:51.051+0000 7f97eb400700 1 -- 172.21.15.62:0/3360763082 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97e41045d0 con 0x7f97e4100e80 2024-04-20T22:22:51.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:51.052+0000 7f97e1600700 1 -- 172.21.15.62:0/3360763082 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f97d000f7f0 con 0x7f97e4100e80 2024-04-20T22:22:51.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:51.053+0000 7f97e1600700 1 --2- 172.21.15.62:0/3360763082 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f97cc06ecc0 0x7f97cc071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:22:51.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:51.053+0000 7f97e1600700 1 -- 172.21.15.62:0/3360763082 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f97d0090410 con 0x7f97e4100e80 2024-04-20T22:22:51.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:51.053+0000 7f97e8a00700 1 --2- 172.21.15.62:0/3360763082 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f97cc06ecc0 0x7f97cc071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:22:51.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:51.054+0000 7f97e8a00700 1 --2- 172.21.15.62:0/3360763082 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f97cc06ecc0 0x7f97cc071170 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f97d8005950 tx=0x7f97d80058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:22:51.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:51.057+0000 7f97e1600700 1 -- 172.21.15.62:0/3360763082 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f97d0059120 con 0x7f97e4100e80 2024-04-20T22:22:51.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:51.380+0000 7f97eb400700 1 -- 172.21.15.62:0/3360763082 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f97e404f2a0 con 0x7f97e4100e80 2024-04-20T22:22:51.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:51.380+0000 7f97e1600700 1 -- 172.21.15.62:0/3360763082 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7f97d005c7e0 con 0x7f97e4100e80 2024-04-20T22:22:51.381 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-20T22:22:51.381 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-20T22:22:51.381 INFO:teuthology.orchestra.run.smithi062.stdout: host smithi189 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi189/config 2024-04-20T22:22:51.381 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-20T22:22:51.381 INFO:teuthology.orchestra.run.smithi062.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi189 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp11670v2q:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:22:51.381 INFO:teuthology.orchestra.run.smithi062.stdout:Traceback (most recent call last): 2024-04-20T22:22:51.381 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-20T22:22:51.381 INFO:teuthology.orchestra.run.smithi062.stdout: return _run_code(code, main_globals, None, 2024-04-20T22:22:51.382 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-20T22:22:51.382 INFO:teuthology.orchestra.run.smithi062.stdout: exec(code, run_globals) 2024-04-20T22:22:51.382 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5618, in 2024-04-20T22:22:51.382 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5606, in main 2024-04-20T22:22:51.382 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 408, in _infer_config 2024-04-20T22:22:51.382 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 323, in _infer_fsid 2024-04-20T22:22:51.382 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 436, in _infer_image 2024-04-20T22:22:51.382 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 310, in _validate_fsid 2024-04-20T22:22:51.382 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 3359, in command_ceph_volume 2024-04-20T22:22:51.382 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-20T22:22:51.382 INFO:teuthology.orchestra.run.smithi062.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi189 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp11670v2q:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:22:51.382 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-20T22:22:51.382 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is degraded 2024-04-20T22:22:51.382 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T22:22:51.383 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T22:22:51.383 INFO:teuthology.orchestra.run.smithi062.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-20T22:22:51.383 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-20T22:22:51.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:51.383+0000 7f97eb400700 1 -- 172.21.15.62:0/3360763082 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f97cc06ecc0 msgr2=0x7f97cc071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:22:51.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:51.383+0000 7f97eb400700 1 --2- 172.21.15.62:0/3360763082 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f97cc06ecc0 0x7f97cc071170 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f97d8005950 tx=0x7f97d80058e0 comp rx=0 tx=0).stop 2024-04-20T22:22:51.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:51.383+0000 7f97eb400700 1 -- 172.21.15.62:0/3360763082 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f97e4100e80 msgr2=0x7f97e4198670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:22:51.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:51.383+0000 7f97eb400700 1 --2- 172.21.15.62:0/3360763082 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f97e4100e80 0x7f97e4198670 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f97d000bd70 tx=0x7f97d000be50 comp rx=0 tx=0).stop 2024-04-20T22:22:51.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:51.384+0000 7f97eb400700 1 -- 172.21.15.62:0/3360763082 shutdown_connections 2024-04-20T22:22:51.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:51.384+0000 7f97eb400700 1 --2- 172.21.15.62:0/3360763082 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f97cc06ecc0 0x7f97cc071170 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:22:51.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:51.384+0000 7f97eb400700 1 --2- 172.21.15.62:0/3360763082 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97e41004e0 0x7f97e4198130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:22:51.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:51.384+0000 7f97eb400700 1 --2- 172.21.15.62:0/3360763082 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f97e4100e80 0x7f97e4198670 unknown :-1 s=CLOSED pgs=1224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:22:51.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:51.384+0000 7f97eb400700 1 -- 172.21.15.62:0/3360763082 >> 172.21.15.62:0/3360763082 conn(0x7f97e40fbda0 msgr2=0x7f97e40fe160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:22:51.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:51.384+0000 7f97eb400700 1 -- 172.21.15.62:0/3360763082 shutdown_connections 2024-04-20T22:22:51.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:22:51.384+0000 7f97eb400700 1 -- 172.21.15.62:0/3360763082 wait complete. 2024-04-20T22:22:51.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:22:51 smithi062 ceph-mon[76322]: from='client.36516 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:22:51.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:22:51 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/3360763082' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:22:51.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:22:51 smithi189 ceph-mon[57022]: from='client.36516 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:22:51.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:22:51 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/3360763082' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:22:52.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:22:52 smithi189 ceph-mon[57022]: pgmap v1430: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:22:53.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:22:52 smithi062 ceph-mon[76322]: pgmap v1430: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:22:54.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:22:54 smithi189 ceph-mon[57022]: pgmap v1431: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:22:55.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:22:54 smithi062 ceph-mon[76322]: pgmap v1431: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:22:56.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:22:56 smithi189 ceph-mon[57022]: pgmap v1432: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:22:57.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:22:56 smithi062 ceph-mon[76322]: pgmap v1432: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:22:58.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:22:58 smithi189 ceph-mon[57022]: pgmap v1433: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:22:59.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:22:58 smithi062 ceph-mon[76322]: pgmap v1433: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:23:00.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:23:00 smithi189 ceph-mon[57022]: pgmap v1434: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:23:01.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:23:00 smithi062 ceph-mon[76322]: pgmap v1434: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:23:02.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:23:02 smithi189 ceph-mon[57022]: pgmap v1435: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:23:03.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:23:02 smithi062 ceph-mon[76322]: pgmap v1435: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:23:04.922 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:23:04 smithi062 ceph-mon[76322]: pgmap v1436: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:23:04.922 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:23:04 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:23:04.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:23:04 smithi189 ceph-mon[57022]: pgmap v1436: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:23:04.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:23:04 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:23:06.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:23:06 smithi189 ceph-mon[57022]: pgmap v1437: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:23:07.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:23:06 smithi062 ceph-mon[76322]: pgmap v1437: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:23:08.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:23:08 smithi189 ceph-mon[57022]: pgmap v1438: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:23:09.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:23:08 smithi062 ceph-mon[76322]: pgmap v1438: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:23:10.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:23:10 smithi189 ceph-mon[57022]: pgmap v1439: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:23:11.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:23:10 smithi062 ceph-mon[76322]: pgmap v1439: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:23:12.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:23:12 smithi189 ceph-mon[57022]: pgmap v1440: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:23:13.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:23:12 smithi062 ceph-mon[76322]: pgmap v1440: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:23:14.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:23:14 smithi189 ceph-mon[57022]: pgmap v1441: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:23:15.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:23:14 smithi062 ceph-mon[76322]: pgmap v1441: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:23:16.962 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:23:16 smithi189 ceph-mon[57022]: pgmap v1442: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:23:17.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:23:16 smithi062 ceph-mon[76322]: pgmap v1442: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:23:18.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:23:18 smithi189 ceph-mon[57022]: pgmap v1443: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:23:19.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:23:18 smithi062 ceph-mon[76322]: pgmap v1443: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:23:19.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:23:19 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:23:20.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:23:19 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:23:20.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:23:20 smithi189 ceph-mon[57022]: pgmap v1444: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:23:21.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:23:20 smithi062 ceph-mon[76322]: pgmap v1444: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:23:21.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:21.564+0000 7f771fe00700 1 -- 172.21.15.62:0/1929126340 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7720102e70 msgr2=0x7f7720106e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:23:21.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:21.564+0000 7f771fe00700 1 --2- 172.21.15.62:0/1929126340 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7720102e70 0x7f7720106e40 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f770c009910 tx=0x7f770c009c20 comp rx=0 tx=0).stop 2024-04-20T22:23:21.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:21.565+0000 7f771fe00700 1 -- 172.21.15.62:0/1929126340 shutdown_connections 2024-04-20T22:23:21.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:21.565+0000 7f771fe00700 1 --2- 172.21.15.62:0/1929126340 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7720102e70 0x7f7720106e40 unknown :-1 s=CLOSED pgs=1225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:23:21.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:21.565+0000 7f771fe00700 1 --2- 172.21.15.62:0/1929126340 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f77201024d0 0x7f77201028a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:23:21.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:21.565+0000 7f771fe00700 1 -- 172.21.15.62:0/1929126340 >> 172.21.15.62:0/1929126340 conn(0x7f77200fdd90 msgr2=0x7f77201001a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:23:21.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:21.565+0000 7f771fe00700 1 -- 172.21.15.62:0/1929126340 shutdown_connections 2024-04-20T22:23:21.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:21.565+0000 7f771fe00700 1 -- 172.21.15.62:0/1929126340 wait complete. 2024-04-20T22:23:21.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:21.566+0000 7f771fe00700 1 Processor -- start 2024-04-20T22:23:21.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:21.566+0000 7f771fe00700 1 -- start start 2024-04-20T22:23:21.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:21.566+0000 7f771fe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f77201024d0 0x7f7720197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:23:21.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:21.566+0000 7f771fe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7720102e70 0x7f77201984c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:23:21.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:21.566+0000 7f771fe00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7720198ba0 con 0x7f7720102e70 2024-04-20T22:23:21.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:21.566+0000 7f771fe00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f772019c930 con 0x7f77201024d0 2024-04-20T22:23:21.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:21.567+0000 7f771e000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7720102e70 0x7f77201984c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:23:21.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:21.567+0000 7f771e000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7720102e70 0x7f77201984c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34836/0 (socket says 172.21.15.62:34836) 2024-04-20T22:23:21.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:21.567+0000 7f771e000700 1 -- 172.21.15.62:0/2961308393 learned_addr learned my addr 172.21.15.62:0/2961308393 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:23:21.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:21.567+0000 7f771ea00700 1 --2- 172.21.15.62:0/2961308393 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f77201024d0 0x7f7720197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:23:21.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:21.567+0000 7f771e000700 1 -- 172.21.15.62:0/2961308393 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f77201024d0 msgr2=0x7f7720197f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:23:21.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:21.567+0000 7f771e000700 1 --2- 172.21.15.62:0/2961308393 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f77201024d0 0x7f7720197f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:23:21.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:21.567+0000 7f771e000700 1 -- 172.21.15.62:0/2961308393 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f770c0095f0 con 0x7f7720102e70 2024-04-20T22:23:21.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:21.568+0000 7f771e000700 1 --2- 172.21.15.62:0/2961308393 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7720102e70 0x7f77201984c0 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f770c0092e0 tx=0x7f770c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:23:21.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:21.568+0000 7f7717400700 1 -- 172.21.15.62:0/2961308393 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f770c01d070 con 0x7f7720102e70 2024-04-20T22:23:21.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:21.568+0000 7f7717400700 1 -- 172.21.15.62:0/2961308393 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f770c00f460 con 0x7f7720102e70 2024-04-20T22:23:21.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:21.568+0000 7f771fe00700 1 -- 172.21.15.62:0/2961308393 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f772019cbb0 con 0x7f7720102e70 2024-04-20T22:23:21.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:21.568+0000 7f7717400700 1 -- 172.21.15.62:0/2961308393 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f770c0175f0 con 0x7f7720102e70 2024-04-20T22:23:21.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:21.568+0000 7f771fe00700 1 -- 172.21.15.62:0/2961308393 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f772019d0a0 con 0x7f7720102e70 2024-04-20T22:23:21.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:21.569+0000 7f771fe00700 1 -- 172.21.15.62:0/2961308393 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f772010a790 con 0x7f7720102e70 2024-04-20T22:23:21.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:21.570+0000 7f7717400700 1 -- 172.21.15.62:0/2961308393 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f770c00f5d0 con 0x7f7720102e70 2024-04-20T22:23:21.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:21.571+0000 7f7717400700 1 --2- 172.21.15.62:0/2961308393 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f770406ecc0 0x7f7704071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:23:21.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:21.571+0000 7f7717400700 1 -- 172.21.15.62:0/2961308393 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f770c090460 con 0x7f7720102e70 2024-04-20T22:23:21.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:21.572+0000 7f771ea00700 1 --2- 172.21.15.62:0/2961308393 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f770406ecc0 0x7f7704071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:23:21.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:21.572+0000 7f771ea00700 1 --2- 172.21.15.62:0/2961308393 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f770406ecc0 0x7f7704071170 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f7710006fd0 tx=0x7f7710008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:23:21.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:21.577+0000 7f7717400700 1 -- 172.21.15.62:0/2961308393 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f770c059170 con 0x7f7720102e70 2024-04-20T22:23:21.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:21.834+0000 7f771fe00700 1 -- 172.21.15.62:0/2961308393 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f772002cca0 con 0x7f770406ecc0 2024-04-20T22:23:21.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:21.837+0000 7f7717400700 1 -- 172.21.15.62:0/2961308393 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f772002cca0 con 0x7f770406ecc0 2024-04-20T22:23:21.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:21.839+0000 7f771fe00700 1 -- 172.21.15.62:0/2961308393 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f770406ecc0 msgr2=0x7f7704071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:23:21.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:21.839+0000 7f771fe00700 1 --2- 172.21.15.62:0/2961308393 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f770406ecc0 0x7f7704071170 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f7710006fd0 tx=0x7f7710008040 comp rx=0 tx=0).stop 2024-04-20T22:23:21.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:21.839+0000 7f771fe00700 1 -- 172.21.15.62:0/2961308393 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7720102e70 msgr2=0x7f77201984c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:23:21.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:21.839+0000 7f771fe00700 1 --2- 172.21.15.62:0/2961308393 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7720102e70 0x7f77201984c0 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f770c0092e0 tx=0x7f770c00be20 comp rx=0 tx=0).stop 2024-04-20T22:23:21.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:21.839+0000 7f771fe00700 1 -- 172.21.15.62:0/2961308393 shutdown_connections 2024-04-20T22:23:21.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:21.839+0000 7f771fe00700 1 --2- 172.21.15.62:0/2961308393 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f770406ecc0 0x7f7704071170 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:23:21.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:21.839+0000 7f771fe00700 1 --2- 172.21.15.62:0/2961308393 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f77201024d0 0x7f7720197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:23:21.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:21.839+0000 7f771fe00700 1 --2- 172.21.15.62:0/2961308393 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7720102e70 0x7f77201984c0 unknown :-1 s=CLOSED pgs=1226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:23:21.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:21.839+0000 7f771fe00700 1 -- 172.21.15.62:0/2961308393 >> 172.21.15.62:0/2961308393 conn(0x7f77200fdd90 msgr2=0x7f77200ff4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:23:21.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:21.839+0000 7f771fe00700 1 -- 172.21.15.62:0/2961308393 shutdown_connections 2024-04-20T22:23:21.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:21.839+0000 7f771fe00700 1 -- 172.21.15.62:0/2961308393 wait complete. 2024-04-20T22:23:21.854 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T22:23:21.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:21.993+0000 7f2ec9e00700 1 -- 172.21.15.62:0/3284274676 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2ec4102470 msgr2=0x7f2ec41028e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:23:21.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:21.993+0000 7f2ec9e00700 1 --2- 172.21.15.62:0/3284274676 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2ec4102470 0x7f2ec41028e0 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7f2eb0009910 tx=0x7f2eb0009c20 comp rx=0 tx=0).stop 2024-04-20T22:23:21.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:21.994+0000 7f2ec9e00700 1 -- 172.21.15.62:0/3284274676 shutdown_connections 2024-04-20T22:23:21.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:21.994+0000 7f2ec9e00700 1 --2- 172.21.15.62:0/3284274676 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2ec4102470 0x7f2ec41028e0 unknown :-1 s=CLOSED pgs=1227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:23:21.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:21.994+0000 7f2ec9e00700 1 --2- 172.21.15.62:0/3284274676 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ec4108470 0x7f2ec4108840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:23:21.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:21.994+0000 7f2ec9e00700 1 -- 172.21.15.62:0/3284274676 >> 172.21.15.62:0/3284274676 conn(0x7f2ec40fdf90 msgr2=0x7f2ec41003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:23:21.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:21.994+0000 7f2ec9e00700 1 -- 172.21.15.62:0/3284274676 shutdown_connections 2024-04-20T22:23:21.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:21.994+0000 7f2ec9e00700 1 -- 172.21.15.62:0/3284274676 wait complete. 2024-04-20T22:23:21.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:21.995+0000 7f2ec9e00700 1 Processor -- start 2024-04-20T22:23:21.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:21.995+0000 7f2ec9e00700 1 -- start start 2024-04-20T22:23:21.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:21.995+0000 7f2ec9e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2ec4102470 0x7f2ec41981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:23:21.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:21.995+0000 7f2ec9e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ec4108470 0x7f2ec4198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:23:21.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:21.995+0000 7f2ec9e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ec4198d80 con 0x7f2ec4102470 2024-04-20T22:23:21.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:21.995+0000 7f2ec9e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ec4198ec0 con 0x7f2ec4108470 2024-04-20T22:23:21.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:21.995+0000 7f2ec2a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2ec4102470 0x7f2ec41981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:23:21.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:21.996+0000 7f2ec2a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2ec4102470 0x7f2ec41981f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34858/0 (socket says 172.21.15.62:34858) 2024-04-20T22:23:21.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:21.996+0000 7f2ec2a00700 1 -- 172.21.15.62:0/3058682042 learned_addr learned my addr 172.21.15.62:0/3058682042 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:23:21.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:21.996+0000 7f2ec2a00700 1 -- 172.21.15.62:0/3058682042 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ec4108470 msgr2=0x7f2ec4198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:23:21.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:21.996+0000 7f2ec2000700 1 --2- 172.21.15.62:0/3058682042 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ec4108470 0x7f2ec4198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:23:21.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:21.996+0000 7f2ec2a00700 1 --2- 172.21.15.62:0/3058682042 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ec4108470 0x7f2ec4198730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:23:21.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:21.996+0000 7f2ec2a00700 1 -- 172.21.15.62:0/3058682042 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2eb00095f0 con 0x7f2ec4102470 2024-04-20T22:23:21.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:21.996+0000 7f2ec2000700 1 --2- 172.21.15.62:0/3058682042 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ec4108470 0x7f2ec4198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T22:23:21.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:21.996+0000 7f2ec2a00700 1 --2- 172.21.15.62:0/3058682042 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2ec4102470 0x7f2ec41981f0 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7f2eb400c930 tx=0x7f2eb400ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:23:21.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:21.997+0000 7f2ec8a00700 1 -- 172.21.15.62:0/3058682042 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2eb4007ad0 con 0x7f2ec4102470 2024-04-20T22:23:21.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:21.997+0000 7f2ec8a00700 1 -- 172.21.15.62:0/3058682042 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2eb4007c30 con 0x7f2ec4102470 2024-04-20T22:23:21.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:21.997+0000 7f2ec8a00700 1 -- 172.21.15.62:0/3058682042 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2eb4018610 con 0x7f2ec4102470 2024-04-20T22:23:21.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:21.997+0000 7f2ec9e00700 1 -- 172.21.15.62:0/3058682042 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ec419cd10 con 0x7f2ec4102470 2024-04-20T22:23:21.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:21.997+0000 7f2ec9e00700 1 -- 172.21.15.62:0/3058682042 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ec419d260 con 0x7f2ec4102470 2024-04-20T22:23:21.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:21.998+0000 7f2ec9e00700 1 -- 172.21.15.62:0/3058682042 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ec410a970 con 0x7f2ec4102470 2024-04-20T22:23:21.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:21.999+0000 7f2ec8a00700 1 -- 172.21.15.62:0/3058682042 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f2eb400f450 con 0x7f2ec4102470 2024-04-20T22:23:21.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.000+0000 7f2ec8a00700 1 --2- 172.21.15.62:0/3058682042 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2ea806ecc0 0x7f2ea8071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:23:21.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.000+0000 7f2ec8a00700 1 -- 172.21.15.62:0/3058682042 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f2eb408eff0 con 0x7f2ec4102470 2024-04-20T22:23:21.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.000+0000 7f2ec2000700 1 --2- 172.21.15.62:0/3058682042 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2ea806ecc0 0x7f2ea8071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:23:21.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.001+0000 7f2ec2000700 1 --2- 172.21.15.62:0/3058682042 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2ea806ecc0 0x7f2ea8071170 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f2eb0006010 tx=0x7f2eb000b540 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:23:22.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.005+0000 7f2ec8a00700 1 -- 172.21.15.62:0/3058682042 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f2eb4057de0 con 0x7f2ec4102470 2024-04-20T22:23:22.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.249+0000 7f2ec9e00700 1 -- 172.21.15.62:0/3058682042 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2ec419d4e0 con 0x7f2ea806ecc0 2024-04-20T22:23:22.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.252+0000 7f2ec8a00700 1 -- 172.21.15.62:0/3058682042 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f2ec419d4e0 con 0x7f2ea806ecc0 2024-04-20T22:23:22.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.254+0000 7f2ec9e00700 1 -- 172.21.15.62:0/3058682042 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2ea806ecc0 msgr2=0x7f2ea8071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:23:22.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.254+0000 7f2ec9e00700 1 --2- 172.21.15.62:0/3058682042 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2ea806ecc0 0x7f2ea8071170 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f2eb0006010 tx=0x7f2eb000b540 comp rx=0 tx=0).stop 2024-04-20T22:23:22.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.254+0000 7f2ec9e00700 1 -- 172.21.15.62:0/3058682042 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2ec4102470 msgr2=0x7f2ec41981f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:23:22.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.254+0000 7f2ec9e00700 1 --2- 172.21.15.62:0/3058682042 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2ec4102470 0x7f2ec41981f0 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7f2eb400c930 tx=0x7f2eb400ccf0 comp rx=0 tx=0).stop 2024-04-20T22:23:22.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.255+0000 7f2ec9e00700 1 -- 172.21.15.62:0/3058682042 shutdown_connections 2024-04-20T22:23:22.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.255+0000 7f2ec9e00700 1 --2- 172.21.15.62:0/3058682042 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2ea806ecc0 0x7f2ea8071170 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:23:22.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.255+0000 7f2ec9e00700 1 --2- 172.21.15.62:0/3058682042 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2ec4102470 0x7f2ec41981f0 unknown :-1 s=CLOSED pgs=1228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:23:22.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.255+0000 7f2ec9e00700 1 --2- 172.21.15.62:0/3058682042 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ec4108470 0x7f2ec4198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:23:22.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.255+0000 7f2ec9e00700 1 -- 172.21.15.62:0/3058682042 >> 172.21.15.62:0/3058682042 conn(0x7f2ec40fdf90 msgr2=0x7f2ec40ff3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:23:22.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.255+0000 7f2ec9e00700 1 -- 172.21.15.62:0/3058682042 shutdown_connections 2024-04-20T22:23:22.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.255+0000 7f2ec9e00700 1 -- 172.21.15.62:0/3058682042 wait complete. 2024-04-20T22:23:22.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.418+0000 7f6269c00700 1 -- 172.21.15.62:0/359641929 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6264108470 msgr2=0x7f6264108840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:23:22.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.418+0000 7f6269c00700 1 --2- 172.21.15.62:0/359641929 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6264108470 0x7f6264108840 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f6254009910 tx=0x7f6254009c20 comp rx=0 tx=0).stop 2024-04-20T22:23:22.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.418+0000 7f6269c00700 1 -- 172.21.15.62:0/359641929 shutdown_connections 2024-04-20T22:23:22.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.418+0000 7f6269c00700 1 --2- 172.21.15.62:0/359641929 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6264102470 0x7f62641028e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:23:22.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.418+0000 7f6269c00700 1 --2- 172.21.15.62:0/359641929 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6264108470 0x7f6264108840 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:23:22.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.418+0000 7f6269c00700 1 -- 172.21.15.62:0/359641929 >> 172.21.15.62:0/359641929 conn(0x7f62640fdf90 msgr2=0x7f62641003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:23:22.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.418+0000 7f6269c00700 1 -- 172.21.15.62:0/359641929 shutdown_connections 2024-04-20T22:23:22.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.418+0000 7f6269c00700 1 -- 172.21.15.62:0/359641929 wait complete. 2024-04-20T22:23:22.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.419+0000 7f6269c00700 1 Processor -- start 2024-04-20T22:23:22.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.419+0000 7f6269c00700 1 -- start start 2024-04-20T22:23:22.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.420+0000 7f6269c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6264102470 0x7f62641981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:23:22.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.420+0000 7f6269c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6264108470 0x7f6264198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:23:22.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.420+0000 7f6269c00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6264198d80 con 0x7f6264108470 2024-04-20T22:23:22.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.420+0000 7f6269c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6264198ec0 con 0x7f6264102470 2024-04-20T22:23:22.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.420+0000 7f6262000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6264108470 0x7f6264198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:23:22.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.420+0000 7f6262000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6264108470 0x7f6264198730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34882/0 (socket says 172.21.15.62:34882) 2024-04-20T22:23:22.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.420+0000 7f6262000700 1 -- 172.21.15.62:0/3895576308 learned_addr learned my addr 172.21.15.62:0/3895576308 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:23:22.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.420+0000 7f6262a00700 1 --2- 172.21.15.62:0/3895576308 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6264102470 0x7f62641981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:23:22.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.421+0000 7f6262000700 1 -- 172.21.15.62:0/3895576308 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6264102470 msgr2=0x7f62641981f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:23:22.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.421+0000 7f6262000700 1 --2- 172.21.15.62:0/3895576308 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6264102470 0x7f62641981f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:23:22.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.421+0000 7f6262000700 1 -- 172.21.15.62:0/3895576308 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62540095f0 con 0x7f6264108470 2024-04-20T22:23:22.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.421+0000 7f6262000700 1 --2- 172.21.15.62:0/3895576308 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6264108470 0x7f6264198730 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7f625000c930 tx=0x7f625000ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:23:22.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.421+0000 7f625b400700 1 -- 172.21.15.62:0/3895576308 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6250007ad0 con 0x7f6264108470 2024-04-20T22:23:22.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.421+0000 7f625b400700 1 -- 172.21.15.62:0/3895576308 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6250007c30 con 0x7f6264108470 2024-04-20T22:23:22.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.421+0000 7f6269c00700 1 -- 172.21.15.62:0/3895576308 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f626419cd10 con 0x7f6264108470 2024-04-20T22:23:22.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.421+0000 7f625b400700 1 -- 172.21.15.62:0/3895576308 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6250018610 con 0x7f6264108470 2024-04-20T22:23:22.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.421+0000 7f6269c00700 1 -- 172.21.15.62:0/3895576308 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f626419d260 con 0x7f6264108470 2024-04-20T22:23:22.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.422+0000 7f6269c00700 1 -- 172.21.15.62:0/3895576308 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f626410a9e0 con 0x7f6264108470 2024-04-20T22:23:22.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.423+0000 7f625b400700 1 -- 172.21.15.62:0/3895576308 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f625000f450 con 0x7f6264108470 2024-04-20T22:23:22.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.424+0000 7f625b400700 1 --2- 172.21.15.62:0/3895576308 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f624806ecc0 0x7f6248071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:23:22.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.425+0000 7f625b400700 1 -- 172.21.15.62:0/3895576308 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f625008f140 con 0x7f6264108470 2024-04-20T22:23:22.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.425+0000 7f6262a00700 1 --2- 172.21.15.62:0/3895576308 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f624806ecc0 0x7f6248071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:23:22.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.426+0000 7f6262a00700 1 --2- 172.21.15.62:0/3895576308 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f624806ecc0 0x7f6248071170 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f6254006010 tx=0x7f625401a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:23:22.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.429+0000 7f625b400700 1 -- 172.21.15.62:0/3895576308 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f6250057e50 con 0x7f6264108470 2024-04-20T22:23:22.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.681+0000 7f6269c00700 1 -- 172.21.15.62:0/3895576308 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f6264199660 con 0x7f624806ecc0 2024-04-20T22:23:22.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.702+0000 7f625b400700 1 -- 172.21.15.62:0/3895576308 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f6264199660 con 0x7f624806ecc0 2024-04-20T22:23:22.701 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T22:23:22.701 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (49m) 10m ago 63m 14.6M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T22:23:22.701 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (63m) 10m ago 63m 11.6M - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T22:23:22.701 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (62m) 10m ago 62m 11.2M - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T22:23:22.702 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (45m) 10m ago 63m 6660k - 19.0.0-3133-ga8c84f59 76bc6b91b726 2825d5ba11c6 2024-04-20T22:23:22.702 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (45m) 10m ago 61m 6665k - 19.0.0-3133-ga8c84f59 76bc6b91b726 20f746fa1bba 2024-04-20T22:23:22.702 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (48m) 10m ago 62m 75.5M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T22:23:22.702 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (40m) 10m ago 55m 22.9M - 19.0.0-3133-ga8c84f59 76bc6b91b726 de80bc90a683 2024-04-20T22:23:22.702 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (41m) 10m ago 55m 23.0M - 19.0.0-3133-ga8c84f59 76bc6b91b726 cd793b7ee98e 2024-04-20T22:23:22.702 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (25m) 10m ago 55m 18.5M - 19.0.0-3133-ga8c84f59 76bc6b91b726 0e7b0bc4d1ab 2024-04-20T22:23:22.702 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (10m) 10m ago 55m 11.4M - 19.0.0-3133-ga8c84f59 76bc6b91b726 0af732ac9e06 2024-04-20T22:23:22.702 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (52m) 10m ago 64m 517M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T22:23:22.702 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (50m) 10m ago 61m 460M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T22:23:22.702 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (46m) 10m ago 65m 86.3M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 019eb1479485 2024-04-20T22:23:22.702 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (46m) 10m ago 61m 58.0M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 55aac560aab6 2024-04-20T22:23:22.702 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (50m) 10m ago 63m 8766k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T22:23:22.702 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (50m) 10m ago 61m 8711k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T22:23:22.702 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (44m) 10m ago 60m 91.3M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 ee2bb901f26e 2024-04-20T22:23:22.702 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (44m) 10m ago 60m 94.9M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 9075152d4162 2024-04-20T22:23:22.703 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (43m) 10m ago 59m 78.5M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 2d4f3e38dc0e 2024-04-20T22:23:22.703 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (43m) 10m ago 58m 94.1M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 8f97b17ae0eb 2024-04-20T22:23:22.703 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (42m) 10m ago 58m 96.1M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 653d8079967d 2024-04-20T22:23:22.703 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (41m) 10m ago 57m 90.9M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 596dded14c12 2024-04-20T22:23:22.703 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (49m) 10m ago 62m 90.4M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T22:23:22.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.705+0000 7f6269c00700 1 -- 172.21.15.62:0/3895576308 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f624806ecc0 msgr2=0x7f6248071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:23:22.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.705+0000 7f6269c00700 1 --2- 172.21.15.62:0/3895576308 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f624806ecc0 0x7f6248071170 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f6254006010 tx=0x7f625401a040 comp rx=0 tx=0).stop 2024-04-20T22:23:22.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.705+0000 7f6269c00700 1 -- 172.21.15.62:0/3895576308 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6264108470 msgr2=0x7f6264198730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:23:22.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.705+0000 7f6269c00700 1 --2- 172.21.15.62:0/3895576308 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6264108470 0x7f6264198730 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7f625000c930 tx=0x7f625000ccf0 comp rx=0 tx=0).stop 2024-04-20T22:23:22.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.705+0000 7f6269c00700 1 -- 172.21.15.62:0/3895576308 shutdown_connections 2024-04-20T22:23:22.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.705+0000 7f6269c00700 1 --2- 172.21.15.62:0/3895576308 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f624806ecc0 0x7f6248071170 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:23:22.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.705+0000 7f6269c00700 1 --2- 172.21.15.62:0/3895576308 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6264102470 0x7f62641981f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:23:22.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.705+0000 7f6269c00700 1 --2- 172.21.15.62:0/3895576308 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6264108470 0x7f6264198730 unknown :-1 s=CLOSED pgs=1230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:23:22.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.705+0000 7f6269c00700 1 -- 172.21.15.62:0/3895576308 >> 172.21.15.62:0/3895576308 conn(0x7f62640fdf90 msgr2=0x7f62641002c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:23:22.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.705+0000 7f6269c00700 1 -- 172.21.15.62:0/3895576308 shutdown_connections 2024-04-20T22:23:22.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.705+0000 7f6269c00700 1 -- 172.21.15.62:0/3895576308 wait complete. 2024-04-20T22:23:22.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.870+0000 7fd210200700 1 -- 172.21.15.62:0/159472609 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2080687a0 msgr2=0x7fd208068b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:23:22.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.870+0000 7fd210200700 1 --2- 172.21.15.62:0/159472609 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2080687a0 0x7fd208068b70 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7fd1fc009910 tx=0x7fd1fc009c20 comp rx=0 tx=0).stop 2024-04-20T22:23:22.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.870+0000 7fd210200700 1 -- 172.21.15.62:0/159472609 shutdown_connections 2024-04-20T22:23:22.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.870+0000 7fd210200700 1 --2- 172.21.15.62:0/159472609 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd208069140 0x7fd208104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:23:22.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.870+0000 7fd210200700 1 --2- 172.21.15.62:0/159472609 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2080687a0 0x7fd208068b70 unknown :-1 s=CLOSED pgs=1231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:23:22.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.870+0000 7fd210200700 1 -- 172.21.15.62:0/159472609 >> 172.21.15.62:0/159472609 conn(0x7fd208075130 msgr2=0x7fd208075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:23:22.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.870+0000 7fd210200700 1 -- 172.21.15.62:0/159472609 shutdown_connections 2024-04-20T22:23:22.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.871+0000 7fd210200700 1 -- 172.21.15.62:0/159472609 wait complete. 2024-04-20T22:23:22.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.872+0000 7fd210200700 1 Processor -- start 2024-04-20T22:23:22.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.872+0000 7fd210200700 1 -- start start 2024-04-20T22:23:22.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.872+0000 7fd210200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2080687a0 0x7fd2081980e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:23:22.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.872+0000 7fd210200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd208069140 0x7fd208198620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:23:22.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.872+0000 7fd210200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd208198d00 con 0x7fd2080687a0 2024-04-20T22:23:22.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.872+0000 7fd210200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd20819ca90 con 0x7fd208069140 2024-04-20T22:23:22.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.873+0000 7fd20d800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2080687a0 0x7fd2081980e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:23:22.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.873+0000 7fd20d800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2080687a0 0x7fd2081980e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34896/0 (socket says 172.21.15.62:34896) 2024-04-20T22:23:22.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.873+0000 7fd20d800700 1 -- 172.21.15.62:0/888772516 learned_addr learned my addr 172.21.15.62:0/888772516 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:23:22.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.873+0000 7fd20ce00700 1 --2- 172.21.15.62:0/888772516 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd208069140 0x7fd208198620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:23:22.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.873+0000 7fd20d800700 1 -- 172.21.15.62:0/888772516 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd208069140 msgr2=0x7fd208198620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:23:22.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.873+0000 7fd20d800700 1 --2- 172.21.15.62:0/888772516 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd208069140 0x7fd208198620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:23:22.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.873+0000 7fd20d800700 1 -- 172.21.15.62:0/888772516 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1fc0095f0 con 0x7fd2080687a0 2024-04-20T22:23:22.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.873+0000 7fd20d800700 1 --2- 172.21.15.62:0/888772516 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2080687a0 0x7fd2081980e0 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7fd1fc0092e0 tx=0x7fd1fc00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:23:22.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.874+0000 7fd206000700 1 -- 172.21.15.62:0/888772516 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd1fc01d070 con 0x7fd2080687a0 2024-04-20T22:23:22.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.874+0000 7fd206000700 1 -- 172.21.15.62:0/888772516 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd1fc005e00 con 0x7fd2080687a0 2024-04-20T22:23:22.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.874+0000 7fd210200700 1 -- 172.21.15.62:0/888772516 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd20819cd10 con 0x7fd2080687a0 2024-04-20T22:23:22.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.874+0000 7fd206000700 1 -- 172.21.15.62:0/888772516 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd1fc017400 con 0x7fd2080687a0 2024-04-20T22:23:22.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.874+0000 7fd210200700 1 -- 172.21.15.62:0/888772516 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd20819d200 con 0x7fd2080687a0 2024-04-20T22:23:22.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.875+0000 7fd210200700 1 -- 172.21.15.62:0/888772516 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd20804f2a0 con 0x7fd2080687a0 2024-04-20T22:23:22.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.876+0000 7fd206000700 1 -- 172.21.15.62:0/888772516 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fd1fc00f460 con 0x7fd2080687a0 2024-04-20T22:23:22.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.877+0000 7fd206000700 1 --2- 172.21.15.62:0/888772516 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd1f006ecc0 0x7fd1f0071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:23:22.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.877+0000 7fd206000700 1 -- 172.21.15.62:0/888772516 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fd1fc090330 con 0x7fd2080687a0 2024-04-20T22:23:22.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.878+0000 7fd20ce00700 1 --2- 172.21.15.62:0/888772516 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd1f006ecc0 0x7fd1f0071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:23:22.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.878+0000 7fd20ce00700 1 --2- 172.21.15.62:0/888772516 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd1f006ecc0 0x7fd1f0071170 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fd1f4005950 tx=0x7fd1f40058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:23:22.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:22.882+0000 7fd206000700 1 -- 172.21.15.62:0/888772516 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fd1fc0590c0 con 0x7fd2080687a0 2024-04-20T22:23:22.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:23:22 smithi189 ceph-mon[57022]: pgmap v1445: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:23:22.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:23:22 smithi189 ceph-mon[57022]: from='client.36524 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:23:22.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:23:22 smithi189 ceph-mon[57022]: from='client.36528 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:23:23.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:23:22 smithi062 ceph-mon[76322]: pgmap v1445: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:23:23.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:23:22 smithi062 ceph-mon[76322]: from='client.36524 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:23:23.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:23:22 smithi062 ceph-mon[76322]: from='client.36528 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:23:23.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:23.213+0000 7fd210200700 1 -- 172.21.15.62:0/888772516 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fd20819d530 con 0x7fd2080687a0 2024-04-20T22:23:23.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:23.214+0000 7fd206000700 1 -- 172.21.15.62:0/888772516 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7fd1fc05c780 con 0x7fd2080687a0 2024-04-20T22:23:23.213 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:23:23.213 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T22:23:23.213 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:23:23.213 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:23:23.213 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T22:23:23.213 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:23:23.213 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:23:23.213 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T22:23:23.213 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 6 2024-04-20T22:23:23.214 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:23:23.214 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T22:23:23.214 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 4 2024-04-20T22:23:23.214 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:23:23.214 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T22:23:23.214 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 14 2024-04-20T22:23:23.214 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T22:23:23.214 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:23:23.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:23.216+0000 7fd210200700 1 -- 172.21.15.62:0/888772516 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd1f006ecc0 msgr2=0x7fd1f0071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:23:23.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:23.217+0000 7fd210200700 1 --2- 172.21.15.62:0/888772516 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd1f006ecc0 0x7fd1f0071170 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fd1f4005950 tx=0x7fd1f40058e0 comp rx=0 tx=0).stop 2024-04-20T22:23:23.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:23.217+0000 7fd210200700 1 -- 172.21.15.62:0/888772516 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2080687a0 msgr2=0x7fd2081980e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:23:23.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:23.217+0000 7fd210200700 1 --2- 172.21.15.62:0/888772516 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2080687a0 0x7fd2081980e0 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7fd1fc0092e0 tx=0x7fd1fc00be20 comp rx=0 tx=0).stop 2024-04-20T22:23:23.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:23.217+0000 7fd210200700 1 -- 172.21.15.62:0/888772516 shutdown_connections 2024-04-20T22:23:23.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:23.217+0000 7fd210200700 1 --2- 172.21.15.62:0/888772516 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd1f006ecc0 0x7fd1f0071170 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:23:23.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:23.217+0000 7fd210200700 1 --2- 172.21.15.62:0/888772516 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2080687a0 0x7fd2081980e0 unknown :-1 s=CLOSED pgs=1232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:23:23.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:23.217+0000 7fd210200700 1 --2- 172.21.15.62:0/888772516 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd208069140 0x7fd208198620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:23:23.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:23.217+0000 7fd210200700 1 -- 172.21.15.62:0/888772516 >> 172.21.15.62:0/888772516 conn(0x7fd208075130 msgr2=0x7fd2080fe7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:23:23.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:23.217+0000 7fd210200700 1 -- 172.21.15.62:0/888772516 shutdown_connections 2024-04-20T22:23:23.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:23.217+0000 7fd210200700 1 -- 172.21.15.62:0/888772516 wait complete. 2024-04-20T22:23:23.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:23.380+0000 7f96b3400700 1 -- 172.21.15.62:0/2741390251 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f96ac068540 msgr2=0x7f96ac068910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:23:23.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:23.380+0000 7f96b3400700 1 --2- 172.21.15.62:0/2741390251 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f96ac068540 0x7f96ac068910 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f96a000b0d0 tx=0x7f96a000b3e0 comp rx=0 tx=0).stop 2024-04-20T22:23:23.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:23.381+0000 7f96b3400700 1 -- 172.21.15.62:0/2741390251 shutdown_connections 2024-04-20T22:23:23.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:23.381+0000 7f96b3400700 1 --2- 172.21.15.62:0/2741390251 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96ac068ee0 0x7f96ac110a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:23:23.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:23.381+0000 7f96b3400700 1 --2- 172.21.15.62:0/2741390251 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f96ac068540 0x7f96ac068910 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:23:23.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:23.381+0000 7f96b3400700 1 -- 172.21.15.62:0/2741390251 >> 172.21.15.62:0/2741390251 conn(0x7f96ac075130 msgr2=0x7f96ac075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:23:23.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:23.381+0000 7f96b3400700 1 -- 172.21.15.62:0/2741390251 shutdown_connections 2024-04-20T22:23:23.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:23.381+0000 7f96b3400700 1 -- 172.21.15.62:0/2741390251 wait complete. 2024-04-20T22:23:23.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:23.382+0000 7f96b3400700 1 Processor -- start 2024-04-20T22:23:23.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:23.382+0000 7f96b3400700 1 -- start start 2024-04-20T22:23:23.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:23.382+0000 7f96b3400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f96ac068540 0x7f96ac19c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:23:23.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:23.382+0000 7f96b3400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96ac068ee0 0x7f96ac19cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:23:23.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:23.382+0000 7f96b3400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96ac19d220 con 0x7f96ac068540 2024-04-20T22:23:23.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:23.382+0000 7f96b3400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96ac1a0fb0 con 0x7f96ac068ee0 2024-04-20T22:23:23.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:23.383+0000 7f96b0a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f96ac068540 0x7f96ac19c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:23:23.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:23.383+0000 7f96b0a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f96ac068540 0x7f96ac19c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34908/0 (socket says 172.21.15.62:34908) 2024-04-20T22:23:23.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:23.383+0000 7f96b0a00700 1 -- 172.21.15.62:0/720579718 learned_addr learned my addr 172.21.15.62:0/720579718 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:23:23.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:23.383+0000 7f96abe00700 1 --2- 172.21.15.62:0/720579718 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96ac068ee0 0x7f96ac19cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:23:23.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:23.383+0000 7f96b0a00700 1 -- 172.21.15.62:0/720579718 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96ac068ee0 msgr2=0x7f96ac19cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:23:23.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:23.383+0000 7f96b0a00700 1 --2- 172.21.15.62:0/720579718 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96ac068ee0 0x7f96ac19cb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:23:23.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:23.383+0000 7f96b0a00700 1 -- 172.21.15.62:0/720579718 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96a0009d00 con 0x7f96ac068540 2024-04-20T22:23:23.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:23.383+0000 7f96abe00700 1 --2- 172.21.15.62:0/720579718 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96ac068ee0 0x7f96ac19cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T22:23:23.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:23.384+0000 7f96b0a00700 1 --2- 172.21.15.62:0/720579718 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f96ac068540 0x7f96ac19c600 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7f96a00062a0 tx=0x7f96a000cdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:23:23.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:23.384+0000 7f96a9600700 1 -- 172.21.15.62:0/720579718 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f96a000e040 con 0x7f96ac068540 2024-04-20T22:23:23.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:23.384+0000 7f96a9600700 1 -- 172.21.15.62:0/720579718 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f96a0004500 con 0x7f96ac068540 2024-04-20T22:23:23.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:23.384+0000 7f96b3400700 1 -- 172.21.15.62:0/720579718 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96ac1a1230 con 0x7f96ac068540 2024-04-20T22:23:23.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:23.384+0000 7f96a9600700 1 -- 172.21.15.62:0/720579718 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f96a001dab0 con 0x7f96ac068540 2024-04-20T22:23:23.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:23.384+0000 7f96b3400700 1 -- 172.21.15.62:0/720579718 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96ac1a1780 con 0x7f96ac068540 2024-04-20T22:23:23.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:23.385+0000 7f96b3400700 1 -- 172.21.15.62:0/720579718 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96ac101410 con 0x7f96ac068540 2024-04-20T22:23:23.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:23.386+0000 7f96a9600700 1 -- 172.21.15.62:0/720579718 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f96a001dc10 con 0x7f96ac068540 2024-04-20T22:23:23.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:23.387+0000 7f96a9600700 1 --2- 172.21.15.62:0/720579718 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f969406ecc0 0x7f9694071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:23:23.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:23.387+0000 7f96a9600700 1 -- 172.21.15.62:0/720579718 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f96a0090880 con 0x7f96ac068540 2024-04-20T22:23:23.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:23.388+0000 7f96abe00700 1 --2- 172.21.15.62:0/720579718 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f969406ecc0 0x7f9694071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:23:23.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:23.388+0000 7f96abe00700 1 --2- 172.21.15.62:0/720579718 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f969406ecc0 0x7f9694071170 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f96ac19dc20 tx=0x7f9698008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:23:23.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:23.392+0000 7f96a9600700 1 -- 172.21.15.62:0/720579718 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f96a0059610 con 0x7f96ac068540 2024-04-20T22:23:23.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:23.680+0000 7f96b3400700 1 -- 172.21.15.62:0/720579718 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f96ac04f2a0 con 0x7f96ac068540 2024-04-20T22:23:23.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:23.681+0000 7f96a9600700 1 -- 172.21.15.62:0/720579718 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 29 v29) v1 ==== 76+0+1887 (secure 0 0 0) 0x7f96a005ccd0 con 0x7f96ac068540 2024-04-20T22:23:23.680 INFO:teuthology.orchestra.run.smithi062.stdout:e29 2024-04-20T22:23:23.680 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T22:23:23.680 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:23:23.680 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T22:23:23.681 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:23:23.681 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T22:23:23.681 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T22:23:23.681 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 20 2024-04-20T22:23:23.681 INFO:teuthology.orchestra.run.smithi062.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-20T22:23:23.681 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T22:23:23.681 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T21:42:01.559061+0000 2024-04-20T22:23:23.681 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T22:23:23.681 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T22:23:23.681 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T22:23:23.681 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T22:23:23.681 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T22:23:23.681 INFO:teuthology.orchestra.run.smithi062.stdout:max_xattr_size 65536 2024-04-20T22:23:23.681 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T22:23:23.681 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T22:23:23.682 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 81 2024-04-20T22:23:23.682 INFO:teuthology.orchestra.run.smithi062.stdout:snap_rstat 1 2024-04-20T22:23:23.682 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:23:23.682 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 2 2024-04-20T22:23:23.682 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1 2024-04-20T22:23:23.682 INFO:teuthology.orchestra.run.smithi062.stdout:up {} 2024-04-20T22:23:23.682 INFO:teuthology.orchestra.run.smithi062.stdout:failed 0,1 2024-04-20T22:23:23.682 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T22:23:23.682 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T22:23:23.682 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T22:23:23.682 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T22:23:23.682 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data enabled 2024-04-20T22:23:23.682 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T22:23:23.682 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T22:23:23.683 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 1 2024-04-20T22:23:23.683 INFO:teuthology.orchestra.run.smithi062.stdout:qdb_cluster leader: 0 members: 2024-04-20T22:23:23.683 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:23:23.683 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:23:23.683 INFO:teuthology.orchestra.run.smithi062.stdout:Standby daemons: 2024-04-20T22:23:23.683 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:23:23.683 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ryqblb{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6826/4113916205,v1:172.21.15.62:6827/4113916205] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:23:23.683 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ozizwi{-1:34476} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6828/3801975165,v1:172.21.15.62:6829/3801975165] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:23:23.683 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.dvimer{-1:44727} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6824/3576699232,v1:172.21.15.189:6825/3576699232] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:23:23.683 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.xbgspw{-1:45075} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6826/212905792,v1:172.21.15.189:6827/212905792] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:23:23.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:23.684+0000 7f96b3400700 1 -- 172.21.15.62:0/720579718 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f969406ecc0 msgr2=0x7f9694071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:23:23.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:23.684+0000 7f96b3400700 1 --2- 172.21.15.62:0/720579718 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f969406ecc0 0x7f9694071170 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f96ac19dc20 tx=0x7f9698008040 comp rx=0 tx=0).stop 2024-04-20T22:23:23.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:23.684+0000 7f96b3400700 1 -- 172.21.15.62:0/720579718 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f96ac068540 msgr2=0x7f96ac19c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:23:23.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:23.684+0000 7f96b3400700 1 --2- 172.21.15.62:0/720579718 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f96ac068540 0x7f96ac19c600 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7f96a00062a0 tx=0x7f96a000cdd0 comp rx=0 tx=0).stop 2024-04-20T22:23:23.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:23.684+0000 7f96b3400700 1 -- 172.21.15.62:0/720579718 shutdown_connections 2024-04-20T22:23:23.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:23.684+0000 7f96b3400700 1 --2- 172.21.15.62:0/720579718 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f969406ecc0 0x7f9694071170 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:23:23.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:23.684+0000 7f96b3400700 1 --2- 172.21.15.62:0/720579718 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f96ac068540 0x7f96ac19c600 unknown :-1 s=CLOSED pgs=1234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:23:23.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:23.684+0000 7f96b3400700 1 --2- 172.21.15.62:0/720579718 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96ac068ee0 0x7f96ac19cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:23:23.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:23.684+0000 7f96b3400700 1 -- 172.21.15.62:0/720579718 >> 172.21.15.62:0/720579718 conn(0x7f96ac075130 msgr2=0x7f96ac10f070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:23:23.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:23.684+0000 7f96b3400700 1 -- 172.21.15.62:0/720579718 shutdown_connections 2024-04-20T22:23:23.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:23.684+0000 7f96b3400700 1 -- 172.21.15.62:0/720579718 wait complete. 2024-04-20T22:23:23.685 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 29 2024-04-20T22:23:23.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:23.852+0000 7fb4b1e00700 1 -- 172.21.15.62:0/3031383962 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb4ac100e80 msgr2=0x7fb4ac104e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:23:23.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:23.852+0000 7fb4b1e00700 1 --2- 172.21.15.62:0/3031383962 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb4ac100e80 0x7fb4ac104e50 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7fb498009910 tx=0x7fb498009c20 comp rx=0 tx=0).stop 2024-04-20T22:23:23.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:23.852+0000 7fb4b1e00700 1 -- 172.21.15.62:0/3031383962 shutdown_connections 2024-04-20T22:23:23.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:23.852+0000 7fb4b1e00700 1 --2- 172.21.15.62:0/3031383962 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb4ac100e80 0x7fb4ac104e50 unknown :-1 s=CLOSED pgs=1235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:23:23.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:23.853+0000 7fb4b1e00700 1 --2- 172.21.15.62:0/3031383962 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb4ac1004e0 0x7fb4ac1008b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:23:23.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:23.853+0000 7fb4b1e00700 1 -- 172.21.15.62:0/3031383962 >> 172.21.15.62:0/3031383962 conn(0x7fb4ac0fbda0 msgr2=0x7fb4ac0fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:23:23.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:23.853+0000 7fb4b1e00700 1 -- 172.21.15.62:0/3031383962 shutdown_connections 2024-04-20T22:23:23.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:23.853+0000 7fb4b1e00700 1 -- 172.21.15.62:0/3031383962 wait complete. 2024-04-20T22:23:23.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:23.854+0000 7fb4b1e00700 1 Processor -- start 2024-04-20T22:23:23.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:23.854+0000 7fb4b1e00700 1 -- start start 2024-04-20T22:23:23.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:23.854+0000 7fb4b1e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb4ac1004e0 0x7fb4ac195f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:23:23.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:23.854+0000 7fb4b1e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb4ac100e80 0x7fb4ac196470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:23:23.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:23.854+0000 7fb4b1e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4ac196b50 con 0x7fb4ac1004e0 2024-04-20T22:23:23.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:23.854+0000 7fb4b1e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4ac19a8e0 con 0x7fb4ac100e80 2024-04-20T22:23:23.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:23.855+0000 7fb4aaa00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb4ac1004e0 0x7fb4ac195f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:23:23.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:23.855+0000 7fb4aaa00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb4ac1004e0 0x7fb4ac195f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34924/0 (socket says 172.21.15.62:34924) 2024-04-20T22:23:23.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:23.855+0000 7fb4aaa00700 1 -- 172.21.15.62:0/3849698839 learned_addr learned my addr 172.21.15.62:0/3849698839 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:23:23.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:23.855+0000 7fb4aa000700 1 --2- 172.21.15.62:0/3849698839 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb4ac100e80 0x7fb4ac196470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:23:23.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:23.855+0000 7fb4aaa00700 1 -- 172.21.15.62:0/3849698839 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb4ac100e80 msgr2=0x7fb4ac196470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:23:23.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:23.855+0000 7fb4aaa00700 1 --2- 172.21.15.62:0/3849698839 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb4ac100e80 0x7fb4ac196470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:23:23.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:23.855+0000 7fb4aaa00700 1 -- 172.21.15.62:0/3849698839 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4980095f0 con 0x7fb4ac1004e0 2024-04-20T22:23:23.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:23.855+0000 7fb4aaa00700 1 --2- 172.21.15.62:0/3849698839 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb4ac1004e0 0x7fb4ac195f30 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7fb49c00d8d0 tx=0x7fb49c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:23:23.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:23.856+0000 7fb4b0a00700 1 -- 172.21.15.62:0/3849698839 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb49c009940 con 0x7fb4ac1004e0 2024-04-20T22:23:23.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:23.856+0000 7fb4b0a00700 1 -- 172.21.15.62:0/3849698839 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb49c010460 con 0x7fb4ac1004e0 2024-04-20T22:23:23.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:23.856+0000 7fb4b0a00700 1 -- 172.21.15.62:0/3849698839 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb49c00f5f0 con 0x7fb4ac1004e0 2024-04-20T22:23:23.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:23.856+0000 7fb4b1e00700 1 -- 172.21.15.62:0/3849698839 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4ac19abc0 con 0x7fb4ac1004e0 2024-04-20T22:23:23.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:23.856+0000 7fb4b1e00700 1 -- 172.21.15.62:0/3849698839 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4ac19b110 con 0x7fb4ac1004e0 2024-04-20T22:23:23.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:23.857+0000 7fb4b1e00700 1 -- 172.21.15.62:0/3849698839 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4ac108850 con 0x7fb4ac1004e0 2024-04-20T22:23:23.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:23.858+0000 7fb4b0a00700 1 -- 172.21.15.62:0/3849698839 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fb49c0105d0 con 0x7fb4ac1004e0 2024-04-20T22:23:23.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:23.859+0000 7fb4b0a00700 1 --2- 172.21.15.62:0/3849698839 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb49006ecc0 0x7fb490071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:23:23.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:23.859+0000 7fb4b0a00700 1 -- 172.21.15.62:0/3849698839 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fb49c08ee10 con 0x7fb4ac1004e0 2024-04-20T22:23:23.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:23.859+0000 7fb4aa000700 1 --2- 172.21.15.62:0/3849698839 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb49006ecc0 0x7fb490071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:23:23.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:23.860+0000 7fb4aa000700 1 --2- 172.21.15.62:0/3849698839 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb49006ecc0 0x7fb490071170 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fb4ac197550 tx=0x7fb49801a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:23:23.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:23.864+0000 7fb4b0a00700 1 -- 172.21.15.62:0/3849698839 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fb49c057ba0 con 0x7fb4ac1004e0 2024-04-20T22:23:23.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:23:23 smithi189 ceph-mon[57022]: from='client.36532 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:23:23.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:23:23 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/888772516' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:23:23.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:23:23 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/720579718' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:23:24.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:23:23 smithi062 ceph-mon[76322]: from='client.36532 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:23:24.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:23:23 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/888772516' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:23:24.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:23:23 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/720579718' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:23:24.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:24.124+0000 7fb4b1e00700 1 -- 172.21.15.62:0/3849698839 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb4ac19b380 con 0x7fb49006ecc0 2024-04-20T22:23:24.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:24.126+0000 7fb4b0a00700 1 -- 172.21.15.62:0/3849698839 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fb4ac19b380 con 0x7fb49006ecc0 2024-04-20T22:23:24.126 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:23:24.126 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T22:23:24.126 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T22:23:24.126 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-20T22:23:24.126 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T22:23:24.126 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr", 2024-04-20T22:23:24.126 INFO:teuthology.orchestra.run.smithi062.stdout: "osd", 2024-04-20T22:23:24.126 INFO:teuthology.orchestra.run.smithi062.stdout: "mds", 2024-04-20T22:23:24.127 INFO:teuthology.orchestra.run.smithi062.stdout: "crash", 2024-04-20T22:23:24.127 INFO:teuthology.orchestra.run.smithi062.stdout: "mon" 2024-04-20T22:23:24.127 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T22:23:24.127 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "16/23 daemons upgraded", 2024-04-20T22:23:24.127 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading mds daemons", 2024-04-20T22:23:24.127 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T22:23:24.127 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:23:24.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:24.130+0000 7fb4b1e00700 1 -- 172.21.15.62:0/3849698839 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb49006ecc0 msgr2=0x7fb490071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:23:24.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:24.130+0000 7fb4b1e00700 1 --2- 172.21.15.62:0/3849698839 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb49006ecc0 0x7fb490071170 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fb4ac197550 tx=0x7fb49801a040 comp rx=0 tx=0).stop 2024-04-20T22:23:24.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:24.130+0000 7fb4b1e00700 1 -- 172.21.15.62:0/3849698839 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb4ac1004e0 msgr2=0x7fb4ac195f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:23:24.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:24.130+0000 7fb4b1e00700 1 --2- 172.21.15.62:0/3849698839 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb4ac1004e0 0x7fb4ac195f30 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7fb49c00d8d0 tx=0x7fb49c00dc90 comp rx=0 tx=0).stop 2024-04-20T22:23:24.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:24.130+0000 7fb4b1e00700 1 -- 172.21.15.62:0/3849698839 shutdown_connections 2024-04-20T22:23:24.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:24.130+0000 7fb4b1e00700 1 --2- 172.21.15.62:0/3849698839 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb49006ecc0 0x7fb490071170 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:23:24.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:24.130+0000 7fb4b1e00700 1 --2- 172.21.15.62:0/3849698839 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb4ac1004e0 0x7fb4ac195f30 unknown :-1 s=CLOSED pgs=1236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:23:24.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:24.130+0000 7fb4b1e00700 1 --2- 172.21.15.62:0/3849698839 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb4ac100e80 0x7fb4ac196470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:23:24.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:24.130+0000 7fb4b1e00700 1 -- 172.21.15.62:0/3849698839 >> 172.21.15.62:0/3849698839 conn(0x7fb4ac0fbda0 msgr2=0x7fb4ac0fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:23:24.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:24.130+0000 7fb4b1e00700 1 -- 172.21.15.62:0/3849698839 shutdown_connections 2024-04-20T22:23:24.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:24.130+0000 7fb4b1e00700 1 -- 172.21.15.62:0/3849698839 wait complete. 2024-04-20T22:23:24.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:24.289+0000 7feabc800700 1 -- 172.21.15.62:0/2583373077 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feab4108470 msgr2=0x7feab4108840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:23:24.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:24.289+0000 7feabc800700 1 --2- 172.21.15.62:0/2583373077 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feab4108470 0x7feab4108840 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7feaa8009910 tx=0x7feaa8009c20 comp rx=0 tx=0).stop 2024-04-20T22:23:24.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:24.289+0000 7feabc800700 1 -- 172.21.15.62:0/2583373077 shutdown_connections 2024-04-20T22:23:24.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:24.289+0000 7feabc800700 1 --2- 172.21.15.62:0/2583373077 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feab4102470 0x7feab41028e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:23:24.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:24.289+0000 7feabc800700 1 --2- 172.21.15.62:0/2583373077 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feab4108470 0x7feab4108840 unknown :-1 s=CLOSED pgs=1237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:23:24.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:24.289+0000 7feabc800700 1 -- 172.21.15.62:0/2583373077 >> 172.21.15.62:0/2583373077 conn(0x7feab40fdf90 msgr2=0x7feab41003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:23:24.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:24.289+0000 7feabc800700 1 -- 172.21.15.62:0/2583373077 shutdown_connections 2024-04-20T22:23:24.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:24.289+0000 7feabc800700 1 -- 172.21.15.62:0/2583373077 wait complete. 2024-04-20T22:23:24.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:24.290+0000 7feabc800700 1 Processor -- start 2024-04-20T22:23:24.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:24.290+0000 7feabc800700 1 -- start start 2024-04-20T22:23:24.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:24.291+0000 7feabc800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feab4102470 0x7feab41981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:23:24.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:24.291+0000 7feabc800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feab4108470 0x7feab4198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:23:24.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:24.291+0000 7feabc800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feab4198e10 con 0x7feab4102470 2024-04-20T22:23:24.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:24.291+0000 7feabc800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feab419cba0 con 0x7feab4108470 2024-04-20T22:23:24.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:24.291+0000 7feab9e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feab4102470 0x7feab41981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:23:24.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:24.291+0000 7feab9e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feab4102470 0x7feab41981f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34938/0 (socket says 172.21.15.62:34938) 2024-04-20T22:23:24.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:24.291+0000 7feab9e00700 1 -- 172.21.15.62:0/3388354486 learned_addr learned my addr 172.21.15.62:0/3388354486 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:23:24.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:24.292+0000 7feab9400700 1 --2- 172.21.15.62:0/3388354486 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feab4108470 0x7feab4198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:23:24.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:24.292+0000 7feab9e00700 1 -- 172.21.15.62:0/3388354486 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feab4108470 msgr2=0x7feab4198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:23:24.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:24.292+0000 7feab9e00700 1 --2- 172.21.15.62:0/3388354486 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feab4108470 0x7feab4198730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:23:24.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:24.292+0000 7feab9e00700 1 -- 172.21.15.62:0/3388354486 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaa80095f0 con 0x7feab4102470 2024-04-20T22:23:24.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:24.292+0000 7feab9e00700 1 --2- 172.21.15.62:0/3388354486 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feab4102470 0x7feab41981f0 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7feaa8006010 tx=0x7feaa800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:23:24.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:24.292+0000 7feaaea00700 1 -- 172.21.15.62:0/3388354486 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feaa801d070 con 0x7feab4102470 2024-04-20T22:23:24.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:24.292+0000 7feaaea00700 1 -- 172.21.15.62:0/3388354486 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7feaa800f460 con 0x7feab4102470 2024-04-20T22:23:24.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:24.292+0000 7feaaea00700 1 -- 172.21.15.62:0/3388354486 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feaa8017630 con 0x7feab4102470 2024-04-20T22:23:24.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:24.292+0000 7feabc800700 1 -- 172.21.15.62:0/3388354486 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feab419ce20 con 0x7feab4102470 2024-04-20T22:23:24.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:24.293+0000 7feabc800700 1 -- 172.21.15.62:0/3388354486 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feab419d310 con 0x7feab4102470 2024-04-20T22:23:24.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:24.294+0000 7feabc800700 1 -- 172.21.15.62:0/3388354486 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feab410a9e0 con 0x7feab4102470 2024-04-20T22:23:24.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:24.295+0000 7feaaea00700 1 -- 172.21.15.62:0/3388354486 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7feaa800f5d0 con 0x7feab4102470 2024-04-20T22:23:24.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:24.296+0000 7feaaea00700 1 --2- 172.21.15.62:0/3388354486 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fea9c06ecc0 0x7fea9c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:23:24.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:24.296+0000 7feaaea00700 1 -- 172.21.15.62:0/3388354486 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7feaa80903b0 con 0x7feab4102470 2024-04-20T22:23:24.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:24.296+0000 7feab9400700 1 --2- 172.21.15.62:0/3388354486 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fea9c06ecc0 0x7fea9c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:23:24.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:24.297+0000 7feab9400700 1 --2- 172.21.15.62:0/3388354486 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fea9c06ecc0 0x7fea9c071170 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7feaa4005950 tx=0x7feaa40058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:23:24.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:24.301+0000 7feaaea00700 1 -- 172.21.15.62:0/3388354486 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7feaa80590c0 con 0x7feab4102470 2024-04-20T22:23:24.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:24.639+0000 7feabc800700 1 -- 172.21.15.62:0/3388354486 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7feab404f2a0 con 0x7feab4102470 2024-04-20T22:23:24.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:24.640+0000 7feaaea00700 1 -- 172.21.15.62:0/3388354486 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7feaa805c780 con 0x7feab4102470 2024-04-20T22:23:24.639 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-20T22:23:24.639 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-20T22:23:24.640 INFO:teuthology.orchestra.run.smithi062.stdout: host smithi189 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi189/config 2024-04-20T22:23:24.640 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-20T22:23:24.640 INFO:teuthology.orchestra.run.smithi062.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi189 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp11670v2q:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:23:24.640 INFO:teuthology.orchestra.run.smithi062.stdout:Traceback (most recent call last): 2024-04-20T22:23:24.640 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-20T22:23:24.640 INFO:teuthology.orchestra.run.smithi062.stdout: return _run_code(code, main_globals, None, 2024-04-20T22:23:24.640 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-20T22:23:24.640 INFO:teuthology.orchestra.run.smithi062.stdout: exec(code, run_globals) 2024-04-20T22:23:24.640 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5618, in 2024-04-20T22:23:24.640 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5606, in main 2024-04-20T22:23:24.640 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 408, in _infer_config 2024-04-20T22:23:24.640 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 323, in _infer_fsid 2024-04-20T22:23:24.640 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 436, in _infer_image 2024-04-20T22:23:24.640 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 310, in _validate_fsid 2024-04-20T22:23:24.641 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 3359, in command_ceph_volume 2024-04-20T22:23:24.641 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-20T22:23:24.641 INFO:teuthology.orchestra.run.smithi062.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi189 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp11670v2q:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:23:24.641 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-20T22:23:24.641 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is degraded 2024-04-20T22:23:24.641 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T22:23:24.641 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T22:23:24.641 INFO:teuthology.orchestra.run.smithi062.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-20T22:23:24.641 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-20T22:23:24.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:24.643+0000 7feabc800700 1 -- 172.21.15.62:0/3388354486 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fea9c06ecc0 msgr2=0x7fea9c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:23:24.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:24.643+0000 7feabc800700 1 --2- 172.21.15.62:0/3388354486 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fea9c06ecc0 0x7fea9c071170 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7feaa4005950 tx=0x7feaa40058e0 comp rx=0 tx=0).stop 2024-04-20T22:23:24.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:24.643+0000 7feabc800700 1 -- 172.21.15.62:0/3388354486 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feab4102470 msgr2=0x7feab41981f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:23:24.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:24.643+0000 7feabc800700 1 --2- 172.21.15.62:0/3388354486 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feab4102470 0x7feab41981f0 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7feaa8006010 tx=0x7feaa800be20 comp rx=0 tx=0).stop 2024-04-20T22:23:24.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:24.643+0000 7feabc800700 1 -- 172.21.15.62:0/3388354486 shutdown_connections 2024-04-20T22:23:24.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:24.643+0000 7feabc800700 1 --2- 172.21.15.62:0/3388354486 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fea9c06ecc0 0x7fea9c071170 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:23:24.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:24.643+0000 7feabc800700 1 --2- 172.21.15.62:0/3388354486 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feab4102470 0x7feab41981f0 unknown :-1 s=CLOSED pgs=1238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:23:24.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:24.643+0000 7feabc800700 1 --2- 172.21.15.62:0/3388354486 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feab4108470 0x7feab4198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:23:24.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:24.643+0000 7feabc800700 1 -- 172.21.15.62:0/3388354486 >> 172.21.15.62:0/3388354486 conn(0x7feab40fdf90 msgr2=0x7feab41002c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:23:24.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:24.644+0000 7feabc800700 1 -- 172.21.15.62:0/3388354486 shutdown_connections 2024-04-20T22:23:24.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:24.644+0000 7feabc800700 1 -- 172.21.15.62:0/3388354486 wait complete. 2024-04-20T22:23:24.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:23:24 smithi189 ceph-mon[57022]: pgmap v1446: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:23:24.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:23:24 smithi189 ceph-mon[57022]: from='client.36544 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:23:24.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:23:24 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/3388354486' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:23:25.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:23:24 smithi062 ceph-mon[76322]: pgmap v1446: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:23:25.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:23:24 smithi062 ceph-mon[76322]: from='client.36544 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:23:25.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:23:24 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/3388354486' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:23:26.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:23:26 smithi189 ceph-mon[57022]: pgmap v1447: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:23:27.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:23:26 smithi062 ceph-mon[76322]: pgmap v1447: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:23:28.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:23:28 smithi189 ceph-mon[57022]: pgmap v1448: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:23:29.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:23:28 smithi062 ceph-mon[76322]: pgmap v1448: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:23:30.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:23:30 smithi189 ceph-mon[57022]: pgmap v1449: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:23:31.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:23:30 smithi062 ceph-mon[76322]: pgmap v1449: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:23:32.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:23:32 smithi189 ceph-mon[57022]: pgmap v1450: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:23:33.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:23:32 smithi062 ceph-mon[76322]: pgmap v1450: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:23:34.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:23:34 smithi189 ceph-mon[57022]: pgmap v1451: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:23:34.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:23:34 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:23:35.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:23:34 smithi062 ceph-mon[76322]: pgmap v1451: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:23:35.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:23:34 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:23:37.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:23:36 smithi062 ceph-mon[76322]: pgmap v1452: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:23:37.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:23:36 smithi189 ceph-mon[57022]: pgmap v1452: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:23:39.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:23:38 smithi062 ceph-mon[76322]: pgmap v1453: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:23:39.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:23:38 smithi189 ceph-mon[57022]: pgmap v1453: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:23:41.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:23:40 smithi062 ceph-mon[76322]: pgmap v1454: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:23:41.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:23:40 smithi189 ceph-mon[57022]: pgmap v1454: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:23:43.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:23:42 smithi062 ceph-mon[76322]: pgmap v1455: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:23:43.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:23:42 smithi189 ceph-mon[57022]: pgmap v1455: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:23:45.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:23:44 smithi062 ceph-mon[76322]: pgmap v1456: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:23:45.040 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:23:44 smithi189 ceph-mon[57022]: pgmap v1456: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:23:47.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:23:46 smithi062 ceph-mon[76322]: pgmap v1457: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:23:47.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:23:46 smithi189 ceph-mon[57022]: pgmap v1457: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:23:49.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:23:48 smithi062 ceph-mon[76322]: pgmap v1458: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:23:49.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:23:48 smithi189 ceph-mon[57022]: pgmap v1458: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:23:50.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:23:49 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:23:50.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:23:49 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:23:51.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:23:50 smithi189 ceph-mon[57022]: pgmap v1459: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:23:51.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:23:50 smithi062 ceph-mon[76322]: pgmap v1459: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:23:53.087 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:23:52 smithi189 ceph-mon[57022]: pgmap v1460: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:23:53.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:23:52 smithi062 ceph-mon[76322]: pgmap v1460: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:23:54.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:54.823+0000 7f951f400700 1 -- 172.21.15.62:0/2945719142 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9518102460 msgr2=0x7f95181028d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:23:54.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:54.823+0000 7f951f400700 1 --2- 172.21.15.62:0/2945719142 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9518102460 0x7f95181028d0 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f9504009910 tx=0x7f9504009c20 comp rx=0 tx=0).stop 2024-04-20T22:23:54.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:54.823+0000 7f951f400700 1 -- 172.21.15.62:0/2945719142 shutdown_connections 2024-04-20T22:23:54.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:54.823+0000 7f951f400700 1 --2- 172.21.15.62:0/2945719142 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9518102460 0x7f95181028d0 unknown :-1 s=CLOSED pgs=1239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:23:54.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:54.823+0000 7f951f400700 1 --2- 172.21.15.62:0/2945719142 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9518108460 0x7f9518108830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:23:54.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:54.823+0000 7f951f400700 1 -- 172.21.15.62:0/2945719142 >> 172.21.15.62:0/2945719142 conn(0x7f95180fdf60 msgr2=0x7f9518100370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:23:54.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:54.824+0000 7f951f400700 1 -- 172.21.15.62:0/2945719142 shutdown_connections 2024-04-20T22:23:54.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:54.824+0000 7f951f400700 1 -- 172.21.15.62:0/2945719142 wait complete. 2024-04-20T22:23:54.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:54.825+0000 7f951f400700 1 Processor -- start 2024-04-20T22:23:54.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:54.825+0000 7f951f400700 1 -- start start 2024-04-20T22:23:54.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:54.825+0000 7f951f400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9518102460 0x7f95181981d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:23:54.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:54.825+0000 7f951f400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9518108460 0x7f9518198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:23:54.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:54.825+0000 7f951f400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9518198df0 con 0x7f9518102460 2024-04-20T22:23:54.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:54.825+0000 7f951f400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f951819cb30 con 0x7f9518108460 2024-04-20T22:23:54.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:54.826+0000 7f951ca00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9518102460 0x7f95181981d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:23:54.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:54.826+0000 7f951ca00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9518102460 0x7f95181981d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47522/0 (socket says 172.21.15.62:47522) 2024-04-20T22:23:54.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:54.826+0000 7f951ca00700 1 -- 172.21.15.62:0/2405548628 learned_addr learned my addr 172.21.15.62:0/2405548628 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:23:54.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:54.826+0000 7f9517e00700 1 --2- 172.21.15.62:0/2405548628 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9518108460 0x7f9518198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:23:54.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:54.826+0000 7f951ca00700 1 -- 172.21.15.62:0/2405548628 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9518108460 msgr2=0x7f9518198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:23:54.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:54.826+0000 7f951ca00700 1 --2- 172.21.15.62:0/2405548628 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9518108460 0x7f9518198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:23:54.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:54.826+0000 7f951ca00700 1 -- 172.21.15.62:0/2405548628 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95040095f0 con 0x7f9518102460 2024-04-20T22:23:54.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:54.827+0000 7f951ca00700 1 --2- 172.21.15.62:0/2405548628 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9518102460 0x7f95181981d0 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f950c00b4b0 tx=0x7f950c00b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:23:54.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:54.827+0000 7f9515600700 1 -- 172.21.15.62:0/2405548628 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f950c011800 con 0x7f9518102460 2024-04-20T22:23:54.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:54.827+0000 7f9515600700 1 -- 172.21.15.62:0/2405548628 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f950c011e20 con 0x7f9518102460 2024-04-20T22:23:54.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:54.827+0000 7f951f400700 1 -- 172.21.15.62:0/2405548628 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f951819ce10 con 0x7f9518102460 2024-04-20T22:23:54.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:54.827+0000 7f9515600700 1 -- 172.21.15.62:0/2405548628 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f950c00f2e0 con 0x7f9518102460 2024-04-20T22:23:54.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:54.827+0000 7f951f400700 1 -- 172.21.15.62:0/2405548628 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f951819d360 con 0x7f9518102460 2024-04-20T22:23:54.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:54.828+0000 7f951f400700 1 -- 172.21.15.62:0/2405548628 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f951810a9f0 con 0x7f9518102460 2024-04-20T22:23:54.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:54.829+0000 7f9515600700 1 -- 172.21.15.62:0/2405548628 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f950c010380 con 0x7f9518102460 2024-04-20T22:23:54.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:54.831+0000 7f9515600700 1 --2- 172.21.15.62:0/2405548628 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f950006ecc0 0x7f9500071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:23:54.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:54.831+0000 7f9515600700 1 -- 172.21.15.62:0/2405548628 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f950c08ea30 con 0x7f9518102460 2024-04-20T22:23:54.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:54.831+0000 7f9517e00700 1 --2- 172.21.15.62:0/2405548628 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f950006ecc0 0x7f9500071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:23:54.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:54.832+0000 7f9517e00700 1 --2- 172.21.15.62:0/2405548628 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f950006ecc0 0x7f9500071170 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f9504005f50 tx=0x7f950401a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:23:54.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:54.836+0000 7f9515600700 1 -- 172.21.15.62:0/2405548628 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f950c057870 con 0x7f9518102460 2024-04-20T22:23:55.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.092+0000 7f951f400700 1 -- 172.21.15.62:0/2405548628 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f951819d630 con 0x7f950006ecc0 2024-04-20T22:23:55.093 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:23:54 smithi062 ceph-mon[76322]: pgmap v1461: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:23:55.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.095+0000 7f9515600700 1 -- 172.21.15.62:0/2405548628 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f951819d630 con 0x7f950006ecc0 2024-04-20T22:23:55.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.098+0000 7f951f400700 1 -- 172.21.15.62:0/2405548628 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f950006ecc0 msgr2=0x7f9500071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:23:55.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.098+0000 7f951f400700 1 --2- 172.21.15.62:0/2405548628 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f950006ecc0 0x7f9500071170 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f9504005f50 tx=0x7f950401a040 comp rx=0 tx=0).stop 2024-04-20T22:23:55.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.098+0000 7f951f400700 1 -- 172.21.15.62:0/2405548628 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9518102460 msgr2=0x7f95181981d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:23:55.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.098+0000 7f951f400700 1 --2- 172.21.15.62:0/2405548628 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9518102460 0x7f95181981d0 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f950c00b4b0 tx=0x7f950c00b870 comp rx=0 tx=0).stop 2024-04-20T22:23:55.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.098+0000 7f951f400700 1 -- 172.21.15.62:0/2405548628 shutdown_connections 2024-04-20T22:23:55.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.098+0000 7f951f400700 1 --2- 172.21.15.62:0/2405548628 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f950006ecc0 0x7f9500071170 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:23:55.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.098+0000 7f951f400700 1 --2- 172.21.15.62:0/2405548628 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9518102460 0x7f95181981d0 unknown :-1 s=CLOSED pgs=1240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:23:55.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.098+0000 7f951f400700 1 --2- 172.21.15.62:0/2405548628 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9518108460 0x7f9518198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:23:55.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.098+0000 7f951f400700 1 -- 172.21.15.62:0/2405548628 >> 172.21.15.62:0/2405548628 conn(0x7f95180fdf60 msgr2=0x7f95180feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:23:55.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.098+0000 7f951f400700 1 -- 172.21.15.62:0/2405548628 shutdown_connections 2024-04-20T22:23:55.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.098+0000 7f951f400700 1 -- 172.21.15.62:0/2405548628 wait complete. 2024-04-20T22:23:55.115 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T22:23:55.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:23:54 smithi189 ceph-mon[57022]: pgmap v1461: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:23:55.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.268+0000 7f6b84800700 1 -- 172.21.15.62:0/1898198327 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6b7c108470 msgr2=0x7f6b7c108840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:23:55.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.268+0000 7f6b84800700 1 --2- 172.21.15.62:0/1898198327 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6b7c108470 0x7f6b7c108840 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7f6b70009910 tx=0x7f6b70009c20 comp rx=0 tx=0).stop 2024-04-20T22:23:55.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.268+0000 7f6b84800700 1 -- 172.21.15.62:0/1898198327 shutdown_connections 2024-04-20T22:23:55.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.268+0000 7f6b84800700 1 --2- 172.21.15.62:0/1898198327 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6b7c102470 0x7f6b7c1028e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:23:55.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.268+0000 7f6b84800700 1 --2- 172.21.15.62:0/1898198327 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6b7c108470 0x7f6b7c108840 unknown :-1 s=CLOSED pgs=1241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:23:55.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.268+0000 7f6b84800700 1 -- 172.21.15.62:0/1898198327 >> 172.21.15.62:0/1898198327 conn(0x7f6b7c0fdf90 msgr2=0x7f6b7c1003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:23:55.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.268+0000 7f6b84800700 1 -- 172.21.15.62:0/1898198327 shutdown_connections 2024-04-20T22:23:55.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.268+0000 7f6b84800700 1 -- 172.21.15.62:0/1898198327 wait complete. 2024-04-20T22:23:55.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.269+0000 7f6b84800700 1 Processor -- start 2024-04-20T22:23:55.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.269+0000 7f6b84800700 1 -- start start 2024-04-20T22:23:55.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.270+0000 7f6b84800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6b7c102470 0x7f6b7c1981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:23:55.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.270+0000 7f6b84800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6b7c108470 0x7f6b7c198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:23:55.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.270+0000 7f6b84800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b7c198d80 con 0x7f6b7c102470 2024-04-20T22:23:55.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.270+0000 7f6b84800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b7c198ec0 con 0x7f6b7c108470 2024-04-20T22:23:55.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.270+0000 7f6b81e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6b7c102470 0x7f6b7c1981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:23:55.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.270+0000 7f6b81e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6b7c102470 0x7f6b7c1981f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47542/0 (socket says 172.21.15.62:47542) 2024-04-20T22:23:55.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.270+0000 7f6b81e00700 1 -- 172.21.15.62:0/2424140120 learned_addr learned my addr 172.21.15.62:0/2424140120 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:23:55.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.271+0000 7f6b81400700 1 --2- 172.21.15.62:0/2424140120 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6b7c108470 0x7f6b7c198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:23:55.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.271+0000 7f6b81e00700 1 -- 172.21.15.62:0/2424140120 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6b7c108470 msgr2=0x7f6b7c198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:23:55.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.271+0000 7f6b81e00700 1 --2- 172.21.15.62:0/2424140120 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6b7c108470 0x7f6b7c198730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:23:55.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.271+0000 7f6b81e00700 1 -- 172.21.15.62:0/2424140120 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b700095f0 con 0x7f6b7c102470 2024-04-20T22:23:55.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.271+0000 7f6b81e00700 1 --2- 172.21.15.62:0/2424140120 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6b7c102470 0x7f6b7c1981f0 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f6b7000b5c0 tx=0x7f6b7000be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:23:55.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.271+0000 7f6b76a00700 1 -- 172.21.15.62:0/2424140120 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6b7001d070 con 0x7f6b7c102470 2024-04-20T22:23:55.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.271+0000 7f6b76a00700 1 -- 172.21.15.62:0/2424140120 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6b7000f460 con 0x7f6b7c102470 2024-04-20T22:23:55.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.271+0000 7f6b84800700 1 -- 172.21.15.62:0/2424140120 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b7c19ccb0 con 0x7f6b7c102470 2024-04-20T22:23:55.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.271+0000 7f6b76a00700 1 -- 172.21.15.62:0/2424140120 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6b70017610 con 0x7f6b7c102470 2024-04-20T22:23:55.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.272+0000 7f6b84800700 1 -- 172.21.15.62:0/2424140120 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b7c19d1a0 con 0x7f6b7c102470 2024-04-20T22:23:55.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.273+0000 7f6b84800700 1 -- 172.21.15.62:0/2424140120 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b7c10a9e0 con 0x7f6b7c102470 2024-04-20T22:23:55.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.274+0000 7f6b76a00700 1 -- 172.21.15.62:0/2424140120 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f6b7000f5d0 con 0x7f6b7c102470 2024-04-20T22:23:55.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.275+0000 7f6b76a00700 1 --2- 172.21.15.62:0/2424140120 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6b6406ecc0 0x7f6b64071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:23:55.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.275+0000 7f6b76a00700 1 -- 172.21.15.62:0/2424140120 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f6b70090450 con 0x7f6b7c102470 2024-04-20T22:23:55.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.275+0000 7f6b81400700 1 --2- 172.21.15.62:0/2424140120 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6b6406ecc0 0x7f6b64071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:23:55.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.276+0000 7f6b81400700 1 --2- 172.21.15.62:0/2424140120 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6b6406ecc0 0x7f6b64071170 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f6b6c005ea0 tx=0x7f6b6c005e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:23:55.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.281+0000 7f6b76a00700 1 -- 172.21.15.62:0/2424140120 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f6b700591e0 con 0x7f6b7c102470 2024-04-20T22:23:55.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.537+0000 7f6b84800700 1 -- 172.21.15.62:0/2424140120 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6b7c19d410 con 0x7f6b6406ecc0 2024-04-20T22:23:55.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.539+0000 7f6b76a00700 1 -- 172.21.15.62:0/2424140120 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f6b7c19d410 con 0x7f6b6406ecc0 2024-04-20T22:23:55.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.542+0000 7f6b84800700 1 -- 172.21.15.62:0/2424140120 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6b6406ecc0 msgr2=0x7f6b64071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:23:55.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.542+0000 7f6b84800700 1 --2- 172.21.15.62:0/2424140120 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6b6406ecc0 0x7f6b64071170 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f6b6c005ea0 tx=0x7f6b6c005e30 comp rx=0 tx=0).stop 2024-04-20T22:23:55.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.542+0000 7f6b84800700 1 -- 172.21.15.62:0/2424140120 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6b7c102470 msgr2=0x7f6b7c1981f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:23:55.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.542+0000 7f6b84800700 1 --2- 172.21.15.62:0/2424140120 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6b7c102470 0x7f6b7c1981f0 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f6b7000b5c0 tx=0x7f6b7000be00 comp rx=0 tx=0).stop 2024-04-20T22:23:55.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.542+0000 7f6b84800700 1 -- 172.21.15.62:0/2424140120 shutdown_connections 2024-04-20T22:23:55.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.542+0000 7f6b84800700 1 --2- 172.21.15.62:0/2424140120 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6b6406ecc0 0x7f6b64071170 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:23:55.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.542+0000 7f6b84800700 1 --2- 172.21.15.62:0/2424140120 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6b7c102470 0x7f6b7c1981f0 unknown :-1 s=CLOSED pgs=1242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:23:55.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.542+0000 7f6b84800700 1 --2- 172.21.15.62:0/2424140120 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6b7c108470 0x7f6b7c198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:23:55.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.542+0000 7f6b84800700 1 -- 172.21.15.62:0/2424140120 >> 172.21.15.62:0/2424140120 conn(0x7f6b7c0fdf90 msgr2=0x7f6b7c1002c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:23:55.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.542+0000 7f6b84800700 1 -- 172.21.15.62:0/2424140120 shutdown_connections 2024-04-20T22:23:55.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.543+0000 7f6b84800700 1 -- 172.21.15.62:0/2424140120 wait complete. 2024-04-20T22:23:55.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.713+0000 7fbbfa400700 1 -- 172.21.15.62:0/33769008 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbbf4106b80 msgr2=0x7fbbf4106ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:23:55.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.713+0000 7fbbfa400700 1 --2- 172.21.15.62:0/33769008 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbbf4106b80 0x7fbbf4106ff0 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7fbbdc009910 tx=0x7fbbdc009c20 comp rx=0 tx=0).stop 2024-04-20T22:23:55.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.714+0000 7fbbfa400700 1 -- 172.21.15.62:0/33769008 shutdown_connections 2024-04-20T22:23:55.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.714+0000 7fbbfa400700 1 --2- 172.21.15.62:0/33769008 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbbf4106b80 0x7fbbf4106ff0 unknown :-1 s=CLOSED pgs=1243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:23:55.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.714+0000 7fbbfa400700 1 --2- 172.21.15.62:0/33769008 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbbf410c8f0 0x7fbbf410ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:23:55.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.714+0000 7fbbfa400700 1 -- 172.21.15.62:0/33769008 >> 172.21.15.62:0/33769008 conn(0x7fbbf4076060 msgr2=0x7fbbf4078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:23:55.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.714+0000 7fbbfa400700 1 -- 172.21.15.62:0/33769008 shutdown_connections 2024-04-20T22:23:55.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.714+0000 7fbbfa400700 1 -- 172.21.15.62:0/33769008 wait complete. 2024-04-20T22:23:55.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.715+0000 7fbbfa400700 1 Processor -- start 2024-04-20T22:23:55.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.715+0000 7fbbfa400700 1 -- start start 2024-04-20T22:23:55.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.716+0000 7fbbfa400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbbf4106b80 0x7fbbf419c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:23:55.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.716+0000 7fbbfa400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbbf410c8f0 0x7fbbf419c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:23:55.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.716+0000 7fbbfa400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbf419d060 con 0x7fbbf410c8f0 2024-04-20T22:23:55.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.716+0000 7fbbfa400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbf41a0df0 con 0x7fbbf4106b80 2024-04-20T22:23:55.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.716+0000 7fbbebe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbbf410c8f0 0x7fbbf419c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:23:55.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.716+0000 7fbbebe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbbf410c8f0 0x7fbbf419c980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47564/0 (socket says 172.21.15.62:47564) 2024-04-20T22:23:55.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.716+0000 7fbbebe00700 1 -- 172.21.15.62:0/3016087084 learned_addr learned my addr 172.21.15.62:0/3016087084 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:23:55.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.717+0000 7fbbf3400700 1 --2- 172.21.15.62:0/3016087084 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbbf4106b80 0x7fbbf419c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:23:55.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.717+0000 7fbbebe00700 1 -- 172.21.15.62:0/3016087084 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbbf4106b80 msgr2=0x7fbbf419c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:23:55.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.717+0000 7fbbebe00700 1 --2- 172.21.15.62:0/3016087084 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbbf4106b80 0x7fbbf419c440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:23:55.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.717+0000 7fbbebe00700 1 -- 172.21.15.62:0/3016087084 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbdc0095f0 con 0x7fbbf410c8f0 2024-04-20T22:23:55.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.717+0000 7fbbf3400700 1 --2- 172.21.15.62:0/3016087084 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbbf4106b80 0x7fbbf419c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-20T22:23:55.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.717+0000 7fbbebe00700 1 --2- 172.21.15.62:0/3016087084 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbbf410c8f0 0x7fbbf419c980 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7fbbdc00b5c0 tx=0x7fbbdc00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:23:55.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.717+0000 7fbbf0c00700 1 -- 172.21.15.62:0/3016087084 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbbdc01d070 con 0x7fbbf410c8f0 2024-04-20T22:23:55.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.717+0000 7fbbf0c00700 1 -- 172.21.15.62:0/3016087084 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbbdc005e00 con 0x7fbbf410c8f0 2024-04-20T22:23:55.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.717+0000 7fbbf0c00700 1 -- 172.21.15.62:0/3016087084 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbbdc017400 con 0x7fbbf410c8f0 2024-04-20T22:23:55.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.717+0000 7fbbfa400700 1 -- 172.21.15.62:0/3016087084 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbf41a1070 con 0x7fbbf410c8f0 2024-04-20T22:23:55.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.718+0000 7fbbfa400700 1 -- 172.21.15.62:0/3016087084 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbf41a1560 con 0x7fbbf410c8f0 2024-04-20T22:23:55.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.719+0000 7fbbfa400700 1 -- 172.21.15.62:0/3016087084 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbf410ecc0 con 0x7fbbf410c8f0 2024-04-20T22:23:55.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.720+0000 7fbbf0c00700 1 -- 172.21.15.62:0/3016087084 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fbbdc00f460 con 0x7fbbf410c8f0 2024-04-20T22:23:55.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.721+0000 7fbbf0c00700 1 --2- 172.21.15.62:0/3016087084 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fbbd806ecc0 0x7fbbd8071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:23:55.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.722+0000 7fbbf0c00700 1 -- 172.21.15.62:0/3016087084 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fbbdc090370 con 0x7fbbf410c8f0 2024-04-20T22:23:55.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.722+0000 7fbbf3400700 1 --2- 172.21.15.62:0/3016087084 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fbbd806ecc0 0x7fbbd8071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:23:55.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.723+0000 7fbbf3400700 1 --2- 172.21.15.62:0/3016087084 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fbbd806ecc0 0x7fbbd8071170 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fbbe4005950 tx=0x7fbbe40058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:23:55.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.727+0000 7fbbf0c00700 1 -- 172.21.15.62:0/3016087084 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fbbdc059100 con 0x7fbbf410c8f0 2024-04-20T22:23:55.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.975+0000 7fbbfa400700 1 -- 172.21.15.62:0/3016087084 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fbbf402cca0 con 0x7fbbd806ecc0 2024-04-20T22:23:55.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.988+0000 7fbbf0c00700 1 -- 172.21.15.62:0/3016087084 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fbbf402cca0 con 0x7fbbd806ecc0 2024-04-20T22:23:55.989 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T22:23:55.989 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (49m) 10m ago 63m 14.6M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T22:23:55.989 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (63m) 10m ago 63m 11.6M - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T22:23:55.989 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (62m) 10m ago 62m 11.2M - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T22:23:55.989 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (45m) 10m ago 63m 6660k - 19.0.0-3133-ga8c84f59 76bc6b91b726 2825d5ba11c6 2024-04-20T22:23:55.989 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (45m) 10m ago 62m 6665k - 19.0.0-3133-ga8c84f59 76bc6b91b726 20f746fa1bba 2024-04-20T22:23:55.989 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (49m) 10m ago 63m 75.5M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T22:23:55.989 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (41m) 10m ago 56m 22.9M - 19.0.0-3133-ga8c84f59 76bc6b91b726 de80bc90a683 2024-04-20T22:23:55.989 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (41m) 10m ago 56m 23.0M - 19.0.0-3133-ga8c84f59 76bc6b91b726 cd793b7ee98e 2024-04-20T22:23:55.989 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (26m) 10m ago 56m 18.5M - 19.0.0-3133-ga8c84f59 76bc6b91b726 0e7b0bc4d1ab 2024-04-20T22:23:55.990 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (10m) 10m ago 56m 11.4M - 19.0.0-3133-ga8c84f59 76bc6b91b726 0af732ac9e06 2024-04-20T22:23:55.990 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (52m) 10m ago 65m 517M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T22:23:55.990 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (51m) 10m ago 62m 460M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T22:23:55.990 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (47m) 10m ago 65m 86.3M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 019eb1479485 2024-04-20T22:23:55.990 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (46m) 10m ago 62m 58.0M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 55aac560aab6 2024-04-20T22:23:55.990 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (50m) 10m ago 63m 8766k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T22:23:55.990 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (50m) 10m ago 62m 8711k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T22:23:55.990 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (45m) 10m ago 61m 91.3M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 ee2bb901f26e 2024-04-20T22:23:55.990 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (44m) 10m ago 60m 94.9M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 9075152d4162 2024-04-20T22:23:55.990 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (44m) 10m ago 60m 78.5M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 2d4f3e38dc0e 2024-04-20T22:23:55.990 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (43m) 10m ago 59m 94.1M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 8f97b17ae0eb 2024-04-20T22:23:55.990 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (42m) 10m ago 58m 96.1M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 653d8079967d 2024-04-20T22:23:55.990 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (42m) 10m ago 58m 90.9M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 596dded14c12 2024-04-20T22:23:55.990 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (50m) 10m ago 63m 90.4M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T22:23:55.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.991+0000 7fbbfa400700 1 -- 172.21.15.62:0/3016087084 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fbbd806ecc0 msgr2=0x7fbbd8071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:23:55.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.991+0000 7fbbfa400700 1 --2- 172.21.15.62:0/3016087084 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fbbd806ecc0 0x7fbbd8071170 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fbbe4005950 tx=0x7fbbe40058e0 comp rx=0 tx=0).stop 2024-04-20T22:23:55.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.991+0000 7fbbfa400700 1 -- 172.21.15.62:0/3016087084 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbbf410c8f0 msgr2=0x7fbbf419c980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:23:55.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.991+0000 7fbbfa400700 1 --2- 172.21.15.62:0/3016087084 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbbf410c8f0 0x7fbbf419c980 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7fbbdc00b5c0 tx=0x7fbbdc00be20 comp rx=0 tx=0).stop 2024-04-20T22:23:55.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.992+0000 7fbbfa400700 1 -- 172.21.15.62:0/3016087084 shutdown_connections 2024-04-20T22:23:55.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.992+0000 7fbbfa400700 1 --2- 172.21.15.62:0/3016087084 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fbbd806ecc0 0x7fbbd8071170 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:23:55.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.992+0000 7fbbfa400700 1 --2- 172.21.15.62:0/3016087084 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbbf4106b80 0x7fbbf419c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:23:55.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.992+0000 7fbbfa400700 1 --2- 172.21.15.62:0/3016087084 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbbf410c8f0 0x7fbbf419c980 unknown :-1 s=CLOSED pgs=1244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:23:55.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.992+0000 7fbbfa400700 1 -- 172.21.15.62:0/3016087084 >> 172.21.15.62:0/3016087084 conn(0x7fbbf4076060 msgr2=0x7fbbf4077ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:23:55.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.992+0000 7fbbfa400700 1 -- 172.21.15.62:0/3016087084 shutdown_connections 2024-04-20T22:23:55.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:55.992+0000 7fbbfa400700 1 -- 172.21.15.62:0/3016087084 wait complete. 2024-04-20T22:23:56.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:56.155+0000 7f2c2b200700 1 -- 172.21.15.62:0/421788005 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c2c10c660 msgr2=0x7f2c2c10ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:23:56.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:56.155+0000 7f2c2b200700 1 --2- 172.21.15.62:0/421788005 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c2c10c660 0x7f2c2c10ca30 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f2c1c009910 tx=0x7f2c1c009c20 comp rx=0 tx=0).stop 2024-04-20T22:23:56.156 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:23:55 smithi062 ceph-mon[76322]: from='client.36552 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:23:56.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:56.156+0000 7f2c2b200700 1 -- 172.21.15.62:0/421788005 shutdown_connections 2024-04-20T22:23:56.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:56.156+0000 7f2c2b200700 1 --2- 172.21.15.62:0/421788005 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c2c106b80 0x7f2c2c106ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:23:56.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:56.156+0000 7f2c2b200700 1 --2- 172.21.15.62:0/421788005 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c2c10c660 0x7f2c2c10ca30 unknown :-1 s=CLOSED pgs=1245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:23:56.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:56.156+0000 7f2c2b200700 1 -- 172.21.15.62:0/421788005 >> 172.21.15.62:0/421788005 conn(0x7f2c2c075e90 msgr2=0x7f2c2c0782a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:23:56.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:56.156+0000 7f2c2b200700 1 -- 172.21.15.62:0/421788005 shutdown_connections 2024-04-20T22:23:56.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:56.156+0000 7f2c2b200700 1 -- 172.21.15.62:0/421788005 wait complete. 2024-04-20T22:23:56.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:56.157+0000 7f2c2b200700 1 Processor -- start 2024-04-20T22:23:56.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:56.157+0000 7f2c2b200700 1 -- start start 2024-04-20T22:23:56.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:56.158+0000 7f2c2b200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c2c106b80 0x7f2c2c19c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:23:56.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:56.158+0000 7f2c2b200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c2c10c660 0x7f2c2c19c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:23:56.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:56.158+0000 7f2c2b200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c2c19cea0 con 0x7f2c2c106b80 2024-04-20T22:23:56.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:56.158+0000 7f2c2b200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c2c1a0c30 con 0x7f2c2c10c660 2024-04-20T22:23:56.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:56.158+0000 7f2c29e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c2c106b80 0x7f2c2c19c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:23:56.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:56.158+0000 7f2c29e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c2c106b80 0x7f2c2c19c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47572/0 (socket says 172.21.15.62:47572) 2024-04-20T22:23:56.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:56.158+0000 7f2c29e00700 1 -- 172.21.15.62:0/1427374370 learned_addr learned my addr 172.21.15.62:0/1427374370 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:23:56.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:56.158+0000 7f2c29400700 1 --2- 172.21.15.62:0/1427374370 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c2c10c660 0x7f2c2c19c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:23:56.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:56.159+0000 7f2c29e00700 1 -- 172.21.15.62:0/1427374370 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c2c10c660 msgr2=0x7f2c2c19c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:23:56.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:56.159+0000 7f2c29e00700 1 --2- 172.21.15.62:0/1427374370 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c2c10c660 0x7f2c2c19c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:23:56.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:56.159+0000 7f2c29e00700 1 -- 172.21.15.62:0/1427374370 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c1c0095f0 con 0x7f2c2c106b80 2024-04-20T22:23:56.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:56.159+0000 7f2c29400700 1 --2- 172.21.15.62:0/1427374370 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c2c10c660 0x7f2c2c19c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-20T22:23:56.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:56.159+0000 7f2c29e00700 1 --2- 172.21.15.62:0/1427374370 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c2c106b80 0x7f2c2c19c280 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f2c1c00b5c0 tx=0x7f2c1c00be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:23:56.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:56.160+0000 7f2c22a00700 1 -- 172.21.15.62:0/1427374370 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2c1c01d070 con 0x7f2c2c106b80 2024-04-20T22:23:56.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:56.160+0000 7f2c22a00700 1 -- 172.21.15.62:0/1427374370 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2c1c00f460 con 0x7f2c2c106b80 2024-04-20T22:23:56.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:56.160+0000 7f2c22a00700 1 -- 172.21.15.62:0/1427374370 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2c1c017610 con 0x7f2c2c106b80 2024-04-20T22:23:56.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:56.160+0000 7f2c2b200700 1 -- 172.21.15.62:0/1427374370 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c2c1a0eb0 con 0x7f2c2c106b80 2024-04-20T22:23:56.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:56.160+0000 7f2c2b200700 1 -- 172.21.15.62:0/1427374370 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c2c1a13a0 con 0x7f2c2c106b80 2024-04-20T22:23:56.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:56.161+0000 7f2c2b200700 1 -- 172.21.15.62:0/1427374370 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c2c10eb60 con 0x7f2c2c106b80 2024-04-20T22:23:56.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:56.163+0000 7f2c22a00700 1 -- 172.21.15.62:0/1427374370 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f2c1c00f5d0 con 0x7f2c2c106b80 2024-04-20T22:23:56.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:56.164+0000 7f2c22a00700 1 --2- 172.21.15.62:0/1427374370 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2c1006ecc0 0x7f2c10071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:23:56.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:56.164+0000 7f2c22a00700 1 -- 172.21.15.62:0/1427374370 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f2c1c090460 con 0x7f2c2c106b80 2024-04-20T22:23:56.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:56.165+0000 7f2c29400700 1 --2- 172.21.15.62:0/1427374370 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2c1006ecc0 0x7f2c10071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:23:56.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:56.165+0000 7f2c29400700 1 --2- 172.21.15.62:0/1427374370 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2c1006ecc0 0x7f2c10071170 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f2c2c19d8a0 tx=0x7f2c18005f50 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:23:56.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:56.169+0000 7f2c22a00700 1 -- 172.21.15.62:0/1427374370 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f2c1c0591f0 con 0x7f2c2c106b80 2024-04-20T22:23:56.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:23:55 smithi189 ceph-mon[57022]: from='client.36552 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:23:56.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:56.506+0000 7f2c2b200700 1 -- 172.21.15.62:0/1427374370 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f2c2c062380 con 0x7f2c2c106b80 2024-04-20T22:23:56.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:56.507+0000 7f2c22a00700 1 -- 172.21.15.62:0/1427374370 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f2c1c05c8b0 con 0x7f2c2c106b80 2024-04-20T22:23:56.508 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:23:56.508 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T22:23:56.508 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:23:56.508 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:23:56.508 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T22:23:56.508 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:23:56.508 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:23:56.508 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T22:23:56.508 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 6 2024-04-20T22:23:56.508 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:23:56.509 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T22:23:56.509 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 4 2024-04-20T22:23:56.509 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:23:56.509 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T22:23:56.509 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 14 2024-04-20T22:23:56.509 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T22:23:56.509 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:23:56.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:56.510+0000 7f2c2b200700 1 -- 172.21.15.62:0/1427374370 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2c1006ecc0 msgr2=0x7f2c10071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:23:56.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:56.510+0000 7f2c2b200700 1 --2- 172.21.15.62:0/1427374370 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2c1006ecc0 0x7f2c10071170 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f2c2c19d8a0 tx=0x7f2c18005f50 comp rx=0 tx=0).stop 2024-04-20T22:23:56.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:56.510+0000 7f2c2b200700 1 -- 172.21.15.62:0/1427374370 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c2c106b80 msgr2=0x7f2c2c19c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:23:56.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:56.510+0000 7f2c2b200700 1 --2- 172.21.15.62:0/1427374370 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c2c106b80 0x7f2c2c19c280 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f2c1c00b5c0 tx=0x7f2c1c00be00 comp rx=0 tx=0).stop 2024-04-20T22:23:56.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:56.510+0000 7f2c2b200700 1 -- 172.21.15.62:0/1427374370 shutdown_connections 2024-04-20T22:23:56.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:56.510+0000 7f2c2b200700 1 --2- 172.21.15.62:0/1427374370 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2c1006ecc0 0x7f2c10071170 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:23:56.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:56.510+0000 7f2c2b200700 1 --2- 172.21.15.62:0/1427374370 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c2c106b80 0x7f2c2c19c280 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:23:56.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:56.510+0000 7f2c2b200700 1 --2- 172.21.15.62:0/1427374370 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c2c10c660 0x7f2c2c19c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:23:56.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:56.510+0000 7f2c2b200700 1 -- 172.21.15.62:0/1427374370 >> 172.21.15.62:0/1427374370 conn(0x7f2c2c075e90 msgr2=0x7f2c2c077980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:23:56.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:56.510+0000 7f2c2b200700 1 -- 172.21.15.62:0/1427374370 shutdown_connections 2024-04-20T22:23:56.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:56.511+0000 7f2c2b200700 1 -- 172.21.15.62:0/1427374370 wait complete. 2024-04-20T22:23:56.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:56.673+0000 7f0f3b200700 1 -- 172.21.15.62:0/934421123 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0f34073a00 msgr2=0x7f0f34110d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:23:56.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:56.673+0000 7f0f3b200700 1 --2- 172.21.15.62:0/934421123 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0f34073a00 0x7f0f34110d50 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7f0f1c009910 tx=0x7f0f1c009c20 comp rx=0 tx=0).stop 2024-04-20T22:23:56.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:56.673+0000 7f0f3b200700 1 -- 172.21.15.62:0/934421123 shutdown_connections 2024-04-20T22:23:56.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:56.674+0000 7f0f3b200700 1 --2- 172.21.15.62:0/934421123 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0f34073a00 0x7f0f34110d50 unknown :-1 s=CLOSED pgs=1247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:23:56.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:56.674+0000 7f0f3b200700 1 --2- 172.21.15.62:0/934421123 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f340730f0 0x7f0f340734c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:23:56.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:56.674+0000 7f0f3b200700 1 -- 172.21.15.62:0/934421123 >> 172.21.15.62:0/934421123 conn(0x7f0f340fbda0 msgr2=0x7f0f340fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:23:56.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:56.674+0000 7f0f3b200700 1 -- 172.21.15.62:0/934421123 shutdown_connections 2024-04-20T22:23:56.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:56.674+0000 7f0f3b200700 1 -- 172.21.15.62:0/934421123 wait complete. 2024-04-20T22:23:56.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:56.675+0000 7f0f3b200700 1 Processor -- start 2024-04-20T22:23:56.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:56.675+0000 7f0f3b200700 1 -- start start 2024-04-20T22:23:56.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:56.676+0000 7f0f3b200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0f340730f0 0x7f0f3419c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:23:56.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:56.676+0000 7f0f3b200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f34073a00 0x7f0f3419c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:23:56.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:56.676+0000 7f0f3b200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f3419d050 con 0x7f0f340730f0 2024-04-20T22:23:56.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:56.676+0000 7f0f3b200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f341a0de0 con 0x7f0f34073a00 2024-04-20T22:23:56.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:56.676+0000 7f0f33e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0f340730f0 0x7f0f3419c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:23:56.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:56.676+0000 7f0f33e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0f340730f0 0x7f0f3419c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47582/0 (socket says 172.21.15.62:47582) 2024-04-20T22:23:56.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:56.676+0000 7f0f33e00700 1 -- 172.21.15.62:0/4197588485 learned_addr learned my addr 172.21.15.62:0/4197588485 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:23:56.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:56.676+0000 7f0f2be00700 1 --2- 172.21.15.62:0/4197588485 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f34073a00 0x7f0f3419c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:23:56.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:56.677+0000 7f0f33e00700 1 -- 172.21.15.62:0/4197588485 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f34073a00 msgr2=0x7f0f3419c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:23:56.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:56.677+0000 7f0f33e00700 1 --2- 172.21.15.62:0/4197588485 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f34073a00 0x7f0f3419c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:23:56.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:56.677+0000 7f0f33e00700 1 -- 172.21.15.62:0/4197588485 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f1c0095f0 con 0x7f0f340730f0 2024-04-20T22:23:56.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:56.677+0000 7f0f33e00700 1 --2- 172.21.15.62:0/4197588485 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0f340730f0 0x7f0f3419c430 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f0f2400d8d0 tx=0x7f0f2400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:23:56.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:56.677+0000 7f0f31600700 1 -- 172.21.15.62:0/4197588485 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0f24009940 con 0x7f0f340730f0 2024-04-20T22:23:56.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:56.677+0000 7f0f31600700 1 -- 172.21.15.62:0/4197588485 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0f24010460 con 0x7f0f340730f0 2024-04-20T22:23:56.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:56.677+0000 7f0f3b200700 1 -- 172.21.15.62:0/4197588485 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f341a10c0 con 0x7f0f340730f0 2024-04-20T22:23:56.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:56.677+0000 7f0f31600700 1 -- 172.21.15.62:0/4197588485 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0f2400f5f0 con 0x7f0f340730f0 2024-04-20T22:23:56.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:56.677+0000 7f0f3b200700 1 -- 172.21.15.62:0/4197588485 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f341a1610 con 0x7f0f340730f0 2024-04-20T22:23:56.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:56.679+0000 7f0f3b200700 1 -- 172.21.15.62:0/4197588485 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f3410e4d0 con 0x7f0f340730f0 2024-04-20T22:23:56.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:56.680+0000 7f0f31600700 1 -- 172.21.15.62:0/4197588485 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f0f240105d0 con 0x7f0f340730f0 2024-04-20T22:23:56.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:56.681+0000 7f0f31600700 1 --2- 172.21.15.62:0/4197588485 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f0f1806ecc0 0x7f0f18071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:23:56.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:56.681+0000 7f0f31600700 1 -- 172.21.15.62:0/4197588485 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f0f2408ee10 con 0x7f0f340730f0 2024-04-20T22:23:56.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:56.682+0000 7f0f2be00700 1 --2- 172.21.15.62:0/4197588485 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f0f1806ecc0 0x7f0f18071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:23:56.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:56.682+0000 7f0f2be00700 1 --2- 172.21.15.62:0/4197588485 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f0f1806ecc0 0x7f0f18071170 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f0f3419da50 tx=0x7f0f1c01a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:23:56.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:56.686+0000 7f0f31600700 1 -- 172.21.15.62:0/4197588485 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f0f24057ba0 con 0x7f0f340730f0 2024-04-20T22:23:56.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:56.975+0000 7f0f3b200700 1 -- 172.21.15.62:0/4197588485 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f0f341a1880 con 0x7f0f340730f0 2024-04-20T22:23:56.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:56.977+0000 7f0f31600700 1 -- 172.21.15.62:0/4197588485 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 29 v29) v1 ==== 76+0+1887 (secure 0 0 0) 0x7f0f24016770 con 0x7f0f340730f0 2024-04-20T22:23:56.977 INFO:teuthology.orchestra.run.smithi062.stdout:e29 2024-04-20T22:23:56.977 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T22:23:56.977 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:23:56.977 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T22:23:56.977 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:23:56.977 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T22:23:56.977 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T22:23:56.977 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 20 2024-04-20T22:23:56.978 INFO:teuthology.orchestra.run.smithi062.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-20T22:23:56.978 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T22:23:56.978 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T21:42:01.559061+0000 2024-04-20T22:23:56.978 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T22:23:56.978 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T22:23:56.978 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T22:23:56.978 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T22:23:56.978 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T22:23:56.978 INFO:teuthology.orchestra.run.smithi062.stdout:max_xattr_size 65536 2024-04-20T22:23:56.978 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T22:23:56.978 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T22:23:56.978 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 81 2024-04-20T22:23:56.978 INFO:teuthology.orchestra.run.smithi062.stdout:snap_rstat 1 2024-04-20T22:23:56.979 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:23:56.979 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 2 2024-04-20T22:23:56.979 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1 2024-04-20T22:23:56.979 INFO:teuthology.orchestra.run.smithi062.stdout:up {} 2024-04-20T22:23:56.979 INFO:teuthology.orchestra.run.smithi062.stdout:failed 0,1 2024-04-20T22:23:56.979 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T22:23:56.979 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T22:23:56.979 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T22:23:56.979 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T22:23:56.979 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data enabled 2024-04-20T22:23:56.979 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T22:23:56.979 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T22:23:56.979 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 1 2024-04-20T22:23:56.979 INFO:teuthology.orchestra.run.smithi062.stdout:qdb_cluster leader: 0 members: 2024-04-20T22:23:56.980 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:23:56.980 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:23:56.980 INFO:teuthology.orchestra.run.smithi062.stdout:Standby daemons: 2024-04-20T22:23:56.980 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:23:56.980 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ryqblb{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6826/4113916205,v1:172.21.15.62:6827/4113916205] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:23:56.980 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ozizwi{-1:34476} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6828/3801975165,v1:172.21.15.62:6829/3801975165] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:23:56.980 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.dvimer{-1:44727} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6824/3576699232,v1:172.21.15.189:6825/3576699232] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:23:56.980 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.xbgspw{-1:45075} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6826/212905792,v1:172.21.15.189:6827/212905792] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:23:56.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:56.979+0000 7f0f3b200700 1 -- 172.21.15.62:0/4197588485 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f0f1806ecc0 msgr2=0x7f0f18071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:23:56.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:56.979+0000 7f0f3b200700 1 --2- 172.21.15.62:0/4197588485 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f0f1806ecc0 0x7f0f18071170 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f0f3419da50 tx=0x7f0f1c01a040 comp rx=0 tx=0).stop 2024-04-20T22:23:56.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:56.979+0000 7f0f3b200700 1 -- 172.21.15.62:0/4197588485 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0f340730f0 msgr2=0x7f0f3419c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:23:56.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:56.979+0000 7f0f3b200700 1 --2- 172.21.15.62:0/4197588485 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0f340730f0 0x7f0f3419c430 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f0f2400d8d0 tx=0x7f0f2400dc90 comp rx=0 tx=0).stop 2024-04-20T22:23:56.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:56.980+0000 7f0f3b200700 1 -- 172.21.15.62:0/4197588485 shutdown_connections 2024-04-20T22:23:56.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:56.980+0000 7f0f3b200700 1 --2- 172.21.15.62:0/4197588485 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f0f1806ecc0 0x7f0f18071170 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:23:56.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:56.980+0000 7f0f3b200700 1 --2- 172.21.15.62:0/4197588485 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0f340730f0 0x7f0f3419c430 unknown :-1 s=CLOSED pgs=1248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:23:56.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:56.980+0000 7f0f3b200700 1 --2- 172.21.15.62:0/4197588485 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f34073a00 0x7f0f3419c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:23:56.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:56.980+0000 7f0f3b200700 1 -- 172.21.15.62:0/4197588485 >> 172.21.15.62:0/4197588485 conn(0x7f0f340fbda0 msgr2=0x7f0f34102860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:23:56.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:56.980+0000 7f0f3b200700 1 -- 172.21.15.62:0/4197588485 shutdown_connections 2024-04-20T22:23:56.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:56.980+0000 7f0f3b200700 1 -- 172.21.15.62:0/4197588485 wait complete. 2024-04-20T22:23:56.982 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 29 2024-04-20T22:23:57.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:57.149+0000 7f51d2c00700 1 -- 172.21.15.62:0/1363558152 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51cc1004e0 msgr2=0x7f51cc1008b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:23:57.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:57.149+0000 7f51d2c00700 1 --2- 172.21.15.62:0/1363558152 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51cc1004e0 0x7f51cc1008b0 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f51bc009910 tx=0x7f51bc009c20 comp rx=0 tx=0).stop 2024-04-20T22:23:57.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:57.149+0000 7f51d2c00700 1 -- 172.21.15.62:0/1363558152 shutdown_connections 2024-04-20T22:23:57.150 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:23:56 smithi062 ceph-mon[76322]: pgmap v1462: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:23:57.150 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:23:56 smithi062 ceph-mon[76322]: from='client.36556 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:23:57.150 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:23:56 smithi062 ceph-mon[76322]: from='client.36560 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:23:57.150 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:23:56 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/1427374370' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:23:57.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:57.149+0000 7f51d2c00700 1 --2- 172.21.15.62:0/1363558152 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f51cc100e80 0x7f51cc104e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:23:57.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:57.149+0000 7f51d2c00700 1 --2- 172.21.15.62:0/1363558152 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51cc1004e0 0x7f51cc1008b0 unknown :-1 s=CLOSED pgs=1249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:23:57.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:57.149+0000 7f51d2c00700 1 -- 172.21.15.62:0/1363558152 >> 172.21.15.62:0/1363558152 conn(0x7f51cc0fbda0 msgr2=0x7f51cc0fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:23:57.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:57.150+0000 7f51d2c00700 1 -- 172.21.15.62:0/1363558152 shutdown_connections 2024-04-20T22:23:57.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:57.150+0000 7f51d2c00700 1 -- 172.21.15.62:0/1363558152 wait complete. 2024-04-20T22:23:57.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:57.151+0000 7f51d2c00700 1 Processor -- start 2024-04-20T22:23:57.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:57.151+0000 7f51d2c00700 1 -- start start 2024-04-20T22:23:57.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:57.151+0000 7f51d2c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51cc1004e0 0x7f51cc1980f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:23:57.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:57.151+0000 7f51d2c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f51cc100e80 0x7f51cc198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:23:57.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:57.151+0000 7f51d2c00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51cc198d10 con 0x7f51cc1004e0 2024-04-20T22:23:57.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:57.151+0000 7f51d2c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51cc19caa0 con 0x7f51cc100e80 2024-04-20T22:23:57.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:57.152+0000 7f51cbe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51cc1004e0 0x7f51cc1980f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:23:57.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:57.152+0000 7f51cbe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51cc1004e0 0x7f51cc1980f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47598/0 (socket says 172.21.15.62:47598) 2024-04-20T22:23:57.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:57.152+0000 7f51cbe00700 1 -- 172.21.15.62:0/563528599 learned_addr learned my addr 172.21.15.62:0/563528599 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:23:57.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:57.152+0000 7f51cb400700 1 --2- 172.21.15.62:0/563528599 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f51cc100e80 0x7f51cc198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:23:57.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:57.152+0000 7f51cbe00700 1 -- 172.21.15.62:0/563528599 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f51cc100e80 msgr2=0x7f51cc198630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:23:57.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:57.152+0000 7f51cbe00700 1 --2- 172.21.15.62:0/563528599 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f51cc100e80 0x7f51cc198630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:23:57.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:57.152+0000 7f51cbe00700 1 -- 172.21.15.62:0/563528599 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51bc0095f0 con 0x7f51cc1004e0 2024-04-20T22:23:57.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:57.152+0000 7f51cb400700 1 --2- 172.21.15.62:0/563528599 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f51cc100e80 0x7f51cc198630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T22:23:57.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:57.153+0000 7f51cbe00700 1 --2- 172.21.15.62:0/563528599 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51cc1004e0 0x7f51cc1980f0 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7f51bc0092e0 tx=0x7f51bc00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:23:57.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:57.153+0000 7f51c8c00700 1 -- 172.21.15.62:0/563528599 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f51bc01d070 con 0x7f51cc1004e0 2024-04-20T22:23:57.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:57.153+0000 7f51d2c00700 1 -- 172.21.15.62:0/563528599 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51cc19cd20 con 0x7f51cc1004e0 2024-04-20T22:23:57.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:57.153+0000 7f51c8c00700 1 -- 172.21.15.62:0/563528599 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f51bc005e00 con 0x7f51cc1004e0 2024-04-20T22:23:57.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:57.153+0000 7f51d2c00700 1 -- 172.21.15.62:0/563528599 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51cc19d210 con 0x7f51cc1004e0 2024-04-20T22:23:57.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:57.154+0000 7f51c8c00700 1 -- 172.21.15.62:0/563528599 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f51bc017400 con 0x7f51cc1004e0 2024-04-20T22:23:57.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:57.155+0000 7f51d2c00700 1 -- 172.21.15.62:0/563528599 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51cc1045d0 con 0x7f51cc1004e0 2024-04-20T22:23:57.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:57.156+0000 7f51c8c00700 1 -- 172.21.15.62:0/563528599 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f51bc00f460 con 0x7f51cc1004e0 2024-04-20T22:23:57.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:57.157+0000 7f51c8c00700 1 --2- 172.21.15.62:0/563528599 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f51b006ecc0 0x7f51b0071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:23:57.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:57.157+0000 7f51c8c00700 1 -- 172.21.15.62:0/563528599 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f51bc090350 con 0x7f51cc1004e0 2024-04-20T22:23:57.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:57.157+0000 7f51cb400700 1 --2- 172.21.15.62:0/563528599 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f51b006ecc0 0x7f51b0071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:23:57.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:57.158+0000 7f51cb400700 1 --2- 172.21.15.62:0/563528599 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f51b006ecc0 0x7f51b0071170 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f51b8005fd0 tx=0x7f51b8005ee0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:23:57.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:57.162+0000 7f51c8c00700 1 -- 172.21.15.62:0/563528599 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f51bc0590e0 con 0x7f51cc1004e0 2024-04-20T22:23:57.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:23:56 smithi189 ceph-mon[57022]: pgmap v1462: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:23:57.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:23:56 smithi189 ceph-mon[57022]: from='client.36556 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:23:57.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:23:56 smithi189 ceph-mon[57022]: from='client.36560 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:23:57.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:23:56 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/1427374370' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:23:57.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:57.417+0000 7f51d2c00700 1 -- 172.21.15.62:0/563528599 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f51cc19d540 con 0x7f51b006ecc0 2024-04-20T22:23:57.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:57.420+0000 7f51c8c00700 1 -- 172.21.15.62:0/563528599 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f51cc19d540 con 0x7f51b006ecc0 2024-04-20T22:23:57.420 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:23:57.420 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T22:23:57.420 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T22:23:57.420 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-20T22:23:57.420 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T22:23:57.420 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr", 2024-04-20T22:23:57.420 INFO:teuthology.orchestra.run.smithi062.stdout: "osd", 2024-04-20T22:23:57.420 INFO:teuthology.orchestra.run.smithi062.stdout: "mds", 2024-04-20T22:23:57.420 INFO:teuthology.orchestra.run.smithi062.stdout: "crash", 2024-04-20T22:23:57.420 INFO:teuthology.orchestra.run.smithi062.stdout: "mon" 2024-04-20T22:23:57.420 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T22:23:57.420 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "16/23 daemons upgraded", 2024-04-20T22:23:57.420 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading mds daemons", 2024-04-20T22:23:57.420 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T22:23:57.421 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:23:57.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:57.422+0000 7f51d2c00700 1 -- 172.21.15.62:0/563528599 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f51b006ecc0 msgr2=0x7f51b0071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:23:57.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:57.422+0000 7f51d2c00700 1 --2- 172.21.15.62:0/563528599 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f51b006ecc0 0x7f51b0071170 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f51b8005fd0 tx=0x7f51b8005ee0 comp rx=0 tx=0).stop 2024-04-20T22:23:57.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:57.422+0000 7f51d2c00700 1 -- 172.21.15.62:0/563528599 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51cc1004e0 msgr2=0x7f51cc1980f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:23:57.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:57.422+0000 7f51d2c00700 1 --2- 172.21.15.62:0/563528599 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51cc1004e0 0x7f51cc1980f0 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7f51bc0092e0 tx=0x7f51bc00be20 comp rx=0 tx=0).stop 2024-04-20T22:23:57.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:57.423+0000 7f51d2c00700 1 -- 172.21.15.62:0/563528599 shutdown_connections 2024-04-20T22:23:57.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:57.423+0000 7f51d2c00700 1 --2- 172.21.15.62:0/563528599 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f51b006ecc0 0x7f51b0071170 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:23:57.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:57.423+0000 7f51d2c00700 1 --2- 172.21.15.62:0/563528599 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51cc1004e0 0x7f51cc1980f0 unknown :-1 s=CLOSED pgs=1250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:23:57.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:57.423+0000 7f51d2c00700 1 --2- 172.21.15.62:0/563528599 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f51cc100e80 0x7f51cc198630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:23:57.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:57.423+0000 7f51d2c00700 1 -- 172.21.15.62:0/563528599 >> 172.21.15.62:0/563528599 conn(0x7f51cc0fbda0 msgr2=0x7f51cc0fe160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:23:57.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:57.423+0000 7f51d2c00700 1 -- 172.21.15.62:0/563528599 shutdown_connections 2024-04-20T22:23:57.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:57.423+0000 7f51d2c00700 1 -- 172.21.15.62:0/563528599 wait complete. 2024-04-20T22:23:57.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:57.590+0000 7f47d3800700 1 -- 172.21.15.62:0/1276712861 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f47cc106b80 msgr2=0x7f47cc106ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:23:57.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:57.590+0000 7f47d3800700 1 --2- 172.21.15.62:0/1276712861 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f47cc106b80 0x7f47cc106ff0 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7f47b4009910 tx=0x7f47b4009c20 comp rx=0 tx=0).stop 2024-04-20T22:23:57.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:57.590+0000 7f47d3800700 1 -- 172.21.15.62:0/1276712861 shutdown_connections 2024-04-20T22:23:57.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:57.590+0000 7f47d3800700 1 --2- 172.21.15.62:0/1276712861 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f47cc106b80 0x7f47cc106ff0 unknown :-1 s=CLOSED pgs=1251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:23:57.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:57.590+0000 7f47d3800700 1 --2- 172.21.15.62:0/1276712861 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f47cc10c8f0 0x7f47cc10ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:23:57.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:57.590+0000 7f47d3800700 1 -- 172.21.15.62:0/1276712861 >> 172.21.15.62:0/1276712861 conn(0x7f47cc076060 msgr2=0x7f47cc078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:23:57.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:57.590+0000 7f47d3800700 1 -- 172.21.15.62:0/1276712861 shutdown_connections 2024-04-20T22:23:57.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:57.590+0000 7f47d3800700 1 -- 172.21.15.62:0/1276712861 wait complete. 2024-04-20T22:23:57.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:57.591+0000 7f47d3800700 1 Processor -- start 2024-04-20T22:23:57.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:57.591+0000 7f47d3800700 1 -- start start 2024-04-20T22:23:57.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:57.592+0000 7f47d3800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f47cc106b80 0x7f47cc19c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:23:57.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:57.592+0000 7f47d3800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f47cc10c8f0 0x7f47cc19c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:23:57.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:57.592+0000 7f47d3800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47cc19d090 con 0x7f47cc10c8f0 2024-04-20T22:23:57.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:57.592+0000 7f47d3800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47cc1a0e20 con 0x7f47cc106b80 2024-04-20T22:23:57.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:57.592+0000 7f47c3e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f47cc10c8f0 0x7f47cc19c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:23:57.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:57.592+0000 7f47c3e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f47cc10c8f0 0x7f47cc19c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47606/0 (socket says 172.21.15.62:47606) 2024-04-20T22:23:57.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:57.592+0000 7f47c3e00700 1 -- 172.21.15.62:0/2475652948 learned_addr learned my addr 172.21.15.62:0/2475652948 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:23:57.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:57.592+0000 7f47d0e00700 1 --2- 172.21.15.62:0/2475652948 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f47cc106b80 0x7f47cc19c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:23:57.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:57.592+0000 7f47c3e00700 1 -- 172.21.15.62:0/2475652948 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f47cc106b80 msgr2=0x7f47cc19c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:23:57.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:57.592+0000 7f47c3e00700 1 --2- 172.21.15.62:0/2475652948 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f47cc106b80 0x7f47cc19c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:23:57.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:57.592+0000 7f47c3e00700 1 -- 172.21.15.62:0/2475652948 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47b40095f0 con 0x7f47cc10c8f0 2024-04-20T22:23:57.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:57.593+0000 7f47c3e00700 1 --2- 172.21.15.62:0/2475652948 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f47cc10c8f0 0x7f47cc19c9b0 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f47b4006010 tx=0x7f47b400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:23:57.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:57.593+0000 7f47ca000700 1 -- 172.21.15.62:0/2475652948 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f47b401d070 con 0x7f47cc10c8f0 2024-04-20T22:23:57.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:57.593+0000 7f47ca000700 1 -- 172.21.15.62:0/2475652948 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f47b400f460 con 0x7f47cc10c8f0 2024-04-20T22:23:57.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:57.593+0000 7f47ca000700 1 -- 172.21.15.62:0/2475652948 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f47b4017630 con 0x7f47cc10c8f0 2024-04-20T22:23:57.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:57.593+0000 7f47d3800700 1 -- 172.21.15.62:0/2475652948 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47cc1a10a0 con 0x7f47cc10c8f0 2024-04-20T22:23:57.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:57.593+0000 7f47d3800700 1 -- 172.21.15.62:0/2475652948 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47cc1a1540 con 0x7f47cc10c8f0 2024-04-20T22:23:57.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:57.595+0000 7f47d3800700 1 -- 172.21.15.62:0/2475652948 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47cc10ecc0 con 0x7f47cc10c8f0 2024-04-20T22:23:57.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:57.596+0000 7f47ca000700 1 -- 172.21.15.62:0/2475652948 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f47b400f5d0 con 0x7f47cc10c8f0 2024-04-20T22:23:57.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:57.597+0000 7f47ca000700 1 --2- 172.21.15.62:0/2475652948 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f47b006ecc0 0x7f47b0071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:23:57.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:57.597+0000 7f47ca000700 1 -- 172.21.15.62:0/2475652948 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f47b4090330 con 0x7f47cc10c8f0 2024-04-20T22:23:57.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:57.597+0000 7f47d0e00700 1 --2- 172.21.15.62:0/2475652948 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f47b006ecc0 0x7f47b0071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:23:57.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:57.598+0000 7f47d0e00700 1 --2- 172.21.15.62:0/2475652948 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f47b006ecc0 0x7f47b0071170 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f47bc005950 tx=0x7f47bc0058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:23:57.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:57.602+0000 7f47ca000700 1 -- 172.21.15.62:0/2475652948 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f47b40590c0 con 0x7f47cc10c8f0 2024-04-20T22:23:57.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:57.930+0000 7f47d3800700 1 -- 172.21.15.62:0/2475652948 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f47cc04f2a0 con 0x7f47cc10c8f0 2024-04-20T22:23:57.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:57.930+0000 7f47ca000700 1 -- 172.21.15.62:0/2475652948 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7f47b405c780 con 0x7f47cc10c8f0 2024-04-20T22:23:57.930 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-20T22:23:57.931 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-20T22:23:57.931 INFO:teuthology.orchestra.run.smithi062.stdout: host smithi189 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi189/config 2024-04-20T22:23:57.931 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-20T22:23:57.931 INFO:teuthology.orchestra.run.smithi062.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi189 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp11670v2q:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:23:57.931 INFO:teuthology.orchestra.run.smithi062.stdout:Traceback (most recent call last): 2024-04-20T22:23:57.931 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-20T22:23:57.931 INFO:teuthology.orchestra.run.smithi062.stdout: return _run_code(code, main_globals, None, 2024-04-20T22:23:57.931 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-20T22:23:57.931 INFO:teuthology.orchestra.run.smithi062.stdout: exec(code, run_globals) 2024-04-20T22:23:57.931 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5618, in 2024-04-20T22:23:57.931 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5606, in main 2024-04-20T22:23:57.931 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 408, in _infer_config 2024-04-20T22:23:57.931 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 323, in _infer_fsid 2024-04-20T22:23:57.932 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 436, in _infer_image 2024-04-20T22:23:57.932 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 310, in _validate_fsid 2024-04-20T22:23:57.932 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 3359, in command_ceph_volume 2024-04-20T22:23:57.932 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-20T22:23:57.932 INFO:teuthology.orchestra.run.smithi062.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi189 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp11670v2q:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:23:57.932 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-20T22:23:57.932 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is degraded 2024-04-20T22:23:57.932 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T22:23:57.932 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T22:23:57.932 INFO:teuthology.orchestra.run.smithi062.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-20T22:23:57.932 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-20T22:23:57.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:57.933+0000 7f47d3800700 1 -- 172.21.15.62:0/2475652948 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f47b006ecc0 msgr2=0x7f47b0071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:23:57.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:57.933+0000 7f47d3800700 1 --2- 172.21.15.62:0/2475652948 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f47b006ecc0 0x7f47b0071170 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f47bc005950 tx=0x7f47bc0058e0 comp rx=0 tx=0).stop 2024-04-20T22:23:57.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:57.933+0000 7f47d3800700 1 -- 172.21.15.62:0/2475652948 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f47cc10c8f0 msgr2=0x7f47cc19c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:23:57.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:57.933+0000 7f47d3800700 1 --2- 172.21.15.62:0/2475652948 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f47cc10c8f0 0x7f47cc19c9b0 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f47b4006010 tx=0x7f47b400be20 comp rx=0 tx=0).stop 2024-04-20T22:23:57.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:57.933+0000 7f47d3800700 1 -- 172.21.15.62:0/2475652948 shutdown_connections 2024-04-20T22:23:57.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:57.933+0000 7f47d3800700 1 --2- 172.21.15.62:0/2475652948 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f47b006ecc0 0x7f47b0071170 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:23:57.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:57.933+0000 7f47d3800700 1 --2- 172.21.15.62:0/2475652948 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f47cc106b80 0x7f47cc19c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:23:57.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:57.933+0000 7f47d3800700 1 --2- 172.21.15.62:0/2475652948 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f47cc10c8f0 0x7f47cc19c9b0 unknown :-1 s=CLOSED pgs=1252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:23:57.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:57.933+0000 7f47d3800700 1 -- 172.21.15.62:0/2475652948 >> 172.21.15.62:0/2475652948 conn(0x7f47cc076060 msgr2=0x7f47cc077c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:23:57.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:57.934+0000 7f47d3800700 1 -- 172.21.15.62:0/2475652948 shutdown_connections 2024-04-20T22:23:57.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:23:57.934+0000 7f47d3800700 1 -- 172.21.15.62:0/2475652948 wait complete. 2024-04-20T22:23:58.144 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:23:57 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/4197588485' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:23:58.144 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:23:57 smithi062 ceph-mon[76322]: from='client.36572 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:23:58.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:23:57 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/4197588485' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:23:58.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:23:57 smithi189 ceph-mon[57022]: from='client.36572 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:23:59.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:23:58 smithi189 ceph-mon[57022]: pgmap v1463: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:23:59.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:23:58 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/2475652948' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:23:59.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:23:58 smithi062 ceph-mon[76322]: pgmap v1463: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:23:59.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:23:58 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/2475652948' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:24:01.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:24:00 smithi189 ceph-mon[57022]: pgmap v1464: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:24:01.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:24:00 smithi062 ceph-mon[76322]: pgmap v1464: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:24:03.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:24:02 smithi189 ceph-mon[57022]: pgmap v1465: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:24:03.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:24:02 smithi062 ceph-mon[76322]: pgmap v1465: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:24:05.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:24:04 smithi189 ceph-mon[57022]: pgmap v1466: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:24:05.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:24:04 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:24:05.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:24:04 smithi062 ceph-mon[76322]: pgmap v1466: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:24:05.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:24:04 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:24:07.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:24:06 smithi189 ceph-mon[57022]: pgmap v1467: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:24:07.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:24:06 smithi062 ceph-mon[76322]: pgmap v1467: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:24:09.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:24:08 smithi189 ceph-mon[57022]: pgmap v1468: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:24:09.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:24:08 smithi062 ceph-mon[76322]: pgmap v1468: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:24:11.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:24:10 smithi189 ceph-mon[57022]: pgmap v1469: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:24:11.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:24:10 smithi062 ceph-mon[76322]: pgmap v1469: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:24:13.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:24:12 smithi189 ceph-mon[57022]: pgmap v1470: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:24:13.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:24:12 smithi062 ceph-mon[76322]: pgmap v1470: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:24:15.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:24:14 smithi189 ceph-mon[57022]: pgmap v1471: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:24:15.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:24:14 smithi062 ceph-mon[76322]: pgmap v1471: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:24:17.188 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:24:16 smithi062 ceph-mon[76322]: pgmap v1472: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:24:17.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:24:16 smithi189 ceph-mon[57022]: pgmap v1472: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:24:19.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:24:18 smithi189 ceph-mon[57022]: pgmap v1473: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:24:19.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:24:18 smithi062 ceph-mon[76322]: pgmap v1473: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:24:20.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:24:19 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:24:20.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:24:19 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:24:21.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:24:20 smithi189 ceph-mon[57022]: pgmap v1474: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:24:21.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:24:20 smithi062 ceph-mon[76322]: pgmap v1474: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:24:23.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:24:22 smithi189 ceph-mon[57022]: pgmap v1475: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:24:23.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:24:22 smithi062 ceph-mon[76322]: pgmap v1475: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:24:25.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:24:24 smithi189 ceph-mon[57022]: pgmap v1476: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:24:25.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:24:24 smithi062 ceph-mon[76322]: pgmap v1476: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:24:27.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:24:27 smithi062 ceph-mon[76322]: pgmap v1477: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:24:27.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:24:27 smithi189 ceph-mon[57022]: pgmap v1477: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:24:28.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.113+0000 7f87b9000700 1 -- 172.21.15.62:0/1290034461 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f87b410c8f0 msgr2=0x7f87b410ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:24:28.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.113+0000 7f87b9000700 1 --2- 172.21.15.62:0/1290034461 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f87b410c8f0 0x7f87b410ccc0 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f87a4009910 tx=0x7f87a4009c20 comp rx=0 tx=0).stop 2024-04-20T22:24:28.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.113+0000 7f87b9000700 1 -- 172.21.15.62:0/1290034461 shutdown_connections 2024-04-20T22:24:28.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.113+0000 7f87b9000700 1 --2- 172.21.15.62:0/1290034461 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87b4106b80 0x7f87b4106ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:24:28.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.113+0000 7f87b9000700 1 --2- 172.21.15.62:0/1290034461 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f87b410c8f0 0x7f87b410ccc0 unknown :-1 s=CLOSED pgs=1253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:24:28.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.113+0000 7f87b9000700 1 -- 172.21.15.62:0/1290034461 >> 172.21.15.62:0/1290034461 conn(0x7f87b4076060 msgr2=0x7f87b4078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:24:28.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.113+0000 7f87b9000700 1 -- 172.21.15.62:0/1290034461 shutdown_connections 2024-04-20T22:24:28.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.114+0000 7f87b9000700 1 -- 172.21.15.62:0/1290034461 wait complete. 2024-04-20T22:24:28.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.115+0000 7f87b9000700 1 Processor -- start 2024-04-20T22:24:28.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.115+0000 7f87b9000700 1 -- start start 2024-04-20T22:24:28.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.116+0000 7f87b9000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f87b4106b80 0x7f87b419c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:24:28.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.116+0000 7f87b9000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87b410c8f0 0x7f87b419ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:24:28.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.116+0000 7f87b9000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87b419d100 con 0x7f87b4106b80 2024-04-20T22:24:28.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.116+0000 7f87b9000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87b41a0e90 con 0x7f87b410c8f0 2024-04-20T22:24:28.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.116+0000 7f87b1e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f87b4106b80 0x7f87b419c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:24:28.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.116+0000 7f87b1e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f87b4106b80 0x7f87b419c4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45816/0 (socket says 172.21.15.62:45816) 2024-04-20T22:24:28.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.116+0000 7f87b1e00700 1 -- 172.21.15.62:0/552684490 learned_addr learned my addr 172.21.15.62:0/552684490 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:24:28.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.116+0000 7f87b1400700 1 --2- 172.21.15.62:0/552684490 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87b410c8f0 0x7f87b419ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:24:28.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.117+0000 7f87b1e00700 1 -- 172.21.15.62:0/552684490 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87b410c8f0 msgr2=0x7f87b419ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:24:28.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.117+0000 7f87b1e00700 1 --2- 172.21.15.62:0/552684490 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87b410c8f0 0x7f87b419ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:24:28.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.117+0000 7f87b1e00700 1 -- 172.21.15.62:0/552684490 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87a40095f0 con 0x7f87b4106b80 2024-04-20T22:24:28.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.117+0000 7f87b1400700 1 --2- 172.21.15.62:0/552684490 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87b410c8f0 0x7f87b419ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T22:24:28.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.117+0000 7f87b1e00700 1 --2- 172.21.15.62:0/552684490 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f87b4106b80 0x7f87b419c4e0 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7f87a400b5c0 tx=0x7f87a400be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:24:28.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.117+0000 7f87aaa00700 1 -- 172.21.15.62:0/552684490 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f87a401d070 con 0x7f87b4106b80 2024-04-20T22:24:28.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.117+0000 7f87aaa00700 1 -- 172.21.15.62:0/552684490 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f87a400f460 con 0x7f87b4106b80 2024-04-20T22:24:28.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.117+0000 7f87aaa00700 1 -- 172.21.15.62:0/552684490 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f87a4017610 con 0x7f87b4106b80 2024-04-20T22:24:28.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.117+0000 7f87b9000700 1 -- 172.21.15.62:0/552684490 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87b41a1110 con 0x7f87b4106b80 2024-04-20T22:24:28.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.118+0000 7f87b9000700 1 -- 172.21.15.62:0/552684490 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87b41a15b0 con 0x7f87b4106b80 2024-04-20T22:24:28.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.119+0000 7f87b9000700 1 -- 172.21.15.62:0/552684490 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87b4109b20 con 0x7f87b4106b80 2024-04-20T22:24:28.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.120+0000 7f87aaa00700 1 -- 172.21.15.62:0/552684490 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f87a400f5d0 con 0x7f87b4106b80 2024-04-20T22:24:28.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.121+0000 7f87aaa00700 1 --2- 172.21.15.62:0/552684490 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f879806ecc0 0x7f8798071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:24:28.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.121+0000 7f87aaa00700 1 -- 172.21.15.62:0/552684490 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f87a4090480 con 0x7f87b4106b80 2024-04-20T22:24:28.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.121+0000 7f87b1400700 1 --2- 172.21.15.62:0/552684490 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f879806ecc0 0x7f8798071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:24:28.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.122+0000 7f87b1400700 1 --2- 172.21.15.62:0/552684490 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f879806ecc0 0x7f8798071170 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f87b419db00 tx=0x7f87a0008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:24:28.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.126+0000 7f87aaa00700 1 -- 172.21.15.62:0/552684490 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f87a4059230 con 0x7f87b4106b80 2024-04-20T22:24:28.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.387+0000 7f87b9000700 1 -- 172.21.15.62:0/552684490 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f87b41a18e0 con 0x7f879806ecc0 2024-04-20T22:24:28.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.390+0000 7f87aaa00700 1 -- 172.21.15.62:0/552684490 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f87b41a18e0 con 0x7f879806ecc0 2024-04-20T22:24:28.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.393+0000 7f87b9000700 1 -- 172.21.15.62:0/552684490 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f879806ecc0 msgr2=0x7f8798071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:24:28.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.393+0000 7f87b9000700 1 --2- 172.21.15.62:0/552684490 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f879806ecc0 0x7f8798071170 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f87b419db00 tx=0x7f87a0008040 comp rx=0 tx=0).stop 2024-04-20T22:24:28.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.393+0000 7f87b9000700 1 -- 172.21.15.62:0/552684490 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f87b4106b80 msgr2=0x7f87b419c4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:24:28.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.393+0000 7f87b9000700 1 --2- 172.21.15.62:0/552684490 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f87b4106b80 0x7f87b419c4e0 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7f87a400b5c0 tx=0x7f87a400be00 comp rx=0 tx=0).stop 2024-04-20T22:24:28.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.393+0000 7f87b9000700 1 -- 172.21.15.62:0/552684490 shutdown_connections 2024-04-20T22:24:28.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.393+0000 7f87b9000700 1 --2- 172.21.15.62:0/552684490 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f879806ecc0 0x7f8798071170 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:24:28.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.393+0000 7f87b9000700 1 --2- 172.21.15.62:0/552684490 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f87b4106b80 0x7f87b419c4e0 unknown :-1 s=CLOSED pgs=1254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:24:28.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.393+0000 7f87b9000700 1 --2- 172.21.15.62:0/552684490 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87b410c8f0 0x7f87b419ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:24:28.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.393+0000 7f87b9000700 1 -- 172.21.15.62:0/552684490 >> 172.21.15.62:0/552684490 conn(0x7f87b4076060 msgr2=0x7f87b4077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:24:28.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.393+0000 7f87b9000700 1 -- 172.21.15.62:0/552684490 shutdown_connections 2024-04-20T22:24:28.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.394+0000 7f87b9000700 1 -- 172.21.15.62:0/552684490 wait complete. 2024-04-20T22:24:28.409 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T22:24:28.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.560+0000 7f8e27a00700 1 -- 172.21.15.62:0/3588046085 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e201004e0 msgr2=0x7f8e201008b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:24:28.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.560+0000 7f8e27a00700 1 --2- 172.21.15.62:0/3588046085 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e201004e0 0x7f8e201008b0 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f8e14009910 tx=0x7f8e14009c20 comp rx=0 tx=0).stop 2024-04-20T22:24:28.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.560+0000 7f8e27a00700 1 -- 172.21.15.62:0/3588046085 shutdown_connections 2024-04-20T22:24:28.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.560+0000 7f8e27a00700 1 --2- 172.21.15.62:0/3588046085 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e20100e80 0x7f8e20104e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:24:28.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.560+0000 7f8e27a00700 1 --2- 172.21.15.62:0/3588046085 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e201004e0 0x7f8e201008b0 unknown :-1 s=CLOSED pgs=1255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:24:28.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.560+0000 7f8e27a00700 1 -- 172.21.15.62:0/3588046085 >> 172.21.15.62:0/3588046085 conn(0x7f8e200fbda0 msgr2=0x7f8e200fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:24:28.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.560+0000 7f8e27a00700 1 -- 172.21.15.62:0/3588046085 shutdown_connections 2024-04-20T22:24:28.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.560+0000 7f8e27a00700 1 -- 172.21.15.62:0/3588046085 wait complete. 2024-04-20T22:24:28.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.561+0000 7f8e27a00700 1 Processor -- start 2024-04-20T22:24:28.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.561+0000 7f8e27a00700 1 -- start start 2024-04-20T22:24:28.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.562+0000 7f8e27a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e201004e0 0x7f8e20073050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:24:28.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.562+0000 7f8e27a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e20100e80 0x7f8e20073590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:24:28.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.562+0000 7f8e27a00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e20073ad0 con 0x7f8e201004e0 2024-04-20T22:24:28.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.562+0000 7f8e27a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e20073c10 con 0x7f8e20100e80 2024-04-20T22:24:28.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.562+0000 7f8e25000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e201004e0 0x7f8e20073050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:24:28.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.562+0000 7f8e25000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e201004e0 0x7f8e20073050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45840/0 (socket says 172.21.15.62:45840) 2024-04-20T22:24:28.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.562+0000 7f8e25000700 1 -- 172.21.15.62:0/1659946331 learned_addr learned my addr 172.21.15.62:0/1659946331 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:24:28.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.562+0000 7f8e1fe00700 1 --2- 172.21.15.62:0/1659946331 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e20100e80 0x7f8e20073590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:24:28.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.563+0000 7f8e25000700 1 -- 172.21.15.62:0/1659946331 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e20100e80 msgr2=0x7f8e20073590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:24:28.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.563+0000 7f8e25000700 1 --2- 172.21.15.62:0/1659946331 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e20100e80 0x7f8e20073590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:24:28.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.563+0000 7f8e25000700 1 -- 172.21.15.62:0/1659946331 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e140095f0 con 0x7f8e201004e0 2024-04-20T22:24:28.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.563+0000 7f8e1fe00700 1 --2- 172.21.15.62:0/1659946331 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e20100e80 0x7f8e20073590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T22:24:28.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.563+0000 7f8e25000700 1 --2- 172.21.15.62:0/1659946331 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e201004e0 0x7f8e20073050 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f8e1400b5c0 tx=0x7f8e1400be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:24:28.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.563+0000 7f8e1d600700 1 -- 172.21.15.62:0/1659946331 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8e1401d070 con 0x7f8e201004e0 2024-04-20T22:24:28.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.563+0000 7f8e1d600700 1 -- 172.21.15.62:0/1659946331 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8e1400f460 con 0x7f8e201004e0 2024-04-20T22:24:28.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.563+0000 7f8e27a00700 1 -- 172.21.15.62:0/1659946331 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e20073e90 con 0x7f8e201004e0 2024-04-20T22:24:28.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.563+0000 7f8e1d600700 1 -- 172.21.15.62:0/1659946331 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8e14017610 con 0x7f8e201004e0 2024-04-20T22:24:28.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.564+0000 7f8e27a00700 1 -- 172.21.15.62:0/1659946331 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e201a2490 con 0x7f8e201004e0 2024-04-20T22:24:28.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.565+0000 7f8e27a00700 1 -- 172.21.15.62:0/1659946331 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e201087f0 con 0x7f8e201004e0 2024-04-20T22:24:28.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.566+0000 7f8e1d600700 1 -- 172.21.15.62:0/1659946331 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f8e1400f5d0 con 0x7f8e201004e0 2024-04-20T22:24:28.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.567+0000 7f8e1d600700 1 --2- 172.21.15.62:0/1659946331 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f8e0806ecc0 0x7f8e08071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:24:28.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.567+0000 7f8e1d600700 1 -- 172.21.15.62:0/1659946331 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f8e14090420 con 0x7f8e201004e0 2024-04-20T22:24:28.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.567+0000 7f8e1fe00700 1 --2- 172.21.15.62:0/1659946331 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f8e0806ecc0 0x7f8e08071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:24:28.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.568+0000 7f8e1fe00700 1 --2- 172.21.15.62:0/1659946331 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f8e0806ecc0 0x7f8e08071170 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f8e20074840 tx=0x7f8e0c008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:24:28.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.574+0000 7f8e1d600700 1 -- 172.21.15.62:0/1659946331 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f8e140591b0 con 0x7f8e201004e0 2024-04-20T22:24:28.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.832+0000 7f8e27a00700 1 -- 172.21.15.62:0/1659946331 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8e20061190 con 0x7f8e0806ecc0 2024-04-20T22:24:28.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.834+0000 7f8e1d600700 1 -- 172.21.15.62:0/1659946331 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f8e20061190 con 0x7f8e0806ecc0 2024-04-20T22:24:28.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.837+0000 7f8e27a00700 1 -- 172.21.15.62:0/1659946331 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f8e0806ecc0 msgr2=0x7f8e08071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:24:28.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.837+0000 7f8e27a00700 1 --2- 172.21.15.62:0/1659946331 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f8e0806ecc0 0x7f8e08071170 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f8e20074840 tx=0x7f8e0c008040 comp rx=0 tx=0).stop 2024-04-20T22:24:28.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.837+0000 7f8e27a00700 1 -- 172.21.15.62:0/1659946331 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e201004e0 msgr2=0x7f8e20073050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:24:28.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.837+0000 7f8e27a00700 1 --2- 172.21.15.62:0/1659946331 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e201004e0 0x7f8e20073050 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f8e1400b5c0 tx=0x7f8e1400be00 comp rx=0 tx=0).stop 2024-04-20T22:24:28.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.837+0000 7f8e27a00700 1 -- 172.21.15.62:0/1659946331 shutdown_connections 2024-04-20T22:24:28.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.837+0000 7f8e27a00700 1 --2- 172.21.15.62:0/1659946331 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f8e0806ecc0 0x7f8e08071170 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:24:28.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.837+0000 7f8e27a00700 1 --2- 172.21.15.62:0/1659946331 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e201004e0 0x7f8e20073050 unknown :-1 s=CLOSED pgs=1256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:24:28.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.837+0000 7f8e27a00700 1 --2- 172.21.15.62:0/1659946331 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e20100e80 0x7f8e20073590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:24:28.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.837+0000 7f8e27a00700 1 -- 172.21.15.62:0/1659946331 >> 172.21.15.62:0/1659946331 conn(0x7f8e200fbda0 msgr2=0x7f8e200fd470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:24:28.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.838+0000 7f8e27a00700 1 -- 172.21.15.62:0/1659946331 shutdown_connections 2024-04-20T22:24:28.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.838+0000 7f8e27a00700 1 -- 172.21.15.62:0/1659946331 wait complete. 2024-04-20T22:24:28.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.989+0000 7fbf59600700 1 -- 172.21.15.62:0/40821782 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbf54100e80 msgr2=0x7fbf54104e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:24:28.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.989+0000 7fbf59600700 1 --2- 172.21.15.62:0/40821782 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbf54100e80 0x7fbf54104e50 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7fbf3c009910 tx=0x7fbf3c009c20 comp rx=0 tx=0).stop 2024-04-20T22:24:28.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.989+0000 7fbf59600700 1 -- 172.21.15.62:0/40821782 shutdown_connections 2024-04-20T22:24:28.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.989+0000 7fbf59600700 1 --2- 172.21.15.62:0/40821782 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbf54100e80 0x7fbf54104e50 unknown :-1 s=CLOSED pgs=1257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:24:28.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.989+0000 7fbf59600700 1 --2- 172.21.15.62:0/40821782 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf541004e0 0x7fbf541008b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:24:28.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.989+0000 7fbf59600700 1 -- 172.21.15.62:0/40821782 >> 172.21.15.62:0/40821782 conn(0x7fbf540fbda0 msgr2=0x7fbf540fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:24:28.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.989+0000 7fbf59600700 1 -- 172.21.15.62:0/40821782 shutdown_connections 2024-04-20T22:24:28.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.989+0000 7fbf59600700 1 -- 172.21.15.62:0/40821782 wait complete. 2024-04-20T22:24:28.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.990+0000 7fbf59600700 1 Processor -- start 2024-04-20T22:24:28.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.990+0000 7fbf59600700 1 -- start start 2024-04-20T22:24:28.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.991+0000 7fbf59600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbf541004e0 0x7fbf54198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:24:28.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.991+0000 7fbf59600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf54100e80 0x7fbf541985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:24:28.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.991+0000 7fbf59600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf54198ca0 con 0x7fbf541004e0 2024-04-20T22:24:28.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.991+0000 7fbf59600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf5419ca30 con 0x7fbf54100e80 2024-04-20T22:24:28.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.991+0000 7fbf52a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbf541004e0 0x7fbf54198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:24:28.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.991+0000 7fbf52a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbf541004e0 0x7fbf54198080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45858/0 (socket says 172.21.15.62:45858) 2024-04-20T22:24:28.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.991+0000 7fbf52a00700 1 -- 172.21.15.62:0/492793307 learned_addr learned my addr 172.21.15.62:0/492793307 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:24:28.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.991+0000 7fbf4a000700 1 --2- 172.21.15.62:0/492793307 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf54100e80 0x7fbf541985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:24:28.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.991+0000 7fbf52a00700 1 -- 172.21.15.62:0/492793307 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf54100e80 msgr2=0x7fbf541985c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:24:28.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.992+0000 7fbf52a00700 1 --2- 172.21.15.62:0/492793307 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf54100e80 0x7fbf541985c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:24:28.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.992+0000 7fbf52a00700 1 -- 172.21.15.62:0/492793307 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf3c0095f0 con 0x7fbf541004e0 2024-04-20T22:24:28.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.992+0000 7fbf4a000700 1 --2- 172.21.15.62:0/492793307 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf54100e80 0x7fbf541985c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T22:24:28.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.992+0000 7fbf52a00700 1 --2- 172.21.15.62:0/492793307 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbf541004e0 0x7fbf54198080 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7fbf440060b0 tx=0x7fbf4400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:24:28.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.992+0000 7fbf4be00700 1 -- 172.21.15.62:0/492793307 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbf4400dda0 con 0x7fbf541004e0 2024-04-20T22:24:28.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.992+0000 7fbf4be00700 1 -- 172.21.15.62:0/492793307 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbf4400f040 con 0x7fbf541004e0 2024-04-20T22:24:28.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.992+0000 7fbf4be00700 1 -- 172.21.15.62:0/492793307 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbf44014790 con 0x7fbf541004e0 2024-04-20T22:24:28.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.992+0000 7fbf59600700 1 -- 172.21.15.62:0/492793307 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf5419cd10 con 0x7fbf541004e0 2024-04-20T22:24:28.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.992+0000 7fbf59600700 1 -- 172.21.15.62:0/492793307 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf5419d260 con 0x7fbf541004e0 2024-04-20T22:24:28.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.994+0000 7fbf59600700 1 -- 172.21.15.62:0/492793307 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf541045d0 con 0x7fbf541004e0 2024-04-20T22:24:28.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.995+0000 7fbf4be00700 1 -- 172.21.15.62:0/492793307 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fbf440148f0 con 0x7fbf541004e0 2024-04-20T22:24:28.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.996+0000 7fbf4be00700 1 --2- 172.21.15.62:0/492793307 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fbf3806ecc0 0x7fbf38071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:24:28.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.996+0000 7fbf4be00700 1 -- 172.21.15.62:0/492793307 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fbf4408e8d0 con 0x7fbf541004e0 2024-04-20T22:24:28.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.996+0000 7fbf4a000700 1 --2- 172.21.15.62:0/492793307 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fbf3806ecc0 0x7fbf38071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:24:28.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:28.997+0000 7fbf4a000700 1 --2- 172.21.15.62:0/492793307 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fbf3806ecc0 0x7fbf38071170 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fbf3c006010 tx=0x7fbf3c00b540 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:24:29.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:29.001+0000 7fbf4be00700 1 -- 172.21.15.62:0/492793307 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fbf44057710 con 0x7fbf541004e0 2024-04-20T22:24:29.237 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:24:29 smithi062 ceph-mon[76322]: pgmap v1478: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:24:29.237 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:24:29 smithi062 ceph-mon[76322]: from='client.36580 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:24:29.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:29.237+0000 7fbf59600700 1 -- 172.21.15.62:0/492793307 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fbf5419d530 con 0x7fbf3806ecc0 2024-04-20T22:24:29.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:29.270+0000 7fbf4be00700 1 -- 172.21.15.62:0/492793307 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fbf5419d530 con 0x7fbf3806ecc0 2024-04-20T22:24:29.269 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T22:24:29.269 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (50m) 11m ago 64m 14.6M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T22:24:29.269 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (64m) 11m ago 64m 11.6M - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T22:24:29.270 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (63m) 11m ago 63m 11.2M - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T22:24:29.270 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (46m) 11m ago 64m 6660k - 19.0.0-3133-ga8c84f59 76bc6b91b726 2825d5ba11c6 2024-04-20T22:24:29.270 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (46m) 11m ago 63m 6665k - 19.0.0-3133-ga8c84f59 76bc6b91b726 20f746fa1bba 2024-04-20T22:24:29.270 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (49m) 11m ago 63m 75.5M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T22:24:29.270 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (41m) 11m ago 56m 22.9M - 19.0.0-3133-ga8c84f59 76bc6b91b726 de80bc90a683 2024-04-20T22:24:29.270 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (42m) 11m ago 57m 23.0M - 19.0.0-3133-ga8c84f59 76bc6b91b726 cd793b7ee98e 2024-04-20T22:24:29.270 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (26m) 11m ago 56m 18.5M - 19.0.0-3133-ga8c84f59 76bc6b91b726 0e7b0bc4d1ab 2024-04-20T22:24:29.270 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (11m) 11m ago 56m 11.4M - 19.0.0-3133-ga8c84f59 76bc6b91b726 0af732ac9e06 2024-04-20T22:24:29.270 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (53m) 11m ago 65m 517M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T22:24:29.270 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (51m) 11m ago 62m 460M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T22:24:29.270 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (48m) 11m ago 66m 86.3M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 019eb1479485 2024-04-20T22:24:29.270 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (47m) 11m ago 62m 58.0M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 55aac560aab6 2024-04-20T22:24:29.270 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (51m) 11m ago 64m 8766k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T22:24:29.270 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (51m) 11m ago 62m 8711k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T22:24:29.271 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (45m) 11m ago 61m 91.3M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 ee2bb901f26e 2024-04-20T22:24:29.271 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (45m) 11m ago 61m 94.9M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 9075152d4162 2024-04-20T22:24:29.271 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (44m) 11m ago 60m 78.5M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 2d4f3e38dc0e 2024-04-20T22:24:29.271 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (44m) 11m ago 60m 94.1M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 8f97b17ae0eb 2024-04-20T22:24:29.271 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (43m) 11m ago 59m 96.1M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 653d8079967d 2024-04-20T22:24:29.271 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (42m) 11m ago 58m 90.9M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 596dded14c12 2024-04-20T22:24:29.271 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (50m) 11m ago 63m 90.4M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T22:24:29.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:29.274+0000 7fbf59600700 1 -- 172.21.15.62:0/492793307 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fbf3806ecc0 msgr2=0x7fbf38071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:24:29.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:29.274+0000 7fbf59600700 1 --2- 172.21.15.62:0/492793307 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fbf3806ecc0 0x7fbf38071170 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fbf3c006010 tx=0x7fbf3c00b540 comp rx=0 tx=0).stop 2024-04-20T22:24:29.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:29.274+0000 7fbf59600700 1 -- 172.21.15.62:0/492793307 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbf541004e0 msgr2=0x7fbf54198080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:24:29.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:29.274+0000 7fbf59600700 1 --2- 172.21.15.62:0/492793307 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbf541004e0 0x7fbf54198080 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7fbf440060b0 tx=0x7fbf4400d3b0 comp rx=0 tx=0).stop 2024-04-20T22:24:29.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:29.274+0000 7fbf59600700 1 -- 172.21.15.62:0/492793307 shutdown_connections 2024-04-20T22:24:29.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:29.274+0000 7fbf59600700 1 --2- 172.21.15.62:0/492793307 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fbf3806ecc0 0x7fbf38071170 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:24:29.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:29.274+0000 7fbf59600700 1 --2- 172.21.15.62:0/492793307 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbf541004e0 0x7fbf54198080 unknown :-1 s=CLOSED pgs=1258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:24:29.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:29.274+0000 7fbf59600700 1 --2- 172.21.15.62:0/492793307 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf54100e80 0x7fbf541985c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:24:29.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:29.274+0000 7fbf59600700 1 -- 172.21.15.62:0/492793307 >> 172.21.15.62:0/492793307 conn(0x7fbf540fbda0 msgr2=0x7fbf540fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:24:29.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:29.274+0000 7fbf59600700 1 -- 172.21.15.62:0/492793307 shutdown_connections 2024-04-20T22:24:29.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:29.274+0000 7fbf59600700 1 -- 172.21.15.62:0/492793307 wait complete. 2024-04-20T22:24:29.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:29.419+0000 7fe9f7400700 1 -- 172.21.15.62:0/2813246040 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9f0102470 msgr2=0x7fe9f01028e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:24:29.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:29.419+0000 7fe9f7400700 1 --2- 172.21.15.62:0/2813246040 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9f0102470 0x7fe9f01028e0 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7fe9dc009910 tx=0x7fe9dc009c20 comp rx=0 tx=0).stop 2024-04-20T22:24:29.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:29.419+0000 7fe9f7400700 1 -- 172.21.15.62:0/2813246040 shutdown_connections 2024-04-20T22:24:29.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:29.419+0000 7fe9f7400700 1 --2- 172.21.15.62:0/2813246040 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9f0102470 0x7fe9f01028e0 unknown :-1 s=CLOSED pgs=1259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:24:29.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:29.419+0000 7fe9f7400700 1 --2- 172.21.15.62:0/2813246040 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe9f0108470 0x7fe9f0108840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:24:29.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:29.419+0000 7fe9f7400700 1 -- 172.21.15.62:0/2813246040 >> 172.21.15.62:0/2813246040 conn(0x7fe9f00fdf90 msgr2=0x7fe9f01003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:24:29.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:29.419+0000 7fe9f7400700 1 -- 172.21.15.62:0/2813246040 shutdown_connections 2024-04-20T22:24:29.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:29.419+0000 7fe9f7400700 1 -- 172.21.15.62:0/2813246040 wait complete. 2024-04-20T22:24:29.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:29.420+0000 7fe9f7400700 1 Processor -- start 2024-04-20T22:24:29.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:29.420+0000 7fe9f7400700 1 -- start start 2024-04-20T22:24:29.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:29.421+0000 7fe9f7400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9f0102470 0x7fe9f0198190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:24:29.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:29.421+0000 7fe9f7400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe9f0108470 0x7fe9f01986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:24:29.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:29.421+0000 7fe9f7400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9f0198db0 con 0x7fe9f0102470 2024-04-20T22:24:29.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:29.421+0000 7fe9f7400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9f019cb40 con 0x7fe9f0108470 2024-04-20T22:24:29.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:29.421+0000 7fe9f4a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9f0102470 0x7fe9f0198190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:24:29.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:29.421+0000 7fe9f4a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9f0102470 0x7fe9f0198190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45886/0 (socket says 172.21.15.62:45886) 2024-04-20T22:24:29.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:29.421+0000 7fe9f4a00700 1 -- 172.21.15.62:0/2435292640 learned_addr learned my addr 172.21.15.62:0/2435292640 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:24:29.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:29.421+0000 7fe9efe00700 1 --2- 172.21.15.62:0/2435292640 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe9f0108470 0x7fe9f01986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:24:29.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:29.421+0000 7fe9f4a00700 1 -- 172.21.15.62:0/2435292640 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe9f0108470 msgr2=0x7fe9f01986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:24:29.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:29.421+0000 7fe9f4a00700 1 --2- 172.21.15.62:0/2435292640 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe9f0108470 0x7fe9f01986d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:24:29.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:29.421+0000 7fe9f4a00700 1 -- 172.21.15.62:0/2435292640 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9dc0095f0 con 0x7fe9f0102470 2024-04-20T22:24:29.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:29.421+0000 7fe9efe00700 1 --2- 172.21.15.62:0/2435292640 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe9f0108470 0x7fe9f01986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T22:24:29.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:29.422+0000 7fe9f4a00700 1 --2- 172.21.15.62:0/2435292640 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9f0102470 0x7fe9f0198190 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7fe9e400b4b0 tx=0x7fe9e400b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:24:29.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:29.422+0000 7fe9ed600700 1 -- 172.21.15.62:0/2435292640 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe9e4011800 con 0x7fe9f0102470 2024-04-20T22:24:29.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:29.422+0000 7fe9ed600700 1 -- 172.21.15.62:0/2435292640 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe9e4011e20 con 0x7fe9f0102470 2024-04-20T22:24:29.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:29.422+0000 7fe9f7400700 1 -- 172.21.15.62:0/2435292640 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9f019ce20 con 0x7fe9f0102470 2024-04-20T22:24:29.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:29.422+0000 7fe9ed600700 1 -- 172.21.15.62:0/2435292640 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe9e400f2e0 con 0x7fe9f0102470 2024-04-20T22:24:29.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:29.422+0000 7fe9f7400700 1 -- 172.21.15.62:0/2435292640 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9f019d370 con 0x7fe9f0102470 2024-04-20T22:24:29.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:29.423+0000 7fe9f7400700 1 -- 172.21.15.62:0/2435292640 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9f010aa00 con 0x7fe9f0102470 2024-04-20T22:24:29.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:29.424+0000 7fe9ed600700 1 -- 172.21.15.62:0/2435292640 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe9e4010380 con 0x7fe9f0102470 2024-04-20T22:24:29.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:29.425+0000 7fe9ed600700 1 --2- 172.21.15.62:0/2435292640 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe9d806ecc0 0x7fe9d8071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:24:29.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:29.425+0000 7fe9ed600700 1 -- 172.21.15.62:0/2435292640 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fe9e408ea30 con 0x7fe9f0102470 2024-04-20T22:24:29.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:29.426+0000 7fe9efe00700 1 --2- 172.21.15.62:0/2435292640 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe9d806ecc0 0x7fe9d8071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:24:29.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:29.426+0000 7fe9efe00700 1 --2- 172.21.15.62:0/2435292640 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe9d806ecc0 0x7fe9d8071170 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fe9dc00b5c0 tx=0x7fe9dc005480 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:24:29.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:29.430+0000 7fe9ed600700 1 -- 172.21.15.62:0/2435292640 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fe9e4057870 con 0x7fe9f0102470 2024-04-20T22:24:29.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:24:29 smithi189 ceph-mon[57022]: pgmap v1478: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:24:29.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:24:29 smithi189 ceph-mon[57022]: from='client.36580 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:24:29.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:29.764+0000 7fe9f7400700 1 -- 172.21.15.62:0/2435292640 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fe9f004f2a0 con 0x7fe9f0102470 2024-04-20T22:24:29.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:29.765+0000 7fe9ed600700 1 -- 172.21.15.62:0/2435292640 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7fe9e405af30 con 0x7fe9f0102470 2024-04-20T22:24:29.764 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:24:29.764 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T22:24:29.764 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:24:29.764 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:24:29.765 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T22:24:29.765 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:24:29.765 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:24:29.765 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T22:24:29.765 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 6 2024-04-20T22:24:29.765 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:24:29.765 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T22:24:29.765 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 4 2024-04-20T22:24:29.765 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:24:29.765 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T22:24:29.765 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 14 2024-04-20T22:24:29.765 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T22:24:29.765 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:24:29.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:29.768+0000 7fe9f7400700 1 -- 172.21.15.62:0/2435292640 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe9d806ecc0 msgr2=0x7fe9d8071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:24:29.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:29.768+0000 7fe9f7400700 1 --2- 172.21.15.62:0/2435292640 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe9d806ecc0 0x7fe9d8071170 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fe9dc00b5c0 tx=0x7fe9dc005480 comp rx=0 tx=0).stop 2024-04-20T22:24:29.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:29.768+0000 7fe9f7400700 1 -- 172.21.15.62:0/2435292640 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9f0102470 msgr2=0x7fe9f0198190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:24:29.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:29.768+0000 7fe9f7400700 1 --2- 172.21.15.62:0/2435292640 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9f0102470 0x7fe9f0198190 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7fe9e400b4b0 tx=0x7fe9e400b870 comp rx=0 tx=0).stop 2024-04-20T22:24:29.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:29.768+0000 7fe9f7400700 1 -- 172.21.15.62:0/2435292640 shutdown_connections 2024-04-20T22:24:29.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:29.768+0000 7fe9f7400700 1 --2- 172.21.15.62:0/2435292640 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe9d806ecc0 0x7fe9d8071170 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:24:29.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:29.768+0000 7fe9f7400700 1 --2- 172.21.15.62:0/2435292640 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9f0102470 0x7fe9f0198190 unknown :-1 s=CLOSED pgs=1260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:24:29.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:29.768+0000 7fe9f7400700 1 --2- 172.21.15.62:0/2435292640 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe9f0108470 0x7fe9f01986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:24:29.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:29.768+0000 7fe9f7400700 1 -- 172.21.15.62:0/2435292640 >> 172.21.15.62:0/2435292640 conn(0x7fe9f00fdf90 msgr2=0x7fe9f00feb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:24:29.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:29.769+0000 7fe9f7400700 1 -- 172.21.15.62:0/2435292640 shutdown_connections 2024-04-20T22:24:29.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:29.769+0000 7fe9f7400700 1 -- 172.21.15.62:0/2435292640 wait complete. 2024-04-20T22:24:29.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:29.934+0000 7f81fc000700 1 -- 172.21.15.62:0/2736934714 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f81f40687a0 msgr2=0x7f81f4068b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:24:29.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:29.934+0000 7f81fc000700 1 --2- 172.21.15.62:0/2736934714 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f81f40687a0 0x7f81f4068b70 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f81e8009910 tx=0x7f81e8009c20 comp rx=0 tx=0).stop 2024-04-20T22:24:29.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:29.934+0000 7f81fc000700 1 -- 172.21.15.62:0/2736934714 shutdown_connections 2024-04-20T22:24:29.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:29.934+0000 7f81fc000700 1 --2- 172.21.15.62:0/2736934714 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f81f4069140 0x7f81f4104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:24:29.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:29.934+0000 7f81fc000700 1 --2- 172.21.15.62:0/2736934714 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f81f40687a0 0x7f81f4068b70 unknown :-1 s=CLOSED pgs=1261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:24:29.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:29.934+0000 7f81fc000700 1 -- 172.21.15.62:0/2736934714 >> 172.21.15.62:0/2736934714 conn(0x7f81f4075130 msgr2=0x7f81f4075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:24:29.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:29.935+0000 7f81fc000700 1 -- 172.21.15.62:0/2736934714 shutdown_connections 2024-04-20T22:24:29.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:29.935+0000 7f81fc000700 1 -- 172.21.15.62:0/2736934714 wait complete. 2024-04-20T22:24:29.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:29.936+0000 7f81fc000700 1 Processor -- start 2024-04-20T22:24:29.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:29.936+0000 7f81fc000700 1 -- start start 2024-04-20T22:24:29.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:29.936+0000 7f81fc000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f81f40687a0 0x7f81f40ffca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:24:29.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:29.936+0000 7f81fc000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f81f4069140 0x7f81f41001e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:24:29.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:29.936+0000 7f81fc000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81f4100720 con 0x7f81f4069140 2024-04-20T22:24:29.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:29.936+0000 7f81fc000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81f4100860 con 0x7f81f40687a0 2024-04-20T22:24:29.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:29.937+0000 7f81f8c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f81f4069140 0x7f81f41001e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:24:29.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:29.937+0000 7f81f8c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f81f4069140 0x7f81f41001e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45904/0 (socket says 172.21.15.62:45904) 2024-04-20T22:24:29.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:29.937+0000 7f81f8c00700 1 -- 172.21.15.62:0/770177920 learned_addr learned my addr 172.21.15.62:0/770177920 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:24:29.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:29.937+0000 7f81f9600700 1 --2- 172.21.15.62:0/770177920 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f81f40687a0 0x7f81f40ffca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:24:29.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:29.937+0000 7f81f8c00700 1 -- 172.21.15.62:0/770177920 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f81f40687a0 msgr2=0x7f81f40ffca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:24:29.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:29.937+0000 7f81f8c00700 1 --2- 172.21.15.62:0/770177920 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f81f40687a0 0x7f81f40ffca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:24:29.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:29.937+0000 7f81f8c00700 1 -- 172.21.15.62:0/770177920 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81e80095f0 con 0x7f81f4069140 2024-04-20T22:24:29.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:29.938+0000 7f81f8c00700 1 --2- 172.21.15.62:0/770177920 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f81f4069140 0x7f81f41001e0 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7f81e000b480 tx=0x7f81e000b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:24:29.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:29.938+0000 7f81f2000700 1 -- 172.21.15.62:0/770177920 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f81e00118c0 con 0x7f81f4069140 2024-04-20T22:24:29.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:29.938+0000 7f81f2000700 1 -- 172.21.15.62:0/770177920 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f81e0010380 con 0x7f81f4069140 2024-04-20T22:24:29.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:29.938+0000 7f81fc000700 1 -- 172.21.15.62:0/770177920 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81f41a2130 con 0x7f81f4069140 2024-04-20T22:24:29.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:29.938+0000 7f81f2000700 1 -- 172.21.15.62:0/770177920 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f81e000f4f0 con 0x7f81f4069140 2024-04-20T22:24:29.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:29.938+0000 7f81fc000700 1 -- 172.21.15.62:0/770177920 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81f41a2540 con 0x7f81f4069140 2024-04-20T22:24:29.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:29.939+0000 7f81fc000700 1 -- 172.21.15.62:0/770177920 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81f4108770 con 0x7f81f4069140 2024-04-20T22:24:29.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:29.940+0000 7f81f2000700 1 -- 172.21.15.62:0/770177920 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f81e00104f0 con 0x7f81f4069140 2024-04-20T22:24:29.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:29.941+0000 7f81f2000700 1 --2- 172.21.15.62:0/770177920 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f81dc06ecc0 0x7f81dc071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:24:29.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:29.941+0000 7f81f2000700 1 -- 172.21.15.62:0/770177920 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f81e008ea70 con 0x7f81f4069140 2024-04-20T22:24:29.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:29.942+0000 7f81f9600700 1 --2- 172.21.15.62:0/770177920 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f81dc06ecc0 0x7f81dc071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:24:29.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:29.943+0000 7f81f9600700 1 --2- 172.21.15.62:0/770177920 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f81dc06ecc0 0x7f81dc071170 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f81e8006010 tx=0x7f81e801a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:24:29.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:29.947+0000 7f81f2000700 1 -- 172.21.15.62:0/770177920 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f81e0057800 con 0x7f81f4069140 2024-04-20T22:24:30.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:30.233+0000 7f81fc000700 1 -- 172.21.15.62:0/770177920 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f81f404f2a0 con 0x7f81f4069140 2024-04-20T22:24:30.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:30.234+0000 7f81f2000700 1 -- 172.21.15.62:0/770177920 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 29 v29) v1 ==== 76+0+1887 (secure 0 0 0) 0x7f81e005aec0 con 0x7f81f4069140 2024-04-20T22:24:30.235 INFO:teuthology.orchestra.run.smithi062.stdout:e29 2024-04-20T22:24:30.235 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T22:24:30.235 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:24:30.235 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T22:24:30.235 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:24:30.235 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T22:24:30.235 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T22:24:30.235 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 20 2024-04-20T22:24:30.235 INFO:teuthology.orchestra.run.smithi062.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-20T22:24:30.235 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T22:24:30.235 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T21:42:01.559061+0000 2024-04-20T22:24:30.235 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T22:24:30.236 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T22:24:30.236 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T22:24:30.236 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T22:24:30.236 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T22:24:30.236 INFO:teuthology.orchestra.run.smithi062.stdout:max_xattr_size 65536 2024-04-20T22:24:30.236 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T22:24:30.236 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T22:24:30.236 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 81 2024-04-20T22:24:30.236 INFO:teuthology.orchestra.run.smithi062.stdout:snap_rstat 1 2024-04-20T22:24:30.236 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:24:30.236 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 2 2024-04-20T22:24:30.236 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1 2024-04-20T22:24:30.236 INFO:teuthology.orchestra.run.smithi062.stdout:up {} 2024-04-20T22:24:30.236 INFO:teuthology.orchestra.run.smithi062.stdout:failed 0,1 2024-04-20T22:24:30.236 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T22:24:30.237 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T22:24:30.237 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T22:24:30.237 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T22:24:30.237 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data enabled 2024-04-20T22:24:30.237 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T22:24:30.237 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T22:24:30.237 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 1 2024-04-20T22:24:30.237 INFO:teuthology.orchestra.run.smithi062.stdout:qdb_cluster leader: 0 members: 2024-04-20T22:24:30.237 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:24:30.237 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:24:30.237 INFO:teuthology.orchestra.run.smithi062.stdout:Standby daemons: 2024-04-20T22:24:30.237 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:24:30.237 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ryqblb{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6826/4113916205,v1:172.21.15.62:6827/4113916205] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:24:30.237 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ozizwi{-1:34476} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6828/3801975165,v1:172.21.15.62:6829/3801975165] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:24:30.237 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.dvimer{-1:44727} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6824/3576699232,v1:172.21.15.189:6825/3576699232] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:24:30.237 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.xbgspw{-1:45075} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6826/212905792,v1:172.21.15.189:6827/212905792] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:24:30.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:30.237+0000 7f81fc000700 1 -- 172.21.15.62:0/770177920 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f81dc06ecc0 msgr2=0x7f81dc071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:24:30.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:30.237+0000 7f81fc000700 1 --2- 172.21.15.62:0/770177920 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f81dc06ecc0 0x7f81dc071170 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f81e8006010 tx=0x7f81e801a040 comp rx=0 tx=0).stop 2024-04-20T22:24:30.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:30.237+0000 7f81fc000700 1 -- 172.21.15.62:0/770177920 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f81f4069140 msgr2=0x7f81f41001e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:24:30.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:30.237+0000 7f81fc000700 1 --2- 172.21.15.62:0/770177920 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f81f4069140 0x7f81f41001e0 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7f81e000b480 tx=0x7f81e000b840 comp rx=0 tx=0).stop 2024-04-20T22:24:30.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:30.237+0000 7f81fc000700 1 -- 172.21.15.62:0/770177920 shutdown_connections 2024-04-20T22:24:30.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:30.237+0000 7f81fc000700 1 --2- 172.21.15.62:0/770177920 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f81dc06ecc0 0x7f81dc071170 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:24:30.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:30.237+0000 7f81fc000700 1 --2- 172.21.15.62:0/770177920 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f81f40687a0 0x7f81f40ffca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:24:30.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:30.237+0000 7f81fc000700 1 --2- 172.21.15.62:0/770177920 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f81f4069140 0x7f81f41001e0 unknown :-1 s=CLOSED pgs=1262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:24:30.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:30.237+0000 7f81fc000700 1 -- 172.21.15.62:0/770177920 >> 172.21.15.62:0/770177920 conn(0x7f81f4075130 msgr2=0x7f81f40fe7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:24:30.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:30.238+0000 7f81fc000700 1 -- 172.21.15.62:0/770177920 shutdown_connections 2024-04-20T22:24:30.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:30.238+0000 7f81fc000700 1 -- 172.21.15.62:0/770177920 wait complete. 2024-04-20T22:24:30.239 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 29 2024-04-20T22:24:30.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:30.390+0000 7fda4d200700 1 -- 172.21.15.62:0/1813817094 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fda480739c0 msgr2=0x7fda4810c5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:24:30.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:30.390+0000 7fda4d200700 1 --2- 172.21.15.62:0/1813817094 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fda480739c0 0x7fda4810c5c0 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7fda34009940 tx=0x7fda34009c50 comp rx=0 tx=0).stop 2024-04-20T22:24:30.391 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:24:30 smithi062 ceph-mon[76322]: from='client.36584 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:24:30.391 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:24:30 smithi062 ceph-mon[76322]: from='client.36588 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:24:30.391 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:24:30 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/2435292640' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:24:30.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:30.390+0000 7fda4d200700 1 -- 172.21.15.62:0/1813817094 shutdown_connections 2024-04-20T22:24:30.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:30.390+0000 7fda4d200700 1 --2- 172.21.15.62:0/1813817094 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fda480739c0 0x7fda4810c5c0 unknown :-1 s=CLOSED pgs=1263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:24:30.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:30.390+0000 7fda4d200700 1 --2- 172.21.15.62:0/1813817094 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fda480730b0 0x7fda48073480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:24:30.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:30.390+0000 7fda4d200700 1 -- 172.21.15.62:0/1813817094 >> 172.21.15.62:0/1813817094 conn(0x7fda480fbda0 msgr2=0x7fda480fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:24:30.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:30.391+0000 7fda4d200700 1 -- 172.21.15.62:0/1813817094 shutdown_connections 2024-04-20T22:24:30.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:30.391+0000 7fda4d200700 1 -- 172.21.15.62:0/1813817094 wait complete. 2024-04-20T22:24:30.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:30.392+0000 7fda4d200700 1 Processor -- start 2024-04-20T22:24:30.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:30.392+0000 7fda4d200700 1 -- start start 2024-04-20T22:24:30.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:30.392+0000 7fda4d200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fda480730b0 0x7fda48198210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:24:30.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:30.392+0000 7fda4d200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fda480739c0 0x7fda48198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:24:30.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:30.392+0000 7fda4d200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda48198e30 con 0x7fda480730b0 2024-04-20T22:24:30.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:30.392+0000 7fda4d200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda4819cbc0 con 0x7fda480739c0 2024-04-20T22:24:30.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:30.393+0000 7fda45e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fda480730b0 0x7fda48198210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:24:30.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:30.393+0000 7fda45e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fda480730b0 0x7fda48198210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45916/0 (socket says 172.21.15.62:45916) 2024-04-20T22:24:30.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:30.393+0000 7fda45e00700 1 -- 172.21.15.62:0/2964921234 learned_addr learned my addr 172.21.15.62:0/2964921234 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:24:30.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:30.393+0000 7fda45400700 1 --2- 172.21.15.62:0/2964921234 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fda480739c0 0x7fda48198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:24:30.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:30.393+0000 7fda45e00700 1 -- 172.21.15.62:0/2964921234 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fda480739c0 msgr2=0x7fda48198750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:24:30.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:30.393+0000 7fda45e00700 1 --2- 172.21.15.62:0/2964921234 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fda480739c0 0x7fda48198750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:24:30.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:30.393+0000 7fda45e00700 1 -- 172.21.15.62:0/2964921234 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda340095f0 con 0x7fda480730b0 2024-04-20T22:24:30.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:30.394+0000 7fda45e00700 1 --2- 172.21.15.62:0/2964921234 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fda480730b0 0x7fda48198210 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7fda3800ada0 tx=0x7fda3800c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:24:30.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:30.394+0000 7fda3ea00700 1 -- 172.21.15.62:0/2964921234 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fda3800f070 con 0x7fda480730b0 2024-04-20T22:24:30.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:30.394+0000 7fda4d200700 1 -- 172.21.15.62:0/2964921234 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda4819cea0 con 0x7fda480730b0 2024-04-20T22:24:30.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:30.394+0000 7fda3ea00700 1 -- 172.21.15.62:0/2964921234 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fda38007bf0 con 0x7fda480730b0 2024-04-20T22:24:30.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:30.394+0000 7fda3ea00700 1 -- 172.21.15.62:0/2964921234 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fda38005690 con 0x7fda480730b0 2024-04-20T22:24:30.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:30.394+0000 7fda4d200700 1 -- 172.21.15.62:0/2964921234 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda4819d3f0 con 0x7fda480730b0 2024-04-20T22:24:30.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:30.395+0000 7fda4d200700 1 -- 172.21.15.62:0/2964921234 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda48109cc0 con 0x7fda480730b0 2024-04-20T22:24:30.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:30.396+0000 7fda3ea00700 1 -- 172.21.15.62:0/2964921234 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fda38005900 con 0x7fda480730b0 2024-04-20T22:24:30.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:30.397+0000 7fda3ea00700 1 --2- 172.21.15.62:0/2964921234 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fda2c06ecc0 0x7fda2c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:24:30.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:30.397+0000 7fda3ea00700 1 -- 172.21.15.62:0/2964921234 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fda3808e6a0 con 0x7fda480730b0 2024-04-20T22:24:30.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:30.397+0000 7fda45400700 1 --2- 172.21.15.62:0/2964921234 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fda2c06ecc0 0x7fda2c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:24:30.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:30.398+0000 7fda45400700 1 --2- 172.21.15.62:0/2964921234 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fda2c06ecc0 0x7fda2c071170 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fda3400b870 tx=0x7fda34005fb0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:24:30.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:30.402+0000 7fda3ea00700 1 -- 172.21.15.62:0/2964921234 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fda38057450 con 0x7fda480730b0 2024-04-20T22:24:30.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:24:30 smithi189 ceph-mon[57022]: from='client.36584 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:24:30.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:24:30 smithi189 ceph-mon[57022]: from='client.36588 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:24:30.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:24:30 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/2435292640' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:24:30.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:30.657+0000 7fda4d200700 1 -- 172.21.15.62:0/2964921234 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fda4819d610 con 0x7fda2c06ecc0 2024-04-20T22:24:30.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:30.660+0000 7fda3ea00700 1 -- 172.21.15.62:0/2964921234 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fda4819d610 con 0x7fda2c06ecc0 2024-04-20T22:24:30.660 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:24:30.660 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T22:24:30.660 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T22:24:30.660 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-20T22:24:30.661 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T22:24:30.661 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr", 2024-04-20T22:24:30.661 INFO:teuthology.orchestra.run.smithi062.stdout: "osd", 2024-04-20T22:24:30.661 INFO:teuthology.orchestra.run.smithi062.stdout: "mds", 2024-04-20T22:24:30.661 INFO:teuthology.orchestra.run.smithi062.stdout: "crash", 2024-04-20T22:24:30.661 INFO:teuthology.orchestra.run.smithi062.stdout: "mon" 2024-04-20T22:24:30.661 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T22:24:30.661 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "16/23 daemons upgraded", 2024-04-20T22:24:30.661 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading mds daemons", 2024-04-20T22:24:30.661 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T22:24:30.661 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:24:30.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:30.662+0000 7fda4d200700 1 -- 172.21.15.62:0/2964921234 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fda2c06ecc0 msgr2=0x7fda2c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:24:30.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:30.662+0000 7fda4d200700 1 --2- 172.21.15.62:0/2964921234 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fda2c06ecc0 0x7fda2c071170 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fda3400b870 tx=0x7fda34005fb0 comp rx=0 tx=0).stop 2024-04-20T22:24:30.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:30.662+0000 7fda4d200700 1 -- 172.21.15.62:0/2964921234 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fda480730b0 msgr2=0x7fda48198210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:24:30.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:30.663+0000 7fda4d200700 1 --2- 172.21.15.62:0/2964921234 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fda480730b0 0x7fda48198210 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7fda3800ada0 tx=0x7fda3800c330 comp rx=0 tx=0).stop 2024-04-20T22:24:30.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:30.663+0000 7fda4d200700 1 -- 172.21.15.62:0/2964921234 shutdown_connections 2024-04-20T22:24:30.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:30.663+0000 7fda4d200700 1 --2- 172.21.15.62:0/2964921234 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fda2c06ecc0 0x7fda2c071170 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:24:30.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:30.663+0000 7fda4d200700 1 --2- 172.21.15.62:0/2964921234 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fda480730b0 0x7fda48198210 unknown :-1 s=CLOSED pgs=1264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:24:30.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:30.663+0000 7fda4d200700 1 --2- 172.21.15.62:0/2964921234 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fda480739c0 0x7fda48198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:24:30.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:30.663+0000 7fda4d200700 1 -- 172.21.15.62:0/2964921234 >> 172.21.15.62:0/2964921234 conn(0x7fda480fbda0 msgr2=0x7fda48106e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:24:30.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:30.663+0000 7fda4d200700 1 -- 172.21.15.62:0/2964921234 shutdown_connections 2024-04-20T22:24:30.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:30.663+0000 7fda4d200700 1 -- 172.21.15.62:0/2964921234 wait complete. 2024-04-20T22:24:30.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:30.828+0000 7f2352000700 1 -- 172.21.15.62:0/973037352 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f234c106b80 msgr2=0x7f234c106ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:24:30.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:30.828+0000 7f2352000700 1 --2- 172.21.15.62:0/973037352 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f234c106b80 0x7f234c106ff0 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7f2338009910 tx=0x7f2338009c20 comp rx=0 tx=0).stop 2024-04-20T22:24:30.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:30.829+0000 7f2352000700 1 -- 172.21.15.62:0/973037352 shutdown_connections 2024-04-20T22:24:30.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:30.829+0000 7f2352000700 1 --2- 172.21.15.62:0/973037352 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f234c106b80 0x7f234c106ff0 unknown :-1 s=CLOSED pgs=1265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:24:30.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:30.829+0000 7f2352000700 1 --2- 172.21.15.62:0/973037352 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f234c10c660 0x7f234c10ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:24:30.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:30.829+0000 7f2352000700 1 -- 172.21.15.62:0/973037352 >> 172.21.15.62:0/973037352 conn(0x7f234c075e90 msgr2=0x7f234c0782a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:24:30.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:30.829+0000 7f2352000700 1 -- 172.21.15.62:0/973037352 shutdown_connections 2024-04-20T22:24:30.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:30.829+0000 7f2352000700 1 -- 172.21.15.62:0/973037352 wait complete. 2024-04-20T22:24:30.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:30.830+0000 7f2352000700 1 Processor -- start 2024-04-20T22:24:30.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:30.830+0000 7f2352000700 1 -- start start 2024-04-20T22:24:30.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:30.830+0000 7f2352000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f234c106b80 0x7f234c19c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:24:30.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:30.830+0000 7f2352000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f234c10c660 0x7f234c19c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:24:30.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:30.830+0000 7f2352000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f234c19ce00 con 0x7f234c106b80 2024-04-20T22:24:30.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:30.830+0000 7f2352000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f234c1a0b90 con 0x7f234c10c660 2024-04-20T22:24:30.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:30.831+0000 7f2350c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f234c106b80 0x7f234c19c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:24:30.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:30.831+0000 7f2350c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f234c106b80 0x7f234c19c1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45938/0 (socket says 172.21.15.62:45938) 2024-04-20T22:24:30.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:30.831+0000 7f2350c00700 1 -- 172.21.15.62:0/665094461 learned_addr learned my addr 172.21.15.62:0/665094461 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:24:30.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:30.831+0000 7f2350c00700 1 -- 172.21.15.62:0/665094461 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f234c10c660 msgr2=0x7f234c19c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:24:30.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:30.831+0000 7f234be00700 1 --2- 172.21.15.62:0/665094461 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f234c10c660 0x7f234c19c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:24:30.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:30.831+0000 7f2350c00700 1 --2- 172.21.15.62:0/665094461 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f234c10c660 0x7f234c19c720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:24:30.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:30.831+0000 7f2350c00700 1 -- 172.21.15.62:0/665094461 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23380095f0 con 0x7f234c106b80 2024-04-20T22:24:30.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:30.831+0000 7f2350c00700 1 --2- 172.21.15.62:0/665094461 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f234c106b80 0x7f234c19c1e0 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7f234000c8a0 tx=0x7f234000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:24:30.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:30.832+0000 7f2349600700 1 -- 172.21.15.62:0/665094461 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f234000cea0 con 0x7f234c106b80 2024-04-20T22:24:30.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:30.832+0000 7f2349600700 1 -- 172.21.15.62:0/665094461 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2340004830 con 0x7f234c106b80 2024-04-20T22:24:30.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:30.832+0000 7f2352000700 1 -- 172.21.15.62:0/665094461 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f234c1a0e70 con 0x7f234c106b80 2024-04-20T22:24:30.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:30.832+0000 7f2349600700 1 -- 172.21.15.62:0/665094461 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f23400055b0 con 0x7f234c106b80 2024-04-20T22:24:30.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:30.832+0000 7f2352000700 1 -- 172.21.15.62:0/665094461 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f234c1a13c0 con 0x7f234c106b80 2024-04-20T22:24:30.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:30.833+0000 7f2352000700 1 -- 172.21.15.62:0/665094461 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f234c10eb40 con 0x7f234c106b80 2024-04-20T22:24:30.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:30.834+0000 7f2349600700 1 -- 172.21.15.62:0/665094461 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f23400049a0 con 0x7f234c106b80 2024-04-20T22:24:30.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:30.836+0000 7f2349600700 1 --2- 172.21.15.62:0/665094461 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f233406ecc0 0x7f2334071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:24:30.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:30.836+0000 7f2349600700 1 -- 172.21.15.62:0/665094461 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f234008ebf0 con 0x7f234c106b80 2024-04-20T22:24:30.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:30.836+0000 7f234be00700 1 --2- 172.21.15.62:0/665094461 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f233406ecc0 0x7f2334071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:24:30.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:30.837+0000 7f234be00700 1 --2- 172.21.15.62:0/665094461 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f233406ecc0 0x7f2334071170 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f2338000c00 tx=0x7f2338005fd0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:24:30.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:30.841+0000 7f2349600700 1 -- 172.21.15.62:0/665094461 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f2340057900 con 0x7f234c106b80 2024-04-20T22:24:31.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:31.173+0000 7f2352000700 1 -- 172.21.15.62:0/665094461 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f234c062380 con 0x7f234c106b80 2024-04-20T22:24:31.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:31.174+0000 7f2349600700 1 -- 172.21.15.62:0/665094461 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7f234c062380 con 0x7f234c106b80 2024-04-20T22:24:31.175 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-20T22:24:31.175 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-20T22:24:31.175 INFO:teuthology.orchestra.run.smithi062.stdout: host smithi189 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi189/config 2024-04-20T22:24:31.175 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-20T22:24:31.175 INFO:teuthology.orchestra.run.smithi062.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi189 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp11670v2q:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:24:31.175 INFO:teuthology.orchestra.run.smithi062.stdout:Traceback (most recent call last): 2024-04-20T22:24:31.175 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-20T22:24:31.175 INFO:teuthology.orchestra.run.smithi062.stdout: return _run_code(code, main_globals, None, 2024-04-20T22:24:31.175 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-20T22:24:31.176 INFO:teuthology.orchestra.run.smithi062.stdout: exec(code, run_globals) 2024-04-20T22:24:31.176 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5618, in 2024-04-20T22:24:31.176 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5606, in main 2024-04-20T22:24:31.176 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 408, in _infer_config 2024-04-20T22:24:31.176 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 323, in _infer_fsid 2024-04-20T22:24:31.176 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 436, in _infer_image 2024-04-20T22:24:31.176 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 310, in _validate_fsid 2024-04-20T22:24:31.176 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 3359, in command_ceph_volume 2024-04-20T22:24:31.176 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-20T22:24:31.176 INFO:teuthology.orchestra.run.smithi062.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi189 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp11670v2q:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:24:31.176 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-20T22:24:31.176 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is degraded 2024-04-20T22:24:31.176 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T22:24:31.176 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T22:24:31.177 INFO:teuthology.orchestra.run.smithi062.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-20T22:24:31.177 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-20T22:24:31.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:31.176+0000 7f2352000700 1 -- 172.21.15.62:0/665094461 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f233406ecc0 msgr2=0x7f2334071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:24:31.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:31.176+0000 7f2352000700 1 --2- 172.21.15.62:0/665094461 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f233406ecc0 0x7f2334071170 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f2338000c00 tx=0x7f2338005fd0 comp rx=0 tx=0).stop 2024-04-20T22:24:31.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:31.177+0000 7f2352000700 1 -- 172.21.15.62:0/665094461 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f234c106b80 msgr2=0x7f234c19c1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:24:31.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:31.177+0000 7f2352000700 1 --2- 172.21.15.62:0/665094461 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f234c106b80 0x7f234c19c1e0 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7f234000c8a0 tx=0x7f234000cc60 comp rx=0 tx=0).stop 2024-04-20T22:24:31.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:31.177+0000 7f2352000700 1 -- 172.21.15.62:0/665094461 shutdown_connections 2024-04-20T22:24:31.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:31.177+0000 7f2352000700 1 --2- 172.21.15.62:0/665094461 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f233406ecc0 0x7f2334071170 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:24:31.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:31.177+0000 7f2352000700 1 --2- 172.21.15.62:0/665094461 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f234c106b80 0x7f234c19c1e0 unknown :-1 s=CLOSED pgs=1266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:24:31.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:31.177+0000 7f2352000700 1 --2- 172.21.15.62:0/665094461 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f234c10c660 0x7f234c19c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:24:31.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:31.177+0000 7f2352000700 1 -- 172.21.15.62:0/665094461 >> 172.21.15.62:0/665094461 conn(0x7f234c075e90 msgr2=0x7f234c0778d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:24:31.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:31.177+0000 7f2352000700 1 -- 172.21.15.62:0/665094461 shutdown_connections 2024-04-20T22:24:31.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:24:31.177+0000 7f2352000700 1 -- 172.21.15.62:0/665094461 wait complete. 2024-04-20T22:24:31.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:24:31 smithi189 ceph-mon[57022]: pgmap v1479: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:24:31.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:24:31 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/770177920' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:24:31.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:24:31 smithi062 ceph-mon[76322]: pgmap v1479: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:24:31.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:24:31 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/770177920' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:24:32.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:24:32 smithi189 ceph-mon[57022]: from='client.36600 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:24:32.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:24:32 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/665094461' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:24:32.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:24:32 smithi062 ceph-mon[76322]: from='client.36600 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:24:32.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:24:32 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/665094461' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:24:33.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:24:33 smithi189 ceph-mon[57022]: pgmap v1480: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:24:33.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:24:33 smithi062 ceph-mon[76322]: pgmap v1480: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:24:35.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:24:35 smithi189 ceph-mon[57022]: pgmap v1481: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:24:35.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:24:35 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:24:35.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:24:35 smithi062 ceph-mon[76322]: pgmap v1481: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:24:35.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:24:35 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:24:37.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:24:37 smithi189 ceph-mon[57022]: pgmap v1482: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:24:37.537 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:24:37 smithi062 ceph-mon[76322]: pgmap v1482: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:24:39.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:24:39 smithi189 ceph-mon[57022]: pgmap v1483: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:24:39.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:24:39 smithi062 ceph-mon[76322]: pgmap v1483: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:24:41.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:24:41 smithi189 ceph-mon[57022]: pgmap v1484: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:24:41.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:24:41 smithi062 ceph-mon[76322]: pgmap v1484: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:24:43.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:24:43 smithi062 ceph-mon[76322]: pgmap v1485: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:24:43.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:24:43 smithi189 ceph-mon[57022]: pgmap v1485: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:24:45.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:24:45 smithi189 ceph-mon[57022]: pgmap v1486: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:24:45.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:24:45 smithi062 ceph-mon[76322]: pgmap v1486: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:24:47.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:24:47 smithi189 ceph-mon[57022]: pgmap v1487: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:24:47.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:24:47 smithi062 ceph-mon[76322]: pgmap v1487: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:24:49.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:24:49 smithi189 ceph-mon[57022]: pgmap v1488: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:24:49.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:24:49 smithi062 ceph-mon[76322]: pgmap v1488: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:24:50.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:24:50 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:24:50.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:24:50 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:24:51.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:24:51 smithi189 ceph-mon[57022]: pgmap v1489: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:24:51.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:24:51 smithi062 ceph-mon[76322]: pgmap v1489: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:24:53.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:24:53 smithi189 ceph-mon[57022]: pgmap v1490: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:24:53.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:24:53 smithi062 ceph-mon[76322]: pgmap v1490: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:24:55.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:24:55 smithi189 ceph-mon[57022]: pgmap v1491: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:24:55.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:24:55 smithi062 ceph-mon[76322]: pgmap v1491: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:24:57.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:24:57 smithi189 ceph-mon[57022]: pgmap v1492: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:24:57.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:24:57 smithi062 ceph-mon[76322]: pgmap v1492: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:24:59.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:24:59 smithi189 ceph-mon[57022]: pgmap v1493: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:24:59.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:24:59 smithi062 ceph-mon[76322]: pgmap v1493: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:25:01.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:01.356+0000 7fa57ca00700 1 -- 172.21.15.62:0/1739055600 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa578073a00 msgr2=0x7fa578110d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:25:01.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:01.356+0000 7fa57ca00700 1 --2- 172.21.15.62:0/1739055600 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa578073a00 0x7fa578110d50 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7fa564009910 tx=0x7fa564009c20 comp rx=0 tx=0).stop 2024-04-20T22:25:01.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:01.356+0000 7fa57ca00700 1 -- 172.21.15.62:0/1739055600 shutdown_connections 2024-04-20T22:25:01.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:01.356+0000 7fa57ca00700 1 --2- 172.21.15.62:0/1739055600 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa578073a00 0x7fa578110d50 unknown :-1 s=CLOSED pgs=1267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:25:01.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:01.356+0000 7fa57ca00700 1 --2- 172.21.15.62:0/1739055600 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5780730f0 0x7fa5780734c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:25:01.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:01.356+0000 7fa57ca00700 1 -- 172.21.15.62:0/1739055600 >> 172.21.15.62:0/1739055600 conn(0x7fa5780fbda0 msgr2=0x7fa5780fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:25:01.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:01.356+0000 7fa57ca00700 1 -- 172.21.15.62:0/1739055600 shutdown_connections 2024-04-20T22:25:01.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:01.356+0000 7fa57ca00700 1 -- 172.21.15.62:0/1739055600 wait complete. 2024-04-20T22:25:01.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:01.358+0000 7fa57ca00700 1 Processor -- start 2024-04-20T22:25:01.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:01.358+0000 7fa57ca00700 1 -- start start 2024-04-20T22:25:01.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:01.358+0000 7fa57ca00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5780730f0 0x7fa57819c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:25:01.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:01.358+0000 7fa57ca00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa578073a00 0x7fa57819ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:25:01.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:01.358+0000 7fa57ca00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa57819d140 con 0x7fa578073a00 2024-04-20T22:25:01.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:01.358+0000 7fa57ca00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5781a0ed0 con 0x7fa5780730f0 2024-04-20T22:25:01.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:01.359+0000 7fa575400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa578073a00 0x7fa57819ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:25:01.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:01.359+0000 7fa575400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa578073a00 0x7fa57819ca60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:38208/0 (socket says 172.21.15.62:38208) 2024-04-20T22:25:01.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:01.359+0000 7fa575400700 1 -- 172.21.15.62:0/1425584632 learned_addr learned my addr 172.21.15.62:0/1425584632 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:25:01.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:01.359+0000 7fa575e00700 1 --2- 172.21.15.62:0/1425584632 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5780730f0 0x7fa57819c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:25:01.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:01.359+0000 7fa575400700 1 -- 172.21.15.62:0/1425584632 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5780730f0 msgr2=0x7fa57819c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:25:01.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:01.359+0000 7fa575400700 1 --2- 172.21.15.62:0/1425584632 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5780730f0 0x7fa57819c520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:25:01.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:01.359+0000 7fa575400700 1 -- 172.21.15.62:0/1425584632 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5640095f0 con 0x7fa578073a00 2024-04-20T22:25:01.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:01.359+0000 7fa575e00700 1 --2- 172.21.15.62:0/1425584632 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5780730f0 0x7fa57819c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T22:25:01.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:01.359+0000 7fa575400700 1 --2- 172.21.15.62:0/1425584632 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa578073a00 0x7fa57819ca60 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7fa56400b5c0 tx=0x7fa56400be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:25:01.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:01.360+0000 7fa56ea00700 1 -- 172.21.15.62:0/1425584632 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa56401d070 con 0x7fa578073a00 2024-04-20T22:25:01.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:01.360+0000 7fa56ea00700 1 -- 172.21.15.62:0/1425584632 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa56400f460 con 0x7fa578073a00 2024-04-20T22:25:01.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:01.360+0000 7fa57ca00700 1 -- 172.21.15.62:0/1425584632 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5781a11b0 con 0x7fa578073a00 2024-04-20T22:25:01.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:01.360+0000 7fa56ea00700 1 -- 172.21.15.62:0/1425584632 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa564017630 con 0x7fa578073a00 2024-04-20T22:25:01.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:01.360+0000 7fa57ca00700 1 -- 172.21.15.62:0/1425584632 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5781a1700 con 0x7fa578073a00 2024-04-20T22:25:01.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:01.361+0000 7fa57ca00700 1 -- 172.21.15.62:0/1425584632 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa57810e4d0 con 0x7fa578073a00 2024-04-20T22:25:01.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:01.362+0000 7fa56ea00700 1 -- 172.21.15.62:0/1425584632 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fa56400f5d0 con 0x7fa578073a00 2024-04-20T22:25:01.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:01.363+0000 7fa56ea00700 1 --2- 172.21.15.62:0/1425584632 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa55c06ecc0 0x7fa55c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:25:01.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:01.363+0000 7fa56ea00700 1 -- 172.21.15.62:0/1425584632 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fa564090490 con 0x7fa578073a00 2024-04-20T22:25:01.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:01.364+0000 7fa575e00700 1 --2- 172.21.15.62:0/1425584632 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa55c06ecc0 0x7fa55c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:25:01.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:01.364+0000 7fa575e00700 1 --2- 172.21.15.62:0/1425584632 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa55c06ecc0 0x7fa55c071170 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fa568007900 tx=0x7fa568008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:25:01.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:01.369+0000 7fa56ea00700 1 -- 172.21.15.62:0/1425584632 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fa5640591a0 con 0x7fa578073a00 2024-04-20T22:25:01.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:25:01 smithi189 ceph-mon[57022]: pgmap v1494: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:25:01.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:25:01 smithi062 ceph-mon[76322]: pgmap v1494: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:25:01.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:01.632+0000 7fa57ca00700 1 -- 172.21.15.62:0/1425584632 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa5781a1990 con 0x7fa55c06ecc0 2024-04-20T22:25:01.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:01.635+0000 7fa56ea00700 1 -- 172.21.15.62:0/1425584632 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fa5781a1990 con 0x7fa55c06ecc0 2024-04-20T22:25:01.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:01.638+0000 7fa57ca00700 1 -- 172.21.15.62:0/1425584632 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa55c06ecc0 msgr2=0x7fa55c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:25:01.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:01.638+0000 7fa57ca00700 1 --2- 172.21.15.62:0/1425584632 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa55c06ecc0 0x7fa55c071170 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fa568007900 tx=0x7fa568008040 comp rx=0 tx=0).stop 2024-04-20T22:25:01.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:01.638+0000 7fa57ca00700 1 -- 172.21.15.62:0/1425584632 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa578073a00 msgr2=0x7fa57819ca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:25:01.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:01.638+0000 7fa57ca00700 1 --2- 172.21.15.62:0/1425584632 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa578073a00 0x7fa57819ca60 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7fa56400b5c0 tx=0x7fa56400be00 comp rx=0 tx=0).stop 2024-04-20T22:25:01.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:01.638+0000 7fa57ca00700 1 -- 172.21.15.62:0/1425584632 shutdown_connections 2024-04-20T22:25:01.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:01.638+0000 7fa57ca00700 1 --2- 172.21.15.62:0/1425584632 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa55c06ecc0 0x7fa55c071170 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:25:01.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:01.638+0000 7fa57ca00700 1 --2- 172.21.15.62:0/1425584632 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5780730f0 0x7fa57819c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:25:01.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:01.638+0000 7fa57ca00700 1 --2- 172.21.15.62:0/1425584632 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa578073a00 0x7fa57819ca60 unknown :-1 s=CLOSED pgs=1268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:25:01.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:01.638+0000 7fa57ca00700 1 -- 172.21.15.62:0/1425584632 >> 172.21.15.62:0/1425584632 conn(0x7fa5780fbda0 msgr2=0x7fa578102860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:25:01.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:01.638+0000 7fa57ca00700 1 -- 172.21.15.62:0/1425584632 shutdown_connections 2024-04-20T22:25:01.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:01.638+0000 7fa57ca00700 1 -- 172.21.15.62:0/1425584632 wait complete. 2024-04-20T22:25:01.655 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T22:25:01.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:01.805+0000 7fa849000700 1 -- 172.21.15.62:0/2179047882 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa844073a00 msgr2=0x7fa844110b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:25:01.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:01.805+0000 7fa849000700 1 --2- 172.21.15.62:0/2179047882 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa844073a00 0x7fa844110b40 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7fa830009910 tx=0x7fa830009c20 comp rx=0 tx=0).stop 2024-04-20T22:25:01.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:01.806+0000 7fa849000700 1 -- 172.21.15.62:0/2179047882 shutdown_connections 2024-04-20T22:25:01.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:01.806+0000 7fa849000700 1 --2- 172.21.15.62:0/2179047882 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa844073a00 0x7fa844110b40 unknown :-1 s=CLOSED pgs=1269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:25:01.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:01.806+0000 7fa849000700 1 --2- 172.21.15.62:0/2179047882 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8440730f0 0x7fa8440734c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:25:01.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:01.806+0000 7fa849000700 1 -- 172.21.15.62:0/2179047882 >> 172.21.15.62:0/2179047882 conn(0x7fa8440fbbb0 msgr2=0x7fa8440fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:25:01.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:01.806+0000 7fa849000700 1 -- 172.21.15.62:0/2179047882 shutdown_connections 2024-04-20T22:25:01.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:01.806+0000 7fa849000700 1 -- 172.21.15.62:0/2179047882 wait complete. 2024-04-20T22:25:01.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:01.807+0000 7fa849000700 1 Processor -- start 2024-04-20T22:25:01.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:01.807+0000 7fa849000700 1 -- start start 2024-04-20T22:25:01.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:01.807+0000 7fa849000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8440730f0 0x7fa84419c210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:25:01.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:01.807+0000 7fa849000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa844073a00 0x7fa84419c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:25:01.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:01.807+0000 7fa849000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa84419ce30 con 0x7fa844073a00 2024-04-20T22:25:01.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:01.807+0000 7fa849000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8441a0bc0 con 0x7fa8440730f0 2024-04-20T22:25:01.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:01.808+0000 7fa842a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa844073a00 0x7fa84419c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:25:01.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:01.808+0000 7fa842a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa844073a00 0x7fa84419c750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:38222/0 (socket says 172.21.15.62:38222) 2024-04-20T22:25:01.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:01.808+0000 7fa842a00700 1 -- 172.21.15.62:0/1958874815 learned_addr learned my addr 172.21.15.62:0/1958874815 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:25:01.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:01.808+0000 7fa843400700 1 --2- 172.21.15.62:0/1958874815 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8440730f0 0x7fa84419c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:25:01.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:01.808+0000 7fa842a00700 1 -- 172.21.15.62:0/1958874815 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8440730f0 msgr2=0x7fa84419c210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:25:01.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:01.808+0000 7fa842a00700 1 --2- 172.21.15.62:0/1958874815 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8440730f0 0x7fa84419c210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:25:01.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:01.808+0000 7fa842a00700 1 -- 172.21.15.62:0/1958874815 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8300095f0 con 0x7fa844073a00 2024-04-20T22:25:01.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:01.809+0000 7fa842a00700 1 --2- 172.21.15.62:0/1958874815 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa844073a00 0x7fa84419c750 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7fa83000b5c0 tx=0x7fa83000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:25:01.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:01.809+0000 7fa83be00700 1 -- 172.21.15.62:0/1958874815 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa83001d070 con 0x7fa844073a00 2024-04-20T22:25:01.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:01.809+0000 7fa83be00700 1 -- 172.21.15.62:0/1958874815 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa830005e00 con 0x7fa844073a00 2024-04-20T22:25:01.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:01.809+0000 7fa83be00700 1 -- 172.21.15.62:0/1958874815 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa830017400 con 0x7fa844073a00 2024-04-20T22:25:01.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:01.809+0000 7fa849000700 1 -- 172.21.15.62:0/1958874815 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8441a0ea0 con 0x7fa844073a00 2024-04-20T22:25:01.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:01.809+0000 7fa849000700 1 -- 172.21.15.62:0/1958874815 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8441a13f0 con 0x7fa844073a00 2024-04-20T22:25:01.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:01.810+0000 7fa849000700 1 -- 172.21.15.62:0/1958874815 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa84410e2c0 con 0x7fa844073a00 2024-04-20T22:25:01.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:01.812+0000 7fa83be00700 1 -- 172.21.15.62:0/1958874815 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fa83000f460 con 0x7fa844073a00 2024-04-20T22:25:01.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:01.812+0000 7fa83be00700 1 --2- 172.21.15.62:0/1958874815 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa82806ecc0 0x7fa828071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:25:01.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:01.813+0000 7fa83be00700 1 -- 172.21.15.62:0/1958874815 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fa830090310 con 0x7fa844073a00 2024-04-20T22:25:01.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:01.813+0000 7fa843400700 1 --2- 172.21.15.62:0/1958874815 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa82806ecc0 0x7fa828071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:25:01.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:01.814+0000 7fa843400700 1 --2- 172.21.15.62:0/1958874815 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa82806ecc0 0x7fa828071170 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fa834005950 tx=0x7fa8340058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:25:01.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:01.817+0000 7fa83be00700 1 -- 172.21.15.62:0/1958874815 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fa8300590a0 con 0x7fa844073a00 2024-04-20T22:25:02.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:02.072+0000 7fa849000700 1 -- 172.21.15.62:0/1958874815 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa8441a16d0 con 0x7fa82806ecc0 2024-04-20T22:25:02.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:02.075+0000 7fa83be00700 1 -- 172.21.15.62:0/1958874815 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fa8441a16d0 con 0x7fa82806ecc0 2024-04-20T22:25:02.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:02.078+0000 7fa849000700 1 -- 172.21.15.62:0/1958874815 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa82806ecc0 msgr2=0x7fa828071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:25:02.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:02.078+0000 7fa849000700 1 --2- 172.21.15.62:0/1958874815 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa82806ecc0 0x7fa828071170 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fa834005950 tx=0x7fa8340058e0 comp rx=0 tx=0).stop 2024-04-20T22:25:02.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:02.078+0000 7fa849000700 1 -- 172.21.15.62:0/1958874815 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa844073a00 msgr2=0x7fa84419c750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:25:02.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:02.078+0000 7fa849000700 1 --2- 172.21.15.62:0/1958874815 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa844073a00 0x7fa84419c750 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7fa83000b5c0 tx=0x7fa83000be20 comp rx=0 tx=0).stop 2024-04-20T22:25:02.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:02.078+0000 7fa849000700 1 -- 172.21.15.62:0/1958874815 shutdown_connections 2024-04-20T22:25:02.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:02.078+0000 7fa849000700 1 --2- 172.21.15.62:0/1958874815 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa82806ecc0 0x7fa828071170 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:25:02.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:02.078+0000 7fa849000700 1 --2- 172.21.15.62:0/1958874815 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8440730f0 0x7fa84419c210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:25:02.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:02.078+0000 7fa849000700 1 --2- 172.21.15.62:0/1958874815 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa844073a00 0x7fa84419c750 unknown :-1 s=CLOSED pgs=1270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:25:02.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:02.078+0000 7fa849000700 1 -- 172.21.15.62:0/1958874815 >> 172.21.15.62:0/1958874815 conn(0x7fa8440fbbb0 msgr2=0x7fa844102670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:25:02.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:02.079+0000 7fa849000700 1 -- 172.21.15.62:0/1958874815 shutdown_connections 2024-04-20T22:25:02.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:02.079+0000 7fa849000700 1 -- 172.21.15.62:0/1958874815 wait complete. 2024-04-20T22:25:02.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:02.242+0000 7f32f5a00700 1 -- 172.21.15.62:0/3837554771 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f32f0104130 msgr2=0x7f32f0104500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:25:02.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:02.242+0000 7f32f5a00700 1 --2- 172.21.15.62:0/3837554771 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f32f0104130 0x7f32f0104500 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7f32e000b0a0 tx=0x7f32e000b3b0 comp rx=0 tx=0).stop 2024-04-20T22:25:02.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:02.242+0000 7f32f5a00700 1 -- 172.21.15.62:0/3837554771 shutdown_connections 2024-04-20T22:25:02.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:02.242+0000 7f32f5a00700 1 --2- 172.21.15.62:0/3837554771 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f32f00ffbd0 0x7f32f0100040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:25:02.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:02.242+0000 7f32f5a00700 1 --2- 172.21.15.62:0/3837554771 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f32f0104130 0x7f32f0104500 unknown :-1 s=CLOSED pgs=1271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:25:02.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:02.242+0000 7f32f5a00700 1 -- 172.21.15.62:0/3837554771 >> 172.21.15.62:0/3837554771 conn(0x7f32f0075130 msgr2=0x7f32f0075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:25:02.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:02.242+0000 7f32f5a00700 1 -- 172.21.15.62:0/3837554771 shutdown_connections 2024-04-20T22:25:02.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:02.242+0000 7f32f5a00700 1 -- 172.21.15.62:0/3837554771 wait complete. 2024-04-20T22:25:02.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:02.243+0000 7f32f5a00700 1 Processor -- start 2024-04-20T22:25:02.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:02.243+0000 7f32f5a00700 1 -- start start 2024-04-20T22:25:02.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:02.244+0000 7f32f5a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f32f00ffbd0 0x7f32f0198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:25:02.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:02.244+0000 7f32f5a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f32f0104130 0x7f32f0198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:25:02.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:02.244+0000 7f32f5a00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32f0198c50 con 0x7f32f00ffbd0 2024-04-20T22:25:02.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:02.244+0000 7f32f5a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32f019c9e0 con 0x7f32f0104130 2024-04-20T22:25:02.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:02.244+0000 7f32eea00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f32f00ffbd0 0x7f32f0198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:25:02.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:02.244+0000 7f32eea00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f32f00ffbd0 0x7f32f0198030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:38234/0 (socket says 172.21.15.62:38234) 2024-04-20T22:25:02.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:02.244+0000 7f32eea00700 1 -- 172.21.15.62:0/1822058531 learned_addr learned my addr 172.21.15.62:0/1822058531 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:25:02.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:02.244+0000 7f32eea00700 1 -- 172.21.15.62:0/1822058531 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f32f0104130 msgr2=0x7f32f0198570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:25:02.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:02.244+0000 7f32e6000700 1 --2- 172.21.15.62:0/1822058531 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f32f0104130 0x7f32f0198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:25:02.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:02.244+0000 7f32eea00700 1 --2- 172.21.15.62:0/1822058531 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f32f0104130 0x7f32f0198570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:25:02.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:02.244+0000 7f32eea00700 1 -- 172.21.15.62:0/1822058531 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32e0009d00 con 0x7f32f00ffbd0 2024-04-20T22:25:02.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:02.245+0000 7f32e6000700 1 --2- 172.21.15.62:0/1822058531 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f32f0104130 0x7f32f0198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T22:25:02.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:02.245+0000 7f32eea00700 1 --2- 172.21.15.62:0/1822058531 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f32f00ffbd0 0x7f32f0198030 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7f32e0000f80 tx=0x7f32e000cdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:25:02.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:02.245+0000 7f32e7e00700 1 -- 172.21.15.62:0/1822058531 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f32e000e040 con 0x7f32f00ffbd0 2024-04-20T22:25:02.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:02.245+0000 7f32e7e00700 1 -- 172.21.15.62:0/1822058531 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f32e0003de0 con 0x7f32f00ffbd0 2024-04-20T22:25:02.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:02.245+0000 7f32f5a00700 1 -- 172.21.15.62:0/1822058531 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32f019cc60 con 0x7f32f00ffbd0 2024-04-20T22:25:02.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:02.245+0000 7f32e7e00700 1 -- 172.21.15.62:0/1822058531 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f32e001d790 con 0x7f32f00ffbd0 2024-04-20T22:25:02.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:02.246+0000 7f32f5a00700 1 -- 172.21.15.62:0/1822058531 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32f019d150 con 0x7f32f00ffbd0 2024-04-20T22:25:02.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:02.247+0000 7f32f5a00700 1 -- 172.21.15.62:0/1822058531 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32f004f2a0 con 0x7f32f00ffbd0 2024-04-20T22:25:02.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:02.248+0000 7f32e7e00700 1 -- 172.21.15.62:0/1822058531 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f32e0004040 con 0x7f32f00ffbd0 2024-04-20T22:25:02.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:02.249+0000 7f32e7e00700 1 --2- 172.21.15.62:0/1822058531 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f32d406ecc0 0x7f32d4071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:25:02.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:02.249+0000 7f32e7e00700 1 -- 172.21.15.62:0/1822058531 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f32e0090740 con 0x7f32f00ffbd0 2024-04-20T22:25:02.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:02.249+0000 7f32e6000700 1 --2- 172.21.15.62:0/1822058531 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f32d406ecc0 0x7f32d4071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:25:02.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:02.250+0000 7f32e6000700 1 --2- 172.21.15.62:0/1822058531 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f32d406ecc0 0x7f32d4071170 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f32d8005fd0 tx=0x7f32d8005ee0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:25:02.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:02.254+0000 7f32e7e00700 1 -- 172.21.15.62:0/1822058531 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f32e0059450 con 0x7f32f00ffbd0 2024-04-20T22:25:02.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:02.501+0000 7f32f5a00700 1 -- 172.21.15.62:0/1822058531 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f32f019d4a0 con 0x7f32d406ecc0 2024-04-20T22:25:02.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:02.519+0000 7f32e7e00700 1 -- 172.21.15.62:0/1822058531 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f32f019d4a0 con 0x7f32d406ecc0 2024-04-20T22:25:02.520 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T22:25:02.520 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (50m) 11m ago 64m 14.6M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T22:25:02.521 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (65m) 11m ago 65m 11.6M - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T22:25:02.521 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (63m) 11m ago 63m 11.2M - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T22:25:02.521 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (46m) 11m ago 65m 6660k - 19.0.0-3133-ga8c84f59 76bc6b91b726 2825d5ba11c6 2024-04-20T22:25:02.521 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (46m) 11m ago 63m 6665k - 19.0.0-3133-ga8c84f59 76bc6b91b726 20f746fa1bba 2024-04-20T22:25:02.521 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (50m) 11m ago 64m 75.5M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T22:25:02.521 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (42m) 11m ago 57m 22.9M - 19.0.0-3133-ga8c84f59 76bc6b91b726 de80bc90a683 2024-04-20T22:25:02.521 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (42m) 11m ago 57m 23.0M - 19.0.0-3133-ga8c84f59 76bc6b91b726 cd793b7ee98e 2024-04-20T22:25:02.521 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (27m) 11m ago 57m 18.5M - 19.0.0-3133-ga8c84f59 76bc6b91b726 0e7b0bc4d1ab 2024-04-20T22:25:02.521 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (11m) 11m ago 57m 11.4M - 19.0.0-3133-ga8c84f59 76bc6b91b726 0af732ac9e06 2024-04-20T22:25:02.521 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (53m) 11m ago 66m 517M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T22:25:02.521 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (52m) 11m ago 63m 460M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T22:25:02.521 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (48m) 11m ago 66m 86.3M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 019eb1479485 2024-04-20T22:25:02.521 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (48m) 11m ago 63m 58.0M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 55aac560aab6 2024-04-20T22:25:02.521 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (52m) 11m ago 64m 8766k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T22:25:02.522 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (51m) 11m ago 63m 8711k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T22:25:02.522 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (46m) 11m ago 62m 91.3M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 ee2bb901f26e 2024-04-20T22:25:02.522 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (45m) 11m ago 61m 94.9M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 9075152d4162 2024-04-20T22:25:02.522 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (45m) 11m ago 61m 78.5M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 2d4f3e38dc0e 2024-04-20T22:25:02.522 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (44m) 11m ago 60m 94.1M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 8f97b17ae0eb 2024-04-20T22:25:02.522 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (43m) 11m ago 60m 96.1M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 653d8079967d 2024-04-20T22:25:02.522 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (43m) 11m ago 59m 90.9M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 596dded14c12 2024-04-20T22:25:02.522 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (51m) 11m ago 64m 90.4M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T22:25:02.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:02.522+0000 7f32f5a00700 1 -- 172.21.15.62:0/1822058531 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f32d406ecc0 msgr2=0x7f32d4071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:25:02.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:02.522+0000 7f32f5a00700 1 --2- 172.21.15.62:0/1822058531 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f32d406ecc0 0x7f32d4071170 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f32d8005fd0 tx=0x7f32d8005ee0 comp rx=0 tx=0).stop 2024-04-20T22:25:02.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:02.522+0000 7f32f5a00700 1 -- 172.21.15.62:0/1822058531 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f32f00ffbd0 msgr2=0x7f32f0198030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:25:02.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:02.522+0000 7f32f5a00700 1 --2- 172.21.15.62:0/1822058531 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f32f00ffbd0 0x7f32f0198030 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7f32e0000f80 tx=0x7f32e000cdd0 comp rx=0 tx=0).stop 2024-04-20T22:25:02.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:02.523+0000 7f32f5a00700 1 -- 172.21.15.62:0/1822058531 shutdown_connections 2024-04-20T22:25:02.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:02.523+0000 7f32f5a00700 1 --2- 172.21.15.62:0/1822058531 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f32d406ecc0 0x7f32d4071170 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:25:02.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:02.523+0000 7f32f5a00700 1 --2- 172.21.15.62:0/1822058531 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f32f00ffbd0 0x7f32f0198030 unknown :-1 s=CLOSED pgs=1272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:25:02.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:02.523+0000 7f32f5a00700 1 --2- 172.21.15.62:0/1822058531 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f32f0104130 0x7f32f0198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:25:02.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:02.523+0000 7f32f5a00700 1 -- 172.21.15.62:0/1822058531 >> 172.21.15.62:0/1822058531 conn(0x7f32f0075130 msgr2=0x7f32f01095e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:25:02.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:02.523+0000 7f32f5a00700 1 -- 172.21.15.62:0/1822058531 shutdown_connections 2024-04-20T22:25:02.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:02.523+0000 7f32f5a00700 1 -- 172.21.15.62:0/1822058531 wait complete. 2024-04-20T22:25:02.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:02.687+0000 7f140c000700 1 -- 172.21.15.62:0/1787746875 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f14041004e0 msgr2=0x7f14041008b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:25:02.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:02.687+0000 7f140c000700 1 --2- 172.21.15.62:0/1787746875 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f14041004e0 0x7f14041008b0 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7f13f8009910 tx=0x7f13f8009c20 comp rx=0 tx=0).stop 2024-04-20T22:25:02.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:02.687+0000 7f140c000700 1 -- 172.21.15.62:0/1787746875 shutdown_connections 2024-04-20T22:25:02.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:02.687+0000 7f140c000700 1 --2- 172.21.15.62:0/1787746875 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1404100e80 0x7f1404104e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:25:02.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:02.687+0000 7f140c000700 1 --2- 172.21.15.62:0/1787746875 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f14041004e0 0x7f14041008b0 unknown :-1 s=CLOSED pgs=1273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:25:02.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:02.687+0000 7f140c000700 1 -- 172.21.15.62:0/1787746875 >> 172.21.15.62:0/1787746875 conn(0x7f14040fbda0 msgr2=0x7f14040fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:25:02.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:02.687+0000 7f140c000700 1 -- 172.21.15.62:0/1787746875 shutdown_connections 2024-04-20T22:25:02.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:02.687+0000 7f140c000700 1 -- 172.21.15.62:0/1787746875 wait complete. 2024-04-20T22:25:02.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:02.688+0000 7f140c000700 1 Processor -- start 2024-04-20T22:25:02.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:02.688+0000 7f140c000700 1 -- start start 2024-04-20T22:25:02.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:02.688+0000 7f140c000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f14041004e0 0x7f14041980d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:25:02.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:02.688+0000 7f140c000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1404100e80 0x7f1404198610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:25:02.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:02.688+0000 7f140c000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1404198cf0 con 0x7f1404100e80 2024-04-20T22:25:02.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:02.688+0000 7f140c000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f140419ca80 con 0x7f14041004e0 2024-04-20T22:25:02.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:02.689+0000 7f1408c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1404100e80 0x7f1404198610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:25:02.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:02.689+0000 7f1408c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1404100e80 0x7f1404198610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:38240/0 (socket says 172.21.15.62:38240) 2024-04-20T22:25:02.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:02.689+0000 7f1408c00700 1 -- 172.21.15.62:0/2048499387 learned_addr learned my addr 172.21.15.62:0/2048499387 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:25:02.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:02.689+0000 7f1409600700 1 --2- 172.21.15.62:0/2048499387 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f14041004e0 0x7f14041980d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:25:02.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:02.689+0000 7f1408c00700 1 -- 172.21.15.62:0/2048499387 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f14041004e0 msgr2=0x7f14041980d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:25:02.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:02.689+0000 7f1408c00700 1 --2- 172.21.15.62:0/2048499387 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f14041004e0 0x7f14041980d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:25:02.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:02.689+0000 7f1408c00700 1 -- 172.21.15.62:0/2048499387 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13f80095f0 con 0x7f1404100e80 2024-04-20T22:25:02.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:02.690+0000 7f1408c00700 1 --2- 172.21.15.62:0/2048499387 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1404100e80 0x7f1404198610 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7f13f000d900 tx=0x7f13f000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:25:02.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:02.690+0000 7f1402000700 1 -- 172.21.15.62:0/2048499387 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f13f0009900 con 0x7f1404100e80 2024-04-20T22:25:02.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:02.690+0000 7f1402000700 1 -- 172.21.15.62:0/2048499387 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f13f0010460 con 0x7f1404100e80 2024-04-20T22:25:02.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:02.690+0000 7f1402000700 1 -- 172.21.15.62:0/2048499387 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f13f000f5f0 con 0x7f1404100e80 2024-04-20T22:25:02.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:02.690+0000 7f140c000700 1 -- 172.21.15.62:0/2048499387 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f140419cd60 con 0x7f1404100e80 2024-04-20T22:25:02.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:02.690+0000 7f140c000700 1 -- 172.21.15.62:0/2048499387 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f140419d2b0 con 0x7f1404100e80 2024-04-20T22:25:02.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:02.691+0000 7f140c000700 1 -- 172.21.15.62:0/2048499387 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14041045d0 con 0x7f1404100e80 2024-04-20T22:25:02.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:02.693+0000 7f1402000700 1 -- 172.21.15.62:0/2048499387 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f13f00105d0 con 0x7f1404100e80 2024-04-20T22:25:02.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:02.694+0000 7f1402000700 1 --2- 172.21.15.62:0/2048499387 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f13ec06ecc0 0x7f13ec071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:25:02.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:02.694+0000 7f1402000700 1 -- 172.21.15.62:0/2048499387 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f13f008eca0 con 0x7f1404100e80 2024-04-20T22:25:02.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:02.694+0000 7f1409600700 1 --2- 172.21.15.62:0/2048499387 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f13ec06ecc0 0x7f13ec071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:25:02.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:02.695+0000 7f1409600700 1 --2- 172.21.15.62:0/2048499387 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f13ec06ecc0 0x7f13ec071170 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f13f8006010 tx=0x7f13f801a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:25:02.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:02.699+0000 7f1402000700 1 -- 172.21.15.62:0/2048499387 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f13f0057a30 con 0x7f1404100e80 2024-04-20T22:25:03.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.026+0000 7f140c000700 1 -- 172.21.15.62:0/2048499387 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f140404f2a0 con 0x7f1404100e80 2024-04-20T22:25:03.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.027+0000 7f1402000700 1 -- 172.21.15.62:0/2048499387 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f13f005b0f0 con 0x7f1404100e80 2024-04-20T22:25:03.027 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:25:03.027 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T22:25:03.028 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:25:03.028 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:25:03.028 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T22:25:03.028 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:25:03.028 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:25:03.028 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T22:25:03.028 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 6 2024-04-20T22:25:03.028 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:25:03.028 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T22:25:03.028 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 4 2024-04-20T22:25:03.028 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:25:03.028 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T22:25:03.028 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 14 2024-04-20T22:25:03.028 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T22:25:03.028 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:25:03.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.030+0000 7f140c000700 1 -- 172.21.15.62:0/2048499387 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f13ec06ecc0 msgr2=0x7f13ec071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:25:03.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.030+0000 7f140c000700 1 --2- 172.21.15.62:0/2048499387 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f13ec06ecc0 0x7f13ec071170 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f13f8006010 tx=0x7f13f801a040 comp rx=0 tx=0).stop 2024-04-20T22:25:03.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.030+0000 7f140c000700 1 -- 172.21.15.62:0/2048499387 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1404100e80 msgr2=0x7f1404198610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:25:03.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.030+0000 7f140c000700 1 --2- 172.21.15.62:0/2048499387 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1404100e80 0x7f1404198610 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7f13f000d900 tx=0x7f13f000dcc0 comp rx=0 tx=0).stop 2024-04-20T22:25:03.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.030+0000 7f140c000700 1 -- 172.21.15.62:0/2048499387 shutdown_connections 2024-04-20T22:25:03.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.030+0000 7f140c000700 1 --2- 172.21.15.62:0/2048499387 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f13ec06ecc0 0x7f13ec071170 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:25:03.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.030+0000 7f140c000700 1 --2- 172.21.15.62:0/2048499387 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f14041004e0 0x7f14041980d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:25:03.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.030+0000 7f140c000700 1 --2- 172.21.15.62:0/2048499387 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1404100e80 0x7f1404198610 unknown :-1 s=CLOSED pgs=1274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:25:03.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.030+0000 7f140c000700 1 -- 172.21.15.62:0/2048499387 >> 172.21.15.62:0/2048499387 conn(0x7f14040fbda0 msgr2=0x7f14040fe150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:25:03.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.030+0000 7f140c000700 1 -- 172.21.15.62:0/2048499387 shutdown_connections 2024-04-20T22:25:03.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.030+0000 7f140c000700 1 -- 172.21.15.62:0/2048499387 wait complete. 2024-04-20T22:25:03.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.187+0000 7febd7e00700 1 -- 172.21.15.62:0/3257279571 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7febd8101220 msgr2=0x7febd81015f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:25:03.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.187+0000 7febd7e00700 1 --2- 172.21.15.62:0/3257279571 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7febd8101220 0x7febd81015f0 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7febc800b0a0 tx=0x7febc800b3b0 comp rx=0 tx=0).stop 2024-04-20T22:25:03.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.187+0000 7febd7e00700 1 -- 172.21.15.62:0/3257279571 shutdown_connections 2024-04-20T22:25:03.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.187+0000 7febd7e00700 1 --2- 172.21.15.62:0/3257279571 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7febd8101b30 0x7febd810a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:25:03.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.187+0000 7febd7e00700 1 --2- 172.21.15.62:0/3257279571 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7febd8101220 0x7febd81015f0 unknown :-1 s=CLOSED pgs=1275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:25:03.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.187+0000 7febd7e00700 1 -- 172.21.15.62:0/3257279571 >> 172.21.15.62:0/3257279571 conn(0x7febd80faa60 msgr2=0x7febd80fce70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:25:03.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.187+0000 7febd7e00700 1 -- 172.21.15.62:0/3257279571 shutdown_connections 2024-04-20T22:25:03.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.187+0000 7febd7e00700 1 -- 172.21.15.62:0/3257279571 wait complete. 2024-04-20T22:25:03.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.189+0000 7febd7e00700 1 Processor -- start 2024-04-20T22:25:03.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.189+0000 7febd7e00700 1 -- start start 2024-04-20T22:25:03.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.189+0000 7febd7e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7febd8101220 0x7febd80fee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:25:03.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.189+0000 7febd7e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7febd8101b30 0x7febd80ff3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:25:03.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.189+0000 7febd7e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febd80ff900 con 0x7febd8101b30 2024-04-20T22:25:03.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.189+0000 7febd7e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febd80ffa40 con 0x7febd8101220 2024-04-20T22:25:03.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.189+0000 7febce000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7febd8101b30 0x7febd80ff3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:25:03.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.190+0000 7febce000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7febd8101b30 0x7febd80ff3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:38252/0 (socket says 172.21.15.62:38252) 2024-04-20T22:25:03.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.190+0000 7febce000700 1 -- 172.21.15.62:0/3607583623 learned_addr learned my addr 172.21.15.62:0/3607583623 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:25:03.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.190+0000 7febd6a00700 1 --2- 172.21.15.62:0/3607583623 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7febd8101220 0x7febd80fee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:25:03.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.190+0000 7febce000700 1 -- 172.21.15.62:0/3607583623 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7febd8101220 msgr2=0x7febd80fee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:25:03.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.190+0000 7febce000700 1 --2- 172.21.15.62:0/3607583623 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7febd8101220 0x7febd80fee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:25:03.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.190+0000 7febce000700 1 -- 172.21.15.62:0/3607583623 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febc8009d00 con 0x7febd8101b30 2024-04-20T22:25:03.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.190+0000 7febce000700 1 --2- 172.21.15.62:0/3607583623 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7febd8101b30 0x7febd80ff3c0 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7febc000d8d0 tx=0x7febc000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:25:03.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.191+0000 7febcfe00700 1 -- 172.21.15.62:0/3607583623 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7febc0009940 con 0x7febd8101b30 2024-04-20T22:25:03.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.191+0000 7febcfe00700 1 -- 172.21.15.62:0/3607583623 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7febc0010460 con 0x7febd8101b30 2024-04-20T22:25:03.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.191+0000 7febcfe00700 1 -- 172.21.15.62:0/3607583623 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7febc000f5f0 con 0x7febd8101b30 2024-04-20T22:25:03.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.191+0000 7febd7e00700 1 -- 172.21.15.62:0/3607583623 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febd81a1dd0 con 0x7febd8101b30 2024-04-20T22:25:03.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.191+0000 7febd7e00700 1 -- 172.21.15.62:0/3607583623 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febd81a21e0 con 0x7febd8101b30 2024-04-20T22:25:03.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.192+0000 7febd7e00700 1 -- 172.21.15.62:0/3607583623 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febd8107700 con 0x7febd8101b30 2024-04-20T22:25:03.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.193+0000 7febcfe00700 1 -- 172.21.15.62:0/3607583623 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7febc00105d0 con 0x7febd8101b30 2024-04-20T22:25:03.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.194+0000 7febcfe00700 1 --2- 172.21.15.62:0/3607583623 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7febbc06ecc0 0x7febbc071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:25:03.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.194+0000 7febcfe00700 1 -- 172.21.15.62:0/3607583623 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7febc008ee10 con 0x7febd8101b30 2024-04-20T22:25:03.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.194+0000 7febd6a00700 1 --2- 172.21.15.62:0/3607583623 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7febbc06ecc0 0x7febbc071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:25:03.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.195+0000 7febd6a00700 1 --2- 172.21.15.62:0/3607583623 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7febbc06ecc0 0x7febbc071170 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7febc8009480 tx=0x7febc800b7f0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:25:03.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.199+0000 7febcfe00700 1 -- 172.21.15.62:0/3607583623 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7febc0057ba0 con 0x7febd8101b30 2024-04-20T22:25:03.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:25:03 smithi062 ceph-mon[76322]: pgmap v1495: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:25:03.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:25:03 smithi062 ceph-mon[76322]: from='client.36608 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:25:03.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:25:03 smithi062 ceph-mon[76322]: from='client.36612 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:25:03.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:25:03 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/2048499387' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:25:03.491 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:25:03 smithi189 ceph-mon[57022]: pgmap v1495: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:25:03.491 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:25:03 smithi189 ceph-mon[57022]: from='client.36608 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:25:03.491 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:25:03 smithi189 ceph-mon[57022]: from='client.36612 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:25:03.491 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:25:03 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/2048499387' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:25:03.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.494+0000 7febd7e00700 1 -- 172.21.15.62:0/3607583623 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7febd8062380 con 0x7febd8101b30 2024-04-20T22:25:03.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.495+0000 7febcfe00700 1 -- 172.21.15.62:0/3607583623 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 29 v29) v1 ==== 76+0+1887 (secure 0 0 0) 0x7febc0016770 con 0x7febd8101b30 2024-04-20T22:25:03.497 INFO:teuthology.orchestra.run.smithi062.stdout:e29 2024-04-20T22:25:03.497 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T22:25:03.497 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:25:03.497 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T22:25:03.497 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:25:03.498 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T22:25:03.498 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T22:25:03.498 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 20 2024-04-20T22:25:03.498 INFO:teuthology.orchestra.run.smithi062.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-20T22:25:03.498 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T22:25:03.498 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T21:42:01.559061+0000 2024-04-20T22:25:03.498 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T22:25:03.498 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T22:25:03.498 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T22:25:03.498 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T22:25:03.498 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T22:25:03.498 INFO:teuthology.orchestra.run.smithi062.stdout:max_xattr_size 65536 2024-04-20T22:25:03.498 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T22:25:03.498 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T22:25:03.499 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 81 2024-04-20T22:25:03.499 INFO:teuthology.orchestra.run.smithi062.stdout:snap_rstat 1 2024-04-20T22:25:03.499 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:25:03.499 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 2 2024-04-20T22:25:03.499 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1 2024-04-20T22:25:03.499 INFO:teuthology.orchestra.run.smithi062.stdout:up {} 2024-04-20T22:25:03.499 INFO:teuthology.orchestra.run.smithi062.stdout:failed 0,1 2024-04-20T22:25:03.499 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T22:25:03.499 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T22:25:03.499 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T22:25:03.499 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T22:25:03.499 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data enabled 2024-04-20T22:25:03.499 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T22:25:03.499 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T22:25:03.499 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 1 2024-04-20T22:25:03.500 INFO:teuthology.orchestra.run.smithi062.stdout:qdb_cluster leader: 0 members: 2024-04-20T22:25:03.500 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:25:03.500 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:25:03.500 INFO:teuthology.orchestra.run.smithi062.stdout:Standby daemons: 2024-04-20T22:25:03.500 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:25:03.500 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ryqblb{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6826/4113916205,v1:172.21.15.62:6827/4113916205] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:25:03.500 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ozizwi{-1:34476} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6828/3801975165,v1:172.21.15.62:6829/3801975165] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:25:03.500 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.dvimer{-1:44727} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6824/3576699232,v1:172.21.15.189:6825/3576699232] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:25:03.500 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.xbgspw{-1:45075} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6826/212905792,v1:172.21.15.189:6827/212905792] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:25:03.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.498+0000 7febd7e00700 1 -- 172.21.15.62:0/3607583623 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7febbc06ecc0 msgr2=0x7febbc071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:25:03.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.498+0000 7febd7e00700 1 --2- 172.21.15.62:0/3607583623 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7febbc06ecc0 0x7febbc071170 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7febc8009480 tx=0x7febc800b7f0 comp rx=0 tx=0).stop 2024-04-20T22:25:03.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.498+0000 7febd7e00700 1 -- 172.21.15.62:0/3607583623 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7febd8101b30 msgr2=0x7febd80ff3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:25:03.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.498+0000 7febd7e00700 1 --2- 172.21.15.62:0/3607583623 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7febd8101b30 0x7febd80ff3c0 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7febc000d8d0 tx=0x7febc000dc90 comp rx=0 tx=0).stop 2024-04-20T22:25:03.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.498+0000 7febd7e00700 1 -- 172.21.15.62:0/3607583623 shutdown_connections 2024-04-20T22:25:03.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.498+0000 7febd7e00700 1 --2- 172.21.15.62:0/3607583623 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7febbc06ecc0 0x7febbc071170 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:25:03.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.498+0000 7febd7e00700 1 --2- 172.21.15.62:0/3607583623 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7febd8101220 0x7febd80fee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:25:03.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.498+0000 7febd7e00700 1 --2- 172.21.15.62:0/3607583623 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7febd8101b30 0x7febd80ff3c0 unknown :-1 s=CLOSED pgs=1276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:25:03.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.498+0000 7febd7e00700 1 -- 172.21.15.62:0/3607583623 >> 172.21.15.62:0/3607583623 conn(0x7febd80faa60 msgr2=0x7febd8104840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:25:03.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.499+0000 7febd7e00700 1 -- 172.21.15.62:0/3607583623 shutdown_connections 2024-04-20T22:25:03.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.499+0000 7febd7e00700 1 -- 172.21.15.62:0/3607583623 wait complete. 2024-04-20T22:25:03.502 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 29 2024-04-20T22:25:03.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.666+0000 7f5133800700 1 -- 172.21.15.62:0/1271597385 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f512c0687a0 msgr2=0x7f512c068b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:25:03.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.666+0000 7f5133800700 1 --2- 172.21.15.62:0/1271597385 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f512c0687a0 0x7f512c068b70 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f5120009910 tx=0x7f5120009c20 comp rx=0 tx=0).stop 2024-04-20T22:25:03.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.666+0000 7f5133800700 1 -- 172.21.15.62:0/1271597385 shutdown_connections 2024-04-20T22:25:03.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.666+0000 7f5133800700 1 --2- 172.21.15.62:0/1271597385 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f512c069140 0x7f512c104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:25:03.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.666+0000 7f5133800700 1 --2- 172.21.15.62:0/1271597385 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f512c0687a0 0x7f512c068b70 unknown :-1 s=CLOSED pgs=1277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:25:03.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.666+0000 7f5133800700 1 -- 172.21.15.62:0/1271597385 >> 172.21.15.62:0/1271597385 conn(0x7f512c075130 msgr2=0x7f512c075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:25:03.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.666+0000 7f5133800700 1 -- 172.21.15.62:0/1271597385 shutdown_connections 2024-04-20T22:25:03.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.666+0000 7f5133800700 1 -- 172.21.15.62:0/1271597385 wait complete. 2024-04-20T22:25:03.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.667+0000 7f5133800700 1 Processor -- start 2024-04-20T22:25:03.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.668+0000 7f5133800700 1 -- start start 2024-04-20T22:25:03.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.668+0000 7f5133800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f512c0687a0 0x7f512c195fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:25:03.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.668+0000 7f5133800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f512c069140 0x7f512c1964e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:25:03.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.668+0000 7f5133800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f512c196bc0 con 0x7f512c0687a0 2024-04-20T22:25:03.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.668+0000 7f5133800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f512c19a950 con 0x7f512c069140 2024-04-20T22:25:03.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.668+0000 7f5130e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f512c0687a0 0x7f512c195fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:25:03.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.668+0000 7f5130e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f512c0687a0 0x7f512c195fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:38264/0 (socket says 172.21.15.62:38264) 2024-04-20T22:25:03.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.669+0000 7f5130e00700 1 -- 172.21.15.62:0/2661233664 learned_addr learned my addr 172.21.15.62:0/2661233664 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:25:03.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.669+0000 7f512be00700 1 --2- 172.21.15.62:0/2661233664 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f512c069140 0x7f512c1964e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:25:03.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.669+0000 7f5130e00700 1 -- 172.21.15.62:0/2661233664 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f512c069140 msgr2=0x7f512c1964e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:25:03.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.669+0000 7f5130e00700 1 --2- 172.21.15.62:0/2661233664 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f512c069140 0x7f512c1964e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:25:03.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.669+0000 7f5130e00700 1 -- 172.21.15.62:0/2661233664 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51200095f0 con 0x7f512c0687a0 2024-04-20T22:25:03.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.669+0000 7f5130e00700 1 --2- 172.21.15.62:0/2661233664 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f512c0687a0 0x7f512c195fa0 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f512000bda0 tx=0x7f512000be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:25:03.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.670+0000 7f5129600700 1 -- 172.21.15.62:0/2661233664 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f512001d070 con 0x7f512c0687a0 2024-04-20T22:25:03.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.670+0000 7f5129600700 1 -- 172.21.15.62:0/2661233664 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5120004fb0 con 0x7f512c0687a0 2024-04-20T22:25:03.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.670+0000 7f5129600700 1 -- 172.21.15.62:0/2661233664 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f512000f480 con 0x7f512c0687a0 2024-04-20T22:25:03.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.670+0000 7f5133800700 1 -- 172.21.15.62:0/2661233664 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f512c19abd0 con 0x7f512c0687a0 2024-04-20T22:25:03.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.670+0000 7f5133800700 1 -- 172.21.15.62:0/2661233664 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f512c19b0c0 con 0x7f512c0687a0 2024-04-20T22:25:03.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.671+0000 7f5133800700 1 -- 172.21.15.62:0/2661233664 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f512c108820 con 0x7f512c0687a0 2024-04-20T22:25:03.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.672+0000 7f5129600700 1 -- 172.21.15.62:0/2661233664 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f5120022870 con 0x7f512c0687a0 2024-04-20T22:25:03.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.673+0000 7f5129600700 1 --2- 172.21.15.62:0/2661233664 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f511406ecc0 0x7f5114071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:25:03.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.674+0000 7f5129600700 1 -- 172.21.15.62:0/2661233664 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f5120090370 con 0x7f512c0687a0 2024-04-20T22:25:03.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.674+0000 7f512be00700 1 --2- 172.21.15.62:0/2661233664 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f511406ecc0 0x7f5114071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:25:03.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.675+0000 7f512be00700 1 --2- 172.21.15.62:0/2661233664 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f511406ecc0 0x7f5114071170 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f5118006fd0 tx=0x7f5118008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:25:03.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.678+0000 7f5129600700 1 -- 172.21.15.62:0/2661233664 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f51200591b0 con 0x7f512c0687a0 2024-04-20T22:25:03.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.935+0000 7f5133800700 1 -- 172.21.15.62:0/2661233664 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f512c197440 con 0x7f511406ecc0 2024-04-20T22:25:03.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.937+0000 7f5129600700 1 -- 172.21.15.62:0/2661233664 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f5118009df0 con 0x7f511406ecc0 2024-04-20T22:25:03.937 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:25:03.937 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T22:25:03.938 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T22:25:03.938 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-20T22:25:03.938 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T22:25:03.938 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr", 2024-04-20T22:25:03.938 INFO:teuthology.orchestra.run.smithi062.stdout: "osd", 2024-04-20T22:25:03.938 INFO:teuthology.orchestra.run.smithi062.stdout: "mds", 2024-04-20T22:25:03.938 INFO:teuthology.orchestra.run.smithi062.stdout: "crash", 2024-04-20T22:25:03.938 INFO:teuthology.orchestra.run.smithi062.stdout: "mon" 2024-04-20T22:25:03.938 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T22:25:03.938 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "16/23 daemons upgraded", 2024-04-20T22:25:03.938 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading mds daemons", 2024-04-20T22:25:03.938 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T22:25:03.938 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:25:03.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.940+0000 7f5133800700 1 -- 172.21.15.62:0/2661233664 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f511406ecc0 msgr2=0x7f5114071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:25:03.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.940+0000 7f5133800700 1 --2- 172.21.15.62:0/2661233664 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f511406ecc0 0x7f5114071170 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f5118006fd0 tx=0x7f5118008040 comp rx=0 tx=0).stop 2024-04-20T22:25:03.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.940+0000 7f5133800700 1 -- 172.21.15.62:0/2661233664 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f512c0687a0 msgr2=0x7f512c195fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:25:03.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.940+0000 7f5133800700 1 --2- 172.21.15.62:0/2661233664 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f512c0687a0 0x7f512c195fa0 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f512000bda0 tx=0x7f512000be80 comp rx=0 tx=0).stop 2024-04-20T22:25:03.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.941+0000 7f5133800700 1 -- 172.21.15.62:0/2661233664 shutdown_connections 2024-04-20T22:25:03.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.941+0000 7f5133800700 1 --2- 172.21.15.62:0/2661233664 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f511406ecc0 0x7f5114071170 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:25:03.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.941+0000 7f5133800700 1 --2- 172.21.15.62:0/2661233664 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f512c0687a0 0x7f512c195fa0 unknown :-1 s=CLOSED pgs=1278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:25:03.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.941+0000 7f5133800700 1 --2- 172.21.15.62:0/2661233664 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f512c069140 0x7f512c1964e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:25:03.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.941+0000 7f5133800700 1 -- 172.21.15.62:0/2661233664 >> 172.21.15.62:0/2661233664 conn(0x7f512c075130 msgr2=0x7f512c0fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:25:03.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.941+0000 7f5133800700 1 -- 172.21.15.62:0/2661233664 shutdown_connections 2024-04-20T22:25:03.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:03.941+0000 7f5133800700 1 -- 172.21.15.62:0/2661233664 wait complete. 2024-04-20T22:25:04.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:04.101+0000 7f879b200700 1 -- 172.21.15.62:0/261469817 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f87941003d0 msgr2=0x7f87941007a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:25:04.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:04.101+0000 7f879b200700 1 --2- 172.21.15.62:0/261469817 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f87941003d0 0x7f87941007a0 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f8788009910 tx=0x7f8788009c20 comp rx=0 tx=0).stop 2024-04-20T22:25:04.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:04.102+0000 7f879b200700 1 -- 172.21.15.62:0/261469817 shutdown_connections 2024-04-20T22:25:04.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:04.102+0000 7f879b200700 1 --2- 172.21.15.62:0/261469817 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8794101c30 0x7f87941020a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:25:04.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:04.102+0000 7f879b200700 1 --2- 172.21.15.62:0/261469817 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f87941003d0 0x7f87941007a0 unknown :-1 s=CLOSED pgs=1279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:25:04.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:04.102+0000 7f879b200700 1 -- 172.21.15.62:0/261469817 >> 172.21.15.62:0/261469817 conn(0x7f87940fbbd0 msgr2=0x7f87940fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:25:04.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:04.102+0000 7f879b200700 1 -- 172.21.15.62:0/261469817 shutdown_connections 2024-04-20T22:25:04.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:04.102+0000 7f879b200700 1 -- 172.21.15.62:0/261469817 wait complete. 2024-04-20T22:25:04.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:04.103+0000 7f879b200700 1 Processor -- start 2024-04-20T22:25:04.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:04.103+0000 7f879b200700 1 -- start start 2024-04-20T22:25:04.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:04.103+0000 7f879b200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87941003d0 0x7f879419ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:25:04.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:04.104+0000 7f879b200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8794101c30 0x7f879419bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:25:04.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:04.104+0000 7f879b200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f879419c620 con 0x7f8794101c30 2024-04-20T22:25:04.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:04.104+0000 7f879b200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8794195ad0 con 0x7f87941003d0 2024-04-20T22:25:04.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:04.104+0000 7f8799400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8794101c30 0x7f879419bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:25:04.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:04.104+0000 7f8799400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8794101c30 0x7f879419bf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:38278/0 (socket says 172.21.15.62:38278) 2024-04-20T22:25:04.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:04.104+0000 7f8799400700 1 -- 172.21.15.62:0/590236537 learned_addr learned my addr 172.21.15.62:0/590236537 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:25:04.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:04.104+0000 7f8799e00700 1 --2- 172.21.15.62:0/590236537 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87941003d0 0x7f879419ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:25:04.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:04.104+0000 7f8799400700 1 -- 172.21.15.62:0/590236537 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87941003d0 msgr2=0x7f879419ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:25:04.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:04.104+0000 7f8799400700 1 --2- 172.21.15.62:0/590236537 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87941003d0 0x7f879419ba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:25:04.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:04.104+0000 7f8799400700 1 -- 172.21.15.62:0/590236537 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87880095f0 con 0x7f8794101c30 2024-04-20T22:25:04.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:04.105+0000 7f8799400700 1 --2- 172.21.15.62:0/590236537 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8794101c30 0x7f879419bf90 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f878400d8d0 tx=0x7f878400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:25:04.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:04.105+0000 7f878ea00700 1 -- 172.21.15.62:0/590236537 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8784009940 con 0x7f8794101c30 2024-04-20T22:25:04.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:04.105+0000 7f878ea00700 1 -- 172.21.15.62:0/590236537 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8784010460 con 0x7f8794101c30 2024-04-20T22:25:04.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:04.105+0000 7f879b200700 1 -- 172.21.15.62:0/590236537 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8794195db0 con 0x7f8794101c30 2024-04-20T22:25:04.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:04.105+0000 7f878ea00700 1 -- 172.21.15.62:0/590236537 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f878400f5f0 con 0x7f8794101c30 2024-04-20T22:25:04.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:04.105+0000 7f879b200700 1 -- 172.21.15.62:0/590236537 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8794196380 con 0x7f8794101c30 2024-04-20T22:25:04.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:04.106+0000 7f879b200700 1 -- 172.21.15.62:0/590236537 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8794062380 con 0x7f8794101c30 2024-04-20T22:25:04.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:04.107+0000 7f878ea00700 1 -- 172.21.15.62:0/590236537 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f87840105d0 con 0x7f8794101c30 2024-04-20T22:25:04.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:04.108+0000 7f878ea00700 1 --2- 172.21.15.62:0/590236537 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f877c06ecc0 0x7f877c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:25:04.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:04.108+0000 7f878ea00700 1 -- 172.21.15.62:0/590236537 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f878408ef20 con 0x7f8794101c30 2024-04-20T22:25:04.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:04.108+0000 7f8799e00700 1 --2- 172.21.15.62:0/590236537 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f877c06ecc0 0x7f877c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:25:04.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:04.109+0000 7f8799e00700 1 --2- 172.21.15.62:0/590236537 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f877c06ecc0 0x7f877c071170 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f878800b5c0 tx=0x7f878801a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:25:04.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:04.113+0000 7f878ea00700 1 -- 172.21.15.62:0/590236537 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f878405b760 con 0x7f8794101c30 2024-04-20T22:25:04.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:25:04 smithi062 ceph-mon[76322]: from='client.36616 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:25:04.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:25:04 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/3607583623' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:25:04.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:04.435+0000 7f879b200700 1 -- 172.21.15.62:0/590236537 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f8794196fe0 con 0x7f8794101c30 2024-04-20T22:25:04.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:04.436+0000 7f878ea00700 1 -- 172.21.15.62:0/590236537 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7f878405b2f0 con 0x7f8794101c30 2024-04-20T22:25:04.437 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-20T22:25:04.437 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-20T22:25:04.437 INFO:teuthology.orchestra.run.smithi062.stdout: host smithi189 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi189/config 2024-04-20T22:25:04.437 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-20T22:25:04.437 INFO:teuthology.orchestra.run.smithi062.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi189 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp11670v2q:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:25:04.437 INFO:teuthology.orchestra.run.smithi062.stdout:Traceback (most recent call last): 2024-04-20T22:25:04.437 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-20T22:25:04.437 INFO:teuthology.orchestra.run.smithi062.stdout: return _run_code(code, main_globals, None, 2024-04-20T22:25:04.438 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-20T22:25:04.438 INFO:teuthology.orchestra.run.smithi062.stdout: exec(code, run_globals) 2024-04-20T22:25:04.438 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5618, in 2024-04-20T22:25:04.438 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5606, in main 2024-04-20T22:25:04.438 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 408, in _infer_config 2024-04-20T22:25:04.438 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 323, in _infer_fsid 2024-04-20T22:25:04.438 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 436, in _infer_image 2024-04-20T22:25:04.438 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 310, in _validate_fsid 2024-04-20T22:25:04.438 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 3359, in command_ceph_volume 2024-04-20T22:25:04.438 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-20T22:25:04.438 INFO:teuthology.orchestra.run.smithi062.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi189 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp11670v2q:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:25:04.438 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-20T22:25:04.438 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is degraded 2024-04-20T22:25:04.438 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T22:25:04.439 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T22:25:04.439 INFO:teuthology.orchestra.run.smithi062.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-20T22:25:04.439 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-20T22:25:04.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:04.439+0000 7f879b200700 1 -- 172.21.15.62:0/590236537 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f877c06ecc0 msgr2=0x7f877c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:25:04.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:04.439+0000 7f879b200700 1 --2- 172.21.15.62:0/590236537 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f877c06ecc0 0x7f877c071170 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f878800b5c0 tx=0x7f878801a040 comp rx=0 tx=0).stop 2024-04-20T22:25:04.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:04.439+0000 7f879b200700 1 -- 172.21.15.62:0/590236537 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8794101c30 msgr2=0x7f879419bf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:25:04.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:04.439+0000 7f879b200700 1 --2- 172.21.15.62:0/590236537 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8794101c30 0x7f879419bf90 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f878400d8d0 tx=0x7f878400dc90 comp rx=0 tx=0).stop 2024-04-20T22:25:04.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:04.439+0000 7f879b200700 1 -- 172.21.15.62:0/590236537 shutdown_connections 2024-04-20T22:25:04.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:04.439+0000 7f879b200700 1 --2- 172.21.15.62:0/590236537 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f877c06ecc0 0x7f877c071170 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:25:04.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:04.439+0000 7f879b200700 1 --2- 172.21.15.62:0/590236537 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87941003d0 0x7f879419ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:25:04.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:04.439+0000 7f879b200700 1 --2- 172.21.15.62:0/590236537 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8794101c30 0x7f879419bf90 unknown :-1 s=CLOSED pgs=1280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:25:04.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:04.439+0000 7f879b200700 1 -- 172.21.15.62:0/590236537 >> 172.21.15.62:0/590236537 conn(0x7f87940fbbd0 msgr2=0x7f87940fc4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:25:04.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:04.439+0000 7f879b200700 1 -- 172.21.15.62:0/590236537 shutdown_connections 2024-04-20T22:25:04.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:04.439+0000 7f879b200700 1 -- 172.21.15.62:0/590236537 wait complete. 2024-04-20T22:25:04.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:25:04 smithi189 ceph-mon[57022]: from='client.36616 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:25:04.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:25:04 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/3607583623' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:25:05.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:25:05 smithi189 ceph-mon[57022]: pgmap v1496: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:25:05.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:25:05 smithi189 ceph-mon[57022]: from='client.36628 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:25:05.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:25:05 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/590236537' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:25:05.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:25:05 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:25:05.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:25:05 smithi062 ceph-mon[76322]: pgmap v1496: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:25:05.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:25:05 smithi062 ceph-mon[76322]: from='client.36628 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:25:05.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:25:05 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/590236537' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:25:05.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:25:05 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:25:07.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:25:07 smithi189 ceph-mon[57022]: pgmap v1497: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:25:07.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:25:07 smithi062 ceph-mon[76322]: pgmap v1497: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:25:09.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:25:09 smithi189 ceph-mon[57022]: pgmap v1498: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:25:09.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:25:09 smithi062 ceph-mon[76322]: pgmap v1498: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:25:11.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:25:11 smithi189 ceph-mon[57022]: pgmap v1499: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:25:11.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:25:11 smithi062 ceph-mon[76322]: pgmap v1499: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:25:13.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:25:13 smithi189 ceph-mon[57022]: pgmap v1500: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:25:13.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:25:13 smithi062 ceph-mon[76322]: pgmap v1500: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:25:15.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:25:15 smithi189 ceph-mon[57022]: pgmap v1501: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:25:15.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:25:15 smithi062 ceph-mon[76322]: pgmap v1501: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:25:17.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:25:17 smithi189 ceph-mon[57022]: pgmap v1502: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:25:17.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:25:17 smithi062 ceph-mon[76322]: pgmap v1502: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:25:19.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:25:19 smithi189 ceph-mon[57022]: pgmap v1503: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:25:19.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:25:19 smithi062 ceph-mon[76322]: pgmap v1503: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:25:20.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:25:20 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:25:20.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:25:20 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:25:21.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:25:21 smithi062 ceph-mon[76322]: pgmap v1504: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:25:21.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:25:21 smithi189 ceph-mon[57022]: pgmap v1504: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:25:23.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:25:23 smithi062 ceph-mon[76322]: pgmap v1505: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:25:23.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:25:23 smithi189 ceph-mon[57022]: pgmap v1505: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:25:25.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:25:25 smithi062 ceph-mon[76322]: pgmap v1506: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:25:25.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:25:25 smithi189 ceph-mon[57022]: pgmap v1506: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:25:27.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:25:27 smithi062 ceph-mon[76322]: pgmap v1507: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:25:27.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:25:27 smithi189 ceph-mon[57022]: pgmap v1507: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:25:29.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:25:29 smithi062 ceph-mon[76322]: pgmap v1508: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:25:29.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:25:29 smithi189 ceph-mon[57022]: pgmap v1508: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:25:31.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:25:31 smithi189 ceph-mon[57022]: pgmap v1509: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:25:31.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:25:31 smithi062 ceph-mon[76322]: pgmap v1509: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:25:33.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:25:33 smithi062 ceph-mon[76322]: pgmap v1510: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:25:33.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:25:33 smithi189 ceph-mon[57022]: pgmap v1510: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:25:34.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:34.620+0000 7fec55600700 1 -- 172.21.15.62:0/826542029 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fec50101d30 msgr2=0x7fec5010a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:25:34.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:34.620+0000 7fec55600700 1 --2- 172.21.15.62:0/826542029 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fec50101d30 0x7fec5010a230 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7fec3c009910 tx=0x7fec3c009c20 comp rx=0 tx=0).stop 2024-04-20T22:25:34.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:34.621+0000 7fec55600700 1 -- 172.21.15.62:0/826542029 shutdown_connections 2024-04-20T22:25:34.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:34.621+0000 7fec55600700 1 --2- 172.21.15.62:0/826542029 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fec50101d30 0x7fec5010a230 unknown :-1 s=CLOSED pgs=1281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:25:34.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:34.621+0000 7fec55600700 1 --2- 172.21.15.62:0/826542029 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec50101420 0x7fec501017f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:25:34.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:34.621+0000 7fec55600700 1 -- 172.21.15.62:0/826542029 >> 172.21.15.62:0/826542029 conn(0x7fec500fac80 msgr2=0x7fec500fd090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:25:34.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:34.621+0000 7fec55600700 1 -- 172.21.15.62:0/826542029 shutdown_connections 2024-04-20T22:25:34.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:34.621+0000 7fec55600700 1 -- 172.21.15.62:0/826542029 wait complete. 2024-04-20T22:25:34.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:34.622+0000 7fec55600700 1 Processor -- start 2024-04-20T22:25:34.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:34.623+0000 7fec55600700 1 -- start start 2024-04-20T22:25:34.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:34.623+0000 7fec55600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fec50101420 0x7fec50195ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:25:34.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:34.623+0000 7fec55600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec50101d30 0x7fec50196530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:25:34.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:34.623+0000 7fec55600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec50196c10 con 0x7fec50101420 2024-04-20T22:25:34.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:34.623+0000 7fec55600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec5019a9a0 con 0x7fec50101d30 2024-04-20T22:25:34.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:34.623+0000 7fec4ea00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fec50101420 0x7fec50195ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:25:34.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:34.624+0000 7fec4ea00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fec50101420 0x7fec50195ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44274/0 (socket says 172.21.15.62:44274) 2024-04-20T22:25:34.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:34.624+0000 7fec4ea00700 1 -- 172.21.15.62:0/3428778837 learned_addr learned my addr 172.21.15.62:0/3428778837 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:25:34.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:34.624+0000 7fec4e000700 1 --2- 172.21.15.62:0/3428778837 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec50101d30 0x7fec50196530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:25:34.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:34.624+0000 7fec4ea00700 1 -- 172.21.15.62:0/3428778837 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec50101d30 msgr2=0x7fec50196530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:25:34.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:34.624+0000 7fec4ea00700 1 --2- 172.21.15.62:0/3428778837 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec50101d30 0x7fec50196530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:25:34.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:34.624+0000 7fec4ea00700 1 -- 172.21.15.62:0/3428778837 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec3c0095f0 con 0x7fec50101420 2024-04-20T22:25:34.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:34.624+0000 7fec4ea00700 1 --2- 172.21.15.62:0/3428778837 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fec50101420 0x7fec50195ff0 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7fec4000b4b0 tx=0x7fec4000b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:25:34.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:34.625+0000 7fec47400700 1 -- 172.21.15.62:0/3428778837 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fec40011800 con 0x7fec50101420 2024-04-20T22:25:34.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:34.625+0000 7fec55600700 1 -- 172.21.15.62:0/3428778837 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec5019ac80 con 0x7fec50101420 2024-04-20T22:25:34.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:34.625+0000 7fec47400700 1 -- 172.21.15.62:0/3428778837 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fec40011e20 con 0x7fec50101420 2024-04-20T22:25:34.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:34.625+0000 7fec47400700 1 -- 172.21.15.62:0/3428778837 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fec4000f2e0 con 0x7fec50101420 2024-04-20T22:25:34.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:34.625+0000 7fec55600700 1 -- 172.21.15.62:0/3428778837 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec5019b1d0 con 0x7fec50101420 2024-04-20T22:25:34.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:34.626+0000 7fec55600700 1 -- 172.21.15.62:0/3428778837 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec501078a0 con 0x7fec50101420 2024-04-20T22:25:34.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:34.628+0000 7fec47400700 1 -- 172.21.15.62:0/3428778837 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fec40010380 con 0x7fec50101420 2024-04-20T22:25:34.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:34.629+0000 7fec47400700 1 --2- 172.21.15.62:0/3428778837 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fec3406ecc0 0x7fec34071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:25:34.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:34.629+0000 7fec47400700 1 -- 172.21.15.62:0/3428778837 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fec4008eaa0 con 0x7fec50101420 2024-04-20T22:25:34.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:34.629+0000 7fec4e000700 1 --2- 172.21.15.62:0/3428778837 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fec3406ecc0 0x7fec34071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:25:34.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:34.630+0000 7fec4e000700 1 --2- 172.21.15.62:0/3428778837 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fec3406ecc0 0x7fec34071170 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fec3c000c00 tx=0x7fec3c005fd0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:25:34.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:34.637+0000 7fec47400700 1 -- 172.21.15.62:0/3428778837 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fec400577b0 con 0x7fec50101420 2024-04-20T22:25:34.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:34.895+0000 7fec55600700 1 -- 172.21.15.62:0/3428778837 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fec5019b4c0 con 0x7fec3406ecc0 2024-04-20T22:25:34.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:34.897+0000 7fec47400700 1 -- 172.21.15.62:0/3428778837 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fec5019b4c0 con 0x7fec3406ecc0 2024-04-20T22:25:34.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:34.900+0000 7fec55600700 1 -- 172.21.15.62:0/3428778837 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fec3406ecc0 msgr2=0x7fec34071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:25:34.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:34.900+0000 7fec55600700 1 --2- 172.21.15.62:0/3428778837 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fec3406ecc0 0x7fec34071170 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fec3c000c00 tx=0x7fec3c005fd0 comp rx=0 tx=0).stop 2024-04-20T22:25:34.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:34.900+0000 7fec55600700 1 -- 172.21.15.62:0/3428778837 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fec50101420 msgr2=0x7fec50195ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:25:34.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:34.900+0000 7fec55600700 1 --2- 172.21.15.62:0/3428778837 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fec50101420 0x7fec50195ff0 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7fec4000b4b0 tx=0x7fec4000b870 comp rx=0 tx=0).stop 2024-04-20T22:25:34.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:34.900+0000 7fec55600700 1 -- 172.21.15.62:0/3428778837 shutdown_connections 2024-04-20T22:25:34.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:34.900+0000 7fec55600700 1 --2- 172.21.15.62:0/3428778837 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fec3406ecc0 0x7fec34071170 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:25:34.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:34.900+0000 7fec55600700 1 --2- 172.21.15.62:0/3428778837 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fec50101420 0x7fec50195ff0 unknown :-1 s=CLOSED pgs=1282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:25:34.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:34.900+0000 7fec55600700 1 --2- 172.21.15.62:0/3428778837 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec50101d30 0x7fec50196530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:25:34.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:34.900+0000 7fec55600700 1 -- 172.21.15.62:0/3428778837 >> 172.21.15.62:0/3428778837 conn(0x7fec500fac80 msgr2=0x7fec501049e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:25:34.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:34.901+0000 7fec55600700 1 -- 172.21.15.62:0/3428778837 shutdown_connections 2024-04-20T22:25:34.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:34.901+0000 7fec55600700 1 -- 172.21.15.62:0/3428778837 wait complete. 2024-04-20T22:25:34.918 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T22:25:35.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.070+0000 7f91cba00700 1 -- 172.21.15.62:0/406727460 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91c4106b80 msgr2=0x7f91c4106ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:25:35.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.070+0000 7f91cba00700 1 --2- 172.21.15.62:0/406727460 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91c4106b80 0x7f91c4106ff0 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7f91b0009910 tx=0x7f91b0009c20 comp rx=0 tx=0).stop 2024-04-20T22:25:35.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.071+0000 7f91cba00700 1 -- 172.21.15.62:0/406727460 shutdown_connections 2024-04-20T22:25:35.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.071+0000 7f91cba00700 1 --2- 172.21.15.62:0/406727460 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91c4106b80 0x7f91c4106ff0 unknown :-1 s=CLOSED pgs=1283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:25:35.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.071+0000 7f91cba00700 1 --2- 172.21.15.62:0/406727460 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91c410c8f0 0x7f91c410ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:25:35.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.071+0000 7f91cba00700 1 -- 172.21.15.62:0/406727460 >> 172.21.15.62:0/406727460 conn(0x7f91c4076060 msgr2=0x7f91c4078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:25:35.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.071+0000 7f91cba00700 1 -- 172.21.15.62:0/406727460 shutdown_connections 2024-04-20T22:25:35.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.071+0000 7f91cba00700 1 -- 172.21.15.62:0/406727460 wait complete. 2024-04-20T22:25:35.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.072+0000 7f91cba00700 1 Processor -- start 2024-04-20T22:25:35.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.072+0000 7f91cba00700 1 -- start start 2024-04-20T22:25:35.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.072+0000 7f91cba00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91c4106b80 0x7f91c419c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:25:35.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.072+0000 7f91cba00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91c410c8f0 0x7f91c419c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:25:35.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.072+0000 7f91cba00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91c419d060 con 0x7f91c4106b80 2024-04-20T22:25:35.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.072+0000 7f91cba00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91c41a0df0 con 0x7f91c410c8f0 2024-04-20T22:25:35.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.073+0000 7f91c9000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91c4106b80 0x7f91c419c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:25:35.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.073+0000 7f91c9000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91c4106b80 0x7f91c419c440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44284/0 (socket says 172.21.15.62:44284) 2024-04-20T22:25:35.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.073+0000 7f91c9000700 1 -- 172.21.15.62:0/1938778889 learned_addr learned my addr 172.21.15.62:0/1938778889 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:25:35.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.073+0000 7f91c3e00700 1 --2- 172.21.15.62:0/1938778889 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91c410c8f0 0x7f91c419c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:25:35.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.073+0000 7f91c9000700 1 -- 172.21.15.62:0/1938778889 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91c410c8f0 msgr2=0x7f91c419c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:25:35.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.073+0000 7f91c9000700 1 --2- 172.21.15.62:0/1938778889 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91c410c8f0 0x7f91c419c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:25:35.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.073+0000 7f91c9000700 1 -- 172.21.15.62:0/1938778889 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91b00095f0 con 0x7f91c4106b80 2024-04-20T22:25:35.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.073+0000 7f91c3e00700 1 --2- 172.21.15.62:0/1938778889 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91c410c8f0 0x7f91c419c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T22:25:35.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.074+0000 7f91c9000700 1 --2- 172.21.15.62:0/1938778889 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91c4106b80 0x7f91c419c440 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f91b800d8d0 tx=0x7f91b800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:25:35.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.074+0000 7f91c1600700 1 -- 172.21.15.62:0/1938778889 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f91b8009940 con 0x7f91c4106b80 2024-04-20T22:25:35.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.074+0000 7f91c1600700 1 -- 172.21.15.62:0/1938778889 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f91b8010460 con 0x7f91c4106b80 2024-04-20T22:25:35.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.074+0000 7f91cba00700 1 -- 172.21.15.62:0/1938778889 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91c41a10d0 con 0x7f91c4106b80 2024-04-20T22:25:35.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.074+0000 7f91c1600700 1 -- 172.21.15.62:0/1938778889 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f91b800f5f0 con 0x7f91c4106b80 2024-04-20T22:25:35.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.074+0000 7f91cba00700 1 -- 172.21.15.62:0/1938778889 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91c41a1620 con 0x7f91c4106b80 2024-04-20T22:25:35.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.075+0000 7f91cba00700 1 -- 172.21.15.62:0/1938778889 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91c410ecc0 con 0x7f91c4106b80 2024-04-20T22:25:35.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.076+0000 7f91c1600700 1 -- 172.21.15.62:0/1938778889 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f91b80105d0 con 0x7f91c4106b80 2024-04-20T22:25:35.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.077+0000 7f91c1600700 1 --2- 172.21.15.62:0/1938778889 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f91ac06ecc0 0x7f91ac071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:25:35.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.077+0000 7f91c1600700 1 -- 172.21.15.62:0/1938778889 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f91b808ee10 con 0x7f91c4106b80 2024-04-20T22:25:35.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.078+0000 7f91c3e00700 1 --2- 172.21.15.62:0/1938778889 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f91ac06ecc0 0x7f91ac071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:25:35.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.078+0000 7f91c3e00700 1 --2- 172.21.15.62:0/1938778889 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f91ac06ecc0 0x7f91ac071170 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f91b0006010 tx=0x7f91b000b540 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:25:35.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.083+0000 7f91c1600700 1 -- 172.21.15.62:0/1938778889 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f91b8057ba0 con 0x7f91c4106b80 2024-04-20T22:25:35.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.346+0000 7f91cba00700 1 -- 172.21.15.62:0/1938778889 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f91c41a1890 con 0x7f91ac06ecc0 2024-04-20T22:25:35.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.349+0000 7f91c1600700 1 -- 172.21.15.62:0/1938778889 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f91c41a1890 con 0x7f91ac06ecc0 2024-04-20T22:25:35.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.352+0000 7f91cba00700 1 -- 172.21.15.62:0/1938778889 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f91ac06ecc0 msgr2=0x7f91ac071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:25:35.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.352+0000 7f91cba00700 1 --2- 172.21.15.62:0/1938778889 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f91ac06ecc0 0x7f91ac071170 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f91b0006010 tx=0x7f91b000b540 comp rx=0 tx=0).stop 2024-04-20T22:25:35.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.352+0000 7f91cba00700 1 -- 172.21.15.62:0/1938778889 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91c4106b80 msgr2=0x7f91c419c440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:25:35.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.352+0000 7f91cba00700 1 --2- 172.21.15.62:0/1938778889 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91c4106b80 0x7f91c419c440 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f91b800d8d0 tx=0x7f91b800dc90 comp rx=0 tx=0).stop 2024-04-20T22:25:35.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.352+0000 7f91cba00700 1 -- 172.21.15.62:0/1938778889 shutdown_connections 2024-04-20T22:25:35.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.352+0000 7f91cba00700 1 --2- 172.21.15.62:0/1938778889 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f91ac06ecc0 0x7f91ac071170 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:25:35.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.352+0000 7f91cba00700 1 --2- 172.21.15.62:0/1938778889 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91c4106b80 0x7f91c419c440 unknown :-1 s=CLOSED pgs=1284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:25:35.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.352+0000 7f91cba00700 1 --2- 172.21.15.62:0/1938778889 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91c410c8f0 0x7f91c419c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:25:35.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.352+0000 7f91cba00700 1 -- 172.21.15.62:0/1938778889 >> 172.21.15.62:0/1938778889 conn(0x7f91c4076060 msgr2=0x7f91c4077ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:25:35.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.352+0000 7f91cba00700 1 -- 172.21.15.62:0/1938778889 shutdown_connections 2024-04-20T22:25:35.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.353+0000 7f91cba00700 1 -- 172.21.15.62:0/1938778889 wait complete. 2024-04-20T22:25:35.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.511+0000 7fb4d2800700 1 -- 172.21.15.62:0/3403079455 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb4cc068540 msgr2=0x7fb4cc0689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:25:35.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.511+0000 7fb4d2800700 1 --2- 172.21.15.62:0/3403079455 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb4cc068540 0x7fb4cc0689b0 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7fb4b4009910 tx=0x7fb4b4009c20 comp rx=0 tx=0).stop 2024-04-20T22:25:35.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.511+0000 7fb4d2800700 1 -- 172.21.15.62:0/3403079455 shutdown_connections 2024-04-20T22:25:35.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.511+0000 7fb4d2800700 1 --2- 172.21.15.62:0/3403079455 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb4cc068540 0x7fb4cc0689b0 unknown :-1 s=CLOSED pgs=1285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:25:35.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.511+0000 7fb4d2800700 1 --2- 172.21.15.62:0/3403079455 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb4cc1085b0 0x7fb4cc108980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:25:35.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.511+0000 7fb4d2800700 1 -- 172.21.15.62:0/3403079455 >> 172.21.15.62:0/3403079455 conn(0x7fb4cc075130 msgr2=0x7fb4cc075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:25:35.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.511+0000 7fb4d2800700 1 -- 172.21.15.62:0/3403079455 shutdown_connections 2024-04-20T22:25:35.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.511+0000 7fb4d2800700 1 -- 172.21.15.62:0/3403079455 wait complete. 2024-04-20T22:25:35.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.512+0000 7fb4d2800700 1 Processor -- start 2024-04-20T22:25:35.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.513+0000 7fb4d2800700 1 -- start start 2024-04-20T22:25:35.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.513+0000 7fb4d2800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb4cc068540 0x7fb4cc1980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:25:35.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.513+0000 7fb4d2800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb4cc1085b0 0x7fb4cc1985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:25:35.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.513+0000 7fb4d2800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4cc198cd0 con 0x7fb4cc1085b0 2024-04-20T22:25:35.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.513+0000 7fb4d2800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4cc19ca60 con 0x7fb4cc068540 2024-04-20T22:25:35.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.513+0000 7fb4c3e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb4cc1085b0 0x7fb4cc1985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:25:35.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.513+0000 7fb4c3e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb4cc1085b0 0x7fb4cc1985f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44300/0 (socket says 172.21.15.62:44300) 2024-04-20T22:25:35.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.513+0000 7fb4c3e00700 1 -- 172.21.15.62:0/3782439889 learned_addr learned my addr 172.21.15.62:0/3782439889 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:25:35.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.514+0000 7fb4cb400700 1 --2- 172.21.15.62:0/3782439889 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb4cc068540 0x7fb4cc1980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:25:35.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.514+0000 7fb4c3e00700 1 -- 172.21.15.62:0/3782439889 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb4cc068540 msgr2=0x7fb4cc1980b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:25:35.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.514+0000 7fb4c3e00700 1 --2- 172.21.15.62:0/3782439889 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb4cc068540 0x7fb4cc1980b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:25:35.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.514+0000 7fb4c3e00700 1 -- 172.21.15.62:0/3782439889 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4b40095f0 con 0x7fb4cc1085b0 2024-04-20T22:25:35.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.514+0000 7fb4c3e00700 1 --2- 172.21.15.62:0/3782439889 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb4cc1085b0 0x7fb4cc1985f0 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7fb4b400b5c0 tx=0x7fb4b400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:25:35.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.515+0000 7fb4c8c00700 1 -- 172.21.15.62:0/3782439889 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb4b401d070 con 0x7fb4cc1085b0 2024-04-20T22:25:35.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.515+0000 7fb4c8c00700 1 -- 172.21.15.62:0/3782439889 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb4b4005e00 con 0x7fb4cc1085b0 2024-04-20T22:25:35.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.515+0000 7fb4d2800700 1 -- 172.21.15.62:0/3782439889 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4cc19cce0 con 0x7fb4cc1085b0 2024-04-20T22:25:35.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.515+0000 7fb4c8c00700 1 -- 172.21.15.62:0/3782439889 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb4b4017400 con 0x7fb4cc1085b0 2024-04-20T22:25:35.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.515+0000 7fb4d2800700 1 -- 172.21.15.62:0/3782439889 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4cc19d1d0 con 0x7fb4cc1085b0 2024-04-20T22:25:35.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.516+0000 7fb4d2800700 1 -- 172.21.15.62:0/3782439889 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4cc10aa00 con 0x7fb4cc1085b0 2024-04-20T22:25:35.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.517+0000 7fb4c8c00700 1 -- 172.21.15.62:0/3782439889 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fb4b400f460 con 0x7fb4cc1085b0 2024-04-20T22:25:35.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.518+0000 7fb4c8c00700 1 --2- 172.21.15.62:0/3782439889 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb4b006ecc0 0x7fb4b0071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:25:35.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.518+0000 7fb4c8c00700 1 -- 172.21.15.62:0/3782439889 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fb4b4090370 con 0x7fb4cc1085b0 2024-04-20T22:25:35.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.518+0000 7fb4cb400700 1 --2- 172.21.15.62:0/3782439889 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb4b006ecc0 0x7fb4b0071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:25:35.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.519+0000 7fb4cb400700 1 --2- 172.21.15.62:0/3782439889 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb4b006ecc0 0x7fb4b0071170 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fb4bc005950 tx=0x7fb4bc0058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:25:35.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.523+0000 7fb4c8c00700 1 -- 172.21.15.62:0/3782439889 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fb4b4059100 con 0x7fb4cc1085b0 2024-04-20T22:25:35.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:25:35 smithi189 ceph-mon[57022]: pgmap v1511: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:25:35.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:25:35 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:25:35.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.767+0000 7fb4d2800700 1 -- 172.21.15.62:0/3782439889 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fb4cc02cca0 con 0x7fb4b006ecc0 2024-04-20T22:25:35.768 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:25:35 smithi062 ceph-mon[76322]: pgmap v1511: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:25:35.768 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:25:35 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:25:35.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.783+0000 7fb4c8c00700 1 -- 172.21.15.62:0/3782439889 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fb4cc02cca0 con 0x7fb4b006ecc0 2024-04-20T22:25:35.783 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T22:25:35.784 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (51m) 12m ago 65m 14.6M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T22:25:35.784 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (65m) 12m ago 65m 11.6M - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T22:25:35.784 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (64m) 12m ago 64m 11.2M - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T22:25:35.784 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (47m) 12m ago 65m 6660k - 19.0.0-3133-ga8c84f59 76bc6b91b726 2825d5ba11c6 2024-04-20T22:25:35.784 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (47m) 12m ago 64m 6665k - 19.0.0-3133-ga8c84f59 76bc6b91b726 20f746fa1bba 2024-04-20T22:25:35.784 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (50m) 12m ago 64m 75.5M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T22:25:35.784 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (42m) 12m ago 58m 22.9M - 19.0.0-3133-ga8c84f59 76bc6b91b726 de80bc90a683 2024-04-20T22:25:35.784 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (43m) 12m ago 58m 23.0M - 19.0.0-3133-ga8c84f59 76bc6b91b726 cd793b7ee98e 2024-04-20T22:25:35.784 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (27m) 12m ago 58m 18.5M - 19.0.0-3133-ga8c84f59 76bc6b91b726 0e7b0bc4d1ab 2024-04-20T22:25:35.784 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (12m) 12m ago 57m 11.4M - 19.0.0-3133-ga8c84f59 76bc6b91b726 0af732ac9e06 2024-04-20T22:25:35.784 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (54m) 12m ago 67m 517M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T22:25:35.784 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (53m) 12m ago 64m 460M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T22:25:35.784 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (49m) 12m ago 67m 86.3M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 019eb1479485 2024-04-20T22:25:35.784 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (48m) 12m ago 63m 58.0M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 55aac560aab6 2024-04-20T22:25:35.785 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (52m) 12m ago 65m 8766k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T22:25:35.785 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (52m) 12m ago 64m 8711k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T22:25:35.785 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (46m) 12m ago 62m 91.3M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 ee2bb901f26e 2024-04-20T22:25:35.785 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (46m) 12m ago 62m 94.9M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 9075152d4162 2024-04-20T22:25:35.785 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (45m) 12m ago 61m 78.5M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 2d4f3e38dc0e 2024-04-20T22:25:35.785 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (45m) 12m ago 61m 94.1M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 8f97b17ae0eb 2024-04-20T22:25:35.785 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (44m) 12m ago 60m 96.1M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 653d8079967d 2024-04-20T22:25:35.785 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (43m) 12m ago 59m 90.9M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 596dded14c12 2024-04-20T22:25:35.785 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (52m) 12m ago 64m 90.4M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T22:25:35.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.787+0000 7fb4d2800700 1 -- 172.21.15.62:0/3782439889 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb4b006ecc0 msgr2=0x7fb4b0071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:25:35.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.787+0000 7fb4d2800700 1 --2- 172.21.15.62:0/3782439889 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb4b006ecc0 0x7fb4b0071170 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fb4bc005950 tx=0x7fb4bc0058e0 comp rx=0 tx=0).stop 2024-04-20T22:25:35.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.787+0000 7fb4d2800700 1 -- 172.21.15.62:0/3782439889 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb4cc1085b0 msgr2=0x7fb4cc1985f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:25:35.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.787+0000 7fb4d2800700 1 --2- 172.21.15.62:0/3782439889 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb4cc1085b0 0x7fb4cc1985f0 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7fb4b400b5c0 tx=0x7fb4b400be20 comp rx=0 tx=0).stop 2024-04-20T22:25:35.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.787+0000 7fb4d2800700 1 -- 172.21.15.62:0/3782439889 shutdown_connections 2024-04-20T22:25:35.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.787+0000 7fb4d2800700 1 --2- 172.21.15.62:0/3782439889 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb4b006ecc0 0x7fb4b0071170 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:25:35.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.787+0000 7fb4d2800700 1 --2- 172.21.15.62:0/3782439889 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb4cc068540 0x7fb4cc1980b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:25:35.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.787+0000 7fb4d2800700 1 --2- 172.21.15.62:0/3782439889 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb4cc1085b0 0x7fb4cc1985f0 unknown :-1 s=CLOSED pgs=1286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:25:35.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.787+0000 7fb4d2800700 1 -- 172.21.15.62:0/3782439889 >> 172.21.15.62:0/3782439889 conn(0x7fb4cc075130 msgr2=0x7fb4cc106390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:25:35.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.787+0000 7fb4d2800700 1 -- 172.21.15.62:0/3782439889 shutdown_connections 2024-04-20T22:25:35.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.787+0000 7fb4d2800700 1 -- 172.21.15.62:0/3782439889 wait complete. 2024-04-20T22:25:35.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.951+0000 7f7bbd600700 1 -- 172.21.15.62:0/3012708882 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7bb8106b80 msgr2=0x7f7bb8106ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:25:35.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.951+0000 7f7bbd600700 1 --2- 172.21.15.62:0/3012708882 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7bb8106b80 0x7f7bb8106ff0 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f7ba4009910 tx=0x7f7ba4009c20 comp rx=0 tx=0).stop 2024-04-20T22:25:35.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.951+0000 7f7bbd600700 1 -- 172.21.15.62:0/3012708882 shutdown_connections 2024-04-20T22:25:35.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.951+0000 7f7bbd600700 1 --2- 172.21.15.62:0/3012708882 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7bb8106b80 0x7f7bb8106ff0 unknown :-1 s=CLOSED pgs=1287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:25:35.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.951+0000 7f7bbd600700 1 --2- 172.21.15.62:0/3012708882 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7bb810c8f0 0x7f7bb810ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:25:35.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.951+0000 7f7bbd600700 1 -- 172.21.15.62:0/3012708882 >> 172.21.15.62:0/3012708882 conn(0x7f7bb8076060 msgr2=0x7f7bb8078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:25:35.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.951+0000 7f7bbd600700 1 -- 172.21.15.62:0/3012708882 shutdown_connections 2024-04-20T22:25:35.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.951+0000 7f7bbd600700 1 -- 172.21.15.62:0/3012708882 wait complete. 2024-04-20T22:25:35.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.952+0000 7f7bbd600700 1 Processor -- start 2024-04-20T22:25:35.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.953+0000 7f7bbd600700 1 -- start start 2024-04-20T22:25:35.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.953+0000 7f7bbd600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7bb8106b80 0x7f7bb819c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:25:35.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.953+0000 7f7bbd600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7bb810c8f0 0x7f7bb819c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:25:35.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.953+0000 7f7bbd600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bb819d060 con 0x7f7bb8106b80 2024-04-20T22:25:35.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.953+0000 7f7bbd600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bb81a0df0 con 0x7f7bb810c8f0 2024-04-20T22:25:35.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.953+0000 7f7bb6a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7bb8106b80 0x7f7bb819c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:25:35.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.953+0000 7f7bb6a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7bb8106b80 0x7f7bb819c440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44306/0 (socket says 172.21.15.62:44306) 2024-04-20T22:25:35.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.953+0000 7f7bb6a00700 1 -- 172.21.15.62:0/325966113 learned_addr learned my addr 172.21.15.62:0/325966113 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:25:35.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.954+0000 7f7bb6000700 1 --2- 172.21.15.62:0/325966113 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7bb810c8f0 0x7f7bb819c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:25:35.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.954+0000 7f7bb6a00700 1 -- 172.21.15.62:0/325966113 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7bb810c8f0 msgr2=0x7f7bb819c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:25:35.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.954+0000 7f7bb6a00700 1 --2- 172.21.15.62:0/325966113 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7bb810c8f0 0x7f7bb819c980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:25:35.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.954+0000 7f7bb6a00700 1 -- 172.21.15.62:0/325966113 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ba40095f0 con 0x7f7bb8106b80 2024-04-20T22:25:35.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.954+0000 7f7bb6a00700 1 --2- 172.21.15.62:0/325966113 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7bb8106b80 0x7f7bb819c440 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7f7ba800c8a0 tx=0x7f7ba800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:25:35.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.954+0000 7f7baf400700 1 -- 172.21.15.62:0/325966113 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7ba800cea0 con 0x7f7bb8106b80 2024-04-20T22:25:35.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.954+0000 7f7baf400700 1 -- 172.21.15.62:0/325966113 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7ba8004830 con 0x7f7bb8106b80 2024-04-20T22:25:35.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.954+0000 7f7baf400700 1 -- 172.21.15.62:0/325966113 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7ba80055b0 con 0x7f7bb8106b80 2024-04-20T22:25:35.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.954+0000 7f7bbd600700 1 -- 172.21.15.62:0/325966113 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bb81a10d0 con 0x7f7bb8106b80 2024-04-20T22:25:35.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.955+0000 7f7bbd600700 1 -- 172.21.15.62:0/325966113 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bb81a1620 con 0x7f7bb8106b80 2024-04-20T22:25:35.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.956+0000 7f7bbd600700 1 -- 172.21.15.62:0/325966113 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bb810ecc0 con 0x7f7bb8106b80 2024-04-20T22:25:35.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.957+0000 7f7baf400700 1 -- 172.21.15.62:0/325966113 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f7ba80049a0 con 0x7f7bb8106b80 2024-04-20T22:25:35.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.958+0000 7f7baf400700 1 --2- 172.21.15.62:0/325966113 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7b9c06ecc0 0x7f7b9c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:25:35.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.958+0000 7f7baf400700 1 -- 172.21.15.62:0/325966113 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f7ba808eaf0 con 0x7f7bb8106b80 2024-04-20T22:25:35.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.958+0000 7f7bb6000700 1 --2- 172.21.15.62:0/325966113 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7b9c06ecc0 0x7f7b9c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:25:35.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.959+0000 7f7bb6000700 1 --2- 172.21.15.62:0/325966113 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7b9c06ecc0 0x7f7b9c071170 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f7ba4000c00 tx=0x7f7ba4005fd0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:25:35.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:35.963+0000 7f7baf400700 1 -- 172.21.15.62:0/325966113 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f7ba8057880 con 0x7f7bb8106b80 2024-04-20T22:25:36.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:36.296+0000 7f7bbd600700 1 -- 172.21.15.62:0/325966113 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f7bb81a18f0 con 0x7f7bb8106b80 2024-04-20T22:25:36.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:36.299+0000 7f7baf400700 1 -- 172.21.15.62:0/325966113 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f7ba801d070 con 0x7f7bb8106b80 2024-04-20T22:25:36.300 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:25:36.300 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T22:25:36.300 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:25:36.300 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:25:36.300 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T22:25:36.301 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:25:36.301 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:25:36.301 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T22:25:36.301 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 6 2024-04-20T22:25:36.301 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:25:36.301 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T22:25:36.301 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 4 2024-04-20T22:25:36.301 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:25:36.301 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T22:25:36.301 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 14 2024-04-20T22:25:36.301 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T22:25:36.301 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:25:36.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:36.302+0000 7f7bbd600700 1 -- 172.21.15.62:0/325966113 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7b9c06ecc0 msgr2=0x7f7b9c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:25:36.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:36.302+0000 7f7bbd600700 1 --2- 172.21.15.62:0/325966113 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7b9c06ecc0 0x7f7b9c071170 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f7ba4000c00 tx=0x7f7ba4005fd0 comp rx=0 tx=0).stop 2024-04-20T22:25:36.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:36.302+0000 7f7bbd600700 1 -- 172.21.15.62:0/325966113 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7bb8106b80 msgr2=0x7f7bb819c440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:25:36.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:36.302+0000 7f7bbd600700 1 --2- 172.21.15.62:0/325966113 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7bb8106b80 0x7f7bb819c440 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7f7ba800c8a0 tx=0x7f7ba800cc60 comp rx=0 tx=0).stop 2024-04-20T22:25:36.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:36.303+0000 7f7bbd600700 1 -- 172.21.15.62:0/325966113 shutdown_connections 2024-04-20T22:25:36.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:36.303+0000 7f7bbd600700 1 --2- 172.21.15.62:0/325966113 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7b9c06ecc0 0x7f7b9c071170 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:25:36.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:36.303+0000 7f7bbd600700 1 --2- 172.21.15.62:0/325966113 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7bb8106b80 0x7f7bb819c440 unknown :-1 s=CLOSED pgs=1288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:25:36.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:36.303+0000 7f7bbd600700 1 --2- 172.21.15.62:0/325966113 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7bb810c8f0 0x7f7bb819c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:25:36.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:36.303+0000 7f7bbd600700 1 -- 172.21.15.62:0/325966113 >> 172.21.15.62:0/325966113 conn(0x7f7bb8076060 msgr2=0x7f7bb8077ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:25:36.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:36.303+0000 7f7bbd600700 1 -- 172.21.15.62:0/325966113 shutdown_connections 2024-04-20T22:25:36.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:36.303+0000 7f7bbd600700 1 -- 172.21.15.62:0/325966113 wait complete. 2024-04-20T22:25:36.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:36.463+0000 7f2fe8c00700 1 -- 172.21.15.62:0/30858452 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2fe4100e80 msgr2=0x7f2fe4104e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:25:36.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:36.463+0000 7f2fe8c00700 1 --2- 172.21.15.62:0/30858452 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2fe4100e80 0x7f2fe4104e50 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7f2fcc009910 tx=0x7f2fcc009c20 comp rx=0 tx=0).stop 2024-04-20T22:25:36.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:36.463+0000 7f2fe8c00700 1 -- 172.21.15.62:0/30858452 shutdown_connections 2024-04-20T22:25:36.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:36.463+0000 7f2fe8c00700 1 --2- 172.21.15.62:0/30858452 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2fe4100e80 0x7f2fe4104e50 unknown :-1 s=CLOSED pgs=1289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:25:36.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:36.463+0000 7f2fe8c00700 1 --2- 172.21.15.62:0/30858452 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2fe41004e0 0x7f2fe41008b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:25:36.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:36.463+0000 7f2fe8c00700 1 -- 172.21.15.62:0/30858452 >> 172.21.15.62:0/30858452 conn(0x7f2fe40fbda0 msgr2=0x7f2fe40fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:25:36.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:36.463+0000 7f2fe8c00700 1 -- 172.21.15.62:0/30858452 shutdown_connections 2024-04-20T22:25:36.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:36.463+0000 7f2fe8c00700 1 -- 172.21.15.62:0/30858452 wait complete. 2024-04-20T22:25:36.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:36.464+0000 7f2fe8c00700 1 Processor -- start 2024-04-20T22:25:36.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:36.465+0000 7f2fe8c00700 1 -- start start 2024-04-20T22:25:36.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:36.465+0000 7f2fe8c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2fe41004e0 0x7f2fe4195f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:25:36.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:36.465+0000 7f2fe8c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2fe4100e80 0x7f2fe4196450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:25:36.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:36.465+0000 7f2fe8c00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe4196aa0 con 0x7f2fe4100e80 2024-04-20T22:25:36.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:36.465+0000 7f2fe8c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe4196be0 con 0x7f2fe41004e0 2024-04-20T22:25:36.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:36.466+0000 7f2fdbe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2fe4100e80 0x7f2fe4196450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:25:36.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:36.466+0000 7f2fdbe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2fe4100e80 0x7f2fe4196450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44338/0 (socket says 172.21.15.62:44338) 2024-04-20T22:25:36.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:36.466+0000 7f2fdbe00700 1 -- 172.21.15.62:0/2120480036 learned_addr learned my addr 172.21.15.62:0/2120480036 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:25:36.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:36.466+0000 7f2fe1e00700 1 --2- 172.21.15.62:0/2120480036 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2fe41004e0 0x7f2fe4195f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:25:36.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:36.466+0000 7f2fdbe00700 1 -- 172.21.15.62:0/2120480036 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2fe41004e0 msgr2=0x7f2fe4195f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:25:36.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:36.466+0000 7f2fdbe00700 1 --2- 172.21.15.62:0/2120480036 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2fe41004e0 0x7f2fe4195f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:25:36.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:36.466+0000 7f2fdbe00700 1 -- 172.21.15.62:0/2120480036 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fcc0095f0 con 0x7f2fe4100e80 2024-04-20T22:25:36.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:36.466+0000 7f2fdbe00700 1 --2- 172.21.15.62:0/2120480036 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2fe4100e80 0x7f2fe4196450 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7f2fcc0092e0 tx=0x7f2fcc00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:25:36.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:36.467+0000 7f2fdaa00700 1 -- 172.21.15.62:0/2120480036 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2fcc01d070 con 0x7f2fe4100e80 2024-04-20T22:25:36.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:36.467+0000 7f2fe8c00700 1 -- 172.21.15.62:0/2120480036 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fe419a9d0 con 0x7f2fe4100e80 2024-04-20T22:25:36.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:36.467+0000 7f2fdaa00700 1 -- 172.21.15.62:0/2120480036 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2fcc00f460 con 0x7f2fe4100e80 2024-04-20T22:25:36.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:36.467+0000 7f2fdaa00700 1 -- 172.21.15.62:0/2120480036 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2fcc0175f0 con 0x7f2fe4100e80 2024-04-20T22:25:36.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:36.467+0000 7f2fe8c00700 1 -- 172.21.15.62:0/2120480036 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fe419aec0 con 0x7f2fe4100e80 2024-04-20T22:25:36.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:36.468+0000 7f2fe8c00700 1 -- 172.21.15.62:0/2120480036 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fe4108850 con 0x7f2fe4100e80 2024-04-20T22:25:36.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:36.470+0000 7f2fdaa00700 1 -- 172.21.15.62:0/2120480036 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f2fcc00f5d0 con 0x7f2fe4100e80 2024-04-20T22:25:36.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:36.471+0000 7f2fdaa00700 1 --2- 172.21.15.62:0/2120480036 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2fc806ecc0 0x7f2fc8071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:25:36.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:36.471+0000 7f2fdaa00700 1 -- 172.21.15.62:0/2120480036 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f2fcc090370 con 0x7f2fe4100e80 2024-04-20T22:25:36.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:36.472+0000 7f2fe1e00700 1 --2- 172.21.15.62:0/2120480036 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2fc806ecc0 0x7f2fc8071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:25:36.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:36.473+0000 7f2fe1e00700 1 --2- 172.21.15.62:0/2120480036 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2fc806ecc0 0x7f2fc8071170 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f2fd400ba60 tx=0x7f2fd4005d50 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:25:36.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:36.476+0000 7f2fdaa00700 1 -- 172.21.15.62:0/2120480036 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f2fcc059100 con 0x7f2fe4100e80 2024-04-20T22:25:36.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:25:36 smithi189 ceph-mon[57022]: from='client.36636 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:25:36.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:25:36 smithi189 ceph-mon[57022]: from='client.36640 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:25:36.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:36.764+0000 7f2fe8c00700 1 -- 172.21.15.62:0/2120480036 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f2fe404f2a0 con 0x7f2fe4100e80 2024-04-20T22:25:36.764 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:25:36 smithi062 ceph-mon[76322]: from='client.36636 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:25:36.764 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:25:36 smithi062 ceph-mon[76322]: from='client.36640 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:25:36.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:36.765+0000 7f2fdaa00700 1 -- 172.21.15.62:0/2120480036 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 29 v29) v1 ==== 76+0+1887 (secure 0 0 0) 0x7f2fcc05c7c0 con 0x7f2fe4100e80 2024-04-20T22:25:36.766 INFO:teuthology.orchestra.run.smithi062.stdout:e29 2024-04-20T22:25:36.766 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T22:25:36.766 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:25:36.766 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T22:25:36.766 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:25:36.766 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T22:25:36.767 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T22:25:36.767 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 20 2024-04-20T22:25:36.767 INFO:teuthology.orchestra.run.smithi062.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-20T22:25:36.767 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T22:25:36.767 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T21:42:01.559061+0000 2024-04-20T22:25:36.767 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T22:25:36.768 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T22:25:36.768 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T22:25:36.768 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T22:25:36.768 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T22:25:36.768 INFO:teuthology.orchestra.run.smithi062.stdout:max_xattr_size 65536 2024-04-20T22:25:36.768 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T22:25:36.769 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T22:25:36.769 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 81 2024-04-20T22:25:36.769 INFO:teuthology.orchestra.run.smithi062.stdout:snap_rstat 1 2024-04-20T22:25:36.769 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:25:36.769 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 2 2024-04-20T22:25:36.770 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1 2024-04-20T22:25:36.770 INFO:teuthology.orchestra.run.smithi062.stdout:up {} 2024-04-20T22:25:36.770 INFO:teuthology.orchestra.run.smithi062.stdout:failed 0,1 2024-04-20T22:25:36.770 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T22:25:36.770 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T22:25:36.770 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T22:25:36.771 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T22:25:36.771 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data enabled 2024-04-20T22:25:36.771 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T22:25:36.771 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T22:25:36.771 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 1 2024-04-20T22:25:36.771 INFO:teuthology.orchestra.run.smithi062.stdout:qdb_cluster leader: 0 members: 2024-04-20T22:25:36.772 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:25:36.772 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:25:36.772 INFO:teuthology.orchestra.run.smithi062.stdout:Standby daemons: 2024-04-20T22:25:36.772 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:25:36.772 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ryqblb{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6826/4113916205,v1:172.21.15.62:6827/4113916205] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:25:36.772 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ozizwi{-1:34476} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6828/3801975165,v1:172.21.15.62:6829/3801975165] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:25:36.773 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.dvimer{-1:44727} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6824/3576699232,v1:172.21.15.189:6825/3576699232] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:25:36.773 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.xbgspw{-1:45075} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6826/212905792,v1:172.21.15.189:6827/212905792] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:25:36.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:36.770+0000 7f2fe8c00700 1 -- 172.21.15.62:0/2120480036 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2fc806ecc0 msgr2=0x7f2fc8071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:25:36.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:36.770+0000 7f2fe8c00700 1 --2- 172.21.15.62:0/2120480036 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2fc806ecc0 0x7f2fc8071170 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f2fd400ba60 tx=0x7f2fd4005d50 comp rx=0 tx=0).stop 2024-04-20T22:25:36.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:36.770+0000 7f2fe8c00700 1 -- 172.21.15.62:0/2120480036 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2fe4100e80 msgr2=0x7f2fe4196450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:25:36.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:36.770+0000 7f2fe8c00700 1 --2- 172.21.15.62:0/2120480036 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2fe4100e80 0x7f2fe4196450 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7f2fcc0092e0 tx=0x7f2fcc00be20 comp rx=0 tx=0).stop 2024-04-20T22:25:36.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:36.770+0000 7f2fe8c00700 1 -- 172.21.15.62:0/2120480036 shutdown_connections 2024-04-20T22:25:36.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:36.770+0000 7f2fe8c00700 1 --2- 172.21.15.62:0/2120480036 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f2fc806ecc0 0x7f2fc8071170 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:25:36.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:36.770+0000 7f2fe8c00700 1 --2- 172.21.15.62:0/2120480036 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2fe41004e0 0x7f2fe4195f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:25:36.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:36.770+0000 7f2fe8c00700 1 --2- 172.21.15.62:0/2120480036 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2fe4100e80 0x7f2fe4196450 unknown :-1 s=CLOSED pgs=1290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:25:36.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:36.770+0000 7f2fe8c00700 1 -- 172.21.15.62:0/2120480036 >> 172.21.15.62:0/2120480036 conn(0x7f2fe40fbda0 msgr2=0x7f2fe40fe180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:25:36.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:36.770+0000 7f2fe8c00700 1 -- 172.21.15.62:0/2120480036 shutdown_connections 2024-04-20T22:25:36.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:36.770+0000 7f2fe8c00700 1 -- 172.21.15.62:0/2120480036 wait complete. 2024-04-20T22:25:36.776 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 29 2024-04-20T22:25:36.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:36.932+0000 7fb1dde00700 1 -- 172.21.15.62:0/1859966332 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb1d8106b80 msgr2=0x7fb1d8106ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:25:36.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:36.932+0000 7fb1dde00700 1 --2- 172.21.15.62:0/1859966332 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb1d8106b80 0x7fb1d8106ff0 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7fb1c4009910 tx=0x7fb1c4009c20 comp rx=0 tx=0).stop 2024-04-20T22:25:36.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:36.932+0000 7fb1dde00700 1 -- 172.21.15.62:0/1859966332 shutdown_connections 2024-04-20T22:25:36.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:36.932+0000 7fb1dde00700 1 --2- 172.21.15.62:0/1859966332 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb1d8106b80 0x7fb1d8106ff0 unknown :-1 s=CLOSED pgs=1291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:25:36.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:36.932+0000 7fb1dde00700 1 --2- 172.21.15.62:0/1859966332 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb1d810c8f0 0x7fb1d810ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:25:36.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:36.932+0000 7fb1dde00700 1 -- 172.21.15.62:0/1859966332 >> 172.21.15.62:0/1859966332 conn(0x7fb1d8076060 msgr2=0x7fb1d8078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:25:36.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:36.932+0000 7fb1dde00700 1 -- 172.21.15.62:0/1859966332 shutdown_connections 2024-04-20T22:25:36.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:36.932+0000 7fb1dde00700 1 -- 172.21.15.62:0/1859966332 wait complete. 2024-04-20T22:25:36.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:36.933+0000 7fb1dde00700 1 Processor -- start 2024-04-20T22:25:36.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:36.933+0000 7fb1dde00700 1 -- start start 2024-04-20T22:25:36.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:36.934+0000 7fb1dde00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb1d8106b80 0x7fb1d819c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:25:36.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:36.934+0000 7fb1dde00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb1d810c8f0 0x7fb1d819ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:25:36.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:36.934+0000 7fb1dde00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1d819d100 con 0x7fb1d8106b80 2024-04-20T22:25:36.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:36.934+0000 7fb1dde00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1d81a0e90 con 0x7fb1d810c8f0 2024-04-20T22:25:36.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:36.934+0000 7fb1d6a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb1d8106b80 0x7fb1d819c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:25:36.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:36.934+0000 7fb1d6a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb1d8106b80 0x7fb1d819c4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44342/0 (socket says 172.21.15.62:44342) 2024-04-20T22:25:36.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:36.934+0000 7fb1d6a00700 1 -- 172.21.15.62:0/110731765 learned_addr learned my addr 172.21.15.62:0/110731765 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:25:36.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:36.935+0000 7fb1d6a00700 1 -- 172.21.15.62:0/110731765 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb1d810c8f0 msgr2=0x7fb1d819ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:25:36.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:36.935+0000 7fb1d6a00700 1 --2- 172.21.15.62:0/110731765 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb1d810c8f0 0x7fb1d819ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:25:36.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:36.935+0000 7fb1d6a00700 1 -- 172.21.15.62:0/110731765 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1c40095f0 con 0x7fb1d8106b80 2024-04-20T22:25:36.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:36.935+0000 7fb1d6a00700 1 --2- 172.21.15.62:0/110731765 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb1d8106b80 0x7fb1d819c4e0 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7fb1c800b4b0 tx=0x7fb1c800b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:25:36.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:36.935+0000 7fb1dca00700 1 -- 172.21.15.62:0/110731765 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb1c8011800 con 0x7fb1d8106b80 2024-04-20T22:25:36.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:36.935+0000 7fb1dca00700 1 -- 172.21.15.62:0/110731765 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb1c8011e20 con 0x7fb1d8106b80 2024-04-20T22:25:36.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:36.935+0000 7fb1dde00700 1 -- 172.21.15.62:0/110731765 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1d81a1170 con 0x7fb1d8106b80 2024-04-20T22:25:36.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:36.935+0000 7fb1dca00700 1 -- 172.21.15.62:0/110731765 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb1c800f2e0 con 0x7fb1d8106b80 2024-04-20T22:25:36.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:36.935+0000 7fb1dde00700 1 -- 172.21.15.62:0/110731765 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1d81a16c0 con 0x7fb1d8106b80 2024-04-20T22:25:36.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:36.937+0000 7fb1dde00700 1 -- 172.21.15.62:0/110731765 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1d8109b20 con 0x7fb1d8106b80 2024-04-20T22:25:36.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:36.938+0000 7fb1dca00700 1 -- 172.21.15.62:0/110731765 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fb1c8010380 con 0x7fb1d8106b80 2024-04-20T22:25:36.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:36.939+0000 7fb1dca00700 1 --2- 172.21.15.62:0/110731765 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb1bc06ecc0 0x7fb1bc071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:25:36.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:36.939+0000 7fb1dca00700 1 -- 172.21.15.62:0/110731765 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fb1c808eaa0 con 0x7fb1d8106b80 2024-04-20T22:25:36.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:36.939+0000 7fb1d6000700 1 --2- 172.21.15.62:0/110731765 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb1bc06ecc0 0x7fb1bc071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:25:36.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:36.940+0000 7fb1d6000700 1 --2- 172.21.15.62:0/110731765 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb1bc06ecc0 0x7fb1bc071170 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fb1c4006010 tx=0x7fb1c400b540 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:25:36.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:36.944+0000 7fb1dca00700 1 -- 172.21.15.62:0/110731765 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fb1c80577b0 con 0x7fb1d8106b80 2024-04-20T22:25:37.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:37.200+0000 7fb1dde00700 1 -- 172.21.15.62:0/110731765 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb1d81a19b0 con 0x7fb1bc06ecc0 2024-04-20T22:25:37.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:37.202+0000 7fb1dca00700 1 -- 172.21.15.62:0/110731765 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fb1d81a19b0 con 0x7fb1bc06ecc0 2024-04-20T22:25:37.202 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:25:37.202 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T22:25:37.202 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T22:25:37.202 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-20T22:25:37.202 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T22:25:37.202 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr", 2024-04-20T22:25:37.202 INFO:teuthology.orchestra.run.smithi062.stdout: "osd", 2024-04-20T22:25:37.202 INFO:teuthology.orchestra.run.smithi062.stdout: "mds", 2024-04-20T22:25:37.202 INFO:teuthology.orchestra.run.smithi062.stdout: "crash", 2024-04-20T22:25:37.202 INFO:teuthology.orchestra.run.smithi062.stdout: "mon" 2024-04-20T22:25:37.203 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T22:25:37.203 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "16/23 daemons upgraded", 2024-04-20T22:25:37.203 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading mds daemons", 2024-04-20T22:25:37.203 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T22:25:37.203 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:25:37.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:37.204+0000 7fb1dde00700 1 -- 172.21.15.62:0/110731765 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb1bc06ecc0 msgr2=0x7fb1bc071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:25:37.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:37.204+0000 7fb1dde00700 1 --2- 172.21.15.62:0/110731765 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb1bc06ecc0 0x7fb1bc071170 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fb1c4006010 tx=0x7fb1c400b540 comp rx=0 tx=0).stop 2024-04-20T22:25:37.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:37.205+0000 7fb1dde00700 1 -- 172.21.15.62:0/110731765 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb1d8106b80 msgr2=0x7fb1d819c4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:25:37.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:37.205+0000 7fb1dde00700 1 --2- 172.21.15.62:0/110731765 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb1d8106b80 0x7fb1d819c4e0 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7fb1c800b4b0 tx=0x7fb1c800b870 comp rx=0 tx=0).stop 2024-04-20T22:25:37.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:37.205+0000 7fb1dde00700 1 -- 172.21.15.62:0/110731765 shutdown_connections 2024-04-20T22:25:37.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:37.205+0000 7fb1dde00700 1 --2- 172.21.15.62:0/110731765 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb1bc06ecc0 0x7fb1bc071170 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:25:37.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:37.205+0000 7fb1dde00700 1 --2- 172.21.15.62:0/110731765 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb1d8106b80 0x7fb1d819c4e0 unknown :-1 s=CLOSED pgs=1292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:25:37.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:37.205+0000 7fb1dde00700 1 --2- 172.21.15.62:0/110731765 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb1d810c8f0 0x7fb1d819ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:25:37.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:37.205+0000 7fb1dde00700 1 -- 172.21.15.62:0/110731765 >> 172.21.15.62:0/110731765 conn(0x7fb1d8076060 msgr2=0x7fb1d8077a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:25:37.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:37.205+0000 7fb1dde00700 1 -- 172.21.15.62:0/110731765 shutdown_connections 2024-04-20T22:25:37.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:37.205+0000 7fb1dde00700 1 -- 172.21.15.62:0/110731765 wait complete. 2024-04-20T22:25:37.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:37.357+0000 7ff3faa00700 1 -- 172.21.15.62:0/1027211221 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff3f41004e0 msgr2=0x7ff3f41008b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:25:37.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:37.357+0000 7ff3faa00700 1 --2- 172.21.15.62:0/1027211221 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff3f41004e0 0x7ff3f41008b0 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7ff3e400b0a0 tx=0x7ff3e400b3b0 comp rx=0 tx=0).stop 2024-04-20T22:25:37.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:37.358+0000 7ff3faa00700 1 -- 172.21.15.62:0/1027211221 shutdown_connections 2024-04-20T22:25:37.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:37.358+0000 7ff3faa00700 1 --2- 172.21.15.62:0/1027211221 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3f4100e80 0x7ff3f4104e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:25:37.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:37.358+0000 7ff3faa00700 1 --2- 172.21.15.62:0/1027211221 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff3f41004e0 0x7ff3f41008b0 unknown :-1 s=CLOSED pgs=1293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:25:37.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:37.358+0000 7ff3faa00700 1 -- 172.21.15.62:0/1027211221 >> 172.21.15.62:0/1027211221 conn(0x7ff3f40fbda0 msgr2=0x7ff3f40fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:25:37.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:37.358+0000 7ff3faa00700 1 -- 172.21.15.62:0/1027211221 shutdown_connections 2024-04-20T22:25:37.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:37.358+0000 7ff3faa00700 1 -- 172.21.15.62:0/1027211221 wait complete. 2024-04-20T22:25:37.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:37.359+0000 7ff3faa00700 1 Processor -- start 2024-04-20T22:25:37.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:37.359+0000 7ff3faa00700 1 -- start start 2024-04-20T22:25:37.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:37.359+0000 7ff3faa00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3f41004e0 0x7ff3f4196010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:25:37.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:37.359+0000 7ff3faa00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff3f4100e80 0x7ff3f4196550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:25:37.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:37.359+0000 7ff3faa00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3f4196c30 con 0x7ff3f4100e80 2024-04-20T22:25:37.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:37.359+0000 7ff3faa00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3f419a9c0 con 0x7ff3f41004e0 2024-04-20T22:25:37.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:37.360+0000 7ff3eb400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff3f4100e80 0x7ff3f4196550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:25:37.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:37.360+0000 7ff3eb400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff3f4100e80 0x7ff3f4196550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44354/0 (socket says 172.21.15.62:44354) 2024-04-20T22:25:37.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:37.360+0000 7ff3eb400700 1 -- 172.21.15.62:0/107385487 learned_addr learned my addr 172.21.15.62:0/107385487 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:25:37.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:37.360+0000 7ff3f3e00700 1 --2- 172.21.15.62:0/107385487 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3f41004e0 0x7ff3f4196010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:25:37.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:37.360+0000 7ff3eb400700 1 -- 172.21.15.62:0/107385487 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3f41004e0 msgr2=0x7ff3f4196010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:25:37.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:37.360+0000 7ff3eb400700 1 --2- 172.21.15.62:0/107385487 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3f41004e0 0x7ff3f4196010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:25:37.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:37.360+0000 7ff3eb400700 1 -- 172.21.15.62:0/107385487 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3e4009d00 con 0x7ff3f4100e80 2024-04-20T22:25:37.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:37.361+0000 7ff3eb400700 1 --2- 172.21.15.62:0/107385487 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff3f4100e80 0x7ff3f4196550 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7ff3dc00b480 tx=0x7ff3dc00b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:25:37.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:37.361+0000 7ff3f1600700 1 -- 172.21.15.62:0/107385487 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff3dc0118c0 con 0x7ff3f4100e80 2024-04-20T22:25:37.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:37.361+0000 7ff3f1600700 1 -- 172.21.15.62:0/107385487 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff3dc010380 con 0x7ff3f4100e80 2024-04-20T22:25:37.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:37.361+0000 7ff3faa00700 1 -- 172.21.15.62:0/107385487 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3f419aca0 con 0x7ff3f4100e80 2024-04-20T22:25:37.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:37.361+0000 7ff3f1600700 1 -- 172.21.15.62:0/107385487 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff3dc00f4f0 con 0x7ff3f4100e80 2024-04-20T22:25:37.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:37.361+0000 7ff3faa00700 1 -- 172.21.15.62:0/107385487 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3f419b1f0 con 0x7ff3f4100e80 2024-04-20T22:25:37.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:37.362+0000 7ff3faa00700 1 -- 172.21.15.62:0/107385487 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3f41087f0 con 0x7ff3f4100e80 2024-04-20T22:25:37.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:37.364+0000 7ff3f1600700 1 -- 172.21.15.62:0/107385487 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff3dc0104f0 con 0x7ff3f4100e80 2024-04-20T22:25:37.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:37.364+0000 7ff3f1600700 1 --2- 172.21.15.62:0/107385487 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff3d806ecc0 0x7ff3d8071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:25:37.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:37.365+0000 7ff3f1600700 1 -- 172.21.15.62:0/107385487 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7ff3dc08eeb0 con 0x7ff3f4100e80 2024-04-20T22:25:37.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:37.365+0000 7ff3f3e00700 1 --2- 172.21.15.62:0/107385487 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff3d806ecc0 0x7ff3d8071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:25:37.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:37.365+0000 7ff3f3e00700 1 --2- 172.21.15.62:0/107385487 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff3d806ecc0 0x7ff3d8071170 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7ff3e4009480 tx=0x7ff3e400b7f0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:25:37.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:37.368+0000 7ff3f1600700 1 -- 172.21.15.62:0/107385487 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7ff3dc057ce0 con 0x7ff3f4100e80 2024-04-20T22:25:37.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:25:37 smithi062 ceph-mon[76322]: pgmap v1512: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:25:37.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:25:37 smithi062 ceph-mon[76322]: from='client.36644 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:25:37.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:25:37 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/325966113' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:25:37.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:25:37 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/2120480036' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:25:37.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:37.688+0000 7ff3faa00700 1 -- 172.21.15.62:0/107385487 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7ff3f404f2a0 con 0x7ff3f4100e80 2024-04-20T22:25:37.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:37.689+0000 7ff3f1600700 1 -- 172.21.15.62:0/107385487 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7ff3dc05b3a0 con 0x7ff3f4100e80 2024-04-20T22:25:37.689 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-20T22:25:37.690 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-20T22:25:37.690 INFO:teuthology.orchestra.run.smithi062.stdout: host smithi189 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi189/config 2024-04-20T22:25:37.690 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-20T22:25:37.690 INFO:teuthology.orchestra.run.smithi062.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi189 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp11670v2q:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:25:37.690 INFO:teuthology.orchestra.run.smithi062.stdout:Traceback (most recent call last): 2024-04-20T22:25:37.690 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-20T22:25:37.690 INFO:teuthology.orchestra.run.smithi062.stdout: return _run_code(code, main_globals, None, 2024-04-20T22:25:37.690 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-20T22:25:37.690 INFO:teuthology.orchestra.run.smithi062.stdout: exec(code, run_globals) 2024-04-20T22:25:37.690 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5618, in 2024-04-20T22:25:37.690 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5606, in main 2024-04-20T22:25:37.690 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 408, in _infer_config 2024-04-20T22:25:37.691 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 323, in _infer_fsid 2024-04-20T22:25:37.691 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 436, in _infer_image 2024-04-20T22:25:37.691 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 310, in _validate_fsid 2024-04-20T22:25:37.691 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 3359, in command_ceph_volume 2024-04-20T22:25:37.691 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-20T22:25:37.691 INFO:teuthology.orchestra.run.smithi062.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi189 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp11670v2q:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:25:37.691 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-20T22:25:37.691 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is degraded 2024-04-20T22:25:37.691 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T22:25:37.691 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T22:25:37.691 INFO:teuthology.orchestra.run.smithi062.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-20T22:25:37.691 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-20T22:25:37.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:37.692+0000 7ff3faa00700 1 -- 172.21.15.62:0/107385487 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff3d806ecc0 msgr2=0x7ff3d8071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:25:37.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:37.692+0000 7ff3faa00700 1 --2- 172.21.15.62:0/107385487 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff3d806ecc0 0x7ff3d8071170 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7ff3e4009480 tx=0x7ff3e400b7f0 comp rx=0 tx=0).stop 2024-04-20T22:25:37.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:37.692+0000 7ff3faa00700 1 -- 172.21.15.62:0/107385487 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff3f4100e80 msgr2=0x7ff3f4196550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:25:37.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:37.692+0000 7ff3faa00700 1 --2- 172.21.15.62:0/107385487 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff3f4100e80 0x7ff3f4196550 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7ff3dc00b480 tx=0x7ff3dc00b840 comp rx=0 tx=0).stop 2024-04-20T22:25:37.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:37.692+0000 7ff3faa00700 1 -- 172.21.15.62:0/107385487 shutdown_connections 2024-04-20T22:25:37.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:37.692+0000 7ff3faa00700 1 --2- 172.21.15.62:0/107385487 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff3d806ecc0 0x7ff3d8071170 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:25:37.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:37.692+0000 7ff3faa00700 1 --2- 172.21.15.62:0/107385487 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3f41004e0 0x7ff3f4196010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:25:37.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:37.692+0000 7ff3faa00700 1 --2- 172.21.15.62:0/107385487 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff3f4100e80 0x7ff3f4196550 unknown :-1 s=CLOSED pgs=1294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:25:37.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:37.692+0000 7ff3faa00700 1 -- 172.21.15.62:0/107385487 >> 172.21.15.62:0/107385487 conn(0x7ff3f40fbda0 msgr2=0x7ff3f40fd480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:25:37.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:37.692+0000 7ff3faa00700 1 -- 172.21.15.62:0/107385487 shutdown_connections 2024-04-20T22:25:37.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:25:37.692+0000 7ff3faa00700 1 -- 172.21.15.62:0/107385487 wait complete. 2024-04-20T22:25:37.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:25:37 smithi189 ceph-mon[57022]: pgmap v1512: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:25:37.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:25:37 smithi189 ceph-mon[57022]: from='client.36644 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:25:37.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:25:37 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/325966113' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:25:37.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:25:37 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/2120480036' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:25:38.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:25:38 smithi062 ceph-mon[76322]: from='client.36656 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:25:38.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:25:38 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/107385487' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:25:38.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:25:38 smithi189 ceph-mon[57022]: from='client.36656 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:25:38.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:25:38 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/107385487' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:25:39.658 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:25:39 smithi062 ceph-mon[76322]: pgmap v1513: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:25:39.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:25:39 smithi189 ceph-mon[57022]: pgmap v1513: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:25:41.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:25:41 smithi189 ceph-mon[57022]: pgmap v1514: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:25:41.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:25:41 smithi062 ceph-mon[76322]: pgmap v1514: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:25:43.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:25:43 smithi189 ceph-mon[57022]: pgmap v1515: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:25:43.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:25:43 smithi062 ceph-mon[76322]: pgmap v1515: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:25:45.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:25:45 smithi189 ceph-mon[57022]: pgmap v1516: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:25:45.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:25:45 smithi062 ceph-mon[76322]: pgmap v1516: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:25:47.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:25:47 smithi189 ceph-mon[57022]: pgmap v1517: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:25:47.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:25:47 smithi062 ceph-mon[76322]: pgmap v1517: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:25:49.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:25:49 smithi189 ceph-mon[57022]: pgmap v1518: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:25:49.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:25:49 smithi062 ceph-mon[76322]: pgmap v1518: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:25:50.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:25:50 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:25:50.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:25:50 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:25:51.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:25:51 smithi189 ceph-mon[57022]: pgmap v1519: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:25:51.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:25:51 smithi062 ceph-mon[76322]: pgmap v1519: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:25:53.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:25:53 smithi189 ceph-mon[57022]: pgmap v1520: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:25:53.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:25:53 smithi062 ceph-mon[76322]: pgmap v1520: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:25:55.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:25:55 smithi062 ceph-mon[76322]: pgmap v1521: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:25:55.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:25:55 smithi189 ceph-mon[57022]: pgmap v1521: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:25:57.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:25:57 smithi189 ceph-mon[57022]: pgmap v1522: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:25:57.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:25:57 smithi062 ceph-mon[76322]: pgmap v1522: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:25:59.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:25:59 smithi189 ceph-mon[57022]: pgmap v1523: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:25:59.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:25:59 smithi062 ceph-mon[76322]: pgmap v1523: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:26:01.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:26:01 smithi189 ceph-mon[57022]: pgmap v1524: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:26:01.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:26:01 smithi062 ceph-mon[76322]: pgmap v1524: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:26:03.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:26:03 smithi189 ceph-mon[57022]: pgmap v1525: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:26:03.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:26:03 smithi062 ceph-mon[76322]: pgmap v1525: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:26:05.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:26:05 smithi189 ceph-mon[57022]: pgmap v1526: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:26:05.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:26:05 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:26:05.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:26:05 smithi062 ceph-mon[76322]: pgmap v1526: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:26:05.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:26:05 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:26:07.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:26:07 smithi189 ceph-mon[57022]: pgmap v1527: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:26:07.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:26:07 smithi062 ceph-mon[76322]: pgmap v1527: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:26:07.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:07.871+0000 7f3c74800700 1 -- 172.21.15.62:0/2240220347 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3c6c108470 msgr2=0x7f3c6c108840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:26:07.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:07.871+0000 7f3c74800700 1 --2- 172.21.15.62:0/2240220347 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3c6c108470 0x7f3c6c108840 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7f3c60009910 tx=0x7f3c60009c20 comp rx=0 tx=0).stop 2024-04-20T22:26:07.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:07.871+0000 7f3c74800700 1 -- 172.21.15.62:0/2240220347 shutdown_connections 2024-04-20T22:26:07.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:07.871+0000 7f3c74800700 1 --2- 172.21.15.62:0/2240220347 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3c6c102470 0x7f3c6c1028e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:26:07.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:07.871+0000 7f3c74800700 1 --2- 172.21.15.62:0/2240220347 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3c6c108470 0x7f3c6c108840 unknown :-1 s=CLOSED pgs=1295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:26:07.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:07.871+0000 7f3c74800700 1 -- 172.21.15.62:0/2240220347 >> 172.21.15.62:0/2240220347 conn(0x7f3c6c0fdf90 msgr2=0x7f3c6c1003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:26:07.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:07.871+0000 7f3c74800700 1 -- 172.21.15.62:0/2240220347 shutdown_connections 2024-04-20T22:26:07.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:07.871+0000 7f3c74800700 1 -- 172.21.15.62:0/2240220347 wait complete. 2024-04-20T22:26:07.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:07.872+0000 7f3c74800700 1 Processor -- start 2024-04-20T22:26:07.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:07.872+0000 7f3c74800700 1 -- start start 2024-04-20T22:26:07.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:07.873+0000 7f3c74800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3c6c102470 0x7f3c6c1981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:26:07.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:07.873+0000 7f3c74800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3c6c108470 0x7f3c6c198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:26:07.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:07.873+0000 7f3c74800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c6c198e10 con 0x7f3c6c108470 2024-04-20T22:26:07.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:07.873+0000 7f3c74800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c6c19cba0 con 0x7f3c6c102470 2024-04-20T22:26:07.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:07.873+0000 7f3c71400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3c6c108470 0x7f3c6c198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:26:07.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:07.873+0000 7f3c71400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3c6c108470 0x7f3c6c198730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53150/0 (socket says 172.21.15.62:53150) 2024-04-20T22:26:07.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:07.873+0000 7f3c71400700 1 -- 172.21.15.62:0/3803506211 learned_addr learned my addr 172.21.15.62:0/3803506211 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:26:07.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:07.874+0000 7f3c71e00700 1 --2- 172.21.15.62:0/3803506211 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3c6c102470 0x7f3c6c1981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:26:07.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:07.874+0000 7f3c71400700 1 -- 172.21.15.62:0/3803506211 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3c6c102470 msgr2=0x7f3c6c1981f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:26:07.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:07.874+0000 7f3c71400700 1 --2- 172.21.15.62:0/3803506211 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3c6c102470 0x7f3c6c1981f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:26:07.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:07.874+0000 7f3c71400700 1 -- 172.21.15.62:0/3803506211 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c600095f0 con 0x7f3c6c108470 2024-04-20T22:26:07.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:07.874+0000 7f3c71400700 1 --2- 172.21.15.62:0/3803506211 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3c6c108470 0x7f3c6c198730 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f3c5c00d8d0 tx=0x7f3c5c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:26:07.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:07.874+0000 7f3c66a00700 1 -- 172.21.15.62:0/3803506211 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3c5c009940 con 0x7f3c6c108470 2024-04-20T22:26:07.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:07.874+0000 7f3c66a00700 1 -- 172.21.15.62:0/3803506211 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3c5c010460 con 0x7f3c6c108470 2024-04-20T22:26:07.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:07.874+0000 7f3c66a00700 1 -- 172.21.15.62:0/3803506211 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3c5c00f5f0 con 0x7f3c6c108470 2024-04-20T22:26:07.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:07.874+0000 7f3c74800700 1 -- 172.21.15.62:0/3803506211 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c6c19ce80 con 0x7f3c6c108470 2024-04-20T22:26:07.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:07.875+0000 7f3c74800700 1 -- 172.21.15.62:0/3803506211 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c6c19d3d0 con 0x7f3c6c108470 2024-04-20T22:26:07.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:07.876+0000 7f3c74800700 1 -- 172.21.15.62:0/3803506211 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c6c10a9e0 con 0x7f3c6c108470 2024-04-20T22:26:07.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:07.877+0000 7f3c66a00700 1 -- 172.21.15.62:0/3803506211 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f3c5c0105d0 con 0x7f3c6c108470 2024-04-20T22:26:07.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:07.878+0000 7f3c66a00700 1 --2- 172.21.15.62:0/3803506211 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3c5406ecc0 0x7f3c54071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:26:07.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:07.878+0000 7f3c66a00700 1 -- 172.21.15.62:0/3803506211 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f3c5c08ef20 con 0x7f3c6c108470 2024-04-20T22:26:07.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:07.878+0000 7f3c71e00700 1 --2- 172.21.15.62:0/3803506211 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3c5406ecc0 0x7f3c54071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:26:07.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:07.879+0000 7f3c71e00700 1 --2- 172.21.15.62:0/3803506211 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3c5406ecc0 0x7f3c54071170 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f3c60006010 tx=0x7f3c6001a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:26:07.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:07.883+0000 7f3c66a00700 1 -- 172.21.15.62:0/3803506211 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f3c5c057c30 con 0x7f3c6c108470 2024-04-20T22:26:08.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:08.149+0000 7f3c74800700 1 -- 172.21.15.62:0/3803506211 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3c6c199660 con 0x7f3c5406ecc0 2024-04-20T22:26:08.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:08.151+0000 7f3c66a00700 1 -- 172.21.15.62:0/3803506211 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f3c6c199660 con 0x7f3c5406ecc0 2024-04-20T22:26:08.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:08.154+0000 7f3c74800700 1 -- 172.21.15.62:0/3803506211 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3c5406ecc0 msgr2=0x7f3c54071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:26:08.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:08.154+0000 7f3c74800700 1 --2- 172.21.15.62:0/3803506211 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3c5406ecc0 0x7f3c54071170 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f3c60006010 tx=0x7f3c6001a040 comp rx=0 tx=0).stop 2024-04-20T22:26:08.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:08.154+0000 7f3c74800700 1 -- 172.21.15.62:0/3803506211 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3c6c108470 msgr2=0x7f3c6c198730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:26:08.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:08.154+0000 7f3c74800700 1 --2- 172.21.15.62:0/3803506211 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3c6c108470 0x7f3c6c198730 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f3c5c00d8d0 tx=0x7f3c5c00dc90 comp rx=0 tx=0).stop 2024-04-20T22:26:08.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:08.154+0000 7f3c74800700 1 -- 172.21.15.62:0/3803506211 shutdown_connections 2024-04-20T22:26:08.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:08.154+0000 7f3c74800700 1 --2- 172.21.15.62:0/3803506211 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3c5406ecc0 0x7f3c54071170 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:26:08.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:08.154+0000 7f3c74800700 1 --2- 172.21.15.62:0/3803506211 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3c6c102470 0x7f3c6c1981f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:26:08.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:08.154+0000 7f3c74800700 1 --2- 172.21.15.62:0/3803506211 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3c6c108470 0x7f3c6c198730 unknown :-1 s=CLOSED pgs=1296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:26:08.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:08.154+0000 7f3c74800700 1 -- 172.21.15.62:0/3803506211 >> 172.21.15.62:0/3803506211 conn(0x7f3c6c0fdf90 msgr2=0x7f3c6c1002c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:26:08.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:08.155+0000 7f3c74800700 1 -- 172.21.15.62:0/3803506211 shutdown_connections 2024-04-20T22:26:08.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:08.155+0000 7f3c74800700 1 -- 172.21.15.62:0/3803506211 wait complete. 2024-04-20T22:26:08.171 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T22:26:08.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:08.325+0000 7f232b600700 1 -- 172.21.15.62:0/2246791299 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2324106320 msgr2=0x7f23241066f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:26:08.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:08.325+0000 7f232b600700 1 --2- 172.21.15.62:0/2246791299 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2324106320 0x7f23241066f0 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f2318009910 tx=0x7f2318009c20 comp rx=0 tx=0).stop 2024-04-20T22:26:08.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:08.325+0000 7f232b600700 1 -- 172.21.15.62:0/2246791299 shutdown_connections 2024-04-20T22:26:08.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:08.325+0000 7f232b600700 1 --2- 172.21.15.62:0/2246791299 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2324068540 0x7f23240689b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:26:08.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:08.325+0000 7f232b600700 1 --2- 172.21.15.62:0/2246791299 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2324106320 0x7f23241066f0 unknown :-1 s=CLOSED pgs=1297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:26:08.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:08.325+0000 7f232b600700 1 -- 172.21.15.62:0/2246791299 >> 172.21.15.62:0/2246791299 conn(0x7f2324075130 msgr2=0x7f2324075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:26:08.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:08.326+0000 7f232b600700 1 -- 172.21.15.62:0/2246791299 shutdown_connections 2024-04-20T22:26:08.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:08.326+0000 7f232b600700 1 -- 172.21.15.62:0/2246791299 wait complete. 2024-04-20T22:26:08.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:08.327+0000 7f232b600700 1 Processor -- start 2024-04-20T22:26:08.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:08.327+0000 7f232b600700 1 -- start start 2024-04-20T22:26:08.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:08.327+0000 7f232b600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2324068540 0x7f2324198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:26:08.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:08.328+0000 7f232b600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2324106320 0x7f23241985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:26:08.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:08.328+0000 7f232b600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2324198c80 con 0x7f2324106320 2024-04-20T22:26:08.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:08.328+0000 7f232b600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f232419ca10 con 0x7f2324068540 2024-04-20T22:26:08.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:08.328+0000 7f2323e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2324106320 0x7f23241985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:26:08.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:08.328+0000 7f2323e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2324106320 0x7f23241985a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53156/0 (socket says 172.21.15.62:53156) 2024-04-20T22:26:08.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:08.328+0000 7f2323e00700 1 -- 172.21.15.62:0/1742956518 learned_addr learned my addr 172.21.15.62:0/1742956518 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:26:08.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:08.328+0000 7f2328c00700 1 --2- 172.21.15.62:0/1742956518 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2324068540 0x7f2324198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:26:08.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:08.328+0000 7f2323e00700 1 -- 172.21.15.62:0/1742956518 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2324068540 msgr2=0x7f2324198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:26:08.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:08.328+0000 7f2323e00700 1 --2- 172.21.15.62:0/1742956518 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2324068540 0x7f2324198060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:26:08.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:08.329+0000 7f2323e00700 1 -- 172.21.15.62:0/1742956518 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23180095f0 con 0x7f2324106320 2024-04-20T22:26:08.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:08.329+0000 7f2323e00700 1 --2- 172.21.15.62:0/1742956518 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2324106320 0x7f23241985a0 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f231000d8d0 tx=0x7f231000dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:26:08.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:08.329+0000 7f2321600700 1 -- 172.21.15.62:0/1742956518 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f23100098a0 con 0x7f2324106320 2024-04-20T22:26:08.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:08.329+0000 7f2321600700 1 -- 172.21.15.62:0/1742956518 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2310010460 con 0x7f2324106320 2024-04-20T22:26:08.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:08.329+0000 7f2321600700 1 -- 172.21.15.62:0/1742956518 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f231000f5f0 con 0x7f2324106320 2024-04-20T22:26:08.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:08.329+0000 7f232b600700 1 -- 172.21.15.62:0/1742956518 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f232419ccf0 con 0x7f2324106320 2024-04-20T22:26:08.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:08.329+0000 7f232b600700 1 -- 172.21.15.62:0/1742956518 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f232419d240 con 0x7f2324106320 2024-04-20T22:26:08.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:08.331+0000 7f232b600700 1 -- 172.21.15.62:0/1742956518 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f232404f2a0 con 0x7f2324106320 2024-04-20T22:26:08.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:08.332+0000 7f2321600700 1 -- 172.21.15.62:0/1742956518 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f2310009a00 con 0x7f2324106320 2024-04-20T22:26:08.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:08.333+0000 7f2321600700 1 --2- 172.21.15.62:0/1742956518 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f230c06ecc0 0x7f230c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:26:08.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:08.333+0000 7f2321600700 1 -- 172.21.15.62:0/1742956518 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f231008ecd0 con 0x7f2324106320 2024-04-20T22:26:08.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:08.333+0000 7f2328c00700 1 --2- 172.21.15.62:0/1742956518 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f230c06ecc0 0x7f230c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:26:08.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:08.334+0000 7f2328c00700 1 --2- 172.21.15.62:0/1742956518 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f230c06ecc0 0x7f230c071170 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f2318006010 tx=0x7f231801a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:26:08.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:08.337+0000 7f2321600700 1 -- 172.21.15.62:0/1742956518 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f2310057a60 con 0x7f2324106320 2024-04-20T22:26:08.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:08.597+0000 7f232b600700 1 -- 172.21.15.62:0/1742956518 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f232419d5c0 con 0x7f230c06ecc0 2024-04-20T22:26:08.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:08.599+0000 7f2321600700 1 -- 172.21.15.62:0/1742956518 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f232419d5c0 con 0x7f230c06ecc0 2024-04-20T22:26:08.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:08.602+0000 7f232b600700 1 -- 172.21.15.62:0/1742956518 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f230c06ecc0 msgr2=0x7f230c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:26:08.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:08.602+0000 7f232b600700 1 --2- 172.21.15.62:0/1742956518 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f230c06ecc0 0x7f230c071170 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f2318006010 tx=0x7f231801a040 comp rx=0 tx=0).stop 2024-04-20T22:26:08.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:08.602+0000 7f232b600700 1 -- 172.21.15.62:0/1742956518 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2324106320 msgr2=0x7f23241985a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:26:08.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:08.602+0000 7f232b600700 1 --2- 172.21.15.62:0/1742956518 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2324106320 0x7f23241985a0 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f231000d8d0 tx=0x7f231000dbe0 comp rx=0 tx=0).stop 2024-04-20T22:26:08.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:08.603+0000 7f232b600700 1 -- 172.21.15.62:0/1742956518 shutdown_connections 2024-04-20T22:26:08.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:08.603+0000 7f232b600700 1 --2- 172.21.15.62:0/1742956518 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f230c06ecc0 0x7f230c071170 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:26:08.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:08.603+0000 7f232b600700 1 --2- 172.21.15.62:0/1742956518 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2324068540 0x7f2324198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:26:08.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:08.603+0000 7f232b600700 1 --2- 172.21.15.62:0/1742956518 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2324106320 0x7f23241985a0 unknown :-1 s=CLOSED pgs=1298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:26:08.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:08.603+0000 7f232b600700 1 -- 172.21.15.62:0/1742956518 >> 172.21.15.62:0/1742956518 conn(0x7f2324075130 msgr2=0x7f23240fffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:26:08.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:08.603+0000 7f232b600700 1 -- 172.21.15.62:0/1742956518 shutdown_connections 2024-04-20T22:26:08.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:08.603+0000 7f232b600700 1 -- 172.21.15.62:0/1742956518 wait complete. 2024-04-20T22:26:08.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:08.767+0000 7f66c6e00700 1 -- 172.21.15.62:0/2563913180 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f66c01026a0 msgr2=0x7f66c0102a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:26:08.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:08.767+0000 7f66c6e00700 1 --2- 172.21.15.62:0/2563913180 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f66c01026a0 0x7f66c0102a70 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f66b0009910 tx=0x7f66b0009c20 comp rx=0 tx=0).stop 2024-04-20T22:26:08.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:08.767+0000 7f66c6e00700 1 -- 172.21.15.62:0/2563913180 shutdown_connections 2024-04-20T22:26:08.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:08.767+0000 7f66c6e00700 1 --2- 172.21.15.62:0/2563913180 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66c0103040 0x7f66c0107010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:26:08.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:08.767+0000 7f66c6e00700 1 --2- 172.21.15.62:0/2563913180 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f66c01026a0 0x7f66c0102a70 unknown :-1 s=CLOSED pgs=1299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:26:08.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:08.767+0000 7f66c6e00700 1 -- 172.21.15.62:0/2563913180 >> 172.21.15.62:0/2563913180 conn(0x7f66c00fdf40 msgr2=0x7f66c0100350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:26:08.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:08.767+0000 7f66c6e00700 1 -- 172.21.15.62:0/2563913180 shutdown_connections 2024-04-20T22:26:08.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:08.767+0000 7f66c6e00700 1 -- 172.21.15.62:0/2563913180 wait complete. 2024-04-20T22:26:08.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:08.768+0000 7f66c6e00700 1 Processor -- start 2024-04-20T22:26:08.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:08.769+0000 7f66c6e00700 1 -- start start 2024-04-20T22:26:08.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:08.769+0000 7f66c6e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66c01026a0 0x7f66c0198130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:26:08.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:08.769+0000 7f66c6e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f66c0103040 0x7f66c0198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:26:08.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:08.769+0000 7f66c6e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66c0198cc0 con 0x7f66c0103040 2024-04-20T22:26:08.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:08.769+0000 7f66c6e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66c0198e00 con 0x7f66c01026a0 2024-04-20T22:26:08.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:08.769+0000 7f66b7400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f66c0103040 0x7f66c0198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:26:08.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:08.770+0000 7f66b7400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f66c0103040 0x7f66c0198670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53180/0 (socket says 172.21.15.62:53180) 2024-04-20T22:26:08.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:08.770+0000 7f66b7400700 1 -- 172.21.15.62:0/4027643621 learned_addr learned my addr 172.21.15.62:0/4027643621 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:26:08.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:08.770+0000 7f66bfe00700 1 --2- 172.21.15.62:0/4027643621 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66c01026a0 0x7f66c0198130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:26:08.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:08.770+0000 7f66b7400700 1 -- 172.21.15.62:0/4027643621 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66c01026a0 msgr2=0x7f66c0198130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:26:08.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:08.770+0000 7f66b7400700 1 --2- 172.21.15.62:0/4027643621 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66c01026a0 0x7f66c0198130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:26:08.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:08.770+0000 7f66b7400700 1 -- 172.21.15.62:0/4027643621 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66b00095f0 con 0x7f66c0103040 2024-04-20T22:26:08.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:08.770+0000 7f66b7400700 1 --2- 172.21.15.62:0/4027643621 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f66c0103040 0x7f66c0198670 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7f66a800cfd0 tx=0x7f66a800a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:26:08.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:08.771+0000 7f66bd600700 1 -- 172.21.15.62:0/4027643621 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f66a800ad70 con 0x7f66c0103040 2024-04-20T22:26:08.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:08.771+0000 7f66bd600700 1 -- 172.21.15.62:0/4027643621 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f66a80128e0 con 0x7f66c0103040 2024-04-20T22:26:08.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:08.771+0000 7f66c6e00700 1 -- 172.21.15.62:0/4027643621 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66c019cc50 con 0x7f66c0103040 2024-04-20T22:26:08.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:08.771+0000 7f66bd600700 1 -- 172.21.15.62:0/4027643621 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f66a8011880 con 0x7f66c0103040 2024-04-20T22:26:08.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:08.771+0000 7f66c6e00700 1 -- 172.21.15.62:0/4027643621 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66c019d1a0 con 0x7f66c0103040 2024-04-20T22:26:08.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:08.772+0000 7f66c6e00700 1 -- 172.21.15.62:0/4027643621 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66c010a9b0 con 0x7f66c0103040 2024-04-20T22:26:08.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:08.773+0000 7f66bd600700 1 -- 172.21.15.62:0/4027643621 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f66a8012420 con 0x7f66c0103040 2024-04-20T22:26:08.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:08.774+0000 7f66bd600700 1 --2- 172.21.15.62:0/4027643621 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f66a406ecc0 0x7f66a4071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:26:08.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:08.774+0000 7f66bd600700 1 -- 172.21.15.62:0/4027643621 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f66a808ef00 con 0x7f66c0103040 2024-04-20T22:26:08.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:08.774+0000 7f66bfe00700 1 --2- 172.21.15.62:0/4027643621 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f66a406ecc0 0x7f66a4071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:26:08.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:08.775+0000 7f66bfe00700 1 --2- 172.21.15.62:0/4027643621 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f66a406ecc0 0x7f66a4071170 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f66b0006010 tx=0x7f66b001a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:26:08.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:08.779+0000 7f66bd600700 1 -- 172.21.15.62:0/4027643621 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f66a8057c90 con 0x7f66c0103040 2024-04-20T22:26:09.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:09.029+0000 7f66c6e00700 1 -- 172.21.15.62:0/4027643621 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f66c019d470 con 0x7f66a406ecc0 2024-04-20T22:26:09.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:09.070+0000 7f66bd600700 1 -- 172.21.15.62:0/4027643621 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f66c019d470 con 0x7f66a406ecc0 2024-04-20T22:26:09.070 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T22:26:09.070 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (51m) 12m ago 65m 14.6M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T22:26:09.070 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (66m) 12m ago 66m 11.6M - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T22:26:09.071 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (64m) 12m ago 64m 11.2M - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T22:26:09.071 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (48m) 12m ago 66m 6660k - 19.0.0-3133-ga8c84f59 76bc6b91b726 2825d5ba11c6 2024-04-20T22:26:09.071 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (47m) 12m ago 64m 6665k - 19.0.0-3133-ga8c84f59 76bc6b91b726 20f746fa1bba 2024-04-20T22:26:09.071 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (51m) 12m ago 65m 75.5M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T22:26:09.071 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (43m) 12m ago 58m 22.9M - 19.0.0-3133-ga8c84f59 76bc6b91b726 de80bc90a683 2024-04-20T22:26:09.071 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (43m) 12m ago 58m 23.0M - 19.0.0-3133-ga8c84f59 76bc6b91b726 cd793b7ee98e 2024-04-20T22:26:09.071 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (28m) 12m ago 58m 18.5M - 19.0.0-3133-ga8c84f59 76bc6b91b726 0e7b0bc4d1ab 2024-04-20T22:26:09.071 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (12m) 12m ago 58m 11.4M - 19.0.0-3133-ga8c84f59 76bc6b91b726 0af732ac9e06 2024-04-20T22:26:09.071 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (54m) 12m ago 67m 517M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T22:26:09.071 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (53m) 12m ago 64m 460M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T22:26:09.071 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (49m) 12m ago 67m 86.3M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 019eb1479485 2024-04-20T22:26:09.072 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (49m) 12m ago 64m 58.0M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 55aac560aab6 2024-04-20T22:26:09.072 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (53m) 12m ago 66m 8766k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T22:26:09.072 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (53m) 12m ago 64m 8711k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T22:26:09.072 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (47m) 12m ago 63m 91.3M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 ee2bb901f26e 2024-04-20T22:26:09.072 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (46m) 12m ago 62m 94.9M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 9075152d4162 2024-04-20T22:26:09.072 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (46m) 12m ago 62m 78.5M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 2d4f3e38dc0e 2024-04-20T22:26:09.072 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (45m) 12m ago 61m 94.1M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 8f97b17ae0eb 2024-04-20T22:26:09.072 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (45m) 12m ago 61m 96.1M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 653d8079967d 2024-04-20T22:26:09.072 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (44m) 12m ago 60m 90.9M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 596dded14c12 2024-04-20T22:26:09.072 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (52m) 12m ago 65m 90.4M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T22:26:09.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:09.073+0000 7f66c6e00700 1 -- 172.21.15.62:0/4027643621 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f66a406ecc0 msgr2=0x7f66a4071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:26:09.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:09.073+0000 7f66c6e00700 1 --2- 172.21.15.62:0/4027643621 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f66a406ecc0 0x7f66a4071170 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f66b0006010 tx=0x7f66b001a040 comp rx=0 tx=0).stop 2024-04-20T22:26:09.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:09.073+0000 7f66c6e00700 1 -- 172.21.15.62:0/4027643621 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f66c0103040 msgr2=0x7f66c0198670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:26:09.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:09.073+0000 7f66c6e00700 1 --2- 172.21.15.62:0/4027643621 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f66c0103040 0x7f66c0198670 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7f66a800cfd0 tx=0x7f66a800a340 comp rx=0 tx=0).stop 2024-04-20T22:26:09.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:09.073+0000 7f66c6e00700 1 -- 172.21.15.62:0/4027643621 shutdown_connections 2024-04-20T22:26:09.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:09.073+0000 7f66c6e00700 1 --2- 172.21.15.62:0/4027643621 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f66a406ecc0 0x7f66a4071170 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:26:09.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:09.073+0000 7f66c6e00700 1 --2- 172.21.15.62:0/4027643621 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66c01026a0 0x7f66c0198130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:26:09.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:09.074+0000 7f66c6e00700 1 --2- 172.21.15.62:0/4027643621 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f66c0103040 0x7f66c0198670 unknown :-1 s=CLOSED pgs=1300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:26:09.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:09.074+0000 7f66c6e00700 1 -- 172.21.15.62:0/4027643621 >> 172.21.15.62:0/4027643621 conn(0x7f66c00fdf40 msgr2=0x7f66c00ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:26:09.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:09.074+0000 7f66c6e00700 1 -- 172.21.15.62:0/4027643621 shutdown_connections 2024-04-20T22:26:09.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:09.074+0000 7f66c6e00700 1 -- 172.21.15.62:0/4027643621 wait complete. 2024-04-20T22:26:09.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:09.240+0000 7fb820000700 1 -- 172.21.15.62:0/2696213887 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb818108470 msgr2=0x7fb818108840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:26:09.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:09.240+0000 7fb820000700 1 --2- 172.21.15.62:0/2696213887 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb818108470 0x7fb818108840 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7fb80c009910 tx=0x7fb80c009c20 comp rx=0 tx=0).stop 2024-04-20T22:26:09.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:09.240+0000 7fb820000700 1 -- 172.21.15.62:0/2696213887 shutdown_connections 2024-04-20T22:26:09.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:09.240+0000 7fb820000700 1 --2- 172.21.15.62:0/2696213887 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb818102470 0x7fb8181028e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:26:09.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:09.240+0000 7fb820000700 1 --2- 172.21.15.62:0/2696213887 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb818108470 0x7fb818108840 unknown :-1 s=CLOSED pgs=1301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:26:09.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:09.240+0000 7fb820000700 1 -- 172.21.15.62:0/2696213887 >> 172.21.15.62:0/2696213887 conn(0x7fb8180fdf90 msgr2=0x7fb8181003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:26:09.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:09.240+0000 7fb820000700 1 -- 172.21.15.62:0/2696213887 shutdown_connections 2024-04-20T22:26:09.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:09.240+0000 7fb820000700 1 -- 172.21.15.62:0/2696213887 wait complete. 2024-04-20T22:26:09.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:09.241+0000 7fb820000700 1 Processor -- start 2024-04-20T22:26:09.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:09.242+0000 7fb820000700 1 -- start start 2024-04-20T22:26:09.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:09.242+0000 7fb820000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb818102470 0x7fb818198210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:26:09.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:09.242+0000 7fb820000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb818108470 0x7fb818198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:26:09.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:09.242+0000 7fb820000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb818198e30 con 0x7fb818108470 2024-04-20T22:26:09.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:09.242+0000 7fb820000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb81819cbc0 con 0x7fb818102470 2024-04-20T22:26:09.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:09.243+0000 7fb81cc00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb818108470 0x7fb818198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:26:09.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:09.243+0000 7fb81cc00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb818108470 0x7fb818198750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53194/0 (socket says 172.21.15.62:53194) 2024-04-20T22:26:09.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:09.243+0000 7fb81cc00700 1 -- 172.21.15.62:0/1082590863 learned_addr learned my addr 172.21.15.62:0/1082590863 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:26:09.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:09.243+0000 7fb81d600700 1 --2- 172.21.15.62:0/1082590863 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb818102470 0x7fb818198210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:26:09.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:09.243+0000 7fb81cc00700 1 -- 172.21.15.62:0/1082590863 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb818102470 msgr2=0x7fb818198210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:26:09.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:09.243+0000 7fb81cc00700 1 --2- 172.21.15.62:0/1082590863 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb818102470 0x7fb818198210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:26:09.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:09.243+0000 7fb81cc00700 1 -- 172.21.15.62:0/1082590863 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb80c0095f0 con 0x7fb818108470 2024-04-20T22:26:09.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:09.244+0000 7fb81cc00700 1 --2- 172.21.15.62:0/1082590863 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb818108470 0x7fb818198750 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7fb80400b4b0 tx=0x7fb80400b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:26:09.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:09.244+0000 7fb816000700 1 -- 172.21.15.62:0/1082590863 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb804011800 con 0x7fb818108470 2024-04-20T22:26:09.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:09.244+0000 7fb816000700 1 -- 172.21.15.62:0/1082590863 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb804011e20 con 0x7fb818108470 2024-04-20T22:26:09.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:09.244+0000 7fb820000700 1 -- 172.21.15.62:0/1082590863 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb81819cea0 con 0x7fb818108470 2024-04-20T22:26:09.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:09.244+0000 7fb816000700 1 -- 172.21.15.62:0/1082590863 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb80400f2e0 con 0x7fb818108470 2024-04-20T22:26:09.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:09.244+0000 7fb820000700 1 -- 172.21.15.62:0/1082590863 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb81819d3f0 con 0x7fb818108470 2024-04-20T22:26:09.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:09.245+0000 7fb820000700 1 -- 172.21.15.62:0/1082590863 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb81810a9e0 con 0x7fb818108470 2024-04-20T22:26:09.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:09.246+0000 7fb816000700 1 -- 172.21.15.62:0/1082590863 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fb804010380 con 0x7fb818108470 2024-04-20T22:26:09.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:09.247+0000 7fb816000700 1 --2- 172.21.15.62:0/1082590863 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb80006ecc0 0x7fb800071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:26:09.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:09.248+0000 7fb816000700 1 -- 172.21.15.62:0/1082590863 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fb80408ea30 con 0x7fb818108470 2024-04-20T22:26:09.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:09.248+0000 7fb81d600700 1 --2- 172.21.15.62:0/1082590863 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb80006ecc0 0x7fb800071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:26:09.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:09.248+0000 7fb81d600700 1 --2- 172.21.15.62:0/1082590863 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb80006ecc0 0x7fb800071170 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fb80c006010 tx=0x7fb80c01a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:26:09.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:09.252+0000 7fb816000700 1 -- 172.21.15.62:0/1082590863 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fb804057870 con 0x7fb818108470 2024-04-20T22:26:09.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:09.587+0000 7fb820000700 1 -- 172.21.15.62:0/1082590863 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fb818199610 con 0x7fb818108470 2024-04-20T22:26:09.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:09.588+0000 7fb816000700 1 -- 172.21.15.62:0/1082590863 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7fb80405af30 con 0x7fb818108470 2024-04-20T22:26:09.589 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:26:09.589 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T22:26:09.589 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:26:09.589 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:26:09.589 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T22:26:09.589 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:26:09.589 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:26:09.589 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T22:26:09.589 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 6 2024-04-20T22:26:09.589 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:26:09.589 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T22:26:09.589 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 4 2024-04-20T22:26:09.589 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:26:09.589 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T22:26:09.590 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 14 2024-04-20T22:26:09.590 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T22:26:09.590 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:26:09.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:09.591+0000 7fb820000700 1 -- 172.21.15.62:0/1082590863 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb80006ecc0 msgr2=0x7fb800071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:26:09.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:09.591+0000 7fb820000700 1 --2- 172.21.15.62:0/1082590863 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb80006ecc0 0x7fb800071170 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fb80c006010 tx=0x7fb80c01a040 comp rx=0 tx=0).stop 2024-04-20T22:26:09.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:09.591+0000 7fb820000700 1 -- 172.21.15.62:0/1082590863 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb818108470 msgr2=0x7fb818198750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:26:09.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:09.591+0000 7fb820000700 1 --2- 172.21.15.62:0/1082590863 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb818108470 0x7fb818198750 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7fb80400b4b0 tx=0x7fb80400b870 comp rx=0 tx=0).stop 2024-04-20T22:26:09.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:09.592+0000 7fb820000700 1 -- 172.21.15.62:0/1082590863 shutdown_connections 2024-04-20T22:26:09.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:09.592+0000 7fb820000700 1 --2- 172.21.15.62:0/1082590863 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb80006ecc0 0x7fb800071170 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:26:09.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:09.592+0000 7fb820000700 1 --2- 172.21.15.62:0/1082590863 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb818102470 0x7fb818198210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:26:09.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:09.592+0000 7fb820000700 1 --2- 172.21.15.62:0/1082590863 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb818108470 0x7fb818198750 unknown :-1 s=CLOSED pgs=1302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:26:09.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:09.592+0000 7fb820000700 1 -- 172.21.15.62:0/1082590863 >> 172.21.15.62:0/1082590863 conn(0x7fb8180fdf90 msgr2=0x7fb8180febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:26:09.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:09.592+0000 7fb820000700 1 -- 172.21.15.62:0/1082590863 shutdown_connections 2024-04-20T22:26:09.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:09.592+0000 7fb820000700 1 -- 172.21.15.62:0/1082590863 wait complete. 2024-04-20T22:26:09.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:26:09 smithi189 ceph-mon[57022]: pgmap v1528: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:26:09.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:26:09 smithi189 ceph-mon[57022]: from='client.36664 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:26:09.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:09.754+0000 7f762a800700 1 -- 172.21.15.62:0/3599002022 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7624101110 msgr2=0x7f76241014e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:26:09.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:09.754+0000 7f762a800700 1 --2- 172.21.15.62:0/3599002022 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7624101110 0x7f76241014e0 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f7618009910 tx=0x7f7618009c20 comp rx=0 tx=0).stop 2024-04-20T22:26:09.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:09.754+0000 7f762a800700 1 -- 172.21.15.62:0/3599002022 shutdown_connections 2024-04-20T22:26:09.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:09.754+0000 7f762a800700 1 --2- 172.21.15.62:0/3599002022 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7624101a20 0x7f762410a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:26:09.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:09.754+0000 7f762a800700 1 --2- 172.21.15.62:0/3599002022 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7624101110 0x7f76241014e0 unknown :-1 s=CLOSED pgs=1303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:26:09.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:09.754+0000 7f762a800700 1 -- 172.21.15.62:0/3599002022 >> 172.21.15.62:0/3599002022 conn(0x7f76240faa60 msgr2=0x7f76240fce70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:26:09.755 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:26:09 smithi062 ceph-mon[76322]: pgmap v1528: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:26:09.755 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:26:09 smithi062 ceph-mon[76322]: from='client.36664 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:26:09.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:09.754+0000 7f762a800700 1 -- 172.21.15.62:0/3599002022 shutdown_connections 2024-04-20T22:26:09.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:09.754+0000 7f762a800700 1 -- 172.21.15.62:0/3599002022 wait complete. 2024-04-20T22:26:09.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:09.755+0000 7f762a800700 1 Processor -- start 2024-04-20T22:26:09.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:09.755+0000 7f762a800700 1 -- start start 2024-04-20T22:26:09.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:09.756+0000 7f762a800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7624101110 0x7f7624197e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:26:09.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:09.756+0000 7f762a800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7624101a20 0x7f76241983a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:26:09.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:09.756+0000 7f762a800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7624198a80 con 0x7f7624101a20 2024-04-20T22:26:09.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:09.756+0000 7f762a800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f762419c810 con 0x7f7624101110 2024-04-20T22:26:09.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:09.756+0000 7f7628a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7624101a20 0x7f76241983a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:26:09.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:09.756+0000 7f7628a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7624101a20 0x7f76241983a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53202/0 (socket says 172.21.15.62:53202) 2024-04-20T22:26:09.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:09.756+0000 7f7628a00700 1 -- 172.21.15.62:0/1794032650 learned_addr learned my addr 172.21.15.62:0/1794032650 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:26:09.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:09.757+0000 7f7628a00700 1 -- 172.21.15.62:0/1794032650 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7624101110 msgr2=0x7f7624197e60 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-20T22:26:09.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:09.757+0000 7f7628a00700 1 --2- 172.21.15.62:0/1794032650 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7624101110 0x7f7624197e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:26:09.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:09.757+0000 7f7628a00700 1 -- 172.21.15.62:0/1794032650 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76180095f0 con 0x7f7624101a20 2024-04-20T22:26:09.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:09.757+0000 7f7628a00700 1 --2- 172.21.15.62:0/1794032650 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7624101a20 0x7f76241983a0 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7f761000d8d0 tx=0x7f761000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:26:09.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:09.757+0000 7f7622000700 1 -- 172.21.15.62:0/1794032650 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7610009940 con 0x7f7624101a20 2024-04-20T22:26:09.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:09.758+0000 7f762a800700 1 -- 172.21.15.62:0/1794032650 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f762419caf0 con 0x7f7624101a20 2024-04-20T22:26:09.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:09.758+0000 7f7622000700 1 -- 172.21.15.62:0/1794032650 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7610010460 con 0x7f7624101a20 2024-04-20T22:26:09.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:09.758+0000 7f762a800700 1 -- 172.21.15.62:0/1794032650 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f762419d040 con 0x7f7624101a20 2024-04-20T22:26:09.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:09.758+0000 7f7622000700 1 -- 172.21.15.62:0/1794032650 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f761001b620 con 0x7f7624101a20 2024-04-20T22:26:09.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:09.759+0000 7f762a800700 1 -- 172.21.15.62:0/1794032650 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7624109bf0 con 0x7f7624101a20 2024-04-20T22:26:09.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:09.760+0000 7f7622000700 1 -- 172.21.15.62:0/1794032650 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f76100105d0 con 0x7f7624101a20 2024-04-20T22:26:09.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:09.761+0000 7f7622000700 1 --2- 172.21.15.62:0/1794032650 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f760c06ecc0 0x7f760c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:26:09.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:09.761+0000 7f7622000700 1 -- 172.21.15.62:0/1794032650 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f761008eed0 con 0x7f7624101a20 2024-04-20T22:26:09.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:09.761+0000 7f7629400700 1 --2- 172.21.15.62:0/1794032650 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f760c06ecc0 0x7f760c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:26:09.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:09.762+0000 7f7629400700 1 --2- 172.21.15.62:0/1794032650 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f760c06ecc0 0x7f760c071170 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f7618000c00 tx=0x7f7618005fb0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:26:09.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:09.766+0000 7f7622000700 1 -- 172.21.15.62:0/1794032650 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f7610057be0 con 0x7f7624101a20 2024-04-20T22:26:10.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:10.051+0000 7f762a800700 1 -- 172.21.15.62:0/1794032650 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f7624062380 con 0x7f7624101a20 2024-04-20T22:26:10.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:10.052+0000 7f7622000700 1 -- 172.21.15.62:0/1794032650 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 29 v29) v1 ==== 76+0+1887 (secure 0 0 0) 0x7f761005b2a0 con 0x7f7624101a20 2024-04-20T22:26:10.053 INFO:teuthology.orchestra.run.smithi062.stdout:e29 2024-04-20T22:26:10.053 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T22:26:10.053 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:26:10.053 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T22:26:10.053 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:26:10.053 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T22:26:10.054 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T22:26:10.054 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 20 2024-04-20T22:26:10.054 INFO:teuthology.orchestra.run.smithi062.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-20T22:26:10.054 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T22:26:10.054 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T21:42:01.559061+0000 2024-04-20T22:26:10.054 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T22:26:10.054 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T22:26:10.054 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T22:26:10.054 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T22:26:10.054 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T22:26:10.054 INFO:teuthology.orchestra.run.smithi062.stdout:max_xattr_size 65536 2024-04-20T22:26:10.054 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T22:26:10.055 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T22:26:10.055 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 81 2024-04-20T22:26:10.055 INFO:teuthology.orchestra.run.smithi062.stdout:snap_rstat 1 2024-04-20T22:26:10.055 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:26:10.055 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 2 2024-04-20T22:26:10.055 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1 2024-04-20T22:26:10.055 INFO:teuthology.orchestra.run.smithi062.stdout:up {} 2024-04-20T22:26:10.055 INFO:teuthology.orchestra.run.smithi062.stdout:failed 0,1 2024-04-20T22:26:10.055 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T22:26:10.055 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T22:26:10.055 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T22:26:10.056 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T22:26:10.056 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data enabled 2024-04-20T22:26:10.056 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T22:26:10.056 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T22:26:10.056 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 1 2024-04-20T22:26:10.056 INFO:teuthology.orchestra.run.smithi062.stdout:qdb_cluster leader: 0 members: 2024-04-20T22:26:10.056 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:26:10.056 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:26:10.056 INFO:teuthology.orchestra.run.smithi062.stdout:Standby daemons: 2024-04-20T22:26:10.056 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:26:10.056 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ryqblb{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6826/4113916205,v1:172.21.15.62:6827/4113916205] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:26:10.056 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ozizwi{-1:34476} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6828/3801975165,v1:172.21.15.62:6829/3801975165] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:26:10.056 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.dvimer{-1:44727} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6824/3576699232,v1:172.21.15.189:6825/3576699232] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:26:10.056 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.xbgspw{-1:45075} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6826/212905792,v1:172.21.15.189:6827/212905792] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:26:10.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:10.055+0000 7f762a800700 1 -- 172.21.15.62:0/1794032650 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f760c06ecc0 msgr2=0x7f760c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:26:10.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:10.055+0000 7f762a800700 1 --2- 172.21.15.62:0/1794032650 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f760c06ecc0 0x7f760c071170 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f7618000c00 tx=0x7f7618005fb0 comp rx=0 tx=0).stop 2024-04-20T22:26:10.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:10.055+0000 7f762a800700 1 -- 172.21.15.62:0/1794032650 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7624101a20 msgr2=0x7f76241983a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:26:10.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:10.055+0000 7f762a800700 1 --2- 172.21.15.62:0/1794032650 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7624101a20 0x7f76241983a0 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7f761000d8d0 tx=0x7f761000dc90 comp rx=0 tx=0).stop 2024-04-20T22:26:10.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:10.055+0000 7f762a800700 1 -- 172.21.15.62:0/1794032650 shutdown_connections 2024-04-20T22:26:10.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:10.055+0000 7f762a800700 1 --2- 172.21.15.62:0/1794032650 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f760c06ecc0 0x7f760c071170 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:26:10.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:10.055+0000 7f762a800700 1 --2- 172.21.15.62:0/1794032650 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7624101110 0x7f7624197e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:26:10.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:10.055+0000 7f762a800700 1 --2- 172.21.15.62:0/1794032650 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7624101a20 0x7f76241983a0 unknown :-1 s=CLOSED pgs=1304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:26:10.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:10.055+0000 7f762a800700 1 -- 172.21.15.62:0/1794032650 >> 172.21.15.62:0/1794032650 conn(0x7f76240faa60 msgr2=0x7f7624106d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:26:10.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:10.055+0000 7f762a800700 1 -- 172.21.15.62:0/1794032650 shutdown_connections 2024-04-20T22:26:10.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:10.055+0000 7f762a800700 1 -- 172.21.15.62:0/1794032650 wait complete. 2024-04-20T22:26:10.058 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 29 2024-04-20T22:26:10.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:10.217+0000 7fdfbc000700 1 -- 172.21.15.62:0/1768027060 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdfb41004e0 msgr2=0x7fdfb41008b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:26:10.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:10.217+0000 7fdfbc000700 1 --2- 172.21.15.62:0/1768027060 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdfb41004e0 0x7fdfb41008b0 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7fdfa8009910 tx=0x7fdfa8009c20 comp rx=0 tx=0).stop 2024-04-20T22:26:10.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:10.217+0000 7fdfbc000700 1 -- 172.21.15.62:0/1768027060 shutdown_connections 2024-04-20T22:26:10.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:10.217+0000 7fdfbc000700 1 --2- 172.21.15.62:0/1768027060 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdfb4100e80 0x7fdfb4104e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:26:10.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:10.217+0000 7fdfbc000700 1 --2- 172.21.15.62:0/1768027060 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdfb41004e0 0x7fdfb41008b0 unknown :-1 s=CLOSED pgs=1305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:26:10.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:10.217+0000 7fdfbc000700 1 -- 172.21.15.62:0/1768027060 >> 172.21.15.62:0/1768027060 conn(0x7fdfb40fbda0 msgr2=0x7fdfb40fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:26:10.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:10.217+0000 7fdfbc000700 1 -- 172.21.15.62:0/1768027060 shutdown_connections 2024-04-20T22:26:10.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:10.217+0000 7fdfbc000700 1 -- 172.21.15.62:0/1768027060 wait complete. 2024-04-20T22:26:10.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:10.218+0000 7fdfbc000700 1 Processor -- start 2024-04-20T22:26:10.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:10.219+0000 7fdfbc000700 1 -- start start 2024-04-20T22:26:10.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:10.219+0000 7fdfbc000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdfb41004e0 0x7fdfb41980d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:26:10.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:10.219+0000 7fdfbc000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdfb4100e80 0x7fdfb4198610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:26:10.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:10.219+0000 7fdfbc000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfb4198cf0 con 0x7fdfb4100e80 2024-04-20T22:26:10.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:10.219+0000 7fdfbc000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfb419ca80 con 0x7fdfb41004e0 2024-04-20T22:26:10.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:10.219+0000 7fdfb8c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdfb4100e80 0x7fdfb4198610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:26:10.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:10.219+0000 7fdfb8c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdfb4100e80 0x7fdfb4198610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53220/0 (socket says 172.21.15.62:53220) 2024-04-20T22:26:10.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:10.219+0000 7fdfb8c00700 1 -- 172.21.15.62:0/3837994407 learned_addr learned my addr 172.21.15.62:0/3837994407 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:26:10.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:10.220+0000 7fdfb9600700 1 --2- 172.21.15.62:0/3837994407 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdfb41004e0 0x7fdfb41980d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:26:10.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:10.220+0000 7fdfb8c00700 1 -- 172.21.15.62:0/3837994407 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdfb41004e0 msgr2=0x7fdfb41980d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:26:10.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:10.220+0000 7fdfb8c00700 1 --2- 172.21.15.62:0/3837994407 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdfb41004e0 0x7fdfb41980d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:26:10.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:10.220+0000 7fdfb8c00700 1 -- 172.21.15.62:0/3837994407 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfa80095f0 con 0x7fdfb4100e80 2024-04-20T22:26:10.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:10.220+0000 7fdfb8c00700 1 --2- 172.21.15.62:0/3837994407 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdfb4100e80 0x7fdfb4198610 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7fdfa000c8f0 tx=0x7fdfa000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:26:10.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:10.220+0000 7fdfb2000700 1 -- 172.21.15.62:0/3837994407 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdfa0007a50 con 0x7fdfb4100e80 2024-04-20T22:26:10.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:10.220+0000 7fdfb2000700 1 -- 172.21.15.62:0/3837994407 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdfa0007bb0 con 0x7fdfb4100e80 2024-04-20T22:26:10.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:10.220+0000 7fdfb2000700 1 -- 172.21.15.62:0/3837994407 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdfa0018630 con 0x7fdfb4100e80 2024-04-20T22:26:10.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:10.220+0000 7fdfbc000700 1 -- 172.21.15.62:0/3837994407 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfb419cd60 con 0x7fdfb4100e80 2024-04-20T22:26:10.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:10.221+0000 7fdfbc000700 1 -- 172.21.15.62:0/3837994407 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfb419d2b0 con 0x7fdfb4100e80 2024-04-20T22:26:10.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:10.222+0000 7fdfbc000700 1 -- 172.21.15.62:0/3837994407 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfb41045d0 con 0x7fdfb4100e80 2024-04-20T22:26:10.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:10.223+0000 7fdfb2000700 1 -- 172.21.15.62:0/3837994407 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fdfa0007d20 con 0x7fdfb4100e80 2024-04-20T22:26:10.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:10.223+0000 7fdfb2000700 1 --2- 172.21.15.62:0/3837994407 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fdf9c06ecc0 0x7fdf9c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:26:10.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:10.224+0000 7fdfb2000700 1 -- 172.21.15.62:0/3837994407 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fdfa008ee00 con 0x7fdfb4100e80 2024-04-20T22:26:10.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:10.224+0000 7fdfb9600700 1 --2- 172.21.15.62:0/3837994407 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fdf9c06ecc0 0x7fdf9c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:26:10.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:10.225+0000 7fdfb9600700 1 --2- 172.21.15.62:0/3837994407 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fdf9c06ecc0 0x7fdf9c071170 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fdfa8000c00 tx=0x7fdfa8005fd0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:26:10.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:10.229+0000 7fdfb2000700 1 -- 172.21.15.62:0/3837994407 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fdfa0057b90 con 0x7fdfb4100e80 2024-04-20T22:26:10.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:10.487+0000 7fdfbc000700 1 -- 172.21.15.62:0/3837994407 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fdfb419d580 con 0x7fdf9c06ecc0 2024-04-20T22:26:10.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:10.491+0000 7fdfb2000700 1 -- 172.21.15.62:0/3837994407 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fdfb419d580 con 0x7fdf9c06ecc0 2024-04-20T22:26:10.491 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:26:10.491 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T22:26:10.491 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T22:26:10.491 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-20T22:26:10.491 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T22:26:10.491 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr", 2024-04-20T22:26:10.491 INFO:teuthology.orchestra.run.smithi062.stdout: "osd", 2024-04-20T22:26:10.491 INFO:teuthology.orchestra.run.smithi062.stdout: "mds", 2024-04-20T22:26:10.492 INFO:teuthology.orchestra.run.smithi062.stdout: "crash", 2024-04-20T22:26:10.492 INFO:teuthology.orchestra.run.smithi062.stdout: "mon" 2024-04-20T22:26:10.492 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T22:26:10.492 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "16/23 daemons upgraded", 2024-04-20T22:26:10.492 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading mds daemons", 2024-04-20T22:26:10.492 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T22:26:10.492 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:26:10.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:10.493+0000 7fdfbc000700 1 -- 172.21.15.62:0/3837994407 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fdf9c06ecc0 msgr2=0x7fdf9c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:26:10.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:10.493+0000 7fdfbc000700 1 --2- 172.21.15.62:0/3837994407 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fdf9c06ecc0 0x7fdf9c071170 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fdfa8000c00 tx=0x7fdfa8005fd0 comp rx=0 tx=0).stop 2024-04-20T22:26:10.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:10.493+0000 7fdfbc000700 1 -- 172.21.15.62:0/3837994407 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdfb4100e80 msgr2=0x7fdfb4198610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:26:10.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:10.493+0000 7fdfbc000700 1 --2- 172.21.15.62:0/3837994407 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdfb4100e80 0x7fdfb4198610 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7fdfa000c8f0 tx=0x7fdfa000ccb0 comp rx=0 tx=0).stop 2024-04-20T22:26:10.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:10.494+0000 7fdfbc000700 1 -- 172.21.15.62:0/3837994407 shutdown_connections 2024-04-20T22:26:10.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:10.494+0000 7fdfbc000700 1 --2- 172.21.15.62:0/3837994407 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fdf9c06ecc0 0x7fdf9c071170 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:26:10.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:10.494+0000 7fdfbc000700 1 --2- 172.21.15.62:0/3837994407 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdfb41004e0 0x7fdfb41980d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:26:10.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:10.494+0000 7fdfbc000700 1 --2- 172.21.15.62:0/3837994407 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdfb4100e80 0x7fdfb4198610 unknown :-1 s=CLOSED pgs=1306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:26:10.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:10.494+0000 7fdfbc000700 1 -- 172.21.15.62:0/3837994407 >> 172.21.15.62:0/3837994407 conn(0x7fdfb40fbda0 msgr2=0x7fdfb40fe150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:26:10.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:10.494+0000 7fdfbc000700 1 -- 172.21.15.62:0/3837994407 shutdown_connections 2024-04-20T22:26:10.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:10.494+0000 7fdfbc000700 1 -- 172.21.15.62:0/3837994407 wait complete. 2024-04-20T22:26:10.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:10.661+0000 7f390d200700 1 -- 172.21.15.62:0/4024176735 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3908073090 msgr2=0x7f3908073460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:26:10.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:10.661+0000 7f390d200700 1 --2- 172.21.15.62:0/4024176735 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3908073090 0x7f3908073460 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7f38f8009940 tx=0x7f38f8009c50 comp rx=0 tx=0).stop 2024-04-20T22:26:10.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:10.661+0000 7f390d200700 1 -- 172.21.15.62:0/4024176735 shutdown_connections 2024-04-20T22:26:10.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:10.661+0000 7f390d200700 1 --2- 172.21.15.62:0/4024176735 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39080739a0 0x7f390810c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:26:10.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:10.661+0000 7f390d200700 1 --2- 172.21.15.62:0/4024176735 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3908073090 0x7f3908073460 unknown :-1 s=CLOSED pgs=1307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:26:10.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:10.661+0000 7f390d200700 1 -- 172.21.15.62:0/4024176735 >> 172.21.15.62:0/4024176735 conn(0x7f39080fbda0 msgr2=0x7f39080fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:26:10.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:10.661+0000 7f390d200700 1 -- 172.21.15.62:0/4024176735 shutdown_connections 2024-04-20T22:26:10.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:10.661+0000 7f390d200700 1 -- 172.21.15.62:0/4024176735 wait complete. 2024-04-20T22:26:10.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:10.662+0000 7f390d200700 1 Processor -- start 2024-04-20T22:26:10.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:10.663+0000 7f390d200700 1 -- start start 2024-04-20T22:26:10.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:10.663+0000 7f390d200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3908073090 0x7f39081981c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:26:10.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:10.663+0000 7f390d200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39080739a0 0x7f3908198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:26:10.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:10.663+0000 7f390d200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3908198de0 con 0x7f39080739a0 2024-04-20T22:26:10.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:10.663+0000 7f390d200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f390819cb70 con 0x7f3908073090 2024-04-20T22:26:10.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:10.663+0000 7f3905400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39080739a0 0x7f3908198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:26:10.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:10.664+0000 7f3905400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39080739a0 0x7f3908198700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53238/0 (socket says 172.21.15.62:53238) 2024-04-20T22:26:10.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:10.664+0000 7f3905400700 1 -- 172.21.15.62:0/3000057328 learned_addr learned my addr 172.21.15.62:0/3000057328 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:26:10.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:10.664+0000 7f3905e00700 1 --2- 172.21.15.62:0/3000057328 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3908073090 0x7f39081981c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:26:10.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:10.664+0000 7f3905400700 1 -- 172.21.15.62:0/3000057328 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3908073090 msgr2=0x7f39081981c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:26:10.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:10.664+0000 7f3905400700 1 --2- 172.21.15.62:0/3000057328 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3908073090 0x7f39081981c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:26:10.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:10.664+0000 7f3905400700 1 -- 172.21.15.62:0/3000057328 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38f80095f0 con 0x7f39080739a0 2024-04-20T22:26:10.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:10.665+0000 7f3905400700 1 --2- 172.21.15.62:0/3000057328 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39080739a0 0x7f3908198700 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7f38f400b4b0 tx=0x7f38f400b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:26:10.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:10.665+0000 7f38fea00700 1 -- 172.21.15.62:0/3000057328 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f38f4011800 con 0x7f39080739a0 2024-04-20T22:26:10.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:10.665+0000 7f38fea00700 1 -- 172.21.15.62:0/3000057328 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f38f4011e20 con 0x7f39080739a0 2024-04-20T22:26:10.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:10.665+0000 7f390d200700 1 -- 172.21.15.62:0/3000057328 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f390819ce50 con 0x7f39080739a0 2024-04-20T22:26:10.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:10.665+0000 7f38fea00700 1 -- 172.21.15.62:0/3000057328 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f38f400f2e0 con 0x7f39080739a0 2024-04-20T22:26:10.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:10.665+0000 7f390d200700 1 -- 172.21.15.62:0/3000057328 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f390819d3a0 con 0x7f39080739a0 2024-04-20T22:26:10.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:10.666+0000 7f390d200700 1 -- 172.21.15.62:0/3000057328 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3908109ce0 con 0x7f39080739a0 2024-04-20T22:26:10.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:10.668+0000 7f38fea00700 1 -- 172.21.15.62:0/3000057328 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f38f4010380 con 0x7f39080739a0 2024-04-20T22:26:10.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:10.670+0000 7f38fea00700 1 --2- 172.21.15.62:0/3000057328 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f38ec06ecc0 0x7f38ec071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:26:10.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:10.670+0000 7f38fea00700 1 -- 172.21.15.62:0/3000057328 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f38f408ea30 con 0x7f39080739a0 2024-04-20T22:26:10.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:10.670+0000 7f3905e00700 1 --2- 172.21.15.62:0/3000057328 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f38ec06ecc0 0x7f38ec071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:26:10.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:10.671+0000 7f3905e00700 1 --2- 172.21.15.62:0/3000057328 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f38ec06ecc0 0x7f38ec071170 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f38f8006010 tx=0x7f38f8005a90 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:26:10.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:10.677+0000 7f38fea00700 1 -- 172.21.15.62:0/3000057328 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f38f4057870 con 0x7f39080739a0 2024-04-20T22:26:10.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:26:10 smithi189 ceph-mon[57022]: from='client.36668 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:26:10.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:26:10 smithi189 ceph-mon[57022]: from='client.36672 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:26:10.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:26:10 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/1082590863' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:26:10.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:26:10 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/1794032650' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:26:10.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:26:10 smithi062 ceph-mon[76322]: from='client.36668 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:26:10.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:26:10 smithi062 ceph-mon[76322]: from='client.36672 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:26:10.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:26:10 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/1082590863' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:26:10.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:26:10 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/1794032650' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:26:11.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:11.020+0000 7f390d200700 1 -- 172.21.15.62:0/3000057328 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f390804f2a0 con 0x7f39080739a0 2024-04-20T22:26:11.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:11.020+0000 7f38fea00700 1 -- 172.21.15.62:0/3000057328 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7f38f405af30 con 0x7f39080739a0 2024-04-20T22:26:11.021 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-20T22:26:11.021 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-20T22:26:11.021 INFO:teuthology.orchestra.run.smithi062.stdout: host smithi189 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi189/config 2024-04-20T22:26:11.021 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-20T22:26:11.021 INFO:teuthology.orchestra.run.smithi062.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi189 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp11670v2q:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:26:11.021 INFO:teuthology.orchestra.run.smithi062.stdout:Traceback (most recent call last): 2024-04-20T22:26:11.021 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-20T22:26:11.021 INFO:teuthology.orchestra.run.smithi062.stdout: return _run_code(code, main_globals, None, 2024-04-20T22:26:11.021 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-20T22:26:11.022 INFO:teuthology.orchestra.run.smithi062.stdout: exec(code, run_globals) 2024-04-20T22:26:11.022 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5618, in 2024-04-20T22:26:11.022 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5606, in main 2024-04-20T22:26:11.022 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 408, in _infer_config 2024-04-20T22:26:11.022 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 323, in _infer_fsid 2024-04-20T22:26:11.022 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 436, in _infer_image 2024-04-20T22:26:11.022 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 310, in _validate_fsid 2024-04-20T22:26:11.022 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 3359, in command_ceph_volume 2024-04-20T22:26:11.022 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-20T22:26:11.022 INFO:teuthology.orchestra.run.smithi062.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi189 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp11670v2q:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:26:11.022 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-20T22:26:11.022 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is degraded 2024-04-20T22:26:11.022 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T22:26:11.022 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T22:26:11.023 INFO:teuthology.orchestra.run.smithi062.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-20T22:26:11.023 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-20T22:26:11.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:11.023+0000 7f390d200700 1 -- 172.21.15.62:0/3000057328 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f38ec06ecc0 msgr2=0x7f38ec071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:26:11.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:11.023+0000 7f390d200700 1 --2- 172.21.15.62:0/3000057328 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f38ec06ecc0 0x7f38ec071170 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f38f8006010 tx=0x7f38f8005a90 comp rx=0 tx=0).stop 2024-04-20T22:26:11.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:11.023+0000 7f390d200700 1 -- 172.21.15.62:0/3000057328 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39080739a0 msgr2=0x7f3908198700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:26:11.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:11.023+0000 7f390d200700 1 --2- 172.21.15.62:0/3000057328 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39080739a0 0x7f3908198700 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7f38f400b4b0 tx=0x7f38f400b870 comp rx=0 tx=0).stop 2024-04-20T22:26:11.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:11.023+0000 7f390d200700 1 -- 172.21.15.62:0/3000057328 shutdown_connections 2024-04-20T22:26:11.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:11.023+0000 7f390d200700 1 --2- 172.21.15.62:0/3000057328 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f38ec06ecc0 0x7f38ec071170 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:26:11.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:11.023+0000 7f390d200700 1 --2- 172.21.15.62:0/3000057328 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3908073090 0x7f39081981c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:26:11.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:11.023+0000 7f390d200700 1 --2- 172.21.15.62:0/3000057328 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39080739a0 0x7f3908198700 unknown :-1 s=CLOSED pgs=1308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:26:11.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:11.023+0000 7f390d200700 1 -- 172.21.15.62:0/3000057328 >> 172.21.15.62:0/3000057328 conn(0x7f39080fbda0 msgr2=0x7f3908106e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:26:11.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:11.024+0000 7f390d200700 1 -- 172.21.15.62:0/3000057328 shutdown_connections 2024-04-20T22:26:11.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:11.024+0000 7f390d200700 1 -- 172.21.15.62:0/3000057328 wait complete. 2024-04-20T22:26:11.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:26:11 smithi189 ceph-mon[57022]: pgmap v1529: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:26:11.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:26:11 smithi189 ceph-mon[57022]: from='client.36684 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:26:11.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:26:11 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/3000057328' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:26:11.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:26:11 smithi062 ceph-mon[76322]: pgmap v1529: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:26:11.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:26:11 smithi062 ceph-mon[76322]: from='client.36684 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:26:11.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:26:11 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/3000057328' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:26:13.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:26:13 smithi189 ceph-mon[57022]: pgmap v1530: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:26:13.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:26:13 smithi062 ceph-mon[76322]: pgmap v1530: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:26:15.733 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:26:15 smithi062 ceph-mon[76322]: pgmap v1531: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:26:15.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:26:15 smithi189 ceph-mon[57022]: pgmap v1531: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:26:17.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:26:17 smithi189 ceph-mon[57022]: pgmap v1532: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:26:17.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:26:17 smithi062 ceph-mon[76322]: pgmap v1532: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:26:19.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:26:19 smithi189 ceph-mon[57022]: pgmap v1533: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:26:19.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:26:19 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:26:19.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:26:19 smithi062 ceph-mon[76322]: pgmap v1533: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:26:19.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:26:19 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:26:21.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:26:21 smithi189 ceph-mon[57022]: pgmap v1534: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:26:21.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:26:21 smithi062 ceph-mon[76322]: pgmap v1534: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:26:23.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:26:23 smithi062 ceph-mon[76322]: pgmap v1535: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:26:23.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:26:23 smithi189 ceph-mon[57022]: pgmap v1535: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:26:25.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:26:25 smithi062 ceph-mon[76322]: pgmap v1536: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:26:25.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:26:25 smithi189 ceph-mon[57022]: pgmap v1536: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:26:27.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:26:27 smithi062 ceph-mon[76322]: pgmap v1537: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:26:27.898 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:26:27 smithi189 ceph-mon[57022]: pgmap v1537: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:26:29.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:26:29 smithi062 ceph-mon[76322]: pgmap v1538: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:26:29.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:26:29 smithi189 ceph-mon[57022]: pgmap v1538: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:26:31.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:26:31 smithi189 ceph-mon[57022]: pgmap v1539: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:26:31.537 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:26:31 smithi062 ceph-mon[76322]: pgmap v1539: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:26:33.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:26:33 smithi189 ceph-mon[57022]: pgmap v1540: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:26:33.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:26:33 smithi062 ceph-mon[76322]: pgmap v1540: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:26:35.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:26:35 smithi189 ceph-mon[57022]: pgmap v1541: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:26:35.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:26:35 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:26:35.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:26:35 smithi062 ceph-mon[76322]: pgmap v1541: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:26:35.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:26:35 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:26:37.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:26:37 smithi189 ceph-mon[57022]: pgmap v1542: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:26:37.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:26:37 smithi062 ceph-mon[76322]: pgmap v1542: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:26:39.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:26:39 smithi189 ceph-mon[57022]: pgmap v1543: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:26:39.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:26:39 smithi062 ceph-mon[76322]: pgmap v1543: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:26:41.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:41.202+0000 7fbe44e00700 1 -- 172.21.15.62:0/3040637341 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe4010c8f0 msgr2=0x7fbe4010ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:26:41.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:41.202+0000 7fbe44e00700 1 --2- 172.21.15.62:0/3040637341 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe4010c8f0 0x7fbe4010ccc0 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7fbe30009910 tx=0x7fbe30009c20 comp rx=0 tx=0).stop 2024-04-20T22:26:41.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:41.202+0000 7fbe44e00700 1 -- 172.21.15.62:0/3040637341 shutdown_connections 2024-04-20T22:26:41.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:41.202+0000 7fbe44e00700 1 --2- 172.21.15.62:0/3040637341 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe40106b80 0x7fbe40106ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:26:41.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:41.202+0000 7fbe44e00700 1 --2- 172.21.15.62:0/3040637341 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe4010c8f0 0x7fbe4010ccc0 unknown :-1 s=CLOSED pgs=1309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:26:41.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:41.202+0000 7fbe44e00700 1 -- 172.21.15.62:0/3040637341 >> 172.21.15.62:0/3040637341 conn(0x7fbe40076060 msgr2=0x7fbe40078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:26:41.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:41.202+0000 7fbe44e00700 1 -- 172.21.15.62:0/3040637341 shutdown_connections 2024-04-20T22:26:41.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:41.202+0000 7fbe44e00700 1 -- 172.21.15.62:0/3040637341 wait complete. 2024-04-20T22:26:41.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:41.203+0000 7fbe44e00700 1 Processor -- start 2024-04-20T22:26:41.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:41.204+0000 7fbe44e00700 1 -- start start 2024-04-20T22:26:41.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:41.204+0000 7fbe44e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe40106b80 0x7fbe4019c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:26:41.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:41.204+0000 7fbe44e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe4010c8f0 0x7fbe4019ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:26:41.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:41.204+0000 7fbe44e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe4019d120 con 0x7fbe4010c8f0 2024-04-20T22:26:41.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:41.204+0000 7fbe44e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe401a0eb0 con 0x7fbe40106b80 2024-04-20T22:26:41.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:41.204+0000 7fbe37e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe4010c8f0 0x7fbe4019ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:26:41.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:41.204+0000 7fbe37e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe4010c8f0 0x7fbe4019ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45944/0 (socket says 172.21.15.62:45944) 2024-04-20T22:26:41.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:41.204+0000 7fbe37e00700 1 -- 172.21.15.62:0/361852922 learned_addr learned my addr 172.21.15.62:0/361852922 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:26:41.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:41.205+0000 7fbe3de00700 1 --2- 172.21.15.62:0/361852922 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe40106b80 0x7fbe4019c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:26:41.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:41.205+0000 7fbe37e00700 1 -- 172.21.15.62:0/361852922 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe40106b80 msgr2=0x7fbe4019c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:26:41.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:41.205+0000 7fbe37e00700 1 --2- 172.21.15.62:0/361852922 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe40106b80 0x7fbe4019c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:26:41.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:41.205+0000 7fbe37e00700 1 -- 172.21.15.62:0/361852922 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe300095f0 con 0x7fbe4010c8f0 2024-04-20T22:26:41.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:41.205+0000 7fbe3de00700 1 --2- 172.21.15.62:0/361852922 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe40106b80 0x7fbe4019c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T22:26:41.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:41.205+0000 7fbe37e00700 1 --2- 172.21.15.62:0/361852922 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe4010c8f0 0x7fbe4019ca40 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7fbe2800d8d0 tx=0x7fbe2800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:26:41.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:41.206+0000 7fbe36a00700 1 -- 172.21.15.62:0/361852922 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbe28009940 con 0x7fbe4010c8f0 2024-04-20T22:26:41.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:41.206+0000 7fbe36a00700 1 -- 172.21.15.62:0/361852922 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbe28010460 con 0x7fbe4010c8f0 2024-04-20T22:26:41.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:41.206+0000 7fbe44e00700 1 -- 172.21.15.62:0/361852922 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe401a1190 con 0x7fbe4010c8f0 2024-04-20T22:26:41.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:41.206+0000 7fbe36a00700 1 -- 172.21.15.62:0/361852922 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbe2800f5f0 con 0x7fbe4010c8f0 2024-04-20T22:26:41.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:41.206+0000 7fbe44e00700 1 -- 172.21.15.62:0/361852922 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe401a16e0 con 0x7fbe4010c8f0 2024-04-20T22:26:41.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:41.207+0000 7fbe44e00700 1 -- 172.21.15.62:0/361852922 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe40109b20 con 0x7fbe4010c8f0 2024-04-20T22:26:41.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:41.208+0000 7fbe36a00700 1 -- 172.21.15.62:0/361852922 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fbe280105d0 con 0x7fbe4010c8f0 2024-04-20T22:26:41.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:41.209+0000 7fbe36a00700 1 --2- 172.21.15.62:0/361852922 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fbe2406ecc0 0x7fbe24071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:26:41.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:41.209+0000 7fbe36a00700 1 -- 172.21.15.62:0/361852922 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fbe2808ee10 con 0x7fbe4010c8f0 2024-04-20T22:26:41.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:41.209+0000 7fbe3de00700 1 --2- 172.21.15.62:0/361852922 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fbe2406ecc0 0x7fbe24071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:26:41.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:41.210+0000 7fbe3de00700 1 --2- 172.21.15.62:0/361852922 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fbe2406ecc0 0x7fbe24071170 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fbe30006010 tx=0x7fbe3000b540 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:26:41.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:41.214+0000 7fbe36a00700 1 -- 172.21.15.62:0/361852922 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fbe28057ba0 con 0x7fbe4010c8f0 2024-04-20T22:26:41.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:41.476+0000 7fbe44e00700 1 -- 172.21.15.62:0/361852922 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fbe401a19c0 con 0x7fbe2406ecc0 2024-04-20T22:26:41.477 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:26:41 smithi062 ceph-mon[76322]: pgmap v1544: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:26:41.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:41.478+0000 7fbe36a00700 1 -- 172.21.15.62:0/361852922 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fbe401a19c0 con 0x7fbe2406ecc0 2024-04-20T22:26:41.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:41.480+0000 7fbe44e00700 1 -- 172.21.15.62:0/361852922 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fbe2406ecc0 msgr2=0x7fbe24071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:26:41.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:41.481+0000 7fbe44e00700 1 --2- 172.21.15.62:0/361852922 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fbe2406ecc0 0x7fbe24071170 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fbe30006010 tx=0x7fbe3000b540 comp rx=0 tx=0).stop 2024-04-20T22:26:41.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:41.481+0000 7fbe44e00700 1 -- 172.21.15.62:0/361852922 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe4010c8f0 msgr2=0x7fbe4019ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:26:41.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:41.481+0000 7fbe44e00700 1 --2- 172.21.15.62:0/361852922 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe4010c8f0 0x7fbe4019ca40 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7fbe2800d8d0 tx=0x7fbe2800dc90 comp rx=0 tx=0).stop 2024-04-20T22:26:41.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:41.481+0000 7fbe44e00700 1 -- 172.21.15.62:0/361852922 shutdown_connections 2024-04-20T22:26:41.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:41.481+0000 7fbe44e00700 1 --2- 172.21.15.62:0/361852922 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fbe2406ecc0 0x7fbe24071170 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:26:41.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:41.481+0000 7fbe44e00700 1 --2- 172.21.15.62:0/361852922 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe40106b80 0x7fbe4019c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:26:41.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:41.481+0000 7fbe44e00700 1 --2- 172.21.15.62:0/361852922 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe4010c8f0 0x7fbe4019ca40 unknown :-1 s=CLOSED pgs=1310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:26:41.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:41.481+0000 7fbe44e00700 1 -- 172.21.15.62:0/361852922 >> 172.21.15.62:0/361852922 conn(0x7fbe40076060 msgr2=0x7fbe40077aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:26:41.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:41.481+0000 7fbe44e00700 1 -- 172.21.15.62:0/361852922 shutdown_connections 2024-04-20T22:26:41.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:41.481+0000 7fbe44e00700 1 -- 172.21.15.62:0/361852922 wait complete. 2024-04-20T22:26:41.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:26:41 smithi189 ceph-mon[57022]: pgmap v1544: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:26:41.497 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T22:26:41.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:41.650+0000 7fce55400700 1 -- 172.21.15.62:0/1936435861 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce501012f0 msgr2=0x7fce501016c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:26:41.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:41.650+0000 7fce55400700 1 --2- 172.21.15.62:0/1936435861 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce501012f0 0x7fce501016c0 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7fce40009910 tx=0x7fce40009c20 comp rx=0 tx=0).stop 2024-04-20T22:26:41.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:41.650+0000 7fce55400700 1 -- 172.21.15.62:0/1936435861 shutdown_connections 2024-04-20T22:26:41.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:41.650+0000 7fce55400700 1 --2- 172.21.15.62:0/1936435861 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce50101c00 0x7fce5010a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:26:41.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:41.650+0000 7fce55400700 1 --2- 172.21.15.62:0/1936435861 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce501012f0 0x7fce501016c0 unknown :-1 s=CLOSED pgs=1311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:26:41.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:41.650+0000 7fce55400700 1 -- 172.21.15.62:0/1936435861 >> 172.21.15.62:0/1936435861 conn(0x7fce500fac60 msgr2=0x7fce500fd070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:26:41.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:41.651+0000 7fce55400700 1 -- 172.21.15.62:0/1936435861 shutdown_connections 2024-04-20T22:26:41.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:41.651+0000 7fce55400700 1 -- 172.21.15.62:0/1936435861 wait complete. 2024-04-20T22:26:41.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:41.652+0000 7fce55400700 1 Processor -- start 2024-04-20T22:26:41.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:41.652+0000 7fce55400700 1 -- start start 2024-04-20T22:26:41.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:41.652+0000 7fce55400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce501012f0 0x7fce50195ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:26:41.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:41.652+0000 7fce55400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce50101c00 0x7fce50196400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:26:41.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:41.652+0000 7fce55400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce50196ae0 con 0x7fce50101c00 2024-04-20T22:26:41.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:41.652+0000 7fce55400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce5019a820 con 0x7fce501012f0 2024-04-20T22:26:41.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:41.653+0000 7fce4de00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce50101c00 0x7fce50196400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:26:41.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:41.653+0000 7fce4de00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce50101c00 0x7fce50196400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:57204/0 (socket says 172.21.15.62:57204) 2024-04-20T22:26:41.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:41.653+0000 7fce4de00700 1 -- 172.21.15.62:0/225203091 learned_addr learned my addr 172.21.15.62:0/225203091 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:26:41.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:41.653+0000 7fce4e800700 1 --2- 172.21.15.62:0/225203091 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce501012f0 0x7fce50195ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:26:41.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:41.653+0000 7fce4de00700 1 -- 172.21.15.62:0/225203091 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce501012f0 msgr2=0x7fce50195ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:26:41.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:41.653+0000 7fce4de00700 1 --2- 172.21.15.62:0/225203091 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce501012f0 0x7fce50195ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:26:41.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:41.653+0000 7fce4de00700 1 -- 172.21.15.62:0/225203091 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce400095f0 con 0x7fce50101c00 2024-04-20T22:26:41.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:41.654+0000 7fce4de00700 1 --2- 172.21.15.62:0/225203091 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce50101c00 0x7fce50196400 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7fce3c00e920 tx=0x7fce3c00ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:26:41.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:41.654+0000 7fce47400700 1 -- 172.21.15.62:0/225203091 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce3c009790 con 0x7fce50101c00 2024-04-20T22:26:41.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:41.654+0000 7fce47400700 1 -- 172.21.15.62:0/225203091 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fce3c00cba0 con 0x7fce50101c00 2024-04-20T22:26:41.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:41.654+0000 7fce55400700 1 -- 172.21.15.62:0/225203091 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce5019ab00 con 0x7fce50101c00 2024-04-20T22:26:41.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:41.654+0000 7fce47400700 1 -- 172.21.15.62:0/225203091 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce3c018650 con 0x7fce50101c00 2024-04-20T22:26:41.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:41.654+0000 7fce55400700 1 -- 172.21.15.62:0/225203091 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce5019b050 con 0x7fce50101c00 2024-04-20T22:26:41.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:41.655+0000 7fce55400700 1 -- 172.21.15.62:0/225203091 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce50107970 con 0x7fce50101c00 2024-04-20T22:26:41.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:41.656+0000 7fce47400700 1 -- 172.21.15.62:0/225203091 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fce3c00cd10 con 0x7fce50101c00 2024-04-20T22:26:41.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:41.657+0000 7fce47400700 1 --2- 172.21.15.62:0/225203091 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fce3406ecc0 0x7fce34071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:26:41.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:41.657+0000 7fce47400700 1 -- 172.21.15.62:0/225203091 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fce3c014070 con 0x7fce50101c00 2024-04-20T22:26:41.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:41.658+0000 7fce4e800700 1 --2- 172.21.15.62:0/225203091 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fce3406ecc0 0x7fce34071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:26:41.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:41.659+0000 7fce4e800700 1 --2- 172.21.15.62:0/225203091 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fce3406ecc0 0x7fce34071170 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fce4000b760 tx=0x7fce40005a90 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:26:41.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:41.663+0000 7fce47400700 1 -- 172.21.15.62:0/225203091 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fce3c053ff0 con 0x7fce50101c00 2024-04-20T22:26:41.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:41.920+0000 7fce55400700 1 -- 172.21.15.62:0/225203091 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fce5019b2e0 con 0x7fce3406ecc0 2024-04-20T22:26:41.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:41.923+0000 7fce47400700 1 -- 172.21.15.62:0/225203091 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fce5019b2e0 con 0x7fce3406ecc0 2024-04-20T22:26:41.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:41.925+0000 7fce55400700 1 -- 172.21.15.62:0/225203091 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fce3406ecc0 msgr2=0x7fce34071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:26:41.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:41.925+0000 7fce55400700 1 --2- 172.21.15.62:0/225203091 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fce3406ecc0 0x7fce34071170 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fce4000b760 tx=0x7fce40005a90 comp rx=0 tx=0).stop 2024-04-20T22:26:41.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:41.925+0000 7fce55400700 1 -- 172.21.15.62:0/225203091 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce50101c00 msgr2=0x7fce50196400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:26:41.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:41.925+0000 7fce55400700 1 --2- 172.21.15.62:0/225203091 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce50101c00 0x7fce50196400 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7fce3c00e920 tx=0x7fce3c00ece0 comp rx=0 tx=0).stop 2024-04-20T22:26:41.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:41.925+0000 7fce55400700 1 -- 172.21.15.62:0/225203091 shutdown_connections 2024-04-20T22:26:41.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:41.925+0000 7fce55400700 1 --2- 172.21.15.62:0/225203091 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fce3406ecc0 0x7fce34071170 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:26:41.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:41.925+0000 7fce55400700 1 --2- 172.21.15.62:0/225203091 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce501012f0 0x7fce50195ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:26:41.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:41.925+0000 7fce55400700 1 --2- 172.21.15.62:0/225203091 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce50101c00 0x7fce50196400 unknown :-1 s=CLOSED pgs=1312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:26:41.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:41.925+0000 7fce55400700 1 -- 172.21.15.62:0/225203091 >> 172.21.15.62:0/225203091 conn(0x7fce500fac60 msgr2=0x7fce50104ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:26:41.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:41.926+0000 7fce55400700 1 -- 172.21.15.62:0/225203091 shutdown_connections 2024-04-20T22:26:41.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:41.926+0000 7fce55400700 1 -- 172.21.15.62:0/225203091 wait complete. 2024-04-20T22:26:42.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:42.077+0000 7fa619400700 1 -- 172.21.15.62:0/765268736 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa614100290 msgr2=0x7fa614100660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:26:42.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:42.077+0000 7fa619400700 1 --2- 172.21.15.62:0/765268736 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa614100290 0x7fa614100660 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7fa604009910 tx=0x7fa604009c20 comp rx=0 tx=0).stop 2024-04-20T22:26:42.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:42.077+0000 7fa619400700 1 -- 172.21.15.62:0/765268736 shutdown_connections 2024-04-20T22:26:42.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:42.077+0000 7fa619400700 1 --2- 172.21.15.62:0/765268736 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa614100c30 0x7fa614104c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:26:42.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:42.077+0000 7fa619400700 1 --2- 172.21.15.62:0/765268736 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa614100290 0x7fa614100660 unknown :-1 s=CLOSED pgs=1313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:26:42.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:42.077+0000 7fa619400700 1 -- 172.21.15.62:0/765268736 >> 172.21.15.62:0/765268736 conn(0x7fa6140fbb10 msgr2=0x7fa6140fdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:26:42.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:42.077+0000 7fa619400700 1 -- 172.21.15.62:0/765268736 shutdown_connections 2024-04-20T22:26:42.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:42.078+0000 7fa619400700 1 -- 172.21.15.62:0/765268736 wait complete. 2024-04-20T22:26:42.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:42.079+0000 7fa619400700 1 Processor -- start 2024-04-20T22:26:42.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:42.079+0000 7fa619400700 1 -- start start 2024-04-20T22:26:42.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:42.079+0000 7fa619400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa614100290 0x7fa614073050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:26:42.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:42.079+0000 7fa619400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa614100c30 0x7fa614073590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:26:42.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:42.079+0000 7fa619400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa614073ad0 con 0x7fa614100c30 2024-04-20T22:26:42.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:42.079+0000 7fa619400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa614073c10 con 0x7fa614100290 2024-04-20T22:26:42.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:42.079+0000 7fa613400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa614100c30 0x7fa614073590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:26:42.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:42.080+0000 7fa613400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa614100c30 0x7fa614073590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:57214/0 (socket says 172.21.15.62:57214) 2024-04-20T22:26:42.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:42.080+0000 7fa613400700 1 -- 172.21.15.62:0/590585077 learned_addr learned my addr 172.21.15.62:0/590585077 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:26:42.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:42.080+0000 7fa613e00700 1 --2- 172.21.15.62:0/590585077 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa614100290 0x7fa614073050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:26:42.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:42.080+0000 7fa613400700 1 -- 172.21.15.62:0/590585077 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa614100290 msgr2=0x7fa614073050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:26:42.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:42.080+0000 7fa613400700 1 --2- 172.21.15.62:0/590585077 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa614100290 0x7fa614073050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:26:42.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:42.080+0000 7fa613400700 1 -- 172.21.15.62:0/590585077 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6040095f0 con 0x7fa614100c30 2024-04-20T22:26:42.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:42.080+0000 7fa613400700 1 --2- 172.21.15.62:0/590585077 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa614100c30 0x7fa614073590 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7fa60000d900 tx=0x7fa60000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:26:42.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:42.081+0000 7fa610c00700 1 -- 172.21.15.62:0/590585077 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa600009900 con 0x7fa614100c30 2024-04-20T22:26:42.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:42.081+0000 7fa610c00700 1 -- 172.21.15.62:0/590585077 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa600010460 con 0x7fa614100c30 2024-04-20T22:26:42.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:42.081+0000 7fa619400700 1 -- 172.21.15.62:0/590585077 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6141a1e00 con 0x7fa614100c30 2024-04-20T22:26:42.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:42.081+0000 7fa610c00700 1 -- 172.21.15.62:0/590585077 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa60000f5f0 con 0x7fa614100c30 2024-04-20T22:26:42.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:42.081+0000 7fa619400700 1 -- 172.21.15.62:0/590585077 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6141a2210 con 0x7fa614100c30 2024-04-20T22:26:42.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:42.082+0000 7fa619400700 1 -- 172.21.15.62:0/590585077 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6141085a0 con 0x7fa614100c30 2024-04-20T22:26:42.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:42.083+0000 7fa610c00700 1 -- 172.21.15.62:0/590585077 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fa6000105d0 con 0x7fa614100c30 2024-04-20T22:26:42.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:42.084+0000 7fa610c00700 1 --2- 172.21.15.62:0/590585077 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa5f806ecc0 0x7fa5f8071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:26:42.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:42.084+0000 7fa610c00700 1 -- 172.21.15.62:0/590585077 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fa60008ecc0 con 0x7fa614100c30 2024-04-20T22:26:42.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:42.084+0000 7fa613e00700 1 --2- 172.21.15.62:0/590585077 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa5f806ecc0 0x7fa5f8071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:26:42.085 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:42.085+0000 7fa613e00700 1 --2- 172.21.15.62:0/590585077 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa5f806ecc0 0x7fa5f8071170 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fa60400b5c0 tx=0x7fa60401a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:26:42.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:42.089+0000 7fa610c00700 1 -- 172.21.15.62:0/590585077 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fa600057a70 con 0x7fa614100c30 2024-04-20T22:26:42.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:26:42 smithi062 ceph-mon[76322]: from='client.36692 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:26:42.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:42.325+0000 7fa619400700 1 -- 172.21.15.62:0/590585077 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fa61402ccf0 con 0x7fa5f806ecc0 2024-04-20T22:26:42.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:42.358+0000 7fa610c00700 1 -- 172.21.15.62:0/590585077 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fa61402ccf0 con 0x7fa5f806ecc0 2024-04-20T22:26:42.359 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T22:26:42.359 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (52m) 13m ago 66m 14.6M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T22:26:42.359 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (66m) 13m ago 66m 11.6M - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T22:26:42.359 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (65m) 13m ago 65m 11.2M - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T22:26:42.359 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (48m) 13m ago 66m 6660k - 19.0.0-3133-ga8c84f59 76bc6b91b726 2825d5ba11c6 2024-04-20T22:26:42.359 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (48m) 13m ago 65m 6665k - 19.0.0-3133-ga8c84f59 76bc6b91b726 20f746fa1bba 2024-04-20T22:26:42.359 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (51m) 13m ago 66m 75.5M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T22:26:42.359 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (44m) 13m ago 59m 22.9M - 19.0.0-3133-ga8c84f59 76bc6b91b726 de80bc90a683 2024-04-20T22:26:42.359 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (44m) 13m ago 59m 23.0M - 19.0.0-3133-ga8c84f59 76bc6b91b726 cd793b7ee98e 2024-04-20T22:26:42.360 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (28m) 13m ago 59m 18.5M - 19.0.0-3133-ga8c84f59 76bc6b91b726 0e7b0bc4d1ab 2024-04-20T22:26:42.360 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (13m) 13m ago 59m 11.4M - 19.0.0-3133-ga8c84f59 76bc6b91b726 0af732ac9e06 2024-04-20T22:26:42.360 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (55m) 13m ago 68m 517M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T22:26:42.360 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (54m) 13m ago 65m 460M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T22:26:42.360 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (50m) 13m ago 68m 86.3M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 019eb1479485 2024-04-20T22:26:42.360 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (49m) 13m ago 64m 58.0M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 55aac560aab6 2024-04-20T22:26:42.360 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (53m) 13m ago 66m 8766k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T22:26:42.360 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (53m) 13m ago 65m 8711k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T22:26:42.360 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (47m) 13m ago 64m 91.3M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 ee2bb901f26e 2024-04-20T22:26:42.360 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (47m) 13m ago 63m 94.9M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 9075152d4162 2024-04-20T22:26:42.360 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (46m) 13m ago 62m 78.5M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 2d4f3e38dc0e 2024-04-20T22:26:42.360 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (46m) 13m ago 62m 94.1M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 8f97b17ae0eb 2024-04-20T22:26:42.360 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (45m) 13m ago 61m 96.1M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 653d8079967d 2024-04-20T22:26:42.360 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (44m) 13m ago 61m 90.9M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 596dded14c12 2024-04-20T22:26:42.361 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (53m) 13m ago 65m 90.4M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T22:26:42.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:42.361+0000 7fa619400700 1 -- 172.21.15.62:0/590585077 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa5f806ecc0 msgr2=0x7fa5f8071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:26:42.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:42.361+0000 7fa619400700 1 --2- 172.21.15.62:0/590585077 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa5f806ecc0 0x7fa5f8071170 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fa60400b5c0 tx=0x7fa60401a040 comp rx=0 tx=0).stop 2024-04-20T22:26:42.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:42.361+0000 7fa619400700 1 -- 172.21.15.62:0/590585077 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa614100c30 msgr2=0x7fa614073590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:26:42.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:42.361+0000 7fa619400700 1 --2- 172.21.15.62:0/590585077 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa614100c30 0x7fa614073590 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7fa60000d900 tx=0x7fa60000dcc0 comp rx=0 tx=0).stop 2024-04-20T22:26:42.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:42.362+0000 7fa619400700 1 -- 172.21.15.62:0/590585077 shutdown_connections 2024-04-20T22:26:42.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:42.362+0000 7fa619400700 1 --2- 172.21.15.62:0/590585077 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa5f806ecc0 0x7fa5f8071170 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:26:42.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:42.362+0000 7fa619400700 1 --2- 172.21.15.62:0/590585077 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa614100290 0x7fa614073050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:26:42.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:42.362+0000 7fa619400700 1 --2- 172.21.15.62:0/590585077 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa614100c30 0x7fa614073590 unknown :-1 s=CLOSED pgs=1314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:26:42.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:42.362+0000 7fa619400700 1 -- 172.21.15.62:0/590585077 >> 172.21.15.62:0/590585077 conn(0x7fa6140fbb10 msgr2=0x7fa6140fd1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:26:42.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:42.362+0000 7fa619400700 1 -- 172.21.15.62:0/590585077 shutdown_connections 2024-04-20T22:26:42.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:42.362+0000 7fa619400700 1 -- 172.21.15.62:0/590585077 wait complete. 2024-04-20T22:26:42.490 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:26:42 smithi189 ceph-mon[57022]: from='client.36692 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:26:42.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:42.522+0000 7f946ca00700 1 -- 172.21.15.62:0/279659922 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9468102470 msgr2=0x7f94681028e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:26:42.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:42.522+0000 7f946ca00700 1 --2- 172.21.15.62:0/279659922 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9468102470 0x7f94681028e0 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f9454009910 tx=0x7f9454009c20 comp rx=0 tx=0).stop 2024-04-20T22:26:42.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:42.522+0000 7f946ca00700 1 -- 172.21.15.62:0/279659922 shutdown_connections 2024-04-20T22:26:42.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:42.522+0000 7f946ca00700 1 --2- 172.21.15.62:0/279659922 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9468102470 0x7f94681028e0 unknown :-1 s=CLOSED pgs=1315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:26:42.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:42.522+0000 7f946ca00700 1 --2- 172.21.15.62:0/279659922 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9468108470 0x7f9468108840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:26:42.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:42.522+0000 7f946ca00700 1 -- 172.21.15.62:0/279659922 >> 172.21.15.62:0/279659922 conn(0x7f94680fdf90 msgr2=0x7f94681003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:26:42.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:42.523+0000 7f946ca00700 1 -- 172.21.15.62:0/279659922 shutdown_connections 2024-04-20T22:26:42.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:42.523+0000 7f946ca00700 1 -- 172.21.15.62:0/279659922 wait complete. 2024-04-20T22:26:42.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:42.524+0000 7f946ca00700 1 Processor -- start 2024-04-20T22:26:42.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:42.524+0000 7f946ca00700 1 -- start start 2024-04-20T22:26:42.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:42.524+0000 7f946ca00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9468102470 0x7f94681981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:26:42.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:42.524+0000 7f946ca00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9468108470 0x7f9468198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:26:42.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:42.524+0000 7f946ca00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9468198e10 con 0x7f9468108470 2024-04-20T22:26:42.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:42.524+0000 7f946ca00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f946819cba0 con 0x7f9468102470 2024-04-20T22:26:42.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:42.525+0000 7f9465400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9468108470 0x7f9468198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:26:42.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:42.525+0000 7f9465400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9468108470 0x7f9468198730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:57224/0 (socket says 172.21.15.62:57224) 2024-04-20T22:26:42.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:42.525+0000 7f9465400700 1 -- 172.21.15.62:0/4105547728 learned_addr learned my addr 172.21.15.62:0/4105547728 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:26:42.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:42.525+0000 7f9465400700 1 -- 172.21.15.62:0/4105547728 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9468102470 msgr2=0x7f94681981f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:26:42.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:42.525+0000 7f9465400700 1 --2- 172.21.15.62:0/4105547728 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9468102470 0x7f94681981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:26:42.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:42.525+0000 7f9465400700 1 -- 172.21.15.62:0/4105547728 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94540095f0 con 0x7f9468108470 2024-04-20T22:26:42.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:42.525+0000 7f9465400700 1 --2- 172.21.15.62:0/4105547728 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9468108470 0x7f9468198730 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f94540092e0 tx=0x7f945400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:26:42.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:42.526+0000 7f945ea00700 1 -- 172.21.15.62:0/4105547728 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f945401d070 con 0x7f9468108470 2024-04-20T22:26:42.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:42.526+0000 7f945ea00700 1 -- 172.21.15.62:0/4105547728 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9454005e00 con 0x7f9468108470 2024-04-20T22:26:42.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:42.526+0000 7f946ca00700 1 -- 172.21.15.62:0/4105547728 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f946819ce20 con 0x7f9468108470 2024-04-20T22:26:42.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:42.526+0000 7f945ea00700 1 -- 172.21.15.62:0/4105547728 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9454017400 con 0x7f9468108470 2024-04-20T22:26:42.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:42.526+0000 7f946ca00700 1 -- 172.21.15.62:0/4105547728 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f946819d310 con 0x7f9468108470 2024-04-20T22:26:42.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:42.527+0000 7f946ca00700 1 -- 172.21.15.62:0/4105547728 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f946810a970 con 0x7f9468108470 2024-04-20T22:26:42.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:42.528+0000 7f945ea00700 1 -- 172.21.15.62:0/4105547728 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f945400f460 con 0x7f9468108470 2024-04-20T22:26:42.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:42.529+0000 7f945ea00700 1 --2- 172.21.15.62:0/4105547728 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f944c06ecc0 0x7f944c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:26:42.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:42.529+0000 7f945ea00700 1 -- 172.21.15.62:0/4105547728 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f9454090330 con 0x7f9468108470 2024-04-20T22:26:42.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:42.530+0000 7f9465e00700 1 --2- 172.21.15.62:0/4105547728 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f944c06ecc0 0x7f944c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:26:42.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:42.530+0000 7f9465e00700 1 --2- 172.21.15.62:0/4105547728 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f944c06ecc0 0x7f944c071170 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f9458005950 tx=0x7f94580058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:26:42.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:42.534+0000 7f945ea00700 1 -- 172.21.15.62:0/4105547728 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f94540590c0 con 0x7f9468108470 2024-04-20T22:26:42.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:42.867+0000 7f946ca00700 1 -- 172.21.15.62:0/4105547728 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f946810a350 con 0x7f9468108470 2024-04-20T22:26:42.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:42.868+0000 7f945ea00700 1 -- 172.21.15.62:0/4105547728 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f945405c780 con 0x7f9468108470 2024-04-20T22:26:42.869 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:26:42.869 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T22:26:42.869 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:26:42.869 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:26:42.869 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T22:26:42.869 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:26:42.869 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:26:42.869 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T22:26:42.870 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 6 2024-04-20T22:26:42.870 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:26:42.870 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T22:26:42.870 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 4 2024-04-20T22:26:42.870 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:26:42.870 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T22:26:42.870 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 14 2024-04-20T22:26:42.870 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T22:26:42.870 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:26:42.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:42.872+0000 7f946ca00700 1 -- 172.21.15.62:0/4105547728 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f944c06ecc0 msgr2=0x7f944c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:26:42.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:42.872+0000 7f946ca00700 1 --2- 172.21.15.62:0/4105547728 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f944c06ecc0 0x7f944c071170 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f9458005950 tx=0x7f94580058e0 comp rx=0 tx=0).stop 2024-04-20T22:26:42.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:42.872+0000 7f946ca00700 1 -- 172.21.15.62:0/4105547728 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9468108470 msgr2=0x7f9468198730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:26:42.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:42.872+0000 7f946ca00700 1 --2- 172.21.15.62:0/4105547728 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9468108470 0x7f9468198730 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f94540092e0 tx=0x7f945400be20 comp rx=0 tx=0).stop 2024-04-20T22:26:42.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:42.872+0000 7f946ca00700 1 -- 172.21.15.62:0/4105547728 shutdown_connections 2024-04-20T22:26:42.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:42.872+0000 7f946ca00700 1 --2- 172.21.15.62:0/4105547728 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f944c06ecc0 0x7f944c071170 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:26:42.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:42.872+0000 7f946ca00700 1 --2- 172.21.15.62:0/4105547728 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9468102470 0x7f94681981f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:26:42.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:42.872+0000 7f946ca00700 1 --2- 172.21.15.62:0/4105547728 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9468108470 0x7f9468198730 unknown :-1 s=CLOSED pgs=1316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:26:42.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:42.872+0000 7f946ca00700 1 -- 172.21.15.62:0/4105547728 >> 172.21.15.62:0/4105547728 conn(0x7f94680fdf90 msgr2=0x7f94680ff3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:26:42.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:42.872+0000 7f946ca00700 1 -- 172.21.15.62:0/4105547728 shutdown_connections 2024-04-20T22:26:42.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:42.872+0000 7f946ca00700 1 -- 172.21.15.62:0/4105547728 wait complete. 2024-04-20T22:26:43.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.019+0000 7f7611600700 1 -- 172.21.15.62:0/944546444 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f760c106b80 msgr2=0x7f760c106ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:26:43.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.019+0000 7f7611600700 1 --2- 172.21.15.62:0/944546444 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f760c106b80 0x7f760c106ff0 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f75f4009910 tx=0x7f75f4009c20 comp rx=0 tx=0).stop 2024-04-20T22:26:43.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.019+0000 7f7611600700 1 -- 172.21.15.62:0/944546444 shutdown_connections 2024-04-20T22:26:43.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.019+0000 7f7611600700 1 --2- 172.21.15.62:0/944546444 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f760c106b80 0x7f760c106ff0 unknown :-1 s=CLOSED pgs=1317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:26:43.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.019+0000 7f7611600700 1 --2- 172.21.15.62:0/944546444 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f760c10c8f0 0x7f760c10ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:26:43.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.019+0000 7f7611600700 1 -- 172.21.15.62:0/944546444 >> 172.21.15.62:0/944546444 conn(0x7f760c076060 msgr2=0x7f760c078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:26:43.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.020+0000 7f7611600700 1 -- 172.21.15.62:0/944546444 shutdown_connections 2024-04-20T22:26:43.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.020+0000 7f7611600700 1 -- 172.21.15.62:0/944546444 wait complete. 2024-04-20T22:26:43.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.021+0000 7f7611600700 1 Processor -- start 2024-04-20T22:26:43.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.021+0000 7f7611600700 1 -- start start 2024-04-20T22:26:43.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.021+0000 7f7611600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f760c106b80 0x7f760c19c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:26:43.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.021+0000 7f7611600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f760c10c8f0 0x7f760c19ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:26:43.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.021+0000 7f7611600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f760c19d100 con 0x7f760c10c8f0 2024-04-20T22:26:43.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.021+0000 7f7611600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f760c1a0e20 con 0x7f760c106b80 2024-04-20T22:26:43.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.022+0000 7f760a000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f760c10c8f0 0x7f760c19ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:26:43.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.022+0000 7f760a000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f760c10c8f0 0x7f760c19ca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:57244/0 (socket says 172.21.15.62:57244) 2024-04-20T22:26:43.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.022+0000 7f760a000700 1 -- 172.21.15.62:0/2783381445 learned_addr learned my addr 172.21.15.62:0/2783381445 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:26:43.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.022+0000 7f760aa00700 1 --2- 172.21.15.62:0/2783381445 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f760c106b80 0x7f760c19c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:26:43.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.022+0000 7f760a000700 1 -- 172.21.15.62:0/2783381445 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f760c106b80 msgr2=0x7f760c19c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:26:43.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.022+0000 7f760a000700 1 --2- 172.21.15.62:0/2783381445 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f760c106b80 0x7f760c19c4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:26:43.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.022+0000 7f760a000700 1 -- 172.21.15.62:0/2783381445 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75f40095f0 con 0x7f760c10c8f0 2024-04-20T22:26:43.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.022+0000 7f760aa00700 1 --2- 172.21.15.62:0/2783381445 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f760c106b80 0x7f760c19c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T22:26:43.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.022+0000 7f760a000700 1 --2- 172.21.15.62:0/2783381445 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f760c10c8f0 0x7f760c19ca20 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f75f400b5c0 tx=0x7f75f400be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:26:43.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.023+0000 7f7603400700 1 -- 172.21.15.62:0/2783381445 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f75f401d070 con 0x7f760c10c8f0 2024-04-20T22:26:43.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.023+0000 7f7603400700 1 -- 172.21.15.62:0/2783381445 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f75f400f460 con 0x7f760c10c8f0 2024-04-20T22:26:43.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.023+0000 7f7603400700 1 -- 172.21.15.62:0/2783381445 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f75f4017610 con 0x7f760c10c8f0 2024-04-20T22:26:43.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.023+0000 7f7611600700 1 -- 172.21.15.62:0/2783381445 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f760c1a10a0 con 0x7f760c10c8f0 2024-04-20T22:26:43.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.023+0000 7f7611600700 1 -- 172.21.15.62:0/2783381445 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f760c1a1590 con 0x7f760c10c8f0 2024-04-20T22:26:43.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.024+0000 7f7611600700 1 -- 172.21.15.62:0/2783381445 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f760c109b20 con 0x7f760c10c8f0 2024-04-20T22:26:43.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.025+0000 7f7603400700 1 -- 172.21.15.62:0/2783381445 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f75f400f5d0 con 0x7f760c10c8f0 2024-04-20T22:26:43.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.026+0000 7f7603400700 1 --2- 172.21.15.62:0/2783381445 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f75f006ecc0 0x7f75f0071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:26:43.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.026+0000 7f7603400700 1 -- 172.21.15.62:0/2783381445 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f75f4090440 con 0x7f760c10c8f0 2024-04-20T22:26:43.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.026+0000 7f760aa00700 1 --2- 172.21.15.62:0/2783381445 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f75f006ecc0 0x7f75f0071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:26:43.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.027+0000 7f760aa00700 1 --2- 172.21.15.62:0/2783381445 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f75f006ecc0 0x7f75f0071170 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f75fc007900 tx=0x7f75fc008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:26:43.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.031+0000 7f7603400700 1 -- 172.21.15.62:0/2783381445 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f75f40591d0 con 0x7f760c10c8f0 2024-04-20T22:26:43.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:26:43 smithi062 ceph-mon[76322]: pgmap v1545: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:26:43.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:26:43 smithi062 ceph-mon[76322]: from='client.36696 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:26:43.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:26:43 smithi062 ceph-mon[76322]: from='client.36700 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:26:43.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:26:43 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/4105547728' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:26:43.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.304+0000 7f7611600700 1 -- 172.21.15.62:0/2783381445 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f760c1a18c0 con 0x7f760c10c8f0 2024-04-20T22:26:43.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.305+0000 7f7603400700 1 -- 172.21.15.62:0/2783381445 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 29 v29) v1 ==== 76+0+1887 (secure 0 0 0) 0x7f75f405c890 con 0x7f760c10c8f0 2024-04-20T22:26:43.305 INFO:teuthology.orchestra.run.smithi062.stdout:e29 2024-04-20T22:26:43.305 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T22:26:43.305 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:26:43.306 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T22:26:43.306 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:26:43.306 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T22:26:43.306 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T22:26:43.306 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 20 2024-04-20T22:26:43.306 INFO:teuthology.orchestra.run.smithi062.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-20T22:26:43.306 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T22:26:43.306 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T21:42:01.559061+0000 2024-04-20T22:26:43.306 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T22:26:43.306 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T22:26:43.306 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T22:26:43.306 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T22:26:43.306 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T22:26:43.306 INFO:teuthology.orchestra.run.smithi062.stdout:max_xattr_size 65536 2024-04-20T22:26:43.307 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T22:26:43.307 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T22:26:43.307 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 81 2024-04-20T22:26:43.307 INFO:teuthology.orchestra.run.smithi062.stdout:snap_rstat 1 2024-04-20T22:26:43.307 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:26:43.307 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 2 2024-04-20T22:26:43.307 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1 2024-04-20T22:26:43.307 INFO:teuthology.orchestra.run.smithi062.stdout:up {} 2024-04-20T22:26:43.307 INFO:teuthology.orchestra.run.smithi062.stdout:failed 0,1 2024-04-20T22:26:43.307 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T22:26:43.307 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T22:26:43.307 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T22:26:43.307 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T22:26:43.307 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data enabled 2024-04-20T22:26:43.307 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T22:26:43.308 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T22:26:43.308 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 1 2024-04-20T22:26:43.308 INFO:teuthology.orchestra.run.smithi062.stdout:qdb_cluster leader: 0 members: 2024-04-20T22:26:43.308 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:26:43.308 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:26:43.308 INFO:teuthology.orchestra.run.smithi062.stdout:Standby daemons: 2024-04-20T22:26:43.308 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:26:43.308 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ryqblb{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6826/4113916205,v1:172.21.15.62:6827/4113916205] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:26:43.308 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ozizwi{-1:34476} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6828/3801975165,v1:172.21.15.62:6829/3801975165] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:26:43.308 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.dvimer{-1:44727} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6824/3576699232,v1:172.21.15.189:6825/3576699232] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:26:43.308 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.xbgspw{-1:45075} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6826/212905792,v1:172.21.15.189:6827/212905792] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:26:43.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.307+0000 7f7611600700 1 -- 172.21.15.62:0/2783381445 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f75f006ecc0 msgr2=0x7f75f0071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:26:43.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.307+0000 7f7611600700 1 --2- 172.21.15.62:0/2783381445 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f75f006ecc0 0x7f75f0071170 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f75fc007900 tx=0x7f75fc008040 comp rx=0 tx=0).stop 2024-04-20T22:26:43.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.307+0000 7f7611600700 1 -- 172.21.15.62:0/2783381445 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f760c10c8f0 msgr2=0x7f760c19ca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:26:43.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.307+0000 7f7611600700 1 --2- 172.21.15.62:0/2783381445 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f760c10c8f0 0x7f760c19ca20 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f75f400b5c0 tx=0x7f75f400be00 comp rx=0 tx=0).stop 2024-04-20T22:26:43.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.307+0000 7f7611600700 1 -- 172.21.15.62:0/2783381445 shutdown_connections 2024-04-20T22:26:43.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.307+0000 7f7611600700 1 --2- 172.21.15.62:0/2783381445 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f75f006ecc0 0x7f75f0071170 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:26:43.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.307+0000 7f7611600700 1 --2- 172.21.15.62:0/2783381445 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f760c106b80 0x7f760c19c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:26:43.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.307+0000 7f7611600700 1 --2- 172.21.15.62:0/2783381445 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f760c10c8f0 0x7f760c19ca20 unknown :-1 s=CLOSED pgs=1318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:26:43.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.307+0000 7f7611600700 1 -- 172.21.15.62:0/2783381445 >> 172.21.15.62:0/2783381445 conn(0x7f760c076060 msgr2=0x7f760c077a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:26:43.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.308+0000 7f7611600700 1 -- 172.21.15.62:0/2783381445 shutdown_connections 2024-04-20T22:26:43.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.308+0000 7f7611600700 1 -- 172.21.15.62:0/2783381445 wait complete. 2024-04-20T22:26:43.310 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 29 2024-04-20T22:26:43.377 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:26:43 smithi189 ceph-mon[57022]: pgmap v1545: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:26:43.377 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:26:43 smithi189 ceph-mon[57022]: from='client.36696 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:26:43.377 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:26:43 smithi189 ceph-mon[57022]: from='client.36700 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:26:43.377 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:26:43 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/4105547728' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:26:43.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.470+0000 7f5902000700 1 -- 172.21.15.62:0/3787754230 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f58fc0739a0 msgr2=0x7f58fc10c5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:26:43.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.470+0000 7f5902000700 1 --2- 172.21.15.62:0/3787754230 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f58fc0739a0 0x7f58fc10c5e0 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7f58e8009940 tx=0x7f58e8009c50 comp rx=0 tx=0).stop 2024-04-20T22:26:43.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.470+0000 7f5902000700 1 -- 172.21.15.62:0/3787754230 shutdown_connections 2024-04-20T22:26:43.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.470+0000 7f5902000700 1 --2- 172.21.15.62:0/3787754230 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f58fc0739a0 0x7f58fc10c5e0 unknown :-1 s=CLOSED pgs=1319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:26:43.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.470+0000 7f5902000700 1 --2- 172.21.15.62:0/3787754230 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f58fc073090 0x7f58fc073460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:26:43.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.470+0000 7f5902000700 1 -- 172.21.15.62:0/3787754230 >> 172.21.15.62:0/3787754230 conn(0x7f58fc0fbda0 msgr2=0x7f58fc0fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:26:43.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.470+0000 7f5902000700 1 -- 172.21.15.62:0/3787754230 shutdown_connections 2024-04-20T22:26:43.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.471+0000 7f5902000700 1 -- 172.21.15.62:0/3787754230 wait complete. 2024-04-20T22:26:43.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.472+0000 7f5902000700 1 Processor -- start 2024-04-20T22:26:43.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.472+0000 7f5902000700 1 -- start start 2024-04-20T22:26:43.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.472+0000 7f5902000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f58fc073090 0x7f58fc198220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:26:43.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.472+0000 7f5902000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f58fc0739a0 0x7f58fc198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:26:43.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.472+0000 7f5902000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58fc198db0 con 0x7f58fc073090 2024-04-20T22:26:43.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.472+0000 7f5902000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58fc198ef0 con 0x7f58fc0739a0 2024-04-20T22:26:43.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.472+0000 7f58fb400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f58fc073090 0x7f58fc198220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:26:43.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.472+0000 7f58fb400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f58fc073090 0x7f58fc198220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:57268/0 (socket says 172.21.15.62:57268) 2024-04-20T22:26:43.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.473+0000 7f58fb400700 1 -- 172.21.15.62:0/617371762 learned_addr learned my addr 172.21.15.62:0/617371762 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:26:43.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.473+0000 7f58fb400700 1 -- 172.21.15.62:0/617371762 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f58fc0739a0 msgr2=0x7f58fc198760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:26:43.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.473+0000 7f58fb400700 1 --2- 172.21.15.62:0/617371762 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f58fc0739a0 0x7f58fc198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:26:43.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.473+0000 7f58fb400700 1 -- 172.21.15.62:0/617371762 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58e80095f0 con 0x7f58fc073090 2024-04-20T22:26:43.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.473+0000 7f58fb400700 1 --2- 172.21.15.62:0/617371762 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f58fc073090 0x7f58fc198220 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7f58ec00c930 tx=0x7f58ec00ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:26:43.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.473+0000 7f58f3e00700 1 -- 172.21.15.62:0/617371762 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f58ec007ad0 con 0x7f58fc073090 2024-04-20T22:26:43.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.474+0000 7f58f3e00700 1 -- 172.21.15.62:0/617371762 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f58ec007c30 con 0x7f58fc073090 2024-04-20T22:26:43.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.474+0000 7f58f3e00700 1 -- 172.21.15.62:0/617371762 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f58ec018610 con 0x7f58fc073090 2024-04-20T22:26:43.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.474+0000 7f5902000700 1 -- 172.21.15.62:0/617371762 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58fc19cd40 con 0x7f58fc073090 2024-04-20T22:26:43.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.474+0000 7f5902000700 1 -- 172.21.15.62:0/617371762 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58fc19d240 con 0x7f58fc073090 2024-04-20T22:26:43.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.475+0000 7f5902000700 1 -- 172.21.15.62:0/617371762 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58fc109ce0 con 0x7f58fc073090 2024-04-20T22:26:43.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.476+0000 7f58f3e00700 1 -- 172.21.15.62:0/617371762 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f58ec00f450 con 0x7f58fc073090 2024-04-20T22:26:43.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.477+0000 7f58f3e00700 1 --2- 172.21.15.62:0/617371762 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f58e006ecc0 0x7f58e0071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:26:43.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.477+0000 7f58f3e00700 1 -- 172.21.15.62:0/617371762 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f58ec08f120 con 0x7f58fc073090 2024-04-20T22:26:43.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.477+0000 7f58faa00700 1 --2- 172.21.15.62:0/617371762 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f58e006ecc0 0x7f58e0071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:26:43.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.478+0000 7f58faa00700 1 --2- 172.21.15.62:0/617371762 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f58e006ecc0 0x7f58e0071170 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f58e8006010 tx=0x7f58e8000bc0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:26:43.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.482+0000 7f58f3e00700 1 -- 172.21.15.62:0/617371762 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f58ec057e30 con 0x7f58fc073090 2024-04-20T22:26:43.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.758+0000 7f5902000700 1 -- 172.21.15.62:0/617371762 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f58fc19d4c0 con 0x7f58e006ecc0 2024-04-20T22:26:43.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.761+0000 7f58f3e00700 1 -- 172.21.15.62:0/617371762 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f58fc19d4c0 con 0x7f58e006ecc0 2024-04-20T22:26:43.761 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:26:43.761 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T22:26:43.761 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T22:26:43.761 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-20T22:26:43.762 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T22:26:43.762 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr", 2024-04-20T22:26:43.762 INFO:teuthology.orchestra.run.smithi062.stdout: "osd", 2024-04-20T22:26:43.762 INFO:teuthology.orchestra.run.smithi062.stdout: "mds", 2024-04-20T22:26:43.762 INFO:teuthology.orchestra.run.smithi062.stdout: "crash", 2024-04-20T22:26:43.762 INFO:teuthology.orchestra.run.smithi062.stdout: "mon" 2024-04-20T22:26:43.762 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T22:26:43.762 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "16/23 daemons upgraded", 2024-04-20T22:26:43.762 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading mds daemons", 2024-04-20T22:26:43.762 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T22:26:43.762 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:26:43.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.764+0000 7f5902000700 1 -- 172.21.15.62:0/617371762 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f58e006ecc0 msgr2=0x7f58e0071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:26:43.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.764+0000 7f5902000700 1 --2- 172.21.15.62:0/617371762 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f58e006ecc0 0x7f58e0071170 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f58e8006010 tx=0x7f58e8000bc0 comp rx=0 tx=0).stop 2024-04-20T22:26:43.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.764+0000 7f5902000700 1 -- 172.21.15.62:0/617371762 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f58fc073090 msgr2=0x7f58fc198220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:26:43.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.764+0000 7f5902000700 1 --2- 172.21.15.62:0/617371762 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f58fc073090 0x7f58fc198220 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7f58ec00c930 tx=0x7f58ec00ccf0 comp rx=0 tx=0).stop 2024-04-20T22:26:43.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.764+0000 7f5902000700 1 -- 172.21.15.62:0/617371762 shutdown_connections 2024-04-20T22:26:43.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.764+0000 7f5902000700 1 --2- 172.21.15.62:0/617371762 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f58e006ecc0 0x7f58e0071170 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:26:43.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.764+0000 7f5902000700 1 --2- 172.21.15.62:0/617371762 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f58fc073090 0x7f58fc198220 unknown :-1 s=CLOSED pgs=1320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:26:43.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.764+0000 7f5902000700 1 --2- 172.21.15.62:0/617371762 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f58fc0739a0 0x7f58fc198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:26:43.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.764+0000 7f5902000700 1 -- 172.21.15.62:0/617371762 >> 172.21.15.62:0/617371762 conn(0x7f58fc0fbda0 msgr2=0x7f58fc106e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:26:43.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.764+0000 7f5902000700 1 -- 172.21.15.62:0/617371762 shutdown_connections 2024-04-20T22:26:43.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.765+0000 7f5902000700 1 -- 172.21.15.62:0/617371762 wait complete. 2024-04-20T22:26:43.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.923+0000 7f7b42c00700 1 -- 172.21.15.62:0/3922295270 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7b3c0ff350 msgr2=0x7f7b3c0ff720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:26:43.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.923+0000 7f7b42c00700 1 --2- 172.21.15.62:0/3922295270 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7b3c0ff350 0x7f7b3c0ff720 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f7b2c009910 tx=0x7f7b2c009c20 comp rx=0 tx=0).stop 2024-04-20T22:26:43.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.923+0000 7f7b42c00700 1 -- 172.21.15.62:0/3922295270 shutdown_connections 2024-04-20T22:26:43.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.923+0000 7f7b42c00700 1 --2- 172.21.15.62:0/3922295270 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7b3c0ffc60 0x7f7b3c10a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:26:43.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.923+0000 7f7b42c00700 1 --2- 172.21.15.62:0/3922295270 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7b3c0ff350 0x7f7b3c0ff720 unknown :-1 s=CLOSED pgs=1321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:26:43.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.923+0000 7f7b42c00700 1 -- 172.21.15.62:0/3922295270 >> 172.21.15.62:0/3922295270 conn(0x7f7b3c0748b0 msgr2=0x7f7b3c074cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:26:43.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.923+0000 7f7b42c00700 1 -- 172.21.15.62:0/3922295270 shutdown_connections 2024-04-20T22:26:43.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.924+0000 7f7b42c00700 1 -- 172.21.15.62:0/3922295270 wait complete. 2024-04-20T22:26:43.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.925+0000 7f7b42c00700 1 Processor -- start 2024-04-20T22:26:43.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.925+0000 7f7b42c00700 1 -- start start 2024-04-20T22:26:43.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.925+0000 7f7b42c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7b3c0ff350 0x7f7b3c100530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:26:43.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.925+0000 7f7b42c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7b3c0ffc60 0x7f7b3c100a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:26:43.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.925+0000 7f7b42c00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b3c100fb0 con 0x7f7b3c0ff350 2024-04-20T22:26:43.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.925+0000 7f7b42c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b3c1010f0 con 0x7f7b3c0ffc60 2024-04-20T22:26:43.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.925+0000 7f7b3be00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7b3c0ff350 0x7f7b3c100530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:26:43.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.925+0000 7f7b3be00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7b3c0ff350 0x7f7b3c100530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:57282/0 (socket says 172.21.15.62:57282) 2024-04-20T22:26:43.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.925+0000 7f7b3be00700 1 -- 172.21.15.62:0/402786915 learned_addr learned my addr 172.21.15.62:0/402786915 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:26:43.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.926+0000 7f7b3b400700 1 --2- 172.21.15.62:0/402786915 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7b3c0ffc60 0x7f7b3c100a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:26:43.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.926+0000 7f7b3be00700 1 -- 172.21.15.62:0/402786915 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7b3c0ffc60 msgr2=0x7f7b3c100a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:26:43.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.926+0000 7f7b3be00700 1 --2- 172.21.15.62:0/402786915 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7b3c0ffc60 0x7f7b3c100a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:26:43.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.926+0000 7f7b3be00700 1 -- 172.21.15.62:0/402786915 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b2c0095f0 con 0x7f7b3c0ff350 2024-04-20T22:26:43.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.926+0000 7f7b3be00700 1 --2- 172.21.15.62:0/402786915 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7b3c0ff350 0x7f7b3c100530 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f7b2c00bda0 tx=0x7f7b2c00be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:26:43.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.926+0000 7f7b38c00700 1 -- 172.21.15.62:0/402786915 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b2c01d070 con 0x7f7b3c0ff350 2024-04-20T22:26:43.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.927+0000 7f7b38c00700 1 -- 172.21.15.62:0/402786915 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7b2c004fb0 con 0x7f7b3c0ff350 2024-04-20T22:26:43.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.927+0000 7f7b38c00700 1 -- 172.21.15.62:0/402786915 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b2c00f460 con 0x7f7b3c0ff350 2024-04-20T22:26:43.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.927+0000 7f7b42c00700 1 -- 172.21.15.62:0/402786915 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b3c101370 con 0x7f7b3c0ff350 2024-04-20T22:26:43.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.927+0000 7f7b42c00700 1 -- 172.21.15.62:0/402786915 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b3c1a2470 con 0x7f7b3c0ff350 2024-04-20T22:26:43.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.928+0000 7f7b42c00700 1 -- 172.21.15.62:0/402786915 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b3c107dd0 con 0x7f7b3c0ff350 2024-04-20T22:26:43.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.929+0000 7f7b38c00700 1 -- 172.21.15.62:0/402786915 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f7b2c022800 con 0x7f7b3c0ff350 2024-04-20T22:26:43.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.930+0000 7f7b38c00700 1 --2- 172.21.15.62:0/402786915 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7b2006ecc0 0x7f7b20071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:26:43.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.930+0000 7f7b38c00700 1 -- 172.21.15.62:0/402786915 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f7b2c090420 con 0x7f7b3c0ff350 2024-04-20T22:26:43.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.930+0000 7f7b3b400700 1 --2- 172.21.15.62:0/402786915 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7b2006ecc0 0x7f7b20071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:26:43.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.931+0000 7f7b3b400700 1 --2- 172.21.15.62:0/402786915 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7b2006ecc0 0x7f7b20071170 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f7b28006fd0 tx=0x7f7b28008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:26:43.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:43.935+0000 7f7b38c00700 1 -- 172.21.15.62:0/402786915 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f7b2c059300 con 0x7f7b3c0ff350 2024-04-20T22:26:44.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:44.265+0000 7f7b42c00700 1 -- 172.21.15.62:0/402786915 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f7b3c02cc30 con 0x7f7b3c0ff350 2024-04-20T22:26:44.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:44.266+0000 7f7b38c00700 1 -- 172.21.15.62:0/402786915 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7f7b2c05c9c0 con 0x7f7b3c0ff350 2024-04-20T22:26:44.266 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-20T22:26:44.266 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-20T22:26:44.267 INFO:teuthology.orchestra.run.smithi062.stdout: host smithi189 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi189/config 2024-04-20T22:26:44.267 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-20T22:26:44.267 INFO:teuthology.orchestra.run.smithi062.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi189 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp11670v2q:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:26:44.267 INFO:teuthology.orchestra.run.smithi062.stdout:Traceback (most recent call last): 2024-04-20T22:26:44.267 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-20T22:26:44.267 INFO:teuthology.orchestra.run.smithi062.stdout: return _run_code(code, main_globals, None, 2024-04-20T22:26:44.267 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-20T22:26:44.267 INFO:teuthology.orchestra.run.smithi062.stdout: exec(code, run_globals) 2024-04-20T22:26:44.267 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5618, in 2024-04-20T22:26:44.267 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5606, in main 2024-04-20T22:26:44.267 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 408, in _infer_config 2024-04-20T22:26:44.267 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 323, in _infer_fsid 2024-04-20T22:26:44.267 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 436, in _infer_image 2024-04-20T22:26:44.267 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 310, in _validate_fsid 2024-04-20T22:26:44.268 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 3359, in command_ceph_volume 2024-04-20T22:26:44.268 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-20T22:26:44.268 INFO:teuthology.orchestra.run.smithi062.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi189 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp11670v2q:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:26:44.268 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-20T22:26:44.268 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is degraded 2024-04-20T22:26:44.268 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T22:26:44.268 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T22:26:44.268 INFO:teuthology.orchestra.run.smithi062.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-20T22:26:44.268 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-20T22:26:44.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:44.269+0000 7f7b42c00700 1 -- 172.21.15.62:0/402786915 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7b2006ecc0 msgr2=0x7f7b20071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:26:44.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:44.269+0000 7f7b42c00700 1 --2- 172.21.15.62:0/402786915 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7b2006ecc0 0x7f7b20071170 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f7b28006fd0 tx=0x7f7b28008040 comp rx=0 tx=0).stop 2024-04-20T22:26:44.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:44.269+0000 7f7b42c00700 1 -- 172.21.15.62:0/402786915 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7b3c0ff350 msgr2=0x7f7b3c100530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:26:44.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:44.269+0000 7f7b42c00700 1 --2- 172.21.15.62:0/402786915 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7b3c0ff350 0x7f7b3c100530 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f7b2c00bda0 tx=0x7f7b2c00be80 comp rx=0 tx=0).stop 2024-04-20T22:26:44.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:44.270+0000 7f7b42c00700 1 -- 172.21.15.62:0/402786915 shutdown_connections 2024-04-20T22:26:44.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:44.270+0000 7f7b42c00700 1 --2- 172.21.15.62:0/402786915 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7b2006ecc0 0x7f7b20071170 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:26:44.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:44.270+0000 7f7b42c00700 1 --2- 172.21.15.62:0/402786915 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7b3c0ff350 0x7f7b3c100530 unknown :-1 s=CLOSED pgs=1322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:26:44.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:44.270+0000 7f7b42c00700 1 --2- 172.21.15.62:0/402786915 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7b3c0ffc60 0x7f7b3c100a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:26:44.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:44.270+0000 7f7b42c00700 1 -- 172.21.15.62:0/402786915 >> 172.21.15.62:0/402786915 conn(0x7f7b3c0748b0 msgr2=0x7f7b3c104f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:26:44.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:44.270+0000 7f7b42c00700 1 -- 172.21.15.62:0/402786915 shutdown_connections 2024-04-20T22:26:44.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:26:44.270+0000 7f7b42c00700 1 -- 172.21.15.62:0/402786915 wait complete. 2024-04-20T22:26:44.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:26:44 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/2783381445' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:26:44.538 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:26:44 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/2783381445' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:26:45.389 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:26:45 smithi189 ceph-mon[57022]: pgmap v1546: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:26:45.390 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:26:45 smithi189 ceph-mon[57022]: from='client.36712 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:26:45.390 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:26:45 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/402786915' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:26:45.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:26:45 smithi062 ceph-mon[76322]: pgmap v1546: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:26:45.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:26:45 smithi062 ceph-mon[76322]: from='client.36712 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:26:45.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:26:45 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/402786915' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:26:47.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:26:47 smithi189 ceph-mon[57022]: pgmap v1547: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:26:47.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:26:47 smithi062 ceph-mon[76322]: pgmap v1547: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:26:49.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:26:49 smithi189 ceph-mon[57022]: pgmap v1548: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:26:49.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:26:49 smithi062 ceph-mon[76322]: pgmap v1548: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:26:50.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:26:50 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:26:50.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:26:50 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:26:51.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:26:51 smithi189 ceph-mon[57022]: pgmap v1549: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:26:51.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:26:51 smithi062 ceph-mon[76322]: pgmap v1549: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:26:53.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:26:53 smithi189 ceph-mon[57022]: pgmap v1550: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:26:53.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:26:53 smithi062 ceph-mon[76322]: pgmap v1550: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:26:55.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:26:55 smithi189 ceph-mon[57022]: pgmap v1551: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:26:55.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:26:55 smithi062 ceph-mon[76322]: pgmap v1551: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:26:57.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:26:57 smithi062 ceph-mon[76322]: pgmap v1552: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:26:57.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:26:57 smithi189 ceph-mon[57022]: pgmap v1552: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:26:59.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:26:59 smithi189 ceph-mon[57022]: pgmap v1553: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:26:59.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:26:59 smithi062 ceph-mon[76322]: pgmap v1553: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:27:01.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:27:01 smithi189 ceph-mon[57022]: pgmap v1554: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:27:01.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:27:01 smithi062 ceph-mon[76322]: pgmap v1554: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:27:03.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:27:03 smithi189 ceph-mon[57022]: pgmap v1555: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:27:03.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:27:03 smithi062 ceph-mon[76322]: pgmap v1555: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:27:05.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:27:05 smithi189 ceph-mon[57022]: pgmap v1556: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:27:05.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:27:05 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:27:05.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:27:05 smithi062 ceph-mon[76322]: pgmap v1556: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:27:05.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:27:05 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:27:07.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:27:07 smithi189 ceph-mon[57022]: pgmap v1557: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:27:07.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:27:07 smithi062 ceph-mon[76322]: pgmap v1557: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:27:09.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:27:09 smithi189 ceph-mon[57022]: pgmap v1558: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:27:09.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:27:09 smithi062 ceph-mon[76322]: pgmap v1558: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:27:11.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:27:11 smithi189 ceph-mon[57022]: pgmap v1559: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:27:11.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:27:11 smithi062 ceph-mon[76322]: pgmap v1559: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:27:13.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:27:13 smithi189 ceph-mon[57022]: pgmap v1560: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:27:13.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:27:13 smithi062 ceph-mon[76322]: pgmap v1560: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:27:14.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:14.449+0000 7fcc9d800700 1 -- 172.21.15.62:0/3932673779 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcc98073070 msgr2=0x7fcc98073440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:27:14.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:14.449+0000 7fcc9d800700 1 --2- 172.21.15.62:0/3932673779 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcc98073070 0x7fcc98073440 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7fcc88009940 tx=0x7fcc88009c50 comp rx=0 tx=0).stop 2024-04-20T22:27:14.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:14.449+0000 7fcc9d800700 1 -- 172.21.15.62:0/3932673779 shutdown_connections 2024-04-20T22:27:14.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:14.449+0000 7fcc9d800700 1 --2- 172.21.15.62:0/3932673779 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc98073980 0x7fcc9810c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:27:14.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:14.449+0000 7fcc9d800700 1 --2- 172.21.15.62:0/3932673779 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcc98073070 0x7fcc98073440 unknown :-1 s=CLOSED pgs=1323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:27:14.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:14.449+0000 7fcc9d800700 1 -- 172.21.15.62:0/3932673779 >> 172.21.15.62:0/3932673779 conn(0x7fcc980fbda0 msgr2=0x7fcc980fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:27:14.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:14.449+0000 7fcc9d800700 1 -- 172.21.15.62:0/3932673779 shutdown_connections 2024-04-20T22:27:14.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:14.449+0000 7fcc9d800700 1 -- 172.21.15.62:0/3932673779 wait complete. 2024-04-20T22:27:14.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:14.451+0000 7fcc9d800700 1 Processor -- start 2024-04-20T22:27:14.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:14.451+0000 7fcc9d800700 1 -- start start 2024-04-20T22:27:14.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:14.451+0000 7fcc9d800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcc98073070 0x7fcc981981d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:27:14.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:14.451+0000 7fcc9d800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc98073980 0x7fcc98198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:27:14.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:14.451+0000 7fcc9d800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc98198df0 con 0x7fcc98073070 2024-04-20T22:27:14.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:14.451+0000 7fcc9d800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc9819cb80 con 0x7fcc98073980 2024-04-20T22:27:14.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:14.452+0000 7fcc96a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcc98073070 0x7fcc981981d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:27:14.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:14.452+0000 7fcc96a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcc98073070 0x7fcc981981d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60016/0 (socket says 172.21.15.62:60016) 2024-04-20T22:27:14.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:14.452+0000 7fcc96a00700 1 -- 172.21.15.62:0/2792835806 learned_addr learned my addr 172.21.15.62:0/2792835806 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:27:14.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:14.452+0000 7fcc8e000700 1 --2- 172.21.15.62:0/2792835806 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc98073980 0x7fcc98198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:27:14.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:14.452+0000 7fcc96a00700 1 -- 172.21.15.62:0/2792835806 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc98073980 msgr2=0x7fcc98198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:27:14.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:14.452+0000 7fcc96a00700 1 --2- 172.21.15.62:0/2792835806 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc98073980 0x7fcc98198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:27:14.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:14.452+0000 7fcc96a00700 1 -- 172.21.15.62:0/2792835806 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc880095f0 con 0x7fcc98073070 2024-04-20T22:27:14.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:14.452+0000 7fcc8e000700 1 --2- 172.21.15.62:0/2792835806 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc98073980 0x7fcc98198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-20T22:27:14.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:14.453+0000 7fcc96a00700 1 --2- 172.21.15.62:0/2792835806 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcc98073070 0x7fcc981981d0 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7fcc880055f0 tx=0x7fcc880056d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:27:14.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:14.453+0000 7fcc8fe00700 1 -- 172.21.15.62:0/2792835806 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcc8801d070 con 0x7fcc98073070 2024-04-20T22:27:14.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:14.453+0000 7fcc8fe00700 1 -- 172.21.15.62:0/2792835806 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcc8800f460 con 0x7fcc98073070 2024-04-20T22:27:14.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:14.453+0000 7fcc9d800700 1 -- 172.21.15.62:0/2792835806 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc9819ce00 con 0x7fcc98073070 2024-04-20T22:27:14.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:14.453+0000 7fcc8fe00700 1 -- 172.21.15.62:0/2792835806 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcc88021640 con 0x7fcc98073070 2024-04-20T22:27:14.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:14.454+0000 7fcc9d800700 1 -- 172.21.15.62:0/2792835806 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc9819d2f0 con 0x7fcc98073070 2024-04-20T22:27:14.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:14.455+0000 7fcc9d800700 1 -- 172.21.15.62:0/2792835806 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc98109d00 con 0x7fcc98073070 2024-04-20T22:27:14.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:14.456+0000 7fcc8fe00700 1 -- 172.21.15.62:0/2792835806 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fcc8800fa20 con 0x7fcc98073070 2024-04-20T22:27:14.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:14.457+0000 7fcc8fe00700 1 --2- 172.21.15.62:0/2792835806 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fcc7c06ecc0 0x7fcc7c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:27:14.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:14.457+0000 7fcc8fe00700 1 -- 172.21.15.62:0/2792835806 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fcc88090160 con 0x7fcc98073070 2024-04-20T22:27:14.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:14.457+0000 7fcc8e000700 1 --2- 172.21.15.62:0/2792835806 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fcc7c06ecc0 0x7fcc7c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:27:14.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:14.458+0000 7fcc8e000700 1 --2- 172.21.15.62:0/2792835806 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fcc7c06ecc0 0x7fcc7c071170 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fcc80005950 tx=0x7fcc800058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:27:14.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:14.462+0000 7fcc8fe00700 1 -- 172.21.15.62:0/2792835806 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fcc88058e70 con 0x7fcc98073070 2024-04-20T22:27:14.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:14.724+0000 7fcc9d800700 1 -- 172.21.15.62:0/2792835806 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fcc9819d510 con 0x7fcc7c06ecc0 2024-04-20T22:27:14.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:14.727+0000 7fcc8fe00700 1 -- 172.21.15.62:0/2792835806 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fcc9819d510 con 0x7fcc7c06ecc0 2024-04-20T22:27:14.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:14.730+0000 7fcc9d800700 1 -- 172.21.15.62:0/2792835806 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fcc7c06ecc0 msgr2=0x7fcc7c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:27:14.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:14.730+0000 7fcc9d800700 1 --2- 172.21.15.62:0/2792835806 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fcc7c06ecc0 0x7fcc7c071170 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fcc80005950 tx=0x7fcc800058e0 comp rx=0 tx=0).stop 2024-04-20T22:27:14.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:14.730+0000 7fcc9d800700 1 -- 172.21.15.62:0/2792835806 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcc98073070 msgr2=0x7fcc981981d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:27:14.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:14.730+0000 7fcc9d800700 1 --2- 172.21.15.62:0/2792835806 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcc98073070 0x7fcc981981d0 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7fcc880055f0 tx=0x7fcc880056d0 comp rx=0 tx=0).stop 2024-04-20T22:27:14.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:14.730+0000 7fcc9d800700 1 -- 172.21.15.62:0/2792835806 shutdown_connections 2024-04-20T22:27:14.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:14.730+0000 7fcc9d800700 1 --2- 172.21.15.62:0/2792835806 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fcc7c06ecc0 0x7fcc7c071170 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:27:14.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:14.730+0000 7fcc9d800700 1 --2- 172.21.15.62:0/2792835806 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcc98073070 0x7fcc981981d0 unknown :-1 s=CLOSED pgs=1324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:27:14.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:14.730+0000 7fcc9d800700 1 --2- 172.21.15.62:0/2792835806 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc98073980 0x7fcc98198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:27:14.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:14.730+0000 7fcc9d800700 1 -- 172.21.15.62:0/2792835806 >> 172.21.15.62:0/2792835806 conn(0x7fcc980fbda0 msgr2=0x7fcc98106e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:27:14.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:14.730+0000 7fcc9d800700 1 -- 172.21.15.62:0/2792835806 shutdown_connections 2024-04-20T22:27:14.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:14.730+0000 7fcc9d800700 1 -- 172.21.15.62:0/2792835806 wait complete. 2024-04-20T22:27:14.747 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T22:27:14.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:14.897+0000 7fb3a4200700 1 -- 172.21.15.62:0/624326415 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb39c108470 msgr2=0x7fb39c108840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:27:14.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:14.897+0000 7fb3a4200700 1 --2- 172.21.15.62:0/624326415 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb39c108470 0x7fb39c108840 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7fb390009910 tx=0x7fb390009c20 comp rx=0 tx=0).stop 2024-04-20T22:27:14.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:14.898+0000 7fb3a4200700 1 -- 172.21.15.62:0/624326415 shutdown_connections 2024-04-20T22:27:14.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:14.898+0000 7fb3a4200700 1 --2- 172.21.15.62:0/624326415 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb39c102470 0x7fb39c1028e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:27:14.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:14.898+0000 7fb3a4200700 1 --2- 172.21.15.62:0/624326415 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb39c108470 0x7fb39c108840 unknown :-1 s=CLOSED pgs=1325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:27:14.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:14.898+0000 7fb3a4200700 1 -- 172.21.15.62:0/624326415 >> 172.21.15.62:0/624326415 conn(0x7fb39c0fdf90 msgr2=0x7fb39c1003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:27:14.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:14.898+0000 7fb3a4200700 1 -- 172.21.15.62:0/624326415 shutdown_connections 2024-04-20T22:27:14.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:14.898+0000 7fb3a4200700 1 -- 172.21.15.62:0/624326415 wait complete. 2024-04-20T22:27:14.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:14.899+0000 7fb3a4200700 1 Processor -- start 2024-04-20T22:27:14.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:14.899+0000 7fb3a4200700 1 -- start start 2024-04-20T22:27:14.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:14.899+0000 7fb3a4200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb39c102470 0x7fb39c1981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:27:14.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:14.899+0000 7fb3a4200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb39c108470 0x7fb39c198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:27:14.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:14.899+0000 7fb3a4200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb39c198e10 con 0x7fb39c102470 2024-04-20T22:27:14.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:14.899+0000 7fb3a4200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb39c19cba0 con 0x7fb39c108470 2024-04-20T22:27:14.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:14.900+0000 7fb3a1800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb39c102470 0x7fb39c1981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:27:14.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:14.900+0000 7fb3a1800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb39c102470 0x7fb39c1981f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60034/0 (socket says 172.21.15.62:60034) 2024-04-20T22:27:14.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:14.900+0000 7fb3a1800700 1 -- 172.21.15.62:0/869498541 learned_addr learned my addr 172.21.15.62:0/869498541 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:27:14.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:14.900+0000 7fb3a0e00700 1 --2- 172.21.15.62:0/869498541 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb39c108470 0x7fb39c198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:27:14.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:14.900+0000 7fb3a1800700 1 -- 172.21.15.62:0/869498541 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb39c108470 msgr2=0x7fb39c198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:27:14.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:14.900+0000 7fb3a1800700 1 --2- 172.21.15.62:0/869498541 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb39c108470 0x7fb39c198730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:27:14.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:14.900+0000 7fb3a1800700 1 -- 172.21.15.62:0/869498541 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3900095f0 con 0x7fb39c102470 2024-04-20T22:27:14.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:14.900+0000 7fb3a0e00700 1 --2- 172.21.15.62:0/869498541 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb39c108470 0x7fb39c198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T22:27:14.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:14.901+0000 7fb3a1800700 1 --2- 172.21.15.62:0/869498541 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb39c102470 0x7fb39c1981f0 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7fb39000b5c0 tx=0x7fb39000be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:27:14.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:14.901+0000 7fb39a000700 1 -- 172.21.15.62:0/869498541 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb39001d070 con 0x7fb39c102470 2024-04-20T22:27:14.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:14.901+0000 7fb39a000700 1 -- 172.21.15.62:0/869498541 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb39000f460 con 0x7fb39c102470 2024-04-20T22:27:14.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:14.901+0000 7fb3a4200700 1 -- 172.21.15.62:0/869498541 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb39c19ce20 con 0x7fb39c102470 2024-04-20T22:27:14.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:14.901+0000 7fb39a000700 1 -- 172.21.15.62:0/869498541 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb390017610 con 0x7fb39c102470 2024-04-20T22:27:14.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:14.901+0000 7fb3a4200700 1 -- 172.21.15.62:0/869498541 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb39c19d310 con 0x7fb39c102470 2024-04-20T22:27:14.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:14.902+0000 7fb3a4200700 1 -- 172.21.15.62:0/869498541 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb39c10a9e0 con 0x7fb39c102470 2024-04-20T22:27:14.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:14.903+0000 7fb39a000700 1 -- 172.21.15.62:0/869498541 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fb39000f5d0 con 0x7fb39c102470 2024-04-20T22:27:14.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:14.904+0000 7fb39a000700 1 --2- 172.21.15.62:0/869498541 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb38406ecc0 0x7fb384071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:27:14.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:14.904+0000 7fb39a000700 1 -- 172.21.15.62:0/869498541 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fb390090510 con 0x7fb39c102470 2024-04-20T22:27:14.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:14.905+0000 7fb3a0e00700 1 --2- 172.21.15.62:0/869498541 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb38406ecc0 0x7fb384071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:27:14.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:14.905+0000 7fb3a0e00700 1 --2- 172.21.15.62:0/869498541 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb38406ecc0 0x7fb384071170 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fb388005fd0 tx=0x7fb388005dc0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:27:14.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:14.909+0000 7fb39a000700 1 -- 172.21.15.62:0/869498541 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fb390059220 con 0x7fb39c102470 2024-04-20T22:27:15.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:15.170+0000 7fb3a4200700 1 -- 172.21.15.62:0/869498541 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb39c199690 con 0x7fb38406ecc0 2024-04-20T22:27:15.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:15.173+0000 7fb39a000700 1 -- 172.21.15.62:0/869498541 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fb3880086c0 con 0x7fb38406ecc0 2024-04-20T22:27:15.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:15.176+0000 7fb3a4200700 1 -- 172.21.15.62:0/869498541 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb38406ecc0 msgr2=0x7fb384071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:27:15.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:15.176+0000 7fb3a4200700 1 --2- 172.21.15.62:0/869498541 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb38406ecc0 0x7fb384071170 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fb388005fd0 tx=0x7fb388005dc0 comp rx=0 tx=0).stop 2024-04-20T22:27:15.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:15.176+0000 7fb3a4200700 1 -- 172.21.15.62:0/869498541 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb39c102470 msgr2=0x7fb39c1981f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:27:15.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:15.176+0000 7fb3a4200700 1 --2- 172.21.15.62:0/869498541 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb39c102470 0x7fb39c1981f0 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7fb39000b5c0 tx=0x7fb39000be00 comp rx=0 tx=0).stop 2024-04-20T22:27:15.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:15.176+0000 7fb3a4200700 1 -- 172.21.15.62:0/869498541 shutdown_connections 2024-04-20T22:27:15.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:15.176+0000 7fb3a4200700 1 --2- 172.21.15.62:0/869498541 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb38406ecc0 0x7fb384071170 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:27:15.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:15.176+0000 7fb3a4200700 1 --2- 172.21.15.62:0/869498541 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb39c102470 0x7fb39c1981f0 unknown :-1 s=CLOSED pgs=1326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:27:15.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:15.176+0000 7fb3a4200700 1 --2- 172.21.15.62:0/869498541 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb39c108470 0x7fb39c198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:27:15.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:15.176+0000 7fb3a4200700 1 -- 172.21.15.62:0/869498541 >> 172.21.15.62:0/869498541 conn(0x7fb39c0fdf90 msgr2=0x7fb39c1002c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:27:15.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:15.176+0000 7fb3a4200700 1 -- 172.21.15.62:0/869498541 shutdown_connections 2024-04-20T22:27:15.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:15.176+0000 7fb3a4200700 1 -- 172.21.15.62:0/869498541 wait complete. 2024-04-20T22:27:15.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:15.344+0000 7f9ab4200700 1 -- 172.21.15.62:0/3151436619 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9aac10c8f0 msgr2=0x7f9aac10ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:27:15.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:15.344+0000 7f9ab4200700 1 --2- 172.21.15.62:0/3151436619 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9aac10c8f0 0x7f9aac10ccc0 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7f9aa0009910 tx=0x7f9aa0009c20 comp rx=0 tx=0).stop 2024-04-20T22:27:15.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:15.344+0000 7f9ab4200700 1 -- 172.21.15.62:0/3151436619 shutdown_connections 2024-04-20T22:27:15.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:15.344+0000 7f9ab4200700 1 --2- 172.21.15.62:0/3151436619 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9aac106b80 0x7f9aac106ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:27:15.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:15.344+0000 7f9ab4200700 1 --2- 172.21.15.62:0/3151436619 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9aac10c8f0 0x7f9aac10ccc0 unknown :-1 s=CLOSED pgs=1327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:27:15.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:15.344+0000 7f9ab4200700 1 -- 172.21.15.62:0/3151436619 >> 172.21.15.62:0/3151436619 conn(0x7f9aac076020 msgr2=0x7f9aac078430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:27:15.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:15.344+0000 7f9ab4200700 1 -- 172.21.15.62:0/3151436619 shutdown_connections 2024-04-20T22:27:15.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:15.344+0000 7f9ab4200700 1 -- 172.21.15.62:0/3151436619 wait complete. 2024-04-20T22:27:15.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:15.345+0000 7f9ab4200700 1 Processor -- start 2024-04-20T22:27:15.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:15.346+0000 7f9ab4200700 1 -- start start 2024-04-20T22:27:15.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:15.346+0000 7f9ab4200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9aac106b80 0x7f9aac19c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:27:15.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:15.346+0000 7f9ab4200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9aac10c8f0 0x7f9aac19c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:27:15.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:15.346+0000 7f9ab4200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9aac19cff0 con 0x7f9aac10c8f0 2024-04-20T22:27:15.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:15.346+0000 7f9ab4200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9aac1a0d80 con 0x7f9aac106b80 2024-04-20T22:27:15.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:15.346+0000 7f9ab0e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9aac10c8f0 0x7f9aac19c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:27:15.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:15.346+0000 7f9ab0e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9aac10c8f0 0x7f9aac19c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60054/0 (socket says 172.21.15.62:60054) 2024-04-20T22:27:15.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:15.346+0000 7f9ab0e00700 1 -- 172.21.15.62:0/1690146986 learned_addr learned my addr 172.21.15.62:0/1690146986 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:27:15.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:15.347+0000 7f9ab1800700 1 --2- 172.21.15.62:0/1690146986 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9aac106b80 0x7f9aac19c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:27:15.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:15.347+0000 7f9ab0e00700 1 -- 172.21.15.62:0/1690146986 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9aac106b80 msgr2=0x7f9aac19c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:27:15.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:15.347+0000 7f9ab0e00700 1 --2- 172.21.15.62:0/1690146986 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9aac106b80 0x7f9aac19c3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:27:15.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:15.347+0000 7f9ab0e00700 1 -- 172.21.15.62:0/1690146986 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9aa00095f0 con 0x7f9aac10c8f0 2024-04-20T22:27:15.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:15.347+0000 7f9ab1800700 1 --2- 172.21.15.62:0/1690146986 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9aac106b80 0x7f9aac19c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T22:27:15.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:15.347+0000 7f9ab0e00700 1 --2- 172.21.15.62:0/1690146986 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9aac10c8f0 0x7f9aac19c910 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f9a9800d900 tx=0x7f9a9800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:27:15.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:15.347+0000 7f9aaa000700 1 -- 172.21.15.62:0/1690146986 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9a980041f0 con 0x7f9aac10c8f0 2024-04-20T22:27:15.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:15.348+0000 7f9aaa000700 1 -- 172.21.15.62:0/1690146986 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9a98004cf0 con 0x7f9aac10c8f0 2024-04-20T22:27:15.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:15.348+0000 7f9ab4200700 1 -- 172.21.15.62:0/1690146986 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9aac1a1060 con 0x7f9aac10c8f0 2024-04-20T22:27:15.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:15.348+0000 7f9aaa000700 1 -- 172.21.15.62:0/1690146986 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9a9800b750 con 0x7f9aac10c8f0 2024-04-20T22:27:15.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:15.348+0000 7f9ab4200700 1 -- 172.21.15.62:0/1690146986 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9aac1a15b0 con 0x7f9aac10c8f0 2024-04-20T22:27:15.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:15.349+0000 7f9ab4200700 1 -- 172.21.15.62:0/1690146986 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9aac04f2a0 con 0x7f9aac10c8f0 2024-04-20T22:27:15.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:15.350+0000 7f9aaa000700 1 -- 172.21.15.62:0/1690146986 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f9a98004350 con 0x7f9aac10c8f0 2024-04-20T22:27:15.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:15.351+0000 7f9aaa000700 1 --2- 172.21.15.62:0/1690146986 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9a9406ecc0 0x7f9a94071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:27:15.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:15.351+0000 7f9aaa000700 1 -- 172.21.15.62:0/1690146986 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f9a9808f0b0 con 0x7f9aac10c8f0 2024-04-20T22:27:15.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:15.351+0000 7f9ab1800700 1 --2- 172.21.15.62:0/1690146986 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9a9406ecc0 0x7f9a94071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:27:15.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:15.352+0000 7f9ab1800700 1 --2- 172.21.15.62:0/1690146986 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9a9406ecc0 0x7f9a94071170 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f9aa000b5c0 tx=0x7f9aa00053a0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:27:15.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:15.356+0000 7f9aaa000700 1 -- 172.21.15.62:0/1690146986 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f9a98057dc0 con 0x7f9aac10c8f0 2024-04-20T22:27:15.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:27:15 smithi189 ceph-mon[57022]: pgmap v1561: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:27:15.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:27:15 smithi062 ceph-mon[76322]: pgmap v1561: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:27:15.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:15.602+0000 7f9ab4200700 1 -- 172.21.15.62:0/1690146986 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f9aac1a1a90 con 0x7f9a9406ecc0 2024-04-20T22:27:15.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:15.640+0000 7f9aaa000700 1 -- 172.21.15.62:0/1690146986 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f9aac1a1a90 con 0x7f9a9406ecc0 2024-04-20T22:27:15.642 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T22:27:15.642 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (53m) 13m ago 66m 14.6M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T22:27:15.642 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (67m) 13m ago 67m 11.6M - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T22:27:15.642 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (65m) 13m ago 65m 11.2M - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T22:27:15.642 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (49m) 13m ago 67m 6660k - 19.0.0-3133-ga8c84f59 76bc6b91b726 2825d5ba11c6 2024-04-20T22:27:15.642 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (49m) 13m ago 65m 6665k - 19.0.0-3133-ga8c84f59 76bc6b91b726 20f746fa1bba 2024-04-20T22:27:15.642 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (52m) 13m ago 66m 75.5M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T22:27:15.642 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (44m) 13m ago 59m 22.9M - 19.0.0-3133-ga8c84f59 76bc6b91b726 de80bc90a683 2024-04-20T22:27:15.642 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (45m) 13m ago 59m 23.0M - 19.0.0-3133-ga8c84f59 76bc6b91b726 cd793b7ee98e 2024-04-20T22:27:15.642 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (29m) 13m ago 59m 18.5M - 19.0.0-3133-ga8c84f59 76bc6b91b726 0e7b0bc4d1ab 2024-04-20T22:27:15.642 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (14m) 13m ago 59m 11.4M - 19.0.0-3133-ga8c84f59 76bc6b91b726 0af732ac9e06 2024-04-20T22:27:15.642 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (55m) 13m ago 68m 517M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T22:27:15.642 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (54m) 13m ago 65m 460M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T22:27:15.643 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (50m) 13m ago 68m 86.3M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 019eb1479485 2024-04-20T22:27:15.643 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (50m) 13m ago 65m 58.0M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 55aac560aab6 2024-04-20T22:27:15.643 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (54m) 13m ago 67m 8766k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T22:27:15.643 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (54m) 13m ago 65m 8711k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T22:27:15.643 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (48m) 13m ago 64m 91.3M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 ee2bb901f26e 2024-04-20T22:27:15.643 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (47m) 13m ago 63m 94.9M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 9075152d4162 2024-04-20T22:27:15.643 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (47m) 13m ago 63m 78.5M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 2d4f3e38dc0e 2024-04-20T22:27:15.643 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (46m) 13m ago 62m 94.1M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 8f97b17ae0eb 2024-04-20T22:27:15.643 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (46m) 13m ago 62m 96.1M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 653d8079967d 2024-04-20T22:27:15.643 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (45m) 13m ago 61m 90.9M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 596dded14c12 2024-04-20T22:27:15.643 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (53m) 13m ago 66m 90.4M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T22:27:15.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:15.643+0000 7f9ab4200700 1 -- 172.21.15.62:0/1690146986 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9a9406ecc0 msgr2=0x7f9a94071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:27:15.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:15.643+0000 7f9ab4200700 1 --2- 172.21.15.62:0/1690146986 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9a9406ecc0 0x7f9a94071170 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f9aa000b5c0 tx=0x7f9aa00053a0 comp rx=0 tx=0).stop 2024-04-20T22:27:15.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:15.644+0000 7f9ab4200700 1 -- 172.21.15.62:0/1690146986 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9aac10c8f0 msgr2=0x7f9aac19c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:27:15.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:15.644+0000 7f9ab4200700 1 --2- 172.21.15.62:0/1690146986 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9aac10c8f0 0x7f9aac19c910 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f9a9800d900 tx=0x7f9a9800dcc0 comp rx=0 tx=0).stop 2024-04-20T22:27:15.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:15.644+0000 7f9ab4200700 1 -- 172.21.15.62:0/1690146986 shutdown_connections 2024-04-20T22:27:15.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:15.644+0000 7f9ab4200700 1 --2- 172.21.15.62:0/1690146986 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9a9406ecc0 0x7f9a94071170 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:27:15.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:15.644+0000 7f9ab4200700 1 --2- 172.21.15.62:0/1690146986 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9aac106b80 0x7f9aac19c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:27:15.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:15.644+0000 7f9ab4200700 1 --2- 172.21.15.62:0/1690146986 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9aac10c8f0 0x7f9aac19c910 unknown :-1 s=CLOSED pgs=1328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:27:15.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:15.644+0000 7f9ab4200700 1 -- 172.21.15.62:0/1690146986 >> 172.21.15.62:0/1690146986 conn(0x7f9aac076020 msgr2=0x7f9aac077940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:27:15.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:15.644+0000 7f9ab4200700 1 -- 172.21.15.62:0/1690146986 shutdown_connections 2024-04-20T22:27:15.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:15.644+0000 7f9ab4200700 1 -- 172.21.15.62:0/1690146986 wait complete. 2024-04-20T22:27:15.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:15.799+0000 7f3d5bc00700 1 -- 172.21.15.62:0/2840610655 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3d541026a0 msgr2=0x7f3d54102a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:27:15.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:15.799+0000 7f3d5bc00700 1 --2- 172.21.15.62:0/2840610655 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3d541026a0 0x7f3d54102a70 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f3d48009910 tx=0x7f3d48009c20 comp rx=0 tx=0).stop 2024-04-20T22:27:15.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:15.799+0000 7f3d5bc00700 1 -- 172.21.15.62:0/2840610655 shutdown_connections 2024-04-20T22:27:15.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:15.799+0000 7f3d5bc00700 1 --2- 172.21.15.62:0/2840610655 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3d54103040 0x7f3d54107010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:27:15.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:15.799+0000 7f3d5bc00700 1 --2- 172.21.15.62:0/2840610655 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3d541026a0 0x7f3d54102a70 unknown :-1 s=CLOSED pgs=1329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:27:15.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:15.799+0000 7f3d5bc00700 1 -- 172.21.15.62:0/2840610655 >> 172.21.15.62:0/2840610655 conn(0x7f3d540fdf40 msgr2=0x7f3d54100350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:27:15.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:15.800+0000 7f3d5bc00700 1 -- 172.21.15.62:0/2840610655 shutdown_connections 2024-04-20T22:27:15.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:15.800+0000 7f3d5bc00700 1 -- 172.21.15.62:0/2840610655 wait complete. 2024-04-20T22:27:15.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:15.801+0000 7f3d5bc00700 1 Processor -- start 2024-04-20T22:27:15.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:15.801+0000 7f3d5bc00700 1 -- start start 2024-04-20T22:27:15.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:15.801+0000 7f3d5bc00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3d541026a0 0x7f3d54198130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:27:15.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:15.801+0000 7f3d5bc00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3d54103040 0x7f3d54198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:27:15.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:15.801+0000 7f3d5bc00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d54198d50 con 0x7f3d54103040 2024-04-20T22:27:15.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:15.801+0000 7f3d5bc00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d5419cae0 con 0x7f3d541026a0 2024-04-20T22:27:15.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:15.802+0000 7f3d53e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3d54103040 0x7f3d54198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:27:15.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:15.802+0000 7f3d53e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3d54103040 0x7f3d54198670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60066/0 (socket says 172.21.15.62:60066) 2024-04-20T22:27:15.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:15.802+0000 7f3d53e00700 1 -- 172.21.15.62:0/1338837653 learned_addr learned my addr 172.21.15.62:0/1338837653 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:27:15.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:15.802+0000 7f3d59200700 1 --2- 172.21.15.62:0/1338837653 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3d541026a0 0x7f3d54198130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:27:15.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:15.802+0000 7f3d53e00700 1 -- 172.21.15.62:0/1338837653 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3d541026a0 msgr2=0x7f3d54198130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:27:15.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:15.802+0000 7f3d53e00700 1 --2- 172.21.15.62:0/1338837653 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3d541026a0 0x7f3d54198130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:27:15.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:15.802+0000 7f3d53e00700 1 -- 172.21.15.62:0/1338837653 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d480095f0 con 0x7f3d54103040 2024-04-20T22:27:15.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:15.803+0000 7f3d53e00700 1 --2- 172.21.15.62:0/1338837653 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3d54103040 0x7f3d54198670 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7f3d4000d900 tx=0x7f3d4000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:27:15.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:15.803+0000 7f3d51600700 1 -- 172.21.15.62:0/1338837653 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3d40009900 con 0x7f3d54103040 2024-04-20T22:27:15.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:15.803+0000 7f3d51600700 1 -- 172.21.15.62:0/1338837653 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3d40010460 con 0x7f3d54103040 2024-04-20T22:27:15.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:15.803+0000 7f3d5bc00700 1 -- 172.21.15.62:0/1338837653 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d5419cdc0 con 0x7f3d54103040 2024-04-20T22:27:15.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:15.803+0000 7f3d51600700 1 -- 172.21.15.62:0/1338837653 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3d4000f5f0 con 0x7f3d54103040 2024-04-20T22:27:15.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:15.803+0000 7f3d5bc00700 1 -- 172.21.15.62:0/1338837653 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d5419d310 con 0x7f3d54103040 2024-04-20T22:27:15.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:15.804+0000 7f3d5bc00700 1 -- 172.21.15.62:0/1338837653 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d5410a9b0 con 0x7f3d54103040 2024-04-20T22:27:15.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:15.805+0000 7f3d51600700 1 -- 172.21.15.62:0/1338837653 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f3d400105d0 con 0x7f3d54103040 2024-04-20T22:27:15.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:15.806+0000 7f3d51600700 1 --2- 172.21.15.62:0/1338837653 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3d3c06ecc0 0x7f3d3c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:27:15.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:15.806+0000 7f3d51600700 1 -- 172.21.15.62:0/1338837653 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f3d4008e9b0 con 0x7f3d54103040 2024-04-20T22:27:15.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:15.807+0000 7f3d59200700 1 --2- 172.21.15.62:0/1338837653 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3d3c06ecc0 0x7f3d3c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:27:15.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:15.807+0000 7f3d59200700 1 --2- 172.21.15.62:0/1338837653 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3d3c06ecc0 0x7f3d3c071170 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f3d48006010 tx=0x7f3d4801a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:27:15.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:15.811+0000 7f3d51600700 1 -- 172.21.15.62:0/1338837653 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f3d40057740 con 0x7f3d54103040 2024-04-20T22:27:16.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:16.145+0000 7f3d5bc00700 1 -- 172.21.15.62:0/1338837653 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f3d5404f2a0 con 0x7f3d54103040 2024-04-20T22:27:16.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:16.146+0000 7f3d51600700 1 -- 172.21.15.62:0/1338837653 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f3d4005ae00 con 0x7f3d54103040 2024-04-20T22:27:16.146 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:27:16.146 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T22:27:16.146 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:27:16.146 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:27:16.146 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T22:27:16.147 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:27:16.147 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:27:16.147 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T22:27:16.147 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 6 2024-04-20T22:27:16.147 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:27:16.147 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T22:27:16.147 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 4 2024-04-20T22:27:16.147 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:27:16.147 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T22:27:16.147 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 14 2024-04-20T22:27:16.147 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T22:27:16.147 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:27:16.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:16.148+0000 7f3d5bc00700 1 -- 172.21.15.62:0/1338837653 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3d3c06ecc0 msgr2=0x7f3d3c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:27:16.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:16.148+0000 7f3d5bc00700 1 --2- 172.21.15.62:0/1338837653 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3d3c06ecc0 0x7f3d3c071170 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f3d48006010 tx=0x7f3d4801a040 comp rx=0 tx=0).stop 2024-04-20T22:27:16.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:16.148+0000 7f3d5bc00700 1 -- 172.21.15.62:0/1338837653 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3d54103040 msgr2=0x7f3d54198670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:27:16.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:16.148+0000 7f3d5bc00700 1 --2- 172.21.15.62:0/1338837653 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3d54103040 0x7f3d54198670 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7f3d4000d900 tx=0x7f3d4000dc10 comp rx=0 tx=0).stop 2024-04-20T22:27:16.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:16.149+0000 7f3d5bc00700 1 -- 172.21.15.62:0/1338837653 shutdown_connections 2024-04-20T22:27:16.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:16.149+0000 7f3d5bc00700 1 --2- 172.21.15.62:0/1338837653 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3d3c06ecc0 0x7f3d3c071170 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:27:16.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:16.149+0000 7f3d5bc00700 1 --2- 172.21.15.62:0/1338837653 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3d541026a0 0x7f3d54198130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:27:16.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:16.149+0000 7f3d5bc00700 1 --2- 172.21.15.62:0/1338837653 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3d54103040 0x7f3d54198670 unknown :-1 s=CLOSED pgs=1330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:27:16.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:16.149+0000 7f3d5bc00700 1 -- 172.21.15.62:0/1338837653 >> 172.21.15.62:0/1338837653 conn(0x7f3d540fdf40 msgr2=0x7f3d540ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:27:16.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:16.149+0000 7f3d5bc00700 1 -- 172.21.15.62:0/1338837653 shutdown_connections 2024-04-20T22:27:16.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:16.149+0000 7f3d5bc00700 1 -- 172.21.15.62:0/1338837653 wait complete. 2024-04-20T22:27:16.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:16.300+0000 7fcf5e200700 1 -- 172.21.15.62:0/3502181555 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf580687a0 msgr2=0x7fcf58068b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:27:16.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:16.300+0000 7fcf5e200700 1 --2- 172.21.15.62:0/3502181555 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf580687a0 0x7fcf58068b70 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7fcf48009910 tx=0x7fcf48009c20 comp rx=0 tx=0).stop 2024-04-20T22:27:16.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:16.300+0000 7fcf5e200700 1 -- 172.21.15.62:0/3502181555 shutdown_connections 2024-04-20T22:27:16.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:16.300+0000 7fcf5e200700 1 --2- 172.21.15.62:0/3502181555 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcf58069140 0x7fcf58104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:27:16.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:16.300+0000 7fcf5e200700 1 --2- 172.21.15.62:0/3502181555 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf580687a0 0x7fcf58068b70 unknown :-1 s=CLOSED pgs=1331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:27:16.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:16.300+0000 7fcf5e200700 1 -- 172.21.15.62:0/3502181555 >> 172.21.15.62:0/3502181555 conn(0x7fcf58075130 msgr2=0x7fcf58075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:27:16.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:16.300+0000 7fcf5e200700 1 -- 172.21.15.62:0/3502181555 shutdown_connections 2024-04-20T22:27:16.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:16.300+0000 7fcf5e200700 1 -- 172.21.15.62:0/3502181555 wait complete. 2024-04-20T22:27:16.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:16.301+0000 7fcf5e200700 1 Processor -- start 2024-04-20T22:27:16.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:16.301+0000 7fcf5e200700 1 -- start start 2024-04-20T22:27:16.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:16.301+0000 7fcf5e200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf580687a0 0x7fcf581980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:27:16.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:16.301+0000 7fcf5e200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcf58069140 0x7fcf581985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:27:16.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:16.301+0000 7fcf5e200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf58198cd0 con 0x7fcf580687a0 2024-04-20T22:27:16.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:16.301+0000 7fcf5e200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf5819ca60 con 0x7fcf58069140 2024-04-20T22:27:16.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:16.302+0000 7fcf57400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf580687a0 0x7fcf581980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:27:16.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:16.302+0000 7fcf57400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf580687a0 0x7fcf581980b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60080/0 (socket says 172.21.15.62:60080) 2024-04-20T22:27:16.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:16.302+0000 7fcf57400700 1 -- 172.21.15.62:0/2542649882 learned_addr learned my addr 172.21.15.62:0/2542649882 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:27:16.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:16.302+0000 7fcf57400700 1 -- 172.21.15.62:0/2542649882 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcf58069140 msgr2=0x7fcf581985f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:27:16.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:16.302+0000 7fcf57400700 1 --2- 172.21.15.62:0/2542649882 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcf58069140 0x7fcf581985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:27:16.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:16.302+0000 7fcf57400700 1 -- 172.21.15.62:0/2542649882 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf480095f0 con 0x7fcf580687a0 2024-04-20T22:27:16.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:16.302+0000 7fcf57400700 1 --2- 172.21.15.62:0/2542649882 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf580687a0 0x7fcf581980b0 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7fcf480092e0 tx=0x7fcf4800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:27:16.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:16.303+0000 7fcf4fe00700 1 -- 172.21.15.62:0/2542649882 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcf4801d070 con 0x7fcf580687a0 2024-04-20T22:27:16.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:16.303+0000 7fcf4fe00700 1 -- 172.21.15.62:0/2542649882 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcf48005e00 con 0x7fcf580687a0 2024-04-20T22:27:16.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:16.303+0000 7fcf4fe00700 1 -- 172.21.15.62:0/2542649882 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcf48017400 con 0x7fcf580687a0 2024-04-20T22:27:16.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:16.303+0000 7fcf5e200700 1 -- 172.21.15.62:0/2542649882 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf5819cce0 con 0x7fcf580687a0 2024-04-20T22:27:16.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:16.303+0000 7fcf5e200700 1 -- 172.21.15.62:0/2542649882 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf5819d1d0 con 0x7fcf580687a0 2024-04-20T22:27:16.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:16.304+0000 7fcf5e200700 1 -- 172.21.15.62:0/2542649882 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf5804f2a0 con 0x7fcf580687a0 2024-04-20T22:27:16.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:16.305+0000 7fcf4fe00700 1 -- 172.21.15.62:0/2542649882 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fcf4800f460 con 0x7fcf580687a0 2024-04-20T22:27:16.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:16.306+0000 7fcf4fe00700 1 --2- 172.21.15.62:0/2542649882 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fcf3c06ecc0 0x7fcf3c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:27:16.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:16.306+0000 7fcf4fe00700 1 -- 172.21.15.62:0/2542649882 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fcf48090590 con 0x7fcf580687a0 2024-04-20T22:27:16.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:16.306+0000 7fcf56a00700 1 --2- 172.21.15.62:0/2542649882 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fcf3c06ecc0 0x7fcf3c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:27:16.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:16.307+0000 7fcf56a00700 1 --2- 172.21.15.62:0/2542649882 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fcf3c06ecc0 0x7fcf3c071170 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fcf44005950 tx=0x7fcf440058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:27:16.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:16.310+0000 7fcf4fe00700 1 -- 172.21.15.62:0/2542649882 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fcf480593a0 con 0x7fcf580687a0 2024-04-20T22:27:16.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:27:16 smithi062 ceph-mon[76322]: from='client.36720 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:27:16.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:27:16 smithi062 ceph-mon[76322]: from='client.36724 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:27:16.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:27:16 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/1338837653' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:27:16.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:16.594+0000 7fcf5e200700 1 -- 172.21.15.62:0/2542649882 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fcf5819d550 con 0x7fcf580687a0 2024-04-20T22:27:16.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:16.595+0000 7fcf4fe00700 1 -- 172.21.15.62:0/2542649882 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 29 v29) v1 ==== 76+0+1887 (secure 0 0 0) 0x7fcf4805ca60 con 0x7fcf580687a0 2024-04-20T22:27:16.595 INFO:teuthology.orchestra.run.smithi062.stdout:e29 2024-04-20T22:27:16.595 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T22:27:16.595 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:27:16.595 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T22:27:16.595 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:27:16.596 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T22:27:16.596 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T22:27:16.596 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 20 2024-04-20T22:27:16.596 INFO:teuthology.orchestra.run.smithi062.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-20T22:27:16.596 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T22:27:16.596 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T21:42:01.559061+0000 2024-04-20T22:27:16.596 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T22:27:16.596 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T22:27:16.596 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T22:27:16.596 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T22:27:16.596 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T22:27:16.596 INFO:teuthology.orchestra.run.smithi062.stdout:max_xattr_size 65536 2024-04-20T22:27:16.596 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T22:27:16.596 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T22:27:16.596 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 81 2024-04-20T22:27:16.597 INFO:teuthology.orchestra.run.smithi062.stdout:snap_rstat 1 2024-04-20T22:27:16.597 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:27:16.597 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 2 2024-04-20T22:27:16.597 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1 2024-04-20T22:27:16.597 INFO:teuthology.orchestra.run.smithi062.stdout:up {} 2024-04-20T22:27:16.597 INFO:teuthology.orchestra.run.smithi062.stdout:failed 0,1 2024-04-20T22:27:16.597 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T22:27:16.597 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T22:27:16.597 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T22:27:16.597 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T22:27:16.597 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data enabled 2024-04-20T22:27:16.597 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T22:27:16.597 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T22:27:16.597 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 1 2024-04-20T22:27:16.597 INFO:teuthology.orchestra.run.smithi062.stdout:qdb_cluster leader: 0 members: 2024-04-20T22:27:16.598 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:27:16.598 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:27:16.598 INFO:teuthology.orchestra.run.smithi062.stdout:Standby daemons: 2024-04-20T22:27:16.598 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:27:16.598 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ryqblb{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6826/4113916205,v1:172.21.15.62:6827/4113916205] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:27:16.598 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ozizwi{-1:34476} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6828/3801975165,v1:172.21.15.62:6829/3801975165] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:27:16.598 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.dvimer{-1:44727} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6824/3576699232,v1:172.21.15.189:6825/3576699232] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:27:16.598 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.xbgspw{-1:45075} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6826/212905792,v1:172.21.15.189:6827/212905792] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:27:16.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:16.598+0000 7fcf5e200700 1 -- 172.21.15.62:0/2542649882 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fcf3c06ecc0 msgr2=0x7fcf3c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:27:16.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:16.598+0000 7fcf5e200700 1 --2- 172.21.15.62:0/2542649882 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fcf3c06ecc0 0x7fcf3c071170 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fcf44005950 tx=0x7fcf440058e0 comp rx=0 tx=0).stop 2024-04-20T22:27:16.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:16.598+0000 7fcf5e200700 1 -- 172.21.15.62:0/2542649882 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf580687a0 msgr2=0x7fcf581980b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:27:16.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:16.598+0000 7fcf5e200700 1 --2- 172.21.15.62:0/2542649882 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf580687a0 0x7fcf581980b0 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7fcf480092e0 tx=0x7fcf4800be20 comp rx=0 tx=0).stop 2024-04-20T22:27:16.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:16.598+0000 7fcf5e200700 1 -- 172.21.15.62:0/2542649882 shutdown_connections 2024-04-20T22:27:16.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:16.598+0000 7fcf5e200700 1 --2- 172.21.15.62:0/2542649882 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fcf3c06ecc0 0x7fcf3c071170 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:27:16.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:16.598+0000 7fcf5e200700 1 --2- 172.21.15.62:0/2542649882 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf580687a0 0x7fcf581980b0 unknown :-1 s=CLOSED pgs=1332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:27:16.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:16.598+0000 7fcf5e200700 1 --2- 172.21.15.62:0/2542649882 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcf58069140 0x7fcf581985f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:27:16.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:16.598+0000 7fcf5e200700 1 -- 172.21.15.62:0/2542649882 >> 172.21.15.62:0/2542649882 conn(0x7fcf58075130 msgr2=0x7fcf58101e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:27:16.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:16.598+0000 7fcf5e200700 1 -- 172.21.15.62:0/2542649882 shutdown_connections 2024-04-20T22:27:16.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:16.598+0000 7fcf5e200700 1 -- 172.21.15.62:0/2542649882 wait complete. 2024-04-20T22:27:16.600 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 29 2024-04-20T22:27:16.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:27:16 smithi189 ceph-mon[57022]: from='client.36720 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:27:16.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:27:16 smithi189 ceph-mon[57022]: from='client.36724 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:27:16.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:27:16 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/1338837653' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:27:16.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:16.749+0000 7fe3f0c00700 1 -- 172.21.15.62:0/2972988896 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe3ec108470 msgr2=0x7fe3ec108840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:27:16.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:16.749+0000 7fe3f0c00700 1 --2- 172.21.15.62:0/2972988896 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe3ec108470 0x7fe3ec108840 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7fe3dc009910 tx=0x7fe3dc009c20 comp rx=0 tx=0).stop 2024-04-20T22:27:16.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:16.749+0000 7fe3f0c00700 1 -- 172.21.15.62:0/2972988896 shutdown_connections 2024-04-20T22:27:16.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:16.749+0000 7fe3f0c00700 1 --2- 172.21.15.62:0/2972988896 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3ec102470 0x7fe3ec1028e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:27:16.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:16.749+0000 7fe3f0c00700 1 --2- 172.21.15.62:0/2972988896 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe3ec108470 0x7fe3ec108840 unknown :-1 s=CLOSED pgs=1333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:27:16.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:16.749+0000 7fe3f0c00700 1 -- 172.21.15.62:0/2972988896 >> 172.21.15.62:0/2972988896 conn(0x7fe3ec0fdf90 msgr2=0x7fe3ec1003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:27:16.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:16.749+0000 7fe3f0c00700 1 -- 172.21.15.62:0/2972988896 shutdown_connections 2024-04-20T22:27:16.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:16.749+0000 7fe3f0c00700 1 -- 172.21.15.62:0/2972988896 wait complete. 2024-04-20T22:27:16.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:16.750+0000 7fe3f0c00700 1 Processor -- start 2024-04-20T22:27:16.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:16.750+0000 7fe3f0c00700 1 -- start start 2024-04-20T22:27:16.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:16.751+0000 7fe3f0c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3ec102470 0x7fe3ec1981d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:27:16.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:16.751+0000 7fe3f0c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe3ec108470 0x7fe3ec198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:27:16.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:16.751+0000 7fe3f0c00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3ec198df0 con 0x7fe3ec108470 2024-04-20T22:27:16.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:16.751+0000 7fe3f0c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3ec19cb80 con 0x7fe3ec102470 2024-04-20T22:27:16.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:16.751+0000 7fe3e9400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe3ec108470 0x7fe3ec198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:27:16.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:16.751+0000 7fe3e9400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe3ec108470 0x7fe3ec198710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60086/0 (socket says 172.21.15.62:60086) 2024-04-20T22:27:16.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:16.751+0000 7fe3e9400700 1 -- 172.21.15.62:0/629076338 learned_addr learned my addr 172.21.15.62:0/629076338 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:27:16.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:16.751+0000 7fe3e9e00700 1 --2- 172.21.15.62:0/629076338 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3ec102470 0x7fe3ec1981d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:27:16.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:16.752+0000 7fe3e9400700 1 -- 172.21.15.62:0/629076338 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3ec102470 msgr2=0x7fe3ec1981d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:27:16.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:16.752+0000 7fe3e9400700 1 --2- 172.21.15.62:0/629076338 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3ec102470 0x7fe3ec1981d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:27:16.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:16.752+0000 7fe3e9400700 1 -- 172.21.15.62:0/629076338 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3dc0095f0 con 0x7fe3ec108470 2024-04-20T22:27:16.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:16.752+0000 7fe3e9e00700 1 --2- 172.21.15.62:0/629076338 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3ec102470 0x7fe3ec1981d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T22:27:16.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:16.752+0000 7fe3e9400700 1 --2- 172.21.15.62:0/629076338 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe3ec108470 0x7fe3ec198710 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7fe3d800b4b0 tx=0x7fe3d800b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:27:16.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:16.752+0000 7fe3e2a00700 1 -- 172.21.15.62:0/629076338 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe3d8011800 con 0x7fe3ec108470 2024-04-20T22:27:16.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:16.752+0000 7fe3e2a00700 1 -- 172.21.15.62:0/629076338 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe3d8011e20 con 0x7fe3ec108470 2024-04-20T22:27:16.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:16.752+0000 7fe3f0c00700 1 -- 172.21.15.62:0/629076338 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3ec19ce60 con 0x7fe3ec108470 2024-04-20T22:27:16.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:16.752+0000 7fe3e2a00700 1 -- 172.21.15.62:0/629076338 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe3d800f2e0 con 0x7fe3ec108470 2024-04-20T22:27:16.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:16.752+0000 7fe3f0c00700 1 -- 172.21.15.62:0/629076338 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3ec19d3b0 con 0x7fe3ec108470 2024-04-20T22:27:16.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:16.754+0000 7fe3f0c00700 1 -- 172.21.15.62:0/629076338 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3ec10a9e0 con 0x7fe3ec108470 2024-04-20T22:27:16.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:16.755+0000 7fe3e2a00700 1 -- 172.21.15.62:0/629076338 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe3d8010380 con 0x7fe3ec108470 2024-04-20T22:27:16.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:16.755+0000 7fe3e2a00700 1 --2- 172.21.15.62:0/629076338 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe3d006ecc0 0x7fe3d0071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:27:16.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:16.756+0000 7fe3e2a00700 1 -- 172.21.15.62:0/629076338 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fe3d808eaa0 con 0x7fe3ec108470 2024-04-20T22:27:16.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:16.756+0000 7fe3e9e00700 1 --2- 172.21.15.62:0/629076338 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe3d006ecc0 0x7fe3d0071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:27:16.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:16.756+0000 7fe3e9e00700 1 --2- 172.21.15.62:0/629076338 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe3d006ecc0 0x7fe3d0071170 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fe3dc00b5c0 tx=0x7fe3dc005480 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:27:16.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:16.760+0000 7fe3e2a00700 1 -- 172.21.15.62:0/629076338 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fe3d80577b0 con 0x7fe3ec108470 2024-04-20T22:27:17.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:17.007+0000 7fe3f0c00700 1 -- 172.21.15.62:0/629076338 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe3ec19d630 con 0x7fe3d006ecc0 2024-04-20T22:27:17.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:17.009+0000 7fe3e2a00700 1 -- 172.21.15.62:0/629076338 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fe3ec19d630 con 0x7fe3d006ecc0 2024-04-20T22:27:17.009 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:27:17.009 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T22:27:17.009 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T22:27:17.009 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-20T22:27:17.009 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T22:27:17.009 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr", 2024-04-20T22:27:17.009 INFO:teuthology.orchestra.run.smithi062.stdout: "osd", 2024-04-20T22:27:17.009 INFO:teuthology.orchestra.run.smithi062.stdout: "mds", 2024-04-20T22:27:17.009 INFO:teuthology.orchestra.run.smithi062.stdout: "crash", 2024-04-20T22:27:17.010 INFO:teuthology.orchestra.run.smithi062.stdout: "mon" 2024-04-20T22:27:17.010 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T22:27:17.010 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "16/23 daemons upgraded", 2024-04-20T22:27:17.010 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading mds daemons", 2024-04-20T22:27:17.010 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T22:27:17.010 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:27:17.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:17.012+0000 7fe3f0c00700 1 -- 172.21.15.62:0/629076338 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe3d006ecc0 msgr2=0x7fe3d0071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:27:17.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:17.012+0000 7fe3f0c00700 1 --2- 172.21.15.62:0/629076338 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe3d006ecc0 0x7fe3d0071170 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fe3dc00b5c0 tx=0x7fe3dc005480 comp rx=0 tx=0).stop 2024-04-20T22:27:17.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:17.012+0000 7fe3f0c00700 1 -- 172.21.15.62:0/629076338 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe3ec108470 msgr2=0x7fe3ec198710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:27:17.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:17.012+0000 7fe3f0c00700 1 --2- 172.21.15.62:0/629076338 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe3ec108470 0x7fe3ec198710 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7fe3d800b4b0 tx=0x7fe3d800b870 comp rx=0 tx=0).stop 2024-04-20T22:27:17.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:17.012+0000 7fe3f0c00700 1 -- 172.21.15.62:0/629076338 shutdown_connections 2024-04-20T22:27:17.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:17.012+0000 7fe3f0c00700 1 --2- 172.21.15.62:0/629076338 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe3d006ecc0 0x7fe3d0071170 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:27:17.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:17.012+0000 7fe3f0c00700 1 --2- 172.21.15.62:0/629076338 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3ec102470 0x7fe3ec1981d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:27:17.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:17.012+0000 7fe3f0c00700 1 --2- 172.21.15.62:0/629076338 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe3ec108470 0x7fe3ec198710 unknown :-1 s=CLOSED pgs=1334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:27:17.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:17.012+0000 7fe3f0c00700 1 -- 172.21.15.62:0/629076338 >> 172.21.15.62:0/629076338 conn(0x7fe3ec0fdf90 msgr2=0x7fe3ec0feb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:27:17.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:17.012+0000 7fe3f0c00700 1 -- 172.21.15.62:0/629076338 shutdown_connections 2024-04-20T22:27:17.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:17.013+0000 7fe3f0c00700 1 -- 172.21.15.62:0/629076338 wait complete. 2024-04-20T22:27:17.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:17.164+0000 7fc06ba00700 1 -- 172.21.15.62:0/3152512580 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc064100530 msgr2=0x7fc064100900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:27:17.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:17.164+0000 7fc06ba00700 1 --2- 172.21.15.62:0/3152512580 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc064100530 0x7fc064100900 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7fc054009910 tx=0x7fc054009c20 comp rx=0 tx=0).stop 2024-04-20T22:27:17.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:17.164+0000 7fc06ba00700 1 -- 172.21.15.62:0/3152512580 shutdown_connections 2024-04-20T22:27:17.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:17.164+0000 7fc06ba00700 1 --2- 172.21.15.62:0/3152512580 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc064100ed0 0x7fc064104ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:27:17.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:17.164+0000 7fc06ba00700 1 --2- 172.21.15.62:0/3152512580 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc064100530 0x7fc064100900 unknown :-1 s=CLOSED pgs=1335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:27:17.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:17.164+0000 7fc06ba00700 1 -- 172.21.15.62:0/3152512580 >> 172.21.15.62:0/3152512580 conn(0x7fc0640fbe10 msgr2=0x7fc0640fe220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:27:17.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:17.164+0000 7fc06ba00700 1 -- 172.21.15.62:0/3152512580 shutdown_connections 2024-04-20T22:27:17.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:17.164+0000 7fc06ba00700 1 -- 172.21.15.62:0/3152512580 wait complete. 2024-04-20T22:27:17.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:17.166+0000 7fc06ba00700 1 Processor -- start 2024-04-20T22:27:17.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:17.166+0000 7fc06ba00700 1 -- start start 2024-04-20T22:27:17.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:17.166+0000 7fc06ba00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc064100530 0x7fc064195e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:27:17.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:17.166+0000 7fc06ba00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc064100ed0 0x7fc064196370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:27:17.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:17.166+0000 7fc06ba00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc064196a50 con 0x7fc064100ed0 2024-04-20T22:27:17.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:17.166+0000 7fc06ba00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc06419a7e0 con 0x7fc064100530 2024-04-20T22:27:17.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:17.167+0000 7fc05be00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc064100ed0 0x7fc064196370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:27:17.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:17.167+0000 7fc05be00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc064100ed0 0x7fc064196370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60102/0 (socket says 172.21.15.62:60102) 2024-04-20T22:27:17.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:17.167+0000 7fc05be00700 1 -- 172.21.15.62:0/3677295541 learned_addr learned my addr 172.21.15.62:0/3677295541 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:27:17.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:17.167+0000 7fc069000700 1 --2- 172.21.15.62:0/3677295541 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc064100530 0x7fc064195e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:27:17.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:17.167+0000 7fc05be00700 1 -- 172.21.15.62:0/3677295541 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc064100530 msgr2=0x7fc064195e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:27:17.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:17.167+0000 7fc05be00700 1 --2- 172.21.15.62:0/3677295541 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc064100530 0x7fc064195e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:27:17.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:17.167+0000 7fc05be00700 1 -- 172.21.15.62:0/3677295541 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0540095f0 con 0x7fc064100ed0 2024-04-20T22:27:17.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:17.168+0000 7fc05be00700 1 --2- 172.21.15.62:0/3677295541 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc064100ed0 0x7fc064196370 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7fc04c00d900 tx=0x7fc04c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:27:17.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:17.168+0000 7fc062000700 1 -- 172.21.15.62:0/3677295541 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc04c009900 con 0x7fc064100ed0 2024-04-20T22:27:17.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:17.168+0000 7fc062000700 1 -- 172.21.15.62:0/3677295541 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc04c010460 con 0x7fc064100ed0 2024-04-20T22:27:17.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:17.168+0000 7fc06ba00700 1 -- 172.21.15.62:0/3677295541 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc06419aac0 con 0x7fc064100ed0 2024-04-20T22:27:17.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:17.168+0000 7fc062000700 1 -- 172.21.15.62:0/3677295541 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc04c00f5f0 con 0x7fc064100ed0 2024-04-20T22:27:17.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:17.168+0000 7fc06ba00700 1 -- 172.21.15.62:0/3677295541 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc06419b010 con 0x7fc064100ed0 2024-04-20T22:27:17.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:17.169+0000 7fc06ba00700 1 -- 172.21.15.62:0/3677295541 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc064108840 con 0x7fc064100ed0 2024-04-20T22:27:17.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:17.170+0000 7fc062000700 1 -- 172.21.15.62:0/3677295541 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fc04c0105d0 con 0x7fc064100ed0 2024-04-20T22:27:17.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:17.171+0000 7fc062000700 1 --2- 172.21.15.62:0/3677295541 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc04806ecc0 0x7fc048071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:27:17.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:17.171+0000 7fc062000700 1 -- 172.21.15.62:0/3677295541 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fc04c08eca0 con 0x7fc064100ed0 2024-04-20T22:27:17.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:17.172+0000 7fc069000700 1 --2- 172.21.15.62:0/3677295541 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc04806ecc0 0x7fc048071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:27:17.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:17.172+0000 7fc069000700 1 --2- 172.21.15.62:0/3677295541 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc04806ecc0 0x7fc048071170 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fc054006010 tx=0x7fc05401a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:27:17.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:17.176+0000 7fc062000700 1 -- 172.21.15.62:0/3677295541 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fc04c057a30 con 0x7fc064100ed0 2024-04-20T22:27:17.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:27:17 smithi062 ceph-mon[76322]: from='client.36728 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:27:17.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:27:17 smithi062 ceph-mon[76322]: pgmap v1562: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:27:17.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:27:17 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/2542649882' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:27:17.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:17.497+0000 7fc06ba00700 1 -- 172.21.15.62:0/3677295541 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fc06404f2a0 con 0x7fc064100ed0 2024-04-20T22:27:17.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:17.498+0000 7fc062000700 1 -- 172.21.15.62:0/3677295541 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7fc04c05b0f0 con 0x7fc064100ed0 2024-04-20T22:27:17.500 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-20T22:27:17.500 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-20T22:27:17.500 INFO:teuthology.orchestra.run.smithi062.stdout: host smithi189 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi189/config 2024-04-20T22:27:17.500 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-20T22:27:17.501 INFO:teuthology.orchestra.run.smithi062.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi189 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp11670v2q:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:27:17.501 INFO:teuthology.orchestra.run.smithi062.stdout:Traceback (most recent call last): 2024-04-20T22:27:17.501 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-20T22:27:17.501 INFO:teuthology.orchestra.run.smithi062.stdout: return _run_code(code, main_globals, None, 2024-04-20T22:27:17.504 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-20T22:27:17.504 INFO:teuthology.orchestra.run.smithi062.stdout: exec(code, run_globals) 2024-04-20T22:27:17.504 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5618, in 2024-04-20T22:27:17.505 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5606, in main 2024-04-20T22:27:17.505 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 408, in _infer_config 2024-04-20T22:27:17.505 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 323, in _infer_fsid 2024-04-20T22:27:17.506 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 436, in _infer_image 2024-04-20T22:27:17.506 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 310, in _validate_fsid 2024-04-20T22:27:17.506 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 3359, in command_ceph_volume 2024-04-20T22:27:17.506 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-20T22:27:17.507 INFO:teuthology.orchestra.run.smithi062.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi189 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp11670v2q:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:27:17.508 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-20T22:27:17.508 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is degraded 2024-04-20T22:27:17.508 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T22:27:17.509 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T22:27:17.509 INFO:teuthology.orchestra.run.smithi062.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-20T22:27:17.509 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-20T22:27:17.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:17.501+0000 7fc06ba00700 1 -- 172.21.15.62:0/3677295541 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc04806ecc0 msgr2=0x7fc048071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:27:17.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:17.501+0000 7fc06ba00700 1 --2- 172.21.15.62:0/3677295541 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc04806ecc0 0x7fc048071170 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fc054006010 tx=0x7fc05401a040 comp rx=0 tx=0).stop 2024-04-20T22:27:17.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:17.501+0000 7fc06ba00700 1 -- 172.21.15.62:0/3677295541 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc064100ed0 msgr2=0x7fc064196370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:27:17.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:17.501+0000 7fc06ba00700 1 --2- 172.21.15.62:0/3677295541 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc064100ed0 0x7fc064196370 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7fc04c00d900 tx=0x7fc04c00dcc0 comp rx=0 tx=0).stop 2024-04-20T22:27:17.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:17.501+0000 7fc06ba00700 1 -- 172.21.15.62:0/3677295541 shutdown_connections 2024-04-20T22:27:17.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:17.501+0000 7fc06ba00700 1 --2- 172.21.15.62:0/3677295541 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc04806ecc0 0x7fc048071170 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:27:17.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:17.501+0000 7fc06ba00700 1 --2- 172.21.15.62:0/3677295541 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc064100530 0x7fc064195e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:27:17.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:17.501+0000 7fc06ba00700 1 --2- 172.21.15.62:0/3677295541 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc064100ed0 0x7fc064196370 unknown :-1 s=CLOSED pgs=1336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:27:17.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:17.501+0000 7fc06ba00700 1 -- 172.21.15.62:0/3677295541 >> 172.21.15.62:0/3677295541 conn(0x7fc0640fbe10 msgr2=0x7fc0640fd1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:27:17.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:17.501+0000 7fc06ba00700 1 -- 172.21.15.62:0/3677295541 shutdown_connections 2024-04-20T22:27:17.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:17.501+0000 7fc06ba00700 1 -- 172.21.15.62:0/3677295541 wait complete. 2024-04-20T22:27:17.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:27:17 smithi189 ceph-mon[57022]: from='client.36728 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:27:17.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:27:17 smithi189 ceph-mon[57022]: pgmap v1562: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:27:17.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:27:17 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/2542649882' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:27:18.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:27:18 smithi062 ceph-mon[76322]: from='client.36740 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:27:18.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:27:18 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/3677295541' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:27:18.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:27:18 smithi189 ceph-mon[57022]: from='client.36740 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:27:18.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:27:18 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/3677295541' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:27:19.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:27:19 smithi062 ceph-mon[76322]: pgmap v1563: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:27:19.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:27:19 smithi189 ceph-mon[57022]: pgmap v1563: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:27:20.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:27:20 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:27:20.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:27:20 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:27:21.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:27:21 smithi062 ceph-mon[76322]: pgmap v1564: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:27:21.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:27:21 smithi189 ceph-mon[57022]: pgmap v1564: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:27:23.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:27:23 smithi062 ceph-mon[76322]: pgmap v1565: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:27:23.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:27:23 smithi189 ceph-mon[57022]: pgmap v1565: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:27:25.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:27:25 smithi189 ceph-mon[57022]: pgmap v1566: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:27:25.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:27:25 smithi062 ceph-mon[76322]: pgmap v1566: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:27:27.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:27:27 smithi189 ceph-mon[57022]: pgmap v1567: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:27:27.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:27:27 smithi062 ceph-mon[76322]: pgmap v1567: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:27:29.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:27:29 smithi189 ceph-mon[57022]: pgmap v1568: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:27:29.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:27:29 smithi062 ceph-mon[76322]: pgmap v1568: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:27:31.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:27:31 smithi189 ceph-mon[57022]: pgmap v1569: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:27:31.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:27:31 smithi062 ceph-mon[76322]: pgmap v1569: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:27:33.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:27:33 smithi062 ceph-mon[76322]: pgmap v1570: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:27:33.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:27:33 smithi189 ceph-mon[57022]: pgmap v1570: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:27:35.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:27:35 smithi189 ceph-mon[57022]: pgmap v1571: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:27:35.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:27:35 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:27:35.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:27:35 smithi062 ceph-mon[76322]: pgmap v1571: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:27:35.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:27:35 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:27:37.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:27:37 smithi189 ceph-mon[57022]: pgmap v1572: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:27:37.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:27:37 smithi062 ceph-mon[76322]: pgmap v1572: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:27:39.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:27:39 smithi189 ceph-mon[57022]: pgmap v1573: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:27:39.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:27:39 smithi062 ceph-mon[76322]: pgmap v1573: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:27:41.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:27:41 smithi189 ceph-mon[57022]: pgmap v1574: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:27:41.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:27:41 smithi062 ceph-mon[76322]: pgmap v1574: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:27:43.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:27:43 smithi189 ceph-mon[57022]: pgmap v1575: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:27:43.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:27:43 smithi062 ceph-mon[76322]: pgmap v1575: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:27:45.540 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:27:45 smithi189 ceph-mon[57022]: pgmap v1576: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:27:45.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:27:45 smithi062 ceph-mon[76322]: pgmap v1576: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:27:47.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:47.683+0000 7fbb86200700 1 -- 172.21.15.62:0/952720921 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbb8010c8f0 msgr2=0x7fbb8010ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:27:47.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:47.683+0000 7fbb86200700 1 --2- 172.21.15.62:0/952720921 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbb8010c8f0 0x7fbb8010ccc0 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7fbb70009910 tx=0x7fbb70009c20 comp rx=0 tx=0).stop 2024-04-20T22:27:47.684 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:27:47 smithi062 ceph-mon[76322]: pgmap v1577: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:27:47.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:47.683+0000 7fbb86200700 1 -- 172.21.15.62:0/952720921 shutdown_connections 2024-04-20T22:27:47.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:47.683+0000 7fbb86200700 1 --2- 172.21.15.62:0/952720921 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb80106b80 0x7fbb80106ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:27:47.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:47.683+0000 7fbb86200700 1 --2- 172.21.15.62:0/952720921 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbb8010c8f0 0x7fbb8010ccc0 unknown :-1 s=CLOSED pgs=1337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:27:47.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:47.683+0000 7fbb86200700 1 -- 172.21.15.62:0/952720921 >> 172.21.15.62:0/952720921 conn(0x7fbb80076060 msgr2=0x7fbb80078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:27:47.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:47.683+0000 7fbb86200700 1 -- 172.21.15.62:0/952720921 shutdown_connections 2024-04-20T22:27:47.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:47.684+0000 7fbb86200700 1 -- 172.21.15.62:0/952720921 wait complete. 2024-04-20T22:27:47.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:47.685+0000 7fbb86200700 1 Processor -- start 2024-04-20T22:27:47.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:47.685+0000 7fbb86200700 1 -- start start 2024-04-20T22:27:47.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:47.685+0000 7fbb86200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbb80106b80 0x7fbb8019c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:27:47.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:47.685+0000 7fbb86200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb8010c8f0 0x7fbb8019ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:27:47.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:47.685+0000 7fbb86200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb8019d120 con 0x7fbb80106b80 2024-04-20T22:27:47.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:47.685+0000 7fbb86200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb801a0eb0 con 0x7fbb8010c8f0 2024-04-20T22:27:47.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:47.686+0000 7fbb7f400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbb80106b80 0x7fbb8019c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:27:47.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:47.686+0000 7fbb7f400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbb80106b80 0x7fbb8019c500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:58824/0 (socket says 172.21.15.62:58824) 2024-04-20T22:27:47.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:47.686+0000 7fbb7f400700 1 -- 172.21.15.62:0/3237644717 learned_addr learned my addr 172.21.15.62:0/3237644717 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:27:47.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:47.686+0000 7fbb7ea00700 1 --2- 172.21.15.62:0/3237644717 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb8010c8f0 0x7fbb8019ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:27:47.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:47.686+0000 7fbb7f400700 1 -- 172.21.15.62:0/3237644717 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb8010c8f0 msgr2=0x7fbb8019ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:27:47.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:47.686+0000 7fbb7f400700 1 --2- 172.21.15.62:0/3237644717 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb8010c8f0 0x7fbb8019ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:27:47.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:47.686+0000 7fbb7f400700 1 -- 172.21.15.62:0/3237644717 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb700095f0 con 0x7fbb80106b80 2024-04-20T22:27:47.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:47.687+0000 7fbb7f400700 1 --2- 172.21.15.62:0/3237644717 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbb80106b80 0x7fbb8019c500 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7fbb700092e0 tx=0x7fbb7000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:27:47.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:47.687+0000 7fbb77e00700 1 -- 172.21.15.62:0/3237644717 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbb7001d070 con 0x7fbb80106b80 2024-04-20T22:27:47.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:47.687+0000 7fbb77e00700 1 -- 172.21.15.62:0/3237644717 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbb70005e00 con 0x7fbb80106b80 2024-04-20T22:27:47.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:47.687+0000 7fbb77e00700 1 -- 172.21.15.62:0/3237644717 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbb70017400 con 0x7fbb80106b80 2024-04-20T22:27:47.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:47.687+0000 7fbb86200700 1 -- 172.21.15.62:0/3237644717 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb801a1130 con 0x7fbb80106b80 2024-04-20T22:27:47.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:47.687+0000 7fbb86200700 1 -- 172.21.15.62:0/3237644717 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb801a1620 con 0x7fbb80106b80 2024-04-20T22:27:47.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:47.688+0000 7fbb86200700 1 -- 172.21.15.62:0/3237644717 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb80109b20 con 0x7fbb80106b80 2024-04-20T22:27:47.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:47.689+0000 7fbb77e00700 1 -- 172.21.15.62:0/3237644717 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fbb7000f460 con 0x7fbb80106b80 2024-04-20T22:27:47.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:47.690+0000 7fbb77e00700 1 --2- 172.21.15.62:0/3237644717 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fbb6406ecc0 0x7fbb64071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:27:47.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:47.691+0000 7fbb77e00700 1 -- 172.21.15.62:0/3237644717 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fbb70090330 con 0x7fbb80106b80 2024-04-20T22:27:47.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:47.691+0000 7fbb7ea00700 1 --2- 172.21.15.62:0/3237644717 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fbb6406ecc0 0x7fbb64071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:27:47.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:47.692+0000 7fbb7ea00700 1 --2- 172.21.15.62:0/3237644717 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fbb6406ecc0 0x7fbb64071170 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c005950 tx=0x7fbb6c0058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:27:47.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:47.696+0000 7fbb77e00700 1 -- 172.21.15.62:0/3237644717 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fbb700590c0 con 0x7fbb80106b80 2024-04-20T22:27:47.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:27:47 smithi189 ceph-mon[57022]: pgmap v1577: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:27:47.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:47.953+0000 7fbb86200700 1 -- 172.21.15.62:0/3237644717 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fbb801a1950 con 0x7fbb6406ecc0 2024-04-20T22:27:47.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:47.956+0000 7fbb77e00700 1 -- 172.21.15.62:0/3237644717 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fbb801a1950 con 0x7fbb6406ecc0 2024-04-20T22:27:47.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:47.959+0000 7fbb86200700 1 -- 172.21.15.62:0/3237644717 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fbb6406ecc0 msgr2=0x7fbb64071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:27:47.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:47.959+0000 7fbb86200700 1 --2- 172.21.15.62:0/3237644717 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fbb6406ecc0 0x7fbb64071170 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c005950 tx=0x7fbb6c0058e0 comp rx=0 tx=0).stop 2024-04-20T22:27:47.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:47.959+0000 7fbb86200700 1 -- 172.21.15.62:0/3237644717 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbb80106b80 msgr2=0x7fbb8019c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:27:47.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:47.959+0000 7fbb86200700 1 --2- 172.21.15.62:0/3237644717 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbb80106b80 0x7fbb8019c500 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7fbb700092e0 tx=0x7fbb7000be20 comp rx=0 tx=0).stop 2024-04-20T22:27:47.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:47.959+0000 7fbb86200700 1 -- 172.21.15.62:0/3237644717 shutdown_connections 2024-04-20T22:27:47.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:47.959+0000 7fbb86200700 1 --2- 172.21.15.62:0/3237644717 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fbb6406ecc0 0x7fbb64071170 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:27:47.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:47.959+0000 7fbb86200700 1 --2- 172.21.15.62:0/3237644717 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbb80106b80 0x7fbb8019c500 unknown :-1 s=CLOSED pgs=1338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:27:47.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:47.959+0000 7fbb86200700 1 --2- 172.21.15.62:0/3237644717 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb8010c8f0 0x7fbb8019ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:27:47.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:47.959+0000 7fbb86200700 1 -- 172.21.15.62:0/3237644717 >> 172.21.15.62:0/3237644717 conn(0x7fbb80076060 msgr2=0x7fbb80077aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:27:47.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:47.959+0000 7fbb86200700 1 -- 172.21.15.62:0/3237644717 shutdown_connections 2024-04-20T22:27:47.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:47.959+0000 7fbb86200700 1 -- 172.21.15.62:0/3237644717 wait complete. 2024-04-20T22:27:47.976 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T22:27:48.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:48.125+0000 7f4819800700 1 -- 172.21.15.62:0/2603325545 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4814100290 msgr2=0x7f4814100660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:27:48.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:48.125+0000 7f4819800700 1 --2- 172.21.15.62:0/2603325545 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4814100290 0x7f4814100660 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7f4804009910 tx=0x7f4804009c20 comp rx=0 tx=0).stop 2024-04-20T22:27:48.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:48.125+0000 7f4819800700 1 -- 172.21.15.62:0/2603325545 shutdown_connections 2024-04-20T22:27:48.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:48.125+0000 7f4819800700 1 --2- 172.21.15.62:0/2603325545 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4814100c30 0x7f4814104c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:27:48.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:48.125+0000 7f4819800700 1 --2- 172.21.15.62:0/2603325545 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4814100290 0x7f4814100660 unknown :-1 s=CLOSED pgs=1339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:27:48.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:48.125+0000 7f4819800700 1 -- 172.21.15.62:0/2603325545 >> 172.21.15.62:0/2603325545 conn(0x7f48140fbb10 msgr2=0x7f48140fdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:27:48.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:48.126+0000 7f4819800700 1 -- 172.21.15.62:0/2603325545 shutdown_connections 2024-04-20T22:27:48.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:48.126+0000 7f4819800700 1 -- 172.21.15.62:0/2603325545 wait complete. 2024-04-20T22:27:48.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:48.127+0000 7f4819800700 1 Processor -- start 2024-04-20T22:27:48.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:48.127+0000 7f4819800700 1 -- start start 2024-04-20T22:27:48.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:48.127+0000 7f4819800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4814100290 0x7f4814073050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:27:48.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:48.127+0000 7f4819800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4814100c30 0x7f4814073590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:27:48.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:48.127+0000 7f4819800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4814073ad0 con 0x7f4814100c30 2024-04-20T22:27:48.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:48.127+0000 7f4819800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4814073c10 con 0x7f4814100290 2024-04-20T22:27:48.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:48.127+0000 7f480be00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4814100c30 0x7f4814073590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:27:48.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:48.128+0000 7f480be00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4814100c30 0x7f4814073590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:58856/0 (socket says 172.21.15.62:58856) 2024-04-20T22:27:48.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:48.128+0000 7f480be00700 1 -- 172.21.15.62:0/1400658098 learned_addr learned my addr 172.21.15.62:0/1400658098 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:27:48.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:48.128+0000 7f4813e00700 1 --2- 172.21.15.62:0/1400658098 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4814100290 0x7f4814073050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:27:48.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:48.128+0000 7f480be00700 1 -- 172.21.15.62:0/1400658098 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4814100290 msgr2=0x7f4814073050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:27:48.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:48.128+0000 7f480be00700 1 --2- 172.21.15.62:0/1400658098 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4814100290 0x7f4814073050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:27:48.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:48.128+0000 7f480be00700 1 -- 172.21.15.62:0/1400658098 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48040095f0 con 0x7f4814100c30 2024-04-20T22:27:48.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:48.128+0000 7f4813e00700 1 --2- 172.21.15.62:0/1400658098 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4814100290 0x7f4814073050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T22:27:48.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:48.128+0000 7f480be00700 1 --2- 172.21.15.62:0/1400658098 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4814100c30 0x7f4814073590 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7f47fc00b480 tx=0x7f47fc00b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:27:48.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:48.129+0000 7f4811600700 1 -- 172.21.15.62:0/1400658098 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f47fc0118c0 con 0x7f4814100c30 2024-04-20T22:27:48.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:48.129+0000 7f4811600700 1 -- 172.21.15.62:0/1400658098 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f47fc010380 con 0x7f4814100c30 2024-04-20T22:27:48.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:48.129+0000 7f4819800700 1 -- 172.21.15.62:0/1400658098 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48141a1ee0 con 0x7f4814100c30 2024-04-20T22:27:48.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:48.129+0000 7f4811600700 1 -- 172.21.15.62:0/1400658098 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f47fc00f4f0 con 0x7f4814100c30 2024-04-20T22:27:48.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:48.129+0000 7f4819800700 1 -- 172.21.15.62:0/1400658098 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48141a22f0 con 0x7f4814100c30 2024-04-20T22:27:48.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:48.130+0000 7f4819800700 1 -- 172.21.15.62:0/1400658098 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48141085a0 con 0x7f4814100c30 2024-04-20T22:27:48.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:48.131+0000 7f4811600700 1 -- 172.21.15.62:0/1400658098 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f47fc0104f0 con 0x7f4814100c30 2024-04-20T22:27:48.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:48.132+0000 7f4811600700 1 --2- 172.21.15.62:0/1400658098 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f47f806ecc0 0x7f47f8071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:27:48.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:48.132+0000 7f4811600700 1 -- 172.21.15.62:0/1400658098 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f47fc08eb30 con 0x7f4814100c30 2024-04-20T22:27:48.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:48.132+0000 7f4813e00700 1 --2- 172.21.15.62:0/1400658098 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f47f806ecc0 0x7f47f8071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:27:48.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:48.133+0000 7f4813e00700 1 --2- 172.21.15.62:0/1400658098 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f47f806ecc0 0x7f47f8071170 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f4804006010 tx=0x7f480400b540 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:27:48.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:48.137+0000 7f4811600700 1 -- 172.21.15.62:0/1400658098 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f47fc057840 con 0x7f4814100c30 2024-04-20T22:27:48.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:48.413+0000 7f4819800700 1 -- 172.21.15.62:0/1400658098 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f481402ccf0 con 0x7f47f806ecc0 2024-04-20T22:27:48.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:48.416+0000 7f4811600700 1 -- 172.21.15.62:0/1400658098 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f481402ccf0 con 0x7f47f806ecc0 2024-04-20T22:27:48.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:48.419+0000 7f4819800700 1 -- 172.21.15.62:0/1400658098 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f47f806ecc0 msgr2=0x7f47f8071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:27:48.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:48.419+0000 7f4819800700 1 --2- 172.21.15.62:0/1400658098 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f47f806ecc0 0x7f47f8071170 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f4804006010 tx=0x7f480400b540 comp rx=0 tx=0).stop 2024-04-20T22:27:48.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:48.419+0000 7f4819800700 1 -- 172.21.15.62:0/1400658098 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4814100c30 msgr2=0x7f4814073590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:27:48.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:48.419+0000 7f4819800700 1 --2- 172.21.15.62:0/1400658098 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4814100c30 0x7f4814073590 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7f47fc00b480 tx=0x7f47fc00b840 comp rx=0 tx=0).stop 2024-04-20T22:27:48.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:48.419+0000 7f4819800700 1 -- 172.21.15.62:0/1400658098 shutdown_connections 2024-04-20T22:27:48.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:48.419+0000 7f4819800700 1 --2- 172.21.15.62:0/1400658098 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f47f806ecc0 0x7f47f8071170 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:27:48.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:48.419+0000 7f4819800700 1 --2- 172.21.15.62:0/1400658098 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4814100290 0x7f4814073050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:27:48.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:48.419+0000 7f4819800700 1 --2- 172.21.15.62:0/1400658098 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4814100c30 0x7f4814073590 unknown :-1 s=CLOSED pgs=1340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:27:48.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:48.419+0000 7f4819800700 1 -- 172.21.15.62:0/1400658098 >> 172.21.15.62:0/1400658098 conn(0x7f48140fbb10 msgr2=0x7f48140fd2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:27:48.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:48.420+0000 7f4819800700 1 -- 172.21.15.62:0/1400658098 shutdown_connections 2024-04-20T22:27:48.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:48.420+0000 7f4819800700 1 -- 172.21.15.62:0/1400658098 wait complete. 2024-04-20T22:27:48.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:48.572+0000 7f8963200700 1 -- 172.21.15.62:0/2826131582 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f895c102470 msgr2=0x7f895c1028e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:27:48.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:48.572+0000 7f8963200700 1 --2- 172.21.15.62:0/2826131582 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f895c102470 0x7f895c1028e0 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f8948009910 tx=0x7f8948009c20 comp rx=0 tx=0).stop 2024-04-20T22:27:48.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:48.572+0000 7f8963200700 1 -- 172.21.15.62:0/2826131582 shutdown_connections 2024-04-20T22:27:48.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:48.572+0000 7f8963200700 1 --2- 172.21.15.62:0/2826131582 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f895c102470 0x7f895c1028e0 unknown :-1 s=CLOSED pgs=1341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:27:48.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:48.572+0000 7f8963200700 1 --2- 172.21.15.62:0/2826131582 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f895c108470 0x7f895c108840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:27:48.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:48.572+0000 7f8963200700 1 -- 172.21.15.62:0/2826131582 >> 172.21.15.62:0/2826131582 conn(0x7f895c0fdf90 msgr2=0x7f895c1003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:27:48.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:48.572+0000 7f8963200700 1 -- 172.21.15.62:0/2826131582 shutdown_connections 2024-04-20T22:27:48.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:48.572+0000 7f8963200700 1 -- 172.21.15.62:0/2826131582 wait complete. 2024-04-20T22:27:48.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:48.573+0000 7f8963200700 1 Processor -- start 2024-04-20T22:27:48.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:48.573+0000 7f8963200700 1 -- start start 2024-04-20T22:27:48.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:48.574+0000 7f8963200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f895c102470 0x7f895c198190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:27:48.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:48.574+0000 7f8963200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f895c108470 0x7f895c1986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:27:48.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:48.574+0000 7f8963200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f895c198db0 con 0x7f895c102470 2024-04-20T22:27:48.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:48.574+0000 7f8963200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f895c19cb40 con 0x7f895c108470 2024-04-20T22:27:48.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:48.574+0000 7f895be00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f895c102470 0x7f895c198190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:27:48.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:48.574+0000 7f895be00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f895c102470 0x7f895c198190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:58882/0 (socket says 172.21.15.62:58882) 2024-04-20T22:27:48.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:48.574+0000 7f895be00700 1 -- 172.21.15.62:0/3302457038 learned_addr learned my addr 172.21.15.62:0/3302457038 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:27:48.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:48.574+0000 7f895b400700 1 --2- 172.21.15.62:0/3302457038 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f895c108470 0x7f895c1986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:27:48.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:48.574+0000 7f895be00700 1 -- 172.21.15.62:0/3302457038 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f895c108470 msgr2=0x7f895c1986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:27:48.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:48.574+0000 7f895be00700 1 --2- 172.21.15.62:0/3302457038 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f895c108470 0x7f895c1986d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:27:48.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:48.574+0000 7f895be00700 1 -- 172.21.15.62:0/3302457038 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89480095f0 con 0x7f895c102470 2024-04-20T22:27:48.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:48.575+0000 7f895be00700 1 --2- 172.21.15.62:0/3302457038 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f895c102470 0x7f895c198190 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7f894c00b4b0 tx=0x7f894c00b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:27:48.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:48.575+0000 7f8958c00700 1 -- 172.21.15.62:0/3302457038 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f894c011800 con 0x7f895c102470 2024-04-20T22:27:48.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:48.575+0000 7f8963200700 1 -- 172.21.15.62:0/3302457038 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f895c19ce20 con 0x7f895c102470 2024-04-20T22:27:48.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:48.575+0000 7f8958c00700 1 -- 172.21.15.62:0/3302457038 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f894c011e20 con 0x7f895c102470 2024-04-20T22:27:48.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:48.575+0000 7f8958c00700 1 -- 172.21.15.62:0/3302457038 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f894c00f2e0 con 0x7f895c102470 2024-04-20T22:27:48.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:48.575+0000 7f8963200700 1 -- 172.21.15.62:0/3302457038 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f895c19d370 con 0x7f895c102470 2024-04-20T22:27:48.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:48.576+0000 7f8963200700 1 -- 172.21.15.62:0/3302457038 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f895c10aa00 con 0x7f895c102470 2024-04-20T22:27:48.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:48.577+0000 7f8958c00700 1 -- 172.21.15.62:0/3302457038 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f894c010380 con 0x7f895c102470 2024-04-20T22:27:48.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:48.579+0000 7f8958c00700 1 --2- 172.21.15.62:0/3302457038 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f894006ecc0 0x7f8940071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:27:48.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:48.579+0000 7f8958c00700 1 -- 172.21.15.62:0/3302457038 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f894c08eaa0 con 0x7f895c102470 2024-04-20T22:27:48.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:48.579+0000 7f895b400700 1 --2- 172.21.15.62:0/3302457038 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f894006ecc0 0x7f8940071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:27:48.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:48.580+0000 7f895b400700 1 --2- 172.21.15.62:0/3302457038 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f894006ecc0 0x7f8940071170 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f895c1997b0 tx=0x7f894801a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:27:48.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:48.584+0000 7f8958c00700 1 -- 172.21.15.62:0/3302457038 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f894c0577b0 con 0x7f895c102470 2024-04-20T22:27:48.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:48.836+0000 7f8963200700 1 -- 172.21.15.62:0/3302457038 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f895c19d650 con 0x7f894006ecc0 2024-04-20T22:27:48.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:48.870+0000 7f8958c00700 1 -- 172.21.15.62:0/3302457038 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f895c19d650 con 0x7f894006ecc0 2024-04-20T22:27:48.871 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T22:27:48.871 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (53m) 14m ago 67m 14.6M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T22:27:48.871 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (67m) 14m ago 67m 11.6M - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T22:27:48.871 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (66m) 14m ago 66m 11.2M - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T22:27:48.871 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (49m) 14m ago 67m 6660k - 19.0.0-3133-ga8c84f59 76bc6b91b726 2825d5ba11c6 2024-04-20T22:27:48.871 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (49m) 14m ago 66m 6665k - 19.0.0-3133-ga8c84f59 76bc6b91b726 20f746fa1bba 2024-04-20T22:27:48.871 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (53m) 14m ago 67m 75.5M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T22:27:48.872 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (45m) 14m ago 60m 22.9M - 19.0.0-3133-ga8c84f59 76bc6b91b726 de80bc90a683 2024-04-20T22:27:48.872 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (45m) 14m ago 60m 23.0M - 19.0.0-3133-ga8c84f59 76bc6b91b726 cd793b7ee98e 2024-04-20T22:27:48.872 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (29m) 14m ago 60m 18.5M - 19.0.0-3133-ga8c84f59 76bc6b91b726 0e7b0bc4d1ab 2024-04-20T22:27:48.872 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (14m) 14m ago 60m 11.4M - 19.0.0-3133-ga8c84f59 76bc6b91b726 0af732ac9e06 2024-04-20T22:27:48.872 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (56m) 14m ago 69m 517M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T22:27:48.872 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (55m) 14m ago 66m 460M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T22:27:48.872 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (51m) 14m ago 69m 86.3M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 019eb1479485 2024-04-20T22:27:48.872 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (50m) 14m ago 66m 58.0M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 55aac560aab6 2024-04-20T22:27:48.872 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (54m) 14m ago 67m 8766k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T22:27:48.872 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (54m) 14m ago 66m 8711k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T22:27:48.872 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (49m) 14m ago 65m 91.3M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 ee2bb901f26e 2024-04-20T22:27:48.872 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (48m) 14m ago 64m 94.9M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 9075152d4162 2024-04-20T22:27:48.872 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (48m) 14m ago 63m 78.5M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 2d4f3e38dc0e 2024-04-20T22:27:48.872 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (47m) 14m ago 63m 94.1M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 8f97b17ae0eb 2024-04-20T22:27:48.872 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (46m) 14m ago 62m 96.1M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 653d8079967d 2024-04-20T22:27:48.873 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (46m) 14m ago 62m 90.9M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 596dded14c12 2024-04-20T22:27:48.873 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (54m) 14m ago 66m 90.4M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T22:27:48.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:48.873+0000 7f8963200700 1 -- 172.21.15.62:0/3302457038 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f894006ecc0 msgr2=0x7f8940071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:27:48.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:48.873+0000 7f8963200700 1 --2- 172.21.15.62:0/3302457038 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f894006ecc0 0x7f8940071170 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f895c1997b0 tx=0x7f894801a040 comp rx=0 tx=0).stop 2024-04-20T22:27:48.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:48.873+0000 7f8963200700 1 -- 172.21.15.62:0/3302457038 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f895c102470 msgr2=0x7f895c198190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:27:48.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:48.873+0000 7f8963200700 1 --2- 172.21.15.62:0/3302457038 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f895c102470 0x7f895c198190 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7f894c00b4b0 tx=0x7f894c00b870 comp rx=0 tx=0).stop 2024-04-20T22:27:48.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:48.873+0000 7f8963200700 1 -- 172.21.15.62:0/3302457038 shutdown_connections 2024-04-20T22:27:48.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:48.873+0000 7f8963200700 1 --2- 172.21.15.62:0/3302457038 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f894006ecc0 0x7f8940071170 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:27:48.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:48.873+0000 7f8963200700 1 --2- 172.21.15.62:0/3302457038 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f895c102470 0x7f895c198190 unknown :-1 s=CLOSED pgs=1342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:27:48.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:48.873+0000 7f8963200700 1 --2- 172.21.15.62:0/3302457038 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f895c108470 0x7f895c1986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:27:48.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:48.873+0000 7f8963200700 1 -- 172.21.15.62:0/3302457038 >> 172.21.15.62:0/3302457038 conn(0x7f895c0fdf90 msgr2=0x7f895c0feb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:27:48.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:48.873+0000 7f8963200700 1 -- 172.21.15.62:0/3302457038 shutdown_connections 2024-04-20T22:27:48.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:48.873+0000 7f8963200700 1 -- 172.21.15.62:0/3302457038 wait complete. 2024-04-20T22:27:49.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:49.035+0000 7fe6edc00700 1 -- 172.21.15.62:0/536532963 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6e81085b0 msgr2=0x7fe6e8108980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:27:49.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:49.035+0000 7fe6edc00700 1 --2- 172.21.15.62:0/536532963 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6e81085b0 0x7fe6e8108980 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7fe6d8009910 tx=0x7fe6d8009c20 comp rx=0 tx=0).stop 2024-04-20T22:27:49.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:49.035+0000 7fe6edc00700 1 -- 172.21.15.62:0/536532963 shutdown_connections 2024-04-20T22:27:49.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:49.035+0000 7fe6edc00700 1 --2- 172.21.15.62:0/536532963 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe6e8068540 0x7fe6e80689b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:27:49.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:49.035+0000 7fe6edc00700 1 --2- 172.21.15.62:0/536532963 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6e81085b0 0x7fe6e8108980 unknown :-1 s=CLOSED pgs=1343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:27:49.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:49.035+0000 7fe6edc00700 1 -- 172.21.15.62:0/536532963 >> 172.21.15.62:0/536532963 conn(0x7fe6e8075130 msgr2=0x7fe6e8075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:27:49.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:49.035+0000 7fe6edc00700 1 -- 172.21.15.62:0/536532963 shutdown_connections 2024-04-20T22:27:49.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:49.035+0000 7fe6edc00700 1 -- 172.21.15.62:0/536532963 wait complete. 2024-04-20T22:27:49.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:49.037+0000 7fe6edc00700 1 Processor -- start 2024-04-20T22:27:49.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:49.037+0000 7fe6edc00700 1 -- start start 2024-04-20T22:27:49.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:49.037+0000 7fe6edc00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe6e8068540 0x7fe6e81980e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:27:49.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:49.037+0000 7fe6edc00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6e81085b0 0x7fe6e8198620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:27:49.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:49.037+0000 7fe6edc00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e8198d00 con 0x7fe6e81085b0 2024-04-20T22:27:49.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:49.037+0000 7fe6edc00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e819ca90 con 0x7fe6e8068540 2024-04-20T22:27:49.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:49.038+0000 7fe6e6000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6e81085b0 0x7fe6e8198620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:27:49.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:49.038+0000 7fe6e6000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6e81085b0 0x7fe6e8198620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:58900/0 (socket says 172.21.15.62:58900) 2024-04-20T22:27:49.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:49.038+0000 7fe6e6000700 1 -- 172.21.15.62:0/3335045656 learned_addr learned my addr 172.21.15.62:0/3335045656 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:27:49.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:49.038+0000 7fe6e6a00700 1 --2- 172.21.15.62:0/3335045656 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe6e8068540 0x7fe6e81980e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:27:49.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:49.038+0000 7fe6e6000700 1 -- 172.21.15.62:0/3335045656 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe6e8068540 msgr2=0x7fe6e81980e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:27:49.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:49.038+0000 7fe6e6000700 1 --2- 172.21.15.62:0/3335045656 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe6e8068540 0x7fe6e81980e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:27:49.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:49.038+0000 7fe6e6000700 1 -- 172.21.15.62:0/3335045656 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6d80095f0 con 0x7fe6e81085b0 2024-04-20T22:27:49.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:49.038+0000 7fe6e6000700 1 --2- 172.21.15.62:0/3335045656 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6e81085b0 0x7fe6e8198620 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7fe6d400c8a0 tx=0x7fe6d400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:27:49.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:49.039+0000 7fe6df400700 1 -- 172.21.15.62:0/3335045656 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe6d400cea0 con 0x7fe6e81085b0 2024-04-20T22:27:49.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:49.039+0000 7fe6df400700 1 -- 172.21.15.62:0/3335045656 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe6d4004830 con 0x7fe6e81085b0 2024-04-20T22:27:49.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:49.039+0000 7fe6df400700 1 -- 172.21.15.62:0/3335045656 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe6d40055b0 con 0x7fe6e81085b0 2024-04-20T22:27:49.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:49.039+0000 7fe6edc00700 1 -- 172.21.15.62:0/3335045656 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6e819cd70 con 0x7fe6e81085b0 2024-04-20T22:27:49.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:49.039+0000 7fe6edc00700 1 -- 172.21.15.62:0/3335045656 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6e819d2c0 con 0x7fe6e81085b0 2024-04-20T22:27:49.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:49.040+0000 7fe6edc00700 1 -- 172.21.15.62:0/3335045656 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6e810aa00 con 0x7fe6e81085b0 2024-04-20T22:27:49.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:49.041+0000 7fe6df400700 1 -- 172.21.15.62:0/3335045656 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe6d40049a0 con 0x7fe6e81085b0 2024-04-20T22:27:49.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:49.042+0000 7fe6df400700 1 --2- 172.21.15.62:0/3335045656 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe6cc06ecc0 0x7fe6cc071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:27:49.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:49.042+0000 7fe6df400700 1 -- 172.21.15.62:0/3335045656 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fe6d408eaf0 con 0x7fe6e81085b0 2024-04-20T22:27:49.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:49.042+0000 7fe6e6a00700 1 --2- 172.21.15.62:0/3335045656 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe6cc06ecc0 0x7fe6cc071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:27:49.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:49.043+0000 7fe6e6a00700 1 --2- 172.21.15.62:0/3335045656 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe6cc06ecc0 0x7fe6cc071170 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fe6d8000c00 tx=0x7fe6d8005fd0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:27:49.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:49.047+0000 7fe6df400700 1 -- 172.21.15.62:0/3335045656 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fe6d4057880 con 0x7fe6e81085b0 2024-04-20T22:27:49.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:49.384+0000 7fe6edc00700 1 -- 172.21.15.62:0/3335045656 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fe6e819d590 con 0x7fe6e81085b0 2024-04-20T22:27:49.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:49.385+0000 7fe6df400700 1 -- 172.21.15.62:0/3335045656 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7fe6d401d070 con 0x7fe6e81085b0 2024-04-20T22:27:49.386 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:27:49.386 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T22:27:49.386 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:27:49.386 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:27:49.387 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T22:27:49.387 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:27:49.387 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:27:49.387 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T22:27:49.387 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 6 2024-04-20T22:27:49.387 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:27:49.387 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T22:27:49.387 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 4 2024-04-20T22:27:49.387 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:27:49.387 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T22:27:49.387 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 14 2024-04-20T22:27:49.387 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T22:27:49.387 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:27:49.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:49.388+0000 7fe6edc00700 1 -- 172.21.15.62:0/3335045656 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe6cc06ecc0 msgr2=0x7fe6cc071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:27:49.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:49.388+0000 7fe6edc00700 1 --2- 172.21.15.62:0/3335045656 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe6cc06ecc0 0x7fe6cc071170 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fe6d8000c00 tx=0x7fe6d8005fd0 comp rx=0 tx=0).stop 2024-04-20T22:27:49.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:49.388+0000 7fe6edc00700 1 -- 172.21.15.62:0/3335045656 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6e81085b0 msgr2=0x7fe6e8198620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:27:49.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:49.388+0000 7fe6edc00700 1 --2- 172.21.15.62:0/3335045656 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6e81085b0 0x7fe6e8198620 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7fe6d400c8a0 tx=0x7fe6d400cc60 comp rx=0 tx=0).stop 2024-04-20T22:27:49.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:49.388+0000 7fe6edc00700 1 -- 172.21.15.62:0/3335045656 shutdown_connections 2024-04-20T22:27:49.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:49.388+0000 7fe6edc00700 1 --2- 172.21.15.62:0/3335045656 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe6cc06ecc0 0x7fe6cc071170 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:27:49.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:49.388+0000 7fe6edc00700 1 --2- 172.21.15.62:0/3335045656 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe6e8068540 0x7fe6e81980e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:27:49.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:49.388+0000 7fe6edc00700 1 --2- 172.21.15.62:0/3335045656 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6e81085b0 0x7fe6e8198620 unknown :-1 s=CLOSED pgs=1344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:27:49.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:49.388+0000 7fe6edc00700 1 -- 172.21.15.62:0/3335045656 >> 172.21.15.62:0/3335045656 conn(0x7fe6e8075130 msgr2=0x7fe6e8106430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:27:49.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:49.388+0000 7fe6edc00700 1 -- 172.21.15.62:0/3335045656 shutdown_connections 2024-04-20T22:27:49.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:49.389+0000 7fe6edc00700 1 -- 172.21.15.62:0/3335045656 wait complete. 2024-04-20T22:27:49.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:49.546+0000 7f7cdce00700 1 -- 172.21.15.62:0/744555595 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7cd80739c0 msgr2=0x7f7cd810c5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:27:49.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:49.546+0000 7f7cdce00700 1 --2- 172.21.15.62:0/744555595 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7cd80739c0 0x7f7cd810c5c0 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7f7cc0009940 tx=0x7f7cc0009c50 comp rx=0 tx=0).stop 2024-04-20T22:27:49.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:49.546+0000 7f7cdce00700 1 -- 172.21.15.62:0/744555595 shutdown_connections 2024-04-20T22:27:49.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:49.546+0000 7f7cdce00700 1 --2- 172.21.15.62:0/744555595 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7cd80739c0 0x7f7cd810c5c0 unknown :-1 s=CLOSED pgs=1345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:27:49.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:49.546+0000 7f7cdce00700 1 --2- 172.21.15.62:0/744555595 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7cd80730b0 0x7f7cd8073480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:27:49.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:49.546+0000 7f7cdce00700 1 -- 172.21.15.62:0/744555595 >> 172.21.15.62:0/744555595 conn(0x7f7cd80fbda0 msgr2=0x7f7cd80fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:27:49.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:49.546+0000 7f7cdce00700 1 -- 172.21.15.62:0/744555595 shutdown_connections 2024-04-20T22:27:49.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:49.546+0000 7f7cdce00700 1 -- 172.21.15.62:0/744555595 wait complete. 2024-04-20T22:27:49.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:49.547+0000 7f7cdce00700 1 Processor -- start 2024-04-20T22:27:49.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:49.548+0000 7f7cdce00700 1 -- start start 2024-04-20T22:27:49.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:49.548+0000 7f7cdce00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7cd80730b0 0x7f7cd8198210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:27:49.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:49.548+0000 7f7cdce00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7cd80739c0 0x7f7cd8198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:27:49.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:49.548+0000 7f7cdce00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cd8198e30 con 0x7f7cd80730b0 2024-04-20T22:27:49.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:49.548+0000 7f7cdce00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cd819cbc0 con 0x7f7cd80739c0 2024-04-20T22:27:49.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:49.548+0000 7f7cd5e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7cd80730b0 0x7f7cd8198210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:27:49.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:49.548+0000 7f7cd5e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7cd80730b0 0x7f7cd8198210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:58918/0 (socket says 172.21.15.62:58918) 2024-04-20T22:27:49.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:49.548+0000 7f7cd5e00700 1 -- 172.21.15.62:0/3738509244 learned_addr learned my addr 172.21.15.62:0/3738509244 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:27:49.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:49.549+0000 7f7cd5e00700 1 -- 172.21.15.62:0/3738509244 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7cd80739c0 msgr2=0x7f7cd8198750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:27:49.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:49.549+0000 7f7cd5e00700 1 --2- 172.21.15.62:0/3738509244 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7cd80739c0 0x7f7cd8198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:27:49.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:49.549+0000 7f7cd5e00700 1 -- 172.21.15.62:0/3738509244 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cc00095f0 con 0x7f7cd80730b0 2024-04-20T22:27:49.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:49.549+0000 7f7cd5e00700 1 --2- 172.21.15.62:0/3738509244 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7cd80730b0 0x7f7cd8198210 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7f7cc800c8a0 tx=0x7f7cc800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:27:49.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:49.549+0000 7f7ccea00700 1 -- 172.21.15.62:0/3738509244 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7cc800cea0 con 0x7f7cd80730b0 2024-04-20T22:27:49.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:49.549+0000 7f7ccea00700 1 -- 172.21.15.62:0/3738509244 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7cc8004830 con 0x7f7cd80730b0 2024-04-20T22:27:49.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:49.549+0000 7f7cdce00700 1 -- 172.21.15.62:0/3738509244 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cd819cea0 con 0x7f7cd80730b0 2024-04-20T22:27:49.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:49.549+0000 7f7ccea00700 1 -- 172.21.15.62:0/3738509244 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7cc80055b0 con 0x7f7cd80730b0 2024-04-20T22:27:49.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:49.550+0000 7f7cdce00700 1 -- 172.21.15.62:0/3738509244 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cd819d3f0 con 0x7f7cd80730b0 2024-04-20T22:27:49.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:49.551+0000 7f7cdce00700 1 -- 172.21.15.62:0/3738509244 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cd8109cc0 con 0x7f7cd80730b0 2024-04-20T22:27:49.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:49.552+0000 7f7ccea00700 1 -- 172.21.15.62:0/3738509244 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f7cc80049a0 con 0x7f7cd80730b0 2024-04-20T22:27:49.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:49.553+0000 7f7ccea00700 1 --2- 172.21.15.62:0/3738509244 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7cbc06ecc0 0x7f7cbc071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:27:49.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:49.553+0000 7f7ccea00700 1 -- 172.21.15.62:0/3738509244 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f7cc808eaf0 con 0x7f7cd80730b0 2024-04-20T22:27:49.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:49.553+0000 7f7ccfe00700 1 --2- 172.21.15.62:0/3738509244 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7cbc06ecc0 0x7f7cbc071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:27:49.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:49.554+0000 7f7ccfe00700 1 --2- 172.21.15.62:0/3738509244 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7cbc06ecc0 0x7f7cbc071170 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f7cc0006010 tx=0x7f7cc0000bc0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:27:49.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:49.557+0000 7f7ccea00700 1 -- 172.21.15.62:0/3738509244 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f7cc8057880 con 0x7f7cd80730b0 2024-04-20T22:27:49.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:27:49 smithi189 ceph-mon[57022]: pgmap v1578: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:27:49.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:27:49 smithi189 ceph-mon[57022]: from='client.36748 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:27:49.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:27:49 smithi189 ceph-mon[57022]: from='client.36752 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:27:49.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:27:49 smithi062 ceph-mon[76322]: pgmap v1578: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:27:49.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:27:49 smithi062 ceph-mon[76322]: from='client.36748 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:27:49.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:27:49 smithi062 ceph-mon[76322]: from='client.36752 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:27:49.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:49.853+0000 7f7cdce00700 1 -- 172.21.15.62:0/3738509244 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f7cd8199640 con 0x7f7cd80730b0 2024-04-20T22:27:49.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:49.854+0000 7f7ccea00700 1 -- 172.21.15.62:0/3738509244 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 29 v29) v1 ==== 76+0+1887 (secure 0 0 0) 0x7f7cc801d070 con 0x7f7cd80730b0 2024-04-20T22:27:49.855 INFO:teuthology.orchestra.run.smithi062.stdout:e29 2024-04-20T22:27:49.855 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T22:27:49.855 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:27:49.855 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T22:27:49.855 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:27:49.855 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T22:27:49.855 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T22:27:49.855 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 20 2024-04-20T22:27:49.855 INFO:teuthology.orchestra.run.smithi062.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-20T22:27:49.855 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T22:27:49.856 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T21:42:01.559061+0000 2024-04-20T22:27:49.856 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T22:27:49.856 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T22:27:49.856 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T22:27:49.856 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T22:27:49.856 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T22:27:49.856 INFO:teuthology.orchestra.run.smithi062.stdout:max_xattr_size 65536 2024-04-20T22:27:49.856 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T22:27:49.856 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T22:27:49.856 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 81 2024-04-20T22:27:49.856 INFO:teuthology.orchestra.run.smithi062.stdout:snap_rstat 1 2024-04-20T22:27:49.856 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:27:49.856 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 2 2024-04-20T22:27:49.856 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1 2024-04-20T22:27:49.856 INFO:teuthology.orchestra.run.smithi062.stdout:up {} 2024-04-20T22:27:49.857 INFO:teuthology.orchestra.run.smithi062.stdout:failed 0,1 2024-04-20T22:27:49.857 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T22:27:49.857 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T22:27:49.857 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T22:27:49.857 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T22:27:49.857 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data enabled 2024-04-20T22:27:49.857 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T22:27:49.857 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T22:27:49.857 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 1 2024-04-20T22:27:49.857 INFO:teuthology.orchestra.run.smithi062.stdout:qdb_cluster leader: 0 members: 2024-04-20T22:27:49.857 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:27:49.857 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:27:49.857 INFO:teuthology.orchestra.run.smithi062.stdout:Standby daemons: 2024-04-20T22:27:49.857 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:27:49.857 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ryqblb{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6826/4113916205,v1:172.21.15.62:6827/4113916205] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:27:49.858 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ozizwi{-1:34476} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.62:6828/3801975165,v1:172.21.15.62:6829/3801975165] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:27:49.858 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.dvimer{-1:44727} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6824/3576699232,v1:172.21.15.189:6825/3576699232] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:27:49.858 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.xbgspw{-1:45075} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6826/212905792,v1:172.21.15.189:6827/212905792] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:27:49.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:49.857+0000 7f7cdce00700 1 -- 172.21.15.62:0/3738509244 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7cbc06ecc0 msgr2=0x7f7cbc071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:27:49.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:49.857+0000 7f7cdce00700 1 --2- 172.21.15.62:0/3738509244 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7cbc06ecc0 0x7f7cbc071170 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f7cc0006010 tx=0x7f7cc0000bc0 comp rx=0 tx=0).stop 2024-04-20T22:27:49.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:49.857+0000 7f7cdce00700 1 -- 172.21.15.62:0/3738509244 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7cd80730b0 msgr2=0x7f7cd8198210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:27:49.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:49.857+0000 7f7cdce00700 1 --2- 172.21.15.62:0/3738509244 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7cd80730b0 0x7f7cd8198210 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7f7cc800c8a0 tx=0x7f7cc800cc60 comp rx=0 tx=0).stop 2024-04-20T22:27:49.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:49.857+0000 7f7cdce00700 1 -- 172.21.15.62:0/3738509244 shutdown_connections 2024-04-20T22:27:49.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:49.858+0000 7f7cdce00700 1 --2- 172.21.15.62:0/3738509244 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7cbc06ecc0 0x7f7cbc071170 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:27:49.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:49.858+0000 7f7cdce00700 1 --2- 172.21.15.62:0/3738509244 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7cd80730b0 0x7f7cd8198210 unknown :-1 s=CLOSED pgs=1346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:27:49.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:49.858+0000 7f7cdce00700 1 --2- 172.21.15.62:0/3738509244 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7cd80739c0 0x7f7cd8198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:27:49.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:49.858+0000 7f7cdce00700 1 -- 172.21.15.62:0/3738509244 >> 172.21.15.62:0/3738509244 conn(0x7f7cd80fbda0 msgr2=0x7f7cd8106e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:27:49.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:49.858+0000 7f7cdce00700 1 -- 172.21.15.62:0/3738509244 shutdown_connections 2024-04-20T22:27:49.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:49.858+0000 7f7cdce00700 1 -- 172.21.15.62:0/3738509244 wait complete. 2024-04-20T22:27:49.860 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 29 2024-04-20T22:27:50.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:50.024+0000 7f841b200700 1 -- 172.21.15.62:0/2330212371 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f841c102e70 msgr2=0x7f841c106e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:27:50.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:50.024+0000 7f841b200700 1 --2- 172.21.15.62:0/2330212371 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f841c102e70 0x7f841c106e40 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7f8404009910 tx=0x7f8404009c20 comp rx=0 tx=0).stop 2024-04-20T22:27:50.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:50.025+0000 7f841b200700 1 -- 172.21.15.62:0/2330212371 shutdown_connections 2024-04-20T22:27:50.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:50.025+0000 7f841b200700 1 --2- 172.21.15.62:0/2330212371 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f841c102e70 0x7f841c106e40 unknown :-1 s=CLOSED pgs=1347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:27:50.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:50.025+0000 7f841b200700 1 --2- 172.21.15.62:0/2330212371 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f841c1024d0 0x7f841c1028a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:27:50.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:50.025+0000 7f841b200700 1 -- 172.21.15.62:0/2330212371 >> 172.21.15.62:0/2330212371 conn(0x7f841c0fdd90 msgr2=0x7f841c1001a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:27:50.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:50.025+0000 7f841b200700 1 -- 172.21.15.62:0/2330212371 shutdown_connections 2024-04-20T22:27:50.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:50.025+0000 7f841b200700 1 -- 172.21.15.62:0/2330212371 wait complete. 2024-04-20T22:27:50.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:50.026+0000 7f841b200700 1 Processor -- start 2024-04-20T22:27:50.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:50.026+0000 7f841b200700 1 -- start start 2024-04-20T22:27:50.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:50.026+0000 7f841b200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f841c1024d0 0x7f841c197e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:27:50.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:50.026+0000 7f841b200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f841c102e70 0x7f841c198390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:27:50.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:50.026+0000 7f841b200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f841c198a70 con 0x7f841c102e70 2024-04-20T22:27:50.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:50.026+0000 7f841b200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f841c19c800 con 0x7f841c1024d0 2024-04-20T22:27:50.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:50.027+0000 7f8411400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f841c102e70 0x7f841c198390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:27:50.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:50.027+0000 7f8411400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f841c102e70 0x7f841c198390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:58936/0 (socket says 172.21.15.62:58936) 2024-04-20T22:27:50.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:50.027+0000 7f8411400700 1 -- 172.21.15.62:0/178391573 learned_addr learned my addr 172.21.15.62:0/178391573 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:27:50.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:50.027+0000 7f8419e00700 1 --2- 172.21.15.62:0/178391573 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f841c1024d0 0x7f841c197e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:27:50.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:50.027+0000 7f8411400700 1 -- 172.21.15.62:0/178391573 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f841c1024d0 msgr2=0x7f841c197e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:27:50.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:50.027+0000 7f8411400700 1 --2- 172.21.15.62:0/178391573 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f841c1024d0 0x7f841c197e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:27:50.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:50.027+0000 7f8411400700 1 -- 172.21.15.62:0/178391573 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84040095f0 con 0x7f841c102e70 2024-04-20T22:27:50.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:50.028+0000 7f8411400700 1 --2- 172.21.15.62:0/178391573 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f841c102e70 0x7f841c198390 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f84040092e0 tx=0x7f840400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:27:50.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:50.028+0000 7f8413400700 1 -- 172.21.15.62:0/178391573 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f840401d070 con 0x7f841c102e70 2024-04-20T22:27:50.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:50.028+0000 7f8413400700 1 -- 172.21.15.62:0/178391573 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8404005dc0 con 0x7f841c102e70 2024-04-20T22:27:50.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:50.028+0000 7f841b200700 1 -- 172.21.15.62:0/178391573 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f841c19ca80 con 0x7f841c102e70 2024-04-20T22:27:50.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:50.028+0000 7f8413400700 1 -- 172.21.15.62:0/178391573 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8404017400 con 0x7f841c102e70 2024-04-20T22:27:50.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:50.028+0000 7f841b200700 1 -- 172.21.15.62:0/178391573 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f841c19cf70 con 0x7f841c102e70 2024-04-20T22:27:50.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:50.029+0000 7f841b200700 1 -- 172.21.15.62:0/178391573 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f841c10a750 con 0x7f841c102e70 2024-04-20T22:27:50.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:50.030+0000 7f8413400700 1 -- 172.21.15.62:0/178391573 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f840400f460 con 0x7f841c102e70 2024-04-20T22:27:50.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:50.031+0000 7f8413400700 1 --2- 172.21.15.62:0/178391573 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f840006ecc0 0x7f8400071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:27:50.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:50.032+0000 7f8413400700 1 -- 172.21.15.62:0/178391573 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f8404090350 con 0x7f841c102e70 2024-04-20T22:27:50.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:50.032+0000 7f8419e00700 1 --2- 172.21.15.62:0/178391573 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f840006ecc0 0x7f8400071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:27:50.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:50.033+0000 7f8419e00700 1 --2- 172.21.15.62:0/178391573 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f840006ecc0 0x7f8400071170 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f840c005950 tx=0x7f840c0058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:27:50.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:50.036+0000 7f8413400700 1 -- 172.21.15.62:0/178391573 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f84040590e0 con 0x7f841c102e70 2024-04-20T22:27:50.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:50.294+0000 7f841b200700 1 -- 172.21.15.62:0/178391573 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f841c02cd10 con 0x7f840006ecc0 2024-04-20T22:27:50.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:50.297+0000 7f8413400700 1 -- 172.21.15.62:0/178391573 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f841c02cd10 con 0x7f840006ecc0 2024-04-20T22:27:50.297 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:27:50.297 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T22:27:50.297 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T22:27:50.297 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-20T22:27:50.297 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T22:27:50.297 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr", 2024-04-20T22:27:50.297 INFO:teuthology.orchestra.run.smithi062.stdout: "osd", 2024-04-20T22:27:50.297 INFO:teuthology.orchestra.run.smithi062.stdout: "mds", 2024-04-20T22:27:50.297 INFO:teuthology.orchestra.run.smithi062.stdout: "crash", 2024-04-20T22:27:50.297 INFO:teuthology.orchestra.run.smithi062.stdout: "mon" 2024-04-20T22:27:50.297 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T22:27:50.298 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "16/23 daemons upgraded", 2024-04-20T22:27:50.298 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading mds daemons", 2024-04-20T22:27:50.298 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T22:27:50.298 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:27:50.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:50.299+0000 7f841b200700 1 -- 172.21.15.62:0/178391573 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f840006ecc0 msgr2=0x7f8400071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:27:50.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:50.299+0000 7f841b200700 1 --2- 172.21.15.62:0/178391573 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f840006ecc0 0x7f8400071170 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f840c005950 tx=0x7f840c0058e0 comp rx=0 tx=0).stop 2024-04-20T22:27:50.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:50.299+0000 7f841b200700 1 -- 172.21.15.62:0/178391573 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f841c102e70 msgr2=0x7f841c198390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:27:50.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:50.299+0000 7f841b200700 1 --2- 172.21.15.62:0/178391573 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f841c102e70 0x7f841c198390 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f84040092e0 tx=0x7f840400be20 comp rx=0 tx=0).stop 2024-04-20T22:27:50.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:50.300+0000 7f841b200700 1 -- 172.21.15.62:0/178391573 shutdown_connections 2024-04-20T22:27:50.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:50.300+0000 7f841b200700 1 --2- 172.21.15.62:0/178391573 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f840006ecc0 0x7f8400071170 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:27:50.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:50.300+0000 7f841b200700 1 --2- 172.21.15.62:0/178391573 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f841c1024d0 0x7f841c197e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:27:50.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:50.300+0000 7f841b200700 1 --2- 172.21.15.62:0/178391573 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f841c102e70 0x7f841c198390 unknown :-1 s=CLOSED pgs=1348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:27:50.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:50.300+0000 7f841b200700 1 -- 172.21.15.62:0/178391573 >> 172.21.15.62:0/178391573 conn(0x7f841c0fdd90 msgr2=0x7f841c1066b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:27:50.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:50.300+0000 7f841b200700 1 -- 172.21.15.62:0/178391573 shutdown_connections 2024-04-20T22:27:50.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:50.300+0000 7f841b200700 1 -- 172.21.15.62:0/178391573 wait complete. 2024-04-20T22:27:50.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:50.455+0000 7f7a19c00700 1 -- 172.21.15.62:0/103710193 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a14104130 msgr2=0x7f7a14104500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:27:50.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:50.455+0000 7f7a19c00700 1 --2- 172.21.15.62:0/103710193 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a14104130 0x7f7a14104500 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f7a04009910 tx=0x7f7a04009c20 comp rx=0 tx=0).stop 2024-04-20T22:27:50.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:50.455+0000 7f7a19c00700 1 -- 172.21.15.62:0/103710193 shutdown_connections 2024-04-20T22:27:50.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:50.455+0000 7f7a19c00700 1 --2- 172.21.15.62:0/103710193 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7a140ffbd0 0x7f7a14100040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:27:50.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:50.455+0000 7f7a19c00700 1 --2- 172.21.15.62:0/103710193 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a14104130 0x7f7a14104500 unknown :-1 s=CLOSED pgs=1349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:27:50.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:50.455+0000 7f7a19c00700 1 -- 172.21.15.62:0/103710193 >> 172.21.15.62:0/103710193 conn(0x7f7a14075130 msgr2=0x7f7a14075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:27:50.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:50.455+0000 7f7a19c00700 1 -- 172.21.15.62:0/103710193 shutdown_connections 2024-04-20T22:27:50.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:50.455+0000 7f7a19c00700 1 -- 172.21.15.62:0/103710193 wait complete. 2024-04-20T22:27:50.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:50.456+0000 7f7a19c00700 1 Processor -- start 2024-04-20T22:27:50.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:50.456+0000 7f7a19c00700 1 -- start start 2024-04-20T22:27:50.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:50.457+0000 7f7a19c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a140ffbd0 0x7f7a14195e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:27:50.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:50.457+0000 7f7a19c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7a14104130 0x7f7a14196390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:27:50.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:50.457+0000 7f7a19c00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a14196a70 con 0x7f7a140ffbd0 2024-04-20T22:27:50.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:50.457+0000 7f7a19c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a1419a800 con 0x7f7a14104130 2024-04-20T22:27:50.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:50.457+0000 7f7a12a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a140ffbd0 0x7f7a14195e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:27:50.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:50.457+0000 7f7a12a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a140ffbd0 0x7f7a14195e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:58946/0 (socket says 172.21.15.62:58946) 2024-04-20T22:27:50.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:50.457+0000 7f7a12a00700 1 -- 172.21.15.62:0/468580021 learned_addr learned my addr 172.21.15.62:0/468580021 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:27:50.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:50.457+0000 7f7a12000700 1 --2- 172.21.15.62:0/468580021 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7a14104130 0x7f7a14196390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:27:50.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:50.457+0000 7f7a12a00700 1 -- 172.21.15.62:0/468580021 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7a14104130 msgr2=0x7f7a14196390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:27:50.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:50.457+0000 7f7a12a00700 1 --2- 172.21.15.62:0/468580021 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7a14104130 0x7f7a14196390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:27:50.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:50.457+0000 7f7a12a00700 1 -- 172.21.15.62:0/468580021 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a040095f0 con 0x7f7a140ffbd0 2024-04-20T22:27:50.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:50.457+0000 7f7a12000700 1 --2- 172.21.15.62:0/468580021 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7a14104130 0x7f7a14196390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T22:27:50.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:50.458+0000 7f7a12a00700 1 --2- 172.21.15.62:0/468580021 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a140ffbd0 0x7f7a14195e50 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f7a040092e0 tx=0x7f7a0400be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:27:50.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:50.458+0000 7f7a0b400700 1 -- 172.21.15.62:0/468580021 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7a0401d070 con 0x7f7a140ffbd0 2024-04-20T22:27:50.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:50.458+0000 7f7a0b400700 1 -- 172.21.15.62:0/468580021 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7a0400f460 con 0x7f7a140ffbd0 2024-04-20T22:27:50.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:50.458+0000 7f7a19c00700 1 -- 172.21.15.62:0/468580021 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a1419aa80 con 0x7f7a140ffbd0 2024-04-20T22:27:50.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:50.458+0000 7f7a0b400700 1 -- 172.21.15.62:0/468580021 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7a04017630 con 0x7f7a140ffbd0 2024-04-20T22:27:50.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:50.458+0000 7f7a19c00700 1 -- 172.21.15.62:0/468580021 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a1419af70 con 0x7f7a140ffbd0 2024-04-20T22:27:50.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:50.459+0000 7f7a19c00700 1 -- 172.21.15.62:0/468580021 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a141086a0 con 0x7f7a140ffbd0 2024-04-20T22:27:50.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:50.460+0000 7f7a0b400700 1 -- 172.21.15.62:0/468580021 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f7a0400f5d0 con 0x7f7a140ffbd0 2024-04-20T22:27:50.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:50.461+0000 7f7a0b400700 1 --2- 172.21.15.62:0/468580021 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f79f806ecc0 0x7f79f8071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:27:50.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:50.461+0000 7f7a0b400700 1 -- 172.21.15.62:0/468580021 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f7a04090430 con 0x7f7a140ffbd0 2024-04-20T22:27:50.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:50.461+0000 7f7a12000700 1 --2- 172.21.15.62:0/468580021 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f79f806ecc0 0x7f79f8071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:27:50.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:50.462+0000 7f7a12000700 1 --2- 172.21.15.62:0/468580021 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f79f806ecc0 0x7f79f8071170 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f7a00005fd0 tx=0x7f7a00005dc0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:27:50.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:50.466+0000 7f7a0b400700 1 -- 172.21.15.62:0/468580021 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f7a040591c0 con 0x7f7a140ffbd0 2024-04-20T22:27:50.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:27:50 smithi189 ceph-mon[57022]: from='client.36756 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:27:50.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:27:50 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/3335045656' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:27:50.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:27:50 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:27:50.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:27:50 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/3738509244' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:27:50.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:27:50 smithi062 ceph-mon[76322]: from='client.36756 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:27:50.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:27:50 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/3335045656' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:27:50.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:27:50 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:27:50.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:27:50 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/3738509244' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:27:50.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:50.793+0000 7f7a19c00700 1 -- 172.21.15.62:0/468580021 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f7a141972c0 con 0x7f7a140ffbd0 2024-04-20T22:27:50.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:50.793+0000 7f7a0b400700 1 -- 172.21.15.62:0/468580021 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4493 (secure 0 0 0) 0x7f7a0405c880 con 0x7f7a140ffbd0 2024-04-20T22:27:50.794 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-20T22:27:50.794 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-20T22:27:50.794 INFO:teuthology.orchestra.run.smithi062.stdout: host smithi189 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi189/config 2024-04-20T22:27:50.794 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-20T22:27:50.794 INFO:teuthology.orchestra.run.smithi062.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi189 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp11670v2q:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:27:50.794 INFO:teuthology.orchestra.run.smithi062.stdout:Traceback (most recent call last): 2024-04-20T22:27:50.794 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-20T22:27:50.794 INFO:teuthology.orchestra.run.smithi062.stdout: return _run_code(code, main_globals, None, 2024-04-20T22:27:50.794 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-20T22:27:50.794 INFO:teuthology.orchestra.run.smithi062.stdout: exec(code, run_globals) 2024-04-20T22:27:50.794 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5618, in 2024-04-20T22:27:50.795 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5606, in main 2024-04-20T22:27:50.795 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 408, in _infer_config 2024-04-20T22:27:50.795 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 323, in _infer_fsid 2024-04-20T22:27:50.795 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 436, in _infer_image 2024-04-20T22:27:50.795 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 310, in _validate_fsid 2024-04-20T22:27:50.795 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 3359, in command_ceph_volume 2024-04-20T22:27:50.795 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-20T22:27:50.795 INFO:teuthology.orchestra.run.smithi062.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi189 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp11670v2q:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:27:50.795 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-20T22:27:50.795 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is degraded 2024-04-20T22:27:50.795 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T22:27:50.795 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T22:27:50.795 INFO:teuthology.orchestra.run.smithi062.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-20T22:27:50.795 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-20T22:27:50.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:50.796+0000 7f7a19c00700 1 -- 172.21.15.62:0/468580021 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f79f806ecc0 msgr2=0x7f79f8071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:27:50.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:50.796+0000 7f7a19c00700 1 --2- 172.21.15.62:0/468580021 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f79f806ecc0 0x7f79f8071170 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f7a00005fd0 tx=0x7f7a00005dc0 comp rx=0 tx=0).stop 2024-04-20T22:27:50.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:50.796+0000 7f7a19c00700 1 -- 172.21.15.62:0/468580021 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a140ffbd0 msgr2=0x7f7a14195e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:27:50.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:50.796+0000 7f7a19c00700 1 --2- 172.21.15.62:0/468580021 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a140ffbd0 0x7f7a14195e50 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f7a040092e0 tx=0x7f7a0400be00 comp rx=0 tx=0).stop 2024-04-20T22:27:50.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:50.797+0000 7f7a19c00700 1 -- 172.21.15.62:0/468580021 shutdown_connections 2024-04-20T22:27:50.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:50.797+0000 7f7a19c00700 1 --2- 172.21.15.62:0/468580021 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f79f806ecc0 0x7f79f8071170 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:27:50.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:50.797+0000 7f7a19c00700 1 --2- 172.21.15.62:0/468580021 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a140ffbd0 0x7f7a14195e50 unknown :-1 s=CLOSED pgs=1350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:27:50.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:50.797+0000 7f7a19c00700 1 --2- 172.21.15.62:0/468580021 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7a14104130 0x7f7a14196390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:27:50.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:50.797+0000 7f7a19c00700 1 -- 172.21.15.62:0/468580021 >> 172.21.15.62:0/468580021 conn(0x7f7a14075130 msgr2=0x7f7a140fe7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:27:50.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:50.797+0000 7f7a19c00700 1 -- 172.21.15.62:0/468580021 shutdown_connections 2024-04-20T22:27:50.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:27:50.797+0000 7f7a19c00700 1 -- 172.21.15.62:0/468580021 wait complete. 2024-04-20T22:27:51.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:27:51 smithi189 ceph-mon[57022]: pgmap v1579: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:27:51.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:27:51 smithi189 ceph-mon[57022]: from='client.36768 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:27:51.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:27:51 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/468580021' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:27:51.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:27:51 smithi062 ceph-mon[76322]: pgmap v1579: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:27:51.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:27:51 smithi062 ceph-mon[76322]: from='client.36768 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:27:51.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:27:51 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/468580021' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:27:53.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:27:53 smithi189 ceph-mon[57022]: pgmap v1580: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:27:53.773 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:27:53 smithi062 ceph-mon[76322]: pgmap v1580: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:27:55.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:27:55 smithi189 ceph-mon[57022]: pgmap v1581: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:27:55.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:27:55 smithi062 ceph-mon[76322]: pgmap v1581: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:27:57.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:27:57 smithi189 ceph-mon[57022]: pgmap v1582: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:27:57.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:27:57 smithi062 ceph-mon[76322]: pgmap v1582: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:27:59.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:27:59 smithi189 ceph-mon[57022]: pgmap v1583: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:27:59.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:27:59 smithi062 ceph-mon[76322]: pgmap v1583: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:28:01.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:01 smithi189 ceph-mon[57022]: pgmap v1584: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:28:01.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:01 smithi062 ceph-mon[76322]: pgmap v1584: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:28:03.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:03 smithi189 ceph-mon[57022]: pgmap v1585: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:28:03.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:03 smithi062 ceph-mon[76322]: pgmap v1585: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:28:05.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:05 smithi189 ceph-mon[57022]: pgmap v1586: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:28:05.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:05 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:28:05.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:05 smithi062 ceph-mon[76322]: pgmap v1586: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:28:05.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:05 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:28:07.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:07 smithi189 ceph-mon[57022]: pgmap v1587: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:28:07.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:07 smithi062 ceph-mon[76322]: pgmap v1587: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:28:09.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:09 smithi062 ceph-mon[76322]: pgmap v1588: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:28:09.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:09 smithi189 ceph-mon[57022]: pgmap v1588: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:28:11.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:11 smithi189 ceph-mon[57022]: pgmap v1589: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:28:11.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:11 smithi062 ceph-mon[76322]: pgmap v1589: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:28:13.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:13 smithi189 ceph-mon[57022]: pgmap v1590: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:28:13.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:13 smithi062 ceph-mon[76322]: pgmap v1590: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:28:15.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:15 smithi189 ceph-mon[57022]: pgmap v1591: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:28:15.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:15 smithi062 ceph-mon[76322]: pgmap v1591: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:28:17.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:17 smithi189 ceph-mon[57022]: pgmap v1592: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:28:17.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:17 smithi062 ceph-mon[76322]: pgmap v1592: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:28:19.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:19 smithi189 ceph-mon[57022]: pgmap v1593: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:28:19.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:19 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:28:19.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:19 smithi062 ceph-mon[76322]: pgmap v1593: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:28:19.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:19 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:28:20.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:20.994+0000 7f91c7800700 1 -- 172.21.15.62:0/3152928122 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91c01026a0 msgr2=0x7f91c0102a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:28:20.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:20.994+0000 7f91c7800700 1 --2- 172.21.15.62:0/3152928122 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91c01026a0 0x7f91c0102a70 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7f91b400b0a0 tx=0x7f91b400b3b0 comp rx=0 tx=0).stop 2024-04-20T22:28:20.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:20.994+0000 7f91c7800700 1 -- 172.21.15.62:0/3152928122 shutdown_connections 2024-04-20T22:28:20.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:20.994+0000 7f91c7800700 1 --2- 172.21.15.62:0/3152928122 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91c0103040 0x7f91c0107010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:28:20.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:20.994+0000 7f91c7800700 1 --2- 172.21.15.62:0/3152928122 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91c01026a0 0x7f91c0102a70 unknown :-1 s=CLOSED pgs=1351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:28:20.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:20.994+0000 7f91c7800700 1 -- 172.21.15.62:0/3152928122 >> 172.21.15.62:0/3152928122 conn(0x7f91c00fdf40 msgr2=0x7f91c0100350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:28:20.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:20.994+0000 7f91c7800700 1 -- 172.21.15.62:0/3152928122 shutdown_connections 2024-04-20T22:28:20.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:20.994+0000 7f91c7800700 1 -- 172.21.15.62:0/3152928122 wait complete. 2024-04-20T22:28:20.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:20.995+0000 7f91c7800700 1 Processor -- start 2024-04-20T22:28:20.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:20.995+0000 7f91c7800700 1 -- start start 2024-04-20T22:28:20.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:20.996+0000 7f91c7800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91c01026a0 0x7f91c0198210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:28:20.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:20.996+0000 7f91c7800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91c0103040 0x7f91c0198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:28:20.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:20.996+0000 7f91c7800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91c0198e30 con 0x7f91c01026a0 2024-04-20T22:28:20.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:20.996+0000 7f91c7800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91c019cbc0 con 0x7f91c0103040 2024-04-20T22:28:20.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:20.996+0000 7f91c4e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91c01026a0 0x7f91c0198210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:28:20.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:20.996+0000 7f91c4e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91c01026a0 0x7f91c0198210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59204/0 (socket says 172.21.15.62:59204) 2024-04-20T22:28:20.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:20.996+0000 7f91c4e00700 1 -- 172.21.15.62:0/2560560240 learned_addr learned my addr 172.21.15.62:0/2560560240 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:28:20.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:20.996+0000 7f91bfe00700 1 --2- 172.21.15.62:0/2560560240 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91c0103040 0x7f91c0198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:28:20.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:20.996+0000 7f91c4e00700 1 -- 172.21.15.62:0/2560560240 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91c0103040 msgr2=0x7f91c0198750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:28:20.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:20.996+0000 7f91c4e00700 1 --2- 172.21.15.62:0/2560560240 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91c0103040 0x7f91c0198750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:28:20.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:20.996+0000 7f91c4e00700 1 -- 172.21.15.62:0/2560560240 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91b4009d00 con 0x7f91c01026a0 2024-04-20T22:28:20.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:20.996+0000 7f91bfe00700 1 --2- 172.21.15.62:0/2560560240 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91c0103040 0x7f91c0198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T22:28:20.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:20.997+0000 7f91c4e00700 1 --2- 172.21.15.62:0/2560560240 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91c01026a0 0x7f91c0198210 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f91b400b070 tx=0x7f91b400cdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:28:20.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:20.997+0000 7f91bd600700 1 -- 172.21.15.62:0/2560560240 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f91b400e040 con 0x7f91c01026a0 2024-04-20T22:28:20.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:20.997+0000 7f91bd600700 1 -- 172.21.15.62:0/2560560240 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f91b4003e60 con 0x7f91c01026a0 2024-04-20T22:28:20.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:20.997+0000 7f91bd600700 1 -- 172.21.15.62:0/2560560240 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f91b401d910 con 0x7f91c01026a0 2024-04-20T22:28:20.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:20.997+0000 7f91c7800700 1 -- 172.21.15.62:0/2560560240 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91c019ce40 con 0x7f91c01026a0 2024-04-20T22:28:20.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:20.997+0000 7f91c7800700 1 -- 172.21.15.62:0/2560560240 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91c019d330 con 0x7f91c01026a0 2024-04-20T22:28:20.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:20.999+0000 7f91c7800700 1 -- 172.21.15.62:0/2560560240 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91c010a9b0 con 0x7f91c01026a0 2024-04-20T22:28:20.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:20.999+0000 7f91bd600700 1 -- 172.21.15.62:0/2560560240 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f91b4004820 con 0x7f91c01026a0 2024-04-20T22:28:21.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.001+0000 7f91bd600700 1 --2- 172.21.15.62:0/2560560240 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f91a806ecc0 0x7f91a8071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:28:21.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.001+0000 7f91bd600700 1 -- 172.21.15.62:0/2560560240 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f91b4090780 con 0x7f91c01026a0 2024-04-20T22:28:21.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.001+0000 7f91bfe00700 1 --2- 172.21.15.62:0/2560560240 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f91a806ecc0 0x7f91a8071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:28:21.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.002+0000 7f91bfe00700 1 --2- 172.21.15.62:0/2560560240 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f91a806ecc0 0x7f91a8071170 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f91c0199830 tx=0x7f91ac008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:28:21.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.006+0000 7f91bd600700 1 -- 172.21.15.62:0/2560560240 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f91b4059510 con 0x7f91c01026a0 2024-04-20T22:28:21.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.264+0000 7f91c7800700 1 -- 172.21.15.62:0/2560560240 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f91c019d5a0 con 0x7f91a806ecc0 2024-04-20T22:28:21.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.266+0000 7f91bd600700 1 -- 172.21.15.62:0/2560560240 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f91c019d5a0 con 0x7f91a806ecc0 2024-04-20T22:28:21.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.269+0000 7f91c7800700 1 -- 172.21.15.62:0/2560560240 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f91a806ecc0 msgr2=0x7f91a8071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:28:21.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.269+0000 7f91c7800700 1 --2- 172.21.15.62:0/2560560240 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f91a806ecc0 0x7f91a8071170 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f91c0199830 tx=0x7f91ac008040 comp rx=0 tx=0).stop 2024-04-20T22:28:21.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.269+0000 7f91c7800700 1 -- 172.21.15.62:0/2560560240 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91c01026a0 msgr2=0x7f91c0198210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:28:21.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.269+0000 7f91c7800700 1 --2- 172.21.15.62:0/2560560240 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91c01026a0 0x7f91c0198210 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f91b400b070 tx=0x7f91b400cdd0 comp rx=0 tx=0).stop 2024-04-20T22:28:21.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.270+0000 7f91c7800700 1 -- 172.21.15.62:0/2560560240 shutdown_connections 2024-04-20T22:28:21.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.270+0000 7f91c7800700 1 --2- 172.21.15.62:0/2560560240 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f91a806ecc0 0x7f91a8071170 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:28:21.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.270+0000 7f91c7800700 1 --2- 172.21.15.62:0/2560560240 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91c01026a0 0x7f91c0198210 unknown :-1 s=CLOSED pgs=1352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:28:21.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.270+0000 7f91c7800700 1 --2- 172.21.15.62:0/2560560240 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91c0103040 0x7f91c0198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:28:21.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.270+0000 7f91c7800700 1 -- 172.21.15.62:0/2560560240 >> 172.21.15.62:0/2560560240 conn(0x7f91c00fdf40 msgr2=0x7f91c00ff930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:28:21.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.270+0000 7f91c7800700 1 -- 172.21.15.62:0/2560560240 shutdown_connections 2024-04-20T22:28:21.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.270+0000 7f91c7800700 1 -- 172.21.15.62:0/2560560240 wait complete. 2024-04-20T22:28:21.287 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T22:28:21.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.432+0000 7f85c3200700 1 -- 172.21.15.62:0/2848223552 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f85c4103e50 msgr2=0x7f85c41042c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:28:21.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.432+0000 7f85c3200700 1 --2- 172.21.15.62:0/2848223552 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f85c4103e50 0x7f85c41042c0 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f85b0009910 tx=0x7f85b0009c20 comp rx=0 tx=0).stop 2024-04-20T22:28:21.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.433+0000 7f85c3200700 1 -- 172.21.15.62:0/2848223552 shutdown_connections 2024-04-20T22:28:21.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.433+0000 7f85c3200700 1 --2- 172.21.15.62:0/2848223552 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f85c4103e50 0x7f85c41042c0 unknown :-1 s=CLOSED pgs=1353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:28:21.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.433+0000 7f85c3200700 1 --2- 172.21.15.62:0/2848223552 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f85c41025f0 0x7f85c41029c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:28:21.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.433+0000 7f85c3200700 1 -- 172.21.15.62:0/2848223552 >> 172.21.15.62:0/2848223552 conn(0x7f85c40fddb0 msgr2=0x7f85c41001c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:28:21.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.433+0000 7f85c3200700 1 -- 172.21.15.62:0/2848223552 shutdown_connections 2024-04-20T22:28:21.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.433+0000 7f85c3200700 1 -- 172.21.15.62:0/2848223552 wait complete. 2024-04-20T22:28:21.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.434+0000 7f85c3200700 1 Processor -- start 2024-04-20T22:28:21.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.434+0000 7f85c3200700 1 -- start start 2024-04-20T22:28:21.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.434+0000 7f85c3200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f85c41025f0 0x7f85c4197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:28:21.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.434+0000 7f85c3200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f85c4103e50 0x7f85c4198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:28:21.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.434+0000 7f85c3200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85c4198c00 con 0x7f85c4103e50 2024-04-20T22:28:21.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.434+0000 7f85c3200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85c419c990 con 0x7f85c41025f0 2024-04-20T22:28:21.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.435+0000 7f85c1400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f85c4103e50 0x7f85c4198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:28:21.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.435+0000 7f85c1400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f85c4103e50 0x7f85c4198520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53124/0 (socket says 172.21.15.62:53124) 2024-04-20T22:28:21.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.435+0000 7f85c1400700 1 -- 172.21.15.62:0/3061508544 learned_addr learned my addr 172.21.15.62:0/3061508544 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:28:21.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.435+0000 7f85c1e00700 1 --2- 172.21.15.62:0/3061508544 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f85c41025f0 0x7f85c4197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:28:21.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.435+0000 7f85c1400700 1 -- 172.21.15.62:0/3061508544 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f85c41025f0 msgr2=0x7f85c4197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:28:21.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.435+0000 7f85c1400700 1 --2- 172.21.15.62:0/3061508544 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f85c41025f0 0x7f85c4197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:28:21.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.435+0000 7f85c1400700 1 -- 172.21.15.62:0/3061508544 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85b00095f0 con 0x7f85c4103e50 2024-04-20T22:28:21.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.436+0000 7f85c1400700 1 --2- 172.21.15.62:0/3061508544 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f85c4103e50 0x7f85c4198520 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7f85b000b5c0 tx=0x7f85b000be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:28:21.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.436+0000 7f85baa00700 1 -- 172.21.15.62:0/3061508544 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f85b001d070 con 0x7f85c4103e50 2024-04-20T22:28:21.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.436+0000 7f85baa00700 1 -- 172.21.15.62:0/3061508544 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f85b000f460 con 0x7f85c4103e50 2024-04-20T22:28:21.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.436+0000 7f85c3200700 1 -- 172.21.15.62:0/3061508544 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85c419cc10 con 0x7f85c4103e50 2024-04-20T22:28:21.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.436+0000 7f85baa00700 1 -- 172.21.15.62:0/3061508544 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f85b0017610 con 0x7f85c4103e50 2024-04-20T22:28:21.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.436+0000 7f85c3200700 1 -- 172.21.15.62:0/3061508544 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85c419d180 con 0x7f85c4103e50 2024-04-20T22:28:21.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.437+0000 7f85c3200700 1 -- 172.21.15.62:0/3061508544 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85c410a8d0 con 0x7f85c4103e50 2024-04-20T22:28:21.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.439+0000 7f85baa00700 1 -- 172.21.15.62:0/3061508544 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f85b000f5d0 con 0x7f85c4103e50 2024-04-20T22:28:21.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.439+0000 7f85baa00700 1 --2- 172.21.15.62:0/3061508544 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f85a806ecc0 0x7f85a8071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:28:21.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.440+0000 7f85baa00700 1 -- 172.21.15.62:0/3061508544 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f85b0090450 con 0x7f85c4103e50 2024-04-20T22:28:21.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.440+0000 7f85c1e00700 1 --2- 172.21.15.62:0/3061508544 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f85a806ecc0 0x7f85a8071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:28:21.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.441+0000 7f85c1e00700 1 --2- 172.21.15.62:0/3061508544 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f85a806ecc0 0x7f85a8071170 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f85b4006fd0 tx=0x7f85b4008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:28:21.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.445+0000 7f85baa00700 1 -- 172.21.15.62:0/3061508544 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f85b00591e0 con 0x7f85c4103e50 2024-04-20T22:28:21.537 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:21 smithi062 ceph-mon[76322]: pgmap v1594: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:28:21.537 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:21 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T22:28:21.537 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:21 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T22:28:21.537 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:21 smithi062 ceph-mon[76322]: pgmap v1595: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:28:21.537 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:21 smithi062 ceph-mon[76322]: pgmap v1596: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:28:21.537 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:21 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:28:21.537 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:21 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T22:28:21.537 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:21 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:28:21.537 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:21 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:28:21.537 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:21 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:28:21.537 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:21 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:28:21.537 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:21 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:28:21.537 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:21 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:28:21.538 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:21 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi062.ozizwi"}]: dispatch 2024-04-20T22:28:21.538 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:21 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi062.ozizwi"}]': finished 2024-04-20T22:28:21.538 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:21 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi062.ryqblb"}]: dispatch 2024-04-20T22:28:21.538 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:21 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi062.ryqblb"}]': finished 2024-04-20T22:28:21.538 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:21 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi189.dvimer"}]: dispatch 2024-04-20T22:28:21.538 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:21 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi189.dvimer"}]': finished 2024-04-20T22:28:21.538 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:21 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi189.xbgspw"}]: dispatch 2024-04-20T22:28:21.538 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:21 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi189.xbgspw"}]': finished 2024-04-20T22:28:21.538 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:21 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]: dispatch 2024-04-20T22:28:21.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.708+0000 7f85c3200700 1 -- 172.21.15.62:0/3061508544 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f85c419d3f0 con 0x7f85a806ecc0 2024-04-20T22:28:21.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.710+0000 7f85baa00700 1 -- 172.21.15.62:0/3061508544 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f85c419d3f0 con 0x7f85a806ecc0 2024-04-20T22:28:21.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.713+0000 7f85c3200700 1 -- 172.21.15.62:0/3061508544 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f85a806ecc0 msgr2=0x7f85a8071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:28:21.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.713+0000 7f85c3200700 1 --2- 172.21.15.62:0/3061508544 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f85a806ecc0 0x7f85a8071170 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f85b4006fd0 tx=0x7f85b4008040 comp rx=0 tx=0).stop 2024-04-20T22:28:21.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.713+0000 7f85c3200700 1 -- 172.21.15.62:0/3061508544 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f85c4103e50 msgr2=0x7f85c4198520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:28:21.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.713+0000 7f85c3200700 1 --2- 172.21.15.62:0/3061508544 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f85c4103e50 0x7f85c4198520 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7f85b000b5c0 tx=0x7f85b000be00 comp rx=0 tx=0).stop 2024-04-20T22:28:21.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.713+0000 7f85c3200700 1 -- 172.21.15.62:0/3061508544 shutdown_connections 2024-04-20T22:28:21.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.713+0000 7f85c3200700 1 --2- 172.21.15.62:0/3061508544 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f85a806ecc0 0x7f85a8071170 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:28:21.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.713+0000 7f85c3200700 1 --2- 172.21.15.62:0/3061508544 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f85c41025f0 0x7f85c4197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:28:21.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.713+0000 7f85c3200700 1 --2- 172.21.15.62:0/3061508544 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f85c4103e50 0x7f85c4198520 unknown :-1 s=CLOSED pgs=1354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:28:21.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.713+0000 7f85c3200700 1 -- 172.21.15.62:0/3061508544 >> 172.21.15.62:0/3061508544 conn(0x7f85c40fddb0 msgr2=0x7f85c40fe810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:28:21.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.713+0000 7f85c3200700 1 -- 172.21.15.62:0/3061508544 shutdown_connections 2024-04-20T22:28:21.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.713+0000 7f85c3200700 1 -- 172.21.15.62:0/3061508544 wait complete. 2024-04-20T22:28:21.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:21 smithi189 ceph-mon[57022]: pgmap v1594: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:28:21.739 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:21 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T22:28:21.739 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:21 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T22:28:21.739 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:21 smithi189 ceph-mon[57022]: pgmap v1595: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:28:21.739 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:21 smithi189 ceph-mon[57022]: pgmap v1596: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-20T22:28:21.739 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:21 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:28:21.739 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:21 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T22:28:21.739 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:21 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:28:21.739 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:21 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:28:21.739 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:21 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:28:21.739 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:21 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:28:21.739 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:21 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:28:21.739 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:21 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:28:21.740 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:21 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi062.ozizwi"}]: dispatch 2024-04-20T22:28:21.740 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:21 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi062.ozizwi"}]': finished 2024-04-20T22:28:21.740 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:21 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi062.ryqblb"}]: dispatch 2024-04-20T22:28:21.740 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:21 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi062.ryqblb"}]': finished 2024-04-20T22:28:21.740 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:21 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi189.dvimer"}]: dispatch 2024-04-20T22:28:21.740 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:21 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi189.dvimer"}]': finished 2024-04-20T22:28:21.740 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:21 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi189.xbgspw"}]: dispatch 2024-04-20T22:28:21.740 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:21 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi189.xbgspw"}]': finished 2024-04-20T22:28:21.740 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:21 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]: dispatch 2024-04-20T22:28:21.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.877+0000 7f17b0a00700 1 -- 172.21.15.62:0/421383463 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f17ac10c8f0 msgr2=0x7f17ac10ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:28:21.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.877+0000 7f17b0a00700 1 --2- 172.21.15.62:0/421383463 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f17ac10c8f0 0x7f17ac10ccc0 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7f179c009910 tx=0x7f179c009c20 comp rx=0 tx=0).stop 2024-04-20T22:28:21.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.877+0000 7f17b0a00700 1 -- 172.21.15.62:0/421383463 shutdown_connections 2024-04-20T22:28:21.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.877+0000 7f17b0a00700 1 --2- 172.21.15.62:0/421383463 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f17ac106b80 0x7f17ac106ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:28:21.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.877+0000 7f17b0a00700 1 --2- 172.21.15.62:0/421383463 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f17ac10c8f0 0x7f17ac10ccc0 unknown :-1 s=CLOSED pgs=1355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:28:21.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.877+0000 7f17b0a00700 1 -- 172.21.15.62:0/421383463 >> 172.21.15.62:0/421383463 conn(0x7f17ac076060 msgr2=0x7f17ac078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:28:21.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.877+0000 7f17b0a00700 1 -- 172.21.15.62:0/421383463 shutdown_connections 2024-04-20T22:28:21.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.877+0000 7f17b0a00700 1 -- 172.21.15.62:0/421383463 wait complete. 2024-04-20T22:28:21.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.878+0000 7f17b0a00700 1 Processor -- start 2024-04-20T22:28:21.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.879+0000 7f17b0a00700 1 -- start start 2024-04-20T22:28:21.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.879+0000 7f17b0a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f17ac106b80 0x7f17ac19c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:28:21.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.879+0000 7f17b0a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f17ac10c8f0 0x7f17ac19c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:28:21.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.879+0000 7f17b0a00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17ac19d050 con 0x7f17ac106b80 2024-04-20T22:28:21.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.879+0000 7f17b0a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17ac1a0de0 con 0x7f17ac10c8f0 2024-04-20T22:28:21.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.879+0000 7f17a9e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f17ac106b80 0x7f17ac19c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:28:21.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.879+0000 7f17a9e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f17ac106b80 0x7f17ac19c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53128/0 (socket says 172.21.15.62:53128) 2024-04-20T22:28:21.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.879+0000 7f17a9e00700 1 -- 172.21.15.62:0/1609108118 learned_addr learned my addr 172.21.15.62:0/1609108118 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:28:21.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.880+0000 7f17a9e00700 1 -- 172.21.15.62:0/1609108118 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f17ac10c8f0 msgr2=0x7f17ac19c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:28:21.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.880+0000 7f17a1400700 1 --2- 172.21.15.62:0/1609108118 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f17ac10c8f0 0x7f17ac19c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:28:21.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.880+0000 7f17a9e00700 1 --2- 172.21.15.62:0/1609108118 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f17ac10c8f0 0x7f17ac19c970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:28:21.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.880+0000 7f17a9e00700 1 -- 172.21.15.62:0/1609108118 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f179c0095f0 con 0x7f17ac106b80 2024-04-20T22:28:21.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.880+0000 7f17a9e00700 1 --2- 172.21.15.62:0/1609108118 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f17ac106b80 0x7f17ac19c430 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f179c0092e0 tx=0x7f179c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:28:21.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.880+0000 7f17a3400700 1 -- 172.21.15.62:0/1609108118 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f179c01d070 con 0x7f17ac106b80 2024-04-20T22:28:21.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.880+0000 7f17b0a00700 1 -- 172.21.15.62:0/1609108118 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17ac1a1060 con 0x7f17ac106b80 2024-04-20T22:28:21.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.880+0000 7f17a3400700 1 -- 172.21.15.62:0/1609108118 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f179c005e00 con 0x7f17ac106b80 2024-04-20T22:28:21.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.881+0000 7f17b0a00700 1 -- 172.21.15.62:0/1609108118 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17ac1a1550 con 0x7f17ac106b80 2024-04-20T22:28:21.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.881+0000 7f17a3400700 1 -- 172.21.15.62:0/1609108118 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f179c02b440 con 0x7f17ac106b80 2024-04-20T22:28:21.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.882+0000 7f17b0a00700 1 -- 172.21.15.62:0/1609108118 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17ac109b20 con 0x7f17ac106b80 2024-04-20T22:28:21.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.883+0000 7f17a3400700 1 -- 172.21.15.62:0/1609108118 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f179c00f460 con 0x7f17ac106b80 2024-04-20T22:28:21.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.884+0000 7f17a3400700 1 --2- 172.21.15.62:0/1609108118 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f179006ecc0 0x7f1790071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:28:21.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.884+0000 7f17a3400700 1 -- 172.21.15.62:0/1609108118 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f179c090440 con 0x7f17ac106b80 2024-04-20T22:28:21.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.884+0000 7f17a1400700 1 --2- 172.21.15.62:0/1609108118 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f179006ecc0 0x7f1790071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:28:21.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.885+0000 7f17a1400700 1 --2- 172.21.15.62:0/1609108118 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f179006ecc0 0x7f1790071170 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f17ac19da50 tx=0x7f1794009040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:28:21.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:21.889+0000 7f17a3400700 1 -- 172.21.15.62:0/1609108118 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f179c059150 con 0x7f17ac106b80 2024-04-20T22:28:22.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:22.215+0000 7f17b0a00700 1 -- 172.21.15.62:0/1609108118 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f17ac1a1880 con 0x7f179006ecc0 2024-04-20T22:28:22.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:22.238+0000 7f17a3400700 1 -- 172.21.15.62:0/1609108118 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f17ac1a1880 con 0x7f179006ecc0 2024-04-20T22:28:22.238 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T22:28:22.238 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (54m) 15m ago 68m 14.6M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T22:28:22.239 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (68m) 15m ago 68m 11.6M - 18.2.0 dc2bc1663786 2feecf1d6e0c 2024-04-20T22:28:22.239 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (67m) 15m ago 67m 11.2M - 18.2.0 dc2bc1663786 0f32c55dea8f 2024-04-20T22:28:22.239 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (50m) 15m ago 68m 6660k - 19.0.0-3133-ga8c84f59 76bc6b91b726 2825d5ba11c6 2024-04-20T22:28:22.239 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (50m) 15m ago 66m 6665k - 19.0.0-3133-ga8c84f59 76bc6b91b726 20f746fa1bba 2024-04-20T22:28:22.239 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (53m) 15m ago 67m 75.5M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T22:28:22.239 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (45m) 15m ago 60m 22.9M - 19.0.0-3133-ga8c84f59 76bc6b91b726 de80bc90a683 2024-04-20T22:28:22.239 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (46m) 15m ago 60m 23.0M - 19.0.0-3133-ga8c84f59 76bc6b91b726 cd793b7ee98e 2024-04-20T22:28:22.239 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (30m) 15m ago 60m 18.5M - 19.0.0-3133-ga8c84f59 76bc6b91b726 0e7b0bc4d1ab 2024-04-20T22:28:22.239 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (15m) 15m ago 60m 11.4M - 19.0.0-3133-ga8c84f59 76bc6b91b726 0af732ac9e06 2024-04-20T22:28:22.239 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (57m) 15m ago 69m 517M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T22:28:22.239 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (55m) 15m ago 66m 460M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T22:28:22.239 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (51m) 15m ago 70m 86.3M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 019eb1479485 2024-04-20T22:28:22.239 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (51m) 15m ago 66m 58.0M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 55aac560aab6 2024-04-20T22:28:22.240 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (55m) 15m ago 68m 8766k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T22:28:22.240 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (55m) 15m ago 66m 8711k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T22:28:22.240 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (49m) 15m ago 65m 91.3M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 ee2bb901f26e 2024-04-20T22:28:22.240 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (49m) 15m ago 65m 94.9M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 9075152d4162 2024-04-20T22:28:22.240 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (48m) 15m ago 64m 78.5M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 2d4f3e38dc0e 2024-04-20T22:28:22.240 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (48m) 15m ago 63m 94.1M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 8f97b17ae0eb 2024-04-20T22:28:22.240 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (47m) 15m ago 63m 96.1M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 653d8079967d 2024-04-20T22:28:22.240 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (46m) 15m ago 62m 90.9M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 596dded14c12 2024-04-20T22:28:22.240 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (54m) 15m ago 67m 90.4M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T22:28:22.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:22.242+0000 7f17b0a00700 1 -- 172.21.15.62:0/1609108118 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f179006ecc0 msgr2=0x7f1790071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:28:22.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:22.242+0000 7f17b0a00700 1 --2- 172.21.15.62:0/1609108118 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f179006ecc0 0x7f1790071170 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f17ac19da50 tx=0x7f1794009040 comp rx=0 tx=0).stop 2024-04-20T22:28:22.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:22.242+0000 7f17b0a00700 1 -- 172.21.15.62:0/1609108118 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f17ac106b80 msgr2=0x7f17ac19c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:28:22.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:22.242+0000 7f17b0a00700 1 --2- 172.21.15.62:0/1609108118 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f17ac106b80 0x7f17ac19c430 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f179c0092e0 tx=0x7f179c00be20 comp rx=0 tx=0).stop 2024-04-20T22:28:22.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:22.242+0000 7f17b0a00700 1 -- 172.21.15.62:0/1609108118 shutdown_connections 2024-04-20T22:28:22.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:22.242+0000 7f17b0a00700 1 --2- 172.21.15.62:0/1609108118 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f179006ecc0 0x7f1790071170 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:28:22.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:22.242+0000 7f17b0a00700 1 --2- 172.21.15.62:0/1609108118 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f17ac106b80 0x7f17ac19c430 unknown :-1 s=CLOSED pgs=1356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:28:22.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:22.242+0000 7f17b0a00700 1 --2- 172.21.15.62:0/1609108118 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f17ac10c8f0 0x7f17ac19c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:28:22.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:22.242+0000 7f17b0a00700 1 -- 172.21.15.62:0/1609108118 >> 172.21.15.62:0/1609108118 conn(0x7f17ac076060 msgr2=0x7f17ac0779a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:28:22.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:22.243+0000 7f17b0a00700 1 -- 172.21.15.62:0/1609108118 shutdown_connections 2024-04-20T22:28:22.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:22.243+0000 7f17b0a00700 1 -- 172.21.15.62:0/1609108118 wait complete. 2024-04-20T22:28:22.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:22.404+0000 7fdd2b400700 1 -- 172.21.15.62:0/1198948188 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd24068540 msgr2=0x7fdd24068910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:28:22.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:22.404+0000 7fdd2b400700 1 --2- 172.21.15.62:0/1198948188 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd24068540 0x7fdd24068910 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7fdd18009940 tx=0x7fdd18009c50 comp rx=0 tx=0).stop 2024-04-20T22:28:22.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:22.404+0000 7fdd2b400700 1 -- 172.21.15.62:0/1198948188 shutdown_connections 2024-04-20T22:28:22.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:22.404+0000 7fdd2b400700 1 --2- 172.21.15.62:0/1198948188 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd24068ee0 0x7fdd24110a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:28:22.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:22.404+0000 7fdd2b400700 1 --2- 172.21.15.62:0/1198948188 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd24068540 0x7fdd24068910 unknown :-1 s=CLOSED pgs=1357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:28:22.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:22.404+0000 7fdd2b400700 1 -- 172.21.15.62:0/1198948188 >> 172.21.15.62:0/1198948188 conn(0x7fdd24075130 msgr2=0x7fdd24075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:28:22.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:22.404+0000 7fdd2b400700 1 -- 172.21.15.62:0/1198948188 shutdown_connections 2024-04-20T22:28:22.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:22.404+0000 7fdd2b400700 1 -- 172.21.15.62:0/1198948188 wait complete. 2024-04-20T22:28:22.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:22.406+0000 7fdd2b400700 1 Processor -- start 2024-04-20T22:28:22.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:22.406+0000 7fdd2b400700 1 -- start start 2024-04-20T22:28:22.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:22.406+0000 7fdd2b400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd24068ee0 0x7fdd241a2580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:28:22.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:22.406+0000 7fdd2b400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd241a2ac0 0x7fdd2419c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:28:22.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:22.406+0000 7fdd2b400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd241a3060 con 0x7fdd24068ee0 2024-04-20T22:28:22.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:22.406+0000 7fdd2b400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd2419cab0 con 0x7fdd241a2ac0 2024-04-20T22:28:22.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:22.407+0000 7fdd28a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd24068ee0 0x7fdd241a2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:28:22.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:22.407+0000 7fdd28a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd24068ee0 0x7fdd241a2580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53146/0 (socket says 172.21.15.62:53146) 2024-04-20T22:28:22.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:22.407+0000 7fdd28a00700 1 -- 172.21.15.62:0/2699923622 learned_addr learned my addr 172.21.15.62:0/2699923622 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:28:22.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:22.407+0000 7fdd23e00700 1 --2- 172.21.15.62:0/2699923622 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd241a2ac0 0x7fdd2419c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:28:22.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:22.407+0000 7fdd28a00700 1 -- 172.21.15.62:0/2699923622 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd241a2ac0 msgr2=0x7fdd2419c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:28:22.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:22.407+0000 7fdd28a00700 1 --2- 172.21.15.62:0/2699923622 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd241a2ac0 0x7fdd2419c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:28:22.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:22.407+0000 7fdd28a00700 1 -- 172.21.15.62:0/2699923622 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd180095f0 con 0x7fdd24068ee0 2024-04-20T22:28:22.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:22.408+0000 7fdd28a00700 1 --2- 172.21.15.62:0/2699923622 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd24068ee0 0x7fdd241a2580 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7fdd1800b580 tx=0x7fdd180056a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:28:22.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:22.408+0000 7fdd21600700 1 -- 172.21.15.62:0/2699923622 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdd1801d070 con 0x7fdd24068ee0 2024-04-20T22:28:22.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:22.408+0000 7fdd21600700 1 -- 172.21.15.62:0/2699923622 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdd18005dc0 con 0x7fdd24068ee0 2024-04-20T22:28:22.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:22.408+0000 7fdd21600700 1 -- 172.21.15.62:0/2699923622 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdd18021410 con 0x7fdd24068ee0 2024-04-20T22:28:22.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:22.408+0000 7fdd2b400700 1 -- 172.21.15.62:0/2699923622 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd2419cce0 con 0x7fdd24068ee0 2024-04-20T22:28:22.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:22.409+0000 7fdd2b400700 1 -- 172.21.15.62:0/2699923622 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd2419d230 con 0x7fdd24068ee0 2024-04-20T22:28:22.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:22.410+0000 7fdd2b400700 1 -- 172.21.15.62:0/2699923622 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd24101410 con 0x7fdd24068ee0 2024-04-20T22:28:22.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:22.412+0000 7fdd21600700 1 -- 172.21.15.62:0/2699923622 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fdd1800f6f0 con 0x7fdd24068ee0 2024-04-20T22:28:22.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:22.414+0000 7fdd21600700 1 --2- 172.21.15.62:0/2699923622 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fdd0c06ecc0 0x7fdd0c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:28:22.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:22.414+0000 7fdd21600700 1 -- 172.21.15.62:0/2699923622 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fdd1808fed0 con 0x7fdd24068ee0 2024-04-20T22:28:22.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:22.414+0000 7fdd23e00700 1 --2- 172.21.15.62:0/2699923622 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fdd0c06ecc0 0x7fdd0c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:28:22.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:22.415+0000 7fdd23e00700 1 --2- 172.21.15.62:0/2699923622 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fdd0c06ecc0 0x7fdd0c071170 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fdd10005950 tx=0x7fdd100058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:28:22.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:22.419+0000 7fdd21600700 1 -- 172.21.15.62:0/2699923622 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fdd18058c60 con 0x7fdd24068ee0 2024-04-20T22:28:22.496 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:22 smithi062 ceph-mon[76322]: Upgrade: Setting container_image for all mds 2024-04-20T22:28:22.497 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:22 smithi062 ceph-mon[76322]: Upgrade: Setting filesystem cephfs Joinable 2024-04-20T22:28:22.497 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:22 smithi062 ceph-mon[76322]: from='client.36776 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:28:22.497 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:22 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]': finished 2024-04-20T22:28:22.497 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:22 smithi062 ceph-mon[76322]: Standby daemon mds.cephfs.smithi062.ryqblb assigned to filesystem cephfs as rank 0 2024-04-20T22:28:22.497 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:22 smithi062 ceph-mon[76322]: Standby daemon mds.cephfs.smithi062.ozizwi assigned to filesystem cephfs as rank 1 2024-04-20T22:28:22.497 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:22 smithi062 ceph-mon[76322]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-20T22:28:22.497 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:22 smithi062 ceph-mon[76322]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-20T22:28:22.497 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:22 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "1"}]: dispatch 2024-04-20T22:28:22.497 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:22 smithi062 ceph-mon[76322]: fsmap cephfs:2/2 {0=cephfs.smithi062.ryqblb=up:replay,1=cephfs.smithi062.ozizwi=up:replay} 2 up:standby 2024-04-20T22:28:22.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:22.778+0000 7fdd2b400700 1 -- 172.21.15.62:0/2699923622 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fdd2404f2a0 con 0x7fdd24068ee0 2024-04-20T22:28:22.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:22.779+0000 7fdd21600700 1 -- 172.21.15.62:0/2699923622 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7fdd1805c320 con 0x7fdd24068ee0 2024-04-20T22:28:22.779 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:28:22.780 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T22:28:22.780 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:28:22.780 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:28:22.780 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T22:28:22.780 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:28:22.780 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:28:22.780 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T22:28:22.780 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 6 2024-04-20T22:28:22.780 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:28:22.780 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T22:28:22.780 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 4 2024-04-20T22:28:22.780 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:28:22.780 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T22:28:22.780 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 14 2024-04-20T22:28:22.780 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T22:28:22.781 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:28:22.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:22.782+0000 7fdd2b400700 1 -- 172.21.15.62:0/2699923622 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fdd0c06ecc0 msgr2=0x7fdd0c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:28:22.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:22.782+0000 7fdd2b400700 1 --2- 172.21.15.62:0/2699923622 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fdd0c06ecc0 0x7fdd0c071170 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fdd10005950 tx=0x7fdd100058e0 comp rx=0 tx=0).stop 2024-04-20T22:28:22.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:22.782+0000 7fdd2b400700 1 -- 172.21.15.62:0/2699923622 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd24068ee0 msgr2=0x7fdd241a2580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:28:22.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:22.782+0000 7fdd2b400700 1 --2- 172.21.15.62:0/2699923622 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd24068ee0 0x7fdd241a2580 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7fdd1800b580 tx=0x7fdd180056a0 comp rx=0 tx=0).stop 2024-04-20T22:28:22.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:22.783+0000 7fdd2b400700 1 -- 172.21.15.62:0/2699923622 shutdown_connections 2024-04-20T22:28:22.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:22.783+0000 7fdd2b400700 1 --2- 172.21.15.62:0/2699923622 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fdd0c06ecc0 0x7fdd0c071170 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:28:22.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:22.783+0000 7fdd2b400700 1 --2- 172.21.15.62:0/2699923622 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd24068ee0 0x7fdd241a2580 unknown :-1 s=CLOSED pgs=1358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:28:22.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:22.783+0000 7fdd2b400700 1 --2- 172.21.15.62:0/2699923622 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd241a2ac0 0x7fdd2419c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:28:22.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:22.783+0000 7fdd2b400700 1 -- 172.21.15.62:0/2699923622 >> 172.21.15.62:0/2699923622 conn(0x7fdd24075130 msgr2=0x7fdd241106d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:28:22.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:22.783+0000 7fdd2b400700 1 -- 172.21.15.62:0/2699923622 shutdown_connections 2024-04-20T22:28:22.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:22.783+0000 7fdd2b400700 1 -- 172.21.15.62:0/2699923622 wait complete. 2024-04-20T22:28:22.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:22 smithi189 ceph-mon[57022]: Upgrade: Setting container_image for all mds 2024-04-20T22:28:22.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:22 smithi189 ceph-mon[57022]: Upgrade: Setting filesystem cephfs Joinable 2024-04-20T22:28:22.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:22 smithi189 ceph-mon[57022]: from='client.36776 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:28:22.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:22 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]': finished 2024-04-20T22:28:22.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:22 smithi189 ceph-mon[57022]: Standby daemon mds.cephfs.smithi062.ryqblb assigned to filesystem cephfs as rank 0 2024-04-20T22:28:22.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:22 smithi189 ceph-mon[57022]: Standby daemon mds.cephfs.smithi062.ozizwi assigned to filesystem cephfs as rank 1 2024-04-20T22:28:22.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:22 smithi189 ceph-mon[57022]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-20T22:28:22.989 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:22 smithi189 ceph-mon[57022]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-20T22:28:22.989 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:22 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "1"}]: dispatch 2024-04-20T22:28:22.989 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:22 smithi189 ceph-mon[57022]: fsmap cephfs:2/2 {0=cephfs.smithi062.ryqblb=up:replay,1=cephfs.smithi062.ozizwi=up:replay} 2 up:standby 2024-04-20T22:28:23.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:23.010+0000 7fd0c8400700 1 -- 172.21.15.62:0/1684024414 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd0c0068540 msgr2=0x7fd0c00689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:28:23.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:23.010+0000 7fd0c8400700 1 --2- 172.21.15.62:0/1684024414 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd0c0068540 0x7fd0c00689b0 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7fd0ac009910 tx=0x7fd0ac009c20 comp rx=0 tx=0).stop 2024-04-20T22:28:23.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:23.010+0000 7fd0c8400700 1 -- 172.21.15.62:0/1684024414 shutdown_connections 2024-04-20T22:28:23.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:23.010+0000 7fd0c8400700 1 --2- 172.21.15.62:0/1684024414 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd0c0068540 0x7fd0c00689b0 secure :-1 s=CLOSED pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7fd0ac009910 tx=0x7fd0ac009c20 comp rx=0 tx=0).stop 2024-04-20T22:28:23.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:23.010+0000 7fd0c8400700 1 --2- 172.21.15.62:0/1684024414 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd0c0101090 0x7fd0c0101460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:28:23.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:23.010+0000 7fd0c8400700 1 -- 172.21.15.62:0/1684024414 >> 172.21.15.62:0/1684024414 conn(0x7fd0c0075130 msgr2=0x7fd0c0075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:28:23.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:23.011+0000 7fd0c8400700 1 -- 172.21.15.62:0/1684024414 shutdown_connections 2024-04-20T22:28:23.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:23.011+0000 7fd0c8400700 1 -- 172.21.15.62:0/1684024414 wait complete. 2024-04-20T22:28:23.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:23.012+0000 7fd0c8400700 1 Processor -- start 2024-04-20T22:28:23.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:23.012+0000 7fd0c8400700 1 -- start start 2024-04-20T22:28:23.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:23.013+0000 7fd0c8400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd0c0101090 0x7fd0c01982a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:28:23.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:23.013+0000 7fd0c8400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd0c01987e0 0x7fd0c019cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:28:23.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:23.013+0000 7fd0c8400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0c0198df0 con 0x7fd0c01987e0 2024-04-20T22:28:23.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:23.013+0000 7fd0c8400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0c0198f60 con 0x7fd0c0101090 2024-04-20T22:28:23.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:23.013+0000 7fd0c5000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd0c01987e0 0x7fd0c019cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:28:23.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:23.013+0000 7fd0c5a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd0c0101090 0x7fd0c01982a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:28:23.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:23.013+0000 7fd0c5a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd0c0101090 0x7fd0c01982a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.62:60906/0 (socket says 172.21.15.62:60906) 2024-04-20T22:28:23.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:23.013+0000 7fd0c5a00700 1 -- 172.21.15.62:0/3560433524 learned_addr learned my addr 172.21.15.62:0/3560433524 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:28:23.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:23.014+0000 7fd0c5000700 1 -- 172.21.15.62:0/3560433524 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd0c0101090 msgr2=0x7fd0c01982a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:28:23.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:23.014+0000 7fd0c5000700 1 --2- 172.21.15.62:0/3560433524 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd0c0101090 0x7fd0c01982a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:28:23.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:23.014+0000 7fd0c5000700 1 -- 172.21.15.62:0/3560433524 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0ac0095f0 con 0x7fd0c01987e0 2024-04-20T22:28:23.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:23.015+0000 7fd0c5000700 1 --2- 172.21.15.62:0/3560433524 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd0c01987e0 0x7fd0c019cc00 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7fd0c00695e0 tx=0x7fd0ac00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:28:23.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:23.015+0000 7fd0be000700 1 -- 172.21.15.62:0/3560433524 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd0ac01d070 con 0x7fd0c01987e0 2024-04-20T22:28:23.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:23.015+0000 7fd0be000700 1 -- 172.21.15.62:0/3560433524 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd0ac005dc0 con 0x7fd0c01987e0 2024-04-20T22:28:23.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:23.015+0000 7fd0be000700 1 -- 172.21.15.62:0/3560433524 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd0ac00fe80 con 0x7fd0c01987e0 2024-04-20T22:28:23.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:23.016+0000 7fd0c8400700 1 -- 172.21.15.62:0/3560433524 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0c019d1a0 con 0x7fd0c01987e0 2024-04-20T22:28:23.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:23.016+0000 7fd0c8400700 1 -- 172.21.15.62:0/3560433524 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0c019d6a0 con 0x7fd0c01987e0 2024-04-20T22:28:23.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:23.018+0000 7fd0c8400700 1 -- 172.21.15.62:0/3560433524 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0c010a9b0 con 0x7fd0c01987e0 2024-04-20T22:28:23.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:23.019+0000 7fd0be000700 1 -- 172.21.15.62:0/3560433524 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fd0ac00f6f0 con 0x7fd0c01987e0 2024-04-20T22:28:23.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:23.020+0000 7fd0be000700 1 --2- 172.21.15.62:0/3560433524 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd0a806ed10 0x7fd0a80711c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:28:23.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:23.020+0000 7fd0be000700 1 -- 172.21.15.62:0/3560433524 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fd0ac08ff90 con 0x7fd0c01987e0 2024-04-20T22:28:23.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:23.021+0000 7fd0c5a00700 1 --2- 172.21.15.62:0/3560433524 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd0a806ed10 0x7fd0a80711c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:28:23.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:23.023+0000 7fd0c5a00700 1 --2- 172.21.15.62:0/3560433524 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd0a806ed10 0x7fd0a80711c0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fd0b4005950 tx=0x7fd0b400b500 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:28:23.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:23.028+0000 7fd0be000700 1 -- 172.21.15.62:0/3560433524 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fd0ac058da0 con 0x7fd0c01987e0 2024-04-20T22:28:23.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:23.477+0000 7fd0c8400700 1 -- 172.21.15.62:0/3560433524 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fd0c004f2a0 con 0x7fd0c01987e0 2024-04-20T22:28:23.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:23.478+0000 7fd0be000700 1 -- 172.21.15.62:0/3560433524 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 32 v32) v1 ==== 76+0+1969 (secure 0 0 0) 0x7fd0ac05c460 con 0x7fd0c01987e0 2024-04-20T22:28:23.479 INFO:teuthology.orchestra.run.smithi062.stdout:e32 2024-04-20T22:28:23.479 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T22:28:23.479 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:28:23.479 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T22:28:23.479 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:28:23.479 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T22:28:23.479 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T22:28:23.479 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 32 2024-04-20T22:28:23.479 INFO:teuthology.orchestra.run.smithi062.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-20T22:28:23.479 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T22:28:23.479 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T22:28:21.959073+0000 2024-04-20T22:28:23.479 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T22:28:23.479 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T22:28:23.479 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T22:28:23.480 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T22:28:23.480 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T22:28:23.480 INFO:teuthology.orchestra.run.smithi062.stdout:max_xattr_size 65536 2024-04-20T22:28:23.480 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T22:28:23.480 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T22:28:23.480 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 81 2024-04-20T22:28:23.480 INFO:teuthology.orchestra.run.smithi062.stdout:snap_rstat 1 2024-04-20T22:28:23.480 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-20T22:28:23.480 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 2 2024-04-20T22:28:23.480 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1 2024-04-20T22:28:23.480 INFO:teuthology.orchestra.run.smithi062.stdout:up {0=34444,1=34476} 2024-04-20T22:28:23.480 INFO:teuthology.orchestra.run.smithi062.stdout:failed 2024-04-20T22:28:23.480 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T22:28:23.480 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T22:28:23.481 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T22:28:23.481 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T22:28:23.481 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data enabled 2024-04-20T22:28:23.481 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T22:28:23.481 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T22:28:23.481 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 1 2024-04-20T22:28:23.481 INFO:teuthology.orchestra.run.smithi062.stdout:qdb_cluster leader: 0 members: 2024-04-20T22:28:23.481 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ryqblb{0:34444} state up:replay seq 1 join_fscid=1 addr [v2:172.21.15.62:6826/4113916205,v1:172.21.15.62:6827/4113916205] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:28:23.481 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ozizwi{1:34476} state up:replay seq 1 join_fscid=1 addr [v2:172.21.15.62:6828/3801975165,v1:172.21.15.62:6829/3801975165] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:28:23.481 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:28:23.481 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:28:23.481 INFO:teuthology.orchestra.run.smithi062.stdout:Standby daemons: 2024-04-20T22:28:23.481 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:28:23.481 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.dvimer{-1:44727} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6824/3576699232,v1:172.21.15.189:6825/3576699232] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:28:23.481 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.xbgspw{-1:45075} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.189:6826/212905792,v1:172.21.15.189:6827/212905792] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:28:23.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:23.482+0000 7fd0c8400700 1 -- 172.21.15.62:0/3560433524 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd0a806ed10 msgr2=0x7fd0a80711c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:28:23.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:23.482+0000 7fd0c8400700 1 --2- 172.21.15.62:0/3560433524 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd0a806ed10 0x7fd0a80711c0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fd0b4005950 tx=0x7fd0b400b500 comp rx=0 tx=0).stop 2024-04-20T22:28:23.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:23.482+0000 7fd0c8400700 1 -- 172.21.15.62:0/3560433524 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd0c01987e0 msgr2=0x7fd0c019cc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:28:23.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:23.482+0000 7fd0c8400700 1 --2- 172.21.15.62:0/3560433524 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd0c01987e0 0x7fd0c019cc00 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7fd0c00695e0 tx=0x7fd0ac00bd70 comp rx=0 tx=0).stop 2024-04-20T22:28:23.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:23.482+0000 7fd0c8400700 1 -- 172.21.15.62:0/3560433524 shutdown_connections 2024-04-20T22:28:23.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:23.482+0000 7fd0c8400700 1 --2- 172.21.15.62:0/3560433524 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd0a806ed10 0x7fd0a80711c0 secure :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fd0b4005950 tx=0x7fd0b400b500 comp rx=0 tx=0).stop 2024-04-20T22:28:23.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:23.482+0000 7fd0c8400700 1 --2- 172.21.15.62:0/3560433524 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd0c0101090 0x7fd0c01982a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:28:23.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:23.482+0000 7fd0c8400700 1 --2- 172.21.15.62:0/3560433524 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd0c01987e0 0x7fd0c019cc00 secure :-1 s=CLOSED pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7fd0c00695e0 tx=0x7fd0ac00bd70 comp rx=0 tx=0).stop 2024-04-20T22:28:23.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:23.482+0000 7fd0c8400700 1 -- 172.21.15.62:0/3560433524 >> 172.21.15.62:0/3560433524 conn(0x7fd0c0075130 msgr2=0x7fd0c00fdcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:28:23.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:23.483+0000 7fd0c8400700 1 -- 172.21.15.62:0/3560433524 shutdown_connections 2024-04-20T22:28:23.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:23.484+0000 7fd0c8400700 1 -- 172.21.15.62:0/3560433524 wait complete. 2024-04-20T22:28:23.485 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 32 2024-04-20T22:28:23.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:23.651+0000 7f8d86a00700 1 -- 172.21.15.62:0/3529794435 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d80069140 msgr2=0x7f8d80104eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:28:23.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:23.651+0000 7f8d86a00700 1 --2- 172.21.15.62:0/3529794435 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d80069140 0x7f8d80104eb0 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7f8d6c009910 tx=0x7f8d6c009c20 comp rx=0 tx=0).stop 2024-04-20T22:28:23.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:23.651+0000 7f8d86a00700 1 -- 172.21.15.62:0/3529794435 shutdown_connections 2024-04-20T22:28:23.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:23.651+0000 7f8d86a00700 1 --2- 172.21.15.62:0/3529794435 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d80069140 0x7f8d80104eb0 unknown :-1 s=CLOSED pgs=1361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:28:23.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:23.651+0000 7f8d86a00700 1 --2- 172.21.15.62:0/3529794435 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d800687a0 0x7f8d80068b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:28:23.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:23.651+0000 7f8d86a00700 1 -- 172.21.15.62:0/3529794435 >> 172.21.15.62:0/3529794435 conn(0x7f8d80075130 msgr2=0x7f8d80075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:28:23.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:23.651+0000 7f8d86a00700 1 -- 172.21.15.62:0/3529794435 shutdown_connections 2024-04-20T22:28:23.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:23.651+0000 7f8d86a00700 1 -- 172.21.15.62:0/3529794435 wait complete. 2024-04-20T22:28:23.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:23.652+0000 7f8d86a00700 1 Processor -- start 2024-04-20T22:28:23.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:23.653+0000 7f8d86a00700 1 -- start start 2024-04-20T22:28:23.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:23.653+0000 7f8d86a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d800687a0 0x7f8d80198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:28:23.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:23.653+0000 7f8d86a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d80069140 0x7f8d80198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:28:23.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:23.653+0000 7f8d86a00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d80198c70 con 0x7f8d800687a0 2024-04-20T22:28:23.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:23.653+0000 7f8d86a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d8019ca00 con 0x7f8d80069140 2024-04-20T22:28:23.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:23.653+0000 7f8d7fe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d800687a0 0x7f8d80198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:28:23.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:23.654+0000 7f8d7fe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d800687a0 0x7f8d80198050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53184/0 (socket says 172.21.15.62:53184) 2024-04-20T22:28:23.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:23.654+0000 7f8d7fe00700 1 -- 172.21.15.62:0/682876151 learned_addr learned my addr 172.21.15.62:0/682876151 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:28:23.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:23.654+0000 7f8d7f400700 1 --2- 172.21.15.62:0/682876151 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d80069140 0x7f8d80198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:28:23.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:23.654+0000 7f8d7fe00700 1 -- 172.21.15.62:0/682876151 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d80069140 msgr2=0x7f8d80198590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:28:23.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:23.654+0000 7f8d7fe00700 1 --2- 172.21.15.62:0/682876151 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d80069140 0x7f8d80198590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:28:23.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:23.654+0000 7f8d7fe00700 1 -- 172.21.15.62:0/682876151 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d6c0095f0 con 0x7f8d800687a0 2024-04-20T22:28:23.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:23.655+0000 7f8d7fe00700 1 --2- 172.21.15.62:0/682876151 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d800687a0 0x7f8d80198050 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7f8d7000d8d0 tx=0x7f8d7000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:28:23.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:23.655+0000 7f8d7cc00700 1 -- 172.21.15.62:0/682876151 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8d70009940 con 0x7f8d800687a0 2024-04-20T22:28:23.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:23.655+0000 7f8d7cc00700 1 -- 172.21.15.62:0/682876151 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8d70010460 con 0x7f8d800687a0 2024-04-20T22:28:23.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:23.655+0000 7f8d86a00700 1 -- 172.21.15.62:0/682876151 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d8019cce0 con 0x7f8d800687a0 2024-04-20T22:28:23.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:23.655+0000 7f8d86a00700 1 -- 172.21.15.62:0/682876151 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d8019d230 con 0x7f8d800687a0 2024-04-20T22:28:23.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:23.655+0000 7f8d7cc00700 1 -- 172.21.15.62:0/682876151 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8d7000f5f0 con 0x7f8d800687a0 2024-04-20T22:28:23.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:23.657+0000 7f8d86a00700 1 -- 172.21.15.62:0/682876151 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d60005320 con 0x7f8d800687a0 2024-04-20T22:28:23.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:23.659+0000 7f8d7cc00700 1 -- 172.21.15.62:0/682876151 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f8d70010ad0 con 0x7f8d800687a0 2024-04-20T22:28:23.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:23.660+0000 7f8d7cc00700 1 --2- 172.21.15.62:0/682876151 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f8d6406ecc0 0x7f8d64071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:28:23.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:23.660+0000 7f8d7cc00700 1 -- 172.21.15.62:0/682876151 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f8d7008ee40 con 0x7f8d800687a0 2024-04-20T22:28:23.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:23.662+0000 7f8d7f400700 1 --2- 172.21.15.62:0/682876151 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f8d6406ecc0 0x7f8d64071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:28:23.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:23.664+0000 7f8d7f400700 1 --2- 172.21.15.62:0/682876151 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f8d6406ecc0 0x7f8d64071170 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f8d80199670 tx=0x7f8d6c01a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:28:23.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:23.671+0000 7f8d7cc00700 1 -- 172.21.15.62:0/682876151 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f8d70057bd0 con 0x7f8d800687a0 2024-04-20T22:28:23.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:23 smithi062 ceph-mon[76322]: from='client.36780 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:28:23.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:23 smithi062 ceph-mon[76322]: Upgrade: Enabling allow_standby_replay on filesystem cephfs 2024-04-20T22:28:23.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:23 smithi062 ceph-mon[76322]: from='client.36784 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:28:23.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:23 smithi062 ceph-mon[76322]: pgmap v1597: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 4.5 MiB/s rd, 1 op/s 2024-04-20T22:28:23.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:23 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/2699923622' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:28:23.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:23 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "1"}]': finished 2024-04-20T22:28:23.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:23 smithi062 ceph-mon[76322]: fsmap cephfs:2/2 {0=cephfs.smithi062.ryqblb=up:replay,1=cephfs.smithi062.ozizwi=up:replay} 2 up:standby 2024-04-20T22:28:23.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:23 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:28:23.787 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:23 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:28:23.787 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:23 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:28:23.787 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:23 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:28:23.787 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:23 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:28:23.787 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:23 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:28:23.787 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:23 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/3560433524' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:28:23.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:23.953+0000 7f8d86a00700 1 -- 172.21.15.62:0/682876151 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8d60000bf0 con 0x7f8d6406ecc0 2024-04-20T22:28:23.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:23.954+0000 7f8d7cc00700 1 -- 172.21.15.62:0/682876151 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+441 (secure 0 0 0) 0x7f8d60000bf0 con 0x7f8d6406ecc0 2024-04-20T22:28:23.955 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:28:23.955 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T22:28:23.955 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T22:28:23.955 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-20T22:28:23.955 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T22:28:23.955 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr", 2024-04-20T22:28:23.955 INFO:teuthology.orchestra.run.smithi062.stdout: "osd", 2024-04-20T22:28:23.955 INFO:teuthology.orchestra.run.smithi062.stdout: "mds", 2024-04-20T22:28:23.955 INFO:teuthology.orchestra.run.smithi062.stdout: "crash", 2024-04-20T22:28:23.955 INFO:teuthology.orchestra.run.smithi062.stdout: "mon" 2024-04-20T22:28:23.955 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T22:28:23.955 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "16/23 daemons upgraded", 2024-04-20T22:28:23.955 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading ceph-exporter daemons", 2024-04-20T22:28:23.955 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T22:28:23.956 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:28:23.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:23.957+0000 7f8d86a00700 1 -- 172.21.15.62:0/682876151 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f8d6406ecc0 msgr2=0x7f8d64071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:28:23.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:23.957+0000 7f8d86a00700 1 --2- 172.21.15.62:0/682876151 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f8d6406ecc0 0x7f8d64071170 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f8d80199670 tx=0x7f8d6c01a040 comp rx=0 tx=0).stop 2024-04-20T22:28:23.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:23.957+0000 7f8d86a00700 1 -- 172.21.15.62:0/682876151 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d800687a0 msgr2=0x7f8d80198050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:28:23.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:23.957+0000 7f8d86a00700 1 --2- 172.21.15.62:0/682876151 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d800687a0 0x7f8d80198050 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7f8d7000d8d0 tx=0x7f8d7000dc90 comp rx=0 tx=0).stop 2024-04-20T22:28:23.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:23.958+0000 7f8d86a00700 1 -- 172.21.15.62:0/682876151 shutdown_connections 2024-04-20T22:28:23.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:23.958+0000 7f8d86a00700 1 --2- 172.21.15.62:0/682876151 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f8d6406ecc0 0x7f8d64071170 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:28:23.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:23.958+0000 7f8d86a00700 1 --2- 172.21.15.62:0/682876151 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d800687a0 0x7f8d80198050 unknown :-1 s=CLOSED pgs=1362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:28:23.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:23.958+0000 7f8d86a00700 1 --2- 172.21.15.62:0/682876151 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d80069140 0x7f8d80198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:28:23.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:23.958+0000 7f8d86a00700 1 -- 172.21.15.62:0/682876151 >> 172.21.15.62:0/682876151 conn(0x7f8d80075130 msgr2=0x7f8d80101560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:28:23.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:23.958+0000 7f8d86a00700 1 -- 172.21.15.62:0/682876151 shutdown_connections 2024-04-20T22:28:23.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:23.959+0000 7f8d86a00700 1 -- 172.21.15.62:0/682876151 wait complete. 2024-04-20T22:28:23.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:23 smithi189 ceph-mon[57022]: from='client.36780 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:28:23.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:23 smithi189 ceph-mon[57022]: Upgrade: Enabling allow_standby_replay on filesystem cephfs 2024-04-20T22:28:23.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:23 smithi189 ceph-mon[57022]: from='client.36784 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:28:23.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:23 smithi189 ceph-mon[57022]: pgmap v1597: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 4.5 MiB/s rd, 1 op/s 2024-04-20T22:28:23.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:23 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/2699923622' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:28:23.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:23 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "1"}]': finished 2024-04-20T22:28:23.989 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:23 smithi189 ceph-mon[57022]: fsmap cephfs:2/2 {0=cephfs.smithi062.ryqblb=up:replay,1=cephfs.smithi062.ozizwi=up:replay} 2 up:standby 2024-04-20T22:28:23.989 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:23 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:28:23.989 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:23 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:28:23.989 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:23 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:28:23.989 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:23 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:28:23.989 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:23 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:28:23.989 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:23 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:28:23.989 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:23 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/3560433524' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:28:24.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:24.133+0000 7f8e6aa00700 1 -- 172.21.15.62:0/1250635280 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e64108470 msgr2=0x7f8e64108840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:28:24.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:24.133+0000 7f8e6aa00700 1 --2- 172.21.15.62:0/1250635280 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e64108470 0x7f8e64108840 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7f8e54009910 tx=0x7f8e54009c20 comp rx=0 tx=0).stop 2024-04-20T22:28:24.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:24.133+0000 7f8e6aa00700 1 -- 172.21.15.62:0/1250635280 shutdown_connections 2024-04-20T22:28:24.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:24.133+0000 7f8e6aa00700 1 --2- 172.21.15.62:0/1250635280 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e64102470 0x7f8e641028e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:28:24.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:24.133+0000 7f8e6aa00700 1 --2- 172.21.15.62:0/1250635280 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e64108470 0x7f8e64108840 secure :-1 s=CLOSED pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7f8e54009910 tx=0x7f8e54009c20 comp rx=0 tx=0).stop 2024-04-20T22:28:24.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:24.133+0000 7f8e6aa00700 1 -- 172.21.15.62:0/1250635280 >> 172.21.15.62:0/1250635280 conn(0x7f8e640fdf90 msgr2=0x7f8e641003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:28:24.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:24.134+0000 7f8e6aa00700 1 -- 172.21.15.62:0/1250635280 shutdown_connections 2024-04-20T22:28:24.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:24.134+0000 7f8e6aa00700 1 -- 172.21.15.62:0/1250635280 wait complete. 2024-04-20T22:28:24.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:24.135+0000 7f8e6aa00700 1 Processor -- start 2024-04-20T22:28:24.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:24.136+0000 7f8e6aa00700 1 -- start start 2024-04-20T22:28:24.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:24.136+0000 7f8e6aa00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e64102470 0x7f8e6419d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:28:24.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:24.136+0000 7f8e6aa00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e6419db70 0x7f8e641a24b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:28:24.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:24.136+0000 7f8e6aa00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e6419e180 con 0x7f8e6419db70 2024-04-20T22:28:24.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:24.136+0000 7f8e6aa00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e6419e2f0 con 0x7f8e64102470 2024-04-20T22:28:24.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:24.137+0000 7f8e63e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e64102470 0x7f8e6419d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:28:24.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:24.137+0000 7f8e63e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e64102470 0x7f8e6419d630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.62:60932/0 (socket says 172.21.15.62:60932) 2024-04-20T22:28:24.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:24.137+0000 7f8e63e00700 1 -- 172.21.15.62:0/390122519 learned_addr learned my addr 172.21.15.62:0/390122519 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:28:24.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:24.137+0000 7f8e63400700 1 --2- 172.21.15.62:0/390122519 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e6419db70 0x7f8e641a24b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:28:24.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:24.138+0000 7f8e63400700 1 -- 172.21.15.62:0/390122519 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e64102470 msgr2=0x7f8e6419d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:28:24.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:24.138+0000 7f8e63400700 1 --2- 172.21.15.62:0/390122519 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e64102470 0x7f8e6419d630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:28:24.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:24.138+0000 7f8e63400700 1 -- 172.21.15.62:0/390122519 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e540095f0 con 0x7f8e6419db70 2024-04-20T22:28:24.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:24.139+0000 7f8e63400700 1 --2- 172.21.15.62:0/390122519 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e6419db70 0x7f8e641a24b0 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7f8e5000d900 tx=0x7f8e5000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:28:24.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:24.139+0000 7f8e60c00700 1 -- 172.21.15.62:0/390122519 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8e50009900 con 0x7f8e6419db70 2024-04-20T22:28:24.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:24.139+0000 7f8e60c00700 1 -- 172.21.15.62:0/390122519 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8e50010460 con 0x7f8e6419db70 2024-04-20T22:28:24.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:24.139+0000 7f8e60c00700 1 -- 172.21.15.62:0/390122519 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8e5000f5f0 con 0x7f8e6419db70 2024-04-20T22:28:24.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:24.139+0000 7f8e6aa00700 1 -- 172.21.15.62:0/390122519 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e641a2ab0 con 0x7f8e6419db70 2024-04-20T22:28:24.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:24.139+0000 7f8e6aa00700 1 -- 172.21.15.62:0/390122519 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e641a3000 con 0x7f8e6419db70 2024-04-20T22:28:24.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:24.141+0000 7f8e6aa00700 1 -- 172.21.15.62:0/390122519 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e6410a9e0 con 0x7f8e6419db70 2024-04-20T22:28:24.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:24.143+0000 7f8e60c00700 1 -- 172.21.15.62:0/390122519 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f8e50009a60 con 0x7f8e6419db70 2024-04-20T22:28:24.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:24.144+0000 7f8e60c00700 1 --2- 172.21.15.62:0/390122519 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f8e4806ed10 0x7f8e480711c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:28:24.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:24.145+0000 7f8e60c00700 1 -- 172.21.15.62:0/390122519 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f8e5008eb10 con 0x7f8e6419db70 2024-04-20T22:28:24.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:24.145+0000 7f8e63e00700 1 --2- 172.21.15.62:0/390122519 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f8e4806ed10 0x7f8e480711c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:28:24.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:24.147+0000 7f8e63e00700 1 --2- 172.21.15.62:0/390122519 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f8e4806ed10 0x7f8e480711c0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f8e5400b5c0 tx=0x7f8e5401a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:28:24.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:24.151+0000 7f8e60c00700 1 -- 172.21.15.62:0/390122519 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f8e5005b3d0 con 0x7f8e6419db70 2024-04-20T22:28:24.654 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:24 smithi062 ceph-mon[76322]: Upgrade: Setting container_image for all rgw 2024-04-20T22:28:24.654 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:24 smithi062 ceph-mon[76322]: Upgrade: Setting container_image for all rbd-mirror 2024-04-20T22:28:24.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:24.795+0000 7f8e6aa00700 1 -- 172.21.15.62:0/390122519 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f8e641a32d0 con 0x7f8e6419db70 2024-04-20T22:28:24.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:24.796+0000 7f8e60c00700 1 -- 172.21.15.62:0/390122519 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+8336 (secure 0 0 0) 0x7f8e50057030 con 0x7f8e6419db70 2024-04-20T22:28:24.798 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_WARN failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data 2024-04-20T22:28:24.798 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-20T22:28:24.798 INFO:teuthology.orchestra.run.smithi062.stdout: host smithi189 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi189/config 2024-04-20T22:28:24.801 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-20T22:28:24.801 INFO:teuthology.orchestra.run.smithi062.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi189 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpi9rrfoe5:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:28:24.801 INFO:teuthology.orchestra.run.smithi062.stdout:Traceback (most recent call last): 2024-04-20T22:28:24.801 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-20T22:28:24.801 INFO:teuthology.orchestra.run.smithi062.stdout: return _run_code(code, main_globals, None, 2024-04-20T22:28:24.801 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-20T22:28:24.801 INFO:teuthology.orchestra.run.smithi062.stdout: exec(code, run_globals) 2024-04-20T22:28:24.802 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5618, in 2024-04-20T22:28:24.802 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5606, in main 2024-04-20T22:28:24.802 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 408, in _infer_config 2024-04-20T22:28:24.802 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 323, in _infer_fsid 2024-04-20T22:28:24.802 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 436, in _infer_image 2024-04-20T22:28:24.802 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 310, in _validate_fsid 2024-04-20T22:28:24.802 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 3359, in command_ceph_volume 2024-04-20T22:28:24.802 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-20T22:28:24.802 INFO:teuthology.orchestra.run.smithi062.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi189 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpi9rrfoe5:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:28:24.802 INFO:teuthology.orchestra.run.smithi062.stdout: host smithi062 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T22:28:24.802 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-20T22:28:24.802 INFO:teuthology.orchestra.run.smithi062.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi062 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpfparopx2:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:28:24.802 INFO:teuthology.orchestra.run.smithi062.stdout:Traceback (most recent call last): 2024-04-20T22:28:24.802 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-20T22:28:24.803 INFO:teuthology.orchestra.run.smithi062.stdout: return _run_code(code, main_globals, None, 2024-04-20T22:28:24.803 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-20T22:28:24.803 INFO:teuthology.orchestra.run.smithi062.stdout: exec(code, run_globals) 2024-04-20T22:28:24.803 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5618, in 2024-04-20T22:28:24.803 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5606, in main 2024-04-20T22:28:24.803 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 408, in _infer_config 2024-04-20T22:28:24.803 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 323, in _infer_fsid 2024-04-20T22:28:24.803 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 436, in _infer_image 2024-04-20T22:28:24.803 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 310, in _validate_fsid 2024-04-20T22:28:24.803 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 3359, in command_ceph_volume 2024-04-20T22:28:24.803 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-20T22:28:24.803 INFO:teuthology.orchestra.run.smithi062.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi062 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpfparopx2:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:28:24.803 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-20T22:28:24.803 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs is degraded 2024-04-20T22:28:24.803 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T22:28:24.804 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T22:28:24.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:24.800+0000 7f8e6aa00700 1 -- 172.21.15.62:0/390122519 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f8e4806ed10 msgr2=0x7f8e480711c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:28:24.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:24.800+0000 7f8e6aa00700 1 --2- 172.21.15.62:0/390122519 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f8e4806ed10 0x7f8e480711c0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f8e5400b5c0 tx=0x7f8e5401a040 comp rx=0 tx=0).stop 2024-04-20T22:28:24.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:24.800+0000 7f8e6aa00700 1 -- 172.21.15.62:0/390122519 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e6419db70 msgr2=0x7f8e641a24b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:28:24.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:24.800+0000 7f8e6aa00700 1 --2- 172.21.15.62:0/390122519 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e6419db70 0x7f8e641a24b0 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7f8e5000d900 tx=0x7f8e5000dc10 comp rx=0 tx=0).stop 2024-04-20T22:28:24.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:24.801+0000 7f8e6aa00700 1 -- 172.21.15.62:0/390122519 shutdown_connections 2024-04-20T22:28:24.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:24.801+0000 7f8e6aa00700 1 --2- 172.21.15.62:0/390122519 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f8e4806ed10 0x7f8e480711c0 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:28:24.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:24.801+0000 7f8e6aa00700 1 --2- 172.21.15.62:0/390122519 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e64102470 0x7f8e6419d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:28:24.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:24.801+0000 7f8e6aa00700 1 --2- 172.21.15.62:0/390122519 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e6419db70 0x7f8e641a24b0 unknown :-1 s=CLOSED pgs=1364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:28:24.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:24.801+0000 7f8e6aa00700 1 -- 172.21.15.62:0/390122519 >> 172.21.15.62:0/390122519 conn(0x7f8e640fdf90 msgr2=0x7f8e640ff490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:28:24.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:24.801+0000 7f8e6aa00700 1 -- 172.21.15.62:0/390122519 shutdown_connections 2024-04-20T22:28:24.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:24.801+0000 7f8e6aa00700 1 -- 172.21.15.62:0/390122519 wait complete. 2024-04-20T22:28:24.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:24 smithi189 ceph-mon[57022]: Upgrade: Setting container_image for all rgw 2024-04-20T22:28:24.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:24 smithi189 ceph-mon[57022]: Upgrade: Setting container_image for all rbd-mirror 2024-04-20T22:28:25.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:25 smithi062 ceph-mon[76322]: from='client.36796 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:28:25.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:25 smithi062 ceph-mon[76322]: pgmap v1598: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 12 MiB/s rd, 5 op/s 2024-04-20T22:28:25.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:25 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/390122519' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:28:25.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:25 smithi189 ceph-mon[57022]: from='client.36796 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:28:25.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:25 smithi189 ceph-mon[57022]: pgmap v1598: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 12 MiB/s rd, 5 op/s 2024-04-20T22:28:25.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:25 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/390122519' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:28:27.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:26 smithi189 ceph-mon[57022]: Upgrade: Updating ceph-exporter.smithi062 (1/2) 2024-04-20T22:28:27.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:26 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:28:27.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:26 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi062", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-20T22:28:27.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:26 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T22:28:27.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:26 smithi189 ceph-mon[57022]: Deploying daemon ceph-exporter.smithi062 on smithi062 2024-04-20T22:28:27.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:26 smithi189 ceph-mon[57022]: pgmap v1599: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 16 MiB/s rd, 6 op/s 2024-04-20T22:28:27.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:26 smithi062 ceph-mon[76322]: Upgrade: Updating ceph-exporter.smithi062 (1/2) 2024-04-20T22:28:27.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:26 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:28:27.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:26 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi062", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-20T22:28:27.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:26 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T22:28:27.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:26 smithi062 ceph-mon[76322]: Deploying daemon ceph-exporter.smithi062 on smithi062 2024-04-20T22:28:27.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:26 smithi062 ceph-mon[76322]: pgmap v1599: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 16 MiB/s rd, 6 op/s 2024-04-20T22:28:29.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:28 smithi189 ceph-mon[57022]: mds.? [v2:172.21.15.62:6828/3801975165,v1:172.21.15.62:6829/3801975165] up:resolve 2024-04-20T22:28:29.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:28 smithi189 ceph-mon[57022]: fsmap cephfs:2/2 {0=cephfs.smithi062.ryqblb=up:replay,1=cephfs.smithi062.ozizwi=up:resolve} 2 up:standby 2024-04-20T22:28:29.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:28 smithi189 ceph-mon[57022]: pgmap v1600: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 28 MiB/s rd, 9 op/s 2024-04-20T22:28:29.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:28 smithi062 ceph-mon[76322]: mds.? [v2:172.21.15.62:6828/3801975165,v1:172.21.15.62:6829/3801975165] up:resolve 2024-04-20T22:28:29.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:28 smithi062 ceph-mon[76322]: fsmap cephfs:2/2 {0=cephfs.smithi062.ryqblb=up:replay,1=cephfs.smithi062.ozizwi=up:resolve} 2 up:standby 2024-04-20T22:28:29.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:28 smithi062 ceph-mon[76322]: pgmap v1600: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 28 MiB/s rd, 9 op/s 2024-04-20T22:28:30.980 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:30 smithi062 ceph-mon[76322]: mds.? [v2:172.21.15.62:6826/4113916205,v1:172.21.15.62:6827/4113916205] up:resolve 2024-04-20T22:28:30.980 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:30 smithi062 ceph-mon[76322]: fsmap cephfs:2/2 {0=cephfs.smithi062.ryqblb=up:resolve,1=cephfs.smithi062.ozizwi=up:resolve} 2 up:standby 2024-04-20T22:28:31.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:30 smithi189 ceph-mon[57022]: mds.? [v2:172.21.15.62:6826/4113916205,v1:172.21.15.62:6827/4113916205] up:resolve 2024-04-20T22:28:31.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:30 smithi189 ceph-mon[57022]: fsmap cephfs:2/2 {0=cephfs.smithi062.ryqblb=up:resolve,1=cephfs.smithi062.ozizwi=up:resolve} 2 up:standby 2024-04-20T22:28:31.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:30 smithi189 ceph-mon[57022]: pgmap v1601: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 30 MiB/s rd, 10 op/s 2024-04-20T22:28:31.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:30 smithi062 ceph-mon[76322]: pgmap v1601: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 30 MiB/s rd, 10 op/s 2024-04-20T22:28:31.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:31 smithi189 ceph-mon[57022]: reconnect by client.14524 192.168.0.1:0/3910576540 after 0.00299995 2024-04-20T22:28:31.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:31 smithi189 ceph-mon[57022]: reconnect by client.14524 192.168.0.1:0/3910576540 after 0.00299995 2024-04-20T22:28:31.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:31 smithi189 ceph-mon[57022]: reconnect by client.24329 192.168.0.1:0/2755061771 after 0.00399993 2024-04-20T22:28:31.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:31 smithi189 ceph-mon[57022]: reconnect by client.24329 192.168.0.1:0/2755061771 after 0.00399993 2024-04-20T22:28:31.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:31 smithi189 ceph-mon[57022]: mds.? [v2:172.21.15.62:6828/3801975165,v1:172.21.15.62:6829/3801975165] up:reconnect 2024-04-20T22:28:31.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:31 smithi189 ceph-mon[57022]: mds.? [v2:172.21.15.62:6826/4113916205,v1:172.21.15.62:6827/4113916205] up:reconnect 2024-04-20T22:28:31.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:31 smithi189 ceph-mon[57022]: fsmap cephfs:2/2 {0=cephfs.smithi062.ryqblb=up:reconnect,1=cephfs.smithi062.ozizwi=up:reconnect} 2 up:standby 2024-04-20T22:28:32.222 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:31 smithi062 ceph-mon[76322]: reconnect by client.14524 192.168.0.1:0/3910576540 after 0.00299995 2024-04-20T22:28:32.222 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:31 smithi062 ceph-mon[76322]: reconnect by client.14524 192.168.0.1:0/3910576540 after 0.00299995 2024-04-20T22:28:32.222 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:31 smithi062 ceph-mon[76322]: reconnect by client.24329 192.168.0.1:0/2755061771 after 0.00399993 2024-04-20T22:28:32.222 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:31 smithi062 ceph-mon[76322]: reconnect by client.24329 192.168.0.1:0/2755061771 after 0.00399993 2024-04-20T22:28:32.222 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:31 smithi062 ceph-mon[76322]: mds.? [v2:172.21.15.62:6828/3801975165,v1:172.21.15.62:6829/3801975165] up:reconnect 2024-04-20T22:28:32.222 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:31 smithi062 ceph-mon[76322]: mds.? [v2:172.21.15.62:6826/4113916205,v1:172.21.15.62:6827/4113916205] up:reconnect 2024-04-20T22:28:32.222 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:31 smithi062 ceph-mon[76322]: fsmap cephfs:2/2 {0=cephfs.smithi062.ryqblb=up:reconnect,1=cephfs.smithi062.ozizwi=up:reconnect} 2 up:standby 2024-04-20T22:28:33.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:33 smithi062 ceph-mon[76322]: mds.? [v2:172.21.15.62:6826/4113916205,v1:172.21.15.62:6827/4113916205] up:rejoin 2024-04-20T22:28:33.287 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:33 smithi062 ceph-mon[76322]: mds.? [v2:172.21.15.62:6828/3801975165,v1:172.21.15.62:6829/3801975165] up:rejoin 2024-04-20T22:28:33.287 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:33 smithi062 ceph-mon[76322]: fsmap cephfs:2/2 {0=cephfs.smithi062.ryqblb=up:rejoin,1=cephfs.smithi062.ozizwi=up:rejoin} 2 up:standby 2024-04-20T22:28:33.287 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:33 smithi062 ceph-mon[76322]: daemon mds.cephfs.smithi062.ozizwi is now active in filesystem cephfs as rank 1 2024-04-20T22:28:33.287 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:33 smithi062 ceph-mon[76322]: daemon mds.cephfs.smithi062.ryqblb is now active in filesystem cephfs as rank 0 2024-04-20T22:28:33.287 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:33 smithi062 ceph-mon[76322]: pgmap v1602: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 26 MiB/s rd, 170 B/s wr, 8 op/s 2024-04-20T22:28:33.287 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:33 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:28:33.287 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:33 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:28:33.470 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:32 smithi189 ceph-mon[57022]: mds.? [v2:172.21.15.62:6826/4113916205,v1:172.21.15.62:6827/4113916205] up:rejoin 2024-04-20T22:28:33.470 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:33 smithi189 ceph-mon[57022]: mds.? [v2:172.21.15.62:6828/3801975165,v1:172.21.15.62:6829/3801975165] up:rejoin 2024-04-20T22:28:33.470 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:33 smithi189 ceph-mon[57022]: fsmap cephfs:2/2 {0=cephfs.smithi062.ryqblb=up:rejoin,1=cephfs.smithi062.ozizwi=up:rejoin} 2 up:standby 2024-04-20T22:28:33.470 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:33 smithi189 ceph-mon[57022]: daemon mds.cephfs.smithi062.ozizwi is now active in filesystem cephfs as rank 1 2024-04-20T22:28:33.470 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:33 smithi189 ceph-mon[57022]: daemon mds.cephfs.smithi062.ryqblb is now active in filesystem cephfs as rank 0 2024-04-20T22:28:33.470 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:33 smithi189 ceph-mon[57022]: pgmap v1602: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 26 MiB/s rd, 170 B/s wr, 8 op/s 2024-04-20T22:28:33.470 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:33 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:28:33.470 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:33 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:28:34.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:34 smithi189 ceph-mon[57022]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-20T22:28:34.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:34 smithi189 ceph-mon[57022]: mds.? [v2:172.21.15.62:6828/3801975165,v1:172.21.15.62:6829/3801975165] up:active 2024-04-20T22:28:34.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:34 smithi189 ceph-mon[57022]: mds.? [v2:172.21.15.62:6826/4113916205,v1:172.21.15.62:6827/4113916205] up:active 2024-04-20T22:28:34.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:34 smithi189 ceph-mon[57022]: fsmap cephfs:2 {0=cephfs.smithi062.ryqblb=up:active,1=cephfs.smithi062.ozizwi=up:active} 2 up:standby 2024-04-20T22:28:34.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:34 smithi189 ceph-mon[57022]: fsmap cephfs:2 {0=cephfs.smithi062.ryqblb=up:active,1=cephfs.smithi062.ozizwi=up:active} 2 up:standby-replay 2024-04-20T22:28:34.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:34 smithi062 ceph-mon[76322]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-20T22:28:34.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:34 smithi062 ceph-mon[76322]: mds.? [v2:172.21.15.62:6828/3801975165,v1:172.21.15.62:6829/3801975165] up:active 2024-04-20T22:28:34.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:34 smithi062 ceph-mon[76322]: mds.? [v2:172.21.15.62:6826/4113916205,v1:172.21.15.62:6827/4113916205] up:active 2024-04-20T22:28:34.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:34 smithi062 ceph-mon[76322]: fsmap cephfs:2 {0=cephfs.smithi062.ryqblb=up:active,1=cephfs.smithi062.ozizwi=up:active} 2 up:standby 2024-04-20T22:28:34.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:34 smithi062 ceph-mon[76322]: fsmap cephfs:2 {0=cephfs.smithi062.ryqblb=up:active,1=cephfs.smithi062.ozizwi=up:active} 2 up:standby-replay 2024-04-20T22:28:35.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:35 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:28:35.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:35 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:28:35.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:35 smithi189 ceph-mon[57022]: pgmap v1603: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 25 MiB/s rd, 767 B/s wr, 11 op/s 2024-04-20T22:28:35.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:35 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:28:35.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:35 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:28:35.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:35 smithi062 ceph-mon[76322]: pgmap v1603: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 25 MiB/s rd, 767 B/s wr, 11 op/s 2024-04-20T22:28:37.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:37 smithi189 ceph-mon[57022]: Upgrade: Updating ceph-exporter.smithi189 (2/2) 2024-04-20T22:28:37.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:37 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:28:37.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:37 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi189", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-20T22:28:37.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:37 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T22:28:37.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:37 smithi189 ceph-mon[57022]: Deploying daemon ceph-exporter.smithi189 on smithi189 2024-04-20T22:28:37.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:37 smithi189 ceph-mon[57022]: pgmap v1604: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 23 MiB/s rd, 1.4 KiB/s wr, 11 op/s 2024-04-20T22:28:37.507 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:37 smithi062 ceph-mon[76322]: Upgrade: Updating ceph-exporter.smithi189 (2/2) 2024-04-20T22:28:37.507 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:37 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:28:37.507 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:37 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi189", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-20T22:28:37.507 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:37 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T22:28:37.507 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:37 smithi062 ceph-mon[76322]: Deploying daemon ceph-exporter.smithi189 on smithi189 2024-04-20T22:28:37.507 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:37 smithi062 ceph-mon[76322]: pgmap v1604: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 23 MiB/s rd, 1.4 KiB/s wr, 11 op/s 2024-04-20T22:28:38.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:38 smithi189 ceph-mon[57022]: pgmap v1605: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 32 MiB/s rd, 8.2 KiB/s wr, 18 op/s 2024-04-20T22:28:39.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:38 smithi062 ceph-mon[76322]: pgmap v1605: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 32 MiB/s rd, 8.2 KiB/s wr, 18 op/s 2024-04-20T22:28:40.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:40 smithi189 ceph-mon[57022]: pgmap v1606: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 29 MiB/s rd, 12 KiB/s wr, 20 op/s 2024-04-20T22:28:41.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:40 smithi062 ceph-mon[76322]: pgmap v1606: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 29 MiB/s rd, 12 KiB/s wr, 20 op/s 2024-04-20T22:28:43.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:42 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:28:43.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:42 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:28:43.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:42 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T22:28:43.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:42 smithi189 ceph-mon[57022]: pgmap v1607: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 25 MiB/s rd, 12 KiB/s wr, 19 op/s 2024-04-20T22:28:43.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:42 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:28:43.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:42 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:28:43.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:42 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:28:43.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:42 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:28:43.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:42 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:28:43.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:42 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:28:43.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:42 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T22:28:43.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:42 smithi062 ceph-mon[76322]: pgmap v1607: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 25 MiB/s rd, 12 KiB/s wr, 19 op/s 2024-04-20T22:28:43.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:42 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:28:43.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:42 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:28:43.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:42 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:28:43.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:42 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:28:44.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:44 smithi189 ceph-mon[57022]: pgmap v1608: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 26 MiB/s rd, 12 KiB/s wr, 20 op/s 2024-04-20T22:28:45.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:44 smithi062 ceph-mon[76322]: pgmap v1608: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 26 MiB/s rd, 12 KiB/s wr, 20 op/s 2024-04-20T22:28:46.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:46 smithi062 ceph-mon[76322]: pgmap v1609: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 24 MiB/s rd, 12 KiB/s wr, 16 op/s 2024-04-20T22:28:46.953 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:46 smithi189 ceph-mon[57022]: pgmap v1609: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 24 MiB/s rd, 12 KiB/s wr, 16 op/s 2024-04-20T22:28:48.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:48 smithi189 ceph-mon[57022]: pgmap v1610: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 21 MiB/s rd, 11 KiB/s wr, 14 op/s 2024-04-20T22:28:48.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:48 smithi062 ceph-mon[76322]: pgmap v1610: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 21 MiB/s rd, 11 KiB/s wr, 14 op/s 2024-04-20T22:28:50.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:50 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:28:50.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:50 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:28:50.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:50 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:28:50.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:50 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:28:50.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:50 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:28:50.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:50 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:28:50.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:50 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:28:50.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:50 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:28:50.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:50 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:28:50.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:50 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:28:51.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:51 smithi189 ceph-mon[57022]: pgmap v1611: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 8.1 MiB/s rd, 4.3 KiB/s wr, 6 op/s 2024-04-20T22:28:51.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:51 smithi062 ceph-mon[76322]: pgmap v1611: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 8.1 MiB/s rd, 4.3 KiB/s wr, 6 op/s 2024-04-20T22:28:52.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:52 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:28:52.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:52 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:28:52.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:52 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:28:52.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:52 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:28:52.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:52 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:28:52.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:52 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:28:52.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:52 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:28:52.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:52 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:28:53.699 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:53 smithi062 ceph-mon[76322]: pgmap v1612: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.8 MiB/s rd, 4 op/s 2024-04-20T22:28:53.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:53 smithi189 ceph-mon[57022]: pgmap v1612: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.8 MiB/s rd, 4 op/s 2024-04-20T22:28:54.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:54 smithi189 ceph-mon[57022]: pgmap v1613: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.5 MiB/s rd, 4 op/s 2024-04-20T22:28:54.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:54 smithi062 ceph-mon[76322]: pgmap v1613: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.5 MiB/s rd, 4 op/s 2024-04-20T22:28:54.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:54.974+0000 7fc57b200700 1 -- 172.21.15.62:0/1794137113 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc574100e80 msgr2=0x7fc574104e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:28:54.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:54.974+0000 7fc57b200700 1 --2- 172.21.15.62:0/1794137113 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc574100e80 0x7fc574104e50 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7fc560009910 tx=0x7fc560009c20 comp rx=0 tx=0).stop 2024-04-20T22:28:54.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:54.974+0000 7fc57b200700 1 -- 172.21.15.62:0/1794137113 shutdown_connections 2024-04-20T22:28:54.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:54.974+0000 7fc57b200700 1 --2- 172.21.15.62:0/1794137113 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc574100e80 0x7fc574104e50 unknown :-1 s=CLOSED pgs=1366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:28:54.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:54.974+0000 7fc57b200700 1 --2- 172.21.15.62:0/1794137113 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5741004e0 0x7fc5741008b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:28:54.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:54.974+0000 7fc57b200700 1 -- 172.21.15.62:0/1794137113 >> 172.21.15.62:0/1794137113 conn(0x7fc5740fbda0 msgr2=0x7fc5740fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:28:54.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:54.974+0000 7fc57b200700 1 -- 172.21.15.62:0/1794137113 shutdown_connections 2024-04-20T22:28:54.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:54.974+0000 7fc57b200700 1 -- 172.21.15.62:0/1794137113 wait complete. 2024-04-20T22:28:54.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:54.976+0000 7fc57b200700 1 Processor -- start 2024-04-20T22:28:54.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:54.976+0000 7fc57b200700 1 -- start start 2024-04-20T22:28:54.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:54.976+0000 7fc57b200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc5741004e0 0x7fc574195f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:28:54.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:54.976+0000 7fc57b200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc574100e80 0x7fc574196470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:28:54.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:54.976+0000 7fc57b200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc574196b50 con 0x7fc5741004e0 2024-04-20T22:28:54.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:54.976+0000 7fc57b200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc57419a8e0 con 0x7fc574100e80 2024-04-20T22:28:54.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:54.977+0000 7fc573e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc5741004e0 0x7fc574195f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:28:54.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:54.977+0000 7fc573e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc5741004e0 0x7fc574195f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44224/0 (socket says 172.21.15.62:44224) 2024-04-20T22:28:54.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:54.977+0000 7fc573e00700 1 -- 172.21.15.62:0/1759263902 learned_addr learned my addr 172.21.15.62:0/1759263902 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:28:54.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:54.977+0000 7fc573400700 1 --2- 172.21.15.62:0/1759263902 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc574100e80 0x7fc574196470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:28:54.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:54.977+0000 7fc573e00700 1 -- 172.21.15.62:0/1759263902 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc574100e80 msgr2=0x7fc574196470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:28:54.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:54.977+0000 7fc573e00700 1 --2- 172.21.15.62:0/1759263902 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc574100e80 0x7fc574196470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:28:54.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:54.977+0000 7fc573e00700 1 -- 172.21.15.62:0/1759263902 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5600095f0 con 0x7fc5741004e0 2024-04-20T22:28:54.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:54.978+0000 7fc573e00700 1 --2- 172.21.15.62:0/1759263902 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc5741004e0 0x7fc574195f30 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7fc56400c8a0 tx=0x7fc56400cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:28:54.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:54.978+0000 7fc570c00700 1 -- 172.21.15.62:0/1759263902 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc5640078e0 con 0x7fc5741004e0 2024-04-20T22:28:54.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:54.978+0000 7fc570c00700 1 -- 172.21.15.62:0/1759263902 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc56400f450 con 0x7fc5741004e0 2024-04-20T22:28:54.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:54.978+0000 7fc57b200700 1 -- 172.21.15.62:0/1759263902 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc57419abc0 con 0x7fc5741004e0 2024-04-20T22:28:54.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:54.978+0000 7fc570c00700 1 -- 172.21.15.62:0/1759263902 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc56400e5e0 con 0x7fc5741004e0 2024-04-20T22:28:54.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:54.979+0000 7fc57b200700 1 -- 172.21.15.62:0/1759263902 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc57419b110 con 0x7fc5741004e0 2024-04-20T22:28:54.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:54.980+0000 7fc57b200700 1 -- 172.21.15.62:0/1759263902 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc574108850 con 0x7fc5741004e0 2024-04-20T22:28:54.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:54.981+0000 7fc570c00700 1 -- 172.21.15.62:0/1759263902 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fc56400f5c0 con 0x7fc5741004e0 2024-04-20T22:28:54.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:54.982+0000 7fc570c00700 1 --2- 172.21.15.62:0/1759263902 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc55806ecc0 0x7fc558071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:28:54.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:54.982+0000 7fc570c00700 1 -- 172.21.15.62:0/1759263902 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fc56408ee80 con 0x7fc5741004e0 2024-04-20T22:28:54.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:54.982+0000 7fc573400700 1 --2- 172.21.15.62:0/1759263902 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc55806ecc0 0x7fc558071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:28:54.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:54.983+0000 7fc573400700 1 --2- 172.21.15.62:0/1759263902 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc55806ecc0 0x7fc558071170 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fc574197550 tx=0x7fc56001a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:28:54.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:54.987+0000 7fc570c00700 1 -- 172.21.15.62:0/1759263902 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fc564057c30 con 0x7fc5741004e0 2024-04-20T22:28:55.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:55.250+0000 7fc57b200700 1 -- 172.21.15.62:0/1759263902 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc57419b390 con 0x7fc55806ecc0 2024-04-20T22:28:55.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:55.253+0000 7fc570c00700 1 -- 172.21.15.62:0/1759263902 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+466 (secure 0 0 0) 0x7fc57419b390 con 0x7fc55806ecc0 2024-04-20T22:28:55.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:55.256+0000 7fc57b200700 1 -- 172.21.15.62:0/1759263902 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc55806ecc0 msgr2=0x7fc558071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:28:55.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:55.256+0000 7fc57b200700 1 --2- 172.21.15.62:0/1759263902 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc55806ecc0 0x7fc558071170 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fc574197550 tx=0x7fc56001a040 comp rx=0 tx=0).stop 2024-04-20T22:28:55.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:55.256+0000 7fc57b200700 1 -- 172.21.15.62:0/1759263902 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc5741004e0 msgr2=0x7fc574195f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:28:55.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:55.256+0000 7fc57b200700 1 --2- 172.21.15.62:0/1759263902 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc5741004e0 0x7fc574195f30 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7fc56400c8a0 tx=0x7fc56400cbb0 comp rx=0 tx=0).stop 2024-04-20T22:28:55.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:55.256+0000 7fc57b200700 1 -- 172.21.15.62:0/1759263902 shutdown_connections 2024-04-20T22:28:55.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:55.256+0000 7fc57b200700 1 --2- 172.21.15.62:0/1759263902 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc55806ecc0 0x7fc558071170 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:28:55.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:55.256+0000 7fc57b200700 1 --2- 172.21.15.62:0/1759263902 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc5741004e0 0x7fc574195f30 unknown :-1 s=CLOSED pgs=1367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:28:55.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:55.256+0000 7fc57b200700 1 --2- 172.21.15.62:0/1759263902 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc574100e80 0x7fc574196470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:28:55.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:55.256+0000 7fc57b200700 1 -- 172.21.15.62:0/1759263902 >> 172.21.15.62:0/1759263902 conn(0x7fc5740fbda0 msgr2=0x7fc5740fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:28:55.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:55.257+0000 7fc57b200700 1 -- 172.21.15.62:0/1759263902 shutdown_connections 2024-04-20T22:28:55.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:55.257+0000 7fc57b200700 1 -- 172.21.15.62:0/1759263902 wait complete. 2024-04-20T22:28:55.273 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T22:28:55.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:55.432+0000 7f634b800700 1 -- 172.21.15.62:0/2259655975 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f63440687a0 msgr2=0x7f6344068b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:28:55.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:55.432+0000 7f634b800700 1 --2- 172.21.15.62:0/2259655975 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f63440687a0 0x7f6344068b70 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f6338009910 tx=0x7f6338009c20 comp rx=0 tx=0).stop 2024-04-20T22:28:55.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:55.432+0000 7f634b800700 1 -- 172.21.15.62:0/2259655975 shutdown_connections 2024-04-20T22:28:55.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:55.432+0000 7f634b800700 1 --2- 172.21.15.62:0/2259655975 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6344069140 0x7f6344104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:28:55.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:55.432+0000 7f634b800700 1 --2- 172.21.15.62:0/2259655975 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f63440687a0 0x7f6344068b70 unknown :-1 s=CLOSED pgs=1368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:28:55.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:55.432+0000 7f634b800700 1 -- 172.21.15.62:0/2259655975 >> 172.21.15.62:0/2259655975 conn(0x7f6344075130 msgr2=0x7f6344075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:28:55.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:55.433+0000 7f634b800700 1 -- 172.21.15.62:0/2259655975 shutdown_connections 2024-04-20T22:28:55.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:55.433+0000 7f634b800700 1 -- 172.21.15.62:0/2259655975 wait complete. 2024-04-20T22:28:55.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:55.434+0000 7f634b800700 1 Processor -- start 2024-04-20T22:28:55.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:55.434+0000 7f634b800700 1 -- start start 2024-04-20T22:28:55.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:55.434+0000 7f634b800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f63440687a0 0x7f63440ffca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:28:55.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:55.434+0000 7f634b800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6344069140 0x7f63441001e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:28:55.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:55.435+0000 7f634b800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6344100720 con 0x7f63440687a0 2024-04-20T22:28:55.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:55.435+0000 7f634b800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6344100860 con 0x7f6344069140 2024-04-20T22:28:55.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:55.435+0000 7f6348e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f63440687a0 0x7f63440ffca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:28:55.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:55.435+0000 7f6348e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f63440687a0 0x7f63440ffca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44254/0 (socket says 172.21.15.62:44254) 2024-04-20T22:28:55.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:55.435+0000 7f6348e00700 1 -- 172.21.15.62:0/1231111704 learned_addr learned my addr 172.21.15.62:0/1231111704 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:28:55.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:55.435+0000 7f6343e00700 1 --2- 172.21.15.62:0/1231111704 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6344069140 0x7f63441001e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:28:55.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:55.435+0000 7f6348e00700 1 -- 172.21.15.62:0/1231111704 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6344069140 msgr2=0x7f63441001e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:28:55.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:55.435+0000 7f6348e00700 1 --2- 172.21.15.62:0/1231111704 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6344069140 0x7f63441001e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:28:55.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:55.435+0000 7f6348e00700 1 -- 172.21.15.62:0/1231111704 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63380095f0 con 0x7f63440687a0 2024-04-20T22:28:55.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:55.436+0000 7f6348e00700 1 --2- 172.21.15.62:0/1231111704 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f63440687a0 0x7f63440ffca0 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7f63380092e0 tx=0x7f633800b6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:28:55.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:55.436+0000 7f6341600700 1 -- 172.21.15.62:0/1231111704 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f633801d070 con 0x7f63440687a0 2024-04-20T22:28:55.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:55.436+0000 7f6341600700 1 -- 172.21.15.62:0/1231111704 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f63380052a0 con 0x7f63440687a0 2024-04-20T22:28:55.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:55.436+0000 7f634b800700 1 -- 172.21.15.62:0/1231111704 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6344100ae0 con 0x7f63440687a0 2024-04-20T22:28:55.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:55.436+0000 7f6341600700 1 -- 172.21.15.62:0/1231111704 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6338017400 con 0x7f63440687a0 2024-04-20T22:28:55.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:55.437+0000 7f634b800700 1 -- 172.21.15.62:0/1231111704 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63441a23e0 con 0x7f63440687a0 2024-04-20T22:28:55.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:55.438+0000 7f634b800700 1 -- 172.21.15.62:0/1231111704 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6344108790 con 0x7f63440687a0 2024-04-20T22:28:55.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:55.439+0000 7f6341600700 1 -- 172.21.15.62:0/1231111704 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f633800f820 con 0x7f63440687a0 2024-04-20T22:28:55.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:55.440+0000 7f6341600700 1 --2- 172.21.15.62:0/1231111704 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f632c06ecc0 0x7f632c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:28:55.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:55.440+0000 7f6341600700 1 -- 172.21.15.62:0/1231111704 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f6338090220 con 0x7f63440687a0 2024-04-20T22:28:55.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:55.440+0000 7f6343e00700 1 --2- 172.21.15.62:0/1231111704 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f632c06ecc0 0x7f632c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:28:55.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:55.441+0000 7f6343e00700 1 --2- 172.21.15.62:0/1231111704 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f632c06ecc0 0x7f632c071170 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f6344101490 tx=0x7f6330009260 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:28:55.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:55.446+0000 7f6341600700 1 -- 172.21.15.62:0/1231111704 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f6338058f30 con 0x7f63440687a0 2024-04-20T22:28:55.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:55.761+0000 7f634b800700 1 -- 172.21.15.62:0/1231111704 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6344061190 con 0x7f632c06ecc0 2024-04-20T22:28:55.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:55.763+0000 7f6341600700 1 -- 172.21.15.62:0/1231111704 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+466 (secure 0 0 0) 0x7f6344061190 con 0x7f632c06ecc0 2024-04-20T22:28:55.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:55.765+0000 7f634b800700 1 -- 172.21.15.62:0/1231111704 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f632c06ecc0 msgr2=0x7f632c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:28:55.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:55.765+0000 7f634b800700 1 --2- 172.21.15.62:0/1231111704 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f632c06ecc0 0x7f632c071170 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f6344101490 tx=0x7f6330009260 comp rx=0 tx=0).stop 2024-04-20T22:28:55.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:55.765+0000 7f634b800700 1 -- 172.21.15.62:0/1231111704 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f63440687a0 msgr2=0x7f63440ffca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:28:55.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:55.765+0000 7f634b800700 1 --2- 172.21.15.62:0/1231111704 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f63440687a0 0x7f63440ffca0 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7f63380092e0 tx=0x7f633800b6c0 comp rx=0 tx=0).stop 2024-04-20T22:28:55.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:55.765+0000 7f634b800700 1 -- 172.21.15.62:0/1231111704 shutdown_connections 2024-04-20T22:28:55.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:55.765+0000 7f634b800700 1 --2- 172.21.15.62:0/1231111704 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f632c06ecc0 0x7f632c071170 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:28:55.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:55.765+0000 7f634b800700 1 --2- 172.21.15.62:0/1231111704 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f63440687a0 0x7f63440ffca0 unknown :-1 s=CLOSED pgs=1369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:28:55.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:55.765+0000 7f634b800700 1 --2- 172.21.15.62:0/1231111704 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6344069140 0x7f63441001e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:28:55.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:55.765+0000 7f634b800700 1 -- 172.21.15.62:0/1231111704 >> 172.21.15.62:0/1231111704 conn(0x7f6344075130 msgr2=0x7f63440fe7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:28:55.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:55.766+0000 7f634b800700 1 -- 172.21.15.62:0/1231111704 shutdown_connections 2024-04-20T22:28:55.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:55.766+0000 7f634b800700 1 -- 172.21.15.62:0/1231111704 wait complete. 2024-04-20T22:28:55.912 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:55 smithi062 ceph-mon[76322]: from='client.36808 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:28:55.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:55.916+0000 7f538f200700 1 -- 172.21.15.62:0/3640935375 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5390100c50 msgr2=0x7f5390104c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:28:55.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:55.916+0000 7f538f200700 1 --2- 172.21.15.62:0/3640935375 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5390100c50 0x7f5390104c20 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7f5378009910 tx=0x7f5378009c20 comp rx=0 tx=0).stop 2024-04-20T22:28:55.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:55.917+0000 7f538f200700 1 -- 172.21.15.62:0/3640935375 shutdown_connections 2024-04-20T22:28:55.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:55.917+0000 7f538f200700 1 --2- 172.21.15.62:0/3640935375 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5390100c50 0x7f5390104c20 unknown :-1 s=CLOSED pgs=1370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:28:55.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:55.917+0000 7f538f200700 1 --2- 172.21.15.62:0/3640935375 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53901002b0 0x7f5390100680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:28:55.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:55.917+0000 7f538f200700 1 -- 172.21.15.62:0/3640935375 >> 172.21.15.62:0/3640935375 conn(0x7f53900fbb50 msgr2=0x7f53900fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:28:55.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:55.917+0000 7f538f200700 1 -- 172.21.15.62:0/3640935375 shutdown_connections 2024-04-20T22:28:55.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:55.917+0000 7f538f200700 1 -- 172.21.15.62:0/3640935375 wait complete. 2024-04-20T22:28:55.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:55.918+0000 7f538f200700 1 Processor -- start 2024-04-20T22:28:55.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:55.918+0000 7f538f200700 1 -- start start 2024-04-20T22:28:55.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:55.918+0000 7f538f200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53901002b0 0x7f5390195ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:28:55.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:55.918+0000 7f538f200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5390100c50 0x7f53901961e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:28:55.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:55.918+0000 7f538f200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5390196870 con 0x7f5390100c50 2024-04-20T22:28:55.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:55.918+0000 7f538f200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53901995a0 con 0x7f53901002b0 2024-04-20T22:28:55.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:55.919+0000 7f538d400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5390100c50 0x7f53901961e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:28:55.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:55.919+0000 7f538d400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5390100c50 0x7f53901961e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44266/0 (socket says 172.21.15.62:44266) 2024-04-20T22:28:55.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:55.919+0000 7f538d400700 1 -- 172.21.15.62:0/2304951217 learned_addr learned my addr 172.21.15.62:0/2304951217 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:28:55.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:55.919+0000 7f538de00700 1 --2- 172.21.15.62:0/2304951217 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53901002b0 0x7f5390195ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:28:55.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:55.919+0000 7f538d400700 1 -- 172.21.15.62:0/2304951217 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53901002b0 msgr2=0x7f5390195ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:28:55.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:55.919+0000 7f538d400700 1 --2- 172.21.15.62:0/2304951217 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53901002b0 0x7f5390195ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:28:55.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:55.919+0000 7f538d400700 1 -- 172.21.15.62:0/2304951217 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53780095f0 con 0x7f5390100c50 2024-04-20T22:28:55.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:55.920+0000 7f538d400700 1 --2- 172.21.15.62:0/2304951217 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5390100c50 0x7f53901961e0 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7f537800bda0 tx=0x7f537800bdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:28:55.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:55.920+0000 7f5386a00700 1 -- 172.21.15.62:0/2304951217 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f537801d070 con 0x7f5390100c50 2024-04-20T22:28:55.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:55.920+0000 7f5386a00700 1 -- 172.21.15.62:0/2304951217 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5378005dc0 con 0x7f5390100c50 2024-04-20T22:28:55.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:55.920+0000 7f5386a00700 1 -- 172.21.15.62:0/2304951217 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5378017400 con 0x7f5390100c50 2024-04-20T22:28:55.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:55.920+0000 7f538f200700 1 -- 172.21.15.62:0/2304951217 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5390199820 con 0x7f5390100c50 2024-04-20T22:28:55.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:55.920+0000 7f538f200700 1 -- 172.21.15.62:0/2304951217 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5390199d10 con 0x7f5390100c50 2024-04-20T22:28:55.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:55.922+0000 7f538f200700 1 -- 172.21.15.62:0/2304951217 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5390062380 con 0x7f5390100c50 2024-04-20T22:28:55.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:55.923+0000 7f5386a00700 1 -- 172.21.15.62:0/2304951217 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f537800f460 con 0x7f5390100c50 2024-04-20T22:28:55.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:55.924+0000 7f5386a00700 1 --2- 172.21.15.62:0/2304951217 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f537406ecc0 0x7f5374071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:28:55.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:55.924+0000 7f5386a00700 1 -- 172.21.15.62:0/2304951217 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f53780902e0 con 0x7f5390100c50 2024-04-20T22:28:55.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:55.924+0000 7f538de00700 1 --2- 172.21.15.62:0/2304951217 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f537406ecc0 0x7f5374071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:28:55.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:55.925+0000 7f538de00700 1 --2- 172.21.15.62:0/2304951217 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f537406ecc0 0x7f5374071170 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f5388009550 tx=0x7f53880091b0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:28:55.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:55.929+0000 7f5386a00700 1 -- 172.21.15.62:0/2304951217 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f5378058ff0 con 0x7f5390100c50 2024-04-20T22:28:55.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:55 smithi189 ceph-mon[57022]: from='client.36808 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:28:56.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:56.176+0000 7f538f200700 1 -- 172.21.15.62:0/2304951217 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f5390199f80 con 0x7f537406ecc0 2024-04-20T22:28:56.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:56.189+0000 7f5386a00700 1 -- 172.21.15.62:0/2304951217 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f5390199f80 con 0x7f537406ecc0 2024-04-20T22:28:56.190 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T22:28:56.190 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (54m) 6s ago 68m 14.4M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T22:28:56.190 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (24s) 6s ago 68m 7876k - 19.0.0-3133-ga8c84f59 76bc6b91b726 a994518fe0b6 2024-04-20T22:28:56.191 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (14s) 6s ago 67m 7105k - 19.0.0-3133-ga8c84f59 76bc6b91b726 3f5760a59416 2024-04-20T22:28:56.191 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (50m) 6s ago 68m 6660k - 19.0.0-3133-ga8c84f59 76bc6b91b726 2825d5ba11c6 2024-04-20T22:28:56.191 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (50m) 6s ago 67m 6665k - 19.0.0-3133-ga8c84f59 76bc6b91b726 20f746fa1bba 2024-04-20T22:28:56.191 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (54m) 6s ago 68m 75.6M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T22:28:56.191 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (46m) 6s ago 61m 62.3M - 19.0.0-3133-ga8c84f59 76bc6b91b726 de80bc90a683 2024-04-20T22:28:56.191 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (46m) 6s ago 61m 64.2M - 19.0.0-3133-ga8c84f59 76bc6b91b726 cd793b7ee98e 2024-04-20T22:28:56.191 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (31m) 6s ago 61m 62.8M - 19.0.0-3133-ga8c84f59 76bc6b91b726 0e7b0bc4d1ab 2024-04-20T22:28:56.191 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (15m) 6s ago 61m 60.7M - 19.0.0-3133-ga8c84f59 76bc6b91b726 0af732ac9e06 2024-04-20T22:28:56.191 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (57m) 6s ago 70m 521M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T22:28:56.191 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (56m) 6s ago 67m 460M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T22:28:56.191 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (52m) 6s ago 70m 103M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 019eb1479485 2024-04-20T22:28:56.192 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (51m) 6s ago 67m 66.9M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 55aac560aab6 2024-04-20T22:28:56.192 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (55m) 6s ago 68m 8975k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T22:28:56.192 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (55m) 6s ago 67m 8850k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T22:28:56.192 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (50m) 6s ago 66m 98.6M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 ee2bb901f26e 2024-04-20T22:28:56.192 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (49m) 6s ago 65m 104M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 9075152d4162 2024-04-20T22:28:56.192 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (49m) 6s ago 65m 86.8M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 2d4f3e38dc0e 2024-04-20T22:28:56.192 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (48m) 6s ago 64m 104M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 8f97b17ae0eb 2024-04-20T22:28:56.192 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (47m) 6s ago 63m 103M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 653d8079967d 2024-04-20T22:28:56.192 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (47m) 6s ago 63m 97.4M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 596dded14c12 2024-04-20T22:28:56.192 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (55m) 6s ago 68m 89.5M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T22:28:56.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:56.192+0000 7f538f200700 1 -- 172.21.15.62:0/2304951217 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f537406ecc0 msgr2=0x7f5374071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:28:56.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:56.192+0000 7f538f200700 1 --2- 172.21.15.62:0/2304951217 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f537406ecc0 0x7f5374071170 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f5388009550 tx=0x7f53880091b0 comp rx=0 tx=0).stop 2024-04-20T22:28:56.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:56.192+0000 7f538f200700 1 -- 172.21.15.62:0/2304951217 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5390100c50 msgr2=0x7f53901961e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:28:56.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:56.192+0000 7f538f200700 1 --2- 172.21.15.62:0/2304951217 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5390100c50 0x7f53901961e0 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7f537800bda0 tx=0x7f537800bdd0 comp rx=0 tx=0).stop 2024-04-20T22:28:56.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:56.192+0000 7f538f200700 1 -- 172.21.15.62:0/2304951217 shutdown_connections 2024-04-20T22:28:56.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:56.192+0000 7f538f200700 1 --2- 172.21.15.62:0/2304951217 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f537406ecc0 0x7f5374071170 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:28:56.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:56.192+0000 7f538f200700 1 --2- 172.21.15.62:0/2304951217 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53901002b0 0x7f5390195ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:28:56.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:56.192+0000 7f538f200700 1 --2- 172.21.15.62:0/2304951217 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5390100c50 0x7f53901961e0 unknown :-1 s=CLOSED pgs=1371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:28:56.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:56.192+0000 7f538f200700 1 -- 172.21.15.62:0/2304951217 >> 172.21.15.62:0/2304951217 conn(0x7f53900fbb50 msgr2=0x7f53900fd150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:28:56.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:56.193+0000 7f538f200700 1 -- 172.21.15.62:0/2304951217 shutdown_connections 2024-04-20T22:28:56.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:56.193+0000 7f538f200700 1 -- 172.21.15.62:0/2304951217 wait complete. 2024-04-20T22:28:56.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:56.349+0000 7fe658c00700 1 -- 172.21.15.62:0/88078993 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe654106b80 msgr2=0x7fe654106ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:28:56.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:56.349+0000 7fe658c00700 1 --2- 172.21.15.62:0/88078993 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe654106b80 0x7fe654106ff0 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7fe63c009910 tx=0x7fe63c009c20 comp rx=0 tx=0).stop 2024-04-20T22:28:56.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:56.350+0000 7fe658c00700 1 -- 172.21.15.62:0/88078993 shutdown_connections 2024-04-20T22:28:56.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:56.350+0000 7fe658c00700 1 --2- 172.21.15.62:0/88078993 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe654106b80 0x7fe654106ff0 unknown :-1 s=CLOSED pgs=1372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:28:56.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:56.350+0000 7fe658c00700 1 --2- 172.21.15.62:0/88078993 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe65410c660 0x7fe65410ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:28:56.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:56.350+0000 7fe658c00700 1 -- 172.21.15.62:0/88078993 >> 172.21.15.62:0/88078993 conn(0x7fe654075e90 msgr2=0x7fe6540782a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:28:56.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:56.350+0000 7fe658c00700 1 -- 172.21.15.62:0/88078993 shutdown_connections 2024-04-20T22:28:56.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:56.350+0000 7fe658c00700 1 -- 172.21.15.62:0/88078993 wait complete. 2024-04-20T22:28:56.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:56.351+0000 7fe658c00700 1 Processor -- start 2024-04-20T22:28:56.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:56.351+0000 7fe658c00700 1 -- start start 2024-04-20T22:28:56.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:56.352+0000 7fe658c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe654106b80 0x7fe65419c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:28:56.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:56.352+0000 7fe658c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe65410c660 0x7fe65419c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:28:56.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:56.352+0000 7fe658c00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe65419ce00 con 0x7fe65410c660 2024-04-20T22:28:56.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:56.352+0000 7fe658c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6541a0b90 con 0x7fe654106b80 2024-04-20T22:28:56.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:56.352+0000 7fe64be00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe65410c660 0x7fe65419c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:28:56.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:56.352+0000 7fe64be00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe65410c660 0x7fe65419c720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44276/0 (socket says 172.21.15.62:44276) 2024-04-20T22:28:56.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:56.352+0000 7fe64be00700 1 -- 172.21.15.62:0/753223763 learned_addr learned my addr 172.21.15.62:0/753223763 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:28:56.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:56.352+0000 7fe653400700 1 --2- 172.21.15.62:0/753223763 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe654106b80 0x7fe65419c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:28:56.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:56.352+0000 7fe64be00700 1 -- 172.21.15.62:0/753223763 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe654106b80 msgr2=0x7fe65419c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:28:56.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:56.352+0000 7fe64be00700 1 --2- 172.21.15.62:0/753223763 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe654106b80 0x7fe65419c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:28:56.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:56.352+0000 7fe64be00700 1 -- 172.21.15.62:0/753223763 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe63c0095f0 con 0x7fe65410c660 2024-04-20T22:28:56.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:56.353+0000 7fe64be00700 1 --2- 172.21.15.62:0/753223763 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe65410c660 0x7fe65419c720 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7fe63c00b5c0 tx=0x7fe63c00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:28:56.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:56.353+0000 7fe650c00700 1 -- 172.21.15.62:0/753223763 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe63c01d070 con 0x7fe65410c660 2024-04-20T22:28:56.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:56.353+0000 7fe658c00700 1 -- 172.21.15.62:0/753223763 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6541a0e10 con 0x7fe65410c660 2024-04-20T22:28:56.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:56.353+0000 7fe650c00700 1 -- 172.21.15.62:0/753223763 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe63c005e00 con 0x7fe65410c660 2024-04-20T22:28:56.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:56.353+0000 7fe650c00700 1 -- 172.21.15.62:0/753223763 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe63c017400 con 0x7fe65410c660 2024-04-20T22:28:56.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:56.353+0000 7fe658c00700 1 -- 172.21.15.62:0/753223763 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6541a1300 con 0x7fe65410c660 2024-04-20T22:28:56.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:56.354+0000 7fe658c00700 1 -- 172.21.15.62:0/753223763 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe65410eb40 con 0x7fe65410c660 2024-04-20T22:28:56.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:56.355+0000 7fe650c00700 1 -- 172.21.15.62:0/753223763 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe63c00f6f0 con 0x7fe65410c660 2024-04-20T22:28:56.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:56.356+0000 7fe650c00700 1 --2- 172.21.15.62:0/753223763 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe63806ecc0 0x7fe638071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:28:56.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:56.356+0000 7fe650c00700 1 -- 172.21.15.62:0/753223763 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fe63c0902f0 con 0x7fe65410c660 2024-04-20T22:28:56.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:56.357+0000 7fe653400700 1 --2- 172.21.15.62:0/753223763 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe63806ecc0 0x7fe638071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:28:56.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:56.357+0000 7fe653400700 1 --2- 172.21.15.62:0/753223763 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe63806ecc0 0x7fe638071170 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fe644005950 tx=0x7fe64400b410 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:28:56.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:56.361+0000 7fe650c00700 1 -- 172.21.15.62:0/753223763 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fe63c0590a0 con 0x7fe65410c660 2024-04-20T22:28:56.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:56.697+0000 7fe658c00700 1 -- 172.21.15.62:0/753223763 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fe654062380 con 0x7fe65410c660 2024-04-20T22:28:56.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:56.698+0000 7fe650c00700 1 -- 172.21.15.62:0/753223763 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7fe63c05c760 con 0x7fe65410c660 2024-04-20T22:28:56.698 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:28:56.698 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T22:28:56.698 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:28:56.699 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:28:56.699 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T22:28:56.699 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:28:56.699 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:28:56.699 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T22:28:56.699 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 6 2024-04-20T22:28:56.699 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:28:56.699 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T22:28:56.699 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 4 2024-04-20T22:28:56.699 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:28:56.699 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T22:28:56.699 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 14 2024-04-20T22:28:56.699 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T22:28:56.699 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:28:56.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:56.701+0000 7fe658c00700 1 -- 172.21.15.62:0/753223763 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe63806ecc0 msgr2=0x7fe638071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:28:56.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:56.701+0000 7fe658c00700 1 --2- 172.21.15.62:0/753223763 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe63806ecc0 0x7fe638071170 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fe644005950 tx=0x7fe64400b410 comp rx=0 tx=0).stop 2024-04-20T22:28:56.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:56.701+0000 7fe658c00700 1 -- 172.21.15.62:0/753223763 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe65410c660 msgr2=0x7fe65419c720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:28:56.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:56.701+0000 7fe658c00700 1 --2- 172.21.15.62:0/753223763 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe65410c660 0x7fe65419c720 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7fe63c00b5c0 tx=0x7fe63c00bd70 comp rx=0 tx=0).stop 2024-04-20T22:28:56.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:56.701+0000 7fe658c00700 1 -- 172.21.15.62:0/753223763 shutdown_connections 2024-04-20T22:28:56.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:56.701+0000 7fe658c00700 1 --2- 172.21.15.62:0/753223763 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe63806ecc0 0x7fe638071170 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:28:56.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:56.701+0000 7fe658c00700 1 --2- 172.21.15.62:0/753223763 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe654106b80 0x7fe65419c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:28:56.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:56.701+0000 7fe658c00700 1 --2- 172.21.15.62:0/753223763 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe65410c660 0x7fe65419c720 unknown :-1 s=CLOSED pgs=1373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:28:56.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:56.701+0000 7fe658c00700 1 -- 172.21.15.62:0/753223763 >> 172.21.15.62:0/753223763 conn(0x7fe654075e90 msgr2=0x7fe6540778d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:28:56.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:56.701+0000 7fe658c00700 1 -- 172.21.15.62:0/753223763 shutdown_connections 2024-04-20T22:28:56.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:56.702+0000 7fe658c00700 1 -- 172.21.15.62:0/753223763 wait complete. 2024-04-20T22:28:56.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:56.864+0000 7f93e2400700 1 -- 172.21.15.62:0/843660633 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f93dc100e80 msgr2=0x7f93dc104e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:28:56.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:56.864+0000 7f93e2400700 1 --2- 172.21.15.62:0/843660633 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f93dc100e80 0x7f93dc104e50 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f93c8009910 tx=0x7f93c8009c20 comp rx=0 tx=0).stop 2024-04-20T22:28:56.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:56.864+0000 7f93e2400700 1 -- 172.21.15.62:0/843660633 shutdown_connections 2024-04-20T22:28:56.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:56.864+0000 7f93e2400700 1 --2- 172.21.15.62:0/843660633 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f93dc100e80 0x7f93dc104e50 unknown :-1 s=CLOSED pgs=1374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:28:56.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:56.864+0000 7f93e2400700 1 --2- 172.21.15.62:0/843660633 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f93dc1004e0 0x7f93dc1008b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:28:56.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:56.864+0000 7f93e2400700 1 -- 172.21.15.62:0/843660633 >> 172.21.15.62:0/843660633 conn(0x7f93dc0fbda0 msgr2=0x7f93dc0fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:28:56.865 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:56 smithi062 ceph-mon[76322]: from='client.36812 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:28:56.865 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:56 smithi062 ceph-mon[76322]: from='client.36816 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:28:56.865 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:56 smithi062 ceph-mon[76322]: pgmap v1614: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-20T22:28:56.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:56.864+0000 7f93e2400700 1 -- 172.21.15.62:0/843660633 shutdown_connections 2024-04-20T22:28:56.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:56.864+0000 7f93e2400700 1 -- 172.21.15.62:0/843660633 wait complete. 2024-04-20T22:28:56.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:56.866+0000 7f93e2400700 1 Processor -- start 2024-04-20T22:28:56.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:56.866+0000 7f93e2400700 1 -- start start 2024-04-20T22:28:56.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:56.866+0000 7f93e2400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f93dc1004e0 0x7f93dc073050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:28:56.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:56.866+0000 7f93e2400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f93dc100e80 0x7f93dc073590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:28:56.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:56.866+0000 7f93e2400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93dc073ad0 con 0x7f93dc100e80 2024-04-20T22:28:56.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:56.866+0000 7f93e2400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93dc073c10 con 0x7f93dc1004e0 2024-04-20T22:28:56.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:56.867+0000 7f93daa00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f93dc100e80 0x7f93dc073590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:28:56.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:56.867+0000 7f93daa00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f93dc100e80 0x7f93dc073590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44298/0 (socket says 172.21.15.62:44298) 2024-04-20T22:28:56.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:56.867+0000 7f93daa00700 1 -- 172.21.15.62:0/1857954457 learned_addr learned my addr 172.21.15.62:0/1857954457 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:28:56.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:56.867+0000 7f93db400700 1 --2- 172.21.15.62:0/1857954457 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f93dc1004e0 0x7f93dc073050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:28:56.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:56.867+0000 7f93daa00700 1 -- 172.21.15.62:0/1857954457 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f93dc1004e0 msgr2=0x7f93dc073050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:28:56.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:56.867+0000 7f93daa00700 1 --2- 172.21.15.62:0/1857954457 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f93dc1004e0 0x7f93dc073050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:28:56.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:56.867+0000 7f93daa00700 1 -- 172.21.15.62:0/1857954457 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93c80095f0 con 0x7f93dc100e80 2024-04-20T22:28:56.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:56.868+0000 7f93daa00700 1 --2- 172.21.15.62:0/1857954457 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f93dc100e80 0x7f93dc073590 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f93c800bda0 tx=0x7f93c800be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:28:56.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:56.868+0000 7f93d3e00700 1 -- 172.21.15.62:0/1857954457 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f93c801d070 con 0x7f93dc100e80 2024-04-20T22:28:56.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:56.868+0000 7f93d3e00700 1 -- 172.21.15.62:0/1857954457 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f93c800f460 con 0x7f93dc100e80 2024-04-20T22:28:56.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:56.868+0000 7f93e2400700 1 -- 172.21.15.62:0/1857954457 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93dc073e90 con 0x7f93dc100e80 2024-04-20T22:28:56.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:56.868+0000 7f93d3e00700 1 -- 172.21.15.62:0/1857954457 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f93c8017610 con 0x7f93dc100e80 2024-04-20T22:28:56.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:56.868+0000 7f93e2400700 1 -- 172.21.15.62:0/1857954457 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93dc1a2410 con 0x7f93dc100e80 2024-04-20T22:28:56.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:56.869+0000 7f93e2400700 1 -- 172.21.15.62:0/1857954457 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93dc108850 con 0x7f93dc100e80 2024-04-20T22:28:56.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:56.871+0000 7f93d3e00700 1 -- 172.21.15.62:0/1857954457 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f93c800fa40 con 0x7f93dc100e80 2024-04-20T22:28:56.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:56.872+0000 7f93d3e00700 1 --2- 172.21.15.62:0/1857954457 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f93c006ecc0 0x7f93c0071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:28:56.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:56.872+0000 7f93d3e00700 1 -- 172.21.15.62:0/1857954457 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f93c80902e0 con 0x7f93dc100e80 2024-04-20T22:28:56.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:56.872+0000 7f93db400700 1 --2- 172.21.15.62:0/1857954457 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f93c006ecc0 0x7f93c0071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:28:56.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:56.873+0000 7f93db400700 1 --2- 172.21.15.62:0/1857954457 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f93c006ecc0 0x7f93c0071170 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f93cc005950 tx=0x7f93cc0058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:28:56.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:56.877+0000 7f93d3e00700 1 -- 172.21.15.62:0/1857954457 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f93c8059070 con 0x7f93dc100e80 2024-04-20T22:28:56.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:56 smithi189 ceph-mon[57022]: from='client.36812 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:28:56.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:56 smithi189 ceph-mon[57022]: from='client.36816 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:28:56.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:56 smithi189 ceph-mon[57022]: pgmap v1614: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-20T22:28:57.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:57.152+0000 7f93e2400700 1 -- 172.21.15.62:0/1857954457 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f93dc108230 con 0x7f93dc100e80 2024-04-20T22:28:57.153 INFO:teuthology.orchestra.run.smithi062.stdout:e38 2024-04-20T22:28:57.153 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T22:28:57.153 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:28:57.153 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T22:28:57.153 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:28:57.153 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T22:28:57.153 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T22:28:57.154 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 38 2024-04-20T22:28:57.154 INFO:teuthology.orchestra.run.smithi062.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-20T22:28:57.154 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T22:28:57.154 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T22:28:33.008821+0000 2024-04-20T22:28:57.154 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T22:28:57.154 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T22:28:57.154 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T22:28:57.154 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T22:28:57.154 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T22:28:57.154 INFO:teuthology.orchestra.run.smithi062.stdout:max_xattr_size 65536 2024-04-20T22:28:57.154 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T22:28:57.154 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T22:28:57.154 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 81 2024-04-20T22:28:57.154 INFO:teuthology.orchestra.run.smithi062.stdout:snap_rstat 1 2024-04-20T22:28:57.154 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-20T22:28:57.155 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 2 2024-04-20T22:28:57.155 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1 2024-04-20T22:28:57.155 INFO:teuthology.orchestra.run.smithi062.stdout:up {0=34444,1=34476} 2024-04-20T22:28:57.155 INFO:teuthology.orchestra.run.smithi062.stdout:failed 2024-04-20T22:28:57.155 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T22:28:57.155 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T22:28:57.155 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T22:28:57.155 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T22:28:57.155 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data enabled 2024-04-20T22:28:57.155 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T22:28:57.155 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T22:28:57.155 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 1 2024-04-20T22:28:57.155 INFO:teuthology.orchestra.run.smithi062.stdout:qdb_cluster leader: 34444 members: 34476,34444 2024-04-20T22:28:57.155 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ryqblb{0:34444} state up:active seq 699 join_fscid=1 addr [v2:172.21.15.62:6826/4113916205,v1:172.21.15.62:6827/4113916205] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:28:57.155 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.dvimer{0:44727} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.189:6824/3576699232,v1:172.21.15.189:6825/3576699232] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:28:57.155 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ozizwi{1:34476} state up:active seq 691 join_fscid=1 addr [v2:172.21.15.62:6828/3801975165,v1:172.21.15.62:6829/3801975165] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:28:57.156 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.xbgspw{1:45075} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.189:6826/212905792,v1:172.21.15.189:6827/212905792] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:28:57.156 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:28:57.156 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:28:57.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:57.153+0000 7f93d3e00700 1 -- 172.21.15.62:0/1857954457 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 38 v38) v1 ==== 76+0+1981 (secure 0 0 0) 0x7f93c805c730 con 0x7f93dc100e80 2024-04-20T22:28:57.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:57.157+0000 7f93e2400700 1 -- 172.21.15.62:0/1857954457 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f93c006ecc0 msgr2=0x7f93c0071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:28:57.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:57.157+0000 7f93e2400700 1 --2- 172.21.15.62:0/1857954457 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f93c006ecc0 0x7f93c0071170 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f93cc005950 tx=0x7f93cc0058e0 comp rx=0 tx=0).stop 2024-04-20T22:28:57.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:57.157+0000 7f93e2400700 1 -- 172.21.15.62:0/1857954457 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f93dc100e80 msgr2=0x7f93dc073590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:28:57.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:57.157+0000 7f93e2400700 1 --2- 172.21.15.62:0/1857954457 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f93dc100e80 0x7f93dc073590 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f93c800bda0 tx=0x7f93c800be80 comp rx=0 tx=0).stop 2024-04-20T22:28:57.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:57.157+0000 7f93e2400700 1 -- 172.21.15.62:0/1857954457 shutdown_connections 2024-04-20T22:28:57.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:57.157+0000 7f93e2400700 1 --2- 172.21.15.62:0/1857954457 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f93c006ecc0 0x7f93c0071170 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:28:57.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:57.157+0000 7f93e2400700 1 --2- 172.21.15.62:0/1857954457 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f93dc1004e0 0x7f93dc073050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:28:57.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:57.157+0000 7f93e2400700 1 --2- 172.21.15.62:0/1857954457 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f93dc100e80 0x7f93dc073590 unknown :-1 s=CLOSED pgs=1375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:28:57.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:57.157+0000 7f93e2400700 1 -- 172.21.15.62:0/1857954457 >> 172.21.15.62:0/1857954457 conn(0x7f93dc0fbda0 msgr2=0x7f93dc0fe180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:28:57.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:57.157+0000 7f93e2400700 1 -- 172.21.15.62:0/1857954457 shutdown_connections 2024-04-20T22:28:57.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:57.157+0000 7f93e2400700 1 -- 172.21.15.62:0/1857954457 wait complete. 2024-04-20T22:28:57.158 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 38 2024-04-20T22:28:57.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:57.318+0000 7f63fcc00700 1 -- 172.21.15.62:0/3006807048 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f63f8102470 msgr2=0x7f63f81028e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:28:57.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:57.318+0000 7f63fcc00700 1 --2- 172.21.15.62:0/3006807048 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f63f8102470 0x7f63f81028e0 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f63e4009910 tx=0x7f63e4009c20 comp rx=0 tx=0).stop 2024-04-20T22:28:57.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:57.318+0000 7f63fcc00700 1 -- 172.21.15.62:0/3006807048 shutdown_connections 2024-04-20T22:28:57.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:57.318+0000 7f63fcc00700 1 --2- 172.21.15.62:0/3006807048 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f63f8102470 0x7f63f81028e0 unknown :-1 s=CLOSED pgs=1376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:28:57.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:57.318+0000 7f63fcc00700 1 --2- 172.21.15.62:0/3006807048 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63f8108470 0x7f63f8108840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:28:57.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:57.318+0000 7f63fcc00700 1 -- 172.21.15.62:0/3006807048 >> 172.21.15.62:0/3006807048 conn(0x7f63f80fdf90 msgr2=0x7f63f81003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:28:57.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:57.319+0000 7f63fcc00700 1 -- 172.21.15.62:0/3006807048 shutdown_connections 2024-04-20T22:28:57.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:57.319+0000 7f63fcc00700 1 -- 172.21.15.62:0/3006807048 wait complete. 2024-04-20T22:28:57.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:57.319+0000 7f63fcc00700 1 Processor -- start 2024-04-20T22:28:57.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:57.320+0000 7f63fcc00700 1 -- start start 2024-04-20T22:28:57.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:57.320+0000 7f63fcc00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63f8102470 0x7f63f81981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:28:57.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:57.320+0000 7f63fcc00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f63f8108470 0x7f63f8198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:28:57.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:57.320+0000 7f63fcc00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63f8198e10 con 0x7f63f8108470 2024-04-20T22:28:57.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:57.320+0000 7f63fcc00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63f819cba0 con 0x7f63f8102470 2024-04-20T22:28:57.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:57.320+0000 7f63f5400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f63f8108470 0x7f63f8198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:28:57.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:57.320+0000 7f63f5400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f63f8108470 0x7f63f8198730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44306/0 (socket says 172.21.15.62:44306) 2024-04-20T22:28:57.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:57.320+0000 7f63f5400700 1 -- 172.21.15.62:0/2853716341 learned_addr learned my addr 172.21.15.62:0/2853716341 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:28:57.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:57.321+0000 7f63f5e00700 1 --2- 172.21.15.62:0/2853716341 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63f8102470 0x7f63f81981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:28:57.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:57.321+0000 7f63f5400700 1 -- 172.21.15.62:0/2853716341 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63f8102470 msgr2=0x7f63f81981f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:28:57.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:57.321+0000 7f63f5400700 1 --2- 172.21.15.62:0/2853716341 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63f8102470 0x7f63f81981f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:28:57.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:57.321+0000 7f63f5400700 1 -- 172.21.15.62:0/2853716341 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63e40095f0 con 0x7f63f8108470 2024-04-20T22:28:57.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:57.321+0000 7f63f5400700 1 --2- 172.21.15.62:0/2853716341 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f63f8108470 0x7f63f8198730 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7f63e400b5c0 tx=0x7f63e400be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:28:57.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:57.321+0000 7f63eea00700 1 -- 172.21.15.62:0/2853716341 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f63e401d070 con 0x7f63f8108470 2024-04-20T22:28:57.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:57.321+0000 7f63eea00700 1 -- 172.21.15.62:0/2853716341 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f63e400f460 con 0x7f63f8108470 2024-04-20T22:28:57.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:57.321+0000 7f63eea00700 1 -- 172.21.15.62:0/2853716341 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f63e4017610 con 0x7f63f8108470 2024-04-20T22:28:57.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:57.322+0000 7f63fcc00700 1 -- 172.21.15.62:0/2853716341 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63f819ce20 con 0x7f63f8108470 2024-04-20T22:28:57.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:57.322+0000 7f63fcc00700 1 -- 172.21.15.62:0/2853716341 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63f819d310 con 0x7f63f8108470 2024-04-20T22:28:57.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:57.323+0000 7f63fcc00700 1 -- 172.21.15.62:0/2853716341 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63f810a970 con 0x7f63f8108470 2024-04-20T22:28:57.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:57.324+0000 7f63eea00700 1 -- 172.21.15.62:0/2853716341 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f63e400f5d0 con 0x7f63f8108470 2024-04-20T22:28:57.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:57.325+0000 7f63eea00700 1 --2- 172.21.15.62:0/2853716341 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f63dc06ecc0 0x7f63dc071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:28:57.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:57.325+0000 7f63eea00700 1 -- 172.21.15.62:0/2853716341 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f63e4090440 con 0x7f63f8108470 2024-04-20T22:28:57.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:57.325+0000 7f63f5e00700 1 --2- 172.21.15.62:0/2853716341 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f63dc06ecc0 0x7f63dc071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:28:57.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:57.326+0000 7f63f5e00700 1 --2- 172.21.15.62:0/2853716341 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f63dc06ecc0 0x7f63dc071170 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f63e8005950 tx=0x7f63e8005cf0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:28:57.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:57.330+0000 7f63eea00700 1 -- 172.21.15.62:0/2853716341 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f63e40591d0 con 0x7f63f8108470 2024-04-20T22:28:57.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:57.587+0000 7f63fcc00700 1 -- 172.21.15.62:0/2853716341 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f63f819d640 con 0x7f63dc06ecc0 2024-04-20T22:28:57.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:57.589+0000 7f63eea00700 1 -- 172.21.15.62:0/2853716341 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+466 (secure 0 0 0) 0x7f63f819d640 con 0x7f63dc06ecc0 2024-04-20T22:28:57.589 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:28:57.589 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1", 2024-04-20T22:28:57.590 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-04-20T22:28:57.590 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-20T22:28:57.590 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-04-20T22:28:57.590 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr", 2024-04-20T22:28:57.590 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph-exporter", 2024-04-20T22:28:57.590 INFO:teuthology.orchestra.run.smithi062.stdout: "osd", 2024-04-20T22:28:57.590 INFO:teuthology.orchestra.run.smithi062.stdout: "mds", 2024-04-20T22:28:57.590 INFO:teuthology.orchestra.run.smithi062.stdout: "crash", 2024-04-20T22:28:57.590 INFO:teuthology.orchestra.run.smithi062.stdout: "mon" 2024-04-20T22:28:57.590 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-20T22:28:57.590 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "18/23 daemons upgraded", 2024-04-20T22:28:57.590 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading ceph-exporter daemons", 2024-04-20T22:28:57.590 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T22:28:57.590 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:28:57.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:57.592+0000 7f63fcc00700 1 -- 172.21.15.62:0/2853716341 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f63dc06ecc0 msgr2=0x7f63dc071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:28:57.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:57.592+0000 7f63fcc00700 1 --2- 172.21.15.62:0/2853716341 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f63dc06ecc0 0x7f63dc071170 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f63e8005950 tx=0x7f63e8005cf0 comp rx=0 tx=0).stop 2024-04-20T22:28:57.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:57.592+0000 7f63fcc00700 1 -- 172.21.15.62:0/2853716341 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f63f8108470 msgr2=0x7f63f8198730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:28:57.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:57.592+0000 7f63fcc00700 1 --2- 172.21.15.62:0/2853716341 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f63f8108470 0x7f63f8198730 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7f63e400b5c0 tx=0x7f63e400be00 comp rx=0 tx=0).stop 2024-04-20T22:28:57.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:57.592+0000 7f63fcc00700 1 -- 172.21.15.62:0/2853716341 shutdown_connections 2024-04-20T22:28:57.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:57.592+0000 7f63fcc00700 1 --2- 172.21.15.62:0/2853716341 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f63dc06ecc0 0x7f63dc071170 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:28:57.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:57.592+0000 7f63fcc00700 1 --2- 172.21.15.62:0/2853716341 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63f8102470 0x7f63f81981f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:28:57.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:57.592+0000 7f63fcc00700 1 --2- 172.21.15.62:0/2853716341 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f63f8108470 0x7f63f8198730 unknown :-1 s=CLOSED pgs=1377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:28:57.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:57.592+0000 7f63fcc00700 1 -- 172.21.15.62:0/2853716341 >> 172.21.15.62:0/2853716341 conn(0x7f63f80fdf90 msgr2=0x7f63f80ff3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:28:57.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:57.593+0000 7f63fcc00700 1 -- 172.21.15.62:0/2853716341 shutdown_connections 2024-04-20T22:28:57.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:57.593+0000 7f63fcc00700 1 -- 172.21.15.62:0/2853716341 wait complete. 2024-04-20T22:28:57.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:57.763+0000 7fc8a6600700 1 -- 172.21.15.62:0/1522155906 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8a01024d0 msgr2=0x7fc8a01028a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:28:57.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:57.763+0000 7fc8a6600700 1 --2- 172.21.15.62:0/1522155906 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8a01024d0 0x7fc8a01028a0 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7fc894009910 tx=0x7fc894009c20 comp rx=0 tx=0).stop 2024-04-20T22:28:57.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:57.763+0000 7fc8a6600700 1 -- 172.21.15.62:0/1522155906 shutdown_connections 2024-04-20T22:28:57.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:57.763+0000 7fc8a6600700 1 --2- 172.21.15.62:0/1522155906 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc8a0102e70 0x7fc8a0106e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:28:57.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:57.763+0000 7fc8a6600700 1 --2- 172.21.15.62:0/1522155906 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8a01024d0 0x7fc8a01028a0 unknown :-1 s=CLOSED pgs=1378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:28:57.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:57.763+0000 7fc8a6600700 1 -- 172.21.15.62:0/1522155906 >> 172.21.15.62:0/1522155906 conn(0x7fc8a00fdd90 msgr2=0x7fc8a01001a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:28:57.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:57.763+0000 7fc8a6600700 1 -- 172.21.15.62:0/1522155906 shutdown_connections 2024-04-20T22:28:57.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:57.763+0000 7fc8a6600700 1 -- 172.21.15.62:0/1522155906 wait complete. 2024-04-20T22:28:57.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:57.764+0000 7fc8a6600700 1 Processor -- start 2024-04-20T22:28:57.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:57.764+0000 7fc8a6600700 1 -- start start 2024-04-20T22:28:57.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:57.765+0000 7fc8a6600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc8a01024d0 0x7fc8a0197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:28:57.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:57.765+0000 7fc8a6600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8a0102e70 0x7fc8a0198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:28:57.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:57.765+0000 7fc8a6600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8a0198b50 con 0x7fc8a0102e70 2024-04-20T22:28:57.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:57.765+0000 7fc8a6600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8a019c8e0 con 0x7fc8a01024d0 2024-04-20T22:28:57.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:57.765+0000 7fc89fe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8a0102e70 0x7fc8a0198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:28:57.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:57.765+0000 7fc89fe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8a0102e70 0x7fc8a0198470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44316/0 (socket says 172.21.15.62:44316) 2024-04-20T22:28:57.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:57.765+0000 7fc89fe00700 1 -- 172.21.15.62:0/1062930222 learned_addr learned my addr 172.21.15.62:0/1062930222 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:28:57.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:57.765+0000 7fc8a5200700 1 --2- 172.21.15.62:0/1062930222 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc8a01024d0 0x7fc8a0197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:28:57.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:57.766+0000 7fc89fe00700 1 -- 172.21.15.62:0/1062930222 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc8a01024d0 msgr2=0x7fc8a0197f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:28:57.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:57.766+0000 7fc89fe00700 1 --2- 172.21.15.62:0/1062930222 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc8a01024d0 0x7fc8a0197f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:28:57.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:57.766+0000 7fc89fe00700 1 -- 172.21.15.62:0/1062930222 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8940095f0 con 0x7fc8a0102e70 2024-04-20T22:28:57.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:57.766+0000 7fc89fe00700 1 --2- 172.21.15.62:0/1062930222 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8a0102e70 0x7fc8a0198470 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7fc88c00b480 tx=0x7fc88c00b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:28:57.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:57.766+0000 7fc89d600700 1 -- 172.21.15.62:0/1062930222 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc88c0118c0 con 0x7fc8a0102e70 2024-04-20T22:28:57.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:57.766+0000 7fc89d600700 1 -- 172.21.15.62:0/1062930222 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc88c010380 con 0x7fc8a0102e70 2024-04-20T22:28:57.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:57.766+0000 7fc89d600700 1 -- 172.21.15.62:0/1062930222 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc88c00f4f0 con 0x7fc8a0102e70 2024-04-20T22:28:57.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:57.766+0000 7fc8a6600700 1 -- 172.21.15.62:0/1062930222 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8a019cbc0 con 0x7fc8a0102e70 2024-04-20T22:28:57.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:57.767+0000 7fc8a6600700 1 -- 172.21.15.62:0/1062930222 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8a019d110 con 0x7fc8a0102e70 2024-04-20T22:28:57.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:57.768+0000 7fc8a6600700 1 -- 172.21.15.62:0/1062930222 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8a010a7e0 con 0x7fc8a0102e70 2024-04-20T22:28:57.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:57.769+0000 7fc89d600700 1 -- 172.21.15.62:0/1062930222 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fc88c0104f0 con 0x7fc8a0102e70 2024-04-20T22:28:57.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:57.771+0000 7fc89d600700 1 --2- 172.21.15.62:0/1062930222 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc88806ecc0 0x7fc888071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:28:57.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:57.771+0000 7fc89d600700 1 -- 172.21.15.62:0/1062930222 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fc88c08ea70 con 0x7fc8a0102e70 2024-04-20T22:28:57.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:57.771+0000 7fc8a5200700 1 --2- 172.21.15.62:0/1062930222 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc88806ecc0 0x7fc888071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:28:57.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:57.772+0000 7fc8a5200700 1 --2- 172.21.15.62:0/1062930222 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc88806ecc0 0x7fc888071170 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fc894006010 tx=0x7fc89401a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:28:57.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:57.775+0000 7fc89d600700 1 -- 172.21.15.62:0/1062930222 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fc88c057800 con 0x7fc8a0102e70 2024-04-20T22:28:57.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:57 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/753223763' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:28:57.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:57 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:28:57.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:57 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:28:57.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:57 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/1857954457' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:28:58.030 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:57 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/753223763' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:28:58.030 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:57 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:28:58.030 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:57 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:28:58.030 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:57 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/1857954457' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:28:58.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:58.119+0000 7fc8a6600700 1 -- 172.21.15.62:0/1062930222 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fc8a0062380 con 0x7fc8a0102e70 2024-04-20T22:28:58.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:58.120+0000 7fc89d600700 1 -- 172.21.15.62:0/1062930222 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+8240 (secure 0 0 0) 0x7fc88c05aec0 con 0x7fc8a0102e70 2024-04-20T22:28:58.120 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_WARN failed to probe daemons or devices; 1 filesystem with deprecated feature inline_data 2024-04-20T22:28:58.120 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-20T22:28:58.120 INFO:teuthology.orchestra.run.smithi062.stdout: host smithi189 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi189/config 2024-04-20T22:28:58.120 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-20T22:28:58.120 INFO:teuthology.orchestra.run.smithi062.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi189 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpi9rrfoe5:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:28:58.120 INFO:teuthology.orchestra.run.smithi062.stdout:Traceback (most recent call last): 2024-04-20T22:28:58.120 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-20T22:28:58.120 INFO:teuthology.orchestra.run.smithi062.stdout: return _run_code(code, main_globals, None, 2024-04-20T22:28:58.121 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-20T22:28:58.121 INFO:teuthology.orchestra.run.smithi062.stdout: exec(code, run_globals) 2024-04-20T22:28:58.121 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5618, in 2024-04-20T22:28:58.121 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5606, in main 2024-04-20T22:28:58.121 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 408, in _infer_config 2024-04-20T22:28:58.121 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 323, in _infer_fsid 2024-04-20T22:28:58.121 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 436, in _infer_image 2024-04-20T22:28:58.121 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 310, in _validate_fsid 2024-04-20T22:28:58.121 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 3359, in command_ceph_volume 2024-04-20T22:28:58.121 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-20T22:28:58.121 INFO:teuthology.orchestra.run.smithi062.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi189 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpi9rrfoe5:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:28:58.121 INFO:teuthology.orchestra.run.smithi062.stdout: host smithi062 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T22:28:58.121 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-20T22:28:58.121 INFO:teuthology.orchestra.run.smithi062.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi062 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpfparopx2:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:28:58.122 INFO:teuthology.orchestra.run.smithi062.stdout:Traceback (most recent call last): 2024-04-20T22:28:58.122 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-20T22:28:58.122 INFO:teuthology.orchestra.run.smithi062.stdout: return _run_code(code, main_globals, None, 2024-04-20T22:28:58.122 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-20T22:28:58.122 INFO:teuthology.orchestra.run.smithi062.stdout: exec(code, run_globals) 2024-04-20T22:28:58.122 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5618, in 2024-04-20T22:28:58.122 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 5606, in main 2024-04-20T22:28:58.122 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 408, in _infer_config 2024-04-20T22:28:58.122 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 323, in _infer_fsid 2024-04-20T22:28:58.122 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 436, in _infer_image 2024-04-20T22:28:58.122 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 310, in _validate_fsid 2024-04-20T22:28:58.122 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/__main__.py", line 3359, in command_ceph_volume 2024-04-20T22:28:58.122 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/cephadm.632956097c576d0ec9244cad5b14f642b02543f0b519abdb4fc441decfe6d478/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-20T22:28:58.123 INFO:teuthology.orchestra.run.smithi062.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 -e NODE_NAME=smithi062 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpfparopx2:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 inventory --format=json-pretty --filter-for-batch 2024-04-20T22:28:58.123 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T22:28:58.123 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T22:28:58.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:58.124+0000 7fc8a6600700 1 -- 172.21.15.62:0/1062930222 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc88806ecc0 msgr2=0x7fc888071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:28:58.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:58.124+0000 7fc8a6600700 1 --2- 172.21.15.62:0/1062930222 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc88806ecc0 0x7fc888071170 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fc894006010 tx=0x7fc89401a040 comp rx=0 tx=0).stop 2024-04-20T22:28:58.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:58.124+0000 7fc8a6600700 1 -- 172.21.15.62:0/1062930222 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8a0102e70 msgr2=0x7fc8a0198470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:28:58.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:58.124+0000 7fc8a6600700 1 --2- 172.21.15.62:0/1062930222 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8a0102e70 0x7fc8a0198470 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7fc88c00b480 tx=0x7fc88c00b840 comp rx=0 tx=0).stop 2024-04-20T22:28:58.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:58.124+0000 7fc8a6600700 1 -- 172.21.15.62:0/1062930222 shutdown_connections 2024-04-20T22:28:58.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:58.124+0000 7fc8a6600700 1 --2- 172.21.15.62:0/1062930222 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fc88806ecc0 0x7fc888071170 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:28:58.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:58.124+0000 7fc8a6600700 1 --2- 172.21.15.62:0/1062930222 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc8a01024d0 0x7fc8a0197f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:28:58.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:58.124+0000 7fc8a6600700 1 --2- 172.21.15.62:0/1062930222 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8a0102e70 0x7fc8a0198470 unknown :-1 s=CLOSED pgs=1379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:28:58.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:58.124+0000 7fc8a6600700 1 -- 172.21.15.62:0/1062930222 >> 172.21.15.62:0/1062930222 conn(0x7fc8a00fdd90 msgr2=0x7fc8a00ff580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:28:58.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:58.125+0000 7fc8a6600700 1 -- 172.21.15.62:0/1062930222 shutdown_connections 2024-04-20T22:28:58.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:28:58.125+0000 7fc8a6600700 1 -- 172.21.15.62:0/1062930222 wait complete. 2024-04-20T22:28:59.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:58 smithi189 ceph-mon[57022]: from='client.36828 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:28:59.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:58 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:28:59.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:58 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:28:59.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:58 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T22:28:59.239 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:58 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T22:28:59.239 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:58 smithi189 ceph-mon[57022]: pgmap v1615: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.6 KiB/s rd, 4 op/s 2024-04-20T22:28:59.239 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:58 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:28:59.239 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:58 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/1062930222' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:28:59.239 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:58 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T22:28:59.239 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:58 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:28:59.239 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:58 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:28:59.239 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:58 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:28:59.239 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:58 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:28:59.239 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:58 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:28:59.239 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:58 smithi189 ceph-mon[57022]: Upgrade: Setting filesystem cephfs Joinable 2024-04-20T22:28:59.239 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:58 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]: dispatch 2024-04-20T22:28:59.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:58 smithi062 ceph-mon[76322]: from='client.36828 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:28:59.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:58 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:28:59.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:58 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:28:59.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:58 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T22:28:59.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:58 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T22:28:59.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:58 smithi062 ceph-mon[76322]: pgmap v1615: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.6 KiB/s rd, 4 op/s 2024-04-20T22:28:59.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:58 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:28:59.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:58 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/1062930222' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:28:59.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:58 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T22:28:59.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:58 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:28:59.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:58 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:28:59.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:58 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:28:59.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:58 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:28:59.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:58 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:28:59.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:58 smithi062 ceph-mon[76322]: Upgrade: Setting filesystem cephfs Joinable 2024-04-20T22:28:59.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:58 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]: dispatch 2024-04-20T22:29:00.239 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:59 smithi189 ceph-mon[57022]: Health check cleared: CEPHADM_REFRESH_FAILED (was: failed to probe daemons or devices) 2024-04-20T22:29:00.239 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:59 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]': finished 2024-04-20T22:29:00.239 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:59 smithi189 ceph-mon[57022]: fsmap cephfs:2 {0=cephfs.smithi062.ryqblb=up:active,1=cephfs.smithi062.ozizwi=up:active} 2 up:standby-replay 2024-04-20T22:29:00.239 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:59 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:29:00.239 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:59 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:29:00.239 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:59 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:29:00.239 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:59 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:29:00.239 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:59 smithi189 ceph-mon[57022]: Upgrade: Setting container_image for all ceph-exporter 2024-04-20T22:29:00.239 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:59 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:29:00.240 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:59 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter.smithi062"}]: dispatch 2024-04-20T22:29:00.240 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:59 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter.smithi062"}]': finished 2024-04-20T22:29:00.240 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:59 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter.smithi189"}]: dispatch 2024-04-20T22:29:00.240 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:59 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter.smithi189"}]': finished 2024-04-20T22:29:00.240 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:59 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:29:00.240 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:59 smithi189 ceph-mon[57022]: Upgrade: Setting container_image for all iscsi 2024-04-20T22:29:00.240 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:59 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:29:00.240 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:59 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:29:00.240 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:59 smithi189 ceph-mon[57022]: Upgrade: Setting container_image for all nfs 2024-04-20T22:29:00.240 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:59 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:29:00.240 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:59 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:29:00.240 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:59 smithi189 ceph-mon[57022]: Upgrade: Setting container_image for all nvmeof 2024-04-20T22:29:00.240 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:59 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:29:00.241 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:59 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:29:00.241 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:59 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:29:00.241 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:59 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:29:00.241 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:59 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:29:00.241 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:59 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:29:00.241 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:59 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:29:00.241 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:59 smithi189 ceph-mon[57022]: Upgrade: Finalizing container_image settings 2024-04-20T22:29:00.241 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:59 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-20T22:29:00.242 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:59 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-04-20T22:29:00.242 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:59 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-20T22:29:00.242 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:59 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-04-20T22:29:00.242 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:59 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-20T22:29:00.242 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:59 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-04-20T22:29:00.242 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:59 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-20T22:29:00.242 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:59 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-04-20T22:29:00.242 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:59 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-20T22:29:00.242 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:59 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-04-20T22:29:00.242 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:59 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-20T22:29:00.242 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:59 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-04-20T22:29:00.242 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:59 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-20T22:29:00.243 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:59 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-04-20T22:29:00.243 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:59 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-20T22:29:00.243 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:59 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-20T22:29:00.243 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:59 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-04-20T22:29:00.243 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:59 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-20T22:29:00.243 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:59 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-04-20T22:29:00.243 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:59 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-20T22:29:00.243 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:59 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-04-20T22:29:00.243 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:59 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-20T22:29:00.243 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:59 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-04-20T22:29:00.243 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:59 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-20T22:29:00.243 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:59 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-20T22:29:00.243 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:59 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-20T22:29:00.243 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:59 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-20T22:29:00.244 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:59 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-20T22:29:00.244 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:59 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-20T22:29:00.244 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:59 smithi189 ceph-mon[57022]: Upgrade: Complete! 2024-04-20T22:29:00.244 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:59 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-20T22:29:00.244 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:59 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-04-20T22:29:00.244 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:59 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T22:29:00.244 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:59 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T22:29:00.244 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:59 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T22:29:00.244 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:59 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:29:00.244 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:59 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T22:29:00.244 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:59 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T22:29:00.244 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:59 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T22:29:00.244 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:28:59 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:29:00.287 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:59 smithi062 ceph-mon[76322]: Health check cleared: CEPHADM_REFRESH_FAILED (was: failed to probe daemons or devices) 2024-04-20T22:29:00.287 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:59 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]': finished 2024-04-20T22:29:00.287 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:59 smithi062 ceph-mon[76322]: fsmap cephfs:2 {0=cephfs.smithi062.ryqblb=up:active,1=cephfs.smithi062.ozizwi=up:active} 2 up:standby-replay 2024-04-20T22:29:00.287 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:59 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:29:00.287 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:59 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:29:00.287 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:59 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:29:00.287 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:59 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:29:00.287 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:59 smithi062 ceph-mon[76322]: Upgrade: Setting container_image for all ceph-exporter 2024-04-20T22:29:00.288 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:59 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:29:00.288 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:59 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter.smithi062"}]: dispatch 2024-04-20T22:29:00.288 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:59 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter.smithi062"}]': finished 2024-04-20T22:29:00.288 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:59 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter.smithi189"}]: dispatch 2024-04-20T22:29:00.288 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:59 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter.smithi189"}]': finished 2024-04-20T22:29:00.288 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:59 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:29:00.288 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:59 smithi062 ceph-mon[76322]: Upgrade: Setting container_image for all iscsi 2024-04-20T22:29:00.288 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:59 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:29:00.288 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:59 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:29:00.288 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:59 smithi062 ceph-mon[76322]: Upgrade: Setting container_image for all nfs 2024-04-20T22:29:00.288 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:59 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:29:00.288 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:59 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:29:00.288 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:59 smithi062 ceph-mon[76322]: Upgrade: Setting container_image for all nvmeof 2024-04-20T22:29:00.288 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:59 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:29:00.289 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:59 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:29:00.289 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:59 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:29:00.289 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:59 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:29:00.289 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:59 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:29:00.289 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:59 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:29:00.289 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:59 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:29:00.289 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:59 smithi062 ceph-mon[76322]: Upgrade: Finalizing container_image settings 2024-04-20T22:29:00.290 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:59 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-20T22:29:00.290 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:59 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-04-20T22:29:00.290 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:59 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-20T22:29:00.290 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:59 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-04-20T22:29:00.290 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:59 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-20T22:29:00.290 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:59 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-04-20T22:29:00.290 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:59 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-20T22:29:00.290 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:59 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-04-20T22:29:00.290 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:59 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-20T22:29:00.290 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:59 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-04-20T22:29:00.290 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:59 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-20T22:29:00.290 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:59 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-04-20T22:29:00.291 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:59 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-20T22:29:00.291 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:59 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-04-20T22:29:00.291 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:59 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-20T22:29:00.291 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:59 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-20T22:29:00.291 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:59 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-04-20T22:29:00.291 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:59 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-20T22:29:00.291 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:59 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-04-20T22:29:00.291 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:59 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-20T22:29:00.291 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:59 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-04-20T22:29:00.291 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:59 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-20T22:29:00.291 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:59 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-04-20T22:29:00.291 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:59 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-20T22:29:00.291 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:59 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-20T22:29:00.291 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:59 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-20T22:29:00.292 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:59 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-20T22:29:00.292 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:59 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-20T22:29:00.292 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:59 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-20T22:29:00.292 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:59 smithi062 ceph-mon[76322]: Upgrade: Complete! 2024-04-20T22:29:00.292 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:59 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-20T22:29:00.292 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:59 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-04-20T22:29:00.292 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:59 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T22:29:00.292 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:59 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T22:29:00.292 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:59 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T22:29:00.292 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:59 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:29:00.292 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:59 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T22:29:00.292 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:59 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T22:29:00.292 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:59 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T22:29:00.292 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:28:59 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:29:01.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:29:00 smithi189 ceph-mon[57022]: pgmap v1616: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-20T22:29:01.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:29:00 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:29:01.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:29:00 smithi062 ceph-mon[76322]: pgmap v1616: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-20T22:29:01.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:29:00 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:29:03.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:29:02 smithi062 ceph-mon[76322]: pgmap v1617: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-20T22:29:03.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:29:02 smithi189 ceph-mon[57022]: pgmap v1617: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-20T22:29:05.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:29:05 smithi189 ceph-mon[57022]: pgmap v1618: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-20T22:29:05.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:29:05 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:29:05.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:29:05 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:29:05.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:29:05 smithi062 ceph-mon[76322]: pgmap v1618: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-20T22:29:05.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:29:05 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:29:05.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:29:05 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:29:07.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:29:07 smithi189 ceph-mon[57022]: pgmap v1619: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-20T22:29:07.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:29:07 smithi062 ceph-mon[76322]: pgmap v1619: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-20T22:29:09.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:29:09 smithi189 ceph-mon[57022]: pgmap v1620: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.6 KiB/s rd, 4 op/s 2024-04-20T22:29:09.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:29:09 smithi062 ceph-mon[76322]: pgmap v1620: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.6 KiB/s rd, 4 op/s 2024-04-20T22:29:11.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:29:11 smithi189 ceph-mon[57022]: pgmap v1621: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:29:11.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:29:11 smithi062 ceph-mon[76322]: pgmap v1621: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:29:13.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:29:13 smithi189 ceph-mon[57022]: pgmap v1622: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:29:13.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:29:13 smithi062 ceph-mon[76322]: pgmap v1622: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:29:15.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:29:15 smithi062 ceph-mon[76322]: pgmap v1623: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:29:15.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:29:15 smithi189 ceph-mon[57022]: pgmap v1623: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:29:17.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:29:17 smithi062 ceph-mon[76322]: pgmap v1624: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:29:17.790 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:29:17 smithi189 ceph-mon[57022]: pgmap v1624: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:29:19.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:29:19 smithi189 ceph-mon[57022]: pgmap v1625: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:29:19.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:29:19 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:29:19.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:29:19 smithi062 ceph-mon[76322]: pgmap v1625: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:29:19.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:29:19 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:29:21.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:29:21 smithi062 ceph-mon[76322]: pgmap v1626: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:29:21.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:29:21 smithi189 ceph-mon[57022]: pgmap v1626: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:29:23.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:29:23 smithi062 ceph-mon[76322]: pgmap v1627: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:29:23.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:29:23 smithi189 ceph-mon[57022]: pgmap v1627: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:29:25.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:29:25 smithi062 ceph-mon[76322]: pgmap v1628: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:29:25.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:29:25 smithi189 ceph-mon[57022]: pgmap v1628: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:29:27.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:29:27 smithi062 ceph-mon[76322]: pgmap v1629: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:29:27.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:29:27 smithi189 ceph-mon[57022]: pgmap v1629: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:29:28.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:28.309+0000 7f0427800700 1 -- 172.21.15.62:0/1052150026 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0420100e80 msgr2=0x7f0420104e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:29:28.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:28.309+0000 7f0427800700 1 --2- 172.21.15.62:0/1052150026 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0420100e80 0x7f0420104e50 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7f040c009910 tx=0x7f040c009c20 comp rx=0 tx=0).stop 2024-04-20T22:29:28.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:28.309+0000 7f0427800700 1 -- 172.21.15.62:0/1052150026 shutdown_connections 2024-04-20T22:29:28.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:28.309+0000 7f0427800700 1 --2- 172.21.15.62:0/1052150026 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0420100e80 0x7f0420104e50 unknown :-1 s=CLOSED pgs=1380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:29:28.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:28.309+0000 7f0427800700 1 --2- 172.21.15.62:0/1052150026 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f04201004e0 0x7f04201008b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:29:28.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:28.310+0000 7f0427800700 1 -- 172.21.15.62:0/1052150026 >> 172.21.15.62:0/1052150026 conn(0x7f04200fbda0 msgr2=0x7f04200fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:29:28.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:28.310+0000 7f0427800700 1 -- 172.21.15.62:0/1052150026 shutdown_connections 2024-04-20T22:29:28.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:28.310+0000 7f0427800700 1 -- 172.21.15.62:0/1052150026 wait complete. 2024-04-20T22:29:28.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:28.311+0000 7f0427800700 1 Processor -- start 2024-04-20T22:29:28.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:28.311+0000 7f0427800700 1 -- start start 2024-04-20T22:29:28.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:28.311+0000 7f0427800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04201004e0 0x7f0420073050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:29:28.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:28.311+0000 7f0427800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0420100e80 0x7f0420073590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:29:28.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:28.311+0000 7f0427800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0420073ad0 con 0x7f04201004e0 2024-04-20T22:29:28.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:28.311+0000 7f0427800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0420073c10 con 0x7f0420100e80 2024-04-20T22:29:28.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:28.312+0000 7f0424e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04201004e0 0x7f0420073050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:29:28.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:28.312+0000 7f0424e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04201004e0 0x7f0420073050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53096/0 (socket says 172.21.15.62:53096) 2024-04-20T22:29:28.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:28.312+0000 7f0424e00700 1 -- 172.21.15.62:0/3468649898 learned_addr learned my addr 172.21.15.62:0/3468649898 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:29:28.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:28.312+0000 7f041fe00700 1 --2- 172.21.15.62:0/3468649898 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0420100e80 0x7f0420073590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:29:28.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:28.312+0000 7f0424e00700 1 -- 172.21.15.62:0/3468649898 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0420100e80 msgr2=0x7f0420073590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:29:28.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:28.312+0000 7f0424e00700 1 --2- 172.21.15.62:0/3468649898 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0420100e80 0x7f0420073590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:29:28.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:28.312+0000 7f0424e00700 1 -- 172.21.15.62:0/3468649898 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f040c0095f0 con 0x7f04201004e0 2024-04-20T22:29:28.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:28.313+0000 7f0424e00700 1 --2- 172.21.15.62:0/3468649898 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04201004e0 0x7f0420073050 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7f041400c930 tx=0x7f041400ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:29:28.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:28.313+0000 7f041d600700 1 -- 172.21.15.62:0/3468649898 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0414007ad0 con 0x7f04201004e0 2024-04-20T22:29:28.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:28.313+0000 7f041d600700 1 -- 172.21.15.62:0/3468649898 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0414007c30 con 0x7f04201004e0 2024-04-20T22:29:28.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:28.313+0000 7f0427800700 1 -- 172.21.15.62:0/3468649898 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04201a20e0 con 0x7f04201004e0 2024-04-20T22:29:28.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:28.313+0000 7f041d600700 1 -- 172.21.15.62:0/3468649898 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0414018610 con 0x7f04201004e0 2024-04-20T22:29:28.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:28.313+0000 7f0427800700 1 -- 172.21.15.62:0/3468649898 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04201a2440 con 0x7f04201004e0 2024-04-20T22:29:28.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:28.314+0000 7f0427800700 1 -- 172.21.15.62:0/3468649898 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0420108850 con 0x7f04201004e0 2024-04-20T22:29:28.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:28.316+0000 7f041d600700 1 -- 172.21.15.62:0/3468649898 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f041400f450 con 0x7f04201004e0 2024-04-20T22:29:28.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:28.317+0000 7f041d600700 1 --2- 172.21.15.62:0/3468649898 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f040806ecc0 0x7f0408071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:29:28.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:28.317+0000 7f041d600700 1 -- 172.21.15.62:0/3468649898 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f041408f120 con 0x7f04201004e0 2024-04-20T22:29:28.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:28.317+0000 7f041fe00700 1 --2- 172.21.15.62:0/3468649898 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f040806ecc0 0x7f0408071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:29:28.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:28.318+0000 7f041fe00700 1 --2- 172.21.15.62:0/3468649898 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f040806ecc0 0x7f0408071170 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f040c006010 tx=0x7f040c00b540 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:29:28.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:28.322+0000 7f041d600700 1 -- 172.21.15.62:0/3468649898 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f0414057e30 con 0x7f04201004e0 2024-04-20T22:29:28.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:28.580+0000 7f0427800700 1 -- 172.21.15.62:0/3468649898 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0420061190 con 0x7f040806ecc0 2024-04-20T22:29:28.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:28.583+0000 7f041d600700 1 -- 172.21.15.62:0/3468649898 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+175 (secure 0 0 0) 0x7f0420061190 con 0x7f040806ecc0 2024-04-20T22:29:28.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:28.585+0000 7f0427800700 1 -- 172.21.15.62:0/3468649898 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f040806ecc0 msgr2=0x7f0408071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:29:28.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:28.585+0000 7f0427800700 1 --2- 172.21.15.62:0/3468649898 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f040806ecc0 0x7f0408071170 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f040c006010 tx=0x7f040c00b540 comp rx=0 tx=0).stop 2024-04-20T22:29:28.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:28.585+0000 7f0427800700 1 -- 172.21.15.62:0/3468649898 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04201004e0 msgr2=0x7f0420073050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:29:28.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:28.585+0000 7f0427800700 1 --2- 172.21.15.62:0/3468649898 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04201004e0 0x7f0420073050 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7f041400c930 tx=0x7f041400ccf0 comp rx=0 tx=0).stop 2024-04-20T22:29:28.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:28.586+0000 7f0427800700 1 -- 172.21.15.62:0/3468649898 shutdown_connections 2024-04-20T22:29:28.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:28.586+0000 7f0427800700 1 --2- 172.21.15.62:0/3468649898 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f040806ecc0 0x7f0408071170 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:29:28.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:28.586+0000 7f0427800700 1 --2- 172.21.15.62:0/3468649898 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04201004e0 0x7f0420073050 unknown :-1 s=CLOSED pgs=1381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:29:28.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:28.586+0000 7f0427800700 1 --2- 172.21.15.62:0/3468649898 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0420100e80 0x7f0420073590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:29:28.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:28.586+0000 7f0427800700 1 -- 172.21.15.62:0/3468649898 >> 172.21.15.62:0/3468649898 conn(0x7f04200fbda0 msgr2=0x7f04200fe180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:29:28.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:28.586+0000 7f0427800700 1 -- 172.21.15.62:0/3468649898 shutdown_connections 2024-04-20T22:29:28.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:28.586+0000 7f0427800700 1 -- 172.21.15.62:0/3468649898 wait complete. 2024-04-20T22:29:29.450 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -e sha1=a8c84f5997950fd7d008204839253291d52e6eda -- bash -c 'ceph orch ps' 2024-04-20T22:29:29.568 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:29:29 smithi062 ceph-mon[76322]: pgmap v1630: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:29:29.834 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:29:29 smithi189 ceph-mon[57022]: pgmap v1630: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:29:29.897 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T22:29:30.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:29:30 smithi062 ceph-mon[76322]: from='client.36836 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:29:30.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:29:30 smithi189 ceph-mon[57022]: from='client.36836 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:29:31.754 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:29:31 smithi062 ceph-mon[76322]: pgmap v1631: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:29:31.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:29:31 smithi189 ceph-mon[57022]: pgmap v1631: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:29:32.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:32.346+0000 7f6d2de00700 1 -- 172.21.15.62:0/807334946 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d280730f0 msgr2=0x7f6d280734c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:29:32.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:32.346+0000 7f6d2de00700 1 --2- 172.21.15.62:0/807334946 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d280730f0 0x7f6d280734c0 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7f6d18009910 tx=0x7f6d18009c20 comp rx=0 tx=0).stop 2024-04-20T22:29:32.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:32.346+0000 7f6d2de00700 1 -- 172.21.15.62:0/807334946 shutdown_connections 2024-04-20T22:29:32.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:32.346+0000 7f6d2de00700 1 --2- 172.21.15.62:0/807334946 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6d28073a00 0x7f6d28110d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:29:32.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:32.346+0000 7f6d2de00700 1 --2- 172.21.15.62:0/807334946 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d280730f0 0x7f6d280734c0 unknown :-1 s=CLOSED pgs=1382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:29:32.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:32.346+0000 7f6d2de00700 1 -- 172.21.15.62:0/807334946 >> 172.21.15.62:0/807334946 conn(0x7f6d280fbda0 msgr2=0x7f6d280fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:29:32.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:32.346+0000 7f6d2de00700 1 -- 172.21.15.62:0/807334946 shutdown_connections 2024-04-20T22:29:32.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:32.346+0000 7f6d2de00700 1 -- 172.21.15.62:0/807334946 wait complete. 2024-04-20T22:29:32.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:32.347+0000 7f6d2de00700 1 Processor -- start 2024-04-20T22:29:32.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:32.348+0000 7f6d2de00700 1 -- start start 2024-04-20T22:29:32.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:32.348+0000 7f6d2de00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d280730f0 0x7f6d2819c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:29:32.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:32.348+0000 7f6d2de00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6d28073a00 0x7f6d2819c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:29:32.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:32.348+0000 7f6d2de00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d2819d000 con 0x7f6d280730f0 2024-04-20T22:29:32.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:32.348+0000 7f6d2de00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d281a0d90 con 0x7f6d28073a00 2024-04-20T22:29:32.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:32.348+0000 7f6d26a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d280730f0 0x7f6d2819c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:29:32.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:32.348+0000 7f6d26a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d280730f0 0x7f6d2819c3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55772/0 (socket says 172.21.15.62:55772) 2024-04-20T22:29:32.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:32.348+0000 7f6d26a00700 1 -- 172.21.15.62:0/2654363740 learned_addr learned my addr 172.21.15.62:0/2654363740 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:29:32.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:32.349+0000 7f6d26a00700 1 -- 172.21.15.62:0/2654363740 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6d28073a00 msgr2=0x7f6d2819c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:29:32.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:32.349+0000 7f6d26a00700 1 --2- 172.21.15.62:0/2654363740 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6d28073a00 0x7f6d2819c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:29:32.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:32.349+0000 7f6d26a00700 1 -- 172.21.15.62:0/2654363740 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d180095f0 con 0x7f6d280730f0 2024-04-20T22:29:32.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:32.349+0000 7f6d26a00700 1 --2- 172.21.15.62:0/2654363740 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d280730f0 0x7f6d2819c3e0 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7f6d1800b5c0 tx=0x7f6d1800be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:29:32.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:32.350+0000 7f6d2ca00700 1 -- 172.21.15.62:0/2654363740 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6d1801d070 con 0x7f6d280730f0 2024-04-20T22:29:32.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:32.350+0000 7f6d2de00700 1 -- 172.21.15.62:0/2654363740 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d281a1070 con 0x7f6d280730f0 2024-04-20T22:29:32.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:32.350+0000 7f6d2ca00700 1 -- 172.21.15.62:0/2654363740 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6d1800f460 con 0x7f6d280730f0 2024-04-20T22:29:32.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:32.350+0000 7f6d2ca00700 1 -- 172.21.15.62:0/2654363740 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6d18017630 con 0x7f6d280730f0 2024-04-20T22:29:32.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:32.350+0000 7f6d2de00700 1 -- 172.21.15.62:0/2654363740 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d281a15c0 con 0x7f6d280730f0 2024-04-20T22:29:32.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:32.351+0000 7f6d2de00700 1 -- 172.21.15.62:0/2654363740 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d2810e4d0 con 0x7f6d280730f0 2024-04-20T22:29:32.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:32.352+0000 7f6d2ca00700 1 -- 172.21.15.62:0/2654363740 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f6d1800f5d0 con 0x7f6d280730f0 2024-04-20T22:29:32.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:32.353+0000 7f6d2ca00700 1 --2- 172.21.15.62:0/2654363740 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6d0c06ecc0 0x7f6d0c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:29:32.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:32.353+0000 7f6d2ca00700 1 -- 172.21.15.62:0/2654363740 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f6d18090490 con 0x7f6d280730f0 2024-04-20T22:29:32.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:32.353+0000 7f6d1e000700 1 --2- 172.21.15.62:0/2654363740 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6d0c06ecc0 0x7f6d0c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:29:32.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:32.354+0000 7f6d1e000700 1 --2- 172.21.15.62:0/2654363740 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6d0c06ecc0 0x7f6d0c071170 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f6d2819da00 tx=0x7f6d10008540 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:29:32.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:32.359+0000 7f6d2ca00700 1 -- 172.21.15.62:0/2654363740 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f6d180591a0 con 0x7f6d280730f0 2024-04-20T22:29:32.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:32.608+0000 7f6d2de00700 1 -- 172.21.15.62:0/2654363740 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f6d281a1850 con 0x7f6d0c06ecc0 2024-04-20T22:29:32.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:32.636+0000 7f6d2ca00700 1 -- 172.21.15.62:0/2654363740 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f6d281a1850 con 0x7f6d0c06ecc0 2024-04-20T22:29:32.635 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T22:29:32.635 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (55m) 43s ago 69m 14.4M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T22:29:32.635 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (60s) 43s ago 69m 7876k - 19.0.0-3133-ga8c84f59 76bc6b91b726 a994518fe0b6 2024-04-20T22:29:32.636 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (51s) 42s ago 68m 7105k - 19.0.0-3133-ga8c84f59 76bc6b91b726 3f5760a59416 2024-04-20T22:29:32.636 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (51m) 43s ago 69m 6660k - 19.0.0-3133-ga8c84f59 76bc6b91b726 2825d5ba11c6 2024-04-20T22:29:32.636 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (51m) 42s ago 68m 6665k - 19.0.0-3133-ga8c84f59 76bc6b91b726 20f746fa1bba 2024-04-20T22:29:32.636 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (54m) 43s ago 68m 75.6M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T22:29:32.636 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (46m) 43s ago 61m 62.3M - 19.0.0-3133-ga8c84f59 76bc6b91b726 de80bc90a683 2024-04-20T22:29:32.636 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (47m) 43s ago 62m 64.2M - 19.0.0-3133-ga8c84f59 76bc6b91b726 cd793b7ee98e 2024-04-20T22:29:32.636 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (31m) 42s ago 62m 62.8M - 19.0.0-3133-ga8c84f59 76bc6b91b726 0e7b0bc4d1ab 2024-04-20T22:29:32.636 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (16m) 42s ago 61m 60.7M - 19.0.0-3133-ga8c84f59 76bc6b91b726 0af732ac9e06 2024-04-20T22:29:32.636 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (58m) 43s ago 71m 521M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T22:29:32.636 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (56m) 42s ago 67m 460M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T22:29:32.636 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (53m) 43s ago 71m 103M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 019eb1479485 2024-04-20T22:29:32.636 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (52m) 42s ago 67m 66.9M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 55aac560aab6 2024-04-20T22:29:32.636 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (56m) 43s ago 69m 8975k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T22:29:32.636 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (56m) 42s ago 68m 8850k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T22:29:32.636 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (50m) 43s ago 66m 98.6M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 ee2bb901f26e 2024-04-20T22:29:32.637 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (50m) 43s ago 66m 104M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 9075152d4162 2024-04-20T22:29:32.637 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (49m) 43s ago 65m 86.8M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 2d4f3e38dc0e 2024-04-20T22:29:32.637 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (49m) 42s ago 65m 104M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 8f97b17ae0eb 2024-04-20T22:29:32.637 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (48m) 42s ago 64m 103M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 653d8079967d 2024-04-20T22:29:32.637 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (47m) 42s ago 63m 97.4M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 596dded14c12 2024-04-20T22:29:32.637 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (55m) 43s ago 68m 89.5M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T22:29:32.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:32.639+0000 7f6d2de00700 1 -- 172.21.15.62:0/2654363740 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6d0c06ecc0 msgr2=0x7f6d0c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:29:32.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:32.639+0000 7f6d2de00700 1 --2- 172.21.15.62:0/2654363740 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6d0c06ecc0 0x7f6d0c071170 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f6d2819da00 tx=0x7f6d10008540 comp rx=0 tx=0).stop 2024-04-20T22:29:32.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:32.639+0000 7f6d2de00700 1 -- 172.21.15.62:0/2654363740 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d280730f0 msgr2=0x7f6d2819c3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:29:32.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:32.639+0000 7f6d2de00700 1 --2- 172.21.15.62:0/2654363740 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d280730f0 0x7f6d2819c3e0 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7f6d1800b5c0 tx=0x7f6d1800be00 comp rx=0 tx=0).stop 2024-04-20T22:29:32.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:32.639+0000 7f6d2de00700 1 -- 172.21.15.62:0/2654363740 shutdown_connections 2024-04-20T22:29:32.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:32.639+0000 7f6d2de00700 1 --2- 172.21.15.62:0/2654363740 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f6d0c06ecc0 0x7f6d0c071170 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:29:32.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:32.639+0000 7f6d2de00700 1 --2- 172.21.15.62:0/2654363740 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d280730f0 0x7f6d2819c3e0 unknown :-1 s=CLOSED pgs=1383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:29:32.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:32.639+0000 7f6d2de00700 1 --2- 172.21.15.62:0/2654363740 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6d28073a00 0x7f6d2819c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:29:32.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:32.639+0000 7f6d2de00700 1 -- 172.21.15.62:0/2654363740 >> 172.21.15.62:0/2654363740 conn(0x7f6d280fbda0 msgr2=0x7f6d28102860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:29:32.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:32.639+0000 7f6d2de00700 1 -- 172.21.15.62:0/2654363740 shutdown_connections 2024-04-20T22:29:32.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:32.640+0000 7f6d2de00700 1 -- 172.21.15.62:0/2654363740 wait complete. 2024-04-20T22:29:33.390 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -e sha1=a8c84f5997950fd7d008204839253291d52e6eda -- bash -c 'ceph orch upgrade status' 2024-04-20T22:29:33.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:29:33 smithi189 ceph-mon[57022]: pgmap v1632: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:29:33.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:29:33 smithi062 ceph-mon[76322]: pgmap v1632: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:29:33.837 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T22:29:34.838 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:29:34 smithi062 ceph-mon[76322]: from='client.36840 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:29:34.838 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:29:34 smithi062 ceph-mon[76322]: pgmap v1633: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:29:34.838 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:29:34 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:29:34.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:29:34 smithi189 ceph-mon[57022]: from='client.36840 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:29:34.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:29:34 smithi189 ceph-mon[57022]: pgmap v1633: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:29:34.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:29:34 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:29:36.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:36.020+0000 7f41aa200700 1 -- 172.21.15.62:0/510150021 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41a40730f0 msgr2=0x7f41a40734c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:29:36.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:36.020+0000 7f41aa200700 1 --2- 172.21.15.62:0/510150021 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41a40730f0 0x7f41a40734c0 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f419400b0a0 tx=0x7f419400b3b0 comp rx=0 tx=0).stop 2024-04-20T22:29:36.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:36.020+0000 7f41aa200700 1 -- 172.21.15.62:0/510150021 shutdown_connections 2024-04-20T22:29:36.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:36.020+0000 7f41aa200700 1 --2- 172.21.15.62:0/510150021 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f41a4073a00 0x7f41a4110d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:29:36.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:36.020+0000 7f41aa200700 1 --2- 172.21.15.62:0/510150021 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41a40730f0 0x7f41a40734c0 unknown :-1 s=CLOSED pgs=1384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:29:36.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:36.020+0000 7f41aa200700 1 -- 172.21.15.62:0/510150021 >> 172.21.15.62:0/510150021 conn(0x7f41a40fbda0 msgr2=0x7f41a40fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:29:36.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:36.020+0000 7f41aa200700 1 -- 172.21.15.62:0/510150021 shutdown_connections 2024-04-20T22:29:36.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:36.020+0000 7f41aa200700 1 -- 172.21.15.62:0/510150021 wait complete. 2024-04-20T22:29:36.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:36.021+0000 7f41aa200700 1 Processor -- start 2024-04-20T22:29:36.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:36.021+0000 7f41aa200700 1 -- start start 2024-04-20T22:29:36.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:36.022+0000 7f41aa200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f41a40730f0 0x7f41a419c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:29:36.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:36.022+0000 7f41aa200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41a4073a00 0x7f41a419c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:29:36.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:36.022+0000 7f41aa200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41a419d030 con 0x7f41a4073a00 2024-04-20T22:29:36.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:36.022+0000 7f41aa200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41a41a0dc0 con 0x7f41a40730f0 2024-04-20T22:29:36.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:36.022+0000 7f41a2a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41a4073a00 0x7f41a419c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:29:36.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:36.022+0000 7f41a2a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41a4073a00 0x7f41a419c950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55792/0 (socket says 172.21.15.62:55792) 2024-04-20T22:29:36.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:36.022+0000 7f41a2a00700 1 -- 172.21.15.62:0/1763433857 learned_addr learned my addr 172.21.15.62:0/1763433857 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:29:36.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:36.022+0000 7f41a3400700 1 --2- 172.21.15.62:0/1763433857 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f41a40730f0 0x7f41a419c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:29:36.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:36.023+0000 7f41a2a00700 1 -- 172.21.15.62:0/1763433857 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f41a40730f0 msgr2=0x7f41a419c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:29:36.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:36.023+0000 7f41a2a00700 1 --2- 172.21.15.62:0/1763433857 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f41a40730f0 0x7f41a419c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:29:36.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:36.023+0000 7f41a2a00700 1 -- 172.21.15.62:0/1763433857 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4194009d00 con 0x7f41a4073a00 2024-04-20T22:29:36.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:36.023+0000 7f41a2a00700 1 --2- 172.21.15.62:0/1763433857 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41a4073a00 0x7f41a419c950 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7f419000d8d0 tx=0x7f419000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:29:36.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:36.023+0000 7f419be00700 1 -- 172.21.15.62:0/1763433857 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4190009940 con 0x7f41a4073a00 2024-04-20T22:29:36.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:36.023+0000 7f419be00700 1 -- 172.21.15.62:0/1763433857 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4190010460 con 0x7f41a4073a00 2024-04-20T22:29:36.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:36.023+0000 7f419be00700 1 -- 172.21.15.62:0/1763433857 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f419000f5f0 con 0x7f41a4073a00 2024-04-20T22:29:36.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:36.023+0000 7f41aa200700 1 -- 172.21.15.62:0/1763433857 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41a41a10a0 con 0x7f41a4073a00 2024-04-20T22:29:36.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:36.024+0000 7f41aa200700 1 -- 172.21.15.62:0/1763433857 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41a41a15f0 con 0x7f41a4073a00 2024-04-20T22:29:36.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:36.025+0000 7f41aa200700 1 -- 172.21.15.62:0/1763433857 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41a410e4d0 con 0x7f41a4073a00 2024-04-20T22:29:36.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:36.026+0000 7f419be00700 1 -- 172.21.15.62:0/1763433857 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f41900105d0 con 0x7f41a4073a00 2024-04-20T22:29:36.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:36.027+0000 7f419be00700 1 --2- 172.21.15.62:0/1763433857 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f418806ecc0 0x7f4188071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:29:36.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:36.027+0000 7f419be00700 1 -- 172.21.15.62:0/1763433857 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f419008ee10 con 0x7f41a4073a00 2024-04-20T22:29:36.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:36.027+0000 7f41a3400700 1 --2- 172.21.15.62:0/1763433857 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f418806ecc0 0x7f4188071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:29:36.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:36.028+0000 7f41a3400700 1 --2- 172.21.15.62:0/1763433857 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f418806ecc0 0x7f4188071170 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f41940060b0 tx=0x7f4194006000 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:29:36.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:36.033+0000 7f419be00700 1 -- 172.21.15.62:0/1763433857 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f4190057ba0 con 0x7f41a4073a00 2024-04-20T22:29:36.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:36.297+0000 7f41aa200700 1 -- 172.21.15.62:0/1763433857 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f41a41a1860 con 0x7f418806ecc0 2024-04-20T22:29:36.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:36.300+0000 7f419be00700 1 -- 172.21.15.62:0/1763433857 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+175 (secure 0 0 0) 0x7f41a41a1860 con 0x7f418806ecc0 2024-04-20T22:29:36.299 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:29:36.300 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": null, 2024-04-20T22:29:36.300 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": false, 2024-04-20T22:29:36.300 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "", 2024-04-20T22:29:36.300 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [], 2024-04-20T22:29:36.300 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": null, 2024-04-20T22:29:36.300 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "", 2024-04-20T22:29:36.300 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-04-20T22:29:36.300 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:29:36.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:36.303+0000 7f41aa200700 1 -- 172.21.15.62:0/1763433857 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f418806ecc0 msgr2=0x7f4188071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:29:36.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:36.303+0000 7f41aa200700 1 --2- 172.21.15.62:0/1763433857 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f418806ecc0 0x7f4188071170 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f41940060b0 tx=0x7f4194006000 comp rx=0 tx=0).stop 2024-04-20T22:29:36.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:36.303+0000 7f41aa200700 1 -- 172.21.15.62:0/1763433857 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41a4073a00 msgr2=0x7f41a419c950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:29:36.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:36.303+0000 7f41aa200700 1 --2- 172.21.15.62:0/1763433857 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41a4073a00 0x7f41a419c950 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7f419000d8d0 tx=0x7f419000dc90 comp rx=0 tx=0).stop 2024-04-20T22:29:36.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:36.303+0000 7f41aa200700 1 -- 172.21.15.62:0/1763433857 shutdown_connections 2024-04-20T22:29:36.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:36.303+0000 7f41aa200700 1 --2- 172.21.15.62:0/1763433857 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f418806ecc0 0x7f4188071170 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:29:36.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:36.303+0000 7f41aa200700 1 --2- 172.21.15.62:0/1763433857 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f41a40730f0 0x7f41a419c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:29:36.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:36.303+0000 7f41aa200700 1 --2- 172.21.15.62:0/1763433857 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41a4073a00 0x7f41a419c950 unknown :-1 s=CLOSED pgs=1385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:29:36.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:36.303+0000 7f41aa200700 1 -- 172.21.15.62:0/1763433857 >> 172.21.15.62:0/1763433857 conn(0x7f41a40fbda0 msgr2=0x7f41a4102860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:29:36.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:36.304+0000 7f41aa200700 1 -- 172.21.15.62:0/1763433857 shutdown_connections 2024-04-20T22:29:36.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:36.304+0000 7f41aa200700 1 -- 172.21.15.62:0/1763433857 wait complete. 2024-04-20T22:29:37.150 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -e sha1=a8c84f5997950fd7d008204839253291d52e6eda -- bash -c 'ceph health detail' 2024-04-20T22:29:37.225 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:29:36 smithi062 ceph-mon[76322]: pgmap v1634: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:29:37.226 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:29:36 smithi062 ceph-mon[76322]: from='client.36844 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:29:37.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:29:36 smithi189 ceph-mon[57022]: pgmap v1634: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:29:37.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:29:36 smithi189 ceph-mon[57022]: from='client.36844 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:29:37.583 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T22:29:39.189 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:29:38 smithi062 ceph-mon[76322]: pgmap v1635: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:29:39.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:29:38 smithi189 ceph-mon[57022]: pgmap v1635: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:29:39.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:39.725+0000 7ff040200700 1 -- 172.21.15.62:0/1053484279 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff038100e80 msgr2=0x7ff038104e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:29:39.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:39.725+0000 7ff040200700 1 --2- 172.21.15.62:0/1053484279 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff038100e80 0x7ff038104e50 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7ff024009910 tx=0x7ff024009c20 comp rx=0 tx=0).stop 2024-04-20T22:29:39.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:39.725+0000 7ff040200700 1 -- 172.21.15.62:0/1053484279 shutdown_connections 2024-04-20T22:29:39.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:39.725+0000 7ff040200700 1 --2- 172.21.15.62:0/1053484279 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff038100e80 0x7ff038104e50 unknown :-1 s=CLOSED pgs=1386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:29:39.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:39.725+0000 7ff040200700 1 --2- 172.21.15.62:0/1053484279 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff0381004e0 0x7ff0381008b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:29:39.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:39.725+0000 7ff040200700 1 -- 172.21.15.62:0/1053484279 >> 172.21.15.62:0/1053484279 conn(0x7ff0380fbda0 msgr2=0x7ff0380fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:29:39.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:39.725+0000 7ff040200700 1 -- 172.21.15.62:0/1053484279 shutdown_connections 2024-04-20T22:29:39.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:39.725+0000 7ff040200700 1 -- 172.21.15.62:0/1053484279 wait complete. 2024-04-20T22:29:39.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:39.726+0000 7ff040200700 1 Processor -- start 2024-04-20T22:29:39.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:39.727+0000 7ff040200700 1 -- start start 2024-04-20T22:29:39.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:39.727+0000 7ff040200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff0381004e0 0x7ff038195e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:29:39.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:39.727+0000 7ff040200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff038100e80 0x7ff0381963d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:29:39.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:39.727+0000 7ff040200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff038196ab0 con 0x7ff038100e80 2024-04-20T22:29:39.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:39.727+0000 7ff040200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff03819a840 con 0x7ff0381004e0 2024-04-20T22:29:39.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:39.728+0000 7ff03ce00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff038100e80 0x7ff0381963d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:29:39.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:39.728+0000 7ff03ce00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff038100e80 0x7ff0381963d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55822/0 (socket says 172.21.15.62:55822) 2024-04-20T22:29:39.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:39.728+0000 7ff03ce00700 1 -- 172.21.15.62:0/608104375 learned_addr learned my addr 172.21.15.62:0/608104375 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:29:39.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:39.728+0000 7ff03d800700 1 --2- 172.21.15.62:0/608104375 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff0381004e0 0x7ff038195e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:29:39.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:39.728+0000 7ff03ce00700 1 -- 172.21.15.62:0/608104375 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff0381004e0 msgr2=0x7ff038195e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:29:39.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:39.728+0000 7ff03ce00700 1 --2- 172.21.15.62:0/608104375 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff0381004e0 0x7ff038195e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:29:39.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:39.728+0000 7ff03ce00700 1 -- 172.21.15.62:0/608104375 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0240095f0 con 0x7ff038100e80 2024-04-20T22:29:39.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:39.728+0000 7ff03ce00700 1 --2- 172.21.15.62:0/608104375 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff038100e80 0x7ff0381963d0 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7ff02400bd70 tx=0x7ff02400be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:29:39.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:39.729+0000 7ff036000700 1 -- 172.21.15.62:0/608104375 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff02401d070 con 0x7ff038100e80 2024-04-20T22:29:39.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:39.729+0000 7ff040200700 1 -- 172.21.15.62:0/608104375 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff03819aac0 con 0x7ff038100e80 2024-04-20T22:29:39.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:39.729+0000 7ff036000700 1 -- 172.21.15.62:0/608104375 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff024005dc0 con 0x7ff038100e80 2024-04-20T22:29:39.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:39.729+0000 7ff036000700 1 -- 172.21.15.62:0/608104375 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff024017400 con 0x7ff038100e80 2024-04-20T22:29:39.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:39.729+0000 7ff040200700 1 -- 172.21.15.62:0/608104375 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff03819afb0 con 0x7ff038100e80 2024-04-20T22:29:39.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:39.730+0000 7ff040200700 1 -- 172.21.15.62:0/608104375 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff038108760 con 0x7ff038100e80 2024-04-20T22:29:39.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:39.731+0000 7ff036000700 1 -- 172.21.15.62:0/608104375 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff02400f7f0 con 0x7ff038100e80 2024-04-20T22:29:39.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:39.732+0000 7ff036000700 1 --2- 172.21.15.62:0/608104375 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff02006ecc0 0x7ff020071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:29:39.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:39.733+0000 7ff036000700 1 -- 172.21.15.62:0/608104375 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7ff024090410 con 0x7ff038100e80 2024-04-20T22:29:39.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:39.733+0000 7ff03d800700 1 --2- 172.21.15.62:0/608104375 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff02006ecc0 0x7ff020071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:29:39.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:39.733+0000 7ff03d800700 1 --2- 172.21.15.62:0/608104375 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff02006ecc0 0x7ff020071170 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7ff02c005950 tx=0x7ff02c0058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:29:39.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:39.737+0000 7ff036000700 1 -- 172.21.15.62:0/608104375 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7ff024059120 con 0x7ff038100e80 2024-04-20T22:29:40.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:40.074+0000 7ff040200700 1 -- 172.21.15.62:0/608104375 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7ff03804f2a0 con 0x7ff038100e80 2024-04-20T22:29:40.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:40.075+0000 7ff036000700 1 -- 172.21.15.62:0/608104375 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7ff02405c7e0 con 0x7ff038100e80 2024-04-20T22:29:40.075 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-04-20T22:29:40.076 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T22:29:40.076 INFO:teuthology.orchestra.run.smithi062.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T22:29:40.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:40.079+0000 7ff040200700 1 -- 172.21.15.62:0/608104375 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff02006ecc0 msgr2=0x7ff020071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:29:40.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:40.079+0000 7ff040200700 1 --2- 172.21.15.62:0/608104375 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff02006ecc0 0x7ff020071170 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7ff02c005950 tx=0x7ff02c0058e0 comp rx=0 tx=0).stop 2024-04-20T22:29:40.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:40.079+0000 7ff040200700 1 -- 172.21.15.62:0/608104375 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff038100e80 msgr2=0x7ff0381963d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:29:40.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:40.079+0000 7ff040200700 1 --2- 172.21.15.62:0/608104375 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff038100e80 0x7ff0381963d0 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7ff02400bd70 tx=0x7ff02400be50 comp rx=0 tx=0).stop 2024-04-20T22:29:40.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:40.079+0000 7ff040200700 1 -- 172.21.15.62:0/608104375 shutdown_connections 2024-04-20T22:29:40.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:40.079+0000 7ff040200700 1 --2- 172.21.15.62:0/608104375 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff02006ecc0 0x7ff020071170 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:29:40.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:40.079+0000 7ff040200700 1 --2- 172.21.15.62:0/608104375 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff0381004e0 0x7ff038195e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:29:40.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:40.079+0000 7ff040200700 1 --2- 172.21.15.62:0/608104375 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff038100e80 0x7ff0381963d0 unknown :-1 s=CLOSED pgs=1387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:29:40.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:40.079+0000 7ff040200700 1 -- 172.21.15.62:0/608104375 >> 172.21.15.62:0/608104375 conn(0x7ff0380fbda0 msgr2=0x7ff0381046c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:29:40.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:40.079+0000 7ff040200700 1 -- 172.21.15.62:0/608104375 shutdown_connections 2024-04-20T22:29:40.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:40.079+0000 7ff040200700 1 -- 172.21.15.62:0/608104375 wait complete. 2024-04-20T22:29:40.812 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -e sha1=a8c84f5997950fd7d008204839253291d52e6eda -- bash -c 'ceph versions' 2024-04-20T22:29:41.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:29:40 smithi062 ceph-mon[76322]: pgmap v1636: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:29:41.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:29:40 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/608104375' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:29:41.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:29:40 smithi189 ceph-mon[57022]: pgmap v1636: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:29:41.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:29:40 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/608104375' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-20T22:29:41.260 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T22:29:43.125 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:29:42 smithi062 ceph-mon[76322]: pgmap v1637: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:29:43.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:29:42 smithi189 ceph-mon[57022]: pgmap v1637: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:29:43.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:43.403+0000 7f0057200700 1 -- 172.21.15.62:0/2074268153 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0058100c70 msgr2=0x7f0058104c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:29:43.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:43.403+0000 7f0057200700 1 --2- 172.21.15.62:0/2074268153 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0058100c70 0x7f0058104c40 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7f0044009910 tx=0x7f0044009c20 comp rx=0 tx=0).stop 2024-04-20T22:29:43.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:43.403+0000 7f0057200700 1 -- 172.21.15.62:0/2074268153 shutdown_connections 2024-04-20T22:29:43.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:43.403+0000 7f0057200700 1 --2- 172.21.15.62:0/2074268153 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0058100c70 0x7f0058104c40 unknown :-1 s=CLOSED pgs=1388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:29:43.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:43.403+0000 7f0057200700 1 --2- 172.21.15.62:0/2074268153 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f00581002d0 0x7f00581006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:29:43.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:43.403+0000 7f0057200700 1 -- 172.21.15.62:0/2074268153 >> 172.21.15.62:0/2074268153 conn(0x7f00580fbbb0 msgr2=0x7f00580fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:29:43.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:43.403+0000 7f0057200700 1 -- 172.21.15.62:0/2074268153 shutdown_connections 2024-04-20T22:29:43.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:43.403+0000 7f0057200700 1 -- 172.21.15.62:0/2074268153 wait complete. 2024-04-20T22:29:43.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:43.404+0000 7f0057200700 1 Processor -- start 2024-04-20T22:29:43.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:43.405+0000 7f0057200700 1 -- start start 2024-04-20T22:29:43.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:43.405+0000 7f0057200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f00581002d0 0x7f0058197eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:29:43.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:43.405+0000 7f0057200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0058100c70 0x7f00581983f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:29:43.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:43.405+0000 7f0057200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0058198a40 con 0x7f0058100c70 2024-04-20T22:29:43.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:43.405+0000 7f0057200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0058198b80 con 0x7f00581002d0 2024-04-20T22:29:43.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:43.405+0000 7f0055400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0058100c70 0x7f00581983f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:29:43.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:43.405+0000 7f0055400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0058100c70 0x7f00581983f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:32902/0 (socket says 172.21.15.62:32902) 2024-04-20T22:29:43.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:43.405+0000 7f0055400700 1 -- 172.21.15.62:0/3816559957 learned_addr learned my addr 172.21.15.62:0/3816559957 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:29:43.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:43.406+0000 7f0055400700 1 -- 172.21.15.62:0/3816559957 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f00581002d0 msgr2=0x7f0058197eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:29:43.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:43.406+0000 7f0055e00700 1 --2- 172.21.15.62:0/3816559957 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f00581002d0 0x7f0058197eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:29:43.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:43.406+0000 7f0055400700 1 --2- 172.21.15.62:0/3816559957 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f00581002d0 0x7f0058197eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:29:43.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:43.406+0000 7f0055400700 1 -- 172.21.15.62:0/3816559957 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00440095f0 con 0x7f0058100c70 2024-04-20T22:29:43.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:43.406+0000 7f0055400700 1 --2- 172.21.15.62:0/3816559957 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0058100c70 0x7f00581983f0 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7f004400bda0 tx=0x7f004400be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:29:43.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:43.407+0000 7f004ea00700 1 -- 172.21.15.62:0/3816559957 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f004401d070 con 0x7f0058100c70 2024-04-20T22:29:43.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:43.407+0000 7f0057200700 1 -- 172.21.15.62:0/3816559957 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f005819c970 con 0x7f0058100c70 2024-04-20T22:29:43.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:43.407+0000 7f004ea00700 1 -- 172.21.15.62:0/3816559957 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0044004fb0 con 0x7f0058100c70 2024-04-20T22:29:43.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:43.407+0000 7f004ea00700 1 -- 172.21.15.62:0/3816559957 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f004400f480 con 0x7f0058100c70 2024-04-20T22:29:43.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:43.407+0000 7f0057200700 1 -- 172.21.15.62:0/3816559957 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f005819ce60 con 0x7f0058100c70 2024-04-20T22:29:43.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:43.408+0000 7f0057200700 1 -- 172.21.15.62:0/3816559957 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0058062380 con 0x7f0058100c70 2024-04-20T22:29:43.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:43.409+0000 7f004ea00700 1 -- 172.21.15.62:0/3816559957 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f0044022870 con 0x7f0058100c70 2024-04-20T22:29:43.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:43.411+0000 7f004ea00700 1 --2- 172.21.15.62:0/3816559957 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f003c06ecc0 0x7f003c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:29:43.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:43.411+0000 7f004ea00700 1 -- 172.21.15.62:0/3816559957 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f0044090440 con 0x7f0058100c70 2024-04-20T22:29:43.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:43.411+0000 7f0055e00700 1 --2- 172.21.15.62:0/3816559957 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f003c06ecc0 0x7f003c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:29:43.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:43.412+0000 7f0055e00700 1 --2- 172.21.15.62:0/3816559957 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f003c06ecc0 0x7f003c071170 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f004800ba60 tx=0x7f0048005d50 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:29:43.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:43.416+0000 7f004ea00700 1 -- 172.21.15.62:0/3816559957 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f0044059200 con 0x7f0058100c70 2024-04-20T22:29:43.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:43.746+0000 7f0057200700 1 -- 172.21.15.62:0/3816559957 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f0058199210 con 0x7f0058100c70 2024-04-20T22:29:43.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:43.748+0000 7f004ea00700 1 -- 172.21.15.62:0/3816559957 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f004405c8c0 con 0x7f0058100c70 2024-04-20T22:29:43.748 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:29:43.748 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T22:29:43.748 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:29:43.748 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:29:43.748 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T22:29:43.748 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:29:43.748 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:29:43.749 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T22:29:43.749 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 6 2024-04-20T22:29:43.749 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:29:43.749 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T22:29:43.749 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 4 2024-04-20T22:29:43.749 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:29:43.749 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T22:29:43.749 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 14 2024-04-20T22:29:43.749 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T22:29:43.749 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:29:43.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:43.751+0000 7f0057200700 1 -- 172.21.15.62:0/3816559957 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f003c06ecc0 msgr2=0x7f003c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:29:43.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:43.751+0000 7f0057200700 1 --2- 172.21.15.62:0/3816559957 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f003c06ecc0 0x7f003c071170 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f004800ba60 tx=0x7f0048005d50 comp rx=0 tx=0).stop 2024-04-20T22:29:43.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:43.751+0000 7f0057200700 1 -- 172.21.15.62:0/3816559957 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0058100c70 msgr2=0x7f00581983f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:29:43.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:43.751+0000 7f0057200700 1 --2- 172.21.15.62:0/3816559957 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0058100c70 0x7f00581983f0 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7f004400bda0 tx=0x7f004400be80 comp rx=0 tx=0).stop 2024-04-20T22:29:43.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:43.751+0000 7f0057200700 1 -- 172.21.15.62:0/3816559957 shutdown_connections 2024-04-20T22:29:43.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:43.751+0000 7f0057200700 1 --2- 172.21.15.62:0/3816559957 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f003c06ecc0 0x7f003c071170 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:29:43.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:43.751+0000 7f0057200700 1 --2- 172.21.15.62:0/3816559957 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f00581002d0 0x7f0058197eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:29:43.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:43.751+0000 7f0057200700 1 --2- 172.21.15.62:0/3816559957 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0058100c70 0x7f00581983f0 unknown :-1 s=CLOSED pgs=1389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:29:43.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:43.751+0000 7f0057200700 1 -- 172.21.15.62:0/3816559957 >> 172.21.15.62:0/3816559957 conn(0x7f00580fbbb0 msgr2=0x7f00580fd130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:29:43.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:43.751+0000 7f0057200700 1 -- 172.21.15.62:0/3816559957 shutdown_connections 2024-04-20T22:29:43.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:29:43.751+0000 7f0057200700 1 -- 172.21.15.62:0/3816559957 wait complete. 2024-04-20T22:29:44.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:29:43 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/3816559957' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:29:44.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:29:43 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/3816559957' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:29:44.544 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -e sha1=a8c84f5997950fd7d008204839253291d52e6eda -- bash -c 'echo "wait for servicemap items w/ changing names to refresh"' 2024-04-20T22:29:44.988 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T22:29:45.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:29:44 smithi062 ceph-mon[76322]: pgmap v1638: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:29:45.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:29:44 smithi189 ceph-mon[57022]: pgmap v1638: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:29:47.201 INFO:teuthology.orchestra.run.smithi062.stdout:wait for servicemap items w/ changing names to refresh 2024-04-20T22:29:47.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:29:46 smithi062 ceph-mon[76322]: pgmap v1639: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:29:47.290 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:29:46 smithi189 ceph-mon[57022]: pgmap v1639: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:29:47.939 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -e sha1=a8c84f5997950fd7d008204839253291d52e6eda -- bash -c 'sleep 60' 2024-04-20T22:29:48.371 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T22:29:49.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:29:49 smithi189 ceph-mon[57022]: pgmap v1640: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:29:49.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:29:49 smithi062 ceph-mon[76322]: pgmap v1640: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:29:50.183 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:29:50 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:29:50.490 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:29:50 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:29:51.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:29:51 smithi062 ceph-mon[76322]: pgmap v1641: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:29:51.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:29:51 smithi189 ceph-mon[57022]: pgmap v1641: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:29:53.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:29:53 smithi062 ceph-mon[76322]: pgmap v1642: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:29:53.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:29:53 smithi189 ceph-mon[57022]: pgmap v1642: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:29:55.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:29:55 smithi062 ceph-mon[76322]: pgmap v1643: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:29:55.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:29:55 smithi189 ceph-mon[57022]: pgmap v1643: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:29:57.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:29:57 smithi189 ceph-mon[57022]: pgmap v1644: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:29:57.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:29:57 smithi062 ceph-mon[76322]: pgmap v1644: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:29:59.179 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:29:59 smithi062 ceph-mon[76322]: pgmap v1645: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:29:59.302 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:29:59 smithi189 ceph-mon[57022]: pgmap v1645: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:30:00.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:30:00 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T22:30:00.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:30:00 smithi062 ceph-mon[76322]: Health detail: HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-04-20T22:30:00.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:30:00 smithi062 ceph-mon[76322]: [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T22:30:00.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:30:00 smithi062 ceph-mon[76322]: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T22:30:00.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:30:00 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T22:30:00.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:30:00 smithi189 ceph-mon[57022]: Health detail: HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-04-20T22:30:00.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:30:00 smithi189 ceph-mon[57022]: [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-20T22:30:00.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:30:00 smithi189 ceph-mon[57022]: fs cephfs has deprecated feature inline_data enabled. 2024-04-20T22:30:01.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:30:01 smithi189 ceph-mon[57022]: pgmap v1646: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:30:01.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:30:01 smithi062 ceph-mon[76322]: pgmap v1646: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:30:02.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:30:02 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T22:30:02.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:30:02 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T22:30:02.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:30:02 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:30:02.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:30:02 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T22:30:02.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:30:02 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T22:30:02.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:30:02 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:30:03.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:30:03 smithi062 ceph-mon[76322]: pgmap v1647: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:30:03.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:30:03 smithi189 ceph-mon[57022]: pgmap v1647: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:30:05.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:30:05 smithi062 ceph-mon[76322]: pgmap v1648: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:30:05.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:30:05 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:30:05.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:30:05 smithi189 ceph-mon[57022]: pgmap v1648: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:30:05.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:30:05 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:30:07.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:30:07 smithi062 ceph-mon[76322]: pgmap v1649: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:30:07.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:30:07 smithi189 ceph-mon[57022]: pgmap v1649: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:30:09.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:30:09 smithi062 ceph-mon[76322]: pgmap v1650: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:30:09.647 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:30:09 smithi189 ceph-mon[57022]: pgmap v1650: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:30:11.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:30:11 smithi062 ceph-mon[76322]: pgmap v1651: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:30:11.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:30:11 smithi189 ceph-mon[57022]: pgmap v1651: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:30:13.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:30:13 smithi062 ceph-mon[76322]: pgmap v1652: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:30:13.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:30:13 smithi189 ceph-mon[57022]: pgmap v1652: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:30:15.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:30:15 smithi189 ceph-mon[57022]: pgmap v1653: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:30:15.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:30:15 smithi062 ceph-mon[76322]: pgmap v1653: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:30:17.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:30:17 smithi189 ceph-mon[57022]: pgmap v1654: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:30:17.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:30:17 smithi062 ceph-mon[76322]: pgmap v1654: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:30:19.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:30:19 smithi189 ceph-mon[57022]: pgmap v1655: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:30:19.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:30:19 smithi062 ceph-mon[76322]: pgmap v1655: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:30:20.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:30:20 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:30:20.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:30:20 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:30:21.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:30:21 smithi189 ceph-mon[57022]: pgmap v1656: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:30:21.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:30:21 smithi062 ceph-mon[76322]: pgmap v1656: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:30:23.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:30:23 smithi189 ceph-mon[57022]: pgmap v1657: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:30:23.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:30:23 smithi062 ceph-mon[76322]: pgmap v1657: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:30:25.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:30:25 smithi189 ceph-mon[57022]: pgmap v1658: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:30:25.739 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:30:25 smithi062 ceph-mon[76322]: pgmap v1658: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:30:27.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:30:27 smithi189 ceph-mon[57022]: pgmap v1659: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:30:27.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:30:27 smithi062 ceph-mon[76322]: pgmap v1659: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:30:29.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:30:29 smithi062 ceph-mon[76322]: pgmap v1660: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:30:29.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:30:29 smithi189 ceph-mon[57022]: pgmap v1660: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:30:31.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:30:31 smithi189 ceph-mon[57022]: pgmap v1661: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:30:31.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:30:31 smithi062 ceph-mon[76322]: pgmap v1661: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:30:33.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:30:33 smithi189 ceph-mon[57022]: pgmap v1662: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:30:33.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:30:33 smithi062 ceph-mon[76322]: pgmap v1662: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:30:35.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:30:35 smithi189 ceph-mon[57022]: pgmap v1663: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:30:35.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:30:35 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:30:35.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:30:35 smithi062 ceph-mon[76322]: pgmap v1663: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:30:35.787 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:30:35 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:30:37.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:30:37 smithi189 ceph-mon[57022]: pgmap v1664: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:30:37.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:30:37 smithi062 ceph-mon[76322]: pgmap v1664: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:30:39.637 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:30:39 smithi062 ceph-mon[76322]: pgmap v1665: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:30:39.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:30:39 smithi189 ceph-mon[57022]: pgmap v1665: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:30:41.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:30:41 smithi062 ceph-mon[76322]: pgmap v1666: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:30:41.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:30:41 smithi189 ceph-mon[57022]: pgmap v1666: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:30:43.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:30:43 smithi189 ceph-mon[57022]: pgmap v1667: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:30:43.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:30:43 smithi062 ceph-mon[76322]: pgmap v1667: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:30:45.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:30:45 smithi189 ceph-mon[57022]: pgmap v1668: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:30:45.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:30:45 smithi062 ceph-mon[76322]: pgmap v1668: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:30:47.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:30:47 smithi189 ceph-mon[57022]: pgmap v1669: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:30:47.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:30:47 smithi062 ceph-mon[76322]: pgmap v1669: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:30:49.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:30:49 smithi189 ceph-mon[57022]: pgmap v1670: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:30:49.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:30:49 smithi062 ceph-mon[76322]: pgmap v1670: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:30:50.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:30:50 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:30:50.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:30:50 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:30:51.508 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -e sha1=a8c84f5997950fd7d008204839253291d52e6eda -- bash -c 'ceph orch ps' 2024-04-20T22:30:51.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:30:51 smithi189 ceph-mon[57022]: pgmap v1671: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:30:51.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:30:51 smithi062 ceph-mon[76322]: pgmap v1671: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:30:51.949 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T22:30:53.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:30:53 smithi062 ceph-mon[76322]: pgmap v1672: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:30:53.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:30:53 smithi189 ceph-mon[57022]: pgmap v1672: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:30:54.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:30:54.544+0000 7fe7b9c00700 1 -- 172.21.15.62:0/616259940 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe7b40739a0 msgr2=0x7fe7b410c5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:30:54.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:30:54.544+0000 7fe7b9c00700 1 --2- 172.21.15.62:0/616259940 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe7b40739a0 0x7fe7b410c5e0 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7fe79c009940 tx=0x7fe79c009c50 comp rx=0 tx=0).stop 2024-04-20T22:30:54.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:30:54.544+0000 7fe7b9c00700 1 -- 172.21.15.62:0/616259940 shutdown_connections 2024-04-20T22:30:54.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:30:54.544+0000 7fe7b9c00700 1 --2- 172.21.15.62:0/616259940 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe7b40739a0 0x7fe7b410c5e0 unknown :-1 s=CLOSED pgs=1390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:30:54.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:30:54.544+0000 7fe7b9c00700 1 --2- 172.21.15.62:0/616259940 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe7b4073090 0x7fe7b4073460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:30:54.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:30:54.544+0000 7fe7b9c00700 1 -- 172.21.15.62:0/616259940 >> 172.21.15.62:0/616259940 conn(0x7fe7b40fbda0 msgr2=0x7fe7b40fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:30:54.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:30:54.544+0000 7fe7b9c00700 1 -- 172.21.15.62:0/616259940 shutdown_connections 2024-04-20T22:30:54.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:30:54.544+0000 7fe7b9c00700 1 -- 172.21.15.62:0/616259940 wait complete. 2024-04-20T22:30:54.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:30:54.545+0000 7fe7b9c00700 1 Processor -- start 2024-04-20T22:30:54.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:30:54.545+0000 7fe7b9c00700 1 -- start start 2024-04-20T22:30:54.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:30:54.546+0000 7fe7b9c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe7b4073090 0x7fe7b41981b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:30:54.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:30:54.546+0000 7fe7b9c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe7b40739a0 0x7fe7b41986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:30:54.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:30:54.546+0000 7fe7b9c00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7b4198dd0 con 0x7fe7b4073090 2024-04-20T22:30:54.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:30:54.546+0000 7fe7b9c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7b419cb60 con 0x7fe7b40739a0 2024-04-20T22:30:54.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:30:54.546+0000 7fe7b2a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe7b4073090 0x7fe7b41981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:30:54.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:30:54.546+0000 7fe7b2a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe7b4073090 0x7fe7b41981b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43338/0 (socket says 172.21.15.62:43338) 2024-04-20T22:30:54.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:30:54.546+0000 7fe7b2a00700 1 -- 172.21.15.62:0/4216667091 learned_addr learned my addr 172.21.15.62:0/4216667091 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:30:54.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:30:54.546+0000 7fe7abe00700 1 --2- 172.21.15.62:0/4216667091 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe7b40739a0 0x7fe7b41986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:30:54.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:30:54.547+0000 7fe7b2a00700 1 -- 172.21.15.62:0/4216667091 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe7b40739a0 msgr2=0x7fe7b41986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:30:54.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:30:54.547+0000 7fe7b2a00700 1 --2- 172.21.15.62:0/4216667091 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe7b40739a0 0x7fe7b41986f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:30:54.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:30:54.547+0000 7fe7b2a00700 1 -- 172.21.15.62:0/4216667091 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe79c0095f0 con 0x7fe7b4073090 2024-04-20T22:30:54.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:30:54.547+0000 7fe7b2a00700 1 --2- 172.21.15.62:0/4216667091 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe7b4073090 0x7fe7b41981b0 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7fe7a400d8d0 tx=0x7fe7a400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:30:54.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:30:54.547+0000 7fe7ab400700 1 -- 172.21.15.62:0/4216667091 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe7a4009940 con 0x7fe7b4073090 2024-04-20T22:30:54.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:30:54.547+0000 7fe7ab400700 1 -- 172.21.15.62:0/4216667091 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe7a4010460 con 0x7fe7b4073090 2024-04-20T22:30:54.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:30:54.547+0000 7fe7ab400700 1 -- 172.21.15.62:0/4216667091 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe7a400f5f0 con 0x7fe7b4073090 2024-04-20T22:30:54.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:30:54.547+0000 7fe7b9c00700 1 -- 172.21.15.62:0/4216667091 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7b419ce40 con 0x7fe7b4073090 2024-04-20T22:30:54.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:30:54.548+0000 7fe7b9c00700 1 -- 172.21.15.62:0/4216667091 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7b419d390 con 0x7fe7b4073090 2024-04-20T22:30:54.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:30:54.549+0000 7fe7b9c00700 1 -- 172.21.15.62:0/4216667091 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7b4109ce0 con 0x7fe7b4073090 2024-04-20T22:30:54.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:30:54.550+0000 7fe7ab400700 1 -- 172.21.15.62:0/4216667091 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe7a40105d0 con 0x7fe7b4073090 2024-04-20T22:30:54.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:30:54.551+0000 7fe7ab400700 1 --2- 172.21.15.62:0/4216667091 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe79806ecc0 0x7fe798071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:30:54.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:30:54.551+0000 7fe7ab400700 1 -- 172.21.15.62:0/4216667091 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fe7a408ee20 con 0x7fe7b4073090 2024-04-20T22:30:54.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:30:54.551+0000 7fe7abe00700 1 --2- 172.21.15.62:0/4216667091 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe79806ecc0 0x7fe798071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:30:54.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:30:54.552+0000 7fe7abe00700 1 --2- 172.21.15.62:0/4216667091 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe79806ecc0 0x7fe798071170 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fe7b41997d0 tx=0x7fe79c005ab0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:30:54.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:30:54.556+0000 7fe7ab400700 1 -- 172.21.15.62:0/4216667091 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fe7a4057bb0 con 0x7fe7b4073090 2024-04-20T22:30:54.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:30:54.791+0000 7fe7b9c00700 1 -- 172.21.15.62:0/4216667091 --> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fe7b4199580 con 0x7fe79806ecc0 2024-04-20T22:30:54.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:30:54.822+0000 7fe7ab400700 1 -- 172.21.15.62:0/4216667091 <== mgr.34102 v2:172.21.15.62:6800/3925433479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fe7b4199580 con 0x7fe79806ecc0 2024-04-20T22:30:54.823 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-20T22:30:54.823 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.smithi062 smithi062 *:9093,9094 running (56m) 2m ago 70m 14.4M - 0.25.0 c8568f914cd2 bac011d08d18 2024-04-20T22:30:54.823 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi062 smithi062 running (2m) 2m ago 70m 7876k - 19.0.0-3133-ga8c84f59 76bc6b91b726 a994518fe0b6 2024-04-20T22:30:54.823 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-exporter.smithi189 smithi189 running (2m) 2m ago 69m 7105k - 19.0.0-3133-ga8c84f59 76bc6b91b726 3f5760a59416 2024-04-20T22:30:54.824 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi062 smithi062 running (52m) 2m ago 70m 6660k - 19.0.0-3133-ga8c84f59 76bc6b91b726 2825d5ba11c6 2024-04-20T22:30:54.824 INFO:teuthology.orchestra.run.smithi062.stdout:crash.smithi189 smithi189 running (52m) 2m ago 69m 6665k - 19.0.0-3133-ga8c84f59 76bc6b91b726 20f746fa1bba 2024-04-20T22:30:54.824 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.smithi062 smithi062 *:3000 running (56m) 2m ago 70m 75.6M - 9.4.12 2bacad6d85d8 d0acb027b77b 2024-04-20T22:30:54.824 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ozizwi smithi062 running (48m) 2m ago 63m 62.3M - 19.0.0-3133-ga8c84f59 76bc6b91b726 de80bc90a683 2024-04-20T22:30:54.824 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi062.ryqblb smithi062 running (48m) 2m ago 63m 64.2M - 19.0.0-3133-ga8c84f59 76bc6b91b726 cd793b7ee98e 2024-04-20T22:30:54.824 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.dvimer smithi189 running (32m) 2m ago 63m 62.8M - 19.0.0-3133-ga8c84f59 76bc6b91b726 0e7b0bc4d1ab 2024-04-20T22:30:54.824 INFO:teuthology.orchestra.run.smithi062.stdout:mds.cephfs.smithi189.xbgspw smithi189 running (17m) 2m ago 63m 60.7M - 19.0.0-3133-ga8c84f59 76bc6b91b726 0af732ac9e06 2024-04-20T22:30:54.824 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi062.rqxuxi smithi062 *:8443,9283,8765 running (59m) 2m ago 72m 521M - 19.0.0-3133-ga8c84f59 76bc6b91b726 8a885074cb1a 2024-04-20T22:30:54.824 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.smithi189.tawavz smithi189 *:8443,9283,8765 running (58m) 2m ago 69m 460M - 19.0.0-3133-ga8c84f59 76bc6b91b726 6464a4a9e03f 2024-04-20T22:30:54.824 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi062 smithi062 running (54m) 2m ago 72m 103M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 019eb1479485 2024-04-20T22:30:54.824 INFO:teuthology.orchestra.run.smithi062.stdout:mon.smithi189 smithi189 running (53m) 2m ago 69m 66.9M 2048M 19.0.0-3133-ga8c84f59 76bc6b91b726 55aac560aab6 2024-04-20T22:30:54.824 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi062 smithi062 *:9100 running (57m) 2m ago 70m 8975k - 1.5.0 0da6a335fe13 01dc044a3123 2024-04-20T22:30:54.824 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.smithi189 smithi189 *:9100 running (57m) 2m ago 69m 8850k - 1.5.0 0da6a335fe13 28475b8921e5 2024-04-20T22:30:54.824 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (52m) 2m ago 68m 98.6M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 ee2bb901f26e 2024-04-20T22:30:54.825 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (51m) 2m ago 67m 104M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 9075152d4162 2024-04-20T22:30:54.825 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (51m) 2m ago 66m 86.8M 1216M 19.0.0-3133-ga8c84f59 76bc6b91b726 2d4f3e38dc0e 2024-04-20T22:30:54.825 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi189 running (50m) 2m ago 66m 104M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 8f97b17ae0eb 2024-04-20T22:30:54.825 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi189 running (49m) 2m ago 65m 103M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 653d8079967d 2024-04-20T22:30:54.825 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi189 running (49m) 2m ago 65m 97.4M 2240M 19.0.0-3133-ga8c84f59 76bc6b91b726 596dded14c12 2024-04-20T22:30:54.825 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.smithi062 smithi062 *:9095 running (57m) 2m ago 70m 89.5M - 2.43.0 a07b618ecd1d 8b67d41aefa8 2024-04-20T22:30:54.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:30:54.825+0000 7fe7b9c00700 1 -- 172.21.15.62:0/4216667091 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe79806ecc0 msgr2=0x7fe798071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:30:54.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:30:54.826+0000 7fe7b9c00700 1 --2- 172.21.15.62:0/4216667091 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe79806ecc0 0x7fe798071170 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fe7b41997d0 tx=0x7fe79c005ab0 comp rx=0 tx=0).stop 2024-04-20T22:30:54.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:30:54.826+0000 7fe7b9c00700 1 -- 172.21.15.62:0/4216667091 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe7b4073090 msgr2=0x7fe7b41981b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:30:54.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:30:54.826+0000 7fe7b9c00700 1 --2- 172.21.15.62:0/4216667091 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe7b4073090 0x7fe7b41981b0 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7fe7a400d8d0 tx=0x7fe7a400dc90 comp rx=0 tx=0).stop 2024-04-20T22:30:54.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:30:54.826+0000 7fe7b9c00700 1 -- 172.21.15.62:0/4216667091 shutdown_connections 2024-04-20T22:30:54.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:30:54.826+0000 7fe7b9c00700 1 --2- 172.21.15.62:0/4216667091 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe79806ecc0 0x7fe798071170 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:30:54.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:30:54.826+0000 7fe7b9c00700 1 --2- 172.21.15.62:0/4216667091 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe7b4073090 0x7fe7b41981b0 unknown :-1 s=CLOSED pgs=1391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:30:54.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:30:54.826+0000 7fe7b9c00700 1 --2- 172.21.15.62:0/4216667091 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe7b40739a0 0x7fe7b41986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:30:54.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:30:54.826+0000 7fe7b9c00700 1 -- 172.21.15.62:0/4216667091 >> 172.21.15.62:0/4216667091 conn(0x7fe7b40fbda0 msgr2=0x7fe7b4106e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:30:54.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:30:54.826+0000 7fe7b9c00700 1 -- 172.21.15.62:0/4216667091 shutdown_connections 2024-04-20T22:30:54.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:30:54.826+0000 7fe7b9c00700 1 -- 172.21.15.62:0/4216667091 wait complete. 2024-04-20T22:30:55.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:30:55 smithi062 ceph-mon[76322]: pgmap v1673: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:30:55.665 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -e sha1=a8c84f5997950fd7d008204839253291d52e6eda -- bash -c 'ceph versions' 2024-04-20T22:30:55.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:30:55 smithi189 ceph-mon[57022]: pgmap v1673: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:30:56.105 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T22:30:56.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:30:56 smithi062 ceph-mon[76322]: from='client.36856 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:30:56.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:30:56 smithi189 ceph-mon[57022]: from='client.36856 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T22:30:57.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:30:57 smithi189 ceph-mon[57022]: pgmap v1674: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:30:57.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:30:57 smithi062 ceph-mon[76322]: pgmap v1674: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:30:58.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:30:58.374+0000 7f7f3a600700 1 -- 172.21.15.62:0/802353261 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f34073a00 msgr2=0x7f7f3410c580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:30:58.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:30:58.374+0000 7f7f3a600700 1 --2- 172.21.15.62:0/802353261 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f34073a00 0x7f7f3410c580 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7f7f20009910 tx=0x7f7f20009c20 comp rx=0 tx=0).stop 2024-04-20T22:30:58.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:30:58.374+0000 7f7f3a600700 1 -- 172.21.15.62:0/802353261 shutdown_connections 2024-04-20T22:30:58.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:30:58.374+0000 7f7f3a600700 1 --2- 172.21.15.62:0/802353261 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f34073a00 0x7f7f3410c580 unknown :-1 s=CLOSED pgs=1392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:30:58.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:30:58.374+0000 7f7f3a600700 1 --2- 172.21.15.62:0/802353261 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f340730f0 0x7f7f340734c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:30:58.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:30:58.374+0000 7f7f3a600700 1 -- 172.21.15.62:0/802353261 >> 172.21.15.62:0/802353261 conn(0x7f7f340fbda0 msgr2=0x7f7f340fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:30:58.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:30:58.374+0000 7f7f3a600700 1 -- 172.21.15.62:0/802353261 shutdown_connections 2024-04-20T22:30:58.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:30:58.374+0000 7f7f3a600700 1 -- 172.21.15.62:0/802353261 wait complete. 2024-04-20T22:30:58.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:30:58.375+0000 7f7f3a600700 1 Processor -- start 2024-04-20T22:30:58.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:30:58.375+0000 7f7f3a600700 1 -- start start 2024-04-20T22:30:58.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:30:58.376+0000 7f7f3a600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f340730f0 0x7f7f34197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:30:58.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:30:58.376+0000 7f7f3a600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f34073a00 0x7f7f341985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:30:58.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:30:58.376+0000 7f7f3a600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f3419c930 con 0x7f7f340730f0 2024-04-20T22:30:58.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:30:58.376+0000 7f7f3a600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f3419caa0 con 0x7f7f34073a00 2024-04-20T22:30:58.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:30:58.376+0000 7f7f33400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f340730f0 0x7f7f34197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:30:58.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:30:58.376+0000 7f7f33400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f340730f0 0x7f7f34197f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43360/0 (socket says 172.21.15.62:43360) 2024-04-20T22:30:58.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:30:58.376+0000 7f7f33400700 1 -- 172.21.15.62:0/1152200710 learned_addr learned my addr 172.21.15.62:0/1152200710 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:30:58.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:30:58.376+0000 7f7f32a00700 1 --2- 172.21.15.62:0/1152200710 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f34073a00 0x7f7f341985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:30:58.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:30:58.377+0000 7f7f33400700 1 -- 172.21.15.62:0/1152200710 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f34073a00 msgr2=0x7f7f341985f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:30:58.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:30:58.377+0000 7f7f33400700 1 --2- 172.21.15.62:0/1152200710 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f34073a00 0x7f7f341985f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:30:58.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:30:58.377+0000 7f7f33400700 1 -- 172.21.15.62:0/1152200710 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f200095f0 con 0x7f7f340730f0 2024-04-20T22:30:58.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:30:58.377+0000 7f7f33400700 1 --2- 172.21.15.62:0/1152200710 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f340730f0 0x7f7f34197f80 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f7f2400e920 tx=0x7f7f2400ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:30:58.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:30:58.377+0000 7f7f2be00700 1 -- 172.21.15.62:0/1152200710 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7f24009790 con 0x7f7f340730f0 2024-04-20T22:30:58.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:30:58.377+0000 7f7f2be00700 1 -- 172.21.15.62:0/1152200710 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7f2400cba0 con 0x7f7f340730f0 2024-04-20T22:30:58.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:30:58.377+0000 7f7f3a600700 1 -- 172.21.15.62:0/1152200710 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f3419cd80 con 0x7f7f340730f0 2024-04-20T22:30:58.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:30:58.377+0000 7f7f2be00700 1 -- 172.21.15.62:0/1152200710 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7f24018650 con 0x7f7f340730f0 2024-04-20T22:30:58.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:30:58.378+0000 7f7f3a600700 1 -- 172.21.15.62:0/1152200710 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f3419d2d0 con 0x7f7f340730f0 2024-04-20T22:30:58.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:30:58.379+0000 7f7f3a600700 1 -- 172.21.15.62:0/1152200710 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f34109c80 con 0x7f7f340730f0 2024-04-20T22:30:58.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:30:58.380+0000 7f7f2be00700 1 -- 172.21.15.62:0/1152200710 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f7f24010430 con 0x7f7f340730f0 2024-04-20T22:30:58.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:30:58.381+0000 7f7f2be00700 1 --2- 172.21.15.62:0/1152200710 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7f1806ecc0 0x7f7f18071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:30:58.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:30:58.381+0000 7f7f2be00700 1 -- 172.21.15.62:0/1152200710 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f7f24014070 con 0x7f7f340730f0 2024-04-20T22:30:58.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:30:58.381+0000 7f7f32a00700 1 --2- 172.21.15.62:0/1152200710 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7f1806ecc0 0x7f7f18071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:30:58.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:30:58.382+0000 7f7f32a00700 1 --2- 172.21.15.62:0/1152200710 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7f1806ecc0 0x7f7f18071170 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f7f2000b760 tx=0x7f7f20005fd0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:30:58.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:30:58.386+0000 7f7f2be00700 1 -- 172.21.15.62:0/1152200710 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f7f24053ff0 con 0x7f7f340730f0 2024-04-20T22:30:58.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:30:58.720+0000 7f7f3a600700 1 -- 172.21.15.62:0/1152200710 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f7f341992e0 con 0x7f7f340730f0 2024-04-20T22:30:58.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:30:58.721+0000 7f7f2be00700 1 -- 172.21.15.62:0/1152200710 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f7f2405bb00 con 0x7f7f340730f0 2024-04-20T22:30:58.721 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-20T22:30:58.721 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-04-20T22:30:58.722 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:30:58.722 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:30:58.722 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-04-20T22:30:58.722 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 2 2024-04-20T22:30:58.722 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:30:58.722 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-04-20T22:30:58.722 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 6 2024-04-20T22:30:58.722 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:30:58.722 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-04-20T22:30:58.722 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 4 2024-04-20T22:30:58.722 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-20T22:30:58.722 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-04-20T22:30:58.722 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)": 14 2024-04-20T22:30:58.722 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-20T22:30:58.723 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-20T22:30:58.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:30:58.724+0000 7f7f3a600700 1 -- 172.21.15.62:0/1152200710 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7f1806ecc0 msgr2=0x7f7f18071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:30:58.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:30:58.724+0000 7f7f3a600700 1 --2- 172.21.15.62:0/1152200710 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7f1806ecc0 0x7f7f18071170 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f7f2000b760 tx=0x7f7f20005fd0 comp rx=0 tx=0).stop 2024-04-20T22:30:58.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:30:58.724+0000 7f7f3a600700 1 -- 172.21.15.62:0/1152200710 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f340730f0 msgr2=0x7f7f34197f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:30:58.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:30:58.724+0000 7f7f3a600700 1 --2- 172.21.15.62:0/1152200710 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f340730f0 0x7f7f34197f80 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f7f2400e920 tx=0x7f7f2400ece0 comp rx=0 tx=0).stop 2024-04-20T22:30:58.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:30:58.725+0000 7f7f3a600700 1 -- 172.21.15.62:0/1152200710 shutdown_connections 2024-04-20T22:30:58.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:30:58.725+0000 7f7f3a600700 1 --2- 172.21.15.62:0/1152200710 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7f1806ecc0 0x7f7f18071170 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:30:58.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:30:58.725+0000 7f7f3a600700 1 --2- 172.21.15.62:0/1152200710 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f340730f0 0x7f7f34197f80 unknown :-1 s=CLOSED pgs=1393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:30:58.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:30:58.725+0000 7f7f3a600700 1 --2- 172.21.15.62:0/1152200710 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f34073a00 0x7f7f341985f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:30:58.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:30:58.725+0000 7f7f3a600700 1 -- 172.21.15.62:0/1152200710 >> 172.21.15.62:0/1152200710 conn(0x7f7f340fbda0 msgr2=0x7f7f34106dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:30:58.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:30:58.725+0000 7f7f3a600700 1 -- 172.21.15.62:0/1152200710 shutdown_connections 2024-04-20T22:30:58.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:30:58.725+0000 7f7f3a600700 1 -- 172.21.15.62:0/1152200710 wait complete. 2024-04-20T22:30:59.477 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -e sha1=a8c84f5997950fd7d008204839253291d52e6eda -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 1'"'"'' 2024-04-20T22:30:59.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:30:59 smithi189 ceph-mon[57022]: pgmap v1675: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:30:59.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:30:59 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/1152200710' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:30:59.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:30:59 smithi062 ceph-mon[76322]: pgmap v1675: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:30:59.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:30:59 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/1152200710' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:30:59.916 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T22:31:01.717 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:31:01 smithi062 ceph-mon[76322]: pgmap v1676: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:31:01.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:31:01 smithi189 ceph-mon[57022]: pgmap v1676: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:31:02.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:02.023+0000 7ffad4400700 1 -- 172.21.15.62:0/2861732741 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffacc106320 msgr2=0x7ffacc1066f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:02.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:02.023+0000 7ffad4400700 1 --2- 172.21.15.62:0/2861732741 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffacc106320 0x7ffacc1066f0 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7ffac0009910 tx=0x7ffac0009c20 comp rx=0 tx=0).stop 2024-04-20T22:31:02.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:02.023+0000 7ffad4400700 1 -- 172.21.15.62:0/2861732741 shutdown_connections 2024-04-20T22:31:02.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:02.023+0000 7ffad4400700 1 --2- 172.21.15.62:0/2861732741 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffacc068540 0x7ffacc0689b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:02.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:02.023+0000 7ffad4400700 1 --2- 172.21.15.62:0/2861732741 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffacc106320 0x7ffacc1066f0 unknown :-1 s=CLOSED pgs=1394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:02.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:02.023+0000 7ffad4400700 1 -- 172.21.15.62:0/2861732741 >> 172.21.15.62:0/2861732741 conn(0x7ffacc075130 msgr2=0x7ffacc075530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:31:02.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:02.024+0000 7ffad4400700 1 -- 172.21.15.62:0/2861732741 shutdown_connections 2024-04-20T22:31:02.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:02.024+0000 7ffad4400700 1 -- 172.21.15.62:0/2861732741 wait complete. 2024-04-20T22:31:02.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:02.025+0000 7ffad4400700 1 Processor -- start 2024-04-20T22:31:02.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:02.025+0000 7ffad4400700 1 -- start start 2024-04-20T22:31:02.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:02.026+0000 7ffad4400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffacc068540 0x7ffacc195e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:02.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:02.026+0000 7ffad4400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffacc106320 0x7ffacc1963b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:02.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:02.026+0000 7ffad4400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffacc196a90 con 0x7ffacc068540 2024-04-20T22:31:02.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:02.026+0000 7ffad4400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffacc19a820 con 0x7ffacc106320 2024-04-20T22:31:02.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:02.026+0000 7ffad1a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffacc068540 0x7ffacc195e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:02.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:02.026+0000 7ffad1a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffacc068540 0x7ffacc195e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49120/0 (socket says 172.21.15.62:49120) 2024-04-20T22:31:02.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:02.026+0000 7ffad1a00700 1 -- 172.21.15.62:0/999225746 learned_addr learned my addr 172.21.15.62:0/999225746 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:31:02.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:02.027+0000 7ffad1000700 1 --2- 172.21.15.62:0/999225746 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffacc106320 0x7ffacc1963b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:02.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:02.027+0000 7ffad1a00700 1 -- 172.21.15.62:0/999225746 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffacc106320 msgr2=0x7ffacc1963b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:02.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:02.027+0000 7ffad1a00700 1 --2- 172.21.15.62:0/999225746 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffacc106320 0x7ffacc1963b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:02.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:02.027+0000 7ffad1a00700 1 -- 172.21.15.62:0/999225746 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffac00095f0 con 0x7ffacc068540 2024-04-20T22:31:02.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:02.028+0000 7ffad1a00700 1 --2- 172.21.15.62:0/999225746 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffacc068540 0x7ffacc195e70 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7ffac0006010 tx=0x7ffac000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:31:02.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:02.028+0000 7ffaca000700 1 -- 172.21.15.62:0/999225746 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffac001d070 con 0x7ffacc068540 2024-04-20T22:31:02.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:02.028+0000 7ffaca000700 1 -- 172.21.15.62:0/999225746 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ffac0005dc0 con 0x7ffacc068540 2024-04-20T22:31:02.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:02.028+0000 7ffad4400700 1 -- 172.21.15.62:0/999225746 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffacc19aaa0 con 0x7ffacc068540 2024-04-20T22:31:02.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:02.028+0000 7ffad4400700 1 -- 172.21.15.62:0/999225746 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffacc19af90 con 0x7ffacc068540 2024-04-20T22:31:02.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:02.029+0000 7ffaca000700 1 -- 172.21.15.62:0/999225746 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffac0017400 con 0x7ffacc068540 2024-04-20T22:31:02.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:02.029+0000 7ffad4400700 1 -- 172.21.15.62:0/999225746 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffacc108790 con 0x7ffacc068540 2024-04-20T22:31:02.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:02.031+0000 7ffaca000700 1 -- 172.21.15.62:0/999225746 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ffac000f460 con 0x7ffacc068540 2024-04-20T22:31:02.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:02.032+0000 7ffaca000700 1 --2- 172.21.15.62:0/999225746 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ffab406ecc0 0x7ffab4071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:02.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:02.032+0000 7ffaca000700 1 -- 172.21.15.62:0/999225746 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7ffac00902e0 con 0x7ffacc068540 2024-04-20T22:31:02.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:02.033+0000 7ffad1000700 1 --2- 172.21.15.62:0/999225746 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ffab406ecc0 0x7ffab4071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:02.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:02.033+0000 7ffad1000700 1 --2- 172.21.15.62:0/999225746 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ffab406ecc0 0x7ffab4071170 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7ffacc197490 tx=0x7ffab800b410 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:31:02.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:02.042+0000 7ffaca000700 1 -- 172.21.15.62:0/999225746 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7ffac00590b0 con 0x7ffacc068540 2024-04-20T22:31:02.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:02.381+0000 7ffad4400700 1 -- 172.21.15.62:0/999225746 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7ffacc02cc30 con 0x7ffacc068540 2024-04-20T22:31:02.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:02.382+0000 7ffaca000700 1 -- 172.21.15.62:0/999225746 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7ffac005c770 con 0x7ffacc068540 2024-04-20T22:31:02.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:02.385+0000 7ffad4400700 1 -- 172.21.15.62:0/999225746 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ffab406ecc0 msgr2=0x7ffab4071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:02.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:02.385+0000 7ffad4400700 1 --2- 172.21.15.62:0/999225746 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ffab406ecc0 0x7ffab4071170 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7ffacc197490 tx=0x7ffab800b410 comp rx=0 tx=0).stop 2024-04-20T22:31:02.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:02.385+0000 7ffad4400700 1 -- 172.21.15.62:0/999225746 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffacc068540 msgr2=0x7ffacc195e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:02.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:02.385+0000 7ffad4400700 1 --2- 172.21.15.62:0/999225746 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffacc068540 0x7ffacc195e70 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7ffac0006010 tx=0x7ffac000bd70 comp rx=0 tx=0).stop 2024-04-20T22:31:02.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:02.385+0000 7ffad4400700 1 -- 172.21.15.62:0/999225746 shutdown_connections 2024-04-20T22:31:02.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:02.385+0000 7ffad4400700 1 --2- 172.21.15.62:0/999225746 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ffab406ecc0 0x7ffab4071170 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:02.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:02.385+0000 7ffad4400700 1 --2- 172.21.15.62:0/999225746 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffacc068540 0x7ffacc195e70 unknown :-1 s=CLOSED pgs=1395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:02.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:02.385+0000 7ffad4400700 1 --2- 172.21.15.62:0/999225746 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffacc106320 0x7ffacc1963b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:02.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:02.385+0000 7ffad4400700 1 -- 172.21.15.62:0/999225746 >> 172.21.15.62:0/999225746 conn(0x7ffacc075130 msgr2=0x7ffacc0fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:31:02.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:02.386+0000 7ffad4400700 1 -- 172.21.15.62:0/999225746 shutdown_connections 2024-04-20T22:31:02.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:02.386+0000 7ffad4400700 1 -- 172.21.15.62:0/999225746 wait complete. 2024-04-20T22:31:02.403 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-04-20T22:31:02.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:31:02 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T22:31:02.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:31:02 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/999225746' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:31:02.751 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:31:02 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T22:31:02.752 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:31:02 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/999225746' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:31:03.177 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -e sha1=a8c84f5997950fd7d008204839253291d52e6eda -- bash -c 'ceph versions | jq -e '"'"'.overall | keys'"'"' | grep $sha1' 2024-04-20T22:31:03.599 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T22:31:03.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:31:03 smithi189 ceph-mon[57022]: pgmap v1677: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:31:03.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:31:03 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T22:31:03.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:31:03 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T22:31:03.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:31:03 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:31:03.754 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:31:03 smithi062 ceph-mon[76322]: pgmap v1677: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:31:03.754 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:31:03 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T22:31:03.754 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:31:03 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T22:31:03.754 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:31:03 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:31:04.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:31:04 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:31:04.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:31:04 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:31:05.651 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:31:05 smithi062 ceph-mon[76322]: pgmap v1678: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:31:05.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:05.926+0000 7f5abda00700 1 -- 172.21.15.62:0/3771266259 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5ab81004e0 msgr2=0x7f5ab81008b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:05.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:05.926+0000 7f5abda00700 1 --2- 172.21.15.62:0/3771266259 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5ab81004e0 0x7f5ab81008b0 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7f5aa8009910 tx=0x7f5aa8009c20 comp rx=0 tx=0).stop 2024-04-20T22:31:05.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:05.927+0000 7f5abda00700 1 -- 172.21.15.62:0/3771266259 shutdown_connections 2024-04-20T22:31:05.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:05.927+0000 7f5abda00700 1 --2- 172.21.15.62:0/3771266259 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ab8100e80 0x7f5ab8104e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:05.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:05.927+0000 7f5abda00700 1 --2- 172.21.15.62:0/3771266259 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5ab81004e0 0x7f5ab81008b0 unknown :-1 s=CLOSED pgs=1396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:05.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:05.927+0000 7f5abda00700 1 -- 172.21.15.62:0/3771266259 >> 172.21.15.62:0/3771266259 conn(0x7f5ab80fbda0 msgr2=0x7f5ab80fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:31:05.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:05.927+0000 7f5abda00700 1 -- 172.21.15.62:0/3771266259 shutdown_connections 2024-04-20T22:31:05.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:05.927+0000 7f5abda00700 1 -- 172.21.15.62:0/3771266259 wait complete. 2024-04-20T22:31:05.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:05.928+0000 7f5abda00700 1 Processor -- start 2024-04-20T22:31:05.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:05.928+0000 7f5abda00700 1 -- start start 2024-04-20T22:31:05.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:05.928+0000 7f5abda00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ab81004e0 0x7f5ab8073050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:05.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:05.929+0000 7f5abda00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5ab8100e80 0x7f5ab8073590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:05.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:05.929+0000 7f5abda00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ab8073ad0 con 0x7f5ab8100e80 2024-04-20T22:31:05.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:05.929+0000 7f5abda00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ab8073c10 con 0x7f5ab81004e0 2024-04-20T22:31:05.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:05.929+0000 7f5aafe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5ab8100e80 0x7f5ab8073590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:05.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:05.929+0000 7f5aafe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5ab8100e80 0x7f5ab8073590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49140/0 (socket says 172.21.15.62:49140) 2024-04-20T22:31:05.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:05.929+0000 7f5aafe00700 1 -- 172.21.15.62:0/1439050388 learned_addr learned my addr 172.21.15.62:0/1439050388 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:31:05.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:05.929+0000 7f5aafe00700 1 -- 172.21.15.62:0/1439050388 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ab81004e0 msgr2=0x7f5ab8073050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:05.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:05.929+0000 7f5aafe00700 1 --2- 172.21.15.62:0/1439050388 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ab81004e0 0x7f5ab8073050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:05.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:05.929+0000 7f5aafe00700 1 -- 172.21.15.62:0/1439050388 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5aa80095f0 con 0x7f5ab8100e80 2024-04-20T22:31:05.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:05.930+0000 7f5aafe00700 1 --2- 172.21.15.62:0/1439050388 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5ab8100e80 0x7f5ab8073590 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7f5aa000c960 tx=0x7f5aa000cd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:31:05.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:05.930+0000 7f5aaf400700 1 -- 172.21.15.62:0/1439050388 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5aa0007a70 con 0x7f5ab8100e80 2024-04-20T22:31:05.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:05.930+0000 7f5aaf400700 1 -- 172.21.15.62:0/1439050388 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5aa0007bd0 con 0x7f5ab8100e80 2024-04-20T22:31:05.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:05.930+0000 7f5aaf400700 1 -- 172.21.15.62:0/1439050388 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5aa00186c0 con 0x7f5ab8100e80 2024-04-20T22:31:05.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:05.930+0000 7f5abda00700 1 -- 172.21.15.62:0/1439050388 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ab81a20c0 con 0x7f5ab8100e80 2024-04-20T22:31:05.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:05.930+0000 7f5abda00700 1 -- 172.21.15.62:0/1439050388 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ab81a2420 con 0x7f5ab8100e80 2024-04-20T22:31:05.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:05.931+0000 7f5abda00700 1 -- 172.21.15.62:0/1439050388 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ab81087f0 con 0x7f5ab8100e80 2024-04-20T22:31:05.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:05.933+0000 7f5aaf400700 1 -- 172.21.15.62:0/1439050388 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f5aa000faa0 con 0x7f5ab8100e80 2024-04-20T22:31:05.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:05.934+0000 7f5aaf400700 1 --2- 172.21.15.62:0/1439050388 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5a9c06ecc0 0x7f5a9c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:05.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:05.934+0000 7f5aaf400700 1 -- 172.21.15.62:0/1439050388 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f5aa008ef10 con 0x7f5ab8100e80 2024-04-20T22:31:05.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:05.934+0000 7f5ab6a00700 1 --2- 172.21.15.62:0/1439050388 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5a9c06ecc0 0x7f5a9c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:05.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:05.935+0000 7f5ab6a00700 1 --2- 172.21.15.62:0/1439050388 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5a9c06ecc0 0x7f5a9c071170 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f5aa800b5c0 tx=0x7f5aa8005fd0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:31:05.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:05.939+0000 7f5aaf400700 1 -- 172.21.15.62:0/1439050388 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f5aa005b750 con 0x7f5ab8100e80 2024-04-20T22:31:05.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:31:05 smithi189 ceph-mon[57022]: pgmap v1678: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:31:06.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:06.274+0000 7f5abda00700 1 -- 172.21.15.62:0/1439050388 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f5ab804f2a0 con 0x7f5ab8100e80 2024-04-20T22:31:06.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:06.275+0000 7f5aaf400700 1 -- 172.21.15.62:0/1439050388 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f5aa005b2e0 con 0x7f5ab8100e80 2024-04-20T22:31:06.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:06.278+0000 7f5abda00700 1 -- 172.21.15.62:0/1439050388 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5a9c06ecc0 msgr2=0x7f5a9c071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:06.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:06.278+0000 7f5abda00700 1 --2- 172.21.15.62:0/1439050388 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5a9c06ecc0 0x7f5a9c071170 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f5aa800b5c0 tx=0x7f5aa8005fd0 comp rx=0 tx=0).stop 2024-04-20T22:31:06.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:06.278+0000 7f5abda00700 1 -- 172.21.15.62:0/1439050388 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5ab8100e80 msgr2=0x7f5ab8073590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:06.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:06.278+0000 7f5abda00700 1 --2- 172.21.15.62:0/1439050388 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5ab8100e80 0x7f5ab8073590 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7f5aa000c960 tx=0x7f5aa000cd20 comp rx=0 tx=0).stop 2024-04-20T22:31:06.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:06.278+0000 7f5abda00700 1 -- 172.21.15.62:0/1439050388 shutdown_connections 2024-04-20T22:31:06.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:06.278+0000 7f5abda00700 1 --2- 172.21.15.62:0/1439050388 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f5a9c06ecc0 0x7f5a9c071170 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:06.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:06.278+0000 7f5abda00700 1 --2- 172.21.15.62:0/1439050388 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ab81004e0 0x7f5ab8073050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:06.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:06.278+0000 7f5abda00700 1 --2- 172.21.15.62:0/1439050388 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5ab8100e80 0x7f5ab8073590 unknown :-1 s=CLOSED pgs=1397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:06.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:06.279+0000 7f5abda00700 1 -- 172.21.15.62:0/1439050388 >> 172.21.15.62:0/1439050388 conn(0x7f5ab80fbda0 msgr2=0x7f5ab80fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:31:06.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:06.279+0000 7f5abda00700 1 -- 172.21.15.62:0/1439050388 shutdown_connections 2024-04-20T22:31:06.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:06.279+0000 7f5abda00700 1 -- 172.21.15.62:0/1439050388 wait complete. 2024-04-20T22:31:06.297 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-3133-ga8c84f59 (a8c84f5997950fd7d008204839253291d52e6eda) squid (dev)" 2024-04-20T22:31:06.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:31:06 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/1439050388' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:31:06.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:31:06 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/1439050388' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-20T22:31:07.084 DEBUG:teuthology.parallel:result is None 2024-04-20T22:31:07.084 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-20T22:31:07.096 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi062.front.sepia.ceph.com 2024-04-20T22:31:07.096 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 -- bash -c 'ceph fs dump' 2024-04-20T22:31:07.531 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/mon.smithi062/config 2024-04-20T22:31:07.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:31:07 smithi062 ceph-mon[76322]: pgmap v1679: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:31:07.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:31:07 smithi189 ceph-mon[57022]: pgmap v1679: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:31:09.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:09.717+0000 7f86cf800700 1 -- 172.21.15.62:0/3467243541 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f86c810c8f0 msgr2=0x7f86c810ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:09.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:09.717+0000 7f86cf800700 1 --2- 172.21.15.62:0/3467243541 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f86c810c8f0 0x7f86c810ccc0 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7f86b8009910 tx=0x7f86b8009c20 comp rx=0 tx=0).stop 2024-04-20T22:31:09.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:09.717+0000 7f86cf800700 1 -- 172.21.15.62:0/3467243541 shutdown_connections 2024-04-20T22:31:09.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:09.717+0000 7f86cf800700 1 --2- 172.21.15.62:0/3467243541 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f86c8106b80 0x7f86c8106ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:09.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:09.717+0000 7f86cf800700 1 --2- 172.21.15.62:0/3467243541 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f86c810c8f0 0x7f86c810ccc0 unknown :-1 s=CLOSED pgs=1398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:09.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:09.717+0000 7f86cf800700 1 -- 172.21.15.62:0/3467243541 >> 172.21.15.62:0/3467243541 conn(0x7f86c8076020 msgr2=0x7f86c8078430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:31:09.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:09.717+0000 7f86cf800700 1 -- 172.21.15.62:0/3467243541 shutdown_connections 2024-04-20T22:31:09.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:09.717+0000 7f86cf800700 1 -- 172.21.15.62:0/3467243541 wait complete. 2024-04-20T22:31:09.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:09.718+0000 7f86cf800700 1 Processor -- start 2024-04-20T22:31:09.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:09.719+0000 7f86cf800700 1 -- start start 2024-04-20T22:31:09.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:09.719+0000 7f86cf800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f86c8106b80 0x7f86c819c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:09.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:09.719+0000 7f86cf800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f86c810c8f0 0x7f86c819c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:09.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:09.719+0000 7f86cf800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86c819cff0 con 0x7f86c810c8f0 2024-04-20T22:31:09.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:09.719+0000 7f86cf800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86c81a0d80 con 0x7f86c8106b80 2024-04-20T22:31:09.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:09.719+0000 7f86bfe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f86c810c8f0 0x7f86c819c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:09.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:09.720+0000 7f86bfe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f86c810c8f0 0x7f86c819c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49160/0 (socket says 172.21.15.62:49160) 2024-04-20T22:31:09.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:09.720+0000 7f86bfe00700 1 -- 172.21.15.62:0/3991504318 learned_addr learned my addr 172.21.15.62:0/3991504318 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:31:09.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:09.720+0000 7f86cce00700 1 --2- 172.21.15.62:0/3991504318 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f86c8106b80 0x7f86c819c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:09.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:09.720+0000 7f86bfe00700 1 -- 172.21.15.62:0/3991504318 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f86c8106b80 msgr2=0x7f86c819c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:09.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:09.720+0000 7f86bfe00700 1 --2- 172.21.15.62:0/3991504318 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f86c8106b80 0x7f86c819c3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:09.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:09.720+0000 7f86bfe00700 1 -- 172.21.15.62:0/3991504318 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86b80095f0 con 0x7f86c810c8f0 2024-04-20T22:31:09.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:09.720+0000 7f86bfe00700 1 --2- 172.21.15.62:0/3991504318 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f86c810c8f0 0x7f86c819c910 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7f86b000d8d0 tx=0x7f86b000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:31:09.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:09.721+0000 7f86c6000700 1 -- 172.21.15.62:0/3991504318 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f86b0009940 con 0x7f86c810c8f0 2024-04-20T22:31:09.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:09.721+0000 7f86c6000700 1 -- 172.21.15.62:0/3991504318 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f86b0010460 con 0x7f86c810c8f0 2024-04-20T22:31:09.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:09.721+0000 7f86c6000700 1 -- 172.21.15.62:0/3991504318 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f86b000f5f0 con 0x7f86c810c8f0 2024-04-20T22:31:09.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:09.721+0000 7f86cf800700 1 -- 172.21.15.62:0/3991504318 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86c81a1060 con 0x7f86c810c8f0 2024-04-20T22:31:09.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:09.721+0000 7f86cf800700 1 -- 172.21.15.62:0/3991504318 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86c81a1560 con 0x7f86c810c8f0 2024-04-20T22:31:09.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:09.722+0000 7f86cf800700 1 -- 172.21.15.62:0/3991504318 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86c804f2a0 con 0x7f86c810c8f0 2024-04-20T22:31:09.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:09.723+0000 7f86c6000700 1 -- 172.21.15.62:0/3991504318 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f86b00105d0 con 0x7f86c810c8f0 2024-04-20T22:31:09.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:09.724+0000 7f86c6000700 1 --2- 172.21.15.62:0/3991504318 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f86ac06ecc0 0x7f86ac071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:09.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:09.724+0000 7f86c6000700 1 -- 172.21.15.62:0/3991504318 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f86b008ef20 con 0x7f86c810c8f0 2024-04-20T22:31:09.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:09.725+0000 7f86cce00700 1 --2- 172.21.15.62:0/3991504318 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f86ac06ecc0 0x7f86ac071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:09.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:09.725+0000 7f86cce00700 1 --2- 172.21.15.62:0/3991504318 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f86ac06ecc0 0x7f86ac071170 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f86b8006010 tx=0x7f86b801a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:31:09.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:09.729+0000 7f86c6000700 1 -- 172.21.15.62:0/3991504318 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f86b0057c30 con 0x7f86c810c8f0 2024-04-20T22:31:09.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:31:09 smithi062 ceph-mon[76322]: pgmap v1680: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:31:09.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:31:09 smithi189 ceph-mon[57022]: pgmap v1680: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:31:10.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:10.017+0000 7f86cf800700 1 -- 172.21.15.62:0/3991504318 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f86c81a1a20 con 0x7f86c810c8f0 2024-04-20T22:31:10.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:10.018+0000 7f86c6000700 1 -- 172.21.15.62:0/3991504318 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 39 v39) v1 ==== 76+0+1981 (secure 0 0 0) 0x7f86b005b2f0 con 0x7f86c810c8f0 2024-04-20T22:31:10.022 INFO:teuthology.orchestra.run.smithi062.stdout:e39 2024-04-20T22:31:10.022 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T22:31:10.022 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-20T22:31:10.022 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-04-20T22:31:10.022 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:31:10.022 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-04-20T22:31:10.022 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-04-20T22:31:10.022 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 39 2024-04-20T22:31:10.022 INFO:teuthology.orchestra.run.smithi062.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-20T22:31:10.023 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-20T21:27:21.032337+0000 2024-04-20T22:31:10.023 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-04-20T22:28:58.231260+0000 2024-04-20T22:31:10.023 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-04-20T22:31:10.023 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-04-20T22:31:10.023 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 60 2024-04-20T22:31:10.023 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-04-20T22:31:10.023 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-04-20T22:31:10.023 INFO:teuthology.orchestra.run.smithi062.stdout:max_xattr_size 65536 2024-04-20T22:31:10.023 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-04-20T22:31:10.023 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-04-20T22:31:10.023 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 81 2024-04-20T22:31:10.023 INFO:teuthology.orchestra.run.smithi062.stdout:snap_rstat 1 2024-04-20T22:31:10.023 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-20T22:31:10.023 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 2 2024-04-20T22:31:10.023 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1 2024-04-20T22:31:10.023 INFO:teuthology.orchestra.run.smithi062.stdout:up {0=34444,1=34476} 2024-04-20T22:31:10.024 INFO:teuthology.orchestra.run.smithi062.stdout:failed 2024-04-20T22:31:10.024 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-04-20T22:31:10.024 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-04-20T22:31:10.024 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3] 2024-04-20T22:31:10.024 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-04-20T22:31:10.024 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data enabled 2024-04-20T22:31:10.024 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-04-20T22:31:10.024 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-04-20T22:31:10.024 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 1 2024-04-20T22:31:10.024 INFO:teuthology.orchestra.run.smithi062.stdout:qdb_cluster leader: 34444 members: 34444,34476 2024-04-20T22:31:10.024 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ryqblb{0:34444} state up:active seq 699 join_fscid=1 addr [v2:172.21.15.62:6826/4113916205,v1:172.21.15.62:6827/4113916205] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:31:10.024 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.dvimer{0:44727} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.189:6824/3576699232,v1:172.21.15.189:6825/3576699232] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:31:10.024 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi062.ozizwi{1:34476} state up:active seq 691 join_fscid=1 addr [v2:172.21.15.62:6828/3801975165,v1:172.21.15.62:6829/3801975165] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:31:10.024 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.cephfs.smithi189.xbgspw{1:45075} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.189:6826/212905792,v1:172.21.15.189:6827/212905792] compat {c=[1],r=[1],i=[1fff]}] 2024-04-20T22:31:10.024 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:31:10.025 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:31:10.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:10.021+0000 7f86cf800700 1 -- 172.21.15.62:0/3991504318 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f86ac06ecc0 msgr2=0x7f86ac071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:10.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:10.021+0000 7f86cf800700 1 --2- 172.21.15.62:0/3991504318 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f86ac06ecc0 0x7f86ac071170 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f86b8006010 tx=0x7f86b801a040 comp rx=0 tx=0).stop 2024-04-20T22:31:10.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:10.021+0000 7f86cf800700 1 -- 172.21.15.62:0/3991504318 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f86c810c8f0 msgr2=0x7f86c819c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:10.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:10.021+0000 7f86cf800700 1 --2- 172.21.15.62:0/3991504318 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f86c810c8f0 0x7f86c819c910 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7f86b000d8d0 tx=0x7f86b000dc90 comp rx=0 tx=0).stop 2024-04-20T22:31:10.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:10.022+0000 7f86cf800700 1 -- 172.21.15.62:0/3991504318 shutdown_connections 2024-04-20T22:31:10.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:10.022+0000 7f86cf800700 1 --2- 172.21.15.62:0/3991504318 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f86ac06ecc0 0x7f86ac071170 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:10.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:10.022+0000 7f86cf800700 1 --2- 172.21.15.62:0/3991504318 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f86c8106b80 0x7f86c819c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:10.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:10.022+0000 7f86cf800700 1 --2- 172.21.15.62:0/3991504318 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f86c810c8f0 0x7f86c819c910 unknown :-1 s=CLOSED pgs=1399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:10.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:10.022+0000 7f86cf800700 1 -- 172.21.15.62:0/3991504318 >> 172.21.15.62:0/3991504318 conn(0x7f86c8076020 msgr2=0x7f86c8077940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:31:10.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:10.022+0000 7f86cf800700 1 -- 172.21.15.62:0/3991504318 shutdown_connections 2024-04-20T22:31:10.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:10.022+0000 7f86cf800700 1 -- 172.21.15.62:0/3991504318 wait complete. 2024-04-20T22:31:10.026 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 39 2024-04-20T22:31:10.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:31:10 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/3991504318' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:31:10.843 INFO:teuthology.run_tasks:Running task fs.post_upgrade_checks... 2024-04-20T22:31:10.855 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-20T22:31:10.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:31:10 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/3991504318' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-20T22:31:11.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:11.082+0000 7f9bae600640 1 -- 172.21.15.62:0/3058428301 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ba8104190 msgr2=0x7f9ba80fdda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:11.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:11.083+0000 7f9bae600640 1 --2- 172.21.15.62:0/3058428301 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ba8104190 0x7f9ba80fdda0 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f9b9000a990 tx=0x7f9b9002d020 comp rx=0 tx=0).stop 2024-04-20T22:31:11.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:11.083+0000 7f9bae600640 1 -- 172.21.15.62:0/3058428301 shutdown_connections 2024-04-20T22:31:11.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:11.083+0000 7f9bae600640 1 --2- 172.21.15.62:0/3058428301 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ba8104190 0x7f9ba80fdda0 unknown :-1 s=CLOSED pgs=1400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:11.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:11.083+0000 7f9bae600640 1 --2- 172.21.15.62:0/3058428301 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ba8066630 0x7f9ba80fd860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:11.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:11.083+0000 7f9bae600640 1 -- 172.21.15.62:0/3058428301 >> 172.21.15.62:0/3058428301 conn(0x7f9ba80f9320 msgr2=0x7f9ba80fb730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:31:11.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:11.083+0000 7f9bae600640 1 -- 172.21.15.62:0/3058428301 shutdown_connections 2024-04-20T22:31:11.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:11.083+0000 7f9bae600640 1 -- 172.21.15.62:0/3058428301 wait complete. 2024-04-20T22:31:11.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:11.084+0000 7f9bae600640 1 Processor -- start 2024-04-20T22:31:11.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:11.084+0000 7f9bae600640 1 -- start start 2024-04-20T22:31:11.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:11.084+0000 7f9bae600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ba8066630 0x7f9ba8197460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:11.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:11.084+0000 7f9bae600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ba8104190 0x7f9ba81979a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:11.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:11.084+0000 7f9bae600640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ba8198030 con 0x7f9ba8066630 2024-04-20T22:31:11.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:11.084+0000 7f9bae600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ba819be00 con 0x7f9ba8104190 2024-04-20T22:31:11.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:11.085+0000 7f9ba7400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ba8066630 0x7f9ba8197460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:11.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:11.085+0000 7f9ba7400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ba8066630 0x7f9ba8197460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49186/0 (socket says 172.21.15.62:49186) 2024-04-20T22:31:11.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:11.085+0000 7f9ba7400640 1 -- 172.21.15.62:0/1021889814 learned_addr learned my addr 172.21.15.62:0/1021889814 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:31:11.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:11.085+0000 7f9ba7400640 1 -- 172.21.15.62:0/1021889814 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ba8104190 msgr2=0x7f9ba81979a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:11.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:11.085+0000 7f9ba6a00640 1 --2- 172.21.15.62:0/1021889814 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ba8104190 0x7f9ba81979a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:11.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:11.085+0000 7f9ba7400640 1 --2- 172.21.15.62:0/1021889814 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ba8104190 0x7f9ba81979a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:11.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:11.085+0000 7f9ba7400640 1 -- 172.21.15.62:0/1021889814 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b9000a670 con 0x7f9ba8066630 2024-04-20T22:31:11.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:11.085+0000 7f9ba6a00640 1 --2- 172.21.15.62:0/1021889814 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ba8104190 0x7f9ba81979a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T22:31:11.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:11.086+0000 7f9ba7400640 1 --2- 172.21.15.62:0/1021889814 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ba8066630 0x7f9ba8197460 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f9b9800d8d0 tx=0x7f9b9800dd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:31:11.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:11.086+0000 7f9b9fe00640 1 -- 172.21.15.62:0/1021889814 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9b98009a30 con 0x7f9ba8066630 2024-04-20T22:31:11.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:11.086+0000 7f9b9fe00640 1 -- 172.21.15.62:0/1021889814 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9b98009b90 con 0x7f9ba8066630 2024-04-20T22:31:11.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:11.086+0000 7f9bae600640 1 -- 172.21.15.62:0/1021889814 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ba819c0e0 con 0x7f9ba8066630 2024-04-20T22:31:11.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:11.086+0000 7f9b9fe00640 1 -- 172.21.15.62:0/1021889814 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9b9800f680 con 0x7f9ba8066630 2024-04-20T22:31:11.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:11.086+0000 7f9bae600640 1 -- 172.21.15.62:0/1021889814 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ba819c640 con 0x7f9ba8066630 2024-04-20T22:31:11.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:11.087+0000 7f9bae600640 1 -- 172.21.15.62:0/1021889814 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ba804e000 con 0x7f9ba8066630 2024-04-20T22:31:11.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:11.088+0000 7f9b9fe00640 1 -- 172.21.15.62:0/1021889814 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f9b98009d00 con 0x7f9ba8066630 2024-04-20T22:31:11.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:11.089+0000 7f9b9fe00640 1 --2- 172.21.15.62:0/1021889814 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9b8c06ecf0 0x7f9b8c0711a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:11.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:11.090+0000 7f9b9fe00640 1 -- 172.21.15.62:0/1021889814 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f9b9808eca0 con 0x7f9ba8066630 2024-04-20T22:31:11.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:11.090+0000 7f9ba6a00640 1 --2- 172.21.15.62:0/1021889814 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9b8c06ecf0 0x7f9b8c0711a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:11.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:11.090+0000 7f9ba6a00640 1 --2- 172.21.15.62:0/1021889814 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9b8c06ecf0 0x7f9b8c0711a0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f9b900057c0 tx=0x7f9b90035600 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:31:11.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:11.095+0000 7f9b9fe00640 1 -- 172.21.15.62:0/1021889814 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f9b98057a30 con 0x7f9ba8066630 2024-04-20T22:31:11.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:11.305+0000 7f9bae600640 1 -- 172.21.15.62:0/1021889814 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9ba810a7f0 con 0x7f9ba8066630 2024-04-20T22:31:11.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:11.308+0000 7f9b9fe00640 1 -- 172.21.15.62:0/1021889814 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+16652 (secure 0 0 0) 0x7f9b98017090 con 0x7f9ba8066630 2024-04-20T22:31:11.309 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:31:11.309 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":85,"fsid":"61ae3628-ff5b-11ee-bc93-c7b262605968","created":"2024-04-20T21:18:19.951305+0000","modified":"2024-04-20T22:13:07.093942+0000","last_up_change":"2024-04-20T21:41:54.404166+0000","last_in_change":"2024-04-20T21:25:19.201764+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":20,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-20T21:24:02.500375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-04-20T21:27:19.577559+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-04-20T21:27:20.560814+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a3f8216-c641-4338-813d-d836efc3f4e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":48,"up_from":52,"up_thru":77,"down_at":49,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":1209367302},{"type":"v1","addr":"172.21.15.62:6803","nonce":1209367302}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":1209367302},{"type":"v1","addr":"172.21.15.62:6805","nonce":1209367302}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":1209367302},{"type":"v1","addr":"172.21.15.62:6809","nonce":1209367302}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":1209367302},{"type":"v1","addr":"172.21.15.62:6807","nonce":1209367302}]},"public_addr":"172.21.15.62:6803/1209367302","cluster_addr":"172.21.15.62:6805/1209367302","heartbeat_back_addr":"172.21.15.62:6809/1209367302","heartbeat_front_addr":"172.21.15.62:6807/1209367302","state":["exists","up"]},{"osd":1,"uuid":"f0c451a4-9002-4ddc-be0e-d8ddebd4418d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":14,"last_clean_end":53,"up_from":57,"up_thru":77,"down_at":54,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":2879835300},{"type":"v1","addr":"172.21.15.62:6811","nonce":2879835300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":2879835300},{"type":"v1","addr":"172.21.15.62:6813","nonce":2879835300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":2879835300},{"type":"v1","addr":"172.21.15.62:6817","nonce":2879835300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":2879835300},{"type":"v1","addr":"172.21.15.62:6815","nonce":2879835300}]},"public_addr":"172.21.15.62:6811/2879835300","cluster_addr":"172.21.15.62:6813/2879835300","heartbeat_back_addr":"172.21.15.62:6817/2879835300","heartbeat_front_addr":"172.21.15.62:6815/2879835300","state":["exists","up"]},{"osd":2,"uuid":"e1a8769d-5da2-49c8-942a-83a94c7ac761","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":19,"last_clean_end":58,"up_from":62,"up_thru":77,"down_at":59,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":4264123649},{"type":"v1","addr":"172.21.15.62:6819","nonce":4264123649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":4264123649},{"type":"v1","addr":"172.21.15.62:6821","nonce":4264123649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":4264123649},{"type":"v1","addr":"172.21.15.62:6825","nonce":4264123649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6822","nonce":4264123649},{"type":"v1","addr":"172.21.15.62:6823","nonce":4264123649}]},"public_addr":"172.21.15.62:6819/4264123649","cluster_addr":"172.21.15.62:6821/4264123649","heartbeat_back_addr":"172.21.15.62:6825/4264123649","heartbeat_front_addr":"172.21.15.62:6823/4264123649","state":["exists","up"]},{"osd":3,"uuid":"eaa2fe2b-87f8-4d01-8ddf-e72f244f31b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":63,"up_from":67,"up_thru":77,"down_at":64,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":3977023570},{"type":"v1","addr":"172.21.15.189:6801","nonce":3977023570}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":3977023570},{"type":"v1","addr":"172.21.15.189:6803","nonce":3977023570}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":3977023570},{"type":"v1","addr":"172.21.15.189:6807","nonce":3977023570}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":3977023570},{"type":"v1","addr":"172.21.15.189:6805","nonce":3977023570}]},"public_addr":"172.21.15.189:6801/3977023570","cluster_addr":"172.21.15.189:6803/3977023570","heartbeat_back_addr":"172.21.15.189:6807/3977023570","heartbeat_front_addr":"172.21.15.189:6805/3977023570","state":["exists","up"]},{"osd":4,"uuid":"72be8597-52e5-45fd-b4f2-06ac50df77e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":34,"last_clean_end":68,"up_from":72,"up_thru":77,"down_at":69,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":2794032930},{"type":"v1","addr":"172.21.15.189:6809","nonce":2794032930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":2794032930},{"type":"v1","addr":"172.21.15.189:6811","nonce":2794032930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":2794032930},{"type":"v1","addr":"172.21.15.189:6815","nonce":2794032930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":2794032930},{"type":"v1","addr":"172.21.15.189:6813","nonce":2794032930}]},"public_addr":"172.21.15.189:6809/2794032930","cluster_addr":"172.21.15.189:6811/2794032930","heartbeat_back_addr":"172.21.15.189:6815/2794032930","heartbeat_front_addr":"172.21.15.189:6813/2794032930","state":["exists","up"]},{"osd":5,"uuid":"c650c63b-41eb-4d4f-8e14-907c7b3614c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":39,"last_clean_end":73,"up_from":77,"up_thru":77,"down_at":74,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2760890917},{"type":"v1","addr":"172.21.15.189:6817","nonce":2760890917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2760890917},{"type":"v1","addr":"172.21.15.189:6819","nonce":2760890917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2760890917},{"type":"v1","addr":"172.21.15.189:6823","nonce":2760890917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2760890917},{"type":"v1","addr":"172.21.15.189:6821","nonce":2760890917}]},"public_addr":"172.21.15.189:6817/2760890917","cluster_addr":"172.21.15.189:6819/2760890917","heartbeat_back_addr":"172.21.15.189:6823/2760890917","heartbeat_front_addr":"172.21.15.189:6821/2760890917","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T21:22:45.605185+0000","dead_epoch":48},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T21:23:21.062873+0000","dead_epoch":53},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T21:24:00.502044+0000","dead_epoch":58},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T21:24:32.944093+0000","dead_epoch":63},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T21:25:06.278988+0000","dead_epoch":68},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T21:25:41.872982+0000","dead_epoch":73}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.189:6825/237262514":"2024-04-21T21:42:01.558630+0000","172.21.15.62:0/2348636520":"2024-04-21T21:30:35.075229+0000","172.21.15.62:6801/871423274":"2024-04-21T21:18:52.990685+0000","172.21.15.62:6801/831506538":"2024-04-21T21:21:03.778739+0000","172.21.15.62:0/4251078301":"2024-04-21T21:36:34.221905+0000","172.21.15.189:0/1714270321":"2024-04-21T21:31:36.545789+0000","172.21.15.62:6826/1147127123":"2024-04-21T21:42:01.558599+0000","172.21.15.62:0/868116748":"2024-04-21T21:30:35.075229+0000","172.21.15.62:0/2525292829":"2024-04-21T21:18:52.990685+0000","172.21.15.62:0/1116048562":"2024-04-21T21:21:03.778739+0000","172.21.15.62:0/634566145":"2024-04-21T21:18:52.990685+0000","172.21.15.62:6800/831506538":"2024-04-21T21:21:03.778739+0000","172.21.15.189:0/4026364416":"2024-04-21T21:31:36.545789+0000","172.21.15.62:0/1789120375":"2024-04-21T21:36:34.221905+0000","172.21.15.62:0/3416563026":"2024-04-21T21:18:52.990685+0000","172.21.15.62:0/3673769314":"2024-04-21T21:30:35.075229+0000","172.21.15.62:0/717708557":"2024-04-21T21:30:35.075229+0000","172.21.15.62:0/4274508569":"2024-04-21T21:19:34.965981+0000","172.21.15.189:0/3430221175":"2024-04-21T21:31:36.545789+0000","172.21.15.189:6824/237262514":"2024-04-21T21:42:01.558630+0000","172.21.15.189:0/100133912":"2024-04-21T21:31:36.545789+0000","172.21.15.189:0/441007232":"2024-04-21T21:31:36.545789+0000","172.21.15.62:6800/4116451241":"2024-04-21T21:30:35.075229+0000","172.21.15.62:6827/1147127123":"2024-04-21T21:42:01.558599+0000","172.21.15.62:6801/4116451241":"2024-04-21T21:30:35.075229+0000","172.21.15.62:6800/871423274":"2024-04-21T21:18:52.990685+0000","172.21.15.189:6828/2010473407":"2024-04-21T21:31:36.545789+0000","172.21.15.189:6829/2010473407":"2024-04-21T21:31:36.545789+0000","172.21.15.189:0/1726963148":"2024-04-21T21:31:36.545789+0000","172.21.15.62:0/3192460755":"2024-04-21T21:21:03.778739+0000","172.21.15.62:6801/4147738586":"2024-04-21T21:19:34.965981+0000","172.21.15.62:0/4158176989":"2024-04-21T21:19:34.965981+0000","172.21.15.62:0/301716985":"2024-04-21T21:21:03.778739+0000","172.21.15.62:0/678960466":"2024-04-21T21:36:34.221905+0000","172.21.15.62:0/2136121731":"2024-04-21T21:36:34.221905+0000","172.21.15.62:6800/4147738586":"2024-04-21T21:19:34.965981+0000","172.21.15.62:6800/1585760929":"2024-04-21T21:36:34.221905+0000","172.21.15.62:0/771955286":"2024-04-21T21:19:34.965981+0000","172.21.15.62:6801/1585760929":"2024-04-21T21:36:34.221905+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-20T22:31:11.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:11.311+0000 7f9bae600640 1 -- 172.21.15.62:0/1021889814 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9b8c06ecf0 msgr2=0x7f9b8c0711a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:11.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:11.312+0000 7f9bae600640 1 --2- 172.21.15.62:0/1021889814 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9b8c06ecf0 0x7f9b8c0711a0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f9b900057c0 tx=0x7f9b90035600 comp rx=0 tx=0).stop 2024-04-20T22:31:11.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:11.312+0000 7f9bae600640 1 -- 172.21.15.62:0/1021889814 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ba8066630 msgr2=0x7f9ba8197460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:11.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:11.312+0000 7f9bae600640 1 --2- 172.21.15.62:0/1021889814 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ba8066630 0x7f9ba8197460 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f9b9800d8d0 tx=0x7f9b9800dd90 comp rx=0 tx=0).stop 2024-04-20T22:31:11.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:11.312+0000 7f9bae600640 1 -- 172.21.15.62:0/1021889814 shutdown_connections 2024-04-20T22:31:11.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:11.312+0000 7f9bae600640 1 --2- 172.21.15.62:0/1021889814 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f9b8c06ecf0 0x7f9b8c0711a0 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:11.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:11.312+0000 7f9bae600640 1 --2- 172.21.15.62:0/1021889814 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ba8104190 0x7f9ba81979a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:11.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:11.312+0000 7f9bae600640 1 --2- 172.21.15.62:0/1021889814 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ba8066630 0x7f9ba8197460 unknown :-1 s=CLOSED pgs=1401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:11.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:11.312+0000 7f9bae600640 1 -- 172.21.15.62:0/1021889814 >> 172.21.15.62:0/1021889814 conn(0x7f9ba80f9320 msgr2=0x7f9ba80fab70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:31:11.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:11.312+0000 7f9bae600640 1 -- 172.21.15.62:0/1021889814 shutdown_connections 2024-04-20T22:31:11.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:11.312+0000 7f9bae600640 1 -- 172.21.15.62:0/1021889814 wait complete. 2024-04-20T22:31:11.331 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-20T21:24:02.500375+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-04-20T21:27:19.577559+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 1, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-04-20T21:27:20.560814+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-20T22:31:11.332 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-20T22:31:11.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:11.536+0000 7fa0abe00640 1 -- 172.21.15.62:0/510207498 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0ac105480 msgr2=0x7fa0ac105850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:11.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:11.536+0000 7fa0abe00640 1 --2- 172.21.15.62:0/510207498 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0ac105480 0x7fa0ac105850 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7fa09c009880 tx=0x7fa09c02bce0 comp rx=0 tx=0).stop 2024-04-20T22:31:11.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:11.537+0000 7fa0abe00640 1 -- 172.21.15.62:0/510207498 shutdown_connections 2024-04-20T22:31:11.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:11.537+0000 7fa0abe00640 1 --2- 172.21.15.62:0/510207498 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0ac0ff470 0x7fa0ac0ff8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:11.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:11.537+0000 7fa0abe00640 1 --2- 172.21.15.62:0/510207498 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0ac105480 0x7fa0ac105850 unknown :-1 s=CLOSED pgs=1402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:11.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:11.537+0000 7fa0abe00640 1 -- 172.21.15.62:0/510207498 >> 172.21.15.62:0/510207498 conn(0x7fa0ac0faf70 msgr2=0x7fa0ac0fd380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:31:11.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:11.537+0000 7fa0abe00640 1 -- 172.21.15.62:0/510207498 shutdown_connections 2024-04-20T22:31:11.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:11.537+0000 7fa0abe00640 1 -- 172.21.15.62:0/510207498 wait complete. 2024-04-20T22:31:11.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:11.538+0000 7fa0abe00640 1 Processor -- start 2024-04-20T22:31:11.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:11.538+0000 7fa0abe00640 1 -- start start 2024-04-20T22:31:11.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:11.538+0000 7fa0abe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0ac0ff470 0x7fa0ac197450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:11.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:11.538+0000 7fa0abe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0ac105480 0x7fa0ac197990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:11.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:11.538+0000 7fa0abe00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0ac198020 con 0x7fa0ac105480 2024-04-20T22:31:11.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:11.538+0000 7fa0abe00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0ac19bdf0 con 0x7fa0ac0ff470 2024-04-20T22:31:11.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:11.539+0000 7fa0aa000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0ac105480 0x7fa0ac197990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:11.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:11.539+0000 7fa0aa000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0ac105480 0x7fa0ac197990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37154/0 (socket says 172.21.15.62:37154) 2024-04-20T22:31:11.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:11.539+0000 7fa0aa000640 1 -- 172.21.15.62:0/240870089 learned_addr learned my addr 172.21.15.62:0/240870089 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:31:11.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:11.539+0000 7fa0aaa00640 1 --2- 172.21.15.62:0/240870089 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0ac0ff470 0x7fa0ac197450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:11.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:11.539+0000 7fa0aa000640 1 -- 172.21.15.62:0/240870089 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0ac0ff470 msgr2=0x7fa0ac197450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:11.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:11.539+0000 7fa0aa000640 1 --2- 172.21.15.62:0/240870089 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0ac0ff470 0x7fa0ac197450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:11.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:11.539+0000 7fa0aa000640 1 -- 172.21.15.62:0/240870089 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa09c009560 con 0x7fa0ac105480 2024-04-20T22:31:11.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:11.540+0000 7fa0aa000640 1 --2- 172.21.15.62:0/240870089 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0ac105480 0x7fa0ac197990 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7fa09800d6f0 tx=0x7fa09800dbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:31:11.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:11.540+0000 7fa0a3400640 1 -- 172.21.15.62:0/240870089 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa098009a30 con 0x7fa0ac105480 2024-04-20T22:31:11.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:11.540+0000 7fa0a3400640 1 -- 172.21.15.62:0/240870089 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa0980054e0 con 0x7fa0ac105480 2024-04-20T22:31:11.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:11.540+0000 7fa0a3400640 1 -- 172.21.15.62:0/240870089 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa098010460 con 0x7fa0ac105480 2024-04-20T22:31:11.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:11.540+0000 7fa0abe00640 1 -- 172.21.15.62:0/240870089 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0ac19c0d0 con 0x7fa0ac105480 2024-04-20T22:31:11.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:11.540+0000 7fa0abe00640 1 -- 172.21.15.62:0/240870089 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0ac19c630 con 0x7fa0ac105480 2024-04-20T22:31:11.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:11.541+0000 7fa0abe00640 1 -- 172.21.15.62:0/240870089 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0ac04e000 con 0x7fa0ac105480 2024-04-20T22:31:11.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:11.543+0000 7fa0a3400640 1 -- 172.21.15.62:0/240870089 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fa098009b90 con 0x7fa0ac105480 2024-04-20T22:31:11.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:11.544+0000 7fa0a3400640 1 --2- 172.21.15.62:0/240870089 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa09006ec30 0x7fa0900710e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:11.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:11.544+0000 7fa0a3400640 1 -- 172.21.15.62:0/240870089 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fa09808eab0 con 0x7fa0ac105480 2024-04-20T22:31:11.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:11.544+0000 7fa0aaa00640 1 --2- 172.21.15.62:0/240870089 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa09006ec30 0x7fa0900710e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:11.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:11.545+0000 7fa0aaa00640 1 --2- 172.21.15.62:0/240870089 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa09006ec30 0x7fa0900710e0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fa09c000c00 tx=0x7fa09c02cf90 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:31:11.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:11.548+0000 7fa0a3400640 1 -- 172.21.15.62:0/240870089 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fa098090e60 con 0x7fa0ac105480 2024-04-20T22:31:11.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:11.761+0000 7fa0abe00640 1 -- 172.21.15.62:0/240870089 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa0ac10c760 con 0x7fa0ac105480 2024-04-20T22:31:11.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:11.762+0000 7fa0a3400640 1 -- 172.21.15.62:0/240870089 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v85) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa098020020 con 0x7fa0ac105480 2024-04-20T22:31:11.762 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 1 2024-04-20T22:31:11.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:11.765+0000 7fa0abe00640 1 -- 172.21.15.62:0/240870089 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa09006ec30 msgr2=0x7fa0900710e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:11.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:11.765+0000 7fa0abe00640 1 --2- 172.21.15.62:0/240870089 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa09006ec30 0x7fa0900710e0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fa09c000c00 tx=0x7fa09c02cf90 comp rx=0 tx=0).stop 2024-04-20T22:31:11.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:11.765+0000 7fa0abe00640 1 -- 172.21.15.62:0/240870089 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0ac105480 msgr2=0x7fa0ac197990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:11.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:11.765+0000 7fa0abe00640 1 --2- 172.21.15.62:0/240870089 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0ac105480 0x7fa0ac197990 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7fa09800d6f0 tx=0x7fa09800dbb0 comp rx=0 tx=0).stop 2024-04-20T22:31:11.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:11.765+0000 7fa0abe00640 1 -- 172.21.15.62:0/240870089 shutdown_connections 2024-04-20T22:31:11.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:11.765+0000 7fa0abe00640 1 --2- 172.21.15.62:0/240870089 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa09006ec30 0x7fa0900710e0 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:11.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:11.765+0000 7fa0abe00640 1 --2- 172.21.15.62:0/240870089 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0ac105480 0x7fa0ac197990 unknown :-1 s=CLOSED pgs=1403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:11.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:11.765+0000 7fa0abe00640 1 --2- 172.21.15.62:0/240870089 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0ac0ff470 0x7fa0ac197450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:11.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:11.765+0000 7fa0abe00640 1 -- 172.21.15.62:0/240870089 >> 172.21.15.62:0/240870089 conn(0x7fa0ac0faf70 msgr2=0x7fa0ac0fb970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:31:11.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:11.765+0000 7fa0abe00640 1 -- 172.21.15.62:0/240870089 shutdown_connections 2024-04-20T22:31:11.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:11.765+0000 7fa0abe00640 1 -- 172.21.15.62:0/240870089 wait complete. 2024-04-20T22:31:11.782 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:31:11 smithi062 ceph-mon[76322]: pgmap v1681: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:31:11.782 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:31:11 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/1021889814' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-20T22:31:11.784 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-04-20T22:31:11.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:31:11 smithi189 ceph-mon[57022]: pgmap v1681: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:31:11.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:31:11 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/1021889814' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-20T22:31:12.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.001+0000 7f0fc7e00640 1 -- 172.21.15.62:0/924054206 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0fc8072f10 msgr2=0x7f0fc810ff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:12.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.001+0000 7f0fc7e00640 1 --2- 172.21.15.62:0/924054206 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0fc8072f10 0x7f0fc810ff80 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f0fb0009880 tx=0x7f0fb002bd80 comp rx=0 tx=0).stop 2024-04-20T22:31:12.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.001+0000 7f0fc7e00640 1 -- 172.21.15.62:0/924054206 shutdown_connections 2024-04-20T22:31:12.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.001+0000 7f0fc7e00640 1 --2- 172.21.15.62:0/924054206 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0fc8072f10 0x7f0fc810ff80 unknown :-1 s=CLOSED pgs=1404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:12.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.001+0000 7f0fc7e00640 1 --2- 172.21.15.62:0/924054206 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0fc8072600 0x7f0fc80729d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:12.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.001+0000 7f0fc7e00640 1 -- 172.21.15.62:0/924054206 >> 172.21.15.62:0/924054206 conn(0x7f0fc80fafb0 msgr2=0x7f0fc80fd3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:31:12.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.001+0000 7f0fc7e00640 1 -- 172.21.15.62:0/924054206 shutdown_connections 2024-04-20T22:31:12.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.001+0000 7f0fc7e00640 1 -- 172.21.15.62:0/924054206 wait complete. 2024-04-20T22:31:12.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.002+0000 7f0fc7e00640 1 Processor -- start 2024-04-20T22:31:12.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.002+0000 7f0fc7e00640 1 -- start start 2024-04-20T22:31:12.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.003+0000 7f0fc7e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0fc8072600 0x7f0fc81a15a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:12.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.003+0000 7f0fc7e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0fc8072f10 0x7f0fc81a1ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:12.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.003+0000 7f0fc7e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fc819b610 con 0x7f0fc8072600 2024-04-20T22:31:12.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.003+0000 7f0fc7e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fc819b780 con 0x7f0fc8072f10 2024-04-20T22:31:12.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.003+0000 7f0fc6a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0fc8072600 0x7f0fc81a15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:12.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.003+0000 7f0fc6a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0fc8072600 0x7f0fc81a15a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37174/0 (socket says 172.21.15.62:37174) 2024-04-20T22:31:12.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.003+0000 7f0fc6a00640 1 -- 172.21.15.62:0/488416000 learned_addr learned my addr 172.21.15.62:0/488416000 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:31:12.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.004+0000 7f0fbe000640 1 --2- 172.21.15.62:0/488416000 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0fc8072f10 0x7f0fc81a1ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:12.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.004+0000 7f0fc6a00640 1 -- 172.21.15.62:0/488416000 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0fc8072f10 msgr2=0x7f0fc81a1ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:12.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.004+0000 7f0fc6a00640 1 --2- 172.21.15.62:0/488416000 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0fc8072f10 0x7f0fc81a1ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:12.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.004+0000 7f0fc6a00640 1 -- 172.21.15.62:0/488416000 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fb0009560 con 0x7f0fc8072600 2024-04-20T22:31:12.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.004+0000 7f0fc6a00640 1 --2- 172.21.15.62:0/488416000 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0fc8072600 0x7f0fc81a15a0 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7f0fb800b420 tx=0x7f0fb800b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:31:12.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.004+0000 7f0fbfe00640 1 -- 172.21.15.62:0/488416000 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0fb8011800 con 0x7f0fc8072600 2024-04-20T22:31:12.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.004+0000 7f0fbfe00640 1 -- 172.21.15.62:0/488416000 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0fb8011e20 con 0x7f0fc8072600 2024-04-20T22:31:12.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.004+0000 7f0fc7e00640 1 -- 172.21.15.62:0/488416000 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fc819ba60 con 0x7f0fc8072600 2024-04-20T22:31:12.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.004+0000 7f0fbfe00640 1 -- 172.21.15.62:0/488416000 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0fb800f2e0 con 0x7f0fc8072600 2024-04-20T22:31:12.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.004+0000 7f0fc7e00640 1 -- 172.21.15.62:0/488416000 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fc819bfc0 con 0x7f0fc8072600 2024-04-20T22:31:12.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.005+0000 7f0fc7e00640 1 -- 172.21.15.62:0/488416000 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fc804e000 con 0x7f0fc8072600 2024-04-20T22:31:12.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.007+0000 7f0fbfe00640 1 -- 172.21.15.62:0/488416000 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f0fb8010380 con 0x7f0fc8072600 2024-04-20T22:31:12.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.008+0000 7f0fbfe00640 1 --2- 172.21.15.62:0/488416000 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f0fac06ec30 0x7f0fac0710e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:12.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.008+0000 7f0fbfe00640 1 -- 172.21.15.62:0/488416000 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f0fb808e790 con 0x7f0fc8072600 2024-04-20T22:31:12.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.008+0000 7f0fbe000640 1 --2- 172.21.15.62:0/488416000 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f0fac06ec30 0x7f0fac0710e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:12.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.009+0000 7f0fbe000640 1 --2- 172.21.15.62:0/488416000 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f0fac06ec30 0x7f0fac0710e0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f0fb0036040 tx=0x7f0fb0000bc0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:31:12.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.013+0000 7f0fbfe00640 1 -- 172.21.15.62:0/488416000 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f0fb8057520 con 0x7f0fc8072600 2024-04-20T22:31:12.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.223+0000 7f0fc7e00640 1 -- 172.21.15.62:0/488416000 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7f0fc810d670 con 0x7f0fc8072600 2024-04-20T22:31:12.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.224+0000 7f0fbfe00640 1 -- 172.21.15.62:0/488416000 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v85) v1 ==== 107+0+11 (secure 0 0 0) 0x7f0fb805abe0 con 0x7f0fc8072600 2024-04-20T22:31:12.224 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 32 2024-04-20T22:31:12.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.227+0000 7f0fc7e00640 1 -- 172.21.15.62:0/488416000 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f0fac06ec30 msgr2=0x7f0fac0710e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:12.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.227+0000 7f0fc7e00640 1 --2- 172.21.15.62:0/488416000 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f0fac06ec30 0x7f0fac0710e0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f0fb0036040 tx=0x7f0fb0000bc0 comp rx=0 tx=0).stop 2024-04-20T22:31:12.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.227+0000 7f0fc7e00640 1 -- 172.21.15.62:0/488416000 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0fc8072600 msgr2=0x7f0fc81a15a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:12.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.227+0000 7f0fc7e00640 1 --2- 172.21.15.62:0/488416000 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0fc8072600 0x7f0fc81a15a0 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7f0fb800b420 tx=0x7f0fb800b8e0 comp rx=0 tx=0).stop 2024-04-20T22:31:12.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.227+0000 7f0fc7e00640 1 -- 172.21.15.62:0/488416000 shutdown_connections 2024-04-20T22:31:12.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.227+0000 7f0fc7e00640 1 --2- 172.21.15.62:0/488416000 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f0fac06ec30 0x7f0fac0710e0 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:12.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.227+0000 7f0fc7e00640 1 --2- 172.21.15.62:0/488416000 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0fc8072f10 0x7f0fc81a1ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:12.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.227+0000 7f0fc7e00640 1 --2- 172.21.15.62:0/488416000 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0fc8072600 0x7f0fc81a15a0 unknown :-1 s=CLOSED pgs=1405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:12.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.227+0000 7f0fc7e00640 1 -- 172.21.15.62:0/488416000 >> 172.21.15.62:0/488416000 conn(0x7f0fc80fafb0 msgr2=0x7f0fc8101a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:31:12.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.227+0000 7f0fc7e00640 1 -- 172.21.15.62:0/488416000 shutdown_connections 2024-04-20T22:31:12.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.227+0000 7f0fc7e00640 1 -- 172.21.15.62:0/488416000 wait complete. 2024-04-20T22:31:12.247 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-04-20T22:31:12.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.452+0000 7f23fdc00640 1 -- 172.21.15.62:0/3675737017 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f23f81055c0 msgr2=0x7f23f8105990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:12.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.452+0000 7f23fdc00640 1 --2- 172.21.15.62:0/3675737017 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f23f81055c0 0x7f23f8105990 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7f23e8009880 tx=0x7f23e802bce0 comp rx=0 tx=0).stop 2024-04-20T22:31:12.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.452+0000 7f23fdc00640 1 -- 172.21.15.62:0/3675737017 shutdown_connections 2024-04-20T22:31:12.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.452+0000 7f23fdc00640 1 --2- 172.21.15.62:0/3675737017 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23f8067310 0x7f23f8067760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:12.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.452+0000 7f23fdc00640 1 --2- 172.21.15.62:0/3675737017 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f23f81055c0 0x7f23f8105990 unknown :-1 s=CLOSED pgs=1406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:12.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.452+0000 7f23fdc00640 1 -- 172.21.15.62:0/3675737017 >> 172.21.15.62:0/3675737017 conn(0x7f23f80743d0 msgr2=0x7f23f80747d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:31:12.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.452+0000 7f23fdc00640 1 -- 172.21.15.62:0/3675737017 shutdown_connections 2024-04-20T22:31:12.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.452+0000 7f23fdc00640 1 -- 172.21.15.62:0/3675737017 wait complete. 2024-04-20T22:31:12.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.453+0000 7f23fdc00640 1 Processor -- start 2024-04-20T22:31:12.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.453+0000 7f23fdc00640 1 -- start start 2024-04-20T22:31:12.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.454+0000 7f23fdc00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23f8067310 0x7f23f8197350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:12.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.454+0000 7f23fdc00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f23f81055c0 0x7f23f8197890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:12.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.454+0000 7f23fdc00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23f8197f20 con 0x7f23f81055c0 2024-04-20T22:31:12.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.454+0000 7f23fdc00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23f819bcf0 con 0x7f23f8067310 2024-04-20T22:31:12.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.454+0000 7f23f7400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f23f81055c0 0x7f23f8197890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:12.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.454+0000 7f23f7400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f23f81055c0 0x7f23f8197890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37186/0 (socket says 172.21.15.62:37186) 2024-04-20T22:31:12.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.454+0000 7f23f7400640 1 -- 172.21.15.62:0/2167303065 learned_addr learned my addr 172.21.15.62:0/2167303065 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:31:12.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.455+0000 7f23f7400640 1 -- 172.21.15.62:0/2167303065 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23f8067310 msgr2=0x7f23f8197350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:12.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.455+0000 7f23f7e00640 1 --2- 172.21.15.62:0/2167303065 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23f8067310 0x7f23f8197350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:12.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.455+0000 7f23f7400640 1 --2- 172.21.15.62:0/2167303065 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23f8067310 0x7f23f8197350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:12.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.455+0000 7f23f7400640 1 -- 172.21.15.62:0/2167303065 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23e8009560 con 0x7f23f81055c0 2024-04-20T22:31:12.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.455+0000 7f23f7400640 1 --2- 172.21.15.62:0/2167303065 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f23f81055c0 0x7f23f8197890 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7f23e000cc60 tx=0x7f23e000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:31:12.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.455+0000 7f23f4c00640 1 -- 172.21.15.62:0/2167303065 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f23e0004b20 con 0x7f23f81055c0 2024-04-20T22:31:12.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.455+0000 7f23f4c00640 1 -- 172.21.15.62:0/2167303065 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f23e0004c80 con 0x7f23f81055c0 2024-04-20T22:31:12.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.455+0000 7f23fdc00640 1 -- 172.21.15.62:0/2167303065 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23f819bfd0 con 0x7f23f81055c0 2024-04-20T22:31:12.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.455+0000 7f23f4c00640 1 -- 172.21.15.62:0/2167303065 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f23e000f640 con 0x7f23f81055c0 2024-04-20T22:31:12.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.455+0000 7f23fdc00640 1 -- 172.21.15.62:0/2167303065 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23f819c530 con 0x7f23f81055c0 2024-04-20T22:31:12.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.457+0000 7f23fdc00640 1 -- 172.21.15.62:0/2167303065 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23f804e000 con 0x7f23f81055c0 2024-04-20T22:31:12.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.458+0000 7f23f4c00640 1 -- 172.21.15.62:0/2167303065 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f23e00055e0 con 0x7f23f81055c0 2024-04-20T22:31:12.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.460+0000 7f23f4c00640 1 --2- 172.21.15.62:0/2167303065 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f23d806ec30 0x7f23d80710e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:12.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.460+0000 7f23f4c00640 1 -- 172.21.15.62:0/2167303065 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f23e008f090 con 0x7f23f81055c0 2024-04-20T22:31:12.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.460+0000 7f23f7e00640 1 --2- 172.21.15.62:0/2167303065 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f23d806ec30 0x7f23d80710e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:12.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.461+0000 7f23f7e00640 1 --2- 172.21.15.62:0/2167303065 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f23d806ec30 0x7f23d80710e0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f23e8036040 tx=0x7f23e802cf90 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:31:12.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.465+0000 7f23f4c00640 1 -- 172.21.15.62:0/2167303065 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f23e0057e20 con 0x7f23f81055c0 2024-04-20T22:31:12.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.672+0000 7f23fdc00640 1 -- 172.21.15.62:0/2167303065 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7f23f80ff350 con 0x7f23f81055c0 2024-04-20T22:31:12.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.674+0000 7f23f4c00640 1 -- 172.21.15.62:0/2167303065 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v85) v1 ==== 107+0+11 (secure 0 0 0) 0x7f23e001d0b0 con 0x7f23f81055c0 2024-04-20T22:31:12.674 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 32 2024-04-20T22:31:12.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.676+0000 7f23fdc00640 1 -- 172.21.15.62:0/2167303065 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f23d806ec30 msgr2=0x7f23d80710e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:12.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.676+0000 7f23fdc00640 1 --2- 172.21.15.62:0/2167303065 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f23d806ec30 0x7f23d80710e0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f23e8036040 tx=0x7f23e802cf90 comp rx=0 tx=0).stop 2024-04-20T22:31:12.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.676+0000 7f23fdc00640 1 -- 172.21.15.62:0/2167303065 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f23f81055c0 msgr2=0x7f23f8197890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:12.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.676+0000 7f23fdc00640 1 --2- 172.21.15.62:0/2167303065 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f23f81055c0 0x7f23f8197890 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7f23e000cc60 tx=0x7f23e000a620 comp rx=0 tx=0).stop 2024-04-20T22:31:12.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.676+0000 7f23fdc00640 1 -- 172.21.15.62:0/2167303065 shutdown_connections 2024-04-20T22:31:12.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.676+0000 7f23fdc00640 1 --2- 172.21.15.62:0/2167303065 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f23d806ec30 0x7f23d80710e0 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:12.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.676+0000 7f23fdc00640 1 --2- 172.21.15.62:0/2167303065 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f23f81055c0 0x7f23f8197890 unknown :-1 s=CLOSED pgs=1407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:12.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.676+0000 7f23fdc00640 1 --2- 172.21.15.62:0/2167303065 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23f8067310 0x7f23f8197350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:12.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.676+0000 7f23fdc00640 1 -- 172.21.15.62:0/2167303065 >> 172.21.15.62:0/2167303065 conn(0x7f23f80743d0 msgr2=0x7f23f8106580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:31:12.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.676+0000 7f23fdc00640 1 -- 172.21.15.62:0/2167303065 shutdown_connections 2024-04-20T22:31:12.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.676+0000 7f23fdc00640 1 -- 172.21.15.62:0/2167303065 wait complete. 2024-04-20T22:31:12.695 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-20T22:31:12.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:31:12 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/240870089' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-20T22:31:12.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:31:12 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/488416000' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-20T22:31:12.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.912+0000 7fce77200640 1 -- 172.21.15.62:0/554910781 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce780750a0 msgr2=0x7fce7810fdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:12.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.912+0000 7fce77200640 1 --2- 172.21.15.62:0/554910781 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce780750a0 0x7fce7810fdf0 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7fce60009880 tx=0x7fce6002bce0 comp rx=0 tx=0).stop 2024-04-20T22:31:12.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.913+0000 7fce77200640 1 -- 172.21.15.62:0/554910781 shutdown_connections 2024-04-20T22:31:12.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.913+0000 7fce77200640 1 --2- 172.21.15.62:0/554910781 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce780750a0 0x7fce7810fdf0 unknown :-1 s=CLOSED pgs=1408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:12.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.913+0000 7fce77200640 1 --2- 172.21.15.62:0/554910781 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce78074790 0x7fce78074b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:12.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.913+0000 7fce77200640 1 -- 172.21.15.62:0/554910781 >> 172.21.15.62:0/554910781 conn(0x7fce780fd170 msgr2=0x7fce780ff580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:31:12.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.913+0000 7fce77200640 1 -- 172.21.15.62:0/554910781 shutdown_connections 2024-04-20T22:31:12.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.913+0000 7fce77200640 1 -- 172.21.15.62:0/554910781 wait complete. 2024-04-20T22:31:12.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.914+0000 7fce77200640 1 Processor -- start 2024-04-20T22:31:12.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.914+0000 7fce77200640 1 -- start start 2024-04-20T22:31:12.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.914+0000 7fce77200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce78074790 0x7fce7819b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:12.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.914+0000 7fce77200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce780750a0 0x7fce7819bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:12.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.914+0000 7fce77200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce7819c1e0 con 0x7fce78074790 2024-04-20T22:31:12.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.914+0000 7fce77200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce7819ffb0 con 0x7fce780750a0 2024-04-20T22:31:12.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.915+0000 7fce75e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce78074790 0x7fce7819b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:12.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.915+0000 7fce75e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce78074790 0x7fce7819b610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37200/0 (socket says 172.21.15.62:37200) 2024-04-20T22:31:12.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.915+0000 7fce75e00640 1 -- 172.21.15.62:0/2290937989 learned_addr learned my addr 172.21.15.62:0/2290937989 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:31:12.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.915+0000 7fce75e00640 1 -- 172.21.15.62:0/2290937989 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce780750a0 msgr2=0x7fce7819bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:12.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.915+0000 7fce6fe00640 1 --2- 172.21.15.62:0/2290937989 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce780750a0 0x7fce7819bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:12.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.915+0000 7fce75e00640 1 --2- 172.21.15.62:0/2290937989 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce780750a0 0x7fce7819bb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:12.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.915+0000 7fce75e00640 1 -- 172.21.15.62:0/2290937989 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce60009560 con 0x7fce78074790 2024-04-20T22:31:12.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.916+0000 7fce75e00640 1 --2- 172.21.15.62:0/2290937989 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce78074790 0x7fce7819b610 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7fce6800d8d0 tx=0x7fce6800dd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:31:12.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.916+0000 7fce6ea00640 1 -- 172.21.15.62:0/2290937989 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce68009c00 con 0x7fce78074790 2024-04-20T22:31:12.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.916+0000 7fce6ea00640 1 -- 172.21.15.62:0/2290937989 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fce68009d60 con 0x7fce78074790 2024-04-20T22:31:12.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.916+0000 7fce6ea00640 1 -- 172.21.15.62:0/2290937989 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce680049d0 con 0x7fce78074790 2024-04-20T22:31:12.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.916+0000 7fce77200640 1 -- 172.21.15.62:0/2290937989 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce781a0290 con 0x7fce78074790 2024-04-20T22:31:12.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.916+0000 7fce77200640 1 -- 172.21.15.62:0/2290937989 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce781a07f0 con 0x7fce78074790 2024-04-20T22:31:12.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.918+0000 7fce77200640 1 -- 172.21.15.62:0/2290937989 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce7804e850 con 0x7fce78074790 2024-04-20T22:31:12.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.919+0000 7fce6ea00640 1 -- 172.21.15.62:0/2290937989 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fce68004c40 con 0x7fce78074790 2024-04-20T22:31:12.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.920+0000 7fce6ea00640 1 --2- 172.21.15.62:0/2290937989 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fce5c06ecf0 0x7fce5c0711a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:12.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.920+0000 7fce6ea00640 1 -- 172.21.15.62:0/2290937989 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fce6808ebe0 con 0x7fce78074790 2024-04-20T22:31:12.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.920+0000 7fce6fe00640 1 --2- 172.21.15.62:0/2290937989 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fce5c06ecf0 0x7fce5c0711a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:12.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.921+0000 7fce6fe00640 1 --2- 172.21.15.62:0/2290937989 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fce5c06ecf0 0x7fce5c0711a0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fce60036040 tx=0x7fce6002cf90 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:31:12.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:12.925+0000 7fce6ea00640 1 -- 172.21.15.62:0/2290937989 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fce68057970 con 0x7fce78074790 2024-04-20T22:31:12.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:31:12 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/240870089' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-20T22:31:12.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:31:12 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/488416000' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-20T22:31:13.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:13.173+0000 7fce77200640 1 -- 172.21.15.62:0/2290937989 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fce7810d4d0 con 0x7fce78074790 2024-04-20T22:31:13.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:13.174+0000 7fce6ea00640 1 -- 172.21.15.62:0/2290937989 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) v1 ==== 94+0+5269 (secure 0 0 0) 0x7fce6808e020 con 0x7fce78074790 2024-04-20T22:31:13.175 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:31:13.175 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":39,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":39,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-20T21:27:21.032337+0000","modified":"2024-04-20T22:28:58.231260+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34444,"mds_1":34476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34444":{"gid":34444,"name":"cephfs.smithi062.ryqblb","rank":0,"incarnation":31,"state":"up:active","state_seq":699,"addr":"172.21.15.62:6827/4113916205","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":4113916205},{"type":"v1","addr":"172.21.15.62:6827","nonce":4113916205}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34476":{"gid":34476,"name":"cephfs.smithi062.ozizwi","rank":1,"incarnation":31,"state":"up:active","state_seq":691,"addr":"172.21.15.62:6829/3801975165","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":3801975165},{"type":"v1","addr":"172.21.15.62:6829","nonce":3801975165}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_44727":{"gid":44727,"name":"cephfs.smithi189.dvimer","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6825/3576699232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":3576699232},{"type":"v1","addr":"172.21.15.189:6825","nonce":3576699232}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_45075":{"gid":45075,"name":"cephfs.smithi189.xbgspw","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6827/212905792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":212905792},{"type":"v1","addr":"172.21.15.189:6827","nonce":212905792}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":34444,"qdb_cluster":[34444,34476]},"id":1}]} 2024-04-20T22:31:13.175 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 39 2024-04-20T22:31:13.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:13.177+0000 7fce77200640 1 -- 172.21.15.62:0/2290937989 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fce5c06ecf0 msgr2=0x7fce5c0711a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:13.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:13.177+0000 7fce77200640 1 --2- 172.21.15.62:0/2290937989 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fce5c06ecf0 0x7fce5c0711a0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fce60036040 tx=0x7fce6002cf90 comp rx=0 tx=0).stop 2024-04-20T22:31:13.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:13.177+0000 7fce77200640 1 -- 172.21.15.62:0/2290937989 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce78074790 msgr2=0x7fce7819b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:13.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:13.177+0000 7fce77200640 1 --2- 172.21.15.62:0/2290937989 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce78074790 0x7fce7819b610 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7fce6800d8d0 tx=0x7fce6800dd90 comp rx=0 tx=0).stop 2024-04-20T22:31:13.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:13.178+0000 7fce77200640 1 -- 172.21.15.62:0/2290937989 shutdown_connections 2024-04-20T22:31:13.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:13.178+0000 7fce77200640 1 --2- 172.21.15.62:0/2290937989 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fce5c06ecf0 0x7fce5c0711a0 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:13.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:13.178+0000 7fce77200640 1 --2- 172.21.15.62:0/2290937989 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce780750a0 0x7fce7819bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:13.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:13.178+0000 7fce77200640 1 --2- 172.21.15.62:0/2290937989 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce78074790 0x7fce7819b610 unknown :-1 s=CLOSED pgs=1409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:13.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:13.178+0000 7fce77200640 1 -- 172.21.15.62:0/2290937989 >> 172.21.15.62:0/2290937989 conn(0x7fce780fd170 msgr2=0x7fce780ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:31:13.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:13.178+0000 7fce77200640 1 -- 172.21.15.62:0/2290937989 shutdown_connections 2024-04-20T22:31:13.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:13.178+0000 7fce77200640 1 -- 172.21.15.62:0/2290937989 wait complete. 2024-04-20T22:31:13.196 DEBUG:tasks.fs:checking fs fscid=1,name=cephfs state = {'epoch': 14, 'max_mds': 2, 'flags': 50} 2024-04-20T22:31:13.196 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 15 2024-04-20T22:31:13.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:13.403+0000 7ff3a3000640 1 -- 172.21.15.62:0/1754129174 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff39c105880 msgr2=0x7ff39c105c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:13.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:13.403+0000 7ff3a3000640 1 --2- 172.21.15.62:0/1754129174 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff39c105880 0x7ff39c105c50 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7ff38c009c20 tx=0x7ff38c02c1b0 comp rx=0 tx=0).stop 2024-04-20T22:31:13.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:13.403+0000 7ff3a3000640 1 -- 172.21.15.62:0/1754129174 shutdown_connections 2024-04-20T22:31:13.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:13.403+0000 7ff3a3000640 1 --2- 172.21.15.62:0/1754129174 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff39c0ff870 0x7ff39c0ffce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:13.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:13.403+0000 7ff3a3000640 1 --2- 172.21.15.62:0/1754129174 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff39c105880 0x7ff39c105c50 unknown :-1 s=CLOSED pgs=1410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:13.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:13.403+0000 7ff3a3000640 1 -- 172.21.15.62:0/1754129174 >> 172.21.15.62:0/1754129174 conn(0x7ff39c0fb370 msgr2=0x7ff39c0fd780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:31:13.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:13.403+0000 7ff3a3000640 1 -- 172.21.15.62:0/1754129174 shutdown_connections 2024-04-20T22:31:13.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:13.403+0000 7ff3a3000640 1 -- 172.21.15.62:0/1754129174 wait complete. 2024-04-20T22:31:13.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:13.404+0000 7ff3a3000640 1 Processor -- start 2024-04-20T22:31:13.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:13.404+0000 7ff3a3000640 1 -- start start 2024-04-20T22:31:13.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:13.405+0000 7ff3a3000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff39c0ff870 0x7ff39c197710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:13.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:13.405+0000 7ff3a3000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff39c105880 0x7ff39c197c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:13.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:13.405+0000 7ff3a3000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff39c198330 con 0x7ff39c105880 2024-04-20T22:31:13.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:13.405+0000 7ff3a3000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff39c19b000 con 0x7ff39c0ff870 2024-04-20T22:31:13.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:13.405+0000 7ff393400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff39c105880 0x7ff39c197c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:13.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:13.405+0000 7ff393400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff39c105880 0x7ff39c197c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37226/0 (socket says 172.21.15.62:37226) 2024-04-20T22:31:13.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:13.405+0000 7ff393400640 1 -- 172.21.15.62:0/4110178247 learned_addr learned my addr 172.21.15.62:0/4110178247 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:31:13.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:13.405+0000 7ff39be00640 1 --2- 172.21.15.62:0/4110178247 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff39c0ff870 0x7ff39c197710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:13.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:13.405+0000 7ff393400640 1 -- 172.21.15.62:0/4110178247 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff39c0ff870 msgr2=0x7ff39c197710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:13.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:13.405+0000 7ff393400640 1 --2- 172.21.15.62:0/4110178247 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff39c0ff870 0x7ff39c197710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:13.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:13.405+0000 7ff393400640 1 -- 172.21.15.62:0/4110178247 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff38c009900 con 0x7ff39c105880 2024-04-20T22:31:13.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:13.406+0000 7ff393400640 1 --2- 172.21.15.62:0/4110178247 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff39c105880 0x7ff39c197c50 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7ff38400d8d0 tx=0x7ff38400dd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:31:13.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:13.406+0000 7ff399600640 1 -- 172.21.15.62:0/4110178247 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff384009c00 con 0x7ff39c105880 2024-04-20T22:31:13.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:13.406+0000 7ff399600640 1 -- 172.21.15.62:0/4110178247 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff384009d60 con 0x7ff39c105880 2024-04-20T22:31:13.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:13.406+0000 7ff3a3000640 1 -- 172.21.15.62:0/4110178247 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff39c19b2e0 con 0x7ff39c105880 2024-04-20T22:31:13.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:13.406+0000 7ff399600640 1 -- 172.21.15.62:0/4110178247 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff3840049d0 con 0x7ff39c105880 2024-04-20T22:31:13.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:13.406+0000 7ff3a3000640 1 -- 172.21.15.62:0/4110178247 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff39c19b840 con 0x7ff39c105880 2024-04-20T22:31:13.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:13.407+0000 7ff3a3000640 1 -- 172.21.15.62:0/4110178247 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff39c04e000 con 0x7ff39c105880 2024-04-20T22:31:13.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:13.409+0000 7ff399600640 1 -- 172.21.15.62:0/4110178247 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff384004c90 con 0x7ff39c105880 2024-04-20T22:31:13.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:13.410+0000 7ff399600640 1 --2- 172.21.15.62:0/4110178247 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff38006ec30 0x7ff3800710e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:13.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:13.410+0000 7ff399600640 1 -- 172.21.15.62:0/4110178247 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7ff38408ecb0 con 0x7ff39c105880 2024-04-20T22:31:13.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:13.410+0000 7ff39be00640 1 --2- 172.21.15.62:0/4110178247 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff38006ec30 0x7ff3800710e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:13.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:13.411+0000 7ff39be00640 1 --2- 172.21.15.62:0/4110178247 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff38006ec30 0x7ff3800710e0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7ff38c004af0 tx=0x7ff38c004a40 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:31:13.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:13.415+0000 7ff399600640 1 -- 172.21.15.62:0/4110178247 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7ff3840579c0 con 0x7ff39c105880 2024-04-20T22:31:13.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:13.663+0000 7ff3a3000640 1 -- 172.21.15.62:0/4110178247 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 15, "format": "json"} v 0) v1 -- 0x7ff39c066860 con 0x7ff39c105880 2024-04-20T22:31:13.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:13.665+0000 7ff399600640 1 -- 172.21.15.62:0/4110178247 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 15, "format": "json"}]=0 dumped fsmap epoch 15 v39) v1 ==== 107+0+4908 (secure 0 0 0) 0x7ff38405b080 con 0x7ff39c105880 2024-04-20T22:31:13.670 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:31:13.670 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":15,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":15,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-20T21:27:21.032337+0000","modified":"2024-04-20T21:30:11.050729+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14458,"mds_1":24279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14458":{"gid":14458,"name":"cephfs.smithi062.ryqblb","rank":0,"incarnation":3,"state":"up:active","state_seq":6,"addr":"172.21.15.62:6827/1147127123","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":1147127123},{"type":"v1","addr":"172.21.15.62:6827","nonce":1147127123}]},"join_fscid":1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14474":{"gid":14474,"name":"cephfs.smithi062.ozizwi","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":3,"addr":"172.21.15.62:6829/3494174293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":3494174293},{"type":"v1","addr":"172.21.15.62:6829","nonce":3494174293}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24279":{"gid":24279,"name":"cephfs.smithi189.dvimer","rank":1,"incarnation":8,"state":"up:active","state_seq":5,"addr":"172.21.15.189:6825/237262514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":237262514},{"type":"v1","addr":"172.21.15.189:6825","nonce":237262514}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24291":{"gid":24291,"name":"cephfs.smithi189.xbgspw","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":2,"addr":"172.21.15.189:6827/1495356258","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":1495356258},{"type":"v1","addr":"172.21.15.189:6827","nonce":1495356258}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-20T22:31:13.671 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 15 2024-04-20T22:31:13.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:13.667+0000 7ff3a3000640 1 -- 172.21.15.62:0/4110178247 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff38006ec30 msgr2=0x7ff3800710e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:13.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:13.667+0000 7ff3a3000640 1 --2- 172.21.15.62:0/4110178247 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff38006ec30 0x7ff3800710e0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7ff38c004af0 tx=0x7ff38c004a40 comp rx=0 tx=0).stop 2024-04-20T22:31:13.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:13.667+0000 7ff3a3000640 1 -- 172.21.15.62:0/4110178247 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff39c105880 msgr2=0x7ff39c197c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:13.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:13.667+0000 7ff3a3000640 1 --2- 172.21.15.62:0/4110178247 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff39c105880 0x7ff39c197c50 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7ff38400d8d0 tx=0x7ff38400dd90 comp rx=0 tx=0).stop 2024-04-20T22:31:13.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:13.667+0000 7ff3a3000640 1 -- 172.21.15.62:0/4110178247 shutdown_connections 2024-04-20T22:31:13.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:13.667+0000 7ff3a3000640 1 --2- 172.21.15.62:0/4110178247 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff38006ec30 0x7ff3800710e0 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:13.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:13.667+0000 7ff3a3000640 1 --2- 172.21.15.62:0/4110178247 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff39c105880 0x7ff39c197c50 unknown :-1 s=CLOSED pgs=1411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:13.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:13.668+0000 7ff3a3000640 1 --2- 172.21.15.62:0/4110178247 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff39c0ff870 0x7ff39c197710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:13.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:13.668+0000 7ff3a3000640 1 -- 172.21.15.62:0/4110178247 >> 172.21.15.62:0/4110178247 conn(0x7ff39c0fb370 msgr2=0x7ff39c0fcb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:31:13.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:13.668+0000 7ff3a3000640 1 -- 172.21.15.62:0/4110178247 shutdown_connections 2024-04-20T22:31:13.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:13.668+0000 7ff3a3000640 1 -- 172.21.15.62:0/4110178247 wait complete. 2024-04-20T22:31:13.685 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 16 2024-04-20T22:31:13.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:13.891+0000 7f3d9e800640 1 -- 172.21.15.62:0/2797200250 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3d981034f0 msgr2=0x7f3d981038c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:13.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:13.891+0000 7f3d9e800640 1 --2- 172.21.15.62:0/2797200250 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3d981034f0 0x7f3d981038c0 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7f3d8c009880 tx=0x7f3d8c02bce0 comp rx=0 tx=0).stop 2024-04-20T22:31:13.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:13.891+0000 7f3d9e800640 1 -- 172.21.15.62:0/2797200250 shutdown_connections 2024-04-20T22:31:13.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:13.891+0000 7f3d9e800640 1 --2- 172.21.15.62:0/2797200250 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3d980fedc0 0x7f3d980ff230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:13.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:13.891+0000 7f3d9e800640 1 --2- 172.21.15.62:0/2797200250 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3d981034f0 0x7f3d981038c0 unknown :-1 s=CLOSED pgs=1412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:13.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:13.891+0000 7f3d9e800640 1 -- 172.21.15.62:0/2797200250 >> 172.21.15.62:0/2797200250 conn(0x7f3d980743d0 msgr2=0x7f3d980747d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:31:13.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:13.892+0000 7f3d9e800640 1 -- 172.21.15.62:0/2797200250 shutdown_connections 2024-04-20T22:31:13.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:13.892+0000 7f3d9e800640 1 -- 172.21.15.62:0/2797200250 wait complete. 2024-04-20T22:31:13.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:13.893+0000 7f3d9e800640 1 Processor -- start 2024-04-20T22:31:13.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:13.893+0000 7f3d9e800640 1 -- start start 2024-04-20T22:31:13.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:13.893+0000 7f3d9e800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3d980fedc0 0x7f3d98197300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:13.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:13.893+0000 7f3d9e800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3d981034f0 0x7f3d98197840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:13.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:13.893+0000 7f3d9e800640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d98197ed0 con 0x7f3d981034f0 2024-04-20T22:31:13.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:13.893+0000 7f3d9e800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d9819bca0 con 0x7f3d980fedc0 2024-04-20T22:31:13.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:13.893+0000 7f3d9ca00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3d981034f0 0x7f3d98197840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:13.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:13.893+0000 7f3d9ca00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3d981034f0 0x7f3d98197840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37244/0 (socket says 172.21.15.62:37244) 2024-04-20T22:31:13.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:13.893+0000 7f3d9ca00640 1 -- 172.21.15.62:0/3937666721 learned_addr learned my addr 172.21.15.62:0/3937666721 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:31:13.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:13.894+0000 7f3d9ca00640 1 -- 172.21.15.62:0/3937666721 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3d980fedc0 msgr2=0x7f3d98197300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:13.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:13.894+0000 7f3d9ca00640 1 --2- 172.21.15.62:0/3937666721 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3d980fedc0 0x7f3d98197300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:13.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:13.894+0000 7f3d9ca00640 1 -- 172.21.15.62:0/3937666721 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d8c009560 con 0x7f3d981034f0 2024-04-20T22:31:13.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:13.894+0000 7f3d9ca00640 1 --2- 172.21.15.62:0/3937666721 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3d981034f0 0x7f3d98197840 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7f3d8400c8f0 tx=0x7f3d8400cdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:31:13.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:13.894+0000 7f3d96000640 1 -- 172.21.15.62:0/3937666721 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3d84007b80 con 0x7f3d981034f0 2024-04-20T22:31:13.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:13.894+0000 7f3d96000640 1 -- 172.21.15.62:0/3937666721 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3d840054e0 con 0x7f3d981034f0 2024-04-20T22:31:13.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:13.894+0000 7f3d96000640 1 -- 172.21.15.62:0/3937666721 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3d8400f450 con 0x7f3d981034f0 2024-04-20T22:31:13.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:13.894+0000 7f3d9e800640 1 -- 172.21.15.62:0/3937666721 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d9819bf80 con 0x7f3d981034f0 2024-04-20T22:31:13.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:13.895+0000 7f3d9e800640 1 -- 172.21.15.62:0/3937666721 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d9819c560 con 0x7f3d981034f0 2024-04-20T22:31:13.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:13.896+0000 7f3d9e800640 1 -- 172.21.15.62:0/3937666721 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d9804e000 con 0x7f3d981034f0 2024-04-20T22:31:13.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:13.897+0000 7f3d96000640 1 -- 172.21.15.62:0/3937666721 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f3d84004f60 con 0x7f3d981034f0 2024-04-20T22:31:13.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:13.898+0000 7f3d96000640 1 --2- 172.21.15.62:0/3937666721 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3d8006ec30 0x7f3d800710e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:13.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:13.898+0000 7f3d96000640 1 -- 172.21.15.62:0/3937666721 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f3d8408ec90 con 0x7f3d981034f0 2024-04-20T22:31:13.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:13.898+0000 7f3d9d400640 1 --2- 172.21.15.62:0/3937666721 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3d8006ec30 0x7f3d800710e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:13.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:13.899+0000 7f3d9d400640 1 --2- 172.21.15.62:0/3937666721 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3d8006ec30 0x7f3d800710e0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f3d8c02efd0 tx=0x7f3d8c000bc0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:31:13.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:13.902+0000 7f3d96000640 1 -- 172.21.15.62:0/3937666721 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f3d84057a20 con 0x7f3d981034f0 2024-04-20T22:31:13.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:31:13 smithi189 ceph-mon[57022]: pgmap v1682: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:31:13.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:31:13 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/2167303065' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-20T22:31:13.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:31:13 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/2290937989' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-20T22:31:14.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:31:13 smithi062 ceph-mon[76322]: pgmap v1682: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:31:14.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:31:13 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/2167303065' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-20T22:31:14.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:31:13 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/2290937989' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-20T22:31:14.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:14.139+0000 7f3d9e800640 1 -- 172.21.15.62:0/3937666721 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 16, "format": "json"} v 0) v1 -- 0x7f3d98198650 con 0x7f3d981034f0 2024-04-20T22:31:14.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:14.141+0000 7f3d96000640 1 -- 172.21.15.62:0/3937666721 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 16, "format": "json"}]=0 dumped fsmap epoch 16 v39) v1 ==== 107+0+4909 (secure 0 0 0) 0x7f3d8401d090 con 0x7f3d981034f0 2024-04-20T22:31:14.141 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:31:14.141 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":16,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-20T21:27:21.032337+0000","modified":"2024-04-20T21:31:14.906617+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14458,"mds_1":24279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14458":{"gid":14458,"name":"cephfs.smithi062.ryqblb","rank":0,"incarnation":3,"state":"up:active","state_seq":6,"addr":"172.21.15.62:6827/1147127123","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":1147127123},{"type":"v1","addr":"172.21.15.62:6827","nonce":1147127123}]},"join_fscid":1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14474":{"gid":14474,"name":"cephfs.smithi062.ozizwi","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":3,"addr":"172.21.15.62:6829/3494174293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":3494174293},{"type":"v1","addr":"172.21.15.62:6829","nonce":3494174293}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24279":{"gid":24279,"name":"cephfs.smithi189.dvimer","rank":1,"incarnation":8,"state":"up:active","state_seq":5,"addr":"172.21.15.189:6825/237262514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":237262514},{"type":"v1","addr":"172.21.15.189:6825","nonce":237262514}]},"join_fscid":1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24291":{"gid":24291,"name":"cephfs.smithi189.xbgspw","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":2,"addr":"172.21.15.189:6827/1495356258","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":1495356258},{"type":"v1","addr":"172.21.15.189:6827","nonce":1495356258}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-20T22:31:14.141 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 16 2024-04-20T22:31:14.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:14.143+0000 7f3d9e800640 1 -- 172.21.15.62:0/3937666721 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3d8006ec30 msgr2=0x7f3d800710e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:14.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:14.143+0000 7f3d9e800640 1 --2- 172.21.15.62:0/3937666721 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3d8006ec30 0x7f3d800710e0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f3d8c02efd0 tx=0x7f3d8c000bc0 comp rx=0 tx=0).stop 2024-04-20T22:31:14.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:14.143+0000 7f3d9e800640 1 -- 172.21.15.62:0/3937666721 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3d981034f0 msgr2=0x7f3d98197840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:14.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:14.143+0000 7f3d9e800640 1 --2- 172.21.15.62:0/3937666721 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3d981034f0 0x7f3d98197840 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7f3d8400c8f0 tx=0x7f3d8400cdb0 comp rx=0 tx=0).stop 2024-04-20T22:31:14.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:14.143+0000 7f3d9e800640 1 -- 172.21.15.62:0/3937666721 shutdown_connections 2024-04-20T22:31:14.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:14.143+0000 7f3d9e800640 1 --2- 172.21.15.62:0/3937666721 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3d8006ec30 0x7f3d800710e0 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:14.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:14.143+0000 7f3d9e800640 1 --2- 172.21.15.62:0/3937666721 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3d981034f0 0x7f3d98197840 unknown :-1 s=CLOSED pgs=1413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:14.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:14.143+0000 7f3d9e800640 1 --2- 172.21.15.62:0/3937666721 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3d980fedc0 0x7f3d98197300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:14.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:14.143+0000 7f3d9e800640 1 -- 172.21.15.62:0/3937666721 >> 172.21.15.62:0/3937666721 conn(0x7f3d980743d0 msgr2=0x7f3d98067ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:31:14.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:14.144+0000 7f3d9e800640 1 -- 172.21.15.62:0/3937666721 shutdown_connections 2024-04-20T22:31:14.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:14.144+0000 7f3d9e800640 1 -- 172.21.15.62:0/3937666721 wait complete. 2024-04-20T22:31:14.162 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 17 2024-04-20T22:31:14.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:14.361+0000 7f91e3200640 1 -- 172.21.15.62:0/4158076693 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91e40ff470 msgr2=0x7f91e40ff8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:14.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:14.361+0000 7f91e3200640 1 --2- 172.21.15.62:0/4158076693 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91e40ff470 0x7f91e40ff8e0 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7f91cc009880 tx=0x7f91cc02bce0 comp rx=0 tx=0).stop 2024-04-20T22:31:14.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:14.361+0000 7f91e3200640 1 -- 172.21.15.62:0/4158076693 shutdown_connections 2024-04-20T22:31:14.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:14.361+0000 7f91e3200640 1 --2- 172.21.15.62:0/4158076693 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91e40ff470 0x7f91e40ff8e0 unknown :-1 s=CLOSED pgs=1414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:14.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:14.361+0000 7f91e3200640 1 --2- 172.21.15.62:0/4158076693 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91e4105480 0x7f91e4105850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:14.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:14.361+0000 7f91e3200640 1 -- 172.21.15.62:0/4158076693 >> 172.21.15.62:0/4158076693 conn(0x7f91e40faf70 msgr2=0x7f91e40fd380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:31:14.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:14.361+0000 7f91e3200640 1 -- 172.21.15.62:0/4158076693 shutdown_connections 2024-04-20T22:31:14.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:14.361+0000 7f91e3200640 1 -- 172.21.15.62:0/4158076693 wait complete. 2024-04-20T22:31:14.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:14.362+0000 7f91e3200640 1 Processor -- start 2024-04-20T22:31:14.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:14.363+0000 7f91e3200640 1 -- start start 2024-04-20T22:31:14.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:14.363+0000 7f91e3200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91e40ff470 0x7f91e4197350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:14.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:14.363+0000 7f91e3200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91e4105480 0x7f91e4197890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:14.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:14.363+0000 7f91e3200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91e4197f20 con 0x7f91e4105480 2024-04-20T22:31:14.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:14.363+0000 7f91e3200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91e419bcf0 con 0x7f91e40ff470 2024-04-20T22:31:14.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:14.363+0000 7f91dbe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91e4105480 0x7f91e4197890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:14.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:14.363+0000 7f91dbe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91e4105480 0x7f91e4197890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37266/0 (socket says 172.21.15.62:37266) 2024-04-20T22:31:14.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:14.363+0000 7f91dbe00640 1 -- 172.21.15.62:0/651765538 learned_addr learned my addr 172.21.15.62:0/651765538 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:31:14.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:14.364+0000 7f91dbe00640 1 -- 172.21.15.62:0/651765538 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91e40ff470 msgr2=0x7f91e4197350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:14.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:14.364+0000 7f91dbe00640 1 --2- 172.21.15.62:0/651765538 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91e40ff470 0x7f91e4197350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:14.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:14.364+0000 7f91dbe00640 1 -- 172.21.15.62:0/651765538 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91cc009560 con 0x7f91e4105480 2024-04-20T22:31:14.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:14.364+0000 7f91dbe00640 1 --2- 172.21.15.62:0/651765538 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91e4105480 0x7f91e4197890 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7f91cc000c00 tx=0x7f91cc004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:31:14.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:14.364+0000 7f91daa00640 1 -- 172.21.15.62:0/651765538 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f91cc0037c0 con 0x7f91e4105480 2024-04-20T22:31:14.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:14.364+0000 7f91daa00640 1 -- 172.21.15.62:0/651765538 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f91cc003de0 con 0x7f91e4105480 2024-04-20T22:31:14.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:14.364+0000 7f91e3200640 1 -- 172.21.15.62:0/651765538 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91e419bf70 con 0x7f91e4105480 2024-04-20T22:31:14.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:14.364+0000 7f91daa00640 1 -- 172.21.15.62:0/651765538 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f91cc034460 con 0x7f91e4105480 2024-04-20T22:31:14.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:14.364+0000 7f91e3200640 1 -- 172.21.15.62:0/651765538 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91e419c470 con 0x7f91e4105480 2024-04-20T22:31:14.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:14.366+0000 7f91e3200640 1 -- 172.21.15.62:0/651765538 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91e404e000 con 0x7f91e4105480 2024-04-20T22:31:14.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:14.367+0000 7f91daa00640 1 -- 172.21.15.62:0/651765538 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f91cc003920 con 0x7f91e4105480 2024-04-20T22:31:14.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:14.368+0000 7f91daa00640 1 --2- 172.21.15.62:0/651765538 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f91c806ec30 0x7f91c80710e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:14.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:14.368+0000 7f91daa00640 1 -- 172.21.15.62:0/651765538 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f91cc0b1400 con 0x7f91e4105480 2024-04-20T22:31:14.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:14.368+0000 7f91e1e00640 1 --2- 172.21.15.62:0/651765538 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f91c806ec30 0x7f91c80710e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:14.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:14.369+0000 7f91e1e00640 1 --2- 172.21.15.62:0/651765538 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f91c806ec30 0x7f91c80710e0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f91d400a8f0 tx=0x7f91d4005bf0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:31:14.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:14.372+0000 7f91daa00640 1 -- 172.21.15.62:0/651765538 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f91cc07a1d0 con 0x7f91e4105480 2024-04-20T22:31:14.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:14.628+0000 7f91e3200640 1 -- 172.21.15.62:0/651765538 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 17, "format": "json"} v 0) v1 -- 0x7f91e410c730 con 0x7f91e4105480 2024-04-20T22:31:14.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:14.629+0000 7f91daa00640 1 -- 172.21.15.62:0/651765538 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 17, "format": "json"}]=0 dumped fsmap epoch 17 v39) v1 ==== 107+0+4923 (secure 0 0 0) 0x7f91cc07d890 con 0x7f91e4105480 2024-04-20T22:31:14.630 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:31:14.630 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":17,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-20T21:27:21.032337+0000","modified":"2024-04-20T21:36:45.237235+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14458,"mds_1":24279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14458":{"gid":14458,"name":"cephfs.smithi062.ryqblb","rank":0,"incarnation":3,"state":"up:active","state_seq":6,"addr":"172.21.15.62:6827/1147127123","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":1147127123},{"type":"v1","addr":"172.21.15.62:6827","nonce":1147127123}]},"join_fscid":1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14474":{"gid":14474,"name":"cephfs.smithi062.ozizwi","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":3,"addr":"172.21.15.62:6829/3494174293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":3494174293},{"type":"v1","addr":"172.21.15.62:6829","nonce":3494174293}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24279":{"gid":24279,"name":"cephfs.smithi189.dvimer","rank":1,"incarnation":8,"state":"up:active","state_seq":5,"addr":"172.21.15.189:6825/237262514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":237262514},{"type":"v1","addr":"172.21.15.189:6825","nonce":237262514}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24291":{"gid":24291,"name":"cephfs.smithi189.xbgspw","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":2,"addr":"172.21.15.189:6827/1495356258","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":1495356258},{"type":"v1","addr":"172.21.15.189:6827","nonce":1495356258}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":14458,"qdb_cluster":[14458,24279]},"id":1}]} 2024-04-20T22:31:14.631 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 17 2024-04-20T22:31:14.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:14.632+0000 7f91e3200640 1 -- 172.21.15.62:0/651765538 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f91c806ec30 msgr2=0x7f91c80710e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:14.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:14.632+0000 7f91e3200640 1 --2- 172.21.15.62:0/651765538 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f91c806ec30 0x7f91c80710e0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f91d400a8f0 tx=0x7f91d4005bf0 comp rx=0 tx=0).stop 2024-04-20T22:31:14.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:14.632+0000 7f91e3200640 1 -- 172.21.15.62:0/651765538 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91e4105480 msgr2=0x7f91e4197890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:14.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:14.632+0000 7f91e3200640 1 --2- 172.21.15.62:0/651765538 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91e4105480 0x7f91e4197890 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7f91cc000c00 tx=0x7f91cc004270 comp rx=0 tx=0).stop 2024-04-20T22:31:14.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:14.632+0000 7f91e3200640 1 -- 172.21.15.62:0/651765538 shutdown_connections 2024-04-20T22:31:14.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:14.632+0000 7f91e3200640 1 --2- 172.21.15.62:0/651765538 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f91c806ec30 0x7f91c80710e0 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:14.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:14.632+0000 7f91e3200640 1 --2- 172.21.15.62:0/651765538 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91e4105480 0x7f91e4197890 unknown :-1 s=CLOSED pgs=1415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:14.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:14.632+0000 7f91e3200640 1 --2- 172.21.15.62:0/651765538 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91e40ff470 0x7f91e4197350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:14.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:14.632+0000 7f91e3200640 1 -- 172.21.15.62:0/651765538 >> 172.21.15.62:0/651765538 conn(0x7f91e40faf70 msgr2=0x7f91e40fb850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:31:14.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:14.633+0000 7f91e3200640 1 -- 172.21.15.62:0/651765538 shutdown_connections 2024-04-20T22:31:14.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:14.633+0000 7f91e3200640 1 -- 172.21.15.62:0/651765538 wait complete. 2024-04-20T22:31:14.652 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 18 2024-04-20T22:31:14.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:14.862+0000 7fe903200640 1 -- 172.21.15.62:0/751452049 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe904072600 msgr2=0x7fe9040729d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:14.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:14.862+0000 7fe903200640 1 --2- 172.21.15.62:0/751452049 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe904072600 0x7fe9040729d0 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7fe8f4009880 tx=0x7fe8f402bd70 comp rx=0 tx=0).stop 2024-04-20T22:31:14.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:14.863+0000 7fe903200640 1 -- 172.21.15.62:0/751452049 shutdown_connections 2024-04-20T22:31:14.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:14.863+0000 7fe903200640 1 --2- 172.21.15.62:0/751452049 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe904072f10 0x7fe90410ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:14.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:14.863+0000 7fe903200640 1 --2- 172.21.15.62:0/751452049 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe904072600 0x7fe9040729d0 unknown :-1 s=CLOSED pgs=1416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:14.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:14.863+0000 7fe903200640 1 -- 172.21.15.62:0/751452049 >> 172.21.15.62:0/751452049 conn(0x7fe9040faf70 msgr2=0x7fe9040fd380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:31:14.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:14.863+0000 7fe903200640 1 -- 172.21.15.62:0/751452049 shutdown_connections 2024-04-20T22:31:14.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:14.863+0000 7fe903200640 1 -- 172.21.15.62:0/751452049 wait complete. 2024-04-20T22:31:14.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:14.864+0000 7fe903200640 1 Processor -- start 2024-04-20T22:31:14.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:14.864+0000 7fe903200640 1 -- start start 2024-04-20T22:31:14.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:14.864+0000 7fe903200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe904072600 0x7fe9041a1570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:14.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:14.864+0000 7fe903200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe904072f10 0x7fe9041a1ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:14.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:14.864+0000 7fe903200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe90419b5e0 con 0x7fe904072600 2024-04-20T22:31:14.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:14.864+0000 7fe903200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe90419b750 con 0x7fe904072f10 2024-04-20T22:31:14.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:14.864+0000 7fe901e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe904072600 0x7fe9041a1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:14.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:14.864+0000 7fe901e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe904072600 0x7fe9041a1570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37296/0 (socket says 172.21.15.62:37296) 2024-04-20T22:31:14.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:14.864+0000 7fe901e00640 1 -- 172.21.15.62:0/2715886208 learned_addr learned my addr 172.21.15.62:0/2715886208 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:31:14.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:14.865+0000 7fe901e00640 1 -- 172.21.15.62:0/2715886208 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe904072f10 msgr2=0x7fe9041a1ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:14.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:14.865+0000 7fe901e00640 1 --2- 172.21.15.62:0/2715886208 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe904072f10 0x7fe9041a1ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:14.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:14.865+0000 7fe901e00640 1 -- 172.21.15.62:0/2715886208 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8f4009560 con 0x7fe904072600 2024-04-20T22:31:14.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:14.865+0000 7fe901e00640 1 --2- 172.21.15.62:0/2715886208 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe904072600 0x7fe9041a1570 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7fe8f4036040 tx=0x7fe8f4004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:31:14.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:14.865+0000 7fe8faa00640 1 -- 172.21.15.62:0/2715886208 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe8f4004aa0 con 0x7fe904072600 2024-04-20T22:31:14.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:14.865+0000 7fe8faa00640 1 -- 172.21.15.62:0/2715886208 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe8f4003cc0 con 0x7fe904072600 2024-04-20T22:31:14.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:14.865+0000 7fe8faa00640 1 -- 172.21.15.62:0/2715886208 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe8f403e450 con 0x7fe904072600 2024-04-20T22:31:14.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:14.865+0000 7fe903200640 1 -- 172.21.15.62:0/2715886208 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe90419b9d0 con 0x7fe904072600 2024-04-20T22:31:14.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:14.866+0000 7fe903200640 1 -- 172.21.15.62:0/2715886208 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe90419bf30 con 0x7fe904072600 2024-04-20T22:31:14.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:14.867+0000 7fe903200640 1 -- 172.21.15.62:0/2715886208 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe90404e000 con 0x7fe904072600 2024-04-20T22:31:14.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:14.868+0000 7fe8faa00640 1 -- 172.21.15.62:0/2715886208 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe8f4005060 con 0x7fe904072600 2024-04-20T22:31:14.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:14.869+0000 7fe8faa00640 1 --2- 172.21.15.62:0/2715886208 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe8e806ec30 0x7fe8e80710e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:14.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:14.869+0000 7fe8faa00640 1 -- 172.21.15.62:0/2715886208 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fe8f40b1ad0 con 0x7fe904072600 2024-04-20T22:31:14.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:14.869+0000 7fe8fbe00640 1 --2- 172.21.15.62:0/2715886208 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe8e806ec30 0x7fe8e80710e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:14.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:14.870+0000 7fe8fbe00640 1 --2- 172.21.15.62:0/2715886208 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe8e806ec30 0x7fe8e80710e0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fe9040fd720 tx=0x7fe8ec006c90 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:31:14.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:14.873+0000 7fe8faa00640 1 -- 172.21.15.62:0/2715886208 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fe8f407a860 con 0x7fe904072600 2024-04-20T22:31:14.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:31:14 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/4110178247' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 15, "format": "json"}]: dispatch 2024-04-20T22:31:14.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:31:14 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/3937666721' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 16, "format": "json"}]: dispatch 2024-04-20T22:31:15.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:31:14 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/4110178247' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 15, "format": "json"}]: dispatch 2024-04-20T22:31:15.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:31:14 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/3937666721' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 16, "format": "json"}]: dispatch 2024-04-20T22:31:15.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:15.121+0000 7fe903200640 1 -- 172.21.15.62:0/2715886208 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 18, "format": "json"} v 0) v1 -- 0x7fe90410d630 con 0x7fe904072600 2024-04-20T22:31:15.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:15.122+0000 7fe8faa00640 1 -- 172.21.15.62:0/2715886208 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 18, "format": "json"}]=0 dumped fsmap epoch 18 v39) v1 ==== 107+0+4922 (secure 0 0 0) 0x7fe8f4043090 con 0x7fe904072600 2024-04-20T22:31:15.123 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:31:15.123 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":18,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-20T21:27:21.032337+0000","modified":"2024-04-20T21:37:06.018300+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14458,"mds_1":24279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14458":{"gid":14458,"name":"cephfs.smithi062.ryqblb","rank":0,"incarnation":3,"state":"up:active","state_seq":6,"addr":"172.21.15.62:6827/1147127123","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":1147127123},{"type":"v1","addr":"172.21.15.62:6827","nonce":1147127123}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14474":{"gid":14474,"name":"cephfs.smithi062.ozizwi","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":3,"addr":"172.21.15.62:6829/3494174293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":3494174293},{"type":"v1","addr":"172.21.15.62:6829","nonce":3494174293}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24279":{"gid":24279,"name":"cephfs.smithi189.dvimer","rank":1,"incarnation":8,"state":"up:active","state_seq":5,"addr":"172.21.15.189:6825/237262514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":237262514},{"type":"v1","addr":"172.21.15.189:6825","nonce":237262514}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24291":{"gid":24291,"name":"cephfs.smithi189.xbgspw","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":2,"addr":"172.21.15.189:6827/1495356258","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":1495356258},{"type":"v1","addr":"172.21.15.189:6827","nonce":1495356258}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":14458,"qdb_cluster":[24279,14458]},"id":1}]} 2024-04-20T22:31:15.123 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 18 2024-04-20T22:31:15.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:15.125+0000 7fe903200640 1 -- 172.21.15.62:0/2715886208 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe8e806ec30 msgr2=0x7fe8e80710e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:15.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:15.125+0000 7fe903200640 1 --2- 172.21.15.62:0/2715886208 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe8e806ec30 0x7fe8e80710e0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fe9040fd720 tx=0x7fe8ec006c90 comp rx=0 tx=0).stop 2024-04-20T22:31:15.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:15.125+0000 7fe903200640 1 -- 172.21.15.62:0/2715886208 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe904072600 msgr2=0x7fe9041a1570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:15.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:15.125+0000 7fe903200640 1 --2- 172.21.15.62:0/2715886208 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe904072600 0x7fe9041a1570 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7fe8f4036040 tx=0x7fe8f4004270 comp rx=0 tx=0).stop 2024-04-20T22:31:15.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:15.125+0000 7fe903200640 1 -- 172.21.15.62:0/2715886208 shutdown_connections 2024-04-20T22:31:15.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:15.125+0000 7fe903200640 1 --2- 172.21.15.62:0/2715886208 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fe8e806ec30 0x7fe8e80710e0 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:15.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:15.125+0000 7fe903200640 1 --2- 172.21.15.62:0/2715886208 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe904072f10 0x7fe9041a1ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:15.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:15.125+0000 7fe903200640 1 --2- 172.21.15.62:0/2715886208 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe904072600 0x7fe9041a1570 unknown :-1 s=CLOSED pgs=1417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:15.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:15.125+0000 7fe903200640 1 -- 172.21.15.62:0/2715886208 >> 172.21.15.62:0/2715886208 conn(0x7fe9040faf70 msgr2=0x7fe904101a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:31:15.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:15.125+0000 7fe903200640 1 -- 172.21.15.62:0/2715886208 shutdown_connections 2024-04-20T22:31:15.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:15.125+0000 7fe903200640 1 -- 172.21.15.62:0/2715886208 wait complete. 2024-04-20T22:31:15.144 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 19 2024-04-20T22:31:15.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:15.353+0000 7fb7df000640 1 -- 172.21.15.62:0/1570903944 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb7d8072f10 msgr2=0x7fb7d810b770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:15.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:15.353+0000 7fb7df000640 1 --2- 172.21.15.62:0/1570903944 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb7d8072f10 0x7fb7d810b770 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7fb7c4009880 tx=0x7fb7c402bd80 comp rx=0 tx=0).stop 2024-04-20T22:31:15.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:15.353+0000 7fb7df000640 1 -- 172.21.15.62:0/1570903944 shutdown_connections 2024-04-20T22:31:15.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:15.353+0000 7fb7df000640 1 --2- 172.21.15.62:0/1570903944 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb7d8072f10 0x7fb7d810b770 unknown :-1 s=CLOSED pgs=1418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:15.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:15.353+0000 7fb7df000640 1 --2- 172.21.15.62:0/1570903944 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7d8072600 0x7fb7d80729d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:15.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:15.353+0000 7fb7df000640 1 -- 172.21.15.62:0/1570903944 >> 172.21.15.62:0/1570903944 conn(0x7fb7d80faf70 msgr2=0x7fb7d80fd380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:31:15.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:15.353+0000 7fb7df000640 1 -- 172.21.15.62:0/1570903944 shutdown_connections 2024-04-20T22:31:15.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:15.353+0000 7fb7df000640 1 -- 172.21.15.62:0/1570903944 wait complete. 2024-04-20T22:31:15.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:15.354+0000 7fb7df000640 1 Processor -- start 2024-04-20T22:31:15.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:15.354+0000 7fb7df000640 1 -- start start 2024-04-20T22:31:15.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:15.354+0000 7fb7df000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7d8072600 0x7fb7d81974a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:15.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:15.354+0000 7fb7df000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb7d8072f10 0x7fb7d81979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:15.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:15.354+0000 7fb7df000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d8198070 con 0x7fb7d8072f10 2024-04-20T22:31:15.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:15.354+0000 7fb7df000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d819bdf0 con 0x7fb7d8072600 2024-04-20T22:31:15.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:15.355+0000 7fb7dd200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb7d8072f10 0x7fb7d81979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:15.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:15.355+0000 7fb7dd200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb7d8072f10 0x7fb7d81979e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37312/0 (socket says 172.21.15.62:37312) 2024-04-20T22:31:15.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:15.355+0000 7fb7dd200640 1 -- 172.21.15.62:0/402685852 learned_addr learned my addr 172.21.15.62:0/402685852 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:31:15.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:15.355+0000 7fb7dd200640 1 -- 172.21.15.62:0/402685852 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7d8072600 msgr2=0x7fb7d81974a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:15.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:15.355+0000 7fb7ddc00640 1 --2- 172.21.15.62:0/402685852 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7d8072600 0x7fb7d81974a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:15.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:15.355+0000 7fb7dd200640 1 --2- 172.21.15.62:0/402685852 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7d8072600 0x7fb7d81974a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:15.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:15.355+0000 7fb7dd200640 1 -- 172.21.15.62:0/402685852 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7c4009560 con 0x7fb7d8072f10 2024-04-20T22:31:15.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:15.355+0000 7fb7ddc00640 1 --2- 172.21.15.62:0/402685852 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7d8072600 0x7fb7d81974a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T22:31:15.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:15.356+0000 7fb7dd200640 1 --2- 172.21.15.62:0/402685852 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb7d8072f10 0x7fb7d81979e0 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7fb7c4036040 tx=0x7fb7c4004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:31:15.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:15.356+0000 7fb7d6000640 1 -- 172.21.15.62:0/402685852 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7c40037c0 con 0x7fb7d8072f10 2024-04-20T22:31:15.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:15.356+0000 7fb7d6000640 1 -- 172.21.15.62:0/402685852 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7c4003de0 con 0x7fb7d8072f10 2024-04-20T22:31:15.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:15.356+0000 7fb7d6000640 1 -- 172.21.15.62:0/402685852 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7c403e450 con 0x7fb7d8072f10 2024-04-20T22:31:15.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:15.356+0000 7fb7df000640 1 -- 172.21.15.62:0/402685852 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7d819c070 con 0x7fb7d8072f10 2024-04-20T22:31:15.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:15.356+0000 7fb7df000640 1 -- 172.21.15.62:0/402685852 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7d819c5d0 con 0x7fb7d8072f10 2024-04-20T22:31:15.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:15.357+0000 7fb7df000640 1 -- 172.21.15.62:0/402685852 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7d804e000 con 0x7fb7d8072f10 2024-04-20T22:31:15.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:15.358+0000 7fb7d6000640 1 -- 172.21.15.62:0/402685852 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fb7c4005060 con 0x7fb7d8072f10 2024-04-20T22:31:15.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:15.359+0000 7fb7d6000640 1 --2- 172.21.15.62:0/402685852 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb7c006ec30 0x7fb7c00710e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:15.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:15.360+0000 7fb7d6000640 1 -- 172.21.15.62:0/402685852 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fb7c40b1bf0 con 0x7fb7d8072f10 2024-04-20T22:31:15.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:15.360+0000 7fb7ddc00640 1 --2- 172.21.15.62:0/402685852 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb7c006ec30 0x7fb7c00710e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:15.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:15.360+0000 7fb7ddc00640 1 --2- 172.21.15.62:0/402685852 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb7c006ec30 0x7fb7c00710e0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc006fd0 tx=0x7fb7cc008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:31:15.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:15.364+0000 7fb7d6000640 1 -- 172.21.15.62:0/402685852 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fb7c407a940 con 0x7fb7d8072f10 2024-04-20T22:31:15.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:15.613+0000 7fb7df000640 1 -- 172.21.15.62:0/402685852 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 19, "format": "json"} v 0) v1 -- 0x7fb7d810e8d0 con 0x7fb7d8072f10 2024-04-20T22:31:15.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:15.614+0000 7fb7d6000640 1 -- 172.21.15.62:0/402685852 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 19, "format": "json"}]=0 dumped fsmap epoch 19 v39) v1 ==== 107+0+3348 (secure 0 0 0) 0x7fb7c4043d30 con 0x7fb7d8072f10 2024-04-20T22:31:15.615 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:31:15.615 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-20T21:27:21.032337+0000","modified":"2024-04-20T21:42:00.238941+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14458,"mds_1":24279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14458":{"gid":14458,"name":"cephfs.smithi062.ryqblb","rank":0,"incarnation":3,"state":"up:active","state_seq":6,"addr":"172.21.15.62:6827/1147127123","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":1147127123},{"type":"v1","addr":"172.21.15.62:6827","nonce":1147127123}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24279":{"gid":24279,"name":"cephfs.smithi189.dvimer","rank":1,"incarnation":8,"state":"up:active","state_seq":5,"addr":"172.21.15.189:6825/237262514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":237262514},{"type":"v1","addr":"172.21.15.189:6825","nonce":237262514}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":14458,"qdb_cluster":[14458,24279]},"id":1}]} 2024-04-20T22:31:15.615 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 19 2024-04-20T22:31:15.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:15.617+0000 7fb7df000640 1 -- 172.21.15.62:0/402685852 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb7c006ec30 msgr2=0x7fb7c00710e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:15.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:15.617+0000 7fb7df000640 1 --2- 172.21.15.62:0/402685852 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb7c006ec30 0x7fb7c00710e0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc006fd0 tx=0x7fb7cc008040 comp rx=0 tx=0).stop 2024-04-20T22:31:15.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:15.617+0000 7fb7df000640 1 -- 172.21.15.62:0/402685852 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb7d8072f10 msgr2=0x7fb7d81979e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:15.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:15.617+0000 7fb7df000640 1 --2- 172.21.15.62:0/402685852 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb7d8072f10 0x7fb7d81979e0 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7fb7c4036040 tx=0x7fb7c4004270 comp rx=0 tx=0).stop 2024-04-20T22:31:15.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:15.617+0000 7fb7df000640 1 -- 172.21.15.62:0/402685852 shutdown_connections 2024-04-20T22:31:15.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:15.617+0000 7fb7df000640 1 --2- 172.21.15.62:0/402685852 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb7c006ec30 0x7fb7c00710e0 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:15.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:15.617+0000 7fb7df000640 1 --2- 172.21.15.62:0/402685852 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb7d8072f10 0x7fb7d81979e0 unknown :-1 s=CLOSED pgs=1419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:15.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:15.617+0000 7fb7df000640 1 --2- 172.21.15.62:0/402685852 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7d8072600 0x7fb7d81974a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:15.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:15.617+0000 7fb7df000640 1 -- 172.21.15.62:0/402685852 >> 172.21.15.62:0/402685852 conn(0x7fb7d80faf70 msgr2=0x7fb7d8105fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:31:15.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:15.617+0000 7fb7df000640 1 -- 172.21.15.62:0/402685852 shutdown_connections 2024-04-20T22:31:15.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:15.617+0000 7fb7df000640 1 -- 172.21.15.62:0/402685852 wait complete. 2024-04-20T22:31:15.640 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 19 2024-04-20T22:31:15.640 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 20 2024-04-20T22:31:15.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:15.857+0000 7fabd5e00640 1 -- 172.21.15.62:0/2995386573 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fabd00750a0 msgr2=0x7fabd010fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:15.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:15.857+0000 7fabd5e00640 1 --2- 172.21.15.62:0/2995386573 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fabd00750a0 0x7fabd010fff0 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7fabb8009880 tx=0x7fabb802bce0 comp rx=0 tx=0).stop 2024-04-20T22:31:15.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:15.858+0000 7fabd5e00640 1 -- 172.21.15.62:0/2995386573 shutdown_connections 2024-04-20T22:31:15.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:15.858+0000 7fabd5e00640 1 --2- 172.21.15.62:0/2995386573 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fabd00750a0 0x7fabd010fff0 unknown :-1 s=CLOSED pgs=1420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:15.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:15.858+0000 7fabd5e00640 1 --2- 172.21.15.62:0/2995386573 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fabd0074790 0x7fabd0074b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:15.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:15.858+0000 7fabd5e00640 1 -- 172.21.15.62:0/2995386573 >> 172.21.15.62:0/2995386573 conn(0x7fabd00fd390 msgr2=0x7fabd00ff7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:31:15.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:15.858+0000 7fabd5e00640 1 -- 172.21.15.62:0/2995386573 shutdown_connections 2024-04-20T22:31:15.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:15.858+0000 7fabd5e00640 1 -- 172.21.15.62:0/2995386573 wait complete. 2024-04-20T22:31:15.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:15.859+0000 7fabd5e00640 1 Processor -- start 2024-04-20T22:31:15.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:15.859+0000 7fabd5e00640 1 -- start start 2024-04-20T22:31:15.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:15.859+0000 7fabd5e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fabd0074790 0x7fabd01a1670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:15.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:15.859+0000 7fabd5e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fabd00750a0 0x7fabd01a1bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:15.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:15.859+0000 7fabd5e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabd019b6e0 con 0x7fabd0074790 2024-04-20T22:31:15.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:15.859+0000 7fabd5e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabd019b850 con 0x7fabd00750a0 2024-04-20T22:31:15.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:15.859+0000 7fabcea00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fabd0074790 0x7fabd01a1670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:15.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:15.860+0000 7fabcea00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fabd0074790 0x7fabd01a1670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37318/0 (socket says 172.21.15.62:37318) 2024-04-20T22:31:15.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:15.860+0000 7fabcea00640 1 -- 172.21.15.62:0/1327736185 learned_addr learned my addr 172.21.15.62:0/1327736185 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:31:15.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:15.860+0000 7fabce000640 1 --2- 172.21.15.62:0/1327736185 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fabd00750a0 0x7fabd01a1bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:15.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:15.860+0000 7fabcea00640 1 -- 172.21.15.62:0/1327736185 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fabd00750a0 msgr2=0x7fabd01a1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:15.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:15.860+0000 7fabcea00640 1 --2- 172.21.15.62:0/1327736185 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fabd00750a0 0x7fabd01a1bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:15.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:15.860+0000 7fabcea00640 1 -- 172.21.15.62:0/1327736185 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabb8009560 con 0x7fabd0074790 2024-04-20T22:31:15.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:15.860+0000 7fabcea00640 1 --2- 172.21.15.62:0/1327736185 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fabd0074790 0x7fabd01a1670 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7fabc000c8f0 tx=0x7fabc000cdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:31:15.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:15.861+0000 7fabd4a00640 1 -- 172.21.15.62:0/1327736185 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fabc0007b80 con 0x7fabd0074790 2024-04-20T22:31:15.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:15.861+0000 7fabd4a00640 1 -- 172.21.15.62:0/1327736185 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fabc00054e0 con 0x7fabd0074790 2024-04-20T22:31:15.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:15.861+0000 7fabd5e00640 1 -- 172.21.15.62:0/1327736185 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabd019bb30 con 0x7fabd0074790 2024-04-20T22:31:15.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:15.861+0000 7fabd4a00640 1 -- 172.21.15.62:0/1327736185 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fabc000f450 con 0x7fabd0074790 2024-04-20T22:31:15.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:15.861+0000 7fabd5e00640 1 -- 172.21.15.62:0/1327736185 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabd019c090 con 0x7fabd0074790 2024-04-20T22:31:15.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:15.862+0000 7fabd5e00640 1 -- 172.21.15.62:0/1327736185 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabd004e000 con 0x7fabd0074790 2024-04-20T22:31:15.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:15.863+0000 7fabd4a00640 1 -- 172.21.15.62:0/1327736185 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fabc0004f60 con 0x7fabd0074790 2024-04-20T22:31:15.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:15.864+0000 7fabd4a00640 1 --2- 172.21.15.62:0/1327736185 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fabb406ec30 0x7fabb40710e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:15.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:15.864+0000 7fabd4a00640 1 -- 172.21.15.62:0/1327736185 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fabc008ecc0 con 0x7fabd0074790 2024-04-20T22:31:15.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:15.864+0000 7fabce000640 1 --2- 172.21.15.62:0/1327736185 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fabb406ec30 0x7fabb40710e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:15.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:15.865+0000 7fabce000640 1 --2- 172.21.15.62:0/1327736185 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fabb406ec30 0x7fabb40710e0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fabb802c220 tx=0x7fabb802cf90 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:31:15.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:15.869+0000 7fabd4a00640 1 -- 172.21.15.62:0/1327736185 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fabc0057a70 con 0x7fabd0074790 2024-04-20T22:31:15.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:31:15 smithi189 ceph-mon[57022]: pgmap v1683: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:31:15.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:31:15 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/651765538' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 17, "format": "json"}]: dispatch 2024-04-20T22:31:15.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:31:15 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/2715886208' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 18, "format": "json"}]: dispatch 2024-04-20T22:31:16.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:31:15 smithi062 ceph-mon[76322]: pgmap v1683: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:31:16.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:31:15 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/651765538' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 17, "format": "json"}]: dispatch 2024-04-20T22:31:16.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:31:15 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/2715886208' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 18, "format": "json"}]: dispatch 2024-04-20T22:31:16.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:16.121+0000 7fabd5e00640 1 -- 172.21.15.62:0/1327736185 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 20, "format": "json"} v 0) v1 -- 0x7fabd010d6d0 con 0x7fabd0074790 2024-04-20T22:31:16.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:16.123+0000 7fabd4a00640 1 -- 172.21.15.62:0/1327736185 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 20, "format": "json"}]=0 dumped fsmap epoch 20 v39) v1 ==== 107+0+3316 (secure 0 0 0) 0x7fabc001dd10 con 0x7fabd0074790 2024-04-20T22:31:16.123 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:31:16.123 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34430,"name":"cephfs.smithi062.ozizwi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6829/4253521164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":4253521164},{"type":"v1","addr":"172.21.15.62:6829","nonce":4253521164}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":44285,"name":"cephfs.smithi189.xbgspw","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6827/3996040526","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":3996040526},{"type":"v1","addr":"172.21.15.189:6827","nonce":3996040526}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":20,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-20T21:27:21.032337+0000","modified":"2024-04-20T21:42:01.559061+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-20T22:31:16.123 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 20 2024-04-20T22:31:16.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:16.125+0000 7fabd5e00640 1 -- 172.21.15.62:0/1327736185 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fabb406ec30 msgr2=0x7fabb40710e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:16.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:16.125+0000 7fabd5e00640 1 --2- 172.21.15.62:0/1327736185 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fabb406ec30 0x7fabb40710e0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fabb802c220 tx=0x7fabb802cf90 comp rx=0 tx=0).stop 2024-04-20T22:31:16.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:16.125+0000 7fabd5e00640 1 -- 172.21.15.62:0/1327736185 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fabd0074790 msgr2=0x7fabd01a1670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:16.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:16.125+0000 7fabd5e00640 1 --2- 172.21.15.62:0/1327736185 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fabd0074790 0x7fabd01a1670 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7fabc000c8f0 tx=0x7fabc000cdb0 comp rx=0 tx=0).stop 2024-04-20T22:31:16.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:16.125+0000 7fabd5e00640 1 -- 172.21.15.62:0/1327736185 shutdown_connections 2024-04-20T22:31:16.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:16.125+0000 7fabd5e00640 1 --2- 172.21.15.62:0/1327736185 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fabb406ec30 0x7fabb40710e0 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:16.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:16.125+0000 7fabd5e00640 1 --2- 172.21.15.62:0/1327736185 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fabd00750a0 0x7fabd01a1bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:16.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:16.125+0000 7fabd5e00640 1 --2- 172.21.15.62:0/1327736185 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fabd0074790 0x7fabd01a1670 unknown :-1 s=CLOSED pgs=1421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:16.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:16.125+0000 7fabd5e00640 1 -- 172.21.15.62:0/1327736185 >> 172.21.15.62:0/1327736185 conn(0x7fabd00fd390 msgr2=0x7fabd00fdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:31:16.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:16.126+0000 7fabd5e00640 1 -- 172.21.15.62:0/1327736185 shutdown_connections 2024-04-20T22:31:16.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:16.126+0000 7fabd5e00640 1 -- 172.21.15.62:0/1327736185 wait complete. 2024-04-20T22:31:16.145 DEBUG:tasks.fs:max_mds not reduced in epoch 20 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-04-20T22:31:16.145 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 20 2024-04-20T22:31:16.145 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 21 2024-04-20T22:31:16.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:16.357+0000 7f7257200640 1 -- 172.21.15.62:0/2902457928 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72580725a0 msgr2=0x7f7258072970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:16.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:16.357+0000 7f7257200640 1 --2- 172.21.15.62:0/2902457928 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72580725a0 0x7f7258072970 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7f7248009880 tx=0x7f724802bd70 comp rx=0 tx=0).stop 2024-04-20T22:31:16.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:16.357+0000 7f7257200640 1 -- 172.21.15.62:0/2902457928 shutdown_connections 2024-04-20T22:31:16.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:16.357+0000 7f7257200640 1 --2- 172.21.15.62:0/2902457928 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7258072eb0 0x7f725810b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:16.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:16.357+0000 7f7257200640 1 --2- 172.21.15.62:0/2902457928 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72580725a0 0x7f7258072970 unknown :-1 s=CLOSED pgs=1422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:16.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:16.357+0000 7f7257200640 1 -- 172.21.15.62:0/2902457928 >> 172.21.15.62:0/2902457928 conn(0x7f72580faf70 msgr2=0x7f72580fd380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:31:16.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:16.357+0000 7f7257200640 1 -- 172.21.15.62:0/2902457928 shutdown_connections 2024-04-20T22:31:16.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:16.357+0000 7f7257200640 1 -- 172.21.15.62:0/2902457928 wait complete. 2024-04-20T22:31:16.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:16.358+0000 7f7257200640 1 Processor -- start 2024-04-20T22:31:16.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:16.358+0000 7f7257200640 1 -- start start 2024-04-20T22:31:16.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:16.359+0000 7f7257200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72580725a0 0x7f7258197370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:16.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:16.359+0000 7f7257200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7258072eb0 0x7f72581978b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:16.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:16.359+0000 7f7257200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7258197f40 con 0x7f72580725a0 2024-04-20T22:31:16.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:16.359+0000 7f7257200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f725819bd10 con 0x7f7258072eb0 2024-04-20T22:31:16.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:16.359+0000 7f7255e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72580725a0 0x7f7258197370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:16.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:16.359+0000 7f7255e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72580725a0 0x7f7258197370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37328/0 (socket says 172.21.15.62:37328) 2024-04-20T22:31:16.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:16.359+0000 7f7255e00640 1 -- 172.21.15.62:0/3338760044 learned_addr learned my addr 172.21.15.62:0/3338760044 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:31:16.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:16.359+0000 7f7255e00640 1 -- 172.21.15.62:0/3338760044 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7258072eb0 msgr2=0x7f72581978b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:16.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:16.359+0000 7f7255e00640 1 --2- 172.21.15.62:0/3338760044 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7258072eb0 0x7f72581978b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:16.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:16.360+0000 7f7255e00640 1 -- 172.21.15.62:0/3338760044 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7248009560 con 0x7f72580725a0 2024-04-20T22:31:16.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:16.360+0000 7f7255e00640 1 --2- 172.21.15.62:0/3338760044 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72580725a0 0x7f7258197370 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7f7248036040 tx=0x7f724802c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:31:16.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:16.360+0000 7f724ea00640 1 -- 172.21.15.62:0/3338760044 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f72480035f0 con 0x7f72580725a0 2024-04-20T22:31:16.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:16.360+0000 7f724ea00640 1 -- 172.21.15.62:0/3338760044 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7248003c10 con 0x7f72580725a0 2024-04-20T22:31:16.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:16.360+0000 7f724ea00640 1 -- 172.21.15.62:0/3338760044 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7248005690 con 0x7f72580725a0 2024-04-20T22:31:16.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:16.360+0000 7f7257200640 1 -- 172.21.15.62:0/3338760044 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f725819bf90 con 0x7f72580725a0 2024-04-20T22:31:16.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:16.360+0000 7f7257200640 1 -- 172.21.15.62:0/3338760044 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f725819c4a0 con 0x7f72580725a0 2024-04-20T22:31:16.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:16.361+0000 7f7257200640 1 -- 172.21.15.62:0/3338760044 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f725804e000 con 0x7f72580725a0 2024-04-20T22:31:16.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:16.363+0000 7f724ea00640 1 -- 172.21.15.62:0/3338760044 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f7248004ed0 con 0x7f72580725a0 2024-04-20T22:31:16.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:16.364+0000 7f724ea00640 1 --2- 172.21.15.62:0/3338760044 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f723c06ec30 0x7f723c0710e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:16.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:16.364+0000 7f724ea00640 1 -- 172.21.15.62:0/3338760044 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f72480b1a00 con 0x7f72580725a0 2024-04-20T22:31:16.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:16.364+0000 7f7255400640 1 --2- 172.21.15.62:0/3338760044 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f723c06ec30 0x7f723c0710e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:16.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:16.365+0000 7f7255400640 1 --2- 172.21.15.62:0/3338760044 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f723c06ec30 0x7f723c0710e0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f7244005fd0 tx=0x7f7244005c70 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:31:16.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:16.369+0000 7f724ea00640 1 -- 172.21.15.62:0/3338760044 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f724807a790 con 0x7f72580725a0 2024-04-20T22:31:16.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:16.621+0000 7f7257200640 1 -- 172.21.15.62:0/3338760044 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 21, "format": "json"} v 0) v1 -- 0x7f725810e840 con 0x7f72580725a0 2024-04-20T22:31:16.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:16.623+0000 7f724ea00640 1 -- 172.21.15.62:0/3338760044 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 21, "format": "json"}]=0 dumped fsmap epoch 21 v39) v1 ==== 107+0+4874 (secure 0 0 0) 0x7f724807de50 con 0x7f72580725a0 2024-04-20T22:31:16.623 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:31:16.623 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34430,"name":"cephfs.smithi062.ozizwi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6829/4253521164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":4253521164},{"type":"v1","addr":"172.21.15.62:6829","nonce":4253521164}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":34440,"name":"cephfs.smithi062.ryqblb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6827/985180362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":985180362},{"type":"v1","addr":"172.21.15.62:6827","nonce":985180362}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":21},{"gid":44285,"name":"cephfs.smithi189.xbgspw","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6827/3996040526","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":3996040526},{"type":"v1","addr":"172.21.15.189:6827","nonce":3996040526}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":44289,"name":"cephfs.smithi189.dvimer","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6825/1743111843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":1743111843},{"type":"v1","addr":"172.21.15.189:6825","nonce":1743111843}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":20,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-20T21:27:21.032337+0000","modified":"2024-04-20T21:42:01.559061+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-20T22:31:16.623 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 21 2024-04-20T22:31:16.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:16.626+0000 7f7257200640 1 -- 172.21.15.62:0/3338760044 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f723c06ec30 msgr2=0x7f723c0710e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:16.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:16.626+0000 7f7257200640 1 --2- 172.21.15.62:0/3338760044 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f723c06ec30 0x7f723c0710e0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f7244005fd0 tx=0x7f7244005c70 comp rx=0 tx=0).stop 2024-04-20T22:31:16.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:16.626+0000 7f7257200640 1 -- 172.21.15.62:0/3338760044 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72580725a0 msgr2=0x7f7258197370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:16.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:16.626+0000 7f7257200640 1 --2- 172.21.15.62:0/3338760044 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72580725a0 0x7f7258197370 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7f7248036040 tx=0x7f724802c8e0 comp rx=0 tx=0).stop 2024-04-20T22:31:16.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:16.626+0000 7f7257200640 1 -- 172.21.15.62:0/3338760044 shutdown_connections 2024-04-20T22:31:16.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:16.626+0000 7f7257200640 1 --2- 172.21.15.62:0/3338760044 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f723c06ec30 0x7f723c0710e0 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:16.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:16.626+0000 7f7257200640 1 --2- 172.21.15.62:0/3338760044 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7258072eb0 0x7f72581978b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:16.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:16.626+0000 7f7257200640 1 --2- 172.21.15.62:0/3338760044 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72580725a0 0x7f7258197370 unknown :-1 s=CLOSED pgs=1423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:16.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:16.626+0000 7f7257200640 1 -- 172.21.15.62:0/3338760044 >> 172.21.15.62:0/3338760044 conn(0x7f72580faf70 msgr2=0x7f7258106010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:31:16.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:16.627+0000 7f7257200640 1 -- 172.21.15.62:0/3338760044 shutdown_connections 2024-04-20T22:31:16.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:16.627+0000 7f7257200640 1 -- 172.21.15.62:0/3338760044 wait complete. 2024-04-20T22:31:16.646 DEBUG:tasks.fs:max_mds not reduced in epoch 21 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-04-20T22:31:16.646 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 21 2024-04-20T22:31:16.646 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 22 2024-04-20T22:31:16.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:16.852+0000 7f7086c00640 1 -- 172.21.15.62:0/179964338 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f708010ac40 msgr2=0x7f708010b010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:16.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:16.852+0000 7f7086c00640 1 --2- 172.21.15.62:0/179964338 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f708010ac40 0x7f708010b010 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7f7070009880 tx=0x7f707002bcf0 comp rx=0 tx=0).stop 2024-04-20T22:31:16.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:16.852+0000 7f7086c00640 1 -- 172.21.15.62:0/179964338 shutdown_connections 2024-04-20T22:31:16.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:16.852+0000 7f7086c00640 1 --2- 172.21.15.62:0/179964338 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7080106090 0x7f70801064e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:16.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:16.852+0000 7f7086c00640 1 --2- 172.21.15.62:0/179964338 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f708010ac40 0x7f708010b010 unknown :-1 s=CLOSED pgs=1424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:16.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:16.852+0000 7f7086c00640 1 -- 172.21.15.62:0/179964338 >> 172.21.15.62:0/179964338 conn(0x7f7080075490 msgr2=0x7f70800778a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:31:16.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:16.852+0000 7f7086c00640 1 -- 172.21.15.62:0/179964338 shutdown_connections 2024-04-20T22:31:16.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:16.852+0000 7f7086c00640 1 -- 172.21.15.62:0/179964338 wait complete. 2024-04-20T22:31:16.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:16.853+0000 7f7086c00640 1 Processor -- start 2024-04-20T22:31:16.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:16.853+0000 7f7086c00640 1 -- start start 2024-04-20T22:31:16.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:16.854+0000 7f7086c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7080106090 0x7f708019b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:16.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:16.854+0000 7f7086c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f708010ac40 0x7f708019beb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:16.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:16.854+0000 7f7086c00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f708019c590 con 0x7f7080106090 2024-04-20T22:31:16.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:16.854+0000 7f7086c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70801a0340 con 0x7f708010ac40 2024-04-20T22:31:16.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:16.854+0000 7f707fe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7080106090 0x7f708019b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:16.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:16.854+0000 7f707fe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7080106090 0x7f708019b970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37348/0 (socket says 172.21.15.62:37348) 2024-04-20T22:31:16.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:16.854+0000 7f707fe00640 1 -- 172.21.15.62:0/3715637505 learned_addr learned my addr 172.21.15.62:0/3715637505 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:31:16.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:16.854+0000 7f7077e00640 1 --2- 172.21.15.62:0/3715637505 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f708010ac40 0x7f708019beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:16.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:16.854+0000 7f707fe00640 1 -- 172.21.15.62:0/3715637505 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f708010ac40 msgr2=0x7f708019beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:16.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:16.854+0000 7f707fe00640 1 --2- 172.21.15.62:0/3715637505 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f708010ac40 0x7f708019beb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:16.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:16.855+0000 7f707fe00640 1 -- 172.21.15.62:0/3715637505 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7070009560 con 0x7f7080106090 2024-04-20T22:31:16.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:16.855+0000 7f707fe00640 1 --2- 172.21.15.62:0/3715637505 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7080106090 0x7f708019b970 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7f7070009230 tx=0x7f7070005e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:31:16.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:16.855+0000 7f707d600640 1 -- 172.21.15.62:0/3715637505 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f70700044a0 con 0x7f7080106090 2024-04-20T22:31:16.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:16.855+0000 7f7086c00640 1 -- 172.21.15.62:0/3715637505 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70801a05c0 con 0x7f7080106090 2024-04-20T22:31:16.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:16.855+0000 7f707d600640 1 -- 172.21.15.62:0/3715637505 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7070003ab0 con 0x7f7080106090 2024-04-20T22:31:16.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:16.855+0000 7f707d600640 1 -- 172.21.15.62:0/3715637505 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7070005620 con 0x7f7080106090 2024-04-20T22:31:16.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:16.855+0000 7f7086c00640 1 -- 172.21.15.62:0/3715637505 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70801a0ba0 con 0x7f7080106090 2024-04-20T22:31:16.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:16.856+0000 7f7086c00640 1 -- 172.21.15.62:0/3715637505 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f708004e000 con 0x7f7080106090 2024-04-20T22:31:16.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:16.857+0000 7f707d600640 1 -- 172.21.15.62:0/3715637505 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f7070004e60 con 0x7f7080106090 2024-04-20T22:31:16.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:16.859+0000 7f707d600640 1 --2- 172.21.15.62:0/3715637505 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f706406ebe0 0x7f7064071090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:16.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:16.859+0000 7f707d600640 1 -- 172.21.15.62:0/3715637505 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f70700b1710 con 0x7f7080106090 2024-04-20T22:31:16.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:16.859+0000 7f7077e00640 1 --2- 172.21.15.62:0/3715637505 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f706406ebe0 0x7f7064071090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:16.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:16.860+0000 7f7077e00640 1 --2- 172.21.15.62:0/3715637505 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f706406ebe0 0x7f7064071090 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f708019cf80 tx=0x7f7068005f90 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:31:16.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:16.864+0000 7f707d600640 1 -- 172.21.15.62:0/3715637505 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f707007a4c0 con 0x7f7080106090 2024-04-20T22:31:16.874 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:31:16 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/402685852' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 19, "format": "json"}]: dispatch 2024-04-20T22:31:16.874 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:31:16 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/1327736185' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 20, "format": "json"}]: dispatch 2024-04-20T22:31:17.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:31:16 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/402685852' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 19, "format": "json"}]: dispatch 2024-04-20T22:31:17.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:31:16 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/1327736185' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 20, "format": "json"}]: dispatch 2024-04-20T22:31:17.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:17.115+0000 7f7086c00640 1 -- 172.21.15.62:0/3715637505 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 22, "format": "json"} v 0) v1 -- 0x7f7080112e60 con 0x7f7080106090 2024-04-20T22:31:17.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:17.116+0000 7f707d600640 1 -- 172.21.15.62:0/3715637505 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 22, "format": "json"}]=0 dumped fsmap epoch 22 v39) v1 ==== 107+0+4098 (secure 0 0 0) 0x7f707007db80 con 0x7f7080106090 2024-04-20T22:31:17.116 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:31:17.116 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34430,"name":"cephfs.smithi062.ozizwi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6829/4253521164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":4253521164},{"type":"v1","addr":"172.21.15.62:6829","nonce":4253521164}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":44285,"name":"cephfs.smithi189.xbgspw","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6827/3996040526","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":3996040526},{"type":"v1","addr":"172.21.15.189:6827","nonce":3996040526}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":44289,"name":"cephfs.smithi189.dvimer","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6825/1743111843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":1743111843},{"type":"v1","addr":"172.21.15.189:6825","nonce":1743111843}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":20,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-20T21:27:21.032337+0000","modified":"2024-04-20T21:42:01.559061+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-20T22:31:17.116 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 22 2024-04-20T22:31:17.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:17.119+0000 7f7086c00640 1 -- 172.21.15.62:0/3715637505 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f706406ebe0 msgr2=0x7f7064071090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:17.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:17.119+0000 7f7086c00640 1 --2- 172.21.15.62:0/3715637505 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f706406ebe0 0x7f7064071090 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f708019cf80 tx=0x7f7068005f90 comp rx=0 tx=0).stop 2024-04-20T22:31:17.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:17.119+0000 7f7086c00640 1 -- 172.21.15.62:0/3715637505 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7080106090 msgr2=0x7f708019b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:17.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:17.119+0000 7f7086c00640 1 --2- 172.21.15.62:0/3715637505 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7080106090 0x7f708019b970 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7f7070009230 tx=0x7f7070005e50 comp rx=0 tx=0).stop 2024-04-20T22:31:17.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:17.119+0000 7f7086c00640 1 -- 172.21.15.62:0/3715637505 shutdown_connections 2024-04-20T22:31:17.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:17.119+0000 7f7086c00640 1 --2- 172.21.15.62:0/3715637505 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f706406ebe0 0x7f7064071090 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:17.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:17.119+0000 7f7086c00640 1 --2- 172.21.15.62:0/3715637505 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f708010ac40 0x7f708019beb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:17.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:17.119+0000 7f7086c00640 1 --2- 172.21.15.62:0/3715637505 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7080106090 0x7f708019b970 unknown :-1 s=CLOSED pgs=1425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:17.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:17.119+0000 7f7086c00640 1 -- 172.21.15.62:0/3715637505 >> 172.21.15.62:0/3715637505 conn(0x7f7080075490 msgr2=0x7f7080075f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:31:17.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:17.119+0000 7f7086c00640 1 -- 172.21.15.62:0/3715637505 shutdown_connections 2024-04-20T22:31:17.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:17.119+0000 7f7086c00640 1 -- 172.21.15.62:0/3715637505 wait complete. 2024-04-20T22:31:17.137 DEBUG:tasks.fs:max_mds not reduced in epoch 22 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-04-20T22:31:17.137 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 22 2024-04-20T22:31:17.137 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 23 2024-04-20T22:31:17.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:17.357+0000 7f752ca00640 1 -- 172.21.15.62:0/2270092834 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7528067310 msgr2=0x7f7528067760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:17.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:17.357+0000 7f752ca00640 1 --2- 172.21.15.62:0/2270092834 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7528067310 0x7f7528067760 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7f7510009880 tx=0x7f751002bce0 comp rx=0 tx=0).stop 2024-04-20T22:31:17.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:17.357+0000 7f752ca00640 1 -- 172.21.15.62:0/2270092834 shutdown_connections 2024-04-20T22:31:17.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:17.357+0000 7f752ca00640 1 --2- 172.21.15.62:0/2270092834 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7528067310 0x7f7528067760 unknown :-1 s=CLOSED pgs=1426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:17.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:17.357+0000 7f752ca00640 1 --2- 172.21.15.62:0/2270092834 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75281055c0 0x7f7528105990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:17.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:17.357+0000 7f752ca00640 1 -- 172.21.15.62:0/2270092834 >> 172.21.15.62:0/2270092834 conn(0x7f75280743d0 msgr2=0x7f75280747d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:31:17.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:17.358+0000 7f752ca00640 1 -- 172.21.15.62:0/2270092834 shutdown_connections 2024-04-20T22:31:17.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:17.358+0000 7f752ca00640 1 -- 172.21.15.62:0/2270092834 wait complete. 2024-04-20T22:31:17.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:17.359+0000 7f752ca00640 1 Processor -- start 2024-04-20T22:31:17.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:17.359+0000 7f752ca00640 1 -- start start 2024-04-20T22:31:17.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:17.360+0000 7f752ca00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7528067310 0x7f75281974c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:17.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:17.360+0000 7f752ca00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75281055c0 0x7f7528197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:17.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:17.360+0000 7f752ca00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7528198090 con 0x7f7528067310 2024-04-20T22:31:17.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:17.360+0000 7f752ca00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f752819be60 con 0x7f75281055c0 2024-04-20T22:31:17.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:17.360+0000 7f7526800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7528067310 0x7f75281974c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:17.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:17.360+0000 7f7526800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7528067310 0x7f75281974c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37372/0 (socket says 172.21.15.62:37372) 2024-04-20T22:31:17.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:17.360+0000 7f7526800640 1 -- 172.21.15.62:0/4204706039 learned_addr learned my addr 172.21.15.62:0/4204706039 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:31:17.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:17.360+0000 7f751fe00640 1 --2- 172.21.15.62:0/4204706039 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75281055c0 0x7f7528197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:17.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:17.360+0000 7f7526800640 1 -- 172.21.15.62:0/4204706039 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75281055c0 msgr2=0x7f7528197a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:17.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:17.360+0000 7f7526800640 1 --2- 172.21.15.62:0/4204706039 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75281055c0 0x7f7528197a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:17.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:17.361+0000 7f7526800640 1 -- 172.21.15.62:0/4204706039 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7510009560 con 0x7f7528067310 2024-04-20T22:31:17.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:17.361+0000 7f7526800640 1 --2- 172.21.15.62:0/4204706039 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7528067310 0x7f75281974c0 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7f751800b420 tx=0x7f751800b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:31:17.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:17.361+0000 7f751f400640 1 -- 172.21.15.62:0/4204706039 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7518011800 con 0x7f7528067310 2024-04-20T22:31:17.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:17.361+0000 7f751f400640 1 -- 172.21.15.62:0/4204706039 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7518011e20 con 0x7f7528067310 2024-04-20T22:31:17.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:17.361+0000 7f751f400640 1 -- 172.21.15.62:0/4204706039 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f751800f2e0 con 0x7f7528067310 2024-04-20T22:31:17.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:17.361+0000 7f752ca00640 1 -- 172.21.15.62:0/4204706039 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f752819c140 con 0x7f7528067310 2024-04-20T22:31:17.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:17.361+0000 7f752ca00640 1 -- 172.21.15.62:0/4204706039 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f752819c650 con 0x7f7528067310 2024-04-20T22:31:17.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:17.363+0000 7f752ca00640 1 -- 172.21.15.62:0/4204706039 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f752804e000 con 0x7f7528067310 2024-04-20T22:31:17.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:17.364+0000 7f751f400640 1 -- 172.21.15.62:0/4204706039 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f7518010380 con 0x7f7528067310 2024-04-20T22:31:17.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:17.365+0000 7f751f400640 1 --2- 172.21.15.62:0/4204706039 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f750c06ec30 0x7f750c0710e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:17.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:17.365+0000 7f751f400640 1 -- 172.21.15.62:0/4204706039 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f751808e880 con 0x7f7528067310 2024-04-20T22:31:17.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:17.365+0000 7f751fe00640 1 --2- 172.21.15.62:0/4204706039 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f750c06ec30 0x7f750c0710e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:17.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:17.366+0000 7f751fe00640 1 --2- 172.21.15.62:0/4204706039 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f750c06ec30 0x7f750c0710e0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f7528198ad0 tx=0x7f7510000bc0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:31:17.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:17.370+0000 7f751f400640 1 -- 172.21.15.62:0/4204706039 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f7518057590 con 0x7f7528067310 2024-04-20T22:31:17.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:17.621+0000 7f752ca00640 1 -- 172.21.15.62:0/4204706039 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 23, "format": "json"} v 0) v1 -- 0x7f7528198810 con 0x7f7528067310 2024-04-20T22:31:17.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:17.623+0000 7f751f400640 1 -- 172.21.15.62:0/4204706039 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 23, "format": "json"}]=0 dumped fsmap epoch 23 v39) v1 ==== 107+0+4945 (secure 0 0 0) 0x7f7518055e60 con 0x7f7528067310 2024-04-20T22:31:17.623 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:31:17.623 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34430,"name":"cephfs.smithi062.ozizwi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6829/4253521164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":4253521164},{"type":"v1","addr":"172.21.15.62:6829","nonce":4253521164}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":34444,"name":"cephfs.smithi062.ryqblb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6827/4113916205","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":4113916205},{"type":"v1","addr":"172.21.15.62:6827","nonce":4113916205}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":23},{"gid":44285,"name":"cephfs.smithi189.xbgspw","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6827/3996040526","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":3996040526},{"type":"v1","addr":"172.21.15.189:6827","nonce":3996040526}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":44289,"name":"cephfs.smithi189.dvimer","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6825/1743111843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":1743111843},{"type":"v1","addr":"172.21.15.189:6825","nonce":1743111843}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":20,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-20T21:27:21.032337+0000","modified":"2024-04-20T21:42:01.559061+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-20T22:31:17.624 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 23 2024-04-20T22:31:17.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:17.626+0000 7f752ca00640 1 -- 172.21.15.62:0/4204706039 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f750c06ec30 msgr2=0x7f750c0710e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:17.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:17.626+0000 7f752ca00640 1 --2- 172.21.15.62:0/4204706039 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f750c06ec30 0x7f750c0710e0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f7528198ad0 tx=0x7f7510000bc0 comp rx=0 tx=0).stop 2024-04-20T22:31:17.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:17.626+0000 7f752ca00640 1 -- 172.21.15.62:0/4204706039 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7528067310 msgr2=0x7f75281974c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:17.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:17.626+0000 7f752ca00640 1 --2- 172.21.15.62:0/4204706039 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7528067310 0x7f75281974c0 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7f751800b420 tx=0x7f751800b8e0 comp rx=0 tx=0).stop 2024-04-20T22:31:17.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:17.627+0000 7f752ca00640 1 -- 172.21.15.62:0/4204706039 shutdown_connections 2024-04-20T22:31:17.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:17.627+0000 7f752ca00640 1 --2- 172.21.15.62:0/4204706039 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f750c06ec30 0x7f750c0710e0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:17.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:17.627+0000 7f752ca00640 1 --2- 172.21.15.62:0/4204706039 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75281055c0 0x7f7528197a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:17.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:17.627+0000 7f752ca00640 1 --2- 172.21.15.62:0/4204706039 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7528067310 0x7f75281974c0 unknown :-1 s=CLOSED pgs=1427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:17.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:17.627+0000 7f752ca00640 1 -- 172.21.15.62:0/4204706039 >> 172.21.15.62:0/4204706039 conn(0x7f75280743d0 msgr2=0x7f75280fceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:31:17.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:17.627+0000 7f752ca00640 1 -- 172.21.15.62:0/4204706039 shutdown_connections 2024-04-20T22:31:17.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:17.627+0000 7f752ca00640 1 -- 172.21.15.62:0/4204706039 wait complete. 2024-04-20T22:31:17.647 DEBUG:tasks.fs:max_mds not reduced in epoch 23 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-04-20T22:31:17.647 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 23 2024-04-20T22:31:17.647 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 24 2024-04-20T22:31:17.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:17.866+0000 7ff062600640 1 -- 172.21.15.62:0/2085378425 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff05c101670 msgr2=0x7ff05c101ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:17.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:17.866+0000 7ff062600640 1 --2- 172.21.15.62:0/2085378425 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff05c101670 0x7ff05c101ae0 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7ff048009880 tx=0x7ff04802bce0 comp rx=0 tx=0).stop 2024-04-20T22:31:17.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:17.867+0000 7ff062600640 1 -- 172.21.15.62:0/2085378425 shutdown_connections 2024-04-20T22:31:17.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:17.867+0000 7ff062600640 1 --2- 172.21.15.62:0/2085378425 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff05c101670 0x7ff05c101ae0 unknown :-1 s=CLOSED pgs=1428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:17.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:17.867+0000 7ff062600640 1 --2- 172.21.15.62:0/2085378425 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff05c107680 0x7ff05c107a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:17.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:17.867+0000 7ff062600640 1 -- 172.21.15.62:0/2085378425 >> 172.21.15.62:0/2085378425 conn(0x7ff05c0fd170 msgr2=0x7ff05c0ff580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:31:17.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:17.867+0000 7ff062600640 1 -- 172.21.15.62:0/2085378425 shutdown_connections 2024-04-20T22:31:17.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:17.867+0000 7ff062600640 1 -- 172.21.15.62:0/2085378425 wait complete. 2024-04-20T22:31:17.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:17.868+0000 7ff062600640 1 Processor -- start 2024-04-20T22:31:17.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:17.868+0000 7ff062600640 1 -- start start 2024-04-20T22:31:17.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:17.869+0000 7ff062600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff05c101670 0x7ff05c1972a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:17.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:17.869+0000 7ff062600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff05c107680 0x7ff05c1977e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:17.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:17.869+0000 7ff062600640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff05c197e70 con 0x7ff05c101670 2024-04-20T22:31:17.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:17.869+0000 7ff062600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff05c19bc40 con 0x7ff05c107680 2024-04-20T22:31:17.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:17.869+0000 7ff061200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff05c101670 0x7ff05c1972a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:17.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:17.869+0000 7ff061200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff05c101670 0x7ff05c1972a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37392/0 (socket says 172.21.15.62:37392) 2024-04-20T22:31:17.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:17.869+0000 7ff061200640 1 -- 172.21.15.62:0/830251765 learned_addr learned my addr 172.21.15.62:0/830251765 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:31:17.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:17.869+0000 7ff05be00640 1 --2- 172.21.15.62:0/830251765 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff05c107680 0x7ff05c1977e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:17.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:17.870+0000 7ff061200640 1 -- 172.21.15.62:0/830251765 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff05c107680 msgr2=0x7ff05c1977e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:17.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:17.870+0000 7ff061200640 1 --2- 172.21.15.62:0/830251765 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff05c107680 0x7ff05c1977e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:17.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:17.870+0000 7ff061200640 1 -- 172.21.15.62:0/830251765 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff048009560 con 0x7ff05c101670 2024-04-20T22:31:17.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:17.870+0000 7ff05be00640 1 --2- 172.21.15.62:0/830251765 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff05c107680 0x7ff05c1977e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T22:31:17.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:17.870+0000 7ff061200640 1 --2- 172.21.15.62:0/830251765 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff05c101670 0x7ff05c1972a0 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7ff05000d6f0 tx=0x7ff05000dbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:31:17.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:17.870+0000 7ff059600640 1 -- 172.21.15.62:0/830251765 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff050009a30 con 0x7ff05c101670 2024-04-20T22:31:17.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:17.870+0000 7ff059600640 1 -- 172.21.15.62:0/830251765 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff050009b90 con 0x7ff05c101670 2024-04-20T22:31:17.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:17.870+0000 7ff059600640 1 -- 172.21.15.62:0/830251765 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff050003e70 con 0x7ff05c101670 2024-04-20T22:31:17.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:17.870+0000 7ff062600640 1 -- 172.21.15.62:0/830251765 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff05c19bf20 con 0x7ff05c101670 2024-04-20T22:31:17.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:17.871+0000 7ff062600640 1 -- 172.21.15.62:0/830251765 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff05c19c500 con 0x7ff05c101670 2024-04-20T22:31:17.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:17.872+0000 7ff062600640 1 -- 172.21.15.62:0/830251765 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff05c04e000 con 0x7ff05c101670 2024-04-20T22:31:17.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:17.873+0000 7ff059600640 1 -- 172.21.15.62:0/830251765 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff050009d00 con 0x7ff05c101670 2024-04-20T22:31:17.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:17.874+0000 7ff059600640 1 --2- 172.21.15.62:0/830251765 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff04406ec30 0x7ff0440710e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:17.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:17.874+0000 7ff059600640 1 -- 172.21.15.62:0/830251765 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7ff05008e8c0 con 0x7ff05c101670 2024-04-20T22:31:17.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:17.874+0000 7ff05be00640 1 --2- 172.21.15.62:0/830251765 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff04406ec30 0x7ff0440710e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:17.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:17.875+0000 7ff05be00640 1 --2- 172.21.15.62:0/830251765 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff04406ec30 0x7ff0440710e0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7ff048036040 tx=0x7ff04802cf90 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:31:17.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:17.879+0000 7ff059600640 1 -- 172.21.15.62:0/830251765 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7ff050057650 con 0x7ff05c101670 2024-04-20T22:31:17.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:31:17 smithi189 ceph-mon[57022]: pgmap v1684: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:31:17.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:31:17 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/3338760044' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 21, "format": "json"}]: dispatch 2024-04-20T22:31:17.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:31:17 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/3715637505' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 22, "format": "json"}]: dispatch 2024-04-20T22:31:18.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:31:17 smithi062 ceph-mon[76322]: pgmap v1684: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:31:18.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:31:17 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/3338760044' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 21, "format": "json"}]: dispatch 2024-04-20T22:31:18.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:31:17 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/3715637505' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 22, "format": "json"}]: dispatch 2024-04-20T22:31:18.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:18.131+0000 7ff062600640 1 -- 172.21.15.62:0/830251765 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 24, "format": "json"} v 0) v1 -- 0x7ff05c1985f0 con 0x7ff05c101670 2024-04-20T22:31:18.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:18.132+0000 7ff059600640 1 -- 172.21.15.62:0/830251765 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 24, "format": "json"}]=0 dumped fsmap epoch 24 v39) v1 ==== 107+0+4166 (secure 0 0 0) 0x7ff05005ad10 con 0x7ff05c101670 2024-04-20T22:31:18.132 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:31:18.132 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34444,"name":"cephfs.smithi062.ryqblb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6827/4113916205","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":4113916205},{"type":"v1","addr":"172.21.15.62:6827","nonce":4113916205}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":23},{"gid":44285,"name":"cephfs.smithi189.xbgspw","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6827/3996040526","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":3996040526},{"type":"v1","addr":"172.21.15.189:6827","nonce":3996040526}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":44289,"name":"cephfs.smithi189.dvimer","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6825/1743111843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":1743111843},{"type":"v1","addr":"172.21.15.189:6825","nonce":1743111843}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":20,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-20T21:27:21.032337+0000","modified":"2024-04-20T21:42:01.559061+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-20T22:31:18.133 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 24 2024-04-20T22:31:18.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:18.135+0000 7ff062600640 1 -- 172.21.15.62:0/830251765 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff04406ec30 msgr2=0x7ff0440710e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:18.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:18.135+0000 7ff062600640 1 --2- 172.21.15.62:0/830251765 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff04406ec30 0x7ff0440710e0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7ff048036040 tx=0x7ff04802cf90 comp rx=0 tx=0).stop 2024-04-20T22:31:18.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:18.135+0000 7ff062600640 1 -- 172.21.15.62:0/830251765 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff05c101670 msgr2=0x7ff05c1972a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:18.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:18.135+0000 7ff062600640 1 --2- 172.21.15.62:0/830251765 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff05c101670 0x7ff05c1972a0 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7ff05000d6f0 tx=0x7ff05000dbb0 comp rx=0 tx=0).stop 2024-04-20T22:31:18.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:18.135+0000 7ff062600640 1 -- 172.21.15.62:0/830251765 shutdown_connections 2024-04-20T22:31:18.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:18.135+0000 7ff062600640 1 --2- 172.21.15.62:0/830251765 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff04406ec30 0x7ff0440710e0 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:18.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:18.135+0000 7ff062600640 1 --2- 172.21.15.62:0/830251765 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff05c107680 0x7ff05c1977e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:18.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:18.135+0000 7ff062600640 1 --2- 172.21.15.62:0/830251765 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff05c101670 0x7ff05c1972a0 unknown :-1 s=CLOSED pgs=1429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:18.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:18.135+0000 7ff062600640 1 -- 172.21.15.62:0/830251765 >> 172.21.15.62:0/830251765 conn(0x7ff05c0fd170 msgr2=0x7ff05c0feb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:31:18.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:18.135+0000 7ff062600640 1 -- 172.21.15.62:0/830251765 shutdown_connections 2024-04-20T22:31:18.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:18.135+0000 7ff062600640 1 -- 172.21.15.62:0/830251765 wait complete. 2024-04-20T22:31:18.155 DEBUG:tasks.fs:max_mds not reduced in epoch 24 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-04-20T22:31:18.155 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 24 2024-04-20T22:31:18.155 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 25 2024-04-20T22:31:18.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:18.357+0000 7fa962a00640 1 -- 172.21.15.62:0/554779467 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa95c106090 msgr2=0x7fa95c1064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:18.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:18.357+0000 7fa962a00640 1 --2- 172.21.15.62:0/554779467 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa95c106090 0x7fa95c1064e0 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7fa950009880 tx=0x7fa95002bce0 comp rx=0 tx=0).stop 2024-04-20T22:31:18.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:18.358+0000 7fa962a00640 1 -- 172.21.15.62:0/554779467 shutdown_connections 2024-04-20T22:31:18.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:18.358+0000 7fa962a00640 1 --2- 172.21.15.62:0/554779467 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa95c106090 0x7fa95c1064e0 unknown :-1 s=CLOSED pgs=1430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:18.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:18.358+0000 7fa962a00640 1 --2- 172.21.15.62:0/554779467 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa95c10bb60 0x7fa95c10bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:18.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:18.358+0000 7fa962a00640 1 -- 172.21.15.62:0/554779467 >> 172.21.15.62:0/554779467 conn(0x7fa95c075260 msgr2=0x7fa95c077670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:31:18.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:18.358+0000 7fa962a00640 1 -- 172.21.15.62:0/554779467 shutdown_connections 2024-04-20T22:31:18.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:18.358+0000 7fa962a00640 1 -- 172.21.15.62:0/554779467 wait complete. 2024-04-20T22:31:18.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:18.359+0000 7fa962a00640 1 Processor -- start 2024-04-20T22:31:18.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:18.359+0000 7fa962a00640 1 -- start start 2024-04-20T22:31:18.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:18.360+0000 7fa962a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa95c106090 0x7fa95c1a1620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:18.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:18.360+0000 7fa962a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa95c10bb60 0x7fa95c1a1b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:18.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:18.360+0000 7fa962a00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa95c19b690 con 0x7fa95c10bb60 2024-04-20T22:31:18.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:18.360+0000 7fa962a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa95c19b800 con 0x7fa95c106090 2024-04-20T22:31:18.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:18.360+0000 7fa960c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa95c10bb60 0x7fa95c1a1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:18.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:18.360+0000 7fa960c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa95c10bb60 0x7fa95c1a1b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37410/0 (socket says 172.21.15.62:37410) 2024-04-20T22:31:18.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:18.360+0000 7fa960c00640 1 -- 172.21.15.62:0/2906985441 learned_addr learned my addr 172.21.15.62:0/2906985441 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:31:18.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:18.360+0000 7fa960c00640 1 -- 172.21.15.62:0/2906985441 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa95c106090 msgr2=0x7fa95c1a1620 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-20T22:31:18.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:18.360+0000 7fa960c00640 1 --2- 172.21.15.62:0/2906985441 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa95c106090 0x7fa95c1a1620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:18.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:18.360+0000 7fa960c00640 1 -- 172.21.15.62:0/2906985441 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa950009560 con 0x7fa95c10bb60 2024-04-20T22:31:18.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:18.361+0000 7fa960c00640 1 --2- 172.21.15.62:0/2906985441 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa95c10bb60 0x7fa95c1a1b60 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7fa950000c00 tx=0x7fa95002c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:31:18.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:18.361+0000 7fa95a000640 1 -- 172.21.15.62:0/2906985441 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa950004520 con 0x7fa95c10bb60 2024-04-20T22:31:18.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:18.361+0000 7fa95a000640 1 -- 172.21.15.62:0/2906985441 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa950003ab0 con 0x7fa95c10bb60 2024-04-20T22:31:18.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:18.361+0000 7fa95a000640 1 -- 172.21.15.62:0/2906985441 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa950034440 con 0x7fa95c10bb60 2024-04-20T22:31:18.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:18.361+0000 7fa962a00640 1 -- 172.21.15.62:0/2906985441 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa95c19ba80 con 0x7fa95c10bb60 2024-04-20T22:31:18.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:18.361+0000 7fa962a00640 1 -- 172.21.15.62:0/2906985441 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa95c19bf30 con 0x7fa95c10bb60 2024-04-20T22:31:18.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:18.363+0000 7fa962a00640 1 -- 172.21.15.62:0/2906985441 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa95c04e000 con 0x7fa95c10bb60 2024-04-20T22:31:18.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:18.364+0000 7fa95a000640 1 -- 172.21.15.62:0/2906985441 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fa950003c20 con 0x7fa95c10bb60 2024-04-20T22:31:18.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:18.365+0000 7fa95a000640 1 --2- 172.21.15.62:0/2906985441 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa94406ec30 0x7fa9440710e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:18.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:18.365+0000 7fa95a000640 1 -- 172.21.15.62:0/2906985441 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fa9500b1210 con 0x7fa95c10bb60 2024-04-20T22:31:18.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:18.365+0000 7fa961600640 1 --2- 172.21.15.62:0/2906985441 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa94406ec30 0x7fa9440710e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:18.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:18.366+0000 7fa961600640 1 --2- 172.21.15.62:0/2906985441 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa94406ec30 0x7fa9440710e0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fa94c00b990 tx=0x7fa94c00b370 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:31:18.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:18.370+0000 7fa95a000640 1 -- 172.21.15.62:0/2906985441 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fa950079fc0 con 0x7fa95c10bb60 2024-04-20T22:31:18.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:18.620+0000 7fa962a00640 1 -- 172.21.15.62:0/2906985441 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 25, "format": "json"} v 0) v1 -- 0x7fa95c10def0 con 0x7fa95c10bb60 2024-04-20T22:31:18.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:18.622+0000 7fa95a000640 1 -- 172.21.15.62:0/2906985441 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 25, "format": "json"}]=0 dumped fsmap epoch 25 v39) v1 ==== 107+0+5013 (secure 0 0 0) 0x7fa95007d680 con 0x7fa95c10bb60 2024-04-20T22:31:18.622 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:31:18.622 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34444,"name":"cephfs.smithi062.ryqblb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6827/4113916205","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":4113916205},{"type":"v1","addr":"172.21.15.62:6827","nonce":4113916205}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":23},{"gid":34476,"name":"cephfs.smithi062.ozizwi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6829/3801975165","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":3801975165},{"type":"v1","addr":"172.21.15.62:6829","nonce":3801975165}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25},{"gid":44285,"name":"cephfs.smithi189.xbgspw","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6827/3996040526","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":3996040526},{"type":"v1","addr":"172.21.15.189:6827","nonce":3996040526}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":44289,"name":"cephfs.smithi189.dvimer","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6825/1743111843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":1743111843},{"type":"v1","addr":"172.21.15.189:6825","nonce":1743111843}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":20,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-20T21:27:21.032337+0000","modified":"2024-04-20T21:42:01.559061+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-20T22:31:18.622 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 25 2024-04-20T22:31:18.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:18.625+0000 7fa962a00640 1 -- 172.21.15.62:0/2906985441 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa94406ec30 msgr2=0x7fa9440710e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:18.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:18.625+0000 7fa962a00640 1 --2- 172.21.15.62:0/2906985441 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa94406ec30 0x7fa9440710e0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fa94c00b990 tx=0x7fa94c00b370 comp rx=0 tx=0).stop 2024-04-20T22:31:18.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:18.625+0000 7fa962a00640 1 -- 172.21.15.62:0/2906985441 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa95c10bb60 msgr2=0x7fa95c1a1b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:18.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:18.625+0000 7fa962a00640 1 --2- 172.21.15.62:0/2906985441 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa95c10bb60 0x7fa95c1a1b60 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7fa950000c00 tx=0x7fa95002c850 comp rx=0 tx=0).stop 2024-04-20T22:31:18.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:18.625+0000 7fa962a00640 1 -- 172.21.15.62:0/2906985441 shutdown_connections 2024-04-20T22:31:18.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:18.625+0000 7fa962a00640 1 --2- 172.21.15.62:0/2906985441 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa94406ec30 0x7fa9440710e0 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:18.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:18.625+0000 7fa962a00640 1 --2- 172.21.15.62:0/2906985441 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa95c10bb60 0x7fa95c1a1b60 unknown :-1 s=CLOSED pgs=1431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:18.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:18.625+0000 7fa962a00640 1 --2- 172.21.15.62:0/2906985441 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa95c106090 0x7fa95c1a1620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:18.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:18.625+0000 7fa962a00640 1 -- 172.21.15.62:0/2906985441 >> 172.21.15.62:0/2906985441 conn(0x7fa95c075260 msgr2=0x7fa95c076d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:31:18.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:18.625+0000 7fa962a00640 1 -- 172.21.15.62:0/2906985441 shutdown_connections 2024-04-20T22:31:18.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:18.625+0000 7fa962a00640 1 -- 172.21.15.62:0/2906985441 wait complete. 2024-04-20T22:31:18.644 DEBUG:tasks.fs:max_mds not reduced in epoch 25 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-04-20T22:31:18.644 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 25 2024-04-20T22:31:18.644 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 26 2024-04-20T22:31:18.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:18.859+0000 7fac5a600640 1 -- 172.21.15.62:0/928570738 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fac54072f10 msgr2=0x7fac5410ff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:18.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:18.859+0000 7fac5a600640 1 --2- 172.21.15.62:0/928570738 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fac54072f10 0x7fac5410ff80 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7fac3c009880 tx=0x7fac3c02bd80 comp rx=0 tx=0).stop 2024-04-20T22:31:18.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:18.860+0000 7fac5a600640 1 -- 172.21.15.62:0/928570738 shutdown_connections 2024-04-20T22:31:18.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:18.860+0000 7fac5a600640 1 --2- 172.21.15.62:0/928570738 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fac54072f10 0x7fac5410ff80 unknown :-1 s=CLOSED pgs=1432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:18.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:18.860+0000 7fac5a600640 1 --2- 172.21.15.62:0/928570738 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac54072600 0x7fac540729d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:18.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:18.860+0000 7fac5a600640 1 -- 172.21.15.62:0/928570738 >> 172.21.15.62:0/928570738 conn(0x7fac540fafb0 msgr2=0x7fac540fd3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:31:18.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:18.860+0000 7fac5a600640 1 -- 172.21.15.62:0/928570738 shutdown_connections 2024-04-20T22:31:18.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:18.860+0000 7fac5a600640 1 -- 172.21.15.62:0/928570738 wait complete. 2024-04-20T22:31:18.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:18.861+0000 7fac5a600640 1 Processor -- start 2024-04-20T22:31:18.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:18.861+0000 7fac5a600640 1 -- start start 2024-04-20T22:31:18.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:18.862+0000 7fac5a600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fac54072600 0x7fac5419b710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:18.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:18.862+0000 7fac5a600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac54072f10 0x7fac5419bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:18.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:18.862+0000 7fac5a600640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac5419c2a0 con 0x7fac54072600 2024-04-20T22:31:18.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:18.862+0000 7fac5a600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac5419f000 con 0x7fac54072f10 2024-04-20T22:31:18.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:18.862+0000 7fac59200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fac54072600 0x7fac5419b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:18.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:18.862+0000 7fac59200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fac54072600 0x7fac5419b710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37420/0 (socket says 172.21.15.62:37420) 2024-04-20T22:31:18.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:18.862+0000 7fac59200640 1 -- 172.21.15.62:0/2338384799 learned_addr learned my addr 172.21.15.62:0/2338384799 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:31:18.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:18.863+0000 7fac59200640 1 -- 172.21.15.62:0/2338384799 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac54072f10 msgr2=0x7fac5419bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:18.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:18.863+0000 7fac59200640 1 --2- 172.21.15.62:0/2338384799 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac54072f10 0x7fac5419bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:18.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:18.863+0000 7fac59200640 1 -- 172.21.15.62:0/2338384799 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac3c009560 con 0x7fac54072600 2024-04-20T22:31:18.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:18.863+0000 7fac59200640 1 --2- 172.21.15.62:0/2338384799 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fac54072600 0x7fac5419b710 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7fac4400cc60 tx=0x7fac4400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:31:18.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:18.863+0000 7fac51600640 1 -- 172.21.15.62:0/2338384799 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fac4400f450 con 0x7fac54072600 2024-04-20T22:31:18.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:18.863+0000 7fac51600640 1 -- 172.21.15.62:0/2338384799 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fac4400fa70 con 0x7fac54072600 2024-04-20T22:31:18.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:18.863+0000 7fac5a600640 1 -- 172.21.15.62:0/2338384799 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac5419f2e0 con 0x7fac54072600 2024-04-20T22:31:18.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:18.863+0000 7fac51600640 1 -- 172.21.15.62:0/2338384799 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fac44018b80 con 0x7fac54072600 2024-04-20T22:31:18.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:18.864+0000 7fac5a600640 1 -- 172.21.15.62:0/2338384799 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac5419f840 con 0x7fac54072600 2024-04-20T22:31:18.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:18.865+0000 7fac5a600640 1 -- 172.21.15.62:0/2338384799 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac5404e000 con 0x7fac54072600 2024-04-20T22:31:18.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:18.866+0000 7fac51600640 1 -- 172.21.15.62:0/2338384799 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fac44018d00 con 0x7fac54072600 2024-04-20T22:31:18.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:18.867+0000 7fac51600640 1 --2- 172.21.15.62:0/2338384799 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fac3806ec30 0x7fac380710e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:18.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:18.867+0000 7fac51600640 1 -- 172.21.15.62:0/2338384799 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fac4408f490 con 0x7fac54072600 2024-04-20T22:31:18.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:18.867+0000 7fac53e00640 1 --2- 172.21.15.62:0/2338384799 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fac3806ec30 0x7fac380710e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:18.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:18.868+0000 7fac53e00640 1 --2- 172.21.15.62:0/2338384799 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fac3806ec30 0x7fac380710e0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fac3c036040 tx=0x7fac3c000bc0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:31:18.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:18.872+0000 7fac51600640 1 -- 172.21.15.62:0/2338384799 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fac440581a0 con 0x7fac54072600 2024-04-20T22:31:18.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:31:18 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/4204706039' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 23, "format": "json"}]: dispatch 2024-04-20T22:31:18.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:31:18 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/830251765' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 24, "format": "json"}]: dispatch 2024-04-20T22:31:19.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:31:18 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/4204706039' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 23, "format": "json"}]: dispatch 2024-04-20T22:31:19.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:31:18 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/830251765' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 24, "format": "json"}]: dispatch 2024-04-20T22:31:19.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:19.120+0000 7fac5a600640 1 -- 172.21.15.62:0/2338384799 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 26, "format": "json"} v 0) v1 -- 0x7fac5410d670 con 0x7fac54072600 2024-04-20T22:31:19.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:19.122+0000 7fac51600640 1 -- 172.21.15.62:0/2338384799 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 26, "format": "json"}]=0 dumped fsmap epoch 26 v39) v1 ==== 107+0+4231 (secure 0 0 0) 0x7fac4405b860 con 0x7fac54072600 2024-04-20T22:31:19.122 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:31:19.122 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34444,"name":"cephfs.smithi062.ryqblb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6827/4113916205","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":4113916205},{"type":"v1","addr":"172.21.15.62:6827","nonce":4113916205}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":23},{"gid":34476,"name":"cephfs.smithi062.ozizwi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6829/3801975165","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":3801975165},{"type":"v1","addr":"172.21.15.62:6829","nonce":3801975165}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25},{"gid":44285,"name":"cephfs.smithi189.xbgspw","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6827/3996040526","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":3996040526},{"type":"v1","addr":"172.21.15.189:6827","nonce":3996040526}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":20,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-20T21:27:21.032337+0000","modified":"2024-04-20T21:42:01.559061+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-20T22:31:19.123 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 26 2024-04-20T22:31:19.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:19.124+0000 7fac5a600640 1 -- 172.21.15.62:0/2338384799 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fac3806ec30 msgr2=0x7fac380710e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:19.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:19.124+0000 7fac5a600640 1 --2- 172.21.15.62:0/2338384799 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fac3806ec30 0x7fac380710e0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fac3c036040 tx=0x7fac3c000bc0 comp rx=0 tx=0).stop 2024-04-20T22:31:19.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:19.124+0000 7fac5a600640 1 -- 172.21.15.62:0/2338384799 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fac54072600 msgr2=0x7fac5419b710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:19.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:19.124+0000 7fac5a600640 1 --2- 172.21.15.62:0/2338384799 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fac54072600 0x7fac5419b710 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7fac4400cc60 tx=0x7fac4400a620 comp rx=0 tx=0).stop 2024-04-20T22:31:19.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:19.124+0000 7fac5a600640 1 -- 172.21.15.62:0/2338384799 shutdown_connections 2024-04-20T22:31:19.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:19.124+0000 7fac5a600640 1 --2- 172.21.15.62:0/2338384799 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fac3806ec30 0x7fac380710e0 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:19.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:19.124+0000 7fac5a600640 1 --2- 172.21.15.62:0/2338384799 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac54072f10 0x7fac5419bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:19.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:19.124+0000 7fac5a600640 1 --2- 172.21.15.62:0/2338384799 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fac54072600 0x7fac5419b710 unknown :-1 s=CLOSED pgs=1433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:19.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:19.124+0000 7fac5a600640 1 -- 172.21.15.62:0/2338384799 >> 172.21.15.62:0/2338384799 conn(0x7fac540fafb0 msgr2=0x7fac54101a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:31:19.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:19.125+0000 7fac5a600640 1 -- 172.21.15.62:0/2338384799 shutdown_connections 2024-04-20T22:31:19.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:19.125+0000 7fac5a600640 1 -- 172.21.15.62:0/2338384799 wait complete. 2024-04-20T22:31:19.145 DEBUG:tasks.fs:max_mds not reduced in epoch 26 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-04-20T22:31:19.145 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 26 2024-04-20T22:31:19.146 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 27 2024-04-20T22:31:19.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:19.356+0000 7fde45a00640 1 -- 172.21.15.62:0/2629927027 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fde40101890 msgr2=0x7fde40101d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:19.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:19.356+0000 7fde45a00640 1 --2- 172.21.15.62:0/2629927027 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fde40101890 0x7fde40101d00 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7fde2c009880 tx=0x7fde2c02bce0 comp rx=0 tx=0).stop 2024-04-20T22:31:19.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:19.356+0000 7fde45a00640 1 -- 172.21.15.62:0/2629927027 shutdown_connections 2024-04-20T22:31:19.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:19.356+0000 7fde45a00640 1 --2- 172.21.15.62:0/2629927027 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fde40101890 0x7fde40101d00 unknown :-1 s=CLOSED pgs=1434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:19.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:19.356+0000 7fde45a00640 1 --2- 172.21.15.62:0/2629927027 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde401078a0 0x7fde40107c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:19.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:19.356+0000 7fde45a00640 1 -- 172.21.15.62:0/2629927027 >> 172.21.15.62:0/2629927027 conn(0x7fde400fd390 msgr2=0x7fde400ff7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:31:19.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:19.356+0000 7fde45a00640 1 -- 172.21.15.62:0/2629927027 shutdown_connections 2024-04-20T22:31:19.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:19.356+0000 7fde45a00640 1 -- 172.21.15.62:0/2629927027 wait complete. 2024-04-20T22:31:19.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:19.357+0000 7fde45a00640 1 Processor -- start 2024-04-20T22:31:19.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:19.358+0000 7fde45a00640 1 -- start start 2024-04-20T22:31:19.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:19.358+0000 7fde45a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fde40101890 0x7fde40197510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:19.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:19.358+0000 7fde45a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde401078a0 0x7fde40197a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:19.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:19.358+0000 7fde45a00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde401980e0 con 0x7fde40101890 2024-04-20T22:31:19.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:19.358+0000 7fde45a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde4019beb0 con 0x7fde401078a0 2024-04-20T22:31:19.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:19.358+0000 7fde3ea00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fde40101890 0x7fde40197510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:19.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:19.358+0000 7fde3ea00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fde40101890 0x7fde40197510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37434/0 (socket says 172.21.15.62:37434) 2024-04-20T22:31:19.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:19.358+0000 7fde3ea00640 1 -- 172.21.15.62:0/63796650 learned_addr learned my addr 172.21.15.62:0/63796650 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:31:19.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:19.359+0000 7fde3e000640 1 --2- 172.21.15.62:0/63796650 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde401078a0 0x7fde40197a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:19.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:19.359+0000 7fde3ea00640 1 -- 172.21.15.62:0/63796650 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde401078a0 msgr2=0x7fde40197a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:19.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:19.359+0000 7fde3ea00640 1 --2- 172.21.15.62:0/63796650 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde401078a0 0x7fde40197a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:19.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:19.359+0000 7fde3ea00640 1 -- 172.21.15.62:0/63796650 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde2c009560 con 0x7fde40101890 2024-04-20T22:31:19.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:19.359+0000 7fde3ea00640 1 --2- 172.21.15.62:0/63796650 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fde40101890 0x7fde40197510 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7fde3000d8d0 tx=0x7fde3000dd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:31:19.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:19.359+0000 7fde37400640 1 -- 172.21.15.62:0/63796650 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fde30009a30 con 0x7fde40101890 2024-04-20T22:31:19.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:19.359+0000 7fde37400640 1 -- 172.21.15.62:0/63796650 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fde30009b90 con 0x7fde40101890 2024-04-20T22:31:19.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:19.359+0000 7fde37400640 1 -- 172.21.15.62:0/63796650 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fde3000f680 con 0x7fde40101890 2024-04-20T22:31:19.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:19.359+0000 7fde45a00640 1 -- 172.21.15.62:0/63796650 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde4019c190 con 0x7fde40101890 2024-04-20T22:31:19.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:19.359+0000 7fde45a00640 1 -- 172.21.15.62:0/63796650 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde4019c770 con 0x7fde40101890 2024-04-20T22:31:19.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:19.361+0000 7fde45a00640 1 -- 172.21.15.62:0/63796650 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde4004e000 con 0x7fde40101890 2024-04-20T22:31:19.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:19.361+0000 7fde37400640 1 -- 172.21.15.62:0/63796650 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fde30009d00 con 0x7fde40101890 2024-04-20T22:31:19.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:19.363+0000 7fde37400640 1 --2- 172.21.15.62:0/63796650 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fde2406ec30 0x7fde240710e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:19.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:19.363+0000 7fde37400640 1 -- 172.21.15.62:0/63796650 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fde3008eca0 con 0x7fde40101890 2024-04-20T22:31:19.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:19.363+0000 7fde3e000640 1 --2- 172.21.15.62:0/63796650 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fde2406ec30 0x7fde240710e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:19.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:19.364+0000 7fde3e000640 1 --2- 172.21.15.62:0/63796650 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fde2406ec30 0x7fde240710e0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fde2c036040 tx=0x7fde2c02cf90 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:31:19.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:19.368+0000 7fde37400640 1 -- 172.21.15.62:0/63796650 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fde30057a30 con 0x7fde40101890 2024-04-20T22:31:19.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:19.650+0000 7fde45a00640 1 -- 172.21.15.62:0/63796650 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 27, "format": "json"} v 0) v1 -- 0x7fde40198860 con 0x7fde40101890 2024-04-20T22:31:19.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:19.651+0000 7fde37400640 1 -- 172.21.15.62:0/63796650 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 27, "format": "json"}]=0 dumped fsmap epoch 27 v39) v1 ==== 107+0+5081 (secure 0 0 0) 0x7fde30017090 con 0x7fde40101890 2024-04-20T22:31:19.651 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:31:19.652 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34444,"name":"cephfs.smithi062.ryqblb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6827/4113916205","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":4113916205},{"type":"v1","addr":"172.21.15.62:6827","nonce":4113916205}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":23},{"gid":34476,"name":"cephfs.smithi062.ozizwi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6829/3801975165","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":3801975165},{"type":"v1","addr":"172.21.15.62:6829","nonce":3801975165}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25},{"gid":44285,"name":"cephfs.smithi189.xbgspw","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6827/3996040526","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":3996040526},{"type":"v1","addr":"172.21.15.189:6827","nonce":3996040526}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":44727,"name":"cephfs.smithi189.dvimer","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6825/3576699232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":3576699232},{"type":"v1","addr":"172.21.15.189:6825","nonce":3576699232}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-20T21:27:21.032337+0000","modified":"2024-04-20T21:42:01.559061+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-20T22:31:19.652 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 27 2024-04-20T22:31:19.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:19.655+0000 7fde45a00640 1 -- 172.21.15.62:0/63796650 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fde2406ec30 msgr2=0x7fde240710e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:19.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:19.655+0000 7fde45a00640 1 --2- 172.21.15.62:0/63796650 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fde2406ec30 0x7fde240710e0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fde2c036040 tx=0x7fde2c02cf90 comp rx=0 tx=0).stop 2024-04-20T22:31:19.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:19.655+0000 7fde45a00640 1 -- 172.21.15.62:0/63796650 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fde40101890 msgr2=0x7fde40197510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:19.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:19.655+0000 7fde45a00640 1 --2- 172.21.15.62:0/63796650 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fde40101890 0x7fde40197510 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7fde3000d8d0 tx=0x7fde3000dd90 comp rx=0 tx=0).stop 2024-04-20T22:31:19.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:19.655+0000 7fde45a00640 1 -- 172.21.15.62:0/63796650 shutdown_connections 2024-04-20T22:31:19.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:19.655+0000 7fde45a00640 1 --2- 172.21.15.62:0/63796650 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fde2406ec30 0x7fde240710e0 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:19.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:19.655+0000 7fde45a00640 1 --2- 172.21.15.62:0/63796650 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde401078a0 0x7fde40197a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:19.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:19.655+0000 7fde45a00640 1 --2- 172.21.15.62:0/63796650 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fde40101890 0x7fde40197510 unknown :-1 s=CLOSED pgs=1435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:19.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:19.655+0000 7fde45a00640 1 -- 172.21.15.62:0/63796650 >> 172.21.15.62:0/63796650 conn(0x7fde400fd390 msgr2=0x7fde400fed80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:31:19.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:19.655+0000 7fde45a00640 1 -- 172.21.15.62:0/63796650 shutdown_connections 2024-04-20T22:31:19.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:19.656+0000 7fde45a00640 1 -- 172.21.15.62:0/63796650 wait complete. 2024-04-20T22:31:19.674 DEBUG:tasks.fs:max_mds not reduced in epoch 27 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-04-20T22:31:19.674 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 27 2024-04-20T22:31:19.674 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 28 2024-04-20T22:31:19.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:19.885+0000 7fbfea600640 1 -- 172.21.15.62:0/431614579 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbfe41016b0 msgr2=0x7fbfe4101b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:19.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:19.885+0000 7fbfea600640 1 --2- 172.21.15.62:0/431614579 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbfe41016b0 0x7fbfe4101b20 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7fbfd0009880 tx=0x7fbfd002bce0 comp rx=0 tx=0).stop 2024-04-20T22:31:19.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:19.886+0000 7fbfea600640 1 -- 172.21.15.62:0/431614579 shutdown_connections 2024-04-20T22:31:19.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:19.886+0000 7fbfea600640 1 --2- 172.21.15.62:0/431614579 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbfe41016b0 0x7fbfe4101b20 unknown :-1 s=CLOSED pgs=1436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:19.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:19.886+0000 7fbfea600640 1 --2- 172.21.15.62:0/431614579 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbfe41076c0 0x7fbfe4107a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:19.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:19.886+0000 7fbfea600640 1 -- 172.21.15.62:0/431614579 >> 172.21.15.62:0/431614579 conn(0x7fbfe40fd1b0 msgr2=0x7fbfe40ff5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:31:19.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:19.886+0000 7fbfea600640 1 -- 172.21.15.62:0/431614579 shutdown_connections 2024-04-20T22:31:19.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:19.886+0000 7fbfea600640 1 -- 172.21.15.62:0/431614579 wait complete. 2024-04-20T22:31:19.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:19.887+0000 7fbfea600640 1 Processor -- start 2024-04-20T22:31:19.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:19.887+0000 7fbfea600640 1 -- start start 2024-04-20T22:31:19.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:19.887+0000 7fbfea600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbfe41016b0 0x7fbfe4197370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:19.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:19.887+0000 7fbfea600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbfe41076c0 0x7fbfe41978b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:19.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:19.887+0000 7fbfea600640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe4197eb0 con 0x7fbfe41076c0 2024-04-20T22:31:19.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:19.887+0000 7fbfea600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe4198020 con 0x7fbfe41016b0 2024-04-20T22:31:19.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:19.888+0000 7fbfe3e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbfe41076c0 0x7fbfe41978b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:19.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:19.888+0000 7fbfe3e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbfe41076c0 0x7fbfe41978b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37464/0 (socket says 172.21.15.62:37464) 2024-04-20T22:31:19.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:19.888+0000 7fbfe3e00640 1 -- 172.21.15.62:0/465596542 learned_addr learned my addr 172.21.15.62:0/465596542 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:31:19.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:19.888+0000 7fbfe9200640 1 --2- 172.21.15.62:0/465596542 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbfe41016b0 0x7fbfe4197370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:19.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:19.888+0000 7fbfe3e00640 1 -- 172.21.15.62:0/465596542 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbfe41016b0 msgr2=0x7fbfe4197370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:19.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:19.888+0000 7fbfe3e00640 1 --2- 172.21.15.62:0/465596542 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbfe41016b0 0x7fbfe4197370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:19.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:19.888+0000 7fbfe3e00640 1 -- 172.21.15.62:0/465596542 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfd0009560 con 0x7fbfe41076c0 2024-04-20T22:31:19.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:19.889+0000 7fbfe3e00640 1 --2- 172.21.15.62:0/465596542 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbfe41076c0 0x7fbfe41978b0 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7fbfd0009230 tx=0x7fbfd0004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:31:19.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:19.889+0000 7fbfe1600640 1 -- 172.21.15.62:0/465596542 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbfd00035f0 con 0x7fbfe41076c0 2024-04-20T22:31:19.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:19.889+0000 7fbfea600640 1 -- 172.21.15.62:0/465596542 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfe419be50 con 0x7fbfe41076c0 2024-04-20T22:31:19.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:19.889+0000 7fbfe1600640 1 -- 172.21.15.62:0/465596542 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbfd0003c10 con 0x7fbfe41076c0 2024-04-20T22:31:19.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:19.889+0000 7fbfe1600640 1 -- 172.21.15.62:0/465596542 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbfd0034440 con 0x7fbfe41076c0 2024-04-20T22:31:19.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:19.889+0000 7fbfea600640 1 -- 172.21.15.62:0/465596542 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfe419c350 con 0x7fbfe41076c0 2024-04-20T22:31:19.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:19.890+0000 7fbfea600640 1 -- 172.21.15.62:0/465596542 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfe404e000 con 0x7fbfe41076c0 2024-04-20T22:31:19.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:19.891+0000 7fbfe1600640 1 -- 172.21.15.62:0/465596542 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fbfd0004fd0 con 0x7fbfe41076c0 2024-04-20T22:31:19.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:19.892+0000 7fbfe1600640 1 --2- 172.21.15.62:0/465596542 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fbfcc06ec30 0x7fbfcc0710e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:19.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:19.892+0000 7fbfe1600640 1 -- 172.21.15.62:0/465596542 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fbfd00b1210 con 0x7fbfe41076c0 2024-04-20T22:31:19.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:19.892+0000 7fbfe9200640 1 --2- 172.21.15.62:0/465596542 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fbfcc06ec30 0x7fbfcc0710e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:19.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:19.893+0000 7fbfe9200640 1 --2- 172.21.15.62:0/465596542 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fbfcc06ec30 0x7fbfcc0710e0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fbfd80058c0 tx=0x7fbfd8005850 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:31:19.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:19.897+0000 7fbfe1600640 1 -- 172.21.15.62:0/465596542 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fbfd007a040 con 0x7fbfe41076c0 2024-04-20T22:31:19.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:31:19 smithi189 ceph-mon[57022]: pgmap v1685: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:31:19.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:31:19 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/2906985441' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 25, "format": "json"}]: dispatch 2024-04-20T22:31:19.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:31:19 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/2338384799' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 26, "format": "json"}]: dispatch 2024-04-20T22:31:19.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:31:19 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:31:20.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:31:19 smithi062 ceph-mon[76322]: pgmap v1685: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:31:20.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:31:19 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/2906985441' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 25, "format": "json"}]: dispatch 2024-04-20T22:31:20.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:31:19 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/2338384799' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 26, "format": "json"}]: dispatch 2024-04-20T22:31:20.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:31:19 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:31:20.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:20.144+0000 7fbfea600640 1 -- 172.21.15.62:0/465596542 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 28, "format": "json"} v 0) v1 -- 0x7fbfe41986c0 con 0x7fbfe41076c0 2024-04-20T22:31:20.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:20.146+0000 7fbfe1600640 1 -- 172.21.15.62:0/465596542 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 28, "format": "json"}]=0 dumped fsmap epoch 28 v39) v1 ==== 107+0+4299 (secure 0 0 0) 0x7fbfd007d700 con 0x7fbfe41076c0 2024-04-20T22:31:20.146 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:31:20.146 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34444,"name":"cephfs.smithi062.ryqblb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6827/4113916205","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":4113916205},{"type":"v1","addr":"172.21.15.62:6827","nonce":4113916205}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":23},{"gid":34476,"name":"cephfs.smithi062.ozizwi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6829/3801975165","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":3801975165},{"type":"v1","addr":"172.21.15.62:6829","nonce":3801975165}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25},{"gid":44727,"name":"cephfs.smithi189.dvimer","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6825/3576699232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":3576699232},{"type":"v1","addr":"172.21.15.189:6825","nonce":3576699232}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-20T21:27:21.032337+0000","modified":"2024-04-20T21:42:01.559061+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-20T22:31:20.146 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 28 2024-04-20T22:31:20.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:20.148+0000 7fbfea600640 1 -- 172.21.15.62:0/465596542 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fbfcc06ec30 msgr2=0x7fbfcc0710e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:20.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:20.148+0000 7fbfea600640 1 --2- 172.21.15.62:0/465596542 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fbfcc06ec30 0x7fbfcc0710e0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fbfd80058c0 tx=0x7fbfd8005850 comp rx=0 tx=0).stop 2024-04-20T22:31:20.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:20.148+0000 7fbfea600640 1 -- 172.21.15.62:0/465596542 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbfe41076c0 msgr2=0x7fbfe41978b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:20.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:20.148+0000 7fbfea600640 1 --2- 172.21.15.62:0/465596542 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbfe41076c0 0x7fbfe41978b0 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7fbfd0009230 tx=0x7fbfd0004040 comp rx=0 tx=0).stop 2024-04-20T22:31:20.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:20.149+0000 7fbfea600640 1 -- 172.21.15.62:0/465596542 shutdown_connections 2024-04-20T22:31:20.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:20.149+0000 7fbfea600640 1 --2- 172.21.15.62:0/465596542 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fbfcc06ec30 0x7fbfcc0710e0 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:20.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:20.149+0000 7fbfea600640 1 --2- 172.21.15.62:0/465596542 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbfe41076c0 0x7fbfe41978b0 unknown :-1 s=CLOSED pgs=1437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:20.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:20.149+0000 7fbfea600640 1 --2- 172.21.15.62:0/465596542 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbfe41016b0 0x7fbfe4197370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:20.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:20.149+0000 7fbfea600640 1 -- 172.21.15.62:0/465596542 >> 172.21.15.62:0/465596542 conn(0x7fbfe40fd1b0 msgr2=0x7fbfe40feb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:31:20.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:20.149+0000 7fbfea600640 1 -- 172.21.15.62:0/465596542 shutdown_connections 2024-04-20T22:31:20.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:20.149+0000 7fbfea600640 1 -- 172.21.15.62:0/465596542 wait complete. 2024-04-20T22:31:20.170 DEBUG:tasks.fs:max_mds not reduced in epoch 28 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-04-20T22:31:20.170 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 28 2024-04-20T22:31:20.170 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 29 2024-04-20T22:31:20.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:20.377+0000 7fbb26000640 1 -- 172.21.15.62:0/840246909 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbb20103af0 msgr2=0x7fbb20067310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:20.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:20.377+0000 7fbb26000640 1 --2- 172.21.15.62:0/840246909 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbb20103af0 0x7fbb20067310 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7fbb0c009880 tx=0x7fbb0c02bce0 comp rx=0 tx=0).stop 2024-04-20T22:31:20.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:20.377+0000 7fbb26000640 1 -- 172.21.15.62:0/840246909 shutdown_connections 2024-04-20T22:31:20.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:20.377+0000 7fbb26000640 1 --2- 172.21.15.62:0/840246909 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbb20103af0 0x7fbb20067310 unknown :-1 s=CLOSED pgs=1438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:20.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:20.377+0000 7fbb26000640 1 --2- 172.21.15.62:0/840246909 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb20103150 0x7fbb20103520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:20.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:20.377+0000 7fbb26000640 1 -- 172.21.15.62:0/840246909 >> 172.21.15.62:0/840246909 conn(0x7fbb200743d0 msgr2=0x7fbb200747d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:31:20.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:20.377+0000 7fbb26000640 1 -- 172.21.15.62:0/840246909 shutdown_connections 2024-04-20T22:31:20.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:20.377+0000 7fbb26000640 1 -- 172.21.15.62:0/840246909 wait complete. 2024-04-20T22:31:20.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:20.378+0000 7fbb26000640 1 Processor -- start 2024-04-20T22:31:20.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:20.378+0000 7fbb26000640 1 -- start start 2024-04-20T22:31:20.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:20.378+0000 7fbb26000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbb20103150 0x7fbb201a1600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:20.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:20.378+0000 7fbb26000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb20103af0 0x7fbb201a1b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:20.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:20.378+0000 7fbb26000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb2019b670 con 0x7fbb20103150 2024-04-20T22:31:20.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:20.378+0000 7fbb26000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb2019b7e0 con 0x7fbb20103af0 2024-04-20T22:31:20.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:20.379+0000 7fbb24c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbb20103150 0x7fbb201a1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:20.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:20.379+0000 7fbb24c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbb20103150 0x7fbb201a1600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37470/0 (socket says 172.21.15.62:37470) 2024-04-20T22:31:20.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:20.379+0000 7fbb24c00640 1 -- 172.21.15.62:0/3840147796 learned_addr learned my addr 172.21.15.62:0/3840147796 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:31:20.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:20.379+0000 7fbb1fe00640 1 --2- 172.21.15.62:0/3840147796 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb20103af0 0x7fbb201a1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:20.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:20.379+0000 7fbb24c00640 1 -- 172.21.15.62:0/3840147796 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb20103af0 msgr2=0x7fbb201a1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:20.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:20.379+0000 7fbb24c00640 1 --2- 172.21.15.62:0/3840147796 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb20103af0 0x7fbb201a1b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:20.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:20.379+0000 7fbb24c00640 1 -- 172.21.15.62:0/3840147796 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb0c009560 con 0x7fbb20103150 2024-04-20T22:31:20.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:20.379+0000 7fbb24c00640 1 --2- 172.21.15.62:0/3840147796 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbb20103150 0x7fbb201a1600 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7fbb1000ba50 tx=0x7fbb1000bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:31:20.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:20.380+0000 7fbb1d600640 1 -- 172.21.15.62:0/3840147796 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbb1000cb30 con 0x7fbb20103150 2024-04-20T22:31:20.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:20.380+0000 7fbb1d600640 1 -- 172.21.15.62:0/3840147796 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbb1000cc90 con 0x7fbb20103150 2024-04-20T22:31:20.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:20.380+0000 7fbb1d600640 1 -- 172.21.15.62:0/3840147796 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbb10007740 con 0x7fbb20103150 2024-04-20T22:31:20.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:20.380+0000 7fbb26000640 1 -- 172.21.15.62:0/3840147796 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb2019bac0 con 0x7fbb20103150 2024-04-20T22:31:20.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:20.380+0000 7fbb26000640 1 -- 172.21.15.62:0/3840147796 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb2019c0a0 con 0x7fbb20103150 2024-04-20T22:31:20.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:20.381+0000 7fbb26000640 1 -- 172.21.15.62:0/3840147796 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb2004e000 con 0x7fbb20103150 2024-04-20T22:31:20.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:20.382+0000 7fbb1d600640 1 -- 172.21.15.62:0/3840147796 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fbb10014b90 con 0x7fbb20103150 2024-04-20T22:31:20.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:20.383+0000 7fbb1d600640 1 --2- 172.21.15.62:0/3840147796 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fbb0406ec30 0x7fbb040710e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:20.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:20.383+0000 7fbb1d600640 1 -- 172.21.15.62:0/3840147796 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fbb1008e800 con 0x7fbb20103150 2024-04-20T22:31:20.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:20.383+0000 7fbb1fe00640 1 --2- 172.21.15.62:0/3840147796 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fbb0406ec30 0x7fbb040710e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:20.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:20.384+0000 7fbb1fe00640 1 --2- 172.21.15.62:0/3840147796 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fbb0406ec30 0x7fbb040710e0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fbb0c02c220 tx=0x7fbb0c02cf90 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:31:20.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:20.389+0000 7fbb1d600640 1 -- 172.21.15.62:0/3840147796 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fbb1005b170 con 0x7fbb20103150 2024-04-20T22:31:20.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:20.636+0000 7fbb26000640 1 -- 172.21.15.62:0/3840147796 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 29, "format": "json"} v 0) v1 -- 0x7fbb2019c4f0 con 0x7fbb20103150 2024-04-20T22:31:20.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:20.638+0000 7fbb1d600640 1 -- 172.21.15.62:0/3840147796 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 29, "format": "json"}]=0 dumped fsmap epoch 29 v39) v1 ==== 107+0+5146 (secure 0 0 0) 0x7fbb1005ad00 con 0x7fbb20103150 2024-04-20T22:31:20.638 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:31:20.638 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34444,"name":"cephfs.smithi062.ryqblb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6827/4113916205","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":4113916205},{"type":"v1","addr":"172.21.15.62:6827","nonce":4113916205}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":23},{"gid":34476,"name":"cephfs.smithi062.ozizwi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6829/3801975165","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":3801975165},{"type":"v1","addr":"172.21.15.62:6829","nonce":3801975165}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25},{"gid":44727,"name":"cephfs.smithi189.dvimer","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6825/3576699232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":3576699232},{"type":"v1","addr":"172.21.15.189:6825","nonce":3576699232}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":27},{"gid":45075,"name":"cephfs.smithi189.xbgspw","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6827/212905792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":212905792},{"type":"v1","addr":"172.21.15.189:6827","nonce":212905792}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":20,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-20T21:27:21.032337+0000","modified":"2024-04-20T21:42:01.559061+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-20T22:31:20.638 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 29 2024-04-20T22:31:20.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:20.641+0000 7fbb26000640 1 -- 172.21.15.62:0/3840147796 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fbb0406ec30 msgr2=0x7fbb040710e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:20.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:20.641+0000 7fbb26000640 1 --2- 172.21.15.62:0/3840147796 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fbb0406ec30 0x7fbb040710e0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fbb0c02c220 tx=0x7fbb0c02cf90 comp rx=0 tx=0).stop 2024-04-20T22:31:20.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:20.641+0000 7fbb26000640 1 -- 172.21.15.62:0/3840147796 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbb20103150 msgr2=0x7fbb201a1600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:20.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:20.641+0000 7fbb26000640 1 --2- 172.21.15.62:0/3840147796 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbb20103150 0x7fbb201a1600 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7fbb1000ba50 tx=0x7fbb1000bf10 comp rx=0 tx=0).stop 2024-04-20T22:31:20.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:20.641+0000 7fbb26000640 1 -- 172.21.15.62:0/3840147796 shutdown_connections 2024-04-20T22:31:20.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:20.641+0000 7fbb26000640 1 --2- 172.21.15.62:0/3840147796 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fbb0406ec30 0x7fbb040710e0 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:20.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:20.641+0000 7fbb26000640 1 --2- 172.21.15.62:0/3840147796 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb20103af0 0x7fbb201a1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:20.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:20.641+0000 7fbb26000640 1 --2- 172.21.15.62:0/3840147796 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbb20103150 0x7fbb201a1600 unknown :-1 s=CLOSED pgs=1439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:20.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:20.641+0000 7fbb26000640 1 -- 172.21.15.62:0/3840147796 >> 172.21.15.62:0/3840147796 conn(0x7fbb200743d0 msgr2=0x7fbb200fcc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:31:20.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:20.641+0000 7fbb26000640 1 -- 172.21.15.62:0/3840147796 shutdown_connections 2024-04-20T22:31:20.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:20.641+0000 7fbb26000640 1 -- 172.21.15.62:0/3840147796 wait complete. 2024-04-20T22:31:20.663 DEBUG:tasks.fs:max_mds not reduced in epoch 29 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-04-20T22:31:20.663 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 29 2024-04-20T22:31:20.663 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 30 2024-04-20T22:31:20.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:20.872+0000 7faadb200640 1 -- 172.21.15.62:0/3914258537 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faadc0ff470 msgr2=0x7faadc0ff8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:20.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:20.872+0000 7faadb200640 1 --2- 172.21.15.62:0/3914258537 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faadc0ff470 0x7faadc0ff8e0 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7faac8009880 tx=0x7faac802bce0 comp rx=0 tx=0).stop 2024-04-20T22:31:20.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:20.872+0000 7faadb200640 1 -- 172.21.15.62:0/3914258537 shutdown_connections 2024-04-20T22:31:20.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:20.872+0000 7faadb200640 1 --2- 172.21.15.62:0/3914258537 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faadc0ff470 0x7faadc0ff8e0 unknown :-1 s=CLOSED pgs=1440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:20.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:20.872+0000 7faadb200640 1 --2- 172.21.15.62:0/3914258537 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faadc105480 0x7faadc105850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:20.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:20.872+0000 7faadb200640 1 -- 172.21.15.62:0/3914258537 >> 172.21.15.62:0/3914258537 conn(0x7faadc0faf70 msgr2=0x7faadc0fd380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:31:20.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:20.872+0000 7faadb200640 1 -- 172.21.15.62:0/3914258537 shutdown_connections 2024-04-20T22:31:20.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:20.872+0000 7faadb200640 1 -- 172.21.15.62:0/3914258537 wait complete. 2024-04-20T22:31:20.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:20.873+0000 7faadb200640 1 Processor -- start 2024-04-20T22:31:20.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:20.873+0000 7faadb200640 1 -- start start 2024-04-20T22:31:20.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:20.873+0000 7faadb200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faadc0ff470 0x7faadc197350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:20.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:20.873+0000 7faadb200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faadc105480 0x7faadc197890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:20.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:20.873+0000 7faadb200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faadc197f20 con 0x7faadc0ff470 2024-04-20T22:31:20.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:20.873+0000 7faadb200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faadc19bcf0 con 0x7faadc105480 2024-04-20T22:31:20.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:20.874+0000 7faad9e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faadc0ff470 0x7faadc197350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:20.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:20.874+0000 7faad9e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faadc0ff470 0x7faadc197350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37488/0 (socket says 172.21.15.62:37488) 2024-04-20T22:31:20.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:20.874+0000 7faad9e00640 1 -- 172.21.15.62:0/1487961031 learned_addr learned my addr 172.21.15.62:0/1487961031 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:31:20.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:20.874+0000 7faad9e00640 1 -- 172.21.15.62:0/1487961031 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faadc105480 msgr2=0x7faadc197890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:20.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:20.874+0000 7faad9400640 1 --2- 172.21.15.62:0/1487961031 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faadc105480 0x7faadc197890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:20.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:20.874+0000 7faad9e00640 1 --2- 172.21.15.62:0/1487961031 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faadc105480 0x7faadc197890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:20.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:20.874+0000 7faad9e00640 1 -- 172.21.15.62:0/1487961031 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faac8009560 con 0x7faadc0ff470 2024-04-20T22:31:20.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:20.874+0000 7faad9e00640 1 --2- 172.21.15.62:0/1487961031 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faadc0ff470 0x7faadc197350 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7faacc00d6f0 tx=0x7faacc00dbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:31:20.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:20.875+0000 7faad2a00640 1 -- 172.21.15.62:0/1487961031 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faacc009a30 con 0x7faadc0ff470 2024-04-20T22:31:20.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:20.875+0000 7faad2a00640 1 -- 172.21.15.62:0/1487961031 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7faacc0054e0 con 0x7faadc0ff470 2024-04-20T22:31:20.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:20.875+0000 7faadb200640 1 -- 172.21.15.62:0/1487961031 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faadc19bfd0 con 0x7faadc0ff470 2024-04-20T22:31:20.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:20.875+0000 7faad2a00640 1 -- 172.21.15.62:0/1487961031 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faacc010460 con 0x7faadc0ff470 2024-04-20T22:31:20.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:20.875+0000 7faadb200640 1 -- 172.21.15.62:0/1487961031 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faadc19c530 con 0x7faadc0ff470 2024-04-20T22:31:20.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:20.876+0000 7faadb200640 1 -- 172.21.15.62:0/1487961031 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faadc04e000 con 0x7faadc0ff470 2024-04-20T22:31:20.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:20.877+0000 7faad2a00640 1 -- 172.21.15.62:0/1487961031 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7faacc009b90 con 0x7faadc0ff470 2024-04-20T22:31:20.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:20.878+0000 7faad2a00640 1 --2- 172.21.15.62:0/1487961031 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7faac006ec30 0x7faac00710e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:20.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:20.878+0000 7faad2a00640 1 -- 172.21.15.62:0/1487961031 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7faacc08e9f0 con 0x7faadc0ff470 2024-04-20T22:31:20.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:20.879+0000 7faad9400640 1 --2- 172.21.15.62:0/1487961031 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7faac006ec30 0x7faac00710e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:20.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:20.879+0000 7faad9400640 1 --2- 172.21.15.62:0/1487961031 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7faac006ec30 0x7faac00710e0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7faac8036040 tx=0x7faac802cf90 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:31:20.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:20.883+0000 7faad2a00640 1 -- 172.21.15.62:0/1487961031 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7faacc057780 con 0x7faadc0ff470 2024-04-20T22:31:20.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:31:20 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/63796650' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 27, "format": "json"}]: dispatch 2024-04-20T22:31:20.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:31:20 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/465596542' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 28, "format": "json"}]: dispatch 2024-04-20T22:31:21.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:31:20 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/63796650' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 27, "format": "json"}]: dispatch 2024-04-20T22:31:21.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:31:20 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/465596542' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 28, "format": "json"}]: dispatch 2024-04-20T22:31:21.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:21.133+0000 7faadb200640 1 -- 172.21.15.62:0/1487961031 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 30, "format": "json"} v 0) v1 -- 0x7faadc1986a0 con 0x7faadc0ff470 2024-04-20T22:31:21.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:21.134+0000 7faad2a00640 1 -- 172.21.15.62:0/1487961031 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 30, "format": "json"}]=0 dumped fsmap epoch 30 v39) v1 ==== 107+0+5145 (secure 0 0 0) 0x7faacc017090 con 0x7faadc0ff470 2024-04-20T22:31:21.135 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:31:21.135 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34444,"name":"cephfs.smithi062.ryqblb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6827/4113916205","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":4113916205},{"type":"v1","addr":"172.21.15.62:6827","nonce":4113916205}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":23},{"gid":34476,"name":"cephfs.smithi062.ozizwi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6829/3801975165","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":3801975165},{"type":"v1","addr":"172.21.15.62:6829","nonce":3801975165}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25},{"gid":44727,"name":"cephfs.smithi189.dvimer","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6825/3576699232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":3576699232},{"type":"v1","addr":"172.21.15.189:6825","nonce":3576699232}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":27},{"gid":45075,"name":"cephfs.smithi189.xbgspw","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6827/212905792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":212905792},{"type":"v1","addr":"172.21.15.189:6827","nonce":212905792}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-20T21:27:21.032337+0000","modified":"2024-04-20T22:28:20.949553+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-20T22:31:21.135 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 30 2024-04-20T22:31:21.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:21.138+0000 7faadb200640 1 -- 172.21.15.62:0/1487961031 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7faac006ec30 msgr2=0x7faac00710e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:21.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:21.138+0000 7faadb200640 1 --2- 172.21.15.62:0/1487961031 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7faac006ec30 0x7faac00710e0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7faac8036040 tx=0x7faac802cf90 comp rx=0 tx=0).stop 2024-04-20T22:31:21.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:21.138+0000 7faadb200640 1 -- 172.21.15.62:0/1487961031 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faadc0ff470 msgr2=0x7faadc197350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:21.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:21.138+0000 7faadb200640 1 --2- 172.21.15.62:0/1487961031 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faadc0ff470 0x7faadc197350 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7faacc00d6f0 tx=0x7faacc00dbb0 comp rx=0 tx=0).stop 2024-04-20T22:31:21.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:21.138+0000 7faadb200640 1 -- 172.21.15.62:0/1487961031 shutdown_connections 2024-04-20T22:31:21.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:21.138+0000 7faadb200640 1 --2- 172.21.15.62:0/1487961031 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7faac006ec30 0x7faac00710e0 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:21.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:21.138+0000 7faadb200640 1 --2- 172.21.15.62:0/1487961031 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faadc105480 0x7faadc197890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:21.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:21.138+0000 7faadb200640 1 --2- 172.21.15.62:0/1487961031 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faadc0ff470 0x7faadc197350 unknown :-1 s=CLOSED pgs=1441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:21.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:21.138+0000 7faadb200640 1 -- 172.21.15.62:0/1487961031 >> 172.21.15.62:0/1487961031 conn(0x7faadc0faf70 msgr2=0x7faadc0fb850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:31:21.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:21.138+0000 7faadb200640 1 -- 172.21.15.62:0/1487961031 shutdown_connections 2024-04-20T22:31:21.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:21.138+0000 7faadb200640 1 -- 172.21.15.62:0/1487961031 wait complete. 2024-04-20T22:31:21.154 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 30 2024-04-20T22:31:21.155 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 31 2024-04-20T22:31:21.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:21.363+0000 7f22ee800640 1 -- 172.21.15.62:0/394103177 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f22e80ff4b0 msgr2=0x7f22e80ff920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:21.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:21.363+0000 7f22ee800640 1 --2- 172.21.15.62:0/394103177 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f22e80ff4b0 0x7f22e80ff920 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7f22d4009880 tx=0x7f22d402bce0 comp rx=0 tx=0).stop 2024-04-20T22:31:21.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:21.363+0000 7f22ee800640 1 -- 172.21.15.62:0/394103177 shutdown_connections 2024-04-20T22:31:21.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:21.363+0000 7f22ee800640 1 --2- 172.21.15.62:0/394103177 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f22e80ff4b0 0x7f22e80ff920 unknown :-1 s=CLOSED pgs=1442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:21.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:21.363+0000 7f22ee800640 1 --2- 172.21.15.62:0/394103177 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f22e81054c0 0x7f22e8105890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:21.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:21.363+0000 7f22ee800640 1 -- 172.21.15.62:0/394103177 >> 172.21.15.62:0/394103177 conn(0x7f22e80fafb0 msgr2=0x7f22e80fd3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:31:21.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:21.363+0000 7f22ee800640 1 -- 172.21.15.62:0/394103177 shutdown_connections 2024-04-20T22:31:21.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:21.363+0000 7f22ee800640 1 -- 172.21.15.62:0/394103177 wait complete. 2024-04-20T22:31:21.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:21.364+0000 7f22ee800640 1 Processor -- start 2024-04-20T22:31:21.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:21.364+0000 7f22ee800640 1 -- start start 2024-04-20T22:31:21.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:21.364+0000 7f22ee800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f22e80ff4b0 0x7f22e81952a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:21.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:21.364+0000 7f22ee800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f22e81054c0 0x7f22e81957e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:21.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:21.364+0000 7f22ee800640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22e8195e70 con 0x7f22e81054c0 2024-04-20T22:31:21.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:21.364+0000 7f22ee800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22e8199c40 con 0x7f22e80ff4b0 2024-04-20T22:31:21.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:21.365+0000 7f22eca00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f22e81054c0 0x7f22e81957e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:21.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:21.365+0000 7f22eca00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f22e81054c0 0x7f22e81957e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53290/0 (socket says 172.21.15.62:53290) 2024-04-20T22:31:21.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:21.365+0000 7f22eca00640 1 -- 172.21.15.62:0/1553731626 learned_addr learned my addr 172.21.15.62:0/1553731626 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:31:21.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:21.365+0000 7f22ed400640 1 --2- 172.21.15.62:0/1553731626 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f22e80ff4b0 0x7f22e81952a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:21.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:21.365+0000 7f22eca00640 1 -- 172.21.15.62:0/1553731626 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f22e80ff4b0 msgr2=0x7f22e81952a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:21.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:21.365+0000 7f22eca00640 1 --2- 172.21.15.62:0/1553731626 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f22e80ff4b0 0x7f22e81952a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:21.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:21.365+0000 7f22eca00640 1 -- 172.21.15.62:0/1553731626 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22d4009560 con 0x7f22e81054c0 2024-04-20T22:31:21.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:21.366+0000 7f22eca00640 1 --2- 172.21.15.62:0/1553731626 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f22e81054c0 0x7f22e81957e0 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7f22d4000c00 tx=0x7f22d4004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:31:21.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:21.366+0000 7f22e6000640 1 -- 172.21.15.62:0/1553731626 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f22d40037c0 con 0x7f22e81054c0 2024-04-20T22:31:21.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:21.366+0000 7f22e6000640 1 -- 172.21.15.62:0/1553731626 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f22d4003de0 con 0x7f22e81054c0 2024-04-20T22:31:21.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:21.366+0000 7f22ee800640 1 -- 172.21.15.62:0/1553731626 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22e8199ec0 con 0x7f22e81054c0 2024-04-20T22:31:21.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:21.366+0000 7f22e6000640 1 -- 172.21.15.62:0/1553731626 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f22d4034460 con 0x7f22e81054c0 2024-04-20T22:31:21.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:21.366+0000 7f22ee800640 1 -- 172.21.15.62:0/1553731626 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22e819a3c0 con 0x7f22e81054c0 2024-04-20T22:31:21.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:21.367+0000 7f22ee800640 1 -- 172.21.15.62:0/1553731626 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22e804e000 con 0x7f22e81054c0 2024-04-20T22:31:21.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:21.368+0000 7f22e6000640 1 -- 172.21.15.62:0/1553731626 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f22d4003920 con 0x7f22e81054c0 2024-04-20T22:31:21.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:21.369+0000 7f22e6000640 1 --2- 172.21.15.62:0/1553731626 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f22d006ec30 0x7f22d00710e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:21.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:21.369+0000 7f22e6000640 1 -- 172.21.15.62:0/1553731626 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f22d40b1340 con 0x7f22e81054c0 2024-04-20T22:31:21.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:21.369+0000 7f22ed400640 1 --2- 172.21.15.62:0/1553731626 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f22d006ec30 0x7f22d00710e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:21.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:21.370+0000 7f22ed400640 1 --2- 172.21.15.62:0/1553731626 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f22d006ec30 0x7f22d00710e0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f22dc009890 tx=0x7f22dc008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:31:21.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:21.374+0000 7f22e6000640 1 -- 172.21.15.62:0/1553731626 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f22d407a190 con 0x7f22e81054c0 2024-04-20T22:31:21.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:21.623+0000 7f22ee800640 1 -- 172.21.15.62:0/1553731626 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 31, "format": "json"} v 0) v1 -- 0x7f22e81079b0 con 0x7f22e81054c0 2024-04-20T22:31:21.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:21.624+0000 7f22e6000640 1 -- 172.21.15.62:0/1553731626 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 31, "format": "json"}]=0 dumped fsmap epoch 31 v39) v1 ==== 107+0+5236 (secure 0 0 0) 0x7f22d407d850 con 0x7f22e81054c0 2024-04-20T22:31:21.625 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:31:21.625 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":44727,"name":"cephfs.smithi189.dvimer","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6825/3576699232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":3576699232},{"type":"v1","addr":"172.21.15.189:6825","nonce":3576699232}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":27},{"gid":45075,"name":"cephfs.smithi189.xbgspw","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6827/212905792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":212905792},{"type":"v1","addr":"172.21.15.189:6827","nonce":212905792}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":31,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-20T21:27:21.032337+0000","modified":"2024-04-20T22:28:21.949822+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34444,"mds_1":34476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34444":{"gid":34444,"name":"cephfs.smithi062.ryqblb","rank":0,"incarnation":31,"state":"up:replay","state_seq":1,"addr":"172.21.15.62:6827/4113916205","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":4113916205},{"type":"v1","addr":"172.21.15.62:6827","nonce":4113916205}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34476":{"gid":34476,"name":"cephfs.smithi062.ozizwi","rank":1,"incarnation":31,"state":"up:replay","state_seq":1,"addr":"172.21.15.62:6829/3801975165","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":3801975165},{"type":"v1","addr":"172.21.15.62:6829","nonce":3801975165}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-20T22:31:21.625 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 31 2024-04-20T22:31:21.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:21.627+0000 7f22ee800640 1 -- 172.21.15.62:0/1553731626 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f22d006ec30 msgr2=0x7f22d00710e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:21.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:21.627+0000 7f22ee800640 1 --2- 172.21.15.62:0/1553731626 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f22d006ec30 0x7f22d00710e0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f22dc009890 tx=0x7f22dc008040 comp rx=0 tx=0).stop 2024-04-20T22:31:21.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:21.627+0000 7f22ee800640 1 -- 172.21.15.62:0/1553731626 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f22e81054c0 msgr2=0x7f22e81957e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:21.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:21.627+0000 7f22ee800640 1 --2- 172.21.15.62:0/1553731626 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f22e81054c0 0x7f22e81957e0 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7f22d4000c00 tx=0x7f22d4004270 comp rx=0 tx=0).stop 2024-04-20T22:31:21.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:21.627+0000 7f22ee800640 1 -- 172.21.15.62:0/1553731626 shutdown_connections 2024-04-20T22:31:21.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:21.627+0000 7f22ee800640 1 --2- 172.21.15.62:0/1553731626 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f22d006ec30 0x7f22d00710e0 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:21.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:21.627+0000 7f22ee800640 1 --2- 172.21.15.62:0/1553731626 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f22e81054c0 0x7f22e81957e0 unknown :-1 s=CLOSED pgs=1443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:21.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:21.627+0000 7f22ee800640 1 --2- 172.21.15.62:0/1553731626 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f22e80ff4b0 0x7f22e81952a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:21.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:21.627+0000 7f22ee800640 1 -- 172.21.15.62:0/1553731626 >> 172.21.15.62:0/1553731626 conn(0x7f22e80fafb0 msgr2=0x7f22e80fba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:31:21.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:21.628+0000 7f22ee800640 1 -- 172.21.15.62:0/1553731626 shutdown_connections 2024-04-20T22:31:21.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:21.628+0000 7f22ee800640 1 -- 172.21.15.62:0/1553731626 wait complete. 2024-04-20T22:31:21.647 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 31 2024-04-20T22:31:21.647 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 32 2024-04-20T22:31:21.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:21.861+0000 7f84e9a00640 1 -- 172.21.15.62:0/1701560058 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f84e40ff4b0 msgr2=0x7f84e40ff920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:21.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:21.861+0000 7f84e9a00640 1 --2- 172.21.15.62:0/1701560058 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f84e40ff4b0 0x7f84e40ff920 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7f84cc009880 tx=0x7f84cc02bcf0 comp rx=0 tx=0).stop 2024-04-20T22:31:21.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:21.861+0000 7f84e9a00640 1 -- 172.21.15.62:0/1701560058 shutdown_connections 2024-04-20T22:31:21.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:21.861+0000 7f84e9a00640 1 --2- 172.21.15.62:0/1701560058 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f84e40ff4b0 0x7f84e40ff920 unknown :-1 s=CLOSED pgs=1444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:21.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:21.861+0000 7f84e9a00640 1 --2- 172.21.15.62:0/1701560058 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f84e41054c0 0x7f84e4105890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:21.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:21.861+0000 7f84e9a00640 1 -- 172.21.15.62:0/1701560058 >> 172.21.15.62:0/1701560058 conn(0x7f84e40fafb0 msgr2=0x7f84e40fd3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:31:21.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:21.861+0000 7f84e9a00640 1 -- 172.21.15.62:0/1701560058 shutdown_connections 2024-04-20T22:31:21.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:21.861+0000 7f84e9a00640 1 -- 172.21.15.62:0/1701560058 wait complete. 2024-04-20T22:31:21.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:21.862+0000 7f84e9a00640 1 Processor -- start 2024-04-20T22:31:21.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:21.863+0000 7f84e9a00640 1 -- start start 2024-04-20T22:31:21.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:21.863+0000 7f84e9a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f84e40ff4b0 0x7f84e4197370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:21.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:21.863+0000 7f84e9a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f84e41054c0 0x7f84e41978b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:21.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:21.863+0000 7f84e9a00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84e4197eb0 con 0x7f84e41054c0 2024-04-20T22:31:21.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:21.863+0000 7f84e9a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84e4198020 con 0x7f84e40ff4b0 2024-04-20T22:31:21.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:21.863+0000 7f84dbe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f84e41054c0 0x7f84e41978b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:21.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:21.863+0000 7f84dbe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f84e41054c0 0x7f84e41978b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53320/0 (socket says 172.21.15.62:53320) 2024-04-20T22:31:21.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:21.863+0000 7f84dbe00640 1 -- 172.21.15.62:0/2755068501 learned_addr learned my addr 172.21.15.62:0/2755068501 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:31:21.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:21.864+0000 7f84e3e00640 1 --2- 172.21.15.62:0/2755068501 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f84e40ff4b0 0x7f84e4197370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:21.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:21.864+0000 7f84dbe00640 1 -- 172.21.15.62:0/2755068501 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f84e40ff4b0 msgr2=0x7f84e4197370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:21.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:21.864+0000 7f84dbe00640 1 --2- 172.21.15.62:0/2755068501 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f84e40ff4b0 0x7f84e4197370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:21.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:21.864+0000 7f84dbe00640 1 -- 172.21.15.62:0/2755068501 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84cc009560 con 0x7f84e41054c0 2024-04-20T22:31:21.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:21.864+0000 7f84e3e00640 1 --2- 172.21.15.62:0/2755068501 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f84e40ff4b0 0x7f84e4197370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T22:31:21.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:21.864+0000 7f84dbe00640 1 --2- 172.21.15.62:0/2755068501 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f84e41054c0 0x7f84e41978b0 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7f84cc036040 tx=0x7f84cc004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:31:21.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:21.864+0000 7f84e1600640 1 -- 172.21.15.62:0/2755068501 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f84cc0035f0 con 0x7f84e41054c0 2024-04-20T22:31:21.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:21.865+0000 7f84e1600640 1 -- 172.21.15.62:0/2755068501 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f84cc003c10 con 0x7f84e41054c0 2024-04-20T22:31:21.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:21.865+0000 7f84e1600640 1 -- 172.21.15.62:0/2755068501 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f84cc034440 con 0x7f84e41054c0 2024-04-20T22:31:21.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:21.865+0000 7f84e9a00640 1 -- 172.21.15.62:0/2755068501 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84e419be50 con 0x7f84e41054c0 2024-04-20T22:31:21.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:21.865+0000 7f84e9a00640 1 -- 172.21.15.62:0/2755068501 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84e419c3b0 con 0x7f84e41054c0 2024-04-20T22:31:21.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:21.866+0000 7f84e9a00640 1 -- 172.21.15.62:0/2755068501 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84e404e000 con 0x7f84e41054c0 2024-04-20T22:31:21.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:21.867+0000 7f84e1600640 1 -- 172.21.15.62:0/2755068501 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f84cc005060 con 0x7f84e41054c0 2024-04-20T22:31:21.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:21.868+0000 7f84e1600640 1 --2- 172.21.15.62:0/2755068501 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f84c806ec30 0x7f84c80710e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:21.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:21.868+0000 7f84e1600640 1 -- 172.21.15.62:0/2755068501 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f84cc0b13a0 con 0x7f84e41054c0 2024-04-20T22:31:21.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:21.868+0000 7f84e3e00640 1 --2- 172.21.15.62:0/2755068501 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f84c806ec30 0x7f84c80710e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:21.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:21.869+0000 7f84e3e00640 1 --2- 172.21.15.62:0/2755068501 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f84c806ec30 0x7f84c80710e0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f84d400baa0 tx=0x7f84d4005d90 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:31:21.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:21.873+0000 7f84e1600640 1 -- 172.21.15.62:0/2755068501 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f84cc07a0b0 con 0x7f84e41054c0 2024-04-20T22:31:21.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:31:21 smithi189 ceph-mon[57022]: pgmap v1686: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:31:21.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:31:21 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/3840147796' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 29, "format": "json"}]: dispatch 2024-04-20T22:31:21.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:31:21 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/1487961031' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 30, "format": "json"}]: dispatch 2024-04-20T22:31:22.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:31:21 smithi062 ceph-mon[76322]: pgmap v1686: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:31:22.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:31:21 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/3840147796' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 29, "format": "json"}]: dispatch 2024-04-20T22:31:22.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:31:21 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/1487961031' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 30, "format": "json"}]: dispatch 2024-04-20T22:31:22.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:22.123+0000 7f84e9a00640 1 -- 172.21.15.62:0/2755068501 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 32, "format": "json"} v 0) v1 -- 0x7f84e41986c0 con 0x7f84e41054c0 2024-04-20T22:31:22.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:22.125+0000 7f84e1600640 1 -- 172.21.15.62:0/2755068501 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 32, "format": "json"}]=0 dumped fsmap epoch 32 v39) v1 ==== 107+0+5235 (secure 0 0 0) 0x7f84cc07d770 con 0x7f84e41054c0 2024-04-20T22:31:22.125 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:31:22.125 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":44727,"name":"cephfs.smithi189.dvimer","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6825/3576699232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":3576699232},{"type":"v1","addr":"172.21.15.189:6825","nonce":3576699232}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":27},{"gid":45075,"name":"cephfs.smithi189.xbgspw","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6827/212905792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":212905792},{"type":"v1","addr":"172.21.15.189:6827","nonce":212905792}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-20T21:27:21.032337+0000","modified":"2024-04-20T22:28:21.959073+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34444,"mds_1":34476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34444":{"gid":34444,"name":"cephfs.smithi062.ryqblb","rank":0,"incarnation":31,"state":"up:replay","state_seq":1,"addr":"172.21.15.62:6827/4113916205","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":4113916205},{"type":"v1","addr":"172.21.15.62:6827","nonce":4113916205}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34476":{"gid":34476,"name":"cephfs.smithi062.ozizwi","rank":1,"incarnation":31,"state":"up:replay","state_seq":1,"addr":"172.21.15.62:6829/3801975165","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":3801975165},{"type":"v1","addr":"172.21.15.62:6829","nonce":3801975165}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-20T22:31:22.125 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 32 2024-04-20T22:31:22.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:22.128+0000 7f84e9a00640 1 -- 172.21.15.62:0/2755068501 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f84c806ec30 msgr2=0x7f84c80710e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:22.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:22.128+0000 7f84e9a00640 1 --2- 172.21.15.62:0/2755068501 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f84c806ec30 0x7f84c80710e0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f84d400baa0 tx=0x7f84d4005d90 comp rx=0 tx=0).stop 2024-04-20T22:31:22.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:22.128+0000 7f84e9a00640 1 -- 172.21.15.62:0/2755068501 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f84e41054c0 msgr2=0x7f84e41978b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:22.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:22.128+0000 7f84e9a00640 1 --2- 172.21.15.62:0/2755068501 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f84e41054c0 0x7f84e41978b0 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7f84cc036040 tx=0x7f84cc004270 comp rx=0 tx=0).stop 2024-04-20T22:31:22.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:22.128+0000 7f84e9a00640 1 -- 172.21.15.62:0/2755068501 shutdown_connections 2024-04-20T22:31:22.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:22.128+0000 7f84e9a00640 1 --2- 172.21.15.62:0/2755068501 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f84c806ec30 0x7f84c80710e0 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:22.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:22.128+0000 7f84e9a00640 1 --2- 172.21.15.62:0/2755068501 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f84e41054c0 0x7f84e41978b0 unknown :-1 s=CLOSED pgs=1445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:22.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:22.128+0000 7f84e9a00640 1 --2- 172.21.15.62:0/2755068501 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f84e40ff4b0 0x7f84e4197370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:22.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:22.128+0000 7f84e9a00640 1 -- 172.21.15.62:0/2755068501 >> 172.21.15.62:0/2755068501 conn(0x7f84e40fafb0 msgr2=0x7f84e40724f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:31:22.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:22.128+0000 7f84e9a00640 1 -- 172.21.15.62:0/2755068501 shutdown_connections 2024-04-20T22:31:22.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:22.128+0000 7f84e9a00640 1 -- 172.21.15.62:0/2755068501 wait complete. 2024-04-20T22:31:22.147 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 33 2024-04-20T22:31:22.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:22.357+0000 7fa2db200640 1 -- 172.21.15.62:0/3074985745 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa2dc1076c0 msgr2=0x7fa2dc107a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:22.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:22.357+0000 7fa2db200640 1 --2- 172.21.15.62:0/3074985745 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa2dc1076c0 0x7fa2dc107a90 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7fa2cc009880 tx=0x7fa2cc02bce0 comp rx=0 tx=0).stop 2024-04-20T22:31:22.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:22.358+0000 7fa2db200640 1 -- 172.21.15.62:0/3074985745 shutdown_connections 2024-04-20T22:31:22.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:22.358+0000 7fa2db200640 1 --2- 172.21.15.62:0/3074985745 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2dc1016b0 0x7fa2dc101b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:22.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:22.358+0000 7fa2db200640 1 --2- 172.21.15.62:0/3074985745 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa2dc1076c0 0x7fa2dc107a90 unknown :-1 s=CLOSED pgs=1446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:22.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:22.358+0000 7fa2db200640 1 -- 172.21.15.62:0/3074985745 >> 172.21.15.62:0/3074985745 conn(0x7fa2dc0fd1b0 msgr2=0x7fa2dc0ff5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:31:22.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:22.358+0000 7fa2db200640 1 -- 172.21.15.62:0/3074985745 shutdown_connections 2024-04-20T22:31:22.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:22.358+0000 7fa2db200640 1 -- 172.21.15.62:0/3074985745 wait complete. 2024-04-20T22:31:22.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:22.359+0000 7fa2db200640 1 Processor -- start 2024-04-20T22:31:22.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:22.359+0000 7fa2db200640 1 -- start start 2024-04-20T22:31:22.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:22.360+0000 7fa2db200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa2dc1016b0 0x7fa2dc197310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:22.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:22.360+0000 7fa2db200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2dc1076c0 0x7fa2dc197850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:22.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:22.360+0000 7fa2db200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2dc197ee0 con 0x7fa2dc1016b0 2024-04-20T22:31:22.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:22.360+0000 7fa2db200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2dc19bcb0 con 0x7fa2dc1076c0 2024-04-20T22:31:22.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:22.360+0000 7fa2d9e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa2dc1016b0 0x7fa2dc197310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:22.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:22.360+0000 7fa2d9e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa2dc1016b0 0x7fa2dc197310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53348/0 (socket says 172.21.15.62:53348) 2024-04-20T22:31:22.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:22.360+0000 7fa2d9e00640 1 -- 172.21.15.62:0/1087000665 learned_addr learned my addr 172.21.15.62:0/1087000665 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:31:22.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:22.360+0000 7fa2d3e00640 1 --2- 172.21.15.62:0/1087000665 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2dc1076c0 0x7fa2dc197850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:22.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:22.361+0000 7fa2d9e00640 1 -- 172.21.15.62:0/1087000665 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2dc1076c0 msgr2=0x7fa2dc197850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:22.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:22.361+0000 7fa2d9e00640 1 --2- 172.21.15.62:0/1087000665 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2dc1076c0 0x7fa2dc197850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:22.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:22.361+0000 7fa2d9e00640 1 -- 172.21.15.62:0/1087000665 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2cc009560 con 0x7fa2dc1016b0 2024-04-20T22:31:22.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:22.361+0000 7fa2d3e00640 1 --2- 172.21.15.62:0/1087000665 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2dc1076c0 0x7fa2dc197850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T22:31:22.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:22.361+0000 7fa2d9e00640 1 --2- 172.21.15.62:0/1087000665 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa2dc1016b0 0x7fa2dc197310 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7fa2cc009230 tx=0x7fa2cc004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:31:22.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:22.361+0000 7fa2d2a00640 1 -- 172.21.15.62:0/1087000665 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa2cc0037c0 con 0x7fa2dc1016b0 2024-04-20T22:31:22.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:22.361+0000 7fa2d2a00640 1 -- 172.21.15.62:0/1087000665 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa2cc003de0 con 0x7fa2dc1016b0 2024-04-20T22:31:22.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:22.361+0000 7fa2d2a00640 1 -- 172.21.15.62:0/1087000665 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa2cc034440 con 0x7fa2dc1016b0 2024-04-20T22:31:22.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:22.361+0000 7fa2db200640 1 -- 172.21.15.62:0/1087000665 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2dc19bf30 con 0x7fa2dc1016b0 2024-04-20T22:31:22.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:22.362+0000 7fa2db200640 1 -- 172.21.15.62:0/1087000665 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2dc19c460 con 0x7fa2dc1016b0 2024-04-20T22:31:22.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:22.363+0000 7fa2db200640 1 -- 172.21.15.62:0/1087000665 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2dc04e000 con 0x7fa2dc1016b0 2024-04-20T22:31:22.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:22.364+0000 7fa2d2a00640 1 -- 172.21.15.62:0/1087000665 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fa2cc004fb0 con 0x7fa2dc1016b0 2024-04-20T22:31:22.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:22.365+0000 7fa2d2a00640 1 --2- 172.21.15.62:0/1087000665 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa2c006ec30 0x7fa2c00710e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:22.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:22.365+0000 7fa2d2a00640 1 -- 172.21.15.62:0/1087000665 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fa2cc0b13f0 con 0x7fa2dc1016b0 2024-04-20T22:31:22.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:22.365+0000 7fa2d3e00640 1 --2- 172.21.15.62:0/1087000665 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa2c006ec30 0x7fa2c00710e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:22.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:22.366+0000 7fa2d3e00640 1 --2- 172.21.15.62:0/1087000665 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa2c006ec30 0x7fa2c00710e0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fa2dc198920 tx=0x7fa2c4005d30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:31:22.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:22.370+0000 7fa2d2a00640 1 -- 172.21.15.62:0/1087000665 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fa2cc07a100 con 0x7fa2dc1016b0 2024-04-20T22:31:22.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:22.645+0000 7fa2db200640 1 -- 172.21.15.62:0/1087000665 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 33, "format": "json"} v 0) v1 -- 0x7fa2dc0008d0 con 0x7fa2dc1016b0 2024-04-20T22:31:22.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:22.645+0000 7fa2d2a00640 1 -- 172.21.15.62:0/1087000665 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 33, "format": "json"}]=0 dumped fsmap epoch 33 v39) v1 ==== 107+0+5238 (secure 0 0 0) 0x7fa2cc07d7c0 con 0x7fa2dc1016b0 2024-04-20T22:31:22.646 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:31:22.646 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":44727,"name":"cephfs.smithi189.dvimer","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6825/3576699232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":3576699232},{"type":"v1","addr":"172.21.15.189:6825","nonce":3576699232}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":27},{"gid":45075,"name":"cephfs.smithi189.xbgspw","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6827/212905792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":212905792},{"type":"v1","addr":"172.21.15.189:6827","nonce":212905792}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":33,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-20T21:27:21.032337+0000","modified":"2024-04-20T22:28:27.926039+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34444,"mds_1":34476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34444":{"gid":34444,"name":"cephfs.smithi062.ryqblb","rank":0,"incarnation":31,"state":"up:replay","state_seq":1,"addr":"172.21.15.62:6827/4113916205","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":4113916205},{"type":"v1","addr":"172.21.15.62:6827","nonce":4113916205}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34476":{"gid":34476,"name":"cephfs.smithi062.ozizwi","rank":1,"incarnation":31,"state":"up:resolve","state_seq":688,"addr":"172.21.15.62:6829/3801975165","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":3801975165},{"type":"v1","addr":"172.21.15.62:6829","nonce":3801975165}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-20T22:31:22.646 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 33 2024-04-20T22:31:22.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:22.648+0000 7fa2db200640 1 -- 172.21.15.62:0/1087000665 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa2c006ec30 msgr2=0x7fa2c00710e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:22.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:22.648+0000 7fa2db200640 1 --2- 172.21.15.62:0/1087000665 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa2c006ec30 0x7fa2c00710e0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fa2dc198920 tx=0x7fa2c4005d30 comp rx=0 tx=0).stop 2024-04-20T22:31:22.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:22.648+0000 7fa2db200640 1 -- 172.21.15.62:0/1087000665 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa2dc1016b0 msgr2=0x7fa2dc197310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:22.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:22.648+0000 7fa2db200640 1 --2- 172.21.15.62:0/1087000665 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa2dc1016b0 0x7fa2dc197310 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7fa2cc009230 tx=0x7fa2cc004270 comp rx=0 tx=0).stop 2024-04-20T22:31:22.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:22.648+0000 7fa2db200640 1 -- 172.21.15.62:0/1087000665 shutdown_connections 2024-04-20T22:31:22.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:22.648+0000 7fa2db200640 1 --2- 172.21.15.62:0/1087000665 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa2c006ec30 0x7fa2c00710e0 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:22.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:22.648+0000 7fa2db200640 1 --2- 172.21.15.62:0/1087000665 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2dc1076c0 0x7fa2dc197850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:22.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:22.648+0000 7fa2db200640 1 --2- 172.21.15.62:0/1087000665 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa2dc1016b0 0x7fa2dc197310 unknown :-1 s=CLOSED pgs=1447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:22.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:22.648+0000 7fa2db200640 1 -- 172.21.15.62:0/1087000665 >> 172.21.15.62:0/1087000665 conn(0x7fa2dc0fd1b0 msgr2=0x7fa2dc0feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:31:22.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:22.649+0000 7fa2db200640 1 -- 172.21.15.62:0/1087000665 shutdown_connections 2024-04-20T22:31:22.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:22.649+0000 7fa2db200640 1 -- 172.21.15.62:0/1087000665 wait complete. 2024-04-20T22:31:22.665 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:31:22 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/1553731626' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 31, "format": "json"}]: dispatch 2024-04-20T22:31:22.665 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:31:22 smithi062 ceph-mon[76322]: pgmap v1687: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:31:22.665 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:31:22 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/2755068501' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 32, "format": "json"}]: dispatch 2024-04-20T22:31:22.669 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 34 2024-04-20T22:31:22.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:22.875+0000 7f1ebbe00640 1 -- 172.21.15.62:0/2947281267 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1eb41078a0 msgr2=0x7f1eb4107c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:22.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:22.875+0000 7f1ebbe00640 1 --2- 172.21.15.62:0/2947281267 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1eb41078a0 0x7f1eb4107c70 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f1ea4009c20 tx=0x7f1ea402c1b0 comp rx=0 tx=0).stop 2024-04-20T22:31:22.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:22.875+0000 7f1ebbe00640 1 -- 172.21.15.62:0/2947281267 shutdown_connections 2024-04-20T22:31:22.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:22.875+0000 7f1ebbe00640 1 --2- 172.21.15.62:0/2947281267 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1eb4101890 0x7f1eb4101d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:22.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:22.875+0000 7f1ebbe00640 1 --2- 172.21.15.62:0/2947281267 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1eb41078a0 0x7f1eb4107c70 unknown :-1 s=CLOSED pgs=1448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:22.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:22.875+0000 7f1ebbe00640 1 -- 172.21.15.62:0/2947281267 >> 172.21.15.62:0/2947281267 conn(0x7f1eb40fd390 msgr2=0x7f1eb40ff7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:31:22.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:22.875+0000 7f1ebbe00640 1 -- 172.21.15.62:0/2947281267 shutdown_connections 2024-04-20T22:31:22.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:22.875+0000 7f1ebbe00640 1 -- 172.21.15.62:0/2947281267 wait complete. 2024-04-20T22:31:22.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:22.876+0000 7f1ebbe00640 1 Processor -- start 2024-04-20T22:31:22.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:22.877+0000 7f1ebbe00640 1 -- start start 2024-04-20T22:31:22.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:22.877+0000 7f1ebbe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1eb4101890 0x7f1eb41975b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:22.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:22.877+0000 7f1ebbe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1eb41078a0 0x7f1eb4197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:22.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:22.877+0000 7f1ebbe00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb4198180 con 0x7f1eb4101890 2024-04-20T22:31:22.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:22.877+0000 7f1ebbe00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb419bf50 con 0x7f1eb41078a0 2024-04-20T22:31:22.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:22.877+0000 7f1eb9400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1eb4101890 0x7f1eb41975b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:22.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:22.877+0000 7f1eb9400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1eb4101890 0x7f1eb41975b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53364/0 (socket says 172.21.15.62:53364) 2024-04-20T22:31:22.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:22.877+0000 7f1eb9400640 1 -- 172.21.15.62:0/180613651 learned_addr learned my addr 172.21.15.62:0/180613651 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:31:22.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:22.878+0000 7f1eb8a00640 1 --2- 172.21.15.62:0/180613651 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1eb41078a0 0x7f1eb4197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:22.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:22.878+0000 7f1eb9400640 1 -- 172.21.15.62:0/180613651 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1eb41078a0 msgr2=0x7f1eb4197af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:22.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:22.878+0000 7f1eb9400640 1 --2- 172.21.15.62:0/180613651 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1eb41078a0 0x7f1eb4197af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:22.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:22.878+0000 7f1eb9400640 1 -- 172.21.15.62:0/180613651 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ea4009900 con 0x7f1eb4101890 2024-04-20T22:31:22.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:22.878+0000 7f1eb8a00640 1 --2- 172.21.15.62:0/180613651 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1eb41078a0 0x7f1eb4197af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T22:31:22.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:22.878+0000 7f1eb9400640 1 --2- 172.21.15.62:0/180613651 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1eb4101890 0x7f1eb41975b0 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7f1ea4032b60 tx=0x7f1ea4032b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:31:22.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:22.878+0000 7f1eb2000640 1 -- 172.21.15.62:0/180613651 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ea4034820 con 0x7f1eb4101890 2024-04-20T22:31:22.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:22.878+0000 7f1eb2000640 1 -- 172.21.15.62:0/180613651 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1ea4003e30 con 0x7f1eb4101890 2024-04-20T22:31:22.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:22.879+0000 7f1ebbe00640 1 -- 172.21.15.62:0/180613651 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1eb419c1d0 con 0x7f1eb4101890 2024-04-20T22:31:22.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:22.879+0000 7f1eb2000640 1 -- 172.21.15.62:0/180613651 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ea403f440 con 0x7f1eb4101890 2024-04-20T22:31:22.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:22.879+0000 7f1ebbe00640 1 -- 172.21.15.62:0/180613651 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1eb419c750 con 0x7f1eb4101890 2024-04-20T22:31:22.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:22.880+0000 7f1ebbe00640 1 -- 172.21.15.62:0/180613651 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1eb404e000 con 0x7f1eb4101890 2024-04-20T22:31:22.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:22.881+0000 7f1eb2000640 1 -- 172.21.15.62:0/180613651 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f1ea403b040 con 0x7f1eb4101890 2024-04-20T22:31:22.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:22.882+0000 7f1eb2000640 1 --2- 172.21.15.62:0/180613651 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f1e9806ec30 0x7f1e980710e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:22.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:22.882+0000 7f1eb2000640 1 -- 172.21.15.62:0/180613651 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f1ea40b1c90 con 0x7f1eb4101890 2024-04-20T22:31:22.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:22.882+0000 7f1eb8a00640 1 --2- 172.21.15.62:0/180613651 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f1e9806ec30 0x7f1e980710e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:22.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:22.883+0000 7f1eb8a00640 1 --2- 172.21.15.62:0/180613651 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f1e9806ec30 0x7f1e980710e0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f1ea0006fd0 tx=0x7f1ea0008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:31:22.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:22.887+0000 7f1eb2000640 1 -- 172.21.15.62:0/180613651 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f1ea407a9a0 con 0x7f1eb4101890 2024-04-20T22:31:22.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:31:22 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/1553731626' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 31, "format": "json"}]: dispatch 2024-04-20T22:31:22.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:31:22 smithi189 ceph-mon[57022]: pgmap v1687: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:31:22.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:31:22 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/2755068501' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 32, "format": "json"}]: dispatch 2024-04-20T22:31:23.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:23.137+0000 7f1ebbe00640 1 -- 172.21.15.62:0/180613651 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 34, "format": "json"} v 0) v1 -- 0x7f1eb4198900 con 0x7f1eb4101890 2024-04-20T22:31:23.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:23.139+0000 7f1eb2000640 1 -- 172.21.15.62:0/180613651 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 34, "format": "json"}]=0 dumped fsmap epoch 34 v39) v1 ==== 107+0+5241 (secure 0 0 0) 0x7f1ea407e060 con 0x7f1eb4101890 2024-04-20T22:31:23.139 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:31:23.139 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":44727,"name":"cephfs.smithi189.dvimer","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6825/3576699232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":3576699232},{"type":"v1","addr":"172.21.15.189:6825","nonce":3576699232}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":27},{"gid":45075,"name":"cephfs.smithi189.xbgspw","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6827/212905792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":212905792},{"type":"v1","addr":"172.21.15.189:6827","nonce":212905792}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":34,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-20T21:27:21.032337+0000","modified":"2024-04-20T22:28:29.867604+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34444,"mds_1":34476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34444":{"gid":34444,"name":"cephfs.smithi062.ryqblb","rank":0,"incarnation":31,"state":"up:resolve","state_seq":696,"addr":"172.21.15.62:6827/4113916205","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":4113916205},{"type":"v1","addr":"172.21.15.62:6827","nonce":4113916205}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34476":{"gid":34476,"name":"cephfs.smithi062.ozizwi","rank":1,"incarnation":31,"state":"up:resolve","state_seq":688,"addr":"172.21.15.62:6829/3801975165","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":3801975165},{"type":"v1","addr":"172.21.15.62:6829","nonce":3801975165}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-20T22:31:23.139 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 34 2024-04-20T22:31:23.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:23.142+0000 7f1ebbe00640 1 -- 172.21.15.62:0/180613651 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f1e9806ec30 msgr2=0x7f1e980710e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:23.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:23.142+0000 7f1ebbe00640 1 --2- 172.21.15.62:0/180613651 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f1e9806ec30 0x7f1e980710e0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f1ea0006fd0 tx=0x7f1ea0008040 comp rx=0 tx=0).stop 2024-04-20T22:31:23.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:23.142+0000 7f1ebbe00640 1 -- 172.21.15.62:0/180613651 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1eb4101890 msgr2=0x7f1eb41975b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:23.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:23.142+0000 7f1ebbe00640 1 --2- 172.21.15.62:0/180613651 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1eb4101890 0x7f1eb41975b0 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7f1ea4032b60 tx=0x7f1ea4032b90 comp rx=0 tx=0).stop 2024-04-20T22:31:23.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:23.142+0000 7f1ebbe00640 1 -- 172.21.15.62:0/180613651 shutdown_connections 2024-04-20T22:31:23.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:23.142+0000 7f1ebbe00640 1 --2- 172.21.15.62:0/180613651 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f1e9806ec30 0x7f1e980710e0 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:23.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:23.142+0000 7f1ebbe00640 1 --2- 172.21.15.62:0/180613651 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1eb41078a0 0x7f1eb4197af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:23.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:23.142+0000 7f1ebbe00640 1 --2- 172.21.15.62:0/180613651 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1eb4101890 0x7f1eb41975b0 unknown :-1 s=CLOSED pgs=1449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:23.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:23.142+0000 7f1ebbe00640 1 -- 172.21.15.62:0/180613651 >> 172.21.15.62:0/180613651 conn(0x7f1eb40fd390 msgr2=0x7f1eb40fee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:31:23.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:23.142+0000 7f1ebbe00640 1 -- 172.21.15.62:0/180613651 shutdown_connections 2024-04-20T22:31:23.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:23.142+0000 7f1ebbe00640 1 -- 172.21.15.62:0/180613651 wait complete. 2024-04-20T22:31:23.161 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 35 2024-04-20T22:31:23.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:23.374+0000 7f276be00640 1 -- 172.21.15.62:0/4120019992 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f276c1076c0 msgr2=0x7f276c107a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:23.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:23.374+0000 7f276be00640 1 --2- 172.21.15.62:0/4120019992 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f276c1076c0 0x7f276c107a90 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7f275c009880 tx=0x7f275c02bce0 comp rx=0 tx=0).stop 2024-04-20T22:31:23.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:23.375+0000 7f276be00640 1 -- 172.21.15.62:0/4120019992 shutdown_connections 2024-04-20T22:31:23.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:23.375+0000 7f276be00640 1 --2- 172.21.15.62:0/4120019992 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f276c1016b0 0x7f276c101b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:23.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:23.375+0000 7f276be00640 1 --2- 172.21.15.62:0/4120019992 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f276c1076c0 0x7f276c107a90 unknown :-1 s=CLOSED pgs=1450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:23.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:23.375+0000 7f276be00640 1 -- 172.21.15.62:0/4120019992 >> 172.21.15.62:0/4120019992 conn(0x7f276c0fd1b0 msgr2=0x7f276c0ff5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:31:23.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:23.375+0000 7f276be00640 1 -- 172.21.15.62:0/4120019992 shutdown_connections 2024-04-20T22:31:23.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:23.375+0000 7f276be00640 1 -- 172.21.15.62:0/4120019992 wait complete. 2024-04-20T22:31:23.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:23.376+0000 7f276be00640 1 Processor -- start 2024-04-20T22:31:23.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:23.376+0000 7f276be00640 1 -- start start 2024-04-20T22:31:23.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:23.376+0000 7f276be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f276c1016b0 0x7f276c197310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:23.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:23.376+0000 7f276be00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f276c1076c0 0x7f276c197850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:23.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:23.376+0000 7f276be00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f276c197ee0 con 0x7f276c1016b0 2024-04-20T22:31:23.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:23.376+0000 7f276be00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f276c19bcb0 con 0x7f276c1076c0 2024-04-20T22:31:23.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:23.377+0000 7f276aa00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f276c1016b0 0x7f276c197310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:23.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:23.377+0000 7f276aa00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f276c1016b0 0x7f276c197310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53386/0 (socket says 172.21.15.62:53386) 2024-04-20T22:31:23.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:23.377+0000 7f276aa00640 1 -- 172.21.15.62:0/3039662315 learned_addr learned my addr 172.21.15.62:0/3039662315 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:31:23.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:23.377+0000 7f276aa00640 1 -- 172.21.15.62:0/3039662315 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f276c1076c0 msgr2=0x7f276c197850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:23.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:23.377+0000 7f276aa00640 1 --2- 172.21.15.62:0/3039662315 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f276c1076c0 0x7f276c197850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:23.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:23.377+0000 7f276aa00640 1 -- 172.21.15.62:0/3039662315 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f275c009560 con 0x7f276c1016b0 2024-04-20T22:31:23.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:23.377+0000 7f276aa00640 1 --2- 172.21.15.62:0/3039662315 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f276c1016b0 0x7f276c197310 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7f275c02c220 tx=0x7f275c02c7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:31:23.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:23.378+0000 7f2763400640 1 -- 172.21.15.62:0/3039662315 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f275c004520 con 0x7f276c1016b0 2024-04-20T22:31:23.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:23.378+0000 7f2763400640 1 -- 172.21.15.62:0/3039662315 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f275c003ab0 con 0x7f276c1016b0 2024-04-20T22:31:23.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:23.378+0000 7f2763400640 1 -- 172.21.15.62:0/3039662315 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f275c034440 con 0x7f276c1016b0 2024-04-20T22:31:23.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:23.378+0000 7f276be00640 1 -- 172.21.15.62:0/3039662315 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f276c19bf30 con 0x7f276c1016b0 2024-04-20T22:31:23.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:23.378+0000 7f276be00640 1 -- 172.21.15.62:0/3039662315 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f276c19c4b0 con 0x7f276c1016b0 2024-04-20T22:31:23.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:23.379+0000 7f276be00640 1 -- 172.21.15.62:0/3039662315 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f276c04e000 con 0x7f276c1016b0 2024-04-20T22:31:23.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:23.380+0000 7f2763400640 1 -- 172.21.15.62:0/3039662315 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f275c003c20 con 0x7f276c1016b0 2024-04-20T22:31:23.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:23.381+0000 7f2763400640 1 --2- 172.21.15.62:0/3039662315 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f275006ec30 0x7f27500710e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:23.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:23.381+0000 7f2763400640 1 -- 172.21.15.62:0/3039662315 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f275c0b1130 con 0x7f276c1016b0 2024-04-20T22:31:23.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:23.382+0000 7f2763e00640 1 --2- 172.21.15.62:0/3039662315 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f275006ec30 0x7f27500710e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:23.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:23.382+0000 7f2763e00640 1 --2- 172.21.15.62:0/3039662315 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f275006ec30 0x7f27500710e0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f2754005fd0 tx=0x7f2754005850 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:31:23.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:23.386+0000 7f2763400640 1 -- 172.21.15.62:0/3039662315 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f275c079f60 con 0x7f276c1016b0 2024-04-20T22:31:23.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:23.639+0000 7f276be00640 1 -- 172.21.15.62:0/3039662315 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 35, "format": "json"} v 0) v1 -- 0x7f276c198660 con 0x7f276c1016b0 2024-04-20T22:31:23.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:23.640+0000 7f2763400640 1 -- 172.21.15.62:0/3039662315 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 35, "format": "json"}]=0 dumped fsmap epoch 35 v39) v1 ==== 107+0+5245 (secure 0 0 0) 0x7f275c07d620 con 0x7f276c1016b0 2024-04-20T22:31:23.640 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:31:23.640 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":44727,"name":"cephfs.smithi189.dvimer","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6825/3576699232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":3576699232},{"type":"v1","addr":"172.21.15.189:6825","nonce":3576699232}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":27},{"gid":45075,"name":"cephfs.smithi189.xbgspw","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6827/212905792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":212905792},{"type":"v1","addr":"172.21.15.189:6827","nonce":212905792}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":35,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-20T21:27:21.032337+0000","modified":"2024-04-20T22:28:30.029451+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34444,"mds_1":34476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34444":{"gid":34444,"name":"cephfs.smithi062.ryqblb","rank":0,"incarnation":31,"state":"up:reconnect","state_seq":697,"addr":"172.21.15.62:6827/4113916205","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":4113916205},{"type":"v1","addr":"172.21.15.62:6827","nonce":4113916205}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34476":{"gid":34476,"name":"cephfs.smithi062.ozizwi","rank":1,"incarnation":31,"state":"up:reconnect","state_seq":689,"addr":"172.21.15.62:6829/3801975165","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":3801975165},{"type":"v1","addr":"172.21.15.62:6829","nonce":3801975165}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-20T22:31:23.640 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 35 2024-04-20T22:31:23.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:23.642+0000 7f276be00640 1 -- 172.21.15.62:0/3039662315 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f275006ec30 msgr2=0x7f27500710e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:23.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:23.642+0000 7f276be00640 1 --2- 172.21.15.62:0/3039662315 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f275006ec30 0x7f27500710e0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f2754005fd0 tx=0x7f2754005850 comp rx=0 tx=0).stop 2024-04-20T22:31:23.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:23.642+0000 7f276be00640 1 -- 172.21.15.62:0/3039662315 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f276c1016b0 msgr2=0x7f276c197310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:23.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:23.642+0000 7f276be00640 1 --2- 172.21.15.62:0/3039662315 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f276c1016b0 0x7f276c197310 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7f275c02c220 tx=0x7f275c02c7a0 comp rx=0 tx=0).stop 2024-04-20T22:31:23.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:23.642+0000 7f276be00640 1 -- 172.21.15.62:0/3039662315 shutdown_connections 2024-04-20T22:31:23.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:23.642+0000 7f276be00640 1 --2- 172.21.15.62:0/3039662315 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f275006ec30 0x7f27500710e0 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:23.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:23.642+0000 7f276be00640 1 --2- 172.21.15.62:0/3039662315 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f276c1076c0 0x7f276c197850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:23.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:23.642+0000 7f276be00640 1 --2- 172.21.15.62:0/3039662315 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f276c1016b0 0x7f276c197310 unknown :-1 s=CLOSED pgs=1451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:23.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:23.642+0000 7f276be00640 1 -- 172.21.15.62:0/3039662315 >> 172.21.15.62:0/3039662315 conn(0x7f276c0fd1b0 msgr2=0x7f276c0feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:31:23.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:23.642+0000 7f276be00640 1 -- 172.21.15.62:0/3039662315 shutdown_connections 2024-04-20T22:31:23.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:23.643+0000 7f276be00640 1 -- 172.21.15.62:0/3039662315 wait complete. 2024-04-20T22:31:23.661 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 36 2024-04-20T22:31:23.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:23.860+0000 7fb951800640 1 -- 172.21.15.62:0/406228861 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb94c067310 msgr2=0x7fb94c067760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:23.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:23.860+0000 7fb951800640 1 --2- 172.21.15.62:0/406228861 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb94c067310 0x7fb94c067760 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7fb934009880 tx=0x7fb93402bce0 comp rx=0 tx=0).stop 2024-04-20T22:31:23.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:23.860+0000 7fb951800640 1 -- 172.21.15.62:0/406228861 shutdown_connections 2024-04-20T22:31:23.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:23.860+0000 7fb951800640 1 --2- 172.21.15.62:0/406228861 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb94c067310 0x7fb94c067760 unknown :-1 s=CLOSED pgs=1452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:23.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:23.860+0000 7fb951800640 1 --2- 172.21.15.62:0/406228861 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb94c1055c0 0x7fb94c105990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:23.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:23.860+0000 7fb951800640 1 -- 172.21.15.62:0/406228861 >> 172.21.15.62:0/406228861 conn(0x7fb94c0743d0 msgr2=0x7fb94c0747d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:31:23.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:23.860+0000 7fb951800640 1 -- 172.21.15.62:0/406228861 shutdown_connections 2024-04-20T22:31:23.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:23.860+0000 7fb951800640 1 -- 172.21.15.62:0/406228861 wait complete. 2024-04-20T22:31:23.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:23.861+0000 7fb951800640 1 Processor -- start 2024-04-20T22:31:23.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:23.862+0000 7fb951800640 1 -- start start 2024-04-20T22:31:23.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:23.862+0000 7fb951800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb94c067310 0x7fb94c195140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:23.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:23.862+0000 7fb951800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb94c1055c0 0x7fb94c195680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:23.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:23.862+0000 7fb951800640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb94c195d10 con 0x7fb94c1055c0 2024-04-20T22:31:23.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:23.862+0000 7fb951800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb94c199ae0 con 0x7fb94c067310 2024-04-20T22:31:23.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:23.862+0000 7fb943400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb94c1055c0 0x7fb94c195680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:23.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:23.862+0000 7fb943400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb94c1055c0 0x7fb94c195680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53406/0 (socket says 172.21.15.62:53406) 2024-04-20T22:31:23.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:23.862+0000 7fb943400640 1 -- 172.21.15.62:0/3436786603 learned_addr learned my addr 172.21.15.62:0/3436786603 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:31:23.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:23.863+0000 7fb943400640 1 -- 172.21.15.62:0/3436786603 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb94c067310 msgr2=0x7fb94c195140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:23.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:23.863+0000 7fb943400640 1 --2- 172.21.15.62:0/3436786603 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb94c067310 0x7fb94c195140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:23.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:23.863+0000 7fb943400640 1 -- 172.21.15.62:0/3436786603 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb934009560 con 0x7fb94c1055c0 2024-04-20T22:31:23.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:23.863+0000 7fb943400640 1 --2- 172.21.15.62:0/3436786603 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb94c1055c0 0x7fb94c195680 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7fb934000c00 tx=0x7fb934004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:31:23.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:23.864+0000 7fb949600640 1 -- 172.21.15.62:0/3436786603 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb9340035f0 con 0x7fb94c1055c0 2024-04-20T22:31:23.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:23.864+0000 7fb949600640 1 -- 172.21.15.62:0/3436786603 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb934003c10 con 0x7fb94c1055c0 2024-04-20T22:31:23.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:23.864+0000 7fb949600640 1 -- 172.21.15.62:0/3436786603 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb934034440 con 0x7fb94c1055c0 2024-04-20T22:31:23.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:23.864+0000 7fb951800640 1 -- 172.21.15.62:0/3436786603 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb94c199d60 con 0x7fb94c1055c0 2024-04-20T22:31:23.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:23.864+0000 7fb951800640 1 -- 172.21.15.62:0/3436786603 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb94c19a210 con 0x7fb94c1055c0 2024-04-20T22:31:23.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:23.865+0000 7fb951800640 1 -- 172.21.15.62:0/3436786603 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb94c04e000 con 0x7fb94c1055c0 2024-04-20T22:31:23.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:23.866+0000 7fb949600640 1 -- 172.21.15.62:0/3436786603 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fb934003750 con 0x7fb94c1055c0 2024-04-20T22:31:23.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:23.867+0000 7fb949600640 1 --2- 172.21.15.62:0/3436786603 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb93006ec30 0x7fb9300710e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:23.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:23.867+0000 7fb949600640 1 -- 172.21.15.62:0/3436786603 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fb9340b13c0 con 0x7fb94c1055c0 2024-04-20T22:31:23.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:23.867+0000 7fb94be00640 1 --2- 172.21.15.62:0/3436786603 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb93006ec30 0x7fb9300710e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:23.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:23.868+0000 7fb94be00640 1 --2- 172.21.15.62:0/3436786603 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb93006ec30 0x7fb9300710e0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fb93c005fd0 tx=0x7fb93c005850 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:31:23.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:23.872+0000 7fb949600640 1 -- 172.21.15.62:0/3436786603 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fb93407a0d0 con 0x7fb94c1055c0 2024-04-20T22:31:23.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:31:23 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/1087000665' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 33, "format": "json"}]: dispatch 2024-04-20T22:31:23.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:31:23 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/180613651' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 34, "format": "json"}]: dispatch 2024-04-20T22:31:24.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:31:23 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/1087000665' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 33, "format": "json"}]: dispatch 2024-04-20T22:31:24.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:31:23 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/180613651' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 34, "format": "json"}]: dispatch 2024-04-20T22:31:24.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:24.120+0000 7fb951800640 1 -- 172.21.15.62:0/3436786603 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 36, "format": "json"} v 0) v1 -- 0x7fb94c10c720 con 0x7fb94c1055c0 2024-04-20T22:31:24.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:24.122+0000 7fb949600640 1 -- 172.21.15.62:0/3436786603 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 36, "format": "json"}]=0 dumped fsmap epoch 36 v39) v1 ==== 107+0+5239 (secure 0 0 0) 0x7fb93407d790 con 0x7fb94c1055c0 2024-04-20T22:31:24.122 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:31:24.122 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":44727,"name":"cephfs.smithi189.dvimer","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6825/3576699232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":3576699232},{"type":"v1","addr":"172.21.15.189:6825","nonce":3576699232}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":27},{"gid":45075,"name":"cephfs.smithi189.xbgspw","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6827/212905792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":212905792},{"type":"v1","addr":"172.21.15.189:6827","nonce":212905792}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":36,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-20T21:27:21.032337+0000","modified":"2024-04-20T22:28:30.976090+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34444,"mds_1":34476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34444":{"gid":34444,"name":"cephfs.smithi062.ryqblb","rank":0,"incarnation":31,"state":"up:rejoin","state_seq":698,"addr":"172.21.15.62:6827/4113916205","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":4113916205},{"type":"v1","addr":"172.21.15.62:6827","nonce":4113916205}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34476":{"gid":34476,"name":"cephfs.smithi062.ozizwi","rank":1,"incarnation":31,"state":"up:rejoin","state_seq":690,"addr":"172.21.15.62:6829/3801975165","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":3801975165},{"type":"v1","addr":"172.21.15.62:6829","nonce":3801975165}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-20T22:31:24.122 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 36 2024-04-20T22:31:24.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:24.124+0000 7fb951800640 1 -- 172.21.15.62:0/3436786603 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb93006ec30 msgr2=0x7fb9300710e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:24.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:24.124+0000 7fb951800640 1 --2- 172.21.15.62:0/3436786603 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb93006ec30 0x7fb9300710e0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fb93c005fd0 tx=0x7fb93c005850 comp rx=0 tx=0).stop 2024-04-20T22:31:24.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:24.124+0000 7fb951800640 1 -- 172.21.15.62:0/3436786603 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb94c1055c0 msgr2=0x7fb94c195680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:24.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:24.124+0000 7fb951800640 1 --2- 172.21.15.62:0/3436786603 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb94c1055c0 0x7fb94c195680 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7fb934000c00 tx=0x7fb934004040 comp rx=0 tx=0).stop 2024-04-20T22:31:24.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:24.125+0000 7fb951800640 1 -- 172.21.15.62:0/3436786603 shutdown_connections 2024-04-20T22:31:24.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:24.125+0000 7fb951800640 1 --2- 172.21.15.62:0/3436786603 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fb93006ec30 0x7fb9300710e0 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:24.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:24.125+0000 7fb951800640 1 --2- 172.21.15.62:0/3436786603 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb94c1055c0 0x7fb94c195680 unknown :-1 s=CLOSED pgs=1453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:24.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:24.125+0000 7fb951800640 1 --2- 172.21.15.62:0/3436786603 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb94c067310 0x7fb94c195140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:24.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:24.125+0000 7fb951800640 1 -- 172.21.15.62:0/3436786603 >> 172.21.15.62:0/3436786603 conn(0x7fb94c0743d0 msgr2=0x7fb94c0fcdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:31:24.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:24.125+0000 7fb951800640 1 -- 172.21.15.62:0/3436786603 shutdown_connections 2024-04-20T22:31:24.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:24.125+0000 7fb951800640 1 -- 172.21.15.62:0/3436786603 wait complete. 2024-04-20T22:31:24.145 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 37 2024-04-20T22:31:24.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:24.346+0000 7f92dfe00640 1 -- 172.21.15.62:0/1148015982 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f92e01016b0 msgr2=0x7f92e0101b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:24.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:24.346+0000 7f92dfe00640 1 --2- 172.21.15.62:0/1148015982 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f92e01016b0 0x7f92e0101b20 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7f92cc009880 tx=0x7f92cc02bcf0 comp rx=0 tx=0).stop 2024-04-20T22:31:24.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:24.346+0000 7f92dfe00640 1 -- 172.21.15.62:0/1148015982 shutdown_connections 2024-04-20T22:31:24.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:24.346+0000 7f92dfe00640 1 --2- 172.21.15.62:0/1148015982 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f92e01016b0 0x7f92e0101b20 unknown :-1 s=CLOSED pgs=1454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:24.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:24.346+0000 7f92dfe00640 1 --2- 172.21.15.62:0/1148015982 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f92e01076c0 0x7f92e0107a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:24.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:24.346+0000 7f92dfe00640 1 -- 172.21.15.62:0/1148015982 >> 172.21.15.62:0/1148015982 conn(0x7f92e00fd1b0 msgr2=0x7f92e00ff5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:31:24.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:24.346+0000 7f92dfe00640 1 -- 172.21.15.62:0/1148015982 shutdown_connections 2024-04-20T22:31:24.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:24.346+0000 7f92dfe00640 1 -- 172.21.15.62:0/1148015982 wait complete. 2024-04-20T22:31:24.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:24.347+0000 7f92dfe00640 1 Processor -- start 2024-04-20T22:31:24.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:24.347+0000 7f92dfe00640 1 -- start start 2024-04-20T22:31:24.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:24.347+0000 7f92dfe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f92e01016b0 0x7f92e0197370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:24.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:24.347+0000 7f92dfe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f92e01076c0 0x7f92e01978b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:24.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:24.347+0000 7f92dfe00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92e0197eb0 con 0x7f92e01016b0 2024-04-20T22:31:24.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:24.347+0000 7f92dfe00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92e0198020 con 0x7f92e01076c0 2024-04-20T22:31:24.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:24.348+0000 7f92dea00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f92e01016b0 0x7f92e0197370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:24.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:24.348+0000 7f92dea00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f92e01016b0 0x7f92e0197370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53414/0 (socket says 172.21.15.62:53414) 2024-04-20T22:31:24.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:24.348+0000 7f92dea00640 1 -- 172.21.15.62:0/3023880843 learned_addr learned my addr 172.21.15.62:0/3023880843 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:31:24.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:24.348+0000 7f92de000640 1 --2- 172.21.15.62:0/3023880843 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f92e01076c0 0x7f92e01978b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:24.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:24.348+0000 7f92dea00640 1 -- 172.21.15.62:0/3023880843 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f92e01076c0 msgr2=0x7f92e01978b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:24.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:24.348+0000 7f92dea00640 1 --2- 172.21.15.62:0/3023880843 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f92e01076c0 0x7f92e01978b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:24.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:24.348+0000 7f92dea00640 1 -- 172.21.15.62:0/3023880843 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92cc009560 con 0x7f92e01016b0 2024-04-20T22:31:24.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:24.349+0000 7f92dea00640 1 --2- 172.21.15.62:0/3023880843 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f92e01016b0 0x7f92e0197370 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7f92d000c8f0 tx=0x7f92d000cdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:31:24.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:24.349+0000 7f92e4a00640 1 -- 172.21.15.62:0/3023880843 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f92d00041f0 con 0x7f92e01016b0 2024-04-20T22:31:24.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:24.349+0000 7f92e4a00640 1 -- 172.21.15.62:0/3023880843 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f92d0007e10 con 0x7f92e01016b0 2024-04-20T22:31:24.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:24.349+0000 7f92dfe00640 1 -- 172.21.15.62:0/3023880843 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92e019beb0 con 0x7f92e01016b0 2024-04-20T22:31:24.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:24.349+0000 7f92e4a00640 1 -- 172.21.15.62:0/3023880843 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f92d000f450 con 0x7f92e01016b0 2024-04-20T22:31:24.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:24.349+0000 7f92dfe00640 1 -- 172.21.15.62:0/3023880843 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92e019c410 con 0x7f92e01016b0 2024-04-20T22:31:24.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:24.350+0000 7f92dfe00640 1 -- 172.21.15.62:0/3023880843 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92e004e000 con 0x7f92e01016b0 2024-04-20T22:31:24.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:24.351+0000 7f92e4a00640 1 -- 172.21.15.62:0/3023880843 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f92d0007950 con 0x7f92e01016b0 2024-04-20T22:31:24.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:24.352+0000 7f92e4a00640 1 --2- 172.21.15.62:0/3023880843 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f92c406ec30 0x7f92c40710e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:24.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:24.353+0000 7f92e4a00640 1 -- 172.21.15.62:0/3023880843 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f92d0013070 con 0x7f92e01016b0 2024-04-20T22:31:24.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:24.353+0000 7f92de000640 1 --2- 172.21.15.62:0/3023880843 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f92c406ec30 0x7f92c40710e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:24.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:24.353+0000 7f92de000640 1 --2- 172.21.15.62:0/3023880843 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f92c406ec30 0x7f92c40710e0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f92e0198980 tx=0x7f92cc000bc0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:31:24.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:24.357+0000 7f92e4a00640 1 -- 172.21.15.62:0/3023880843 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f92d0057d10 con 0x7f92e01016b0 2024-04-20T22:31:24.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:24.595+0000 7f92dfe00640 1 -- 172.21.15.62:0/3023880843 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 37, "format": "json"} v 0) v1 -- 0x7f92e01986c0 con 0x7f92e01016b0 2024-04-20T22:31:24.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:24.597+0000 7f92e4a00640 1 -- 172.21.15.62:0/3023880843 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 37, "format": "json"}]=0 dumped fsmap epoch 37 v39) v1 ==== 107+0+5254 (secure 0 0 0) 0x7f92d005b3d0 con 0x7f92e01016b0 2024-04-20T22:31:24.597 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:31:24.597 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":44727,"name":"cephfs.smithi189.dvimer","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6825/3576699232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":3576699232},{"type":"v1","addr":"172.21.15.189:6825","nonce":3576699232}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":27},{"gid":45075,"name":"cephfs.smithi189.xbgspw","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6827/212905792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":212905792},{"type":"v1","addr":"172.21.15.189:6827","nonce":212905792}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":37,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-20T21:27:21.032337+0000","modified":"2024-04-20T22:28:32.985522+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34444,"mds_1":34476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34444":{"gid":34444,"name":"cephfs.smithi062.ryqblb","rank":0,"incarnation":31,"state":"up:active","state_seq":699,"addr":"172.21.15.62:6827/4113916205","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":4113916205},{"type":"v1","addr":"172.21.15.62:6827","nonce":4113916205}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34476":{"gid":34476,"name":"cephfs.smithi062.ozizwi","rank":1,"incarnation":31,"state":"up:active","state_seq":691,"addr":"172.21.15.62:6829/3801975165","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":3801975165},{"type":"v1","addr":"172.21.15.62:6829","nonce":3801975165}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":34444,"qdb_cluster":[34444,34476]},"id":1}]} 2024-04-20T22:31:24.597 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 37 2024-04-20T22:31:24.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:24.600+0000 7f92dfe00640 1 -- 172.21.15.62:0/3023880843 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f92c406ec30 msgr2=0x7f92c40710e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:24.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:24.600+0000 7f92dfe00640 1 --2- 172.21.15.62:0/3023880843 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f92c406ec30 0x7f92c40710e0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f92e0198980 tx=0x7f92cc000bc0 comp rx=0 tx=0).stop 2024-04-20T22:31:24.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:24.600+0000 7f92dfe00640 1 -- 172.21.15.62:0/3023880843 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f92e01016b0 msgr2=0x7f92e0197370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:24.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:24.600+0000 7f92dfe00640 1 --2- 172.21.15.62:0/3023880843 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f92e01016b0 0x7f92e0197370 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7f92d000c8f0 tx=0x7f92d000cdb0 comp rx=0 tx=0).stop 2024-04-20T22:31:24.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:24.600+0000 7f92dfe00640 1 -- 172.21.15.62:0/3023880843 shutdown_connections 2024-04-20T22:31:24.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:24.600+0000 7f92dfe00640 1 --2- 172.21.15.62:0/3023880843 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f92c406ec30 0x7f92c40710e0 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:24.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:24.600+0000 7f92dfe00640 1 --2- 172.21.15.62:0/3023880843 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f92e01076c0 0x7f92e01978b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:24.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:24.600+0000 7f92dfe00640 1 --2- 172.21.15.62:0/3023880843 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f92e01016b0 0x7f92e0197370 unknown :-1 s=CLOSED pgs=1455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:24.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:24.600+0000 7f92dfe00640 1 -- 172.21.15.62:0/3023880843 >> 172.21.15.62:0/3023880843 conn(0x7f92e00fd1b0 msgr2=0x7f92e00feb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:31:24.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:24.600+0000 7f92dfe00640 1 -- 172.21.15.62:0/3023880843 shutdown_connections 2024-04-20T22:31:24.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:24.600+0000 7f92dfe00640 1 -- 172.21.15.62:0/3023880843 wait complete. 2024-04-20T22:31:24.621 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 38 2024-04-20T22:31:24.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:24.825+0000 7f98d6800640 1 -- 172.21.15.62:0/3392142496 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f98d0106090 msgr2=0x7f98d01064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:24.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:24.825+0000 7f98d6800640 1 --2- 172.21.15.62:0/3392142496 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f98d0106090 0x7f98d01064e0 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7f98bc009880 tx=0x7f98bc02bce0 comp rx=0 tx=0).stop 2024-04-20T22:31:24.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:24.825+0000 7f98d6800640 1 -- 172.21.15.62:0/3392142496 shutdown_connections 2024-04-20T22:31:24.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:24.825+0000 7f98d6800640 1 --2- 172.21.15.62:0/3392142496 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f98d0106090 0x7f98d01064e0 unknown :-1 s=CLOSED pgs=1456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:24.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:24.825+0000 7f98d6800640 1 --2- 172.21.15.62:0/3392142496 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98d010bb60 0x7f98d010bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:24.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:24.825+0000 7f98d6800640 1 -- 172.21.15.62:0/3392142496 >> 172.21.15.62:0/3392142496 conn(0x7f98d0075260 msgr2=0x7f98d0077670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:31:24.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:24.825+0000 7f98d6800640 1 -- 172.21.15.62:0/3392142496 shutdown_connections 2024-04-20T22:31:24.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:24.825+0000 7f98d6800640 1 -- 172.21.15.62:0/3392142496 wait complete. 2024-04-20T22:31:24.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:24.826+0000 7f98d6800640 1 Processor -- start 2024-04-20T22:31:24.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:24.827+0000 7f98d6800640 1 -- start start 2024-04-20T22:31:24.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:24.827+0000 7f98d6800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98d0106090 0x7f98d01a15e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:24.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:24.827+0000 7f98d6800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f98d010bb60 0x7f98d01a1b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:24.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:24.827+0000 7f98d6800640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98d019b5c0 con 0x7f98d010bb60 2024-04-20T22:31:24.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:24.827+0000 7f98d6800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98d019b700 con 0x7f98d0106090 2024-04-20T22:31:24.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:24.827+0000 7f98d4a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f98d010bb60 0x7f98d01a1b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:24.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:24.827+0000 7f98d4a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f98d010bb60 0x7f98d01a1b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53428/0 (socket says 172.21.15.62:53428) 2024-04-20T22:31:24.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:24.827+0000 7f98d4a00640 1 -- 172.21.15.62:0/4088891697 learned_addr learned my addr 172.21.15.62:0/4088891697 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:31:24.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:24.828+0000 7f98d5400640 1 --2- 172.21.15.62:0/4088891697 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98d0106090 0x7f98d01a15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:24.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:24.828+0000 7f98d4a00640 1 -- 172.21.15.62:0/4088891697 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98d0106090 msgr2=0x7f98d01a15e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:24.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:24.828+0000 7f98d4a00640 1 --2- 172.21.15.62:0/4088891697 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98d0106090 0x7f98d01a15e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:24.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:24.828+0000 7f98d4a00640 1 -- 172.21.15.62:0/4088891697 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98bc009560 con 0x7f98d010bb60 2024-04-20T22:31:24.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:24.828+0000 7f98d4a00640 1 --2- 172.21.15.62:0/4088891697 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f98d010bb60 0x7f98d01a1b20 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7f98bc009230 tx=0x7f98bc004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:31:24.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:24.829+0000 7f98ce000640 1 -- 172.21.15.62:0/4088891697 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f98bc0035f0 con 0x7f98d010bb60 2024-04-20T22:31:24.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:24.829+0000 7f98ce000640 1 -- 172.21.15.62:0/4088891697 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f98bc003c10 con 0x7f98d010bb60 2024-04-20T22:31:24.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:24.829+0000 7f98d6800640 1 -- 172.21.15.62:0/4088891697 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98d019b980 con 0x7f98d010bb60 2024-04-20T22:31:24.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:24.829+0000 7f98ce000640 1 -- 172.21.15.62:0/4088891697 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f98bc005670 con 0x7f98d010bb60 2024-04-20T22:31:24.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:24.829+0000 7f98d6800640 1 -- 172.21.15.62:0/4088891697 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98d019be80 con 0x7f98d010bb60 2024-04-20T22:31:24.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:24.830+0000 7f98d6800640 1 -- 172.21.15.62:0/4088891697 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98d004e000 con 0x7f98d010bb60 2024-04-20T22:31:24.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:24.831+0000 7f98ce000640 1 -- 172.21.15.62:0/4088891697 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f98bc004dd0 con 0x7f98d010bb60 2024-04-20T22:31:24.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:24.833+0000 7f98ce000640 1 --2- 172.21.15.62:0/4088891697 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f98b806ec30 0x7f98b80710e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:24.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:24.833+0000 7f98ce000640 1 -- 172.21.15.62:0/4088891697 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f98bc0b1170 con 0x7f98d010bb60 2024-04-20T22:31:24.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:24.833+0000 7f98d5400640 1 --2- 172.21.15.62:0/4088891697 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f98b806ec30 0x7f98b80710e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:24.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:24.834+0000 7f98d5400640 1 --2- 172.21.15.62:0/4088891697 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f98b806ec30 0x7f98b80710e0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f98c400bbe0 tx=0x7f98c400b390 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:31:24.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:24.837+0000 7f98ce000640 1 -- 172.21.15.62:0/4088891697 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f98bc079fb0 con 0x7f98d010bb60 2024-04-20T22:31:24.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:31:24 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/3039662315' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 35, "format": "json"}]: dispatch 2024-04-20T22:31:24.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:31:24 smithi189 ceph-mon[57022]: pgmap v1688: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:31:24.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:31:24 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/3436786603' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 36, "format": "json"}]: dispatch 2024-04-20T22:31:24.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:31:24 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/3023880843' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 37, "format": "json"}]: dispatch 2024-04-20T22:31:25.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:31:24 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/3039662315' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 35, "format": "json"}]: dispatch 2024-04-20T22:31:25.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:31:24 smithi062 ceph-mon[76322]: pgmap v1688: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:31:25.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:31:24 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/3436786603' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 36, "format": "json"}]: dispatch 2024-04-20T22:31:25.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:31:24 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/3023880843' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 37, "format": "json"}]: dispatch 2024-04-20T22:31:25.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:25.083+0000 7f98d6800640 1 -- 172.21.15.62:0/4088891697 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 38, "format": "json"} v 0) v1 -- 0x7f98d0112c70 con 0x7f98d010bb60 2024-04-20T22:31:25.085 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:25.085+0000 7f98ce000640 1 -- 172.21.15.62:0/4088891697 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 38, "format": "json"}]=0 dumped fsmap epoch 38 v39) v1 ==== 107+0+5269 (secure 0 0 0) 0x7f98bc07d670 con 0x7f98d010bb60 2024-04-20T22:31:25.085 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:31:25.085 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":38,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-20T21:27:21.032337+0000","modified":"2024-04-20T22:28:33.008821+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34444,"mds_1":34476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34444":{"gid":34444,"name":"cephfs.smithi062.ryqblb","rank":0,"incarnation":31,"state":"up:active","state_seq":699,"addr":"172.21.15.62:6827/4113916205","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":4113916205},{"type":"v1","addr":"172.21.15.62:6827","nonce":4113916205}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34476":{"gid":34476,"name":"cephfs.smithi062.ozizwi","rank":1,"incarnation":31,"state":"up:active","state_seq":691,"addr":"172.21.15.62:6829/3801975165","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":3801975165},{"type":"v1","addr":"172.21.15.62:6829","nonce":3801975165}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_44727":{"gid":44727,"name":"cephfs.smithi189.dvimer","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6825/3576699232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":3576699232},{"type":"v1","addr":"172.21.15.189:6825","nonce":3576699232}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_45075":{"gid":45075,"name":"cephfs.smithi189.xbgspw","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6827/212905792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":212905792},{"type":"v1","addr":"172.21.15.189:6827","nonce":212905792}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":34444,"qdb_cluster":[34476,34444]},"id":1}]} 2024-04-20T22:31:25.085 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 38 2024-04-20T22:31:25.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:25.088+0000 7f98d6800640 1 -- 172.21.15.62:0/4088891697 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f98b806ec30 msgr2=0x7f98b80710e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:25.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:25.088+0000 7f98d6800640 1 --2- 172.21.15.62:0/4088891697 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f98b806ec30 0x7f98b80710e0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f98c400bbe0 tx=0x7f98c400b390 comp rx=0 tx=0).stop 2024-04-20T22:31:25.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:25.088+0000 7f98d6800640 1 -- 172.21.15.62:0/4088891697 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f98d010bb60 msgr2=0x7f98d01a1b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:25.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:25.088+0000 7f98d6800640 1 --2- 172.21.15.62:0/4088891697 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f98d010bb60 0x7f98d01a1b20 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7f98bc009230 tx=0x7f98bc004040 comp rx=0 tx=0).stop 2024-04-20T22:31:25.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:25.088+0000 7f98d6800640 1 -- 172.21.15.62:0/4088891697 shutdown_connections 2024-04-20T22:31:25.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:25.088+0000 7f98d6800640 1 --2- 172.21.15.62:0/4088891697 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f98b806ec30 0x7f98b80710e0 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:25.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:25.088+0000 7f98d6800640 1 --2- 172.21.15.62:0/4088891697 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f98d010bb60 0x7f98d01a1b20 unknown :-1 s=CLOSED pgs=1457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:25.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:25.088+0000 7f98d6800640 1 --2- 172.21.15.62:0/4088891697 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98d0106090 0x7f98d01a15e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:25.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:25.088+0000 7f98d6800640 1 -- 172.21.15.62:0/4088891697 >> 172.21.15.62:0/4088891697 conn(0x7f98d0075260 msgr2=0x7f98d0076cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:31:25.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:25.088+0000 7f98d6800640 1 -- 172.21.15.62:0/4088891697 shutdown_connections 2024-04-20T22:31:25.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:25.088+0000 7f98d6800640 1 -- 172.21.15.62:0/4088891697 wait complete. 2024-04-20T22:31:25.110 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-04-20T22:31:25.120 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-04-20T22:31:25.121 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-04-20T22:31:25.121 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-20T22:31:25.179 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-04-20T22:31:25.179 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-20T22:31:25.232 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-20T22:31:25.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:25.442+0000 7f53aec00640 1 -- 172.21.15.62:0/2132749600 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f53a8100620 msgr2=0x7f53a81009f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:25.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:25.442+0000 7f53aec00640 1 --2- 172.21.15.62:0/2132749600 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f53a8100620 0x7f53a81009f0 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7f5398009c20 tx=0x7f539802c1b0 comp rx=0 tx=0).stop 2024-04-20T22:31:25.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:25.442+0000 7f53aec00640 1 -- 172.21.15.62:0/2132749600 shutdown_connections 2024-04-20T22:31:25.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:25.442+0000 7f53aec00640 1 --2- 172.21.15.62:0/2132749600 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53a8100f30 0x7f53a8109420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:25.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:25.442+0000 7f53aec00640 1 --2- 172.21.15.62:0/2132749600 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f53a8100620 0x7f53a81009f0 unknown :-1 s=CLOSED pgs=1458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:25.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:25.442+0000 7f53aec00640 1 -- 172.21.15.62:0/2132749600 >> 172.21.15.62:0/2132749600 conn(0x7f53a80f9e30 msgr2=0x7f53a80fc240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:31:25.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:25.442+0000 7f53aec00640 1 -- 172.21.15.62:0/2132749600 shutdown_connections 2024-04-20T22:31:25.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:25.442+0000 7f53aec00640 1 -- 172.21.15.62:0/2132749600 wait complete. 2024-04-20T22:31:25.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:25.443+0000 7f53aec00640 1 Processor -- start 2024-04-20T22:31:25.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:25.444+0000 7f53aec00640 1 -- start start 2024-04-20T22:31:25.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:25.444+0000 7f53aec00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f53a8100620 0x7f53a81951e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:25.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:25.444+0000 7f53aec00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53a8100f30 0x7f53a8195720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:25.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:25.444+0000 7f53aec00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53a8195db0 con 0x7f53a8100620 2024-04-20T22:31:25.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:25.444+0000 7f53aec00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53a8199b80 con 0x7f53a8100f30 2024-04-20T22:31:25.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:25.444+0000 7f53ad800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f53a8100620 0x7f53a81951e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:25.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:25.444+0000 7f53ad800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f53a8100620 0x7f53a81951e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53458/0 (socket says 172.21.15.62:53458) 2024-04-20T22:31:25.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:25.444+0000 7f53ad800640 1 -- 172.21.15.62:0/2342590549 learned_addr learned my addr 172.21.15.62:0/2342590549 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:31:25.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:25.445+0000 7f53ad800640 1 -- 172.21.15.62:0/2342590549 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53a8100f30 msgr2=0x7f53a8195720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:25.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:25.445+0000 7f53ace00640 1 --2- 172.21.15.62:0/2342590549 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53a8100f30 0x7f53a8195720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:25.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:25.445+0000 7f53ad800640 1 --2- 172.21.15.62:0/2342590549 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53a8100f30 0x7f53a8195720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:25.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:25.445+0000 7f53ad800640 1 -- 172.21.15.62:0/2342590549 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5398009900 con 0x7f53a8100620 2024-04-20T22:31:25.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:25.445+0000 7f53ace00640 1 --2- 172.21.15.62:0/2342590549 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53a8100f30 0x7f53a8195720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T22:31:25.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:25.445+0000 7f53ad800640 1 --2- 172.21.15.62:0/2342590549 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f53a8100620 0x7f53a81951e0 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7f5398009bf0 tx=0x7f5398032af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:31:25.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:25.446+0000 7f53a6000640 1 -- 172.21.15.62:0/2342590549 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5398034820 con 0x7f53a8100620 2024-04-20T22:31:25.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:25.446+0000 7f53a6000640 1 -- 172.21.15.62:0/2342590549 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5398003e30 con 0x7f53a8100620 2024-04-20T22:31:25.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:25.446+0000 7f53aec00640 1 -- 172.21.15.62:0/2342590549 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53a8199e00 con 0x7f53a8100620 2024-04-20T22:31:25.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:25.446+0000 7f53a6000640 1 -- 172.21.15.62:0/2342590549 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f539803f440 con 0x7f53a8100620 2024-04-20T22:31:25.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:25.446+0000 7f53aec00640 1 -- 172.21.15.62:0/2342590549 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53a819a300 con 0x7f53a8100620 2024-04-20T22:31:25.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:25.447+0000 7f53aec00640 1 -- 172.21.15.62:0/2342590549 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53a804e850 con 0x7f53a8100620 2024-04-20T22:31:25.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:25.448+0000 7f53a6000640 1 -- 172.21.15.62:0/2342590549 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f539803b040 con 0x7f53a8100620 2024-04-20T22:31:25.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:25.449+0000 7f53a6000640 1 --2- 172.21.15.62:0/2342590549 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f538c06ecf0 0x7f538c0711a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:25.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:25.449+0000 7f53a6000640 1 -- 172.21.15.62:0/2342590549 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f53980b1b50 con 0x7f53a8100620 2024-04-20T22:31:25.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:25.449+0000 7f53ace00640 1 --2- 172.21.15.62:0/2342590549 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f538c06ecf0 0x7f538c0711a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:25.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:25.450+0000 7f53ace00640 1 --2- 172.21.15.62:0/2342590549 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f538c06ecf0 0x7f538c0711a0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f5394006fd0 tx=0x7f5394008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:31:25.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:25.454+0000 7f53a6000640 1 -- 172.21.15.62:0/2342590549 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f539807a980 con 0x7f53a8100620 2024-04-20T22:31:25.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:25.665+0000 7f53aec00640 1 -- 172.21.15.62:0/2342590549 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f53a810c6e0 con 0x7f53a8100620 2024-04-20T22:31:25.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:25.668+0000 7f53a6000640 1 -- 172.21.15.62:0/2342590549 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+16652 (secure 0 0 0) 0x7f539807e040 con 0x7f53a8100620 2024-04-20T22:31:25.668 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:31:25.668 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":85,"fsid":"61ae3628-ff5b-11ee-bc93-c7b262605968","created":"2024-04-20T21:18:19.951305+0000","modified":"2024-04-20T22:13:07.093942+0000","last_up_change":"2024-04-20T21:41:54.404166+0000","last_in_change":"2024-04-20T21:25:19.201764+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":20,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-20T21:24:02.500375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-04-20T21:27:19.577559+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-04-20T21:27:20.560814+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a3f8216-c641-4338-813d-d836efc3f4e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":48,"up_from":52,"up_thru":77,"down_at":49,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":1209367302},{"type":"v1","addr":"172.21.15.62:6803","nonce":1209367302}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":1209367302},{"type":"v1","addr":"172.21.15.62:6805","nonce":1209367302}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":1209367302},{"type":"v1","addr":"172.21.15.62:6809","nonce":1209367302}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":1209367302},{"type":"v1","addr":"172.21.15.62:6807","nonce":1209367302}]},"public_addr":"172.21.15.62:6803/1209367302","cluster_addr":"172.21.15.62:6805/1209367302","heartbeat_back_addr":"172.21.15.62:6809/1209367302","heartbeat_front_addr":"172.21.15.62:6807/1209367302","state":["exists","up"]},{"osd":1,"uuid":"f0c451a4-9002-4ddc-be0e-d8ddebd4418d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":14,"last_clean_end":53,"up_from":57,"up_thru":77,"down_at":54,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":2879835300},{"type":"v1","addr":"172.21.15.62:6811","nonce":2879835300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":2879835300},{"type":"v1","addr":"172.21.15.62:6813","nonce":2879835300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":2879835300},{"type":"v1","addr":"172.21.15.62:6817","nonce":2879835300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":2879835300},{"type":"v1","addr":"172.21.15.62:6815","nonce":2879835300}]},"public_addr":"172.21.15.62:6811/2879835300","cluster_addr":"172.21.15.62:6813/2879835300","heartbeat_back_addr":"172.21.15.62:6817/2879835300","heartbeat_front_addr":"172.21.15.62:6815/2879835300","state":["exists","up"]},{"osd":2,"uuid":"e1a8769d-5da2-49c8-942a-83a94c7ac761","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":19,"last_clean_end":58,"up_from":62,"up_thru":77,"down_at":59,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":4264123649},{"type":"v1","addr":"172.21.15.62:6819","nonce":4264123649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":4264123649},{"type":"v1","addr":"172.21.15.62:6821","nonce":4264123649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":4264123649},{"type":"v1","addr":"172.21.15.62:6825","nonce":4264123649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6822","nonce":4264123649},{"type":"v1","addr":"172.21.15.62:6823","nonce":4264123649}]},"public_addr":"172.21.15.62:6819/4264123649","cluster_addr":"172.21.15.62:6821/4264123649","heartbeat_back_addr":"172.21.15.62:6825/4264123649","heartbeat_front_addr":"172.21.15.62:6823/4264123649","state":["exists","up"]},{"osd":3,"uuid":"eaa2fe2b-87f8-4d01-8ddf-e72f244f31b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":63,"up_from":67,"up_thru":77,"down_at":64,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":3977023570},{"type":"v1","addr":"172.21.15.189:6801","nonce":3977023570}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":3977023570},{"type":"v1","addr":"172.21.15.189:6803","nonce":3977023570}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":3977023570},{"type":"v1","addr":"172.21.15.189:6807","nonce":3977023570}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":3977023570},{"type":"v1","addr":"172.21.15.189:6805","nonce":3977023570}]},"public_addr":"172.21.15.189:6801/3977023570","cluster_addr":"172.21.15.189:6803/3977023570","heartbeat_back_addr":"172.21.15.189:6807/3977023570","heartbeat_front_addr":"172.21.15.189:6805/3977023570","state":["exists","up"]},{"osd":4,"uuid":"72be8597-52e5-45fd-b4f2-06ac50df77e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":34,"last_clean_end":68,"up_from":72,"up_thru":77,"down_at":69,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":2794032930},{"type":"v1","addr":"172.21.15.189:6809","nonce":2794032930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":2794032930},{"type":"v1","addr":"172.21.15.189:6811","nonce":2794032930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":2794032930},{"type":"v1","addr":"172.21.15.189:6815","nonce":2794032930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":2794032930},{"type":"v1","addr":"172.21.15.189:6813","nonce":2794032930}]},"public_addr":"172.21.15.189:6809/2794032930","cluster_addr":"172.21.15.189:6811/2794032930","heartbeat_back_addr":"172.21.15.189:6815/2794032930","heartbeat_front_addr":"172.21.15.189:6813/2794032930","state":["exists","up"]},{"osd":5,"uuid":"c650c63b-41eb-4d4f-8e14-907c7b3614c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":39,"last_clean_end":73,"up_from":77,"up_thru":77,"down_at":74,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2760890917},{"type":"v1","addr":"172.21.15.189:6817","nonce":2760890917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2760890917},{"type":"v1","addr":"172.21.15.189:6819","nonce":2760890917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2760890917},{"type":"v1","addr":"172.21.15.189:6823","nonce":2760890917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2760890917},{"type":"v1","addr":"172.21.15.189:6821","nonce":2760890917}]},"public_addr":"172.21.15.189:6817/2760890917","cluster_addr":"172.21.15.189:6819/2760890917","heartbeat_back_addr":"172.21.15.189:6823/2760890917","heartbeat_front_addr":"172.21.15.189:6821/2760890917","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T21:22:45.605185+0000","dead_epoch":48},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T21:23:21.062873+0000","dead_epoch":53},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T21:24:00.502044+0000","dead_epoch":58},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T21:24:32.944093+0000","dead_epoch":63},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T21:25:06.278988+0000","dead_epoch":68},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T21:25:41.872982+0000","dead_epoch":73}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.189:6825/237262514":"2024-04-21T21:42:01.558630+0000","172.21.15.62:0/2348636520":"2024-04-21T21:30:35.075229+0000","172.21.15.62:6801/871423274":"2024-04-21T21:18:52.990685+0000","172.21.15.62:6801/831506538":"2024-04-21T21:21:03.778739+0000","172.21.15.62:0/4251078301":"2024-04-21T21:36:34.221905+0000","172.21.15.189:0/1714270321":"2024-04-21T21:31:36.545789+0000","172.21.15.62:6826/1147127123":"2024-04-21T21:42:01.558599+0000","172.21.15.62:0/868116748":"2024-04-21T21:30:35.075229+0000","172.21.15.62:0/2525292829":"2024-04-21T21:18:52.990685+0000","172.21.15.62:0/1116048562":"2024-04-21T21:21:03.778739+0000","172.21.15.62:0/634566145":"2024-04-21T21:18:52.990685+0000","172.21.15.62:6800/831506538":"2024-04-21T21:21:03.778739+0000","172.21.15.189:0/4026364416":"2024-04-21T21:31:36.545789+0000","172.21.15.62:0/1789120375":"2024-04-21T21:36:34.221905+0000","172.21.15.62:0/3416563026":"2024-04-21T21:18:52.990685+0000","172.21.15.62:0/3673769314":"2024-04-21T21:30:35.075229+0000","172.21.15.62:0/717708557":"2024-04-21T21:30:35.075229+0000","172.21.15.62:0/4274508569":"2024-04-21T21:19:34.965981+0000","172.21.15.189:0/3430221175":"2024-04-21T21:31:36.545789+0000","172.21.15.189:6824/237262514":"2024-04-21T21:42:01.558630+0000","172.21.15.189:0/100133912":"2024-04-21T21:31:36.545789+0000","172.21.15.189:0/441007232":"2024-04-21T21:31:36.545789+0000","172.21.15.62:6800/4116451241":"2024-04-21T21:30:35.075229+0000","172.21.15.62:6827/1147127123":"2024-04-21T21:42:01.558599+0000","172.21.15.62:6801/4116451241":"2024-04-21T21:30:35.075229+0000","172.21.15.62:6800/871423274":"2024-04-21T21:18:52.990685+0000","172.21.15.189:6828/2010473407":"2024-04-21T21:31:36.545789+0000","172.21.15.189:6829/2010473407":"2024-04-21T21:31:36.545789+0000","172.21.15.189:0/1726963148":"2024-04-21T21:31:36.545789+0000","172.21.15.62:0/3192460755":"2024-04-21T21:21:03.778739+0000","172.21.15.62:6801/4147738586":"2024-04-21T21:19:34.965981+0000","172.21.15.62:0/4158176989":"2024-04-21T21:19:34.965981+0000","172.21.15.62:0/301716985":"2024-04-21T21:21:03.778739+0000","172.21.15.62:0/678960466":"2024-04-21T21:36:34.221905+0000","172.21.15.62:0/2136121731":"2024-04-21T21:36:34.221905+0000","172.21.15.62:6800/4147738586":"2024-04-21T21:19:34.965981+0000","172.21.15.62:6800/1585760929":"2024-04-21T21:36:34.221905+0000","172.21.15.62:0/771955286":"2024-04-21T21:19:34.965981+0000","172.21.15.62:6801/1585760929":"2024-04-21T21:36:34.221905+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-20T22:31:25.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:25.670+0000 7f53aec00640 1 -- 172.21.15.62:0/2342590549 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f538c06ecf0 msgr2=0x7f538c0711a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:25.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:25.670+0000 7f53aec00640 1 --2- 172.21.15.62:0/2342590549 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f538c06ecf0 0x7f538c0711a0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f5394006fd0 tx=0x7f5394008040 comp rx=0 tx=0).stop 2024-04-20T22:31:25.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:25.670+0000 7f53aec00640 1 -- 172.21.15.62:0/2342590549 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f53a8100620 msgr2=0x7f53a81951e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:25.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:25.670+0000 7f53aec00640 1 --2- 172.21.15.62:0/2342590549 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f53a8100620 0x7f53a81951e0 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7f5398009bf0 tx=0x7f5398032af0 comp rx=0 tx=0).stop 2024-04-20T22:31:25.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:25.670+0000 7f53aec00640 1 -- 172.21.15.62:0/2342590549 shutdown_connections 2024-04-20T22:31:25.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:25.670+0000 7f53aec00640 1 --2- 172.21.15.62:0/2342590549 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f538c06ecf0 0x7f538c0711a0 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:25.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:25.670+0000 7f53aec00640 1 --2- 172.21.15.62:0/2342590549 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53a8100f30 0x7f53a8195720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:25.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:25.670+0000 7f53aec00640 1 --2- 172.21.15.62:0/2342590549 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f53a8100620 0x7f53a81951e0 unknown :-1 s=CLOSED pgs=1459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:25.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:25.670+0000 7f53aec00640 1 -- 172.21.15.62:0/2342590549 >> 172.21.15.62:0/2342590549 conn(0x7f53a80f9e30 msgr2=0x7f53a8103c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:31:25.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:25.671+0000 7f53aec00640 1 -- 172.21.15.62:0/2342590549 shutdown_connections 2024-04-20T22:31:25.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:25.671+0000 7f53aec00640 1 -- 172.21.15.62:0/2342590549 wait complete. 2024-04-20T22:31:25.688 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-20T21:24:02.500375+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-04-20T21:27:19.577559+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 1, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-04-20T21:27:20.560814+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-20T22:31:25.689 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-20T22:31:25.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:25.888+0000 7f91ada00640 1 -- 172.21.15.62:0/1728108142 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91a8105880 msgr2=0x7f91a8105c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:25.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:25.888+0000 7f91ada00640 1 --2- 172.21.15.62:0/1728108142 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91a8105880 0x7f91a8105c50 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7f9198009880 tx=0x7f919802bcf0 comp rx=0 tx=0).stop 2024-04-20T22:31:25.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:25.889+0000 7f91ada00640 1 -- 172.21.15.62:0/1728108142 shutdown_connections 2024-04-20T22:31:25.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:25.889+0000 7f91ada00640 1 --2- 172.21.15.62:0/1728108142 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91a80ff870 0x7f91a80ffce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:25.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:25.889+0000 7f91ada00640 1 --2- 172.21.15.62:0/1728108142 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91a8105880 0x7f91a8105c50 unknown :-1 s=CLOSED pgs=1460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:25.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:25.889+0000 7f91ada00640 1 -- 172.21.15.62:0/1728108142 >> 172.21.15.62:0/1728108142 conn(0x7f91a80fb370 msgr2=0x7f91a80fd780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:31:25.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:25.889+0000 7f91ada00640 1 -- 172.21.15.62:0/1728108142 shutdown_connections 2024-04-20T22:31:25.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:25.889+0000 7f91ada00640 1 -- 172.21.15.62:0/1728108142 wait complete. 2024-04-20T22:31:25.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:25.890+0000 7f91ada00640 1 Processor -- start 2024-04-20T22:31:25.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:25.890+0000 7f91ada00640 1 -- start start 2024-04-20T22:31:25.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:25.890+0000 7f91ada00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91a80ff870 0x7f91a8197710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:25.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:25.890+0000 7f91ada00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91a8105880 0x7f91a8197c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:25.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:25.890+0000 7f91ada00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91a81982a0 con 0x7f91a8105880 2024-04-20T22:31:25.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:25.890+0000 7f91ada00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91a819b000 con 0x7f91a80ff870 2024-04-20T22:31:25.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:25.890+0000 7f91a6000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91a8105880 0x7f91a8197c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:25.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:25.891+0000 7f91a6000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91a8105880 0x7f91a8197c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53478/0 (socket says 172.21.15.62:53478) 2024-04-20T22:31:25.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:25.891+0000 7f91a6000640 1 -- 172.21.15.62:0/3966558065 learned_addr learned my addr 172.21.15.62:0/3966558065 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:31:25.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:25.891+0000 7f91a6a00640 1 --2- 172.21.15.62:0/3966558065 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91a80ff870 0x7f91a8197710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:25.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:25.891+0000 7f91a6000640 1 -- 172.21.15.62:0/3966558065 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91a80ff870 msgr2=0x7f91a8197710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:25.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:25.891+0000 7f91a6000640 1 --2- 172.21.15.62:0/3966558065 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91a80ff870 0x7f91a8197710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:25.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:25.891+0000 7f91a6000640 1 -- 172.21.15.62:0/3966558065 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9198009560 con 0x7f91a8105880 2024-04-20T22:31:25.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:25.891+0000 7f91a6a00640 1 --2- 172.21.15.62:0/3966558065 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91a80ff870 0x7f91a8197710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T22:31:25.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:25.891+0000 7f91a6000640 1 --2- 172.21.15.62:0/3966558065 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91a8105880 0x7f91a8197c50 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7f919400c8a0 tx=0x7f919400cd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:31:25.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:25.892+0000 7f919f400640 1 -- 172.21.15.62:0/3966558065 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9194007b80 con 0x7f91a8105880 2024-04-20T22:31:25.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:25.892+0000 7f919f400640 1 -- 172.21.15.62:0/3966558065 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9194007ce0 con 0x7f91a8105880 2024-04-20T22:31:25.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:25.892+0000 7f919f400640 1 -- 172.21.15.62:0/3966558065 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9194018810 con 0x7f91a8105880 2024-04-20T22:31:25.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:25.892+0000 7f91ada00640 1 -- 172.21.15.62:0/3966558065 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91a819b2e0 con 0x7f91a8105880 2024-04-20T22:31:25.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:25.892+0000 7f91ada00640 1 -- 172.21.15.62:0/3966558065 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91a819b870 con 0x7f91a8105880 2024-04-20T22:31:25.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:25.893+0000 7f91ada00640 1 -- 172.21.15.62:0/3966558065 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91a804e000 con 0x7f91a8105880 2024-04-20T22:31:25.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:25.894+0000 7f919f400640 1 -- 172.21.15.62:0/3966558065 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f9194018a80 con 0x7f91a8105880 2024-04-20T22:31:25.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:25.895+0000 7f919f400640 1 --2- 172.21.15.62:0/3966558065 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f918c06ec30 0x7f918c0710e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:25.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:25.895+0000 7f919f400640 1 -- 172.21.15.62:0/3966558065 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f919408f120 con 0x7f91a8105880 2024-04-20T22:31:25.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:25.895+0000 7f91a6a00640 1 --2- 172.21.15.62:0/3966558065 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f918c06ec30 0x7f918c0710e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:25.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:25.896+0000 7f91a6a00640 1 --2- 172.21.15.62:0/3966558065 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f918c06ec30 0x7f918c0710e0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f9198036040 tx=0x7f9198005fd0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:31:25.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:25.900+0000 7f919f400640 1 -- 172.21.15.62:0/3966558065 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f9194057e30 con 0x7f91a8105880 2024-04-20T22:31:25.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:31:25 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/4088891697' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 38, "format": "json"}]: dispatch 2024-04-20T22:31:26.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:31:25 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/4088891697' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 38, "format": "json"}]: dispatch 2024-04-20T22:31:26.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:26.097+0000 7f91ada00640 1 -- 172.21.15.62:0/3966558065 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f91a810ca40 con 0x7f91a8105880 2024-04-20T22:31:26.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:26.098+0000 7f919f400640 1 -- 172.21.15.62:0/3966558065 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v85) v1 ==== 93+0+10 (secure 0 0 0) 0x7f919405b4f0 con 0x7f91a8105880 2024-04-20T22:31:26.098 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 1 2024-04-20T22:31:26.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:26.100+0000 7f91ada00640 1 -- 172.21.15.62:0/3966558065 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f918c06ec30 msgr2=0x7f918c0710e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:26.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:26.100+0000 7f91ada00640 1 --2- 172.21.15.62:0/3966558065 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f918c06ec30 0x7f918c0710e0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f9198036040 tx=0x7f9198005fd0 comp rx=0 tx=0).stop 2024-04-20T22:31:26.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:26.100+0000 7f91ada00640 1 -- 172.21.15.62:0/3966558065 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91a8105880 msgr2=0x7f91a8197c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:26.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:26.100+0000 7f91ada00640 1 --2- 172.21.15.62:0/3966558065 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91a8105880 0x7f91a8197c50 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7f919400c8a0 tx=0x7f919400cd60 comp rx=0 tx=0).stop 2024-04-20T22:31:26.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:26.100+0000 7f91ada00640 1 -- 172.21.15.62:0/3966558065 shutdown_connections 2024-04-20T22:31:26.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:26.100+0000 7f91ada00640 1 --2- 172.21.15.62:0/3966558065 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f918c06ec30 0x7f918c0710e0 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:26.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:26.100+0000 7f91ada00640 1 --2- 172.21.15.62:0/3966558065 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91a8105880 0x7f91a8197c50 unknown :-1 s=CLOSED pgs=1461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:26.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:26.100+0000 7f91ada00640 1 --2- 172.21.15.62:0/3966558065 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91a80ff870 0x7f91a8197710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:26.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:26.100+0000 7f91ada00640 1 -- 172.21.15.62:0/3966558065 >> 172.21.15.62:0/3966558065 conn(0x7f91a80fb370 msgr2=0x7f91a80fcb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:31:26.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:26.101+0000 7f91ada00640 1 -- 172.21.15.62:0/3966558065 shutdown_connections 2024-04-20T22:31:26.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:26.101+0000 7f91ada00640 1 -- 172.21.15.62:0/3966558065 wait complete. 2024-04-20T22:31:26.120 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-04-20T22:31:26.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:26.334+0000 7ff951e00640 1 -- 172.21.15.62:0/3112628781 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff94c106090 msgr2=0x7ff94c1064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:26.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:26.334+0000 7ff951e00640 1 --2- 172.21.15.62:0/3112628781 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff94c106090 0x7ff94c1064e0 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7ff938009880 tx=0x7ff93802bce0 comp rx=0 tx=0).stop 2024-04-20T22:31:26.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:26.334+0000 7ff951e00640 1 -- 172.21.15.62:0/3112628781 shutdown_connections 2024-04-20T22:31:26.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:26.334+0000 7ff951e00640 1 --2- 172.21.15.62:0/3112628781 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff94c106090 0x7ff94c1064e0 unknown :-1 s=CLOSED pgs=1462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:26.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:26.334+0000 7ff951e00640 1 --2- 172.21.15.62:0/3112628781 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff94c10bb60 0x7ff94c10bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:26.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:26.334+0000 7ff951e00640 1 -- 172.21.15.62:0/3112628781 >> 172.21.15.62:0/3112628781 conn(0x7ff94c075260 msgr2=0x7ff94c077670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:31:26.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:26.334+0000 7ff951e00640 1 -- 172.21.15.62:0/3112628781 shutdown_connections 2024-04-20T22:31:26.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:26.334+0000 7ff951e00640 1 -- 172.21.15.62:0/3112628781 wait complete. 2024-04-20T22:31:26.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:26.335+0000 7ff951e00640 1 Processor -- start 2024-04-20T22:31:26.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:26.336+0000 7ff951e00640 1 -- start start 2024-04-20T22:31:26.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:26.336+0000 7ff951e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff94c106090 0x7ff94c1a1620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:26.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:26.336+0000 7ff951e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff94c10bb60 0x7ff94c1a1b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:26.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:26.336+0000 7ff951e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff94c19b690 con 0x7ff94c10bb60 2024-04-20T22:31:26.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:26.336+0000 7ff951e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff94c19b800 con 0x7ff94c106090 2024-04-20T22:31:26.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:26.336+0000 7ff94be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff94c10bb60 0x7ff94c1a1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:26.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:26.336+0000 7ff94be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff94c10bb60 0x7ff94c1a1b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53504/0 (socket says 172.21.15.62:53504) 2024-04-20T22:31:26.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:26.336+0000 7ff94be00640 1 -- 172.21.15.62:0/1828267440 learned_addr learned my addr 172.21.15.62:0/1828267440 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:31:26.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:26.337+0000 7ff94be00640 1 -- 172.21.15.62:0/1828267440 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff94c106090 msgr2=0x7ff94c1a1620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:26.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:26.337+0000 7ff94be00640 1 --2- 172.21.15.62:0/1828267440 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff94c106090 0x7ff94c1a1620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:26.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:26.337+0000 7ff94be00640 1 -- 172.21.15.62:0/1828267440 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff938009560 con 0x7ff94c10bb60 2024-04-20T22:31:26.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:26.337+0000 7ff94be00640 1 --2- 172.21.15.62:0/1828267440 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff94c10bb60 0x7ff94c1a1b60 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7ff938000c00 tx=0x7ff93802c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:31:26.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:26.337+0000 7ff949600640 1 -- 172.21.15.62:0/1828267440 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff938004520 con 0x7ff94c10bb60 2024-04-20T22:31:26.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:26.337+0000 7ff951e00640 1 -- 172.21.15.62:0/1828267440 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff94c19ba80 con 0x7ff94c10bb60 2024-04-20T22:31:26.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:26.337+0000 7ff949600640 1 -- 172.21.15.62:0/1828267440 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff938003ab0 con 0x7ff94c10bb60 2024-04-20T22:31:26.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:26.337+0000 7ff949600640 1 -- 172.21.15.62:0/1828267440 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff938034440 con 0x7ff94c10bb60 2024-04-20T22:31:26.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:26.338+0000 7ff951e00640 1 -- 172.21.15.62:0/1828267440 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff94c19bf80 con 0x7ff94c10bb60 2024-04-20T22:31:26.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:26.339+0000 7ff951e00640 1 -- 172.21.15.62:0/1828267440 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff94c04e000 con 0x7ff94c10bb60 2024-04-20T22:31:26.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:26.340+0000 7ff949600640 1 -- 172.21.15.62:0/1828267440 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff938003c20 con 0x7ff94c10bb60 2024-04-20T22:31:26.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:26.341+0000 7ff949600640 1 --2- 172.21.15.62:0/1828267440 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff93406ec30 0x7ff9340710e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:26.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:26.341+0000 7ff949600640 1 -- 172.21.15.62:0/1828267440 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7ff9380b11c0 con 0x7ff94c10bb60 2024-04-20T22:31:26.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:26.341+0000 7ff950a00640 1 --2- 172.21.15.62:0/1828267440 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff93406ec30 0x7ff9340710e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:26.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:26.342+0000 7ff950a00640 1 --2- 172.21.15.62:0/1828267440 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff93406ec30 0x7ff9340710e0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7ff940005fd0 tx=0x7ff940005850 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:31:26.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:26.345+0000 7ff949600640 1 -- 172.21.15.62:0/1828267440 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7ff938079f50 con 0x7ff94c10bb60 2024-04-20T22:31:26.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:26.555+0000 7ff951e00640 1 -- 172.21.15.62:0/1828267440 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7ff94c10def0 con 0x7ff94c10bb60 2024-04-20T22:31:26.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:26.556+0000 7ff949600640 1 -- 172.21.15.62:0/1828267440 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v85) v1 ==== 107+0+11 (secure 0 0 0) 0x7ff94c10def0 con 0x7ff94c10bb60 2024-04-20T22:31:26.556 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 32 2024-04-20T22:31:26.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:26.558+0000 7ff951e00640 1 -- 172.21.15.62:0/1828267440 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff93406ec30 msgr2=0x7ff9340710e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:26.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:26.559+0000 7ff951e00640 1 --2- 172.21.15.62:0/1828267440 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff93406ec30 0x7ff9340710e0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7ff940005fd0 tx=0x7ff940005850 comp rx=0 tx=0).stop 2024-04-20T22:31:26.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:26.559+0000 7ff951e00640 1 -- 172.21.15.62:0/1828267440 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff94c10bb60 msgr2=0x7ff94c1a1b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:26.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:26.559+0000 7ff951e00640 1 --2- 172.21.15.62:0/1828267440 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff94c10bb60 0x7ff94c1a1b60 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7ff938000c00 tx=0x7ff93802c850 comp rx=0 tx=0).stop 2024-04-20T22:31:26.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:26.559+0000 7ff951e00640 1 -- 172.21.15.62:0/1828267440 shutdown_connections 2024-04-20T22:31:26.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:26.559+0000 7ff951e00640 1 --2- 172.21.15.62:0/1828267440 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7ff93406ec30 0x7ff9340710e0 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:26.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:26.559+0000 7ff951e00640 1 --2- 172.21.15.62:0/1828267440 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff94c10bb60 0x7ff94c1a1b60 unknown :-1 s=CLOSED pgs=1463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:26.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:26.559+0000 7ff951e00640 1 --2- 172.21.15.62:0/1828267440 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff94c106090 0x7ff94c1a1620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:26.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:26.559+0000 7ff951e00640 1 -- 172.21.15.62:0/1828267440 >> 172.21.15.62:0/1828267440 conn(0x7ff94c075260 msgr2=0x7ff94c076d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:31:26.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:26.559+0000 7ff951e00640 1 -- 172.21.15.62:0/1828267440 shutdown_connections 2024-04-20T22:31:26.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:26.559+0000 7ff951e00640 1 -- 172.21.15.62:0/1828267440 wait complete. 2024-04-20T22:31:26.579 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-04-20T22:31:26.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:26.798+0000 7f7fd5200640 1 -- 172.21.15.62:0/3484195902 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7fd0105480 msgr2=0x7f7fd0105850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:26.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:26.798+0000 7f7fd5200640 1 --2- 172.21.15.62:0/3484195902 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7fd0105480 0x7f7fd0105850 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7f7fc0009c20 tx=0x7f7fc002c1b0 comp rx=0 tx=0).stop 2024-04-20T22:31:26.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:26.799+0000 7f7fd5200640 1 -- 172.21.15.62:0/3484195902 shutdown_connections 2024-04-20T22:31:26.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:26.799+0000 7f7fd5200640 1 --2- 172.21.15.62:0/3484195902 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7fd00ff470 0x7f7fd00ff8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:26.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:26.799+0000 7f7fd5200640 1 --2- 172.21.15.62:0/3484195902 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7fd0105480 0x7f7fd0105850 unknown :-1 s=CLOSED pgs=1464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:26.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:26.799+0000 7f7fd5200640 1 -- 172.21.15.62:0/3484195902 >> 172.21.15.62:0/3484195902 conn(0x7f7fd00faf70 msgr2=0x7f7fd00fd380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:31:26.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:26.799+0000 7f7fd5200640 1 -- 172.21.15.62:0/3484195902 shutdown_connections 2024-04-20T22:31:26.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:26.799+0000 7f7fd5200640 1 -- 172.21.15.62:0/3484195902 wait complete. 2024-04-20T22:31:26.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:26.800+0000 7f7fd5200640 1 Processor -- start 2024-04-20T22:31:26.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:26.800+0000 7f7fd5200640 1 -- start start 2024-04-20T22:31:26.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:26.800+0000 7f7fd5200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7fd00ff470 0x7f7fd0191e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:26.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:26.800+0000 7f7fd5200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7fd0105480 0x7f7fd01923b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:26.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:26.800+0000 7f7fd5200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fd0196010 con 0x7f7fd0105480 2024-04-20T22:31:26.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:26.800+0000 7f7fd5200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fd01928f0 con 0x7f7fd00ff470 2024-04-20T22:31:26.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:26.800+0000 7f7fcea00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7fd0105480 0x7f7fd01923b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:26.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:26.800+0000 7f7fcea00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7fd0105480 0x7f7fd01923b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53510/0 (socket says 172.21.15.62:53510) 2024-04-20T22:31:26.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:26.800+0000 7f7fcea00640 1 -- 172.21.15.62:0/3886227876 learned_addr learned my addr 172.21.15.62:0/3886227876 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:31:26.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:26.801+0000 7f7fcf400640 1 --2- 172.21.15.62:0/3886227876 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7fd00ff470 0x7f7fd0191e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:26.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:26.801+0000 7f7fcea00640 1 -- 172.21.15.62:0/3886227876 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7fd00ff470 msgr2=0x7f7fd0191e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:26.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:26.801+0000 7f7fcea00640 1 --2- 172.21.15.62:0/3886227876 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7fd00ff470 0x7f7fd0191e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:26.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:26.801+0000 7f7fcea00640 1 -- 172.21.15.62:0/3886227876 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fc0009900 con 0x7f7fd0105480 2024-04-20T22:31:26.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:26.801+0000 7f7fcf400640 1 --2- 172.21.15.62:0/3886227876 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7fd00ff470 0x7f7fd0191e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T22:31:26.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:26.801+0000 7f7fcea00640 1 --2- 172.21.15.62:0/3886227876 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7fd0105480 0x7f7fd01923b0 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7f7fb800d6f0 tx=0x7f7fb800dbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:31:26.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:26.802+0000 7f7fc7e00640 1 -- 172.21.15.62:0/3886227876 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7fb8009a30 con 0x7f7fd0105480 2024-04-20T22:31:26.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:26.802+0000 7f7fc7e00640 1 -- 172.21.15.62:0/3886227876 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7fb80054e0 con 0x7f7fd0105480 2024-04-20T22:31:26.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:26.802+0000 7f7fc7e00640 1 -- 172.21.15.62:0/3886227876 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7fb8010460 con 0x7f7fd0105480 2024-04-20T22:31:26.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:26.802+0000 7f7fd5200640 1 -- 172.21.15.62:0/3886227876 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fd0192b70 con 0x7f7fd0105480 2024-04-20T22:31:26.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:26.802+0000 7f7fd5200640 1 -- 172.21.15.62:0/3886227876 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fd01a5b20 con 0x7f7fd0105480 2024-04-20T22:31:26.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:26.803+0000 7f7fd5200640 1 -- 172.21.15.62:0/3886227876 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fd004e000 con 0x7f7fd0105480 2024-04-20T22:31:26.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:26.804+0000 7f7fc7e00640 1 -- 172.21.15.62:0/3886227876 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f7fb8009b90 con 0x7f7fd0105480 2024-04-20T22:31:26.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:26.805+0000 7f7fc7e00640 1 --2- 172.21.15.62:0/3886227876 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7fb406ec30 0x7f7fb40710e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:26.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:26.805+0000 7f7fc7e00640 1 -- 172.21.15.62:0/3886227876 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f7fb808e9f0 con 0x7f7fd0105480 2024-04-20T22:31:26.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:26.806+0000 7f7fcf400640 1 --2- 172.21.15.62:0/3886227876 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7fb406ec30 0x7f7fb40710e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:26.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:26.806+0000 7f7fcf400640 1 --2- 172.21.15.62:0/3886227876 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7fb406ec30 0x7f7fb40710e0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f7fc0009430 tx=0x7f7fc0009360 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:31:26.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:26.810+0000 7f7fc7e00640 1 -- 172.21.15.62:0/3886227876 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f7fb8057780 con 0x7f7fd0105480 2024-04-20T22:31:26.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:31:26 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/2342590549' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-20T22:31:26.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:31:26 smithi189 ceph-mon[57022]: pgmap v1689: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:31:26.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:31:26 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/3966558065' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-20T22:31:26.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:31:26 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/1828267440' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-20T22:31:27.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.018+0000 7f7fd5200640 1 -- 172.21.15.62:0/3886227876 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7f7fd010c660 con 0x7f7fd0105480 2024-04-20T22:31:27.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.019+0000 7f7fc7e00640 1 -- 172.21.15.62:0/3886227876 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v85) v1 ==== 107+0+11 (secure 0 0 0) 0x7f7fb8020020 con 0x7f7fd0105480 2024-04-20T22:31:27.019 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 32 2024-04-20T22:31:27.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.022+0000 7f7fd5200640 1 -- 172.21.15.62:0/3886227876 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7fb406ec30 msgr2=0x7f7fb40710e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:27.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.022+0000 7f7fd5200640 1 --2- 172.21.15.62:0/3886227876 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7fb406ec30 0x7f7fb40710e0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f7fc0009430 tx=0x7f7fc0009360 comp rx=0 tx=0).stop 2024-04-20T22:31:27.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.022+0000 7f7fd5200640 1 -- 172.21.15.62:0/3886227876 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7fd0105480 msgr2=0x7f7fd01923b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:27.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.022+0000 7f7fd5200640 1 --2- 172.21.15.62:0/3886227876 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7fd0105480 0x7f7fd01923b0 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7f7fb800d6f0 tx=0x7f7fb800dbb0 comp rx=0 tx=0).stop 2024-04-20T22:31:27.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.022+0000 7f7fd5200640 1 -- 172.21.15.62:0/3886227876 shutdown_connections 2024-04-20T22:31:27.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.022+0000 7f7fd5200640 1 --2- 172.21.15.62:0/3886227876 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f7fb406ec30 0x7f7fb40710e0 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:27.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.022+0000 7f7fd5200640 1 --2- 172.21.15.62:0/3886227876 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7fd0105480 0x7f7fd01923b0 unknown :-1 s=CLOSED pgs=1465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:27.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.022+0000 7f7fd5200640 1 --2- 172.21.15.62:0/3886227876 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7fd00ff470 0x7f7fd0191e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:27.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.022+0000 7f7fd5200640 1 -- 172.21.15.62:0/3886227876 >> 172.21.15.62:0/3886227876 conn(0x7f7fd00faf70 msgr2=0x7f7fd00fb910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:31:27.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.022+0000 7f7fd5200640 1 -- 172.21.15.62:0/3886227876 shutdown_connections 2024-04-20T22:31:27.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.022+0000 7f7fd5200640 1 -- 172.21.15.62:0/3886227876 wait complete. 2024-04-20T22:31:27.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:31:26 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/2342590549' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-20T22:31:27.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:31:26 smithi062 ceph-mon[76322]: pgmap v1689: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:31:27.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:31:26 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/3966558065' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-20T22:31:27.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:31:26 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/1828267440' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-20T22:31:27.040 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-20T22:31:27.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.251+0000 7f3a4e600640 1 -- 172.21.15.62:0/2890424335 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3a48107690 msgr2=0x7f3a48107a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:27.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.251+0000 7f3a4e600640 1 --2- 172.21.15.62:0/2890424335 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3a48107690 0x7f3a48107a60 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7f3a38009880 tx=0x7f3a3802bce0 comp rx=0 tx=0).stop 2024-04-20T22:31:27.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.251+0000 7f3a4e600640 1 -- 172.21.15.62:0/2890424335 shutdown_connections 2024-04-20T22:31:27.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.251+0000 7f3a4e600640 1 --2- 172.21.15.62:0/2890424335 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a48101680 0x7f3a48101af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:27.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.251+0000 7f3a4e600640 1 --2- 172.21.15.62:0/2890424335 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3a48107690 0x7f3a48107a60 unknown :-1 s=CLOSED pgs=1466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:27.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.251+0000 7f3a4e600640 1 -- 172.21.15.62:0/2890424335 >> 172.21.15.62:0/2890424335 conn(0x7f3a480fd160 msgr2=0x7f3a480ff570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:31:27.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.251+0000 7f3a4e600640 1 -- 172.21.15.62:0/2890424335 shutdown_connections 2024-04-20T22:31:27.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.251+0000 7f3a4e600640 1 -- 172.21.15.62:0/2890424335 wait complete. 2024-04-20T22:31:27.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.252+0000 7f3a4e600640 1 Processor -- start 2024-04-20T22:31:27.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.253+0000 7f3a4e600640 1 -- start start 2024-04-20T22:31:27.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.253+0000 7f3a4e600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a48101680 0x7f3a481972a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:27.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.253+0000 7f3a4e600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3a48107690 0x7f3a481977e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:27.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.253+0000 7f3a4e600640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a48197e70 con 0x7f3a48107690 2024-04-20T22:31:27.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.253+0000 7f3a4e600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a4819bc40 con 0x7f3a48101680 2024-04-20T22:31:27.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.253+0000 7f3a47e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3a48107690 0x7f3a481977e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:27.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.254+0000 7f3a47e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3a48107690 0x7f3a481977e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53528/0 (socket says 172.21.15.62:53528) 2024-04-20T22:31:27.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.254+0000 7f3a47e00640 1 -- 172.21.15.62:0/3567450713 learned_addr learned my addr 172.21.15.62:0/3567450713 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:31:27.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.254+0000 7f3a4d200640 1 --2- 172.21.15.62:0/3567450713 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a48101680 0x7f3a481972a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:27.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.254+0000 7f3a47e00640 1 -- 172.21.15.62:0/3567450713 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a48101680 msgr2=0x7f3a481972a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:27.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.254+0000 7f3a47e00640 1 --2- 172.21.15.62:0/3567450713 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a48101680 0x7f3a481972a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:27.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.254+0000 7f3a47e00640 1 -- 172.21.15.62:0/3567450713 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a38009560 con 0x7f3a48107690 2024-04-20T22:31:27.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.254+0000 7f3a47e00640 1 --2- 172.21.15.62:0/3567450713 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3a48107690 0x7f3a481977e0 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7f3a3400ea00 tx=0x7f3a3400eec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:31:27.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.255+0000 7f3a45600640 1 -- 172.21.15.62:0/3567450713 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3a3400cd50 con 0x7f3a48107690 2024-04-20T22:31:27.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.255+0000 7f3a45600640 1 -- 172.21.15.62:0/3567450713 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3a34005020 con 0x7f3a48107690 2024-04-20T22:31:27.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.255+0000 7f3a45600640 1 -- 172.21.15.62:0/3567450713 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3a340116a0 con 0x7f3a48107690 2024-04-20T22:31:27.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.255+0000 7f3a4e600640 1 -- 172.21.15.62:0/3567450713 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a4819bf20 con 0x7f3a48107690 2024-04-20T22:31:27.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.255+0000 7f3a4e600640 1 -- 172.21.15.62:0/3567450713 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a4819c500 con 0x7f3a48107690 2024-04-20T22:31:27.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.256+0000 7f3a4e600640 1 -- 172.21.15.62:0/3567450713 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a4804e850 con 0x7f3a48107690 2024-04-20T22:31:27.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.258+0000 7f3a45600640 1 -- 172.21.15.62:0/3567450713 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f3a34005190 con 0x7f3a48107690 2024-04-20T22:31:27.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.259+0000 7f3a45600640 1 --2- 172.21.15.62:0/3567450713 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3a2c06ec80 0x7f3a2c071130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:27.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.259+0000 7f3a45600640 1 -- 172.21.15.62:0/3567450713 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f3a3408f150 con 0x7f3a48107690 2024-04-20T22:31:27.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.259+0000 7f3a4d200640 1 --2- 172.21.15.62:0/3567450713 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3a2c06ec80 0x7f3a2c071130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:27.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.260+0000 7f3a4d200640 1 --2- 172.21.15.62:0/3567450713 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3a2c06ec80 0x7f3a2c071130 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f3a38036040 tx=0x7f3a38005fd0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:31:27.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.264+0000 7f3a45600640 1 -- 172.21.15.62:0/3567450713 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f3a34057f80 con 0x7f3a48107690 2024-04-20T22:31:27.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.483+0000 7f3a4e600640 1 -- 172.21.15.62:0/3567450713 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f3a480008d0 con 0x7f3a48107690 2024-04-20T22:31:27.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.485+0000 7f3a45600640 1 -- 172.21.15.62:0/3567450713 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 39 entries v85) v1 ==== 81+0+2306 (secure 0 0 0) 0x7f3a3401f920 con 0x7f3a48107690 2024-04-20T22:31:27.485 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.189:6825/237262514 2024-04-21T21:42:01.558630+0000 2024-04-20T22:31:27.485 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:0/2348636520 2024-04-21T21:30:35.075229+0000 2024-04-20T22:31:27.485 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:6801/871423274 2024-04-21T21:18:52.990685+0000 2024-04-20T22:31:27.485 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:6801/831506538 2024-04-21T21:21:03.778739+0000 2024-04-20T22:31:27.485 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:0/4251078301 2024-04-21T21:36:34.221905+0000 2024-04-20T22:31:27.485 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.189:0/1714270321 2024-04-21T21:31:36.545789+0000 2024-04-20T22:31:27.485 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:6826/1147127123 2024-04-21T21:42:01.558599+0000 2024-04-20T22:31:27.486 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:0/868116748 2024-04-21T21:30:35.075229+0000 2024-04-20T22:31:27.486 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:0/2525292829 2024-04-21T21:18:52.990685+0000 2024-04-20T22:31:27.486 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:0/1116048562 2024-04-21T21:21:03.778739+0000 2024-04-20T22:31:27.486 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:0/634566145 2024-04-21T21:18:52.990685+0000 2024-04-20T22:31:27.486 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:6800/831506538 2024-04-21T21:21:03.778739+0000 2024-04-20T22:31:27.486 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.189:0/4026364416 2024-04-21T21:31:36.545789+0000 2024-04-20T22:31:27.486 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:0/1789120375 2024-04-21T21:36:34.221905+0000 2024-04-20T22:31:27.486 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:0/3416563026 2024-04-21T21:18:52.990685+0000 2024-04-20T22:31:27.486 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:0/3673769314 2024-04-21T21:30:35.075229+0000 2024-04-20T22:31:27.486 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:0/717708557 2024-04-21T21:30:35.075229+0000 2024-04-20T22:31:27.486 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:0/4274508569 2024-04-21T21:19:34.965981+0000 2024-04-20T22:31:27.486 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.189:0/3430221175 2024-04-21T21:31:36.545789+0000 2024-04-20T22:31:27.486 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.189:6824/237262514 2024-04-21T21:42:01.558630+0000 2024-04-20T22:31:27.486 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.189:0/100133912 2024-04-21T21:31:36.545789+0000 2024-04-20T22:31:27.487 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.189:0/441007232 2024-04-21T21:31:36.545789+0000 2024-04-20T22:31:27.487 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:6800/4116451241 2024-04-21T21:30:35.075229+0000 2024-04-20T22:31:27.487 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:6827/1147127123 2024-04-21T21:42:01.558599+0000 2024-04-20T22:31:27.487 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:6801/4116451241 2024-04-21T21:30:35.075229+0000 2024-04-20T22:31:27.487 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:6800/871423274 2024-04-21T21:18:52.990685+0000 2024-04-20T22:31:27.487 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.189:6828/2010473407 2024-04-21T21:31:36.545789+0000 2024-04-20T22:31:27.487 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.189:6829/2010473407 2024-04-21T21:31:36.545789+0000 2024-04-20T22:31:27.487 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.189:0/1726963148 2024-04-21T21:31:36.545789+0000 2024-04-20T22:31:27.487 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:0/3192460755 2024-04-21T21:21:03.778739+0000 2024-04-20T22:31:27.487 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:6801/4147738586 2024-04-21T21:19:34.965981+0000 2024-04-20T22:31:27.487 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:0/4158176989 2024-04-21T21:19:34.965981+0000 2024-04-20T22:31:27.487 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:0/301716985 2024-04-21T21:21:03.778739+0000 2024-04-20T22:31:27.487 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:0/678960466 2024-04-21T21:36:34.221905+0000 2024-04-20T22:31:27.487 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:0/2136121731 2024-04-21T21:36:34.221905+0000 2024-04-20T22:31:27.488 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:6800/4147738586 2024-04-21T21:19:34.965981+0000 2024-04-20T22:31:27.488 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:6800/1585760929 2024-04-21T21:36:34.221905+0000 2024-04-20T22:31:27.488 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:0/771955286 2024-04-21T21:19:34.965981+0000 2024-04-20T22:31:27.488 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:6801/1585760929 2024-04-21T21:36:34.221905+0000 2024-04-20T22:31:27.489 INFO:teuthology.orchestra.run.smithi062.stderr:listed 39 entries 2024-04-20T22:31:27.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.488+0000 7f3a4e600640 1 -- 172.21.15.62:0/3567450713 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3a2c06ec80 msgr2=0x7f3a2c071130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:27.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.488+0000 7f3a4e600640 1 --2- 172.21.15.62:0/3567450713 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3a2c06ec80 0x7f3a2c071130 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f3a38036040 tx=0x7f3a38005fd0 comp rx=0 tx=0).stop 2024-04-20T22:31:27.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.488+0000 7f3a4e600640 1 -- 172.21.15.62:0/3567450713 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3a48107690 msgr2=0x7f3a481977e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:27.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.488+0000 7f3a4e600640 1 --2- 172.21.15.62:0/3567450713 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3a48107690 0x7f3a481977e0 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7f3a3400ea00 tx=0x7f3a3400eec0 comp rx=0 tx=0).stop 2024-04-20T22:31:27.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.488+0000 7f3a4e600640 1 -- 172.21.15.62:0/3567450713 shutdown_connections 2024-04-20T22:31:27.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.488+0000 7f3a4e600640 1 --2- 172.21.15.62:0/3567450713 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3a2c06ec80 0x7f3a2c071130 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:27.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.488+0000 7f3a4e600640 1 --2- 172.21.15.62:0/3567450713 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3a48107690 0x7f3a481977e0 unknown :-1 s=CLOSED pgs=1467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:27.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.488+0000 7f3a4e600640 1 --2- 172.21.15.62:0/3567450713 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a48101680 0x7f3a481972a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:27.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.488+0000 7f3a4e600640 1 -- 172.21.15.62:0/3567450713 >> 172.21.15.62:0/3567450713 conn(0x7f3a480fd160 msgr2=0x7f3a480fe970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:31:27.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.488+0000 7f3a4e600640 1 -- 172.21.15.62:0/3567450713 shutdown_connections 2024-04-20T22:31:27.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.488+0000 7f3a4e600640 1 -- 172.21.15.62:0/3567450713 wait complete. 2024-04-20T22:31:27.509 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-04-20T22:31:27.510 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-20T22:31:27.564 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-20T22:31:27.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.756+0000 7f3b1a800640 1 -- 172.21.15.62:0/4148628910 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3b14101870 msgr2=0x7f3b14101ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:27.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.756+0000 7f3b1a800640 1 --2- 172.21.15.62:0/4148628910 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3b14101870 0x7f3b14101ce0 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7f3afc009880 tx=0x7f3afc02bcf0 comp rx=0 tx=0).stop 2024-04-20T22:31:27.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.756+0000 7f3b1a800640 1 -- 172.21.15.62:0/4148628910 shutdown_connections 2024-04-20T22:31:27.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.756+0000 7f3b1a800640 1 --2- 172.21.15.62:0/4148628910 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3b14101870 0x7f3b14101ce0 unknown :-1 s=CLOSED pgs=1468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:27.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.756+0000 7f3b1a800640 1 --2- 172.21.15.62:0/4148628910 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b14107880 0x7f3b14107c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:27.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.756+0000 7f3b1a800640 1 -- 172.21.15.62:0/4148628910 >> 172.21.15.62:0/4148628910 conn(0x7f3b140fd370 msgr2=0x7f3b140ff780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:31:27.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.756+0000 7f3b1a800640 1 -- 172.21.15.62:0/4148628910 shutdown_connections 2024-04-20T22:31:27.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.756+0000 7f3b1a800640 1 -- 172.21.15.62:0/4148628910 wait complete. 2024-04-20T22:31:27.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.757+0000 7f3b1a800640 1 Processor -- start 2024-04-20T22:31:27.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.757+0000 7f3b1a800640 1 -- start start 2024-04-20T22:31:27.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.758+0000 7f3b1a800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b14101870 0x7f3b14197390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:27.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.758+0000 7f3b1a800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3b14107880 0x7f3b141978d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:27.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.758+0000 7f3b1a800640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b14197ed0 con 0x7f3b14107880 2024-04-20T22:31:27.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.758+0000 7f3b1a800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b14198040 con 0x7f3b14101870 2024-04-20T22:31:27.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.758+0000 7f3b0aa00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3b14107880 0x7f3b141978d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:27.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.758+0000 7f3b0aa00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3b14107880 0x7f3b141978d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53552/0 (socket says 172.21.15.62:53552) 2024-04-20T22:31:27.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.758+0000 7f3b0aa00640 1 -- 172.21.15.62:0/2021562814 learned_addr learned my addr 172.21.15.62:0/2021562814 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:31:27.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.758+0000 7f3b13400640 1 --2- 172.21.15.62:0/2021562814 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b14101870 0x7f3b14197390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:27.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.759+0000 7f3b0aa00640 1 -- 172.21.15.62:0/2021562814 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b14101870 msgr2=0x7f3b14197390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:27.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.759+0000 7f3b0aa00640 1 --2- 172.21.15.62:0/2021562814 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b14101870 0x7f3b14197390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:27.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.759+0000 7f3b0aa00640 1 -- 172.21.15.62:0/2021562814 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3afc009560 con 0x7f3b14107880 2024-04-20T22:31:27.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.759+0000 7f3b0aa00640 1 --2- 172.21.15.62:0/2021562814 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3b14107880 0x7f3b141978d0 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7f3afc036040 tx=0x7f3afc004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:31:27.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.759+0000 7f3b10c00640 1 -- 172.21.15.62:0/2021562814 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3afc0035f0 con 0x7f3b14107880 2024-04-20T22:31:27.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.759+0000 7f3b10c00640 1 -- 172.21.15.62:0/2021562814 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3afc003c10 con 0x7f3b14107880 2024-04-20T22:31:27.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.759+0000 7f3b10c00640 1 -- 172.21.15.62:0/2021562814 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3afc034440 con 0x7f3b14107880 2024-04-20T22:31:27.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.759+0000 7f3b1a800640 1 -- 172.21.15.62:0/2021562814 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b1419be70 con 0x7f3b14107880 2024-04-20T22:31:27.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.760+0000 7f3b1a800640 1 -- 172.21.15.62:0/2021562814 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b1419c400 con 0x7f3b14107880 2024-04-20T22:31:27.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.761+0000 7f3b1a800640 1 -- 172.21.15.62:0/2021562814 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b1404e000 con 0x7f3b14107880 2024-04-20T22:31:27.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.762+0000 7f3b10c00640 1 -- 172.21.15.62:0/2021562814 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f3afc005060 con 0x7f3b14107880 2024-04-20T22:31:27.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.763+0000 7f3b10c00640 1 --2- 172.21.15.62:0/2021562814 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3af806ecf0 0x7f3af80711a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:27.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.763+0000 7f3b10c00640 1 -- 172.21.15.62:0/2021562814 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f3afc0b1300 con 0x7f3b14107880 2024-04-20T22:31:27.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.763+0000 7f3b13400640 1 --2- 172.21.15.62:0/2021562814 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3af806ecf0 0x7f3af80711a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:27.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.764+0000 7f3b13400640 1 --2- 172.21.15.62:0/2021562814 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3af806ecf0 0x7f3af80711a0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f3b04005c40 tx=0x7f3b04005bd0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:31:27.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.768+0000 7f3b10c00640 1 -- 172.21.15.62:0/2021562814 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f3afc07a0b0 con 0x7f3b14107880 2024-04-20T22:31:27.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:31:27 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/3886227876' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-20T22:31:27.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:31:27 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/3567450713' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-20T22:31:27.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.976+0000 7f3b1a800640 1 -- 172.21.15.62:0/2021562814 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3b140008d0 con 0x7f3b14107880 2024-04-20T22:31:27.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.980+0000 7f3b10c00640 1 -- 172.21.15.62:0/2021562814 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+16652 (secure 0 0 0) 0x7f3afc07d770 con 0x7f3b14107880 2024-04-20T22:31:27.980 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-20T22:31:27.980 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":85,"fsid":"61ae3628-ff5b-11ee-bc93-c7b262605968","created":"2024-04-20T21:18:19.951305+0000","modified":"2024-04-20T22:13:07.093942+0000","last_up_change":"2024-04-20T21:41:54.404166+0000","last_in_change":"2024-04-20T21:25:19.201764+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":20,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-20T21:24:02.500375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-04-20T21:27:19.577559+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-04-20T21:27:20.560814+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a3f8216-c641-4338-813d-d836efc3f4e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":48,"up_from":52,"up_thru":77,"down_at":49,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":1209367302},{"type":"v1","addr":"172.21.15.62:6803","nonce":1209367302}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":1209367302},{"type":"v1","addr":"172.21.15.62:6805","nonce":1209367302}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":1209367302},{"type":"v1","addr":"172.21.15.62:6809","nonce":1209367302}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":1209367302},{"type":"v1","addr":"172.21.15.62:6807","nonce":1209367302}]},"public_addr":"172.21.15.62:6803/1209367302","cluster_addr":"172.21.15.62:6805/1209367302","heartbeat_back_addr":"172.21.15.62:6809/1209367302","heartbeat_front_addr":"172.21.15.62:6807/1209367302","state":["exists","up"]},{"osd":1,"uuid":"f0c451a4-9002-4ddc-be0e-d8ddebd4418d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":14,"last_clean_end":53,"up_from":57,"up_thru":77,"down_at":54,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":2879835300},{"type":"v1","addr":"172.21.15.62:6811","nonce":2879835300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":2879835300},{"type":"v1","addr":"172.21.15.62:6813","nonce":2879835300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":2879835300},{"type":"v1","addr":"172.21.15.62:6817","nonce":2879835300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":2879835300},{"type":"v1","addr":"172.21.15.62:6815","nonce":2879835300}]},"public_addr":"172.21.15.62:6811/2879835300","cluster_addr":"172.21.15.62:6813/2879835300","heartbeat_back_addr":"172.21.15.62:6817/2879835300","heartbeat_front_addr":"172.21.15.62:6815/2879835300","state":["exists","up"]},{"osd":2,"uuid":"e1a8769d-5da2-49c8-942a-83a94c7ac761","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":19,"last_clean_end":58,"up_from":62,"up_thru":77,"down_at":59,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":4264123649},{"type":"v1","addr":"172.21.15.62:6819","nonce":4264123649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":4264123649},{"type":"v1","addr":"172.21.15.62:6821","nonce":4264123649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":4264123649},{"type":"v1","addr":"172.21.15.62:6825","nonce":4264123649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6822","nonce":4264123649},{"type":"v1","addr":"172.21.15.62:6823","nonce":4264123649}]},"public_addr":"172.21.15.62:6819/4264123649","cluster_addr":"172.21.15.62:6821/4264123649","heartbeat_back_addr":"172.21.15.62:6825/4264123649","heartbeat_front_addr":"172.21.15.62:6823/4264123649","state":["exists","up"]},{"osd":3,"uuid":"eaa2fe2b-87f8-4d01-8ddf-e72f244f31b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":63,"up_from":67,"up_thru":77,"down_at":64,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":3977023570},{"type":"v1","addr":"172.21.15.189:6801","nonce":3977023570}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":3977023570},{"type":"v1","addr":"172.21.15.189:6803","nonce":3977023570}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":3977023570},{"type":"v1","addr":"172.21.15.189:6807","nonce":3977023570}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":3977023570},{"type":"v1","addr":"172.21.15.189:6805","nonce":3977023570}]},"public_addr":"172.21.15.189:6801/3977023570","cluster_addr":"172.21.15.189:6803/3977023570","heartbeat_back_addr":"172.21.15.189:6807/3977023570","heartbeat_front_addr":"172.21.15.189:6805/3977023570","state":["exists","up"]},{"osd":4,"uuid":"72be8597-52e5-45fd-b4f2-06ac50df77e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":34,"last_clean_end":68,"up_from":72,"up_thru":77,"down_at":69,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":2794032930},{"type":"v1","addr":"172.21.15.189:6809","nonce":2794032930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":2794032930},{"type":"v1","addr":"172.21.15.189:6811","nonce":2794032930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":2794032930},{"type":"v1","addr":"172.21.15.189:6815","nonce":2794032930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":2794032930},{"type":"v1","addr":"172.21.15.189:6813","nonce":2794032930}]},"public_addr":"172.21.15.189:6809/2794032930","cluster_addr":"172.21.15.189:6811/2794032930","heartbeat_back_addr":"172.21.15.189:6815/2794032930","heartbeat_front_addr":"172.21.15.189:6813/2794032930","state":["exists","up"]},{"osd":5,"uuid":"c650c63b-41eb-4d4f-8e14-907c7b3614c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":39,"last_clean_end":73,"up_from":77,"up_thru":77,"down_at":74,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2760890917},{"type":"v1","addr":"172.21.15.189:6817","nonce":2760890917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2760890917},{"type":"v1","addr":"172.21.15.189:6819","nonce":2760890917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2760890917},{"type":"v1","addr":"172.21.15.189:6823","nonce":2760890917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2760890917},{"type":"v1","addr":"172.21.15.189:6821","nonce":2760890917}]},"public_addr":"172.21.15.189:6817/2760890917","cluster_addr":"172.21.15.189:6819/2760890917","heartbeat_back_addr":"172.21.15.189:6823/2760890917","heartbeat_front_addr":"172.21.15.189:6821/2760890917","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T21:22:45.605185+0000","dead_epoch":48},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T21:23:21.062873+0000","dead_epoch":53},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T21:24:00.502044+0000","dead_epoch":58},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T21:24:32.944093+0000","dead_epoch":63},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T21:25:06.278988+0000","dead_epoch":68},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T21:25:41.872982+0000","dead_epoch":73}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.189:6825/237262514":"2024-04-21T21:42:01.558630+0000","172.21.15.62:0/2348636520":"2024-04-21T21:30:35.075229+0000","172.21.15.62:6801/871423274":"2024-04-21T21:18:52.990685+0000","172.21.15.62:6801/831506538":"2024-04-21T21:21:03.778739+0000","172.21.15.62:0/4251078301":"2024-04-21T21:36:34.221905+0000","172.21.15.189:0/1714270321":"2024-04-21T21:31:36.545789+0000","172.21.15.62:6826/1147127123":"2024-04-21T21:42:01.558599+0000","172.21.15.62:0/868116748":"2024-04-21T21:30:35.075229+0000","172.21.15.62:0/2525292829":"2024-04-21T21:18:52.990685+0000","172.21.15.62:0/1116048562":"2024-04-21T21:21:03.778739+0000","172.21.15.62:0/634566145":"2024-04-21T21:18:52.990685+0000","172.21.15.62:6800/831506538":"2024-04-21T21:21:03.778739+0000","172.21.15.189:0/4026364416":"2024-04-21T21:31:36.545789+0000","172.21.15.62:0/1789120375":"2024-04-21T21:36:34.221905+0000","172.21.15.62:0/3416563026":"2024-04-21T21:18:52.990685+0000","172.21.15.62:0/3673769314":"2024-04-21T21:30:35.075229+0000","172.21.15.62:0/717708557":"2024-04-21T21:30:35.075229+0000","172.21.15.62:0/4274508569":"2024-04-21T21:19:34.965981+0000","172.21.15.189:0/3430221175":"2024-04-21T21:31:36.545789+0000","172.21.15.189:6824/237262514":"2024-04-21T21:42:01.558630+0000","172.21.15.189:0/100133912":"2024-04-21T21:31:36.545789+0000","172.21.15.189:0/441007232":"2024-04-21T21:31:36.545789+0000","172.21.15.62:6800/4116451241":"2024-04-21T21:30:35.075229+0000","172.21.15.62:6827/1147127123":"2024-04-21T21:42:01.558599+0000","172.21.15.62:6801/4116451241":"2024-04-21T21:30:35.075229+0000","172.21.15.62:6800/871423274":"2024-04-21T21:18:52.990685+0000","172.21.15.189:6828/2010473407":"2024-04-21T21:31:36.545789+0000","172.21.15.189:6829/2010473407":"2024-04-21T21:31:36.545789+0000","172.21.15.189:0/1726963148":"2024-04-21T21:31:36.545789+0000","172.21.15.62:0/3192460755":"2024-04-21T21:21:03.778739+0000","172.21.15.62:6801/4147738586":"2024-04-21T21:19:34.965981+0000","172.21.15.62:0/4158176989":"2024-04-21T21:19:34.965981+0000","172.21.15.62:0/301716985":"2024-04-21T21:21:03.778739+0000","172.21.15.62:0/678960466":"2024-04-21T21:36:34.221905+0000","172.21.15.62:0/2136121731":"2024-04-21T21:36:34.221905+0000","172.21.15.62:6800/4147738586":"2024-04-21T21:19:34.965981+0000","172.21.15.62:6800/1585760929":"2024-04-21T21:36:34.221905+0000","172.21.15.62:0/771955286":"2024-04-21T21:19:34.965981+0000","172.21.15.62:6801/1585760929":"2024-04-21T21:36:34.221905+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-20T22:31:27.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.982+0000 7f3b1a800640 1 -- 172.21.15.62:0/2021562814 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3af806ecf0 msgr2=0x7f3af80711a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:27.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.982+0000 7f3b1a800640 1 --2- 172.21.15.62:0/2021562814 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3af806ecf0 0x7f3af80711a0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f3b04005c40 tx=0x7f3b04005bd0 comp rx=0 tx=0).stop 2024-04-20T22:31:27.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.983+0000 7f3b1a800640 1 -- 172.21.15.62:0/2021562814 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3b14107880 msgr2=0x7f3b141978d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:27.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.983+0000 7f3b1a800640 1 --2- 172.21.15.62:0/2021562814 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3b14107880 0x7f3b141978d0 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7f3afc036040 tx=0x7f3afc004270 comp rx=0 tx=0).stop 2024-04-20T22:31:27.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.983+0000 7f3b1a800640 1 -- 172.21.15.62:0/2021562814 shutdown_connections 2024-04-20T22:31:27.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.983+0000 7f3b1a800640 1 --2- 172.21.15.62:0/2021562814 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f3af806ecf0 0x7f3af80711a0 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:27.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.983+0000 7f3b1a800640 1 --2- 172.21.15.62:0/2021562814 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3b14107880 0x7f3b141978d0 unknown :-1 s=CLOSED pgs=1469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:27.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.983+0000 7f3b1a800640 1 --2- 172.21.15.62:0/2021562814 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b14101870 0x7f3b14197390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:27.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.983+0000 7f3b1a800640 1 -- 172.21.15.62:0/2021562814 >> 172.21.15.62:0/2021562814 conn(0x7f3b140fd370 msgr2=0x7f3b140feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:31:27.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.983+0000 7f3b1a800640 1 -- 172.21.15.62:0/2021562814 shutdown_connections 2024-04-20T22:31:27.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:27.983+0000 7f3b1a800640 1 -- 172.21.15.62:0/2021562814 wait complete. 2024-04-20T22:31:27.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:31:27 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/3886227876' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-20T22:31:27.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:31:27 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/3567450713' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-20T22:31:28.002 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-20T21:24:02.500375+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-04-20T21:27:19.577559+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 1, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-04-20T21:27:20.560814+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-20T22:31:28.002 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-20T22:31:28.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:28.214+0000 7fa22c200640 1 -- 172.21.15.62:0/286154987 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa22410ac40 msgr2=0x7fa22410b010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:28.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:28.214+0000 7fa22c200640 1 --2- 172.21.15.62:0/286154987 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa22410ac40 0x7fa22410b010 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7fa218009c20 tx=0x7fa21802c1b0 comp rx=0 tx=0).stop 2024-04-20T22:31:28.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:28.214+0000 7fa22c200640 1 -- 172.21.15.62:0/286154987 shutdown_connections 2024-04-20T22:31:28.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:28.215+0000 7fa22c200640 1 --2- 172.21.15.62:0/286154987 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa224106090 0x7fa2241064e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:28.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:28.215+0000 7fa22c200640 1 --2- 172.21.15.62:0/286154987 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa22410ac40 0x7fa22410b010 unknown :-1 s=CLOSED pgs=1470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:28.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:28.215+0000 7fa22c200640 1 -- 172.21.15.62:0/286154987 >> 172.21.15.62:0/286154987 conn(0x7fa224075490 msgr2=0x7fa2240778a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:31:28.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:28.215+0000 7fa22c200640 1 -- 172.21.15.62:0/286154987 shutdown_connections 2024-04-20T22:31:28.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:28.215+0000 7fa22c200640 1 -- 172.21.15.62:0/286154987 wait complete. 2024-04-20T22:31:28.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:28.217+0000 7fa22c200640 1 Processor -- start 2024-04-20T22:31:28.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:28.217+0000 7fa22c200640 1 -- start start 2024-04-20T22:31:28.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:28.218+0000 7fa22c200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa224106090 0x7fa2241a1680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:28.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:28.218+0000 7fa22c200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa22410ac40 0x7fa2241a1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:28.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:28.218+0000 7fa22c200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa22419b6f0 con 0x7fa224106090 2024-04-20T22:31:28.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:28.218+0000 7fa22c200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa22419b860 con 0x7fa22410ac40 2024-04-20T22:31:28.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:28.218+0000 7fa229800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa224106090 0x7fa2241a1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:28.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:28.218+0000 7fa229800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa224106090 0x7fa2241a1680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53578/0 (socket says 172.21.15.62:53578) 2024-04-20T22:31:28.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:28.218+0000 7fa229800640 1 -- 172.21.15.62:0/2994118830 learned_addr learned my addr 172.21.15.62:0/2994118830 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:31:28.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:28.218+0000 7fa228e00640 1 --2- 172.21.15.62:0/2994118830 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa22410ac40 0x7fa2241a1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:28.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:28.218+0000 7fa229800640 1 -- 172.21.15.62:0/2994118830 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa22410ac40 msgr2=0x7fa2241a1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:28.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:28.219+0000 7fa229800640 1 --2- 172.21.15.62:0/2994118830 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa22410ac40 0x7fa2241a1bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:28.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:28.219+0000 7fa229800640 1 -- 172.21.15.62:0/2994118830 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa218009900 con 0x7fa224106090 2024-04-20T22:31:28.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:28.219+0000 7fa229800640 1 --2- 172.21.15.62:0/2994118830 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa224106090 0x7fa2241a1680 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7fa218037040 tx=0x7fa218032b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:31:28.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:28.219+0000 7fa222000640 1 -- 172.21.15.62:0/2994118830 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa218034820 con 0x7fa224106090 2024-04-20T22:31:28.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:28.219+0000 7fa222000640 1 -- 172.21.15.62:0/2994118830 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa218003e30 con 0x7fa224106090 2024-04-20T22:31:28.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:28.219+0000 7fa22c200640 1 -- 172.21.15.62:0/2994118830 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa22419bae0 con 0x7fa224106090 2024-04-20T22:31:28.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:28.219+0000 7fa222000640 1 -- 172.21.15.62:0/2994118830 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa21803f440 con 0x7fa224106090 2024-04-20T22:31:28.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:28.219+0000 7fa22c200640 1 -- 172.21.15.62:0/2994118830 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa22419c060 con 0x7fa224106090 2024-04-20T22:31:28.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:28.221+0000 7fa22c200640 1 -- 172.21.15.62:0/2994118830 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa22404e000 con 0x7fa224106090 2024-04-20T22:31:28.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:28.222+0000 7fa222000640 1 -- 172.21.15.62:0/2994118830 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fa21803b040 con 0x7fa224106090 2024-04-20T22:31:28.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:28.224+0000 7fa222000640 1 --2- 172.21.15.62:0/2994118830 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa20c06ec30 0x7fa20c0710e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:28.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:28.224+0000 7fa222000640 1 -- 172.21.15.62:0/2994118830 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fa2180b1c60 con 0x7fa224106090 2024-04-20T22:31:28.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:28.224+0000 7fa228e00640 1 --2- 172.21.15.62:0/2994118830 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa20c06ec30 0x7fa20c0710e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:28.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:28.225+0000 7fa228e00640 1 --2- 172.21.15.62:0/2994118830 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa20c06ec30 0x7fa20c0710e0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fa210005fd0 tx=0x7fa210005850 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:31:28.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:28.228+0000 7fa222000640 1 -- 172.21.15.62:0/2994118830 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fa21807aa90 con 0x7fa224106090 2024-04-20T22:31:28.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:28.440+0000 7fa22c200640 1 -- 172.21.15.62:0/2994118830 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa22419c370 con 0x7fa224106090 2024-04-20T22:31:28.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:28.441+0000 7fa222000640 1 -- 172.21.15.62:0/2994118830 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v85) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa218008220 con 0x7fa224106090 2024-04-20T22:31:28.441 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 1 2024-04-20T22:31:28.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:28.444+0000 7fa22c200640 1 -- 172.21.15.62:0/2994118830 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa20c06ec30 msgr2=0x7fa20c0710e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:28.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:28.444+0000 7fa22c200640 1 --2- 172.21.15.62:0/2994118830 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa20c06ec30 0x7fa20c0710e0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fa210005fd0 tx=0x7fa210005850 comp rx=0 tx=0).stop 2024-04-20T22:31:28.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:28.444+0000 7fa22c200640 1 -- 172.21.15.62:0/2994118830 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa224106090 msgr2=0x7fa2241a1680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:28.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:28.444+0000 7fa22c200640 1 --2- 172.21.15.62:0/2994118830 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa224106090 0x7fa2241a1680 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7fa218037040 tx=0x7fa218032b60 comp rx=0 tx=0).stop 2024-04-20T22:31:28.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:28.445+0000 7fa22c200640 1 -- 172.21.15.62:0/2994118830 shutdown_connections 2024-04-20T22:31:28.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:28.445+0000 7fa22c200640 1 --2- 172.21.15.62:0/2994118830 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fa20c06ec30 0x7fa20c0710e0 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:28.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:28.445+0000 7fa22c200640 1 --2- 172.21.15.62:0/2994118830 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa22410ac40 0x7fa2241a1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:28.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:28.445+0000 7fa22c200640 1 --2- 172.21.15.62:0/2994118830 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa224106090 0x7fa2241a1680 unknown :-1 s=CLOSED pgs=1471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:28.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:28.445+0000 7fa22c200640 1 -- 172.21.15.62:0/2994118830 >> 172.21.15.62:0/2994118830 conn(0x7fa224075490 msgr2=0x7fa224075e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:31:28.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:28.445+0000 7fa22c200640 1 -- 172.21.15.62:0/2994118830 shutdown_connections 2024-04-20T22:31:28.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:28.445+0000 7fa22c200640 1 -- 172.21.15.62:0/2994118830 wait complete. 2024-04-20T22:31:28.464 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-04-20T22:31:28.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:28.680+0000 7f74c5400640 1 -- 172.21.15.62:0/3187398000 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f74c0072f10 msgr2=0x7f74c0110120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:28.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:28.680+0000 7f74c5400640 1 --2- 172.21.15.62:0/3187398000 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f74c0072f10 0x7f74c0110120 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7f74a8009880 tx=0x7f74a802bd90 comp rx=0 tx=0).stop 2024-04-20T22:31:28.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:28.680+0000 7f74c5400640 1 -- 172.21.15.62:0/3187398000 shutdown_connections 2024-04-20T22:31:28.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:28.680+0000 7f74c5400640 1 --2- 172.21.15.62:0/3187398000 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f74c0072f10 0x7f74c0110120 unknown :-1 s=CLOSED pgs=1472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:28.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:28.680+0000 7f74c5400640 1 --2- 172.21.15.62:0/3187398000 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f74c0072600 0x7f74c00729d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:28.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:28.680+0000 7f74c5400640 1 -- 172.21.15.62:0/3187398000 >> 172.21.15.62:0/3187398000 conn(0x7f74c00fb170 msgr2=0x7f74c00fd580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:31:28.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:28.680+0000 7f74c5400640 1 -- 172.21.15.62:0/3187398000 shutdown_connections 2024-04-20T22:31:28.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:28.680+0000 7f74c5400640 1 -- 172.21.15.62:0/3187398000 wait complete. 2024-04-20T22:31:28.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:28.681+0000 7f74c5400640 1 Processor -- start 2024-04-20T22:31:28.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:28.681+0000 7f74c5400640 1 -- start start 2024-04-20T22:31:28.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:28.682+0000 7f74c5400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f74c0072600 0x7f74c01a1680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:28.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:28.682+0000 7f74c5400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f74c0072f10 0x7f74c01a1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:28.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:28.682+0000 7f74c5400640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74c01a2210 con 0x7f74c0072600 2024-04-20T22:31:28.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:28.682+0000 7f74c5400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74c019b6f0 con 0x7f74c0072f10 2024-04-20T22:31:28.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:28.682+0000 7f74be800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f74c0072600 0x7f74c01a1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:28.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:28.682+0000 7f74be800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f74c0072600 0x7f74c01a1680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53590/0 (socket says 172.21.15.62:53590) 2024-04-20T22:31:28.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:28.682+0000 7f74be800640 1 -- 172.21.15.62:0/3573669126 learned_addr learned my addr 172.21.15.62:0/3573669126 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:31:28.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:28.682+0000 7f74b5e00640 1 --2- 172.21.15.62:0/3573669126 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f74c0072f10 0x7f74c01a1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:28.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:28.682+0000 7f74be800640 1 -- 172.21.15.62:0/3573669126 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f74c0072f10 msgr2=0x7f74c01a1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:28.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:28.682+0000 7f74be800640 1 --2- 172.21.15.62:0/3573669126 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f74c0072f10 0x7f74c01a1bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:28.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:28.682+0000 7f74be800640 1 -- 172.21.15.62:0/3573669126 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74a8009560 con 0x7f74c0072600 2024-04-20T22:31:28.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:28.683+0000 7f74be800640 1 --2- 172.21.15.62:0/3573669126 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f74c0072600 0x7f74c01a1680 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7f74b000c8a0 tx=0x7f74b000cd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:31:28.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:28.683+0000 7f74b7e00640 1 -- 172.21.15.62:0/3573669126 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f74b0007b80 con 0x7f74c0072600 2024-04-20T22:31:28.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:28.683+0000 7f74b7e00640 1 -- 172.21.15.62:0/3573669126 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f74b0007ce0 con 0x7f74c0072600 2024-04-20T22:31:28.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:28.683+0000 7f74b7e00640 1 -- 172.21.15.62:0/3573669126 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f74b0018810 con 0x7f74c0072600 2024-04-20T22:31:28.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:28.683+0000 7f74c5400640 1 -- 172.21.15.62:0/3573669126 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74c019b9a0 con 0x7f74c0072600 2024-04-20T22:31:28.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:28.683+0000 7f74c5400640 1 -- 172.21.15.62:0/3573669126 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74c019bf00 con 0x7f74c0072600 2024-04-20T22:31:28.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:28.684+0000 7f74c5400640 1 -- 172.21.15.62:0/3573669126 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74c004e000 con 0x7f74c0072600 2024-04-20T22:31:28.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:28.686+0000 7f74b7e00640 1 -- 172.21.15.62:0/3573669126 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f74b0018970 con 0x7f74c0072600 2024-04-20T22:31:28.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:28.687+0000 7f74b7e00640 1 --2- 172.21.15.62:0/3573669126 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f74a406ec30 0x7f74a40710e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:28.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:28.687+0000 7f74b7e00640 1 -- 172.21.15.62:0/3573669126 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f74b008efe0 con 0x7f74c0072600 2024-04-20T22:31:28.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:28.687+0000 7f74b5e00640 1 --2- 172.21.15.62:0/3573669126 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f74a406ec30 0x7f74a40710e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:28.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:28.688+0000 7f74b5e00640 1 --2- 172.21.15.62:0/3573669126 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f74a406ec30 0x7f74a40710e0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f74a8036040 tx=0x7f74a8005fd0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:31:28.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:28.692+0000 7f74b7e00640 1 -- 172.21.15.62:0/3573669126 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f74b0057e10 con 0x7f74c0072600 2024-04-20T22:31:28.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:28.895+0000 7f74c5400640 1 -- 172.21.15.62:0/3573669126 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7f74c019cc00 con 0x7f74c0072600 2024-04-20T22:31:28.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:28.896+0000 7f74b7e00640 1 -- 172.21.15.62:0/3573669126 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v85) v1 ==== 107+0+11 (secure 0 0 0) 0x7f74b005b4d0 con 0x7f74c0072600 2024-04-20T22:31:28.896 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 32 2024-04-20T22:31:28.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:28.900+0000 7f74c5400640 1 -- 172.21.15.62:0/3573669126 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f74a406ec30 msgr2=0x7f74a40710e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:28.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:28.900+0000 7f74c5400640 1 --2- 172.21.15.62:0/3573669126 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f74a406ec30 0x7f74a40710e0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f74a8036040 tx=0x7f74a8005fd0 comp rx=0 tx=0).stop 2024-04-20T22:31:28.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:28.900+0000 7f74c5400640 1 -- 172.21.15.62:0/3573669126 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f74c0072600 msgr2=0x7f74c01a1680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:28.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:28.900+0000 7f74c5400640 1 --2- 172.21.15.62:0/3573669126 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f74c0072600 0x7f74c01a1680 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7f74b000c8a0 tx=0x7f74b000cd60 comp rx=0 tx=0).stop 2024-04-20T22:31:28.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:28.900+0000 7f74c5400640 1 -- 172.21.15.62:0/3573669126 shutdown_connections 2024-04-20T22:31:28.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:28.900+0000 7f74c5400640 1 --2- 172.21.15.62:0/3573669126 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f74a406ec30 0x7f74a40710e0 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:28.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:28.900+0000 7f74c5400640 1 --2- 172.21.15.62:0/3573669126 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f74c0072f10 0x7f74c01a1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:28.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:28.900+0000 7f74c5400640 1 --2- 172.21.15.62:0/3573669126 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f74c0072600 0x7f74c01a1680 unknown :-1 s=CLOSED pgs=1473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:28.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:28.900+0000 7f74c5400640 1 -- 172.21.15.62:0/3573669126 >> 172.21.15.62:0/3573669126 conn(0x7f74c00fb170 msgr2=0x7f74c0101c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:31:28.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:28.900+0000 7f74c5400640 1 -- 172.21.15.62:0/3573669126 shutdown_connections 2024-04-20T22:31:28.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:28.900+0000 7f74c5400640 1 -- 172.21.15.62:0/3573669126 wait complete. 2024-04-20T22:31:28.918 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:31:28 smithi062 ceph-mon[76322]: pgmap v1690: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:31:28.918 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:31:28 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/2021562814' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-20T22:31:28.918 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:31:28 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/2994118830' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-20T22:31:28.920 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-04-20T22:31:28.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:31:28 smithi189 ceph-mon[57022]: pgmap v1690: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:31:28.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:31:28 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/2021562814' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-20T22:31:28.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:31:28 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/2994118830' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-20T22:31:29.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:29.108+0000 7f882a000640 1 -- 172.21.15.62:0/1206106850 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8824067310 msgr2=0x7f8824067780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:29.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:29.108+0000 7f882a000640 1 --2- 172.21.15.62:0/1206106850 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8824067310 0x7f8824067780 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7f8810009880 tx=0x7f881002bce0 comp rx=0 tx=0).stop 2024-04-20T22:31:29.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:29.108+0000 7f882a000640 1 -- 172.21.15.62:0/1206106850 shutdown_connections 2024-04-20T22:31:29.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:29.108+0000 7f882a000640 1 --2- 172.21.15.62:0/1206106850 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8824067310 0x7f8824067780 unknown :-1 s=CLOSED pgs=1474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:29.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:29.108+0000 7f882a000640 1 --2- 172.21.15.62:0/1206106850 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8824100290 0x7f8824100660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:29.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:29.108+0000 7f882a000640 1 -- 172.21.15.62:0/1206106850 >> 172.21.15.62:0/1206106850 conn(0x7f88240743d0 msgr2=0x7f88240747d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:31:29.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:29.108+0000 7f882a000640 1 -- 172.21.15.62:0/1206106850 shutdown_connections 2024-04-20T22:31:29.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:29.109+0000 7f882a000640 1 -- 172.21.15.62:0/1206106850 wait complete. 2024-04-20T22:31:29.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:29.110+0000 7f882a000640 1 Processor -- start 2024-04-20T22:31:29.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:29.110+0000 7f882a000640 1 -- start start 2024-04-20T22:31:29.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:29.110+0000 7f882a000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8824067310 0x7f88241972f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:29.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:29.110+0000 7f882a000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8824100290 0x7f8824197830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:29.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:29.110+0000 7f882a000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8824197ec0 con 0x7f8824067310 2024-04-20T22:31:29.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:29.110+0000 7f882a000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f882419bc90 con 0x7f8824100290 2024-04-20T22:31:29.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:29.110+0000 7f8828c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8824067310 0x7f88241972f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:29.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:29.110+0000 7f8828c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8824067310 0x7f88241972f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53612/0 (socket says 172.21.15.62:53612) 2024-04-20T22:31:29.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:29.110+0000 7f8828c00640 1 -- 172.21.15.62:0/3188317367 learned_addr learned my addr 172.21.15.62:0/3188317367 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:31:29.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:29.111+0000 7f8823e00640 1 --2- 172.21.15.62:0/3188317367 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8824100290 0x7f8824197830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:29.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:29.111+0000 7f8828c00640 1 -- 172.21.15.62:0/3188317367 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8824100290 msgr2=0x7f8824197830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:29.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:29.111+0000 7f8828c00640 1 --2- 172.21.15.62:0/3188317367 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8824100290 0x7f8824197830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:29.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:29.111+0000 7f8828c00640 1 -- 172.21.15.62:0/3188317367 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8810009560 con 0x7f8824067310 2024-04-20T22:31:29.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:29.111+0000 7f8823e00640 1 --2- 172.21.15.62:0/3188317367 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8824100290 0x7f8824197830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-20T22:31:29.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:29.111+0000 7f8828c00640 1 --2- 172.21.15.62:0/3188317367 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8824067310 0x7f88241972f0 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7f881800d6f0 tx=0x7f881800dbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:31:29.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:29.111+0000 7f8821600640 1 -- 172.21.15.62:0/3188317367 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8818009a30 con 0x7f8824067310 2024-04-20T22:31:29.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:29.112+0000 7f8821600640 1 -- 172.21.15.62:0/3188317367 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f88180054e0 con 0x7f8824067310 2024-04-20T22:31:29.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:29.112+0000 7f882a000640 1 -- 172.21.15.62:0/3188317367 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f882419bf70 con 0x7f8824067310 2024-04-20T22:31:29.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:29.112+0000 7f8821600640 1 -- 172.21.15.62:0/3188317367 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8818010460 con 0x7f8824067310 2024-04-20T22:31:29.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:29.112+0000 7f882a000640 1 -- 172.21.15.62:0/3188317367 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f882419c550 con 0x7f8824067310 2024-04-20T22:31:29.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:29.113+0000 7f882a000640 1 -- 172.21.15.62:0/3188317367 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f882404e000 con 0x7f8824067310 2024-04-20T22:31:29.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:29.114+0000 7f8821600640 1 -- 172.21.15.62:0/3188317367 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f8818009b90 con 0x7f8824067310 2024-04-20T22:31:29.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:29.115+0000 7f8821600640 1 --2- 172.21.15.62:0/3188317367 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f880c06ec30 0x7f880c0710e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:29.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:29.115+0000 7f8821600640 1 -- 172.21.15.62:0/3188317367 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f881808ea10 con 0x7f8824067310 2024-04-20T22:31:29.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:29.116+0000 7f8823e00640 1 --2- 172.21.15.62:0/3188317367 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f880c06ec30 0x7f880c0710e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:29.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:29.116+0000 7f8823e00640 1 --2- 172.21.15.62:0/3188317367 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f880c06ec30 0x7f880c0710e0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f8810005ec0 tx=0x7f8810005fd0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:31:29.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:29.120+0000 7f8821600640 1 -- 172.21.15.62:0/3188317367 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7f88180577c0 con 0x7f8824067310 2024-04-20T22:31:29.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:29.324+0000 7f882a000640 1 -- 172.21.15.62:0/3188317367 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7f882410eb20 con 0x7f8824067310 2024-04-20T22:31:29.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:29.325+0000 7f8821600640 1 -- 172.21.15.62:0/3188317367 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v85) v1 ==== 107+0+11 (secure 0 0 0) 0x7f8818020020 con 0x7f8824067310 2024-04-20T22:31:29.325 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 32 2024-04-20T22:31:29.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:29.327+0000 7f882a000640 1 -- 172.21.15.62:0/3188317367 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f880c06ec30 msgr2=0x7f880c0710e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:29.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:29.327+0000 7f882a000640 1 --2- 172.21.15.62:0/3188317367 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f880c06ec30 0x7f880c0710e0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f8810005ec0 tx=0x7f8810005fd0 comp rx=0 tx=0).stop 2024-04-20T22:31:29.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:29.328+0000 7f882a000640 1 -- 172.21.15.62:0/3188317367 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8824067310 msgr2=0x7f88241972f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:29.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:29.328+0000 7f882a000640 1 --2- 172.21.15.62:0/3188317367 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8824067310 0x7f88241972f0 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7f881800d6f0 tx=0x7f881800dbb0 comp rx=0 tx=0).stop 2024-04-20T22:31:29.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:29.328+0000 7f882a000640 1 -- 172.21.15.62:0/3188317367 shutdown_connections 2024-04-20T22:31:29.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:29.328+0000 7f882a000640 1 --2- 172.21.15.62:0/3188317367 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7f880c06ec30 0x7f880c0710e0 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:29.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:29.328+0000 7f882a000640 1 --2- 172.21.15.62:0/3188317367 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8824100290 0x7f8824197830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:29.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:29.328+0000 7f882a000640 1 --2- 172.21.15.62:0/3188317367 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8824067310 0x7f88241972f0 unknown :-1 s=CLOSED pgs=1475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:29.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:29.328+0000 7f882a000640 1 -- 172.21.15.62:0/3188317367 >> 172.21.15.62:0/3188317367 conn(0x7f88240743d0 msgr2=0x7f88240fcc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:31:29.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:29.328+0000 7f882a000640 1 -- 172.21.15.62:0/3188317367 shutdown_connections 2024-04-20T22:31:29.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:29.328+0000 7f882a000640 1 -- 172.21.15.62:0/3188317367 wait complete. 2024-04-20T22:31:29.346 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-20T22:31:29.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:29.545+0000 7fd2d0a00640 1 -- 172.21.15.62:0/1409712066 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2cc067f20 msgr2=0x7fd2cc1040a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:29.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:29.545+0000 7fd2d0a00640 1 --2- 172.21.15.62:0/1409712066 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2cc067f20 0x7fd2cc1040a0 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7fd2b4009880 tx=0x7fd2b402bce0 comp rx=0 tx=0).stop 2024-04-20T22:31:29.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:29.546+0000 7fd2d0a00640 1 -- 172.21.15.62:0/1409712066 shutdown_connections 2024-04-20T22:31:29.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:29.546+0000 7fd2d0a00640 1 --2- 172.21.15.62:0/1409712066 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2cc067f20 0x7fd2cc1040a0 unknown :-1 s=CLOSED pgs=1476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:29.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:29.546+0000 7fd2d0a00640 1 --2- 172.21.15.62:0/1409712066 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2cc067580 0x7fd2cc067950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:29.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:29.546+0000 7fd2d0a00640 1 -- 172.21.15.62:0/1409712066 >> 172.21.15.62:0/1409712066 conn(0x7fd2cc0743e0 msgr2=0x7fd2cc0747e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:31:29.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:29.546+0000 7fd2d0a00640 1 -- 172.21.15.62:0/1409712066 shutdown_connections 2024-04-20T22:31:29.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:29.546+0000 7fd2d0a00640 1 -- 172.21.15.62:0/1409712066 wait complete. 2024-04-20T22:31:29.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:29.547+0000 7fd2d0a00640 1 Processor -- start 2024-04-20T22:31:29.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:29.547+0000 7fd2d0a00640 1 -- start start 2024-04-20T22:31:29.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:29.547+0000 7fd2d0a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2cc067580 0x7fd2cc195110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:29.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:29.547+0000 7fd2d0a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2cc067f20 0x7fd2cc195650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:29.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:29.547+0000 7fd2d0a00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2cc195ce0 con 0x7fd2cc067580 2024-04-20T22:31:29.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:29.547+0000 7fd2d0a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2cc199ab0 con 0x7fd2cc067f20 2024-04-20T22:31:29.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:29.548+0000 7fd2ca800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2cc067580 0x7fd2cc195110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:29.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:29.548+0000 7fd2ca800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2cc067580 0x7fd2cc195110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53636/0 (socket says 172.21.15.62:53636) 2024-04-20T22:31:29.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:29.548+0000 7fd2ca800640 1 -- 172.21.15.62:0/2836708447 learned_addr learned my addr 172.21.15.62:0/2836708447 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-20T22:31:29.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:29.548+0000 7fd2c3e00640 1 --2- 172.21.15.62:0/2836708447 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2cc067f20 0x7fd2cc195650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:29.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:29.548+0000 7fd2ca800640 1 -- 172.21.15.62:0/2836708447 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2cc067f20 msgr2=0x7fd2cc195650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:29.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:29.548+0000 7fd2ca800640 1 --2- 172.21.15.62:0/2836708447 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2cc067f20 0x7fd2cc195650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:29.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:29.548+0000 7fd2ca800640 1 -- 172.21.15.62:0/2836708447 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2b4009560 con 0x7fd2cc067580 2024-04-20T22:31:29.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:29.549+0000 7fd2ca800640 1 --2- 172.21.15.62:0/2836708447 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2cc067580 0x7fd2cc195110 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7fd2bc00d6f0 tx=0x7fd2bc00dbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:31:29.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:29.549+0000 7fd2c3400640 1 -- 172.21.15.62:0/2836708447 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd2bc009a30 con 0x7fd2cc067580 2024-04-20T22:31:29.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:29.549+0000 7fd2d0a00640 1 -- 172.21.15.62:0/2836708447 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2cc199d90 con 0x7fd2cc067580 2024-04-20T22:31:29.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:29.549+0000 7fd2c3400640 1 -- 172.21.15.62:0/2836708447 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd2bc0054e0 con 0x7fd2cc067580 2024-04-20T22:31:29.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:29.549+0000 7fd2d0a00640 1 -- 172.21.15.62:0/2836708447 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2cc19a2c0 con 0x7fd2cc067580 2024-04-20T22:31:29.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:29.549+0000 7fd2c3400640 1 -- 172.21.15.62:0/2836708447 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd2bc010460 con 0x7fd2cc067580 2024-04-20T22:31:29.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:29.551+0000 7fd2d0a00640 1 -- 172.21.15.62:0/2836708447 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2a8005280 con 0x7fd2cc067580 2024-04-20T22:31:29.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:29.552+0000 7fd2c3400640 1 -- 172.21.15.62:0/2836708447 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 38) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fd2bc0046c0 con 0x7fd2cc067580 2024-04-20T22:31:29.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:29.553+0000 7fd2c3400640 1 --2- 172.21.15.62:0/2836708447 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd2b006ec80 0x7fd2b0071130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-20T22:31:29.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:29.553+0000 7fd2c3400640 1 -- 172.21.15.62:0/2836708447 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fd2bc08e870 con 0x7fd2cc067580 2024-04-20T22:31:29.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:29.553+0000 7fd2c3e00640 1 --2- 172.21.15.62:0/2836708447 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd2b006ec80 0x7fd2b0071130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-20T22:31:29.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:29.554+0000 7fd2c3e00640 1 --2- 172.21.15.62:0/2836708447 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd2b006ec80 0x7fd2b0071130 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fd2b4006010 tx=0x7fd2b4005a20 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-20T22:31:29.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:29.558+0000 7fd2c3400640 1 -- 172.21.15.62:0/2836708447 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188381 (secure 0 0 0) 0x7fd2bc0576a0 con 0x7fd2cc067580 2024-04-20T22:31:29.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:29.775+0000 7fd2d0a00640 1 -- 172.21.15.62:0/2836708447 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fd2a8005ab0 con 0x7fd2cc067580 2024-04-20T22:31:29.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:29.776+0000 7fd2c3400640 1 -- 172.21.15.62:0/2836708447 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 39 entries v85) v1 ==== 81+0+2306 (secure 0 0 0) 0x7fd2bc020920 con 0x7fd2cc067580 2024-04-20T22:31:29.776 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.189:6825/237262514 2024-04-21T21:42:01.558630+0000 2024-04-20T22:31:29.776 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:0/2348636520 2024-04-21T21:30:35.075229+0000 2024-04-20T22:31:29.776 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:6801/871423274 2024-04-21T21:18:52.990685+0000 2024-04-20T22:31:29.776 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:6801/831506538 2024-04-21T21:21:03.778739+0000 2024-04-20T22:31:29.776 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:0/4251078301 2024-04-21T21:36:34.221905+0000 2024-04-20T22:31:29.776 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.189:0/1714270321 2024-04-21T21:31:36.545789+0000 2024-04-20T22:31:29.776 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:6826/1147127123 2024-04-21T21:42:01.558599+0000 2024-04-20T22:31:29.777 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:0/868116748 2024-04-21T21:30:35.075229+0000 2024-04-20T22:31:29.777 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:0/2525292829 2024-04-21T21:18:52.990685+0000 2024-04-20T22:31:29.777 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:0/1116048562 2024-04-21T21:21:03.778739+0000 2024-04-20T22:31:29.777 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:0/634566145 2024-04-21T21:18:52.990685+0000 2024-04-20T22:31:29.777 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:6800/831506538 2024-04-21T21:21:03.778739+0000 2024-04-20T22:31:29.777 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.189:0/4026364416 2024-04-21T21:31:36.545789+0000 2024-04-20T22:31:29.777 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:0/1789120375 2024-04-21T21:36:34.221905+0000 2024-04-20T22:31:29.777 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:0/3416563026 2024-04-21T21:18:52.990685+0000 2024-04-20T22:31:29.777 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:0/3673769314 2024-04-21T21:30:35.075229+0000 2024-04-20T22:31:29.777 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:0/717708557 2024-04-21T21:30:35.075229+0000 2024-04-20T22:31:29.777 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:0/4274508569 2024-04-21T21:19:34.965981+0000 2024-04-20T22:31:29.777 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.189:0/3430221175 2024-04-21T21:31:36.545789+0000 2024-04-20T22:31:29.777 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.189:6824/237262514 2024-04-21T21:42:01.558630+0000 2024-04-20T22:31:29.777 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.189:0/100133912 2024-04-21T21:31:36.545789+0000 2024-04-20T22:31:29.777 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.189:0/441007232 2024-04-21T21:31:36.545789+0000 2024-04-20T22:31:29.778 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:6800/4116451241 2024-04-21T21:30:35.075229+0000 2024-04-20T22:31:29.778 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:6827/1147127123 2024-04-21T21:42:01.558599+0000 2024-04-20T22:31:29.778 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:6801/4116451241 2024-04-21T21:30:35.075229+0000 2024-04-20T22:31:29.778 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:6800/871423274 2024-04-21T21:18:52.990685+0000 2024-04-20T22:31:29.779 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.189:6828/2010473407 2024-04-21T21:31:36.545789+0000 2024-04-20T22:31:29.779 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.189:6829/2010473407 2024-04-21T21:31:36.545789+0000 2024-04-20T22:31:29.779 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.189:0/1726963148 2024-04-21T21:31:36.545789+0000 2024-04-20T22:31:29.779 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:0/3192460755 2024-04-21T21:21:03.778739+0000 2024-04-20T22:31:29.779 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:6801/4147738586 2024-04-21T21:19:34.965981+0000 2024-04-20T22:31:29.779 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:0/4158176989 2024-04-21T21:19:34.965981+0000 2024-04-20T22:31:29.779 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:0/301716985 2024-04-21T21:21:03.778739+0000 2024-04-20T22:31:29.779 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:0/678960466 2024-04-21T21:36:34.221905+0000 2024-04-20T22:31:29.779 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:0/2136121731 2024-04-21T21:36:34.221905+0000 2024-04-20T22:31:29.779 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:6800/4147738586 2024-04-21T21:19:34.965981+0000 2024-04-20T22:31:29.779 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:6800/1585760929 2024-04-21T21:36:34.221905+0000 2024-04-20T22:31:29.780 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:0/771955286 2024-04-21T21:19:34.965981+0000 2024-04-20T22:31:29.780 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:6801/1585760929 2024-04-21T21:36:34.221905+0000 2024-04-20T22:31:29.781 INFO:teuthology.orchestra.run.smithi062.stderr:listed 39 entries 2024-04-20T22:31:29.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:29.778+0000 7fd2d0a00640 1 -- 172.21.15.62:0/2836708447 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd2b006ec80 msgr2=0x7fd2b0071130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:29.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:29.778+0000 7fd2d0a00640 1 --2- 172.21.15.62:0/2836708447 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd2b006ec80 0x7fd2b0071130 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fd2b4006010 tx=0x7fd2b4005a20 comp rx=0 tx=0).stop 2024-04-20T22:31:29.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:29.778+0000 7fd2d0a00640 1 -- 172.21.15.62:0/2836708447 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2cc067580 msgr2=0x7fd2cc195110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-20T22:31:29.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:29.778+0000 7fd2d0a00640 1 --2- 172.21.15.62:0/2836708447 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2cc067580 0x7fd2cc195110 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7fd2bc00d6f0 tx=0x7fd2bc00dbb0 comp rx=0 tx=0).stop 2024-04-20T22:31:29.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:29.778+0000 7fd2d0a00640 1 -- 172.21.15.62:0/2836708447 shutdown_connections 2024-04-20T22:31:29.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:29.778+0000 7fd2d0a00640 1 --2- 172.21.15.62:0/2836708447 >> [v2:172.21.15.62:6800/3925433479,v1:172.21.15.62:6801/3925433479] conn(0x7fd2b006ec80 0x7fd2b0071130 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:29.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:29.778+0000 7fd2d0a00640 1 --2- 172.21.15.62:0/2836708447 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2cc067f20 0x7fd2cc195650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:29.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:29.778+0000 7fd2d0a00640 1 --2- 172.21.15.62:0/2836708447 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2cc067580 0x7fd2cc195110 unknown :-1 s=CLOSED pgs=1477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-20T22:31:29.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:29.778+0000 7fd2d0a00640 1 -- 172.21.15.62:0/2836708447 >> 172.21.15.62:0/2836708447 conn(0x7fd2cc0743e0 msgr2=0x7fd2cc0fd8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-20T22:31:29.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:29.779+0000 7fd2d0a00640 1 -- 172.21.15.62:0/2836708447 shutdown_connections 2024-04-20T22:31:29.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-20T22:31:29.779+0000 7fd2d0a00640 1 -- 172.21.15.62:0/2836708447 wait complete. 2024-04-20T22:31:29.795 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi062.front.sepia.ceph.com... 2024-04-20T22:31:29.795 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-20T22:31:29.795 DEBUG:teuthology.orchestra.run.smithi062:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-04-20T22:31:29.871 INFO:teuthology.orchestra.run:waiting for 300 2024-04-20T22:31:29.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:31:29 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/3573669126' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-20T22:31:29.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:31:29 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/3188317367' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-20T22:31:30.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:31:29 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/3573669126' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-20T22:31:30.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:31:29 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/3188317367' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-20T22:31:30.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:31:30 smithi189 ceph-mon[57022]: from='client.? 172.21.15.62:0/2836708447' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-20T22:31:30.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:31:30 smithi189 ceph-mon[57022]: pgmap v1691: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:31:31.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:31:30 smithi062 ceph-mon[76322]: from='client.? 172.21.15.62:0/2836708447' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-20T22:31:31.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:31:30 smithi062 ceph-mon[76322]: pgmap v1691: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:31:33.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:31:33 smithi062 ceph-mon[76322]: pgmap v1692: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:31:33.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:31:33 smithi189 ceph-mon[57022]: pgmap v1692: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:31:35.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:31:35 smithi062 ceph-mon[76322]: pgmap v1693: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:31:35.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:31:35 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:31:35.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:31:35 smithi189 ceph-mon[57022]: pgmap v1693: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:31:35.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:31:35 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:31:37.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:31:37 smithi062 ceph-mon[76322]: pgmap v1694: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:31:37.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:31:37 smithi189 ceph-mon[57022]: pgmap v1694: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:31:39.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:31:39 smithi062 ceph-mon[76322]: pgmap v1695: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:31:39.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:31:39 smithi189 ceph-mon[57022]: pgmap v1695: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:31:41.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:31:41 smithi062 ceph-mon[76322]: pgmap v1696: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:31:41.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:31:41 smithi189 ceph-mon[57022]: pgmap v1696: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:31:43.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:31:43 smithi062 ceph-mon[76322]: pgmap v1697: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:31:43.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:31:43 smithi189 ceph-mon[57022]: pgmap v1697: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:31:45.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:31:45 smithi189 ceph-mon[57022]: pgmap v1698: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:31:45.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:31:45 smithi062 ceph-mon[76322]: pgmap v1698: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:31:47.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:31:47 smithi189 ceph-mon[57022]: pgmap v1699: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:31:47.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:31:47 smithi062 ceph-mon[76322]: pgmap v1699: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:31:49.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:31:49 smithi189 ceph-mon[57022]: pgmap v1700: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:31:49.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:31:49 smithi062 ceph-mon[76322]: pgmap v1700: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:31:50.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:31:50 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:31:50.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:31:50 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:31:51.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:31:51 smithi189 ceph-mon[57022]: pgmap v1701: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:31:51.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:31:51 smithi062 ceph-mon[76322]: pgmap v1701: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:31:53.341 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:31:53 smithi062 ceph-mon[76322]: pgmap v1702: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:31:53.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:31:53 smithi189 ceph-mon[57022]: pgmap v1702: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:31:55.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:31:55 smithi189 ceph-mon[57022]: pgmap v1703: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:31:55.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:31:55 smithi062 ceph-mon[76322]: pgmap v1703: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:31:57.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:31:57 smithi189 ceph-mon[57022]: pgmap v1704: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:31:57.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:31:57 smithi062 ceph-mon[76322]: pgmap v1704: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:31:59.469 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:31:59 smithi062 ceph-mon[76322]: pgmap v1705: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:31:59.487 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:31:59 smithi189 ceph-mon[57022]: pgmap v1705: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:32:01.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:32:01 smithi189 ceph-mon[57022]: pgmap v1706: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:32:01.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:32:01 smithi062 ceph-mon[76322]: pgmap v1706: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:32:03.408 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:32:03 smithi189 ceph-mon[57022]: pgmap v1707: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:32:03.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:32:03 smithi062 ceph-mon[76322]: pgmap v1707: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:32:04.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:32:04 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T22:32:04.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:32:04 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T22:32:05.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:32:05 smithi189 ceph-mon[57022]: pgmap v1708: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-20T22:32:05.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:32:05 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:32:05.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:32:05 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T22:32:05.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:32:05 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T22:32:05.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:32:05 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:32:05.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:32:05 smithi062 ceph-mon[76322]: pgmap v1708: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-20T22:32:05.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:32:05 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:32:05.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:32:05 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T22:32:05.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:32:05 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T22:32:05.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:32:05 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:32:07.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:32:07 smithi189 ceph-mon[57022]: pgmap v1709: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-20T22:32:07.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:32:07 smithi062 ceph-mon[76322]: pgmap v1709: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-20T22:32:09.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:32:09 smithi189 ceph-mon[57022]: pgmap v1710: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-20T22:32:09.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:32:09 smithi062 ceph-mon[76322]: pgmap v1710: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-20T22:32:11.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:32:11 smithi189 ceph-mon[57022]: pgmap v1711: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-20T22:32:11.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:32:11 smithi062 ceph-mon[76322]: pgmap v1711: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-20T22:32:13.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:32:13 smithi189 ceph-mon[57022]: pgmap v1712: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-20T22:32:13.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:32:13 smithi062 ceph-mon[76322]: pgmap v1712: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-20T22:32:15.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:32:15 smithi189 ceph-mon[57022]: pgmap v1713: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-20T22:32:15.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:32:15 smithi062 ceph-mon[76322]: pgmap v1713: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-20T22:32:17.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:32:17 smithi189 ceph-mon[57022]: pgmap v1714: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:32:17.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:32:17 smithi062 ceph-mon[76322]: pgmap v1714: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:32:19.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:32:19 smithi189 ceph-mon[57022]: pgmap v1715: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:32:19.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:32:19 smithi062 ceph-mon[76322]: pgmap v1715: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:32:20.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:32:20 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:32:20.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:32:20 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:32:21.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:32:21 smithi189 ceph-mon[57022]: pgmap v1716: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:32:21.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:32:21 smithi062 ceph-mon[76322]: pgmap v1716: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:32:23.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:32:23 smithi189 ceph-mon[57022]: pgmap v1717: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:32:23.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:32:23 smithi062 ceph-mon[76322]: pgmap v1717: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:32:25.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:32:25 smithi189 ceph-mon[57022]: pgmap v1718: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:32:25.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:32:25 smithi062 ceph-mon[76322]: pgmap v1718: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:32:27.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:32:27 smithi189 ceph-mon[57022]: pgmap v1719: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:32:27.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:32:27 smithi062 ceph-mon[76322]: pgmap v1719: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:32:29.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:32:29 smithi189 ceph-mon[57022]: pgmap v1720: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:32:29.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:32:29 smithi062 ceph-mon[76322]: pgmap v1720: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:32:31.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:32:31 smithi189 ceph-mon[57022]: pgmap v1721: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:32:31.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:32:31 smithi062 ceph-mon[76322]: pgmap v1721: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:32:33.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:32:33 smithi189 ceph-mon[57022]: pgmap v1722: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:32:33.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:32:33 smithi062 ceph-mon[76322]: pgmap v1722: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:32:35.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:32:35 smithi189 ceph-mon[57022]: pgmap v1723: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:32:35.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:32:35 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:32:35.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:32:35 smithi062 ceph-mon[76322]: pgmap v1723: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:32:35.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:32:35 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:32:37.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:32:37 smithi189 ceph-mon[57022]: pgmap v1724: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:32:37.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:32:37 smithi062 ceph-mon[76322]: pgmap v1724: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:32:39.524 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:32:39 smithi189 ceph-mon[57022]: pgmap v1725: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:32:39.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:32:39 smithi062 ceph-mon[76322]: pgmap v1725: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:32:41.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:32:41 smithi062 ceph-mon[76322]: pgmap v1726: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:32:41.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:32:41 smithi189 ceph-mon[57022]: pgmap v1726: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:32:43.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:32:43 smithi062 ceph-mon[76322]: pgmap v1727: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:32:43.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:32:43 smithi189 ceph-mon[57022]: pgmap v1727: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:32:45.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:32:45 smithi062 ceph-mon[76322]: pgmap v1728: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:32:45.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:32:45 smithi189 ceph-mon[57022]: pgmap v1728: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:32:47.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:32:47 smithi062 ceph-mon[76322]: pgmap v1729: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:32:47.540 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:32:47 smithi189 ceph-mon[57022]: pgmap v1729: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:32:49.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:32:49 smithi062 ceph-mon[76322]: pgmap v1730: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:32:49.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:32:49 smithi189 ceph-mon[57022]: pgmap v1730: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:32:50.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:32:50 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:32:50.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:32:50 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:32:51.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:32:51 smithi189 ceph-mon[57022]: pgmap v1731: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:32:51.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:32:51 smithi062 ceph-mon[76322]: pgmap v1731: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:32:53.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:32:53 smithi189 ceph-mon[57022]: pgmap v1732: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:32:53.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:32:53 smithi062 ceph-mon[76322]: pgmap v1732: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:32:55.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:32:55 smithi189 ceph-mon[57022]: pgmap v1733: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:32:55.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:32:55 smithi062 ceph-mon[76322]: pgmap v1733: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:32:57.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:32:57 smithi189 ceph-mon[57022]: pgmap v1734: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:32:57.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:32:57 smithi062 ceph-mon[76322]: pgmap v1734: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:32:59.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:32:59 smithi189 ceph-mon[57022]: pgmap v1735: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:32:59.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:32:59 smithi062 ceph-mon[76322]: pgmap v1735: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:33:01.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:33:01 smithi189 ceph-mon[57022]: pgmap v1736: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:33:01.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:33:01 smithi062 ceph-mon[76322]: pgmap v1736: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:33:03.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:33:03 smithi189 ceph-mon[57022]: pgmap v1737: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:33:03.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:33:03 smithi062 ceph-mon[76322]: pgmap v1737: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:33:05.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:33:05 smithi189 ceph-mon[57022]: pgmap v1738: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:33:05.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:33:05 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:33:05.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:33:05 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T22:33:05.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:33:05 smithi062 ceph-mon[76322]: pgmap v1738: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:33:05.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:33:05 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:33:05.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:33:05 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T22:33:07.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:33:07 smithi189 ceph-mon[57022]: pgmap v1739: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:33:07.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:33:07 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T22:33:07.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:33:07 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T22:33:07.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:33:07 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:33:07.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:33:07 smithi062 ceph-mon[76322]: pgmap v1739: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:33:07.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:33:07 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T22:33:07.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:33:07 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T22:33:07.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:33:07 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:33:09.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:33:09 smithi189 ceph-mon[57022]: pgmap v1740: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:33:09.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:33:09 smithi062 ceph-mon[76322]: pgmap v1740: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:33:11.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:33:11 smithi189 ceph-mon[57022]: pgmap v1741: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:33:11.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:33:11 smithi062 ceph-mon[76322]: pgmap v1741: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:33:13.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:33:13 smithi189 ceph-mon[57022]: pgmap v1742: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:33:13.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:33:13 smithi062 ceph-mon[76322]: pgmap v1742: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:33:15.716 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:33:15 smithi189 ceph-mon[57022]: pgmap v1743: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:33:15.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:33:15 smithi062 ceph-mon[76322]: pgmap v1743: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:33:17.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:33:17 smithi189 ceph-mon[57022]: pgmap v1744: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:33:17.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:33:17 smithi062 ceph-mon[76322]: pgmap v1744: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:33:19.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:33:19 smithi189 ceph-mon[57022]: pgmap v1745: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:33:19.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:33:19 smithi062 ceph-mon[76322]: pgmap v1745: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:33:20.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:33:20 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:33:20.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:33:20 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:33:21.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:33:21 smithi189 ceph-mon[57022]: pgmap v1746: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:33:21.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:33:21 smithi062 ceph-mon[76322]: pgmap v1746: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:33:23.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:33:23 smithi189 ceph-mon[57022]: pgmap v1747: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:33:23.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:33:23 smithi062 ceph-mon[76322]: pgmap v1747: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:33:25.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:33:25 smithi189 ceph-mon[57022]: pgmap v1748: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:33:25.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:33:25 smithi062 ceph-mon[76322]: pgmap v1748: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:33:27.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:33:27 smithi189 ceph-mon[57022]: pgmap v1749: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:33:27.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:33:27 smithi062 ceph-mon[76322]: pgmap v1749: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:33:29.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:33:29 smithi189 ceph-mon[57022]: pgmap v1750: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:33:29.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:33:29 smithi062 ceph-mon[76322]: pgmap v1750: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:33:31.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:33:31 smithi189 ceph-mon[57022]: pgmap v1751: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:33:31.685 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:33:31 smithi062 ceph-mon[76322]: pgmap v1751: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:33:33.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:33:33 smithi189 ceph-mon[57022]: pgmap v1752: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:33:33.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:33:33 smithi062 ceph-mon[76322]: pgmap v1752: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:33:35.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:33:35 smithi189 ceph-mon[57022]: pgmap v1753: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:33:35.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:33:35 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:33:35.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:33:35 smithi062 ceph-mon[76322]: pgmap v1753: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:33:35.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:33:35 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:33:37.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:33:37 smithi189 ceph-mon[57022]: pgmap v1754: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:33:37.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:33:37 smithi062 ceph-mon[76322]: pgmap v1754: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:33:39.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:33:39 smithi189 ceph-mon[57022]: pgmap v1755: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:33:39.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:33:39 smithi062 ceph-mon[76322]: pgmap v1755: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:33:41.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:33:41 smithi189 ceph-mon[57022]: pgmap v1756: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:33:41.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:33:41 smithi062 ceph-mon[76322]: pgmap v1756: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:33:43.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:33:43 smithi189 ceph-mon[57022]: pgmap v1757: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:33:43.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:33:43 smithi062 ceph-mon[76322]: pgmap v1757: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:33:45.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:33:45 smithi189 ceph-mon[57022]: pgmap v1758: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:33:45.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:33:45 smithi062 ceph-mon[76322]: pgmap v1758: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:33:47.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:33:47 smithi189 ceph-mon[57022]: pgmap v1759: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:33:47.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:33:47 smithi062 ceph-mon[76322]: pgmap v1759: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:33:49.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:33:49 smithi062 ceph-mon[76322]: pgmap v1760: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:33:49.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:33:49 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:33:49.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:33:49 smithi189 ceph-mon[57022]: pgmap v1760: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:33:49.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:33:49 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:33:51.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:33:51 smithi062 ceph-mon[76322]: pgmap v1761: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:33:51.902 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:33:51 smithi189 ceph-mon[57022]: pgmap v1761: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:33:53.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:33:53 smithi062 ceph-mon[76322]: pgmap v1762: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:33:53.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:33:53 smithi189 ceph-mon[57022]: pgmap v1762: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:33:55.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:33:55 smithi062 ceph-mon[76322]: pgmap v1763: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:33:55.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:33:55 smithi189 ceph-mon[57022]: pgmap v1763: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:33:57.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:33:57 smithi062 ceph-mon[76322]: pgmap v1764: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:33:57.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:33:57 smithi189 ceph-mon[57022]: pgmap v1764: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:33:59.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:33:59 smithi062 ceph-mon[76322]: pgmap v1765: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:33:59.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:33:59 smithi189 ceph-mon[57022]: pgmap v1765: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:34:01.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:34:01 smithi062 ceph-mon[76322]: pgmap v1766: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:34:01.987 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:34:01 smithi189 ceph-mon[57022]: pgmap v1766: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:34:03.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:34:03 smithi189 ceph-mon[57022]: pgmap v1767: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:34:04.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:34:03 smithi062 ceph-mon[76322]: pgmap v1767: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:34:04.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:34:04 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:34:05.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:34:04 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:34:05.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:34:05 smithi189 ceph-mon[57022]: pgmap v1768: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:34:06.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:34:05 smithi062 ceph-mon[76322]: pgmap v1768: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:34:07.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:34:07 smithi189 ceph-mon[57022]: pgmap v1769: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:34:07.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:34:07 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T22:34:07.915 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:34:07 smithi062 ceph-mon[76322]: pgmap v1769: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:34:07.915 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:34:07 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T22:34:08.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:34:08 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T22:34:08.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:34:08 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T22:34:08.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:34:08 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:34:09.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:34:08 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T22:34:09.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:34:08 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T22:34:09.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:34:08 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:34:09.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:34:09 smithi189 ceph-mon[57022]: pgmap v1770: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:34:10.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:34:09 smithi062 ceph-mon[76322]: pgmap v1770: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:34:11.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:34:11 smithi189 ceph-mon[57022]: pgmap v1771: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:34:12.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:34:11 smithi062 ceph-mon[76322]: pgmap v1771: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:34:13.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:34:13 smithi189 ceph-mon[57022]: pgmap v1772: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:34:14.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:34:13 smithi062 ceph-mon[76322]: pgmap v1772: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:34:15.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:34:15 smithi189 ceph-mon[57022]: pgmap v1773: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:34:16.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:34:15 smithi062 ceph-mon[76322]: pgmap v1773: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:34:17.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:34:17 smithi189 ceph-mon[57022]: pgmap v1774: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:34:18.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:34:17 smithi062 ceph-mon[76322]: pgmap v1774: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:34:19.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:34:19 smithi189 ceph-mon[57022]: pgmap v1775: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:34:19.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:34:19 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:34:20.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:34:19 smithi062 ceph-mon[76322]: pgmap v1775: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:34:20.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:34:19 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:34:21.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:34:21 smithi189 ceph-mon[57022]: pgmap v1776: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:34:22.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:34:21 smithi062 ceph-mon[76322]: pgmap v1776: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:34:23.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:34:23 smithi062 ceph-mon[76322]: pgmap v1777: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:34:23.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:34:23 smithi189 ceph-mon[57022]: pgmap v1777: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:34:25.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:34:25 smithi189 ceph-mon[57022]: pgmap v1778: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:34:26.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:34:25 smithi062 ceph-mon[76322]: pgmap v1778: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:34:26.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:34:26 smithi189 ceph-mon[57022]: pgmap v1779: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:34:27.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:34:26 smithi062 ceph-mon[76322]: pgmap v1779: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:34:29.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:34:29 smithi189 ceph-mon[57022]: pgmap v1780: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:34:29.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:34:29 smithi062 ceph-mon[76322]: pgmap v1780: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:34:31.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:34:31 smithi189 ceph-mon[57022]: pgmap v1781: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:34:31.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:34:31 smithi062 ceph-mon[76322]: pgmap v1781: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:34:33.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:34:33 smithi189 ceph-mon[57022]: pgmap v1782: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:34:33.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:34:33 smithi062 ceph-mon[76322]: pgmap v1782: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:34:35.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:34:35 smithi189 ceph-mon[57022]: pgmap v1783: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:34:35.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:34:35 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:34:35.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:34:35 smithi062 ceph-mon[76322]: pgmap v1783: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:34:35.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:34:35 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:34:37.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:34:37 smithi189 ceph-mon[57022]: pgmap v1784: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:34:37.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:34:37 smithi062 ceph-mon[76322]: pgmap v1784: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:34:39.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:34:39 smithi062 ceph-mon[76322]: pgmap v1785: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:34:39.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:34:39 smithi189 ceph-mon[57022]: pgmap v1785: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:34:41.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:34:41 smithi189 ceph-mon[57022]: pgmap v1786: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:34:41.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:34:41 smithi062 ceph-mon[76322]: pgmap v1786: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:34:43.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:34:43 smithi189 ceph-mon[57022]: pgmap v1787: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:34:43.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:34:43 smithi062 ceph-mon[76322]: pgmap v1787: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:34:45.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:34:45 smithi189 ceph-mon[57022]: pgmap v1788: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:34:45.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:34:45 smithi062 ceph-mon[76322]: pgmap v1788: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:34:47.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:34:47 smithi189 ceph-mon[57022]: pgmap v1789: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:34:47.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:34:47 smithi062 ceph-mon[76322]: pgmap v1789: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:34:49.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:34:49 smithi189 ceph-mon[57022]: pgmap v1790: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:34:49.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:34:49 smithi062 ceph-mon[76322]: pgmap v1790: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:34:50.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:34:50 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:34:50.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:34:50 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:34:51.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:34:51 smithi189 ceph-mon[57022]: pgmap v1791: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:34:51.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:34:51 smithi062 ceph-mon[76322]: pgmap v1791: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:34:53.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:34:53 smithi189 ceph-mon[57022]: pgmap v1792: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:34:53.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:34:53 smithi062 ceph-mon[76322]: pgmap v1792: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:34:55.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:34:55 smithi189 ceph-mon[57022]: pgmap v1793: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:34:55.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:34:55 smithi062 ceph-mon[76322]: pgmap v1793: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:34:57.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:34:57 smithi189 ceph-mon[57022]: pgmap v1794: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:34:57.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:34:57 smithi062 ceph-mon[76322]: pgmap v1794: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:34:59.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:34:59 smithi189 ceph-mon[57022]: pgmap v1795: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-20T22:34:59.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:34:59 smithi062 ceph-mon[76322]: pgmap v1795: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-20T22:35:01.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:35:01 smithi189 ceph-mon[57022]: pgmap v1796: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-20T22:35:01.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:35:01 smithi062 ceph-mon[76322]: pgmap v1796: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-20T22:35:03.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:35:03 smithi189 ceph-mon[57022]: pgmap v1797: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-20T22:35:03.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:35:03 smithi062 ceph-mon[76322]: pgmap v1797: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-20T22:35:05.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:35:05 smithi189 ceph-mon[57022]: pgmap v1798: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-20T22:35:05.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:35:05 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:35:05.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:35:05 smithi062 ceph-mon[76322]: pgmap v1798: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-20T22:35:05.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:35:05 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:35:07.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:35:07 smithi189 ceph-mon[57022]: pgmap v1799: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-20T22:35:07.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:35:07 smithi062 ceph-mon[76322]: pgmap v1799: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-20T22:35:09.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:35:09 smithi189 ceph-mon[57022]: pgmap v1800: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-20T22:35:09.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:35:09 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T22:35:09.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:35:09 smithi062 ceph-mon[76322]: pgmap v1800: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-20T22:35:09.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:35:09 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T22:35:10.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:35:10 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T22:35:10.301 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:35:10 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T22:35:10.301 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:35:10 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:35:10.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:35:10 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T22:35:10.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:35:10 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T22:35:10.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:35:10 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:35:11.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:35:11 smithi189 ceph-mon[57022]: pgmap v1801: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:35:11.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:35:11 smithi062 ceph-mon[76322]: pgmap v1801: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:35:13.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:35:13 smithi189 ceph-mon[57022]: pgmap v1802: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:35:13.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:35:13 smithi062 ceph-mon[76322]: pgmap v1802: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:35:15.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:35:15 smithi189 ceph-mon[57022]: pgmap v1803: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:35:15.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:35:15 smithi062 ceph-mon[76322]: pgmap v1803: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:35:17.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:35:17 smithi189 ceph-mon[57022]: pgmap v1804: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:35:17.535 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:35:17 smithi062 ceph-mon[76322]: pgmap v1804: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:35:19.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:35:19 smithi062 ceph-mon[76322]: pgmap v1805: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:35:19.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:35:19 smithi189 ceph-mon[57022]: pgmap v1805: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:35:20.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:35:20 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:35:20.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:35:20 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:35:21.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:35:21 smithi062 ceph-mon[76322]: pgmap v1806: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:35:21.737 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:35:21 smithi189 ceph-mon[57022]: pgmap v1806: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:35:23.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:35:23 smithi062 ceph-mon[76322]: pgmap v1807: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:35:23.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:35:23 smithi189 ceph-mon[57022]: pgmap v1807: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:35:25.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:35:25 smithi062 ceph-mon[76322]: pgmap v1808: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:35:25.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:35:25 smithi189 ceph-mon[57022]: pgmap v1808: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:35:27.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:35:27 smithi062 ceph-mon[76322]: pgmap v1809: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:35:27.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:35:27 smithi189 ceph-mon[57022]: pgmap v1809: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:35:29.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:35:29 smithi062 ceph-mon[76322]: pgmap v1810: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:35:29.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:35:29 smithi189 ceph-mon[57022]: pgmap v1810: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:35:31.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:35:31 smithi189 ceph-mon[57022]: pgmap v1811: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:35:31.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:35:31 smithi062 ceph-mon[76322]: pgmap v1811: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:35:33.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:35:33 smithi189 ceph-mon[57022]: pgmap v1812: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:35:33.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:35:33 smithi062 ceph-mon[76322]: pgmap v1812: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:35:35.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:35:35 smithi189 ceph-mon[57022]: pgmap v1813: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:35:35.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:35:35 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:35:35.784 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:35:35 smithi062 ceph-mon[76322]: pgmap v1813: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:35:35.784 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:35:35 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:35:37.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:35:37 smithi189 ceph-mon[57022]: pgmap v1814: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:35:37.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:35:37 smithi062 ceph-mon[76322]: pgmap v1814: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:35:39.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:35:39 smithi189 ceph-mon[57022]: pgmap v1815: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:35:39.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:35:39 smithi062 ceph-mon[76322]: pgmap v1815: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:35:41.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:35:41 smithi189 ceph-mon[57022]: pgmap v1816: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:35:41.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:35:41 smithi062 ceph-mon[76322]: pgmap v1816: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:35:43.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:35:43 smithi189 ceph-mon[57022]: pgmap v1817: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:35:43.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:35:43 smithi062 ceph-mon[76322]: pgmap v1817: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:35:45.607 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:35:45 smithi189 ceph-mon[57022]: pgmap v1818: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:35:45.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:35:45 smithi062 ceph-mon[76322]: pgmap v1818: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:35:47.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:35:47 smithi189 ceph-mon[57022]: pgmap v1819: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:35:47.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:35:47 smithi062 ceph-mon[76322]: pgmap v1819: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:35:49.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:35:49 smithi189 ceph-mon[57022]: pgmap v1820: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:35:49.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:35:49 smithi062 ceph-mon[76322]: pgmap v1820: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:35:50.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:35:50 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:35:50.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:35:50 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:35:51.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:35:51 smithi189 ceph-mon[57022]: pgmap v1821: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:35:51.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:35:51 smithi062 ceph-mon[76322]: pgmap v1821: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:35:53.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:35:53 smithi189 ceph-mon[57022]: pgmap v1822: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:35:53.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:35:53 smithi062 ceph-mon[76322]: pgmap v1822: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:35:55.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:35:55 smithi189 ceph-mon[57022]: pgmap v1823: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:35:55.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:35:55 smithi062 ceph-mon[76322]: pgmap v1823: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:35:57.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:35:57 smithi189 ceph-mon[57022]: pgmap v1824: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:35:57.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:35:57 smithi062 ceph-mon[76322]: pgmap v1824: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:35:59.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:35:59 smithi189 ceph-mon[57022]: pgmap v1825: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:35:59.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:35:59 smithi062 ceph-mon[76322]: pgmap v1825: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:36:01.668 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:36:01 smithi062 ceph-mon[76322]: pgmap v1826: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:36:01.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:36:01 smithi189 ceph-mon[57022]: pgmap v1826: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:36:03.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:36:03 smithi189 ceph-mon[57022]: pgmap v1827: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:36:03.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:36:03 smithi062 ceph-mon[76322]: pgmap v1827: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:36:05.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:36:05 smithi189 ceph-mon[57022]: pgmap v1828: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:36:05.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:36:05 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:36:05.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:36:05 smithi062 ceph-mon[76322]: pgmap v1828: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:36:05.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:36:05 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:36:07.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:36:07 smithi189 ceph-mon[57022]: pgmap v1829: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:36:07.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:36:07 smithi062 ceph-mon[76322]: pgmap v1829: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:36:09.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:36:09 smithi189 ceph-mon[57022]: pgmap v1830: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:36:09.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:36:09 smithi062 ceph-mon[76322]: pgmap v1830: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:36:10.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:36:10 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T22:36:10.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:36:10 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T22:36:11.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:36:11 smithi189 ceph-mon[57022]: pgmap v1831: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:36:11.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:36:11 smithi062 ceph-mon[76322]: pgmap v1831: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:36:12.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:36:12 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T22:36:12.779 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:36:12 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T22:36:12.779 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:36:12 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:36:12.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:36:12 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T22:36:12.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:36:12 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T22:36:12.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:36:12 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' 2024-04-20T22:36:13.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:36:13 smithi189 ceph-mon[57022]: pgmap v1832: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:36:14.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:36:13 smithi062 ceph-mon[76322]: pgmap v1832: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:36:15.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:36:15 smithi189 ceph-mon[57022]: pgmap v1833: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:36:16.035 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:36:15 smithi062 ceph-mon[76322]: pgmap v1833: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:36:17.889 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:36:17 smithi189 ceph-mon[57022]: pgmap v1834: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:36:18.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:36:17 smithi062 ceph-mon[76322]: pgmap v1834: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:36:19.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:36:19 smithi189 ceph-mon[57022]: pgmap v1835: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:36:19.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:36:19 smithi189 ceph-mon[57022]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:36:20.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:36:19 smithi062 ceph-mon[76322]: pgmap v1835: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:36:20.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:36:19 smithi062 ceph-mon[76322]: from='mgr.34102 172.21.15.62:0/3067961676' entity='mgr.smithi062.rqxuxi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-20T22:36:20.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:36:20 smithi189 ceph-mon[57022]: pgmap v1836: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:36:21.036 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:36:20 smithi062 ceph-mon[76322]: pgmap v1836: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:36:23.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:36:23 smithi189 ceph-mon[57022]: pgmap v1837: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:36:23.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:36:23 smithi062 ceph-mon[76322]: pgmap v1837: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:36:25.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:36:25 smithi189 ceph-mon[57022]: pgmap v1838: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:36:25.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:36:25 smithi062 ceph-mon[76322]: pgmap v1838: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:36:27.238 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:36:27 smithi189 ceph-mon[57022]: pgmap v1839: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:36:27.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:36:27 smithi062 ceph-mon[76322]: pgmap v1839: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-20T22:36:28.939 ERROR:tasks.cephfs.fuse_mount:process failed to terminate after unmount. This probably indicates a bug within ceph-fuse. 2024-04-20T22:36:28.940 ERROR:teuthology.run_tasks:Manager failed: ceph-fuse Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_a8c84f5997950fd7d008204839253291d52e6eda/qa/tasks/ceph_fuse.py", line 183, in task mount.umount_wait() File "/home/teuthworker/src/git.ceph.com_ceph-c_a8c84f5997950fd7d008204839253291d52e6eda/qa/tasks/cephfs/fuse_mount.py", line 403, in umount_wait run.wait([self.fuse_daemon], timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2024-04-20T22:36:28.941 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-20T22:36:28.952 INFO:tasks.cephadm:Teardown begin 2024-04-20T22:36:28.952 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_a8c84f5997950fd7d008204839253291d52e6eda/qa/tasks/cephadm.py", line 2246, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_a8c84f5997950fd7d008204839253291d52e6eda/qa/tasks/ceph_fuse.py", line 183, in task mount.umount_wait() File "/home/teuthworker/src/git.ceph.com_ceph-c_a8c84f5997950fd7d008204839253291d52e6eda/qa/tasks/cephfs/fuse_mount.py", line 403, in umount_wait run.wait([self.fuse_daemon], timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2024-04-20T22:36:28.953 DEBUG:teuthology.orchestra.run.smithi062:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-20T22:36:29.049 DEBUG:teuthology.orchestra.run.smithi189:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-20T22:36:29.148 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-20T22:36:29.148 DEBUG:teuthology.orchestra.run.smithi062:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-20T22:36:29.207 DEBUG:teuthology.orchestra.run.smithi189:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-20T22:36:29.262 INFO:tasks.cephadm:Stopping all daemons... 2024-04-20T22:36:29.262 INFO:tasks.cephadm.mon.smithi062:Stopping mon.smithi062... 2024-04-20T22:36:29.262 DEBUG:teuthology.orchestra.run.smithi062:> sudo systemctl stop ceph-61ae3628-ff5b-11ee-bc93-c7b262605968@mon.smithi062 2024-04-20T22:36:29.285 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:36:29 smithi062 ceph-mon[76322]: pgmap v1840: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:36:29.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:36:29 smithi189 ceph-mon[57022]: pgmap v1840: 65 pgs: 65 active+clean; 315 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-20T22:36:29.786 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:36:29 smithi062 systemd[1]: Stopping Ceph mon.smithi062 for 61ae3628-ff5b-11ee-bc93-c7b262605968... 2024-04-20T22:36:30.202 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:36:29 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-mon-smithi062[76318]: 2024-04-20T22:36:29.913+0000 7f615ba00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi062 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-20T22:36:30.202 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:36:29 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-mon-smithi062[76318]: 2024-04-20T22:36:29.913+0000 7f615ba00700 -1 mon.smithi062@0(leader) e3 *** Got Signal Terminated *** 2024-04-20T22:36:30.536 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:36:30 smithi062 podman[140871]: 2024-04-20 22:36:30.202595315 +0000 UTC m=+0.450157833 container died 019eb14794852cc40f5e7beeaa6d021dc8a8cf60ed30bdb363900c0cd8f6e1d7 (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-mon-smithi062, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-20T22:36:31.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:36:30 smithi062 podman[140871]: 2024-04-20 22:36:30.873565142 +0000 UTC m=+1.121127653 container cleanup 019eb14794852cc40f5e7beeaa6d021dc8a8cf60ed30bdb363900c0cd8f6e1d7 (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-mon-smithi062, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-04-20T22:36:31.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:36:30 smithi062 bash[140871]: ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-mon-smithi062 2024-04-20T22:36:31.286 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Apr 20 22:36:31 smithi062 podman[140897]: 2024-04-20 22:36:31.083591315 +0000 UTC m=+0.866160538 container remove 019eb14794852cc40f5e7beeaa6d021dc8a8cf60ed30bdb363900c0cd8f6e1d7 (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-mon-smithi062, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-04-20T22:36:31.628 DEBUG:teuthology.orchestra.run.smithi062:> sudo pkill -f 'journalctl -f -n 0 -u ceph-61ae3628-ff5b-11ee-bc93-c7b262605968@mon.smithi062.service' 2024-04-20T22:36:31.833 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-20T22:36:31.833 INFO:tasks.cephadm.mon.smithi062:Stopped mon.smithi062 2024-04-20T22:36:31.833 INFO:tasks.cephadm.mon.smithi189:Stopping mon.smithi189... 2024-04-20T22:36:31.834 DEBUG:teuthology.orchestra.run.smithi189:> sudo systemctl stop ceph-61ae3628-ff5b-11ee-bc93-c7b262605968@mon.smithi189 2024-04-20T22:36:32.237 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:36:31 smithi189 systemd[1]: Stopping Ceph mon.smithi189 for 61ae3628-ff5b-11ee-bc93-c7b262605968... 2024-04-20T22:36:32.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:36:32 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-mon-smithi189[57018]: 2024-04-20T22:36:32.461+0000 7f40e4000700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi189 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-20T22:36:32.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:36:32 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-mon-smithi189[57018]: 2024-04-20T22:36:32.461+0000 7f40e4000700 -1 mon.smithi189@1(peon) e3 *** Got Signal Terminated *** 2024-04-20T22:36:32.738 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:36:32 smithi189 podman[93097]: 2024-04-20 22:36:32.710110837 +0000 UTC m=+0.430173726 container died 55aac560aab6cf464252775feab1579f97a33fad9d546da9a8dc6e47acd49d99 (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-mon-smithi189, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5) 2024-04-20T22:36:33.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:36:33 smithi189 podman[93097]: 2024-04-20 22:36:33.196900254 +0000 UTC m=+0.916963166 container cleanup 55aac560aab6cf464252775feab1579f97a33fad9d546da9a8dc6e47acd49d99 (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-mon-smithi189, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-20T22:36:33.488 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:36:33 smithi189 bash[93097]: ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-mon-smithi189 2024-04-20T22:36:33.988 INFO:journalctl@ceph.mon.smithi189.smithi189.stdout:Apr 20 22:36:33 smithi189 podman[93111]: 2024-04-20 22:36:33.682324376 +0000 UTC m=+0.952556579 container remove 55aac560aab6cf464252775feab1579f97a33fad9d546da9a8dc6e47acd49d99 (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-mon-smithi189, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-04-20T22:36:34.047 DEBUG:teuthology.orchestra.run.smithi189:> sudo pkill -f 'journalctl -f -n 0 -u ceph-61ae3628-ff5b-11ee-bc93-c7b262605968@mon.smithi189.service' 2024-04-20T22:36:34.161 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-20T22:36:34.161 INFO:tasks.cephadm.mon.smithi189:Stopped mon.smithi189 2024-04-20T22:36:34.161 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-04-20T22:36:34.162 DEBUG:teuthology.orchestra.run.smithi062:> sudo systemctl stop ceph-61ae3628-ff5b-11ee-bc93-c7b262605968@osd.0 2024-04-20T22:36:34.533 INFO:journalctl@ceph.osd.0.smithi062.stdout:Apr 20 22:36:34 smithi062 systemd[1]: Stopping Ceph osd.0 for 61ae3628-ff5b-11ee-bc93-c7b262605968... 2024-04-20T22:36:34.787 INFO:journalctl@ceph.osd.0.smithi062.stdout:Apr 20 22:36:34 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-0[88146]: 2024-04-20T22:36:34.725+0000 7fe7cde00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-20T22:36:34.788 INFO:journalctl@ceph.osd.0.smithi062.stdout:Apr 20 22:36:34 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-0[88146]: 2024-04-20T22:36:34.725+0000 7fe7cde00700 -1 osd.0 85 *** Got signal Terminated *** 2024-04-20T22:36:34.788 INFO:journalctl@ceph.osd.0.smithi062.stdout:Apr 20 22:36:34 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-0[88146]: 2024-04-20T22:36:34.725+0000 7fe7cde00700 -1 osd.0 85 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-20T22:36:40.536 INFO:journalctl@ceph.osd.0.smithi062.stdout:Apr 20 22:36:40 smithi062 podman[141015]: 2024-04-20 22:36:40.265039909 +0000 UTC m=+5.683045578 container died ee2bb901f26e6a4de27453ff07542d1965f02e9ac6d2b6d88d4004da7d789b60 (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-20T22:36:40.955 INFO:journalctl@ceph.osd.0.smithi062.stdout:Apr 20 22:36:40 smithi062 podman[141015]: 2024-04-20 22:36:40.720276064 +0000 UTC m=+6.138281763 container cleanup ee2bb901f26e6a4de27453ff07542d1965f02e9ac6d2b6d88d4004da7d789b60 (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS) 2024-04-20T22:36:40.955 INFO:journalctl@ceph.osd.0.smithi062.stdout:Apr 20 22:36:40 smithi062 bash[141015]: ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-0 2024-04-20T22:36:41.286 INFO:journalctl@ceph.osd.0.smithi062.stdout:Apr 20 22:36:40 smithi062 podman[141035]: 2024-04-20 22:36:40.954730805 +0000 UTC m=+0.676527728 container remove ee2bb901f26e6a4de27453ff07542d1965f02e9ac6d2b6d88d4004da7d789b60 (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5) 2024-04-20T22:36:42.160 INFO:journalctl@ceph.osd.0.smithi062.stdout:Apr 20 22:36:41 smithi062 podman[141129]: 2024-04-20 22:36:41.750870738 +0000 UTC m=+0.035091046 image pull 76bc6b91b726a03b8b736eeeb35fc1ffd1469e23c7d60f5b444a217528d8906d quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 2024-04-20T22:36:42.160 INFO:journalctl@ceph.osd.0.smithi062.stdout:Apr 20 22:36:41 smithi062 podman[141129]: 2024-04-20 22:36:41.86432066 +0000 UTC m=+0.148540992 container create 65828a005943f6126f2037e93ba628a4516307d5f7d719f5442e06f1071b322b (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-0-deactivate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-20T22:36:42.160 INFO:journalctl@ceph.osd.0.smithi062.stdout:Apr 20 22:36:42 smithi062 podman[141129]: 2024-04-20 22:36:42.159953344 +0000 UTC m=+0.444173719 container init 65828a005943f6126f2037e93ba628a4516307d5f7d719f5442e06f1071b322b (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-0-deactivate, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-04-20T22:36:42.536 INFO:journalctl@ceph.osd.0.smithi062.stdout:Apr 20 22:36:42 smithi062 podman[141129]: 2024-04-20 22:36:42.171832256 +0000 UTC m=+0.456052561 container start 65828a005943f6126f2037e93ba628a4516307d5f7d719f5442e06f1071b322b (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-0-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2) 2024-04-20T22:36:42.536 INFO:journalctl@ceph.osd.0.smithi062.stdout:Apr 20 22:36:42 smithi062 podman[141129]: 2024-04-20 22:36:42.275220585 +0000 UTC m=+0.559440957 container attach 65828a005943f6126f2037e93ba628a4516307d5f7d719f5442e06f1071b322b (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-0-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, org.label-schema.build-date=20240416, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-04-20T22:36:43.036 INFO:journalctl@ceph.osd.0.smithi062.stdout:Apr 20 22:36:42 smithi062 podman[141129]: 2024-04-20 22:36:42.637252113 +0000 UTC m=+0.921472485 container died 65828a005943f6126f2037e93ba628a4516307d5f7d719f5442e06f1071b322b (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-0-deactivate, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-20T22:36:43.638 DEBUG:teuthology.orchestra.run.smithi062:> sudo pkill -f 'journalctl -f -n 0 -u ceph-61ae3628-ff5b-11ee-bc93-c7b262605968@osd.0.service' 2024-04-20T22:36:43.759 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-20T22:36:43.760 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-04-20T22:36:43.760 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-20T22:36:43.760 DEBUG:teuthology.orchestra.run.smithi062:> sudo systemctl stop ceph-61ae3628-ff5b-11ee-bc93-c7b262605968@osd.1 2024-04-20T22:36:44.286 INFO:journalctl@ceph.osd.1.smithi062.stdout:Apr 20 22:36:43 smithi062 systemd[1]: Stopping Ceph osd.1 for 61ae3628-ff5b-11ee-bc93-c7b262605968... 2024-04-20T22:36:44.786 INFO:journalctl@ceph.osd.1.smithi062.stdout:Apr 20 22:36:44 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-1[93530]: 2024-04-20T22:36:44.399+0000 7f221ec00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-20T22:36:44.786 INFO:journalctl@ceph.osd.1.smithi062.stdout:Apr 20 22:36:44 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-1[93530]: 2024-04-20T22:36:44.399+0000 7f221ec00700 -1 osd.1 85 *** Got signal Terminated *** 2024-04-20T22:36:44.786 INFO:journalctl@ceph.osd.1.smithi062.stdout:Apr 20 22:36:44 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-1[93530]: 2024-04-20T22:36:44.399+0000 7f221ec00700 -1 osd.1 85 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-20T22:36:50.206 INFO:journalctl@ceph.osd.1.smithi062.stdout:Apr 20 22:36:49 smithi062 podman[141235]: 2024-04-20 22:36:49.842826934 +0000 UTC m=+5.595368272 container died 9075152d41629e2a60658c8ccccf136d6a2a45811f0c7a9786e49a741d6d62ca (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-1, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-04-20T22:36:50.536 INFO:journalctl@ceph.osd.1.smithi062.stdout:Apr 20 22:36:50 smithi062 podman[141235]: 2024-04-20 22:36:50.341069503 +0000 UTC m=+6.093610878 container cleanup 9075152d41629e2a60658c8ccccf136d6a2a45811f0c7a9786e49a741d6d62ca (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-1, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-20T22:36:50.536 INFO:journalctl@ceph.osd.1.smithi062.stdout:Apr 20 22:36:50 smithi062 bash[141235]: ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-1 2024-04-20T22:36:51.036 INFO:journalctl@ceph.osd.1.smithi062.stdout:Apr 20 22:36:50 smithi062 podman[141256]: 2024-04-20 22:36:50.592594429 +0000 UTC m=+0.739696103 container remove 9075152d41629e2a60658c8ccccf136d6a2a45811f0c7a9786e49a741d6d62ca (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-1, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-04-20T22:36:51.786 INFO:journalctl@ceph.osd.1.smithi062.stdout:Apr 20 22:36:51 smithi062 podman[141346]: 2024-04-20 22:36:51.408234097 +0000 UTC m=+0.043791165 image pull 76bc6b91b726a03b8b736eeeb35fc1ffd1469e23c7d60f5b444a217528d8906d quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 2024-04-20T22:36:51.786 INFO:journalctl@ceph.osd.1.smithi062.stdout:Apr 20 22:36:51 smithi062 podman[141346]: 2024-04-20 22:36:51.510274538 +0000 UTC m=+0.145831602 container create b9e15e429330a58690509387665f06b2c9cc001ec4fb1c03e6a027a6056984b2 (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-1-deactivate, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, CEPH_POINT_RELEASE=) 2024-04-20T22:36:52.263 INFO:journalctl@ceph.osd.1.smithi062.stdout:Apr 20 22:36:51 smithi062 podman[141346]: 2024-04-20 22:36:51.788987074 +0000 UTC m=+0.424544164 container init b9e15e429330a58690509387665f06b2c9cc001ec4fb1c03e6a027a6056984b2 (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-1-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-04-20T22:36:52.263 INFO:journalctl@ceph.osd.1.smithi062.stdout:Apr 20 22:36:51 smithi062 podman[141346]: 2024-04-20 22:36:51.80239723 +0000 UTC m=+0.437954283 container start b9e15e429330a58690509387665f06b2c9cc001ec4fb1c03e6a027a6056984b2 (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-04-20T22:36:52.263 INFO:journalctl@ceph.osd.1.smithi062.stdout:Apr 20 22:36:51 smithi062 podman[141346]: 2024-04-20 22:36:51.887689056 +0000 UTC m=+0.523246180 container attach b9e15e429330a58690509387665f06b2c9cc001ec4fb1c03e6a027a6056984b2 (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-1-deactivate, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-20T22:36:52.263 INFO:journalctl@ceph.osd.1.smithi062.stdout:Apr 20 22:36:52 smithi062 podman[141346]: 2024-04-20 22:36:52.263845346 +0000 UTC m=+0.899402450 container died b9e15e429330a58690509387665f06b2c9cc001ec4fb1c03e6a027a6056984b2 (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-1-deactivate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2) 2024-04-20T22:36:53.076 DEBUG:teuthology.orchestra.run.smithi062:> sudo pkill -f 'journalctl -f -n 0 -u ceph-61ae3628-ff5b-11ee-bc93-c7b262605968@osd.1.service' 2024-04-20T22:36:53.176 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-20T22:36:53.177 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-20T22:36:53.177 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-20T22:36:53.177 DEBUG:teuthology.orchestra.run.smithi062:> sudo systemctl stop ceph-61ae3628-ff5b-11ee-bc93-c7b262605968@osd.2 2024-04-20T22:36:53.786 INFO:journalctl@ceph.osd.2.smithi062.stdout:Apr 20 22:36:53 smithi062 systemd[1]: Stopping Ceph osd.2 for 61ae3628-ff5b-11ee-bc93-c7b262605968... 2024-04-20T22:36:54.286 INFO:journalctl@ceph.osd.2.smithi062.stdout:Apr 20 22:36:53 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-2[99074]: 2024-04-20T22:36:53.800+0000 7fdc0d600700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-20T22:36:54.286 INFO:journalctl@ceph.osd.2.smithi062.stdout:Apr 20 22:36:53 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-2[99074]: 2024-04-20T22:36:53.800+0000 7fdc0d600700 -1 osd.2 85 *** Got signal Terminated *** 2024-04-20T22:36:54.286 INFO:journalctl@ceph.osd.2.smithi062.stdout:Apr 20 22:36:53 smithi062 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-2[99074]: 2024-04-20T22:36:53.800+0000 7fdc0d600700 -1 osd.2 85 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-20T22:36:59.725 INFO:journalctl@ceph.osd.2.smithi062.stdout:Apr 20 22:36:59 smithi062 podman[141451]: 2024-04-20 22:36:59.392154376 +0000 UTC m=+5.716408602 container died 2d4f3e38dc0e10db7d042a56a546f30d4579ae68724f5b36ef4fce943f2b08a3 (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-2, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-20T22:37:00.038 INFO:journalctl@ceph.osd.2.smithi062.stdout:Apr 20 22:36:59 smithi062 podman[141451]: 2024-04-20 22:36:59.861840071 +0000 UTC m=+6.186094286 container cleanup 2d4f3e38dc0e10db7d042a56a546f30d4579ae68724f5b36ef4fce943f2b08a3 (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-2, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-04-20T22:37:00.038 INFO:journalctl@ceph.osd.2.smithi062.stdout:Apr 20 22:36:59 smithi062 bash[141451]: ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-2 2024-04-20T22:37:00.536 INFO:journalctl@ceph.osd.2.smithi062.stdout:Apr 20 22:37:00 smithi062 podman[141470]: 2024-04-20 22:37:00.121953149 +0000 UTC m=+0.720993414 container remove 2d4f3e38dc0e10db7d042a56a546f30d4579ae68724f5b36ef4fce943f2b08a3 (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416) 2024-04-20T22:37:00.988 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 20 22:37:00 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-4[71392]: 2024-04-20T22:37:00.516+0000 7f2f33c00700 -1 osd.4 85 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-04-20T22:36:36.242121+0000 front 2024-04-20T22:36:36.242237+0000 (oldest deadline 2024-04-20T22:37:00.341764+0000) 2024-04-20T22:37:01.143 INFO:journalctl@ceph.osd.2.smithi062.stdout:Apr 20 22:37:00 smithi062 podman[141574]: 2024-04-20 22:37:00.788035286 +0000 UTC m=+0.034159394 image pull 76bc6b91b726a03b8b736eeeb35fc1ffd1469e23c7d60f5b444a217528d8906d quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 2024-04-20T22:37:01.143 INFO:journalctl@ceph.osd.2.smithi062.stdout:Apr 20 22:37:00 smithi062 podman[141574]: 2024-04-20 22:37:00.889259063 +0000 UTC m=+0.135383199 container create 0fbb9952db49b791af92ec37555abaa3272091c9740fa1f317b4af0c8934fd45 (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-2-deactivate, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-20T22:37:01.144 INFO:journalctl@ceph.osd.2.smithi062.stdout:Apr 20 22:37:01 smithi062 podman[141574]: 2024-04-20 22:37:01.142910501 +0000 UTC m=+0.389034652 container init 0fbb9952db49b791af92ec37555abaa3272091c9740fa1f317b4af0c8934fd45 (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-2-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-20T22:37:01.536 INFO:journalctl@ceph.osd.2.smithi062.stdout:Apr 20 22:37:01 smithi062 podman[141574]: 2024-04-20 22:37:01.154372874 +0000 UTC m=+0.400496994 container start 0fbb9952db49b791af92ec37555abaa3272091c9740fa1f317b4af0c8934fd45 (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-2-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-20T22:37:01.536 INFO:journalctl@ceph.osd.2.smithi062.stdout:Apr 20 22:37:01 smithi062 podman[141574]: 2024-04-20 22:37:01.249861848 +0000 UTC m=+0.495986037 container attach 0fbb9952db49b791af92ec37555abaa3272091c9740fa1f317b4af0c8934fd45 (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-2-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-20T22:37:01.979 INFO:journalctl@ceph.osd.2.smithi062.stdout:Apr 20 22:37:01 smithi062 podman[141574]: 2024-04-20 22:37:01.61055571 +0000 UTC m=+0.856679857 container died 0fbb9952db49b791af92ec37555abaa3272091c9740fa1f317b4af0c8934fd45 (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-2-deactivate, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416) 2024-04-20T22:37:01.988 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 20 22:37:01 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-4[71392]: 2024-04-20T22:37:01.512+0000 7f2f33c00700 -1 osd.4 85 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-04-20T22:36:36.242121+0000 front 2024-04-20T22:36:36.242237+0000 (oldest deadline 2024-04-20T22:37:00.341764+0000) 2024-04-20T22:37:02.398 DEBUG:teuthology.orchestra.run.smithi062:> sudo pkill -f 'journalctl -f -n 0 -u ceph-61ae3628-ff5b-11ee-bc93-c7b262605968@osd.2.service' 2024-04-20T22:37:02.508 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-20T22:37:02.509 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-20T22:37:02.509 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-04-20T22:37:02.509 DEBUG:teuthology.orchestra.run.smithi189:> sudo systemctl stop ceph-61ae3628-ff5b-11ee-bc93-c7b262605968@osd.3 2024-04-20T22:37:02.919 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 20 22:37:02 smithi189 systemd[1]: Stopping Ceph osd.3 for 61ae3628-ff5b-11ee-bc93-c7b262605968... 2024-04-20T22:37:02.919 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 20 22:37:02 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-4[71392]: 2024-04-20T22:37:02.556+0000 7f2f33c00700 -1 osd.4 85 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-04-20T22:36:36.242121+0000 front 2024-04-20T22:36:36.242237+0000 (oldest deadline 2024-04-20T22:37:00.341764+0000) 2024-04-20T22:37:03.238 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 20 22:37:03 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-3[66504]: 2024-04-20T22:37:03.117+0000 7f05ada00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-20T22:37:03.238 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 20 22:37:03 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-3[66504]: 2024-04-20T22:37:03.117+0000 7f05ada00700 -1 osd.3 85 *** Got signal Terminated *** 2024-04-20T22:37:03.238 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 20 22:37:03 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-3[66504]: 2024-04-20T22:37:03.117+0000 7f05ada00700 -1 osd.3 85 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-20T22:37:03.238 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 22:37:02 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-5[76275]: 2024-04-20T22:37:02.918+0000 7f3164200700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-04-20T22:36:38.027525+0000 front 2024-04-20T22:36:38.027687+0000 (oldest deadline 2024-04-20T22:37:02.727494+0000) 2024-04-20T22:37:03.920 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 20 22:37:03 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-4[71392]: 2024-04-20T22:37:03.523+0000 7f2f33c00700 -1 osd.4 85 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-04-20T22:36:36.242121+0000 front 2024-04-20T22:36:36.242237+0000 (oldest deadline 2024-04-20T22:37:00.341764+0000) 2024-04-20T22:37:04.238 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 22:37:03 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-5[76275]: 2024-04-20T22:37:03.920+0000 7f3164200700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-04-20T22:36:38.027525+0000 front 2024-04-20T22:36:38.027687+0000 (oldest deadline 2024-04-20T22:37:02.727494+0000) 2024-04-20T22:37:04.908 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 20 22:37:04 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-3[66504]: 2024-04-20T22:37:04.657+0000 7f05a9200700 -1 osd.3 85 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-04-20T22:36:38.377096+0000 front 2024-04-20T22:36:38.377291+0000 (oldest deadline 2024-04-20T22:37:03.676835+0000) 2024-04-20T22:37:04.908 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 20 22:37:04 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-4[71392]: 2024-04-20T22:37:04.565+0000 7f2f33c00700 -1 osd.4 85 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-04-20T22:36:36.242121+0000 front 2024-04-20T22:36:36.242237+0000 (oldest deadline 2024-04-20T22:37:00.341764+0000) 2024-04-20T22:37:05.238 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 22:37:04 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-5[76275]: 2024-04-20T22:37:04.907+0000 7f3164200700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-04-20T22:36:38.027525+0000 front 2024-04-20T22:36:38.027687+0000 (oldest deadline 2024-04-20T22:37:02.727494+0000) 2024-04-20T22:37:05.908 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 20 22:37:05 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-3[66504]: 2024-04-20T22:37:05.624+0000 7f05a9200700 -1 osd.3 85 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-04-20T22:36:38.377096+0000 front 2024-04-20T22:36:38.377291+0000 (oldest deadline 2024-04-20T22:37:03.676835+0000) 2024-04-20T22:37:05.908 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 20 22:37:05 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-4[71392]: 2024-04-20T22:37:05.582+0000 7f2f33c00700 -1 osd.4 85 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-04-20T22:36:36.242121+0000 front 2024-04-20T22:36:36.242237+0000 (oldest deadline 2024-04-20T22:37:00.341764+0000) 2024-04-20T22:37:06.238 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 22:37:05 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-5[76275]: 2024-04-20T22:37:05.907+0000 7f3164200700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-04-20T22:36:38.027525+0000 front 2024-04-20T22:36:38.027687+0000 (oldest deadline 2024-04-20T22:37:02.727494+0000) 2024-04-20T22:37:06.935 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 20 22:37:06 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-3[66504]: 2024-04-20T22:37:06.670+0000 7f05a9200700 -1 osd.3 85 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-04-20T22:36:38.377096+0000 front 2024-04-20T22:36:38.377291+0000 (oldest deadline 2024-04-20T22:37:03.676835+0000) 2024-04-20T22:37:06.935 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 20 22:37:06 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-4[71392]: 2024-04-20T22:37:06.581+0000 7f2f33c00700 -1 osd.4 85 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-04-20T22:36:36.242121+0000 front 2024-04-20T22:36:36.242237+0000 (oldest deadline 2024-04-20T22:37:00.341764+0000) 2024-04-20T22:37:07.238 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 22:37:06 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-5[76275]: 2024-04-20T22:37:06.934+0000 7f3164200700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-04-20T22:36:38.027525+0000 front 2024-04-20T22:36:38.027687+0000 (oldest deadline 2024-04-20T22:37:02.727494+0000) 2024-04-20T22:37:07.906 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 20 22:37:07 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-3[66504]: 2024-04-20T22:37:07.692+0000 7f05a9200700 -1 osd.3 85 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-04-20T22:36:38.377096+0000 front 2024-04-20T22:36:38.377291+0000 (oldest deadline 2024-04-20T22:37:03.676835+0000) 2024-04-20T22:37:07.906 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 20 22:37:07 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-4[71392]: 2024-04-20T22:37:07.535+0000 7f2f33c00700 -1 osd.4 85 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-04-20T22:36:36.242121+0000 front 2024-04-20T22:36:36.242237+0000 (oldest deadline 2024-04-20T22:37:00.341764+0000) 2024-04-20T22:37:08.196 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 22:37:07 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-5[76275]: 2024-04-20T22:37:07.905+0000 7f3164200700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-04-20T22:36:38.027525+0000 front 2024-04-20T22:36:38.027687+0000 (oldest deadline 2024-04-20T22:37:02.727494+0000) 2024-04-20T22:37:08.874 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 20 22:37:08 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-4[71392]: 2024-04-20T22:37:08.533+0000 7f2f33c00700 -1 osd.4 85 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-04-20T22:36:36.242121+0000 front 2024-04-20T22:36:36.242237+0000 (oldest deadline 2024-04-20T22:37:00.341764+0000) 2024-04-20T22:37:08.874 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 20 22:37:08 smithi189 podman[93234]: 2024-04-20 22:37:08.675126534 +0000 UTC m=+5.730465042 container died 8f97b17ae0ebb32dda871a6d08fd56aaba57016a65a45f736c36aecb0de36b6f (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-3, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-20T22:37:09.239 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 22:37:08 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-5[76275]: 2024-04-20T22:37:08.873+0000 7f3164200700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-04-20T22:36:38.027525+0000 front 2024-04-20T22:36:38.027687+0000 (oldest deadline 2024-04-20T22:37:02.727494+0000) 2024-04-20T22:37:09.239 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 20 22:37:09 smithi189 podman[93234]: 2024-04-20 22:37:09.075550458 +0000 UTC m=+6.130889006 container cleanup 8f97b17ae0ebb32dda871a6d08fd56aaba57016a65a45f736c36aecb0de36b6f (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-3, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5) 2024-04-20T22:37:09.239 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 20 22:37:09 smithi189 bash[93234]: ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-3 2024-04-20T22:37:09.541 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 20 22:37:09 smithi189 podman[93252]: 2024-04-20 22:37:09.385307819 +0000 UTC m=+0.692974354 container remove 8f97b17ae0ebb32dda871a6d08fd56aaba57016a65a45f736c36aecb0de36b6f (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-3, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-20T22:37:09.912 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 20 22:37:09 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-4[71392]: 2024-04-20T22:37:09.540+0000 7f2f33c00700 -1 osd.4 85 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-04-20T22:36:36.242121+0000 front 2024-04-20T22:36:36.242237+0000 (oldest deadline 2024-04-20T22:37:00.341764+0000) 2024-04-20T22:37:10.217 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 22:37:09 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-5[76275]: 2024-04-20T22:37:09.912+0000 7f3164200700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-04-20T22:36:38.027525+0000 front 2024-04-20T22:36:38.027687+0000 (oldest deadline 2024-04-20T22:37:02.727494+0000) 2024-04-20T22:37:10.488 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 20 22:37:10 smithi189 podman[93342]: 2024-04-20 22:37:10.122361477 +0000 UTC m=+0.032369153 image pull 76bc6b91b726a03b8b736eeeb35fc1ffd1469e23c7d60f5b444a217528d8906d quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 2024-04-20T22:37:10.488 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 20 22:37:10 smithi189 podman[93342]: 2024-04-20 22:37:10.236453102 +0000 UTC m=+0.146460815 container create d3216fd78293260d384fecb866d83406a8053d506d90338a147a879b91230047 (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-3-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-20T22:37:10.919 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 20 22:37:10 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-4[71392]: 2024-04-20T22:37:10.586+0000 7f2f33c00700 -1 osd.4 85 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-04-20T22:36:36.242121+0000 front 2024-04-20T22:36:36.242237+0000 (oldest deadline 2024-04-20T22:37:00.341764+0000) 2024-04-20T22:37:10.920 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 20 22:37:10 smithi189 podman[93342]: 2024-04-20 22:37:10.517644716 +0000 UTC m=+0.427652436 container init d3216fd78293260d384fecb866d83406a8053d506d90338a147a879b91230047 (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-3-deactivate, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-04-20T22:37:10.920 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 20 22:37:10 smithi189 podman[93342]: 2024-04-20 22:37:10.527745174 +0000 UTC m=+0.437752894 container start d3216fd78293260d384fecb866d83406a8053d506d90338a147a879b91230047 (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-3-deactivate, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-04-20T22:37:10.920 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 20 22:37:10 smithi189 podman[93342]: 2024-04-20 22:37:10.6054702 +0000 UTC m=+0.515477977 container attach d3216fd78293260d384fecb866d83406a8053d506d90338a147a879b91230047 (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-20T22:37:11.238 INFO:journalctl@ceph.osd.3.smithi189.stdout:Apr 20 22:37:10 smithi189 podman[93342]: 2024-04-20 22:37:10.986668793 +0000 UTC m=+0.896676513 container died d3216fd78293260d384fecb866d83406a8053d506d90338a147a879b91230047 (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-3-deactivate, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-20T22:37:11.238 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 22:37:10 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-5[76275]: 2024-04-20T22:37:10.919+0000 7f3164200700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-04-20T22:36:38.027525+0000 front 2024-04-20T22:36:38.027687+0000 (oldest deadline 2024-04-20T22:37:02.727494+0000) 2024-04-20T22:37:11.686 DEBUG:teuthology.orchestra.run.smithi189:> sudo pkill -f 'journalctl -f -n 0 -u ceph-61ae3628-ff5b-11ee-bc93-c7b262605968@osd.3.service' 2024-04-20T22:37:11.790 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-20T22:37:11.790 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-04-20T22:37:11.790 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-04-20T22:37:11.790 DEBUG:teuthology.orchestra.run.smithi189:> sudo systemctl stop ceph-61ae3628-ff5b-11ee-bc93-c7b262605968@osd.4 2024-04-20T22:37:11.880 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 20 22:37:11 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-4[71392]: 2024-04-20T22:37:11.577+0000 7f2f33c00700 -1 osd.4 85 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-04-20T22:36:36.242121+0000 front 2024-04-20T22:36:36.242237+0000 (oldest deadline 2024-04-20T22:37:00.341764+0000) 2024-04-20T22:37:11.880 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 20 22:37:11 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-4[71392]: 2024-04-20T22:37:11.577+0000 7f2f33c00700 -1 osd.4 85 heartbeat_check: no reply from 172.21.15.62:6814 osd.1 since back 2024-04-20T22:36:46.545455+0000 front 2024-04-20T22:36:46.545774+0000 (oldest deadline 2024-04-20T22:37:10.645306+0000) 2024-04-20T22:37:12.238 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 20 22:37:11 smithi189 systemd[1]: Stopping Ceph osd.4 for 61ae3628-ff5b-11ee-bc93-c7b262605968... 2024-04-20T22:37:12.238 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 22:37:11 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-5[76275]: 2024-04-20T22:37:11.954+0000 7f3164200700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-04-20T22:36:38.027525+0000 front 2024-04-20T22:36:38.027687+0000 (oldest deadline 2024-04-20T22:37:02.727494+0000) 2024-04-20T22:37:12.238 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 22:37:11 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-5[76275]: 2024-04-20T22:37:11.954+0000 7f3164200700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.62:6814 osd.1 since back 2024-04-20T22:36:46.229550+0000 front 2024-04-20T22:36:46.229526+0000 (oldest deadline 2024-04-20T22:37:11.529195+0000) 2024-04-20T22:37:12.738 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 20 22:37:12 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-4[71392]: 2024-04-20T22:37:12.369+0000 7f2f3d400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-20T22:37:12.739 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 20 22:37:12 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-4[71392]: 2024-04-20T22:37:12.369+0000 7f2f3d400700 -1 osd.4 85 *** Got signal Terminated *** 2024-04-20T22:37:12.739 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 20 22:37:12 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-4[71392]: 2024-04-20T22:37:12.369+0000 7f2f3d400700 -1 osd.4 85 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-20T22:37:12.739 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 20 22:37:12 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-4[71392]: 2024-04-20T22:37:12.616+0000 7f2f33c00700 -1 osd.4 85 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-04-20T22:36:36.242121+0000 front 2024-04-20T22:36:36.242237+0000 (oldest deadline 2024-04-20T22:37:00.341764+0000) 2024-04-20T22:37:12.739 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 20 22:37:12 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-4[71392]: 2024-04-20T22:37:12.616+0000 7f2f33c00700 -1 osd.4 85 heartbeat_check: no reply from 172.21.15.62:6814 osd.1 since back 2024-04-20T22:36:46.545455+0000 front 2024-04-20T22:36:46.545774+0000 (oldest deadline 2024-04-20T22:37:10.645306+0000) 2024-04-20T22:37:13.344 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 22:37:12 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-5[76275]: 2024-04-20T22:37:12.989+0000 7f3164200700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-04-20T22:36:38.027525+0000 front 2024-04-20T22:36:38.027687+0000 (oldest deadline 2024-04-20T22:37:02.727494+0000) 2024-04-20T22:37:13.344 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 22:37:12 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-5[76275]: 2024-04-20T22:37:12.989+0000 7f3164200700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.62:6814 osd.1 since back 2024-04-20T22:36:46.229550+0000 front 2024-04-20T22:36:46.229526+0000 (oldest deadline 2024-04-20T22:37:11.529195+0000) 2024-04-20T22:37:13.738 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 20 22:37:13 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-4[71392]: 2024-04-20T22:37:13.576+0000 7f2f33c00700 -1 osd.4 85 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-04-20T22:36:36.242121+0000 front 2024-04-20T22:36:36.242237+0000 (oldest deadline 2024-04-20T22:37:00.341764+0000) 2024-04-20T22:37:13.738 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 20 22:37:13 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-4[71392]: 2024-04-20T22:37:13.576+0000 7f2f33c00700 -1 osd.4 85 heartbeat_check: no reply from 172.21.15.62:6814 osd.1 since back 2024-04-20T22:36:46.545455+0000 front 2024-04-20T22:36:46.545774+0000 (oldest deadline 2024-04-20T22:37:10.645306+0000) 2024-04-20T22:37:14.488 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 22:37:14 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-5[76275]: 2024-04-20T22:37:14.022+0000 7f3164200700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-04-20T22:36:38.027525+0000 front 2024-04-20T22:36:38.027687+0000 (oldest deadline 2024-04-20T22:37:02.727494+0000) 2024-04-20T22:37:14.488 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 22:37:14 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-5[76275]: 2024-04-20T22:37:14.022+0000 7f3164200700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.62:6814 osd.1 since back 2024-04-20T22:36:46.229550+0000 front 2024-04-20T22:36:46.229526+0000 (oldest deadline 2024-04-20T22:37:11.529195+0000) 2024-04-20T22:37:14.988 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 20 22:37:14 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-4[71392]: 2024-04-20T22:37:14.558+0000 7f2f33c00700 -1 osd.4 85 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-04-20T22:36:36.242121+0000 front 2024-04-20T22:36:36.242237+0000 (oldest deadline 2024-04-20T22:37:00.341764+0000) 2024-04-20T22:37:14.988 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 20 22:37:14 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-4[71392]: 2024-04-20T22:37:14.558+0000 7f2f33c00700 -1 osd.4 85 heartbeat_check: no reply from 172.21.15.62:6814 osd.1 since back 2024-04-20T22:36:46.545455+0000 front 2024-04-20T22:36:46.545774+0000 (oldest deadline 2024-04-20T22:37:10.645306+0000) 2024-04-20T22:37:15.488 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 22:37:15 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-5[76275]: 2024-04-20T22:37:15.050+0000 7f3164200700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-04-20T22:36:38.027525+0000 front 2024-04-20T22:36:38.027687+0000 (oldest deadline 2024-04-20T22:37:02.727494+0000) 2024-04-20T22:37:15.711 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 22:37:15 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-5[76275]: 2024-04-20T22:37:15.050+0000 7f3164200700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.62:6814 osd.1 since back 2024-04-20T22:36:46.229550+0000 front 2024-04-20T22:36:46.229526+0000 (oldest deadline 2024-04-20T22:37:11.529195+0000) 2024-04-20T22:37:15.988 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 20 22:37:15 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-4[71392]: 2024-04-20T22:37:15.584+0000 7f2f33c00700 -1 osd.4 85 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-04-20T22:36:36.242121+0000 front 2024-04-20T22:36:36.242237+0000 (oldest deadline 2024-04-20T22:37:00.341764+0000) 2024-04-20T22:37:15.988 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 20 22:37:15 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-4[71392]: 2024-04-20T22:37:15.584+0000 7f2f33c00700 -1 osd.4 85 heartbeat_check: no reply from 172.21.15.62:6814 osd.1 since back 2024-04-20T22:36:46.545455+0000 front 2024-04-20T22:36:46.545774+0000 (oldest deadline 2024-04-20T22:37:10.645306+0000) 2024-04-20T22:37:16.488 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 22:37:16 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-5[76275]: 2024-04-20T22:37:16.050+0000 7f3164200700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-04-20T22:36:38.027525+0000 front 2024-04-20T22:36:38.027687+0000 (oldest deadline 2024-04-20T22:37:02.727494+0000) 2024-04-20T22:37:16.488 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 22:37:16 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-5[76275]: 2024-04-20T22:37:16.050+0000 7f3164200700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.62:6814 osd.1 since back 2024-04-20T22:36:46.229550+0000 front 2024-04-20T22:36:46.229526+0000 (oldest deadline 2024-04-20T22:37:11.529195+0000) 2024-04-20T22:37:16.988 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 20 22:37:16 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-4[71392]: 2024-04-20T22:37:16.595+0000 7f2f33c00700 -1 osd.4 85 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-04-20T22:36:36.242121+0000 front 2024-04-20T22:36:36.242237+0000 (oldest deadline 2024-04-20T22:37:00.341764+0000) 2024-04-20T22:37:16.988 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 20 22:37:16 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-4[71392]: 2024-04-20T22:37:16.595+0000 7f2f33c00700 -1 osd.4 85 heartbeat_check: no reply from 172.21.15.62:6814 osd.1 since back 2024-04-20T22:36:46.545455+0000 front 2024-04-20T22:36:46.545774+0000 (oldest deadline 2024-04-20T22:37:10.645306+0000) 2024-04-20T22:37:17.488 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 22:37:17 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-5[76275]: 2024-04-20T22:37:17.085+0000 7f3164200700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-04-20T22:36:38.027525+0000 front 2024-04-20T22:36:38.027687+0000 (oldest deadline 2024-04-20T22:37:02.727494+0000) 2024-04-20T22:37:17.488 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 22:37:17 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-5[76275]: 2024-04-20T22:37:17.085+0000 7f3164200700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.62:6814 osd.1 since back 2024-04-20T22:36:46.229550+0000 front 2024-04-20T22:36:46.229526+0000 (oldest deadline 2024-04-20T22:37:11.529195+0000) 2024-04-20T22:37:18.051 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 20 22:37:17 smithi189 podman[93459]: 2024-04-20 22:37:17.744362121 +0000 UTC m=+5.553716184 container died 653d8079967d3f53e729f038004fbcabe76e431cdaed1f685e973f1464f8534a (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-4, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, GIT_CLEAN=True) 2024-04-20T22:37:18.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 20 22:37:18 smithi189 podman[93459]: 2024-04-20 22:37:18.16306819 +0000 UTC m=+5.972422247 container cleanup 653d8079967d3f53e729f038004fbcabe76e431cdaed1f685e973f1464f8534a (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-4, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-04-20T22:37:18.461 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 20 22:37:18 smithi189 bash[93459]: ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-4 2024-04-20T22:37:18.461 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 22:37:18 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-5[76275]: 2024-04-20T22:37:18.076+0000 7f3164200700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-04-20T22:36:38.027525+0000 front 2024-04-20T22:36:38.027687+0000 (oldest deadline 2024-04-20T22:37:02.727494+0000) 2024-04-20T22:37:18.461 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 22:37:18 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-5[76275]: 2024-04-20T22:37:18.076+0000 7f3164200700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.62:6814 osd.1 since back 2024-04-20T22:36:46.229550+0000 front 2024-04-20T22:36:46.229526+0000 (oldest deadline 2024-04-20T22:37:11.529195+0000) 2024-04-20T22:37:18.738 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 20 22:37:18 smithi189 podman[93476]: 2024-04-20 22:37:18.497522969 +0000 UTC m=+0.736841769 container remove 653d8079967d3f53e729f038004fbcabe76e431cdaed1f685e973f1464f8534a (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-4, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-04-20T22:37:19.341 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 22:37:19 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-5[76275]: 2024-04-20T22:37:19.040+0000 7f3164200700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-04-20T22:36:38.027525+0000 front 2024-04-20T22:36:38.027687+0000 (oldest deadline 2024-04-20T22:37:02.727494+0000) 2024-04-20T22:37:19.341 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 22:37:19 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-5[76275]: 2024-04-20T22:37:19.040+0000 7f3164200700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.62:6814 osd.1 since back 2024-04-20T22:36:46.229550+0000 front 2024-04-20T22:36:46.229526+0000 (oldest deadline 2024-04-20T22:37:11.529195+0000) 2024-04-20T22:37:19.694 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 20 22:37:19 smithi189 podman[93563]: 2024-04-20 22:37:19.248002234 +0000 UTC m=+0.039523691 image pull 76bc6b91b726a03b8b736eeeb35fc1ffd1469e23c7d60f5b444a217528d8906d quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 2024-04-20T22:37:19.694 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 20 22:37:19 smithi189 podman[93563]: 2024-04-20 22:37:19.399398824 +0000 UTC m=+0.190920364 container create a6638a3803f1f4a733e7616283523d0c50503e21af03a43bdf15764901f39065 (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-4-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD) 2024-04-20T22:37:19.694 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 20 22:37:19 smithi189 podman[93563]: 2024-04-20 22:37:19.694128714 +0000 UTC m=+0.485650235 container init a6638a3803f1f4a733e7616283523d0c50503e21af03a43bdf15764901f39065 (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-4-deactivate, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-04-20T22:37:19.988 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 20 22:37:19 smithi189 podman[93563]: 2024-04-20 22:37:19.705459146 +0000 UTC m=+0.496980676 container start a6638a3803f1f4a733e7616283523d0c50503e21af03a43bdf15764901f39065 (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-04-20T22:37:19.988 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 20 22:37:19 smithi189 podman[93563]: 2024-04-20 22:37:19.792830369 +0000 UTC m=+0.584351926 container attach a6638a3803f1f4a733e7616283523d0c50503e21af03a43bdf15764901f39065 (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-20T22:37:20.488 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 22:37:19 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-5[76275]: 2024-04-20T22:37:19.997+0000 7f3164200700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-04-20T22:36:38.027525+0000 front 2024-04-20T22:36:38.027687+0000 (oldest deadline 2024-04-20T22:37:02.727494+0000) 2024-04-20T22:37:20.488 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 22:37:19 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-5[76275]: 2024-04-20T22:37:19.997+0000 7f3164200700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.62:6814 osd.1 since back 2024-04-20T22:36:46.229550+0000 front 2024-04-20T22:36:46.229526+0000 (oldest deadline 2024-04-20T22:37:11.529195+0000) 2024-04-20T22:37:20.489 INFO:journalctl@ceph.osd.4.smithi189.stdout:Apr 20 22:37:20 smithi189 podman[93588]: 2024-04-20 22:37:20.215737002 +0000 UTC m=+0.035744039 container died a6638a3803f1f4a733e7616283523d0c50503e21af03a43bdf15764901f39065 (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-4-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416) 2024-04-20T22:37:20.909 DEBUG:teuthology.orchestra.run.smithi189:> sudo pkill -f 'journalctl -f -n 0 -u ceph-61ae3628-ff5b-11ee-bc93-c7b262605968@osd.4.service' 2024-04-20T22:37:21.020 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-20T22:37:21.020 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-04-20T22:37:21.020 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-04-20T22:37:21.020 DEBUG:teuthology.orchestra.run.smithi189:> sudo systemctl stop ceph-61ae3628-ff5b-11ee-bc93-c7b262605968@osd.5 2024-04-20T22:37:21.134 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 22:37:21 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-5[76275]: 2024-04-20T22:37:21.046+0000 7f3164200700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-04-20T22:36:38.027525+0000 front 2024-04-20T22:36:38.027687+0000 (oldest deadline 2024-04-20T22:37:02.727494+0000) 2024-04-20T22:37:21.135 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 22:37:21 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-5[76275]: 2024-04-20T22:37:21.046+0000 7f3164200700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.62:6814 osd.1 since back 2024-04-20T22:36:46.229550+0000 front 2024-04-20T22:36:46.229526+0000 (oldest deadline 2024-04-20T22:37:11.529195+0000) 2024-04-20T22:37:21.135 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 22:37:21 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-5[76275]: 2024-04-20T22:37:21.046+0000 7f3164200700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.62:6822 osd.2 since back 2024-04-20T22:36:54.430813+0000 front 2024-04-20T22:36:54.430729+0000 (oldest deadline 2024-04-20T22:37:20.330306+0000) 2024-04-20T22:37:21.487 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 22:37:21 smithi189 systemd[1]: Stopping Ceph osd.5 for 61ae3628-ff5b-11ee-bc93-c7b262605968... 2024-04-20T22:37:21.988 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 22:37:21 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-5[76275]: 2024-04-20T22:37:21.646+0000 7f316da00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-20T22:37:21.988 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 22:37:21 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-5[76275]: 2024-04-20T22:37:21.646+0000 7f316da00700 -1 osd.5 85 *** Got signal Terminated *** 2024-04-20T22:37:21.988 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 22:37:21 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-5[76275]: 2024-04-20T22:37:21.646+0000 7f316da00700 -1 osd.5 85 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-20T22:37:22.488 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 22:37:22 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-5[76275]: 2024-04-20T22:37:22.002+0000 7f3164200700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-04-20T22:36:38.027525+0000 front 2024-04-20T22:36:38.027687+0000 (oldest deadline 2024-04-20T22:37:02.727494+0000) 2024-04-20T22:37:22.488 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 22:37:22 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-5[76275]: 2024-04-20T22:37:22.002+0000 7f3164200700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.62:6814 osd.1 since back 2024-04-20T22:36:46.229550+0000 front 2024-04-20T22:36:46.229526+0000 (oldest deadline 2024-04-20T22:37:11.529195+0000) 2024-04-20T22:37:22.488 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 22:37:22 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-5[76275]: 2024-04-20T22:37:22.002+0000 7f3164200700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.62:6822 osd.2 since back 2024-04-20T22:36:54.430813+0000 front 2024-04-20T22:36:54.430729+0000 (oldest deadline 2024-04-20T22:37:20.330306+0000) 2024-04-20T22:37:23.488 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 22:37:23 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-5[76275]: 2024-04-20T22:37:23.038+0000 7f3164200700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-04-20T22:36:38.027525+0000 front 2024-04-20T22:36:38.027687+0000 (oldest deadline 2024-04-20T22:37:02.727494+0000) 2024-04-20T22:37:23.488 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 22:37:23 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-5[76275]: 2024-04-20T22:37:23.038+0000 7f3164200700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.62:6814 osd.1 since back 2024-04-20T22:36:46.229550+0000 front 2024-04-20T22:36:46.229526+0000 (oldest deadline 2024-04-20T22:37:11.529195+0000) 2024-04-20T22:37:23.488 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 22:37:23 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-5[76275]: 2024-04-20T22:37:23.038+0000 7f3164200700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.62:6822 osd.2 since back 2024-04-20T22:36:54.430813+0000 front 2024-04-20T22:36:54.430729+0000 (oldest deadline 2024-04-20T22:37:20.330306+0000) 2024-04-20T22:37:24.488 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 22:37:24 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-5[76275]: 2024-04-20T22:37:24.080+0000 7f3164200700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-04-20T22:36:38.027525+0000 front 2024-04-20T22:36:38.027687+0000 (oldest deadline 2024-04-20T22:37:02.727494+0000) 2024-04-20T22:37:24.488 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 22:37:24 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-5[76275]: 2024-04-20T22:37:24.080+0000 7f3164200700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.62:6814 osd.1 since back 2024-04-20T22:36:46.229550+0000 front 2024-04-20T22:36:46.229526+0000 (oldest deadline 2024-04-20T22:37:11.529195+0000) 2024-04-20T22:37:24.488 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 22:37:24 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-5[76275]: 2024-04-20T22:37:24.080+0000 7f3164200700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.62:6822 osd.2 since back 2024-04-20T22:36:54.430813+0000 front 2024-04-20T22:36:54.430729+0000 (oldest deadline 2024-04-20T22:37:20.330306+0000) 2024-04-20T22:37:25.488 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 22:37:25 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-5[76275]: 2024-04-20T22:37:25.103+0000 7f3164200700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-04-20T22:36:38.027525+0000 front 2024-04-20T22:36:38.027687+0000 (oldest deadline 2024-04-20T22:37:02.727494+0000) 2024-04-20T22:37:25.488 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 22:37:25 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-5[76275]: 2024-04-20T22:37:25.103+0000 7f3164200700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.62:6814 osd.1 since back 2024-04-20T22:36:46.229550+0000 front 2024-04-20T22:36:46.229526+0000 (oldest deadline 2024-04-20T22:37:11.529195+0000) 2024-04-20T22:37:25.488 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 22:37:25 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-5[76275]: 2024-04-20T22:37:25.103+0000 7f3164200700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.62:6822 osd.2 since back 2024-04-20T22:36:54.430813+0000 front 2024-04-20T22:36:54.430729+0000 (oldest deadline 2024-04-20T22:37:20.330306+0000) 2024-04-20T22:37:26.488 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 22:37:26 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-5[76275]: 2024-04-20T22:37:26.070+0000 7f3164200700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-04-20T22:36:38.027525+0000 front 2024-04-20T22:36:38.027687+0000 (oldest deadline 2024-04-20T22:37:02.727494+0000) 2024-04-20T22:37:26.488 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 22:37:26 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-5[76275]: 2024-04-20T22:37:26.070+0000 7f3164200700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.62:6814 osd.1 since back 2024-04-20T22:36:46.229550+0000 front 2024-04-20T22:36:46.229526+0000 (oldest deadline 2024-04-20T22:37:11.529195+0000) 2024-04-20T22:37:26.488 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 22:37:26 smithi189 ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-5[76275]: 2024-04-20T22:37:26.070+0000 7f3164200700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.62:6822 osd.2 since back 2024-04-20T22:36:54.430813+0000 front 2024-04-20T22:36:54.430729+0000 (oldest deadline 2024-04-20T22:37:20.330306+0000) 2024-04-20T22:37:27.353 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 22:37:27 smithi189 podman[93667]: 2024-04-20 22:37:27.09078521 +0000 UTC m=+5.634779668 container died 596dded14c124b91e080c03569bfae21c361718e40c11237aac1d50c29eb3166 (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-20T22:37:27.693 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 22:37:27 smithi189 podman[93667]: 2024-04-20 22:37:27.473672724 +0000 UTC m=+6.017667243 container cleanup 596dded14c124b91e080c03569bfae21c361718e40c11237aac1d50c29eb3166 (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-5, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-04-20T22:37:27.693 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 22:37:27 smithi189 bash[93667]: ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-5 2024-04-20T22:37:27.988 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 22:37:27 smithi189 podman[93686]: 2024-04-20 22:37:27.691647604 +0000 UTC m=+0.586822572 container remove 596dded14c124b91e080c03569bfae21c361718e40c11237aac1d50c29eb3166 (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, maintainer=Guillaume Abrioux ) 2024-04-20T22:37:28.738 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 22:37:28 smithi189 podman[93774]: 2024-04-20 22:37:28.383609332 +0000 UTC m=+0.038881354 image pull 76bc6b91b726a03b8b736eeeb35fc1ffd1469e23c7d60f5b444a217528d8906d quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1 2024-04-20T22:37:28.738 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 22:37:28 smithi189 podman[93774]: 2024-04-20 22:37:28.517724739 +0000 UTC m=+0.172996818 container create eef93541c9ea365b39c0a9af6326f9fedb49cb1cf9783aab1cf8f1856a571ffd (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-5-deactivate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True) 2024-04-20T22:37:29.226 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 22:37:28 smithi189 podman[93774]: 2024-04-20 22:37:28.763316571 +0000 UTC m=+0.418588630 container init eef93541c9ea365b39c0a9af6326f9fedb49cb1cf9783aab1cf8f1856a571ffd (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-5-deactivate, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-04-20T22:37:29.226 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 22:37:28 smithi189 podman[93774]: 2024-04-20 22:37:28.774724516 +0000 UTC m=+0.429996565 container start eef93541c9ea365b39c0a9af6326f9fedb49cb1cf9783aab1cf8f1856a571ffd (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-5-deactivate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-20T22:37:29.226 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 22:37:28 smithi189 podman[93774]: 2024-04-20 22:37:28.853608472 +0000 UTC m=+0.508880577 container attach eef93541c9ea365b39c0a9af6326f9fedb49cb1cf9783aab1cf8f1856a571ffd (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-5-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, ceph=True, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-20T22:37:29.898 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 20 22:37:29 smithi189 podman[93774]: 2024-04-20 22:37:29.226725982 +0000 UTC m=+0.881998087 container died eef93541c9ea365b39c0a9af6326f9fedb49cb1cf9783aab1cf8f1856a571ffd (image=quay.ceph.io/ceph-ci/ceph@sha256:556b2d4db8c1fdd86d8938d44019b738fe91b025623acee0a51cae93523136b1, name=ceph-61ae3628-ff5b-11ee-bc93-c7b262605968-osd-5-deactivate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240416.193735-a8c84f5, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416) 2024-04-20T22:37:29.950 DEBUG:teuthology.orchestra.run.smithi189:> sudo pkill -f 'journalctl -f -n 0 -u ceph-61ae3628-ff5b-11ee-bc93-c7b262605968@osd.5.service' 2024-04-20T22:37:30.063 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-20T22:37:30.063 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-04-20T22:37:30.063 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 --force --keep-logs 2024-04-20T22:37:30.339 INFO:teuthology.orchestra.run.smithi062.stdout:Deleting cluster with fsid: 61ae3628-ff5b-11ee-bc93-c7b262605968 2024-04-20T22:37:37.602 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi062.stderr:ceph-fuse[54098]: fuse finished with error 0 and tester_r 0 2024-04-20T22:38:07.612 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 --force --keep-logs 2024-04-20T22:38:07.890 INFO:teuthology.orchestra.run.smithi189.stdout:Deleting cluster with fsid: 61ae3628-ff5b-11ee-bc93-c7b262605968 2024-04-20T22:38:26.552 DEBUG:teuthology.orchestra.run.smithi062:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-20T22:38:26.650 DEBUG:teuthology.orchestra.run.smithi189:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-20T22:38:26.740 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-20T22:38:26.742 DEBUG:teuthology.misc:Transferring archived files from smithi062:/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash to /home/teuthworker/archive/rishabh-2024-04-20_20:50:37-fs-wip-rishabh-testing-20240416.193735-testing-default-smithi/7665388/remote/smithi062/crash 2024-04-20T22:38:26.742 DEBUG:teuthology.orchestra.run.smithi062:> sudo tar c -f - -C /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash -- . 2024-04-20T22:38:26.832 INFO:teuthology.orchestra.run.smithi062.stderr:tar: /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-20T22:38:26.833 INFO:teuthology.orchestra.run.smithi062.stderr:tar: Error is not recoverable: exiting now 2024-04-20T22:38:26.839 DEBUG:teuthology.misc:Transferring archived files from smithi189:/var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash to /home/teuthworker/archive/rishabh-2024-04-20_20:50:37-fs-wip-rishabh-testing-20240416.193735-testing-default-smithi/7665388/remote/smithi189/crash 2024-04-20T22:38:26.840 DEBUG:teuthology.orchestra.run.smithi189:> sudo tar c -f - -C /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash -- . 2024-04-20T22:38:26.924 INFO:teuthology.orchestra.run.smithi189.stderr:tar: /var/lib/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-20T22:38:26.924 INFO:teuthology.orchestra.run.smithi189.stderr:tar: Error is not recoverable: exiting now 2024-04-20T22:38:26.928 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-20T22:38:26.928 DEBUG:teuthology.orchestra.run.smithi062:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v OSD_DOWN | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | head -n 1 2024-04-20T22:38:27.035 INFO:teuthology.orchestra.run.smithi062.stdout:2024-04-20T21:40:00.000493+0000 mon.smithi062 (mon.0) 394 : cluster [WRN] pg 2.1 is active+undersized+degraded, acting [1,0] 2024-04-20T22:38:27.036 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-04-20T22:38:27.036 INFO:tasks.cephadm:Compressing logs... 2024-04-20T22:38:27.036 DEBUG:teuthology.orchestra.run.smithi062:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-20T22:38:27.040 DEBUG:teuthology.orchestra.run.smithi189:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-20T22:38:27.101 INFO:teuthology.orchestra.run.smithi062.stderr:find: ‘/var/log/rbd-target-api’: No such file or directorygzip 2024-04-20T22:38:27.101 INFO:teuthology.orchestra.run.smithi062.stderr: -5 --verbose -- /var/log/ceph/ceph-client.0.log 2024-04-20T22:38:27.102 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-20T22:38:27.102 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/ceph-mgr.smithi062.rqxuxi.log 2024-04-20T22:38:27.103 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.0.log: /var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/ceph.log 2024-04-20T22:38:27.104 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/ceph-mgr.smithi062.rqxuxi.log: gzip -5 --verbose -- /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/ceph-osd.2.log 2024-04-20T22:38:27.106 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/ceph.audit.log 2024-04-20T22:38:27.107 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/ceph-mon.smithi062.log 2024-04-20T22:38:27.109 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/ceph-mds.cephfs.smithi062.ryqblb.log 2024-04-20T22:38:27.112 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/ceph-mon.smithi062.log: /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/ceph-mds.cephfs.smithi062.ryqblb.log: gzip -5 --verbose -- /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/ceph-mds.cephfs.smithi062.ozizwi.log 2024-04-20T22:38:27.117 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/ceph-mds.cephfs.smithi062.ozizwi.log: gzip -5 89.9% -- replaced with /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/ceph.log.gz 2024-04-20T22:38:27.117 INFO:teuthology.orchestra.run.smithi062.stderr: --verbose -- /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/ceph-osd.1.log 2024-04-20T22:38:27.118 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/ceph-osd.0.log 2024-04-20T22:38:27.120 INFO:teuthology.orchestra.run.smithi062.stderr: 91.3% -- replaced with /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/ceph.audit.log.gz 2024-04-20T22:38:27.124 INFO:teuthology.orchestra.run.smithi189.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-04-20T22:38:27.124 INFO:teuthology.orchestra.run.smithi189.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.log 2024-04-20T22:38:27.126 INFO:teuthology.orchestra.run.smithi189.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-20T22:38:27.127 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/ceph-client.1.log: gzip -5 --verbose -- /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/ceph.log 2024-04-20T22:38:27.127 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/ceph-osd.1.log: /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/ceph.cephadm.log 2024-04-20T22:38:27.128 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/ceph.audit.log 2024-04-20T22:38:27.129 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/ceph-client.ceph-exporter.smithi062.log 2024-04-20T22:38:27.129 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/ceph-mon.smithi189.log 2024-04-20T22:38:27.130 INFO:teuthology.orchestra.run.smithi189.stderr: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-20T22:38:27.130 INFO:teuthology.orchestra.run.smithi062.stderr: 91.5% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-20T22:38:27.131 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/ceph-mds.cephfs.smithi189.dvimer.log 2024-04-20T22:38:27.133 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/ceph-volume.log 2024-04-20T22:38:27.133 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/ceph-mon.smithi189.log: gzip -5 --verbose -- /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/ceph-osd.5.log 2024-04-20T22:38:27.134 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/ceph-mds.cephfs.smithi189.dvimer.log: gzip -5 --verbose -- /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/ceph-osd.3.log 2024-04-20T22:38:27.135 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/ceph-client.ceph-exporter.smithi062.log: /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/ceph.cephadm.log: 85.6% -- replaced with /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-20T22:38:27.136 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/ceph-mgr.smithi189.tawavz.log 2024-04-20T22:38:27.138 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/ceph-osd.3.log: 90.0% -- replaced with /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/ceph.log.gz 2024-04-20T22:38:27.139 INFO:teuthology.orchestra.run.smithi189.stderr:gzip -5 --verbose -- /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/ceph-osd.4.log 2024-04-20T22:38:27.140 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/ceph-mgr.smithi189.tawavz.log: 91.3% -- replaced with /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/ceph.audit.log.gz 2024-04-20T22:38:27.141 INFO:teuthology.orchestra.run.smithi189.stderr:gzip -5 --verbose -- /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/ceph.cephadm.log 2024-04-20T22:38:27.143 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/ceph-volume.log: 94.5% -- replaced with /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/ceph-client.ceph-exporter.smithi062.log.gz 2024-04-20T22:38:27.144 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/ceph-osd.4.log: /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/ceph-client.ceph-exporter.smithi189.log 2024-04-20T22:38:27.147 INFO:teuthology.orchestra.run.smithi189.stderr: 85.6%/var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/ceph-client.ceph-exporter.smithi189.log: gzip -5 --verbose -- /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/ceph-mds.cephfs.smithi189.xbgspw.log 2024-04-20T22:38:27.148 INFO:teuthology.orchestra.run.smithi189.stderr: -- replaced with /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-20T22:38:27.151 INFO:teuthology.orchestra.run.smithi189.stderr:gzip -5 --verbose -- /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/ceph-volume.log 2024-04-20T22:38:27.166 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/ceph-mds.cephfs.smithi189.xbgspw.log: /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/ceph-volume.log: 94.5% -- replaced with /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/ceph-client.ceph-exporter.smithi189.log.gz 2024-04-20T22:38:27.170 INFO:teuthology.orchestra.run.smithi062.stderr: 91.4% -- replaced with /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/ceph-volume.log.gz 2024-04-20T22:38:27.179 INFO:teuthology.orchestra.run.smithi189.stderr: 91.2% -- replaced with /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/ceph-volume.log.gz 2024-04-20T22:38:27.208 INFO:teuthology.orchestra.run.smithi189.stderr: 90.2% -- replaced with /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/ceph-mgr.smithi189.tawavz.log.gz 2024-04-20T22:38:28.023 INFO:teuthology.orchestra.run.smithi062.stderr: 89.7% -- replaced with /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/ceph-mgr.smithi062.rqxuxi.log.gz 2024-04-20T22:38:28.325 INFO:teuthology.orchestra.run.smithi189.stderr: 92.5% -- replaced with /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/ceph-mon.smithi189.log.gz 2024-04-20T22:38:30.549 INFO:teuthology.orchestra.run.smithi062.stderr: 89.6% -- replaced with /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/ceph-mon.smithi062.log.gz 2024-04-20T22:38:31.708 INFO:teuthology.orchestra.run.smithi189.stderr: 93.6% -- replaced with /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/ceph-osd.4.log.gz 2024-04-20T22:38:33.302 INFO:teuthology.orchestra.run.smithi062.stderr: 93.6% -- replaced with /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/ceph-osd.1.log.gz 2024-04-20T22:38:34.092 INFO:teuthology.orchestra.run.smithi062.stderr: 93.4% -- replaced with /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/ceph-osd.2.log.gz 2024-04-20T22:38:34.545 INFO:teuthology.orchestra.run.smithi062.stderr: 93.5% -- replaced with /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/ceph-osd.0.log.gz 2024-04-20T22:38:34.788 INFO:teuthology.orchestra.run.smithi189.stderr: 93.6% -- replaced with /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/ceph-osd.3.log.gz 2024-04-20T22:38:34.819 INFO:teuthology.orchestra.run.smithi189.stderr: 93.7% -- replaced with /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/ceph-osd.5.log.gz 2024-04-20T22:38:36.159 INFO:teuthology.orchestra.run.smithi189.stderr: 95.0% -- replaced with /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/ceph-mds.cephfs.smithi189.xbgspw.log.gz 2024-04-20T22:38:36.956 INFO:teuthology.orchestra.run.smithi062.stderr: 95.0% -- replaced with /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/ceph-mds.cephfs.smithi062.ozizwi.log.gz 2024-04-20T22:38:43.352 INFO:teuthology.orchestra.run.smithi189.stderr:gzip: /var/log/ceph/ceph-client.1.log: file size changed while zipping 2024-04-20T22:38:43.352 INFO:teuthology.orchestra.run.smithi189.stderr: 93.0% -- replaced with /var/log/ceph/ceph-client.1.log.gz 2024-04-20T22:38:44.042 INFO:teuthology.orchestra.run.smithi062.stderr: 93.0% -- replaced with /var/log/ceph/ceph-client.0.log.gz 2024-04-20T22:39:11.529 INFO:teuthology.orchestra.run.smithi189.stderr: 93.5% -- replaced with /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/ceph-mds.cephfs.smithi189.dvimer.log.gz 2024-04-20T22:39:11.533 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-04-20T22:39:11.533 INFO:teuthology.orchestra.run.smithi189.stderr:real 0m44.439s 2024-04-20T22:39:11.533 INFO:teuthology.orchestra.run.smithi189.stderr:user 1m24.558s 2024-04-20T22:39:11.534 INFO:teuthology.orchestra.run.smithi189.stderr:sys 0m6.239s 2024-04-20T22:39:30.017 INFO:teuthology.orchestra.run.smithi062.stderr: 93.3% -- replaced with /var/log/ceph/61ae3628-ff5b-11ee-bc93-c7b262605968/ceph-mds.cephfs.smithi062.ryqblb.log.gz 2024-04-20T22:39:30.021 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-20T22:39:30.021 INFO:teuthology.orchestra.run.smithi062.stderr:real 1m2.946s 2024-04-20T22:39:30.021 INFO:teuthology.orchestra.run.smithi062.stderr:user 1m46.842s 2024-04-20T22:39:30.021 INFO:teuthology.orchestra.run.smithi062.stderr:sys 0m7.838s 2024-04-20T22:39:30.022 INFO:tasks.cephadm:Archiving logs... 2024-04-20T22:39:30.023 DEBUG:teuthology.misc:Transferring archived files from smithi062:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-04-20_20:50:37-fs-wip-rishabh-testing-20240416.193735-testing-default-smithi/7665388/remote/smithi062/log 2024-04-20T22:39:30.023 DEBUG:teuthology.orchestra.run.smithi062:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-20T22:39:44.067 DEBUG:teuthology.misc:Transferring archived files from smithi189:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-04-20_20:50:37-fs-wip-rishabh-testing-20240416.193735-testing-default-smithi/7665388/remote/smithi189/log 2024-04-20T22:39:44.068 DEBUG:teuthology.orchestra.run.smithi189:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-20T22:39:55.049 INFO:tasks.cephadm:Removing cluster... 2024-04-20T22:39:55.049 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 --force 2024-04-20T22:39:55.339 INFO:teuthology.orchestra.run.smithi062.stdout:Deleting cluster with fsid: 61ae3628-ff5b-11ee-bc93-c7b262605968 2024-04-20T22:39:56.275 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 61ae3628-ff5b-11ee-bc93-c7b262605968 --force 2024-04-20T22:39:56.550 INFO:teuthology.orchestra.run.smithi189.stdout:Deleting cluster with fsid: 61ae3628-ff5b-11ee-bc93-c7b262605968 2024-04-20T22:39:57.430 INFO:tasks.cephadm:Removing cephadm ... 2024-04-20T22:39:57.431 DEBUG:teuthology.orchestra.run.smithi062:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-20T22:39:57.493 DEBUG:teuthology.orchestra.run.smithi189:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-20T22:39:57.550 INFO:tasks.cephadm:Teardown complete 2024-04-20T22:39:57.550 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-20T22:39:57.561 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_a8c84f5997950fd7d008204839253291d52e6eda/qa/tasks/ceph_fuse.py", line 183, in task mount.umount_wait() File "/home/teuthworker/src/git.ceph.com_ceph-c_a8c84f5997950fd7d008204839253291d52e6eda/qa/tasks/cephfs/fuse_mount.py", line 403, in umount_wait run.wait([self.fuse_daemon], timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2024-04-20T22:39:57.561 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-20T22:39:57.561 DEBUG:teuthology.orchestra.run.smithi062:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-20T22:39:57.569 DEBUG:teuthology.orchestra.run.smithi189:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-20T22:39:57.698 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-20T22:39:57.709 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-20T22:39:57.709 DEBUG:teuthology.orchestra.run.smithi062:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-20T22:39:57.717 DEBUG:teuthology.orchestra.run.smithi189:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-20T22:39:57.767 INFO:teuthology.orchestra.run.smithi062.stderr:bash: line 1: ntpq: command not found 2024-04-20T22:39:57.770 INFO:teuthology.orchestra.run.smithi189.stderr:bash: line 1: ntpq: command not found 2024-04-20T22:39:57.777 INFO:teuthology.orchestra.run.smithi062.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-20T22:39:57.777 INFO:teuthology.orchestra.run.smithi062.stdout:=============================================================================== 2024-04-20T22:39:57.778 INFO:teuthology.orchestra.run.smithi062.stdout:^+ hv01.front.sepia.ceph.com 2 8 377 141 -633us[ -732us] +/- 80ms 2024-04-20T22:39:57.778 INFO:teuthology.orchestra.run.smithi062.stdout:^* hv02.front.sepia.ceph.com 3 7 377 12 -598us[ -686us] +/- 58ms 2024-04-20T22:39:57.778 INFO:teuthology.orchestra.run.smithi062.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 10 +1142us[+1142us] +/- 69ms 2024-04-20T22:39:57.778 INFO:teuthology.orchestra.run.smithi062.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-04-20T22:39:57.780 INFO:teuthology.orchestra.run.smithi189.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-20T22:39:57.780 INFO:teuthology.orchestra.run.smithi189.stdout:=============================================================================== 2024-04-20T22:39:57.780 INFO:teuthology.orchestra.run.smithi189.stdout:^+ hv01.front.sepia.ceph.com 2 8 377 71 -686us[ -686us] +/- 81ms 2024-04-20T22:39:57.781 INFO:teuthology.orchestra.run.smithi189.stdout:^* hv02.front.sepia.ceph.com 3 9 377 204 -608us[ -671us] +/- 55ms 2024-04-20T22:39:57.781 INFO:teuthology.orchestra.run.smithi189.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 6 +1081us[+1081us] +/- 69ms 2024-04-20T22:39:57.781 INFO:teuthology.orchestra.run.smithi189.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-04-20T22:39:57.782 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-20T22:39:57.793 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-20T22:39:57.794 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-20T22:39:57.804 DEBUG:teuthology.orchestra.run.smithi062:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-20T22:39:57.992 DEBUG:teuthology.orchestra.run.smithi189:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-20T22:39:58.158 DEBUG:teuthology.orchestra.run.smithi062:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-20T22:39:58.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-20T22:39:58.256 DEBUG:teuthology.orchestra.run.smithi189:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-20T22:39:58.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-20T22:39:58.348 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-20T22:39:58.359 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-20T22:39:58.385 INFO:teuthology.task.internal:Duration was 5269.396730 seconds 2024-04-20T22:39:58.385 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-20T22:39:58.396 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-20T22:39:58.396 DEBUG:teuthology.orchestra.run.smithi062:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-20T22:39:58.404 DEBUG:teuthology.orchestra.run.smithi189:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-20T22:39:58.539 INFO:teuthology.orchestra.run.smithi062.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-20T22:39:58.548 INFO:teuthology.orchestra.run.smithi189.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-20T22:39:58.878 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-20T22:39:58.878 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi062.front.sepia.ceph.com 2024-04-20T22:39:58.878 DEBUG:teuthology.orchestra.run.smithi062:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-20T22:39:58.946 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi189.front.sepia.ceph.com 2024-04-20T22:39:58.946 DEBUG:teuthology.orchestra.run.smithi189:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-20T22:39:59.018 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-20T22:39:59.018 DEBUG:teuthology.orchestra.run.smithi062:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-20T22:39:59.025 DEBUG:teuthology.orchestra.run.smithi189:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-20T22:39:59.110 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-20T22:39:59.110 DEBUG:teuthology.orchestra.run.smithi062:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-20T22:39:59.115 DEBUG:teuthology.orchestra.run.smithi189:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-20T22:40:00.753 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-20T22:40:00.764 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-20T22:40:00.764 DEBUG:teuthology.orchestra.run.smithi062:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-20T22:40:00.841 DEBUG:teuthology.orchestra.run.smithi189:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-20T22:40:00.930 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-20T22:40:00.943 DEBUG:teuthology.orchestra.run.smithi062:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-20T22:40:00.951 DEBUG:teuthology.orchestra.run.smithi189:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-20T22:40:01.042 INFO:teuthology.orchestra.run.smithi189.stdout:kernel.core_pattern = core 2024-04-20T22:40:01.046 INFO:teuthology.orchestra.run.smithi062.stdout:kernel.core_pattern = core 2024-04-20T22:40:01.113 DEBUG:teuthology.orchestra.run.smithi062:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-20T22:40:01.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-20T22:40:01.166 DEBUG:teuthology.orchestra.run.smithi189:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-20T22:40:01.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-20T22:40:01.219 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-20T22:40:01.231 INFO:teuthology.task.internal:Transferring archived files... 2024-04-20T22:40:01.232 DEBUG:teuthology.misc:Transferring archived files from smithi062:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-04-20_20:50:37-fs-wip-rishabh-testing-20240416.193735-testing-default-smithi/7665388/remote/smithi062 2024-04-20T22:40:01.232 DEBUG:teuthology.orchestra.run.smithi062:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-20T22:40:01.361 DEBUG:teuthology.misc:Transferring archived files from smithi189:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-04-20_20:50:37-fs-wip-rishabh-testing-20240416.193735-testing-default-smithi/7665388/remote/smithi189 2024-04-20T22:40:01.361 DEBUG:teuthology.orchestra.run.smithi189:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-20T22:40:01.459 INFO:teuthology.task.internal:Removing archive directory... 2024-04-20T22:40:01.459 DEBUG:teuthology.orchestra.run.smithi062:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-20T22:40:01.468 DEBUG:teuthology.orchestra.run.smithi189:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-20T22:40:01.525 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-20T22:40:01.538 INFO:teuthology.task.internal:Not uploading archives. 2024-04-20T22:40:01.538 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-20T22:40:01.586 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-20T22:40:01.586 DEBUG:teuthology.orchestra.run.smithi062:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-20T22:40:01.594 DEBUG:teuthology.orchestra.run.smithi189:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-20T22:40:01.641 INFO:teuthology.orchestra.run.smithi062.stdout: 393229 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Apr 20 22:40 /home/ubuntu/cephtest 2024-04-20T22:40:01.641 INFO:teuthology.orchestra.run.smithi062.stdout: 685431 4 d--------- 2 ubuntu ubuntu 4096 Apr 20 21:28 /home/ubuntu/cephtest/mnt.0 2024-04-20T22:40:01.642 INFO:teuthology.orchestra.run.smithi062.stderr:find: ‘/home/ubuntu/cephtest/mnt.0’: Permission denied 2024-04-20T22:40:01.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-20T22:40:01.644 INFO:teuthology.orchestra.run.smithi062.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-04-20T22:40:01.644 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_a8c84f5997950fd7d008204839253291d52e6eda/qa/tasks/ceph_fuse.py", line 183, in task mount.umount_wait() File "/home/teuthworker/src/git.ceph.com_ceph-c_a8c84f5997950fd7d008204839253291d52e6eda/qa/tasks/cephfs/fuse_mount.py", line 403, in umount_wait run.wait([self.fuse_daemon], timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi062 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-04-20T22:40:01.645 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-20T22:40:01.658 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-20T22:40:01.671 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2024-04-20T22:40:01.671 INFO:teuthology.nuke:Checking targets against current locks 2024-04-20T22:40:01.694 DEBUG:teuthology.nuke:shortname: smithi062 2024-04-20T22:40:01.695 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-20T22:40:01.719 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi062.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-04-20_20:50:37-fs-wip-rishabh-testing-20240416.193735-testing-default-smithi/7665388', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-20 20:51:38.764805', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFWpe9SJsDurBtnuJZcMu88UOJClsGajFv7glq/3P42ZmjzSSU8PSr3FioBnSz6RNYQ5UHRSycxlgSv0tNeaubI='} 2024-04-20T22:40:01.723 DEBUG:teuthology.nuke:shortname: smithi189 2024-04-20T22:40:01.723 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-20T22:40:01.745 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi189.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-04-20_20:50:37-fs-wip-rishabh-testing-20240416.193735-testing-default-smithi/7665388', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-20 20:51:38.769118', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBG1JxUynIoCSQeK70sAnxX+sG5vDLI88pwfpaS1csAIL2TcSKN7hwWd/eHA8/2yTiI7gnIpEtIoQkvwNeb5Rjks='} 2024-04-20T22:40:01.773 INFO:teuthology.orchestra.console.smithi062:Power off 2024-04-20T22:40:01.773 DEBUG:teuthology.orchestra.console.smithi062:pexpect command: ipmitool -H smithi062.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-20T22:40:01.794 DEBUG:teuthology.orchestra.console.smithi062:power off output: Chassis Power Control: Down/Off 2024-04-20T22:40:01.794 DEBUG:teuthology.orchestra.console.smithi062:pexpect command: ipmitool -H smithi062.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-20T22:40:01.810 INFO:teuthology.orchestra.console.smithi189:Power off 2024-04-20T22:40:01.810 DEBUG:teuthology.orchestra.console.smithi189:pexpect command: ipmitool -H smithi189.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-20T22:40:01.821 DEBUG:teuthology.orchestra.console.smithi062:check power output: Chassis Power is on 2024-04-20T22:40:01.832 DEBUG:teuthology.orchestra.console.smithi189:power off output: Chassis Power Control: Down/Off 2024-04-20T22:40:01.832 DEBUG:teuthology.orchestra.console.smithi189:pexpect command: ipmitool -H smithi189.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-20T22:40:01.854 DEBUG:teuthology.orchestra.console.smithi189:check power output: Chassis Power is on 2024-04-20T22:40:05.822 DEBUG:teuthology.orchestra.console.smithi062:pexpect command: ipmitool -H smithi062.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-20T22:40:05.854 DEBUG:teuthology.orchestra.console.smithi189:pexpect command: ipmitool -H smithi189.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-20T22:40:05.933 DEBUG:teuthology.orchestra.console.smithi062:check power output: Chassis Power is on 2024-04-20T22:40:05.963 DEBUG:teuthology.orchestra.console.smithi189:check power output: Chassis Power is on 2024-04-20T22:40:09.936 DEBUG:teuthology.orchestra.console.smithi062:pexpect command: ipmitool -H smithi062.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-20T22:40:09.964 DEBUG:teuthology.orchestra.console.smithi189:pexpect command: ipmitool -H smithi189.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-20T22:40:10.046 DEBUG:teuthology.orchestra.console.smithi062:check power output: Chassis Power is off 2024-04-20T22:40:10.073 DEBUG:teuthology.orchestra.console.smithi189:check power output: Chassis Power is off 2024-04-20T22:40:10.147 INFO:teuthology.orchestra.console.smithi062:Power off completed 2024-04-20T22:40:10.174 INFO:teuthology.orchestra.console.smithi189:Power off completed 2024-04-20T22:40:10.317 INFO:teuthology.run:Summary data: description: fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} duration: 5269.396730422974 failure_reason: reached maximum tries (51) after waiting for 300 seconds flavor: default host.a-kernel-sha1: 806ec36b21129f113ce356c007dd59694d340b37 host.b-kernel-sha1: 806ec36b21129f113ce356c007dd59694d340b37 owner: scheduled_rishabh@teuthology status: fail success: false 2024-04-20T22:40:10.317 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-20T22:40:10.405 INFO:teuthology.run:FAIL